microsoft word news_eblip4.doc evidence based library and information practice 2006, 1:4  81 evidence based library and information practice     news      submission deadline for eblip4 extended to jan 8, 2007        © 2006 eblip. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    the evidence based library and  information practice conference (eblip4)  is an exciting international event that has  emerged in response to the growing interest  among all types of libraries in using the best  available research‐based evidence to  improve information practice. the  conference on may 6‐9, 2007 in chapel hill,  north carolina will be followed by two  days of ce. the conference provides a  forum for the presentation of high quality  papers and posters as well as examples of  how eblip is being implemented in library  and information settings around the globe.  eblip4 invites submissions for contributed  papers and posters including both original  research and innovative applications of  eblip in library and information  management. papers that deal with  library support of evidence‐based practice in  other fields such as health, social work and  public policy are also welcome. additional  info may be found at www.eblip4.unc.edu  important dates    january 8, 2006  submission deadline for abstracts for papers  and posters    february 11, 2007  final decisions for accepted papers     february 15, 2007  final decisions for accepted posters     march 15, 2007  submission deadline for full papers    if you have previously experience  difficulties in submitting an abstract please  try again using the new address provided.  should you have any difficulties, please  contact carol perryman, eblip4 co‐chair,  at: cp1757@gmail.com.   http://creativecommons.org/licenses/by/2.0 microsoft word news8_1498_final evidence based library and information practice 2008, 3:1  93 evidence based library and information practice       news     online tutorials for librarians interested in systematic reviews      © 2008. this is an open access article distributed under the terms of the creative commons attribution  license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and  reproduction in any medium, provided the original work is properly cited.       the uk higher education academy has  commissioned and made available four  online modules for librarians interested in  undertaking systematic reviews.    the units are aimed at undergraduates in  library and information studies preparing  for their final project or dissertation.  postgraduates in library and information  science should also find the materials  relevant to their research training.  using  examples from the library and information  science literature the modules take the user  through topics needed to carry out a  systematic review including: what is a  systematic review, formulating searches for  research evidence, producing a systematic  review (sifting and grading evidence) and  meta‐analysis, meta‐synthesis and  guidelines. the units complement, but do  not replace existing research methods  modules.  authored by dr. christine  urquhart, alison yeoman and dina  tbaishat from the university of  aberystwyth, uk, and alison brettle from  the university of salford, uk, the modules  are available online or to download from   http://www.ics.heacademy.ac.uk/resources/r los/systematic_review/.         news/announcements   carroll preston baber research grant call for proposals      2015. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   do you have a project that is just waiting for the right funding?  are you thinking about ways that libraries can improve services to users?   the american library association (ala) gives an annual grant for those conducting research that will lead to the improvement of services to users.  the carroll preston baber research grant is given to one or more librarians or library educators who will conduct innovative research that could lead to an improvement in services to any specified group of people.   the grant, up to $3,000, will be given to a proposed project that aims to answer a question of vital importance to the library community that is national in scope. among the review panel criteria are:   ·         the research problem is clearly defined, with a specific question or questions that can be answered by collecting data. the applicant(s) clearly describe a strategy for data collection whose methods are appropriate to the research question(s). a review of the literature, methodologies, etc. is not considered research (e.g., methodology review rather than application of a methodology) for purposes of the award, except where the literature review is the primary method of collecting data. ·         the research question focuses on benefits to library users and should be applied and have practical value as opposed to theoretical. ·         the applicant(s) demonstrate ability to undertake and successfully complete the project. the application provides evidence that sufficient time and resources have been allocated to the effort. appropriate institutional commitment to the project has been secured.   any ala member may apply, and the jury would welcome projects that involve both a practicing librarian and a researcher.   deadline is january 15, 2016.   procedures and an application form are available at http://www.ala.org/offices/ors/orsawards/baberresearchgrant/babercarroll.  see the section on how to apply.   also see related documents linked near the bottom of the page for:   ·         schedule and procedures http://www.ala.org/offices/ors/orsawards/baberresearchgrant/schedandprocedures   ·         proposal requirements and application cover sheet: http://www.ala.org/offices/ors/orsawards/baberresearchgrant/requirements   the full press release is available at: http://www.ala.org/news/baber-submissions-2016     questions?   contact billie peterson-lugo, baylor university, at billie_peterson@baylor.edu.   microsoft word news1_1469_final evidence based library and information practice 2008, 3:1  84 evidence based library and information practice       news     eblig sponsors cla 2008 pre‐conference workshop    © 2008. this is an open access article distributed under the terms of the creative commons attribution  license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and  reproduction in any medium, provided the original work is properly cited.       the evidence-based librarianship interest group (eblig) is pleased to offer a pre-conference session at cla in vancouver. the conference website http://www.cla.ca/conference/2008/ has been updated with program information. reading between the lines: how to study a paper whether you are using an evidencebased model of practice, participating in a journal club, acting as a peer reviewer, or are a regular reader of research articles, learning how to critically evaluate a paper is an essential skill. critical evaluation requires the reader to ask specific questions regarding the research methodology, data analysis and the presentation of results. these questions will be identified and discussed in this workshop, as several evaluation tools will be presented. participants will work in groups in which they will critically evaluate a research article and present their findings. this lively, hands-on workshop will provide participants with the necessary tools to approach and challenge research with inquisitiveness. speaker: lindsay glynn acting head, public services health sciences library memorial university of newfoundland registration fees: cla eblig members: $90.00 cla members: $100.00 non-members: $120.00 1:00 pm 5:00 pm wed., may 21st, 2008 session is limited to 35 delegates. registration includes: one coffee break. this session is organized by: evidence-based librarianship interest group (eblig) of cla   ebl 101   evaluating the results of evidence application, part two: at the practice level   virginia wilson client services librarian murray library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   originally published in: evidence based library and information practice, 5(4), 130–131. https://ejournals.library.ualberta.ca/index.php/eblip/article/view/9388/7539     received: 01 nov. 2010 accepted: 02 nov. 2010      2016 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   evaluation after implementation of evidence is a step that can be easily overlooked. however, its importance cannot be overestimated. last time i wrote about evaluation at the practitioner level, or reflection on your own performance as an evidence based practitioner. this time, it is evaluation at the practice level—you want to discover if, as booth puts it, “the service that you introduced or modified as a result of undertaking the evidence-based process actually made the anticipated difference” (p. 127). evaluation will do one of two things: confirm that the actions taken had the anticipated effect, or it may lead you to rethink your original issue. either way, evaluation can generate valuable information for your practice.   evaluation or assessment is not exclusive to evidence based library and information practice.   many libraries are cultivating a culture of assessment. there is a conference that focuses on assessment (library assessment conference, http://libraryassessment.org), academic libraries are using tools such as libqual+ (http://library.queensu.ca/webir/canlibqual/carl-libqual.htm) for assessment activities, and there are even assessment librarians in place in some libraries. the value of looking inward at the overall practice of the institution, in whatever library sector you may be situated, is high. while libqual+ is a massive assessment tool, evaluation can be done on a much smaller scale for individual evidence based projects.   ideally, plans for evaluation are included at the beginning of an evidence based project, or at least at the start of the implementation of whatever change is being made. the exact steps to be taken to perform the evaluation will vary depending on the scope of the project and what changes are occurring or being evaluated. evaluation for a newly implemented reference model will be different than evaluation of a new instructional design approach. evaluation that is undertaken as the project is ongoing is referred to as formative evaluation, and focuses on the process. summative evaluation, undertaken at the end of a project, illuminates the ultimate effectiveness of the implementation. both can be helpful when looking at an evidence based project.   there are several questions to ask when thinking about evaluation:   exactly what is being evaluated? how will it be measured? how will the data be collected and analyzed? what indicates improvement or success? what are the goals to be accomplished? what is the timeline for the evaluation? who will be involved in the evaluation process?   there are many different methods of evaluation, depending on what you are attempting to evaluate and what data you need to collect. approaches such as focus groups, interviews, surveys, questionnaires, and observation are all ways to assess whether or not your changes have achieved the desired effect.   while it is far beyond the scope of this column to explore the explicit details of these various kinds of evaluation, there are many resources out there to help you out. here is a brief bibliography to get you started.   crawford, j. (2006). the culture of evaluation in library and information services. oxford, uk: chandos.   mathison, s. (ed.). (2005). encyclopedia of evaluation. london: sage.   matthews, j.r. (2007). the evaluation and measurement of library services. westport.   ct.: libraries unlimited.   wallace, d.p. & van fleet, c. (2001). library evaluation: a casebook and can-do guide. englewood, co: libraries unlimited.   in practical terms, evaluating the implementation of evidence from the perspective of practice involves undertaking the steps of evidence based library and information practice again: formulate a question (decide what it is you are evaluating or measuring), find the evidence (in this case, you will be generating your own evidence by evaluating the changes made), appraise the evidence (this will involve making sure your evaluation methods and techniques are sound), and apply the evidence (by holding it up to your indicators of success). in terms of the evaluation step in this instance, you can be reflective and evaluate your own work as an evidence based practitioner.   as with anything in the evidence based process, starting small can help overcome feeling overwhelmed. eventually, the process will become more streamlined and easier to manage. consulting with colleagues as well as the literature can help to get you going. all you need to do is take that first step.   next time in ebl101, i will take a look at the process of disseminating your research. you have done an evidence based project, now what? get it out there to help others in the library world.   reference   booth, a. (2004). evaluating your performance. in a. booth & a. brice (eds.), evidence–based practice for information professionals: a handbook (pp. 127-137). london: facet.   microsoft word es 1326_haley_final evidence based library and information practice 2008, 3:2 34 evidence based library and information practice evidence summary canadian research librarians have little time for scholarship a review of: fox, david. “finding time for scholarship: a survey of canadian research university librarians.” portal: libraries and the academy 7.4 (2007): 451-62. reviewed by: pamela haley manager, library services, stormont, dundas & glengarry county library cornwall, ontario, canada e-mail: phaley@sdglibrary.ca received: 10 february 2008 accepted: 25 april 2008 © 2008 haley. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – to provide comparative data from canadian research universities regarding the time spent on scholarly activities by research librarians. design – qualitative study employing a bilingual survey consisting of thirty-nine questions. setting – canadian association of research libraries (carl) located at the twentyseven carl universities during the spring and fall of 2006. subjects – carl university librarians for whom e-mail addresses could be retrieved. methods – the survey was distributed to 1052 carl librarians during the spring and fall of 2006 via e-mail. problems with the clarity of two questions became evident during the receipt of responses. the questions were revised and resubmitted to the same population. main results – five hundred and twenty responses (49.4%) were received, with 441 (84.8%) in english and 15.2% in french. a total of 53 surveys were unusable, leaving 467 (44.4%) cases as the basis for data analysis. responses to the survey revealed that 51.4% of participants were required or encouraged to undertake scholarship. of these, 35% were expected, in addition to sabbatical and study leaves, to make scholarship an integral and ongoing part of their professional responsibilities. due to the individualized and subjective nature of the responses, no clear data emerged on the balance between scholarship and other evidence based library and information practice 2008, 3:2 35 professional activities. the majority of research librarians, on average, spent less than five hours per week on scholarship activities. for the 290 full-time librarians surveyed, the average time spent per week on all activities (professional and scholarly) was 47.4 hours. almost one third of the full time librarians worked fifty hours or more per week – the equivalent to the time commitment of the average university professor. participants indicated that in an ideal world they would like to spend 10% less time on assigned duties. francophone respondents spent 5% more of their time on professional responsibilities and 5% less time on scholarship. participation rates in scholarship related leaves are low, with less than 25% of those surveyed engaging in these opportunities. conclusion – based on the study’s findings, research librarians are not participating in scholarship to any great degree due to the perceived lack of time. commentary this paper just touches the surface of what could be an interesting analysis between expectations and delivery. it is felt that the author’s superficial reporting and analysis detracts from the scholarly nature of the work. the author states that “this paper will attempt to provide some comparative data from a canadian research university library perspective,” (451) but discusses only a limited number of the survey findings. to be “comparative” implies that one set of the data is compared to another set of data. given the introduction to the paper, one expects some comparison with jeanne brown’s work as well as to data about scholarship activities undertaken by university professors. unfortunately, this level of analysis is missing from the article. evaluation of the survey instrument itself is difficult as very little information is provided. it would have been helpful had the full survey and attendant results been included. since this was not done, one is not sure that the results have not been cherrypicked to support the thesis. nor is it clear what method of analysis was used. the author highlights the university of saskatchewan as a model for encouraging scholarship. it would have been more interesting if the policies of the other carl libraries had been presented and these then compared with the responses from the librarians at each university. indeed, a breakdown of responses by university may well have revealed some interesting information. reference is made to the average number of hours a university professor works without any data/source to back this up. insufficient data was proffered to support the claim that “the requirement for formal scholarship by canadian research university librarians appears to be a growing trend” (452). terminology is not always clear, for instance, there is no mention as to what “8r” refers to. a footnote is certainly warranted. in figure 5 an explanation of what “other” includes is also lacking. the author implies that the uptake for scholarly leaves is low because of perceived lack of time. his focus on this perception results in a narrowness in the analysis of the results, leaving many questions unanswered. for instance, what degree of real support is given to librarians as opposed to token support? a survey of administrators’ perceptions would supplement the thesis of the article. also, the differences within the carl libraries are delineated for the reader. yet, no analysis is made to compare responses to library “type.” evidence based library and information practice 2008, 3:2 36 while this paper is weak in a number of areas, it is a valuable first attempt to investigate a level of scholarship amongst professional staff in university libraries. a more in-depth, analytical research paper would certainly be worthwhile. evidence summary   cataloguing remains an important skill at public libraries in the modern metadata landscape of norway   a review of: preminger, m., rype, i., ådland, m.k., massey, d., & tallerås, k. (2020). the public library metadata landscape, the case of norway 2017–2018. cataloging & classification quarterly, 58(2), 127–148. https://doi.org/10.1080/01639374.2020.1711836   reviewed by: jordan patterson cataloging and metadata librarian memorial university of newfoundland st. john’s, newfoundland and labrador, canada email: jpatterson13@mun.ca   received: 1 june 2020                                                               accepted:  17 july 2020      2020 patterson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29788     abstract   objective – to understand cataloguing practices in norwegian public libraries through the analysis of a set of marc records.   design – quantitative content analysis.   setting – 2 central cataloguing agencies and 49 public libraries in norway.   subjects – 21,275 cataloguing agency records and 116,029 public library catalogue records.   methods – the researchers derived a sample set of marc records from the central cataloguing agencies and public libraries. matching records from each agency (i.e., records for the same manifestation catalogued separately at each agency) were compared. then, marc records exported from public libraries were compared to matching records from the central agencies.   main results – the two central agencies differed in some cataloguing practices while still adhering to the accepted standards. public libraries made few changes to records imported from central libraries, and among public libraries, larger libraries were more likely to alter agency-derived marc records.   conclusion – current practices indicate that despite the prevalence and efficiency of centralized cataloguing, training in cataloguing remains important in public libraries, particularly in larger libraries.   commentary   the late age of marc, when rda is under constant revision and bibframe is not yet ready for implementation, is the perfect time for cataloguing librarians to cast a critical eye upon their role in libraries. this is especially the case with regard to preparing future cataloguers to enter the complex, shifting world of library metadata. a broad assessment of current practices, such as that undertaken in this study, can provide a better understanding of the cataloguing needs of public libraries and thus inform the training requirements of the next generation of cataloguers and cataloguing librarians.   when assessed with glynn’s critical appraisal tool (2006), this study achieves a standard of validity. within the boundaries of norway, the researchers obtained a representative sample of cataloguing work by collecting records from both central cataloguing agencies and a fair cross-section of public libraries of varying sizes. the researchers noted that they had to modify their sample late in the process due to the revelation that not all libraries received full records from the central agency, resulting in a smaller sample size.   the researchers were more confident in their analysis and comparison of records between the two central agencies. in their dataset, they found 5,815 “directly comparable” pairs of records (p. 133). this one-to-one correspondence presented a relatively clear pattern of difference in cataloguing practices between the two agencies as well as a clear idea of the origin of that difference. when the study proceeded to the analysis of public library records, derived from a sample of 49 catalogues, the researchers were clear that they were less confident about the results because of the necessarily smaller-than-intended sample size.   while agency records are, theoretically, original creations conforming to stringent standards and produced in controlled environments, public library records have a larger number of potential sources and, through transmission, sources of interference. where the researchers detected differences between records from the public library and those of the central agencies, they were limited to stating that differences exist and what those differences were. the motivations for those changes necessarily remain speculative, although the researchers made educated hypotheses. the ambitious scope of the present study and its high altitude did not lend themselves to the kind of detailed scrutiny of complex bibliographic records the researchers aimed to perform. such a study would require more specific research questions and greater precision in sampling. future research could address these issues and more properly include a more fine-grained analysis on the exact nature of record modifications.   the complementary nature of the cataloguing work performed by agencies and public libraries is a key insight of the study. central agencies are not equipped to address the local concerns of every public library, but they can provide clean, objective bibliographic records. public libraries, conversely, may not have the resources to provide original cataloguing for every item, but they do possess an expert understanding of their own users’ needs and can modify subjective elements of the bibliographic record accordingly. since both central agencies and public libraries perform important cataloguing work, each contributing where the other cannot, cataloguing remains an essential skill in libraries of all sizes.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387–399. https://doi.org/10.1108/07378830610692154   evidence based library and information practice 2011, 6.4 124 evidence based library and information practice evidence summary inter-library loans and document supply services in italy appear to supplement journal subscriptions rather than replace them a review of: bernardini, e., & mangiaracini, s. (2011). the relationship between ill/document supply and journal subscriptions. interlending and document supply, 39(1), 9-25. doi: 10.1108/02641611111112101] reviewed by: kathryn oxborrow senior customer service team member (librarian) hutt city libraries lower hutt, new zealand email: kathryn.oxborrow@huttcity.govt.nz received: 23 may 2011 accepted: 23 oct. 2011 2011 oxborrow. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objectives – to examine patterns of interlibrary loans and document supply (ill) in a large network of libraries over a period of five years, in order to establish whether ill services are being used to replace journal subscriptions. the authors also aimed to establish which journal titles were most requested in their network, to inform future acquisitions policy. design – longitudinal study using transactional data collected from the italian network for inter-library document exchange (nilde) for the period 2005-2009. setting – the italian library sector. subjects – member libraries of nilde, which is the largest ill network in italy with several hundred libraries. these consist primarily of university libraries, but also hospitals and health research institutions, public research institutions, and not-for-profit and public organisations. methods – ill request data collected from the nilde network software were analyzed. figures were retrieved for the number of different journal titles requested per year of the study overall and by individual institution. mailto:kathryn.oxborrow@huttcity.govt.nz� evidence based library and information practice 2011, 6.4 125 further analysis was undertaken on requests for more recent articles, those published up to five years prior to being requested. this involved creating a list of the most requested titles for each year, and then compiling a core collection of journals that were requested 20 or more times in each year of the study. these core titles were analyzed for trends by subject and publisher, and for any significant correlations between either impact factors (ifs) or citation counts and ill requests for particular journals. main results – the data revealed that the number of ills processed through nilde increased every year during the period of the study. the majority of journals were only requested a small number of times in the five year period of the study, with 60% being requested five times or less. in the majority of instances, institutions were not borrowing the same title regularly. analysis of the core collection of journals revealed that these repeated requests of the same title were mainly in the biomedical sciences and science and technology subject areas, and that these journals were often produced by smaller publishers who were not included in consortia purchasing. there was no correlation between journal impact factors (ifs) and ill requests, but there was a statistically significant correlation between citation counts and ill requests. conclusions – ill numbers are increasing despite big deals and consortia purchasing. the majority of requests are for articles that are two years old or older, and the authors suggest that this indicates that ills do not influence journal subscriptions. the authors suggest that ills may have increased during the course of the study (and may continue to do so) due to the current global financial crisis and its impact on library acquisitions. commentary this article adds to an established body of literature on the topic of ill and its impact on journal subscriptions. it follows on from an earlier study about the nilde network of libraries in italy, which the authors state is the most popular ill network in italy. the data set is large, covering tens of thousands of ill requests over a five year period, which lends a great deal of weight to the study. data for the study were anonymized, so individual libraries could not be identified. many of the observations the authors make are interesting, such as the predominance of science and technology and biomedical sciences titles in the most requested journals, and a large proportion of requests being for journals from small publishers. there are some good applications for the italian library community, such as using the data to inform national acquisitions policy. the published literature backs up many of the findings, and the authors make good suggestions for further work, such as investigating the impact of the global recession on the situation regarding ill and journal subscriptions. there are difficulties, however. the conclusion that ills do not have an impact on journal subscriptions is extrapolated from a range of factors, such as the number of requests per journal and the age of the article requested. while each factor is clearly referred to in the results, they could have been drawn together more effectively in the conclusions section. as it stands, readers may be confused by the seeming contradiction between the conclusion based on the large proportion of ill requests for articles that are two years old or older, and the reported annual rise in requests for newer articles. another confusing element is that the authors make no mention of their hypotheses at the beginning of the article, which are then referred to in the results. overall, the scale of the study is impressive, and the authors mention some practical ways in which the results have impacted practice in the italian context, such as in journal subscription negotiations. the study also leaves plenty of room for further work in the area. future work evidence based library and information practice 2011, 6.4 126 could use a mixed-methods approach to investigate the perceptions of acquisitions librarians about this topic. given the strength of its methodology, this study should be replicated in future years, and in other contexts, in order to keep track of changes in the field over time. / evidence based library and information practice evidence based library and information practice 2012, 7.1 105 evidence based library and information practice evidence summary public library clients prefer formal classes for initial training on library’s online resources and informal, on-demand assistance for further training a review of: ruthven, j. (2010). training needs and preferences of adult public library clients in the use of online resources. the australian library journal, 59(3), 108-117. reviewed by: diana k. wakimoto online literacy librarian, california state university, east bay doctoral student, san jose-qut gateway program hayward, california, united states of america email: diana.wakimoto@csueastbay.edu received: 8 sept. 2011 accepted: 4 jan. 2012 2012 wakimoto. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to discover public library clients’ needs and preferences for modes of training on the use of the internet and the libraries’ online resources and to apply these findings to improve training offered by public library staff. design – multiple exploratory case study. setting – two public libraries in new south wales, australia: a regional library (mudgee branch of the mid-western regional council library service) and a metropolitan library (marrickville central library). subjects – a total of 24 public library clients. the participants were split evenly between the two libraries, with 12 from the mudgee branch and 12 from the marrickville central. the respondents were further subdivided into two groups based on age (35 to 44 years old and 65 or older) and evenly distributed by sex within the groups. methods – this study used naturalistic inquiry to frame the multiple exploratory case study of two public libraries. ruthven used maximum variation sampling to guide the selection of participants. library staff helped the researcher to identify possible participants at marrickville, while the researcher advertised for participants at mudgee library and at an internet/database course taught at the mudgee business enterprise centre. she used snowball sampling to find additional participants at both sites. ruthven conducted semi-structured mailto:diana.wakimoto@csueastbay.edu� http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/� evidence based library and information practice 2012, 7.1 106 interviews with the participants, with questions covering their preferences, recommendations, and needs for online resource training. the data from the interviews and search logs were analyzed using inductive data analysis. main results – participants preferred small group, face-to-face, formalized instruction for initial training on online resources. for further training, participants preferred individualized assistance and immediate support instead of formal classes. they noted a lack of training opportunities and a lack of help from library staff as sources of frustration when trying to learn to use online resources at the public libraries. conclusion – public library staff should offer formalized classes for those beginning to learn about using online resources, and focus on ad hoc, individualized assistance for more advanced learners. since offering this type of instructional program is dependent on staff knowledge and staff availability, library staff members need to be trained in the use of online resources and classroom presentation skills. commentary ruthven’s research will be of interest to those involved in computer or online training classes and planning the best allocation of resources for these classes. the literature review’s strength is its overview of previous research related to user preferences in computer training and internet usage, with international coverage. however, spending less time on a discussion of online tutorials, which was “not a primary focus of this study” (p. 110) would have allowed ruthven to discuss more fully the research base directly related to her study, especially the training for older adults (dickinson, eisma, gregor, syme, & milne, 2005; webb, 2003). while ruthven used a relevant methodology for her study, her procedures were not fully explained except for her selection of interview participants, which was well-reasoned. the inclusion of the basic interview guide would have been appreciated. additionally, ruthven noted that she analyzed data “obtained from interviews and search logs” (p. 112), but did not explain what these logs contained or from where she obtained them. the results were clearly reported both in the narrative and in the tables; however, while the participants were divided into multiple subgroups, the results were only reported in the aggregate. while the tables noted some differences between the two libraries in terms of modes and content of training desired, a discussion of these results within the text, along with the percentage of participants whose responses led to these results, would have strengthened the article. it appears from the author’s biography that this article is based on ruthven’s doctoral thesis. detail that is lacking in this article is most likely found in her thesis. this detail would have clarified the methodology and reasons for selective reporting of the findings. the conclusions appear to be supported by the reported findings, and ruthven does not overgeneralize her findings. tying her findings to communications richness theory enhanced their validity, thereby strengthening her conclusions, which would have been further strengthened if quotes from the participants had been included. this study of public library computer users’ training preferences covers an important topic for public librarians and administrators who provide instruction and want to improve the effectiveness of the content and delivery mode. future research could replicate the study in other libraries to determine the generalizability of her findings. also, research into the effectiveness of implementing ruthven’s findings would be useful to determine how they impact adult learners. evidence based library and information practice 2012, 7.1 107 references dickinson, a., eisma, r., gregor, p., syme, a., & milne, s. (2005). strategies for teaching older people to use the world wide web. universal access in the information society, 4(1), 3-15. doi:10.1007/s10209-003-0082-6 webb, l. m. (2002). availability of internet training programs for elderly public library patrons. the reference librarian, 37(77), 137-147. doi:10.1300/j120v37n77_11 / evidence based library and information practice microsoft word es_ed_1.doc evidence based library and information practice 2006, 1:1    81 evidence based library and information practice       editorial    small steps forward through critical appraisal       denise koufogiannakis  associate editor (evidence summaries)    collections and acquisitions coordinator, university of alberta libraries  edmonton, alberta, canada  e‐mail: denise.koufogiannakis@ualberta.ca      © 2006 koufogiannakis. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      as a vocal proponent of evidence based  information practice, i have actively  encouraged my co‐workers and anyone else  who might listen, to take small steps in  order to move towards incorporating  research evidence into their decision making.  i prompt colleagues to search in databases  for research articles that may be useful,  gather these articles and incorporate  research results as part of their decision  making process. i am often told of the  difficulties faced despite one’s best attempts  to make their practice more evidence‐based.   a major difficulty is access to the research  itself since librarians and information  professionals often lack access to databases  and journals in our field, particularly those  of us working outside of academic libraries.  an even more daunting question is how to  critically read the existing research to  determine whether it is valuable in a specific  situation. not all research is good, and how  do we sort out the good from the bad? this  inaugural issue of evidence based library and  information practice brings us one small step  closer to addressing some of these concerns  by equipping practitioners with information  to help in their decision making process.     every issue of evidence based library and  information practice will incorporate  approximately 10 evidence summaries to  help readers sort through recently published  research and determine if that research was  done well.  evidence summaries provide a  critical appraisal synthesis for a specific  research article, so that practitioners may  more readily determine if the evidence in  that research study is valid and reliable, and  whether they can apply it to their own  practice.     evidence summaries are indeed a small step,  but an important one.  published in an  open‐access forum, these summaries will  address three barriers to evidence based  practice that we have faced in the past. first  of all, they will bring awareness of  previously published research to readers  who may otherwise have missed this work.  secondly, they will bring that research to life,  by engaging a dialogue with what has been  published rather than allowing that  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:1    82 published research to quietly wait to be  discovered. and most importantly, the  evidence summaries will allow for an  objective critique of research, which in turn  allows library and information professionals  to make more informed decisions about the  quality of the research and weigh this into  their local decision making. indirectly,  reading critical appraisals informs us all of  the questions we should be asking when we  approach a research article and allows us to  become more familiar with a critical  approach to reading the literature of our  field.    i do not think that we could have a journal  called evidence based library and information  practice without a section dedicated to  critical appraisal of the existing research  literature. a huge part of evidence based  practice consists of filtering through the  published research evidence to determine  whether that evidence is valid, reliable and  applicable to one’s own practice.  publishing  such critical appraisal and sharing it with  the whole lis community is a central part of  this journal and what we are striving to  accomplish.     the evidence summaries team is  comprised of 10 members from australia,                                   canada, new zealand, the united kingdom  and the united states of america.  they  work in, and bring varying skills and   knowledge from, academic, health, public  and special library sectors. this is a diverse  group of people who have dedicated  themselves to writing one evidence  summary per issue for the first year of  publication.  the evidence summaries  follow a standardized format and undergo  double‐blind peer review.  a wide number  of journals are scanned for potential  research articles to review, and suggestions  for review are most welcome.     as editor of the evidence summaries, i hope  that you find the format we have adopted  useful, and that you will encourage  colleagues to search this open‐access journal  when they are looking for pre‐appraised  evidence to support their decision making.    evidence summary   publication numbers are increasing at american research universities   a review of: budd, j. (2017). faculty publications and citations: a longitudinal examination. college & research libraries, 78(1), 80–89. https://doi.org/10.5860/crl.78.1.80   reviewed by: jennifer kaari librarian east orange public library east orange, new jersey, united states of america jkaari@eopl.org   received: 7 sept. 2019                                                               accepted:  31 oct. 2019      2019 kaari. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29647     abstract   objective – to study the publishing output and citation activity of faculty at research universities.   design – bibliometric and citation analysis.   setting – academic citation databases.   subjects – institutions in the united states that are members of the association of research libraries (arl).   methods – this study builds on three previous studies conducted by the author looking at faculty publication productivity, which were conducted for three different time periods beginning in 1991. for the present study, the author searched scopus by institution to collect the total number of publications and citations for the faculty of more than 100 association of research libraries (arl) member universities, covering the years 2011 to 2013. the author acquired the total number of faculty at each institution from the arl website. the faculty number from the arl website and publication and citation data from scopus were used to calculate the per capita publication and citation numbers for each institution. the author calculated the total mean number of publications and the mean number of per capita publications per university. chi tests were used to compare the means for statistical significance.    main results – the number of both total and per capita publications for each institution went up over the course of all three studies. the mean number of total publications per university for 1991 to 1993, the first time period studied, was 4,595.8; for the time period of the current study, 2011 to 2013, the mean was 9,662.0. for per capita publications, the mean for 1991 to 1993 was 3.56 and the mean for the present study was 5.96. based on chi-square tests, the results were found to be statistically significant.   conclusions – the study found that the number of total publications increased significantly over time, exceeding the author’s statistical expectations based on previous work.   commentary   the adage “publish or perish” points to the well-known pressure academics face to publish in the scholarly literature to receive tenure or promotion. this environment can lead to the perception that scholars are emphasizing quantity over quality in their publication output (fanelli & larivière, 2012). examining the total number of scholarly publications overtime can provide valuable insights into the validity of these perceptions. many studies looking at scholarly productivity focus on an individual discipline (walters, 2016; griffin, bolkan, & dahlbach, 2017; ford, richman, mayes, pagel, & bartels, 2019). by looking at total publication numbers across disciplines and institutions, this study provides a valuable, high-level look at the wider field of scholarly publications.   when assessed using glynn’s critical appraisal tool for library and information research, this study has an 86% validity rating (2006). a major strength of the study is that it builds on years of previous studies conducted by the author. the statistical analysis is well-described, and the tables provided present the results clearly. however, the author presents the publication numbers for only the top twenty institutions; future researchers would benefit from having access to all of the data for all the universities analyzed for the study.   a potential limitation of this study did emerge while examining the study methodology. the author presents this study as an examination of faculty publications and citations, but it is unclear how faculty and non-faculty publications and citations were differentiated in the data collection process. if scopus has a functionality that allows for filtering between faculty and non-faculty publications, this should have been mentioned in the study text. if there was no differentiation, this is a serious limitation regarding how useful this data is in considering faculty productivity specifically. the data would still be useful as a look at the publication output and citation numbers for the institution as a whole but not when it comes to examining faculty output specifically.   this study is more informative than it is a basis for actionable practice for librarians. it would be of interest to analyze these results in the context of other studies on scholarly productivity that use different methods or are discipline specific to determine how strong the evidence is for a substantial increase in scholarly productivity over time. the results of the study also provide a foundation for potential future research as noted by the author, including further exploring the relationship between increasing publication numbers and institutional pressures, how publication quantity relates to quality, and the value of citation metrics as evaluation criteria, all of which are vital issues for academic librarians.   references   griffin, d., bolka, s., & dahlbach, b.j. (2018). scholarly productivity in communication studies: five-year review 2012-2016. communication education, 67(1), 88–101. https://doi.org/10.1080/03634523.2017.1385820   fanelli d., & larivière v. (2016). researchers’ individual publication rate has not increased in a century. plos one 11(3): e0149504. https://doi.org/10.1371/journal.pone.0149504   ford, d. k., richman, a., mayes, l. m., pagel, p. s., & bartels, k. (2019). progressive increase in scholarly productivity of new american board of anesthesiology diplomates from 2006 to 2016: a bibliometric analysis. anesthesia & analgesia, 128(4), 796–801. https://doi.org/10.1213/ane.0000000000003926   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387–399. https://doi.org/10.1108/07378830610692154   walters, w. h. (2016). the faculty subculture, the librarian subculture, and librarians’ scholarly productivity. portal: libraries and the academy, 16(4), 817–843. https://doi.org/10.1353/pla.2016.0054       evidence based library and information practice 2012, 7.4 108 evidence based library and information practice evidence summary several factors of library publishing services facilitate scholarly communication functions a review of: park, j.-h., & shim, j. (2011). exploring how library publishing services facilitate scholarly communication. journal of scholarly publishing, 43(1), 76-89. doi: 10.1353/scp.2011.0038 reviewed by: leslie bussert head of instruction/literature and humanities librarian university of washington bothell, cascadia community college bothell, washington, united states of america email: lbussert@uwb.edu received: 11 may 2012 accepted: 27 sept. 2012 2012 bussert. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to identify and examine the factors of library publishing services that facilitate scholarly communication. design – analysis of library publishing service programs. setting – north american research libraries. subjects – eight research libraries selected from the signatories for the compact for openaccess publishing equity (cope) cornell university library’s center for innovative publishing; dartmouth college library’s digital publishing program and scholars portal project; mit libraries’ office of scholarly publishing and licensing; columbia university libraries’ center for digital research and scholarship; university of michigan library’s scholarly publishing office; duke university library’s office of scholarly communications; university of calgary libraries and cultural resources’ centre for scholarly communication; and simon fraser university library’s scholarly publishing. methods – the authors used roosendaal and geurt’s (1997) four functions of scholarly communication to analyze and categorize library publishing services provided by libraries included in the study. the four functions of scholarly communication include registration, certification, awareness, and archiving. mailto:lbussert@uwb.edu http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/ evidence based library and information practice 2012, 7.4 109 main results – analysis of the registration functions provided by library publishing services in this study revealed three types of facilitating factors: intellectual property, licensing, and publishing. these include services such as repositories for digital scholarly work and research, isbn/issn registration, and digital publishing. analysis of archiving functions demonstrated that most programs in the study focus on repositoryrelated services in support of digital content preservation of papers, datasets, technical reports, etc. analysis of certification functions provided by these services exposed a focus on expert review and research support. these include services like professional assessment of information sources, consultation on appropriate literature and information-seeking tools, and writing or copyright advisory services. analysis of awareness function showed search aids and knowledge-sharing platforms to be the main facilitating factors. these include services like metadata application, schema, and standards or scholarly portals enabling knowledge-sharing among scholars. conclusion – this study identified several services offered by these library publishing programs which can be categorized as facilitators under roosendaal and geurt’s (1997) four functions of scholarly communication. the majority of the libraries in the study treated library publishing services as part of broader scholarly communication units or initiatives. digital publishing (registration function) was offered by all programs analyzed in the study, while traditional peerreview services (certification function) were not. widely adopted among programs in the study were the use of social networking tools (awareness function) and self-publishing (archiving function). the authors recommend developing services that facilitate peer review and assert the need to provide a knowledgesharing mechanism within the academic community that facilitates the scholarly communication process. commentary this study contributes to a growing body of literature exploring library publishing services in the broader context of scholarly communication. it uniquely and explicitly ties these services to specific functions across the scholarly communication process while other studies focus on service or business models for these programs. it is also interesting to consider this study in light of the recent findings of mullins et al. (2012) in their library publishing services report released by the scholarly publishing and academic resources coalition (sparc), which found approximately half (55%) of survey respondents have or want to develop such services, and that those with existing programs anticipate increasing their capacity within the next year (p. 6). this study fills a gap in the literature but comes with some limitations. its primary weakness is the lack of in-depth analysis of the findings and their applicability, and the absence of suggestions for further research. further discussion on this could enhance understanding of the role of academic libraries in scholarly communication and help readers consider the application of this work to their library’s practice and context. the sample size is small and includes mostly association of research libraries members, many of which have been at the forefront of establishing library publishing services. lastly, some methodological details are absent, such as how information was obtained about the programs and the process of mapping services to the chosen theoretical construct. roosendaal and guert’s (1997) framework outlining the four key functions of scholarly communication is effective and appropriate for this analysis, and is a structure libraries could utilize for program evaluation and planning. the mullins et al. (2012) sparc report recommends “treating academic publishing support as a holistic endeavor and assuming responsibility for acquiring a comprehensive understanding of editor and author needs” (p. 2). this framework could be used to identify areas where services need to be expanded or added, and could be helpful for libraries planning to offer publishing services. it can evidence based library and information practice 2012, 7.4 110 also help managers of library publishing services understand scholarly communication functions in both conceptual and specific terms in order to better address the needs of their scholarly community or to strategically develop and leverage collaborative partnerships across campus to ensure fulfillment of all scholarly communication functions. the mullins et al. (2012) sparc report also notes that further articulation of the important role library publishing services play in scholarly communication is needed (p. 3). this study contributes to that goal, and can be enhanced by replicating the study with a larger and more diverse sample size, examining which library publishing service or business models best facilitate the four functions of scholarly communication, and investigating the feasibility of roosendaal and geurt’s (1997) framework as a program development and assessment tool. references mullins, j. l., murray-rust, c., ogburn, j. l., crow, r., ivins, o., mower, a., nesdill, d., newton, m. … watkinson, c. (2012). library publishing services: strategies for success: final research report. retrieved 19 oct. 2012 from http://docs.lib.purdue.edu/purduepres s_ebooks/24 roosendaal, h. e., & geurts, p. a. t. m. (1997). forces and functions in scientific communication: an analysis of their interplay. first international workshop on cooperative information systems in physics, oldenburg, germany. retrieved 19 oct. 2012 from http://www.physik.unioldenburg.de/conferences/crisp97/roos endaal.html evidence summary   increasing access to digital archives is a complex problem, and more collaboration between archivists and users is needed to enact solutions    a review of: jaillant, l. (2022). how can we make born-digital and digitised archives more accessible? identifying obstacles and solutions. archival science, 22, 417-436. https://doi.org/10.1007/s10502-022-09390-7   reviewed by: christine fena undergraduate success librarian stony brook university libraries stony brook, new york, united states of america email: christine.fena@stonybrook.edu   received: 1 june 2023                                                               accepted:  10 july 2023      2023 fena. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30380     abstract   objective – to outline current levels of access to digitized and born-digital collections, investigate and identify obstacles to increasing access, and suggest possible solutions.   design – semi-structured online interviews.   setting – archives, libraries, and museums based in the uk, ireland, and the united states.   subjects – a total of 26 practitioners in archives, libraries, and museums including 12 women and 14 men.   methods – the researchers recruited participants from existing personal contacts and those contacts’ colleagues, with attention toward diversifying in the areas of gender, career stage, institution size, and geographical location. twelve interview questions were sent to interviewees in advance, but the questions were tailored to each interviewee during the interview with follow-up questions asked as necessary. a team of three digital humanities scholars conducted 21 interviews with the 26 subjects, and all but three interviewees agreed to be named in the resulting article. all interviews were conducted in may 2021, except one, which was conducted in november 2020.   main results – the author discusses relevant paraphrases and quotations from the interviewees under four headings: “obstacles to access to digitised collections,” “born-digital collections: from creation to access,” “current levels of access to digital collections,” and “possible solutions to the problems of access.” key obstacles to access that emerge throughout the discussion include technological obsolescence, copyright and permissions, data protection of sensitive materials, lack of a market for born-digital records, and the problem of scale and skill gaps. strategies to increase access include enhanced collections, less restrictive legislation, new access interfaces including virtual reading room software, use of artificial intelligence to increase discoverability, and web archives. the author makes distinctions between born-digital (e.g., emails) and digitized (e.g., scanned photographs) content throughout the discussion of results.   conclusion – there is a paradox between the focus on data analysis in current research and the difficulty researchers have in accessing cultural data through digital archives, but increasing access to digital collections remains a challenging and complex problem. the author highlights some possible solutions that emerged from the interviews, including artificial intelligence, but also emphasizes the need to bring together an interdisciplinary community of both archivists and users, to continue shifting the conversation surrounding digital collections from focusing on preservation to focusing on access, and to advocate for changes to legislation, digitization practices, and copyright clearance.   commentary   discourse surrounding digital collections has recently been changing. a digital preservation coalition report suggests that the focus has shifted away from preservation and toward appraisal and processing (prom, 2019). additionally, the born-digital archives working group recently released “levels of born-digital access,” which focuses exclusively on access (peltzman et al., 2020). making digital collections more accessible and usable is a formidable problem facing archivists and users alike, with significant impacts toward researchers' ability to analyze and understand history, culture, and society in the digital age. solutions involve probing issues related to copyright law, ethics, and whether the archivist needs to take a more active role in reorganizing and interpreting content.   this article was appraised using “the cat: a generic critical appraisal tool” (perryman & rathbun-grubb, 2014). the study’s strengths include the author’s significant expertise; she has years of experience as a digital humanist, extensive publications, and collaborations with multiple funded projects that focus on digital archives. in addition, the author includes the interview questions within the body of the article, and the interviewees represent a variety of geographic areas and different types of institutions including archives, libraries, and museums.   however, there are several limitations to this work. as the author acknowledges, large cultural institutions are over-represented, and the method of recruitment, through existing networks, makes it possible that the results are distorted. no inclusion or exclusion criteria are specified. no institutions from asia or continental europe are included, nor institutions from anywhere in the global south.   another significant issue is that the author never explains the method of analysis. it is unclear whether the four headings in the results narrative emerged from analyzing the text of the interviews, as they might if qualitative content analysis was used, or if the author had already chosen these headings as preconceived categories. there is also no discussion of whether a theoretical framework was used to create the interview questions. finally, the quotations and paraphrases from the interviews are mixed in with quotations and paraphrases from existing literature. this sometimes makes it difficult to parse which ideas emerge from the new research.   despite these weaknesses, the article provides a rich and detailed discussion of the many challenges facing access to and use of digital collections. the quotations and paraphrases from the interviewees reflect a variety of perspectives from over two dozen archivists, as well as different strategies practitioners are trying to address the problems they are seeing in the field. the author does not try to simplify a complicated array of issues, makes clear distinctions between born-digital and digitized content, and focuses on the need for archivists and users to work together to better understand and confront the many challenges facing digital collection access. although the author does not make a detailed or prescriptive proposal for how to move forward, the discussion is a good starting point for those looking to fine-tune their understanding of the many problems facing access to digital archives, as well as some possible solutions.   references   jaillant, l. (2022). how can we make born-digital and digitised archives more accessible? identifying obstacles and solutions. archival science, 22, 417-436. https://doi.org/10.1007/s10502-022-09390-7   peltzman, s., dietz, b., butler, d., walker, p., farrell, j., arroyo-ramirez, e., macquarie, c. bolding, k., helms, a. m., & venlet, j. (2020). levels of born-digital access. digital library federation, council on library and information resources. https://osf.io/r5f78/   perryman, c., & rathbun-grubb, s. (2014). the cat: a generic critical appraisal tool. http://www.jotform.us/cp1757/thecat   prom, c. j. (2019). preserving email. (2nd ed.). digital preservation coalition. http://doi.org/10.7207/twr19-01   evidence based library and information practice 2012, 7.3 95 evidence based library and information practice evidence summary applying the narrow forms of pubmed methods-based and topic-based filters increases nephrologists’ search efficiency a review of: shariff, s. z., sontrop, j. m., haynes, r. b., iansavichus, a. v., mckibbon, k. a., wilczynski, n. l., weir, m. a., speechley, m. r., thind, a. … garg, a. x. (2012). impact of pubmed search filters on the retrieval of evidence by physicians. cmaj: canadian medical association journal, 184(3), e184-e190. doi: 10.1503/cmaj.101661 reviewed by: kate kelly chief librarian royal college of surgeons in ireland dublin, ireland email: katekelly@rcsi.ie received: 13 june 2012 accepted: 15 aug. 2012 2012 kelly. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to determine whether the use of pubmed methods-based filters and topic-based filters, alone or in combination, improves physician searching. design – mixed methods, survey questionnaire, comparative. setting – canada. subjects – random sample of canadian nephrologists (n=153), responses (n=115), excluded (n=15), total (n=100). methods – the methods are described in detail in a previously published study protocol by a subset of the authors (shariff et al., 2010). one hundred systematic reviews on renal therapy were identified using the evidenceupdates service (http://plus.mcmaster.ca/evidenceupdates) and a clinical question was derived from each review. randomly-selected canadian nephrologists were randomly assigned a unique clinical question derived from the reviews and asked, by survey, to provide the search query they would use to search pubmed. the survey was administered until one valid search query for each of the one hundred questions was received. mailto:katekelly@rcsi.ie http://plus.mcmaster.ca/evidenceupdates evidence based library and information practice 2012, 7.3 96 the physician search was re-executed and compared to searches where either or both methods-based and topic-based filters were applied. nine searches for each question were conducted: the original physician search, a broad and narrow form of the clinical queries therapy filter, a broad and narrow form of the nephrology topic filter and combinations of broad and narrow forms of both filters. significance tests of comprehensiveness (proportion of relevant articles found) and efficiency (ratio of relevant to non-relevant articles) of the filtered and unfiltered searches were conducted. the primary studies included in the systematic reviews were set as the reference standard for relevant articles. as physicians indicated they did not scan beyond two pages of default pubmed results, primary analysis was also repeated on search results restricted to the first 40 records. the ability of the filters to retrieve highlyrelevant or highly-cited articles was also tested, with an article being considered highlyrelevant if referenced by uptodate and highly-cited if its citation count was greater than the median citation count of all relevant articles for that question – there was an average of eight highly-cited articles per question. to reduce the risk of type i error, the conservative method of bonferroni was applied so that tests with a p<0.003 were interpreted as statistically significant. main results – response rate 75%. physicianprovided search terms retrieved 46% of relevant articles and a ratio of relevant to nonrelevant articles of 1:16 (p<0.003). applying the narrow forms of both the nephrology and clinical queries filters together produced the greatest overall improvement, with efficiency improving by 16% and comprehensiveness remaining unchanged. applying a narrow form of the clinical queries filter increased efficiency by 17% (p<0.003) but decreased comprehensiveness by 8% (p<0.003). no combination of search filters produced improvements in both comprehensiveness and efficiency. when results were restricted to the first 40 citations, the use of the narrow form of the clinical queries filter alone improved overall search performance – comprehensiveness improved from 13% to 26 % and efficiency from 5.5% to 23%. for highly-cited or highly-relevant articles the combined use of the narrow forms of both filters produced the greatest overall improvement in efficiency but no significant change in comprehensiveness. conclusion – the use of pubmed search filters improves the efficiency of physician searches and saves time and frustration. applying clinical filters for quick clinical searches can significantly improve the efficiency of physician searching. improved search performance has the potential to enhance the transfer of research into practice and improve patient care. commentary the authors state that this study moves beyond the development of filters to testing their functionality in the real-world context of physician searching and that, to their knowledge, the latter has only been attempted twice before and never with methodsand topic-based filters in combination. this reviewer agrees with those statements although arguably the study by lokker, haynes, wilczynski, mckibbon and walter (2011), which includes 40 practising physicians and a study objective “to determine the yield of relevant citations and physician satisfaction while searching for diagnostic and treatment studies using the clinical queries page of pubmed compared with searching pubmed without these filters” could be added to the two studies referenced. the only minor criticism of this study is that it cannot be read or replicated without reading the published study protocol. however the evidence based library and information practice 2012, 7.3 97 study protocol describes the methodology in great detail and is worth reading by itself as an example of the level of thinking behind a study design. this reviewer found that any questions arising from reading the study were answered either further on in the paper or in the study protocol and the three online appendices which accompanied the study. the response rate of 75% is impressive and, as the study protocol indicates that the total population of nephrologists in canada is 519, the sample size of 100 is large enough. the authors clearly identify the strengths and limitations of the study, indicate areas of further research and reach valid conclusions. for health sciences librarians this is further evidence to support the use of the clinical queries filter and to include teaching the use of clinical filters in pubmed training – the results suggest using the methods-based filters are generalizable to other sub-specialties. in addition, as the authors suggest, the research methodology can be applied to other specialties to further evaluate the performance of filters in the real world. the dearth of literature on this topic indicates real research potential in this area. references lokker, c., haynes, r. b. wilczynski, n. l. mckibbon, k. a., & walter, s. d. (2011). retrieval of diagnostic and treatment studies for clinical use through pubmed and pubmed's clinical queries filters. journal of the american medical informatics association, 18(5), 652-659. doi:10.1136/amiajnl-2011-000233 shariff, s. z., cuerden, m. s., haynes, r. b., mckibbon k. a., wilczynski, n. l., iansavichus, a. v., speechley, m. r., thind, a., & garg, a. x. (2010). evaluating the impact of medline filters on evidence retrieval: study protocol. implementation science, 5, 58. doi:10.1186/1748-5908-5-58 evidence based library and information practice 2011, 6.3 1 evidence based library and information practice evidence summary while collaboration is increasing in the profession the lis dissertation remains a soloauthored monograph a review of: sugimoto, c. r. (2011). collaboration in information and library science doctoral education. library & information science research, 33, 3-11. doi:10.1016/j.lisr.2010.05.003 reviewed by: diana k. wakimoto online literacy librarian, california state university, east bay doctoral student, san jose-qut gateway program hayward, california, united states of america email: diana.wakimoto@csueastbay.edu received: 5 may 2011 accepted: 22 july 2011 © 2011 wakimoto. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to investigate collaboration in lis doctoral education, in particular the extent and perception of collaboration between advisors and advisees, and the dissertation as a collaborative product. design – quantitative and qualitative analysis of questionnaire data. qualitative analysis of interviews. bibliometric analysis of curricula vitae (cvs) and dissertation citations. setting – american library association (ala)accredited, doctorate-granting schools in the united states and canada. subjects – a total of 374 full-time, tenured faculty members with the rank of associate or full professor (advisor group) and 294 assistant professors (advisee group) comprised the pool of faculty members (n=668) who were sent the questionnaire. of these, 30 individuals participated in follow-up telephone interviews, which were equally split between the two groups. there were 97 faculty members from the original pool of 668 faculty members were included in the bibliometric analyses. mailto:diana.wakimoto@csueastbay.edu� evidence based library and information practice 2011, 6.3 62 methods – the author developed two questionnaires, one for the advisors (associate and full professors) and one for the advisees (assistant professors), and sent the surveys to faculty members at ala-accredited schools in the united states and canada. the questionnaires gathered information about the extent of collaboration and perceptions of collaboration in lis doctoral education. the author also collected contact information from those interested in participating in a follow-up interview. the author selected the first 30 individuals who responded as the interview participants. the interview participants were split equally between advisors and advisees. a separate subpopulation of 97 faculty members was chosen for the bibliometric analysis phase of the study. these faculty members were chosen with the following criteria: graduation from an ala-accredited school; full-text of dissertation available online; and a current, full cv available online. cvs were searched to determine the level of co-authoring before and after graduation. main results – a total of 215 faculty members completed the questionnaires. the results from the surveys showed that more than 61% of the advisors reported collaborating with at least half of their advisees, while 58% of the advisees reported collaborating with their advisors. both advisors and advisees defined collaboration mainly as publishing, researching, and presenting together. more than 50% of the advisors reported co-publishing with half of their advisees during the advisees’ doctoral education. the advisors reported co-publishing with less than 30% of their advisees after the students completed their doctoral education. advisees reported similar numbers: 44% and 31%, respectively. following graduation, the majority of advisees (96%) planned to publish from their dissertations. of these, 78% did not plan to include their advisor as co-author in these publications. 42% of the advisors reported that none of their advisees included them as coauthors, while 3% of advisors stated that their advisees always included them as co-authors. after the 30 interview transcripts were coded using inductive and deductive approaches, the results showed that advisees saw research as a process whereby they became collaborators with their advisors. advisees also found collaboration with other doctoral students as “kind of key” (p. 7). advisors saw collaboration as a form of mentorship. however, both advisees and advisors reported that the dissertation itself was not a collaborative product, with the responsibilities of the dissertation tasks falling more heavily on the advisees than the advisors, except in the realm of reviewing and approving the final version of the dissertation. analysis of the cvs for co-publishing between advisees and their advisor and/or committee members showed that 41% of the advisees published with their advisors and 34% published with at least one committee member before receiving their doctorate. after receiving their doctorates, 31% of the advisees published with their advisors and 32% published with a committee member. conclusion – the author concluded that a majority of advisors and advisees see collaboration as joint publication during the period of doctoral studies. both advisors and advisees see the doctoral dissertation as a soloauthored monograph and not a collaborative product. however, other forms of collaboration among advisees and their advisors, committee members, and fellow doctoral students are viewed as important parts of the doctoral education experience. based on these findings, the author suggests that the profession may need to adapt its model of doctoral education to become more aligned with the increasingly collaborative nature of lis research. evidence based library and information practice 2011, 6.3 63 commentary this study examined the collaboration between doctoral students (advisees) and their advisors during and after the successful completion of the dissertation. while advisors and their advisees have similar positive views of collaboration in general and its growing importance in lis research, neither group views the dissertation as a collaborative product. the study is important for our understanding of current collaboration in the doctoral process and will be of interest to those involved in lis doctoral education. it may also provide an impetus for reconsidering how students prepare for “collaborative models of scholarly productivity” (p. 4). the author clearly explained the study’s multiple methods of data collection and analysis. all methods were appropriate for the research questions. it would have improved the paper to have broken down the numbers of advisors versus advisees who responded to the surveys. this information can be deduced from figure 1, but it would improve readability to include the numbers in the text. in addition, reporting the overlap of advisors and advisees used in the bibliographic analyses compared with the surveys and interviews would have strengthened the study. the paper would have also benefited from presenting the statistics in more readable tables to facilitate understanding the key results. however, these are minor points in an otherwise methodologically strong paper. the author did note the limitations in representing only “one kind of ‘successful’ doctoral experience” (p. 5) and not including either unsuccessful doctoral students or those who did not become faculty members at alaaccredited schools in the study. overall, this is a clear, well-written article covering a very important topic for the lis profession. while it may be an oversimplification to assert that we are “requiring a product of 19th century science from scholars who we expect to contribute to 21st century science” (p. 10) by requiring a soloauthored dissertation and therefore are not adequately preparing students for the rigors of collaborative research, this article is important for the ongoing conversation about lis doctoral education. as suggested by sugimoto, future research that explores the impact of “doctoral student colleagues in the intellectual development of their peers” (p. 9), the impact of grant funding on lis research, and how the lis model of doctoral education relates to other social science disciplines will continue to increase our knowledge of and ability to improve lis doctoral education. / evidence based library and information practice evidence summary   focused bedside education may improve engagement of hospitalized patients with their patient portals   a review of: greysen, s.r., harrison, j.d., rareshide, c., magan, y., seghal, n., rosenthal, j., jacolbia, r., & auerbach, a.d. (2018). a randomized controlled trial to improve engagement of hospitalized patients with their patient portals. journal of the american medical informatics association, 25(12), 1626-1633. https://doi.org/10.1093/jamia/ocy125   reviewed by: joanne m. muellenbach library director and associate professor california health sciences university clovis, california, united states of america email: jmuellenbach@chsu.edu   received: 6 nov. 2019                                                                   accepted:  15 jan. 2020      2020 muellenbach. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29671     abstract   objectives – to study hospitalized patients who were provided with tablet computers and the extent to which having access to these computers increased their patient portal engagement during hospitalization and following their discharge.      design – prospective, randomized controlled trial (rct) within a larger, observational study of patient engagement in discharge planning.   setting – a large, academic medical centre in the western united states of america.   subjects – of a total of 250 potential subjects from a larger observational study, 137 declined to participate in this one; of the remaining 113 subjects, 16 were unable to access the patient portal, leaving 97 adult (18 years of age or older) patients in the final group. all subjects (50 intervention and 47 control) were randomized but not blinded, had been admitted to medical service, and spoke english. in addition, all participants were supplied with tablet computers for one day during their inpatient stay and were provided with limited assistance to the portal registration and login process as needed. they were also required to have access to a tablet or home computer when discharged.   methods – the intervention group participants received focused bedside structured education by trained research assistants (ras) who demonstrated portal key functions and explained the importance of these functions for their upcoming transition to post-discharge care. following enrolment and consent, ras administered a brief pre-study survey to assess baseline technology use. then, at the end of the observation day, the ras performed a debrief interview in which participants were asked to demonstrate their ability to perform key portal tasks. the ras recorded which tasks were accomplished or if the ras had provided assistance. patient demographics and clinical information were obtained from the electronic health record (ehr).      main results – of the 97 patients who were enrolled in the rct, 57% logged into their portals at least once within seven days of their discharge. the mean number of logins and specific portal tasks performed was higher for the intervention group than for the control group. in addition, while in the hospital, the intervention group was better able to log in and navigate the portal. only one specific portal task reached statistical significance—the use of the tab for viewing the messaging interaction with the provider. the time needed to deliver the intervention was brief—less than 15 minutes for 80% of participants. the intervention group’s overall satisfaction with the bedside tablet to access the portal was high.   conclusion – data analysis revealed that the bedside tablet educational intervention succeeded in increasing patient engagement in the use of the patient portal, both during hospitalization and following discharge. as the interest and demand for patient access to ehrs increases among patients, caregivers, and healthcare providers, more rigorous studies will be needed to guide the implementation of patient portals during and after hospitalization.   commentary   this is the first randomized clinical trial to test an intervention for increasing patient engagement with their portals in hospital and post-discharge settings. the study, through its strong design, provides a model for other studies of patient portal use.    a critical appraisal tool (glynn, 2006) revealed the study’s strengths and weaknesses. the authors received irb approval and provided appendices outlining the inclusion criteria, enrolment, and randomization details as well as other details (the mychart tutorial, post-study survey, and final questionnaire). however, the study would have been stronger if it had included patients from more than one hospital, as noted by the authors together with other study limitations in the “discussion” section (p. 1632). with regard to the training provided by the ras, the effect of the study not being blinded is unclear, as is the effect of ras being allowed to “tailor the depth of the explanations to the needs of individual patients” (p. 1627). in future studies, a separate “limitations” section might assist readers in considering study limitations.   patient demographics and clinical information were adequately summarized. however, when logistic regression analysis was performed, it revealed that an area of difference between the two groups was prior mychart registration: 34 participants in the intervention group (68%) were previously registered versus 18 participants (38%) in the control group. given this difference, the study would have been more robust if it had randomized for previous mychart registration.     future studies might be enriched by incorporating a “teach to goal” approach for such skills as effective use of inhalers for patients with respiratory conditions and might investigate additional patient characteristics (acute care needs, use of different devices, and age/income effects, for example). several recent studies have focused on how inpatient portals have furthered patient empowerment. a qualitative, retrospective study (mcalearney, 2019), found early indications that inpatient portals “promoted independence, reduced anxiety, informed families, and increased patient empowerment, overall.” a randomized trial (masterson, 2019) concluded that inpatient portals resulted in a decreased 30-day readmission rate. a systematic review (dendere, 2019) found some evidence that patient portals benefited by discovering medical errors, improving medication adherence, and enhancing patient-provider communication, and these predictors for portal use could be used in future studies.   clinical, consumer health, or hospital-based librarians may wish to collaborate with health providers in further studies in order to contribute to the development of new evidence-based, consumer health resources that could be linked within the patient portals and to understand how library resources might be integrated into such portals. medical librarians might also benefit from understanding the degree of engagement for patients who receive consumer health educational interventions using patient portals and might contribute to identifying patients with the greatest need for portals, such as those with serious health conditions.         references   dendere, r., slade, c., burton-jones, a., sullivan, c., staib, a., & janda, m. (2019). patient portals facilitating engagement with inpatient electronic medical records: a systematic review. journal of medical internet research, 21(4), e12779. https://doi.org/10.2196/12779   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154   masterson, c. r., grossman, l. v., ryan, b., qian, m., polubriaginof, f. c. g., restaino, s., bakken, s., hripcsak, g., & vawdrey, d. k. engaging hospitalized patients with personalized health information: a randomized trial of an inpatient portal. journal of the american medical informatics association, 26(2), 115-123. https://doi.org/10.1093/jamia/ocy146   mcalearney, a. s., fareed, n., gaughan, a., macewan, s. r., volney, j., & sieck, c. j. (2019). empowering patients during hospitalization: perspectives on inpatient portal use. applied clinical informatics, 10(01), 103-112. https://doi.org/10.1055/s-0039-1677722 evidence based library and information practice 2012, 7.4 119 evidence based library and information practice evidence summary doctoral students in new zealand have low awareness of institutional repository existence, but positive attitudes toward open access publication of their work a review of: stanton, k. v., & liew, c. l. (2012). open access theses in institutional repositories: an exploratory study of the perceptions of doctoral students. information research, 17(1), paper 507. available from http://informationr.net/ir/17-1/paper507.html reviewed by: theresa s. arndt associate director for library resources & administration waidner-spahr library, dickinson college carlisle, pennsylvania, united states of america email: arndtt@dickinson.edu received: 1 sept. 2012 accepted: 26 oct. 2012 2012 arndt. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to investigate doctoral students' knowledge of and attitudes toward open access models of scholarly communication and institutional repositories, and to examine their willingness to comply with a mandatory institutional repository (ir) submission policy. design – mixed method, sequential exploratory design. setting – a large, multi-campus new zealand university that mandates ir deposit of doctoral theses. subjects – two doctoral students from each of four university colleges were interviewed. all 901 doctoral students were subsequently sent a survey, with 251 responding. methods – semi-structured interviews with eight subjects selected by purposive sampling, followed by a survey sent to all doctoral students. the authors used nvivo 8 for analysis of interview data, along with a twophase approach to coding. first, they analyzed transcripts from semi-structured interviews line-by-line to identify themes. in the second phase, authors employed focused coding to analyze the most common themes and to merge or drop peripheral themes. themes were mapped against rogers' diffusion of innovation theory and social exchange theory constructs to aid interpretation. the results were used to develop a survey with a fixed set file://umd-home.umd.umich.edu/mkdunawa/private/my%20documents/eblip/7(4)/arndtt@dickinson.edu http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/ evidence based library and information practice 2012, 7.4 120 of response choices. authors then analyzed survey results using excel and surveymonkey, first as a single data set and then by discipline. main results – the authors found that general awareness of open access was high (62%), and overall support for open access publication was 86.3%. awareness of irs as a general concept was much lower at 48%. those subject to a mandatory ir deposit policy for doctoral theses overwhelmingly indicated willingness to comply (92.6%), as did those matriculating prior to the policy (83.3%), although only 77.3% of all respondents agreed that deposit should be mandatory. only 17.6% of respondents had deposited their own work in an ir, while 31.7% reported directly accessing a repository for research. the greatest perceived benefits of ir participation were removal of cost for readers, ease of sharing research, increased exposure and citing of one's work, and professional networking. the greatest perceived risks were plagiarism, loss of ability to publish elsewhere, and less prestige relative to traditional publication. the reason most given for selecting a specific publication outlet was recommendation of a doctoral supervisor. disciplinary differences in responses were not sizable. for additional interpretation, the authors applied rogers’s diffusion of innovations theory to determine the extent to which irs are effective innovations. the authors posit that repositories will become a more widely adopted innovations as awareness of irs in general increases, and through increased awareness that ir content is discoverable through major search engines such as google scholar, thus improving usability and increasing dissemination of research. using the social exchange theory framework, the authors found that respondents’ expressed willingness to deposit their work in irs demonstrated altruistic motives for sharing their research freely with others, appreciation for the reciprocity of gaining access to others’ research, and awareness of the potential direct reward of having their work cited more often. conclusion – authors identified that lack of awareness, rather than resistance to deposit, as the main barrier to ir depository participation. major benefits perceived for participating included the public good of knowledge sharing and increased exposure for one’s work. concerns included copyright and plagiarism issues. these findings have implications for communication and marketing campaigns to promote doctoral students' deposit of their work in institutional repositories. while respondents reported low direct use of irs for conducting research, the vast majority reported using google scholar, and so may have unknowingly accessed open access repository content. this finding suggests that attention be given to enhanced metadata for optimizing discoverability of ir content through general search engines. commentary [critical appraisal tool used: booth & brice (2003). cristal checklist for appraising a user study. available from: http://nettingtheevidence.pbwiki.com/f/use.do c] the authors have made a useful contribution to the literature on attitudes toward new modes of scholarly communication, and provide a thorough literature review of prior studies. colleges and universities relying on a “build it and they will come” approach for ir participation have typically been disappointed. studies of faculty have identified multiple barriers to participating in open access publication in general and irs in particular. faculty concerns about increased plagiarism, copyright, and tenure qualification have been widely reported. prior studies have also found significant disciplinary differences in open access support. by focusing on doctoral students, this study sheds light on the attitudes of emerging researchers. it is to be expected that awareness and attitudes of doctoral students would be largely consistent with studies of faculty, who transmit the social norms of the academy to their students. however, the authors' unanticipated finding of weak disciplinary differences suggests that the next generation of researchers may be more receptive to ir participation regardless of their evidence based library and information practice 2012, 7.4 121 disciplines' traditional form of scholarly communication. the objectives of the current study and the study population have been clearly defined. the mixed method approach is of benefit in conducting research into attitudes about complex concepts and practices. eight doctoral students were interviewed, and criteria for interview subject selection, other than the college within the university, are not stated. the study would have been enriched by additional interview results. the overall survey response rate of 28% on the survey is also disappointing, though not uncommon in this type of research, and the authors do note the possibility of response bias. additional clarity is provided by reporting both percentages and number of responses for specific questions. no tests were applied to determine statistical significance of the findings. this leaves the reader to conclude whether a particular result may or may not be meaningful. however given the consistency with prior studies, the findings are credible. library practitioners could readily replicate the methodology to gauge local attitudes. as colleges and universities develop institutional repositories, it is important to understand the motivating factors and barriers affecting researchers' willingness to participate. mandatory deposit policies may be useful, but cannot substitute for individual researcher's buy-in to both the concept and practice of open access publication. as the next generation of academic faculty, doctoral students' awareness and understanding of ir benefits and risks will be crucial in shaping this new form of scholarly communication. by understanding these attitudes, librarians and others involved in ir development can create more effective communication and marketing programs. the finding that doctoral supervisor recommendation most influenced choice of publication venue suggests that marketing and advocacy aimed at this group may greatly influence ir participation. [these columns are not symmetrical in length] microsoft word es_931_bogel evidence based library and information practice 2008, 3:2 61 evidence based library and information practice evidence summary social marketing and the school library: an effective path to collaboration? a review of: immroth, barbara and w. bernard lukenbill. “teacher-school library media specialist collaboration through social marketing strategies: an information behavior study.” school library media research 10 (2007). 22 april 2008 <http://www.ala.org/ala/aasl/aaslpubsandjournals/slmrb/slmrcontents/volume10/immrot h_teacherslmscollaboration.cfm>. reviewed by: gayle bogel director, learning resources and information technology, john read middle school redding, connecticut united states of america e-mail: gbogel@gmail.com received: 03 december 2007 accepted: 15 april 2008 © 2008 bogel. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – the study attempted to apply the strategies of social marketing theory to collaboration between school librarians and teachers. design – based on the 1972 theory of social marketing by zaltman, kotler and kaufman, a cohort of students in a graduate-level practicum established a collaborative unit with selected teachers within their school. in addition, two focus groups were conducted in alternate schools to gauge the overall attitudes of teachers toward collaboration with school librarians. subjects – students (student librarians) in a graduate-level certification class for texas school librarians, and both teachers and librarians in host schools/districts for the graduate students’ practicum experiences methods – researchers used qualitative approaches, both case study and focus groups, to gather data about the collaborative interactions between teachers and school librarians. the interactions were designed using the social marketing aida model (attention, interest, desire, and action). social marketing, based on models of commercial marketing, assumes that social goodwill is a motivator for establishing interactions between groups – evidence based library and information practice 2008, 3:2 62 or selling a service that is for the greater good. students in a graduate-level practicum were instructed to develop a strategy based on the aida model to elicit and carry out a collaborative unit with teachers in their host schools. they were given specific guidelines by the principal investigators that included: • instructions for designing announcements, leaflets, and conferences as marketing strategies • instructional unit designs for subject content and information literacy skills • incentive payments of $200 to be used for library resources as an incentive to collaborate. • the steps to engaging in the collaborative process • procedural guidelines for taking field notes, unobtrusive observations and informal evidence. summative evaluation was based on a reflective journaling exercise by both student librarian and teacher. measurements and recordings were analysed using accepted case study methods. main results – social marketing model the researchers evaluated the study in each of the four aspects of the social marketing model. attention (a) – gaining attention and convincing. efforts to gain attention through student choices of flyers to teachers were not successful. e-mail announcements were more effective, but it appeared that direct librarian-teacher contact was the most effective. the monetary incentive also did not appear to have an effect on response rate. host librarians did make suggestions regarding the appropriateness of when and how to distribute the flyers in some cases. researchers concluded that perhaps such a straightforward advertising approach did not fit in the established relationships, and may be a better choice for new librarians who are establishing their presence in schools. interest (i) -promoting interest in services and/or products researchers noted that initial strategies did not promote interest in the field study project. teachers cited time and test– related curriculum restraints, and viewed the project as an “extra” responsibility. the researchers note the need to establish the value of the collaborative instruction to long-term goals for both teachers and librarians. the focus groups showed more interest in collaboration, and an awareness of the value of librarians’ collaboration in promoting effective teaching and improving student achievement. desire (d) and action (a) – recognizing values and taking action. field test responses did not reflect desire on the part of teachers to collaborate with student librarians. only two teachers responded directly to the advertisement. the offer of monetary incentive ($200 in library supplies) also did not appear to increase motivation of teachers to participate. results after the field test showed that overall, teachers gained an appreciation of the value of collaboration with school librarians, and indicated they would be open to future projects. action process themes of successful marketing campaigns were evident in the results of the study and benefits in being evidence based library and information practice 2008, 3:2 63 exposed to new resources and information approaches were reported by teachers. the concept of territoriality of teachers, and how much authority is shared with librarians in a collaborative setting, was an aspect not explored by the study, although indications from both the field test and the focus groups showed that the perception of the competency of the student librarian, and the teacher’s personal approach (structured vs. more relaxed) affected the release of teaching authority. the librarian bringing ideas, concepts and directions to teachers can enhance collaboration. opportunities to collaborate based on objectives of state-mandated exams to develop specific skills can also foster collaboration. main results – collaborative research this project reflected much of the earlier research in collaboration and added data to support the importance of the findings of the landmark mettessich and monsey (1972) study of collaboration. shared interest, mutual trust, flexibility, adaptability and clear roles and policies were all reflected as needs in the current study. in addition, clear communication, shared goals and purposes and the need to have leadership from the school librarians in establishing collaborative interactions was reiterated. predictive behaviour of teachers toward collaboration included time and overall commitments to other teaching responsibilities. confidence in the skills and knowledge of the librarian also affected the teachers’ willingness to collaborate. conclusions – social exchange theory and community psychology were cited by the researchers as two theoretical concepts that affected the design and interpretation of data. they suggest that these two strategies may be most helpful in situations that have less than optimal environments for collaboration, where librarians have not been successful, or are not considered equal to the tasks. commentary the findings of this study present a structured method for school librarians to approach collaboration. while largely unsuccessful, the methods used in the study were illuminating in that they clearly illustrate the challenges in creating a collaborative environment in a school setting. the act of offering monetary rewards seemed curiously at odds with the educational setting and the research objectives, and although no reference was made to teacher reactions (other than stating that only two teachers were motivated by the reward), it seems likely that the monetary aspect may have been a deterrent rather than a motivation for many teachers. although monetary incentives are routinely used in commercial marketing ventures (and commonly given to focus group participants), they may be less appropriate in school settings. more concrete analysis of the findings would help practitioners understand the results: 1. the researchers state that “most, if not all” of student librarians chose to use flyers. it would have been helpful to see a chart of exactly how many respondents chose flyers, and what the other choices for gaining attention were, with the problems and successes clearly noted. 2. stating the total number of participants in these focus groups, evidence based library and information practice 2008, 3:2 64 and the makeup of the sample: disciplines, ages, years of experience, and comparative percentages that would clarify the strength of the conclusions. one well documented aspect of collaboration is the need to establish trust with the collaborative partner (matessich and monsey, 1992), and it appeared that student librarians had not established the necessary trust with teachers or, in some cases their mentor librarians who may have been unwilling to risk their existing relationships with teachers and administrators by endorsing the somewhat unknown skills of student librarians. it appears that the shared goal of “the greater good,” the approach of social marketing as defined by the researchers, does not match the current emphasis on student achievement that is the overwhelming concern of both teachers and school librarians. student learning, while ultimately defined by educators as the “greater good”, was not clearly stated as a possible shared gain in the approaches used by the student librarians. they would benefit from reframing the goals of collaboration to clearly link both teacher and librarian in activities that promote collaboration as a means to affect measurable outcomes of student learning. works cited mattessich, paul w. and barbara r. monsey. collaboration: what makes it work. a review of research literature on factors influencing successful collaboration. st. paul, mn: amherst h. wilder foundation, 1992. 22 april 2008 <http://eric.ed.gov/ericdocs/data/e ricdocs2sql/content_storage_01/0000 019b/80/14/53/ba.pdf>. zaltman, gerald p., philip kotler, and ira kaufman, eds. creating social change. new york: holt, rinehart, and winston, 1972 microsoft word es_lewis.doc evidence based library and information practice 2006, 1:2      40 evidence based library and information practice       evidence summary    pda use by clinicians has a positive impact on clinical decision making    a review of:  dee, cheryl r., marilyn teolis, and andrew d. todd. “physicians’ use of the personal digital  assistant (pda) in clinical decision making.” journal of the medical library association  93.4 (october 2005): 480‐6.    reviewed by:  suzanne p. lewis   library manager, central coast sector  northern sydney central coast health   gosford, new south wales, australia  email: slewis@nsccahs.health.nsw.gov.au    received: 2 march 2006           accepted: 13 april 2006      © 2006 lewis. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective –  to examine how frequently  attending physicians and physicians in  training (medical students, interns and  residents) used pdas for patient care and to  explore physicians’ perceptions of the  impact of pda use on several aspects of  clinical care.    design – user study via a questionnaire.    setting – teaching hospitals in tennessee,  florida, alabama, kentucky, and  pennsylvania in the united states.    subjects – a convenience sample of fifty‐ nine attending physicians and forty‐nine  physicians in training (108 total), spread  unevenly across the five states.   methods – subjects were recruited by  librarians at teaching hospitals to answer a  questionnaire which was distributed and  collected at medical meetings, as well as by  email, mail, and fax. the subjects were  required to have and use a pda, but prior  training on pda use was not a requirement,  nor was it offered to the subjects before the  study. most of the questions required the  respondent to choose from five likert scale  answers regarding frequency of pda use:  almost always, often, a few times, rarely, or  never. in the reporting of results, the options  ‘almost always’ and ‘often’ were combined  and reported as ‘frequent’, and the options  ‘a few times’ and ‘rarely’, were combined  and reported as ‘occasional’. subjects could  also record comments for each question, but  only for affirmative responses.    http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:2      41 subjects were asked about their frequency of  pda use before, during, or after a patient  encounter. they were also asked if pda use  had influenced one or more of five aspects  of clinical care – decision making, diagnosis,  treatment, test ordering, and in‐patient  hospital length of stay.    data analysis included chi square tests to  assess differences between attending  physicians and physicians in training  regarding frequency of pda use and the  influence of pda use on the five aspects of  clinical care. the subject population was  also divided into frequent and occasional  users of pdas, and chi square testing was  used to assess differences between these two  groups regarding the influence of pda use  on clinical care. a significance value of  p<0.05 was considered statistically  significant.    main results – ninety‐four (87%) of the 108  respondents used pdas for patient  encounters. of this group, 59 were frequent  users and 35 were occasional users. there  were no significant differences between  attending physicians and physicians in  training with regard to frequency of pda  use in patient encounters.    sixty‐seven percent of the 108 respondents  reported that using a pda had influenced  their clinical decision making; over 50%  reported that pda use had influenced  changes in patient treatment; 16% reported  that pda use had helped avoid unnecessary  tests; 10% reported that pda use had helped  change a patient’s diagnosis; and 6%  reported that pda use had helped shorten a  patient’s length of stay. within these results,  there were no statistically significant  differences between the attending  physicians and the physicians in training.    more than 85% of the frequent pda users  (n=59), and 60% of the occasional pda users  (n=35), reported that pda use had  influenced their clinical decision making.  the difference between these two groups  was statistically significant (p<0.007) with  regard to the influence of pda use.     fifty‐six percent of respondents recorded  comments on the survey. more than 75% of  the comments included using pdas to  access drug‐related information for clinical  decision making.    conclusion – pda use has a positive impact  on clinical decision making and patient care.  frequency of pda use appears to be a factor  in determining the perceived impact of  pdas on clinical decision making. however,  even those physicians who used pdas only  occasionally reported a positive impact of  pda use on clinical decision making. the  status of physicians (attending or in training)  does not appear to be a determining factor  in the frequency of pda use for patient  encounters. health sciences librarians are  well positioned to provide resources and  training for pda use by clinicians.    commentary    this is an interesting study which builds on  recent research into clinicians’ use of pdas.  not only does this study examine frequency  of pda use for patient encounters, it also  explores physicians’ perceptions of the  impact of pda use on clinical decision  making, diagnosis, treatment, test ordering,  and length of stay. the researchers  successfully achieved their stated objectives  and relate their results to previous relevant  studies.    the study is not without faults. the sample  size of this study (n=108) is much smaller  than in the comparable studies (n≥946) cited  in the article. the authors acknowledge this  and go on to say, “the data from this study  cannot be broadly generalized to the entire  population of pda users, because the small  sample is a sample of convenience . . . .”  (484). there is potential response bias due to  the questionnaire being self‐reporting.  additionally, neither the sample size nor the  response rate is clearly reported; the reader  evidence based library and information practice 2006, 1:2      42 must assume that all respondents (n=108)  comprised the entire sample, and that the  response rate was 100%.    the report provides useful information  about the study subjects, particularly a  definition of ‘physicians in training’,  allowing readers unfamiliar with this term  to translate it to their own settings. one of  the most interesting findings is that the level  of medical training achieved by the subjects  was not found to be a significant factor in  either the frequency of pda use or in the  perceived impact of pda use.     inclusion of the survey instrument in an  appendix, plus sufficient information in the  methodology section about how the results  were collated and analyzed, means that the  study could be repeated by other  researchers in similar clinical settings. the  authors do not state whether the survey  instrument was validated, whether they  devised it themselves, or whether they  adapted it from another study.    the only statistically significant result was  that frequency of pda use was a factor in  determining the perceived impact of pda  use in clinical decision making. analysis of  the respondents’ comments is valuable in  highlighting prescribing as the area of  clinical activity where most value is seen to  be derived from pda use.     this research makes a valuable contribution  to understanding physicians’ perceptions of  the impact of pda use in clinical care. the  methodology is reliable and applicable to  most health care settings in developed  countries. the study is relevant to health  librarians seeking to understand the  attitudes of physicians towards emerging  technology. the results suggest that health  librarians can have a positive impact on  clinical decision making by supporting pda  use with training and resources.    a valuable suggestion for future study  would be to quantify the potential savings  in drug costs, reduction in prescribing errors,  and/or reduction in adverse drug  interactions that could be achieved by  physicians’ routine use of pdas in patient  encounters. this would provide an evidence  base to those health librarians who wish to  present a business case for providing pda  resources and training to clinicians.   evidence based library and information practice 2011, 6.3 84 evidence based library and information practice using evidence in practice building a user blog with evidence: the health information skills academic library blog rachel adams academic support librarian the library, university of salford salford, united kingdom email: r.adams@salford.ac.uk received: 08 july 2011 accepted: 04 aug. 2011 © 2011 adams. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. setting the college of health and social care within the university of salford is supported by a team of academic support librarians who deliver training, support and advice to staff and student users of library resources and services. the team continually seeks ways to educate, inform and assist users. in 2010, staff began a blog to provide a new support format, with the aim of building a repository of training and information. this article sets a year of blog experience in the context of a literature review and content analysis. problem students often study from a distance or spend time away from campus in clinical settings. previous attempts by the library team to support those away from university included provision of a blackboard site within the institution’s virtual learning environment (vle). however users needed to log in to access the resource, and the formal structure of vle sites made delivering and accessing support content difficult. users quickly lost interest. many users request librarian help with library resources and with wider information literacy skills. often these arrive as email requests, some of which can be answered immediately. others require an appointment to meet with the librarian. queries that can be answered quickly are received frequently by three librarians, who send individual responses. they felt that providing that same information online would significantly reduce the time spent responding to simpler queries. from march to august, 2010, the team developed the salford health information skills mailto:r.adams@salford.ac.uk� evidence based library and information practice 2011, 6.3 85 blog to begin the process of building a repository of information. the aim was creating what godwin (2007) describes as “an additional weapon in our armoury” of existing support (p. 105). more complex questions are still most frequently and effectively dealt with by means of an appointment with the librarian. however, as the blog developed, librarians wrote blog posts on some of the topics discussed in individual appointments. librarians refer students to these posts as an initial response, or as a refresher for those who have attended a session. from september 2010, library staff has promoted the blog to students and college staff. promotional approaches included: using the blog address in email signatures; mentioning the blog in library training sessions and in the team’s newsletters; and presenting a poster at a university good practice day which provided the opportunity to both promote to and gain feedback from academic staff. evidence a literature review revealed that the accessibility of blog software, coupled with the flexibility of use, means that blogs are still one of the best ways for libraries to communicate with their users (johnson, 2008). the blog aims to provide a repository of information, as content is built and accumulated over time leading to a ‘stock’ of information and guidance to which users can refer. the library team anticipated that as this repository grew it would assist them in responding to user enquiries (godwin, 2007). by referring users to blog posts in answer to a query the team could not only reduce the time spent responding but also effectively promotes the blog as a useful resource that user might visit again in the future. the blog functions as an online information source, much like the library’s official website, and has not moved forward to take on the role of a communication or interaction tool. the opportunity to engage directly with blog content through a comments tool is a key benefit that blogs have over traditional websites. much of the literature refers to a lack of success regarding blog comments within the academic library environment (toth, 2010). this has certainly been the case with the salford health information skills blog, where users do not comment on posts. the health information skills blog primarily targets students within the college, although it is also of use to academic staff. determining our primary audience allowed us to identify the type of information that would be most useful (often taken from the queries received from users). knowing the audience also helps set the conversational tone and level of detail most appropriate for the blog (stover, 2006). blogs are easy to create and are generally free to use so represent a good opportunity to reach users. the main issue with blogs is that they require regular updates and maintenance. when a blog is provided by a team, as is the case with the health information skills blog, the team shares the workload of creating new content and maintaining the site. if the blog reduces existing workloads around user enquiries, then time saved reproducing standard responses can instead be spent creating blog posts for wider audiences. even blogs with regular updates require evaluation of their use, readership and impact. user statistics are a useful tool (chan & cmor, 2009), as blog content is only valuable if it has an audience. the team made use of the statcounter tool to see which posts are popular, the number of new users, how many users make return visits, the length of time people spend viewing blog content, and the path they take coming into and leaving the site. the team performed content analysis in april 2011, analysing 30 blogs from uk academic evidence based library and information practice 2011, 6.3 86 institutions where content had been updated in 2011 (see appendix a). we compared the structure, content, and use of the blog with others in the sector. at least 50% of examined blogs employ five key features and it is reassuring to note that the health information skills blog uses all of these features. the five features are a search option, archives, tag or category list to help users navigate to related content, links to the library website to set the blog content in context, and provision of rss feeds to connect users with new content as it develops. a full list of features identified can be seen in figure 1. one key feature of blogs is the use of tags or category lists. the majority of blogs employed tags in either cloud or list form, so reviewing the tags in depth presented a snapshot of the content coverage. where slight differences between terms existed (for example ebook, e‐ book, ebooks), the team normalized the tags to one option for comparison. the health information skills blog currently has 14 tags (see figure 2), some of which overlap with the top 20 tags identified in the content analysis (figure 3). this suggests coverage in the health blog is broadly similar to that in other academic library blogs. while some language is specific to salford university, tags around news, electronic systems and resources, databases, and training appear on both lists. the literature review highlighted the issue of user reluctance to engage with blog content through comments. this issue also appeared in the content analysis. two blogs did not provide the option for comments. of the remaining blogs, there was a 50/50 split between those with no comments and those with some. further examination of blogs with comments showed that while a few had regular comments, half of the blogs examined received fewer than five comments in a year. this finding suggests that many academic library blogs struggle to engage users through comments. blogs with regular comments appeared in online learning figure 1 blog features analysed evidence based library and information practice 2011, 6.3 87 environments (open university) or were provided by an individual rather than a service where blogger and reader may feel more personally connected. implementation as a result of the literature review and content analysis, the team identified some key amendments and additions for the health information skills blog. first, further promotion is needed. the literature suggests that promoting the benefit to the user encourages people to engage with yet another information source. posters around the library and flyers for enquiry desk staff to hand out are one promotional option. the team will also consider delivering an existing newsletter directly through the blog. figure 2 tags used in the health information skills blog figure 3 top 20 tags idenfied in the content analysis of uk academic blogs evidence based library and information practice 2011, 6.3 88 the blog already has an rss feed; however, there is evidence that users are unsure how rss works so an additional ‘subscribe via email’ option using feedburner is now being added. this may encourage more users to sign up, and also compiles blog statistics. the team may post a brief explanation of how to use the rss tool, perhaps near the start of the new academic year, to encourage new readers to make use of that option. over the past academic year librarians posted irregularly. we created a more formalised agreement regarding the frequency of posting and the division among team members to ensure the blog is regularly updated. the blog does allow comments and the team would welcome the use of this feature by users. increasing both the overall audience and the number and frequency of return visits might encourage users to interact using this feature. it may be useful to add a brief post encouraging users to engage in this way. outcome the evidence within the literature and our analysis of blog activity in academic libraries indicates that there is a place for a resource such as the health information skills blog. the blog sits well within the context of both literature review and a comparison with other, similar blogs. the team will modify the blog throughout the following academic year, focusing primarily on promotion to increase the audience, and will re‐ evaluate the time needed to maintain such a resource in terms of audience statistics and feedback. reflection although blogging is not without its issues it would appear to be a worthwhile tool (johnson, 2008). the health team implemented the blog as an additional method of easily accessible support to provide information, training, news, and advice to the users we already support. the initial aim of the blog was to create a repository of information to refer users and this is being achieved over time. as the blog develops the team feels positive about its potential, and now has a plan for further development. the team is now turning its attention to increasing the blog’s audience and its impact with the hope of creating an engaging and valuable information source. this research led us to the conclusion that continuing the blog is worthwhile. we discovered that our experience is similar to that of others, and we can learn from others who are perhaps further ahead in their blog development or use. without this research evidence the team would have no context within which to reflect on its experience, and no external input to take this endeavour into the future. references chan, c., & cmor, d. (2009). blogging toward information literacy: engaging students and facilitating peer learning. reference services review, 37(4), 395‐407. doi: 10.1108/00907320911007001 godwin, p. (2007). information literacy meets web 2.0: how the new tools affect our own training and our teaching. new review of information networking, 13(2), 101‐112. doi: 10.1080/13614570801900005 johnson. k. (2008). are blogs here to stay? an examination of the longevity and currency of a static list of library and information science weblogs. serials review, 34(3), 199‐204. doi: 10.1016/j.serrev.2008.06.009 evidence based library and information practice 2011, 6.3 89 stover, j. s. (2006). making marketing work for your library blog. internet reference services quarterly, 11(4), 155‐167. toth, m. (2010). are users interested in library blogs? bobcatsss 2010. retrieved march 24, 2011 from http://hdl.handle.net/1889/1259 / evidence based library and information practice microsoft word es8_1002_final_final_final evidence based library and information practice 2008, 3:1  78 evidence based library and information practice       evidence summary    clinician‐selected electronic information resources do not guarantee accuracy in  answering primary care physicians’ information needs    a review of:   mckibbon, k. ann, and douglas b. fridsma. “effectiveness of clinician‐selected electronic  information resources for answering primary care physicians’ information needs.”  journal of the american medical informatics association 13.6 (2006): 653‐9.      reviewed by:  martha ingrid preddie  doctoral student, school of information and library science, university of north carolina at  chapel hill, chapel hill, north carolina, united states of america   e‐mail: mipreddie@gmail.com      received: 24 december 2007     accepted:  1 march 2008      © 2008 preddie. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to determine if electronic  information resources selected by primary  care physicians improve their ability to  answer simulated clinical questions.      design – an observational study utilizing  hour‐long interviews and think‐aloud  protocols.      setting – the offices and clinics of primary  care physicians in canada and the united  states.      subjects – twenty‐five primary care  physicians of whom 4 were women, 17 were  from canada, 22 were family physicians,  and 24 were board certified.       methods – participants provided responses  to 23 multiple‐choice questions. each  physician then chose two questions and  looked for the answers utilizing information  resources of their own choice. the search  processes, chosen resources and search  times were noted. these were analyzed  along with data on the accuracy of the  answers and certainties related to the  evidence based library and information practice 2008, 3:1  79 answer to each clinical question prior to the  search.       main results – twenty‐three physicians  sought answers to 46 simulated clinical  questions. utilizing only electronic  information resources, physicians spent a  mean of 13.0 (sd 5.5) minutes searching for  answers to the questions, an average of 7.3  (sd 4.0) minutes for the first question and  5.8 (sd 2.2) minutes to answer the second  question.  on average, 1.8 resources were  utilized per question. resources that  summarized information, such as the  cochrane database of systematic reviews,  uptodate and clinical evidence, were  favored 39.2% of the time, medline (ovid  and pubmed) 35.7%, and internet resources  including google 22.6%. almost 50% of the  search and retrieval strategies were  keyword‐based, while mesh, subheadings  and limiting were used less frequently. on  average, before searching physicians  answered 10 of 23 (43.5%) questions  accurately. for questions that were searched  using clinician‐selected electronic resources,  18 (39.1%) of the 46 answers were accurate  before searching, while 19 (42.1%) were  accurate after searching. the difference of  one correct answer was due to the answers  from 5 (10.9%) questions changing from  correct to incorrect, while the answers to 6  questions (13.0%) changed from incorrect to  correct. the ability to provide correct  answers differed among the various  resources. google and cochrane provided  the correct answers about 50% of the time  while pubmed, ovid medline, uptodate,  ovid evidence based medicine reviews and  infopoems were more likely to be  associated with incorrect answers.  physicians also seemed unable to determine  when they needed to search for information  in order to make an accurate decision.     conclusion – clinician‐selected electronic  information resources did not guarantee  accuracy in the answers provided to  simulated clinical questions. at times the  use of these resources caused physicians to  change self‐determined correct answers to  incorrect ones. the authors state that this  was possibly due to factors such as poor  choice of resources, ineffective search  strategies, time constraints and automation  bias. library and information practitioners  have an important role to play in identifying  and advocating for appropriate information  resources to be integrated into the electronic  medical record systems provided by health  care institutions to ensure evidence based  health care delivery.     commentary     this research adds to knowledge base  created by westbrook, coirea and gosling’s  investigation of clinicians’ use of  information resources to answer simulated  clinical questions. inclusion criteria were  family physicians in canada and the united  states as well as general internists in the  united states who were attending to  patients regularly in a clinic. there was no  predetermined sampling. instead,  physicians were selected based on  availability. this resulted in the majority of  the participants being primary care  physicians from the mcmaster online  rating of evidence (more) system, a  project in which clinicians are involved in  rating published research studies and  systematic review articles. participants  included faculty members from  departments of family medicine at a  canadian university and the university of  pittsburgh. this occurrence introduced  some measure of bias into the study since  those physicians can hardly be regarded as  representative of primary care practitioners  in general.      while the use of interviews was appropriate  in order to discern participants’ experience  and use of computers, electronic resources  and the internet, the article failed to fully  evidence based library and information practice 2008, 3:1  80 describe the application of think‐aloud  protocols, why they were utilized, as well as  how they benefited the study. this  shortcoming makes it difficult to determine  the feasibility of replicating this study in a  different subject group. the study also  failed to indicate how the correct answers to  clinical questions were determined.  however, the majority of the questions  utilized for answering and searching were  used previously in a 2002 study conducted  by hersh and colleagues. this served to  validate the main data collection instrument.      the researchers acknowledged other  limitations: the small number of  participants, essentially volunteers who  possessed a fervent interest in evidence  based medicine, the focus on primary care  physicians only, and the use of simulated  questions rather than the observation of  physicians seeking information during  actual clinical encounters. despite these  shortcomings, the findings of the study are  reliable since they mirror the results of  earlier studies in terms of the occurrence of  a considerable number of changed answers  after searching including health care  practitioners changing answers from correct  to incorrect.  [hersh et al., “factors  associated with successful answering of  clinical questions,” (328); hersh et al.,  “factors associated with success in  searching medline,” (289)]. however, the  results differ somewhat from westbrook et  al.’s study of clinicians in australia, where  the findings revealed that notwithstanding  some errors, the use of online information  retrieval systems resulted in “significant  improvement in the quality of answers  provided by clinicians to typical clinical  problems” (315).       while the original study does not suggest  further areas for research, this study could  have benefited from an investigation of the  reasons for physicians’ choice of resources,  search strategies, and acceptance of  provided answers, especially when these  changed from correct to incorrect. future  research could also explore the relationship  between physicians’ original self‐assessed  skills pertinent to research, database  searching and the internet, and their  performance in answering clinical questions.  additionally, the successful use of google  and cochrane, along with the unsuccessful  use of the highly touted uptodate, warrant  further investigation.      the significance of this article to library and  information practice is its    acknowledgement that the use of electronic  information resources by primary care  physicians does not guarantee accuracy in  answering their information needs. the  integration of evidence based information  resources into electronic medical record  systems has the potential to improve  accuracy. library and information  professionals should assume the  responsibility to advise parent institutions  that, in the integration of information  resources into electronic medical record  systems, designs should facilitate the  retrieval of accurate information by  clinicians, regardless of individual search  competencies. such systems should provide  easy and effective search capabilities, as well  as the ability to prompt physicians to seek  information from designated evidence based  sources, when an improvement in care is  possible.        works cited      hersh, william r., m. katherine crabtree,  david h. hickam, lynetta rose,  and charles p. friedman. “factors  associated with successful  answering of clinical questions  using an information retrieval  system.” bulletin of the medical  library association 88.4 (2000): 323‐ 31.  evidence based library and information practice 2008, 3:1  81   hersh, william r., m. katherine crabtree,  david h. hickam, lynetta sacherek,  charles p. friedman, patricia  tidmarsh, craig mosbaek, and dale  kraemer. “factors associated with  success in searching medline and  applying evidence to answer  clinical questions.” journal of the  american medical informatics  association 9.3 (2002): 283‐93.    westbrook, johanna i., enrico w. coiera,  and a. sophie gosling. “do online  information retrieval systems help  experienced clinicians answer  clinical questions?” journal of the  american medical informatics  association 12.3 (2005): 315‐21.  evidence based library and information practice using evidence in practice   improving reference service with evidence   bonnie r. nelson professor and associate librarian for information systems, lloyd sealy library john jay college of criminal justice, city university of new york new york, new york, united states of america email: bnelson@jjay.cuny.edu   received: 19 nov. 2015  accepted: 25 jan. 2016      2016 nelson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     setting   john jay college of criminal justice is a senior college of the city university of new york (cuny), serving a student population of approximately 11,000 fte (full-time equivalent students). while john jay now offers majors in a variety of fields, traditionally our focus has been on criminal justice, public management, forensic science, and forensic psychology.  our motto remains “educating for justice.” our students are typical graduates of new york city public schools, who often find the idea of writing a research paper using academic resources to be a challenge. john jay’s reference librarians aim to help students meet that challenge with library instruction in selected classes (especially first-year writing classes and research methods classes), outreach to faculty, and by providing reference desk service every hour the library is open.  the lloyd sealy library has a print collection selected to meet the needs of an undergraduate population as well as a research-level collection in criminal justice that serves the needs of a doctoral program and researchers around the world. our online resources are very strong for a public college our size, owing to long-standing cooperative arrangements among cuny libraries and support from the cuny central office.   problem   in december 2014, the library department assessment committee met to review the longitudinal statistics we had been maintaining as part of our participation in both the acrl annual and als biennial library statistics reporting programs. the library faculty of john jay college of criminal justice have always taken these measures very seriously and been as assiduous as possible about maintaining both accuracy and consistency in counting methods. as a result, we felt we had fairly reliable numbers going back more than 20 years. we met to review these numbers to see what they could reveal about the work we had been doing and where we could improve.   many of the trends over this 20-24 year period were expected from our knowledge of the history of john jay college and general library trends: john jay college’s full time equivalent (fte) student numbers increased dramatically before leveling off and then dropping slightly; circulation of materials from the general collection declined as electronic journals and e-books became commonplace; both collection and total expenditures increased over time, but decreased when inflation was factored in and even more so on a per student fte basis. the library’s gate count numbers, however, were somewhat erratic, most likely fluctuating in response to the use of space elsewhere in the college that resulted in more or less free space for students to study. but the gate counts never showed a serious decline in use and informal observation confirmed that the library continued to be a popular place for students to study alone or in groups, with students sometimes sitting on the floor at the height of the semester.   the most troubling and glaring trend observed by the assessment committee was the long-term, steep decline in the number of reference questions asked. the decline was in absolute numbers, as well as in questions per fte student (see figure 1). the sealy library faculty had often discussed the proper staffing of the library reference desk, prioritizing this service as perhaps the single most important way to help students succeed, but these numbers made us question the wisdom of staff hours devoted to reference service. experienced reference librarians pointed out that although the questions were fewer in number, they tended to be complicated and required more time to sort through the students’ needs. still the decline in numbers was so steep and troubling that it became the one statistic the assessment committee chose to focus on (see figure 1).     figure 1 decline in reference transactions     a month-long discussion ensued involving the entire library department. several librarians noted that the library literature indicated that the decline in reference questions was ubiquitous in academic libraries (stevens, 2013) and that recent ethnographic studies had shown students’ reluctance to ask librarians for help (green, 2012) (miller & murillo, 2012). these studies, plus librarians’ own observations, showed that students frequently needed the help of a skilled librarian even when they did not ask. therefore, we challenged ourselves to increase the number of reference questions answered beginning with the spring 2015 semester.   evidence   two recent changes in statistics collection made it possible to measure the effectiveness of our efforts. first, after years of relying on a “typical week” mode of counting reference questions, in august 2013 we had switched to a locally-developed, simple means of counting every reference transaction. this was developed primarily as a means to evaluate how fully to staff the reference desk, but it also allowed us to see what kinds of questions we were getting and other trends.   secondly, in september 2014 we re-instituted a chat reference service using libraryh3lp (nub games https://libraryh3lp.com/). the sealy library had previously used questionpoint (oclc http://www.questionpoint.org/), but dropped the service after concluding that it provided insufficient benefits to our students. for the previous few years we had been relying on email reference and infrequently-used texting to service off-campus users. after our disappointment with the earlier chat experience, the new chat service was launched with muted expectations but a desire to provide online reference service to students in john jay’s first online master’s degree program, which also started in september 2014. we were able to provide the new service during peak hours of reference desk use, from monday-thursday, 11:00a.m. – 5:00p.m. we announced the new service in our library news blog, and added a chat widget to both the library home page and to the “ask a librarian” page. otherwise we did not publicize the service. libraryh3lp provides excellent statistics on duration of chat, ip address of questioner, and url of the page where the chat initiated.    looking at reference statistics in isolation, however, would not necessarily provide a complete picture. the number of reference questions asked is also related to the number of fte students, the number of classes we teach (since those students tend to be heavy library users), and the number of students entering the library, among other things, so we needed to look at reference questions in relation to the other statistics we keep.    implementation   we took several steps to try to encourage the asking of more reference questions.   to increase in-person reference:   signage identifying the reference desk was reviewed and improved reference librarians were encouraged get up from behind the desk and walk around to be more approachable reference librarians were encouraged to actively approach students who looked like they might need help student staff at the circulation, reserve, and library computer lab desks were reminded to refer patrons needing help to the reference desk   to increase chat reference, we:   added four chat hours per week, from 5:00p.m. to 6:00p.m. monday-thursday added a chat widget to our ezproxy login error page added a chat widget to the results page in all our ebscohost databases added a link to our “ask a librarian” web page (where a chat widget is located) on proquest databases added a chat widget to some of our libguides   outcome   a review of reference statistics at the end of the spring 2015 semester[i] indicated that our interventions were successful (table 1).   without implementing our chat reference service in fall 2014, total reference questions asked from fall 2013 to fall 2014 would have continued their long-term decline; the addition of chat reference reversed that by a very modest .96%. however, we took more aggressive steps for spring 2016 (discussed above) and the number of reference questions asked increased  by nearly 11% compared to the previous spring. even without the chat service, the increase would have been a respectable 5%.   a look at the other statistics we keep indicated that such an unexpected increase in usage was not reflected elsewhere (figure 2).   from spring 2014 to spring 2015 there was actually a small decrease in the number of students at john jay. there was a sharp decline in the number of library instruction classes taught, the usual driver of students to the reference desk. there was a 5% increase in the number of users passing through our security gates. however, in prior years, when we used the “typical week” method of estimating usage statistics, there was little relationship between gate count and reference questions. in those years gate counts went up and down, but reference questions consistently dropped. use of our electronic resources, as measured by proxy server connections, showed a much bigger increase in fall than in spring.     table 1 change in reference transactions, 2013/14 to 2014/15 total reference questions chats total without chat change with chat change without chat fall 2013 total 5744 5744 spring 2014 total 4547 4547 fall 2014 total 5799 167 5632 spring 2015 total 5040 250 4790 change fall 2013 to fall 2014 0.96% -1.95% change spring 2014 to spring 2015 10.84% 5.34%     figure 2 change in library activity: by fall and spring semesters     figure 3 source of web chats: fall 2014, spring 2015     reflection   the effectiveness of both our traditional and our chat interventions needed to be examined. the chat question was fairly easily answered by looking at the source of the chats, as shown by our libraryh3lp logs (figure 3).   whereas the source of over half of our web chat sessions in the fall was our home page, in the spring, after adding additional chat access points, the home page accounted for only 26% of our chats while 43% of our chats came from these new sources. also, 13% of our chat sessions came from the hour added between 5:00p.m. and 6:00p.m. it should be noted that we did not do any additional publicizing of the chat service, although word of mouth and repeat users may account for some of the increase. randomly selected chat transcripts confirmed that the questions coming from ebscohost were indeed questions from users confused about how to search for information, or how to interpret what they were finding. this insight, along with the increased usage, confirmed what we believed to be true: that we are improving our services to students by adding our chat widget to all possible locations.   ironically, in mid-fall 2015, after this study, we realized that most of the questions coming from the chat widget on the ezproxy login error page were from students incorrectly entering their usernames. we have attempted to revise our login pages to eliminate confusion. if we succeed we will improve service but reduce our chat counts. this is a paradoxical result but a reminder that the numbers we collect can never tell the full story.   whether or not we were successful in our attempts to increase in-person reference was less clear. a 5% increase in in-person reference questions asked (over the previous spring) would have been unlikely had it not been for the proactive approach on the part of the librarians, particularly in light of the sharp decrease in library instruction classes. but it was certainly possible that our department-wide discussion of reference statistics resulted in more assiduous recording of the activity, rather than greater efforts to engage our students. to attempt to answer this question, the writer asked all john jay reference librarians to fill out a simple two-question survey, asking whether they were aware that we were trying to increase the number of reference questions asked and whether they had changed their behavior in any way in order to elicit more questions.    out of 19 reference librarians, 15 responded. ten were aware of the program, but seven librarians felt they did nothing different last spring and five said that they recorded the reference questions more assiduously. however, three said that they walked around the reference area to be more approachable and six said that they directly addressed students who looked like they needed help.  in comments, two of the librarians indicated that better signage might have been the primary reason for any increase in the number of reference questions. it is clear that at least some of the reference librarians took actions that resulted in more students getting the help they need.   conclusion   we found that a decrease in the number of reference questions is not inevitable and that both in-person and remote questions will increase if librarians reach out to connect to users where they are, whether sitting in the library being puzzled or working at home with a database they find confusing. this conclusion seems obvious and almost trite, but it was only by looking at the evidence of decreasing reference use that we were motivated to make changes. and, hopefully, seeing how effective these actions have been will encourage us to expand on these changes even further.   references   green, d. (2012). supporting the academic success of hispanic students. in l. duke, & a. asher, college libraries and student culture: what we now know (pp. 87-89). chicago: american library association.   miller, s., & murillo, n. (2012). why don't students ask librarians for help? undergraduate help-seeking behavior in three academic libraries. in l. duke, & a. asher, college libraries and student culture: what we know now (pp. 49-70). chicago: american library association.   stevens, c. (2013). reference reviewed and re-envisioned: revamping librarian and desk-centric services with libstars and libanswers. the journal of academic librarianship, 39(2), 202-214. http://dx.doi.org/10.1016/j.acalib.2012.11.006h     [i] all statistical comparisons are fall to fall and spring to spring, since experience has revealed that library usage changes dramatically every year from the fall to the spring semester. microsoft word news2_1471_final evidence based library and information practice 2008, 3:1  85 evidence based library and information practice       news     call for papers: evidence based library and information practice      © 2008. this is an open access article distributed under the terms of the creative commons attribution  license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and  reproduction in any medium, provided the original work is properly cited.       evidence based library and information  practice seeks papers on all areas of  ebl/eblip including, but not limited to:  • ebl application  • qualitative and quantitative  research  • management and administrative  issues related to ebp  • research tools (statistics, data  collection methods, etc.)  • research education in library  schools  • collaborations with other  disciplines  • evidence based practice from other  disciplines applicable to ebl  • harnessing evidence to support  new innovations  • developing and applying evidence  based tools    papers may be submitted to the following  sections:  • research articles (peer reviewed,  original research)  • commentaries (non‐peer reviewed,  opinion‐based)    submission deadlines for 2008:  june 1st (for september 15th issue)  september 1st (for december 15th issue)  december 1st (for march 15th, 2009 issue)    further information regarding author  guidelines and the online submission  process can be found on the submission  section of the eblip website  http://ejournals.library.ualberta.ca/index.ph p/eblip/index. microsoft word news_studyday2.doc evidence based library and information practice 2006, 1:3  103 evidence based library and information practice     news      study day: everything you ever wanted to know about journals (but were afraid to  ask)      © 2006 eblip. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    the uk nhs south west workforce  development confederation’s  knowledge resources development unit   presents:     “everything you ever wanted to know  about journals (but were afraid to ask)”     journals are an essential building block in  every library service, but what do you really  know about how they are put together?    at this study day, a variety of speakers will  give the benefit of their knowledge about  the world of journal publishing. delegates  will be actively encouraged to arrive armed  with queries, ask questions, and pick the  collective brain.    speakers include:    • alison brettle, associate editor,  evidence based library and  information practice    • sally marchant, editor, midirs  midwifery digest    • jayne bailey, editorial manager,  journal of psychopharmacology    date: thursday 2nd november 2006    10.30am‐3.30pm  registration / coffee 10am   course commences 10.30am    venue:  cst tutorial room, ubht  education centre,  upper maudlin street,  bristol, bs2 8ae    enquiries to john loy, learning resources  manager, avon and wiltshire mental  health partnership nhs trust:   john.loy@awp.nhs.uk                  http://creativecommons.org/licenses/by/2.0 microsoft word news_ifla.doc evidence based library and information practice 2006, 1:4  82 evidence based library and information practice     news      call for papers:  world library and information congress: 73rd ifla general conference and council,  durban, south africa, 19‐23 august 2007        © 2006 eblip. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    libraries for the future: progress,  development and partnerships    the ifla social science libraries section  standing committee invites library and  information science professionals to submit  paper proposals on the theme:    “evidence based practice in social science  libraries: using research and empirical data  to improve service”    proposals should focus on one or more of  the following areas within social science  library settings:  • case studies that demonstrate the use of  evidence based practice to improve or  create new library services  • case studies that focus on the use of  evidence based practice to guide  professional development of librarians  • essays that provide theoretical or  practical approaches to evidence based  practice for social science libraries (this  may include the application of  qualitative and quantitative research  methodologies such as fieldwork and  observation, interviewing, qualitative  inquiry, meta‐analysis, evaluation  studies etc…)    important dates    please e‐mail abstracts (maximum 500  words) by 1 february 2007 to:    steve witt, standing committee chair,  swwitt@uiuc.edu      accompanied by the following information:  • abstract  • names of presenter(s)  • position or title of presenter (s)  • employer or affiliated institution   • mailing address  • telephone/fax numbers  • e‐mail address  • short biographical statement and  resume    notifications of abstracts acceptance will be  issued by 1 march 2006.  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:4  83 the deadline for submission of full papers is  1 may 2006.    important notes    regrettably, no financial support can be  provided, but a special invitation can be sent  to authors of accepted papers.    abstracts and papers must be submitted in  one of the official ifla languages (chinese,  english, french, german, russian, and  spanish).          news/announcements   eblip9, the ninth in the successful series of international conferences to promote evidence based approaches to library and information practice, has been awarded to philadelphia, usa to take place in 2017      2016. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     john wiggins, director of library services and quality improvement, and danuta a. nitecki, dean of libraries, both from drexel university libraries, coordinated the philadelphia bid.   criteria for selection included the location, facilities, regional and national support, the strength of the organizing team, and unique selling points. philadelphia was commended for the overall quality of its proposal including a good understanding of the requirements of the conference and excellent support for the eblip movement.   "drexel university is delighted to host the ninth international evidence based library and information practice conference,” said john wiggins, co-chair of the local organizing committee. “we look forward to a dynamic and engaging conference that will stimulate and excite new and existing eblip research community members.”   the eblip conference is a biennial event designed to promote the use of the best available evidence to improve library and information practice in all types of libraries. it aims to bring together practitioners, researchers, and students involved in critical and reflective information practice to think through new ways to address contemporary issues of evidence based practice in less isolated, more collaborative ways.   the first ebl(ip) conference took place in sheffield, uk in 2001. subsequent conferences have been hosted in edmonton (canada, 2003), brisbane (australia, 2005), chapel hill (usa, 2007), stockholm (sweden, 2009), salford (uk, 2011), saskatoon (canada, 2013) and brisbane (australia, 2015).   information on last year’s conference that was held at the queensland university of technology in brisbane australia can be found at http://eblip8.info   queries about eblip9 can be directed to eblip9@drexel.edu   evidence summary   positive correlation between academic library services and high-impact practices for student retention   a review of: murray, a. (2015). academic libraries and high-impact practices for student retention: library deans’ perspectives. portal: libraries and the academy, 15(3), 471-487. http://dx.doi.org/10.1353/pla.2015.0027   reviewed by: saori wendy herman, mlis, ahip education and liaison librarian hofstra northwell school of medicine hempstead, new york, united states of america email: saori.w.yoshioka@gmail.com   received: 5 dec. 2015     accepted: 12 feb. 2016      2016 herman. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to investigate the perceived alignment between academic library services and high-impact practices (hips) that affect student retention.   design – survey questionnaire.   setting – public comprehensive universities in the united states of america with a carnegie classification of master’s level as of january 2013.   subjects – 68 library deans or directors out of the 271 who were originally contacted.   methods – the author used qualtrics software to create a survey based on the hips, tested the survey for reliability, and then distributed it to 271 universities. library services were grouped into 1 of 3 library scales: library collection, library instruction, or library facilities. the survey consisted of a matrix of 10 likert-style questions addressing the perceived level of alignment between the library scales and the hips. each question provided an opportunity for the respondent to enter a “brief description of support practices” (p 477). additional demographic questions addressed the years of experience of the respondent, undergraduate student enrollment of the university, and whether librarians held faculty rank.   main results – the author measured pearson correlation coefficients and found a positive correlation between the library scales and the hips. all three library scales displayed a moderately strong positive correlation between first-year seminars and experiences (hip 1), common intellectual experiences (hip 2), writing-intensive courses (hip 4), undergraduate research (hip 6), diversity and global learning (hip 7), service learning and community-based learning (hip 8), internships (hip 9), and capstone courses and projects (hip 10). the library collections scale and library facilities scale displayed a moderately strong correlation with learning communities (hip 3) and collaborative assignments and projects (hip 5). the library instruction scale displayed a strong positive correlation with hip 3 and a very strong positive correlation with hip 5. each of the positive correlations was of high significance. as the rating of library alignment with each hip increased, so did the total rating of each library scale. along with the quantitative data, various themes for each hip relating to the library’s support practices emerged from the qualitative feedback. no significant trends were noted from the demographic questions.   conclusion – library deans or directors can utilize the conceptual framework presented in this study to connect the impact of library services to terminology and practices commonly understood by university administrators. further research using the conceptual framework would benefit future discussion on how academic libraries measure impact or success of their library services.   commentary   this study presents a fascinating perspective on the perceived correlation between library services and student retention rates. as the author indicates, there are a number of studies that examine the relationship between libraries and student retention, but a vast majority of them focus on a student’s library usage behaviour rather than library services as a whole (soria, fransen, & nackerud, 2013). furthermore, none of these studies attempt to study the alignment with the hips.   the reviewer critically appraised the article using glynn’s ebl critical appraisal checklist (2006). the overall validity was 76% and so this study falls within the range for validity. section validity yielded 80% for population, 67% for data collection, 100% for study design, and 67% for results. the percentage for the data collection and results sections did not fall within the range for validity and should therefore be questioned.   the author does not include the survey instrument in the publication. the sample question that appears in the manuscript provides one example of the instructions and questions posed on the instrument, but was not sufficient enough for the reviewer to conclude whether all questions posed were clear enough to elicit precise answers. the author provides a link to additional information on the reliability testing for the instrument, but an attempt to access the provided url resulted in an error, therefore making the supplemental material unusable. the exclusion of the instrument compromised the data collection validity. on a separate note, it should be common practice for an author to include the survey instrument, in its entirety when possible, as an appendix, supplement, or table. this practice allows for transparency and promotes reproducibility.   for the results section, some but not all variables were addressed. furthermore, some variables, such as the demographics of the respondents, were only analyzed as a means to identify future research topics rather than to identify the impact of these variables within the study. in addition, the results were not externally valid. the target population of the study is limited to library deans or directors in the united states of america with a carnegie classification of master’s level as of january 2013. the reviewer agrees with the author that the population is not representative of all users. as a result, the findings cannot be generalized to a broader population. the study, however, can be generalized and applied to institutions with different carnegie classifications.   this article presents a compelling framework to align the perceived impact of library services to student retention concepts. with some modifications, this study is worth exploring for future research. library deans and directors should take note of this research as it provides a unique process for measuring library impact.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154   soria, k. m., fransen, j., & nackerud, s. (2013). library use and undergraduate student outcomes: new evidence for students’ retention and academic success. portal: libraries and the academy, 13(2), 147-164. http://dx.doi.org/10.1353/pla.2013.0010   soria, k. m., fransen, j., & nackerud, s. (2014). stacks, serials, search engines, and students’ success: first-year undergraduate students’ library use, academic achievement, and retention. journal of academic librarianship, 40(1), 84-91. http://dx.doi.org/10.1016/j.acalib.2013.12.002       evidence based library and information practice 2011, 6.3 96 evidence based library and information practice news/announcements call for expressions of interest to host the 7th international evidence based library and information practice conference in 2013 © 2011. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. the international advisory committee for the international evidence based library and information practice conference series warmly invites expressions of interest from individuals, organisations and associations working in the public sector (e.g. academic, school, local and regional authorities, national associations) to host the 7th international evidence based library and information practice conference in 2013. more details and an expression of interest form are available from: http://eblipexpress.pbworks.com candidates wishing to host the conference must apply to andrew booth, the convenor of the iac, a.booth@sheffield.ac.uk monday 14th november 2011 using the official expression of interest template. applicants are encouraged to provide additional information supporting their bid. the eblip conference is a biennial conference designed to promote the use of best available evidence to improve library and information practice in all types of libraries and to thus address issues around inequitable use of resources, unacceptable provision of services and ineffective comprehension and use of information that continue to plague human society. it aims to bring together practitioners, researchers and students involved in critical and reflective information practice to think through new ways to address contemporary issues of evidence based practice in less isolated, more collaborative, ways. the first ebl(ip) conference took place in sheffield, u.k. in 2001. the success of this conference led to further successful conferences in edmonton (canada, 2003), brisbane (australia, 2005), chapel hill (u.s.a., 2007), stockholm (sweden, 2009) and salford (u.k., 2011). brief details are as follows: eblip conferences typically last between 2.5 and 3 days. they are typically accompanied by a continuing education programme of workshops of between 0.5 and 2 days duration. (the continuing education programme is optional for delegates and is not included in the registration fees for the conference). a social programme typically includes a welcome reception and conference dinner (both included in the registration fees) and additional mailto:a.booth@sheffield.ac.uk� evidence based library and information practice 2011, 6.3 97 optional pay‐as‐you‐go social events. day registration rates are typically provided to encourage attendance by local participants. conference venues have previously included academic, hotel and commercial conference facilities. local organising committees (locs) should be coordinated by a professional working in the public sector (e.g. academic, public, school, national, professional organisation). conferences are run on a not‐for‐ profit basis with any surplus (after all deductions) being handed on to the successive local organising committee. an exhibition of commercial sponsors is often provided as a necessary source of conference income. the conference programme is the responsibility of the international programme committee (ipc) co‐ordinated by an ipc chair/co‐chairs working in liaison with the loc. the conference programme will typically include up to five invited keynote speakers (whose travel, accommodation and registration is typically covered by conference income), twenty to fifty submitted papers (selected by the ipc) and fifteen to forty submitted posters (again selected separately by the ipc). awards for best paper and best poster are typically made by the ipc with commendation certificates provided to shortlisted presentations. an eblip international advisory committee (iac) (comprised of representatives from the previous hosts) will select the venue for the conference based on submissions of expressions of interest from potential locs. more details are available from http://eblipexpress.pbworks.com. http://eblipexpress.pbworks.com/� / evidence based library and information practice news/announcements call for expressions of interest to host the 7th international evidence based library and information practice conference in 2013 microsoft word edresp evidence based library and information practice 2008, 3:4 3 evidence based library and information practice editorial responsibilities © 2008. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. editor in chief: lindsay glynn associate editor (articles): alison brettle, denise koufogiannakis associate editor (evidence summaries): lorie kloda associate editor (classics): denise koufogiannakis associate editor (commentaries, news & announcements, ebl 101): lindsay glynn publication editor: katrine mallan copyeditors: heather ganshorn, heather pretty, lisa shen, priscilla l. stephenson, dale storie, darren wall, elizabeth zeeuw evidence summary   survey of canadian academic librarians outlines integration of traditional and emerging services   a review of: ducas, a., michaud-oystryk, n., & speare, m. (2020). reinventing ourselves: new and emerging roles of academic librarians in canadian research-intensive universities. college & research libraries, 81(1), 43–65. https://doi.org/10.5860/crl.81.1.43   reviewed by: laura costello virtual reference librarian rutgers university libraries new brunswick, new jersey, united states of america email: laura.costello@rutgers.edu   received: 1 june 2020                                                               accepted:  15 july 2020      2020 costello. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29789     abstract   objective – to identify new and emerging roles for librarians and understand how those new roles impact their confidence, training needs, and job satisfaction. to understand how librarians conceptualize the impact of these new roles on the academic enterprise.   design – electronic survey.   setting – academic research libraries at canadian research-intensive universities.     subjects – 205 academic librarians.   methods – an electronic survey was distributed to all librarians working at the 15 research-intensive universities in canada. archivists were included in this population, but senior administrators, such as university librarians, deans, and associate administrators, were not included. the 38-question survey was produced in english and french. five focus areas for emerging skills were drawn from the literature and a review of job postings. librarians were asked about their participation in particular activities associated with the different focus areas and about their training and confidence in those areas. the survey was sent to 743 librarians and had a 27% response rate with a total of 205 complete responses. librarians participated from each of the 15 research universities and institutional response rates ranged from 14% to 51%. survey monkey was used to distribute the online survey. cronbach’s alpha was used to measure reliability for each section of the survey and ranged from .735 in the confidence area to .934 in the job satisfaction area, indicating sufficient internal consistency. the data were analyzed using spss and rstudio.   main results – in the general area of research support, a majority (75%) of participants reported that they provided information discovery services like consultations and literature reviews, 28% engaged in grant application support, 27% provided assistance with systematic reviews, 26% provided bibliometric services, and 23% provided data management services. in the teaching and learning area, 78% of participants provided classroom teaching to students, 75% provided one-on-one instruction, 48% created tutorials, 47% taught workshops for faculty, and 43% conducted copyright consultations. only around half of participants offered digital scholarship services, and copyright consultations were the most frequently offered service in this area, with 36% of participants indicating that they offered this service. the area of user experience had the highest number of respondents, and the top services offered in this area included liaison services for staff and faculty (87%), library services assessment (46%), and student engagement initiatives (41%). in the scholarly communication area, 49% of respondents indicated that they provided consultation on alternative publishing models, including open access, and 41% provided copyright and intellectual property services.   the majority of librarians were confident that they could perform their duties in the five focus areas. teaching and learning had the highest confidence rate, with 75% of respondents indicating that they felt confident or very confident in their roles. digital scholarship had the lowest confidence rating, with only 50% indicating that they felt confident or very confident about these roles. the survey also asked participants about their training and skills acquisition in the five areas. most participants indicated that they acquired these skills through professional work experience and self-teaching. based on the calculations from the survey focusing on participation in new and traditional roles, 13% of librarian participants performed only new roles, 44% performed only traditional roles, and 44% performed some new and some traditional roles. additionally, 45% of librarians spent the majority of their time delivering traditional services, 19% delivering new services, and 36% dividing their time between new and traditional services. job satisfaction and new or traditional roles were also examined, and statistically significant results indicated that librarians performing new roles were more satisfied with assigned duties (p = 0.009084), more satisfied with opportunities for challenge (p = 0.02499), and less satisfied with opportunities for independent action (p = 0.02904). librarians performing new roles perceived a higher impact on scholarly communication (p = 0.02621) and supporting researchers (p = 0.0002126) than those performing traditional roles. librarians performing new roles perceived a lower impact on contributing to student success (p = 0.003686) and supporting teaching and learning at the classroom level (p = 0.01473) than librarians performing traditional roles.   conclusion – results demonstrate that librarians are still engaged in traditional roles, but new roles are emerging particularly in the areas of copyright and publishing, bibliometrics, online learning initiatives, and new communication strategies. job satisfaction and confidence in these roles are similar between traditional and emerging roles. overall, participants felt that they had a significant impact on the academic enterprise when performing new or traditional roles but that the roles had different areas of impact. this study is meant to be a baseline for future investigations in the trends and developments of roles for canadian librarians. the survey and data are available from the university of manitoba’s dataverse repository: https://doi.org/10.5203/fk2/rhoffu   commentary   this study focuses on the new and emerging roles for canadian academic librarians. the results represent a strong foundation of traditional library services, services that have evolved to meet emerging needs, and newly developed services that expand the role of the library. though librarians reported feeling confident and satisfied by both new and traditional roles, the free response section of the survey demonstrates a diversity of opinions about the role of new services in the work of academic libraries. some comments emphasized the importance of specialization and expertise in librarianship, while others advocated for greater institutional integration and collaboration within and beyond the library. the comments are aligned with the issues around hybrid roles in libraries raised by cox and corrall (2013).   the survey population included librarians at the 15 research-intensive universities in canada. the survey collected data from 27% of the eligible population. the respondents represented all 15 universities and had diverse specializations and a range of experience levels. while the small population size and lack of randomization mean that the study is not broadly generalizable, the population is sufficient to meet the goals of the researchers in establishing a baseline for librarians’ roles in canadian academic universities that further studies can develop. the instrument and results have been published open access to facilitate this process. glynn (2006) notes that publishing the instrument is critical to replication.    because of the broad goals of this study, the questions and results focus on a diversity of service types in academic libraries. the researchers endeavored toward straightforward language, though terms for new service strategies may have been novel for some participants, particularly in the research support area, which filtered on service classification before asking participants to select the service action. interviews or work time studies might expand on this research to help develop a deeper understanding of how librarians divide time between new and traditional roles. while this study focused only on research institutions, future research may want to address libraries of other types or focus on the experiences of librarians through qualitative methods.   this study is important both for establishing an understanding of emerging and developing roles for academic librarians and how these roles are applied in libraries. this study found that 44% of librarians provided both traditional and emerging services and that librarians predominantly learned new roles through self-study and on-the-job practice. this indication of the complexity and continuous development of academic library roles is intriguing, and further research into the way librarians manage this process may help stakeholders support this development toward the library roles of the future. this study may be of particular use to library administrators developing and supporting librarians in providing emerging services.   references    cox, a. m., & corrall, s. (2013). evolving academic library specialties. journal of the american society for information science and technology, 64(8), 1526–1542. https://doi.org/10.1002/asi.22847   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387–399. https://doi.org/10.1108/07378830610692154   evidence based library and information practice 2012, 7.3 71 evidence based library and information practice editorial improvements to evidence summaries: an evidence based approach lorie kloda associate editor (evidence summaries) assessment librarian mcgill university montreal, quebec, canada email: lorie.kloda@mcgill.ca 2012 kloda. this is an open access article distributed under the terms of the creative commonsattributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. with the current issue of evidence based library and information practice (eblip), we will have published over 250 evidence summaries. that’s an average of 8 per issue, or 32 per year. when the journal was created, the goal of publishing these summaries was to provide concise evidence to assist with knowledge transfer, with the eventual hope that readers would use the evidence to inform decision making and practice (koufogiannakis, 2006). over the past seven years, this goal remains unchanged, and members of the editorial board have undertaken two separate studies to determine the effectiveness of evidence summaries in meeting it. the first study (kloda, koufogiannakis, & mallan, 2011), a content analysis, revealed that evidence summaries tend to convey mixed messages about the quality and applicability of the research being summarized; while the second study (kloda, koufogiannakis, & brettle, 2012) entailed the development and testing of a tool to assess the impact of evidence summaries on professionals’ knowledge, practice, and user community. preliminary findings from the second study suggested that evidence summaries add to librarians’ knowledge and, occasionally, to their professional practice and decision making. there is, as yet, little indication that evidence summaries contribute to larger changes in the workplace, or that these changes impact users of library and information services. evidence summaries are brief critical appraisal reviews of current research articles. the summaries follow a standardized format to ensure consistency and ease of use for readers. they are produced by our team of approximately 20 writers, who are selected through an application process. all evidence summaries undergo double-blind peer review by at least two reviewers before being considered for acceptance. typically, revisions are required before an evidence summary is accepted. evidence summaries cover a range of topics in all domains relevant to library and information practice, including education (including information literacy), collections, reference, management, information storage and retrieval, professional issues, and scholarly publishing, and cover a range of settings, including academic, health, school, public, and specialized libraries. mailto:lorie.kloda@mcgill.ca http://creativecommons.org/licenses/by-nc-sa/2.5/ca/%29 evidence based library and information practice 2012, 7.3 72 the 2011 content analysis demonstrated that the evidence summaries published in eblip tended to have rather lengthy commentaries in which authors focused on describing all the shortcomings of the methodology. little space remained for discussion of applicability of research findings to information practice. we therefore decided to make some changes to the evidence summaries format, changes which the editorial team hope will be received as improvements by the journal’s readers, and perhaps increase the readership of evidence summaries. in this issue, we introduce eight evidence summaries that follow revised guidelines. each summary now includes:  a title describing the major finding(s) of the study being appraised.  a structured abstract providing an overview of the key elements of the research. the format has been revised to be more concise, highlighting each study’s features more generally and summarizing main findings. this revised format is designed to allow the reader to quickly determine the relevance and importance of a study.  a commentary, generally not exceeding 450 words, which briefly situates the research study in the broader context of research on the topic, and addresses the strength of the evidence provided. the commentary concludes with a statement on the significance of the research as well as its practical applications. to date, eblip has made incredible progress in creating and disseminating these research summaries, and we will continue to do so. as the associate editor for evidence summaries, it is my hope that these improvements will be welcomed by our readers. i welcome your feedback, and suggestions for further improvements. references kloda, l., koufogiannakis, d. & brettle, a. (2012, may). assessing the impact of evidence summaries in library and information studies: a mixed methods approach. medical library association annual meeting, seattle, wa, usa. retrieved 7 sept. 2012 from http://www.slideshare.net/lkloda/klo da-mla-2012-impact kloda, l. a., koufogiannakis, d., & mallan, k. (2011). transferring evidence into practice: what evidence summaries of library and information studies research tell practitioners. information research, 16(1), paper 465. retrieved 7 sept. 2012 from http://informationr.net/ir/161/paper465.html koufogiannakis, d. (2006). small steps forward through critical appraisal. evidence based library and information practice, 1(1), 81-82. retreived 7 sept. 2012 from http://ejournals.library.ualberta.ca/in dex.php/eblip/article/view/26/64 http://www.slideshare.net/lkloda/kloda-mla-2012-impact http://www.slideshare.net/lkloda/kloda-mla-2012-impact http://informationr.net/ir/16-1/paper465.html http://informationr.net/ir/16-1/paper465.html http://ejournals.library.ualberta.ca/index.php/eblip/article/view/26/64 http://ejournals.library.ualberta.ca/index.php/eblip/article/view/26/64 microsoft word es_1774 evidence based library and information practice 2008, 3:3 61 evidence based library and information practice evidence summary undergraduates prefer federated searching to searching databases individually a review of: belliston, c. jeffrey, jared l. howland, & brian c. roberts. “undergraduate use of federated searching: a survey of preferences and perceptions of value-added functionality.” college & research libraries 68.6 (nov. 2007): 472-86. reviewed by: genevieve gore coordinator, escholarship@mcgill mcgill university library montreal, qc, canada e-mail: genevieve.gore@mcgill.ca received: 28 may 2008 accepted: 13 july 2008 © 2008 gore. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – to determine whether use of federated searching by undergraduates saves time, meets their information needs, is preferred over searching databases individually, and provides results of higher quality. design – crossover study. setting – three american universities, all members of the consortium of church libraries & archives (ccla): byu (brigham young university, a large research university); byuh (brigham young university – hawaii, a small baccalaureate college); and byui (brigham young university – idaho, a large baccalaureate college) subjects – ninety-five participants recruited via e-mail invitations sent to a random sample of currently enrolled undergraduates at byu, byuh, and byui. methods – participants were given written directions to complete a literature search for journal articles on two biology-related topics using two search methods: 1. federated searching with webfeat® (implemented in the same way for this study at the three universities) and 2. a hyperlinked list of databases to search individually. both evidence based library and information practice 2008, 3:3 62 methods used the same set of seven databases. each topic was assigned in random order to one of the two search methods, also assigned in random order, for a total of two searches per participant. the time to complete the searches was recorded. students compiled their list of citations, which were later normalized and graded. to analyze the quality of the citations, one quantitative rubric was created by librarians and one qualitative rubric was approved by a faculty member at byu. the librariancreated rubric included the journal impact factor (from isi’s journal citation reports®), the proportion of citations from peerreviewed journals (determined from ulrichsweb.com™) to total citations, and the timeliness of the articles. the facultyapproved rubric included three criteria: relevance to the topic, quality of the individual citations (good quality: primary research results, peer-reviewed sources), and number of citations. data were then analysed using anova and manova. finally, librarians at the acrl 13th national conference presentation were polled about their perceptions of the time savings of federated searching, whether the method meets undergraduates’ information needs, undergraduate preference for searching, and the quality of citations found. main results – seventy percent of all participants preferred federated searching. for all schools combined, there was no statistically significant difference between the average time taken using federated searching (20.34 minutes) vs. non-federated searching (22.72 minutes). for all schools combined, there was a statistically significant difference in satisfaction of results favouring federated searching (5.59/7 vs. 4.80/7 for non-federated searching, α = .05). according to the librarian-created rubric, citations retrieved from federated searching were a statistically significant 6% lower in quality than citations retrieved from non-federated searching (α = .05). the faculty-approved rubric did not detect a difference in the quality of the citations retrieved using the 2 methods. librarians’ perceptions as assessed at the acrl 13th national conference presentation generally matched the authors’ findings. conclusion – overall, students in this study preferred federated searching, were more satisfied with the results of federated searching, and saved time (although the savings were not statistically significant). the quality of citations retrieved via both methods was judged to be similar. the study provides useful information for librarians interested in users’ experiences and perceptions of federated searching, and indicates future studies worth conducting. commentary this article includes a good review of the current state of federated searching as well as the literature related to it. overall, the study provides useful results in an area about which many assumptions have been made in the literature but few user surveys have been soundly conducted to validate them. the study used a strong research methodology to analyze undergraduates’ experiences and perceptions of federated vs. non-federated searching. the population was limited to undergraduates and the results may not therefore be generalizable, although the authors speculate that the results would hold. the context of the study (i.e. its implications for the renewal or nonrenewal of the webfeat® licence within the ccla) may have introduced an element of bias. participants were randomly contacted but it is unclear how randomization was done or whether there was a self-selection bias in the type of respondent who agreed to participate (response rates were not provided). the number of participants in evidence based library and information practice 2008, 3:3 63 each group was sufficient to conduct statistical tests of significance on the data. written instructions were given to participants to minimize inter-observer and intra-observer bias (it is unclear how many observers were used, however). instruments were included in appendices, although no screenshot of the federated search implementation was provided and the authors acknowledged that the implementation could have had an effect on the results of the study. the survey questions were clearly posed. it infrastructure differences between the institutions was not addressed as a possible source of noise when comparing time savings between the two methods. it is unclear whether the two rubrics used to analyze quality were validated, and interrater reliability was not assessed for the faculty-approved rubric: it fell upon one undergraduate biology student to assess the quality of the citations retrieved. one could ask why the librarian-created rubric used the impact factor for journal citations instead of using citation counts for each article, although citation counts would have required normalization for publication date. in addition, one might ask why the statistically significant difference favouring individual database searches over metasearching, captured by the librarian-created rubric, was essentially dismissed by the authors in the discussion section. it is also unclear whether the number of databases being searched affected the participants’ perceptions of federated vs. non-federated searching. the number was based on the average number of databases listed by subject librarians and assumed to be optimal. with respect to the librarian survey conducted at the acrl conference, it is unclear how the respondents were chosen from the audience. this research contributes to our understanding of undergraduates’ experiences and perceptions of federated vs. non-federated searching. this study is much needed given the growing presence of federated search tools in libraries and can provide empirical support to institutions thinking about purchasing such a system for their own library. microsoft word news_ingenta.doc evidence based library and information practice 2006, 1:3  104 evidence based library and information practice     news      ingenta research award       © 2006 eblip. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    purpose    the ingenta research award is given  annually by the library research round  table of the american library association  to support research projects about  acquisition, use, and preservation of digital  information.    example areas of research include:    • the analysis of online journal usage data  to develop conclusions and predictive  models which may be used by libraries  and publishers in determining future  behavior;  • an investigation of the issues  surrounding institutional archiving,  particularly costs, preservation and  securing the participation of faculty;  • a study of information seeking behavior  of readers and/or authors; and  • the development of future models for  verifying the relative usefulness of  publications.    other topics related to digital publications  will also be considered.     eligibility    applications are welcome from practicing  librarians, faculty and students at schools of  library and information science, and  independent scholars.     criteria    the ingenta award jury will evaluate  applications on the basis of the following  criteria:    • appropriateness of the proposed project  to understanding of seeking and use of  digital information;  • significance of the problem;  • design of the study;  • qualifications of the investigator(s); and  • realism of the timetable.    amount    the grant consists of up to $6,000 for  research and up to $1,000 for travel to a  national or international conference to  present the results of the research.   expenditures must directly support research;  the award does not cover indirect costs or  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:3  105 overhead. half of the research amount will  be paid within one month of the selection of  the awardee; the remaining half will be  provided approximately six months later  upon the receipt of a satisfactory progress  report as determined by the ingenta award  jury chair and the ala staff liaison to the  ingenta award jury.     how to apply    send:    1. proposal of no more than 6 double‐ spaced pages that provides:  • overall statement of the project  • relation of the project to previous  research  • research questions  • method/plan of investigation  • timetable for the work  • significance of the project  • plan to disseminate the results  2. budget (1 page)  3. curriculum vitae (2 pages)    deadline    all submissions must reach the ala  address on or before friday, march 31, 2007.   you will receive confirmation via e‐mail  within two days.     email the proposal, budget and curriculum  vitae to:    denise davis  american library association  50 east huron street  chicago, il 60611  phone: (800) 545‐2433, ext. 1‐4273  fax: (312)280‐4392  e‐mail: dmdavis@ala.org                      evidence summary   public librarians hold critical and evolving role as community facilitators of government information   a review of: zhu, x., winberry, j., mcbee, k., cowell, e., & headrick, j. s. (2022). serving the community with trustworthy government information and data: what can we learn from the public librarians? public library quarterly, 41(6), 574–595. https://doi.org/10.1080/01616846.2021.1994312   reviewed by: lisa shen business librarian & director of public services newton gresham library sam houston state university huntsville, texas, united states of america  email: lshen@shsu.edu   received: 1 june 2023                                                                accepted:  17 july 2023      2023 shen. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30381     abstract   objective – to understand public librarians’ experiences in addressing their communities’ government information and data needs.   design – semi-structured interviews.   setting – 4 public county library systems in 2 southern states in the united states in early 2019, prior to onset of the covid-19 pandemic.   subjects – 31 public service librarians, recruited through a combination of theoretical and convenience sampling strategies.   methods – the researchers conducted individual interviews, ranging between 30 and 60 minutes, with each participant. interview recordings were transcribed and processed through the qualitative data software nvivo, using a grounded theory approach with open inductive coding followed by thematic analysis.   main results – six major findings were identified through thematic coding, including variability and complexity of reference questions, diversity in patron demographics, need for advanced knowledge of the local community context, preparedness of librarians to provide reference consultation for government information, balance between information and interpretation, and trust issues related to government sources. challenges related to digital literacy level was a shared factor across multiple themes, as patrons’ government information needs are increasingly impacted by their ability to access web, mobile, and computer technologies, navigate online resources, and interpret bureaucratic vocabulary. some librarians also expressed their own eroding trust towards the validity of government sources, such as climate change information from the environmental protection agency under the trump administration.   conclusion – a majority of the findings were consistent with past literature, including the breadth and depth of varying government informational needs of public library patrons and the trust patrons have for their public libraries and librarians. researchers also noted limited initiatives by public libraries to proactively educate patrons about open data or misinformation and recommended that libraries and library science educators better prepare current and future librarians for their role as government information mediators.   commentary   this study provides timely advocacy for the value of public libraries in assisting their communities with identifying, accessing, and interpreting government information and data in an increasingly complex information and technology landscape. even though data was collected prior to covid-19, the findings remain relevant and applicable for the current socio-political context. moreover, the authors skillfully provided a concise literature review of pertinent research, which would be valuable to those interested in public libraries’ roles in promoting equity and inclusion through government information reference.   two appraisal tools were consulted to assess this study: glynn’s ebl critical appraisal checklist (2006), which provided a framework for the review, and letts et al.’s critical review form (2007), which supplemented qualitative elements not fully addressed by glynn’s more quantitatively oriented checklist. overall, the study’s purpose was clearly stated, and the selections of theoretical framework and methodology were appropriate for the research questions. the researchers also provided sound rationale for population selection and recruitment decisions, and they addressed limitations for the generalization of the findings. informed consent was obtained, and data analysis tools and approaches were clearly identified.   yet, despite many elements that strengthen validity of the research, several omissions weakened the transferability of the study (letts et al., 2007). firstly, the authors did not provide interview questions and a protocol. it is also unclear which of the five coauthors served as interviewers of the 31 subjects and which participated in coding and analysis. details on how the researchers approached norming during their coding process were vague; for example, researchers noted that seven general areas for analysis had initially emerged, but they did not elaborate on how they selected the final themes from those topics.   finally, the researchers may not have fully considered how their own positionalities could have influenced their analysis and interpretation of the results. for instance, the authors had expected to find public librarians proactively “fighting misinformation” and “promoting open data” and were surprised that interviewees showed “a lack of concern” in those topics (zhu et al., 2022, p. 589). however, given the digital divide and digital literacy challenges experienced by many public library patrons and the increasingly polarizing political climate in many communities, it may not be reasonable to assume that public librarians would be as actively engaged in emerging lis topics as a research group consisting of university faculty, doctoral students, and recent graduates of mlis programs.   because of these validity limitations, readers are advised to consider the representativeness of the discussion with some reservation. nonetheless, this article highlights a critical and unique role of public librarians as trusted facilitators and mediators of government information and data. the researchers also convincingly identified gaps in current lis curriculum and the need for educators and administrators to provide specialized government information reference, information literacy, and digital literacy curriculum or professional development opportunities for aspiring and practicing librarians.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387–399. https://doi.org/10.1108/07378830610692154   letts, l., wilkins, s., law, m., stewart, d., bosch, j., & westmorland, m. (2007). critical review form – qualitative studies (version 2.0). http://www.peelregion.ca/health/library/eidmtools/qualreview_version2_0.pdf    zhu, x., winberry, j., mcbee, k., cowell, e., & headrick, j. s. (2022). serving the community with trustworthy government information and data: what can we learn from the public librarians? public library quarterly, 41(6), 574–595. https://doi.org/10.1080/01616846.2021.1994312     microsoft word comm_1690_birdsall evidence based library and information practice 2008, 3:2 65 evidence based library and information practice commentary the chiasmus of librarianship and collaborative research for evidence based practice william f. birdsall library consultant bedford, nova scotia canada e-mail: billbirdsall@accesswave.ca received: 28 april 2008 accepted: 28 may 2008 © 2008 birdsall. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. introduction it is my contention librarianship will experience throughout much of the twenty-first century a profound chiasmus or flip from the universal homogeneity of values and practice currently dominating the profession, to the particularistic heterogeneity characteristic of nineteenth century librarianship. this transformation will arise out of the necessity to meet the needs of specific communities and their unique ways of knowing through the collaborative development of evidence based practice for multiple knowledge systems. the paper concludes that if evidence based practice is to make a substantive contribution to this the chiasmus of librarianship it will need to embrace research methodologies developed in collaboration with multiple communities of knowing. from particularistic heterogeneity to universal homogeneity up to the closing decades of the nineteenth century, library practice was locally orientated to meet the needs of specific communities. as there were no professional associations or institutions promulgating national values, standards or methodology it was a time of much experimentation. consequently, library practice was heterogeneous, responding to the needs of particular communities. this heterogeneous environment began to change as a result of the unprecedented acceleration of social, economic and technological developments during the closing decades of the nineteenth century. the creation of a political, economic and cultural mass society was in the making. one consequence of this change was the creation of a sufficient critical mass of evidence based library and information practice 2008, 3:2 66 libraries and practitioners to launch the professionalizing of librarianship. as other professions, foremost medicine, made science the foundation of their professional legitimacy, librarianship also adopted this strategy. since the eighteenth century, science was gaining in ascendancy as the prevailing way of knowing among advanced western countries. with the beginning of the twentieth century, science was close to achieving the status of the dominant way of knowing. because of its significant contributions to world war ii military efforts, science gained immense power as the prevailing knowledge system. the scientific way of knowing strives to formulate universal laws culminating in a universal theory of everything. toward this end, science adheres to strict methods of experimentation, hierarchical taxonomies and controlled vocabularies. its ethos, then, embodies universality, rationality and efficiency. as science became accepted as the superior way of knowing other ways were characterized as subjective, irrational myths or unverifiable intuitions and their communities of knowing marginalized. the scientific valuing of universality and rationale efficiency became hallmarks of librarianship. indeed, the father of librarianship, melvil dewey, had an obsessive mania for promoting efficiency. librarianship became committed to universally applied homogeneous values, techniques, and bureaucratic efficiency. echoing science, librarians dreamed of creating the universal library of everything (a dream they now share with google inc.). encouraged by emerging professional associations and university based training programs, library staff became local representatives of a larger cosmopolitan ethos infused with a universal, scientific rationality. by the 1930s, library practice was promoted as a library science; schools of library economy became schools of library science. as the twentieth century drew to a close, the drive for homogenization led to ever greater centralization and standardization in the practice of librarianship. individual libraries were incorporated into ever larger administrative jurisdictions, consortia and networks. striving for greater efficiencies, methods and services were developed remote from any substantive input by local library clientele or from those with their own ways of knowing. when knowledge generated by communities embracing other ways of knowing and organizing knowledge could not be easily incorporated into librarianship’s science-based framework, they were marginalized or ignored. the movement promoting evidence based library and information practice is a significant effort to strengthen the link between research and practice. however, it is crucial, in my mind, for those promoting evidence based practice to consider its methodology in the larger context of the transformation of librarianship; that is, a shift back to heterogeneous library practice arising out of the recognition there are multiple communities of knowing. recognizing that there are ways of knowing other than science is not a question of one way being more valid than another. rather, there are many communities of knowing, each having a right to communicate according to knowledge evidence based library and information practice 2008, 3:2 67 systems that meet their particular needs; needs that are currently unmet by contemporary library practice. chiasmus of librarianship a shift from homogeneity back to heterogeneity in librarianship can be seen in a larger context of social, cultural and technological change, a process characterized by marshall mcluhan as chiasmus. chiasmus is defined by the oxford english dictionary as “a grammatical figure by which the order of words in one of two parallel clauses is inverted in the other." an example from the bible is matthew 23:11-12: “whoever exalts himself will be humbled, and whoever humbles himself will be exalted." an earthier example is mae west’s quip: “it's not the men in my life, it's the life in my men." in his analysis of change, marshall mcluhan translated the rhetorical idea of chiasmus into a “law” of reversal: when any human artifact, be it an idea, art, technique, or process, is pushed to its limit, there is a reversal or flip to characteristics of an earlier state. however, chiasmus does not mean a total return to old ways. rather, it is a metamorphosis of them to fit the new context. as mcluhan observed: a breakdown can lead to a breakthrough. in the laws of media (1988) and the global village (1989) mcluhan illustrated how chiasmus could be demonstrated for a whole range of ideas and techniques from acoustic space to the zipper. to assert that library and information practice is embarking on a process of chiasmus does not mean it is doomed to some kind of deterministic law of change. mcluhan asserted determinism can be avoided if attention is paid to the potential implications of change. using mcluhan’s application of chiasmus to analyzing change stimulates us to pay attention to and prepare for a profound transformation in twenty-first century practice: a chiasmus of librarianship from the state of universal homogeneity of values and practice characteristic of twentieth century librarianship to a state of particularistic heterogeneity during the twenty-first century. interestingly, mcluhan predicted that with the expansion of global electronic communication westerners, accustomed to the science mode of perceiving the world, would experience a chiasmus to a world view not unlike indigenous peoples. indeed, mcluhan expected that as society moved further into the era of the home computer and global interactive communication, the mass society library could become obsolete. however, obsolescence does not have to be its fate if practitioners embark on finding new models of practice embracing collaboration with multiple communities of knowing. mcluhan’s reference to indigenous peoples is prescient as they serve as an excellent example to illustrate how ways of knowing can differ from the sciencebased model. indigenous knowledge systems are the most widely recognized of diverse communities of knowing. indigenous knowledge systems as communities of knowing according to unesco the world-wide population of indigenous peoples is about 350 million individuals in over seventy countries representing over 5,000 languages and cultures (unesco). indigenous peoples provide a vivid and readily available example of distinct knowledge systems because they are especially successful in evidence based library and information practice 2008, 3:2 68 articulating how their ways of knowing differ from the prevailing science way of knowing. there is an extensive and growing body of literature on all aspects of indigenous ways of knowing and knowledge organization dealing with such issues as intellectual property, cultural heritage preservation, education, language rights, and so forth (a few examples are: semali and kincheloe; battiste and henderson; battiste 2000; battiste 2002; mihesuah and cavender ; riley; bastien, mistaken, kremer; anderson). nonetheless, because of their uniqueness and diversity indigenous ways of knowing have been considered existing outside the mainstream science mode of knowing and, therefore, of local interest only to their particular communities. a brief comparison between contemporary library and information practice and those of indigenous peoples demonstrates how other ways of knowing can possess values and practice dramatically different from those of twentieth century librarianship. following the linear rationalism of the science model, librarians support the intellectual property rights of the individual creator; see knowledge as a public good that should be universally accessible; are committed to its preservation as fixed, authenticated texts whose accumulation forms a linear narrative embodied in a textual canon. in stark contrast, there is a great diversity among indigenous knowledge systems. their unique ways of knowing often derive from spiritual cosmologies intimately connected to a longstanding identification with a specific geographic locale. the community may rely on the oral transmission of knowledge through community elders. because of this mode of transmission the body of knowledge is flexible, dynamic, organic, and lacking in conical texts. as knowledge is generated out of the community’s unique experience with its specific locale, space is privileged over the linear time of scientific rationalism (shreve 372). while, the body of knowledge can be the possession of the entire community over any individual, all or parts of it can be considered sacred, only accessible to select members of the community. modes of preservation may be through not only memory and oral transmission but also ceremonies and ritual and a wide variety of artifacts (birdsall and shearer). the characteristics found among diverse indigenous knowledge systems differ substantially from current library and information practice. consequently indigenous peoples feel compelled to take matters into their own hands (battiste 2002; battiste and henderson; roy). these efforts include establishing their own library organizations such as the american indian library association and the torres strait islanders library and information network. there are biennial conferences of the international indigenous librarians’ forum. while indigenous peoples serve as a vivid example of a diversity of communities whose knowledge systems can differ substantially form the prevailing science knowledge system there is evidence there are other ways of knowing deserving attention. a recent study undertaken by the canadian association of research libraries (carl) revealed that multiple ways of knowing can be found among various ethnic, linguistic, disciplinary, and cultural communities. in the carl evidence based library and information practice 2008, 3:2 69 study a panel of canadian academic researchers identified five scholarly communication research priorities, the first being knowledge systems. the panel stressed that the creation of knowledge “takes place in complex cultural, linguistic, and regional contexts.” in addition to canadian indigenous knowledge systems, the panel recognized other sources of knowledge, such as dance, theatre and linguistic and cultural communities can also constitute unique communities of knowing. the panel concluded there is a need for strategies of knowledge organization that respond to the heterogeneity of the origins and uses of knowledge. such strategies should be formulated in collaboration with the specific communities of knowing (birdsall et al.). shifting back to a particularistic heterogeneity there are several examples that can be interpreted as modest evidence of a growing awareness of the challenges of multiple ways of knowing and the need for collaboration to address them. the success of indigenous peoples in gaining recognition of the diversity of their many ways of knowing can itself be seen as evidence of a shift to greater awareness of heterogeneous ways of knowing. their success is especially noteworthy in its expression in the human rights discourse at the international level (ivison and patton,). after twenty years of negotiations the united nations adopted in september, 2007, a declaration on the rights of indigenous peoples (drip) by a vote of 143 to 4 (australia, canada, new zealand, and the united states opposing). the declaration includes many articles giving communities of indigenous peoples rights relating to preserving and accessing all forms of their cultural expression and artifacts (united nations). the drip specifically addresses issues critical to modern librarianship including access to information, intellectual property, privacy, modes of preservation, authenticity of knowledge, control of the media, and cultural development (birdsall 2008). as the drip is a declaration only it does not have legal status but it does carry much moral force to support those at the national and international levels working to get the declared rights entrenched in national and international law. as a result of the efforts of the indigenous peoples movement there is in the library and information field increased efforts to respond to the needs of indigenous peoples. i already noted the issue arose in the carl study on scholarly communication. international and national professional associations are responding: the international federation of library associations and institutions (ifla) created a section on library services to multicultural populations and adopted a multicultural library manifesto; the canadian library association established an interest group on library and information needs of native peoples and passed an aboriginal services resolution. schools of library and information studies are beginning to incorporate courses into their curriculum in response to the greater awareness of the needs of indigenous peoples. the university of british columbia school of library, archival, and information studies launched a first nations curriculum concentration in its master’s programs. library systems are attempting to establish enhanced service for evidence based library and information practice 2008, 3:2 70 indigenous peoples. for example, a library services for saskatchewan aboriginal peoples (lssap) committee was established in 1991. because archivists have always had to be conscious of the contextual sources of cultural records they appear to be moving faster towards a greater sensitivity to multiple ways of knowing and the need for collaboration than other groups within library and information practice. for example, katie shilton and ramesh srinvasan, university of california-los angeles graduate school of education and information studies, examine “participatory appraisal and arrangement for multicultural archival collections” (shilton and srinvasan). they note that archives have traditionally “appropriated the histories of marginalized communities, creating archives about rather than of communities” (shilton and srinvasan 2). as a consequence, “members of marginalized groups have taken preservation into their own hands, building archives and museums devoted to community history” (shilton and srinvasan 5). with regard to the processes of arrangement and description of archives they advocate “participatory processes to facilitate the preservation of representative, empowered narratives” (shilton and srinvasan 3). they envisage “participatory archiving” practice as a system that should draw upon community knowledge “through methods of participatory design, a movement within the information technology research world that positions users as the designers of their own systems” (shilton and srinvasan 8). examples noted of archivists collaborating with specific communities include the south east asian archives at the university of california (uc) irvine, the chicano studies archives at uc los angeles and santa barbara, and the northern coast indian collection at the portland museum of art. further evidence library and information practice may be moving towards greater heterogeneity is the web 2.0 movement and its derivative library 2.0 a common theme throughout the discourse surrounding these concepts is the call for an increased participatory role for those for whom systems are being designed. there are references to “an architecture of participation;” “harnessing collective intelligence;” “rich user experience;” “trusting users as co-developers;” development in a state of “perpetual beta”; “the wisdom of crowds” (o’reilly). others talk about usercentered participation in the creation of content and services; socially rich communication between users; being communally innovative (maness). principles enunciated for library 2.0 include “freeing of data,” “participative,” “work for the user,” “sharing,” “communication and facilitating community,” “trust” (miller, 2005; miller, 2006). library 2.0 values are significant because of the great extent to which they include: user participation in research and development; a non-hierarchical relationship through collaboration among users, developers, and service providers; a commitment to community building and to the needs of distinct communities; and a recognition that technological developments are always in a state of “beta development” (birdsall 2007) evidence based library and information practice 2008, 3:2 71 the above examples are examples of a growing awareness of the need for greater collaboration with diverse communities of knowing. but they do not as yet represent a fundamental challenge to the current institutional and professional structures of twentieth century librarianship. going for the breakthrough: collaboration with communities of knowing the universalizing mode of knowledge organization has reached its limit of effectiveness as a model for librarianship in a global environment of multiple ways of knowing. the traditional gatekeeper role to information cherished by librarians during the twentieth century will become obsolete. the time is ripe for chiasmus. remembering mcluhan’s admonition, this chiasmus should not be seen as a breakdown but as an opportunity for a breakthrough to a fundamental transformation of library and information practice based on collaborative evidence based practice with multiple communities of knowing to meet their specific knowledge needs. this collaboration will be a profound change in the power relationship between professional and client. for example, despite the creation of the lssap initiative in saskatchewan, it failed to attract as much participation of aboriginal peoples communities as was hoped for. consequently, a key recommendation of a minister's advisory committee on library services for aboriginal peoples, created in 2001, is that: first nations retain responsibility for developing and funding local, onreserve library services, in cooperation with the regional libraries and the federal government. it is their responsibility to determine the kind and level of public library services they wish to establish on reserves, such as, stand alone public libraries, school-housed public libraries, book mobiles, book drops, computer, or van delivery (sinclair-sparvier). edgardo civallero, national university of cordoba, argentina, also stresses the necessity to address the needs of the diversity of indigenous people’s communities through a collaborative strategy that acknowledges each community knows best its problems and what it wants for the future. he states “…it is both necessary and urgent to hear the voices of the final users, those with whom we want to collaborate.” consequently, “any collaboration program should start just by listening [to] them, understanding them and working with them (grass-root development)” (civallero 4). in this context he warns that librarians should not see themselves as “heroes or saviours, but as helping hands...” a heterogeneous model of librarianship will require building a new professional knowledge base through collaborative, participatory research with different communities of knowing. as civallero observes: it is not a matter of simply adapting a widely known and already used model to special circumstances: strange transplants are meant to fail and to be refused by any organic system. it is about creating a new, unique, imaginative model, likely to be continuously evidence based library and information practice 2008, 3:2 72 adapted to the expected development of any human group by using action-research” (his emphasis) (civallero 5). cora weber-pillwax, a canadian authority on indigenous research methodologies, also advocates actionresearch and the need for research objectives and methodology to contribute to the real community needs (weber-pillwax). indeed, a key element of participatory action research is the shift in power from the current one of power being held solely by the researcher/practitioner to being a model where power is held by the diverse communities of knowing, a situation that raises professional and political challenges to traditional models of research (cornwall and jewkes; grenier). recognizing there are multiple communities of knowing requires the collaborative research of the fundamentals of contemporary library practice, addressing such questions as: what is the appropriate professional/client relationship? what constitutes authentic knowledge? who “owns” it? how can knowledge be most effectively distributed and in what forms? how should access be provided and to whom? what are the appropriate modes of preservation and the appropriate institutions to do so? what happens when knowledge is translated from one medium to another? who is responsible for developing the methodologies of a specific community’s knowledge needs and system? conclusion the evidence based library and information practice movement is taking the lead in attempting to link research and practice. however, in my view, if it is to make a substantive contribution to a twenty-first century shift to heterogeneous practice it must incorporate into its research ethos collaboration with specific communities of knowing. the transformation to a model of serving communities of knowing is a fundamental challenge to the ethos, institutional and legal structures, and methodology of library and information practice. this will not be an easy objective to achieve as resistance could arise from a number of fronts. the scientific ethos is strongly entrenched throughout library and information practice, training, research, values, and institutions. also, the institutional structure of library and information practice is what could be called jurisdictional, that is, libraries are funded by and serve specific legal jurisdictions: municipalities, counties, states; higher educational institutions; elementary and secondary schools; corporate bodies. they are not orientated to serve specific communities of knowing. the shift to serving communities of knowing both within and outside traditional jurisdictions raises many legal, political, and financial issues. there is also the cult of efficiency within librarianship to overcome. any major change involves experimentation, risk, failure and is time. shilton and srinivasan warn that participative methodologies can be “particularly labor-intensive” (shilton and srinivasan 12). while there may be resistance to moving to a particularistic evidence based library and information practice 2008, 3:2 73 heterogeneous model, the adoption of the declaration on the rights of indigenous peoples could be interpreted as a harbinger of other communities of knowing demanding a role in the development of their own knowledge systems, a harbinger to which library and information practitioners should direct their attention and energy. as canadian lawyer and library trustee merillee rasmussen reminds us that in the current environment of global electronic communication it is possible for individuals “to belong to a community in an era when the nature of community is changing” (rasmussen 142). thus, we could see, in addition to the demands of traditional communities of knowing, new communities calling for collaborative initiatives for developing their unique knowledge systems. the closing decades of the nineteenth century were a period of tremendous creativity for the emerging profession of librarianship, including the annus mirabilis of 1876. perhaps the closing decades of the twenty-first century will witness another burst of comparable creativity, thereby completing the chiasmus of a evidence based, collaborative librarianship. works cited anderson, jane. 2005. access and control of indigenous knowledge in libraries and archives: ownership and future use. new york: american library association and the macarthur foundation, 2005. http://conservationcommons.or g/media/document/docui64cij.pdf. bastien, betty, jurgen w kremer, and duane mistaken. blackfoot ways of knowing: the worldview of the siksikaitsitapi. calgary, alberta: university of calgary press, 2004. battiste, marie. ed. reclaiming indigenous voice and vision. vancouver: ubc press, 2004 battiste, marie. indigenous knowledge and pedagogy in first nations education: a literature review with recommendations. ottawa: indian and northern affairs canada, 2002. http://www.ainc-inac.gc.ca-prpubkrw-ikp_e.pdf. battiste, marie; james s.y.henderson. protecting indigenous knowledge and heritage: a global challenge. saskatoon: purich publishing, 2000. birdsall, william f. et al. toward an integrated knowledge ecosystem. ottawa: canadian association of research libraries, 2005. also available at http://www.carlabrc.ca/projects/kdstudy/publi c_html/results.html. birdsall, william f. “web 2.0 as a social movement.” webology. 4.2 (2007). 05 march 2008. http://www.webology.ir/2007/ v4n2/a40.html. birdsall, william f. “constructing a right to communicate: the un declaration on the rights of indigenous peoples as a case study.” global media journal: evidence based library and information practice 2008, 3:2 74 american edition. (special fall issue 2008). forthcoming. birdsall, william f., kathleen shearer. “an open model of organization of diverse knowledge systems.” la interdiscilinariedad y la transdisciplinariedad en la organizaticion del conocimiento cientifico/interdisciplinarity and transdisciplinarity in the organization of scientific knowledge. actas del viii congreso isko-espana, leon, 18, 19 y 20 de abril de 2007. edited by blanca rodriguez bravo and ma luisa alvite diez. pp. 43-50. civallero, edgardo. “libraries, indigenous peoples, identity and inclusion.” world library and information congress: 73rd ifla general conference and council. 19-23 august, 2007. durban, south africa. http://www.ifla.org/iv/ifla73/ papers/128-civallero-transen.pdf. grenier, louise. working with indigenous knowledge: a guide for researchers. ottawa: international development research centre, 1997. 03 march 2008 http://www.idrc.ca/en/ev28700-201-1-do_topic.html. ivison, duncan, paul patton. eds. political theory and rights of indigenous peoples. cambridge, uk: cambridge university press, 2000. maness, j. m. 2006. “ library 2.0 theory: web2.0 and its implications for libraries.” webology. 3.2 (2006). 04 march 2008. http://www.webology.ir/2006/ v3n2/a25.html. mcluhan, marshall. the laws of media: the new science. toronto: university of toronto press, 1988. mcluhan, marshall. the global village: transformations in world life and media in the 21st century. new york: oxford university press, 1989. mihesuah, devon abbot, wilson, angela cavendar. 2004. eds. indigenizing the academy: transforming scholarship and empowering communities. lincoln: university of nebraska press, 2004. miller, p. “web 2.0: building the new library.” ariadne. issue 45 (october) (2005). 05 march 2008 http://www.ariadne.ac.uk/issu e45/miller/. miller, paul. “coming together around library 2.0.” d-lib magazine. 12.4 (2006). 02 march 2008 http://www.dlib.org/dlib/april 06/miller/04miller.html. o’reilly, t. 2005. “what is web 2.0: design patterns and business models for the next generation of software.” 04 march 2008 http://www.oreilly.com/pub/a /oreilly/tim/news/2005/09/30 /what-is-web-20.html. rasmussen, merrilee .” information rights and the law: the right evidence based library and information practice 2008, 3:2 75 to communicate in the canadian charter.”. access to information in a digital world. eds. karen g. adams and william f. birdsall. ottawa: canadian library association: 2004. riley, mary. ed. indigenous intellectual property rights: legal obstacles and innovative solutions. walnut creek, calif.: altamira press, 2004. roy, loriene. “indigenous libraries and innovative multicultural service.” keynote presentation. international federation of library associations and institutions, pretoria, 2007. sinclair-sparvier, wendy. 2002. “improving and delivering effective library services for aboriginal peoples in saskatchewan.” world libraries. 12.1 (2002). 05 march 2008 http://www.worlib.org/vol12n o1/sinclair_v12n1.shtml world library and information congress: 73rd ifla general conference and council. pretoria, south africa. august 15, 2007. 03 march 2008 http://www.ischool.utexas.edu /~loriene/prez_elect/media/pu blications/ifla-keynotelorieneroy.pdf. semali, ladislaus m., joe l. kincheloe. eds. what is indigenous knowledge?: voices from the academy. new york: falmer press, 1999. shreve, bradley g. 2007. “of gods and broken rainbows: native american religions, western rationalism, and the problem of sacred lands.” new mexico historical review. 82(summer): 369-390. unesco. action in favour of indigenous peoples. 2007. 05 march 2008 http://portal.unesco.org/cultur e/admin/ev.php?url_id=2946 &url_do=do_topic&url_s ection=201 united nations. united nations declaration on the rights of indigenous peoples. general assembly. sixty first session. 2 october, 2007. resolution 61/295. 05 march 2008.http://daccessdds.un.org/ doc/undoc/gen/n06/512/0 7/pdf/n0651207.pdf?openele ment. weber-pillwax, cora. “what is indigenous research?” canadian journal of native education. 25.2 (2001): 166-74. microsoft word news_eblip_au.doc evidence based library and information practice 2006, 1:3  98 evidence based library and information practice     announcement    libraries using evidence – eblip.net.au      lisa cotter  school librarian, nursing and midwifery  the university of newcastle and northern sydney central coast health  gosford, new south wales, australia  email: lisa@eblip.net.au    suzanne lewis  manager, central coast health service library  northern sydney central coast health  gosford, new south wales, australia  email: suzanne@eblip.net.au         © 2006 cotter and lewis. this is an open access article distributed under the terms of the creative  commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    northern sydney central coast health and  the university of newcastle, australia are  proud to introduce libraries using evidence  – eblip.net.au (http://www.eblip.net.au), a  new australian site being developed to  support library practitioners from all sectors  adopting the theory of evidence based  library and information practice (eblip).  the site will include a directory of current  evidence‐based research projects and  activities, a current awareness feed, a  gateway to support from the international  evidence‐based library and information  practice community, and pathways to help  put the theory into practice including the  eblip toolkit (october 2006) and pebl – a  project methodology for evidence‐based  libraries (early 2007).   the libraries using evidence advisory  group (http://eblip.net.au/libraries using  evidence charter.pdf) has been brought  together to oversee development of  libraries using evidence – eblip.net.au. this  international group, stretching across  australia to hong kong, and as far as  sheffield, england, will ensure the site  remains current, relevant, and useful to  library practitioners across the world who  want to make better decisions in daily  practice by applying principles of evidence  based library and information practice.    libraries using evidence – eblip.net.au will  be formally launched in october 2006 when  the eblip toolkit, a fundamental  component of the site, is made available.  http://creativecommons.org/licenses/by/2.0 http://www.eblip.net.au http://eblip.net.au/libraries evidence based library and information practice 2006, 1:3  99 the project to develop the eblip toolkit is  driven by the focused, answerable question:     “what resources/tools should be  included in an eblip toolkit  at libraries using evidence –  eblip.net.au to ensure it is a resource  that library practitioners across the  world will find useful to support  application of eblip? how can  library 2.0 techniques further  enhance the toolkit’s usefulness?ʺ    a draft version of the toolkit includes  modules containing tools, resources, and  information supporting each stage of the  evidence based practice process. features  include easy access to the tools needed to  find research evidence, checklists needed to  critically appraise the evidence found, and  support for effectively evaluating the impact  of the decisions made. keep up‐to‐date with  development of the site at  http://eblipnetauupdates.blogspot.com/.    pebl – a project methodology for evidence‐ based libraries (http://pebl.eblip.net.au) is a  project planning tool being developed by  lisa cotter and suzanne lewis to  incorporate the evidence based process into  standard project methodology. it is  designed to be used in conjunction with the  eblip toolkit as a stepping stone between  the theory of eblip and application of the  theory to practice. a call for project plans is  currently open to allow examination of  standard project planning tools to identify  where they fall short of supporting an  evidence based approach to decision  making. if you would like to contribute a  project plan to support this initiative, please  send it to pebl@eblip.net.au. see  http://www.eblip.net.au/pebl/call for project  plans.pdf for more information.    as with project management, synergies  have been identified between evidence  based library and information practice and  the principles behind library 2.0. in  developing libraries using evidence –  eblip.net.au, the project team is taking the  opportunity to explore how library 2.0  techniques can be used to enhance the  usefulness of the site. casey and savastinuk  consider “any service, physical or virtual,  that successfully reaches users, is evaluated  frequently, and makes use of customer input  is a library 2.0 service” (2006). initial  strategies identified to achieve this include  use of del.icio.us (http://del.icio.us) to  manage a dynamic current awareness feed  embedded in http://eblip.net.au and a list of  available evidence summaries; use of  podcasts to introduce major issues and each  stage of the eblip process; a wiki to collate  examples of eblip being modeled by  practitioners; and blogs to report progress,  discuss issues and enable wisdom of the crowd  to influence ongoing development of  libraries using evidence – eblip.net.au.  once the site is launched, its success will  largely rely upon the participation and  contribution of practitioners across the globe.     the current awareness feed is one aspect of  the site that is now live. the list may be  viewed within http://eblip.net.au/ under  “recent articles of interest”, or by  subscribing to  http://del.icio.us/rss/ebliptoolkit/current  using your rss reader. contribute to this  feed by tagging items of interest in  del.icio.us, as “for:ebliptoolkit.” gosford  hospital library staff will contribute to the  feed through selections found by keyword  alerts of web pages, blogs, and databases;  and by monitoring the top research journals  for each domain of librarianship as  identified by a content analysis of the library  and information science literature  (koufogiannakis, slater and crumley 2004).    when the eblip toolkit is launched in  october 2006, the libraries using evidence  advisory group will be calling for  volunteers (via the evidence‐based‐libraries  http://eblipnetauupdates.blogspot.com/ http://pebl.eblip.net.au http://www.eblip.net.au/pebl/call http://del.icio.us http://eblip.net.au http://eblip.net.au/ http://del.icio.us/rss/ebliptoolkit/current evidence based library and information practice 2006, 1:3  100 listserv) to peer review the toolkit. please  advise toolkit@eblip.net.au if you would like  to participate in this process.    visit and bookmark libraries using  evidence – eblip.net.au today!      works cited  casey, michael e., and laura c. savastinuk.  ʺlibrary 2.0.ʺ library journal 131.14  (2006): 40‐2.  koufogiannakis, denise, linda slater, and  ellen crumley. ʺa content analysis of  librarianship research.ʺ journal of  information science 30.3 (2004): 227‐39.                            article   determinants of health information use for self-efficacy in lifestyle modification for chronic disease patients   ebele n. anyaoku college medical librarian medical library college of health sciences nnamdi azikiwe university nnewi, anambra state, nigeria email: ebeleanyaoku@yahoo.com   obiora c. nwosu professor of library and information science department of library and information science nnamdi azikiwe university awka, anambra state, nigeria email: nwosu420@yahoo.com   received: 8 sept. 2015    accepted: 11 apr. 2016       2016 anyaoku and nwosu. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objectives – various efforts are being made to disseminate lifestyle modification information. what is the role of health information in building patients self-efficacy in lifestyle modification? the research examined level of access to lifestyle modification information for patients with chronic diseases in two federal government teaching hospitals in south east nigeria. it explored the relationship between self-efficacy and access to lifestyle modification information and also factors that are associated with self-efficacy when patients have access to lifestyle modification information.   methods – the research is a cross-sectional correlation study that used a questionnaire to collect data. (see appendix a.) sample was 784 patients with chronic diseases. questionnaires were distributed to the patients as they attended clinics in the medical and surgical outpatients’ clinics of the hospitals.   results – findings showed access to lifestyle modification information was significantly and positively correlated with self-efficacy. multiple regression analysis suggest that age, type of illness, and length of treatment in the teaching hospitals were associated with self-efficacy when patients have access to lifestyle modification information.   conclusion – it will be pertinent that demographic and disease factors are considered when making lifestyle modification information available to patients for greater self-efficacy.   introduction   various professional groups including health professionals, health educators, promoters, and medical librarians (pullen, jones & timm, 2011) provide information and education to people. this is done to help individuals modify lifestyle behaviours in order to prevent, as well as to live successfully with, chronic diseases. this is being done through various information media including traditional print, mass media such as radio and television, and the internet. it is important that people make effective use of the information to achieve the desired lifestyle modifications.   the use of information refers to what people do with information after they have acquired it.   information use occurs when the individual selects and processes information which leads to a change in the individual’s capacity to make sense or to take action. the outcome of information use is a change in the individual’s state of knowledge or capacity to act. thus, information use typically involves the selection and processing of information in order to answer a question, solve a problem, make a decision, negotiate a position, or make sense of a situation. (choo, et al., 2006)   bandura (1986, 1997) in social cognitive theory presented self-efficacy as an outcome of information use. self-efficacy is defined as a person’s perception of one’s own ability to undertake a given task or behaviour. self-efficacy affects the choice of behaviour, settings in which behaviours are performed, and the amount of effort and persistence to be spent on performance of a specific task. people who have high self-efficacy will be more likely to perform a related behaviour than those with low self-efficacy (lawrance & mcleroy, 1986; davies, 2011).   in addition to medication administration people living with chronic diseases are frequently required to modify lifestyle behaviour such as changing to a healthier diet, limiting alcohol consumption, increasing the amount and intensity of physical activity, or stopping smoking to improve their health (dunbar-jacob, 2007, osório, 2010). individuals need high self-efficacy to undertake these behaviour modifications. chronic disease self-efficacy is defined as the individual’s judgment of confidence to carry out tasks specific to chronic disease self-management (rapley, passmore & phillips, 2003). the self-management behaviours include coping with the illness, adherence to treatment, and undertaking required lifestyle modifications. self-efficacy may determine who can successfully perform these required self-management behaviours.   pálsdóttir (2008) noted that self-efficacy has been examined extensively in relation to healthy behaviour. the connection between people's health information behaviour and their judgments about how capable they are of managing their health in a successful way, has not gained much interest. in the face of the global increase in chronic diseases and its attendant increase in mortality and morbidity (alwan, 2011) there is need to explore various avenues to stem the tide of the impact of these diseases on the general populace. the focus of the research is therefore to ascertain the extent to which access to health information can contribute to building people’s capacity to undertake lifestyle modifications necessary for living successfully with chronic diseases.   health information use and self-efficacy in lifestyle modification   according to social cognitive theory, individuals form their self-efficacy beliefs by interpreting information primarily from four sources: mastery or direct experiences, vicarious experience, social persuasions, and physiological reactions. social persuasions include suggestions, exhortations, self-instructions, health promotions, and health education. despite all efforts to provide information for health living, considerable opinions have been expressed on the role of health information in achieving behavioural changes or maintaining a healthy lifestyle. while some authors highlight the importance of information in this process (clayton, 2010; burkell, wolfe, potter, & jutai, 2006; pinnock & sheikh, 2004) others argued that information alone does not guarantee healthy behaviour. according to murray, et al. (2005) in promoting health behaviours, the expectation is that well-informed people are more likely to follow healthy behaviours than poorly informed people are.   however, this assumption is probably only partially correct – if knowledge were all that was needed to promote healthy behaviour, smoking, for example, would not be as prevalent as it is. ash et al. (2008) posited that a good relationship with patients and providing sound education and advice are clearly necessary but are not in themselves effective in inducing patients to comply with advice. patients do not necessary take physicians’ advice to alter behaviour beyond simply taking medicine. again, spehr and curnow, (2011) wrote that on its own information is very unlikely to lead to a measurable change in behaviour. information may improve people’s knowledge of a problem or contribute to a change in their attitude towards it, but there is a vast gulf between knowing about a problem and doing something about it, as anyone who’s tried to give up smoking or lose weight knows.   these different opinions tend to be corroborated by empirical reports. sharaf (2010) assessed the impact of health education on diet, smoking, and exercise among patients with chronic diseases in al qassim region in saudi arabia. among chronic disease patients, significant improvements in smoking, diet, and exercise habits were observed at end-line survey compared to baseline. similarly, tawalbeh and ahmad (2014) found that implementing cardiac educational programs help enhance knowledge and adherence to healthy lifestyle among patients with coronary artery disease in north of jordan. çevik, özcan, and satman (2015) found the training program for reducing cvd risk factors in patients with type 2 diabetes was effective in improving nutrition and lifestyle behaviours. contrary, bohaty, rocole, wehling, and waltman (2008) found no behavioural change in dietary intake of calcium and vitamin d after increasing patients’ knowledge of osteoporosis. equally, stadler, oettingen, and gollwitzer (2010) found that combining information with self-regulation strategies had a better effect on eating fruits and vegetables than an information-only intervention over a two year period.   it is important to note that the insignificant association reports did not dispute the fact that health information has some roles to play in behavioural and lifestyle changes. the authors were of the view that information needs to be supported with other interventions to achieve the desired aims. for instance, fisher and fisher (1992) in postulating the information-motivation-behavioural-skills theory noted that information relates to the basic knowledge about a medical condition, and is an essential but not necessarily sufficient in isolation. a favourable intervention would establish the baseline levels of information, and target information gaps. if information needs to be supported by other interventions to make it effective in building peoples’ perceptions of their capability to undertake behavioural changes, then a study of individual’s variables and their relationship to lifestyle modification becomes imperative. the main purpose of this study was to find out if any relationship exists between patients’ access to lifestyle modification information and patient’s self-efficacy in managing chronic diseases. it also sought the predictive effects of demographic and disease variables on self-efficacy when patients have access to lifestyle modification information. the findings can serve as reference to support and help information providers design and offer appropriate user-centered information services in tertiary health institutions.   hypotheses   h1:         patients who report higher access to lifestyle modification information will report higher perceptions of self-efficacy in lifestyle modification.   h2:         patients’ demographic and disease variables are positively related to self-efficacy when they have access to lifestyle modification information.   methods   study design   the research is a cross-sectional correlation study and is part of a larger study on access to health information for patients with chronic diseases.   population and sample   the study was carried out in university of nigeria teaching hospital ituku ozalla, enugu, enugu state and nnamdi azikiwe university teaching hospital, nnewi, anambra state. participants were patients with chronic diseases in the two hospitals. the study sample was calculated to be 784 respondents using ‘yaro yamane’ formula for finite population as presented by uzoagulu (1998). the study was approved by the medical ethics committee of the two teaching hospitals in question.   instrument   a questionnaire was the instrument for data collection. a patient’s health information access questionnaire which also included a lifestyle modification self-efficacy scale was used to collect data for the study. access to lifestyle modification information was measured using six items (α = .859) that examined patients’ level of access to information on lifestyle modifications in the areas of diet, exercise, and relaxation. respondents were asked to rate how much information they received on lifestyle modification on a four-point scale: of (4) much information, (3) some information, (2) little information, (1) no information. the lifestyle modification self-efficacy scale (α. =799) was patterned like the validated self-efficacy for managing chronic disease scale developed by stanford patient education research center. patients were requested to rate their level of confidence to undertake lifestyle modifications on a five point scale of not confident to completely confident. the questionnaire was validated by two medical doctors and two lecturers from nnamdi azikiwe university and was subjected to an internal consistency test using cronbach’s alpha (α) test. the reliability test used 15 patients at a large missionary hospital in anambra state.   the questionnaires were administered by the first researcher and research assistants to the patients in the outpatient’s clinics comprising: the medical outpatient clinics, the oncology clinics, and the retroviral disease clinics. the respondents were requested to complete and return the questionnaires on receipt. in all, 1,080 questionnaires were distributed to the patients before the requisite sample of 784 properly completed copies were obtained. of the questionnaires distributed, 234 were not properly completed and 62 were not returned.   table 1 demographic characteristics of study participants   frequency % gender female male   480 304   61.2 38.8 age 18-29 30-44 45-64 65+   84 271 272 157   10.7 34.6 34.7 20 educational level none primary secondary tertiary   109 226 211 238   13.9 28.8 26.9 30.4 nature of illness hypertension diabetes hypertension / diabetes hiv and aids cancer  kidney disease   204 131 52 337 40 20   26.0 16.7 6.6 43.0 5.1 2.6 duration of illness 1-11 months 1-5 years 6-10 years 11+ years   146 341 182 115   18.6 43.5 23.2 14.7 duration of treatment in teaching hospital 1-11 months 1-5 years 6-10 years 11+ years   239 369 134 42   30.5 47.1 17.1 5.4     method of data analysis   descriptive statistics were employed to ascertain the extent of access to lifestyle modification information and level of patients’ self-efficacy. pearson product moment correlation coefficient (r) was used to test the relationships between self-efficacy and access to lifestyle modification information. multiple regression analysis was done to explore the predictive effect of demographic and disease variables on access to lifestyle modification information and self-efficacy. self-efficacy was used as the dependent variable. lifestyle modification information, demographic, and disease variables were the independent variables. the ‘enter method’ was used for the multiple regression analysis. patients’ demographic and disease variables were recorded and dichotomized 0 and 1. recoding was as follows: gender (female, vs male), age (young adulthood vs. older adulthood), education (below tertiary vs. tertiary), disease type– (hypertension and diabetes vs. others). duration of illness (≤5 years vs. ≥ 6 years). duration of treatment in teaching hospital (≤5 years vs. ≥ 6 years).   results   level of patients’ access to lifestyle modification information   table 2 shows patients’ level of access to lifestyle modification information. analysis of data in terms of high access (much or some information) and low access (little or no information) show that more than two-thirds of respondents reported high access on how to take fruits and vegetables(86.2%), proper nutrition (82.4%), alcohol intake moderation (73.3%), relaxation and stress reduction techniques (71.2%), and proper exercise (68.1% ). the least percentage (56.3%) of the respondents indicated high access on how to maintain a healthy weight. however, about one quarter of the respondents indicated having little or no information access to various aspects of lifestyle modification.     table 2 level of patients’ access to lifestyle modification information health information no information a little information some information much information mean std. dev. proper nutrition 82 (10.5%) 56 (7.1%) 125 (15.9%) 521 (66.5%) 3.38 1.00 fruits and vegetable intake 73 (9.3%) 35 (4.5%) 99(12.6%) 577 (73.6%) 3.51 .94 exercise 182 (23.2%) 68 (8.7%) 133(17.0 %) 401 (51.1%) 2.96 1.24 maintaining healthy weight   258 (32.9%) 85 (10.8%) 121 (15.5%) 320 (40.8%) 2.64 1.31 alcohol intake and smoking cessation 171 (21.8%) 38 (4.9%) 92 (11.7%) 483 (61.6%) 3.13 1.23 relaxation and stress reduction 160 (20.4%) 66 (8.4%) 147 (18.8%) 411 (52.4%) 3.03 1.19     patients’ self-efficacy in undertaking lifestyle modification   table 3 summarizes patients’ perceptions of their ability to undertake various lifestyle modification behaviours. analysis of the mean score of the variables in table 3 shows that respondents rated their self–efficacy very high (vhs) in taking drugs appropriately > 4.50. they also indicated having high self-efficacy (hs) > 4.00 for taking fruits and vegetables, taking proper nutrition, and self-efficacy in keeping appointments and judging when to see a doctor. the respondents rated their self-efficacy moderate (ms) >3.00 to undertake exercise.     table 3 patients’ self-efficacy in undertaking lifestyle modification self – efficacy items not confident a little confident moderately confident highly confident completely confident mean std. dev confidence to take proper nutrition   20 (2.6%) 55 (7.0%) 100 (12.8%) 184 (23.5%) 425 (54.2%) 4.20 1.07 confidence to take fruits and vegetables   23 (2.9%) 46 (5.9%) 86 (11.0%) 188 (24.0%) 441 (56.3%) 4.25 1.05 confidence to take drugs appropriately   14 (1.8%) 21 (2.7%) 60 (7.7%) 134 (17.1%) 555 (70.8%) 4.52 .88 confidence to undertake exercise 116 (14.8%) 131 (16.7%) 147 (18.8%) 107 (13.6%) 283 (36.1%) 3.39 1.48 confidence to keep appointments and judge when to see a doctor 57 (7.3%) 52 (6.6%) 56 (7.1%) 142 (18.1%) 477 (60.8%) 4.19 1.25     correlations          table 4 shows the correlation between self-efficacy and all variables of lifestyle modification information. the result shows that self-efficacy is positively and significantly correlated with all the variables of lifestyle modification information (p<0.05). patients’ access to information on relaxation and stress reduction has the highest correlation with self-efficacy showing a moderately positive correlation (r=.420 p< .001). other variables namely information on fruits and vegetable intake (r=.397 p< .001), smoking cessation and alcohol intake(r=.382 p< .001), proper nutrition (r=.374 p< .001), exercise (r=.345 p< .001), and maintaining healthy weight (r=.310 p< .001), are also positively but lowly correlated with self-efficacy. based on the significant correlations (p<0.05), h1 of the study is supported. patients who reported higher access to lifestyle modification information also report higher self-efficacy in lifestyle modification.     table 4 correlation matrix of access to lifestyle modification information and self-efficacy   1 2 3 4 5 6 7 1 – self-efficacy 1 .374** .397** .345** .310** .382** .420** 2 – proper nutrition .374** 1           3 – fruits and vegetable intake .397** .750** 1         4 – exercise .345** .469** .560** 1       5 – maintaining healthy weight .310** .414** .426** .661** 1     6 – smoking cessation & alcohol intake .382** .508** .492** .450** .462** 1   7 – relaxation and stress reduction .420** .485** .478** .458** .478** .490** 1     regression   regression analysis was done to explore the predictive effect of socio-demographic variables on access to health information and self-efficacy. self-efficacy was used as the dependent variable. lifestyle modification information, demographics, and disease variables were the independent variables. the regression was a poor fit (r2 = 31%) but the overall relationship was significant (f (12,723) = 26.767, p<0.05). an inspection of independent variables shows that with other variables held constant, self-efficacy had  significant negative association to age (beta = -090, p<0.05),and significantly positive association to disease type (beta =.127, p<0.05) and duration of treatment (beta=.082, p>0.05). conversely, patients’ gender (beta = -.014, p>0.05), educational attainment (beta =-.034, p>0.05), and duration of illness (beta = .047, p>0.05) are not significantly associated with patients’ self-efficacy.   based on the significant associations found, the second hypothesis of the study is supported. patients’ demographic and disease variables are positively related to self-efficacy when they have access to lifestyle modification information.     table 5 association between patients’ demographic and disease variables, access to lifestyle modification information and self-efficacy   r2 f* beta se p   background characteristics gender age education .311 26.767     -.014 -.090 -.034     .054 .065 .056     .667 .023** .288 disease variable disease type duration of illness duration of treatment       .127 -.047 .082   .066 .067 .077 . 002** .241 .041** lifestyle modification information proper nutrition fruit and vegetable intake exercise maintaining healthy weight alcohol intake and smoking cessation relaxation and stress reduction       .065 .121 .057 -.004 .123 .214   .041 .045 .030 .027 .027 .027   .182 .016** .208 .917 .003** .001**     discussion   results of the study showed that a larger percentage of patients reported high access to the six variables of lifestyle modification information studied. however, about one quarter of respondents have little or no information on alcohol intake moderation, relaxation, stress reduction techniques, and proper exercise. close to half of the patients do not have access to information on how to maintain a healthy body weight. this indicates that some patients do not have access to the important information needed to live successfully with chronic disease.   correlation analysis showed that access to lifestyle modification information was significantly and positively connected with self-efficacy. respondents who reported higher access to information on proper nutrition, fruit and vegetable intake, proper exercise, maintaining healthy weight, smoking cessation, alcohol intake, relaxation, and stress reduction also reported higher confidence to undertake these changes. the study therefore supports the positive relationship between patients’ access to lifestyle modification information and patients’ self-efficacy as found in some studies (çevik, özcan, and satman, 2015; tawalbeh and ahmad, 2014; sharaf, 2010;kavathe, 2009).   to improve health information use in lifestyle modification, it is important to know factors that influence self-efficacy. findings suggest that with access to lifestyle modification information, patients who are older, have hypertension and diabetes, and recently received treatment at teaching hospitals reported higher self-efficacy in lifestyle modification. considering that older patients reported more self-efficacy, younger patients can be targeted with specific massages using sources that appeal to them such as social media sites, the internet, and peer groups. according to noar, harrington, van stee, and aldrich (2011) “to date, a large literature has amassed showing the promise of tailored programs delivered via print, internet, local computer/kiosk, telephone, and interpersonal channels. numerous studies demonstrate that these programs are capable of significant impacts on smoking cessation, dietary change, physical activity, and multiple behavior change.”   findings of the study also showed that patients’ gender, educational attainment, and duration of illness were not significant predictors of self-efficacy in managing chronic disease when they have access to health information. this result suggests that there is no difference in self-efficacy in lifestyle modification for male and female patients when they have access to lifestyle modification information. equally low or high level of education does not predispose a patient to higher level of self-efficacy when provided access to lifestyle modification. materials at all levels of patient education should be acquired as all will benefit from the information irrespective of their educational background. duration of illness was also not a significant predictor of self-efficacy. when patients have access to health information, all will have the same level of self-efficacy irrespective of how long their illness lasted. so information dissemination and access should be a continuous process in the disease continuum.   health information is a topic that transcends the boundaries of many disciplines. librarians, health professionals, and health educators have some role in its dissemination. to mitigate the effect of chronic disease on the populace, everyone must be involved. health professionals, medical librarians, and other information providers should develop strategies that will ensure patients have access to the right information at the right time and have it tailored to their individual needs. there is a need for effective collaboration between health professionals and other information providers in tertiary health institutions in providing health information for the benefit of patients.   conclusion   in providing information for effective lifestyle changes, it is important to understand the factors that can support people’s efforts to undertake these changes. the result of the study shows there is a significant relationship between patients’ access to health information and patients’ self-efficacy in lifestyle modification. findings also suggest that patients who are older, have hypertension and diabetes, and recently received treatment at teaching hospitals, reported higher self-efficacy in lifestyle modification when they have access to appropriate information. this means that strategies are needed to enhance the effectiveness of health information dissemination for chronic disease management especially for younger patients. however, the study is limited by the nature of data collected. the responses are perceived, self-reported, and relative. there was no knowledge test to confirm patients’ health knowledge level or quality of information received. despite this limitation, the result is significant for provision of health information. it has revealed areas where patients feel they lack information and factors that can be manipulated to make information dissemination more effective. therefore the findings of the study should serve mainly as a benchmark for providing quality health information. patient perceived gaps and factors for tailoring information to ensure effective delivery should be noted.   further research can focus on a longitudinal study of information intervention and actual self-efficacy in lifestyle modification for patients with chronic diseases in nigerian tertiary health institutions. due to differences in availability of various media for health information dissemination, especially in the extensive usage of digital media in many developed countries, the result on level of access to lifestyle modification information may not be generalizable to a wider population outside nigeria. however, noting the significant correlations and the general need to improve information use in lifestyle modification, it is pertinent that the identified demographic and disease variables which predict self-efficacy are considered and exploited by health information providers.   references   alwan, a. (2011). global status report on non-communicable diseases 2010. geneva: world health organization. retrieved from www.who.int/nmh/publications/ncd_report2010   ash, l. m., modic, m. t., obuchowski, n. a., ross, j. s., brant-zawadzki m. n., & grooff, p. n. (2008). effects of diagnostic information, per se, on patient outcomes in acute radiculopathy and low back pain. american journal of neuroradiology, 29(6), 1098-1103. doi:10.3174/ajnr.a0999   bandura, a. (1986). social foundations of thought and action: a social cognitive theory. englewood cliffs, nj: prentice hall.   bandura, a. (1997). self-efficacy: the exercise of control. new york: freeman.   bohaty, k., rocole, h., wehling, k., & waltman, n. (2008).testing the effectiveness of an educational intervention to increase dietary intake of calcium and vitamin d in young adult women. journal of american academy nursing practitioners, 20(2), 93-99. doi:10.1111/j.1745-7599.2007.00281.x   burkell, j. a., wolfe, d. l., potter, p. j., & jutai, j. w. (2006). information needs and information sources of individuals living with spinal cord injury. health information & libraries journal, 23(4), 257–265. doi:10.1111/j.1471-1842.2006.00686.x   çevik, a. b., özcan, s., & satman, i. (2015). reducing the modifiable risks of cardiovascular disease in turkish patients with type 2 diabetes: the effectiveness of training. clinical nursing research, 24(3), 299-317. doi:10.1177/1054773814531288   choo, c. w., furness, c., paquette, s., van den berg, h., detlor, b., bergeron, p., & heaton, l. (2006).working with information: information management and culture in a professional services organization. journal of information science, 32(6), 491-510. doi:10.1177/0165551506068159   clayton, l. h. (2010). strategies for selecting effective patient nutrition education materials. nutrition in clinical practice, 25(5), 436-442. doi:10.1177/0884533610379605   davies, n. (2011). healthier lifestyles: behaviour change. nursing times, 107(23), 20-23   dunbar-jacob, j. (2007). models for changing patient behavior: creating successful self-care plans. the american journal of nursing, 107(suppl. 6), 20-25. http://www.jstor.org/stable/40384535   fisher, j. d., & fisher, w. a. (1992).the information-motivation behavioural skills model. in: diclemente, r., crosby r. j., & kegler, m. (eds.) emerging theories in health promotion practice and research. san francisco, ca: jossey bass publishers, 40–70.   kavathe, r. s. (2009). patterns of access and use of online health information among internet users: a case study (doctoral dissertation).bowling green state university. retrieved from http://rave.ohiolink.edu/etdc/view?acc_num=bgsu1250696451   lawrance, l., & mcleroy, k. r. (1986).self‐efficacy and health education. journal of school health, 56(8), 317-321.   murray, e., burns, j., see, t.s., lai, r., & nazareth, i. (2005). interactive health communication applications for people with chronic disease. cochrane database of systematic reviews, 4. doi:10.1002/14651858.cd004274.pub4   noar, s. m., harrington, n. g., van stee, s. k.,& aldrich, r. s. (2011). tailored health communication to change lifestyle behaviors. american journal of lifestyle medicine.5 (2) 112-122. doi:10.1177/1559827610387255   osório, j. (2010) bringing about lifestyle change. nature reviews cardiology, 7, 537. doi:10.1038/nrcardio.2010.142   pálsdóttir, á. (2008). information behaviour, health self-efficacy beliefs and health behaviour in icelanders' everyday life. information research, 13(1). http://informationr.net/ir/13-1/paper334.html   pinnock, h., & sheikh, a. (2004).meeting the information needs of patients with allergic disorders: partnership is the key. clinical and experimental allergy, 34(9), 1333–1335.   pullen, k., jones, d., & timm, d. f. (2011). healthy connections for consumers: national library of medicine web sites. journal of hospital librarianship, 11(1), 23-38. doi:10.1080/15323269.2011.537979   rapley, p., passmore, a., & phillips, m. (2003). review of the psychometric properties of the diabetes self-efficacy scale: australian longitudinal study. nursing & health sciences, 5(4), 289-97.   sharaf, f. (2010).impact of health education on compliance among patients of chronic diseases in al qassim, saudi arabia. international journal of health sciences (qassim), 4(2), 139-148.   spehr, k., & curnow, r. (2011). are your objectives really about behaviour change? or are they really about knowledge &attitudes? community change http://www.communitychange.com.au/insights-and-tools/improving-your-program/are-your-objectives-really-about-behaviour-change.html stadler, g., oettingen, g., & gollwitzer, p. m. (2010). intervention effects of information and self-regulation on eating fruits and vegetables over two years. health psychology, 29(3), 274-283. doi:10.1037/a0018644   stanford patient education research center (n.d.). chronic disease self-efficacy scales. retrieved from http://patienteducation.stanford.edu/research/secd32.pdf   tawalbeh, l. i., & ahmad, m. m. (2014).the effect of cardiac education on knowledge and adherence to healthy lifestyle. clinical nursing research, 23(3), 245-258.   uzoagulu, a. e. (1998). practical guide to writing research project reports in tertiary institutions. enugu: john jacob’s classic.     appendix a questionnaire: determinants of health information use for self-efficacy in lifestyle modification for chronic disease patients   please tick √ in the appropriate box 1. sex:    female_____      male_____                 2. age:   18-29_____          30-44_____          45-64_____          65+_____   3. occupation:  _____________________________________________   4. educational level:         no formal educ_____      primary_____   secondary (wasce)_____            tertiary_____            5. what is the nature of your illness? tick all that apply    hypertension_____          diabetes_____    hiv_____            kidney diseases _____ cancers: please state cancer type__________         other illnesses __________   6. how long have you had this illness?  1-11 months_____           1 – 5 years_____                 6 – 10 years_____             11+ yrs_____   7. how long have you received treatment in this hospital? ________________________________________   8. how much information did you receive on lifestyle modification while receiving treatment such as? (please tick √ in the appropriate box)   much information some information little information no information   proper nutrition         intake of fruits and vegetables           how to exercise and keep fit         how to lose weight or maintain a healthy weight           alcohol intake moderation or smoking cessation           reduction of stress e.g.  relaxation techniques                                            self-efficacy scale: we would like to know how confident or sure you are in doing certain activities required to help you manage your illness at the present time. please tick √ in the appropriate box that shows your degree of confidence for each item.   completely confident. 5 high confident 4 moderately confident. 3 a little confident 2 not confident 1 how confident or sure are you that you can take all medications prescribed by your doctor without missing a dose at any point in time?           how confident or sure are you that you can stick to a healthy and balanced diet so as to maintain your weight and health?           how confident or sure are you that you can increase your fruit and vegetable intake so as to maintain your health?           how confident or sure are you that you can exercise 30 to 45 minutes, 4 to 5 times a week so as to maintain your health?           how confident or sure are you that you can keep your appointment or judge when the changes in your illness mean you should visit a doctor?             evidence based library and information practice evidence based library and information practice 2011, 6.2 59 evidence based library and information practice evidence summary investigation of factors affecting information literacy student learning outcomes fails to undercover significant findings a review of: detlor, b., julien, h., willson, r., serenko, a., & lavallee, m. (2011). learning outcomes of information literacy instruction at business schools. journal of the american society for information science and technology, 62(3), 572-585. reviewed by: jason martin associate librarian university of central florida libraries orlando, florida, united states of america email: jason.martin@ucf.edu received: 25 mar. 2011 accepted: 25 may 2011 2011 martin. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to ascertain the factors influencing student learning during information literacy instruction (ili) and create a theoretical model based on those factors. design – mixed methodology consisting of interviews and an assessment test. setting – three canadian business schools. subjects – seven librarians, 4 library administrators, 16 business faculty, and 52 undergraduate business students were interviewed, and the standardized assessment of information literacy skills (sails) test was administered to 1,087 undergraduate business students across three different business schools. methods – the authors used an interview script to conduct interviews with librarians, library administrators, business school faculty, and undergraduate business school students at three business schools in canada. the authors also administered the sails test to undergraduate business students at the same three canadian business schools. main results – ili works best when it is related to an assignment, part of the curriculum, periodically evaluated, adequately funded, timely, mandatory, interactive, uses handouts, provides the proper amount of mailto:jason.martin@ucf.edu� evidence based library and information practice 2011, 6.2 60 information, and favourably viewed within the school. ili student learning outcomes are affected by whether the students find the ili beneficial and relevant, their year in the program, gender, status as international or domestic student, and overall academic achievement. conclusion – creation of theoretical model consisting of the three main factors influencing student learning outcomes in information literacy instruction: learning environment, information literacy components, and student demographics. commentary the authors of this research study set about to understand what factors influence student learning outcomes in ili and develop a theoretical model for ili. their findings are basic; anyone who has experience teaching library instruction, or any subject matter, would be familiar with the factors the authors state influence learning outcomes. sadly, many of these factors are out of the control of librarians and teachers. however, the basic nature of their findings works to their favour in that they add credence to the inclusion of these factors in a theoretical model. if the authors had stopped with these findings, then they would have succeeded in developing a modest model, albeit one which would need more research and evidence to bolster its credibility. even “common sense” notions such as the idea that ili should be given “just-in-time” and tied to an assignment benefit from credible supporting evidence. however, the authors include findings which seem to be not only outside their objective, but do not support their theory. first, the authors included a discussion of the differences in perception of ili between students and librarians, library administrators, and teaching faculty. while this discussion might make for an interesting article in and of itself, it adds nothing to the overall aim to create a theoretical model of factors influencing ili. second, the authors administered the sails test to the student participants in the three business schools. the overall scores for the three schools were not good, and only two scores of a possible 24 were statistically different. these results seem to not support the component of learning environment included in the authors’ model. the inclusion of this extra data muddles the true intent and purpose of the article. the authors’ theoretical model provides a starting point for researchers looking to understand what factors influence student learning during library instruction. more research is needed to validate the model and possibly find more factors within the three areas that influence student learning outcomes. / evidence based library and information practice evidence summary microsoft word es_lewis.doc evidence based library and information practice 2006, 1:1  92 evidence based library and information practice       evidence summary    the majority of library clients still use person‐to‐person interaction when asking  reference questions    a review of:   de groote, sandra l. “questions asked at the virtual and physical health sciences reference  desk: how do they compare and what do they tell us?” medical reference services  quarterly 24.2 (summer 2005): 11‐23.    reviewed by:   suzanne p. lewis   library manager, central coast sector, northern sydney central coast health  gosford, new south wales, australia  email: slewis@nsccahs.health.nsw.gov.au    received: 15 december 2005 accepted: 14 february 2006     © 2006 lewis. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to identify similarities and  differences in the questions asked at the  virtual and physical reference desks of a  health sciences library, in order to better  understand user needs and highlight areas  for service improvement. also to  retrospectively analyze reference statistics  collected over the previous six years.    design – use study; retrospective study of  reference statistics for the period july 1997  to june 2003; literature review.    setting – large academic health sciences  library in the united states.    subjects – all questions asked at the  reference and information desks, plus  questions submitted to the university‐wide  virtual reference service and answered by a  health sciences librarian, over a period of  one month. the questions were asked by  faculty, staff, students and members of the  public.     methods – a literature review was carried  out to examine the types of  information/reference questions typically  asked in health sciences libraries both before  and after the mass introduction of remote  end‐user searching of online resources and  the establishment of virtual reference  services.    next, the reference statistics collected at the  university of illinois at chicago (uic)  library of the health sciences between july  1997 and june 2003 were examined. for  most of this period a digital reference  service was offered using a listserv address  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:1  93 to which patrons would submit email  queries. beginning in march 2003, a formal  virtual reference service (chat and email)  was provided using commercial software.     finally, data was gathered on questions  answered by a health sciences librarian, and  clients who asked the questions, at either the  physical or virtual reference desk, during  the month of november 2003 at the uic  library of the health sciences. library staff  completed an online survey form for each  question, and if a client asked more than one  question, each question was coded  individually. data included: status of client  using the service (faculty/staff,  undergraduate student, graduate student,  non‐uic, unknown); mode of submission  (email, chat, phone, in person); and type of  question asked (directional, ready reference,  in‐depth/mediated, instructional, technical,  accounts/status and other).  in subsequent  analysis, the original seven types of  questions were further broken down into 19  categories.     main results – it was not possible to  undertake a meta‐analysis or systematic  review of the studies identified in the  literature review because of differences in  time frames, settings and the categories used  to code reference questions. however the  following trends emerged: directional  questions accounted for between 30 and  35% of questions asked at both physical and  virtual reference desks; the remainder of  questions were generally about known item  searches, library policies and services,  research, database use and quick reference.    the statistics collected at uic library of the  health sciences over the period july 1997 to  june 2003 were analyzed. coded reference  questions fell into one of four categories:  ready reference, in‐depth reference,  mediated searches and digital reference.  there was a noticeable drop in the number  of reference questions received in 1999/2000  which reflects trends reported in some of the  studies identified in the literature review.  the number of mediated searches decreased  from 154 in 1997/98 to 4 in 2002/2003, but the  number of digital reference questions  increased from 0 to 508 in the same period.    statistics were collected over the month of  november 2003 for 939 questions asked at  the reference and information desks which  included: 38 e‐mail; 48 chat; 156 phone; and  697 in person. the major findings were as  follows:    • approximately 55% of questions were  reference questions (33.5% ready  reference, 9.7% in‐depth/mediated, 9.7%  instructional); 30% were directional; and  10% were technical; it is not stated what  the remaining 5% of questions were;    • library clients who asked the questions  comprised graduate students (26%),  faculty (24%), undergraduate students  (24%) and non‐uic patrons (22%);    • all groups of clients were most likely to  ask their reference questions in person;    • graduate students were the largest users  of email reference (34%),  undergraduates were the largest group  to use chat (35%) and faculty/staff were  the largest group to use the telephone  (39%);    • 28% of the questions were answered  from library’s staff’s general knowledge  of the library; 22% using the online  catalogue or electronic journal list; 22%  by referring the patron to, or using, an  online database or resource; 13% by  referring the patron to another  department in the library (such as  circulation); 7% by consulting another  individual; and 5% by using print  resources.    evidence based library and information practice 2006, 1:1  94 conclusion – the results of the three parts  of this study – literature review, study of  1997‐2003 statistics, and in‐depth collection  of statistics for november 2003 – are  difficult to compare with each other.  however, the general trend emerging from  the results is that some kinds of questions  asked by health sciences library clients have  remained very much the same regardless of  the rise of remote end‐user searching and  the mode of transmission of the questions.  these include questions about library  policies and services, journal and book  holdings, database searching and  instructional support.  questions about  citation verification and consumer health  appear to be decreasing while technical  questions and questions about accessing  remote databases and online journals are  increasing. the majority of reference  questions are still asked in person.    commentary    the author clearly states the objective of this  study – to examine the types of questions  asked and the demographic categories of the  users of the virtual and physical reference  desks in a health sciences library. however  the article loses focus on the objective by  starting with an overly long literature  review. the studies described in the  literature review are too disparate to be  directly comparable with each other and the  november 2003 in‐depth study does not  appear to use the methodology of any of the  earlier studies identified in the review.    the subjects of the study are the questions  asked during november 2003 by clients of  the information desk and physical reference  desk at the uic library of the health  sciences, plus questions asked at the  university‐wide virtual reference desk that  were answered by a health sciences librarian.  the total number of questions included in  the study is 939 which is an adequate  sample size.   the data collection instrument used in this  study – an online survey form – was not  provided in the article, thereby limiting the  replicability of the study. the coding of the  questions is difficult to determine, as neither  a comprehensive list nor definitions of  coding categories are given. the data was  recorded by the librarians providing the  service being studied. the author does not  discuss any possible measures taken to  reduce bias beyond stating that the library  staff received training on filling out the  forms to clarify definitions. the response  rate to this study was presumably 100%, but  the article does not state whether all online  survey forms were filled out in sufficient  detail to be used in the study.    the author states that “the data gathering  was to be unobtrusive to the users”. it is not  clear whether clients of the reference desk  knew they were participating in a study, or  whether consent to participate was sought  or given.    there is some analysis of the reference  statistics collected by the uic health  sciences library staff over the period july  1997 to june 2003. the november 2003 study  may have replicated some aspects of the  1997‐2003 statistics collection, but this is not  clearly stated. more value could have been  derived from the retrospective statistics if  more detail was provided on how they were  collected and how the original coding of  reference questions relates to the coding  used in the later study. the author does not  state clearly whether the online survey form  used in the november 2003 study is the  same instrument used to capture the earlier  statistics.    the results of the study are generally clearly  presented, although this reviewer found  several of the graphs too small to read easily.  on occasion, different terms are used for the  same type of reference question. for  example, the article refers to “database  evidence based library and information practice 2006, 1:1  95 searches to find articles” and “how to find  articles on a particular topic,” but it is not  clear whether these are the same, or  different, coding categories.     overall, the study is valid, but its reliability  is compromised by not supplying the reader  with enough information, and its  applicability is limited. the reliability of the  study is reduced by the author’s failure to  include all relevant information and results.  the author states that “coded information  was … analyzed using spss statistical  software” but complete results including  statistical significance are not given. not  enough detail is given in this article to  enable another library to repeat the study;  the data collection instrument and data  coding would be needed. the study is  directly applicable to academic health  sciences libraries and could be adapted for  use by other libraries if further information  could be obtained from the author.                                                  this is a useful study in that it demonstrates  that the majority of library clients still use  person‐to‐person interaction when asking  reference questions. only 25% of the  questions received during the in‐depth  study period (november 2003) were asked  via phone, email or chat; the remaining 75%  of questions were asked in person. the  study provides useful evidence for health  sciences librarians to present to  administrators who may question the value  of maintaining and funding a physical  library. the study also highlights the fact  that some library clients are more likely to  use phone, email or chat to ask questions  than others. this is one possible area of  future study and has implications for  staffing virtual reference desk services and  promoting the service to different client  groups.  evidence summary   secondary school students ascribe value to presentation, accuracy, and currency in their evaluation of web-based information   a review of: pickard, a. j., shenton, a. k., & johnson, a. (2014). young people and the evaluation of information on the world wide web: principles, practice and beliefs. journal of librarianship and information science, 46(1), 3-20. http://dx.doi.org/10.1177/0961000612467813   reviewed by: kimberly miller learning technologies librarian albert s. cook library towson university towson, maryland, united states of america email: kimberlymiller@towson.edu   received: 20 nov. 2015 accepted: 10 feb. 2016      2016 miller. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to measure the importance students place on criteria used to evaluate web-based information.   design – online, self-report questionnaire.   setting – secondary school in the united kingdom.   subjects – 149 students aged 13-18 years, representing a response rate of approximately 21% of the 713 students sampled.   methods – the authors used themes generated in a previous study of web-based information evaluation (pickard, gannon-leary, & coventry, 2010) to create a 10-item questionnaire about the importance of criteria used to evaluate web-based information. criteria represented in the questionnaire included accuracy, authority (2 statements), currency (2 statements), coverage, presentation, affiliation, source motivation, and citations. students used a four-point scale from “very important” to “not at all important” to indicate how significant they considered each criteria to be when they evaluated websites.   students received an email invitation to participate in the study, with a link to the questionnaire in the school’s sharepoint environment. two subsequent email reminders were sent approximately 8-10 weeks after the initial invitation to participate. teachers at the school were also asked to promote the questionnaire in their classes.   main results – over 75% of the 149 student respondents rated statements about presentation (n=116), accuracy (n=114), and currency (n=116) as “very important” or “quite important.” a majority of students (over 50%) rated the two statements about website authorship as being only “a little important” or “not at all important” (n=92, and n=86). however, 62% of students (n=92) indicated that a website’s sponsoring organization is “very important” or “quite important.” the authors suggest there were some differences between responses from older and younger students, with older students more likely to rate statements about coverage, citations, organization sponsorship, and source motivation as “very important” or “quite important.”   conclusion – the authors recommend that instruction about information evaluation for teenagers does not need to take a “back to basics” approach (p. 16), as most questionnaire respondents indicated they already find several criteria to be important when evaluating information. instead, instruction should address student opinions and misconceptions about web-based information in the context of their school assignments or other information needs. for example, students may be more motivated to learn about and apply evaluative criteria that are generated through discussion with their peers. students may also be more receptive to expanding information evaluation criteria when they are researching topics they find interesting or important. finally, the authors recommend that instruction should take into account the context or situations in which various evaluation criteria may be most important.   commentary   evaluating web-based information remains a key skill in today's technology-saturated world. documents such as the association of college and research libraries’ framework for information literacy for higher education and the american association of school librarians’ standards for the 21st century learner place evaluation skills at the heart of modern information literacy education and practice.   this study begins with an engaging literature review that describes the challenges underlying contemporary information evaluation. this review includes the historical significance of evaluation, the complex cognitive abilities required to formulate complete evaluations, and the centrality of evaluation in critical thinking and information literacy. the review provides insights into the complicated nature of studying, measuring, and teaching information evaluation skills.   material design, data collection, and data analysis methods are described in a clear and replicable manner, but questionnaire validity, data reporting, and result analysis are more difficult to evaluate when reviewed against glynn’s (2006) critical appraisal checklist. the data collection instrument was constructed using findings from a previous study of information evaluation criteria in higher education (pickard et al., 2010). constructing a questionnaire based on themes from a previous study signals instrument validity and allows the current study’s results to be compared to previous research results. there is less discussion about why teenagers’ “internal cues” (p. 9) are expected to match those of higher education users’ aside from the authors’ observation that existing information literacy instruction usually targets similar criteria.   the study results are presented in a table of descriptive statistics per questionnaire item and response option, while the narrative outlines underlying patterns in evaluation cues students deemed more or less important. a visual representation of other results, such as a list of the top three most and least important internal cues, or a table summarizing results by cue, would have helped the reader quickly interpret trends. sample size limitations restrict the study's ability to detect differences between students’ grade levels. this means the influence of confounding factors, like students’ developmental and ability differences, is unclear in the analysis. the study authors acknowledge these limitations and do not overextend their interpretation of the evidence.   librarians should find the study’s themes useful when designing information evaluation instruction. in particular, a study observation about how the classroom curriculum may influence information evaluation is worth further consideration in practice. for example, the study authors hypothesize that students value evaluation criteria presented in their regular course work, such as the emphasis teachers place on attention to grammar and presentation. more often than not, library or information literacy instruction takes place embedded within existing classes. this means understanding the greater curricular context in which students develop and internalize information skills and attitudes is essential for coherent, meaningful information literacy instruction.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154   pickard, a. j., gannon-leary, p., & coventry, l. (2010). users’ trust in information resources in the web environment: a status report. retrieved from http://ie-repository.jisc.ac.uk/470/2/jisc_user_trust_final_report.pdf    microsoft word news3_1472_final evidence based library and information practice 2008, 3:1  86 evidence based library and information practice       news     call for editorial advisors: evidence based library and information practice      © 2008. this is an open access article distributed under the terms of the creative commons attribution  license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and  reproduction in any medium, provided the original work is properly cited.       evidence based library and information  practice is expanding its editorial advisory  team and is seeking qualified persons to  apply for a two year term.  the international  editorial advisory team currently consists  of information professionals representing  numerous areas of library and information  studies.  advisors are expected to review  approximately 4 manuscripts per year.   manuscripts include original research and  evidence summaries.  advisors should be  familiar with evidence based practice and  research methods.  we are particularly  interested in applicants with experience in  school and public librarianship, information  technology, archives, and non‐traditional  library services and environments.     interested persons should send their  resume, along with a list of 5 areas of  interest/specialization, to lindsay glynn  (lglynn@mun.ca) no later than april 1, 2008.               conference paper   library support for indigenous university students: moving from the periphery to the mainstream   joanna hare faculty librarian bond university robina, queensland, australia email: joannahare@outlook.com     wendy abbott university librarian bond university robina, queensland, australia email: wendy_abbott@bond.edu.au   received: 15 aug. 2015  accepted: 08 oct. 2015         2015 hare and abbot. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – this research project explored the models of indigenous support programs in australian academic libraries, and how they align with the needs of the students they support. the research objective was to gather feedback from indigenous students and obtain evidence of good practice models from australian academic libraries to inform the development and enhancement of indigenous support programs. the research presents the viewpoints of both indigenous students and librarians.   methods – the research methods comprised an online survey using surveymonkey and a focus group. the survey was conducted nationally in australia to gather evidence on the different models of indigenous support provided by academic libraries. the survey explored the nature of support services such as specialized study spaces and resources, information literacy education, and liaison services for indigenous students. the survey also asked respondents to comment on the challenges they encountered and improvements they would recommend in providing indigenous student support.   to provide a student perspective, a small cohort of indigenous students at a small university in south east queensland was interviewed in a focus group about their library experiences. the focus group explored indigenous students’ perceptions of the library, their frequency of use and where they go for help with their studies.   results – the survey found that 84% of academic libraries provide some specific support for indigenous students with 89% of those support services being conducted in a place other than the library. across the sector, australian academic libraries have a strong commitment to the success of indigenous students and considerable engagement with indigenous issues.   the focus group found that indigenous students’ needs and concerns about using the library were not differentiated by their cultural background. rather their concerns were similar to issues being raised in the broader student population.   conclusion – the survey results indicated that the main areas in which support for indigenous students might be improved are greater inter-departmental communication and collaboration within the university, increased training of library staff in indigenous cultural sensitivity, and the employment of indigenous library staff members. the focus group was valuable in opening the communication channels between indigenous students and library staff and highlighted the importance of engaging with students using both formal and informal channels. introduction   access to education, especially the tertiary sector, is widely seen as a stepping stone to economic and social success in modern australian society. indigenous disadvantage is a major deficit in australia which can be addressed in part through improving educational outcomes for indigenous students. universities are endeavouring to provide culturally sensitive, in-depth support to ensure more indigenous students complete a university education, leading to greater workforce participation and personal success (pechenkina & anderson, 2011, p.5). in this article, the term ‘indigenous’ has been used to describe people who identify as being of aboriginal and/or torres strait islander background.   in most australian universities, special support programs for indigenous students are provided by an indigenous education centre which coordinates support from faculties and administration, and may also provide pastoral or cultural care. page and asmar (2008) argue that these visible support structures are just the “tip of an iceberg” (p. 109), and that informal and invisible modes of support are difficult to measure. the 2012 review of higher education access and outcomes for aboriginal and torres strait islander people final report (hereafter the iher report) recommended a shift away from these isolated indigenous education centres to a “whole-of-university effort” (department of education and training, 2012, p. xii), thereby sharing the responsibility for supporting indigenous students across a university.   academic libraries are well placed to make a significant contribution to indigenous student retention and success rates. as this research will show, academic libraries not only support students in indigenous education centres, but also provide informal support through service desk encounters and participation in cultural events. this is important, because as asmar, page, and radloff (2011) found:   …there is a direct link between how [indigenous] students perceive institutional support, and whether or not they intend to depart prior to completion. the issue of support, therefore, is far from peripheral to the optimizing of indigenous student engagement. rather, it is crucial. (p. 9)   in providing support to indigenous students, libraries can refer to university policies as well as professional practice guidelines such as the aboriginal and torres strait islander protocols for libraries, archives and information services (hereafter the protocols) to ensure their services are appropriately targeted and culturally sensitive. introduced in 1995, the protocols set new standards aimed at ensuring the rights and needs of indigenous people in libraries. much of the literature that references the protocols has focused on collecting and archiving indigenous material, leaving a gap in understanding how libraries provide face-to-face and online services to indigenous people (protocol 4: accessibility and use).   this research seeks to present two perspectives on support services for indigenous students in academic libraries: that of the academic libraries providing the support, and the perspectives of a sample of indigenous higher education students at a small regional university in south east queensland. as stated by foley (1996), the thoughts and opinions of indigenous students regarding the support they receive in an academic setting has rarely been sought. requesting the input of indigenous students seeks to “ensure meaningful aboriginal and torres strait islander participation in effective development, adoption and implementation of relevant policies” (aboriginal and torres strait islander library information and resource network, 2012). the research also aims to ensure future efforts in indigenous support by academic libraries address specific student needs.   literature review   the current literature about the relationship between libraries and indigenous communities in australia largely focuses on the collecting, archiving, and maintenance of indigenous collections (as addressed by protocols 2, 3, 5, 6 and 7), rather than how libraries provide customer service and support to indigenous people (protocol 4). the connection between access to accurately-described indigenous collections and the interest of indigenous communities in libraries in general cannot be underestimated, as described by nakata, byrne, nakata and gardiner (2006a): “…both the goal of preserving indigenous documentary heritage and the goal of increasing the relevance of libraries for indigenous peoples is assisted by local documentation of indigenous knowledge and indigenous perspectives of historical experience” (p. 15). if a library demonstrates a commitment to engaging indigenous customers in collecting and archiving material, the interest, comfort and confidence of indigenous people in using libraries may increase. that being said, this literature review will focus on the limited number of articles about how libraries provide customer service and support to indigenous people (protocol 4), as well as a small number of australian articles about the interaction between indigenous higher education students and academic libraries.   novak and robinson (1998) highlight the importance of consulting and collaborating with indigenous students in designing library services. focus groups were conducted with indigenous university students to gather their perspectives on using the library. the students shared their perspective that the library space felt both physically and psychologically intimidating. the students also found library staff intimidating:   ...alienation from staff was [an] issue for these students, with the feeling that some staff have no time for them… when seeking help in the library, some were ashamed to ask for help and very reluctant to ask a second time if the first exchange did not solve their problem. (novak & robinson, 1998, p. 20)   first-time users in particular may find libraries to be intimidating, a matter garwood-houng (2006) addresses, stating that “all staff who work on reception desks… need to be able to deal with indigenous people, information and issues” (pp. 129–130). nakata, byrne, nakata, and gardiner (2006b) also discuss the importance of welcoming and sensitive customer service, arguing that “...five minutes of helpful and friendly attention to an indigenous client could be the difference between that person using a library or archive for a lifetime or not ever entering one again” (p. 169).   in identifying models of library service for indigenous higher education students, nakata, byrne, nakata, and gardiner (2006b) indicate that some academic libraries have liaison librarians for indigenous students, designated study spaces, and targeted orientation programs. a 2009 survey of the six members of the libraries of australian technology network (latn) and two partner libraries found that all the surveyed libraries have a relationship with the university indigenous education centre, and all but one provide special outreach programs for indigenous students (vautier, mcdonald, & byrne, 2009). the survey also found that indigenous cultural awareness training is limited and often only offered as part of staff orientation programs.   nakata et al. (2006a) comment that while indigenous people should have the same equitable access to library services and collections as non-indigenous people, the library and information sector also needs to acknowledge that “both in the collective sense and in terms of many indigenous individuals and communities, there are indigenous needs and interests that are distinct from those of other groups of users” (p. 14). nakata and langton (2006) advocate for collaboration with indigenous communities in the design of library services. this should move beyond simple consultation to:   …dialogue, conversation, education, and working through things together… providing the opportunity and means for indigenous people to be part of what they determine should be done. it is not about being focused just long enough to fix a problem, but is about investment in the issues for the long term and for future generations. (nakata & langton, 2006, pp. 4–5)   this review of the literature shows that a detailed description of how indigenous students are supported by academic libraries is lacking. this paper seeks to address this gap by reporting findings from a nation-wide survey of australian academic libraries together with the outcomes of an indigenous student focus group on their perspectives and experiences of using the library.   aim   the aim of this research was threefold:   1.       to survey the australian academic library community for models of service and examples of good practice in supporting indigenous students. 2.       to develop an understanding of the library needs of indigenous university students at a small regional university in south east queensland by means of a focus group. 3.       to distil and draw on indigenous student feedback and good practice models to inform the development and enhancement of indigenous support programs at a small regional university in south east queensland.   other academic libraries both locally and internationally may draw on the findings to inform the design of support for indigenous students in their own institutions.   methods   to compare the different models of indigenous support provided by academic libraries, primary data was gathered through two instruments: a nationwide survey of australian academic libraries regarding the types of services being offered and a focus group conducted with indigenous students at a small regional university in south east queensland.   the survey was circulated to all 39 australian academic libraries via the council of australian university librarians (caul) email distribution list and was conducted online using surveymonkey. the survey investigated the range of library support provided to indigenous students including the availability of specialized study spaces and resources, information literacy education, and liaison services. the survey asked respondents to comment on the factors that informed or influenced the nature of the library support provided, the challenges they encountered, and the improvements they would recommend in providing indigenous student services. please refer to appendix a for the survey questions.   to gather the perspectives of indigenous students at the university, one-on-one interviews were proposed. all students at the university who self-identify as aboriginal or torres strait islander were invited via email to participate in the interviews with the indigenous library liaison officer (also an author of this paper). staff at the indigenous education centre provided the students’ email addresses, and encouraged students to participate by word of mouth. however, a low response rate meant the approach had to be changed. instead, the same students were contacted a second time to invite them to participate in a focus group. the indigenous library liaison officer makes scheduled weekly visits to the centre at a prearranged time, and so the focus group was scheduled for the same time. five students responded to the email request to take part in the focus group, and a sixth student who happened to be at the centre at the time volunteered to participate. the focus group sought qualitative evidence of how the students use and perceive the library. rather than to generalize across the indigenous student population, the intention of the focus group was to develop a nuanced understanding of how specific indigenous students manage in an academic library environment.   as the research involved a minority group, a full ethics review was conducted by the university’s research ethics committee and the research methodology was approved. the australian institute of aboriginal and torres strait islander studies (aiatsis) guidelines for ethical research in australian indigenous studies were followed to ensure the students were treated respectfully and that there was agreement about how their data would be used. results   in reporting the results of the survey, percentages have been used to represent findings of quantitative questions. whole numbers have been used to show patterns in qualitative responses.   survey responses   thirty one responses to the survey were received; a response rate of 79%. the main finding is that 84% of the libraries that responded to the survey provide some specific support for indigenous students, whether that is in services, resources, or facilities. the majority (89%) of these services are conducted in a place other than the library, such as in the university’s indigenous education centre.   the survey found the most commonly provided supports are a designated library staff member in a liaison role and targeted information literacy classes (figure 1). figure 1 what is the nature of indigenous student support provided by library staff?     amongst the ‘other’ category, four libraries referred to unique library services for remote indigenous students, such as extended borrowing privileges, resources provided in print or on usb sticks for students who may not have access to a reliable internet connection, and intensive library training workshops for remote students visiting campus. also mentioned were writing workshops, attendance at and support of indigenous cultural events on campus, research workshops about using indigenous materials, consultation with the indigenous education centre regarding collection development, and the representation of the indigenous education centre on the library advisory board. the variety of responses shared in the ‘other’ category reflects the willingness of library staff to adapt their services based on the needs of diverse indigenous user groups.   in terms of resources designed to support indigenous students, 86% of surveyed libraries provide online resources, 32% offer print resources, and 18% offer videos. libguides were frequently mentioned as an online resource for indigenous students, but these were most often designed to support a course or subject in indigenous australian studies, rather than to support indigenous students. this supports nakata et al. (2006b) who found that collection in academic libraries tends to be in aid of supporting the curriculum, rather than for archival purposes.   the majority (92%) of libraries do not provide dedicated computing facilities for indigenous students. few libraries (17%) provide specific spaces for indigenous students. this is almost certainly because access to spaces and computing facilities are typically made available by indigenous education centres.   respondents were asked to comment on the provision of cultural sensitivity training at their institution. indigenous cultural awareness training is given to library staff at 65% of the libraries that responded to the survey. of these libraries, 45% reported that all library staff are given training, while 5% reported that only staff providing indigenous support were given training. this selective approach to cultural sensitivity training was criticized by one respondent who commented:   the library believes it is not good enough to have a few designated staff to support indigenous students. we are open long hours and offer a range of services. indigenous students should be confident of getting appropriate support whoever they encounter in the library.   the survey responses make clear there is little consistency across academic libraries in terms of the nature of cultural sensitivity training, and this is similar to the findings from the 2009 latn survey (vautier et al., 2009) referenced in the literature review. six respondents indicated training was offered or “encouraged” but was not compulsory. the type of training varies, from short sessions to half and whole-day workshops run by university departments such as human resources, “local indigenous people”, or external providers. two libraries are developing comprehensive training programs for library staff, one that includes a twelve month review.   when asked to comment on what factors informed and influenced the design of library support for indigenous students, 12 respondents reported referring to university reconciliation statements or other formal policies pertaining to indigenous people. one respondent stated the university reconciliation statement is “an important document underpinning… planning and review processes” while another said the reconciliation statement is “the framework for the provision of information sources to support indigenous studies and enable all students to develop understanding and respect for indigenous traditional and contemporary cultures”. finally, one respondent stated the university’s reconciliation statement did not have a direct impact on service design but had strengthened their relationships with staff in the indigenous education centre and indigenous student representatives.   beyond formal institutional policies and documents, a range of consultation strategies provided guidance in the design of library support for indigenous students. for example, 14 libraries consulted indigenous staff regarding student needs; 9 of these libraries specified that the staff consulted were academic staff. four libraries reported incorporating feedback from indigenous students, and three reported membership on various university committees that discuss indigenous matters. one library said that they consulted with indigenous tutorial assistance scheme (itas) tutors. three libraries reported that cultural training for staff informed support.   surveyed libraries were asked to comment on how closely their library’s support for indigenous students follows the protocols, which elicited a range of responses. some libraries are aware of the protocols and follow them to a degree, for example, committing to aspects of the protocols such as recruitment targets. six libraries demonstrated a strong, ongoing commitment to the protocols which are built into daily practice, with one library saying they refer to the protocols “constantly” and another saying their work is regularly reviewed against the protocols to “ensure consistency of library service”. when asked about their awareness of the protocols, 6 respondents reported they were not aware of them. however, upon being made aware of the protocols, 3 respondents commented they were satisfied their programs adequately addressed them.   some of the most varied and impassioned responses in the survey were received when respondents were asked to comment on the challenges of supporting indigenous students, and how services for indigenous students might be improved. for example, one respondent commented that “building confidence of the students [is a challenge,] that they are not only welcome in the library but will be actively and appropriately support [sic] to achieve the study outcomes they want.” another respondent shared their perspective that:   indigenous students should not just receive “special services” but should come to [the university] and know that they belong. they should be represented within the student and staff population; they should be represented in the curricula they are learning; and they should see their culture in the spaces they use; and they should have opportunities to participate in cultural events.   three libraries reported that developing personal relationships with students could be challenging. staff cultural sensitivity was identified as a challenge by four of the respondents, with one library adding it has been their “biggest challenge.” eleven respondents suggested that if the library had greater interaction, cooperation, and collaboration with other departments in the university the support services offered to indigenous students would improve. specific departments mentioned were learning support services and itas tutors, academic advisors and staff who support indigenous students, and indigenous student groups. five respondents reported that employing an indigenous person to provide direct support to indigenous students would improve their services in alignment with the protocols.   on a practical level, some libraries addressed the issue of remote indigenous students. this ranged from how to effectively support remote students who may not have consistent internet access to the return of items by their due dates. two libraries reported the difficulties in reaching all enrolled indigenous students, with one library stating “it would be helpful if the library could somehow ‘capture’ all the students. there are some continuing students that i have never met.” one respondent reported that they visit the indigenous education centre regularly, but not all the students use the centre. another said their roving librarian service to the indigenous education centre was not well used. despite this, they felt that a roving librarian in the centre was valuable for development of relationships with other staff that support students, not necessarily the students themselves.   focus group responses   the focus group comprised five undergraduate students and one postgraduate student. therefore, the following presentation of their perspectives must be understood as primarily that of undergraduate students. the students’ course progression ranged from first semester, to half way through their program, to their final semester. the majority of students reported using the library spaces every day or once a week, with increased usage closer to exams. the students use the library for printing, scanning, borrowing books, lecture streaming, access to the library computers and for group and individual study. one student reported that working at the library was “more motivating than studying at home”. all students said they preferred the communal study areas and only use the silent study areas occasionally. students commented the library could improve on the availability of scanners and computers.”   when asked how they learned about the library, several of the students reported that a subject-specific liaison librarian had visited their class to tell them about the library services in general and specific resources for that subject. one student mentioned that he had a librarian visit two of his classes in his first semester. when asked if reiteration of the content of classes was helpful, he said “when you’re in your first semester, yeah, because after a while you kind of get into the habit of knowing where to go.” two students explained that in their first two or three semesters they didn’t understand that different library databases included different content, and they could use a combination of these databases for their research. they felt that this could have been emphasized more by library and academic staff.   when asked where they go for help with their studies, one student said his main point of contact was other students. two students said they would ask their lecturers for assistance, but not always with success. one student said “sometimes lecturers are reluctant to help you. and they’ll just be like: it’s easy; you’ll know how to do it!” one student in her first semester said:   i find it difficult to go and talk to people about it. i usually just try and search for it on my own but the only problem with that is i don’t know how to right now. i spent like three hours trying to do it on my own when i probably should have just asked for help.   the students universally agree that if a lecturer is unhelpful once they would not approach them for help a second time.   in general the students perceive the help they receive from the library as satisfactory, preferring to visit the help desk than use online tools like email or chat. one student commented that “some of [the library staff] aren’t very helpful. they will try, but they are usually a dead end anyway. and i’m like, never mind, i’ll just go find it.” another student says “you kind of know who to approach now. because i’ve been here so long now that i’m like, i won’t talk to you because last time i talked to you it didn’t go down so well.” another student says that some staff have “barriers”.  these sentiments reflect the work of novak and robinson (1998) and nakata et al. (2006b), as outlined in the literature review, that library staff and spaces can sometimes be unfriendly or intimidating for indigenous students.   at the conclusion of the focus group the indigenous library liaison officer pointed out to the students that even though the purpose of the discussion was to find out what they as indigenous students needed from the library, they had not made any comments about their indigeneity. the students were prompted to talk about how their needs or experiences might differ from the broader student population, but the only comment made was to enquire as to whether the library had an indigenous collection.    discussion   this research was primarily an information gathering exercise. the aims of the research were to survey the australian academic library community regarding their support for indigenous students, and to gather the perspectives of indigenous students at the university. the information gathered by the survey and the focus group has identified areas of improvement at the authors’ institution and has generated ideas about new ways of engaging and supporting indigenous students. the survey results were shared with all caul libraries, and prior to publication the authors were contacted by two other libraries interested in the research findings as inspiration for improving their support for indigenous students. this demonstrates the value of the research to the wider academic library community. the following section will discuss the results of the survey and the focus group, and how those findings have prompted change at the authors’ institution.   with such a small group of students, and with the diversity of the indigenous student cohort, the goal of the focus group was not to draw blanket conclusions or generalize across the cohort. the exercise was valuable for opening the channel of communication with the students. therefore, the authors will seek to meet with indigenous students in this way regularly to ensure services continue to meet the needs of a changing student population. a formal, yearly focus group to interact and share feedback will be supplemented by informal communications, such as serendipitous conversation or emails. this allows for casual feedback to be made and provides opportunities for students who may be intimidated by formal consultation processes to share their thoughts.   the comments in the survey of australian academic libraries regarding increased inter-departmental consultation and cooperation echo the 2012 iher report. the iher report recommended a “fundamental shift from often marginalized indigenous education units bearing responsibility for supporting aboriginal and torres strait islander students, to a whole-of-university effort” (department of education and training, 2012, p. xii). making connections with the various departments that contribute to the success of indigenous students will strengthen the library’s and the university’s approach to supporting indigenous students and ensure there are no gaps or inconsistencies in assisting students through the various stages of their academic careers. at the institution where this research was conducted, the library was already connected to the indigenous education centre via the indigenous library liaison officer. this positive relationship and findings of the survey and focus group have motivated the library to engage indigenous students more directly. this has involved working closely with the indigenous students’ representative group to contribute to cultural events on campus, such as national aboriginal and islander day observance committee (naidoc) week. the library has also formally adopted the indigenous literacy foundation as their endorsed charity, and collaborates with the indigenous students’ representative group to host fundraising events. these occasions raise the profile of indigenous students and groups on campus, while the community aspect allows many different groups on campus to attend and participate.   the survey shows that respondents believe increasing the number of indigenous employees in libraries would improve services for indigenous students. the survey found that 8 percent of the surveyed libraries employ an indigenous person. employment of indigenous people in libraries is also addressed by protocol 8, which states libraries should “aim to reflect the composition of the client/community population in each organization’s staffing profile” and “take affirmative action to recruit and promote aboriginal and torres strait islander peoples” (aboriginal and torres strait islander library information and resource network, 2012, protocols 8.1 and 8.2). increasing indigenous employment figures in libraries requires a long term commitment from library management to actively recruit in this area. libraries can also make a commitment where possible to engage indigenous library and information students in work placements and internships. this is an area identified for improvement at the institution where this research was conducted.   cultural awareness training for staff is crucial in ensuring indigenous students feel welcome and supported in the library. as garwood-houng (2006) argues, all staff who work on service desks “need to be able to deal with indigenous people, information and issues” (pp. 129–130). responses to the survey suggest cultural sensitivity training is carried out inconsistently across the sector, and comments from the student focus group reiterate that one unpleasant experience with the library can define a student’s library experience long term. as a highly visible point of customer service within universities, libraries have an obligation to provide a high quality of customer service universally to all students. to do so, the number of library staff receiving cultural competency training in indigenous perspectives and issues should be closer to 100%. training library staff in indigenous cultural competency has many benefits beyond direct customer service; for example, in developing culturally sensitive information literacy sessions and embedding indigenous perspectives in library materials. at the institution where this research was conducted, an indigenous cultural sensitivity training program has been implemented specifically for library staff. this training has been developed with the indigenous cultural support officer who also presents the training. the training is a one hour session that is mandatory for all full time staff. it is planned to extend the training to casual staff, and to provide an updated training session each year to maintain awareness and ensure all new staff receive training.   the main limitation experienced in the course of this research was in the recruitment of students to participate in interviews, and subsequently, the focus group. this was also addressed as a limitation in research undertaken by novak and robinson (1998). furthermore, the diversity of the group could have been greater as five of the six participants were undergraduate students and there were no mature age students. a larger student group that also included some mature age students would have been preferable for this research project.   the findings from this research point to many different areas for future research. for example, interviews or focus groups with indigenous students from multiple universities, with a mixture of ages and degree levels (undergraduate and postgraduate), would provide a more detailed picture of how students experience academic libraries across the country. future research might also look to how libraries support indigenous higher degree researchers and indigenous academic staff. in-depth research into whether indigenous students’ and non-indigenous students’ information seeking and searching behaviour differ would be beneficial to the field. the nature of and potential for cultural sensitivity training in libraries would also benefit from further research. finally a collaborative approach to modelling library support and surveying indigenous students from different countries, such as canada or new zealand, would enhance knowledge in this field.   conclusion   this paper reports the findings of a national survey of australian academic libraries regarding their support for indigenous higher education students. the survey found that libraries are providing a range of support programs and services, responding to the diversity of the indigenous student population. ultimately the survey showed australian academic libraries have a strong commitment to the success of indigenous students and considerable engagement with indigenous issues. the main areas for improvement identified by the survey are greater inter-departmental communication and collaboration within the university, increased training of library staff in indigenous cultural sensitivity, and the employment of indigenous library staff members.   the focus group with indigenous students was a valuable exercise. firstly, it provided a platform for indigenous students to share their ideas and feedback with library staff. secondly, the collection of ideas and feedback provides an opportunity to incorporate indigenous perspectives into the design of library services. finally, the most beneficial part of conducting the focus group was making connections with students. it is recommended that making opportunities for the collection of authentic feedback from indigenous students and staff should be a regular occurrence at australian academic libraries.   working toward the recommendations made in this paper is aided by building the aboriginal and torres strait islander protocols for libraries, archives and information services into professional library practice. in doing so, library services will develop in ways that not only benefit indigenous students but the wider student and university community. a library service that is culturally sensitive and celebrates indigenous culture and knowledge can contribute to a “whole-of-university approach to aboriginal and torres strait islander student success” where “faculties and mainstream support services have primary responsibility for supporting aboriginal and torres strait islander students” (department of education and training, 2012, p. xix). a culturally competent library can take a leadership role in embedding indigenous perspectives not only in library services, resources and collections but in the overall university culture.   references   aboriginal and torres strait islander library information and resource network. (2012). aboriginal and torres strait islander protocols for libraries, archives and information services. retrieved from http://www.aiatsis.gov.au/atsilirn/protocols.php   asmar, c., page, s., & radloff, a. (2011). dispelling myths: indigenous students’ engagement with university. australasian survey of student engagement (ausse) research briefing, 10(april). retrieved from http://www.acer.edu.au/documents/aussereports/ausse_research_briefing_vol10.pdf   department of education and training. (2012). review of higher education access and outcomes for aboriginal and torres strait islander people: final report. retrieved from https://education.gov.au/review-higher-education-access-and-outcomes-aboriginal-and-torres-strait-islander-people   foley, d. (1996). perspectives on effective student support for indigenous students in a tertiary institution. the australian journal of indigenous education, 24(2), 53–55. http://doi.org/10.1017/s1326011100002477   garwood-houng, a. (2006). protocols: meeting the challenges of indigenous information needs. in m. nakata & m. langton (eds.), australian indigenous knowledge and libraries (pp. 125–131). sydney: uts epress. retrieved from https://opus.lib.uts.edu.au/research/handle/2100/835   nakata, m., byrne, a., nakata, v., & gardiner, g. (2006a). indigenous knowledge, the library and information service sector, and protocols. in m. nakata & m. langton (eds.), australian indigenous knowledge and libraries (pp. 7–18). sydney: uts epress. retrieved from https://opus.lib.uts.edu.au/research/handle/2100/835   nakata, m., byrne, a., nakata, v., & gardiner, g. (2006b). libraries, indigenous australians and a developing protocols strategy for the library and information sector. in m. nakata & m. langton (eds.), australian indigenous knowledge and libraries (pp. 162–172). sydney: uts epress. retrieved from https://opus.lib.uts.edu.au/research/handle/2100/835   nakata, m., & langton, m. (2006). introduction. in m. nakata & m. langton (eds.), australian indigenous knowledge and libraries (pp. 3–6). sydney: uts epress. retrieved from https://opus.lib.uts.edu.au/research/handle/2100/835   novak, j., & robinson, g. (1998). “you tell us”: indigenous students talk to a tertiary library. australian academic & research libraries, 29(1), 13–22. http://doi.org/10.1080/00048623.1998.10755034   page, s., & asmar, c. (2008). beneath the teaching iceberg: exposing the hidden support dimensions of indigenous academic work. australian journal of indigenous education, 37(2008), 109–117.   pechenkina, e., & anderson, i. (2011). background paper on indigenous australian higher education: trends, initiatives and policy implications. department of education and training. retrieved from http://docs.education.gov.au/documents/background-paper-indigenous-australian-higher-education-trends-initiatives-and-policy   vautier, c., mcdonald, j., & byrne, a. (2009). indigenous working group report: where we are in 2009. unpublished manuscript, libraries of australian technology network, australia     appendix a survey questions sent to 39 members of the council of australian university librarians   please note: the survey was designed and circulated using surveymonkey, so the following questions do not reflect the full survey logic.   introduction   1.       library name 2.       name of person completing survey 3.       name of person who can be contacted for additional information 4.       does your library provide support specifically designed for indigenous students (services, resources, facilities)? yes/no   staff services   5.       if yes, what is the nature of the library support provided (please tick all applicable boxes)? a.        employment of one or more indigenous staff to provide support                                         b.        designation of one or more staff members in a liaison/support role                    c.        allocation of staff hours to provide support                                                                            d.       targeted information literacy classes                                                                                       e.        other (please describe)                                                                                                                                  6.       are some or all of these staff services conducted in a place other than the library (e.g.: in separate university indigenous support centres)? yes/no/please comment   library resources   7.       please indicate any specially designed resources for indigenous students that your library provides:                                a.        videos                                                                                                                                                            b.        print guides/leaflets/booklets                                                                                                     c.        libguide(s)                                                                                                                                   d.       other web-based resources                                                                                                          e.        any other resources (please describe)                                                                                                                                          8.       does your library provide dedicated information technology for indigenous students (e.g. computers, wireless devices, etc.)? yes/no/please describe   9.       does your library provide dedicated library spaces for indigenous students? yes/no/please describe.   10.    please describe any other support that your library provides for indigenous students that has not been covered by previous responses.   design of library support for indigenous students   11.    how closely does your library's support for indigenous students follow the aboriginal and torres strait islander protocols for libraries, archives and information services? please comment.   12.    please describe any other factors that have informed your indigenous students support (e.g. input from indigenous people, advice, consultation or research).   13.    please describe any factors external to the library that have influenced your indigenous support (e.g. university, state or national policies, funding, etc.).   evaluation   14.    has your library undertaken any formal evaluation of your indigenous student support? yes/no/please comment   15.    if yes, based on the evaluation, how successful is your library’s indigenous student support? (circle one) a.        not successful at all b.        not very successful c.        successful d.       very successful e.        please comment   if no, based on your own impressions, how successful is your library’s indigenous student support? (circle one) a.        not successful at all b.        not very successful c.        successful d.       very successful e.        please comment   16.    how would you improve your library’s indigenous student support?   17.    what are the challenges in providing library support to indigenous students?   18.    to what extent does your library collaborate with the university's indigenous education centre in providing support for aboriginal and torres strait islander students? please describe:   19.    are library staff given any training in indigenous cultural issues? yes/no   20.    if yes, which staff are given training? a.        all library staff                                                                                                                                            b.        only staff providing indigenous support                                                                                                c.        other / please comment   21.    please describe the nature of the training provided.       microsoft word es5_9322_final evidence based library and information practice 2008, 3:1  68 evidence based library and information practice   evidence summary    local purchasing of journals is required in addition to a nationally purchased  collection to meet the information needs of nhs staff.     a review of:  glover, steven william, john addison, colette gleghorn, and john bramwell. “journal usage  in nhs hospitals: a comparison report of total usage at an acute nhs trust and a  specialist nhs trust in the north west of england.” health information and  libraries journal 24.3 (2007): 193‐9.      reviewed by:  jennie kelson  library services manager, buckinghamshire hospitals nhs trust  amersham, united kingdom  e‐mail: jennie.kelson@buckshosp.nhs.uk       received: 3 december 2007      accepted: 17 february 2008      © 2008 kelson. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    abstract    objective ‐ to compare journal usage  between an acute national health service  (nhs) trust and a specialist nhs trust  located in north west england to provide  some evidence as to how well the national  core content collection (provided by  proquest) meets the needs of staff in these  settings.    design ‐ comparative study     setting ‐ an acute nhs trust, comprising  four hospital sites, and a cancer specialist  nhs trust based on a single site. both  trusts are located in north west england.  the cancer specialist nhs trust is a  teaching hospital with undergraduate  nurses, medical students, and student  radiographers. this trust is also closely  associated with an adjoining cancer  research institute. the acute nhs trust  has a large number of healthcare staff in  training and was not described as a  teaching hospital.    subjects ‐ staff of the respective nhs  trusts. the staff numbers for each  organisation were not provided.      methods ‐ counter usage statistics of  online journals, obtained from publisher  administration tools, were collected for  one year covering the period 1 december  2005 to 30 november 2006. where  evidence based library and information practice 2008, 3:1  69 available, the number of photocopies  made from print journals during the same  period by library users for their own use  was also included. all full‐text downloads  of journal articles were counted as part of  this study, hence the possibility of double  counting if a single article was requested  in both html and pdf versions. details  of free or open access articles accessed  without the need for a username and  password were not included in the study.    to encourage use of the electronic  journals, library services at both trusts  implemented a number of initiatives to  maximize publicity. these included direct  e‐mails to staff, posters, and presentations  to staff. athens registration, required for  access to the electronic journal collections,  was promoted as part of the induction  process for new library users. staff  members were encouraged to apply for  the electronic table of contents alert for  journals in their area of specialty. an a‐z  list of journals was accessible via the  trusts’ intranet and internet sites, and  direct links to electronic journals were  added to the nhs dialog/datastar  databases and on pubmed.    main results ‐ the libraries at both trusts  in the study provide the majority of their  journals in electronic‐only format. in  addition to the national core content  titles, the cancer specialist trust provides  access to an additional five journal  collections: science direct health sciences,  blackwell synergy medical and nursing  collection, five journals selected from  nature publishing group, selected society  journals via highwire press and oxford  university press titles. staff members of  the acute trust have access to an  additional three journal collections:  science direct health sciences, the bmj  publishing group (bmjpg) collection and  16 lww titles via ovid.    during the study period, a total of 93,376  articles were downloaded or copied. of  these, 47,079 articles were downloaded or  copied by staff of the cancer specialist  nhs trust and 46,297 articles were  downloaded or copied by staff of the acute  nhs trust. the usage of each of the  journal collections during the study period  for each trust is shown in the table below  with the most used resource shown in  bold.    content  provider  cancer  trust  acute  trust  highwire press  hosted journals  19,773  (42%)  n/a  science direct  health sciences  10,808  (23%)  15,844  (34.2%)  nature  publishing  group   6,034  (12.8%)  n/a  blackwell  synergy medical  and nursing    4,487  (9.5%)  n/a  oxford  university press   3,387  (7.2%)  n/a  national core  content  proquest  2,118  (4.5%)  13,834  (29.9%)  bmjpg journals  n/a  5,440  (11.8%)  lww via ovid  n/a  2,279  (4.9%)  photocopies from  print journals  472 (1%)  8,900  (19.2%)  total 47,079  46,297    general information regarding some of the  commonly used journal titles by staff of  each trust was included in the  commentary, but levels of use were not  enumerated.  staff of the cancer specialist  nhs trust favoured cancer‐related titles,  whilst staff of the acute nhs trust  accessed journal titles over a greater range  of subject areas. details of the top titles  accessed from the national core content  collection were not provided for either  trust. a possible reason given for the  lower usage level of the national core  content collection by the cancer specialist  nhs trust compared to the acute nhs  trust was the presence of embargoes on  evidence based library and information practice 2008, 3:1  70 many cancer‐related titles within the  national core content collection.    information about training offered to staff,  their level of access to computer facilities,  or familiarity with accessing electronic  journal collections was not provided.      conclusion ‐ staff of both trusts in the  study recorded a high level of journal use,  with a total of 93,376 articles downloaded  or photocopied during the study period.    there was a marked difference in the  usage patterns of the national core  content journals between the two trusts  studied with the acute nhs trust showing  a higher proportion of usage (29.9%)  compared with the cancer specialist trust  (4.5%). staff members of the acute nhs  trust accessed a greater range of subject  areas, while staff those at the cancer  specialist nhs trust favoured cancer‐ related titles.    the results indicated that the national  core content collection did not meet the  information needs of the specialist cancer  trust as well as it met the needs of the  more generalist acute trust. the national  core content collection appears  insufficient to meet the diverse  information needs of all nhs staff. local  purchasing of journals, in addition to  national level provision, is therefore  required to ensure that the needs of local  specialties are adequately supported.    commentary    this study compares overall usage of  journal collections by staff of a cancer  specialist nhs trust with that of staff at  an acute nhs trust. the main conclusion  from the study was that the national core  content collection does not met the  information needs of a specialist cancer  trust as well as it does a more generalist  acute trust. however, there are a number  of uncertainties in the study design that  make it difficult to draw firm conclusions  from the study results.     no justification is provided for the choice  of trusts included in the study. whilst  they may have been chosen for  convenience, it would have been useful if  the authors had clarified how these  particular settings addressed the research  question.     the validity of the study results would  have been stronger if more information  had been provided about the various  journal collections available to staff as part  of the study. for example, it would have  been useful to know the extent of any  overlap between the various online  collections, whether any of the print  journals duplicated those also available in  electronic format, and how many print  journal titles the respective libraries held.      although told that both trusts provided  an a‐z list of journals available, we do not  know whether staff members were given  the option to access duplicated journal  titles from more than one provider. if staff  were offered multiple access options,  either via the a‐z list or via the direct  links from the dialog/datastar or pubmed  databases, we do not know whether a  particular provider was displayed first.  either of these situations may result in  increased usage of journals from a  particular provider. it would also have  been useful to know how many staff used  the electronic table of contents alerts  facility. usage of journals from a particular  provider would be expected to be higher if  a greater number of staff were accessing  articles through the table of contents  alerts. in addition, the level of familiarity  with the publisher interface for the various  journal collections may have had a bearing  on whether staff preferred to use a  particular collection. we do not know  when the various journal collections were  first purchased by the respective trusts  nor the extent of any training and  evidence based library and information practice 2008, 3:1  71 computer facilities provided to help staff  use these resources.    overall, given the limitations of the study  design, the data presented provides little  firm evidence to support the authors’  conclusions other than in a general way.   review article   information literacies of phd students in the health sciences: a review of scholarly articles (2009 2018)   elisabeth nylander research librarian jönköping university library jönköping, sweden e-mail: elisabeth.nylander@ju.se margareta hjort instruction librarian jönköping university library jönköping, sweden e-mail: margareta.hjort@ju.se   received: 26 aug. 2019                                                                  accepted: 1 nov. 2019      2020 nylander and hjort. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29630     acknowledgements   this project was made possible in part by the institute of museum and library services (re-95-17-0025-17). we thank the research training institute of the medical library association for its training, support, and encouragement to carry out this research. we thank our library director mattias lorentzi for providing us with the opportunity to conduct this project. we also thank anna abelsson and thomas mattsson for comments on the manuscript.   abstract   objective – doctoral studies offer a unique phase in the development and legitimization of researchers, in which phd students shift from the consumption to the production of knowledge. if librarians are to support this process in an evidence based manner, it is essential to understand the distinct practices of this user population. while recent reviews exist concerning the information behaviours of graduate students and researchers, there is little knowledge synthesis focused on the information literacies of phd students in specific disciplines. the aim of this article is to explore the depth and breadth of recent evidence which describes the information literacies of students pursuing a doctoral degree in the health sciences.   methods – strategic searches were performed in databases, hand-searched key journals, and reference lists. records were screened independently by both authors based on pre-determined criteria. general trends within the literature were mapped based on the extraction of the following data: geographic location, population, study aims, and method of investigation. further analysis of the articles included charting the academic disciplines represented, summarizing major findings related to phd students in health sciences, and which databases indexed the relevant articles.   results – many studies fail to treat doctoral studies as a unique process. phd students are often grouped together with other graduate students or researchers. studies tend to be based on small populations, and the number of phd students involved is either unclear or only equals a few individuals within the entire group of study. in addition, of the limited number of studies which focus exclusively on phd students, few conduct explicit examination of information practices in the health sciences. the result is that this user group is underrepresented within recent journal publications.   conclusion – this review highlights the need for more primary, in-depth research on the information literacies of phd students in the health sciences. in addition, librarians are encouraged to share their knowledge in scholarly publications which can reach beyond their own professional circles.     introduction   a practical objective of library and information science (lis) is to investigate the information practices of different groups in order to be able to invest in appropriate information resources and services. understanding the information literacies of phd students is of particular importance to academic libraries, since these students are often present or future faculty, and librarians can provide support in the transformation from students to scholars (fleming-may & yuro, 2009).   information literacy – debate and definitions   information literacy (il) has traditionally been defined by organizations in terms of explicit learning goals for the use of information. several models for il have been put forth over the past few decades, including the recent association of college and research libraries’ (acrl) “framework for information literacy for higher education”. this document represents an effort to shift from normative standards to a more nuanced definition of il as “the set of integrated abilities encompassing the reflective discovery of information, the understanding of how information is produced and valued, and the use of information in creating new knowledge and participating ethically in communities of learning” (american library association, 2015).   as a research field, il has been under development and debate for several decades (bruce, 2000; pinto, cordón, & díaz, 2010; tuominen, savolainen, & talja, 2005); however, it can be asserted that il is now a well-established concept within its own mature research domain (bruce, 2016). for the purposes of this review, the concept information literacies is used in the plural form to denote dynamic learning activities that take place through interactions within specific social contexts. in other words, information skills evolve in domain specific areas such as disciplines or communities of practice (lave & wenger, 1991; nicolini, gherardi, & yanow, 2003). this situated understanding of il “calls for empirical research efforts to analyze how specific communities use various conceptual, cultural, and technical tools to access printed and digital documents and to evaluate and create knowledge” (tuominen et al., 2005, p. 342).   information literacies of phd students   there is a wealth of research regarding information practices within educational settings, but few studies have concentrated on phd students as a discrete group. in a meta-synthesis of the literature on graduate students’ information-seeking behaviour, catalano (2013) only found 11 studies published between 1997 and 2012 that focus specifically on phd students. these studies typically center around efforts to improve library services, e.g., identifying information source preferences or investigating research and writing processes during a literature review. catalano’s (2013) review considered graduate students on both the master’s and doctoral level, and only a few patterns of behaviour were pointed out as unique to phd students. like master’s students, phd students were found to begin their research on the internet. however, phd students were also more inclined to consult their faculty advisors when seeking information.   spezi (2016) augmented catalano’s (2013) findings through a narrative review covering the years 2010 – 2015, focusing on whether there has been a change in phd students’ information seeking behaviours due to developments in information and communications technologies. only a handful of the identified research looked solely at phd students, instead most studies grouped phd students together with other graduate students or with other researchers. spezi’s (2016) review confirms catalano’s (2013) earlier observation that phd students are inclined to begin their searches on the internet and that this is now an established and recognized trend. at the same time, library e-resources are, after “a period of disenchantment”, still useful enough to compete with web searches. spezi (2016) also points to the previously documented importance of academic journals to phd students during the research process, and that more articles tend to be read in the medical and life sciences. phd students were also found to over-estimate their ability to search for information effectively, e.g., constructing effective search strategies.   disciplinary differences and the health sciences   in lis research, there has been a tendency to generalize about metadisciplines, i.e., group fields into broad discipline categories such as science or the humanities (case & given, 2016). against this backdrop, a common assumption is that scholars within the natural sciences mainly use journals and humanities scholars mainly use archives and books. these generalizations “may be true as they go, but they do not further our understanding of the important mechanisms of information seeking, nor are they particularly useful in application, as in designing university information systems to serve particular disciplines” (case & given, 2016, p. 288). as noted previously, there is a strand of lis research which asserts the importance of the disciplinary context. disciplines have different research cultures and traditions (talja, vakkari, fry, & wouters, 2007) and an “academic discipline ‘disciplines’ its members to behave in certain ways” (sundin, limberg, & lundh, 2008, p. 22). for this review, the health sciences as a concept is defined as narrower than a metadiscipline but wide enough as a field to encompass several smaller disciplines of science, which focus on health or health care, e.g., medicine or nursing.   aims   phd students are a unique library user group, marked by a period of transition. they are not merely graduate students; they are researchers in training. academic disciplines provide the social contexts through which phd students learn what it means to be information literate in their fields. although there have been recent reviews about the information-seeking behaviour of phd students, there is little knowledge synthesis about these students that is connected to the broader concept of information literacies or to the discipline-specific culture of the health sciences.   the aim of this article is therefore to explore the depth and breadth of research in scholarly articles concerning the information literacies of phd students within the health sciences.   methods   to “identify the nature and extent of research evidence” and to provide “a preliminary assessment of the potential size and scope of available research literature” (grant & booth, 2009, p. 101), a review of scholarly articles was conducted based on scoping review methodologies (arksey & o'malley, 2005; o’brien et al., 2016; peters et al., 2015).   this review involved structured searches of subject-specific as well as multidisciplinary databases for the years 2009 – 2018. this date range was chosen in order to locate the most current publications available on the topic of il. while the phrase information literacy was introduced as early as the 1970s (zurkowski, 1974), it can be argued that il has only recently been established as a research domain (bruce, 2000; bruce, 2016).   different databases were searched to identify the scope of the evidence, i.e., not only what research is available but also where. lisa (library & information science abstracts) was chosen to find research within lis and eric (education resource information center) for education. cinahl (cumulative index to nursing and allied health literature), medline, and psycinfo were used to broaden the search to encompass the health sciences. scopus was chosen in order to cover multidisciplinary publications. search strings were constructed using the individual databases’ thesauri in combination with variations of the keywords doctoral student and information literacy. searches were conducted in september 2018 and detailed documentation of the strategies, including which database platforms were used, is provided in the appendix.   through a series of test searches, the following four journals were recognized as particularly relevant for additional hand-searching: college & research libraries, eblip (evidence based library & information practice), journal of academic librarianship, and journal of information literacy.   the inclusion criteria of this review are reflected within its search strategies and screening criteria. it was limited to peer-reviewed journal articles that report on empirical evidence written in english. this narrow focus was used to identify publications that are commonly used and perhaps most valued by professionals supporting phd students within the health sciences, e.g., academic supervisors and medical or academic librarians. commentaries and essays were excluded, as were theses and dissertations, conference proceedings, book chapters, and policy papers, since these documents tend to be secondary sources rather than primary studies. review articles were also excluded, but only after consulting the reference lists of these articles to identify additional original studies.   the included research articles could employ qualitative, quantitative, or mixed methods. the information literacies of phd students could be examined from the perspective of the phd students themselves or from other groups involved in doctoral studies, such as thesis supervisors or librarians. however, the articles had to clearly identify and examine phd students within the health sciences as a distinct group.   both authors conducted the the literature search, screening, and data extraction.the search results were imported into endnote desktop for de-duplication according to a comprehensive and strategic method (bramer, giustini, de jong, holland, & bekhuis, 2016) and then independently screened using rayyan (ouzzani, hammady, fedorowicz, & elmagarmid, 2016). conflicting decisions were discussed in order to reach consensus. the following data was charted from the included studies: geographic location, population, aims, methodology, academic disciplines represented, major findings related to health sciences, and which databases indexed the relevant articles.   results   identification of relevant articles   an adapted version of the prisma flow diagram (moher, liberati, tetzlaff, altman, & group, 2009) summarizes the search and screening process (figure 1). the literature search identified 2,685 records and an additional 188 records were found through hand-searching. after duplicates were removed, the titles and abstracts of the 2,317 remaining records were screened independently by both authors.   full-text was retrieved for 129 articles. following full-text screening, only seven articles (0.3% of the initial data set of 2,317 records) met the inclusion criteria.   figure 1 modified prisma diagram.   indexing practices   table 1 shows which databases indexed the relevant articles. the journals represented were mainly within lis (six of the seven), apart from one journal within education. none of the databases indexed all seven articles, but all the articles were retrieved through scopus, a multidisciplinary database. lisa indexed all six lis journals, but not the education journal. in addition to the education journal, eric indexed two of the six lis journals. none of the articles were found in the databases covering nursing or medical research, i.e., cinahl and medline. however, psycinfo indexed the education journal and five of the six lis journals.   table 1 included articles – respective journals and database indexing article published in cinahl eric medline psycinfo scopus (carpenter, 2012) information services & use no no no yes yes (edwards & jones, 2014) evidence based library and information practice no no no yes yes (green, 2010) journal of academic librarianship no yes no yes yes (grigas, juzeniene, & velickaite, 2017) information research no yes no no yes (ramlogan, 2014) library philosophy and practice no no no yes yes (trafford & leshem, 2009) innovations in education and teaching international no yes no yes yes (warburton & macauley, 2014) australian academic and research libraries no no no yes yes     the nature of the evidence   this review revealed that the literature rarely treats doctoral studies as a unique process. phd students are usually grouped together with other graduate students or researchers (65 articles with an unclear population, 30 articles with phd students named but not the focus). studies tend to be based on small populations, and the number of phd students involved is either unclear or only equals a few individuals within the entire group of study. few studies were about phd students in the health sciences (27 with unclear or wrong discipline). in other words, we assert that phd students in the health sciences are underrepresented in recent scholarly journals.   only seven articles met the inclusion criteria. the following synthesis is based on the extracted data from these articles. table 2 provides an overview of the geographic locations as well as the populations, aims, and methodologies reported in the articles. the studies took place in predominately english-speaking countries such as the uk, the us, and australia. phd populations varied greatly in size from under 20 (green, 2010) to several thousand (carpenter, 2012), but the exact number of phd students representing health sciences was often unclear (carpenter, 2012; green, 2010; ramlogan, 2014; trafford & leshem, 2009). four studies were about the phd students themselves or their academic supervisors/librarians (carpenter, 2012; green, 2010; trafford & leshem, 2009; warburton & macauley, 2014), and three studies investigated written scholarly output, i.e., citation practices (edwards & jones, 2014; grigas et al., 2017; ramlogan, 2014).   table 2 overview of included articles article country population aim methodology (carpenter, 2012) uk 6,161 generation y (born 1982 1994) doctoral students and 7,432 older doctoral students; cohort of 30 full-time doctoral students identify the research behaviour among doctoral students in generation y mixed; three annual surveys and a longitudinal, qualitative cohort study (edwards & jones, 2014) us 107 doctoral dissertations compare how well library collections support doctoral research quantitative; citation analysis (green, 2010) us and australia 42 participants including 5 american and 6 australian librarians, 8 american and 10 australian doctoral candidates, and 6 american and 7 doctoral advisors examine and reconsider the assumption that doctoral students are information illiterate qualitative; interviews coded through grounded theory (grigas et al., 2017) lithuania 39 doctoral theses evaluate how useful freely available full-text information sources can be when writing phd theses; determine to what extent the library may be an information resource provider and intermediator quantitative; citation analysis (ramlogan, 2014) jamaica 696 theses/dissertation checks examine the service of thesis and dissertation checking provided by liaison librarians quantitative; statistical analysis (trafford & leshem, 2009) uk 55 phds, 7 supervisors, texts from examiners identify the difficulties that doctoral candidates encounter qualitative; open-ended questionnaire, discussions, and text analysis, all coded into vignettes (warburton & macauley, 2014) australia 79 phd candidates and 32 phd supervisors profile phd candidate usage of research consultation service; explore if consultations make a difference in the early stages of the phd candidature mixed; open-ended questionnaire, survey, both online   both quantitative and qualitative methods were used to achieve the aims of the studies. citation analysis was employed within two of the articles (edwards & jones, 2014; grigas et al., 2017) to evaluate various aspects of library services for phd students, e.g., relevance of library collections and the usefulness of freely available full-text information. numerical evidence concerning the provision of thesis/dissertation checking was presented in one study (ramlogan, 2014). interviews and grounded theory were used to challenge the assumption that phd students are information illiterate in one study (green, 2010). several qualitative methods, including coding into vignettes, were used within one study to identify the difficulties that phd students encounter (trafford & leshem, 2009). mixed methods, i.e., a combination of surveys and interviews, were used within two of the articles (carpenter, 2012; warburton & macauley, 2014). the aims of these latter studies included comparing research behaviour based on generational differences and determining the impact of early research consultation services during candidature.   the breadth and depth of the evidence   table 3 charts the discipline-specific data collected from the articles. while all the articles focused on phd students, it was often difficult to locate data specific to the health sciences. for two of the articles (ramlogan, 2014; trafford & leshem, 2009), no data could be identified relating to health science phd students in particular. for one article (green, 2010), the data fit under generalizations made for the entire population of study regardless of discipline. for the remaining four articles (carpenter, 2012; edwards & jones, 2014; grigas et al., 2017; warburton & macauley, 2014), the reporting was clearer concerning which findings pertained to phd students in the health sciences, but the amount of data was limited. the health science most commonly named was medicine (carpenter, 2012; grigas et al., 2017; ramlogan, 2014; trafford & leshem, 2009; warburton & macauley, 2014), and all the studies examined other disciplines outside the health sciences at the same time, e.g., education or engineering.   table 3 health science (hs) discipline-specific data from included articles article hs population hs discipline(s) other discipline(s) usability of findings major hs findings (carpenter, 2012) number of respondents from hs disciplines is unclear medicine, dentistry & health, veterinary sciences social sciences, engineering & computer sciences, arts & humanities, biomedical sciences, physical sciences, biological sciences 2012 report based on studies performed in 2007 and 2009; limited amount of data that could be identified as specific to hs discipline e-journals dominate as a research resource for hs students; cohort students strongly indicate that difficulty accessing and obtaining relevant resources due to licensing is a severe constraint on their research; citation databases and e-journal search interfaces are equally as popular as google; with the exception of veterinary sciences, phd students work alone and not in collaborating research teams (edwards & jones, 2014) out of 107 dissertations, 28 (26%) within psychology and 22 (21%) within social welfare psychology, social welfare education clear disciplinespecific reporting yet limited amount of data that could be charted psychology students cited the highest percentage of journals; social welfare students cited free web resources (primarily government documents or reports from ngos and advocacy groups) but psychology students did not; both disciplines cited older material than anticipated; surprisingly cross-disciplinary nature of research, e.g., social welfare students frequently cited journals in psychology (green, 2010) number of respondents from hs discipline is unclear nursing education, physical & biological science limited amount of data that could be identified as specific to hs discipline phd students from all disciplines indicated that they used the strategy of backward and forward citation tracking to evaluate the quality of sources and expand their bibliographies; most phd students developed their literacy skills without direct instruction; librarians are predisposed toward the view that phd students are information illiterate (grigas et al., 2017) out of 39 theses, 2 (5%) within psychology and 5 (13%) within medicine psychology, medicine humanities, social sciences, biomedical sciences, technological sciences, physical sciences clear discipline-specific reporting yet limited amount of data that could be charted phd students from the biomedical sciences are substantial users of peer-reviewed e-journals; biomedical sciences students use books and e-books less than students within the humanities (ramlogan, 2014) unclear reporting for 176 theses/dissertation checks; out of 520 theses/dissertation checks, 47 (9%) within medical sciences yet unclear if on master’s or phd level medical sciences science & agriculture, humanities & education, engineering, social sciences focus is on prevalence of the service rather than the impact it has on phd students; no data that could be charted as specific to hs discipline not applicable (trafford & leshem, 2009) number of respondents/documents from hs discipline is unclear bio-medical sciences botany, management and business, education, english, geography, history, law, linguistics, surveying no data that could be identified as specific to hs discipline not applicable (warburton & macauley, 2014) 43.4% of phd candidates and 40.6% of phd supervisors within medicine, dentistry and health sciences (mdhs) mdhs arts, education, engineering, architecture, building & planning, veterinary science, business & economics clear discipline-specific reporting yet limited amount of data that could be charted more than half of part-time mdhs candidates rated their information skills as "less than adequate"; 78.8% of mdhs students spoke of information "chaos", "floundering" and "random" approaches to locating information; the main reasons mdhs students sought library research assistance were for help with search terms and keywords, and for literature searching strategy design; 100% of mdhs students thought library consultations could assist in refining literature search strategies and 88% thought consultations could assist in undertaking thorough or systematic literature searching   summarising the evidence   overall, the studies identified in this review provide a regrettably limited amount of data about the information literacies of phd students in the health sciences. it has already been noted that few studies in recent lis literature are devoted solely to phd students, and the results of this review confirm this knowledge gap. even fewer studies were found addressing information literacies specific to health science disciplines, and the few studies identified were mainly found in lis journals.   discussion   comparing the evidence   the small amount of relevant data available for analysis (tables 2 and 3) corresponds well with the previous discoveries of catalano (2013) and spezi (2016).   as established by catalano (2013), several studies were centered around efforts to improve library services (edwards & jones, 2014; grigas et al., 2017; ramlogan, 2014; warburton & macauley, 2014). in keeping with a common assumption in lis research (case & given, 2016, p. 288), spezi (2016) confirmed the importance of academic journals to phd students and that more articles tend to be read in the medical sciences. the same evidence is found in the article reporting on the largest population (carpenter, 2012) as well as the two smaller studies based on citation analysis (edwards & jones, 2014; grigas et al., 2017). both catalano (2013) and spezi (2016) observed that phd students are inclined to begin searching on the internet; however, spezi also argued that library e-resources are still able to compete with web searches. this varied approach was also reported in the carpenter article (2012).   a few new findings were discerned from the limited data of the included articles. while spezi (2016) described how phd students over-estimate their ability to search for information effectively, the students in the study by warburton and macauley (2014) often rated their skills as “less than adequate” and spoke of information “chaos”, “floundering”, and “random” approaches to locating information. green (2010) asserts that librarians are predisposed toward the view of phd students as information illiterate and calls for the profession to question this assumption; in part this is because the students in green’s study were found to develop their literacy skills without direct instruction.   additional findings moved beyond information-seeking and discovery into the realm of “how information is produced and valued” (american library association, 2015). while edwards and jones (2014) found students cited older material than anticipated, green (2010) reported that students strategically tracked citations backward and forward in order to evaluate the quality of sources and expand their bibliographies. regarding “the use of information in creating new knowledge and participating ethically in communities of learning” (american library association, 2015), warburton and macauley (2014) discovered that students mainly sought library research assistance for their information-seeking, i.e., search terms, keywords, and strategy design. it should be noted that their respondents were very confident in library research support, e.g., 100% thought that library consultations could refine search strategies and 88% thought they could get help with thorough or systematic searches. with regards to communities of learning, carpenter (2012) reported that phd students in medicine, dentistry, and health generally worked alone and not in collaborating research teams.   charting the evidence base   as indicated in table 1, the few relevant articles identified in this review were mainly found in lis journals. however, if librarians wish to inform faculty about il and how librarians can help, it is the disciplinary publications which faculty value that can serve as the most effective medium (stevens, 2007). in the health sciences, these publications are usually scholarly articles found in databases such as pubmed.   within the lis community, there is a call for evidence based library and information practice (booth, 2002; crumley & koufogiannakis, 2002) and a concern that there is not enough research from which to draw conclusions. as a former editor of the journal of the medical library association remarked, “we have many articles; we do not have a body of evidence” (plutchak, 2005). in an overview identifying research gaps, koufogiannaikis and crumley (2006) also noted several issues that librarians face when publishing articles, including a lack of indexing and open access options in lis journals.   where is the evidence about information literacy to be found and who is publishing this research? in a small-scale reference analysis of articles on how academic libraries contribute to student success, findings suggest an uneven relationship between lis and other disciplines. more specifically, lis is borrowing concepts and methods from the field of education, but other disciplines rarely cite lis research (kogut, 2019). another exploratory study investigating the visibility of librarians as authors in scholarly journals within higher education, teaching, and learning between 2000 and 2012, found that less than 2% of articles published in these journals were written by librarians; while il was the most common topic for librarians, most articles were theoretical and not based on empirical research (folk, 2014). pilerot (2014) notes in another small-scale investigation how the established assumption is that there is a disconnect between research and practice, and that the prevailing gap-metaphor should be abandoned to allow for a more nuanced discussion between librarians as a professional group and lis faculty. is there a gap in the evidence base concerning the il of phd students in the health sciences? this review points to the possibility, but perhaps there is also too little communication between library practice, library research, and those who benefit from both.   limitations   this review is not without its limitations. very few studies met the narrow inclusion criteria. generally, the populations of the studies were small and researchers rarely ascribed their findings to discipline-specific practices, resulting in findings that are almost anecdotal in nature, making it difficult to track larger trends. il was mapped as an established concept, but more studies might have been located if the search strategies had included classic lis terminology such as information-seeking or literacy skills, or if the date range had been extended to include research from earlier decades. this review may have also missed articles where il was not named, but rather described as a particular strategy such as help from librarians or using journal articles. additional studies might have also been found if phd students had not been treated as unique user group, i.e., labels like graduate students or researchers were used. moreover, the inclusion of professional doctorates such as md or dpharm might have also led to a broader review.   a great deal of investigative work devoted to this population is probably being carried out by lis professionals, and not just by lis researchers. if more health science librarians were to disseminate the results of their own research, a solid evidence base could be established within our profession (koufogiannakis & crumley, 2006). more knowledge about how phd students interact with libraries is likely to be found in librarians’ grey literature, such as conference posters and institutional reports. therefore, future attempts to map this user population should also include searches of the grey literature. in addition, if enough original studies are found devoted to this population, these should be subjected to some form of critical analysis before data extraction, to increase the trustworthiness of any resulting synthesis.   conclusions   this review found that phd students in the health sciences are underrepresented in current scholarly journals. out of over 2,500 possible records, only seven articles met the inclusion criteria. from these seven, six were found in lis journals, resulting in a lack of evidence about how to support the information literacies of this population. future lis research should address this deficiency by studying phd students as a unique group operating within discipline-specific communities. furthermore, it is recommended that more health science librarians share their professional experiences in publications that reach beyond their own institutions or organizations, e.g., peer-reviewed articles in journals which are indexed in databases such as cinahl or medline.   references   american library association. (2015). framework for information literacy for higher education. retrieved from http://www.ala.org/acrl/standards/ilframework   arksey, h., & o'malley, l. (2005). scoping studies: towards a methodological framework. international journal of social research methodology, 8(1), 19-32. https://doi.org/10.1080/1364557032000119616   booth, a. (2002). evidence-based librarianship: one small step. health information and libraries journal, 19(2), 116-119. https://doi.org/10.1046/j.1471-1842.2002.00373.x   bramer, w.m., giustini, d., de jong, g.b., holland, l., & bekhuis, t. (2016). de-duplication of database search results for systematic reviews in endnote. journal of the medical library association, 104(3), 240-243. https://doi.org/10.3163%2f1536-5050.104.3.014   bruce, c. (2000). information literacy research: dimensions of the emerging collective consciousness. australian academic and research libraries, 31(2), 91-109. https://doi.org/10.1080/00048623.2000.10755119   bruce, c.s. (2016). information literacy research: dimensions of the emerging collective consciousness. a reflection. australian academic and research libraries, 47(4), 239-244. https://doi.org/10.1080/00048623.2016.1248236   carpenter, j. (2012). researchers of tomorrow: the research behaviour of generation y doctoral students. information services & use, 32(1-2), 3-17. https://doi.org/10.3233/isu-2012-0637   case, d.o., & given, l.m. (2016). looking for information: a survey of research on information seeking, needs, and behavior. bingley: emerald.   catalano, a. (2013). patterns of graduate students' information seeking behavior: a meta-synthesis of the literature. journal of documentation, 69(2), 243-274. https://doi.org/10.1108/00220411311300066   crumley, e., & koufogiannakis, d. (2002). developing evidence-based librarianship: practical steps for implementation. health information and libraries journal, 19(2), 61-70. https://doi.org/10.1046/j.1471-1842.2002.00372.x   edwards, s., & jones, l. (2014). assessing the fitness of an academic library for doctoral research. evidence based library & information practice, 9(2), 4-15. https://doi.org/10.18438/b81k5t   fleming-may, r., & yuro, l. (2009). from student to scholar: the academic library and social sciences phd students' transformation. portal: libraries and the academy, 9(2), 199-221. https://doi.org/10.1353/pla.0.0040   folk, a.l. (2014). librarians as authors in higher education and teaching and learning journals in the twenty-first century: an exploratory study. the journal of academic librarianship, 40(1), 76-83. https://doi.org/10.1016/j.acalib.2013.12.001   grant, m.j., & booth, a. (2009). a typology of reviews: an analysis of 14 review types and associated methodologies. health information and libraries journal, 26(2), 91-108. https://doi.org/10.1111/j.1471-1842.2009.00848.x   green, r. (2010). information illiteracy: examining our assumptions. the journal of academic librarianship, 36(4), 313-319. https://doi.org/10.1016/j.acalib.2010.05.005   grigas, v., juzeniene, s., & velickaite, j. (2017). “just google it” – the scope of freely available information sources for doctoral thesis writing. information research, 22(1). retrieved from http://informationr.net/ir/22-1/paper738.html   kogut, a. (2019). information transfer in articles about libraries and student success. the journal of academic librarianship, 45(2), 137-145. https://doi.org/10.1016/j.acalib.2019.02.002   koufogiannakis, d., & crumley, e. (2006). research in librarianship: issues to consider. library hi tech, 24(3), 324-340. https://doi.org/10.1108/07378830610692109   lave, j., & wenger, e. (1991). situated learning: legitimate peripheral participation. cambridge: cambridge university press.   moher, d., liberati, a., tetzlaff, j., altman, d.g., & the prisma group. (2009). preferred reporting items for systematic reviews and meta-analyses: the prisma statement. annals of internal medicine, 151(4), 264-269. https://doi.org/10.7326/0003-4819-151-4-200908180-00135   nicolini, d., gherardi, s., & yanow, d. (2003). introduction: toward a practice-based view of knowing and learning in organizations. in d. nicolini, s. gherardi, & d. yanow (eds.), knowing in organizations: a practice-based approach (pp. 3-31). armonk, newyork: m.e. sharpe.   o’brien, k.k., colquhoun, h., levac, d., baxter, l., tricco, a.c., straus, s., wickerson, l., nayar, a., moher, d., & o’malley, l. (2016). advancing scoping study methodology: a web-based survey and consultation of perceptions on terminology, definition and methodological steps. bmc health services research, 16, 305. https://doi.org/10.1186/s12913-016-1579-z   ouzzani, m., hammady, h., fedorowicz, z., & elmagarmid, a. (2016). rayyan-a web and mobile app for systematic reviews. systematic reviews, 5(1), 210. https://doi.org/10.1186/s13643-016-0384-4   peters, m.d.j., godfrey, c.m., khalil, h., mcinerney, p., parker, d., & soares, c.b. (2015). guidance for conducting systematic scoping reviews. international journal of evidence-based healthcare, 13(3), 141-146. https://doi.org/10.1097/xeb.0000000000000050   pilerot, o. (2014). connections between research and practice in the information literacy narrative: a mapping of the literature and some propositions. journal of librarianship and information science, 48(4), 313-321. https://doi.org/10.1177/0961000614559140   pinto, m., cordon, j.a., & diaz, r.g. (2010). thirty years of information literacy (1977-2007): a terminological, conceptual and statistical analysis. journal of librarianship and information science, 42(1), 3-19. https://doi.org/10.1177%2f0961000609345091   plutchak, t.s. (2005). building a body of evidence. journal of the medical library association, 93(2), 193-195. retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/pmc1082934/   ramlogan, r. (2014). theses and dissertations at the university of the west indies, st. augustine: a liaison librarian's input. library philosophy and practice, 1-12. retrieved from https://digitalcommons.unl.edu/libphilprac/1150/   spezi, v. (2016). is information-seeking behavior of doctoral students changing?: a review of the literature (2010–2015). new review of academic librarianship, 22(1), 78-106. https://doi.org/10.1080/13614533.2015.1127831   stevens, c.r. (2007). beyond preaching to the choir: information literacy, faculty outreach, and disciplinary journals. the journal of academic librarianship, 33(2), 254-267. https://doi.org/10.1016/j.acalib.2006.08.009   sundin, o., limberg, l., & lundh, a. (2008). constructing librarians' information literacy expertise in the domain of nursing. journal of librarianship and information science, 40(1), 21-30. https://doi.org/10.1177%2f0961000607086618   talja, s., vakkari, p., fry, j., & wouters, p. (2007). impact of research cultures on the use of digital library resources. journal of the american society for information science and technology, 58(11), 1674-1685. https://doi.org/10.1002/asi.20650   trafford, v., & leshem, s. (2009). doctorateness as a threshold concept. innovations in education and teaching international, 46(3), 305-316. https://doi.org/10.1080/14703290903069027   tuominen, k., savolainen, r., & talja, s. (2005). information literacy as a sociotechnical practice. library quarterly, 75(3), 329-345. https://doi.org/10.1086/497311   warburton, j., & macauley, p. (2014). wrangling the literature: quietly contributing to hdr completions. australian academic and research libraries, 45(3), 159-175. https://doi.org/10.1080/00048623.2014.928992   zurkowski, p.g. (1974). the information service environment: relationships and priorities. related paper no. 5. washington, d.c.: national commission on libraries and information science.   appendix database searches   database search string limiters cinahl (ebscohost) (mh "students, graduate" or ti (doctoral or doctorate or post-graduate or postgraduate or graduate or phd or “doctor of philosophy”) or ab (doctoral or doctorate or post-graduate or postgraduate or graduate or phd or “doctor of philosophy”)) and (mh "information literacy" or ti (information and (literacy or literacies)) or ab (information and (literacy or literacies))) peer reviewed; published date: 20090101-20180831; english language eric (ebscohost) (de "doctoral programs" or ti (doctoral or doctorate or post-graduate or postgraduate or graduate or phd or “doctor of philosophy”) or ab (doctoral or doctorate or post-graduate or postgraduate or graduate or phd or “doctor of philosophy”)) and (de "information literacy" or ti (information and (literacy or literacies)) or ab (information and (literacy or literacies))) peer reviewed; published date: 20090101-20180831; english language lisa (proquest) (mainsubject.exact("graduate studies") or ti(doctoral or doctorate or post-graduate or postgraduate or graduate or phd or "doctor of philosophy") or ab(doctoral or doctorate or post-graduate or postgraduate or graduate or phd or "doctor of philosophy")) and (mainsubject.exact("information literacy") or ti(information and (literacy or literacies)) or ab(information and (literacy or literacies))) peer reviewed; date: from 2009 january 01 to 2018 august 31; english language medline (ebscohost) (mh "education,graduate" or ti (doctoral or doctorate or post-graduate or postgraduate or graduate or phd or “doctor of philosophy”) or ab (doctoral or doctorate or post-graduate or postgraduate or graduate or phd or “doctor of philosophy”)) and (mh "information literacy" or ti (information and (literacy or literacies)) or ab (information and (literacy or literacies))) published date: 20090101-20180831; english language psycinfo (proquest) (mainsubject.exact("postgraduate students") or ti(doctoral or doctorate or post-graduate or postgraduate or graduate or phd or "doctor of philosophy") or ab(doctoral or doctorate or post-graduate or postgraduate or graduate or phd or "doctor of philosophy")) and (mainsubject.exact("information literacy") or ti(information and (literacy or literacies)) or ab(information and (literacy or literacies))) peer reviewed; date: from 2009 january 01 to 2018 august 31; english language scopus (elsevier) title-abs-key (information and (literacy or literacies)) and (doctoral or doctorate or post-graduate or postgraduate or graduate or phd or "doctor of philosophy") and doctype(ar) and pubyear > 2009 and language(english) all searches were performed on september 21, 2018.   evidence based library and information practice 2011, 6.4 127 evidence based library and information practice evidence summary adaptation of impact questions from an existing toolkit provided clear assessment of valued service elements and desirable service improvements in a primary health care library and information service a review of: urquhart, c., thomas, r., ovens, j., lucking, w., & villa, j. (2010). planning changes to health library services on the basis of impact assessment. health information and libraries journal, 27(4), 277-285. doi: 10.1111/j.1471-1842.2010.00900.x reviewed by: kate kelly librarian royal college of surgeons in ireland dublin 2, ireland email: katekelly@rcsi.ie received: 01 june 2011 accepted: 05 oct. 2011 2011 kelly. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to provide an action plan for the knowledge, resource and information service (kris) based on an impact assessment of current services, satisfaction with current services, and views on desirable improvements to service and service delivery. design – questionnaire for kris service users and interviews with kris staff. setting – two locations served by kris in the north and south of bristol city in the uk – one a health promotion service and one a national health service (nhs) teaching hospital. subjects – a convenience sample of a total of 244 users of the library services at the two locations, 121 users at the health promotion service site and 123 users at the hospital site. methods – a questionnaire designed for a previous nhs library service impact study was adapted for use with staff other than health mailto:katekelly@rcsi.ie evidence based library and information practice 2011, 6.4 128 workers, since teachers and youth workers, for example, also used the health promotion service. the researchers circulated the questionnaire by mail and email to prospective respondents. the questionnaire asked participants to reflect on the most recent time they had used kris services and provide details on the purpose of use, what elements of the service they used, satisfaction with the service or the information provided, the immediate impact on their work, and its probable contribution to future work. it also asked about desirable improvements and how kris contributed to the respondents’ work and continuing professional development. the researchers interviewed kris staff face to face and asked for their views on the history of the service and future developments. main results – the overall response rate was 62.3% (152/244), with similar responses from each site. community nurses and midwives were the largest group of respondents (n=31, 20.4%), followed by managers and administrators (n=24, 15.8%). both sites reported health promotion activities as the dominant reason for use. health promotion leaflets (n=94, 61.8%) and resources for loan (n=57, 37.5%) were the top two service elements used, with literature searching third (n=46, 30.3%). the service completely met the needs of 93% (n=141) of respondents, with 97.4% (n=148) reporting being wholly satisfied that information was delivered in time. in terms of the immediate cognitive impact of the information obtained, 93% (n=141) reported obtaining relevant information, 76% (n=115) current information, and 72% (n=109) accurate information. nearly 70% were distributing information or resources to other people, and 68% were sharing or discussing information provided with colleagues. nobody claimed that they had found little or nothing of value. advice to patients, clients or carers was the most frequently cited impact category (n=104, 68%), followed by advice to colleagues (n=84, 55.3%). the main strengths of the service were the perceived reliability of the service, the current awareness and alerting service run from one site, and literature searching support for commissioning and public health enquiries. respondents reported saving time, and the main cognitive impact was the perceived relevance of the information obtained. service weaknesses identified related to accessibility issues such as parking, opening hours, and staff coverage, and outreach was identified as the main service development opportunity. conclusion – the adaptation of an existing questionnaire provided a clear assessment of the effectiveness of a primary health care information service to a broad spectrum of users. respondents identified what they valued about the service and how it could be improved. commentary while at first glance this research may seem specific to the commissioning agency, kris, it is relevant to all those with an interest in measuring and defining the impact and effectiveness of library and information services, particularly in clinical settings. it is well written and links the research to the literature, including the u.k. impact tool kit referenced within the article (weightman, urquhart, spink, & thomas, 2009) and the information assessment method (iam) developed at mcgill university (information technology primary care research group, 2009). it clearly identifies implications for practice and implications for policy. as a result of this research, kris took actions to change opening hours and to increase the marketing and promotion of the service to potential users such as schools and colleges. it evidence based library and information practice 2011, 6.4 129 also planned to introduce an interactive website with online ordering of materials. this work validates the adaptation of the original tool, and the actions subsequently taken by kris on the basis of the research should signal to readers that this is a tool that can be used. the inclusion of the adapted questionnaire as a supplement to the online article makes it very easy to do this, and reassuringly for practitioners, it is a simple and relatively short questionnaire with nine out of eleven questions being answered using check boxes. the original guidance document is also available as an online appendix and provides detailed instructions on implementing an impact study using the toolkit. the authors provide suggestions on how it could be adapted, and also discuss adapting impact frameworks for other library settings. the authors identify minor limitations of the study. for example, a randomized sample may have yielded more robust evidence of the service impacts compared to the convenience sample used. overall this is a very pragmatic and reproducible piece of research – a real example of using evidence to plan and improve services. references information technology primary care research group. (2009). the information assessment method. department of family medicine, mcgill university. retrieved 30 may 2011 from http://iam2009.pbworks.com/frontpage. (archived by webcite® at http://www.webcitation.org/ 5gsxvzicm) weightman, a., urquhart, c., spink, s., & thomas, r. (2009). the value and impact of information provided through library services for patient care: developing guidance for best practice. health information and libraries journal, 26(1), 63-71. evidence based library and information practice 2012, 7.4 35 evidence based library and information practice article teaching literacy: methods for studying and improving library instruction meggan houlihan coordinator of instruction/reference american university in cairo library new cairo, egypt email: mhoulihan@aucegypt.edu amanda click phd student and elime fellow school of information and library science university of north carolina at chapel hill chapel hill, north carolina, united states of america email: aclick@live.unc.edu received: 22 feb. 2012 accepted: 9 oct. 2012 2012 houlihan and click. this is an open access article distributed under the terms of the creative commons‐ attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – the aim of this paper is to evaluate teaching effectiveness in one-shot information literacy (il) instruction sessions. the authors used multiple methods, including plus/delta forms, peer evaluations, and instructor feedback surveys, in an effort to improve student learning, individual teaching skill, and the overall il program at the american university in cairo. methods – researchers implemented three main evaluation tools to gather data in this study. librarians collected both quantitative and qualitative data using student plus/delta surveys, peer evaluation, and faculty feedback in order to draw overall conclusions about the effectiveness of one-shot il sessions. by designing a multi-method study, and gathering information from students, faculty, and instruction librarians, results represented the perspectives of multiple stakeholders. mailto:mhoulihan@aucegypt.edu mailto:aclick@live.unc.edu evidence based library and information practice 2012, 7.4 36 results – the data collected using the three evaluation tools provided insight into the needs and perspectives of three stakeholder groups. individual instructors benefit from the opportunity to improve teaching through informed reflection, and are eager for feedback. faculty members want their students to have more hands-on experience, but are pleased overall with instruction. students need less lecturing and more authentic learning opportunities to engage with new knowledge. conclusion – including evaluation techniques in overall information literacy assessment plans is valuable, as instruction librarians gain opportunities for self-reflection and improvement, and administrators gather information about teaching skill levels. the authors gathered useful data that informed administrative decision making related to the il program at the american university in cairo. the findings discussed in this paper, both practical and theoretical, can help other college and university librarians think critically about their own il programs, and influence how library instruction sessions might be evaluated and improved. introduction assessment is one of the most popular topics in academic libraries today. much research has been conducted, and many papers written, on this topic, and they are generally valuable additions to the body of library literature. this article, however, is not about assessment. it is about evaluation. although these terms are sometimes used interchangeably, they are not the same and do not entail the same processes. assessment requires that the skills or knowledge that students are expected to develop during a class or library session are stated explicitly prior to instruction. the ability of students to demonstrate these skills or knowledge is then measured following the instruction session to assess the effectiveness of the instructor or other teaching tool (association of college and research libraries, 2011). evaluation, however, involves “rating the performance of services, programs, or individual instructors,” in order to identify strengths, weaknesses, and areas for improvement (rabine & cardwell, 2000, p. 320). the focus of this study was to gather information from multiple stakeholders about the effectiveness of teachers: this is evaluation. assessment and evaluation, while not identical or even interchangeable, can be closely related. for example, the results of an evaluation project may provide insight into the areas of instruction that need the most improvement, thus informing the design of an assessment study. at the american university in cairo (auc) main library, library instruction falls under the responsibility of the department of research and information services. the information literacy program is made up of a required semester-long il course (lalt 101) intended to be taken by freshmen, as well as individual “one-shot” instruction sessions tailored to specific classes. the demand for these one-shot sessions increased noticeably for the 2010-2011 academic year – from 43 in 2009-2010 to 101 for fall 2010 and spring 2011. the majority of these sessions are taught by a core group of eight librarians who serve as departmental liaisons and provide reference and instruction services, and about half of these sessions are for classes within the freshman writing program. every session is designed to address predetermined student learning outcomes that are established through collaboration with the professors. little had been done to evaluate these one-shot sessions in recent years. in fall of 2010, researchers began development of an evaluation plan to examine one-shots from multiple perspectives and to improve information literacy training to auc students. the project evidence based library and information practice 2012, 7.4 37 included multiple methods – plus/delta forms, faculty feedback, and peer observation – in order to collect data from students, faculty, and librarians, and was scheduled to take place in the beginning of the spring semester. literature review a review of the literature indicates that evaluating and assessing library instruction have become a priority for many libraries (matthews, 2007; oakleaf, 2009; shonrock, 1996; zald & gilchrist, 2008). the process may seem intimidating for librarians who have never undertaken such a project, but the literature included in this review indicates that many would not be dissuaded. teaching skills the focus of our study differs from much of the literature in that we focused primarily on the evaluation of the teaching skills of librarian instructors. walter (2006) argues that “teacher training is still a relatively minor part of the professional education for librarians even as it becomes an increasingly important part of their daily work,” and so “instructional improvement” (p. 216) should be pursued by all instruction librarians, novice or experienced. it has also been shown that librarians, particularly those with less than five years of experience, are not confident in maintaining student interest, classroom management, and public speaking (click & walker, 2010). the evaluation of instruction can provide feedback that allows teaching librarians to develop in these areas. instruction librarians can use a variety of techniques to improve teaching, such as reflection (belanger, bliquez, & mondal, 2012), peer observation (samson & mccrae, 2008), or small group analysis (zanin-yost & crow, 2012). assessment and evaluation zanin-yost and crow (2012) describe assessment as a “multistep process that includes collecting and interpreting information that will assist the instructor in making decisions about what methods of course delivery to use, when to teach course content, and how to manage the class” (p. 208). others define assessment simply as the measuring of outcomes, while evaluation denotes “an overall process of reviewing inputs, curriculum and instruction” (judd, tims, farrow, & periatt, 2004, p. 274). the idea that assessment and evaluation are not synonyms is rarely discussed in the library literature. popular methods the use of preand post-tests in order to assess il skill development appears regularly in the literature on assessing the effectiveness of library instruction. hsieh and holden (2010) employed preand post-testing as well as student surveys in an effort to discover what students actually learned from one-shot sessions, and whether or not these sessions were effective. they found that “it is just as incorrect to say that single-session information literacy instruction is useless as it is to believe that it is all that is needed to achieve a high level of il among college students” (p. 468). furno and flanagan (2008) developed a questionnaire that was given to students before and after il instruction, designed to test students on three topics: formulating research strategies, evaluating resources, and resource recognition. their research illustrated that there were several areas to improve upon, specifically teaching students to use the boolean “or,” but most importantly it showed them that creating a culture of assessment in the library would lead to improved il instruction sessions. research like this has a clear practical purpose, since it helps discover areas in which il sessions might be improved. furno and flanagan’s research was of particular interest to us because it was conducted at an american-style overseas university, the american university of sharjah, a setting which is similar to auc. wong, chan, and chu (2006) provided an additional international perspective from the hong kong university of science and technology, utilizing a delayed survey to collect student impressions evidence based library and information practice 2012, 7.4 38 of il instruction four to eight weeks after the session. although the survey did not test for knowledge specifically, results encouraged librarians to make changes to session length and handout content (wong, chan, & chu, 2006). like furno and flanagan, wong, chan, and woo analyzed data to make improvements to individual instruction sessions, but they also used this assessment technique to create an assessment program at their university. using multiple methods rabine and cardwell’s (2000) multi-method assessment helped with the development of this project, as they used student and faculty feedback, peer evaluation, and self-assessment. their study allowed them to gather a great deal of data from all stakeholders, so that they might “attempt to reach common understandings and establish ‘best practices’” (p. 328) for one-shot sessions. bowles-terry (2012) chose a mixedmethods approach to collect both quantitative and qualitative data, which provided “a more complete picture” (p. 86). the results of her study offered more than just statistical correlations: she was able to form a strong argument in support of developing a tiered il program. although the use of multiple or mixed methods in assessment has not been common in the library literature, the technique is gaining popularity in the field. plus/delta, faculty feedback, and peer evaluation there is little to be found in the library literature about using the plus/delta chart, which is simply a piece of paper on which students write a plus side for one thing that they have learned in the class session, and on the other side a delta sign for one thing about which they are still confused. mcclanahan and mcclanahan (2010) described this concept as a simple way to obtain feedback about what is and is not working in the classroom. collaboration between librarians and teaching faculty is regularly encouraged in the literature (arp, woodard, lindstrom, & shonrock, 2006; belanger, bliquez, & mondal, 2012; black, crest, & volland, 2001). rabine and cartwell (2000) solicited faculty feedback on specific one-shot sessions in order to make improvements to teaching methods and content. black, crest, and volland (2001) surveyed over 100 faculty members who had utilized library instruction in the past and were able to identify where programmatic changes should be made. gathering feedback from these crucial stakeholders supports the assessment of il by “putting these various perspectives in conversation with each other” and fostering “a dialogue between faculty and librarians about shared instructional aims” (belanger, bliquez, & mondal, 2012, p. 70). samson and mccrea (2008) provided background for using peer evaluation in il instruction, a topic that is not often addressed in the library literature. they note that the experience benefits all instructors: “new teaching faculty garnered ideas and pedagogy from their more experienced colleagues, but experienced librarians were also inspired by the fresh perspectives and insights of newer teachers” (p. 66). middleton’s (2002) analysis of the peer evaluation program at oregon state university provided a framework for setting up a system of evaluation instead of just creating a snapshot of teaching effectiveness. she noted that “the most significant benefit to the reference department and the library administration was the establishment of a peer evaluation of instruction process, incorporating both summative and formative evaluation depending upon the type of review selected and/or needed” (p. 75). methods in spring of 2011, with the assistance and enthusiasm of the research and information services department at auc, researchers prepared to evaluate teaching effectiveness of evidence based library and information practice 2012, 7.4 39 one-shot il instruction sessions by conducting an institutional review board–approved study using three assessment methods. we designed a plus/delta form to measure student input, a peer evaluation form to measure input from instruction librarians, and an online survey to collect faculty feedback. the goal of these three evaluation instruments is to examine instruction and delivery from the perspective of three different stakeholders. by including and collecting data from all stakeholders, the authors were able to identify individual and overarching trends in assessment data and thus form stronger conclusions. plus/delta instructor librarians distributed plus/delta forms to all students at the end of all il one-shot instruction sessions during a one-and-a-halfmonth survey period; 232 students chose to participate. they were asked to identify one concept they learned (the plus) and one concept about which they were still confused at the conclusion of the session (the delta). the plus symbol represents strengths; instructors use this positive feedback to identify areas of instruction and delivery in which they excel. the delta symbol represents change; instructors use this feedback to make adjustments and improvements to their teaching. the authors compiled the plus/delta forms and then transcribed to allow for better organization and analysis. all comments were grouped by theme – such as specific skills, resources, services, and general comments – to analyze which concepts were being taught well. this quick and simple information gathering tool allowed students to provide anonymous commentary, and librarians were able to use this immediate feedback to identify the strengths and weakness in their presentation. examples of completed plus/delta forms can be found below. figure 1 examples of completed plus/delta forms evidence based library and information practice 2012, 7.4 40 peer evaluations all eight instruction librarians were asked to participate in a peer evaluation program as both observer and observed. participation was optional. in order to prepare, an instruction meeting was held to discuss the background, evaluation process, and criteria for evaluating teaching effectiveness, as defined by rabine and cardwell (2000). during this time, all questions were answered related to the study, and participants discussed the benefits of being both the observer and observed. librarians were asked to observe four sessions and to be observed by two of their peers in two separate sessions. to increase effectiveness and reduce bias in the peer evaluation process, two librarians were assigned to provide feedback on each observed class. a peer evaluation form (appendix a) was designed and piloted in the fall of 2010, and updated and used to collect data in spring 2011. peers were asked to comment on preparation, instruction and delivery, class management, and instruction methods. critical feedback was provided and teaching effectiveness was measured. faculty survey twenty-two instructors were emailed an instructor evaluation form (appendix b) prior to instruction sessions so that they could observe the appropriate aspects of the sessions and report their personal evaluation. this survey was designed to measure teaching efficacy and asked participants to rank effectiveness and provide qualitative feedback regarding what they would have changed or what they particularly appreciated about the session. fourteen instructors returned the survey with critical feedback. the qualitative comments were grouped by theme to look for programmatic problems, while the individual instructor comments were summarized and given to the participating librarians. results the majority of requested one-shot instruction sessions were taught in february and march 2011, and we collected a great deal of data. despite the fact that historically fewer sessions are taught in the spring, and that the egyptian revolution caused the semester to be shortened by several weeks, in 31 one-shot sessions, 232 plus/delta forms were collected, 15 sessions were observed by colleagues, and 14 feedback surveys were returned by faculty. plus/delta the plus/delta forms provided useful feedback regarding what students had learned, or at least what they remembered from the sessions. out of 383 students surveyed, a total of 232 (77%) returned the survey. students seemed hesitant to complete the “something that i still find confusing” portion of the form, despite the promise of anonymity. perhaps they were uncomfortable with criticizing a perceived authority figure or perhaps they had been so unfamiliar with library resources prior to the one-shot that they were unable recognize what was still unclear. regardless, because of the large number of responses, we were able to draw some useful conclusions. we carefully sifted through all the plus/delta forms, and organized responses by specific theme (e.g., choosing keywords, academic search complete) and then by broader themes (e.g., specific skills, specific resources). choosing keywords would fall under “specific skills” and academic search complete under “specific resources.” additional broad themes under both plus and delta categories included “services” and “general comments.” general comments such as “developed research techniques” and “learned about library databases” came up frequently, as did general praise, such as “very helpful, thanks!” see appendix c for a complete list of identified themes. evidence based library and information practice 2012, 7.4 41 in september 2010, the auc library implemented the serials solutions product summon, a discovery platform for searching library resources. the platform was branded library one search (l1s), and is now the main search box on the library website. this new platform has been a focus of library instruction sessions, and many students referenced it under “something that i learned.” there were 53 references to l1s, often by name but sometimes by other terminology such as the “library search engine,” “library website search,” or other variations on these phrases. in all 53 instances, however, it was clear that the student respondent was referring to l1s. in total, we found 89 instances of general commentary under the plus responses. general comments under the delta heading included unspecified confusion, information overload, and having received similar training in other classes. peer evaluations although peer evaluations may have been the least methodologically sound assessment used in the study – as a result of issues related to peers judging peers (see discussion for more details) – they offered valuable insight on teaching strengths and opportunities for improvement. a total of 8 instruction librarians participated in 28 peer evaluations, where they were asked to evaluate the strengths and weaknesses of their colleagues’ teaching abilities. due to the egyptian revolution, scheduling observations and instruction sessions became extremely difficult as class started two weeks after the scheduled date and numerous other class days were cancelled due to unrest. as a result, we were unable to fulfill the 32 anticipated observations, and only 28 were collected via observation of 15 different class sessions. four librarians observed three sessions and two librarians observed only one session. four librarians were observed four times, two librarians were observed three times, and one librarian was observed once. the authors contributed 50% of the collected observations, instead of 25% as originally planned, as a result of scheduling challenges. seven of the classes observed were instruction sessions for rhetoric 201, a sophomore-level rhetoric and writing class in which students are required to write research papers. the other eight sessions were all discipline-specific instruction sessions ranging from art to biology. researchers asked librarians to provide qualitative data on four different aspects of their colleagues’ teaching skills: preparation, instruction and delivery, class management, and instruction methods. when asked to comment on preparation, all observing librarians stated that their peers were clearly prepared for the instruction session – through various methods such as preparing an outline, providing examples, and conducting discussion related to course content. comments included, “session was well planned. it followed a clearly defined outline,” and “clearly prepared for this session – all of her examples were related to student topics.” comments related to instruction and delivery and class management proved to be informative and helpful for librarian instructors. issues with voice tone and library jargon were frequently mentioned when discussing instruction and delivery. twelve observers mentioned that the teaching librarian talked with a clear and concise voice, while three observers mentioned that the teaching librarian talked too quickly and used too much library jargon. there were seven references to library instructors’ clearly identifying and clarifying library terminology. these comments are extremely important since the majority of auc students are non–native english speakers, and often unfamiliar with library resources and services. there were two references related to better classroom management, due to the inability to keep students’ attention and clearly explain concepts, such as, “her enthusiasm for and thorough knowledge of the resources sometimes led to longer explanations and details, which may have been less effective than a brief answer would have been.” evidence based library and information practice 2012, 7.4 42 although library instructors try to engage students, evaluation results show that far too much time is spent on lecturing and demonstrating. there were 28 references to library instructors using lecture and demonstration as the primary means of instructing students. we found nine references to actions meant to keep students engaged, such as providing students with the opportunity to work in class and soliciting questions from the class. observers were asked to rank their colleagues’ overall teaching effectiveness on a scale of one to ten. on average, librarian instructors received a rating of 7.85, with 6 being the lowest score received and 9 the highest score. see table 1. faculty survey the faculty feedback survey, created using surveymonkey and distributed via email, allowed faculty instructors the opportunity to provide feedback related to the perceived effectiveness of the library instruction session. researchers asked faculty members to provide qualitative feedback related to what they especially liked about the session and what they would have changed. in total, 22 surveys were distributed, and respondents completed and returned 14 surveys. in some cases, librarians forgot to distribute the survey. the return rate was surprisingly high considering there were four general guide to graduate research workshops assessed, which were general library sessions and student participation was optional. these latter sessions were not attended by faculty members. faculty members were asked to rank seven statements related to the success and instructional design of the session (see table 2). overall, faculty members strongly agreed that the session met their expectations, was focused on skills that were relevant to course assignments, and that the instructor clearly explained concepts. when asked if instructional activities were appropriate, five instructors strongly agreed, five agreed, and one was neutral. this figure indicated that new active learning activities could be implemented to engage students in the learning process. similarly, when asked if the instruction session better prepared students for research, five instructors strongly agreed, five instructors agreed, and one instructor was neutral. the most engaging and informative data was collected in the second part of the survey, in which faculty instructors were asked to describe what they particularly liked about the instruction session and what they would have changed. in order to analyze the open-ended responses, we coded and categorized comments to reflect specific skills and concepts, the same process used to analyze the plus/delta data. table 1 observations and frequency observation number of occurrences librarian clearly prepared for session 28 librarian spoke clearly 12 librarian explained unfamiliar terminology 7 librarian kept students engaged 9 librarian used lecture primarily 28 librarian spoke too quickly and used too much jargon 3 evidence based library and information practice 2012, 7.4 43 when asked to list what the faculty instructor particularly liked about the session, 16 respondents provided comments. an overwhelming majority of the positive responses from faculty dealt with teaching students how to find resources, both general and specific. six comments were related to finding general resources, such as, “showed them various ways to find resources in the library,” and “students were introduced to library database.” two comments also reflected the importance of effectively using books and library one search in the research process. there were two comments generally related to the presentation skills of librarians (“she was just great”). two comments directly addressed the librarians’ willingness to assist students and answer questions, for example, “stress and repetition of the librarian’s availability to answer questions at any time.” in response to what faculty members would have changed about the instruction session, seven instructors stated that they were satisfied with the session and did not have any changes to suggest. three faculty members listed specific resources and concepts they would have liked their students to learn, for example, “using resource from outside the university and interlibrary resources” and “how to refine a search.” three professors also commented on the structure of the class, suggesting variations table 2 statements ranked by faculty respondents answer options strongly agree agree neutral disagree strongly disagree n/a the session met my expectations. 8 3 0 0 0 0 the session focused on skills that are relevant to current course assignments. 11 1 0 0 0 0 the session instructor was clear in explaining concepts. 10 1 0 0 0 0 instructional materials (e.g., handouts, web pages, etc.) were useful. 6 3 0 0 0 2 instructional activities (e.g., discussions, planned searching exercises etc.) were appropriate. 5 5 1 0 0 0 in general, students are more prepared to conduct research for class assignments as a result of this session. 5 5 1 0 0 0 if there was hands-on computer time, i believe that students found the activities useful. 10 3 0 0 0 1 evidence based library and information practice 2012, 7.4 44 in how instructors deal with lecturing and allowing students to practice the skills they learned. two professors stated, “i would get students to engage as a group with instructor vs. one on one,” and “i would have built more time into the presentation for the students to use the skills they learned to research their own topics.” these comments emphasize two major points we discovered in the faculty survey and peer evaluation – more active learning techniques, such as group problem-solving activities, are needed to engage students in the learning process, and adjustments should be made to session structure. generally most librarians received positive feedback on their teaching. the use of multiple methods to evaluate teaching effectiveness, including plus/delta, peer evaluation, and instructor feedback surveys, provided the research and information services department with the data needed to improve teaching, student learning, and the overall instruction program. common themes found within the three evaluation tools showed an overall positive opinion of instruction librarians, but specific themes, such as a lack of active learning techniques, were identified throughout all evaluation tools. students, instructors, and observing librarians stated there was not enough time to engage with or utilize new knowledge. instruction librarians were most critical about the use of lecture and demonstration to teach library resources and skills – clearly, librarians need to engage with their students more effectively. all three assessments also showed that some instructors struggle with explaining certain concepts; for example, one instructor was noted for use of excessive library jargon. overall, the results from all three evaluation tools showed that students are learning new material and librarians deliver instruction sessions that are perceived in a positive way by teaching faculty and colleagues. discussion the results of this study were beneficial to the auc main library il program in two fundamental ways. first, we were able to identify larger issues that should be acknowledged and addressed program-wide. second, participating instruction librarians benefited from opportunities for reflection and growth. we were pleased that the use of multimethod evaluation provided a “big picture” view of the il program by including the perspectives of multiple shareholders, as has been demonstrated elsewhere in the literature (bowles-terry, 2012; rabine & cardwell, 2000). individual instructor growth opportunities at the end of the study, researchers provided all instruction librarians with a comprehensive feedback file, compiled by the authors, so that instructors would have the opportunity to review feedback and spend time on selfreflection in order to improve specific skills. in this way, those that needed to work on, for example, eliminating or explaining library jargon became aware of this opportunity for growth and improvement. an added benefit to using the peer evaluation method was the number of librarians who enjoyed observing their colleagues, which led to personal reflection and the incorporation of new teaching strategies. in addition, some of the observed librarians were eager to receive their own feedback for self-improvement purposes. creating this culture of evaluation can improve relationships between librarians and others on campus, thus leading to more effective collaboration. this university-wide culture of assessment and the library’s role within it has become an increasingly popular topic in the library assessment literature (sobel & wolf, 2010). departmental developments by using multiple methods and involving three main stakeholder groups, we were able to collect valuable information. it is certainly beneficial to repeat this type of evaluation annually, as teaching librarians develop and staff changes. the teaching reports we assembled at the end of the study were useful individually, for faculty evidence based library and information practice 2012, 7.4 45 reports, personal development, and for the instruction librarians as a group. since the results of the study were last analyzed, several librarians have taken advantage of professional development opportunities related to improving teaching. the auc main library is planning two series of workshops, the first of which will provide instruction librarians the opportunity to brush up on learning theory and teaching pedagogy. the second series of workshops will be provided to faculty members, either within or outside the library, who wish to learn more about information literacy and how they can make the most of one-shot instruction sessions. issues with student engagement all three of the evaluation tools revealed that the structure of one-shot sessions should be reconsidered in order to avoid too much lecture and demonstration. instructors might consider addressing the problem of time constraints by including less content but more group work so that students can learn from and teach one another. wong, chan, and chu (2006) found similar problems with student engagement, and adjusted the length of instruction sessions. students might also remain engaged and retain more information if active learning techniques were included when possible. for a variety of activities and ideas for increasing active learning in the library classroom, we suggest consulting the library instruction cookbook (sittler & cook, 2009). limitations when developing the plus/delta survey, we were confident that this evaluation technique would appeal to students because it was quick, simple, and immediate. however, as mentioned previously, it seems that some students were hesitant to give critical feedback. in the future, perhaps asking the professor to distribute the forms to students at their next class meeting or providing more specific prompts would be a better plan. students would feel more anonymous, and feedback might be more useful if it is not so immediate; librarians would discover what stuck with students after a couple of days. creating an online form to be completed at the end of the instruction session might have given the students the feeling that all submissions were anonymous, rather than completing and handing in an evaluation form to the library instructor. also, students might respond more clearly to more specific questions: they may have found the plus/delta format to be confusing or intimidating. in developing the faculty feedback portion of the study, we were faced with the decision of anonymity versus utility of feedback. we had access to all of the returned surveys, and faculty were aware of this fact. had the survey been anonymous, faculty might have felt more comfortable giving constructive criticism, but we would have been unable to trace the feedback to specific sessions and library instructors. requesting that faculty provide both anonymous and identifiable feedback could solve this problem, but may be asking too much. instead of asking faculty if they had prior contact with instruction librarians, we could have framed the question to reflect whether or not librarians helped with instructional design of assignments and if so, was it helpful? this would have allowed us to gauge whether or not librarian participation is effective beyond the one-shot sessions. the peer evaluation certainly provided some valuable guidance for instruction librarians, although this process was difficult for both the observer and observed. some librarians were nervous about the presence of colleagues in the classroom, and some librarians were uncomfortable ranking their colleagues. these issues, however, are unavoidable if this technique is utilized. the qualitative results were definitely more useful than the rating scale – we discovered that no one was willing to rank another librarian below a six, regardless of performance. although librarians hesitated to rank their peers, there were numerous qualitative suggestions and comments related to evidence based library and information practice 2012, 7.4 46 teaching effectiveness, classroom management, and delivery. recommendations we support developing and implementing a system of evaluation and recommend the following:  create a system of evaluation that is a continuous ongoing project, instead of focusing on a one-semester snapshot of teaching effectiveness. this will encourage instruction librarians to actively and continually improve their teaching.  beyond handing out comprehensive feedback files to instruction librarians intended for self-reflection, schedule individual meetings with librarians to discuss evaluations.  have instruction librarians set yearly goals related to specific skills they would like to improve. provide assistance and help develop these skills.  focus on creating a discussion of teaching effectiveness in your library and campus. work with your centre for teaching and learning to promote teaching and information literacy by planning and co-sponsoring workshops and other educational opportunities. conclusions evaluating effective teaching using multiple methods is useful in developing and maintaining a successful information literacy program. by involving all stakeholders in the evaluation process, a study can benefit from multiple perspectives on teaching effectiveness and ability. a cumulative look at all information collected and analyzed provides instruction librarians with information about areas in which teaching can be improved and also highlights areas of excellence. this study indicates that in general, instruction librarians, students, and faculty members are satisfied with il sessions, but there is room for improvement. individual librarian instructors benefit from opportunities to improve teaching through informed reflection. faculty members want their students to have more hands-on experience in the classroom. students need less lecturing and more authentic learning opportunities to engage with new knowledge. the overall evaluation of il instruction and programs sessions goes beyond measuring student learning outcomes, and should also focus heavily on effective teaching. we advocate for further research in this area to encourage a system of evaluation and assessment. it should be noted that this was a time-consuming process, and should be scaled to the available library resources. however, the improvement of instruction in academic libraries is a worthwhile endeavour, and serves to emphasize the importance of library resources and services for students and faculty. references arp, l., woodard, b. s., lindstrom, j., & shonrock, d. d. (2006). faculty-librarian collaboration to achieve integration of information literacy. reference & user services quarterly, 46(1), 18-23. association of college and research libraries. (2011). introduction. in standards for libraries in higher education. retrieved 4 dec. 2012 from http://www.ala.org/acrl/standards/stand ardslibraries. belanger, j., bliquez, r., & mondal, s. (2012). developing a collaborative facultylibrarian information literacy assessment project. library review, 61(2), 68-91. evidence based library and information practice 2012, 7.4 47 black, c., crest, s., & volland, m. (2001). building a successful information literacy infrastructure on the foundation of librarian-faculty collaboration. research strategies, 18(3), 215-225. bowles-terry, m. (2012). library instruction and academic success: a mixed-methods assessment of a library instruction program. evidence based library and information practice, 7(1), 82-95. retrieved 4 dec. 2012 from http://ejournals.library.ualberta.ca/index .php/eblip/article/view/12373 click, a., & walker, c. life after library school: on-the-job training for new instruction librarians. endnotes: the journal of the new members round table, 1(1). retrieved 4 dec. 2012 from http://www.ala.org/ala/mgrps/rts/nmrt/ oversightgroups/comm/schres/endnotes vol1is1/2li eafterlibrarysch.pdf furno, c., & flanagan, d. (2008). information literacy: getting the most from your 60 minutes. reference service review, 36(3), 264-271. hsieh, m. l., & holden, h. a. (2010). the effectiveness of a university’s singlesession information literacy instruction. reference service review, 38(3), 458-473. judd, v., tims, b., farrow, l., & periatt, j. (2004). evaluation and assessment of a library instruction component of an introduction to business course: a continuous process. reference services review, 32(3), 274-283. matthews, j. r. (2007). library assessment in higher education. westport, ct: libraries unlimited. middleton, c. (2002). evolution of peer evaluation of library instruction at oregon state university libraries. portal: libraries and the academy, 2(1), 6978. mcclanahan, e. b., & mcclanahan, l. l. (2010). active learning in a non-majors biology class: lessons learned. college teaching, 50(3), 92-96. oakleaf, m. (2009). the information literacy instruction assessment cycle: a guide for increasing student learning and improving librarian instructional skills. journal of documentation, 65(4), 539-560. rabine, j., & cardwell, c. (2000). start making sense: practical approaches to outcomes assessment for libraries. research strategies, 17, 319-335. samson, s., & mccrae, d. e. (2008). using peerreview to foster good teaching. reference services review, 36(1), 61-70. sittler, r. l., & cook, d. (eds.). (2009). the library instruction cookbook. chicago: association of college and research libraries. shonrock, d. d. (ed.). (1996). evaluating library instruction: sample questions, forms, and strategies for practical use. chicago: american library association. sobel, k., & wolf, k. (2010). updating your toolbelt: redesigning assessments of learning in the library. reference & user services quarterly, 50(3), 245-258. walter, s. (2006). instructional improvement: building capacity for the professional development of librarians as teachers. reference & user services quarterly, 45(3), 213-218. wong, g., chan, d., & chu, s. (2006). assessing the enduring impact of library evidence based library and information practice 2012, 7.4 48 instruction programs. the journal of academic librarianship, 32(4), 384-395. zald, a. e., gilchrist, d. (2008). instruction and program design through assessment. in c. n. cox & e. b. lindsay (eds.), information literacy instruction handbook (pp. 164-192). chicago: association of college and research libraries. appendix a peer evaluation 1. observer: 2. librarian instructor: 3. instructor and class (e.g. rhet 201, bob ross) 4. preparation: 5. instruction and delivery: 6. class management: 7. instruction methods: 8. on a scale from 1 to 10, how would you rate the effectiveness of this instruction session? comments: evidence based library and information practice 2012, 7.4 49 appendix b faculty feedback form 1. what was the date of the library instruction session? 2. how did you communicate with the librarian that taught the one shot prior to the session? __ in person __ on the phone __ via email __ didn’t communicate with the instructor 3. strongly agree agree neutral disagree strongly disagree n/a a. the session met my expectations. b. the session focused on skills that are relevant to current course assignments. c. the session instructor was clear in explaining concepts. d. instructional materials (e.g., handouts, web pages, etc.) were useful. e. instructional activities (e.g., discussions, planned searching exercises, etc.) were appropriate. f. in general, students are more prepared to conduct research for class assignments as a result of this session. g. if there was hands-on computer time, i believe that students evidence based library and information practice 2012, 7.4 50 found the activities useful. 4. what did you particularly like about the session? 5. what would you change about the session? appendix c plus/delta themes plus # of responses delta # of responses specific skills specific skills library one search 53 finding a book/call numbers 9 narrowing results 32 using online resources/library website 6 search connectors 30 evaluating sources 6 primary sources 14 accessing articles 5 finding books/call number 13 searching by discipline 5 keywords 12 narrowing a search 4 databases by major 12 citations 3 citations 10 search strategies 3 building a search statement 9 subject terms 2 search punctuation () "" * 8 finding fulltext 2 developing a research question 6 more online searching 1 database tools 5 search connectors 1 finding scholarly sources 4 types of resources 1 subject terms 4 building a search statement 1 using synonyms 3 finding fulltext 2 specific resources evaluating sources 1 refworks 5 identifying types of resources 1 other databases 5 arabic sources 4 specific resources catalog 1 refworks 10 print resources 1 academic search complete 8 subject guides 8 services google scholar 8 document delivery 7 proquest theses & dissertations 5 technical problems 3 historical newspapers 3 reserve 1 psychology databases 3 recommending books for purchase 1 evidence based library and information practice 2012, 7.4 51 digital archive & research repository 2 printing 1 political science complete 1 evening services 1 web of science 1 business source complete 1 general comments opposing viewpoints 1 lots of information/need to practice 9 vague confusion 7 services needed this information previously 5 document delivery 24 delivery too fast 3 help desk 1 general comments research techniques 34 databases 28 vague praise 27 evidence based library and information practice evidence based library and information practice 2011, 6.4 72 evidence based library and information practice article recording database searches for systematic reviews what is the value of adding a narrative to peer-review checklists? a case study of nice interventional procedures guidance jenny craven information specialist national institute for health and clinical excellence manchester, united kingdom email: jennifer.craven@nice.org.uk paul levay information specialist national institute for health and clinical excellence manchester, united kingdom email: paul.levay@nice.org.uk received: 25 august 2011 accepted: 05 november 2011 2011 craven and levay. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract this paper discusses the value of open and transparent methods for recording systematic database search strategies, showing how they have been applied at the national institute for health and clinical excellence (nice, see appendix c for definitions) in the united kingdom (uk). objective – the objectives are to: 1) discuss the value of search strategy recording methods. 2) assess any limitations to the practical application of a checklist approach. 3) make recommendations for recording systematic database searches. methods – the procedures for recording searches for interventional procedures guidance at nice were examined. a sample of current methods for recording systematic searches identified in the literature was compared to the nice processes. the case study analyses the search conducted for evidence about an interventional procedure and shows the practical issues involved in recording the database strategies. mailto:jennifer.craven@nice.org.uk mailto:paul.levay@nice.org.uk evidence based library and information practice 2011, 6.4 73 the case study explores why relevant papers were not retrieved by a search strategy meeting all of the criteria on the checklist used to peer review it. the evidence was required for guidance on non-rigid stabilisation techniques for the treatment of low back pain. results – the analysis shows that amending the medline strategy to make it more sensitive would have increased its yield by 6614 articles. examination of the search records together with correspondence between the analyst and the searcher reveals the peer reviewer had approved the search because its sensitivity was appropriate for the purpose of producing interventional procedures guidance. the case study demonstrates the limitations of relying on a checklist to ensure the quality of a database search without having any contextual information. conclusion – it is difficult for the peer reviewer to assess the subjective elements of a search without knowing why it has a particular structure or what the searcher intended. there is a risk that the peer reviewer will concentrate on the technical details, such as spelling mistakes, without having the contextual information. it is beneficial if the searcher records correspondence on key decisions and reports a summary alongside the search strategy. the narrative describes the major decisions that shaped the strategy and gives the peer reviewer an insight into the rationale for the search approach. introduction systematic reviews aim to provide more reliable conclusions than the individual studies that they contain by synthesising empirical evidence in a manner that minimises bias. systematic reviews have become a standard tool in the healthcare sector, as the concept of evidence-based medicine has become increasingly popular (bastian, glasziou, & chalmers, 2010). the cochrane collaboration, one of the leading producers of systematic reviews, states that their key characteristics include following an “explicit, reproducible methodology”. the manual for producing cochrane reviews goes on to show that the methodology relies on “a systematic search that attempts to identify all studies that would meet the eligibility criteria” (higgins & green, 2011, section 1.2.2). a rigorous search is required to produce a valid systematic review that is useful for clinicians and other decision makers. a full systematic search should employ a number of techniques for interrogating the literature, including citation searching, consulting experts and hand searching journals (crumley, wiebe, cramer, klassen, & hartling, 2005). this case study only covers one aspect of the process, i.e, systematically searching the available electronic databases. the choice of database and search platform is an important issue which has been explored elsewhere (booth, 2010; younger & boddy, 2009) and that issue is not covered here. there are two aspects to ensuring the systematic integrity of a database search. the first is to submit the search to a quality assurance process before it is run and the references downloaded. the second aspect is recording the actual search and presenting it alongside the review so that someone unfamiliar with the study can understand the methods. the purpose of this paper is to discuss the benefits of a checklist approach to peer reviewing the pre-search phase and to consider the implications of this for reporting at the post-search stage. the objectives are to assess the value of search strategy recording methods and to discuss any limitations to the evidence based library and information practice 2011, 6.4 74 practical application of a checklist approach. finally, it will make recommendations on recording systematic database searches. background the case study explores the process that the united kingdom (uk) national institute for health and clinical excellence (nice) uses to record the searches for its interventional procedures guidance (ipg). the searches are undertaken by a team of information professionals at nice, based in the information services department. nice is an independent organisation responsible for providing national guidance on the promotion of good health and the prevention and treatment of ill health (national institute for health and clinical excellence [nice], 2010a). nice interventional procedures guidance advises the uk national health service (nhs) on whether interventional procedures used for diagnosis or treatment are safe enough and work well enough to be recommended for routine use (nice, 2009a). interventional procedures involve making cuts in the body, gaining access to body cavities or using electromagnetic radiation. more than four hundred pieces of guidance have been published since the programme was launched in 2003. publication is one step in a longer process of encouraging the uptake of innovative technologies and procedures in clinical practice (lourenco, grant, burr, & vale, 2010). interventional procedures follow a series of structured steps, which take approximately nine months, to ensure that they are based on the best available evidence and meet the principles of openness and transparency (nice, 2007; nice, 2009a). literature review literature search methods the case study reported in this paper originated with a literature search that aimed to understand current practice and to identify any evidence-based recommendations for recording search strategies. this was not intended to be a systematic literature review. the search, undertaken in march 2011, took an iterative approach and included a number of techniques: searching medline using the ovid platform, the cochrane library using the wiley platform, library, information science and technology abstracts (lista ) using the ebsco platform, library and information science abstracts (lisa ) using the proquest platform and google scholar; scanning the tables of contents for health information and libraries journal, journal of the medical library association and evidence based library and information practice (eblip); consulting colleagues and systematic reviewers; and checking the reference lists of articles already identified. the searches combined free-text terms for systematic reviews, search strategies, search reporting, search recording, and checklists with subject headings, such as the mesh headings review literature as topic, meta-analysis as topic, and guidelines as topic. recording search strategies the purpose of recording a search strategy is to make the strategy used explicit and reproducible. the first step is to check the quality of the search and one method is to subject it to peer review. the canadian agency for drugs and technologies in health (cadth) investigated how consistently searches were being peer reviewed (sampson, mcgowan, lefebvre, moher, & grimshaw, 2008a). the systematic review conducted for the cadth report found that 26 tools to perform peer reviews were already in existence, although none of them had been validated against database searches (sampson et al., 2008a, p. 31). the report identifies a number of elements that could affect a search and ranks them in three tiers according to the level of impact they can have on performance. the output from the cadth report was a checklist called press ebc: peer review of electronic search strategies evidence based checklist (press) (mcgowan, sampson, & lefebvre, 2010). this checklist can be used to assess whether a good quality search has been evidence based library and information practice 2011, 6.4 75 performed, since the seven elements have been shown to reduce the precision or recall of a search if they have not been “skilfully executed” (mcgowan et al., 2008, p. 150). reporting search strategies once the search has been peer reviewed, the references screened, and the review written, the methods must be reported to an external audience. the literature review confirmed that it is “good evidence-based practice” to record the search process used to prepare a systematic review (deluca, mullins, lyles, & crepaz, 2008, p. 5). however, deluca et al. (2008) go on to argue that systematic reviews “often provide a limited explanation of the search methods used to capture the literature”, even though a poor search can have “critical implications” for the review (p. 5). where search records do exist there is “no clear consensus regarding optimum reporting of a systematic review search methods and commonly recommended items show suboptimal reporting” (sampson, mcgowan, tetzlaff, cogo, & moher, 2008b). in a more recent survey, niederstadt and droste (2010) conclude that “no generally accepted standard of reporting of [information retrieval] in [health technology assessment] exists”(p 450). the united states agency for healthcare research and quality has similarly concluded, in the context of comparative effectiveness research, that “better reporting and further research on search strategies is needed to develop additional evidence-based recommendations” (relevo & balshem, 2011, p. 1168). the value of consistent reporting procedures is highlighted by the work of fehrmann and thomas (2011) which shows that readers’ confidence in a review can be affected by how well the search methods have been reported. the cochrane handbook sets out requirements for documenting the search process (lefebvre, manheimer, & glanville, 2011, sections 6.6.1-6.6.3). the handbook stresses that reviews are only reproducible if there is a record of the full search strategy that was actually run on each database. there have been several studies assessing compliance with the cochrane search reporting standards, which are summarised in yoshii, plaut, mcgraw, anderson and wellik (2009). the centre for reviews and dissemination ([crd], 2009, appendix 3) has provided useful guidance on reporting the search process. the guidance makes a clear distinction between the very detailed search records that the information specialists should retain and the summary that should be reported in the published version (p. 249). the detailed records could still be made available as an appendix or online-only document. guidelines for systematic reviews there are a number of guidelines available for reporting the entire systematic review process (moher et al., 2011). the preferred reporting items for systematic reviews and metaanalyses (prisma) instrument provides 27 elements to include in a systematic review report and two of these relate to searching (liberati et al., 2009). prisma recommends using press to check searches. the institute of medicine (2011) has issued guidance on conducting systematic reviews and standard 3.1 clearly states that one of the required elements for conducting a comprehensive search for evidence is to “use an independent librarian or other information specialist to peer review the search strategy” (p 84). the appraisal of guidelines research and evaluation (agree) instrument can be used to assess the quality of clinical practice guidelines (agree collaboration, 2001). agree includes a search dimension and assessors have to judge how far they agree with the statement that “systematic methods were used to search for evidence”(p 10). systematic reviews in other disciplines press was developed for health technology assessments and it does not transfer directly to searches for qualitative evidence, which have evidence based library and information practice 2011, 6.4 76 their own requirements and generally make more use of purposeful literature sampling. booth (2006) has proposed the starlite checklist for qualitative searches. reporting standards have been adapted to other disciplines and they should be appropriate to the types of evidence being used. the social care institute for excellence (rutter, francis, coren, & fisher, 2010) provides an example of procedures and guidance in a different sector. the campbell collaboration produces systematic reviews on education, criminal justice and social welfare topics, and its manual incorporates guidance on searching similar to cochrane (hammerstrøm, wade, & jørgensen, 2010). limitations of the current recommendations a search strategy must be recorded properly before it can be peer reviewed with a checklist. a search would be rejected if any of the elements were missing, for example, if there were no indication of which database platform had been used. the current checklists are limited because they present the peer reviewer with the final search strategy without any background information. the checklists do not prompt the peer reviewer to consider how the strategy was developed, even though the final version may have changed considerably since its first draft. the various iterations could reveal important information about how the search for evidence has been conceptualised. this is not to suggest that the peer reviewer needs to see each draft of the search. the developmental process can be summarised into a narrative description that provides a rationale for the approach and is far richer than a search strategy seen in isolation. there are already some instructions for searchers to record this information. the crd guidance encourages the searcher to record and explain any decisions made during searching (crd, 2009, p. 21). the blank template provided in the campbell manual encourages searchers to record the purpose of the search and advises keeping contemporaneous notes on any decisions (hammerstrøm, wade, & jørgensen, 2010, pp. 31, 69). it would be helpful if these recommendations were given more prominence and if they were reported consistently to facilitate peer review. this paper describes the peer reviewing and recording of search strategies for interventional procedures guidance at nice. the case study will demonstrate the limits of peer reviewing with a checklist when the rationale has not been reported along with the search strategy. methods a case study approach has been used to explore the process that nice uses to record searches and to assess the benefits of the checklist approach for peer reviewing. the existing methods used to peer review searches and record search strategies identified in the literature search were compared to the processes at nice and the interventional procedures programme in particular. the information services team has its own process documents detailing how it supports the various nice programmes. the case study shows the similarities between the lessons learned at nice and the experiences reported in the literature. interventional procedures search process the information services team at nice works closely with analysts from the centre for health technology evaluation (chte) to locate the evidence for interventional procedures guidance. the analyst uses the literature to write an overview document which forms the evidence presented to the interventional procedures advisory committee for use in its discussions. a search strategy is developed in medline, using the ovid interface. the search might go through several iterations to ensure that it is retrieving the relevant literature. a senior member of the team peer reviews all searches against a checklist (see appendix a), which has been tailored to the nice process and preevidence based library and information practice 2011, 6.4 77 dates press by several years, although there are a number of similarities. the strategy is also run in a core set of databases, such as the cochrane library, embase, and cinahl. the medline strategy is translated into the appropriate vocabulary and syntax of the other databases and platforms. the quality of these search strategies is also assured through a process of peer review to maintain standards and minimise errors. a spreadsheet for recording the names of the searcher and the reviewer is useful for dealing with any subsequent enquiries from analysts or committee members. the process usually allows 14 days for designing the searches, peer reviewing and then actually performing the search and downloading the results. a template has been developed for recording the search information, which includes fields such as the name of the database or website searched, date searched, version used and the number of hits retrieved. copies of all the search strategies, as they were actually run, are also retained. the template ensures that the information is recorded consistently across topics, regardless of who performed the search. the information is clearly laid out which makes it straightforward to reproduce the search if it needs to be updated at a later date. the completed templates form the appendix to the overview document presented to the committee (see nice, 2009b, appendix c for an example). this process ensures that the committee members have both the evidence they need and an explanation of how that evidence was located. results the case study concerned the production of interventional procedure guidance number ipg 366, which was published in november 2010 (nice, 2010b) and replaced ipg 183 from june 2006 (nice, 2006). the final guidance document recommends that non-rigid implants between two or more vertebrae can be routinely offered as a treatment option to nhs patients with low back pain. the evidence on non-rigid implants was collected according to the process described above, with information services running the database searches and the analyst screening the abstracts. the evidence was collated into an overview document and this was presented to a committee including experts on spinal surgery. the committee members are a useful resource for identifying additional studies because they are familiar with the literature and the most up-to-date evidence in their field. three papers identified at the committee stage were not in the first draft of the overview document. the references, identified through this contact with experts, were included in the next version of the overview document (nice, 2009b) and they were available when the committee made its final recommendations. the searches met the criteria in the peer review checklist, however the three papers identified by the experts, still were not retrieved. the case study analysed why plev and sutcliffe (2005), kanayama et al. (2005) and grevitt et al. (1995) were missed by the database searches. the case study involved rerunning the original search from october 2009 (reported in nice 2009b) and then modifying it to see if it could have found the three papers. date limits were applied to the new search, performed in july 2011, to give an approximation of the results that would have been available in october 2009. the full search strategy is reported in appendix b. the obvious first question to ask was whether the articles were actually indexed in the databases that were searched. the plev and sutcliffe (2005) study is not indexed in any of the databases (medline, embase, cinahl and the cochrane library). the journal in question, spine journal: official journal of the north american spine society, is indexed in medline and embase; however further investigation revealed that the article appeared in a special supplement reporting conference abstracts that had not been indexed in either database. evidence based library and information practice 2011, 6.4 78 the initial check verified that the other two articles were available through the ovid interface to medline [appendix b, lines 5859]. the next question was whether the article was available in the database at the time of searching (spreckelsen, deserno, & spitzer, 2010). the complete medline record for the kanayama et al. (2005) paper shows that it was added to the database on 6 april 2006 and the search was conducted on 1 october 2009. grevitt et al. (1995) was added to medline in november 1995. it has been established that two of the papers were available in medline at the time of the search. it is worth going back to the quality assurance checklist (whether the nice or press version) and retracing the peer review process. the checklist should be completed for each database to obtain the complete picture of what happened. the illustration here only considers medline because this is the key strategy used and any issues would have translated to the other databases. the spellings, syntax, line numbers, truncation, boolean operators and proximity operators were applied correctly and the terms were all nested appropriately, in accordance with the criteria in appendix a. a filter had been applied to remove animal studies but this was not responsible for the two papers being eliminated [lines 69-70]. the search used a combination of mesh headings and natural language to capture relevant articles. the subject headings were exploded and appropriate sub-headings had been used. the free-text terms included synonyms and alternative spellings. the strategy in appendix b appears to address the research question by combining terms for the procedure and condition. a closer look at how grevitt et al. (1995) was indexed in medline reveals the problem. the article was indexed with the mesh term low back pain and this was contained in the strategy [lines 35-36]. the search was structured [line 46] so that it retrieved articles with at least one term for the procedure and one for the condition (whether in the subject headings or free text). the problem must therefore lie with how the interventional procedure was described in the strategy. several mesh headings attached to the medline record appear in the search: (bone screws [line 11], lumbar vertebrae surgery [line 20], and internal fixators [line 12]). the issue is that these subject headings have been combined using a boolean and with the free text terms for flexible, dynamic, and non-rigid [lines 30-31]. none of the free-text terms for non-rigid appear in the title or abstract of grevitt et al. (1995) thus the article was not found [line 61]. the same terms also account for the absence of kanayama et al. (2005) from the final results [line 62]. this explains why the search missed the articles but what accounts for the search being constructed in this way? the reconstructed search in appendix b shows that the two papers are retrieved if the free-text terms to describe non-rigid are removed from the strategy [lines 67-68]. the reconstructed search also shows that this line of free-text terms has a major impact on the yield; the original strategy retrieves 695 hits [line 57], whereas the version omitting the terms for non-rigid results in 7309 hits [line 66]. this means that an additional 6614 papers would have to be screened to locate grevitt et al. (1995) and kanayama et al. (2005). the reconstructed search shows that similar figures would have occurred at the time of the original search in october 2009 (with a difference of 6614 again between 554 [line 72] and 7168 [line 73]). these figures represent only the medline results and the number of additional papers required for screening would be much higher if the non-rigid terms were removed from the cinahl, embase and cochrane database strategies. the line of non-rigid free-text terms increases the precision of the search and reduces the number of articles to be screened by over 90%. the line reduces the number of irrelevant hits on rigid stabilisation techniques that would otherwise have been screened. the changes to the search were balanced by the contact with evidence based library and information practice 2011, 6.4 79 experts, and for this reason the other articles excluded by the line of free-text were not reviewed. the three papers confirm that contact with experts (mcmanus et al., 1998) and hand searching (hopewell, clarke, lefebvre, & scherer, 2007) can be useful supplements to database searches. including a narrative explanation of the search would not have resulted in the articles being retrieved from medline, although it would have provided a clear explanation for why they had not been found. the search met the checklist criteria because the peer reviewer understood the context and the reasons for the strategy being made precise. nice must issue timely guidance to the nhs and this affects the deadlines for reviews, the time available for screening references and the appropriate level of sensitivity in the searches. the original search was sufficiently robust for the purpose of producing interventional procedures guidance. the same search might have failed the peer-review process if a different organisation, with more time available, had been undertaking the systematic review. booth (2010) has discussed in more detail the implications of adopting the optimal approach to searching and the effects it can have on the comprehensiveness of results. the nice template that is used to record the search strategies has since been amended to prompt the searcher to record a short narrative on the purpose and structure of the strategy. discussion the case study illustrates the limitations of using a checklist for peer review without understanding why a strategy has a particular structure. a full peer review requires “an ascertainment that no technical errors have been made and a more subjective assessment of the adequacy of term selection” (sampson et al., 2008a, p. 32). the cadth report acknowledges that these two factors require different levels of knowledge to evaluate the search, with spelling mistakes a “largely mechanical” requirement, compared to the expertise required to judge how well the research question has been translated into a series of interlinked search concepts. the subjective elements are best judged when the searcher provides the contextual information that explains the structure of the search, as this case study has demonstrated. the searcher needs to retain correspondence regarding changes to the search, including the results from any test searches. the peer reviewer needs to understand the iterations that have been tried, the terms tested and removed, the reasons for terms being added, or the choice of a search filter. this background information could, for example, show that the search is sufficiently sensitive for its purpose and help the peer reviewer to avoid suggesting subject headings that have already been tested and removed from the search. it would not be feasible to report all of the communication or test results but a short paragraph summarising the decisions would be a valuable addition. the narrative could be useful for readers of the published review who need to decide whether it is a valid study. prisma (liberati et al., 2009) and crd (2009) acknowledge that the word limits in journals restrict the amount of information that can be reported but they both encourage authors to make their detailed records available, for example in online appendices. the narrative would be a useful addition to these search reports and provide the reader with the rationale for the search approach. the narrative would be an additional tool for facilitating critical appraisal and promoting transparent methods. two searches could legitimately be structured quite differently, even if they were attempting to answer similar research questions. a nice interventional procedure guidance search has a different purpose than a cochrane review. the differences in the purpose of the search, the time available, and other external factors can influence the appropriate level of sensitivity in the search. the peer reviewer has to take this into account when assuring the quality of the strategy. this means that any checklists used at the pre-search stage will evidence based library and information practice 2011, 6.4 80 need to be adapted to the needs of the organisation undertaking the search. it also means that any standardised checklists for peer reviewing published reviews, such as press, will be limited if the strategy is not accompanied by a narrative justifying its structure. conclusion the case study of interventional procedures guidance at nice has illustrated issues with the way that database searches are peer reviewed and reported. the checklist approach has been beneficial for facilitating consistency, accuracy and transparency in assessing searches. the risk with a checklist is that it focuses on the technical details of a search and quality assurance becomes something of a mechanical task. the peer reviewer performs a more effective role when contextual information is available to help judge the subjective elements of the search. the contextual information is only available if the searcher has recorded the decision-making process and presented it in a convenient format, such as a short narrative. the narrative should be recorded for the internal peer review process and it is also useful to report it to external readers of the report. the search strategy shows how the evidence was located and the narrative explains why it was done this way. a checklist for quality assurance and a narrative record of the major decisions are valuable for demonstrating that a search has been done systematically and that it followed an explicit, reproducible methodology. references agree collaboration. (2001). appraisal of guidelines for research & evaluation instrument. london: agree collaboration. retrieved 13 nov. 2011 from http://apps.who.int/rhl/agreeinstrume ntfinal.pdf bastian, h., glasziou, p., & chalmers, i. (2010). seventy-five trials and eleven systematic reviews a day: how will we ever keep up? plos medicine, 7(9), e1000326. doi:10.1371/journal.pmed.1000326 booth, a. (2006). “brimful of starlite”: toward standards for reporting literature searches. journal of the medical library association, 94(4), 421429, e205. booth, a. (2010). how much searching is enough? comprehensive versus optimal retrieval for technology assessments. international journal of technology assessment in health care, 26(4), 431-435. doi:10.1017/s0266462310000966 centre for reviews and dissemination. (2009). systematic reviews: crd's guidance for undertaking reviews in health care. york: centre for reviews and dissemination, university of york. retrieved 13 nov. 2011 from http://www.york.ac.uk/inst/crd/index_ guidance.htm crumley, e., wiebe, n., cramer, k., klassen, t., & hartling, l. (2005). which resources should be used to identify rct/ccts for systematic reviews: a systematic review. bmc medical research methodology, 5, 24. doi:10.1186/1471-2288-5-24 deluca, j.b., mullins, m.m., lyles, c.m., crepaz, n., kay, l., & thadiparthi, s. (2008). developing a comprehensive search strategy for evidence based systematic reviews. evidence based library and information practice, 3(1), 332. retrieved 13 nov. 2011 from http://ejournals.library.ualberta.ca/ind ex.php/eblip/article/view/855 fehrmann, p. & thomas, j. (2011). comprehensive computer searches and reporting in systematic reviews. evidence based library and information practice 2011, 6.4 81 research synthesis methods, 2(1), 15-32. doi:10.1002/jrsm.31 grevitt, m.p., gardner, a.d., spilsbury, j., shackleford, i.m., baskerville, r., pursell, l.m., hassaan a.,& mulholland, r.c. (1995). the graf stabilisation system: early results in 50 patients. european spine journal, 4(3), 169-175. doi:10.1007/bf00298241 hammerstrøm, k., wade, a., & jørgensen, a.m.k. (2010). searching for studies: a guide to information retrieval for campbell systematic reviews. oslo: campbell collaboration. retrieved 13 nov. 2011 from http://www.campbellcollaboration.org /news_/new_information_retrieval_gu ide.php higgins, j.p.t., & green, s. (eds.). (2011). cochrane handbook for systematic reviews of interventions version 5.1.0 [updated march 2011]. oxford: the cochrane collaboration. retrieved 13 nov. 2011 from http://www.cochranehandbook.org/ hopewell, s., clarke, m., lefebvre, c., & scherer, r. (2007). handsearching versus electronic searching to identify reports of randomized trials. cochrane database of systematic reviews, 2007(issue 2), mr000001. doi:10.1002/14651858.mr000001.pub2 institute of medicine (u.s.). committee on standards for systematic reviews of comparative effectiveness research.. (2011). finding what works in health care: standards for systematic reviews. washington, d.c.: national academies press. retrieved 13 nov. 2011 from http://www.iom.edu/reports/2011/fin ding-what-works-in-health-carestandards-for-systematicreviews.aspx kanayama, m., hashimoto, t., shigenobu, k., oha, f., ishida, t., & yamane, s. (2005). non-fusion surgery for degenerative spondylolisthesis using artificial ligament stabilization: surgical indication and clinical results. spine, 30(5), 588-592. doi:10.1097/01.brs.0000154766.74637.5e lefebvre c., manheimer e., & glanville j. (2011). chapter 6: searching for studies. in higgins, j.p.t., & green, s. (eds.), cochrane handbook for systematic reviews of interventions version 5.1.0 [updated march2011]. oxford: the cochrane collaboration. retrieved 13 nov. 2011 from http://www.cochranehandbook.org/ liberati, a., altman, d.g., tetzlaff, j., mulrow, c., gotzsche, p.c., ioannidis, j.p.a., clarke, m., devereaus, p.j.,kleiinen j., & moher, d. (2009). the prisma statement for reporting systematic reviews and meta-analysis of studies that evaluate health care interventions: explanation and elaboration. plos medicine, 6(7) e1000100. doi:10.1371/journal.pmed.1000100 lourenco, t., grant a.m., burr, j.m., & vale, l. (2011). the introduction of new interventional procedures in the british national health service a qualitative study. health policy, 100, 35-42. doi:10.1016/j.healthpol.2010.08.012 mcgowan, j., sampson, m., & lefebvre, c. (2010). an evidence based checklist for the peer review of electronic search strategies (press ebc). evidence based library and information practice, 5(1), 149-154. retrieved 13 nov. 2011 from http://ejournals.library.ualberta.ca/ind ex.php/eblip/article/view/7402 mcmanus, r.j., wilson, s., delaney, b.c., fitzmaurice, d.a., hyde, c.j., tobias, r.s., jowett, s,& hobbs, f.d.r. (1998). evidence based library and information practice 2011, 6.4 82 review of the usefulness of contacting other experts when conducting a literature search for systematic reviews. bmj, 317(7172), 1562-1563. moher d., weeks, l., ocampo, m., seely, d., sampson, m., altman, d.g., schultz, k.f., miller, d., simera, i., grimshaw, j, & hoey, j. (2011). describing reporting guidelines for health research: a systematic review. journal of clinical epidemiology, 64(7), 718-742. doi:10.1016/j.clinepi.2010.09.013 national institute for health and clinical excellence. (2006). non-rigid stabilisation techniques for the treatment of low back pain (interventional procedures guidance ipg183). london: national institute for health and clinical excellence. retrieved 13 nov. 2011 from http://www.nice.org.uk/nicemedia/pd f/ipg183guidance.pdf national institute for health and clinical excellence. (2007). interventional procedures programme methods guide. london: national institute for health and clinical excellence. retrieved 13 nov. 2011 from http://www.nice.org.uk/aboutnice/ho wwework/developingniceintervention alprocedures/interventionalprocedure sprogrammemanual/interventionalpro ceduresprogrammemanual.jsp?domed ia=1&mid=70c7f0d8-19b9-e0b5d464c50580f580b1 national institute for health and clinical excellence. (2009a). interventional procedures programme process guide. london: national institute for health and clinical excellence. retrieved 13 nov. 2011 from http://www.nice.org.uk/aboutnice/ho wwework/developingniceintervention alprocedures/interventionalprocedure sprogrammemanual/interventionalpro ceduresprogrammemanual.jsp?domed ia=1&mid=96186dbc-19b9-e0b5d4b7abf517dfc88b national institute for health and clinical excellence. (2009b) interventional procedure overview of non-rigid stabilisation techniques for the treatment of low back pain. london: national institute for health and clinical excellence. retrieved 13 nov. 2011 from http://www.nice.org.uk/nicemedia/liv e/11204/49541/49541.pdf national institute for health and clinical excellence. (2010a). about nice guidance: what does it mean for me? a guide for patients, carers and the public. london: national institute for health and clinical excellence. retrieved 13 nov. 2011 from http://www.nice.org.uk/media/b88/ef /niceaboutguidancefeb2010.pdf national institute for health and clinical excellence. (2010b). non-rigid stabilisation techniques for the treatment of low back pain (interventional procedures guidance ipg366). london: national institute for health and clinical excellence. retrieved 13 nov. 2011 from http://www.nice.org.uk/ipg366 niederstadt, c., & droste, s. (2010). reporting and presenting information retrieval processes: the need for optimizing common practice in health technology assessment. international journal of technology assessment in health care, 26(4), 450-457. doi:10.1017/s0266462310001066 plev d., & sutcliffe, j.c. (2005). outcome and complications using a dynamic neutralization and stabilization pedicle screw system (dynesys): is this a "soft fusion"? spine journal, 5(4 suppl. 1), s141-s142. doi:10.1016/j.spinee.2005.05.282 evidence based library and information practice 2011, 6.4 83 relevo, r. & balshem, h. (2011). finding evidence for comparing medical interventions: agency for healthcare research and quality (ahrq) and the effective health care program. journal of clinical epidemiology, 64(11), 11681177. doi:10.1016/j.jclinepi.2010.11.022 rutter, d., francis, j., coren, e., & fisher, m. (2010). scie systematic research reviews: guidelines, 2nd ed. london: social care institute for excellence. retrieved 13 nov. 2011 from http://www.scie.org.uk/publications/r esearchresources/rr01.pdf sampson, m., mcgowan, j., lefebvre, c., moher, d., & grimshaw, j. (2008a). press: peer review of electronic search strategies. ottawa: canadian agency for drugs and technologies in health. retrieved from http://www.cadth.ca/publication/781 sampson, m., mcgowan, j., tetzlaff, j., cogo e., & moher, d. (2008b). no consensus exists on search reporting methods for systematic reviews. journal of clinical epidemiology, 61(8), 748-754. doi:10.1016/j.clinepi.2007.10.009 spreckelsen, c., deserno, t.m., & spitzer, k. (2010). the publication echo: effects of retrieving literature in pubmed by year of publication. international journal of medical informatics, 79(4), 297-303. doi: 10.1016/j.ijmedinf.2010.01.007 yoshii, a., plaut, d.a., mcgraw, k.a., anderson, m.j., & wellik, k.e. (2009). analysis of the reporting of search strategies in cochrane systematic reviews. journal of the medical library association, 97(1), 21-29. doi:10.3163/1536-5050.97.1.004 younger, p. & boddy, k. (2009). when is a search not a search? a comparison of searching the amed complementary health database via ebscohost, ovid and dialog. health information & libraries journal, 26(2):126-135. doi:10.1111/j.1471-1842.2008.00785.x evidence based library and information practice 2011, 6.4 84 appendix a. nice checklist to evaluate the quality of internal search strategies:  all relevant concepts are included in the strategy  all relevant mesh and free text terms have been identified and included  boolean logic is accurate, within and between concepts  proximity operators have been used appropriately  truncations are appropriate  brackets are in the right place  there are no spelling mistakes  device names have been included in the strategy as appropriate  there are no spurious results which may indicate an error, e.g. lines with zero hits  the filter for eliminating animal studies has been applied correctly  for review searches, the search has been limited by entry date from the date of the last search  any other limits have been applied as appropriate  the analyst has been consulted if anything is unclear in the literature search request form evidence based library and information practice 2011, 6.4 85 appendix b. modified search strategy for nice ipg 366 database(s): ovid medline(r) 1948 to july week 1 2011 date searched: 17 july 2011 1 (flexi$ adj3 (screw$ or implant$ or device$ or instrument$)).tw. (1110) 2 (rotat$ adj3 (screw$ or implant$ or device$ or instrument$)).tw. (1016) 3 (dynesis or dynesys).tw. (50) 4 dynamic neutrali?ation system$.tw. (8) 5 (dynamic adj2 (fus$ or stabili$)).tw. (1118) 6 or/1-5 (3222) 7 (interspin$ adj3 implant$).tw. (58) 8 (graf$ adj3 soft$ adj3 stabili$ adj3 system$).tw. (1) 9 orthopedic fixation devices/ or bone nails/ or bone plates/ or bone screws/ or bone wires/ or internal fixators/ or splints/ or suture anchors/ (43487) 10 (orthoped$ adj3 fix$ adj3 device$).tw. (25) 11 (bone$ adj3 (nail$ or plate$ or screw$ or wire$)).tw. (4388) 12 (internal adj3 fix$).tw. (9746) 13 splint$.tw. (9308) 14 (suture$ adj3 anchor$).tw. (1142) 15 exp arthrodesis/ (20186) 16 arthrodesis$.tw. (7188) 17 (spin$ adj3 fus$).tw. (4642) 18 exp laminectomy/ (7134) 19 laminectom$.tw. (5354) 20 exp lumbar vertebrae/su [surgery] (8983) 21 (lumbar$ adj3 vertebr$).tw. (5118) 22 ((lumbar or pedicle) adj3 fus$).tw. (2324) 23 ((ligament$ or fusion$) adj3 (bone graft or pedical screw) adj3 lumbar).tw. (16) 24 intervertebral disk/ (9070) 25 "prostheses and implants"/ (34063) 26 24 and 25 (295) 27 (prosthet$ adj3 (interverteb$ adj3 (disc or disk))).tw. (6) 28 or/7-23 (90011) 29 26 or 27 or 28 (90124) [terms for the interventional procedure] 30 (flexib$ or dynamic or non-rigid or non rigid).tw. (210337) [free text terms for non-rigid] 31 29 and 30 (3592) [terms for the interventional procedure combined with free text terms for nonrigid] 32 6 or 31 (6514) [expansion of the terms for the interventional procedure] 33 exp spinal stenosis/ (3455) 34 (spin$ adj3 stenos$).tw (2891). 35 (low$ adj3 back$ adj3 pain$).tw. (14058) 36 low back pain/ or failed back surgery syndrome/ (11335) 37 exp spondylolysis/ or spondylolisthesis/ (3618) 38 spondylolisthesis.tw. (2613) 39 spondylolysis.tw. (916) 40 (lumbar$ adj3 decompress$).tw. (413) 41 (lumbar adj3 dis$ adj3 disease$).tw. (1152) 42 degenerative dis$ disease$.tw. (911) 43 ((disc or disk) adj3 herniat$).tw. (5313) 44 listhesis$.tw. (80) evidence based library and information practice 2011, 6.4 86 45 (flexion$ adj3 instab$).tw. (109) 46 or/33-45 (30123) [terms for the condition] 47 32 and 46 (340) [terms for the interventional procedure and terms for the condition] 48 fass.tw. (122) 49 diam implant$.tw. (4) 50 interspinous u.tw. (4) 51 x-stop.tw. (45) 52 mims.tw. (244) 53 (wallis adj5 stabili$).tw. (3) 54 or/48-53 (420) [alternative names for the device] 55 47 or 54 (749) [terms for the procedure and the condition combined with names for the device] 56 animals/ not humans/ (3533433) [filter to exclude animal studies] 57 55 not 56 (695) [search results with animal studies removed] 58 graf stabilisation system: early results in 50 patients.ti. (1) [grevitt et al.] 59 non-fusion surgery for degenerative spondylolisthesis using artificial ligament stabilization.ti. (1) [kanayama et al.] 60 (outcome and complications using a dynamic neutralization and stabilization pedicle screw system).ti. (0) [plev et al. is not in medline] 61 57 and 58 (0) [grevitt et al. not retrieved by the original search] 62 57 and 59 (0) [kanayama et al. not retrieved by the original search] 63 6 or 29 (92860) [terms for the procedure with non-rigid free-text terms excluded] 64 63 and 46 (7137) [terms for the procedure and condition with non-rigid free-text terms excluded] 65 64 or 54 (7512) [terms for the procedure and condition combined with names for the device, with non-rigid free-text terms excluded] 66 65 not 56 (7309) [animals filter applied to the new search with the non-rigid free-text terms excluded] 67 66 and 58 (1) [grevitt et al. retrieved when non-rigid free-text terms are excluded] 68 66 and 59 (1) [kanayama et al. retrieved when non-rigid free-text terms are excluded] 69 55 and 58 (0) [grevitt et al. still not retrieved by the original search when the animals filter is not used] 70 55 and 59 (0) [kanayama et al. still not retrieved by the original search when the animals filter is not used] 71 limit 57 to ed=20091001-20110717 (141) [items added between the original search in october 2009 and the new search for this paper] 72 57 not 71 (554) [approximation of the results in october 2009 when the search was originally run] 73 66 not 71 (7168) [approximation of the search results if the non-rigid terms had been excluded in october 2009] 74 72 and 58 (0) [grevitt et al. still not retrieved when strategy restricted to the dates of the original search] 75 72 and 59 (0) [kanayama et al. still not retrieved when strategy restricted to the dates of the original search] 76 73 and 58 (1) [grevitt et al. would have been retrieved if the non-rigid terms had been excluded in october 2009] 77 73 and 59 (1) [kanayama et al. would have been retrieved if the non-rigid terms had been excluded in october 2009] evidence based library and information practice 2011, 6.4 87 appendix c. abbreviations agree appraisal of guidelines research and evaluation cadth canadian agency for drugs and technologies in health chte centre for health technology evaluation (nice) crd centre for reviews and dissemination hta health technology assessment ipg interventional procedures guidance ipac interventional procedures advisory committee mesh medical subject headings nice national institute for health and clinical excellence nhs national health service press peer review of electronic search strategies press ebc peer review of electronic search strategies evidence based checklist prisma preferred reporting items for systematic reviews and meta-analyses starlite sampling type approach range limits inclusion terms electronic sources evidence summary   facet use in search tools is influenced by the interface but remains difficult to predict   a review of: dahlen, s. p. c., haeger, h., hanson, k., & montellano, m. (2020). almost in the wild: student search behaviors when librarians aren’t looking. journal of academic librarianship, 46(1), 102096. https://doi.org/10.1016/j.acalib.2019.102096   reviewed by: scott goldstein coordinator, web services & library technology mcgill university library montréal, québec, canada email: scott.goldstein@mcgill.ca   received: 1 june 2020                                                               accepted:  22 july 2020      2020 goldstein. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29790     abstract   objective – to examine the relationship between student search behaviours and the quality of scholarly sources chosen from among library search tools.   design – unmonitored search sessions in a facilitated library setting.   setting – a mid-sized public university in the united states of america.   subjects – 50 upper-level undergraduate students in the social and behavioural sciences.   methods – recruited participants were given one of two search prompts and asked to use ebsco’s social science abstracts and two configurations of proquest’s summon, with one being pre-scoped to exclude newspapers and include subject areas within the social sciences. the search tools were assigned in random order. in each case, the participant was asked to find two of the “best quality” articles (p. 3). a librarian was present in the room but did not observe participants; instead, all sessions were recorded using camtasia relay. afterwards, participants were interviewed about the process they used and their impressions of the search tools. they also completed a survey collecting information on their gpa and whether they had previously had library instruction.   main results – facet use differed significantly between the ebsco database and summon, though not between the two different configurations of summon. there was a significant relationship between high use of facets in one platform being connected to high use in the other platforms. in contrast to some previous studies, a non-trivial proportion of participants went beyond the first page of search results. in support of most previous studies, participants infrequently searched on the subject field or changed the default sort order. summon’s article suggestion feature was noted as being especially helpful, and clicking on suggested articles was significantly correlated with the number of article records viewed.   conclusion – the choice of search tool has a large influence on students’ subsequent search behaviour. many of the advanced features are still missed by students, although in this study the majority of sources picked were of high quality. the authors note the importance of configuring the interface so that facets and other features deemed worthwhile by librarians are higher up on the page. the researchers reason that the prominent display of facets leads to greater uptake. despite finding no association between library instruction and facet use, teaching students how to use facets remains an advisable strategy.   commentary   this article, a continuation of research previously conducted by two of the authors, examines a few related questions that all revolve around the use of discovery systems and similar search tools in academic libraries. foremost among them is the extent to which users take advantage of facets (or limiters). previous research has shown that many students do not use—and perhaps do not fully understand—facets (bloom & deyrup, 2015). the authors add to this literature by using a within-subjects design testing three interfaces (two configurations of summon and the social science abstracts database) to determine how this might affect facet use. however, the study measures several other variables including the amount of time taken during the searches, which links were clicked, and ratings of authority and relevance of the articles that were selected.   this commentary relies on the cat critical appraisal tool (perryman & rathbun-grubb, 2014). the study is a well-motivated project with an extensive literature review. the research question is somewhat broad and does not explicitly mention facets, although this is a major aspect of the study. the methodology is well described and appropriate to the analyses. for the most part, the analyses are clear, but the article might have benefited from some screenshots of the search platforms, especially for readers who are less familiar with them. some of the statistical results were presented without clear explanation. for example, it seems like pearson correlations were used which requires interval or ratio level data, but it is difficult to interpret variables like “use of the scholarly facet” or “clicking on a suggested article” as meeting that criterion without more details (pp. 5–6). other limitations of the study, such as the convenience sampling and recruiting of a narrow subset of students, are acknowledged and discussed.   this study is laudable for laying out concrete and refreshingly clear advice on how librarians should customize search tools to increase the use of facets and other advanced features: the higher up and more visible, the better. salience in interface design is usually taken as common sense, but it can sometimes (rather ironically) get buried in the practitioner literature. the careful planning, execution, and analysis of this study is to be admired, but it also raises the question of whether future studies could achieve a similar level of thoroughness using more automated means. could some of this data be captured programmatically using browser plugins or server logs rather than screencast software, from which data extraction is extremely labour-intensive? if so, this might encourage more librarians to do this much needed work.   references   bloom, b., & deyrup, m. m. (2015). the shu research logs: student online search behaviors trans-scripted. journal of academic librarianship, 41(5), 593–601. https://doi.org/10.1016/j.acalib.2015.07.002   perryman, c., & rathbun-grubb, s. (2014). the cat: a generic critical appraisal tool. in jotform – form builder. retrieved from https://www.jotform.us/cp1757/thecat research article   age as a predictor of burnout in russian public librarians   nikita kolachev research assistant international laboratory of positive psychology of personality and motivation national research university higher school of economics moscow, russian federation email: nkolachev@hse.ru   igor novikov scientific secretary moscow governorate universal library moscow, russian federation email: novikov@gumbo.ru   received: 20 mar. 2019                                                              accepted: 17 sep. 2020      2020 kolachev and novikov. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29753     abstract   objective – increasing life expectancy leads to an increase in the mean age of the workforce. the aging workforce implies new challenges for management and human resources. existing findings on relations between age and burnout are controversial and scarce. also, the problem of burnout amongst library workers in russia has received little attention from researchers.   methods – the studied sample consisted of 620 public librarians from 166 public libraries of different regions (the moscow region, yaroslavl, chelyabinsk, novosibirsk, astrakhan, and republic of buryatia) of the russian federation, who completed a self-reported online survey. for measuring burnout, a new burnout assessment tool was implemented. to examine the associations of interest, we used structural equation modeling with a group correction approach. in addition, library location, general self-efficacy, and length of employment at the current workplace were utilized as predictors. all statistical analysis was performed in r.     results – findings confirmed the hypotheses partially and revealed negative links between exhaustion, mental distance, and cognitive control and age, while reduced emotional control did not relate to age. urban librarians tended to demonstrate higher levels of mental distance and had more significant problems with emotional regulation than their rural counterparts. also, the non-moscow region librarians did not demonstrate correlations between age and reduced cognitive control. moreover, they showed a positive link between age and reduced emotional control.    conclusion – the current paper confirmed some previous results on the negative relations between burnout symptoms and chronological age. the results suggest the existence of higher risks of burnout for younger library workers. potential mechanisms underlying the resilience of older workers are discussed.     introduction   increasing life expectancy leads to an increase in the mean age of the workforce (adams & shultz, 2018). the aging workforce implies new challenges for management and human resources. according to a canadian report of 2018, for one employee of 25 to 34 years, there was one employee aged 55 and older (statistics canada, 2019). a similar situation is observed in the european union. according to eurostat (2020), in the first quarter of 2012, there were 13% of employees aged 55-64; in the first quarter of 2015, there were 15% of employees aged 55-64; and in the first quarter of 2019, there were 17% of employees aged 55-64. librarianship is not an exception. according to wilder’s (2018) findings, the average age of employees of the association of research libraries increased by 2015 as the proportion of workers aged 60 and over grew. in this sample, the mean age was 49 years. unfortunately, in the russian federation, there are no appropriate library statistics; however, the situation seems analogous. according to a statistical report from the russian state library, the mean age of its personnel in 2019 was 48.90 years (russian state library, 2020). in a recent study of librarians of the moscow region, the participants’ mean age was 48.05 (kolachev, osin, schaufeli, & desart, 2019).   burnout amongst russian library workers has received little attention from researchers. librarians are not considered a part of the more socially important professions, like teachers, nurses, physicians, and social workers. however, librarianship belongs to the human services, where short-term contacts with clients are the primary source of stress (salyers et al., 2019). some researchers refer to librarianship as a helping profession, in which assistance to those who are staying in need, the frequency of such requirements, and the limitation of available resources often lead to stress (smith & nelson, 1983). in addition to general burnout factors (i.e., gender, age, personality, locus of control, expectations) and organizational factors (i.e., excessive workload, underemployment, employee conflict, role conflict), mccormack and cotter (2013) mentioned a specific stress factor for library workers: boredom with the routine nature of library work and little intellectual stimulation. in addition to the harm of burnout for librarians themselves, it can degrade the quality of services provided, thereby affecting the satisfaction of library visitors.   literature review   a crucial problem that influences an employee’s performance is work-related stress and its severe form: burnout (penz et al., 2018). to date, the most common definition of the term “burnout” is maslach, schaufeli, and leiter's (2001) interpretation: burnout is a state of physical and psychological exhaustion, which develops as a reaction to stressful long-term working conditions. according to the authors, burnout consists of three separate but interrelated constructs: emotional exhaustion, cynicism/depersonalization, and lack of accomplishment (inefficacy). emotional exhaustion is the most common symptom of burnout and is an emotional and physical sensation of exhaustion from excessive workload. cynicism implies an excessively detached attitude to various aspects of work. lack of accomplishment refers to a sense of incompetence and reduced production of labour. the model has been common in various studies for almost 30 years, but it is not quite up-to-date with current understandings of burnout syndrome, since we know that burnout also links to emotional and cognitive impairments (deligkaris, panagopoulou, montgomery, & masoura, 2014).   in a recently developed model by schaufeli, de witte, and desart (2019), burnout syndrome is characterized by extreme fatigue, reduced ability to regulate cognitive and emotional processes, as well as detachment in solving problems, depressed mood, as well as non-specific psychological and psychosomatic symptoms. according to the authors, this is developed by an imbalance between high job demands and low levels of organizational resources (for reviewing the job demands-resources model, xanthopoulou, bakker, demerouti, & schaufeli, 2007). primary symptoms include emotional exhaustion, mental distance (the same as cynicism/depersonalization in maslach et al.’s model), reduced emotional control, and reduced cognitive control. emotional exhaustion refers to a feeling of either physical or mental exhaustion, or lack of energy. mental distance is about aversion to work, such as avoiding contact with others at work. reduced emotional control (emotional impairment) includes irritability and emotional overreacting. reduced cognitive control (cognitive impairment) supposes attention and memory problems such as forgetfulness or concentration deficits.     in the field of burnout research, much attention has been paid to the dispositional and organizational factors that associate with this syndrome (bakker & demerouti, 2007; bianchi, 2018; mcmanus, keeling, & paice, 2004). in some studies, age negatively predicted burnout symptoms such as exhaustion and cynicism/depersonalization (maslach et al., 2001; rutledge & francis, 2004); however, it was usually used as a control variable, and a limited number of researchers found that it played a role.   existing scientific works on the connections between age and burnout emphasize that younger employees are more prone to experience burnout (randall, 2007). in a representative sample of the finnish population, ahola, honkonen, virtanen, aromaa, and lönnqvist (2008) confirmed that the negative association between age and burnous is solely attributed to a subsample of young female persons. in recent research, marchand, blanc, and beauregard (2018) showed that age non-linearly relates to burnout and its components. in particular, there was a positive connection between age and either burnout or exhaustion until the age of 30, then a negative one until the age of 55 (quadratic polynomial), and then again, a positive pattern (cubic polynomial). at the same time, cynicism and a lack of personal accomplishment negatively linked to age. recent research conducted of librarians of the moscow region showed that age was a significant predictor of general burnout, pointing out that younger workers are more prone to experience burnout (kolachev et al., 2019).   results concerning exhaustion and cynicism/depersonalization are quite clear. recently discovered symptoms (reduced emotional control and reduced cognitive control) of burnout are of interest. according to johnson, machowski, holdsworth, kern, and zapf's (2017) findings, age predicts less burnout because of emotion regulation strategies. this means that older workers who have more emotional experience apply effective coping strategies against burnout; moreover, they pay more attention to emotional states than their younger counterparts. many other studies support the idea of higher emotional control in older people (doerwald, scheibe, zacher, & van yperen, 2016; mauno, ruokolainen, & kinnunen, 2013). one of the explanations for this phenomenon is that older people demonstrate a higher motivation to avoid negative situations and try to enjoy life more, as they realize the finiteness of existence (johnson et al., 2017). so, we may expect that older workers are more effective when dealing with emotional disturbances associated with burnout.   another factor that enters into this picture is the role of cognitive abilities, and it is well known that cognitive abilities decline with aging. theorists admit that more severe changes occur in attention since performance on complex attentional tasks is worse in older people (murman, 2015). moreover, so-called fluid cognitive functions, such as processing speed and reasoning, also diminish with aging (deary et al., 2009). burnout does not improve cognitive functions either. in their systematic review, deligkaris et al. (2014) revealed that burnout primarily links to problems with executive functions (working memory, inhibitory control, and task switching). primarily, respondents with lower levels of burnout perform better on n-back and stroop tasks (diestel, cosmar, & schmidt, 2013); in other words, people with higher levels of burnout demonstrate less inhibitory control and working memory capacity. therefore, we may predict that age positively correlates with reduced cognitive control in burnout, since the older respondents are, the more cognitive deficits they have.   aims   in this study, we aimed to examine whether linear patterns between age and burnout symptoms are observed in librarians. in addition, we were interested in revealing whether the new constructs of reduced emotional and reduced cognitive control link to age in a manner similar to exhaustion and distance (depersonalization).   based on the literature review, we hypothesized:   ·        h1 – younger workers experience higher levels of exhaustion and mental distance; ·        h2 – younger librarians demonstrate lower emotional control than their older counterparts; ·        h3 – older librarians demonstrate lower cognitive control than their younger counterparts.   methods   context   according to the federalniy zakon №78 [federal law №78] of 1994, the library system in russia is represented by the following types of libraries: national, federal, regional, municipal, research, university, organizational, private, and funded by citizen groups. the four former libraries constitute the system of public libraries (zverevich, 2014). there are 41,821 public libraries; amongst them, 79% are rural libraries, and 21% are urban (national library of russia, 2020). the number of registered users who visit a library 9.4 times per year was 43,371,700 persons in 2018 (main information and computing center of the ministry of culture of russia, 2018), which is equal to approximately 30% of the total population (federal state statistics service, 2019).     table 1 descriptive statistics of the demographic variables   m/% sd age 47.36 11.91 gender (% of females) 95% education (% of those who have a higher education degree) 76% length of employment 15.09 12.47 library location (% of participants from urban areas) 61%     participants   initially, we sent invitation letters to library directors across the country. we got responses from 600 libraries. then we took a random sample of 305 libraries and sent the link to the questionnaire. the invitation to participate was sent to a library director who distributed the survey to all staff in the library. in total, 620 librarians from 166 public libraries of different regions of the russian federation completed the survey (response rate at the library level = 54%). sixty-six percent of the participants were from libraries in the moscow region (central russia), 8% from novosibirsk (siberia), 7% from chelyabinsk (ural), 12% from yaroslavl (central russia), 4% from astrakhan (southern russia), and 3% from the republic of buryatia (siberia).   participants were reached by email. every library had a unique link to the online questionnaire created on the 1ka.si survey platform. participation was entirely voluntary and did not involve any financial reward. respondents were informed that, by completing the survey, they were giving consent to their inclusion in the study. all ethical standards have been followed.   table 1 demonstrates the descriptive statistics of the sample. the mean participants’ age is 47.36 (the median age is 49); the standard deviation equals to 11.91; range: 17-72. the majority of the sample consists of females (95%). most respondents have a higher education degree (76%). usually, the average librarian has a bachelor’s degree; it is rare to have a master’s or a doctoral degree among russian librarians. in our sample, only 3% of the participants have a master’s or doctoral degree. the average length of employment at the current workplace equals 15.09 years (standard deviation = 12.47). sixty-one percent of the participants are from urban libraries, and 39% are from rural.   measures   for measuring burnout, the russian version of the burnout assessment tool (schaufeli et al., 2019) was used. it includes four subscales: exhaustion (cronbach’s α = .89, mcdonald’s ω = .89), distance (cronbach’s α = .76, mcdonald’s ω = .77), reduced emotional control (cronbach’s α = .84, mcdonald’s ω = .85), and reduced cognitive control (cronbach’s α = .85, mcdonald’s ω = .85). the response involves the five-point scale: 1 = never, 5 = always. the russian version of the burnout instrument was validated on librarians of the moscow region; the factorial validity (using confirmatory factor analysis framework), convergent validity (correlations with optimism, hardiness, and self-efficacy), and content validity were confirmed (kolachev et al., 2019).    age was measured in years. we also included length of employment (in years) and type of library (0 = rural, 1 = urban). length of employment is an important predictor of burnout because less experienced employees tend to burn out more (dimunová & nagyová, 2012; maslach et al., 2001). the location of the library could be important because, in urban libraries, there are more visitors than in rural ones; it produces more stress factors, which leads to higher levels of burnout. for instance, according to saijo et al. (2013), urban hospital physicians experience higher levels of burnout than rural hospital physicians.   additionally, we used the general self-efficacy scale (schwarzer & jerusalem, 1995). it is one of the determinants of stress-related outcomes (shoji et al., 2016). moreover, self-efficacy is a personal resource whose lower levels connect with higher levels of burnout (luthans, avolio, avey, & norman, 2007). the instrument contains 10 items with the four-point response scale: 1 not at all true, 4 exactly true (cronbach’s α = .91, mcdonald’s ω = .91). the scale was validated in many countries (scholz, doña, sud, & schwarzer, 2002): the authors demonstrated factorial validity (using confirmatory factor analysis framework), concurrent validity (correlations with optimism, anxiety, social support), and measurement invariance (also using confirmatory factor analysis framework).   gender and education were not included in the model due to little variation in these variables.   data analysis plan   first, we presented bivariate correlations to examine how the variables are interrelated, not accounting for the effects of the other variables. it is also essential to demonstrate that the predictors are independent (not highly correlated) to be included in the structural model.   as a preliminary analysis, we performed confirmatory factor analysis corrected for the clustered nature of the data to examine the factor structure of the burnout measurement model before the structural model was tested. the cluster correction is needed when the observations are nested within clusters (in our case, librarians are nested within libraries). observations within clusters are more similar than between them (hox, moerbeek, & van de schoot, 2010). this implies that our observations are not independent, which requires correction for non-independence.   the maximum likelihood estimation with robust standard errors and a satorra-bentler scaled test statistic (mlm) was used, which is appropriate for five-point rating scales (rhemtulla et al., 2012).   for observing connections of interest, structural equation modeling with cluster correction was used. structural equation modeling is a useful technique because it estimates all parameters simultaneously, including latent variables, and provides fit indices (kline, 2011). structural equation modeling incorporates measurement errors so that researchers can get unbiased estimates of the effects between predictors and outcomes (bollen & hoyle, 2012). structural variables included age, length of employment at the current workplace, library location, and self-efficacy variables.   all statistical analysis was performed in r version 3.6.1 (r core team, 2016) using such packages as lavaan (measurement and structural models; rosseel, 2012), lavaan.survey (cluster correction for the measurement and structural models; oberski, 2014), and sjplot (correlational matrix; lüdecke, 2018; wickham, 2016).   the data described in this article are openly available in csv format in the open science framework at https://osf.io/m7nwk/.   results   for the confirmatory factor analysis, the following indicators are the quality criteria: rmsea < .08, cfi and tli > .90, srmr < .08 (kline, 2011). the model with four first-order factors (exhaustion, distance, reduced emotional control, and reduced cognitive control) was fitted; it was identified through fixing the variance of the latent variables to 1. the results are the following: χ2 (224, n = 620) = 418.25, scaling factor = 1.31, cfi = .96, tli = .96, rmsea = .04 90% ci [.04;.05], srmr = .04. all factor loadings, except one, exceeded .60 and were significant. therefore, the measurement model demonstrated a good fit.     table 2 bivariate correlations of the variables of interest variable  1 2 3 4 5 6 7 1. exhaustion     2. distance .65***             3. reduced emotional control .64*** .55***           4. reduced cognitive control .59*** .63*** .61***         5. age -.18*** -.17*** -.01 -.14***       6. length of employment -.08* -.07 -.00 -.12** .60***     7. library location (0 = rural, 1 = urban) .08* .13** .10* .03 -.05 -.08   8. self-efficacy -.43*** -.42*** -.39*** -.46*** -.05 -.06 .01 note. computed correlation used the spearman method. * p < .05. ** p < .01. *** p < .001.     figure 1 structural equation model of relations between age, length of employment, library location, self-efficacy, and the factors of burnout (standardized solution; n = 620).     table 2 depicts the bivariate correlations between variables tested in the structural model. we can see that burnout dimensions are highly interrelated and negatively correlate with self-efficacy. age significantly links to exhaustion, distance, and reduced cognitive control. length of employment correlates significantly with exhaustion, reduced cognitive control, and age. library location relates to exhaustion, distance, and reduced emotional control. also, age, length of employment, and library location do not correlate with self-efficacy. structural model    figure 1 displays the paths and coefficients of the tested model. first, authors tested the structural model of the links between age and burnout symptoms controlling for length of employment, library location, and self-efficacy.   table 3 depicts standardized and unstandardized regression coefficients. unstandardized coefficients are estimates of relationships in real units of measurement. standardized coefficients are the same as the correlation coefficients. table 3 shows that the model fits the data well because cfi and tli > .90, rmsea < .08, and srmr < .08 (see note in table 3). among all predictors, self-efficacy significantly predicts each of the burnout dimensions, controlling for age, length of employment, and library location. age negatively links to exhaustion and distance. also, age predicts significantly reduced cognitive control. as in the bivariate correlations demonstrated, age does not relate to reduced emotional control. length of employment does not predict any of the burnout factors. there is a difference in distance and reduced emotional control between rural and urban library workers: those who work at urban libraries have higher levels of distance and a lower level of emotional control. also, respondents from urban libraries demonstrate higher levels of distance and greater problems with emotional control compared to rural library workers. although these differences between urban and rural librarians are small. predictors explained 27% of the variance in exhaustion, 26% of the variance in distance, 17% of the variance in reduced emotional control, and 29% of the variance in reduced cognitive control dispersion.   also, we conducted an additional correlational analysis on the non-moscow region part of the sample. we found that exhaustion correlates negatively with age (r = -.16, p = .02); there is a negative correlation between age and distance (r = -.14, p = .04); age and impaired emotional control are linked positively (r = .15, p = .03); there is no significant correlation between age and impaired cognitive control (r = .01, p = .93).   discussion   in the present study, the main aim was to examine how age relates to burnout components such as exhaustion, distance, reduced emotional control, and reduced cognitive control in a librarian sample. as predicted (h1), age linearly and negatively links to exhaustion and distance. contrary to the predictions (h2), age does not relate to reduced emotional control. not in line with our expectations (h3), younger workers reported greater problems with cognitive control at work compared to their older colleagues. the length of employment did not predict any of the burnout factors. in addition, there is a difference between rural and urban library workers in mental distance and reduced emotional control. urban librarians tend to demonstrate higher levels of distance and have more reduced emotional regulation than their rural counterparts.   age more strongly predicts exhaustion; therefore, younger employees are more likely to experience emotional exhaustion. younger librarians are also more prone to distance themselves from work than their older counterparts. these results are in correspondence with salyers et al. (2019), who in a sample of librarians of the indiana state library, found that emotional exhaustion and cynicism were related negatively to burnout: r = -.19 and r = -.15, respectively. in our sample, the correlation of exhaustion and age was -.18, between distance (the same as cynicism) and age was -.17. wood, guimaraes, holm, hayes, and brooks (2020), in a sample of 1,628 academic librarians employed within the united states, found that age was related to burnout significantly and negatively. however, martini, viotti, converso, battaglia, and loera (2019), in a sample of 167 italian public library workers, found that controlling for job demands, job resources, and some demographic variables age was linked to exhaustion positively while the link between age and cynicism was insignificant.     table 3 unstandardized, standardized coefficients, and significance levels for the paths of the structural equation model (standard errors in parentheses; n = 620) predictor -> outcome unstandardized coefficients standardized coefficients p age → exhaustion -0.02 (0.005) -.24 .00 age → distance -0.02 (0.006) -.22 .00 age → reduced emotional control 0.00 (0.005) .02 .69 age → reduced cognitive control -0.01 (0.006) -.13 .03 length of employment → exhaustion 0.00 (0.004) .04 .38 length of employment → distance 0.005 (0.003) .05 .14 length of employment → reduced emotional control 0.00 (0.004) -.00 .93 length of employment → reduced cognitive control -0.005 (0.005) -.05 .33 library location → exhaustion 0.20 (0.14) .09 .14 library location → distance 0.38 (0.11) .16 .00 library location → reduced emotional control 0.26 (0.12) .11 .03 library location → reduced cognitive control 0.09 (0.15) .04 .54 self-efficacy → exhaustion -0.53 (0.06) -.46 .00 self-efficacy → distance -0.51 (0.06) -.44 .00 self-efficacy → reduced emotional control -0.44 (0.05) -.40 .00 self-efficacy → reduced cognitive control -0.61 (0.06) -.51 .00 note. χ2(572) = 1070.38, p < .001, scaling factor = 1.23; cfi = .94; tli = .93; rmsea = .04, 90% ci [.04, .05]; srmr = .04.     also, our results overlap marchand et al.’s (2018) findings partially. in canadian employees of the private sector, they found that age was linearly and negatively linked to cynicism (b = -0.12, 95% ci [-0.17, -0.07]). however, the authors found that age was non-linearly related to exhaustion (cubic predictor was significant). moreover, they revealed that the relations are different for males and females, demonstrating non-linear pattern with exhaustion and cynicism in women. in men, associations were linear. a similar non-linear pattern of results in women was obtained by ahola et al. (2008). probably, their results are attributable to the higher heterogeneity in terms of different professions of the representative sample. instead, brewer and shapard (2004), in a meta-analysis dedicated to employees’ burnout, found that the mean correlation between age and exhaustion was -.18, corrected for heterogeneity of the sample equaled to -.23. our results confirm linear relations of exhaustion, distance (the same as cynicism), and impaired cognitive control with age.   the only study of russian librarians dedicated to burnout showed linear relations between age and composite burnout accounting for gender, length of employment, personal resources, and library location (kolachev et al., 2019). however, in this study, the authors did not pay attention to the relations between age and the four factors of burnout. although our sample partially overlaps the sample in kolachev et al. (2019) in terms of regions, our data include librarians from other regions. moreover, our correlational analysis conducted in librarians from regions other than moscow revealed that there are no differences between librarians from the moscow region and librarians from other regions in our sample in the correlation pattern of age with exhaustion and distance. however, there is a difference in relations with reduced emotional control: older librarians not from the moscow region tend to report more problems with emotional control than their younger colleagues. also, in non-moscow region librarians, there is no age difference in reduced cognitive control. this is another characteristic that differentiates the librarians of the moscow region.   concerning reduced emotional control, participants of different ages reported similar levels of emotional regulation. the relationship between emotion, age, and burnout appears to be complex. several researchers suggest that older adults are more likely to suppress affect and inhibit emotional responses due to increased cognitive complexity (mcconatha & huba, 1999; orgeta, 2009). however, modern studies mention the role of culture in emotional expression. sheldon et al. (2017) claimed that russian people tend to inhibit their emotions no matter whom they encounter – themselves or their countrymen. russians are more emotionally self-distanced than their western counterparts and are less prone to reflect in the case of experiencing negative situations (grossmann & kross, 2010). according to the results obtained on the non-moscow region part of our sample, older workers are more prone to experience problems with emotional control. this could mean that for this population, the idea that older workers are more successful in emotional regulation is applicable, but it requires further investigation. these results contradict mauno et al.’s (2013) conclusions that older workers have better emotional regulation in relation to negative feelings.   regarding the relations of reduced cognitive control and age, it can be assumed that, according to socioemotional selectivity theory, older people may pay little attention to negative stimuli (martins, florjanczyk, jackson, gatz, & mather, 2018). at the same time, younger respondents tend to focus on negative stimuli that distract them during the work. the increased engagement with negative information leads to more problems with cognitive control in younger workers. research conducted by martins, sheppes, gross, and mather (2016) confirmed that older people become less distracted when exposed to positive stimuli and more distracted when exposed to negative ones. the non-moscow region part of our sample demonstrates an absence of the significant link between age and reduced cognitive control. potentially it could be another specific feature of the non-moscow region librarians and requires more empirical evidence.   comparing the present results with those obtained from studies of other professions demonstrates that the relationship between age and burnout is complex and may be situation-dependent. for instance, thomas, kohli, and choi (2014), on a sample of californian human service workers, found that when controlling for education and caseload size, age was a positive predictor of burnout (the standardized regression coefficient was .18) while years of experience did not link to burnout. however, chou, li, and hu (2014), on a sample of the medical staff of a hospital in taiwan, revealed that older employees were less prone to experience burnout. therefore, it would appear that the relationship between burnout and age may differ by profession.   practical implications   the present findings have some practical implications. it is of use to implement burnout screening in personnel so that supervisors could propose some psychological or even medical assistance to those who are at risk of burnout. there are national and european laws and regulations that oblige employers to assess psychosocial risks among their employees periodically and to implement policies to prevent burnout and work stress (schaufeli et al., 2019). in several european countries, burnout is acknowledged as an occupational disease or work-related disorder, and there is some compensation for workers included in social insurance (lastovkova et al., 2017). in the context of librarianship, two directions can be distinguished. first, it is essential to organize an annual program for monitoring employees’ emotional state in order to prevent attrition or reduced job performance. in the absence of a specialized hr department, this program can be implemented by library methodologists. second, if alarming indicators of stress or burnout are found, job rotation might be applied. for example, a service department employee could work in the acquisition department. caputo (1991) claims that librarians would appreciate a work rotation that equitably distributes unpopular tasks, such as a particularly heavy reference shift or equipment service calls. also, if possible, it could be beneficial to give some professional and psychological guidance to newcomers so that they adapt successfully to working conditions. for instance, as smith, bazalar, and wheeler (2020) point out, “pre-service librarians might shadow a public librarian who works at the reference desk or a staff member with administrative duties to get a first-hand glimpse into how to navigate job duties” (p. 426-7).   limitations the current research is not free of several limitations. first, the study included a voluntary sample of participants, which means that the results may reflect self-selection bias. this may lead to another explanation of the results by the phenomenon of the survivor’s bias, since if older employees experience lower levels of burnout, this may mean that only the most persistent and resilient ones have remained at work (maslach et al., 2001). second, there are many potential explanations of the results due to the absence of control for cohort differences. cohort differences may reflect different work attitudes and values. finally, results may not be generalizable; as data mostly came from libraries in the central region of the russian federation, these librarians may differ from the rest of the country, as we noticed differences concerning reduced emotional and reduced cognitive control.      conclusion   despite the limitations mentioned above, the current paper confirms some previous results on negative relations between burnout symptoms and chronological age. this study is a first attempt to scrutinize burnout in librarians using the instrument based on a new burnout framework that better describes the phenomenon. the results obtained indicate that young employees are at risk not only for exhaustion and depersonalization at work but also for problems with cognitive functions, in particular with attention. they may need psychological help not only in terms of rest to diminish exhaustion and mental distance but a strengthening of attentional skills. also, in relation to the aging workforce problem, the current study proposes a new challenge for the management and human resources fields in librarianship: how to help young workers experience less burnout or avoid it altogether? if we account for self-reporting bias, it is of practical importance to maintain and boost subjective well-being in younger employees. moreover, the resilient potential of older workers remains unclear and requires further investigation.   references   adams, g. a. & shultz, k. s. (2018). introduction and overview. in g.a. adams & k. s. shultz (eds.), aging and work in the 21st century (pp. 34-58). routledge. https://doi.org/10.4324/9781315167602-1   ahola, k., honkonen, t., virtanen, m., aromaa, a., & lönnqvist, j. (2008). burnout in relation to age in the adult working population. journal of occupational health, 50(4), 362-365. https://doi.org/10.1539/joh.m8002   bakker, a. b., & demerouti, e. (2007). the job demands-resources model: state of the art. journal of managerial psychology, 22(3), 309–328. https://doi.org/10.1108/02683940710733115   bianchi r. (2018). burnout is more strongly linked to neuroticism than to work-contextualized factors. psychiatry research, 270, 901-905. https://doi.org/10.1016/j.psychres.2018.11.015   bollen, k. a., & hoyle, r. h. (2012). latent variables in structural equation modeling. in r. h. hoyle (ed.), handbook of structural equation modeling (p. 56-67). the guilford press.     brewer, e. w., & shapard, l. (2004). employee burnout: a meta-analysis of the relationship between age or years of experience. human resource development review, 3(2), 102-123. https://doi.org/10.1177/1534484304263335   caputo, j. s. (1991). stress and burnout in library service. oryx press.   chou, l. p., li, c. y., & hu, s. c. (2014). job stress and burnout in hospital employees: comparisons of different medical professions in a regional hospital in taiwan. b.m.j. open, 4(2). https://doi.org/10.1136/bmjopen-2013-004185      deary, i.j., corley, j., gow, a.j., harris, s.e., houlihan, l.m., marioni, r.e., penke, l., rafnsson, s.b., & starr, j.m. (2009). age-associated cognitive decline. british medical bulletin, 92(1), 135-152. https://doi.org/10.1093/bmb/ldp033   deligkaris, p., panagopoulou, e., montgomery, a. j., & masoura, e. (2014). job burnout and cognitive functioning: a systematic review. work & stress, 28(2), 107–123.   diestel, s., cosmar, m., & schmidt, k. h. (2013). burnout and impaired cognitive functioning: the role of executive control in the performance of cognitive tasks. work & stress, 27(2), 164-180. https://doi.org/10.1080/02678373.2013.790243   dimunová, l., & nagyová, i. (2012). the relationship between burnout and the length of work experience in nurses and midwives in the slovak republic. profese online, 5(1), 1-4. https://doi.org/10.5507/pol.2012.001      doerwald, f., scheibe, s., zacher, h., & van yperen, n. w. (2016). emotional competencies across adulthood: state of knowledge and implications for the work context. work, aging and retirement, 2(2), 159-216. https://doi.org/10.1093/workar/waw013   eurostat. (2020). employees by sex, age and occupation (1,000). retrieved from https://ec.europa.eu/eurostat/databrowser/view/lfsq_eegais/default/table?lang=en   federal state statistics service. (2019). chislennost' postoyannogo naseleniya na 1 yanvarya [total population number as of january 1]. retrieved from https://showdata.gks.ru/report/278928/   federalniy zakon №78 “o bibliotechnom dele” [federal law №78 “on librarianship”]. (1994 & supp. 2019). retrieved from http://www.consultant.ru/document/cons_doc_law_5434/c..   grossmann, i., & kross, e. (2010). the impact of culture on adaptive versus maladaptive self-reflection. psychological science, 21(8), 1150-1157. https://doi.org/10.1177/0956797610376655   hox, j. j., moerbeek, m., & van de schoot, r. (2010). multilevel analysis: techniques and applications. routledge.     johnson, s. j., machowski, s., holdsworth, l., kern, m., & zapf, d. (2017). age, emotion regulation strategies, burnout, and engagement in the service sector: advantages of older workers. journal of work and organizational psychology, 33(3), 205-216. https://doi.org/10.1016/j.rpto.2017.09.001   kline, r. b. (2011). principles and practice of structural equation modeling (3rd ed.). guilford press.   kolachev, n., osin., e., schaufeli, w., & desart, s. (2019). personal resources and burnout: evidence from a study among librarians of moscow region. organizational psychology, 9(2), 129-147.   lastovkova, a., carder, m., rasmussen, h. m., sjoberg, l., de groene, g. j., sauni, r., vevoda, j., vevodova, s., lasfargues, g., svartengren, m., varga, m., colosio, c., & pelclova, d. (2018). burnout syndrome as an occupational disease in the european union: an exploratory study. industrial health, 56(2), 160-165. https://doi.org/10.2486/indhealth.2017-0132   lüdecke, d. (2018). sjplot: data visualization for statistics in social science. retrieved from https://cran.r-project.org/package=sjplot   luthans, f., avolio, b. j., avey, j. b., & norman, s. m. (2007). positive psychological capital: measurement and relationship with performance and satisfaction. personnel psychology, 60(3), 541-572. https://doi.org/10.1111/j.1744-6570.2007.00083.x      marchand, a., blanc, m. e., & beauregard, n. (2018). do age and gender contribute to workers’ burnout symptoms? occupational medicine, 68(6), 405-411. https://doi.org/10.1093/occmed/kqy088   martini, m., viotti, s., converso, d., battaglia, j., & loera, b. (2019). when social support by patrons protects against burnout: a study among italian public library workers. journal of librarianship and information science, 51(4), 1091-1102. https://doi.org/10.1177/0961000618763716   martins, b., florjanczyk, j., jackson, n. j., gatz, m., & mather, m. (2018). age differences in emotion regulation effort: pupil response distinguishes reappraisal and distraction for older but not younger adults. psychology and aging, 33(2), 338-349. https://doi.org/10.1037/pag0000227   martins, b., sheppes, g., gross, j. j., & mather, m. (2016). age differences in emotion regulation choice: older adults use distraction less than younger adults in high-intensity positive contexts. the journals of gerontology: series b, 73(4), 603-611. https://doi.org/10.1093/geronb/gbw028   maslach, c., schaufeli, w. b., & leiter, m. p. (2001). job burnout. annual review of psychology, 52(1), 397-422. https://doi.org/10.1146/annurev.psych.52.1.397   mauno, s., ruokolainen, m., & kinnunen, u. (2013). does aging make employees more resilient to job stress? age as a moderator in the job stressor–well-being relationship in three finnish occupational samples. aging & mental health, 17(4), 411-422. https://doi.org/10.1080/13607863.2012.747077   mcconatha, j. t., & huba, h. m. (1999). primary, secondary, and emotional control across adulthood. current psychology, 18(2), 164-170. https://doi.org/10.1007/s12144-999-1025-z   mccormack, n., & cotter, c. (2013). managing burnout in the workplace: a guide for information professionals. elsevier. https://doi.org/10.1016/b978-1-84334-734-7.50010-4   mcmanus, i. c., keeling, a., & paice, e. (2004). stress, burnout and doctors’ attitudes to work are determined by personality and learning style: a twelve year longitudinal study of uk medical graduates. bmc medicine, 2, 29. https://doi.org/10.1186/1741-7015-2-29   murman, d. l. (2015). the impact of age on cognition. seminars in hearing, 36(3), 111-121.  https://doi.org/10.1055/s-0035-1555115   national library of russia. (2020). yezhegodnyy doklad o sostoyanii seti obshchedostupnykh bibliotek rossiyskoy federatsii: po itogam monitoringa 2019 g. [annual report on the state of the network of public libraries in the russian federation: based on the results of monitoring in 2019]. retrieved from http://clrf.nlr.ru/images/sitedocum/news/2020/ed_set_bibliotek_19.pdf   oberski, d. (2014). lavaan. survey: an r package for complex survey analysis of structural equation models. journal of statistical software, 57(1), 1-27. https://doi.org/10.18637/jss.v057.i01   orgeta, v. (2009). specificity of age differences in emotion regulation. aging and mental health, 13(6), 818-826. https://doi.org/10.1080/13607860902989661   penz, m., wekenborg, m. k., pieper, l., beesdo‐baum, k., walther, a., miller, r., stadler, t., & kirschbaum, c. (2018). the dresden burnout study: protocol of a prospective cohort study for the bio‐psychological investigation of burnout. international journal of methods in psychiatric research, 27(2), e1613. https://doi.org/10.1002/mpr.1613   r core team. (2016). r: a language and environment for statistical computing. retrieved from https://www.r-project.org/   randall, k. j. (2007). examining the relationship between burnout and age among anglican clergy in england and wales. mental health, religion & culture, 10(1), 39-46. https://doi.org/10.1080/13674670601012303   rhemtulla, m., brosseau-liard, p. é., savalei, v. (2012). when can categorical variables be treated as continuous? a comparison of robust continuous and categorical sem estimation methods under suboptimal conditions. psychological methods, 17(3), 354-373. https://doi.org/10.1037/a0029315   rosseel, y. (2012). lavaan: an r package for structural equation modeling. journal of statistical software, 48(2), 1-36. http://dx.doi.org/10.18637/jss.v048.i02   russian state library. (2020). kratkaya statisticheskaya spravka o rossiyskoy gosudarstvennoy biblioteke v 2019 godu (po sostoyaniyu na 01.01.2020) [brief statistical information about the russian state library in 2019 (as of 01.01.2020). retrieved from https://www.rsl.ru/ru/about/documents/statisticheskaya-spravka-o-rabote-rgb/spravka-2019   saijo, y., chiba, s., yoshioka, e., kawanishi, y., nakagi, y., ito, t., sugioka, y., kitaoka-higashiguchi, k., & yoshida, t. (2013). job stress and burnout among urban and rural hospital physicians in japan. the australian journal of rural health, 21(4), 225-231. https://doi.org/10.1111/ajr.12040      salyers, m. p., watkins, m. a., painter, a., snajdr, e. a., gilmer, l. o., garabrant, j. m., & henry, n. h. (2019). predictors of burnout in public library employees. journal of librarianship and information science, 51(4), 974-983. https://doi.org/10.1177/0961000618759415   schaufeli, w.b., de witte, h. & desart, s. (2019). manual burnout assessment tool (bat). ku leuven, belgium: unpublished internal report.   scholz, u., doña, b. g., sud, s., & schwarzer, r. (2002). is general self-efficacy a universal construct? psychometric findings from 25 countries. european journal of psychological assessment, 18(3), 242-251. https://doi.org/10.1027/1015-5759.18.3.242   schwarzer, r., & jerusalem, m. (1995). generalized self-efficacy scale. in j. weinman, s. wright, & m. johnston (eds.), measures in health psychology: a user’s portfolio. (pp. 35-37). nfer-nelson.   sheldon, k. m., titova, l., gordeeva, t. o., osin, e. n., lyubomirsky, s., & bogomaz, s. (2017). russians inhibit the expression of happiness to strangers: testing a display rule model. journal of cross-cultural psychology, 48(5), 718-733. https://doi.org/10.1177/0022022117699883   shoji, k., cieslak, r., smoktunowicz, e., rogala, a., benight, c. c., & luszczynska, a. (2016). associations between job burnout and self-efficacy: a meta-analysis. anxiety, stress, & coping, 29(4), 367-386. https://doi.org/10.1080/10615806.2015.1058369   smith, n. m., & nelson, v. c. (1983). burnout: a survey of academic reference librarians (research note). college & research libraries, 44(3), 245-250. https://doi.org/10.5860/crl_44_03_245   smith, d. l., bazalar, b., & wheeler, m. (2020). public librarian job stressors and burnout predictors. journal of library administration, 60(4), 412-429. https://doi.org/10.1080/01930826.2020.1733347   statistics canada. (2019). study: occupations with older workers. retrieved from https://www150.statcan.gc.ca/n1/en/daily-quotidien/190725/dq190725b-eng.pdf?st=ugdkvr71   thomas, m., kohli, v., & choi, j. (2014). correlates of job burnout among human services workers: implications for workforce retention. the journal of sociology and social welfare, 41(4), 69-90. https://scholarworks.wmich.edu/jssw/vol41/iss4/5     wickham, h. (2016). ggplot2: elegant graphics for data analysis (2nd ed.). springer. https://doi.org/10.1007/978-3-319-24277-4_9   wilder, s. (2018). delayed retirements and the youth movement among arl library professionals. research library issues, 295, 6-16. https://doi.org/10.29242/rli.295.2   wood, b. a., guimaraes, a. b., holm, c. e., hayes, s. w., & brooks, k. r. (2020). academic librarian burnout: a survey using the copenhagen burnout inventory (cbi). journal of library administration, 60(5), 512-531. https://doi.org/10.1080/01930826.2020.1729622   xanthopoulou, d., bakker, a. b., demerouti, e., & schaufeli, w. b. (2007). the role of personal resources in the job demands-resources model. international journal of stress management, 14(2), 121-141. https://doi.org/10.1037/1072-5245.14.2.121   zverevich, v. (2014). developing the library network in postcommunist russia: trends, issues, and perspectives. library trends, 63(2), 144-160. https://doi.org/10.1353/lib.2014.0039     evidence based library and information practice commentary   building a home for evidence based library and information practice   denise koufogiannakis collections and acquisitions coordinator university of alberta libraries edmonton, alberta, canada email: denise.koufogiannakis@ualberta.ca   pam ryan director, collections & technology edmonton public library edmonton, alberta, canada email: pryan@epl.ca   lindsay alcock head of public services health sciences library memorial university of newfoundland st. john’s, newfoundland and labrador, canada email: lalcock@mun.ca   susan cleyle director, distance education, learning and teaching support (delts) memorial university of newfoundland st. john’s, newfoundland and labrador, canada email: scleyle@mun.ca   received: 19 jan. 2016   accepted: 19 jan. 2016      2016 koufogiannakis, ryan, alcock, and cleyle. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     a 10th anniversary is a good time for both celebration and reflection. as the group of people who started this journal, we wanted to look back at what we began working on in the spring of 2005 and provide some memories about how the journal came to be, considering our struggles and victories along the way.   the idea to start a journal began with email conversations between denise and susan at the end of april 2005. together with lindsay, we had recently begun the evidence based librarianship interest group (eblig) of the canadian library association (cla), and we had a mutual interest in advancing evidence based practice via more concrete steps. our two main ideas were for the creation of a database of evidence and for an international, open access journal focused on evidence based practice in librarianship. the latter seemed more feasible, and therefore our continued discussions focused on the implementation of a journal. lindsay came on board soon after, and within two months pam joined our small group.   in three months, we went from an idea to having the bones of our journal set up within open journal systems (ojs) hosted by the university of alberta libraries (ual). looking back on it now, the time frame seems quite impossible, given that ual had not yet begun its journal hosting service. in fact, evidence based library and information practice (eblip) kick-started a new service for ual, one that now hosts more than 30 journals using ojs, the open source software from the public knowledge project (pkp).   basically, we dove into the notion that librarianship needed this journal and that we were at the perfect point in time to make it happen. and this turned out to be true. in the spring of 2005, the evidence based library and information practice (eblip) movement had been active for approximately eight years and was gaining momentum. two international conferences had been held, another was being planned, many articles had been published, and a year earlier, andrew booth and anne brice had published their landmark book on the topic. the focus of evidence based librarianship, as it was widely referred to at that time, was still on health sciences libraries. we felt it was important to have a professional forum for evidence based practice across all library sectors, and since no scholarly publishing venue existed, there was a need and opportunity to fill this void.   it was also critically important to all of us that the journal be open access. as librarians, we felt that open access was the way that scholarly publishing had to evolve, and we wanted to be part of that change. more importantly, we felt that evidence based practice could not become a reality unless all information professionals had access to this body of literature. we were all academic librarians at the time with excellent access to the library and information studies (lis) research, but we knew that most librarians were not so fortunate, and this was a good opportunity to help make a difference. fortunately, pkp’s ojs was released in 2001 and had been operational for almost four years by the time we were getting started. likewise, the ual was at the point of being ready to support open access publishing with the infrastructure and in-kind technical support needed to make it happen.   of course, the process wasn’t easy or straightforward, and there certainly were detractors. we initially thought it would be best to partner with cla, under which the eblig had formed; the thought being that this would assist with stability and governance of the journal. however, the cla did not share our excitement regarding the need and value for this journal, so we quickly walked away and decided to go it on our own with the support of the university of alberta, who made a firm and generous commitment at the incubator stage. eblip continues to function under a grassroots model of governance, built and revised by those who are passionate about evidence based practice and the journal. it is 100% maintained by volunteers from around the world. the only money that has ever been spent was $200cad to design the logo and print some postcards, both of which we still use today! with institutional and collaborative support due to initiatives such as ojs, it is certainly possible to build a well-functioning, growing, and successful open access journal at a very low cost.   we engaged the library community from the start by letting people know of our plan via a mailing list and soliciting input on the name of the journal. this prompted much conversation and debate because not everyone thought that such a journal was warranted. the journal was tentatively to be called journal of evidence based librarianship, but andrew booth, well known for his ability come up with a good acronym, challenged this name and suggested evidence based librarianship and information practice, or eblip. carol perryman suggested a slight change to evidence based library & information practice, and by the end of june 2005, we decided upon evidence based library and information practice.   at the third ebl conference in australia on october 19, 2005, we officially launched and promoted the journal, unveiling the journal’s website and putting out calls for peer reviewers and evidence summary team members as well as for papers. our initial team was quite small with the four of us as editors, 10 evidence summary writers, and 20 peer reviewers. in fact, we did not have a copy editor position at the time of our first issue—the editors did that work themselves! the launch of our first issue on march 15, 2006, was a defining moment and a scary one because the need to maintain momentum and continually find quality content became apparent. like any new start-up, our early years were somewhat lean and were carried by the evidence summaries and other regular sections, such as ebl101. recognizing that eblip was not well known in the early days meant that it was not always authors’ first choice for submitting research articles. to solicit submissions, the editors combed through conference programs for promising research presentations and contacted authors directly and invited them to consider eblip.   but how things have changed and for the better! with our 10th anniversary, eblip no longer has any trouble getting research article submissions, and we are able to have regular features through partnerships with relevant conferences. google scholar shows eblip articles as having 1074 citations, 846 of which have been since 2011, a sign of the journal’s growth and acceptance within the lis research community. our volunteer cohort has grown to include an editorial team of 8, an editorial advisory team of 4, 11 copyeditors, over 100 peer reviewers, 21 evidence summary writers, 4 members who provide writing assistance, and 1 person who provides indexing support. more than 4000 readers have registered with the journal, but because we are an open access publication, we know there are many more readers who have not registered. in the one-year period from may 2014 through april 2015, the number of users visiting the site was 56,173, resulting in 73,546 unique sessions and 226,246 page views. our readership is truly international with 35% of visits from the united states, 13% from canada, 11% from the uk, and 7% from australia. the remaining 34% comes from many diverse countries; the countries with the next highest use include india, philippines, malaysia, china, sweden, and nigeria.   between the big moments of deciding to go ahead with a journal, finding the journal’s home, naming our publication, building our core team, and publishing that first issue, a lot of work happened to get things off the ground, including many decisions about how we would run the journal, sections to include, formatting style, layout style, and so much more. we developed guidelines for submissions, evidence summaries, peer reviewers, copyeditors, the roles of the editors, and so on. it was a lot of work. but we all agree that it was worth it. our guidelines for peer review have been adopted and adapted by other journals including partnership: the canadian journal of library and information practice and research and journal of the canadian health libraries association. we certainly do not want to give the impression that that the four of us were the only contributors to all this work. everything we started with has evolved and been made better by numerous other individuals who have been the bedrock of this journal. we could never name everyone, but there are some who deserve special mention for their numerous and long standing contributions over the years, including alison brettle, lorie kloda, katrine mallan, heather pretty, michelle dunaway, and jonathan eldredge. we also want to thank those who have contributed content to the journal, especially in the early days. it is worth remembering that a journal lives or dies based on the quality of its content, and we have been very fortunate. authors, peer reviewers, and editors all contribute to this quality, and eblip is truly a journal that has emerged from a strong community of practice within lis.   thinking back on 10 years of eblip, as founders we are proud of how the journal has grown and adapted with different people stepping up to lead at different points in time. it certainly stands as a measure and model of how a community can initiate and sustain a successful open access journal. while we believed this journal would be successful, the long term success and number of people who see value in the journal have far surpassed our expectations. in order to continue to remain relevant and be successful for at least another 10 years, it is important that the journal continues to exist as a venue for what matters to the wider evidence based practice community and that new voices join and engage in the conversation.   microsoft word news_riwa.doc evidence based library and information practice 2006, 1:4  84 evidence based library and information practice     news      research in the workplace award 2006/2007        © 2006 eblip. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    the research in the workplace award  (riwa)* is a biennial grant that seeks to  fund small lis‐led workplace research  projects.     the award of £3000 gbp/$5900 usd/$6800  cad aims to encourage and support those  new to research.  projects can relate to any  aspect of service provision, development or  theory.  advice is available throughout the  lifetime of your project, which should be  achievable within 12 months.  the award  fund must constitute at least 55% of the  overall project funding.     if you have an idea for a small work‐based  research project, why not consider applying  for riwa 2006/7?  a copy of the 2 page application form is  available from:  http://ifmh.org.uk/riwa.html      submission deadline: 22nd december 2006.     for further details contact maria on +44 (0)  161 295 6423 or email:  m.j.grant@salford.ac.uk      * riwa 2006/7 is sponsored by the national  library for health cpd forum, ifm  healthcare, the health libraries group, the  university medical school librarians group,  the university health sciences libraries and  libraries for nursing.    http://creativecommons.org/licenses/by/2.0 http://ifmh.org.uk/riwa.html evidence based library and information practice 2012, 7.2 81 evidence based library and information practice evidence summary perception and information behaviour of institutional repository end-users provides valuable insight for future development a review of: st. jean, b., rieh, s. y., yakel, e., & markey, k. (2011). unheard voices: institutional repository endusers. college & research libraries, 72(1), 21-42. reviewed by: lisa shen reference librarian and assistant professor newton gresham library, sam houston state university huntsville, texas, united states of america email: lshen@shsu.edu received: 12 sept. 2011 accepted: 4 jan. 2012 2012 shen. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to determine the perceptions and information behavior of institutional repository (ir) end-users. design – semi-structured interviews. setting – the interviews were conducted over the telephone. subjects – twenty end-users of five different irs were interviewed for the study. seventeen of the interviewees were recruited via recruitment forms the researchers placed on ir homepages and the other three interviewees were referred to researchers by ir managers. the interviewees’ academic backgrounds varied, including six undergraduates, four masters’ students, three doctorial students, five faculty, and two library or museum staff members. they represented disciplines in arts and humanities (5), science and health sciences (10), and social sciences (5). fifteen of the 20 interviewees were recruited through their own institution’s ir. all except two of the interviewees had used the ir for which they were recruited less than six times. methods – forty-three potential interviewees were recruited using web recruitment forms and ir manager recommendations. researchers subsequently excluded 23 (53.5%) of the interviewees because they were primarily ir contributors rather than endusers, or could not be reached by phone. mailto:lshen@shsu.edu evidence based library and information practice 2012, 7.2 82 twenty interviews ranging from 17 to 60 minutes were conducted between january and june 2008. the average interview time was 34 minutes. the recordings were transcribed then analyzed using qualitative data analysis software nvivo7. coding categories were developed using both the original research questions and emerging themes from the actual transcripts. the final coding scheme had a holsi coefficient of reliability of 0.732 for inter-coder reliability. main results – researchers identified six common themes from the results: how do end-users characterize irs? while most interviewees recognized that there is a relationship between the ir and its host institution, their understandings of the function and content of irs varied widely. interviewees likened the irs they used to a varying array of information resources and tools, including databases, interface, server, online forums, and “static wikipedia” (p. 27). furthermore, six of the interviewees had never heard of the actual term “institutional repository” (p. 27). how do end-users access and use irs? the most common methods of accessing irs included selecting the link on their institution library’s website and google searches. many interviewees found out about the irs they are using through recommendations from professors, peers, or library workshops. other interviewees found out about particular irs “simply because a google search had landed them there” (p. 29). interviewees’ preferred method of interacting with an ir were divided between browsing and keyword searching. however, these preferences may have been the result of an ir’s content or interface limitations. for instance, some interviewees expressed difficulties with browsing a particular ir, while another interviewee preferred browsing because “there wasn’t much going on” when searching for a specific topic of interest (p. 30). for what purposes do end-users use irs? interviewees commonly cited keeping abreast with research projects from their own university as a reason to access their institutions’ irs. student interviewees also used irs to find examples of theses and dissertations they would be expected to complete. identifying people doing similar work across different departments in the same institution for collaboration and networking opportunities was another unique purpose for using irs. how do end-users perceive the credibility of information from irs? many interviewees perceived irs to be more “trustworthy” than google scholar (p. 33). in their view, an ir’s credibility was assured by the reputation of its affiliated institution. on the other hand, many interviewees viewed a lack of comprehensiveness in content negatively when judging the credibility of an information source, which placed most irs in a less favorable light. additionally, researchers noted conflicting assumptions made by interviewees about irs in the evaluation process for their content. some interviewees believed all the content of an ir has been vetted through an approval process, while others distrusted all ir content that was not peer-reviewed. to what extent are end-users willing to return to an ir or recommend it to their peers? the great majority of interviews indicated they were likely to use irs again in the future, and nearly all indicated they would recommend irs to their peers. however, most interviewees did not know of any people using irs. the few interviewees who did often knew of ir contributors rather than end-users. how do irs fit into end-users’ information seeking behavior? many interviewees noted that irs provided them with content that was not commonly available through traditional publishing channels, including conference papers and dissertations. others felt irs made content available more quickly than other information sources. however, the results also suggested evidence based library and information practice 2012, 7.2 83 that most interviewees did not include irs in their routine research process. conclusion – this study identified current end-users’ perceptions of irs and highlighted several areas for future ir development. areas of improvement for irs included intensifying publicity efforts; increasing content recruitment; making content recruitment policies more transparent; and improving appearance and navigation functionalities. the findings also suggested new directions for ir marketing, such as emphasizing on the networking and collaborating benefits of using ir. commentary this exploratory study uncovered several insights for ir development. study results indicated end-users were largely unfamiliar with the purpose and scope of irs. a significant portion of end-users surveyed were also unsatisfied with the collection size and usability of irs they have accessed. these findings provided valuable directions for ir improvement, especially in user-experience related areas such as interface design and marketing. nonetheless, this study was exploratory and its findings were meant to generate new research ideas and encourage further scholarship, not to serve as generalized conclusions. there were also several shortcomings in this study that future research could improve upon. one flaw of the study lies in its subject recruitment through sign-up forms posted on ir homepages. as the authors themselves noted, past studies found that majority of enduser reaches irs via google or google scholar, which bypassed ir homepages. since majority of users from the five irs were excluded from the recruitment process, then, one cannot conclude the interviewees’ comments were representative of perceptions of general ir end-users. moreover, while the researchers noted difficulties with differentiating between ir end-users and contributors, their actual methods for distinguishing them were not specified. five (25%) of the interviewees were both ir end-users and contributors, and this inclusion could have negatively affected study results. for instance, part of the investigation included interviewees’ perception an ir’s content quality, and interviewees with contribution experience and familiarity with an ir’s content recruitment policy would likely have had a different perspective than endusers. lastly, the researchers’ rationale for selecting the five particular irs for recruitment was not specified, nor did the researchers identify these irs. providing access to the irs reviewed in the interviews would allow audiences to better understand some of the interviewees’ comments. in one instance, the researchers noted conflicting interviewee opinions on whether irs were better for browsing or searching. such preference variations could have been influenced by specific ir designs that interviewees were familiar with, but this hypothesis could be not verified since the irs discussed were not identified. due to these limitations in data collection, the overall validity of this study is less than 75% based on the ebl critical appraisal checklist (glynn, 2006). this validity score suggests readers should not use the results for generalized conclusions. even so, this study provided valuable contribution to current literature because it highlighted unique challenges face by ir end-users and provided directions for future ir designs. references glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. research article   assessment of online information literacy learning objects for first year community college english composition   mara bordignon coordinator, teaching and learning librarian seneca college toronto, ontario, canada email: mara.bordignon@senecacollege.ca   alana otis reference and information literacy technician seneca college toronto, ontario, canada email: alana.otis@senecacollege.ca   adele georgievski information literacy and liaison librarian seneca college toronto, ontario, canada email: adele.georgievski@senecacollege.ca   jennifer peters teaching and learning technologies librarian seneca college toronto, ontario, canada email: jennifer.peters@senecacollege.ca   gail strachan information literacy and liaison librarian seneca college toronto, ontario, canada email: gstrachan@ocls.ca   joy muller associate director of library services and copyright management seneca college toronto, ontario, canada email : joy.muller@senecacollege.ca   rana tamin associate professor / assistant dean for research and graduate studies college of education zayed university dubai, united arab emirates email: rana.tamim@zu.ac.ae     received: 15 july 2015    accepted: 8 apr. 2016          2016 bordignon, otis, georgievski, peters, strachan, muller, and tamin. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – the main objective was to determine whether information literacy (il) learning objects (los) impact student il competency, specifically in a foundational first year english composition course. the primary research question was: what is the effectiveness of il los compared to face-to-face instruction in terms of students’ skill acquisition?   methods – the methods involved testing student il competency through a multiple-choice test given preand post-il intervention. effectiveness was measured by assessing whether il competency improves after exposure to one of two interventions: online il los or face-to-face librarian-led workshop. over two semesters, equal sections of the course were tested for each of these interventions. for the il los group, students first completed a pre-test, then they worked independently through three online il los. the three il los were videos comprised of animation, screen casting, and video capture on these topics: finding articles at seneca libraries (hereafter referred to as finding articles), finding articles on current issues, and popular and scholarly sources. the students were then given the same test again. for the face-to-face group, the preand post-tests were also required for the same number of sections. this study was conducted under institutional ethics approval.   results – descriptive analysis revealed student test scores increased for both interventions, il los and face-to-face. test scores increased, on average, between 14 to 37%. in comparing post-tests, results revealed a statistically significant difference only with the first topic, finding articles. in this case, the il los (video) group outperformed the face-to-face group by at least 10%. no significance, in terms of performance from preand post-test scores, was found for the other two topics.   conclusion – both il lo and face-to-face library led workshop interventions had a positive impact on students’ il skill acquisition as evidenced by an overall increase in average test scores. one il lo on finding articles significantly outperformed the face-to-face class equivalent. further study is needed to track individual student performance.   introduction   seneca libraries has been an innovator in creating learning objects (los) to teach students information literacy (il) skills. we realized early the need to integrate online learning into our instruction strategy. the seneca libraries il team collects statistics and analyzes data to inform strategic planning and assure quality and continuous improvement. we analyzed two sets of statistics in fall 2010 and winter 2011. the first set of statistics considered the total number of one-shot il classes in foundational english composition courses. one in five, or approximately 20%, of all il classes taught by the library were for foundational english composition courses, either english & communication eac149 (non-credit developmental course in reading, writing, and oral expression that prepares students for eac150), or college english eac150 (compulsory, introductory college writing and reading course fundamental to successful college studies). this represented a significant amount of staff time spent on instruction.   approximately 80% of other il classes were taught in the program disciplines within which students major. there is currently an initiative to embed and integrate il within the program-specific curriculum. allocating staff to increase the number of classes taught for english composition would come at the expense of work already underway embedding il skills directly into the program specific courses. even if more staff could be allocated to english composition, there would still be scheduling challenges making it nearly impossible for staff to reach every section face-to-face.     the second set of statistics looked at the number of eac149 and eac150 sections taught over these two semesters, as a percentage of the total number of sections (table 1). we discovered that library instructional staff taught approximately 24-27% of all sections of eac150, and approximately 13-17% of all sections of eac149. this indicated that the majority of sections for both courses received no form of il instruction.     in addition to these statistics, we also had to take into consideration that while eac150 is compulsory, students are not obligated to take it in their first semester. therefore, it could not be certain that every first year student was receiving il instruction. if a student took the course in their last semester before graduating, they would not have had the opportunity to practice these skills in other courses, or benefit from the library’s strategic scaffolding of il skills throughout their programs.   table 1 information literacy classes taught for seneca college english composition courses semester total number of eac150 sections total number of eac150 sections taught by library percentage (%) of eac150 il sections taught by library   total number of eac149 sections   total number of eac149 sections taught by library percentage (%) of eac149 il classes taught by library   fall 2010 132 36 27 105 18 17 winter 2011 111 27 24 67 9 13   in the late 1990s, seneca libraries, in collaboration with professors, developed an online tutorial, library research success, for the business management program at seneca college.  this tutorial addressed basic business information literacy skills for first year students deemed foundational. students would work through the tutorial either in class or on their own time allowing flexibility in terms of when and where they learned. students were also required to complete a low-weighted, graded research assignment. as reviewing the il lo was a requirement of the course, we reached every student. when delivering face-to-face this is not always the case, given the staffing limitations and scheduling conflicts in the high-enrollment program. donaldson (2000), a seneca librarian and co-creator of the tutorial, published a qualitative, anecdotal techniques study that collected data in the form of reviewing completed student assignments for the tutorial and comments (which were optional) that revealed students’ perceptions. business professors were also asked to provide informal feedback through personal interviews. overall, students performed well on the assignments, and feedback from students and faculty was positive. the adoption and success of this tutorial allowed for adaption and customization in other programs, primarily for use by first year students. however, as over a decade had passed since this tutorial was created, new technologies and software had rendered the tutorial outdated.   there were several issues to be taken into consideration about the english composition course at seneca college. limited staffing and increasing enrollment meant an inability to reach every course section. librarians also wanted to make sure students received il instruction early in their studies. finally, the outdated tutorial needed a significant upgrade. how could these problems be solved? the answer was a strategic approach to the development of online learning objects.   in a survey of best practices in developing online il tutorials, holland et al. (2013) found that nearly all librarians felt it was important for the library to create its own tutorials in order to showcase their institution and its materials.   seneca libraries recognized that the development of online il los as a strategic initiative should be aligned with the institution’s goals, whereby “every seneca graduate will demonstrate competency in the seneca core literacies” (seneca college, 2012, p. 10), of which il is identified as one of the core literacies, and “faculty will model digital literacy through use of a variety of media and/or mobile technologies to engage students as partners in learning” (seneca college, 2012, p. 13).     the il team adopted the following process in order to reach seneca libraries’ strategic goal in developing online il learning objects:   1.  needs analysis. surveys were sent to library teaching staff and english faculty to determine which il topics were most commonly taught in class, and which were perceived to be the most challenging or difficult for students. these results helped identify and prioritize the il topics to be developed into los. the following were identified as priority, in order of preference: database searching, academic honesty, evaluating information, analysis and application, library website, and library catalogue searching.   2.  analysis of current best practices in the field. national and international electronic mail lists were queried and responses were taken into consideration. seneca librarians’ lesson plans and teaching materials were also reviewed. these internal documents included learning outcomes based on the acrl’s information literacy competency standards for higher education (acrl, 2000). a literature search on the development of il learning objects was conducted. from these sources, the most common il topics developed into online learning objects were:   ·         using online library tools (book catalogue, databases, libguides, etc.); ·         evaluating material and selecting resources; ·         defining a research topic; ·         searching skills for the internet (including google scholar); ·         documenting your research; ·         locating a known journal article.    instructional design and development best practices were incorporated into creating our own set of design principles to optimize student engagement and learning.   3.  inventory of los already developed by seneca libraries. comparing the list of recommended topics to be developed to the list of existing los, identifying gaps, and prioritizing objects for development.   4.  development of los. allocation of library resources (e.g., staffing, software), collaborating with english faculty to design objects, building prototypes, testing prototypes with small user groups, modifying and reviewing prototypes and launching beta objects.   an lo is “a reusable instructional resource, usually digital and web-based, that is developed to support learning” (mestre, 2012b, p. 261).  examples of learning objects can include tutorials, videos, games, and quizzes. a series of il los were developed over the 2012 spring and summer semesters, and were released in september 2012 for the start of the fall semester. a learning objects committee, under the seneca library’s information literacy (slil) team, was tasked with this project. the committee was made up of several librarians and library technicians. the committee chair, the library’s elearning technologies librarian, was both project manager and technical support. once the initial process was completed (needs analysis, best practices, and inventory), the committee broke into smaller groups responsible for developing individual los by topic. these groups consisted of one to two librarians delegated as content leads whose main responsibilities were scripting, storyboarding, and quiz creation. they were partnered with at least one library technician who provided support for filming, animations, and editing. each group was further supported by the committee lead and a library media technician, both of whom helped with filming, animation, screen casting, audio capture, and software support. each group was given permission to proceed with filming and production only after their scripts were reviewed and approved by the entire committee.   the il los consist of short, one to three minute videos that include live action recordings, screen casting, and animations. the main software used was camtasia. the videos are all closed-captioned and include a text-based transcript. for introductory il videos there is a pdf summary, and for demonstration videos there are pdf step-by-step instructions with screenshots. by offering the lessons in both video and text-based formats we hope to offer flexible options for learning. all los have learning outcomes tied to assessments, typically multiple-choice questions. los, accompanying assessments, and documentation are also bundled into library cartridges, which are zip files that can be imported as one unit into blackboard, the institution’s course management system. for consistency, il los will be herein referred to as videos.    while usability and design were tested throughout the development process, what remained to be assessed was the impact the newly created videos had on student il competency. considering the time and effort invested and the goal to teach more students online, it was vital that these videos contributed positively to student learning. we determined that the videos needed to be assessed for their effectiveness in terms of student il skill acquisition. in early 2013, we were granted ethics approval from our institution to conduct a research study to investigate this issue.   literature review   evaluation and assessment of online learning objects   it was clear we needed to update seneca’s first generation of tutorials, and developing a strategy to evaluate and assess them was paramount. the abundant amount of literature on learning object development and creation indicates interest and activity in this area, especially studies which review and survey best practices (blummer & kritskaya, 2009; mestre, 2012a; somoza-fernández & abadal, 2009; su & kuo, 2010; yang, 2009; zhang, 2006). these studies also identified the importance of building in evaluation and assessment as part of the development process in order to measure success and effectiveness.     mestre (2012a) noted the importance of assessment as a way of measuring success. mestre (2012a) also stated that assessment should focus on students’ learning, as well as outcomes and opinions and lists various ways to document evidence as to whether the goals of the learning object were accomplished: checkpoints, statistical tracking, log file analysis, web page analytics, tracking new accounts, evaluation of student work preand post-tests, student debriefing, and surveys.   measuring success: usability, student learning, student perceptions or all of the above?   the issue on what aspect to evaluate or assess was evident in several studies. lindsay, cummings, johnson, and scales (2006) grappled with this dilemma when they asked “is it more important to measure student learning or to study how well the tool can be navigated and utilized?” (p. 431). they settled on capturing both areas, but without one-on-one usability testing, instead designing “the assessment modules to gather data from the students about their use of resources, attitudes towards the libraries, and perceptions of the utility of the online tutorials” (lindsay et al., 2006, p. 432). befus and byrne (2011, as cited in thornes, 2012), found that the success of a tutorial can be difficult to quantify. they found that despite students obtaining lower than anticipated scores in the associated test, the tutorial was still successful because it reached more students with greater flexibility.   comparisons in library instructional delivery methods   other studies investigated whether online learning modules were as effective as more traditional modes of instruction, such as librarian-led, face-to-face classroom sessions, and most found that the modules were equally effective. bracke and dickenson (2002) found that “using an assignment-specific web tutorial in conjunction with an instructor-led, in-class preparatory exercise is an effective method of delivering library instruction to large classes” (p. 335). silver and nickel (2005) developed and embedded a multiple module tutorial for a psychology course, which was animated and interactive. post-tests on material covered, including questions on confidence level and preferred mode of instruction, showed that there was no difference between the tutorial and classroom instruction in terms of quiz results (silver & nickel, 2005).  koufogiannakis and wiebe’s (2006) systematic review of 122 unique studies found that instruction provided electronically was just as effective as more traditional instruction. specifically, “fourteen studies compared [computer assisted instruction] cai with traditional instruction (ti), and 9 of these showed a neutral result. meta‐analysis of 8 of these studies agreed with this neutral result” (koufogiannakis & wiebe, 2006, p. 4). kraemer et al. (2007) compared three instructional methods: online instruction only, live instruction, and a hybrid combination in a first-year writing course. they concluded with a “high degree of confidence that significant improvement in test performance occurred for all subjects following library instruction, regardless of the format of that instruction” (kraemer et. al., 2007, p. 336). similarly, as part of the curriculum for a general education course, anderson and may (2010) tested the following il topics across three conditions: library catalog, academic databases, boolean searching, and evaluation of sources materials. their results indicated that the way in which instruction is delivered does not affect the students’ ability to retain the information taught (anderson & may, 2010). sachs et al. (2013) also found that millennial students learned equally well from both html-based tutorials and dynamic, interactive audio/video tutorials. however, they also found that “students expressed a much higher level of satisfaction from the tutorial designed to be ‘millennial friendly’” (sachs et. al., 2013, p. 1).   instructional effectiveness of online learning objects   while previous studies point out that online tutorials can be just as effective as face-to-face classroom instruction and in effect, compare modes of delivery, another branch of literature compares different types of online tutorials for their instructional effectiveness. mestre (2012b) found “that a screencast tutorial with images can be more effective than a screencast video tutorial” (p. 273) for 16 out of 21 students tested.  in contrast, mery et al. (2014) found that there was no impact on student performance between two types of instruction, one form of receiving information from passively watching a screencast, and the other form rooted in active learning, the guide on the side. despite limitations to the study, mery et al. (2014) still asserted that “database instruction can successfully be taught online in a number of ways from static tutorials to highly interactive ones” (p. 78).     mixed methodology studies   as mentioned earlier, most studies invariably have some form of usability testing, along with some measure on student learning through testing content, pedagogical approaches, or student learning styles or preferences. johnston (2010) investigated first year social work students’ opinions on il, while also gathering feedback on the tutorial, and assessing students’ skills. they employed a mixed methods approach with quantitative and qualitative research methods that included a survey, focus groups, empirical data from task results, and observations (johnston, 2010, p. 211). the majority of students were given tasks to complete and researchers evaluated if those tasks were completed efficiently; however, an exact measurement was not specified or elaborated on. findings indicate that students efficiently completed their tasks involving evaluating websites and finding cited and relevant information using google, while they struggled with tasks involving databases, including search techniques, and differentiating between databases and other sources of information (johnston, 2010). an observational study by bowles-terry et al. (2010) “examined the usability of brief instructional videos but also investigated whether watching a video tutorial enabled a student to complete the task described in the tutorial” (p. 21). their findings informed best practices in the following categories: pace, length, content, look and feel, video vs. text, findability, and interest in using video tutorials (bowles-terry et al., 2010). they also pointed out that future research is needed, particularly performance-based assessments as they “would give great insight into how well videos can be used to teach and whether their effectiveness is restricted to students with particular learning styles and/or specific content, for example, procedural, rather than conceptual” (bowles-terry et al., 2010, p. 27). adapting these models of evaluation or assessment with a focus on measuring student learning, particularly through quantitative methods, seemed to make the most sense for our learning objects. taking into consideration that usability studies have been done throughout the development and prototype cycle of our project, measuring how our learning objects impact student learning seemed to be the most pressing issue to investigate.   aims   the aim of this preliminary quantitative study is to ascertain whether library-developed il los impact student il competency in comparison to traditional face-to-face instruction in a first year english composition foundation course. if the los impact student il competency in the same way, or to a greater degree as face-to-face instruction, then this evidence can be used to inform the use, development, and assessment of il los in the library’s il program. no previous research of this kind has been carried out by seneca libraries. the secondary aim was to measure, through preand post-testing, if there is a statistically significant difference in student performance for any one of the three pre-selected il topics as a success indicator for one method of instruction, e.g. online or traditional face-to-face. results of this study can help inform the lo development process, in addition to future assessment studies of il los. it can also be used to add to the wider discussion of the use and development of il los in secondary education.   methods   type of assessment   the literature distinguishes between two different types of evaluation and assessment: 1. measurement throughout the development and prototype cycle in order to inform design or structural changes in the form of usability testing, and; 2. measurement of student learning by testing different pedagogical approaches and student learning behaviour. most studies invariably have some form of usability testing, along with some type of measurement on student learning.   in our case, adoption of best practices meant that informal usability testing occurred throughout the development and prototype cycle for learning object development, albeit informally and therefore inconsistently. in specific, two methods of assessment, as identified by mestre (2012a) were used, and would fall under the first type of assessment mentioned above:   pilot (beta) testing. during script and storyboard development, student library workers, individually or in small groups of two or three, were sporadically recruited and asked for input. student feedback. informal feedback was obtained either individually during reference interviews, or as small groups, during in-class il sessions. general, open-ended questions were asked and responses recorded by a library technician or librarian. questions were not standardized.   in this way, design could be continually improved to meet the needs of the users. with a reasonable amount of confidence, we felt that the second generation of modules we were building had solid design principles based on the best practices and experiences set by other academic libraries. the main variation with our modules was the customization to the local context so that seneca libraries’ resources, students, and course-specific research challenges were represented. recommendations from usability studies helped guide our learning object development (bury & oud, 2005; lund & pors, 2012; mestre, 2012b).   this preliminary study focused instead on the second type of assessment, measuring student learning. while building on earlier similar studies (anderson & may, 2010; gunn & miree, 2012; johnston, 2010; kraemer et. al., 2007; mery et al., 2014; zhang, goodman, & xie, 2015), the departure lies mainly with a focused or narrow method by testing only student performance. quantitative student test results were analyzed through determining statistical significance for each of three information literacy topics.   data collection   to measure the effectiveness of the videos in terms of students’ skill acquisition, a preliminary quantitative study was initiated. ethics approval was obtained from the institution and all students consented to take part in the study. participation was optional and students could choose to exit the study at any time. results were anonymous and did not impact student grades.   we decided to conduct our study in the foundational english composition course, college english eac150. this is a compulsory course for students and so an ideal student population to test for basic il skills. more importantly, librarians had been partnering with english faculty for several years, delivering face-to-face one-shot instructional sessions tailored to the learning outcome in the course syllabus. students were required to produce effective research writing through the completion of a research project. students had incentive to participate as the information learned through the study would help them complete the research project in the course.   the study was carried out over two semesters; 75 students participated in the winter (january to april) 2013 semester (herein referred to as group 1), and 35 students participated in the fall (september to december) 2013 semester (herein referred to as group 2). a librarian and a library technician led each group. in each, the students were first assessed for their il skills competency through completing an online pre-test of multiple-choice questions. the students were then exposed to one of two interventions: online videos or face-to-face, librarian-led instruction. after the intervention, the students were given the same test again. for the videos intervention, these consisted of three newly created online videos that were produced in house: finding articles, finding articles on current issues, and popular and scholarly sources.     the learning outcomes were standardized across the two interventions so that the face-to-face classes taught to the same learning outcomes as the videos. the learning outcomes for finding articles were (the learner will be able to…): 1. select appropriate database(s) by subject or discipline as related to their research topic; 2. perform a basic search in a database; and 3. understand various mechanisms for retrieving articles (printing, emailing, saving). the learning outcomes match the lower-order skills of bloom’s taxonomy which fall under knowledge or remembering (krathwohl, 2002). the learning outcomes for finding articles on current issues were (the learner will be able to…): 1. select social sciences, news and current events databases; 2. perform searches based on research topic; and 3. evaluate results for relevancy. the learning outcomes for popular and scholarly sources were (the learner will be able to…): 1. differentiate between popular and scholarly literature; 2. identify characteristics of a scholarly article; and 3. select the appropriate type of article for their research needs. the learning outcomes for these last two videos match higher-order skills under analysis according to bloom’s taxonomy (krathwohl, 2002).   for the video intervention, students were asked to view the videos independently using their own headphones, or headphones were made available and distributed. students then completed the online test and results were gathered through the online tool, surveymonkey. all questions were multiple choice and were based on the content in the videos. the questions were written by librarians who developed the videos and were the main assessment tools used to test student understanding of the content found in each video. the questions were independently reviewed by a library technician who matched each question against the script (content) in the video as a measure for quality control. for the face-to-face, librarian-led instruction intervention, students were presented with the same content (and learning outcomes) as the three videos. the same library staff moderated both interventions, for the same campus location, to ensure consistency in pacing and content. if students had technical issues with the online test, library staff provided support. if students had any additional questions in regards to the content, e.g. seeking help with question clarification, library staff would provide guidance but were mindful of not providing overt clues that could inadvertently point to the correct answers.   in group 1, 40 students were exposed to the online videos intervention, and 35 were exposed to the face-to-face, librarian-led instruction. the online test consisted of fifteen multiple-choice questions (appendix a), in which there were five questions for each of the three videos.   in group 2, 18 students were exposed to the online videos intervention, and 17 were exposed to the face-to-face, librarian-led instruction. the online test consisted of 14 multiple-choice questions, in which there were 5 questions for 2 videos, and 4 questions were given for the video finding articles on current issues (appendix a). unfortunately, one question had to be withdrawn from the test because it no longer made sense in light of a significant structural change to the homepage of the library’s website. we decided to delete the question, rather than replace it, since the answers were not likely to be comparable when analyzing results.   the main research question was:  what is the effectiveness of videos, in comparison to face-to-face instruction, in terms of students’ skill acquisition?   statistical analysis   general descriptive statistics were run for the individual pre and post-tests for each of the groups. considering that the current research project was preliminary in nature, comparisons were only made between the pre-tests of the videos and face-to-face groups for each of the topics as well as the post-tests of the videos and face-to-face groups for each of the topics through independent samples t-tests. unfortunately, repeated measures could not be used to compare pre-tests and post-tests for each topic due to the fact that the tests were anonymous and it was not possible to match the pre-test and post-test for each participant.   results   pre-test measurement of students, in each of the three topic areas, was done to determine pre-existing skill level. we anticipated that the post-test measurement would be affected after applying an intervention, either exposure to an online module or a face-to-face class. in either case, we hoped that an increase in test scores would indicate learning.   findings showed that test scores improved regardless of intervention. the lowest test score increase, averaged across a group of 35 students, was 14.6% for face-to-face (figure 1). the highest test score increase, averaged across a group of 18 students, was 37.5% for videos (figure 1).     when pooling results for both groups, and running a t-test between the video group pre-test and face-to-face group pre-test for each of the topics, results indicated that both groups were not significantly different in their knowledge of the three topics.   similarly, a t-test was used in comparing post-tests results for video to the face-to-face across both groups for each of the topics. independent samples test results revealed a statistically significant difference with the first topic, finding articles, t(110) = 2.25 and p = 0.026. the videos group outperformed the face-to-face group by at least 10%. no significance, in terms of performance from preand post-test scores, was found for the other two topics: finding articles on current events, t(110) = -1.11 and p = 0.2688, and popular & scholarly t(110) = -0.009 and p = 0.993.     figure 1 change in preand post-test scores amongst group 1 and group 2 for both interventions, online videos and face-to-face instruction.     for the first topic, finding articles, scores for both groups 1 and 2 increased on average 34.7% and 15.3% respectively for the video group (figure 1). in comparison, scores increased on average 14.6% and 26.2% respectively for the face-to-face group. the highest average post-test scores were found for the video group (figure 1). on average, the mean test scores were higher in the post-test for both groups (figure 2).   for the second topic, finding articles on current issues, scores for both groups 1 and 2 increased on average 23.3% and 37.5% respectively for the video group (figure 1). in comparison, scores increased on average 18.2% and 27.6% respectively for the face-to-face group. in this case, preand post-test scores were consistently the lowest (figure 2).   for the third topic, scholarly & popular sources, scores for both groups 1 and 2 increased on average 33.4% and 27.1% respectively for the video group (figure 1). in comparison, scores increased on average 33.7% and 26.1% respectively for the face-to-face group. similar to the first topic, this topic also had the highest post-test scores in the video group (figure 2).     figure 2 mean test scores: groups 1 and 2 combined. *please note that for topic 2, data set for group 2 normalized to 5 from 4.     discussion   similar to previous studies (anderson & may, 2010; kraemer et. al., 2007; koufogiannakis & wiebe, 2006; silver & nickel, 2005) this preliminary study reaffirmed that exposure to il instruction, regardless of method of delivery—either through online modules or face-to-face librarian instruction—increases il skills of students. overall, for both groups there was an increase in test scores after online and face-to-face instruction. on average, test scores increased between 14 to 37% where the lowest test score increase, averaged across a group of 35 students, was 14.6% for face-to-face and the highest test score increase, averaged across a group of 18 students, was 37.5% for videos. however, as this analysis was descriptive in nature, we also sought to determine if there was real statistical significance to these increases.   when comparing online modules to face-to-face instruction, we found one instance in which online modules outperformed face-to-face library instruction. for both groups, the difference in post-test scores for students exposed to online videos compared to those exposed to face-to-face instruction, was statistically significant only for one topic, finding articles. in this instance, we can say with a reasonable amount of confidence, that the video outperformed face-to-face instruction. for this topic, students exposed to the videos outperformed those students exposed to face-to-face instruction by at least 10%. perhaps this topic was better suited for online learning because the learning outcomes for this particular lo were task-based, and required lower-order thinking. perhaps these simple step-by-step tasks and instructions were better demonstrated through an online, video-based environment. further observation would be needed to understand why this may be the case.    there was no statistical significance in results for the other two topics, finding articles on current issues, and popular and scholarly sources. for these two topics, whether instruction is delivered online or face-to-face had no impact on student performance, unlike the finding articles topic. one reason for this may be that the learning outcomes for these topics required higher-order thinking, thus making it more difficult to learn, regardless of whether it was taught online or face-to-face.    we can therefore conclude that a video, built following best practices and customized to a program’s curriculum and student body, can have the same, if not better, impact on students’ uptake of il skills in comparison to live, face-to-face librarian-led led classes. in addition, because our findings showed statistical significance with one topic (finding articles), it indicates particular il topics are better suited for delivery in an online environment. this area of study, applying statistical significance through t-tests as it relates to specific il topics, is less represented in the literature than the overall usability and effectiveness of il tutorials or modules.   another point of discussion is whether or not the text-based transcripts of each video had an impact on student learning. this was not studied separately, but could be considered another method of instruction in addition to online video and face-to-face instruction that would need further investigation. the proven efficacy of the il los have encouraged further usage of the text-based transcripts and summaries in subsequent los.   this preliminary study had limitations. firstly, while we did perform an independent t-test to show differences in group averages, we could not perform a paired, or dependent, t-test which would have been possible had we tracked the identity of each individual participant. a paired, or dependent, t-test analysis would have looked at the sampling distribution of the differences between scores, not the scores themselves. thus, we would have been able to track differences in test scores, for each individual student, rather than looking at pooled averages.   secondly, a mixed methodology approach would have been useful. more data would be captured for interpretation through combining quantitative and qualitative methods. measuring the differences in student performance for teaching method (online vs. face-to-face) and il topic (three different topics) was the quantitative measurements. we combined this with the measurements of collecting demographic data on students, focus groups, and observational user testing. we would not only have the ability to analyze test scores, but would also have the ability to see correlations.   thirdly, while the sample size was reasonable, at 110 participant students we did not obtain the total number of students enrolled in all sections of college english, eac150 for those two semesters. we cannot assume that our sample size accurately represents the average or normal behaviour of all students enrolled in this course. we would need to obtain this figure, and compare our smaller sample size as a percentage.   conclusions   this preliminary quantitative study gathered evidence in helping to determine whether library developed il los impact student il competency in comparison to traditional face-to-face instruction in a first year foundational english composition course. this study found that both il los (videos) and face-to-face instruction have a positive impact by increasing students’ il test scores. only one video on the topic finding articles outperformed face-to-face instruction. further work, in the form of a mixed methodology study, would be beneficial in identifying how specific characteristics, for both online modules and face-to-face instruction, impact student acquisition of il skills.   references   anderson, k., & may, f. a. (2010). does the method of instruction matter? an experimental examination of information literacy instruction in the online, blended, and face-to-face classrooms. the journal of academic librarianship, 36(6), 495–500. http://doi.org/doi:10.1016/j.acalib.2010.08.005   association of college & research libraries. (2000). information literacy competency standards for higher education [guidelines, standards, and frameworks]. retrieved from http://www.ala.org/acrl/standards/informationliteracycompetency.   befus, r., & byrne, k. (2011). redesigned with them in mind: evaluating an online library information literacy tutorial. urban library journal, 17(1) retrieved from http://ojs.gc.cuny.edu/index.php/urbanlibrary/index     blummer, b. a., & kritskaya, o. (2009). best practices for creating an online tutorial: a literature review. journal of web librarianship, 3(3), 199-216. http://dx.doi.org/10.1080/19322900903050799   bracke, p. j., & dickstein, r. (2002). web tutorials and scalable instruction: testing the waters. reference services review, 40(3), 330–337.  http://dx.doi.org/10.1108/00907320210451321   bury, s., & oud, j. (2005). usability testing of an online information literacy tutorial. reference services review, 33(1), 54–65.  http://dx.doi.org/10.1108/00907320510581388   donaldson, k. a. (2000). library research success: designing an online tutorial to teach information literacy skills to first-year students. the internet and higher education, 2(4), 237-251. http://dx.doi.org/10.1016/s1096-7516(00)00025-7   gunn, m., & miree, c. e. (2012). business information literacy teaching at different academic levels: an exploration of skills and implications for instructional design. journal of information literacy, 6(1), 17-34. http://dx.doi.org/10.11645/6.1.1671     johnston, n. (2010). is an online learning module an effective way to develop information literacy skills? australian academic and research libraries, 41(3), 207–218. http://dx.doi.org/10.1080/00048623.2010.10721464   koufogiannakis, d., & wiebe, n. (2006). effective methods for teaching information literacy skills to undergraduate students: a systematic review and meta-analysis. evidence based library and information practice, 1(3), 3–43. http://dx.doi.org/10.18438/b8ms3d   kraemer, e. w., lombardo, s. v., & lepkowski, f. j. (2007). the librarian, the machine, or a little of both: a comparative study of three information literacy pedagogies at oakland university. college & research libraries, 68(4), 330–342. http://dx.doi.org/10.5860/crl.68.4.330   krathwohl, d. r. (2002). a revision of bloom’s taxonomy: an overview. theory into practice, 41(4), 212-218. http://dx.doi.org/10.1207/s15430421tip4104_2   lindsay, e. b., cummings, l., johnson, c. m., & scales, b. j. (2006). if you build it, will they learn? assessing online information literacy tutorials. college & research libraries, 67(5), 429–445. http://dx.doi.org/10.5860/crl.67.5.429   lund, h., & pors, n. o. (2012). web-tutorials in context: affordances and usability perspectives. performance measurement and metrics, 13(3), 197–211. http://dx.doi.org/10.1108/14678041211284731   mery, y., defrain, e., kline, e., & sult, l. (2014). evaluating the effectiveness of tools for online database instruction. communications in information literacy, 8(1), 70–81.  retrieved from http://www.comminfolit.org/     mestre, l. s. (2012a). designing effective library tutorials:  a guide for accommodating multiple learning styles. retrieved from proquest safari books online database.   mestre, l. s. (2012b). student preference for tutorial design: a usability study. reference services review, 40(2), 258-276. http://dx.doi.org/10.1108/00907321211228318   sachs, d. e., langana, k. a., leatherman, c. c., & walters, j. l. (2013). assessing the effectiveness of online information literacy tutorials for millennial undergraduates. university libraries faculty & staff publications. paper 29. retrieved from http://scholarworks.wmich.edu/library_pubs/29   seneca college. (2012). academic plan 2012 2017 [planning documentation]. retrieved from http://www.senecacollege.ca/about/reports/academic-plan/index.html   silver, s. l., & nickel, l. t. (2005). are online tutorials effective? a comparison of online and classroom library instruction methods. research strategies, 20(4), 389–396. http://dx.doi.org/doi:10.1016/j.resstr.2006.12.012   somoza-fernández, m., & abadal, e. (2009). analysis of web-based tutorials created by academic libraries. the journal of academic librarianship, 35(2), 126–131. http://dx.doi.org/doi:10.1016/j.acalib.2009.01.010   su, s.-f., & kuo, j. (2010). design and development of web-based information literacy tutorials. the journal of academic librarianship, 36(4), 320–328. http://dx.doi.org/doi:10.1016/j.acalib.2010.05.006   thornes, s. l. (2012). creating an online tutorial to develop academic and research skills. journal of information literacy, 6(1), 82–95. http://dx.doi.org/10.11645/6.1.1654   yang, s. (2009). information literacy online tutorials: an introduction to rationale and technological tools in tutorial creation. the electronic library, 27(4), 684–693. http://dx.doi.org/10.1108/02640470910979624   zhang, l. (2006). effectively incorporating instructional media into web-based information literacy. the electronic library, 24(3), 294–306. http://dx.doi.org/10.1108/02640470610671169   zhang, q., goodman, m., & xie, s. (2015). integrating library instruction into the course management system for a first-year engineering class: an evidence-based study measuring the effectiveness of blended learning on students’ information literacy levels. college & research libraries, 76(7), 934-958. http://dx.doi.org/10.5860/crl.76.7.934     appendix a pre and post test questions (please note italicized indicates correct answer)   topic: finding articles 1. where do you go on the library website to find databases? a)       library catalogue b)       articles tab c)       repositories d)      all of the above   2. to find a database with articles about canadian politics, you should try: a)       browsing the alphabetical list of databases b)       any database will have the articles on your topic c)        select the subject that best matches your topic from the drop down list of subjects d)      all of the above   3. where in an article record will you find article information like journal title, date of publication, and page number? a)       abstract b)       source c)       subject terms d)      author   4. what should you do if the database you are searching doesn’t have enough articles on your topic? a)       try a different database b)       go to google c)       use the library catalogue d)      give up   5. what are your options for saving articles? a)       print b)       bookmark c)       email d)       all of the above   topic: finding articles on current issues   1. you are doing a research assignment and need information on a topic that was recently covered in the news. where is the best place to start? a)       google b)       a specific database for current events c)       wikipedia d)      the library catalogue   2. which category of databases is the best to use to find articles on current issues?* a)       general b)       science and technology c)       business d)       news and current events *[please note that this question was withdrawn from the test for group 2 only as it no longer was relevant in light of a significant structural change to the homepage of the library’s website. it was decided it was best to delete the question, rather than replace it since the answers were not likely to be comparable when analyzing results.]   3. of the following list, which database offers a concise list of current events? a)       adforum b)       academic onefile c)        opposing viewpoints d)      canadian newsstand   4. what information can be found about a current issue in the database opposing viewpoints? a)       statistics b)       journal articles c)       viewpoints d)       all of the above   5. how can you search for current issues in the database opposing viewpoints? a)       click browse issues or type in an issue of your own b)       click latest news and choose from a list c)       click resources and choose a category d)      click search history to see what issues other people have searched   topic: popular and scholarly sources   1. when searching for information, the best place to start is… a)       google b)       itunes u c)       twitter d)       seneca libraries website   2. popular articles can be… a)       news stories b)       reviews c)       topic overviews d)       all of the above   3. scholarly articles usually come from... a)       journals b)       newspapers c)       magazines d)      blogs   4. it is sometimes difficult to determine whether or not an article comes from a journal. which statement does not apply to scholarly articles? a)       are usually several pages long b)       does not need to contain a list of references c)       are divided into sections, the first section of which is usually an abstract or synopsis. d)      are written by a scholar or expert within the subject discipline   5. in order to ensure quality, journals are often… a)       board reviewed b)       peer reviewed c)       panel reviewed d)      technically reviewed   microsoft word es 4539 evidence based library and information practice 2009, 4:1 evidence based library and information practice evidence summary access of digitized print originals in us and uk higher education libraries combined with print circulation indicates increased usage of traditional forms of reading materials a review of: joint, nicholas. “is digitisation the new circulation?: borrowing trends, digitisation and the nature of reading in us and uk libraries.” library review 57.2 (2008): 87-95. reviewed by: kurt blythe university of north carolina at chapel hill 151 davis library, chapel hill, nc 27514 e-mail: kcblythe@email.unc.edu received: 12 december 2008 accepted: 10 february 2009 © 2009. blythe. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. objective – to discern the statistical accuracy of reports that print circulation is in decline in libraries, particularly higher education libraries in the united states (us) and united kingdom (uk), and to determine if circulation patterns reflect a changing dynamic in patron reading habits. design – comparative statistical analysis. setting – library circulation statistics from as early as 1982 to as recent as 2006, culled from various sources with specific references to statistics gathered by the national endowment for the arts (nea), the library and information statistics unit (lisu), the association of research libraries (arl), the national center for education statistics (nces), and the association of college and research libraries (acrl). subjects – higher education institutions in the united states and united kingdom, along with public libraries to a lesser extent. methods – this study consists of an analysis of print circulation statistics in public and higher education libraries in the us and uk, combined with data on multimedia circulation in public libraries and instances of digital access in university libraries. specifically, nea statistics provided data on print readership levels in the us from 1982 to 2002; lisu statistics were analyzed for evidence based library and information practice 2009, 4:1 circulation figures and gate counts in uk public libraries; arl statistics from 1996 to 2006 provided circulation data for large north american research libraries; nces statistics from 1990 to 2004 contributed data on circulation in “tertiary level” us higher education libraries; and acrl statistics were analyzed for more circulation numbers for us post-secondary education libraries. the study further includes data on uk trends in print readership and circulation in uk higher education libraries, and trends in us public library circulation of non-print materials. main results – analysis of the data indicates that print circulation is down in us and uk public libraries and in arlmember libraries, while it is up in the nonarl higher education libraries represented and in uk higher education libraries. however, audio book circulation in us public libraries supplements print circulation to the point where overall circulation of book materials is increasing, and the access of digital literature supplements print circulation in arlmember libraries (although the statistics are difficult to measure and meld with print circulation statistics). essentially, the circulation of book material is increasing in most institutions when all formats are considered. according to the author, library patrons are reading more than ever; the materials patrons are accessing are traditional in content regardless of the means by which the materials are accessed. conclusion – the author contends that print circulation is in decline only where digitization efforts are extensive, such as in arl-member libraries; when digital content is factored into the equation the access of book-type materials is up in most libraries. the author speculates that whether library patrons use print or digital materials, the content of those materials is largely traditional in nature, thereby resulting in the act of “literary” reading remaining a focal point of library usage. modes of reading and learning have not changed, at least insofar as these things may be inferred from studying circulation statistics. the author asserts that digital access is favorable to patrons and that libraries should attempt to follow the arl model of engaging in largescale digitization projects in order to provide better service to their patrons; the author goes on to argue that uk institutions with comparable funding to arls will have greater success in this endeavour if uk copyright laws are relaxed. commentary the article does a good job of confronting with statistical evidence what the author perceives to be the widely-held belief that print circulation is declining as a result of changing modes of learning and habits in reading on the part of higher education students and public library patrons. on the contrary, the author finds that print circulation is actually up in many institutions, and he reasons that when audio book circulation and digital access of print originals are combined with print circulation, the overall circulation of booktype materials is up in nearly all libraries; joint contends that these figures indicate that "reflective" reading has not been overthrown in favor of non-linear reading habits. the cognitive processes of younger, “digital natives” are not changing. the article is less effective at proving that because traditional materials are being accessed—regardless of format—they are being consumed in traditional ways, although proving this is not the thrust of the article. likewise, relatively little data are provided regarding audio book circulation statistics and what, how often, and for what reason digital collections are used. the author points out that visits to uk public evidence based library and information practice 2009, 4:1 libraries are increasing even as print circulation is decreasing, but fails to provide more than speculation that those patrons are using the library for anything more than “net surfing and game-playing.” the article attempts to argue that differences in us and uk copyright laws account for differences in the effectiveness of digitization efforts, but neither closely analyzes differences in copyright nor studies to what extent copyright impacts digitization. neither does the author really prove the correlation between increased digitization and decreased circulation; it simply remains unproven that those elements of digital content accessed by patrons are traditional in nature, and that methods of consuming those materials are also traditional in nature. nonetheless, the article does well to compare the potential for digital books to the situation with journals moving online wherein the mode of access changes while the content and method of consumption remain largely the same. moreover, the article is useful to libraries questioning their role in an increasingly technologydominated environment. although the argument for public libraries receives less attention, and detracts somewhat from the overall effectiveness of the article; some good points are made for the case of traditional scholarly content, regardless of format, in retaining its use-value in higher education. while the article is perhaps too far-reaching, moving from statistics and speculation on usage and the cognitive processes of patrons to copyright, and more speculative than is entirely useful the overall argument that traditional, reflective reading is still being done is strong, as is the argument for digitizing print originals. in regions where copyright law (and funding) allows for wide-spread digitization of print originals, as seen in the arl model, library patrons are provided with an excellent service; one that combines content with format, and does not weaken the library’s position in the discovery process. evidence based library and information practice evidence based library and information practice 2011, 6.2 88 evidence based library and information practice news/announcements ken ladd receives 2011 robert h. blackburn distinguished paper award 2011 canadian library association. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/bync-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. the canadian association of college and university libraries (cacul) is pleased to announce that the winner of the 2011 robert h. blackburn distinguished paper award is ken ladd, who is honoured for his 2010 paper entitled, “an examination of the failure rate and content equivalency of electronic surrogates and the implications for print equivalent preservation,” published in evidence based library and information practice (5:4). this well-written and detailed study addresses an important subject for both libraries and the scholars we serve. in his paper, ladd asked whether there was evidence to suggest that libraries need to preserve print equivalent journal collections. print collections would be preserved for possible digitization to replace poor quality or defective electronic copies. his research sought data on the “failure rate” of electronic surrogates of print journals (how often content was incomplete, missing, or illegible), and how often damage or other irregularities in print journals limited their use for digitization purposes. his project compared the content of randomly selected journal titles, volumes and issues from seven e-journal archives and their print equivalents held at the university of saskatchewan library. the archives were obtained from five vendors representing humanities, social sciences, technology, science, and medicine. the study examines several types of failure for print journals and their electronic surrogates, and presents a wealth of data demonstrating the need for preserving print equivalent journal titles for at least the short (less than 5 years) to medium term (up to 10 years), during which time poorly digitized materials can be identified, replaced, and digitally preserved. ladd notes that, while online surrogates of image-rich scholarly papers are more likely to have quality issues, the study found that some text-only pdf scholarly documents were illegible. he cautions against the disposal of text-heavy titles, as our assumptions about the quality of online surrogates may be incorrect. he notes a significant absence of supplemental content in electronic surrogates. finally, his observations about the number of print journals that could not be used for digitization suggest that the number of copies required for a full set of preserved journals over a specific time period may be greater than anticipated. the robert h. blackburn distinguished paper award is named for dr. blackburn, the first president of cacul (1963-64). the award evidence based library and information practice 2011, 6.2 89 annually recognizes notable research published by cacul members. the robert h. blackburn distinguished paper award will be presented at the cacul annual general meeting on friday, may 27, during the cla national conference in halifax, nova scotia. the canadian association of college and university libraries (cacul) is a division of the canadian library association (cla). cla is a national association representing the interests of public, academic, school and special libraries, professional librarians, library workers, and all those concerned about enhancing the quality of life for canadians through information and literacy. for details, visit www.cla.ca. http://www.cla.ca/� / evidence based library and information practice news/announcements ken ladd receives 2011 robert h. blackburn distinguished paper award evidence based library and information practice 2012, 7.2 49 evidence based library and information practice article at your leisure: establishing a popular reading collection at ubc library bailey diers university of british columbia library vancouver, british columbia, canada email: bailey.diers@gmail.com shannon simpson university of british columbia library vancouver, british columbia, canada email: shannon.delay@gmail.com received: 23 nov. 2011 accepted: 21 jan. 2012 2012 diers and simpson. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objectives – this study investigated the leisure reading habits and preferences of students, faculty, staff, and community members at the university of british columbia (ubc) in order to determine if a leisure reading collection would fulfill a need and, if so, what form that collection should take to best serve the population. methods – this study, conducted in october 2010, consisted of a 19-question online questionnaire distributed to a random sample of ubc undergraduate students, graduate students, faculty, and community library users and an identical, open participation questionnaire for the entire ubc community, including staff and community members. in addition to some demographic information, the questionnaire gathered information about leisure reading habits, tendencies, and the participants’ preferences for a potential future leisure reading collection at ubc library. results – there were 467 valid responses out of 473 total responses received from ubc undergraduate students, graduate students, faculty, staff, and community members. of the valid responses, 244 were received from the 1,500 random sample invitations (a 16.3% response rate). additionally, the questionnaire was advertised for open participation for those not invited, resulting in the remaining 229 responses. results of this study indicated overwhelming support for a leisure reading collection mailto:bailey.diers@gmail.com mailto:shannon.delay@gmail.com evidence based library and information practice 2012, 7.2 50 at ubc library, with 94% of respondents stating they might or would use a leisure reading collection. this study also revealed strong leisure reading habits among all response groups. however, only 6% of respondents currently acquire most of their leisure reading materials from ubc library. additional analysis found that ubc library already owns 81% of the titles and authors requested by respondents in the survey. conclusions – based on the findings, the strong support for a leisure reading collection, and the fact that many ubc campus residents are not eligible for a free municipal public library card and borrowing privileges, there is a genuine need for a leisure reading collection at ubc library. the data indicates that if accessible and convenient, a leisure reading collection could provide an opportunity for those who do not already read for leisure to do so. additionally, a ubc library leisure reading collection could attract community members, including those who are not ubc library cardholders. in response to the results of the study, a pilot leisure reading collection was created in september 2011. this will make leisure reading materials easier to access and will allow the library to further analyze the potential of such a collection, ultimately determining its future. introduction it has been identified that reading, in general, has valuable outcomes, and that leisure reading can support an academic library’s educational mission. “leisure reading,” also referred to as pleasure or popular reading in the literature, is used in this article to denote reading fiction or non-fiction books of one’s own accord for pleasure or one’s own enrichment, rather than for work or school. due to the perception that leisure reading collections are outside a typical academic library’s goals and mission, many college and university libraries, including the university of british columbia (ubc) (vancouver, canada), have been hesitant to house a leisure reading collection (dewan, 2010). some at ubc argue that providing leisure reading is the role of a public library. however, in the ubc context, many campus residents, including those living at undergraduate residences, do not have access to free public library services because the campus is not located in the city of vancouver (vancouver public library, 2011), and so there is a potential need for a leisure reading collection at ubc library. the scope of this research was to answer the following: 1. would the implementation of a leisure reading collection fulfill a need among ubc library users? 2. what form should that collection take to best serve the population? the feedback from all user groups – faculty, staff, graduate students, undergraduate students, and community members – helped to develop a broad picture of leisure reading habits and the appeal of a leisure reading collection across all campus groups. conducting a questionnaire of this kind that includes many different constituencies within the ubc community expands the traditional definition of an academic library patron. many studies reviewed for this article have either lacked input from library patrons or focused solely on undergraduate users. this research study aims to fill the void in the existing literature and presents a methodology that can be easily replicated at other institutions. literature review history of leisure reading collections historically, promoting leisure reading was an important function of academic librarians. according to zauha (1993), it was common for libraries in the 1920s and 1930s to have several recreational reading collections or “browsing collections” throughout the campus in libraries, dormitories, and student union buildings. since then, the roles and mission of evidence based library and information practice 2012, 7.2 51 the academic library have migrated toward a more research-focused collection built on university curricula, resulting in a steady decline in browsing collections (zauha, 1993). leisure reading habits among students it is important to consider the reading habits and interests of patrons in order to build a successful leisure reading collection. the literature on leisure reading habits of students is rather uneven, mainly due to the varying definitions of leisure reading and what formats constitute leisure reading. salter and brook (2007) suggest that literary reading is in decline whereas general reading is on the rise. they attribute this rise to technology and specifically the general reading that is conducted on the internet. they conclude that students do read and it is one of the choices they make for their leisure time, although it is not the predominant choice. the top choice for leisure time among the students surveyed by salter and brook is watching television or movies. while it would have been helpful for our purposes to see how available leisure reading materials were to these students, such as proximity of public libraries or popular collections within their academic libraries, salter and brook’s questionnaire and sound methodology are valuable resources and can be applied to explaining how academic libraries can use their resources to meet the recreational demands of their users. academic uses for leisure reading collections while leisure reading collections are often created for library patrons’ recreational pursuits, there has been some research on how creating and exhibiting bestseller collections can promote academic research on past and current popular culture. clendenning (2003) specifically addresses the creation of exhibits to promote and celebrate popular books in an academic environment for the purpose of academic research. similarly, crawford and harris (2001) argue that academic libraries should consider establishing bestseller collections as a resource for future popular culture studies. like crawford and harris, van fleet (2003) underscores the importance of collecting popular fiction for the sake of popular culture scholars, but also believes that providing popular fiction collections for leisure reading is necessary on its own merits. collection policies little has been written discussing the presence of leisure reading collection guidelines in academic library collection policies. hsieh and runner (2005) surveyed 99 academic libraries in the united states and reviewed 30 collection development policies to see which type of library is most likely to have a policy on leisure reading, if and how patrons influence the collection, and additional “environmental factors,” including public library access for students and the percentage of students living on campus. they found that in total, only 14% of all libraries surveyed have “no-purchase” policies for leisure reading materials and that most of those with such policies resulted from either budget constraints or the library’s stance that leisure reading is not part of the library’s mission. while hsieh and runner’s article did not discuss collection development policies that support leisure reading purchases, they did state that an increasing number of academic libraries are beginning to adjust their policies to include leisure reading material. models for leisure reading collections the success or failure of various leisure reading collection models and formats such as e-books is also not heavily discussed in the current literature. one model that many libraries use is a book-leasing plan. zauha’s (1998) article comparing brodart’s mcnaughton lease plan and baker & taylor’s book leasing program, the two most popular book-lease plans, is important for its outline of what libraries need to consider before embarking on a leased collection project. zauha also addresses the disadvantages of a leased popular reading collection, namely that such leasing programs are susceptible to being cut when budgets are tight. similarly, odess-harnish (2002) conducted a survey of 22 academic libraries in the united evidence based library and information practice 2012, 7.2 52 states who use the mcnaughton lease plan for 200-1,000 titles, in addition to or instead of purchasing books. her study examined why the lease option was chosen, how successful it had been, and other factors that may have affected the use of the collection. overall, odess-harnish found that libraries utilizing leased collections had positive responses; 54% of the libraries surveyed reported circulation statistics at least as high as originally expected and 23% reported that the titles circulate more than expected. with recent developments in e-book technologies, libraries are also beginning to consider the use of e-readers for leisure literature collections. while there is emerging research on e-books and users’ reactions to ebooks in academic environments, such as rowlands, nicholas, jamali, and huntington (2007), there is very little on e-readers and their application in libraries in general. in an experiment at penn state university libraries, sony collaborated with the school to provide 100 e-readers for one academic school year. behler (2009) discusses the results of the experiment and the techniques used to gather feedback on the devices. housing a leisure reading collection when embarking on creating a popular reading collection, libraries need to think about not only what type of model they will follow and what type of materials they will acquire, but where the collection will be located and how the physical space will appear. research on design of the physical space of a leisure reading collection constitutes a gap in the literature. woodward (2009), in her book, creating the customer-driven academic library, dealing with the physical space of academic libraries, suggests a location for the popular reading collection (near the café) and mentions the need to include books that appeal to undergraduates. she does not specifically discuss the physical space of a popular reading collection and how to entice non-undergraduate users. leisure reading collection user studies looking back at how successful leisure reading collections have functioned at other institutions is an excellent way to develop a vision of future leisure reading collections. rathe and blankenship (2006) at the university of northern colorado conducted a survey among users of their year-old popular reading collection and found that while the popular reading collection is mostly used for leisure, 11% of respondents noted that they used the collection for class work as well. other studies, such as sanders (2009), have surveyed both users and library staff. a common theme found in such studies is that library staff members’ perceptions of their leisure reading collection often are not supported by user experiences or usage data. summary of literature review in general, there is a large chronological gap in the literature about leisure reading and leisure reading collections in academic libraries. there is some discussion of leisure reading collections and the academic library’s role in promoting reading before 1940, but since academic libraries transitioned around 1940 to more directly supporting curricula and public libraries took over the recreational reading role, the literature has declined sharply. discussion of popular reading collections in academic libraries has slowly been increasing in the last 20 years as leisure reading collections have gained more support, but much work remains to be done in this context (dewan, 2010). in addition to the lack of research on leisure reading collections in academic libraries, the findings in this area are often contradictory, because “reading” and specifically “leisure reading” are defined differently in every study. because of the variety of definitions and conclusions, existing studies on literacy and leisure reading need to be read very carefully as a whole. evidence based library and information practice 2012, 7.2 53 overall, the literature is supportive of leisure reading collections, but ignores opponents of such collections in academic libraries. the negative aspects that may be involved are barely discussed, especially regarding changes in workflow, increases in workloads, and allocation of budgets and resources, all of which are important considerations for any library. van fleet (2003) and elliott (2007) briefly mention the challenges of maintaining popular reading collections, but their statements are based on opinion rather than evidence. additionally, research based on questionnaires given out to users of leisure reading collections usually focus on student patrons, often ignoring faculty, staff, and other users. it is important to address these users’ needs, especially since both rathe and blankenship (2006) and odess-harnish (2002) noted that faculty and staff were initially the largest group of users for their leisure reading collections. looking at who uses leisure reading collections, why they use them, and what they would like to see done differently are useful questions that have yet to be addressed in the literature. previous studies are narrow in scope, excluding student views and surveying users only after the leisure reading collection was established. this study differs in two regards. first, the breadth of those invited to participate – the inclusion of students, as well as staff, faculty, and community members – exceeds that of other studies. secondly, surveying potential users before the collection is created is an excellent opportunity to see how well users’ perceived needs can be met with such a collection and will provide a baseline for future assessment. this study, therefore, aims to fill the above gaps in the literature. additionally, the lack of generalizable results and contradictions among local surveys such as rathe and blankenship (2006) and sanders (2009) reinforces the inability of the literature to predict local needs, and therefore required us to conduct our own study relevant to the ubc context. methods in the spring and summer of 2010, we worked to assemble the questions for the leisure reading online questionnaire (see appendix). the questions were formulated from information gleaned from the literature review and derived from our research questions. we used the vovici efm survey tool to create an online version of the questionnaire. quantitative analysis was done using spss and qualitative responses were coded with atlas.ti. three pilot tests of the questionnaire, consisting of a total of five participants – representing community members, faculty, and students – were held to determine the quality and flow of the survey. additionally, we met with the ubc library collections advisory and development committee to receive library staff input. discussing the questionnaire with the pilot participants and library staff was invaluable for the revision process, resulting in substantial changes to the wording and order of questions. two different methods for sampling the various user populations were used. the first included targeting participants based on a random sample. the second method was advertising the questionnaire for open participation, which allowed participants not included in the random sample to complete the survey. the 19-question questionnaire was a conditional survey in that participants who indicated they would not use a leisure reading collection were asked a final follow-up question, whereas those who indicated they would or would maybe use a leisure reading collection were asked additional questions regarding space and format preferences for leisure reading. by eliminating those who would not use the leisure reading collection, we were able to isolate the responses pertaining to the collection itself to those users who would use the collection. the majority of the questions asked were multiple-choice and covered demographic information, participants’ leisure reading habits and evidence based library and information practice 2012, 7.2 54 preferences, and methods used to obtain leisure reading materials. open-ended questions were also included for participants to share their opinions and even note particular authors, titles, and genres they would like to see in a leisure reading collection. this study received approval from the university of british columbia’s ethical review board. the questionnaire ran for two weeks at the end of october 2010. an email reminder was sent to all invited participants one week into the study, with one week remaining. sampling methods invited sample an email with a link to the online questionnaire was sent to a random sample of each of the user groups – ubc undergraduate students, graduate students, faculty, and library community members (individuals in the vicinity with ubc library cards). due to ubc library survey procedures, our sample size was restricted to a maximum of 1,500 and no individual group could exceed 500. working within these parameters, we set sample sizes for each population in order to achieve enough responses for acceptable confidence levels and intervals. a base of 500 was set for undergraduates, the largest population on campus. the other groups were selected in increments of 100 relative to their campus population (see table 1). in all, 1,500 email invitations were sent. open sample additionally, the survey was advertised for open participation on both the library website “carousel,” a rotating image of library news and events, and in the local university neighbourhoods association community online newsletter. the open participation survey was a separate but identical survey included to capture responses from groups like ubc staff and community members that could not be directly targeted. limitations it is important to note that we realize the survey is somewhat biased towards support for a leisure reading collection. when presented with an added service, it is likely that many people will positively respond. additionally, the data suggest that those respondents who replied to the open participation survey – who are likely already library users, considering many accessed the survey through the library website – were more likely to say that they would definitely use a leisure reading collection compared to those who were randomly selected to take the survey (see figure 1). where appropriate, we identify the differences between the two samples in our discussion. table 1 sample size chart population size number of survey invitations sent respondents (includes open survey) confidence level confidence interval undergrads 37,781 500 140 95% 8.27 graduates 9,008 400 133 95% 8.44 faculty 4,502 300 56 95% 13.02 total of known populations 51,291 1,200 329 95% 5.39 community & other unknown 300 100 staff unknown not included in sample 37 evidence based library and information practice 2012, 7.2 55 results and discussion in all, 1,500 email invitations were sent and 244 completed responses were received (16.3% response rate). while this response rate may appear low, it is consistent with response rates of other surveys conducted by the university library, including libqual+® 2010 (university of british columbia library, 2010a). the open participation survey, a separate but identical survey that was included to capture responses from groups like ubc staff and community members that could not be directly targeted, resulted in an additional 229 responses, contributing to the final total of 473 responses. due to the similarity of the responses from both the open and invited groups, we do not feel that the seemingly low response rate has an effect on the overall results. the respondents from the open survey did indicate higher amounts of leisure reading than those who took the invited survey, so it is possible that some people who were already excited about leisure reading self-selected to participate in the open call. however, when comparing the open and invited responses, there are no significant differences between answers to questions pertaining to the potential leisure reading collection at ubc. all results provided in this discussion are taken from the total valid responses. overall, there was strong indication that a leisure reading collection would be used: 62% of respondents stated that they would check out books from a leisure reading collection, 32% said they maybe would check out books from a leisure reading collection, and 6% indicated they would not use a leisure reading collection. respondents who live on campus were 20% more likely to state they would use a leisure reading collection. currently, bookstores, public libraries, and the internet are the main sources of leisure reading materials (in order from most responses to least). only 6% (invited survey 3.2%, open survey 8.6%) of respondents get most of their leisure reading materials from ubc library, presenting an opportunity for ubc library to increase its circulation and patron base through fulfilling leisure reading needs. demographics the demographic breakdown between the open and invited surveys differed somewhat. graduate students made up the majority of the invited survey results (34.6%), while undergraduate students made up the majority of the open survey results (36.3%). faculty made up a greater percentage of the invited survey (17.5% vs. 6.2%), whereas more staff responded to the open survey (15% vs. 0.8%). figure 1 invited and open participation survey responses: support for a leisure reading collection evidence based library and information practice 2012, 7.2 56 more community members, which included university alumni, responded to the invited survey than the open survey (20.5% vs. 18.8%). members of the faculty of arts comprised the majority of responses, followed by the faculty of science. these are the two largest faculties at ubc. there were 29% of respondents who do not have a local public library card. of the total number of respondents, 25% live on the ubc campus; of these, 41% do not have a public library card, indicating that while leisure reading may be outside the traditional academic library’s purview and a public library responsibility, in the ubc campus context, public library access is limited. therefore, the academic library is the primary source for borrowable leisure reading resources. reading habits only 6% (invited survey 8.3%, open survey 3.5%) of the population stated that they did not read any books for leisure in the three months prior to the questionnaire, whereas 26% (invited survey 18.3%, open survey 33.6%) of respondents read more than six books for leisure in the same time period. when asked how many hours per week respondents read for leisure, only 3% (invited survey 4.6%, open survey 1.3%) indicated that they did not read for leisure at all and 49% (invited survey 41.5%, open survey 56%) of respondents read three or more hours per week. those who do not read for leisure are in the minority and in fact the vast majority of respondents read for leisure. figure 2 average hours per week of leisure reading by group (combined sample) evidence based library and information practice 2012, 7.2 57 undergraduate students since most of the literature on leisure reading collections in academic libraries has focused on undergraduates, our undergraduate data can be compared with other studies. gallick (1999) found that when undergraduates were in session, only 37% read more than two hours per week for leisure. our study, with an almost identical response rate as gallick’s, actually found that 48% of undergraduates read two or more hours per week for leisure. additionally, our study shows that over 92% of undergraduate students read for leisure in some form (see figure 2). interestingly, while only 10 total undergraduate students stated that they do not read for leisure, 9 of those students said that they would maybe or definitely use a leisure reading collection. this suggests that a leisure reading collection could entice those undergraduates who currently do not read for leisure to perhaps devote more time to leisure reading. graduate students our survey indicates that graduate students read for leisure even more than undergraduates, with 62% reading two or more hours per week. even though it is perceived that time inhibits leisure reading, especially for graduate students, our findings indicate that most graduate students do make time for leisure reading. faculty and staff of all of the user groups, faculty and staff spend the most time reading for leisure. there were 85% who read two or more hours per week and none indicated that they do not read for leisure. this would make faculty and staff prime leisure reading collection users to target based on their leisure reading habits. community community members included alumni, ubc library community cardholders, and ubc campus residents (non-students). similar to faculty and staff, there weren’t any community members who indicated they do not read for leisure and 76% read two or more hours per week. while 42% of these community members do not have an active ubc library card, 96% indicated they would or would maybe use a leisure reading collection at ubc library, suggesting that community members could be an untapped patron base for the academic library. genres and titles following salter and brook’s (2007) lead, we asked respondents what genres, titles, and authors they would like to see in a potential leisure reading collection. these lists are extremely helpful in providing a direction for selection procedures and making specific selections for a leisure reading collection. by including user-suggested authors and titles in the leisure reading collection, we hope to increase the success of the collection and reveal the library to be responsive to user requests. bestselling fiction, award winners, bestselling non-fiction, classics, historical fiction, biographies, science fiction, mystery/suspense, and short stories were the most prominent genre choices for a leisure reading collection. however, some respondents suggested that the ubc library could differentiate its leisure reading collection from public libraries and bookstores by promoting books of critical acclaim rather than popular appeal. while there is concern from some about the overlap of a leisure reading collection at ubc with the collections in the public libraries, this concern may not be wholly applicable in the ubc context because some individuals in the university community have no free access to public libraries. many specified a desire for the collection to include magazines such as the new yorker, the economist, vanity fair, and reader’s digest. respondents also mentioned preferring paperbacks and making sure there are enough copies of popular books to reduce wait time. when they were asked what languages would be desired, english was overwhelmingly the language of choice. according to our collection analysis, ubc library owns 81% of the specific titles and authors mentioned by respondents. evidence based library and information practice 2012, 7.2 58 considering that only 6% of the respondents obtain most of their leisure reading resources from ubc library, it is possible that many are simply not aware of the materials, particularly in regard to leisure reading, at ubc library. the fact that leisure reading materials are not merchandized and not easily browsable means that the patron looking for these materials must know specific titles or authors, search the catalogue for call numbers, and then locate various titles on dispersed shelves. this process does not naturally lend itself to leisure reading needs. the results of our genre, title, and author analysis suggest that ubc library could utilize many resources already owned to meet many of the leisure reading desires of its users by creating a separate browsable collection. format print was overwhelmingly the format of choice for a leisure reading collection, with 97% of respondents indicating a preference for print, and 39% indicating some interest in an ereader format. the kindle, ipad, and ipod touch were the most favoured, but overall there was substantial variation on the brand of e-reader preferred. while providing e-formats for a leisure reading collection might be something to consider in the future, the lack of consensus on format or device makes it difficult to pursue. look and feel the two largest and most frequented libraries on campus were the preferred locations for a leisure reading collection. most requests about the collection space itself were for a separate leisure reading area with comfortable couches and chairs and good lighting. a coffee shop or coffee vending machine is also highly desirable. qualitative data regarding access to a leisure reading collection refer primarily to the need for the collection to be in a central location that can be easily browsed both physically and online. in addition, several respondents stated the desire to have leisure reading materials available electronically because of their distance from or infrequent visits to campus. using the online catalogue and browsing the physical shelves are the preferred methods of searching for books in the leisure reading collection, according to the quantitative data. some, particularly graduate students, expressed interest in virtual bookshelves such as librarything or goodreads. since our survey did not define what a virtual bookshelf is, this method for browsing may be underrepresented. overall perceptions positive sentiment response to the survey was strongly positive. many stated in the additional comments that a leisure reading collection would be an “excellent idea” or a “great initiative.” people feel leisure reading is an important activity and find that ubc library currently does not satisfy their leisure reading needs. several respondents went as far as to suggest ideas for implementing a leisure reading collection at ubc, including building the collection with donations, having a shorter loan period, and developing a plan for keeping the collection sustainable. in addition to providing ideas for building a successful collection, respondents gave input on programs and techniques to promote the collection: book clubs, author talks, staff picks, themes, and displays. many respondents indicated that a leisure reading collection would be a worthwhile addition to ubc library if there was a large selection and the collection was properly promoted. negative sentiment odess-harnish (2002) found that of the 22 staff, all from different libraries, who responded to her survey, none believed that they should be doing more to support or collect popular reading titles. in that study, library staff reported the following reasons: there are time, budget, and space constraints; the books are not a part of the curriculum; there is no interest or demand from the patrons; students do not have time for pleasure reading; and if they so desire, there is a public library nearby. these claims mirror evidence based library and information practice 2012, 7.2 59 the negative sentiments found in our qualitative data among all respondents. in our study, even some of the 94% of respondents who stated they would or would maybe use a leisure reading collection voiced concerns that leisure reading does not fall under the academic library’s mission, the library does not have the funding for such a collection, the collection could not be comprehensive enough to suit the community’s needs, and patrons simply do not have time for leisure reading. those who would not use such a collection were not against the idea of a leisure reading collection, but simply cited lack of time as their main reason for answering no. this suggests that if a leisure reading collection were established in a location convenient and accessible to these individuals, they might find they do have time for leisure reading. implications for practice: establishing a pilot collection the results of this study have led to the development of a pilot leisure reading collection (called great reads) at ubc library. the collection is populated with the genres most requested in the survey. using our collection analysis, which found that the library already owned the majority of titles requested by respondents, we were successfully able to populate the collection with books the library already owns. as of spring 2012 the great reads collection has been well received. the success of the collection pilot at the main library has led to a second collection, and a third great reads collection will be opening in the summer of 2012. conclusions overall, this study suggests strong leisure reading habits among all user groups, as well as support for a leisure reading collection at ubc library. the data indicates that if provided in an accessible and convenient form, a leisure reading collection could provide an opportunity for those who do not read for leisure to do so. additionally, a ubc library leisure reading collection could attract community members, especially those who are not current ubc library cardholders. within the ubc context, this would further the strategic direction outlined in the library’s strategic plan to engage with the community (university of british columbia, 2010b). the next step in this project is to conduct a comprehensive evaluation of the pilot leisure reading collections implemented in response to this study and assess their success through circulation analysis and user feedback. though changes in collection policies may be necessary for the long-term maintenance of such a collection, the leisure reading pilot currently underway is using only titles ubc library already owns, merchandized in a separate, browsable leisure reading collection space. it is hoped that the assessment of the pilot can be compared with the results of this study to provide a before-and-after picture of leisure reading at an academic library. acknowledgements we would like to acknowledge the ubc library for its assistance on this project as well as rick kopak and jo anne newyear-ramirez for their encouragement and support throughout this process. references behler, a. (2009). e-readers in action: an academic library teams with sony to assess the technology. american libraries, 40(10), 56-59. clendenning, l. f. (2003). rave reviews for popular american fiction. reference & user services quarterly, 42(3), 224-228. crawford, g. a., & harris, m. (2001). bestsellers in academic libraries. college & research libraries, 62(3), 216-225. dewan, p. (2010). why your academic library needs a popular reading collection now more than ever. college & undergraduate libraries, 17(1), 44-64. evidence based library and information practice 2012, 7.2 60 elliott, j. (2007). academic libraries and extracurricular reading promotion. reference & user services quarterly, 46(3), 34-43. gallick, j. d. (1999). do they read for pleasure? recreational reading habits of college students. journal of adolescent & adult literacy, 42(6), 480-488. hsieh, c., & runner, r. (2005). textbooks, leisure reading, and the academic library. library collections, acquisitions, and technical services, 29(2), 192-204. odess-harnish, k. (2002). making sense of leased popular literature collections. collection management, 27(2), 55-74. rathe, b., & blankenship, l. (2006). recreational reading collections in academic libraries. collection management, 30(2), 73-85. rowlands, i., nicholas, d., jamali, h. r., & huntington, p. (2007). what do faculty and students really think about ebooks? aslib proceedings, 59(6), 489-511. salter, a., & brook, j. (2007). are we becoming an aliterate society? the demand for recreational reading among undergraduates at two universities. college & undergraduate libraries, 14(3), 2743. doi:10.1300/j106v14n03_02 sanders, m. (2009). popular reading collections in public university libraries: a survey of three southeastern states. public services quarterly, 5(3), 174-183. university of british columbia library. (2010a). ubc library libqual+® results data 2010. unpublished raw data. university of british columbia library. (2010b). ubc library strategic directions, goals, and actions. retrieved 1 feb. 2012 from http://strategicplan.library.ubc.ca/files/ 2010/03/ubc-library-strategicdirections.pdf vancouver public library. (2011). how to get a library card. retrieved 1 feb. 2012 from http://www.vpl.ca/library/details/how _to_get_a_library_card van fleet, c. (2003). popular fiction collections in academic and public libraries. the acquisitions librarian, 15(29), 63-85. doi:10.1300/j101v15n29_07 woodward, j. a. (2009). creating the customerdriven academic library. chicago, il: american library association. zauha, j. m. (1993). recreational reading in academic browsing rooms: resources for readers’ advisory. collection building, 12(3-4), 57-62. doi:10.1108/eb023344 zauha, j. m. (1998). options for fiction provision in academic libraries: book lease plans. the acquisitions librarian, 10(19), 45-54. doi:10.1300/j101v10n19_04 evidence based library and information practice 2012, 7.2 61 appendix questionnaire ubc library leisure reading study tell us what you want in your library! we invite you to participate in this questionnaire as we consider the possibility of building a leisure reading collection at ubc. university of british columbia library leisure reading study participant consent form principal investigators: dr. rick kopak, school of library, archival & information studies at the university of british columbia. he can be reached at 604-822-2898. co-investigator(s): bailey diers and shannon simpson, school of library, archival & information studies at the university of british columbia and jo anne newyear ramirez at the university of british columbia library. bailey or shannon can be reached at 604-822-2404. this research is being undertaken to fulfill a course requirement for a graduate degree within the school of library, archival & information studies program. the university of british columbia library will be given the aggregated results of this questionnaire to better enhance library services. purpose: you are being invited to take part in this research study because you are a member of the university of british columbia community. the university of british columbia library is considering the possibility of developing a leisure reading collection. we are investigating the need for a leisure reading collection and determining what users would like to see in such a collection. study procedures: if you are at least 19 years of age and agree to participate in this study, you will have the opportunity to fill out an online questionnaire. this questionnaire should take no more than 10 minutes; most participants finished the questionnaire in about 5 minutes. potential risks: this questionnaire will not pose any risks greater than you would incur with normal computer use. potential benefits: by participating in this questionnaire, you may directly benefit by helping contribute to the university of british columbia library collections. if you would like to receive the results of the study after the completion of this study, please email delay@interchange.ubc.ca. confidentiality: we do not ask for any identifying information in the questionnaire and there will be no way to connect you to your questionnaire results. your email address, which is not connected to your responses, will only be used to enter you in the prize drawing and will not be used for any other purposes. evidence based library and information practice 2012, 7.2 62 remuneration/compensation: in thanks for taking the time to complete the questionnaire, you will have the opportunity to win one of four $50.00 gift cards to chapters bookstores. please be aware that if you exit or close the questionnaire window you will not be entered in the drawing. if you wish to withdraw from the survey at any time and still would like to be considered for the prize, you may proceed to the end of the survey to click submit. contact for information about the study: if you have any questions or desire further information with respect to this study, you may contact bailey diers or shannon simpson at delay@interchange.ubc.ca. contact for concerns about the rights of research subjects: if you have any concerns about your treatment or rights as a research subject, you may contact the research subject information line in the ubc office of research services at 604-822-8598 or if long distance e-mail to rsil@ors.ubc.ca or toll free 1-877-822-8598. consent: your participation in this study is entirely voluntary and you may refuse to participate or withdraw from the study at any time. 1) do you consent to participate in this questionnaire?  yes, i consent to the terms above and am at least 19 years of age.  no, i do not consent to the terms above or i am under the age of 19. (by clicking next page you will exit the survey) 2) please select your ubc status.  undergraduate student  graduate student  faculty  staff  community member (without active ubc library account)  community member (with active ubc library account)  ubc alumnus (without active ubc library account)  ubc alumnus (with active ubc library account)  other (please specify) if you selected other, please specify ______________________________________________________________________ 3) please select your faculty.  does not apply  applied science, faculty of  arts, faculty of  business, sauder school of  education, faculty of  forestry/land and food sciences, faculties of  health sciences  law, faculty of  science, faculty of evidence based library and information practice 2012, 7.2 63  undecided  other (please specify) if you selected other, please specify ______________________________________________________________________ 4) do you live on the ubc campus or ubc endowment lands?  yes  no 5) do you have a public library card from vancouver public library or any public library in the lower mainland?  yes  no 6) definition of leisure reading: in this survey, leisure reading means fiction or non-fiction books to be read for the sake of reading, on one’s own accord for pleasure or one’s own enrichment, and not for work or a class. how many books (excluding school or work related books) have you read for leisure in the past 3 months?  none  1-2  3-4  5-6  more than 6 7) on average, how many hours do you spend reading for leisure each week?  i don’t read for leisure  less than 1  between 1-2  between 2-3  between 3-6  more than 6 8) if ubc libraries had a leisure reading collection would you check out books from this collection?  yes [respondent will skip question #9]  maybe [respondent will skip question #9]  no [respondent will be sent to question #9] 9) since you selected no, is there anything that would make a potential leisure reading collection appeal to you? __________________________________________________________________________________________ __________________________________________________________________________________________ ______________________________________________________ evidence based library and information practice 2012, 7.2 64 10) where do you get most of your leisure reading material?  ubc library  vancouver public library  other public library  bookstores  online  other (please specify) if you selected other, please specify ______________________________________________________________________ 11) on average, how many of your leisure reading books do you get from ubc libraries?  none  some  most  all 12) what library on campus would you prefer to use for browsing and checking out leisure reading books? (select all that apply)  asian library  david lam library  education library  irving k. barber learning centre  koerner library  law library  music library  woodward library  xwi7xwa library  other (please specify) if you selected other, please specify ______________________________________________________________________ ______________________________________________________________________ 13) how would you prefer to search for books within a leisure reading collection? (select all that apply)  online ubc library catalogue  virtual bookshelf (i.e. librarything, goodreads)  browse the physical shelves of the collection  other (please specify) if you selected other, please specify ______________________________________________________________________ ______________________________________________________________________ 14) which genres from a potential ubc library leisure reading collection would you be interested in reading? (select all that apply)  award winners  bestselling fiction evidence based library and information practice 2012, 7.2 65  bestselling non-fiction  biographies  children’s  classics  comic books or graphic novels  fantasy  historical fiction  how-to  mystery/suspense  poetry  romance  science fiction  self-improvement  short stories  young adult  other (please specify) if you selected other, please specify ______________________________________________________________________ 15) which languages in a potential ubc leisure reading collection would you be interested in reading? (select all that apply)  chinese  english  french  japanese  korean  persian  punjabi  spanish  tagalog  vietnamese  other (please specify) if you selected other, please specify ________________________________________________ 16) what format of leisure reading would you prefer? (select all that apply)  printed books  e-books on a computer  e-books on your personal e-reader  e-books on an e-reader that you check out from the library  audiobooks 17) if you selected e-readers as your response in the previous question, what brand of reader do you use or would you prefer to use?  ipad  ipod touch  kindle  nook evidence based library and information practice 2012, 7.2 66  sony e-reader  none  other (please specify) if you selected other, please specify ______________________________________________________________________ 18) please recommend any authors or types of books that you would like to see in a potential leisure reading collection: __________________________________________________________________________________________ __________________________________________________________________________________________ ______________________________________________________ 19) please let us know of anything else you think we should consider in providing a leisure reading collection at ubc. __________________________________________________________________________________________ __________________________________________________________________________________________ ______________________________________________________ thank you for participating in this questionnaire. we appreciate your feedback. you will be entered into a randomly selected drawing for a chance to win a gift certificate to chapters bookstores. if you have any comments or questions regarding this questionnaire, please contact shannon simpson. evidence summary   continuing education and data training initiatives are needed to positively impact academic librarians providing data services   a review of: fuhr, j. (2022). developing data services skills in academic libraries. college & research libraries, 83(3), 474. https://doi.org/10.5860/crl.83.3.474   reviewed by: nandi prince assistant professor ursula c. schwerin library new york city college of technology new york, new york, united states of america email: nandi.prince25@citytech.cuny.edu   received: 1 june 2023                                                               accepted:  11 july 2023      2023 prince. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30382     abstract   objective – to measure the existing data services skills of academic librarians and gather information on the preferred training programs available to enhance those skills.   design – survey questionnaire.   setting – libraries in canada, the united states, the united kingdom, and australia.   subjects – one hundred and twenty respondents who self-identified as providing data services. most (85%) worked in academic libraries with 7% in hospital libraries, 3% in government libraries and 5% in other types of libraries.   methods – permission was received from the institution ethics board to administer an incentivized survey. all respondents received a 22-question survey which consisted of a mix of likert-scale questions, multiple choice, open-ended, and short answer questions. the survey was open for two months, beginning on february 20, 2020. one hundred and twenty responses were collected from librarians. a regression analysis was run for the four-skill set categories: general data services, programming languages and software, library instruction, and soft skills. the four variables measured were: geographic region, percentage of time spent performing data management services, length of time served in the data services role, and overall length of time spent in the library science field.   main results – the strongest data services skill sets were soft skills and instruction. the weakest skill set was programming languages and software. the more time a librarian spent providing data services, the higher their self-assessed score was for programming languages and software and general data services. librarians from the united states rated themselves higher than canadian librarians in data analysis software, data visualization, data mining, programming languages, text editors and project management. preferred forms of professional development were learning by doing and self-directed learning. biggest impediments to professional development were lack of time (34%), high cost (28%), and lack of support from administrators and supervisors (26%). qualitative comments revealed challenges related to a lack of support, a lack of direction, and a lack of defined roles.   conclusion – the survey revealed that additional training and development skills initiatives are necessary for practitioners supporting data services in academic libraries. academic data librarianship is an emerging field with vaguely articulated roles for the data practitioner in a broad range of settings. furthermore, the skills and training needed are not clearly defined. the standardization of education, training and the core competencies needed for the mechanics of the roles are challenging to define because of diversity within the field. libraries embarking on providing data management services need to explore what services their community of researchers needs and plan to equip their staff with appropriate skill sets.   commentary   the author provided an overview of the issues pertaining to the emerging field of data librarianship and established the significance of the study to the profession. the survey advances knowledge in this emerging field and brings attention to uncertainties surrounding the role of academic librarians rendering and supporting new data services for researchers. the validity of the study was evaluated using glynn’s tool, and found to be acceptable (glynn, 2006). the author had a data-centric approach and attempted to gather data from a large population set, as evidenced by the census approach. they aimed to be inclusive by recruiting at academic libraries and beyond, and by recruiting librarians and other practitioners engaged in data services.   the methodology selected by the author was suitable for achieving the aims of the study; the questions asked were clearly defined and the charts in the “findings” section provided a good visual sequence to interpret the four high-level categories of the study. the survey used for obtaining the data is consistent with accepted practices in the library and information studies (lis) field. additionally, the survey can be replicated because the survey tool questions were appended. however, it is not clear if respondents were functioning in the role of data librarian or if they were hired as an expert in other data-related roles to carry out data services work.   the author noted limitations and areas where more research is needed. the sample size was small, especially regarding representation from the united kingdom and australia. additional studies would extend this research work. exploring data service trends in academic libraries was one of the author’s objectives.  the study analyzed the length of time librarians spent providing users with this service, and the self-assessed level of proficiency of the respondents. although the results of the study are significant, reliance on self-reported data is an area of concern because of the potential for respondents’ own interpretations and biases. the author did not include the demographic data of survey respondents, such as gender. this omission may impact the results of the analysis.   this study will help librarians of all experience levels to better understand the work of data librarianship. administrators who are planning on expanding data services to their research community may use this study to identify core competencies needed by librarian staff as the author outlines specific skill sets needed. the insights gained are particularly useful because they are the perceptions of current practitioners performing diverse work in this emerging field.   references   fuhr, j. (2022). developing data services skills in academic libraries. college & research libraries, 83(3), 474. https://doi.org/10.5860/crl.83.3.474   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154.     microsoft word news4_1473_final_final evidence based library and information practice 2008, 3:1  87 evidence based library and information practice       news     2008 chla/absc conference announcement      © 2008. this is an open access article distributed under the terms of the creative commons attribution  license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and  reproduction in any medium, provided the original work is properly cited.       iʹm pleased to announce that registration for  the 2008 chla/absc conference to be held  in halifax, ns, from may 26 to 30, is now  open. visit the conference website at  http://www.chla‐absc.ca/2008/ and click on  ʺregistrationʺ. the rate for the full  conference for members is $300 for early  bird registration. after april 11, a late  charge of $80 will be charged.    ce details for most sessions are now on the  website. please note that registration for  each of these sessions is limited to 25. a  wide array of topics is being presented.    project management  joanne fraser     learning styles: are you smarter than a  millennial? the challenges of generations  working and learning together  daniel phelan  and sarah wickett     getting started in research  andrew booth    evidence‐based practice   ann mckibbon    grey matters! finding grey literature  sarah normandin  and amanda hodgson     canadian copyright law: current issues for  librarians  teresa scassa    current awareness tools – web 2.0   ryan deschamps and kelli wooshue    creating online tutorials  gwendolyn macnairn     come for the programme, come for the  social events, come for the lobster, come for  the history, come for the famous down east  hospitality! a lot of effort is being expended  to ensure this yearʹs conference will be a  very successful one.    tim ruggles  publicity committee  chla/absc 2008             evidence based library and information practice 2012, 7.4 96 evidence based library and information practice evidence summary graduate students report strong acceptance and loyal usage of google scholar a review of: cothran, t. (2011). google scholar acceptance and use among graduate students: a quantitative study. library and information science research, 33(4), 293-301. doi: 10.1016/j.lisr.2011.02.001 reviewed by: lisa shen business reference librarian sam houston state university huntsville, texas, united states of america email: lshen@shsu.edu received: 2 july 2012 accepted: 4 nov. 2012 2012 shen. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to determine the frequency of graduate students’ google scholar usage, and the contributing factors to their adoption. the researchers also aimed to examine whether the technology acceptance model (tam) is applicable to graduate students’ acceptance of google scholar. design – web-based survey questionnaire. setting – the survey was conducted over the internet through email invitations. subjects – 1,114 graduate students enrolled at the twin cities campus of the university of minnesota. methods – 9,998 graduate students were invited via email to participate in a study about their perceptions of google scholar in the fall of 2009. a follow-up email and a raffle of two $25 gift certificates were used to provide participation incentive. the survey measurements, which consisted of 53 items in 15 questions, were based on modifications to the validated tam using measurements adopted by other studies using the same instrument. each item was scored using five-point scales ranging from 1 (“strongly disagree”) to 5 (“strongly agree”). because the tam model is based on direct user experience, only responses from those who have used google scholar in the past were included in the data analysis. mailto:lshen@shsu.edu http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/ evidence based library and information practice 2012, 7.4 97 main results – the survey had a response rate of 11.4%, with 73% of the respondents reporting having used google scholar at least once before. however, only 45% of those who had used google scholar reported linking to full text articles through the customized library link “frequently or always.” on average, respondents found google scholar easy to use (m=4.09 out of 5) and access (m=3.86). they also perceived google scholar as a useful resource for their research (m=3.98), which enhanced their searching effectiveness (m=3.89). however, respondents were less enthusiastic when asked whether they often found what they were looking for using google scholar (m=3.33) or whether it had enough resources for their research (m=3.14). nonetheless, most still felt they made the correct decision to use google scholar (m=3.94), even if their loyalty towards google scholar was limited (m=3.23). the researcher categorized survey measurements into 9 tam-based variables and performed regression analysis (all with p<0.001) to analyze the relationships. overall, accessibility (β=0.32) and system quality (β=0.53) were significant determinants of respondents’ perceived ease of use of google scholar, while perceived ease of use (β=0.33) and comprehensiveness (β=0.53) were significant determinants of respondents’ perceived usefulness of google scholar. in turn, perceived usefulness (β=0.45), loyalty (β=0.38), and perceived ease of use (β=0.12) were the main factors contributing to respondents’ actual intention to use google scholar. lastly, respondents’ loyalty towards google scholar was largely attributed to their satisfaction with the search engine (r²=0.532). conclusion – this study found several factors that strongly influence graduate students’ intention to use google scholar, including students’ perceived usefulness of google scholar, their sense of loyalty towards the search engine, and its perceived ease of use. moreover, the findings also showed that tam is an applicable model for explaining graduate students’ use of google scholar. these findings provide useful insights for librarians seeking to understand graduate students’ perception of google scholar and practical implications on how to best promote new information resources to graduate students. commentary this study examines graduate students’ perception and usage of google scholar. the findings should be of interest to academic librarians seeking to strike a balance between the promotion of library resources and google scholar for student research. not only did the author highlight major determinants for graduate students’ use of google scholar, but she also drew attention to the frequency at which respondents already use this search tool. the findings make a strong argument for librarians to focus on improving the usability and accessibility of library resources and the linking between library databases and google scholar, instead of simply discouraging students’ google scholar usage. a close examination of the research using the ebl critical appraisal checklist (glynn, 2006) indicated an overall validity of 78.3%. in addition, validity scoring was consistent for each appraisal checklist section. therefore, it can be safely concluded that the study is valid. the survey instrument was adopted from past tam studies and published with the article. moreover, cronbach’s alpha, a reliability measurement for internal consistency, showed that all the survey variables were above the accepted standards of 0.70. this provided further evidence to the soundness of the survey construct. however, this study does have some limitations. first, there is a lack of comparative data to place study findings in context. for instance, based on factors such as users’ perceived search effectiveness (3.89 out of 5) of google scholar, the author observed that user loyalty (3.23 out of 5) towards google scholar is rather limited. however, respondents’ loyalty ratings of google scholar may actually be significantly higher compared to their loyalty ratings of most library databases, or vice versa. evidence based library and information practice 2012, 7.4 98 in addition, the perceptions and attitudes of respondents who have never used google scholar and those who use it less than once per semester were excluded in the data analysis. the reasons for graduate students to choose not to use google scholar can be just as valuable as their reasons for adopting it as a regular search tool. additional research exploring students’ non-use of google scholar and usage of library databases based on tam would provide valuable insights for academic librarians. nonetheless, this survey study is well constructed and investigates a pertinent and timely issue in academic librarianship. it provides valuable contributions to the limited current literature on user studies examining google scholar. moreover, the study validates tam for examining user perception towards information resources and provides detailed methodology for those interested in expanding this field of study. references glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi: 10.1108/07378830610692154 evidence based library and information practice evidence based library and information practice 2011, 6.4 155 evidence based library and information practice evidence summary nutrition, food science, and dietetics faculty have information needs similar to basic and medical sciences faculty – online access to electronic journals, pubmed/medline, and google a review of: shpilko, i. (2011). assessing information-seeking patterns and needs of nutrition, food science, and dietetics faculty. library & information science research, 33(2), 151-157. reviewed by: mê-linh lê health sciences librarian health sciences library, university of saskatchewan saskatoon, saskatchewan, canada email: me-linh.le@usask.ca received: 15 july 2011 accepted: 05 oct. 2011 2011 lé. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to determine the information needs of nutrition, food science, and dietetics faculty members by specifically examining how they locate and access information sources and which scholarly journals are consulted for teaching, research, and current awareness; and identifying any perceived information service needs (e.g., training). design – online survey questionnaire. setting – four senior colleges within the city university of new york (cuny) system. subjects – nutrition, food science, and dietetics faculty members. methods – using institutional websites and the assistance of relevant affiliated librarians, 29 full-time and adjunct nutrition, food science, and dietetics faculty members were identified at queens college, brooklyn college, hunter college, and lehman college (all part of the cuny system). a survey was emailed in june and july 2007 and had 14 (48.4%) responses. the study was temporarily halted in late 2007. when resumed in january 2009, the survey was re-sent to the initial nonrespondents; five additional responses were received for a final 65.5% (n=19) response rate. main results – the majority of respondents held a phd in their field of study (63.1%), were full-time faculty (no percentage given), and female (89.5%). information sources were mailto:me-linh.le@usask.ca� evidence based library and information practice 2011, 6.4 156 ranked for usage by respondents, with scholarly journals unsurprisingly ranked highly (100%), followed by conference and seminar proceedings (78.9%), search engines (73.6%), government sources (68.4%), and information from professional organizations (68.4%). respondents ranked the top ten journals they used for current awareness and for research and teaching purposes. perhaps due to a lack of distinction by faculty in terms of what they use journals for, the two journal lists differ by only two titles. the majority browse e-journals (55.6%) rather than print, obtain access to e-journals through home or work computers (23.6%), and obtain access to print through personal collections (42.1%). databases were cited as the most effective way to locate relevant information (63.1%); pubmed was the most heavily used database (73.7%), although medline (via ebsco), science direct, and academic search premier were also used. respondents were asked how they preferred to obtain online research skills (e.g., on their own, via a colleague, via a librarian, or in some other way). the linked data does not answer this question, however, and instead supplies figures on what types of sessions respondents had attended in the past (44.4% attended library instruction sessions, while others were self-taught, consulted colleagues, attended seminars, or obtained skills through their phd research). conclusion – strong public interest in nutritional issues is a growing trend in the western world. for those faculty members and scholars researching and teaching on nutrition and related areas, more work on their information needs is required. this study begins to address that gap and found that nutrition, food science, and dietetics faculty share strong similarities with researchers in medicine and the other basic sciences with regard to information needs and behaviours. the focus is on electronic journals, pubmed/medline, and online access to resources. important insights include the fact that print journals are still in modest use, researchers use grey literature (e.g., government sources) and other nontraditional formats (e.g., conference proceedings and electronic mail lists) as information sources, and training sessions need to be offered in a variety of formats in order to address individual preferences. commentary this study contributes to our understanding of the information needs and behaviour of nutrition, food science, and dietetics faculty members – a fairly understudied group in the literature. specific areas of interest such as the top journals in the field, the use of grey literature, web-based search engines, and personal journal subscriptions will be valuable to liaison librarians when planning services and ordering resources. overall the study is a fairly straightforward and well-done information needs survey. however, the article would have benefited from more detail concerning the survey instrument and data collection. very little information is given on the instrument, such as how it was developed, the survey tool used (e.g., survey monkey), the number and types of questions, the wording employed, or how data analysis was performed. with regard to data collection, the author does not state whether upon resumption of the study in 2009 (the study was halted from late 2007 until early 2009), any attempts were made to determine if any new faculty members had been hired in the interim. if so, including them could have increased the relatively small sample size. additionally, there is no discussion on whether some demographic information (rank and credentials) and information preferences may be skewed towards full-time faculty and adjuncts. no mention is made of sessionals (staff hired to teach on a contract, i.e., nonpermanent, basis), who are not always listed on departmental websites or may not make use of liaison librarians (from whom the author retrieved her contact lists). as evidence based library and information practice 2011, 6.4 157 sessionals are typically younger researchers at the beginning of their careers, responses from older, more established researchers could have a significant impact on the data concerning the types of resources used (e.g., older researchers may be more inclined to use print-based personal library collections than their younger colleagues (connaway, radford, dickey, williams, & confer, 2008)). future updates of this study may want to consider additional means of collecting complete contact lists, such as by contacting the department secretary. a further area requiring clarity is the issue of private vs. library journal subscriptions. shpilko argues that faculty value, and make high use of, their personal journal subscriptions, resulting in fewer visits to the physical library. she also notes that few faculty use library computers to access ejournals, instead preferring to use their work and home computers. while this is one argument that can be made based on the data, another is that faculty members are unable to distinguish between personal and institutional e-journal access. many institutions now provide access to their e-resources via ip ranges that provide a seamless way of obtaining online resources anywhere on campus. in many cases users may not realize that the access is coming via the library system. since the survey instrument is not provided, there is no way of knowing how the original question(s) on private journal subscriptions were phrased. the article also makes no mention of how e-resources are accessed in the cuny system (e.g., via ip ranges). while the author recognizes some other limitations of her study, specifically the small sample size and lack of clarity for survey questions on intended journal use (i.e., research and teaching vs. keeping up to date), her suggestions for future research would both minimize these limitations and increase the external validity through the surveying of faculty in other subject areas and in clinical settings. references connaway, l. s., radford, m. l., dickey, t. j., williams, j. d. a., & confer, p. (2008). sense-making and synchronicity: information-seeking behaviors of millennials and baby boomers. libri, 58(2), 123-135. / evidence based library and information practice    evidence based library and information practice       editorial   the costs of open access   lorie kloda editor-in-chief assessment librarian, mcgill university library montreal, quebec, canada email: lorie.kloda@mcgill.ca      2015 kloda. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     since its inception almost 10 years ago, the eblip journal has been open access. yet, it seems that open access continues to be an important, and possibly controversial, topic in scholarly communications and within library and information practice. though many open access journals have emerged (and some even folded) in the past decade, i get the impression that librarians, scholars, and publishers continue to discuss and debate the same issues surrounding open access.   eblip is a gold open access journal according to the definition used by suber  (2015). this means that there are no article processing charges (apcs) that authors are required to pay for their submission to be published. in fact, according to suber’s research, most open access journals do not charge author fees, which means they employ a business model whereby the costs to produce the publication are covered in some other way. recently, crawford (2015) noted in his study of journals indexed in the directory of open access journals that only 4% of library science open access articles appear in journals that have apcs. in other words, gold open access is the norm in open access publishing, and especially in the field of librarianship. nevertheless, discussion continues about the sustainability of open access (see most recently, for example, banks, 2015).   i like to assume that readers of this journal, or at least long-time readers, are aware of how the journal operates. i can understand that those new to the journal, or considering submitting a manuscript here for the first time, will have some questions. but though our website clearly does not make any mention of apcs, i still regularly receive inquiries about fees. i would like to take the opportunity to clear up these questions for current, and hopefully future readers.   the way that eblip functions is based entirely on voluntary work of a group of professionals – librarian and information professionals and professors in library and information studies. these individuals volunteer their time, knowledge, and skills to keep this journal in motion. eblip is published quarterly and has disseminated four issues every year since 2006. these issues are typically published online on the 15th of march, june, september and december. we have never missed an issue, and several issues include an additional feature section with supplemental papers on a particular theme or related to a recent conference.   the journal is published using the open journal system platform, developed by the public knowledge project, and is hosted by the university of alberta’s learning services. the journal relies on the editorial team, consisting of editors, copyeditors, and editorial advisors, as well as the peer reviewers and writing assistants, to manage publications from submission to final layout and publication. none of these individuals are paid a salary or receive any money for completing work related to eblip. rather, these individuals work at various institutions that allow them the flexibility to make a contribution to research and scholarship, or else they contribute their personal time as part of their commitment to the journal.   there are various models for sustaining open access publications, and no single model suits the needs of all journals, even within library and information practice. eblip, however, has its costs absorbed from various institutions and individuals so that authors can focus on the quality of their submissions.   references   banks, m. (2015). open access publishing: what it is and how to sustain it. american libraries, 46(9/10), 58-61. retrieved from http://americanlibrariesmagazine.org/     crawford, w. (2015, may 17). oa articles involving apcs: more complete 2014 table. [blog post]. retrieved from http://walt.lishost.org/2015/05/oa-articles-involving-apcs-more-complete-2014-table/   suber, p. (2015, july 7). open access overview. retrieved from http://legacy.earlham.edu/~peters/fos/overview.htm   microsoft word art_langille.doc evidence based library and information practice 2007, 2:1  23 evidence based library and information practice     feature article    navigating the road to success: a systematic approach to preparing competitive  grant proposals       lynn langille  research consultant   atlantic health promotion research centre   dalhousie university  halifax, nova scotia, canada  email: lynn.langille@dal.ca     theresa mackenzie  manager, knowledge exchange and research capacity development   newfoundland and labrador centre for applied health research   memorial university  st. john’s, newfoundland and labrador, canada  email: tmackenz@mun.ca      received: 29 november 2006    accepted: 13 january 2007        © 2007 langille and mackenzie. this is an open access article distributed under the terms of the creative  commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    abstract    purpose ‐ difficulty in securing research funding has been cited as one barrier to the  involvement of more librarians and information professionals in conducting original  research. this article seeks to support the work of librarians who wish to secure research  funding by describing a systematic approach to the creation of successful grant applications.     approach ‐ the authors draw on more than fifteen years collective experience in supporting  the development of successful research grant proposals. eleven grant‐writing best practices  or ‘key approaches’ are described, and a planning timeline is suggested.     conclusions: use of these best practices can assist researchers in creating successful  research grant proposals that will also help streamline the research process once it is  underway. it is important to recognize the competitive nature of research grant  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:1  24 competitions, obtain feedback from an internal review panel, and use feedback from  funding agencies to strengthen future grant applications.       introduction  it is clear from reports in this and other  research journals that librarians and  information professionals are active  researchers who value the use of evidence in  their work. recent studies have pointed to  the need for more research in the field, but  barriers exist that may hinder librarians’  ability to conduct research (powell et al).  one such barrier may be the inability of  researchers to secure funding for this work  (mcnicol, koufogiannakis and crumley).    the guidelines in this article draw on the  authors’ more than fifteen collective years of  experience in writing, and supporting others  to write, collaborative research grants  (atlantic health promotion research centre,  newfoundland and labrador centre for  applied health research).  the purpose of  the article is to distill from these experiences  the key aspects of a successful grant  proposal, and to suggest a systematic  approach to grant‐writing, that will help  librarians to secure research funding.                whether a researcher is applying for  research grants from his/her employing  organization or from a national or  international granting body, the qualities of  a successful grant proposal are similar.  some of the most important approaches to  writing successful grant proposals are  summarized in table 1 and outlined in the  sections that follow.     1.  address a specific audience  there are two main audiences for a grant  proposal – the funding agency, with its  specific mandate and priorities, and a  committee of reviewers who may or may  not be experts in your field or discipline.      researchers can learn more about funding  agencies by studying the organization’s  websites and publications. additionally,  researchers can obtain information by  contacting a program officer within the  agency to confirm that the proposal fits with  the program objectives of the funding  agency.      table 1. a systematic approach to writing a successful grant proposal.  a systematic approach to writing a  successful grant proposal  1. address a specific audience   2. be persuasive   3. be innovative  4. involve stakeholders   5. define your objectives and outcomes  6. include a detailed research plan   7. address ethical issues  8. describe the use of research results   9. describe required human and material resources   10. be focused and write clearly   11. tell a story  evidence based library and information practice 2007, 2:1  25 the agency’s website will often provide  direct contact information for a program  officer in its request for proposals. when  this information is not provided, contact the  agency and request this information.  it is  helpful to establish a contact person for  information as you navigate through the  application process.      members of review committees are typically  busy researchers who are in demand for  their expertise and may have many  proposals to review.  successful grant  proposals capture the attention of the  reviewers and stand out from other  submissions in their clarity and focus.      to begin, determine your eligibility for the  granting initiative, e.g., does the agency  have citizenship or academic requirements  that you must meet?  then, explicitly follow  the application instructions provided by the  funding agency.  if a proposal is not  complete, or does not adhere to the specified  format, it may be rejected by the agency in  their preliminary review before it even  reaches the review committee.  funding  agencies often provide assessment criteria  within the application instructions.  use this  information to assess your own proposal  and to ensure that it meets the agency’s  requirements.      other useful tools for critically appraising  grant proposals are available for clinical  research (inouye and fiellin) and  community based research (ontario healthy  communities coalition).       the summary  funding agencies often require a one‐page  summary of the proposed project.  the  summary distills the essential components  of the proposal into a concise, clear  description of the project (innouye and  fiellin).  it should contain a few sentences  on each of the following:  the problem, the  credibility of the applicants, the general and  specific objectives, methods to achieve the  objectives, and anticipated outcomes.  the  summary will make an important first  impression in the minds of the reviewers  assigned to your proposal.  it is often the  only section of the proposal that will be read  by all members of the review committee.  a  well‐crafted summary is a critical  component of a successful grant application.    2.  be persuasive   in competitive funding contexts, there are  likely to be many proposals that are  technically sound.  in these cases,  persuasion will be the deciding factor in  success. applicants must provide evidence  to explain the need for research in a  particular area and demonstrate their ability  to carry it out.  explain who will benefit  from the research and how.  spark the  interest and imagination of reviewers by  using a variety of types of information and  formats including statistics, stories, quotes,  frameworks and tables.      soliciting feedback from experienced  researchers, both internal and external to  your field or discipline, will improve the  persuasiveness of your proposal.  look for  researchers within your institution who  have had funding success, or approach  specialists in your subject area who work at  other institutions. inouye & fiellin  recommend a structured internal review  process in which the proposal is distributed  about 3 weeks before the deadline to an  internal review panel consisting of three  members (2 in the same field and 1 outside  the field).  the panel meets with the  principal investigator(s) about 2 weeks  before the deadline to provide feedback on  the overall grant and quality of presentation  and to address specific issues within each  section.  an internal review process can  improve the presentation and scientific  content of a grant proposal, give members of  the panel experience in reviewing grants,  evidence based library and information practice 2007, 2:1  26 increase collegiality, and help applicants  meet their deadlines (inouye and fiellin).      3.  be innovative  innovation is an assessment criterion in  most funding initiatives.  funding agencies  are not interested in funding the same  research repeatedly.  they want to produce  new knowledge that builds on what is  already known.  the current push toward  applied research through knowledge  translation includes innovation in  partnerships and collaborations as well as in  knowledge development.   successful grant‐ writers indicate clearly how their proposal is  unique, responds to an identified need,  builds on previous work, avoids previous  errors, creates stronger collaborations, and  makes an important contribution to their  field.    4.  involve stakeholders  increasingly, funding agencies want to  know that the research they are supporting  is going to be applied to real‐world  problems and challenges.  including a range  of stakeholders for whom your research is  relevant increases the likelihood that the  research results will be used.  stakeholders  may participate in the design of the research  question, in the development of the  proposal, or in the application or  dissemination of results. a good proposal  describes the role of stakeholders at each  stage of the project.        selected grant writing terms and acronyms  stakeholder   a person, group, or organization that has an interest in the outcomes of  the research project  principal  investigator (pi)  generally, the main applicant in a research proposal; the person who will  lead the research project and be accountable for the conduct of the  research and reporting to the funding agency.   co‐investigator  (ci)  an individual who works with the principal investigator in the  development or implementation of a research project.   kt/kte  knowledge transfer or knowledge translation and exchange: sharing of  information in the process of conducting a research project or in  dissemination of results. implies two‐way exchange with stakeholders.   loi  letter of intent: a letter that registers your intent to apply to a funding  opportunity or agency. not all agencies or programs require an loi.  check with the funding agency for specific requirements.   reb/irb  research ethics board or institutional review board: the committee or  board of an institution or jurisdiction that is responsible for ensuring that  research proposals meet ethical standards. the name of the reb/irb  varies by institution and jurisdiction.   cfa/rfa/rfp  call for applications/ request for applications/ request for proposals: an  announcement from a funding agency that indicates the start of a new  grant competition. it contains specific instructions for applying to the  initiative.     table 2. selected grant writing terms and acronyms.            evidence based library and information practice 2007, 2:1  27 5.  define your objectives and outcomes  research proposals usually describe a  general objective, specific objectives, and  expected outcomes.  general objectives  describe the larger problem or the  motivation behind the research. they are  goals that cannot be achieved by a single  project or organization and typically require  long‐term efforts. specific objectives can be  achieved within the timeframe of the project  or program described in the proposal.  outcomes are the products of the research  or the contribution towards resolving the  general and specific objectives.     strong proposals strike a balance between  ambition and achievability. grant reviewers  want to know what the outcome of your  research is expected to be.  will it be a new  theory, a better explanation or  understanding of a social or technical  problem, a new method or approach?  in the  proposal, outcomes should be linked to the  specific objectives and detailed in the  research plan.      in addition to clarifying why the research is  important and where it fits in the larger  scheme of things, the process of developing  objectives and describing outcomes for the  proposal also helps the researcher provide a  framework for the selection of research  methods and knowledge exchange  strategies.     6. include a detailed research plan  a comprehensive research plan is one of the  most important factors determining the  success of a proposal.  about half of the  pages in the proposal should be dedicated to  the research plan, which provides a detailed  description of how the researchers will  achieve their specific objectives.  explicitly  state the connection between the objectives  and methods, call attention to potential  challenges that may be encountered, and  provide possible strategies for addressing  them.  the research plan will include  information on the sample or population,  sample recruitment, data collection and data  analysis.  describe what will be done, how it  will be done, and by whom and when.   clarify the roles and contributions of the  research team and partners. include a  timeline showing when each of the  components of the research plan will take  place and when the outcomes will be  realized. displaying the key components of  the research plan in table format is a useful  way to provide a ‘snapshot’ of the research  and related activities for the reviewers, and  can help improve the internal consistency of  the plan as it is being developed.        components of a research plan  description of objectives  general description of methods and how  they will achieve objectives  specific methodology: sample or  population; recruitment; data collection and  analysis  potential challenges and strategies for  addressing them  roles of research team members and  partners  timeline    table 3. components of a research plan.  evidence based library and information practice 2007, 2:1  28   7.  address ethical issues  all research involving human subjects  requires ethical approval from institutional  or government ethics review boards.    in some cases, ethical approval for research  is required before an application is accepted  by a funding agency.  more often, ethical  approval is sought after a grant is funded  but before any aspects of the research are  initiated.  even if formal ethics approval is  not required by the funding agency to which  you are applying, your proposal should  broadly describe the ethical dimensions of  the proposed research (e.g., informed  consent, confidentiality) and the approaches  that will be taken to ensure that the research  meets ethical standards.    8.  describe the use of research results   funding agencies often suggest that at least  10% of the total project budget be devoted to  the dissemination and application of  research results.  over the past decade, the  application of research results in health and  social sciences has become a field of study in  itself, framed within concepts such as  knowledge translation, knowledge exchange,  and knowledge mobilization.  the canadian  health services research foundation has  developed a 1:3:25 format (a one‐page  summary, a three‐page description and a 25‐ page report) for the dissemination of  research to policy makers in the health  services field (www.chsrf.ca).  the  provincial centre of excellence for child  and youth mental health at the children’s  hospital of eastern ontario has produced a  toolkit called doing more with what you  know which includes a wide range of tools  and processes for knowledge exchange  (www.alafinepointe.ca).  these tools can be  easily adapted for librarian researchers and  can be used to identify multiple target  audiences for research results and to  promote creativity in developing content  and formats.      9. describe required human and material  resources  the project budget outlines the human and  material resources needed to carry out the  project.  a well‐planned budget reflects  attention to detail.  align the budget with  the proposed activities, current costs, and  the expenditures allowed by the funding  agency. link the required personnel to the  specific aims of the project.  many funding  agencies will require that dollar figures are  attached to ‘in‐kind’ contributions such as  the time contributions of existing staff, the  use of existing equipment, and the value of  office and meeting space.  carefully  calculate both the resources that are being  contributed by partners in the project and  those you are requesting from the funding  agency.  keep in mind that there are often  guidelines for salaries and other costs within  your institution, or within the funding  agency, that must be followed.  funding  agencies and your employing institution  may provide guidance in this area; check  with them for sample budgets and other  useful information.  make sure the budget  figures add up correctly.    10.  be focused and write clearly  proposals are likely to be viewed in a  positive light if they are pleasurable to read  and easy to assess according to the criteria  set out by the funding agency.  proposals  that are vague and unfocused will not be  well received by reviewers.  begin each  section and paragraph in the proposal with  a strong lead sentence that boosts the  readers’ interest and tells them what the  paragraph is about (mcinnes et al.).  define  key concepts and use the same information  and terms consistently throughout the  proposal. write generally for all members of  the review committee but with enough  specificity to satisfy the reviewers who are  knowledgeable in your field.      evidence based library and information practice 2007, 2:1  29 write in the active voice using short, clear  sentences.  writing clearly means avoiding  jargon and acronyms as much as possible,  paying close attention to good grammar and  spelling, and practicing good formatting  (e.g., using the exact margins and font sizes  specified by the funding agency).   additional strategies for improving grant‐ writing skills include asking colleagues who  have been successful in obtaining grant  funding for copies of their proposals and  becoming a member of a peer review  committee.     11. tell a story  a grant proposal is a narrative.  like any  good narrative, it has a beginning that  draws the reader in, a middle that provides  vivid detail to sustain interest and provide  the kinds of information the reader is  seeking, and an ending that draws the  threads together.  a strong beginning is  imperative to set the stage for what follows.   the middle of the proposal succinctly  reviews the research context and provides   details of the research plan.  the ending re‐ states the importance of the work being  proposed, assures the reader of the  researchers’ credibility and preparedness to  take on the tasks outlined, and reiterates  how the outcomes of the project will  contribute to research and broader social  goals.    grant writing timeline  grant writing generally consists of three  phases:  pre‐writing, writing and  submission.  in phase i, the pre‐writing  phase, the general and specific objectives for  the research are drafted, links are  established with research users, preliminary  work (e.g., pilot testing, needs assessment)  is conducted, and contact is made with the  program officer in the funding agency.  phase ii, the writing phase, starts about 3  months before the application deadline    grant writing timeline    period of time before  the deadline  activities  p h as e  1  6 months to 1 year  survey funding opportunities and decide which funding  programs you will target.  draft the general and specific objectives for your research.  establish links with research users/ stakeholders.  conduct pilot tests, needs assessments and other preliminary  work as required.  contact the program officer in the funding agency.  12 weeks    assemble your multidisciplinary team (if applicable).  write the general and specific objectives.  start gathering materials.  6 weeks  start writing, some every day.  4 weeks  finish gathering materials.  solicit letters of support, as appropriate.  p h as e  2  3 weeks  distribute draft to internal review panel.  2 weeks  meet with internal review panel.  submit for institutional processing and signatures.  p h as e  3  1 week  make final revisions, carefully proofread, and submit.  adapted from mcinnes, andrews & rachubinski (2005)    table 4. grant writing timeline.  evidence based library and information practice 2007, 2:1  30 it includes gathering relevant materials,  writing the background and research plan,  preparing the budget, completing the  various forms required by the funding  agency, and carrying out an internal review  of the proposal. in phase iii, the submission  phase, the grant is revised and submitted to  the funding agency according to the  requirements set out in the request for  applications.  a typical schedule for the  three phases is outlined in table 4.      conclusion  the preparation of a grant proposal is time  consuming, but the time is well spent if it  enables researchers to carry out meaningful  research and contribute to their field.  attention to detail in the preparation stage  also streamlines the research process.      some common errors made by new grant‐ writers include failure to carefully follow  the funding guidelines and process, failure  to demonstrate understanding of the  funding agency’s priorities, omission of  required information, or failure to think  through the implementation of the project.  these mistakes can be easily avoided by  paying careful attention to the details  outlined in the request for proposals.      if funded, a comprehensive, well‐written  grant proposal can be used to help clarify  partner roles and responsibilities, to provide  the basis for the preparation of ethics  submission(s) to relevant institutional or  research ethics boards, and to establish a  framework for the management of the  project.  if a proposal is not funded, try not  to take it personally (rejection is more  common than acceptance), read the reviews  carefully, use constructive criticism to revise,  and re‐submit the proposal at the next  opportunity.  grant‐writing skills improve  with time and effort, and each attempt  builds on the learning process.      acknowledgements  ʺthe authors would like to thank sandra  crowell, managing director of the atlantic  health promotion research centre, for her  editorial assistance, and nancy dawe for her  research assistance.  the article has also  benefited from helpful comments from  eblip reviewers.ʺ      works cited   atlantic health promotion research centre.  guidelines for preparing research  proposals:  navigating the road to  success. (1999). 14 nov. 2006  <http://www.ahprc.dal.ca/guideline s.pdf>.    canadian health services research  foundation. communication notes,  reader‐friendly writing – 1:3:25 (2004).  3 nov. 2006  <http://www.chsrf.ca/knowledge_trans fer/pdf/cn‐1325_e.pdf>    inouye, sharon, and david fiellin. “an  evidence‐based guide to writing grant  proposals for clinical research.” annals  of internal medicine. 142 (2005): 274‐ 282.      koufogiannakis, denise and ellen crumley.  “research in librarianship: issues to  consider.” library hi tech. 24 (2006):  324‐340.    mcinnes, roderick, brenda andrews, and  richard rachubinski. guidebook for  new principal investigators.  canadians institutes of health  research, institute of genetics (2005).  15 nov. 2006 <http://www.cihr‐ irsc.gc.ca/e/documents/ig_guide_for_n ew_pis_e.pdf>    mcnicol, sarah.  “is research an untapped  resource in the library and  information profession?” journal of  http://www.ahprc.dal.ca/guideline http://www.chsrf.ca/knowledge_trans http://www.cihr%e2%80%90irsc%00 evidence based library and information practice 2007, 2:1  31 librarianship and information science,  6.3 (2004):119‐26.    newfoundland and labrador centre for  applied health research.  grantsmanship resources (2004). 14  nov. 2006  <www.nlcahr.mun.ca/resources_team/ writing_tools.php>     ontario healthy communities coalition.  strategies for effective proposal  writing. (2nd ed.) (no date). 7 nov. 2006  <http://www.healthycommunities.on.c a/publications/misc/fundproposal.pdf>    powell, ronald, lynda baker, and joseph  mika. “library and information science  practitioners and research.” library &  information science research. 24  (2002): 49‐72.    provincial centre of excellence for child  and youth mental health at cheo.  doing more with what you know: a  tool kit on knowledge exchange.  (2005). 7 nov.  2006  <http://www.alafinepointe.ca/kec/docu ments/ketoolkit.pdf>       additional resources  society of research administrators,  international grant resources  http://www.srainternational.org/newweb/gr antsweb/index.cfm    special libraries association research  statement  http://www.sla.org/content/resources/resear ch/rsrchstatement.cfm    online computer library centre research  resources  http://www.oclc.org/research/default.htm    library research services  http://www.lrs.org/        http://www.healthycommunities.on.c http://www.alafinepointe.ca/kec/docu http://www.srainternational.org/newweb/gr http://www.sla.org/content/resources/resear http://www.oclc.org/research/default.htm http://www.lrs.org/ evidence summary   embedded library guides in learning management systems help students get started on research assignments   a review of: murphy, s. a. & black, e. l. (2013). embedding guides where students learn: do design choices and librarian behavior make a difference? the journal of academic librarianship, 39(6), 528-534. http://dx.doi.org/10.1016/j.acalib.2013.06.007   reviewed by: dominique daniel humanities librarian for history and modern languages oakland university rochester, michigan, united states of america email: daniel@oakland.edu     received: 24 nov. 2015 accepted: 10 feb. 2016      2016 daniel. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine whether library guides embedded in learning management systems (lms) get used by students, and to identify best practices for the creation and promotion of these guides by librarians.   design – mixed methods combining quantitative and qualitative data collection and analysis (survey, interviews, and statistical analysis).   setting – a large public university in the united states of america.   subjects – 100 undergraduate students and 14 librarians.   methods – the researchers surveyed undergraduate students who were participating in a project information literacy study about their use of library guides in the learning management system (lms) for a given quarter. at that university, all course pages in the lms are automatically assigned a library guide. in addition, web usage data about the course-embedded guides was analyzed and high use guides were identified, namely guides that received an average of at least two visits per student enrolled in a course. the researchers also conducted a qualitative analysis of the layout of the high use guides, including the number of widgets (or boxes) and links. finally, librarians who created high use library guides were interviewed. these mixed methods were designed to address four research questions: 1) were students finding the guides in the lms, and did they find the guides useful? 2) did high use guides differ in design and composition? 3) were the guides designed for a specific course, or for an entire department or college? and, 4) how did the librarians promote use?   main results – only 33% of the students said they noticed the library guide in the lms course page, and 21% reported using the guide. among those who used the guide, the majority were freshmen (possibly because embedding of library guides in the lms had just started at the university). library guides with high use in relation to class enrollment did not significantly differ from low use guides in terms of numbers of widgets and links, although high use guides tended to have slightly fewer widgets. of those guides, 55% were assigned at the course level, 30% at the department level and 13% at the college level. over half the librarians with at least one high use guide conducted a library instruction session in which they used or promoted that guide. for 39% of the courses with high-use guides, the librarian was actively engaged with the faculty and students via the lms, but others reported no specific involvement in courses.   conclusion – those students who used library guides reported the guides helped them get started on their research paper or assignment and find research materials, two areas for which previous studies show students have great difficulty. since the majority of students did not notice the link to the library guide in the lms, librarians could emphasize it in the news section of the course, which gets much more attention. within library guides, simpler groupings of links might be easier for students to use, but this conclusion would require further research to confirm. in any case, nearly half of all high use guides were not promoted in any way by librarians, but simply automatically embedded in the lms, a sign that passive embedding may provide an easy way for the library to reach a large number of students early in their academic career. since the automatic embedding of guides began, guides have seen a dramatic increase in usage.   commentary   although there are many articles describing how library guides have been automatically embedded into learning management systems (lms), there are not many assessments of their effectiveness. for this reason, this study is thought-provoking. the researchers do not just rely on web usage statistics but also on student and librarian feedback to see whether students found the guides useful and what design practices improve use. their conclusions add to the growing body of research about patron use of guides embedded in course pages (wakeham, roberts, shelley, & wells, 2012; chiware, 2014).   the combination of qualitative and quantitative methods involving both creators and users of guides yields interesting results. one finding challenges the generally accepted view that students prefer course-specific guides over general subject guides (reeb and gibbons, 2004; barr, 2010; leighton & may, 2013). the researchers believe that generic subject guides may be as highly used as customized course guides if they can easily be found in lms course pages. the authors also argue that library content delivered passively, without instructional or other promotional effort, can still be successful, since nearly half of all high-use guides were created by librarians who did not do any instruction. however, promotion in the lms or in the classroom somewhat encourages student discovery. this counters the commonly accepted view that promotion of guides during instruction sessions is essential for guide use (chiware, 2014).   unfortunately, the student survey relied on a small sample population that limits the implications of the study. the study does not specify if the respondents who expressed satisfaction used subject or course-specific guides, and what discipline they were from. it would also be interesting to see how helpful the guides truly were by comparing the skill level of students who used them and of those who did not. the placement and discoverability of the guide are probably as important as its content and design regarding the number of users, but the guide’s success, as measured by student learning, does depend on its content and design.   still, the study has important practical implications for librarians. when preparing guides, librarians should weigh the time spent creating guides versus their impact as measured by student use and learning. the decision to create course-specific guides or more general guides should be based on numerous factors, including the nature of the course and level of students, the discipline involved, the purpose of the guides, their intended life span, and the instructors’ intended promotion and use of the guide in class. librarians should decide between guides “as a teaching tool” and guides “as an answer to a student’s research” (gessner, chandler & wilcox, 2015, p. 505). this study’s findings can help librarians identify the most effective strategy.   references   barr, d. (2010). reaching students where they go: embedding library resources in course content. science & technology libraries, 29(4), 289-297. http://dx.doi.org/10.1080/0194262x.2010.523305   chiware, m. s. (2014). the efficacy of course-specific library guides to support essay writing at the university of cape town. south african journal of libraries and information science, 80(2), 27-35. http://dx.doi.org/10.7553/80-2-1522   gessner, g. c., chandler, a., & wilcox, w. s. (2015) are you reaching your audience?: the intersection between libguide authors and libguide users. reference services review, 43(3), 491-508. http://dx.doi.org/10.1108/rsr-02-2015-0010   leighton, h. v., & may, d. (2013). the library course page and instruction: perceived helpfulness and use among students. internet reference services quarterly, 18(2), 127-138. http://dx.doi.org/10.1080/10875301.2013.804019   reeb, b., & gibbons, s. (2004). students, librarians and subject guides: improving a poor rate of return. portal: libraries and the academy, 4(1), 123-130. http://dx.doi.org/10.1353/pla.2004.0020   wakeham, m., roberts, a., shelley, j., & wells, p. (2012). library subject guides: a case study of evidence-informed library development. journal of librarianship and information science, 44(3), 199-207. http://dx.doi.org/10.1177/0961000611434757   evidence based library and information practice 2012, 7.2 79 evidence based library and information practice evidence summary teams are now used by many technical services departments in academic libraries a review of: zhu, l. (2011). use of teams in technical services in academic libraries. library collections, acquisitions, & technical services 35, 69-82. doi:10.1016/j.lcats.2011.03.013 reviewed by: kirsty thomson subject librarian university library, heriot-watt university edinburgh, scotland, united kingdom email: k.s.thomson@hw.ac.uk received: 29 feb. 2012 accepted: 12 apr. 2012 2012 thomson. this is an open access article distributed under the terms of the creative commons‐ attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – an investigation of the use of teams in technical services, provision of training on team-working, characteristics of technical services teams, and the effectiveness of teams. design – survey comprising of 19 closed questions and one open question. setting – technical services departments in academic libraries. subjects– responses were received from 322 library staff members. of those, 294 answered the survey question about team-based technical services and 55.9% of respondents completed the full survey. methods – an online survey was promoted via seven technical services electronic mail lists and was conducted using surveymonkey. main results – the survey found that 39% of technical services were entirely team-based, 18% were partly team-based, and 43% did not use teams. information was gathered about the number of teams, team nomenclature, and how long teams have been used. this research highlighted the lack of provision of training and documentation about working in teams. conclusion – many respondents have teambased technical services, and most participants found that working in teams had a positive impact. a systematic application of this survey is planned for the future. mailto:k.s.thomson@hw.ac.uk evidence based library and information practice 2012, 7.2 80 commentary survey participants were recruited via technical services electronic mail lists. this was not “simple random sampling” as stated in the paper (p. 72). instead, participants were selfselecting. there is a strong likelihood of sample bias in this study as participants needed to both be a member of at least one of the electronic mail lists and choose to complete the survey. unfortunately, promoting the survey on multiple mail lists means that response rates cannot be calculated, as the targeted population cannot be measured. the author does not mention if she identified multiple responses from the same institution, which would also skew the results. an approach of directly contacting institutions would have gathered a more representative, measurable set of participants, and the author states that she is planning to conduct a systematic survey in the future. although the survey was based on previously published research, there are some omissions in the questionnaire and in the presentation of the results. it will be difficult for organizations considering teams to make use of the data from the survey without more information about team structures. for example, participants were asked about the number of technical services teams in their organization, but not about the size of the teams or overall staffing levels. some questions have been cross-tabulated, such as the relationship between team-based libraries and team-based technical services, but there is little statistical analysis of the results beyond reporting of percentages. the survey will have found additional information, e.g., the relationship between team autonomy and morale, but the lack of analysis means this has not been revealed. this paper includes a large number of tables, which don’t always appear next to the relevant text. including the survey question numbering in the table captions would have improved the readability of the paper. some comments from participants are included in the narrative of this paper. these must have been gathered via the open “any other comments” question. it would have been beneficial to include further open questions in the survey as this would have gathered extra qualitative information about the use of teams. although the survey found “more and more” technical services were forming teams (51 had teams more than 10 years ago, while 63 had formed teams in the last 10 years) this finding may have been affected by the self-selecting nature of the survey (p. 80). future research should consider if any technical services have abandoned a team structure. evidence based library and information practice 2012, 7.2 67 evidence based library and information practice evidence summary university students are unaware of the role of academic librarians a review of: bickley, r. & corral, s. (2011). student perceptions of staff in the information commons: a survey at the university of sheffield. reference services review, 39(2), 223-243. doi:10.1108/00907321111135466 reviewed by: kirsty thomson subject librarian heriot-watt university edinburgh, scotland, united kingdom email: k.s.thomson@hw.ac.uk received: 31 aug. 2011 accepted: 4 mar. 2012 2012 thomson. this is an open access article distributed under the terms of the creative commons‐ attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to discover students’ perceptions of information commons staff, and to determine how these perceptions influence the use of library resources. design – post-experience survey with one follow-up interview. setting – the university of sheffield, a postsecondary institution in england. subjects – all undergraduate and postgraduate students were invited to take part. just over 1% of the student population, or 250 students, completed the survey. methods – information about the survey was sent to students’ institutional email addresses. one follow up interview was carried out via email using the critical incident technique. main results – students do not understand the academic roles of librarians. they are unlikely to approach library staff for academic support, preferring to turn to instructors, other students, friends, and family. most students had positive opinions about assistance received in the information commons, but a small number reflected on previous bad experiences with staff, or on a fear of being made to feel foolish. the vast majority of students who did not seek help in the information commons stated that this was because they did not require assistance. most mailto:k.s.thomson@hw.ac.uk evidence based library and information practice 2012, 7.2 68 students do not perceive a difference between information commons staff and library staff. conclusion – students have positive views of information commons staff at the university of sheffield, but have low awareness of the roles of professional librarians. librarians need to develop partnerships with academic staff and strengthen their presence in both physical and online learning environments to promote their academic roles. commentary this is a well-written paper reporting on findings that have serious implications for professional librarians, especially those working in post-secondary education. most of the data for the paper was gathered via an online survey. researchers used email to contact university students since the survey was conducted during the summer vacation. the paper does not explain the decision to implement the survey when many students were away from the university, but the lead author was undertaking a master’s degree and is likely to have been restricted to this time period due to the structure of her course. just over 1% of potential participants completed and submitted survey responses, a very small sample size. this may result in sample bias, as students with strong opinions (positive or negative) may have been more motivated to complete the survey. unfortunately the article does not include key details about the survey design and methodology, e.g., whether reminder emails were sent to encourage completion. also, post-experience questionnaires can be problematic if there is a time gap between the activity and the survey. in this study students were asked about the previous academic year, and may have provided inaccurate data about experiences from earlier months. the authors refer to a copy of the survey provided as an appendix (p. 229) but this is missing from the e-journal. therefore it is not possible to review the exact wording used in the survey questions and it is unclear if students were asked about online contact with the information commons, or only about physical visits. respondents did not correctly identify the activities carried out by academic librarians, but there may have been confusion around the different job titles of library employees: were job titles clarified in the survey design, or did some participants think all university library employees were academic librarians? the findings of the survey clearly demonstrate that university libraries are not effectively communicating the roles of academic librarians. more than half of the survey respondents did not know the identity of their academic librarian, and many did not understand the academic support role of library staff. though researchers made attempts to arrange follow-up interviews with respondents, only one interview was conducted, and this interview is not fully described in the paper. further interviews and alternative methods of gathering information, such as focus groups, could prove helpful in future studies in this area. evidence based library and information practice 2011, 6.3 92 evidence based library and information practice news/announcements evidence based library and information practice seeks associate editor (articles) © 2011. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. evidence based library and information practice (eblip) is looking for an associate editor, responsible for articles, to join our editorial team. the current associate editor (articles) will be taking over as editor‐in‐chief beginning january 2012, at which time the successful candidate will take over her responsibilities for articles. the associate editor (articles) is responsible for: • overseeing the complete editorial process for items submitted to the articles section (this includes assignment of peer reviewers, monitoring the peer review process, communicating with authors and peer reviewers, making acceptance decisions, ensuring required changes to manuscripts are made, communicating with copyeditors and the production editor to ensure the final copy is as expected). • communicating and consulting with the editor‐in‐chief on a regular basis. • attending editorial team meetings on a monthly basis and participating in the overall governance of the journal. • ensuring that all necessary deadlines are met. • communicating with potential authors and responding to queries. the ideal candidate will be well‐versed in evidence based practice and research methods. this position requires dedicated time on a regular basis, and it is therefore essential that interested persons ensure available time to devote to this position prior to applying. it is estimated that the workload is approximately 10‐15 hours per month. interested persons should send a statement of interest, indicating areas of strength they would bring to the role, as well as a brief resume to denise koufogiannakis, editor‐in‐chief, at denise.koufogiannakis@ualberta.ca by october 31, 2011. the current associate editor (articles) is happy to answer specific queries about the role on request. please email a.brettle@salford.ac.uk. **please note that evidence based library and information practice is a non‐profit, open access journal and all positions are voluntary and unpaid. evidence based library and information practice 2011, 6.3 93 please visit the evidence based library and information practice web site (http://ejournals.library.ualberta.ca/index.php/e blip) for further information about the journal. / evidence based library and information practice microsoft word art_cotter.doc evidence based library and information practice 2006, 1:1      3 evidence based library and information practice       article    adding spice to a library intranet site: a recipe to enhance usability      lisa cotter  school librarian, nursing and midwifery  the university of newcastle and northern sydney central coast health   gosford, nsw, australia  e‐mail: lisa.cotter@newcastle.edu.au    larnich harije  technical services librarian, central coast health service library,   northern sydney central coast health   gosford, nsw, australia  e‐mail: lharije@nsccahs.health.nsw.gov.au    suzanne lewis  manager, central coast health service library  northern sydney central coast health   gosford, nsw, australia  e‐mail: slewis@nsccahs.health.nsw.gov.au    ingrid tonnison  client services librarian, central coast health service library,   northern sydney central coast health   gosford, nsw, australia  e‐mail: itonnison@nsccahs.health.nsw.gov.au      received: 9 december 2005    accepted: 6 march 2006      © 2006 cotter, harije, lewis and tonnison. this is an open access article distributed under the terms of  the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits  unrestricted use, distribution, and reproduction in any medium, provided the original work is properly  cited.      abstract    objective ‐ to produce a highly‐usable intranet site, use the project to explore the  practical application of evidence‐based librarianship (ebl), and refine the library’s  project management methodology.   methods ‐ evidence was gathered via a literature review, an online survey, scenario‐ based usability testing, and completion of a usability checklist. usability issues were  then addressed, guided by the research‐based web design and usability guidelines.    http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:1      4  results ‐ after a preliminary revision, the site achieved a usability index of 79% after  application of the “raward library usability analysis tool”. finding the information  and supporting user tasks were identified as areas of weakness. usability testing and  client feedback supported these findings. after these issues were addressed by a major  site redevelopment, the usability index increased to 98%.    conclusions ‐ raward’s checklist is an easy and effective tool for measuring and  identifying usability issues. its value was enhanced by scenario‐based usability testing,  which yielded rich, client‐specific information. the application of ebl and project  management principles enhanced the outcomes of the project, and the professional  development of the project team.      objective     the central coast health service (cchs)  library in new south wales, australia,  provides information services to meet the  clinical, research and learning needs of the  central coast sector (803 beds) of  northern sydney central coast health  (nscch). the primary client groups  include the cchs staff (3665 fte) and  health students based at cchs. the main  site, gosford hospital, is a teaching  hospital of the university of newcastle,  australia. this partnership in health  education and training has been  particularly successful in the area of  library services, with a strong, mutually  beneficial relationship maintained  between the cchs library and the  university of newcastle library.    in may 2003, librarians from the cchs  library were introduced to the evidence  based librarianship (ebl) framework via a  course offered through the national  electronic library for health (nelh)  (booth ʺbriefingʺ). following an exercise  which asked participants to identify an  aspect of current service provision that  was based on poor evidence, the cchs  librarians acknowledged that the library’s  intranet site was a service based on very  poor evidence, and had outgrown its  original structure and purpose. the  following year, a project team was  established to thoroughly revise both the  architecture and content of the site, with a  focus on improving usability. members of  the team included the client services  librarian, technical services librarian,  university liaison librarian, and the  library manager.  the objectives of the project were to  produce a highly usable library intranet  site; to undertake the project as an  opportunity to explore the practice of ebl;  and to use the project as a means of  refining the library’s newly‐adopted  project management methodology.  accordingly, the problem – a user‐ unfriendly intranet site – was reframed in  terms of the following research question:     “how can the usability of the cchs library’s  intranet site be improved to enhance the siteʹs  effectiveness as a gateway to the library’s  services and resources, for the staff and  students of the organisation?”    the primary objective – to produce a  highly usable intranet site – had to be  achieved within the framework of an  existing organisational template which  restricted the modification of many design  elements. organisational culture was  another issue as central coast health was  in the process of merging with a  neighbouring health service. at the time of  this writing, a combined, area‐wide  intranet is being developed. this project  also provided an ideal opportunity to  achieve the secondary objectives of  exploring the practice of ebl, and refining  the library’s project management  methodology. the team envisaged  synergies between ebl and project  management and planned to explore this  idea as a potential tool for bridging the  theory‐practice gap. evidence based library and information practice 2006, 1:1      5 method    the library’s project methodology, based on  the project management body of knowledge  (a guide), was used to plan the  redevelopment of the library’s intranet site.  the methodology was enhanced by  incorporating the principles of evidence based  librarianship. to effectively assimilate the ebl  process into the project plan, a series of  workshops were scheduled to develop the  team’s understanding of the theory, and to  explore its potential for integration into  practice. the program included six journal  club sessions structured around the ebl  process, and used the library’s current projects  (carter and grimmond; cotter et al.) as case  studies throughout the series. project team  members were responsible for preparing and  presenting one session each. an introductory  overview session was followed by five  workshops examining each stage of the ebl  process in turn.    after a research question was framed,  librarian‐observed, user‐reported, and  research‐derived evidence was gathered and  appraised (booth ʺexceedingʺ). findings were  applied to the project in varying degrees –  directly, by derivation, conditionally, or just by  providing enlightenment (booth ʺwhatʺ).  finally, the project outcomes were evaluated  and disseminated. like the phases of a project  plan, the main processes of ebl are not  necessarily independent and consecutive  activities. they are inter‐related,  interdependent, and can occur concurrently.    the main phases of the project were to:  • undertake a preliminary revision of  the site to eliminate incorrect, obsolete,  unnecessary, and poorly presented  pages. this provided a suitable  starting point from which further  evidence‐based improvements could  be made.   • locate, retrieve, and examine the  evidence to identify required  modifications. the evidence was taken  from the literature, usability testing,  and feedback from library staff and  clients.   • implement the recommended changes,  and publish the redeveloped and now  highly‐usable intranet site.   • define guidelines for the ongoing  maintenance and development of the  site.  • evaluate the new intranet site and the  project team’s utilisation of ebl.  • disseminate the results via a launch of  the new site, presentation of a hot  topic at the 3rd international evidence  based librarianship conference, and  publication of the project report   refining the question    the spice anatomy (booth ʺformulatingʺ 65‐ 66) was used to focus the direction of the  project and refine the research question (table  1):      s  setting  cchs library intranet site  p  perspective  staff and students of the organisation  gateway to our services & resources    i  intervention  site improvements  c  comparison  original site  e  evaluation  usability (as a determiner of effectiveness)  table 1. spice breakdown  evidence based library and information practice 2006, 1:1      6 asking a clear and answerable question at the  start of the project was vital to its successful  completion. one well‐formulated question had  been developed to focus the project’s scope,  but additional questions were needed,  particularly at the literature review stage. the  literature review became problematic because  the project team attempted to use one search  strategy to answer several different questions  such as “what makes a good intranet site?”  and “how do you conduct usability testing?”    literature review    preliminary searching revealed a large amount  of literature on design and evaluation of  internet/intranet sites. initially, the research  question used the terms usability and  effectiveness interchangeably. the review  clarified the distinctions between these two  terms and related terms such as usefulness. this  awareness led to the original question being  focussed to specify usability as the evaluation  measure, thereby limiting the number of  potential evaluation tools that would require  examination.    a search strategy was designed, with inclusion  and exclusion criteria. databases covering the  literature of library & information science  (lisa, alisa, library literature, apais/apaft,  emerald library), health (medline, cinahl,  embase, austhealth), and technology (gartner  core research, ieeexplore) were selected.  several multidisciplinary databases (proquest  5000, expanded academic index, megafile premier,  blackwell synergy, sciencedirect) were also used.  the internet was also searched. the project  team considered it an appropriate source since  many organisations publish their usability  studies on their websites (campbell et al.;  clairmont, dickstein and mills; mit libraries).  a few core texts provided background  information (krug; norlin). responsibility for  searching was divided among the project team.  one team member aggregated the individual  summaries into a documented literature  review allowing the team’s individual efforts  to be considered as a whole.    the main themes to emerge from the literature  review were: distinctions between usefulness,  effectiveness, and usability; usability testing in  practice; and web design guidelines.     various definitions of usability were identified,  the most useful version provided by the  international organization for standardization  (iso 13407): “the usability of an interface is a  measure of the effectiveness, efficiency and  satisfaction with which specified users can  achieve specified goals in a particular  environment with that interface”. the project  plan acknowledges that usability is only one  indicator of the overall effectiveness of an  intranet site, and that other components, such  as accessibility and the extent to which clients  needs are being met, also contribute to  effectiveness. such factors, however, are  beyond the scope of this project.    usability testing “does not tell you who is  actually using your web‐based services,  whether their needs are being met, or whether  you are reaching your target clientele” (pearce  20).this is directly applicable to the present  study, as the project team were only testing the  existing content of the site, not resources or  services that potentially could be included.  also, the research participants were all users  of the organisation’s intranet site at the time of  their participation, so the project was not  reaching non‐users, or potential users, nor  identifying barriers to use.    the literature search yielded a large number of  resources describing practical approaches to  usability testing. mit libraries’ web site  usability test, and an entire issue of library  technology reports on usability testing (pace)  were particularly helpful. they provided  practical guidance in carrying out usability  testing under real‐life conditions and included  examples of test protocols, sample questions,  and data recording sheets. steve krug’s book  don’t make me think! was useful in providing  details on how to administer the usability  sessions, including a sample script for the  testers to use, room setup, and how to interact  with the subjects of the study.    evidence based library and information practice 2006, 1:1      7 as well as such case reports, numerous  guidelines and lists of recommendations were  uncovered, with varying strengths of evidence  and degrees of applicability. much of the  literature deals with commercial websites, but  libraries and intranet sites may have a  different focus (pace 35). critical appraisal  resulted in two instruments being selected for  use in this project. they were research‐based  web design and usability guidelines (koyani,  bailey and nall) which includes 187  recommendations underpinned by research,  with each carrying a strength of evidence, and  a relative importance score; and the “raward  library usability analysis tool” (raward,  ʺstudyʺ 226‐30), which is a best practice  checklist for the design of library websites  intended to be a cost‐effective way to conduct,  user‐centred studies without user involvement  (raward, ʺreportʺ).     results of the literature review were appraised  for validity, reliability, and applicability; yet,  the team concede that “intrinsic factors that  relate to research design and aspects of  methodology” (booth and brice 105) were only  appraised superficially. use of appraisal tools,  such as the cristal checklists (booth,  ʺcristalʺ), would have allowed critical  assessment of the intrinsic factors.    client feedback    an e‐mail sent organisation‐wide, together  with a link prominently displayed on the  library’s intranet home page, invited feedback  and suggestions via a simple online survey.  the objective of the survey was to establish the  use which was actually being made of the  intranet site, and to reveal the client  perspective of the functions the site should be  performing. the survey asked:    • is the library intranet site easy to use?  • what are your main reasons for using  the library site?  • what would you like to use the library  site for?  • would you use links to selected  internet sites for your speciality?    the intent of the survey was to ensure the  project goals and client needs were aligned in  order to avoid the pitfall of “broccoli  librarianship” – librarians imposing on clients  what they should do because that is what is  good for them (vaughn and callicott). in  conjunction with observations made by library  staff, results of the survey were used to  compile a list of the major functions the library  intranet site was expected to perform. a  further function of the survey was to enlist  volunteers to take part in the scenario‐based  usability testing of the site.     qualitative data which represented client  feelings about the usability of the site was  elicited via questions posed during a post‐test  discussion (see appendix a). capturing  participant views immediately after they used  the site to perform scenario‐based tasks  enhanced the quality of the responses. the  post‐test discussion was used in lieu of focus  groups as the project team was unable to  recruit volunteers from busy clinical areas for  focus groups.    client feedback from the online survey and  post‐test discussions was appraised for  validity and contextual relevance.    usability testing    usability testing was undertaken in two forms.  the raward usability checklist was applied to  the intranet site by individual members of the  project team to identify how the site fell short  of recognised best practice for usability.  library clients performed scenario‐based tasks  while under observation to provide a usability  assessment based on specific, local needs and  use of the site.     checklist analysis  the “raward library usability analysis tool”  (raward, ʺstudyʺ 226‐30) was applied to the  library’s intranet site before and after a  preliminary revision of the site based only on  the project team’s professional judgement. the  raward checklist was next applied to the site  after its major redesign at the end of 2005 (the  current site), with the intention of applying it  evidence based library and information practice 2006, 1:1      8 at each major revision of the site. this tool, a  103 question checklist, is the result of an  australian librarian’s research project which  examined the usability of library websites.  although the tool was developed specifically  for academic library websites, it was deemed  applicable for this project on the basis that it  was designed for non‐commercial, service‐ oriented organisations. each time the tool was  used, at least two independent evaluators  applied it to the intranet site and results were  compared.    scenario‐ based tasks  scenario‐based usability testing of the library’s  intranet site was based upon the fourteen main  functions of the site derived from the survey.     in response to a challenge made to librarians  who conduct usability testing to consider the  role of a library’s website in relation to the  library itself (vaughn and callicott 16), the  project team determined that the library’s  intranet site assumed both the roles of a  substitution for and an extension of the  physical library. vaughn & callicott argue that,  if the site is regarded as a substitute for the  library, then the usability test instrument  should not contain any hints for the users, for  example, a simple statement such as  “databases allow you to search for articles on  various topics”. without such hints, the  usability test will test ease of use. if the site is  seen as an extension of the library, then  minimal instructional statements should be  included in the test instrument and the focus  of the test will change to usefulness. in both  cases, the tasks posed must be sure to test the  usability of the site, and not the information  literacy skill of the participant.    the substitution/extension approach becomes  problematic when the site has to fill both roles.  the cchs library is part of a 24/7  organisation. when the library is open, the  intranet site functions as an extension of the  library, but when the library is closed, some  parts of the organisation (inpatient wards,  emergency department) continue to function  and rely on the intranet site as a substitute for  the library. throughout the testing process,  this was one issue which was not satisfactorily  resolved. however, formation of the scenario‐ based tasks ‐ meant to reflect real‐life use of  the site ‐ tended towards testing ease of use.  the scenarios posed in this usability study are  included as appendix b.    clients who completed the online survey and  indicated on the survey form their willingness  to participate in the scenario‐based testing  sessions were contacted to organise session  times. renowned usability expert jakob  nielsen states that “the best results come from  testing no more than 5 users and running as  many small tests as you can afford” (nielsen).  according to nielsen, a usability study with  five subjects will identify 85% of a site’s  usability problems. more users would be  required if highly distinctive user groups are  being represented, in which case, three to four  from each group should be selected. on this  recommendation, the project team, which  plans to undertake a round of usability testing  at each stage of the intranet redevelopment,  will conduct in each round two tests of seven  scenarios with participants representing the  intranet’s major client groups ‐ hospital staff  (clinical and administrative) and university  students. each volunteer will only be eligible  to participate in one round of testing, but the  participant profile at each round will be  consistent.     at the time of this writing, the first round of  testing has been completed. eight hospital staff  members, representing the clinical and  administrative client groups, participated.  students were excluded from the first round of  usability testing on the basis that the site was  not yet a seamless interface for the two distinct  client groups. testing seamlessness at this  early stage of the project would not have  yielded useful results. the first round of  revisions addressed this problem and so the  next round of usability testing will examine  the impact of these changes by including  students in the participant cohort.    the scenario‐based usability test sessions were  run by two members of the project team: one  as administrator, and one as recorder. the  evidence based library and information practice 2006, 1:1      9 administrator welcomed the volunteer,  administered the participant profile  questionnaire, conducted the scenarios, and  facilitated the post‐test discussion (see  appendices c, b and a). the recorder had  minimal interaction with both the volunteers  and the administrator and, instead,  concentrated on the documentation of the  sessions on recording sheets (see appendix d).  for each scenario there was a recording sheet  which set out the preferred navigation path.  the recorder ticked off each step and noted  any variation, as the volunteer moved through  the intranet site. other details recorded  included the number of clicks needed to  complete the scenario, whether the volunteer  was successful or unsuccessful in completing  the scenario and any problems of navigation,  architecture, content, terminology,  presentation, search tool, or user support  which were encountered. the administrator  kept a separate record (see appendix e) and  each session was audio‐taped. screen capture  software would have been ideal for recording  the usability test sessions. unfortunately, the  project team did not have access to the  necessary equipment or budget required to  purchase it. in order to obtain the most  accurate record of the session possible, the  administrator and recorder reviewed each  session immediately after the volunteer left  and combined notes (see appendix f).     usability testing was conducted in the library  rather than on the wards, which would have  better emulated real‐life use of the site. testing  in the library was a practical necessity due to  the difficulties of computer access and the  interruptions which would be expected to  occur in the busy clinical workplace.    results of the scenario‐based usability testing  were appraised for validity by examining the  actual responses of the participants to each  scenario against the information the project  team expected to gain from each scenario. the  methodology was also appraised for reliability.  for example, there was some inconsistency in  the way in which the scenario‐based test  sessions were administered and recorded by  the project team. also, the project team were  administering the tests largely to participants  known to them, and were also responsible for  redesigning the intranet site, and therefore  some bias in the results was inevitable.    research based guidelines    once usability issues had been identified, the  usability checklist applied, and the scenario‐ based usability testing conducted, research‐  based web design & usability guidelines  (koyani, bailey and nall) was referred to for  evidence based solutions. each guideline in  this document includes ʺa brief statement of  the overarching principle that is the  foundation of the guideline, further  explanatory comments on the research,  graphic examples of the guideline in practice  and citations to relevant websites, technical  and research reports supporting the guidelineʺ  (booth, ʺuntanglingʺ 71). to assist in the  process of applying the recommendations  locally, each guideline is assigned a strength of  evidence and relative importance score.     once relevant guidelines were identified, they  were applied, taking into account local  conditions such as a fixed intranet template  and organisational structure and culture.     results    raward’s checklist was an effective way to  identify where the site fell short of best  practice in design for usability. keevil explains  that usability checklists “can measure the  usability index of a website and determine,  expressed as a percentage, how closely the  features of a particular website match  generally accepted usability guidelines” (qtd.  in raward, ʺstudyʺ 109). not only can the  usability scores of versions of the site be  compared, but also, by virtue of the  characteristics marked no, a list of  recommendations to improve usability can be  extracted. the checklist identified several  aspects of the site which needed attention, and  this method alone would have resulted in a  revised site with significantly enhanced  usability. evidence based library and information practice 2006, 1:1      10   42 79 98 0 20 40 60 80 100 usability index (%) dec 03 jun 05 nov 05 library intranet site   figure 1. raward library usability index when the checklist was applied to the original  site (an archived version as at december 2003),  it recorded a usability index of only 42%. after  the initial revision based only upon  professional judgement (june 2005), the  usability index improved considerably,  achieving 79%. when the site was substantially  redesigned in november 2005 to address the  issues identified by the usability study, the  score again improved, this time to 98% (figure  1).  the results of the scenario‐based usability  testing demonstrated a fairly usable site,  though one with plenty of room for  improvement. overall, participants were able  to successfully complete the tasks 87% of the  time, but only 60% of the time by the preferred  path. scenario‐based testing supported the  findings of the usability checklist and had the  added benefit of allowing the project team to  observe how clients used the site to satisfy  particular information needs. not only were  problems with the site identified, but some  insight into why the problem existed was also  gained. valuable feedback on other library  services and a positive public relations exercise  were unforeseen benefits.    analysis of the library intranet site using  raward’s checklist showed that the site was  strong in the areas of technical accuracy and  providing information that is easy to  understand; less strong in presenting  information (including navigation and  formatting); and relatively weak in supporting  user tasks (including an index, site map, help  screens, and faqs). usability testing  supported these findings by pinpointing   specific functions that clients expected to be  able to carry out using the intranet site, but  which were difficult in practice. as a result,  the following major changes were made to the  cchs library intranet site:  • inclusion of a site index  • phone, fax and, postal address  included on home page  • details for contacting library staff  included on each page  • commonly‐used links made available  on each page  • a link to the home page added to each  page  • headings and links made jargon‐free  and task‐based (“renew loans”  replaced “mylibrary”)  •  “online resources” page redesigned  to group related elements together  • provision of an a‐z list of electronic  journals  • hyperlink labels synonymous with  their destination page titles  • images no longer used to represent  hyperlinks    the redesign includes a new look, presentation,  and format, and the site has been formally  launched. a second round of usability testing  is scheduled for early 2006.        evidence based library and information practice 2006, 1:1      11 discussion    usability testing lies within the ambit of  evidence based librarianship in that it is  practical, grounded testing carried out in order  to obtain evidence with which to address  relevant and answerable questions facing the  practice of librarianship.   usability testing is part of a continuous cycle  of development, implementation, and  evaluation. ebl is based on a cyclical process  of evaluation and improvement, and was,  therefore, a suitable framework for this project.  although standard project methodologies  emphasise formal closure of the project as a  final stage, an ongoing program of usability  testing would transcend the project lifecycle  and become a part of the ongoing maintenance  and development of the site.    the methodology used to enhance the  usability of this library intranet site would be  just as applicable to a usability study of other  library internet/intranet sites. it is anticipated  that the findings of the project will be used to  propose the launch of an internet site for the  cchs library.     with usability addressed, future projects could  examine how to enhance other factors that  contribute to the library site’s effectiveness. is  the site meeting the needs of clients? is the site  accessible by clients? do the potential users of  the site have the skills necessary to use the  resources provided? if such investigation leads  to further revision of the site, modifications  would be subject to usability testing, as a  matter of course.     usability testing  the raward library usability analysis tool  was extremely valuable in the early stages of  site development, promoting adoption of  recognised best practice. ongoing regular use  of the checklist is limited by the subjectivity of  certain sections of the assessment. however, it  remains an extremely useful tool for  maintaining a focus on best practice guidelines.    the project team found that, despite requiring  a large investment of time, scenario‐based  testing yielded valuable information about the  library’s intranet site and exposed usability  issues obscured by the limitations of the  checklist analysis. the next round of scenario‐ based testing is expected to demonstrate that  the usability index of 98% scored in november  2005 is an artificially high indicator of usability.    applying raward’s checklist to the intranet  site required a much smaller commitment of  time and effort than conducting scenario‐based  testing. writing the scenarios, scripts, and test  protocol; making appointments with subjects;  setting up the room; administering the tests;  analysing the results; and so on, consumed  large amounts of time. conducting research as  a team makes it a challenge to ensure  consistency. procedures for administration of  the tests and recording of data were  standardised, but variations still occurred. in  contrast, each time raward’s checklist was  applied to the intranet site (december 2003,  june 2005, november 2005), two or more  members of the project team applied the  checklist independently of each other but  obtained almost identical results. this  confirmed the tool’s suitability for the purpose  for which it was designed – a quick and easy  test of website usability requiring only a sole  evaluator.    the raward checklist and scenario‐based  testing were found to be complementary tools  for identifying usability issues. to maximise  the benefit of scenario‐based testing (especially  because it is so resource intensive), the  methodology used here could be improved by  implementing the changes recommended by  raward’s best‐practice guidelines, first. this  would avoid scenario‐based test subjects  experiencing problems with usability, which  could have been easily identified and resolved  in the first instance, thereby maximising the  project team’s opportunity to gather valuable  local, client‐specific feedback.    this project did not test enough subjects to  enable each usability objective (such as the  time taken to complete a task, number of clicks,  or completion via the preferred pathway) to be  measured to a particular confidence level. to  evidence based library and information practice 2006, 1:1      12 do so was beyond the resources of the project  team. however, enough subjects (eight in total)  were tested to obtain meaningful results for  this type of research. it was the experience of  the project team that after the first two or three  usability tests, the subjects tended to identify  the same problems and experience very similar  difficulties with the site. for example, none of  the subjects were able to successfully complete  scenario 4 (renewing loans online) due to the  absence of a link for this function on the home  page as well as unclear instructions for its use  if/when the link was located.     application of ebl  a project such as this cannot be achieved in  isolation from the normal operation and  management of the library. the project team  undertook the usability study, not only as a  means of improving the library’s intranet site  but also “to explore the realities and  practicalities of the eblip process within the  constraints offered by a library practitioner  role” (booth, ʺaustralianʺ 71).    cchs library staff developed a strong  understanding of ebl and its practical  application over the course of this project  largely due to the workshop series, prepared  and presented by the project team.     one barrier to the ebl approach was that the  project team did not have easy access to major  library and information science databases with  international coverage, e.g., library and  information science abstracts (lisa) and library  literature. access to the indexed professional  literature is required for ebl to make the  transition from theory to practice. the project  has also highlighted the difficulties of  applying ebl when tools such as appraisal  checklists are still under development or  difficult to locate.    following through all the stages presented by  the ebl model, rather than just engaging with  specific components, not only enhanced  project outcomes but also enriched the  professional development experience. for  instance, project evaluation normally marks  the end of a project. however, the ebl model  encourages dissemination of results as a final  stage, to enhance the evidence base available  to the profession for future decision making.  knowledge that the project would be  presented to an audience of peers, made the  project team more rigorous at every stage ‐  from designing methodology to analysing  results.    project management  the experience of the project team has been  that project management and ebl are  complementary and can enhance the outcomes  of any project. ebl was identified as a way of  ensuring best practice; project management  ensures it is achieved efficiently. just as  flavour is enhanced by adding spice when  cooking, embedding ebl in a project’s  methodology should produce a blueprint to  enable a good project to become even better.     the usability project has demonstrated that  ebl is not a “bolt‐on extra” but “can be  integrated within the accepted structure of the  project management cycle” (booth,  ʺaustralianʺ 71).  the cchs library’s  standard project methodology has been  refined to allow for the ebl model to be  incorporated locally on an ongoing basis. the  project team is now developing this project  planning tool for wider application. it is  anticipated that provision of such an  instrument will help to bridge the gap between  evidence based librarianship as a theory  presented in a text book, and evidence based  librarianship routinely applied in the  workplace to enhance decision making.       acknowledgement    a version of this paper was presented at the 3rd  international evidence based librarianship  conference, held in brisbane, australia  (october 2005). see:   <http://conferences.alia.org.au/ebl2005/cotter. pdf>.          http://conferences.alia.org.au/ebl2005/cotter evidence based library and information practice 2006, 1:1      13 works cited    booth, andrew. ʺbriefing paper #1. facilitated  online learning interactive opportunity  (folio): evidence based librarianship  (ebl)ʺ. 2003. national electronic library  for health. 29 april 2003.  <http://www.shef.ac.uk/scharr/eblib/briefi ng1.htm>.    ‐‐‐. ʺaustralian supermodel?‐‐a practical  example of evidence‐based library and  information practice (eblip).ʺ health  information and libraries journal 23.1  (2006): 69‐72.    ‐‐‐. ʺcristal onlineʺ. 2003. 5 september 2005.  <http://www.shef.ac.uk/scharr/eblib/>.    ‐‐‐. ʺexceeding expectations: achieving  professional excellence by getting  research into practice.ʺ lianza 2000.  christchurch, new zealand, 2000.    ‐‐‐. ʺformulating answerable questions.ʺ  evidence‐based practice for information  professionals: a handbook. eds. andrew  booth and anne brice. london: facet, 2004.  61‐70.     ‐‐‐. ʺuntangling the web. using research in  practice.ʺ health information and libraries  journal 21.1 (2004): 70‐73.    ‐‐‐. ʺwhat research studies do practitioners  actually find useful?ʺ health information  and libraries journal 21.3 (2004): 197‐200.    booth, andrew, and anne brice. ʺappraising  the evidence.ʺ evidence‐based practice for  information professionals: a handbook.  eds. andrew booth and anne brice.  london: facet, 2004. 104‐17.     campbell, nicole, et al. ʺdesigning for the  user: how to test for usabilityʺ. 2001. 27  may 2005.  <http://www.vancouver.wsu.edu/fac/camp bell/usability/acrl/>.    carter, sharna, and mary grimmond.  ʺrehabilitating the stroke collection.ʺ 3rd  international evidence based  librarianship conference. evolution of  evidence : global perspectives on linking  research with practice. brisbane, australia,  2005.    clairmont, michelle, ruth dickstein, and vicki  mills. ʺtesting for usability in the design  of a new information gatewayʺ. 30 may  2005.  <http://www.library.arizona.edu/library/te ams/access9798/lft2paper.htm>.    cotter, lisa, et al. ʺadding spice to our  library intranet site: a recipe to enhance  usability.ʺ 3rd international evidence  based librarianship conference. evolution  of evidence : global perspectives on  linking research with practice. brisbane,  australia, 2005.    a guide to the project management body of  knowledge: pmbok guide. 3rd ed.  newtown square, pa: project  management institute, inc., 2004.    iso 13407: human‐centred design processes  for interactive systems. geneva:  international organization for  standardization, 1999.    koyani, sanjay j, robert w bailey, and janice r  nall. research‐based web design &  usability guidelines. washington, d.c.:  united states department of health and  human services, 2003.    krug, steve. donʹt make me think! : a  common sense approach to web  usability. indianapolis, ind.: new riders,  2000.    mit libraries. ʺweb site usability testʺ. 1999.  mit libraries. 27 may 2005.  <http://macfadden.mit.edu/webgroup/usa bility/results/index.html>.    nielsen, jakob. ʺwhy you only need to test  with 5 usersʺ. 2000. jakob nielsenʹs  http://www.shef.ac.uk/scharr/eblib/briefi http://www.shef.ac.uk/scharr/eblib/ http://www.vancouver.wsu.edu/fac/camp http://www.library.arizona.edu/library/te http://macfadden.mit.edu/webgroup/usa evidence based library and information practice 2006, 1:1      14 alertbox. 20 june 2005.  <http://www.useit.com/alertbox/20000319. html>.    norlin, elaina. usability testing for library  websites: a hands‐on guide / elaina  norlin, cm! winters. chicago: american  library association, 2002.    pace, andrew k. ʺoptimizing library web  services: a usability approach.ʺ library  technology reports 38.2 (2002): 2‐81.  pearce, judith. ʺtesting the usability of the  nla web site.ʺ national library of  australia gateways 51 (2001): 20‐21.    raward, roslyn a. ʺa report on the  development of a usability analysis tool  for the evaluation of library websites.ʺ  sydney, 2003. information online 2003:  11th conference and exhibition. (21‐23  january 2003).  <http://conferences.alia.org.au/online2003/ papers/raward.html>.    ‐‐‐. ʺa study of best practice design  guidelines and the development of a  usability analysis tool for the evaluation  of australian academic library web  sites.ʺ thesis (m.a.). university of  canberra, 2002.    vaughn, debbie, and burton callicott.  ʺbroccoli librarianship and google‐bred  patrons, or whatʹs wrong with usability  testing?ʺ college & undergraduate  libraries 10.2 (2003): 1‐18. http://www.useit.com/alertbox/20000319 http://conferences.alia.org.au/online2003/ evidence based library and information practice 2006, 1:1      15 appendix a   post test discussion      thank you for your participation. please take a moment to answer the following:      1. when you were completing the tasks, how did you feel (frustrated, confident, etc.)?                  2. what do you like about the site?                   3. if you were to change anything about the site, what would you change?                   4. any other comments?                  5. what is your overall assessment of the site? assign a mark out of 10.  (1 is poor, 10 is great)    1  2  3  4  5  6  7  8  9  10  evidence based library and information practice 2006, 1:1      16 appendix b   scenario‐based usability test tasks  scenario 1  you would like to request a change in the gosford or wyong hospital library’s opening hours. how  would you submit your suggestion via the library’s intranet site?    objective: can the client provide feedback to the library? can the client locate library forms to  request services?    scenario 2  you have determined that the journal that has the article you need is not held in the gosford nor  wyong hospital libraries, nor is it available online. find an electronic form that you can use to  request a photocopy of this article from another library.    objective: can the client request journal articles not directly available through cch library services?  can the client locate library forms to request services?    scenario 3  you need to talk to the librarian who liaises with the university of newcastle. locate their contact  details on the library’s intranet site.    objective: can the client contact the library staff member best able to answer their inquiry?    scenario 4  its 10.00pm and you have just started your shift when you realise that your library books are due  today, but the library is closed. demonstrate how you would renew your books using the library’s  intranet site.    objective: can the client manage their library borrower account (such as renew books)?    scenario 5 (cch staff only)  your manager/num has asked you to review a procedure commonly performed in your  department/ward. any changes you make must be evidence‐based. you would like the librarian to do  a search for the latest information on this topic. it is after hours and there is no phone help available.  show how you would submit a search request using the library’s intranet site?    objective: can the client locate library forms to request services?    scenario 6  cch staff : you would like to update your information‐seeking skills and learn more about ciap.  using the intranet site, how will you find out about training offered by the library?    evidence based library and information practice 2006, 1:1      17 university :you would like to update your information‐seeking skills and learn more about the  university of newcastle’s online resources. using the intranet site, how will you find out about  training offered by the library?    objective: can the client seek help to improve their information skills? can the client contact the  library staff member best able to answer their inquiry?     scenario 7  you are looking for a book on cardiac nursing and wonder whether the gosford or wyong hospital  libraries have any. how would you find out using the library’s intranet site?    objective: can the client identify the catalogue, newcat?    scenario 8  you need to search mims and the print copies have all gone missing, so you have to look up the  online version. show us how you would access mims online using the library’s intranet site?    objective: can the client access specific databases such as medline, or mims? can the client access  ciap or the university of newcastle’s online resources as appropriate?    scenario 9 (cch staff only)  your colleague has told you there is a direct link on the library’s intranet site to the online edition of  the european respiratory journal. view the current issue of this journal.     objective: can the client locate full‐text journal articles, or the online edition of a journal?    scenario 10  you need to read an article in the latest issue of the lancet. demonstrate how you would access the  online version of the lancet via the library’s intranet site?      objective: can the client locate full‐text journal articles, or the online edition of a journal? can the  client access ciap or the university of newcastle’s online resources as appropriate?     scenario 11  cch staff :you have heard about ciap and are keen to know more. go to ciap via the library’s  intranet site so that you are able to browse the resources available.    university : you have heard about the university of newcastle’s journal databases and are keen to  know more. go to the list of online databases provided by the university of newcastle so that  you are able to browse the resources available.    objective: can the client access ciap or the university of newcastle’s online resources?     evidence based library and information practice 2006, 1:1      18 scenario 12  you need to search the health literature for information on a work/study related topic. the library is  not open to seek help. demonstrate how you would begin.     objective: can the client select appropriate information resources to find required information?    scenario 13  you would like to recommend that the library purchase a particular book. demonstrate how you  would make this recommendation using the library intranet site.    objective: can the client submit suggestions for new book purchases? can the client locate library  forms to request services?    scenario 14  you would like to know what is published in your favourite journal each month. you have heard table  of contents alerts are available, but are unsure as to how to go about setting them up. use the library’s  intranet site to find help with this.     objective: can the client locate table of contents (toc) and other current awareness services?      assistance guidelines    if participant seeks help as they are lost, general advice would be to “return to the library  homepage and start again” and/or refer the participant back to the question.    if participant is prompted to enter a password and seeks help – first, explain which password is  required and encourage them to enter it if they know it. if they don’t know it or don’t have one,  the test administrator should enter one, and the test monitor should note this as an issue. in the  post‐test interview, clear up any password issues.    if the participant is obviously on the wrong track (due to misinterpretation of the question), the  test administrator should clarify the scenario.    usability test sessions    the 14 tasks have been divided into 2 separate tests. each test will be administered to a group of  participants with a similar profile. the 7 tasks in each test will be administered in random order to  each participant. tasks 5 and 9 will only be administered to cch staff.    test a  tasks 1, 2, 7, 9, 11, 12, 14    test b  tasks 3, 4, 5, 6, 8, 10, 13      evidence based library and information practice 2006, 1:1      19 the questions will be presented to the participants in the following order:    test a    participant  task order  a1  9  2  7  14  11  1  12  a2  14  12  7  11  9  1  2  a3  12  14  2  1  11  7  9  a4  7  12  14  9  1  11  2  a5  14  1  9  12  7  2  11  a6  12  7  1  14  2  9  11  a7  12  11  7  2  9  1  14      test b    participant  task order  b1  6  8  10  5  13  3  4  b2  3  10  6  8  5  13  4  b3  5  6  3  4  13  8  10  b4  3  4  10  5  13  8  6  b5  3  13  10  6  5  4  8  b6  3  5  13  10  4  6  8  b7  13  3  5  10  8  6  4      evidence based library and information practice 2006, 1:1      20 appendix c   participant profile questionnaire    tester number :   _____________________________       test version :    a   b  test date/time :    _____________________________                                 personal profile    domain :      allied health   medicine   nursing   student   other   __________________________    which organisation’s online resources are you eligible to access ?   central coast health         the university of newcastle     is english your first language ?   yes         no       computer experience    how often do you use a computer (at home or at work) ?   daily         weekly         monthly         almost never         i have never used a computer    how often do you use the hospital’s intranet ?   daily         weekly         monthly         almost never         i have never used the hospital’s intranet    how often do you use the cch library services’ intranet site ?   daily         weekly         monthly         almost never         i have never used the library’s intranet site    do you agree to an audio‐recording of the test session being made?   yes         no   evidence based library and information practice 2006, 1:1      21 appendix d  usability test recording sheet – recorder (sample)  scenario 1  you would like to request a change in the gosford or wyong hospital library’s opening hours. how would  you submit your suggestion via the library’s intranet site?    objective: can the client provide feedback to the library? can the client locate library forms to request  services?    criteria for success: ask a question form is located, or appropriate contact details are identified.    preferred path: ask a question logo/link    or  contact us | identify the library manager, general inquiries, or ask a question form    time allowed: 3 minutes (maximum)      browsing   note path taken & navigation tools used:     top navigation bar   left navigation bar   quicklinks   logos      search tool  was the search  tool used?       yes        no        search terms used :  did the search tool  provide the  answer?     yes        no    comments:    describe issues/problems:                  summary      successful via expected path   successful via unusual path   error made, though recovered   error made, did not recover  evidence based library and information practice 2006, 1:1      22 appendix e  usability test recording sheet – administrator (sample)  scenario 1  you would like to request a change in the gosford or wyong hospital library’s opening hours. how would  you submit your suggestion via the library’s intranet site?    objective: can the client provide feedback to the library? can the client locate library forms to request  services?    criteria for success: ask a question form is located, or appropriate contact details are identified.    preferred path: ask a question logo/link    or  contact us | identify the library manager, general inquiries, or ask a question form    time allowed: 3 minutes (maximum)      clicks                          password issues     internet   ciap   database   newcat pin    comments :    comments  evidence based library and information practice 2006, 1:1      23 summary   successfully completed within time allowed ?    yes      no     number of clicks:      evidence based library and information practice 2006, 1:1      24 appendix f  usability test recording sheet – summary (sample)  scenario 1  you would like to request a change in the gosford or wyong hospital library’s opening hours. how would  you submit your suggestion via the library’s intranet site?    objective: can the client provide feedback to the library? can the client locate library forms to request  services?    criteria for success: ask a question form is located, or appropriate contact details are identified.    preferred path: ask a question logo/link    or  contact us | identify the library manager, general inquiries, or ask a question form    time allowed: 3 minutes (maximum)      summary     successfully completed within time allowed ?     successful via expected path   successful via unusual path   error made, though recovered   error made, did not recover            yes     no        number of clicks:    ______      navigation     top navigation bar   left navigation bar   quicklinks   logos      issues identified      level of concern     comment  architecture…………………. .   major    minor    ………………………………………………………  content      – missing…………………..   major    minor    ………………………………………………………  – out‐of‐date / erroneous...   major    minor    ………………………………………………………  navigation       – cch intranet site……….   major    minor    ………………………………………………………  – library services site……   major    minor    ………………………………………………………  presentation………………....   major    minor    ………………………………………………………  search tool…………………..   major    minor    ………………………………………………………  terminology……………….... .   major    minor    ………………………………………………………  user support / help………….   major    minor    ………………………………………………………        evidence based library and information practice 2006, 1:1      25   other      password issues     internet   ciap   database   newcat pin        evidence based library and information practice 2012, 7.1 102 evidence based library and information practice evidence summary public library summer reading programs contribute to reading progress and proficiency a review of: roman, s., & fiore, c. (2010). do public library summer reading programs close the achievement gap? children and libraries, (winter), 27-31. reviewed by: gayle bogel assistant professor, educational technology/school library media graduate school of education and allied professions fairfield university fairfield, connecticut, united states of america email: gbogel@fairfield.edu received: 28 nov. 2011 accepted: 28 jan. 2012 2012 bogel. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objectives – to explore summer reading partnerships between public libraries and school libraries, and the impact on student achievement in reading. design – the design is mixed methods: tests, interviews and surveys. setting – eleven us sites involving school and public library partners. subjects – a total of 357 elementary school students entering fourth grade that met specific criteria. parents, teachers, school librarians and public librarians were also included. methods – this study occurred over a three year period from 2006-2009. it was developed as a partnership between dominican university, the colorado state library agency and the texas library and archives commission. additionally, the center for summer learning at johns hopkins university was contracted to conduct the research. it was designed to explore the research question central to a 1978 study by barbara heyns. an advisory committee, with representatives from each partner agency, developed evaluation questions and established the objective selection criteria for participants in the study. the criteria included: at least 50% of students qualifying for free and reduced price mailto:gbogel@fairfield.edu� evidence based library and information practice 2012, 7.1 103 meals; at least 85 percent of school population would take the reading proficiency test in english; public library summer reading programs with a minimum of six weeks of programming; a history of collaboration between the paired school and public library applicants; both school and public library would sign a partnership agreement and participate in conference calls. in spring of 2008, students at participating sites were given a pre-test using the scholastic reading inventory, and also provided with special summer reading logs developed for the study, to be used during their subsequent participation in the public library summer reading programs. in fall of 2008, the same children were tested on the scholastic reading inventory. in addition, surveys of students, parents, teachers and library staff were administered, as well as a structured interview of library staff. main results – the students who participated in the public library summer reading program did not experience summer loss in reading as measured by the post test administered in their schools. the researchers note patterns in the demographics of students who chose to participate (participation was self-selected and voluntary): predominantly caucasian girls above poverty level, who generally scored well on reading tests. those who participated in the study also used libraries, had more books in their homes, and parents who used libraries and provided literacy activities at home. teachers found that students who participated in the program started the school year ready to learn, had improved reading achievement and displayed stronger motivation, enjoyment and confidence in reading. parents of students who participated in the program also strongly agreed that their children where better prepared to begin the school year. conclusion – the results from this study confirmed findings from an earlier study (heyns, 1978) and subsequent related research that summer reading programs in public libraries can contribute to maintaining reading progress and proficiency, and that the recreational reading that is available to all students regardless of socioeconomic status through the public library can make a difference in reading scores. in this article the researchers presented results and conclusions based on selected subsets of the results in the full study. the participants were self-reported and no control group was established. the researchers also use the results of the study as a starting point to provide a call to action that highlights the value of public library summer reading programs, and the need for the education community to invest in partnerships with public libraries, particularly in communities that serve children and families in low-economic or depressed areas. they also note the need to include parents and grandparents in outreach efforts. commentary while there is significant research to support the value of public libraries in teaching children to read – not as instructional supports, but to provide the vitally important recreational reading that helps children develop as engaged readers – the education community has been slow to focus on the value of partnerships between school and public libraries. public libraries provide support for independent reading that builds on the instructional role of the school and classroom teachers, and augments the school library collections, particularly during the summer months. the authors’ methodology was designed to measure the effect of summer reading programs on at-risk students, but, by the description of the participants and actual data collected, it is not clear whether they were able to achieve this goal since it appears that the majority of actual participants were not at-risk students. the authors did, however, translate the entirety of the research to a call for action evidence based library and information practice 2012, 7.1 104 that encompasses the potential value of collaborative efforts between agencies and is applicability across several states. previous research has consistently shown that pleasurable reading builds fluency and confidence, and children who enjoy reading a wide variety of materials are likely to have higher scores on reading achievement tests (krashen, 2004). particularly for disadvantaged children, the availability of public library reading materials and the support of summer reading programs may help combat the longterm educational consequences (lower achievement in all areas) that result from cumulative effects of summer learning loss. the value of public libraries summer reading programs has been well documented (allington, et.al, 2010; gordon, 2008; gordon & lu, 2008) and this study supports the view that partnerships between schools and public libraries can provide significant support for children from all socioeconomic groups in establishing and maintaining consistent progress in learning to read. the final section, the call to action, outlines specific strategies that can be implemented in daily practice to capitalize on the findings of this study. references allington, r. l., mcgill-frazen a., camilli g., williams l., graff j., zeig j., zmach c., & nowak r. (2010). addressing summer reading setback among economically disadvantaged elementary students. reading psychology, 31(5), 411-427. alexander, k., entwistle d., & olsen l. (2007). lasting consequences of the summer learning gap. american sociological review, 72, 167-180. gordon, c. (2008). a never-ending story: action research meets summer reading. knowledge quest, 37(2) 34-41. gordon, c., & lu, y.l. (2008) i hate to read, or do i? low achievers and their reading. school library media research, 11. retrieved 13 feb. 2012 from http://www.ala.org/ala/mgrps/divs/a asl/aaslpubsandjournals/slmrb/slmrc ontents/ volume11/gordon_lu.cfm heyns, b. (1978). summer learning and the effects of schooling. new york: academic press. krashen, s. (2004) the power of reading: insights from the research. portsmouth, nh: heinemann. / evidence based library and information practice microsoft word news5_1474_final_final evidence based library and information practice 2008, 3:1  88 evidence based library and information practice       news     11th european conference of medical and health libraries      © 2008. this is an open access article distributed under the terms of the creative commons attribution  license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and  reproduction in any medium, provided the original work is properly cited.       welcome to the 11th european conference  of medical and health libraries.  towards a new information space:  innovations and renovations,   helsinki, finland, 23rd ‐ 28th june 2008.    the finnish medical library association,  bibliothecarii medicinae fenniae (bmf), in  collaboration with the national library of  health sciences, has the great honour and  joy of inviting you to the 11th eahil  (european association for health  information and libraries) conference.    the eahil 2008 helsinki conference offers  a high quality scientific program. the  proceedings will cover many interesting and  current themes, especially the following  topics inspired the authors: virtual  communities and virtual libraries, evidence‐ based practice, education and professional  development and new technologies and  applications. in addition, a number of  inspirational continuing education courses  will be offered.    registration deadline for early birds:  march  31, 2008 (380 €)  registration: april 1, 2008 onwards  (430 €)  accompanying person: 230 €  continuing education courses: 60 € ‐ 80 €    registration:  http://www.congreszon.fi/eahil_2008/registr ation/    please visit the eahil 2008 helsinki home  page http://www.congreszon.fi/eahil_2008/  for information.     for current discussions about the  conference, please visit the eahil 2008  helsinki blog at   http://eahil2008.blogspot.com/.         evidence based library and information practice evidence based library and information practice 2011, 6.2 90 evidence based library and information practice news/announcements call for papers: special issue of library and information research 2011 library and information research. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/bync-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. the journal library and information research (lir) is seeking papers for a special issue on research methodology in library and information studies (lis), to be published january/february 2012. all papers will be peer-reviewed and submission details are given below. final date for submission of papers is friday 14 october. please address any queries and expressions of interest to the guest editor for this issue, judith broadypreston, department of information studies, aberystwyth university (jbp@aber.ac.uk) this special issue is an opportunity to share contemporary ideas in research methodologies, models and methods originating from, or of relevance to any branch of the lis discipline, sub-discipline and wider information profession. theoretical papers relating to research methodologies, models and methods are especially welcomed, as are philosophical papers, and papers presenting a fully developed process ready for wider use. in all cases concepts and underlying principles should be emphasised, with enough background information to orient any reader who is not a specialist in the particular subject area. the work should not have been published elsewhere nor submitted to another journal or conference. topics may include, but are not limited to: action research case study research method (single and multiple) computer modelling and simulation critical theory research delphi method design-oriented research evaluative and performance research, including conjoint analysis evidence-based and reflective professional practice research grounded theory positivist paradigm relativist approaches to qualitative research. reflexivity and subjectivity research design and triangulation research ethics and protocols theory development information for authors to find out about writing for library and information research please see the 'information for authors', available from the right hand menu bar on the journal home page: http://www.cilipjournals.org.uk/lir. mailto:jbp@aber.ac.uk� http://www.cilipjournals.org.uk/lir� evidence based library and information practice 2011, 6.2 91 submitting a paper to library and information research to submit a paper to library and information research you will first need to register with the journal using the ‘register’ link at the top of the journal’s home page. please be sure to register both as a reader and an author (there are check boxes for this on the registration page). once you have registered you may log in to the journal. you should navigate to the author guidelines (via ‘information for authors as above) and download the template for articles (from the link under ‘manuscript preparation’). when you are ready to submit you should go to your user home screen and select 'author'. this will offer you the option to 'click here to start the submission process'. you will then be taken step by step through the submission process. if you have any difficulty with submitting your work please contact the library and information research editor at lirg.lirteam@gmail.com. mailto:lirg.lirteam@gmail.com� / evidence based library and information practice ebl 101   evidence based librarianship backgrounder   su cleyle associate university librarian memorial university of newfoundland st. john’s, newfoundland, canada e-mail: scleyle@mun.ca   julie mckenna deputy library director regina public library regina, saskatchewan, canada email: jmckenna@reginalibrary.ca    originally published in: evidence based library and information practice, 3(3), 91–93. https://ejournals.library.ualberta.ca/index.php/eblip/article/view/4061/3329     received: 13 april 2008 accepted: 17 june 2008      2016 cleyle and mckenna. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   evidence based librarianship (ebl) is a means to improve the profession of librarianship by asking questions, finding, critically appraising and incorporating research evidence from library science (and other disciplines) into daily practice. it also involves encouraging librarians to conduct research (koufogiannakis and crumley, 112).    welcome to ebl 101! this new column is designed to offer guidance into the workings of evidence based practice and answer that question: “how can i implement ebl in my library?”  the intent is to offer short, simple columns on a variety of ebl topics allowing any librarian, regardless of library type or size, to practice evidence based librarianship. so let’s get to it, shall we? evidence based practice (ebp) is a term that we have all heard. usually it is associated with the health professions and originated the area of clinical medicine. the medical profession forged the way for many professions to embrace evidence based practice (ebm.) back in the 1990’s, canadian doctors sought to create an environment of lifelong learning and clinical practice that utilitized research to answer clinical questions.   there are 5 steps to evidence based medicine:   formulate an answerable question. track down the best evidence critically appraise the evidence (i.e. find out how good it is). apply the evidence (integrate the results with clinical expertise and patient values). evaluate the effectiveness and efficiency of the process (to improve next time) (glasziou 23).   as you can see the concept of ebm is a simple one – gather evidence to help with answering questions and helping patients.  for the discipline of medicine, the body of evidence is rich in these areas and it is conceivable that most questions can be answered through use of existing research evidence.  but even though the steps seem simple enough, a variety of skills are needed to ensure each step is completed properly. for example, asking the right question is crucial to finding the best evidence; evaluating the evidence is pivotal to determining the best course of action, and so on.   evidence based librarianship (ebl)   evidence based practice is a practical approach to finding answers to questions and for professionals to stay abreast of current trends and research. it is also a useful model for contributing to the body of evidence.  but will this model work in a discipline that is not grounded in the research practices associated with the collection and use of empirical data?  the social sciences research base is very different from that in the sciences.  it is possible to apply the evidence based practice model to social science disciplines, including librarianship?    ebp, quite simply, can encompass original research and the evaluation and use of existing research.  koufogiannakis, crumley, and slater reviewed several content analysis reports and note the “the variation in the interpretation of what constitutes a ‘research’ article…” (koufogiannakis, crumley, and slater 228). of the 2664 articles reviewed from the 2001 publishing year, 30.3% were identified as research articles. this is a rate similar to previous content reviews for librarianship, although there are variations in scope and definition of what “constitutes a ‘research’ article”. clearly, the higher percentage of articles appearing in our professional literature is not research oriented.   librarianship is not primarily comprised of scholars or researchers. it is comprised of practitioners and administrators. thus, research has not necessarily made its way into our professional literature and our decision making processes. many of us in the profession now recognize the need to formalize our research and our decision making processes to ensure that we base our decisions on the best possible evidence.    based on the ebm model, the steps for ebl are similar:   define problem find evidence appraise evidence apply results of appraisal evaluate change redefine problem (booth and brice, 2003)   the definition of the problem, or “the question”, is pivotal to the entire process. the next ebl 101 column will focus on asking the right question.   works cited   booth, andrew and anne brice: “evidence based librarianship: the first steps.” 2nd international evidence based librarianship conference, edmonton, alberta, canada, june 2003. april, 2005 <http://www.shef.ac.uk/scharr/eblib/conf2003.htm>.   glasziou, paul. evidence based medicine workbook. london, gbr: bmj publishing group, 2003. august, 2008 <http://site.ebrary.com/lib/memorial/doc?id=10049688&ppg=29>.   koufogiannakis, denise, and ellen crumley. “evidence based librarianship” feliciter, 3 (2002): 112-4.    koufogiannakis, denise, ellen crumley, and linda slater. “a content analysis of librarianship research.” journal of information science 30.3 (2004): 227-39.   evidence summary   multiple databases are needed to search the journal literature on computer science   a review of: cavacini, a. (2015). what is the best database for computer science journal articles? scientometrics 102(3): 2059-2071. http://dx.doi.org/10.1007/s11192-014-1506-1   reviewed by: giovanna badia liaison librarian schulich library of science & engineering mcgill university montreal, quebec, canada email: giovanna.badia@mcgill.ca   received: 18 sep. 2015   accepted: 04 nov. 2015      2015 badia. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   objective – to compare the coverage of computer science literature in four bibliographic databases by checking the indexing of a selection of journal articles. the purpose of this comparison was to identify the most comprehensive database in computer science and determine whether more than one database is needed to search for articles on computer science topics.    design – comparative database evaluation using citation analysis.   setting – computer science journal literature found within the inspec, scopus, web of science, and dblp databases.   subjects – 1,135 computer science journal articles published by an italian university’s researchers from 1979 to 2014.   methods – the university of milan’s institutional repository (air), containing publications authored by the university’s researchers, was searched in october 2014 for journal articles that were assigned the subject heading “informatica” (the word for computer science in italian). the author then searched the titles of these journal articles in each of the databases to check whether they were indexed. for articles indexed in all four databases, the author also examined the quality of the bibliographic records by looking for the presence of 20 elements (e.g., the “cited by” option, ranking of search results, precision of results, etc.) in each database’s record. these overlapping articles were also searched in google scholar to help compare the quality of the records between the databases.     main results – scopus indexed 75.86% of the journal articles found in air, web of science indexed 64.49%, dblp indexed 61.15%, and inspec indexed 53.39%. web of science and inspec put together covered 74.80% of the articles, which is comparable to the amount indexed by scopus. dblp and scopus contained the highest number of references to articles that were not found in the other databases, about 4% each. out of the 1,135 journal articles, 391 (34.45%) were indexed by all four databases, with web of science scoring the highest for providing the best quality bibliographic records for these articles.   conclusions – according to the author, the findings showed that inspec, scopus, web of science, and dblp “complemented each other, in a way that neither one could replace the other” (p. 2068) when searching the computer science literature. while there was overlap between databases, they each also contained unique articles.    commentary   based on the author’s literature review, there are many published studies that have compared two or more of the same databases being contrasted in this study, and examined one or more databases that index the computer science literature. however, it seems that none, thus far, have compared inspec, scopus, web of science, and dblp for their coverage of the computer science literature, or used publications from an institutional repository as the source of data for making database comparisons.   the ebl critical appraisal checklist (glynn, 2006) was employed to help objectively determine the strengths and limitations of this study. its strengths lie with the data collection and study design. the author clearly outlines the procedure he utilized to conduct the research and describes the results concisely. readers would be able to easily replicate the methodology.    the major limitation of this study concerns the study population, in this case the source of data used to compare the databases. as noted by the author, only journal article titles that were assigned the computer science subject heading were extracted from the university of milan’s institutional repository and used to compare the four databases. these journal articles accounted for only 29.64% of the total number of documents (1,135 out of 3,828 documents) in the repository about computer science. the author states that “in the computer science field, proceedings are usually a prime avenue for publications, and selected conference proceedings are as prestigious as journal articles” (p. 2069). conference papers account for more of the published literature in computer science than journal articles. therefore, the sample used as the data source in this study is not representative of the computer science literature, which means the results cannot be generalized to searching the entire corpus of published literature in computer science.   additionally, another limitation pointed out by the author is that databases continuously update their list of indexed journals so that “identical searches might give different results if repeated over time” (p. 2069). the results in the study are a snapshot in time, requiring that the study be repeated to confirm the findings.    the author suggests that a future direction for research would involve searching the databases under investigation for the conference papers found in the institutional repository. this reviewer thinks that the future direction should have been done in this study in order to validate the findings. the author could have randomly selected documents from the repository to obtain a more representative sample or searched the titles of all 3,828 documents in the different databases.   despite its limitations, this study will be of interest to librarians seeking to compare databases in a specific discipline for teaching, reference, or collection development purposes. the study successfully demonstrates that the source of data for a comparative database evaluation can also be taken from an institutional repository that provides references to all the scholarly output of its researchers. readers should take a representative sample of documents from the institutional repository to ensure the validity of their results.   reference   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692145   microsoft word art_1330_toner_final evidence based library and information practice 2008, 3:2 18 evidence based library and information practice article non-use of library services by students in a uk academic library lisa toner site library manager, harold bridges library st. martin’s college, lancaster,uk email: lisa.toner@cumbria.ac.uk received: 26 february 2008 accepted: 08 may 2008 © 2008 toner. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – this study examined low or non-use of the library at st. martin’s college, uk and determined the possible reasons for this. additionally, this study investigated the other sources of information non-users were accessing. the results were then fed into the library’s future strategic planning cycle. methods – using data from the talis management system, a postal survey was distributed to students who registered as low or non-users. non-users were considered a hard-to-reach target group, and a further classroom-based survey was administered. results – the findings showed that students in part-time and distance learning courses were the largest group of non-users. other factors influencing non-use included the purchase of books, use of the internet, and a lack of awareness of services available. conclusions – the library service needs to target non-traditional student groups, particularly those taught off-campus, with innovative induction techniques. this survey highlighted the need to embed information skills teaching into the curriculum and a recent restructure of the department has enabled this to happen with a more focused approach to academic liaisonship. evidence based library and information practice 2008, 3:2 19 introduction and background higher education is changing rapidly. like much of the world, the uk system is becoming increasingly open and diverse with the underlying strategic aim of being available to all. this is evidenced by the uk government’s widening participation target of having 50% of all 18-30 year olds in higher education by 2010 (dfes 63). people from disadvantaged backgrounds are being encouraged to consider entry, and this increased emphasis on students from nontraditional backgrounds has caused student retention to surface as a major strategic agenda item. as a consequence, uk institutions in the 21st century are becoming more responsive to the needs of their customers than ever before. advances in technology have made possible virtual classrooms, online courses, and distance learning. this, coupled with the growth in society’s access to information via ict, has altered student perceptions of what the library has to offer. if libraries are to maintain their relevance in the cycle of student needs, then they must adapt and change. listening to the needs of those who choose not to use our services is one way of being responsive to that change. philip payne, head of learning support services at leeds metropolitan university, argues that, “there is an increasing focus in academic libraries on users and in meeting their needs. measuring the impact of our services potentially forms part of having a customer focus” (everest and payne 18). the corporate plan of st. martin’s college outlines in its vision for the future that st. martin’s will, “improve its ability to listen to and respond to students, their needs and aspirations” (st. martin’s college 6). in line with that sentiment, the library service strategic plan states in aim 1.2 that the service desires to “critically evaluate library services to assess impact and introduce service enhancements” (library services 23). a lack of knowledge regarding what potential users want makes it difficult for the library to provide suitable services for them. as peter brophy, director of cerlim at manchester metropolitan university suggested, “users do not use services in the way we think they do, partly because librarians are inconsistent in our promotion of services” (everest and payne 19). library staff are extremely good at surveying the needs of their users. various methodologies have been in use throughout the sector, most commonly the exit survey or questionnaire. they are used primarily to establish student satisfaction levels and to determine improvements they would like to see to services and facilities. however, comprehensive user survey methodologies which produce statistically valid findings are expensive and timely to produce. this is evidenced by the work done at the university of california san diego libraries, which employed a consultant to assist with the construction of a large scale project (talbot 358). within the field of academic librarianship in the uk, the methodology advocated by the society of college, national and university libraries (sconul) is commonly used and has been employed at st. martin’s college library for the previous three years. libraries are less proficient at canvassing the opinions of those who do not use their services, although some attempts have been made within the public sector as part of general community sampling or as surveys of lapsed users. in an academic setting however, it is taken for granted that the students will use the library of their institution. higher education institutions have historically functioned largely as closed communities with only students registered on recognised courses allowed evidence based library and information practice 2008, 3:2 20 borrowing access to materials; as such, the library stock and services are directly aimed at specific course needs. in practice, however, a significant minority do not borrow books from the library. the perception on the ground is that this situation is increasing. the purpose of this research was to discover if this is the case at st. martin’s college and what the possible reasons would be for low or non-use. the results could then be fed into the library’s future strategic planning cycle. institutional context st. martin’s college is a higher education institution based in the north west of england primarily lancaster, carlisle and ambleside with satellite sites in barrow, whitehaven, tower hamlets and greenwich (the latter two in the south of england). st. martin’s has 11,274 students (6,839 fte) of those 5,918 are on part time courses and of those 181 are taught on a designated distance learning course. the college is the largest national provider of initial teacher training and education. a major part of the college’s work is specialist provision in relation to the allied health professions. a significant proportion of students study on degrees in the arts, humanities and social sciences with a particular emphasis on sports and outdoor studies. st. martin’s library aims to provide a common service framework enabling students to receive the same quality of service regardless of location. the library provides fully staffed library services on the three principal sites of lancaster, ambleside and carlisle. library services are provided jointly at the satellite sites by a range of agreements with the health trusts, tower hamlets schools library service, and woolwich boys polytechnic technical school. literature review a range of databases were searched to identify relevant literature. this included lisa, assia, infotrac, web of science, and electronic journal service. although some work has been done on non-use of library services, non-user surveys in academic libraries proved to be scarce. sridhar has examined the theoretical concepts behind the terms “non-user” and “marginal user,” and he includes a case study from the isro satellite centre in bangalore which concludes that it is important to identify the characteristics and needs of non-users if libraries are to increase their market penetration (sridhar 21). he acknowledges that opinion is divided in the profession as to the relevance of non-user data; as such, surveys can be costly and complex. they also often result merely in enhancing the service for existing users. he goes on, however, to argue that the absolute non-users may be stimulated into using the library if it can modify its services to suit their requirements (14). early work in the field of academic library non-users includes that of lubans. although his work predates the mass electronic information era, he does identify that, “non use to any degree is a serious problem” (362). lubans’ study centers around the rensselaer polytechnic institute library in new york and concludes that patterns of non use are more prevalent amongst undergraduates than other client groups of the library. martell has examined downward trends in circulation figures amongst association of research libraries members in the usa over a nine-year period. the results show a comparable increase in non-user statistics which he attributes to an increase in the use of virtual technologies being the preferred method of accessing information (438, 443). evidence based library and information practice 2008, 3:2 21 new technologies themselves are not the panacea of all information needs. the university of abertay, dundee, discovered this in its survey in the spring of 2004 which was aimed at student use of electronic resources only. this research found that, “the main reasons for infrequent or non-use of resources is lack of awareness and perceived lack of relevance. lack of time is also a barrier” (millar 68). non-user surveys have been conducted in coventry, leeds, and warrington in the uk public library sector (strategic planning and marketing, market research, molyneux). these were conducted generally through the post, as a questionnaire or as street surveys using a professional market research company. an article on a lapsed borrower survey in essex, ”where have all the issues gone?” found that changes in ict usage and the internet were the prime causes of non use of public libraries, along with conflicting demands on leisure time from a competitive society. the most cited cause was “lack of time” to read (hawkins 10). further examples of non-user surveys were discovered. louise flowers from upper goulburn, australia comments that, “as yet the professional literature does not acknowledge the topic of non-users in its own right, probably because there is only a small body of research that specifically addresses this issue” (flowers 67). this is confirmed by sridhar, who notes the difficulty in compiling non-user surveys and suggests this has led to an imbalance in the research literature (sridhar 2). the literature highlights limited research into reasons for non-use of academic libraries that could be used for strategic planning. therefore, this project aimed to investigate the level of non-use of traditional library services at st. martin’s college and to ascertain the reasons why some students were not engaging with the printed material available to them. to this end several objectives were put forward: 1. to discover what sources were being used to satisfy the students’ information needs given that they were not borrowing books 2. to ascertain whether age, gender, or mode of study had any bearing on non-use 3. to establish whether opening hours were convenient for this user group 4. to calculate how many of the target group had access to a pc and how much use was made of the internet and other electronic resources 5. to determine what level of training these students had received to access resources electronically 6. to discover what effect changing patterns of teaching and learning have had on the use of traditional library services 7. to establish whether the target group received a library induction in any form and how this was perceived by those who had received it 8. to establish the main reasons why these students made little or no use of traditional library services 9. to discover how this group of students perceived the library service examining these research objectives would enable the library to draw conclusions about how it could improve and encourage this group of students to make more use of the services available. subsequently, service improvements via the library service strategic plan that would encourage higher levels of use or realignments of service provisions in accordance with the needs of this group could be made. evidence based library and information practice 2008, 3:2 22 methodology the research project was conducted over a three-month period between march and may 2005. a pilot questionnaire was trialled with two groups of students to ascertain if the wording was appropriate. following amendment, 2,189 questionnaires (see appendix) were posted to students who registered as low or non-users according to talis, the library database. low use was defined following a discussion amongst the management team as students who had borrowed three items or less in the previous twelve-month period. this amounted to 21% of total student numbers. a further 800 questionnaires were distributed to classrooms. lectures were identified on a specific day via the timetabling office and tutors were asked to distribute and collect the survey forms at the start of their session. although this created the possibility of duplication with the postal survey, students were asked not to complete the survey twice. there is, of course, a chance that some students may have ignored this request resulting in duplicate responses. the results were collated using the spss statistical software package where the data was analysed against a number of variables such as cohort groups or location to provide a comprehensive report. findings five hundred eighty-six forms were returned by post equating to a response rate of 26.5%. three hundred eighty-two surveys distributed via classrooms were returned; however only those who indicated that they rarely or never used the library (n=24 or 6.28%) were analysed as part of the research. overall, the project returned a 27.3% response rate. half these responses were found to be from satellite sites of the college, and it was subsequently discovered that, due to differences in the library systems at the different sites, many of these users were local borrowers at the satellite site rather than non-library users. these were also removed from the sample, reducing the non-user response rate of traditional library services to just under 14%. the results presented below are therefore based on actual non-users (14% of students at st. martins college). non-users the faculty of health and social care produced the largest number of non-users (n=213), of those over half of them do use other university or nhs libraries. faculty frequency percent not known 52 12.8 health 213 52.6 education 76 18.8 arts, humanities and social science 64 15.8 total 405 100.0 table 1. non-users by faculty evidence based library and information practice 2008, 3:2 23 two programmes in particular, nurse practitioners and preparing mentors for professional learning and development, showed the highest levels of non-use. the mentoring program is taught almost exclusively off-campus in remote locations throughout the region. these areas have been targeted for further work by library and academic staff. demographics the data confirmed that 318 (78%) of the non-users in the college were female. this is in line with the college’s intake ratio of 4:1 female to male. three hundred forty five respondents or 85% were over the age of 24. it would appear from these results that the incidence of non-use of library services does increase with age. however, these results could also indicate that older people are more diligent in returning their survey forms. mode of study the majority of respondents (n=298, 73%) indicated that they were on a part time course. a further analysis of the age ranges based on those who are in part-time courses virtually eliminates those in the 18-24 age range. those over 40 are the highest nonusers, as they are overall (192 or 47.4%). thus, mode of study and age do seem to be predictors of use of the library services. year group students in the first year of their study make up the largest group of non-users. these may be students who have not yet been assigned work that requires use of library resources or students who have not yet found their way into the library. however, given the timing of the survey (after two full terms), it is reasonable to expect them to be using the library facilities. those on diploma courses (short modules lasting less than one 18 24 25 40 over 40 age range 0 50 100 150 200 c o u n t figure 1. age of respondents study mode frequency percent not answered 19 4.7 full time 88 21.7 part time 298 73.6 total 405 100.0 table 2. mode of study 18 24 25 40 over 40 age range 0 50 100 150 200 c o u n t figure 2. mode of study by age evidence based library and information practice 2008, 3:2 24 year) also made up a large group of nonusers. information sources used the results highlighted that non-users not only made no use of traditional library services but only made limited use of the electronic resources provided by the library. just under a third of non-users (n=120) did not make any use of the college’s electronic resources and fewer than 10% (n=39) accessed the e-books collection. in contrast to this, the majority (n=357, 88%) use the internet as a source of information for their assignments. induction to library service inductions are organised by academic staff, particularly for courses taught off campus. induction was analysed in terms of take up with tours being seen as more useful than powerpoint presentations. a third of all respondents (n=120) reported that they had not been offered any form of induction. these were mainly students taught from remote locations such as church halls or school based situations where the college is working with the community. the library must creatively develop more alternative induction packages for those not taught on a main campus. information sources used to complete assignments 0 50 100 150 200 250 300 350 400 in te rn et s m c e jo ur na ls s m c d at ab as es e bo ok s li br ar y bo ok s r ef b oo ks b uy o w n co pi es le nt b y fri en ds p ho to co pi es source s n o o f re s p o n d e n ts figure 3. information services used by library non-users no answer tour power point presentation opted out none offered induction 0 50 100 150 c o u n t figure 4. methods of induction evidence based library and information practice 2008, 3:2 25 access to it use of pcs was predominantly in the home, with 74% (n=298) using them there. substantial numbers of students used work or library pcs to access it facilities. however, the following other locations were also identified: reasons for non-use the majority of respondents cited distance from the main campus (n=275, 68%) as the main reason for non-use of the library (an option not anticipated on the questionnaire and thus not shown in figure 5). almost a third (n=130, 32%) respondents used other university or nhs libraries instead of st. martin’s library. this is perhaps due to the successful introduction of the uk libraries plus scheme that allows reciprocal access at local academic libraries for students in part-time and distance learning programs. also, many of the students in health-related courses are able to use the libraries at the hospital where they undertake placement. other reasons cited were no time, (n=63,16%), inconvenient opening hours (n=26, 6.4%), difficulties finding items on shelves (n=23, 5.6%), lack of help from staff (n=15, 3.7%), no need 15 (3.7%), uninviting (n=11, 2.9%) and finally 1% (n=5) do not use the library because they do not like libraries at all. discussion given that the target group for the survey was students who were perceived to be not engaging with the library, a response rate of 26.5% was deemed to be above that expected. this meant that, although numbers at individual sites were low, the overall responses were indicative of the group and findings could be extrapolated to form trends that had the potential to improve the whole service to make it more attractive to non-users. the timing of the project was determined by institutional funding arrangements and had to be dovetailed with other project work within the library. as a consequence, the post-graduates and 3rd year cohorts of students were not present for the classroombased questionnaires. they did have the opportunity to respond via the postal survey so were not excluded from the research. changing patterns of learning have affected library use, with 168 or 41% of respondents reported to be on a course supported by open and distance learning. this correlates with the finding that distance from the library was the single biggest factor, cited by 275 respondents (68%) of non-use. respondents did, however, display some confusion over the term “distance learning,” with some stating that they were in a distance learning course even though the college does not provide their course on such a basis. students often assumed that, because they lived at a distance from the college, that they were in a distance learning course. the college as a whole perhaps needs to examine its use of frequency percent pc at home 298 73.6 pc at work 106 26.2 pc in library 76 18.8 friend's 11 2.7 hospital site 6 1.5 public library 4 1.0 learn direct 1 .2 wireless laptop 1 .2 other hei 1 .2 in the gym 1 .2 parents 1 .2 table 3. access to it evidence based library and information practice 2008, 3:2 26 terminology and ensure that it is clear and understandable by all. some distance learning courses make heavy use of blackboard, the institution’s virtual learning environment where tutors may post reading materials in digitized form. a new digitization service has been established within the library service but at the time of the survey this was not widespread; therefore, material provided in digitized formats through the vle is not likely to account for the lack of use of library materials. the responses indicate that many students are not aware of all the library services offered. this highlights the need to do much more in terms of marketing, publicity, and promotion. this was particularly apparent in the area of electronic resources. students reported not knowing about the electronic services available. many reported that they either did not know about the resources or that they did not know how to use them. several students cited lack of confidence as a factor in non-use of electronic journals, databases and e-books, and many requested further training aimed in the form of drop in sessions and printed literature posted out to their home addresses. it is clear that some students completely misunderstood the nature of electronic resources. for example, 27 students (3%) answered that they did not use them because they thought that they were “too far to travel,” whilst a further 18 (2%) thought that athens was a database rather than a password authentication system. more training was requested by students, particularly in ict skills for mature students and those who struggle with computers. an interesting factor was that 32 respondents said that they bought their own copies of books and an additional 44 borrowed from their friends. this could be explained by the fact that many are mature reasons for non use 0 20 40 60 80 100 120 140 o pe ni ng h ou rs u se o th e r lib s n o tim e d on 't ge t he lp n o ne ed u ni nv iti n g b uy o w n b oo ks d on 't lik e li b s d iff to fi n d bo ok s reason n o o f re s p o n d e n ts figure 5. reasons for non-use evidence based library and information practice 2008, 3:2 27 students who perhaps have more demands on their time in terms of work and family commitments. as one student said, “i buy all my essay books from amazon and have them delivered to my house.” this demonstrates a significant shift away from the concept of the library being the provider of all the research and information needs of students, and it is noted that the private or second-hand market is becoming increasingly competitive in this area. the fact that reciprocal access schemes are so popular again diminishes the monopoly once held by individual academic institutions. organisation and support of students in flexible and distributed learning courses and part-time students who live at a distance are important considerations for the library. a postal loan service is provided, but these students did not appear to be using it or were unaware of its existence. this was demonstrated by a comment from one student who asked if it would be possible to set up a postal service. again, this indicates areas where the library needs to market its facilities and services more explicitly. actual distances from the library did not in itself appear to play a part in non-use, as 26% (105) of respondents lived within 10 miles of a main campus library. the acquisition of 12,000 new e-book titles as a result of an agreement with net library has the potential to support students at a distance, providing the issues relating to electronic resources noted above can be addressed. students who do not borrow books are on the whole not using other library services either. two hundred ninety-eight respondents access the computer from home (73%) or 106 from work (26%). those who do visit the library building do so mainly for photocopying services, 101 (25%) or private study. other factors in non-use of st. martin’s library services was use of other university and nhs libraries by 33% of respondents, which is indicative of the success of uk libraries plus and other reciprocal access schemes in meeting the needs of part time and distance learners. in contrast to the surveys of non-users in public libraries (molyneux 20, strategic planning 14, market research uk 11) ‘lack of time’ was only a factor in n= 61 or 15% of cases. st. martin’s does not open its libraries 24/7, and opening hours, which were expected to be a prime reason for non-use, were cited by a small number (n= 26, 6.5%). when asked to consider improvements to the service to make it more attractive for them to use, a range of comments were made. students inevitably requested more stock, in particular books and journals. they asked for improved inductions and training from the library, and suggested “an idiot’s step-by-step guide to take away and use from home.” some asked for easier online access, and this was particularly in relation to password difficulties via athens authentication protocols for the databases. the college is currently working towards the establishment of a single sign-on for the network. comments about staffing, like “have more people on the desk – there is always a big queue,” tended to acknowledge that staff were extremely busy and often have little time. there were also several positive comments received, such as “whenever i use the library the service has been excellent.” the fact that few responses came from the main body of undergraduate or taught postgraduate campus-based courses suggests that the work which the library is currently doing with these groups is working well and that they are being taught library skills and subsequently using the service. the main groups of students who do not use traditional library services are evidence based library and information practice 2008, 3:2 28 those who may feel on the fringes of the college due to their part time or distance mode of learning. even then, one third (n=133) of those are using library services near to their home or place of work. the survey was successful in reaching a proportion of those students who were not using the library service; however, further data could have been usefully gathered from this group by follow-up telephone interviews with a sample of respondents. time and funding were factors in preventing this further drilling down into the data. conclusion this survey of a uk academic library highlighted that the main reasons for nonuse of the library services were students studying on part time or distance learning course who make heavy use of other libraries or buy their own texts. these students use the internet to access information but are not using the electronic resources provided by the library. the results of the survey have shown that the library needs to become much more creative in its marketing, publicity, and promotion to non-user groups, particularly part-time and distance learners. as a consequence, the faculty liaison teams have been strengthened within a new structure and library staff now work more closely with course and program leaders to ensure that all students receive meaningful induction and subsequent training on databases, information searching, and critical thinking skills. an increasing number of students are taught not to rely totally on the internet for their information needs. work has been done to embed this training into the curriculum wherever possible and library staff are increasingly included in the training of students at remote locations. further research is needed to analyse the effectiveness of our actions in 2-3 years time when it is hoped that much of the impact of increased training and promotion will have had a positive effect on usage figures. works cited department of education and skills (dfes). the future of higher education. norwich, the stationery office ltd, 2003. 23 may 2008 <http://www.dfes.gov.uk/hegatewa y/strategy/hestrategy/pdfs/dfeshighereducation.pdf>. everest, katherine, and philip payne. "the impact of libraries on learning, teaching and research; report of the lirg seminar held in leeds, 30th october 2001." library and information research news winter (2001): 18-21. flowers, louise. "non-users of the upper goldburn library service." the australian library journal may (1995): 67-85. hawkins, margaret. "where have all the issues gone? lapsed borrower survey in essex libraries." public library journal 6.1 (1991): 10-2. lubans, john. "nonuse of an academic library." college and research libraries 32.5 (1971): 362-7. market research uk limited. leeds city council libraries survey draft report., 2001. martell, charles. "the elusive user: changing use patterns in academic libraries 1995-2004." college and research libraries 68.5 (2007): 43544. evidence based library and information practice 2008, 3:2 29 molyneux, wendy. lymm library lapsed user survey. warrington borough council, 2002. millar, s. “news from member libraries”. sconul focus 32 summer/autumn (2004): 68. society of college, national and university libraries (sconul)"sconul survey templates." 31 may 2008 <http://www.sconul.ac.uk/groups/p erformance_improvement/surveys/t emplates.html>. sridhar, m. s. "non-use and non-users of libraries." library science with a slant to documentation and information studies 31.3 (1994): 11528. st. martin's college. corporate plan 2004-9., 2003. ---. library services strategic plan 2005 2009., 2004. strategic planning and marketing cultural development. non-user survey september october 2000. coventry libraries, 2000. talbot, dawn. "from the users' perspective: the ucsd libraries user survey project." journal of academic librarianship 24.5 (1998): 357-64. evidence based library and information practice 2011, 6.3 1 evidence based library and information practice evidence summary millennial students’ online search strategies are associated with their mental models of search a review of: holman, l. (2011). millennial students’ mental models of search: implications for academic librarians and database developers. journal of academic librarianship, 37(1), 19-27. doi:10.1016/j.acalib.2010.10.003 reviewed by: leslie bussert research & instruction librarian university of washington bothell, cascadia community college campus library bothell, washington, united states of america email: lbussert@uwb.edu received: 01 june 2011 accepted: 02 aug. 2011 © 2011 bussert. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to examine first-year college students’ information seeking behaviours and determine whether their mental models of the search process influence their ability to effectively search for and find scholarly materials. design – mixed methods including contextual inquiry, concept mapping, observation, and interviews. setting – university of baltimore, a public institution in maryland, united states of america, offering undergraduate, graduate, and professional degrees. subjects – a total of 21 first-year undergraduate students, ages 16 to 19 years, undertaking research assignments for which they chose to use online resources. methods – first-year students were recruited in the fall of 2008 and met with the researcher in a university usability lab for about one hour over a three week period. the researcher observed mailto:lbussert@uwb.edu� evidence based library and information practice 2011, 6.3 78 and videotaped the students as they conducted research in their chosen search engines or article databases. the searches were captured using software, and students were encouraged to think aloud about their research process, search strategies, and anticipated search results. observation sessions concluded with a 10question interview incorporating a review of the keywords the student used, the student’s reflection on the success of his or her searches, and possible alternate keywords. the interview also offered prompts to help the researcher learn about students’ conceptualizations of search tools’ utilization of keywords to generate results. the researcher then asked the students to provide a visual diagram of the relationship between their search terms and the items retrieved in the search tool. data were analyzed by identifying the 21 different search tools used by the students and categorizing all 210 searches and student diagrams for further analysis. a scheme similar to guinee, eagleton, and hall’s (2003) characterized the student searches into four categories: simple single-term searches, topic plus focus searches, phrase searches, and advanced searches employing multiple boolean operators. students’ diagrams were put into three different groups: process view, hierarchical view, and network view. the researcher then analyzed the relationships between the students’ search behaviours and their mental models to develop further conclusions. main results – analysis revealed that this population of students had a limited mental model of the search process and used narrow sets of fairly simple search strategies for retrieving information online. search engines were used for the majority (61.9%) of total searches and 72.3% of those conducted in search engines were in google. the majority of students (76%) began their search process with a search engine while other students began searching in online encyclopedias (10%) or online databases (14%). academic search premiere was used for 73.8% of the database searches. some students (5%) also performed searches in individual websites (6.3%), for an overall total of 224 searches conducted. students performed four varieties of searches: simple searches using short phrases conveying a single concept (34% of total searches); topic plus focus searches using a single boolean and (30%); phrase searches consisting of multipleword descriptive phrases or sentence fragments (17.4%); and advanced boolean searches combining two or more distinct concepts (13.8%). generally, students used the same search terms and structure whether they were in a search engine or database, particularly with phrase searches. nearly 71% of the advanced boolean searches were inappropriately formed, particularly when used in the databases. of the few students employing boolean logic beyond a single and, only two used it correctly, and only one with successful results. students were unable to recognize or explain why a search failed or why they got the results they did. they made frequent incorrect use of punctuation, spelling, and syntax, leading to limited or no search results. students assumed that obtaining few results indicated a problem of keyword choices rather than search query structure. when faced with no results in the databases, they assumed there were no articles on their topics and did not re-evaluate their search queries. those with unsuccessful boolean searches did not recognize that their errors were due to logic, and instead changed their keywords or began a new search altogether. several students understood keywords as concepts versus literal strings of letters, yet thought the search tools determined search results based solely on what was typed into the search box. of those employing phrase searches, some believed that each word was queried, while others thought only the “primary terms” were queried. most students (61%) offered evidence based library and information practice 2011, 6.3 79 analogies to print resources to explain how search engines process queries, and all the students’ descriptions included their ideas about what a search tool contained, rather than how the search tools organized information. attempts to expand or narrow searches were haphazard. while most students (57%) employed the strategy of adding keywords to narrow searches, only a few (11%) recognized the function of this technique and used it regularly, while others tended to return to their original broad searches in a different tool. some had a limited understanding that adding terms narrows and reducing terms broadens the search, but their boolean errors negated the use of synonyms or alternative terms for those purposes. other strategies included using the search tool’s “advanced search” features or quotes, although all who used the latter did so incorrectly and some mistakenly thought parentheses served the same purpose. all subjects drew representations of their views of the relationship between keywords used and search results retrieved, though few were able to clearly visualize how a search engine processes a query, or address ideas such as expanding or narrowing searches or synonym use. three categories of diagrams emerged: the process view, hierarchical view, and network view. the process view displayed a task flow diagram. these students demonstrated the least formed mental models and experienced the search tool as a “black box” that gives results, showing little understanding of how they are generated. they performed the fewest overall searches (11.6%), the majority (79%) of which were simple or phrase searches with no use of boolean operators. the hierarchical view displayed a broad subject with subtopics, or results highlighting specific aspects of the subject. these students performed nearly 30% of the total searches, 17% of which included the use of boolean logic. the network view displayed models of interconnected terms. these students performed the majority of the searches (58.7%), and also constructed the most sophisticated queries. many of their searches employed boolean logic (83%), and 65% were either boolean or topic plus focus searches. students with this mental model tended to focus more on the queries themselves than the results received. students indicated feelings of success in their searching and were comfortable relying on simple searches retrieving large results sets. while not central to the research design or driving questions, students’ evaluation of search results was observed and found to be weak. students displayed rapid searching, scanning, and evaluation processes which may have played a role in many of their mistakes when repeating or attempting to correct faulty searches. conclusion – the results show students did not have strong conceptual models of the search process or how search queries impacted results, and were often unable to recognize or troubleshoot problems with searches in order to improve results. students displaying stronger mental models used more complex search strategies, but still performed unsuccessful searches and demonstrated challenges in remedying defective searches. students skimmed search results quickly, rarely looking beyond the first two pages, and did not take time to evaluate them for topic relevance. the findings suggest that librarians should rethink how millennial students are taught search strategies and evaluation, to focus more on problem solving or critical thinking. they also suggest that database developers should continue developing search algorithms and tools, considering this population’s conceptualizations of search. further research on millennials’ information processing, critical thinking, and evaluation skills in the context of academic work is needed. commentary this study builds upon others investigating search conceptualization and behaviour in the evidence based library and information practice 2011, 6.3 80 fields of library and information science and computer science. the literature review offers a context for millennials’ information and technology behaviours and a foundation for understanding the role of mental models in information retrieval. the identified relationships between mental models and search skills can inform librarians’ approaches to teaching millennials how to search, as well as the design or modification of user interfaces and functionality of search tools. the mixed methods approach facilitated the collection of multiple data from which the researcher could draw conclusions. the small sample size was admitted, yet other limitations or variables remain unaddressed, such as: how subjects were recruited; whether they received library instruction in prior educational settings; whether subjects were researching for the same assignment, warranting the same types of sources; whether the interview data was formally coded for analysis; or the possibility that subjects modified typical search behaviour due to being observed in close proximity. some confusion is introduced through the organization of information presented in the article. findings on narrowing searches are discussed across multiple sections when it would be clearer if it was all contained within one section. the author’s definition of “search engine” is also hard to track. the majority of the article uses this term to describe search tools freely available online, such as google, and does not include library databases, yet in later sections the author appears to use it more broadly to incorporate both. this introduces a lack of clarity regarding the type of search tools around which the students’ mental models were constructed, and consequently a reader’s understanding of the findings may be confused. the provided recommendations for instruction librarians and database developers are valuable, yet not groundbreaking. instruction librarians have emphasized critical thinking since the late 1980’s, and database vendors have been developing search discovery tools better suited for simple searching since 1998 (bodi, 1988; webfeat, 2011). it would have been helpful if instead the author had shared suggestions for instruction librarians on ways to teach students about relationships between search queries and results, or discussed how these findings might apply to other user populations or search behaviour in discovery and federated search tools. the author’s findings can also be illuminated when placed within the broader study of human information seeking behaviours. in particular, mansourian and ford (2007a, 2007b) have identified theoretical frameworks to help understand web-based information seeking behaviours such as satisficing, persistence in searching, and decision making. they utilized simon’s (1955, 1956) bounded rationality theory, cooper’s (1976) utility theory, and kraft and lee’s (1979) three stopping rules. they also framed their work around prior research relevant to this study and the practice of instruction librarians. for example, ren (2000) identified a link between self-efficacy and search performance and found that search training can increase self-efficacy, while thompson, meriac, and cope (2002) identified links between selfefficacy and the number of items correctly retrieved, as well as increased searcher persistence when specific task instructions (versus general directions) were provided. references bodi, s. (1988). critical thinking and bibliographic instruction. journal of academic librarianship, 14(3), 150-153. cooper, w. s. (1976). the paradoxical role of unexamined documents in evaluation of retrieval effectiveness. information processing and management, 12(6), 367375. evidence based library and information practice 2011, 6.3 81 guinee, k., eagleton, m. b., & hall, t. e. (2003). adolescents’ internet search strategies: drawing upon familiar cognitive paradigms when accessing electronic information sources. journal of education computing research, 29(3), 363-374. kraft, d. h., & lee, t. (1979). stopping rules and their effect on expected search length. information processing and management, 15(1), 47-58. ren, w. h. (2000). library instruction and college student self-efficacy in electronic information searching. journal of academic librarianship, 26(5), 323-328. mansourian, y. and ford, n. (2007). web searchers’ attributions of success and failure: an empirical study. journal of documentation, 63(5), 659-679. doi: 10.1108/00220410710827745 mansourian, y. and ford, n. (2007). search persistence and failure on the web: a bounded rationality and satisficing analysis. journal of documentation, 63(5), 680-701. doi: 10.1108/00220410710827754 simon, h. a. (1955). a behavioral model of rational choice. quarterly journal of economics, 69(1), 99-118. simon, h. a. (1956). rational choice and the structure of the environment. psychological review, 63(2), 129-138. thompson, l. f., meriac, j.p., & cope, j. g. (2002). motivating online performance: the influences of goal setting and internet self-efficacy. social science computer review, 20(2), 149-160. webfeat. webfeat: a brief history. retrieved 29 may 2011 from http://www.webfeat.org/about.htm / evidence based library and information practice evidence summary   public libraries can be open science laboratories for citizen science projects   a review of: cigarini, a., bonhoure, i., vicens, j., & perelló, j. (2021). public libraries embrace citizen science: strengths and challenges. library & information science research, 43(2), 101090. https://doi.org/10.1016/j.lisr.2021.101090   reviewed by: matthew bridgeman information and education librarian robert wood johnson library of the health sciences rutgers, the state university of new jersey new brunswick, new jersey, united states of america email: mcb226@libraries.rutgers.edu   received: 5 june 2023                                                               accepted:  10 july 2023      2023 bridgeman. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30385     abstract   objective – the objective of this study was to evaluate the potential of libraries supporting citizen scientist (cs) projects.   design – mixed methods program evaluation study.   setting – 24 public libraries in barcelona, spain.   subjects – public librarians and library users.   methods – it is a mixed methods and mixed population study done in several phases. the first phase involved training 30 librarians how to conduct a citizen science project. they were given a pre and post survey about their perceptions of citizen science and comfort-level in conducting a project. the second phase involved a project run by the now-trained librarians with library user participation. at this phase a questionnaire was given to the users at the start and end of the project. finally, a focus group of librarians was asked about their project. the responses were evaluated through thematic analysis. seven libraries participated in the focus groups.   main results – during the first phase of the study, the survey found the librarians were pessimistic about user participation in a citizen science project, both at the beginning (75%) and at the end (79%) of the session. though they felt confident in discussing citizen science (100%) and had high satisfaction in the training (70%), only 42% felt confident to conduct a project on their own. the second phase involved the users, 94% of whom had never participated in a cs project. at the end, 70% of users said the project positively changed their perceptions of the library and 70% were satisfied with the experiment. during the focus groups, librarians said the project brought new users into the library and had the potential to build more relationships among participants and with the community. major challenges discussed were user commitment to the project and the workload required by librarians, however they all answered positively when asked about continuing with cs projects.   conclusion – this study showed that citizen science projects can be successfully implemented in public libraries. public libraries are facing challenges caused by societal change, the rise of open science, and more transparent and novel democratic ways of knowledge production. updating public library infrastructure would be needed to support these projects more fully. this may involve building partnerships and developing new guidelines. there is potential for public libraries to be leaders and innovators in citizen science.   commentary   this study was evaluated using the cat: a generic critical appraisal tool (perryman & rathbun-grubb, 2014). cigarini et al. (2001) are information and knowledge, engineering, and physics researchers at universitat de barcelona and universitat oberta de catalunya. they are well versed in stem (science, technology, engineering, and math) research but lack a library science perspective. while the library science component is not necessary, it can be beneficial in future studies. the authors began by providing a history of citizen science and potential of libraries as a project space. the question of the study—can libraries be hubs for citizen scientists?—is adequately addressed by the data. though the data visualization used is unclear at first, it does match the data after some review. they used a similar survey to evaluate subject responses before and after the project. the final focus group also answered some of the challenges faced throughout the course of the study. their conclusions matched what the data say. overall, the study is well done, but could be represented better.   this is a project full of promise. cigarini et al. (2001) defined citizen science as: “beyond open access to data, publications, and other research outputs, citizen science facilitates the active participation of citizens in the scientific research process.” conceptually, the public library is the perfect location to nurture citizen scientists. it is free and open to citizens, and librarians can provide access to open science resources. it provides space and support structure. public library involvement with citizen science is attempting to make the library analogous to a laboratory. this correlation is not unfamiliar in the library world. parallels can be drawn to the rise of makerspaces. makerspaces are a popular trend in the public library space, reinventing the library from a passive repository of knowledge to a creative place and including the use of 3d printers, electronics, cricut, and more (kim, 2022). the free library of philadelphia even has a kitchen in it for patrons to use. libraries even share some of the same challenges as citizen science projects related to roll out, particularly user engagement. perhaps the next project can draw from lessons learned in implemented makerspaces.   in their article, ross and sennyey (2008) argued that libraries must make fundamental changes to adapt to the new information world. while they discussed academic libraries specifically, and though libraries themselves have proven more resilient than they suggested, the main point still stands and is relevant to public libraries. creating a citizen science center may be one useful adaptation. the fact that there was positive engagement is a promising sign, and it should be encouraged at other public libraries. the main challenge could be spreading the word on citizen science and linking that with the library.   while the project was not perfect, it still showed possibility. perhaps there can be collaboration between public librarians and local academic librarians. public librarians can provide support to citizens, while academic librarians can help with open science resources. there are also opportunities to build relationships with schools to provide more learning experiences and relate data to real world examples. the possibilities are exciting; more projects like this should be encouraged.   references   cigarini, a., bonhoure, i., vicens, j., & perelló, j. (2021). public libraries embrace citizen science: strengths and challenges. library & information science research, 43(2), 101090. https://doi.org/10.1016/j.lisr.2021.101090     kim, s. h., jung, y. j., & choi, g. w. (2022). a systematic review of library makerspaces research. library & information science research, 44(4), 101202. https://doi.org/10.1016/j.lisr.2022.101202   perryman, c. & rathbun-grubb, s. (2014). the cat: a generic critical appraisal tool. http://www.jotform.us/cp1757/thecat    ross, l., & sennyey, p. (2008). the library is dead, long live the library! the practice of academic librarianship and the digital revolution. the journal of academic librarianship, 34(2), 145–152. https://doi.org/10.1016/j.acalib.2007.12.006     research article   measuring scholarly productivity of long island educational institutions: using web of science and scopus as a tool   clara tran science librarian science and engineering library stony brook university stony brook, new york, united states of america email: yuet.tran@stonybrook.edu   selenay aytac associate professor b. davis schwartz memorial library long island university brookville, new york, united states of america email: selenay.aytac@liu.edu   received: 1 jan. 2016       accepted: 25 may 2016        2016 tran and aytac. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – this paper explores how to utilize two well-known library databases, thomson reuter’s web of science and elsevier’s scopus, to quantify long island educational institutions’ scholarly productivity.   methods – institutions located in the long island region and within nassau and suffolk counties, including the state university of new york (suny) colleges, private institutions, and technical schools, were examined for the last 14 years (2000–2013). eight long island institutions were represented in both databases and were included in the study.   results – of the eight institutions, stony brook university produced the most publications indexed in web of science and scopus during the period of 2000–2013. cold spring harbor laboratory yielded the second most publications during 2000–2013 in both web of science and scopus, but it produced the highest quality publications compared with other institutions excluding stony brook university. although the annual growth rates of farmingdale state college and new york institute of technology increased dramatically in both web of science and scopus, the large proportional increase did not represent a large increase in total value. additionally, some institutions had a higher number of publications indexed in web of science than in scopus, and others had a higher number of publications indexed in scopus than in web of science.   conclusions – data were collected from institutions in long island with various institutional sizes, the number of faculty members employed may have made an impact on the number of publications. thus, publication data in this study cannot be used to compare their rankings. institutions with a similar type and similar size of faculty members should be selected for comparison. due to the different coverage and scope of web of science and scopus, institutions should use both databases to examine their scholarly output. furthermore, institutions should consider using altmetrics to capture various impacts of the scholarly output to complement the traditional metrics.   introduction   for decades, the traditional assessment of institutions’ scholarly or research productivity has relied on scholarly publishing (slutsky & aytac, 2014). as research is needed for institutions to remain relevant and sustain their reputation for knowledge discovery, monitoring scholarly productivity assessment data is useful for individual, departmental, and university level evaluations. university administrators can use scholarly productivity data for institutional productivity assessment and annual budget decisions. additionally, they can provide information on the overall performance of their institutions to obtain government funding and support accreditation decisions (amara, landry, & halilem, 2015).   although the first statistical analysis of scientific literature was conducted by alfred j. lotka in 1926, “bibliometrics” was coined separately by pritchard as well as nalimov and mulchenko in 1969 (glanzel, 2003). roemer and borchardt (2015) defined bibliometrics as a quantitative tool to measure and analyze research impact on print-based scholarly productivity that can be obtained by using proprietary databases or free online ranking resources. bibliometric analysis is not only a useful method to study scholarly productivity and institutions’ citation impact (wang, fu, & ho, 2011) but also one of the most used quantitative data collection methods for investigating publication patterns within a given field (aytac, 2010). common bibliometric indicators include the number of publications, number of citations, and journal impact factors (wang et al., 2011). according to roemer and borchardt (2015), bibliometrics include times cited (which measures the individual contribution level); impact factor, immediacy index, cited half-life, eigenfactor and article influence score, scimago journal rankings, h5-index and h5-median (all of which measure the journal impact level); h-index and i10 index (each measures the author level); and essential science indicators rankings, scimago institutions rankings, and snowball metrics (all of which measure the institutional level).   in recent years, interest in institutional scholarly productivity ratings has increased. these ratings are generally created by using bibliometric research tools, such as thomson reuter’s web of science (wos), elsevier’s scopus, and google’s google scholar.   for over forty years, wos was the only database that tracked citation references (meho & yang, 2006; li, burnham, lemley, & britton, 2010) and produced large scale bibliometric statistics (archambault, campbell, gingras & larivière, 2009). with its development in 2004, scopus became a good alternative to wos (manafy, 2005; dess, 2006; vieira & gomes, 2009). likewise, google scholar, also created in 2004 (adriaanse & rensleigh, 2013), can be utilized for scholarly productivity data collection (orduna-malea & aytac, 2015).   moreover, wos and scopus provide scholarly productivity data for institutions, departments, and individual faculty members. they also deliver reliable and comparable trend data that can be used to compare the research strength of institutions. the authors of this study used wos and scopus to explore the institutional scholarly productivity of nassau and suffolk counties in long island. these bibliometric tools served the following purposes in this study: (1) obtaining scholarly productivity data for each institution, (2) collecting the citation data and h-index for each institution, and (3) benchmarking these institutions for annual trend data.   data for long island institutions for the period of 2000–2013 were collected in january 2015. eight long island institutions were represented in both databases and were included in this study. due to the varying sizes of the institutions examined, this study was limited to analyzing the growth of each institution instead of comparing rankings among the institutions.   literature review   wos and scopus have been widely used for bibliometric analysis. in 2011, sicilia, sánchez-alonso, and garcía-barriocanal compared computer science-related journals and found that journal impact factors included in wos and scopus ranking lists were highly correlated and comparable. sarkozy, slyman and wu (2015) studied the publication and citation activity for individual researchers in three health sciences departments and suggested that faculty and administrators should not completely rely on citation counts as a measure of productivity due to name ambiguities and database limitations. in 2012, bergman studied the citations of social work literature and found that wos provided the fewest citation counts while scopus provided the highest citation counts, even though both databases had a similar coverage pattern. archambault et al. (2009) compared science, natural sciences, and engineering data based on “the number of papers and citations received by country” and analyzed the correlation between a country’s production and its ranking among the countries examined in their study (p.1320). they concluded that wos and scopus are “robust tools for measuring science at the country level” and suggested the study be repeated at the institutional level (p. 1325). to understand how different wos and scopus are in indexing publications at the institutional level, this study examined the long island educational institutions’ scholarly output from the period of 2000–2013. the wos was further used to collect institutional citation data and h-index for measuring their research quality.   to some extent, limitations exist in bibliometric analysis. roemer and borchardt (2015) suggested scholars check other sources for times cited numbers as the content overlapping in wos, scopus, and google scholar varies in disciplines. furthermore, impact factor does not appropriately apply to disciplines that are not focused on journals and journal articles and also does not include essays and extensive opinion works that have scholarly value. these limitations can result in an increase or decrease of the impact factor (roemer & borchardt, 2015). levine-clark and gil (2009) found that wos does not fully measure a scholar’s actual impact since it does not index all peer-reviewed journals and “other types of resources” (p. 45). in his study of journals indexed in google scholar, pubmed, and scopus, chen (2013) found that scopus does not index green oa (open access), which “refers to self-archived articles hosted on oa web sites such as institutional repositories” (p. 244). because no single metric can fully measure the true impact factor, librarians should advise researchers, faculty, and graduate students to look for traditional and nontraditional measures for a better reflection of their scholarly works’ impact factor (roemer & borchardt, 2015).   to understand the coverage of wos and scopus, the authors retrieved information from the respective products’ websites. thomson reuters (2016a) indicates that wos core collection includes five indexes and two chemistry databases:   science citation index expanded (sci) from 1900 to present with over 8,500 journals across 150 disciplines social sciences citation index (ssci) from 1900 to present with over 3,000 journals across 55 social science disciplines arts & humanities citation index from 1975 to present with over 1,700 arts and humanities journals conference proceedings citation index from 1990 to present book citation index from 2005 to present with over 50,000 selected books current chemical reactions dated to 1986 with over one million reactions and the inpi archives from 1840 to 1985 index chemicus dated to 1993 with over 2.6 million compounds   elsevier (2016b) also indicates scopus’s coverage of various types of materials, including the following:   over 21,500 peer-reviewed journals, including 4,200 open access journals 360 trade publications over 530 book series over 7.2 million conference papers over 5,000 journals with “articles-in-press” more than 116,000 books more than 38 million records with references dating back to 1996 over 22 million records pre-1996, dating to as far back as 1823 over 27 million patent records   there have been studies to compare the coverage, scope, and methodology of wos and scopus. lópez-illescas, moya-anegón, and moed (2008) agreed that the two databases differ in scope, data volume, and coverage. levine-clark and gil (2009) stated that in addition to covering mostly journals, scopus also “includes conference proceedings, book series, and trade publications” (p. 33). gravel and iselid (2008) also found that scopus covers a larger number of serial publications than wos. in 2009, levine-clark and gil studied citations for business and economics journals and reported that scopus retrieved slightly more citations than wos since scopus includes 8,000 more journals than wos. dess (2006), and li et al. (2010) found that due to different coverage, wos allows for a longer period of citation tracking than scopus. scopus only covers citation tracking from 1996 onward (li et al., 2010). during their study of content verification and quality of the south african environmental sciences journals, adriaanse and rensleigh (2013) found that scopus provides the most comprehensive coverage of title, author, and volume number compared to wos and google scholar.   the scope of disciplines covered by the two databases also varies. elsevier (2016b) shows that scopus’s subject areas include the life sciences (15%), health sciences (32%), physical sciences (29%), and social sciences (24%). dess's study in 2006 showed that scopus is heavily focused on the health and life sciences with less emphasis on physical science, mathematics, psychology, and social sciences and even less emphasis on business and marketing. li et al. (2010) agreed that scopus provides strong coverage in health sciences and physical sciences but not the other disciplines.   wos provides two categories of searches: bibliographic search and cited reference search (li et al., 2010). dess (2006), and li et al. (2010) stated that bibliographic information can be found using the basic, advanced, and author searches. the basic or the advanced search allows users to obtain specific information from search results, such as the “numbers of articles in subject areas, document type, authors, source titles, publication years, institutions, funding agencies, languages, and countries” (li et al., 2010, p. 198). furthermore, users can obtain a citation report that includes “the search results found, sum of the times cited, average citations per item, and h-index number” from search results (p. 198). wos also includes unique features, such as distinct author set and citation map. additionally, wos provides a useful statistical tool, journal citation reports, which measures journal impact factor (levine-clark & gil, 2009).   likewise, li et al. (2010) described scopus’s author identifier, which retrieves matches from “their affiliation, address, subject area, source title, dates of publication citations, and co-authors,” as the strength of the database (p. 201). similar to wos, scopus provides a cited reference list when searching for an author. citation analysis allows users to “view the articles that cited the original articles” and the h-index provides graphs that display publication record strength (p. 201). furthermore, scopus provides a journal analyzer that allows users to compare journals in terms of “number of citations, articles published, and percentage not cited” (p. 202). gavel and iselid (2008) observed that it is more difficult to study the overlapping coverage at an article level than at the journal level because overlapping coverage at the article level requires users to identify “the bibliographic subfields of individual articles cited” (p. 9).   although wos and scopus provide reliable bibliographic data for institutions, they have limitations. first, the two databases have different criteria for indexing publications. goodwin (2014) stated that the “organization-enhanced” option does not include all organizations that are indexed in wos. in scopus, a document that does not have sufficient citation information may not be correctly assigned to the affiliation from which the publication originates (elsevier, 2016a). the two databases include different document types, disciplines, languages, and time periods (zhang, 2014). the databases have other issues related to published journals. first, the databases have limited scholarly journal coverage based on the information provided on the products' sites. second, the databases have limited coverage of open access journals, although wos includes over 12,000 high impact journals, including open access journals (thomson reuters, 2016a), and scopus indexes 4,200 open access journals (elsevier, 2016a), but neither database includes all the journal titles in the directory of open access journals (doaj, 2016). third, the databases have limited coverage of non-periodical resources, such as monographs and dissertations. further, scopus covers patents (elsevier, 2016b) but wos does not. additionally, limited coverage for non-western publications and the language bias of these indexes may affect publication count.   although bibliometric research methods, particularly citation indexes, have received considerable attention in the literature, some limitations of these indexes have been noted by researchers. okubo and miquel (1990) pointed out that for some cases, authors’ affiliations are not always the true indicator of the corresponding research’s origin. since co-authorships are the primary indicators of affiliations and can only be tracked by authors’ affiliation data, the amount of co-authorship studies in wos’s indexes may be limited.   limitations of the ssci, and particularly its “representativity” problem, which corresponds to the equal representation of each country’s research publication, are underlined by schoepflin (1990). however, the main problem that corresponds with representativity is largely related to the publication language of journals. unfortunately, journal articles published in non-mainstream languages are not likely to be in both indexes of wos. similarly, braun, glanzel, and schubert (2000) evaluated the representativeness of the sci’s journal coverage at the level of countries. this is a very valid issue especially for non-western or non-english speaking countries. for instance, only a few turkish journals are listed in journal citation reports, and both the sci and ssci indexes are lacking in terms of representation of most of developing countries due to language bias. as english is the lingua franca of science, the research done in non-english languages is oftentimes lost. the language bias of the wos database was repeatedly discussed in the literature. cole and phelan (1999), osareh and wilson (1997), and barrios, borrego, ollé, vilaginés, and somoza (2008) have pointed out this as a limitation in reaching those non-english scientific journals. in the same vein, mongeon and paul-hus (2016) reported that scopus has similar aforementioned limitations despite its much larger coverage. the authors can conclude that both databases wos and scopus have similar limitations.   methods   there are numerous ways to quantify an institution’s research or scholarly productivity. one way is counting the number of scholarly outputs produced by the institution. this data generally consists of the number of publications made by faculty, students, and staff affiliated with the institution. in january 2015, the period of 2000–2013 was chosen (instead of 2000–2014 for data collection because publications in 2014 might not have been fully indexed in wos and scopus). document types including articles, reviews, proceedings, books, and book chapters were included in this study. in addition to collecting the publication counts for measuring research productivity using the two databases, citation counts were also collected for measuring research quality on april 19, 2016, using only wos. scopus was not able to provide a large dataset for many of the institutions for the selected period of 2000–2013 in a single query.   samples   a period of fourteen years (2000–2013) of scholarly productivity of eighteen long island institutions were identified for data collection. cold spring harbor laboratory was included in the study because of its phd program in biological sciences. brookhaven national laboratory was not considered for this study because it is not an academic institution.   the bibliometric analysis revealed that only eight of the eighteen institutions were represented in both wos and scopus databases. the data of institutions, which range from private to public colleges and universities, were used for further analysis. below is the list of the eight institutions:   adelphi university cold spring harbor laboratory (watson school of biological sciences) farmingdale state college hofstra university long island university new york institute of technology stony brook university suny old westbury   procedures   the data of the eighteen institutions were collected from the wos and scopus databases at the end of january 2015. data collection involved several steps. the annual number of scholarly productions per institution was extracted from wos and scopus and exported from the databases to an excel spreadsheet for analysis and calculations. then, because the eight institutions were represented in both databases, their data were filtered for further analysis. finally, the annual growth rate of these eight institutions’ scholarly productivity was calculated for each year using this formula:         for farmingdale state college, the calculations in wos were based on the year of 2003 as no prior publications from 2000 to 2002 were recorded.   during the data collection process, the “organization-enhanced” option in wos and the “affiliation search” option in scopus were used. in wos, the “organization-enhanced” option allows users to find publications from institutions with name variants. users can either enter the organization name in the search field or click the “select from index” link to search for the organization. this “select from index” link provides users with options to either select the organization name from the “organization-enhanced” list or enter the name in the “find” field.  additionally, selecting the preferred name from the list or entering the preferred name in the “find” field yields a more accurate result because the result is retrieved from the addresses linked to that organization (thomson reuters, 2015). in this study, preferred names were mainly collected from the “organization-enhanced” list without expanding the “view details” option to add or exclude any affiliations. data for farmingdale state college and the new york institute of technology, however, were collected using the search field.   like wos, scopus also allows users to search for an organization using the “affiliation search” option. when a list of affiliations is generated, the affiliations of the institutions can be selected from the list. this list of affiliations provides links to documents and any available information about the affiliations, such as affiliation id, name variations, and address information (elsevier 2016a). data for farmingdale state college, however, were collected using the "document search," followed by "affiliation name."   results   scholarly productivity data 2000–2013 for each institution   based on wos from 2000 to 2013, stony brook university produced the most scholarly publications (33,406) as shown in table 1 and table 1a. stony brook university was followed by cold spring harbor laboratory (2,935), hofstra university (2,507), adelphi university (1,446), long island university (1,237), suny old westbury (424), farmingdale state college (61), and new york institute of technology (16).   in scopus and searching from 2000 to 2013, stony brook university also produced the most scholarly publications (30,759) as shown in table 2 and table 2a. stony brook university was followed by cold spring harbor laboratory (2,834), long island university (2,369), hofstra university (2,229), adelphi university (1,415), new york institute of technology (1,040), suny old westbury (320), and farmingdale state college (96).   times cited and h-index of each institution   data which were collected in april 2016 revealed that cold spring harbor laboratory produced the highest quality research papers, followed by hofstra university, long island university, adelphi university, suny old westbury, farmingdale state college, and new york institute of technology as shown in table 3. stony brook university was not included in this comparison. according to thomson reuters (2016b), the citation report feature in wos only allows a search of citation activity for up to 10,000 records. as stony brook university’s scholarly output record from 2000 to 2013 was 33,790, multiple searches for citation activities were required (see table 4). additionally, data showed that some institutions had a slightly higher scholarly output number than data that were collected in january 2015; this did not affect the analysis’s result.     table 1 web of science – institutional scholarly productivity from 2000–2006 name of institution 2000 2001 2002 2003 2004 2005 2006 adelphi university 71 79 57 52 92 82 94 cold spring harbor laboratory 165 174 210 208 219 213 208 farmingdale state college 0 0 0 3 2 1 3 hofstra university 122 139 135 143 136 173 168 long island university 67 91 90 87 80 80 77 new york institute of technology 1 0 1 1 2 0 0 stony brook university 2102 2127 2059 2087 2274 2258 2359 suny old westbury 43 26 30 22 27 45 40     table 1a web of science – institutional scholarly productivity from 2007–20013 name of institution 2007 2008 2009 2010 2011 2012 2013 total adelphi university 112 123 105 132 133 154 160 1446 cold spring harbor laboratory 221 231 208 201 216 218 243 2935 farmingdale state college 5 6 6 8 5 9 13 61 hofstra university 140 184 178 192 263 275 259 2507 long island university 79 109 105 96 89 102 85 1237 new york institute of technology 1 1 0 2 3 0 4 16 stony brook university 2418 2487 2558 2466 2586 2766 2859 33406 suny old westbury 21 34 37 30 22 26 21 424     table 2 scopus – institutional scholarly productivity from 2000–2006 name of institution 2000 2001 2002 2003 2004 2005 2006 adelphi university 43 43 41 55 81 83 99 cold spring harbor laboratory 152 149 152 210 215 236 197 farmingdale state college 1 2 1 0 1 3 3 hofstra university 56 81 86 131 131 193 168 long island university 109 114 129 123 147 165 185 new york institute of technology 22 23 24 39 46 36 71 stony brook university 1754 1643 1653 1846 2045 2188 2334 suny old westbury 35 18 23 21 25 33 23     table 2a scopus – institutional scholarly productivity from 2007–2013 name of institution 2007 2008 2009 2010 2011 2012 2013 total adelphi university 135 105 128 131 155 157 159 1415 cold spring harbor laboratory 221 217 203 189 206 230 257 2834 farmingdale state college 7 8 13 23 11 13 10 96 hofstra university 156 182 183 192 231 237 202 2229 long island university 179 202 193 217 185 213 208 2369 new york institute of technology 81 95 98 106 112 132 155 1040 stony brook university 2333 2334 2368 2349 2502 2611 2799 30759 suny old westbury 14 22 27 23 19 19 18 320     table 3 web of science – institutional scholarly output, times cited, and h-index from 2000–2013 name of institution scholarly output times cited h-index adelphi university 1452 11387 48 cold spring harbor laboratory 2960 313392 255 farmingdale state college 62 319 11 hofstra university 3270 26042 63 long island university 1242 13517 51 new york institute of technology 17 75 5 stony brook university* 33790 * * suny old westbury 424 5427 37 *please see table 4.     table 4 web of science – stony brook university scholarly output, times cited, and h-index from 2000–2013 year scholarly output times cited h-index 2000–2003 8462 310600 211 2004–2007 9389 278826 197 2008–2010 7578 153301 144 2011–2013 8361 99894 103 total 33790 842621       benchmark institutions for annual trend data   each institution’s publication growth was measured by the percent increase in annual growth indexed in wos and scopus. the following figures provide benchmarking of all the institutions and give a clear view of productivity growth for each of the institutions between the years 2000 and 2013.   in 2013, farmingdale state college as indexed in wos topped an annual growth rate of +333%, followed by new york institute of technology (+300%), adelphi university (+125%), hofstra university (+112%), cold spring harbor laboratory (+47%), stony brook university (+36%), long island university (+27%), and suny old westbury (-51%). these growths rates are shown in figure 1.     figure 1 web of science – institutional annual growth rate from 2000–2013.     figure 2 scopus – institutional annual growth rate from 2000–2013.     in 2013, as shown in figure 2, farmingdale state college as indexed in scopus also topped an annual growth rate of (+900), followed by new york institute of technology (+605%), adelphi university (+270%), hofstra university (+261%), long island university (+91%), cold spring harbor laboratory (+69%), stony brook university (+60%), and suny old westbury (-49%).   comparison of the two databases on scholarly productivity of each long island institution for 14 years from 2000–2013   table 5 provides the institutional annual comparison between the wos and scopus. referring to the adelphi university, the numbers of years with publications indexed in wos (7) and scopus (7) were the same.      table 5 institutional annual comparison between web of science and scopus from 2000–2013   name of institution number of years with publications indexed in web of science   number of years with publications indexed in scopus number of years with publications indexed in web of science and scopus adelphi university 7 7 0 cold spring harbor laboratory 9 4 1 hofstra university 9 3 2 farmingdale state college 3 10 1 long island university 0 14 0 new york institute of technology 0 14 0 stony brook university 14 0 0 suny old westbury 14 0 0     cold spring harbor laboratory had a higher number of years with publications indexed in wos (9) than in scopus (4) with one year that had the same number of articles in both databases. similarly, hofstra university had a higher number of years with publications indexed in wos (9) than in scopus (3) with two years that had the same number of articles in both databases. on the other hand, farmingdale state college had more years with publications indexed in scopus (10) than in wos (3) with one year that had the same coverage in both databases.   data from long island university and new york institute of technology showed that both institutions had a higher number of years with publications indexed in scopus (14) than in wos (0) for every single year from 2000 to 2013. to the contrary, stony brook university and suny   old westbury had a higher number of years with publications indexed in wos (14) than in scopus (0) for every single year from 2000 to 2013.   discussion   in terms of publications, data showed that stony brook university produced the most publications during 2000–2013 in both wos and scopus. the carnegie classification of higher education (n.d.) showed that stony brook university is classified as a research university with very high research activity. additionally, stony brook university employed 2,471 faculty members in the fall of 2013 (stony brook university, 2015). the number of faculty members employed may make an impact on the number of publications. although cold spring harbor laboratory produced the second most scholarly output during 2000–2013 in both wos and scopus among the eight institutions, it produced the highest quality publications compared with six institutions.   regarding the institutional annual growth rate, figures 1 and 2 revealed that the annual growth rates of farmingdale state college and new york institute of technology increased dramatically in both wos and scopus. slutsky and aytac (2014) explained that a large proportional increase in annual productivity does not represent a large increase in total value; the data presented should be viewed as trend data and no conclusion should be made from these observations. however, the presented data can be useful to see the general trend in scholarly growth among the long island institutions.    additionally, the graphs provided background information regarding annual scholarly productivity per institution for this investigation. for instance, adelphi university, cold spring harbor laboratory, hofstra university, stony brook university, and suny old westbury, all of which had the same or a higher number of publications indexed in wos during 2000–2013, are either affiliated with medical schools or heavily involved in scientific research. hence, the scholarly productivity is higher. goodwin (2014) also stated that the publications that wos indexes are heavily weighted towards the sciences, particularly towards the life sciences. on the other hand, farmingdale state college, long island university, and new york institute of technology had a higher number of publications indexed in scopus due to the publications of materials such as dissertations and theses in the humanities. archambault et al. (2009) observed that wos and scopus do not have the same system of categorizing documents; the two databases may “label the same documents differently” (p.1321). to see the spread of types was for each database, the documents from long island university in wos and scopus were identified for further analysis as shown in table 6. the data were taken from wos and scopus in the month of june of 2015 for this specific case.     table 6 document types in wos and scopus for long island university in 2013 long island university document type wos scopus article 62 168 book 0 4 book chapter 0 12 book review 3 0 conference paper/proceeding paper 1 8 editorial/editorial material 2 3 meeting abstract 14 0 note 0 5 review 3 9 total 85 209     figure 3 number of documents in wos and scopus as well as the overlapping citations in both databases for long island university in 2013.     long island university had 61 items indexed in both wos and scopus. see figure 3. among these 61 items, the document types included article, review, editorial, book chapter, and conference proceeding as displayed in table 7. in this subset, scopus indexed 54 articles while wos indexed 55 articles. scopus indexed 2 conference papers while wos indexed only one. in this case, the insignificant difference in number was not sufficient to show that either scopus or wos labeled journal articles and conference papers very differently. however, institutions should use both wos and scopus to examine their scholarly output as the databases’ coverage and scope are different.   google scholar compliments scholarly productivity findings from traditional approaches. google scholar, an academic search engine launched in november 2004, indexes and retrieves academic content throughout the internet. google scholar recently released an automatic institutional affiliation tool that gathers all authors belonging to one institution. google scholar, however, cannot directly retrieve the number of documents published by one university as opposed to wos or scopus can. instead, specific queries can be performed to retrieve the number of documents stored on the official university website. considering the role of institutional repositories, this procedure might represent a proxy (orduna-malea, ayllón, martín-martín, & lópez-cózar, 2015; orduna-malea & lópez-cózar, 2014; orduna-malea, serrano-cobos, & lloret-romero, 2009). table 8 displays the results from google scholar for each institution included in the study. the hit count estimates (number of documents stored within each official university website) were retrieved from google.com with the "site" search command. the data were collected on april 1, 2016.     table 7 overlapping document type in wos and scopus overlapping document type wos scopus article 55 54 review 3 3 editorial material; book chapter/editorial 1 1 article; book chapter/book chapter 1 1 proceeding paper/conference paper 1 2 total 61 61     table 8 google scholar – institutional scholarly productivity from 2000–2013 name of institutions domain names google scholar (2000–2013) google scholar (all years) adelphi university adelphi.edu 41 52 cold spring harbor laboratory (watson school of biological sciences) cshl.edu 182 242 farmingdale state college farmingdale.edu 12 19 hofstra university hofstra.edu 660 1140 long island university liu.edu 36 63 new york institute of technology nyit.edu 77 105 stony brook university stonybrook.edu 2040 2040 suny old westbury oldwestbury.edu 1 1 total 3049 3662     when interpreting data from google scholar, a small amount does not mean low productivity. an institution may publish a large quantity of papers, but if these materials are not deposited on the website (especially in an institutional repository), the number of items indexed in google scholar will be low. more importantly, a high number may mean, with some confidence, great performance and good visibility online. if citation data is needed, citations must be manually created for every item with the query "site:url". additionally, having information management strategies, particularly institutional repositories, may help universities be better represented on google scholar.   conclusion   one of the well-accepted goals of institutions is to increase institutional research. educational institutions would find it beneficial to use wos and scopus more systematically to obtain scholarly productivity data on student, faculty, and staff engagement in research activities.  these data can be also used to shape institutions’ decisions on strategic planning, research allocations, and research funding.   as data were collected from institutions in long island with various types, missions, and institutional sizes, publication data in this study cannot be used to compare their rankings. for instance, both hofstra and stony brook have medical schools, but hofstra established its medical school recently, and the two universities have different sizes of faculty bodies, making them very difficult to compare. this study should be repeated for another cluster of new york institutions, such as suny campuses or cuny colleges, with a similar size of faculty members. additionally, similar type of institution should be examined, such as two-year community colleges or four-year research universities.   additionally, our findings suggest that the use of the publication indicator does not cover the full research profile of the long island institutions that were selected as a sample; nevertheless, they do provide a sense of research growth for each institution. in order to gain a comprehensive awareness of the research activity of each institution, a future study may involve analyses of the scholarly productivity data in relation to clusters of strength in different research disciplines in science, engineering, social sciences, humanities, and medicine, with a focus on wos or scopus.   another scholarly productivity data source, google scholar, provided trend data for the long island institutions in this paper. it is important to note that materials deposited on the internet or an institutional repository may yield a higher number of items indexed in google scholar. additionally, altmetrics should be considered to capture various impacts of the scholarly output to complement the traditional metrics.   as outlined previously, the limitations of these two databases will not allow a full examination of the scholarly productivity of each institution. however, the micro level of data collection procedures that was provided should be helpful to obtain the institutional scholarly output. future projects can be built on these findings to extend the knowledge and understanding of the scholarly productivity of long island scholars.   references   adriaanse, l. s., & rensleigh c. (2013). web of science, scopus and google scholar: a content comprehensiveness comparison. the electronic library, 31(6), 727-744. http://doi.org/10.1108/el-12-2011-0174   amara, n., landry, r., & halilem, n. (2015). what can university administrators do to increase the publication and citation scores of their faculty members? scientometrics, 103(2), 489-530. http://doi.org/10.1007/s11192-015-1537-2   archambault, é., campbell, d., gingras, y., & larivière, v. (2009). comparing bibliometric statistics obtained from the web of science and scopus. journal of the american society for information science and technology, 60(7), 1320-1326. http://doi.org/10.1002/asi.21062   aytac, s. (2010). scientific international collaboration of turkey, greece, poland, and portugal: a bibliometric analysis. proceedings of the american society for information science and technology, 47(1), 1-3. http://doi.org/10.1002/meet.14504701305   barrios, m., borrego, a., ollé, c., vilaginés, a., & somoza, m. (2008). a bibliometric study of psychological research on tourism. scientometrics, 77(3), 453-467. http://doi.org/10.1007/s11192-007-1952-0   bergman, e. m. l. (2012). finding citations to social work literature: the relative benefits of using web of science, scopus, or google scholar. the journal of academic librarianship, 38(6), 370-379. http://doi.org/10.1016/j.acalib.2012.08.002   braun, t., glanzel, w., & schubert, a. (2000). collaboration networks in science. in b. cronin & h. b. atkins (eds.), the web of knowledge (pp. 251-277). medford, nj: information today.   chen, x. (2013). journal article retrieval in an age of open access: how journal indexes indicate open access articles. journal of web librarianship, 7, 243-254. http://dx.doi.org/10.1080/19322909.2013.795426   cole, s., & phelan, t. j. (1999). the scientific productivity of nations. minerva, 37(1), 1-23.   dess, h. m. (2006). database reviews and reports: scopus. issues in science and technology librarianship, 46. http://doi.org/10.5062/f4x0650t   doaj. (2016). directory of open access journals. retrieved april 21, 2016, from https://doaj.org/   elsevier. (2016a). scopus: affiliation search. retrieved april 20, 2016, from http://help.elsevier.com/app/answers/detail/a_id/2312/p/8150/kw/affiliation%20search/search/1   elsevier. (2016b). scopus: content. retrieved april 20, 2016, from http://www.elsevier.com/solutions/scopus/content   gavel, y., & iselid, l. (2008). web of science and scopus: a journal title overlap study. online information review, 32(1), 8-21. http://dx.doi.org/10.1108/14684520810865958   glanzel, w. (2003). bibliometrics as a research field: a course on theory and application of bibliometric indicators. national science digital library. retrieved from http://nsdl.niscair.res.in/jspui/handle/123456789/968   goodwin, c. (2014). advisor reviews-standard review: web of science. the charleston advisor, 16(2), 55-61. http://dx.doi.org/10.5260/chara.16.2.55   levine-clark, m., & gil, e. l. (2009). a comparative citation of web of science, scopus, and google scholar. journal of business & finance librarianship, 14, 32-46. http://dx.doi.org/10.1080/08963560802176348   li, j., burnham, j. f., lemley, t., & britton, r. m. (2010). citation analysis: comparison of web of science, scopus, scifinder, and google scholar. journal of electronic resources in medical libraries, 7, 196-217. http://dx.doi.org/10.1080/15424065.2010.505518   lópez-illescas, c., moya-anegón, f. d., & moed, h. f. (2008). coverage and citation impact of oncological journals in the web of science and scopus. journal of informetrics, 2, 304-316. http://dx.doi.org/10.1016/j.joi.2008.08.001   manafy, m. (2005). scopus of influence: content selection committee announced. econtent, 28(10), 12.   meho, l. i., & yang k. (2007). impact of data sources on citation counts and rankings of lis faculty: web of science vs scopus and google scholar. journal of american society for information science and technology, 58(13), 2105-2125. http://dx.doi.org/10.1002/asi.20677   mongeon, p., & paul-hus, a. (2016). the journal coverage of web of science and scopus: a comparative analysis. scientometrics, 106(1), 213-228. http://dx.doi.org/10.1007/s11192-015-1765-5   okubo, y., & miquel, j. f. (1990). international cooperation in basic science. in p. weingart, r. sehringer, & m. winterhage (eds.), representations of science and technology: proceedings of the international conference on science and technology indicators (pp. 124-143). leiden: dswo press.   orduna-malea, e., ayllón, j. m., martín-martín, a., & lópez-cózar, e. d. (2015). methods for estimating the size of google scholar. scientometrics, 104(3), 931-949. http://dx.doi.org/10.1007/s11192-015-1614-6   orduna-malea, e., & aytac, s. (2015). revealing the online network between university and industry: the case of turkey. scientometrics, 105(3), 1849-1866. http://dx.doi.org/10.1007/s11192-015-1596-4   orduna-malea, e., delgado lópez-cózar, e. (2014). google scholar metrics evolution: an analysis according to languages. scientometrics, 98(3), 2353-2367. http://dx.doi.org/10.1007/s11192-013-1164-8   orduna-malea, e., serrano-cobos, j., & lloret-romero, n. (2009). spanish public universities in google scholar: presence, evolution and coverage of their scientific output. el profesional de la información, 18(5), 493-500. http://dx.doi.org/10.3145/epi.2009.sep.02   osareh, f., & wilson, c. (1997). third world countries (twc) research publications by disciplines: a country-by-country citation analysis. scientometrics, 39(3), 253-266. http://dx.doi.org/10.1007/bf02458529   roemer, r. c., & borchardt, r. (2015). meaningful metrics: a 21st-century librarian’s guide to bibilometrics, altmetrics, and research impact. chicago, il: association of college and research libraries.   sarkozy, a., slyman, a., & wu, w. (2015). capturing citation activity in three health sciences departments: a comparison study of scopus and web of science. medical reference services quarterly, 34(2), 190-201.  http://dx.doi.org/10.1080/02763869.2015.1019747   sicilia, m.-a., sánchez-alonso, s., & garcía-barriocanal, e. (2011).  comparing impact factors from two different citation databases: the case of computer science. journal of informetrics, 5, 698-704. http://dx.doi.org/10.1016/j.joi.2011.01.007   slutsky, b. & aytac, s. (2014). publication patterns of science, technology, and medical librarians: review of the 2008-2012 published research. science & technology libraries, 33(4), 369-382. http://dx.doi.org/10.1080/0194262x.2014.952486   schoepflin, u. (1990). problems of representativity in the social sciences citation index. in p. weingart, r. sehringer, & m. winterhager (eds.), representations of science and technology: proceedings of the international conference on science and technology indicators (pp. 177-188). leiden: dswo press.   stony brook university. (2015). institutional research, planning & effectiveness. human resources (faculty & staff): faculty. retrieved from http://www.stonybrook.edu/commcms/irpe/data/hr/index.html   the carnegie classification of institutions of higher education. (n.d.). institution profile: stony brook university. retrieved april 26, 2015, from http://carnegieclassifications.iu.edu/lookup/view_institution.php?unit_id=196097&start_page=lookup.php&clq=%7b%22first_letter%22%3a%22s%22%7d   thomson reuters. (2015). web of science core collection help: searching the organizations-enhanced field. retrieved june 15, 2015, from https://images.webofknowledge.com/wokrs518b4/help/wos/hs_organizations_enhanced.html   thomson reuters. (2016a). the world’s most trusted citation index: web of science core collection covering the leading scholarly literature. retrieved april 21, 2016, from http://wokinfo.com/products_tools/multidisciplinary/webofscience/   thomson reuters. (2016b). web of science: analysis the knowledge to decide. retrieved april 21, 2016, from http://wokinfo.com/benefits/whywok/ouranalysistools/   vieira, e. s., & gomes, j.a.n.f. (2009). a comparison of scopus and web of science for a typical university. scientometrics, 81(2), 587-600. http://dx.doi.org/10.1007/s11192-009-2178-0   wang, m.-h., fu, h.-z., & ho, y.-s. (2011). comparison of universities’ scientific performance using bibliometric indicators. malaysian journal of library & information science, 16(2), 1-19.   zhang, l. (2014). the impact of data source on the ranking of computer scientists based on citation indicators: a comparison of web of science and scopus. issues in science and technology librarianship, 75. http://doi.org/10.5062/f4d798cw     news/announcements    call for applications: evidence summary writers      2015. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   the evidence based library and information practice journal would like to announce opportunity to fulfil a role within the eblip journal as an evidence summary writer.  evidence based library and information practice (eblip) is a non-profit, open access journal and all positions are voluntary and unpaid.   call for applications:  evidence based library and information practice evidence summaries writers eblip seeks to add five writers to the evidence summaries team. evidence summaries provide critical appraisal syntheses for specific research articles. these research synopses provide readers with information regarding the original research article's validity and reliability, thus providing information on the presence or absence of evidence with which to make informed decisions. evidence summaries team members are required to write two evidence summaries per year, with a two year commitment to the journal. evidence summaries cover all areas of library and information studies and we encourage applications from information professionals in areas such as school, public, and special libraries, as well as academic settings. interested persons should send a statement of interest, indicating areas of strength they would bring to the role, as well as a brief resumé to heather pretty (associate editor, evidence summaries) hjpretty@mun.ca by january 15, 2016. applicants who are shortlisted* will be asked to submit a sample evidence summary. please note that evidence based library and information practice is a non-profit, open access journal and all positions are voluntary and unpaid. the positions are an excellent opportunity for continuing professional development and gaining experience in reviewing or critically appraising library-related research. *only those applicants who are selected or shortlisted will be contacted by the editors. about the journal: published quarterly by the university of alberta, this peer reviewed, open access journal is targeted at all library and information professionals interested in an evidence based model of practice. by facilitating access to library and information studies research via original research articles and evidence summaries of relevant research from the library literature, evidence based library and information practice enables information professionals to practice their profession in an evidence-based manner. please visit the evidence based library and information practice website (http://ejournals.library.ualberta.ca/index.php/eblip) for further information about the journal. thank you, lorie kloda editor-in-chief   evidence based library and information practice commentary   how to develop a validated geographic search filter: five key steps   lynda ayiku information specialist national institute for health and care excellence (nice) manchester, united kingdom email: lynda.ayiku@nice.org.uk   jenny craven information specialist national institute for health and care excellence (nice) manchester, united kingdom email: jenny.craven@nice.org.uk   thomas hudson information specialist national institute for health and care excellence (nice) manchester, united kingdom email: thomas.hudson@nice.org.uk   paul levay information specialist national institute for health and care excellence (nice) manchester, united kingdom email: paul.levay@nice.org.uk   received: 5 sept. 2019                                                                  accepted: 24 jan. 2020      2020 ayiku, craven, hudson, and levay. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29633   introduction   the purpose of this commentary is to increase awareness of the existing validated geographic search filters and to encourage the creation of new filters for additional places in the world.   search filters are collections of search terms that are designed to find evidence with a common feature (glanville et al., 2008). they differ from search strategies because their retrieval ability has been tested (validated) against a set of relevant references (glanville et al., 2008). this provides users with an indication of how successfully filters work for retrieving the type of evidence that they wish to identify.   most filters aim to retrieve evidence with a specific study design (damarell, may, hammond, sladek & tieman, 2019). information professionals will probably be most familiar with those for systematic reviews or randomized controlled trials. however, an increasing number of “topic search filters” have been developed for clinical conditions, demography, health care delivery issues, and geographic locations (damarell et al., 2019).   geographic search filters are applied to literature searches with the aim of retrieving evidence about geographic locations such as continents or countries. as of 2020, only three validated geographic filters are available in published literature (glanville, lefebvre & wright, 2020):   1.       spain: pubmed (valderas, mendivil, parada, losada‐yáñez,& alonso, 2006) 2.       africa: pubmed and embase (pienaar, grobler, busgeeth, eisinga, & siegfried, 2011) 3.       uk: medline and embase, ovid platform (ayiku et al., 2017, 2019)   there are search strategies for other geographic locations that are labelled as “search filters”, but these have not been created and validated using recognized filter development methods (ayiku et al., 2017).   geographic restrictions are not always applied to searches with a geographic focus when validated geographic filters are unavailable. for instance, in a post-development study for the national institute for health and care excellence (nice) uk filters, 100 uk-focused systematic reviews were identified that had no geographic restrictions in their searches (the searches were conducted before the uk filters were available publicly) (ayiku & finnegan, 2019). a potential reason for this is that information professionals may have concerns about excluding relevant geographic evidence by accident through the use of untested search approaches. however, when restrictions are not applied, references about a specific location need to be identified from a larger set of irrelevant geographic literature. this approach is time-consuming and inefficient.   geographic filters enable effective and efficient literature searches for topics with a geographic focus. they can retrieve most of the evidence about a geographic region while limiting the retrieval of irrelevant references about other geographic regions (ayiku et al., 2017, 2019). geographic filters therefore save time and associated resource costs spent on selecting evidence for topics about specific regions.   developing and validating geographic search filters: five key steps   the following steps are based on filter development methodologies (jenkins, 2004; sampson et al., 2006; glanville et al., 2008) in addition to the authors’ knowledge gained during the creation of the nice uk filters for medline and embase (ayiku et al., 2017, 2019). the process for developing geographic filters is outlined in figure 1.   figure 1 process for developing a geographic search filter. step 1. define the geographic region   official definitions can help to specify the geographic region for the filter if required.   step 2. find references for the region   2a. identifying references   a set of references about the geographic region for the filter is required to develop and validate geographic search filters. this set is called a “gold standard” (also known as a “reference set”) (jenkins, 2004). evidence based sources such as systematic reviews or guidelines usually provide descriptions about the geographic setting of the references that informed them. the gold standard set can be created by pooling relevant references that have informed evidence-based sources (sampson et al., 2006). the aim is to enable the pragmatic collection of references that have been previously identified for the topic of the filter. this method of reference identification is used to validate filters via the “relative recall” approach and it is quicker than finding relevant references by hand searching journals (sampson et al., 2006). however, hand searching can be used to create a gold standard set for geographic filters if preferred.   the authors identified references with a uk setting for the gold standard set from nice guidance documents to develop the nice uk filters (ayiku et al., 2017, 2019). 2b. how many references for the gold standard set are needed?   the authors advise that at least 300 references about a geographic location should be identified for the gold standard set. this is because it is possible that some references will not be available in the bibliographic database for the filter. in addition, the references will need to be divided into the following sets:   1.       development set: used to create filters 2.       validation set: used to validate filters   sampson et al. (2006) suggest that at least 100 references are required to validate filters because this sample size will provide a reasonable confidence interval (assuming that the filter retrieves 90% of the validation set references). finding a minimum of 300 references will help to ensure that there are at least 100 references for the validation set.   step 3. form the gold standard set   3a. locating references in the bibliographic database for the filter   when 300 or more references have been identified, their availability in the database for the filter needs to be checked. to locate the references in the database, enter key bibliographic details (such as title and author) for each reference into the database. the references that are available will form the gold standard set.   the existing geographic filters have been designed for the pubmed, medline, and embase bibliographic databases (valderas et al., 2006; pienaar et al., 2011; ayiku et al., 2017, 2019). however, it may be appropriate to design a filter for another database if it is relevant to do so. 3b. creating the development and validation sets   next, the references in the gold standard set need to be split into a development set and a validation set. for rigor, the references should be randomized prior to their division. to do this, assign each of the references a number (this could simply be their number order). a free online randomizer tool can be used to randomize the numbers. the authors used random.org (randomness and integrity services ltd, 2020) for the nice uk filters (ayiku et al., 2017, 2019).   once the references have been randomized and divided, create two search strategies in the database for the filter; one for the development set references and another for the validation set references. for both search strategies, combine the references at the end using the or boolean operator. as an example, the nice uk filter search strategies for the development set and validation set references were structured as follows:   1.       langford i (author) and “the potential effects of climate change on winter mortality in england and wales” (title) and 1995 (year) 2.       chahal r (author) and “a study of the morbidity, mortality and long-term survival following radical cystectomy and radical radiotherapy in the treatment of invasive bladder cancer in yorkshire” (title) and 2003 (year) 3.       saka o (author) and “cost of stroke in the united kingdom” (title) and 2009 (year) 4.       etc… 5.       1 or 2 or 3 or 4…   save both search strategies in the database account so that they can be re-run to test the retrieval ability of the filter during steps four and five.   step 4. develop filter   4a. development set   the purpose of the development set references is to identify the most relevant search fields and search terms to create the geographic filter. creating filters using fields and terms from the development set references will help to ensure that the most relevant details for the filter are identified (hausner, waffenschmidt, kaiser, & simon, 2012). filters that are created in this way are known as “objectively-derived” filters (jenkins, 2004). the authors used this approach to create the nice uk filters (ayiku et al., 2017, 2019).   identifying relevant search fields   an excel spreadsheet can be used to identify relevant search fields from the development set references. if the filter is for an ovid database, the “excel sheet” export option can be used to transfer the database records for development set references into excel. using the “csv” export option will work in a similar way to transfer database records into excel if the filter is for pubmed.   in the excel spreadsheet, the content for each search field from the development set database records is displayed in separate columns. the search fields that contain geographic setting details about your region of interest will be the relevant fields for your filter.   the most relevant fields found in excel for the nice uk filters (ayiku et al., 2017, 2019) were:   ·         subject heading ·         title ·         abstract ·         journal name ·         institution   uk setting terms also appeared in the ‘country of publication’ field but it was not included in the final version of the filter. this is because several uk-based publishing companies produce journals that contain international content. however, it may be useful to add the ‘country of publication’ field if your filter is for a country in which publishing companies are more likely to publish geographic-specific content.   identifying search terms   once the relevant search fields have been identified, word frequency analysis can be conducted to find candidate geographic setting search terms for the filter. the authors used the writewords (2020) word and phrase counter tool to conduct the frequency analysis for the nice uk filters (ayiku et al., 2017, 2019). writewords (2020) is available for free online. other free online counters are available such as databasic (bhargava & d'ignazio, 2020) and commercial counters can be used too.   for the nice uk filters, the authors copied the content contained in each relevant search field from excel and pasted it into writewords (2020) one field at a time. the frequency of single words up to phrases containing four words was then recorded for each field. next, the high frequency words and phrases used to describe uk settings were examined. the most frequent uk settings identified from the development set references were:   ·         countries ·         nationalities ·         cities ·         uk national health service (nhs)   4b. constructing the filter   a geographic filter can be drafted once the relevant search fields and geographic setting terms have been identified. save the draft filter in the database account so that it can be easily re-run to test its retrieval ability.   as an example of a geographic filter structure, an outline of the nice uk filters is provided in figure 2. the full nice uk filters for medline and embase can be found in published journal articles (ayiku et al., 2017, 2019) and in the intertasc information specialists’ sub‐group (issg) search filter resource section on geographic search filters (glanville et al., 2020). figure 2 example structure for a geographic filter to retrieve evidence about a country. 4c. internal validity test   when the geographic filter is drafted, the next step is to test how successfully it retrieves the references that were used to create it. this is known as an “internal validity” test (jenkins, 2004). to do this, run the saved search strategy for the development set references. next, run the saved search strategy for the draft filter and apply it to the development set search strategy using the and boolean operator. for example, the search strategy structure used to test the retrieval ability of the nice uk filters was as follows:   1.       langford i (author) and “the potential effects of climate change on winter mortality in england and wales” (title) and 1995 (year) 2.       chahal r (author) and “a study of the morbidity, mortality and long-term survival following radical cystectomy and radical radiotherapy in the treatment of invasive bladder cancer in yorkshire” (title) and 2003 (year) 3.       saka o (author) and “cost of stroke in the united kingdom” (title) and 2009 (year) 4.       etc… 5.       1 or 2 or 3 or 4… 6.       draft uk geographic search filter 7.       5 and 6   it is unlikely that the draft filter will retrieve all of the development set references because it is rare for search filters to have a 100% retrieval rate. for instance, some references will contain no details about their geographic setting in their database records (ayiku et al., 2017, 2019).   if the draft filter retrieves all of the references in the development set, it can be validated using the instructions in step five. if the draft filter does not retrieve all of the references, the reasons why the missing references were not retrieved must be investigated. carefully look through the database records for the missing references to see if any geographic setting details are contained within them. consider making modifications to the filter to retrieve missing references that contain setting details for the region. ensure that you record any changes you make to the filter and provide explanations about why the changes were made. also make a record of any references that cannot retrieved by the draft filter and explain why the references were not retrieved. save the final version of the filter in the database account so that it can be easily re-run to validate the filter (see step five).   step 5. validate filter   validation is the final process for filter development. the validation set contains references that have not been used previously to develop the filter and it is used to assess the filter’s “external validity” (glanville et al., 2008). validating filters using an independent set of references provides an indication of how well filters perform in retrieving relevant evidence in any search (glanville et al., 2008).   to validate the filter, run the saved search strategy for the validation set references. next, run the saved search strategy for the final version of the filter. apply the filter to the validation set search strategy using the and boolean operator following the same example structure shown above in step four.   the filter’s recall can now be calculated. “recall”, also known as “sensitivity”, is used to measure a filter’s ability to retrieve a set of known relevant references and it is calculated as follows (jenkins, 2004):   ·         number of relevant records retrieved by filter/total number of relevant records (× 100 to express as a percentage)   the term “relative recall” is more accurate than “recall” when the relative recall approach has been used to identify references pooled from multiple evidence based sources for the validation set (sampson et al., 2006), however, in practice both terms are used.   it is unlikely that the filter will achieve 100% recall and the reasons why missing references were not retrieved should be investigated and recorded. there is no standard definition of “high” recall. however, 90% or above has been used as a threshold in previous studies (beynon et al., 2013). the existing geographic filters performed as follows:   ·         spain filter: pubmed: 88.1% recall (valderas et al., 2006) ·         africa filters: pubmed: 74% recall, embase: 73% recall (pienaar et al., 2011) ·         nice uk filters: medline uk filter: 99.5% recall, embase uk filter: 99.8% recall (for references with uk identifiers) (ayiku et al., 2017, 2019)   note that no changes can be made to the filter once its recall against the validation set has been calculated. another validation set containing at least 100 previously unused references will need to be created if filter modifications are required to increase recall. in this case, the former validation set becomes a “test set” that was used to inform the filter’s development.   tips for creating filters   seek advice   it may be helpful to seek advice from a professional peer with relevant experience if needed.   limiting retrieval of irrelevant results   some setting names for the geographic region of the filter may be found elsewhere in the world. using the not boolean operator can help to minimize the retrieval of irrelevant geographic references. for example, the nice uk filters included the following strategy to help minimize the retrieval of irrelevant geographic references about the us: york not “new york” (ayiku et al., 2017, 2019).   language variations   if relevant, use language variations for the geographic region. for instance, the spain filter included the following language variations for the country: spain, espagne, espana, and spagna (valderas et al., 2006).   retrieving references by language   consider retrieving references by language if the filter is for a region with a language that is uncommon in other geographic locations. the search strategy to retrieve references by language is: “language.lg” for ovid databases or “language.la” for pubmed (e.g., welsh.lg or welsh.la). add the language search strategy to the rest of the filter using the or boolean operator.   share the filter   the filter should be published along with the accompanying filter development processes to make it widely available. it will be added to the issg search filter resource section on geographic search filters when it is published which will increase its dissemination (glanville et al., 2020). in addition, the filter could be promoted at conferences and on social media.   acknowledge limitations   no filter is perfect, it is unlikely that the filter will achieve 100% recall. make sure to explain why the filter does not retrieve certain geographic references so that users understand its limitations.   keep the filter up to date   make sure that the filter is kept updated with any changes to the geographic setting terms. the updated filter may not be validated but the original recall level can still be considered as a baseline for this type of change.   conclusion   geographic search filters enable effective and efficient systematic literature searches for topics with a geographic focus. there are currently only three validated filters identified in the published literature for spain, africa and the uk (glanville et al., 2020). the authors hope that this commentary has increased awareness of the existing filters and encourages the creation of new geographic filters for additional places in the world.   references   ayiku, l., & finnegan, a. (2019). op23 smart searches for context-sensitive topics: geographic search filters. international journal of technology assessment in health care, 35(s1), 5. https://doi.org/10.1017/s0266462319000953   ayiku, l., levay, p., hudson, t., craven, j., barrett, e., finnegan, a., & adams, r. (2017). the medline uk filter: development and validation of a geographic search filter to retrieve research about the uk from ovid medline. health information and libraries journal, 34(3), 200–216. https://doi.org/10.1111/hir.12187   ayiku, l., levay, p., hudson, t., craven, j., finnegan, a., adams, r., & barrett, e. (2019). the embase uk filter: validation of a geographic search filter to retrieve research about the uk from ovid embase. health information and libraries journal, 36(2), 121-133. https://doi.org/10.1111/hir.12252   beynon, r., leeflang, m. m., mcdonald, s., eisinga, a., mitchell, r. l., whiting, p., & glanville, j. m. (2013). search strategies to identify diagnostic accuracy studies in medline and embase. cochrane database of systematic reviews (9), mr000022. https://doi.org/10.1002/14651858.mr000022.pub3   bhargava, r., & d'ignazio, c. (2020). databasic word counter. emerson college and university of massachusetts, massachusetts, usa. retrieved from  https://databasic.io/en/wordcounter/   damarell, r. a., may, n., hammond, s., sladek r. m., & tieman, j. j. (2019). topic search filters: a systematic scoping review. health information and libraries journal, 36(1), 4-40. https://doi.org/10.1111/hir.12244   glanville, j., bayliss, s., booth, a., dunda, y., fernandes, h., fleeman, n. d., foster, l., fraser, c., fry-smith, a., golder, s., lefebvre, c., miller, c., paisley, s., payne, l., price, a., welch, k. (2008). so many filters, so little time: the development of a search filter appraisal checklist. journal of the medical library association, 96(4), 356–361. https://doi.org/10.3163/1536-5050.96.4.011.   glanville, j., lefebvre, c., & wright, k. (2020). the intertasc information specialists’ sub‐group search filter resource: filters to find studies of geographic locations. issg filters resource. university of york, york, uk. retrieved from https://sites.google.com/a/york.ac.uk/issg-search-filters-resource/other-filters/filters-to-find-studies-of-geographic-locations   hausner e., waffenschmidt, s., kaiser, t., & simon, m. (2012). routine development of objectively derived search strategies. systematic reviews, 1(19). https://doi.org/10.1186/2046-4053-1-19   jenkins, m. (2004). evaluation of methodological search filters – a review. health information and libraries journal, 21(3), 148–163. https://doi.org/10.1111/j.1471-1842.2004.00511.x   pienaar, e., grobler, l., busgeeth, k., eisinga, a., & siegfried, n. (2011). developing a geographic search filter to identify randomised controlled trials in africa: finding the optimal balance between sensitivity and precision. health information and libraries journal, 28(3), 210–215. https://doi.org/10.1111/j.1471-1842.2011.00936.x   randomness and integrity services ltd. (2020). random.org. retrieved from https://www.random.org/   sampson, m., zhang, l., morrison, a., barrowman, n. j., clifford, t. j., platt, r. w., klassen, t. p., & moher, d. (2006). an alternative to the hand searching gold standard: validating methodological search filters using relative recall. bmc medical research methodology, 6(33). https://doi.org/10.1186/1471-2288-6-33   valderas, j., mendivil, j., parada, a., losada‐yáñez, m. & alonso, j. (2006). development of a geographic filter for pubmed to identify studies performed in spain. revista española de cardiología, 59(12), 1244–1251. https://doi.org/10.1016/s1885-5857(07)60080-2   writewords. (2020). writewords frequency counters. retrieved from http://www.writewords.org.uk/word_count.asp   evidence based library and information practice 2012, 7.2 87 evidence based library and information practice evidence summary low response rate and other factors render academic health science library system study ungeneralizable a review of: folb, b. l., wessel, c. b., & czechowski, l. j. (2011). clinical and academic use of electronic and print books: the health sciences library system e-book study at the university of pittsburgh. journal of the medical library association, 99(3), 218-228. doi:10.3163/1536-5050.99.3.009 reviewed by: maria melssen medical librarian ohio, united states of america email: mariamelssen@gmail.com received: 31 jan. 2012 accepted: 7 apr. 2012 2012 melssen. this is an open access article distributed under the terms of the creative commons‐ attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to determine the factors, barriers and facilitators, preference, and intended use of e-book compared to print book usage by all patrons in a health science library system, which serves a university with health science degree programs and a hospital system. design – two online surveys. setting – university of pittsburgh health sciences library system, which includes the university of pittsburgh’s six schools of health sciences (medicine, dental medicine, nursing, pharmacy, public health, and rehabilitation) and the university of pittsburgh medical center hospitals and programs. subjects – all health sciences library system users, including faculty, researchers, clinicians, residents, fellows, employees, and students. methods – two versions of the survey were deployed in 2009 using opinio. there were 46 questions for the university of pittsburgh medical center (upmc) survey and 47 questions for the university of pittsburgh (pitt) survey. the surveys were pilot tested by health sciences library system (hsls) librarians and graduate students in a survey methods class. the survey was edited based on the feedback provided and received institutional review board approval as an exempt study. a total of 5,292 email addresses were randomly selected by spss from a pool of mailto:mariamelssen@gmail.com evidence based library and information practice 2012, 7.2 88 9,472 upmc and pitt patrons registered with a hsls remote access password; 2,684 patrons from upmc and 2,608 patrons from pitt were selected. hsls librarians were excluded from the survey. participants were emailed a link to the survey in march of 2009. three email reminders were sent at five day intervals. data was collected for 22 days and exported from opinio to spss statistics software. survey results were analyzed using basic descriptive statistics and cross-tabulations. main results – of the 5,292 emails sent, 979 surveys were submitted and 871 were completed fully. the 108 partially completed the surveys were analyzed using pair wise deletion. all hsls user groups were represented and all rated their confidence in computer skills high. the mean age of respondents was 39.9 with the majority of respondents being female. of the 871 completed surveys, over half (55.4%) of the respondents reported using hsls e-books: 66.7% men and 54.9% women. hsls e-books were used for in-depth reading by 53.4% of men and 36.8% of women. at upmc, 70% of attending physicians, interns, residents, fellows, and pitt postdoctoral/fellows use hsls e-books. the primary use of the e-books was for clinical care, by 75.3% of attending physicians; 86% of interns, residents, and fellows; and 38.9% of nurses. hsls e-books are also used by 61.8% of respiratory care and physical therapists, 28.6% of administrators, and 56.8% of researchers. at pitt, 73% of postdoctoral students or fellows and 64.7% of faculty used hsls e-books. the primary use of the e-books was to support research. 76.5% of postdoctoral students and fellows and 54.1% of faculty used e-books for this purpose. only 21.3% of faculty assigned e-books for class readings. though 14% of undergraduate and 33.5% of medical students responded that they had been assigned readings from e-books, 51% of undergraduates and 62.1% of graduate and medical students used an e-book to complete an assignment. over half (65.5%) of respondents saw information about hsls e-books on the hsls website and 55.4% of respondents used an hsls e-book. when using an e-book, 56.6% look up brief, factual information while 41.9% use e-books for in-depth study. uses of hsls e-book search tools were rated: the federated full text search tool was used by 67.2% of respondents and 74.3% of those who use this tool rated it as moderately to extremely useful. google books and the library catalog were also rated moderately to extremely useful by respondents. the catalog received the lowest rating of the hsls e-book search tools. more respondents (95.4%) use the library’s website than come to the physical library (63.8%); however, 66.9% say they use both the website and physical library. of the 63.8% of respondents who came to one of the hsls libraries, 67.2% borrowed or used a hsls print book. when using a book at the library, 23.4% only use print, 14.8% only use e-books, 44.7% use both, and 17.1% use neither. fewer respondents (46.4%) agreed or completely agreed they could locate an e-book compared to those who agreed or completely agreed they could locate a print book (66.7%). nearly half (45.3%) agreed that both hsls ebooks and print books were accessible where they needed to use them; however, only 27.9% agreed or completely agreed that they had time to go to the library and use a print book when they needed it. the closer a respondent worked to the library the more likely they used the physical library. those also within one block of the library were greater users of hsls e-books (67% of respondents) than those who worked more than two blocks from the library (52.3% of respondents). when respondents did come to the library, 84.3% used a hsls print book in the past year and 64.7% used an hsls e-book. of the respondents who did not have time to come to the library, 55.3% used a hsls print book and 55.3% used a hsls e-book. when using e-books, respondents preferred such features as printing, saving, and searching over features such as bookmarking, evidence based library and information practice 2012, 7.2 89 highlighting, and annotating. respondents also preferred e-books for general reference and pharmaceutical reference, and print books for textbooks and handbooks. a finding of significance is that “those preferring print were more flexible about using e-books than those preferring e-books were about using print” (p. 224). conclusion – hsls e-book use varied depending on the respondent’s role at their institution (e.g., clinical physician, researcher) and type of book (e.g., reference book) they used. the heaviest hsls e-book users were students, postdoctoral fellows, researchers, and clinical physicians. respondents who used hsls e-books most often were also those who used print books most often, and respondents within one block of the library were some of the heaviest hsls e-book users. respondents felt that reference and pharmaceutical books were more suitable as e-books. also of note was that though faculty were not using ebooks heavily for assigned readings, students were using hsls e-books to complete assignments. the greatest drive to choosing between a print and e-book was the respondent’s information need and which book format was most convenient to access at that time. respondents were flexible in their use of print books and ebooks: respondents “would be willing to use a less preferred format if it were more convenient at the time of need” (p.226). in light of respondents’ flexibility between e-book and print book usage, the authors suggest that collection development librarians could reduce the duplication of book formats. regarding awareness of e-books, survey results from this study were comparable to that of other studies. also, the respondent’s comments indicate that the survey itself prompted e-book awareness: respondents felt that more advertising of e-books should be done. such responses show that passive advertisement of e-books though the library’s catalog and on the website are not enough. ebooks should be advertised during library instructional sessions. respondents also prefer web access to hsls ebooks as well as the hsls federated e-book search rather than to access hsls e-books from the library catalog. the authors’ recommendation is to make sure users can easily access e-book catalog records through the web in order to best facilitate patrons’ use of e-books. despite the conclusions that were drawn, there were several limitations of this study. though the sample size was large enough and all hsls users were included, the response rate was very low. bias could be an issue as well: nonresponse bias as well as an overestimation of the number of hsls e-book users could be contributing factors to the low response rate. in addition to the small sample size and possible bias, the lack of completed responses (11%) was also a concern. finally, respondents expressed confusion over how “e-books” were defined in the survey. because of these issues, results of this survey may not be generalizable to other libraries. commentary this study offers valuable insights into the ebook and print book usage of health science library system users; however, there are various aspects of the study that are cause for concern. first, there are various writing errors within the study. “upmc” is never defined in the article; a search on the upmc web site followed by a phone call to upmc was required to determine that upmc was the “university of pittsburgh medical center.” there is also no consistency in describing “university of pittsburgh;” it is referred to as either “university” or “pitt.” there is also inconsistency in the respondent type “physician;” the labels “attending physician,” “clinical physician” and “physician” are used interchangeably. the methodology of the study is also problematic. the authors do not discuss why there are different number in questions for upmc and pitt. the survey questions were also tested on graduate students as opposed to a sample of library users the study was evidence based library and information practice 2012, 7.2 90 targeting. perhaps if the questions were tested on such a sample group the issue of confusion over the definition of e-books could have been avoided. though the authors acknowledge the study’s low response rate, it is of a major concern: the authors estimated that they would have a 25% rate, but only 16.5% of their sample completed the survey. the authors believed that an email cover letter from the director and three email reminders to be enough of an incentive for respondents to complete the survey; however, it clearly was not enough. a unique feature of this study was the variety of users they surveyed. though surveying all of these users may provide the institution with a comprehensive view of how e-books are being used, this also causes the study to become ungeneralizable. few other institutions may have the same user population as pitt. also, there is an overwhelming amount of data. it is a challenge to digest this much information in a single article. it would have been beneficial to break this study up over several articles or to only survey one patron population at a time. despite these drawbacks, this study does provide valuable data on how different health science patrons use e-books compared to print books. large academic health center collection development librarians may find this study’s findings beneficial in making their own determinations for e-book vs. print book purchases. review article   mixed methods research in library and information science: a methodological review   richard hayman associate professor & digital initiatives librarian mount royal university calgary, alberta, canada email: rhayman@mtroyal.ca   erika e. smith assistant professor & faculty development consultant academic development centre mount royal university calgary, alberta, canada email: eesmith@mtroyal.ca   received: 12 sept. 2019                                                                  accepted: 6 jan. 2020      2020 hayman and smith. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   data availability: hayman, r., & smith, e. e. (2019). mixed methods in library and information science bibliographic records (2008-2018) [data set]. edmonton, canada: ual dataverse, ver. 1.0, https://doi.org/10.7939/dvn/cxuw6a   doi: 10.18438/eblip29648     abstract   objective to review mixed methods research trends in the field of library and information science (lis). in particular, we examine the extent to which research about or using mixed methods has been occurring in library and information science over the past decade (2008-2018), and how much of that mixed methods research is done in health contexts.   methods we conducted a methodological review and analysis of mixed methods research (mmr) in lis for published articles indexed in lista and web of science. after deduplication and verification for inclusion, we coded 417 articles to identify contributions using or about mmr. given the connections between evidence based practice in health and lis, we also identified whether articles about or using mixed methods were health-focused.   results we found mmr to be a tiny proportion (less than 0.5%) of the overall lis research literature. in terms of observable trends, while contributions about mmr remain fairly static, there has been an increase in articles using mixed methods. of the 417 included articles, 373 (89.5%) primarily used mixed methods and 44 (10.5%) were primarily about mmr. results also demonstrated that health-related research both using and about mixed methods has a strong presence in the lis literature, with 136 published articles (32.6% of the total).   conclusion confirming findings of prior analyses of research methods in lis, our methodological review shows current opportunities to adopt and expand the use of mixed methods research processes. further contributions about mixed methods research, and ideally connecting research and practice in lis, are needed. despite the small proportion of mmr in lis research, there is an observable increase in the number of publications using mixed methods during this timeframe. the lis research community can promote additional growth by leveraging this momentum around using mixed methods, and look to translate lessons learned about mixed methods research and practice in health contexts to other lis settings. recommendations include developing educational opportunities and learning resources that facilitate wider engagement with mmr in lis contexts.     introduction   for those interested in evidence based practice (ebp), there is an increasing array of research methods, strategies, and approaches available today that can be leveraged to foster praxis. various analyses of the literature point to untapped opportunities for researchers and practitioners in the field of library and information science (lis) to expand the range of research methods and methodologies utilized, including mixed methods approaches (aytac & slutsky, 2014; chu, 2015; gauchi risso, 2016; ullah & ameen, 2018). in prompting those in lis to ask “are we there yet?” regarding adoption of mixed methods research, fidel’s (2008) analysis demonstrated that mmr was not commonly used or discussed in lis, concluding that increased awareness would be advantageous to the field. we revisit this overarching question regarding whether lis has been using or discussing mixed methods during the decade following fidel’s work. as researcher-practitioners who have realized the value of using mixed methods research (mmr) for scholarship and evidence based practice in our own contexts, we see benefits to an evidence based discussion of current trends and the potential value of mmr. with the goal of exploring ways to expand engagement with mixed methods research in lis contexts in mind, the purpose of this article is to take stock of mixed methods research trends and issues through a broad methodological review of the lis literature over a ten-year span.   aims   to support mixed methods practice plano clark and ivankova (2016) argued that there is great value in consulting literature analyses about the status of mixed methods in the context of a particular research community, especially in the form of methodological reviews and discipline-based discussions. with this in mind, we conducted a methodological review and analysis of mixed methods research in lis published over the past decade (2008-2018) to address the following research question:   rq1: to what extent is research about or using mixed methods occurring in library and information science?   additionally, given the established connections between evidence based practice and evidence based medicine, and their intersection in health librarianship, we also explored the following related research question:   rq2: over the same decade, what literature about or using mmr in library and information science has occurred within health contexts?   in light of these research questions, our approach specifically sought to capture the breadth of mixed methods research occurring over time and across a considerable, representative dataset. adopting this broader approach enabled us to compare findings from other lis research methods analyses, and using a larger sample than if we had focused on a particular subset of journals.   to encourage further development and application of mmr in ways that are clear and relevant for this disciplinary context we outline recommendations connected to lis practice. our goal is to promote further consideration of mixed methods research in ways that can beneficially inform new ways of collecting, using, and integrating evidence in lis contexts.   defining mixed methods research   there are several definitions of mixed method research, but a common component of most definitions is that researchers must deliberately combine two or more (usually qualitative and quantitative) research methods in a single study to provide the most comprehensive means of addressing the research problem and questions at hand. recognizing mixed methods as a research process, creswell (2008) defined mmr thus:   a broad umbrella term encompassing perspectives that see it as a research method of data collection and analysis, a methodology that spans the process of research from philosophical assumptions to interpretations, a philosophy of research, and a set of procedures used within existing research designs such as case studies, experiments, and narrative projects. (p. 2)   mixing methods increases our ways of viewing issues, providing more evidence than we would using a single method. in their seminal work on mmr, johnson, onwuegbuzie, and turner (2007) argued that mmr was increasingly being understood as a third research paradigm alongside existing qualitative and quantitative research paradigms, providing opportunities “to consider multiple viewpoints, perspectives, positions, and standpoints” (p.113). mmr helps bridge the divide between quantitative and qualitative research (creswell & plano-clark, 2018), and many researchers relate these aspects of mmr to triangulation, a way of cross-validating information from several sources (gorman & clayton, 2005; connaway & radford, 2017; wilson, 2014).   methods   to examine current research trends surrounding mixed methods, we integrated key strategies outlined by mmr experts plano clark and ivankova (2016) for methodological reviews. they reinforce the value of such work for research and practice, acknowledging the “long history of scholars conducting disciplinary-based methodological reviews in the field of mixed methods research” (p. 256). we follow their recommendation to report the procedures used for identifying the sample of published mixed methods research, and analyze specific dimensions and features reported within those publications to provide insights into patterns and trends, such as the prevalence rate of mixed methods. our methodological review also draws on useful scoping and mapping review techniques (grant & booth, 2009) to illustrate issues over the course of a decade through figures and diagrams.   in examining evidence from the literature in ways that are relevant for those in the field, a methodological review should outline strengths and weaknesses and how these may “constrain or open up opportunities for learning” (elsevier, n.d., p. 4). huynh, hatton-bowers, and smith (2019) remind us that conducting a methodological review within a disciplinary context helps identify trends and opportunities for using and improving mmr practices. finally, onwuegbuzie, leech, and collins (2011) noted that a methodological review can be an end in itself, highlighting the benefit of such reviews for informing practice and understanding the topic being explored.   sources and search strategies   our search focused on two primary information resources that index research from lis contexts: library, information science & technology abstracts with full text (lista, from ebsco), and web of science core collection (wos, from clarivate analytics). we selected these based on their disciplinary coverage and the fact that both were accessible through our current institutional subscriptions. to be exhaustive with our wos search we included six main indices from the wos core collection: science citation index expanded; social sciences citation index, arts & humanities citation index; conference proceedings citation index-science; conference proceedings citation index-social science & humanities; and emerging sources citation index.   to maintain our focus on the use of mmr in current research while also ensuring feasibility and manageability of the project, we restricted all searches to english-language journal articles published from 2008-2018. we identified and used a variety of phrases to describe our primary topic based on our own knowledge of the subject and research being explored. these phrases reflect the popular terminology used extensively in existing mmr literature, and in many cases echoed the language and labels that authors had used in their studies. test searches allowed us to refine this list, leading to the search strategies outlined below. search strategy for lista as a discipline-specific database, lista was our starting place to test keywords/phrases and to focus on lis-related literature.   search strategy for wos as a large, interdisciplinary index, we relied on built-in tools for limiting to only those publications that belong to lis. since wos has a specific subject category for “information science & library science” we used this for our first search before searching for keywords/phrases.   together lista and wos revealed 636 results for further analysis. figure 1 is a high-level illustration of our process starting from the point when these results were combined, deduplicated, and then checked against include/exclude criteria. only the final 417 included articles were subsequently coded.   table 1 searches conducted within lista search terms and limits results s1 (no keyword/phrase used to find all results) limiters: ●        publication date: 2008-01-01 to 2018-12-31 ●        publication type: academic journal ●        document type: article ●        language: english 98,343 s2 (de "mixed methods research") or "mixed methods research" or mmr or "mixed methodology" or "mixed research" or "mixed methods sampling" or "mixed design" or "mixed method design" or "combined methods" or "mixed methods approach" or "mixed methods study" 504 s3 s1 and s2 354   table 2 searches conducted within wos search terms and limits results s1 (no keyword/phrase used to find all results) wc=(information science & library science) and language: (english) and document types: (article) indexes=sci-expanded, ssci, a&hci, cpci-s, cpci-ssh, esci timespan=2008-2018 51,873 s2 ts="mixed methods research" or all="mixed methods research" or all=mmr or all="mixed methodology" or all="mixed research" or all="mixed methods sampling" or all="mixed design" or all="mixed method design" or all="combined methods" or all="mixed methods approach" or all="mixed methods study"   18,518 s3 s1 and s2 282   figure 1 process for methodological review starting with records captured from lista and wos.   deduplication   we imported the 636 citations into citation management software zotero (https://www.zotero.org/), which includes a built-in deduplication function that compares several metadata fields and flags suspected matches. we reviewed each flagged match before removing items that were duplicates, then reviewed the full list again to manually remove additional duplicates that were not flagged as part of the automatic deduplication. the remaining 473 items were sent to the include/exclude process.   inclusion and exclusion criteria   both authors reviewed the 473 potential articles remaining after deduplication to determine whether they met our inclusion criteria. we examined each study for the following:   ●        a research article published in a journal; ●        situated in library or information science contexts, as determined by the subject matter or source publication; and ●        evidence that the study involved mmr processes, whether ○        reporting on an original study using mmr; or ○        discussing mmr as part of a larger methodological discussion; or ○        a protocol study wherein mmr was part of the proposal and the mmr process was evident.   while titles and abstracts typically served as primary sources of information to determine the mmr processes involved, in several cases these provided insufficient evidence that the study was in fact mmr-based. in such cases we then examined the full text, focusing on the methods section, which proved a reliable way to determine each study’s mmr status. we used traditional subscription databases, open access resources, and third-party tools (e.g., researchgate) to find full-text versions.   in the rare circumstance where we were unable to locate full text, we decided to err on the side of caution. in these very few cases we based our decision to include or exclude using the available abstract in tandem with their peer-reviewed status. if the abstract described these works as mixed-methods, and reviewers and journal editors had deemed them fit to be published as such, then we would include these few publications in our sample.   while screening articles for inclusion or exclusion, we identified several articles where authors indicated their study used mmr, but upon reading the article it was clear that they reported on only a single phase or method. for example, we found several studies using a survey or questionnaire with closedand open-ended questions that described themselves as mixed methods. however, creswell and hirose (2019) mark the distinction between survey methodologies, which can include openand close-ended questions, and mixed methods research proper, which may involve a survey or questionnaire but ultimately requires a combination and integration of multiple research approaches. based on this definition, we excluded survey-only mmr reports from our dataset. similarly, since intentionally mixing methods is an essential characteristic of mmr, we excluded studies that merely reported on a single stage of a larger mmr project (e.g., only reporting the qualitative or quantitative phase) when they did not situate or report that data within the wider context, methods, and findings of the rest of the mmr study.   we also excluded obvious false hits, such as a few articles that used our mmr acronym keyword for something other than mixed methods research (e.g., articles discussing vaccines for measles, mumps, and rubella). since published research articles were our focus, we removed results that had been tagged as articles in their source database but were merely conference abstracts or grey literature reports. the include/exclude process resulted in 417 articles that were sent for coding.   coding   since our approach specifically sought to capture the breadth of mixed methods occurring in this dataset, both researchers were in agreement that coding of the remaining articles should be sufficiently high-level in order to support feasibility of this wide scope of research. we aimed to generate a general picture and position of mmr in lis research over the last decade, rather than focus on the specifics of how mmr manifests. both authors reviewed the 417 included articles and independently coded each according to whether it was a study that used mmr or whether it was about or discussing mmr. within these two main categories we also identified those that involved medicalor health-related research. both researchers reviewed and discussed these categorizations to ensure consensus.   results   publication sources   we briefly explored the source publications for these 417 articles. concerning rq1, mmr articles appeared in 121 different publications representing the breadth and depth of lis research over the past decade. the top five publications and the number of articles from each are in table 3.   table 3 top five publication sources by number of mmr articles published publication title no. of articles journal of medical internet research 62 qualitative health research 20 information research 10 journal of the american medical informatics association 10 journal of the association for information science & technology (formerly journal of the american society for information science & technology) 10   we accounted for identifiable journal title changes that occurred during the decade in question and standardized for slight differences in journal titles within citation information (e.g., evidence based library and information practice vs. evidence based library & information practice). a full list of publication titles and article counts is available as appendix a.   prevalence of mmr   concerning rq1 and the extent that research using or about mmr is occurring in lis, the main results are reported in table 4. we identified 373 (89.5%) articles that primarily used mixed methods as part of the research process, and another 44 (10.5%) articles were that were primarily about mmr and related methodological discussions. addressing rq2, nearly one-third (n=118, 31.6%) of the 373 articles using mixed methods processes were situated in a health context. similarly, more than one-third (n=18, 40.9%) of the 44 articles about mixed methods or research methodologies occurred in health contexts. when combined, these health-focused articles comprised 136 published articles (32.6% of the total) related to health or medical sciences within the overarching lis literature.   table 4 number and percentage of each article type type no. of articles % of total all "using" articles 373 89.45%   articles using mmr 255 61.15%   articles using mmr in health contexts 118 28.30% all "about" articles 44 10.55%   articles about mmr 26 6.24%   articles about mmr in health contexts 18 4.32%   mmr over time   we tracked the number of articles published per year to look for developmental trends over the decade (figure 2). this distribution demonstrates an increasing trend in the use and discussion of mixed methods processes within lis research. we also combined the articles using mmr with those using mmr in health contexts, to compare them against all of the articles about mmr combined with those about mmr in health contexts. this comparison, shown in figure 3, reveals that the trend in research about mmr is fairly static, and that it is the studies using mmr that drive the overall increasing trend.   figure 2 distribution by publication year for all included articles (n=417).   figure 3 distribution by publication year comparing articles using mmr (n=373) and articles about mmr (n=44).   discussion   mmr trends and patterns   as shown in our search strategies for lista and wos we could isolate the total results for lis generally before including our mmr-related phrases and keywords. we found that proportionally mmr makes up a tiny fraction of the corpus of lis research literature. consider that:   ●        the 354 results found in lista represent 0.36% of the 98,343 total lista results when searching with search limiters, but not using keywords; and ●        the 282 results found in wos represent 0.52% of the 51,873 total wos results when searching with search limiters, but not using keywords.   while this is an imprecise measure, our use of disciplinary and other search limiters (i.e., date range, language, document type, publication type) together help us significantly refine the corpus of available, published lis literature. these figures provide a compelling case for identifying an overall lack of mmr processes within lis research.   despite the small number of mmr contributions overall, the upward trend does show some growth in the use of mmr within lis (see figure 2). we see this as a promising area for future research. however, in contrast with the growth seen via the increase in the number of articles where mmr was used, we found that much fewer articles discussed mixed methods as a research process, and those that did most often occurred in health contexts. the prevalence of articles about mmr has remained relatively static (see figure 3), an indication that, in addition to fostering momentum around expanded use of mmr, there are likely opportunities for further research contributions aimed at discussing mixed methods processes and related meta-research aspects within lis. as an example, venkatesh, brown, and bala’s (2013) guidelines for conducting mmr in information systems appears to have met, or created, an appetite for contributions about mixed methods, with their article having received nearly 2,000 citations according to google scholar at the time this article was drafted. our methodological review indicates that further scholarly contributions that intentionally and explicitly connect mmr with lis would be valuable, and likely necessary.   focusing again on health contexts (rq2), our analysis shows that health-related mixed methods research appears to be prominent, with 31.6% of articles using mmr and 40.9% of articles about mixed methods or methodologies being situated in a health context. within the list of top ten publications containing mmr articles there is a substantial representation of mmr with a health focus. within the top five journals with mmr publications (see table 3), three of these are health-focused, and collectively, these three journals published 92 (22.1%) of the articles we examined. since ebp in lis has known connections to evidence based medicine (ebm) and ebp in health settings, perhaps this is unsurprising. however, these results do underscore that health-related research has a strong presence in the lis literature that either uses or is about mixed methods.   researchers and practitioners in lis who are interested in mmr may look to health-related research to determine practices that could help bolster mmr in other topic areas or contexts. for example, o’cathain, murphy, and nicholl (2008) identified that collaboration is often an important part of mixed methods research, emphasizing that mmr in health settings often involves large interdisciplinary or multidisciplinary teams bringing together people with a variety of expertise. it could be the case that engagement with mmr is occurring in health contexts to a large extent due to the involvement of experts within and beyond lis on these larger interdisciplinary teams. further exploration of the role of lis researchers and other information professionals on such mmr teams could provide insights into effective research practices and other lessons learned that could help extend mixed methods approaches (and mmr in ebp) from these health-focused research projects to the broader lis research community.   connections to the research methods literature   to place our findings in the context of wider work on research methods we consulted the lis literature generally, seeking connections between our methodological review of mmr to overall research trends in lis via a discipline-focused discussion. the literature reveals that the discipline draws heavily on quantitative research approaches and surveys, though there are some signs that this could be changing. booth and brice (2004) found that “lis research typically utilizes designs of limited applicability, such as the user survey” (p. 91), while koufogiannakis, slater, and crumley’s (2004) content analysis of librarianship research found that descriptive research, mainly using a survey or questionnaire method, was the highest proportion of research published (p. 232). such points have been an ongoing refrain in the field of library and information science.   a decade later, turcios, agarwal, and watkins (2014) demonstrated that surveys were still the most popular research method used. similarly, aytac and slutsky’s (2014) analysis of lis research published from 2008-2012 found very few studies (1%) using multiple or combined method approaches. descriptive research and surveys remained the most popular in lis, with a majority of the studies employing solely quantitative analysis (69%). they predicted an ongoing growth in practitioner research, but cautioned against over-use of descriptive statistical analysis, instead encouraging practitioners “[to] seek out training in more advanced statistical methods” (p. 152).   these and other authors contend that, although there are a variety of research methodologies employed across lis, mixed methods approaches have not gained adequate recognition in the field. gauchi risso’s (2016) analysis of research methods from 1970-2010 similarly showed the prevalence of descriptive methodologies while stating that “lis needs new methodological developments, which should combine qualitative and quantitative approaches” (p. 74). likewise, ullah and ameen’s (2018) analysis demonstrated a predominance of quantitative, descriptive, and empirical methodologies in lis, with survey research still being the most widely used method.   all of this points to the need for lis researchers to give more consideration to, and increase their awareness of, other research approaches, including mixed methods. wilson (2013) advocated that those who support evidence based practice in lis would benefit from expanding their methodological approaches to include mixed methods, that approaching “a research question from multiple methodological perspectives in the same study will add a depth and breadth to the findings, and open up options for data collection and analysis” (p. 277).   studies exploring mmr approaches in lis continue to show low uptake of mmr. fidel (2008) found that only 5% of lis articles employed mixed methods and that “recognition of mmr by name or as a research method was absent from these articles and from the methodological literature in lis” (p. 265). this 5% figure was also reported by venkatesh et al. (2013). chu’s (2015) analysis found somewhat more variety in research methods used in the field, but the overall results underscored a need to continue expanding and developing research methods and their application to lis. chu concluded that “more efforts in the form of education, training, and advocacy are needed to promote the use of multiple methods” (p. 40).   research methods trends have implications for research of specific topics in lis today. matusiak’s (2017) analysis of methodologies in information behaviour research reflected the same themes of overall lis research practices, finding a majority studies were quantitative and used the common approaches (i.e., surveys). this shows a lack of growth in qualitative and mixed methods, ultimately reinforcing the need to increase awareness in lis about these research areas. moreover, in exploring the long-standing over-reliance on surveys and quantitative methodologies for research of technology-acceptance models and information systems (is) generally, wu (2012) emphasized that “a mixed methods approach combining both qualitative and quantitative techniques deserves more attention from is researchers” (p. 175). these trends from the wider research methods literature support our findings and confirm the underexplored opportunities for current lis researchers and practitioners to consider ways to expand their suite of approaches to adopt mmr (and other methods), increasing and enhancing strategies available for collecting, analysing, and using evidence in research and practice.   this is not to say that quantitative methodologies, descriptive research, and survey methods do not have their place, as we know they can be valuable. koufogiannakis et al. (2004) noted that lis is not unique in its tendency toward descriptive research, explaining that these approaches are likely ubiquitous in the field because “they are inexpensive and relatively easy to conduct, can be carried out in a short period of time, and the results are generally easy to analyze” (p. 233). common research approaches such as surveys likely continue to be popular within and beyond lis precisely because they offer an appropriate means of addressing particular research questions and problems.   ultimately, it is important for those conducting any research to consider whether and how a particular methodology and the related method(s) are aligned with, and appropriate for, understanding the phenomenon being explored. we see merit in mmr and join our voices with those arguing for increased adoption of mmr processes for ebp, yet we also recognize that mmr is not always the best or most appropriate choice. we strongly agree with scholars such as venkatesh et al. (2013) that “the decision to conduct mixed methods research should hinge on the research question, purpose, and context” (emphasis in original, p. 22). nevertheless, the findings from our methodological review of mmr, as well as the experiences described by researcher-practitioners such as ourselves, together inform our assertion that there are untapped opportunities and potential within lis to continue to go beyond traditional research approaches and increase the adoption of mmr processes. the field can benefit from engagement with mmr as a way to facilitate creative research and to rigorously combine approaches that can and will foster new forms of inquiry.   limitations   we have presented a broad methodological review examining mixed methods research within lis published from 2008 through 2018. we did not set out to employ the methods of a focused systematic review or meta-analysis, nor did we complete detailed quantitative or qualitative analyses of all of the included research artifacts, though future research employing these strategies would certainly be valuable. instead, our comprehensive “wide lens” approach addresses a gap in the extant literature and enables us to better position our findings alongside other methodological and disciplinary discussions. though we limited our searches to discipline-specific databases available through our current institutional subscriptions and note that both lista and wos provide significant coverage of lis research publications, these sources are not exhaustive. we acknowledge that other subscription products (e.g., library & information science abstracts), indexing services (e.g., google scholar), web search engines, and other tools may reveal additional published and grey literature that are relevant. also, we note that though it appears to be the most common terminology used today, the term mixed methods research is not universally used across the discipline. our search strategies focused on phrases rather than keywords to reflect the reality that mmr studies are sometimes mislabelled, and that this term may not appear on mixed methods work at all. this leads to the possibility that the growing trend identified in the results could be due to improvements toward consistent labelling strategies and terminology for mmr that are otherwise difficult to capture. like all research projects, this study may have benefitted from a larger research team, particularly for greater access to search indices and sources, increased scope including grey literature and conference materials, and additional experts participating in verification and consensus steps.   conclusions   our findings show that there are still untapped opportunities to extend scholarly contributions about and using mixed methods in research for library and information science contexts, and further confirms findings from the wider lis research methods literature. it is our hope that by outlining the following recommendations for developing mmr in ebp, we can encourage other researchers and practitioners in their developing their understanding of mixed methods processes, ideally embracing the benefits and opportunities that mmr offers.   recommendations   the methodological review of the mmr literature, as well as the authors’ own experiences conducting mixed methods research (hayman, smith, & storrs, 2019; smith, 2016), inform our outline of current needs and related recommendations to extend the development and application of mixed methods in lis. one recommendation is to encourage researchers and researcher-practitioners to undertake mmr when appropriate. we echo the calls from chu (2015) and others to promote further understanding of mmr through education, training, and advocacy. efforts to expand engagement with mmr through informal, non-formal, and formal education, including in graduate curriculum for library and information schools, could help to develop scholarship not just using mmr, but also about mixed methods processes and aspects of meta-research. intentionally integrating such pedagogical strategies aligns with crumley and koufogiannakis’ (2002) assertion that learning research skills is “essential for the growth of ebl [evidence based librarianship] within the entire profession” (p. 69). we note that this need to support the teaching and learning of research methods that includes mmr is not limited to lis. tashakkori and teddlie (2010) created their handbook on mmr for the social sciences broadly, and the field of education specifically, based on their practical experience working with graduate students on research methods training. they include a section with specific recommendations for pedagogy since this topic emerged “as one of the most difficult and controversial areas in mixed methods” (p. xi). given these complexities are widespread, lis could certainly take advantage of emerging and established educational developments through cross-disciplinary collaborations with other areas such as health and education. our findings showing the prevalence of mmr research in health-related contexts makes this point clear.   resources such as handbooks and toolkits from mixed methodologists can be helpful. a related recommendation is for graduate-level research methods courses to explore ways to enhance their coverage of mixed methods research. while some graduate programs may increasingly recognize the use of mmr in the research process, further scaffolding and building of expertise within and across disciplines, including those in lis, is warranted as a means of mitigating the challenges of mmr with the goal of realizing the benefits. as mmr evolves, the creation of courses and open resources that outline the theoretical, empirical, and practical considerations for mixed methods and its designs that can be easily accessed beyond the academy would also be beneficial in this regard. so would continuous professional development (cpd) opportunities on evolving research methods and mmr – for example, cpd connected to professional associations, conferences, and journals – that provide venues for lis researchers and practitioners at all levels who wish to reflect the principles of ebp and expand their methodological repertoire.   summary   in returning to fidel’s question of whether lis is “there yet” in engaging with mmr, we find that while there has been some growth in the use of mixed methods over the past decade, our methodological review demonstrates that mmr still represents only a small fraction of current lis literature. these findings indicate that further contributions about mmr processes and approaches are still needed, including those explicitly connecting research with practice. our results also show some momentum in mmr use, with an observable increase in the number of publications using mixed methods in lis during the decade in question, and that there is room for future research to explore this trend. health research contexts have a particularly strong presence in scholarship using and about mmr in lis, highlighting an opportunity to translate lessons learned about mmr and practice from health-focused areas into other lis settings. based on our findings, we recommend that the lis research community look to actively facilitate greater engagement with mixed methods, so that wider awareness and understanding of mmr can be fostered through educational development initiatives that build pedagogical strategies and resources for mmr, especially those supporting graduate programs and bridging academic and practitioner communities. enhancing ways to develop and apply mixed methods research in lis contexts in ways that take advantage of the affordances of mmr will benefit evidence based library and information practice.   data availability statement a dataset (hayman & smith, 2019) including the combined 636 citations exported from lista and wos is available in bibtex (.bib) format.   references   aytac, s., & slutsky, b. (2014). published librarian research, 2008 through 2012: analyses and perspectives. collaborative librarianship, 6(4), 147-159. retrieved from https://digitalcommons.du.edu/collaborativelibrarianship/vol6/iss4/6/   booth, a., & brice, a. (eds.). (2004). evidence-based practice for information professionals: a handbook. london, uk: facet.   chu, h. (2015). research methods in library and information science: a content analysis. library & information science research, 37(1), 36-41. https://doi.org/10.1016/j.lisr.2014.09.003   connaway, l. s., & radford, m. l. (2017). research methods in library and information science. santa barbara, ca: libraries unlimited.   creswell, j. w. (2008). mixed methods research. in l. m. given (ed.), the sage encyclopedia of qualitative research methods (pp. 527-530). https://doi.org/10.4135/9781412963909.n269   creswell, j. w., & hirose, m. (2019). mixed methods and survey research in family medicine and community health. family medicine and community health, 7(2), e000086. https://doi.org/10.1136/fmch-2018-000086   creswell, j. w., & plano clark, v. l. (2018). designing and conducting mixed methods research (3rd ed.). los angeles, ca: sage.   crumley, e., & koufogiannakis, d. (2002). developing evidence-based librarianship: practical steps for implementation. health information and libraries journal, 19(2), 61-70. https://doi.org/10.1046/j.1471-1842.2002.00372.x   elsevier. (n.d.). a guide for writing scholarly articles or reviews for the educational research review. retrieved from https://www.elsevier.com/__data/promis_misc/edurevreviewpaperwriting.pdf   fidel, r. (2008). are we there yet? mixed methods research in library and information science. library & information science research, 30(4), 265-272. https://doi.org/10.1016/j.lisr.2008.04.001   gauchi risso, v. (2016). research methods used in library and information science during the 1970-2010. new library world, 117(1/2), 74-93. https://doi.org/10.1108/nlw-08-2015-0055   gorman, g. e., & clayton, p. (2005). qualitative research for the information professional: a practical handbook (2nd ed.). london, uk: facet.   grant, m. j., & booth, a. (2009). a typology of reviews: an analysis of 14 review types and associated methodologies. health information & libraries journal, 26(2), 91-108. https://doi.org/10.1111/j.1471-1842.2009.00848.x hayman, r., & smith, e. e. (2019). mixed methods in library and information science bibliographic records (2008-2018) [data set]. edmonton, canada: ual dataverse, ver. 1.0, https://doi.org/10.7939/dvn/cxuw6a   hayman, r., smith, e. e., & storrs, h. (2019). information behaviour of undergraduate students using facebook confessions for educational purposes. information research, 24(3), paper rails1809. http://informationr.net/ir/24-3/rails/rails1809.html   huynh, t., hatton-bowers, h., & smith, m. h. (2019). a critical methodological review of mixed methods designs used in mindfulness research. mindfulness, 10(5), 786-798. https://doi.org/10.1007/s12671-018-1038-5   johnson, r. b., onwuegbuzie, a. j., & turner, l. a. (2007). toward a definition of mixed methods research. journal of mixed methods research, 1(2), 112-133. https://doi.org/10.1177/1558689806298224   koufogiannakis, d., slater, l., & crumley, e. (2004). a content analysis of librarianship research. journal of information science, 30(3), 227-239. https://doi.org/10.1177/0165551504044668   matusiak, k. k. (2017). studying information behavior of image users: an overview of research methodology in lis literature, 2004-2015. library & information science research, 1(39), 53-60. https://doi.org/10.1016/j.lisr.2017.01.008   o’cathain, a., murphy, e., & nicholl, j. (2008). multidisciplinary, interdisciplinary, or dysfunctional? team working in mixed-methods research. qualitative health research, 18(11), 1574-1585. https://doi.org/10.1177/1049732308325535   onwuegbuzie, a., leech, n. & collins, k. (2011). innovative qualitative data collection techniques for conducting literature reviews/research syntheses. in m. williams & w. p. vogt (eds.), the sage handbook of innovation in social research methods (pp. 182-204). london, uk: sage. https://doi.org/10.4135/9781446268261.n13   plano clark, v., & ivankova, n. (2016). mixed methods research: a guide to the field. thousand oaks, ca: sage. https://doi.org/10.4135/9781483398341   smith, e. e. (2016). “a real double-edged sword:” undergraduate perceptions of social media in their learning. computers & education, 103, 44-58. https://doi.org/10.1016/j.compedu.2016.09.009   tashakkori, a., & teddlie, c. (2010). the sage handbook of mixed methods in social & behavioural research (2nd ed.). los angeles, ca: sage. https://doi.org/10.4135/9781506335193   turcios, m. e., agarwal, n. k., & watkins, l. (2014). how much of library and information science literature qualifies as research? the journal of academic librarianship, 40(5), 473-479. https://doi.org/10.1016/j.acalib.2014.06.003   ullah, a., & ameen, k. (2018). account of methodologies and methods applied in lis research: a systematic review. library & information science research, 40(1), 53-60. https://doi.org/10.1016/j.lisr.2018.03.002   venkatesh, v., brown, s. a., & bala, h. (2013). bridging the qualitative-quantitative divide: guidelines for conducting mixed methods research in information systems. mis quarterly, 37(1), 21-54. https://doi.org/10.25300/misq/2013/37.1.02   wilson, v. (2013). research methods: mixed methods research. evidence based library and information practice, 8(2), 275-277. https://doi.org/10.18438/b8801m   wilson, v. (2014). research methods: triangulation. evidence based library and information practice, 9(1), 74-75. https://doi.org/10.18438/b8ww3x   wu, p. f. (2012). a mixed methods approach to technology acceptance research. journal of the association for information systems, 13(3), 172-187. https://doi.org/10.17705/1jais.00287   appendix a   list of publication titles and corresponding number of articles published that were included in the findings (n=417) publication title no. of articles journal of medical internet research 62 qualitative health research 20 information research 10 journal of the american medical informatics association 10 journal of the association for information science & technology (formerly journal of the american society for information science & technology) 10 aslib journal of information management (formerly aslib proceedings) 8 internet & higher education 8 journal of health communication 8 information technology & people 7 journal of librarianship & information science 7 mis quarterly 7 south african journal of information management 7 college & research libraries 6 education for information 6 evidence based library & information practice 6 first monday 6 information development 6 journal of documentation 6 qualitative & quantitative methods in libraries 6 electronic library 5 information, communication & society 5 journal of the association for information systems 5 mousaion 5 new review of academic librarianship 5 technology, pedagogy & education 5 health informatics journal 4 health information & libraries journal 4 information and learning science (formerly new library world) 4 international journal of information management 4 journal of enterprise information management 4 journal of the medical library association 4 library & information science research 4 library management 4 public library quarterly 4 south african journal of libraries & information science 4 african journal of library, archives & information science 3 canadian journal of information & library sciences 3 european journal of information systems 3 ifla journal 3 information & management 3 international information & library review 3 internet research 3 journal of education for library & information science 3 journal of information & knowledge management 3 journal of nonprofit & public sector marketing 3 journal of strategic information systems 3 library hi tech 3 library trends 3 online information review 3 portal: libraries & the academy 3 reference services review 3 research evaluation 3 telematics and informatics 3 transforming government: people process and policy 3 information processing & management 2 information systems journal 2 information technology for development 2 innovation 2 international journal of information & communication technology education 2 journal of academic librarianship 2 journal of the australian library & information association (formerly australian library journal) 2 journal of information science 2 journal of information technology & politics 2 journal of organizational & end user computing 2 journal of technology in human services 2 library review 2 libri: international journal of libraries & information services 2 new zealand library & information management journal 2 open learning 2 school libraries worldwide 2 social science computer review 2 accountability in research: policies & quality assurance 1 archival science 1 archives & manuscripts 1 behaviour & information technology 1 college & undergraduate libraries 1 communications in information literacy 1 community & junior college libraries 1 computers in the schools 1 data base for advances in information systems 1 data technologies and applications 1 electronic journal of knowledge management 1 hypothesis: journal of the research section of mla 1 iassist quarterly 1 informatics for health & social care 1 information & organization 1 information discovery and delivery 1 information polity: the international journal of government & democracy in the information age 1 information services & use 1 information society 1 information systems research 1 information technology & management 1 international journal of computer-supported collaborative learning 1 international journal of electronic government research 1 international journal of technology and human interaction 1 international journal of web based communities 1 journal of access services 1 journal of information systems education 1 journal of information technology 1 journal of informetrics 1 journal of intellectual capital 1 journal of knowledge management 1 journal of library & information services in distance learning 1 journal of library administration 1 journal of library metadata 1 journal of scholarly publishing 1 knowledge organization 1 learned publishing 1 library philosophy & practice 1 music reference services quarterly 1 new review of information networking 1 pakistan library & information science journal 1 partnership: the canadian journal of library & information practice & research 1 performance measurement & metrics 1 public services quarterly 1 publications 1 publishing research quarterly 1 reference & user services quarterly 1 teacher librarian 1 universal access in the information society 1 vine: the journal of information & knowledge management systems 1 total no. of articles 417     evidence based library and information practice evidence based library and information practice 2011, 6.4 149 evidence based library and information practice evidence summary study of limited value in exploring irish hospital clinicians’ information behaviour and attitudes towards the clinical informationist a review of: flynn, m. g., & mcguinness, c. (2011). hospital clinicians’ information behaviour and attitudes towards the ‘clinical informationist’: an irish survey. health information & libraries journal, 28(1), 23-32. doi: 10.1111/j.1471-1842.2010.00917.x reviewed by: maria c. melssen head of learning and information services florida international university miami, florida, united states of america email: mmelssen@fiu.edu received: 15 july 2011 accepted: 05 oct. 2011 2011 melssen. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to determine the self-perceived information needs, information-seeking strategies, and skill levels of hospital clinicians, in addition to their opinions on the inclusion of a clinical informationist (ci) in their clinical teams. design – questionnaire survey. setting – two public, medium-sized teaching hospitals (200-250 beds) in ireland. subjects – hospital clinicians. methods – a 33-item questionnaire (pilot tested on nurses) was deployed using surveymonkey. participants were recruited using snowball sampling and were emailed the link to the questionnaire in june, july, and august of 2008. hard copies of the questionnaire were also distributed at one of the hospitals; the librarian at this hospital manually entered the responses into surveymonkey. survey results were analyzed using descriptive statistics. main results – of 230 eligible hospital clinicians (hcs), 22 participated in the survey. of the hcs surveyed, 90% spend “over 21 hours per week engaged in patient care” (p. 26). during this time the hcs generate an average of 1-5 clinical questions each. the hcs surveyed frequently required information on the latest research on a specific topic, evidence based library and information practice 2011, 6.4 150 treatment, or patient problem. information on diagnosis, drugs, prognosis, new therapies and products, or the latest information on a disease area, was required less frequently, but still by at least one-third of participants. not having the time to conduct searches was the greatest barrier to information seeking for hcs. hcs spend from 1-10 hours per week on investigating answers to clinical questions. most of this information seeking occurs outside of working hours, either at home or during breaks at work. to answer their clinical questions, 90% of the hcs use published, medical literature. the resources used by hcs include textbooks (30%), journals (30%), the internet (19%), colleagues (17%), and databases (4%). the most important factors that influence resource choices are access to electronic or internet resources: 60% prefer electronic resources rather than print resources. additional influential factors included whether the resource was evidence-based, if it provided concise summaries of the information provided, and if the information could be found in a paper copy. hcs in this study also consult colleagues regularly, and their proximity to a colleague for consultation was a factor. the hcs rated their search skills very high: over half (55%) rated their ability to find information as good, 15% believed they were very good, 25% felt they were average, and 5% rated themselves as poor. the hcs were also confident in finding information to meet their needs: 70% claim that they find the information they require more than half of the time. of those hcs, 25% claim they are successful more than three quarters of the time. 65% of the hcs experience difficulties in keeping current with evidence based medicine. evidence-based resources such as the cochrane collaboration are used less frequently (25%) than resources such as medline (65%) and google (75%). when hcs were provided with a definition of clinical informationists (ci), 68% were not familiar with the role of cis and only 32% of clinicians were familiar with the term “clinical informationist.” the hcs were then asked their feelings regarding the idea of involving a ci in their hospital: 18 of the 19 responses were positive. various suggestions for how cis could be used and the benefits of cis were provided by the participants. only three disadvantages of cis were noted. of 18 responses, 72% perceived that the inclusion of cis would have a positive impact on patient care while 27.7% were neutral. conclusion – overall, the hcs surveyed in this study ask fewer questions, have different information needs, and are more confident in their search skills than clinicians found in previous studies; however, the authors state that previous studies had been done with clinicians in office settings rather than clinicians in hospital settings. hcs in this study identified lack of time as their main barrier to researching clinical questions and when they do find the time to search for clinical questions, it is either during breaks in their day or after work at home. their preferred resources are those found electronically. though they value evidencebased resources, hcs rarely use them. these factors point to a need for information professionals to provide either remote access to electronic medical information resources from home, or provide a service that would allow hospital clinicians to quickly and easily find information during the work day. this is an area in which a ci might play a role. though many hcs were not familiar with cis, they were receptive to having a ci on their clinical team. the hcs provided various suggestions for where a ci could be involved as well as desired skills and qualifications of cis. the only possible disadvantages that the clinicians could foresee was cost, the deskilling of clinicians’ own informationseeking skills, and medico-legal issues. the authors identified several limitations of this study which include the small sample size, the snowball sampling method and the possibility of bias in subject recruitment, and not evidence based library and information practice 2011, 6.4 151 including other health care professionals in this study. further research regarding the information behaviour, seeking and skills of other health professionals is needed, as well as research on training and accreditation of cis. commentary though the data collected could provide evidence for the need for clinical informationists (cis) in the two hospitals surveyed, there are several areas of concern. first, the snowball sampling method of selecting participants could present bias: the participants in the study were contacts of the authors. a second issue is the small sample size: roughly 10% of eligible hospital clinicians (hcs) participated in the study. the diversity of hc specialization, which ranges from orthopedics and plastic surgery to cardiology and rheumatology, is also problematic. different specialists may have different information needs. it is hard to generalize the information needs of such a diverse group of clinical practitioners. various terms used in the article also lack clear definitions. the reader is left to assume that “hospital clinicians” only include doctors and do not include other medical professionals, such as physician assistants. also, the term “internet” is used to describe one of the various resources hcs utilize to answer clinical questions, but it is not clear how the authors define “internet.” “internet” could include databases, search engines, or other electronic resources. an additional point of confusion is why the survey was first tested on nurses but then given to hcs. nurses and hcs may have different clinical information needs and perceptions of cis. the questions asked of nurses would be different from the questions that would be asked of hcs. finally, there is no information regarding informed consent. the reporting of the findings is also problematic. mathematical discrepancies are first found in the area of specialization for the hcs at hospital b: the total percentage is over 100%. another error was found in figure 1: the total percentage for the number of clinical questions that hcs generate per week is over 100%. a third miscalculation was in the hcs self rating of their search skills: the number of responses analyzed by the authors (n=20) is less than the number of reported responses. the corresponding chart for this data (figure 6) is also incorrect: “average” should be 25%, not 5%. such inaccuracies call into question the quality of this study’s findings. it is important to note that participants were able to select multiple responses for several questions, which is why the percentages for the questions regarding reasons why hcs do not pursue clinical questions, where hcs need additional clinical information, and cochrane, medline, and google use preference add up to over 100%. despite the drawbacks outlined above, the research methodology utilized for the group studied is appropriate and this study could be replicated. the authors detail areas of limitations with the study as well as discuss areas for further research. future researchers can also learn from the errors in this study and be mindful of their own research methodologies and data analysis. hospital librarians who are interested in ci services may find this study useful as a model for conducting a similar assessment in their own hospital. also, this study may provide additional ideas for hospitals that currently provide ci services. / evidence based library and information practice review article   career advancement and writing about women librarians: a literature review   kathleen delong associate university librarian university of alberta libraries edmonton, alberta, canada email: kathleen.delong@ualberta.ca   received: 7 aug. 2012     accepted: 10 feb. 2013      2013 delong. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – this review of the literature provides a framework for understanding the professional experiences of women library directors in academic libraries. it focuses upon career advancement and writing about women librarians in the united states and canada from the 1930s to 2012.   methods – databases from the disciplines of library science and business and management, including the larger social sciences, were searched for references to sources that dealt with career advancement and progression of women, specifically women librarians, from the 1930s to 2012. similarly, these databases were also searched for sources pertaining to writing about women, especially women in libraries. sources were also culled from major bibliographies on women in libraries. articles and monographs were selected for inclusion in the review if they reported research findings related to these broad topics. in some cases sources from the professional literature were included if they offered a unique perspective on lived experience.   results – evidence shows the number of women in senior leadership roles has increased over the years. from the 1930s to the 1950s it was the natural order for men to be heads of academic libraries, particularly major research libraries. research studies of the decades from the 1960s to the 1980s provide evidence of a shift from the assumption that various personal and professional characteristics could be identified to account for differences in the number of men and of women recruited into senior positions in academic libraries. despite this, women remained vastly under-represented in director positions in academic libraries. from the 1990s to the present, the evidence shows the number of women in senior leadership roles increased, despite factors such as mobility, career interruptions, or lack of advanced degrees that were traditionally identified as limitations to career growth. while women have gained in terms of the number of senior positions in academic libraries in the u.s. and canada they are still not proportionately represented. the results section concludes with a review of sources that pertain to writing about women library leaders. this emphasizes that the professional lives of women librarians are largely unknown, as is the importance of their contribution to the development of libraries and librarianship. these sources were included to highlight the critical importance, but lack of material that speaks to writing about women and their professional lives and experiences.   conclusions – research into the lives of women library leaders is important because women traditionally represent 75-80% of library professionals, yet the story of their career advancement and leadership within librarianship is bounded by characteristics – real or perceived – that affect their career progression. future research focusing on collecting current data about career advancement of women in canadian academic libraries as well as the contributions of women to development of libraries is suggested.     introduction   the specific focus of this review is women academic librarians’ career advancement and progression as well as writing about women in libraries. it was judged important to include the latter review of works as there is a dearth of material that speaks to writing about women and their professional lives and experiences. the literature review primarily includes sources from the united states and canada from the 1930s to 2012, although there is some brief reference to sources of importance from other countries for comparative purposes. the year 1930 was chosen as a beginning date for the review as no major studies or surveys of library staff could be located prior to that point in time. while the focus is on women academic librarians, many research studies include data about male librarians and their career advancement or in some cases the career advancement of women librarians in general.   the story of women in librarianship in the united states and canada is strikingly similar and the entrance of women into the library profession and their career advancement can be found in chronicles or accounts of the time, primarily from the united states and predominantly written by male librarians (hildenbrand, 1992). in one account from c. f. mccombs (an american surveying library services in canada in the early 1940s), the story of women in canadian librarians is found in the backdrop of a recounting of staffing that acknowledges that women’s salaries were significantly lower and opportunities for advancement much more restricted than those of men in similar positions (buxton & acland, 1998).   beginning in the late 1960s, some u.s. researchers asked a crucial question related to career advancement: “what is the status of women in librarianship?” (schiller, 1979, p. 222). both bradley (1968) and schiller (1969) found a similar trend of hiring men rather than women as chief librarians in u.s. libraries. in canada, the closest attempt to systematically investigate the status of women in canadian libraries was that of librarians sherrill cheda and phyllis yaffe, along with sociologists, dr. linda fischer and mary ann wasylycia-coe. their research in the early 1970s was completed with the help of a grant from the canada council and the data were analyzed but the study was never published in its entirety for reasons that are still unclear. lack of time, interest, funds; problems in methodology, research design, and implementation; possibly even political suppression, have all been suggested as reasons (futas, 1983). however, several articles were published using data from the study, and cited widely as illustrative of the fairly low status and salaries of women in canadian libraries and the different career structures that existed for men and women during the 1970s, as first detailed in the article by cheda, fischer, wasylycia-coe, and yaffe (1978).  kristy (1983) examined the status of women in librarianship on a cross-national basis. her study encompassed western style democracies (including the united states, canada, france, and new zealand), soviet bloc countries (bulgaria, cuba, hungary, czechoslovakia, and the ussr) and developing nations (nigeria, india, and brazil). with the exception of nigeria, librarianship was found to be a profession in which women dominated numerically, however all countries exhibited patterns of intra-occupational segregation which was defined as different career tracks for men and women in the same profession. overall, men were found to have attained higher levels of position and salary than their female counterparts.   the trend to hire male library administrators began to reverse itself by 1990. wilder (2003), who enumerated the gender of directors of arl libraries from 1980 to 2000, described greater balance between the genders; the number of male directors decreased from 84.5% in 1980 to 54.1% in 2000. recent survey data from arl shows a further decline in the number of male directors to 40% in 2009 (association of research libraries, 2010). data compiled by the american federation of labor and congress of industrial organizations, which is affiliated with over 57 unions, indicate the majority of library workers in the united states are female while men make up less than 20% of the library profession (afl-cio, department for professional employees, 2009). further, academic (arl and non-arl) library directors are 47% male and public library directors are 35% male. in canada, a 2005 study by the 8rs research team determined that 79% of library professionals were women; 41% of senior administrators in carl libraries were male, as were 29% of senior administrators in culc (canadian urban libraries council) libraries. while more women are heads of academic libraries in the u.s. and canada in the twenty-first century than in the past, the 80/20 rule of women and men making up the profession has remained fairly constant throughout the years, and on this proportional basis women are still under-represented at the most senior rank of librarianship, including academic libraries.    turning to writing about women, there is little evidence that women have written openly about themselves or their accomplishments, and this is true of women in the library profession where there is little documentation or history of women’s professional experience, including their career progression and development as leaders. gerda lerner, a distinguished historian, has written extensively about why the history of women’s experience matters (lerner, 1997), and in an interview about her work she states its importance:    and now people may think that's not so very important, but the fact is that our ideas about what is possible for the future are formed out of our knowledge of what was possible in the past. and if we have no past, if a group is deprived of its past, it cannot imagine a future for itself. it can only imagine a future for the people that it thinks have done the historic work in the past, and that's men. (lerner, n.d.)   thus, reviewing the literature and understanding the past is timely as the current generation of women who aspire to senior administrative and leadership roles in academic libraries prepare themselves for the future.   aims   this review of the literature provides a framework for understanding the past experiences of women library directors. it focuses upon career advancement and writing about women librarians, almost exclusively in the united states and canada from the 1930s to 2012. why is this important? as the current generation of library directors moves into retirement, women library leaders will look to the experiences of their predecessors as they prepare to assume senior management positions and senior roles. while they will find evidence of change in career patterns of male and female librarians, they will also appreciate that these changes are of relatively recent date and that hiring trends and the status of women in libraries cannot be taken for granted. finally, the literature review will be concluded by suggestions for areas for further research with a focus on women academic librarians in canada in particular.   methods   databases from the disciplines of library science and business and management, including the larger social sciences, were searched for references to sources that dealt with career advancement and progression of women, specifically women librarians in academic libraries. these databases included: library and information science source (ebsco), lisa: library and information science abstracts, academic search complete, business source complete, abi inform global, cbca complete, and eric (ebsco).   additionally, it is critical to acknowledge a primary source of works about women in librarianship, the bibliographies on account of sex: an annotated bibliography on the status of women in librarianship, published from 1977 to 2002, and its forerunner the role of women in librarianship, 1897-1976: the entry, advancement and struggle for equalization in our profession provide descriptions of works important to the literature review. as maack (2002) noted, historical and biographical works make up a large part of the entries and “we now have bibliographic access to over 120 years of writings that vividly demonstrate that there have always been women in librarianship who were not silent or complacent or lacking in ambition” (p. 242). these bibliographies were also scanned for works important to the topics of career advancement and writing about women librarians.   although the emphasis of the literature review is upon women in academic libraries, there are relatively few references that pertain solely to academic librarianship, so works on women’s careers and leadership in libraries generally were examined as important source materials and included as appropriate. as stated in the introduction, this review primarily includes sources from the united states and canada with some brief reference to sources of importance from other countries for comparative purposes. studies were selected for inclusion if they provided research data that added to the discussion of career progression of women in librarianship or were helpful in outlining the thinking about women in libraries and their status in the profession during the time period in which they were written. most works receive brief treatment with a basic outline of their results or conclusions, unless they provide evidence of a change in the number of women in administrative positions in academic libraries or add substantially to the discussion of personal characteristics and professional qualifications of women.   results   the results of the literature review fall into the two categories identified as the objectives of the review. first, the section on career progression of women in academic libraries highlights demographic profiles and professional characteristics and qualifications of female librarians as well as issues such as mobility and career interruptions. evidence shows the number of women in senior leadership roles has increased over the years. however, women are still underrepresented today in leadership positions in academic libraries. second, the results section concludes with a review of sources that pertain to writing about women library leaders. the noted dearth of writing about women librarians emphasizes that the professional lives of women librarians are largely unknown, as is the importance of their contribution to the development of libraries and librarianship. these sources were included to highlight the critical importance of but lack of material that speaks to writing about women and their professional lives and experiences.   career progression of women in academic libraries   studies of career progression of women in academic libraries vary, but most focus on demographic and personal characteristics and professional qualifications of women librarians. bradley (1968) defines personal characteristics as factors that differentiate librarians from one another but have little or no relationship to the ability to perform in a job, and qualifications as those things that distinguish librarians from each other and have an assumed relationship to performing well. the personal characteristics of job mobility and career interruption are ones that are often identified as issues and commonly highlighted for discussion as these are factors most closely associated with women’s careers. most often the data or results of the research are described in career profiles and examined for career patterns that have determined women’s professional lives. in conjunction with data about women, characteristics and qualifications of male librarians are often gathered and used to compare and contrast, so may illuminate the differences men and women experience in career progression or advancement in academic libraries.   from the 1930s to the 1950s in the u.s. and canada: limited prospects for women   irvine (1985) reviewed the writing on gender in academic library administration beginning in the 1930s with randall’s (1932) acknowledgement that the administration of college libraries was primarily a male exercise. alvarez (1938), williamson (1939), and mcdiarmid (1942) pointed to the importance of advanced degrees and experience in administration for recruitment into senior positions in academic libraries. while alvarez argued women have just as many opportunities to become chief librarians as men, both williamson and mcdiarmid acknowledged that prospects were much more limited for women. o’brien (1983), who reviewed a number of articles published in the library literature from 1941-1950, concluded that “the fact that men were preferred as administrators was acknowledged both within and without the library profession during the 1940s” (p. 60). various reasons were advanced for this, including bannister’s assessment (as cited in o’brien, 1983) that women were unable to handle prolonged periods of responsibility.   samuel rothstein, a noted canadian library educator, commented upon the tenor of professional life in canadian libraries in the 1930s and 1940s and recalled that:   yes, most of us were women (perhaps nine out of ten professionals), but this marked disproportion, which was certainly one of the most salient and influential features of the library community, was also one of the least discussed subjects. oh, one heard adjurations about the desirability of getting more men into the profession but scarcely a word about women being passed over in appointment to high-level positions. it was as though most women librarians saw themselves as natural subordinates and their careers as temporary jobs to be held only until marriage. (1990, p. 6)   as rothstein (1990) concluded, “in other words, when almost the only alternatives were nursing, social work, or teaching, librarianship found it easy to recruit women of very superior ability who were willing to dedicate themselves to demanding jobs for small practical rewards” (p. 7).   in the 1950s, schick (1950) and harvey (1958) both observed the tendencies for women chief librarians to be clustered in smaller academic institutions, to be promoted from within, and to lack both academic degrees and job mobility. it was also common for these women to be single and younger than male chief librarian colleagues and have fewer years of professional library experience.   examining the writing and research over this time period provides evidence that women’s administrative and leadership abilities were often recognized, but given greatest credence under specific circumstances. career advancement for women, who were the majority in libraries, was bounded by the size of the responsibilities, usually also determined by the size of the institution, by job mobility, and by years of education and experience. it was the natural order for men to be heads of academic libraries, particularly major research libraries, and the male minority presumably advanced the careers of other men.   the 1960s to 1980s in the u.s. and canada: “who are these women?”   in the 1970s, cohn (1976) and parsons (1976) both retrospectively reviewed the recruitment of directors into large research libraries. parsons, who compared the number of women directors in 1958 to that of 1973, found that five female directors had been recruited during that period. cohn’s work, which spanned 1933 to 1973, identified two women appointed between 1934 and 1969 and five between 1970 and 1973. all of these director appointments were internal. wong and zubatsky (1983), who studied the demographic and professional characteristics of first-time appointed college and university library directors between the years 1970 to 1980, found women were most often appointed at smaller institutions. since they tended to be unmarried, it was presumed they were more mobile. when morrison (1969) surveyed academic library administrators he found female chief librarians had less academic training, but also found there was not a great difference in job mobility of male and female chief librarians. men and women differed on two additional characteristics, however; male chief librarians tended to enter the professional later than their female counterparts and they tended to be published more frequently. bradley (1968), in his study of 100 heads of large u.s. academic and public libraries, noted there was a definite tendency for men to replace women at this position level, reaching its zenith among large academic library directors, all of whom were men. schiller’s (1969) work on academic librarians does not separate the characteristics of chief librarians from the population of over 2,200 librarians she studied, but does describe how the tendency to hire male librarians to fill administrative positions had become more apparent over the last few decades. she found that as women directors retired men were hired to take their places. she also noted that men apparently have greater job mobility. schiller’s work (1979) on the status of female librarians, with respect to salary, position level, and career advancement, “has become a benchmark against which to measure women’s progress” (maack, 2002, p. 244).   the question of mobility in job seeking and its influence on career advancement tended to dominate discussions of career progression. in a large scale study of american library association members, almost 2,000 respondents provided career and personal data. the data indicated that overall career patterns of male and female members were relatively similar (heim, 1983). women, however, were twice as likely to report geographic location as a factor in accepting positions and career advancement was often related to job mobility. studies by metz (1978), martin (1979), and maag (1981) provided data on recruitment of academic library directors, job mobility and advancement. metz, who examined succession patterns of academic library directors, found external male candidates were most often recruited into academic libraries. if women were hired they were most often internal candidates. he speculated this may be due in part to their more limited geographic mobility. martin’s findings did not support a statistically significant difference in the mobility rates of men and women in arl libraries, although there were important predictor variables for position level (e.g., career work continuity and working in a large number of libraries). maag found that appointments of academic library directors typically went to external male candidates and woman were more likely to be internal appointees.   in 1983, robinson, who studied mobility and career patterns in a large population of female and male academic librarians, found a statistically significant relationship between mobility and position level. her research also showed that men gained greater position levels through mobility, and even when men and women were moving in order to advance their careers, rather than for personal or family reasons, men achieved greater position advancement. robinson examined career interruptions and leaves of both men and women in order to determine the influence of work interruptions on careers. while women more frequently took leaves from work, and these were more usually personal or family related leaves, there was no statistically significant difference in position advancement between women who had interrupted their careers and those who did not. robinson concluded neither the issue of job mobility nor career interruption could be held responsible for the differences in career progression of men and women in academic librarianship.   fennell (1983), referenced fairchild’s (1904) oft quoted query about whether women would ever hold the highest administrative positions in libraries to note that fairchild’s question was still open. she asked a follow-up question of her own, “the most important positions of leadership and administration in the academic-library profession are reserved for men, or, at best, include only a small proportion of women. yet, who are these women?” (p. 209). fennell explored personal and professional backgrounds, education and training, the role of work in their lives, and factors that influenced career achievement to develop a composite career profile of 11 (from a total of 17) women directors of the 164 largest academic libraries in the united states. the women administrators were asked what recommendations they would have for women interested in academic library director positions. their advice was summarized as “get an advanced degree, work hard, be mobile, and choose your job because of the person for whom you will be working rather than for any other reason” (p. 239). the career profile of these women did not necessarily exemplify their advice; they believed they had worked hard and been in the right place at the right time but for the most part had not been deliberate in planning their careers.   at the same time that fennell was examining the career profiles of female library directors, moran (1983) was studying the comparative career progression of over 400 male and female academic library administrators who had been either assistant or associate library directors ten years previously. she explored personal characteristics, professional qualifications, and the relationship of these factors to having attained a director’s position. there were distinctive differences by gender. for women there was no statistically significant relationship between personal characteristics such as marital status and number of geographic relocations and becoming a director. men, however, were much more likely to have attained a director position if they were married with children and had made a number of moves to assume administrative positions. similarly, professional qualifications showed little influence in determining whether or not the women in the study group became directors, but did effect whether men did. even when examining types of academic libraries, from college to large research libraries, these same patterns held true. moran concluded that the personal and professional factors that are commonly used to explain why women do not attain the director position in academic libraries are not related to success in achieving that role. an explanation lies in the different career patterns of men and women in academic librarianship. male librarians who achieve advanced degrees, participate in professional associations, and are published in the professional literature are likely to become directors in academic libraries; women following the same career pattern would not be as likely to attain a directorship. the only variable likely to influence the success of women in achieving the director’s position was whether they were an internal candidate for the position, having been an assistant or associate director in the same institution. several years earlier, metz (1978) had reported this same pattern of internal succession for women – a pattern that is much less likely for men.   it is worthy of note that the advice given to women interested in academic library directorships by the female directors in the fennell study is similar to the male career pattern moran (1983) describes. olsgaard (1984) developed a set of attributes of the successful academic librarian by studying academic librarians listed in who’s who in library and information services published in 1982 and provided a composite of the successful academic librarian. among other attributes, the successful academic librarian is most likely to be a male administrator working in a large academic library. olsgaard stressed that these composite characteristics are not necessarily the most desirable and that a set of standards for success would be useful for the profession.   several studies published in the 1980s examined personal and professional characteristics of male and female librarians. they found few differences other than the number of men and women recruited into senior positions in academic libraries. swisher and dumont (1984) studied the professional qualifications of over 300 academic librarians in an effort to ascertain how these factors influenced attainment of administrative positions. they identified tangible job qualifications (educational background, previous experience, publications, and professional activity) and compared men and women to show that even with almost identical qualifications men were more likely to advance to administrative positions. irvine (1985) examined the characteristics of directors, associate directors, and assistant directors of the 99 academic libraries that were members of arl in 1980. of the 371 individuals identified, 256 men and 115 women answered survey questions about demographic characteristics and career patterns that could have influenced their attainment of administrative positions. overall, irvine found that women displayed many of the characteristics of their male colleagues and far fewer gender related differences than expected. women, on average, were younger, had worked fewer years in libraries, and had only half the administrative experience as men in comparable positions. irvine attributed these differences to the passing of title ix of the education amendments of 1972, coupled with affirmative action programs, as having given women access to administrative posts and increased the number of women in administrative positions. the enactment of federal laws and regulations likely encouraged women to apply for administrative posts, as women perceived that positions were accessible to them. irvine (1985) also pointed to arl data that indicated that there were no women directors and only 16% of associate and assistant director positions filled by women in 1970, but by 1976/1977 11% of arl directors and 30% of associate and assistant directors were women.   in 1989, moran replicated the research on career progression she had done some ten years earlier. she found there had been significant changes in the career progression patterns of female academic library administrators. the personal characteristics of male and female library directors remained unchanged, but professional characteristics showed statistically significant differences. in the 1983 study, neither educational level nor professional activities were associated with female success in career advancement, and internal succession was the norm. by 1989, professional characteristics of advanced degrees, publications, professional activity, and mobility were associated with success of both men and women.   looking specifically at the canadian context from the 1960s to the 1980s, several studies are illustrative of the trends found in the u.s. professional library community. in 1974, cheda, fischer, wasylycia-coe, and yaffe examined the career patterns of canadian librarians. the research group broke up before the study was published in its entirety (futas, 1983), but the group later produced a study which examined salary differentials of male and female librarians (cheda, fischer, wasylycia-coe, & yaffe, 1978). wasylycia-coe (1981) also profiled canadian chief librarians by gender, analyzing data collected in the larger study. she compared male and female chief librarians on a number of career characteristics and social background variables and noted that male and female heads of libraries were fairly equally represented in small to large public libraries and college libraries, whereas women in special libraries outnumbered male heads almost two to one. there were, however, no women heading large university libraries in the sample data. she also found men were much more likely to have obtained a first job as chief librarian, and it was more usual for a man than a woman to become a chief librarian early in his career. the salary and advancement findings that cheda, fischer, wasylycia-coe, and yaffe (1978) first noted were echoed by plate and siegel (1979) in their study of ontario librarians. almost 10 years after the cheda, fischer, wasylycia-coe, and yaffe study, harris and monk (1986) surveyed masters of library science graduates from the university of western ontario and undertook a “prestige analysis” of professional tasks and work settings to find that men were most often performing higher status tasks, such as administrative tasks, more of the time than were women, and that men were paid more, on average, than women in every setting except for community college libraries. in 1987, bowron tabulated the number of men heading large canadian university and public libraries in 1948 and in 1986, and found that there had been an increase in the male majority.   overall, the career progression of canadian library directors is similar to that of their u.s. counterparts. harris and tague (1989) explored the career paths of 26 male and female directors in academic, government, and large public libraries. noting work done by u.s. researchers such as moran, the canadian investigators studied whether similar career patterns might be found in canadian librarianship. by and large the findings were similar: male directors did not have higher degrees in education, higher publication levels, or higher professional activity than female directors, and mobility was not found to be a factor in career advancement. deschatelets and saint-marseille (1991) reported similar findings in their study of directors of large francophone libraries in canada.   summing up to the 1990s, the research studies of the decades from the 1960s to the 1980s provided evidence of a shift from the assumption that various personal and professional characteristics could be identified to account for differences in the number of men and of women recruited into senior positions in academic libraries. overall, women began to display many of the characteristics of their male colleagues and far fewer gender related differences than expected. despite this, they remained vastly underrepresented in director positions in academic libraries.   the 1990s and beyond in the u.s. and canada: “is the revolution over?”   kirkland (1997), who surveyed 135 women academic library directors, identified the variables participants had found most important in their career advancement. while variables such as mobility, academic qualifications, service in professional organizations, and tenacity and perseverance were most commonly chosen, mentoring was ranked most highly in importance for career success and attaining the director position. kirkland also asked women librarians to identify anti-mentoring or deprivation behavior, which can be defined as experiences that discourage women from achieving administrative positions. deprivation or discouraging behaviors can occur in the areas of responsibility, information, recognition and approval, and solidarity (pitting women against one another). deprivation behavior, which kirkland defined as a subtle form of gender bias, discourages female ambition and supports the glass ceiling many women experience in their career progression.   moran, leonard, and zellers (2009), who analyzed the academic library workforce, concluded parity between men and women administrators had not been accomplished, but women holding directorships increased from 2% in 1972 to almost 61% in 2004. they attributed the change to the high turnover of director positions over the period between 1994 and 2004 and the number of women who replaced men as they vacated the director role. encouraging as these results are, they noted that this trend can still be improved if women are to be as proportionally represented in academic administrator positions as they are in the profession. this highlighting of the issue of parity is in response to the question posed by deyrup (2004) when she asked “is the revolution over?” (p. 249). deyrup, noting the impressive gain in top administrator positions women had made over the past thirty years, determined that both salary and professional parity had been achieved. however, she also acknowledged that these gains must be maintained and women must be encouraged to enter and then be supported in senior positions.   in canada, a 2005 study by the 8rs research team looked at variables of recruitment, remuneration, retention, retirement, reaccreditation, rejuvenation, repatriation, and restructuring throughout the canadian library workforce. although the study did not look specifically at career advancement of women or men, demographic and position data were gathered for all librarians who reported in the practitioner’s survey – over 4,697 individuals. as a follow up to the study, sorensen (2012) reanalyzed these data to provide further insights into the demographics of almost 600 senior administrators in canadian libraries, including 151 chief librarians or directors in academic libraries. while noting that very little research attention had been paid to senior positions, she concluded that   notably, however, females comprise a smaller portion of carl, other academic, and culc senior administrators, while other public, government, non-profit, and school senior administrators are slightly more likely to be female than their professional librarian counterparts. hence, while female representation is higher in the library sector than in most other occupations at all career levels, women are less likely to be found heading up the largest libraries in the country. (sorensen, 2012, p. 53)   in examining sources, primarily from the 1930s to the 2000s on the career advancement of women librarians in academic libraries, the evidence shows the number of women in senior leadership roles has increased, despite the personal characteristics and professional qualifications such as mobility, career interruptions, and lack of advanced degrees that were traditionally identified as limitations to career growth. however, women are still underrepresented in leadership positions in academic libraries and this picture should also be viewed within the larger sociocultural context of women’s advancement in the workplace. while the pace of career advancement for all women accelerated, particularly during the 1970s to 1990s, there is some suspicion that it might now be slowing. for example, in 2007 eagly and carli reminded readers that women still cannot assume that they will find gender equality in the workplace:   contemporary women still face many challenges, especially in relation to male-dominated leadership roles. they must be brave, resourceful, creative, and smart to be successful, because they can face the most elaborate of labyrinths on their path to leadership. (p. 199)   in the same article, eagly and carli also discuss how the march towards equality now shows a much slower pace. women who aspire to leadership positions in libraries should be aware that the pace of change and acceptance of women in leadership roles continues to be slow, perhaps even slackening, and they will continue to find barriers and obstacles to surmount in attaining the careers and leadership roles that they desire.   writing about women library leaders   writing about women’s lives, the female experience, has received a lot of attention from scholars over the past quarter century. seminal works by heilbrun (1988) and bateson (1990) discussed the discordance between the published or public life and the private lives of women, and how most women have not written openly about themselves or their accomplishments. similarly, biographies often describe women’s lives in ways that are not based on female truth or experiences, but contrive to be in keeping with the societal norms of the day.   works that describe women’s lives as librarians can be found in the professional literature. many of these are biographies, interviews or brief descriptions of women’s professional lives that were compiled so that names and achievements would not be lost once the memory of an individual or an institution had faded. grotzinger (1983b) examined a number of biographical studies of librarians to find that biographies of male librarians far outnumber the biographies of female librarians. she concluded that this dearth of writing about women librarians is recognition denied and “removes the evidence that would permit a solid assessment of their roles and contributions” (p. 373). this evidence is critical to decide whether their contribution was notable, and if women can be determined to be leaders in their times. grotzinger (1983a) also maintained that one of the perils of biographical studies of women librarians is the lack of apparent methodological rigor and critical analysis, as a great many of these biographical works fall into the classification of non-scholarly tributes. she examined these works as well as theses, dissertations, and collective scholarly biographies, and concluded there are a number of critical flaws in biographical works about women, some of them due to lack of primary source materials and failure to cite sources.   hildenbrand (1992) examined library history (defined as the body of published historical writing) and women’s place in history to reveal: “from their leadership positions men have shaped not only the profession but also the writing of its history” (p. 19). she refers to the difficulty of understanding the experiences of early women librarians as they “left fewer personal records than any other similarly educated women of the period” (p. 24). hildenbrand decries that histories of librarianship that omit women are still being published and points to the work of grotzinger (1983a) as critical to understanding what the lives of women librarians were like and the centrality of their contribution to library development. as hildenbrand (1996) pointed out, it is only through gendered history or an understanding of the position of women librarians in relation to their male counterparts that one can approach larger questions such as inequality in the workplace. and, women’s individual and collective biographies must not only attempt to redress the paucity of writing about women librarians but should “present an invaluable picture of early library women as active agents, choosing their work and making valuable contributions in the face of enormous obstacles” (p. 14). one such contemporary work is fitzpatrick’s mrs. magavero: a history based on the life of an academic librarian (2007). the work on magavero, who was the first female librarian at a maritime college, is set firmly within the professional writings and events of her time (1949-2003). her experience of working within a male-dominated organization for low pay and marginal status is described in her own voice:   now in those days, fort schuyler was really a male bastion, and i was coming on as the only professional woman. they had women as clerks, but i was the only professional woman. and the library was manned by the director, mr. hoverter, and i was the only other professional person in the library at the time. (p. 66)   she continues:   and here i am, with much more education, and i was hired as a clerk. now i questioned that of the librarian at the time, you know. i thought, “why should i be in the clerical line?” he said, “well there’s nothing i could do, you know this is the way it is,” and you know i accepted it. (p. 67)   it was not until 1961 or 1962 that mrs. magavero was put into a professional position. as fitzpatrick (2007) concludes, “the library world has come a long way since 1949, but still has a long way to go” (p. 60).   the stories that people tell about themselves or are told by others are powerful because they not only situate the individual in time and within context or situation but also speak to the manner in which individuals develop understanding or make meaning of their experience. mrs. magavero reflected upon her experience in terms of striving for acceptance and seeking recognition of her competency in an organization that was dominated by men in positions of power and influence. her story is compelling for the reader because it offers a firsthand account of the struggle that many women have faced in making and advancing a career in academic libraries.    discussion   research into the lives of women library leaders is important because women traditionally represent 75-80% of library professionals and yet the story of their career advancement and leadership within librarianship is bounded by characteristics – real or perceived – that affect their career progression. analysis of the works reviewed shows that the research and writing that exists pertains primarily to the united states, although a few canadian studies are noted and they show findings similar to those of u.s. studies. few studies of women’s career advancement in libraries exist outside of the u.s. and canada.   clearly, additional research is needed. today there is a generation of women library directors who came up through the ranks of academic libraries and were overwhelmingly led by male librarians, and who may have been perceived as less “desirable” than their male colleagues for promotion into senior positions (kronus & grimm, 1971). nonetheless, these women attained senior roles and account in some part for the reversal in the number of males and females in administrative positions in major libraries. the current generation of women directors is likely to have learned important lessons about the effect that being female has had on their careers. these women are also likely to retire within the next 5 to 10 years, with little documentation of the issues and challenges that shaped their experience. as positions as director or chief librarian become vacant, female candidates will be interested and competitive in recruitment into these positions, and it is likely that they can learn from the experiences of their predecessors. it is also very likely that women who aspire to senior administrative and leadership roles in academic libraries will encounter some of the same issues and obstacles to career advancement and leadership development as the current generation of women academic library directors did in the past. just as lerner (1997) advocated for all women and their history, female librarians should have the opportunity to understand the historical role of women in librarianship and celebrate the accomplishments of these women as a prelude to defining their own futures as library leaders.   looking specifically at the canadian scene, few research studies provide a baseline of data upon which further studies of career progression or leadership development in academic libraries can be grounded. future research in canada should focus on collecting current data about career advancement of women in academic libraries. additionally, few biographical works about women librarians exist, and the contributions of women in building canadian libraries should be documented and acknowledged.   the future of human resources in canadian libraries by the 8rs research team is the sole comprehensive source of general demographic and human resources data for canadian libraries. however, other than the percentage of senior administrators by sex and by library sector, the original 8rs data were not comprehensive or exhaustive enough to provide a more detailed profile for males and females in canadian academic libraries. ideally, the 8rs data should be updated on a regular basis, with particular attention paid to analyzing data by sex and potentially by the other three groups identified in canadian equity legislation. this would provide a firm foundation of demographic data on which to base further study of women in academic libraries and of all sectors.   limitations of this review include inclusion of only studies that dealt specifically with career advancement as well as writing about women in libraries. in doing so, many studies and much writing about the sociocultural forces that were shaping the lives of women in the workforce in the 1960s to the 1980s have not been acknowledged or represented, and yet these forces have influenced the choices that women in libraries have made about their careers. neither does the literature review capture any studies reporting changes to the operating environment of libraries and the internal changes, such as unionization, that were bred in the tumult of social change. libraries as organizations have not been immune from societal change and influences, and any resulting legislation or policy, as well as the changing role and status of women within library organizations, has been the subject of debate and question by both administrators and library staff. although over time established attitudes and behaviors regarding women librarians no longer seemed appropriate, in some instances the career aspirations and advancement of women was still met with ignorance, discrimination, and opposition. and, as eagly and carli (2007) remind us, this can still happen today; gender equality in the workplace cannot be assumed. additionally, the review does not include research about career development of women in general, career aspirations, or the development of skills or competencies that support leadership growth of the individual. all of these elements are useful in understanding the context for career advancement and writing about women in libraries, but this review was not designed to be comprehensive enough to reflect overall context.   conclusions   a review of the literature regarding career advancement and writing about women revealed that overall women have gained in terms of the number of senior positions in academic libraries in the u.s and canada, although they are still not proportionately represented. noteworthy studies described the demographic profiles and professional characteristics and qualifications of women librarians and their significance and change over time. the dearth of writing about women librarians is highlighted to emphasize that the professional lives of women librarians are largely unknown as is the importance of their contribution to the development of libraries and librarianship, whether or not they achieved senior administrative positions. future research focusing on collecting current data about career advancement of women in academic libraries as well as the contributions of women to development of libraries is suggested.     references   8rs research team. (2005). the future of human resources in canadian libraries. in university of alberta learning services. retrieved 6 sept. 2009 from http://www.ls.ualberta.ca/8rs/8rsfutureofhrlibraries.pdf   afl-cio, department for professional employees. (2009). library workers: facts and figures. in afl-cio, department for professional employees. retrieved 6 sept. 2009 from http://www.dpeaflcio.org/programs/factsheets/fs_2009_library_workers.htm   alvarez, r. s. (1938). women’s place in librarianship. wilson bulletin for libraries, 13, 175-178.   association of research libraries. (2010). arl annual salary survey 2009-2010. in association of research libraries. retrieved 14 july 2010 from http://www.arl.org/stats/annualsurveys/salary/sal0910.shtml   bateson, m. c. (1990). composing a life. new york: plume books.   bowron, a. (1987). statistics tell the tale: men still getting top jobs. canadian library journal, 44(4), 211-212.   bradley, b. w. (1968). a study of the characteristics, qualifications and succession patterns of heads of large u.s. academic and public libraries. unpublished manuscript, university of texas, austin, texas.   buxton, w. j., & acland, c. r. (1998). american philanthropy and canadian libraries: the politics of knowledge and information, accompanied by “report on canadian libraries” as submitted to the rockefeller foundation (1941). montreal: mcgill university graduate school of library and information studies.   cheda, s., fischer, l., wasylycia-coe, m. & yaffe, p. (1978). salary differentials of female and male librarians in canada. emergency librarian, 5(3), 3-13.   cohn, w. l. (1976). an overview of arl directors, 1933-1973. college & research libraries, 37(2), 137-144. retrieved 18 feb. 2013 from http://crl.acrl.org/content/37/2/137.full.pdf   deschatelets, g., & saint-marseille, j. (1991). bibliothécaires ou administrateurs? profils de carrière des directeurs des grandes bibliothèques francophones du canada [librarians or managers? career profiles of the directors of large francophone libraries in canada]. documentation et bibliothèques, 37(1), 25-34.   deyrup, m. m. (2004). is the revolution over? gender, economic, and professional parity in academic library leadership positions. college & research libraries, 65(3), 242-250. retrieved 18 feb. 2013 from http://crl.acrl.org/content/65/3/242.full.pdf   eagly, a. h., & carli, l. l. (2007). through the labyrinth: the truth about how women become leaders. boston: harvard business school press.   fairchild, s.c. (1904). women in american libraries. library journal, 29, 157-162.   fennell, j. c. (1983). the woman academic-library administrator: a career profile. in k.m. heim (ed.), the status of women in librarianship: historical, sociological, and economic issues (pp. 207-241). new york: neal-schuman publishers.   fitzpatrick, j. b. (2007). mrs. magavero: a history based on the career of an academic librarian. duluth, mn: library juice press.   futas, e. (1983). an analysis of the study: career paths of male and female librarians in canada. in k. m. heim (ed.), the status of women in librarianship: historical, sociological, and economic issues (pp. 393-413). new york: neal-schuman.   grotzinger, l. a. (1983a). biographical research on women librarians: its paucity, perils and pleasures. in k.m. heim (ed.), the status of women in librarianship: historical, sociological, and economic issues (pp. 139-190). new york: neal-schuman publishers.   grotzinger, l. a. (1983b). biographical research: recognition denied. journal of american library history, 18(4), 372-381.   harris, r. m., & monk, s. (1986). sex differences in prestige and salaries. canadian library journal, 43(3), 149-156.   harris, r. m., & tague, j. m. (1989). reaching the top in canadian librarianship: a biographical study of sex differences in career development. the library quarterly, 59(2), 116-130. retrieved 18 feb. 2013 from http://www.jstor.org/stable/4308350   harvey, j. f. (1958). variety in the experience of chief librarians. college & research libraries, 19(2), 107-110. retrieved 18 feb. 2013 from http://crl.acrl.org/content/19/2/107.full.pdf   heilbrun, c. g. (1988). writing a woman’s life. new york: ballantine books.   heim, k. m., estabrook, l. s. (1983). career profiles and sex discrimination in the library profession. chicago: american library association.   hildenbrand, s. (1992). a historical perspective on gender issues in american librarianship. canadian journal of information science, 17(3), 18-28.   hildenbrand, s. (1996). from the politics of library history to the history of library politics. in s. hildenbrand (ed.). reclaiming the american library past: writing the women in (pp. 1-23). norwood, nj: ablex publishing.   irvine, b. (1985). sex segregation in librarianship: demographic and career patterns of academic library administrators. westport, ct: greenwood press.   kirkland, j. j. (1997). the missing women library directors: deprivation versus mentoring. college & research libraries, 58(4), 375-383. retrieved 18 feb. 2013 from http://crl.acrl.org/content/58/4/375.full.pdf   kristy, k. k. (1983). women in librarianship: a cross-national problem study. retrieved 19 feb. 2013 from eric database (ed238453).   kronus, c. l., & grimm, j. w. (1971). women in librarianship: the majority rules? protean, 1, 4-9.   lerner, g. (n.d.) interview: women and history part ii: the creation of feminist consciousness. retrieved 19 feb. 2013 from http://www.intuition.org/txt/lerner2.htm   lerner, g. (1997). why history matters: life and thought. new york: oxford university press.   maack, m. n. (2002). documenting one hundred twenty years of writings on women’s entry, advancement, and struggle for equalization in librarianship. the library quarterly, 72, 241-246. retrieved 18 feb. 2013 from http://www.jstor.org/stable/4309602   maag, a. f. (1981). so you want to be a director? journal of academic librarianship, 7(4), 213-217.   martin, j. k. (1979). academic library management: a comparison of females and males. library management bulletin, 2(4), 9-11.   mcdiarmid, e. w. (1942). the place of experience in developing college and university librarians. the library quarterly 12(3), 614-621. retrieved 18 feb. 2013 from http://www.jstor.org/stable/4302993   metz, p. (1978). administrative succession in the academic library. college & research libraries, 39(5), 358-364. retrieved 18 feb. 2013 from http://crl.acrl.org/content/39/5/358.full.pdf   moran, b. b. (1983). career patterns of academic library administrators. college & research libraries, 44(5), 334-344.   moran, b. b. (1989). a reexamination of the career patterns of academic library administrators. in j. c. fennell (ed.), building on the first century: proceedings of the fifth national conference of the association of college and research libraries. (pp. 276-280). chicago: association of college and research libraries.   moran, b. b., leonard, e., & zellers, j. (2009). women administrators in academic libraries: three decades of change. library trends, 58(2), 215-228. doi:10.1353/lib.0.0088   morrison, p. d. (1969). the career of the academic librarian: a study of the social origins educational attainments, vocational experiences, and personality characteristics of a group of american academic librarians. chicago: american library association.   o’brien, n. p. (1983). the recruitment of men into librarianship following world war ii. in k.m. heim (ed.), the status of women in librarianship: historical, sociological, and economic issues. (pp. 51-66). new york: neal-schuman publishers inc.   olsgaard, j. n. (1984). characteristics of “success” among academic librarians. college & research libraries, 45(1), 5-14.   parsons, j. l. (1976). how have they changed? characteristics of research library directors, 1958 and 1973. wilson library bulletin, 50(8), 613-617.   plate, k. h., & seigel, j. p. (1979). career patterns of ontario librarians. canadian library journal, 36(3), 143-148.   randall, w. m. (1932). the college library. chicago: american library association and the university of chicago press.   robinson, j. s. (1983). geographic mobility and career advancement of male and female librarians. in k. m. heim (ed.), the status of women in librarianship: historical, sociological, and economic issues (pp. 345-391). new york: neal-schuman.   rothstein, s. (1990, spr.). what’s past is prologue: an historical view of canadian librarianship’s future. epilogue/épilogue: canadian bulletin for the history of books, libraries and archives, 9, 1-17.   schick, f. l. (1950). meet the college librarian. library journal, 74, 1017-1019.   schiller, a. r. (1969). characteristics of professional personnel in college and university libraries. springfield, il: illinois state library.   schiller, a. r. (1979). women in librarianship. in k. weibel and k. m. heim (eds.). the role of women in librarianship 1876-1976: the entry, advancement and struggle for equalization in one profession. (pp. 222-256). phoenix, az: oryx press.   sorensen, m. (2012). senior administrators in canadian libraries: a profile. in k. de long and a. sivak (eds.), the future is now: responses to the 8rs canadian library human resources study. (pp. 52-66). ottawa: canadian library association.   swisher, r., & du mont, r. r. (1984). sex structuring in academic libraries: searching for explanations. the library quarterly, 54(2), 137-156. retrieved 18 feb. 2013 from http://www.jstor.org/stable/4307718   wasylycia-coe, m. a. (1981). profile: canadian chief librarians by sex. canadian library journal, 38(3), 159-163.   wilder, s. j. (2003). demographic change in academic librarianship. washington, dc: association of research libraries.   williamson, c. c. (1939). training of university librarians–ii. college & research libraries, 1(1), 30-32. retrieved 14 feb. 2013 from http://crl.acrl.org/content/1/1/22.full.pdf   wong, w. s., & zubatsky, d. s. (1983). the first-time appointed academic library director 1970-1980: a profile. journal of library administration, 4(1), 41-49. doi:10.1300/j111v04n01_05     article   health professions students’ lifelong learning orientation: associations with information skills and self-efficacy    misa mi associate professor medical library oakland university william beaumont school of medicine rochester, mi, usa email: mi@oakland.edu   cheryl riley-doucet professor school of nursing oakland university rochester, mi, usa email: rileydou@oakland.edu   received: 7 dec. 2015     accepted: 16 apr. 2016       2016 mi and riley-doucet. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – this study aimed to investigate the relationships among health professions students’ lifelong learning orientation, self-assessed information skills, and information self-efficacy.   methods – this was a descriptive study with a cross-sectional research design. participants included 850 nursing students and 325 medical students. a total of 419 students responded to a survey questionnaire that was comprised of 3 parts: demographic information, the jefferson scale of lifelong learning (jeffsll-hps), and an information self-efficacy scale.   results – findings of the study show a significant correlation between students’ lifelong learning orientation and information self-efficacy. average jeffsll-hps total scores for undergraduate nursing students (m = 41.84) were significantly lower than the scores for graduate nursing students (m = 46.20). average information self-efficacy total scores were significantly lower for undergraduate nursing students (m = 63.34) than the scores for graduate nursing students (m = 65.97). there were no significant differences among cohorts of medical students for jeffsll-hps total scores. however, for information self-efficacy, first year medical students (m = 55.62) and second year medical students (m = 58.00) had significantly lower scores than third/fourth year students (m = 64.42).   conclusion – findings from the study suggest implications for librarians seeking ways to advance the value and utility of information literacy instruction in educational curricula. as such instruction has the potential to lead to high levels of information self-efficacy associated with lifelong learning; various strategies could be developed and incorporated into the instruction to cultivate students’ information self-efficacy.   introduction   there has been an increasing emphasis on quality improvement in health care, patient satisfaction, and evidence-based practice in patient care to ensure quality patient outcomes. evidence-based practice is a process of problem-solving, self-directed, and lifelong learning in which caring for one’s own patients creates the need for clinically important information about different health care issues. to become lifelong learners and provide high-quality care guided by the best evidence, health professions students need to learn a new set of information skills. how students’ lifelong learning attitude or orientation is associated with their information skills has not been explored, in spite of the potential effect of information skills on one’s own lifelong learning orientation. results of any research on the relationships would inform health sciences librarians and health professions educators in developing programs, instructional strategies, or learning activities integrated into curriculums that inculcate health professions students’ information skills and self-efficacy for lifelong learning over the span of their future careers.   literature review   lifelong learning   lifelong learning is considered an indicator of professionalism for healthcare professionals (arnold, 2002; duff, 2002; nelson, 1998; novak, palladino, ange, & richardson, 2014). it is important for any healthcare providers to engage in lifelong learning because they work with human life--meeting patients’ medical or healthcare needs (muliira, etyang, muliira, & kizza, 2012). lifelong learning is defined as “an attribute involving a set of self-initiated activities and information-seeking skills with sustained motivation to learn and the ability to recognize one’s own learning needs” (hojat et al., 2003; hojat, veloski, nasca, erdmann, & gonnella, 2006). physicians must be lifelong learners throughout their professional careers, taking time to keep abreast of new developments and advancement in their specialty (afonso, ramos, saraiva, moreira, & figueira, 2014). to be successful as physicians, they must commit themselves to a lifetime of self-directed learning or self-education (duff, 2002).   the institute of medicine released a report titled the future of nursing: leading change, advancing health that emphasizes the importance of lifelong learning in developing a more highly-educated workforce (institute of medicine, 2010). the report states that “nursing education should serve as a platform for continued lifelong learning and should include opportunities for seamless transition to higher degree programs” (institute of medicine, 2010, p. 4). to lead change and advance health, nurses should continuously seek out and actively participate in activities that promote lifelong learning (rishel, 2013). it is essential for nurses to deliberately plan and execute strategies for lifelong learning by moving beyond simple competency (woodruff, 2012) which benner (1984) considers as a low level of professional practice. in woodruff’s view, “competencies may be the stimulus for additional learning, but do not by themselves result in lifelong learning” (2012, p. 12). an inquiry into the relationship between health professions students’ information skills and their lifelong learning would contribute to our understanding of and development of strategies and activities to promote lifelong learning.   lifelong learning is recognized as an obligation for healthcare professionals; however, engaging in the process of lifelong learning is not necessarily something that comes naturally; it takes a personal commitment to pursue learning throughout one’s professional career (rishel, 2013). with an exponential growth of medical knowledge and rapid development of biomedical advances, health care professionals are facing challenges of staying current and applying the growing medical knowledge to caring for individual patients. hojat and colleagues (2009) maintain that it is important and timely to empirically study physicians’ lifelong learning, development, its predictors, and its outcomes. since lifelong learning is considered as an element of professionalism for healthcare professionals, it is necessary to examine how health professions students develop lifelong learning across the lifespan of their educational career and how lifelong learning interacts with other behavioral manifestations and outcomes. findings from research on students’ lifelong learning may inform curriculum planning and evaluation to improve teaching and learning.   information literacy   information literacy lays the foundation for lifelong learning. it initiates, sustains, and extends lifelong learning through a cluster of abilities such as using technologies, finding, using, evaluating, and managing information. developing lifelong learners is central to the mission of higher education institutions and any profession’s education programs (association of college & research libraries, 2000). information literacy competency extends learning beyond formal classroom settings and provides practice with self-directed investigations as individuals move into professional positions and are entrusted with increasing responsibilities in all walks of life. “gaining skills in information literacy multiplies the opportunities for students’ self-directed learning” (association of college & research libraries, 2000). because information literacy augments students’ competency with finding, using, evaluating, and managing information, it is now considered by accreditation bodies as one key outcome for students (association of college & research libraries, 2000; liaison committee on medical education, 2013). information literacy has been, to a certain extent, incorporated into health professions educational programs. nevertheless, the relationship between students’ lifelong learning orientation and their information literacy has received little attention in the literature.   self-efficacy   “efficacy beliefs influence how people think, feel, motivate themselves, and act”(bandura, 1995b). bandura (1986) defined self-efficacy as people’s belief in their ability to successfully perform a given behavior or task or their capabilities to organize and execute the course of action required to attain a goal. self-efficacy for information literacy has been defined as the extent to which college students feel capable of conducting online information searches to identify appropriate sources and retrieve relevant information for academic or research purposes (ren, 2000). “self-efficacy beliefs provide the foundation for human motivation, well-being, and personal accomplishment” (kurbanoglu, 2003, p. 638). people are more likely to engage in activities in which they feel confident or efficacious.   learning certain skills is far from enough; individuals should also develop confidence in the skills that they are learning (bandura, 1977). success is not simply based on the procession of necessary knowledge and skills for performance, it also requires the efficacy belief or confidence to apply the knowledge and skills effectively (kurbanoglu, 2003). a strong belief in one’s capabilities regulates one’s learning, motivation, and attitude; thus, building students’ efficacy beliefs in their capabilities would likely enhance and sustain their motivation to learn and develop lifelong learning skills. strong self-efficacy perception for information literacy is a must for accomplishing lifelong learning (kurbanoglu, 2003).   efficacy perceptions develop from a gradual attainment of skills and mastery of experiences over time (bandura, 1986). previous studies document the effect of information literacy instruction on levels of students’ self-efficacy and academic performance (ren, 2000; tang & tseng, 2013). given the close link between lifelong learning, the attainment of self-efficacy, and information literacy, investigating efficacy beliefs related to information literacy is a worthwhile endeavor to examine the psychological factors underlining the development of information literacy, which would likely affect one’s own lifelong learning orientation.   aims   the study was undertaken to generate empirical evidence demonstrating the relationship between students’ lifelong learning orientation, information skills, and information self-efficacy. it addressed the following research questions:   1.       what was the relationship between health professions students’ lifelong learning orientation and information self-efficacy? 2.       what was the relationship between health professions students’ demographic variables (student characteristics) and their lifelong learning orientation and information self-efficacy? 3.       what was the relationship between students’ self-assessed information skills and their lifelong learning orientation and information self-efficacy?   perceived academic self-efficacy is defined as beliefs in or personal judgments of one’s capabilities to organize and execute the course of action required to attain designated types of education performances (bandura, 1977, 1995b). for this study, information self-efficacy is measured by levels of confidence that students can perform given tasks in information literacy.   methods   this study employed a cross-sectional research design. participants included 850 nursing students matriculating at the oakland university school of nursing and 325 medical students enrolled at the oakland university william beaumont school of medicine. the study was conducted with approval of the institutional review board (irb).   instruments   a survey was administered to nursing and medical students. it was comprised of three parts: demographic information, the jefferson scale of lifelong learning, and an information self-efficacy scale. the jefferson scale of lifelong learning is a validated instrument measuring the lifelong learning orientation of health professions students (jeffsll-hps) with 14 items (novak, palladino, ange, & richardson, 2014). respondents indicate their agreement or disagreement with each statement on a 4-point likert scale from 1 (strongly disagree) to 4 (strongly agree). total score is the sum of all item scores. higher scores indicate a more positive orientation that the student has toward lifelong learning. possible range of scores is from 14 to 56.   the information self-efficacy scale developed and used by one of the authors in her instructional sessions (mm) consists of 17 items examining students’ perceived level of self-efficacy for information literacy. it measures the degree of certainty that students can perform various information tasks in using library resources, searching for information, and in evidence-based practice. students were asked to rate their confidence in ability to execute these information skills on a 5-point likert scale from 1 (extremely unconfident) to 5 (extremely confident). total score is the sum of all item scores, ranging from 17 to 85. higher scores suggest a stronger self-efficacy belief in information skills. the scale also included four additional questions: one on frequency of information seeking for their coursework/ project, one on satisfaction with search results, and the other two on searching skills. participants had approximately 10 minutes to complete the survey. (the scale is available upon request from the authors).   procedure   two survey modes (print and online) are recommended to tailor a selfadministered survey procedure to specific situations and resource constraints at study locations (dillman, 2000). the print survey was distributed to first(m1) and second-year (m2) medical students at the end of one of their course sessions. third-year (m3) and fourth-year (m4) medical students had clinical rotations in various hospital settings. they received an online version of the same survey by email through surveymonkey. the print survey was distributed to nursing students, including undergraduates and graduate students in the master degree program (np), at the end of one of their class sessions. dnp (doctor of nursing practice) students who took courses online received an email message with a link to the online survey.   data analysis   data collected were checked for completeness and accuracy. they were entered into and analyzed with spss. internal consistency for the two scales was estimated using coefficient alpha to test their reliability. scale scores were summed up as an average of constituent items. anova, chi-square test, pearson correlations, and spearman rank-order correlations were computed to examine the relationships between students’ lifelong learning orientation, information skills, self-efficacy, and demographic characteristics.   results   participant demographics   out of 419 respondents to the survey (an overall response rate of 36%), 87.8% (368/419) completed the print survey while 12.2% (51/419) completed the online one via surveymonkey. the majority of the sample was nursing students, 68.5% (287/419) in comparison to medical students, 31.5% (132/419). detailed demographic information is illustrated in table 1.   psychometric properties of scales     table 1 participants’ demographics (n=419) variable frequency % medical students (n=132) 128          m1 51 12.2        m2 50 11.9        m3 13 3.1        m4 14 3.3 nursing students (n=287)            undergraduate students 209 49.9        graduate students (np) 72 17.2        graduate students (dnp)   10 2.4 age            18-25 220 52.5        26-36 132 31.5        37-47 47 11.2        >48 19 4.5 gender            male 98 23.4        female 321 76.6 race            american indian/alaska native 1 .2        asian/pacific islander 50 11.9        african american/black 23 5.5        hispanic/latino 8 1.9        caucasian/white 320 76.4        other 17 4.1 enrollment            full-time student 355 84.7        part-time student 63 15.0     table 2 correlations between jeffsll-hps total scores and information self-efficacy total scores   n r p all participants 388 .380 <.001* nursing students 265 .441 <.001* medical students 123 .345 <.001* *denotes statistical significance.     prior to running analyses using the jeffsll-hps and information self-efficacy scales, it was necessary to assess the reliability of these scales prior to creating total composite (additive) scores. there was a high degree of internal consistency for both the jeffsll-hps scale, with a cronbach’s alpha value of .82, and for the information self-efficacy scale, with a cronbach’s alpha value of .91.    correlation between lifelong learning and information self-efficacy   overall, there was a significant positive correlation between jeffsll-hps total scores and information self-efficacy total scores, r (386) = .380, p < .001, as illustrated in table 2. there was also a significant positive correlation between jeffsll-hps total scores and information self-efficacy total scores for both nursing students, r (263) = .441, p < .001, and for medical students, r (121) = .345, p < .001.   additionally, average jeffsll-hps total scores for nursing students (m = 43.04) were not significantly different from average jeffsll-hps total scores for medical students (m = 43.04), t (407) = -0.01, p = .995. however, average information self-efficacy total scores were significantly higher for nursing students (m = 64.01) than for medical students (m = 58.53), t (392) = 5.44, p < .001 (table 3).   demographic variables and lifelong learning and self-efficacy   a chi-square test of independence was conducted to assess the relationships between class status (nursing vs. medical) and gender (male vs. female). the results indicated a dependent relationship between class status and gender, c2 (1; p < .001) = 33.01. it was of additional interest to assess differences between genders for total scores of jeffsll-hps and information self-efficacy within class status. there were no significant gender differences for either nursing students or medical students in jeffsll-hps total scores or information self-efficacy total scores.   average jeffsll-hps total scores for undergraduate nursing students (m = 41.84) were significantly lower than average jeffsll-hps total scores for graduate nursing students (m = 46.20), t (283) = -6.89, p < .001 (table 4). average information self-efficacy total scores were significantly lower for undergraduate nursing students (m = 63.34) than those for graduate nursing students (m = 65.97), t (270) = -2.40, p = .031 (table 4).   it was also of interest to compare jeffsll-hps and information self-efficacy total scores across different cohorts of medical students. for this analysis, groups were classified as m1 medical students, m2 medical students, and m3/m4 year medical students. a one-way analysis of variance showed no significant differences among cohorts of medical students for jeffsll-hps total scores, f (2, 121) = 2.13, p = .123. however, for information self-efficacy total scores, m1 medical students (m = 55.62) and m2 medical students (m = 58.00) had significantly lower scores than m3/m4 year students (m = 64.42), f (2, 119) = 5.16, p = .007.     table 3 comparisons with means (standard deviations) of jeffsll-hps total scores and information self-efficacy total scores for nursing versus medical students   nursing (n = 281) medical (n = 128) p jeffsll-hps m = 43.04 (5.14) m = 43.04 (5.13) .995 information self-efficacy m = 64.01 (8.15) m = 58.03 (11.41) <.001* *denotes statistical significance.     table 4 comparisons with means (standard deviations) of jeffsll-hps total scores and information self-efficacy total scores for undergraduate versus graduate nursing students   undergraduate graduate p jeffsll-hps m = 41.84 (4.56) (n = 206) m = 46.20 (5.31) (n = 79) <.001* information self-efficacy m = 63.34 (7.57) (n = 198) m = 65.97 (9.25) (n = 74) .031 *denotes statistical significance.     information seeking frequency and lifelong learning and self-efficacy   question 18 of the information skills section of the survey gathered data on frequency of information seeking and contained response options ranging from “never” to “several times a day” that were rank ordered to reflect information seeking frequency (magnitude). for all respondents, there was a significant spearman rank-order correlation between information seeking frequency and jeffsll-hps scores, rs (391) = .195, p < .001, and between information seeking frequency and information self-efficacy, rs (379) = .125, p = .015.   for nursing students, there was a significant spearman rank-order correlation between information seeking frequency and jeffsll-hps scores, rs (266) = .206, p = .001, and between information seeking frequency and information self-efficacy, rs (256) = .157, p = .012.    however, for medical students, the rank-order correlations for frequency of information seeking were not significant for jeffsll-hps, rs (123) = .161, p = .072, or for information self-efficacy, rs (121) = .130, p = .152.   satisfaction with search results and lifelong learning and information self-efficacy   question 19 of the information skills section of the survey gathered data on satisfaction with search results. response options ranged from “very seldom” to “all of the time” and were rank ordered to reflect satisfaction with search results (magnitude). for all respondents, there was a significant spearman rank-order correlation between satisfaction and jeffsll-hps, rs (401) = .176, p < .001, and between satisfaction and information efficacy, rs (388) = .444, p < .001.   for nursing students, there was a significant spearman rank-order correlation between satisfaction and jeffsll-hps scores, rs (273) = .159, p = .008, and between satisfaction and information self-efficacy, rs (264) = .390, p < .001.   for medical students, there was also a significant spearman rank-order correlation between satisfaction and jeffsll-hps scores, rs (126) = .208, p = .019, and between satisfaction and information self-efficacy, rs (122) = .565, p < .001.   search skills and lifelong learning orientation and information self-efficacy   question 20 tested students’ skills in using boolean operators. a chi-square test of independence was conducted to assess the relationship between class status (nursing vs. medical) and accuracy (correct versus incorrect). the results indicated a dependent relationship between class status and accuracy, c2 (1; p < .001) = 20.17. medical students identified the correct answer with 67.4% response accuracy versus nursing students who identified the correct answer with 43.6% accuracy.   question 21 tested students’ skill in recognizing an effective search strategy. the results of the same statistical analysis indicated a dependent relationship between class status and response accuracy, c2 (1; p < .001) = 62.13. medical students identified the correct answer with 78.0% accuracy versus nursing students who identified the correct answer with 36.6% accuracy.   discussion   the findings of the study demonstrate a very good internal consistency reliability coefficient of .82 for the jeffsll-hps scale, which was in line with the findings of previous studies showing its sound psychometric properties (wetzel et al., 2010). there was also a satisfactory internal consistency reliability coefficient of .91 for the information self-efficacy scale, suggesting that it may be appropriate for use to examine self-efficacy beliefs of different groups of health professions students in their information literacy. among five sources of evidence to test validity of measures is relations to other variables (downing & haladyna, 2009). the significant correlation between the jeffsll-hps and information self-efficacy scales provides a source of convergent validity evidence for each of the scales.   the finding that students’ lifelong learning orientation was significantly correlated with their information self-efficacy indicates that students with a stronger lifelong learning orientation were more likely to have a higher level of self-efficacy beliefs in information skills. from the finding, we could hypothesize that information self-efficacy could affect students’ attitude toward lifelong learning or vice versa. this is the first study that has been conducted to establish the significant link between the two constructs. lifelong learning is considered an element of professionalism (arnold, 2002) and is critical for safe and competent patient care practice throughout the careers of all health care professionals (novak et al., 2014). accreditation bodies for health professions educational programs mandate educational experiences in curriculums that prepare students for lifelong learning as future healthcare providers. to promote lifelong learning, librarians should partner with health professions educators to develop instructional strategies to provide students with information mastery experiences to develop their strong sense of efficacy for information skills that could contribute to a strong lifelong learning orientation.   most of the studies dealing with self-efficacy had participants comprised of homogeneous samples of students (e.g., undergraduates, library school students). the present study is the first one that recruited heterogeneous groups of students including undergraduate nursing students, graduate nursing students in programs of master’s (np) and doctor of nursing practice degrees (dnp), medical students, and students taking classes on campus or online. clearly, it merits research on how students’ demographic variables (gender, class status, educational program) could have an impact on their lifelong learning orientation and information self-efficacy. the study examined these demographic variables in relation to the constructs of lifelong learning orientation and information self-efficacy.   there were no differences in lifelong learning orientation between nursing students and medical students, suggesting that both groups of health professions students may perceive lifelong learning with an equal value regardless of their diverse educational backgrounds and learning experiences in different health educational programs. while nursing students’ self-efficacy for information skills was significantly higher, their accuracy response rate for two skill assessment questions was significantly lower in comparison to medical students. medical students may possess higher entry level information skills than undergraduate nursing students. another possible explanation for such a difference is that nursing students may have an inflated view of their ability relative to their peers or counterparts. as pajares (2002) points out, “belief and reality are seldom perfectly matched…it is not unusual for individuals to overor underestimate their abilities.” kruger and dunning (1999) posited that people are likely to hold overly favorable views of their abilities in many social and intellectual domains. their finding that undergraduate students overestimated their test performance and ability led to the conclusion that people overestimated their abilities because they may not be in a position to accurately assess their skills due to a lack of skills in intellectual domains. maughan (2001) found that graduating seniors had a higher opinion of their ability to access information and to conduct library research than they were able to demonstrate by their test scores. other researchers confirm the notion that, with regard to assessing their own knowledge and skills, students demonstrated an overconfidence in their abilities (ehrlinger, johnson, banner, dunning, & kruger, 2008; monoi, o'hanlon, & diaz, 2005; ren, 2000). nursing students in the present study may have expressed overconfidence in their self-appraisal of their information skills, which concurs with the findings of previous studies. the significant difference in the accuracy response rate for the two questions may also be partly explained by possible differences in levels of exposure to formal information literacy instruction in curricula of nursing and medical education programs.   the results of the study show no significant difference in lifelong learning orientation and information self-efficacy between male and female students, which echo the conclusion of previous studies demonstrating no gender-based differences in information self-efficacy beliefs (bronstein, 2014; hojat et al., 2009; multon, brown, & lent, 1991; ren, 2000).   in terms of class status, graduate nursing students had a higher lifelong learning orientation and higher sense of information self-efficacy than undergraduate nursing students. graduate nursing students work independently in a clinical area while pursuing their graduate study; they must know how to retrieve clinical information for their practice. the authentic, concrete clinical experience propels them to apply their information skills to support evidence-based practice and perform clinical tasks. the clinical context fosters their mastery of information skills. according to bandura (1995b), mastery experience is the most effective way of creating a strong sense of efficacy. furthermore, graduate nursing students’ course work may be more demanding than undergraduates as the programs are likely to have a narrowly specialized focus and be more research-oriented. for this reason, graduate students may develop a strong sense of professional accomplishment. as a result, graduate nursing students may develop stronger efficacy beliefs in their information skills. the research evidence was in agreement with the findings of previous studies (hojat et al., 2009; muliira et al., 2012). hojat and colleagues (2009) studied physicians’ lifelong learning orientation in relation to their academic status (full-time clinicians with exclusive responsibility for patient care and academic clinicians with more involvement in teaching and research). they found a significant link between physicians’ lifelong learning scores and indicators of physicians’ learning motivation and professional accomplishments. muliira and his associates (2012) examined nurses’ orientation toward lifelong learning, reporting that it increased as they gained a higher level of education.   for medical students, however, lifelong learning orientation did not change across class levels, suggesting that medical students had a consistent view about the importance of lifelong learning over the course of their medical education. on the information self-efficacy scale, the combined group of m3/m4 medical students scored higher than m1 and m2 students respectively. for m1 and m2 students, learning mostly occurs in the classroom setting; while m3 and m4 students’ learning takes place in a clinical setting comparable to the clinical environment for graduate nursing students. the clinical learning experience and clinical patient care tasks promote information seeking to develop information mastery experience. as a result, m3 and m4 medical students may develop a strong sense of self-efficacy in their capabilities to access, retrieve, and use information resources. as bandura (1995a) pointed out, to remain task-orientated in the face of pressing demands and other trying situations, it is important to have a strong sense of efficacy. the learning environments for both graduate nursing students and medical students in their clinical years may account for their perceived higher self-efficacy in information skills in this study. further research with large samples of graduate nursing students and upper class medical students across institutions would need to confirm such findings, so information literacy instruction focusing on evidence-based searching skills could be developed and tailored to students’ learning needs in patient care settings.   tella and colleagues (2007) examined how undergraduates’ and postgraduates’ self-efficacy was associated with their use of electronic information and academic performance. they found that there was a significant correlation among the three variables and that students with high self-efficacy used electronic information more, and performed better on an aptitude test, than those with low self-efficacy. in another study on college students’ self-efficacy in electronic information searching before and after library instruction, ren (2000) discovered that frequent information users continued to hold relatively higher self-efficacy than non-frequent users prior to or following library training. the results of the present study suggest that the overall levels of nursing students’ lifelong learning orientation and information self-efficacy were predictive of students’ information seeking frequency. the significant correlation between the variables in this study was consistent with previous research findings (ren, 2000; tella et al., 2008). however, variation analysis reveals that medical students’ information seeking frequency was not strongly correlated with their lifelong learning orientation or information self-efficacy. the finding suggests that medical students’ lifelong learning orientation or self-efficacy may play a minimal role in affecting their information seeking frequency. on the contrary, nursing students’ lifelong learning orientation and efficacy beliefs may be strong determinants of their information seeking frequency. further research with an experimental research design would need to confirm the findings and establish a causal relationship between the variables.   one question on the information self-efficacy scale addressed the affective or emotional element of the information seeking process, namely students’ satisfaction with their search results. the study discovered that the more satisfied nursing or medical students were with their search results the stronger their lifelong learning orientation and self-efficacy beliefs in their information skills. further research should investigate how students’ affective state or emotion experienced during their search could exert a certain influence on their attitude to their lifelong learning and self-efficacy for information skills.   research limitations   it must be noted that there are several limitations inherent in this study. students volunteered to participate in the study, which could potentially create a risk of bias as volunteers may have been more likely to perceive positively their ability to succeed in library-related tasks. the self-reported nature of the study method may have led to social desirability bias in responses provided by participants who may have tried to appear highly oriented toward lifelong learning (novak et al., 2014) and show a higher level of perceived efficacy in performing information tasks. this is a single institutional study with a relatively low response rate that utilized a convenience sampling technique for recruiting participants; any findings of the study should be interpreted with caution and may not be generalizable to all populations of nursing and medical students across health education programs in different institutions.   implications for information literacy instruction   interprofessional education is gaining more attention in health professions education. health sciences librarians and health professions educators need to appreciate the similarities and differences among health care professionals to develop instructional interventions to optimize student success (horsburgh, lamdin, & williamson, 2001; reid, bruce, allstaff, & mclernon, 2006). information literacy instruction should be designed to tailor to learning needs and skill levels of health professions students in different learning contexts to promote the development of a strong lifelong learning orientation.   “the ultimate goal of the educational system is to shift to the individual the burden of pursuing his own education” (gardner, 1963, p. 21). to enable students to reach the self-education goal, zimmerman (1995) stresses that schools must go beyond teaching intellectual skills by developing students’ self-beliefs and self-regulatory capabilities to educate themselves throughout a lifetime. in addition to helping students acquire necessary knowledge and information skills to meet their study requirement, it is necessary for librarians to develop strategies or methods to increase and evaluate the level of students’ perceived self-efficacy regarding these knowledge and skills, and to investigate how self-efficacy beliefs affect information problem solving and lifelong learning skill building (kurbanoglu, 2003).   the findings of the present study suggest implications for librarians seeking approaches to     advancing the value and utility of information literacy instruction in educational curricula. as such instruction has the potential to lead to high levels of information self-efficacy associated with lifelong learning. strategies should be developed and incorporated into the instruction to cultivate students’ information self-efficacy. examples of strategies include encouraging students to self-observe and reflect on their use of information resources, search terms, and search process; helping students to become more personally aware of their improved information searching effectiveness and capability; creating checklists for students to self-regulate or monitor their learning and performances of given tasks. the lifelong learning and information self-efficacy scales that have been validated in this study may be used as alternative measures to assess the affective dimension of information mastery, which is an important but often overlooked aspect of information literacy instruction (monoi et al., 2005).   conclusions   the present study yields evidence demonstrating that health professions students with a higher level of self-efficacy for information skills tended to have a stronger lifelong learning orientation, and that their information use and satisfaction with their searches were associated with their strength of self belief in performing various information tasks. the study contributes to the body of literature dealing with information literacy education to foster students’ information self-efficacy. such an efficacious outlook would play an important role in the sustainable development of health professions students’ lifelong learning.   acknowledgement   a special thanks to dr. patrick doyle for his assistance with data analysis and presentation.     references   afonso, p., ramos, m. r., saraiva, s., moreira, c. a., & figueira, m. l. (2014). assessing the relation between career satisfaction in psychiatry with lifelong learning and scientific activity. psychiatry research, 217(3), 210-214. doi: 10.1016/j.psychres.2014.03.044   arnold, l. (2002). assessing professional behavior: yesterday, today, and tomorrow. academic medicine, 77(6), 502-515.   association of college & research libraries. (2000). information literacy competency standards for higher education. retrieved from http://www.ala.org/acrl/sites/ala.org.acrl/files/content/standards/standards.pdf   bandura, a. (1977). self-efficacy: toward a unifying theory of behavior. psychological review, 84(2), 191-215.   bandura, a. (1986). social foundation of thought and action: a social cognitive theory. englewood cliffs, nj: prentice hall.   bandura, a. (1995a). exercise of personal and collective efficacy in changing societies. in a. bandura (ed.), self-efficacy in changing societies (pp. 1-45). cambridge; new york: cambridge university press.   bandura, a. (1995b). self-efficacy in changing societies. new york, ny: cambridge university press.   benner, p. (1984). from novice to expert: excellence and power in clinical nursing practice. upper saddle river, nj: prentice hall.   bronstein, j. (2014). the role of perceived self-efficacy in the information seeking behavior of library and information science students. the journal of academic librarianship, 40(2), 101-106.   downing, s. m., & haladyna, t. m. (2009). validity and its threats. in s. m. downing & r. yudkowsky (eds.), assessment in health professions education (pp. 21-55). new york, ny: routledge.   duff, p. (2002). professionalism in medicine: an a-z primer. obstetrics and gynecology, 99(6), 1127-1128.   ehrlinger, j., johnson, k., banner, m., dunning, d., & kruger, j. (2008). why the unskilled are unaware: further explorations of (absent) self-insight among the incompetent. organizational behavior and human decision processes, 105(1), 98-121.   gardner, j. w. (1963). self-renewal: the individual and the innovative society. new york, ny: harper & row.   hojat, m., nasca, t. j., erdmann, j. b., frisby, a. j., veloski, j. j., & gonnella, j. s. (2003). an operational measure of physician lifelong learning: its development, components and preliminary psychometric data. medical teacher, 25(4), 433-437. doi:10.1080/0142159031000137463   hojat, m., veloski, j., nasca, t. j., erdmann, j. b., & gonnella, j. s. (2006). assessing physicians' orientation toward lifelong learning. journal of general internal medicine, 21(9), 931-936. doi:10.1111/j.1525-1497.2006.00500.x   hojat, m., veloski, j. j., & gonnella, j. s. (2009). measurement and correlates of physicians' lifelong learning. academic medicine, 84(8), 1066-1074. doi:10.1097/acm.0b013e3181acf25f.   horsburgh, m., lamdin, r., & williamson, e. (2001). multiprofessional learning: the attitudes of medical, nursing, and pharmacy students to shared learning. medical education, 35(9), 876-883.   institute of medicine. (2010). the future of nursing: leading change, advancing health. washington, dc: national academies press. retrieved from http://iom.nationalacademies.org/reports/2010/the-future-of-nursing-leading-change-advancing-health.aspx   kruger, j., & dunning, d. (1999). unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead to inflated self-assessments. journal of personality and social psychology, 77(6), 1121-1134.   kurbanoglu, s. s. (2003). self-efficacy: a concept closely linked to information literacy and lifelong learning. journal of documentation, 59(6), 635-646. http://dx.doi.org/10.1108/00220410310506295   liaison committee on medical education. (2013). functions and structure of a medical school: standards for accreditation of medical education programs leading to the m.d. degree. edmonton, ab: university of alberta.   novak, m.k., palladino, c., ange, b., & richardson, d. (2014). measuring health professions students' orientation toward lifelong learning. journal of allied health, 43(3), 146-149.   maughan, p. d. (2001). assessing information literacy among undergraduates: a discussion of the literature and the university of california-berkeley assessment experience. college & research libraries, 62(1), 71-85. doi:10.5860/crl.62.1.71   monoi, s., o'hanlon, n., & diaz, k. r. (2005). online searching skills: development of an inventory to assess self-efficacy. the journal of academic librarianship, 31(2), 98-105.   muliira, j. k., etyang, c., muliira, r. s., & kizza, i. b. (2012). nurses' orientation toward lifelong learning: a case study of uganda's national hospital. journal of continuing education in nursing, 43(2), 90-96. doi:10.3928/00220124-20111003-03   multon, k. d., brown, s. d., & lent, r. w. (1991). relation of self-efficacy beliefs to academic outcomes: a meta-analysis investigation. journal of counseling psychology, 38(1), 30-38. http://dx.doi.org/10.1037/0022-0167.38.1.30   nelson, a. r. (1998). medicine: business or profession, art or science? american journal of obstetrics and gynecology, 178(4), 755-758.   pajares, f. (2002). overview of social cognitive theory and of self-efficacy. retrieved from http://www.uky.edu/~eushe2/pajares/eff.html   reid, r., bruce, d., allstaff, k., & mclernon, d. (2006). validating the readiness for interprofessional learning scale (ripls) in the postgraduate context: are health care professionals ready for ipl? medical education, 40(5), 415-422.   ren, w. (2000). library instruction and college student self-sufficiency in electronic information searching. journal of academic librarianship, 26(5), 323-328.   rishel, c. j. (2013). professional development for oncology nurses: a commitment to lifelong learning. oncology nursing forum, 40(6), 537-539. doi:10.1188/13.onf.537-539   tang, y., & tseng, h. w. (2013). distance learners' self-efficacy and information literacy skills. the journal of academic librarianship, 39(6), 517-521.   tella, a., tella, a., ayeni, c. o., & omoba, r. o. (2007). self-efficacy and use of electronic information as predictors of academic performance. electronic journal of academic and special librarianship, 8(2). http://southernlibrarianship.icaap.org/content/v08n02/tella_a01.html   wetzel, a. p., mazmanian, p. e., hojat, m., kreutzer, k. o., carrico, r. j., carr, c., . . . rafiq, a. (2010). measuring medical students' orientation toward lifelong learning: a psychometric evaluation. academic medicine, 85(10 suppl.), s41-s44. doi:10.1097/acm.0b013e3181ed1ae9   woodruff, d. w. (2012). the necessity of lifelong learning. medsurg matters!, 21(3/4), 12.   zimmerman, b. j. (1995). self-efficacy and educational development. in a. bandura (ed.), self-efficacy in changing societies. new york, ny: cambridge university press.   microsoft word es 4553_new evidence based library and information practice 2009, 4:1 24 evidence based library and information practice evidence summary purchase of journal portfolios by research libraries is not cost-effective and may lead to normalization of collections a review of: murphy, sarah ann. “the effects of portfolio purchasing on scientific subject collections.” college & research libraries july 2008: 332-40. reviewed by: stephanie walker acting chief librarian brooklyn college – city university of new york brooklyn, ny usa email: swalker@brooklyn.cuny.edu received: 14 december 2008 accepted: 12 february 2009 © 2009. walker. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. objectives – to determine whether the purchase of journal portfolios (i.e., packages of journals purchased as a group from publishers, such as elsevier’s sciencedirect) from publishers is an effective means of meeting research needs for faculty in the life, medical, physical, and applied sciences, and to determine the effects of such purchases on research library collections. design – citation analysis. setting – ohio state university libraries in the life, medical, physical, and applied sciences. subjects – a total of 253,604 citations from 6,815 articles published between the years 2003 and 2005 by ohio state university faculty in the life, medical, physical, and applied sciences were analyzed using the bradford distribution (an explanation of the bradford distribution is provided later in this review). methods – using isi’s science citation index, the author generated a list of articles published by ohio state university (osu) faculty in the life, medical, physical, and applied sciences between the years 2003 and 2005. the author then assigned each article to a specific discipline, according to the osu college of the first osu author listed. for example, if an article was written by several co-authors, and the first osu author listed was a faculty member in osu’s college of dentistry, the article would be designated a dentistry article. multidisciplinary works were assigned to the college of the first osu author listed. (the osu colleges evidence based library and information practice 2009, 4:1 25 considered to be part of the study were the college of biological sciences; the college of dentistry; the college of engineering; food, agriculture, and environmental sciences; the college of mathematical and physical sciences; the college of medicine; the college of optometry; the college of pharmacy; and the college of veterinary medicine.) books, conference proceedings, theses, and other nonjournal materials were excluded from the set of citations considered. next, the author pulled journal citations from each article, again utilizing science citation index. the references were analyzed to determine the number of times each individual journal had been cited. the author then created a list of journals which had been cited in articles by osu faculty in the various colleges, grouped by college. the journals were arranged in descending order, according to the number of times each journal had been cited. thus there would be, for example, a list of all journals cited in articles published by faculty members in the osu college of dentistry between 2003 and 2005. most journals had been cited only once over the three-year period. a total of 2,407 journal titles were cited 10 or more times. in total, the author analyzed 253,604 citations from 6,815 articles. a bradford distribution of journal citations was calculated, and journals were divided into three categories. the three categories were called zones 1, 2, and 3, with zone 1 being core journals for the faculty, zone 2 being more secondary titles, and zone 3 being those cited least frequently. for those not familiar with this type of analysis, a definition of bradford’s law is available on the u.s. national institute for standards and technology website. it is included here for ease of reference: “journals in a field can be divided into three parts, each with about one-third of all articles: 1) a core of a few journals; 2) a second zone, with more journals; and 3) a third zone, with the bulk of journals. the number of journals is 1:n:n². note thatbradford formulated his law after studying a bibliography of geophysics, covering 326 journals in the field. he discovered that 9 journals contained 429 articles, 59 contained 499 articles, and 258 contained 404 articles. although bradford's law is not statistically accurate, librarians commonly use it as a guideline” (black). the author then determined how the osu libraries purchased access to each title. the three options analyzed were: 1) through ohiolink (through which osu libraries purchase the bulk of the journal portfolios to which they subscribe), 2) through the independent purchase of an electronic subscription, or 3) through the independent purchase of a print subscription. the cost for each title was calculated by taking the amount paid for ohiolink subscriptions and removing the cost of non-scientific journals from the total amount. pricing for the nonscientific journals was obtained using ebsco’s librarian’s handbook 2006-2007 and ulrich’s periodical directory. to account for inflation, any 2007 prices were adjusted by 6. the above activities were designed to calculate both the cost of each title as purchased through ohiolink, and what the osu libraries would have paid for each individual title if it had been purchased separately. main results – of all journals cited by osu faculty in the life, medical, physical, and applied sciences during the years studied, only 7% were available in print format only. the percentage of cited journal titles that were included in portfolio purchases varied considerably across the colleges. the college for which the greatest percentage of cited journals were obtained via ohiolink was the college of mathematical and physical sciences; 85.7% of journals cited by this evidence based library and information practice 2009, 4:1 26 college were purchased via ohiolink. overall figures for the cited journals analyzed were as follows: 52.0% were purchased via ohiolink portfolio purchases, and 26.3% were purchased individually in electronic format by the osu libraries. of all journals listed in zone 1 (those designated as “core journals” for the fields in question), 100% had electronic versions, though osu libraries continued to subscribe to the print version in addition to the electronic version for five titles, due to embargoes of 4-12 months in the electronic subscriptions. in terms of how the zone 1 journals were acquired, 35.5% were purchased via ohiolink as part of a portfolio purchase, and 62.2% were individually purchased. for the college of biological sciences; the college of food, agricultural, and environmental sciences; the college of medicine; the college of nursing; the college of pharmacy; and the college of veterinary medicine, fewer than 40% of the zone 1 (core, most highly cited) titles for their disciplines were purchased via ohiolink. for the college of mathematical and physical sciences, 75.5% of zone 1 titles were purchased via ohiolink. this figure was 60.5% in the college of engineering. by contrast, over 50% of the titles in zone 1 for the colleges of dentistry, nursing, pharmacy, and veterinary medicine were purchased individually, and not via portfolio purchases from ohiolink. the author notes that in these fields, the majority of research is published in journals from professional societies or smaller publishers, which have neither the high profile nor the market that some journals in other fields have, and thus are frequently not included in portfolios available via consortia such as ohiolink. the author also provides a numerical breakdown, showing exactly how many titles in each of zones 1, 2, and 3 were purchased via ohiolink, how many were purchased directly by osu in electronic form, and how many were purchased in print form, for each college and for all colleges combined. for all colleges combined, the overall results are as follows: • zone 1 included 45 cited journal titles. of these, 16 were purchased via ohiolink, 28 were purchased in electronic format directly by osu, none were purchased in print, and one was considered “other” (not at osu, ceased, or cancelled). • zone 2 included 299 cited journal titles. of these, 167 were purchased via ohiolink, 109 were purchased in electronic format directly by osu, 13 were purchased in print, and 10 fell under “other”. • zone 3 included 2,063 cited journal titles. of these, 1,068 were purchased via ohiolink, 497 were purchased in electronic format directly by osu, 155 were purchased in print, and 343 fell under “other”. the author also provides a list of the top 50 journals cited, including the number of citations linked to each title and how the title was purchased. of the top 50 journals, 32 were purchased directly by osu libraries in electronic format, and only 18 were purchased via ohiolink. interestingly, however, 70% of osu libraries’ total expenditures on titles in the life, medical, physical, and allied sciences are devoted to ohiolink. the author notes that if osu had not had ohiolink, they would have paid 61.4% more to directly purchase the journals cited in this analysis which they currently obtain by portfolio purchases. however, if they purchased only those titles which the faculty in question had cited 10 or more times, the cost would be 30% more. if they purchased only the titles which had been cited 15 or more times, osu would only have paid an 8.9% premium to buy the titles directly from the publisher rather than through ohiolink. conclusion – as the author points out, her findings raise the question as to whether the large amount of content provided by buying evidence based library and information practice 2009, 4:1 27 into the “big deal” portfolio purchases (as they are frequently called) is really worth it for osu libraries. the author notes that other articles have asserted that portfolio purchases form a significant barrier to libraries wishing to purchase individual titles, as the amount spent on portfolio purchases can limit a library’s financial flexibility. even when other individual titles may more closely meet faculty needs, it can be difficult to justify cancelling portfolio purchases that offer a larger number of journals in the field. the advantages and disadvantages of portfolio purchasing at ohio state university libraries are clear from the author’s research: while some fields are well-served by portfolio purchases, others are not, with large percentages of the journals which are most important in their fields not being available through such portfolios. furthermore, due to the percentage of the osu libraries’ budget dedicated to ohiolink portfolio purchases, flexibility to purchase titles not in portfolios is indeed limited. the author’s pricing calculations lead to the conclusion that osu libraries pay between an 8.9%-30.0% premium to maintain access to 3,813 titles (75.4%) which were cited fewer than 10 times over the three year period between 2003 and 2005. the author concludes that the premium paid to access over three-quarters of the journals available in portfolios should be reconsidered, as they are relatively infrequently used and thus may not be meeting faculty research needs. the author recommends that large research libraries (including osu libraries) consider a return to à la carte purchasing. additionally, the author notes that purchase of portfolios by a large percentage of research libraries may lead to normalization of library collections and loss of the ability to support non-commercial publishers who publish strong research in specialized fields. commentary the author does not explain why a journal would need to be cited specifically 10 times or more over a three year period in order to be considered a zone 1 journal. at another point in the article, fifteen citations is suggested as a benchmark which might be used to justify the purchase of a particular title; the rationale for choosing either number is not specified. the methods used by the author to determine pricing of individual journals within portfolios seems reasonable, but there could be variations among journal titles or between fields. this reviewer acknowledges the difficulty of determining a reasonable calculation for the pricing of an individual electronic journal within a portfolio. it is somewhat less difficult to determine the price of a print journal, and the author’s method of averaging the price discount for previous years appears reasonable, but again, there could be great variations among individual titles. it might have proven more accurate if the librarian’s handbook and ulrich’s periodical directory for each of the years between 2003 and 2005 had been consulted, though this would undoubtedly have been more laborintensive. these issues, however, are not sufficiently substantial as to invalidate the conclusions of the study. the author’s conclusions will most likely be of greatest interest to large academic research libraries and library systems comparable to those of osu. similar analyses of the citation patterns prevalent at other large academic research institutions could be quite valuable, and it would be worthwhile to see if the numbers are as striking at other institutions. one might not expect major differences, but the types of colleges, the disciplines studied, and the research undertaken at other universities might provide interesting comparisons. it would also be interesting to undertake similar studies at small and mid-size institutions. some libraries which are not part of large research systems but which nevertheless support a number of evidence based library and information practice 2009, 4:1 28 graduate programs in specific fields also subscribe to big deal portfolios. it would be interesting to analyze citations in articles published by faculty at these institutions to determine whether their research needs are better served by the portfolios (which undoubtedly do allow libraries to purchase access to a larger number of journal titles overall), or whether they too would be better served by returning to individual title purchasing practices, at least in certain fields. the author also notes that some libraries, most notably those of cornell university, have cancelled portfolio purchases from at least one large publisher (elsevier) and others (such as the university of wisconsin-madison) have never been involved in portfolio purchasing. it would also be interesting to obtain further information on factors or studies which informed these decisions, and to compare these to the work done by the author at ohio state; this information might usefully inform further research by other academic libraries. in any case, libraries might be well advised to examine the benefit to their own faculty of the big deal portfolio purchases, and to use the information thereby gleaned to inform their own purchasing practices. work cited black, p.e. "bradford's law." dictionary of algorithms and data structures, paul e. black, ed. 17 december 2004. u.s. national institute of standards and technology. 8 january 2009 <http://www.itl.nist.gov/div897/sqg/dad s/html/bradfordslaw.html>. evidence based library and information practice 2012, 7.2 73 evidence based library and information practice evidence summary librarian and lis faculty participation in self-archiving practice needs improvement a review of: xia, j., wilhoite, s. k., & myers, r. l. (2011). a “librarian-lis faculty” divide in open access practice. journal of documentation, 67(5), 791-805. doi:10.1108/00220411111164673 reviewed by: annie m. hughes reference librarian wilson dental library, university of southern california los angeles, california, united states of america email: amhughes@usc.edu received: 1 mar. 2012 accepted: 18 apr. 2012 2012 hughes. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to compare librarian and lis faculty open access (oa) self-archiving behavior. design – logistical regression analysis. setting – twenty top-ranked library and information science journals published in 2006. subjects – a total of 812 research articles in lis journals. methods – for the purpose of data collection, the researchers first compiled a list of library and information science journals utilizing journal citation reports (jcr) from 2006. twenty journals were selected by considering impact factor and the list compiled was checked against ulrich’s periodicals directory following a similar methodology utilized in a 2007 article by anita coleman. the sample included was representative of both library and information science journals, and there were exclusions of 3 types of journals: free online journals where oa participation could not be measured; subscription based journals that do not supply free articles; and annual review journals. here, oa participation or oa practice is considered to be author selfarchiving of articles that are not freely available online. research articles were included in the sample; however, editorials and book reviews were excluded. the researchers also collected information about the article itself, including the title, name of the journal and name of the author. only first author’s status as librarian or lis faculty was considered in data collection. one difficulty in mailto:amhughes@usc.edu evidence based library and information practice 2012, 7.2 74 collecting data about the authors was that their professional status was not always clear. the researchers collected information on whether the author’s status was librarian or faculty; when an author’s status was unclear, researchers searched online to determine it. if the author’s status still could not be determined via online searching, the authors chose to exclude those articles. after the articles were collected, google scholar was searched in order to determine oa status. the articles that were deemed oa were opened and if the article was downloadable, it was included; otherwise, it was not included. researchers also avoided linking to articles through their own library portal which would have allowed for access to articles through their own library’s subscription. other data was collected using web of science and included citation information; length of articles; and number of references, authors, and self-citations. analysis of data was performed utilizing logistic regression. the researchers selected the professional status (librarian or faculty) as the dependent variable, assigning 1 to librarian status and 0 to faculty status. the independent variables included the oa status of an article, citation count, self-citation counts, number of authors, length in pages, and number of references. researchers also chose to normalize data by adding one citation to every article in the dataset because many did not receive citations. spss was utilized as the statistical analysis tool. main results – researchers were able to recognize a divide with regard to where librarians and librarian faculty publish. librarians tend to avoid journals that focus on information science and publish more in journals related to the practice of librarianship. after performing the logistic regression analysis, researchers also chose to look at the “dichotomous dependent variable” and the “dichotomous predictor variable.” the dependent variable was article availability in oa and the predictor variable is professional status of the author. they ran a 2x2 contingency table and the p-value was below 0.05; therefore, there was a failure to reject the null hypotheses that there is no difference between librarian and faculty publication behaviors. an odds ratio was also calculated that reveals that librarians are only 1.029 times more likely to self-archive their articles than faculty. results from the logistic regression model analysis also included information that librarians and faculty have similar behavior with regard to self-archiving the oa version of their publication, and the researchers provide data in table format in order to depict the relationships between predictor variables and dependent variables. with regard to author status and citation counts, faculty have more citation numbers, self-citations are not related to author status, and faculty tend to self-cite more often. librarians favor co-authorship more than faculty, and faculty articles tend to be lengthier and utilize more references. effectiveness of the logistic regression analysis was tested using the hosmer-lemeshow test for goodness of fit and showed that logistic regression analysis was the proper method for analyzing data collected with a 74.8 success rate. according to the article, there is no difference between librarian and faculty with regard to self-archiving oa activity; faculty members receive more citations regardless of oa status; and there are differences between the two with regard to other variables of articles. conclusion – because librarians and lis faculty are considered to be advocates for selfarchiving and are often involved with institutional repositories, it is surprising that they are not themselves participating more in self-archiving behaviors. commentary the actual results of this study are much narrower than the researchers imply. the study discusses oa practice and the authors’ language leads one to believe that the scope of evidence based library and information practice 2012, 7.2 75 the article is larger. oa practice in this article means “librarian and lis faculty self-archiving behavior,” but this was not always made clear throughout. the differentiation between librarian and faculty was also unclear. it would be beneficial for the authors to clearly state whether the librarians included in the study are academic librarians who are not faculty status, or if they are practitioners outside of the academic arena. oa may not be as much of a concern for librarians who are not working in an academic setting. the researchers identified limitations with regard to their methodology. one issue is that they were not completely able to identify whether an article was oa. only google scholar was used to search for articles and no other databases were considered. another limitation is that they had difficulty at times discovering the status of an author and also did not consider the status of co-authors, which may skew the data. the data collected regarding oa citation numbers and consumption is interesting, but the main research question stated in the article is whether a librarian-faculty divide exists in oa contribution with regard to article selfarchiving. analyzing data related to oa article consumption would be better presented in another article and the focus should remain on the analysis of oa self-archiving between the two groups. while there are some questions with regard to data collection, the data analysis was executed nicely. the researchers decided to employ a direct measure instead of using a survey instrument to determine whether there is a divide between librarians and faculty. they chose to use a logistic regression analysis and then went on to utilize a goodness-of-fit test, hosmer-lemeshow, which indicated that it was a good model to use for the purposes of this paper. the main goal of this article was to present a divide between librarians and lis faculty with regard to self-archiving. while there is no significant difference between the two groups, the research does present that self-archiving behavior is happening less than one would think. the results of the research could lead to implementation of self-archiving policies or mandated contribution to institutional or digital repositories. librarians and lis-faculty tend to educate scholars about self-archiving, but perhaps more education must take place within their own profession. references coleman, a. (2007). self-archiving and the copyright transfer agreements of isiranked library and information science journals. journal of the american society for information science and technology, 58(2), 286-296. article   embedded, participatory research: creating a grounded theory with teenagers   shannon crawford barniskis doctoral student school of information studies university of wisconsin-milwaukee milwaukee, wisconsin, united states email: crawfo55@uwm.edu   received: 7 sept. 2012     accepted: 7 jan. 2013      2013 crawford barniskis. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – this project, based on a study of the impact of art programs in public libraries on the teenaged participants, sought to show how library practitioners can perform embedded, participatory research by adding participants to their research team. embedded participatory techniques, when paired with grounded theory methods, build testable theories from the ground up, based on the real experiences of those involved, including the librarian. this method offers practical solutions for other librarians while furthering a theoretical research agenda.   methods – this example of embedded, participatory techniques used grounded theory methods based on the experiences of teens who participated in art programs at a public library. fourteen teens participated in interviews, and six of them assisted in coding, analyzing, and abstracting the data, and validating the resulting theory.   results – employing the teenagers within the research team resulted in a teen-validated theory. the embedded techniques of the practitioner-researcher resulted in a theory that can be applied to practice.   conclusions – this research framework develops the body of literature based on real-world contexts and supports hands-on practitioners. it also provides evidence-based theory for funding agencies and assessment. in addition, practitioner-based research that incorporates teens as research partners activates teens’ voices. it gives them a venue to speak for themselves with support from an interested and often advocacy-minded adult.     introduction   recently i studied how art programs in public libraries affect teens, using a metric of civic engagement. i was curious to learn how teenagers describe the impacts of the manga-drawing, poetry-writing, and craft-making classes offered at many public libraries. were these programs just fun for individual participants? were they for the purpose of making wonderful art? both “fun” and “great art” are perfectly acceptable intrinsic reasons for individuals to participate in art programs in libraries, but do programs like these positively impact the teens as a group? are there extrinsic, community-wide benefits to arts programming? this line of research has implications that could shift how funders view programs that they sometimes consider as outside of a public library’s mission. but the most interesting part of this study was the unique research method my team used. my research team was not a bevy of professionals: it was the teens whom i was studying, and me, an embedded researcher-practitioner. over the course of the research project, the teens and i built a grounded theory that described their experiences of the programs, and discovered how the programs increased social capital (crawford barniskis, 2012b), using embedded, participatory techniques. these techniques, when paired with grounded theory methods, build testable theories from the ground up, based on the real experiences of those involved, including the librarian. this method offers practical solutions for other librarians while furthering a theoretical research agenda.   in line with the reflexive nature of this research, i am writing in the first person, as an “embedded author” telling a story in which i am not merely a narrator, but a co-creator of meaning.   background   in library and information science (lis), as in many fields, practitioners do little research, and those working in public libraries do even less. even though they comprise 28 percent of all librarians (bureau of labor statistics, 2012), public librarians have contributed at most 3.37 percent of the research (buttlar, 1991). the latest published count by penta and mckenzie (2005) finds that public librarians contribute a mere 3 percent to the general lis literature. jasist, the largest publisher of scholarly lis work, published work by 1,011 authors, according to the penta and mckenzie study. of those 1,011, not one was a public librarian. in 2010, of 405 articles in 11 top lis journals, only 2 percent of the authors identified solely as public librarians. those public librarians who did publish research published only in public-library-specific journals, even though the subjects they studied were often applicable to a wider audience (crawford barniskis, 2010). academics do the research in the lis field, and they are generally academic faculty, not practitioners.   as moeller, pettee, and leeper (2011) point out, the lack of the “teen voice” in library research is especially problematic; they call for teens to participate in the research process, and for their points of view to be fully represented:   those of us who are granted the privilege by young adults to use their voices in our research know that their voices give our work depth, interest, and relevance. those who further engage their young adult participants in the process of member-checking more soundly validate their research and the young adult’s experience. to produce robust, informative research, we as scholars need to talk to young adults, not just about them (para. 6).   while i did this research before this call for more thoroughly integrating teens in studies, this method is a response to the same concerns of marginalization. however, the embedded, heuristic research method i used could be effective with adult populations as well. there is no other evidence of lis studies done by public librarians as embedded and participatory members of the group being studied, nor other studies in which the participants under study are active members of the research team.   literature review   the concept of the embedded researcher is not new. anthropologists have incorporated themselves into the environments under study using naturalistic field methods since the 1970s (clarke, 1975). since the  1960s, research methodologies and their epistemic justifications have become more connected, and more aware of contingencies and biases (code, 1997; robson, 2011). researchers practicing embedded techniques are able to be more aware and connected than aloof, “unbiased” research approaches may require. sociologist valerie jenness (2008) defines an embedded researcher as similar to an embedded journalist, one who is “occupying multiple locations within and under the control of a single field of play while also moving from one site to another, one host to another, one level of analysis to the other, and one constituency to another” (p. 6). jenness describes the work embedded researchers do as “systematic back-translation” (p. 4).   in this article, the term “embedded researcher” reflects the research i did while also acting as a practitioner. in this framework, the researcher uses heuristic techniques to derive a praxis-based result, which is verified in partnership with the people being researched. this definition of “embedded” is different from “embedded design” methodologies, in which one set of data supports or explicates the other (creswell, 2012). instead, this is collaborative research in which a researcher is a practitioner, or is temporarily submersed in a practitioner’s environment. there are several related or overlapping frameworks that researchers use instead of, or in addition to, “embedded research.”   table 1 describes several types of methodologies that have one or more points in common with the type of embedded, participatory research described in this article. in each of these methods, the researcher participates in the experience under study, utilizes a practice-informed framework to guide the study, or is particularly sensitive to the context of the experience under investigation. these research paths can stem from critical analysis of power inequities by foucault, bourdieu, and friere (leckie, given, & buschman, 2010), which offer useful paradigms for examining teen perspectives or the perspectives of other marginalized groups. the pragmatic nature of these approaches means that whichever framework a researcher chooses, the end result will have some immediate utility to practitioners. however, the embedded, participatory framework described in this article combines many of these frameworks into a single method.   i will not delve deeply into the epistemic pros and cons of the embedded researcher model i espouse here, but i will touch on a few points. the justification for this sort of idiographic knowledge goes back to the qualitative/quantitative break that many researchers analyze and the tension between positivist and constructivist methodologies.      table 1 examples of similar frameworks and methodologies similar frameworks and methodologies potential similarities potential differences descriptions in the literature heuristic research hands-on, practice-informed research design. not necessarily embedded researcher, may be interventive or modelling. (poulter, 2006) interdisciplinary or transdisciplinary research often praxis-based, using multiple frameworks, hermeneutics. may be quantitative, no embedded researcher necessary. (wickson, carew, & russell, 2006) situated research may mean ethnographically situated in the context being studied. may also mean “situated” in the sense that the research or theory is situated in the larger corpus of research (e.g., what i am doing here with the “embedded researcher” concept). (miller & goodnow, 1995) action research often involves a practitioner-researcher; results in practical recommendations for action. may be only a local solution, not generalizable or abstracted. (kuhne & quigley, 1997) participatory research like action research, can “enable local people to seek their own solutions according to their priorities.” allows those being researched to also be in the powerful position of researcher. may be only a local solution, not generalizable or abstracted. (cornwall & jewkes, 1995, p. 1668) design-based research similar to action research, a practical partnership between researcher and practitioner. focused on interventions, “doing,” and measuring rather than simply examining. (anderson & shattuck, 2012)       the benefits of participatory, embedded research include:   ·         this technique builds relationships between concepts, academics and practitioners, and the researchers and the researched. ·         it reduces power disparities between practitioners and researchers (nutley, jung, & walter, 2008). ·         researchers have a framework and impetus for increased reflexivity and examination of biases (hoskins, 2000; miller & goodnow, 1995; reis, 2011). ·         the research is more relevant for practitioners, and may be more relevant for library boards or funding agencies. ·         this technique incorporates the practitioner’s tacit knowledge into the research, both methodologically and as an actionable response to the analysis (miller & goodnow, 1995). ·         researchers who are aware of the subtleties of the researched environment may more sensitively translate experiences of marginalized populations (li, 2008). ·         the research narrows the gap between the actual and the empirical, and the events and the studied, recalled experiences (tsoukas, 1989).   some of the limitations include:   ·         the contingent nature of this research requires significant abstraction for generalizability. ·         sometimes interactions can feel awkward or forced (hoskins, 2000). ·         practitioners or “teachers are usually too busy and often ill trained to conduct rigorous research” (anderson & shattuck, 2012, p. 17). ·         challenges abound regarding the “soft” nature of any qualitative research in its credibility and bias, but especially one in which the researcher lacks “scientific” distance from the researched. ·         while collaborative methods can bridge the lack of research knowledge of practitioners and the lack of tacit knowledge of researchers, such collaboration is costly, time-intensive, or unlikely to occur in locations where there are no local academics interested in partnerships.   neither the limitations nor the benefits of being an embedded researcher can be ignored, but one benefit is paramount: adolescents, in general, perceive that they lack a voice, power, and advocates (e.g., levine, 2008; marshall & arvay, 1999; ross, 2006). practitioner-based research that incorporates teens as research partners activates the teens’ voice. it gives them a venue to speak for themselves with the support of an interested and often advocacy-minded adult.   methods   the research project used in this study generated a grounded theory through a qualitative case study structure (crawford barniskis, 2012b). i advertised for applicants in several media outlets, through library flyers, and in the morning announcements at the local high school, and chose 11 teenagers out of a pool of 20 applicants. those chosen represented the widest-possible range of demographics in age, gender, family income level, church attendance, volunteerism, language spoken at home, and educational background of parents and teens. the group included five boys and six girls, ages 12 to 18, though one boy had to leave the study due to scheduling conflicts. some were regular library users; some had never been to the library. two lived in different towns entirely. this group of teens had the opportunity to participate in six art programs at the library and earn a ten-dollar gift card for each weekly program they attended. they attended at least five programs and the focus group interview after the program series ended. the art was varied, including poetry readings, a modern dance performance, manga and graffiti workshops, and artist trading card and photography classes. many other teens attended these programs as well.   at first the participants didn’t know which social aspects were under investigation. at the beginning of the first program, the teens filled out a survey asking how strongly they agreed with statements such as, “i feel like a valued member of my community” and “i think playing sports or exercising regularly is important.” at the end of the final program they filled out the same survey. i never intended the preand post-test surveys to generate statistically significant data because the sample size was too small. instead, i illustrated the survey data using comparative pie charts, which were a jumping-off point for discussion during the focus group and individual interviews.   the focus group interview, held after the series of programs ended, lasted two hours and involved junk food, 10 teens, and me – armed with a list of questions, charts, and an audio recorder. the interview covered questions such as, “how did the art programs you attended here change – or not change – how you feel about the people in the program? the library? the community?” these were big questions, and required big answers. i expected that some of the teens would avoid the hard questions with a joke or a brief answer, e.g. “the programs didn’t change anything for me.” but i was intrigued to receive complex and reflective answers from all of the participants, regardless of age or personality. early in the planning process for this project, i decided to take the unusual step of inviting the teens into the researcher’s role by asking any interested participants to work on the grounded theory coding with me, after they had finished the interview process and their answers could not be biased by seeing preliminary results. when i saw that the teens were being careful and conscientious scientists, i knew i was on the right track. one teen untangled correlation from causation when she said:   somebody who makes art is pretty much probably a smarter person. as someone who writes a letter to the editor is also a thinking person. and they are likely to do both of those things, but one does not lead to the other.   even the youngest participant, aged 12, stepped back from her own experiences and biases to clarify her conjectures about others’ motivations: “it depends on the person.” every teen interviewed tested the falsifiability of the theory they were generating, establishing the parameters of “trueness” by exploring who could not or would not be likely to be affected by library art programs. they did this with little prompting from me. they were good scientists.   researchers deduce top-down theories, such as those described in a meta-synthesis of the literature on this topic (crawford barniskis, 2012a), based on previous research (all of which had been done exclusively by adults, and never in this intersection of art, libraries, and teens). we inductively developed a “bottom-up” theory based on the teens’ perceptions, building a testable theory using charmaz’s (2006) active coding method. this approach allowed the teens to speak their language with as little mediation as possible.   grounded theory is a useful method when a phenomenon is established but little work has been done to examine it. in this instance, public libraries have offered art programs to teens for years. no one had published research on this phenomenon, but practitioners wrote about it in the form of how-to manuals and descriptive articles. a phenomenological method would have been appropriate to simply describe the experience of the teens in these programs, but i was seeking a deeper, more correlative examination of the impact of the programs – the “why” and “how” as opposed to the “what.”   the basic steps in constructivist grounded theory construction are iterative (charmaz, 2006):   ·         coding line by line, or capturing the essence of the statements using slightly simplified terms. these codes are “open”; the researcher is not attempting to make them similar or standard at this point ·         adding more data in the form of memos. ·         coding sentence by sentence, often at a slightly more theoretical distance, or other larger granularities. ·         sorting these codes into groups that make sense. ·         using constant comparative methods. when describing this to teens, i used the visual explanation at http://www.youtube.com/watch?v=nxierzx3aqq  (researchjimminy, 2009), which was more instructive than anything i could write. ·         discovering the themes in the sorted “piles” of codes. ·         acquiring more data when the theory needs fleshing out, using theoretical sampling. ·         describing the themes to create the theory. ·         connecting the themes as far as is possible, to make the theory.   in this project, i recorded and transcribed the focus group interview, then followed these steps. using atlas.ti software, i lightly coded each line of the interview transcript. i often listened to the recording to get a sense of the context of the comment, and to refresh my memory of the non-verbal communication. i coded each line with simple gerund-based codes that captured the activity occurring in each line. for example, one teen said:   yeah, i definitely do feel closer to the libraries that i go to after going through a program. so it’s like the more programs that you go to the more it feels kind of like home in a way. not as really home, but like a really comforting awesome place.   for this statement, the codes included “feeling closer to library” and “feeling comforted.”   at this point i returned to six of the teens who had volunteered to be part of the research team, to make sure i was coding using language with which they felt comfortable, and that i was capturing what they were intending to say. this process went beyond what is known as member-checking, in which researchers show their analysis to participants for basic validation (lincoln & guba, 1985, p. 314). i met with two teams of teens. the first session included three teens, and occurred just after i had finished my initial open coding of the focus group interview. i showed the group a brief video on the constant comparative process, and spoke briefly about how coding and categorizing worked. the teens mostly learned by doing. they used my codes, tweaking some of them to better reflect the intended meaning, and added codes of their own. the teen coders, for instance, changed the original code “feeling comforted.” they believed that the library was the critical reason for the comforting feeling this teen experienced, and had to be included in the code as the active force: “library comforting.” the teen coders reasoned aloud as they changed and added codes. i recorded, transcribed, coded, and analyzed the teens’ discussion during the coding sessions, because, as one of the originators of the grounded theory method notes, “all is data” (glaser, 2001, p. 145). the second meeting with the teen researchers included one returning teen and three teens who had not coded before. in this session, i repeated the video and basic instruction. we worked at a more abstract distance from the focus group and some individual interview transcripts. this team of coders was also encouraged to change or add codes, at either open, descriptive levels, deeper more theoretical levels, or to sort the codes into categories. they reasoned aloud as they worked with the data.   i treated the transcriptions of these two coding sessions as data for coding, and as memos of the researchers’ thought processes. writing down one’s own feelings, biases, intuitions, and reasoning is central to the grounded theory method. my memos, as these diary-like notes are called, were treated as data, as were the thought processes of the teen researchers. by coding and comparing these memos, i was able to notice and challenge some of my own biases and assumptions, clarify the teens’ thought processes, and build a theory more grounded in the teens’ experiences. in addition, the teen coders validated the nascent theory, keeping me on track.   as a theoretical sample to flesh out the theory, i interviewed four teens not part of the original focus group, who had attended some of the art programs. i interviewed these teens one at a time, until new concepts appeared exhausted, and the theory was fully fleshed out. these teens were just as sensitive to the contextual experiences of art program participants as were the focus group teens, and were equally careful scientists. in addition, the one-on-one interviews offered much more data than the original focus group interview. while the point of the focus group interview was that teens could feel comfortable and bounce ideas off each other – and that did happen – some teens spoke up more than others and depth was sometimes sacrificed in the interest of everyone having a chance to speak.   results   in the end, the teens and i had compared over 2,000 lines of open coding, and discovered 60 categories, which we grouped into 7 themes. the central theme was “the library can make a difference for us,” with the library as locus of change. the remaining themes described what change occurs through art programming in libraries and how it occurs:   art moves us. it’s an adult’s world. we want to connect, we want to open up. creating a community that supports us. we want to help, but don’t push us. does our engagement shift?   these themes describe the contexts, processes, and activities of teens who have experienced library art programs and how these experiences affected their civic engagement. i wrote a paper describing the theory and how we generated it. five of the teen researchers read my written interpretation of the data and verified the theory and implications. the original intent of the research project came full circle with the teen validation. the teens themselves decided how legitimate the theory was, whether statements meant what i assumed they meant, and ensured that the theory remained firmly grounded in their perspective. validating the findings of a research process with the people involved in the study can address issues of bias. it can also signal to the participants whether their concerns were heard and understood, which can be helpful when dealing with marginalized groups.   discussion   our research described how art programs in public libraries positively affect the social capital of teens, including their sense of power and capability. the programs revealed new role models, friends and advocates, as well as new ways to civically engage. teens experienced heightened empathy, and shifted their concept of the library from book-place to creation-space in a way that empowered both creativity and social connectedness. the teens felt more valued by their community after the programs and were able to pinpoint why and how this shift occurred. the art programs supported the skills, values, and motivation for civic engagement through these social connections and the activities in the programs. however, we revealed little evidence that library art programs bridge the “activation gap” (rheingold, 2008) between the desire to engage and engagement behaviour such as volunteering or political activism. the gap between the building blocks of civic engagement and the actual engagement behaviour is significant. the teens in this study described their intent to engage inchoately and fragmentarily, and the factors of active engagement need to be explored more thoroughly in further research. nevertheless, the data reveals an intent for the teens to bridge the activation gap and become more civically engaged with the understanding that they would be supported by public library programming in specific ways that address their concerns of adult hegemony and limited time to engage. since a grounded theory should reveal why and how something occurs, researchers can use quantitative methods to determine the extent to which the themes play out in a larger sample population. practitioners can examine their programs to see if or how they would want to address this gap.   implications   this project discovered implications for further research. the embedded participatory research process reveals an ethnographic description of a problem or experience, but also allows for the discovery of conceptual associations for further examination. in this project, we were able to uncover several themes, and qualitative studies can now test these themes. many library services could benefit from similar studies. libraries are often under attack during the funding process and must justify their existence to those who do not value the intrinsic benefits of funding such programs.   research is needed to addresses teen participation as research partners and the effects of this participation. participation as research partners may improve skills in research or critical thinking, teen self-image, and teen perception of the institution that offers them an opportunity to participate. one may wonder if research participation opportunities, such as the one described in this paper, increase the social capital of teens. if this is true in this research project, how much of the teens’ shift in civic engagement and social capital derived from the art program participation, and how much derived from participating as researchers? it is difficult to say. still, the teens described most of the increases in social capital before they acted as part of the research team, and even before that they knew they would have the opportunity to do so. in addition, those teens who did not participate as researchers described similar shifts. further research should consider whether acting as a researcher impacts the teens’ social capital, educational outcomes, or otherwise positively affects teens. in this way, the body of theoretical knowledge for lis scholars can be improved with more embedded, heuristic research.   the project also made visible several implications for practice. outcome-based research can help to justify a library’s need for funds. how do library services affect civic engagement, or student satisfaction with their school, or bullying, or … ? pick an outcome that a funding agency cares about, and determine if or how your program helps. such research may ignore or marginalize the individual, intrinsic benefits of library programs, such as having a good time and making friends, but few funding agencies appear to care about such intrinsic benefits. they are more interested in how a library service furthers their social agenda. social agendas such as that of educational attainment and civic engagement are good touchpoints for funders interested in attaining educational and democratic goals. the pedagogical benefits for students who learn research skills could impact their educational attainment.   librarians need more evidence-based practice, especially research grounded in the users’ experience. librarians already collect and share assessments of various programs, often through surveys or informal interviews. expanding local assessments into generalizable research may be a more or less seamless process, leveraging these surveys and interviews into user-validated results. dissemination of research results can help improve the practice of other librarians, because the research is often based on pragmatic situations which practitioners can emulate.   the act of research itself also may improve practice. in my case, i became more aware of the smallness of the teen world, their lack of opportunities to create and display art, and the way they feel systematically silenced by the institutions that rule their lives. this improved my work as a librarian. during this research project, i was able to extrapolate from the findings to offer many ways for public librarians to support civic engagement for teens. the teens offered many of these suggestions, and validated the ones i generated. such recommendations are responses to the librarian’s need for more evidence based practices. as practitioner-researchers delve deeply into their work and the impact it has on those they serve, the feedback can reinforce the best of their work and help better align outcomes with the needs of users. in fact, this research method blurs the line between research and practice. this research project bled into my practice, and emerged from it, being at the same time a recursive “cause” and “effect.” while the research project was a formal, albeit embedded, grounded theory study, it evolved into the participatory heuristic described in this paper. the method is the message, to paraphrase mcluhan.   finally, doing research with teens is fun. though it can be difficult to strip away assumptions based on one’s own teen years, or adult “wisdom,” hearing the teens describe their experiences is enlightening and enjoyable. reflexively, i know that enjoying being in a position to ensure teens are heard is ego-based. still, it is satisfying to identify a problem (teens feel voiceless), offer a small, local, and temporary solution (a few are given a venue to speak), and perhaps a more lasting and pervasive one (if others take on such research, many teens will be offered venues to speak). this research process can act as an amplification of the participants’ voices.   conclusion   the method described in this paper is recursive not only in the methodology – in the way it coded the coding process and utilized the voices of the participants to drive the research – but also in that the phenomenon under study, social capital, may have been furthered by the research method itself. the framework straddles the intersection of research and practice. yet the method is emergent, and needs further trial. it has both theoretical and practical implications for librarians, researchers, communities, and users of library services. practitioners who do research in partnership with those they are serving can make new connections between their practice and the big picture of librarianship. they can support transformational experiences for those they serve, aside from and in concert with the services themselves. librarians further expand, improve, legitimize, and advocate for their field of work when they publish research. embedded, participatory methods build testable worldviews from the ground up, based on the real experiences of those involved, offering practical solutions for other librarians while furthering a useful research agenda. users and communities benefit from these embedded, participatory research processes when their stories are authentically conveyed and conceptualized and their voices are amplified, and they sample the skills and habits of research. libraries can benefit from the establishment of user-verified outcomes of programs and services. this is especially valuable when libraries are trying to demonstrate how their services benefit the communities that fund them.     references   anderson, t., & shattuck, j. (2012). design-based research: a decade of progress in education research? educational researcher, 41(1), 16-25. doi:10.3102/0013189x11428813   bureau of labor statistics, u.s. department of labor (2012). librarians. in occupational outlook handbook (2012-13 ed.) retrieved 15 aug. 2012, from http://www.bls.gov/ooh/education-training-and-library/librarians.htm   buttlar, l. (1991). analyzing the library periodical literature: content and authorship. college & research libraries, 52(1), 38-53.   charmaz, k. (2006). constructing grounded theory: a practical guide through qualitative analysis. london: sage publications.   clarke, m. (1975). survival in the field: implications of personal experience in fieldwork. theory and society, 2(1), 95-123.   code, l. (1997). voice and voicelessness: a modest proposal? in j. kourany (ed.), philosophy in a different voice (pp. 204-230). princeton, ct: princeton university press.   cornwall, a., & jewkes, r. (1995). what is participatory research? social science & medicine, 41(12), 1667-1676.   crawford barniskis, s. (2010). public practitioner-researchers: how lis research is informed by praxis. paper presented at the student research symposium 2010, milwaukee, wi.   crawford barniskis, s. (2012). graffiti, poetry, dance: how public library art programs affect teens. part 1: introduction & literature review. journal of research on libraries and young adults, 2(3). retrieved 3 mar. 2013 from http://www.yalsa.ala.org/jrlya/2012/09/graffiti-poetry-dance-how-public-library-art-programs-affect-teens-part-1-introduction-literature-review/     crawford barniskis, s. (2012). graffiti, poetry, dance: how public library art programs affect teens. part 2: the research study and its practical implications. journal of research on libraries and young adults, 2(3). retrieved 3 mar. 2013 from http://www.yalsa.ala.org/jrlya/2012/09/graffiti-poetry-dance-how-public-library-art-programs-affect-teens-part-2-the-research-study-and-its-practical-implications/   creswell, j. w. (2012). qualitative inquiry and research design: choosing among five approaches (3rd ed.). thousand oaks, ca: sage publications.   finlay, l. (2002). negotiating the swamp: the opportunity and challenge of reflexivity in research practice. qualitative research, 2(2), 209-230.   flowers, s. (2010). young adults deserve the best: yalsa's competencies in action. chicago, il: ala editions.   glaser, b. g. (2001). the grounded theory perspective: conceptualization contrasted with description. mill valley, ca: sociology press.   hoskins, m. l. (2000). living research: the experience of researching self, other, and discourse. journal of constructivist psychology, 13(1), 47-66.   jenness, v. (2008). pluto, prisons, and plaintiffs: notes on systematic back-translation from an embedded researcher. social problems, 55(1), 1-22.   kuhne, g. w., & quigley, b. a. (1997). understanding and using action research in practice settings. in b. a. quigley & g. w. kuhne (eds.), creating practical knowledge through action research (pp. 23-40). san francisco: jossey-bass.   leckie, g. j., given, l. m., & buschman, j. e. (eds.). (2010). critical theory for library and information science: exploring the social across the disciplines. santa barbara, ca: abc-clio.   levine, p. (2008). a public voice for youth: the audience problem in digital media and civic education. in w. l. bennett (ed.), civic life online: learning how digital media can engage youth (pp. 119-138). cambridge, ma: the mit press.   li, j. (2008). ethical challenges in participant observation: a reflection on ethnographic fieldwork. the qualitative report, 13(1), 100-115.   lincon, y. s., & guba, e. g. (1985). naturalistic inquiry. newbury park, ca, sage publications.   marshall, a. m., & arvay, m. (1999). perspectives on voice and sense of self among young adolescents. professional school counseling, 3(1), 43-51.   miller, p. j., & goodnow, j. j. (1995). cultural practices: toward an integration of culture and development. new directions for child and adolescent development, 1995(67), 5-16.   moeller, r., pattee, a., & leeper, a. (2011). the young adult voice in research about young adults. journal of research on libraries and young adults. retrieved 30 nov. 2011 from http://www.yalsa.ala.org/jrlya/2011/11/the-young-adult-voice-in-research-about-young-adults/     nutley, s., jung, t., & walter, i. (2008). the many forms of research-informed practice: a framework for mapping diversity. cambridge journal of education, 38(1), 53-71.   penta, m., & mckenzie, p. j. (2005). the big gap remains: public librarians as authors in us journals, 199-2003. public library quarterly, 24(1), 33-46.   poulter, j. (2006). the two embedded research functions of heuristic case practice. australian social work, 59(3), 328-341.   reis, a. c. (2011). bringing my creative self to the fore. creative approaches to research, 4(1), 2-18.   researchjimminy. (2009, june 21). constant comparison stop motion demo [video file]. retrieved 20 jan. 2013 from http://www.youtube.com/watch?v=nxierzx3aqq     rheingold, h. (2008). using participatory media and public voice to encourage civic engagement. in w. l. bennett (ed.), civic life online: learning how digital media can engage youth (pp. 97-118). cambridge, ma: the mit press.   robson, c. (2011). real world research: a resource for users of social-research mthods in applied settings (3rd ed.). oxford, uk: wiley-blackwell.   ross, l. (2006). where do we belong? urban adolescents' struggle for place and voice. american journal of community psychology, 37(3/4), 293-301. doi:10.1007/s10464-006-9041-8   tsoukas, h. (1989). the validity of idiographic research explanations. the academy of management review, 14(4), 551-561.   vittek, r. e. (2010). the people in your neighborhood: using local collaboration to advocate for teen patrons. young adult library services, 9(1), 13-14.   wickson, f., carew, a. l., & russell, a. w. (2006). transdisciplinary research: characteristics, quandaries and quality. futures, 38(9), 1046-1059. doi:10.1016/j.futures.2006.02.011   microsoft word es_loy.doc evidence based library and information practice 2006, 1:1  104 evidence based library and information practice       evidence summary    the randomised controlled trial in medical research: using bibliometric methods to  identify core journals    a review of: tsay, migh‐yueh, and yen‐hsu yang. “bibliometric analysis of the literature of     randomized controlled trials.” journal of the medical library association 93.4 (october  2005): 450‐58.      reviewed by: john loy  learning resources manager, avon & wiltshire mental health partnership nhs trust, barrow  hospital  bristol, united kingdom  e‐mail: john.loy@awp.nhs.uk    received: 12 december 2005 accepted: 21 february 2006     © 2006 loy. this is an open access article distributed under the terms of the creative commons attribution  license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and  reproduction in any medium, provided the original work is properly cited.      abstract    objective – to explore the characteristics  and distribution of randomized controlled  trials (rcts) in the medical literature. the  study aims to identify the growth patterns  of the rct, key subject matter, country and  language of publication, and determine a list  of core journals which contain a substantial  proportion of the rct literature.    design – retrospective analysis of rcts.     setting – medical journal literature.    subjects – a total of 160,213 articles  published between 1965‐2001. detailed  analysis of a subset numbering 114,850  articles published from 1990‐2001.  methods – the study seeks to identify all  rcts in medline from 1965‐2001, and  examines the growth rate of the rct. the  authors then do a more detailed analysis on  a subset of data from 1990‐2001, using  access database and excel spreadsheet  software, and perl programming language.  the references were analyzed by five fields  within medline; publication type, source,  language, country of publication, and  descriptor (subject index).    main results – an exponential growth rate  for the rct is demonstrated, suggesting  that in the medical literature development  has not yet matured and that research using  this method continues to grow. a growth  rate for the rct of 11.2% per annum is  identified.    http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:1  105   the most common form of publication is the  journal article, making up approximately  98% of the rct literature. approximately  75% of the rcts are multicentre trials  indicating that this is the design of choice  adopted by researchers.    the united states proves to be the greatest  source of rct literature, with 39.9% of  journals and 50.6% of articles originating  there. after the usa, the most productive  countries are england (15.8% of journals and  21.7% articles) and germany (6.5% journals  and 6.1% articles). as might be expected,  english is the predominant language  providing 92.9% of the total publications. of  the remaining 7%, german is the most  common language accounting for 2.2%.    the top three areas being researched are:     1. drug therapy for hypertension    2291 citations  2. anticancer drug combinations    2140 citations  3. drug therapy and asthma     1397 citations    bradford’s law of scattering is successfully  applied, identifying four zones of journals  which each publish approximately 26,000  articles.     conclusion – the results indicate that  bibliometric methods can be applied to the  medical literature, and highlight those  disciplines in which rcts more often occur.  a core list of 42 journal titles is presented,  providing busy practitioners with  invaluable guidance as to which journals are  most likely to publish the greater number of  rcts.    commentary    deciding upon which journal titles to  purchase is a fundamental part of library  management, and anything which provides  good solid evidence on which to base these       decisions is naturally very welcome. while  there have been previous studies employing  bibliometric methods to explore rcts in  specific areas of medicine (curry, reeves,  and stringer; latronico et al.), this is  believed to be the first such study to include  all areas of the health sciences.     bradford’s law of scattering was first  proposed in 1934, and for anyone a little  rusty in their knowledge, the following  example may serve as a refresher. if a  literature search identifies 300 references,  youʹll likely discover that 100 of those came  from a core group of 5 journals, the next 100  came from a group of 25 journals further  away from the core, and the final 100  citations came from 125 journals on the  outer fringes. naturally anyone with limited  time and money will focus on the core 5  titles for the greatest return on their  investment. what tsay and yang’s research  does is utilize bradford’s law to provide  just such a core list for medical libraries.    the single greatest concern with this article,  which appeared in the journal of the medical  library association, is that it presents itself as  covering all of the literature, whereas in  actual fact it concentrates exclusively on the  medline literature. this does help  propagate the myth that if you can’t find it  on medline it doesn’t exist. while the  choice of medline, generally viewed as  the gold standard, is understandable, this  does introduce potential bias. the database  is produced in north america and  sometimes faces the accusation of being  biased towards the us publications. why  not also search embase which originates in  the netherlands and which some view as a  european version of medline?     the research undoubtedly contributes  towards the range of tools healthcare  librarians can draw upon when making  decisions about our collections. the final list  of 42 core journals will not hold too many  surprises for an experienced practitioner,     evidence based library and information practice 2006, 1:1  106   though interestingly, 8 of the top 20 titles  identified are not included in the national  library of medicine’s list of core clinical  journals.    the article could have been enhanced with  more methodological detail. we are told  that “each bibliographic record was  downloaded and analyzed” but we are not  provided with the details about what this  actually involved. we do know that analysis  of citations by mesh (medical subject  headings, the controlled vocabulary  devised by the national library of medicine)  takes place, and that the authors used the  major rather than minor descriptors to do  this. however, there is no mention of how  access, excel, and perl programming  language were used to analyze the data.    the authors have published several pieces  on bibliometric analysis in recent years,  including an article from 2003 with an  almost identical title which appears to cover  very similar ground (tsay and yang).  curiously this earlier publication is not  referenced in the jmla article, and  questions about the relationship between  the two remain unanswered.    an additional minor criticism would be  with presentation of results, as the authors  do have a tendency to move back and forth  between the large data set (1960‐2001) and  the smaller subset of 1990‐2001. the subset  provides the analytical core of the article,  and the reader must keep a close eye on  what exactly is being reported in each table  in order to avoid confusion.    while this research undoubtedly addresses  a worthwhile issue and provides the ever‐    pressed librarian with more evidence to aid  decision making, it must be said that it will  be of most interest to those working in  medical libraries. the final list of 42 journals  is very medically orientated, and those  employed in the wider healthcare arena will  probably view the study as having a more  limited value. of the top 20 subjects for  rcts, virtually all of them involve some  form of drug therapy. when it comes to  clinical decision making, rcts are  acknowledged as one of the most effective  sources of reliable information but they are  certainly not the only form of worthwhile  research. while the rct is a hugely valuable  research method, it is not appropriate for  every form of therapeutic intervention, and  is not the sole basis upon which to build a  library collection.    works cited    tsay, ming‐yueh, and yen‐hsu yang. “a  bibliometric study on the literature of  randomized controlled trials.” journal  of information, communication and  library science 9.4 (june 2003): 27‐51.    curry, joe i, barnaby reeves, and mark d.  stringer. “randomized controlled  trials in pediatric surgery: could we  do better?” journal of pediatric surgery  38.4 (april 2003): 556‐9.    latronico, nicola, et al. “quality of  reporting of randomised controlled  trials in the intensive care literature: a  systematic analysis of papers published  in intensive care medicine over 26  years.” intensive care medicine 28.9  (september 2002): 1316‐23.  evidence based library and information practice evidence based library and information practice 2011, 6.3 28 evidence based library and information practice article development and testing of a literature search protocol for evidence based nursing: an applied student learning experience andy hickneradministrative specialist patient safety enhancement program university of michigan medical school ann arbor, michigan, united states of america email: hicknera@med.umich.edu christopher r. friese assistant professor university of michigan school of nursing ann arbor, michigan, united states of america email: cfriese@umich.edu margaret irwin research associate oncology nursing society pittsburgh, pennsylvania, united states of america email: mirwin@ons.org received: 29 mar. 2011 accepted: 20 july 2011 2011 hickner, friese, and irwin. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/bync-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – the study aimed to develop a search protocol and evaluate reviewers' satisfaction with an evidence-based practice (ebp) review by embedding a library science student in the process. methods – the student was embedded in one of four review teams overseen by a professional organization for oncology nurses (ons). a literature search protocol was developed by the student following discussion and feedback from the review team. organization staff provided process feedback. reviewers from both case and control groups completed a questionnaire to assess satisfaction with the literature search phases of the review process. mailto:hicknera@med.umich.edu� mailto:cfriese@umich.edu� mailto:mirwin@ons.org� evidence based library and information practice 2011, 6.3 29 results – a protocol was developed and refined for use by future review teams. the collaboration and the resulting search protocol were beneficial for both the student and the review team members. the questionnaire results did not yield statistically significant differences regarding satisfaction with the search process between case and control groups. conclusions – evidence-based reviewers' satisfaction with the literature searching process depends on multiple factors and it was not clear that embedding an lis specialist in the review team improved satisfaction with the process. future research with more respondents may elucidate specific factors that may impact reviewers' assessment. introduction research teams in evidence-based practice (ebp) must master numerous skills and follow established procedures to produce high quality products that adhere to methodologically rigorous scientific standards. the oncology nursing society's (ons) putting evidence into practice (onspep™) program has produced evidence-based summaries for oncology nurses for managing symptoms and side effects of cancer and cancer treatments (eaton, tipton, & oncology nursing society, 2009). these summaries are prepared by small teams consisting of staff nurses, advanced practice nurses, and nurse scientists. by 2008, 16 ons-pep™ reviews had been disseminated among the ons membership in various forms, including print books, online resources, and quick reference cards (doorenbos et al., 2008). as of 2009, ons had distributed to oncology nurses over 147,000 copies of the reference cards (saca-hazboun, 2009). according to ons (2008), ons-pep™ reviews classify interventions into six categories. these categories are ranked from “recommended for practice,” where there is “strong evidence from rigorously-designed studies, meta-analyses, or systematic reviews, and for which expectation of harms is small compared with the benefits,” to “not recommended for practice,” i.e. interventions in which there is clear evidence of harm, ineffectiveness, or cost or burden exceeding likely benefit (oncology nursing society, 2008). gobel, beck, and o’leary (2006) outlined the schema for evidence rating used in ons reviews. rutledge, depalma, and cunningham (2004) described the triad model of research synthesis used in ons-pep™ reviews. the triad model emphasizes collaboration between advance practice clinicians, researchers and educators. mcgowan and harris have documented numerous examples of the roles that trained medical librarians play in the development of systematic reviews used to guide ebp (harris, 2005; mcgowan & sampson, 2005). these include clarifying the research question, the reference interview, electronic reference management, and obtaining the full text of studies as well as expert searching. weller (2004) argued for the inclusion of medical librarians in the development of meta-analyses to maximize the rigor of the search strategies employed and to improve documentation and reproduction of those strategies. it is necessary to document and standardize the search component of an ebp synthesis so that it can be repeated by other researchers. however, patrick, et al., found that many meta-analyses failed to do so (2004). before the 2009-2010 reviews, there was no formal protocol for ons-pep™ investigators to guide the literature search phase of the review process. responsibility for searching typically fell to one or two team members who volunteer to conduct the search; these team members may have consulted with a medical librarian, but it was not mandated that they do so. the level of detail provided about the search strategies evidence based library and information practice 2011, 6.3 30 employed varied among reviews as well. for example, a lymphedema review listed the specific databases searched (poage, singer, armer, poundall, & shellabarger, 2008). rutledge, et al., (2004) documented the process of formulating the final strategy, including teams’ consultation with librarians. page, berger, and johnson (2006) provided no description of the literature search strategy in the otherwise detailed review on sleep-wake disturbances. this inconsistency with describing the search process within systematic reviews is not unusual (zhang et al. 2006). consequently, one of the authors, a previous ons-pep™ investigator serving on the faculty of the university of michigan's (um) school of nursing, sought to standardize and improve the literature review experience for review teams. he partnered with a master's student at um’s school of information to develop a protocol for the literature search process of ons-pep™ reviews. this goal was embraced by ons staff to ensure that the synthesis of evidence for ons-pep™ resources would be grounded in a clear and reproducible literature search process. following the completion of the 2009-2010 reviews, ons staff surveyed ons-pep™ reviewers on their experiences with the reviews. the aim of this paper is to describe how the student was embedded into the process. it additionally explains the protocol developed for the literature search component of the review in order to assess reviewers’ satisfaction with the search process. methods an intervention was devised which embedded the student in a review team. the student was assigned to the team responsible for developing the hormonal side effects review. the team subsequently narrowed the scope of the review to “hot flashes,” one specific hormonal side effect. study participants included the student, his colleagues on the hot flashes team, and the members of the other 2009-2010 review teams. the primary outcome of interest included the completion of a literature search protocol by the student for use in future reviews. the secondary outcome of interest was the satisfaction with the review search process of the team exposed to the intervention (“hot flashes”), as compared to that of the other review teams (the control). the investigators employed a mixed-methods evaluation consisting of both qualitative (team member interviews) and quantitative elements (a survey). protocol development andeembeddingsstudent in reviewtteam the student developed a literature search protocol for use in future reviews, based on his experiences in this team, as described below. the first step in developing the protocol was to synthesize relevant studies about the literature review gathering process and evaluate the literature in the context of the current ons-pep™ process. a search of medline using pubmed, incorporating the terms “evidence-based medicine,” “librarians,” “libraries,” and “systematic review” was conducted. additional literature was identified through the process of citation chasing, defined as examining sources referenced in the most relevant articles from the medline search. literature was reviewed on both the systematic review process and on ebp to identify best practices. ons-pep™ publications were also reviewed in order to understand literature search approaches, the databases searched, and the search limits previous reviewers applied. based on a review of case studies and other articles that examined the librarian’s role in ebp, the cochrane handbook was identified as a potential model for adoption. the handbook provides detailed instructions for literature searching and citation management for researchers conducting cochrane systematic reviews (the cochrane collaboration, 2008). based on experience with other evidence-based resources at an early phase in the project, the student hypothesized the cochrane handbook could provide an appropriate roadmap for ons-pep™ reviews. evidence based library and information practice 2011, 6.3 31 the student interviewed relevant staff members at ons including the ons librarian, the ons-pep™ coordinator, and a research associate assisting with the reviews. (these three individuals are hereafter referred to as “ons staff.”) based on their feedback, a memorandum outlining a proposal for the student’s participation in the review was submitted. after obtaining additional feedback on this proposal and approval from the ons-pep™ coordinator, the student was embedded in a review team to pilot test and refine the protocol. the student and his faculty advisor submitted a completed protocol for ons, and performed a debriefing to review lessons learned and strategies for the future. details for these activities performed are described below. the faculty mentor, student, and the ons staff members previously mentioned discussed project options by telephone. topics included the purpose and expected benefits of a revised searching protocol, student goals and objectives, and idealized outcomes of the proposed changes for ons-pep™ reviewers. discussions occurred over the proposed protocol’s scope (i.e., determining which databases to search), and refinement of the scope of each ons-pep™ review using the pico (patient-intervention-comparisonoutcome) framework (sackett, 2000). the telephone calls also established key deliverables and timelines. a formal memorandum outlining the final proposal and explaining the nature of the student’s participation in the review teams was submitted. from august until late october 2009, the student participated in conference calls with members of the hot flashes team every two weeks. the team was comprised of six clinical nurses and nurse practitioners, one doctorallyprepared nurse researcher, and the student. the student conducted the literature search and citation management for the review. the student departed from the team when the search stage was completed and the article review stage began. concurrently, the student enrolled in an evidence-based health information class at the um school of information to provide the theoretical and practical knowledge necessary for project execution. among many other topics, the course content included principles of evidence-based medicine; how to critically appraise a scientific study; how to find a variety of information sources such as practice guidelines, unpublished studies, and research articles; and an overview of the process for conducting a systematic review. the initial protocol draft numbered 17 pages. the protocol was revised based on comments and discussions with the student, mentor and ons staff (appendix). evaluation the process was evaluated in a number of ways. an ongoing observation was carried out by the student when developing the protocol and working with the hot flashes team. these observations were refined and worked into the protocol that was accepted for future pep teams. the hot flashes team participated in a structured discussion by telephone which was designed to elicit their views on embedding an lis student in the pep process. positive and negative experiences were sought as well as their suggestions for future improvement. approximately one month after the embedded experience, the mentor and student met with the ons staff to identify lessons learned and to consider the impact on the ons-pep™ review process. one of the ons staff members involved in the initial planning of the study conducted semi-structured phone interviews with groups of reviewers from the hot flashes team. the interviews were designed to elicit responses from team members regarding their experience of the entire pep process. all team members participated in this phone interview and responded to open-ended questions to identify positive and negative aspects of their experiences, as well as to provide recommendations for process improvement. evidence based library and information practice 2011, 6.3 32 responses from all teams were recorded and summarized by the interviewer. a quantitative evaluation beginning in august 2010 was conducted by the ons staff, the faculty mentor, and the student. the ons staff developed a web-based questionnaire, using the zarca® (zarca interactive, inc., herndon, va) electronic survey system, and disseminated it by email to all 27 members of the 2009-10 pep teams. twenty –two members completed the survey, for a response rate of 81.5%. a case-control method was employed, with respondents from the hot flashes team analyzed as cases and respondents from all other teams categorized as controls. the questionnaire included several questions pertaining to the literature search phase of the reviews. after responses to the questionnaire were compiled, ons staff analyzed the results using predictive analytics software® (spss, inc., chicago, il), version 18.0. results qualitative results protocol development the search protocol was refined iteratively (see appendix for final version). the protocol required significant adaptations to the procedures outlined in the cochrane handbook (higgins & green, 2009) because ons-pep™ reviews differ from cochrane systematic reviews in purpose, methods and scope. cochrane reviews typically evaluate the evidence pertaining to one specific intervention for a given condition, and limit results to randomized controlled trials (rct). in contrast, ons-pep™ reviews are broader, evaluating multiple interventions for a condition. rather than limiting the type of research design to randomized trials, onspep™ reviews include all types of research and summarize the limitations of the evidence. the protocol development process helped identify differences in assumptions, attitudes, and approaches among the ons-pep™ leadership team. this encouraged debate and acknowledgment of different perspectives among team members. following this discussion, clearer processes emerged and were communicated across review teams. for example, while the cochrane handbook mandates the searching of grey (unpublished) literature for each review, ons determined that grey literature should only be identified to "tip the balance" in cases where evidence for an intervention is ambiguous. reasons for doing so were sound, but have important methodological implications for the results of each review. the final protocol explicated this justification. key components of an effective search protocol that emerged during the process included specifying database and bibliographic management requirements. experience during the hot flashes review, demonstrated that at the beginning of the review process, reviewers need to clearly identify the databases to be searched, search limits, and the citation management software best suited to the project. before searching, teams should also identify ways to automatically sort and organize results to optimize the process of reviewing the retrieved studies; for example, separately filtering studies by topic (such as pharmacological versus alternative therapies) or level of evidence (such as secondary from primary level sources). this organization was seen to facilitate summary and review of various interventions. due to time constraints, the range of experience of clinician reviewers, and the purpose of the review series, the cochrane standards were not always appropriate or practical. however, cochrane standards were used as a frame of reference to guide the final protocol. based on feedback from ons staff, the investigators found that the recommendations as originally drafted needed to be simplified for adoption by busy health care professionals who were unfamiliar with literature searching nuances and terminology. the major adaptations from the initial draft included the preparation of a one-page protocol synopsis for reviewers. the synopsis summarized the most essential points from the full protocol evidence based library and information practice 2011, 6.3 33 and outlined sequentially the key tasks and milestones that each ons-pep™ team would need to accomplish (see appendix, “the literature search process for putting evidence into practice (pep) review teams”). the student also modified the sequence of the search and citation management activities. the reviewers encountered other challenges in the process that made changes to the protocol necessary. key challenges and adaptations to the resulting guidelines included issues in forming the research question and external constraints that affected the process. clarifying the research question initially there was disagreement among reviewers over the scope of the question. the process was delayed by the consensus-based nature of ons-pep™ team decision-making, and accordingly influenced the literature search process and timeline. the approach for review and synthesis of evidence is somewhat unique in the ons-pep™ team process. specifically, the summary, synthesis and categorization, or weighting of evidence is done by team consensus in the application of specified characteristics of the body of the evidence. teams are purposely constructed to include a range of nurses specialized in oncology, including nurse scientists, nurse educators, advanced practice nurses and staff nurses. this mix of individuals is used to ensure that the synthesis and production of evidence-based resources incorporates a range of knowledge, expertise and perspectives. the pico format, along with the direct guidance and collaboration of the student enabled the team to resolve these differences and adopt a clearer charge for content review, according to ons staff and pep team members. one key factor that enabled the hot flashes team to move forward was narrowing the scope of the research question. other teams found the pico format confusing, suggesting that the inclusion of an information science specialist can be a critical factor in facilitating this aspect of the process. consequently, the formulation of the pico question assumed a greater prominence in the guidelines. process constraints two major constraints in the process emerged in the project: reviewer time limitations and the nature of the available evidence in the topic area. after the team narrowed the review scope from hormonal side effects to solely hot flashes, ten days remained to conduct the search and share the results with colleagues. the majority of time was consumed by eliminating duplicates, technical problems with citation management and transferring refworks™ (proquest llc, ann arbor, mi) libraries between the student and the ons lead for the review team. the authors changed the guideline to increase the emphasis on the citation management process and citation management guidelines. for the hot flashes team, high-quality literature and consistent research findings for particular interventions were scant. in response, the final protocol provided guidance for targeted search strategies in such instances. the experience also identified improved outcomes with didactic and applied education, and benefits of collaboration between ebp reviewers and library/information science (lis) students, which will be discussed later. quantitative results results from the questionnaire are presented in table 1. the questions were scored on a tenpoint likert scale, where 0 = highly dissatisfied to 10 = highly satisfied with the exception of the question “how difficult was it for your pep team to define your literature search strategy?” (where lower numbers equaled less reported difficulty). satisfaction levels were generally moderate for each question. the highest responses were for the questions “how confident are you that all appropriate and relevant evidence was captured in your pep team literature search?” and “how satisfied are you with the final product the pep team created?” the lowest responses were for questions one and two: the teams found it moderately difficult to define the search strategy and were only somewhat evidence based library and information practice 2011, 6.3 34 satisfied with the process of defining the strategy. a trend of increased satisfaction with the literature search process was not observed when the student participated in the review group. indeed, there was a trend toward negative findings in the case group compared with the control groups; however, these differences failed to reach statistical significance. discussion the qualitative evaluation suggested that embedding lis students in ebp projects is mutually beneficial, and that american library association-accredited lis science programs should collaborate with ebp investigators to advance both clinical care and students' professional interests. in their debriefing, ons leadership reported they appreciated the outside perspective of a librarian, as well as a skill set that complemented and extended the team review. they have now incorporated ways to embed librarians or library students in future rounds of searching to support review teams. as specified in the protocol, pico search terms and inclusion and exclusion criteria are clearly defined for the literature search and review phases of the process. team members first categorize, then summarize the evidence they have retrieved. combined didactic and applied learning can improve outcomes for both lis students and ebp reviewers. one example of the impact of formal education on the applied experience was a change in the preferred interface for searching medline. initially, the student conducted the search for the hot flashes team using pubmed. an experienced systematic reviewer pointed out that the saved strategies table 1 satisfaction with review process case group n= 6 control group n= 13* p question mean (sd) 1. how difficult was it for your pep team to define your literature search strategy? 5.00 (2.68) 4.31 (2.32) .57 2. how satisfied were you with the process of defining your literature search strategy? 5.83 (3.13) 7.15 (2.08) .29 3. how confident are you that all appropriate and relevant evidence was captured in your pep team literature search? 7.67 (1.51) 8.15 (1.52) .52 4. how satisfied were you with the process of summarizing and categorizing the evidence? 6.33 (1.37) 6.46 (2.11) .89 5. how satisfied are you with your overall experience in participating in this activity? 6.50 (2.510) 8.31 (1.653) .07 6. how satisfied are you with the final product the pep team created? 7.50 (2.07) 8.92 (0.95) .16 *total response on the entire survey was 22 people, but the data reported are from 13 subjects due to missing data. evidence based library and information practice 2011, 6.3 35 in ovid medline were significantly clearer and more elegant in presentation compared to the strategy in pubmed. the pubmed search details included numerous nested parentheses, making the structure of the strategy hard to unravel and to read. thus, the final protocol was revised with the recommendation to use ovid medline if and when team members had access to it (with the understanding that this is not always the case). this activity and study findings have resulted in practice changes for the conduct of the pep processes. the quantitative evaluation provided mixed results. the responses to questions 3 and 6 suggest hot flashes team members were satisfied with their final product, and that they felt confident that all appropriate and relevant evidence was captured by the literature search. responses to other questions suggest that the contribution of an lis student to evidence-based reviews does not necessarily increase reviewers’ satisfaction with the literature search process. unfortunately, the questionnaire did not probe respondents’ specific reasons for their satisfaction with the assistance of the student. the study also does not seek to objectively evaluate the quality of the search or the search results from either the case or control teams. the team members’ initial debate over the scope of the research question and the resulting time constraints may have influenced the results. it is possible that the presence of, or supervision by, an lis professional with greater experience and expertise would have yielded different results. consequently, the impact of the level of the lis specialist’s experience on reviewer satisfaction merits further study, alongside the need for research on the impact of lis specialist involvement on the quality of the search process which was identified by zhang et al., (2006). the experience in this study suggests that if protocols are to be adopted by clinicianreviewers, a user-centered perspective is essential. to adhere to cochrane standards (or any other protocols) at the expense of the user experience is to risk crafting a document no one will read, and mandating steps no one can or will follow. groups that wish to develop a similar protocol for their ebp projects may find it helpful to borrow practices from human computer interaction and design such as holtzblatt, wendell, and wood’s contextual inquiry (2005) and to conduct them before drafting and piloting a protocol. since cochrane reviews examine narrowly defined medical interventions, it may also be more appropriate to examine literature about searching in relation to more complex topics (eg golder et al, 2008; mcnally and alborz, 2004; ogilvie et al, 2005 ). how do these findings fit into the broader literature on this topic? there is an abundance of studies on the impact of information professionals on various outcomes in the health sciences. for example, a systematic review by rankin, grefsheim, and canto (2008) summarizes a number of studies on the impact of the informationist, defined as “a new professional… with responsibility for providing highly specialized information services in the clinical setting.” others have examined the impact of librarian services on clinical practices (mcgowan, et al., 2010). however, to the authors’ knowledge, there have been no formal evaluations of the impact of an embedded lis specialist on their outcome of interest, the satisfaction of ebp review team members. the issues that arose in the qualitative evaluation are a reminder that librarians and informationists seeking to evaluate their own effectiveness need to build evaluation into projects from the beginning. a prospective, rather than retrospective, study design for evaluation may be helpful. having teams keep diaries or journals of each phase of the project could provide qualitative data to help identify confounding factors. future studies could include measurement development to examine: time spent during literature search; time spent on abstract review; time spent on paper inclusion and exclusion; number of irrelevant references retrieved that had to be discarded, and team member satisfaction with the review process. evidence based library and information practice 2011, 6.3 36 as evidence-based reviews proliferate, clinical agencies and reviewers face multiple challenges in the conduct and dissemination of review results. the authors’ experience suggests that embedding an lis student is a feasible approach to facilitate review as well as a valuable learning experience for the student. future investigators should consider a prospective evaluation. they should also solicit specific feedback on the evaluation of the specific role of an lis specialist within a review team. with such data, evidence-based reviews in the future can be more confident in the deployment of valuable human resources. this study contained a number of limitations stemming from the retrospective rather than prospective nature of the evaluation. quantitative evaluation of this project was designed to answer the question, “is there a difference among review team members in satisfaction with the process and results as a result of embedding an information specialist?” this approach did not allow us to evaluate the potential difference within a topic team with or without an information specialist. thus there is no way to determine the impact of the specialist within the team itself – how the results would have differed in the same group of reviewers without the assistance of the information specialist. quantitative findings were also limited by the small sample size, which did not have sufficient power to detect a significant difference between groups. a number of global factors may also confound the results. group dynamics and experience may have contributed to perceived satisfaction with the review process for both case and control groups. overall satisfaction with the process within the hot flashes ons-pep™ team may have also been influenced by the fact that little high quality evidence in this topic area was found. conclusions this project sought to embed and evaluate an lis student within an evidence based review process. the project resulted in a standard protocol to be used for searches conducted by future ons-pep™ teams. the protocol anticipates problems and issues encountered in this experience and provides guidelines for search documentation that improves the transparency and reproducibility of the process. the authors found that combining applied and didactic learning resulted in improvements to the protocol. the project was evaluated by both quantitative and qualitative methods. qualitative feedback from those involved in the review which received the embedded experience demonstrated that team members were satisfied and appreciated having the perspective of an lis professional on the team. in contrast, the quantitative evaluation suggested that the student’s contribution failed to increase reviewer satisfaction in the literature search process, however differences in survey responses failed to reach statistical significance. the experience gained in this project reinforces the need to evaluate student learning projects from the beginning in order to maximize the quality of data collected. references doorenbos, a., berger, a., brohard-holbert, c., eaton, l., kozachik, s., lobiondo wood, g., varricchio, c. (2008). oncology nursing society putting evidence into practice resources: where are we now and what is next? clinical journal of oncology nursing, 12(6), 965-970. eaton, l. h., tipton, j. m., & oncology nursing society. (2009). putting evidence into practice: improving oncology patient resources. pittsburgh, pa: oncology nursing society. gobel, b. h., beck, s. l., & o'leary, c. (2006). nursing-sensitive patient outcomes: the development of the putting evidence into practice resources for nursing practice. clinical journal of oncology nursing, 10(5), 621-624. evidence based library and information practice 2011, 6.3 37 golder, s., mason, a., & spilsbury, k. (2008) systematic searches for the effectiveness of respite care, journal of the medical library association, 96 (2), pp 147-152. harris, m. r. (2005). the librarian's roles in the systematic review process: a case study. journal of the medical library association, 93(1), 81. higgins, j., & green, s. (eds.). (2009). cochrane handbook for systematic reviews of interventions version 5.0. 2 [updated september 2009] holtzblatt, k., wendell, j. b., & wood, s. (2005). rapid contextual design: a how-to guide to key techniques for user-centered design. san francisco: elsevier/morgan kaufmann. mcgowan, j., hogg, w., rader, t., salzwedel, d., worster, d., cogo, e., & rowan, m. (2010). a rapid evidence -based service by librarians provided information to answer primary care clinical questions. health information & libraries journal, 27(1), 11-21. mcgowan, j., & sampson, m. (2005). systematic reviews need systematic searchers. journal of the medical library association, 93(1), 74. mcnally., r., & alborz, a. ( 2004). developing methods for systematic reviewing in health services delivery and organization: an example from a review of access to health care for people with learning disabilities. part 1. identifying the literature, health information & libraries journal, 21 (3), pp 182-192. ogilvie, d., hamilton, v., egan, m., & petticrew, m. (2005) systematic reviews of health effects of social interventions: 1. finding the evidence: how far should you go?, journal of epidemiology and community health, 59 (9), pp 804-808. oncology nursing society. (2008). ons pep (putting evidence into practice) weight of evidence classification schema: decision rules for summative evaluation of a body of evidence. pittsburgh, pa: mitchell, s.a., & friese, c.r.. page, m., berger, a., & johnson, l. (2006). putting evidence into practice: evidence-based interventions for sleep-wake disturbances. clinical journal of oncology nursing, 10(6), 753 767. patrick, t. b., demiris, g., folk, l. c., moxley, d. e., mitchell, j. a., & tao, d. (2004). evidence-based retrieval in evidence based medicine. journal of the medical library association, 92(2), 196. poage, e., singer, m., armer, j., poundall, m., & shellabarger, m. j. (2008). demystifying lymphedema: development of the lymphedema putting evidence into practice card. clinical journal of oncology nursing, 12(6), 951-964. rankin, j. a., grefsheim, s. f., & canto, c. c. (2008). the emerging informationist specialty: a systematic review of the literature. journal of the medical library association: jmla, 96(3), 194. rutledge, d., depalma, j., & cunningham, m. (2004). a process model for evidence based literature syntheses. oncology nursing forum, 31(3), 543-550. saca-hazboun, h. (2009). putting evidence into practice. outcomes of using the ons pep resources. ons connect, 24(4), 8-11. sackett, d. l. (2000). evidence-based medicine: how to practice and teach ebm. edinburgh; new york: churchill livingstone. the cochrane collaboration. (2008). the cochrane library: evidence for healthcare evidence based library and information practice 2011, 6.3 38 decision-making. retrieved april 15, 2010, from http://www3.interscience.wiley.com/a boutus/sharedfiles/brochures flyers/cochrane_library_brochure.pdf weller, a. c. (2004). mounting evidence that librarians are essential for comprehensive literature searches for meta-analyses and cochrane reports. journal of the medical library association, 92(2), 163. zhang, l., sampson, m., & mcgowan, j. (2006). reporting of the role of the expert searcher in cochrane reviews. evidence based library and information practice, 1(4), 3-16. retrieved from http://ejournals.library.ualberta.ca/ind ex.php/eblip/article/view/57 evidence based library and information practice 2011, 6.3 39 appendix the literature search process for putting evidence into practice (pep) review teams the purpose of the guide is to make the complex and often intimidating process of a comprehensive literature search easier for all team members involved. the guidelines are intended to be a framework. individual team members are free to exercise their own discretion in determining whether a recommendation is appropriate for their particular topic. individual teams, because of the question under study, may alter these suggestions. suggestions for revision are not just welcomed, but encouraged first conference call • identify who will conduct searches, and any additional sources to search • work through pico question as group, clarify scope • confirm time frame and due date for final search results identifying who will conduct the literature searches: the team researcher and educator are the most likely candidates for searcher(s). however, the search team could include any other team member who is comfortable with searching, and is affiliated with an institution with a robust set of database subscriptions. the following databases are recommended for all project teams: cinahl, medline via ovid sp, embase, and cochrane central. additional sources, including professional conference proceedings, may be considered individually by project teams. developing and refining the pico question: pico is a method of forming a focused clinical question. carefully working through the pico question up front saves the searcher and team a lot of time down the road. this can be done as a team to come up with a comprehensive list of concepts and terms for the population examined, interventions searched, and outcomes measured. don’t forget to incorporate survivorship concepts into your pico question. confirm time frame and due date for final search results : at the time of the first team call, the searchers should confer with the team leader to clarify the date that final search results are due, and to ensure that the timeframe for them to do this is adequate. since pep researchers are often working full time at their own jobs, it may be best to allow two weeks or more to complete this. by the second conference call, searches should: • complete the pico outline tool (and/or other pico worksheets) • complete search planning form from “literature search process: protocols for researchers” • compare/share pico and search planning forms; be prepared to share and discuss any disagreements with the rest of the team during call second conference call • share pico and search planning strategy with team • clarify any remaining questions/resolve any debates about scope of pico question with team share pico and search planning strategy with team: searchers should be prepared to briefly share the results of their pico work with the rest of the team. they should resolve what differences they can prior to the meeting and if any remain outstanding, they can share these with the team for further discussion. nb: the full version of the protocol and associated documents are available from the authors on request. / evidence based library and information practice/ / evidence based library and information practice 2012, 7.1 120 evidence based library and information practice evidence summary canadian public libraries are aware of their role as information literacy training providers, but face several challenges a review of: lai, h.-j. (2011). information literacy training in public libraries: a case from canada. educational technology & society, 14(2), 81-88. reviewed by: laura newton miller science & engineering librarian carleton university ottawa, ontario, canada email: laura_newtonmiller@carleton.ca received: 30 nov. 2011 accepted: 20 jan. 2012 2012 newton miller. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to explore the current state of information literacy (il) training in canadian public libraries, and to identify strategies used for improving il training skills for staff and patrons. design – mixed-methods approach, including document analysis, observations, and focus group interviews. setting – two libraries of a large public library system in canada: the central library and one branch library. subjects – six staff members (manager, administrator, training coordinator, instructor, and computer technician) who have been involved in designing and teaching information literacy courses for library patrons and staff. methods – the researcher analyzed internal and external library documents related to information literacy, including, but not limited to, reports, posters, lesson plans, newsletters, and training scripts. he also observed interactions and behaviours of patrons during il training sessions. finally, he conducted a focus group with people involved in il training, asking questions about facilities and resources, programs, patron reaction, librarian knowledge of il theory, and impediments and benefits of il training programs in public libraries. main results – staff were aware of the importance of il training in the library. mailto:laura_newtonmiller@carleton.ca� http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/� evidence based library and information practice 2012, 7.1 121 attracting more library patrons (including building partnerships with other organizations), improving staff il and training skills, employing effective strategies for running training programs, and dealing with financial issues were all concerns about running il training that were highlighted. conclusion – canadian public libraries are well aware of their role as il training providers, but they still face several challenges in order to improve their effectiveness. commentary lai presents an interesting study on an information literacy training program within a large public library system, providing background on the importance of lifelong and self-directed learning when discussing adult learners. he then centres on library staff attitudes toward il training as the focus of this study. this reviewer would consider the paper a case study of a particular library system. although the research is of interest to others working with il training in public libraries, the study may be difficult to generalize and to replicate because of the uniqueness of the subjects studied. the researcher conducted only one focus group of six people with different viewpoints of the library. issues of administrators or management are very different from those of a training coordinator, instructor, or computer technician. because the make-up of the focus group is limiting, there is a disconnect between what is said and how generalizable these results are to all canadian public libraries. one group member assumes that most staff members have library science degrees but tend to ignore the theories behind teaching, and that “some staff members are resistant to embrace their teaching role in providing il instruction.” (p. 86) who is saying this? and what is the educational background not only of focus group participants, but of staff providing il training? including more people in several focus groups would make for a better informed study. we might see very different results if there were a focus group of just instructors and another of just management. for example, perhaps there are different reasons why it seems that staff do not value the training opportunities afforded to them, but staff may be unwilling to talk about these in front of their employer. observing only two training sessions is also very limiting. perhaps it would have been more beneficial for the researcher to sit in on more than two sessions, but because a script was provided for sessions, maybe this wasn’t necessary. but are people actually following a script? readers do not know. although document analysis was conducted, very little is known about what was actually found in the library documentation to support the researcher’s findings. expansion of the research through interviews with more library staff and knowledge of educational backgrounds would be beneficial for further research. although this paper is a good start in examining guidelines for effective il training in public libraries, a more rigorous and systematic method would lead to more sound, valid, and replicable results. / evidence based library and information practice evidence based library and information practice 2011, 6.4 12 evidence based library and information practice commentary project output versus influence in practice: impact as a dimension of research quality hazel hall professor and director, centre for social informatics edinburgh napier university executive secretary, library and information science research coalition (part-time secondment) edinburgh, scotland, united kingdom email: h.hall@napier.ac.uk received: 15 oct. 2011 accepted: 17 oct. 2011 2011 hall. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. in an environment where resources are few, research funders are expected to focus on projects that demonstrate value for money. thus, in the context of service provision, proposed research projects should actively contribute to the building of an evidence base that both supports decision-making and is actively deployed in practice. the research in librarianship impact evaluation study (rilies) (http://lisresearch.org/rilies-project/), which ran from february to july 2011, was initiated by the library and information science research coalition to investigate the extent to which funded research projects in the domain of library and information science (lis) influence practice in the u.k. it focused particularly on identifying factors that increase or hinder the impact of research findings on those who deliver library and information services. the project findings derived from a review of the lis literature on impact, a practitioner poll, case studies of five lis research projects identified as “impactful”, three sector-specific focus groups, and a validation survey. by the end of june 2011 the rilies team was in the latter stages of data collection for the project, and it was possible to report some of the preliminary findings at the 6th international evidence based library and information practice conference (eblip6). the rilies team also took advantage of the large number of eblip6 delegates who work in healthcare and medical settings to conduct the last of the three sector-specific focus groups (http://lisresearch.org/2011/07/04/links-betweenresearch-and-practice-the-health-and-medicallibrarians-perspectives/) at the conference. the preliminary findings of the rilies project were relayed to the eblip6 conference audience mailto:h.hall@napier.ac.uk� evidence based library and information practice 2011, 6.4 13 in the keynote presentation entitled “project output versus influence in practice: impact as a dimension of research quality” delivered by hazel hall on thursday, june 30, 2011. the context for these initial findings was set against a discussion of impact as conceived in librarianship and information science, with acknowledgement of both the difficulties of measuring impact and the importance of doing so. the full set of slides for the presentation can be found at: (http://www.dcs.napier.ac.uk/~hazelh/esis/ hall_eblip6_2011.ppt). by june 2011 some strong messages had already emerged from the rilies empirical data. the factors identified to date as determining a research project’s level of impact in practice related to: (1) initial project conception and implementation; (2) the means by which research output is disseminated; (3) the nature of research output; (4) the level of interest that the project generates amongst key lis research information hubs; and (5) the context in which the target practitioner audience for the research operates. in terms of where research effort should be directed, the london and perth focus groups (academic and public librarians respectively), for example, emphasized the need for studies that are of direct relevance to practitioners, both in terms of “subject” and “context” (time, place, sector), and that these need to be on a scale for the project recommendations to be applicable across a range of contexts. three of the “impactful” case studies highlighted the importance of research approach. projects that deploy action research, mix in development and engagement, encourage community creation around the project, and involve high profile, prestigious partners are most likely to have strong impact amongst the practitioner community. in terms of dissemination, face-toface routes are favoured most by practitioners. textual sources need to be accessible, both in terms of their presentation and in terms of “physical” access. it was interesting in the context of presenting these preliminary findings that three of the five individuals identified in the early poll for the rilies project as lis research hubs were in the lecture hall: andrew booth, alison brettle and hazel hall (the others were rhona arthur and phil bradley). the last area for consideration in this presentation referred to the contexts in which lis practitioners operate, and how often this hinders access to research, for example when training and travel budgets are cut. the full analysis of the empirical data for the project was completed at the end of july 2011. this largely confirmed the findings from the earlier literature review: that there is a disconnect between lis research and the practitioner community; that the level of impact a project enjoys depends mainly on how it is planned and conceived, the extent to which practitioners are involved in its execution, and how its findings are reported. organizational factors that support a receptive target audience for research output are also of important to the question of impact. the project’s findings also generated new insights related to the roles of research leadership and sponsorship, and the means of involving practitioners in research projects. in particular, the findings highlighted a greater preference for face-to-face channels for the dissemination of research results than was previously reported, and revealed the role of social media in raising awareness of research for the first time in a study on this theme. the rilies project has confirmed that where impact is measured as a dimension of research quality, a number of strategies should be deployed to extend project outcome reach. these need to ensure that the lis research undertaken has high level support; the execution of lis research involves practitioners; dissemination plans for lis research take into account practitioner preferences for consuming research output; lis research output is accessible to the target audience; and practitioners are given support to engage with research by their employers and professional evidence based library and information practice 2011, 6.4 14 bodies, drawing on good practice within the broad community of librarians and information scientists. the research information network (rin) (http://www.rin.ac.uk) will publish the full rilies project report on behalf of the lis research coalition later this year. in addition, the project team will be disseminating the findings in several other ways, for example, in a presentation at online 2011 at london olympia on thursday, december 1, 2011 (http://www.onlineinformation.co.uk/online2011/visiting/seminar_ description_online.html?presentation_id=1700), in a series of posts to the lis research coalition blog at (http://lisresearch.org/) and in the peerreviewed journal literature. acknowledgements this paper is based on a keynote address given at the 6th international evidence based library and information practice conference (eblip6), held at the university of salford, june 27-30, 2011. the rilies team members hazel hall, peter cruickshank, ella taylor-smith and jenny gebel would like to thank all those involved in the project, in particular michael jubb and stephanie kenna, and all those who contributed to the data collection exercises, including alison brettle and maria grant who made it possible to conduct one of the focus groups at eblip6. / evidence based library and information practice microsoft word news6_1477_final_final_final_final evidence based library and information practice 2008, 3:1  89 evidence based library and information practice       news     unesco “training the trainers in information literacy workshop”: call for  participants      © 2008. this is an open access article distributed under the terms of the creative commons attribution  license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and  reproduction in any medium, provided the original work is properly cited.       first announcement and call for  participants    the unesco “training the  trainers in information literacy  workshopʺ, hosted by the  department of information  management & faculty of  education of hacettepe university,  will take place in ankara, turkey,  from 3‐5 september 2008.      the main aim of this workshop is to  provide information literacy  instructors with specialized,  authoritative, comprehensive and  up‐to‐date training knowledge, and  bring them together to discuss and  share information on current issues,  theories, teaching and learning  approaches and methods, and  professional “best practices”  regarding information literacy  education and training.    every individual and organization  in today’s global information  society needs to master various  kinds of critical 21st century skills     and competencies.  none is more  important than lifelong learning and  information literacy.  fluency in  those skills will enable people and  institutions both to succeed in a  constantly changing and  challenging, and highly competitive  world.     information literacy is a set of skills  that embrace the entire life cycle of  knowing how to define one’s  information needs, then to find,  evaluate, communicate and  effectively use the retrieved  information. information literacy is,  in turn, the keystone to lifelong  learning. an information literate  individual is a self‐directed  individual capable of knowing how  to learn throughout his/her entire  lifespan. librarians, educators and  other information professionals play  a central role in equipping  individuals with information  literacy skills, as well as providing  them with opportunities to  strengthen and keep current these  skills. therefore, training librarians,  evidence based library and information practice 2008, 3:1  90 educators, archivists, curators, and  other professionals in information  literacy concepts and practices is  crucially important to every nation,  its institutions, and other elements  of its society, both public and  private.    the main topics of the workshop are  as follows:  • explaining the concept, and  importance of information  literacy and lifelong  learning;   • explaining how information  literacy and lifelong  learning are “twin”  concepts that are ideally  counter‐dependent;   • illustrating information  literacy models,  programmes and standards  that have been developed  and tested;   • sharing “best practices”  information literacy  examples;   • developing a course  syllabus for teaching  information literacy;   • amplifying a “needs  analysis”;   • describing the  characteristics of learners  (i.e. learning styles);  • sharing training content and  materials development  ideas;   • discussing information  literacy teaching methods  and techniques;   • listing effective information  literacy evaluation  techniques; and   • integrating information  literacy into curricula.    sessions on the main topics of the  workshop will include class  exercises and small group work,  and will be facilitated by expert  presenters. participants will be  expected, and strongly encouraged,  to contribute to the discussions  guided by expert presenters.   school and university librarians,  teachers and faculty, school  principals and other administrators  who are involved and/or in charge  of developing and delivering  information literacy instruction and  pedagogical development are  invited to apply. a maximum of 50  participants will be accepted to the  workshop mainly from the caspian  and black sea neighboring  countries, southwestern asian, and  eastern european/balkan countries.  a certificate of completion will be  awarded to those who successfully  complete the workshop.     applicants must fill out the online  application form, which will be  shortly available on the workshop  website  http://www.tttworkshop.net.  applicants will be placed on a  mailing list after their application is  received, and in due course, will be  notified whether their candidacy for  the workshop has been accepted or  not.  if accepted, additional  guidance as to next steps and  arrangements, including  authorization to proceed with travel  arrangements, will be provided.     it is strongly recommended that  interested participants who desire to  apply to the workshop, or who have  already applied and are awaiting  evidence based library and information practice 2008, 3:1  91 notification as to acceptance or not,  regularly check the workshop  website for the latest information.  communications of all kinds  (views, queries, etc.) should be  directed to  tttwshop@hacettepe.edu.tr.    please refer to the following  websites for more information about  the workshop series:   http://portal.unesco.org/ci/admin/ev .php?url_id=25623&url_do=do _topic&url_section=20    http://www.albertkb.nl/pageid_536 6125.html      looking forward to your  participation in the workshop.  evidence summary   multiple factors influence undergraduates’ intent to use online library resources   a review of: joo, s., & choi, n. (2015). factors affecting undergraduates’ selection of online library resources in academic tasks. library hi tech, 33(2), 272-291. doi: 10.1108/lht-01-2015-0008   reviewed by: eamon c. tewell reference & instruction librarian brooklyn campus library long island university brooklyn, new york, united states of america email: eamon.tewell@liu.edu   received: 20 sep. 2015   accepted: 19 oct. 2015      2015 tewell. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to investigate multiple factors that may affect undergraduate students’ selection of online library resources. usefulness and ease of use, quality, and user differences were each explored as factors influencing undergraduates’ use intention.   design – survey questionnaire.   setting – a state university located in the united states of america.   subjects – 332 randomly selected undergraduate students.   methods – a survey designed to measure the intent to utilize online library resources was administered to an undergraduate population. the results, including 11 factors of use intention, were analyzed quantitatively using inferential statistics such as structural equation modeling, multiple regression, t-tests, anovas, and linear regression.   main results – the factors of usefulness and ease of use were reported to have a positive relationship with undergraduates’ intent to use online library resources (regression weights = 0.473 and 0.408, p < 0.01). respondents who answered that they were “very or extremely familiar” with online library resources had higher use intention of these sources (mean = 6.17) than other groups moderately or not at all familiar (mean = 5.74 and 4.95, respectively). experience in a library instruction program was not found to influence use intention (t = -0.368, p > 0.05).   conclusions – the authors conclude that multiple factors influence online library resource selection behavior among undergraduates. the results indicate that usefulness and ease of use are important factors in use intention. the effect of “resource quality” factors, indicated by credibility, format, accessibility, currency, and coverage, suggested that all five factors positively impact use intention. accessibility is most likely to increase the likelihood of online library resource selection while the credibility of a source has the weakest effect on selection. familiarity with online library resources and self-reported strong search skills also positively influenced use intention.   commentary   as the selection and use of information sources continues to occur in online spaces, lis researchers and practitioners in higher education settings investigate why users choose the resources that they do. a number of studies have found that convenience and ease of use contribute highly to undergraduate students’ selection of sources for their academic work, including currie et al. (2010) and connaway et al. (2011). the authors of the study at hand examine whether the self-reported intent to use online library resources can be explained by three groups of variables pertaining to usefulness and ease of use, resource quality, and individual differences.   among the many strengths of this study is an extensive review of the literature that grounds the study’s findings in the context of other works examining student preferences in relation to online sources, and the clearly stated intent and outcomes of the research. the description of the data collection process and research participants lacks information for the findings to be fully evaluated. details were omitted regarding how students were recruited for the study, how many students were initially invited to participate, the randomization process, whether the students invited and the students who participated in the study were representative of the undergraduate student population at that university, means of survey distribution, and the name of the university at which the data was collected.   while very thorough in its design and methodological rigor, there are some aspects of the study to acknowledge when considering the results presented. given that the data and findings are based upon survey results, which are necessarily self-reported behaviors, the use of another data collection method to achieve triangulation (such as a qualitative measure including in-depth interviews, focus groups, or observations of student behavior) would strongly bolster the findings’ validity. the study frequently uses causal language such as “influence” or “effect” when terms that describe relationships and correlations would be more appropriate and accurate. the inclusion of a survey instrument would allow for reader evaluation of the instrument and the possibility of replicating the study. additionally, the discussion of the study’s limitations and of potential areas for future research could benefit from additional information.   this study confirms the findings of other research examining students’ preferences for online sources, including that undergraduates are likely to prioritize convenience, ease of use, and familiarity when seeking information. a number of practice implications are outlined, including: librarians could emphasize the suitability of library resources for academic tasks during information literacy instruction, design google-like library search interfaces to increase ease of use, and develop library collections with varied and up to date information, all with the intent of encouraging undergraduates to use more reliable resources in their academic work. the practice recommendations are made in absolute language. however, when there is not always a direct relationship between a finding and practice, less directive language would be more appropriate.   as a suggestion for future studies in the area of the information seeking behavior of undergraduates, researchers and practitioners should attempt to move past the narrative of “digital natives” and, in particular, the assumption that undergraduate students all share similar experiences and expertise. studies that consider information use as a contextualized and local practice will encourage the understanding of learners as individuals with unique backgrounds, and allow for the profession to discard the idea of students as a monolithic construct.   references   connaway, l. s., dickey, t. j., and radford, m. l. (2011). if it is too inconvenient i’m not going after it: convenience as a critical factor in information-seeking behaviors. library & information science research, 33(3), 179-190. doi: 10.1016/j.lisr.2010.12.002   currie, l., devlin, f., emde, j., & graves, k. (2010). undergraduate search strategies and evaluation criteria: searching for credible sources. new library world, 111(3/4), 113-124. doi: 10.1108/03074801011027628     ebl 101   disseminating your research   virginia wilson client services librarian murray library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   originally published in: evidence based library and information practice, 6(1), 99–100. https://ejournals.library.ualberta.ca/index.php/eblip/article/view/9625/7854     received: 22 dec. 2010  accepted: 05 jan. 2011      2016 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   in terms of looking at all the steps of evidence based library and information practice, we're almost home! the final thing to consider once you have gone through the process is disseminating your work. whether you have used the evidence base to inform your practice or decision making, or you have conducted your own research study to do the same, disseminating the results is beneficial for a number of reasons. crumley and koufogiannakis (2002) state that "perhaps the largest obstacle in finding library research is that librarians generally do not publish their research" (p. 69).   when thinking of dissemination, keep three things in mind: ease of comprehension, ease of access, and multiple forums. strive for accessible language and format, and consider publishing in an open access journal so that the information is freely available to your colleagues. in terms of multiple forums, there are several venues to consider when thinking about getting the word out.   first up is publishing. why might you want to publish your own work on your own practice? an important reason is to increase the evidence base. in order to practice ebl, we need to have the knowledge base. and that means publishing research activities in peer-reviewed scholarly journals. eldredge (2006) contends that we as a profession have "an obligation to ensure that results are communicated, even if these results are not dramatic or new, to build a more solid foundation to our knowledge base" (351). but publishing in scholarly journals is not the only way to get the word out. publishing online on blogs, wikis, and personal or professional websites, just to name a few online avenues, also serves to get the message out.   another way to disseminate your message is by educational or professional development means. present a paper or take part in a poster session at a conference. run a workshop, session, or webinar based on the topic of your research. present your findings at a journal club, discussion, group, or any other gathering in a professional setting.   the research that librarians undertake to inform their practice can be integrated into library management and policy in a few ways. making sure that the ultimate decision-makers in the organization are aware of the research activities, or conversely, aware of a lack of evidence for certain issues, can prompt administrators to go forward with an evidence based approach.  presenting evidence either self-generated or found in the literature at library meetings or other internal venues can be the spark needed to push new ideas forward, and can contribute to an evidence based culture shift within the organization.   and finally, personal communication is an excellent way to get the word out about your results and to share what you have found with colleagues. listservs and email work, as do social networking tools such facebook, twitter, wikis, and blogs. chapter 12 in evidence-based practice for information professionals: a handbook by crumley and koufogiannakis, fleshes out all of these examples and is a must-read if you are thinking of getting the results from the evidence based process out there.   it is difficult to practice in a vacuum. and in evidence based library and information practice, just as we need to reach out to find the evidence, so does everyone else who is endeavouring to practice in that way. there needs to be evidence in the evidence base. i believe it is safe to assume that because you had a question you needed to explore, others have or will have the same question. eblip is a way to work collaboratively without even meeting face to face. sharing the evidence; making it available to inform practice, helps everyone in librarianship.   now that i have finished taking you through the steps of ebl, next time i will embark on a new and exciting adventure in ebl 101. i will be looking at various research methods, qualitative and quantitative, and will be giving overviews of the methods including types of research studies for which they would be appropriate. first up, an overview of qualitative and quantitative research.   references   crumley, e. and koufogiannakis, d. (2002). developing evidence-based librarianship: practical steps for implementation. health information and libraries journal 19(2), 61-70.   crumley, e. and koufogiannakis, d. (2004). disseminating the lessons of evidence-based practice. in a. booth & a. brice (eds.), evidence‐based practice for information professionals: a handbook  (pp. 138-143). london: facet.   eldredge, j. (2006). evidence-based librarianship: the ebl process. library hi tech 24(3), 341-354.   evidence based library and information practice 2011, 6.3 59 evidence based library and information practice evidence summary structure may be key to incorporating library school interns in academic library environments a review of: sargent, a. r., becker, b. w., & klingberg, s. (2011). incorporating library school interns on academic library subject teams. the journal of academic librarianship, 37(1), 28-33. doi: 10.1016/j.acalib.2010.10.004 reviewed by: heather r. williams strategist, content processes & services robert w. woodruff library, emory university atlanta, georgia, united states of america email: hrwilli@emory.edu received: 01 june 2011 accepted: 02 aug. 2011 © 2011 williams. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. objective – to evaluate the effectiveness of the san jose state university library internship program. design – focus group; single point in time; qualitative design. setting – large academic library in the united states of america. subjects – nine former interns of the san jose state university (sjsu) library. methods nine former interns of the sjsu library internship program participated in a single 90-minute session. no inducements for participation were offered. a moderator asked a series of 10 questions designed to gather feedback in three areas: 1) “the internship as part of the masters program,” 2) “the internship’s role in the realization of personal objectives and professional development,” and 3) “the experience of working in team based activities.” a digital voice recorder captured the participants’ responses, allowing for detailed analysis of the responses after the session. mailto:hrwilli@emory.edu� evidence based library and information practice 2011, 6.3 60 main results – the interns deemed their overall experience successful, as all indicated they achieved their professional development objectives for the internship. however, the interns also indicated their experience could have been improved by the appointment of a single dedicated coordinator for recruitment and oversight, as well as more feedback on the quality of their work, especially for courserelated instruction. conclusion – the sjsu library determined that the internship program was advantageous to both the library and the interns. all of the interns who participated in the focus group achieved their profession development objectives for the internship. additionally, the library received valuable feedback for improving the program. suggestions included appointing a dedicated internship coordinator, allowing interns more of an opportunity to choose their projects, and ensuring that interns are offered frequent feedback about the quality of their work. commentary this study evaluated the effectiveness of the san jose state university library internship program by gathering feedback from an intern focus group. the results of this study will be of interest to those libraries that have existing internship programs as well as those that are looking to begin programs. the authors clearly described the methodology employed for the focus group, including the fact that although a design of multiple focus groups of six to eight participants would be ideal, they were only able to hold one focus group due to the fact that only nine former interns remained in the san jose area. this single focus group design, however, raises potential anonymity and confidentiality concerns. the participants are likely to be easily identifiable due to their limited number and their discussion of specific internship situations and projects. did this pose any issues for the study? would the participants have responded differently if their anonymity had been guaranteed? the authors did not address these questions, nor did they mention whether this design posed a problem for their university research ethics board. not all university research ethics boards require that all studies pass through their purview, but this could pose an issue for a library attempting to replicate this study. the authors did not indicate how the coding or interpretation of the recorded responses was completed. did the authors have a methodical way of selecting key responses? regardless, the authors did make conclusions, specifically, that the program would be improved by appointing a dedicated coordinator and by offering more frequent opportunities for feedback. other libraries would likely improve or establish more successful internship programs by providing such structure for their library school interns. an internship program should ultimately be advantageous to both the library and the interns. the authors cite the primary benefit of the internship program as being the opportunity to hire experienced former interns. an additional benefit is the unique skills that some interns bring to the internship. ultimately, further research could be done relating to a cost benefit analysis. do the benefits received by the institution outweigh the costs involved in training and mentoring a temporary staff member? the authors acknowledge this area of further research and plan for a survey of both supervisors and faculty about their experiences. / evidence based library and information practice ebl 101   asking the right question   lorie kloda phd student mcgill university school of information studies montreal, quebec, canada email: lorie.kloda@mcgill.ca   originally published in: evidence based library and information practice, 3(4), 79–81. https://ejournals.library.ualberta.ca/index.php/eblip/article/view/4426/3725        2016 kloda. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   for librarians, the idea of “asking the right question” is nothing new. as information professionals, we know that the real question is not the same as the first thing a patron asks at the outset of the reference encounter. similarly, those teaching information literacy recognize the importance of understanding one’s information needs as one of the first steps in the research process.   the first step in the evidence based librarianship (ebl) process is to formulate an answerable question. eldredge draws a parallel between this step and the first step of problem based learning, in which learners are encouraged to express their uncertainties as precise information needs that can be answered using the literature. in the same way, even though you often begin with vague uncertainties regarding your information practice, ebl requires that you turn those uncertainties into more refined questions.   for a question to be answerable, it must be precise or detailed enough to be conceivably answered by research. of course, it is easier to create a detailed question if you are familiar with the subject area, and formulating answerable question takes practice. the benefit to creating a precise, answerable question is that you will be more likely to make a decision based on the answer, should you find one.   another benefit to formulating an answerable question is that it also enables efficient retrieval. as librarians, we all know the value of retrieving a set of literature that is not only high in recall, but high in precision as well. in other words, the concepts present in a detailed question will enable you to develop a search strategy that retrieves only very relevant results.   formulating an answerable question, though, does not always mean that an answer will be available. lewis and cotter found a gap between the topics of questions asked by practitioners (mostly management and education) and those addressed by researchers (mostly information access and retrieval, and collections).   in evidence based medicine and other health disciplines, the formulation of clinical questions is guided by the pico structure (for person or problem, intervention, comparison, and outcome). this structure, proposed by richardson and colleagues, was meant to be helpful in guiding physicians to formulate precise clinical questions. the pico structure, which allows for flexibility (some of the elements cab be omitted) continues to be employed by many health professionals. in library and information practice, the spice structure has been proposed by booth:   setting: the context (e.g., an academic library, law firm) perspective: the  stakeholder (group or individual) interested (e.g., graduate student, manager) intervention: the service being offered (e.g., chat reference, library instruction workshop) comparison: the service to which it is being compared (note that there may be no comparison) evaluation: the measure used to determine success (e.g., usage statistics, visit to the reference desk after regular hours)   as an example, an academic librarian work in a health sciences library may want to know if there are any disadvantages to staffing a chat reference service with paraprofessionals. in order to refine this question into a detailed, answerable question, the librarian can use the spice structure:   setting: academic health sciences library perspective: students, faculty members intervention: chat reference offered by professional librarian comparison: chat reference offered by a paraprofessional evaluation: user satisfaction   using this example, the question can be restated as follows: in an academic health sciences library, does staffing a chat reference service with a librarian instead of a paraprofessional result in greater user satisfaction?   keep in mind that asking questions is an iterative process, as librarians will recognize from the reference interview. it is a necessary and worthwhile endeavour to continually refine and reframe a question until it captures precisely the uncertainty you wish to resolve. this process takes some time and thought, and it is a good idea to make sure you, and if applicable, your colleagues, are in agreement on what exactly is the question before attempting to find an answer, otherwise you risk wasting time looking for, appraising, and applying evidence that is not even relevant to your original uncertainty!   once you have formulated an answerable question, the next step is to identify the appropriate level of evidence for answering it. the next ebl 101 column will focus on matching question types to study designs.   works cited   booth, andrew. “formulating answerable questions.” evidence based practice: an information professional’s handbook. eds. andrew booth and anne brice. london: facet, 2004. 61-70.   eldredge, jonathan deforest. “evidence-based librarianship: formulating ebl questions.” bibliotheca medica canadiana 22.2 (winter 2000): 74-7.   lewis, suzanne and lisa cotter. “have the most relevant and answerable research questions facing librarians changed between 2001 and 2006?” evidence based library and information practice 2.1 (mar. 14, 2007): 107-20.   richardson, w. scott, marc c. wilson, jim nishikawa, and robert s. a. hayward. “the well-built clinical question: a key to evidence-based decisions.” acp journal club 123 (nov./dec. 1995): a12.   microsoft word article_okello_10 font evidence based library and information practice 2008, 3:3 39 evidence based library and information practice article electronic information access and utilization by makerere university students in uganda constant okello-obura lecturer, east african school of library and information science makerere university kampala, uganda e-mail: obura@easlis.mak.ac.ug. elisam magara associate professor, east african school of library and information science makerere university kampala, uganda e-mail: magara@easlis.mak.ac.ug received: 04 december 2007 accepted: 27 july 2008 © 2008 okello-obura and magara. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objectives – the objectives of this study were to establish the level of computer utilization skills of makerere university (uganda) library and information science (lis) students; to determine the use of electronic information resources by lis students; to determine the attitudes of lis students towards electronic information resources; and to establish the problems faced by lis students in accessing electronic information resources. methods – a questionnaire survey was used for data collection. results – the majority of library and information science students at makerere university depend on university computers for their work, and very few of them access the library’s e-resources. the few who access e-resources are self-taught. the majority of students surveyed were unaware of emerald and ebsco databases evidence based library and information practice 2008, 3:3 40 relevant to library and information science students, and they found accessing eresources time-consuming. conclusion – the study concluded that a concerted effort is needed by both lis lecturers and university librarians in promoting use of the library’s electronic resources. introduction makerere university is uganda’s premier institution of higher learning, and enrolls over 30,000 students. makerere is one of the largest universities in east and central africa. the university is also one of the oldest and most prestigious universities in africa. it was established in 1922 as a technical school, and it currently has 22 departments offering day, evening, and distance study programs. in recent years there have been a number of changes in the higher education sector in uganda and, in particular, at makerere university. these dramatic changes include the way in which information is provided to the university community. the university, through a number of donor support programs, has embraced the electronic provision of information to facilitate study programs and research. with the integration of virtual and digital libraries in its library systems, makerere university attempts to disseminate information by means of an electronic hybrid online public access catalogue (opac). the major objectives of the adoption of e-resources are to facilitate access to internet-based information resources, as well as the timely dissemination of both local and international research output. this should result in more timely access to current issues of international research journals and improve user satisfaction with library services. following an analysis of reference lists submitted with assignments in 2004-2005, lis lecturers discovered that students in the library and information science (lis) discipline were not effectively using eresources for their coursework assignments and were not including references located from e-resources. this prompted a number of questions including: • what are the perceptions of makerere university lis students regarding the availability of electronic information? • how frequently do lis students use the e-resources available to them through the library web site? • what are lis students’ attitudes towards substituting e-resources for printed formats? • what problems do makerere university students face in accessing electronic resources? by gaining a better understanding of the situation, lecturers aimed to identify whether students’ information needs were being met effectively and to develop proposals to ensure effective and efficient electronic information resource utilization by lis students at makerere university. this study set out to establish whether students really needed e-resources and to determine their attitudes towards electronic resource utilization and the problems they faced in accessing electronic resources in the university library. evidence based library and information practice 2008, 3:3 41 literature review the need for electronic information resources according to shuling (2007), electronic information has gradually become a major resource in every university library in recent years. the growth and diversity of electronic resources, especially e-journals, has led many to predict the extinction of the printed journal. it has been suggested that a new paradigm is sweeping scholarship (liew, foo, and chennupati; harper et al.). majid and abozova argued that technological advancements opened up new horizons for the creation, storage, access, distribution and presentation of information. in the global information communications technology (ict)dominated world, “place” is less important (ferguson). “the impact of moving from text-based to resource-based learning has involved heavier use of library materials and a demand for more and varied media sources” (kinengyere). this makes the provision and use of electronic information systems in academic libraries a critical issue for those working in information and library services (armstrong et al.; elam). the pace at which information resources are being produced and converted into electronic formats is greater today than in previous years (armstrong et al.). in today's information age it would seem that library users would not only be eager to take advantage of the convenience electronic resources have to offer, but would be fully immersed in the new technologies (elam). electronic information resources offer today's students new opportunities not available to previous generations. liew, foo, and chennupati (302) argue that while reading an e-journal is not the same as reading a printed issue, many people now acknowledge that electronic documents offer users advanced features and novel forms of functionality beyond those possible in printed form. brophy noted that the advantages of electronic resources over print include speed, ease of use, ability to search multiple files at the same time, ability to save, print and repeat searches, more frequent updating, and the ability to access documents from outside the library (a particular advantage for the distance learner). electronic resources are invaluable research tools that complement the print-based resources in a traditional library. their advantages include access to information that might be restricted to the user due to geographical location or finances, access to more current information, and provision of extensive links to additional resources or related content (dadzie). the arguments for students using electronic resources are compelling. however, a knowledge of computers and retrieval techniques is needed to search these resources effectively. it is therefore necessary to determine what computer skills students need to access library electronic information resources. information retrieval skills tella et al. noted that students’ ability to find and retrieve information effectively is a transferable skill useful for their future lives as well as enabling the positive and successful use of academic electronic resources. the ability to explore the digital environment is a requirement for academic success today. students are increasingly expected to use electronic information resources at the university. in order to make use of the growing range of electronic resources, students must acquire and practice the skills necessary to exploit them. skills learning is essential in a technology driven environment, and it can be enhanced through the use of innovative learning evidence based library and information practice 2008, 3:3 42 strategies (lawson). the skills required to achieve the maximum potential of electronic resources are much greater than those required for searching printed sources (ray and day). these skills include knowledge of the structure of the database, the instructions that the searcher must enter into the computer, and an understanding of the ways these instructions are linked to one another. furthermore, it has been suggested that libraries should ensure that all university students acquire information skills to enable them to operate in an information society once outside the university environment (brophy). the literature highlights the advantages of electronic over print sources of information, and it also suggests the need for certain skills in order to access and use electronic resources effectively. given their apparent failure to use electronic resources, this study sets out to determine if lis students at makerere university have the required skills to access and utilize electronic resources. objectives of the study the objectives of this study were to • establish the level of computer utilization skills of makerere university lis students. • determine the use of electronic information resources by lis students at makerere university. • determine the attitudes of lis students towards electronic information resources. • establish the problems faced by lis students in accessing electronic information resources. • recommend appropriate measures to improve electronic information access and utilization at makerere university. methodology the study was conducted at the east african school of library and information science, makerere university, uganda. the study population included a total of 250 second and third-year bachelor of library and information science (blis) students. these students were selected as the study population because they had been exposed to information and communication technologies (icts) in their first year of studies, and because they were expected to use electronic resources for their coursework assignments and research. the study used a quantitative approach in which a structured survey questionnaire was used for data collection. the questionnaire (appendix 1) was pre-tested on ten secondand ten thirdyear bachelor of library and information science students who did not participate in the main study. following pre-testing, some questions were re-worded for clarity. one researcher administered the questionnaire during participants’ lectures (both day and evening students) over a two-day period. the responses were checked for legibility by a researcher and entered into an excel spreadsheet by a data entry assistant. a researcher performed further quality checking of the data before analysis. excel was used to perform simple descriptive statistics that are highlighted below. findings characteristics of the respondents of the 250 targeted students, 190 responded, resulting in a response rate of 76%. the majority of students (55.3%) were female, while 44.7% were male. computer utilization skills of the respondents computer utilization skills were defined for the purposes of this project as “the skills a evidence based library and information practice 2008, 3:3 43 computer user needs in order to use the computer.” the study established that all respondents had some basic computer utilization skills (table 1). the majority (60.5%) had acquired their computer utilization skills at secondary level (prior to joining the university), whereas 26.3% had acquired them at degree level. only 13.2% of the respondents had acquired their computer utilization skills at primary level. none of the participants had acquired computer skills at diploma level. the majority of the respondents did not have skills in a wide range of computer applications. students were most proficient in word processing, spreadsheets, database management, and desktop publishing. table 1 respondents’ computer utilization skills computer packages (skills) responses number (percentage) word processing 91 (47.9%) spreadsheet 75 (39.5%) database management 75 (39.5%) desktop publishing 75 (39.5%) internet and e-mail use 52 (27.4%) programming 24 (12.6%) web page design 24 (12.6%) computer repairs and maintenance 10 ( 5.3%) the majority of lis students did not believe they had the knowledge or skills for eresources retrieval (table 2). table 2 respondents’ knowledge and skills with eresources retrieval knowledge and skills responses number (percentage) knowledge on the determination of appropriate search term 83 (43.7%) use of boolean logic searching 78 (41.1%) knowledge on database structure 75 (39.5%) using different search engines 50 (26.3%) to understand whether respondents seek assistance when they have computer problems, respondents were asked to state how often they asked for assistance. the majority (55.3%) of respondents admitted to sometimes asking for assistance, 32.6% said they rarely did so, and only 12.1% said they always asked for assistance. no respondent admitted to never asking for assistance. electronic information resources usage one of the objectives of the study was to establish the usage of electronic resources by lis students at makerere university. almost three-quarters of the students (74.7%) depended on the university for computers to access to the networked resources, and the remainder (25.3%) accessed computers either at home or at their hostels. respondents stated that they used networked computers for academic (study) purposes (41.1%); e-mail services (53.7%); and sending text messages to cell phones (44.2%). when the respondents were asked to review a list of the library’s electronic resources and to select the products they used, the results evidence based library and information practice 2008, 3:3 44 indicate that 55.8% used the internet; 30.5% used the library opac; and 5.3% used cdrom titles. survey respondents reported little use of the library’s full-text databases (table 3). emerald and ebsco full-text databases contain titles that would be most relevant to the academic needs of lis students, and these were the most frequently used products. nevertheless, the use of these two products was only minimal (6.32%). the remainder of the products were each accessed by 1% of respondents or less. respondents were asked how they learned to access electronic resources and how they acquired database search skills. the majority are self-taught (73.7%), and a large percentage learns from fellow students (47.4%), trial and error (42.1%), or lecturers (42.1%). only 5.3% acquired their skills in use of electronic resources from guidance given by library staff. survey respondents indicated a variety of perceived benefits related to use of the library’s electronic resources (table 4). while the majority (70%) perceived eresources as providing access to a wider range of information, only 18.4% believed these resources provided easier access to information. this might be attributed to the lack of available personal computers (table 5). it is significant that 33% of the students did not consider that access to quality information via e-resources led to improved academic performance. table 3 utilization of full-text databases available at makerere university* full-text databases yes, accessed and utilized the full-text database (number of respondents) percentage (%)* agora 1 0.53% blackwell synergy 0 0 cambridge journals online 1 0.53% cochrane library 0 0 ebsco (journal collections) 12 6.32% emerald group publishing ltd. 12 6.32% gale (thomas learning) 0 0 hinari 1 0.53% institute of physic publishing (iopp) 0 0 oxford university press (oup) 2 1.1% palgrave macmillan journals 2 1.1% royal society of london 1 0.53% springerlink 0 0 * percentage computed based responses from 190 respondents. evidence based library and information practice 2008, 3:3 45 table 4 benefits students derive from e-resources benefits derived from electronic information resources responses number (percentage) access to a wider range of information? 133 (70.0%) faster access to information? 107 (56.3%) access to current/up-todate information? 76 (40.0%) easier access to information? 35 (18.4%) improved academic performance as a result of access to quality information? 63 (33.2%) other? (e.g., it is modern, saves money in photocopying etc) 49 (25.8%) attitudes of lis students towards electronic information resources the questionnaire asked survey participants to respond to the following statements to help determine students’ attitudes toward electronic information resources, i feel that the standard of my academic work would suffer without electronic information resources. a large majority (90%) of the respondents agreed or strongly agreed with the statement that the standard of their academic work would suffer without electronic resources; only 10% disagreed. i can avoid the utilization of electronic information resources and still perform better. the respondents were equally divided on this statement. fifty percent of the respondents strongly agreed with the statement, while 50% strongly disagreed. none of the respondents chose the “agree” or the “i don’t know” options. given the opportunity to choose between electronic resources and printed resources at makerere university to do my coursework, i would choose printed materials. responses were mixed concerning printed versus electronic resources. sixty percent of the respondents either “strongly preferred” or “preferred” printed materials over electronic resources, and 40% of the respondents did not choose printed materials over electronic resources. the majority of the students preferred to teach themselves or have lecturers teach them how to use e-resources (figure 1). evidence based library and information practice 2008, 3:3 46 problems faced in accessing electronic resources the study also sought to identify the main problems users had in accessing electronic resources. respondents were asked to select from a list of possible problems listed in table 5. respondents said the main barrier to their being able to access e-resources was their limited access to a computer terminal (90%). time, lack of it knowledge, and retrieval of too much information were also cited as barriers by large numbers of students. together these can be viewed as a result of ineffective information retrieval skills as effective information skills are likely to result in the retrieval of more manageable amounts of information and a more efficient use of time. uncooperative staff members were also perceived as a problem by 20% of the respondents. who should make you use electronic resources? personal initiative lecturers librarians computer technicians fig. 1. students’ opinions about who should teach them to use e-resources table 5 problems students faced in accessing e-resources problems faced in accessing e-resources responses number (percentage) limited access to a computer terminal 171 (90.0%) it is time consuming and wastes a lot of my time 97 (51.1%) lack of it knowledge to effectively use the services 76 (40.0%) there is too much information retrieved 76 (49.0%) uncooperative staff to facilitate easy access 38 (20.0%) using electronic resources often distracts me from doing other work 1 ( 1.0%) evidence based library and information practice 2008, 3:3 47 proposals to improve on e-resources access and utilization in makerere university respondents’ views were canvassed on a range of proposals that would potentially improve e-resources access (table 6). there was a high level of agreement with almost all of the proposals listed. students recommended an improved awareness campaign for e-resources and compulsory use of e-resources for coursework assignments, but a suggestion to introduce a fee to support use of e-resources was supported by only a very small percentage of the students. discussion the massive impact of ict on the profession of librarianship has changed the way librarians and support staff do their jobs and interact with users and colleagues (oduwole and sowole). however, the question of knowing how to use a computer will remain a challenge to most students in developing countries such as uganda, where access to computers is a serious problem. learning basic computer skills and software applications is increasingly necessary to function in today's workplace or to pursue personal interests in an electronic environment. this knowledge gives library users a practical understanding about how their computers and printers operate, how to troubleshoot problems, how to locate an internet web site, and a host of other technology-based skills that help a library user become more successful in the technological world (lawson). for the effective use of electronic resources, computer utilization and information literacy skills are essential. an analysis of the findings indicates that the majority of university students at makerere university did not have the necessary computer skills and had problems accessing personal computers. some but not all students arrive at makerere university with computer utilization skills, this raises a question concerning the level at which university information technology training should start. with increasing numbers of students entering higher education and the numbers of staff not keeping pace with this increase, it is not table 6 recommendations on ways to improve accessibility and utilization of e-resources at makerere university suggestions responses number (percentage) computer skills of students should be improved 182 ( 95.8%) first-year students should be introduced to icts and information literacy 187 ( 98.4%) lecturers should insist that students use e-resources for coursework/assignments 99 (52.1%) librarians should be trained to work with students 188 (98.9%) more networked computers should be purchased by the university 190 (100.0%) university libraries should improve an awareness campaign of e-resources 103 (54.2%) text message services should be integrated into library e-resources services provision 187 (98.4%) university should introduce student-paid e-resources fee 11 ( 5.8%) others, please, specify 4 ( 2.1%) evidence based library and information practice 2008, 3:3 48 possible to ensure that all students have acquired the needed skills. this poses serious challenges for educators, especially in terms of those students who do not seek assistance or guidance from lecturers. according to ray and day, extensive experience in user education programs has shown that teaching information retrieval skills to students should be embedded in the curriculum, and instruction should be given at a time when users can understand its appropriateness. this training should be adapted to the varying abilities of the users. if students are aware that the skills required for using electronic resources are not insular, and that these skills will provide them with valuable transferable lifelong skills employers are seeking, they may be more likely to learn how to use them (ray and day; oduwole and sowole). to access electronic resources, networked computers are necessary ingredients that any university should strive to have. this study established that 74.7% of the respondents depended on the university to access networked computers, and that 90% had problems accessing personal computers (table 5). further, 100% of the respondents proposed that more networked computers should be purchased by the university to help students access e-resources (table 6). it is likely that very few of the students surveyed would be able to afford access to electronic resources without university support. a more difficult question remains as to whether the university library can afford to increase the number of networked computers to match the increasing number of students at makerere university. possible solutions might be national and external lobbying for support or charging fees to enable students to access electronic resources. this finding however, sharply contrasts with a study conducted in the united kingdom at glasgow caledonian university, where crawford noted that internet access from home and from work has caused a declining demand for internet and electronic information services usage on campus. this should be taken into account on the future provision of library services for developed countries, but developing countries such as uganda need to address the issue of having adequate networked computers on campus. students’ reasons for use of the university’s networked computers were not solely related to academic work or course preparation. although a large proportion (41.2%) used the campus computers for academic study, the majority used networked computers for sending emails (53.7%) and short messages (44.2%). compulsory use of e-resources for assignments would improve usage, but this would require more financial support or collaboration to acquire additional computers. the makerere university library through a number of donor initiatives subscribes to a number of databases, including emerald and ebsco, which provide full-text journal articles for studies in library and information science. these databases are not used to their maximum potential. this lack of use could be attributed to a number of factors. in addition to access, there may be a lack of awareness amongst the students of the availability of the databases, and they may not be aware of their relevance and value to their lis studies. although the library has made an effort in promoting awareness of these resources within the university community, there is still a need for a more vigorous awareness campaign. experience has shown that because of the importance students attach to grades on course assignments, using lecturers to make use of electronic resources compulsory might be a positive way forward. lecturers may not currently be actively encouraging students to familiarize themselves with these resources and use evidence based library and information practice 2008, 3:3 49 them. academic staff members also seem reluctant to use electronic resources, possibly because of a lack of awareness of the resources and their functionality. training from library staff might be useful, but further investigations surrounding these assumptions are necessary. myers, saunders, and rogers warn that although the concept of the hybrid and virtual library has been much discussed with the advent of electronic resources, librarians in some developing countries have also been slow to move away from traditional library concepts and formats. as a result, many libraries simply emulate dry, dull replicas of the traditional library catalogue or bibliographies in their design of links to web-based resources (myers, saunders, and rogers). conyers argued that with the advent of electronic journals, libraries now have the opportunity, as never before, to obtain robust quantitative data about levels of use of their periodicals and to analyse how far their investment in their journals collections represents value for money for their institutions. this is an opportunity that makerere university should utilize, and the library should design strategic interventions to promote and monitor e-resources utilization. using a monitoring system, the university library could identify lecturers who encourage students to use e-resources with prizes for the most use. another way might be to send lecturers e-mail alert messages when changes to the database are announced. although the level of usage is low, it is encouraging to find that the majority of students make an effort to teach themselves how to use electronic resources. the minimal contribution which appears to be made by librarians raises a number of questions, such as: are librarians not willing to help students or do the students not consider librarians important in their literature search? al-baridi and ahmed note that an increasingly important function of academic libraries, today, is the provision of information in electronic formats, including indexes, full-text articles, complete journals and web resources. students need to understand that librarians can be very useful in assisting them in their electronic information searching. librarians need to assert their positions and make their positions (roles) seen by library users through promotion and efficient service delivery. a number of respondents indicated that staff members were uncooperative, and this is a concern for improving e-resources utilization and the development of the university library. okon asserted that the core skills traditionally associated with information professionals, which include information handling skills, training and facilitating skills, evaluation skills and concern for the customer, are all still relevant. it is possible that library staff would benefit from further training, perhaps in customer care. although the majority (70%) of the respondents accepted that e-resources provide access to a wide range of information, lack of access to networked computers remained the largest reported barrier to computer use (table 5). dickinson argued that electronic resources greatly increase access to information. the lack of computers and frequent power outages limited students’ ability to effectively access electronic information resources, and these are issues that need to be addressed in both the university’s main and branch libraries in the university. the problem of lack of adequate access to networked computers confirms dadzie’s argument that most studies carried out by african authors cited inadequate pcs as one of the major barriers to electronic or internet access. a study carried out by shuling in the library of shaanxi university of science and technology, xianyang, shaanxi, china also found that a lack of access to the internet evidence based library and information practice 2008, 3:3 50 and computer are the main obstacles for the readers using electronic resources. this appears to be a problem for developing countries where the costs of technology and communication infrastructures far exceed their allocated budgets. although students have problems in accessing electronic resources, the majority (75.3%) agree that their standard of academic work would suffer without electronic resources, suggesting a positive attitude towards them. the university library should address the problem of lack of retrieval skills for e-resources among students. although kinengyere clearly outlined efforts made by individual programs to increase e-resources usage including organising training sessions on how to access the opac and other end userprograms, more is required to enable efficient utilization of e-resources by students. this could include training on searching open access journals and subscription based databases, using boolean logic, training on database structure, or providing searching aids for them to at least some degree (shuling). the libraries of the future will be more of a portal through which students and staff will access the vast information resources of the world and less of a place where information is kept. therefore, the need to use electronic resources is of paramount importance to developing countries if access to up-to-date e-resources is to be realized. conclusions and recommendations modern university education now depends on a robust program of automated information services to support and facilitate teaching, learning, research, and management. developing countries need to adapt to the new situation of handling library services. makerere university is trying to provide this support despite inadequate access to networked computers. the findings of this study on the use of electronic information resources show that students could exploit the benefits of electronic resources in their academic work. however, a number of issues need to be addressed internally beforehand. these include: • the current method of training lis students in icts at the beginning of their career at the east african school of library and information science (easlis) should be maintained in order to promote computer utilization skills. information technology, especially online retrieval and database searching, should be integrated into the curriculum and delivery of all lis programs in makerere university. as sani and tiamiyu noted for nigerian universities, programs should be initiated to enable all university teachers to learn how to use modern information technologies to deliver courses and to interact with students (e.g., e-mail, file sharing) in all programs. this will help provide the necessary ict skills for eresources access. • the university library could improve the promotion of electronic resources to lis students by providing training in e-resources retrieval skills as part of their curriculum. it may be necessary for library staff to receive training in customer care and e-resource utilization in order to provide training more effectively. • encouragement by academic staff should be explored as a way of promoting the usefulness of electronic resources to students. if academic staff were to promote electronic resources by providing evidence based library and information practice 2008, 3:3 51 references for students to locate, this might increase the number of students acquiring the necessary information retrieval skills. lecturers are crucial in the promotion of electronic information resources utilization. coursework assignments requiring the use of electronic resources should be developed. this will compel students to utilize electronic resources. • university libraries should intensify their efforts to promote the availability of electronic resources. the university library should consider using e-mail alert messages, text messages, and prizes as methods of promoting use of the library’s electronic resources. • more networked computers should be provided on campus. this could be achieved through soliciting external support and charging eresource fees to be paid by students. lis schools should employ computer laboratory assistants who are interested and willing to offer assistance to students during information retrieval. • there should be a university-wide study on students’ access and use of electronic information resources at makerere university. if lis students who should know the importance of electronic information resources are not optimally utilizing them, what is the situation with students from other faculties? • a course in information literacy should be made compulsory for all students irrespective of their discipline. this will go a long way in increasing the knowledge level of the learners regarding the use of electronic information. works cited al-baridi, saleh, and syed sajjad ahmed. “developing electronic resources at the kfupm library.” collection building 19.3 (2000): 109-16. 21 aug. 2008 <http://www.emeraldinsight.com/10 .1108/01604950010337669>. armstrong, chris, roger fenton, ray lonsdale, david stoker, rhian thomas, and christine urquhart. “a study of the use of electronic information systems by higher education students in the uk.” program: electronic library and information systems 35.3 (2001): 241-62. 21 aug. 2008 <http://www.emeraldinsight.com/10 .1108/eum0000000006949>. brophy, peter. “networking in british academic libraries.” british journal of academic librarianship 8.1 (1993): 49-60. conyers, angela. “e-measures: developing statistical measures for electronic information services.” vine: the journal of information and knowledge management systems 34.4 (2004): 148-53. 21 aug. 2008 <http://www.emeraldinsight.com/10 .1108/03055720410570911>. crawford, john. “the use of electronic information services by students at glasgow caledonian university.” vine: the journal of information and knowledge management systems 34.3 (2004): 113-8. 21 aug. 2008 <http://www.emeraldinsight.com/10 .1108/03055720410563478>. evidence based library and information practice 2008, 3:3 52 dadzie, perpetua s. “electronic resources: access and usage at ashesi university college.” campus-wide information systems 22.5 (2005): 290-7. 21 aug. 2008 <http://www.emeraldinsight.com/10 .1108/10650740510632208>. dickinson, gail k. selection and evaluation of electronic resources, englewood, co: libraries unlimited, 1994. elam, barbara. “readiness or avoidance: eresources and the art historian.” collection building, 26.1 (2007): 4-6. 21 aug. 2008 <http://www.emeraldinsight.com/10 .1108/01604950710721539>. ferguson, anthony w. “print to electronic: the university of hong kong case.” collection building 25.3 (2006): 85-8. 21 aug. 2008 <http://www.emeraldinsight.com/10 .1108/01604950610677558>. harper, paulina v., kyrille goldbeck, margaret hogarth, david greenebaum, david magolis, and millie jackson. the 1st electronic resources and libraries conference: a report. library hi tech news 23.5 (2006): 12-22. kinengyere, alison annet. “the effect of information literacy on the utilization of electronic information resources in selected academic and research institutions in uganda.” the electronic library 25.3 (2007): 328-41. 21 aug. 2008 <http://emeraldinsight.com/10.1108/ 02640470710754832>. lawson , karen g. “using eclectic digital resources to enhance instructional methods for adult learners.” oclc systems & services: international digital library perspectives 21.1 (2005): 49-60. 21 aug. 2008 <http://www.emeraldinsight.com/10 .1108/10650750510578154>. liew, chern li, schubert foo, and k.r. chennupati. “a study of graduate student end-users’ use and perception of electronic journals.” online information review 24.2(2000): 302-15. 21 aug. 2008 <http://www.emeraldinsight.com/10 .1108/14684520010350650>. majid, shaheen, and alfia fanilievna abazova. “computer literacy and use of electronic information sources by academics: a case study of international islamic university of malaysia.” asian libraries 8.4 (1999): 100-11. 21 aug. 2008 <http://www.emeraldinsight.com/10 .1108/10176749910275867>. myers, glenda, suzanne saunders, and geoff rogers. “beyond the virtual library: electronic curriculum web resources.” the electronic library 20.6 (2002): 473-80. 21 aug. 2008 <http://www.emeraldinsight.com/10 .1108/02640470210453958>. neuman, wl. “social research methods: qualitative and quantitative approaches,” 5th ed. boston: allyn and bacon, 2003. oduwole, a.a., and a.o. sowole. “utilisation and impact of the essential electronic agricultural database (teeal) on library services in a nigerian university of agriculture.” program: electronic library and information systems 40.2 (2006): 157-67. 21 aug. 2008. <http://www.emeraldinsight.com/in evidence based library and information practice 2008, 3:3 53 sight/viewcontentservlet? filename=published/emeraldfullte xtarticle/articles/2800400205.html>. okon, henry itohowo. effective communication and smooth administration of academic libraries in the 21st century: a new paradigm in nigeria. library philosophy and practice 8.1 (fall 2005). 21 aug. 2008 <http://www.webpages.uidaho.edu/ ~mbolin/okon.htm>. ray, kathryn, and joan day. "student attitudes towards electronic information resources." information research 4.2 (1998). 21 aug. 2007 <http://informationr.net/ir/42/paper54.html>. sani, abdulraheem, and muta tiamiyu. “evaluation of automated services in nigerian universities.” the electronic library 23.3(2005): 27488. 21 aug. 2007 <http://www.emeraldinsight.com/10 .1108/02640470510603679>. shuling, wu. “investigation and analysis of current use of electronic resources in university libraries.” library management 28.1/2 (2007): 72-88. 21 aug. 2008 <http://www.emeraldinsight.com/10 .1108/01435120710723563>. tella, adeyinka, adedeji tella, c.o. ayeni, and r.o. omoba. “self-efficacy and use of electronic information as predictors of academic performance.” electronic journal of academic and special librarianship 8.2 (summer 2007). 21 aug. 2008 <http://southernlibrarianship.icaap. org/content/v08n02/tella_a01.html>. evidence based library and information practice 2008, 3:3 54 questionnaire topic: electronic information access and utilization by makerere university students dear respondent, this is to kindly request you to candidly answer the questions asked in this questionnaire. the answers will be used to provide strategies to improve on e-resources utilization in makerere university. all answers will be treated as confidential. section a: background information 1. gender [1] male [2] female 2. year of study………………………………….. section b: computer skills 3. at what level of education did you learn how to use a computer? [ 1] nursery [2] primary [3] secondary [ 4] diploma [5] university 4. a. what computer skills do you have? [1] word-processing [2] spreadsheets [3] database management [4] internet and e-mail use [5] programming [6] web page design [7] desktop publishing [8] computer repair and maintenance. [9] other, please, specify b. which knowledge or skills do you have for e-resources retrieval? [ ] word processing [ ] determination of appropriate search terms [ ] use of boolean logic [ ]knowledge on database structure [ ] using different search engines [ ] others, please specify………………… c. when you are using computer to do your work, how often do you ask for assistance when you have problem with using the software? [ ] always [ ] sometimes [ ] rarely [ ] never section c: electronic resources usage 5. do you get access to a networked computer from: a: makerere university? [1] yes [2] no b. home/hostel [1] yes [2] no 6. when you access a networked computer, what do you mostly use it for? tick as applicable [1] academic purposes [2] viewing pornography [3] e-mail services [4] sending sms [5] other, please, specify…………………….. 7.a. which electronic resources do you use at makerere university? [1] cdroms [2] internet [3] e-mail [4] electronic journals [5] database of thesis and dissertation (datad) [6] opac [7] other, please specify………. b. which of the following full text journal databases have you ever accessed from makerere library: [1] agora: http://www.aginternetwork.org/en/ evidence based library and information practice 2008, 3:3 55 [2] blackwell-synergy: http://www.blackwell-synergy.com [3] cambridge journals online: http://journals.cambridge.org [4] the cochrane library: http://www3.interscience.wiley.com/cgibin/mrwhome/106568753/home [5] ebsco: http://search.epnet.com [6] emerald group publishing limited: http://www.emeraldinsight.com/login [7] gale (thomas learing): http://infotrac.london.galegroup.com/itweb/makuni [8] hinari: http://www.healthinternetwork.org/scipub.php [9] institute of physics publishing (iopp): http://www.iop.org/ej [10] oxford university press (oup): http://www3.oup.co.uk/jnls/ [11] palgrave macmillan journals: http://www.palgrave-journals.com/pal/jnlsubject.html [12] the royal society of london: http://www.pubs.royalsoc.ac.uk/ [13] springerlink: http://www.springerlink.com c. how did you learn how to use computer to access electronic resources? (choose as applicable) [1] trial and error [2] guidance from other students [3] guidance from library staff [4] self taught [5] courses offered by university or college [6] guidance from lecturers [7] external courses [8] guidance from technicians 8. a. what benefits have you derived from electronic information resources utilization? [1] access to a wider range of information [2] faster access to information [3] access to current up-to-date information [4] easier access to information [5] improved academic performance as a result of access to quality information [6] other, please, specify section d: attitudes towards electronic resources please, rate your feelings or opinions to the following statements: 9. a. i feel that the standard of my academic work would suffer without electronic resources [1] strongly agree [2] agree [3] i don’t agree [4] i don’t know [5] i don’t use it b. “i can avoid the utilisation of electronic information resources and still perform better” [1] strongly agree [2] agree [3] i don’t agree [4] i don’t know c. given the opportunity to choose between electronic resources and printed resources at makerere university to do my coursework, i would choose printed materials. [ ] strongly agree [ ] agree [ ] disagree d. who should make you use electronic resources? evidence based library and information practice 2008, 3:3 56 [1] my self – personal initiative [2] lecturers [3] librarians [ ] computer technicians section e: problems faced with electronic resources 10. what problems do you face with electronic resources? [1] there is too much information retrieved [2]time consuming and waste a lot of my time [3] limited access to a computer terminal [4] lack of it knowledge to effectively utilise the services [5] using electronic resources often detracts me from doing my other work [6] uncooperative staff to facilitate easy access [7] other, please, specify…………………………………………….. 11. suggest ways to improve e-resources utilisation in makerere university. [ ] computer skills of students should be improved [ ] first year students should be introduced to icts and information literacy [ ] lecturers should insist on students using e-resources for coursework [ ] librarians should be trained to know how to work with students [ ] more networked computers should be purchased by the university [ ] university libraries should improve on awareness campaign of e-resources [ ] phone short text message services should be integrated into library e-resources services provision [ ] the university should introduce e-resources fee to be paid by students [ ] others specify……………………………………………………………………….. thank you very much for your time spent evidence summary   differences in work/life balance and stress at work between male and female academic librarians   a review of: galbraith, q., fry, l., and garrison, m. (2016). the impact of faculty status and gender on employee well-being in academic libraries. college & research libraries, 77(1), 71-86. https://doi.org/10.5860/crl.77.1.71     reviewed by: alisa howlett coordinator, evidence based practice university of southern queensland library springfield, queensland, australia email: alisa.howlett@usq.edu.au   received: 9 sept. 2019                                                               accepted:  18 oct. 2019      2019 howlett. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29649     abstract   objective – to measure job satisfaction, personal fulfilment, work/life balance, and stress levels of male and female academic librarians.   design – survey.   setting – arl institutions.   subjects – male and female librarians who work in arl institutions.   methods – the survey was emailed to deans of 110 arl libraries for completion by professional librarians. participants were asked to rate their work/life balance, job satisfaction, stress at work, and personal fulfillment on likert scales (1 low -7 high). overall, 846 librarians from 25 arl libraries responded to the survey. in total, 719 valid responses were analysed using a 2-tailed 2-sample t-test and multiple linear regression to explore variables.   main results – results of this study indicate that differences exist between male and female librarians’ well-being in academic libraries. differences in work/life balance and stress at work were most significant. however, at non-faculty institutions this difference was smaller between male and female librarians than faculty institutions. hours worked per week and the number of years worked at the library were found to have a statistically significant impact on work/life balance. data analysis also suggested that there is no association between gender and job satisfaction and personal fulfillment. tenure at faculty institutions also did not have a statistically significant impact on job satisfaction.   conclusion – the study concluded that support for workplace flexibility and well-being may make the most difference in reducing stress and promoting work/life balance by librarians at arl institutions.   commentary   despite constant change to librarian roles and our library and information science practice over recent decades, little is known about how we experience our work and work-related issues such as stress and fulfillment. student behaviour, our roles and work culture, and expectations of self are sources of stress identified in personal experiences (farler and broady-preston, 2012; larrivee, 2014). linden, salo and jansson (2016) suggest that individuals and workplace interventions should work together to manage stress and burnout. galbraith, fry, and garrison’s study (2016) of how male and female librarians differ in their work experiences aims to fill this knowledge gap by focusing specifically on the potential impact of gender and faculty status on well-being.   galbraith, fry, and garrison use a quantitative survey design with likert scales to assess well-being as measured by work/life balance, job satisfaction, stress at work, and personal fulfillment. appraisal of this study, using a checklist from the centre for evidence-based management, revealed that the overall research design was appropriate to achieve its aims. description about how the survey was developed however, lacks transparency with authors basing the design on experience in organisational behaviour. related literature was not cited in informing the design. also, the authors do not indicate whether the survey was piloted prior to data collection. an appendix for the survey was mentioned within the article but was not found. the authors did demonstrate rigour in analysis by detailing how the sample was refined and the methods were applied to the survey data, including statistical significance.   the authors caution that the t-test results should be considered together with the multiple regression analysis for a more reliable insight into the relationships between gender, faculty status, and other variables on employee well-being. although over 700 survey responses were considered valid and included in the analysis, less than 25% of arl libraries were represented in the data sample. further to this, the response rate is not known, therefore it is difficult to determine whether the unequal sample sizes between male and female librarians are representative of the target population. results must be interpreted with caution and readers must note that the study relates specifically to arl institutions and not libraries in general.   results of this study provide an insight to the interplay between factors impacting on employee well-being in the lis profession. although the authors do not discuss specific practical applications, the results show that both male and female academic librarians are relatively personally fulfilled by the work they do. library organisations should continue to converse, and work with employees, particularly in relation to workload expectations and the number of hours worked per week, to address concerns and identify supports and interventions that may be needed to enhance well-being at a local level.   references   center for evidence-based management. (n.d). critical appraisal of a survey. retrieved from https://www.cebma.org/wp-content/uploads/critical-appraisal-questions-for-a-survey.pdf   farler, l. and broady-preston, j. (2012). workplace stress in libraries: a case study. aslib proceedings, 64(3), 225-240. https://doi.org/10.1108/00012531211244509   larrivee, a. (2014). exploring the stressors of new librarians. public services quarterly, 10(1), 1-10. https://doi.org/10.1080/15228959.2014.875766   lindén, m., salo, i. and jansson, a. (2018). organizational stressors and burnout in public librarians. journal of librarianship and information science, 50(2), 199-204. https://doi.org/10.1177/0961000616666130   evidence summary   open access complements interlibrary loan services, but additional user education is needed   a review of: baich, t. (2015). open access: help or hindrance to resource sharing? interlending & document supply, 43(2), 68-75. http://dx.doi.org/10.1108/ilds-01-2015-0003   reviewed by: richard hayman assistant professor & digital initiatives librarian mount royal university calgary, alberta, canada email: rhayman@mtroyal.ca   received: 29 nov. 2015 accepted: 18 jan. 2015      2016 hayman. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract    objective – to examine interlibrary loan (ill) request rates for open access (oa) materials and determine how oa may affect resource sharing. this research updates the author’s previous study.   design – quantitative analysis.   setting – a large, urban, public research university library system in the united states of america.   subjects – 1,557 open access ill material requests among 23,531 total ill requests submitted during the 2012 and 2013 fiscal years (july 2011-june 2013).   methods – the library has tracked and recorded oa requests that appear among ill material requests since 2009. using oclc’s illiad software to manage ill requests, they have implemented two custom routines. one routine is for open access searching on standard items, and uses software plugins to search across various open resources. all materials published prior to 1923 are treated as being in the public domain, so requests for these materials are automatically routed to this queue. the second custom routine is used for searching for oa electronic theses and dissertations, and is employed when the requested resource is not found in the library’s subscription resources. other article requests are routed to the rapidill service for open access availability.   main results – the research presented reveals that ill requests for oa materials exhibited a steady increase year over year, while overall ill requests decreased slightly. this finding is true both for the fiscal years reported in this study and also the years since the author’s original study in 2011 (baich, 2012). of the 1,557 oa requests examined, 72% (n=1,135) were for journal articles, 8% (n=125) were for books or book chapters, 9% (n=140) were for theses or dissertations, 3% (n=54) were for conference papers, and 7% (n=105) were for reports.   library staff typically fill these article requests using gold oa or green oa sources. the researcher notes the difficulty in refining by source, though confirmed that 15% of articles requested (n=170) were filled using a gold oa source, and that another 30 article requests (~2.6%) were filled with materials available in the public domain. this leads to the conclusion that the majority of article requests are filled using green oa sources. as the library also includes oa collections within its electronic resources, staff filled 13% of ill article requests (n=152) using journals and repositories from these sources. another 16% of article requests were filled using a combination of various online open repositories, including subject repositories (n=83), institutional repositories (n=84), or national or consortial repositories (n=16).   the author includes a similar breakdown of fulfillment rates and sources for the other main categories explored – books and book chapters, theses and dissertations, conference papers, and reports – representing a combined 27% of all oa ill requests. regarding this content, it is noteworthy that overall open access requests for these material categories has dropped across each category when compared to the author’s previous study, with the exception of report requests, which more than doubled compared to that previous study.   the study includes a brief overview of the user status for users making the various open access requests, with undergraduate students (n=283) and graduate students (n=807) combined making 70% of all requests. subject areas are also briefly examined, with ill requests coming from 63 different schools or departments across the library system. the top 15 are reported, with psychology being the top requester (n=198), followed closely by engineering & technology (n=182). the author notes that 7 of the top 15 are stem or health science disciplines.   conclusion – the rate of ill requests for oa materials shows that library users continue to struggle with information retrieval. the researcher concludes that in many cases, making an ill request is easier for the user than completing a thorough search. since staff resources are being redirected to fill user requests for materials that are readily available through open access, this use of staff time may have impacts on resource sharing and the library’s ability to fill ill requests. the author identifies benefits of using oa resources, including an increased ability of staff to fulfill ill requests, especially when providing grey literature, theses and dissertations, and conference papers and reports. another identified benefit was the decreased turnaround time for securing materials, with immediate availability via oa saving 1.15 days to deliver materials to the user. finally, the library estimates cost savings of over $27,000 (usd), based on estimated traditional per unit ill costs.   commentary   this research reinforces the need for additional user training around locating oa materials. in particular, the results reveal that information literacy education that includes the use of open access materials might be of particular use to undergraduate or graduate students, especially since the majority of oa ill requests identified in this study came from those user groups. keane (2012) revealed that while approximately 70% of surveyed academic librarians believe it is important to educate students about oa, nearly 60% of respondents indicated their institutions “do not do any promotion, much less education” around oa (p. 348). more recently, allen and weber (2014) identified the need for instruction librarians to teach open access searching just as they teach traditional library search sources. the researcher does indicate that ill practitioners may have a role to play in enhancing instruction efforts that improve user skills for finding oa materials.   much of this report is dedicated to explaining operational considerations and workflow processes aligned for the library setting. this information might be significant for understanding project background, but does not contribute much to the overall discussion of the research. moving beyond these sections reveals insights that make the study useful. using glynn’s (2006) critical appraisal tool to test for validity, the use of a large, localized dataset is appropriate for this study’s outcomes, as are the inclusion criteria and overall methods used. it is worth noting that the data collection relies on the library’s vendor products and applications alongside locally customized ill routines, potentially limiting the opportunity for other institutions to conduct similar research using their own ill request data, at least not without some modification to the methods. the study is strengthened by the author’s previous research, allowing for the use of a previously tested method and for an informed comparative analysis.   this research has implications for understanding and enhancing a library’s ill user services and overall customer service goals. filling requests for oa materials using ill staff may seem a misallocation of resources, but the opportunity to assist users with their information needs, and in a timely manner compared to traditional ill, has a direct benefit for the user and indirectly may benefit the library’s image as being helpful and user-focused. until such time as users are knowledgeable about finding and accessing oa materials, to the point where ill requests for oa sources significantly decline, individual libraries can use this research to help determine whether the benefits to end-users outweigh the potential local resource impacts. there is opportunity for future and ongoing research to understand trends and developments regarding oa material ill requests, user satisfaction, and budget and resource sharing, as well as room for research on improving user oa search skills and the resulting impact on ill and document delivery services.   references   allen, e. j., & weber, r. k. (2014). the library and the web: graduate students' selection of open access journals for empirical literature searches. journal of web librarianship, 8(3), 243-262. http://dx.doi.org/10.1080/19322909.2014.927745   baich, t. (2012). opening interlibrary loan to open access. interlending & document supply, 40(1), 55-60. http://dx.doi.org/10.1108/02641611211214305   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154   keane, e. p. (2012). librarian viewpoints on teaching open access publishing principles to college students. serials librarian, 63(3-4), 333-349. http://dx.doi.org/10.1080/0361526x.2012.724377 microsoft word es 1355_lewis evidence based library and information practice 2008, 3:2 38 evidence based library and information practice evidence summary e-book discovery and use behaviour is complex a review of: rowland, ian, david nicholas, hamid r. jamali, and paul huntington. “what do faculty and students really think about e-books?” aslib proceedings: new information perspectives; 59.6 (2007): 489-511. reviewed by: suzanne lewis manager, gosford and wyong hospital libraries northern sydney central coast health gosford, nsw, australia email: slewis@nsccahs.health.nsw.gov.au received: 31 december 2007 accepted: 19 march 2008 © 2008 lewis. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – to assess academic users’ awareness, perceptions and levels of use of e-books. also to discover the purposes for which e-books were used and identify the most effective library marketing strategies for e-books. design – survey. setting – university college london (ucl). subjects – 1,818 ucl staff and students. methods – in november 2006, staff and students of ucl were asked to participate in an online survey, administered using surveymonkey software. the survey ran november 1-18, 2006. survey results were analysed using software package for social sciences (spss). main results – the response rate to the survey was at least 6.7%. a total of 1,818 completed surveys were received from approximately 27,000 potential respondents, although it is not known whether all e-mails announcing the survey were successfully delivered. no statistically significant differences were found between the demographic profile of the survey sample and the profile of the total ucl population. data regarding e-book usage were collected from the sub-group of respondents who were existing e-book users, and data regarding use of print collections and book evidence based library and information practice 2008, 3:2 39 discovery were collected from all respondents. forty-four per cent of respondents had used e-books, with age a good predictor of usage. however additional data analysis revealed complex demographic interactions underlying e-book usage, making broad generalisations too simplistic. of existing ebook users, 61% sourced e-books independently of the ucl library. deeper analysis showed that men were more “library independent” than women and doctoral students were more so than other students and staff. forty-eight per cent of existing e-book users preferred reading from a screen rather than paper, with men more likely to read from a screen than women, and undergraduates more likely to do so than other groups. responses to questions about the purpose of reading showed that existing e-book users consulted e-books primarily for work and study, and tended to obtain these from libraries. they were less likely to use e-books for leisure, but if they did so, were likely to obtain them from nonlibrary sources. e-books were compared to traditional print across a range of factors and scored very favourably for ease of copying, currency, space requirements, 24/7 accessibility, convenience and ease of navigation. however e-books scored poorly compared to print for ease of reading, ease of marking a place and ease of annotation. regarding use of library print titles, data from all respondents indicated that women (42%) were more likely to be regular users of print than men (35%). print book discovery behaviour is complex, and age, gender and subject area all influenced book discovery preferences. analysis of data regarding satisfaction with ucl’s current provision of print library books showed that 41% rated this service as “excellent” or “good,” but further analysis by gender, age and subject area revealed pockets of low satisfaction which warrant further attention. students were much more aware of e-book availability through the ucl library than academic and research staff, with differences in awareness also displayed between different faculties. the library’s web site and catalogue were the main channels for e-book awareness, with respondents themselves suggesting the library web site and e-mail user guides as the most effective e-book awareness mechanisms. conclusion – this study reveals a significant level of interest in and use of ebooks in one academic community, but with differences determined by age, gender, academic sub-group and subject area. it builds on the findings of previous studies of e-book usage and indicates key areas for further study. these include whether reallife information behaviour correlates with the self-reporting of respondents, and the intersection of gender and self-reported information behaviour. this information, plus the patterns of book discovery behaviour emerging from this study, will be of interest to publishers, booksellers and libraries. commentary the survey reported in this article is part of the larger superbook project at the centre for information behaviour and the evaluation of research (ciber) at ucl. this project aims to apply a range of research methodologies to e-book user behaviour and integration of e-books with elearning. wiley and emerald, two major ebook providers, are funding the project so it would have been appropriate for the authors to provide a statement outlining how conflict of interest will be avoided. approximately 3,000 e-texts have been made available to the ucl community and, evidence based library and information practice 2008, 3:2 40 again, it would have been useful for the authors to explain how these texts were selected and by whom – the publishers or academics or both. it is possible that e-text and/or publisher selection could influence how the ucl community discovers and uses e-books. user behaviour will be examined using deep log analysis and interviews or focus groups. before this happens, however, the survey reported in this paper was carried out to provide baseline contextual data regarding the ucl academic community’s awareness and usage of e-books. an excellent literature review places this article in context with other studies of ebook usage. this article builds on the results of levine-clark, whose study of e-book usage at the university of denver has been the subject of an earlier evidence summary in this journal (see hannigan). generally, results are reported logically and clearly, although there may be too much detail for some readers. there appears to be one error in the reporting of the results. regarding reading format preferences, the authors note that the youngest group of respondents prefer reading from the screen “but the overall pattern appears not to be very agedependent, except for a marked fall off (on the basis of a small population) after the age of 65” (rowland 497). however the accompanying graph (figure 7) shows the opposite trend. this may simply be an accidental reversal of information in figure 7. there is a wealth of detail in this report and some interesting trends are identified regarding the interaction of age, gender, academic sub-group and subject area with ebook use, library use and satisfaction with current library services. potentially the most valuable information derived from the data is the identification of predictors of e-book use, and “hotspots” of dissatisfaction with current library provision of printed books. however, while this information will be of great use to the ucl library management, readers cannot apply it to their local populations with confidence. as the authors themselves note, some of the results may be locally determined by e-book availability rather than by characteristics of the study population itself that could be extrapolated to other academic communities. the analysis of book discovery behaviour and preferences will, however, be of interest to many readers. complex analysis of responses to the question of how dependent readers were on a range of formal and informal strategies for book discovery revealed a hierarchical classification of three clusters of strategies. the first cluster consists of formal systems of literature control outside the academic library such as other libraries, publishers’ catalogues and book reviews; the second cluster comprises informal, personal activities including searching web sites such as amazon and google and visiting bookshops; the third cluster of activities (reading lists, ucl library, ucl catalogue, recommendations) is focused on the academic institution. these clusters are further modified by significant demographic differences. as the authors note: “service planning and delivery might well benefit from a better understanding of how people find books and … librarians might do well to segment their offerings in a much more sophisticated way” (rowland 504). the survey was conducted online, but a list of the survey questions and response options would have been a useful appendix to this article. some of the response options appear imprecise, particularly for a selfreported user survey such as this. for example, regarding current use of library print collections, response options such as “use regularly,”“use as required” and “rarely use” are open to interpretation – is regular use once a day or once a week? it evidence based library and information practice 2008, 3:2 41 will be interesting to see the comparison between the self-reported results elicited by this survey and the “real-life” behaviour revealed by deep log analysis, the next stage of the superbook project. works cited armstrong, chris, ray lonsdale, and david nicholas. ciber superbook project. 24 oct. 2007. university college london. 27 march. 2008 <http://www.ucl.ac.uk/slais/research /ciber/superbook/>. hannigan, gale g. “users’ awareness of electronic books is limited.” evidence based library and information practice 2.2 (2007): 104106. 27 march 2008 <http://ejournals.library.ualberta.ca/ index.php/eblip/article/view/235> rowland, ian, david nicholas, hamid r. jamali, and paul huntington. “what do faculty and students really think about e-books?” aslib proceedings: new information perspectives 59.6 (2007): 489-511. evidence based library and information practice 2012, 7.1 129 evidence based library and information practice ebl 101 research methods: focus groups virginia wilson liaison librarian, murray library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca received: 31 jan. 2012 accepted: 4 feb. 2012 2012 wilson. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. conducting focus groups is a qualitative research method that allows researchers to collect a large amount of data from a substantial group of people in a relatively short amount of time. focus groups explore how people perceive, feel about, or view a certain service, product, topic, etc. this method gained popularity in marketing and market research in the last several decades. there is an unverified anecdote that tells of the ford focus vehicle being named by an extremely bored focus group! however, focus groups need not be boring, and the use of these groups for research has become more popular in the social sciences and the health sciences (bell, 2010, p. 165). the data produced is unique as well. given the “synergistic potentials” of focus groups, they “often produce data that are seldom produced through individual interviewing and observation and that result in especially powerful interpretive insights” (kamberelis & dimitriadas, 2005, p. 903). in other words, get people talking and the insights will often go further because of the group dynamic. focus groups are more commonly conducted face to face, with participants, researchers, and focus group facilitator all together in one place. however, it is becoming more common for focus groups to be conducted virtually, via real time chat or video conferencing. although beck and manuel (2008) assert that “library and information science practitionerresearchers have not yet used electronic interviews or virtual focus groups to any significant degree,” these methods are less costly in terms of administration and data transcription (p. 94-95). it is conceivable that the benefits will outweigh the challenges of conducting virtual research into the future. now, what are the basics of conducting focus groups? the basic structure consists of a group of six to ten participants taking part in an organized but flexible conversation that will last one to two hours. a moderator facilitates the discussion, notes are taken, and the entire conversation is usually recorded for later transcription. the role of the moderator is an mailto:virginia.wilson@usask.ca� evidence based library and information practice 2012, 7.1 130 important one, and if an expert moderator is not available, training should be explored. the “moderator sets the tone of the session . . . [and] needs to strike a balance between ease and formality that encourages the free flow of information but that also requests that participants take the session seriously” (beck & manuel, 2008, p.96). of course there are issues that can arise when a group meets for a discussion. some people tend to dominate group discussion while others rarely speak. it is the moderator’s job to make sure that all viewpoints are heard. beck and manuel (2008) outline some types of focus group participants that can be challenging in a group discussion: dominant talkers, longwinded participants, the expert, the argumentative type, the shy person. it is important to think ahead of ways to deal with the various personalities one may encounter in a focus group. careful consideration should go into populating a focus group. once the topic is decided, you need to think of who would be the best people to inquire about it. other things to think about are how many participants in total will the study include? how many groups will be conducted? should each group contain a varied mix of people or be more homogeneous? beck and manuel (2008) point out that “the research problem itself primarily drives the answers to these questions through its purposes and goals” (p. 87). if you are considering conducting focus group research, there are many resources out there to get you started, including the following books: glitz, b. (1998). focus groups for libraries and librarians. new york, ny: forbes. greenbaum, t. l. (2000). moderating focus groups: a practical guide for group facilitation. thousand oaks, ca: sage. liamputtong, p. (2011). focus group methodology: principles and practices. los angeles: sage. walden, g. r. (2008). focus groups: a selective annotated bibliography. lanham, md: scarecrow press. additionally, it’s a good idea to review research articles that have used focus groups as a research method: carlock, d. m., & perry, a. m. (2008). exploring faculty experiences with ebooks: a focus group. library hi tech, 26(2), 244-254. doi:10.1108/07378830810880342 courtois, m. p., & turtle, e. c. (2008). using faculty focus groups to launch a scholarly communication program. oclc systems & services: international digital library perspectives, 24(3), 160166. doi:10.1108/10650750810898192 fagerheim, b. a., & weingart, s. j. (2005). using focus groups to assess student needs. library review, 54(9), 524-530. doi:10.1108/00242530510629542 macmillan, d., mckee, s., & sadler, s. (2007). getting everyone on the same page: a staff focus group study for library web site redesign. reference services review, 35(3), 425-433. doi:10.1108/00907320710774292 waters, m. (1996). a children's focus group discussion in a public library: part one. public library quarterly, 15(2), 5-6. focus groups can generate quick and plentiful data for a research project. their benefits include a relatively low cost, highly detailed data, synergistic results from group participation, and flexibility. some drawbacks of focus groups include problems with dominant personalities, potential scheduling hassles, the possibility of group think (comes from the desire for harmony within a group which is trying to minimize potential conflict), and outcomes that are highly dependent on the moderator (adapted from beck & manuel, 2008, p. 79). however, when you want to discover how a group of people perceives a particular service or topic, a focus group can http://dx.doi.org/10.1108/10650750810898192� http://dx.doi.org/10.1108/00242530510629542� evidence based library and information practice 2012, 7.1 131 be the way to go. next time, we will take a look at a method that is related to focus groups – the interview. references beck, s. e., & manuel, k. (2008). practical research methods for librarians and information professionals. new york, ny: neal-schuman. bell, j. (2010). doing your research project: a guide for first-time researchers in education, health and social sciences. new york, ny: open university. kamberelis, g., & dimitriadas, g. (2005). focus groups: strategic articulations of pedagogy, politics, and inquiry. in n. k. denzin & y. s. lincoln (eds.), the sage handbook of qualitative research, (3rd ed.) (pp. 887-907). thousand oaks, ca: sage. / evidence based library and information practice evidence based library and information practice 2012, 7.2 1 evidence based library and information practice editorial learning from others about research evidence alison brettle editor-in-chief senior lecturer, school of nursing midwifery and social work university of salford salford, united kingdom email: a.brettle@salford.ac.uk 2012 brettle. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. welcome to the june issue of eblip, our first to be published with an html version as well as pdfs for each article. i hope you enjoy and find the alternative formats useful. as usual the issue comprises an interesting range of evidence summaries and articles that i hope you will find useful in applying evidence to your practice. when considering evidence, two recent trips to edinburgh got me thinking about the wide range of study designs or methods that are useful for generating evidence, and also how we can learn about their use from other professions. the first trip was as part of the cadre of the lis dream project (http://lisresearch.org/dreamproject/). dream has been set up by the lis research coalition to develop a sustainable lis research network in the uk. as part of this, a series of workshops aims to introduce lis practitioners to a wider range of research methods, thus expanding the methods used in lis research. indeed, a quick scan of the contents of this issue show a preponderance of surveys, interviews, and citation analysis, suggesting that broadening our knowledge of methods may well be a useful idea. the workshops are highly interactive and, at each session experts from outside the lis discipline introduce particular research methods and outline how they could be used in lis applications. as a result, i can see the value and understand when to use research methods such as social network analysis, horizon scanning, ethnography, discourse analysis, and repertory grids – as well as knowing that data mining is something i’m likely to avoid! so far i’ve shared my new knowledge with a phd student who was considering her methodology and incorporated my new knowledge of horizon scanning into a bid for research funding. the next (and more exciting) step is to think of a situation where i can apply one of these methods to examining an aspect of lis practice. the second trip was the british association of counselling and psychotherapy research conference, an event which i've attended for the last few years (don’t ask!). each time, i've mailto:a.brettle@salford.ac.uk evidence based library and information practice 2012, 7.2 2 been struck by both the similarities and differences between counselling and lis research in the uk. counselling research is conducted by a relatively small number of individuals and, as in lis, the vast majority of practitioners don’t engage in writing any research up for publication (clapton, 2010). particular types of research dominate in counselling, but most are highly qualitative in manner, e.g. using biographical approaches. i can’t immediately see how these could become widely used in lis, but i do find it fascinating to hear about different approaches and the evidence this provides. like many of the things that lis professionals do, counselling and psychotherapy is a complex intervention and it is not always immediately apparent what has caused an effect. it may well be that the counsellor is only one of a number of elements that has led to a positive outcome or a change in effect. this makes it difficult to generate evidence about the effectiveness of counselling, similar, for example, to trying to generate evidence regarding the effectiveness of information literacy. due to political drivers there is an increasing interest (and resistance) to a more evidence based approach in counselling and psychotherapy. one of the main areas of resistance towards evidence based practice (ebp) in counselling is that the medical model or paradigm of ebp and the view that the randomized controlled trial (rct) is the method of choice for providing high quality evidence on the effectiveness of services doesn't fit with the way counsellors provide services to their clients. each client is seen as an individual and therapy is provided according to a client’s particular needs at that time rather than following a set manual or course. this makes it impossible to assess in a "randomized controlled" manner, before even beginning to worry about the ethical and practical implications of conducting an experimental study. the unsuitability of the rct has also been raised regarding generating evidence for eblip (e.g. banks, 2008); however, “best evidence” doesn’t need to be an rct. the definition of eblip provided by booth (2006) mentions best quality evidence (generated from research, among other elements, but makes no mention of particular research designs). in addition, both eldredge (2004) and crumley and koufogiannakis (2002) have argued for the consideration of a wide type of study designs as evidence within eblip, a viewpoint with which i have long agreed. after all, it is much more important to choose a design that is suitable to answer the question at hand and provide good quality evidence, rather than trying to use a "good quality" design at the expense of finding relevant evidence. bearing that in mind, i'm racking my brains to think of how i can use webometrics and techniques from history to investigate my practice. at the same time, i urge you to think widely about research evidence and try exploring some different methodologies and see what evidence they can reveal. references banks, m.a. (2008). friendly skepticism about evidence based library and information practice, evidence based library and information practice, 3(3): 86-90. booth, a. (2006). counting what counts: performance measurement and evidence-based practice. performance measurement and metrics, 7(2), 63-74. clapton, j. (2010). library and information science practitioners writing for publication: motivations, barriers and supports. library and information research, 34(106): 7-21. crumley, e., & koufogiannakis, d. (2002). developing evidence-based librarianship: practical steps for implementation. health information & libraries journal, 19(2), 61. eldredge, j. d. (2004). inventory of research methods for librarianship and informatics. journal of the medical library association, 92(1), 83-90. evidence based library and information practice 2012, 7.2 3 note: videos and material from the dream events is available from http://lisresearch.org/dream-project/dreamworkshops/ http://lisresearch.org/dream-project/dream-workshops/ http://lisresearch.org/dream-project/dream-workshops/ feature editorial   the 8th international evidence based library and information practice conference (eblip8): special issue and reflections   helen partridge pro vice-chancellor, scholarly information and learning services executive director, australian digital futures institute university of southern queensland toowoomba, queensland, australia email: helen.partridge@usq.edu.au      2015 partridge. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   the 8th international evidence based library and information practice conference (eblip8) was held in brisbane, australia on july 6-8, 2015. the conference theme “evidence and practice: working together” reflected our focus on narrowing the gap between practice and evidence, and in promoting evidence based practice in all parts of the information profession.   evidence based library and information practice (eblip), once based on the medical paradigm, is changing and evolving. we are developing practices and processes particular to the library and information profession and reconfiguring the traditional evidence-based philosophy to suit individual circumstances and locations. collaboration across institutions and sectors is fundamental to learning through sharing experiences, publication of quality research results, and a sustainable and connected eblip community.   over 175 delegates attended eblip8 from 12 countries: australia, new zealand, canada, doha, hong kong, malaysia, united states of america, norway, saudi arabia, sweden, united kingdom and cameroon. the conference provided a rich and diverse array of experiences; it included 3 keynote addresses, 27 paper presentations, 17 posters, 1 panel session, and 6 workshops. topics were many and varied, with evidence based practice being critically explored in the context of big data, decision making, information literacy, leadership, open access, and website design. the full conference program is available at http://eblip8.info/program. the conference is an excellent opportunity to network, and eblip8 organizers ensured there were plenty of opportunities for delegates to meet colleagues from around the world with a welcome reception, conference dinner, and a farewell reception.   this special issue begins with reflections by suzanne lewis and virginia wilson on the challenges and opportunities in being a solo practitioner of evidence based library and information practice. they remind us that the possibilities are endless. eight of the papers that were presented at the conference follow this editorial. the papers present eight perspectives on evidence based practice and its application within the library and information profession.   sarah hayman opens the discussion by acknowledging the important role that effective literature searching has in supporting evidence based practice. the paper outlines the development and evaluation of smart searching, a free, self-paced online resource to help librarians to apply an evidence based approach to their own searching practice. mary somerville and niki chatzipanagiotou note that contemporary organisations must develop workplace environments that enable nimble decision-making and action taking provide a critical discussion of informed systems, and have contributed a theoretically grounded approach to support evidence based library and information practice. matthew kelly describes the development of a methodology for subject based collection analysis for public libraries. he suggests that the methodology has the potential to improve the practice of collection development by linking subjective decision making with a scientifically based approach to managing knowledge resources.   claudia davies and richard vankoningsveld outline how evidence based practice has been used at legal aid queensland to make decisions around the future training needs and activities of library clients. their training needs analysis involving a blend of qualitative and quantitative evidence enabled the refinement of training activities to more closely align with client needs and organisational objectives. anna hare and wendy abbott explore the models of indigenous support programs in australian academic libraries and how they align with the needs of the students they support. evidence gathered through focus groups and an online questionnaire was used to establish empirically informed recommendations on how academic libraries could improve indigenous student support. sandy rae and margaret hunn discuss the design of research and writing for business students, an online resource for first year students. the positive impact of the resource on student success was demonstrated by the gathering of evidence informed by the five senses of success framework.   gillian hallam and robyn ellard report on a study examining the skills requirements of staff across public libraries in the australian state of victoria. they used interviews, focus groups and questionnaires to collect evidence on the current skills, and to anticipate future requirements, for the public library workforce. they conclude that research should stimulate debate between practitioners, educators, and professional associations about the future direction of library and information education with the goal of ensuring a strong future for the sector. ellen forsyth, cameron morley, and kate o’grady discuss how the state library of new south wales established the public library network research committee in 2002. recognising the lack of research about public libraries in new south wales, the committee had input into identifying, recommending, and implementing research projects that will help advance public libraries. the authors discuss the committee’s approach of using evidence to inform practice for the long-term benefit to public libraries.   the special issue concludes with commentaries and reflections by four conference delegates. a big thank you to ruby warren, megan fitzgibbons, kristie jones, and brenda strachan for sharing their invaluable insights.   an international conference relies on many people contributing their time and energy. eblip8 could not have happened without the members of the local organising committee and the international program committee, as well as the student volunteers and session chairs.   we also warmly thank our sponsors who helped make this event possible: university of saskatchewan’s centre for evidence based library and information practice; charles sturt university’s school of information studies; queensland university of technology’s information studies group; state library of new south wales; university of southern queensland; state library of queensland; queensland university of technology library services; university of queensland library services; and elsevier.   the call for expressions of interest to host eblip9 went out recently, with the location to be announced early 2016. i hope to see you at eblip9!   evidence based library and information practice 2012, 7.3 126 evidence based library and information practice news/announcements planning is well underway for the 7th international evidence based library and information practice conference (eblip7) to be held in saskatoon, saskatchewan, canada – july 15-18, 2013 2012. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. it’s hard to believe that it’s been well over a year since eblip6 in salford, but eblip7 is fast approaching. the conference theme is “the possibilities are endless…” and it is a theme which we hope will evoke not only the wide open spaces of saskatchewan on the canadian prairies, but also the notion that in evidence based library and information practice, the possibilities are indeed endless when it comes to exploring and implementing the best available evidence into a practice situation. as well, the possibilities are endless when it comes to delving into practice-based research in order to inform and enable best practices. these are exciting times in eblip, and you are invited to the gathering of the eblip community and to join in the ever-changing and always interesting conversations. in the weeks ahead, you will be invited to submit abstracts for conference sessions which will be peer reviewed by the international program committee. a call for posters will also be initiated. there will be details of continuing education programs which will be held on monday, july 15, 2013. conference registration will open in the new year. very soon, a functional eblip7 website will be unveiled and the opening and closing keynote speakers will be announced. as well, information about the opening reception, conference banquet, conference accommodations, and other pertinent details will be posted to the website. there are lots of ways to keep abreast of elbip7 news:  bookmark http://eblip7.library.usask.ca/  follow us on facebook (http://www.facebook.com/groups/ebli p7/ ) and twitter (@eblip7)  always check for news and information in the evidence based library and information practice journal  watch for announcement on various listservs if you have any questions, email us at eblip7@library.usask.ca. http://eblip7.library.usask.ca/ http://www.facebook.com/groups/eblip7/ http://www.facebook.com/groups/eblip7/ mailto:eblip7@library.usask.ca evidence based library and information practice 2012, 7.2 84 evidence based library and information practice evidence summary perceived convenience, compatibility, and media richness contribute significantly to dedicated e-book reader acceptance a review of: lai, j.-y., & chang, c.-y. (2011). user attitudes toward dedicated e-book readers for reading: the effects of convenience, compatibility and media richness. online information review, 35(4), 558580. reviewed by: theresa s. arndt associate director for library resources & administration waidner-spahr library, dickinson college carlisle, pennsylvania, united states of america email: arndtt@dickinson.edu received: 1 mar. 2012 accepted: 16 may 2012 2012 kelly. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/),which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – investigates the effects of perceived convenience, compatibility and media richness on users’ attitudes toward dedicated e-book readers. design – convenience sample survey. setting – taiwanese university. subjects – a total of 288 students at the senior secondary (5%), four-year university (78%), and graduate student (17%) levels. malefemale participation was approximately equal. methods – students completed a 23-item survey on dedicated e-book readers, with questions on perceived usefulness, perceived ease of use, intention to use, convenience, compatibility, and media richness. data was analyzed using the partial least squares statistical technique. main results – users state an increased intention to use dedicated e-book readers if they perceive the technology to be compatible with what they desire in a “book,” if the device delivers rich media content, and if the device is convenient. compatibility was found to significantly affect perceived ease of use, and was found to be the strongest influence on intent to use a dedicated e-book reader. compatibility, media richness and convenience also increased the perceived usefulness of dedicated e-book readers. mailto:arndtt@dickinson.edu http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ evidence based library and information practice 2012, 7.2 85 conclusion – users will prefer dedicated ebook readers that are compatible with their preferences in a “book,” that deliver mediarich content, and that they find convenient. the study has implications for the design and development of e-book reading devices. commentary the primary theoretical basis for this study is the information systems field’s technology acceptance model (tam) (davis, 1989). tam has been extensively used and built upon as a framework for studying what influences users to accept and use technology. additionally, the authors draw from theory in sociology, organizational management, and marketing. the survey was created by adapting questions from pre-existing published instruments, and was pre-tested on users and experts. completed surveys were returned by 326 students; 288 surveys were deemed valid for analysis. a significant limitation, acknowledged by the authors, was the use of a convenience sample creating the possibility of self-selection bias. the authors further acknowledge the limited subject selection (taiwanese university students), and recommend further research across different groups and cultures. use of the partial least squares technique for data analysis is appropriate for this type of study. construct reliability and validity were established using confirmatory factor analysis. the authors never acknowledge the critiques that have been made of the technology acceptance model which forms the foundation of their study. bagozzi (2007) has articulated problems with overreliance on tam, commenting that it has led to “the fallacy of simplicity” in the information systems field (p. 244). the authors’ paper would have been strengthened had they addressed this criticism directly. they do acknowledge that their study does not examine social and cultural factors which might influence dedicated e-book reader acceptance. careful editing would have addressed some inconsistencies in the narrative. the authors occasionally veer into speculative territory unsupported by their findings. after describing the finding that “compatibility shows greater influence on acceptance…than convenience,” the authors state that “[m]ore than compatibility, convenience may be a critical factor attracting users to e-book readers” (p. 572). the authors also make some incongruous statements based on dated references (“the e-book market has failed to materialise” p. 562) that contradict their point as well as current reality. the target audience for this study is e-book reader device developers, not librarians. the authors discuss implications for e-book reader feature development and design, including specific features they believe would enhance compatibility, media richness, and convenience. the study is indirectly relevant to librarians who may be considering loaning actual e-book devices or providing e-books for use by their patrons on their own devices. the factors examined in this study could be included in librarians’ evaluative criteria for deciding which e-books or devices may be more acceptable to patrons. however, recently published library science studies conducted from a less theoretical perspective will likely be of more direct interest to practicing librarians (duncan, 2011; shelburne, 2009). references bagozzi, r. p. (2007). the legacy of the technology acceptance model and a proposal for a paradigm shift. journal of the association for information systems, 8(4), 244–254. davis, f. d. (1989), perceived usefulness, perceived ease of use, and user acceptance of information technology. mis quarterly, 13(3), 319–340. duncan, r. (2011). ebooks and beyond: update on a survey of library users. aplis, 24(4), 182-193. evidence based library and information practice 2012, 7.2 86 shelburne, w. (2009). e-book usage in an academic library: user attitudes and behaviors. library collections, acquisitions, & technical services, 33(2/3), 59-72. microsoft word art_martina.doc evidence based library and information practice 2006, 1:1      26 evidence based library and information practice       article    employing evidence: does it have a job in vocational libraries?   cecily martina  liaison librarian, southbank institute  briabane, queensland, australia  e‐mail: cecily.e.martina@det.qld.gov.au    brad jones  liaison librarian, bremer institute of tafe  queensland, australia  e‐mail: bradley.jones@det.qld.gov.au      received: 15 december 2005    accepted: 6 march 2006      © 2006 martina and jones. this is an open access article distributed under the terms of the creative  commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted  use, distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective ‐ evidence based librarianship (ebl) springs from medical and academic  origins. as librarians are tertiary educated (only occasionally with supplementary  qualifications covering research and statistics) ebl has had an academic focus. the ebl  literature has significant content from school and university perspectives, but has had  little, if any, vocational content. this paper suggests a possible evidence based  librarianship context for vocational libraries.    methods ‐ a multidisciplinary scan of evidence based literature was undertaken,  covering medicine and allied health, librarianship, law, science and education. national  and international vocational education developments were examined. the concept and  use of evidence in vocational libraries was considered.    results ‐ library practice can generally benefit from generic empirical science  methodologies used elsewhere. different areas, however, may have different concepts  of what constitutes evidence and appropriate methodologies. libraries also need to  reflect the evidence used in their host organisations. the australian vocational librarian  has been functioning in an evidence based educational sector: national, transportable,  prescriptive, competency based and outcome driven training packages.  these require  a qualitatively different concept of evidence compared to other educational sectors as  they reflect pragmatic, economic, employability outcomes.    conclusions ‐ vocational and other librarians have been doing research but need to be  more systematic about design and analysis.  librarians need to develop ‘evidence  literacy’ as one of their professional evaluation skills.  libraries will need to utilise  evidence relevant to their host organisations to establish and maintain credibility, and  in the vocational sector this is set in a competency based framework.  competency  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:1      27  based measures are becoming increasingly relevant in school and university (including  medical) education.      librarians: generic and specialist  evidence skills?    whilst evidence based philosophies have  been with us for some time, the current  evolving debate over the nature and  practical application of evidence in  librarianship is relatively recent.      in australia, librarians are tertiary  educated and for the most part move in  generalist circles (public, vocational and  university for example).  sectors such as  health and law have a history of speciality,  and there are even areas like teacher  librarianship where the library  qualification is secondary.  some librarians  hold additional qualifications including  research components (partridge and  hallam “developing a culture of  evidence based practice”), but library  education itself has only relatively  recently increased its focus on research  methodology and statistics.     after moving into the library workforce it  is reasonable to suggest that librarians  absorb something of the culture and skill  set of their host organisations due to the  sharing of goals, collaboration with non– library colleagues and the information  with which they deal. in some cases this  involves research and statistics. this may  in part explain the emergence of interest in  evidence based librarianship (ebl) in  health and academic libraries, both of  which exist in research rich environments.    the medical fraternity is often critical of  the evidence base behind work done in its  own area, due to factors such as the  quality of the work itself, the  interpretation of evidence, or  contamination by vested economic  interests (ioannidis; smith). evidence  based medicine (ebm) itself has not been  universally or uncritically accepted in  medical & allied areas (straus and  mcalister; grahame‐smith).       there is also an argument that, contrary to  the simplified view that some areas have  failed to capitalise on more advanced  research methods, different “evidence”  may apply in different fields. one  significant difference, for example, has  been a comparison with the legal concept  of evidence ‐ particularly the challenges  posed by the daubert decision and ‘good  science’ implications (berger; wagner).  the countering of scientific argument to  serve vested interests reminds us that  evidence has its place – often behind  political and economic interests.  given  the adversarial host culture, legal  librarians may have different views of  evidence.    educators also have different perspectives  on evidence.  groundwater‐smith goes so  far as to “argue that education can lay  claim to a broader and richer  understanding of [evidence based practice]  growing out of a tradition of action  enquiry and practitioner research.” todd  raises the relevance of the host  organisation when he suggests that  evidence must demonstrate the  contribution of the library to the school’s  learning goals, through a constructivist,  inquiry based framework, at the local level.      it may also be argued that evidence varies  not only between disciplines, but within  different sectors of the same basic  discipline, such as education. outside the  school setting, partridge and hallam  (“developing a culture of evidence based  practice”) examined the university level  training of professional librarians. their  preferred educational outcome favoured  the evidence based “reflective  practitioner” rather than researcher  (except at research masters and phd  levels). they noted difficulties in keeping  courses relevant to employers’ needs,  increased curriculum congestion, ongoing  review of the curriculum at their  institution, and invite “other teaching and  evidence based library and information practice 2006, 1:1      28 learning models (to be) presented and  critically discussed.”      the partridge and hallam study did not  include the views of vocational education  and training (vet) librarians. this sector  does not have as significant a research and  publication culture as medicine and  academia. the post‐compulsory  vocational education sector reflects a  different and increasingly relevant  teaching and learning model, with a very  different perspective on what constitutes  evidence.    it is important to move beyond medicine  and academia to more accurately reflect  reality. table 1 displays the significance of  the vet sector in australia while not  having the profile of medicine and  academia, vet qualifications are held by  nearly a third of the australian working  age population, and it has a participation  rate almost twice that of the tertiary sector.  the pervasiveness of vet training seems  to give it a degree of invisibility. as the  training sector underpinning the economy,  the authors believe that this area, and its  concept of evidence, warrants attention  and also has a considerable amount to  offer.    vet librarians train at university and are  familiar with academic assessment.   however, they work in the same sector  that trains paraprofessionals according to  a very different competency based regime.   professionals and paraprofessionals,  however, often do quite similar work.   teece (quoted in partridge and hallam  “the double helix”) notes that  paraprofessionals are now “routinely  doing quite complex work formerly seen  as exclusively the province of professional  librarians” (1). carroll states “one of the   key dilemmas that has faced the library  industry…has been the degree to which  the two sectors of library employment  converge” (117). vet sector librarians  must live with the discomfort of differing  emphasis on serving goals related to  supply‐driven liberal holistic education  ends (as with their own training) and  demand‐driven pragmatic employability  outcomes (as with paraprofessionals).    while there are international variants, the  competency based approach is the  preferred vocational education option in  competitive economies. in australia, the  vet sector differs from other educational  sectors in that:   • ties with industry needs are built into  the system, as curricula are  determined and regularly reviewed by  industry (via industry skills councils);  • curricula are not developed in  individual institutions. instead, highly  prescriptive, nationally consistent  training packages are used; and,    • assessment is not on the basis of  holistic measures, but on explicitly  defined competencies using “evidence  guides.    how significant is vocational education and training?  qualifications held 2004  (15‐64 age group)    bachelor degree or above    adv diploma or below (vet)    18.9%    31.3%  participation rates  (2003)    university    vet    930,000    1.7m    (source: australian bureau of statistics)    table 1: significance of vet in australia     evidence based library and information practice 2006, 1:1      29 the authors are of the view that relevant  evidence reflects the generic / discipline  based skills debate (partridge and hallam  “the double helix”; smith and martina).  in this sense, generic scientific  methodologies, research strategies and  statistical analyses applied in areas such as  agriculture, psychology and medicine  might also be directly applied in  librarianship. many disciplines, however,  may also have their own unique  perspectives on what constitutes evidence,  and these may require different  approaches to be taken by libraries  servicing these areas.    ebl may need to consider both a  “generic” science based perspective, as  well as a “host discipline” relevancy  approach to evidence. ebl relies on the  argument that up until now librarianship  has lacked a sophisticated “generic”  science base, relying instead on lower  levels of evidence. “we often espouse the  deficiency of our usersʹ information  literacy yet we too have deficiencies and  particularly with regard to research”  (macauley).  eldredge laments “that  librarianship does not offer a better  representation of the more rigorous  methods at the higher levels of evidence”  (298).  “librarians place a great emphasis  on anecdote and experience” (brice, booth  and bexon 16), with approaches such as  case studies resulting in an  “overwhelming positive‐outcome bias”  (eldredge 297).  ebl should result in “an  increased number of research projects  conducted at the higher levels of evidence  that are capable of facilitating practical  decisions” (eldredge 298).      library practitioners outside medicine and  academia may find this empirical science  basis unfamiliar, with a resultant tendency  to fall back on a rebadging of literature  research and traditional library  approaches as evidence. this misses the  primary point of ebl: that more  sophisticated methodological and  statistical analysis of what goes on in  libraries is required.      the “discipline” based approach within a  host organisation is a different concept. in  educational libraries, evidence should  reflect the accepted evidence in education,  particularly attending to the forms of  assessment used in each sector. to this  end, school and university levels are  primarily holistic, while vet has for some  time been shifting to highly prescriptive  economic approach measures. further,  this approach has been widely accepted  globally (kearns) and aspects are moving  into other educational and professional  development arenas. this is a pivotal  development for the concept of evidence  across all educational sectors and,  consequently, for librarians in these  educational areas.    a brief history of vocational education in  australia    it may be helpful to provide a brief  historical background of vet, examine the  current vet structure, touch upon some  recent library examples, and reflect upon  the potential role of evidence in the  vocational context.    the male–trade stereotype of the  australian vet sector dominated until the  1960’s and 70s. while the stereotype  persists, the reality faded due to  international social and economic changes,  with its australian variant appearing in  the landmark 1974 kangan report, which  effectively combined the various state vet  agencies into a new national educational  sector.  “for the first time, tafe  [technical and further education] was  regarded as part of the tertiary education  sector” (employment and skills formation  council, quoted in australian education  union).    in the late 1980s, with vocational  education also receiving much attention  overseas, there was a significant shift from  liberal education goals to a solid  economically driven agenda, though again  with an australian flavour. the australian  national training authority (anta) was  established in 1992 with heavy industry  involvement. user choice saw private  evidence based library and information practice 2006, 1:1      30 businesses supplement the public tafe  system. globalisation, with international  movements of capital and skills,  encouraged pragmatic vocational  education and various countries  repositioned their training systems for this  new environment.     the late 1990s brought the national  training framework, a national approach  to qualifications through industry‐defined  competencies and the development of  training packages.  anta was abolished  and its functions subsumed by the  department of education science and  training (dest), with a new direction  outlined in dest’s skilling australia  paper.  this builds on the competencies  and training package concept, with  training policies, priorities and delivery  driven by industry and business needs, as  well as greater flexibility emphasising  employability skills and a revision of  quality control measures.    the current basis of vocational education    the vocational sector, while still retaining  liberal holistic goals relating to the  development of individual potential, has  had a significant international shift in  primary emphasis back to pragmatic  economic outcomes. mammon effectively  drives vocational education. there is little  doubt about potential conflicts of interest,  or motivation for evidence to be doctored.      educational qualifications in australia are  detailed under the australian  qualifications framework (see table 2).  this framework connects the schools, vet  and higher education sectors in a single  framework, and details qualification titles  and guidelines. while assessments at  school and university tend to the holistic,  the vet sector takes on an approach based  on ‘competence’ within a ‘training  packages’ context.  the content agenda is  driven by business and industry via  industry skills councils.  the australian training package  arrangements are detailed on the national  training information service’s website  <http://www.ntis.gov.au/>. training  packages are highly prescriptive in that  they outline all the competencies  considered relevant to a particular  industry grouping. qualifications can be  at different levels; each qualification is  made up of a package of competencies;  each competency is made up of a number  of elements; each element is assessed  according to an evidence guide       aqf table of qualifications (by sector of accreditation), march 2005    schools sector  accreditation  vocational education and  training sector accreditation  higher education sector  accreditation                    senior secondary  certificate of education        vocational graduate diploma  vocational graduate certificate    advanced diploma  diploma  certificate iv  certificate iii  certificate ii  certificate i    doctoral degree  masters degree  graduate diploma  graduate certificate  bachelor degree  associate degree, advanced diploma  diploma      table 2: the australian qualifications framework   reproduced with permission of the australian qualifications framework advisory committee    http://www.ntis.gov.au/ evidence based library and information practice 2006, 1:1      31 (acceptable observable evidence that  complies with specific auditable criteria).  thus, the educational evidence is already  defined. industry requires a particular  predetermined set of competencies to be  available.  individuals may have existing  skills acknowledged by recognition of  prior learning (rpl).  learners are  assessed as either “competent” or “not yet  competent”. they then work on the  remaining competencies until they can  demonstrate evidence of achievement. the  competencies may be sequenced. if  competencies are not achieved, efforts  concentrate on bringing individuals up to  the specified level. when the packaged  requisite set of competencies has been  evidenced, a qualification is issued.      this competency based approach has  extended to overlap with other sectors.   many students now begin vocational  studies in the post‐compulsory years of  high school, with these studies recognised  upon subsequent entry to the vet system.  at the higher end, vet has articulation  into university courses.      educational outcomes aside, the entire  sector is subject to heavily measured  outcomes, accountability and quality  assurance processes, and subject to  ongoing review. employer surveys,  specific client target groups, student  retention and completion rates, student  outcomes and earnings, and economic  input/output measures all feature. this  approach has been described as evidence  based education (mcdonald). while  similar pressures exist in most sectors, in  vet they are housed within a competency  based context.      the vet experience    within the queensland tafe sector,  southbank institute has had considerable  involvement with the massive redefinition  of skill sets expected of library  practitioners. current and recent library  staff have published or presented on a  broad range of topics that provide a  credible record, irrespective of ebl  limitations, for any institution: metadata,  the establishment of elearn centres, the  evolving collaborative relationship  between librarians and teachers in  working towards national vet objectives,  digital repositories, information literacy,  digital video and free / budget practical  software.    of particular note is a paper by smith and  martina, which targeted the ‘information  literacy’ needs of bakers. surveys and  other evidence were collected from  teachers, business and industry. the  correlation between vets prescriptive  evidence based key competencies,  information literacy principles and  industry sourced “employability skills”,  showed the interdependence between  them and it was concluded that  information literacy skills in the  workplace are essential. this preliminary  study recommended that information  literacy should be embedded in training  packages, and that librarians be involved  in the training package review process.  thus, “by using methods familiar to our  colleagues outside librarianship, librarians  can open doors to future multidisciplinary  collaboration” (eldredge 292).    does evidence have a role in vocational  libraries?      the question is, of course, rhetorical, as  evidence has several roles in vocational  libraries. in the “generic” empirical  science sense, all libraries face issues that  can benefit from tighter methodologies  than have been adopted in the past. some  evidence may come from other disciplines:  information literacy instruction, for  example, could draw upon work done  previously in the broader field of  education. other aspects are undoubtedly  unique to libraries, including some  specific to vocational libraries, and require  original research.      libraries have faced considerable  pressures and changes for quite some time,  and this situation shows no sign of abating.  to apply higher level scientific  experimental designs and statistical  evidence based library and information practice 2006, 1:1      32 analyses would require the acceptance of  yet another new perspective and skill set.  at present there are few signs that this can  be achieved within the bounds of  everyday librarianship. but these are early  days, and efforts may be made toward  raising awareness and developing such an  “evidence literacy” skill set. there are  dangers in yet another brief workplace  training session, but in the short term this  may well be the most likely option to  increase general understanding. it is  unlikely that there will be an employment  market for specialist, scientifically trained,  librarians; and while there has been  increased attention on ebl content  recently, the development of more  structured training electives and research  could be further encouraged. another  option is to collaborate with research  experts. ultimately, the application of  more evidence based methods must have  an impact beyond an increasingly  sophisticated audience of professional  librarians.    turning to the second major aspect of  evidence in vocational (and other) libraries,  their ultimate credibility and possibly  survival may depend upon adopting the  educational variants of evidence used by  the libraries’ respective host institutions.  the evidence used in australian  vocational institutions is different to that  adopted in school and university  education. at present, and for the  foreseeable future, evidence is based on  the concept of competencies.  vocational  librarians need to be able to dissect the  training packages, identify those  competencies where they may be able to  partly or wholly make a contribution, and  examine the evidence guides for  measuring these competencies. thereafter,  librarians will need to work these into an  analytical, methodological and statistical  framework in order to produce usable,  relevant evidence. they might also be on  the alert for gaps in competencies and  employability skills, and lobby for  inclusions or variation in these areas.    while educational evidence may relate to  competencies, other evidence is also of  critical relevance in vet, though it  remains set in a competency based,  pragmatic economic outcomes framework.  these are the organisational performance  evaluations such as employer surveys,  retention and completion rates, and  economic cost /benefit measures.  following from mcdonald, in 2003 dawe  (8) made specific reference to the influence  of ebm, and provided advance notice of a  national council for vocational education  research (ncver) centralised resource  for vet systematic reviews. in 2005,  thomson, anlezark, dawe and hayman  advised that they had undertaken “the  first systematic review conducted in a  vocational education and training …  context in australia” (9), and anlezark,  dawe and hayman concurrently  developed “a replicable framework and  infrastructure for further systematic  reviews of research” (9). as a starting  point, they utilised the cochrane derived  models of the campbell collaboration,  evidence for policy and practice  information and co‐ordinating centre  (eppi‐centre) and the learning and skills  development agency.      a search of the aforementioned sites  reveals that librarians are largely not “in  evidence”. as the emphasis is primarily  on higher level evidence, fields with  approaches considered to be lower forms  of evidence tend to be overlooked. hence,  there is little library related content, and it  appears much of the literature remains  within library circles. the voced  database contains library and related  content, but minimal material on  conceptual frameworks for library  evidence in the evolving vocational  context, what it is, or appropriate  methodologies for research. ebl needs a  higher profile, with contributions related  to the host organisation, if libraries are  going to be appropriately recognised in  the increasingly accountable, higher level  evidence, measured, vet environment.     conclusion    in this paper the authors set out to raise  the profile of vocational libraries in an  evidence based library and information practice 2006, 1:1      33 evidence based framework, currently  overshadowed by medical and tertiary  education precedent. vocational libraries  can undoubtedly benefit from a greater  ebl emphasis. however, there are  benefits in both directions as the  vocational sector has an educational  variant of evidence that may be applicable  to other sectors. university education does  have vocational content, and medicine is,  after all, another vocation.    there is, of course, substantial literature  on medical education with the usual  theoretical / practical spectrum. in 1999,  the best evidence in medical education  collaboration (beme) was formed, using  the cochrane and campbell collaboration  format, though it noted that the idea of eb  medical education had been hampered by  antipathy towards ebm, and that there  were few good models to go on from  general and higher education.    in britain, in august 2005, the department  of health raised some now familiar  themes. using the same basic argument as  ebm ‐ to ensure “optimum patient care  and safety” and “quality of medical care” ‐  and using evidence in the form of  “standards … set … by senior healthcare  professionals”, “doctors who are starting  their first year after medical school … will  have to demonstrate explicitly that they  are competent in a number of areas  including communication and  consultation skills, patient safety and  teamwork, as well as the more traditional  skills”. this “programme based on the  achievement of competence, rather than  time served” … “is just the beginning of a  much wider ranging change in medical  training”.    thus, even hippocrates may be  confronting, at least in part, the  competency based / employability skills  approach that has been the framework of  the vocational sector for some time.      partridge and hallam (“developing a  culture of evidence based practice”) also  note the issue of keeping university  courses relevant to employers’ needs in  their call for other teaching and learning  models to be presented and critically  discussed.  at bonn in september 2005,  unesco‐unevoc held a seminar,  involving the european centre for higher  education and the international centre for  technical and vocational education and  training, which focused on the complex  need for growing vocational content in  higher education. issues such as evidence  and increased vocational content in  schools and higher education, together  with competencies, articulation, credit  transfer, and lifelong learning issues in the  knowledge based economy, were  significant at this forum.    rather than being overshadowed by  medicine and university education, it  seems that the evidence that has been  driving vocational education for some  time is now spreading into increasing  vocational prominence in school and  university courses, including medicine.  accordingly, the authors draw attention to  the competency based evidence model.      without a strong and pervasive research  culture, librarians in vet have yet to  properly embed library measures in a  competency based and performance  measured evidential framework. as the  education systems further reconcile the  conflict between liberal holistic measures  at one end of the spectrum, and pragmatic  employability goals at the other, the  educational evidence may well be couched  in terms and concepts related to  competencies. of similar relevance are the  performance evaluation measures used by  vocational institutions. libraries will need  to employ evidence in such a framework  to prove themselves relevant to their host  organisations and beyond.    acknowledgement    a version of this paper was presented at  the 3rd international evidence based  librarianship conference, held in brisbane,  australia (october 2005). see:   <http://conferences.alia.org.au/ebl2005/ma rtina.pdf>.    http://conferences.alia.org.au/ebl2005/ma evidence based library and information practice 2006, 1:1      34 works cited    anlezark, alison, susan dawe and sarah  hayman. an aid to systematic  reviews of research in vocational  education and training in australia.   adelaide, national centre for  vocational education research, 2005.  voced database. 12 december 2005  <http://www.ncver.edu.au/research/co mmercial/op320a.pdf >.    australian education union, tafe  division. policy on the role of tafe in  education. (2001). 6 july 2005  <http://www.aeufederal.org.au/tafe/r oleoftafepolicy.pdf>.    australian qualifications framework.  2005. 12 aug. 2005   <http://www.aqf.edu.au/aqfqual.htm>.    berger, margaret a.  “what has a decade  of daubert wrought?” american  journal of public health: supplement  on scientific evidence and public  policy 95.s1 (july 2005): s59‐s65. 12  aug. 2005  <http://www.defendingscience.org/loa der.cfm?url=/commonspot/security/ge tfile.cfm&pageid=2407> .        best evidence in medical education  collaboration. 2005. 1 dec. 2005  <http://www.bemecollaboration.org/>.    brice, anne, andrew booth, and nicola  bexon. “evidence‐based librarianship:  a case study in the social sciences.”  libraries‐a voyage of discovery.  world library and information  congress: 71th ifla general  conference and council.  oslo,  norway. 14‐18 august, 2005. 10  august 2005  <http://www.ifla.org/iv/ifla71/papers/ 111e_brice_booth_bexon.pdf>.    campbell collaboration (c2). 2005. 12 dec.  2005  <http://www.campbellcollaboration.or g/ >.    carroll, mary. “the well‐worn path.”  australian library journal 51.2 (2002):  117‐25.  7 july 2005  <http://www.alia.org.au/publishing/alj /51.2/full.text/path.html>.    dawe, susan. “basing policy and practice  on sound evidence.” australasian  evaluation society international  conference. auckland, new zealand.  18 sept. 2003. national centre for  vocational education research.   voced database.10 may 2005  <http://www.ncver.edu.au/pubs/confs /cp03conf.pdf>.    defendingscience.org. american journal  of public health: supplement on  scientific evidence and public policy  95.s1 (july 2005). 12 aug. 2005  <http://www.defendingscience.org/a mjpublichealth‐supplement.cfm>.    department of education, science and  training. skilling australia: new  directions for vocational education  and training. canberra:  commonwealth of australia, 2005. 6  july 2005  <http://www.dest.gov.au/nr/rdonlyre s/3e9602e2‐c6da‐431c‐bf65‐ e053006ac4db/2236/skilling_australi a.pdf>.    department of health. “major shake‐up in  medical training”. government news  network. 9 aug 2005. 10 aug. 2005  <http://www.gnn.gov.uk/content/det ail.asp?releaseid=166188&newsareai d=2>    eldredge, jonathan d. “evidence‐based  librarianship: an overview.” bulletin  of the medical library association  88.4 (2000): 289‐302.    evidence for policy and practice  information and co‐ordinating centre  (eppi‐centre). 2005. 12 dec.2005  <http://eppi.ioe.ac.uk/eppiweb/home. aspx >.        http://www.ncver.edu.au/research/co http://www.aeufederal.org.au/tafe/r http://www.aqf.edu.au/aqfqual.htm http://www.defendingscience.org/loa http://www.bemecollaboration.org/ http://www.ifla.org/iv/ifla71/papers/ http://www.campbellcollaboration.or http://www.alia.org.au/publishing/alj http://www.ncver.edu.au/pubs/confs http://www.defendingscience.org/a http://www.dest.gov.au/nr/rdonlyre http://www.gnn.gov.uk/content/det http://eppi.ioe.ac.uk/eppiweb/home evidence based library and information practice 2006, 1:1      35 grahame‐smith, david. “evidence based  medicine: socratic dissent.” british  medical journal 310 (1995): 1126‐27. 8  july 2005  <http://bmj.bmjjournals.com/cgi/conte nt/full/310/6987/1126>.    groundwater‐smith, susan. “evidence  based practice – towards whole school  improvement.” annual conference  australian association for research in  education.  sydney. 4‐7 dec 2000. 7  july 2005  <http://www.aare.edu.au/00pap/gro00 303.htm>.    ioannidis, john p.a. “why most published  research findings are false.”  plos  medicine (pre‐issue), 2.8: e124 (2005):  1‐6.  20 aug. 2005  <http://www.pubmedcentral.nih.gov/ articlerender.fcgi?tool=pubmed&pub medid=16060722>.    kearns, peter. vet in the 21st century   global knowledge economy: an  overview of international  developments in vocational education  and training. kambah, act: peter  kearns and associates, 2004. 12 aug.  2005  <http://www.dest.gov.au/nr/rdonlyre s/2e5924cc‐3463‐4956‐9d10‐ 3cfeeabe3026/6342/vet_global_kn owledge_economy_peter_kearns.rtf>.    learning and skills development agency  (lsda). 2005. 12 dec. 2005  <http://www.lsda.org.uk/home.asp>.    macauley, peter. “the doctoring of  evidence based librarianship”.  evolution of evidence: global  perspectives on linking evidence with  practice. 3rd international evidence  based librarianship conference, 16‐19  oct 2005. brisbane, queensland. 12  aug. 2005  <http://conferences.alia.org.au/ebl2005 /keynoteabstracts.html#macauley>.    mcdonald, rod. towards evidence‐based  vocational education and training.   adelaide, national centre for  vocational education research, 1999.  voced database. 10 may 2005  <http://www.ncver.edu.au/research/pr oj/nr9002.pdf>.    national council for vocational education  research. 2005. 12 dec. 2005  <http://www.ncver.edu.au/>.    national training information service.  2005. 12 aug. 2005  <http://www.ntis.gov.au/>.    partridge, helen and gillian hallam. “the  double helix: a personal account of  the discovery of the structure of [the  information professional’s] dna.”   australian library and information  association (alia) biennial  conference. gold coast, australia. 21‐ 24 september 2004. 29 june 2005  <http://conferences.alia.org.au/alia200 4/pdfs/partridge.h.paper.pdf>.    ‐‐‐.  “developing a culture of evidence  based practice within the library and  information profession: the impact of  library science education. a teaching  and learning model from the  queensland university of  technology.” ifla world library and  information congress, division vii  management and marketing section.  2005. 12 aug. 2005  <http://eprints.qut.edu.au/archive/000 01973/01/1973.pdf>.    smith, elizabeth and cecily martina.  “keeping the dough rising:  considering information in the  workplace with reference to the  bakery trade”. in danaher, p.a., c.  macpherson, f. nouwens, & d. orr   (eds.). (2004). lifelong learning: whose  responsibility & what is your  contribution?: refereed papers from  the 3rd international lifelong learning  conference. rockhampton, qld.:  central queensland university press.    smith, richard. “medical journals are an  extension of the marketing arm of  pharmaceutical companies.”  plos  medicine, 2.5: e138 (2005): 1‐6.  8 july  http://bmj.bmjjournals.com/cgi/conte http://www.aare.edu.au/00pap/gro00 http://www.pubmedcentral.nih.gov/ http://www.dest.gov.au/nr/rdonlyre http://www.lsda.org.uk/home.asp http://conferences.alia.org.au/ebl2005 http://www.ncver.edu.au/research/pr http://www.ncver.edu.au/ http://www.ntis.gov.au/ http://conferences.alia.org.au/alia200 http://eprints.qut.edu.au/archive/000 evidence based library and information practice 2006, 1:1      36 2005  <http://medicine.plosjournals.org/perl serv/?request=get‐ document&doi=10.1371/journal.pmed. 0020138>.    straus, sharon e. and finlay a mcalister.  “evidence based medicine: a  commentary on common criticisms”  cmaj/jamc 163.7 (2000): 837‐841          12 aug 2005   <http://www.cmaj.ca/cgi/reprint/163/7 /837>.     thomson, peter, alison anlezark, susan  dawe, and sarah hayman. the  mature‐aged and skill development  activities: a systematic review of  research. adelaide, national centre  for vocational education research,  2005. voced database. 12 dec. 2005  <http://www.ncver.edu.au/research/co mmercial/op320b.pdf>.    todd, ross.  “transitions for preferred  futures of school libraries: knowledge  space, not information place  connections, not collections actions,  not positions evidence, not advocacy”.   iasl conference. auckland, new  zealand. 9‐12 july 2001. 30 march 2005  <http://www.iasl‐ slo.org/virtualpaper2001.html>.    unesco‐unevoc international centre  for technical and vocational  education and training.  vocational  content in mass higher education?  responses to challenges for the  labour market and the work‐place.   bonn, 8‐10 september 2005. 10 oct.  2005  <http://portal.unesco.org/education/en /ev.php‐ url_id=38189&url_do=do_topi c&url_section=201.html>.                  voced database. 2005. 12 dec. 2005  <http://www.voced.edu.au/index.htm    >.    wagner, wendy. “the perils of relying on  interested parties to evaluate scientific  quality.”  american journal of  public health: supplement on  scientific evidence and public policy  95.s1 (july 2005): s99‐s106. 12 aug.  2005  <http://www.defendingscience.org/loa der.cfm?url=/commonspot/security/ge tfile.cfm&pageid=2412>.      http://medicine.plosjournals.org/perl http://www.cmaj.ca/cgi/reprint/163/7 http://www.ncver.edu.au/research/co http://www.iasl%e2%80%90slo%00 http://portal.unesco.org/education/en http://www.voced.edu.au/index.htm http://www.defendingscience.org/loa evidence based library and information practice 2011, 6.4 120 evidence based library and information practice evidence summary librarians view instruction as integral to their professional identities a review of: julien, h., & genuis, s. k. (2011). librarians’ experience of the teaching role: a national survey of librarians. library & information science research, 33(2), 103-111. doi: 10.1016/j.lisr.2010.09.005 reviewed by: ann medaille reference & instruction librarian university of nevada, reno reno, nevada, united states of america e-mail: amedaille@unr.edu received: 15 july 2011 accepted: 05 oct. 2011 2011 medaille. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to explore the ways that professional and non-professional library staff experience and relate to their instructional roles. design – online survey. setting – all types of canadian libraries, including public, school, post-secondary, medical, special, and other libraries. subjects – a total of 788 library staff persons with instructional responsibilities. methods – in 2009, the authors constructed a 20minute anonymous survey that contained questions about the nature of librarians’ instructional work, their preparation for doing instruction, and their experiences as instructors. subjects were recruited via several electronic mail lists. the authors used spss to analyze the quantitative data and nvivo to analyze the qualitative data. main results – the study found that the majority of subjects believed instruction to be integral to their professional identities, although some viewed it as an imposition. the nature of instructional work varied greatly, but included short presentations; a series of sessions; mailto:amedaille@unr.edu� evidence based library and information practice 2011, 6.4 121 semester-length courses; and one-on-one instruction. subjects prepared for instruction through on-the-job training; reading professional literature; attending workshops and conferences; taking a formal course in instruction; and other methods. on the whole, training helped library staff to feel more prepared for teaching and to embrace instructional work as integral to their professional identities. study participants derived enjoyment from instruction in the form of satisfaction with facilitating student learning; relationship building; personal development; task variety; and appreciation of the heightened profile of library staff. subjects also described several barriers to teaching, including administrative, technological, and logistical barriers; client and faculty interactions; and interpersonal challenges such as nervousness or lack of preparation. finally, subjects described the ways that instruction has changed with the impact of new technologies, increased expectations, and changing pedagogical practices. conclusion – library administrators should support the teaching duties of librarians and library staff by helping to provide them with adequate preparation time, resources, emotional support, and training. in addition, formal preparation for instruction should be integrated into professional library training programs, including mlis programs, to better prepare librarians and other library staff to participate in information literacy instruction. commentary to support the growing need for information literacy skills, librarians are increasingly called upon to serve as instructors in the formal classroom, through distance learning, or through one-on-one interactions at the reference desk. the demands brought about by the changing information landscape naturally raise the question: are librarians and other library staff adequately prepared to embrace this new instructional role? in many cases it would appear that they are either unprepared to teach or are unsure about their role as teachers. this study provides a valuable analysis of this issue and contains important implications for library administrators and library science programs. in this study the authors have built upon their previous work, in which they analyzed librarians’ experiences of the instructional role through interviews and diary data (julien & pecoskie, 2009; julien & genuis, 2009). in previous studies the authors focused on librarians’ often complex and unequal relationships with teaching faculty (julien & pecoskie, 2009) and the emotional stressors that come with doing library instruction (julien & genuis, 2009). although the population sample used in this study is not random, the high level of participation from staff from a variety of library types provides critical insights into the changing role of instruction in library work. the study received ethics board approval, and the survey instrument, which is included as an appendix, was pretested twice to ensure its validity. as the authors observed, a weakness of the study design is that participants were self-selected and therefore may have particularly strong feelings about instruction. the choice of the survey method, however, is appropriate for the study objectives. the authors found that significant relationships existed among a number of important factors with regard to preparation, expectations, and the instructional role. for example, those who took formal instruction courses felt that they were better prepared to teach and viewed instruction as important to their professional identities; this was also true of those who attended professional development activities related to instruction. in addition, those with mlis or equivalent degrees were more likely to expect to teach and to view instruction as an important part of their library roles. although these findings are significant, the most evidence based library and information practice 2011, 6.4 122 important finding to emerge from this research is that a good portion of today’s librarians and library staff do indeed view instruction as integral to their professional identities. what makes this study particularly interesting is its use of identity theory and role theory to provide a lens through which readers may view the issue of librarians’ self-conception in relation to instruction. from this theoretical standpoint, roles and identities are constructed in response to social interactions that shape expectations for patterns of behaviour. as the roles of librarians change in response to new technologies, instruction is becoming more prevalent in librarians’ work tasks, but numerous external and internal factors influence the ways that librarians view these new roles. faculty interactions, changing student needs and preferences, new technologies, teacher training, and administrative support all have an impact on the ways that librarians relate to their new roles as information literacy instructors. as this study demonstrates, the effect of these compounding factors may be that librarians experience some ambivalence about the teaching role, while at the same time they believe teaching to be important to their professional identities. one of this study’s implications concerns how the role of instruction in library work has been only minimally addressed in library school curricula. although librarians largely acquire teaching skills on the job, many would prefer to acquire these skills through their library school educations (westbrock & fabian, 2010). however, many library schools do not require students to take an instructional course or even offer any course in information literacy instruction (julien, 2005). for mlis programs that do offer instructional courses, their focus is often on school libraries and media centers, which is of limited use to those pursuing careers in different types of libraries (sproles, johnson, & farison, 2008). thus, the importance of addressing information literacy instruction in the formal training for library careers should not be underemphasized. while taking instructional courses is an important part of the preparation for library work, the process of learning to teach (and learning to teach better) is ongoing, occurring throughout one’s library career. to competently and confidently embrace their instructional roles, librarians and library staff must engage in a process of continual education about new teaching strategies and new technologies through participation in conferences, seminars, webinars, workshops, peer reviews, evaluations, self-reflection, and reading the literature. in addition to its implications for library school curricula, this study is especially important for library administrators, since they play a critical role in creating a supportive work atmosphere that provides librarians and library staff with ample time and resources to prepare for teaching and to engage in professional development. thus, this study should resonate broadly with library administrators who recognize the value of supporting librarians in their roles as information literacy instructors. references julien, h. (2005). education for information literacy instruction: a global perspective. journal of education for library and information science, 46(3), 210-216. retrieved 25 november 2011 from http://www.jstor.org/stable/40323846 julien, h., & genuis, s. k. (2009). emotional labour in librarians’ instruction work. journal of documentation, 65(6), 926-937. doi: 10.1108/00220410910998924 julien, h., & pecoskie, j. (2009). librarians’ experiences of the teaching role. library and information science research, 31, 149154. doi: 10.1016/j.lisr.2010.09.005 evidence based library and information practice 2011, 6.4 123 sproles, c., johnson, a. m., & farison, l. (2008). what the teachers are teaching: how mlis programs are preparing academic librarians for instructional roles. journal of education for library and information science, 49(3), 195-209. westbrock, t., & fabian, s. (2010). proficiencies for instruction librarians: is there still a disconnect between professional education and professional responsibilities? college & research libraries, 71(6), 569-590. retrieved 25 november 2011 from http://crl.acrl.org/content/71/6/569.full.p df / evidence based library and information practice evidence based library and information practice 2009, 4:2 152 evidence based library and information practice evidence summary libraries demonstrate low adherence to virtual reference service guidelines a review of: shachaf, pnina, and sarah m. horowitz. “virtual reference service evaluation: adherence to rusa behavioral guidelines and ifla digital reference guidelines.” library & information science research 30.2 (2008): 122-37. reviewed by: elise cogo information specialist consultant ottawa, on, canada email: ecogo@uwo.ca received: 16 december 2008 accepted: 02 april 2009 © 2009 cogo. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. objectives – this study evaluates the level to which virtual (asynchronous e-mail) reference services adhere to professional guidelines. specifically, it addresses the following research questions: 1) to what extent do virtual reference services adhere to the american library association (ala) reference and user services association (rusa) and the international federation of library associations (ifla) guidelines? 2) how does the level of adherence to rusa or ifla guidelines vary based on request type, user name, and institution? 3) is there a correlation between outcome measures of reference transactions (accuracy, completeness, and satisfaction) and the level of adherence to rusa or ifla guidelines? design – unobtrusive evaluation of researchergenerated queries. setting – fifty-four academic libraries in north america. subjects – a total of 324 queries were sent to the 54 libraries, with each library receiving six different types of requests from six different user names. methods – researchers developed two coding schemes for the guidelines (34 codes and 12 attributes for the rusa guidelines and 33 codes and 10 attributes for the ifla guidelines). each of the six user names used represented an ethnic and/or religious group identity: mary anderson (caucasian, christian), moshe cohen (caucasian, jewish), ahmed ibrahim (arab), mailto:ecogo@uwo.ca evidence based library and information practice 2009, 4:2 153 latoya johnson (african american), rosa manuz (hispanic), and chang su (asian). the six request types were designed so that three would be answered (questions 1-3) and three would be out of scope and not answered (questions 4-6). the following queries were sent, individualized for each institution: 1) dissertation query; 2) sports team query; 3) population query; 4) subject query; 5) article query; 6) request for a pdf copy. the 324 queries were uploaded into nvivo 2 software, and all e-mail transactions were coded and analyzed. main results – analysis of the 324 transactions from 54 libraries showed the following results: 1) low levels of adherence to both sets of guidelines; 2) varied levels of adherence based on request types and user names on both sets of guidelines; 3) variation in institutional rank according to different sets of guidelines; 4) no correlation between user satisfaction and adherence to either set of guidelines. conclusion – this study suggests that higher levels of virtual reference service effectiveness could be achieved by automatically integrating some less observed behaviours (e.g., thank you notes) into replies sent to users and by increasing librarians’ awareness of professional guidelines through training and detailed institutional policies. the authors also suggest that librarians should be aware of their tendencies to react differently to different user groups, and that administrators can facilitate this by providing diversity workshops. commentary one of the primary strengths of this study is that it provides a detailed methodology for systematically analyzing transactions in light of ideal professional standards, and it provides an empirical benchmark for evaluating virtual reference services and for objective comparison with other libraries. the authors note that a few previous evaluative studies of virtual reference services have used these professional guidelines but only parts of the guidelines were used and those that were used were not treated systematically. the article includes many tables with the coding schemes used and transaction results and frequencies, thereby providing clear and transparent methods and results. the large number of libraries and transactions evaluated in the study adds strength to the key finding that there was an overall low level of adherence to professional guidelines. the study also aimed to evaluate the effects of several reference query variables (e.g., request type and user name,), which were included in the interpretation of the results. the use of nvivo software to facilitate the qualitative analysis allowed for the coding of text as well as the linking and modeling of qualitative data. a detailed coding scheme was developed to represent each guideline criterion in order that systematic interpretations and analyses could be made. this allowed a large amount of qualitative information to be analyzed quantitatively and for key statistics to be generated. in addition, the software’s searching and co-occurrence capabilities allowed for better understanding of the frequencies that were generated from the transactions. one limitation that the authors noted is that some of the frequencies need to be interpreted with caution since they may not apply in each situation. for example, certain request types may violate institutional policy. other limitations of this study, which the authors suggest as avenues for future research, include the inability to assess real information needs, real-time interactions, or user affiliation. one of the implications of this study is the need for further research and training within the library profession with regard to service to various ethnic and/or religious groups. overall, this study makes a very significant contribution evidence based library and information practice 2009, 4:2 154 to the methodology of virtual reference service evaluation and provides a benchmark for the status of virtual reference service guidelines adherence in academic libraries. microsoft word es_wilson.doc evidence based library and information practice 2007, 2:1  134 evidence based library and information practice       evidence summary    a content analysis of google scholar: coverage varies by discipline and by database       a review of:  neuhaus, chris, ellen neuhaus, alan asher, and clint wrede. “the depth and breadth  of google scholar: an empirical study.” portal: libraries and the academy 6.2  (apr. 2006): 127‐41.    reviewed by:   virginia wilson  shirp coordinator, health sciences library, university of saskatchewan  saskatoon, saskatchewan, canada  e‐mail: v_e_wilson@hotmail.com      received: 30 november 2006    accepted: 8 january 2007      © 2007 wilson. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to ascertain the coverage by  discipline, publication date, publication  language, and upload frequency of the  scholarly articles found in google scholar.    design – comparative content analyses.    setting – electronic information resources  accessible via the internet (both freely  accessible and for‐fee databases).    subjects – forty‐seven online databases and  google scholar.    methods – the study compared the content  of 47 databases (21 internet resources freely  available to the general public; 26 restricted‐ access databases) covering a variety of  subjects with the content of google scholar.  each database was assigned to one of the  following discipline categories: business,  education, humanities, science and medicine,  social science, and multidisciplinary. from  april through july 2005, researchers  generated random samples of 50 article titles  from each of the 47 databases and searched  the titles on google scholar to determine  inclusion.     related studies were conducted for  publication date and publication language  analysis, and for the google scholar upload  frequency study. for the publication date  study, random samples from one database  (psycinfo) with a high degree of variability  in google scholar coverage were searched for  1990, 2000, and 2004. for the publication  language study, google scholar coverage of  psycinfo articles in english was compared  to coverage of psycinfo articles published  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:1  135 in non‐english languages. for the upload  frequency study, two databases chosen for  their high degree of coverage (biomed  central and pubmed) were monitored to  determine how often the new content was  uploaded to google scholar.    main results – this study revealed that  content covered by google scholar varies  greatly from database to database and from  discipline to discipline. of the 47 databases  studied, coverage ranged from 6% to 100%.  mean and median values of coverage for all  databases were both 60%. the mean  discipline category scores varied from the  humanities databases at 10% coverage, to  the social sciences and education at 39% and  41% respectively, to science and medicine  databases at 76% coverage. mean coverage  was 77% for the multidisciplinary databases.  mean coverage of open access journal  databases was 95%, freely accessible  databases had 84% mean coverage, and  single publisher databases had 83% mean  coverage.    the publication language study found a bias  towards english language publications. as  well, a publication date bias was found –  coverage of earlier dates was not as  thorough as coverage of more recent  publications. in the upload frequency study,  for biomed central and pubmed there  appears to be an approximately 15‐week  delay in the uploading of new material to  google scholar.    conclusions – the results of this study  serve to alert researchers and information  professionals that google scholar (in beta test  mode at the time of the study) has poor  coverage in certain areas. to those with  access to commercial databases, this serves  as a cautionary tale. to those with a dearth  of commercial databases, google scholar is a  welcome site and can provide at least some  information. the researchers state that the  search engine itself could make future  content studies unnecessary if it decides to  make its content collection methodology  transparent to users. upload frequency,  google scholar’s linking services, the  advanced search option, and the “cited by”  feature could all be subjects of future studies.  for its first year in operation, google scholar  offers a broad range of discipline coverage  with substantial depth in some areas. at the  time of the study, google scholar was  working with libraries and vendors to  connect search results to library‐licensed full  text.    commentary    google scholar has certainly evoked mixed  reactions from library and information  professionals since its appearance. this  study is a revealing one, as google scholar  does not release information as to what  content is included and how it is chosen.  there has been no other content analysis  undertaken to this depth thus far on google  scholar. however, there are some issues  about the methodology of the study that call  the results into question. for example, the  investigators included 47 databases in their  comparative analysis. they go to great  lengths to describe how article title samples  are randomly generated from these chosen  databases for inclusion into the study.  however, they do not describe – or even  mention – how the 47 databases are chosen.     the researchers divide the 47 databases into  disciplinary categories. this is a necessary  step in order to determine coverage by  discipline. however, while there are 15  databases in the science and medicine  category, there is only one business database  included.  other databases broken down by  discipline include 3 in education, 5 in the  humanities, 7 in the social sciences, and 16  multidisciplinary databases. the  unevenness of the databases by discipline  calls into question the validity of the results.  in particular, generalizing about business  evidence based library and information practice 2007, 2:1  136 coverage based on one database does not  give accurate results as to how business as a  discipline is covered by google scholar. the  inclusion of so many multidisciplinary  databases is problematic, as they skew the  “by discipline” results. the randomly  generated article titles from the  multidisciplinary databases presumably  included articles from a variety of  disciplines. the percentage of coverage in  the humanities, for example, does not  include the humanities articles that may  have been generated from the  multidisciplinary databases. and finally,  assigning databases to disciplinary  categories can be an interpretive exercise.  for this study, should the atla religion  database be categorized as humanities? or  should it remain in social science, where the  researchers included it? as well, library  literature could have been classified as  social science, whereas the researchers  chose education.    additionally, it is interesting to note that in  the coverage portion of the study, pubmed  had 100% coverage in google scholar. in the  upload testing portion of the study, it is  clear that there is significant lag time in the  uploading of recent material from pubmed.  this discrepancy points to the inadequacy of  the 50 article per database sample size. a  larger sample size, particularly as some  databases contain millions of citations,  would have given a more accurate picture of  actual coverage of any given database.    this study is useful to information  practitioners in a provisionary way. it is  good to know what kind of content is  covered by google scholar and what the  deficits might be when helping patrons to  navigate google scholar. it would have been  more useful had the article focused solely on  the content inclusion analysis, and saved the  publication date and publication language  study and the upload frequency study  results for another paper. packing all the  results into one relatively short paper did all  of the studies a disservice. this article is  usable for librarians across all sectors, and in  particular information professionals who do  not have access to a wide range of for‐fee  databases. however, google scholar must  still be navigated carefully, as this study is  certainly not the definitive answer as to  what the database/search engine includes.  article   the collision of two lexicons: librarians, composition instructors and the vocabulary of source evaluation   toni m. carter reference and instruction librarian reference department auburn university libraries auburn, al, usa email: tcarter@auburn.edu   todd aldridge graduate teaching assistant department of english auburn university auburn, al, usa email: tja0004@auburn.edu   received: 21 july 2015    accepted: 5 jan. 2016           2016 carter and aldridge. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – the study has two aims. the first is to identify words and phrases from information literacy and rhetoric and composition that students used to justify the comparability of two sources. the second is to interpret the effectiveness of students’ application of these evaluative vocabularies and explore the implications for librarians and first-year composition instructors’ collaborations.   methods – a librarian and a first-year composition instructor taught a class on source evaluation using the language of information literacy, composition, and rhetorical analysis (i.e., classical, aristotelian, rhetorical appeals). students applied the information learned from the instruction session to help them locate and select two sources of comparable genre and rigor for the purpose of an essay assignment. the authors assessed this writing assignment for students’ evaluative diction to identify how they could improve their understanding of each other’s discourse.   results – the authors’ analysis of the student writing sample exposes struggles in how students understand, apply, and integrate the jargon of information literacy and rhetoric and composition. assessment shows that students chose the language of rhetoric and composition rather than the language of information literacy, they selected the broadest and/or vaguest terms to evaluate their sources, and they applied circular reasoning when justifying their choices. when introduced to analogous concepts or terms between the two discourses, students cherry-picked the terms that allowed for the easiest, albeit, least-meaningful evaluations.   conclusion – the authors found that their unfamiliarity with each other’s discourse revealed itself in both the class and the student writing. they discovered that these miscommunications affected students’ language use in their written source evaluations. in fact, the authors conclude that this oversight in addressing the subtle differences between the two vocabularies was detrimental to student learning. to improve communication and students’ source evaluation, the authors consider developing a common vocabulary for more consistency between the two lexicons.     introduction   phrases such as library jargon, library terminology, and library vocabulary evoke references to services and objects, such as circulation desks, monographs, and reserves. much has been written about librarians’ efforts to help patrons understand this language (adedibu & ajala, 2011; ayre, smith, & cleeve, 2006; chaudhry & choo, 2001; dewey, 1999; doran, 1998; foster, 2010; houdyshell, 1998; hutcherson, 2004; imler & eichelberger, 2014; naismith & stein, 1989; pinto, cordon, & gómez diaz, 2010; sonsteby & dejonghe, 2013; spivey, 2000; swanson & green, 2011). rather than alluding to tangible objects and services, information literacy jargon, on the other hand, may elicit abstract thoughts and actions that require a higher-degree of critical thinking to comprehend and apply (pinto, cordon, & gómez diaz, 2010). possibly due to time limitations or misconceptions of students’ prior knowledge, librarians can easily overwhelm first-year composition students with this terminology during library instruction classes. for instance, in a “source evaluation” session, a librarian might hand students a checklist that describes evaluative criteria such as authority, accuracy, currency, purpose, relevancy, objectivity/bias, among others. in addition to exposing students to this laundry list of terms, checklists neglect the complexities and nuances of source evaluation; they fail to consider information need and encourage a dichotomous assessment of information (benjas-small, archer, tucker, vassady, & resor whicker, 2013; burkholder, 2010; meola, 2004). this “checklist” approach has been under increased scrutiny since the creation of the acrl framework for information literacy (association of college & research libraries, 2015). the framework encourages a more holistic and authentic pedagogy which focuses on the information-creation process, and how this process affects credibility and the appropriateness of a source. despite this gradual departure from “checklists,” librarians continue to use the same or similar words to teach evaluation skills, and students must still understand the meanings and usages of such terms as authority, purpose, and bias.   the casual blending of librarians’ language with that of composition instructors can further confuse a discussion on source evaluation. in a one-shot library session, librarians tend to approach source evaluation as locating and identifying a “credible” source that meets the students’ information needs. librarians teach students to evaluate a source’s authority, purpose, audience, and so forth. a first-year composition instructor might concur with this pedagogy, but could have different ideas of what makes a source “credible”, “reliable,” “reputable,” etc., than that of the librarian. further, instructors view source evaluation through the lens of rhetorical analysis – a concept that requires students to evaluate the author’s argument, in addition to the credibility of the source from which it is found (mazziotti & grettano, 2011). students must consider authors’ logic, persuasiveness, and ethos. these subtle distinctions in purposes may not be obvious to librarian and instructor, and this oversight can spill over into their use of language in the classroom. through the examination of students’ written work, the authors of this paper – an instruction librarian and an instructor of first-year composition – illustrate how inconsistencies in language-use and meaning between these two groups can negatively affect student learning. we consider the development of a common vocabulary as a possible solution.   background auburn university is a land, sea, and space grant university in east alabama with an enrollment of approximately 26,000 students. the english composition program serves about 4,000 undergraduate students in nearly 250 classes each fall and spring. engl1100 is the introductory course on academic reading and writing and focuses on the development of writing processes and rhetorical awareness of audience and style. taking up the writing skills from engl1100, engl1120 emphasizes argumentative writing and academic research that requires library instruction sessions. engl1120 is based on a scaffolded curriculum, in which students write several shorter essays throughout the semester, culminating in a final research paper. each engl1120 class takes part in 2 to 3 library sessions, which make up the bulk of the 600-700 information literacy classes taught each year by the university librarians. this is also where the majority of assessment for the core curriculum’s information literacy student learning outcome occurs. sessions concentrate on basic information literacy concepts such as keyword development, search strategies, and source evaluation.   literature review   collaborations between librarians and composition instructors, the inherent relationship between information literacy and writing, and the concept of information literacy as a situated literacy within composition have all received substantial coverage in the literature (barclay & barclay, 1994; birmingham et al., 2008; bowles-terry, davis, & holliday, 2010; fister, 1992; hlavaty & townsend, 2010; jacobs & jacobs, 2009; mazziotti & grettano, 2011; mounce, 2009; palsson & mcdade, 2014; shields, 2014; sult & mills, 2006; white-farnham & caffrey gardner, 2014). this review focuses on a few additional works that most closely relate to our research.   about a little over a decade ago, rolf norgaard (2003) contended that incorporating concepts from rhetoric and composition into information literacy “would help yield a more situated, process-oriented literacy relevant to a broad range of rhetorical and intellectual activities” (p. 125). he insisted that this collaboration would help to transform information literacy practices from skills-based into a more dynamic practice of intellectual and contextual inquiry (2003, p.125; 2004, p.221). in exchange, norgaard believed information literacy would help to legitimize the study of rhetoric and composition, usually viewed as part of the ivory tower, by investing it in real-world actions (2004, p. 225). norgaard argued that blending information literary and rhetoric and composition would help strengthen instruction and contribute to the development of a “situated” or “rhetoricized” information literacy (2004, p. 221). for norgaard, these adaptations would help both fields move beyond surface features and rote search tasks (e.g., grammar and citation) into new territories of mutual, disciplinary growth.   in this “provocation” for an integrated approach to information literacy, norgaard (2004) touches on mutual benefits for each discipline (p. 225) without entertaining, in depth, potential shortcomings. one possible complication from this collaborative approach comes out of norgaard’s discussion of the language of information literacy. he explained that instructors of composition and librarians both seek to make research accessible, relevant, and transparent for students. considering that jargon occludes this entry and command of information literacy, he asserted that framing literacy practices in ordinary language would help to establish common ground for the complex work that instructors of composition and librarians do together (2003, p. 126). however, using the language of everyday speech to draw the fields together and enrich each other, we contend, is far more complex than sharing theories and pedagogies in mutually respectful teaching and research environments. librarians and composition instructors may speak the same language, but in a manner of speaking, they do not. for students, overlapping vocabularies produce confusing and sometimes competing conceptions of how to access and evaluate information.   research into student evaluative skills primarily comes from studies of composition and linguistics, in particular the work of siew mei wu. assessing the language of evaluation in argumentative essays by student writers, wu and allison (2005) found student writers who supported their thesis statements with clear, evaluative expression performed better academically than those who relied primarily on exposition. as students who performed poorly tended to discuss a topic rather than develop an argument, wu and allison found that “the high-rated essay writers tend to maintain a more dialogically expansive stance to soften the assertiveness level of the claims” (p. 124). in part, this “dialogical expanse” develops through fluid integration of sources that supports clear assertions from the student writer. examining another sample of student argumentative writing, wu (2008) explained that student writers often lack the disciplinary discourse and jargon to assess and form their own arguments in writing, but assignments often require them to read the discourse, understand it, and participate in that academic conversation (p. 59). students not only have to comprehend the discourse of the documents that they assess but also understand the language of evaluation used in the classroom. in general, the implications of wu’s research (2008) suggest that students who “situated” their arguments outperformed those who provided vague details and explanations, namely those who used the language of evaluation discretely (p. 71). the language of evaluative expression reflects criteria (e.g., bias, citation, credibility) from both rhetoric and composition and information literacy. these studies indicate that academic readers assess the quality of an argument through evaluative language, underscoring the importance of consistent vocabulary and conceptual frameworks in this process.   as wu (2008) points out the complex web of discourses that students try to untangle in order to analyze and evaluate information, holliday and rogers (2013) – in a librarian and writing instructor’s collaboration – report on the language of information literacy used in the classroom (i.e., by them and by students) and how that affects students’ engagement with information. they suggest the language of evaluation used by the two groups alike affects student ability to achieve meaningful and coherent source evaluation. they assert that the “[i]nstructional discourse” at play between the librarian and the writing instructor contribute to student researchers and student writers producing artificial evaluations (p. 259). the authors propose a shift in both language and “instructional attention” in the classroom from “finding sources” to a focus on “learning about” sources (p.268). our study also examines discourse, but we focus on how two languages come together – that of information literacy and rhetoric and composition – and how this union affects how students evaluate sources in their written work.   one of the goals of information literacy and composition is to teach students methods of source evaluation, applicable to many assignments and situations in order to assess the quality of a text and its argument. these goals seem mutually beneficial and congruent, but we contend that our language gets in the way of student uptake and application. to borrow a metaphor from holliday and rogers (2013), these heuristics (e.g., “checklist method”) for source evaluation are tools that we intend for students to learn and apply in their research and writing (p. 259). however, we claim that combining information literacy and rhetoric and composition is like dumping two boxes of tools onto our students; instead of a smooth and soluble integration, merging discourses produces a pile of symbolic tools, some similar, some different, some redundant, and some incomprehensible, that all parties involve need to sort through.   aims   we conducted a semester-long study of one engl1120 class, in order to assess how well students transferred the skills and concepts learned in course-integrated library instruction sessions to their assignments. from the assessments, we hoped to identify outcomes for which the librarian could train the instructor to further discuss with students after the sessions. the unpredictable nature of assessment led us down a different path, however.   examination of topic proposals written after a class on source evaluation revealed students’ reliance on rhetoric and composition vocabulary to evaluate information; this occurred, despite explicit instructions to consider what they had learned from the librarian. the few “information literacy” words used were closely woven within rhetoric and composition terminology, although most often ineffectively. we realized that during our planning session for the class, we had omitted a thorough discussion of each other’s source evaluation discourse. we contend that this resulted in muddled and superficial evaluations by students. the discussion below examines how students appropriated the diction and vocabulary of information literacy and rhetoric and composition following a session on source evaluation. we share the consequences of a glossed-over understanding of each other’s language – a somewhat inconspicuous topic that needs more attention in the literature.   methods   the engl1120 class that is the focus of this study consisted of 27 students: 20 freshmen, 6 sophomores, and 1 junior with majors in the liberal arts, science and math, engineering, business, education, and nursing. the first major essay of this course was a rhetorical analysis of one text, and did not require a library session. the second essay asked students to locate two texts that were of comparable genre and rigor on a topic related to cultural diversity. the instructor hoped that setting limits on the types of sources that students could compare would help eliminate weak and unbalanced comparisons that he had graded in previous classes (for instance, comparing arguments in a peer-reviewed article to the opinions of a blogger). students would analyze the sources’ rhetoric guided by instructions provided in the assignment prompt and explain which author made the better argument and why. before students began this essay, the instructor assigned a topic proposal assignment in the form of a short writing (roughly 250-300 words) to determine whether students understood the expectations of the larger assignment. the topic proposal required students to 1) provide a brief summary of each of the authors’ claims; 2) justify why the articles were of comparable rigor and genre (students were to consider the information from the library session); 3) defend the better argument and include the criteria they used to determine this; and, 4) include a plan to support their (the students’) argument and ideas. the instructor kept the word count low to ensure succinct and well-thought out evaluations.   during our pre-class meeting, we outlined a team-taught library session on source evaluation to coincide with this assignment. in an effort to deemphasize the evaluation “checklist” and the superficial assessment of information that it encourages, the librarian suggested a lesson plan centered on the “information lifecycle.” we also acknowledged and incorporated two terms on the assignment that the librarian normally did not use; genre, defined by the instructor as “a category of writing or art that share similarities in form and style” and rigor, “the thoroughness and accuracy of a source.” focus on genre would transform a dichotomous discussion of the differences in the format of “popular” and “scholarly” sources to a closer examination of both the way a source looks and the way it is written. rigor would discourage the assessment of peer-reviewed articles as being the “best” type of source, but instead characterize the review process as a factor to consider – and one that should strengthen – as we moved around the information lifecycle. throughout the semester, the instructor framed the concept of rhetorical analysis using the three aristotelian, persuasive appeals: logos (e.g., logic, reasoning, evidence), pathos (e.g., emotionally charged language, anecdotes, narration), and ethos (e.g., credibility, diction, tone). the pre-class planning session did not include a discussion of this particular discourse and its relation to source evaluation. we phrased our learning outcome as follows: “students will learn that information is disseminated in different formats and that the accuracy and thoroughness (rigor) of information is often related to the length of time it takes to produce the information and the format in which it is reported” (carter & aldridge, 2015).   we began the library session with a review of the concept of genre, which the instructor had introduced in a previous class. he explained that sources within the same genres share comparable patterns of arguments, and provided characteristics to consider when identifying a genre: length, tone, sentence complexity, level of formality and informality, use of visuals, kinds of evidence, depth of research, and presence or absence of documentation (ramage, bean, & johnson, 2010). the discussion included examples of types of genres, such as op-ed pieces and scholarly articles, and the librarian asked students from which genres they might find sources for their assignment.   the librarian next introduced a current event in cultural diversity that would serve as a class topic. she placed students in groups of two, and then distributed to each group a piece of paper with a pre-selected source written on it. the sources represented a variety of genres: broadcast news, online and print newspapers, magazines, trade magazines, scholarly journals, and books. keeping the example topic in mind, students answered worksheet questions about their source that addressed the information creation process. prior to class, the librarian had set up five stations around the classroom that represented a point in the information lifecycle. she had labeled the stations “one day,” “one week,” “one month,” “one year,” and “longer than one year.” after they completed the worksheet, she asked each group to tape their source at the station along the information lifecycle that most closely matched its speed of publication. she then led a discussion about the rigor of each as they moved around the room, and essentially around the information lifecycle. the discussion incorporated familiar terms such as authority, accuracy, and purpose, but not in conjunction with a checklist. for a revised version of this lesson plan, see carter & aldridge (2015). the class concluded with an introduction to the academic search premier database and time for the students to search. after class, the instructor shared the completed topic proposals with the librarian, and they met several times to discuss the results. approval from the university’s institutional review board was required to conduct the study, and age of consent in alabama is 19. this library session occurred early in the semester, before approximately half of the students turned 19. therefore, out of the 27 students, we are only able to report on the artifacts of 13 students.             apart from one another, we each assessed the topic proposal assignments that students completed after the library session with the use of two simple rubrics. the first rubric measured student success at finding articles of similar genre and rigor. if we graded a paper as “sufficient” or “accomplished” at this task, we each applied the second rubric to determine how well they justified their choices. to earn an “accomplished” rating on this second task, a student “succeeded in convincingly justifying their selections,” while a “sufficient” rating indicated that the student tried to justify their selections, but fell short. we reserved “insufficient” ratings for those students who put forth little effort. then we came together to discuss each other’s results. we found that we had each applied the rubric similarly, informally norming the rubric. in an effort to conduct an organized review of word choices, we identified categories of evaluative language that the students used in their topic proposals. the categories listed here were developed through our discussion of the patterns that each of us identified when we read throught the proposals separate from one another: logic (logos, evidence, facts, organization, reasoning); emotion (pathos, personal stories, anecdotes, charged language); credibility (ethos, ethics, credentials, character, authority); surface features (mention of length of article, credentials mentioned without analysis, bias, citations/references); genre (identified a specific genre). we then read through the papers one last time separately, color-coding for each of the categories. this enhanced our later discussions by providing a visualization of the vocabulary patterns.   results and discussion   while most students could locate the “right” types of sources (of similar rigor and genre), the majority of their attempts to evaluate the sources involved sweeping statements using the broadest terminology possible. this strategy did not result in what we considered accomplished evaluations, and we identified three possible reasons for this poor performance: 1) flawed assignment design – students were asked to do too much with too few words, therefore could not be as precise as we would have liked; 2) an ineffective information literacy session; and/or 3) a lack of clear understanding of evaluative language. while all three represent crucial pieces of the puzzle, language-use rose to the top for us as the most stimulating finding of this assessment. we thought this focus would touch on the other two potential factors as well.   we moved forward by labeling each evaluative word choice as either an “information literacy” or “rhetoric and composition” term. we based these labels on the language each of us most commonly used in our respective classes. table 1 illustrates students’ choice of words divided by the authors into these two categories.   the majority these evaluative word choices fell into three categories: logic, emotion, and credibility. our discussion below is based on this framework.     table 1 word choices related to information literacy or rhetoric and composition information literacy rhetoric and composition credibility logic authority evidence format-related features (e.g., length, credentials, citations) facts   organization   reasoning   emotion   pathos   the use of personal stories   the use of “charged” language   character   genre     logic   figure 1 shows the words students chose when referring to the logic of an author. the librarian discussed reference lists and citing sources in the information literacy session, and the importance of each in determining authority and accuracy. however, students mostly chose less-specific concepts found in their composition reader.     figure 1 student word choices related to logic.     for example, rather than explaining that an author had “cited sources,” students preferred the term “evidence.” this may seem like a trivial difference, until the applications of the word and its effect on student performance is considered. student 1 in our sample used the word evidence three separate times. the student describes, “[the author] begins by explaning he first believed that gun control was a positive move forward, but later changed his thesis after considering evidence. although the student impressively applies the language of rhetorical analysis (e.g, “his thesis”), he or she makes no clear point about what type of evidence swayed the author to change positions. evidence seems like whatever material the author uses to support his or her point. later, the student gets somewhat more specific, adding adjectival modifiers to differentiate types of evidence: “[both articles] support their evidence through historical evidence.” this seems like a firm step toward specific evaluative analysis. narrowing evidence into manageable categories begins to demonstrate the student’s awareness of different types of evidence and their potential uses. however, the student made up this category of evidence on the spot, since it was not introduced in discussions during the library sessions or during writing class. what consitutes historical evidence and why it matters in evaluating the author and the source’s effectiveness are simply mentioned and then abandoned. next, the student continues to bring the discussion into more focus: “[the author] uses some factual based evidence, but lacking a proper amount of citation and logical appeal.” factual-based evidence seems like a straightforward categorization – it’s evidence based on facts. questions remain, however, and many go unanswered or unaddressed about the nature and origin of those facts. what, moreover, is the “proper amount of citation” to appeal logically and appropriately to the audience? why is that the case? why are some facts more persuasive, reliable, and fitting for one audience over others? how and why? it’s repetitive questioning, no doubt, but important for evaluating a source beyond its surface functions, parts, and pieces. those deep level analytical questions are left in favor of shallow responses. this student’s identifications of evidence capture a dominent trend that runs through the entire sample –students disregarded the suasive function of the types of evidence used to describe the article. if it has evidence, it is a good source. if it has more evidence than the other source, it is likely better. there is little mention of the quality of evidence, or sources, consulted. immersing students in this vague terminology provided them with the flexibility to make words mean what they wanted them to mean – it required less thoughtful evaluation and less critical thinking.   emotion   as we identified parallels within our two lexicons, we ran into a discrepancy when considering pathos and emotion. the librarian considered emotion to be connected to bias, while the composition instructor argued that bias, although it might be included under pathos, primarily falls within ethos and speaks to the credibility of an author. this in itself serves as a valuable illustration of lexical mismatch. for the purposes of this discussion, we favor the composition instructor’s view. as seen in figure 2, students pulled from a limited vocabulary when discussing pathos.   although students generally made empty evaluations of pathos (similar to those discussed in the logos section), they found a lack of emotional appeal as a real problem. beyond unsound logic, unreasonable beliefs, or tenuous support, students often condemned an author who failed to move readers emotionally. one student claimed that strong pathos was the crucial evaluative element between his or her authors: “the main point i will make in my essay, though, is the lack of pathos in [the author’s] article. [the other author] fills his article with emotional appeal, which makes it very strong.” although the students made clear claims about the effectiveness of pathos, what their proposals leave out are specific details. how do these authors affect readers? why is that specific method weak or strong or persuasive? these questions we both want to know.   figure 2 student word choices related to emotion.     figure 3 student word choices related to credibility.     however, one student stood out from this sample. this student often leads with generalities, most of the writers from our sample do, but he or she steps toward specific evaluation through analysis. analyzing and evaluating two sources about the legalization of marijuana, the student starts with a vague statement, introducing the rhetorical category: “[‘article title’] will be addressed by showing [the author’s] lack of logical evidence and no emotional appeal to his audience.” the student’s claim to “no emotional appeal” is sweeping and inaccurate, since no document completely lacks the ability to affect an audience, even if it produces boredom or contempt. he restates this position further: “[the author] also does nothing to connect with the people that don’t use the drug while he is arguing against the legalization of marijuana thus showing his lack of emotional appeal.” although the student made weak claims about pathos of the article considered rhetorically weaker, he or she treated the source considered rhetorically effective more precisely. “[the other author’s] article is more effective than ineffective because of his use of strong emotional appeal and his use of situations that his intended audience can easily relate to.” the student doesn’t connect that “his use of situations” are a facet of “strong emotional appeal,” which is an enormous category. he reiterates later, “[the author’s] use of situations that make his argument easy to relate to.” by situations, the student means to describe narratives, anecdotes, or descriptions that the author uses to concretize the policies for which he argues. in addition to identifying a specific emotional strategy used by the author, this student reaches a solid conclusion regarding the emotions that the author intends to elicit from the audience: “he also evokes sympathy and happiness at different times as he shows the marijuana dispensaries being shut down and the excitement of citizens in states where the drug was legalized.” the mention of “sympathy and happiness” seems like small steps and still somewhat vague, yet, unlike most of our sample, this student actually proposed specific emotions rather than simply mentioning “appeals to emotion” or “uses pathos.” this is the type of evaluation and analysis that we encourage in student evaluation because it, at the very least, displays a measure of critical, evaluative thinking.   the many emphatic criticisms levied against an author’s pathos suggest that students may need instruction on how to distance their personal point of view during source evaluation yet register their reactions to emotive language carefully. that they produce circular evaluations about pathos might seem removed from the librarian’s goals, but the student’s fixation on emotional appeal and swift criticism of the lack thereof might suggest that they are not thinking critically about finding and evaluating the best sources. in fact, their examinations of and references to pathos suggest that affective language may influence student source decisions in a detrimental way.   credibility   arguing for an author’s ethos, authority, credibility, under whatever name, eluded many students from our sample. a factor contributing to students’ poor performance could be our two similar but competing definitions of authority. the librarian took a traditional (albeit changing) approach to teaching authority by focusing on an author’s credentials. for composition, ethos takes this oversimplified view into consideration, but also requires evaluating how an author exhibits authority through proper diction and ethical claims. furthermore, a credible author should present counterarguments fairly, and if he or she does not, then their manipulation of information or bias might compromise their ethos, their credibility. the information literacy approach promoted a surface-level evaluation, while the rhetorical analysis of authority required a thorough reading and comprehension of the source. students applied the librarian’s definition of authority and tried to make it fit the rhetorical analysis ethos-framework. rather than using the term authority, however, they chose to use credible – a word not formally defined in class, but rather tossed around loosely by the librarian without considering the implications for student understanding.   for example, one student acknowledged, “both of the authors appeal to ethos almost equally due to their credibility and background.” another student remarked, “the author in article one has many ethos that help to [sic] his argument which make him credible.” essentially, the students decided that the author is credible because he or she is credible. although they show awareness of credibility and its importance for evaluating sources, their circular arguments demonstrate missing analytical tools for identifying specific appeals to credibility.   few students constructed nuanced analyses and evaluation of credibility. these examples, however, indicate awareness of the implicit nature of ethos assembled by the parts of writing. one student claims, “throughout the article she [the author] is very sincere and seems to really care that texting while driving should be banned.” this student reaches a conclusion based on synthesizing the parts of the argument. this may seem vague, but his or her point speaks to the tone and voice of the author in the article, rather than the external factors, like credentials or publisher, on which most students remarked. the same student states, “[the other author] uses ways of relating to both sides,” an instrumental gesture for conveying fairness and comprehension in an argument. had the student identified what specific components of writing and argument made this first author appear “very sincere” or seeming “to really care” or what “ways of relating to both sides” used by the second author, he or she would advance toward strong evaluation of ethos. that he or she sees beyond the surface, beyond the literal demonstrates analysis and evaluation that we encourage and endeavor to replicate in student scholars. another student, writing about atheism and theism, drew out that one of his or her author’s was more persuasive than the other because of her “understanding and placating tone, and her experience with both sides of a theistic existence.” this student recognizes the author’s intention of forging common ground over a contentious topic with a potentially hostile audience comes through how she writes and not simply what. his or her evaluation of the author’s ethos, though minor, stands out from our sample because the student compresses several dimensions of ethical credibility into one single sentence. however, these evaluative statements were the exception and not the rule.   in addition, limited class time meant that some crucial points were glossed over. unfortunately, students walked away with the impression that bias was bad, and could most often be identified as one-sided. they ignored discussions of bias led by the instructor throughout the semester in which he presented one-sided as arguments slanted toward an audience in favor of the topic (e.g., arguing for a better football stadium to football fans) as well as an argument without consideration or acknowledgement of counterarguments. for instance, one student, comparing argumentative articles on the issue of abortion, accuses an author of a pro-life argument to be “rhetorically ineffective” because of “presenting a one-sided argument.” this conclusion may be the building block of a strong, detailed evaluation. however, in comparing the two articles, the student concludes, “they have similar genres in which the author has a one-sided point of view and uses specific detail to argue pro-life or pro-choice.” in the same paper, the student changes use of the term “one-sided,” maintaining it as a point of comparison between the two sources. for this student writer, the “one-sided” approach clarifies the author’s argument and intention, forgoing engagement with counterarguments or introducing alternative perspectives in a fair and comprehensive manner. the phrase contributed to dichotomous evaluation of information.   limitations of study   in consideration of the problems with our instruction, one issue stands above the rest. the composition instructor’s assignment asked students to say too much in very few words. the instructor had intended for students to demonstrate basic understanding of the major essay’s purpose and show that they had, in fact, started the writing and researching process. however, compressing a summary, a justification, and a developing argument into 250 to 300 words, simply could not be done well. rather than increase word count, it may be more productive to cut the objectives and focus the assignment on justifying their choices. this might yield more developed and thoughtful conclusions. having said this, we believe if provided with a larger sample, we would most likely see similar trends as seen in figures 2, 3, and 4 – students’ inclination to use broad, somewhat meaningless words. we cannot prove this with our data, however, but hope others will take up this research and move it forward.   that the language of information literacy was mostly missing from their analyses raises an important question. how effective was the instruction session when most of the information literacy terms were never used in the students’ writing? is it realistic to expect students to remember a multitude of terms, comprehend the meanings of these terms, and apply them appropriately after a 50-minute information literacy session? instruction prior to and during the information literacy session may have steered students in the wrong direction. before the session, the composition instructor presented information on genre to students using genres that he hypothesized they were familiar with (e.g., action films, teen dystopian novels, etc.) as a way to help them approach analyzing and evaluating more conventional college-level sources (e.g., op-ed pieces, peer-reviewed articles, etc.). using examples from popular culture only to build common ground for understanding the concept of genre, however, may have stunted students’ ability to see the transferability of these skills to the evaluation of academic sources. moreover, the librarian fell short in her attempt to fully move away from the “checklist approach” by encouraging students to rely on author’s credentials for authority, rather than considering information need, or the instructor’s definition of authority.   implications and conclusion   reflecting back on our project, students spent more time with the instructor and had more incentive to use his language given that he graded their work. our research showed that the words used by the instructor – ethos, evidence, so forth – took on different forms for the librarian, such as sources, references, etc. it may seem that the instructor and librarian were saying the same things, but just using different words. based on the students’ writing and the instructor and librarian’s consultations afterwards, however, these seemingly similar words had different meanings. it seems like we’re arguing semantics here, which is commonly seen as nitpicky and frustrating. but, in this case, semantics matter. are our languages similar enough that we can have a common vocabulary or do we need a better understanding of our languages so we’re not working at cross-purposes? are we enabling students to take the easy way out because of the inconsistencies in the languages that we use?   we contend that both sides’ concept of sources could serve as a starting point for a more blended discourse. in the information literacy session discussed above, the word genre replaced source and format in the traditional framework of popular versus scholarly sources. through our post-class discussions, we learned that we both meant basically the same thing, but just expressed it differently. burkholder (2010) speaks to this by arguing for the use of genre theory to redefine sources by “bridging the gap between what a form really is and what it is actually designed to do” (p. 2). bizup (2008) argues in favor of a vocabulary to describe how writers use sources, rather than for types of sources (p. 75). this could be a perfect opportunity to combine ideas from rhetoric and composition and information literacy to create a mutually-endorsed descriptor. however, this requires a higher-level of understanding of each other’s discourse. siloing our thoughts and concepts into distinct teaching responsibilities (i.e., you teach this, i teach that) will no longer suffice. composition represents only the beginning of the journey, as discourse becomes more complex as students’ progress through their majors. frank conversations with faculty about the purpose of information literacy instruction and their expectations of student performance must also include a discussion of disciplinary discourse. clear language serves as the crux of comprehension.   at the end of our analysis, the paths forward split in many directions. one way is toward further standardization of conceptual vocabulary for source evaluation. if instructors of composition and librarians shared identical language and methods, confusions and redundancies in our respective approaches and wordings would likely decrease. however, another way forward is to keep going in the same direction, stay the course, in other words. composition instructors and librarians join together and meld their methods organically. though this process may be messy, the results may better mirror the challenge for our students who have to navigate through the unfamiliar terrain of source evaluation in the information age. standardization may strip away our students’ creative edge needed to cut away ambiguity and fabrication of authority during a time when information flows as freely as air and can likely be as insubstantial. by bringing various languages of source evaluation together, our process becomes one of many methods, not the method but a method, available to students who need to learn to adapt to varying audiences and demands in order to evaluate work in meaningful ways, rather than blankly repeating vocabulary.   the endless flexibility in and between different academic disciplines challenges first-year students. when the language of rhetoric and composition and information literacy collide in the classroom, expect a crash in the students’ minds. they have to learn to adapt to multiple discourses, sets of words and principles of knowing, in a single classroom for each assignment. assuming a “fake-it-until-you-make-it” voice in their academic writing helps them gesture toward the clear and specific evaluations that we strive to teach our students. despite the limitations of our study, we feel as though we have stumbled upon an issue relevant to all librarians who communicate with students, composition instructors, and disciplinary faculty. understanding the role discourse plays in student learning should be embedded in our advocacy for information literacy.   references   adedibu, l.o. & ajala, i.o. (2011). recognition of library terms and concepts by undergraduate students. library philosophy and practice (e-journal). paper 449. retrieved from http://digitalcommons.unl.edu/libphilprac/449/   association of college & research libraries. (2015). framework for information literacy for higher education. chicago: american libraries association. retrieved from http://www.ala.org/acrl/standards/ilframework   ayre, c., smith, i.a., & cleeve, m. (2006). electronic library glossaries: jargonbusting essentials or wasted resource? the electronic librarian, 24(2), 126-134. http://dx.doi.org/10.1108/02640470610660323   barclay, d.a. & barclay, d.r. (1994). the role of freshmen writing in academic bibliographic instruction. the journal of academic librarianship, 20(4), 213-217. http://dx.doi.org/10.1016/0099-1333(94)90101-5   benjas-small, c., archer, a., tucker, k., vassady, l., & resor whicker, j. (2013). teaching web evaluation: a cognitive development approach. communications in information literacy, 7(1), 39-49. retrieved from http://www.comminfolit.org/index.php?journal=cil&page=article&op=view&path[]=v7i1p39   birmingham, e.j., chinwongs, l., flaspohler, m.r., hearn, c., kvanvig, d., portmann, r. (2008). first-year writing teachers, perceptions of students’ information literacy competencies, and a call for a collaborative approach. communications in information literacy, 2(1), 6-24. retrieved from http://www.comminfolit.org/index.php?journal=cil&page=article&op=viewarticle&path[]=spring2008ar1&path[]=67   bizup, j. (2008). beam: a rhetorical vocabulary for teaching research-based writing. rhetoric review, 27(1), 72-86. http://dx.doi.org/10.1080/07350190701738858   bowles-terry, m., davis, e., & holliday, w. (2010). “writing information literacy revisited”: application of theory to practice in the classroom. reference & user services quarterly, 49(3), 225-230.   burkholder, j.m.(2010). redefining sources as social acts: genre theory in information literacy instruction. library philosophy and practice, paper 413. retrieved from http://digitalcommons.unl.edu/libphilprac/413   carter, t.m. & aldridge, t. (2015). information life cycle. in p. bravender, h. mcclure, & g.schaub (eds.), teaching information literacy threshold concepts: lesson plans for librarians, 94-98. chicago: association of college and research libraries.   chaudhry, a. s. & choo, m. (2001). understanding of library jargon in the information seeking process. journal of information science, 27(5), 343-349.   dewey, b.i. (1999). in search of services: analyzing the findability of links on cic university libraries’ web pages. information technology and libraries, 18(4), 210-213. retrieved from http://www.ala.org/acrl/sites/ala.org.acrl/files/content/conferences/pdf/dewey99.pdf   doran, k. (1998). lamenting librarians’ linguistic lapses. american libraries, 29(7), 43.   fister, b. (1992). common ground: the composition/bibliographic instruction connection. in t. kirk (ed.), academic libraries: achieving excellence in higher education (pp. 154-158). chicago: association of college and research libraries.   foster, e.h. (2010). an examination of the effectiveness of an online library jargon glossary. (unpublished master’s thesis). university of north carolina, chapel hill, nc. retrieved from http://dc.lib.unc.edu/cdm/ref/collection/s_papers/id/1214   holliday, w. & rogers, j. (2013). talking about information literacy: the mediating role of discourse in a college writing classroom. libraries and the academy, 13(3), 257-271. http://dx.doi.org/ 10.1353/pla.2013.0025   houdyshell, m.l. (1998). bring out the best in your bi or converting confusion into confidence. college & undergraduate libraries, 5(1), 95-101. http://dx.doi.org/ 10.1300/j106v05n01_11   hutcherson, n. b. (2004). library jargon: student recognition of terms and concepts commonly used by librarians in the classroom 1. college and research libraries, 65(4), 349-354. http://dx.doi.org/ 10.5860/crl.65.4.349   hlavaty, g. & townsend, m. (2010). the library’s new relevance: fostering the first-year student’s acquisition, evaluation, and integration of print and electronic materials. teaching english in the two-year college, 38(2), 149-160. retrieved from http://www.ncte.org/library/nctefiles/resources/journals/tetyc/0382-dec2010/tetyc0382librarys.pdf   imler, b. & eichelberger, m. (2014). commercial database design vs. library terminology comprehension: why do students print abstracts instead of full-text articles? college & research libraries, 75(3), 284-297.  http://dx.doi.org/ 10.5860/crl12-426   jacobs, h. l. m., & jacobs, d. (2009). transforming the one-shot library session into pedagogical collaboration: information literacy and the english composition class. reference & user services quarterly, 49(1), 72-82. retrieved from http://www.jstor.org/stable/20865180   mazziotti, d. & grettano, t. (2011). “hanging together”: collaboration between information literacy and writing programs based on the acrl standards and wpa outcomes. in d.m. mueller (ed.), declaration of interdependence: the proceedings of the acrl 2011 conference, march 30-april 2, 2011, philadelphia, pa (pp. 180-190). chicago, il: association of college and research libraries. retrieved from http://www.ala.org/acrl/sites/ala.org.acrl/files/content/conferences/confsandpreconfs/national/2011/papers/hanging_together.pdf   meola, m. (2004). chucking the checklist: a contextual approach to teaching undergraduates web-site evaluation. libraries and the academy, 4(3), 331-344. http://dx.doi.org/10.1353/pla.2004.0055   mounce, m. (2009). academic librarian and english composition instructor collaboration: a selective annotated bibliography, 1998-2007. reference services review 37(1), 44-53. http://dx.doi.org/10.1108/00907320910934986   naismith, r., & stein, j. (1989). library jargon: student comprehension of technical language used by librarians. college and research libraries, 50(5), 543-552. http://dx.doi.org/ 10.5860/crl_50_05_543   norgaard, r. (2003). writing information literacy: contributions to a concept. reference & user services quarterly, 43(2), 124-130. retrieved from http://www.jstor.org/stable/20864155   norgaard, r. (2004). writing information literacy in the classroom: pedagogical enactments and implications. reference & user services quarterly, 43(3), 220-226. retrieved from http://www.jstor.org/stable/20864203   palsson, f., & mcdade, c. l. (2014). factors affecting the successful implementation of a common assignment for first-year composition information literacy. college & undergraduate libraries, 21(2), 193-209. http://dx.doi.org/10.1080/10691316.2013.829375   pinto, m., cordon, j. a., & gomez diaz, r.  (2010). thirty years of information literacy (1977-2007): a terminological, conceptual and statistical analysis. journal of librarianship and information science, 42(1), 3-19. http://dx.doi.org/ 10.1177/0961000609345091   ramage, j.d., bean, j.c., & johnson, j. (2010). writing arguments: a rhetoric with readings (custom ed. for auburn university). boston: pearson.   shields, k. (2014). research partners, teaching partners: a collaboration between fyc faculty and librarians to study students’ research and writing habits. internet reference services quarterly, 19(3-4), 207-218. http://dx.doi.org/10.1080/10875301.2014.983286    sonsteby, a. & dejonghe, j. (2013). usability testing, user-centered design, and libguides subject guides: a case study. journal of web librarianship, 7(1), 83-94. http://dx.doi.org/10.1080/19322909.2013.747366   spivey, m. a. (2000). the vocabulary of library home pages: an influence on diverse and remote end-users. information technology and libraries, 19(3), 151-156.   sult, l. & mills, v. (2006). a blended method for integrating information literacy instruction into english composition classes. reference services review, 34(3), 368-388. http://dx.doi.org/10.1108/00907320610685328   swanson, t.a. & green, j. (2011). why we are not google: lessons from a library web site usability study. the journal of academic librarianship, 37(3), 222-229. http://dx.doi.org/10.1016/j.acalib.2011.02.014   white-farnham, j. & caffrey gardner, c. (2014).crowdsourcing the curriculum: information literacy instruction in first-year writing. reference services review, 42(2), 277-292. http:dx.doi.org/10.1108/rsr-09-2013-0046   wu, s.m. (2008). investigating the effectiveness of arguments in undergraduate essays from an evaluation perspective. prospect, 23(3), 59-75. retrieved from http://www.ameprc.mq.edu.au/docs/prospect_journal/volume_23_no_3/23_3_art_5.pdf   wu, s.m. & allison, d. (2005). evaluative expressions in analytical arguments: aspects of appraisal in assigned english language essays. journal of applied linguistics, 2(1), 105-127. http://dx.doi.org/10.1558/japl.2005.2.1.105   microsoft word es_kloda.doc evidence based library and information practice 2006, 1:4      46 evidence based library and information practice       evidence summary    academic librarians should be sensitive to language and cultural barriers when  providing reference service to international students    a review of:  curry, ann and deborah copeman. “reference service to international students: a field  stimulation research study.” journal of academic librarianship 31.5 (sep. 2005): 409‐20.      reviewed by:  lorie a. kloda  instructional technology librarian  life sciences library  mcgill university  montreal, quebec, canada  e‐mail: lorie.kloda@mcgill.ca      received: 2 september 2006            accepted: 21 september 2006      © 2006 kloda. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to evaluate the quality of  reference service provided to non‐native,  english‐speaking international students in  academic libraries.    design – field stimulation (unobtrusive  testing).    setting – eleven college and university  libraries in the lower mainland of british  columbia, canada, in the fall of 2003.    subjects – library staff offering reference  service at one of the participating libraries.    methods – the study utilized field  stimulation, whereby an individual, or  “proxy,” posed as a library user and  initiated a reference encounter with library  staff at each institution. in each case the  proxy asked the same question to the library  staff member. after the interaction was  completed the proxy recorded all observed  behaviours. data were collected using a  checklist of actions; a narrative record  written by the proxy; and several evaluative  questions. each library was visited by the  same proxy on two separate occasions for a  total of 22 visits, of which 20 instances  resulted in usable data. the narrative  recordings of the reference encounters were  analyzed using an open coding process.  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:4      47 main results – in 75% of the cases, the proxy  was “‘satisfied” or “very satisfied” with  help received from the library staff member  and was “likely to” or “definitely would  return to the staff member” in the future.  the reference encounters lasted between a  few minutes to half an hour in length, with  most lasting between 5 and 15 minutes.  encounters that were brief (less than 5  minutes) resulted in an evaluation of  “dissatisfied” or “very dissatisfied” and  “not likely to” or “definitely would not  return.” encounters where the library staff  member extended an invitation to the proxy  to return in the future were all rated with  “high satisfaction” and “willingness to  return.” the following reference service  actions were observed in at least half of the  encounters:  • asked questions for clarification (20)  • avoided overwhelming the user  with information (19)  • provided instruction on how to use  information sources (18)  • explained what he / she was doing  at every stage (17)  • demonstrated awareness of  language barriers and modified his /  her behaviour accordingly (16)  • had a respectful attitude toward the  user and her question (16)  • looked approachable (15)  • used library jargon (12)    the remaining reference service actions  from the checklist were observed in less  than half the encounters:  • accompanied the user to  information sources (9)  • invited the user to return if she  needed more help (6)  • asked the user if she had found  what she needed (4)  • referred the user to someone else (2)    the data collected from the narrative  recordings of the reference encounters  resulted in the identification of 17 themes.  most of these themes corresponded with the  literature reviewed as important qualities  for positive reference interactions. nine of  these themes were found to correlate with  the proxy’s positive evaluation of the  reference encounter: approachability,  awareness of language barriers, asking  questions, rephrasing, explanation, library  jargon, instruction, early termination of  interview, patience, and follow up.    conclusion – this preliminary study  documents the actions of reference staff in  academic libraries when answering a  question from an international student. the  researchers found a relationship between  some library staff behaviours and the user’s  level of satisfaction and likelihood to return  to the staff member in the future. the  research suggests that reference staff pay  special attention to the needs of non‐native  english speakers in order to provide a  positive reference encounter.    commentary    this study looks at reference service  provided to a distinct but significant portion  of the academic population—international  students. as the authors point out, these  students originate from all over the world,  and in many cases are unfamiliar with  north american or western culture. many  international students speak english as a  second or even third language, and speak it  with a heavy accent. these students are at a  disadvantage compared to native english  speakers not only due to the language  barrier, but to the anxiety that can arise in  anticipation of the reference encounter.  library staff, who are already trained in  proper reference interview techniques, need  to pay special attention to the unique needs  of this population.     the field stimulation method employed for  this study was appropriate due to the nature  of the information sought. engaging a proxy  evidence based library and information practice 2006, 1:4      48 with heavily accented english to pose as a  student and approach library staff in their  natural setting is an ideal method to gather  data to answer questions about staff  behaviour. the researchers note the  advantage of using a single proxy who is  also an mlis student for maintaining  consistency. in addition, the mlis student  posing as a “regular” student was likely  better able to observe desirable actions and  was less anxious about the encounter. the  researchers also note the possibility of bias  as the proxy’s knowledge of library jargon,  as well as her expectations of the reference  interview may have influenced both her  conduct and her interpretation of the  encounters. another issue could be the  limiting nature of having a single individual  represent the so‐called “international  student.” the diversity of library users  cannot be represented by a single user and  thus the encounters observed cannot be  generalized to all international students. an  assurance of the validity of the data  collected from the narrative recordings is  their similarity to the literature on the topic.     one drawback of the study design was the  inability to verify the status of the library  staff being observed. it was assumed that  these individuals were reference librarians,  although the authors acknowledge the  possibility that these individuals could have  been support staff. it is important to keep in  mind that whatever their status, the library  staff observed in the study were in fact  responsible for reference service at their  respective institutions. another drawback of  observing study subjects in their natural  setting, mentioned by the researchers, is the  difficulty of replicating the study.                this preliminary study was conducted in a  rather specific location on the west coast of  canada, an ideal setting due to its diverse  population of students from many countries.  due to the sampling method, these results  cannot be generalized to all international  students in british columbia, let alone  north america. yet the results are of great  interest to all academic librarians, and other  information professionals working in  multicultural environments. this research  reinforces the importance of a good  reference interview and proper  communication skills, and it highlights the  actions or behaviours that are most  important when interacting with an  international student.    evidence based library and information practice evidence based library and information practice 2010, 5.4 138 evidence based library and information practice news/announcements call for proposals – 3rd qualitative and quantitative methods in libraries international conference this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. the 3rd qualitative and quantitative methods in libraries international conference (qqml2011) will take place may 24-27, 2011, in athens, greece. since 2009, qqml has provided an excellent framework for the presentation of new trends and developments in every aspect of library and information science technology, applications and research. papers are invited for this international conference. key themes include, but are not limited to, the following: • management • financial strength and sustainability • marketing • communication strategies • data analysis and data mining • digital libraries special sessions & workshops you may send proposals for special sessions (4-6 papers) or workshops (more than 2 sessions), including the title and a brief description, to the conference committee at: secretariat@isast.org or via the electronic submission page at: http://www.isast.org/abstractpaperregister.html abstract/paper submissions you may submit proposals for contributed abstracts/papers via the electronic submission page at http://www.isast.org/abstractpaperregister.html contributions could include one of the following: • structured abstracts (not exceeding 500 words) and presentation; • full papers (not exceeding 7,000 words); and • posters (not exceeding 2,500 words). in all cases, at least one of the authors is required to be registered for the conference. abstracts and full papers should be submitted electronically according to the timetable available at: http://www.isast.org/importantdates.html abstracts and full papers should follow the author guidelines available at: http://www.isast.org/abstractpaperregister.html all abstracts will be published in the conference book of abstracts and on the conference website. papers will be published on the conference website, with the permission of the author(s). evidence based library and information practice 2010, 5.4 139 student submissions the conference also offers postgraduate students and phd candidates an opportunity to participate in a paper competition. students who submit abstracts to the conference can participate, but can only submit only one paper to the competition. students who enter the competition must register in the conference and be in attendance. the papers will be judged by a scientific committee according to the significance of the research theme, the research methodology, the description of the results and the organization of the presentation. professors and supervisors are encouraged to organize conference sessions of postgraduate theses and dissertations. please direct any questions regarding the qqml 2011 conference and student research competition to anthi katsirikou at anthi@asmda.com or to the secretary of the conference at secretariat@isast.org. / evidence based library and information practice evidence summary   the impact of social responsibility and organizational accountability on the performance of public librarians in iran   a review of: soltani-nejad, n., jahanshahi, m., karim saberi, m., ansari, n., & zarei-maram, n. (2022). the relationship between social responsibility and public libraries accountability: the mediating role of professional ethics and conscientiousness. journal of librarianship and information science, 54(2), 306–324. https://doi.org/10.1177/09610006211014260   reviewed by: andrea miller-nesbitt associate librarian schulich library of physical sciences, life sciences, and engineering mcgill university library montreal, quebec, canada email: andrea.miller-nesbitt@mcgill.ca   received: 1 mar. 2023                                                                    accepted:  28 mar. 2023      2023 miller-nesbitt. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30338     abstract   objective – to determine how librarians' perceptions of public libraries' social responsibility and accountability within society affect their professional ethics and conscientiousness.   design – quantitative, survey questionnaire.   setting – public libraries in iran.                 subjects – survey respondents (n=362) were public librarians* (see note below).   methods – the authors created a theoretical model based on six separate hypotheses, describing the relationship between the following variables: corporate social responsibility, organizational accountability, professional ethics, and conscientiousness. a questionnaire was distributed to the participants. spss 21.0 was used for the analysis of demographic data and smart pls 3.0 was used to assess the theoretical model.   main results – the results show a significant, positive, and direct relationship between the variables being studied (corporate social responsibility, organizational accountability, professional ethics, and conscientiousness), therefore confirming the relevance of the authors’ theoretical model.   conclusion – the results of this study demonstrate the importance of promoting the social responsibility and organizational accountability efforts of public libraries. the data suggest that doing so will strengthen the positive perception of the library amongst employees, which will in turn have a positive effect on their professional ethics and conscientiousness. the authors suggest that library managers need to create a culture of accountability and ethics within libraries. they can do so by incorporating ethics and social responsibility in decision-making and policies. additionally, the authors propose that professional ethics training in library curricula and continuing education would provide librarians with the knowledge necessary when encountering ethical dilemmas on the job.   commentary   there is a plethora of literature discussing the four main variables considered in this study: corporate social responsibility, organizational accountability, professional ethics, and conscientiousness. much of the literature comes from business and management, but these concepts are well established within the library and information science (lis) literature as well (esdaile, 1933). in fact, social responsibility and professional ethics are described as core tenets of librarianship by professional associations such as the american library association and the international federation of library associations (ala council, 2019; garcia-febo et al., 2012). this study is the first to consider the relationships between all four variables together.   this study was appraised using the evidence based librarianship critical appraisal checklist developed by glynn (2006). key details regarding the population and aspects of the data collection are missing. for example, it is unclear how the participants were selected, how the questionnaire was distributed and in what format and language, what the response rate was, and whether informed consent was received from participants.   the authors adopted four separate questionnaires from previously published literature and combined them to create the survey tool used in this study. the authors state that they chose to modify questions from previously published research in order to ensure the validity of the survey tool. these questionnaires were not from lis literature and, although an attempt was made to modify the questions to fit the content of this research, several of the questions do not lend themselves to the context of public libraries.   the strength of this study is the rigorous data analysis. the authors used structural equation modeling (sem) to test their theoretical model. they provide very detailed statistical analysis of the relationship between the four main variables.   the authors are cognizant of the limitations of the study, notably how cultural and societal norms in iran likely shape respondents’ perceptions of social responsibility and accountability. areas of further research proposed include conducting a similar study in other parts of the world or in other types of libraries (e.g., academic or special libraries), and investigating the relationship between the variables using a qualitative methodology.   public libraries play an important role within democratic societies. results from this study are particularly important for library managers and administrators to consider. managers need to find meaningful ways to strengthen the positive perception of the library by advancing the library’s commitment to social responsibility and improving organizational accountability. doing so will enhance librarians’ commitment to their work and improve their job performance and inner satisfaction, which will in turn benefit the communities they serve.    *note: the authors refer to the participants as “librarians,” “library employees,” and “library staff” at various points in the study. according to the demographic breakdown, less than 40% of respondents have a master's degree or higher. i do not know what the requirements are to be a librarian in iran, but i have also used the term “librarian” in this evidence summary for the sake of consistency.   references   american library association council. (2019, january). core values of librarianship. the american library association. https://www.ala.org/advocacy/intfreedom/corevalues   esdaile, a. (1933). the social responsibility of the modern library. bulletin of the american library association, 27(13), 572–576. https://www.jstor.org/stable/25688031   garcia-febo, l., hustad, a., rösch, h., sturges, p., & vallotton, a. (2012). ifla code of ethics for librarians and other information workers. international federation of library associations and institutions. https://www.ifla.org/publications/ifla-code-of-ethics-for-librarians-and-other-information-workers-full-version/   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387–399. https://doi.org/10.1108/07378830610692154   soltani-nejad, n., jahanshahi, m., karim saberi, m., ansari, n., & zarei-maram, n. (2022). the relationship between social responsibility and public libraries accountability: the mediating role of professional ethics and conscientiousness. journal of librarianship and information science, 54(2), 306–324. https://doi.org/10.1177/09610006211014260     conference paper   an evidence based methodology to facilitate public library non-fiction collection development   matthew kelly phd candidate department of information studies, curtin university perth, western australia, australia email: mattkelly.curtin@gmail.com   received: 18 aug. 2015  accepted: 30 oct. 2015         2015 kelly. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – this research was designed as a pilot study to test a methodology for subject based collection analysis for public libraries.   methods – worldcat collection data from eight australian public libraries was extracted using the collection evaluation application. the data was aggregated and filtered to assess how the sample’s titles could be compared against the oclc conspectus subject categories. a hierarchy of emphasis emerged and this was divided into tiers ranging from <0.1% of the sample to >1% of the sample. these tiers were further analysed to quantify their representativeness against both the sample’s titles and the subject categories taken as a whole. the interpretive aspect of the study sought to understand the types of knowledge embedded in the tiers and was underpinned by hermeneutic phenomenology.   results – the study revealed that there was a marked tendency for a small percentage of subject categories to constitute a large proportion of the potential topicality that might have been represented in these types of collections. the study also found that distribution of the aggregated collection conformed to a power law distribution (80/20) so that approximately 80% of the collection was represented by 20% of the subject categories. the study also found that there were significant commonalities in the types of subject categories that were found in the designated tiers and that it may be possible to develop ontologies that correspond to the collection tiers.   conclusions – the evidence-based methodology developed in this pilot study has the potential for further development to help to improve the practice of collection development. the introduction of the concept of the epistemic role played by collection tiers is a promising aid to inform our understanding of knowledge organization for public libraries. the research shows a way forward to help to link subjective decision making with a scientifically based approach to managing knowledge resources. introduction   there remains in the broader information management specialization known as collection development, a tension between advocates of the traditional view of collection development as an art, and those who are interested in how scientific methods can be adapted to get the right information to users. put simply, the question would be, “can any bibliometric method provide the basis by which we sublimate the axiological values that underpin topical choice and subject representation?” is there a method of collection analysis that allows us to understand what the general selection choices in public libraries look like, why they look the way they do and what should they look like? this study was formulated to help to answer these questions at the level of the non-fiction collection in the public library setting where the need for a wide range of potential topicality is, arguably, at its broadest (public libraries having to meet the knowledge needs of all sectors of civil society).   literature review   identifying what material deserves a place in a public library's collection has been debated for many decades, as has the process that allows for evaluation of a collection in order for it to both maximize its usefulness to a cohort of users, and also, to determine how it stands as a set of documents that best represent the viable knowledge on a topic (agee, 2005; wilson, 1968). the difficulty in framing a method that is sufficiently objective to receive general endorsement, given the inherently subjective nature of collection evaluation (evans, 2000) can be linked to attempts to understand the relationships between various branches of knowledge. the search for a well-reasoned approach to objectivity that explicates its source in intersubjectivity (alexander, 2012), rather than in an ideal state, can aid in this goal which also has found significant expression in the field known as the sociology of knowledge (berger & luckman, 1971; bernstein, 1983; hekman, 1986; mannheim, 1972; scheler, 1980; stark, 1967).   we are seeking a system that can: 1) “guide the systematic selection of the world's recorded knowledge...according to a rationale founded upon priorities that have been identified to serve the community most effectively” (osburn, 1979, p. 10) and, 2) incorporate the “dynamism inherent in the interactions and potential interactions of the community and the information universe via collection management” (osburn, 2005, p. 10). questions relating to how consideration is given to those domains that find either minimal or no representation in collections have primarily been approached from a standpoint involving checking collections against bibliographies which were thought to reveal what libraries should own. such an approach could not reveal, however, items within a collection that should not have been included.   elzy and lancaster (1990) identified an innovative means by which the reciprocal or interdependent relationship between bibliographies and collections might be checked to determine measures of complementarity and quality. evaluating materials within collections, based on ranking data, and hence audience levels (in this context “audience levels” means that high ranking across compared libraries largely indicates a popular work in the sense of it not having scholarly content — the inference being that scholarly content largely belongs in scholarly libraries), emerged from white's brief tests of collection strength (1995). white's approach compared short lists of items to library holdings (the so-called “brief tests”) and included research libraries group conspectus levels as part of how assessment was conducted. white's approach enabled collection level descriptions to be established quickly without the need for either extensive checks of bibliographies or the assumption of subject knowledge (lesniaski, 2004). the “brief test” method was followed by an elaboration of the original method, the “coverage power test” remedy — which aimed to shore up a number of perceived shortcomings. this method involved testing a collection against an absolute scale of holdings counts (all of worldcat’s holdings) rather than the earlier iteration (a bibliography composed by an expert) (white, 2008). white's methods were tested and determined to be efficacious by twiss (2001), lesniaski (2004), bernstein (2006), beals and gilmour (2007), and mcminn (2010). other scholars have attempted to use worldcat’s collection analysis application to look to better understand collections with various levels of efficacy (genoni & wright, 2010; lavoie, connaway & o’neill, 2007; monroe-gulick & currie, 2011; o’neil, connaway & dickey, 2008; jensen, 2012; perrault, 2004).   the standing committee of the international federation of library associations and institutions’ (ifla) acquisition and collection development section (2001) noted how the process of outlining a collection policy relating to subject breadth and depth contributes to reducing personal bias and reducing gaps in a collection (pp. 2-6). they advocated an approach using the oclc conspectus to aid evaluation that can contribute to libraries holding a more extensive range of subjects or a deeper coverage of those subjects. by approaching subject range and depth as not only capable of being assessed qualitatively, but also as a desirable precursor to answering questions relating to how subjective and objective approaches to knowledge domains and subject representation are contextualized within civil society settings, we begin to develop a more resilient (social) epistemological basis for the model of knowledge that we choose to promote in public libraries (budd, 2001; egan & shera, 1952; fallis, 2006). matthews and stephens (2010) describe this in a general sense as “the optimization of systems of knowledge acquisition through an appreciation of social strategies and motivations” (p. 541). such an approach is also present in capurro’s (1992) information hermeneutics that looks to move beyond the simple question of what is the best way to promote collection development and ask more fundamental questions such as what is collection development for, and moreover, how might it serve the interests of civil society in the context of public libraries?   aims   this research was designed as a pilot study to test the methodology for subject-based collection analysis that will help:   to determine how subjects in adult non-fiction monograph collections in australian  public libraries are distributed; to describe the commonalities in distribution that reveal subject priorities or subject gaps; to assess how subjects in adult non-fiction monograph collections in public libraries are distributed in terms of a weighting that indicates range and depth of coverage; to identify subjects that have measurable priority or omission, and to assess if this is associated with the range of materials available for acquisition or if the epistemic values of selectors drive an imbalance in collections.   by grounding the broader research in these specific factors, a further qualitatively-based aspect of the research will, it is expected, be better placed at a later date to determine the criteria that selectors bring to bear on their selection and evaluation decisions for non-fiction monograph collections in public libraries. this later aim is to examine what selectors consider to be “core knowledge” as well as what knowledge represented in subject domains is considered to be crucial to meeting the educational, informational and recreational needs of public library users.   methods   from the cohort of 31 municipal public library services in australia that agreed to take part in the ongoing research project, 8 libraries were selected to take part, based on the similarity of the level of their reported collection holdings in worldcat.  a survey was conducted in 2014 that totaled the eight libraries’ combined holdings in worldcat at over 2.2 million items. the worldcat holdings data was extracted using oclc’s proprietary collection evaluation application; this application was accessed online.   the collection evaluation application requires comparisons to be made against the holdings of an “anchor library” (in a practitioner setting this is the practitioner’s own library). in this case, the data from the “anchor library” — the first library which agreed to participate —was extracted first. this data was not included in the study and is referenced here to illustrate part of the process of working with collection evaluation as a research tool. following this a “one-to-many” analysis was conducted with 8 other libraries’ collections and then further refined by using the “benchmarking” filter on the collection evaluation website which was set to show that the titles to be delivered in the ftp transfer were “not held” by the “anchor library.”   the data was also pre-filtered in the collection evaluation website for print and e-books. the spreadsheet file was downloaded through the oclc ftp server and the print/e-book filter was applied, resulting in 1,557,380 items held in worldcat in june 2015. of these, 1,023,453 were unique titles; these formed the basis of this study. the ratio of unique to shared titles was 70:30 for the initial pre-active-filtered data set downloaded from worldcat with 306,663 items/titles shared by 1 or more libraries and 254 items/titles shared by all. further spreadsheet filtering took place to ensure that only adult, non-fiction print or e-books were represented. this process included deleting 187,934 juvenile titles, 139,112 non-english language titles, 140,616 audiovisual titles, 34,981 non-book titles, 241,062 fiction titles, 205,607 items described as unknown classification, and 9,025 titles whose subject categories crossed over between foreign language and literature. where foreign language was specifically dealt with in the subject category, the item was included as it was likely not to be a fictional work; the aim was to exclude foreign language fiction classified within subject categories dealing with “language and literature.” it was necessary to actively search in other data designations to ensure that the desired sample was as accurate as possible. for example, filtering english-language-only did not automatically remove all non-english works, and filtering for print and e-books did not eliminate all other formats in the initial file download. paring each of these qualifiers down to achieve the desired set resulted in 334,544 titles (21.48 % of the data provided in the output file by collection evaluation as print/e-book format). the instances of “subject category” from this data set were then transferred to another spreadsheet. the individual titles data was no longer of any use at this point and was retired along with all other criteria that had accompanied “subject category” in the file download.   the subject categories were sorted to create a hierarchy and the 334,544 titles were tabulated in the 437 subject categories that emerged from the sample. the sample was divided into five tiers:   ·         1% of the sample ·         0.5% 1% of the sample ·         0.25% 0.5% of the sample ·         0.1% 0.25% of the sample ·         < 0.1% of the sample   the tiers were then interrogated for their relationship to three factors:   ·         number of subject categories in the tier ·         percentage of the sample titles in the tier ·         number of subjects in the tier as a percentage of total subject categories in the sample   an interpretive phenomenological method (hermeneutic phenomenology) was introduced to assess the subject agglomerations (or clusters) that were evident in the tiers to ascertain, in a preliminary way, what the sample might reveal about the selection decisions made by the collection development librarians who had created this combined collection. hermeneutic phenomenology provides a well-defined ontological grounding in how we conceptualize the nature of the knowledge that an information expert (such as a collection developer in a public library) might be called upon to deploy and to engage — in a dialectical sense — with the community of users and the community of knowledge creators (benediktsson, 1989; bruce, 1999; budd, 2005; capurro, 1992; hansson, 2005;  savolainen, 2008; suorsa, 2015; suorsa & huotari, 2014; vanscoy & evenstad, 2015).   fundamental to this approach to knowing is an acknowledgment that human subjects are engaged, already and always, in a process of creation and co-creation of the knowledge environments in which they are cast. budd (2008) outlines how this hermeneutic approach “introduces the realization that knowledge, information and searching are not solitary acts, but are undertaken in a communicative relationship with another creator of knowledge and information. information seeking and retrieval is, in short, dialogical” (p. 91). in this study, an attempt was made to use the results of the subject category structure to look to how civil society, through the public library, structures knowledge organization. in doing so, the approach sought a way to invoke what rorty (1979) calls the “kantian notion of philosophy as metacriticism of the special disciplines” (p. 166) and to join this with an equally kantian approach which rehabilitates “intuitions and concepts” as tools with which we can rework not only a theory of knowledge, as rorty hopes to do, but a theory of its organization. with these factors in mind, and with reference to the emergent knowledge organization tradition, and especially svenonius’s (2004) explication of varieties of theories of meaning in this context, an amalgam of the operational, referential and instrumental approaches was sought in an attempt to graft an epistemological framework on to the sample results so as to uncover something of the design of knowledge representations in the setting of civil society library collections.   results   the ranking of subject categories, their percentage of the total sample and their inclusion in one of the five tiers referred to above is outlined in appendix a.   taking the results as providing a symptomatic and indicative (rather than conclusive) referencing of the state of adult non-fiction collections in the australian public library sector, it seems reasonable to acknowledge that there is a tendency for widely divergent level of emphasis on subject categories in the civil society knowledge context within which the public library operates. what this means is that a small number of subjects that are statistically insignificant when regarded against the entire matrix of possible subject categories contribute an inordinate amount of material to the libraries they serve.   the study found that 0.9% of possible subjects contribute 16% of the collection’s titles in the area and format (adult non-fiction books and e-books) under investigation. assuming that the pilot study does reflect the generalized state of the sector, we should ask — and investigate — why this is not an acknowledged problem for librarians. the breakdown of how identified segments constitute the collection can be further schematized across tiers where we can see the bibliometric relationship of subject categories’ share of sample to the proportion of titles (table 1). we see here that the most numerous 10% of subject categories in the sample give up 60% of the titles; the most numerous 20% of subject categories in the sample provide 77% of the titles, the most numerous 40% of subject categories in the sample account for 92% of titles, while the balance of 60% of subject categories in the sample provide only 8% of the titles. the tier weighting remedy in table 1 is explained below.   this study indicated that there is a strong tendency for a limited number of subject categories to represent the varieties of knowledge considered suitable for civil society settings. the relationship of category to title holdings in this study showed a strong correlation to what is described in a statistical sense as a power law or a pareto distribution. bradford’s law of scattering is a representation of a similar statistical distribution. to date, there has been no specifically collection-oriented identification of a bradford-style power law at work in library monograph collections (although a significant body of work exists on serials collection management with reference to bradford’s law). the notion that this type of distribution is to be expected in a range of information environments is a common theme in early bibliometric (buckland & hindle, 1969; fairthorne, 1969; drott, 1981) and scientometric commentary (de solla price, 1976). in figure 1, the heavy weighting of the top 20% of subject categories is evident.     table 1 share of categories vs proportion of titles & tier weighting remedy subject categories’ share of sample   proportion of titles proportion of titles after tier weighting applied     result most numerous 10% of subject categories in sample   60%   44 %   tiers 1 and 2 reduced by 16%   most numerous 20% of subject categories in sample   77%   64%   tiers 1,2 and 3 reduced by 13%   most numerous 40% of subject categories in sample   92%   82%   tiers 1,2,3 and 4 reduced by 10%   balance of 60% of subject categories in sample   8%   18%   tier 5 increased by 10%     figure 1 title numbers by subject category.     once the structural nature of the collection is identified, it becomes possible to locate how a non-bibliometric selection technique creates significant imbalances in a collection’s focus. in order to rebalance a collection, it is necessary to look to the best available method to ensure that the most representative collection, in terms of subject categories relevant to civil society users, is instantiated through bibliometric planning. a preliminary attempt was made to formulate such a methodology so as to flatten the distribution through a weighting technique.  this involved:   ·         multiplying the top subject category for tier 1 by 0.25 and transposing the result on the remainder of tier 1 subject categories (the top 22 categories are reduced substantially while the bottom 3 categories increase marginally); ·         multiplying tiers 2 and 3 by 1.2 and transposing the result on the remainder of tiers 2 and 3 subject categories (a simple 20% increase); ·         multiplying tier 4 by 1.25 and transposing the result on the remainder of tier 4 subject categories (a simple 25% increase); ·         creating an artificially homogenous tier 5 through multiplying the first number of the tier (the subject category ranked number 173 in the sample) by 0.7, thus creating approximately a 30% differentiation from tier 4. this differentiation, which was not evident in the sample (the last subject category of tier 4 and first subject category of tier 5 were numerically separated by only 9 titles) allowed all tier 5 categories to move from a range of statistical significance that in the sample is in the range 2.98914e-06 to 0.09% to a constant 0.07% (42 tier 5 subject categories would lose titles while 224 would gain titles).   the visual representation of this process can be seen in figure 2.   so while the changes that such a weighting approach might make are potentially significant to the makeup of a collection in tiers 1 and 5, they are only relatively minor for the collection as a whole when conceptualized as a grouping of tiers. the general shape of the collection, with the exception of an activity that might profitably be described as “capping” (and which refers to the limiting of tier 1 subject categories to one percent of the collection), remains essentially the same. what this approach promises is the ability to plan for collection development by identifying a percentage-based increase or decrease for each subject category based on its location within a particular tier of the collection (see table 1).   the point here is not that particular tiers should have a nominated reduction or increase but that the analysis can be done so as to effect a more balanced collection and that the application of a tier-weighted approach is likely to ensure that the collection — assuming it is reasonably balanced — is able to be worked on to help to ensure that the broader domains that the tiers represent are not disturbed. there is, as yet, no ontological assumption built into the model which would see changes directed toward the themes or domains that each tier might represent. the tier-weighted approach makes the assumption that the there are levels of tolerance that exist within each of the tiers, such that the addition or deletion of an entire subject category (and its commensurate level of titles holdings) would not substantially affect how a collection delivered the broader information domain.   working with the idea that a tier-based breakdown might reveal a significant bibliometric relationship between types of knowledge in the civil society context, the sample revealed the following data regarding how collection tiers were constituted (table 2).     figure 2 modified title numbers by subject category.     table 2 collection tiers and relationships to title and subject categories collection tier subject   categories percentage of                       total titles percentage of total subject categories percentage   of  total collection per subject by tier (mean) tier 1: the self: home and family 25 48.15% 5.72% 1.92% tier 2: outside of the self: the civilized mind 17 12.09% 3.89% 0.71% tier 3: onward the enlightenment: specialized science, history and culture 58 20.50% 13.27% 0.35% tier 4: democratizing knowledge: the world of generalities 73 11.33% 16.71%   0.15%   tier 5: deep natural and social science: the borders of academic knowledge 264 7.93% 60.41% 0.03%     discussion   as “understanding always involves understanding from within a framework which makes sense for us” and that learning from the past involves a dialectical engagement with it through “posing questions to the past in light of our conceptual preoccupations in the present” (benhabib, 1986, p. xi) we should not be surprised at the difficulty of aligning a collection to meet universal, worldly and pragmatic requirements. such an analysis is cognizant of, but in no way driven by, the current needs of users. it can also never be more than the sum of knowledge aggregated by the non-fiction publishing industry over a given period of time.   this research takes heed of the need for its own foundation to be ultimately grounded, in a comparative sense, with the types of collection profiles that collecting libraries maintain. by comparing and differentiating “collect everything collections” and “circulating collections,” it is possible to ensure that, where subject category priority can be identified in the latter, for instance in the practical arts of domestic life or the generalizable narrative of history, then these domains are more specifically articulated as knowledge that defines — perhaps more pertinently than other domains — the types of knowledge that civil society demands and deserves in its libraries.   but it also should be said that while the civil society library is to some extent a creature of its times, it also has an educative mission that should reject the relativist position that all knowledge is equally as valid and that no knowledge can deserve to be maintained in situ. while these questions are beyond the scope of this paper, it is worthwhile acknowledging that the types of knowledge that we do maintain in civil society libraries reflect the epistemic priorities that we set. such priorities are, surprisingly, rarely interrogated for what they represent about our critical or hermeneutic assumptions.   as this specifically epistemic concern was foundational to this research (in part, the research was designed to seek preliminary answers to such questions), it is worth acknowledging this briefly, so as to provide both proper context for the work conducted on the eight library collections and for a report on the findings to date for interested scholars and practitioners. this will also assist in explaining how the tiers received their thematic designations outlined in table 2.   a process of investigation was conducted, using a hermeneutically-based phenomenological method. at its core such a method is a philosophically-oriented approach which seeks to “…acquire the essence of the research process as this is opened up in the philosophical literature... [the researcher seeks to] attune themselves towards the ontological nature of phenomenon while learning to “see” pre-reflective, taken-for-granted, and essential understandings through the lens of [existing] pre-understandings and prejudices.” (kafle, 2011, p.188)   a project to outline the significant elements of the researcher’s pre-understandings in the area of public library collection development was undertaken and the results presented and published across a range of fora (kelly, 2014, 2015a, 2015b, 2015c). in so doing, the prejudices that the researcher brought to the project relating to the theory and practice of collection development were made explicit to better facilitate dispassionate engagement with the qualitative issues at the core of the inquiry. analysis commenced, using the following hermeneutic and phenomenological approaches:   the hermeneutic circle of “reading, reflective writing and interpretation” (kafle, 2011, p. 195) within and around information science and philosophy; a type of engagement that is extended, temporal and oriented toward development of provisional documents (the quantitative data delivered by oclc’s worldcat union catalog) and how meaning develops for both the researcher and research participant (which in this case included the collection developers whose collection and epistemic choices were central to the study); the nature of the rhetorical basis of types of language use pertinent to the theme investigated so as to link the subject categories together in a meaningful way.   with this framework as the basis for the qualitative aspect of the research, the subject categories that emerged from the bibliometric inquiry were assessed to attempt to link them together in a common theme.    commencing the analysis with the most popular and continuing through to the least popular, a standout theme emerged for the tier 1 results (>1% of subject categories and 25 out of 437 subject categories). this theme was identified as home, family and self. since cooking, sports, arts and crafts, family, sexuality, gardening, psychology and local history are all prominent, there should be no real surprise at the assignation. more than a quarter of all books and e-books, according to this study, involved the following 10 subject categories: domestic engineering (4.62%), sports (3.02%), history— oceania, south seas (mainly australian history) (2.91%), handicrafts, arts & crafts (2.81%), decorative arts, applied arts (2.61%), history, general (2.40%), family, marriage, women, sexual life (2.33%), history—great britain (2.25%), plant culture (2.18%), individual psychology (1.88%).it is argued here that seven of these ten subject categories fall within the ambit of the householder who has an interest in improving the quality of their own life and that of their immediate family, whether in terms of recipes for meals, maintaining the home, engaging in a craft-oriented hobby or gardening. this also links with the importance of sport to households as a recreational, social, activity. the psychology of family life is prevalent here as well. two of the three history subject categories can be considered parochial in nature. taken together, these 25 subjects comprised 48% of the sample and it is not unreasonable to start to build a picture, albeit it a speculative and preliminary one at this stage, of how 6% of the possible topicality equates to nearly half of the sampled collection. this leads to the question, “is this all that civil society cohorts are interested in reading?” or is there a more or less unstated assumption by librarians that they should be selecting very heavily in the home, family, self space?   the interpretive label outside of the self: the civilized mind was chosen for tier 2. the term “the civilized mind” is not meant to carry any baggage but is used in the same way that raymond williams uses it to refer to “an achieved state or condition of organized social life” (1976, p. 57). this tier constituted a grouping of topical interests that demonstrated a tendency towards inquiry into matters that were less likely to be easily linked just with the world of home and family. connections could, however, be made. while the importance of motion pictures and the theater could be linked to the notion of entertainment, they are a specific type of skilled entertainment that does not generally link with the orientation of the hobbyist that links so many of the tier 1 topics (in tier 2 games and amusements might be emended to this tier 1 group). english philology and language, along with literature —collections, covered a wide range of literary technique and anecdote. the presence of the so-called occult sciences and of psychiatry within this list abutting one another was a serendipitous aid in orienting the tier 2 topicality and showed similar concerns for the mind conceived of beyond the normative realm. similar levels of title holdings were evident for social work/social and public welfare and for therapeutics and pharmacology, which were also present in this tier. geography and the four separate subject categories of history of africa, history of the middle east, history of italy, history of france (ancient history has no conspectus category) were a prominent grouping. the separate subject categories of genealogy and biography can be reasonably linked with the notion that readers (and selectors) want to be able to discover the self through engagement with the selves of others. the subject category practical theology while effectively only dealing with christianity, deals with its social articulation and practice (which is the notable thematic expression that the research identifies in this tier). without wishing to psychologize the process, the tier 2 group, in contrast to the tier 1 group, might reasonably be said to deal with topicality that touches on the enlightenment movement into disciplinary knowledge and “the civilized mind.”  while it might surprise some, it might well be argued on the basis of this research that this process of moving the locus of the major considerations of civil society knowledge away from the home and into an “open world” is still in the process of development, even in western countries.   the description of the general domain evident in tier 3 takes the enlightenment metaphor developed above a stage further. designating this category “onward the enlightenment” in reference to the emergent themes in the tier 2 domain, this tier was further qualified as “specialized science, history and culture.” it seemed to deal with specialized knowledge and, with a handful of exceptions, does not touch upon the topicality of home, family, health, spirituality, hobbies, customs and personal/spiritual matters. tier 4 was designated “democratizing knowledge” and is identified as dealing with “the world of generalities.” while most can be identified as having a humanistic or social scientific base with the balance comprising natural sciences or technical/applied sciences, further delineation has not been attempted. the final tier, tier 5, was designated “deep natural and social science” in recognition that a majority of the subject categories might be seen as quite reasonably likely to match an identifiable specialization in natural or social science. the tier was further qualified as “the borders of academic knowledge” in recognition that the titles included in these subject categories might reasonably be expected to comprise not only introductory works dealing with these fields but also works that assume significant foundational knowledge to be of use to a reader. not all subject categories fitted this description and the delineation between tier 4 as specialized and tier 5 as deep knowledge is somewhat arbitrary. with further refinement in methods for sorting the large data sets it is expected that the “long tail” — that is, the tier 4 and 5 set of subject categories — may reveal more about how individual libraries select for this type of deeper or specialized knowledge that constituted 73% of subject categories but only 18.5% of the titles.   conclusion   just as the methodology outlined here offers promise to improve the practice of collection development, it also provides a starting point for assessing how the epistemic role of collection tiers can inform our understanding of knowledge organization. while the approach was designed to aid public libraries in their quest to meet the information needs of all of civil society it may in fact have applicability in more specific knowledge spheres as well. while many of these findings remain provisional in nature (such as the apparent identification of a power law at work in such collections) and will require further verification, the tier-based method outlined here offers the following benefits: it is simple, replicable, rigorously defined and enables, through providing a relatively objective methodology for making decisions in various parts of the non-fiction collection, the important interpretive aspects of selection and evaluation of information resources to be grounded in the hermeneutic and critical faculties of the librarian. it offers the promise that the inevitably subjective decisions that are made in support of quality collection development might also be referenced to a scientifically-based approach to managing the knowledge resources underpinning these important deliberative activities. such a process offers a considerable opportunity for growth, in terms of the ability to better target resources to the communities who need them, but also, in promoting the level of scientific and informetric engagement of the public librarians tasked to facilitate this.   acknowledgment   the author wishes to acknowledge maria c. tan who made many useful suggestions to improve this paper.   references   agee, j. (2005). collection evaluation: a foundation for collection development. collection building, 24(3), 92-95. http://dx.doi.org/10.1108/01604950510608267   alexander, f. (2012). assessing information taxonomies using epistemology and the sociology of science. journal of documentation, 68(5), 725-743. http://dx.doi.org/10.1108/00220411211256058   beals, j. b., & gilmour, r. (2007). assessing collections using brief tests and worldcat collection analysis. collection building, 26(4), 104-107.   benediktsson, d. (1989). hermeneutics: dimensions toward lis thinking. library and information science research, 11(3), 201-34.   benhabib, s. (1986). critique, norm, and utopia: a study of the foundations of critical theory. new york: columbia university press.   berger, p. l., & luckman, t. (1971). the social construction of reality. london: penguin. (originally published in 1966).   bernstein, j. h. (2006). from the ubiquitous to the nonexistent: a demographic study of oclc worldcat. library resources & technical services, 50, 79-90. http://dx.doi.org/10.5860/lrts.50n2.79   bernstein, r. j. (1983). beyond objectivism and relativism: science, hermeneutics, and praxis. philadelphia: university of pennsylvania press.   bruce, c. s. (1999). phenomenography: opening a new territory for library and information science research. the new review of information and library research, 5(1), 31-48.   buckland, μ. κ., & hindle, a. (1969) library zipf. journal of documentation, 25(1), 52-60.   budd, j. m. (2001). knowledge and knowing in library and information science: a philosophical framework. lanham, md and london: scarecrow press.   budd, j. m. (2005). phenomenology and information studies. journal of documentation, 61(1) pp. 44-59.   budd, j. (2008). self-examination: the present and future of librarianship. westport, ct: libraries unlimited.   capurro, r. (1992). what is information science for? a philosophical reflection. in p. vakkari and b. cronin (eds.), conceptions of library and information science: historical, empirical and theoretical perspectives, (pp. 82-98). london: taylor graham.   de solla price, d. j. (1976). a general theory of bibliometric and other cumulative advantage processes. journal of the american society of information science, 27, 292-306.   drott, m. c. (1981). bradford's law: theory, empiricism and the gaps between. library trends, 30(1), 41-52.   egan, m. e. & shera, j. h., (1952). foundations of a theory of bibliography. the library quarterly, 22(20), 125-137.   elzy, c. a., & lancaster, f.w. (1990). looking at a collection in different ways. collection management, 12(3-4), 1-10. http://dx.doi.org/10.1300/j105v12n03_01   evans, g. e. (2000). developing library and information center collections (4th ed.). englewood, co.: libraries unlimited.   fairthorne, r. a. (1969). empirical hyperbolic distributions (bradford-zipf-mandelbrot) for bibliometric description and prediction. journal of documentation, 25(4), 319-343.   fallis, d. (2006). social epistemology and information science. annual review of information science and technology, 40, 475-519. http://dx.doi.org/10.1002/aris.1440400119   genoni, p., & wright, j. (2010). assessing the collective wealth of australian research libraries: measuring overlap using worldcat collection analysis. the australian library journal, 59(4), 197-207.   hansson, j. (2005). hermeneutics as a bridge between the modern and the postmodern in library and information science. journal of documentation, 61(1), 102-113.   hekman, s.j. (1986). hermeneutics and the sociology of knowledge. cambridge: polity press.   jensen, k. (2012). data-driven decisions for library liaisons: exploring strategies for effectively managing diminishing monograph collections. collection management, 37(1), 9-2.   kafle, n. p. (2011). hermeneutic phenomenological research method simplified. bodhi: an interdisciplinary journal, 5(1), 181-200. http://dx.doi.org/10.3126/bodhi.v5i1.8053   kelly, m. (2014). assessing the relative value of domain knowledge for civil society's libraries: the role of core collections. proceedings of the 14th libraries in the digital age conference. paper presented at the 14th libraries in the digital age conference, 16-20 june 2014, university of zadar, zadar, croatia. retrieved from http://ozk.unizd.hr/proceedings/ index.php/lida   kelly, m. (2015a). developing optimal subject representation for adult non-fiction public library collections: the intersections of hermeneutics and technology. journal of technologies and knowledge sharing, 10(2), 11-19.   kelly, m. (2015b). collection development policies in public libraries in australia: a qualitative content analysis. public library quarterly, 34(1), 44-62. http://dx.doi.org/10.1080/01616846.2015.1000783   kelly, m. (2015c). the materials-centered approach to public library collection development: a defense. library philosophy and practice (e-journal). paper 1232. retrieved from http://digitalcommons.unl.edu/libphilprac/1232   lavoie, b. f., connaway, l.s., & o’neill, e.t. (2007). mapping worldcat’s digital landscape. library resources and technical services, 51, 106-115.   lesniaski, d. (2004). evaluating collections: a discussion and extension of brief tests of collection strength. college and undergraduate libraries, 11(1), 11-24. http://dx.doi.org/10.1300/j106v11n01_02   mcminn, h. s. (2010). evaluation of motor vehicles, aeronautics, astronautics collections using white's power method of collection analysis. collection management, 36(1), 29-52.   mannheim, k. (1972). essays on the sociology of knowledge. london: routledge & kegan paul. (originally published 1952).   matthews, p., & stephens, r. (2010). sociable knowledge sharing online: philosophy, patterns and intervention. aslib proceedings, 62(6), 539-553. http://dx.doi.org/10.1108/00012531011089667   monroe-gulick, a., & currie, l. (2011). using the worldcat collection analysis tool: experiences from the university of kansas libraries. collection management, 36(4), 203-216. http://dx.doi.org/10.1080/01462679.2011.604907   o'neill, e. t., connaway, l.s., &  dickey, t. j. (2008). estimating the audience level for library resources. journal of the american society for information science & technology, 59, 2042-2050. http://dx.doi.org/10.1002/asi.20908   osburn, c. b. (1979). some practical observations on the writing, implementation, and revision of collection development policy. library resources and technical services, 23(1), 7-16.   osburn, c. b. (2005). collection evaluation: a reconsideration.  advances in library administration and organization, 22, 1-21. http://dx.doi.org/10.1016/s0732-0671(05)22001-1   perrault, a. (2004). the role of worldcat in resources sharing. collection management,  28(1-2), 63-75. http://dx.doi.org/10.1300/j105v28n01_07   rorty, r. (1979). philosophy and the mirror of nature. princeton: princeton university press.   savolainen, r. (2008). everyday information practices: a social phenomenological perspective. lanham md: scarecrow press.   scheler, m. (1980) problems of a sociology of knowledge. (m. s. frings, trans.). london: routledge and kegan paul. (originally published in 1924).   standing committee of the ifla acquisition and collection development section (2001). guidelines for a collection development policy using the conspectus model.  retrieved from http://www.ifla.org/files/assets/acquisition-collection-development/publications /gcdp-en.pdf     stark, w. (1967). the sociology of knowledge: an essay in aid of a deeper understanding of the history of ideas. london: routledge & kegan paul.   svenonius, e. (2004). the epistemological foundations of knowledge representations. library trends, 52(3), 571-587.   twiss, t. m. (2001). a validation of “brief tests of collection strength”. collection management, 25(3), 23-37.   vanscoy, a., & evenstad, s.b. (2015). interpretative phenomenological analysis for lis research. journal of documentation, 71(2), 338-357.   white, h. (1995). brief tests of collection strength: a methodology for all types of libraries. westport, ct: greenwood press.   white, h. (2008). better than brief tests: coverage power tests of collection strength. college and research libraries, 69(2), 155-174. http://dx.doi.org/10.5860/crl.69.2.155   williams, r. (1976). keywords: a vocabulary of culture and society. new york: oxford university press.   wilson, p. (1968). two kinds of power: an essay on bibliographical control. berkeley: university of california press.   appendix a subject category percentages subject resultssubject category sample % domestic engineering 4.62% sports 3.02% history oceania, south seas 2.91% handicrafts, arts & crafts 2.81% decorative arts, applied arts 2.61% history, general 2.40% family, marriage, women, sexual life 2.33% history great britain 2.25% plant culture 2.18% individual psychology 1.88% business, business administration 1.71% literature on music 1.67% painting 1.66% graphic arts, drawing, design 1.57% motor vehicles, aeronautics, astronautics 1.53% economics industries, land use, labor 1.52% english philology & language 1.51% visual arts in general 1.47% animal culture 1.43% criminology, criminal justice 1.38% photography 1.22% architecture 1.17% public health, public aspects of medicine 1.15% history eastern asia, s.e. asia, far east 1.14% religions, mythology, rationalism 1.01% motion pictures 0.93% occult sciences 0.88% psychiatry 0.88% geography, general 0.84% the theater 0.79% history s.w. asia, middle east 0.76% social work, social & public welfare 0.76% therapeutics, pharmacology 0.75% history africa 0.71% finance, general 0.70% genealogy 0.63% practical theology 0.63% literature collections 0.60% games & amusements 0.59% history italy 0.57% biography 0.54% history france, andorra, monaco 0.53% building construction 0.49% law of the pacific area & antarctica 0.49% manners & customs, general 0.48% libraries library science 0.48% pediatrics 0.48% computer software 0.46% sociology, general & theoretical 0.45% astronomy 0.44% literature authorship & criticism 0.44% transportation & communication, general 0.43% parapsychology 0.43% doctrinal theology 0.43% economic history & conditions 0.43% electrical engineering 0.42% physical training 0.42% bible 0.41% labor, general 0.41% roman catholic church 0.40% special industries & trades, general 0.40% history southern asia, indian ocean 0.39% diseases of organs, glands, systems 0.39% ethnology. social and cultural anthropology 0.39% vocal music 0.38% manufactures 0.37% computer networks 0.37% recreation 0.37% general technology 0.36% broadcasting 0.36% gynecology & obstetrics 0.36% history united states, since the civil war 0.35% history united states, colonial, special topics 0.35% buddhism 0.35% physiology 0.34% science, general 0.33% botany, general 0.32% natural history 0.31% military science, general 0.31% economic theory 0.31% islam, bahaism, theosophy, etc. 0.30% chemical technology 0.30% christianity 0.30% neurosciences, neurology 0.30% ethics 0.28% history germany 0.28% philosophy modern (1450/1600) 0.28% botany, specific fields 0.27% special aspects of education 0.26% history russia. soviet union 0.26% folklore 0.26% birds 0.26% prose technique 0.26% communities, classes, races 0.26% journalism, the periodical press 0.26% office automation 0.26% philology, linguistics 0.26% computer programming & programming languages 0.25% arts in general 0.25% protestantism 0.24% history south america 0.24% naval architecture, shipbuilding, etc. 0.24% history greece 0.24% zoology, general 0.23% air force 0.23% agriculture, general 0.22% medicine, general 0.22% mechanical engineering & machinery 0.22% early childhood, preschool, kindergarten & primary 0.22% mathematics, general 0.22% theory & practice of education 0.22% political inst. & public admin., general 0.21% environmental technology 0.21% political inst. & public admin. asia/africa/australia   0.21% social history, social problems, social reform 0.20% political theory, theory of the state 0.20% biology, general 0.19%   dancing 0.19% history of civilization & culture 0.19% music instruction & study 0.19% special topics in computer science 0.18% history americas, general, indian,    north america 0.18% history balkan peninsula 0.17% neoplasms, tumors, oncology 0.17% history spain 0.17% meteorology 0.16% philosophy ancient, medieval, renaissance 0.16% sculpture 0.16% print media, printmaking, engraving 0.16% speculative philosophy 0.16% practice of medicine 0.16% state & local history n. england, atlantic coast 0.15% invertebrates 0.15% armies organization, distribution, etc. 0.15% physics, general 0.15% writing 0.15% military engineering 0.15% general engineering 0.14% atlases. globes 0.14% printing 0.14% special computers & systems 0.14% surgery 0.14% instrumental music 0.13% psychology 0.13% commerce, general 0.13% history of europe, general 0.13% state & local history pacific states. territories 0.12% history united states, slavery & civil war 0.12% economics industry, general 0.11% animal behavior, anatomy, embryology 0.11% french language, provencal language & literature 0.11% law, general 0.11% railroad engineering 0.11% environmental sciences 0.11% international relations 0.11% subject bibliography 0.11% computers, general 0.10% veterinary medicine 0.10% socialism, communism, utopias, anarchism 0.10% local government 0.10% bookselling & publishing 0.10% law united states, federal 0.10% naval science, general 0.10% general bibliography 0.10% military administration 0.10% navigation, merchant marine 0.10% pathology 0.10% judaism 0.10% anthropology, general 0.10% spanish language 0.10% nervous system 0.10% history asia, general 0.09% history mexico 0.09% public finance, general 0.09% school administration & organization 0.09% ecology 0.09% history northern europe, scandinavia 0.09% heraldry 0.09% state & local history south, gulf states 0.09% special aspects 0.09% major theories & systems 0.09% prehistoric archaeology 0.09% mining engineering & metallurgy 0.09% encyclopedias 0.08% history british/french/dutch america. canada 0.08% social usages, etiquette 0.08% archaeology, general 0.08% history west indies. caribbean area 0.08% italian language, sardinian language & lit 0.08% human ecology, anthropogeography 0.08% aquaculture & fisheries 0.08% political inst. & public admin. united states 0.08% japanese language 0.08% computer science general 0.08% immunologic, nutritional & metabolic diseases 0.08% reptiles & amphibians 0.08% history of medicine 0.08% other systems of medicine 0.07% political inst. & public admin. europe 0.07% forestry 0.07% fishes 0.07% german language 0.07% nursing 0.07% history central america 0.07% public health 0.07% state & local history midwest, old northwest 0.07% health professions 0.07% musculoskeletal system 0.07% history of scholarship & learning 0.07% societies secret, benevolent, etc. 0.07% constitution & properties of matter 0.07% immigration & emigration 0.06% collections 0.06% dictionaries, general reference 0.06% higher education 0.06% history of education 0.06% circuses, carnivals, etc. 0.06% genetics 0.06% history hungary, czechoslovakia 0.06% gynecology 0.06% paleozoology, paleobotany, palynology 0.06% statistics 0.06% chinese language 0.06% political science, general 0.06% military science maintenance & transportation 0.06% infantry 0.06% state & local history the west 0.06% special types of drama 0.06% medical centers, hospitals, clinics 0.05% navies organization, distribution, etc. 0.05% dermatology 0.05% algebra 0.05% dynamic & structural geology 0.05% geology, general 0.05% law of the united kingdom and ireland 0.05% special situations & cond. geriatric, sport 0.05% individual institutions asia, africa, oceania 0.05% history netherlands, low countries & belgium 0.05% human anatomy 0.05% oceanography 0.05% numeration, arithmetic, elementary mathematics 0.05% probabilities, math. stats., interpolation, numeri 0.05% geriatrics.  chronic disease 0.05% parlor magic & tricks 0.04% endocrine system 0.04% history: austria,austro-hungarian empire, liechtenstein 0.04% pharmacology 0.04% numismatics 0.04% internal medicine, general 0.04% biochemistry 0.04% latin literature 0.04% international law, international relations 0.04% paleontology 0.04% superintendent of documents publications 0.04% mineralogy 0.04% legislative & executive papers 0.04% secondary & middle school education 0.04% information resources 0.04% electricity, magnetism, nuclear physics 0.04% poetry, general 0.04% hunting sports 0.04% mathematical analysis 0.04% infectious & parasitic diseases 0.04% obstetrics 0.03% ophthalmology 0.03% history eastern europe, general 0.03% geomorphology 0.03% philosophy periodicals, societies, congresses 0.03% hydrology 0.03% animal biochemistry 0.03% digestive system 0.03% hydraulic engineering 0.03% central asian & far eastern republics 0.03% artillery 0.03% geometry, topology 0.03% museums, collectors & collecting 0.03% cardiovascular system 0.03% iranian philology & literature 0.03% communicable diseases 0.03% conservation of natural resources, land conservation   0.03% history latin america, spanish america, general 0.03% history portugal 0.03% greek language 0.03% metabolic diseases 0.03% microbiology 0.03% history united states, 1790-1861 0.03% russian language. belarusian language & literature 0.03% homeopathy 0.03% urogenital system 0.03% plant ecology 0.03% cartography 0.02% state and non-u.s. government documents 0.02% history mediterranean region, greco-roman world 0.02% drama, general 0.02% logic 0.02% toxicology 0.02% pharmacy & materia medica 0.02% aesthetics 0.02% plant physiology 0.02% history united states, revolutionary period 0.02% history switzerland 0.02% social sciences general 0.02% international law & relations 0.02% latin language 0.02% academies & learned societies 0.02% physical & theoretical chemistry 0.02% plant anatomy 0.02% history poland 0.02% minor services of navies 0.02% law of africa 0.02% regional geology 0.02% cybernetics 0.02% proverbs 0.02% highway engineering 0.02% history central europe, general 0.02% diplomatics. archives 0.02% national bibliography asia/africa/australia/oceania 0.02% eastern christian churches & ecumenism 0.02% reproduction & life 0.02% wildlife management 0.02% cytology 0.02% online data processing 0.02% performing arts & show biz 0.02% education & training of teachers 0.02% optics, light, radiation 0.02% portuguese language 0.01% microbiology and immunology 0.01% bridge engineering 0.01% otorhinolaryngology 0.01% petrology 0.01% otolaryngology 0.01% natural disasters 0.01% immunologic diseases.  collagen diseases. 0.01% law of asia & eurasia 0.01% organic chemistry, general 0.01% marines 0.01% forensic medicine 0.01% korean language 0.01% physical geography 0.01% law of europe, except uk & ireland 0.01% colonies & colonization 0.01% respiratory system 0.01% old norse literature: old icelandic & old norwegian 0.01% dutch language 0.01% inorganic chemistry, general 0.01% individual institutions europe 0.01% military science other services 0.01% copyright, intellectual property 0.01% cryptography. manuscripts. paleography 0.01% chronology 0.01% yearbooks, almanacs, directories 0.01% weights & measures 0.01% cavalry, armor 0.01% gypsies 0.01% dentistry 0.01% naval administration 0.01% national bibliography america, united states 0.01% hospitals and other health facilities 0.01% national bibliography europe 0.01% history of books 0.01% medical geography & climatology 0.01% personal bibliography 0.01% management information systems 0.01% dentistry.  oral surgery. 0.01% modern languages (general) 0.01% swedish language 0.01% general music 0.01% analytical mechanics 0.01% individual institutions united states 0.00% classical literature, general 0.00% hemic and lymphatic systems 0.00% heat 0.00% acoustics, sound 0.00% stratigraphy 0.00% political inst. & public admin. canada, latin america 0.00% african languages 0.00% semitic philology & assyrian & sumerian language & literature 0.00% optical data processing 0.00% chordates vertebrates 0.00% nutrition disorders 0.00% radiology 0.00% experimental mechanics 0.00% naval maintenance 0.00% mixed languages creole, pidgin english, etc. 0.00% u.s. states & territories 0.00% maps 0.00% textbooks 0.00% germanic philology & languages (general) 0.00% microscopy 0.00% norwegian language 0.00% general education 0.00% clinical pathology 0.00% oriental philology & literature (general) 0.00% parasitology 0.00% virology 0.00% geophysics, geomagnetism 0.00% american indian languages & literature 0.00% artificial languages & literature. secret languages, esperanto 0.00% classical philology 0.00% military astronautics, space warfare 0.00% periodicals 0.00% indo-aryan languages 0.00% mathematical geography 0.00% danish language 0.00% plant poisons 0.00% epigraphy, inscriptions 0.00% crystallography 0.00% analytical chemistry 0.00% auxiliary sciences of history, general 0.00% seals 0.00% diseases of regions of the body 0.00% indo-iranian philology & literature (general) 0.00% animal poisons 0.00% economic biology 0.00% government of canada publications 0.00% machine theory, abstract automata 0.00% student fraternities & societies, united states 0.00% tuberculosis 0.00% individual institutions america, except u.s. 0.00% naval ordnance 0.00% botanic, thomsonian, eclectic medicine 0.00% law of the americas, except the us & canada 0.00% diseases & injuries caused by physical agents 0.00% law of canada 0.00% indexes 0.00% college & school magazines & papers 0.00% national bibliography mexico, central & south america 0.00% newspapers 0.00% modeling & simulation 0.00% constitutional diseases (general) 0.00% naval seamen 0.00% congenital disorders 0.00% law of the sea 0.00%   note. subject categories showing 0% were actually represented by 1-16 titles. the level of 80% of the titles in the sample was achieved at subject category no.99 (philology, linguistics) and 80% of the subject categories were represented after that. ebl 101   matching question types to study designs   virginia wilson shirp coordinator health sciences library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   originally published in: evidence based library and information practice, 4(1), 51–52. https://ejournals.library.ualberta.ca/index.php/eblip/article/view/5063/5115     received: 31 jan. 2009   accepted: 31 jan 2009      2016 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   in the last ebl 101 column, lorie kloda discussed asking the right questions, and outlined the formulation of an answerable question. in evidence based librarianship, the question is the foundation upon which everything else rests. the question needs to be focused enough to find precise evidence while taking into account the key concepts involved in the situation. however, before rushing off to begin searching the literature after constructing the best question ever, take a moment to think about the type of question you’ve formulated. the question type can point you in the direction of the study design best suited to answering your particular question.   often, one’s prior level of knowledge of the topic will determine the type of question asked. first, decide if the question is a background question or a foreground question. a background question is one that is more general in nature and one that asks about fundamentals and facts. these types of questions might arise among novice practitioners, or among librarians who are encountering a new issue in the field for the first time. an example of a background question is, “what are the possible solutions to plagiarism in a post-secondary situation?” these types of questions may be answered by consulting a handbook, by conducting a literature review, or by eliciting thoughts and opinions from colleagues.   by contrast, the foreground question presumes prior knowledge of the subject, and the practitioner usually has a couple of alternatives in mind from which to choose. when a foreground question is generated, one is most likely at the point of decision making. an example of a foreground question is, “among teen public library patrons, do after school study programs result in higher marks at school?” these types of questions are often answered by turning to the literature and finding relevant research studies (booth 62).    but what type of research study should one look for? this is where the next grouping of question types comes into play. jonathan eldredge has written about question types and levels of evidence. he determined that there are three types of questions generated by library professionals:   prediction questions intervention questions exploration questions   prediction questions typically predict an outcome under particular circumstances. an example of such a question would be, “are students who have attended information literacy sessions more likely to continue their studies?”  these types of questions are often answered by using a cohort study; that is, a study that involves a defined population that is closely monitored over time to determine the outcome of being exposed to a particular phenomenon. andrew booth reported that prediction questions, and thus cohort studies, have investigated topics such as information resource use, outreach, education, and marketing (63).   intervention questions are aimed at finding particular outcomes by comparing different actions (or interventions). these questions often compare an innovation to a traditional way of doing things. an intervention question might look like this: “do medical students learn searching skills more effectively from librarians or teaching faculty?” (eldredge 11). the classic research design for an intervention question is a randomized controlled trial (rtc). an rtc involves taking two similar groups and exposing them to the different actions; that is, one group is taught searching skills by librarians and one group is taught by teaching faculty. because the groups are similar in their make-up, any changes can most likely be attributed to the intervention. intervention questions could involve teaching, delivering a reference service, or maintaining a collection (booth 64).   exploration questions closely resemble background questions. these questions typically ask or imply a “why” query. one example is, “why do non-library users not use their library?” qualitative research methods are best suited for the exploration question as these types of questions are more likely to be open-ended, and to have a need to explore ranges of behaviors and reasons for those behaviors. study designs include focus groups, ethnographic studies, observation, interviewing, and historical analysis (booth 65).   now that you have decided what type of study design is best suited to your question, it is time to examine the research evidence. next time: looking to the literature—domains to help determine where to look.   works cited   booth, andrew. “formulating answerable questions.” evidence based practice: an information professional’s handbook. eds. andrew booth and anne brice. london: facet, 2004. 61-70.   eldredge, jon. “evidence-based librarianship levels of evidence,” hypothesis 16.3 (2002): 10-14.   evidence summary   one-shot library instruction sessions may not increase student use of academic journals or diversity of sources   a review of: howard, k., nicholas, t., hayes, t., & appelt, c. w. (2014). evaluating one-shot library sessions: impact on the quality and diversity of student source use. community & junior college libraries, 20(1-2), 27-38. http://dx.doi.org/10.1080/02763915.2014.1009749   reviewed by: aislinn conway phd fellow national university of ireland galway galway, ireland email: a.conway18@nuigalway.ie   received: 21 sept. 2015 accepted: 4 nov. 2015      2015 conway. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to assess the effect of one-shot library research workshops focused on database searching on student coursework bibliographies.   design – comparative bibliometric analysis of student bibliographies.   setting – career and transfer program at a community college in illinois, united states of america.   subjects – students taking an english 101 course.   methods – during the study, 39 sections of english 101 occurred. an optional library instruction session was offered to faculty and as a result students from 18 sections participated. each session consisted of a 45-minute lecture and 30 minutes of independent research time. the librarian delivering the session introduced students to keyword searching and demonstrated the online library catalogue and two core electronic databases; academic search complete, and opposing viewpoints in context (ovc), and other databases of their choosing. students in each session were required to submit a variety of assignments in an exit portfolio at the end of the semester. sections of students were excluded if the instructors did not submit the students’ portfolios and they no longer taught at the community college, making it impossible to track down the portfolios. exclusion also occurred in cases where sections were taught by the researchers. seventeen sections who had attended library instruction group and 14 sections who had not attended the library instruction group were included in the study and randomised.   researchers evaluated portfolios according to the following criteria: whether the student who submitted the portfolio attended library instruction; whether their portfolio earned a pass or fail mark; total number of citations in bibliographies; number of each type of source (e.g., book, journal, internet resource, or other; and dates of sources).   main results – data were collected from 115 portfolios submitted by students who had attended a library session and 92 portfolios from students who had not attended a library session. student pass or fail status was not reported. attending library instruction did not have a significant effect on the mean number of total citations: 5.513 for attendees vs. 6.076 for non-attendees.  of 205 student portfolios evaluated, there was no difference in the number of types of resources used by students who had library instruction (2.3 ± 0.1) and those who had none (2.2 ± 0.1; p > 0.05).   conclusion – the library instruction sessions did not increase the use of academic journals or the diversity of sources used. however, students were more likely to use library sources if they attended training. the authors recommend that demonstrating multiple databases should be covered in later sessions and more conceptual information literacy instruction should be the first step.  librarians could teach the value of different types of sources; connect the sources to the tools needed to locate and access them; and demonstrate how to effectively evaluate sources. the authors recommend further research to evaluate objectives, content and outcomes of this type of library instruction.   commentary   to date, several studies report that one-off workshops, tailored to the needs of student groups, may result in the increased use of specific databases (rafferty, 2013; van epps, 2013). similar to the results of this article, these studies found that the educational interventions used had a positive impact on student assignments. current evidence is insufficient to provide us with a systematic understanding of how library instruction contributes to student-focused outcomes. the authors attempt to tackle this issue. heterogeneity in the conduct and reporting of previous library instruction sessions make them difficult to generalise to wider populations.   elements of the results section of the reliant (reader’s guide to the literature on interventions addressing the need for education and training) instrument were used to critically appraise the study. this is a tool generally used by library professionals for appraising published reports of educational and training interventions.   the authors address an important question regarding one-shot library sessions: how can they be designed to improve the standard of student citations? our knowledge of the impact of library instruction sessions is largely based on small studies; however, these authors collected an adequate sample size. they make important recommendations to focus more on information literacy for students.   the study lacks baseline information and there are too many unacknowledged variables. for example, we do not know the level of prior experience students had with literature searching, database usage, or additional training that might have impacted the quality of their portfolios. this information would be a valuable addition, allowing readers to judge whether the study could have external validity to other populations. while p-values are given for outcomes, confidence intervals are not reported.   the researchers describe how they altered their instruction as a result of their findings.   the researchers measured and recorded short-term outcomes based on the bibliographies in the students’ portfolios. therefore, it is not possible to predict whether students continued to benefit from the instruction in the long-term. the author concludes that the order of demonstration of the databases in the session was the reason a high number of students consulted the final database (ovc) more frequently. this reasoning does not account for factors such as ease of use, accessibility, or relevance to the assignments. in addition, the authors state that they made “numerous assumptions” during the study and that the way they determined whether a citation was a library or non-library resource was “highly subjective.” these limitations undermine the overall research findings and conclusions. therefore, data reported in the study may not support the authors’ conclusions.   this article is significant to library and information practice because it adds to the growing body of evidence that draws attention to the limitations of one-shot library instruction sessions. it points to the need to capture long-term learning outcomes. it highlights the lack of a reporting guideline in this field which could improve the quality and consistency between studies to facilitate comparison. the study demonstrates the complex challenges related to making claims about the impact of library instruction. library and information professionals who are planning and developing one-shot library instruction sessions should aim to prioritise active learning and information literacy instruction over database demonstrations to increase the potential depth of student learning and knowledge transfer.   references   koufogiannakis, d., booth, a., & brettle, a. (2006). reliant: reader’s guide to the literature on interventions addressing the need for education and training. library & information research, 30(94), 44-51. retrieved from http://www.lirgjournal.org.uk/lir/ojs/index.php/lir/article/view/271/318   rafferty, r. s. (2013). the impact of library instruction: do first-year medical students use library resources specifically highlighted during instructional sessions? journal of the medical library association, 101(3), 213-217. http://dx.doi.org/10.3163/1536-5050.101.3.011   van epps, a. & nelson, m. s. (2013). one-shot or embedded? assessing different delivery timing for information resources relevant to assignments. evidence based library and information practice, 8(1), 4-18. retrieved from https://ejournals.library.ualberta.ca/index.php/eblip/article/view/18027/14854     evidence based library and information practice evidence based library and information practice 2011, 6.4 146 evidence based library and information practice evidence summary ovid medline instruction can be evaluated using a validated search assessment tool a review of: rana, g. k., bradley, d. r., hamstra, s. j., ross, p. t., schumacher, r. e., frohna, j. g., & lypson, m. l. (2011). a validated search assessment tool: assessing practice-based learning and improvement in a residency program. journal of the medical library association, 99(1), 77-81. doi:10.3163/1536-5050.99.1.013 reviewed by: giovanna badia liaison librarian schulich library of science and engineering, mcgill university montreal, quebec, canada email: giovanna.badia@mcgill.ca received: 01 june 2011 accepted: 17 oct. 2011 2011 badia. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to determine the construct validity of a search assessment instrument that is used to evaluate search strategies in ovid medline. design – cross-sectional, cohort study. setting – the academic medical center of the university of michigan. subjects – all 22 first-year residents in the department of pediatrics in 2004 (cohort 1); 10 senior pediatric residents in 2005 (cohort 2); and 9 faculty members who taught evidence based medicine (ebm) and published on ebm topics. methods – two methods were employed to determine whether the university of michigan medline search assessment instrument (ummsa) could show differences between searchers’ construction of a medline search strategy. the first method tested the search skills of all 22 incoming pediatrics residents (cohort 1) after they received medline training in 2004, and again upon graduation in 2007. only 15 of these residents were tested upon graduation; seven were either no longer in the residency program, or had quickly left the institution after graduation. the search test asked study participants to read a clinical scenario, identify the search question in the scenario, and perform an ovid medline evidence based library and information practice 2011, 6.4 147 search. two librarians scored the blinded search strategies. the second method compared the scores of the 22 residents with the scores of ten senior residents (cohort 2) and nine faculty volunteers. unlike the first cohort, the ten senior residents had not received any medline training. the faculty members’ search strategies were used as the gold standard comparison for scoring the search skills of the two cohorts. main results – the search strategy scores of the 22 first-year residents, who received training, improved from 2004 to 2007 (mean improvement: 51.7 to 78.7; t(14)=5.43, p<0.0001). the graduation scores were also significantly higher for this first cohort compared to the second cohort, who received no training (median 85.0 vs. 65.0; wilcoxon chi-square(1)=4.09, p=0.043). the graduation scores of the first cohort were similar to those of faculty volunteers (wilcoxon chisquare(1)=3.82, p=0.050) . conclusion – according to the authors, “the results of this study provide evidence for the validity of an instrument to evaluate medline search strategies” (p. 81), since the instrument under investigation was able to measure improvements and differences in the search performances of the study’s participants. a validated search assessment instrument can effectively measure improvements in residents’ search skills to demonstrate training effectiveness, as well as satisfy practice-based learning competency requirements from the accreditation council for graduate medical education. commentary the authors state that they are unaware of any validated search assessment instruments that measure residents’ medline search performances. this study fills a gap in the existing literature by presenting and validating an instrument, the university of michigan medline search assessment (ummsa), to evaluate ovid medline search strategies. instructional librarians can use this study to create and validate their own instruments to measure search performance improvement as a result of medline training. proof for the validity of the ummsa consisted of the following: 1. the instrument was created by expert librarians and provided useful information about searchers’ abilities when used; 2. the instrument was piloted in other settings; and 3. the instrument successfully recorded differences in the search performances of clients with different searching expertise. the ummsa was modeled on a search assessment instrument developed by nesbit and glover in 2002. the authors declared that this instrument “allowed greater objectivity and provided a more efficient means to measure search skills” (p. 78) than the tools they had used in the past. this reviewer would have liked an explanation of how nesbit and glover’s instrument allowed greater objectivity and efficiency. the authors also briefly discuss how they used item-total score correlations from the ummsa, and their judgment as expert searchers, to identify the five most important elements in an effective medline search strategy. according to the authors, an effective search strategy in medline should include: 1. all search concepts; 2. medical subject headings; 3. appropriate search limits; 4. boolean operators for all search concepts; and 5. an efficient search history (strategies should not include too many search sets that are not combined in the final search string) it is not clear from the article how item-total score correlations from the ummsa are an appropriate means to identify the critical evidence based library and information practice 2011, 6.4 148 elements in a medline search. a summary of the published literature on the elements required to perform a successful search would have further supported the authors’ argument. further research could confirm the importance of the authors’ five critical elements for conducting a successful medline search. after which, the authors could shorten the search assessment instrument from the eleven search criteria used in the ummsa tool to only the five critical elements. investigators could then retest this revised instrument in different populations to establish its construct validity. references nesbit, k., & glover, j. (2002). ovid medline search strategy scoring sheet. rochester, ny: university of rochester. / evidence based library and information practice evidence summary   differences in faculty approaches to plagiarism deterrence are an opportunity for increased collaboration in information literacy instruction   a review of: michalak, r., rysavy, m., hunt, k., worden, j., & smith, b. (2018). faculty perceptions of plagiarism: insight for librarians’ information literacy programs. college and research libraries, 79(6), 747-767. https://doi.org/10.5860/crl.79.6.747     reviewed by: sarah schroeder research & instruction librarian university of washington bothell/cascadia college campus library bothell, washington, united states of america email: sarahkb6@uw.edu   received: 23 nov. 2019                                                                 accepted:  16 jan. 2020      2020 schroeder. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29651     abstract   objective – to learn how faculty members define plagiarism and what actions (if any) they are taking in their classes to educate students about plagiarism.   design – online survey.   setting – a small private college in the northeastern united states of america.   subjects – a total of 79 full-time and adjunct faculty members in arts and business.   methods – participants completed an online survey, modified from a survey in the plagiarism handbook, in which they provided their definition of plagiarism. they then answered yes/no questions regarding their knowledge levels and methods of plagiarism instruction used in class.  the authors collected data on the faculty members’ age, discipline, years of experience, and their status as either adjunct or full-time faculty. after analyzing the results independently, the authors later collaborated to discuss codes and identify clear themes in the list of definitions.   main results – an analysis of faculty members’ plagiarism definitions determined that most define plagiarism in a way that roughly aligns with the university’s definition, but identified inconsistencies regarding severity, student knowledge, the role of intent, and the necessity of a source attribution when determining what constitutes plagiarism. the themes in their responses clearly illustrate the major differences in approaches to plagiarism.   the authors also found that while 87% of respondents reported discussing plagiarism in their classes, they usually did so only “a little” or “a moderate amount.” furthermore, just over 53% of respondents did not provide their students with materials on plagiarism, though 55% reported including a definition of plagiarism in their course syllabi. researchers also asked whether or not faculty members had invited a librarian to speak to their class about plagiarism, to which 74% of faculty members responded no.   conclusion – this study suggested that librarians should consider differing perspectives on plagiarism when collaborating with faculty members and that librarian-faculty collaboration on information literacy instruction can help to mitigate the effects of inconsistent practices regarding plagiarism. the study’s authors are integrating their research findings into anti-plagiarism training modules for students at the institution where this study was conducted. future studies based on this research are planned to further explore the intersections of plagiarism and information literacy.   commentary   plagiarism is a persistent topic in the library literature. some recent work on plagiarism has focused on faculty perceptions of what constitutes plagiarism. for example, one study at queensborough community college surveyed faculty members in english and speech & theater about their attitudes toward various plagiarism scenarios (marcus & beck, 2011), while another multi-university study specifically surveyed faculty members on their views of student self-plagiarism (hallupa & bollinger, 2013). the authors of the present study noted that they believe their method of asking faculty members to provide their own definitions of plagiarism is the first of its kind and the additional data they provided on how often faculty members are communicating plagiarism information to their students is also unusual in the wider body of literature.   this summary uses an appraisal tool developed by lindsay glynn to evaluate library and information science research (2006). this tool addresses four sections: population, data collection, study design, and results. the researchers conducted their survey in the context of broader efforts to address student plagiarism through their information literacy program. as employees of a small school with only business and arts & sciences faculties, the size and diversity of the population investigated was limited, meaning results could likely not be generalized.   the full survey is included as an appendix, and it shows that faculty members were asked about several topics not addressed in the results section. for example, faculty members were asked about their experiences with encountering and reporting honour code violations on the survey, but no data on this subject is shared in the final article. another question gauged how faculty members perceived plagiarism expectations and consequences for both domestic and international students. the data from this part of the survey was not shared in the article, and it is unclear why it was omitted.   academic librarians who are seeking to collaborate with faculty members on plagiarism deterrence can build upon the survey data provided here and may consider the findings in discussions with faculty members. the themes identified in the study can act as a springboard to further discussion with teaching faculty regarding what constitutes plagiarism and how best to address it.   references   bennett, k., behrendt, l., & boothby, j. (2011). instructor perceptions of plagiarism: are we finding common ground? teaching of psychology, 38(1), 29-35. https://doi.org/10.1177/0098628310390851      glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154       halupa, c., & bolliger, d. (2013). faculty perceptions of student self plagiarism: an exploratory multi-university study. journal of academic ethics, 11(4), 297-310. https://doi.org/10.1007/s10805-013-9195-6   marcus, s, & beck, s. (2011). faculty perceptions of plagiarism at queensborough community college. community & junior college libraries, 17(2), 63-73. https://doi.org/10.1080/02763915.2011.591709     evidence summary   analysis of static and dynamic e-reference content at a multi-campus university shows that updated content is associated with greater annual usage   a review of: lamothe, a. r. (2015). comparing usage between dynamic and static e-reference collections. collection building, 34(3), 78-88. http://dx.doi.org/10.1108/cb-04-2015-0006   reviewed by: laura costello head of research & emerging technologies stony brook university libraries stony brook, new york, united states of america email: laura.costello@stonybrook.edu   received: 30 nov. 2015 accepted: 2 feb. 2016      2016 costello. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to discover whether there is a difference in use over time between dynamically updated and changing subscription e-reference titles and collections, and static purchased e-reference titles and collections.   design – case study.   setting – a multi-campus canadian university with 9,200 students enrolled in both graduate and undergraduate programs.   subjects – e-reference book packages and individual e-reference titles.    methods – the author compared data from individual e-reference books and packages. first, individual subscription e-reference books that periodically added updated content were compared to individually purchased e-reference books that remained static after purchase. the author then compared two e-reference book packages that provided new and updated content to two static e-reference book packages. the author compared data from patron usage to new content added over time using regression analysis.   main results – as the library acquired e-reference titles, dynamic title subscriptions added to the collection were associated with 2,246 to 4,635 views per subscription while static title additions were associated with 8 to 123 views per purchase. the author also found that there was a strong linear relationship between views and dynamic titles added to the collection (r2=0.79) and a very weak linear relationship (r2=0.18) with views when static titles are added to the collection. regression analysis of dynamic e-reference collections revealed that the number of titles added to each collection was strongly associated with views of the material (r2=0.99), while static e-reference collections were less strongly linked (r2=0.43).   conclusion – dynamic e-reference titles and collections experienced increases in usage each year while static titles and collections experienced decreases in usage. this indicates that collections and titles that offer new content to users each year will continue to see growth in usage while static collections and titles will see maximum usage within a few years and then begin to decline as they get older. fresh content is strongly associated with usage in e-reference titles, which mirrors the author’s previous work examining static and dynamic content in e-monographs.   commentary   this study represents a quantitative comparison of dynamic e-reference titles and collections, and static e-reference titles and collections with attention to the value of these collections to academic library patrons. the author contextualized this study with findings from his previous works which showed that updated content had a positive impact on e-monograph usage. though collection development literature now frequently features quantitative studies of electronic materials, this article represents one of the first attempts to compare usage between purchased and subscription e-reference materials.   the researcher in this study also provides these materials as a service at the university, but the two types of collections compared in the study were already owned or set up as recurring purchases by the author’s institution at the time of writing. a standardized measure of use from the author’s other works was used to determine the value to patrons for each title and collection. additions to packages and purchased titles were determined using simple counts and included all eligible e-reference materials obtained by the library between 2002 and 2014. the author used a multiple regression analysis to calculate the usage of materials over time. this is an appropriate model, but we have few benchmarks for comparing the usage data from this research with print and e-reference data from other institutions.   the results represent the usage of one library from a collection specifically selected for the particular needs of that library. the resources, particularly static individual e-reference titles, may be tied to individual course assignments that changed over the observation period. the work represents a template that could be tested in other institutions to overcome this limitation. calculating usage over time, particularly for static e-reference materials, could be used to shape purchasing decisions for the future, even if a sufficient return on investment has already been reached. the results suggest that subscription e-reference titles and packages are a better investment for libraries than static content, especially if annual subscription prices are lower than purchasing titles outright.   the author also makes clear that the content of dynamic subscription e-reference materials is often fundamentally different from the content of static purchase collections and titles. encyclopedias, dictionaries, and style manuals tend to feature subscription payment models and also contain information that users consult many times in the course of their research and scholarship. dynamic titles are often also general resources with broad appeal, while many static titles are discipline-specific. dynamic reference materials represent a much larger lifetime investment for libraries, but that investment may be justified with continuing high levels of usage. evidence based library and information practice 2012, 7.3 125 evidence based library and information practice news/announcements call for submissions: evidence based library and information practice 2012. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. evidence based library and information practice (eblip) is a peer reviewed, open access journal, which provides a forum for librarians and other information professionals to discover research that may contribute to decision making in professional practice. eblip is published quarterly by the university of alberta learning services. eblip publishes original research and commentary on the topic of evidence based library and information practice, as well as reviews of previously published research (evidence summaries) on a wide number of topics. eblip seeks submissions for the following sections:  classics: contributions to the “classics” section follow a structured format designed to highlight, summarize, and critically appraise research studies that stood the test of time and continue to have an impact on library and information practice.  commentaries: commentaries are opinion pieces on a topic related to evidence based practice. this is a non peer-reviewed section.  review articles: review articles provide a way for librarians to obtain an overview of the evidence on a particular topic, and stay current with the literature.  using evidence in practice: the “using evidence in practice” section features brief reports of lis practitioners' use of evidence to assist with decision making. this is a nonpeer reviewed section. for more information about these sections, please visit http://ejournals.library.ualberta.ca/index.php/e blip/about/submissions. for more information about the journal, please visit the evidence based library and information practice web site http://ejournals.library.ualberta.ca/index.php/e blip/index. thank you, alison brettle editor-in-chief a.brettle@salford.ac.uk http://www.ls.ualberta.ca/ http://www.ls.ualberta.ca/ http://ejournals.library.ualberta.ca/index.php/eblip/about/submissions http://ejournals.library.ualberta.ca/index.php/eblip/about/submissions http://ejournals.library.ualberta.ca/index.php/eblip/index http://ejournals.library.ualberta.ca/index.php/eblip/index mailto:a.brettle@salford.ac.uk evidence based library and information practice 2012, 7.1 41 evidence based library and information practice article the use of volunteers in local study library projects: a case study of the walter gardiner photography project beth hewitt assistant information adviser information services, university of brighton brighton, united kingdom email: b.hewitt@brighton.ac.uk juliet eve principal lecturer school of computing, mathematics and engineering, university of brighton brighton, united kingdom email: j.eve@brighton.ac.uk received: 31 aug. 2011 accepted: 7 feb. 2012 2012 hewitt and eve. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objectives – interviews with library staff and volunteers were conducted to evaluate the use of volunteers in uk public libraries via a case study of the walter gardiner photographic project, a digitisation project based in worthing library, to inform future guidelines on volunteer usage and to make recommendations to existing practice. methods – fourteen semi-structured interviews were carried out to explore the perceptions and experiences of both staff and volunteers of the project. all interviews were fully transcribed and then coded to identify emergent themes. results – key positives for volunteers were professional training, good time management and organization by staff, the friendliness and approachability of staff, and the informal nature of the volunteering. enjoyment of the work and forming close relationships with others were key motivating factors. for staff, the completion of work which would have otherwise been impossible was the most positive outcome. problem areas identified by volunteers were lack of contact time with project staff and feeling isolated from other library staff. for project staff, a lack of professionalism on behalf of some volunteers was the primary negative. key issues to emerge were the mailto:b.hewitt@brighton.ac.uk� mailto:j.eve@brighton.ac.uk� evidence based library and information practice 2012, 7.1 42 need to strike a balance between formal and informal management, the need for good integration between the volunteers and host organization, and the importance of acknowledging the nature of the voluntary commitment. conclusions – the project proved overall to be a successful example of using volunteers in public library projects with good examples of volunteer recruitment, training, and management being demonstrated. areas of conflict that did arise stemmed from differing expectations of levels of service between staff and volunteers. clarification on these expectations through a written volunteer agreement is advocated for further projects. introduction the use of volunteers in u.k. public libraries is (and has been) a controversial and complex issue, balancing the potential benefits (e.g., increased community involvement, expansion of existing services, and injection of enthusiasm, new skills, and knowledge) against the possible disadvantages: damage to staff relations, expense incurred through time and money for training, and management of volunteer work and expectations (gale, 1999; jervis, 2000; cookman, haynes, & streatfield, 2000; cookman, 2001; mcdiarmid & auster, 2004). over the last 18 months, the twin political and economic influences of the uk coalition government’s “big society” agenda and the severe cuts to local authority budgets have re-ignited the controversy around the issue: “community-managed” local libraries have been posited as an alternative to both library closures initiated by local authorities and as an ideological alternative to state-run public services. consequently, there is a more fundamental need than ever for the library profession to address the subject of library volunteerism and for library services to consider best practice strategies for managing volunteers to ensure productive, mutually beneficial relationships with volunteers. this article documents the key findings of a qualitative research study carried out in 2009, which sought to explore the perceptions and experiences of both staff and volunteer participants of the walter gardiner photographic project, a local studies project based at worthing library, a public library within the west sussex library service in the south east uk. the project was set up in 2007 to preserve, digitize, and catalogue a photographic collection of historical significance to the local area (worthing). following a publicity campaign and an interview process, 12 volunteers were recruited to work with over 5,500 paper photographs, glass negatives, film negatives, documents, and ephemera from the pre-1946 part of the collection. items were numbered, cleaned, re-housed in archival sleeves and boxes, researched, and listed on a computer. nearly 1,000 originals were selected for more detailed research, high quality scanning, and digital restoration. the project culminated in an exhibition of the best photographs from the collection, which volunteers stewarded, talking to the public about the images and giving demonstrations of the archival work they had undertaken. the project was the first of its kind run by the local studies team and was to be used as a benchmark for further volunteer-involved projects. the research set out to evaluate project participants’ experiences based broadly around the themes of recruitment, training, and management, as experienced by both staff and volunteers. at a local level the findings of this research were to form the basis of a “volunteer’s toolkit,” a set of good practice guidelines for future use in west sussex library service projects involving volunteers. on a broader level the project sought to contribute to the limited research on library volunteers, to document the hitherto underrepresented ”volunteer perspective” on library volunteering, and to move away from the existing “advantages and disadvantages” evidence based library and information practice 2012, 7.1 43 discourse which has traditionally dominated lis research on volunteerism (mcdiarmid & auster, 2004). further, the project sought to focus on volunteer experience, as opposed to recent policy literature, which focuses only on volunteer usage and best practice for management of volunteers (e.g., cookman et al., 2000; capital planning information limited, 2000; howlett, machin, & malmersio, 2005). a qualitative methodology was employed to allow for exploration of the social processes of the project and to reflect the complexities of volunteer and organizational relationships. literature review volunteers play an active part in the delivery of u.k. public library services. in 2000 cookman et al. reported 85% of english authorities, 82% of scottish, and 63% of welsh authorities were using volunteers, and deployed them across (amongst others) housebound services, children’s story time, it instruction, newspaper indexing, and library promotion services. recent figures suggest little sign of this changing, and show an increase of 8.5% between 2008-2009 and 20092010, with a total of 16,271 volunteers providing libraries over half a million person hours (museums, libraries and archives council, 2010). existing research despite this topic being one of long standing interest to the lis community, only a small body of research studies exist to complement the much larger amount of anecdotal literature which is available. for example, the studies by lock (1994) and flood (2004) offer brief accounts of the work undertaken, but reflect little on the nature of volunteering. central to existing lis research on volunteers is a discussion around the advantages and disadvantages of using volunteers (mcdiarmid & auster, 2004). a number of perceived common benefits of volunteers are evidenced in gale’s (1999) study of london library managers, in jervis’ (2000) study of 25 u.k. national services, and in the work of cookman et al. (2000). cookman (2001) further notes advantages in the opportunity to forge links with the local community, to demonstrate in a tangible manner the library as a community resource, to promote library services informally, and the opportunity to take advantage of local knowledge. volunteers are used to add value to existing services by undertaking “extra” library projects (such as the walter gardiner photographic project), which may not be otherwise achievable. for example, howlett et al. (2005) found that 74% of library services surveyed cited “allowing us to do things we would not normally be able to do” as the most advantageous aspect of volunteers (p. 12). similarly, freeing paid staff from more routine tasks meant they could contribute more time to such projects. staff morale can be increased by the enthusiasm and new skills displayed by volunteers, and opportunities exist to improve management skills by working with diverse people (jervis, 2000). roy (1988) found that structured volunteer programs were favourably viewed by library management, and mccune and nelson (1995) suggest that staff felt more socially and professionally confident in their jobs when working with people from various backgrounds. a disadvantage of using volunteers reported by jervis (2000) was the difficulty of recruiting suitable people, alongside concerns about reliability and professionalism (gale, 1999). similarly, mcdiarmid and auster (2004) found that the major pitfall perceived by volunteer managers in hospital libraries was poor attendance and commitment. additionally, there was a worry that “services could run the risk of being seen as second class” if overly staffed with volunteers with no professional guidance (jervis, 2000, p.16). debates in the usa echo this sentiment: those opposed to volunteer programs are concerned with a decline in professionalism in libraries, threats to staff employment, and a decreased probability for adequate public funding. (nicol & johnson, 2008, p. 154) evidence based library and information practice 2012, 7.1 44 many of the concerns relating to recruitment reported by cookman et al. (2000) were as much to do with maintaining levels of volunteers to sustain service delivery standards, as they were about taking volunteers on at all. the strong commitment of volunteers was often more acknowledged than concerns about their durability. without the incentive of a pay packet it is understandable that managers may be concerned over the commitment given by volunteers; however the nature of the voluntary relationship between volunteer and organization is one which is not underpinned by the same motives and incentives as paid employees, and should not be judged as such (gay, 2001; zimmeck, 2000; gaskin, 2003). this research found that the commitment of the volunteers was equal to and perhaps greater than that of paid staff. the “bind” to the organization was not a contractual one but rather a moral one which was perceived as being of more value than a piece of written documentation. the enthusiasm of volunteers was actually cited as an advantage of using volunteers by one member of staff: i think what has really shone through is the enthusiasm. they’ve come here because they’ve really wanted to. (member of staff b) library and government policy in 2001 the then library association published volunteer guidelines, covering the three areas of policy, recruitment, and management, and emphasized the need for a clearly defined policy before volunteers are recruited. the guidelines advised that volunteers should have access to effective supervision, reviews, and necessary support mechanisms such as training, which reflects the more formalized, work-based approach that developed in the wider voluntary sector (mccurley & lynch, 1998). how many library services have such a policy is not measurable, however the research of howlett et al. (2005) showed that 57% of responding libraries in their survey (174) had a formal policy, 10% more than in 2001. the current u.k. government, via the championing of its “big society” concept (see http://www.cabinetoffice.gov.uk/content/bigsociety-overview), has increased the pressure to make use of volunteers in libraries as part of a wider ideological move to shift the running of services away from local authorities. local government should seek to devolve to the most local level possible and to encourage communities to take over services. one example would be libraries . . . (downey, kirby, & sherlock, 2010, p. 12) the future libraries programme, for example, has established 10 projects to “test drive an ambitious change programme for libraries” (“future libraries programme”, 2010) including exploring “increased use of volunteers to run libraries or to work alongside professionals to support opening hours and services” (local government group, 2011, p.13). a museums, libraries and archives council (mla) report drawing on 29 existing community libraries suggests that already volunteers are replacing, rather than complementing professional staff: the majority (86%) of community libraries are not staffed by professional librarians or library service staff, with volunteers taking on the roles not only of frontline staff but also of managing events and activities, stock development and rotation, and premises and budget management. (museums, libraries and archives council, 2011, p. 7) the chartered institute of library and information professionals (cilip), the successor body to the library association, reiterated in 2010 that volunteers enrich libraries and help to sustain their viability. however, cilip noted that volunteers are not “free” and need proper management, training, and development, and that their use should be: part of a professionally managed public library service that has at its core sufficient paid staff to ensure evidence based library and information practice 2012, 7.1 45 the direction, development and quality of the service provided. (chartered institute of library and information professionals, 2010) whether volunteers take over the running of libraries completely, or merely continue to complement existing services, this trend is likely to continue, which makes it vital for libraries that they are able to manage the process of recruiting, training, and managing volunteers. methods in order to investigate issues of volunteering in depth, a case study approach was adopted, based on the assumption that [via] an in-depth investigation of a discrete entity . . . it is possible to derive knowledge of the wider phenomenon from intensive investigation of a specific instance or case. (gorman, clayton, rice-lively, & gorman, 1997, p. 50) selecting this approach was appropriate for a number of reasons. the walter gardiner photographic project was “a functioning specific” case (stake, 1995) in that it had clearly defined parameters and it was therefore possible to study it as a ‘discrete entity’. as denscombe (2010) identifies, case studies are an appropriate means to study detailed workings and social processes, rather than restricting attention to outcomes. the aims and objectives of the research were to obtain an indepth, holistic account of the project from both staff and volunteer perspectives. the case study approach thus facilitated an understanding of the perspectives of all those involved and explored the complexities of the relationship between volunteers and staff. the project can be seen as a “typical instance” of its type (denscombe, 2010, p. 57), and thus provides opportunities to generalize from the findings. caution must be taken when generalizing from such a small scale study, however the authors suggest that the findings “allow for transferability . . . based on contextual applicability” (pickard, 2007, p. 93), as the case study is similar enough to others of its type to allow for this. based on a qualitative approach, project staff and volunteer interviews were the sole method of data collection employed. fourteen interviews, lasting up to an hour each, were conducted over a 3 week period in july and august 2009: 4 library staff and 10 out of 12 volunteers participated. interviewees were provided with full information on the research project, and an interview consent form was signed by both the interviewee and the researcher prior to each interview. as the research aimed to record the perceptions, emotions, and motivations of the volunteers and staff, semi-structured interviews were identified as the most appropriate means of acquiring this descriptive data (denscombe, 2010). an interview schedule was developed and used as a guide for the interviews, based on a number of key themes arising from the review of the literature: volunteer recruitment, training, and management. to avoid misrepresentation, respondent validation was sought after the initial interviews to check the factual accuracy, and also to allow for the researcher’s understanding to be confirmed. all fourteen interviews were transcribed verbatim, concurrent with the interview process. a line by line analysis of the transcripts was undertaken and the material was assigned a series of codes. codes here refer to: . . . tags or labels for assigning units of meaning to information compiled during a study. codes are usually attached to chunks of varying size . . . they can take the form of a straightforward category, label or a more complex one. (miles & huberman, 1994, p.56) an element of pre-coding existed at the outset due to the broad thematic structure of the interview schedule. the research aimed to explore “social processes” and not merely provide a simple analysis of themes, moving the coding beyond “data reduction,” towards evidence based library and information practice 2012, 7.1 46 “data complication.” the analysis was used to “expand, transform and reconceptualise data and open the data to further analysis, using codes as heuristic devices” (coffey & atkinson 1996, p. 29). codes were refined so that synonyms were removed, making the codes standardized, and colour coding was used to denote concepts within the transcripts to aid “data display.” a systematic examination of commonalities and differences among interviewees was undertaken, drawing out a number of themes but also highlighting complexities. subjects the request for volunteers was made through the local media and the internet from november/december 2008. advertising was placed in local newspapers and television and an estimated 300-400 people expressed an interest in the project. interviews took place in january 2008 conducted by the county local studies librarian and a member of the digitisation unit. fifty volunteers were interviewed and 12 chosen, of which 9 were female and three male. of these volunteers two female volunteers did not participate in the research. of the 10 interviewed for the research, all were white british, and 8 of the 10 were retired. the final selection was representative of those who had made initial inquiries. of the 50 interviewed, staff estimate that 75% were women and at least 89% were retired people, with the great majority being over the age of 65. this is consistent with other research on volunteers in the museums, libraries, and archives sector. howlett et al. (2005) found that volunteers across museums, libraries, and archives in the uk are predominantly white (on average, 96%), aged 55 years and over (72%), and female (68%). two of the volunteers had previous library experience: one had been an information librarian and the other had worked as a library assistant, both for west sussex library service. four members of staff were interviewed: two local studies librarians and the two project officers in the west sussex digitisation unit. key findings and discussion key issues emerging from the data were organized thematically around the main areas of recruitment, training, and management of volunteers. the results from volunteer and staff perspectives are interwoven, which provides a detailed picture of the successes and limitations of the project, and highlights some areas of tension which speak to the sometimes complex nature of the relationships between volunteers and paid staff in institutions. recruitment all volunteers interviewed stated that the interview and recruitment processes were well done. all agreed that they had an opportunity to talk about their skills and interest in the project at interview, and received an appropriate introduction to the project. two thirds of the volunteers commented on the informality of the interview, which was viewed positively. this supports the evidence in volunteer recruitment literature, which emphasizes veering away from a formal “workplace” model of interviewing, and concludes that volunteer interviews work best as an informal two-way process (mccurley & lynch, 1998; volunteering england, 2011). at odds with this desire for informality is an increased expectation for libraries to have more rigorous recruitment policies in place in which volunteer agreements outline details such as the length of a project, its expected outcomes, and an anticipated number of volunteer hours (library association, 2001). eight of the 10 volunteers in this project indicated a formalization of the recruitment process, such as providing a reference, would be acceptable; however, all also felt that whilst it would be necessary to have a reference if you were working with children or vulnerable people, for a local studies project it was too officious, and “a bit county council” (volunteer f). evidence based library and information practice 2012, 7.1 47 training all 10 volunteers felt the training they had received had equipped them with the right skills and knowledge to do the job. an initial practical session with a conservator was identified by 8 of the 10 volunteers as the most successful element of the training and over half emphasized his professionalism and thoroughness: very professional more so than we needed really. (volunteer a) even though at least six of the volunteers emphasized the work was “commonsense,” having professional training made them feel more confident about their skills, and half of the volunteers mentioned that it also gave them confidence that staff were serious enough about the work they were undertaking to invest time and resources in them. all four staff participants stated that the volunteer training was a time consuming process, corroborating the research of howlett et al. (2005) who found it to be the most cited concern amongst library personnel about involvement of volunteers. however, in line with other studies – for example those of jervis (2000), gale (1999), and cookman et al. (2000) – this was not seen as a “disadvantage” but just a necessary part of using volunteers: i’d never use the term pitfalls or disadvantages. you have to create a lot of documents for them and you have to spend a lot of time thinking about them. you have to think more carefully than with staff. . . . as long as you are aware of that i don’t regard it as a disadvantage. (member of staff d) management nine of the 10 volunteers believed they were well managed and could not think of any major improvements. the most frequently mentioned positive aspects were: friendliness and welcoming attitude of the staff, the informality of the project staff-volunteer relationship, and the feeling of being valued by staff. of these, particular weight was placed on the importance of staff being friendly and the informality of the volunteering experience in both work schedule and interpersonal relationships. it was good management the training everything it was a real pleasure. (volunteer d) all the staff are very friendly and good at listening to any problems that we had. (volunteer a) this informal culture was deliberately fostered by the library staff: i like to have fun in sessions as well. that’s always been motivational. it’s the way i work. not to sit there in silence . . . that’s our methodology. (member of staff d) further to this approach was the reinforcement of their usefulness to the service. every volunteer showed an awareness of their purpose and function in “doing something that otherwise could not be done.” i motivate people by making them realise they are extremely important and they are important part of an important project . . . (member of staff d) as locke, ellis, and smith (2003) suggest, being able to put one’s contribution into some context and clarify one’s purpose can be motivational. all staff reflected that time management was one of the biggest difficulties of the project, due to a lot of hands-on management, and as one interviewee reflected: i think maybe if we made more time at the beginning . . . it probably would have saved time in the long run, because things keep coming up . . . (member of staff c) this was reiterated by another interviewee: it was fine in the early days [answering queries]. but it’s evidence based library and information practice 2012, 7.1 48 not useful to do that because they become dependent on you. even after five or six months. i’d get [name removed] to deal with stuff that was routine by then. there wasn’t a set policy for that and that would have been useful actually, and i could have defined roles different, better. i could have got [name removed] to be a first point of call. (member of staff d) evidentially, it would have been an improvement to clarify the supervisory structure and define early on in the project the roles allocated to each member of staff, so that volunteers would have had a clear point of reference. formality vs. informality an interesting emergent theme was the juxtaposition of formality against informality. on a number of occasions during the project, inconsistency between the less formal approach in certain contexts and the adherence to more rigid, formal hierarchies resulted in difficulties for both staff and volunteers. in particular this occurred when the volunteers were used in a customer service role as exhibition stewards. four volunteers recounted incidents concerning disagreements with library patrons. common among these incidents were volunteers attempting to deal with problems which, following library protocol, should have been referred to the library staff member on duty. as one member of staff stated: they still have to represent the service in the same way that a paid member of staff would do, and i think again that’s a tricky area. (member of staff c) one particular incident involved a volunteer disagreeing with the library first-aider over the treatment of a visitor at the exhibition who unexpectedly felt dizzy. common to most policybound public sector workplaces, the first-aider was expected to follow a set procedure which involved calling an ambulance, which was deemed totally unnecessary by the volunteer. although an isolated incident, this exemplifies the potential difficulty of involving those employed on an informal basis in contexts where strict adherence to protocol is expected. relative to this issue is the difficulty in using suitable disciplinary measures. the incident with the first-aider related above resulted in the volunteer being “reprimanded.” the difficulty here was that this contrasted starkly with the hitherto informal relationship between staff and volunteer, and led the volunteer to believe the project was “badly managed” (volunteer b). one of the tensions of volunteer management is that the relationship between volunteer and manager is underpinned by a reciprocal relationship which is not the same as paid employee relations: how much can you ask of them, you know? (member of staff c) zimmeck (2000) and gaskin (2003) have argued that using the same methods of interview, recruitment, contracts, and disciplinary procedures with volunteers is fundamentally flawed: they work, but they are not employees: they do not have to do what they do: they do it in more episodic, circuitous and idiosyncratic ways: they are not paid for doing it: and, if they do not feel that they are properly involved, supported or cherished, they will walk away. (zimmeck, 2000, p. 5) she reflects that a “home grown” model should be more widely adopted, which recognizes the differences between volunteers and employees, but treats them as partners who participate in decision making by consensus and exercise shared authority. this is qualified by recognizing that there is no one size fits all model, but the ethos of understanding what volunteers want and need is the preferred basis from which to manage volunteers. gaskin’s research explored this evidence based library and information practice 2012, 7.1 49 further, proposing a model of progressive volunteer involvement, concluding what volunteers wanted from management was to feel welcome, secure, accepted, respected, informed, well-used, and well-managed (2003). as a result of these reported incidents it was suggested by project staff that a formalization of the volunteer process in implementing policies around wider library management would have been beneficial: maybe if we had some policies in place regarding volunteers operating alongside other members of staff, volunteers being alongside members of the public. looking back on that it probably would have helped. (member of staff c) volunteer responses to the issue of formalizing their relationship with the library service suggest that this might run the risk of undermining the inherent freedom and reciprocity perceived to be characteristic of volunteering. whilst volunteer agreements are advocated as a means of ensuring commitment (cookman et al., 2000) many of the walter gardiner photographic project volunteers demonstrated the potential for the opposite to be true: we all feel more duty bound to complete it and do it properly than we did when we were in paid employment. (volunteer f) in some ways if i feel i’ve got the freedom then i’m more dedicated than if i feel more constricted by something. (volunteer c) volunteer-organization relationship a further significant theme was the ambiguous position of the volunteers within the wider library service and the importance this played in the success of the public aspect of the project. the three volunteers who specifically mentioned staff relationships outside of the immediate project staff were positive about those relationships: they treated us as part of the team. they were never superior or patronising we always felt we were wanted. (volunteer h) the majority of volunteers perceived other library staff as friendly and welcoming. however, evidence from a staff perspective arose suggesting that the introduction of volunteers was not as unproblematic as might be supposed from volunteer accounts: yes and it came from the staff themselves [reinforcing separation]. you know, “who’s that lot downstairs?” you have to feel both sides there. make the staff realise they [the volunteers] are working for us and not getting paid so they’re great . . . and at the same time make the volunteers feel like they’re a part of our team and not excluded. (member of staff c) seven of the ten volunteers reported feeling “not completely part of the library,” whilst one volunteer reflected: name badges might have been a good idea. . . . no matter what level you work on it’s nice to feel which position you’re in. (volunteer e) as documented in gaskin’s (2003) work, the importance of volunteers feeling like they belong within an organization is key to a successful experience for both parties. in this project the self-perceived and staff attributed “otherness” of the volunteers proved to be problematic once the project moved to a customer focused level where volunteers were expected to be aware of and supportive of the organizational culture. recommendations recommendations to library services considering use of volunteers are divided into the three themes: interviewing, training, and management. as discussed above, these recommendations stem from the results of this local study, but may have transferability to similar library projects. at the interview stage, it is useful to maintain informality yet be evidence based library and information practice 2012, 7.1 50 rigorous in discussing volunteers’ motivations, as this will aid in allocating appropriate tasks and thus maximising commitment. training is a key aspect throughout any project; it is perhaps self-evident that induction sessions need to be welcoming, but they also need to facilitate volunteers’ understanding and establishment of their place within the organization. a key recommendation for induction sessions is clarifying paid staff roles, so volunteers know how to access appropriate assistance. induction and ongoing training needs to be professionally designed and delivered, so a clear message is given about the investment a service is willing to make in their volunteers. this can also help to facilitate a professional ethos amongst volunteers, and be used as a forum to introduce the ethos and management practices of the service. if there are specific codes of conduct expected, they should be explained (e.g., health and safety procedures). it is important to be responsive to the ongoing training needs of volunteers and be alert to their initial, developing, and perhaps changing needs. maintaining the informality of the volunteer experience and making it less like a work experience can be key to a project’s success, particularly with certain volunteers (e.g., older, retired volunteers who are not looking to gain “career” skills; statistically likely to be the majority of volunteers). integrating volunteers within the wider organization fosters a sense of community, which can be achieved by inviting volunteers into staff areas and to staff meetings. developing a “volunteer agreement” which sets out mutual responsibilities and expectations (rather than being prescriptive or restrictive) can help with the clarification of roles. expectation of time commitment and hours should be addressed, but this should be done sensitively. the freedom inherent in volunteering is very important for some volunteers, and something that erodes this might be counterproductive. as part of this, establish a procedure for volunteers to have a point of contact elsewhere in the organization (outside of the immediate project management team) to deal with complaints, similar to a grievance officer for paid staff. conclusions the key success factors in this project can be summed up as: good volunteer recruitment strategies at the initial stage of the project, providing training of sufficient quality and depth, and a friendly and informal management style. despite these successes a number of tensions arose – largely as a result of the project moving away from its selfcontained dimension into a more public arena. the volunteers’ lack of familiarity with the wider library organization protocol (e.g., the managerial structures and hierarchy), and the huge importance placed on customer care and public service proved problematic. future projects can learn from this by implementing some of the recommendations listed above, particularly the integration of volunteers into the wider service “ethos.” on a wider policy level, the findings supported the use of a volunteer agreement which would clarify volunteer and organization expectations, but showed clearly that the freedom of choice underpinning the volunteer-organization relationship was of great significance. as such, the key to success is to find the right balance between formality and informality to satisfy both parties. this research is limited by the scale and locality of the project. future research could further explore the role volunteer motivations play in the success or otherwise of a project. future research will also likely need to address the issue from the perspective of a greatly altered policy landscape, which may bring significant changes to the nature and scale of volunteering in libraries. however these develop, it is likely that libraries will need more resources and guidance to support a potentially expanding set of volunteer staff. there may be a role for dedicated paid staff to develop, manage, and train volunteers within contexts where professional librarians are coordinating – and perhaps co-managing – their services with entirely volunteer-run ones. evidence based library and information practice 2012, 7.1 51 references capital planning information limited. (2000). volunteers in library and information services: a report to the library and information commission. library and information commission research report, 53, 75 chartered institute of library and information professionals. (2010). use of volunteers in public libraries. in cilip. retrieved 18 feb. 2012 from http://www.cilip.org.uk/getinvolved/policy/statements/pages/useof-volunteers.aspx cookman, n., haynes, d., & streatfield, d. (2000), the use of volunteers in public libraries: a report to the library association. london: library association. cookman, n. (2001). volunteers, a way of encouraging active community participation? library and information research news, 25(81), 8-11. denscombe, m. (2010). the good research guide: for small-scale social research projects (4th ed.). maidenhead, england: mcgraw-hill/open university press. downey, a., kirby, p., & sherlock, n. (2010). payment for success – how to shift power from whitehall to public service customers. in kpmg in the united kingdom. retrieved 18 feb. 2012 from http://www.kpmg.co.uk/pubs/204000 %20payment%20for%20success%20a ccess.pdf flood, a. (2004).using volunteers in camden local studies and archives. local studies librarian, 23(1), 13-15. future libraries programme. (2010). in department for culture media and sport. retrieved 28 feb. 2012 from http://www.culture.gov.uk/news/news _stories/7381.aspx gale, k. (1999). more than just a helping hand. library association record, 101(8), 471. gaskin, k. (2003). a choice blend: what volunteers want from organisation and management. london: the institute for volunteering research. retrieved 26 feb. 2012 from http://www.ivr.org.uk/images/stories/i nstitute-of-volunteeringresearch/migratedresources/documents/c/choice-blendreport-for-printing.pdf gay, p. (2001). bright future: developing volunteer management. london: the institute for volunteering research. retrieved 26 feb. 2012 from http://www.ivr.org.uk/images/stories/ volunteeringengland/documents/publications/brig htfuture.pdf gorman, g. e., clayton, p., rice-lively, m. l., & gorman, l. (1997). qualitative research for the information professional: a practical handbook. london: library association. howlett, s., machin, j., & malmersjo, g. (2005). volunteering in museums, libraries and archives. in the museums libraries & archives council. retrieved 18 feb. 2012 from http://research.mla.gov.uk/evidence/d ocuments/volunteer_survey_2006_950 0.pdf jervis, b. (2000). unpaid volunteers: burden or bonus? public library journal, 15(1), 1516. library association. (2001). involving volunteers in public libraries: guidelines. in library association, uk, archive site. retrieved 18 feb. 2012 from http://www.libraryassociation.org.uk/ directory/ivpl.pdf local government group. (2011). future libraries: change, options and how to evidence based library and information practice 2012, 7.1 52 get there. in the museums libraries & archives council. retrieved 18 feb. 2012 from http://www.mla.gov.uk/what/program mes/~/media/files/pdf/2011/libraries/f uture-libraries-report lock, a. (1994). living memories of hyde: using volunteers in an oral history project. local studies librarian, 13(2), 11-15. locke, m., ellis, a., & smith, j. d., (2003). hold on to what you’ve got: the volunteer retention literature. institute for volunteering research 5(3), 81-100. mccune, b. f., & nelson, c. (1995). recruiting and managing volunteers in libraries: a how-to-do-it manual. new york: nealschuman publishers. mccurley, s., & lynch, r. (1998). essential volunteer management (rev. 2nd ed.). london: directory of social change. mcdiarmid, m., & auster e. w., (2004). volunteers @ your library: benefits and pitfalls of volunteers in hospital libraries. journal of the canadian health libraries association, 25(1), 5-10. doi:10.5596/c04-005 miles, m. b., & huberman, a. m., (1994). qualitative data analysis: an expanded sourcebook (2nd ed.). thousand oaks, ca: sage publications. museums, libraries and archives council. (2011). community managed libraries. in the museums libraries & archives council. retrieved 17 feb. 2012 from http://www.mla.gov.uk/what/policy_d evelopment/communities/~/media/0f a94424234f4eeba40866bcb5c6f840. ashx 9accessed 26/08/11) museums, libraries and archives council. (2010). trends from cipfa public library. in the museums libraries & archives council. retrieved 17 feb. 2012 from http://www.mla.gov.uk/news_and_vie ws/press_releases/2010/~/media/files/ pdf/2010/research/trends_from_cipf a_public_library_service-dec-2010 nicol, e. a., & johnson, c. m. (2008). volunteers in libraries: program structure, evaluation, and theoretical analysis. reference & user services quarterly, 48(2), 154-163. retrieved 17 feb. 2012 from http://www.rusq.org/wpcontent/uploads/2008/48n2/pdf/nicol.p df pickard, a. j. (2007). research methods in information. london: facet publishing. roy, l. (1988). the use of volunteers in public libraries: a pilot study. public library quarterly, 8(1/2), 127-145. doi:10.1300/j118v08n01_13 stake, r. e. (1995). the art of case study research. thousand oaks, ca: sage publications. volunteering england. (2011). volunteer interviews. london: the institute for volunteering research. zimmeck, m. (2000). the right stuff: new ways of thinking about managing volunteers. london: the institute for volunteering research. retrieved 26 feb. 2012 from http://www.ivr.org.uk/images/stories/i nstitute-of-volunteeringresearch/migratedresources/documents/r/the_right_st uff.pdf / evidence based library and information practice/ / evidence based library and information practice 2012, 7.3 77 evidence based library and information practice evidence summary pleasure reading offers educational, social, and personal benefits for young teenagers a review of: howard, v. (2011). the importance of pleasure reading in the lives of young teens: self-identification, self-construction and self-awareness. journal of librarianship and information science, 43(1), 4655. doi:10.1177/0961000610390992 reviewed by: ann medaille reference & instruction librarian university of nevada, reno reno, nevada, united states of america email: amedaille@unr.edu received: 29 may 2012 accepted: 15 july 2012 2012 medaille. this is an open access article distributed under the terms of the creative commons‐ attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to investigate the role that pleasure reading plays in the lives of young teenagers. design – a series of focus groups were used. setting – focus groups were held in nine junior high schools in an eastern canadian municipality. subjects – participants consisted of 68 students in grades 7, 8, and 9, ranging in age from 12-15 years old. seventy percent of participants were girls and 30% were boys. methods – a semi-structured interview protocol was used. responses were coded and categorized by using qsr nud*ist, and a grounded theory approach was used to analyze the data. main results – this study found that young teenagers derive numerous benefits from pleasure reading. from an educational perspective, pleasure reading helps improve literacy and thinking skills, and helps young teenagers clarify and explore career goals. from a social perspective, pleasure reading helps young teenagers understand historical and current events, helps them develop compassion and empathy, empowers them to develop and act on their beliefs, and helps them to understand the consequences of risky behaviors. from a personal perspective, pleasure reading provides young teenagers mailto:amedaille@unr.edu evidence based library and information practice 2012, 7.3 78 with entertainment, relaxation, reassurance, a creative outlet, and a means of escape. conclusion – reading for pleasure provides a means of everyday life information seeking for young teenagers. it helps them improve skills and learn about themselves, their relationships, and their values, all of which help them to make the transition to adulthood. commentary this study elaborates on the benefits of reading for pleasure (also called pleasure reading or recreational reading or voluntary reading) and describes how those benefits operate within the lives of young teenagers (ages 12-15). this work is consistent with the results of other studies that show that the impact of pleasure reading is considerable. for example, the national endowment for the arts’ (nea) to read or not to read: a question of national consequence reports on the numerous correlations that exist between voluntary reading and academic and career success. the nea also describes some of the benefits of pleasure reading and reports on the troubling decline in reading that often occurs during the teenage years. because of this decline, it is important for librarians and educators to be able to better advocate for the importance of reading and to help teenagers understand the benefits that reading can provide to them. howard’s study is significant for its focus on this age group and its identification of the role that reading plays in the process of maturing to adulthood. the researcher collected information from both active readers (85% of participants) and reluctant readers (15% of participants), and this variety of participants may have added to the diversity of the responses. by collecting data through focus groups, howard enables young teenagers to discuss the advantages of reading from their own points-of-view and by using their own language. although the focus of this study was limited to a single municipality, the researcher randomly selected the schools and classes to be targeted. the use of focus groups is appropriate for this age group and is well-suited to an exploratory study that is designed to gather a range of different thoughts, ideas, and feelings about the benefits gained from pleasure reading. in addition, the author made an appropriate choice in using a grounded theory approach to the data, which enabled a hypothesis about the benefits of pleasure reading to emerge from the data collection process itself, rather than being formulated beforehand. however, this study is also limited in its use of the focus group methodology, which restricts the kinds of conclusions that can be made from the research. thus, no correlations can be drawn between certain reading benefits and age, gender, reader type, reader habits, or other characteristics. this study emphasizes that pleasure reading serves several functions for young teenagers – functions that go beyond the provision of mere entertainment. the author points out that, for young teenagers, pleasure reading can be understood in the context of everyday life information seeking, a model which emphasizes the social and psychological factors that influence information-seeking choices in everyday contexts (savolainen, 2005). thus, young teenagers do not intend to seek information when reading for pleasure; rather, they acquire information as a secondary benefit of this activity. in addition, this study provides support for reader response theory, which emphasizes the active role of the reader in constructing meaning from texts (ross, 2005). in accordance with reader response theory, this study argues that works written for teenage readers should not be diminished because their primary function is to provide entertainment. rather, “active critical reading can take place with any text as stimulus” (p. 54) because readers bring their range of experiences, interests, ideas, feelings, and needs to the act of reading. overall, this study is theoretically solid and carefully executed, and the study results have been clearly presented in the paper. while the author eloquently describes the role of reading in the childhood-to-adulthood transition, the strength of this study lies in its usefulness for evidence based library and information practice 2012, 7.3 79 librarians and other educators who serve this population and who advocate for the benefits of reading. not surprisingly, teenagers’ views of reading differ from those of adults, and librarians and other educators would benefit from understanding the teenage perspective. for youth librarians, this study can certainly aid in decisions related to programming and collection development, but even more important, it can also help to improve their conversations with teenagers about the importance of reading. references national endowment for the arts (2007). to read or not to read: a question of national consequence. retrieved 1 august 2012 from http://www.nea.gov/research/toread. pdf ross, c. s. (2005). reader response theory. in fisher, k. e., erdelez, s., & mckechnie, l. (eds.), theories of information behavior (pp. 303-307). medford, nj: information today. savolainen, r. (2005). everyday life information seeking. in fisher, k. e., erdelez, s., & mckechnie, l. (eds.), theories of information behavior (pp. 143148). medford, nj: information today. http://www.nea.gov/research/toread.pdf http://www.nea.gov/research/toread.pdf evidence based library and information practice 2012, 7.1 96 evidence based library and information practice evidence summary local public libraries serve important functions as meeting places, but demographic variables appear significant, suggesting a need for extensive further research a review of: aabø, s., audunson, r., & vårheim, a. (2010). how do public libraries function as meeting places? library & information science research, 32(1), 16-26. doi: 10.1016/j.lisr.2009.07.008. reviewed by: r. laval hunsucker information and collection specialist emeritus universiteit van amsterdam libraries silversteyn 80, breukelen, the netherlands email: amoinsde@yahoo.com received: 30 nov. 2011 accepted: 31 jan. 2012 2012 hunsucker. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – the investigators hoped to gain an understanding of the extent to which local public libraries are used by their visitors as meeting places, and in what ways. furthermore, they sought to determine whether certain demographic variables correlate with variations in these ways of using the library. finally, they were looking for evidence of a relationship between the degree of the subjects’ general community involvement on the one hand, and their participation in various types of meetings in the library on the other. design – questionnaire-based telephone survey. setting – oslo, norway. subjects – 750 adult residents (eighteen years or older) from 3 of oslo’s 15 boroughs. methods – the researchers selected these boroughs (not identified in this article and referred to, unusually, as “townships”) because they judged them to represent three demographically varying types of urban community. in march of 2006, a professional survey organization drew numbers at random from a database of telephone numbers in each borough, continuing until it had reached the desired number of 250 actual survey respondents, including cell phone users, for each borough. it weighted the sample according to gender and age, and administered the telephone interviews on the basis of a questionnaire which the researchers had evidence based library and information practice 2012, 7.1 97 designed to yield quantitative data for ten independent, and seven dependent, variables. interviewers asked the respondents to answer questions on the basis of their entire recollected personal history of public library use, rather than during a specific defined period. six of the independent variables were demographic: borough of residence, occupational category, age category, educational level, cultural/linguistic background (dichotomous: either nonnorwegian or norwegian), and household income category. the other four were: level of participation in local activities, degree of involvement in community improvement activities, degree to which a subject trusted various community institutions, and frequency of local library use. “meeting intensity,” or the number of different meeting types for which a given subject could remember ever having used the library, was one dependent variable. the others were participation/nonparticipation in each of the six defined meeting types. the researchers employed hierarchical multiple regression analyses for determining degrees of correlation. main results – “meeting intensity” correlated significantly and positively not only with frequency of library use in general, but also with the number of local activities participated in and level of involvement in community improvement activities, as well as with nonnorwegian cultural/linguistic background. it correlated significantly and negatively with household income. the investigators report no significant relationship of meeting intensity with occupational or age category, or with level of education. participation in certain of the defined meeting types did correlate significantly with certain independent variables. respondents tend to turn to the local public library more for “public sphere” meetings as they grow older. participation in this kind of meeting is likewise more common among those with a higher level of community involvement and engagement, but also among the lower-income respondents. high-intensive “joint activities” meetings with friends, acquaintances, colleagues or classmates are especially popular among adults in the lower age categories, as well as among respondents with a lower level of education and with a lower household income. “virtual” meetings (via library internet use), also defined as a high-intensive meeting type, are especially popular with the occupational categories “job seeker” and “homemaker,” as well as with the younger respondents and with those who have a lower household income. use of the local public library for both the “virtual” and the “joint-activities” types of meetings is also considerably more common among those with a non-norwegian cultural/linguistic background. frequency of library use in general was not related to participation in either of these two types of meetings at the library, but it was related to library use for the more low-intensive meeting types (chance meetings and encounters, library as rendezvous point for joint activities elsewhere), as well as to what the investigators term using the library as a “metameeting place,” i.e., a place for finding “information about other arenas and activities” in the local community. conclusion – the local public library seems to serve, for many of its patrons, an important function as venue for meetings of various kinds. in general, using it for meeting purposes appears to be something that appeals more to younger than to older adults, more to those in the lower than to those in the higher income categories, and more to those with an immigrant than to those with an indigenous background. the perhaps even less expected finding that use of the library for a relatively intensive, instrumental kind of meeting activity correlates significantly with a lower level of education would particularly suggest a need for further research. noteworthy, as well, is the apparent fact that those who make use of the local public library as a venue for relatively intensive meeting activity, whether physical or virtual, tend to come to the library expressly for that purpose, and visit the library less often for other reasons than do other library users. the urban districts in which respondents resided were in fact not internally homogeneous enough, nor socio-economically evidence based library and information practice 2012, 7.1 98 distinct enough from one another, to yield correlations of practical evidentiary value. it was the researchers’ working assumption that their three independent variables of community engagement – i.e., level of participation in local activities, degree of involvement in community improvement activities, and degree to which one trusts community institutions – can be taken together to represent the amount of a respondent’s “social capital.” they detected, in general, a positive correlation between the extent of such “social capital” and the use of the library as a meeting place. neither the strength nor the direction of this relationship was clear, however, from the results of this study: both will have to be explored through further research. “does the library contribute to generating social capital,” they ask, “or is the use of the library as a meeting place a result of pre-existing social capital?” (p. 25) they were hoping at least to discover whether the library, specifically in its role as a low-intensive and “public sphere” meeting place, contributes to the generation of “bridging” social capital between citizens of differing cultural backgrounds, with differing values, viewpoints, and interests. though their findings did not justify this conclusion, and skøtt’s (2005) study even contradicts it, the researchers nevertheless express their confidence that, while not a genuine “third place” in the sense intended by oldenburg (1999), “the library as a meeting place plays a substantial role in equalizing the possibilities of being an active citizen across social and economic differences” (p. 25). but however that may be, they are in any case convinced that their questionnaire and categorization scheme for meeting types have now shown their value, and that the grouping of types into “low-intensive” versus “high-intensive” appears to be fruitful. they do concede that their approach still requires more thorough and detailed examination, and that their survey instrument must be further refined and developed. commentary it is likely that this study will prove to be of more practical use to researchers than to library practitioners. in particular, its findings regarding public library use patterns by nonindigenous, by less-educated, by lowerincome, and by younger patrons are of interest but will require extensive testing through additional research, in other settings as well as with other, especially qualitative, methods. highly significant, too, is this study’s implicit conclusion that for many who make relatively intensive and instrumental use of the library as place, it would indeed seem still to be an important community institution although presumably not by virtue of the functions and services more traditionally associated with it. strangely, that conclusion is not one which the authors themselves formulate, although their data, like those of abm-utvikling (2008), strongly indicate that it is justified, as in fact høimyr (2011) likewise suggests. indeed, it is remarkable that this study takes no account whatsoever of those functions and services, nor of certain other factors which can clearly have a considerable influence on whether, the extent to which, and in what ways, patrons will use a public library as a meeting place. we are told nothing about these local libraries’ collections, facilities, size, staffing, or services, and what role those might play. there is no mention of the libraries’ policies and procedures, or the extent to which those may be aimed at facilitating or even encouraging the use of library premises for various meeting purposes. we hear nothing, either, about architecture, arrangement, and design, though, as van slyck has argued, “a [library] building’s plan determines which interactions ... are possible and which are impossible,” and the qualities of its interior spaces as well as its furnishings encourage “users to play certain sanctioned roles, while making others seem unthinkable” (2007, p. 221). surely these factors are, as many have indeed suggested (breeding, 2011; cox, swinbourne, pip, & laing, 2000; johnson, 2010; klopfer & nagata, 2011; ljødal, 2005; may & black, 2010; preer, 2001; servet, 2010; sin & kim, 2008; vårheim, steinmo, & ide, 2008; wahnich, 2011), far from irrelevant to research regarding libraries’ use as meeting places? was it as libraries that the evidence based library and information practice 2012, 7.1 99 respondents found these libraries to be suitable/attractive/meaningful as meeting places? could some or all of the types of meeting just as well have taken place in something other than a library? if not, why not? what specifically, or what combination of factors, makes a library appealing as a meeting place? since the present study does not speak to any of these matters, it is difficult to assess to what degree its results actually amount to evidence which might be of use to practicing librarians. moreover, the sample was not entirely representative; subject self-selection and selfreporting biases were present, as was a degree of language bias; and, oddly, the investigators neglected to establish, even approximately, how often any given respondent had used the library either for meeting purposes altogether or for any specific meeting type. they do not provide a copy of the questionnaire employed. this reader could nonetheless scarcely avoid the impression that some survey questions were multi-interpretable, and some terminology – even some variables – imprecisely defined. this renders it all the more regrettable that the researchers chose to employ a single-method, rather than a triangulated, research design. how can we adequately measure the societal performance, or calculate the social and community value, of a public library? this is clearly a multifaceted and still unresolved question, but nevertheless an important and probably increasingly crucial one (calvert, 1994; debono, 2002; imholz & arns, 2007; klopfer & nagata, 2011; koontz, jue, & lance, 2005; linley & usherwood, 1998; richter, 2011; state library, 2005; wiegand, 2003). in spite of their study’s limitations, aabø and her colleagues quite rightly recognized that at least one important aspect, the library’s actual use as meeting place, had hardly ever been the subject of any empirical research or analysis. their and the abm-utvikling’s (2008) findings on this specific aspect now amount in any case to a welcome, albeit modest, beginning on which further research can build. in the meantime, their study does already at least strongly suggest that the twenty-first century public library has indeed begun to attract a somewhat differently constituted clientele, serving it in a manner different to the traditional and familiar one. and that in consequence, as some others have already observed (alstad & curry, 2003; bonrepaux, 2010; breemer, 2011; cox et al., 2000; shoham & yablonka, 2008; wahnich, 2011; wiegand, 2003), librarians may well have little choice but to adjust their thinking and their practices accordingly. references abm-utvikling (2008). hvem er de og hvor går de? om brukeradferd i norske storbybibliotek. abm-skrift 46. oslo: abm-utvikling. retrieved 30 jan. 2012 from http://www.abmutvikling.no/publisert/abmskrift/hvem-er-de-og-hvor-gar-de-1 alstad, c., & curry, a. (2003). public space, public discourse, and public libraries. libres: library and information science research electronic journal, 13(1). retrieved 30 jan. 2012 from http://libres.curtin.edu.au/libres13n1/p ub_space.htm bonrepaux, c. (15 dec. 2010). le bibliothécaire, un mutant: entretien avec dominique arot. le monde, supplément “le monde éducation”, 20495, 11. breeding, m. (2011). using technology to enhance a library as place. computers in libraries, 31(3), 29-31. breemer, a. v. d. (28 oct. 2011). de beste biebs zitten er bovenop. de volkskrant, 90(26592), v4-v5. calvert, p. j. (1994). library effectiveness: the search for a social context. journal of librarianship and information science, 26(1), 15-21. doi: 10.1177/096100069402600103. cox, e., swinbourne, k., pip, c., & laing, s. evidence based library and information practice 2012, 7.1 100 (june 2000). “a safe place to go”: libraries and social capital. sydney, australia: university of technology, sydney; the state library of new south wales. retrieved 30 jan. 2012 from http://www.sl.nsw.gov.au/services/pu blic_libraries/docs/safe_place.pdf debono, b. (2002). assessing the social impact of public libraries: what the literature is saying. australasian public libraries and information services, 15(2), 80-95. høimyr, t. (2011). east and west oslo: differences in library usage? a survey of two local libraries. scandinavian public library quarterly, 44(2), 22-23. retrieved 30 jan. 2012 from http://splq.info/issues/vol44_2/11.htm imholz, s., & arns, j. w. (2007). worth their weight: an assessment of the evolving field of library valuation. new york: americans for libraries council. retrieved 30 jan. 2012 from http://www.ila.org/advocacy/pdf/wort htheirweight.pdf johnson, c. a. (2010). do public libraries contribute to social capital? a preliminary investigation into the relationship. library & information science research, 32(2), 147-155. doi: 10.1016/j.lisr.2009.12.006. klopfer, l., & nagata, h. (2011). contextual approach to understanding the socio cultural function of a public library in japan. performance measurement and metrics, 12(1), 66-77. doi: 10.1108/14678041111124306. koontz, c. m., jue, d. k., & lance, k. c. (2005). neighborhood-based in-library use performance measures for public libraries: a nationwide study of majority–minority and majority white/low income markets using personal digital data collectors. library & information science research, 27(1), 28-50. doi: 10.1016/j.lisr.2004.09.003. linley, r., & usherwood, b. (1998). new measures for the new library: a social audit of public libraries. british library research & innovation centre report 89. london: british library board. retrieved 30 jan. 2012 from http://www.sheffield.ac.uk/polopoly_f s/1.128118!/file/cplis---newmeasures-for-the-new-library.pdf ljødal, h. k. (2005). folkebiblioteket som offentlig møteplass i en digital tid. abm-skrift 18. oslo: abm-utvikling. retrieved 30 jan. 2012 from http://www.abmutvikling.no/publisert/abmskrift/folkebibl_moteplass.pdf may, f., & black, f. (2010). the life of the space: evidence from nova scotia public libraries. evidence based library and information practice, 5(2), 5-34. retrieved 30 jan. 2012 from http://ejournals.library.ualberta.ca/ind ex.php/eblip/article/view/6497/6958 oldenburg, r. (1999). the great good place: cafés, coffee shops, bookstores, bars, hair salons and other hangouts at the heart of a community. new york: marlowe. preer, j. (2001). where are libraries in bowling alone? american libraries, 32(8), 60-62. richter, s. (2011). die bibliothek als ort und raum: verfahren zur wirkungsmessung. information: wissenschaft & praxis, 62(5), 225-236. servet, m. (2010). les bibliothèques troisième lieu: une nouvelle génération d'établissements culturels. bulletin des bibliothèques de france, 55(4), 57-63. shoham, s., & yablonka, i. (2008). implications of monumental construction for public library services. libri, 58(1), 34-46. doi: 10.1515/libr.2008.004. sin, s.-c. j., & kim, k.-s. (2008). use and non use of public libraries in the information age: a logistic regression evidence based library and information practice 2012, 7.1 101 analysis of household characteristics and library services variables. library & information science research, 30(3), 207-215. doi: 10.1016/j.lisr.2007.11.008. skøtt, b. c. (2005). ethnic diversity in danish public libraries: four stories. in c. g. johannsen & l. kajberg (eds.), new frontiers in public library research (pp. 185-210). lanham, md: scarecrow. state library of victoria (feb. 2005). libraries building communities: the vital contribution of victoria’s public libraries – a research report for the library board of victoria and the victorian public library network: report one: setting the scene. melbourne, australia: state library of victoria. retrieved 30 jan. 2012 from http://www2.slv.vic.gov.au/pdfs/about us/publications/lbc_report1.pdf van slyck, a. (2007). managing pleasure: library architecture and the erotics of reading. in j. e. buschman & g. j. leckie (eds.), the library as place: history, community, and culture (pp. 221-234). westport, ct: libraries unlimited. vårheim, a., steinmo, s., & ide, e. (2008). do libraries matter? public libraries and the creation of social capital. journal of documentation, 64(6), 877-892. doi: 10.1108/00220410810912433. wahnich, s. (2011). à quoi sert une bibliothèque? bulletin des bibliothèques de france, 56(2), 23-26. wiegand, w. a. (2003). to reposition a research agenda: what american studies can teach the lis community about the library in the life of the user. the library quarterly, 73(4), 369-382. / evidence based library and information practice feature   eblip8 delegate reflections      2015. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   ruby warren, user experience librarian, university of manitoba libraries   i attended eblip8 in brisbane, hosted by the queensland university of technology, and as a first eblip experience it couldn’t have been more delightful. the university was a good venue, with rooms and sessions easy to identify and locate, and our host volunteers were both numerous and helpful. the university being located near the city botanic gardens was an unexpected bonus, as the brisbane winter weather was perfect for a canadian like me to enjoy some lunchtime sunshine under the banyan trees. the session area being attached to the cube, a unique arrangement of screens, visualizers, and projectors, was an excellent choice it served as a unique method of poster display, and during session breaks it was refreshing to watch local children and parents play with the visualizers. i can only hope watching foreign adults like me dance in front of the cameras was just as amusing for the other attendees.   the sessions themselves were interesting, and i found the keynotes from virginia wilson and kim tairi invigorating, with a sense of playfulness that is often sorely missed during research-oriented enterprises. virginia’s exploration of the strange position of the practitioner-researcher was an excellent conference opener, one that encouraged me to examine the supports we need and the supports we have in our unique role as someone with work to do and a need to study that work. kim’s encouragement of sketchnoting gave me an important new conference tool, and her exploration of different ways that evidence-based practitioners and researchers manage time and achieve work-life balance (or more accurately, work-other work-more work-life balance) was helpful to any librarian struggling with an administrative load in their move towards evidence-based practice.  in addition to these excellent keynotes, i strongly appreciated the addition of practical, skills-acquisition oriented workshops to the conference program, and was incredibly impressed with my progress in the visualizing the evidence workshop hosted by kim davis. her guidance gave me a stronger perspective on how to visualize data i’ve collected, and the practical resources to create those visualizations.   by far, the best thing about attending eblip8 was the warmth and engagement of the attendees. after arriving knowing hardly anyone, i became collegial and friendly with a large number of participants, and had a wonderful time sharing opinions and research ideas with everyone there. the conference’s strong twitter presence had a level of positivity and fun that is hard to achieve on conference hashtags, and it created a feeling of total engagement and interest in conference sessions and presentations. my thoughts about a presentation can sometimes escape by the time we get to coffee and tiny cakes, and being able to take notes and bounce ideas with others during the presentation itself made the ideas we explored even clearer and more exciting. this community is wonderful, and i’ll definitely be planning attendance to eblip9 in 2017.   megan fitzgibbons, librarian, university of western australia   it’s perhaps an unprofessional way to phrase it, but eblip8 was, in my humble opinion, the best conference ever. it was my first time at an eblip conference, and i felt it was a unique gathering because attendees not just interested in ‘topics’, but rather we are all interested in ways of working and of understanding our profession. it really was a ‘community of practice.’ like any good conference, the discussions transcended any single session and several themes threaded through many of the presentations, leading to a rich dialogue across the event and beyond, in person, on twitter, and through the lark (library applied research kollektive) group.   virginia wilson’s opening keynote prompted me to think deeply about the importance of evidence-based library and information practice and effectively set the stage for my learning through the rest of the conference. for me, one of her most important messages was the accessibility of eblip. as she put it, eblip isn’t a ‘special occasion kind of thing’; it should be integrated as a part of daily practice. further, research is a way of knowing that is relevant to all of us and should therefore be part of our jobs. eblip, she argued, is supported by management and organisational structures, but more importantly, it is fueled by personal work ethic, dedication, and curiosity. instead of completing a project and filing it away in a drawer, an eblip-inspired librarian in virginia’s characterisation wants to know more: did it work? what did the clients think? how does this fit with what we know about information practice theory? i will certainly cultivate this ethic of constantly questioning going forward—because as kim tairi put it in her closing keynote, eblip empowers us to make a difference in our organisations and communities.   the conference also prompted reflection on the nature of research and its relationship to practice. there was a useful emphasis on projects that focused on workplace evaluation, but at the same time, when conducting research, i think that we should be careful to always consider the connections to theory of information practice. in other words, our workplace activities do not take place in isolation. the difference between research and simply ‘evaluation’ is that the former asks why and seeks to situate phenomenon within a larger body of knowledge. one key takeaway from the conference for me was the realisation that the key to evidence based practice is finding questions that matter and finding the best ways to answer them. any report back on eblip8 would be lacking without a mention of the fantastic facilities provided by queensland university of technology. the cube of course was amazing, with its animated simulation of the great barrier reef and later the conference posters, but i was equally impressed by the fact that there was a power point everywhere anyone could conceivably want one. i also had the opportunity to visit qut’s kelvin grove branch library and admire the book canopy as well as the clear consideration of students’ needs in the details of how the space is designed. inspiring!   kristie jones, library manager – qld, endeavour college of natural health, australia   gaspari, s., (2015). viewed 24 sept. 2015. retrieved from https://twitter.com/stefaniegaspari/status/618324652989886465   the colouring-in breakout activity at eblip8 summed up the entire conference for me “today is going to be awesome”. it was awesome indeed to be presenting a poster at my first international conference, held at the queensland university of technology gardens point campus in the heart of brisbane.    186 delegates attended the conference from 12 countries and a range of library and information sectors. it was clear from the presentations that evidence based library and information practice could be incorporated into any type of library – university, public, school, health or special. the variety of the presentations gave all library professionals the opportunity to learn something of interest. my own takeaways included dr carrington’s enthralling keynote on team culture, ruby warren’s funny recount of designing an evidence based intranet, charles sturt university’s models for effective research engagement, seneca college’s development and assessment of online information literacy learning objects and kim tairi’s quirky keynote on research practitioners and role conflict. over all, the sessions emphasised to me the power of eblip – that by applying a methodological approach to gathering and using evidence, it is possible to create change.   the conference was held in the science and engineering centre which included the cube, the perfect venue for holding poster presentations. it was exciting to see my own and other delegates’ posters projected around the cube’s space, and to have the opportunity to discuss my project in such a fun, informal way. tours of the cube were offered every lunch hour and included a glimpse of the science research department with its innovative break out area and solar panelled terrace.    helen partridge and her team provided a welcoming atmosphere where we were encouraged to discuss new ideas with other delegates and put these into practice. at the end of the conference, helen encouraged us to write down a plan for eblip ideas to follow up on or put in place in our own organisations, which would then be emailed to us in a few months’ time. an innovative way of ensuring we wouldn’t forget our learnings at such an inspiring conference!   brenda strachan, campus librarian (fraser coast), university of southern queensland   what’s not to love about eblip8? the conference of choice to contemplate evidence based practice from dawn til late. diverse librarians from all over the place came to brisbane to meet their mates. enthused and confident they soon relate their stories and incorporate the evidence. how did they rate? eight out of eight! evidence based library and information practice 2012, 7.2 69 evidence based library and information practice evidence summary undergraduate library instruction in the humanities increases the use of books over journals a review of: cooke, r. & rosenthal, d. (2011). students use more books after library instruction: an analysis of undergraduate paper citations. college & research libraries, 72(4), 334-343. reviewed by: mê-linh lê health sciences librarian university of saskatchewan saskatoon, saskatchewan, canada email: me-linh.le@usask.ca received: 7 feb. 2012 accepted: 7 apr. 2012 2012 lê. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to assess the impact of in-class library instruction sessions on the quantity, quality, and format of resources cited by undergraduate students. design – citation analysis and literature review. setting – a public university in the united states with approximately 9,000 undergraduate students. subjects – undergraduates in eight first-year composition i classes and five upper-level humanities classes at florida gulf coast university (fgcu). methods – this study consisted of three components. in the first, first-year students with little to no academic library experience from eight classes of first-year composition i were divided into two groups: those who received library instruction and those who did not. the instruction sessions were all taught by the same librarian, were one-hour hands-on classes held in a computer lab, and focused on basic library information, searching the catalogue, as well as searching journal databases. later in the term, the citation pages from papers submitted by the students as a class assignment were analyzed by the authors who looked at the average number of citations employed in each paper, the frequency of scholarly citations, and the frequency of source/format type (e.g., book, article, website, etc.). spss was used for data recording, mailto:me-linh.le@usask.ca evidence based library and information practice 2012, 7.2 70 storage, and to calculate statistics (although it should be noted that the authors do not include any of the descriptive statistics that can be generated by spss). in the second component, which attempted to discern if there were any differences in the citations used by students from the different disciplines, the same form of citation analyses was performed on bibliographies from upper-level students enrolled in five history, art history, art, and english classes who had participated in a library instruction session in the past. the results of the two citation analyses (composition i versus upper-level students) were then compared. the third component compared the results of the citation analyses to data extracted from five similar studies in order to determine if the fgcu findings were typical of undergraduate students or deviated from the norm. main results – the comparison of citations from the composition i students showed that students who received a library instruction session had more average citations per paper (5.3 to 3.2); used slightly more scholarly sources (51.7% to 49.4%); were much more likely to use books (25.6% vs. 6.3%) or magazines and newspapers (18.5% vs. 9.6%) as a source; and were less likely to cite journal articles (16.3% vs. 27.3%) than their counterparts who received no library instruction. students who had not received instruction were more likely to use videos (5.4% vs. 2.8%) or course texts and handouts (11.7% vs. 0%). both groups exhibited a preference for material that could be accessed online, and web sites were the most frequently cited source, accounting for nearly one-third of all citations. when the results from the composition i students who received library instruction were compared to upper-level students who had received instruction in the past, it was found that the average number of citations increased as the course level got higher (i.e., fourth year students used more citations than third year, who used more than second year, etc.). in general, the number of scholarly sources also increased as the course level did. the analysis also showed a strong preference for books over journal articles throughout all classes and course level. preference for other formats (e.g., web sites, reference sources) varied a great deal and in many cases could be attributed to the nature of the assignments. in order to determine whether the fgcu findings were typical of the undergraduate experience, the citation analyses were compared to five other institutions across the u.s. results show that the fgcu findings were similar in some aspects; two other institutions also displayed a preference for books, but usage of journal articles in upperlevel courses was either the same or lower at fgcu compared to other institutions. conclusion – for many academic liaison librarians, instruction is an important and time-consuming part of their job. the nature of many library instruction sessions – frequently one-time classes at the beginning of a semester – means instruction is often given without much attention to the impact of the session on the quality of students’ work. this study addresses this issue in order to determine whether library instructions sessions should continue at fgcu in their present format. the findings broadly indicate that library instruction has a large impact on the number of books used and the overall number of resources cited, and a very small impact on the number of scholarly sources cited. it appears that the increased reliance on books by students comes at the expense of journal articles, which were much more frequently used by students who had not received instruction. the study also found that as students progress in their studies, they cite more material and use more scholarly material. this finding is seen in a number of other citation analysis studies located through a literature search. ultimately, the authors believe that this study demonstrates the usefulness of the library sessions to students, as it causes them to cite more sources, to cite a wider variety of sources, and to cite more books. it is possible that some of the negative findings of the study, specifically related to low journal usage, may be used to alter the structure or content of future library sessions offered by fgcu librarians. evidence based library and information practice 2012, 7.2 71 commentary this study provides insight into the nature of resource use by undergraduates but there are areas of concern, such as a lack of clarity on the study design and variables unaccounted for during the results and comparison, which could have a large impact on the resulting conclusions. aspects of this study would be useful for librarians involved in planning library instruction sessions. the nature of library instruction means that sessions often leave out the important evaluative component. it is hoped, however, that this article will encourage librarians involved in teaching to carry out more evaluations on the impact of instruction on the quality of students’ work. the article raises interesting points on how content covered in class can lead to specific resource uptake (e.g., if instructors spend more time teaching about how books can be found in the catalogue and used in research, more students will cite books) and on the nature of resource use in general by undergraduates (e.g., heavy reliance on websites). however, there are a number of unanswered questions regarding study design that must be raised. specifically, how was student confidentiality ensured or how it was determined if a student has been in an instruction classes or not? for the comparison with upper-level students, how did the authors determine if these students had received instruction? furthermore, for the 400 level classes (roughly equivalent to a senior or fourth-year class) it could have been several years since they had received instruction, and therefore, the content could have been very different, or the class could include transfer students who had not received instruction at all. past sessions may not have taken place in a computer lab, been taught by a different librarian, or had an entirely different focus than what the composition i students received. however the authors draw conclusions from the comparison between the two groups without discussion of this issue. all these questions could have been easily addressed in a more detailed methods section. the authors also consistently use terminology that indicates a lack of understanding about their study design. for example, they call the instructed group their experimental group and the non-instructed group their control group. however, true experimental studies must include certain elements, including an attempt to truly randomize the groups and to make the two groups as equivalent as possible in order to minimize potential confounding variables, which the authors do not seem to have done. the main topic not addressed in significant detail involves the exact requirements of the assignments. when comparing the citations of students who had received library instruction with those that had not, it is mentioned that the papers required only a few sources (anywhere from one to around five). this variable (i.e., which composition i professors required one source versus those who required more), however, is never discussed as playing a potential role in how many resources students were citing. it is entirely possible that students in the non-library instruction group were being instructed by professors who required minimal citations, which would likely result in students including fewer citations. the authors could have dealt with this issue by examining the exact assignment requirements for both groups of students to determine whether this could have impacted the final analyses. it also appears that the authors have chosen to focus on some findings as areas of strength while downgrading the importance of others. for example, the fact that instructed students used more books is touted, but no explanation is given for why they also used more magazines and newspapers. it was not discussed whether these sources were focused on heavily in the session, nor were possible reasons explaining this trend provided. finally, it seems worthwhile to point out while the authors believe that a heavy reliance on books and less reliance on journal articles is a positive thing, this is usually only the case in humanities classes. most social sciences and evidence based library and information practice 2012, 7.2 72 life sciences classes would likely want to see the reverse in terms of books and journals cited. while the article seems to discuss all undergraduates, it is really only discussing students in the humanities. it would be interesting to know whether fgcu library sessions for non-humanities students are structured any differently. <html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/tr/rec-html40"> <head> <meta http-equiv=content-type content="text/html; charset=unicode"> <meta name=progid content=word.document> <meta name=generator content="microsoft word 15"> <meta name=originator content="microsoft word 15"> <link rel=file-list href="news_29757_v2_files/filelist.xml"> <link rel=edit-time-data href="news_29757_v2_files/editdata.mso"> <!--[if !mso]> <style> v\:* {behavior:url(#default#vml);} o\:* {behavior:url(#default#vml);} w\:* {behavior:url(#default#vml);} .shape {behavior:url(#default#vml);} </style> <![endif]--><!--[if gte mso 9]><xml> <o:documentproperties> <o:author>mallan, katrine</o:author> <o:template>normal</o:template> <o:lastauthor>hinrichs, rachel</o:lastauthor> <o:revision>2</o:revision> <o:totaltime>24</o:totaltime> <o:lastprinted>2020-03-26t20:37:00z</o:lastprinted> <o:created>2020-03-26t20:57:00z</o:created> <o:lastsaved>2020-03-26t20:57:00z</o:lastsaved> <o:pages>1</o:pages> <o:words>525</o:words> <o:characters>2998</o:characters> <o:company>microsoft</o:company> <o:lines>24</o:lines> <o:paragraphs>7</o:paragraphs> <o:characterswithspaces>3516</o:characterswithspaces> <o:version>16.00</o:version> </o:documentproperties> <o:officedocumentsettings> <o:allowpng/> </o:officedocumentsettings> </xml><![endif]--> <link rel=datastoreitem href="news_29757_v2_files/item0009.xml" target="news_29757_v2_files/props010.xml"> <link rel=themedata href="news_29757_v2_files/themedata.thmx"> <link rel=colorschememapping href="news_29757_v2_files/colorschememapping.xml"> <!--[if gte mso 9]><xml> <w:worddocument> <w:zoom>120</w:zoom> <w:spellingstate>clean</w:spellingstate> <w:grammarstate>clean</w:grammarstate> <w:trackmoves>false</w:trackmoves> <w:trackformatting/> <w:hyphenationzone>21</w:hyphenationzone> <w:punctuationkerning/> <w:validateagainstschemas/> <w:saveifxmlinvalid>false</w:saveifxmlinvalid> <w:ignoremixedcontent>false</w:ignoremixedcontent> <w:alwaysshowplaceholdertext>false</w:alwaysshowplaceholdertext> <w:donotpromoteqf/> <w:lidthemeother>de</w:lidthemeother> <w:lidthemeasian>zh-cn</w:lidthemeasian> <w:lidthemecomplexscript>x-none</w:lidthemecomplexscript> <w:compatibility> <w:breakwrappedtables/> <w:snaptogridincell/> <w:wraptextwithpunct/> <w:useasianbreakrules/> <w:dontgrowautofit/> <w:splitpgbreakandparamark/> <w:enableopentypekerning/> <w:dontflipmirrorindents/> <w:overridetablestylehps/> </w:compatibility> <w:browserlevel>microsoftinternetexplorer4</w:browserlevel> <m:mathpr> <m:mathfont m:val="cambria math"/> <m:brkbin m:val="before"/> <m:brkbinsub m:val="&#45;-"/> <m:smallfrac m:val="off"/> <m:dispdef/> <m:lmargin m:val="0"/> <m:rmargin m:val="0"/> <m:defjc m:val="centergroup"/> <m:wrapindent m:val="1440"/> <m:intlim m:val="subsup"/> <m:narylim m:val="undovr"/> </m:mathpr></w:worddocument> </xml><![endif]--><!--[if gte mso 9]><xml> <w:latentstyles deflockedstate="false" defunhidewhenused="false" defsemihidden="false" defqformat="false" defpriority="99" latentstylecount="375"> <w:lsdexception locked="false" priority="0" qformat="true" name="normal"/> <w:lsdexception locked="false" priority="9" qformat="true" name="heading 1"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 2"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 3"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 4"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 5"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 6"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 7"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 8"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 9"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 6"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 7"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 8"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 9"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 1"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 2"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 3"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 4"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 5"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 6"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 7"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 8"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 9"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="normal indent"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="footnote text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="annotation text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="header"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="footer"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index heading"/> <w:lsdexception locked="false" priority="35" semihidden="true" unhidewhenused="true" qformat="true" name="caption"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table of figures"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="envelope address"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="envelope return"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="footnote reference"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="annotation reference"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="line number"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="page number"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="endnote reference"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="endnote text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table of authorities"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="macro"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="toa heading"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number 5"/> <w:lsdexception locked="false" priority="10" qformat="true" name="title"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="closing"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="signature"/> <w:lsdexception locked="false" priority="1" semihidden="true" unhidewhenused="true" name="default paragraph font"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text indent"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="message header"/> <w:lsdexception locked="false" priority="11" qformat="true" name="subtitle"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="salutation"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="date"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text first indent"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text first indent 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="note heading"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text indent 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text indent 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="block text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="hyperlink"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="followedhyperlink"/> <w:lsdexception locked="false" priority="22" qformat="true" name="strong"/> <w:lsdexception locked="false" priority="20" qformat="true" name="emphasis"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="document map"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="plain text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="e-mail signature"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html top of form"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html bottom of form"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="normal (web)"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html acronym"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html address"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html cite"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html code"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html definition"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html keyboard"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html preformatted"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html sample"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html typewriter"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html variable"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="normal table"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="annotation subject"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="no list"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="outline list 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="outline list 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="outline list 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table simple 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table simple 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table simple 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table classic 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table classic 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table classic 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table classic 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table colorful 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table colorful 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table colorful 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 6"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 7"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 8"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 6"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 7"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 8"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table 3d effects 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table 3d effects 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table 3d effects 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table contemporary"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table elegant"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table professional"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table subtle 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table subtle 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table web 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table web 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table web 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="balloon text"/> <w:lsdexception locked="false" priority="59" name="table grid"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table theme"/> <w:lsdexception locked="false" semihidden="true" name="placeholder text"/> <w:lsdexception locked="false" priority="1" qformat="true" name="no spacing"/> <w:lsdexception locked="false" priority="60" name="light shading"/> <w:lsdexception locked="false" priority="61" name="light list"/> <w:lsdexception locked="false" priority="62" name="light grid"/> <w:lsdexception locked="false" priority="63" name="medium shading 1"/> <w:lsdexception locked="false" priority="64" name="medium shading 2"/> <w:lsdexception locked="false" priority="65" name="medium list 1"/> <w:lsdexception locked="false" priority="66" name="medium list 2"/> <w:lsdexception locked="false" priority="67" name="medium grid 1"/> <w:lsdexception locked="false" priority="68" name="medium grid 2"/> <w:lsdexception locked="false" priority="69" name="medium grid 3"/> <w:lsdexception locked="false" priority="70" name="dark list"/> <w:lsdexception locked="false" priority="71" name="colorful shading"/> <w:lsdexception locked="false" priority="72" name="colorful list"/> <w:lsdexception locked="false" priority="73" name="colorful grid"/> <w:lsdexception locked="false" priority="60" name="light shading accent 1"/> <w:lsdexception locked="false" priority="61" name="light list accent 1"/> <w:lsdexception locked="false" priority="62" name="light grid accent 1"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 1"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 1"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 1"/> <w:lsdexception locked="false" semihidden="true" name="revision"/> <w:lsdexception locked="false" priority="34" qformat="true" name="list paragraph"/> <w:lsdexception locked="false" priority="29" qformat="true" name="quote"/> <w:lsdexception locked="false" priority="30" qformat="true" name="intense quote"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 1"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 1"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 1"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 1"/> <w:lsdexception locked="false" priority="70" name="dark list accent 1"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 1"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 1"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 1"/> <w:lsdexception locked="false" priority="60" name="light shading accent 2"/> <w:lsdexception locked="false" priority="61" name="light list accent 2"/> <w:lsdexception locked="false" priority="62" name="light grid accent 2"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 2"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 2"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 2"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 2"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 2"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 2"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 2"/> <w:lsdexception locked="false" priority="70" name="dark list accent 2"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 2"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 2"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 2"/> <w:lsdexception locked="false" priority="60" name="light shading accent 3"/> <w:lsdexception locked="false" priority="61" name="light list accent 3"/> <w:lsdexception locked="false" priority="62" name="light grid accent 3"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 3"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 3"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 3"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 3"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 3"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 3"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 3"/> <w:lsdexception locked="false" priority="70" name="dark list accent 3"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 3"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 3"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 3"/> <w:lsdexception locked="false" priority="60" name="light shading accent 4"/> <w:lsdexception locked="false" priority="61" name="light list accent 4"/> <w:lsdexception locked="false" priority="62" name="light grid accent 4"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 4"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 4"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 4"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 4"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 4"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 4"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 4"/> <w:lsdexception locked="false" priority="70" name="dark list accent 4"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 4"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 4"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 4"/> <w:lsdexception locked="false" priority="60" name="light shading accent 5"/> <w:lsdexception locked="false" priority="61" name="light list accent 5"/> <w:lsdexception locked="false" priority="62" name="light grid accent 5"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 5"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 5"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 5"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 5"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 5"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 5"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 5"/> <w:lsdexception locked="false" priority="70" name="dark list accent 5"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 5"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 5"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 5"/> <w:lsdexception locked="false" priority="60" name="light shading accent 6"/> <w:lsdexception locked="false" priority="61" name="light list accent 6"/> <w:lsdexception locked="false" priority="62" name="light grid accent 6"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 6"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 6"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 6"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 6"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 6"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 6"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 6"/> <w:lsdexception locked="false" priority="70" name="dark list accent 6"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 6"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 6"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 6"/> <w:lsdexception locked="false" priority="19" qformat="true" name="subtle emphasis"/> <w:lsdexception locked="false" priority="21" qformat="true" name="intense emphasis"/> <w:lsdexception locked="false" priority="31" qformat="true" name="subtle reference"/> <w:lsdexception locked="false" priority="32" qformat="true" name="intense reference"/> <w:lsdexception locked="false" priority="33" qformat="true" name="book title"/> <w:lsdexception locked="false" priority="37" semihidden="true" unhidewhenused="true" name="bibliography"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" qformat="true" name="toc heading"/> <w:lsdexception locked="false" priority="41" name="plain table 1"/> <w:lsdexception locked="false" priority="42" name="plain table 2"/> <w:lsdexception locked="false" priority="43" name="plain table 3"/> <w:lsdexception locked="false" priority="44" name="plain table 4"/> <w:lsdexception locked="false" priority="45" name="plain table 5"/> <w:lsdexception locked="false" priority="40" name="grid table light"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light"/> <w:lsdexception locked="false" priority="47" name="grid table 2"/> <w:lsdexception locked="false" priority="48" name="grid table 3"/> <w:lsdexception locked="false" priority="49" name="grid table 4"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 1"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 1"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 1"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 1"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 1"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 1"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 1"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 2"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 2"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 2"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 2"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 2"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 2"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 2"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 3"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 3"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 3"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 3"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 3"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 3"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 3"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 4"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 4"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 4"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 4"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 4"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 4"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 4"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 5"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 5"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 5"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 5"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 5"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 5"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 5"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 6"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 6"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 6"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 6"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 6"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 6"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 6"/> <w:lsdexception locked="false" priority="46" name="list table 1 light"/> <w:lsdexception locked="false" priority="47" name="list table 2"/> <w:lsdexception locked="false" priority="48" name="list table 3"/> <w:lsdexception locked="false" priority="49" name="list table 4"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 1"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 1"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 1"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 1"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 1"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 1"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 1"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 2"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 2"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 2"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 2"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 2"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 2"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 2"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 3"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 3"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 3"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 3"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 3"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 3"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 3"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 4"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 4"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 4"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 4"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 4"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 4"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 4"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 5"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 5"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 5"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 5"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 5"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 5"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 5"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 6"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 6"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 6"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 6"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 6"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 6"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 6"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="mention"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="smart hyperlink"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="hashtag"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="unresolved mention"/> </w:latentstyles> </xml><![endif]--> <style> <!- /* font definitions */ @font-face {font-family:wingdings; panose-1:5 0 0 0 0 0 0 0 0 0; mso-font-charset:2; mso-generic-font-family:auto; mso-font-pitch:variable; mso-font-signature:0 268435456 0 0 -2147483648 0;} @font-face {font-family:"ms mincho"; panose-1:2 2 6 9 4 2 5 8 3 4; mso-font-alt:"ｍｓ 明朝"; mso-font-charset:128; mso-generic-font-family:modern; mso-font-pitch:fixed; mso-font-signature:-536870145 1791491579 134217746 0 131231 0;} @font-face {font-family:latha; panose-1:2 0 4 0 0 0 0 0 0 0; mso-font-charset:0; mso-generic-font-family:swiss; mso-font-pitch:variable; mso-font-signature:1048579 0 0 0 1 0;} @font-face {font-family:"cambria math"; panose-1:2 4 5 3 5 4 6 3 2 4; mso-font-charset:0; mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:3 0 0 0 1 0;} @font-face {font-family:calibri; panose-1:2 15 5 2 2 2 4 3 2 4; mso-font-charset:0; mso-generic-font-family:swiss; mso-font-pitch:variable; mso-font-signature:-536858881 -1073732485 9 0 511 0;} @font-face {font-family:"palatino linotype"; panose-1:2 4 5 2 5 5 5 3 3 4; mso-font-charset:0; mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:-536870265 1073741843 0 0 415 0;} @font-face {font-family:tahoma; panose-1:2 11 6 4 3 5 4 4 2 4; mso-font-charset:0; mso-generic-font-family:swiss; mso-font-pitch:variable; mso-font-signature:-520081665 -1073717157 41 0 66047 0;} @font-face {font-family:"\@ms mincho"; panose-1:2 2 6 9 4 2 5 8 3 4; mso-font-charset:128; mso-generic-font-family:modern; mso-font-pitch:fixed; mso-font-signature:-536870145 1791491579 134217746 0 131231 0;} /* style definitions */ p.msonormal, li.msonormal, div.msonormal {mso-style-unhide:no; mso-style-qformat:yes; mso-style-parent:""; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca; mso-fareast-language:en-us;} p.msocommenttext, li.msocommenttext, div.msocommenttext {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"comment text char"; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca; mso-fareast-language:en-us;} p.msoheader, li.msoheader, div.msoheader {mso-style-priority:99; mso-style-link:"header char"; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; tab-stops:center 3.15in right 6.3in; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca; mso-fareast-language:en-us;} p.msofooter, li.msofooter, div.msofooter {mso-style-priority:99; mso-style-link:"footer char"; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; tab-stops:center 3.15in right 6.3in; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca; mso-fareast-language:en-us;} span.msocommentreference {mso-style-noshow:yes; mso-style-priority:99; mso-ansi-font-size:8.0pt; mso-bidi-font-size:8.0pt;} p.msobodytext, li.msobodytext, div.msobodytext {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"body text char"; margin:0in; margin-bottom:.0001pt; mso-pagination:none; mso-layout-grid-align:none; text-autospace:none; font-size:12.0pt; font-family:"arial",sans-serif; mso-fareast-font-family:calibri; mso-bidi-font-family:latha; mso-ansi-language:en-us; mso-fareast-language:en-us;} a:link, span.msohyperlink {mso-style-priority:99; color:blue; text-decoration:underline; text-underline:single;} a:visited, span.msohyperlinkfollowed {mso-style-noshow:yes; mso-style-priority:99; color:purple; mso-themecolor:followedhyperlink; text-decoration:underline; text-underline:single;} p {mso-style-noshow:yes; mso-style-priority:99; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-ansi-language:en-ca; mso-fareast-language:zh-cn;} p.msocommentsubject, li.msocommentsubject, div.msocommentsubject {mso-style-noshow:yes; mso-style-priority:99; mso-style-parent:"comment text"; mso-style-link:"comment subject char"; mso-style-next:"comment text"; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca; mso-fareast-language:en-us; font-weight:bold;} p.msoacetate, li.msoacetate, div.msoacetate {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"balloon text char"; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:8.0pt; font-family:"tahoma",sans-serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca; mso-fareast-language:en-us;} p.msolistparagraph, li.msolistparagraph, div.msolistparagraph {mso-style-priority:34; mso-style-unhide:no; mso-style-qformat:yes; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.5in; margin-bottom:.0001pt; mso-add-space:auto; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca; mso-fareast-language:en-us;} p.msolistparagraphcxspfirst, li.msolistparagraphcxspfirst, div.msolistparagraphcxspfirst {mso-style-priority:34; mso-style-unhide:no; mso-style-qformat:yes; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.5in; margin-bottom:.0001pt; mso-add-space:auto; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca; mso-fareast-language:en-us;} p.msolistparagraphcxspmiddle, li.msolistparagraphcxspmiddle, div.msolistparagraphcxspmiddle {mso-style-priority:34; mso-style-unhide:no; mso-style-qformat:yes; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.5in; margin-bottom:.0001pt; mso-add-space:auto; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca; mso-fareast-language:en-us;} p.msolistparagraphcxsplast, li.msolistparagraphcxsplast, div.msolistparagraphcxsplast {mso-style-priority:34; mso-style-unhide:no; mso-style-qformat:yes; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.5in; margin-bottom:.0001pt; mso-add-space:auto; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca; mso-fareast-language:en-us;} p.msonormal0, li.msonormal0, div.msonormal0 {mso-style-name:msonormal; mso-style-priority:99; mso-style-unhide:no; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-ansi-language:en-ca; mso-fareast-language:zh-cn;} span.commenttextchar {mso-style-name:"comment text char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"comment text"; mso-ansi-font-size:10.0pt; mso-bidi-font-size:10.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; mso-ansi-language:en-ca;} span.headerchar {mso-style-name:"header char"; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:header; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; mso-ansi-language:en-ca;} span.footerchar {mso-style-name:"footer char"; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:footer; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; mso-ansi-language:en-ca;} span.bodytextchar {mso-style-name:"body text char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"body text"; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"arial",sans-serif; mso-ascii-font-family:arial; mso-fareast-font-family:calibri; mso-hansi-font-family:arial; mso-bidi-font-family:latha; mso-ansi-language:en-us;} span.commentsubjectchar {mso-style-name:"comment subject char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-parent:"comment text char"; mso-style-link:"comment subject"; mso-ansi-font-size:10.0pt; mso-bidi-font-size:10.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; mso-ansi-language:en-ca; font-weight:bold;} span.balloontextchar {mso-style-name:"balloon text char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"balloon text"; mso-ansi-font-size:8.0pt; mso-bidi-font-size:8.0pt; font-family:"tahoma",sans-serif; mso-ascii-font-family:tahoma; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:tahoma; mso-bidi-font-family:tahoma; mso-ansi-language:en-ca;} p.default, li.default, div.default {mso-style-name:default; mso-style-priority:99; mso-style-unhide:no; mso-style-parent:""; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; mso-layout-grid-align:none; text-autospace:none; font-size:12.0pt; font-family:"palatino linotype",serif; mso-fareast-font-family:"ms mincho"; mso-bidi-font-family:"palatino linotype"; color:black; mso-ansi-language:en-ca; mso-fareast-language:ja;} span.normalchar1 {mso-style-name:normal__char1; mso-style-priority:99; mso-style-unhide:no; mso-style-parent:""; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman";} span.shorttext {mso-style-name:short_text; mso-style-priority:99; mso-style-unhide:no; mso-style-parent:""; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman";} span.heading2main {mso-style-name:"heading 2 main"; mso-style-unhide:no; mso-style-parent:""; mso-ansi-font-size:12.0pt; font-family:"arial",sans-serif; mso-ascii-font-family:arial; mso-fareast-font-family:arial; mso-hansi-font-family:arial; mso-bidi-font-family:arial; color:black; font-weight:bold; mso-bidi-font-weight:normal; font-style:normal; text-underline:black;} span.spelle {mso-style-name:""; mso-spl-e:yes;} span.grame {mso-style-name:""; mso-gram-e:yes;} .msochpdefault {mso-style-type:export-only; mso-default-props:yes; font-size:10.0pt; mso-ansi-font-size:10.0pt; mso-bidi-font-size:10.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:calibri; mso-fareast-theme-font:minor-latin; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; mso-ansi-language:de;} /* page definitions */ @page {mso-footnote-separator:url("news_29757_v2_files/header.htm") fs; mso-footnote-continuation-separator:url("news_29757_v2_files/header.htm") fcs; mso-endnote-separator:url("news_29757_v2_files/header.htm") es; mso-endnote-continuation-separator:url("news_29757_v2_files/header.htm") ecs;} @page wordsection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in; mso-header-margin:35.4pt; mso-footer-margin:35.4pt; mso-columns:2 even 35.4pt; mso-header:url("news_29757_v2_files/header.htm") h1; mso-footer:url("news_29757_v2_files/header.htm") f1; mso-paper-source:0;} div.wordsection1 {page:wordsection1;} /* list definitions */ @list l0 {mso-list-id:881093513; mso-list-template-ids:1281779048;} @list l0:level1 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:.5in; mso-level-number-position:left; text-indent:-.25in; mso-ansi-font-size:10.0pt; font-family:symbol;} @list l0:level2 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:1.0in; mso-level-number-position:left; text-indent:-.25in; mso-ansi-font-size:10.0pt; font-family:symbol;} @list l0:level3 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:1.5in; mso-level-number-position:left; text-indent:-.25in; mso-ansi-font-size:10.0pt; font-family:symbol;} @list l0:level4 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:2.0in; mso-level-number-position:left; text-indent:-.25in; mso-ansi-font-size:10.0pt; font-family:symbol;} @list l0:level5 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:2.5in; mso-level-number-position:left; text-indent:-.25in; mso-ansi-font-size:10.0pt; font-family:symbol;} @list l0:level6 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:3.0in; mso-level-number-position:left; text-indent:-.25in; mso-ansi-font-size:10.0pt; font-family:symbol;} @list l0:level7 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:3.5in; mso-level-number-position:left; text-indent:-.25in; mso-ansi-font-size:10.0pt; font-family:symbol;} @list l0:level8 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:4.0in; mso-level-number-position:left; text-indent:-.25in; mso-ansi-font-size:10.0pt; font-family:symbol;} @list l0:level9 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:4.5in; mso-level-number-position:left; text-indent:-.25in; mso-ansi-font-size:10.0pt; font-family:symbol;} @list l1 {mso-list-id:924805370; mso-list-type:hybrid; mso-list-template-ids:-178096484 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;} @list l1:level1 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:symbol;} @list l1:level2 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:"courier new";} @list l1:level3 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:wingdings;} @list l1:level4 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:symbol;} @list l1:level5 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:"courier new";} @list l1:level6 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:wingdings;} @list l1:level7 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:symbol;} @list l1:level8 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:"courier new";} @list l1:level9 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:wingdings;} @list l2 {mso-list-id:1851752113; mso-list-type:hybrid; mso-list-template-ids:566934936 269025281 269025283 269025285 269025281 269025283 269025285 269025281 269025283 269025285;} @list l2:level1 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:symbol;} @list l2:level2 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:"courier new";} @list l2:level3 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:wingdings;} @list l2:level4 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:symbol;} @list l2:level5 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:"courier new";} @list l2:level6 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:wingdings;} @list l2:level7 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:symbol;} @list l2:level8 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:"courier new";} @list l2:level9 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:wingdings;} ol {margin-bottom:0in;} ul {margin-bottom:0in;} --> </style> <!--[if gte mso 10]> <style> /* style definitions */ table.msonormaltable {mso-style-name:"table normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; mso-ansi-language:de;} </style> <![endif]--><!--[if gte mso 9]><xml> <o:shapedefaults v:ext="edit" spidmax="2049"/> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext="edit"> <o:idmap v:ext="edit" data="1"/> </o:shapelayout></xml><![endif]--> </head> <body lang=en-us link=blue vlink=purple style='tab-interval:35.4pt'> <div class=wordsection1> <p class=msonormal style='mso-pagination:none;mso-layout-grid-align:none; text-autospace:none'><i style='mso-bidi-font-style:normal'><span lang=en-ca style='font-family:"palatino linotype",serif'>news<o:p></o:p></span></i></p> <p class=msonormal style='mso-pagination:none;mso-layout-grid-align:none; text-autospace:none'><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal><span style='font-size:11.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>call for applications:&nbsp;copyeditors for <i style='mso-bidi-font-style:normal'>evidence based library and information practice</i><br style='mso-special-character:line-break'> <![if !supportlinebreaknewline]><br style='mso-special-character:line-break'> <![endif]></span><span lang=en-ca style='font-family:"palatino linotype",serif'><o:p></o:p></span></p> <p class=msonormal><span style='font-size:9.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-no-proof:yes'><!--[if gte vml 1]><v:shapetype id="_x0000_t75" coordsize="21600,21600" o:spt="75" o:preferrelative="t" path="m@4@5l@4@11@9@11@9@5xe" filled="f" stroked="f"> <v:stroke joinstyle="miter"/> <v:formulas> <v:f eqn="if linedrawn pixellinewidth 0"/> <v:f eqn="sum @0 1 0"/> <v:f eqn="sum 0 0 @1"/> <v:f eqn="prod @2 1 2"/> <v:f eqn="prod @3 21600 pixelwidth"/> <v:f eqn="prod @3 21600 pixelheight"/> <v:f eqn="sum @0 0 1"/> <v:f eqn="prod @6 1 2"/> <v:f eqn="prod @7 21600 pixelwidth"/> <v:f eqn="sum @8 21600 0"/> <v:f eqn="prod @7 21600 pixelheight"/> <v:f eqn="sum @10 21600 0"/> </v:formulas> <v:path o:extrusionok="f" gradientshapeok="t" o:connecttype="rect"/> <o:lock v:ext="edit" aspectratio="t"/> </v:shapetype><v:shape id="picture_x0020_3" o:spid="_x0000_i1025" type="#_x0000_t75" alt="cc-ca_logo_xl" style='width:6.6pt;height:7.8pt;visibility:visible; mso-wrap-style:square'> <v:imagedata src="news_29757_v2_files/image001.png" o:title="cc-ca_logo_xl" cropright="49871f"/> </v:shape><![endif]--><![if !vml]><img width=9 height=10 src="news_29757_v2_files/image002.png" alt="cc-ca_logo_xl" v:shapes="picture_x0020_3"><![endif]><span style='mso-spacerun:yes'> </span>2020</span><span lang=en-ca style='font-size: 9.0pt;font-family:"palatino linotype",serif'>. this is an open access article distributed under the terms of the creative commons‐attribution‐<span class=spelle>noncommercial</span>‐share alike license 4.0 international (</span><span lang=en-ca><a href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><span style='font-size:9.0pt;font-family:"palatino linotype",serif'>http://creativecommons.org/licenses/by-nc-sa/4.0/</span></a></span><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif'>), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif; color:blue'>doi:</span><span lang=en-ca style='font-size:9.0pt;font-family: "palatino linotype",serif'> 10.18438/eblip29757</span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p></o:p></span></p> <div style='mso-element:para-border-div;border:none;border-bottom:solid black 1.0pt; mso-border-bottom-alt:solid black .25pt;padding:0in 0in 1.0pt 0in'> <p class=msobodytext style='border:none;mso-border-bottom-alt:solid black .25pt; padding:0in;mso-padding-alt:0in 0in 1.0pt 0in'><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-ca'><o:p>&nbsp;</o:p></span></p> </div> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-fareast-font-family: "times new roman";mso-fareast-theme-font:minor-fareast;mso-bidi-font-family: calibri;color:#212121;background:white;mso-ansi-language:en-us;mso-fareast-language: zh-cn'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;color:#212121;background:white;mso-ansi-language: en-us;mso-fareast-language:zh-cn'>copyeditors<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;color:#212121;background:white;mso-ansi-language: en-us;mso-fareast-language:zh-cn'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><i style='mso-bidi-font-style:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-fareast-font-family: "times new roman";mso-fareast-theme-font:minor-fareast;mso-bidi-font-family: calibri;color:#212121;background:white;mso-ansi-language:en-us;mso-fareast-language: zh-cn'>evidence based library and information practice (eblip)</span></i><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-fareast-font-family: "times new roman";mso-fareast-theme-font:minor-fareast;mso-bidi-font-family: calibri;color:#212121;background:white;mso-ansi-language:en-us;mso-fareast-language: zh-cn'> is looking for three new copyeditors to join our copyediting team. ideally, candidates will be in place by may 2020, in time to begin work on the june issue.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;color:#212121;background:white;mso-ansi-language: en-us;mso-fareast-language:zh-cn'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;color:#212121;background:white;mso-ansi-language: en-us;mso-fareast-language:zh-cn'>the role of copyeditors is to: <o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;color:#212121;background:white;mso-ansi-language: en-us;mso-fareast-language:zh-cn'><o:p>&nbsp;</o:p></span></p> <p class=msolistparagraphcxspfirst style='text-indent:-.25in;mso-list:l1 level1 lfo4'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol;color:#212121;mso-ansi-language:en-us;mso-fareast-language: zh-cn'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;color:#212121;background:white;mso-ansi-language: en-us;mso-fareast-language:zh-cn'>accept or reject copyediting requests in a timely manner <o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='text-indent:-.25in;mso-list:l1 level1 lfo4'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol;color:#212121;mso-ansi-language:en-us;mso-fareast-language: zh-cn'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;color:#212121;background:white;mso-ansi-language: en-us;mso-fareast-language:zh-cn'>utilize eblip copyediting guidelines alongside apa style guidelines <o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='text-indent:-.25in;mso-list:l1 level1 lfo4'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol;color:#212121;mso-ansi-language:en-us;mso-fareast-language: zh-cn'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;color:#212121;background:white;mso-ansi-language: en-us;mso-fareast-language:zh-cn'>ensure that proper grammar and spelling conventions are addressed<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='text-indent:-.25in;mso-list:l1 level1 lfo4'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol;color:#212121;mso-ansi-language:en-us;mso-fareast-language: zh-cn'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;color:#212121;background:white;mso-ansi-language: en-us;mso-fareast-language:zh-cn'>verify citations and references to the extent possible with available bibliographic resources and advise editor of any not checked<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='text-indent:-.25in;mso-list:l1 level1 lfo4'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol;color:#212121;mso-ansi-language:en-us;mso-fareast-language: zh-cn'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;color:#212121;background:white;mso-ansi-language: en-us;mso-fareast-language:zh-cn'>meet copyediting deadlines set by editors<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='text-indent:-.25in;mso-list:l1 level1 lfo4'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol;color:#212121;mso-ansi-language:en-us;mso-fareast-language: zh-cn'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;color:#212121;background:white;mso-ansi-language: en-us;mso-fareast-language:zh-cn'>identify inconsistencies or changes in format/style as necessary<o:p></o:p></span></p> <p class=msolistparagraphcxsplast style='text-indent:-.25in;mso-list:l1 level1 lfo4'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol;color:#212121;mso-ansi-language:en-us;mso-fareast-language: zh-cn'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;color:#212121;background:white;mso-ansi-language: en-us;mso-fareast-language:zh-cn'>communicate with the lead copyeditor regarding circumstances that might require temporary absences from editing availability<span style='mso-spacerun:yes'>   </span><o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;color:#212121;background:white;mso-ansi-language: en-us;mso-fareast-language:zh-cn'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;color:#212121;background:white;mso-ansi-language: en-us;mso-fareast-language:zh-cn'>the ideal candidate will have copyediting or other detail-oriented editing experience, possess working knowledge of apa style, and have an interest in <span class=grame>evidence based</span> practice. the position requires dedicated time to ensure eblip publishing deadlines are met, and it is therefore essential that interested persons ensure available time to devote to this position prior to applying. it is estimated that the workload is approximately 5 to 10 hours per issue, and much of this work falls over a one-month period, on a quarterly basis, prior to the publication of each issue.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;color:#212121;background:white;mso-ansi-language: en-us;mso-fareast-language:zh-cn'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;color:#212121;background:white;mso-ansi-language: en-us;mso-fareast-language:zh-cn'>interested persons should send a cover letter, indicating areas of strength they would bring to the role, and resume/cv as a single pdf file to heather healy, lead copyeditor, at <a href="mailto:heather-healy@uiowa.edu">heather-healy@uiowa.edu</a> by april 18, 2020. <o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;color:#212121;background:white;mso-ansi-language: en-us;mso-fareast-language:zh-cn'><span style='mso-spacerun:yes'> </span><o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;color:#212121;background:white;mso-ansi-language: en-us;mso-fareast-language:zh-cn'>**please note that <i style='mso-bidi-font-style: normal'>evidence based library and information practice</i> is a non-profit, open access journal and all positions are voluntary and unpaid. the positions are an excellent opportunity for continuing professional development and gaining experience in reviewing or critically appraising library-related research.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;color:#212121;background:white;mso-ansi-language: en-us;mso-fareast-language:zh-cn'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;color:#212121;background:white;mso-ansi-language: en-us;mso-fareast-language:zh-cn'>**only those applicants who are selected or shortlisted will be contacted.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;color:#212121;background:white;mso-ansi-language: en-us;mso-fareast-language:zh-cn'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;color:#212121;background:white;mso-ansi-language: en-us;mso-fareast-language:zh-cn'>about the journal:<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;color:#212121;background:white;mso-ansi-language: en-us;mso-fareast-language:zh-cn'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;color:#212121;background:white;mso-ansi-language: en-us;mso-fareast-language:zh-cn'>published quarterly, and hosted by the university of alberta, this peer-reviewed, open access journal is targeted at all library and information professionals interested in an <span class=grame>evidence based</span> model of practice. by facilitating access to librarianship research via original research articles and evidence summaries of relevant research from the library literature, <i style='mso-bidi-font-style:normal'>evidence based library and information practice</i> will enable librarians to practice their profession in an <span class=grame>evidence based</span> manner.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;color:#212121;background:white;mso-ansi-language: en-us;mso-fareast-language:zh-cn'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;color:#212121;background:white;mso-ansi-language: en-us;mso-fareast-language:zh-cn'>please visit the evidence based library and information practice web site (<a href="http://journals.library.ualberta.ca/index.php/eblip">http://journals.library.ualberta.ca/index.php/eblip</a>) for further information about the journal.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;color:#212121;background:white;mso-ansi-language: en-us;mso-fareast-language:zh-cn'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;color:#212121;background:white;mso-ansi-language: en-us;mso-fareast-language:zh-cn'>thank you,<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;color:#212121;background:white;mso-ansi-language: en-us;mso-fareast-language:zh-cn'>lorie <span class=spelle>kloda</span><o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;color:#212121;background:white;mso-ansi-language: en-us;mso-fareast-language:zh-cn'>editor-in-chief<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;color:#212121;background:white;mso-ansi-language: en-us;mso-fareast-language:zh-cn'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;color:#212121;background:white;mso-ansi-language: en-us;mso-fareast-language:zh-cn'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-fareast-font-family: "times new roman";mso-fareast-theme-font:minor-fareast;mso-bidi-font-family: calibri;color:#212121;background:white;mso-ansi-language:en-us;mso-fareast-language: zh-cn'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> </div> </body> </html> evidence based library and information practice 2012, 7.4 52 evidence based library and information practice article independent searching during one-shot information literacy instruction sessions: is it an effective use of time? rebekah willson assistant professor and librarian, mount royal university phd student, school of information studies, charles sturt university wagga wagga, new south wales, australia email: rwillson@csu.edu.au received: 2 july 2012 accepted: 18 nov. 2012 2012 willson. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to test the assumption that giving students time to research independently during a one-shot information literacy instruction (ili) session, combined with scaffolding, is an effective pedagogical practice and a good use of class time. methods – the study was conducted at a student-focused, four-year undergraduate institution with 8,500 full load equivalent students. following brief, focused instruction in 10 different ili sessions, first-, second-, and third-year students in 80-minute one-shot ili sessions were given time to research independently. the librarian and instructor were present to scaffold the instruction students received. students were asked to track the research they did during class using a research log and to fill out a short web survey about their preparedness to do research and the usefulness of the ili session. results – students agreed to have 83 research logs and 73 web surveys included in the study. students indicated that they felt more prepared to do research for their assignment after the ili session and rated individual help from the librarian as the most useful aspect of the instruction session. students did not rate independent time to do research as valuable as anticipated. examining the research logs indicated that several things are taking place during the ili session, including that students are demonstrating what was taught in the session in their searches, that their searches are progressing in complexity, and that students are using feedback mailto:rwillson@csu.edu.au evidence based library and information practice 2012, 7.4 53 from previous searches to inform the formulation of search queries. while students appear to be putting independent search time to good use, many students’ articulation of their thesis statement remains poor and searches continue to be fairly simplistic. conclusions – this study gives evidence that giving independent research time in ili sessions, with scaffolding, is an effective use of class time. the study also demonstrates that the majority of students are able to use what is taught during classes and that they are using class time effectively, though searching remains fairly simple. the focus of ili sessions is on skill development, and future research should be on integrating il into the curriculum to develop more complex skills and thinking needed in the research process. introduction for all practitioners, evidence based practice is challenging. for those who teach information literacy instruction (ili), evidence based librarianship becomes particularly difficult when teaching sessions are “one shot.” many librarians have a single session with a group of students that will last between 50 and 80 minutes, and a significant amount of content to cover. as student learning is the ultimate goal for ili, the choice of what content to cover (e.g., teaching concepts or skills) and how to teach that content (e.g., lectures or hands-on practice) is of the utmost importance. collecting evidence for any research is challenging; however, when attempting to practice evidence based librarianship in this restricted context, data collection must be quick and unobtrusive, not taking up precious class time. one way to incorporate data collection into one-shot sessions is to use what already takes place in the class. many librarians do this by collecting assignments, either research assignments that have been assigned by the course lecturer (webster & reilly, 2003) or worksheets that have been assigned by the librarian in the ili session (fain, 2011), or a combination of these two methods. for other librarians, course assignments are not available and they may decide worksheets take up more class time than they are willing to give. the author was interested in questions of pedagogy – how best to use class time in the ili session. typical classes were short lectures followed by lengthy periods of time to search independently, combined with one-on-one help termed scaffolding. scaffolding is a technique in which a teacher works with students individually to give them support, gradually removing that support as the student is able to work more independently. (larkin, 2008). the evidence based librarianship project was designed to determine if current practices were helpful to students. the research question was whether giving students time to research independently during a one-shot information literacy instruction session, combined with scaffolding, is an effective use of class time. literature review information literacy instruction is a large part of many librarians’ work. as such it has been the subject of much study. to better understand what takes place in the classroom, it is important to examine both student behaviour and classroom pedagogy. information literacy instruction is a complex research topic, with a multitude of factors impacting the learner, the learning environment, and the instruction. for those who teach searching, understanding how students search for online information is an important aspect. in their information behavior model, urquhart and rowley (2007) identified many micro and macro factors that impact student information behaviours, including information literacy, search strategies, discipline and curriculum, pedagogy, support and training, information evidence based library and information practice 2012, 7.4 54 resource design, and access. many students typically begin searching with google (griffiths & brophy, 2005; urquhart & rowley, 2007), relying less on academic resources (griffiths & brophy, 2005). starting searches with google and limiting use of resources to those that are well known is typical, as ease of use and familiarity are important factors in their choice of resource (dervin & reinhard, 2007; griffiths & brophy, 2005; urquhart & rowley, 2007). searchers in online environments typically include few terms in their search queries, infrequently use advanced search techniques such as boolean operators, and infrequently use advanced search features such as limiters (lau & goh, 2006; markey, 2007a, 2007b; wang, berry, & yang, 2003; willson & given, 2010). in addition to simple searching, students often expect online public access catalogues and databases to work like a search engine (griffiths & brophy, 2005; novotny, 2004). searchers with less search experience spend less time thinking about and planning searches, in addition to using fewer self-aware, metacognitive strategies than do searchers with expertise (tabatabai & shore, 2005). overall, undergraduate students use familiar sources in fairly rudimentary ways to satisfy their information needs. many researchers have examined the effectiveness of ili. some research has focused on the general effectiveness of ili. portmann & roush (2004) found that ili increased library usage, though not library skills. one of koufogiannakis and wiebe’s (2006) findings from their meta-analysis of ili studies was that, overall, instruction of any variety was better than no instruction. other research has focused on the effectiveness of particular interventions. buhay, best, and mcguire (2010) found that student scores on post-tests were statistically significantly higher when they used clickers in ili. marcus and beck (2003) found that students who took part in a treasure-hunt style self-orientation to the library scored higher on questionnaires and rated the tours more positively than those who were in librarian-led orientation groups. bren, hillemann, and topp (1998) found that using a guided, hands-on instructional method increased undergraduate students’ retention of information provided during an ili session. these studies indicate that ili is effective and that particular interventions can be used. what is missing from studies of ili effectiveness is an examination of scaffolding and independent search time. one article mentioned scaffolding as part of the ili instruction (johnson et al., 2011). no articles were found that addressed independent search time. it is difficult to determine whether this apparent lack of literature is due to these pedagogical practices not being researched, or due to differences in terminology that make the literature difficult to find. context and aims context mount royal university is a four-year undergraduate university. the student body is 10,551 full-time students (mount royal university, n.d.a). the institution has four categories within its aims of an undergraduate undergraduate education, with information literacy listed under intellectual and practical skills (mount royal university, n.d.b). in 2009/2010, the library taught 710 ili sessions to over 12,000 students. typically ili sessions are one-shot sessions that are either 50or 80-minutes long and tend to be focused on a specific research project. the vast majority of ili sessions are hands-on and take place in computer labs. aims as part of evidence based practice, the author was interested in examining the information literacy instruction she provides, with a view to improving understanding of whether the independent time to search in the ili session is used effectively and how teaching could be altered to improve learning. feeling that she was trying to include too much content into one-shot sessions, she cut down on the amount of content presented in class in the lecture format, focusing on the specific research assignment and providing as much time as possible for hands-on work. the focus of these evidence based library and information practice 2012, 7.4 55 ili sessions became the students’ research assignments and the individual help (scaffolding) provided to students by the librarian and the class instructor. the researcher felt that giving students time to work on their research assignment is a more active learning technique and that students would have the opportunity to try what had been discussed in class, along with scaffolding. part of scaffolding is to work within students’ “zone of proximal development,” the gap between what a student can achieve on their own and what they can achieve with help. by focusing on individual time with students to scaffold their work, the researcher believed that instruction could be better tailored to students’ specific needs (e.g., working within their zone of proximal development), focus on those needing more help and support by providing additional time and attention, while more confident students could get research done during class time. while this was the reasoning behind the original changes made to the ili sessions, the assumption being made was that independent time to work would be beneficial for students. the author received challenges to this assumption by colleagues in the scholars’ program who wondered if students were prepared for time to work independently and whether giving students an assignment that introduced concepts might be a more beneficial use of class time. while much research has been done into which method of delivering information literacy instruction (e.g., computer-assisted instruction vs. traditional instruction), fewer librarians have researched what specific aspects of a method of instruction make it beneficial. methods participants students from 10 ili sessions were the participants in this study. the classes were at the first-, second-, and third-year level from the religious studies, psychology, and general education disciplines. class sizes ranged from 20-30 students, meaning the participants were drawn from an overall sample of approximately 200-300 students. the classes focused on searching for sources for their assignment, basic search strategies (boolean operators, truncation, and phrase searching) and database searching. in two sections of a third-year psychology course, students were taught to use mesh. the inclusion criteria for the study were that students were attending an ili session that was 80 minutes or longer, as 50-minute sessions were too short to include a web survey. all students in ili sessions were asked to perform the same tasks, fill out a research log during independent searching, and to complete a web-based survey at the end of class. all students’ responses were examined to inform pedagogy and student learning. only the data of those students who agreed to participate were included in the study. in total, 73 students agreed to include their web survey in the study and 83 students agreed to include their research log in the study. this study received approval from the human research ethics board at mount royal university. web survey this quantitative study included both a web survey and research logs. to examine the research question, the researcher designed a web survey and research log. section 1 of the web survey asked whether students had attended a previous ili session (and if so, how many), trying to gauge students’ prior experience (see appendix a). section 2 of the web survey was designed to determine the specific aspects of the ili session – the different pedagogical tools used in the classroom – that students perceived to be most useful by comparing one aspect to another. students were asked to rate the activities that took place during the class from most useful (1) to least useful (9). finally, in section 3, students were also asked to rate their preparedness before and after the ili session on a four-point likert scale. the research referred students to the web survey url via the online subject guide for the class. the web survey was administered at the end of the ili session, taking approximately five minutes to complete. evidence based library and information practice 2012, 7.4 56 research logs the research logs attempted to uncover students’ information behaviours – which resources they use, how they search, how they modify their searches, what they think about their searching (see appendix b). the research logs were examined to determine if students used the skills taught in class during independent searching. other research has described research journals as part of an ongoing research process throughout a class (smith, 2001; warner, 2003). the research log used in this study is intended to capture students’ searching at a particular time and to aid students in recording their search process, similar to kuhlthau’s search logs (2004, pp. 3233). bates’ berrypicking model (1989) and kuhlthaus’ information search process model (2005) were used in the creation of the research log – to help students track their search progression and the change in their thinking that leads to search modification. during the independent search time, students were asked to record their work. the research log was on carbonless paper; students kept the top copy while the author kept the bottom. it is important for students to keep track of their searches to understand where they have searched, to understand what they have searched, and to examine how their research might progress. in addition to being a form for data collection, the researcher employed the research log as a pedagogical tool to try to increase students’ awareness of their searching. the research log was used in preference to computer logs, which do not involve student thought. students were asked to record their topic/thesis statement and ideas/concepts related to their topic. students were also asked to record their searches: the date, the resource searched, the search query, what was found, and notes to self. if students asked questions during the ili session, they were also asked to record their questions: what the question related to, if their question was answered, and what questions they felt might come up later. analysis the researcher analyzed the web surveys using descriptive statistics (frequencies and percentages), which were chosen to summarize the responses from the students in the sample and to provide basic information about the responses. the researcher examined research logs to determine if there was evidence of what took place during the independent search time, while recorded searches were examined to determine where students searched, the search queries students created, the search strategies students used, the ways in which searches changed, and what students wrote about their searching. from looking at how students’ modified their searches and what they wrote about their searches, the researcher developed categories to describe the commonalities seen. after developing operational definitions for the categories, the author categorized the searches. the categories were: search complexity (searches with two or more terms or use of specific search strategies), search progression (series of searches in which students increase the complexity or precision of their searches), use of feedback (series of searches in which students use the results of previous searches to modify or improve searches), and mode of search modification (incremental modifications or jumps from one strategy to another). the researcher used the categories to produce descriptive statistics about how students used their independent search time. results usefulness of ili sessions the web survey asked students to rate the usefulness of the different aspects of the ili session from most useful (1) to least useful (9). students rated individual help from the librarian as the most useful aspect of the ili session with a rating of 3.89 (see table 1). the second most highly rated aspect of the session was discussions of how to use resources (4.46). after the top two rated aspects of the session there was little variance between ratings. the author had hypothesized that students would rate time to work independently as the most evidence based library and information practice 2012, 7.4 57 valuable aspect of the ili session. instead, time to work independently was tied for the eighth most useful aspect of the session. rather than simply valuing the time to work, students rated the one-on-one help they received during that independent search time as more useful. while this survey only asked for student perceptions of usefulness by comparing different pedagogical activities, students valued the scaffolding that took place in the session. as the results from the web survey were too small to run a test for statistical significance, the ratings should be treated with caution and more research is needed to confirm these findings. the researcher examined the research logs to determine if students used what was taught in class. of the 83 research logs included in the study, 77 (93%) contained recorded searches; 43 of those 77 (56%) showed clear evidence of using what had been taught, while 21 (27%) showed some evidence, and 13 (17%) showed no evidence (see table 2). there is evidence that students understood the content of the lesson well enough for them to use it to search during independent searching within the context of the ili session. (table 2) table 1 average rating of aspects of ili sessions from most useful (1) to least useful (9) answer options rating average individual help from librarian 3.89 discussion of how to use the resources 4.46 discussion about search difficulties 4.86 citation discussion 4.93 individual help from class instructor 4.94 discussion of the resources to use 5.00 time to work independently 5.23 working/discussing with class mates 5.23 explanation of the assignment 5.75 table 2 evidence, from three different students, of using what was taught in class evidence of using what was taught in class use (%) n=77 example of evidence clear evidence 43 (56%) bipolar disorder in mm, review articles, linked full text unclear evidence 21 (27%) bipolar disorder, youth no evidence 13 (17%) bipolar disorder evidence based library and information practice 2012, 7.4 58 how students are searching the author also analyzed the research logs for evidence of how students search during the time given for independent searching. in total 237 searches were recorded, representing an average of 3.1 searches per research log (n=77). searches averaged 3.7 words per query. of the 77 research logs containing searches, 52 (68%) included use of boolean operators, 23 (30%) included use of truncation, 19 (25%) used phrase searching, and 16 (21%) recorded the use of a search limit. students used boolean operators most frequently of the search strategies taught during the ili session. of those using boolean operators, 1 (2%, n=52) used them incorrectly; for those using truncation, 4 (17%, n=23) used them incorrectly; and for phrase searching, 6 (32%, n=19) used them incorrectly (see table 3). the percentage of incorrect uses for each of the search strategies may indicate that students are most comfortable using boolean operators and least comfortable using phrase searching. searches were also rated on their complexity. a complex search had more than two ideas, or had two ideas in addition to employing specific search strategies – boolean operators, truncation, subject heading searches, etc. the author found that 34 of 77 (44%) research logs included complex searches, that 28 (36%) did not have complex searches, and that 15 (19%) had elements of complex searches but could not be fully categorized as complex (see table 4). when looking at the number of students that used more than two ideas in their search and different search strategies, the author discovered that the overall searches were relatively simple. while different assignments required differing levels of search complexity, many of the topics students were exploring would retrieve results too great in number or lacking in precision. the researcher also examined how searches were modified, whether in increments, by making small modifications to search strategies or an aspect of a term, or in jumps, such as by changing vocabulary, topics, or resources entirely. of the 60 research logs with multiple searches, 37 (62%) made search modifications using increments, 13 (22%) made search modifications using jumps, and 10 (17%) made search modifications using both. in making incremental modifications, students table 3 search strategy use and examples of incorrect use search strategy use (%) n=77 inappropriate use (%), n example of incorrect use boolean 52 (68%) 1 (2%), n=52 divorce and children and childhood truncation 23 (30%) 4 (17%), n=23 immigration and poverty and canada* phrase 19 (25%) 6 (32%), n=19 “abuse” and “elder”, specific to 65+1 1 referring to database-specific age limit table 4 search complexity and examples of complexity search complexity use (%) n=77 example of search complexity complex search 34 (44%) globalization and relig* and identity not complex search 23 (36%) walmart and globalization elements of complexity 15 (19%) eat* local* evidence based library and information practice 2012, 7.4 59 were searching in a more focused way, testing how small changes to a search will affect the search results. in making jumps in search query modifications, students were searching in a broader way, by exploring what is available or exploring their topic. the researcher examined search modifications (changes made to search queries and/or resources in which the search was carried out over successive searches) to determine the specifics of how such changes were made: keywords used, resources used, and search techniques. changing the keyword used was the most common modification, followed by adding or subtracting keywords, then changing the resource in which the search was carried out, followed by changing search techniques, such as using operators, truncation and/or phrase searching (see table 5). of 60 research logs with recorded search modifications, 19 (32%) revealed the use of multiple search modifications during the search. the data suggest that students view keyword terms as the primary way to change their searches. student preparation to search independently at the end of the ili session students filled out the web survey, rating their preparedness to do research before and after the session. looking back, 41% of students rated their preparedness before the session as “prepared” or “somewhat prepared,” while 100% of students rated their preparedness after the session as “prepared” or “somewhat prepared” (n=73). the author examined research logs to see if this perception was corroborated in behaviour. examining changes in the research logs allowed the researcher to observe how students adapted their searching during time given in class to search independently. searches were examined to determine if they showed progress, which was defined as a series of searches in which students increased the complexity or precision of their search. an example of a student search that demonstrates progression is shown in table 6, while an example that does not demonstrate progression is found in table 7. a total of 60 research logs contained more than one recorded search and were examined for search progression. of those 60 research logs, 41 (68%) showed a progression while 19 (32%) showed no progression. that evidence of progression appeared in the majority of research logs with multiple searches indicates that students were able to use their independent search time to adaptively change their searches. effective changes must use feedback from the results of previous searches. evidence of use of feedback was defined by a series of searches in which students used the results of previous searches to modify and improve their searches, as evidenced by discussion of changes in the results or notes to self fields or the modification of search terms. of the 60 table 5 types of search modifications by number and percentage of research logs search modification number (%) (n=60)* changing keywords 46 (77%) adding or subtracting keywords 37 (62%) changing resource used 22 (37%) putting on or taking off database limits 9 (15%) adding or subtracting boolean operators 5 (8%) adding or subtracting truncation 3 (5%) adding or subtracting phrase searching 2 (3%) other 2 (3%) *multiple search modifications could be used in one research log. evidence based library and information practice 2012, 7.4 60 searches that had multiple recorded searches, 29 (48%) showed evidence of use of feedback, while 31 (52%) showed no evidence. approximately half of the students gave evidence that they used the previous searches to inform their subsequent search choices, indicating that students were learning while they used the independent time to search. a closer look at table 6 and table 7 reveals differing use of feedback. both tables show examples of two different research logs from the same class. the examples can be examined for of several types of searching: progression, use of feedback, demonstrating use of what was taught in class and complexity. in the class from which these research logs come, students were taught to use mesh to search for neurological disorders. the student in table 6 demonstrates progression through their use of the search terms: starting with the name of the disorder, checking it in mesh, using the mesh term as a major subject heading, and then continuing to add words and limits to the search until the student reaches what s/he determines to be a useful search. the student in table 7 does not show progression. the second search used has more ideas, making it more complex; however, there is no indication of how the student arrived at the search or whether the student tried other searches that were more or less successful. the example in table 6 demonstrates the use of feedback from previous results, making comments in the notes to self about how the search could be changed, which are then reflected in the searches and the resources found. the example in table 7 demonstrates no use of changing based on previous results. the first example table 6 example of student searching demonstrating progression in searching resource used search resources found notes to self medline guillain-barre syndrome overwhelming, 5,000+ articles use mesh medline (mm “guillain-barre syndrome”) 1,800+ articles, but interesting subset headings advanced search “etiology” medline (mm “guillain-barre syndrome”) and etiology 500+ articles advanced search check review articles medline (mm “guillain-barre syndrome”) and etiology + review articles 70 results. there are 7 solid articles i can use on first page but there are no copies!!?? sciencedirect advanced → review articles, title/abstract/keyword, guillainbarre syndrome 72 results, 4-5 articles i can use sciencedirect actually has copies! table 7 example of student searching not demonstrating progression in searching resource used search resources found notes to self medline narcolepsy and etiology – keywords many results found – reviewed, emailed link full text. recent articles. sciencedirect narcolepsy sleep disorders rem sleep e-mailed 8 articles for further review sleep disorders, etiology, sleep mri, sleep eeg evidence based library and information practice 2012, 7.4 61 demonstrates what was taught in class – subject headings, subheadings, and the use of limits – while the second example indicates some use example has of the language discussed in class (e.g., etiology) but has used keywords in medline and used no boolean operators or phrase searching while in sciencedirect. also, the first example includes a complex search using many ideas and limits together, while the second example also has both a more and less complex search. the first recorded search is simple in query terms, though it might also include limits making it more complex, whereas the second search has more ideas included despite problems with the search construction. the student whose search is represented in table 6 demonstrates search progression, use of feedback, use of what was taught in class, as well as a complex search. the student whose search is represented in table 7 partially demonstrates what was taught in class and a complex search. without data triangulation through examining finished work it is not possible to determine if students’ recording of their searches was truly indicative of how they searched or of what they wanted to record for later use. discussion usefulness of ili sessions the web survey results indicate that students feel more prepared after the ili session than before, and they rate the help from the librarian as the most useful part of the session. counter to expectations, students rate the scaffolding they receive during the independent time to work much more highly than they rate the time they are given to work independently. without further information about students’ perceptions and expectations, it is not possible to determine whether the ratings were due to perceived usefulness or prior expectations. students may expect individualized help during ili sessions. overall, these results indicate that students do find ili sessions useful in helping them to feel prepared to complete their research assignment. the data indicate that independent time to search is useful so far as it allows scaffolding to take place, as one-on-one help is viewed as more beneficial. how students are searching student searches were not very complex, with searches containing on average 3.7 words, and less than half of the research logs rated as having complex searches. despite this, most students used boolean operators and were able to use them appropriately. students used truncation and phrase searching less frequently, and almost one-third of those using phrase searching could not use this technique appropriately. the majority of students are able to use, at least in part, what was taught in the ili session. while this is encouraging, cmor, chan, and kong (2010) found that while the majority of students could complete information literacy-related exercises in ili sessions, few were able to demonstrate the ability to use new tools and search strategies or incorporate new knowledge into projects. the researcher’s assumption that it is beneficial to give students hands-on time to search independently during the ili session was partially substantiated. while students demonstrated that they could achieve many things during the time given to work independently, including formulating more complex and/or precise searches, using feedback to improve searches, and using what is taught in class, students most valued the individual help they received from the librarian. the research logs revealed several other aspects in which students could use instruction. for example, students’ searches were not very complex, and while not all search topics require complex searching, putting together a search strategy that increases both precision and recall is important. more instruction on search query formulation could be beneficial. in addition, more instruction about truncation and phrase searching could be beneficial; the number of incorrect uses indicates that students may not know what phrase searching will do. evidence based library and information practice 2012, 7.4 62 student preparation to search independently the majority of students who performed multiple searches showed evidence of search progression. this indicates that students are able, even within the course of an ili session, to increase the complexity or precision of their searches. that fact that students are able to demonstrate this during class suggests that they may do the same in the searching that they engage in on their own time. also encouraging was that almost half the student research logs contained multiple searches, evidence which suggests that students were using feedback to modify their searches, something not explicitly taught during ili sessions. additionally, the fact that students use feedback to make search modifications also lines up with the incremental changes seen in most research logs. vocabulary and resources are the most commonly used ways to change searches, with little experimentation of search techniques. while changing terms is one of the best ways to modify a search query, students are not modifying search queries using search techniques, a topic that does not receives much attention during the ili session. limitations there were several limitations to this study, many of which stem from incorporating research into one-shot ili sessions, which are particularly constrained. as the web survey was completed at the end of the session, it had a lower response rate than the research logs, and the timing of the survey may also impact the results. this lower response rate may have been due to students rushing to finish class, which may also have impacted the survey results if students guessed at answers or chose answers which appeared first. in addition, the survey asked for usefulness of pedagogical tools in comparison with one another, making it more difficult to assess how these tools impacted student learning. the survey asked about the general preparedness of students, rather than preparedness related to specific tasks, which may have been less sensitive to differences in students’ levels of preparation and could have been affected by student interpretations of the question. all data in the study were recorded by students. from in-class observations it was clear that some students were doing more searches than they were recording. from markings on the carbonless paper research logs it was clear that some students were also recording searches other places. some students may have found recording their searches onerous. some students might have difficulties performing the searches and making accurate recordings. this may have been the case particularly for students with less search experience, as recording searches adds another task and could increase the mental effort required to complete the work. while recording searches with research logs may have made the task more difficult, it has potential benefits as a pedagogical tool to help students think about their search process. in addition to issues around students recording, the process of data analysis added limitations to the study. the researcher examined research logs together, by collapsing the classes into one group for comparison. collapsing the classes has potential validity issues, though it was the most appropriate way to analyze the data collected. since the research logs did not capture demographic data, other than class, and since the number of participants for each class was low, this means that group divisions were not meaningful. further to issues of validity and reliability, the author categorized the research logs. in the future, to increase reliability and validity of the findings, more than one person should categorize the data. originally, the research project included an additional component to help triangulate the data, to gain further insight into student behaviours and to address whether the ili had impact beyond the classroom. those participants who completed the web survey were asked if they would be interested in being contacted for a follow-up interview. five participants indicated interest; however, only one participant took part in the semistructured interview. because of this, data collection was limited to the classroom and no follow-up information could be gathered. this lack of follow-up data means the results are evidence based library and information practice 2012, 7.4 63 limited to the quantitative results, and understanding reasons for participant behaviour is limited. implications for future practice and research from the progression seen in students’ searches produced during independent search time, it appears that students are making good use of that time. from the students’ point of view, that individual help from the librarian is the most useful part of the ili session. students find that independent time to search, when combined with scaffolding tailored to students’ individual needs, is beneficial or that it meets their expectations of an ili session. based on this evidence the author will continue to provide time for students to work independently and provide help on a one-toone basis. the web survey should be given to more students to determine how prevalent the view that the most important aspect of the session is one-on-one help with the librarian, in addition to what expectations students have of the session. recording searches can be challenging, particularly for students who are less familiar with searching and whose cognitive processing space is being used in doing the actual searches. to improve future research, as well as to help students’ keep track of their search process, new ways for students to accurately keep track of their search process should be explored. database features such as search history, citation management tools, or instruments could be used for this purpose. the majority of students demonstrated in their research logs what they learned in class. however, students are not demonstrating some of the important aspects of the research process. the author hoped for more evidence of using feedback to improve searches and metacognition about the search process. it is difficult to know whether students are not engaging in these activities, or whether the limitations of the situation (little time, computer lab environment, pressure to get work done, research log limitations) contribute to what students do or not do during class time. in addition, these types of higher-level thinking skills are not explicitly taught during class. students are able to demonstrate activities taught during the ili lesson during time given to search independently. it is possible that explicitly teaching skills such as thinking metacognitively and how to use feedback could also elicit those behaviours during independent search time. future research will explore metacognitive aspects of the research process, both what aspects students engage in and how metacognitive thinking can be enhanced. while students demonstrate that they can use what was taught in class, it is unknown if students can take that learning beyond the classroom and some research (e.g., cmor et al., 2010) suggests it is doubtful. in addition, it is difficult to know whether students can extrapolate their learning in class to a greater understanding of research as a process. librarians may want students to engage in metacognitive thinking and learn that research in a complex process, part of which involves library research. however, in one class it is unrealistic to expect students to become information literate. one-shot ili sessions, while they may be effective, should be only one part of an overall library instruction program (webster & rielly, 2003). because librarians often have only one class, working with instructors who have the semester and with programs that set four-year curricula becomes more important. ili sessions need to go beyond teaching skills and into authentic student learning, requiring collaboration with classroom faculty (wakimoto, 2010). integrating ili into what is done at the class and curricula level is necessary for students’ growth in information literacy. conclusions simply by examining what students do during a single ili session, librarians can learn a lot about their own teaching, student information behaviour, and student learning. students report that information literacy instruction sessions help them feel more prepared to do research. more than just time to work independently, students indicate scaffolding, the one-to-one instruction from the librarian, evidence based library and information practice 2012, 7.4 64 as being valuable. within class, students are able to demonstrate generally correct use of skills taught during the ili session, though their search queries are rather simple. while ili sessions are only one part of a larger plan for developing il skills, students report finding them useful. also, demonstrating the usefulness of ili sessions is the fact that many recorded student searches show increasingly complex searches or the use of feedback from previous searches to create more precise search queries. while skills are of immediate importance to students doing research assignments, these are important to students’ learning throughout their degrees. if students are not already demonstrating these understandings in their searches, they should be the focus of instruction. again, this instruction cannot take place in a one-shot session as they are complex and take time to develop. integrating more into courses and curriculum is important if we wish our students to attain these skills. acknowledgements support and funding for this research were provided by the institute for the scholarship of teaching and learning at mount royal university. a special thank you to dr. richard gale, dr. karen manarin, dr. deb bennett, dr. miriam carey, and the 2010 scholar cohort. this research was first presented as a paper at the 6th international evidence based library and information practice (eblip6) conference, manchester, uk, june 2011. references baker, l. (2006). library instruction in the rearview mirror: a reflective look at the evolution of a first-year library program using evidence-based practice. college & undergraduate libraries, 13(2), 1-20. doi: 10.1300/j106v13n02_01 bates, m. j. (1989). the design of browsing and berrypicking techniques for the online search interface. online review, 13(5), 407-424. bren, b., hillemann, b., & topp, v. (1998). effectiveness of hands-on instruction of electronic resources. research strategies, 16(1), 41-51. doi:10.1016/s0734-3310(98)90005-2 buhay, d., best, l. a., & mcguire, k. (2010). the effectiveness of library instruction: do student response systems (clickers) enhance learning? the canadian journal for the scholarship of teaching and learning, 1(1). retrieved 4 dec. 2012 from http://ir.lib.uwo.ca/cjsotl_rcacea/vol1/i ss1/5/ cmor, d., chan, a., & kong, t. (2010). courseintegrated learning outcomes for library database searching: three assessment points on the path of evidence. evidence based library and information practice, 5(1), 64-81. retrieved 4 dec. 2012 from http://ejournals.library.ualberta.ca/ind ex.php/eblip/article/view/6509/6449 dervin, b., & reinhard, c. d. (2006). researchers and practitioners talk about users and each other . making user and audience studies matter – paper 1. information research, 12(1). retrieved 7 dec. 2012 from http://informationr.net/ir/121/paper286.html fain, m. (2011). assessing information literacy skills development in first year students: a multi-year study. the journal of academic librarianship, 37(2), 109-119. griffiths, j. r., & brophy, p. (2005). student searching behavior and the web: use of academic resources and google. library trends, 53(4), 539-554. johnson, c. m., anelli, c. m., galbraith, b. j., & green, k. a. (2011). information literacy instruction and assessment in an honors college science fundamentals course. college & research libraries, 72(6), 533-547. evidence based library and information practice 2012, 7.4 65 koufogiannakis, d., & wiebe, n. (2006). effective methods for teaching information literacy skills to undergraduate students: a systematic review and meta-analysis. evidence based library and information practice, 1(3), 3-43. retrieved 4 dec. 2012 from http://ejournals.library.ualberta.ca/ind ex.php/eblip/article/view/76/153 kuhlthau, c. c. (2005). kulhthau’s information search process. in k. e. fisher, s. erdelez, & l. e. f. mckechnie (eds.), theories of information behavior. (pp. 230-234). medford, nj: published for the american society for information science and technology by information today, inc. kuhlthau, c. c. (2004). seeking meaning: a process approach to library and information services. westport, ct: libraries unlimited. larkin, m. j. (2008). scaffolding. in n. j. salkind (ed.), encyclopedia of educational psychology. (pp. 863-864). thousand oaks, ca: sage publications. lau, e. p., & goh, d. h. l. (2006). in search of query patterns: a case study of a university opac. information processing & management, 42, 13161329. doi:10.1016/j.ipm.2006.02.003 marcus, s., & beck, s. (2003). a library adventure: comparing a treasure hunt with a traditional freshman orientation tour. college & research libraries, 64(1), 23-44. markey, k. (2007a). twenty-five years of enduser searching, part 1: research findings. journal of the american society for information science and technology, 58(8), 1071-1081. doi:10.1002/asi.20462 markey, k. (2007b). twenty-five years of enduser searching, part 2: future research directions. journal of the american society for information science and technology, 58(8), 1082-1099. doi:10.1002/asi.20601 mount royal university. (n.d.a). fast facts. retrieved 5 dec. 2012 from http://mtroyal.ca/aboutmountroyal/f astfacts/index.htm mount royal university. (n.d.b). inspiring learning: mount royal university academic plan 2012-2017. retrieved 5 dec. 2012 from http://www.mtroyal.ca/wcm/groups/p ublic/documents/pdf/academic_plan.p df novotny, e. (2004). i don’t think i click: a protocol analysis study of use of a library online catalog in the internet age. college & research libraries, 65(6), 525-537. portmann, c. a., & roush, a. j. (2004). assessing the effects of library instruction. the journal of academic librarianship, 30(6), 461-465. doi:10.1016/j.acalib.2004.07.004 smith, t. g. (2001). keeping track: librarians, composition instructors, and student writers use the research journal. research strategies, 18, 21-28. tabatabai, d., & shore, b. m. (2005). how experts and novices search the web. library & information science research, 27(2), 222-248. doi:10.1016/j.lisr.2005.01.005 urquhart, c., & rowley, j. (2007). understanding student information behavior in relation to electronic information services: lessons from longitudinal monitoring and evaluation, part 2. journal of the american society for information science & technology, 58(8), 1188-1197. doi:10.1002/asi.20562 warner, d. a. (2003). programmatic assessment: turning process into practice by teaching for learning. the evidence based library and information practice 2012, 7.4 66 journal of academic librarianship, 29(3), 169-176. wakimoto, d. (2010). information literacy instruction assessment and improvement through evidence based practice: a mixed method study. evidence based library and information practice, 5(1), 82-92. retrieved 4 dec. 2012 from http://ejournals.library.ualberta.ca/ind ex.php/eblip/article/view/6456/6447 wang, p., berry, m.w., &yang, y. (2003). mining longitudinal web queries: trends and patterns. journal of the american society for information science and technology, 54(8), 743-758. doi:10.1002/asi.10262 warner, d. a. (2003). programmatic assessment: turning process into practice by teaching for learning. the journal of academic librarianship, 29(3), 169-176. webster, j., & rielly, l. (2003). a library instruction case study: measuring success from multiple perspectives. research strategies, 19, 16-32. doi:10.1016/j.resstr.2003.11.001 willson, r., & given, l. m. (2010). the effect of spelling and retrieval system familiarity on search behavior in online public access catalogs: a mixed methods study. journal of the american society for information science & technology, 61(12), 2461-2476. doi:10.1002/asi.21433 appendix a web survey study title: the impact of one-shot library sessions on student research ⬜ i have read the study information and consent to have my research log used anonymously in rebekah willson’s study, the impact of one-shot library sessions on student research 1. have you ever had a library session for any other class? a. ⬜ yes ⬜ no ⬜ prefer not to respond 2. if yes, how many sessions have you had? 3. please rate the following from most useful (1) to least useful (9) a. explanation of the assignment b. discussion of the resources to use c. discussion of how to use the resources d. time to work independently e. working/discussing with class mates f. individual help from librarian g. individual help from class instructor h. discussion about search difficulties i. citation discussion j. other 4. how prepared did you feel to do research before this session? a. ⬜ unprepared ⬜ somewhat unprepared ⬜ somewhat prepared ⬜ prepared 5. how prepared do you feel to do research after this session? a. ⬜ unprepared ⬜ somewhat unprepared ⬜ somewhat prepared ⬜ prepared evidence based library and information practice 2012, 7.4 67 6. are you interested in possibly participating in an interview to follow up on this survey? if so, please open a new window or tab in your web browser and copy and paste the address below into your address bar. this will allow me to get your e-mail address to contact you without attaching your e-mail to this survey.” appendix b study title: the impact of one-shot library sessions on student research ⬜ i have read the study information and consent to have my research log used anonymously in rebekah willson’s study, the impact of one-shot library sessions on student research research log research logs allow you to keep track of your research – the searches you have done, the resources you have used – and to plan for what you need to do next. they also help prevent you from duplicating the work you’ve done. topic/thesis statement: _____________________________________________________________________________________________ _____________________________________________________________________________________________ _____________________________________________________________________________________________ _____________________________________________________________________________________________ ideas/concepts (and synonyms and words related to ideas/concepts): _____________________________________________________________________________________________ _____________________________________________________________________________________________ _____________________________________________________________________________________________ _____________________________________________________________________________________________ today’s date resource used (e.g. academic search complete) keywords/search (e.g. “global warming” and ocean* in keywords) resources found (e.g. good results, e-mailed smith & jones article) notes to self (e.g. try synonyms for global warming) did you ask questions during the session? ⬜ yes ⬜ no what type of question(s) did you ask? ⬜ my topic ⬜ vocabulary ⬜ articles ⬜ books ⬜ problems searching ⬜ technical problem did your question get answered? ⬜ yes ⬜ no do you still have questions? ⬜ yes ⬜ no what questions do you think may come up later? _____________________________________________________________________________________________ _____________________________________________________________________________________________ _____________________________________________________________________________________________ _____________________________________________________________________________________________ microsoft word ed_1514 evidence based library and information practice 2008, 3:1  1 evidence based library and information practice     editorial    facing my fears    lindsay glynn  editor‐in‐chief    acting head, public services, health sciences library  memorial university of newfoundland  st. john’s, newfoundland, canada  e‐mail: lglynn@mun.ca      © 2008 glynn. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      i’m scared. i’m nervous. in a few short  weeks the contractors and electricians  will take over my library for several  months. they will drill huge gouges in  the concrete floor, hammer, saw, scrape,  move, wire, etc. no doubt they may  have to be asked to keep their voices  down once or twice. half of the print  journal collection will be relocated to  accommodate a new teaching lab that  will also double as an information  commons.  the planning has been going  on for many months.  we have  consulted with other libraries, reviewed  the literature, identified the needs of our  various user groups, measured space,  tested technical possibilities, and met  with architects and engineers.  up until  now the new lab was an organic idea on  paper, discussed over coffee and in  meetings. that’s fairly easy to deal with.  but just around the corner it becomes a  reality and i’m a bag of nerves. have we  made the right decisions? will it address  all our needs? is there anything i forgot  to consider? what if our users don’t like  it? what if it is a complete failure?!  theoretically, it should be ok. i’ve  followed the right steps and worked  with a creative, talented and dedicated  team. this is different from trying out a  new instructional technique or  reorganizing the information desk. this  is big. i talk the evidence based talk  regularly, but now i am walking the  walk in a bigger way than i had ever  imagined.     change can be frightening. moving out  of comfort zones is not easy. having  said that, the challenge can be  invigorating and the change, refreshing.    evidence based library and information practice 2008, 3:1  2 i find myself welcoming the change as  much as i dread it. i’ll face my fears and     see it through to the implementation  and evaluations and beyond. and hey,  no matter what the outcome, it should  make for a good paper. if anyone else  out there is going through a similar  process, i’d be interested in comparing  notes.    i invite you to try something new this  year in your work environment or in  your professional activities. implement  a new service; change a process; publish  a paper; present at a conference; teach a  new workshop. find out what should  work through research and  investigation, and go for it. yes, it may  cause anxiety and you may have to  spend time convincing others to accept  change, but it beats just thinking about  it.     now, if i could just crawl under a  reference book until the lab is built i’ll  be all set. call me when it’s done.           microsoft word comm_ryan.doc evidence based library and information practice 2006, 1:4  77 evidence based library and information practice     commentary      ebl and library assessment: two solitudes?        pam ryan  interim head, science and technology library  university of alberta libraries  edmonton, alberta, canada  e‐mail: pam.ryan@ualberta.ca      © 2006 ryan. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      introduction    as booth notes in a recent commentary on  the conceptual and practical links between  performance measurement and evidence‐ based library and information practice  (eblip), there has been a discernable creep  among segments of the library community  that seemingly existed as two solitudes:  those in evidence based librarianship (ebl)  circles and those in the library assessment  practitioners group (“counting what  counts” 63). beginning in 2005, individuals  from one group have been showing up at  the others’ conferences and events to discuss  their methods, frameworks and processes.   are these separate movements within  librarianship forming theoretical bridges? is  some sort of merger, fusion or takeover in  the future? or are these simply collegial  discussions about our evidence‐based  leanings in librarianship? is all evidence‐ based practice in librarianship, that is, some  form of research‐derived data guiding the  decision‐making of practitioners, subject to  the theoretical framework proposed by the  ebl movement? if so, are the tools and  practices of library assessment rigorous  enough, by ebl theory standards, to afford  equal participation in these evidence‐based  practice circles or will assessment  practitioners forever be relegated to wallow  as devotees of the lowest cells of eldredge’s  (2002) exploratory research evidence chart?  if we are just now coming to understand our  similarities, will our differences be enough  that we wish never to be one movement and  therefore forever remain as two solitudes in  evidence‐based practice?  background     the ebl and library assessment movements  grew up separately but at approximately the  same time, with ebl first appearing in the  literature in 1995 in an editorial by margaret  haines (booth evidence based practice for  information professionals 8) and the initial  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:4  78 calls for a “culture of assessment” in  academic libraries appeared in articles by  stoffle in 1996 and lakos in 1999.  library  assessment has its roots in the need for  academic libraries to redefine measures to  show the library’s real contributions to the  communities they serve and to develop  collections and services based on user needs  and preferences. this user‐centred  movement shares many of the research tools  and methodologies of marketing, business,  and other disciplines of the social sciences.  evidence based librarianship developed  out of the practice of health librarians’  participation in evidence based medicine  (ebm) initiatives and turning that  experience to a reflective review of the use  of research evidence in their own  professional decision‐making practices.     definitions and differences    even though the definition of ebl has  evolved since the earliest of ebm‐influence  days, the core of ebl practice still rests on  finding in the research literature the best  evidence to answer a question or problem  and then appraising, applying and  evaluating specific interventions.  the most  recent definition of ebl, where the name of  the movement is currently proposed as  evidence based library and information  practice (eblip), is that it:    “ʺ….seeks to improve library and  information services and practice by  bringing together the best available  evidence and insights derived from  working experience, moderated by  user needs and preferences. eblip  involves asking answerable questions,  finding, critically appraising and then  utilizing research evidence from  relevant disciplines in daily practice. it  thus attempts to integrate user‐ reported, practitioner‐observed and  research‐derived evidence as an  explicit basis for decision‐making.ʺ  (booth “counting what counts” 65)     the assumption is that an answer is  available in the research literature of  librarianship or related fields and that  knowing how to evaluate research literature  will guide the practitioner to the best  answer. looking to the literature is the first  source when a question is formed. however,  the understanding in the eblip process that  evidence is to be “moderated by user needs  and preferences” shows some common  ground with the library assessment  approach, even though the definition  doesn’t explicitly speak to how those needs  are determined, evaluated or assessed.    a definition of library assessment work that  the assessment community of practitioners  adheres to is elusive. there appears to be no  standing definition in common use. a  helpful, although broad, definition used to  describe the topics of interest for the  libraryassessment.info community discussion  blog is:    “…any activities that seek to measure  the library’s impact on teaching,  learning, and research, as well as  initiatives that seek to identify user  needs or gauge user perceptions or  satisfaction. the overall goal is data‐ based and user‐centered continuous  improvement of library collections and  services.” (kyrillidou and ryan)    while booth’s recent article (“counting  what counts”) focuses on the links between  performance measurement and evidence‐ based library and information practice, it  should be noted that performance  measurement is just one evaluative  approach of the many evaluative methods  available in the library assessment toolkit.  bertot provides a good summary of selected  approaches to evaluative research,  assessment’s parallel to eldgredge’s levels  evidence based library and information practice 2006, 1:4  79 of evidence chart, if you will, with examples  of where the contact with library users is  made.    in library assessment, there is a reliance on  the literature for understanding best  practices in qualitative methodology  processes (i.e. usability testing, focus groups)  and the use and sharing of tested, reliable,  and standardized tools (i.e. questionnaires,  tests, evaluation forms). however, there is  no expectation of answers in the literature.  in assessment work, evidence can only be  local. the nature of assessment work is that  it circles around what users are experiencing,  so the appropriate study design is  frequently a qualitiative, user‐engaging  method. only your users, or the usage trails  they leave behind in your systems, can tell  you where your web site isn’t up to snuff,  that your philosophy collection isn’t being  used, how your study space is insufficient,  or how your staff are perceived. while it is  still the role of the practitioner to determine  how to fix problems, library assessment  practice dictates that problem‐identification  and service satisfaction measures can only  come from your library’s community of  users.     common ground and work ahead    one area where those involved in ebl and  library assessment could work  collaboratively would be in defining areas  where the ebl model and library  assessment practices are best suited, where  they overlap, and where differences excel or  fall short.  for example, an issue that needs  further discussion is how the ebl literature‐ immersion process of “formulate, search,  appraise, assess, evaluate” (eldredge 2006  342) fits into the innovation cycle of new  service development. with continuous  improvement at the core of library  assessment work, more often than not,  practitioners are working on new service  development or new ways of providing  access to collections based on their  systematic work in determining user  satisfaction and local user needs. this is  where library assessment work excels and  where its methods are tightly integrated into  the new service development process. with  the time lag of the research lifecycle, is the  ebl process relevant in today’s dynamic  web 2.0‐influenced service environment? if  so, what does that process model look like?  where does the google‐esque service  development model of rapid, imperfect  development, beta launch, user‐engagement  in evaluation, and then modification based  on user take‐up and feedback, fit within the  ebl theoretical framework? is ebl, so  reliant on research literature results for  decision, only relevant in traditional or  pedagogical areas of librarianship where  change might wait for the research cycle?     to date, progress in the ebl movement has  been made by a group of innovative, like‐ minded individuals mustering early‐ adopting, like‐minded and interested  individuals. this is where individuals in  ebl are meeting with individuals in library  assessment. our discussions on all aspects  of librarianship that include using evidence,  data, evaluation and research methods in  our practice need to continue as many  questions remain. we need not expect or  hope that the outcome is one theoretical,  evidence‐based practice umbrella. a good  start is already underway in the  acknowledgment that we need awareness  and understanding of each others’ methods,  frameworks and processes and that there is  keen interest in working together to move  forward.    acknowledgement    this commentary is based, in part, on  material previously presented at the 2006  canadian library association conference:    ryan, pam. “good libraries use evidence:  cultivating a culture of assessment and  evidence based library and information practice 2006, 1:4  80 evidence‐based decision‐making in  academic libraries.” canadian library  association conference. ottawa, ontario.  june 2006  <http://www.cla.ca/resources/cla2006/presen tations/cla_ebl.pdf>.      works cited  bertot, john carlo. “assessing digital  library services: approaches, issues, and  considerations.” international symposium  on digital libraries and knowledge  communities in networked information  society (dlkcʹ04). university of tsukuba,  tsukuba, ibaraki, japan. march 2004. 3  november 2006  <http://www.kc.tsukuba.ac.jp/dlkc/e‐ proceedings/papers/dlkc04pp72.pdf>.  booth, andrew. “counting what counts:  the link between performance  measurement and evidence based  information practice.” performance  measurement and metrics 7.2 (2006): 63‐74.    booth, andrew, and anne brice, editors.  evidence‐based practice for information  professionals: a handbook. london: facet  publishing, 2004.                                   eldredge, jonathan. “evidence‐based  librarianship: the ebl process.” library hi  tech 24.3 (2006): 341‐54.    eldredge, jonathan. “evidence‐based  librarianship: levels of evidence.”  hypothesis 16.3 (2002): 10‐13.   lakos, amos. “the missing ingredient:  culture of assessment in libraries.”  performance measurement and metrics 1.1  (1999): 3‐7.    kyrillidou, martha, and pam ryan.  libraryassessment.info. 3 november 2006 <  http://libraryassessment.info/?page_id=7>.    stoffle, carla, robert renaud and jerilyn r.  veldof. “choosing our future.” college and   research libraries 57.3 (1996): 213‐25.         http://www.cla.ca/resources/cla2006/presen http://www.kc.tsukuba.ac.jp/dlkc/e%e2%80%90proceedings/ http://libraryassessment.info/?page_id=7 microsoft word practice4580 evidence based library and information practice 2009, 4:1 53 evidence based library and information practice using evidence in practice patron preferences for folksonomy tags: research findings when both hierarchical subject headings and folksonomy tags are used elaine peterson information resources specialist, montana state university libraries bozeman, montana, united states of america email: elainep@montana.edu received: 17 dec 2008 accepted: 31 jan 2009 © 2009 peterson. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. setting montana state university is a comprehensive research university of 13,000 students, offering both masters and doctoral degree programs. the library has been the archival repository for paper copies of all theses and dissertations received from the graduate studies program since its inception. as the state's land-grant university, montana state university emphasizes the sciences, engineering, and architecture. the majority of graduate students’ theses and dissertations reflect this focus. librarians have always catalogued and assigned traditional subject headings to these dissertations and theses using library of congress subject headings (lcsh). in 2005 the library's depository paper copy of theses and dissertations was replaced with an electronic file copy, submitted by the graduate student and transmitted to the library by the graduate studies office. the library created an electronic theses and dissertations (etd) database, making these full texts of graduate research freely available to anyone with internet access. even with the separate etd database, the library continues to apply lc subject headings to each dissertation and thesis. the library adds marc records to the online catalogue using those same lcsh headings. problem with the emergence and popularity of folksonomy as an option for subject tagging, library staff discussed the costs and benefits of continuing to construct and apply traditional subject headings, now that patrons can generate their own tags. could the expensive practice of selecting lcsh headings be dropped? would folksonomy tags be sufficient for the subject indexing of the etds? evidence based library and information practice 2009, 4:1 54 evidence one of the problems in making this decision was that there was little research to inform the discussion. a further complication was that few databases exist that allow the use of both hierarchical subject headings, such as lcsh, as well as patron-applied folksonomy tags. as a result there were no comparisons available for research and informed discussion. traditional databases, such as online catalogues or journal databases, apply hierarchical subject headings or index terms, while newer ventures, such as facebook or flickr, rely on folksonomy tagging. because of the lack of research, the library decided to move ahead and gather data to see if this was a worthwhile course of action. implementation folksonomy tags were added to the etd database in 2006. by placing folksonomy tagging opportunities on the etd home page in a prominent location and making it easy to use, patrons were encouraged to add their own tags. although the feature was added to supplement access to the etds and to provide a more up-todate, collaborative atmosphere for the electronic resource, the result has provided an interesting test bed for two subject cataloguing systems residing together. the home page for the montana state university etd database (figure 1) provides a traditional search interface where the user can browse by author, title, or subject heading, as well as by programs, committee chairs, and dates. these are traditional search options, and the subject search utilizes the constructed lcsh headings. each of these marc records is duplicated in the online catalogue and loaded into the oclc database. thus, there are multiple avenues of access to an etd based on the marc record. figure 1. montana state university library etd home page <http://etd.lib.montana.edu/etd/view> evidence based library and information practice 2009, 4:1 55 on the right side of the home page is a blueshadowed area that focuses on the folksonomy tags. the heading, “most popular etd tags,” leads users to this area. an initial click shows the tags listed alphabetically in columns, with a toggle option to view them as a cloud tag. a click on any of the tags connects the user to the etds tagged with that term. users reading an individual etd are encouraged to add tags to the record. upon completion of a graduate degree and acceptance of the thesis or dissertation, the student submits the final document to the library, with an abstract and list of suggested keywords. the library staff uses those keywords as an aid in subject classification with lcsh subject headings. for example, jon hasenbank’s 2006 thesis for his master’s degree in mathematics, “the effects of a framework for procedural understanding of college algebra students’ procedural skill and understanding,” was added to the etd and classified with these lcsh subject headings: • cognitive psychology • curriculum-based assessment • mathematical readiness • mathematics—study and teaching (higher) sometime after the creation of the etd with its access points, a patron added the folksonomy tag: “wachovia cards.” this is a perfect example of a user-generated tag. there is nothing contained in the keywords from the author of the dissertation to create this unique heading, but one user found the subject “wachovia cards” relevant to the content of the dissertation and added it as a folksonomy tag. outcome with the steady appearance of folksonomy tags, this researcher wanted to know two things: • how often were they added to the etds? • why were they being added? if it were possible to understand why the tags were being applied, perhaps it would then be possible to understand how patron needs were not being met by the lcsh headings. or, perhaps sufficient use of folksonomy tags would warrant dropping the lcsh headings entirely. in december of 2007, one year after the introduction of folksonomy tags, there were 572 etd titles in the database. each of those titles had at least one lcsh heading, although the average was four headings per title. database users had added folksonomy tags to only 14 (2.4%) of the etds. one year later, in december of 2008, the etd database had grown to 678 titles and a total of 55 tags. patron-generated tags had been applied to only 8% of the etds. despite encouragement and ease of use, most users of the etd database do not take advantage of the ability to tag the theses and dissertations. although there has been an increase, there is the question as to why users have not been taking advantage of tagging the etds. when folksonomy tags are applied, what, if anything, does that say about the deficiencies of the database’s subject access points? tags were sorted in order to understand their types and use. the first review of the tags in an alphabetical list revealed that most had been used only once. of the 55 folksonomy tags in the database, only five had been used as a descriptor for more than one etd: • birds • lost in space • sci-fi • star wars • teaching the folksonomy tags in the etd database have been generally unique to the research of a particular author. it may also be true that the tags were unique to the research interests of the library patron who applied the folksonomy tag. a second feature of the tags is that for the most part, they would have been disallowed in a traditional, hierarchical subject cataloguing evidence based library and information practice 2009, 4:1 56 scheme such as lcsh. examples include: • headings used to indicate a subject discussed in only a single, small portion of the entire dissertation. an example is the use of “moa” as a tag in one dissertation, even though it was mentioned in only one paragraph of the entire work • headings not recognized by lcsh, such as “dominant voices” or “fisher” • inverted headings used by lcsh, such as “bridges, concrete” • broad terms that lcsh would not have permitted as descriptors, such as “birds” or “transportation” • misspelled headings such as “foucoult” (instead of “foucault”) that can hinder an effective search for the most part, the folksonomy tags do not duplicate what has already been added as lcsh subject headings, although there are a handful of tags that have been added that are identical to lcsh subject headings. reflection after two years of gathering data, there are a few things that can be said about the patron tags in the etd database. first, patrons are using folksonomy tags, and the usage of the tags is increasing. an increase from 2% to 8% in one year, although small, is still significant. usage of the tags is evidence that permitting folksonomy tags in the etd database has met patrons’ needs. second, there is little chance at this point in time that folksonomy tags supplied by the users will take the place of structured subject headings applied to the etds. the numbers are not there, and there is little overlap between what users want in their tags and those applied by librarians as subject headings. the uses of the subject headings and tags are quite different. when searching for information, a structured search using traditional subject headings is the best tool for efficient retrieval of data. when having a conversation with another researcher, or when using a tag for one’s own research, a user-supplied tag is appropriate. it will be interesting to see if the usage of the lcsh headings and tags achieves more overlap over time. at present, it appears that the uses of lcsh and folksonomy are quite different, and that these parallel modes of access should continue to maximize usability and ease of access to the database. evidence based library and information practice evidence based library and information practice 2011, 6.3 40 evidence based library and information practice article the transition from print to electronic journals: a study of college and university libraries in indiana jo mcclamroch electronic resources acquisitions librarian indiana university wells library e350 bloomington, indiana, united states of america email: jmcclamr@indiana.edu received: 11 may 2011 accepted: 11 june 2011 2011 mcclamroch. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objectives – this study examines what factors are considered by college and university libraries in indiana when making the decision to cancel subscriptions to print journals when an electronic equivalent is available. the study also looks at who the primary decision makers are in this regard. libraries at public and private institutions of varying sizes were included in the study. methods – an online survey was sent to seventy-three libraries in the consortium, academic libraries of indiana. structured interviews with administrators at nine libraries were also conducted. results – academic libraries in indiana use subscription cost, redundancy of formats, student preference, budget reductions and usage as the primary factors in cancelling print journal subscriptions in favour of their electronic counterparts. there is also a preference for the electronic format for new subscriptions even when a print version is also available. conclusions – the study indicates that subscription cost is the most important consideration in the journal cancelation process with other factors also having an effect on the preference of libraries for electronic versions of journals. the study also shows that libraries at public and private colleges and universities are at different mailto:jmcclamr@indiana.edu� evidence based library and information practice 2011, 6.3 41 stages of moving away from print to an online-only journal format. at the same time, there is consensus that a small collection of print titles will still be needed. the primary decision-makers are librarians, faculty, and library administrators. introduction electronic journals have both beguiled and bedeviled academic libraries since being introduced on a broad scale in the early 1990s. librarians were interested yet cautious about this new, untested format. kownacki (1991) posed the question “do our patrons really want these journals in this format?” (p. 81). in a matter of years, that question was answered with a resounding “yes.” as the technology continued to improve and interfaces became more user-friendly, it was evident that e-journals were very much welcomed by patrons. over the past two decades there has been a steady march toward electronic-only journal subscriptions in academic libraries. from the initial uncertainty regarding user acceptance of the electronic format to its current ubiquitous use, users not only embrace but expect online access. academic libraries have moved from a print-centric environment to an electronic world where access to thousands of journal titles is within the reach of even the smallest of institutions. in a time of increasing fiscal constraint reducing print subscriptions is a means of reducing library expenditure, particularly when these are duplicated in an electronic collection. however what are the factors involved when academic libraries make decisions about journal subscriptions and are these factors the same regardless of size or background of the institution? literature review previous research in this area has focused on single aspects of print to e-journal transition. there have been studies of a single institution (kalyan 2002; maple, wright & seeds, 2003), studies of a particular discipline such as medical libraries (gallagher, bauer & dollar, 2005; weston & acton 2009), studies of arl libraries (rupp-serrano, robbins & cain, 2002; prabha 2006), studies of selected academic departments (frazer & morgan, 1999), and a study of small college and university libraries (spencer & millson-martula, 2006). historically the two most common factors driving journal cancelation decisions are cost – increasing subscription prices and shrinking budgets. both commercial and noncommercial publishers have often had subscription price percentage increases in the double digits. johnson and luther (2007) report on interviews with arl member libraries, private and public, which cite these two cost issues as key factors in assessing whether or not they can sustain paying for two formats. in a study of small college and university libraries, spencer and millsonmartula (2007) found that subscription cost was the primary factor prompting a serials review, and it was the third highest factor in cancelation decisions. subscription cost does not stand alone as a factor in journal retention decisions. as libraries have moved from a print-centric environment to one in which e-journals dominate, the target for cost cutting has changed from cancelling duplicate print subscriptions to cancelling print subscriptions that are now redundant to their electronic equivalents including those available in database aggregators. the question is which redundant print titles to cancel and how to determine which factors to utilize in making those decisions. some libraries include use as a factor to employ for decision making. an example of a usage study is one carried out at by gallagher et al. (2005) at the yale university cushing/whitney medical library. staff undertook a three month study of the library’s 1,249 current print journals. they found that evidence based library and information practice 2011, 6.3 42 657 (53%) of titles received no use during the study and only nine titles (7%) were used more than once a month. as a result they cancelled 212 existing print subscriptions when that was permitted by license agreements. in a study of print serials use over a ten-year period 1992-2002, rosati (2006) found that as the library’s electronic resources grew print use plummeted. her study used statistical data to identify the one hundred most used journals in 1992 and then tracked their use over ten years. the top ten titles in that list of the one hundred most used print journals showed a total decrease in print use of 85.4% over the course of the usage study. during that same time period, use of the bottom ten titles decreased by 85.8%. these findings were used to inform print cancellation decisions. in the absence of a systematic usage study, however, even a casual look at a library’s uninhabited reading room is a visible indicator of low use. another factor to consider in the decisionmaking process is the preference by students for the electronic format. there is general agreement that both students and faculty prefer electronic access to e-journals. this preference was confirmed by a study in 2002 comprised of interviews with over 3,000 undergraduate and graduate students, and faculty at colleges and universities. researchers found that students and faculty were regular users of e-journals whether in the library or off-campus (maple et al. 2003). user preference notwithstanding, librarians must question whether e-journals are exact duplicates of the print versions. despite assurances from publishers that the electronic version is equivalent to the print, many case studies have shown otherwise. a study conducted at old dominion university found that there was more content in the print than in the electronic for several full-text titles provided in the iac infotrac searchbank database. the authors found that entire articles were missing as were supplementary materials such as editorials, book reviews, and job postings (frazer & morgan, 1999). in a study at the university of colorado at colorado springs kraemer family library (kfl) an analysis was carried out of 79 peerreviewed journals primarily in the social sciences and humanities all of which were covered in one or more of five database aggregators. in this study only major articles were examined and supplementary material was not evaluated. the findings showed that only 83% of the major articles were available online (sprague & chambers, 2000). in a follow-up study 10 years later, 20 of the original 79 titles were still available in aggregators subscribed at kfl. an improvement was seen as a comparison between the print and electronic versions showed nearly 100% coverage of major articles in the online edition (thohira, sprague & chambers, 2010). in a study at the national library of medicine, 149 titles were examined for what the authors call “divergent content” between the print and electronic editions. this study looked at both articles and supplementary material. findings show that while 63.76% of titles included all articles in both print and electronic formats, 36.24% published some articles only in the electronic edition. in looking at editorials, in 89.69% of instances they were published in both the print and electronic editions, while 3.09% appeared only in the online version and 7.22% were published only in the print. the study found similar instances of divergent content in all other categories of supplementary material such as book reviews and letters/commentary (weston & acton, 2009). in light of findings such as these publishers need to be asked which is the version of record? hunter (2007) raises this question of version control in light of the user demand to see journal articles as soon as possible. what if an article is subsequently updated and re-released? which is now the version of record? as electronic journals continued to displace print journals as the preferred format, concerns arose around the need for a permanent digital archive. unlike print, there was no longer a physical object to retain. in describing their experience at the mit libraries, duranceau, lippert, manhoff and snowden (1996) determined that it was not feasible for them to maintain a local electronic evidence based library and information practice 2011, 6.3 43 archive and that it was likely to be a difficult role in general for research libraries to take on. rupp-serrano et al. (2002) conducted an e-mail survey of arl libraries to learn whether they had collection development policies in place for e-journals. their purpose in collecting and analyzing the data was to develop potential criteria that might assist academic libraries in formulating de-selection policies for redundant print subscriptions. they stated that libraries needed to determine what for them what would constitute a “reliable archive.” they suggested that libraries ask publishers whether they planned to maintain an electronic archive so that libraries would no longer need to continue maintaining a print archive. hunter (2007) noted progress made since the mid-1990s in developing a permanent electronic archive of the digital record, one in which libraries and publishers alike would have confidence. she observed that there was a shared recognition by both libraries and publishers for the need to participate in such an endeavor. to that end, two entities, portico and lockss, were created to collect and maintain “dark archives” for a portion of the digital record. moghaddam (2007) discussed the emergence of the national library of the netherlands as another and larger archival entity where publishers’ electronic output could be deposited with the assurance of longterm preservation. as noted above, much of the previous research focuses on single institutions. this study examines factors considered by college and university libraries across indiana when making the decision to cancel subscriptions to print journals when an electronic equivalent is available. the study also looks at who are the primary decision makers in this regard. the goal of the study was to obtain a snapshot of the current status of the transition from print to electronic journals in this particular group of academic libraries. the consortium which was studied, academic libraries of indiana, offered a demographic cross-section of libraries as it included both public and private colleges and universities of varying sizes. methods sample academic libraries of indiana (ali) is a 73member consortium whose mission is “to enhance and enrich access to the full range of information resources and services required to improve the quality of teaching, learning, research, and engagement in indiana’s colleges, universities, and seminaries through collaboration, resource sharing, and advocacy.” ali members are from both private and public institutions of varying enrollments ranging from 200 to over 50,000. there are 71 unique institutions represented in the ali membership. two universities – indiana university and ivy tech community college – have multiple campuses though each campus is an ali member in its own right. survey an online survey comprising 12 questions (described in the tables below) was sent in july 2010 to the designated contact at each ali member library. the survey was available for two weeks. responses were received from 26 member libraries for a response rate of 36%. not all respondents answered all questions. interviews structured interviews were conducted in august 2010. there were eight open-ended questions for the interviews (see appendix). interviews were conducted by phone or onsite. the interviews were carried out to obtain details for some of the decision factors included in the survey as well as for other questions that could not be easily collected from the survey alone. results survey data in the online survey were collected for three demographic categories: status of parent institution (private or public), fte enrollment, evidence based library and information practice 2011, 6.3 44 and highest degree offered. of the total survey responses 57.7% (15) were from libraries at private colleges and universities and 42.3% (11) were from public colleges and universities. respondents reported enrollments ranging from under 1,000 (26.9%) to over 25,000 (3.8%) (table 1). regarding highest degree offered the responses were associate 8.3%, bachelor 20.8%, master 37.5%, and doctorate 33.3%. table 1 fte enrollment percentage responding libraries responding under 1,000 26.9% 7 1,001 5,000 42.3% 11 5,001 10,000 23.1% 6 10,001 25,000 3.8% 1 data regarding total materials budgets are provided in table 2. the largest group of respondents included seven libraries (30.4%) which reported budgets under $100,000 while the second largest group of five libraries (21.7%) were at the opposite end of the spectrum, reporting materials budgets over $1 million. table 2 total materials budget percentage responding libraries responding under $100,000 30.4% 7 $100,001 $250,000 17.4% 4 $250,001 $500,000 13.0% 3 $500,001 $1,000,000 17.4% 4 over $1,000,000 21.7% 5 there were two questions on the survey addressing how materials budgets were allocated between print and electronic journals. twelve respondents (54.5%) reported spending under $50,000 on print subscriptions and three (13.6%) reported expenditures from $250,001-$500,000 (table 3). table 3 expenditures for print journals percentage responding libraries responding under $50,000 54.5% 12 $50,001 $100,000 22.7% 5 $100,001 $250,000 9.1% 2 $250,001 $500,000 13.6% 3 results for e-journal expenditures –single-title subscriptions, titles available in aggregators or in publisher packages – are shown in table 4. responses covered a range; six libraries reported expenditures under $50,000 and one reported spending over $1 million on ejournals. table 4 expenditures for electronic journals percentage responding libraries responding under $50,000 27.3% 6 $50,001 $100,000 18.2% 4 $100,001 $250,000 22.7% 5 $250,001 $500,000 18.2% 4 $500,001 $1,000,000 9.1% 2 over $1,000,000 4.5% 1 regarding the number of print subscriptions, 58.3% responded they subscribe to under 500 titles and one library reported current print subscriptions from 2,001 – 5,000 (table 5). evidence based library and information practice 2011, 6.3 45 table 5 current print subscriptions percentage responding libraries responding under 500 58.3% 14 501 1,000 25.0% 6 1,001 2,000 12.5% 3 2,001 5,000 4.2% 1 two survey questions addressed responsibility for decision making in journal cancelation decisions (tables 6 and 7). when asked who is involved in the cancelation of print journals, survey responses indicated a high level of involvement by librarians (90.5%), faculty (52.4%) and library administrators (33.3%) (table 6). the responses regarding decision-makers for e-journal cancelations parallel those for print: librarians (95.2%), faculty (57.1%) and library administrators (33.3%) (table 7). three questions central to the survey presented different scenarios regarding specific factors considered in making a decision to cancel a print subscription in favour of its electronic counterpart. each question provided a list of ten factors to consider: • budget reduction • subscription cost • redundancy • faculty recommendation • usage statistics • confidence in perpetual access • space • print used less than electronic • electronic preferred by students • change in curriculum results for each of the three scenarios are found in tables 8, 9, and 10. for the first scenario, general cancelation factors, the top five responses were subscription cost (81.8%), electronic preferred by students (81.8%), table 6 who is involved in the cancelation of single-title print journals? decision-makers percentage responding libraries responding librarians 90.5% 19 faculty 52.4% 11 library administrators 33.3% 7 library committee 14.3% 3 librarian/faculty committee 9.5% 2 faculty committee 4.8% 1 students 0.0% 0 table 7 who is involved in the cancelation of single-title e-journals? decision-makers percentage responding libraries responding librarians 95.2% 20 faculty 57.1% 12 library administrators 33.3% 7 library committee 14.3% 3 librarian/faculty committee 4.8% 1 faculty committee 0.0% 0 students 0.0% 0 evidence based library and information practice 2011, 6.3 46 redundancy (63.6%), print used less than electronic (54.5%), and budget reduction (45.5%) (table 8). the second question offered a scenario regarding considerations when cancelling a title in an aggregator. survey respondents indicated that the top five factors were the same as the general scenario but the second and third highest ranked factors – electronic preferred by students, and redundancy were reversed (table 9). the third scenario sought information on factors taken into account if a journal title were to be cancelled outright. results are shown in table 10. as with the two previous scenarios, the top factor is subscription cost (95.5%), followed by budget reduction (65.6%), usage statistics (59.1%), faculty recommendation (54.5%), and redundancy (40.9%). table 8 when making the decision to cancel a single-title print subscription in favor of its electronic version, what factors are considered? percentage responding libraries responding subscription cost 81.8% 18 electronic preferred by students 81.8% 18 redundancy 63.6% 14 print used less than electronic 54.5% 12 budget reduction 45.5% 10 confidence in perpetual access 40.9% 9 change in curriculum 31.8% 7 faculty recommendation 27.3% 6 usage statistics 27.3% 6 space 27.3% 6 table 9 when making the decision to cancel a single-title print subscription when there is a duplicate version in an aggregated database, what factors are considered? percentage responding libraries responding subscription cost 81.8% 18 redundancy 81.8% 18 electronic preferred by students 72.7% 16 print used less than electronic 54.5% 12 budget reduction 45.5% 10 usage statistics 40.9% 9 confidence in perpetual access 31.8% 7 space 31.8% 7 change in curriculum 27.3% 6 faculty recommendation 18.2% 4 evidence based library and information practice 2011, 6.3 47 interviews interviews took place with library administrators – eight deans and directors, and one head of a periodicals division at nine college and university libraries, five private and four public. in terms of institution size, at the private institutions, enrollments ranged from 200 to 3,000 while at the public institutions enrollments ranged from 4,500 to over 16,000. administrators were asked if they had experienced changes in their budgets in the previous two years. six reported budgets that were either static or had enjoyed a “periodic influx of money” (three public, three private). two libraries reported decreased materials budgets; one private had a 7% decrease and one public university library had a 25% decrease. only one private university library reported a budget increase in each of the two previous years. when asked what percentage of the materials budget is allocated to all e-resources – singletitle subscriptions, aggregators, publisher packages – library administrators had a variety of responses. for the public colleges and universities, the responses were 33, 54, 60, and a high of 80% of funds dedicated to electronic resources. for the private colleges and universities, administrators reported that the current budget allocations for all electronic resources were 20, 30, 35, 50, and a high of 80%. most said the majority of their electronic journal holdings are in database aggregators. most library administrators emphasized the number of journals they have access to now via aggregators as compared to what was available ten years ago. this was often remarked as a sort of “selling point” to faculty as evaluations were being made for potential print cancelations; that though it may be necessary to make some cancelations, the library would remain information rich because of its subscriptions to numerous aggregator databases such as ebscohost academic search premier and lexis-nexis academic universe. another question asked of administrators was if they would see the day their library went totally electronic with journals. one replied “that’s a sad question but i think it will happen.” a couple of administrators said they could see the collection being at least 80% electronic. all see a rapid decline of print journals. several reasons for retaining print subscriptions were articulated. in addition to maintaining a browsing collection and keeping print for titles with graphics, they cited the need for a core list of “must-have titles.” another reason to continue a print subscription was if it were embargoed in an aggregator. if that were the case, they would table 10 when making the decision to cancel a journal subscription outright, what factors are considered? percentage responding libraries responding subscription cost 95.5% 21 budget reduction 63.6% 14 usage statistics 59.1% 13 faculty recommendation 54.5% 12 redundancy 40.9% 9 change in curriculum 40.9% 9 electronic preferred by students 27.3% 6 print used less than electronic 22.7% 5 space 18.2% 4 confidence in perpetual access 13.6% 3 evidence based library and information practice 2011, 6.3 48 consult with faculty to see how long they could wait for a current issue to be available. finally, one administrator noted that their accrediting agencies required that certain core titles be maintained in print. regarding the number of current print subscriptions; public colleges and universities reported print subscriptions of 100-600; private colleges and universities 200-700. deans and directors emphasized the consultative nature of managing electronic collections. while librarians have a predominant role in decision making, those who were interviewed discussed the important role of faculty in the decisionmaking process. all administrators remarked how critical it is to develop and maintain cordial working partnerships with faculty. however, administrators underscored the primary goal of the library is to support the curriculum and not just to support faculty research. a variety of approaches are used to inform and engage faculty in decision-making. most of these libraries use a librarian-liaison arrangement with both formal and informal contacts. a variety of means of communicating cancelation recommendations are employed such as email sent to individual faculty or to all faculty members in a department, or perhaps posting a proposed list of print cancelations to the library’s website. two libraries reported using a systematic journal review by department every three or four years so that faculty can identify what is no longer used or needed. such a review is also an opportunity for faculty to identify titles that should be acquired. another dean said that library/faculty collaboration is educational in that it gives the library a chance to discuss the materials budget and the constraints that entails in making collection decisions. as she put it, “faculty need to understand the budget is finite.” library administrators indicated that they use many of the same cancelation factors as survey respondents reported. four of nine stated that subscription cost is a deciding factor when either substituting electronic for print or cancelling print outright. five of nine directors identified redundancy as a primary factor in a cancelation decision. one director at a public university said “we can’t pay for this level of duplication.” a librarian at a private college said a duplicate print subscription may be cancelled to add a new subscription “that would benefit the many rather than the few.” a dean at a public university remarked that a journal does not have to be redundant to cancel. several directors said there is a clear preference by students for e-journals. one dean observed that if an e-journal is temporarily unavailable he cannot persuade students to use print even when he offers to walk them to the journal. low usage of print journals was mentioned by four librarians as a cancelation factor. though none of the libraries had done a formal usage study, one dean said that “if a print title is gathering dust it is a candidate for cancelation.” another said print cancelations are based on observation and instinct. budget reductions are used as a decision factor as well. print titles may be cancelled “to repurpose funds for something needed.” only one dean at a public university observed that she would cancel print only if the electronic version was a true full-text duplicate of the print. in the survey and interviews, perpetual access is defined as having ongoing access to journals in aggregators when they are no longer provided by that aggregator. when asked if they had concerns regarding perpetual access, six of nine administrators responded in the affirmative noting that the content of aggregators is “churning.” the dean at a private university remarked that “my nightmare is that it will all go away.” on the other hand, another dean at a private university said that “if it happens it happens. it doesn’t worry me much.” four administrators noted that if a needed title was dropped by an aggregator they would reinstate the title. however, they would reinstate as a print subscription only if an electronic version was not available. evidence based library and information practice 2011, 6.3 49 discussion this study and the literature review have areas of commonality as well as of divergence. there is one thing in which all are unanimous. across the board – in the literature, in the survey, and in the interviews – subscription cost is the primary factor considered in cancelling redundant print journals when an electronic equivalent is available. rising subscription costs are one of the fiscal realities libraries must address. the other is declining materials budgets and the two frequently occur in tandem as many libraries have experienced. in such situations the collection decisions libraries must make become that much more difficult. libraries have always had to operate within the constraints of their materials budgets and have always had to make difficult decisions about where to direct resource dollars, of choosing what to forego in order to maintain the necessary. in many libraries there is not much left to cut and to continue print subscriptions to journals that are available electronically – often in more than one aggregator – is becoming more difficult for libraries to justify. regardless of the size or status of an institution, common factors were apparent in decision making regarding journal cancelation. when asked to rank factors involved in cancelation decisions (whether in general or when the title was held in an aggregator), the top five factors were the same though in different orders. one difference is with the factor “electronic preferred by students,” which was ranked second (18 libraries) in general factors and third (16 libraries) when in an aggregator. when asked about outright cancelation (the third question), however, this criterion was ranked sixth (6 libraries). while the preference of students for the electronic format merits consideration in cancelation decisions, only six libraries indicated student preference was a factor in an outright cancelation decision. this seems to speak to the value of the content rather than the format. when merely swapping formats – electronic for print—there is no loss of content. it is the prospect of losing future content to a valued journal that seems to be the difference. it is possible that in such situations libraries might look for other cost-saving measures such as directing materials dollars away from monographs and moving those funds to journals. there has been much written about usage studies and how that data can be applied in making journal cancelation decisions. at interview library administrators agreed that there is value in usage studies but at the same time they had not conducted any formal studies at their own libraries. the survey yielded mixed results regarding usage studies as a factor to consider in their journal cancelation decisions – 27.3% (6 libraries) in the general cancelation scenario, 40.9% (9 libraries) when in an aggregator, and a high of 59.1% (13 libraries) in the cancel outright scenario. a possible explanation for this high of 59.1% is that because an outright cancelation is being considered, actual use data would be beneficial before making that final decision. journal subscriptions are typically a library’s largest single budget expenditure. decisions regarding journals are made with care and consideration, are done in consultation with faculty, and are made to support the teaching and learning mission of the university. regarding “faculty recommendation” as a factor, it was chosen by six libraries in the general cancelation scenario, by four libraries when in an aggregator, but by 12 libraries in the outright cancelation scenario. it is referenced throughout the literature and is common practice that librarians are consultative with their faculty. that it was ranked fourth out of ten factors in the outright cancelation scenario confirms the importance of the role of faculty in building the collection. administrators interviewed also emphasized the role of faculty in collection decisions. this finding confirms how important their input is when it comes to journal cancelation decisions. another survey factor was confidence in perpetual access for content in aggregators with results of 40.9% (nine libraries), 31.8% evidence based library and information practice 2011, 6.3 50 (seven libraries), and 13.6% (three libraries) across the three scenarios. this suggests some measure of confidence in the general stability of aggregators. on the other hand, the matter of a permanent archive is discussed in the literature as a key factor in the overall examination of whether to give up print in exchange for e-only access. this is a more pressing concern for university research libraries that have an obligation to preserve the scholarly record which smaller universities do not have. while a satisfactory long-term solution is being developed (hunter, 2007), most libraries continue to rely on print as their archive. at the same time, small colleges and universities are addressing this issue albeit it on a smaller scale. for example, one dean at a private university told me an effort is underway in indiana to develop cooperative print archiving among peer institutions to alleviate their concerns that publishers can guarantee they will provide stable electronic archives. neither change in curriculum nor space was rated high as a cancelation factor overall. across the three scenarios for change in curriculum, the highest response was nine libraries (40.9%) in the cancel outright scenario where changes in curriculum would have an obvious impact on what titles to retain. regarding space, it did not weigh heavily as a cancelation consideration with responses of six, seven, and four libraries with the high of seven being a factor when the title is in an aggregator. one item not in the survey but about which a great deal has been written is whether or not an electronic version of a journal is an exact duplicate of the print version, i.e., “how full is full-text.” while this shortcoming was largely resolved a decade ago, there remain concerns by some libraries that this has not been addressed satisfactorily. a case in point is the study by weston and acton (2009) at the national library of medicine. though the universal assumption is that e-journals provide cover-to-cover complete coverage of all content in the print version, that study (and others) demonstrated otherwise. however only one library administrator in this study reported that full-text electronic coverage is a factor in determining whether to cancel a print subscription. in this study, the two private institutions reporting the lowest percentages of total materials dollars going to electronic journals – 20 and 30% – are at religious-affiliated institutions whose users’ needs are specialized and narrow. it is a niche market served by small publishers who may not have the financial capability to convert their print publications to electronic. this is not to say that those publishers eschew e-journals but rather that their subscribers are well served by print. at the other end of the print-toelectronic ratio, one private college library and one public university library report that 80% of their materials budgets are given to electronic resources. the public university library suffered a 25% budget cut in a single fiscal year leaving few viable alternatives other than to cancel print titles that were redundant and/or costly. however, since they were already transitioning to more electronic access this deep budget cut has merely accelerated that process. the library at the private university has not experienced a budget cut but rather is deliberately moving away from print. these two libraries report approximately 200 active print subscriptions. though the pace of the change varies, most libraries expect that their print journal collections will continue to shrink. in a study conducted from 2002-2006, prabha (2006) reports on the mix of journal holdings in arl libraries. she examined three formats: printonly, print + electronic, and electronic-only. during this period, print-only subscriptions decreased from 64% in 2002 to 30% in 2006. during that same period, e-only subscriptions increased from 5% in 2002 to 36% in 2006. few libraries, however, have made the transition to a one hundred percent online-only journal collection. even drexel university, who took the dramatic step to move “overnight” from a primarily print collection to an e-journal collection, retained around 200 titles in print (montgomery & sparks, 2000). evidence based library and information practice 2011, 6.3 51 the results demonstrate that decision-makers are librarians, faculty, and library administrators. the percentages of participation were very close whether cancelling print or electronic journals. three committee configurations were also offered as decision-maker options: library committee, faculty committee, and librarian/faculty committee. when cancelling print journals, three libraries used a library committee, one a faculty committee and two chose a librarian/faculty committee. as to the cancelation of e-journals, three libraries worked with a library committee, none with a faculty committee, and one with a librarian/faculty committee. students were not involved in these decisions, in contrast to expectations prior to the survey. the low use of a committee structure seems to suggest that librarians and faculty act in a collaborative way outside the committee format. conclusions this study of college and university libraries in indiana shows a consistent pattern of movement away from print journals to their electronic versions regardless of institution size or status. when considering cancelling journals, subscription cost was the highest ranked factor in each of the three survey questions: cancel in general, cancel when in an aggregator, and cancel outright. two factors – redundancy and electronic preference of students – were ranked second and third for both general cancelations and when in an aggregator. when considering cancelling journals outright, the second and third ranked factors were budget reduction and usage. according to the results of this study, these five factors are those that will most likely influence journal cancelation decisions in college and university libraries in indiana. based on the rankings of the ten cancelation factors, the survey has shown that these libraries are making careful decisions as they proceed down this increasingly electronic path. whether it is to cancel a print subscription available in an aggregator or to cancel a print subscription completely, librarians and library administrators act in a consultative way by including faculty in the decision-making process. though most current print subscriptions are expected to be phased out, there was agreement that there will always be a small core collection of print titles in libraries, such as journals with graphics and browsing-type titles such as newsweeklies. as reported from the interviews, it is not likely we will see the end of print. what we can expect to see is that more publishers will make the conversion to electronic-only journals. references academic libraries of indiana (ali) mission statement. (n.d.). retrieved 25 aug. 2011 from http://ali.bsu.edu/mission.html duranceau, e., lippert, m., manhoff, m. & snowden, c. (1996). electronic journals in the mit libraries: report of the 1995 e-journal subgroup. serials review, 22(1), 47-61. frazer, s.l. & morgan, p.d. (1999). electronic for-print journal substitutions: a case study. serials review 25(2), 1-7. doi:10.1016/s0098-7913(96)90006-4 gallagher, j., bauer, k. & dollar, d.m. (2005). evidence-based librarianship: utilizing data from all available sources to make judicious print cancellation decisions. library collections, acquisitions, & technical services, 29(2), 169-179. doi:10.1016/j.lcats.2005.04.004 hunter, k. (2007). the end of print journals: (in)frequently asked questions. journal of library administration 46(2), 119-132. doi:10.1300/j111v46n02_09 johnson, r.k. & luther, j. (2007). the e-only tipping point for journals: what’s ahead in the print-to-electronic transition zone. arl report. retrieved http://dx.doi.org.login.ezproxy.library.ualberta.ca/10.1016/s0098-7913(96)90006-4� http://dx.doi.org/10.1016/j.lcats.2005.04.004� evidence based library and information practice 2011, 6.3 52 25 aug. 2011 from http://www.arl.org/bm~doc/electronic _transition.pdf kalyan, s. (2002). non-renewal of print journal subscriptions that duplicate titles in selected electronic databases: a case study. library collections, acquisitions, & technical services, 26(4), 409-421. doi:10.1016/s1464-9055(02)00287-7 kownacki, b. (1991). electronic information: the view from reference, in electronic journals: considerations for the present and the future, mcmilllan, g., guest editor. serials review, 22(4), 77 86. doi:10.1016/0098-7913(91)90032-e maple, a., wright, c. & seeds, r. (2003). analysis of format duplication in an academic library collection. library collections, acquisitions, & technical services, 27(4), 425-442. doi:10.1016/j.lcats.2003.09.003 moghaddam, g.g. (2007). archiving challenges of scholarly electronic journals: how do publishers manage them? serials review, 33(2), 81-90. doi:10.1016/j.serrev.2007.03.003 montgomery, c.h. & sparks, j.l. (2000). the transition to an electronic journal collection: managing the organizational changes. serials review, 26(3), 4-18. doi:10.1016/s00987913(00)00073-3 prabha, c. (2007). shifting from print to electronic journals in arl university libraries. serials review, 33(1), 4-13. doi:10.1016/j.serrev.2006.12.001 rosati, k.t. (2005). the decline of print: ten years of print serial use in a small academic medical library. the acquisitions librarian, 18(35/36), 107117. doi:10.1300/j101v18n35_08 rupp-serrano, k., robbins, s. & cain, d. (2002). canceling print serials in favor of electronic: criteria for decision making. library collections, acquisitions, & technical services, 26(4), 369-378. doi:10.1016/s14649055(02)00274-9 spencer, j.s. & millson-martula, c. (2006). serials cancellations in college and small university libraries. the serials librarian, 49(4), 135-155. doi:10.1300/j123v49n04_10 sprague, n. & chambers, m.b. (2000). full-text databases and the journal cancellation process: a case study. serials review 26(3), 19-31. doi:10.1016/s00987913(00)00074-5 thohira, m., chambers, m.b. & sprague, n. (2010). full-text databases: a case study revisited a decade later. serials review, 36(3), 152-160. doi:10.1016/j.serrev.2010.07.001 weston, b. & acton, d. (2009). managing divergence of print and electronic journals. the serials librarian 56(1), 181-198. doi:10.1080/03615260802679697 http://dx.doi.org/10.1016/s1464-9055(02)00287-7� http://dx.doi.org.login.ezproxy.library.ualberta.ca/10.1016/0098-7913(91)90032-e� http://dx.doi.org/10.1016/j.lcats.2003.09.003� http://dx.doi.org/10.1016/j.serrev.2007.03.003� http://dx.doi.org/10.1016/s0098-7913(00)00073-3� http://dx.doi.org/10.1016/s0098-7913(00)00073-3� http://dx.doi.org/10.1016/j.serrev.2006.12.001� http://dx.doi.org.login.ezproxy.library.ualberta.ca/10.1016/s1464-9055(02)00274-9� http://dx.doi.org.login.ezproxy.library.ualberta.ca/10.1016/s1464-9055(02)00274-9� http://dx.doi.org/10.1016/s0098-7913(00)00074-5� http://dx.doi.org/10.1016/s0098-7913(00)00074-5� http://dx.doi.org/10.1016/j.serrev.2010.07.001� / evidence based library and information practice/ / evidence based library and information practice 2012, 7.4 25 evidence based library and information practice article collection usage preand post-summon implementation at the university of manitoba libraries lisa o’hara head, discovery and delivery services university of manitoba libraries winnipeg, manitoba, canada email: lisa_ohara@umanitoba.ca received: 24 oct. 2011 accepted: 11 nov. 2012 2012 o’hara. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objectives – this study examines the use of print and electronic collections both before and after implementation of summon at the university of manitoba libraries. summon is a web-scale discovery service which allows discovery of all of the materials the library owns or has access to from a simple search box on the library’s web page. methods – counter statistics were used to determine database, e-journal, and ebook statistics, including database search statistics (dr1) from the counter database report 1, full-text article downloads from the counter journal report 1 (jr1), and successful section search requests from the counter book report 2 (br2) for electronic resources. sirsi, the university of manitoba’s integrated library system, provided statistics on checkouts for the libraries’ circulating print monograph and serial collections. the percentage change from the pre-summon implementation period to the post-summon implementation period was calculated and these numbers were used to determine whether usage had increased or decreased for both print and electronic collections. results – as expected, searches in citation databases decreased because searches were no longer being carried out in the native database as the metadata from the database is included in summon. e-journal usage increased dramatically and e-book usage also increased for four of six providers examined. print usage decreased, but the results were inconclusive. mailto:lisa_ohara@umanitoba.ca evidence based library and information practice 2012, 7.4 26 conclusions – summon implementation had a favourable impact on collection usage. introduction as illustrated by research dating back to as recently as 2010 or as far back as the 1990s (if not earlier), library discovery systems within the networked online environment have evolved, yet continue to struggle to serve users. as a result, the library (or systems supported and maintained by the library) is often not the first stop for research – or worse, not a stop at all. users have defected, and research continues to illustrate this fact. (vaughan, 2011, p. 7). for many years, the university of manitoba libraries (uml), like many academic libraries, have offered a wide variety of tools to search for information. books, audiovisual (av) materials, and journals were discoverable from the uml’s catalogue, but articles, statistics, and other materials were discoverable only through specific databases, requiring specialized instruction for users in selecting appropriate resources to search. an external website review recommended that the uml implement a single search box as a starting place for all searches for library materials. at the time, the only service on the market was the summon web-scale discovery service, which offered a single search box searching the majority of the uml’s databases and catalogue records. summon was presented to the libraries’ staff in 2009, acquired late that year, and implemented in may 2010. because summon was a new service to uml and to libraries in general, uml staff were interested in monitoring the effectiveness of summon, and one way to do so was to compare collection usage preand post-summon implementation. it was expected that clients would find more materials that were relevant to their searches, thereby increasing the use of the uml collections. summon was implemented at the uml as anout-of-the-box service. at the time, summon allowed very little customization of order and placement of facets, fonts, displayed information, and other aspects of the interface. summon in different libraries would differ in only in the branding and the content searched, dependent on activation in the summon knowledgebase of the individual libraries’ available content. implementation therefore required little decision-making, and the main work for uml’s staff involved making sure that the data from the uml’s catalogue was appearing correctly and linking back to the catalogue, and activating the electronic resources in the summon knowledgebase. the university of manitoba is a doctoral-level university serving over 25,000 students, and the uml’s collections number over 1.8 million titles in 19 libraries, including 8 hospital libraries. administration and technical services are centralized for the 19 libraries, and the uml uses sirsi symphony version 3.3.1, but switched from the web2 interface to elibrary in summer 2011. the uml also uses sfx as its openurl resolver and has implemented verde as its electronic resources management system. access to over 300 separate databases in a variety of subjects is provided by the uml in support of the university’s programs. these databases were made available using a drupal product, although migration to libguides took place in summer 2011. the uml was the second library in canada to adopt summon as its resource discovery tool and made it available on the uml’s homepage as a single search option called “one stop search.” literature review web-scale discovery tools first came onto the market in 2008; articles and books on the topic are now beginning to appear in the literature. there are many articles discussing the role, advantages, and disadvantages of web-scale discovery services, such as hoy’s “an introduction to web scale discovery systems” (2012) and hoeppner’s “the ins and outs of evaluating web-scale discovery services” evidence based library and information practice 2012, 7.4 27 (2012). jason vaughan also published a number of articles on the topic and, in 2011, wrote an ala technology report which discusses web-scale discovery, documents available systems, examines their differences, and suggests questions librarians should ask before acquiring a web-scale discovery system (2011). there have been other reviews of various systems, most notably ronda rowe’s (2010) review of summon, ebsco discovery service, and worldcat local in the charleston advisor, which identifies some areas to consider when selecting a tool. implementation and decisions surrounding system configurations have been discussed in a number of articles and book chapters. nara l. newcomer (2011) and anita breckbill (2012) look at selecting, configuring, and searching web-scale systems for music-related information resources. more generally, implementation decisions for various systems are discussed in the “implementation” section of planning and implementing resource discovery tools in academic libraries, edited by mary pagliero popp and diane dallis (2012). in particular, there are discussions on summon implementation in a consortial environment (christel, koehler, & upfold, 2012), in two british academic libraries (thoburn, coates, & stone, 2012), and at montana state university (babbitt, foster, & rossmann, 2012). search satisfaction and usability testing also receive a good deal of attention in the literature. julia gross and lutie sheridan (2011) examine the user experience with summon, finding that students react favourably to the simplified search, using it even when a different search tool might give them better results. this was similar to a finding at the university of manitoba libraries where the authors determined that while all participants searching summon found relevant materials compared to only 60% of those searching pre-summon, some searches would have retrieved better search results had they been done with more refined tools (o’hara, nicholls, & keiller, 2012). change in collection usage as a result of the implementation of web-scale discovery tools is not as well covered as other topics. doug way (2010) of grand valley state university libraries examined usage statistics over a three-month period post-summon implementation, to find the impact of summon on collection usage at his institution. he determined that there was a dramatic increase in the use of full-text resources and a dramatic decrease in the use of abstracting and indexing databases. jan kemp (2012) at the university of texas at san antonio also examined changes in collection usage post-summon implementation, finding that full-text article downloads increased by 23%. tonia graves (2012), discussing the effects of a discovery layer on usage a year after implementation, reports that “usage reports show a dramatic increase in patron usage of worldcat local. detailed reports from worldcat local indicate that the lowest amount of usage in a single month for the year of 2011 was higher than the highest usage month in 2010” (p. 173). methods to examine changes in collection usage at uml, usage statistics were examined for the year prior to summon implementation (may 2009 to april 2010 inclusive) and the two years after summon implementation (may 2010 to april 2011 and may 2011 to april 2012 inclusive). as way (2010) did, to ensure that measurements for electronic resources were comparable, counter statistics were used in the uml study to find database, e-journal, and e-book statistics. these included database search statistics (dr1) from the counter database report 1 to measure database usage; full-text article downloads from the counter journal report 1 (jr1) to measure electronic journal usage; and successful section search requests from the counter book report 2 (br2) to measure e-book usage. because electronic collections are dynamic and change is a constant, only platforms and databases that were licensed by uml over the entire period of the study were included. as well, only publishers, databases, and providers that reported counter statistics for the full period of the study were included. the majority of uml’s publishers and providers supply counter statistics, covering an evidence based library and information practice 2012, 7.4 28 estimated 95% of the uml’s e-journal collections, less for other electronic collections. for the more static print resources, sirsi statistics were used to determine changes in usage in the preand post-summon implementation periods, just as kemp (2012) used opac circulation statistics for the same purpose in his study. other factors such as the increase or decrease in size of collections, changes in collection policies, and recommendations for the use of specific materials or collections by faculty were not considered except peripherally in the discussion. results print resources print resource circulation increased for monographs fairly steadily until summon implementation, when they began to decline slightly. usage for print journals had been declining steadily, but circulations actually increased by 3% over the previous year in the final year examined. one factor affecting this increase might be the addition of a print journal target to the sfx (openurl resolver) menu at uml during this period. adding the target allowed patrons searching for an article in a citation database or in summon to immediately determine if the uml held the required journal in print. prior to this, patrons had to perform a separate search in the catalogue to find whether the journal was held in print after they identified required journal articles. database use traditional abstracting and indexing databases saw a fairly uniform decrease in usage postsummon, according to the usage statistics reported in the table below. most databases for which usage grew significantly post-summon implementation figure 1 checkouts for print resources 2007-2012. a a note that implementation took place in may 2010; statistics prior to that are pre-summon implementation and are given to provide some context for general trends in print circulations at uml. evidence based library and information practice 2012, 7.4 29 table 1 percent change in database searches by database for the period may 2009-april 2010 (y0, presummon) and may 2010-april 2012 (y1 & y2, post-summon) database name %change y0-y1 %change y0-y2 average change health and safety science abstracts -84.22% -95.29% -89.81% csa social services abstracts -25.84% -84.09% -54.96% conference papers index -26.37% -79.42% -52.90% environmental sciences and pollution abstracts -9.68% 89.09% 49.39% biotechnology research abstracts -20.66% -77.62% -49.14% bacteriology abstracts (microbiology b) -14.97% -74.57% -44.77% csa linguistics and language behaviour abstracts 2.4% -82.37% -39.99% cos scholar universe: social science 5.56% -80.65% -37.54% georef 26.64% -91.57% -32.47% csa worldwide political science abstracts -16.49% -47.44% -31.97% social work abstracts -9.57% -49.57% -29.57% child development & adolescent studies -10.58% -45.41% -27.99% bibliography of native north americans -8.83% -44.07% -26.45% america: history & life -7.38% -42.71% -25.04% anthropology plus -7.44% -41.09% -24.26% historical abstracts -6.99% -40.79% -23.89% atla religion database with serials -5.27% -41.84% -23.56% peace research abstracts -4.15% -39.70% -21.39% rilm abstracts of music literature -3.44% -38.66% -21.05% psycinfo -33% -4.74% -18.87% cab abstracts -3.36% -32.89% -18.13% sportdiscus 1.15% -35.50% -17.17% cinahl -9.36% -13.82% -11.59% food science and technology abstracts 9.85% -19.75% -4.95% geobase -2.45% -0.39% -1.42% compendex 8.63% 0.11% 4.37% avery index to architectural periodicals -14.42% 170.52% 78.05% international pharmaceutical abstracts 106.70% 91.83% 99.26% index islamicus -10.52% 284.74% 137.11% risk abstracts -84.37% 606.72% 261.18% artbibliographies modern -19.64% 664.15% 322.26% metadex -31.79% 1186.49% 577.35% industrial and applied microbiology abstracts -20.45% 2474.9% 1227.22% toxline -19.08% 2561.01% 1270.96% toxicology abstracts -30.78% 3050.9% 1510.06% ecology abstracts -18.82% 3578.33% 1779.75% water resource abstracts -18.93% 3581.54% 1781.30% environmental engineering abstracts -24.67% 8462.44% 4218.89% average -11.65% 668.96% 328.66% evidence based library and information practice 2012, 7.4 30 were migrated to the new proquest platform in 2012, leading staff to suspect that the data were faulty. proquest staff confirmed that the issue had also been reported by other libraries. therefore, the numbers in the table above for “% change y0-y2” cannot be considered accurate for the proquest databases (shaded in grey) and should be discounted; however, they were included in the study because they constitute a significant portion of uml’s citation databases. when the second-year postsummon implementation data for the proquest databases are removed, the average change from the year before summon implementation to the second year postsummon implementation is -11.68%, almost exactly the -11.65% change experienced in the first year post-summon implementation electronic journal use the majority of platforms saw a significant increase in successful journal requests, with an average increase of almost 19% in the first year post-summon and an increase of 43% from the year before summon was implemented to the second year after implementation. electronic book use the uml provide access to all books and ebooks through the library catalogue; a bibliographic record for each e-book is loaded to summon, with new acquisitions being loaded weekly. this means that every e-book is available through summon via the catalogue whether the metadata is available in summon’s knowledgebase or not. average usage of e-book platforms increased post-summon implementation but usage of two platforms dropped overall. discussion the university of manitoba’s examination of usage statistics did not reveal unexpected results and was typical of the results found in the studies carried out by way (2010) and kemp (2012). one area not examined in previous studies was usage of print collections and the decrease in checkouts for print collections at uml was not unexpected. however, there are other factors that might also affect this decrease, including the figure 2 average e-books usage for the period may 2009-april 2010 (y0, pre-summon) and may 2010april 2012 (y1 & y2 post-summon) evidence based library and information practice 2012, 7.4 31 table 2 percent change in successful journal requests by platform for the period may 2009-april 2010 (y0, pre-summon) and may 2010-april 2012 (y1 & y2, post-summon) platform %change y0-y1 %change y0-y2 average change scitation -59.85% -52.46% -56.16% poj -63.27% -42.86% -53.06% newsbank -42.37% -59.27% -50.82% aps -40.18% -47.98% -44.08% acs publications -36.03% -34.7% -35.37% iimp -34.18% -34.32% -34.25% project euclid -25.83% -21.32% -23.57% aps journals -10.14% -32.69% -21.42% mit press journals 14.68% -46.63% -15.98% proquest 5.51% -29.5% -12% acm digital library 10.62% -32.71% -11.04% jstor -15.74% 1.26% -7.24% swetswise -7.21% 4.57% -1.32% highwire press 0.19% 1.43% 0.81% bioone 8.04% -0.21% 3.92% highwire 3.73% 11.22% 7.47% ebscohost 15.55% 11.44% 13.5% content.karger.com 32.13% -5.08% 13.52% biomed central 20.61% 31.34% 25.98% sciencedirect 19.21% 41.6% 30.4% nature.com 14.5% 50.02% 32.26% wiley 34.48% 50.1% 42.29% annual reviews 44.1% 41.6% 42.85% ieee explore 24.47% 67.61% 46.04% project muse 22.79% 71.58% 47.19% metapress 11.32% 87.67% 49.49% rsc.org 31.61% 75.61% 53.61% jnspgonline 10.63% 97.11% 53.87% periodicals archive online 49.96% 66.08% 58.02% ingentaconnect 33.7% 83.18% 58.44% cjo 41.77% 82.46% 62.11% gold 52.05% 75.8% 63.91% internurse.com 124.15% 123.1% 123.13% palgrave-journals.com 136.06% 126.6% 131.33% thieme journals 15.02% 295.6% 155.31% mla journals 95% 232.5% 163.75% cairn 170.48% 303.81% 237.14% average 19.1% 43.07% 31.08% evidence based library and information practice 2012, 7.4 32 purchase of large numbers of backfiles, weeding of the print collection, the movement of materials to off-site storage, and a move to e-preferred purchasing. these factors make it impossible to determine if the decrease in usage is due to summon or to the other factors and might be worth further study. wj although way (2010) described the drop in searches for core subject databases as troubling, it is explained by the fact that usage is no longer tracked in the native interface once citations are included in summon. the almost uniform decrease in usage of citation databases cannot be accurately measured until it is possible to track usage by database within summon itself. at present, the source of citations returned from searches cannot be tracked or measured, so it is impossible to determine whether citation databases within summon are used or useful. the fact that the decrease in usage statistics averaged -11% for both years post-summon implementation demonstrates that a significant amount of research is still being conducted in these native interfaces and that they are still a necessary acquisition for academic libraries. for electronic journals, the uml case study shows similar results to those found in the case study way (2010) conducted at grand valley state university libraries. he found that usage of full-text ejournals increased “regardless of whether the content provider had directly partnered with serials solutions to make their content available in summon” (p. 219). similarly, uml experienced increases in usage for content not in summon, as exemplified by the 13.5% average increase in successful journal requests in the ebscohost platform. the uml results of an average increase of 19% in successful searches in the first year postsummon implementation are very close to the results seen at the university of texas at san antonio, where full-text downloads increased 23% in the same one-year period (2012). e-book usage also increased on average for the platforms surveyed. because the uml are in the process of moving from a mainly printbased monograph collection to an electronic monograph collection, it is more difficult to state confidently that the average increase was due to summon implementation. many other factors – including fluctuations in the size of the e-book collections, availability of print books, implementation of demand-driven acquisitions, not to mention the relevancy ranking algorithm used by summon itself – could affect the e-book usage statistics positively and negatively. it is interesting that the two platforms that saw decreases in usage, blackwell online reference and royal society of chemistry, have somewhat similar content to credo reference online and springer, which both saw increases in usage. this is certainly an area where further research could be done. conclusion although the study carried out at the university of manitoba libraries examined only those platforms where counter statistics were supplied, a number of conclusions can be drawn. summon implementation at uml has resulted in an increase in the use of the uml’s electronic fulltext collections. in fact, the similarity in results among the uml study, way’s study (2010), and kemp’s study (2012) for e-journals suggest that implementation of a web-scale discovery system will increase usage of full-text ejournals in academic libraries. although neither way nor kemp examined e-book usage, the uml study indicates that the same is true for full-text e-books, and so further study is needed in this area. the fact that citation database usage decreased with the implementation of a discovery layer was not a surprise, since the searches are being conducted within the discovery layer rather than in the native interface. in fact, it can be argued that this decrease points to client satisfaction with summon searching, assuming that summon is being used by some segment of the uml user population in place of the native interfaces. however, because there is still significant use of the citation databases recorded, they are still a necessary acquisition. although the decrease in print circulations is troubling, it is not possible to determine evidence based library and information practice 2012, 7.4 33 whether it is the result of the implementation of summon or other factors, and it will be worthwhile to monitor these circulations in the future. it will also be interesting to observe how resource discovery tools will continue to affect collection usage as vendors and publishers provide better metadata, and advances such as semantic web technology increase search effectiveness. references babbitt, e. p., foster, a, & rossmann, d. (2012). implementation of resource discovery: lessons learned. in m. p. popp & d. dallis (eds.). planning and implementing resource discovery tools in academic libraries (pp. 598-607). hershey, pa: igi global. breckbill, a. (2012). the trial of searching for musical works using resource discovery tools. in m. p. popp & d. dallis (eds.). planning and implementing resource discovery tools in academic libraries (pp. 662-676). hershey, pa: igi global. christel, m., koehler, j., & upfold, m. (2012). implementing a discovery layer in a consortial environment. in m. p. popp & d. dallis (eds.). planning and implementing resource discovery tools in academic libraries (pp. 407-418). hershey, pa: igi global. graves, t., & dresselhaus, a. (2012). one academic library – one year of webscale discovery. the serials librarian, 62(1-4), 169-175. doi: 10.1080/0361526x.2012.652915 gross, j., & sheridan, l. (2011). web scale discovery: the user experience. new library world, 112(5/6), 236-247. retrieved 22 oct. 2012 from: http://search.proquest.com/ docview/868247945 hoy, m. b. (2012). an introduction to web scale discovery systems. medical reference services quarterly, 31(3), 323329. doi: 10.1080/02763869.2012.698186 hoeppner, a. (2012). the ins and outs of evaluating web-scale discovery services. computers in libraries, 32(3), 610. retrieved 22 oct. 2012 from http://search.proquest.com/docview/10 08145593?accountid=14569 kemp, j. (2012). does web-scale discovery make a difference? changes in collections use after implementing summon. in m. p. popp & d. dallis (eds.). planning and implementing resource discovery tools in academic libraries (pp. 456-468). hershey, pa: igi global. newcomer, n. l. (2011). the detail behind web-scale: selecting and configuring web-scale discovery tools to meet music information retrieval needs. music reference services quarterly, 14(3), 131-145. doi: 10.1080/10588167.2011.596098 o’hara, l., nicholls, p., & keiller, k. (2012). search success at the university of manitoba libraries preand postsummon implementation. in m. p. popp & d. dallis (eds.). planning and implementing resource discovery tools in academic libraries (pp. 268-287). hershey, pa: igi global. popp, m. p., & dallis, d. (eds.) (2012). planning and implementing resource discovery tools in academic libraries. hershey, pa: igi global. rowe, r. (2010). web-scale discovery: a review of summon, ebsco discovery service, and worldcat local. charleston advisor, 12(1), 5-10. doi: 10.5260/chara.12.1.5 thoburn, j., coates, a., & stone, g. simplifying resource discovery and access in academic libraries: implementing and evaluating summon at huddersfield and http://www.igi-global.com/affiliate/mary-pagliero-popp/228567/ http://www.igi-global.com/affiliate/mary-pagliero-popp/228567/ http://www.igi-global.com/affiliate/diane-dallis/228568/ http://www.igi-global.com/affiliate/mary-pagliero-popp/228567/ http://www.igi-global.com/affiliate/diane-dallis/228568/ http://www.igi-global.com/affiliate/diane-dallis/228568/ http://www.igi-global.com/affiliate/mary-pagliero-popp/228567/ http://www.igi-global.com/affiliate/diane-dallis/228568/ http://www.igi-global.com/affiliate/mary-pagliero-popp/228567/ http://www.igi-global.com/affiliate/diane-dallis/228568/ http://www.igi-global.com/affiliate/mary-pagliero-popp/228567/ http://www.igi-global.com/affiliate/mary-pagliero-popp/228567/ http://www.igi-global.com/affiliate/diane-dallis/228568/ evidence based library and information practice 2012, 7.4 34 northumbria universities. in m. p. popp & d. dallis (eds.). planning and implementing resource discovery tools in academic libraries (pp. 580-597). hershey, pa: igi global. vaughan, j. (2011). web scale discovery services. chicago, il: ala editions. retrieved 22 oct. 2012 from http://site.ebrary.com /docdetail.action?docid=10469319 way, d. (2010). the impact of web-scale discovery on the use of a library collection. serials review, 36(4) 214-220. doi: 10.1016/j.serrev.2010.07.002 http://www.igi-global.com/affiliate/mary-pagliero-popp/228567/ http://www.igi-global.com/affiliate/mary-pagliero-popp/228567/ http://www.igi-global.com/affiliate/diane-dallis/228568/   commentary   the state of systematic reviews in library and information studies   denise koufogiannakis collections and acquisitions coordinator university of alberta libraries edmonton, alberta, canada email: denise.koufogiannakis@ualberta.ca   received: 15 apr. 2012                                                                 accepted: 6 may 2012      2012 koufogiannakis. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     introduction   although the first article outlining what was then referred to as evidence based librarianship (ebl) was published in 1997 (eldredge), it wasn’t until 2000 that significant articles outlining how evidence based practice could work in library and information studies (lis) were published (eldredge, 2000a, 2000b, 2000c; booth, 2000). jonathan eldredge published several keystone articles that year, and in one of those he focused on the challenges of finding the needed evidence to practice in an evidence based manner (2000c). pondering the types of research evidence available in the lis literature, specifically health sciences librarianship, eldredge noted that: “systematic reviews for ebl levels 1-2 … are not known to currently exist in the health sciences librarianship literature” (p. 8). since that time, the evidence based library and information practice (eblip) movement has encouraged the development of publications that synthesize or appraise existing research, such as the evidence summaries published in this journal, so that practitioners do not have to do all the work themselves each and every time they encounter a problem or question. systematic reviews have become an important source of information because they both synthesize the existing research on a topic, as well as critically appraise it and try to draw conclusions from the total body of quality research evidence.   grant and booth (2009) define a systematic review as a type of review that “seeks to systematically search for, appraise and synthesis research evidence, often adhering to guidelines on the conduct of a review” (p. 95). booth and brice (2004) point out that a systematic review helps us “keep up-to-date, define the boundaries of what is known and what is not known and can help us avoid knowing less than has been proven” (p. 111). however, urquhart (2010) points out the complexity of applying such methodology to research in lis due to the diversity of the research methods in the lis knowledge base, as well as the different standpoints taken by the researchers. systematic reviews in medicine have mainly used quantitative studies; however the field of library and information studies also contains a lot of qualitative research and different types of social sciences methodologies. brettle (2009) notes that while most systematic reviews in health care use controlled studies, “it is appropriate for systematic reviews in the library domain to take a wide view of relevant evidence and include a variety of designs appropriate to the topic or review question at hand” (p. 45). it is not necessarily as “easy” to work with the various types of lis research data within the context of a systematic review. however, more recently, there has been work done on qualitative synthesis within systematic reviews, and greater attempts to incorporate different types of research into such reviews (dixon-woods et al., 2006; barnett-page & thomas, 2009).   the creation of a wiki to track systematic reviews   prompted by a desire to determine the growth of systematic reviews since the evidence based practice movement in our field began, i created a wiki (http://lis-systematic-reviews.wikispaces.com) in january 2012 to gather all known systematic reviews in library and information studies. having found no similar source or good way to locate systematic reviews, the wiki was created and those with a known interest in the topic were invited to be collaborators. databases such as library and information science abstracts (lisa), library literature, and google scholar were searched; any articles known to contain citations to systematic reviews in lis (ankem, 2008; brettle, 2009; urquhart, 2010; eldredge, 2004; koufogiannakis & crumley, 2006) were reviewed, and input from colleagues was solicited once a preliminary version of the wiki was produced.   upon the suggestion of andrew booth, a contributor to the wiki, it was decided that systematic reviews should meet one of the following criteria to be included on the wiki site: published in a library journal authored by an lis lead author affiliated with a library or information unit, or academic library or information department include studies conducted in a library setting include at least 25% of included studies from library journals   description of the systematic reviews included to date   as of this writing, there are 37 lis systematic reviews cited on the wiki. these systematic reviews were published between 1997 and 2012. topics cover a wide range, but the vast majority of systematic reviews fall into the health sciences librarianship field (24). others pertain to academic libraries (5), and a number do not focus on a specific type of library (8).   it is no surprise that so many systematic reviews are published on health librarianship topics: eblip grew out of the evidence based medicine (ebm) movement; health sciences librarianship was the first to embrace and apply evidence based principles; and librarians in health sciences librarianship would have been familiar with the systematic review methodology, as many participated in research teams working on systematic reviews in health care.   journals that have published systematic reviews in lis are noted in table 1. health information and libraries journal (hilj) impressively stands out as the one journal that has published a significant number of systematic reviews dating back to 2003, and at least one per year since 2007. hilj has made a concerted effort for the past number of years to attract and publish review articles, and identify what type of a review article it is. hence, systematic reviews published in hilj are quite easy to identify as such.   table 1 journals that have published systematic reviews on lis topics journal title number health libraries and information journal 13 journal of the medical library association /bulletin of the medical library association 4 information research 2 journal of academic librarianship 2 reference services review 2 other journals that have published one systematic review each 14   table 2 number of systematic reviews in lis by domain lis domain number reference 15 education 8 professional issues 8 information access and retrieval 6 collections 1 management 1       when categorized by lis domain (koufogiannakis, slater, & crumley, 2004), many systematic reviews were found to be in the areas of reference, education, and professional issues. the area of information needs research was included in the reference category, because it was seen as knowledge to support the reference needs of particular groups. systematic reviews relating to information needs comprised about half of all the systematic reviews in the reference category. also of note, in the professional issues category are three systematic reviews assessing the literature on clinical librarianship programs, and one assessing the literature on informationist programs, together comprising half of the systematic reviews in this category.   the number of systematic reviews published in lis each year seems to slowly be growing; however the numbers are still small. of the 37 systematic reviews that have been identified, only 3 were published prior to 2003. of those three, one was published in 1996, and two in 1997. there was then a five year period between 1998 and 2002 when it seems that no systematic reviews were published. since 2003, there have been at least two systematic reviews published every year, with a high of six published in 2010. these numbers show that while there were a few systematic reviews prior to the year 2000 when evidence based practice in lis began, growth has really occurred since systematic reviews have been promoted as a useful form of research to assist with evidence based practice.   conclusion   the newly created lis systematic reviews wiki is a starting point for lis researchers and practitioners who are looking to see what research summaries on their topics of interest already exist. where systematic reviews on a topic already exist, the review will save the practitioner or researcher much time in providing an overview of research on that topic up to the point in time in which it was published. since systematic reviews are quite detailed, references to the original research studies will also prove valuable.   yet, the number of systematic reviews in lis is small, and so a review that is on topic may be elusive. the current small number of systematic reviews provides research and publishing opportunities for librarians. for example, only one systematic review was found in each of the categories of collections and management. the research produced in these areas make them ripe for possible systematic reviews.   eblip is beginning a new reviews section within the journal, hoping to provide more of this type of literature for practitioners, since the editorial team believes it is a valuable form of research in its own right, and has the potential to be very useful. reviews do not have to be systematic reviews to be included in the reviews section, but given the current numbers of systematic reviews being published, there is certainly room for more. whether it be in eblip or another journal, i encourage librarians to consider producing a systematic review. the wiki site will be continually updated as new reviews are brought to the author’s attention, and will hopefully remain a useful gathering place for such content. all suggestions are welcome!   acknowledgements   the author would like to thank the following individuals who have contributed content or ideas toward the creation of the lis systematic reviews wiki: andrew booth, lorie kloda, alison brettle, and christine urquhart.     references   ankem, k. (2008). evaluation of method in systematic reviews and meta-analyses published in lis. library and information research, 32(101), 91-104. retrieved 15 may 2012 from http://www.lirgjournal.org.uk/lir/ojs/index.php/lir/article/view/58/118   barnett-page, e., & thomas, j. (2009). methods for the synthesis of qualitative research: a critical review. bmc medical research methodology, 9(59). doi:10.1186/1471-2288-9-59   booth, a. (2000, july 2-5). librarian heal thyself: evidence based librarianship, useful, practical, desirable? 8th international congress on medical librarianship. london, uk.   booth, a., & brice, a. (2004). appraising the evidence. in a. booth & a. brice (eds.). evidence-based practice for information professionals: a handbook. (pp. 104-118). london: facet.   brettle, a. (2009). systematic reviews and evidence based library and information practice. evidence based library and information practice, 4(1), 43-50. retrieved 15 may 2012 from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/5082/5082   dixon-woods, m., bonas, s., booth, a., jones, d. r., miller, t., sutton, a. j., shaw, r. l., smith, j. a., & young, b. (2006). how can systematic reviews incorporate qualitative research? a critical perspective. qualitative research, 6(1), 27-44. doi:10.1177/1468794106058867   eldredge, j. (1997). evidence-based librarianship: a commentary for hypothesis. hypothesis, 11(3), 4-7. retrieved 15 may 2012 from http://research.mlanet.org/hypothesis/hypo11-3.pdf   eldredge, j. d. (2000a). evidence-based librarianship: an overview. bulletin of the medical library association 88(4), 289-302. retrieved 15 may 2012 from http://www.ncbi.nlm.nih.gov/pmc/articles/pmc35250/   eldredge, j. d. (2000b). evidence-based librarianship: formulating ebl questions. bibliotheca medica canadiana 22(2), 74-77.   eldredge, j. d. (2000c). evidence-based librarianship: searching for the needed ebl evidence. medical reference services quarterly, 19(3), 1-18. doi:10.1300/j115v19n03_01   eldredge, j. d. (2004). how good is the evidence base? in a. booth & a. brice (eds.). evidence-based practice for information professionals: a handbook. (pp. 36-48). london: facet.   grant, m., & booth, a. (2009). a typology of reviews: an analysis of 14 review types of associated methodologies. health information and libraries journal, 26(2), 91-108.   koufogiannakis, d., & crumley, e. (2006). research in librarianship: issues to consider. library hi tech, 24(3), 324-340. doi:10.1108/07378830610692109   koufogiannakis, d., slater, l., & crumley, e. (2004). a content analysis of librarianship research. journal of information science, 30(3), 227-239. doi:10.1177/0165551504044668   urquhart, c. (2010). systematic reviewing, meta-analysis and meta-synthesis for evidence-based library and information science. information research, 15(3), colis708. retrieved 3 mar. 2012 from http://informationr.net/ir/15-3/colis7/colis708.html     microsoft word news_call.doc evidence based library and information practice 2006, 1:3  101 evidence based library and information practice     news      call for submissions ‐ 4th international conference on evidence based library and  information practice         © 2006 eblip. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    4th international conference on evidence  based library and information practice   chapel hill, nc, usa may 6‐11, 2007    call for submissions    the evidence based library and  information practice conference (eblip4) is  an exciting international event that has  emerged in response to the growing interest  among all types of libraries in using the best  available evidence to improve information  practice. the conference on may 6‐9, 2007 in  chapel hill, north carolina will be followed  by two days of continuing education. the  conference provides a forum for the  presentation of high quality papers and  posters as well as examples of how eblip is  being implemented in library and  information settings around the globe.    eblip4 invites submissions for contributed  papers and posters including both original  research and innovative applications of  eblip in library and information  management. papers that deal with library  support of evidence‐based practice in other  fields such as health, social work and public  policy are also welcome. full instructions to  authors may be found at:  www.eblip4.unc.edu    important dates    december 1, 2006: submission deadline for  abstracts     february 11, 2007: final decisions for  accepted papers     february 15, 2007: final decisions for  accepted posters     march 15, 2007: submission deadline for full  papers                  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2012, 7.3 73 evidence based library and information practice evidence summary enhanced catalogue records positively impact circulation but are not used to their potential in patron searching a review of: tosaka, y., & weng, c. (2011). reexamining content-enriched access: its effect on usage and discovery. college & research libraries, 72(5), 412-427. reviewed by: cari merkley associate professor mount royal university library calgary, alberta, canada email: cmerkley@mtroyal.ca received: 1 mar. 2012 accepted: 16 june 2012 2012 merkley. this is an open access article distributed under the terms of the creative commons‐ attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to determine how contentenriched catalogue records impact the circulation rates of print resources in four subject areas, and to investigate how this additional metadata influences opac searching and item retrieval. design – analysis of circulation data, bibliographic records, and opac search logs. setting – a library at a four-year undergraduate residential college in the north-eastern united states. subjects – bibliographic records for 88,538 titles; data from 7,782 circulation transactions; and 130 opac search strings and related circulation data. methods – in the first part of the study, bibliographic records for print items published since 1990 were extracted from the library’s integrated library system (ils) in the following library of congress (lc) classes: d, e, f, h, j, l, p, q, r, s, and t. it is assumed that electronic books were excluded from this study because their usage is not tracked in the ils. these lc classes were chosen to correspond to the subject areas targeted by the researchers for comparison – “history, social sciences, language and literature, and science and technology” (p. 416). the data file included the publication date of the title, as well as values for the marc fields identified mailto:cmerkley@mtroyal.ca evidence based library and information practice 2012, 7.3 74 by the researchers as containing contentenriched data. these fields were marc 505 (an item’s table of contents or list of works included), marc 520 (summaries or annotations), and marc 856 (url to electronic location of related material or electronic copy) (p. 416; library of congress network development and marc standards office, 2003, 2008a, 2008b). the authors analyzed records for 88,538 titles and determined the total number of records containing each of the marc fields either singly or in combination. data relating to circulation transactions for items located in these lc classes from january to may 2009 was also identified. like the bibliographic records, circulation data was pulled for print items only. the researchers identified 7,782 circulation transactions that met the study criteria for the period in question. in the second part of the study, circulation data for september 22, 2009 was obtained and sorted into the four subject categories identified in part i of the study. the authors indicate that this date was chosen at random, but do not specify how. researchers compared the records of the 133 titles borrowed that day from the lc classes studied to the opac search logs from september 16-22, 2009 to determine which searches led to the circulation of these items. the authors felt that searches resulting in checkouts on the day in question may have begun earlier in the week. the searches that led to borrowing were recorded and categorized as keyword, title, author, or other searches. if a user entered a title or author name into the keyword field, these were classed as known item searches in the appropriate categories. the authors identified and analyzed 130 searches relating to circulated items. main results – in the first part of the study, the number of catalogue records that contained marc 505, 520, and/or 856 fields significantly increased for titles published between 1990 and 2007, with a slight decrease in 2008. marc 505 was the most common contentenriched field until 2000, after which the presence of marc 856 grew significantly. the marc 520 field was used least often, making it difficult to draw firm conclusions about its impact on circulation. the incidence of enhanced records was very low among older books in the study. only 14.3% of items published between 1990 and 1994, and 19.3% of items published between 1995 and 1999, had records that contained marc 505, 520, or 856 fields. in contrast, the percentage of enhanced records was very high (80.9%) for items published between 2005 and 2008. the authors acknowledged that these stark imbalances created skewed comparison data for items published in these date ranges. as such, they suggested that the data for titles published between 2000 and 2004 offered the most balanced comparison because the numbers of enhanced and non-enhanced records were almost equal. the overall circulation of items with enhanced records published between 2000 and 2004 was 2.9% higher than for items with non-enhanced records, constituting a relative percentage difference of 30.7%. the relative percentage difference in this period was higher for books in science and technology (36.9%), followed by history (34%), language and literature (30.6%), and social sciences (25.7%). enhanced records also had a positive impact on circulation for items published between 1990 and 2000 over their non-enhanced counterparts; however, this positive growth levelled off for items published between 2005 and 2008, with almost equal circulation rates between items with enhanced and non-enhanced records during this period. the impact of the three marc fields was examined, and the presence of the marc 505 field was most associated with increased circulation rates, in part because it was the most commonly used field of the three for the period in question. the number of records with marc 520 and 856 fields was not sufficient to draw firm conclusions about their impact on circulation. while not the focus of the study, the circulation data also suggested a preference for current titles among all four subject areas, most significantly among the social sciences and science and technology. evidence based library and information practice 2012, 7.3 75 the second part of the study found that keyword searching was the most common strategy employed by patrons, with 49.6% of the 130 searches examined falling into this category. keyword searches most commonly led to the borrowing of items from the history lc classes, while title searches were most common in science and technology. known item searches (title or author) accounted for 45.9% of the overall searches analyzed. however, in most cases, the search terms used that led to a title circulating were found in the title and subject fields, rather than in a contentenriched marc field. the researchers suggested that this may be due to the appearance of search results in the opac (brief rather than full record) and the way relevancy sorting was calculated, as contents notes were not given a high weighting in the opac’s formula. conclusion – the study found that enhanced catalogue records led to higher circulation rates in the four subject areas studied. the increased proportion of content-enriched records in the overall catalogue in recent years suggested that their value had been recognized by the library. the limited role these enhanced fields played in the september 22, 2009 searches suggested that further work on improving how this information is displayed to users in the opac and sorted is needed. the researchers identified areas for future research including the role of the publication date and the impact of improvements to the display of content fields in the opac on the circulation of items with content-enriched records. commentary this work adds to an existing argument in the library literature: that enhanced catalogue records are correlated with increased circulation. their presentation of the study data is measured and thorough, acknowledging the disproportionate impact small sample sizes have on the percentage differences in several of their data sets and their limited ability to draw reasonable conclusions from such numbers. it is useful to have both the absolute and relative percentage differences in circulation to better gauge the difference made by content-enriched records. even better, it is always clear which percentage difference, absolute or relative, is the one being stated by the authors. tosaka and weng do acknowledge, however, that enhanced catalogue records may not be the most important factor in determining circulation rates. publication date played a significant role in the study results, with more recent titles circulating more across all disciplines. teasing out whether these circulation gains are due to the fact that newer records are more likely to be enhanced, or newer materials are just of more interest to users, would be a difficult but worthwhile task. the findings of the second part of the study suggest that enhanced records or date of publication are not the only factors impacting circulation rates: how the patron and library technology interacts with them is perhaps the richer field for study moving forward. the authors write, “to achieve content-enriched access, it is necessary to have a well-designed data-mining mechanism to dig out contentenriched components to system retrieval ability and postsearch evaluation” (p. 413). it was striking how small a role the contentenhanced fields appeared to play in the searches and subsequent circulations on the day studied by the researchers. they proposed explanations for the disconnect between the findings of the first and second parts of the study raise important questions for practitioners. why spend the energy, time, and staff dollars on enhancing catalogue records, to then only display the brief view of the record or sort by publication date in your opac? is this a choice on the part of the library, or a consequence of ils display design? alternatively, if publication date is one of the most important criteria for patrons in selection of material in an opac, is default relevancy ranking a help or an obstacle for users? while the question of to enhance or not to enhance records appears to have been largely answered (as evidenced by the prevalence of enriched records today), it is clear that the work of evidence based library and information practice 2012, 7.3 76 creating a better and more responsive access point to our collections is never done. references library of congress network development and marc standards office. (2003). guidelines for the use of field 856. retrieved february 27, 2012, from http://www.loc.gov/marc /bibliographic/bd856.html library of congress network development and marc standards office. (2008a). 505 formatted contents note (r). retrieved february 27, 2012, from http://www.loc.gov /marc/bibliographic/bd5xx.html library of congress network development and marc standards office. (2008.). 520 summary, etc. (r). retrieved february 27, 2012, from http://www.loc.gov/marc /bibliographic/bd520.html http://www.loc.gov/marc/bibliographic/bd856.html http://www.loc.gov/marc/bibliographic/bd856.html http://www.loc.gov/marc/bibliographic/bd5xx.html http://www.loc.gov/marc/bibliographic/bd5xx.html http://www.loc.gov/marc/bibliographic/bd520.html http://www.loc.gov/marc/bibliographic/bd520.html research article   making job postings more equitable: evidence based recommendations from an analysis of data professionals job postings between 2013-2018   joanna thielen biomedical engineering librarian art, architecture & engineering library university of michigan ann arbor, michigan, united states of america email: jethiele@umich.edu   amy neeser consulting & outreach lead research it university of california berkeley berkeley, california, united states of america email: aneeser@berkeley.edu   received: 1 mar. 2020                                                                accepted: 18 june 2020      2020 thielen and neeser. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   data availability: neeser, a., & thielen, j. (2020). making job postings more equitable: evidence-based recommendations from an analysis of data professionals job postings between 2013-2018 (v2) [dataset]. uc berkeley. https://doi.org/10.6078/d1k419   doi: 10.18438/eblip29674     abstract   objective over the last decade, many academic libraries have hired data professionals to offer research data services. as these positions often require different types of experience than traditional librarian positions, there is an increased interest in hiring professionals from outside the typical library and information science (lis) pipeline. more broadly, there has also been an increased interest in academic libraries and higher education to incorporate the principles and practices of diversity, equity, inclusion, and accessibility (dei&a) into their work. these phenomena allow an opportunity to examine the growing area of data professionals and library hiring practices through the lens of dei&a. data was collected from 180 data professional job positions, including education, experiences, and skills, to better understand the evolving and complex landscape of data professionals and to provide evidence based recommendations regarding how the profession can enact meaningful and lasting change in the areas of dei&a.   methods the qualifications and responsibilities listed in data professional job postings from 2013 to 2018 were examined. prior to analyzing the job postings, a codebook of 43 variables was developed. the 177 data professional job postings (corresponding to 180 positions) were independently analyzed, noting the presence of each variable, including the locations and the degrees of complexity sought. after coding, discrepancies were mutually resolved. overall, the coding process had 94% intercoder agreement, which indicates a high level of agreement.   results over one-third of postings (n = 63, 35%) did not use the word “librarian” in the job title. eighty-eight percent (n = 159) required a master’s in lis degree, but 67% (n = 119) also accepted an equivalent degree. over half of the positions (n = 108, 60%) were also looking for an additional degree, most frequently a graduate degree. the median salary of the positions listing a quantitative value was $57,000; however, this value may not be accurate because only 26% of job positions (n = 47) gave a quantitative salary. from the research data management skills mentioned, general data management (n = 155, 86%), data repositories (n = 122, 68%), and data curation (n = 101, 56%) appeared most frequently. libraries were also looking for traditional lis skills and experiences, including instruction (n = 138, 77%), consultation (n = 121, 67%), and a public services perspective (n = 69, 38%).   conclusion the results show that academic libraries are trying to recruit candidates from outside the traditional academic library pipeline. research data activities (a non-traditional area for lis) and traditional lis areas were both frequently mentioned. overall, these job positions should be written through a more intentional lens of dei&a. this would help to make data professional positions more diverse and inclusive, while also helping academic libraries to reach their goal of recruiting outside of lis. a set of concrete dei&a recommendations are provided that are applicable for writing all library positions, so that readers can put these results into action and enact meaningful change within the profession.     introduction   over the last decade, an increasing number of academic libraries have hired data professionals to offer research data services (rds) to facilitate the advancement of research. data professionals help researchers to “address the full data lifecycle, including the data management plan, digital curation (selection, preservation, maintenance, and archiving), and metadata creation and conversion” (tenopir, sandusky, allard, & birch, 2013, p. 70). these positions often require different types of experience than traditional librarian positions, which can create an interest in hiring professionals from outside of the typical library and information science (lis) pipeline. accepting a variety of academic backgrounds and professional experiences naturally increases other forms of diversity because more types of people will apply. furthermore, there is an increased interest in academic libraries and higher education more broadly to incorporate principles and practices of diversity, equity, inclusion, and accessibility (dei&a) into their work. examining the landscape of data professionals working in academic libraries and formulating recommendations for action can help increase diversity in these positions, reducing disparities within the profession and its institutions. the consequence of perpetuating the status quo is to worsen the disparities amongst underprivileged and underrepresented groups. as hiring managers, search committee members, tenure review committee members, advocates, and conversation starters, everyone has a role to play in making our profession more equitable and inclusive for a more diverse groups of professionals. dei&a is much more than simply having a library or institutional statement at the bottom of a job posting. dei&a principles and practices should inform every aspect of a job posting. this evidence based research study presents the data collected from a deductive thematic analysis of 177 data professional job postings, including education, experiences, and skills, to better understand the complex landscape of data professionals. the findings are used to create a set of recommendations for how dei&a principles can be incorporated into any academic library job posting so that the profession can enact meaningful and lasting change.   literature review   research data services in academic libraries   the need for academic libraries to provide rds due to the emergence of more data intensive research, data management mandates from funding agencies, and other factors, has been well-established in the literature (tenopir et al., 2013). further, rds is listed as a top trend in academic libraries in both 2016 and 2018 by the association of college and research libraries (acrl) (acrl research planning and review committee, 2016; acrl research planning and review committee, 2018). as rds is an emerging area within academic librarianship, the literature consists mostly of case studies, focused primarily on assessing the needs of campus researchers and implementing these services, as summarized by tenopir, kaufman, sandusky, and pollock (2019). while this literature provides valuable information about researcher needs and the implementation of rds services, it provides little information on the emerging sub-discipline of data professionals. there is a need to capture data about the responsibilities, qualifications, and other information about data professional positions, such as education, experiences, and skills.   dei&a in academic libraries and higher education   academic libraries have a long history of valuing dei&a. examples include research on accessibility and diversity of library websites (yoon, hulscher, & dols, 2016) and lis student groups advocating for dei&a inclusion in lis curriculum (jardine & zerhusen, 2015). there are several examples of conferences and events on this topic, such as the conference on inclusion and diversity in library & information science (https://cidlis.umd.edu/). other national lis conferences, such as the digital library federation and research data access and preservation association, have tracks or specific foci on these topics. further, national groups such as the american library association and acrl have offices and committees to ensure the prioritization of dei&a.   similarly, higher education institutions have also been incorporating dei&a into their values and work, as seen throughout professional publications such as inside higher ed (willis, 2017) and the chronicle of higher education (brown, 2019). professional associations such as educause (n.d.) have identified dei&a as a critical priority and higher education conferences such as the leadership in higher education (https://www.magnapubs.com/leadership-in-higher-education-conference/) are likewise focusing on these themes. additionally, individual universities have incorporated these principles into many facets of the institution, such as the university of michigan’s diversity, equity, and inclusion certificate (n.d.) for graduate students and the university of california berkeley’s (2018) strategic plan. however, one area that has received less attention from the dei&a perspective is the job search process in academia, which is opaque and favors those on the inside (fernandes et al., 2020).   job posting analyses to create a landscape of data professionals   job postings describe “the duties and responsibilities ... experience, education, skills, knowledge, or other attributes required for the job; and the hiring organization, salary range, and other benefits” (kim & angnakoon, 2016, p. 327). academic libraries can also use job postings to articulate their needs and priorities, especially for areas of expansion such as rds.   subsets of rds job postings have been examined via content analysis. si, zhuang, xing, and guo (2013) compared the core competencies and duties of scientific data specialists in 46 job postings to the current curricula in 38 lis programs. they found that most lis curricula train students in the basics of data curation, but more specialized areas were limited. kim, warga, and moen (2013) studied job postings for digital curation positions and developed a set of competencies for digital curation responsibilities, which were used to create curricula in digital curation and data management. xia and wang (2014) visualized keyword and phrase occurrences of 167 job postings for social science data librarians from 2005-2012. chen and zhang (2017) analyzed 70 data management professionals’ positions, from january to april 2015 using word frequency analysis, finding that 27% of postings mentioned a master’s degree in library and information science (mlis). thematic analysis as a research method   thematic analyses “move beyond counting explicit words or phrases and focus on identifying and describing both implicit and explicit ideas within the data” (guest, macqueen, & namey, 2012, p. 10). this method yields richer results than word frequency analysis because it can “captur[e] the complexities of meaning within a textual data set” (guest et al., 2012, p. 11). this methodology has been previously applied to the analysis of job postings within academic libraries. hall-ellis (2005; 2006) used this confirmatory method to track changing expectations and requirements for entry-level cataloguer positions and managerial cataloguer positions. in addition to coding the appearance of predetermined variables in the job postings, hall-ellis (2005; 2006) also coded for the complexity of each variable, which cannot be done with word frequency analysis. a more rigorous analysis of job postings within rds using thematic analysis is lacking from the literature, with chen and zhang (2017, p. 22) noting that the results of their study shows “a need for a follow-up study to monitor the development of th[is] emerging job area.”   aims   this research project aims to answer the following research questions:   1.      what are the most frequently occurring qualifications (required and preferred) and responsibilities for data professional positions? a.      specifically, what education and experiences occur most frequently? b.      what research data activities occur most frequently? c.      what other responsibilities and skills occur most frequently? 2.      what is the median salary and salary range of data professional positions?   methods   this research study uses deductive thematic analysis to examine data professional job postings that were posted from january 1, 2013 to june 30, 2018. these job postings were gathered from the following electronic mailing lists: 1) acrl science & technology section (n.d), 2) code4lib jobs list (n.d.), 3) digital library federation job board (n.d.), 4) international association of social science information services & technology jobs portal (n.d.) and 5) research data access and preservation association (n.d.). in addition, datacure (an electronic mailing list on google groups) was analyzed for job postings; note that the viewer must be a member before accessing the list but anyone is allowed to join. these data sources were chosen because they are known nationally, attract job postings from a diverse pool of academic libraries, and provide access to job postings during the chosen time frame.   in some cases, the job announcement did not contain the complete job posting. in these cases, links to external websites (usually the university jobs portal), the internet archive wayback machine (n.d.), google searches, and personal communications were used to locate the complete job posting. seven job postings were excluded from this study because the full posting could not be located.   job postings were first evaluated based on the job title. if a job title referenced data or rds, the job posting was downloaded for further analysis. postings were then reviewed to determine if they met the following four inclusion criteria:   1.      full-time, permanent positions 2.      located in an academic library 3.      located within the us 4.      primarily focused on providing rds, which was defined as 50% or more of job responsibilities devoted to these services. the following description of rds from cox and pinfield (2014) was used to determine if the job position fulfilled this criterion and positions that focused on library or administrative data were excluded:   [rds] consists of a number of different activities and processes associated with the data lifecycle, involving the design and creation of data, storage, security, preservation, retrieval, sharing, and reuse, all taking into account technical capabilities, ethical considerations, legal issues and governance frameworks. (cox & pinfield, 2014, p. 300)   once it was concluded that a job met the four inclusion criteria, metadata about the job posting was recorded, including the university name, job title, and posting date (see appendix a for metadata on the job postings). in total, 236 full data professional job postings were gathered. however, this corpus contained duplicates. job postings from the same university posted within 12 months of each other were targeted as possible duplicates. several factors were scrutinized to determine if the postings were duplicates of the same position, including posting date, job title, responsibilities, and qualifications. if the postings had 25% or more difference in their responsibilities or qualifications, they were not considered duplicates and each posting was kept in the corpus. potential duplicate postings were reviewed individually to determine if the posting should be included or excluded. determinations were then discussed and agreement was reached on the inclusion or exclusion for each posting. if postings were duplicated, the posting with the most recent posting date was kept. in total, 59 postings were removed as duplicates, leaving 177 job postings corresponding to 180 job positions (3 job postings were for 2 positions).   to determine patterns in the qualifications and responsibilities for data professionals, a confirmatory approach was taken using a deductive thematic analysis methodology. a codebook of variables and attributes for each variable was determined prior to analyzing the job positions. the codebook was based on hall-ellis’ (2005; 2006) thematic analyses of cataloguing librarian job postings. appendix b shows the complete codebook of 43 variables and corresponding attributes. each variable in the codebook was operationally defined in order to avoid ambiguity. descriptions of when each variable should be used and should not be used were included. variables were grouped into three categories: 1) education, experience, and salary; 2) research data activities; and 3) other responsibilities and skills. for each of the 43 variables, the attribute of location in the job posting was coded (see table 1 for list of attributes). if the variable was mentioned in multiple locations in the job positions, only one location was recorded, based on the following hierarchy: required qualifications > preferred qualifications > responsibilities > description. for example, if the variable “data management plan” appeared in the responsibilities and preferred qualifications sections, it was coded as preferred qualifications. for the variables in the research data activities category and most variables in the other responsibilities or skills category, an interval scale correlating to the stated degree of complexity sought was also coded (table 1). the codebook was reviewed by two academic data professionals (who were not affiliated with the project) and their feedback was incorporated to ensure that the variables were an accurate and thorough representation of the responsibilities and qualifications sought for data professionals.   all job postings were coded independently to ensure consistency and reliability. initially, a small corpus of 15 job postings was coded and the codebook was refined to define variables more clearly, add additional variables, eliminate unneeded variables, and revise attributes. after these revisions, the entire corpus of 177 job postings was coded. coding discrepancies were resolved through discussion. coding reflected a high level of intercoder agreement; percent agreement was 94%, which is higher than the threshold of 80% for good agreement (guest et al., 2012).     table 1 attributes for the variable “data storage” a variable = data storage           attributes         location in the job posting required qualifications (minimum requirements; basic requirements) preferred qualifications (desired qualifications) responsibilities (duties) description not applicable degree of complexity sought experience (ability; demonstrated ability; aptitude) knowledge (understanding; competent; competence) familiarity implied not applicable a synonyms for each attribute are shown in parenthesis. the full codebook is in appendix b.     table 2 the carnegie classification of institutions of higher education for the job positions (n = 180) (shown in descending order of institutional size) carnegie classification n doctoral universities: very high research activity 146 doctoral universities: high research activity 19 doctoral/professional schools 1 master's colleges & universities: larger programs 2 baccalaureate colleges: arts & sciences focus 8 special focus four-year: medical schools & centers 3 special focus four-year: other health professions schools 1     results   metadata about the job positions   the entire corpus contained 177 job postings, corresponding to 180 job positions. all of the following analyses were based on the number of job positions. the number of job positions posted each year over the 2013-2017 time frame remained relatively consistent, ranging from 25 to 38 positions. the positions were geographically dispersed across the us, spread out across 37 states and washington d.c.   most positions were located at doctoral-granting universities with very high research activity (n = 146, 81%), based on the carnegie classification of institutions of higher education (indiana university, 2017). the breakdown of job positions by the carnegie classification of the institutions is shown in table 2.   from the 180 positions, there were 119 unique job titles (job titles were analyzed based on exactly how they appeared in the job posting). the four job titles occurring most frequently were:   ·        data services librarian (n = 23, 13%) ·        data curation librarian (n = 7, 4%) ·        research data management librarian (n = 6, 3%) ·        data librarian (n = 6, 3%)   further, over one-third (n = 63, 35%) of the job titles did not include the word “librarian”, instead using terms such as specialist, consultant, informationist, curator, coordinator, and analyst.   education and experience   of the 180 positions, almost 90% (n = 159) listed an mlis degree as a qualification (figure 1).   however, over 70% of positions (n = 132, 73%) accepted an equivalent degree in lieu of an mlis degree and all mentions of an equivalent degree were located in the required qualifications. one position listed this qualification as “mlis degree or equivalent advanced degree in the social sciences.” figures 2 and 3 show the level and disciplines mentioned for these equivalent degrees (note that a position could list multiple levels or disciplines). the most frequent equivalent degree level sought was an advanced degree (n = 73) and the most frequent discipline of the equivalent degree was relevant (n = 47). while the term “relevant” is ambiguous, it does reflect the terms used in the job postings.     figure 1 the location of an mlis as a qualification for the job position (n = 180).     figure 2 the levels of equivalent degrees mentioned. synonyms for advanced were graduate and professional; a synonym for doctorate was terminal. note that a position could list multiple degree levels.       figure 3 the disciplines of equivalent degrees mentioned. synonyms for relevant were related, appropriate, and comparable. note that a position could list multiple degree disciplines.     in addition to an mlis or equivalent degree, 60% of job positions (n = 108) wanted the candidate to have an additional degree (either undergraduate or graduate). for example, a preferred qualification for one job position was an “additional relevant graduate degree.” the majority (78%, n = 84) of these additional degrees were listed as a preferred qualification. as for the level of the degree, the majority wanted an advanced degree (n = 65; figure 4).   when an additional degree was mentioned, discipline(s) of that degree were sometimes also mentioned. of the 108 positions that listed an additional degree as a qualification, the science, technology, engineering and math (stem; n = 59) and social sciences (n = 47) disciplines were mentioned most frequently (a position could list multiple disciplines and the complete disciplinary list is shown in table 3).     table 3 disciplines listed for an additional degree as a qualification b discipline n stem 59 social sciences 47 data science, data intensive field, and others. 27 business 7 relevant 7 health sciences 5 arts & humanities 4 b note that a position could list multiple disciplines. synonyms for relevant were related, appropriate, and comparable.     figure 4 the level of an additional degree mentioned. synonyms for advanced were graduate and professional; a synonym for doctorate was terminal. note that a position could list multiple degree levels.     of the 117 positions with the word “librarian” in the title, 62% (n = 73) accepted an mlis degree or equivalent degree, while 36% (n = 42) only accepted an mlis degree (figure 5). conversely, of the 63 postings that did not use the word “librarian” in the job title, 65% (n = 41) accepted an mlis or equivalent degree and 2% (n = 1) only accepted an mlis degree.   in addition to educational qualifications, many positions were seeking professional experience. almost half (n = 87, 48%) wanted a candidate who had previous academic library experience, with those mentions split between required (n = 39) and preferred qualifications (n = 48). figure 6 shows the length of academic library experience listed in the job positions, with almost half (n = 43) not specifying a length of time. in terms of previous experience with research data, 60% (n = 108) of positions wanted a candidate with this type of experience, most frequently naming it a required qualification (n = 85). only a few positions (n = 21) listed a length of time for this experience, with 3 to 5 years (n = 11) being the most frequent length of time. for example, one position listed a required qualification as “minimum of three years professional experience working with large research datasets and/or familiarity with major data resources.”   in addition to professional experience, about one-fifth of the job positions (n = 35, 19%) were looking for additional academic experience. almost two-thirds of mentions were for lab or research experience (n = 23), while the remaining one-third of the mentions were for significant coursework or academic background in a discipline (n = 12; note that a position could list multiple types of academic experiences). all mentions of additional academic experience were in the required or preferred qualifications. while these terms for academic experiences are nebulous, they mirror the terms used in job postings. examples of these qualifications are “research laboratory experience” as a preferred qualification and “coursework or experience leading to knowledge of the principles and practices of data curation and long-term digital preservation” as a required qualification.     figure 5 degree requirements for positions with the word “librarian” in the job title (n = 117) and without the word “librarian” in the job title (n = 63).     figure 6 the length of experience in an academic library listed as a qualification (n = 180).     salary   almost half (n = 77, 43%) of the positions did not mention salary. when salary was mentioned, about a third (n = 57, 32%) only used descriptive words such as commensurate or competitive (figure 7). a quarter (n = 47, 25%) gave a quantitative salary value, with or without descriptive words. the range of salaries listed was from $40,000 to $157,000, with a median salary of $57,000, and over half (n = 25) clustered between $54,000 68,000 (figure 8).   research data activities   of the 180 job positions, the most common research data activities mentioned were general data management (n = 154, 86%), data repository (n = 122, 68%), data curation (n = 101, 56%), data discovery (n = 97, 54%) and data documentation (n = 96, 53%; figures 9 and 10 and appendix c). general data management was most commonly mentioned in the preferred qualifications (n = 73) and the degree of complexity sought most frequently was “experience” (n = 58, 37%). the variable “general data management” is vague, but it reflects the actual terminology used in job postings. for example, one job position listed “assists faculty and graduate students with data management” as a responsibility; this is also an example of “implied” as the degree of complexity for this variable. in contrast, the more specific variable “data management plans” was mentioned in over 40% of positions (n = 76, 42%), most commonly mentioned in the required qualifications section (n = 24).   “data repository” was mentioned in more than two-thirds of positions (n = 122, 67%). this was the variable with the highest number of occurrences in the required qualifications (n = 52); but it was also mentioned frequently in the responsibilities (n = 33) and preferred qualifications (n = 31). as for the degree of complexity sought, “experience” (n = 34) and “knowledge” (n = 32) were most common.     figure 7 how salary was described in the job positions (n = 180).     figure 8 histogram of salary values (n = 47). if a salary range was given for the position, the median value was used.     different types of data analysis (general, statistical, spatial, or qualitative) were often mentioned in the job positions. in total, at least 1 type of data analysis was listed in over 60% of positions (n = 111; note that multiple types of data analysis could be listed in a position). “general data analysis”, the variable used when a specific type of data analysis was not mentioned, was mentioned in over 40% of the positions (n = 78, 43%). over half of these mentions occurred in the required qualifications section (n = 42, 53%). additionally, half of these mentions were seeking “experience” for the degree of complexity (n = 39). for example, one job position stated, as a required qualification, “knowledge of quantitative data analysis applications.” statistical (n = 76, 42%), spatial (n = 46, 26%), and qualitative (n = 36, 20%) data analysis were also mentioned in the job positions. statistical analysis (n = 45, 59%) was most frequently listed as a required qualification, while spatial (n = 24, 52%) and qualitative data analysis (n = 18, 50%) were most frequently listed as preferred qualifications. as for the degree of complexity sought, all 3 types of analysis were most frequently seeking “experience” (statistical analysis: n = 45; spatial analysis: n = 24; qualitative analysis: n = 21).   other responsibilities and skills   about one-third (n = 60) of the job positions had faculty status; two-thirds of those with faculty status (n = 40) were also tenure-track. the requirement to research and publish was mentioned in about one-third of the positions (n = 55, 31%), most commonly listed in the responsibilities section (n = 28). having a public or customer service perspective was mentioned in 38% of the postings (n = 69), most frequently mentioned as a required qualification (n = 46, 67%).   instruction was mentioned in over three-fourths of positions (n = 138, 76%). although mentioned in all 4 main locations within a job posting, mentions of instruction were most frequently mentioned in the required qualifications (n = 49) and responsibilities (n = 46). this variable listed “experience” as the most common degree of complexity sought (n = 81, 59%).     figure 9 summary of the degree of complexity sought. raw values are shown in appendix c.     figure 10 summary of b) location in the job posting for 17 research data activities (n = 180). raw values are shown in appendix c.     consultation was mentioned in over two-thirds of the positions (n = 121, 67%), most frequently in the responsibilities section (n = 93). additionally, 85% of these mentions listed “implied” as the degree of complexity sought (n = 103), meaning that a specific degree of complexity was not mentioned. for example, one job position stated in the description that the incumbent will “provid[e] training and consulting services.”   more than 40% of the positions were focused on meeting research data needs within specific disciplines (n = 75, 42%). this variable was most commonly listed in the responsibilities section (n = 42, 23%). of those focused on specific disciplines, the most common discipline was the social sciences (n = 32; table 4 shows the complete disciplinary breakdown).   table 4 disciplines of job positions that focused on the research data needs of specific disciplines c discipline n social sciences 32 stem 22 health sciences 20 business 7 arts & humanities 4 c if specific departments were listed, they were grouped into their broader discipline (multiple disciplines could be listed for a position).   additionally, 28% (n = 51) of the job positions were the liaison to 1 or more departments or units on campus; this variable was most commonly listed in the responsibilities section (n = 40, 22%). of those with liaison responsibilities, three-fourths (n = 37, 73%) listed specific departments or disciplines (table 5) and the remaining positions had a department(s) assigned upon hiring. of the 51 positions listing liaison responsibilities, over 85% (n = 44) also had instruction duties, as opposed to 72% of positions (n = 93) without liaison duties.   table 5 disciplines for job positions that included liaison responsibilities to one or more department or unit d discipline n stem 14 social sciences 13 business 8 health sciences 4 administrative units 3 data science 2 arts & humanities 1 d if specific departments were listed, they were grouped into their broader discipline (multiple disciplines could be listed for a position).   the variable of dei&a related to the position, not the university or library, was mentioned in less than half of the positions (n = 75, 42%). these statements were most often included in the required qualifications section (n = 51), followed by the preferred qualifications section (n = 15). as these statements most often referred to a candidate’s commitment to or understanding of the importance of dei&a, the degree of complexity was not coded. for example, one required qualification was a “commitment to supporting and working in a multicultural and diverse environment.” figure 11 shows that this variable was included in more job positions over time.     figure 11 number of occurrences of dei&a statements relating to the position over time. positions from 2018 were not included because they were only gathered for half of that year.     discussion   what are the required and preferred qualifications and responsibilities for data professional positions?   overall, the education, experiences, and skills mentioned throughout these data professional job positions show that this sub-discipline of academic librarianship is looking for a mixture of traditional (instruction, consultation, and others) and non-traditional areas (general data management, data repositories, and others) for lis. while the skills and experiences of those within the academic library pipeline are still sought, this mixture indicates an eagerness to recruit candidates from outside of the traditional lis pipeline; this is a positive sign towards diversifying academic librarianship. therefore, data professional positions are ripe to accept a variety of academic backgrounds and professional experiences, which naturally attract diverse candidates and thereby increase other forms of diversity.   education and experience   in the degree qualifications, over 70% (n = 132, 67%) accepted an equivalent degree in lieu of the mlis degree. however, most positions were still seeking candidates with a degree beyond a bachelor’s (n = 104). interestingly, for these equivalent degrees, most commonly the term “relevant” (n = 47) was used to describe the discipline or the discipline was not specified (n = 41). if a specific discipline was mentioned, stem was the most common (n = 35). this indicates that libraries are seeking candidates with graduate degrees from all disciplines for their data professional positions, allowing for a diverse set of backgrounds and thus more diverse candidates. many libraries were seeking candidates possessing an additional degree (n = 108, 60%), most frequently mentioned as a preferred qualification (n = 84). again, if a specific discipline was mentioned, stem was most common (n = 59). these degree qualifications are troubling from a dei&a lens because many inequities in our society prevent individuals from obtaining a graduate degree much less multiple graduate degrees (soto & yao, 2010). in 2018, only 10.2% of the us adult population had a master’s degree and only 2.1% had a doctoral degree (oh and kim, 2020). instead of listing these degrees by default, an analysis should be done to demonstrate how the degree(s) would help the candidate to fulfill the job responsibilities (thielen & neeser, 2019). also, see if an institution offers any benefits (such as tuition reimbursement) that would allow a candidate to earn another degree while working, and if so include them in the job posting.   the term “data intensive field” was often used to describe the discipline of an equivalent (n = 30) or additional degree (n = 27). this term is often used in rds. it is hypothesized that libraries are using this term to denote that they would like a candidate with research data experience but do not want to list specific disciplines. however, from a dei&a lens, this term is subjective, perhaps leaving a candidate unsure if their degree meets this qualification. it is suggested to avoid this ambiguous term in job postings. further, individuals from underrepresented groups are less likely to apply to positions if they do not meet all of the qualifications (mohr, 2014), so including ambiguous jargon will make them less likely to apply.   over a third of the data professional positions (n = 63) did not use the word “librarian” in the job title; this may impact the degree qualifications. of the positions that include this word in the job title (n = 117), 36% (n = 42) only accept an mlis degree. conversely, of the positions without this word in the job title (n = 63), 2% (n = 1) only accept an mlis degree. the difference in degree qualifications is an excellent example of how libraries are writing job positions that seek to diversify this sub-discipline.   another indication that many libraries are looking to recruit outside of the lis pipeline is that of the positions that wanted candidates to have previous academic library experience (n = 87), only 45% of these mentions (n = 39) occurred in the required qualifications section.   in addition to degrees, previous experiences mentioned in the job positions also indicate an emphasis on areas traditionally considered outside the scope of lis. experience working with research data was a common qualification (n = 108), most frequently listed as a required qualification. finally, it is important to note that almost 20% of the positions (n = 35) mentioned additional academic experiences (lab or research experience, academic background, and others) as a required or preferred qualification. this could be a way for a candidate to demonstrate knowledge of a particular area without having an academic degree. asking for these types of additional academic experiences, instead of an additional degree, is another excellent way to incorporate dei&a principles into a job posting.   research data activities   overall, the research data activities that were most frequently mentioned in the data professional job positions show that this sub-discipline of academic librarianship values areas traditionally outside of lis (such as general data management, data repositories, and various types of data analysis). general data management (n = 155) was the second most commonly mentioned variable in the job positions, second to the mlis degree (n = 159).   unsurprisingly, general data management was the most frequently mentioned research data activities variable (n = 155). interestingly, although general data management was most commonly mentioned in the preferred qualifications (n = 73), “experience” (n = 58) was the most frequent degree of complexity for this variable. this suggests that libraries want a candidate with experience managing research data, but know that it may not be feasible to ask for this as a required qualification. data repository is the variable with the highest number of occurrences in the required qualifications section (n = 51). this shows that there is much interest in hiring candidates with these skills and, therefore, offering these services on campus. overall, at least 1 of the 4 types of data analysis were mentioned in over 60% of positions (n = 111; note that a position could list multiple types). assisting patrons with data analysis is not a traditional area of lis, but this result indicates that libraries consider this an unmet need that they are trying to fulfill on their campuses.   academic libraries are seeking to hire specialist data professionals as well as generalist data professionals; 42% of the positions (n = 75) were seeking to hire a specialist data professional, while the other 58% (n = 104) were seeking to hire a generalist. the occurrence of these specialist data professional positions is another indication that libraries are trying to recruit candidates from outside the traditional lis pipeline.   other responsibilities and skills   many of the common variables in this section need further explanation or different terminology entirely in order to recruit candidates from outside of lis. public or customer service perspective was mentioned in almost 40% of the postings (n = 69), with two-thirds of those mentions in the required qualifications section. public or customer service is not necessarily a tenant of other fields like it is in lis, so providing further context to this requirement would give candidates a better understanding of what this qualification entails and why it is valued in this context.   liaison duties are another example of library jargon in these positions. almost 30% of positions (n = 51) had liaison duties. it is unlikely that someone outside of lis would understand what the term “liaison” means. instead of saying “liaison to the political science department”, this could be rephrased as “librarian for the political science department.” small changes like this could have a huge impact on whether candidates outside of lis decide to apply for a position. additionally, of those listing liaison duties, three-fourths (n = 37) listed being a liaison to a specific department(s). while listing these departments adds specificity to the job position, it also may discourage applicants who do not have an academic background or experience with the subject area(s). writing something like “departments will be assigned based on the candidate's background and interests,” will help to recruit a more diverse candidate pool.   instruction was mentioned in three-fourths of the positions (n = 138, 76%) and consultation was mentioned in two-thirds of the positions (n = 121, 67%). both of these activities are common across job sectors within the lis profession. the high number of mentions of these two variables shows that academic libraries, while embracing new ways of engaging with patrons, believe that these traditional means of engagement are still vital parts of the services they offer on campus.   it is encouraging to see that the mentions of dei&a have increased during the time period studied (figure 11). however, there is still room for improvement because, over the 5 years in this study, less than half of the positions (n = 75, 42%) included this variable. dei&a related to the position was the focus, as opposed to generic statements about the university or library, because this was felt to be a demonstration of commitment to these principles rather than an human resources requirement. having a required qualification for all job positions related to dei&a could concretize academic libraries’ commitment to these principles and practices.   what is the median salary and salary range of data professional positions?   this study cannot give a definitive answer to this research question because only 26% (n = 47) of the job positions listed a quantitative salary value. most frequently, salary was not mentioned (n = 77, 43%). an additional third of the job positions (n = 57) only used qualitative descriptors for salary such as “competitive” or “commensurate”. however, of the 47 positions listing a salary value or range, the median salary was $57,000.   not mentioning salary or only providing qualitative salary descriptors is problematic from a dei&a lens. this practice favors those already working in academic libraries as they will have inside access to and knowledge about common practices and resources, disadvantaging recent lis graduates, and those outside of the traditional lis pipeline. for example, those already working in academic libraries may have access to internal salary documents and databases or be able to ask their professional networks about salary information and practices. it also favors those working in the part of the country where the job is located, because they may have an idea of data professional salaries in their geographic area. for example, a competitive salary at a university in san francisco, california will be very different from a competitive salary at a university in rural michigan. furthermore, these practices could hinder a candidate’s ability to effectively negotiate salary and individuals from underrepresented groups are less likely to negotiate salaries (silva & galbraith, 2018). listing a salary range indicates that candidates can negotiate; not doing so furthers inequity between those who already hold privilege from those who do not.   additionally, the salary values listed for the job positions may not be an accurate reflection of the person hired for a position. a new employee’s salary could be higher or lower than the stated salary due to their qualifications and experiences. a follow-up study could survey recently hired data professionals, asking them for their salary upon hire.   study limitations   this study does have some limitations. first, the sources of the job postings were chosen because they were known to attract postings for data professionals in academic libraries. however, these sources were not exhaustive for data professional job postings in academic libraries from 2013-2018. additionally, job positions were only included in this study when the full job posting was available. as noted above, seven job positions were excluded because the full job postings were not available. this study also only included job positions within the us; data professionals are a growing sector in academic libraries worldwide. a follow-up study could analyze job postings for data professionals outside of the us.   an inherent limitation of job posting analyses is that job postings tend to be very aspirational, meaning that a data professional’s actual responsibilities could vary greatly from those listed in the job posting. a follow-up study could carry out in-depth interviews with data professionals to compare how their actual responsibilities align with those in the job posting.   finally, this study is undercounting the number of data professionals working in academic libraries, especially those working at master’s or baccalaureate institutions. many could have rds roles or responsibilities added to their job duties after hiring as data needs emerge on campus. additionally, at many small and mid-sized institutions, a librarian may be responsible for providing rds but this responsibility is not large enough to be reflected in their job title (which was the initial screening mechanism to determine if a position should be included in this study).   conclusion   studies such as this do not have an impact unless the results are put into action. the following recommendations will help the reader to use this data to take steps toward incorporating dei&a principles and practices into job postings:   ●       write each and every sentence within a job posting using the lens of dei&a principles and practices ●       list a quantitative salary value; it is a simple way to make the hiring process more transparent and less prone to inequitable practices. listing a range indicates the possibility of negotiation, which is helpful for underrepresented groups ●       carefully consider which degrees to include as required or preferred qualifications. for example, think critically about how an mlis or an additional graduate degree would help the applicant perform the job responsibilities. many positions in this study required an mlis or asked for multiple degrees, which automatically limits the applicant pool. due to inequalities built into our societal and educational systems, not everyone has access to attain a graduate degree. consider undergraduate degrees or academic background as a way for an applicant to demonstrate expertise ●       include dei&a as a required qualification in the job posting to demonstrate that the institution is committed to hiring applicants who understand the value and importance of dei&a ●       write the job description that the candidate will perform; job postings should be realistic not aspirational. one way to accomplish this is to limit preferred qualifications ●       finally, this data can be used to initiate conversations; showing quantitative evidence of how disparities are inadvertently woven into hiring practices and providing evidence based suggestions for improvement can be a valuable tool for data-driving decision-making. this set of recommendations is also transferable to other sub-disciplines of librarianship   job postings are a small yet very important part of the hiring process. it is hoped that this article will inspire reviews of hiring processes as a whole. the data is openly available in the dryad repository https://datadryad.org/stash/dataset/doi:10.6078/d1k419; the authors strongly encourage other researchers to further analyze this data.   acknowledgements   the authors thank kristin briney for reviewing the codebook, as well as marie kennedy, abigail goben, and tina griffin for reviewing a draft of this article and providing valuable feedback.   references   acrl research planning and review committee. (2016). 2016 top trends in academic libraries: a review of the trends and issues affecting academic libraries in higher education. college & research libraries news, 77(6), 274-281. https://doi.org/10.5860/crln.77.6.9505   acrl planning and review committee. (2018). 2018 top ten trends in academic libraries: a review of the trends and issues affecting academic libraries in higher education. college & research libraries news, 79(6), 286-300. https://doi.org/10.5860/crln.79.6.286   association of college & research libraries science & technology section [electronic mailing list]. (n.d.). retrieved from lists.ala.org/sympa/info/sts-l   brown, s. (2019). want a more diverse campus? start at the top. in the chronicle of higher education. retrieved from https://www.chronicle.com/article/want-a-more-diverse-campus-/247285   code4lib jobs [electronic mailing list]. (n.d.). retrieved from jobs.code4lib.org   chen, h., & zhang, y. (2017). educating data management professionals: a content analysis of job descriptions. the journal of academic librarianship, 43(1), 18–24. https://doi.org/10.1016/j.acalib.2016.11.002 0099-1333   cox, a. m., & pinfield, s. (2014). research data management and libraries: current activities and future priorities. journal of librarianship and information science, 46(4), 299–316. https://doi.org/10.1177/0961000613492542   digital library federation job board [electronic mailing list]. (n.d.). retrieved from jobs.diglib.org/   educause diversity, equity, and inclusion. (n.d.). retrieved from https://www.educause.edu/about/diversity-equity-and-inclusion   fernandes, j.d., sarabipour, s., smith, c.t., niemi, n.m., jadavji, n.m., kozik, a.j., holehouse, a.s., pejaver, v., symmons, o., filho, a.w.b., & haage, a. (2020). a survey-based analysis of the academic job market. elife, 9:e54097.  https://doi.org/10.7554/elife.54097   guest, g., macqueen, k., & namey, e. (2012). applied thematic analysis. los angeles, ca: sage publications.   hall-ellis, s. d. (2005). descriptive impressions of entry-level cataloger positions as reflected in american libraries, autocat, and the colorado state library jobline, 2000-2003. cataloging & classification quarterly, 40(2), 33-72. https://doi.org/10.1300/j104v40n02_05   hall-ellis, s. d. (2006). descriptive impressions of managerial and supervisory cataloger positions as reflected in american libraries, autocat, and the colorado state library jobline, 2000-2004: a content analysis of education, competencies, and experience. cataloging & classification quarterly, 42(1), 55-92. https://doi.org/10.1300/j104v42n01_06   indiana university. (2017). about the carnegie classification. in the carnegie classification of institutions of higher education. retrieved from http://carnegieclassifications.iu.edu/   international association for social science information service & technology jobs portal [electronic mailing list]. (n.d.). retrieved from https://iassistdata.org/jobs-repository/   internet archive wayback machine. (n.d.). retrieved from archive.org/web   jardine, f. m., & zerhusen, e. k. (2015). charting the course of equity and inclusion in lis through idiversity. the library quarterly, 85(2), 185-192. https://doi.org/10.1086/680156   kim, j., & angnakoon, p. (2016). research using job advertisements: a methodological assessment. library & information science research, 38(4), 327-335. https://doi.org/10.1016/j.lisr.2016.11.006   kim, j., warga, e., & moen, w. (2013). competencies required for digital curation: an analysis of job advertisements. international journal of digital curation, 8(1), 66-83. https://doi.org/10.2218/ijdc.v8i1.242 mohr, t. s. (2014). why women don’t apply for jobs unless they’re 100% qualified. in harvard business review. retrieved from https://hbr.org/2014/08/why-women-dont-apply-for-jobs-unless-theyre-100-qualified   oh, b., & kim, c. (2020). broken promise of college? new educational sorting mechanisms for intergenerational association in the 21st century. social science research, 86(102375), 1-15. https://doi.org/10.1016/j.ssresearch.2019.102375   research data access & preservation association [electronic mailing list]. (n.d.). retrieved from https://rdapassociation.org   si, l., zhuang, x., xing, w., & guo, w. (2013). the cultivation of scientific data specialists: development of lis education oriented to e-science service requirements. library hi tech, 31(4), 700-724. https://doi.org/10.1108/lht-06-2013-0070   silva, e., & galbraith, q. (2018). salary negotiation patterns between women and men in academic libraries. college & research libraries, 79(3), 324-335. https://doi.org/10.5860/crl.79.3.324   soto, m. & yao, c. (2010). retention of women of color in stem doctoral programs. proceedings of the 29th annual midwest research to practice conference in adult, continuing, community, and extension education. east lansing, michigan, 207-213.   tenopir, c., sandusky, r. j., allard, s., & birch, b. (2013). academic librarians and research data services: preparation and attitudes. ifla journal, 39(1), 70-78. https://doi.org/10.1177/0340035212473089   tenopir, c., kaufman, j., sandusky, r., & pollock, c. (2019). research data services in academic libraries: where are we today? [white paper]. in choice. retrieved from https://choice360.org/librarianship/whitepaper   thielen, j., & neeser, a. (2019). how you can write more inclusive data practitioner job postings. journal of escience librarianship, 8(2), e1167. https://doi.org/10.7191/jeslib.2019.1167   university of california berkeley. (2018). berkeley strategic plan. in berkeley university of california. retrieved from https://strategicplan.berkeley.edu/publications/   university of michigan. (2020). rackham professional development diversity, equity, and inclusion certificate. in rackham graduate school. retrieved from https://rackham.umich.edu/professional-development/dei-certificate/   willis, d. s. (2017). getting up to speed on diversity. in inside higher ed. retrieved from https://www.insidehighered.com/advice/2017/08/21/how-graduate-students-can-demonstrate-commitment-diversity-job-interviews-essay   xia, j., & wang, m. (2014). competencies and responsibilities of social science data librarians: an analysis of job descriptions. college & research libraries, 75(3), 362-388. https://doi.org/10.5860/crl13-435   yoon, k., hulscher, l., & dols, r. (2016). accessibility and diversity in library and information science: inclusive information architecture for library websites. the library quarterly, 86(2), 213-229. https://doi.org/10.1086/685399     appendix a metadata about data professional job postings (note, this appendix only includes the 177 job postings analyzed in this study)   university name carnegie classification position title posting date notes american university doctoral universities: high research activity research data librarian 2013-01   auburn university doctoral universities: very high research activity research data management librarian 2017-03   boston college doctoral universities: very high research activity data and visualization librarian 2016-12   brown university doctoral universities: very high research activity scientific data management specialist 2013-11   brown university doctoral universities: very high research activity scientific data curator 2013-03   bryn mawr college baccalaureate colleges: arts & sciences focus social sciences and data librarian 2018-02   california state university northridge master's colleges & universities: larger programs business & data librarian 2013-01   carnegie mellon university doctoral universities: very high research activity data services librarian 2014-05   case western reserve university doctoral universities: very high research activity research data specialist 2018-01   colby college baccalaureate colleges: arts & sciences focus social sciences data librarian 2014-02   colorado state university doctoral universities: very high research activity data management specialist 2017-11   colorado state university doctoral universities: very high research activity data management specialist 2016-01   columbia university doctoral universities: very high research activity research support & data services librarian 2014-04   columbia university doctoral universities: very high research activity research support & data services librarian 2016-10   columbia university doctoral universities: very high research activity data services librarian 2014-12   columbia university doctoral universities: very high research activity data services & emerging technologies librarian 2014-05   columbia university doctoral universities: very high research activity research support & data services librarian 2015-04   cornell university doctoral universities: very high research activity social science and geospatial data librarian 2017-09   cornell university doctoral universities: very high research activity research data and environmental sciences librarian 2014-02   cuny graduate school and university center doctoral universities: very high research activity data librarian 2014-12   dartmouth college doctoral universities: very high research activity data & visualization librarian 2015-12   depaul university doctoral universities: high research activity data services & government information librarian 2016-06   drake university doctoral/professional schools data and business librarian 2015-04   drexel university doctoral universities: very high research activity director, data & digital stewardship 2015-10   drexel university doctoral universities: very high research activity director, informatics for research engagement 2014-02   duke university doctoral universities: very high research activity senior research data management consultant 2016-08 this posting was for two positions east carolina university doctoral universities: high research activity data services librarian 2017-03   florida institute of technology doctoral universities: high research activity research data specialist 2014-11   florida institute of technology doctoral universities: high research activity data librarian 2018-05   florida state university doctoral universities: very high research activity data research librarian 2013-11   florida state university doctoral universities: very high research activity social sciences research & data librarian 2016-10   george washington university doctoral universities: very high research activity data services librarian 2017-10   george washington university doctoral universities: very high research activity data services librarian 2014-07   georgia southern university doctoral universities: high research activity discovery services and data curation librarian 2014-12   georgia state university doctoral universities: very high research activity team leader, research data services 2016-01   georgia state university doctoral universities: very high research activity quantitative data specialist for the social sciences 2017-08   georgia state university doctoral universities: very high research activity business data services librarian 2014-06   harvard university doctoral universities: very high research activity librarian for the social sciences and visualization 2014-10   harvard university doctoral universities: very high research activity research data management librarian for the sciences 2018-04   harvard university doctoral universities: very high research activity research data & collections librarian 2017-05   indiana university doctoral universities: very high research activity research data management librarian 2016-06   indiana university bloomington doctoral universities: very high research activity research data management librarian 2015-08   indiana university bloomington doctoral universities: very high research activity research data management librarian and head of scholarly communication department 2016-05   johns hopkins university doctoral universities: very high research activity data management services manager 2015-12   johns hopkins university doctoral universities: very high research activity data informationist 2016-03   johns hopkins university doctoral universities: very high research activity data services manager 2017-01   johns hopkins university doctoral universities: very high research activity data management consultant 2015-04   johns hopkins university doctoral universities: very high research activity data management specialist 2016-02   kenyon college baccalaureate colleges: arts & sciences focus social sciences and data librarian 2016-03   lehigh university doctoral universities: high research activity business/data librarian 2015-11   lewis & clark college baccalaureate colleges: arts & sciences focus science & data services librarian 2014-10   louisiana state university doctoral universities: very high research activity data curation librarian 2015-01   massachusetts institute of technology doctoral universities: very high research activity program head, data management services 2016-11   michigan state university doctoral universities: very high research activity data librarian 2016-04   middlebury college baccalaureate colleges: arts & sciences focus data services librarian 2015-05   montana state university doctoral universities: high research activity data management librarian 2013-08   new york university doctoral universities: very high research activity knowledge management librarian 2014-10   new york university doctoral universities: very high research activity data services librarian 2015-03   new york university doctoral universities: very high research activity research data management librarian 2014-11   north carolina state university doctoral universities: very high research activity research data & infrastructure librarian 2018-03   north carolina state university doctoral universities: very high research activity research librarian for engineering and biotechnology 2015-09   northwestern university doctoral universities: very high research activity data scientist 2017-03   oakland university doctoral universities: high research activity research data librarian 2015-11   occidental college baccalaureate colleges: arts & sciences focus data and information specialist for the social sciences 2017-08   ohio state university doctoral universities: very high research activity data management services librarian 2013-05   oregon health & science university special focus four-year: medical schools & centers basic science liaison/research data management librarian 2015-12   oregon state university doctoral universities: very high research activity data management specialist 2015-12   pennsylvania state university doctoral universities: very high research activity science data librarian 2014-11   princeton university doctoral universities: very high research activity data services specialist 2013-06   princeton university doctoral universities: very high research activity interdisciplinary quantitative research librarian 2015-08   purdue university doctoral universities: very high research activity data repository outreach specialist 2015-08   purdue university doctoral universities: very high research activity research data specialist 2015-02   purdue university doctoral universities: very high research activity digital data repository specialist 2014-12   reed college baccalaureate colleges: arts & sciences focus data services librarian 2015-07   rice university doctoral universities: very high research activity data and government information librarian 2017-11   rice university doctoral universities: very high research activity head, kelley center for government information, data & geospatial services 2014-06   rutgers university doctoral universities: high research activity data services librarian 2013-06   san diego state university doctoral universities: high research activity social science & data librarian 2018-01   san jose state university master's colleges & universities: larger programs data services librarian 2017-05   southern california university of health sciences special focus four-year: other health professions schools knowledge management & data specialist 2015-09   stanford university doctoral universities: very high research activity data services and visualization librarian 2017-05   stanford university doctoral universities: very high research activity engineering librarian for data and collections 2018-06   temple university doctoral universities: very high research activity research and data services librarian 2018-05   texas a&m university doctoral universities: very high research activity data librarian 2016-09   tufts university doctoral universities: very high research activity librarian for research data 2016-09   tufts university doctoral universities: very high research activity social science data librarian 2017-05   university of arizona doctoral universities: very high research activity research data management librarian 2017-03   university of arkansas at little rock doctoral universities: high research activity data services librarian 2018-06   university of california irvine doctoral universities: very high research activity e-research and digital scholarship services librarian 2014-10   university of california los angeles doctoral universities: very high research activity sciences data informationist 2016-11   university of california los angeles doctoral universities: very high research activity grand challenges data administrator 2016-09   university of california los angeles doctoral universities: very high research activity director of ucla libraries social science data archive 2016-06   university of california san diego doctoral universities: very high research activity data services and collections librarian 2014-03   university of california san diego doctoral universities: very high research activity director, research data curation services 2013-01   university of california san diego doctoral universities: very high research activity metadata specialist 2018-06   university of california san diego doctoral universities: very high research activity data science librarian 2017-09   university of california san diego doctoral universities: very high research activity director, research data curation services 2013-01   university of california san diego doctoral universities: very high research activity research data metadata librarian 2017-11   university of california san diego doctoral universities: very high research activity research data curation program technical analyst 2013-07   university of california berkeley doctoral universities: very high research activity science data & engineering librarian 2015-07   university of california berkeley doctoral universities: very high research activity business & data librarian 2015-08   university of california berkeley doctoral universities: very high research activity research data management service design analyst 2015-01   university of california berkeley doctoral universities: very high research activity data services librarian 2017-01   university of california davis doctoral universities: very high research activity associate director, data management program 2015-08   university of california davis doctoral universities: very high research activity data management analyst 2017-03   university of california san francisco special focus four-year: medical schools & centers data services and assessment librarian 2016-12   university of california santa barbara doctoral universities: very high research activity humanities data curator 2015-09   university of california santa barbara doctoral universities: very high research activity geospatial data curator 2013-08   university of california santa barbara doctoral universities: very high research activity data services and digital scholarship librarian 2018-05   university of chicago doctoral universities: very high research activity biomedical data librarian 2017-12   university of chicago doctoral universities: very high research activity social science data and sociology librarian 2017-04   university of chicago doctoral universities: very high research activity data research services and biomedical librarian 2017-04   university of colorado boulder doctoral universities: very high research activity data services librarian 2017-07   university of florida doctoral universities: very high research activity data management librarian 2015-04   university of houston doctoral universities: very high research activity social science data librarian 2014-03   university of houston doctoral universities: very high research activity data services librarian 2016-11   university of houston doctoral universities: very high research activity research data management librarian 2018-05   university of illinois urbana-champaign doctoral universities: very high research activity director, research data service and open-rank professor 2013-10   university of illinois urbana-champaign doctoral universities: very high research activity data curation specialist 2014-11 this posting was for two positions university of iowa doctoral universities: very high research activity data services manager 2017-02   university of kansas doctoral universities: very high research activity data services librarian 2013-06   university of maryland doctoral universities: very high research activity data services librarian 2017-01   university of maryland doctoral universities: very high research activity data services librarian 2018-07   university of massachusetts amherst doctoral universities: very high research activity data services librarian 2017-05   university of miami doctoral universities: very high research activity data services librarian 2016-09   university of michigan doctoral universities: very high research activity data workflows specialist 2017-01   university of michigan doctoral universities: very high research activity research data curation librarian 2014-11   university of michigan doctoral universities: very high research activity research data services manager 2013-12   university of michigan doctoral universities: very high research activity data curation librarian 2017-07   university of michigan doctoral universities: very high research activity health sciences data services informationist 2015-11   university of minnesota doctoral universities: very high research activity biosciences liaison librarian and scientific data curator 2017-06   university of minnesota doctoral universities: very high research activity informatics/data services specialist 2013-06   university of minnesota doctoral universities: very high research activity public health liaison and data curation specialist 2015-10   university of nebraska lincoln doctoral universities: very high research activity data curation librarian 2016-08   university of nebraska lincoln doctoral universities: very high research activity data curation librarian 2013-12   university of nevada las vegas doctoral universities: high research activity social sciences data librarian 2014-08   university of new hampshire doctoral universities: very high research activity business and data reference librarian 2015-03   university of new hampshire doctoral universities: very high research activity research data services librarian 2018-01   university of new mexico doctoral universities: very high research activity director of research data services 2013-12   university of new mexico doctoral universities: very high research activity data curation librarian 2017-07   university of north carolina at chapel hill doctoral universities: very high research activity repository librarian 2015-04   university of north carolina at greensboro doctoral universities: high research activity research and data support coordinator 2013-10   university of north carolina wilmington doctoral universities: high research activity digital program and data management librarian 2013-03   university of notre dame doctoral universities: very high research activity digital library data curation developer 2015-07   university of pennsylvania doctoral universities: very high research activity business & data analysis librarian 2018-04   university of pennsylvania doctoral universities: very high research activity scholarly communications & data curation librarian 2016-03   university of pittsburgh doctoral universities: very high research activity data services librarian 2017-07   university of pittsburgh doctoral universities: very high research activity data curation librarian 2018-06   university of rhode island doctoral universities: high research activity data services librarian 2016-05   university of rochester doctoral universities: very high research activity science & engineering outreach librarian (data) 2018-01   university of tennessee doctoral universities: very high research activity data curation librarian 2013-03   university of texas at arlington doctoral universities: very high research activity data & escience librarian 2014-12   university of texas at arlington doctoral universities: very high research activity social sciences data librarian 2014-11   university of texas at austin doctoral universities: very high research activity data management coordinator 2015-09   university of vermont doctoral universities: high research activity science and data librarian 2017-02   university of virginia doctoral universities: very high research activity senior research data scientist 2014-05   university of virginia doctoral universities: very high research activity data and geographical information librarian 2013-01   university of virginia doctoral universities: very high research activity research data specialist 2017-02   university of virginia doctoral universities: very high research activity clinical data research specialist 2017-02   university of washington doctoral universities: very high research activity data management librarian 2015-05   university of wisconsin madison doctoral universities: very high research activity science & engineering data & information specialist 2018-04 this posting was for two positions university of wisconsin madison doctoral universities: very high research activity digital curation coordinator 2017-06   university of wisconsin milwaukee doctoral universities: very high research activity data services librarian 2013-07   upstate medical university special focus four-year: medical schools & centers data services librarian 2018-05   vanderbilt university doctoral universities: very high research activity business and data analysis librarian 2016-12   vassar college baccalaureate colleges: arts & sciences focus social sciences and data librarian 2016-03   villanova university doctoral universities: high research activity social sciences and data services librarian 2015-12   virginia commonwealth university doctoral universities: very high research activity research data librarian 2017-05   virginia polytechnic institute and state university doctoral universities: very high research activity data and informatics consultant 2013-12   virginia polytechnic institute and state university doctoral universities: very high research activity social science data consultant & data educator coordinator 2017-04   virginia polytechnic institute and state university doctoral universities: very high research activity research data consultant 2014-05   washington university in st. louis doctoral universities: very high research activity data specialist 2015-04   western michigan university doctoral universities: high research activity data librarian 2018-02   yale university doctoral universities: very high research activity data librarian 2017-11   yale university doctoral universities: very high research activity data librarian for the health sciences 2018-03   yale university doctoral universities: very high research activity research data support specialist 2016-07   yale university doctoral universities: very high research activity librarian for finance, accounting & business data 2018-04       appendix b codebook     variable attributes education, experience, and salary         mlis degree not applicable description respon. preferred qual. required qual.         equivalent degree not applicable description respon. preferred qual. required qual.         equivalent degree level* not applicable bachelor's master's doctorate advanced not specified       equivalent degree discipline(s)* not applicable arts & humanities social sciences stem data intensive/data science business relevant not specified   academic library experience no 1-2 years 3-5 years 5+ years length not specified         [location in job posting] not applicable description respon. preferred qual. required qual.         research data experience no 1-2 years 3-5 years 5+ years length not specified         [location in job posting] not applicable description respon. preferred qual. required qual.         supervisory experience no 1-2 years 3-5 years 5+ years length not specified         [location in job posting] not applicable description respon. preferred qual. required qual.         additional experience or degree not applicable description respon. preferred qual. required qual.         additional degree level* not applicable bachelor's master's doctorate or phd advanced         [discipline of additional degree*] not applicable arts & humanities social sciences stem data intensive, data science, and others. business relevant not specified   additional experience* not applicable significant coursework or academic background subject knowledge lab or research experience other, specify: [free text]         carnegie classification of institution baccalaureate master's doctoral special focus           [for doctoral institutions, specify the research intensity level] not applicable very high high doctoral/professional           salary information* not applicable commensurate competitive other, specify: [free text]           salary range or minimum not applicable [exact salary values]                                 research data activities   management     general data management not applicable implied familiarity knowledge experienced         [location in job posting] not applicable description respon. preferred qual. required qual.         data management plans not applicable implied familiarity knowledge experienced         [location in job posting] not applicable description respon. preferred qual. required qual.       discovery and re-use     data discovery not applicable implied familiarity knowledge experienced         [location in job posting] not applicable description respon. preferred qual. required qual.       collection                   data organization not applicable implied familiarity knowledge experienced         [location in job posting] not applicable description respon. preferred qual. required qual.         data documentation not applicable implied familiarity knowledge experienced         [location in job posting] not applicable description respon. preferred qual. required qual.       storage     data storage not applicable implied familiarity knowledge experienced         [location in job posting] not applicable description respon. preferred qual. required qual.         data security not applicable implied familiarity knowledge experienced         [location in job posting] not applicable description respon. preferred qual. required qual.       analysis     data visualization not applicable implied familiarity knowledge experienced         [location in job posting] not applicable description respon. preferred qual. required qual.         general data analysis not applicable implied familiarity knowledge experienced         [location in job posting] not applicable description respon. preferred qual. required qual.         statistical data analysis not applicable implied familiarity knowledge experienced         [location in job posting] not applicable description respon. preferred qual. required qual.         spatial data analysis not applicable implied familiarity knowledge experienced         [location in job posting] not applicable description respon. preferred qual. required qual.         qualitative data analysis not applicable implied familiarity knowledge experienced         [location in job posting] not applicable description respon. preferred qual. required qual.         programming languages not applicable implied familiarity knowledge experienced         [location in job posting] not applicable description respon. preferred qual. required qual.         [list programming languages] not applicable [list programming languages]             sharing     data sharing not applicable implied familiarity knowledge experienced         [location in job posting] not applicable description respon. preferred qual. required qual.       preservation     data repository not applicable implied familiarity knowledge experienced         [location in job posting] not applicable description respon. preferred qual. required qual.         data curation not applicable implied familiarity knowledge experienced         [location in job posting] not applicable description respon. preferred qual. required qual.       other     data policy not applicable implied familiarity knowledge experienced         [location in job posting] not applicable description respon. preferred qual. required qual.                           other responsibilities or skills     instruction not applicable implied familiarity knowledge experienced         [location in job posting] not applicable description respon. preferred qual. required qual.         consultation not applicable implied familiarity knowledge experienced         [location in job posting] not applicable description respon. preferred qual. required qual.         public/customer service perspective not applicable description respon. preferred qual. required qual.         faculty status no yes               tenure requirement no yes               research/publishing requirement not applicable description respon. preferred qual. required qual.         liaison to department not applicable description respon. preferred qual. required qual.         [whether depts. are listed] depts. as assigned specific depts. listed not applicable             [list all depts. specified] not applicable [list specific depts.]               research data role focused on specific discipline(s) not applicable description respon. preferred qual. required qual.         [whether disciplines are listed] disciplines as assigned specific discipline listed not applicable             [list all disciplines specified] not applicable [list specific disciplines]               assessment not applicable description respon. preferred qual. required qual.         scholarly communication not applicable description respon. preferred qual. required qual.         outreach not applicable description respon. preferred qual. required qual.         collaboration with other campus units not applicable description respon. preferred qual. required qual.         diversity, equity, inclusion and accessibility not applicable description respon. preferred qual. required qual.           notes                                              * = select all attributes that apply                                                                                                                       synonyms for attributes                                                            doctorate = terminal                                                    advanced = graduate, professional                                                       knowledge = understanding, competent, competence                                    experience = ability, demonstrated ability, aptitude                                        relevant = related, appropriate, comparable                                       commensurate = dependent                                                                                                                hierarchy for location                                                                required qual > preferred qual > responsibilities > description   operational definitions   variable definition when to use when not to use how to use definition source experience, education and salary   mlis degree master's of library or information science degree (often abbreviated mlis, mls, msi, and others) any reference of a master's degree in library and information science graduate degree other than a mlis (or equivalent); undergraduate degree(s) code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description) -   equivalent degree a degree (besides a mlis) that provides a relevant educational background if phase like “equivalent degree” is used to describe the educational background needed for the position additional graduate degree or undergraduate degree; mlis degree code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description) -   equivalent degree level(s) the level of an equivalent degree that provides a relevant educational background if the level of the degree is specified in the phase “equivalent degree” additional graduate degree or undergraduate degree; mlis degree code level of degree: not applicable, bachelor's, master's, doctorate, advanced, not specified -   equivalent degree discipline(s) the discipline of the degree (besides a mlis) that provides a relevant educational background if the discipline of the degree is specified in the phase “equivalent degree” additional graduate degree or undergraduate degree; mlis degree code for all disciplines specified: not applicable, arts & humanities, social sciences, stem, data intensive/data science, business, relevant, not specified -   academic library experience experience working in an academic library any experience working in an academic library (including work as a student) experience working in any setting outside of an academic library 1) code the length of experience (# of years) or length not specified (if not stated, code “no”); 2) code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description) -   research data experience professional experience working with research data, either inside or outside of a library context work experience relating to any aspect of the research data lifecycle, either in an academic library or outside (i.e., experience as a researcher) professional experience working in any other area (either inside or outside of a library); supervisory experience 1) code the length of experience (# of years) or length not specified (if not stated, code “no”); 2) code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description) -   supervisory experience professional experience working as a supervisor or manager supervisory or managerial experience other types of experience 1) code the length of experience (# of years) or length not specified (if not stated, code “no”); 2) code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description) -   additional degree experience or degree (undergraduate or graduate) mentioned in addition to the mlis or equivalent degree experience or degree (undergraduate or graduate) mentioned in addition to the mlis or equivalent degree mlis degree; equivalent degree code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description) -   additional degree level level of degree (undergraduate or graduate) in any discipline other than library and information science level of degree (undergraduate or graduate) in any discipline other than library and information science mlis degree; equivalent degree 1) code level of degree: not applicable, bachelor's, master's, doctorate or phd, advanced; 2) code for all disciplines specified: not applicable, arts & humanities, social sciences, stem, data intensive/data science, business, relevant, not specified -   additional experience additional types of academic or professional experience additional types of academic or professional experience any mentions of degrees code for all experiences specified: not applicable, significant coursework or academic background, subject knowledge, lab or research experience, other, specify: [free text] -   carnegie classification of institution the carnegie classification of the institution which can be found at: http://carnegieclassifications.iu.edu/classification_descriptions/basic.php identify name of the posting institution and then look up the carnegie classification on this website: http://carnegieclassifications.iu.edu/classification_descriptions/basic.php - 1) code this classification by looking up the institution's name on this website: http://carnegieclassifications.iu.edu/classification_descriptions/basic.php; 2) code the level of research activity for doctoral-granting universities or not applicable -   salary information salary information listed in the job posting description of salary information such as “competitive” or “commensurate” numerical salary values; description of any benefits code the salary descriptors used: commensurate, competitive, other, specify: [free text] -   salary range or minimum numerical salary values given exact numerical salary values given (minimum, maximum, range, and others) salary descriptors such as “competitive” or “commensurate”; descriptions of any benefits code exact salary values given (the salary range or minimum) or not applicable - research data activities management             data management process of controlling & managing data, and its associated actions, created during planning and acquisition phases of observation and research any reference to the term “data management” or the actions associated with data management data management plans or other data plans (data sharing plans, data security plans, and others) 1) code degree of complexity sought for this variable (not applicable, implied, familiarity, knowledge, experienced); 2) code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description) adapted from rda term definition tool   data management plans a formal statement describing how research data will be managed and documented throughout a research project and the terms regarding the subsequent deposit of the data with a data repository for long-term management and preservation any reference to data management plans, dmps, data sharing plans or any other type of written data plan required for a grant application data management 1) code degree of complexity sought for this variable (not applicable, implied, familiarity, knowledge, experienced); 2) code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description) casrai dictionary: research data domain discovery and re-use   data discovery process of query or search to find (research) data of interest any reference to locating, discovering or re-using existing datasets (including research data, reference data, government data, and others). other terms could include data access and data identification - 1) code degree of complexity sought for this variable (not applicable, implied, familiarity, knowledge, experienced); 2) code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description) rda term definition tool               collection   data organization process of creating a logical system for storing data files and folders any reference to creating a data file organization system; examples of organization technique: file naming conventions and file structures - 1) code degree of complexity sought for this variable (not applicable, implied, familiarity, knowledge, experienced); 2) code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description) -   data documentation the metadata or information about a data product (e.g., data table, database) that enables one to understand and use the data. such information may include the scientific context underlying the data as well as who collected the data, why the data were collected, and where, when, and how the data were collected; metadata: data about data, data that defines and describes the characteristics of other data any reference to creating documentation (print or electronic format) about data or documenting data (including metadata and metadata standards); reference to cleaning or cleansing research data prior to sharing, publishing, and others; other terms: data quality - 1) code degree of complexity sought for this variable (not applicable, implied, familiarity, knowledge, experienced); 2) code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description) definition of metadata: casrai dictionary research data domain; definition of documentation: dataone best practices primer storage   data storage recording of data on a storage media any reference to how and where to store data, including storage media, storage locations, storage hardware or storage devices data preservation 1) code degree of complexity sought for this variable (not applicable, implied, familiarity, knowledge, experienced); 2) code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description) -   data security measures taken to protect data from unauthorized access, change, destruction, or other threats any reference to data security, preventing unauthorized access, and others. de-identification of data 1) code degree of complexity sought for this variable (not applicable, implied, familiarity, knowledge, experienced); 2) code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description) adapted from society of american archivists' definition analysis   data visualization visual representations of data any reference to data visualization or visualization software (such as tableau, and others.) - 1) code degree of complexity sought for this variable (not applicable, implied, familiarity, knowledge, experienced); 2) code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description) -   general data analysis analyzing data to search for trends or patterns any reference to data analysis that does not specify one or more of the three specific types listed below; quantitative data analysis spatial, geospatial, gis, statistical, or qualitative analysis 1) code degree of complexity sought for this variable (not applicable, implied, familiarity, knowledge, experienced); 2) code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description) -   statistical data analysis using statistics to analyze data for patterns and trends any reference to statistical analysis methods or tests; common tests include anova, chi-square tests, t-tests, factor analysis and cluster analysis. references to common software packages (such as sas, spss, and others) spatial, geospatial or gis analysis 1) code degree of complexity sought for this variable (not applicable, implied, familiarity, knowledge, experienced); 2) code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description) -   spatial data analysis type of geographical analysis which seeks to explain patterns of human behavior and its spatial expression in terms of mathematics and geometry, that is, locational analysis any reference to spatial analysis, geospatial, or gis analysis; mentions of using specific software such as arcgis statistical analysis 1) code degree of complexity sought for this variable (not applicable, implied, familiarity, knowledge, experienced); 2) code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description) dartmouth libraries geospatial information systems research guide   qualitative data analysis the identification, examination, and interpretation of patterns and themes in textual data and determining how these patterns and themes help answer the research questions at hand any reference to qualitative data analysis, including text mining; mentions of qualitative analysis software such as nvivo, dedoose, atlas.ti, and others. any analysis of quantitative data (statistical or spatial) 1) code degree of complexity sought for this variable (not applicable, implied, familiarity, knowledge, experienced); 2) code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description) pell institute evaluation tool kit: analyzing qualitative data   programming languages if the position needs to know one or more computer programming languages (python, c, java, html, and others) specific programming language(s) are mentioned providing programming for the campus community (i.e., planning events) 1) code degree of complexity sought for this variable (not applicable, implied, familiarity, knowledge, experienced); 2) code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description); 3) list the specific programming languages mentioned (if none, use “not applicable”) - sharing   data sharing the practice of making data available for discovery and reuse. this may be done, for example, by depositing the data in a repository or through data publication any reference to sharing or publishing research data (outside of a research team) through a variety of possible avenues (data repository, data journal, and others); mention on assigning persistent identifiers (purls, dois, and others). other terms include data publishing and data dissemination sharing within a research group or collaboration 1) code degree of complexity sought for this variable (not applicable, implied, familiarity, knowledge, experienced); 2) code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description) casrai dictionary: research data domain preservation   data repository a digital archive that provides services for the storage and retrieval of digital content any reference to using, creating, facilitating, and others. a data repository or data archive; other terms could include collecting datasets - 1) code degree of complexity sought for this variable (not applicable, implied, familiarity, knowledge, experienced); 2) code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description) data curation network: data curation terms and activities report   data curation the encompassing work and actions taken by curators of a data repository in order to provide meaningful and enduring access to data. these activities include ingest, appraisal, curation, access and preservation any reference to data curation, curating research data or related data curation activities; other term: data curator - 1) code degree of complexity sought for this variable (not applicable, implied, familiarity, knowledge, experienced); 2) code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description) data curation network: data curation terms and activities report               other   data policy an organization’s stated data/information management processes designed to assist and protect research data assets any reference to data policies (a library's policies, university's policies, funder policies, and others) including data management plan policies, deposit policies, intellectual property policies, data curation policies, and others - 1) code degree of complexity sought for this variable (not applicable, implied, familiarity, knowledge, experienced); 2) code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description) adapted from rda term definition tool               other responsibilities or skills   instruction teaching (online or in-person) researchers about any research data management activities (including the variables listed in the research data activities section of this codebook) reference to teaching (in-person or online) sessions, workshops, courses, and others on research data management; creating or maintaining tutorials, online modules, and others for asynchronous instruction instruction for liaison, scholarly communication or other non-research data roles/responsibilities 1) code degree of complexity sought for this variable (not applicable, implied, familiarity, knowledge, experienced); 2) code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description) -   data consultation a meeting in which a data librarian or research data staff and patron discuss research data management issues and potential solutions any reference to providing consultations or reference interactions for patrons to discuss research data management issues - 1) code degree of complexity sought for this variable (not applicable, implied, familiarity, knowledge, experienced); 2) code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description) -   public/customer service perspective mindset focused on providing high quality public/ customer service description of a mindset focused on providing high quality public/ customer service - code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description) -   faculty status the position has faculty status at the institution (as opposed to being staff, academic staff, and others) faculty status is mentioned tenure-track position code if this variable appears in the job posting (yes, no) -   tenure requirement if this position is a tenure-track position at the institution tenure-track is mentioned status at the institution (faculty, staff, academic staff, and others) code if this variable appears in the job posting (yes, no) -   research/publishing requirement if the successful candidate needs to have a demonstrated record of research/publishing (books, book chapters, journal articles, and others) or they demonstrate the ability to do research/ publish in the future any mention that scholarly research/ publishing is a requirement of the position publishing data for patrons; need to know about current topics in scholarly communication code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description) -   liaison to department this position will serve as the library liaison to one or more departments or units at the institution, in addition to their research data responsibilities; provide reference/ research assistance, instruction, outreach, collection development, and others liaison activities or work are mentioned (either with or without naming specific departments or units that the position will be the liaison to) collaboration with other campus departments/ units; research data role focused on specific disciplines 1) code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description); 2) whether specific departments are listed in the job posting (depts. as assigned, specific depts. listed, not applicable); 3) list the specific depts (free text, not applicable) -   research data role focused on specific discipline(s) this position focuses on the research data management needs of specific disciplines, schools, colleges, and others this position focuses on the research data management needs of specific disciplines, schools, colleges, and others liaison to department; collaboration with other campus departments/units 1) code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description); 2) whether specific disciplines are listed in the job posting (depts. as assigned, specific depts. listed, not applicable); 3) list the specific disciplines (free text, not applicable) -   assessment if the position will be involved in assessment projects, relating to the research data responsibilities assessment is mentioned relating to research data responsibilities (such as assessment of patron satisfaction with the library's research data services) assessment activities related to responsibilities outside of research data responsibilities (such as service work, liaison work, and others) code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description) -   scholarly communication if the position needs to know about the current landscape of scholarly communication mentions of knowing about scholarly communication if the position required to publish code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description) -   outreach if the position will be conducting outreach to the campus community (outside of the library) to advertise the library's research data services mention of outreach, marketing or advertising the library's research data services outreach for responsibilities outside of research data responsibilities (such as liaison activities) code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description) -   collaboration with other campus units if this position will collaborate with campus units outside of the library (such as it, research office, provost's office, and others) on research data projects collaboration with campus units outside of the library liaison duties to campus departments/units code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description) -   diversity, equity, inclusion and accessibility if the applicant needs to know about and recognize the importance of these issues within a library or university any mention of applicant being committed or recognizing the importance of diversity, equity, inclusion, and accessibility (such as having to submit a diversity statement as part of the application or having a commitment to fostering these on campus) language about the university's commitment to diversity, equity, inclusion, and accessibility code where it occurs in the job posting (required qualifications, preferred qualifications, responsibilities, description) -     sources   sources of some variables   hall-ellis (2005).   hall-ellis (2006).   chen, h. l., & zhang, y. (2017). educating data management professionals: a content analysis of job descriptions. the journal of academic librarianship, 43(1), 18-24.   xia & wang (2014).    indiana university. (2017). institution lookup. in the carnegie classification of institutions of higher education. retrieved from https://carnegieclassifications.iu.edu/lookup/lookup.php       sources of some operational definitions   dataone best practices primer https://www.dataone.org/sites/all/documents/dataone_bp_primer_020212.pdf research data alliance (rda) term definition tool https://smw-rda.esc.rzg.mpg.de/index.php/main_page casrai dictionary research data domain http://dictionary.casrai.org/category:research_data_domain society of american archivists glossary https://www2.archivists.org/glossary/terms dartmouth libraries geographical information systems research guide https://researchguides.dartmouth.edu/gis/spatialanalysis pell institute evaluation toolkit: analyzing qualitative data http://toolkit.pellinstitute.org/evaluation-guide/analyze/analyze-qualitative-data/ data curation network: data curation terms and activities report https://conservancy.umn.edu/bitstream/handle/11299/188638/definitionsofdatacurationactivities%20%281%29.pdf?sequence=1&isallowed=y     appendix c supplementary table   summary of mentions of 19 research data management activities: a) degree of complexity sought and b) location in the job posting.   a)   experience knowledge familiarity implied not applicable general data management 58 31 10 55 26 statistical data analysis 45 12 10 9 104 general data analysis 39 7 14 18 102 data repository 34 32 17 38 59 data curation 33 27 1 40 79 data visualization 31 7 7 29 106 data documentation 25 33 10 28 84 spatial data analysis 24 10 7 5 134 qualitative data analysis 21 3 7 5 144 programming languages 21 3 7 5 144 data management plans 18 13 5 40 104 data discovery 13 11 6 67 83 data sharing 7 7 7 64 95 data policy 6 2 3 38 131 data storage 2 6 1 22 149 data organization 1 1 0 17 161 data security 0 3 3 11 163     b)   required qualifications preferred qualifications responsibilities description not applicable data repository 51 32 33 5 59 statistical data analysis 45 23 5 3 104 data documentation 38 30 21 7 84 programming languages 33 28 0 0 119 data visualization 30 15 26 3 106 data management plans 24 12 33 7 104 general data management 24 73 51 6 26 spatial data analysis 24 17 3 2 134 general data analysis 18 42 15 3 102 data curation 17 44 38 2 79 qualitative data analysis 13 18 4 1 144 data sharing 8 12 50 15 95 data discovery 7 23 57 10 83 data policy 6 5 31 7 131 data security 3 3 8 3 163 data storage 2 7 13 9 149 data organization 0 2 6 11 161     evidence based library and information practice 2012, 7.1 7 evidence based library and information practice article the potential of generic social outcomes in promoting the positive impact of the public library: evidence from the national year of reading in yorkshire carolynn rankin senior lecturer faculty of health and social sciences leeds metropolitan university leeds, united kingdom email: c.rankin@leedsmet.ac.uk received: 15 sept. 11 accepted: 20 nov. 11 2012 rankin. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to present the development, planning and implementation of a qualitative research project on the impact of national year of reading in yorkshire. methods – generic social outcomes (gso) were used to develop a theoretical framework. data were gathered via in-depth interviews and focus groups with national year of reading (nyr) steering group partners in calderdale and north lincolnshire, selected as the two case study authorities. the use of maxqda computer-assisted qualitative data analysis software (caqdas) enabled data and coding structures to be stored and facilitated comparison in this longitudinal study. results – the findings using the gso framework show considerable evidence of nyr related activities in supporting the three first-tier social outcomes: ‘stronger and safer communities’, ‘health and well-being’ and ‘strengthening public life.’ conclusion – the gso framework can be used by practitioners to help provide evidence for how public libraries contribute to diverse agendas and demonstrate their value to the community. public library authorities can use this evidence for planning and for advocacy with a range of audiences including local and central government. evidence based library and information practice 2012, 7.1 8 introduction to the national year of reading evaluation project the 2008 national year of reading (nyr) campaign in the uk was about celebrating and encouraging reading in all its forms. its aim was to promote reading in the family and beyond, and to help to build a nation of readers. the nyr campaign ran from january to december with organizations and local authorities asked to pledge and plan their support between january and march. the department for children schools and families (dcsf) commissioned the national literacy trust (nlt), with lead partner the reading agency, to run the nyr campaign. delivery of the nyr was launched in april 2008 to support ongoing work to achieve national literacy targets, engage parents and families in reading with their children, and develop adult literacy. the key values identified for the yearlong social marketing campaign were: • impact both personal and social • celebration positive, enjoyable experiences • diversity of reading experiences and communities • participation co-production of the year with communities and partners • creativity in development and delivery • legacy to create a lasting change in lives and systems the strategy was to create a network of partnerships to provide engagement with the nyr campaign, particularly with education and libraries. the campaign intended to stimulate community-based reading activities and to help publicize and reinforce projects already underway. public libraries were seen as central to the nyr campaign and there was a 100% sign up from the 149 public library authorities (thomson, 2009). the campaign needed to focus on ways of attracting people to start reading or develop their existing skills. a national headline target driven media campaign promoted the year based on several key messages: • everything begins with reading • join your library – everything you could wish to read, for free • reading anything, anytime, anywhere is good a partnership of museums, libraries & archives (mla) yorkshire, renaissance yorkshire, and arts council england (yorkshire region), jointly commissioned a longitudinal evaluation of the social impact of the nyr in yorkshire. researchers at leeds metropolitan university were appointed by mla yorkshire in august 2008 to undertake evaluation research for the clients. this paper outlines the development, planning and implementation of this qualitative research project, and discusses the impact of nyr on the organizations that delivered the campaign and their work with target groups. literature review this select review of the literature will consider the role of the public library as a key partner in the nyr campaign in 2008, and will discuss the importance of literacy in contemporary society. the final section will consider the challenges of measuring the value and impact of public library services and will focus on the mla generic social outcomes (gso) framework used in this evaluation project. the public library and the power of partnerships there is growing interest in the perceived social impact of the public library in contributing to the social cohesion and development of the community served. it is suggested in the literature that public libraries have the potential to act as generators of social capital since they are open to all and provide community meeting spaces and a safe place for people to meet (johnson, 2010; varheim, 2009; bourke, 2005). communities give purpose to libraries and good library services will put the community at the heart of developing and delivering targeted services, engaging with people and responding to their needs. in a guide to evidence based library and information practice 2012, 7.1 9 providing multicultural services the international federation of library associations and institutions (ifla) notes that in reflecting the needs of the population they serve, libraries impact greatly upon people’s lives as learning centres, cultural centres and as information centres (ifla, 2009). this has implications for the role of libraries in supporting family learning, and pateman and vincent (2010) provide a well-argued justification for extending the social justice role of the public library and the provision of needs-based services. koontz and gubbin (2010) note that the public library is the prime community access point designed to respond to a multitude of ever-changing information needs. the challenge for librarians is to implement policies in libraries and work with partner organizations to achieve effective delivery of services. public libraries in the uk have been at the centre of their local communities since the 19th century, providing services to reflect the diversity of the population they serve (brophy, 2007; goulding, 2006; mcmenemy, 2008). the public libraries and museums act 1964 came into force in april 1965 and since then the provision of a public library service is a statutory duty for local councils. in order to meet key legal requirements, a local library service must serve both adults and children and provide value for money, working in partnership with other authorities and agencies. the cilip guidelines, what makes a good library service?, say that good library service should provide a positive experience for local people, and that it will meet key policy objectives by providing a positive future for children and young people; strong safe and sustainable communities; equality, community cohesion and social justice; health improvements, and well-being (cilip, 2010). public libraries were identified as essential partners in the nyr campaign strategy and all 149 library authorities signed up for active involvement within a very short period of notice. the inclusion of public libraries as key partners built on three themes identified as priorities in the 1998 campaign a decade earlier: changing attitudes to reading among different audiences, the role of libraries in developing readers, and working in partnership (national literacy trust, 1999). dolan and ayub (2011) note the nyr campaign involved many partners and “it gave libraries a bigger stage on which to perform” (p. 85). the inclusion of public libraries as key partners acknowledged that public libraries make a measurable and substantial contribution to local economies and help to bridge social divides. they support well-being; encourage reading; spread knowledge; contribute to learning and skills; and help to foster identity, community and a sense of place for people of all ages, backgrounds and cultures (mla, 2008). the public library is also seen as a trusted community resource providing a universal entitlement to the skills and joy of reading, essential information, learning and knowledge at all stages of life and involvement in the social, learning and creative life of the community (dolan, 2007; mla, 2008; mla, 2010). why the nyr 2008 campaign? the overall aim of the nyr campaign was to promote reading in the family and beyond, and to help to build a nation of readers. this involved encouraging reading both for pleasure and as a means of improving learning, achievement and individual prospects. the campaign focused on ways of attracting people to start reading or develop existing skills. competence in literacy is essential for life in contemporary society. moreover, it dramatically contributes to people’s emotional wellbeing, mental health, and economic success. the case for reading for pleasure has been set out in research by bodies such as the organisation for economic cooperation and development (oecd, 2002), which showed that reading for pleasure is more important for children’s educational success than their family’s socio-economic status, and that improvements in literacy, at any point in life, can have a profound effect on an individual. evidence based library and information practice 2012, 7.1 10 literacy changes lives the national literacy trust (nlt) has addressed the national literacy challenge as a priority and was the lead organization for delivering the nyr in 2008. the nyr campaign strategy was to create a network of partnerships that would provide engagement and stimulate community-based reading activities. low literacy levels are a barrier to social justice, producing social, economic and cultural exclusion that scars communities and undermines social cohesion. in 2007 a u.s. perspective on the wider benefits of literacy was reported in to read or not to read. the nlt compiled an equivalent document for england and the literacy changes lives report (dugdale and clark, 2008) draws on a number of sources that have used longitudinal studies that track subjects from birth. this research presents evidence that literacy has a significant impact on person’s happiness and success, and gives a clear indication of the dangers of illiteracy as well as the benefits of improving literacy for the individual, the community, the workforce and the nation (2008). literacy changes lives brings together evidence of the wider benefits of literacy by looking at five key personal areas: economic well-being; aspirations; family life; health; and civic and cultural engagement. in every one of these areas, those with poor literacy had significantly worse outcomes. literacy is not just about reading ability but is a skill central to many facets of life. a unique combination of benefits – measuring the value of the public library service although there is much in the library and information studies (lis) literature about what the public library service provides to the user community, it is more difficult to present this in terms of perceived value. in the uk there is still a paucity of convincing published evidence that is persuasive and likely to be used at a political level. a commissioned report undertaken by burns owens partnership (bop) identifies a lack of qualitative in-depth research that analyses the specific nature of interactions that take place in libraries (dcms/bop consulting, 2009). a recent publication from the paul hamlyn foundation reports on what has been learned from the reading and libraries challenge fund. in the introduction to this review of 60 grants made over several years (an investment of £3.7 million), helen carpenter writes, public libraries have tended to be slow to react to this changing environment. this may be in part because their operating culture is not used to expressing their role in terms of policy objectives emerging in other sectors, such as children’s and youth services. it may also be that libraries have not sufficiently entered the consciousness or collective imaginations of policy makers in other domains, as a resource that can help them achieve their objectives. (carpenter, 2010, p. 3-4) using the generic social outcomes (gso) framework, the national year of reading research project looked for evidence of the public library offering in two contrasting public library authorities. powell (2006), in providing an overview of evaluation research, says that it should enhance knowledge and decision making and lead to practical applications. one way of exploring the challenging question of how libraries contribute to the cohesion and development of their communities is to use the concept of social capital. measuring impact evaluation is more difficult than collating statistics. markless and streatfield (2006) remind us of the problems of getting sidetracked and looking at activities and processes when trying to evaluate impact, rather than concentrating on what difference a public library offering makes. marshall (2007), writing about the future of value and impact studies, identifies two major strands of research on the value of libraries. the first is composed of economic studies that focus on return on investment, cost benefit analysis and other monetary measures. this can demonstrate efficiency of libraries in evidence based library and information practice 2012, 7.1 11 comparison with other government or industry sectors but the measures can appear to be distant from actual services provided to users. she identifies a second strand that focuses on the use of information by particular groups, e.g. library user studies. this is seen as particularly relevant in the health environment where outcome measures for particular populations and evidence based practice are gaining prominence, but there are also messages for public library practitioners. evidence based practice is a concept that originated within healthcare but is migrating to other fields. it involves applying the results from rigorous research studies to professional practice in order to improve the quality of services to clients (booth and brice, 2004) rooney-browne (2011) undertook a literature review of existing qualitative and quantitative evaluation methodologies for demonstrating the value of public libraries in the uk. this report, commissioned by the cilip library and information group, provides an overview of current methods for measuring performance. rooney-browne found that in the uk considerable emphasis is placed on the importance of measuring the performance of public libraries using traditional statistical methods, with the more complex direct and indirect benefits often being overlooked. so what is the value of the public library service and how might this be measured? how can we “measure” that service in a way that is meaningful to stakeholders? the particular challenge is that value is a psychological construct and there are implications for the reflective practitioner in professional practice. the next section will discuss the gso framework developed as a tool to help demonstrate how public libraries contribute to diverse agendas and demonstrate their value to the community. public library authorities can use this evidence for planning and for advocacy with a range of audiences including local and central government. generic social outcomes as an evidence framework public libraries are recognised as being good at reaching “hard-to-reach” groups and good at building partnerships based on reading and family learning. however, there are critical challenges ahead in finding the resources to continue delivering publicly funded library services in times of economic strictures. public libraries may be effective and reliable partners in providing services that the public appreciate, but they are rarely recognized as the lead agency and as such do not yet have the same political voice in promoting their professional worth. this presents a particular dilemma, as rooney-browne (2011) notes: “our ability to produce social value is considered by some to be one of our greatest commodities” (p. 29). mla funded research led to the development of generic social outcomes (gsos) – a national framework developed and piloted by the burns owen partnership (bop) in 2005. this national gso framework was developed as a combination of a bottom up process of developing the framework with practitioners and a top down process of aligning the sector's potential social contribution with key drivers of government policy through consultation with external bodies. the result was the creation of three gsos: stronger and safer communities, strengthening public life, and health and well-being. each of the tier 1 gsos is further subdivided into a set of social outcome themes (tier 2) as shown in table 1. the national gso framework was developed by the mla to help museums, libraries and archives show evidence of the benefit of their services, and is a way of aligning the sector's potential social contribution with key government policy drivers. the ability to demonstrate achievement of outcomes alongside local priorities will give the sector strategic influence and help advocate for resources (mla, 2007). linley and herman (2008), in a report for mla yorkshire, argue that the gso framework is useful for advocacy with potential partners and for funding applications. the gso framework can also be seen as a tool enabling policy makers and practitioners to speak the same language. practitioners can evidence based library and information practice 2012, 7.1 12 then make the case for impact using terminology that is meaningful to local and national government. the mla suggests that the gsos can also support service improvement by helping practitioners to: • plan projects, set objectives and assess delivery against them • develop best practice • fit with the focus on shared outcomes for local people measured by comprehensive area assessments • develop reflective practitioners • encourage partnership working through shared priorities aim and objectives of the nyr evaluation project in yorkshire the aim of this longitudinal research was to investigate the efficacy of the national year of reading campaign in yorkshire as it relates to the place shaping and social inclusion targets of the nyr. the development of the project brief and specification for the consultancy project was coordinated by mla yorkshire and required that qualitative research should incorporate use of the mla’s gsos. the overall objectives were to investigate the impact of the nyr in two contrasting local authorities in relation to: 1) target beneficiaries, and; 2) partnership and cross departmental working. methods the nyr campaign was delivered in 15 local authorities in yorkshire and the humber region. in consultation with the leeds metropolitan university research team, the yorkshire steering group decided to sample two contrasting authorities, one predominantly rural, and the other urban; there was no intention to undertake a comparison of the two case study authorities. calderdale libraries and north lincolnshire libraries were invited to participate, along table 1 tier 1 social outcomes framework and tier 2 social outcomes themes tier 1 1 stronger & safer communities 2 strengthening public life 3 health & well-being tier 2 1.1 improving group and inter-group dialogue and understanding 2.1 encouraging and supporting awareness and participation in local decision-making and wider civic and political engagement 3.1 encouraging healthy lifestyles and contributing to mental and physical well-being 1.2 supporting cultural diversity and identity 2.2 building the capacity of community and voluntary groups 3.2 supporting care and recovery 1.3 encouraging familial ties and relationships 2.3 providing safe, inclusive and trusted public spaces 3.3 supporting older people to live independent lives 1.4 tackling the fear of crime and anti-social behaviour 2.4 enabling community empowerment through the awareness of rights, benefits and external services 3.4 helping children and young people to enjoy life and make a positive contribution 1.5 contributing to crime prevention and reduction 2.5 improving the responsiveness of services to the needs of the local community, including other stakeholders (mla north east) evidence based library and information practice 2012, 7.1 13 with the departments and organizations involved in the nyr steering groups of those two authorities. project planning and timescale the research was designed as a longitudinal project in two phases. contact with the key senior library personnel in calderdale and north lincolnshire was initiated by the regional participation and inclusion advisor for mla yorkshire. for the researchers, this direct route to participants was of particular advantage due to the short timescale for phase one. a project initiation meeting was held in september 2008 to communicate key messages about the project objectives, agree on timescales, and facilitate a partnership approach to undertaking a detailed and intensive analysis of nyr activities. in phase two the researchers re-established contact with the respondents. a variety of research methods were used to collect qualitative data from key stakeholders in north lincolnshire and calderdale and the project budget included funding for a research assistant to assist with the transcription and analysis of the data sets. phase one (october-november 2008) • semi-structured interviews with key library staff • focus group interviews with nyr steering group partners through written responses to key questions • information gathered from documentation and publicity materials provided by each case study authority phase two (may-june 2010) • semi-structured interviews with key library staff • information gathered from documentation and publicity materials provided by each case study authority the theoretical framework for the nyr evaluation project in yorkshire three levels of analysis were used to evaluate the nyr impact issues in the two case study authorities. the mla generic social outcome framework (gsos) and the national indicators (ni) from the new performance framework for local authorities & local authority partnerships (department for communities & local government, 2007) were specified in the project brief. a third level, issues raised by the respondents, was added by the research team, as the “voice of the practitioner” was considered an important aspect of this evaluation project. issues raised by respondents during interviews were coded to enable the capture of data about individual concerns, interests, experiences and reflections. this paper will focus on the use of the gso framework. the generic social outcomes framework a requirement of the research brief was to use the generic social outcomes (gsos). as discussed in the literature review section the gso framework has been developed by the mla to help museums, libraries and archives to deliver against key agendas and maximise their contribution to communities. the framework is built around three key stands – “stronger and safer communities,” “health and well-being,” and “strengthening public life” – providing a means by which museums, libraries and archives can evidence their contribution to outcomes (burns owens partnership, n.d.). the gsos are seen as a key tool that practitioners can use because of the increased emphasis on outcomes as well as outputs. an overview of the data collection strategies the study was primarily qualitative, involving the gathering, analysis, interpretation and presentation of narrative information. an account of the project methodology has previously been published by rankin, brock and matthews (2009). a variety of research methods were used to gather the data. in order to gain depth and rigour of analysis, both the method and the process of analysis were triangulated. bryman (2008) defines triangulation as the use of more than one method or source of data in the study of social phenomena so that findings may be evidence based library and information practice 2012, 7.1 14 crosschecked. this was undertaken through combinations and comparisons of multiple data sources, data collection and analysis procedures. the researchers gained different perspectives on outcomes by gathering the views of staff, partners and users, as well as evidence from other sources using desk research. data was collected through focus group interviews with nyr steering group partners, focus group written responses to key questions, and individual interviews. a range of supporting material in printed, digital and visual format was provided by the case study authorities (this material was readily available and not created specifically for the purposes of the evaluation research), and by using a variety of documentation and publicity materials from each authority. interviewing interviews are a powerful data collection technique because they use one-to-one interaction between researchers and interviewees (teddlie and tashakkori, 2009). it was decided that semi-structured interviews would be the most appropriate method to gather data in support of the research questions (denscombe, 2003; pickard, 2007) and in-depth, face-to-face semi-structured interviews were conducted with key senior library staff in the two case study authorities. an interview schedule was devised covering key questions about the target beneficiaries and partnership and cross-departmental working (see appendix 1 for the phase one schedule). each local authority library setting was visited so that respondents were interviewed in their own work environments. the interviews were digitally recorded with the permission of the interviewees and this facilitated transcription of the data. focus group interviews with nyr steering group members in phase one, members of the nyr steering groups in calderdale and north lincolnshire were invited to take part in a group discussion. in social research, the focus group methodology is used extensively; there is an emphasis in the questioning on a particular fairly tightly defined topic, and on interaction within the group and the joint construction of meaning. this is an effective and efficient way of gaining qualitative data through engaging a small number of people in an informal group discussion (krueger and casey, 2009; bryman, 2008; flick, 2009). the group discussions focussed on a number of key questions designed to encourage discussion about partnership working and nyr legacy issues and the representatives of the partner organizations were invited to record their personal views, experiences and opinions on the nyr activities. data gathered during the group interview discussions were digitally recorded for transcription, analysis and coding, as were written responses to key questions generated during the session by group members working in pairs. as noted by bryman (2008) the group interaction was seen as an important component of the discussion. analysing the data – the value of computerassisted qualitative data analysis software (caqdas) qualitative data from interview transcripts can be difficult to manage and needs to be well organised and structured to allow for coding and theory building. analysing data involves segmenting text and isolating items into categories to look for patterns. the computerassisted qualitative data analysis software (caqdas) maxqda 2007 was used to facilitate the organization of the data for coding, sorting and retrieving as well as for theory building. the advantage of caqdas is that it can be effective in handling large volumes of coded data, enabling the researcher to track, retrieve, map, chart, and redefine it quickly and accurately (silverman, 2010). appropriate coding of the qualitative data is an important part of the interpretative process and maxqda 2007 enabled the researchers to code and validate the data as required by mla (yorkshire) clients by using the gso framework and the psa national outcome and indicator set. a third level of coding was developed to identify issues raised by the interviewees. it is the researcher who must determine the main areas for the analysis of the research and interpret the data – not the evidence based library and information practice 2012, 7.1 15 software package. maxqda 2007 proved to be a very effective tool that enabled the nyr data to be readily accessible and continually re-interrogated. results – the nyr evaluation the project brief and specification for the evaluation project required the researchers to investigate the impact of the nyr in two sample local authorities focusing on target beneficiaries and partnerships, and cross departmental working. this paper does not identify specific locations but rather presents generic findings and selected comments from the phase one and phase two data as evidence. one of the key issues highlighted by the analysis of the qualitative data is that using the gsos is an important way in which the local public library can not only demonstrate their value in the community, but also show how they contribute to other agendas not immediately apparent such as in the areas of health, well-being, equality, and improving cohesion. in terms of nyr related activity the analysis of data from the first phase suggested 6 key themes: • improving services and sharpening the focus of what is on offer • working in partnership and strengthening partnerships • dealing with challenges • the importance of activities and events • stronger communities – with a particular emphasis on improving group and inter-group dialogue and understanding and supporting cultural diversity and identity • legacy of the nyr – what could be sustained and embedded these six themes were still in evidence some eighteen months later in phase two but the emphasis had rather shifted to: • the differences in rural/urban experiences and the squeeze on resources • initiating and consolidating partnerships • the importance of reading for social engagement the findings using the gso framework show considerable evidence of nyr related activities in supporting the three first tier social outcomes of “stronger and safer communities,” “health and well-being,” and “strengthening public life.” the next section provides an overview of the findings with commentary on the related second tier themes. as it is interesting to hear the practitioner voice, sample quotes from the both phases of the project are used to help present the evidence. see table 1 for the structure of the tier 1 social outcomes framework and tier 2 social outcomes themes. gso stronger and safer communities for the project objectives “target beneficiaries” and “partnership and cross departmental working” there is significant evidence for the stronger and safer communities second tier social outcome themes. there is very strong evidence for “improving group and inter-group dialogue and understanding” that came from all layers of analysis and from all participating groups. the coding and analysis demonstrated that this was the strongest outcome overall from the nyr activities. “supporting cultural diversity and identity” was also identified as being a strong theme throughout the data collection. a range of different groups were identified as key target groups and there was varied cultural diversity within the target groups in both case study authorities. the manga event brought all sorts of people in to the library who have never been before. i want to go one step further and consult those people about using our service. (phase one) one of our big successes is the manga. we’d had one before we last saw you now we’ve had three – we get a group of about 150 supposedly hard to reach people… some of those people come back, i see some of the men twice a week sitting on our sofa reading manga and i know they are from that event. (phase two) evidence based library and information practice 2012, 7.1 16 “encouraging familial ties and relationships” was also strongly demonstrated as there was interest in supporting family groups. some projects focused on activities that were aimed at hard to reach groups, some added value to regular users, while others were new ideas inspired by the opportunities of the nyr partnerships. we are doing a lot of great stuff, including breakthrough initiatives for some groups e.g. making materials for homeless people. we have never done that before. (phase one) we are now thinking about legacy. we don’t just want to run events and then stop it at the end of nyr. we are also thinking about community engagement so i want to use some of the people we have made contact with to improve our community engagement in future. (phase one) gso strengthening public life a very strong second tier theme was improving services. this featured in both phases of the project, and this is perhaps to be expected given the focus public libraries have on designing services to meet the needs of their communities. we have various targets and agendas that drive us. the leading agenda is the equalities impact assessment, which in library terms means identifying new communities and providing a service for them. the nyr has pushed towards a service located outside the library. were it not for the nyr we would have contented ourselves with providing stock within the library and the outreach might not have happened. the actuality is that the nyr gave us the steering group that provided us with the contacts that we needed to do that. (phase one) from now people who want to rent council houses will have to bid for them online … so we are doing open planned sessions so people who don’t have computer skills or access to the internet at home can come to the library… and we can show them how to do it. (phase two) there was also evidence that partnership projects were being used to reach target beneficiaries. it has been through the partnerships we have been working with – housing benefits, sheltered housing, and the hospital library – we find it hard to reach those target groups. those kinds of people don’t tend to come into libraries. we need to make contact with the people who work with them on a regular basis. (phase one) other second tier themes that generated an evidence base were “safe, inclusive and trusted public spaces” and “building the capacity of community and voluntary groups.” we are taking out of this year a commitment to changing the pattern of city and local libraries, where most things happen. we will hopefully soon have two more places and hopefully we will have more. in terms of redevelopment, a place becomes a hub if it becomes a place where the community gets used to expecting exciting and valuable reading events, workshops, festivals on a regular basis. i hope that will emerge from the nyr. (phase one) there is space in the library for events and activities. one of our milestones is to make contact with community based organizations that will put on community events in collaboration with us using the library space. (phase two) gso health and well-being within the health and well-being gso the strongest second tier theme was “helping children and young people to enjoy life and make a positive contribution.” “encouraging healthy life styles and contributing to mental and physical well-being” also provided strong evidence. the analysis of the findings show the contribution libraries can make to healthy evidence based library and information practice 2012, 7.1 17 lifestyles and mental and physical welfare. a number of projects and partnerships have lasted beyond the year of nyr; they have become embedded into the organizational planning and are no longer dependent on individuals. we are reaching the homeless, new immigrants, people with mental health problems – the potential is much more than we are doing. i hope that we will maintain the commitment to be creative. (phase one) we have staged a third poetry competition aimed at adult learners across the region. there were lots of entries. people, who had never been in a library read out poems in front of a whole audience. (phase one) the other ongoing success is the literary festival as that started win the nyr… and we run workshops for people who want to improve their writing skills, these are hugely popular and developed out of the nyr... and the whole reader profile – we have 40 readers groups now. (phase two) …the health theme might not be an obvious theme for a library but there is an attempt to work with the local pcts … another initiative is called nhs choices… if someone who comes in wants to look up a health condition we actually log them on and show them how to use it. (phase two) discussion the nyr has certainly had an impact on the visibility of reading in communities and the analysis of the evidence from the yorkshire case study authorities gives an indication of the enthusiasm and professionalism involved in delivering the campaign. public libraries played a key role in delivering nyr campaign outcomes. stronger and safer communities and the improvement of group and inter-group dialogue and understanding proved to be some of the most significant issues resulting from the nyr in yorkshire evaluation. the interviewees spoke about provisions that targeted adults, young people and children; diverse ethnic communities; migrant workers; specific workforce groups; hard to reach groups such as the homeless; and those with mental health needs. the case study data showcases a wide range of activities and events offered under the banner head of the nyr campaign. the voice of the practitioner was considered an important aspect of the qualitative research in this evaluation project as issues raised by respondents during interview were coded to enable the capture of data about individual concerns, interests and experiences. across all the data there were noteworthy issues that are not addressed by the gsos. for any evaluation of library services to be effective it is important to elicit the voices of those engaged in the management and delivery of the services and the development of new initiatives. the voice of the practitioner needs to be heard and taken into account by those who are involved in policy making. the interviewees felt strongly that partnerships were a key aspect of the nyr activities. the steering groups in both case study local authorities had worked successfully in different ways. these were a very positive outcome in the two locations, driven by highly motivated practitioners who were using the nyr as a focal point for drawing together their work, showcasing new projects and partnerships and refreshing ideas about existing provision. but what about the value of the evidence? in this project the gsos provided a framework that enabled the research team to map the evidence about the nyr campaign to show how public libraries contribute to diverse agendas and demonstrate their value to the community. this helps to reinforce the case for an evidence base that is not just regarded as anecdotal. the university-based research team used computer-assisted qualitative data analysis software to store the narrative data and coding, but it is not essential to use this complex software with the gso framework. to help encourage public library practitioners to use such tools, the mla have provided an online gso indicator bank resource which evidence based library and information practice 2012, 7.1 18 includes templates for evidence gathering and access to guidance and case studies. in light of today’s economic climate there is a need for public libraries to be more strategically aware and to be seen to take credit in partnerships so that they are recognized for their impact. gsos can provide a framework for enabling public libraries to show social return on investment and how they can contribute to diverse agendas and demonstrate their value to the community. however, there are issues of translation and interpretation that need to be addressed by practitioners in order to communicate effectively with political stakeholders and in partnership with other organizations and agencies to help achieve strategic objectives. there is a critical need for practitioners to publish and disseminate information about the impact of sustainable projects that benefit their communities (rankin, 2010). public libraries can use gso evidence for advocacy with a range of audiences including local and central government. reflective practitioners must consider that setting outcomes should be at the heart of service planning and improvement. use of an evidence based approach such as the gso framework may contribute to decision making in professional practice in these challenging times. the nyr in 2008 provided an opportunity for public libraries to be in the political limelight for a time as key partners in achieving success for the government sponsored campaign. durcan sees this current time of imminent new austerity as “a good time to reinforce our traditional role and to drive and exploit our potential as the free street-corner, village, town and city-centre access points to positive activity, recreation, skills support, information and knowledge” (durcan, 2011, p. 328). conclusion this paper discussed an evaluation of the national year of reading in yorkshire conducted by leeds metropolitan university in response to a brief from museums, libraries and archives, yorkshire. the national year of reading in 2008 provided an opportunity for public libraries to be in the political limelight for a time, as key partners in achieving success for the government sponsored campaign. libraries need to become more strategically aware and need to be seen to take the lead so that they are recognized for their impact in delivering community based projects, particularly in light of today’s economic climate. this research project has shown how the generic social outcomes framework can be used to help demonstrate how public libraries contribute to diverse agendas and show their value to the community. the challenge is for librarians to drive forward an agenda of demonstrating impact through value-added projects. acknowledgments thanks are due to the funders mla yorkshire, renaissance yorkshire and arts council england, yorkshire; the senior staff at calderdale and north lincolnshire libraries who were interviewed, and members of the nyr steering groups in those authorities for their participation in the focus group discussions. wikireadia is a searchable and editable encyclopaedia of good practice in reading, writing, listening and speaking. it was originally created as part of the 2008 national year of reading and is now managed by the national literacy trust. http://www.wikireadia.org.uk/index.php?title= 2008_national_year_of_reading references booth, a. & brice, a. (eds.) (2004). evidence based practice for information professionals: a handbook. london: facet publishing. bourke, c. (2005). public libraries: building social capital through networking. australasian public libraries and information services (aplis), 18(1), 7175. brophy, p. (2007). the library in the twenty-first evidence based library and information practice 2012, 7.1 19 century. london: facet publishing. bryman, a. (2008). social research methods. oxford: oxford university press. burns owens partnership. (n.d.). mla: social outcomes for museums, libraries and archives. prototype social outcomes framework. retrieved 1 feb. 2012 from http://www.mla.gov.uk/policy/comm unities/gso_overview carpenter, h. (2010). leading questions: learning from the reading and libraries challenge fund. retrieved 1 feb. 2012 from http://issuu.com/paulhamlynfoundatio n/docs/ phf-fund-report_leadingquestions?viewmode=magazine&mod e=embed chartered institute of library and information professionals (2010). what makes a good library service? guidelines on public library provision in england for portfolio holders in local councils. retrieved 1 feb 2012 from http://www.cilip.org.uk/getinvolved/advocacy/publiclibraries/documents/ what_makes_a_good_library_service_ cilip_guidelines denscombe, m. (2003). the good research guide for small scale research projects. buckingham: open university press. department for communities and local government. (2007). the new performance framework for local authorities and local authority partnerships: single set of national indicators. london: dept. for communities & local authorities. department for culture, media and sport/burns owens partnership. (2009). capturing the impact of libraries. retrieved 1 feb 2012 from http://webarchive.nationalarchives.go v.uk/+/http://www.culture.gov.uk/ima ges/publications/capturing_the_impac t_of_libraries.pdf dolan, j. & khan, a. (2011). the more they change, the more they stay the same: public libraries and social inclusion. in d. baker and w. evans (eds.), libraries in society: role, responsibility and future in an age of change. (pp. 81-99) oxford: chandos. dugdale, g. & clark c. (2008). literacy changes lives: an advocacy resource. london: national literacy trust. durcan, t. (2011). the future of and for library and information services: a public library view. in david baker and wendy evans (eds) libraries in society: role, responsibility and future in an age of change. (p. 327). oxford: chandos. flick, u. (2009). an introduction to qualitative research. london: sage publications. goulding, a. (2006). public libraries in the 21st century. defining services and debating the future. ashgate, aldershot. international federation of library associations (2009). multicultural communities: guidelines for library services. retrieved 2 feb 2012 from http://www.ifla.org/files/libraryservices-to-multicultural-populations/ publications/multiculturalcommunities-en.pdf johnson, c. a. (2010). do public libraries contribute to social capital?: a preliminary investigation into the relationship. library and information science research 32(2), 147-155. koontz, c. & gubbin, b. (eds). (2010). ifla public library service guidelines. berlin: ifla publications 147 saur. krueger, ra & casey, m. a. (2009). focus evidence based library and information practice 2012, 7.1 20 groups: a practical guide for applied research. thousand oaks, calif.: sage publications. linley, r. & herman, c. (2008). yorkshire gso pilot programme. retrieved 2 feb 2012 from http://www.mla.gov.uk/about/region/ yorkshire/~/media/files/pdf/2008/ yorkshire_gso_pilot markless, s. & streatfield, d. (2006). evaluating the impact of your library. london: facet publishing. marshall, j. g. (2007). measuring the value and impact of health library and information services: past reflections, future possibilities. health information and libraries journal 24(suppl. 1), 4-17. mcmemeny, d (2008). the public library. london: facet publishing. museums libraries and archives (2007). securing excellence; delivering for communities. museum, libraries and archives and the local government. retrieved 2 feb 2012 from http://www.mla.gov.uk/what/publicati ons/2007 museums libraries and archives north east. (n.d.). generic social outcomes indicator bank for museums, libraries and archives. retrieved 2 feb 2012 from http://www.mla.gov.uk/what/program mes/renaissance/regions/north_east/inf o_for_sector/~/media/north_east/files /2009/generic%20social%20outcome% 20indicators%20introduction national literacy trust. (1999). building a nation of readers: a review of the national year of reading. department for education and employment and the national literacy trust. retrieved 2 feb 2012 from http://www.literacytrust.org.uk/resour ces/ practical_resources_info/749_national_ year_of_reading_1998_1999 organisation for economic cooperation and development (2002). reading for change: performance and engagement across countries. results from pisa 2000. new york: organisation for economic cooperation and development. pateman, j. & vincent, j. (2010). public libraries and social justice. famham, surrey, england: ashgate pickard, a. (2007). research methods in information. london: facet publishing. powell, r. r. (2006). evaluation research: an overview. library trends 55(1), 102-120. rankin, c. (2010) mind the gap. library & information update. 9(11), 34. rankin, c. & brock, a. (2009). delivering the best start: a guide to early years libraries. london: facet publications. rankin, c., brock, a., & matthews, j. (2009). why can’t every year be a national year of reading? an evaluation of the social impact of the national year of reading in yorkshire. library and information research 33(104), 11-25. rooney-browne, c. (2011). methods for demonstrating the value of public libraries in the uk: a literature review. library and information research 35(109), 3-39. silverman, d. (2010). doing qualitative research. london: sage publications. streatfield, d. r., tibbitts, d., swan, r., jefferies, g., & downing, r. (1999). rediscovering reading: an evaluation of the role of public libraries in the national year of reading. twickenham, england: information management associates for the library and information commission. evidence based library and information practice 2012, 7.1 21 teddlie, c. & tashakkori, a. (2009). foundations of mixed methods research : integrating quantitative and qualitative approaches in the social and behavioural sciences. london: sage publications. thomson, a. (2009.) reading: the future. london: national literacy trust. varheim, a. (2009). public libraries: places creating social capital? library hi tech 27(3), 372-381. appendix 1 national year of reading evaluation project phase 1 interview schedule for library staff in the calderdale and north lincolnshire case study authorities section 1: target beneficiaries please can you tell me about what are you doing to promote the nyr in [your authority] with the target groups. what opportunities has the nyr offered for developing new projects and initiatives? in what way has the nyr changed the way you are supporting the target groups. what do you think are the likely legacy benefits for the nyr in your authority? section 2: partnership and cross departmental working how are partnerships with other organisations being used to promote the nyr in [your authority]? what nyr partnerships activities do you feel are working well in [your authority]? what are the challenges for your staff in working in nyr partnership activities? how has the nyr effected what you are doing in your everyday work? / evidence based library and information practice/ / the project brief and specification for the evaluation project required the researchers to investigate the impact of the nyr in two sample local authorities focusing on target beneficiaries and partnerships, and cross departmental working. this paper ... national literacy trust. (1999). building a nation of readers: a review of the national year of reading. department for education and employment and the national literacy trust. retrieved 2 feb 2012 from http://www.literacytrust.org.uk/resources/ practical_resources_info/749_national_year_of_reading_1998_1999 research article   research assessment reform, non-traditional research outputs, and digital repositories: an analysis of the declaration on research assessment (dora) signatories in the united kingdom   christie hurrell associate librarian libraries and cultural resources university of calgary calgary, alberta, canada email: achurrel@ucalgary.ca   received: 21 july 2023                                                               accepted: 11 oct. 2023      2023 hurrell. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30407     abstract   objective – the goal of this study was to better understand to what extent digital repositories at academic libraries are active in promoting the collection of non-traditional research outputs. to achieve this goal, the researcher examined the digital repositories of universities in the united kingdom who are signatories of the declaration on research assessment (dora), which recommends broadening the range of research outputs included in assessment exercises.   methods – the researcher developed a list of 77 universities in the uk who are signatories to dora and have institutional repositories. using this list, the researcher consulted the public websites of these institutions using a structured protocol and collected data to 1) characterize the types of outputs collected by research repositories at dora-signatory institutions and their ability to provide measures of potential impact, and 2) assess whether university library websites promote repositories as a venue for hosting non-traditional research outputs. finally, the researcher surveyed repository managers to understand the nature of their involvement with supporting the aims of dora on their campuses.   results – the analysis found that almost all (96%) of the 77 repositories reviewed contained a variety of non-traditional research outputs, although the proportion of these outputs was small compared to traditional outputs. of these 77 repositories, 82% featured usage metrics of some kind. most (67%) of the same repositories, however, were not minting persistent identifiers for items. of the universities in this sample, 53% also maintained a standalone data repository. of these data repositories, 90% featured persistent identifiers, and all of them featured metrics of some kind. in a review of university library websites promoting the use of repositories, 47% of websites mentioned non-traditional research outputs. in response to survey questions, repository managers reported that the library and the unit responsible for the repository were involved in implementing dora, and managers perceived it to be influential on their campus.   conclusion – repositories in this sample are relatively well positioned to support the collection and promotion of non-traditional research outputs. however, despite this positioning, and repository managers’ belief that realizing the goals of dora is important, most libraries in this sample do not appear to be actively collecting non-traditional outputs, although they are active in other areas to promote research assessment reform.     introduction   universities, governments, and funders in many jurisdictions are increasingly investing time, resources, and energy into changing the way that researchers and the outputs of research are assessed for rewards such as grants, hiring, promotion, and tenure. traditional means of assessing the outputs of research, and by proxy the researchers producing these outputs, have relied on a limited set of outputs (primarily peer-reviewed journal articles, books, and monographs) as well as a narrow range of metrics to measure those outputs (primarily quantitative bibliometrics in many fields, or factors such as the prestige of a press in others). increasingly, it is being recognized that limiting the assessment of research to these outputs and metrics is inequitable and does not align with the stated mission and goals of many actors in the research ecosystem. an early and highly influential force in this shift is the declaration on research assessment (dora), which developed out of a scholarly conference held in 2012. along with a suite of other recommendations, dora pushes institutions and funders to consider a wider range of research outputs in research assessment (san francisco declaration on research assessment, 2012).   widening what is considered a “research output” outside of the traditional paradigm of peer-reviewed journal articles, books, monographs, and conference publications presents challenges. non-traditional research outputs can take a variety of forms, such as digital collections, gis projects, audio-visual materials, datasets, or code. due to their diversity, these outputs are unlikely to find a home with established scholarly publishers (library publishing coalition research committee, 2020). as such, researchers who produce non-traditional research outputs may face barriers to showcasing these items to knowledge users, peers, or assessors in their institutions or beyond.   digital repositories hosted by academic libraries are unlike traditional scholarly publishers in that they can typically host and preserve a wide variety of content and format types, and they are not constrained by the profit motive of most academic publishers. additionally, many repositories have features to track a variety of usage metrics, such as permanent identifiers to help make outputs more discoverable. staff in academic libraries can also advise researchers on topics including copyright, preservation, and impact assessment. as such, institutional repositories may be well placed to help facilitate the recognition of non-traditional outputs in research assessment. however, it is unclear to what extent academic libraries are positioning research repositories as a solution to this challenge, particularly among institutions that have publicly committed to enacting the recommendations of dora.   this study examined the digital repositories of universities in the united kingdom that are signatories of dora. the goals of this exploratory study were to 1) characterize the types of outputs collected by research repositories at dora-signatory institutions and their ability to provide measures of potential impact, 2) assess whether university library websites promote repositories as a venue for hosting non-traditional research outputs, and 3) survey repository managers to understand the nature of their involvement with supporting the aims of dora on their campuses.   literature review   research assessment and the impetus for reform   the push towards reforming the way research is assessed has been spurred on by policies of funders worldwide, by institutions, and by researchers themselves. these groups are reacting to a substantial body of research that demonstrates major limitations in the way research and researchers have traditionally been assessed. one of the major targets of this criticism has been the journal impact factor (jif), a metric that was originally created as a tool to help librarians make journal selection decisions by quantifying the frequency with which the “average article” in a journal has been cited in the past two years (garfield, 2006). over the years, its use has evolved, such that many in the research community use it as a simple proxy for journal quality, and by extension, for quality of individual journal articles and even individual researchers. this is despite well-documented limitations of this metric, including that citation distributions within journals are highly skewed, that impact factors can be manipulated by unethical editorial practices, and that the data used to calculate journal impact factors are neither transparent nor open to the public (sugimoto & larivière, 2017). even when used appropriately, the journal impact factor only captures a narrow portion of potential research impact; namely, impact on scholarship in the form of citations.   the shortcomings of the journal impact factor for research evaluation can be seen as the tip of an iceberg of well-documented biases that disproportionately impact scholars who are not english-speaking white men in all parts of the research ecosystem (see, e.g., caplar et al., 2017; chawla, 2016a; fulvio et al., 2021; mason et al., 2021). these biases influence the ability of scholars to enter academia, publish and disseminate scholarship, progress through tenure and promotion, receive funding, and be competitive for recognition and awards (inefuku & roh, 2016). additionally, the profit-seeking paradigm of traditional scholarly publishers creates artificial scarcity and uses gatekeeping mechanisms to limit the formats, perspectives, and volume of scholarship that is published through their channels (suber, 2012). this paradigm particularly disadvantages scholars whose most important contributions may come in formats such as software, code, datasets, or practice-based research outputs created with or for community partners (chawla, 2016b; parsons et al., 2019; savan et al., 2009).   the role of non-traditional outputs and impacts in research evaluation   despite not being widely accepted in traditional research evaluation exercises, there is evidence that sharing research outputs such as datasets, code, and grey literature can be very important in a variety of ways, including contact and collaboration with a broader range of colleagues, an improvement in the reproducibility of research, influence on policy and practice, and, as citation practices for these outputs mature, increased measures of impact in the form of citations or altmetrics (lawrence et al., 2014; piwowar et al., 2007; van noorden, 2013; vandewalle, 2012). the emerging evidence around the potential benefits of making these outputs more visible and discoverable has led libraries to pursue means of hosting and preserving them, although this endeavor is not without challenges (burpee et al., 2015).   in light of this evidence, and to align with current institutional initiatives around equity, diversity, and inclusion, most calls to reform the way research is assessed look to broaden both the methods used to evaluate research as well as the types of research activities evaluated. the most influential researcher-led initiative in this space, dora explicitly rejects the use of metrics such as the journal impact factor in research assessment, and provides specific recommendations to funding agencies, institutions, publishers, metrics-supplying organizations, and researchers themselves. one of dora’s recommendations is that a wider range of research outputs be considered in assessment exercises (reflecting its origin in the sciences, dora specifically mentions datasets and software) as well as a broad range of impact measures (including influence on policy and practice). alternative metrics (known as “altmetrics”) such as mentions in news media, social media, or citations on wikipedia have been offered as one method to attempt to quantify a broader range of impacts, although debate on how to interpret them, and discussion of their limitations, still exists in the research community. some of the drawbacks of altmetrics include their ability to be “gamed,” bias, data quality, and commercialization (bornmann, 2014; priem et al., 2010; sud & thelwall, 2014).   dora now has over 20,000 signatories including publishers, institutions, and individuals from across more than 160 countries (signers, n.d.). however, it is not the only influential document expressing dissatisfaction with the current state of research assessment and making recommendations for change, including broadening the range of activities that are included. in 2015, uk research and innovation published a report presenting the findings and recommendations of an independent review of the role of metrics in research assessment entitled the metric tide: review of metrics in research assessment. that report mentions a wide range of non-traditional outputs including blog posts, datasets, and software, and recommends that “the use of dois [digital object identifiers] should be extended to cover all research outputs” in order to make them more discoverable and trackable (wilsdon et al., 2015, p. 145). this recommendation was expanded in a 2022 update of the report to acknowledge the utility of a wide variety of persistent identifiers (pids) for a variety of outputs (curry et al., 2022). the same report summarizes the recommendations of 19 documents published by a variety of organizations in the research ecosystem and notes that most of them “include at least one recommendation on widening the range of research activities considered by research assessment” (curry et al., 2022, p. 68). the most commonly mentioned non-traditional outputs in these documents are datasets and code as well as activities such as peer review and mentorship.   a consistent definition of non-traditional outputs is not present in all of these documents. the australian research council defines them as research outputs that “do not take the form of published books, book chapters, journal articles or conference publications” and names several specific output types including original creative works, public exhibitions and events, research reports for an external body, and portfolios (australian research council, 2019, para. 1). a study examining the research, promotion, and tenure documents from over 100 north american universities found mention of 127 different types of scholarly outputs, which the researchers grouped into 12 diverse categories (alperin et al., 2022). these examples make it clear that researchers are producing a wide variety of items outside of the traditional paradigm, and that these outputs may be valued—or at least considered—by research assessors.   funding agencies in a wide variety of contexts are also increasingly adjusting their practices to align with the recommendations of dora and other research assessment reforms. a survey completed by 55 funding agencies from around the world in 2020 found that 34% of them had endorsed dora, and 73% had “adapted their research assessment systems and processes for different research disciplines and fields, or where different research outputs are intended” (curry et al., 2020, p. 32). additionally, 76% of respondents were currently assessing non-publication outputs, with software, code, and algorithms the most commonly mentioned. the open research funders, a group of philanthropic funders worldwide, published an “incentivization blueprint” that urges funders to “provide demonstrable evidence that, while journal articles are important, [they] value and reward all types of research outputs” and promotes research repositories as a way for researchers to disseminate outputs (open research funders group, n.d., p. 1). the global charitable foundation wellcome trust developed guidance for organizations they fund that draws heavily on dora’s recommendations and suggests that candidates for recruitment and promotion be encouraged to “highlight a broad range of research outputs and other contributions, in addition to publications” (wellcome trust, n.d., para. 20).   the role of academic libraries   simultaneously with this broad movement to shift the way research is assessed, academic libraries have been developing and expanding services and roles around scholarly communication and the research lifecycle. to support what lorcan dempsey (2017) terms the “inside-out library,” libraries have increasingly developed infrastructure and staff to support outputs at all stages of the research lifecycle, including non-traditional outputs such as datasets, preprints, digital collections, audiovisual materials, and more. as part of this shift, libraries have introduced infrastructure such as institutional repositories and research data repositories, as well as roles including scholarly communications librarians, repository technicians, research data management librarians, and digital preservation specialists. digital repositories managed by academic libraries are typically quite flexible in terms of the file formats they can host and maintain and may have advanced technological features both to promote discoverability of their contents and to track and communicate indicators around usage and potential impact of outputs. additionally, librarians and other staff supporting these services often bring with them skills in publishing best practices, digital preservation, copyright, and impact evaluation that can provide added value to users of repositories and other library-hosted infrastructure. this has led to calls for more discussion of how libraries can support the publication of a variety of non-traditional research outputs that may not align with the activities of traditional academic publishers (library publishing coalition research committee, 2020). other researchers have pointed out that libraries may need to invest in training to better prepare staff to support research evaluation and impact assessment activities (nicholson & howard, 2018).   some academic libraries have recognized an opportunity to get more involved with hosting—and helping to demonstrate the impact of—non-traditional research outputs in their repositories. in an article published in 2003, when repositories were a relatively new feature in the academic library environment, lynch noted that repositories’ ability to host these new forms had potential to challenge scholarship’s status quo:   preservability is an essential prerequisite to any claims to scholarly legitimacy for authoring in [a] new medium; without being able to claim such works are a permanent part of the scholarly record, it’s very hard to argue that they not only deserve but demand full consideration as contributions to scholarship. (p. 330)   early analyses of the deployment of institutional repositories at academic libraries focused primarily on the united states, where assessments showed that repositories did contain a variety of non-traditional outputs, although the most commonly included types of materials were versions of articles, along with student work including electronic theses and dissertations (bailey et al., 2006; lynch & lippincott, 2005; mcdowell, 2007; rieh et al., 2007). one 2005 study to assess repositories in 13 countries found more variation in content types, with european repositories being more focused on textual content types than u.s. repositories and also more likely to collect metadata-only records than their north american counterparts. the study showed that repositories in the united kingdom were comprised of 74% articles, 16% theses and books, and 9% other materials, including data and multimedia (van westrienen & lynch, 2005).   in the decades since these early analyses, some academic libraries have prioritized the collection of a diverse range of research outputs in their repositories to respond to different institutional priorities. for example, the university of san diego built on their university’s commitment to community engagement to prioritize the collection of items created in collaboration with community groups (makula, 2019). similarly, moore et al. (2020) provide examples from the university of minnesota demonstrating how the university’s institutional repository can support community engagement by hosting and preserving outputs such as newsletters, reports, and other community-focused publications. a study tracking access to non-traditional research outputs in one institutional repository found that four diverse types of outputs were all accessed frequently in the year after they were deposited, garnering on average between 16–25 page views per month (kroth et al., 2010).   libraries have also begun to see repositories as central not only to hosting non-traditional outputs, but to demonstrating their impact, due to the variety of metrics available in many repositories. kingsley (2020) characterizes this as the “impact opportunity” for academic libraries and notes that they can build on their experiences and strengths with research data management and open access to develop infrastructure and processes to capture non-traditional outputs in their repositories. however, she also points out that repositories might need to widen their collections policies to include a broad range of outputs, metadata-only records, or different content types.   the concurrent rise in interest in reforming the way research is assessed, along with academic libraries’ shift towards supporting the outputs of their researchers through the whole research lifecycle, presents potential synergies. for example, if universities are moving towards a more inclusive definition of what constitutes a research output, as well as broadening the ways in which the impacts of research outputs might be measured, then digital repositories may be able to assist with this endeavor.   methods   data to inform the research questions of this study were gathered using two methods: analysis of publicly available website content and a survey of institutional repository managers. to gather the data, a sample of relevant institutions was developed using information found on the dora website. by using the website’s filters, the researcher was able to develop a list of institutions located within the united kingdom that had signed onto dora. from this list, a subset of 77 universities was generated (non-university organizations such as scholarly associations, subject-specific research groups, and publishers were excluded).   website analysis from this list of universities, the researcher gathered urls for each institution’s repository using opendoar, a global directory of open access repositories. using this list, the researcher performed an analysis of three information sources for each university on the list: an analysis of the university’s institutional repository, an analysis of the university’s data repository, and an analysis of university websites describing or promoting the use of these repositories.   the researcher visited the public-facing websites of these institutions to analyze the content and selected features of both institutional and data repositories. the analysis was conducted in 2022 and followed a structured protocol to collect information on variables including:   the types of outputs collected by repositories the proportion of various output types contained in repositories whether or not repositories accepted metadata-only records the types of persistent identifiers (pids) created for items deposited into repositories the types of impact metrics available within repositories   the full data collection protocol can be found in hurrell (2022).   the researcher also examined webpages (such as repository policy documents, library guides, and institutional websites) that provided information about the repositories as well as guidance and instruction to faculty who might use the repositories to deposit their work. the researcher scanned these web pages looking for specific mention of non-traditional research outputs, defined as anything other than peer-reviewed journal articles, book chapters, monographs, or conference proceedings. where specific non-traditional outputs were mentioned, the researcher kept a tally of which specific outputs were named.   survey of repository personnel using the same list of 77 universities mentioned above, and during the content analysis process already described, the researcher collected contact information associated with institutional repositories at each institution. an online survey, administered via the qualtrics platform, was distributed twice during an 8-week period in 2022, resulting in a 29% response rate (n=22). the research was approved by the university of calgary’s conjoint human research ethics board. a list of all survey questions is available in hurrell (2022).   the survey consisted of eight questions designed to elicit additional data about how academic libraries, and more specifically digital repositories and the staff who support them, have been involved (or not) in implementing the recommendations of dora on their campuses; whether or not their repository policies have changed since becoming a signatory to dora; and information about other factors affecting research assessment practices at their institution. most questions were in multiple choice format, with additional data gathered through open-ended options and questions.   results   website analysis all 77 of the universities in the sample were maintaining a publicly accessible open access institutional repository at the time of data collection. as is evident from figure 1, all the institutional repositories contained peer-reviewed outputs, while 96% of repositories contained non-traditional outputs of various types, with grey literature (including white papers and reports), and art or creative performances being the most common. most repositories (76%) contained metadata-only records as well as full-text items.     figure 1 item types contained in institutional repositories (n=77).   the researcher also attempted to characterize the proportion of these item types in the repository data set. these data were difficult to collect accurately, because different repositories used different schemas for categorizing item types, and many repositories did not have clear item type categories for outputs such as grey literature, pre-prints, or teaching and learning objects. additionally, nine of the repositories in the dataset could not be searched or browsed by item type. however, it was clear from the available data that peer-reviewed items made up the majority of content available in the repositories under study, even though most repositories contained a wide variety of content types overall. as shown in figure 2, peer-reviewed outputs comprised 74% of items, with conference and workshop papers comprising 9%, “other” items comprising 9%, and theses and dissertations comprising 5%. other item types were represented in very small numbers, although it is likely that many item types were miscategorized as “other.”   figure 2 proportion of item types contained in institutional repositories (n=68).   persistent identifiers were not minted by the majority of the institutional repositories in this sample. most institutional repositories (67%) did not assign any type of persistent identifier to items, while 24% of repositories minted handles, and 10% assigned digital object identifiers (dois).   a variety of metrics were available from most institutional repositories, with download statistics being the most common type. where commercial altmetrics (e.g., altmetric.com or plumx metrics) were integrated into repositories, these were counted separately. a full list of available metrics is shown in figure 3.    the researcher also ran searches for known items from each institutional repository in google scholar to test whether the repository’s content was being indexed. four known items were searched from each institutional repository using title searches. due to documented issues with google scholar’s indexing of grey literature (haddaway et al., 2015), the researcher chose to search for grey literature item types in this test. this simple test revealed that 96% of tested items were discoverable by google scholar.   figure 3 metrics available in institutional repositories (n=77).   in the analysis of institutional websites to learn whether repositories were promoted as a place to deposit non-traditional outputs, the researcher found an approximately even split between websites naming specific non-traditional outputs as items that could be deposited into institutional repositories (47%, n=36) and websites that did not mention non-traditional outputs at all (49%, n=38). three web pages (4%) could not be assessed because they were not public (i.e., links went to a password-protected intranet). of the 36 web pages that named non-traditional research outputs, a total of 55 unique output types were noted, with datasets, reports, working papers, images, exhibitions, and software being the most commonly named output types. a complete list of outputs mentioned, along with their frequency, can be downloaded from hurrell (2022).   because research data represent an important type of non-traditional research output, and due to the growing practice to collect datasets in repositories specifically designed for this purpose, the researcher also examined institutional websites to ascertain whether the university had a separate research data repository. within the larger sample of 77 universities, 53% (n=41) had a separate data repository. almost half of these repositories (49%) contained metadata-only records as well as records with all data files, while the remaining 51% contained full records with files only.   of the subsample of data repositories, the vast majority (88%) were assigning dois to records, with 5% assigning handles and only 10% not issuing any sort of persistent identifier. similarly, most data repositories offered at least some metrics, with download statistics again being the most common. a full list of available metrics is shown in figure 4.   figure 4 metrics available in data repositories (n=41).   similar to the test run for institutional repositories, the researcher searched for known items contained in the data repositories by searching for specific dataset titles in google dataset search. only 34% of data repositories in this sample were discoverable by google dataset search.   survey of repository personnel   survey results provided additional details from institutional repository managers about how dora was implemented on their university campuses as well as some context on dora’s influence on how scholarship is produced and evaluated at their university. the 22 participants who responded represented institutions that had signed onto dora at a variety of points in time, the earliest being 2014 and the most recent being 2021.   in their responses, repository managers cited the library as being most often responsible for implementing dora, along with their institution’s research office. human resources departments and “other” (described most commonly as committees of academic staff) were less commonly mentioned, as well as other campus units, as shown in figure 5.   figure 5 campus units involved in dora implementation.   when asked more specifically if the unit responsible for the institutional repository had been involved in implementing dora, 75% of respondents indicated that their unit had been involved in some fashion, primarily through outreach and engagement; development of policies, guidelines, or information resources; or through participation in working groups or committees. a smaller number (10%) indicated having delivered workshops or instructional modules. of respondents, 20% indicated that the institutional repository had expanded its inclusion criteria to include non-traditional research outputs since signing on to dora.   when asked to rank dora’s influence on the production and assessment of research as compared to other forces, repository managers ranked it as being quite influential, second only to policies of funding agencies. overall, as shown in figure 6, dora was ranked as having a higher influence than other manifestos and reports aiming to change research assessment, and higher also than the uk’s national research assessment exercise, the research excellence framework (ref).   figure 6 perceived influence of various factors on the production and assessment of research.   discussion   this analysis of repository websites, institutional websites, and repository managers at universities in the united kingdom who are signatories to dora found that although most repositories do contain a variety of non-traditional outputs, the active collection of these materials does not appear to be a strong priority for most repositories, given the volume of these items compared to traditional research outputs. this is despite the fact that repository managers perceive dora to be an influential factor in research assessment, and that libraries are given significant responsibilities for implementing dora on their campuses.   repositories have a number of characteristics that make them ideal locations for the preservation and discoverability of academic outputs, and these characteristics were present in many of the repositories included in this sample. for example, almost all repositories were well indexed in google scholar, making their contents more discoverable, and most incorporated one or more usage metric. these features may not exist in other online locations such as faculty, lab, or community-based research unit websites, and certainly represent a benefit of repositories.   however, past research has shown that researchers are often not aware of these benefits, or they do not value them. even early research interrogating the utility of repositories suggested that the difficulties faculty associated with using repositories (such as time, concerns about copyright, unintuitive software, and inflexible features) vastly outweighed the legitimate yet unappealing benefits such as preservation and the open access citation advantage (salo, 2008).   this analysis also showed that less than half of institutional repositories in the sample were providing a persistent identifier (pid) to deposited items. pids, and particularly dois, are well recognized by researchers and provide benefits beyond a persistent url, including assistance in tracking usage and potential impact (haak et al., 2018; macgregor et al., 2023). perhaps because of their more recent deployment, software tools used as data repositories in this sample were much more likely to integrate dois, although they were less likely to be indexed by google data search.   some institutions have marketed pids, and the metrics they can help drive, to positive effect. for example, imperial college london noted a 206% increase in the deposit of reports in the year following a targeted outreach campaign promoting the features of their repository, including persistent identifiers and metrics (price & murtagh, 2020). imperial college london is a signatory to dora and thus was included in the present study’s sample; their website was by far the most detailed and thorough in promoting the repository for non-traditional outputs.   however, even those institutions that have found success in promoting their repository for non-traditional outputs note significant challenges: first, faculty members often prefer the flexibility, customizability, and perceived ease of using commercial hosting sites or personal websites for depositing outputs; second, repository managers acknowledge the resource and time commitments required to complete other research assessment exercises, especially the uk’s research excellence framework (ref; price & murtagh, 2020). the requirements of the ref—namely that all journal articles and conference papers be deposited in a repository—have vastly increased the volume of content in uk repositories but have also shifted perceptions of the repository from an optional and potentially exciting tool to a compliance requirement. as one repository manager noted in an interview, “all the interesting stuff, talking about all the benefits, or the potential benefits, it's reduced to ... it's just compliance, compliance, compliance” (as reported in ten holter, 2020, p. 7). the respondents in the ten holter study note that the emphasis on depositing traditional outputs as a requirement has left less time, energy, and interest amongst repository managers and researchers for using repositories in other ways. this observation was borne out in the website analysis portion of the current study: all websites and guides promoting repositories contained information about how to comply with the ref’s open access requirement and provided guidance on how to comply with publisher requirements around green open access and self-archiving of journal articles.   collecting non-traditional outputs in repositories requires targeted outreach, engagement, possible investment in infrastructure changes (such as doi registration), and potential changes to policies and workflows. an analysis of over 100 academic repositories in north america discovered that while 95% of them contained outputs such as technical reports and working papers, only 63% of repositories appeared to be making an active effort to collect those items (marsolek et al., 2018). the authors suggest that changes to repository collection policies and scope statements might be required, as well as targeted outreach and metadata enhancement to make deposited items more discoverable. the present study reinforces these findings by gathering data from a different jurisdiction to find similar results.   there are several limitations to the current study. first, the researcher relied on information found on institutions’ public websites to gather data about the contents, features, and functionality of repositories and information used to promote them. it is possible that this analysis missed important information, and it is uninformed by the workflows and procedures that underlie the public interface. the number of repository managers that responded to the survey part of the study was small, and because participants responded anonymously, links between the survey results and the website analysis results cannot be drawn. future research would benefit from more in-depth engagement with repository managers, perhaps in the form of interviews. additionally, further engagement with researchers in the form of interviews and user experience testing may surface additional opportunities for developing effective partnerships to showcase non-traditional research outputs in repositories.   conclusion   this study demonstrates that repositories are well equipped to accept non-traditional research outputs, both from a technical and a policy perspective. most repositories already contain a wide variety of non-traditional outputs, but the volume of these outputs is dwarfed in comparison to traditional, peer-reviewed outputs. this suggests that repository staff and researchers both put a higher priority on ensuring that traditional outputs are reflected in institutional repositories. this is likely influenced by the requirements of existing research assessment exercises and cultures.   this research suggests that if repositories are to make a concerted effort to collect and showcase non-traditional research outputs, they may have to expand beyond the current focus of ensuring that researchers comply with requirements set out by funding agencies, governments, and publishers. the uk’s higher education funding bodies are making changes to the next ref, which will assess research and impact between 2021 and 2027, including to “recognise and reward a broader range of research outputs” (research england et al., 2023, p. 2). the report goes on to state:   supporting and rewarding a diversity of research outputs is important for the progress of research and its dissemination to diverse audiences. there are important output types that contribute to the wider infrastructure of research fields that, as well as being important contributions in their own right, enable the research of others. examples include review articles (including systematic reviews), meta-analyses, replication studies, datasets, software tools, reagents, translations and critical editions. reaching businesses, policymakers and citizens also requires outputs in different formats, such as policy summaries or video or audio content. (p. 8)   it is too soon to tell whether these changes, coupled with the incremental yet growing culture shifts in the research ecosystem towards more holistic and equitable forms of research assessment, will result in repositories gaining a more central role in collecting, disseminating, and showcasing non-traditional research outputs.   acknowledgements   thank you to dr. john brosz for assistance with data visualization.   references   alperin, j. p., schimanski, l. a., la, m., niles, m. t., & mckiernan, e. c. (2022). the value of data and other non-traditional scholarly outputs in academic review, promotion, and tenure in canada and the united states. in a. l. berez-kroeker, b. j. mcdonnell, e. koller, & l. b. collister (eds.), the open handbook of linguistic data management (pp. 171–182). mit press. https://doi.org/10.7551/mitpress/12200.003.0017   australian research council. (2019). non-traditional research outputs (ntros). in state of australian university research 2018–19: era national report. https://dataportal.arc.gov.au/era/nationalreport/2018/pages/section1/non-traditional-research-outputs-ntros/   bailey, c. w., jr., coombs, k., emery, j., mitchell, a., morris, c., simons, s., & wright, r. (2006). institutional repositories (spec kit 292). association of research libraries. https://publications.arl.org/institutional-repositories-spec-kit-292/1   bornmann, l. (2014). do altmetrics point to the broader impact of research? an overview of benefits and disadvantages of altmetrics. journal of informetrics, 8(4), 895–903. https://doi.org/10.1016/j.joi.2014.09.005   burpee, k. j., glushko, b., goddard, l., kehoe, i., & moore, p. (2015). outside the four corners: exploring non­traditional scholarly communication. scholarly and research communication, 6(2), article 0201224. https://doi.org/10.22230/src.2015v6n2a224   caplar, n., tacchella, s., & birrer, s. (2017). quantitative evaluation of gender bias in astronomical publications from citation counts. nature astronomy, 1(6), article 0141. https://doi.org/10.1038/s41550-017-0141   chawla, d. s. (2016a). men cite themselves more than women do. nature. https://doi.org/10.1038/nature.2016.20176   chawla, d. s. (2016b). the unsung heroes of scientific software. nature, 529(7584),115–116. https://doi.org/10.1038/529115a   curry, s., de rijcke, s., hatch, a., pillay, d., van der weijden, i., & wilsdon, j. (2020). the changing role of funders in responsible research assessment: progress, obstacles and the way ahead. research on research institute. https://doi.org/10.6084/m9.figshare.13227914.v1   curry, s., gadd, e., & wilsdon, j. (2022). harnessing the metric tide: indicators, infrastructures & priorities for uk responsible research assessment. research on research institute. https://doi.org/10.6084/m9.figshare.21701624.v2   dempsey, l. (2017). library collections in the life of the user: two directions. liber quarterly: the journal of the association of european research libraries, 26(4), 338–359. https://doi.org/10.18352/lq.10170   fulvio, j. m., akinnola, i., & postle, b. r. (2021). gender (im)balance in citation practices in cognitive neuroscience. journal of cognitive neuroscience, 33(1), 3–7. https://doi.org/10.1162/jocn_a_01643   garfield, e. (2006). the history and meaning of the journal impact factor. jama, 295(1), 90–93. https://doi.org/10.1001/jama.295.1.90   haak, l. l., meadows, a., & brown, j. (2018). using orcid, doi, and other open identifiers in research evaluation. frontiers in research metrics and analytics, 3: article 28. https://doi.org/10.3389/frma.2018.00028   haddaway, n. r., collins, a. m., coughlin, d., & kirk, s. (2015). the role of google scholar in evidence reviews and its applicability to grey literature searching. plos one, 10(9), article e0138237. https://doi.org/10.1371/journal.pone.0138237   hurrell, c. (2022). role of institutional repositories in supporting dora. open science framework. https://osf.io/5kjna/   inefuku, h. w., & roh, c. (2016). agents of diversity and social justice: librarians and scholarly communication. in k. l. smith & k. a. dickson (eds.), open access and the future of scholarly communication: policy and infrastructure (pp. 107–127). rowman & littlefield.   kingsley, d. (2020). the ‘impact opportunity’ for academic libraries through grey literature. the serials librarian, 79(3–4), 281–289. https://doi.org/10.1080/0361526x.2020.1847744   kroth, p. j., phillips, h. e., & hannigan, g. g. (2010). institutional repository access patterns of nontraditionally published academic content: what types of content are accessed the most? journal of electronic resources in medical libraries, 7(3), 189–195. https://doi.org/10.1080/15424065.2010.505515   lawrence, a., houghton, j., thomas, j., & weldon, p. (2014). where is the evidence? realising the value of grey literature for public policy and practice: a discussion paper. swinburne institute for social research. http://doi.org/10.4225/50/5580b1e02daf9   library publishing coalition research committee. (2020). library publishing research agenda. educopia institute. http://doi.org/10.5703/1288284317124   lynch, c. a. (2003). institutional repositories: essential infrastructure for scholarship in the digital age. portal: libraries and the academy, 3(2), 327–336. https://doi.org/10.1353/pla.2003.0039   lynch, c. a., & lippincott, j. k. (2005). institutional repository deployment in the united states as of early 2005. d-lib magazine, 11(9). https://doi.org/10.1045/september2005-lynch   macgregor, g., lancho-barrantes, b. s., & pennington, d. r. (2023). measuring the concept of pid literacy: user perceptions and understanding of pids in support of open scholarly infrastructure. open information science, 7(1): article 20220142. https://doi.org/10.1515/opis-2022-0142   makula, a. (2019). “institutional” repositories, redefined: reflecting institutional commitments to community engagement. against the grain, 31(5): article 17. https://doi.org/10.7771/2380-176x.8431   marsolek, w. r., cooper, k., farrell, s. l., & kelly, j. a. (2018). the types, frequencies, and findability of disciplinary grey literature within prominent subject databases and academic institutional repositories. journal of librarianship and scholarly communication, 6(1), article ep2200. https://doi.org/10.7710/2162-3309.2200   mason, s., merga, m. k., gonzález canché, m. s., & mat roni, s. (2021). the internationality of published higher education scholarship: how do the ‘top’ journals compare? journal of informetrics, 15(2), article 101155. https://doi.org/10.1016/j.joi.2021.101155   mcdowell, c. s. (2007). evaluating institutional repository deployment in american academe since early 2005: repositories by the numbers, part 2. d-lib magazine, 13(9/10). https://doi.org/10.1045/september2007-mcdowell   moore, e. a., collins, v. m., & johnston, l. r. (2020). institutional repositories for public engagement: creating a common good model for an engaged campus. journal of library outreach and engagement, 1(1), 116–129. https://doi.org/10.21900/j.jloe.v1i1.472   nicholson, j., & howard, k. (2018). a study of core competencies for supporting roles in engagement and impact assessment in australia. journal of the australian library and information association, 67(2), 131–146. https://doi.org/10.1080/24750158.2018.1473907   open research funders group. (n.d.). incentivizing the sharing of research outputs through research assessment: a funder implementation blueprint. https://www.orfg.org/s/orfg_funder-incentives-blueprint-_final_with_templated_language.docx   parsons, m. a., duerr, r. e., & jones, m. b. (2019). the history and future of data citation in practice. data science journal, 18, article 52. https://doi.org/10.5334/dsj-2019-052   piwowar, h. a., day, r. s., & fridsma, d. b. (2007). sharing detailed research data is associated with increased citation rate. plos one, 2(3), article e308. https://doi.org/10.1371/journal.pone.0000308   price, r., & murtagh, j. (2020). an institutional repository publishing model for imperial college london grey literature. the serials librarian, 79(3–4), 349–358. https://doi.org/10.1080/0361526x.2020.1847737   priem, j., taraborelli, d., groth, p., & neylon, c. (2010). altmetrics: a manifesto.  http://altmetrics.org/manifesto/   research england, scottish funding council, higher education funding council for wales, & department for the economy, northern ireland. (2023). research excellence framework 2028: initial decisions and issues for further consultation (ref 2028/23/01). https://repository.jisc.ac.uk/9148/1/research-excellence-framework-2028-initial-decisions-report.pdf   rieh, s. y., markey, k., st. jean, b., yakel, e., & kim, j. (2007). census of institutional repositories in the u.s.: a comparison across institutions at different stages of ir development. d-lib magazine, 13(11/12). https://doi.org/10.1045/november2007-rieh   salo, d. (2008). innkeeper at the roach motel. library trends, 57(2), 98–123. https://doi.org/10.1353/lib.0.0031   san francisco declaration on research assessment. (2012). declaration on research assessment. https://sfdora.org/read/   savan, b., flicker, s., kolenda, b., & mildenberger, m. (2009). how to facilitate (or discourage) community-based research: recommendations based on a canadian survey. local environment, 14(8), 783–796. https://doi.org/10.1080/13549830903102177   signers. (n.d.). declaration on research assessment. https://sfdora.org/signers/   suber, p. (2012). open access. mit press. https://doi.org/10.7551/mitpress/9286.001.0001   sud, p., & thelwall, m. (2014). evaluating altmetrics. scientometrics, 98(2), 1131–1143. https://doi.org/10.1007/s11192-013-1117-2   sugimoto, c. r., & larivière, v. (2017). measuring research: what everyone needs to know. oxford university press.   ten holter, c. (2020). the repository, the researcher, and the ref: “it’s just compliance, compliance, compliance.” the journal of academic librarianship, 46(1), article 102079. https://doi.org/10.1016/j.acalib.2019.102079   van noorden, r. (2013). data-sharing: everything on display. nature, 500(7461), 243–245. https://doi.org/10.1038/nj7461-243a   van westrienen, g., & lynch, c. a. (2005). academic institutional repositories: deployment status in 13 nations as of mid 2005. d-lib magazine, 11(9). https://doi.org/10.1045/september2005-westrienen   vandewalle, p. (2012). code sharing is associated with research impact in image processing. computing in science & engineering, 14(4), 42–47. https://doi.org/10.1109/mcse.2012.63   wellcome trust. (n.d.). guidance for research organisations on how to implement responsible and fair approaches for research assessment. https://wellcome.org/grant-funding/guidance/open-access-guidance/research-organisations-how-implement-responsible-and-fair-approaches-research   wilsdon, j., allen, l., belfiore, e., campbell, p., curry, s., hill, s., jones, r., kain, r., kerridge, s., thelwall, m., tinkler, j., viney, i., wouters, p., hill, j., & johnson, b. (2015). the metric tide: report of the independent review of the role of metrics in research assessment and management. https://doi.org/10.13140/rg.2.1.4929.1363 research article   advancing the reference narrative: assessing student learning in research consultations   doreen r. bradley director of learning programs and initiatives university of michigan library ann arbor, michigan, united states of america email: dbradley@umich.edu   angie oehrli learning librarian university of michigan library ann arbor, michigan, united states of america email: jooerhli@umich.edu   soo young rieh professor and associate dean for education school of information, university of texas at austin austin, texas, united states of america email: rieh@ischool.utexas.edu   elizabeth hanley post graduate fellow academic innovation, university of michigan ann arbor, michigan, united states of america email: hanleyel@umich.edu   brian s. matzke digital humanities librarian central connecticut state university library new britain, connecticut, united states of america email: bmatzke@ccsu.edu   received: 30 aug. 2019                                                                  accepted: 9 jan. 2020      2020 bradley, oehrli, rieh, hanley, and matzke. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. doi: 10.18438/eblip29634     abstract   objective – as reference services continue to evolve, libraries must make evidence based decisions about their services. this study seeks to determine the value of reference services in relation to student learning acquired during research consultations, by soliciting students’ and librarians’ perceptions of consultation success and examining the degree of alignment between them.   methods – the alignment of students’ learning outcomes (reported skills and knowledge acquired) with librarians’ expectations for student learning during consultations was assessed. an online questionnaire was conducted to gather responses from students who had sought consultation services; 20 students participated. in-person interviews took place with eight librarians who had provided these consultations. the online questionnaire for students included questions about students’ assessments of their self-identified learning goals through consultation with a librarian and their success at applying the knowledge and skills gained. librarian interviews elicited responses about students’ prior research experience, librarians’ objectives for student learning, librarians’ perceptions of student learning outcomes, and perceived consultation success. the responses of both the students and the librarians were coded, matched, and compared.   results – students and librarians both considered the consultation process to be successful in advancing learning objectives and research skills. all students reported that the consultations met their expectations, and most reported that the skills acquired were applicable to their projects and significantly improved the quality of their work. librarians expressed confidence that students had gained competency in the following skill sets: finding sources, search strategy development, topic exploration, specific tool use, and library organization and access. a high degree of alignment was observed in the identification by both students and librarians of “finding sources” as the skill set most in need of enhancement or assistance, while some disparity was noted in the ranking of “search strategy development,” which librarians ranked second and students ranked last.   conclusion – the data demonstrate that both students and librarians perceived individual research consultations as an effective means to meet student learning expectations. study findings suggest that as reference models continue to change and reference desk usage declines, research consultations remain a valuable element in a library’s service model and an efficient use of human resources.     introduction   librarians are increasingly expected to demonstrate the value of their services for improving student learning and success, and to make informed decisions based on empirical data. while research consultation services have been shown to be useful for students (butler & byrd, 2016), and although users report satisfaction with such services (ishaq & cornick, 1978; magi & mardeusz, 2013; martin & park, 2010; rogers & carrier, 2017), most previous studies evaluating research consultation services have tended to focus on the usage or effectiveness of the service (e.g., attebury, sprague, & young, 2009; watts & mahfood, 2015). we still know little about the extent to which these services affect student learning in academic library settings specifically and in higher education more generally.   our study investigated the value and contributions of research consultation services with respect to student-centered learning objectives. we sought to understand students’ experience beyond the use of the service or the evaluation of the quality of the service. therefore, we conducted an empirical study to examine the value of research consultation services to assess student learning and the direct implications of that learning for student success.   this study was conducted in a u.s. research university with 45,000 students, comprising 30,000 undergraduates and 15,000 graduate students. the university’s library offers various consultation services through which students can meet one-on-one with a librarian for approximately 30 minutes. while the library provides specialist consultation services whereby users can receive assistance from an expert in an academic discipline or technological field, the library also offers a general research consultation service staffed by librarians identified as generalists who have some knowledge in many fields. this study focuses on the consultations provided through this general service.   consultation topics are patron driven, typically centering on questions that students have about research-based academic projects. the consultation format is flexible, determined by students’ self-identified learning objectives. with the purpose of evaluating the extent to which students perceived their learning objectives had been achieved and to better understand the students’ self-identified learning objectives, an online questionnaire was initiated by contacting those students who had used the consultation service. to obtain librarians’ perceptions of those same consultations, all of the librarians who had provided the service to those students who responded to the online questionnaire were interviewed. this method enabled examination of the alignment between students’ reported acquisition of knowledge and skills and the librarians’ expectations and perceptions of student learning during the consultation process.   literature review   the literature on library research consultations dates back to the 1970s, when academic libraries began to offer appointment-based consultation services. in their early study of consultations at the university of north carolina at chapel hill, ishaq and cornick (1978) found high degrees of satisfaction with the program; all of the 49 questionnaire respondents who had utilized the service indicated that they would use the service again and recommend it to others. later studies found similarly high levels of satisfaction with library consultation services at other institutions. for example, a study of the university of idaho’s library research consultations found that an average of 115 students per year utilized the service, a number that remained relatively stable over the 10-year study period and that represented a wide array of departments and levels of study (attebury et al., 2009). in addition, in a recent questionnaire of 80 students, 86% described their consultations as “very useful” and 14% described them as “somewhat useful” (butler & byrd, 2016, p. 85).   much of the recent literature on consultation services focuses on the role of technology in facilitating research consultations. online appointment tools like google calendar and youcanbook.me have been found to decrease student wait times and mitigate library anxiety by enabling students to make appointments without having to contact a librarian directly (cole & reiter, 2017; kuglitsch, tingle, & watkins, 2017). at the same time, employing online note-taking tools like evernote during consultations has been found to help students organize information and provide a research narrative that students can refer back to (kani, 2017).   despite the usefulness of digital tools in consultation sessions, many students describe face-to-face consultations as the easiest and most efficient method for getting help, in comparison to forms of virtual reference such as chat (magi & mardeusz, 2013). for example, when working in collaboration with their university writing center, meyer, forbes, and bowers (2010) described the importance of providing a dedicated, highly visible space for research consultations; having a physical space that served as the “research center” eased students’ anxiety about asking for help and facilitated the promotion of the library’s research services. similarly, rogers and carrier (2017) found that students appreciated the opportunity to meet in a private consultation environment as opposed to the “open” environment of the reference desk.   many research consultation studies center on specific student populations or circumstances. for example, isbell (2009) focused on honors students’ perceptions of a consultation service because such students are highly motivated, study a wide range of disciplines, and tend to overestimate their research abilities. faix, macdonald, and taxakis (2014) surveyed students from both a senior capstone class and a freshman seminar who were required to attend a library research consultation. the study found that upper-level students benefited more from the consultations than freshmen, who were sometimes overwhelmed by the number of resources that consultation sessions helped them locate. in addition, kolendo (2016) identified the extra-credit consultation as a unique circumstance, in which students schedule sessions for the credit only, usually after having already completed their papers.   a persistent challenge is measuring the effectiveness of research consultations. fournier and sikora (2015, 2017) discussed the lack of assessment in scholarly literature, finding that most libraries either practice no form of assessment or rely solely on informal feedback from users. however, the literature demonstrates that more sophisticated analyses have been attempted. sikora, fournier, and rebner (2019) administered preand post-consultation tests, demonstrating statistically significant improvements in students’ search abilities and confidence in their research skills after consultations. reinsfelder (2012) used citation analysis to show that consultations positively impacted the quality and quantity of sources that students used in their papers.   in addition to quantitative metrics, qualitative research methods such as questionnaires (butler & byrd, 2016), interviews (rogers & carrier, 2017), focus groups (watts & mahfood, 2015), and analyses of librarians’ consultation notes (suarez, 2013) provide valuable insights into what students learn during consultation sessions. studies have found that confusion about library terminology can impede student learning (butler & byrd, 2016), but that students value the individualized attention from in-depth engagement with the librarian, as well as the librarians’ perceived subject expertise (rogers & carrier, 2017). relatedly, students who participate in consultations have reported a higher degree of confidence in their research abilities, believing that their research has become more efficient and feeling that they have developed good relationships with the librarian as an educator (watts & mahfood, 2015). however, others have found that students tend to overestimate their information-seeking abilities even when they still struggle to develop search strategies or generate keywords beyond those that are laid out in the assignment prompt (suarez, 2013). in this manner, students in research consultations appear to evince the dunning-kruger effect, the cognitive bias whereby people are unable to recognize their own incompetence (suarez, 2013). on the other hand, librarians have sometimes been found to underestimate the effectiveness of the consultation, a phenomenon known as provider pessimism (butler & byrd, 2016).   aims   this study therefore aims to contribute to this growing body of literature on student learning through research consultations, by providing a more complete and nuanced picture of students’ and librarians’ perceptions of the consultation process. specifically, three research questions are addressed:   1.       how do students who participated in a library consultation perceive their learning objectives and experience? 2.       how do librarians who provided a library consultation conceptualize the student learning from this service? 3.       how aligned are students and librarians in their perceptions of the degree of success of the consultation?   methods   study data was collected using a student questionnaire and in-person interviews with librarians. first, a questionnaire was sent to students who had participated in consultations during the fall 2017 and winter 2018 semesters. the questionnaire had three main foci: (1) understanding students’ self-identified learning objectives; (2) evaluating the degree to which students perceived that these learning objectives were achieved; and (3) understanding students’ perceptions of how they applied the knowledge and skills acquired in the consultations to their course projects. after the student questionnaires were completed, the librarians were interviewed. in order to minimize potential biases, neither the students nor the librarians were informed about the study prior to the consultations.   part 1: student perspectives   participants   during the winter 2018 semester, questionnaires were sent to the 38 students who had participated in research consultations during the fall 2017 or winter 2018 semesters (see appendix a). of those 38, 20 questionnaires were completed for a 53% response rate. researchers administered the questionnaire several months after the consultations occurred in order to permit students sufficient time to complete projects, to receive feedback on their projects, and to reflect upon their learning. students required approximately 30 minutes to complete the questionnaire. a $30 amazon gift card was offered as an incentive to increase the response rate and to motivate students to provide thoughtful and accurate responses.   measures   the questionnaire was distributed via email using qualtrics software and was comprised of 31 items in total, although not all questions were visible to all students due to the use of skip logic. in addition to demographic questions there were open-ended items asking about the students’ self-identified learning objectives (“what did you hope to learn from the consultation?”) and student perceptions of the learning that took place (“what, if anything, did you discuss that was new to you?”). closed-ended items asked about student perceptions of the success of the consultations (“do you feel that the consultation met your expectations?” “to what extent did this service improve the quality of your project/assignment?”). we also asked for specific feedback that students may have received from course instructors on their projects. although student emails were solicited in the questionnaire for possible future contact, follow-up interviews were not conducted.   student learning objectives and student perceptions of learning were coded using four categories as follows:   1.       library tools (use of research tools such as specific databases) 2.       library organization and access (understanding how to access print and digital resources within the library, including the physical library buildings and library website) 3.       research process (topic exploration, search strategy development, and finding, evaluating, and citing sources) 4.       other (goals not covered above, such as earning extra credit for meeting with a librarian)   student perceptions of success were coded along two dimensions using a four-point likert scale:   1.       success, ranging from one (not at all) to four (significantly) 2.       met expectations, ranging from one (not at all) to four (significantly)    after the questionnaire closed, the responses were downloaded from qualtrics in csv format. two researchers then coded the open-ended responses using nvivo software.   part 2: librarian perspectives   participants   after students submitted their questionnaires, the librarians who had conducted the consultations were contacted for interviews. we sought to understand what the librarians believed the students had needed to learn in order to complete their projects and to compare this to the students’ own perceptions of what they themselves needed to learn. therefore, the interviews focused on (1) understanding librarians’ perceptions of student learning needs and (2) evaluating the degree to which librarians believed these learning needs were achieved.   using consultation scheduling software, we identified the names of eight librarians who provided the consultations for all 20 students were identified using consultation scheduling software. one of the eight librarians provided approximately half of the consultations, while each of the other librarians conducted between one and three consultations. all eight librarians were interviewed during the winter 2018 semester. student and librarian responses were matched based on library records of the research consultations. interviews lasted approximately 30 minutes and were audio recorded and transcribed for coding purposes.   measures   to assess librarians’ perspectives of student learning, an interview protocol was developed that contained questions about the students’ prior research experience (“what was your impression of the student’s research skills at the start of the session?”); librarian learning objectives (“what goals did you have for the session? that is, what did you want the students to walk away from the session having learned?”); librarian perceptions of student learning outcomes (“what [skills and concepts] did the student learn?”); and consultation success (“on a scale of one to ten, ten being highly successful, one being not successful, how successful was the session?”) (see appendix b). the interview questions were coded along the same four dimensions outlined for student learning objectives and student perceptions of learning: library tools, library organization and access, research process, and other. the question about consultation success asked librarians to provide a rating on a scale from 1 (very unsuccessful) to 10 (very successful). the transcribed interviews were coded by two researchers using nvivo software. a codebook was developed focusing on the following themes: library organization and access, specific tools, and the research process. we test coded five interviews to assess the feasibility of the coding scheme, to facilitate consensus on the application of the codes, and to ensure inter-coder reliability.   demographic background and consultation length   of the 20 students who had received consultations, 16 were undergraduate students, one was a master’s student, one was a phd student, and two students self-identified as “other.” the students represented a total of 14 disciplines including nursing, economics, social work, political science, history, computer science, international studies, kinesiology, biochemistry, and several other disciplines that included five students with undeclared majors. the duration of the consultations varied; four consultations lasted over 30 minutes, nine were 20–30 minutes long, six were 10–20 minutes long, and one lasted less than 10 minutes. the majority, 15 of the sessions, were in the 10–30 minute range. all of the students responded that they were working on a project; of these, 15 projects were for a course and five were not course related. all consultations were sought to meet immediate, short-term objectives rather than for longer-term projects. the eight librarians had between two and 30 years of reference service experience in academic libraries.   eighteen students reported that they remembered the consultation “well,” while only two reported remembering it “a little.”  therefore, although students completed the questionnaire several months after the consultations occurred, they were able to provide a good level of detail in their responses. likewise, for the librarians, most remembered the consultations well with some having sent follow-up email messages to students. in one case, a librarian was not able to recall enough information about the consultation to assess its level of success.   results   in this section, we examine the results of our study from two perspectives: student perceptions about their own learning and their assessments of the success of the consultations versus librarian perspectives on student learning and consultation success.   student learning: student perspectives   in general, students reported that their self-identified learning objectives were met during the consultations, responding consistently that the consultations had helped them to learn new skills for their projects; the fact that the consultations provided them with search tactics that they could use in the future was appreciated. the students also reported that the consultations had helped them to locate higher quality sources. one respondent wrote, “i was completely lost on where to go. the topic was a little bit peculiar and doing a simple google search was not helping much. the consultation helped me gain more trustworthy sources, which was key.” there were no discernible differences between undergraduate and graduate student participants’ expectations or perceptions about consultation outcomes.   students identified their top four learning objectives as (1) finding sources (n = 19); (2) using specific tools/databases (n = 10); (3) library organization and access, which included navigating both the physical space of the library and the library website (n = 7); and (4) search strategy development (n = 3).   one interesting finding is that students reported that they applied what they had learned to their projects. sharing feedback received from their course instructors, respondents stated: “my instructor said my sources were extremely strong and made my argument more well-rounded”; “my compilation of data was outstanding and everything they were looking for”; and “i got good feedback and a good grade in part because of the thoroughness to which i worked to find meaningful resources.”   one of the questions we asked in the questionnaire was whether students had used skills learned during the consultations to enhance their work on any subsequent projects, as this would demonstrate transferable skills learned. half of the survey respondents (n = 10), indicated that they were able to apply something they discussed during the consultations to a project other than the one that led them to schedule the consultation. one student commented, “i have since used the methods [the librarian] taught me to aid my research in my new political science research assistant job. i have also used them in other courses for other essays and projects.” another offered, “i’m working on a psych project now that i regularly use my database research skills to find articles for.”   some students reported that they shared what they learned from the consultation service with others, such as one respondent who indicated, “i was able to teach these techniques to my research partner to find other sources for our project.” such responses strongly suggest that student-librarian consultations pay themselves forward by helping students to use their enhanced skills and knowledge in subsequent research projects, and by enabling students to teach these skills to others, which extends the impact of consultations beyond a single-project application.   student learning: librarian perspectives   at the beginning of each interview, we asked librarians to rate each student’s pre-consultation level of research experience. most students (n = 12) were rated “low” in previous research experience, while only two students were identified as having “high” skill levels. data analysis revealed that the librarians identified four main skill sets that students needed to acquire or enhance in order to successfully complete work on their projects, with individual students requiring help in several of these skill areas: (1) finding sources (n = 19); search strategy development (n = 15); (3) topic exploration (n = 5); and (4) using specific tools (n = 3).   librarians described in detail how they felt that students displayed their understanding of the concepts covered in the consultation, describing how students suggested synonyms to create better search strategies and used new search strategies and new databases while searching alongside the librarian. while librarians recognized that the students had requested help with specific databases, they felt that students would benefit from more broad-based help, for example, with formulating search strategies or exploring topics through using filters to refine search results. librarians expressed confidence that the students had gained competency in the top learning needs that they had identified (table 1).   table 1 librarian and student assessment of the top four student learning needs librarian perceptions student perceptions 1.  finding sources 1.  finding sources 2.  search strategy development 2.  using specific tools/databases 3.  topic exploration 3.  library organization and access 4.  using specific tools 4.  search strategy development   consultation success: student and librarian perspectives   using a scale of significantly, somewhat, a little, and not at all, all 20 students reported that the consultations had met their expectations, with 15 rating that their expectations had been significantly met and five rating that their expectations had been somewhat met. no students reported that the consultation met their expectations a little or not at all. using a similar scale to assess whether the consultations had any impact on the participants’ projects, 19 students out of 20 felt that the consultations improved their project to some degree. fourteen students responded that the consultations improved their projects significantly, three somewhat, and two a little. only one reported that the consultation did not improve their project at all; this student had already explored a significant amount of resources and was referred to a subject specialist outside of the general reference consultation service.   figure 1 agreement on consultation success between librarians and students.   librarians’ assessments of the success of consultations were similar to those of the students. using a scale of 1–10, ranging from 1 (very unsuccessful) to 10 (very successful), librarians reported that they felt 13 of the consultations were very successful (rated 8–10), and that six were somewhat successful (rated 4–7). none of the librarians considered any consultations to be very unsuccessful (rated 1–3), although one librarian revealed that they could not remember enough details to rate the success of one consultation.   the rates at which the librarians and students agreed on the degree of success were measured using the same scale as above. for 14 consultations, both groups agreed on the level of success. of interest, the librarians rated four consultations as having been more successful than the students rated those consultations. although, one student did rate a consultation as more successful than the librarian did. for the instance in which the librarian could not remember enough about the consultation to attach a level of success, we decided not to compare it with the student-reported level of success (figure 1).   discussion   this study was designed to address three questions: how did students who accessed the library consultation service perceive their learning objectives and experience? how did librarians who provided the library consultation service conceptualize the student learning from this service? how aligned were students and librarians in their perceptions of the degree of success of the consultations?   with regard to learning objectives, there was almost complete agreement (19 cases) between both librarians and students that “finding sources” was the most important area requiring new and additional skills in a general consultation. however, both groups diverged when ranking the other learning objectives (see table 1).   perhaps most striking is the discrepancy in the ranking of “search strategy development,” which was ranked second in importance by librarians but last by students, whereas “specific tools and databases” was ranked second in importance by students and last by librarians. “topic exploration” was third in importance for the librarians, but was not named among the top four learning needs by students. “library organization and access,” named third in importance by students, was not mentioned by librarians among their top four.   when describing gaps in students’ skill sets, the librarians were more likely to discuss broader concepts such as critical thinking, as well as universally applicable competencies like search strategy development and topic exploration. by contrast, students tended to discuss basic needs with concrete outcomes, wanting to learn how to use a particular tool or to find a specific source at the library. this discrepancy is an opportunity for librarians to expand students’ awareness of their own learning needs and to encourage self-reflection.   some research suggests that while students report a higher degree of confidence in their research abilities after a consultation (watts & mahfood, 2015), they may overestimate their information-seeking abilities overall (suarez, 2013). a higher degree of confidence after a consultation may be an indicator of student success for the consultation. librarians have underestimated their effectiveness in consultations in general (butler & byrd, 2016), and it might be concluded that librarians should be more confident than they are about the impact of their work. these prior studies indicate that there is a possible mismatch between what students rate as successful and what librarians perceive as impactful. the findings of this study also show that students and librarians interpret the success of research consultations in slightly different ways. while most of the librarians and students agreed on the level of success of the consultations in this study, in four instances the librarians rated the consultations as more successful than did the students involved in those consultations, a finding that is seemingly inconsistent with butler and byrd’s (2016) work. there was only one case in which the student’s rating of the consultation’s success was higher than the librarian’s. while the discrepancy in perceptions of success among these five cases is interesting, more noteworthy is the fact that some degree of success was indicated by both librarians and students in all cases. we interpret this finding as suggesting the potential benefit of incorporating two routine practices into the consultation process: having students and librarians identify clear learning objectives at the outset of the consultation; and following up consultations by asking students about their level of satisfaction with the process and their success at applying newly developed library skills to additional projects. we believe that these practices are likely to improve student-librarian consensus of perceived success, enhance communication between students and librarians, and provide feedback to aid the ongoing improvement of consultation services.   by a very large measure, both librarians and students felt that the time and effort put forth in the consultations was worthwhile. nearly all students, 19 out of 20, reported that the consultations improved their projects to some degree, while all responded that the consultations met their expectations. consistent with butler and byrd’s (2016) work, the librarians reported feeling that slightly fewer of the consultations (17 out of 20) were very successful. however, when assessing how well the self-identified learning objectives were met during the consultations, the data illustrate that half of the student respondents were able to apply something that they had discussed during the consultations to projects other than the one for which they had scheduled the consultation. this finding indicates the acquisition of transferable skills and demonstrates both the shortand long-term value of consultations for improving students’ research skills.   the consultations in this study were initiated for immediate, short-term needs associated with required projects, rather than for self-directed, longer-term projects. a valuable extension of this study might include consultations with undergraduates who are completing long-term projects, such as honor theses, or consultations with a sampling of graduate students. while the learning objectives in these cases might differ from those in the present study, the measures of perceived consultation success (applicability of new or improved skills, transferability of skills to other contexts, and alignment between librarian and student perceptions of success) would still pertain, thus offering a more complete picture of student learning through consultations.   conclusions   the purpose of this study was to assess student learning from research consultations in the academic library by identifying the students’ own learning goals and measuring the success in achieving those goals in relationship to librarian perceptions of the work completed in those consultations. students and librarians in this study had the same primary goal in these consultations: to find sources for a research project. though there were some differences in perceptions of learning outcomes outside of that primary goal, in most cases, both students and librarians interpreted the degree of success in the consultation similarly. the findings clearly demonstrate that individual research consultations are effective and impactful in meeting student learning needs. as reference models continue to change and reference desk usage declines, general research consultations are a valuable element in librarians’ service model and an efficient use of human resources.   references   attebury, r., sprague, n., & young, n. j. (2009). a decade of personalized research assistance. reference services review, 37(2), 207–220. https://doi.org/10.1108/00907320910957233   butler, k., & byrd, j. (2016). research consultation assessment: perceptions of students and librarians. the journal of academic librarianship, 42(1), 83–86. https://doi.org/10.1016/j.acalib.2015.10.011   cole, c., & reiter, l. (2017). online appointment-scheduling for optimizing a high volume of research consultations. pennsylvania libraries: research & practice, 5(2), 138–143. https://doi.org/10.5195/palrap.2017.155   faix, a., macdonald, a., & taxakis, b. (2014). research consultation effectiveness for freshman and senior undergraduate students. reference services review, 42(1), 4–15. https://doi.org/10.1108/rsr-05-2013-0024   fournier, k., & sikora, l. (2015). individualized research consultations in academic libraries: a scoping review of practice and evaluation methods. evidence based library & information practice, 10(4), 247–267. https://doi.org/10.18438/b8zc7w   fournier, k., & sikora, l. (2017). how canadian librarians practice and assess individualized research consultations in academic libraries: a nationwide survey. performance measurement and metrics, 18(2), 148–157. https://doi.org/10.1108/pmm-05-2017-0022   isbell, d. (2009). a librarian research consultation requirement for university honors students beginning their theses. college & undergraduate libraries, 16(1), 53–57. https://doi.org/10.1080/10691310902754072   ishaq, m. r., & cornick, d. p. (1978). library and research consultations (larc): a service for graduate students. rq, 18(2), 168–176. retrieved from https://www.jstor.org/stable/25826127   kani, j. (2017). evernote in the research consultation: a feasibility study. reference services review, 45(1), 67–78. https://doi.org/10.1108/rsr-05-2016-0034   kolendo, j. (2016). the extra credit consultation in two academic settings. the reference librarian, 57(3), 247–253. https://doi.org/10.1080/02763877.2016.1129246   kuglitsch, r. z., tingle, n., & watkins, a. (2017). facilitating research consultations using cloud services: experiences, preferences, and best practices. information technology and libraries, 36(1), 29–35. https://doi.org/10.6017/ital.v36i1.8923   magi, t. j., & mardeusz, p. e. (2013). why some students continue to value individual, face-to-face research consultations in a technology-rich world. college & research libraries, 74(6), 605–618. https://doi.org/10.5860/crl12-363   martin, p. n., & park, l. (2010). reference desk consultation assignment: an exploratory study of students’ perceptions of reference service. reference & user services quarterly, 49(4), 333–340. https://doi.org/10.5860/rusq.49n4.333   meyer, e., forbes, c., & bowers, j. (2010). the research center: creating an environment for interactive research consultations. reference services review, 38(1), 57–70. https://doi.org/10.1108/00907321011020725   reinsfelder, t. l. (2012). citation analysis as a tool to measure the impact of individual research consultations. college & research libraries, 73(3), 263–277. https://doi.org/10.5860/crl-261   rogers, e., & carrier, h. s. (2017). a qualitative investigation of patrons’ experiences with academic library research consultations. reference services review, 45(1), 18-37. https://doi.org/10.1108/rsr-04-2016-0029   sikora, l., fournier, k., & rebner, j. (2019). exploring the impact of individualized research consultations using pre and posttesting in an academic library: a mixed methods study. evidence based library & information practice, 14(1), 2–21. https://doi.org/10.18438/eblip29500   suarez, d. (2013). making sense of liaison consultations: using reflection to understand information-seeking behavior. new library world, 114(11/12), 527-541. https://doi.org/10.1108/nlw-04-2013-0036   watts, j., & mahfood, s. (2015). collaborating with faculty to assess research consultations for graduate students. behavioral & social sciences librarian, 34(2), 70–87. https://doi.org/10.1080/01639269.2015.1042819   appendix a student consultation questionnaire questions   hello, you had a research consultation with a u-m librarian this past winter semester. how well do you remember this consultation? ●        not at all ●        a little ●        fairly well ●        very well   approximately how long did your consultation take? ●        0–10 minutes ●        10–20 minutes ●        20–30 minutes ●        30+ minutes   what was your program of study at the time of your consultation? ●        bachelor’s ●        master’s ●        phd ●        postdoc ●        other   what is your expected year of graduation?   what major, program, or department were you affiliated with at the time of your consultation?   was this for a class? ●        yes ●        no   (if “was this for a class? yes”) what class was this for? (e.g., english 125) [optional]   can you briefly summarize what your consultation was about?   what did you hope to learn from the consultation?   thinking about your answer above, do you feel that the consultation met your expectations? ●        significantly ●        somewhat ●        a little ●        not at all   (if “thinking about your answer above, do you feel that the consultation met your expectations? not not at all”) in what ways did the consultation meet your expectations?   (if “thinking about your answer above, do you feel that the consultation met your expectations? not at all”) please tell us why the consultation service did not meet your expectations.   what, if anything, did you discuss during the consultation that was new to you?   were you working on a project/assignment when you scheduled your consultation? ●        yes ●        no   (if “were you working on a project/assignment when you scheduled your consultation? yes”) to what extent did this service improve the quality of your project/assignment? ●        significantly ●        somewhat ●        a little ●        not at all   (if “to what extent did this service improve the quality of your project/assignment? not not at all”) in what ways did the consultation service improve the quality of your project/assignment?   (if “to what extent did this service improve the quality of your project/assignment? not at all”) please tell us why you didn’t think that the consultation service improved the quality of your project/assignment.   did you come to this consultation because you needed sources (e.g., materials such as books or articles to use for your research)? ●        yes ●        no   (if “did you come to this consultation because you needed sources (e.g., materials such as books or articles to use for your research)? yes”) were you able to locate higher quality sources than before the consultation? ●        yes ●        no   (if “did you come to this consultation because you needed sources (e.g., materials such as books or articles to use for your research)? no”) please explain why not.   (if “did you come to this consultation because you needed sources (e.g., materials such as books or articles to use for your research)? yes”) what made these sources better for your project/assignment?   (if “were you working on a project/assignment when you scheduled your consultation? yes”) did you get any feedback related to your project/assignment from your instructor or supervisor? ●        yes ●        no   (if “did you get any feedback related to your project/assignment from your instructor or supervisor? yes”) what feedback did you receive from your instructor or supervisor?   were you able to use something that you discussed during your consultation for your project/assignment? ●        yes ●        no   (if “were you able to use something that you discussed during your consultation for your project/assignment? yes”) how were you able to apply what you discussed?   were you able to apply something that you discussed during your consultation to other projects/assignments (i.e., different projects/assignments than the one for which you scheduled a consultation)? ●        yes ●        no   (if “were you able to apply something that you discussed during your consultation to other projects/assignments (i.e., different projects/assignments than the one for which you scheduled a consultation)? yes”) how were you able to apply what you discussed?   how would you describe the consultation service to a friend?   would you recommend the consultation service to a friend? ●        yes ●        no   (if “would you recommend the consultation service to a friend? no”) why not?   if you would be willing to let us contact you with potential follow-up questions, please enter your email address.   appendix b librarian consultation interview questions   introductions   [ask the librarian what they know about the project, then fill in gaps based on what they don’t know yet. if they don’t know about the project, read the summary below.]   for those who conducted multiple consultations, do they want to talk about each consultation individually or all at the same time?   1.       what was the student’s project? 2.       did the research consultation take the full half hour? otherwise, how long did it take? 3.       what prior research on the topic had the student conducted? 4.       what was your impression of the student’s research skills at the start of the session? 5.       what goals did you have for the session? that is, what did you want the students to walk away from the session having learned? 6.       what steps did you take to help the student answer their questions? 7.       what did the student learn? a.       skills b.       concepts 8.       on a scale of one to ten, ten being highly successful, one being not successful, how successful was the session?  why did you give them this rating? 9.       did the student contact you after the consultation? 10.    in your estimation, was the research consultation service the most appropriate mode for what the student needed to learn? a.       if no, what other mode might have been more appropriate? b.       if yes, what did the student learn from this session that would have been difficult to teach in another mode? 11.    is there anything that you wish that the student had learned during the consultation? 12.    may we contact you with any follow up questions?     evidence summary   thematic analysis of videos suggests that ya space design should be user-driven, user-centered, and flexible enough to enable multiple uses   a review of: agosto, d. e., bell, j. p., bernier, a. & kuhlmann, m. (2015). “this is our library, and it’s a pretty cool place”: a user-centered study of public library ya spaces. public library quarterly, 34(1), 23-43. doi: http://dx.doi.org/10.1080/01616846.2015.1000777   reviewed by: ann glusker reference/consumer health librarian business, science and technology department the seattle public library seattle, washington, united states of america email: ann.glusker@spl.org   received: 28 aug. 2015 accepted: 19 oct. 2015      2015 glusker. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   objective – to add empirical evidence to the study of young adult (ya) spaces and creation of related guidelines by investigating librarians’ and teens’ perceptions of ya spaces and their use in public libraries.   design – qualitative thematic analysis of video data content.   setting – 25 public libraries in the united states of america.   subjects – a librarian, and a teen of their choosing, at each of the 25 selected libraries.   methods – the researchers chose 25 public libraries randomly from a list of 257 libraries profiled from 2005-2010 in library journal’s annual index of new and renovated libraries. each of the 25 libraries was sent a video camera, and the selected librarian and teen each recorded a narrated tour of their library’s ya space. the cameras were returned to the researchers, who had the audio of the recordings transcribed, referring back to the video for any narration that needed clarification. thematic analysis was performed on the transcriptions to identify majorand sub-categories of space and space-use characteristics. empirical results were reported as a numerical figure representing the number of videos (n=42 total usable videos) in which a particular theme was mentioned at least once.   main results – five main categories of ya space characteristics recommended by teens and librarians emerged from the thematic analysis. ya spaces need to facilitate: physical comfort; both leisure activity and information needs; both academic activity and information needs; a sense of ownership by teens; and improved marketing of the spaces as well as clear displays of library policies regarding the spaces. sub-categories were also listed in each instance, as were counts and percentages of numbers of videos in which a theme was mentioned at least once.   conclusion – ya spaces in public libraries need to be user-centered, and the arrangements need to be flexible enough to be used for multiple purposes. teens need to be involved in and drivers of their design and ongoing use. in addition, teens’ needs for social interaction, and leisure as well as academic support, must be explicitly incorporated into any planning. librarians and libraries must adjust their focus on resources to better mirror teens’ focus on the activities that happen in ya spaces, both online and in person. in order to serve ya patrons, libraries must also actively promote ya spaces. this initial exploratory empirical investigation can inform future much-needed research on improving ya space planning. research is also needed to examine the evolution of libraries’ ya investments as teen patrons’ needs change over time.   commentary   this study is an admittedly initial exploration of the ways in which empirical research into the development and maintenance of ya spaces might be conducted. with empiricism as their stated goal, the authors could have profitably used existing non-evidence-based practitioner literature as a basis for their exploration, for example, asking participants to rate a list of known ya space characteristics. instead they started from scratch, although excellent publications from the practitioner world exist, one of which has a list of characteristics which almost exactly match the results of this study, and another of which is cited and appreciated in a paper by this study’s third author (gorman & suellentrop, 2009; farrelly, 2011; bernier, males & rickman, 2014).   for this evidence summary, these methodologies were systematically assessed using the critical appraisal checklist by glynn (glynn, 2006). there are multiple inherent limitations in the research method chosen. the authors indicate that analyzing video-based data is “notoriously difficult”. for example, due to institutional review board concerns, they weren’t able to film minor subjects’ faces, so they had to rely on audio transcripts, and thus did not have facial expressions or any sense of demographics. librarian subjects chose the teen subjects for the study, and this, as well as the subjects’ knowledge of participating in the study, created bias. furthermore, the empirical reporting could only go as far as presenting the number of videos in which a theme occurred, not its emphasis within a video — and the details of the methodology for the thematic analysis were not discussed. in the end, given these methodological limitations, the contention that the research findings related to ya space planning are now empirically supported seems uncertain.   however, research of this kind has to start somewhere and this study was carefully crafted and reported, with a useful section on how these findings can be followed up in future research.   this study is one arm of a multi-armed project and it may be worth reading some of the papers published by the project’s other researchers. in fact, kuhlmann, agosto, bell and bernier published a more informal piece about the same data as in the study reviewed here (2014). the kuhlmann et al. article presents this study’s methods and findings more accessibly, and so may be a good access point for understanding its implications. interestingly, kuhlmann and colleagues imply that both librarians and teens “may not be fully aware of what to expect from public spaces in general or the full range of possible improvements their library could provide”. they suggest that tools be developed to help librarians and teens give “meaningful feedback” as to what is working and what is not. this is a focus that was not included in the article reviewed here, and which would have been a useful addition.   references   bernier, a., males, m. & rickman, c. (2014). “it is silly to hide your most active patrons”: exploring user participation of library space designs for young adults in the united states. library quarterly: information, community, policy, 84(2), 165-182. doi: http://dx.doi.org/ 10.1086/675330   farrelly, m. g. (2011). make room for teens!: reflections on developing teen spaces in libraries. santa barbara, ca: libraries unlimited.   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi: http://dx.doi.org/10.1108/07378830610692154   gorman, m. & suellentrop, t. (2009). connecting young adults and libraries: a how-to-do-it manual (4th ed.). new york, ny: neal-schuman publishers, inc.   kuhlmann, m., agosto, d. e., bell, j. p. & bernier, a. (2014). learning from librarians and teens about ya library spaces. public libraries, 53(3), 24-28.     ebl 101   looking to the literature: domains to help determine where to look   virginia wilson shirp coordinator health sciences library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   originally published in: evidence based library and information practice, 4(2), 182–184. https://ejournals.library.ualberta.ca/index.php/eblip/article/view/6174/5359     received: 26 april 2009 accepted: 26 april 2009      2016 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   last time in ebl 101, we looked at matching question types to study designs. the process of determining the type of question you’ve asked can point you in the direction of the research design best suited to answering your question. this time around, we’ll stay with the question in order to help determine where to look for evidence in the literature.   it’s been suggested, and further confirmed through research, that many if not all questions generated in librarianship fall under one or more of the following six domains:   collections education information access and retrieval management professional issues reference/enquiries   this classification scheme was originally proposed to deal with published research in order to make it “easier to map sources, study types and search strategies pertaining to each domain” (koufogiannakis, slater, and crumley 228).  deciding in which domain your question belongs is helpful in the ebl process. the domain can point you to the type of literature likely to contain articles pertaining to your topic.   as an aside, there is a bit of discrepancy around the domains in some ebl literature that was published around 2004. for example, in booth and brice’s evidence-based practice for information professionals: a handbook, reference to the six domains includes marketing as a separate domain. koufogiannakis, slater, and crumley, in their content analysis of librarianship research, tested the six initially proposed domains, and found that evidence for marketing research was limited and removed it as a separate domain from the list, including it instead in the management domain. additionally, based on the results of their content analysis, they added a domain: professional issues. so 2004 saw the move from the initially proposed classification taxonomy to the six domains listed above.    it’s time to take your question and find out in which domain it belongs. each domain has its own definition:   collections: building a high-quality collection of print and electronic materials that is useful, cost-effective and meets the users’ needs. education: incorporating teaching methods and strategies to educate users about library resources and how to improve research skills. lis education is a subset of the education domain, and it specifically pertains to the professional education of librarians. information access and retrieval: creating better systems and methods for information retrieval and access. management: managing people and resources within an organization. this includes marketing and promotion as well as human resources. professional issues: exploring issues that affect librarianship as a profession. reference/enquiries: providing service and information access that meets the needs of library users. (koufogiannakis, slater, and crumley, table 2)   now that you have more information about your question, it’s time to turn to the literature. the challenging part of evidence-based librarianship is that not only should you look in the library and information studies (lis) literature, but also further afield in disciplines that relate to your question’s domain(s). the starting point is usually the traditional lis databases, in addition to the suggestions below.   collections: collection management as a domain is almost completely covered within the library literature. in this case, the library literature includes “the wider scope offered by museum and archival activities” (booth, collection, 187). education: research studies about education can be found in several places: educational resources information center (eric), psycinfo, and the campbell collaboration. however, brice and carlson report that “much education research sits in the ‘grey literature’” (168). information access & retrieval: in the last few decades, more research into digital libraries has been found in the information retrieval research. additionally, information science and computing literature also looks at information retrieval and access. it’s been noted that there is not a lot of cross over between public and commercial sectors and the academic sector. thus, “the wheel is being reinvented” (booth, evidence, 233). therefore, there should be more impetus to search broadly across different disciplines. management: informative research on a variety of management topics can be found in the harvard business review, as well as in general management databases such as abi-inform; in sector-specific management databases and collection, i.e. health business elite; and in resources covering specific facets of management, i.e. marketing, hr, etc. (booth, towards, 201). professional issues: this is a wide-ranging domain that explores issues pertaining to librarianship as a profession. the traditional lis databases will most likely be the only stop to make. however, depending on the question, literature from other professions could be consulted. reference/enquiries: questions that fall under the reference domain can also be explored in literature pertaining to “the wider subject discipline of human interaction and communication” which also includes “research studies from psychology (psycinfo), computer science (inspec), and other behavioural and social sciences” (booth, examining, 150).   it is important to think outside the lis box when looking for evidence to inform your question. thinking about the question in terms of domains can help you do this and in turn broaden the range of useful resources. but what happens if there is no access to for-fee databases? next time: open access sources of lis evidence.   works cited   koufogiannakis, denise, linda slater, and ellen crumley. “a content analysis of librarianship research,” journal of information science 30.3 (2004): 227-39.   brice, anne and cindy carson. the contribution of evidence-based practice to educational activities. evidence based practice: an information professionals handbook. eds. andrew booth and anne brice. london: facet, 2004.164-77.   booth, andrew. an evidence-based approach to collection management. evidence based practice: an information professionals handbook. eds. andrew booth and anne brice. london: facet, 2004. 185-95.   . evidence-based perspectives on information access and retrieval. evidence based practice: an information professionals handbook. eds. andrew booth and anne brice. london: facet, 2004. 231-46.   . examining the evidence base for reference services and enquiry work. evidence based practice: an information professionals handbook. eds. andrew booth and anne brice. london: facet, 2004. 148-58.   . towards evidence-based management. evidence based practice: an information professionals handbook. eds. andrew booth and anne brice. london: facet, 2004. 200-09.   evidence summary   bibliometric analysis identifies publication trends and most common research topics related to internet health information seeking behaviour   a review of: li, f., li, m., guan, p., ma, s., & cui, l. (2015). mapping publication trends and identifying hot spots of research on internet health information seeking behavior: a quantitative and co-word biclustering analysis. journal of medical internet research, 17(3), e81. http://dx.doi.org/10.2196/jmir.3326   reviewed by: lindsay alcock head, public services health sciences library memorial university of newfoundland st. john’s, newfoundland, canada email: lalcock@mun.ca   received: 22 sept. 2015 accepted: 19 nov. 2015      2015 alcock. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to identify research and publication trends related to health-seeking information behaviour on the internet.   design – bibliometric analysis, publication trends, and co-word biclustering analysis.   setting – academic journals.   subjects – journal articles retrieved from pubmed meeting eligibility criteria, and articles selected through hand-searching of the top three journals publishing in the identified area of research.   methods – a search for relevant articles was performed in pubmed and supplemented by manual searching of the top three journals in the field, yielding a total of 2,780 articles. following a high concordance rate on screening agreement, researchers identified a total of 533 articles for inclusion. these articles were considered to be representative of all the articles published on internet health-seeking behaviour as of september 2014. data deemed essential to biclustering co-word analysis included article title, author, institution, country, source, publication year, and mesh terms, and was collected in both xml and medline formats to ensure information exhaustivity for subsequent analysis. analysis of the distribution of data, as well as major mesh frequency ranking, allowed researchers to identify the most active journals in the subject area, while biclustering for highly frequent mesh terms determined hot spots of research. researchers used both mountain and matrix visualization to further illustrate semantic relationships of mesh terms and the framework for the analysis of research hot spots. co-word analysis facilitated the identification of like-articles based on major mesh indexing, while cluster analysis utilized a matrix grouping to identify themes. by combining this information and reorganizing the matrix, researchers were able to highlight the most common themes.   main results – researchers identified ten research “hot spots,” the most prolific research topics, thus providing the top subject areas of research published in the literature related to internet health-seeking behaviour. top subjects include health information seeking behaviour related to hiv infection or sexually transmitted diseases; information seeking behaviour of students and of patients with cancer; consumer trust in online health information; behaviour of internet health information seeking through mobile apps; the interaction between physician-patient relations/communications and internet use; personal preference and computer literacy related to internet use; and the use of social media by parents. in terms of publishing rates, the number of papers published on health information seeking behaviour has increased consistently since 1985, when only one paper was published, to 2013, in which 114 papers were published. authors from 42 countries or regions contributed to the body of relevant literature, with authors from the united states of america accounting for over half of published papers. just over 96% of articles were published in english. of the 253 journals identified as publishers of these articles, eight published over one-third of all the identified articles. the journal of medical internet research published the most articles on this topic.   conclusion – bibliographic analysis identified both subject and publication trends related to internet health information seeking behaviour. publication rates of research in the area of internet health information have increased steadily since the first article was published in 1985. the bulk of the research tends to fall within ten identified hot spots, or research topics, according to a bibliometric analysis of indexing.   commentary   the authors provide several resources supporting the fact that a significant proportion of internet searches are related to health information. bibliometric analyses on research trends in this subject area are sparse and therefore this is a relevant and timely study that helps to fill a gap in the literature.   the articles identified for analysis were obtained through a broad, and arguably inefficient, pubmed keyword search, using only those indexed with mesh terms. it is unclear why only major mesh terms were ultimately utilized in the bibliometric analysis, though this is likely because such terms would identify the primary rather than secondary focus of the article. the eligibility criteria of pubmed search results ensured that articles focused on health information seeking behaviour rather than the evaluation of online health information or other behaviours related to internet use (e.g., online shopping, internet addiction, online gaming). according to the authors, pubmed was selected as the only database to search because it is freely available. they suggest that available research data will increase when other databases become available through open access publication. it is not clear whether the authors had access to additional licensed databases through their institutions, however one would assume that additional resources would be available through academic affiliations. the authors note that using only pubmed is a study limitation, as journals that are not indexed in pubmed could not be included in the analysis. the authors also concede that a lack of standard indexing, along with the timing of the introduction of mesh terms, are factors that may affect the accuracy in identifying research hot spots.   given that complex bibliometric analysis may be unfamiliar to many readers, one of the strengths of this study is the clear definition and description of the methods used in data analysis and interpretation. that said, the research sometimes read as more of a lesson in bibliometric analysis rather than a presentation and discussion of findings to the extent that one would expect in a research publication.   results from this study can be used to inform the development of subsequent research areas since the identification of “hot spots,” by default, implies gaps in the literature. as well, these study results can provide the initial basis of a scoping review on a hot spot topic, aid in decisions related to authors’ journal selection for manuscript submission, and provide data to inform collection development decisions.       evidence summary   health sciences librarianship's status as a profession is unclear, according to its members   a review of: koenig, r. a., rodriguez, v. a., & sima, a. p. (2021). attitudinal attributes of professionalism in health sciences librarians. journal of library administration, 61(1), 1–20. https://doi.org/10.1080/01930826.2020.1845544   reviewed by: samantha j. kaplan research & education librarian, liaison to the school of medicine duke university medical center library & archives durham, north carolina, united states of america email: samantha.kaplan@duke.edu   received: 6 mar. 2023                                                                    accepted:  27 apr. 2023      2023 kaplan. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30340     abstract   objective – to determine health sciences librarians' attitudes toward professionalism and to examine relationships between professionalism attributes and participant characteristics as defined by the richard h. hall professionalism inventory.   design – cross-sectional online survey using the richard h. hall professionalism inventory.   setting – electronic mailing lists of the medical library association (mla), the association of academic health sciences libraries (aahsl), the association of college and research libraries (acrl) health sciences interest group, and the canadian health libraries association (chla).   subjects – there were 430 participants.   methods – the online survey, created in redcap, was distributed electronically across multiple mailing lists during june and july of 2019. quantitative analysis included descriptive statistics and anova conducted in r with reliability determined by cronbach's alpha.   main results – professionalism scores for health sciences librarians were lowest in public service and self-regulation, and highest in professional organization as referent, autonomy, and sense of calling. individuals with a degree in health sciences scored lower on a sense of calling than individuals with library and information science (lis) degrees. faculty benefits such as tenure decreased sense of calling. there were statistically significant differences according to role (e.g., archives, administration). subject specialty librarians had lower scores in most attributes.    conclusion – health sciences librarianship does not clearly meet the criteria of a profession. its heterogeneity of specializations and receptiveness to diverse backgrounds and perspectives are possible threats to its ability to create a cohesive identity. further, duties that can be considered non-library work appear to correlate with lower professionalism scores, even when they are associated with faculty status.   commentary   this study was appraised using the center for evidence based management's critical appraisal checklist for a cross-sectional study (2014). the study addressed a clearly focused question with an appropriate methodology. the sample selection was clearly described and, while it likely introduced bias, the authors acknowledge this in their manuscript, increasing the trust in their findings. assessing the representativeness of this sample is challenging because so much of the sample was derived from subscribers of electronic mailing lists that require membership to a professional organization, a potential financial barrier. however, this does potentially offer insight into why involvement with professional organizations consistently received higher scores than other markers of professionalism. additionally, the utilization of multiple mailing lists means we cannot truly determine if there was a satisfactory response rate, as the number of members is unknown. the proportion of respondents who were members of multiple mailing lists surveyed is also impossible to ascertain. the measurements can be considered valid and reliable, as the authors used a validated tool. statistical significance was assessed, and confidence intervals were provided. overall, despite the challenges of the sample, the results have meaningful applications to medical center and health sciences libraries.   while this study aimed to determine if health sciences librarianship qualifies as a profession according to its members, its most meaningful findings are probably for the administrators and managers of health sciences and medical center libraries. faculty status benefits and expectations cannot consistently be said to improve professional identity. archivists, administrators, and collection development librarians had higher scores than their peers in subject specialty, reference, and research services positions. these latter positions are outwardly patron-focused and typically involve higher interaction rates with library patrons. the authors identify some possibilities to explain this, particularly the likelihood of librarianship as a second career for these individuals and that "librarians working in subject-oriented roles must often learn new skills and expand their services," contributing to burnout (koenig et al., 2021, p. 15). regardless of cause, these findings should be concerning because the individuals most likely to interface with patrons are the least likely to hold a strong professional identity as health sciences librarians. efforts should be made to understand the nuance of these findings. at the same time, other research could explore how to help these librarians balance the perceived and tangible burden of maintained subject matter expertise while preserving connection to librarianship.   the authors do address that the source of their sample (professional electronic mailing lists) likely contributed to professional organization as referent receiving higher scores in general. the potential for bias here is tricky. while it is easy to assert that members of a professional organization will likely value it, it is also true that many health sciences librarians must maintain these memberships and may not view it as a straightforward benefit. thus, while membership in a professional organization or guild is a key component of the richard h. hall professionalism inventory instrument, it is less clear how valuable it is as an indicator for health sciences librarianship. while the richard h. hall criteria are vetted and validated to determine a profession, librarianship is known for its adherence to core values, such as lifelong learning and a right to privacy (ala, 2019). asking respondents about core values and seeing how agreement with those values correlate with the richard h. hall criteria would have provided greater context to the findings.   in conclusion, we cannot confidently consider health sciences librarianship as a profession according to this study, in that scores in professionalism, professional organization, public service, autonomy, self-regulation, and sense of calling were not homogenous and strong. however, the exact reason for this cannot be said, as some members of the proposed profession do strongly identify with some of these attributes. further research is needed to understand the source of these attitudinal differences, both for the long-term future of the profession and the real-time work life of its members.   references   american library association. (2019, january). core values of librarianship.  http://www.ala.org/advocacy/intfreedom/corevalues   center for evidence based management. (2014, july). critical appraisal checklist for a cross-sectional study. https://cebma.org/wp-content/uploads/critical-appraisal-questions-for-a-cross-sectional-study-july-2014-1.pdf   koenig, r. a., rodriguez, v. a., & sima, a. p. (2021). attitudinal attributes of professionalism in health sciences librarians. journal of library administration, 61(1), 1–20. https://doi.org/10.1080/01930826.2020.1845544     microsoft word es6_937_final_final evidence based library and information practice 2008, 3:1   72 evidence based library and information practice         evidence summary    collaborative chat reference service effectiveness varies by question type for public  library patrons    a review of:  kwon, nahyun. ʺpublic library patronsʹ use of collaborative chat reference service: the  effectiveness of question answering by question type.ʺ library & information science  research 29.1 (mar. 2007): 70‐91.      reviewed by:  stephanie hall  chief librarian, powell river public library  powell river, british columbia, canada  e‐mail: shall@powellriverlibrary.ca      received : 4 december 2007      accepted : 11 february 2008      © 2008 hall. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    abstract    objective – to assess the effectiveness of a  collaborative chat reference service in  answering different types of question.   specifically, the study compares the degree  of answer completion and the level of user  satisfaction for simple factual questions vs.  more in‐depth subject‐based reference  questions, and for ‘local’ (pertaining to a  particular library) and non‐local questions.    design – content analysis of 415 transcripts  of reference transactions, which were also  compared to corresponding user satisfaction  survey results.    setting – an online collaborative reference  service offered by a large public library  system (33 branch and regional locations).   this service is part of the metropolitan co‐ operative library system: a virtual reference  consortium of u.s. libraries (public,  academic, special, and corporate) that  provides 24/7 service.     subjects – reference librarians from around  the u.s. (49 different libraries), and users  logging into the service via the public  library system’s portal (primarily patrons of  the 49 libraries).   evidence based library and information practice 2008, 3:1   73 method – content analysis was used to  evaluate virtual reference transcripts  recorded between january and june, 2004.   reliability was enhanced through  triangulation, with researchers comparing  the content analysis of each transcript  against the results of a voluntary exit  survey.  of 1,387 transactions that occurred  during the period of study, 420 users  completed the survey and these formed the  basis of the study, apart from 5 transactions  that were omitted because the questions  were incomprehensible.  questions were  examined and assigned to five categories:  “simple, factual questions; subject‐based  research questions; resource access  questions; circulation‐related questions; and  local library information inquiries” (80‐81).   answers were classed as either “completely  answered, partially answered or  unanswered, referred, and problematic  endings” (82).  lastly, user satisfaction was  surveyed on three measures: satisfaction  with the answer, perceived staff quality, and  willingness to return. in general, the  methods used were clearly described and  appeared reliable.    main results – distribution of question types:   by far the largest group of questions were  circulation‐related (48.9%), with subject‐ based research questions coming next  (25.8%), then simple factual questions  (9.6%), resource access questions (8.9%), and  local library information inquiries (6.8%).    effectiveness of chat reference service by question  type: no statistically significant difference  was found between simple factual questions  and subject‐based research questions in  terms of answer completeness and user  satisfaction.  however, a statistically  significant difference was found when  comparing ‘local’ (circulation and local  library information questions) and ‘non‐ local’ (simple factual and subject‐based  research questions), with both satisfaction  and answer completeness being lower for  local questions.        conclusions – the suggestion that chat  reference may not be as appropriate for in‐ depth, subject‐based research questions as it  is for simple factual questions is not  supported by this research.  in fact, the  author notes that “subject‐based research  questions, when answered, were answered  as completely as factual questions and  found to be the question type that gives the  greatest satisfaction to the patrons among all  question types” (86).    lower satisfaction and answer completion  were found among local vs. non‐local  queries.  additionally, there appeared to be  some confusion among patrons about the  nature of the collaborative service – they  often assumed that the librarian answering  their question was from their local library.   the author suggests some form of triage to  direct local questions to the appropriate  venue from the outset, thus avoiding  confusion and unnecessary referrals. the  emergence of repetitive questions also  signalled the need for the development of  faqs for chat reference staff and the  incorporation of such questions into chat  reference training.    commentary    the study uncovers an important nuance in  the effectiveness of virtual reference service  – that is the lower degree of satisfaction and  answer completion for local vs. non‐local  queries – and both kwon’s question  typology itself and her eminently practical  suggestions for improving user satisfaction  (introduction of question triage, review of  policies, staff training, development of  faqs, coordination of web page design, and  so on) will be of interest both to future  researchers and to consortia and individual  institutions participating in chat reference.   evidence based library and information practice 2008, 3:1   74 it would be interesting to compare the  degree of satisfaction for local and non‐local  queries in the physical vs. the virtual  environment, in addition to looking at how  such questions are distributed in libraries,  and to which types of staff (i.e., clerical vs.  professional).  given the preponderance of  circulation‐related queries in the study’s  sample, it may be that the cost implication  of using professional reference staff to  respond to routine inquiries is large enough  not to be ignored.      kwon notes two limitations of the study  under review.  firstly, it is limited to queries  by patrons in a single library system using a  single type of software.  she suggests further  research in a different setting to validate the  findings.   secondly, she notes that only  those transactions where the patron  completed the voluntary user survey were  included in the study, and that those  patrons may have different characteristics  from those who declined the survey.   additionally, it should be noted that the  study is limited to the u.s. context. in  general, however the study is well‐designed  with various measures being taken to ensure  validity and reliability, for example, the use  of a composite variable to determine user  satisfaction and the testing of inter‐coder  reliability.    evidence based library and information practice 2011, 6.3 94 evidence based library and information practice news/announcements eblip6 conference award winners © 2011. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. the eblip6 conference was held at the university of salford, u.k. from june 27‐30, 2011. the conference was a great success and was attended by 170 delegates from 27 countries. the december issue of evidence based library and information practice will include a feature section on the conference, but in the meantime many of the conference presentations and photos from the conference are available on the conference website at http://www.eblip6.salford.ac.uk. over the three main days of the conference, there were over 60 presentations given in seven parallel sessions and 20 poster presentations, some of whom also presented at “one minute poster madness” sessions. as at previous conferences, presentations and posters were judged by members of the international programme committee (ipc), and for the first time at an eblip conference, the audience was also asked to nominate their favourite presentations and posters. the winner of the best presentation was the same for both the ipc and audience vote: o kate davis, queensland university of technology, and zaana howard, swineburn university of technology – redesigning evidence based practice for wicked problem solving. winner of the ipc best poster was: o philip kroth, holly philips and jonathan eldredge, university of new mexico – evaluation of an evidence‐based scholarly communication conference focused on support of translational investigators using a prospective longitudinal cohort design; best poster as voted for by conference delegates was: o mary dunne, health research board, ireland – barriers and facilitators to research use: the role of library and information services. judging for the best presentation was very close, and a number of presentations received commendations from the ipc. these included: o carolynn rankin and avril brock, leeds metropolitan university – the potential of generic social outcomes in http://www.eblip6.salford.ac.uk/� evidence based library and information practice 2011, 6.3 95 promoting the positive impact of the public library o denise koufogiannakis, university of alberta – considering the place of practice based evidence within eblip o jackie druery, nancy mccormack and sharon murphy, queen’s university – are best practices really best? a review of "best practices" in library literature o joanne marshall, university of north carolina – measuring value of health library and information services: replicating the rochester study o lisa m given, university of alberta – evidence based web design: a user‐ driven approach to providing consumer health information o lorie kloda, mcgill university, denise koufogiannakis, university of alberta and alison brettle, university of salford – the impact of evidence summaries on lis practitioners highly commended posters as voted for by the ipc included: o elaine garratt and lucy reid, royal college of obstetricians and gynaecologists – clinical query services from the royal college of obstetricians and gynaecologists; o karen walshe, the british library – defining a new role: the embedded research information manager; o sandy campbell, dale storie, brettany johnson and robert hayward, university of alberta – using a professional continuing medical education simulation to engage undergraduate medical students in information literacy congratulations to all the presentation and poster winners! / evidence based library and information practice evidence summary   more academic librarians in arkansas with faculty status and rights but decreased benefits and increased responsibility   a review of: vix, h. m., & buckman, k. m. (2011). academic librarians: status, privileges, and rights. journal of academic librarianship, 38(1), 20-25. doi: 10.1016/j.acalib.2011.11.004   reviewed by: mê-linh lê health sciences centre librarian university of manitoba winnipeg, manitoba, canada email: melinh_le@umanitoba.ca   received: 13 sept. 2012 accepted: 21 jan. 2013      2013 lê . this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to provide cross-comparable information on the number of students per librarian, salary, faculty status, contract lengths, and maternity benefits of academic librarians.   design – online questionnaire.   setting – four-year private, four-year public, and two-year public academic institutions in arkansas.   subjects – academic library deans and directors were surveyed three times over a six-year period.   methods – three surveys were sent to library deans and directors of four-year private, four-year public, and two-year public academic institutions in arkansas in 2007, 2009, and 2011. the surveys were created by the college and university library division of the arkansas library association, with questions created based on reports from the association of college and research libraries (e.g., standards and statements). committee members tested the survey before distribution. over the course of the six-year period the questions were modified and were chosen to ensure that respondents could easily answer them (i.e., no questions on topics such as retirements, vacation, which can vary significantly from librarian to librarian). all responses were confidential.   main results – the 2007 survey had a 78% response rate (n=35/45); the 2009 survey had a 93% response rate (n=42/45); and the 2011 survey had a 90% response rate (n=44/49). while the survey covered a number of topics (shown in supplementary material online), the article focused on five areas of interests and had the following findings. 1) the number of students per librarian is increasing at four-year private and two-year public institutions. while the data shows a decrease in the ratio at four-year public institutions the authors believe this is due to the addition of new institutions in the follow-up surveys, one of which had a very low study-to-librarian ration. 2) tenured librarians make more than non-tenured librarians. 3) the number of institutions granting faculty status is increasing at a statistically significant rate at four-year private and two-year public institutions, and has remained relatively constant at four-year public institutions. 4) most libraries have 12-month contracts for librarians, although this has decreased slightly over the survey period. 5) the number of institutions providing paid maternity leave has decreased.   conclusion – this study provides a broad overview of the changing state of academic librarians’ rights and benefits in arkansas over the last six years. some of the trends demonstrated, such as an increase in the number of students per librarian, are potentially troubling and may have a negative impact on the quality of service provided by individual institutions. other trends, such as increases in institutions granting faculty status to librarians and decreasing the number of 12-month contracts, may signal that some institutions are beginning to acknowledge the significance and impact of librarian research. ultimately, this article provides a starting point for other states and provinces to begin collecting similar data in an attempt to understand changing trends in academic libraries.     commentary   while there are large annual surveys on librarian salaries and a large number of works related to librarian rank, there are few studies that look at these issues in order to provide context and a broader understanding on a variety of topics related to academic librarian rights, responsibilities, and status.   although the study methodology is not outlined in great detail, the area that would have benefited from additional clarification is exactly how the academic institutions were identified and why the number of institutions changed over the years. it is also unclear whether all arkansas institutions that could have been contacted were included. furthermore, the issue of faculty status would also be served by more explication. in the literature review, the authors note rightly that with other studies on status, it is hard to make comparisons as institutions have different definitions of what constitutes faculty status and rank. in the summary of results, however, the authors provide no further breakdown of their definition of faculty status. the actual survey asks several detailed questions about the rights of librarians with faculty status (e.g., whether they can participate on the university senate and whether they can be granted tenure), but the article itself does not include any of these findings in its analysis. it is hoped that the authors will include more detailed information in future publications. indeed, the survey instrument shows a large amount of data was collected that was not included in the analysis, but which could be used for future study.   that being said, the information provided in this work could be very helpful for comparison purposes with some caveats. specifically, librarians and administrators must be careful to ensure that they account for other variables that may be at work. for example, when comparing salaries, one question raised in the survey, but not in the article itself, is the issue of merit pay. whether an institution offers merit pay can have a significant impact on a librarian’s total salary, particularly if that librarian has been with the same institution for many years. in addition, if comparing salaries or benefits outlined in this article to other states or provinces, it will be important to take factors such as standard of living costs and taxation levels into consideration.   while the article is brief and collects only data from arkansas librarians, the information included is useful in that it contributes one piece to the broader picture of librarian rights and privileges. it is also one of the few articles that looks at several topics at once (e.g., salary, rank, and benefits). it is hoped that this article will serve as an impetus for other state or provincial library associations to follow suit and begin collecting similar information. if that information is already being collected, then it is suggested that the data be shared publicly in some format. in days of ever-dwindling budgets and cutbacks, any evidence that shows the perceived value of librarians through salary and rank is helpful.   evidence summary   open access papers have a greater citation advantage in the author-pays model compared to toll access papers in springer and elsevier open access journals       a review of: sotudeh, h., ghasempour, z., & yaghtin, m. (2015). the citation advantage of author-pays model: the case of springer and elsevier oa journals. scientometrics, 104(2), 581-608. http://dx.doi.org/10.1007/s11192-015-1607-5   reviewed by: elaine sullo coordinator, information and instructional services himmelfarb health sciences library the george washington university washington, district of columbia, united states of america email: elainej@gwu.edu   received: 30 nov. 2015 accepted: 10 feb. 2015      2016 sullo. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to investigate the citation performance of open access (oa) and toll access (ta) papers published in author-pays open access journals.   design – longitudinal citation analysis.   setting – publications in springer and elsevier’s author-pays open access journals.   subjects – 633 journals published using the author-pays model. this model encompasses both journals where the article processing charge (apc) is required and journals in which authors can request open access and voluntarily pay apcs for accepted manuscripts.   methods – the authors identified apc funded journals (journals funded by mandatory author processing charges as well as those where authors voluntarily paid a fee in order to have their articles openly accessible) from both springer and elsevier, and analyzed papers published in these journals from 2007 to 2011. the authors excluded journals that adopted the apc model later than 2007. to identify springer titles, the authors created a search strategy to identify open access articles in springerlink. a total of 576 journals were identified and double checked in the sherpa-romeo database (a database of copyright and open access self-archiving policies of academic journals) to verify their open access policies. the authors then downloaded the journal content using springerlink, and using springer author-mapper, separated out the open access articles from the toll access articles.    in order to identify the elsevier apc funded journals, the authors referred to “open access journal directory: a-z,” which contained 35 oa journals (p. 584). once the authors consulted “sponsored articles” issued by elsevier and verified titles in sherpa-romeo, they identified 57 journals that fit the “author-pays” model. the bibliographic information was downloaded and oa articles were separated from ta articles. the authors confirmed that all journals were indeed oa publications by downloading the full-text from off-campus locations; they also verified that the journals were using the apc model by visiting each journal’s website.   because of the large number of subject areas of the identified journals, the researchers decided to classify the journals into four broader categories: health sciences, life sciences, natural sciences, and social sciences and humanities. to calculate the impact of oa papers, citation per paper (cpp) was calculated for each subject area. impact values were calculated on an annual basis as well. the researchers calculated the citation advantage of oa articles as the “difference between the open access and toll access impacts in terms of a percentage of the latter” (p. 585).   main results – the authors categorized their findings according to three themes: the growth of apc funded oa papers, the number of oa papers by discipline, and citation advantage of oa vs. ta in general and by subject area.   together, springer and elsevier published 18,654 oa papers in the apc journals; this number represents 4.7% of the 396,760 papers published between 2007 and 2011. while the number of oa and ta papers has been growing annually, the number of oa papers has been growing more rapidly compared to the ta papers.   in terms of subject areas, life sciences had the largest number of oa and ta papers (184,315), followed by health sciences (149,341), natural sciences (121,274), and social sciences and humanities (42,824). natural sciences had the most oa papers (5.7%) in terms of the number of papers in this subject area being oa papers, followed by social sciences and humanities (5.2%), health sciences (4.6%) and life sciences (3.6%).    overall, the researchers found that the impact values of oa papers were larger than those of the ta papers for each year examined. in considering subject areas, in all disciplines except life sciences, the most highly cited paper in the field is an oa paper. in life sciences, the most highly cited ta paper had 2,215 citations, compared to the oa paper, which had 1,501 citations. even though the ta paper had more citations, overall, the oa papers had a higher impact (citation advantage). in health sciences, the most highly cited oa paper received 1,501 citations, which is 1.2 times the most highly cited ta paper, with 1,252 citations. the citation advantage for the oa group is 33.29% higher than the ta group. in natural sciences, the number of citations from the highest cited oa paper is 1,736, or 2.52 times higher than the most highly cited ta paper. the oa papers in this discipline had a 35.95% citation advantage. in social sciences and humanities, the most highly cited oa paper had 681 citations, compared to the ta paper, with 432 citations. for this subject area, the citation impact of the oa paper is 3.14% higher than the ta paper.   conclusions – in sum, the number of article processing charge funded open access papers has grown tremendously in recent years. furthermore, open access papers have a citation advantage over toll access papers, both annually and across disciplines.   commentary   this study will certainly be of great interest to academic librarians, especially those who navigate electronic journal subscriptions and open access publishing options, as well as those who advise faculty on publishing opportunities. the authors note that their research represents the first large scale study to focus on the oa author-pays model for two publishers who have been pioneers in adopting the model. the research adds to the body of published literature related to open access citation advantage. study findings also demonstrate that articles published by the open access author-pays model have an impact on authors’ visibility and impact in their area of scholarly expertise; academic librarians may want to relay this important information to faculty who may be wary about publishing in open access journals.   this study was evaluated using the reliant instrument (koufogiannakis, booth, & brettle, 2006). the significant strengths of this research related to study design include the clearly explained research methodology and data collection procedures. however, readers who are not familiar with the current trends in open access publishing may have to review parts of the article multiple times to understand the types of open access models. the article in general uses many abbreviations – oa, ta, oaca, apc, and cpp, for example – which can be confusing at times. additionally, the way that the citation advantage was calculated might be difficult for readers to understand. fortunately, the authors fully explain, in both tables and in the text, how the calculations translate into meaningful information.   the study had several limitations. readers should keep in mind that the authors’ analysis may have shortcomings, considering that the author-pays oa model is only a small part of the overall oa landscape. also, because many oa papers fall into the “green” model (self-archiving in open access repositories or archives), there may be other confounding variables related to the trends observed in this research. in addition, the research was limited to two large, well-known publishers. as such, the citation advantage of oa papers could be due to factors related to the reputation of the publishers and the associated visibility of journals published by springer and elsevier. furthermore, in assessing the journal citations, other contributing factors were not considered, such as “institution reputation, journals prestige, co-authorship, and impact of the green model” (p. 592). however, due to the large amount of data collected, the broad subject coverage, and a long time span, the conclusions may be generalizable to the entirety of apc open access journals. additionally, the authors noted that in the years studied, the apc open access model was still in its infancy, and thus it may be necessary to conduct further research in the future, when the model is more established.   because of the wide subject area range of the 633 journals that were analyzed, the researchers categorized the journals into broad subject categories. as the authors noted, there is the opportunity for further research in specific disciplines, perhaps analyzing the citation performance in narrower subject areas.    reference   koufogiannakis, d., booth, a., & brettle, a. (2006). reliant: reader’s guide to the literature on interventions addressing the need of education and training. library and information research, 30(94), 44-51. retrieved from http://www.lirgjournal.org.uk/lir/ojs/index.php/lir/article/view/271 editorial   what it means to be an international journal   lorie kloda editor-in-chief associate university librarian, planning and community relations concordia university montreal, quebec, canada email: lorie.kloda@concordia.ca      2015 kloda. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   welcome to the last issue of 2015, and the 40th issue of evidence based library and information practice (eblip)! it’s hard to believe it has been 10 years since the inaugural issue of the journal. since then the journal has grown in many ways. eblip now has a larger editorial board with more associate editors, an editorial intern, and a lead copyeditor. the journal also now boasts a team of editorial advisors composed of past members of the editorial board, as well as a writing assistance team. the content of eblip has also grown in scope. in addition to commentaries, articles, evidence summaries, classics, and news/announcements, it now also includes occasional reviews, using evidence in practice papers, and the research in practice column. over the years, the journal has occasionally included a special features section devoted to particular themes, such as public libraries, or on highlighting a recent conference or symposium, such as this issue’s feature on the recent eblip8 conference, held in brisbane, australia. helen partridge, the conference chair, has written an editorial with an overview of both the conference and the feature section.   the biennial eblip conference is held on different continents, and so far has travelled from the united kingdom to the united states, canada, australia, and sweden. the most recent conference welcomed delegates from 12 countries. like the conference, the eblip journal is international in scope. but what does it mean to be a truly international journal? members of the editorial board frequently discuss questions and issues related to this concept. for example, is our editorial board international enough? how do we include papers by authors who may not have proficiency in the english language? how do we make our journal accessible and interesting to those in non-english speaking regions? moreover, how do we include perspectives of non-english countries that will engage our readers? i don’t have perfect answers to all of the questions, but i think we have made some progress towards becoming increasingly international. in this issue, we not only have the eblip8 conference feature, which includes submissions from all over, and from australia in particular, but also two separate articles by nigerian authors about research pertinent to those in that country. daniel abubakar’s paper on internet access in university libraries demonstrates that seamless internet connectivity in the university library cannot be taken for granted by some students. the article by jerome idiedgbeyan-ose and colleagues reports on the availability and quality of library services in two nigerian rural communities. as the editor-in-chief, i am proud that the journal attracts submissions from all over, and that we are able to continue to publish high quality peer reviewed articles to the evidence base.   i also look forward to the next issue, which will include some extra content in honour of our 10th anniversary. whether you are a long time journal reader or relatively new to eblip, i hope you will continue to find the journal’s content relevant to your practice in the coming decade.   evidence based library and information practice evidence based library and information practice 2011, 6.2 81 evidence based library and information practice using evidence in practice point of need reference service kealin m. mccabe research and learning services librarian university of northern british columbia prince george, british columbia, canada email: mccabek@unbc.ca willow e. fuchs open access adviser (rcs) university of nottingham nottingham, united kingdom email: willow.fuchs@nottingham.ac.uk received: 30 mar. 2011 accepted: 18 apr. 2011 2011 mccabe and fuchs. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. setting the university of northern british columbia is a small, research-intensive university located in prince george, british columbia. staffed by 7 full-time librarians, the geoffrey r. weller library is the sole library on campus, serving just over 4000 full and parttime students. on average, librarians conduct 105 information literacy (il) sessions annually, reaching just over 2,000 students per year. classes are taught in a variety of disciplines, but generally present librarians with only one opportunity to teach students “everything” about the research process in 60 minutes or less. the courses arts 101 and arts 102 present instructing librarians with the unique opportunity to teach four 90 minute sessions. these il sessions and subsequent assignments comprise 20% of the student’s overall course mark. arts 101 and 102 both serve as introductions to the critical thinking, research, writing, and life skills each student needs to successfully complete their degrees. all unbc students are permitted to enrol in these courses, but for students registered in the northern advancement program (nap), they are required. the nap acts as a transitional year for first nations students and students from rural and remote communities entering their first year of university. it bridges the gaps between their life experiences and academic mailto:mccabek@unbc.ca� mailto:willow.fuchs@nottingham.ac.uk� evidence based library and information practice 2011, 6.2 82 culture. the goal of the program is to ensure and enable student learning and success (northern advancement program, program objectives, n.d.). prior to the start of classes in the winter term of 2009, librarians met with instructors to discuss the upcoming term. aside from discussing the assignments and session content, the course instructors asked that emphasis be placed on encouraging students to seek research assistance. with this in mind, an electronic library guide (libguide)1 was created, containing all course materials including handouts and assignments. a chat widget was embedded into the libguide to allow students to access research help anonymously. it seemed only logical to place the widget alongside their assignments so that they could access research help at their virtual point of need. the students were also encouraged to utilize the library’s reference desk and the course librarians’ open-door policy. the open-door policy was intended to serve as a means for students to have further interaction with their librarians. students were invited to stop by for research assistance or simply to say hello. problem everything was organized for the course and the live widget was set up and staffed in anticipation of student requests for research help. as new librarians, we had naïvely expected that we would be inundated with questions relating to their library assignments or assignments from other classes. although students had actively asked for assistance during the il sessions, none of them made use of the services available to them, chat or the open-door policy of the librarians. the fact that they needed help, but were not using any of the services, left us with a sense of bewilderment. in an effort to gain some insight, each student was issued a feedback form on the last day of class. 1 http://libguides.unbc.ca/arts102 there was a common theme to the results. despite having found the sessions overwhelmingly useful, 20% of the students wrote that the wanted more help on an individual basis. specifically, students were asked “what further training or support, if any, would you like? please describe.” their replies were refreshingly straightforward. according to the feedback sheets, students wanted “more one on one sessions”, “maybe more tutorials”, and “one on one work at my pace”. apparently, the students wanted and needed our help, but we were not providing the service in the way that they needed. implementation as a result of the students’ feedback, the librarians approached the course instructors to discuss the idea of having the library conduct reference hours within the first nations centre (fnc), to determine if this would better meet the needs of the students. the first nations centre is a welcoming and nurturing space, which serves as a “home away from home” (first nations centre, n.d.) for many. within this space, students’ academic, social, and technological needs are met. the first nations centre provides students with access to computers, study space, counselling and peer support services, and serves as a supportive community meeting place for all. this is where arts 101 and 102 students were doing their research and completing their assignments. their instructors were also located in the centre. this was the ideal place to be to ensure that students received the one-on-one research and reference assistance they wanted and needed, at their point of need. while the idea to provide the reference hours came from nap student feedback, the service was made available to all students using the first nations centre. point of need reference service started the last week of february 2009, with set hours of service on tuesdays and thursdays from 11 a.m. to 12 p.m.. admittedly, library staff were a little hesitant at the start, wondering what the students http://libguides.unbc.ca/arts102� evidence based library and information practice 2011, 6.2 83 would think of having ‘us’ in ‘their’ space. would students view staff as “intruders in their domain” as experienced by librarians at bowling green state university in the late 1990s (nims, 1998)? our approach to the launch of service was a little unorthodox in that we began providing the service without consulting existing library literature ahead of time. we saw gap in existing library services that needed to be filled. due to the speed with which this service was launched, very little marketing was undertaken. staff publicized the service via inclass announcements and posters placed throughout the fnc, specifically on the walls above the computer terminals, printer, and on the centre’s refrigerator door. on the days that librarians were available for the service, they would make their presence known by introducing themselves to the room, stating their name, position, and why they were there. depending on availability, the librarian would log in to a computer ready to assist. every attempt was made to position ourselves in a highly visible area, whereby we could greet and initiate contact with the students as they arrived. evidence librarians collected very minimal statistics during that first term of the service (winter term 2009). staff created a spreadsheet to record service statistics, recording primarily the date and number of questions asked. in total, participating librarians answered twenty-four questions in twelve service hours. averaging two questions per hour, staff deemed the pilot a success, and decided that the service would continue the following term (fall 2009). prior to the start of the fall 2009 term, staff changed the way that statistics were to be gathered. we hoped that the changes would provide us with a clearer picture of the types of reference transactions occurring in the fnc. participating librarians would now record the nature of the questions they answered. reference transactions occurring in the fnc were to be classified as being either facilitative (directional), or basic or complex in nature (research related). staff defined basic and complex reference transactions based on the amount of time it took to answer the query. basic questions took between one to five minutes to answer, while complex questions required six minutes or more to complete. during the fall term 2009 librarians recorded a total of 26 reference in the first nations centre. using the new recording model, we learned that 15% of questions were facilitative; 46% were classified as basic reference questions, lasting 1-5 minutes in length; and 39% were complex in nature, taking six minutes or more to answer. in total 85% of the 26 questions answered during the fall term were research related. the service was offered again the following term (winter 2010), where 34 questions were asked, a 23.5% increase over the previous term. the recorded data showed 24% of were facilitative whereas the remainder were classified as 41% basic and 35% complex. outcome our hopes were simple, that the students would make use the services available to them. we hoped that our presence in the classroom and in the fnc would help to promote the library and the many services available. as a result of offering reference services in the first nations centre, the number of student appointments with librarians increased dramatically. by the end of the fall 2009 semester, there were only six requests for one-on-one appointments. this number increased dramatically the following term. in the winter 2010 term a total of 20 students made office visits, 11 of which were pop-ins! this pilot project played a significant role in creating and maintaining trusting, meaningful, and truly collaborative relationships with partners outside of the library. as a result of this project, librarians and nap instructors continue to work closely together to create evidence based library and information practice 2011, 6.2 84 meaningful curriculum. the library also enjoys a closer relationship with the students. the students know who we are and what we can do for them. this type of embedded service proved to be a tremendous opportunity to expand the role of the library throughout the university. new opportunities have presented themselves as a result of this new and successful initiative. the library is now involved in nap orientation activities, conducting research and writing workshops for nap students, and conducting workshops for sage (supporting aboriginal graduate enhancement) students. librarians still hold reference service hours in the first nations centre, proving that the influence and importance of the library extends beyond the four walls physically defining traditional library space. reflection upon reflection, the old adage that hindsight is 20/20 holds true. the location and service hours will not change, but the marketing of fnc hours and the collections of statistics would be altered dramatically. as stated previously, very little effort was put into marketing fnc reference hours outside of signs and word of mouth. for the fall 2011 term, additional marketing items will be placed in key orientation materials and acceptance packages. backing hindsight, there is a gaping hole that needs to be filled in relation to the collection of statistics for this service. we know that the service was used, but we forgot to gather information relating to how the students view the service. every effort will be made to solicit user feedback about the service, its location, times, and usefulness to the students and their research needs. such results will be meaningful in shaping the service’s future direction. references nims, j. k., (1998). meeting students on their own turf. research strategies, 16(1), 8589. university of northern british columbia. (n.d.). first nations centre. retrieved 09 may 2011 from http://www.unbc.ca/firstnationscentre/ university of northern britissh columbia. (n.d.). northern advancement program. retrieved 09 may 2011 from http://www.unbc.ca/nap/objectives.html http://www.unbc.ca/firstnationscentre/� http://www.unbc.ca/nap/objectives.html� / evidence based library and information practice setting problem microsoft word ed_glynn.doc evidence based library and information practice       editorial    terminology versus action        lindsay glynn  editor‐in‐chief    public services librarian and instruction coordinator, health sciences library   memorial university of newfoundland   st. johnʹs, newfoundland, canada  e‐mail: lglynn@mun.ca      © 2006 glynn. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      i have heard it said, as many of us have, that  evidence based library and information  practice is an area dominated and led by  librarians in the health sciences.  it is a  logical leap to say that medical librarians  may be more familiar with the evidence  based model of practice because of their  need to be familiar, on some level, with  evidence based medicine.  the idea of  making a decision based on the appropriate  evidence is as familiar to a medical librarian  as is how to effectively search pubmed.   how pervasive is the influence of the  medical profession on this area?  being a  librarian looking for quick information, i  turned to google.  the results on the first  two pages from a google search for evidence  based practice are 100% health/medicine  related.  being a good librarian, i refined my  search to see how the results would differ,  and i added the term library to the search.   this time there were 75% health/medicine  results and 5% representing evidence based  library and information practice (eblip).   note that a high percentage of the  health/medicine hits were library webpages  on evidence based medicine.  being an  obsessive‐compulsive librarian, i changed  my search strategy again by replacing library  with librarianship.  this time there were 30%  health/medicine results and 65% eblip.  a  final search for evidence based information had  this journal as the top hit.  being a busy  librarian with a lot of work to do, i stopped  right there.      ok, so the terminology appears to point  strongly in one direction and weighs heavily  on the health sciences penetration.  let’s  leave terminology aside for a moment and  look at action.  since evidence based library  and information practice is the first journal on  this topic, the list of contributors and their  backgrounds should give an indication on  whether or not there is a concentration of  medical librarians.  approximately two  thirds of the articles that we have published  are non‐health/medicine related.  only 29%  of our evidence summaries are based on  research in the medical information field.   our 52 editorial advisors and 21 evidence  summaries writers come from many  different areas including school libraries,  law libraries, library schools, public libraries,  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:4    2 academic libraries, research institutions, etc.   admittedly, there is some bias here since we  make every attempt to ensure that we have  a balanced team of editorial advisors and  evidence summary writers.  that aside, the  evidence based movement, i’m pleased to  say, appears to be reaching far beyond  medical librarianship.       the proverbial saying that “it’s all in the  word” doesn’t hold here.  you may be  practicing in an evidence based manner  without even realizing it.  do you make  evidence informed decisions?  do you perform  action based research?   have you provided a  solid literature review to support a proposal?   have you presented a thorough evaluation  on an implemented service?  answering yes  to any of these questions does not increase  the likelihood that you are a librarian in the  health sciences.  rather, it demonstrates the  far‐reaching evidence based activities of  people in this most impressive cadre of  professionals in the library and information  field.      this issue of evidence based library and  information practice, like the past 3 issues, is  robust with excellence.  the quality of the  research articles and the evidence  summaries is extraordinary and once again,  surpasses my already high expectations.    we have reached a milestone: this issue  marks the end of our first year as a new  journal.  i would like to take this  opportunity to thank the editorial team for  their remarkable contributions to the success  of this journal.  working with this team  during the past year has been challenging,  exciting, rewarding and humbling.  to  denise koufogiannakis and alison brettle, i  thank you for your tireless work, endless  patience and devotion to excellence.  to  pam ryan, i thank you for keeping us all on  track, for managing the technology that  allows the journal to exist in an open access  format and for your invaluable insight.  to  priscilla stephenson and heather pretty,  your thorough attention to detail as  copyeditors is most impressive and i thank  you for this invaluable contribution.  to erin  alcock, ensuring that all metadata is entered  in doaj means that our articles are more  accessible – thank you.  to our editorial  advisory board and evidence summaries  writers, this journal would not exist without  your continued dedication and expertise.   finally, on behalf of the entire evidence based  library and information practice team, i  extend gratitude to the authors and readers  for your interest, support and inspiring  contributions.  may you all have a safe and  thoroughly enjoyable holiday season.          microsoft word news_eblipnom.doc evidence based library and information practice 2007, 2:1  159 evidence based library and information practice     news    eblip seeks nominations for a feature on classic research studies      © 2007 eblip. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      evidence based library and information  practice (eblip)   http://ejournals.library.ualberta.ca/index.ph p/eblip is soliciting nominations for a  special feature in our december 2007 issue.  we will be featuring summaries of classic  research studies that have impacted practice,  had an influence on lis researchers, and  stood the test of time. we need your help to  identify these classic studies in our field and  commit to writing a summary of that  research.  the summaries will use a format  similar to that of the current evidence  summaries published in eblip, but with a  commentary that focuses on the impact of  the research since it was published. please  give this some thought and consider  nominating a great research article to be  featured in eblip!    for more information, or to nominate a  research article, please contact denise  koufogiannakis  <denise.koufogiannakis@ualberta.ca>.    dates to note:     nomination deadline ‐‐ may 30, 2007     notification of acceptance for the feature  issue ‐‐ june 30, 2007     submission deadline for the summary ‐‐  september 1, 2007     publication date ‐‐ december 15, 2007     nominations should be accompanied by a  full bibliographic citation and an  explanation of the contribution of the  research to the field of library and  information practice.                    evidence summary   americans’ tolerance of racist materials in public libraries remained steady between 1976-2006   a review of: burke, s. k. (2010). social tolerance and racist materials in public libraries. reference and user services quarterly, 49(4), 369-379.   reviewed by: leslie bussert head of instruction / literature and humanities research and instruction librarian cascadia community college campus library university of washington bothell bothell, washington, united states of america email: lbussert@uwb.edu   received: 1 dec. 2011                                                                    accepted: 28 jan. 2012      2012 bussert. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to determine the general public’s levels of social tolerance toward public library materials containing racist content in order to present opinion data to librarians within a framework of scholarly perspectives that they can use for making decisions about intellectual freedom and controversial materials in libraries.   design – percentage and regression analysis of the general social survey longitudinal trend study dataset.   setting – united states, 1976-2006.   subjects – random samples of 26,798 primarily english-speaking adults aged 18 and up.   methods – the author analyzed responses from the well-respected and frequently used general social survey (gss), which has been conducted by the national opinion research center since 1972. the gss is a closed-ended survey including a variety of demographic measures. between the years 1976 and 2006, it also included a question to gauge the support of removing a book with racist beliefs about african americans from the public library. the surveys were conducted irregularly over this thirty-year span, and in total the question was asked nineteen times garnering 26,798 responses. spanish speakers were not included until 2006.   the author examined the data in multilevel cross-tabulations using percentages, and calculated chi-square for independence using frequencies. a multiple regression analysis was conducted to determine the predictive value of the independent variables examined on opinions of book removal. the author examined different variables, including education level, race, age, parental status, sex, geographic factors, religious affiliation, political party, and political conservatism. occupation was not used in the regression analysis because sample sizes in some categories were too small. the two ordinal variables, age and education level, were available as ratio level data that are most appropriate for regression calculations.   due to the large sample size, very small differences in percentages are significant at the .000 level. in these cases the author made judgment as to whether these differences were meaningful, or divided the data into multi-layer cross-tabulations to reduce the sample size and make the significance test more informative.   main results – analysis revealed the most influential predictors of support for book removal from the public library were education level, religious affiliation, and race. age was particularly influential for older respondents, while occupation and living in the south were moderately influential. variables with only slight correlations to support of book removal included political party affiliation and conservatism, parental status, and sex.   across all years of the study only 35.3% of respondents supported removal of racist materials from the public library. levels of support only changed slightly over the decades: in 1976, 38.1% supported removal while in 2006 only 34.5% did.   the mean age of respondents was 44.1 years and the median was 42 years. respondents over 57 years old were more likely to support removal (43.5%) compared to younger ages whose support ranged from 31.1–34.1%. the largest change over time was seen from respondents 57 years and older, whose support for removal dropped in later years of the study.   education level had a strong impact on opinions; the lower one’s education level, the higher their support for removal of the racist book from the public library. of those with less than a high school degree, 50.6% supported removal versus 35.8% of high school diploma holders. respondents with junior college, bachelor’s, and graduate degrees supported removal at 29.2%, 20.5%, and 15.3%, respectively. over time, those with high school degrees maintained their level of support for removal while those with higher levels of education increased their support for removal.   race was strongly related to opinions on removing offensive items from the library. while half of african american respondents supported removing a racist book, only one-third of white respondents did. however, in all but a few subcategories of analysis, the majority of african americans did not support removal, indicating a great deal of social tolerance on their part despite the possibility of being more sensitive to the implications of having racist materials in the library. when cross-tabulated with education level, the same pattern of support for removal was reproduced. there was little variation over time in white respondent’s opinions while african americans’ varied slightly. geographic factors affected opinions supporting removal of racist materials, though place size only had a small impact on opinions. respondents in the south were most likely to support removal (42.1%) and those in new england were least likely (25.2%). about one-third of respondents from the midwest (33%), mid-atlantic (36%), and the west (29.8%) supported removal. opinions over time remained the same in all regions but the south, whose support of removal dropped to 38.8%.   religion was found to correlate with opinions on removing racist books from the library. protestants showed the highest level of support for removal (39.5%), followed by catholics (32.3%), jews (21.7%), and respondents unaffiliated with religion (20.5%). race had a strong impact within some religions on supporting removal, particularly among methodists and those claiming no religion. when opinions by religion were cross-tabulated with education level, at every level baptists were more likely to support removal than other groups, while jews and those without religious affiliation were least likely.   other demographic variables had little effect on opinions concerning removal of racist materials from the library. parents supported removal (37%) while nonparents were less likely to (30%), and men and women were almost equally likely to support removal (33% and 37% respectively). political affiliation and level of conservatism only showed slight effects on opinions supporting removal. by a small margin democrats were most likely to support removal (39.2%) followed by republicans (34%) and independents (32.5%). across the conservatism spectrum, moderates were most likely to support removal (37.7%) followed by conservatives (36.4%) and liberals (29.9%).   the author also examined whether a respondent’s occupation influenced their opinions and focused this inquiry on the professions of library workers and educators. librarians were overwhelmingly against removal of racist materials while library paraprofessionals were less likely to support it than other workers with a similar level of education. college and university teachers in disciplines other than library and information science were divided but in comparison to other similarly educated professions they were less likely to support removal. school teachers were significantly more likely to support removal than other occupations also requiring a bachelor’s degree.   when contrasted with controversial materials of other types, such as those by openly homosexual or communist authors, different patterns of support for removal over time were observed. support for removal of books by homosexuals and communists declined significantly over the decades. similar to the support of the removal of racist materials, education and religious affiliation were the variables most highly correlated to support of removal of these other types of controversial books.   conclusion – the discomfort among americans over the free expression of exclusionary speech about african americans remained relatively consistent over the years of the study (1976 – 2006) despite some shifts within particular demographic categories. tolerance toward free expression by homosexuals and communists increased over time, demonstrating exclusionary speech may be perceived as a different type of social threat. librarians can use this information to: better understand how non-librarians view intellectual freedom in the context of materials with offensive content; inform collection development decisions and predict likelihood of challenges based on the demographics of their user communities; and to educate the public and library stakeholders of the implications of challenging these kinds of items within a library’s collection through upholding their professional values. librarians should continue to serve their communities by acting as champions of intellectual freedom and to uphold the profession’s rigorous standards. the author suggests future research could: address attitudes about materials with racist views of populations other than african americans; look for differences in opinions among library users versus non-users; and differentiate between adult and children’s materials containing controversial topics.     commentary   this study offers thorough analysis of a longitudinal dataset spanning thirty years and covering a variety of variables potentially impacting opinions on removing books from public libraries. the literature review includes other disciplines, such as speech communications, public policy, and race studies, for broader context. limitations of the study are acknowledged and based in the dataset itself. for example, african americans are the only race included in the survey prompt analyzed. the author concedes additional questions would be necessary for a comprehensive analysis from the library and information science perspective, and recognizes one’s opinions do not necessarily correlate to action.   the brief summary of data from related gss survey prompts about support for removal of homosexual or communist materials is valuable, as it helps identify whether opinions on racist books translate to other types of controversial materials, and in this case they did not. this comparison could be expanded by also analyzing the gss data about removal of socialist or militarist materials and more recent prompts about books by anti-american muslim clergymen and general anti-religious materials (icpsr, 2011a, 2011b).   the author’s findings are also illuminated when placed within the broader study of censorship and social tolerance. boyer (2002) has traced shifting patterns of book censorship in american legal and cultural history allowing for further contextualization of these findings. meanwhile, harell’s (2007, 2010) work examines the effects and consequences diverse democratic societies have on social tolerance and how exposure to social diversity impacts attitudes toward exclusionary speech. like burke (2010), harell also presents evidence that individuals are less likely to tolerate exclusionary speech compared to other controversial speech, and argues it is because exposure to social diversity increases empathy toward the victims of such speech (2007). this may help explain why individuals with more education, along with college professors or librarians, were more tolerant of racist materials in libraries. perhaps exposure to diverse ideas and people via education also impact one’s tolerance for exclusionary speech. however, librarian opposition to removal of racist materials exceeded all other educators, indicating they are stronger advocates for free speech.   the findings of this study reaffirm the profession’s stance on intellectual freedom and the author offers several useful ideas for applying them to practice. additional ideas for future research include revisiting the gss data over time for additional analysis and collecting opinions on this issue from non-english speaking populations across the united states.   references   boyer, p. s. (2002). purity in print: book censorship in america from the gilded age to the computer age. madison, wi: university of wisconsin press.   burke, s. k. (2010). social tolerance and racist materials in public libraries. reference & user services quarterl,y 49(4), 369-379.   harell, a. (30 aug 2007). tolerance judgments in multicultural democracies: target groups and the role of social diversity. american political science association annual meeting, hyatt regency chicago and the sheraton chicago hotel and towers, chicago, il, usa. retrieved 11 jan 2011 from http://www.allacademic.com/meta/p209670_index.html   harell, a. (2010). the limits of social tolerance in diverse societies: hate speech and political tolerance norms among young youth. canadian journal of political science, 43(2), 407-432. doi: 10.1017/s000842391000003x   inter-university consortium for political and social research (icpsr). (2011). general social survey, 1972-2006 [data file]. available from http://www.icpsr.umich.edu/icpsrweb/icpsr/access/index.jsp   inter-university consortium for political and social research (icpsr). (2011). general social survey, 1972-2008 [data file]. available from http://www.icpsr.umich.edu/icpsrweb/icpsr/access/index.jsp evidence based library and information practice 2011, 7.1 132 evidence based library and information practice news/announcements 7th international evidence based library and information practice conference to be held in saskatoon, saskatchewan, canada – july 15-18, 2013 2012. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. the university library at the university of saskatchewan is very proud to host the 7th international evidence based library and information practice conference (eblip7), july 15‐18, 2013. with six amazing conferences leading the way, eblip7 promises to be exciting, engaging, and invigorating. the first question you might have if you are not from canada (or maybe even if you are!) is “where is saskatoon, saskatchewan”? saskatchewan is in western canada and is one of the three prairie provinces. the local joke is that saskatchewan is difficult to spell but easy to draw! saskatoon is in the central part of the province and is called the hub city due to its central location. it is also referred to as the city of bridges because of the many bridges spanning the south saskatchewan river that runs through the city. in a province of one million people, saskatoon is the largest city with a population of approximately 230,000 people. the university of saskatchewan (u of s) is a premiere medical‐doctoral institution and is figure 1 image from tourism saskatchewan http://www.sasktourism.com/about‐saskatchewan/quick‐ facts/location http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/� http://www.sasktourism.com/about-saskatchewan/quick-facts/location� http://www.sasktourism.com/about-saskatchewan/quick-facts/location� evidence based library and information practice 2011, 7.1 133 home to over 20,000 undergraduate and graduate students, and over 1,100 faculty. the u of s is one of north america’s most beautiful campuses. with a 100+ year history (which is a long time by canadian standards considering that saskatchewan became a province in 1905), its earliest buildings form what the canadian register of historic places refers to as “the finest grouping of collegiate gothic university buildings in canada” (http://goo.gl/svejo) if you are new to the eblip conference experience, please consider making eblip7 your maiden voyage. eblip is a practical was to practice librarianship and if you are interested in research and using research in practice, then this is the conference for you. eblip7, with the dual draws of a robust and inspiring programme selected by an international programme committee and the chance to network with the vibrant international eblip community provides a good opportunity to learn more. if you are an eblip conference veteran, you will love the saskatchewan experience. from a sumptuous opening reception and conference banquet to the wealth of knowledge and information that the conference program and preconference sessions will bring, you will leave saskatoon filled with enthusiasm and new ideas. when you get back home, you will remember your prairie experience as one filled with warmth, hospitality, and friendship. follow eblip7 on twitter: @eblip7. there is also a conference website at http://eblip7.library.usask.ca/. in the months to come, more information will be posted on all aspects of the conference: travel, accommodations, calls for papers and posters, conference program, special events, events for spouses and families, etc. the university library, u of s, is very excited to host the international eblip community in 2013. http://goo.gl/svejo� http://eblip7.library.usask.ca/� / evidence based library and information practice news/announcements   registration for the c-eblip fall symposium 2015: librarians as researchers is now open!      2015. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   the c-eblip fall symposium is a one-day conference coming up on wednesday, october 14, 2015 at the university of saskatchewan (with complimentary registration) open to any interested librarian. it is designed to facilitate sharing, collaboration, and networking with the focus on librarians in their researcher roles. new this year, a pre-symposium research workshop, entitled transforming ideas into well-designed research questions, will be presented by the university library's researcher in residence, selinda berg, on october 13, 2015. please note that spaces are limited for the workshop.   university of regina librarian and researcher cara bradley will be giving the opening keynote address entitled let's give them something to talk about: sharing librarians' research beyond professional boundaries. you can find the rest of the schedule at a glance here: http://library.usask.ca/ceblip/c-eblip-fall-symposium/symposium-program.php   you can access the online registration form here: http://library.usask.ca/ceblip/c-eblip-fall-symposium/registration.php   save your space now and plan to join us at the centre for evidence based library and information practice, university library, university of saskatchewan in saskatoon this fall.   evidence based library and information practice evidence based library and information practice 2011, 6.4 140 evidence based library and information practice evidence summary master’s students in an information studies program enter the program with excitement and leave with concerns about professional preparation for their chosen fields a review of: cherry, j. m., duff, w. m., singh, n., & freund, l. (2011). student perceptions of the information professions and their master's program in information studies. library & information science research, 33(2), 120-131. doi:10.1016/j.lisr.2010.09.004 reviewed by: christina e. carter associate professor consortium library, university of alaska anchorage anchorage, alaska, united states of america email: trina@uaa.alaska.edu received: 06 sept. 2011 accepted: 19 nov. 2011 2011 carter. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to assess master’s students’ perceptions of their information studies program with regard to the program’s academic quality and professional preparation as it moved to become an ischool. design – longitudinal survey, employing both quantitative analysis of demographics and closed responses, and thematic analysis of open-ended responses. setting – university of toronto, canada, faculty of information, master of information studies (mist) ala-accredited program. subjects – students enrolled in the mist program from fall 2003 to spring 2007. methods – between 2003 and 2007, a selfadministered confidential questionnaire was distributed eight times: a short version of the questionnaire to incoming students in the fall term over the four years, and a longer version to the entire mist student body in the spring term of the four years. thus, individual students participated in the survey multiple times. survey questions fell into four categories: program assessment, perceptions on the information professions, career and personal achievements, and demographics. the first questionnaire was mailed in paper form; after that, web-based questionnaires were used. quantitative data collected was mailto:trina@uaa.alaska.edu� evidence based library and information practice 2011, 6.4 141 analyzed using spss, version 17, and openended responses were examined for recurring themes. main results – across the four years of the survey, researchers obtained about 1,000 completed questionnaires. the response rate was always higher in the fall term than in the spring term, ranging from a high of 67% in fall 2003, to a low of 47% in spring 2007 which seemed to indicate “fatigue” with the study (p. 124). respondents primarily were interested in the information professions and the majority planned to work in one of them (archives, library systems, or library and information science) after graduating. no statistically significant differences relating to the year the survey was completed were found for student perceptions of career prospects or for amount of computer knowledge required. a statistically significant difference was found for perceptions of new students of occupational prestige for archivists and librarians; it increased over the four years. the majority of students surveyed over the four years indicated that: 1) job prospects would grow, 2) required computer knowledge was high and would increase in the next five years, and 3) computer and systems-related tracks garnered higher perceived social status than the archivist and librarian “streams.” students who had been in the program longer (completing nine or more courses) more strongly supported the master’s program’s move to an ischool with more emphasis on technology and computing, and an increased emphasis on professional work. these students were also less positive about their academic programs than the students who had completed less coursework. open-ended responses echoed many other studies pointing to social status assigned to library professionals being lower than the opinion of the students themselves. conclusion – the authors’ longitudinal approach and survey methodology revealed perceptual differences between new and more veteran mist students at the university of toronto of their master’s program, and that as students progressed through the program, they felt the need for more professional preparation. between 2008 and 2010, the authors conducted similar studies regarding the perceptions of students at five other canadian institutions, and planned to survey students at many other north american graduate library science programs via webbased questionnaire to compare findings cross-institutionally. commentary this survey vetted some 1,000 returned questionnaires, a good response rate although administered at only one institution. the method of pretesting for questionnaires is not explained nor are the “small number of changes and additions” made in the four years of its use (p. 123). as the authors note, due to anonymity in questionnaires, individual responses could not be tracked over the four years, which would have supplied valued insight. the study setting’s move to an ischool is not expounded upon in this study, nor what that transition specifically meant to students surveyed. questions regarding future program directions were only asked in the final survey in spring 2007, so conclusions drawn from this question only asked in one term cannot be extrapolated to a larger group. it would be worthwhile to explore the concept of the “ischools phenomenon” (wallace, 2009) more in-depth, perhaps surveying students fully in the midst of it, both within this institution and at others. an internship or other practicum is not required at the program studied, as it is in many other accredited information studies programs. it is not surprising, therefore, that the majority of students commented on the need for more professional preparation without this component. library school educators and practitioners can immediately take from this that they should examine their own programs to ensure that there are sufficient and diverse opportunities such as practica, job shadowing, and paid internships evidence based library and information practice 2011, 6.4 142 available to students throughout their program of study. this study also brings to bear the continuing question of balance between theory and practice in professional study. the data and insights revealed here can further assist those involved in graduate library and information science education in planning and assessment. this study was carried out before some of the more recent explosion in information technology in society at large (social media, ubiquitous nature of handheld devices) of the latter part of the decade and into the present one. in particular, the use of social media is now used in all fields within the profession. in a reiteration of this study, it would be worthwhile to include questions on this topic and how it impacts information studies programs and preparation for employment. additional longitudinal studies could also track perceptions of students just after graduation and well into their professional careers in the three tracks examined (archives, information systems, and library and information science). references wallace, d. p. (2009). the ischools, education for librarianship, and the voice of doom and gloom. the journal of academic librarianship, 35(5), 405-409. doi:10.1016/j.acalib.2009.07.001 / evidence based library and information practice evidence based library and information practice 2011, 6.4 177 evidence based library and information practice ebl 101 research methods: content analysis virginia wilson liaison librarian, murray library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca received: 27 oct. 2011 accepted: 27 oct. 2011 2011 wilson. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. content analysis, a method which can be used qualitatively or quantitatively for systematically analyzing written, verbal, or visual documentation, goes back to the 1950s and the study of mass communication (white & marsh, 2006, p. 22). key themes emerge from the documents after they are classified and coded. the content can come from a wide variety of sources: books, manuscripts, drawings, photographs, recorded conversations, videotaped events, messages on electronic mailings lists and online forums, blog posts, etc. content is analyzed by breaking it up into conceptual chunks that are then coded or named. qualitative analysis develops the categories as the analysis takes place. the results are used to make inferences about the messages in the text. quantitative analysis starts with a hypothesis and a predetermined coding scheme that is designed to test the hypothesis. the results are described using statistics. kimberly neuendorf and klaus krippendorff are two of many contemporary scholars in the area of content analysis. neuendorf sees the method as primarily quantitative, while krippendorf believes that counting is not a prerequisite of content analysis. verbal categories and the listing of quotes are considered as valid as numbers and counting. there are two types of content analysis: conceptual analysis and relational analysis. conceptual analysis is largely what was described above: the content is coded for certain words, concepts, or themes, and the analyst makes inferences based on the patterns that emerge. relational analysis builds on conceptual analysis by delving into the relationships between the concepts and themes that surface from the analyzed text. relational analysis is popular because of its flexibility, but this flexibility can also be a drawback when reliability and trustworthiness are necessary in the research. when the analysis is too flexible, the research becomes impossible to replicate. therefore, a codebook, a coding form, rules, and often more than one coder are necessary to give stability to the content analysis process. mailto:virginia.wilson@usask.ca evidence based library and information practice 2011, 6.4 178 content analysis is a method commonly used in the social sciences and is therefore a viable choice for lis research. in fact, there are many published lis research studies that have used the content analysis methodology. i have selected a few as an illustration: dahl, c. (2001). electronic pathfinders in academic libraries: an analysis of their content and form. college& research libraries, 62(3), 227-237. du, y., stein, b., & martin, r.s. (2007) content analysis of an lis job database: a regional prototype for a collaborative model. libri, 57, 17-26. retrieved from http://www.librijournal.org/pdf/20071pp17-26.pdf julien, h., mckechnie, l. & hart, s. (2004). a content analysis of affective issues in library and information science systems work [summary of a research note delivered at the isic 2004 conference, dublin, 1-3 september, 2004], information research, 10(1). retrieved from http://informationr.net/ir/10-1/abs6 koufogiannakis, d., slater, l., & crumley, e. (2004). a content analysis of librarian research. journal of information science, 30(3), 227-239. doi: 10.1177/0165551504044668 morais, y., & sampson, s. (2010). a content analysis of chat transcripts in the georgetown law library. legal reference services, 29(3), 165-178. yontar, a., & yalvac, m. (2000). problems of library and information science research in turkey: a content analysis of journal articles 1952-1994. ifla journal 26(1), 39-51. the basic steps a researcher takes in approaching a content analysis is as follows: 1. develop a research question (if using qualitative analysis) or a hypothesis (if using quantitative analysis). 2. define the population. this step can take place at several places during the course of the study: at the research question development state or later in the process depending on what is returned from choosing a particular population. then there are different methods of choosing a sample: random sampling methods (simple random sampling, systematic random sampling, cluster sampling, stratified sampling) and non-random sampling (purposive sampling, convenience sampling). 3. select a research design. there are several tasks to be accomplished in this step: choose units of analysis to study; create a coding scheme; and, in the case of quantitative analysis, develop a numbering system. 4. gather data: quality control is a major concern when coding. agreement tests must be conducted between coders to insure acceptable levels of inter-coder reliability. the researcher must try to avoid subjectivity and the appearance of subjectivity. 5. interpreting the evidence: findings must directly address the research question or hypothesis. tables, bar graphs, numbers, etc., all must be explained and interpreted in light of the question asked. (adapted from beck & manuel, 2004, chapter 3) in practice, content analysis can be “timeconsuming and labour-intensive” (beck & manuel, 2004, p. 37). using this research method can help to reveal trends and themes, but it cannot attribute cause. however, it is one of the top research methods used in lis research, and can be just the thing when an analysis of multiple texts is required. coming up in the next issue, a look at using focus groups. works consulted course content page from the university of texas school of information. retrieved from http://search.proquest.com.cyber.usask.ca/docview.issuebrowselink:searchpublicationissue/23477/legal+reference+services/02010y07y01$23jul+2010$3b++vol.+29+$283$29/29/3?site=lisa&t:ac=758112415/citation/1328d82f1ff50a99074/71&t:cp=maintain/resultcitationblocks http://search.proquest.com.cyber.usask.ca/docview.issuebrowselink:searchpublicationissue/23477/ifla+journal/02000y01y01$232000$3b++vol.+26+$281$29/26/1?site=lisa&t:ac=57518212/abstract/1328d82f1ff50a99074/57&t:cp=maintain/resultcitationblocks evidence based library and information practice 2011, 6.4 179 http://www.ischool.utexas.edu/~palmq uis/courses/content.html hsieh, h-f., & shannon, s.e. (2005). three approaches to qualitative content analysis. qualitative health research, 15, 1277-1288, doi: 10.1177/1049732305276687 recommended reading krippendorff, k. (2004). content analysis: an introduction to its methodology (2nd ed.). thousand oaks, ca: sage. neuendorf, k. a. (2002). the content analysis guidebook. thousand oaks, ca: sage. white, m.d., & marsh, e.e. (2006). content analysis: a flexible methodology. library trends, 55(1), 22-45, doi: 10.1353/lib.2006.0053 references beck, s.e., & manuel, k. (2004). practical research methods for librarians and information professionals. new york: neal-schuman. white, m.d., & marsh, e.e. (2006). content analysis: a flexible methodology. library trends 55(1), 22-45 doi: 10.1353/lib.2006.0053 ebl 101   looking to the literature: open access and free sources of lis evidence   virginia wilson shirp coordinator health sciences library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   originally published in: evidence based library and information practice, 4(3), 75–77. https://ejournals.library.ualberta.ca/index.php/eblip/article/view/6521/5536     received: 06 aug. 2009 accepted: 10 aug. 2009      2016 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   in the last ebl 101 column, the discussion centered upon assigning your question to one or more domains in order to determine where to look for research evidence. a rule of thumb is to check out the lis resources first, and then move to other bodies of evidence that may inform your question, such as the business, education, marketing, or management literature, to name a few. one of the barriers to evidence based library and information practice is a lack of access to the research evidence. some library acquisition budgets do not include a line for professional publications, or if they do, the funds allocated to such collections can be relatively small.   in light of this barrier, i have compiled a list of open access and free sources of library and information science resources. according to peter suber, open access literature is “digital, online, free of charge, and free of most copyright and licensing restrictions” (qtd. in morrison: 46). anyone can access and utilize open access resources. stephanie hall and i compiled the following list for the evidence based librarianship toolkit for public libraries <http://ebltoolkit.pbworks.com/>, and have updated it for this column.   databases   library, information science & technology abstracts (lista) <http://www.libraryresearch.com> from ebsco publishing, this free bibliographic database provides coverage on subjects such as librarianship, classification, cataloging, bibliometrics, online information retrieval, information management and more.   journals   directory of open access journals (doaj) <http://www.doaj.org/> under the social sciences heading, there are 96 open access library and information science journals in various languages.   evidence based library and information practice <http://ejournals.library.ualberta.ca/index.php/eblip> an open access journal containing research, evidence summaries, and commentary on evidence based library and information practice. published quarterly. canadian.   partnership: the canadian journal of library and information practice and research <http://journal.lib.uoguelph.ca//index.php/perj> this open access practitioners’ journal contains lis research, conference presentations, profiles, reviews, commentary, and more. published twice yearly. canadian.   libres: library and information science research electronic journal <http://libres.curtin.edu.au//. an open access international refereed electronic journal devoted to new research in library and information science. published twice yearly. australian.   ariadne <http://www.ariadne.ac.uk/> this is a web magazine for information professionals in archives, libraries and museums. published every three months, beginning in january. uk.   e-jasl: the electronic journal of academic and special librarianship <http://southernlibrarianship.icaap.org/> an independent, professional, refereed electronic journal dedicated first and foremost to advancing knowledge and research in the areas of academic and special librarianship (from website). open access. canadian.   information research <http://informationr.net/ir/> this open access, international, peer-reviewed, scholarly journal is dedicated to making accessible the results of research across a wide range of information-related disciplines (from website).   archives   e-lis: e-prints in library and information science <http://eprints.rclis.org/> an open access archive for scientific or technical documents, published or unpublished, on librarianship, information science and technology, and related areas (from website). there are more than 9,400 items in the archive.   dlist: digital library of information science and technology <http://dlist.sir.arizona.edu/> an open access archive for the information sciences. the archive has a global advisory board.   oclc research publications repository <http://www.oclc.org/research/publications/search.htm> this repository contains works produced, sponsored, or submitted by oclc research. in general, the works are research-oriented and are in the subject area of library and information science. many items describe oclc research projects, activities, and programs and were originally published by oclc, while others are from peer-reviewed scholarly journals. it currently contains links to the full text of 303 items (from website).   opendoar: the directory of open access repositories <http://www.opendoar.org/> an authoritative directory of academic open access repositories, each opendoar repository has been visited by project staff to check the information that is recorded there. this in-depth approach does not rely on automated analysis and gives a quality-controlled list of repositories (from website).   if you lack access to a range of lis resources, these should get you started. make sure to check with professional organizations to see if there are resources available to you as part of the membership fee. if you have an open access or free lis resource to add to the list, please contact me: virginia.wilson@usask.ca. i will make any additions available in future ebl 101 columns. but what happens if you cannot find any published research evidence to aid in your decision-making process? next time, conducting your own research: something to consider.   works cited   morrison, heather.  “evidence based librarianship and open access.” evidence based library and information practice 1.2 (2006): 46-50. 6 aug. 2009 <http://ejournals.library.ualberta.ca/index.php/eblip/article/view/49/117>.   evidence based library and information practice 2012, 7.3 3 evidence based library and information practice editorial responsibilities 2012. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. editor-in-chief: alison brettle associate editor (articles): wayne jones associate editor (evidence summaries): lorie kloda associate editor (classics): jonathan eldredge associate editor (reviews): denise koufogiannakis associate editor (commentaries, using evidence in practice, ebl101, news): alison brettle production editor: katrine mallan editorial intern: michelle dunaway copyeditors: heather pretty (lead copyeditor), georgianne bordner, molly des jardin, linda ferguson, richard hayman, stacey penney, mary virginia taylor , alison yeoman indexing support: pam morgan http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/ editorial   evidence summary theme: information access & retrieval   fiona inglis associate editor (evidence summaries) liaison librarian, science wilfrid laurier university waterloo, ontario, canada email: finglis@wlu.ca      2023 inglis. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30328     the evidence summaries in this issue take a very broad view of information access and retrieval. koufogiannakis et al. (2004) define this domain of librarianship as “creating better systems and methods for information retrieval and access” (p. 233), but these evidence summaries expand the scope a little to include information seeking behavior as an important indicator of how our spaces, services, and systems need to function to enable access.   barriers to access can have deep roots. prince reviews a study of a public library system that reveals the impact of historical and ongoing systemic inequities on children’s access to library spaces and materials.   information cannot be accessed if it does not exist. this is an issue highlighted in the study appraised by shen that explores the challenges of accessing accurate breast cancer information for transgender and gender non-conforming populations.   understanding information seeking behaviour is essential if we are to improve access. fena and jasmin review two different studies on information seeking behaviour, one focused on 2slgbtq+ students and the other on older adults.   the layout of both physical and virtual library spaces can also be a barrier to access. bridgemen discusses a multi-method wayfinding study exploring the experiences of students who speak english as an additional language.   outside of the library, online communities are creating their own methods for providing access to information. patterson discusses a study that highlights what libraries and data enthusiasts can learn from each other.   as always, each evidence summary includes a discussion of how findings from the research can be applied to other settings. we hope these will inspire you to ask questions and explore ways to reduce barriers within your own context.   references   koufogiannakis, d., slater, l., & crumley, e. (2004). a content analysis of librarianship research. journal of information science, 30(3), 227–239. https://doi.org/10.1177/0165551504044668   news/announcements   institute for research design in librarianship: call for proposals      2015. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   “irdl not only set me on the path to producing and properly documenting original research, but it set me up with a network of like-minded librarians who will help me achieve my goals.” — 2014 irdl scholar   we are issuing a call for applications for the institute for research design in librarianship (irdl) 2016. we are seeking novice librarian researchers who are employed by academic libraries or research libraries outside an academic setting in the united states to participate in the institute. novice researchers typically may have conducted research but have not yet had a peer-reviewed article published as the primary author or had an individual presentation accepted by a peer-reviewed conference. we define “novice” broadly; if you feel that you would benefit from being guided throughout the entire research design process, we encourage your application. librarians of all levels of professional experience are welcome to apply.   the third workshop will be held on june 6-16, 2016, with arrival on campus on sunday, june 5, and departure on friday, june 17.   the william h. hannon library has received a three-year grant from the institute for museum and library services (imls) to offer this continuing education opportunity for academic and research librarians. each year 21 librarians will receive, at no cost to them, instruction in research design and a full year of peer/mentor support to complete a research project at their home institutions; the learning experience, travel to and from los angeles, ca, accommodations, and food will be supplied to scholars free of charge. the summer irdl workshop is supplemented with pre-institute learning activities and a peer learning network that provides ongoing support. the workshop will be held on the campus of loyola marymount university in los angeles, california. for more information about the project, including the project partners, the san josé state university school of information and the statewide california electronic library consortium (scelc), please see the project website: http://irdlonline.org.   we seek librarians with a passion for research and a desire to improve their research skills. irdl is designed to bring together all that the literature tells us about the necessary conditions for librarians to conduct valid and reliable research in an institutional setting. the cohort will be chosen from a selective submission process, with an emphasis on enthusiasm for research and diversity from a variety of perspectives, including ethnicity and type and size of library. selection criteria:   commitment to the year-long process of communicating with other participants and conducting the proposed study within the 2016-2017academic year; significance of the research problem to the operational success of the applicant’s library or to the profession of librarianship; thoughtfulness, thoroughness, and clarity of the research proposal; enthusiasm for research and a desire to learn.   we will be accepting applications from december 1, 2015 to february 1, 2016.   scholars accepted to the institute will be notified in early april, 2016.   please contact project directors with any questions about the institute or the application process: kristine brancolini, dean of the library, loyola marymount university brancoli@lmu.edu marie kennedy, serials & electronic resources librarian, loyola marymount university marie.kennedy@lmu.edu   evidence summary   third-party tracking in online public library environments in the united states and canada: a statistical analysis   a review of: gardner, g. j. (2021). aiding and abetting: third-party tracking and (in)secure connections in public libraries. the serials librarian, 81(1), 69–87.  https://doi.org/10.1080/0361526x.2021.1943105   reviewed by: david dettman associate professor and library instruction program coordinator university of wisconsin-stevens point stevens point, wisconsin, united states of america email: ddettman@uwsp.edu   received: 6 mar. 2023                                                                    accepted:  20 apr. 2023      2023 dettman. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30342     abstract   objective – to determine through statistical data collection the frequency of tracking by third parties in online public library environments along with the visibility and ease of discovery of online library policies and disclosures related to third-party tracking in particular and data privacy in general.   design – online evaluation of public library websites.   setting – english-language public libraries in the united states and canada.   subjects – 178 public library websites (133 in the united states and 45 in canada). the libraries included in the study were intentionally selected for their membership in either the canadian urban libraries council (culc) or the urban libraries council (ulc) in the unites states, since these libraries have some of the largest systems membership serving predominantly urban and suburban communities in both countries. the included canadian libraries serve nearly 41% of the population in that country while the included libraries in the united states are positioned to serve 28% percent of the total population. the author notes that “these percentage figures serve as hypothetical, upper-bound estimates of the population affected by third-party tracking since not every member of these communities actually uses their local public library” (gardner, 2021, p.72).   methods – in addition to evaluating the public library catalog and website in general with regards to third-party tracking and data privacy, 10 common content sources (databases) available at all of the included libraries were also included in the examination. two browser add-ons designed to detect third-party tracking, ghostery and disconnect, were used in the study due to their popularity and incorporation into previous similar studies. in addition to third-party tracking the author executed word searches on library homepages using ctrl-f for words commonly used to denote privacy or terms of use statements. no qualitative analysis was performed to determine if information shared regarding third-party tracking was accurate, and subpages were not examined. the data collection period lasted a total of three months beginning in march 2017 and running through may 2017.   main results – the data gathered between march and may of 2017 clearly indicates a general disregard among most sampled public libraries regarding the protection of patron data gathered by third-party tracking. of canadian libraries included in the sample 89% (40) enabled third-party tracking, while libraries in the united states allowed it at a rate of 87% (116). both ghostery and disconnect revealed an almost identical number of incidences of third-party tracking in library catalogs and in the 10 popular public library databases examined in the study. certain opacs were associated with higher tracking counts as were certain library databases. libraries were found to be lax when it came to providing a link on the homepage potentially informing users of the presence of third-party tracking. of the 156 total libraries with third-party tracking in their online catalogs, 69 (44%) included a homepage link while the rest did not. the author notes that the presence of a link was all that was examined, and not specific language used to disclose the level of third-party tracking or data privacy. in total, 8 of the 10 common content sources allowed third-party tracking. all 10 provided a link to either privacy or terms of service statements on their landing pages.   conclusion – although patron privacy is an issue addressed in the american library association (ala) code of ethics (american library association, 2021), the author concludes that “together with previous research on usage of privacy-enhancing tools in public libraries, these results suggest that public libraries are accessories to third-party tracking on a large scale” (gardner, 2021, p.69).   commentary   the introduction to the article references the 2019 report issued by a civil grand jury in santa cruz county, california castigating the santa cruz public libraries for neglecting to make users aware of how personal data was being tracked by third-party software, along with reprimanding them for not adhering to ala best practices. they were also reprimanded for entering into agreements with third parties that “raised liability issues related to patron privacy – among other things” (gardner, 2021, p.69).   the author asserts that although public librarians often see themselves as the protectors of patron privacy, complex challenges have presented themselves in an information age where online environments become increasingly the place where information is disseminated. despite this cataclysmic change, the author notes that there are certain measures that can be easily undertaken, for example, adopting https to take advantage of built-in security instead of continuing to deliver websites and catalogs to patrons over http. the study revealed that 46.7% of canadian libraries in the sample and 66.9% of libraries in the united states did not have https configured to run on either their homepage or their opac.   the author rightly asserts that “an underappreciated aspect of recent development over the past decade or so is the rise of e-scores. indeed, they’ve been dubbed ‘the new face of predictive analytics’ by the american marketing association” (gardner, 2021, p.81). this is truly a concern when we consider how patron-harvested data can be used to determine not just consumer habits and personalized advertising content. harvested patron data can also be used for more sinister and nefarious purposes when it is fed into algorithms used to reinforce and bolster systematic and structural forms of racism and oppression.   the author concedes that the data from 2017 may not accurately represent the current state of affairs. they mention that the study was completed prior to the widespread adoption of the european union’s general data protection regulation, which is making it more commonplace for users on commercial websites (and library database websites, in this reviewer’s experiences) to be presented with third-party tracking details with web cookie or other tracking notification. the author also notes that when the data was gathered in 2017 the let’s encrypt certificate authority had been released the previous year, and it is being increasingly adopted by organizations of all kinds. this certificate provides free tsl/ssl certificates that greatly increase measures to protect personal data. this along with unprecedented high-profile incidences of the compromise of personal data “have increased public and governmental awareness of website security and privacy issues” (gardner, 2021, p.83).   the quality of the study was appraised using “the cat: a generic critical appraisal tool,” created by perryman and rathbun-grubb (2014). based on this analysis the quality of the study was found to be high. according to their orcid page the author is the interim associate dean of the university library at california state university, long beach and has published extensively around information-seeking behavior and use as it is mediated through library systems. a useful literature review is included, and the research question is clearly defined. the methods used in the research are clearly communicated. there are also discussions about the limitations of the study and the direction that further research might profitably take.   references   american library association. (2021, june 29). code of ethics. https://www.ala.org/tools/ethics              gardner, g. j. (2021). aiding and abetting: third-party tracking and (in)secure connections in public libraries. the serials librarian, 81(1), 69–87. https://doi.org/10.1080/0361526x.2021.1943105   perryman, c., & rathbun-grubb, s. (2014). the cat: a generic critical appraisal tool.  http://www.jotform.us/cp1757/thecat review article   a systematic review of library services provision in response to covid-19 pandemic   philips o. ayeni phd candidate, school of information studies mcgill university montreal, quebec, canada email: philipsayeni@gmail.com   blessed o. agbaje email: agbajeblessed@gmail.com   maria tippler graduate, master of information studies school of information studies mcgill university montreal, quebec, canada email: maria.tippler@mail.mcgill.ca   received: 30 dec. 2020                                                              accepted: 21 july 2021        2021 ayeni, agbaje, and tippler. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   doi: 10.18438/eblip29902     abstract   objective – libraries have had to temporarily shut their doors because of the covid-19 pandemic, resulting in the provision of online and remote services. this review analyzed services offered by libraries, the technological tools used, and the challenges facing libraries during the pandemic.   methods – this study employed a systematic literature review, following the prisma checklist (moher at al., 2009). the building blocks search strategy was employed to search for keywords of concepts in library and information science abstract (lisa), library and information science technology abstract (lista), library science database, web of science (wos) core collections, and google scholar. a set of inclusion and exclusion criteria was pre-determined by the authors prior to database searching. quality assessment of included studies was performed using the mixed methods appraisal tool (hong et al., 2018). a tabular approach was used to provide a summary of each article allowing the synthesis of results, which led to the identification of eight broad categories of services provided by libraries in included studies.   results – the first set of searches from the 5 databases produced 3,499 results. after we removed duplicates and applied the inclusion and exclusion criteria based on titles and abstracts, 37 potentially relevant articles were identified. further screening of the full-text led to the final inclusion of 23 articles used for the qualitative synthesis. the majority of the studies were conducted in the united states of america (n= 6, 26.1%), followed by india (n=4, 17%), and china (n=2, 8.7%). the remaining studies were carried out in united kingdom, ireland, canada, mexico, romania, czech republic, indonesia, pakistan, nigeria, lesotho, and zimbabwe. the most common method used in selected studies was the case study (n= 11, 48%), followed by survey (n=7, 30.4%), content analysis (n=4, 17.4%), and mixed methods (n=1, 4.3%). the majority of the studies were carried out in academic libraries (74%), while the rest were based on medical, public, and special libraries. findings show that the majority of academic libraries in the included studies are providing and expanding access to electronic resources (n=16, 69.6%) and increasing open access resources and services (n=11, 47.8%). more so, most academic libraries are assisting in virtual education and teaching endeavors of faculty and students (n=13, 56.5%). in addition, some medical and public libraries are bolstering public health safety through health literacy (n=12, 52.2%), supporting research efforts, and engaging in virtual reference services, among others. in order to carry out these services, libraries are harnessing several educational, social networking, communication, and makerspaces technologies. most of the libraries in the included studies reported budgetary challenges, and the need for new ict infrastructure and internet service as they move their services online.   conclusion – this review found that libraries are adapting in a number of ways to continue their roles in meeting patrons’ needs in spite of the growing challenges posed by covid-19 restrictions and lockdown. for libraries to thrive in these trying times, there must be a well-structured approach to ensuring continuity of services. libraries should prioritize the acquisition of electronic resources as well as increase their efforts to digitize resources that are only available in printed copies. as library services have predominantly shifted online, there should be concerted effort and support from government and funding agencies to equip libraries with the technological facilities needed to provide cutting-edge services. the quality assessment of the included studies shows that there is need for rigor and transparency in the methodological description of studies investigating library services provision in a pandemic. this review provides an overview of the ways libraries have responded to the challenges posed by a global pandemic, and hence will be of use and interest to all librarians especially those in health and academic sectors.     introduction   the year 2020 started off with the novel coronavirus disease (hereafter referred to as covid-19), which was first discovered in wuhan, china in 2019. covid-19, caused by sars-cov-2 virus, was declared a pandemic on march 11, 2020, because it is a highly infectious virus that can permanently damage the entire body – from the heart, lungs, nervous system, musculoskeletal structures, and others (world health organization [who], 2020a). transmission happens primarily by being in close contact with an infected individual when sneezing, coughing, talking, or touching one’s face after handling contaminated surfaces (who, 2020b). as of july 18, 2021, over 188 million cases and over 4 million deaths have been reported globally since the start of the pandemic (who, 2021).  the covid-19 pandemic has caused global socioeconomic disruption, leading to shutdowns, quarantines, curfews, closures, cancellations, and other restrictive controls to constrain and prevent its spread (kosciejew, 2021).   libraries around the world have been negatively impacted by the pandemic as many libraries have either closed their buildings or restricted physical access to library facilities (yu & mani, 2020) and yet have adapted in various ways. libraries have a pressing need to acclimatize to an ever-changing environment quickly and efficiently, as services have shifted predominantly online. understanding the current challenges and highlighting the various strategies libraries have undertaken around the world can help provide a cohesive vision of how to best move forward in these difficult times.   aims   this systematic review aims to provide an understanding of how different types of libraries have responded to sudden closure because of the covid-19 pandemic and what services they are providing to meet the needs of users. this review also aims to determine the technological tools used in carrying out these services and the challenges hindering library services provision amid a global pandemic. the findings of this review will provide libraries guidance as regards to how to approach library services provision in a pandemic, as well as the tool and technologies that can be harnessed in carrying out these services, and the challenges to overcome.   methods   we chose the systematic review method of literature review because it has a potential to produce a reliable knowledge base through the accumulation of findings from a range of studies in a systematic and reproducible way (briner & denyer, 2012). according to moher et al. (2009), a systematic review is a “review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyze data from the studies that are included in the review” (p. 1). it follows a clear, easy-to-follow, and replicable process in identifying, analyzing, and reporting what has been published on a particular topic. conducting a systematic review will help synergize previous studies that met certain criteria which are not necessarily considered in other types of review such as a traditional narrative review. hence, we included academic and scholarly journals articles that discussed and researched on library services provision in response to the covid-19 pandemic in different types of libraries.   eligibility criteria   before the search was conducted, the authors clarified and agreed on inclusion and exclusion criteria employed for the systematic review. the review focused on articles published in peer-reviewed journals in english journals, with search limits set between january 2020 and march 2021 in order to retrieve articles published during the pandemic. the inclusion and exclusion assessment are in two stages. the first stage involved the preliminary assessment of whether or not the articles met the objectives of the study before an in-depth analysis of each article to see if they were thorough and could be included for final analysis and synthesis. at the first stage, articles were excluded if: i) they were not published in scholarly journals; ii) full-text of articles was not available; iii) they were short memos, magazines, and news report; iv) they were trade journals since majority of the articles published therein focused more on monetary and budgetary implications of the pandemic with less focus of services; and v) were not written in the english language. at the second stage of the inclusion/exclusion, the assessment of the full-text of articles that met the first criteria was carried out to find out if the articles met the objectives of the study. articles were excluded if: i) there was no discussion about the library services provided in response to covid-19 in the article; ii) they were anecdotal reports from researchers without any methodological evidence or approach and iii) articles discussed more about general library services without emphasis on response to the covid-19 pandemic.   approach to systematic literature review   the preferred reporting items for systematic reviews and meta-analyses (prisma) protocol for was adopted in this study. prisma is mainly used for reporting systematic reviews and meta-analysis of medical interventions and randomized trials (moher et al., 2009; liberati et al., 2009). it was created by an international network of health-based collaborators to provide the framework for systematic literature review (slr) to ensure methodological rigor and quality (pati & lorusso, 2018). since our study focuses on library services provision in response to the covid-19 pandemic, we found it a useful protocol for reporting our approach for systematic literature review. the prisma protocol for systematic review has four stages, which are identification, screening, eligibility, and inclusion.   the first stage of the prisma protocol is identification, as potential articles in core databases in library and information science and other sources were identified. at the second stage, the articles were screened, applying the first stage of inclusion and exclusion criteria to the search results. the title and abstract of articles were screened. if articles seem potentially fit for the study, they were saved to an endnote library where duplicates were removed. at the third stage, full-text of articles were further screened for eligibility using the second stage of eligibility criteria. the full-text of selected articles were critically evaluated to check whether library services in response to covid-19 were discussed by the authors. the last stage is the final stage of the process which shows the articles finally included for systematic review and analysis. for the included articles, we created an excel spreadsheet, where two of the authors (poa and boa) individually noted the findings from the selected articles, including titles, authors, methods, country, library services, tools, and technology used for carrying out those services and the challenges faced by libraries. this was done to ensure rigor and avoid bias in analysing selected articles. authors met virtually via zoom to compare notes to clarify any conflicting issues. we resolved our differing opinions on the selected articles and created a table which summarized the included articles.   sources and search technique   the literature search was conducted in five databases; library and information science abstract (lisa) (1966-week 51, 2020), library and information science technology abstract (lista) (1960-week 51, 2020), library science databases (1970-weeek 51, 2020), and web of science (wos) core collection. the literature search was first conducted on september 16, 2020, and updated on december 13, 2020. as this is a rapidly growing area of research, a final google scholar search was additionally conducted on march 22, 2021, to account for the latest articles that have been published. we also aimed at capturing articles published in open access journals or outlets such as repositories through google scholar which may not be captured in proprietary databases. please see appendices a – d for the search strategies and results for each database.   table 1 search plan concept 1 library services concept 2 covid-19 “library service*” or “information service*” or “bibliographic instruction*” or “book drop” or “reference service*” or “reading promotion” or “research support” or “citation guide*” or “systematic review service*” or “digital scholarship” or “open access” or “inter-library loan” or ill or “teaching support” or literacy or “institutional repositor*” or "user service*” covid-19 or pandemic* or coronavirus* or 2019-ncov-2 or cov-19   to develop the search technique, we created a search plan (see table 1). we began with preliminary keyword searching of the two main concepts of our review, which are “library services” and covid-19. hence, we developed a list of alternative terms that have been used to represent these concepts in the literature. this is important because if an article did not mention library service but discussed services specific to libraries, it potentially would not be caught by our search. keyword searching was imperative to retrieve relevant articles that focused on library services and covid-19. we used quotation marks for key words that are in phrases so databases can recognize them as such and not crawl them as individual words.   as shown in figure 1 below, the full-text of 37 potentially eligible articles were assessed for final inclusion. out of the 37 articles, 23 articles were found eligible and finally included for systematic qualitative analysis. we excluded articles that did not discuss library services provision in response to the covid-19 pandemic, and we prioritized articles that had rigorous methodological approaches rather than personal views, reflections, and reviews. we excluded one article because it only focused on the use of social media by library and information professionals in disseminating covid-19 information, and no other library services were discussed.  similarly, we excluded eight articles because their contents did not relate to the focus of our study even though they discussed covid-19. for example, seven of these articles focused on mis-information, tools for marketing library services, open education resources (oers) and student access, oers consortium and academic libraries, literacy models for stay-home parents, using social annotation in online classes, preventative measures, and strategies for curbing the spread of covid-19. similarly, one article was excluded because the content was not substantial. four articles were excluded because they were anecdotal reports or authors’ perspectives without established methodological approaches. we wanted to make sure included studies were methodologically sound, and that authors collected data in one way or another, either through document analysis, survey, interview, observation, case study, and others. after applying our criteria, we included 23 articles for qualitative systematic analysis.   quality assessment of included studies   quality assessment is important to weighting studies in terms of quality and relevance to answering the research question (gough, 2007). the assessment of quality and relevance of the included studies was done later in the review because it was after mapping and data extraction that there was sufficient information to make the assessment (gough, 2007). the assessment of the methodological quality of the included studies were based on the independent evaluation of two reviewers (boa and poa) and active discussion until consensus was reached in the case of discrepancies. we assessed the methodological quality of the included studies using the mixed methods appraisal tool (mmat) version 2018 (hong et al., 2018).   mmat was chosen because it provides customized questions that can be used for assessing different study designs. these are qualitative, quantitative randomized controlled trials, quantitative non-randomized, quantitative descriptive and mixed methods studies (hong et al., 2018). the tool is not intended to score the studies based on their quality; hence we did not include scores in the quality assessment. however, we provided comments in instances where studies could not answer all the quality assessment questions to aid in interpreting findings of this review. the eligible studies (n=23) were categorized, with reference to mmat, into three types of study design: qualitative case studies (n=14), quantitative descriptive studies (n=8), and mixed-methods study (n=1). the categories on quantitative randomized controlled trials and quantitative non-randomized were excluded as none of the included articles used such study designs.   figure 1 prisma flow diagram   results   this section presents the results of the quality assessment and a synthesis of themes derived from the included studies to meet the aims of this review. our results examine library services provided, technologies used for providing those services, and the challenges facing libraries in the pandemic.   quality appraisal   all studies were subjected to two screening questions: (s1) are there clear research questions? and (s2) do the collected data address the research questions? as stipulated in mmat, if any study cannot answer yes to these questions, they cannot be assessed with mmat (hong et al., 2018). all included studies, except one, answered ‘yes’ to the first two screening questions of the mmat. studies that answered yes to these questions were then assessed using the specific questions relating to their design categories.    for the eligible qualitative studies (n=14), 11 of the studies are judged to be of high quality. two studies were judged to be of medium quality as there were no sufficient description of the methodological approach, which makes it difficult to ascertain if their research questions were properly addressed. one study (i.e., howes et al., 2021) did not have clear research questions, hence could not be assessed. for the eligible quantitative descriptive studies (n=8), there were concerns related to sampling, including issues around sampling technique and lack of clarity on the population to justify the selected sample. out of the eight quantitative studies, three were judged to be of low quality, while four were deemed to be of medium quality and only one study was judged to be of high quality. the only mixed-method study in the included studies was judged to be of low quality as it does not answer three of the five appraisal questions related to integration and interpretation of data and findings from both quantitative and qualitative data. no studies were excluded from the study based on the quality assessment outcome. however, the quality appraisal shows that there is need for rigorous methodological approaches to provide more transparent, measurable, and replicable research focusing on library services provided in response to the covid-19 pandemic. see appendix e for details of the quality assessment of each study.   study characteristics   the majority of the studies were conducted in the united states of america (n= 6, 26.1%), followed by india (n=4, 17%), and china (n=2, 8.7%). the remaining studies were carried out in united kingdom, ireland, canada, mexico, romania, czech republic, indonesia, pakistan, nigeria, lesotho, and zimbabwe. the most common method used in selected studies was the case study (n= 11, 48%), followed by survey (n=7, 30.4%), content analysis (n=4, 17.4%), and mixed methods (n=1, 4.3%). the majority of the studies were carried out in academic libraries (n=17, 74%), while the rest were based on medical, public, and special libraries. this may indicate that academic librarians are most likely to publish, and higher institutions are assiduously seeking ways to ensure library services provision continues to meet the needs of their patrons.   table 2 summary of included articles for qualitative synthesis s/n authors country type of library methods services offered tools and technologies used challenges   howes et al. (2021)   usa medical library case study 1. virtual reference services. 2. instructional workshops 3. literature search services 4. document delivery services through interlibrary loan 5. online tutorials and e-learning through goskills 6. covid-19 libguide 6. expanding access to electronic resources webex, zoom, endnote microsoft teams camtasia connectwise 1. disruption of normal operations procedures 2. rapidly changing protocols and administrative directives   mehta & wang (2020) usa academic library case study 1.digital references services 2.bibliographic instructions (bi) 3.providing electronic resources 4.digital repositories services   zoom, blackboard, microsoft teams, libchat and libanswers   1.challenging digitization process for non-born digital resources 2. library staff's lack of access to a computer 3. difficulty conducting reference services and bi online 4. difficulty with collaboration among academics and administrative units   see (2020) usa academic library case study 1. circulation services 2. equipment/laptop lending 3. research support 4. document delivery services 5. course materials reserve. 6. inter-library loan markerlab, 3d printer laptops, and media (dvds & cds) check-out was eliminated for all production equipment because of difficulty in disinfecting   tolppanen (2021) usa academic libraries survey 1. virtual services 2. inter-library loan 3. circulation services and book drop through courier services 4. document delivery through sending scanned copies telephone, email, chat tools, dvds, cds providing access services in a pandemic   weeks et al. (2020) usa academic health sciences library case study 1. health literacy 2. research and scholarship support 3. user query services through libanswers 4. resources provision for teaching and learning support 5. access to free remote services 6. community and healthcare support and outreach 7. public health information 8. collaboration and partnerships 9. 3d prototype printing of ppe (face masks, shields and visors) and donation to student wellness google hangout and webex for online meetings. youtube, facebook, email, libanswers budget tightening was predicted for the next few years   yu & mani (2020) usa academic medical/health sciences libraries observational study, using content analysis 1. remote access to library resources 2. interlibrary loan 3. checking out laptops and other hardware for remote learning 4. virtual workshop and instruction session 5. access to open resources from publishers. 6. covid research publication search strategies to preprint publications 7 health literacy library website, 3d printer covid-19 open research dataset (“cord-19”) none reported   chakraborty, et al. (2020) india all libraries content analysis 1. providing list of all union territories, state-wise, national and international and help lines, and websites for authentic source of information about covid-19 websites none reported   dadhe & dubey, 2020 india special libraries content analysis   1. remote access to e-resources 2. free and expanded access to e-resources 3. open access to resources, including covid-19 materials 4. information searching and scholarly writing 5. organizing online activities for user engagement 6. access to print materials through scanning and online delivery of documents 7. virtual reference services 8. research support 9. information literacy on covid-19 plagiarism checking tools, email, repositories, websites.   none reported   kaur & mahajan (2020). india academic library mixed method 1. disseminating information about information sources and services 2 providing access to databases whatsapp, gmail none reported   neog (2020). india academic library survey 1. reference services 2. open access services 3. access to covid19 resources 4. access to web opac facebook, twitter, blog, whatsapp 1. users’ low response to library services 2. lack of policy guidelines 3. lack of high speed internet   guo et al. (2020) china academic library survey 1. providing timely covid-19 information 2. providing open-access resources 3. virtual reference services 4. online research support services 5. providing out of library disaster relief 6. replacing paper reports with electronic stamped reports when carrying out citation search services wechat, microblog mobile library app, chaoxing learning link telephone, email   none reported     ma (2020) china academic library case study 1. health and well-being services 2. access to library collections and e-resources 3. instruction and teaching support 4. circulation services through self-checkout 5. research support through bibliographic management training 6. virtual reference services through zoom-with-a-librarian zoom, endnote, refworks, repository none reported   cox & brewster (2020) united kingdom academic libraries survey 1. giving students access to additional learning materials online 2. reassuring messages via social media 3. reorganizing website to emphasize digital support options 4. suspension of fines 5. linking to home university well-being related services 6. recommending leisure reading and content streaming sources 7. listing remotely-available well-being related books 8. highlighting lighthearted and/or distracting content via social media 9. offering webinars on topics students are concerned about and well-being related topics social media none reported   carbery et al. (2020) ireland medical libraries, public libraries, academic libraries case studies 1. teaching information literacy and instructional sessions online 2. providing temporary access to library classroom for health executives to train healthcare professionals 3. 3d printing of face shields, mask clips and personal protective equipment (ppe) 4. creation of online study room – a virtual mediated reading room 5. creation of digital assets and resources addressing mindfulness during the pandemic 6. online educational, cultural, parenting events, online exercise and yoga classes, story time sessions in irish and english, arts and crafts workshop 7. free book delivery service to homes 8. lending 3d printer to surfbox and laptops to essential service workers 9. open access services zoom, microsoft teams, 3d printer 1. moving academic libraries services online 2. providing online service for academic staff   walsh & rana (2020) canada academic library case study 1. instructional services support for remote teaching and learning 2. research support 3. institutional collaborations 4. promoting covid-19 research resources 5. promoting open scholarship through institutional repositories 6. remote and reference information service digital repositories, synchronous chat system   budgetary challenges   saavedra-alamillas et al. (2020) mexico academic libraries document and content analysis 1. document delivery services (dds) 2. inter library loans (ill) whatsapp, google drive 1. problems with remote access to digital library 2. information search problems 3. problems with downloading documents   erich (2020) romania several libraries case study 1. providing covid-19 information 2. providing access to library’s digital resources 3. suspension of fines 4. organizing online activities facebook none reported   pokorna et al. (2020) czech republic academic libraries  case study 1. digital library model with open remote access 2. enabling licence agreements to provide access to digital copies 3. adopting an open-source dl system 4. restriction-free access to monograph kramerius and fedora repository 1. a wide spectrum of documents are not available digitally 2. inaccessibility of copyright-protected literature 3. piracy of copyright-protected documents 4. the digital library fell short at replacing loan services or reference reading rooms   winata et al. (2020) indonesia academic libraries case study 1. book delivery and circulation services in the drive-through 2. provision of e-resources for research support 3. virtual references services using “ask librarians” form 4. transforming final paper project services into online drobox, a computer circulation desk, smartphone, twitter, instagram, youtube, facebook 1. internet and network problems 2. adjusting to online mode of information provision 3. service limitation in working hours 4. budget challenges as a result of converting services to online mode   ishtiaq et al. (2020) pakistan academic libraries survey 1. vpn access 2. email management 3. live chat 4. mobile application services 5.  literacy sessions 6. webinar on crisis 7. virtual reference services gmail, facebook, youtube, linkedin, instagram, facebook messenger, google plus none reported   kasa & yusuf (2020) nigeria academic library survey 1. dissemination of e-books, videos, and sending of links to resources telegram 1. poor internet service 2. collaboration challenges 3. lack of private working space at home   mbambo-thata (2021) lesotho academic library case study 1. providing off-campus access to digital library 2. online information literacy program 3. online reference service on facebook and email chat 4. bearing data cost facebook, gmail, websites none reported   tsekea & chigwada (2020) zimbabwe academic libraries survey 1. research support 2. teaching support through learning management system 3. information literacy 4. collaboration with publishers for free resources 5. virtual references services 6. remote access to e-resources facebook, whatsapp, skype, twitter, youtube 1. lack of internet connection at home 2. negative attitude by the community in using technology 3. lack of technical knowhow on using digital libraries 4. challenges with servers’ connectivity   table 2 provides the summary of library services provided in the included articles, technologies used, and challenges libraries faced in response to the covid-19 pandemic. overall, we identified eight broad library services in the included studies. the most frequently offered library service is remote digital access to library resources (n=16, 69.6%), followed by virtual education and teaching support (n=13, 56.5%), public health safety information provision and literacy (n=12, 52.2%), open access services through collaborative efforts (n=11, 47.8%), and virtual reference services (n=10, 43.5%). others include circulation and lending services (n=9, 39.1%). research support services (n=7, 30.4%), and welfare and outreach services (n=6, 26.1%). we grouped the results by sorting the table by countries (e.g., putting all usa results together).   synthesis of themes   library services provision in a pandemic   we identified eight broad categories of services provided by libraries in the included studies. these are 1) virtual education and teaching support, 2) research support, 3) open access services through collaborative efforts, 4) public health safety, information provision and literacy, 5) virtual reference services, 6) remote digital access to library resources, 7) welfare and outreach, and 8) circulation and lending services.   virtual education and teaching support   despite library closure, findings show that libraries have continued one of their core missions – supporting the educational objectives of their institutions. results show that the majority of academic libraries in the included studies are supporting and facilitating remote access to digital contents in support of teaching and learning in universities (carbery et al., 2020; tsekea & chigwada, 2020; ishtiaq, sehar & shahid, 2020). more specifically, academic librarians are assisting teaching staff to upload contents on learning management systems (tsekea & chigwada, 2020) and providing digital services to teaching faculty and students through digital library and institutional repositories (mehta & wang, 2020). copyright has been an ongoing concern for many libraries. however, since there is an increasing demand for digital contents in the pandemic, several academic libraries are obtaining permissions from publishers and providing students and academic staff with access to digital contents including copyright-protected works for the period of the lockdown (e.g., ma, 2020; pokorna et al., 2020). within universities, libraries are making course readings accessible by linking to digital resource equivalents that are copyright-compliant (e.g., walsh & rana, 2020).   other efforts to support teaching and online education include sitting on curriculum committees, helping with curriculum development through instruction support, collaborating with academic faculty and providing strong liaison supports for both students and staff (weeks et al., 2020). as schools close, school libraries now promote the use of other online course platforms for autonomous learning, examination resources, vocational training, and self-improvement (guo et al., 2020), as well the use of open educational resources (mehta & wang, 2020). in addition, academic libraries are granting students additional access to online resources (cox & brewster, 2020), engaging them in virtual consultations via google hangout meetings and email communications (carbery et al., 2020; weeks et al., 2020), as well as compiling lists of virtual tours to museums, operas, aquariums, zoos and national parks (yu & mani, 2020). this is to ensure students adhere to government lockdown restrictions by studying at home while having the opportunities to virtually visit places they might have visited in-person. in addition, libraries are offering online training modules through goskills (e.g., howes et al., 2021), as well as webinars on topics of concern for students, including accessing resources remotely, well-being and mental health (cox & brewster, 2020).   research support   one of the prominent services discussed in the included articles is research support services. despite the lockdown, the majority of the libraries in the included studies have continued to provide many vital services such as reference services, document deliveries, literature searching and syntheses in support of the research efforts of students, faculty, and other user categories (guo et al. 2020; howes et al., 2021; mehta & wang, 2020; tsekea & chigwada, 2020; walsh & rana, 2020; weeks et al., 2020; winata, fadelina & basuki, 2020). librarians provide vital information services to support students, researchers, academics, medical and paramedical professionals with recent relevant studies, new developments in vaccine production, diagnosing procedures, and intellectual property (guo et al., 2020; walsh & rana, 2020). furthermore, librarians at the medical libraries in included studies are providing covid-19 information and literature in support of the researchers, faculty and medical researchers (dadhe & dubey, 2020; tsekea & chigwada, 2020; weeks et al., 2020). for example, librarians with health sciences expertise at the health sciences library of the university of nevada, las vegas, created a coronavirus guide containing several pages including information for the public, information for heath care providers, crisis support, and also addressing issues of racism and xenophobia during covid-19 (weeks et al., 2020).   other efforts include providing user-tailored support to research ranging from advanced workshops to information sessions, networking events, customised personal assistance, research data management, synchronous online chatting services, and publishing support (walsh & rana, 2020). some academic libraries are engaging in bibliographic management using tools such as endnote and refworks (ma, 2020) and plagiarism checking in research papers (dadhe & dubey, 2020). in some cases, librarians are providing researchers with access to electronic resources containing e-journals, e-books, as well as links of journal publishers and supported databases (winata et al., 2020). academic libraries have witnessed an increase in request for literature searches via email and webform as librarians provide training on literature searching through webex conferences (e.g., howes et al., 2021). some libraries have developed unified certification and resource sharing infrastructures to facilitate online research for college teachers and students (guo et al., 2020; pokorna et al., 2020).   open access through collaboration efforts   findings show that libraries are increasing their effort in providing open access services by increasing partnerships with publishers and databases providers (pokorna et al., 2020; walsh & rana, 2020; weeks et al., 2020). libraries and the society at large have come to better understand the benefits of open access in which research is made open and available for reading and reuse (walsh & rana, 2020). for example, in canada, the university of toronto is enhancing open access to medical research in fighting the spread of covid-19 and helping researchers make their research open through the libraries’ scholarly communications and copyright office. other academic libraries in the included studies are providing open access services through temporary access to resources provided through the library consortium and collaboration, and access to free covid-19 resources (guo et al., 2020; howes et al., 2021; weeks et al., 2020; yu & mani, 2020).   the current situation has improved libraries’ collaboration effort with publishers and database providers to create emergency temporary access services to allow in-copyright physical items to remain accessible (carbery et al., 2020; walsh & rana, 2020). librarians are helping to distribute articles, papers, case reports, and other resources from free or open access initiatives by database providers, publishers, and internet companies (walsh & rana, 2020). other efforts include ensuring that open access collections and resources were accessible and visible to end-users (carbery et al., 2020; dadhe & dubey, 2020) and that open access journals are evaluated for credibility and quality (mbambo-thata, 2021). the current situation has echoed the importance of collaboration usually referred to as a consortium among libraries. beyond library-to-library collaborations, librarians are now collaborating with government and health institutions for increased access to covid-19 resources (walsh & rana, 2020). this is particularly beneficial, as collaborating with other countries who have already developed valuable informational webpages will help those with less resources to do so.   public health safety, information provision and literacy   public health is largely dependent on the acceptance of changes to social norms, which span from individual actions to bigger shifts in socioeconomic institutions. findings show that librarians – especially in medical and public libraries – can help this process by facilitating awareness campaigns and disseminating information to prevent transmission of the virus. this include respecting and supporting governing regulations for remote learning, working from home, quarantining, and library closures (chakraborty et al., 2020; erich, 2020; guo et al., 2020; neog, 2020). libraries are advocating for and promoting the importance of wearing masks, hand washing, use of sanitizers, and social distancing (dadhe & dubey, 2020; guo et al., 2020; weeks et al., 2020), as well as promoting health literacy by providing guidance on evaluating information and fact-checking (yu & mani, 2020). other types of libraries, especially medical and health sciences libraries have promoted public health literacy by creating libguides, lectures, tailored information sessions, and workshops, both on covid-19 and general health recommendations (guo et al., 2020; howes et al., 2021; walsh & rana, 2020; weeks et al., 2020).  libraries now play a significant role in raising awareness and preventing the spread of misinformation through social media channels (i.e., twitter, whatsapp, instagram, facebook, wechat, and others) by sharing reliable information with patrons through institutional and online media accounts (guo et al., 2020).   findings show that several medical libraries are striving to provide access to resources targeted toward improving public health and safety. for example, several libraries are using their makerspaces for 3d printing of face masks and personal protective equipment (ppes) to be used by medical practitioners and staff (carbery et al., 2020; dadhe & dubey, 2020; see, 2020; weeks et al., 2020; yu & mani, 2020). some libraries have responded to national needs by 3d printing a range of kit and tools for frontline workers, including visor shields, mask clips, and other 3d objects to support the needs of community healthcare workers and student wellness centers (carbery et al., 2020; weeks, 2020). the increasing need for ppes has made some libraries continue opening of makerspaces for 3d printing despite closing other sections such as production studios, photography studio, and the virtual reality learning studio (see, 2020). this shows that libraries are important in ensuring that the right health information is provided to users, and also helping to fill the gap in health resources shortages through the printing of ppes.   virtual reference services   one vital service provided by libraries in selected articles is virtual reference services. as a result of the lockdown, there has been a significant increase in virtual reference desk use (ishtiaq et al., 2020; mbambo-thata, 2021; mehta & wang, 2020; see, 2020). virtual references services were provided in the form of literacy sessions, live chat, and email communication (dadhe & dubey, 2020; ishtiaq et al., 2020; see, 2020). academic libraries are providing reference services through chat facility on their websites, while subject-specific queries were answered by subject librarians (mbambo-thata, 2021). for example, at the university of lesotho library, special effort was made to include staff phone numbers, facebook, and e-mail contact in subject libguides so users can interact with humans when writing with subject-specific queries (mbambo-thata, 2021). similarly, at bridgewater state university (bsu), virtual references were provided to cater for users’ research consultations through libchat and libanswers with the use of email, short messaging service (sms), instant messaging (im). and frequently asked questions (faq) (mehta & wang, 2020). this is similar to the virtual reference service provided at the university of toronto libraries through the ask/chat with librarians, a synchronous online information support to researchers (walsh & rana, 2020).   at the health sciences library of the university of nevada, las vegas, reference service was primarily provided by booking online consultations using springshare’s libcal, while library staff use libanswers to collect, monitor, and distribute online reference requests (weeks et al. 2020). other libraries have provided virtual reference services through social media platforms such as whatsapp, facebook, and twitter (neog, 2020; tsekea & chigwada, 2020), remote system troubleshooting through connectwise (howes et al., 2021) and a service such as zoom-with-a-librarian (ma, 2020). to be effective in attending to multiple virtual reference queries at maxwell library of bsu, reference librarians created a schedule of two hours slot between 8:00 a.m. and 4:00 p.m., signing on to libchat and answering questions (mehta & wang, 2020). virtual reference services have remained pivotal and librarians have responded very well while working remotely to provide this important service to their users.   remote digital access to library resources   as there has been a significant increase in the demand for online resources, findings show that libraries have also increased their digitization efforts by scanning articles and book chapters, as well as retrieving dvds and cds from physical collection for digitization purposes (tolppanen, 2021). some libraries are using existing software for registered users and demanding amendments to the copyright act to allow for licence agreements so more digital resources can be utilized for non-profit purposes and research (pokorna et al., 2020). in the czech republic, libraries are granting users remote access to copyrighted documents including monographs and periodicals (pokorna et al., 2020). some libraries have enhanced the level of access to online materials by putting temporary database and other electronic resource trials in place (carbery et al., 2020; cox & brewster, 2020). libraries are making special efforts to manage all the digital infrastructure, library management systems, institutional repositories, and websites to ensure users have access to open access collections and resources being provided by multiple suppliers (carberry et al., 2020). this includes re-organizing library websites to emphasize digital support for accessing online resources, ensuring the digital well-being of patrons (cox & brewster 2020). findings show that libraries are helping users effectively access digital library resources and other subscription databases (erich, 2020; guo et al., 2020), leveraging and expanding access to electronic resources and services (dadhe & dubey, 2020; howes et al., 2021). more so, libraries are making an effort to negotiate and cooperate with database providers to strive for preferential policies for access and expanding digital resource services (guo et al., 2020; mbambo-thata, 2021).   to ensure continued access to digital library resources, suggestions have emerged to make access more user-friendly, as well as making more online and digital sources available (howes et al., 2021; kaur & mahajan, 2020). libraries are making a concerted effort to facilitate off-campus access to digital library and resources. this includes subscribing to remotex, a system that enables off-campus library access (mbambo-thata, 2021), and the use of a virtual private network (vpn) login for remote databases access (ishtiaq et al., 2020; guo et al., 2020). digital resources have been used to support the teaching, learning and research needs of faculty and students, made possible by ensuring remote digital access to libraries resources, databases, and archives (mehta & wang, 2020; neog, 2020; tsekea & chigwada, 2020; winata et al., 2020; yu & mani, 2020), scanning pages of print materials and making them available through online delivery of documents (dadhe & dubey, 2020)   welfare and outreach services   findings from the selected articles show that the mental health and well-being of library users are being prioritized by some libraries. welfare services and activities were carried out in libraries in the form of linking students to home university well-being related services, providing out-of-library disaster relief, providing community healthcare support and outreach (carbery et al., 2020; cox & brewster, 2020; guo et al., 2020; weeks et al., 2020). in some cases, libraries support students’ well-being by acquiring new collections, provision of information, the creation of dedicated spaces or spatial redesign. imaginative approaches, such as animal petting were relatively rare (cox & brewster, 2020). other efforts include using social platforms such as facebook, twitter, and instagram to promote social welfare entitlements, community, and well-being support. in addition, libraries are providing relief support through volunteer services, donating materials and money, providing reading therapy services (guo et al., 2020), as well as providing hand sanitizer at service counters and book sterilizers at circulation points (ma, 2020) for example, wuhan university library provided special collections to centralized isolated zones for medical staff and isolated population to read, and established a reading station for a mobile “cabin hospital” that treated coronavirus infected patients with mild symptoms (guo et al., 2020).   circulation and lending services   the circulation services of libraries have continued despite the lockdown. findings show that libraries are providing inter-library loans and document delivery services through docline and oclc (howes et al., 2021; saavedra-alamillas et al., 2020), while some are lending e-books, videos, and providing access links to e-resources (e.g., kasa & yusuf, 2020). the lockdown has prompted a shift to online lending services. in exceptional cases, some libraries have managed to stay partially open to provide services to a large number of international students who live on campus (see, 2020; tolppanen, 2021). for instance, cline library of northern arizona university has managed to stay open by reducing opening hours, and by providing a modified, self-mediated check-out station with two return carts (see, 2020). some libraries have provided drive-through book delivery services (winata et al., 2020), checking out laptops, media (dvds & cds), and other hardware in a socially distanced library space (carbery et al. 2020; see, 2020; yu & mani, 2020). to return borrowed books, laptops and other media equipment, some libraries have made special provisions such as external book drop through courier service (tolppanen, 2021). others have provided drop boxes and return carts where users can drop borrowed items placed along the service point area without having to physically be in contact with librarians (see, 2020; winata et al., 2020). some libraries have employed book sterilizers for returned books (ma, 2020) and designated a period of time to quarantine returned items (tolppanen, 2021). these efforts have ensured continued access to libraries’ educational and hardware resources. many libraries have suspended overdue fines of borrowed books and physical resources (cox & brewster, 2020; dadhe & dubey, 2020; ma, 2020; tolppanen, 2021). owing to increasing need for 3d printing of ppes, some libraries have been lending 3d objects (see, 2020) and 3d printers, virtual reality tools, and a google jamboard (carbery et al., 2020; weeks et al., 2020).   technological tools used for library services provision   there are several technological tools used for providing library services in response to the covid-19 restrictions in the included studies. we categorized them into four broad areas: i) social networking tools; ii) video conferencing tools; iii) educational technologies, and iv) makerspaces technologies.   social networking tools   social media platforms were used heavily for promoting library services, live streaming, storytelling, and reaching out to patrons. examples of social networks tools used in the selected articles include facebook, facebook messenger, twitter, wechat, whatsapp, instagram, linkedin, telegram, and google plus (carbery et al., 2020; ishtiaq et al., 2020; kasa & yusuf, 2020; guo et al., 2020; winata et al., 2020). with the use of social networking sites, patrons can engage in social tagging to contribute to folksonomies (pokorna et al., 2020). social networking tools have enabled libraries to remain connected with their patrons, offering services, attending to their needs and keeping them abreast of the latest developments regarding research, covid-19 restrictions, and library openings and other related information.   videoconferencing tools   findings show that several videoconferencing tools and technologies are harnessed for virtual meetings, teaching, and presentations. the common videoconferencing tools used in the selected studies include skype, zoom, webex, google hangouts, and microsoft teams (carbery et al., 2020; howes et al., 2021; ma, 2020; mehta & wang, 2020; tsekea & chigwada, 2020; weeks et al., 2020). zoom and webex appear more popular among the libraries in the selected articles. these tools are useful for conducting virtual reference, supported by telephone, email, social media messaging, and other methods of contact (guo et al., 2020). educational technologies   educational technologies include those promoting e-reading, video presentations, educational programs and learning management systems, particularly in academic libraries, and include mobile library applications such as chaoxing learning link (guo et al., 2020), youtube, camtasia for screen recording and blackboard (howe et al., 2021; ishtiaq et al., 2020; tsekea & chigwada, 2020; weeks et al., 2020). blackboard was primarily used as learning management system, and most academic libraries in the included studies use canvas to embed e-resources such as databases, e-books, and online research guides (mehta & wang, 2020). networking tools such as vpns, off-campus direct access, remote access to databases, and servers protected by shibboleth authentication were used to help patrons’ access to online electronic resources and to meet their information needs (guo et al., 2020; pokorna et al., 2020). due to the online and digital nature of resources, there has been an increasing use of cd and dvds from printed books to aid the digitization efforts of libraries (e.g. tolppanen, 2021). to enhance the research activities of students and faculty, libraries are engaging in research data and reference management with the use of tools such as endnote and refworks (howes et al., 2021; ma, 2020). some libraries are harnessing plagiarism checking tools to help researchers check for and prevent plagiarism in their research (dadhe & dubey, 2020). these technologies are imperative to ensure continued access to library resources irrespective of patrons’ geographic location.   makerspaces technologies   makerspaces technologies were used to provide library services in medical libraries in the included articles in the form of 3d printers. these were used for 3d printing of a range of kit and tools for frontline workers, including facemasks, visor shields, mask clips, and other 3d objects to support the needs of community healthcare (carbery et al., 2020; see, 2020; yu & mani, 2020). this shows that libraries have remained relevant in meeting the health needs of community with the inventions from makerspaces.   challenges facing libraries in the pandemic   there are several challenges facing libraries because of the global pandemic. as there were sudden shifts in operation, libraries have had to change or adapt existing policies on physical items by extending loan time or suspending fines (mehta & wang, 2020; erich, 2020; cox & brewster, 2020). in instances where libraries could remain partially open, librarians and library staff experienced increased anxiety of being exposed to the virus (mehta & wang, 2020; see, 2020). most concerns centred around disruption of normal operation procedures (dadhe & dubey, 2020), and shifting services to digital environment as physical spaces were closed (mehta & wang, 2020). even when resources were made available online, there were difficulties with some library staff and patrons not knowing how to download and use e-resources, ict platforms, and digital libraries (mehta & wang, 2020; saavedra-alamillas et al., 2020; tsekea & chigwada, 2020). this led to difficulties in conducting reference services online – especially for collaboration among research and administrative units (mehta & wang, 2020).   some academic and public libraries have expressed a need for new ict structures and internet service to better support reference desks and streaming activities as demand for virtual services increases (mehta & wang, 2020; neog, 2020; kasa & yusuf, 2020; tsekea & chigwada, 2020; winata et al., 2020). in moving academic libraries’ services online and providing online service for academic staff, libraries have faced additional challenges such as budgetary challenges (carbery et al., 2020; winata et al., 2020). several studies reported budgetary and logistics-related challenges in the cost of training and maintaining open library spaces (mehta & wang, 2020; walsh & rana, 2020; weeks et al., 2020; winata et al., 2020), as well as a prediction of budget tightening for the next few years (weeks et al., 2020). more so, some mandates required elimination of check-out for all production equipment because of the difficulty in disinfecting them upon return (see, 2020). in some cases, students were not able to access course reserves that were in high demand because they were only available in physical copies (mehta & wang, 2020).   discussion   this review found that libraries are providing several services in response to the covid-19 pandemic to meet the changing needs of their users. one of the prominent services provided by almost all the libraries in the included studies is remote digital access to library resources. libraries are providing digital support in terms of instruction, webinars, and off-campus access to electronic resources, databases, and archives, as well as negotiating and collaborating with publishers and database vendors to increase access to electronic databases and resources. before the pandemic, several efforts have been made to increase collaboration and partnership among libraries to solve problems that are beyond the ability of a single institution (maurer, gammon, & pollock, 2013; patterson, 2009; simpson, 1997). this perhaps owes to declining financial support, coupled with the ever-increasing need to incorporate new technology into library services (burich, casey, devlin, & ivanitskaya, 2006). in the current milieu, institutional collaboration is particularly important as most libraries, schools, and universities have closed their doors, leading to increasing demand for access to electronic resources by students, researchers, and faculty. this lends support to klucevsek and brungard’s (2020) submisssion that libraries are increasing the provision of e-books, electronic journals, and enhancing access to online databases as physical access is limited or restricted in most cases. this is also beneficial to publishers as digital distribution of resources via publisher websites has greatly reduced the costs of dissemination – although the cost of copyediting, marketing and peer-review remains (tavernier, 2020).   another prominent service provided by libraries is virtual education and teaching support. this review found that academic libraries in particular are supporting and facilitating remote access to digital content in support of teaching and learning in universities. librarians and library staff are assisting faculty by uploading teaching content on learning management systems, sitting on curriculum committees, and advising on curriculum development and copyright issues around digital access to electronic resources. libraries have an inseparable connection with education and are considered a place for transfer of knowledge and information. results show that library staff are working with instructors to make course materials accessible electronically by creating or linking to digital, copyright-compliant equivalents of textual and au­diovisual materials. this is in support with soleymani, mojiri and zadeh (2017) who posit that the library acts as a powerful support for education by providing information resources and the necessary resources for students without the limitation of time and geographic location. this also supports the recommendation by santos (2020) that libraries can host continued education in-house, which includes offering language lessons, reading clubs, and learning programs.   the virtual education support provided by libraries has been bolstered by an increasing need for open access. this review found that more libraries are engaging in open access services in response to the covid-19 pandemic. this is made possible as publishers are removing the paywall for covid-19 information and published articles. database providers and publishers, including leading publishers such as oxford, elsevier, bmj, nature, wiley, emerald, and cambridge are now helping to make more resources available via open access. open access, which advocates that the outcome of scholarly research be made open and freely accessible to everyone with access to the internet (bjork, 2004; suber, 2003) has seen an increasing uptake during the pandemic. this is in line with a recent study by homolak, kodvanj and virag (2020) which echoes the need to adopt open science principles to accelerate the discovery of covid-19-related data for more efficient solutions. as such, the commitment by big publishers including the lancet and cell press in 2020 to share research data and findings relevant to the novel coronavirus has demonstrated the value of open access (tavernier, 2020). however, researchers need to be careful in their quest to use and publish in open access platforms as there is a growing influx of predatory journals with qestionable practices and dubious quality (ayeni & adetoro, 2017; burggren, madasu, hawkins, & halbert, 2018; frederick, 2020; zhao, 2014). researchers have to constantly sieve disinformation and misinformation in research results relating to the covid-19 pandemic (baines & elliott, 2020), which are usually spread through platforms that are open and easily accessible to the public.   in addition, there is an increasing effort from libraries to support the research endeavors of researchers and students, particularly those related to covid-19. findings show that librarians are providing vital information services to support students, researchers, academics, medical and paramedical professionals with recent relevant studies, new developments in vaccine production, diagnostic procedures, and issues around intellectual property. similarly, user-tailored support to research such as advanced workshops to information sessions have been increasingly provided to support research activities. this is in support with the findings of pauwels et al. (2020) that libraries are synthesizing research on treatments, diagnostic testing, specimen testing procedures, personal protection equipment, measures taken in nursing homes, the effect of the virus during pregnancy, and containing hospital spread. as research related to covid-19 has received increasing support and attention from government and funding agencies with over $205.4 billion as of december 20, 2020 (cornish, 2020), the place of libraries as the research hub of higher institutions cannot be overlooked.   this review also found that the circulation and lending efforts of libraries have continued despite the lockdown. this was made possible through inter-library loan, document delivery services, scanning of print collections, e-book and media technology loans such as laptops and dvds. this corroborates the submission by christenson (2011) that libraries are able to ensure their users have access to educational resources by using platforms such as hathitrust, a “shared digital repository owned and operated by a partnership of more than forty major research libraries” (p. 93). hathitrust has seen an upsurge in usage during the pandemic, as more resources are made accessible in digital formats. hathitrust currently enables free online access to full-text of millions of articles which used to be locked to only partner institutions when it was first launched (eden & beaubien, 2012). these innovations are helping libraries provide access to electronic resources needed by students and researchers amidst an elongated lockdown.   results reveal that libraries are providing public health information and ensuring safety of frontline workers through the 3d printing of facemasks, visor shields, and provisions of ppe. this has been helpful in supporting the ongoing efforts by governments to prevent and curb the spread of the covid-19 pandemic. most libraries in the included studies have been respecting and supporting government regulations by providing remote learning, working from home, and quarantining. many libraries have helped advocate the importance of wearing masks, hand washing, the use of sanitizers, and social distancing by providing online platforms and restricting access to physical spaces.  efforts have been made to disseminate public health information within individual libraries, as well as through larger affiliated associations. for example, the international coalition of library consortia (2020) issued a collective statement on the global effects of covid-19 and how libraries can collectively approach the pandemic while adjusting services and resources. the australian library and information association (2020), the association of research libraries (2020), the international federation of library association and institutions (2020) and the institute of museum and library services in the usa (2020) have compiled library services provision since the pandemic while regularly updating their website (ali & gatiti, 2020).   beyond their traditional functions, this review found that libraries are now engaging in welfare services and reaching out to their patrons as well as providing resources that will help sustain the mental health and wellbeing of library users. this is done by providing relief support through volunteer services, donating materials and money, and providing reading therapy services. as the pandemic has brought about travel restrictions and lockdown in most countries, the mental health of library users may be declining. mental health and wellbeing support services from libraries are invaluable. this corroborates the findings of ramsey and aagard (2018) that academic libraries can contribute to students’ mental health and well-being through the creation of leisure reading collections and designation of spaces for contemplation or napping, and dog petting sessions. since most of these services are better provided in-person, it may be challenging to do so in an online space, which explains why there are few libraries with such services in the included articles.   limitations and recommendations for further study   since the focus of this study is a new area of literature, we recognize that this review may not provide an overarching overview of library services provision in all types of libraries all over the world, as the majority of the studies included focused on academic libraries. this may be because academic librarians tend to publish more than other librarians. further study is needed to identify services provided in specific libraries to cater for different user categories. limitation to english only may have introduced bias in this study as articles not written in the english language were not included. similarly, by not including the thesaurus terms for covid-19 through subject heading, we may have missed some alternative terms that have been used in the literature. our search may also be limited in scope as we did not search educational-related and other scientific databases that may have published studies related to the pandemic. nevertheless, we expected that our search in wos core collections and google scholar would have filled this gap.   furthermore, since the world-wide pandemic is still ongoing, new articles are likely to appear in the literature as time goes by. the studies included in this review show a general trend in library services provision, technological tools used in providing those services and challenges facing libraries in the pandemic from several countries of the world. this will help librarians – especially those in the academic and health sectors – in their strategic planning and quest for information services provision during and after the covid-19 pandemic. as shown in the quality assessment section, most of the included studies are case studies with few empirical studies which may weaken the weight of evidence of the included studies. since empirical studies investigating library services provision are currently sparce in the literature, further study is needed to fill this gap as the pandemic moves to an end. other studies could also focus on post-pandemic experiences of different types of libraries, accounting for varying geographical location and languages. this will help libraries navigate the new challenges they may face while ramping up their services in meeting the ever changing and complex user needs.   conclusion   this review analyzed the various services provided by libraries in response to the covid-19 pandemic. whereas eight broad categories of library services were identified in the selected articles, remote digital access, virtual education support, open access services and research support services were provided more by academic libraries. this review found that libraries are adapting to the remote nature of schooling and research as a result of lockdown, and they are providing relevant services to meet the information, research, and well-being needs of patrons. libraries also increased the provision of virtual services ranging from instructional and teaching services to reference services, digital access to e-resources, literature searching, and others. libraries are increasing their collaborative efforts with publishers, government agencies, local hospitals, and communities to ensure that necessary resources are made available to patrons without the barrier of access or geographic location. more resources are being made open access, enabling the continuation of research endeavors of students, researchers, faculty, and health researchers.   in some cases, medical libraries are lending out 3d printers and printing personal protective equipment to facilitate the safety of frontline workers who are short of these provisions. due to the online nature of most library services provided, several technological tools were used, ranging from videoconferencing applications to educational technologies, social networking applications, and makerspaces. some libraries are facing enormous challenges due to the pandemic. this owes largely to technological barriers as the majority of the included studies reported internet connectivity and bandwidth issues. for some libraries that have partially remained open, they reported budgetary restrictions which limited them from getting the necessary tools needed to keep safe (e.g., face masks, hand sanitizers, sanitizing returned items), as well as coping with the changing nature of librarians’ jobs and increased demands for digital services from patrons. for libraries to thrive in these trying times, there must be a well-structured approach to ensuring continuity of services provision while obeying government regulations towards containing the covid-19 pandemic. regardless of the opening date, it is important for libraries to improve on efforts to provide online digital services, which may see an upsurge in usage even when the pandemic is over. libraries should prioritize the acquisition of electronic resources as well as increase their efforts to digitize resources that are only available in print. this is especially important in meeting the needs of students and researchers in the humanities who may have many sources only available in print. as library services have predominantly shifted online, there should be concerted effort and support from government and funding agencies to equip libraries with the technological facilities needed to provide cutting-edge services. there should also be adequate provision of facilities and the equipment needed to keep librarians, library staff, and patrons safe.   libraries should provide a safe environment for circulation services by providing automated return carts or boxes, and self-check-out facilities which should be properly sanitized after every user activity. this has serious implications on increasing budgetary allocations needed to ensure safe and secure library operations. as regards libraries that are completely closed, pending the re-opening regulations from government, patrons’ needs should be prioritized, and services provision should cater for their particular situations. this may involve personalized library services, document delivery, pop-up libraries, information literacy sessions, instructional videos, webinars on how to access library resources, and off-campus access to selected library services, among others. libraries should improve their collaborative efforts with other libraries, publishers, database vendors, health, and government agencies to meet the predominantly digital needs of their patrons, supporting their virtual education, research and development, well-being, and safety.   author contributions   philips o. ayeni: conceptualization, investigation, methodology, formal analysis, project administration, supervision, visualization, writing – original draft, writing – review & editing blessed o. agbaje: conceptualization, investigation, methodology, formal analysis, visualization, writing original draft, writing review & editing maria tippler: investigation, formal analysis, writing original draft   acknowledgments   the authors would like to thank dr. rebekah jane willson, assistant professor, mcgill school of information studies and vera granikov, phd candidate, mcgill school of information studies for reviewing and providing suggestions and feedback on our manuscript.   references   articles included for the systematic review are indicated with asterisk *   ali, m. y. & gatiti, p. (2020). the covid-19 (coronavirus) pandemic: reflections on the roles of librarians and information professionals. health information and libraries journal, 37(2), 158-162. https://doi.org/10.1111/hir.12307   australian library and information association. (2020). australian libraries responding to covid-19. retrieved from https://www.alia.org.au/australian-libraries-responding-covid-19   ayeni, p. o., & adetoro, n. (2017). growth of predatory open access journals: implication for quality assurance in library and information science research. library hi tech news. 34(1), 17-22.  https://doi.org/10.1108/lhtn-10-2016-0046   baines, d., & elliott, r. (2020). defining misinformation, disinformation and malinformation: an urgent need for clarity during the covid-19 infodemic. discussion papers, 20-06 department of economics, university of birmingham. http://www.repec.bham.ac.uk/pdf/20-06.pdf   bjork, b. c. (2004). open access to scientific publications an analysis of the barriers to change? information research: an international electronic journal, 9(2), 15. http://hdl.handle.net/10227/647   briner, r. b., & denyer, d. (2012). systematic review and evidence synthesis as a practice and scholarship tool. in d. rousseau (ed.), the oxford handbook of evidence-based management: companies, classrooms, and research (pp. 112-129). (oxford library of psychology). oxford university press.   burggren, w., madasu, d. k., hawkins, k. s., & halbert, m. (2018). marketing via email solicitation by predatory (and legitimate) journals: an evaluation of quality, frequency and relevance. journal of librarianship and scholarly communication, 6(1). http://dx.doi.org/10.7710/2162-3309.2246   burich, n. j., casey, a. m., devlin, f. a., & ivanitskaya, l. (2006). project management and institutional collaboration in libraries. technical services quarterly, 24(1), 17-36. https://doi.org/10.1300/j124v24n01_02   *carbery, a., fallon, h., higgins, m., kennedy, e., lawton, a., & mccauley, c. (2020). irish libraries and covid-19: first reflections. insights, 33, 1-19. http://doi.org/10.1629/uksg.522   *chakraborty, k., kureshi, p. s. l., gajbe, s. b., upadhyay, n., & devi, d. (2020). role of lis professionals to provide authentic information sources during covid-19 a pandemic crisis. library philosophy and practice, 1-19. https://digitalcommons.unl.edu/libphilprac/4180   christenson, h. (2011). hathitrust: a research library at webscale. library resources & technical services, 55(2), 93-102. https://journals.ala.org/index.php/lrts/article/view/5226/6352   cornish, l. (2020). interactive: who's funding the covid-19 response and what are the priorities? devex. https://www.devex.com/news/interactive-who-s-funding-the-covid-19-response-and-what-are-the-priorities-96833   *cox, a., & brewster, l. (2020). library support for student mental health and well-being in the uk: before and during the covid-19 pandemic. journal of academic librarianship, 46(6), 102256. https://doi.org/10.1016/j.acalib.2020.102256   *dadhe, p. p., & dubey, m. n. (2020). library services provided during covid-19 pandemic: content analysis of websites of premier technological institutions of india. https://digitalcommons.unl.edu/libphilprac/4445/   eden, k., & beaubien, a. k. (2012). hathitrust: digital access at the intersection of interlibrary lending potential and the protection of intellectual property rights. interlending & document supply, 40(2), 94-99.  https://doi.org/10.1108/02641611211239560   *erich, a. t. (2020). libraries reaction of romania to the crisis caused by covid 19. postmodern openings, 11(3), 332-343. https://doi.org/10.18662/po/11.3/218   frederick, d. e. (2020). scholarly communications, predatory publishers and publish or perish culture in the 2020s. library hi tech news, 37(5), 1-11. https://doi.org/10.1108/lhtn-01-2020-0007   *guo, y. j., yang, z. n., yang, z. s., liu, y. q., bielefield, a., & tharp, g. (2020). the provision of patron services in chinese academic libraries responding to the covid-19 pandemic. library hi tech. 39(2), 533-648. https://doi.org/10.1108/lht-04-2020-0098   gough, d. (2007). weight of evidence: a framework for the appraisal of the quality and relevance of evidence. research papers in education, 22(2), 213-228. https://doi.org/10.1080/02671520701296189   homolak, j., kodvanj, i., & virag, d. (2020). preliminary analysis of covid-19 academic information patterns: a call for open science in the times of closed borders. scientometrics, 124(3), 2687-2701. https://doi.org/10.1007/s11192-020-03587-2   hong, q. n. et al. (2018). mixed methods appraisal tool (mmat) version 2018. registration of copyright (#1148552), canadian intellectual property office, industry, canada.   *howes, l., ferrell, l., pettys, g., & roloff, a. (2021). adapting to remote library services during covid-19. medical reference services quarterly, 40(1), 35-47. https://doi.org/10.1080/02763869.2021.1873616   institute of museum and library services. (2020). covid-19 resources for librarians and museums. https://www.imls.gov/coronavirus-covid-19-updates/covid-19-resources-libraries-and-museums   international coalition of library consortia. (2020). statement on the global covid-19 pandemic and its impact on library services and resources. https://icolc.net/statement/statement-global-covid-19-pandemic-and-its-impact-library-services-and-resources   international federation of library association and federation. (2020). covid-19 and the global library field. https://www.ifla.org/covid-19-and-libraries   *ishtiaq, s. m., sehar, n., & shahid, a. (2020). information dissemination during covid-19 and lockdown: the role of university libraries of sindh, pakistan. library philosophy and practice. https://digitalcommons.unl.edu/libphilprac/4280/   *kasa, m., & yusuf, a. (2020). experience of an academic library during the covid-19 pandemic. library philosophy and practice. https://digitalcommons.unl.edu/libphilprac/4456/   *kaur, h., & mahajan, p. (2020). library and information services of a.c. joshi library during covid–19 pandemic: a study of faculty satisfaction. library philosophy and practice, 1-20. https://digitalcommons.unl.edu/libphilprac/4464/ klucevsek, k. m., & brungard, a. b. (2020). digital resources for students: navigating scholarship in a changing terrain. portal : libraries and the academy, 20(4), 597-619. https://doi.org/10.1353/pla.2020.0040   kosciejew, m. (2021). the coronavirus pandemic, libraries and information: a thematic analysis of initial international responses to covid-19. global knowledge, memory and communication, 70(4/5), 304-324.  https://doi.org/10.1108/gkmc-04-2020-0041   liberati, a., altman, d. g., tetzlaff, j., mulrow, c., gøtzsche, p. c., ioannidis, j. p. a., clarke, m., devereaux, p. j., kleijnen, j., moher, d. (2009). the prisma statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. research methods and reporting.  https://doi.org/10.1136/bmj.b2700    *ma, l. f. (2020). academic library services during covid-19: the experience of cuhk library. international information & library review, 52(4), 321-324. https://doi.org/10.1080/10572317.2020.1834251   maurer, m. b., gammon, j. a., & pollock, b. m. (2013). developing best practices for technical services cross-institutional collaboration. cataloging & classification quarterly, 51(1-3), 179-193. https://doi.org/10.1080/01639374.2012.733795   *mbambo-thata, b. (2021). responding to covid-19 in an african university: the case the national university of lesotho library. digital library perspectives, 37(1), 28-38. https://doi.org/10.1108/dlp-07-2020-0061   *mehta, d., & wang, x. c. (2020). covid-19 and digital library services a case study of a university's library. digital library perspectives. 36(4), 351-363. https://doi.org/10.1108/dlp-05-2020-0030   moher, d., liberati, a., tetzlaff, j., altman, d. g., & the prisma group. (2009). preferred reporting items for systematic reviews and meta-analyses: the prisma statement. plos medicine 6(7), e1000100.  https://doi.org/10.1136/bmj.b2535   *neog, s. (2020). library services through social media during lockdown due to covid-19 with special reference to university libraries of assam. library philosophy and practice, 1-12. https://digitalcommons.unl.edu/libphilprac/4262/   paez, a. (2017). grey literature: an important resource in systematic reviews. journal of evidence based medicine, 10(3), 233-240. https://doi.org/10.1111/jebm.12265   pati, d., & lorusso, l. n. (2018). how to write a systematic review of the literature. herd: health environments research & design journal, 11(1), 15-30. https://doi.org/10.1177/1937586717747384   patterson, a. (2009). research support through resource sharing: challenges and opportunities for irish academic libraries. interlending & document supply. 37(2), 87-93. https://doi.org/10.1108/02641610910962328   pauwels, s. n., de meulemeester, a., romagnoli, a., buysse, h., & peleman, r. (2020). medical and health informatics services during and after the covid-19 pandemic should be virtual, tailored, responsive and interactive: a case study in belgium. health information & libraries journal, 38(1), 66-71. https://doi.org/10.1111/hir.12330   *pokorna, l., indrak, m., grman, m., stepanovsky, f., & smetankova, m. (2020). silver lining of the covid-19 crisis for digital libraries in terms of remote access. digital library perspectives. 36(4), 389-401. https://doi.org/10.1108/dlp-05-2020-0026   ramsey, e., & aagard, m. c. (2018). academic libraries as active contributors to student wellness. college & undergraduate libraries, 25(4), 328–334. https://doi.org/10.1080/10691316.2018.1517433   *saavedra-alamillas, c., pacheco-mendoza, j., garcia-melendez, h. e., lopez, a. v., martinez-camacho, h., ortiz-diaz, e. m., & ortega-martinez, e. (2020). document delivering services: a strategy to support research in mexican universities in the pandemic by covid-19. digital library perspectives. 36(4), 337-349. https://doi.org/10.1108/dlp-0505-2020-0031   santos, m. c. (2020). libraries respond to covid-19. texas library journal, 96(2), 64-72. https://www.webjunction.org/content/dam/webjunction/documents/webjunction/2020-08/tlj-summer-2020-innovation-leads-to-virtual-success.pdf   *see, a. (2020). weathering the storm: continuing essential services at an academic library during a global pandemic. against the grain, 32(4), 24-27. https://www.charleston-hub.com/2020/09/v324-weathering-the-storm-continuing-essential-services-at-an-academic-library-during-a-global-pandemic/   simpson, d. b. (1997). solving the challenges presented by electronic resources: creating opportunities through inter-institutional collaboration. journal of library administration, 24(4), 49-60. https://doi.org/10.1300/j111v24n04_06   soleymani, m., mojiri, s., & zadeh, m. (2017). the supporting roles of academic librarians in virtual education. int j educ psychol res, 3, 213-218. https://doi.org/10.4103/2395-2296.225095   suber, p. (2003). scholarly communication: removing barriers to research: an introduction to open access for librarians. college & research libraries news, 64(2), 92-113. https://doi.org/10.5860/crln.64.2.92   tavernier, w. (2020). covid-19 demonstrates the value of open access: what happens next? college & research libraries news, 81(5), 226. https://doi.org/10.5860/crln.81.5.226   *tolppanen, b. p. (2021). a survey of response of access services in academic libraries to covid-19. journal of access services, 1-11. https://doi.org/10.1080/15367967.2021.1871619   *tsekea, s., & chigwada, j. p. (2020). covid-19: strategies for positioning the university library in support of e-learning. digital library perspectives. 36(1), 54-64. https://doi.org/10.1108/dlp-06-2020-0058   *walsh, b., & rana, h. (2020). continuity of academic library services during the pandemic: the university of toronto libraries' response. journal of scholarly publishing, 51(4), 237-245. https://doi.org/10.3138/jsp.51.4.04   *weeks, a., houk, k. m., nugent, r. l., corn, m., & lackey, m. (2020). unlv health sciences library’s initial response to the covid-19 pandemic: how a versatile environment, online technologies, and liaison expertise prepared library faculty in supporting its user communities. medical reference services quarterly, 39(4), 344-358. https://doi.org/10.1080/02763869.2020.1826197   who. (2020a). what we know about the long-term effects of covid-19. https://www.who.int/docs/default-source/coronaviruse/risk-comms-updates/update-36-long-term-symptoms.pdf?sfvrsn=5d3789a6_2   who. (2020b). transmission of sars-cov-2: implications for infection prevention precautions. https://www.who.int/news-room/commentaries/detail/transmission-of-sars-cov-2-implications-for-infection-prevention-precautions   who. (2021). who coronavirus (covid-19) dahsboard – 18 july, 2021. https://covid19.who.int/   *winata, a., & fadelina, r. (2020). new normal and library services in indonesia: a case study of university libraries. digital library perspectives, 1-9. https://doi.org/10.1108/dlp-07-2020-0059   *yu, f., & mani, n. (2020). how american academic medical/health sciences libraries responded to the covid-19 health crisis: an observational study. data and information management, 4(3), 200-208. https://doi.org/10.2478/dim-2020-0013   zhao, l. (2014). riding the wave of open access: providing library research support for scholarly publishing literacy. australian academic & research libraries, 45(1), 3-18. https://doi.org/10.1080/00048623.2014.882873     appendix a lisa search strategy using building blocks approach set# searched for databases results s1 “library service*” or “information service*” or “bibliographic instruction*” or “book drop” or “reference service*” or “reading promotion” or “research support” or “citation guide*” or “systematic review service*” or “digital scholarship” or “open access” or “inter-library loan” or ill or “teaching support” or literacy or “institutional repositor*” or "user service*” library & information science abstracts (lisa) 298492 s2 covid-19 or pandemic* or coronavirus* or 2019-ncov-2 or cov-19 library & information science abstracts (lisa) 1521 s3 (“library service*” or “information service*” or “bibliographic instruction*” or “book drop” or  “reference service*” or  “reading promotion” or  “research support” or  “citation guide*” or  “systematic review service*” or “digital scholarship” or “open access” or  “inter-library loan” or ill or “teaching support” or literacy or “institutional repositor*” or "user service*”) and (covid-19 or pandemic*  or coronavirus* or 2019-ncov-2  or cov-19) library & information science abstracts (lisa) these databases are searched for part of your query. 773 s4 (“library service*” or “information service*” or “bibliographic instruction*” or “book drop” or  “reference service*” or  “reading promotion” or  “research support” or  “citation guide*” or  “systematic review service*” or “digital scholarship” or “open access” or  “inter-library loan” or ill or “teaching support” or literacy or “institutional repositor*” or "user service*”) and (covid-19 or pandemic*  or coronavirus* or 2019-ncov-2  or cov-19) and yr(2020) library & information science abstracts (lisa) these databases are searched for part of your query. 461 s5 (“library service*” or “information service*” or “bibliographic instruction*” or “book drop” or  “reference service*” or  “reading promotion” or  “research support” or  “citation guide*” or  “systematic review service*” or “digital scholarship” or “open access” or  “inter-library loan” or ill or “teaching support” or literacy or “institutional repositor*” or "user service*”) and (covid-19 or pandemic*  or coronavirus* or 2019-ncov-2  or cov-19) and (stype.exact("scholarly journals" not ("trade journals" or "magazines")) and yr(2020)) library & information science abstracts (lisa) these databases are searched for part of your query. 194     appendix b library science database search strategy using building blocks approach set# searched for databases results s1 “library service*” or “information service*” or “bibliographic instruction*” or “book drop” or  “reference service*” or  “reading promotion” or  “research support” or  “citation guide*” or  “systematic review service*” or “digital scholarship” or “open access” or  “inter-library loan” or ill or “teaching support” or literacy or “institutional repositor*” or "user service*” library science database,  library science database 368218 s2 covid-19 or pandemic* or coronavirus* or 2019-ncov-2 or cov-19 library science database,  library science database 3986 s3 (“library service*” or “information service*” or “bibliographic instruction*” or “book drop” or  “reference service*” or  “reading promotion” or  “research support” or  “citation guide*” or  “systematic review service*” or “digital scholarship” or “open access” or  “inter-library loan” or ill or “teaching support” or literacy or “institutional repositor*” or "user service*”) and (covid-19 or pandemic*  or coronavirus* or 2019-ncov-2  or cov-19) library science database,  library science database these databases are searched for part of your query. 1278 s4 (“library service*” or “information service*” or “bibliographic instruction*” or “book drop” or  “reference service*” or  “reading promotion” or  “research support” or  “citation guide*” or  “systematic review service*” or “digital scholarship” or “open access” or  “inter-library loan” or ill or “teaching support” or literacy or “institutional repositor*” or "user service*”) and (covid-19 or pandemic*  or coronavirus* or 2019-ncov-2  or cov-19) and pd(20200101-20201231) library science database,  library science database these databases are searched for part of your query. 739 s5 (“library service*” or “information service*” or “bibliographic instruction*” or “book drop” or  “reference service*” or  “reading promotion” or  “research support” or  “citation guide*” or  “systematic review service*” or “digital scholarship” or “open access” or  “inter-library loan” or ill or “teaching support” or literacy or “institutional repositor*” or "user service*”) and (covid-19 or pandemic*  or coronavirus* or 2019-ncov-2  or cov-19) and (stype.exact("scholarly journals" not ("trade journals" or "blogs, podcasts, & websites" or "magazines")) and pd(20200101-20201231)) library science database,  library science database these databases are searched for part of your query. 158   appendix c lista search strategy using building blocks approach set# searched for databases results s1 ( “library service*” or “information service*” or “bibliographic instruction*” or “book drop” or “reference service*” or “reading promotion” or “research support” or “citation guide*” or “systematic review service*” or “digital scholarship” or “open access” or “inter-library loan” or ill or “teaching support” or literacy or “institutional repositor*” or "user service*” ) and ( covid-19 or pandemic* or coronavirus* or 2019-ncov-2 or cov-19 )  expanders  apply related words; apply equivalent subjects search modes  boolean/phrase   297 s2 ( “library service*” or “information service*” or “bibliographic instruction*” or “book drop” or “reference service*” or “reading promotion” or “research support” or “citation guide*” or “systematic review service*” or “digital scholarship” or “open access” or “inter-library loan” or ill or “teaching support” or literacy or “institutional repositor*” or "user service*” ) and ( covid-19 or pandemic* or coronavirus* or 2019-ncov-2 or cov-19  limiters  publication date: 20200101-20201231 expanders  apply related words; apply equivalent subjects search modes  boolean/phrase database 267 s3 ( “library service*” or “information service*” or “bibliographic instruction*” or “book drop” or “reference service*” or “reading promotion” or “research support” or “citation guide*” or “systematic review service*” or “digital scholarship” or “open access” or “inter-library loan” or ill or “teaching support” or literacy or “institutional repositor*” or "user service*” ) and ( covid-19 or pandemic* or coronavirus* or 2019-ncov-2 or cov-19 )  limiters  publication date: 20200101-20201231 expanders  apply related words; apply equivalent subjects narrow by language: english search modes  boolean/phrase   109   appendix d web of science core collections search strategy using building blocks approach set# results save/search history #4 2,028 #2 and #1  refined by: publication years: ( 2020 ) indexes=sci-expanded, ssci, a&hci, cpci-s, cpci-ssh, bkci-s, bkci-ssh, esci, ccr-expanded, ic timespan=all years #3 3,164 #2 and #1  indexes=sci-expanded, ssci, a&hci, cpci-s, cpci-ssh, bkci-s, bkci-ssh, esci, ccr-expanded, ic timespan=all years #2 107,419 ts=(covid-19 or pandemic* or coronavirus* or 2019-ncov-2 or cov-19)  indexes=sci-expanded, ssci, a&hci, cpci-s, cpci-ssh, bkci-s, bkci-ssh, esci, ccr-expanded, ic timespan=all years #1 404,081 ts=("library service*" or "information service*" or "bibliographic instruction*" or "book drop" or "reference service*" or "reading promotion"  or "research support" or "citation guide*" or "systematic review service*" or "digital scholarship" or "open access" or "inter-library loan" or ill or "teaching support" or literacy or "institutional repositor*" or "user service*")  indexes=sci-expanded, ssci, a&hci, cpci-s, cpci-ssh, bkci-s, bkci-ssh, esci, ccr-expanded, ic timespan=all years   appendix e critical appraisal of methodological quality of included studies. 1.      qualitative studies first author year q 1.1 q 1.2 q 1.3 q 1.4 q 1.5 comments howes 2021 cannot answer the general questions (s.1 and s.2), hence cannot be assessed. mehta 2020 yes yes yes yes yes   see 2020 yes yes yes yes yes   weeks 2020 yes yes yes yes yes   yu 2020 yes yes yes yes yes   chakraborty 2020 yes yes yes yes yes   dadhe 2020 yes yes yes yes yes   ma 2020 no no yes yes yes no sufficient description of the methodological approach. carbery 2020 yes yes yes yes yes   walsh 2020 no no yes yes yes no sufficient description of the methodological approach. erich 2020 yes yes yes yes yes   pokorna 2020 yes yes yes yes yes   winata 2020 yes yes yes yes yes   mbambo-thata 2021 yes yes yes yes yes   q 1.1. is the qualitative approach appropriate to answer the research question? q 1.2. are the qualitative data collection methods adequate to address the research question? q 1.3. are the findings adequately derived from the data? q 1.4. is the interpretation of results sufficiently substantiated by data? q 1.5. is there coherence between qualitative data sources, collection, analysis, and interpretation?   4. quantitative descriptive studies first author year q 4.1 q 4.2 q 4.3 q 4.4 q 4.5 comments tolppanen 2021 no can’t tell yes can’t tell yes no mention of sampling strategy, nor the total population of the study. hence, we cannot tell if the sample is representative or not. neog 2020 no can’t tell yes can’t tell yes no mention of sampling strategy, nor the total population of the study. hence, we cannot tell if the sample is representative or not. guo 2020 yes yes yes na yes this study did not use human participants, but used content analysis. hence, risk of non-response bias is not applicable. cox 2020 yes yes yes no yes with response rate of 31%, the risk of non-response bias is low. saavedra-alamillas 2020 na can’t tell yes na yes this study did not use human participants, but used content analysis. hence, risk of non-response bias is not applicable. ishtiaq 2020 yes yes yes yes yes   kasa 2020 yes yes yes na yes this study did not use human participants, but used content analysis. hence, risk of non-response bias is not applicable. tsekea 2020 yes can’t tell yes yes yes this study did not include the total number of the unit of analysis. hence, we cannot tell if sample was representative or not. q 4.1. is the sampling strategy relevant to address the research question? q 4.2. is the sample representative of the target population? q 4.3. are the measurements appropriate? q 4.4. is the risk of nonresponse bias low? q 4.5. is the statistical analysis appropriate to answer the research question?     5. mixed methods studies first author year q 5.1 q 5.2 q 5.3 q 5.4 q 5.5 comments kaur 2020 yes no no no yes the outputs of the qualitative and quantitative components of this study were interpreted separately. there was no integration of data. data from the qualitative component were insufficiently interpreted and discussed. q 5.1. is there an adequate rationale for using a mixed methods design to address the research question? q 5.2. are the different components of the study effectively integrated to answer the research question? q 5.3. are the outputs of the integration of qualitative and quantitative components adequately interpreted? q 5.4. are divergences and inconsistencies between quantitative and qualitative results adequately addressed? q 5.5. do the different components of the study adhere to the quality criteria of each tradition of the methods involved?     evidence based library and information practice evidence based library and information practice 2011, 6.4 130 evidence based library and information practice evidence summary academic libraries should consider a strategic approach to promotion and marketing of e-books a review of: vasileiou, m. & rowley, j. (2011). marketing and promotion of e-books in academic libraries. journal of documentation, 67(4), 624-643. doi: 10.1108/00220411111145025 reviewed by: nazi torabi reference and instructional librarian university of western ontario london, ontario, canada email: ntorabi@uwo.ca received: 31 aug. 2011 accepted: 23 oct. 2011 2011 torabi. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objectives – the main objectives of this study were the following: 1) to determine the existence of a formal promotion or marketing strategic plan for e-books in academic libraries. 2) to identify the marketing and promotion tools being used for e-book marketing. 3) to understand the role of faculty in promoting e-books. 4) to explore the issues and challenges that the libraries are facing in promoting e-books. 5) to investigate the future plans for the promotion of e-books. design – qualitative study involving individual interviews followed by thematic analysis. setting – the research setting was seven academic libraries with various sizes of e-book collections (~9,000 to ~500,000) and ranging from research leading to teaching focused universities. subjects – the interviews were conducted with 25 academic librarians, including 19 subject librarians, 4 e-resources librarians, 1 collection development manager, and 1 acquisition and metadata officer. methods – using a multiple case approach, the researchers interviewed 25 librarians from 7 academic libraries. the interviews were mailto:ntorabi@uwo.ca� evidence based library and information practice 2011, 6.4 131 digitally recorded. the interview transcripts were coded and analyzed using thematic analysis. the researchers recorded information about the type and size of the university, number of e-books, and the role of interviewees, and then conducted a semistructured interview. the following questions were asked: 1) “do you have a promotion/marketing strategy for e-books (or e-resources)? if not, is there a need for one? 2) what are the marketing and promotion tools you use for e-books? 3) what are the issues and challenges in promoting e-books?” (p. 629) main results – while none of the cases had a formalized marketing strategy, most participants (19 out of 25) acknowledged the importance of developing one. those who did not find it necessary argued that their users had good knowledge of library e-resources, thus, marketing was unnecessary. although there was a lack of a marketing strategic plan across all the cases, some of the participants were practicing e-book marketing and promotion to some extent. the interviewees mentioned a total of 20 different promotional tools. the most frequently used promotional tools were library websites, information literacy sessions, opacs, and e-mails. participants also noted marketing using online help tutorials and various events. the least frequently mentioned promotional tools were announcement boards, a library representative, blogs, bulletins, the display screen, and a marketing campaign. the marketing campaign was only mentioned by librarians from one library. according to some respondents, there are faculty and course instructors who contribute in promotion of e-books dependently or independently from the library. some recommend e-books as course reading materials, some add links to e-books on the virtual learning environment (vle) reading lists, and some even go beyond that and provide instructions on how to use e-books. not all librarians were aware of the use of ebooks by faculty. the authors also investigated the future plans anticipated by the participants. there was only one library that has appointed a marketing officer and intends to develop a marketing strategy. all other cases plan to improve on current marketing practices or expressed innovative approaches in marketing by “development of an information literacy project, use of online chat, promotion of ebooks in course committee meetings, provision of a paper guide for e-books,” cd case display of available e-books, and other strategies (p. 633). marketing a new service comes with its own issues and challenges. the most frequently mentioned issues and challenges that librarians were facing were time constraints, raising users’ expectations, and e-book availability in the library. also, accessibility was of concern to some participants. conclusion – this study underlines the importance of marketing a library’s new resources, particularly e-books, and it confirms the lack of a strategic approach to marketing in academic libraries. some participants recognize the importance of a strategic plan, and practice e-book marketing using a variety of tools. however, most tools are not marketing specific, but are either communication tools or part of service delivery. the authors stated that most interviewees had a reflective approach to marketing, since they were trying to improve the current marketing practices by innovative means or by adapting successful practices from other colleagues. the authors recommend that establishing a well thought-out strategic approach would help overcome several issues and challenges raised by participants. it would help manage the tension between service delivery and promotion. also, it would provide clear guidelines for developing a budget allocation plan for e-book purchasing. in addition, internal marketing would help establish a culture of user-centred service in the library, and highlights the importance of new services among library staff. evidence based library and information practice 2011, 6.4 132 libraries are always dependent on publishers’ decisions concerning the availability of and licensing agreements for e-books. they are considered as “intermediaries in the information or document supply chain” (p. 638). the authors suggest future studies to understand the role of libraries in managing links between licensing issues and service delivery. one interesting observation in this study was the absence of web 2.0 tools as promotional tools. it seems that libraries do not fully take advantage of new technology for marketing. another interesting finding was the inaccurate or indirect reference to the notion of word-ofmouth by respondents. in this study, word-ofmouth was mentioned as a marketing tool by only a few participants. no one perceived the role of academic faculty in promoting e-books by word-of-mouth. while word-of-mouth is considered to be an invaluable marketing strategy because it is a tailored recommendation by consumers to consumers (alire, 2007), it was neglected by most respondents in this study. overall, the lack of knowledge of marketing strategies and users’ information seeking behaviour among academic librarians is evident. commentary this article highlights the importance of ebooks in the future of academic libraries. it points out that both marketing research and marketing practice are neglected in academic libraries. overall, this article contributes to our knowledge of e-book promotion and marketing in academic libraries and encourages more research on this very important topic. while the research design and a multiple case study approach were very appropriate for this type of study, the authors should have been more transparent about the methodology. the selection criteria for the cases are not reported. the authors should have elaborated more on their framework for conducting the thematic analysis and coding. the coding reliability is not defined, since they did not mention how many people were responsible for coding. table 1 is nicely presented. however, the authors do not describe the role of each interviewee, since the job title does not clearly specify what a person might do, and it varies from one university library to another. it would have been better if they had provided more information about each participant’s responsibilities. while the authors reported the use of thematic analysis in this study, it seems that they applied a mixed method of content analysis (ca) and thematic analysis (ta). “in thematic analysis the significance of a theme is not determined by its frequency but by its ‘substantive significance.’ patton refers to ‘substantive significance’ as the consistency of themes across and within study participants” (floersch, 2010, p. 408). also, it is important to identify to what extent the findings are consistent with the existing literature. as the result, ta pays greater attention to the qualitative aspects rather than generating numerical description (marks, 2004, p. 56). in this article, the authors emphasized numerical description and the findings that could be discovered by conducting a survey. for future studies, the authors recommend investigating the marketing and consumer orientations of library staff and understanding the issues associated with managing supply chain issues and the user side of service delivery (pp. 637-638). this work uncovers the dichotomy between strategy and practice among the participants. it refers to e-book marketing as a reflective approach, laissez faire approach, and short-term plan approach for an increasingly important information resource. it will be valuable to understand the reason(s) for the current practice in order to improve it. another important study in this area can be related to establishing best practices in marketing by assessing the value and effectiveness of different promotional tools in academic libraries. evidence based library and information practice 2011, 6.4 133 references alire, c. a. (2007). word-of-mouth marketing: abandoning the academic library ivory tower. new library world, 108(11/12), 545551. doi: 10.1108/03074800710838272 floersch, j., longhofer, j. l., kranke, d., & townsend, l. (2010). integrating thematic, grounded theory and narrative analysis: a case study of adolescent psychotropic treatment. qualitative social work, 9(3), 407-425. doi: 10.1177/1473325010362330 marks, d., & yardley, l. (2004). research methods for clinical and health psychology. thousand oaks, ca: sage publications. / evidence based library and information practice evidence summary   obtaining journal titles via big deals most cost effective compared to individual subscriptions, pay-per-view, and interlibrary loan   a review of: lemley, t., & li, j. (2015). "big deal” journal subscription packages: are they worth the cost? journal of electronic resources in medical libraries, 12(1), 1-10. http://dx.doi.org/10.1080/15424065.2015.1001959     reviewed by: kathleen reed assessment & data librarian vancouver island university nanaimo, british columbia, canada email: kathleen.reed@viu.ca   received: 30 nov. 2015 accepted: 10 feb. 2016      2016 reed. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine if “big deal” journal subscription packages are a cost-effective way to provide electronic journal access to academic library users versus individual subscriptions, pay-per-view, and interlibrary loans (ill).   design – cost-per-article-use analysis.   setting – public research university in the united states of america.   subjects – cost-per-use data from 1) journals in seven big deal packages, 2) individually subscribed journals, 3) pay-per-view from publishers’ websites, and 4) interlibrary loans.   methods – the authors determined cost-per-use for big deal titles by utilizing counter jr1 metric successful full-text article request (sftar) reports. individual journal subscription cost-per-use data were obtained from individual publishers or platforms.  pay-per-view cost was determined by recording the price listed on publishers’ websites. ill cost-per-use was established by reviewing cost-per-article obtained from libraries outside of reciprocal borrowing agreement networks. with the exception of pay-per-view numbers, title cost-per-use was averaged over a three-year period from 2010 through 2012.   main results – cost-per-article use for journals from big deals varied from $2.11 to $9.42. for individually subscribed journals, the average cost-per-article ranged from $0.25 to $84.00. pay-per-view charges ranged from $15.00 to $80.00, with an average cost of $37.72.   conclusion – the authors conclude that big deals are cost effective, but that they consume such a large amount of funds that they limit the purchase of other resources. the authors go on to outline the options for libraries thinking about big deal packages. first, libraries should keep big deal packages in place if the average cost-per-article is less than individual subscriptions. second, libraries could subscribe only to the most-used journals in big deals, cancel the packages, and rely on ill and pay-per-view access. third, consortia could be joined to favourably negotiate big deal package prices. fourth, big deals could be dropped completely. fifth, individual libraries armed with jr1 reports can negotiate with publishers for better deals.   commentary   as a general climate of budget pressures and vendor price increases continues, librarians require tools for evaluating the “big deal,” “an online aggregation of journals that publishers offer as a one-price, one size fits all package” (frazier, 2001). while this article reminds librarians to consider multiple ways of obtaining articles and the associated costs, it has significant flaws and is of limited value.   the article could be strengthened by comparing the value of the same set of journals using various purchasing methods or by shying away from an analysis of all big deals in favour of identifying individual big deals that are comparable with bundles of individual journal titles. one way to compare apples to apples is by analyzing journals by discipline as this factor significantly influences price. the authors did not analyze their data in this fashion; they compared cost-per-use of articles between a biomedical library and a general campus library, which gives a skewed picture of the cost of resources.   the details in this article are very general. for example, the cost-per-article of one-off subscriptions identified by the authors from $0.25 to $84.00 per title. this range is far too varied to help librarians make subscription decisions.   throughout the article critical details are missing. the cost-per-use of ill is never stated, the authors do not specify if they are using counter reports, and the authors mention that they exclude data from articles obtained “for free” via reciprocal borrowing agreements. while there may be no fee for articles obtained via reciprocal agreements, there is a cost associated with staff processing time. if a library chooses to cut titles based on the assumption that ill is cheaper, it should factor in how much the increased staff time will cost.   the authors could improve this paper by going beyond a simple cost-per-use analysis to answer the question of whether or not big deals are worth it. while they do mention in the very last paragraph that “a key issue in discussing any big deal journal package must include library user satisfaction,” (p. 9) this point comes far too late and is not unpacked. the authors are silent on other factors librarians must consider when reviewing big deals, such as curriculum requirements and usability.   while the paper does have a good summary of the pros and cons of big deals, overall it is of limited value. librarians may want to consult this article for ideas but compare groups of similar resources for a more accurate analysis.   reference   frazier, k. (2001). the librarians’ dilemma: contemplating the costs of the “big deal.” d-lib magazine, 7(3). retrieved from http://www.dlib.org/dlib/march01/frazier/03frazier.html   article   taking a page from retail: secret shopping for academic libraries   kathryn crowe associate dean for public services walter clinton jackson library university of north carolina at greensboro greensboro, nc, united states email: kmcrowe@uncg.edu   agnes kathy bradshaw assistant university librarian for organizational development virginia commonwealth university libraries virginia commonwealth university richmond, virginia, united states email: akbradshaw@vcue.edu   received: 4 june 2015     accepted: 27 oct. 2015         2016 crowe and bradshaw. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – the university libraries at the university of north carolina at greensboro (uncg) sought to gain feedback on the customer service experience beyond satisfaction surveys. after reviewing a variety of methods, it was determined to conduct a mystery or secret shopper exercise, a standard practice in the retail and hospitality world.   methods – two mystery shopper assessments were conducted in 2010 and 2012. students were recruited from a hospitality management class to serve as the secret shoppers. “shoppers” completed a rating sheet for each encounter based on customer service values established by the libraries. data was analyzed and presented to staff.   results initial findings were generally quite positive but indicated that we could improve “going the extra mile” and “confirming satisfaction.” as a result, we developed training sessions for public services staff which were delivered during summer 2011. a libguide that included training videos was created for public services student employees who were required to view the videos and provide comments. in addition, we developed more specific public service standards for procedures such as answering the telephone, confirming satisfaction, and referring patrons to other offices. the secret shopper assessment was administered again in spring 2012 to see if scores improved. the results in the second study indicated improvement.   conclusions the mystery shopper exercises provided the uncg university libraries with the opportunity to examine our services and customer service goals more closely. conducting the mystery shopper study identified several areas to address. we realized we needed more clearly defined standards for staff to follow. we saw that we needed to discuss what “going the extra mile” means to us as an organization. we also needed to develop a scalable training method for student employees.      introduction   academic libraries are increasingly emphasizing the entire user experience for their customers and seek to provide not only outstanding collections but also services and programs that contribute to student success and faculty research as well as facilities that provide learning spaces. much of the user experience conversation focuses on efficient online accessibility and discovery. recently, however, bell called for academic libraries to “commit to a total, organization-wide effort to design and implement a systemic ux.” bell also advocated for “shifting the academic library experience from usability to totality” (bell, 2014, p. 370). many libraries are hiring librarians with job titles such as “user experience librarian” and engage in a wide variety of assessments to gain knowledge about what students and faculty seek in library services. much of this research employs ethnographic studies originating with the excellent university of rochester work where they tracked students’ research patterns using a variety of methods such as photo surveys and mapping diaries (foster & gibbons, 2007; foster, 2013). in 2011, the association of research libraries (arl) published a spec kit, library user experience that outlined numerous types of user assessments employed at arl libraries including surveys, facilities studies, focus groups, and usability studies (fox & doshi, 2011).   one aspect of the user experience that remains crucial is excellent customer service both face-to-face and virtual. although libraries seek to make the online and in-house user experience as self-service as possible, customers still require both directional and in-depth assistance to find the information and services they need. furthermore, as libraries seek to become information hubs and learning centers it is necessary that students have a good customer experience so that they view the library as a comfortable and welcoming place. fair or not, we are aware that users compare the customer service we provide in the library to that offered in retail shopping areas such as bricks and mortar book stores and by other retail services such as the apple store. in a 2011 study, bell surveyed college students to compare their experiences in libraries to retail using an instrument from the study of great retail shopping experiences in north america. fortunately, libraries compared well! one factor in the survey includes “engagement” characterized by politeness, caring and listening. bell recommended that academic librarians focus their efforts on less tangible “soft skills” such as eye contact, patience, and making customers feel important (bell, 2011).      with these customer service issues in mind, the university libraries at the university of north carolina at greensboro sought to assess the service experiences of students for both in-house and virtual services. the university of north carolina at greensboro, part of the 17-campus university of north carolina system, is a publicly-supported university with a high research activity carnegie classification. in 2015 the total enrollment was 19,398 with a faculty of approximately 1,000. the university libraries include the walter clinton jackson main library and the harold schiffman music library. at the time of the initial study, jackson library had two public service points; reference and access services (checkout) on the first floor. later, the special collections and university archives (scua) department added a service point on the second floor and was included in the second study. the schiffman music library has one combined service point. these services desks are staffed by professional librarians, paraprofessional staff, and student employees. the two service points (the reference desk in jackson and the front desk at schiffman) both employ graduate students from the libraries and information studies program as interns.   previous assessments conducted by the university libraries indicated positive results for services. in 2008 the libraries conducted libqual+® and the overall perceived mean for “affect of service” was 7.5 on the nine-point scale. every three years the unc system conducts surveys of all sophomores and seniors which include questions about library services. in the 2010 senior survey the libraries scored 3.5 on a four-point scale for “staff responsiveness” and 3.6 for “library services overall”. longitudinally, we showed improvement in these categories since 1998 when we scored 3.2 on both these questions. in the 2010 sophomore survey the libraries received 4.1 out of 5 on “helpfulness of staff.” because this survey was newly revised that year we don’t have longitudinal data for it (uncg university libraries, 2016).   although the libraries performed well on these assessments they were satisfaction surveys rather than in-depth studies focused on the user experience. and, while most qualitative comments on the 2008 libqual+® survey were very positive, some indicated that users had less than satisfactory interactions at service desks:                 “i sometimes find the student staff to be really annoyed at having to help me, even just checking out books.”   “i cannot send my students to the library with confidence that they will be treated with the same respect.”   both jackson and schiffman offer computers with a wide variety of software, group and quiet study space and technology checkout as well as traditional print and av materials. chat, email, and texting are offered in addition to in-house service. jackson library has a 24/5 space that is very popular. together the libraries have over 1 million visitors each year. like many academic libraries, we are realigning service staff to rely more on paraprofessionals for reference service so that librarians may focus on information literacy and specialized liaison services. often these staff members are not part of the research, outreach and instruction department (roi, formerly called the reference & instructional services department) which can present training challenges. the reliance on student employees with a high turnover rate can also make it difficult to provide consistent service. after administering the association of research libraries’ libqual+® survey in 2008 the libraries sought to enhance the quality of the customer experience at service desks and via phone and chat. to begin the process, the associate dean for public services charged a task force in 2009 to develop customer service values to serve as a guide for both external and internal service. these values were vetted among the public service departments and posted on the libraries’ web page along with the libraries’ mission statement, to indicate to both patrons and staff that we are committed to quality service. (uncg university libraries, 2015a). the task force recommended a training program for customer service that “should be shaped through ongoing assessment.”  literature review   mystery shopping is a term that is familiar in industries that are heavily focused on customer service such as financial services, retail, restaurants, and hospitality. in 2010, the mystery shopping business was “estimated to be a $1.5 billion industry, up from roughly $600 million in 2004” (andruss, 2010). many of the industries that use mystery shopping use professional services organizations that hire and train the shoppers. there have also been attempts to utilize the mystery shopping concept in other non-customer-service areas, such as patient satisfaction with health care services. and, while much of the literature once focused on mystery shopping done in person, work is now being conducted to evaluate the quality of services delivered in virtual environments. according to the 14th annual mystery shopping study conducted by the e-tailing group… “the study confirms that merchants are refining online tactics to find, inform, personalize and connect with improved speed and efficiency, while diligently developing social and mobile initiatives” (tierney, 2012). in areas that are profit-driven, mystery shopping has been used to measure up-selling offers (peters, 2011) and identify employees with promotional potential (cocheo, 2011).   an early use of mystery shopping in a library took place in 1996 in a public library in modesto, california. mystery shoppers were used to assess the library’s customer service, as part of the county’s quality service initiative (czopek, 1998). subsequent use of mystery shopping in libraries has been to measure the quality of the customer service experience; there is not, however, a universal definition of quality customer service. in addition, there is not a universal way to assess quality of customer service. is it the amount of time a person has to wait to speak with someone at the reference desk? is it providing free coffee to students at exam time? is it offering resume writing and computer workshops at public libraries in response to the needs of the local community (roy, bolfing & brzozowski, 2010)? another factor that must be considered is that, in many instances, the library may be considered a “self-service” organization; patrons can come into the library or visit the website, and in many instances find what they are looking for without requesting assistance from library personnel. even those that do not find what they are seeking still may not approach a service point (in-house or virtual) for assistance.    the literature also shows that the use of mystery shoppers is as varied as the desired outcomes. for some libraries, when measuring customer service quality, the focus could be on the accuracy of answers received at the reference desk (e.g. kocevar-weidinger, benjes-small & kinman, 2010; tesdell, 2000). there are studies that use mystery shopping to judge the accuracy of answers received during a reference interview as well as an assessment of the appropriateness and accessibility of physical space and signage (tesdell, 2000). another use of mystery shopping is the assessment and development of customer service training needs. the assessment for training needs is not only confined to the front-line public services staff — reference and access services/circulation department staffs — but also internal departments as well, such as the human resources department. in one library, they worked with the state’s small business development center to tailor the mystery shopping process for the needs of their library. various service points were “shopped” and they made sure to include a variety of customers so that they could get a better idea of the needs of diverse populations such as patrons whose first language was not english, parents with children, etc. their shoppers used repeat visits (5 times) in order to relieve employee concerns about the impact of workload variability on the customer service encounter and consistency of responses (backs & kinder, 2007). at florida international university, mystery shopping was used on student employees initially as a way to assess how the service being provided “felt” to the patrons, to determine if additional training would be needed and to determine which areas needed improvement, based on patron feedback. additional shopping trials were used after an organizational change resulted in combined service points. the later mystery shopping assessments focused not only on accuracy of the responses but also on service provider behaviour. (hammill & fojo, 2013)   support and agreement by stakeholders is always crucial in implementing a mystery shopper initiative in a library. for public libraries, authorization by the library board or employee union may be required prior to implementing such a program. for academic libraries, the permission of the university’s institutional review board (irb) will probably be required (benjes-small & kocevar-weidinger, 2011). benjes-small and kocevar-weidinger also discuss the importance of using written guidelines of appropriate behaviour to which all staff are exposed as a way to measure whether or not customer services standards are being met. both authors used students as mystery shoppers. at longwood university, the results of the survey were used as a part of the employees’ performance review, which resulted in revised job descriptions and using the mystery shopper assessment to measure progress (benjes-small & kocevar-weidinger, 2011).   in some instances, the results of mystery shopper evaluations have been received as unwelcome surprises to the library staff. there are also instances in which library staff resist efforts to measure quality library customer service output in the same way as customer service is measured in a retail operation (e.g. deane, 2003; gavillet, 2011; hernon, nitecki & altman, 1999). most of the literature shows that mystery shopping efforts have been focused only on the delivery of customer service to external users and not internal customer service providers, such as cataloguing, acquisitions, or administration.   the majority of efforts to use mystery shopping in libraries occur in the public library sector. depending on the environment (unionized or civil service), there may be barriers to using mystery shopping as a measurement of job performance or as an assessment of promotional potential. academic libraries and public libraries do have many commonalities, but also have differences in their missions as well as a different patron base. one of the commonalities of both academic and public libraries is that, unlike retail establishments, libraries do not have a vested interest in trying to get a patron to “buy” additional products and services; however, library employees should have a vested interest in ensuring that the patron is aware of the products and services that could be of assistance, either at the time of the visit, or during a future one. both academic and public libraries should seek to create an environment where customers (or patrons) are comfortable seeking assistance within any service point. the association of college and research libraries (acrl) 2012 “top ten trends for academic libraries” included “staffing” and “user behaviors and expectations” as important issues (acrl, 2012). library users often base their expectations of customer service on that which is provided in non-library environments. as stated by connaway, dickey, and radford, “librarians are finding that they must compete with other, more convenient, familiar, and easy-to-use information sources. the user once built workflows around the library systems and services, but now increasingly the library must build its services around user workflows” (connaway et al., 2011). failure to assess customer service delivery and the quality of that delivery would mean we are ignoring the needs of our users. users who feel their needs are being ignored will turn to other, more welcoming, resources regardless if they are the best ones for their need.    method and procedures   after reviewing the literature, the libraries determined that the mystery shopper protocol was the best method to assess our service interactions and accomplish our goal of determining if our customer service was indeed meeting the established customer service values. the study completed at radford and longwood universities in 2010 was an excellent model and we adapted their protocol for our project (benjes-small & kocevar-weidinger, 2011. we conducted the first mystery shopper assessment in fall 2010 and included desk and phone service for all service points — reference and checkout in jackson and the service desk in schiffman — and chat service for reference. the research team included the associate dean for public services, the human resources librarian, and the assessment analyst. because secret shopping is a standard in service industries we collaborated with uncg’s hospitality and tourism management department to recruit students as shoppers. a professor agreed to award extra credit to students who participated. we also gave them a $10 credit for the campus food service. we developed a rating sheet (see appendix 1) for the students to use based on the customer service values mentioned above. although we certainly care about accuracy, the emphasis for this assessment was on the customer service experience. we included four behaviours: greeting, follow-up, confirmation of satisfaction and referral, with three levels of rating: 1(poor), 2(satisfactory) and 3(very good). brief descriptions of each behaviour were included on the rating sheet along with criteria for each level and type of service. for example, for greeting at a service desk, the following guidance was provided:   very good – employee made eye contact, acknowledged me and greeted me in a positive manner satisfactory – employee greeted me but not with great enthusiasm poor – employee was distracted and did not acknowledge me   we also had three yes/no questions:   employee treated me with respect employee avoided jargon or technical language employee went the extra mile   because the yes/no questions were quite subjective, we discussed them extensively in the training and provided guidelines for what should be expected from the libraries service staff. we also conducted role-playing and asked the shoppers to evaluate the mock transaction in order to prepare them better for the actual experience. space for additional comments was also included and comments were encouraged   we sought to make the assessment as “real life” and anonymous as possible. we informed staff in the departments to be studied that the exercise would take place sometime during the semester. we did not, however, give exact dates. we met with each department to apprise them of the protocol and assure them it was not part of their performance review but rather an overall assessment of our service so that we could address any issues identified. to that end we did not include any date/time stamps in the results. the questions developed for the survey were constructed around the feedback received from the initial libqual+® results that indicated some patrons did not feel they were treated respectfully by staff. we collaborated with the heads of the roi, access services and schiffman music library to obtain frequently asked questions considered “typical.” questions for the checkout desk emphasized service-related questions that could usually be answered with basic responses, such as: “how many books can i check out at one time?” or “where can i print something in color?” while certain categories of service related questions may seem easy to answer we wanted to ensure that shoppers were being asked the right clarifying questions by employees, not to see if the correct answer was provided since that was not the primary focus of this study. for example, it would be simple to tell a questioner that the library is open 24 hours, 5 days a week but, in reality, that schedule is only applicable to people with a uncg id. for other patrons, the library closes at 12:00 am.    for questions to be asked at the reference desk, the head of the roi provided a list of questions relating to common assignments and citation issues. since often times the reference desk is staffed by paraprofessional staff, we did not want to present a difficult question that would require obtaining additional assistance, or place the questioner in a position which would require him/her to handle questions they could not answer. examples of questions asked of reference staff included: “can you help me find articles on identity theft?” and “i am a uncg graduate, how do i access the databases from home?” or “how to do cite this in apa style?” (see appendix 2 for sample questions).   we required the shoppers to attend a 90-minute training session. during the training, we provided an explanation of the importance of excellent customer service to the libraries as well as the customer service values (and behavioural examples of them) that staff were expected to demonstrate, and we provided instruction on what to look for when observing staff behaviours. each shopper was assigned a question for each service point (reference desk, access services desk and the schiffman music library) and type of service (in-person, telephone and chat) with the exception of the schiffman music library and access services; chat service was not offered in schiffman at the time of the initial survey and is still not available in the access services department. we requested that shoppers vary their times of contact to make their presence as anonymous and unobtrusive as possible. we also wanted to vary the time of contact to avoid staff members feeling as if they were being “targeted” if the questions were only asked during specific time periods.    one question was placed on each rating sheet used by the shoppers. six students completed the exercise with each shopper asking a question for each service. they entered their scores into a qualtrics® form created by the team. qualtrics is an online survey platform licensed on many campuses. they also submitted paper sheets as a backup.    results   for the most part, the libraries received very positive results. scores were particularly high for “greeting” and “referral.” “follow-up” was rated slightly less well and “confirming satisfaction” the lowest. for the yes/no questions, shoppers rated staff well for “treated with respect” and “avoided jargon.” there were, however, issues with “going the extra mile.” below are overall averages for all service points and types of service (figures 1 and 2).   we also compiled results for each department broken down by type of service (figure 3).   follow up the assessment analyst compiled the results and developed graphs for each question that indicated scores for desk, phone, and chat. the results for all services were shared with the entire staff through meetings and email. the associate dean shared results for individual departments with the appropriate department head for discussion among their staff. after examining the results, the team had the following recommendations:   develop “standards of service” that reflect the customer service values. although we had the values we really had no specific standards or guidelines for interacting with staff. for example, we did not have guidelines on how to greet patrons, do a referral, transfer a phone call to another department, or best practices for chat service. established standards are useful to train new staff, both full-time and student employees, so that they know what is expected of them. as our public service desks are staffed by a variety of employees, we determined it was important to establish service standards that would be uniform across all service points to ensure a more consistent experience for users. these standards are based on both industry best practices and library staff input. they include not only procedural guidelines but also advice on how to “go the extra mile” which is subjective in nature and can be difficult to define. advice here includes “walk a patron to a destination rather pointing them, including going to the stacks”, “feel empowered to be flexible in order to provide service”, and “be flexible about staying after hours to provide a consultation for a student who works full time”. these standards are posted on the customer service skills libguide under the “customer service documents” tab (uncg university libraries, 2015b). develop customer service training for full-time library staff that focused on “going the extra mile.” the impetus for this was the feedback from users during the libqual +® results. while the phrase “going the extra mile” is subjective and varies according to the individual being asked, we wanted to convey to staff members that being polite and helpful was not enough. we felt it was important for all staff members to ask enough questions and offer a level of assistance to ensure that all user needs were being met. because that question received lower scores we decided that we needed the opportunity to discuss what we meant by going the extra mile and how we could achieve it. develop online training for student employees. because our students work many shifts in two buildings it is impossible to get them all together for training.  conduct the assessment again after training to see if there was improvement.     figure 1 results for the “four behaviours” questions.   figure 2 results for the yes/no questions.   figure 3 results by type of service.     staff training    training was provided for all library staff members including those that did not have contact with the public. we wanted to ensure that the customer service values we wanted to impart within the library were given to staff members that provided internal service, not just given to those who work at public services desks.    we conducted six sessions (4 hours each, with breaks) and extended an offer to attend training to the managers of the computer labs, which are housed in the library but are not under the organizational control of the library. because the lab is located in the library, students often make an incorrect connection between the computing lab staff and the library staff. the managers of the computer labs were unable to attend, however. sessions were staggered so that those staff members that work during evening hours were able to attend. all employees of the library, with the exception of the dean and the assistant deans, were required to attend the sessions. approximately 90% of the staff, including library faculty completed the training.    the training design was done by the human resources librarian. she also conducted the training sessions, and developed a workbook to use in the training sessions. the program design focused on “going the extra mile” which the team felt would allow the staff not to feel the training was remedial in nature or was being used as a punitive measure. the emphasis in the program design was to improve customer service and eliminate the feeling by patrons that they were not being treated respectfully. we were careful to point out that the libqual+® scores reflected that good customer service was being provided. we let the staff know that the libqual+® qualitative data included comments which said some respondents did not feel the customer service being provided went far enough; it did not “go the extra mile.”    although not planned, the training sessions gave some staff members new information about some of the services offered within the library; staff members who are considered to be internal service providers found the information to be extremely beneficial. the libraries’ customer service values were updated based on staff suggestions.     student training as mentioned above we determined that online training was best for our student employees. the libraries place great emphasis on providing our students with the opportunity to gain skills they can use in the future regardless of what profession they chose. the distance education librarian and a library and information studies (lis) practicum student spent a semester developing customer service videos around the standards. these include basic skills such as approachability, the reference interview, telephone etiquette, referrals and handling a line of customers. additional videos provide tips for dealing with angry customers. we used students in the videos and made them upbeat and humorous so that they would appeal to our employees. libraries’ documents such as the customer service values and standards are included as well. the videos and documents were organized into a libguide for easy access and editing (uncg university libraries, 2015b). once the libguide was completed, student supervisors asked to include videos on general basic success skills such as attitude, attire, and professional image. for these segments we pulled videos from our films on demand subscription. student supervisors were asked to require employees to view the videos and make comments to indicate they had completed them. some comments from students include:   “these skills seem like common sense, but it's amazing how many people you see that don't follow it. you should send this video to the workers in subway.” “i easily get flustered when a person is frustrated at me, however this video taught me how to properly handle the situation and remain calm and respectful” i’ve never thought to look for people who need help because i always assumed they would ask, now i know.”   second study   in the second mystery shopping assessment, staff members were again told that mystery shopping would happen sometime during the spring semester, but were not given a specific timeframe. during the second study, we reached out again to the department of hospitality and tourism management for students to be mystery shoppers and recruited nine students. we reviewed the questions and made some minor changes to them. because our special collections and archives (scua) had added a formal service point it was included in the assessment and questions for that area were added. for this study an lis graduate student assisted us. she helped with the training sessions, prepared the question sheets, and entered data into qualtrics.    the same training was provided for the second group of secret shoppers that was provided for the first group of shoppers. as with the first group of student shoppers, we explained the importance that the library placed on customer service and that we were assessing the customer service experience rather than accuracy of the answers. we shared the newly developed standards of service as well as the customer service values.    results from the 2012 assessment indicate that improvement occurred for all behaviours and questions from the 2010 results (figures 4 and 5).     we were particularly glad to see that the areas with the lowest scores in 2010, “follow up” (increase from 2.24 to 2.73), “confirmed satisfaction” (increase from 1.68 to 2.44 out of 3) and “went the extra mile,” had the largest margin of improvement. in 2010 only 36% of respondents felt that their service went the extra mile; in 2012 that rose to 59%.     . figure 4 results for ‘four behaviour’ questions, 2012.     figure 5 results for yes/no questions, 2012.     we shared the overall results again with all libraries’ staff and posted comparison graphs on our assessment libguide.   similar graphs for each department were also developed and shared with the department heads. the associate dean discussed results in a public services department heads meeting and individually with department heads. she also visited department meetings to discuss the results with staff and gain their input. we also shared results with student employees during the fall 2012 student orientation to show returning students the improvement in their performance and to let new students know that the online training is very important information.    discussion   the libraries conducted libqual+® again in fall 2012 with an increase in the “affect of service” score from 7.5 in 2008 to 7.92. these results, along with changes between the 2010 and 2012 mystery shopper results, indicate substantial improvements in service quality and satisfaction for the libraries. developing standards and providing training reinforced the importance of customer service and the role that all staff members play so that users have a positive experience in the library. staff comments received after the training indicate that the training was helpful and resulted in staff members viewing customer service and their own role as service providers in a different way; a role which is key to having a positive experience in the library. the libraries continue to emphasize the importance of customer service. all new staff receive the customer service values and standards and are strongly encouraged to attend appropriate campus workshops conducted by the campus human resources department to enhance their customer service skills. all new student employees are required to complete the videos on the customer service libguide.     we also continue to examine our services to ensure we are meeting the needs of our patrons. because we are likely to continue staffing with paraprofessionals, future customer service training should include not only going the extra mile, but also providing the skills and knowledge to answer questions accurately. while providing helpful, respectful, and courteous service is a requirement, we recognize that our training needs will shift also to enhancing skill development. examples would include conducting reference interviews and ensuring competence with the wide variety of resources for those staffing the service desks. training will also need to take into account the changing demographics of our customers. for example, we have an increasing number of international students, as well as larger numbers of what would be considered to be “adult students.” as our requests for virtual reference assistance increase, we anticipate that chat inquiries will also become more complex. as mentioned above, our services must respond to changes in academic libraries and higher education and we need to ensure that assessments correspond accordingly.   conclusion   the mystery shopper exercises provided the uncg university libraries with the opportunity to examine our services and customer service goals more closely. the changing nature of our services with moving toward using more paraprofessional staff and the impact of technology on services provided some of the impetus for doing the study. we also wanted to gather additional evidence on issues identified in the 2008 libqual+ ®survey. and finally, we sought more in-depth assessment of the user experience than that provided by satisfaction measures.    conducting the mystery shopper study identified several areas to address. we realized we needed more clearly defined standards for staff to follow. we saw that we needed to discuss what “going the extra mile” means to us as an organization. we also needed to develop a scalable training method for student employees. although our research design and methods did not include tests for validity, the results strongly suggest that standards and training had a positive impact on improvement. it was also very useful to have specific evidence for staff to see where changes needed to be made. and it was equally important to celebrate with staff when there was improvement! the study provided an excellent opportunity for the libraries’ staff to discuss what service means to us as an organization and helped enhance the already established culture of excellent customer service.   it is essential to get buy-in from staff before conducting a mystery shopper study and make the goals of the study clear and transparent. for some staff it may always be perceived as a threat and management needs to assure them that such assessment is necessary in order for the library to remain viable and current and to ensure that we are providing the services and resources that our customers need and desire.   references   acrl (2012) top ten trends in academic libraries. (2012). college & research libraries news, 73(6), 311-320. retrieved from http://crln.acrl.org/   andruss, p. (2010). the case of the missing research insights. marketing news, 44(7), 23-25.   backs, s. m., & kinder, t. (2007). secret shopping at the monroe county public library. indiana libraries, 26(4), 17-19.   bell, s j. (2011, march). delivering a wow user experience: do academic librarians measure up? paper presented at the biennial meeting of the association of college and research libraries, philadelphia, pa. retrieved from http://www.ala.org/acrl/conferences/confsandpreconfs/national/acrl2011papers.   bell, s. j. (2014). staying true to the core: designing the future academic library experience. portal: libraries and the academy, 14(3), 369–382. http://dx.doi.org/10.1353/pla.2014.0021   benjes-small, c., & kocevar-weidinger, e. (2011). secrets to successful mystery shopping: a case study. college & research libraries news, 72(5), 274-287. retrieved from http://crln.acrl.org/   cocheo, s. (2011). mystery shop the whole bank. aba banking journal, 103(1), 13.   connaway, l. s., dickey, t. j., & radford, m. l. (2011). “if it is too inconvenient i'm not going after it”: convenience as a critical factor in information-seeking behaviors. library & information science research, 33(3), 179-190. http://dx.doi.org/10.1016/j.lisr.2010.12.002   czopek, v. (1998). using mystery shoppers to evaluate customer service in the public library. public libraries, 37(6), 370-375. .   deane, g. (2003). bridging the value gap: getting past professional values to customer value in the public library. public libraries, 42(5), 315-319. retrieved from http://www.ala.org/pla/publications/publiclibraries   foster, n. f., & gibbons, s., (2007). studying students: the undergraduate research project at the university of rochester. chicago, il: association of college and research libraries.   foster, n. f. (ed.) (2013). studying students: a second look. chicago, il: association of college and research libraries.   fox, r. & doshi, a. (2011). library user experience. arl spec kit 332. washington, dc: association of research libraries.   gavillet, e. l. (2011). the “just do it” approach to customer service development: a case study. college & research libraries news, 72(4), 229-236. retrieved from http://crln.acrl.org/   hammill, s. j. & fojo, e. (2013). using secret shopping to assess student assistant training. reference services review, 41(3), 514-531. http://dx.doi.org/10.1108/rsr-12-2012-0086   hernon, p., nitecki, d. a., & altman, e. (1999). service quality and customer satisfaction: an assessment and future directions. journal of academic librarianship, 25(1), 9-17. http://dx.doi.org/10.1016/s0099-1333(99)80170-0   kocevar-weidinger, e., benjes-small, c., ackermann, e., & kinman, v. r. (2010). why and how to mystery shop your reference desk. reference services review, 38(1), 28-43. http://dx.doi.org/10.1108/00907321011020707   peters, k. (2011). office depot's president on how "mystery shopping" helped spark a turnaround. harvard business review, 89(11), 47-50. retrieved from https://hbr.org/   roy, l., bolfing, t., & brzozowski, b. (2010). computer classes for job seekers: lis students team with public librarians to extend public services. public library quarterly, 29(3), 193-209, http://dx.doi.org/10.1080/01616846.2010.502028 .   tesdell, k. (2000). evaluating public library service—the mystery shopper approach. public libraries, 39(3), 145.   tierney, j. (2012, february 2). nine merchants earn top marks in online customer service study. multichannel merchant exclusive insight. retrieved from http://multichannelmerchant.com/ecommerce/nine-merchants-earn-top-marks-in-online-customer-service-study-02022012/   uncg university libraries (2015a). mission statement, goals and values. retrieved from http://library.uncg.edu/info/mission_statement.aspx   uncg university libraries (2015b). customer service skills. retrieved from http://uncg.libguides.com/customerservice   uncg university libraries (2016) assessment information. retrieved from http://uncg.libguides.com/libassessment     appendix a mystery shopper questions access services music reference (now research outreach and instruction) special collections and archives (scua) what are your hours today? i just heard a symphony called witches sabbath. do you have a recording of this on cd? when were presidents only serving two terms and what law was that? how many books can i check out at one time? i’m in a wheelchair and i want to come to the library? where can i park and how to i get into the building? i’m not a music student, but i need biographical info on stravinsky for my russian history class. can you help me? i need to research the gaming industry. i’d like to donate some books to the library. who can i talk to about this? i would like to check out an ipad. how long can i keep it and what downloads can i put on it? what are your hours today? i’m researching the travel industry as a possible career. where can i look? i’m looking at your homepage, and i came across the term “finding aid.” what is that? how do i use it in my planned research? how long can i check out items? i need to fax something. can i do that here? i’m supposed to find some blues music for my african american history class. is there something i can find online? can i scan something in the library? i need to change my uncg password. where can i do that? i’d like this cd please. i need to fax something. can i do that here? can i check out materials from special collections and university archives? do you have your policies posted online? if so, can you show me where they are on the library site? is there a place i can meet my group in the library? do you take donations of lp’s? i need to find financial information about the hilton hotel chain. what are your hours today? i need to make a color print. where can i do that? do you have a score of beethoven’s eroica symphony? i’m looking for an article from the nats journal from 1994 and i can’t find it online. when did uncg change from being a women’s college to a co-ed university? i need help with my laptop. where can i go? how long can i check out items? i need to cite this article in apa citation style. my grandmother graduated in 1945; i’d like to find her picture in the yearbook. how do i renew my books? i’m looking for a recording of “alexander’s ragtime band” to use for an american social history class. is there a way i can get that online? which supreme court justice has been on the court the longest and who appointed him or her? my family has a large collection of old papers that seem to be related to greensboro and uncg. can i scan something in the library? i need to find a recording of “brahms requiem.” i’m not a music student. can i check out the cd? i need some films on how to prepare for a job interview. i am completing a research paper for a history class. i used your university archives collection. is there a specific way to cite my sources?     appendix b mystery shopper rating sheet     microsoft word article_macdonald_10pt font evidence based library and information practice 2008, 3:3 18 evidence based library and information practice article healthcare services managers: what information do they need and use? jackie macdonald library and knowledge management services shared services (dalhousie school of nursing – yarmouth site, annapolis valley health, south shore health, south west health) yarmouth, nova scotia, canada centre for health information management research department of information studies, university of sheffield, sheffield, uk e-mail: jmacdonald@swndha.nshealth.ca peter bath senior lecturer in health informatics, and director centre for health information management research (chimr), department of information studies university of sheffield, sheffield, uk e-mail: p.a.bath@sheffield.ac.uk andrew booth director of information resources, and reader in evidence based information practice school of health and related research (scharr), university of sheffield sheffield, uk e-mail: a.booth@sheffield.ac.uk received: 2 june 2008 accepted: 6 august 2008 © 2008 macdonald et al. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objectives – the purpose of this research project was to gain insight into the information behaviour of healthcare services managers as they use information while engaged in decision-making unrelated to individual patient care. methods – this small-scale, exploratory, multiple case study used the critical incident technique in nineteen semi-structured interviews. responses were analyzed using ‘framework,’ a matrix-based content analysis system. evidence based library and information practice 2008, 3:3 19 results – this paper presents findings related to the internal information that healthcare services managers need and use. their decisions are influenced by a wide variety of factors. they must often make decisions without all of the information they would prefer to have. internal information and practical experience set the context for new research-based information, so they are generally considered first. conclusions – healthcare services managers support decisions with both facts and value-based information. these results may inform both delivery of health library services delivery and strategic health information management planning. they may also support librarians who extend their skills beyond managing library collections and teaching published information retrieval skills, to managing internal and external information, teaching information literacy, and supporting information sharing. introduction and context of study internationally, more than $125 billion (u.s. dollars) is spent each year on research designed to improve patient outcomes (global forum 83). however, health research is not consistently translated so that it can be implemented in practice (canadian institutes), and healthcare policy decisions are made with little reference to research evidence (brehaut and juzwishin 4; mitton et al. 1660; zitner 38). some academic researchers have suggested healthcare services managers should apply systematic decision-making approaches to all healthcare decisions (winkler 57). others have wondered why evidence-based approaches are not being applied to all healthcare system decisions (kadane 565), and why accountability to evidence is not required (canadian health 3). this research was initiated when a committee, working to integrate a population health approach into healthcare services managers’ decision-making, asked when in the decision process is the optimal point at which to consider population health issues. this question could not be answered from the existing research literature. this paper presents findings from the initial phase of a two-part research project. this exploratory study examined the information healthcare services managers used to support decisions unrelated to individual patient care. these services included clinical decisions for groups of patients, such as those involved in developing practice guidelines, compliance with patient safety standards, planning chronic disease prevention strategies, and other strategic, tactical, and operational decisions made within the organization. definitions for the purposes of this research, a manager is a paid employee charged with leading an organization or one of its subunits; the manager may or may not have staff to supervise or budgets to manage. a manager may be a senior executive, the ceo, or a vice president who manages a portfolio of services, a director who oversees services of two or more departments, or a department manager. a fourth category, termed “other leaders,” describes managers who oversee specific initiatives within a department or service, such as injury prevention, health planning, or infection control. other leaders generally serve as organizational information gatekeepers with respect to evidence based library and information practice 2008, 3:3 20 their subject areas. a manager’s information behaviour describes his approach to seeking and handling information at work. this study developed working definitions for internal and external information. the research literature discusses the difference between internal and external information (dervin 332), explicit and tacit knowledge (nonaka and takeuchi 8-9), tacit knowledge (polanyi 4), cultural and explicit knowledge (choo 111), and scientific and colloquial evidence (lomas et al. 3). while all these concepts are relevant to this research, each seemed only partially congruent to the phenomena in this study. none of the terms adequately captured the distinction participants reported between the two. within this study, external information is information created outside the organization, consisting of research-based information that describes what other organizations are doing as observed through visits, or reported by experts, or government reports. external information has not yet been applied, implemented, or interpreted. internal information is information created within the organisation and may, incorporate research information that has been applied, absorbed, synthesized or translated within the organization. internal information may be implicit knowledge or explicit information. it may be a by-product of healthcare services or purposefully written as reports, meeting minutes, policies, or practice guidelines. literature review for this research project, a comprehensive search reviewed the literature of library and information sciences, operations research, management science, medicine and the healthcare professions, medical education, health administration, information technology, and computer science. the literature was examined for research related to managers’ information behaviour in general, as well as healthcare professionals’ information behaviour specifically. searches of individual databases included medline; acm digital library; library literature, library and information sciences abstracts; cinahl; abi inform; digital dissertations; and web of science. chaining and citation searching identified key research articles. the literature review began at the proposal stage and continued as an iterative process throughout data analysis and report writing. decision influences organizational knowledge and other aspects of internal information are recognized as important influences on managers’ decisions. less tangible variables consist of perception of the decision’s importance, its importance to the organization, characteristics of the task or problem, time pressures, decision deadlines, and simultaneous decisions or priorities facing the manager at the same time, established interpersonal behaviour patterns involving the manager, and the manager's decision role (saunders and jones 35). decision influences identified in healthcare settings incorporate a variety of personal qualities and capacities such as values and beliefs, leadership, knowledge and skills, resources, organizational support, partnership links, networking, the perceived benefit of change, and the complexity of the innovation itself (bowen and zwi). factors that influence groups engaged in health policy decision-making processes involve usefulness of the innovation, the influence of the individual leading the decision, legislation, and politics (bowen and zwi 0602). mitton and patten (148) observed that in the absence of “good concrete evidence,” healthcare decision makers used intuition, professional experience, knowledge of patient preferences, and situation matching. these were termed evidence based library and information practice 2008, 3:3 21 “soft” evidence and seen to be powerful forces in decision-making. healthcare services managers’ use of other, more tangible information has been noted. moahi (121) observed that managers used government documents, circulating mail and correspondence, office discussions, meetings, other departments, and telephone conversations as information sources in their work. internal information sources from finance and human resources departments (smith and preston) and external community-based information (uk dept. of health 40) that healthcare services managers might use to support decisionmaking have been identified. brehaut and juzwishin (15-20) outlined seven categories of information for consideration in healthrelated public policy development consisting of social and system demographics, technology, environment, economics, politics, legislation, and ethics. two approaches were identified for classifying the information healthcare services managers’ use, one for health information (lomas et al.) and one for organizational information (choo). choo classified organizational knowledge as explicit, tacit, or cultural (136). for choo explicit knowledge is rule-based and typically involves written documents such as policies, guidelines, meeting minutes, union contracts, or position descriptions. (choo 136) . cultural knowledge is background information incorporating shared assumptions and beliefs about the organizations’ goals and capabilities, customers, and competitors. it is used to assign value and significance to new information (choo 136). if it contains taboos, it is less likely to be shared or written down. tacit knowledge is acquired through experience--the unspoken knowledge used by members of an organization to perform their jobs and to make sense of their worlds (polanyi 60). tacit knowledge is hard to verbalize, so is the most difficult form of organizational knowledge to capture. a systematic review identified three forms of evidence used in healthcare decisionmaking “medically oriented effectiveness research,” context free with respect to the decision; context sensitive “social scienceoriented research,” and “colloquial evidence,” the expertise, views, and realities of stakeholders (lomas, et al. 14-5). categories for colloquial evidence consist of professional experience and expertise, judgement, resources, values and decisionmaking context, habits and traditions, lobbyists and pressure groups, and pragmatics and contingencies (davies; lomas et al. 15). researchers have identified categories for social science-oriented context related scientific evidence comprised of implementation evidence, organizational evidence, ethical evidence, attitudinal evidence, organizational capacity evidence, forecasting evidence, and economics/finance evidence (lomas et al. 14). health services managers’ information behaviour aside from work related to the use of research (baker, ginsburg, and langley; caccia-bava, guimaraes, and harrington 205), there have been few studies of healthcare services managers’ information behaviour. a canadian health services research foundation report notes the importance of both values alongside facts in healthcare services managers’ decisionmaking (2). three library and information sciences studies relevant to this research were conducted in the u.k. (head), botswana (moahi) and poland (niedźwiedzka). head interviewed ten healthcare services managers, looking for differences between career managers who entered healthcare services as managers rather than as evidence based library and information practice 2008, 3:3 22 clinicians, and hybrid managers, clinical professionals who later became managers. head’s research indicated that both groups needed internal and external information, and a case was made for health library services to manage both types of information. moahi used observations and interviews in a qualitative study of the information behaviour of twenty-eight healthcare planners, managers, and administrators. the study examined tasks carried out by healthcare services managers, their information needs, motivation for information seeking, information seeking behaviour, information sources and channels, and problems and barriers. moahi concluded that her participants were similar to managers in general, with respect to their information behaviour. she determined that participants’ effectiveness was hampered by a lack of information management infrastructure. niedźwiedzka used a mixed methods approach that involved quantitative analysis of questionnaire responses from 815 managers. her project also included a qualitative analysis of transcripts from five focus groups and ten oral interviews. she examined the needs, preferences, and limitations of healthcare services managers as information users. she also examined the environmental factors that influenced their information behaviour. niedźwiedzka observed that managers tend to use intermediaries, generally other staff, to search for, process, and evaluate information needed. although head, moahi, and niedźwiedzka explored different questions, all three observed the importance of internal or local information to healthcare services. in addition to these three studies, a fourth study of managers in the not-for-profit sector is relevant for its methodology and findings. zach (“modelling” 52, 54) used semi-structured interviews and the critical incident technique in a multiple case study of non-profit arts administrators’ information behaviour. she found that american arts administrators relied heavily on direct personal experiences to fill their information-seeking needs, frequently “satisficing” (zach, “investigation” 32), or settling, for the best decisions they could make under the circumstances. they would continue to work through the rational decision-making model, searching, identifying, and evaluating alternatives, until they were certain they had made the best decision (simon xxv. 240-1). managers, their information behaviour in general and internal information research has determined that managers generally prefer to receive information orally (daft, lengel, and treveno 356; meertens 5; moahi 160). researchers who have considered managers’ decisions in real world settings (berryman 210; lipshitz et al. 341) have classified their decision-making approach as “naturalistic decision-making.” naturalistic decision-making is characterized by time pressures, uncertainty, ill-defined goals, high personal stakes and a focus on using experience and expertise (lipshitz et al. 332-4). other characteristics are pattern matching, forward reasoning, and story-telling to anticipate the decision outcome, rather than searching for new information (lipshitz et al. 341). other research suggests that healthcare services managers may be naturalistic decision makers (baker, ginsburg, and langley 101-7). managers value internal information sources and rely on coworkers and colleagues with whom they have established relationships (mackenzie). information flow is often hierarchical, with managers obtaining most information from the evidence based library and information practice 2008, 3:3 23 subordinates closest to them (jones and mcleod 220-49). there is little research literature related to healthcare services managers’ information behaviour, their decision-making phases, or their use of information to support decisionmaking. the literature review failed to identify any research that reviewed how healthcare services managers deal with information gaps in decision-making. a definitive system for organizing and classifying the different kinds of information needed by healthcare services managers has not been established. this exploratory study aimed to address these issues. study aims and objectives this project sought to examine decisions made by healthcare services managers. it aimed first to identify the types of information used in the decision-making process and then to determine whether there were differences in the types of information used at different points in the process. the study sought to determine whether the information used by healthcare services managers might be classified according to an existing classification system. methods this exploratory study used a multiple case study approach. semi-structured qualitative interviews were conducted using a critical incident technique (cit) (flanagan 1954). the cit is a five-step procedure for gathering facts, and it is designed to isolate the significant or critical factors that contribute to success or failure. it is used frequently with in-depth semi-structured interviews, and it has been adapted for use in different disciplines where specific processes are being examined. all of the nineteen participants interviewed were located in nova scotia, canada. seventeen were selected from the paid leadership of a rural district health authority and two were from volunteer board members, participants were selected based on their work position and leadership status within the organization (i.e., senior executives, directors, managers, other leaders, or board members); by portfolio (i.e., acute care, community health, operations, or administration); and by employer (i.e., single district health authority, or consolidated health service). interview questions (appendix) were organized in three sections: critical incident technique questions, general questions about information seeking, and population health knowledge questions. each interview question had a set of additional probing questions to be used as needed. all interviews were audiotaped and transcribed verbatim. they were indexed categorically with 526 terms in four broad families using atlas.ti 4.1 software, to provide the qualitative data analysis. the interviews were analyzed according to “framework,” a matrix-based content analysis technique developed for applied social policy qualitative research questions by the u.k. national centre for social research (ritchie and spencer). this data analysis tool facilitates withinand betweencase comparisons. it is important to understand work related information needs in the context in which they arise (leckie, pettigrew, and sylvain 101). information behaviour research is traditionally used to study academic scholars (case 296). students and faculty generally focus on one subject at a time as they conduct exhaustive searches of related research literature. information systems and services designed to meet scholars’ needs may not meet the need of healthcare evidence based library and information practice 2008, 3:3 24 services managers working in real world settings and faced with challenges such as multiple simultaneous conflicting priorities. framework of decision-making behaviour after consideration of responses with respect to managers’ roles (mintzberg 59) a conceptual framework was developed from research related to decision complexity. this framework sought to establish a more complete understanding of participants’ decision-making behaviour and incorporated • decision levels (heller et al. 5) • decision modes (lipshitz and strauss 158; march and simon; allison 246; cohen, march, and olsen 1, 16; mintzberg, raisinghani, and thêorét 246-75) • decision types (canadian health 2) • decision structure (simon 31) • decision situations (mintzberg, raisinghani, and thêorét 251). framework of phases of decision-making a second framework was constructed from work by simon (41) and mintzberg et al. (252) to help explain when information was used in the rational decision-making processes, simon identified four phases: • phase 1, “intelligence,” identifying the problem • phase 2, “design,” inventing, developing, and analyzing possible courses of action • phase 3, “choice,” selecting a particular course of action from those available • phase 4, “review,” carrying out decisions and assessing past choices (41). simon noted that “each phase in making a particular decision is itself a complex decision-making process”(43). mintzberg and his colleagues conducted a field study of strategic decision processes across twenty-five organizations (252). they identified three stages in the decision process, parallel to the first three phases of simon’s four-stage decision process: “identification,” “development,” and “selection.” framework of information for organizational decision-making researchers also developed a framework to organize the information that healthcare services managers said they used or needed when making decisions. in the first round of indexing, all information mentioned in participants’ transcripts was indexed by type and source of information. passages indexed as information types or sources were then examined to see whether they influenced aspects of the decision process. these were first indexed as “decision influences,” and then sorted within two broad categories--organizational knowledge and gaps. organizational knowledge was sorted using choo’s framework for explicit, tacit, or cultural organizational knowledge (figure 1). information was indexed as a “gap” when participants specifically mentioned wanting information but not being able to obtain it. boxes at the lowest level of fig are broad headings that may consist of two or more narrower subgroups. for example, participants mentioned patient safety, employee safety, and environmental safety. these are contained in the “safety” box under “organizational considerations.” research results indicate some overlap between subcategories of explicit knowledge, davies’ categories of colloquial evidence, and context sensitive scientific evidence (lomas et al. 14-5). the literature review was initially unable to suggest the best way to sort cultural knowledge. knowledge and information evidence based library and information practice 2008, 3:3 25 fig. 1. organizational knowledge and gaps that influenced decisions evidence based library and information practice 2008, 3:3 26 that influenced decisions and had been initially indexed as cultural knowledge were later sorted into two broad headings – situation variables and environmental variables -during interview data analysis. within this study situational variables were “decision weather,” temporary conditions that might apply to only one decision situation. environmental variables were “decision climate,” of a longer duration, and more general in nature; they might apply to any situation within the department, portfolio or district. any one of these powerful variables could effectively block progression through the decision phases. in some cases where decisions affected by cultural knowledge could not be postponed, external facilitators were used to achieve consensus. after these categories were established and the sorting completed, an article was located where a similar approach was used. mick, lindsey, and callahati identified several levels of variables affecting managers’ information behaviour and labelled them individual variables, situational (task) level variables, and environmental level variables (347). results and discussion overview all of the nineteen interview participants had completed their postsecondary education. eleven had graduate degrees, and four had undergraduate degrees. of the twelve participants who were registered professionals, only one did not also have a university degree. the mean participant age was 51 years old, and the mean length of healthcare career was 21 years. the interviews identified four main themes: • information and decisions • information and sharing • information and seeking, and • information and population health. this paper presents findings about information that influenced decisions from the information and decisions theme. participants’ decisions were complex and multi-level. unstructured group decisions and their decision processes were typical of the naturalistic decision mode (macdonald et al. 23). the findings suggest that healthcare services managers are similar to managers in general as described in the research literature. participants’ descriptions of their own roles while engaged in decision-making (macdonald et al. 21) were similar to the managers’ roles described by mintzberg (59). almost all participants satisficed, that is, they terminated the information search process when they felt they had just enough information for a comfortable decision, recognizing that they did not have all of the information they would have liked to have had. these healthcare services managers’ information sources were also congruent with research on managers in general, in that they obtained information from subordinates (jones and macleod 232), from colleagues within the organization (mackenzie), and from counterparts in other organization (jones and macleod 232). one theory arising from these findings is that research based information is brought into the organization by middleand lower-level managers who have both experience and expertise in a subject and understand its relevance to the organization. some of these managers evolve into the role of information gatekeepers, while others are tasked with monitoring subject areas, contributing to group decisions, and writing guidelines for structured decision. these guidelines would typically be practice guidelines, policies, or service plans to provide guidance for frequent decisions that could be made at evidence based library and information practice 2008, 3:3 27 lower levels. when individuals take responsibility for monitoring a subject, putting new research into context, and sharing it within the organization, decision makers can access relevant research based information earlier in the decision process. further research is needed to determine whether information gatekeepers are an important market for health library services, to determine their information needs and to decide how to best meet those needs. these areas will be explored further in the next stage of the proposed research on information sharing. information use in the decision-making process responses were examined for details on what participants did when first faced with a decision situation, what information they used to support a decision, and what factors influenced the level of effort they would expend in looking for information. all participants looked for internal information, and most also mentioned looking for external information. some participants also sought knowledgeand research-based information. when is information used? using simon’s four-phase decision process outlined above (simon 41) as a framework, responses were examined to determine when information was used in the decisionmaking process. most participants considered internal information and organizational knowledge in phase 2, the design and development phase of the decision process. although no participants said that they found new research-based information that made a difference to decision outcomes, those who searched for research-based and other external information did so in phase 3, the choice and selection phase. this appears to be congruent with dervin who observed that evidence from perception research shows that humans take external information and organize it within their own internal information to make sense out of their world (326). it also supports the definition of colloquial evidence as stated by the canadian health services research foundation, “the role of colloquial evidence is more to inform the scientific evidence — guiding the selection and interpretation of science and filling in gaps when they appear” (2). few cases involved a fourth review phase, where the decision was implemented or evaluated. most cases that did consider information in phase 4 involved only one department, and their critical incidents were resolved by reorganizing staff within the department. there were no patterns observed with respect to these phase 4 cases other than that they drew on tacit and explicit information about staffing and scope of practice. in the rare case where a case that reached the review stage involved more than one department, changes were made to the communication and information management processes between the departments that had led to the critical incident. although this research project aimed to examine healthcare services managers’ information behaviour with respect to both internal and external information, study participants focused on internal information in their responses. references to external research and knowledge-based information such as found in books, journals and libraries were rare. the published literature of the health and medical professions has been indexed and organized by the u.s. national library of medicine in index medicus and medline, and networks of health librarians, professional organizations and publishers provide easy accessibility to health science information. it is possible that evidence based library and information practice 2008, 3:3 28 this convenient access may be the reason why these participants did not express concern about external research based information, libraries, and library services. another reason why libraries, library services, and research based publications were rarely mentioned is that they may still be seen simply as collections of clinical books and journals purchased, processed, and housed just in case they might be needed. healthcare services have evolved into highly specialized organizations where the division of labour and specialization of knowledge has become narrower and more restrictive (glouberman 10). it may be that as the amount of information directly relevant to the work of more than one healthcare worker has decreased, the value and relevance of traditional hospital libraries to information sharing has decreased. it may also be that in real world situations, decision makers rely more on experts who know both the subject and the situation and who can synthesize both in one piece of information. further research is needed. canada’s healthcare industry is estimated to be twenty-five years behind its banking industry with respect to information management (fell). contributing to the inefficiency of canada’s healthcare system is an information management infrastructure constructed from outdated, inadequate and mostly incompatible systems, according to a report from the canadian broadcasting corporation. few canadian healthcare organizations have had the resources for a planned information management infrastructure in the form of either systems to manage internal health information or staff trained in health informatics able to understand and work with clinical data (smith 13). health library user education services often focus on developing clinicians’ skills to retrieve published literature, and some focus on retrieval skills at point of care. the findings of this study suggest that there is a need to develop information retrieval skills at the “point of decision” and to expand skills beyond retrieving the published literature. this is in line with information literacy competency standards that recognize information literacy as more than information retrieval (acrl 2). what information influences healthcare managers’ decisions? the information that influenced healthcare managers’ decisions was varied and indexed as “organizational knowledge” (figure 1). a variety of information needs were identified but not met, these were indexed as “gaps.” these factors are described below, using the categories listed in the headings used in figure 1. explicit organizational knowledge participants’ decisions were influenced by explicit organisational knowledge, such as policies and guidelines. this knowledge was subcategorised into organizational values, organizational considerations, regulations, and resources. organizational values participants described influences on decisions similar to davies’ (5) colloquial evidence category for values and decisionmaking context. these two quotations show how they drew on values such as respect, trust, equity, and accountability, and used internal information as sources: “it will depend on what our philosophy is for the district and how … to incorporate that into our mission vision and values and how that may line up with some of the other work that has been out there in other places.” (other leader) evidence based library and information practice 2008, 3:3 29 “i like the [district] values: respect, integrity, responsibility, accountability … i try to be accountable for it … for it being right when values conflict, yes, or when ethical principles conflict; then trying to find ways to deal with that.” (manager) organizational considerations other explicit knowledge mentioned by participants may fit within several of davies’ categories for context-sensitive scientific evidence (lomas et al. 14). some of the organizational considerations appear in the organization’s mission, vision, and strategic directions; others are represented by committees or positions within the organization that monitor safety, including staff safety, patient safety and environmental safety, quality, risk, ethics, and population health, as shown in the following quotations: “… how that is going to impact … patients and nurses, quality of life for the nurses, and safety for patients.” (other leader) "i would look at ethics principles … those would be the principles that i would go down through and talk about with people around.” (manager) "i actually keep a copy of the strategic plan, population health principles, and my position description; and some of this i carry with me … every day ... and it helps me keep focused on what is significant in terms of my jurisdiction.” (manager) regulations study participants were influenced by regulations, such as legislation, policies, procedures, union contracts, position descriptions, professional standards, and privacy issues. this information was generated both within the organization and externally, often by provincial government or by professional organizations. this is congruent with head (43) and niedźwiedzka (107-8) who observed the importance of legal information, guidelines, policies, and similar regulations to healthcare services managers in their decisions. these two quotations illustrate how legislation, standards and other guidelines influenced participants: “yes, generally if we are looking at a project … we will look at what the regulatory bodies say first … and start building from there.” (director) “… we always look back at the standards of practice.” (other leader) resources information on organizational resources was most commonly mentioned as an unmet information need. this category consisted of human resources, both numbers of staff and appropriate skill sets; financial resources, both costs and available budget; physical resources, equipment and space, and time. organizational resources were examined in terms of past expenditures of resources for service use and this information was used to project future needs. similar to head’s (57) study participants, these healthcare services managers had difficulty matching productivity against available resources, as described in these two quotations: “we had to use internal information, and that was number of staff, where are their positions’ workload?” (director) “doing budget what was spent in the past, why you are over? information around productivity standards for the province, from other hospitals what is already going on if you have high productivity standards and still are not meeting the workload.” (manager) evidence based library and information practice 2008, 3:3 30 cultural organizational knowledge healthcare managers’ decisions were also influenced by cultural organizational knowledge or background information. as discussed above, cultural knowledge was sorted into two categories situational variables that had an impact on the immediate decision and environmental variables that had an impact on any decision made in the same part of the organization in the same time period. situational variables in this study, situational variables consisted of buy-in, level of controversy, conflict of interest, bias, and lack of bias. the following quotations illustrate the influence of situational variables: “you can't do something usually unless you wait … have that buy-in from the top … you struggle and struggle and all of sudden for some reason, the timing is right and it happens.” (director) “the other thing is how controversial the issue is and who is involved with it who is going to challenge me on it?” (director) environmental variables environmental variables were internal or external. politics and power were identified as influences on decision-making. participants described the need to see where their decisions fit within the organization, and in some situations they have to wait for a culture change, as noted in the following quotations: “we tried to think of who we might conscript to work on this with … some other people were not interested in sitting down … we were not sure of the political agendas being played out … we weren't sure who we could trust.” (director) “there is nothing more frustrating in your career than investing a lot of time, a lot of your personal emotional effort to feel strongly about something and then to have it go nowhere, because you don't really have the power to move something …” (director) tacit organizational knowledge healthcare managers’ decisions were also influenced by tacit organizational knowledge, that is, knowledge that involves skills and information gained experientially and through intuition. these were sorted into several categories, including awareness of decision complexity, decision stakes, task importance and participant’s confidence in their own judgement (figure 1). these had more in common with polanyi (60), than with caccia-bava et al. (205) who considered tacit knowledge with respect to healthcare services managers’ knowledge of information technology within a framework of absorptive capacity. participants drew on tacit knowledge to determine decision importance, as noted in these two quotations: “i would look and say how important is this decision and what impact will it have one way or another on what happens. and if i kind of rate it as "this is one hell of a big decision" that has to be made, and it is going to have an impact on a ton of people, then that's the one that i am going to pick to try and take and look at everything to consider and do and take the time. and the driving force on what i do and what i use is, i think, going to be around how important and how relevant this decision is.” (director) “chances are if it is something that comes with a deadline, it is an important decision and would have big ramifications … and the reason there is a deadline is that they need to bring in the thoughts and ideas of other influential people within the organization; and i would never want my thoughts and ideas to go forward without evidence based library and information practice 2008, 3:3 31 having been well researched and well educated, well put together.” (manager) what information is missing when healthcare managers make decisions? participants identified many gaps in the internal information that they needed. these could be factors in deciding whether to continue the decision or to postpone it until additional resources were secured. they also helped participants determine what additional information would be sought in the choice phase of decisionmaking (phase 3), when selections are made between alternative choices. these findings are congruent with dervin’s sense-making theory where she uses a bridge metaphor to explain how an individual who encounters an information gap between his understanding and experience needs information to make sense of his situation and then move on (68). the following quotations are examples of participant comments about information gaps related to their specific critical incidents … “we are starting to have [named outpatient care service] clinics, and so we are trying to gather information on that and figure out how to interpret information. we don’t have as much information and are not 100% sure of what it means.” (director) related to their gaps in information management… “for all of the areas in the portfolio we get almost no information from a data perspective whether we are doing a good job or a bad job … a lot of the information is out there, but we haven't structured it in such a way that it filters back up.” (senior executive) related to their gaps in understanding other departments … “yes, i cannot tell you in [my service area] with any degree of accuracy how much time [my staff] … is spending on programs and activities … because we don't have that information system in place … and that is a problem.” (director) “… people have very little notion of how what they do impacts on other areas.” (senior executive) “they keep a lot of information to get it from them may not be that easy, because the person who has it might be on vacation or they are not on site ... at another hospital and call them there.” (manager) there were other gaps related to resources needed to sustain or implement decisions, such as human resources, space and other physical resources, financial resources, and time. no research was identified that considers what healthcare services managers do when they are not able to find the information they need to bridge a gap. as noted above, these participants satisficed. they reported that they made decisions recognizing that they did not have all of the information they needed. they may also have postponed making a decision. in this study, few cases involved decision-making in the review phase (phase 4), where the decision had been implemented and evaluated. where decisions were not crisis situations, any of these gaps might provide reason not to continue with the decision, perhaps helping explain why in most cases decisions paused or stopped at some point in the process. further research would be needed to determine whether and how gaps influenced the decision process and how healthcare services managers cope with gaps, when they decide to satisfice, and when they decide to postpone decisionmaking. conclusions and implications for further research this paper presents findings about the information that healthcare services evidence based library and information practice 2008, 3:3 32 managers use in decision-making unrelated to individual patient care. it identifies points during the decision process at which information is used. all participants in this interview study drew on information in the intelligence and design phases of decision-making (the first two of four decision phases), and most also engaged with information during the choice (third) phase. they tended to consider internal information that had been created or had already been implemented within the organization in the intelligence (phase 1) and design (phase 2) stages, when they identified the problem and determined possible courses of action. in these phases information was gathered to set the context, and information gaps were identified. in some cases, participants actively searched for additional external information in phase 3, to help select a course of action from alternatives. in other cases, participants satisficed, that is made a decision without all of the information they would have liked. there were few cases where decisions were made in the review phase (phase 4). in cases where internal information suggested a conflict between a course of action and organizational knowledge, where it was not supported by cultural or tacit knowledge, or where crucial gaps encountered could not be bridged, participants postponed the implementation of their decisions or terminated the process. the research findings presented in this paper provide some evidence to support librarians who have expanded beyond traditional health sciences library services. these findings may suggest that healthcare services managers are more likely to use research-based information to support their decision-making if their internal information is well enough managed so that their most basic information needs are met. the same skills that librarians have used to manage collections of publications may be of great value to the organization if they extend their services beyond the walls of the library. librarians might find ways to manage information created within the organization, and to integrate it with external, researchbased information. research to determine how well the academic model of library services delivery meets healthcare services needs may also be useful. this study suggests that healthcare services librarians should look holistically at information literacy within their organizations. when they identify gaps in information literacy skills they should partner with other departments to determine how to address these gaps. more research is needed to determine who makes decisions, including decisions about the care of individual patients and groups of patients, who makes what decisions in healthcare services, and whether these decisions tend to be individual or group decisions. library managers need to know more about how best to integrate information to meet the needs of healthcare services managers. some of these issues will be explored in the second phase of this study, which will examine healthcare services managers’ information sharing to support group decisions. works cited allison, graham t. essence of decision: explaining the cuban missile crisis. boston, ma: little brown, 1971. association of college and research libraries (acrl). information literacy competency standards for higher education. chicago, il: american library assoc., 18 jan. 2000. 25 aug. 2008 <file://localhost/<http/:www.ala.org:ala: acrl:acrlstandards:informationliteracyc ompetency.cfm>. evidence based library and information practice 2008, 3:3 33 baker, g. ross, liane ginsburg, and ann langley. "using knowledge and evidence in health care: multidisciplinary perspectives. ed. l. lemieux-charles and f. champagne. toronto: university of toronto press, 2004. 86-114. berryman, jennifer m. "judgements during information seeking: a naturalistic approach to understanding the assessment of enough information." journal of information science 34.2 (2008): 196-206. doi:10.1177/0165551507082589 bowen, shelley, and anthony b. zwi. "pathways to “evidence-informed” policy and practice: a framework for action." plos med 2.7 (2005): e166. 30 aug. 2008 doi: 10.1371/journal.pmed.0020166 brehaut, jon d., and don juzwishin. bridging the gap: the use of research evidence in policy development, sept. 2005. alberta heritage foundation for medical research, initiative #18. 25 august 2008 <www.ihe.ca/documents/htafr18.pdf>. caccia-bava, maria do carmo, tor guimaraes, and susan j. harrington. "hospital organization culture, capacity to innovate and success in technology adoption." journal of health organization management 20.23 (2006): 194-217. canadian broadcasting corporation. "doctors too slow to embrace electronic health records” 25 may 2007. 30 aug. 2008 <http://www.cbc.ca/health/story/2007/0 5/25/electronihealthrecords.html#skip3 00x250>. canadian health services research foundation. health services research and evidence-based decision-making, june 2000. 30 aug. 2008 <http://www.chsrf.ca/knowledge_tr ansfer/pdf/ebdm_e.pdf>. ---. "weighing up the evidence. making evidence-informed guidance accurate, achievable, and acceptable." [international workshop]. 29 sept. 2005. 2 sept/ 2008 <http://www.chsrf.ca/other_documents /pdf/weighing_up_the_evidence_e.pdf >. canadian institutes of health research. innovation in action. ottawa: canadian institutes of health research, 2004. case, donald o. looking for information: a survey of research on information seeking, needs, and behavior. 2nd ed. amsterdam: academic press, 2007. choo, chun wei. the knowing organization: how organizations use information to construct meaning, create knowledge and make decisions. 2nd ed. new york: oxford university press, 2006. cohen, michael d., james g. march, and johan p. olsen. "a garbage can model of organizational choice." administrative science quarterly 17 (1972): 1-25. daft, richard l., robert h. lengel, and linda klebe treveno. "message equivocality, media selection, and manager performance: implications for information systems." mis quarterly 11.3 (1987): 355-66. evidence based library and information practice 2008, 3:3 34 davies, philip. "evidence-based government: how can we make it happen?” canadian association of paediatric health centres annual meeting, montreal, qc, canada. 9 dec. 2007. dervin, brenda. "from the mind’s eye of the user: the sense-making qualitativequantitative methodology." qualitative research in information management. ed. j.d. glazier and r. r. powell, 1992. 61-84. ---. "strategies for dealing with human information needs: information or communication." journal of broadcasting 20 (1976): 324-51. fell, anthony. s. "letter to mohamed dhanani, resigning as director of the toronto central local health integration network” 2 apr. 2007. 30 aug. 2008 <http://www3.thestar.com/static/pdf/0 70419_fell_letter.pdf>. flanagan, john c. "the critical incident technique." psychological bulletin 51.4 (jul. 1954): 327-58. global forum for health research. monitoring financial flows for health research 2006: the changing landscape of health research for development. eds. andrés de francisco and stephen matlin. 2006. 17 june 2007 <http://www.globalforumhealth.org/fil esupld/monitoring_financial_flows_06/ contents.pdf>. glouberman, sholom. health and everything: what’s next in population health. a workshop presented by the change foundation, 2001. 20 nov. 2003. head, angela l. "an examination of the implications for nhs information providers of staff transferring from functional to managerial roles." msc. economics thesis. university college of wales, aberystwyth, wales, uk 1996. heller, frank, pieter drenth, paul koopman, and veljko rus. decisions in organizations: a three county comparative study. london: sage, 1988. jones, jack william, and raymond mcleod. "the structure of executive information systems: an exploratory analysis." decision sciences 17 (1986): 220-49. kadane, joseph b. "bayesian methods for health-related decision making." statistics in medicine 24.4 (28 feb. 2005): 563-7. leckie, gloria j., karen e. pettigrew, and christian sylvain. "modelling the information seeking of professionals. a general model derived from research on engineers, health care professionals, and lawyers." library quarterly 66 (1996): 161-93. lipshitz, raanan, gary klein, judith orasanu, and eduardo salas. "taking stock of naturalistic decision making." journal of behavioral decision making 14.5 (dec. 2001): 331-52. lipshitz, raanan and orna strauss. "coping with uncertainty: a naturalistic decision-making analysis." organizational behavior and human decision processes 69.2 (feb. 1997): 14963. lomas, jonathan, tony culyer, chris mccutcheon, laura mcauley, and susan law. conceptualizing and evidence based library and information practice 2008, 3:3 35 combining evidence for health system guidance: final report. ottawa. canadian health services research foundation, may 2005. 15 dec. 2007. accessed 30 aug. 2008 <http://www.chsrf.ca/other_docume nts/pdf/evidence_e.pdf>. macdonald, jacqueline m., peter a. bath, and andrew booth. “healthcare managers’ decision making: findings of a small-scale exploratory study.” proceedings of 12th international symposium on health information management research, sheffield, uk, july 2007. eds. bath peter a., albright k. and norris t. university of sheffield, sheffield, uk: july 2007. 1727. mackenzie, maureen l. "managers look to the social network to seek information." information research 10.2 (2005). march, james g., herbert a. simon, and harold steere guetzkow. organizations. 2nd ed. oxford, uk: blackwell, 1993. meertens, lucas o. "changes in information source preference of managers over the decades." 6th univeristy of twente student conference on it, enschede, the netherlands. 2008 feb. 2. 2 sept. 2008 <http://referaat.cs.utwente.nl/new/pape r.php?paperid=184> mick, colin k., georg n. lindsey, and daniel callahati. “toward usable user studies." journal of the american society for information science 31.5 (sept. 1980): 347-56. mintzberg, henry. the nature of managerial work. new york: harper and rowe, 1973. mintzberg, henry, duru raisinghani, and andre thêorét. "the structure of "unstructured" decision processes." administrative science quarterly 21 (1976): 246-75. mitton, craig, san patten, howard waldner, cam donaldson "priority setting in health authorities: a novel approach to a historical activity." social science & medicine 57.9 (nov. 2003): 1653-63. mitton, craig, and san j. patten. "evidencebased priority-setting: what do the decision-makers think?" health services research and policy 9.3 (july 2004): 146-52. moahi, kgomotso h. "a study of the information behavior of health care planners, managers and administrators in botswana and implications for the design of a national health information system.” diss. university of pittsburgh, 2000. niedźwiedzka, barbara. "barriers to evidence-based decision-making among polish health care managers." health services management research 16.2 (may 2003): 106-15. nonaka, iikujiro, and hirotaka takeuchi. the knowledge-creating company: how japanese companies create the dynamics of innovation. new york: oxford university press, 1995. polanyi, michael. the tacit dimension. new york: doubleday, 1966. ritchie, jane, and liz spencer. "qualitative data analysis for applied policy research." analyzing qualitative data. ed. a. bryman and r. g. burgess. london: routledge, 1994. 172-94. evidence based library and information practice 2008, 3:3 36 saunders, carol and jack william jones. "temporal sequences in information acquisition for decision making: a focus on source and medium." academy of management review 15 (1990): 29-46. simon, herbert a. administrative behavior: a study of decision-making processes in administrative organization. new york, ny: macmillan, 1947. smith, jeremy. "wanted cyber clinicians to transform the nation’s healthcare system bit by bit." canadian healthcare manager 12.3 (2005): 14-5. smith, les, and hugh preston. "information management and technology strategy in healthcare: local timescales and national requirements." information research 5.3 (2000). uk department of health. overview and scrutiny of health guidance. london: u.k. department of health, 18 july 2003. 2 sept. 2008 <http://www.dh.gov.uk/en/publication sandstatistics/publications/publications legislation/dh_4009607>. winkler, robert l. "why bayesian analysis hasn't caught on in healthcare decision making." international journal of technology assessment in healthcare 17.1 (winter 2001): 56-66. zach, lisl. "when is “enough” enough? modelling the information-seeking and stopping behavior of senior arts administrators." journal of the american society for information science and technology 56.1(jan. 2005): 23-35. doi: 10.1002/asi.20092 zach, sarah e. "when is “enough” enough? an investigation of the informationseeking and stopping behaviors of senior arts administrators." diss. university of maryland, college park, md, usa, 2002. zitner, david. “is sane management possible in a crazy world?” healthcare papers 3.3 (2003): 36-43, 66-71. evidence based library and information practice 2008, 3:3 37 appendix participants were each asked to think of a critical decision they had made in the very recent past. this was to be a decision outside their normal routine – one they had not encountered before, perhaps one leading to initiating or terminating a new service orprogram, or one that had a direct impact on budget. interview questions 1. how did the decision come about? 2. what did you do first? 3. if you used any information to help make your decision, where did you get it? 4. if you used information that you were given or had already, what did you use, in terms of specific kinds of information? 5. if you didn’t have the information you knew you needed, where did you go first? 6. did you use any other source? 7. what information did you need but couldn’t find? 8. what information did you find most useful? 9. how did you decide when you had enough information? 10. if you were making this decision over again, what would you do differently, if anything. general questions 11. how do you tend to approach information related to your work with avh? which do you tend to do, keep up or look for information when you need it? 12. when you look for information to support a decision or perform a task, which do you tend to do – look for information to support one alternative, or map out two or more possible alternatives and look for information to support both or all of them? 13. what factors influence the level of effort you spend looking for information? (level of effort is the time and trouble needed to obtain information; the cost, number and types of sources checked; their ease of use or familiarity, location, accessibility, and ease of access.) 14. during the course of your work with avh, what are the most common types of decisions or tasks for which you look for information? evidence based library and information practice 2008, 3:3 38 15. generally, if you could pick one information format, which would you prefer verbal, printed, video, or electronic? 16. is there anything else you think i should know about the way you look for information? population health questions 17. during our 2002 accreditation, was the meaning of “the population health approach,” as it is used in the cchsa aim document, clear to you? 18. please share your experiences and opinions on learning experiences as they relate to population health. 19. did you use a population health approach or any other decision making framework when considering the issue we just discussed? 20. are you familiar with, or have you seen, the avh population health framework? microsoft word news_2eip.doc evidence based library and information practice 2006, 1:4  85 evidence based library and information practice     news      2nd evidence in practice award open for entries        © 2006 eblip. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    the 2nd evidence in practice award is now  open for entries.     in approximately 750 words, can you  describe a case study where your work has  influenced clinical practice? we are looking  for examples of good evidence‐based  librarianship practice in a healthcare setting,  examples of where librarians and  information professionals have influenced  clinical practice and patient outcomes. the  competition is open to partnerships of  clinical and health professionals in the uk.       the winning partnership will each receive a  personal digital assistant, £500 each  towards attendance at a professional  conference or course of their choice, plus a  free delegate place at the 3rd clinical  librarian conference. prizes are jointly  sponsored by nlh and bmj group.     entries can be submitted up to 31st march  2007 after which anonymised case studies  will be judged by an independent panel  combining clinical and information  expertise. the judgesʹ decision will be final.    the award will be presented at the 3rd uk  clinical librarian conference, 11th & 12th  june 2007, st williamʹs college, york  minster, where the award winners will an  opportunity to share their example of  successful practice.    online entry for the award is available at:  http://www.insitefulsurveys.com/survey.as p?si=110406111828   http://creativecommons.org/licenses/by/2.0 http://www.insitefulsurveys.com/survey.as news/announcements   canadian library assessment workshop – registration now open      2015. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   the canadian association of research libraries (carl) is committed to helping libraries develop and use outcomes-based measures. the goal of these measures is allow our libraries to better support research, teaching and learning at our institutions. through the delivery of assessment-related programming, information sharing, and support for relevant research, carl supports members as they use evidence to inform a process of ongoing development and improvement.   it is with this strategic initiative in mind that carl and ryerson university invite you to participate in the canadian library assessment workshop (claw) (http://www.carl-abrc.ca/en/research-libraries/claw/2015-canadian-library-assessment-workshop.html) which will take place in toronto on october 14 16, 2015. this workshop will be of interest to all academic and research librarians engaged in assessment. the program consists of 2 to 3 hour workshop style sessions with topics such as:   ·         linking your assessment plan to the library’s strategic plan ·         data collection for decision making ·         data visualization ·         assessing the unbundling of the big deal   participants will leave the workshop with tangible and practical ideas to take back to their libraries. registration is limited.   microsoft word art 1102 evidence based library and information practice 2008, 3:4 34 evidence based library and information practice article new intersections for student engagement in libraries: a qualitative exploration of collaborative learning with multimedia technologies katharine a. webb librarian and associate professor, the ohio state university libraries columbus, ohio, united states of america e-mail: webb.45@osu.edu tingting lu graduate associate and multimedia development specialist, the ohio state university libraries columbus, ohio, united states of america e-mail: lu.162@osu.edu elizabeth l. black assistant professor and systems librarian, the ohio state university libraries columbus, ohio, united states of america e-mail: black.367@osu.edu received: 15 january 2008 accepted: 05 november 2008 © 2008 webb, lu, and black. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – the purpose of this study was to explore new ways librarians can provide meaningful learning experiences for students beyond the traditional classroom assignment and the one-hour library instruction session. methods – the study was done within a qualitative framework using participative, interpretive, and personal experience methods. the research team consisted of two librarians and a graduate student. data collected included transcripts of audio-recorded team meetings and interviews, field notes, and a post-project survey, where students described their experiences negotiating the conceptual and technical processes of authoring a multimedia story. the evidence based library and information practice 2008, 3:4 35 instructional layer was built upon a constructivist approach allowing for a collaborative learning setting to foster learner control and self-efficacy. results – findings illustrate the benefits of collaborative approaches for enhancing the learning experiences of students in the library, in this case with multimedia. the data also suggest promising new ways for librarians to facilitate learning and to engage students in the library. conclusion – through a multimedia project that involves both librarian-guided exploration and collaborative learning processes, libraries can offer students formal and structured opportunities to explore their own interests or underlying curiosities beyond the classroom assignment and the one-hour library instruction session. introduction a recent report released by the association of american colleges and universities (aac&u), outlined the aims and outcomes of a typical twenty-first century college education. the report, representing conclusions of educators and employers, advised greater inclusiveness of students in higher education and a redesign of college learning in response to new global challenges. in addition, it described a dearth of meaningful assessments, enjoining: “student success in college cannot be documented—as it usually is—only in terms of enrollment, persistence, and degree attainment” (aac&u 4). moreover, the report defined twenty-first century learners as intentional, empowered, informed, and responsible. those characteristics also mirror the national school board association (nsba)’s descriptors of tomorrow’s successful students: “intentional architects of their own learning, setting goals, exploring, reflecting, and integrating acquired knowledge and experiences into existing worldviews” (nsba 10). it is an inherent promise of libraries to guide students in ways to help them become “intentional architects of their own learning.” yet, it is the case that the traditional classroom assignment and information skills instruction session remain the prevailing agents of education and student engagement within library walls. these familiar and prescriptive approaches have little influence on students, particularly when libraries can no longer expect students to come to them in the first place. a new model of student engagement that involves personal and meaningful discovery is needed. in meeting the challenges identified by the aac&u and the nsba, libraries can play a viable and necessary role—one that calls for a grassroots approach to cast stronger student relationships with the library. literature review the theoretical foundation of this study applies a constructivist approach, employing learner control in a social, collaborative setting. the instructional layer uses reflection as a sense-making strategy was employed to encourage self-discovery. the constructivist perspective is built largely upon the work of educational theorists vygotsky (1978) and bruner (1967) who brought to bear the understanding that the onus of learning is on the learner and occurs most often experientially, in situ, building upon the learner’s own knowledge evidence based library and information practice 2008, 3:4 36 and experience. further, vygotsky advanced the idea that knowledge is also constructed through social exchange. the theory of social interdependence, as explained by deutsch (1962), is exemplified when learners share common goals and each other’s success, which is determined largely by the quality of interactions; negative interdependence results when a group’s interactions are competitive or opposing (jonassen 7-11, 37-42). importantly, collaborative discussions allow for a social construction of meaning as students express themselves and consider the viewpoints of their peers. while the teaching of information literacy skills is not the objective of this study, the research on constructivist approaches to educating library users points mainly to active learning strategies. although these strategies offer more active roles for learners, they are often embedded in traditional pedagogy which focuses on acquiring a skill set rather than on the thinking patterns and experiences of the learner. naylor and karp summarize the evolution of information literacy instruction from resource-centered to user-centered, a transition that involved a great deal of outreach to infuse the curriculum with information literacy, to improve methods of assessment, and to craft online delivery of content (237-239). yet these advances lack the social, experiential, and metacognitive attributes inherently valuable in the learning process. one such study stressed the importance of a partnership between librarians and faculty in fostering independent, autonomous student research skills, and noted that acquiring a generic set of information literacy skills is not enough to foster autonomous learning (mcdowell 264). from a broader perspective, hensley calls for bringing curiosity and creativity back into learning; he states “ . . . fostering an individual’s sense of curiosity and creativity in tandem with developing his ability to find, locate, and evaluate information is the essence of information literacy” (35). indeed, the “why” question is inescapably individual and personal. consistent with this position is woodard’s examination of the relationship between information literacy, technology, and pedagogy. she concludes with an argument for student involvement in constructivist learning environments and encourages librarians to take on new roles to facilitate this. woodward asserts that “the best uses of educational technology and the most appropriate environments for cultivating information literacy competencies use constructivist approaches to teaching and learning” (186). finally, the results of lloyd’s study point to the benefits of a holistic, context-dependent approach to information literacy instruction, one that is embedded within “socio-cultural and physical experiences that are involved in coming to know an information environment” (2008). these studies indicate the need for a model which librarians can base an active, educationally influencing role that values students’ individual interests, perspectives, and innate curiosities. one important component in a constructivist approach is the idea of learner control. self-determination theory (sdt) underscores the importance of feelings of autonomy built upon competency (selfefficacy) and relatedness—socialenvironmental factors that sustain or enhance intrinsic motivation (ryan and deci 68-79). in social cognitive theory, perceived self-efficacy is embedded in a theory of human agency . . . a belief in one’s capability to organize and execute the courses of action required to given levels of attainments (bandura 79-81). the essential elements of sdt—autonomy, competence, and relatedness—are crucial in a successful evidence based library and information practice 2008, 3:4 37 learning experience. for instance, the uninitiated learner must be persistent and have a sense of his or her own competence in order to construe and negotiate the multiple and often complex pathways to what they seek in the library. bandura notes that “when faced with obstacles, setbacks and failures, those who doubt their abilities slacken their efforts, give up, or settle for mediocre solutions. those who have a strong belief in their capabilities redouble their effort and figure out better ways to master the challenges” (49). although learner control theory has been introduced in information literacy instruction (wang 151-6), in general there has been little change in how students are taught these skills, as students are in general expected to learn and apply them in the context of a classroom. collaborative learning is defined as “a process through which a group creates knowledge for its members, for itself as a system, and for others” (kasl, marsick, dechant 253-76). it has been proven that these environments enhance self-efficacy (moriarity 73-84) and, according to bandura, students who work together and help one another also tend to have positive selfevaluations of capability and perform better academically than those in individualistic or competitive learning environments (71-81). accordingly, the importance of hosting and fostering collaborative learning experiences is valuable to building self-efficacy. in this context, the relatedness factor of sdt may translate into the relationship that a student has with the instructor through perceived and received support related to the course. the immediate outcomes of this relationship may be the amount of creative and exploratory effort shown by a student as well as the quality of the student’s work. in fact, there is evidence that points to a positive relationship between relatedness and autonomy (ryan and deci 74). in addition, slavin reports that motivationrelated attitudes of students who participated in learning together were more intrinsically motivated than were individualistically taught students, indicating that these environments enhance self-efficacy (46-9). research evidence that shows how peer collaborations and social interactions can facilitate learning process and improve learning outcomes has been accumulating (wentzel and watkins 36671). even online learning environments can be designed to be “positive, caring, nonthreatening,” fostering the sharing of personal experiences, expressions of personal growth, and a sense of community (barab, thomas and merrill 132-5). strategies that encourage metacognitive activities, such as reflective thinking, have a key role in fostering learner control. kolb’s “learning by discovery” model highlights the importance of experience, perception, cognition, and behavior in learning as a holistic process (kolb 25-38). in this study, those key elements were supported in collaborative activities that encouraged students to make sense of their experiences by thinking reflectively as they developed their multimedia projects. for instance, the process of story development provided them with opportunities to explore and reflect on crucial points in their experiences. critical and imaginative thinking were required in order to translate ideas and evidences of their experiences into combinations of a variety of multimedia including texts, visuals, and sound. students were also asked to monitor their own progress, allocate their time, prioritize multimedia building and editing tasks, and to seek help when they needed it; all of which are crucial to success in high learnercontrol environments (schmidt and ford). in short, thoughtfully designed collaborative learning experiences can facilitate relatedness, build competence, and evidence based library and information practice 2008, 3:4 38 encourage self-directed exploration; the essential attributes for life-long learning. methodology the purpose of this study was to explore ways librarians can provide meaningful learning experiences for students beyond the traditional classroom assignment and the one-hour library instruction session. the study was done within a qualitative framework using participative, interpretive, and personal experience methods. these methods are rooted in the works of john dewey who viewed education, experience, and life as inseparable and, as explained by clandinin and connelly, “personal experience methods are inevitably relationship methods” offer “an opportunity to create a middle ground where there is a conversation among people with different life experiences.” further, these methods are understood “. . . as ways to enter into and participate with the social world in ways that allow the possibility of transformations and growth” as well as spontaneity, flexibility, and openness among investigators and participants (150-78). personal experience methods “. . . require researchers to set aside preconceptions and to become immersed in individuals’ life world in order to understand how they subjectively constitute and interpret reality” (powell 91-119). data collection points included: team meetings, student interviews, and a post-project survey. five meetings, referred to as “team meetings,” supported the collaborative learning approach. a team of two librarians and a graduate assistant acted as both investigators and instructors throughout the project. while one librarian keyed observations and conversations on a laptop, we each kept a separate field log. additionally, we all participated in actively guiding discussion and encouraging clear and open communication during the meetings. the instructional layer of the study consisted of story development with storyboarding activities; team meetings designed for explorative conversations about story ideas, and translating those into texts, photos, illustrations, video, and audio; and finally, sessions geared toward rendering a composite with common multimedia tools such as dreamweaver, flash, and adobe premier. the students had a ten-week period during the spring quarter to acquire basic multimedia and digital story development skills and to complete their projects. study participants building upon an existing program of student learning in the library, we recruited volunteers from the ohio state university libraries’ peer library tutor (plt) program. plts are paid student employees of what is a highly decentralized, complex research library system. they are trained to assist their peers in using library resources and they help staff a central reference desk. in addition, plts are given assignments each quarter that are designed for active learning; for example, interviewing a subject librarian, exploring and reporting on a special collection, or investigating the operations of a library service. it is important to note that the participants in this study attended librarian-led information literacy sessions covering principles of intellectual property and copyright. recruitment involved an initial informational meeting that was held at the end of the winter quarter. the winter break allowed students time to consider participation in the study in view of course loads and other commitments. at this meeting, students were introduced to the evidence based library and information practice 2008, 3:4 39 study’s purpose and procedures, and to the rubrics of the multimedia project. we emphatically stressed the voluntary nature of the study and the multimedia project, and their right to opt-out of both at any time. handouts detailing the study’s purpose and procedures, a list of students’ rights to anonymity and confidentiality safeguards were made available on the university’s course management system. eight students accepted the invitation. the participant group was comprised of four sophomores, two juniors, and two graduating seniors. gender was split: four female and four male students. procedures to promote collaborative learning, team meetings were held every other week. while meetings were scheduled for 90 minutes, students often lingered to chat or to continue work on their projects. a librarian’s office served as the meeting space since it had two multimedia-enhanced workstations and comfortably accommodated the group. the first 20 minutes of every team meeting were given to students to openly express concerns and frustrations and to ask questions. thirty minutes allowed coverage of one or more of the building blocks of multimedia development, these were: 1) conceptualizing ideas through concept maps and storyboards; 2) refining and articulating a message and determining audience; 3) identifying and acquiring artifacts (photos, music, sounds, and other objects); 4) converting non-digital artifacts; 5) editing or creating media; and, 6) adding interactivity and navigation. the remaining time was used for a “show-and-tell” when students shared their ideas, storyboards, artifacts, and project developments. the findings suggest this collaborative time was crucial to the overall quality of students’ experiences as it affected their learning about themselves while motivating them to stick with their projects. to foster learner control, each student would decide the message, purpose, content, and format of his or her own project. their work would not be graded nor evaluated. a showcase event was planned at the end of the quarter to leverage student motivation and to allow them to receive recognition for their work by the university community. instructional materials consisted of a storyboarding activity sheet, descriptions of available software and digital recording devices, a matrix of questions to foster reflective thinking, links to sample digital stories and portfolios, and a project development timeline used to keep projects moving forward to the final showcase event. all materials were made available on the university’s course management system. data collection as investigators we were most interested in students’ experiences during the ten-week process of developing their multimedia project—a process that necessarily implies transitions, positive or negative, that have bearing on attitudes, perceptions, and understanding of oneself or others. we made note of such changes as important indicators that a transition in a student’s thinking had occurred. as previously mentioned, there were three data collection points: team meetings, student interviews, and a post-project survey, all of which involved students recounting their experiences as they negotiated the conceptual and technical development of their multimedia story. we aimed to record the discussion and interactions as close to the experience as possible. accordingly, an audio-recording was made of each meeting in addition to the investigators’ hand-written and computerevidence based library and information practice 2008, 3:4 40 keyed logs. while the logs contained our separate accounts, we used the margins to indicate the speaker’s voice (feeling and tone), expressions, body language, and other signifying words, behaviors, or interactions. our own observations, perceptions, and questions were recorded and notationally bracketed for later analysis. to ensure student anonymity, paper slips were inscribed with a number, 1-8, and placed in a bowl; each number as drawn was assigned to a student from an alphabetical list. these numbers were referenced in place of names on all recorded and transcribed documents. investigators met before and after each team meeting. these small group discussions were recorded to maintain uniformity in procedures and to capture our discussions of events as they unfolded. further, our discussions helped to ground us in our objectives and most importantly, to position ourselves in neutrality and openness. postmeetings allowed us to debrief while students’ conversations and interactions were fresh in our minds and to check our impressions against the recorded data and notes collected. it was particularly helpful to resolve inconsistencies in this way while the relevant data was immediately at hand. student interviews and post-project survey within a few days after the showcase, each student was interviewed separately. the interviews were audio-recorded with permission. each was asked eight openended questions that invited thoughts on their experiences with the project from its beginning to the showcase event. students were also given a printed survey to take with them to complete and return by campus mail. the survey consisted of five open-ended questions which gave them an opportunity to leave anonymous feedback and to express any thoughts they may not have been comfortable expressing during the interview (appendix). data analysis an inductive analysis procedure as described by miles and huberman (1994) was applied to the four data sets. separately, each investigator performed a chronological, iterative reading of all data in order to gain an initial impression which was then followed by a closer reading of each student’s data in order to note significant attributes such as statements, patterns of thought or expression. we then read through the data sets again to note frequencies of those distinct attributes. each reading allowed us to identify salient themes, forming clusters which ranged from level of student interest, engagement, and collaboration to reflective thinking and positive self-talk. margins were used for noting the speaker by number and associated pages in each set. each investigator wrote a summary account of each student’s data, noting the theme clusters, and then met to synthesize the accounts. the aim of the synthesis process was to disambiguate the theme clusters for consistency in definition and attributes; this brought about a clarified set of the most prominent themes. we then assigned a color for each that aided in locating them within the data sets. the color coding was borrowed from a manual code-and-retrieve method described by richards and richards (214-215). data patterned into outcomes or themes of ownership (learner control), reflectivity, collaboration, and transformative thinking. findings data indicated that student behavior was aligned with the theoretical attributes of learner control and collaborative learning. all but one student demonstrated growth in accepting control and ownership but they all demonstrated growth in confidence in their ideas and the ability to express them through multimedia. the collaborative evidence based library and information practice 2008, 3:4 41 learning approach facilitated engagement in the project and each student gained a measurable degree of skill in using digital media devices and multimedia software. in addition, the value of “library as place” was evidenced in three students’ projects in which they positively characterized their library experience and its personal and educational value. we learned that the multimedia project itself was an inherent motivator. two graduating seniors chose to do digital portfolios or resumés as a way to market themselves to potential employers. one student used the opportunity to develop a multimedia website to promote his barber shop quartet and another created a short film to chronicle his life as a student. in short, the benefits of acquiring multimedia skills were obvious as the students had no trouble coming up with a purpose for their projects. in addition, we discovered that the showcase event was a significant motivator as it kept students on task while providing them the opportunity to present their work to the campus community. learner control, collaboration, and patterns of self-efficacy feelings of autonomy arising from a sense of control, one of the three previously referenced social-environmental factors (autonomy, competence, relatedness) that contribute to intrinsic motivation was indeed available through the freedom students had in choosing their project’s purpose, audience and content. initially, this proved a delicate balance for some students who struggled with that freedom and pressed us for guidance. for instance, one student explained: “i was first kind of really baffled by the project; i didn’t really know what i was going to do. at first i really focused a ton of effort on the actual website rather than on the content . . . i really didn’t know where i was going with it.” and while all the students successfully completed their projects, despite our best efforts to be supportive, one student was ambivalent from beginning to end. he expresses it this way: “ . . . one of the barriers was just motivation—it wasn’t like i was making something that i wanted to do, totally, it’s kind of like they are giving us this idea of what they wanted us to do, and then we kind of have to develop it ourselves . . .” indeed, autonomy was an important component as five of the eight students demonstrated strong and sustained investment in their projects. these same students were also more engaged in group interactions and received more support from others; as this student explains: “it really helped just to pick up a camera and start looking around . . . i found inspirations from a lot of different places.” remarkably, this student claimed she had never used a camera before. another student took responsibility for her ideas from the start and led other students in the same manner despite moments of doubt and pressure to finish her project in time for the showcase. she stated: “. . . i’m a perfectionist, and you know, [i had to] throw things out, and then i had to do things in quicker ways; but then afterwards, when we were showcasing everything and i had people coming up and asking me questions about that, i was really proud of it.” furthermore, student feedback and behavior indicated they felt a commitment to the project. this was demonstrated by their regular attendance at team meetings, sustained engagement in their projects, and trust demonstrated their in openness to giving and receiving feedback and responding to questions that prompted selfreflection. for instance, one student explains how his thinking changed over the course: “. . . at first i was approaching it like something i had to do, but then the more we did, the further i got into it . . . i started evidence based library and information practice 2008, 3:4 42 realizing the whole value of it—it was going to be something really cool.” another student stated: “. . . it was just an assignment to begin with, but it evolved into something that i could take for myself and use for personal experience, and i could also share it with others.” self-efficacy and multimedia technology two of the eight students in the study indicated that they had some prior experience in working with multimedia while most showed initial reticence; one expressed anxiety about using technology altogether. despite these potentially inhibiting factors, by the end of the project all of the students expressed satisfaction and even delight in the technological skills they had acquired; as one student expressed it: “i just think that i gained more confidence as the process developed. i feel i succeeded by being able to work with new technologies and i feel that i got a great deal of satisfaction out of this.” another stated: “i feel a lot more comfortable going up to a computer. i definitely didn’t feel like i could do any sort of basic web page or movie stuff; like that was a whole other world, and it used to be this huge, scary world like all the web pages and internet and computers, uhhh! but now it’s not scary any more, because i have done it.” and another expressed it this way: “i think the technology was the coolest part of it—just learning everything—that was my favorite part;” while learning the technology was evidently rewarding, it was also challenging for some: “it was challenging, but it was fun to learn. but, it was definitely challenging;” and another, “i was not technologically savvy . . . most of the programs i enjoyed, and just got used to them.” collaborative and experiential learning the project was initially received with mild, probably less-than-sincere enthusiasm. however, the more students discussed their ideas, we noticed a rise in general motivation and enthusiasm. the team meetings were critical in reinforcing the collaborative aspects of the project as students shared and received feedback from one another and even began to give each other support through encouragement or help with visualizing or editing. one student expressed it this way: “. . . talking it through with people was one of the things that helped—having that teamwork with everybody was one of the things that helped me the most to be successful in the project.” another said: “those [team meetings] helped a lot—just kind of seeing what everybody was doing helped me focus mine a lot—just bouncing ideas off of each other.” when asked in the interview what was most helpful, one student stated: “it helped to bounce ideas off of each other, to talk about it, to do the little things like, to share what we’re doing and to express the doubts that we have—questioning whether we would make something interesting.” the students’ narratives as well as their observed performance suggested that a collaborative learning environment contributed to their self-efficacy and even encouraged achievement. for example, the student who had earlier expressed ambivalence, later said: “. . . we were all supportive of each other, so that helped you complete it. one wanted it to be good because you know everybody else is doing a good job, putting a lot into it, so you want to make sure you did too.” indeed, the more salient findings point to enhanced learning experiences and selfefficacy due to the collaborative learning environment. in this context, the relatedness factor of sdt may be represented by the relationships among the students and between the students and the investigators, and the perceived and actually received support (from us and their peer group). the evidence based library and information practice 2008, 3:4 43 immediate outcomes associated with this factor may be the amount of time invested or effort made by a student, as well as the quality of the student’s work. indeed, six of the eight students emphasized the importance of the group interaction and support to their learning. as this student expresses it: “all the meetings definitely helped—talking it through with people— having that teamwork was one of the things that helped me be successful in the project.” another said, “if there was an aspect that i didn’t understand or something that needed clarified, hearing others’ input definitely helped a lot.” another added: “at meetings we could just bounce ideas off each other— just us talking and somebody else showing what they were planning, then somebody else going up to the board—just everybody talking about what they’re doing—that helped a ton.” another expressed it this way: “we joked around and if one of us knew or didn’t know how to do something we could ask each other.” finally, as one student poignantly summarized it: “just seeing everybody else’s portfolio was—we were all excited and we’d ask ‘so what’s yours?’” we also witnessed reciprocal benefits as students began to seek each other out, many meeting in small groups of two or three: “i’m not a visual person so i couldn’t draw, and i couldn’t really show what i was trying to say. she’s [student a] more of an artistic person; she was able to draw what i was able to say.” student a said in reply to student b: “i had a lot of feedback from her [student a]. she would look over my page and proofread it.” another example: “she’d [student c] help me if i couldn’t figure out the color or she really helped me figure out when i first had an idea, like i told her what i was thinking and she helped me ‘okay. this is what you’re saying, let’s get it down on paper.’” the role of reflection of the eight students, four indicated they had experienced some level of change in their thinking about some aspect of the project or themselves: “it was more about the learning experience than the final product. you know, i don’t think that we all would have learned as much had you just said, ‘okay, this is what we want at the end.’ i don’t think all of us would have gotten this much out of it. what i came up with i was really proud of and the further i got into it, i started realizing the whole value of it.” another student who did a short video documentary of his senior year stated, “i’ve gained a great sense of accomplishment from this project, and i’m grateful for the positive changes it’s brought to my life.” student narratives also evidenced positive self-talk (personal explanation of events). in fact, there was a commonality among those students who evidenced self-talk, and the overall quality of their projects. for example, one student, a senior, was facing personal setbacks during the project. he explained in his interview: “for awhile my confidence was not quite there in terms of my abilities to do it, but it forced me to look at and reflect upon my life and you know, my current situation, but also to think more about what themes in my life made this process important.” in essence, this student used the project as a focus point and creative outlet. similarly, for many others it was the process of developing their story that brought about changes in their thinking. here are a few more examples that show how self-talk helped students change negative attitudes: “there were times when i thought, ‘i just don’t want to do this stuff but it’s going to get better, it’s going to be marketable—such a good thing for you when you’re done.’ so that was one thing evidence based library and information practice 2008, 3:4 44 that kind of kept me through.” another example: “i was like, you know, there was just too much stuff going on, i needed to focus on school, ‘but these skills are going to help me in the end—just keep going—it’s going to turn out how you want it to—you are going to like your end product and you are going to be happy with it, just keep going through.’” struggling with self-doubt, another student revealed his personal mantra: “you can do it; you’ve done it before, so why can’t you do it again?” the interviews after the showcase evidenced how some students’ thinking had changed during the project. for example, early on this student described her thoughts about the project this way: “this is kind of a random assignment—i felt it kind of defeated the purpose of our job—i didn’t really see how it tied in with the library . . . .” in her interview after the showcase, however, she states: “i gained confidence in myself; i knew there was something to figure out. to pinpoint what it was, was nice.” the process of gathering evidence of one’s life invoked reflection for this student: “before, i didn’t think much about where everything was taking me, and then during, as you were gathering information on your life, you kind of realize what you’re doing and where you’re going.” one student used self-talk to change her mind about getting technical help from her peers: “it made it so much better when i actually use what was available to me instead of just sitting back and ‘oh, i don’t know what to do. well, these people can help you, go ask for it.’” discussion the purpose of this study was to explore new ways librarians can provide meaningful learning experiences for students beyond the traditional classroom assignment and the one-hour library instruction session. more notable were outcomes that suggest new ways librarians might engage students in the library with multimedia assignments and in hosting and facilitating collaborative learning projects. a dedicated space in the library was important as it gave students a place to meet, openly interact with each other and librarians, and also to work on their multimedia projects. it was also a place where they could collaborate without disturbing students who were studying. although not all of the quotes suggest direct ties between their learning and the library environment, the fact that each student benefited from individual and collaborative team work suggests a promising view of the library as a space that nurtures collaborative learning. it is noteworthy that participants in this study were not divided into small groups by the librarians; instead, they formed their own groups and found learning partners among themselves. when a space affords such group activity, students can adapt their behavior to social collaborative learning more easily. also important was the dynamic relationship of the librarians as facilitators rather than instructors to the students. it was the guidance from the librarians as well as a tailored level of learner control that allowed the students to have such positive experiences in the project. lastly, the opportunity to learn multimedia technologies and apply them in personally meaningful ways reveals new opportunities and formats for student learning in the library. there are important limitations in this study that should be acknowledged. foremost, we believe the fact that the students knew each other prior to the project had an effect on the outcome. another potentially limiting factor was the students’ prior knowledge and experience in the library and the fact that they were already identified as motivated evidence based library and information practice 2008, 3:4 45 learners through the plt program. had students been randomly recruited from a diverse, more representative group of undergraduates the outcomes would likely have been more mixed. implications overall, the study evidences a student learning experience that was both positive and meaningful. apart from tools for developing multimedia projects, the library’s resources, space, and infrastructure proved to inspire students’ creativity. librarian-led instruction covering principles of intellectual property, copyright, and the use of information resources, including digital images, were all made available. in fact, three of the eight students chose their library experience as the focus of their multimedia project. these same projects were later used as communication and promotional tools for the library. one of the students conveyed the following: “i wanted to focus on the library and i wanted to make it come off as very alive and moving because people tend to think that it’s boring in the library, and i just wanted to show everyone that it isn’t.” further, working with multimedia technologies requires a rubric of literacies (critical perception, listening, thinking, and writing) and exciting new formats for students to demonstrate learning and articulate their experiences and knowledge of the library. a space within libraries for multimedia development promises a means to integrate new ways of learning with library’s knowledge resources and information technologies. as a teaching and learning resource, multimedia formats provide intersections for libraries to engage with both students and faculty, inside and outside the classroom, promoting a more flexible and a more tangibly relevant library. this is an intersection of student learning in libraries that merits further exploration. the findings also suggest that the concept of collaborative learning within the library holds meaningful potential, one where guidance from librarians can ensure that learning experiences involve principles of information literacy, copyright and intellectual property, and importantly, the mixing of reliable traditional and nontraditional knowledge resources with computer and multimedia technologies. investigators set out to better understand this premise by creating an assignment that would enable to students to use library resources to create a story or message about their own learning. the overall positive findings indicate the power of authentic learning experiences, where students are guided by their own curiosities yet supported in the process of intellectual and personal discovery—the very promise of libraries. recommendations for further study include more exploration into multimedia learning as a library service. other questions that were raised during the course of the study include: in what ways does the library provide a different kind of multimedia learning experience than a computer lab? in what ways can librarians use their subject knowledge to help students working on multimedia assignments? how might library instruction programs blend information and multimedia technologies for effective learning? could we learn something about how students experience the library through their personal explanations of events? evidence based library and information practice 2008, 3:4 46 works cited association of american colleges and universities (aac&u). college learning for the new global century. washington, dc: national leadership council for liberal education & america’s promise, 2007. 13 october 2008 <http://www.aacu.org/leap/docume nts/globalcentury_final.pdf>. bandura, albert. social learning theory. englewood cliffs: prentice-hall, 1977. barab, sasha a., thomas, michael k., and merrill, henry “online learning: from information dissemination to fostering collaboration.” journal of interactive learning research, 12.1 (2001): 105-43. clandinin, jean, d. and f. michael connelly. “personal experience methods.” collecting and interpreting qualitative materials. ed. norman k. denzin and yvonna s. lincoln. thousand oaks: sage, 1998. 150-78. deutsch, morton. “cooperation and trust: some theoretical notes.” nebraska symposium on motivation. ed. m.r. jones. lincoln: university of nebraska press, 1962. 275-310. hensley, randy burke. "curiosity and creativity as attributes of information literacy." reference and user services quarterly 44.2 (fall 2004): 31-5. jonassen, david h., ed. “the nature of cooperative learning” handbook of research on educational communications and technology. 2nd ed. the association for educational communications and technology. mahwah: lawrence erlbaum, 2004. 786-9. jonassen, david h., kyle l. peck, and brent g. wilson. learning with technology: a constructivist perspective. upper saddle river: prentice hall, 1999. kolb, david a. experiential learning: experience as the source of learning and development. englewood cliffs: prentice-hall, 1984. kasl, elizabeth, victoria marsick, and kathleen dechant. “teams as learners: a research based model of team learning.” strategic learning in a knowledge economy: collective, and organizational learning process. ed. rob cross. boston: butterworthheinemann, 2000. 253-276. moriarity, beverley, et al. “the importance of self-efficacy as a mediating variable between learning environments and achievement,” british journal of educational psychology 65 (1995): 73-84. lloyd, annemaree. “recasting information literacy as socio-cultural practice: implications for library and information science researchers.” information research 12.4 (october 2007): 13 october 2008 <http://informationr.net/ir/124/colis/colis34.html>. mcdowell, liz. “electronic information resources in undergraduate education: an exploratory study of opportunities for student learning and independence.” british journal of educational technology 33.3 (june 2002): 255-66. evidence based library and information practice 2008, 3:4 47 miles, matthew b. and a. michael huberman. qualitative data analysis: an expanded sourcebook. 2nd edition. newbury park: sage, 1994. national school boards association (nsba). creating and connecting: research and guidelines on online social and educational networking. alexandria: nsba, july 2007. 13 october 2008 <http://www.nsba.org/secondaryme nu/tln/creatingandconnecting.as px>. naylor, susan and rashelle karp. “the academic library: issues and challenges for the future.” defining relevancy: managing the new academic library. ed. janet mcneil hurlbert. westport: libraries unlimited, 2007. 237-8. powell, ronald r. “recent trends in research: a methodological essay.” library and information science research 21 (1999): 91-119. richards, thomas j. and lyn richards. "using computers in qualitative research." collecting and interpreting qualitative materials. eds. norman k. denzin and yvonna s. lincoln. thousand oaks: sage publications, 1998. 214-5. ryan, richard m. and edward l. deci. “self-determination theory and the facilitation of intrinsic motivation, social development, and wellbeing.” american psychologist 55 (2000): 68-78. schmidt, a. m. and k. ford. “learning within a learner control training environment: the interactive effects of goal orientation and metacognitive instruction on learning outcomes.” personnel psychology 56 (2003): 405-29. wang, li. “sociocultural learning theories and information literacy teaching activities in higher education,” reference & user services quarterly 47 (winter 2007): 149-158. wentzel, kathryn r. and watkins, deborah e. “peer relationships and collaborative learning as contexts for academic enablers.” school psychology review 31.3 (2002): 36677. vygotsky, leanord s. mind in society: the development of higher psychological processes. cambridge, ma: harvard university press, 1978. evidence based library and information practice 2008, 3:4 48 appendix interview face sheet date: interview no: time: end time: (guiding phrases): think back . . . describe . . . characterize . . . tell me about . . . in what ways . . . what was/is . . . how . . . what do you think . . . how do you feel . . . describe your story or what you wanted to convey with your portfolio. . . . has your story changed since the beginning of the project? how . . . can you identify any changes in your thinking about yourself before and during the creation of your portfolio, compared to now . . . . what aspects of the d-story development process helped you . . . can you tell me about a particular incident that was particularly helpful to you . . . what were the difficulties or barriers in your experience . . . what could have helped, assuming the help was possible? did you discover anything about yourself that was particularly surprising . . . what about discoveries about others . . . if you had this project to do over again, what would you do differently . . . what would you want us (co-investigators) to do differently . . . do you have any questions for me? evidence based library and information practice 2012, 7.1 114 evidence based library and information practice evidence summary public libraries in norway help non-western immigrant women to integrate into society a review of: audunson, r., essmat, s., & aabø, s. (2011). public libraries: a meeting place for immigrant women? library & information science research, 33(3), 220-227. doi: 10.1016/j.lisr.2011.01.003 reviewed by: kathryn oxborrow team leader hutt city libraries lower hutt, new zealand email: kathryn.oxborrow@huttcity.govt.nz received: 4 dec. 2011 accepted: 22 feb. 2012 © 2012 oxborrow. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objectives – to discover the ways in which the public library was used by immigrant women, with a particular focus on the library as a meeting place. design – semi-structured qualitative interviews conducted in the participants’ native languages. setting – public libraries in norway. participants lived in one of two cities both with a population of approximately 40,000 and a somewhat lower number of immigrants than the national average. subjects – nine non-western women who had immigrated to norway between 8 months and 17 years prior to the study. three women were from iran, kurdistan and afghanistan respectively. all identified themselves as public library users. methods – participants were interviewed in their native languages and the qualitative results were analyzed in accordance with the theoretical framework set out by the authors. the main areas of focus were the role of the library in the generation of social capital, and the library as a high intensive versus low intensive meeting place. main results – participants used public libraries in various ways. in the initial stages of life in a new country they were used to observe and learn about the majority culture and language. they were also used as a safe place to openly grieve and provide comfort among close mailto:kathryn.oxborrow@huttcity.govt.nz� http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/� evidence based library and information practice 2012, 7.1 115 friends without fear of being seen by other fellow countrymen. over time, participants came to use the library space in more traditional ways such as for information, social, and professional needs. the study also revealed that using public libraries built trust in the institution of libraries and librarians as employees. conclusions – the public library plays a key role in the generation of social capital, both in terms of integrating into the majority culture through observation and spontaneous interactions (bridging social capital) and connecting with others from participants’ home cultures (bonding social capital) for example through the provision of social space and other links to home such as native language materials. the public library was used initially by participants as a high intensive meeting place, for meeting with friends and consoling one another. in time, however, the public library became a mediumto low-intensive meeting place, with participants engaging in library programmes or in spontaneous interactions with other library customers. moreover, the study refutes the swedish library association’s conclusion that library is used more often by immigrants that have lived in the country for long periods of time. commentary this study was conducted as part of the place project (public libraries, arenas for citizenship), a four year project which ends in april 2012. the place project focuses on the role of public libraries in the generation of social capital and cross-cultural communication in norway. each of the three authors is involved with the wider project; audunson is leading the project, aabø is part of the project team, and essmat’s master’s thesis formed the basis for this article. one of the authors is fluent in the three languages spoken by the women in this study (farsi, dari and kurdish) and thus participants were interviewed in their native language. this adds greatly to the study, since it prevented any linguistic barriers which may have made it difficult for the women to complete the interview. the study is well-organized, with detailed explanations of the theoretical framework that includes social capital as well as highand lowintensive meeting places. the study’s procedures are clearly explained, from the selection of participants to the structure of the interviews. the description of the data analysis could have been more detailed, however. the main themes are given with illustrative quotes, but few references are made to the number of participants who made certain comments. the article would be more accessible for readers if appendices had been included, such as the interview questions used, a matrix showing the theoretical framework, and a taxonomy of participant quotes which had elicited the themes described. inclusion of this additional information would have strengthened an already strong study. another aspect omitted by the authors was information as to whether any steps were taken to eliminate interview bias. this is especially important in the context of the long-running place study, which all of the authors are involved in, and previous findings could have affected the interviewer’s expectations of what the findings might be. there were nine participants in total, three each from iran, afghanistan and kurdistan. the authors mention the difficulty of recruiting a larger group of participants, and in light of the fact that this was an exploratory study, the number of participants appears to be satisfactory. the authors also mention the intention to follow-up with quantitative and qualitative research on a larger sample in an area with a higher percentage of immigrants in a future study. however, the authors do not provide ideas or suggestions for successfully engaging a much larger sample. an interesting angle for future work would be to contrast library users with non-library users to find out what factors encourage or discourage immigrant women from using the public library. other possibilities include investigating the effect of public librarians intentionally becoming more aware of the cultures they serve, and a more in-depth study of the possibilities of librarians helping immigrant women to find health information. / evidence based library and information practice evidence based library and information practice evidence based library and information practice 2011, 6.4 28 evidence based library and information practice news/announcements an international association of eblip, redux virginia wilson liaison librarian, murray library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca received: 27 oct. 2011 accepted: 27 oct. 2011 2011 wilson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. at the 6th international evidence based library and information practice conference (eblip6) in manchester, uk in june 2011, a meeting to facilitate further discussion about the formation of an international eblip association was held. a similar meeting was also held in stockholm at eblip5 in 2009, at which time the discussion centered on the potential vulnerability of the eblip conference series. propelled by enthusiastic organizers from several countries on an ad hoc basis, and avid conference participants from even more countries, the biennial conference has essentially driven itself. but, as andrew booth (2009) described in his commentary, “is now the time for an international association of eblip?”: . . . the vulnerability of this model can be simply illustrated in a brief sequence of “what if’s”: what if a future local organising committee decides unilaterally to cancel a conference? what if there is a difference in opinion on the conference ‘vision’ between future local organising and international programme committees? what if a future local organising committee decides to double the conference registration fees? (p. 68‐69) what emerged from the eblip5 meeting in stockholm was a new way to determine who would organize the subsequent conference. this model was implemented for eblip6: applications were accepted from locations, and a committee made up of international eblip figures and representatives from prior conferences decided on where the next conference would be held. this model is in place again for eblip7, and the conference destination will be announced in the new year. there is agreement that an international organization is needed. the eblip movement has matured to a point where a more structured type of organization would be welcome in order to see that the conference is mailto:virginia.wilson@usask.ca� evidence based library and information practice 2011, 6.4 29 reliably supported; that some support be given to the international open access journal, evidence based library and information practice; and that future programs and initiatives can be explored to help support those interested in eblip. however, while this desire seems to have been the case for the past few years, no forward movement has happened thus far to create an international association. it makes sense, really. who will do it? which country will the association be incorporated in? who will decide on what should be done? at the meeting in manchester (eblip6), which included participants from canada, sweden, england, wales, australia, and india, different models of organization were discussed. after a general discussion about the various types of organizations out there and the ways in which they operate, the group decided that volunteers from canada, the uk, and australia would investigate what would be needed to establish a non‐profit status in those countries and how costly it would be to incorporate as an association. the target to reach a decision regarding where to register/incorporate a new association is set for the end of 2011. at this point, the decision to move forward with an international association for eblip has been made. the international association will consist of some kind of governance structure, incorporation in a particular country, an arrangement that sees a portion of the international eblip conference registration go towards the association (rather than annual membership fees), and responsibility for the high level organization of the international eblip conference (choosing the venue from among applicants, serving as a home for the international programme committee, etc.). now it is time to work out the logistics. comments, questions, concerns, and conversations are encouraged and welcomed. register with this journal (evidence based library and information practice) as a reader and attach comments to this document. references booth, a. (2009). is now the time for an international association of eblip? evidence based library and information practice, 4(3), 68‐71. retrieved 2 nov. 2011 from http://ejournals.library.ualberta.ca/ind ex.php/eblip/article/view/6467/5534. / evidence based library and information practice evidence based library and information practice evidence based library and information practice 2011, 6.4 143 evidence based library and information practice evidence summary new, old, indifferent: the united kingdom’s preferences regarding the architecture and design of public libraries a review of: black, a. (2011). ‘we don’t do public libraries like we used to’: attitudes to public library buildings in the uk at the start of the 21st century. journal of librarianship and information science, 43(1), 3045. reviewed by: annie hughes information services librarian wilson dental library, university of southern california los angeles, california, united states of america email: amhughes@usc.edu received: 31 aug. 2011 accepted: 02 nov. 2011 2011 hughes. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – analysis and discussion of attitudes of u.k. citizens toward the architecture and design of the country’s public libraries. design – content analysis of essays submitted to the u.k. mass observation archive (moa). setting – citizens of the united kingdom. subjects – 180 respondents from a panel of 500 mass observation archive (moa) volunteers. methods – the moa originated in the 1930s as a way to gather qualitative evidence regarding everyday life of the british public. most of the data gathered takes the form of variable length essays written by a panel of 500 anonymous volunteers. the volunteers respond to specific directives, and in this article, black summarized responses to a directive he originally posed to the 500 volunteers in 2005: ‘public library buildings’. black issued this particular directive to the panel of volunteers in the autumn of 2005 and results were made available to the public by mid-2006.the moa received a total of 180 responses, of which 121 were from women and 59 were from men. both users of libraries and non-users were included in the sample. the respondents were not a representative sample of the british public because men, ethnic minorities, lower socio-economic mailto:amhughes@usc.edu� evidence based library and information practice 2011, 6.4 144 groups, and those living outside of the south of england were underrepresented. the author analyzed the content of the 180 submitted essays to gain insight on attitudes regarding public library design and architecture. respondents were asked about public library location, environmental fit, architectural style, sensory aspects of the building, and whether or not the building resembled other types of public buildings. although he posed several questions, black focused on answers to three questions: what do you think about the design of modern library buildings? do you prefer them to older style buildings? have you seen older libraries renovated into more modern libraries, and what do you think of them? black then analyzed the responses and grouped them into four major attitudes toward the architecture and design of public libraries. the author chose not to code any of the responses and instead chose to analyze the ‘discourse’ in and not necessarily the ‘content’ of the essays. after analyzing the discourse, black contextualized the evidence he discovered. he then discussed political and cultural issues with relation to the four major attitudes and how these issues affected the current landscape of libraries. main results – the four major categories derived from the essays that black analyzed included: preference for the new; preference for the old; preference for a mixing of the old and the new; architectural indifference, the library as ‘place’ and the concept of ‘libraryness.’ those with a preference for the new preferred the newer, more modern building because it fit better within the world of information technology. these respondents also felt that the older buildings were too intimidating and cold. those who preferred the older architecture and design felt that the buildings allowed them to access a piece of the past, and they thought an older library to be more impressive, historic, and generally have more elaborate and interesting architecture. these respondents pointed out the fewer places to hide in new libraries, and indicated that new architecture is boring and stolid. the third group of respondents preferred an older exterior, but an up-to-date interior with a more modern infrastructure. they enjoyed the large impressive buildings but liked the interior to contain comfortable, modern furniture, good lighting, as well as updated technological tools. finally, the remaining group of respondents did not place importance on the physical space of a library, but more so the services and collections within the physical space. conclusion – the discourse derived from the moa and analyzed in black’s article summarizes the attitudes and preferences that citizens of the uk have regarding public library architecture. among the 180 responses to the ‘public library buildings’ directive, there is a clear tension in these attitudes and preferences. the information gathered in the moa directive on public libraries could also provide political and cultural leaders with evidence of a need for renewal or rethinking of the country’s public libraries. commentary overall, this article is an interesting commentary on preferences for space and design regarding the united kingdom’s public libraries. the author analyzed the data available via the essay responses and came away with four opinions on library space. he not only addressed preferences but contextualized preferences by providing examples of libraries exhibiting the new, old, and a mix of new and old. the author has clearly done extensive research on public libraries as physical spaces in the united kingdom. the author recognizes methodological issues with the moa, and in a previous article (black, 2002) he describes the issues in greater detail. the lack of wide gender, ethnic, and geographic representation is a definite methodology problem in the moa. sample size was also relatively small with a pool of 500 potential participants and only 180 respondents to this particular directive. evidence based library and information practice 2011, 6.4 145 the article lacked quantitative evidence, as the author did not include percentage of responses which fell into each attitude category. providing such statistical data could be useful to those who may decide to refurbish or redesign a public library in the uk or to those interested in researching public library usage and architectural preferences. while the qualitative data gathered via the moa implies certain themes, the research does not lead to a traditional qualitative assessment of public library usage. surveys of library users and non-users throughout the uk intended to explore attitudes, expectations and preferences with regard to the public library would be more useful to government officials or those researching economic feasibility of redesigning space. references black, a., & crann, m. (2002). in the public eye: a mass observation of the public library. journal of librarianship and information science, 34(145), 145-157. / evidence based library and information practice microsoft word es_1781 evidence based library and information practice 2008, 3:3 68 evidence based library and information practice evidence summary half of vermont academic and public libraries lack written confidentiality policies despite directors’ support for principles of patron confidentiality a review of: magi, trina j., “the gap between theory and practice: a study of the prevalence and strength of patron confidentiality policies in public and academic libraries.” library & information science research 29 (2007): 455-70. reviewed by: stephanie walker acting chief librarian brooklyn college – city university of new york brooklyn, ny, united states of america email: swalker@brooklyn.cuny.edu received: 11 december 2006 accepted: 11 january 2007 © 2008 walker. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objectives – to examine the prevalence and strength of patron confidentiality policies and practices at academic and public libraries in vermont; to examine variances in policies by type and size of library and by qualifications of library directors; and to examine the level of support among vermont library directors for strengthening state law to clarify that personally identifiable information about library patrons is confidential and should only be released with a court order. design – survey (non-randomized) of 213 library directors. setting – academic and public libraries in vermont (usa). subjects – the initial audience was defined to be the directors of the 213 academic and public libraries in vermont (as the persons to whom the survey was directed). ultimately, results from 149 returned surveys were used as a basis for analysis. in addition, a written report of the information gathered was provided to the governing bodies of the vermont department of libraries and the vermont library evidence based library and information practice 2008, 3:3 69 association. the author also anticipates that the results of the survey will be of general interest to academic and public librarians. methods – a literature review of previous research on library policies and practices regarding patron confidentiality was completed. as well, the author examined the various state and federal laws and library association codes of ethics which address issues of the confidentiality of library patron data. a paper questionnaire was mailed on january 11, 2006, to directors of 188 municipal and incorporated public libraries (including 11 combined public and school libraries) and 25 college and university libraries, for a total of 213 libraries targeted. cover letters explaining the purpose of the study and assuring respondents of the confidentiality of their responses were included in the mailing. the contact information came from a list supplied by the vermont department of libraries. the questionnaire had previously been tested by five librarians from the aforementioned list, and slight changes had been made to clarify questions and instructions. survey respondents were given an initial deadline of february 1, 2006, but responses were received and included through march 13, 2006. various tactics were used to boost response rates, including electronic discussion list announcements, usage of pre-addressed stamped return envelopes, usage of personalized address information and a personal signature by the researcher on the cover letter, reminder postcards, and a coupon for free ice cream. one hundred and fifty-one questionnaires were returned. two questionnaires were returned incomplete, and were removed from both the target population and the responses, leaving a total of 149 out of 211 possible responses and a response rate of 71%. for this response rate, all calculations were determined to have a margin of error of +/-4.4% with 95% confidence. the survey was constructed to address five research questions, as detailed below: 1) how many requests for personally identifiable patron information have libraries received in the past year, and from whom did those inquiries come? 2) how many libraries have a written confidentiality policy, and what are the characteristics of libraries and library directors that have policies? 3) how current are those policies? 4) how many policies prohibit release of patron information without a court order or other binding legal document (e.g., subpoena or search warrant)? how many policies call for attorney review of any such court orders or other binding legal documents? 5) do librarians support the idea of strengthening the state law relating to confidentiality of library records? the questions were closed-ended in order to allow responses to be coded and easily analyzed, although space was provided for respondents to offer additional comments. though a deadline of february 1, 2006 was set, responses were received as late as march 13, 2006. person’s chi-square and fisher’s exact test were used to measure statistical significance of differences among various respondent groups. main results – the survey respondents estimated that they received a minimum of 1,228 requests for patron information within the past year, and 46% of libraries had received at least one such request. academic libraries, which made up 13% of the sample, received 11% of requests. these queries included requests for personally identifiable information about patron reading interests, use of library materials and use of services. it was not possible to accurately quantify requests received because many respondents indicated that their library had evidence based library and information practice 2008, 3:3 70 received numerous requests, and did not supply specific numbers; in such cases, there was generally no indication that the library kept track of such requests. (and perhaps did not themselves track such requests). of interest was the finding that because of gag order provisions in laws such as the patriot act, which prohibit librarians from reporting that such requests for information have even been made, it is likely that requests made by law enforcement agencies are under-reported. however, according to the responses to the survey, 90% of total inquiries came not from law enforcement officials, but from parents, guardians, spouses, or partners of library patrons or from teachers or professors of library patrons. in academic libraries, 60% of requests were received from teachers or professors of patrons. only 1% of total requests were reported as coming from law enforcement officials. additional groups making requests included college or university administrators (1% of total inquiries, exclusively in academic libraries); persons representing other government agencies but not law enforcement (0.3% of total inquiries, exclusively in public libraries); persons representing companies or non-governmental organizations (0.08%, one inquiry only, in a public library); and ‘others’ (6% of total inquiries, all but one of which were at public libraries). ‘others’ included the president of a friends group, caregivers of elderly or disabled or homebound patrons, friends or acquaintances of patrons, book group participants, and people picking up reserved materials. despite heightened awareness of confidentiality issues (due to the patriot act and related heavily publicized legal actions), only 48% of survey respondents indicated that their library had written policies or procedures detailing how requests for personally identifiable information should be addressed. fifty-six per cent of academic libraries had such policies, as compared to 47% of public libraries. thirty-five per cent of libraries without such policies indicated that they were working on developing these policies or planned to create them within a year. as well, 64% of libraries whose directors hold mls degrees or the vermont department of libraries’ certificate of public librarianship had confidentiality policies, as compared to 38% of libraries where the director held neither qualification. data were also analyzed according to the number of years of experience that the library director had, but results obtained from this comparative analysis were not statistically significant. thirty-six per cent of libraries with two or fewer paid personnel had written confidentiality policies, as compared to 41% of libraries with 3-5 paid personnel, 59% of libraries with 6-10 paid personnel, and 95% of libraries with 11 or more paid personnel. the policies tend to be up-to-date: of libraries with confidentiality policies, 46% had reviewed or updated their policy within the past year, and 86% had done so within the past three years. of libraries with confidentiality policies, 86% require a court order or other binding legal document, such as a subpoena or search warrant, before they will release patron information. there were no statistically significant differences in this matter when data were analyzed according to library type, number of paid personnel, or years of experience of the library director or the number of paid library personnel, but there were differences when data were analyzed according to whether the library director held an mls degree or vermont department of libraries’ certificate of public librarianship, or neither qualification. in libraries where the director held one of the abovementioned qualifications, 93% required presentation of a binding legal document before permitting release of patron information; in libraries evidence based library and information practice 2008, 3:3 71 where the director held neither qualification, this fell to 58%. fifty-two per cent of libraries with confidentiality policies required that an attorney review the binding legal documents requesting patron information before such information could be released. libraries which were more likely to include such a provision in their policies included libraries in which the director held either an mls degree or the vermont department of libraries’ certificate of public librarianship (62%, versus only 8% of libraries where the director holds neither qualification), and libraries with larger numbers of paid personnel (20% of libraries with 2 or fewer personnel had provisions requiring attorney review of binding legal documents, versus 79% of libraries with 3-5 paid personnel, 50% of libraries with 6-10 paid personnel, and 61% of libraries with 11 or more paid personnel). finally, 74% of respondents supported strengthening vermont law to enhance safeguards to confidentiality of library records. twenty-five per cent stated that they would need more information before making a decision, and 1% stated that they would not support such changes. there was a statistically significant difference by library type: 83% of academic library directors supported strengthening the law as described above, as compared to 73% of public library directors. the article also suggested that education and support should be directed at those libraries which have no confidentiality policy. conclusion – in 2005, nearly half of vermont academic and public library directors received at least one request for personally identifiable information about patrons’ use of library resources or services, and in total, at least 1,228 such requests were received. it is difficult to be sure about either the number or prevalence of such requests, as some laws, such as the patriot act, prohibit libraries from revealing that such inquiries have been made. however, only 1% of reported requests for such information were made by law enforcement agencies; it is far more common for requests to come from parents, guardians, spouses, partners, teachers, or professors of library patrons. despite the number and prevalence of requests received, and despite heightened awareness of patron confidentiality issues due to recent publicity on the topic of legal action taken by libraries as a result of requests for information made under the patriot act, fewer than half of the libraries studied have written policies or procedures addressing how such requests should be handled. of libraries that do have such policies, a large majority (86%) require presentation of a binding legal document before they will release patron information, and over half (52%) require attorney review of the binding legal document before they will comply with the request. a large majority (74%) of vermont library directors also support strengthening state law to protect confidentiality of library records. commentary the survey response rate was quite strong, at 71%. the author notes that “a high response rate is important in minimizing non-response bias in the survey results. non-response bias occurs when the opinions of the people who respond are significantly different from the opinions of those who do not. … [several] research methods textbooks suggest that researchers should strive to achieve response rates of at least 50%, 60%, or 75%.” (462). by these criteria, the author has rather successfully dealt with non-response bias; the response rate is under 75% but exceeds other stated acceptable rates. the sample size, however, is small and localized, and it would be very useful to repeat the survey in other states or evidence based library and information practice 2008, 3:3 72 even nationally, and then to compare results. another possible area for further research is how requests for patron information are handled in libraries where no confidentiality policy exists. it would also be interesting to further explore the correlation that was found between the likelihood of a library having a written confidentiality policy and the library director holding an mls degree. the author speculates that this correlation could have something to do with the fact that mls programs may have included information about the importance of confidentiality, or alternatively that libraries which have gone to the trouble of codifying their practices in this area may be more inclined to hire directors with mls degrees. the survey did not explore this, and it could present an additional avenue for further research. likely understatement of the total number of requests for patron information due to the usage by respondents of words like “many” or “countless times” rather than specific numbers is of some concern. it would be preferable, of course, to have precise numbers. it is unclear whether library directors were asked if they formally keep track of exact numbers of requests for patron information. there is also possible inherent (and unavoidable) inaccuracy in numbers of requests reported as coming from law enforcement agencies; under the federal patriot act, such requests cannot be reported. nonetheless, the research presented in this article is likely to be valuable to a number of audiences. the vermont department of libraries and the vermont library association can use the information gathered in designing programs to better serve their constituents. other library associations and state or federal library organizations can use this information in a similar fashion. the information presented (including both the overview of existing research and relevant federal and state legislation, and the summary of this research) is also likely to be useful to library directors or other librarians charged with examining patron confidentiality practices or developing patron confidentiality policies at their libraries. it should be noted that while this study is a us-based study, and perhaps has specific resonance for us-based libraries due to the patriot act, the article will be of interest to libraries in other regions. libraries around the world deal with confidentiality issues in one way or another. research article   use, perceptions, and awareness of libguides among undergraduate and graduate health professions students   john carey head, health professions library hunter college, city university of new york new york, new york, united states of america email: john.carey@hunter.cuny.edu   ajatshatru pathak health and informatics librarian hunter college, city university of new york new york, new york, united states of america email: apathak@hunter.cuny.edu   sarah c. johnson social sciences librarian hunter college, city university of new york new york, new york, united states of america email: sjo0034@hunter.cuny.edu   received: 15 sept. 2019                                                             accepted: 8 june 2020      2020 carey, pathak, and johnson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29653     abstract   objective – this study investigated usage, perceptions, and awareness of library research guides created using springshare’s libguides among undergraduate and graduate health professions students.   methods – the researchers recruited 100 health professions students in april 2017 from hunter college, a senior college within the city university of new york system. participants were asked to complete a paper survey to ascertain their use, perceptions, and awareness of springhare’s libguides.   results – nearly two-thirds of study participants were not aware of library-created libguides and 68% had never used this tool. compared to undergraduates, graduate students were more likely to be aware of libguides. the use of libguides was higher among graduate respondents (43%) than their undergraduate counterparts (30%). the study found low awareness and use of libguides among health professions students overall, regardless of age, gender, academic level, and health sciences concentration. physical therapy students were more likely to use and be familiar with libguides than nursing, medical laboratory sciences, and speech-language pathology and audiology students. participants reported using general subject guides more than course-specific guides, and the most commonly used page was the databases guide. of those participants who had used libguides, the vast majority (97%) said they found them useful in their studies.   conclusion – this study demonstrates low usage and awareness of libguides among health professions students at a large urban public college. findings suggest a need for academic libraries serving such students to develop and implement strategies to promote awareness and increase usage of online research guides. the researchers recommend instructing with libguides during information literacy sessions and demonstrating their usefulness during reference consultations. additional strategies include linking libguides to course sites through learning management systems such as blackboard and collaborating with faculty members to better inform students about the guides.     introduction   libguides is a cloud-based tool offered by the internet software company springshare that enables users to create and easily edit web pages of useful resources in a subject area or for a specific course. libguides often provide links to books from the library’s catalogue, instructions for searching databases, and lists of relevant journals or recommended resources. with this tool librarians can “synthesize vast amounts of information about databases, websites, journals, and other sources, and list only the most relevant sources for a particular subject” (ouellette, 2011, pp. 436-437). today, many academic libraries offer access to libguides or other online research guides from the home pages of their websites.   at hunter college’s health professions library (hpl), librarians utilize research guides to teach bibliographic instruction classes, conduct research consultations, and answer queries at the reference desk. students can access the guides by first clicking on an icon labeled “research guides” on the home page of the library’s website, then choosing from an alphabetical list of subjects. when viewing this list or an individual guide, the guides are identified under the heading “libguides.”   the authors designed this study to gain insight into the use and awareness of libguides among undergraduate and graduate health professions students at hunter college. these students differ from other populations studied as health professions students not only follow different curricula but also work in clinical and hospital settings where they face time constraints and rely on evidence based information to make point-of-care decisions. previous researchers have noted that “[h]ealth care professionals are faced with a need to acquire and apply information in an immediate sense (e.g., students may be asked to evaluate a case study of a patient exhibiting particular symptoms, and must be able to easily locate a valid resource for examining this patient and correctly diagnosing him). nursing students exemplify this demand” (barnett-ellis & restauri, 2007, p. 121). given these needs it is likely that, for example, nursing students who are preparing for careers in medical centers will have different information-seeking practices—including subject guide usage—than humanities or social sciences students who aspire to teach and conduct research in academic settings.   some studies have identified differences in the types of information favored by health sciences students compared with others. in a study of the research habits of doctoral students in the united kingdom, carpenter (2012) found that “google or google scholar . . .  were strongly favoured above other sources by arts and humanities, social science and engineering and computing science students; while citation databases or ejournal search interfaces were equally as popular as google among biological and biomedical sciences students” (carpenter, 2012, p. 6). similarly, in a citation analysis of doctoral theses defended at vilnius university in lithuania, grigas et al. (2017) found that “in biomedical sciences the most popular type of information was peer-reviewed papers. . . . the social sciences and the humanities manifest an analogous situation with printed books” (p. 14). the researchers hope that the findings of this study are valuable to academic reference and instruction librarians and will add to existing library and information science (lis) literature on research guide usage, awareness, and perception among health sciences students.   literature review   for decades, librarians have used subject guides to introduce patrons to topics and support the use of library resources. whether known as “research guides, pathfinders, electronic library guides, [or] webliographies,” in academic libraries these tools have traditionally aimed to “assist students with their research needs” (staley, 2007, p. 119). the web-based springshare product libguides evolved in recent years from academic subject guides and similar tools developed by librarians or offered by vendors. first launched in 2007, libguides differed from many previous “static forms” of subject guides in that the libguides platform “allows content to be dynamically pulled in from other sources, shared across multiple guides, and . . . is also based around the concept of reuse and sharing, not just of in-house material, but also guides created by other institutions, supporting a culture of best practice and minimising duplicated effort” (dalton & pan, 2014, p. 516). the ease with which even basic users can edit and update libguides led to a remarkable expansion in their use; writing from a north american perspective, almeida and tidal (2017) found that libguides “have become ubiquitous in academic library environments” (p. 63). a study conducted from 2011 through 2013 examining subject guides on the websites of u.s. academic libraries belonging to the association of research libraries found that 71% of libraries surveyed used the libguides platform (jackson & stacy-bates, 2016, p. 222). as giullian and zitser noted in 2015, libguides have proliferated internationally as well, based on global statistics tracked on the libguides community site (p. 173). this growth has since continued, reaching 731,795 total guides created at 5,460 institutions in 95 countries by march 2020 (springshare, 2020).   given the prevalence of libguides, it is vital for librarians to understand their impact and effectiveness for patrons. a corresponding body of lis literature has emerged, focused on a few primary themes, including “guide content and arrangement, the use of guides, and promotion of their use” (jackson & stacy-bates, 2016, p. 220). a smaller number of studies have focused on specific aspects of guide usage or types of users. one study conducted at san josé state university examined the use of libguides among more than 1,000 undergraduate nursing, journalism, mass communications, and organization and management students (staley, 2007). findings indicated high usage of the “online databases” research guide, particularly among nursing students (p. 125). ouellette (2011) reviewed the use of libguides at concordia university college of alberta by conducting in-depth interviews with 11 students from diverse academic backgrounds, identifying lack of awareness about libguides as one reason respondents did not use them. by contrast, staley (2007) discovered participants who attended information literacy sessions at the san josé state university library were more likely to utilize subject guides. this indicates the value of raising awareness of libguides at library workshops or instruction sessions.   studies have shown that library users hold a wide range of views toward the usefulness of libguides. this ambivalence is not specific to the libguides brand; as dalton and pan (2014) note, “to date subject guides have received a mixed response in terms of both usage levels and user feedback” (p. 516). in an early study focusing on user perceptions, researchers at george washington university surveyed 210 students about the usefulness of libguides (courtois, higgins, & kapur, 2005). results indicated nearly one third of respondents did not find research guides “helpful,” while slightly more, 35%, described them as “[v]ery helpful” (courtois, higgins, & kapur, 2005, p. 192).   some researchers have focused on usability testing, exploring how issues of guide structure, layout, and navigation affect user experience and educational value. in one usability test conducted at metropolitan state university in minnesota, investigators observed “a great deal of frustration and confusion on the part of the participants” when asked to perform tasks using the library’s libguides (sonsteby & dejonghe, 2013, p. 86). that study identified “six major design issues” with libguides, involving confusion about the placement or purpose of search boxes, ambiguity or inconsistency with language or labels used within or across guides, excessive tabs or clutter, and confusing contact information for assistance (sonsteby & dejonghe, 2013, p. 86). these findings correspond to several “common usability impediments” that thorngate and hoden (2017) identify in their review of the literature on libguides usability and design (p. 845). these obstacles include “inconsistent design” from page to page or from guide to guide, “confusing terminology/reliance on library jargon,” cluttered pages that lack a “focal point,” and “too much content, not appropriately scoped to the task at hand” (p. 846). such impediments risk making libguides harder to use and less pedagogically effective for students.   librarians can address such issues with targeted preparation when creating subject guides. as almeida and tidal (2017) point out, “just because librarians have subject expertise and knowledge of specialized research practices does not necessarily mean they can create digital resources that will be easy for students to use or that will address the information needs students have in different contexts” (p. 63). when crafting research guides, librarians must consider factors such as the selection of high-quality resources, arrangement of those resources, annotations that properly describe guide content, and the accuracy and currency of links (jackson & stacy-bates, 2016, p. 220). librarians in this role should receive sufficient training and maintain awareness of best practices to ensure the guides they promote are as useful and relevant as possible.   some studies have looked at the roles libguides can play in health sciences libraries. britton and li (2019) identified “numerous positive results” the university of south alabama’s biomedical library derived from its implementation of libguides, including consolidating access to e-books that would otherwise be siloed by platform or vendor (p. 36). in a survey of nursing libguides at 50 institutions across the u.s., stankus and parker (2012) identified a “common core” of recommended resources, consisting of “cinahl, some version of pubmed or medline, perhaps one or two point-of-care information services, a relative handful of classic reference sources, a few widely recognized journals, and websites, with an overall emphasis on any resources that deal with evidence-based practice” (p. 254). however, to the best of the authors’ knowledge, previous studies have not explored the use of libguides among physical therapy, medical laboratory sciences, or speech-language pathology and audiology students. in addition, previous studies have not collected data about students’ academic level, undergraduate class level, or demographic factors. this paper attempts to fill these gaps in the lis literature.   aims   the goal of this exploratory study was to examine usage, perceptions, and awareness of libguides among undergraduate and graduate health professions students. drawing from a limited, non-random sample, the investigators sought to collect baseline data regarding four research questions:   ●       rq 1. how do the use and awareness of libguides compare among health professions students of different academic levels (undergraduate or graduate) and undergraduate class levels—freshman (first year), sophomore (second year), junior (third year), or senior (fourth year)? ●       rq 2. how do the use and awareness of libguides compare among nursing, physical therapy, medical laboratory sciences, and speech-language pathology/audiology students? ●       rq 3. which type of libguides (course specific or general subject guides) and guide pages (e.g., finding books, finding articles, citation styles) do health professions students utilize? ●       rq 4. do health professions students find libguides useful in their studies?   this study was conducted at hunter college’s brookdale campus in new york city. hunter college is a senior college within the city university of new york system. it offers undergraduate and graduate degrees in a wide variety of academic disciplines through its six schools (hunter college, 2019a). approximately 17,000 undergraduate and 6,000 graduate students currently attend hunter (hunter college, 2019b).   the brookdale campus is one of hunter’s three campuses in manhattan. it houses the school of health professions and the hunter-bellevue school of nursing. the school of health professions offers graduate degrees in speech-language pathology and audiology, as well as physical therapy. the school of nursing offers degrees at the baccalaureate, master’s, and doctoral levels. the brookdale campus also houses the medical laboratory sciences department (part of the school of arts and sciences), which offers undergraduate and master’s degrees. the health professions library supports all of these programs.   the school of health professions and school of nursing at hunter together hold a total enrollment of 1,355 students. the speech-language pathology and audiology program enrolls 47 students, while physical therapy enrolls 96. the school of nursing serves a much larger body of 431 undergraduate and 629 graduate students. medical laboratory sciences has 110 undergraduate and 42 graduate students.   methods   research methodology, survey instrument, and pilot study   the investigators designed a 16-item paper survey instrument based on the study’s research questions. it consisted of fifteen multiple-choice questions and one open response question. the survey addressed students’ use, awareness, and opinions of libguides; gathered demographic information about the participants; and sought to assess participants’ use of the hunter college libraries website and attendance at library instruction sessions.   to identify gaps in existing research for the questions on the instrument to address, the authors searched databases covering library and information science as well as social sciences. the investigators conducted boolean searches in library and information science source, library and information science and technology abstracts, and academic search complete using the following keywords: (subject guides or subject guide, libguides or libguide, research guides or research guide); (use or usage); (awareness); (undergraduate students or college students or university students); (health science students or health sciences students);  (physical therapy students or nursing students or medical laboratory science students); and (speech-language pathology and audiology students). the authors ran a search in ebsco all databases using su (health sciences or health sciences programs or health science programs) and (“research guides” or “subject guides” or libguides or libguides). in web of science, the authors searched for the topic: (“subject guide”) refined by the categories (information science library science or area studies or nursing) and topic: (health).   the authors conducted a pilot study with five health sciences students (four undergraduate and one graduate) to test the survey instrument. this enabled investigators to identify ambiguous questions and revise them for clarity. investigators also incorporated feedback from a hunter college librarian not involved in the study to further refine the instrument. see the appendix for the text of the final survey instrument.   data collection and analysis   in april 2017, following approval from the institutional review board, the investigators administered the survey to health professions students at the college. the researchers chose april to avoid scheduling conflicts with upcoming final examinations. the researchers recruited participants for the study during one day from 9 a.m. to 7 p.m., choosing a day of the week that is usually busy for patron traffic. conducting the survey from morning through evening allowed recruitment of students enrolled in morning, afternoon, or evening classes at the campus.   researchers approached potential participants in college hallways, in front of classrooms, both outside and inside the hpl, outside the school of health professions building, and inside the campus cafeteria. the researchers set a goal of recruiting 100 participants for this study in order to collect baseline data with a view toward conducting a larger quantitative or qualitative study in the future. they approached 103 potential participants, 100 of whom agreed to take the survey (three declined to participate due to lack of time). all students approached were health sciences majors and thus eligible to participate in the study, not surprising given that this campus houses only health professions programs. the investigators explained the study’s purpose, informed potential subjects that their participation was voluntary and responses would remain anonymous, and provided copies of the informed consent script. afterwards they administered paper questionnaires to participants, none of whom received monetary compensation or other incentives. most participants took three to five minutes to complete the questionnaire, although some requested additional time (up to five to ten minutes) due to language difficulty.   survey population and demographics   all survey respondents were health sciences students. one hundred students completed the survey in full, demonstrating a 97% response rate and a 100% completion rate. among those surveyed, 78% (n = 78) were undergraduates, 21% (n = 21) graduates, and 1% (n = 1) identified as “other” (continuing education). among the 78 undergraduates, 31% (n = 24) were sophomores (second year), 42% (n = 33) were juniors (third year), and 27% (n = 21) seniors (fourth year). respondents included no freshmen as students enter the hunter college undergraduate health sciences programs only in their second year or later. however, the researchers included the freshman demographic in the survey instrument because students from other city university of new york colleges can access the hpl, and some of those institutions admit first-year students into their health sciences programs. therefore, the investigators wanted to give the opportunity to all eligible patrons of the library to participate in the study.   of those surveyed, 22 were male and 78 female. sixty participants were under 25 years of age, 30 between 25 and 34 years of age, five between 35 and 44, four between 45 and 54, and one was 55 or older. study participants also represented a wide range of racial and ethnic groups. forty-six respondents identified as asian, 28 as white, 12 as being from other racial and ethnic groups, seven as black, and seven as hispanic.   in terms of majors, 65 students were pursuing degrees in nursing, seven in physical therapy, 24 in medical laboratory sciences, and three in speech-language pathology and audiology. one participant chose not to answer this question.   results   only a minority of participants (n = 35) reported being aware of libguides, and even fewer (n = 32) reported using them. sixty-eight respondents said they had never used libguides, 65 of whom were unaware of the guides. in addition to the following text summaries that describe the results according to students’ academic level and field of study, table 1 provides a complete record of participant responses regarding use and awareness.   use and awareness of libguides by academic level and class level   both awareness and use of libguides were higher among graduate than undergraduate respondents. approximately 57% (n = 12) of graduate respondents indicated awareness of libguides, while only 30% (n = 23) of undergraduates did so. however, all undergraduates who were aware of libguides also reported using them (n = 23, 30%). among graduate students, use was slightly lower than awareness (n = 9, 43%).   the researchers disaggregated libguide use and awareness data by undergraduate class level. close to 21% (n = 5) of sophomores (second year), 30% (n = 10) of juniors (third year), and 38% (n = 8) of seniors (fourth year) reported awareness of libguides. usage similarly increased with academic level:  approximately 21% (n = 5) of sophomores and 27% (n = 9) of juniors reported having used libguides whereas 43% (n = 9) of seniors had done so.   use and awareness of libguides by health sciences concentration   this study also collected data on libguides use and awareness according to concentrations within the health professions. results show most students within each discipline were not using libguides. among the 65 nursing students—the study’s largest cohort—only 19 were aware of libguides and 46 had not used them. five out of the seven physical therapy students indicated awareness of libguides and 4 had used them. among medical laboratory science students, 10 indicated awareness of libguides yet 15 had never used them. among speech-language pathology and audiology students, only one respondent was aware of libguides and the other two had never used them.   table 1 awareness and use of libguides, by number of respondents and percentage (n = 100)   awareness use yes, n (%) no, n (%) yes, n (%) no, n (%) all participants 35 (35) 65 (65) 32 (32) 68 (68) by academic level ●       undergraduate students 23 (30) 55 (70) 23 (30) 55 (70) ●       graduate students 12 (57) 9 (43) 9 (43) 12 (57) ●       other 0 1 (100) 0 1 (100) by undergraduate class level ●       sophomore 5 (21) 19 (79) 5 (21) 19 (79) ●       junior 10 (30) 23 (70) 9 (27) 24 (73) ●       senior 8 (38) 13 (62) 9 (43) 12 (57) by health sciences concentration ●       medical laboratory sciences 10 (42) 14 (58) 9 (37) 15 (63) ●       nursing 19 (29) 46 (71) 19 (29) 46 (71) ●       physical therapy 5 (71) 2 (29) 4 (57) 3 (43) ●       speech-language pathology and audiology 1 (33) 2 (67) 1 (33) 2 (67) by age group ●       under 25 years of age 17 (28) 43 (72) 18 (30) 42 (70) ●       25-34 14 (47) 16 (53) 12 (40) 18 (60) ●       35-44 2 (40) 3 (60) 1 (20) 4 (80) ●       45-54 2 (50) 2 (50) 1 (25) 3 (75) ●       55 and over 0 1 (100) 0 1 (100) by gender ●       female 28 (36) 50 (64) 28 (36) 50 (64) ●       male 7 (32) 15 (68) 4 (18) 18 (82)   use of libguides by type and page   the researchers further disaggregated libguides usage data by type of guide: course specific or a general subject area. of the 32 respondents who had used libguides, most (63%, n = 20) reported using general subject guides, while 34% (n = 11) used course-specific guides. only one respondent reported using both types of guides (figure 1).   figure 1 use of libguides by type (n = 32).   based on observations drawn from several years of experience working with students in the health professions programs, the researchers suspected that students often remember the specific guides and pages within guides that they access. for instance, after library instruction sessions the second author has received questions from students asking whether they could go directly to a libguides page such as “finding books” rather than navigating from the library’s home page. therefore, the investigators designed one question on the instrument to collect data on participants’ utilization of specific pages within libguides. among the 32 respondents who had used libguides, 21 navigated to pages helping to locate journal articles, and three used pages assisting with apa style. just one respondent used pages locating print and e-books while another used a page about citation management software. six respondents indicated using more than one type of page within the guides (figure 2).   figure 2 use of libguides by page (n = 32).   utility of libguides   the researchers sought to determine whether students found libguides useful in their studies. of the 32 respondents who reported having used libguides, 31 (97%) found them useful and one (3%) said they did not know whether the guides were useful or not.   use and awareness of libguides and library instruction sessions   thirty-eight of the health professions students who took part in the study reported attending a library workshop or instruction session, whereas 61 did not attend. one participant did not answer this question.   of the 38 respondents who had attended a library workshop or information literacy session, 22 (58%) indicated that they were aware of libguides and 16 (42%) that they were unaware; 19 (50%) of these respondents said they had used libguides and 19 had not. of the 61 participants who had not attended a library workshop or instruction session, 13 (21%) said they were aware of subject guides and an equal number reported having used them. forty-eight (79%) of the uninstructed participants said they were not aware of libguides and had not used them. so, the survey data appear to suggest a connection between attending a library workshop or instruction session and increased use and awareness of libguides.   open question responses   the survey instrument included an open question asking participants whether they had any additional comments or suggestions for the librarians. the vast majority of respondents, 86% (n = 86), had no comments. many of the comments received addressed topics not relevant to this study, such as a desire for more charging stations and extended library hours, or the relative helpfulness of library staff. of those comments that did refer to libguides, one student said that they now “will use libguides.” another participant wrote, “we use google for our research projects. but if instructors inform us about libguides, we may look into it more.” another wrote that while they had never heard of libguides, “it sounds helpful for research students should be made aware of the service [sic].”   discussion   the goal of this research project was to ascertain the use, awareness, and perceptions of libguides among baccalaureate, master’s, and doctoral-level students in health sciences. the most compelling discovery was that the vast majority of respondents (65%) were unaware of the existence of research guides and nearly the same proportion do not utilize them. the results indicating low levels of use and awareness persist across various academic levels and demographics. this study found higher use and awareness of libguides among graduate students than undergraduate students. moreover, within the undergraduate cohort both use and awareness increased progressively from second-, to third-, to fourth-year students. one possible explanation for these results could be the extensive use of course-specific libguides by instruction librarians during information literacy sessions at the hpl. as we have seen, the data collected here suggest a link between attendance at these sessions and increased awareness of libguides. librarians at the hpl teach more one-shot information literacy sessions for graduate and upper-level undergraduate students than for others, which may explain the higher rates of use and awareness found among these groups. another possible explanation could be the promotion of subject guides by some health sciences faculty to their students. one physical therapy instructor mentioned during a conversation with the second investigator that he encourages his doctor of physical therapy students to utilize relevant subject guides.   among those respondents who had used libguides, two thirds reported having used the “finding articles: databases” page, which helps students navigate databases and locate journal literature. as jackson and stacy-bates (2016) note, several earlier studies also found high usage for guide pages that link to databases. in the current study, usage of this page far eclipsed that of other pages common to all the guides, such as those addressing use of the library’s catalogue or citation management software. moreover, this was probably the only page many users were viewing, since most respondents who used libguides did not report consulting more than one page within the guides. the popularity of this page among users could be due to the ease with which libguides allow patrons to browse and seamlessly connect to a curated list of databases or other electronic resources, or it could indicate a demand for research support not sufficiently addressed by other means. these results may also reflect a discipline-specific prevalence of assignments in the health professions that require students to retrieve and cite journal articles rather than, for example, books from the circulating collection. as nordsteien, horntvedt, and syse (2017) note in their study of the effect of faculty-library teaching collaborations on norwegian nursing students’ research skills, “it is not enough to teach how to search for information; skills such as formulating a research question and critically appraising, analysing and synthesising the literature are also required” (p. 24). in their description of an embedded information literacy program for nursing students implemented at the memorial university of newfoundland, farrell, goosney and hutchens (2013) detail the research competencies that map to specific stages of the nursing curriculum:   first year: basic searching of the cumulative index of nursing and allied health literature (cinahl) database, and evaluating and distinguishing between popular and scholarly literature. second year: drug information resources, alternative medicine resources, critical evaluation of web-based information, and advanced cinahl searching techniques (subject headings). third year: the principles of evidence informed practice, study types, formulating a research question using the pico (patient, intervention, comparison, outcome) method, searching pubmed using both mesh and clinical queries, searching cinahl using clinical queries, and other evidence-based resources. (p. 166)   while the current study cannot identify a cause for the use of individual pages within or across libguides, the heavy usage of the “databases” page suggests that among the population surveyed, a primary function of the library’s subject guides is to assist with this type of literature searching and appraisal emphasized in the curriculum.   finally, the finding that less than one third of participants in this study had used libguides may lead some to question whether the creation and maintenance of such guides merits the time and attention involved. however, to recommend abandoning such efforts based on the data collected here would be premature. even leaving aside the argument that the impact of subject guides goes beyond student use (many librarians and staff also consult them when delivering public services such as reference or instruction), it is nonetheless promising that among those participants who had used libguides almost all (97%, n = 31) said they found the guides useful. this is not to minimize the challenges librarians face in producing the guides, challenges which go beyond technical mastery of the libguides platform. as one respondent wrote in jackson and stacy-bates’ (2016) survey of heads of reference, “librarians spend a lot of time carefully compiling exhaustive amounts of information, but they could use help packaging it for consumption. librarians need to become more familiar with principles of user-centered design and best practices in writing for the web” (p. 227). addressing this need may involve additional staff training or the development of departmental guidelines or policies. however, the finding that virtually all participants in this study who had accessed libguides perceived some benefit suggests that libraries should explore building on this success before drawing any final conclusions regarding the merits of subject guides.   limitations and recommendations for further research   this study has three primary limitations. first, only 100 health professions students out of an enrollment of 1,355 participated in this survey. this small sample size makes it problematic to draw conclusions applicable to health professions students in general at hunter college or similar institutions. the project also relied on self-reported data, which can introduce response bias in the results. finally, the investigators administered the questionnaire only on campus, thus capturing no data from students taking only online or hybrid courses.   with regard to the data collection, the researchers verbally explained to participants before distributing the questionnaires that if they had not used libguides (that is, answered “no” to item 11), then they should not answer the question regarding whether or not they find libguides useful (item 15). however, given the fact that 88 respondents answered question 15 despite the fact that only 32 had used libguides, perhaps it would have been more effective to include an instruction about this in the text of the instrument. moreover, the researchers did not address the possibility that some students who may have used or known of libguides nonetheless did not know the name of the specific tool, and therefore might answer “no” when in fact they had used a libguide. both of these circumstances introduce potential ambiguity into the results and constitute additional limitations of this study.   in spite of its limitations, the study provides opportunities for further research on this topic. while libguides will naturally vary by institution, further study of their usage at other locales by health sciences students could reveal more about the information needs and behaviors of health sciences students in general. in addition, investigators could design a similar study to explore the use, perceptions, and awareness of research guides by students majoring in other subjects. they could conduct a comparative study to examine responses of students enrolled in online and hybrid courses versus those of traditional students on campus. finally, a study utilizing focus groups could provide qualitative insights about health sciences and other students’ libguides usage and information behaviors.   conclusion   this study examined the use, perceptions, and awareness of subject guides among undergraduate and graduate health professions students. results reveal low use and awareness of libguides among the majority of study participants. this suggests several courses of action for librarians at hunter college or similar institutions. to begin with, librarians should ensure that the libguides they design are as useful and appealing to students as possible by familiarizing themselves with commonly noted usability impediments and best practices. during this process, health sciences librarians must remain mindful of the competencies health professions students need to master and the types of assignments and coursework that these students undertake, so that they can highlight the most relevant resources (such as point-of-care information services or evidence based practice resources) and support development of the most relevant skills (such as advanced searching of journal literature or formulation of research questions).   once librarians have created suitable libguides, they can employ numerous strategies to maximize use and awareness. first, libraries should prominently display links to the guides on the library web page to enhance discoverability, and the catalogue itself should link directly to the guides. instructional librarians should use subject guides as instruction tools to raise awareness among students attending information literacy sessions and highlight them as a key resource during reference interactions. as a form of outreach to teaching faculty, librarians can demonstrate the value of subject guides by mapping guide content to the health professions curriculum and can encourage faculty to showcase libguides on course syllabi or link to them through course management sites. finally, new student and faculty orientations offer opportunities to raise awareness of subject guides. the researchers hope other academic health professions libraries benefit from the results of this project and encourage fellow librarians in other disciplines to undertake further research on this topic.   references   almeida, n., & tidal, j. (2017). mixed methods not mixed messages: improving libguides with student usability data. evidence based library and information practice, 12(4), 62–77. https://doi.org/10.18438/b8cd4t   barnett-ellis, p., & restauri, s. (2007). nursing student library usage patterns in online courses: findings and recommendations. internet reference services quarterly, 11(4): 117-138. https://doi.org/10.1300/j136v11n04_08   britton, r. m., & li, j. (2019). using subject guides to support curriculum and patient care: a multipronged approach. medical reference services quarterly, 38(1), 31-40. https://doi.org/10.1080/02763869.2018.1547048   carpenter, j. (2012). researchers of tomorrow:  the research behavior of generation y doctoral students. information services & use, 32(1-2), 3-17. https://doi.org/10.3233/isu-2012-0637   courtois, m. p., higgins, m. e., & kapur, a. (2005). was this guide helpful? users' perceptions of subject guides. reference services review, 33(2), 188–196. https://doi.org/10.1108/00907320510597381   dalton, m., & pan, r. (2014). snakes or ladders? evaluating a libguides pilot at ucd library. the journal of academic librarianship, 40(5), 515-520. https://doi.org/10.1016/j.acalib.2014.05.006   farrell, a., goosney, j., & hutchens, k. (2013). evaluation of the effectiveness of course integrated library instruction in an undergraduate nursing program. journal of the canadian health libraries association / journal de l’association des bibliothèques de la santé du canada, 34(3), 164-175. https://doi.org/10.5596/c13-061   giullian, j. c., & zitser, e. a. (2015). beyond libguides: the past, present, and future of online research guides. slavic & east european information resources, 16(4), 170–180. https://doi.org/10.1080/15228886.2015.1094718   grigas, v., juzėnienė, s., & veličkaitė, j. (2017). “just google it”—the scope of freely available information resources for doctoral thesis writing. information research, 22(1). http://informationr.net/ir/22-1/paper738.html   hunter college. (2019a). majors and degree tracks. retrieved from http://www.hunter.cuny.edu/academics/majors   hunter college. (2019b). facts and statistics. retrieved from http://www.hunter.cuny.edu/communications/media-relations/facts   jackson, r., & stacy-bates, k. (2016). the enduring landscape of online subject research guides. reference & user services quarterly, 55(3), 219–229. https://doi.org/10.5860/rusq.55n3.219   nordsteien, a., horntvedt, m.-e., & syse, j. (2017). use of research in undergraduate nursing students’ theses: a mixed methods study. nurse education today, 56, 23-28. https://doi.org/10.1016/j.nedt.2017.06.001   ouellette, d. (2011). subject guides in academic libraries: a user-centered study of uses and perceptions. the canadian journal of information and library science, 35(4), 436–451. https://doi.org/10.1353/ils.2011.0024   sonsteby, a., & dejonghe, j. (2013). usability testing, user-centered design, and libguides subject guides:  a case study. journal of web librarianship, 7(1), 83-94. https://doi.org/10.1080/19322909.2013.747366   springshare. (2020). libguides community. retrieved from https://community.libguides.com/   staley, s. m. (2007). academic subject guides: a case study of use at san josé state university. college & research libraries, 68(2), 119–139. https://doi.org/10.5860/crl.68.2.119   stankus, t., & parker, m. a. (2012). the anatomy of nursing libguides. science and technology libraries, 31(2), 242-255. https://doi.org/10.1080/0194262x.2012.678222   thorngate, s., & hoden, a. (2017). exploratory usability testing of user interface options in libguides 2. college & research libraries 78(6), 844-861. https://doi.org/10.5860/crl.78.6.844   appendix questionnaire: use and awareness of libguides   1.     which institution do you attend? please select one of the following; a.      hunter college b.     other cuny college (please specify) c.      other college or institution (please specify) d.     i do not attend a college or institution 2.     what is your academic level? please select one of the following: a.      undergraduate student b.     graduate/professional student c.      continuing education student d.     other (please specify) 3.      if you are an undergraduate student, please select one of the following. otherwise skip this question. a.      freshman b.     sophomore c.      junior d.     senior 4.     what is your gender? please select one. a.      male b.     female c.      transgender 5.     approximately what is your age? please select one. a.      under 25 b.     25-34 years c.      35-44 years d.     45-54 years e.      55 and over 6.     how do you identify yourself? please select one. a.      white b.     black c.      hispanic d.     american indian or alaska native e.      asian f.      arab/middle eastern g.     native hawaiian or other pacific islander h.     multiracial i.       other (please specify) 7.     what is your major or intended major or in what subject area do you have or hope to obtain a degree? please select one. a.      nursing b.     physical therapy c.      medical laboratory sciences d.     speech-language pathology and audiology e.      other (please specify)     8.      how often do you visit your college library’s website? a.      at least once per week b.      at least once per month c.      at least once per semester d.      i never visit the library’s website 9.      have you ever attended a library workshop or instruction session? a.      yes b.      no 10.   libguides (research guides) are collections of web pages that gather together useful resources related to a subject area or to a specific course. libguides often provide links to books from the library’s catalogue, instructions for searching databases, and lists of relevant journals or other recommended resources. hunter’s libguides can be accessed from the library’s home page. are you aware of libguides related to your courses or major? a.      yes b.     no 11.   have you used libguides? a.      yes b.     no 12.  which type of libguides do you use most often? a.      course-specific guides (e.g., nursing 700, nursing 380) b.     general subject area guides 13.  which pages of libguides do you use most often? a.      finding books b.     finding articles: databases c.      managing your references d.     citing: apa style e.      other (please specify)     14.  how often do you visit or check libguides? a.      at least once per week b.     at least once per month c.      at least once per semester d.     i never visit or check libguides 15.   do you find libguides useful for your academic purposes (such as research papers or other assignments)? a.      yes b.      no c.      do not know 16.   any other comments or suggestions:     evidence based library and information practice 2012, 7.3 124 evidence based library and information practice news/announcements eblip seeks new production editor 2012. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. evidence based library and information practice (eblip) is a peer reviewed, open access journal, which provides a forum for librarians and other information professionals to discover research that may contribute to decision making in professional practice. eblip is published quarterly by the university of alberta learning services. eblip publishes original research and commentary on the topic of evidence based library and information practice, as well as reviews of previously published research (evidence summaries) on a wide number of topics. eblip seeks a new production editor, for a three year term commencing december 2012. as with all other eblip editorial positions, the production editor is a voluntary appointment. the production editor role is one of the final steps of the journal production process. responsibilities include laying out the pdf and html of all journal content and working with section editors to make the final publication look professional, attractive and in the house style. advanced experience with microsoft word, good knowledge of html, strong attention to detail and the ability to meet tight deadlines are essential. knowledge of the ojs system would also be beneficial. the role involves a time commitment of approximately 10 hours per month, with the bulk of the work occurring on a quarterly basis prior to journal publication. the production editor should also be available to attend monthly editorial meetings via skype. the new editor should be in a position to commence working with the outgoing production editor by mid november to ensure an effective handover of responsibilities and training. for informal enquiries and more information about the role, please contact katrine mallan – katrine.mallan@nrccnrc.gc.ca. for more information about the journal, please visit the evidence based library and information practice web site http://ejournals.library.ualberta.ca/index.php/e blip/index. to apply, please submit a brief statement of interest and resume including all relevant experience to alison brettle, editor-in-chief a.brettle@salford.ac.uk by 15 october 2012. http://www.ls.ualberta.ca/ http://www.ls.ualberta.ca/ mailto:katrine.mallan@nrc-cnrc.gc.ca mailto:katrine.mallan@nrc-cnrc.gc.ca http://ejournals.library.ualberta.ca/index.php/eblip/index http://ejournals.library.ualberta.ca/index.php/eblip/index mailto:a.brettle@salford.ac.uk microsoft word art_1515_ evidence based library and information practice 2008, 3:2 3 evidence based library and information practice article an evaluation of the five most used evidence based bedside information tools in canadian health libraries alison farrell public services librarian and instruction coordinator health sciences library memorial university of newfoundland st. john's, newfoundland, canada email: alisonr@mun.ca received: 13 march 2008 accepted: 15 may 2008 © 2008 farrell. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – this project sought to identify the five most used evidence based bedside information tools used in canadian health libraries, to examine librarians’ attitudes towards these tools, and to test the comprehensiveness of the tools. methods – the author developed a definition of evidence based bedside information tools and a list of resources that fit this definition. participants were respondents to a survey distributed via the canmedlib electronic mail list. the survey sought to identify information from library staff regarding the most frequently used evidence based bedside information tools. clinical questions were used to measure the comprehensiveness of each resource and the levels of evidence they provided to each question. results – survey respondents reported that the five most used evidence based bedside information tools in their libraries were uptodate, bmj clinical evidence, first consult, bandolier and acp pier. librarians were generally satisfied with the ease of use, efficiency and informative nature of these resources. the resource assessment determined that not all of these tools are comprehensive in terms of their ability to answer clinical questions or with regard to the inclusion of levels of evidence. uptodate was able to provide information for the greatest number of evidence based library and information practice 2008, 3:2 4 clinical questions, but it provided a level of evidence only seven percent of the time. acp pier was able to provide information on only 50% of the clinical questions, but it provided levels of evidence for all of these. conclusion – uptodate and bmj clinical evidence were both rated as easy to use and informative. however, neither product generally includes levels of evidence, so it would be prudent for the practitioner to critically appraise information from these sources before using it in a patient care setting. acp pier eliminates the critical appraisal stage, thus reducing the time it takes to go from forming a clinical question to implementing the answer, but survey respondents did not rate it as high in terms of usability. there remains a need for user-friendly, comprehensive resources that provide evidence summaries relying on levels of evidence to support their conclusions. introduction evidence based medicine can be defined as “the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients” (sackett et al. 71). there are four steps to practicing evidence based medicine. first, one must have a question that is phrased in such a way that it can be searched. the next step is searching the literature. third, one must critically appraise the information found, and lastly, implement the new information in practice (rosenberg and donald 1122). health care professionals often need information in a timely manner at the point of care (i.e., at the patient’s bedside or in the doctor’s office) and may not have the time to follow all four steps. there is too much information, and it takes far too long to read and synthesize that information. evidence based bedside information tools may be a potential solution for this problem. for the purposes of this study, evidence based bedside information tools are defined as web-based, searchable, patient oriented, and pre-digested forms of information that offer evidence summaries of relevant research that has been critically appraised by scholars in the field and are peer reviewed. these resources should demand little work to provide answers to clinical questions. they provide health professionals with the information they need in a summarized format, thus shortening the time needed to find answers. good evidence based bedside information tools should be dependable, reliable, and based on appropriate evidence, thus ensuring the health care professional is getting the best information available to make decisions as quickly as possible. studies have shown that the average physician will spend only two minutes looking for an answer before giving up the quest. (alper et al., “answering” 961; campbell and ash, “an evaluation” 435; ketchell et al. 537). furthermore, most physicians will look at only a small number of documents to find the answer. in the research conducted by magrabi and her colleagues, clinicians examined only two documents for an answer to a clinical question in 56.4% of the searches examined in their research (“clinicians” 299). further, research has revealed that although physicians encounter many questions in the run of an average workday, they often neglect to look for the answers. green and ruff found that residents pursue only 28% of the clinical questions that arise in an evidence based library and information practice 2008, 3:2 5 average day (176). alper et al. showed that while family physicians have an average of six clinical questions per half-day of practice, 70% of those questions go unanswered (“answering” 960). there are many barriers to finding the answers, such as lack of time, difficulty in phrasing a question, lack of awareness of search resources, forgetting to look, and lack of search skills (ketchell et al. 538). one potential solution is to turn to a good, reliable evidence based bedside information tool. there are many of these tools available ranging from tools giving short, informative summaries of individual studies to those that synthesize multiple studies into a concise summary. to date, while there have been studies demonstrating the effectiveness of using evidence based bedside information tools for patient care, there have been few that evaluate specific resources. in a united kingdom study, 60% of the general practitioners who used the attract system reported that the information gained had changed their practices (westbrook, gosling, and coiera 114). according to schwartz, 62% of the answers sought in a clinical practice will modify a physician’s opinion, 56% of the answers will change current patient care, and 70% of the answers will affect future patient care (254). koonce, giuse, and todd found that evidence based resources provided answers for general patient care management questions more frequently than they aided complex clinical questions. they also documented that evidence based resources are successful when used at the point of care. magrabi et al. concluded that general practitioners will use evidence based resources if they are accessible and easy to use in their daily practice (“general practitioners”). campbell and ash performed a usercentered evaluation of five evidence based bedside information tools that resulted in a ranked list. the resources were evaluated based on criteria that practitioners deemed to be most important. the authors argued that the content of a resource is not the only way to measure the usefulness and value of a resource, and that if physicians do not like the interface, design, and organization of a resource they will not use it. they concluded, “a product selected because of excellent content may be rendered useless by a difficult user interface” (“comparing” 104). campbell and ash found that users preferred the interface and design of uptodate and were able to find more answers to their questions in uptodate than in the other four resources evaluated. they also found opinions of acp pier were mixed, with six percent of the users in their study ranking it as the best of the five resources, and thirteen percent ranking it as the worst (“comparing” 103-4). the questions used in campbell and ash’s study were selected only after they proved to be answerable in all resources tested, and thus their study did not test the comprehensiveness of the resources. librarians may wish to take both user satisfaction and depth of content into consideration when deciding which resources to purchase. the literature review did not identify any studies that measured the comprehensiveness of resources in terms of their abilities to answer questions, nor in terms of the inclusion of the levels of evidence on which the answers were based. the inclusion of levels of evidence in a resource eliminates the need for the third step, critical appraisal, in the evidence based process. it also allows physicians to quickly implement the newfound knowledge into their practices – an important consideration for physicians with limited time. alper, white, and ge found that physicians were able to answer more questions with evidence based resources that synthesized and critically appraised the evidence than evidence based library and information practice 2008, 3:2 6 they were with more traditional resources, such as bibliographic databases. as a result, they found that use of resources with a critical appraisal component brought about changes in their clinical practices more frequently (511). health professionals and librarians have a problem in selecting appropriate evidence based resources. this study examined the most frequently used resources among canadian health libraries. findings from this project may assist librarians with purchasing decisions for their libraries and thus aid their clientele, as well. the project began by identifying the five most used tools, the reasons they were being used, and whether or not librarians found them easy to use, efficient, and informative. in addition to librarians’ attitudes towards evidence based bedside information tools, this study tested the comprehensiveness and inclusion of levels of evidence in the five resources. the inclusion of levels of evidence is an important aspect of an evidence based bedside information tool, as it allows the physician to confidently skip the critical appraisal step in evidence based medicine. methods a survey (appendix 1) sent to canadian health librarians via the canmedlib mail list sought to learn the attitudes of librarians towards evidence based bedside information tools. a list of evidence based resources fitting the definition outlined above was developed using the literature, internet, and previous knowledge. in addition to meeting the definition, the study included only resources available to anyone, either freely or for a fee. this excluded, for example, resources developed for specific organizations, such as the clinical information access program (ciap) in australia (westbrook 113), and accessible only to employees of that organization. based on these criteria, the following resources, in alphabetical order, were included for consideration: • acp pier • attract • bandolier • bestbets • bmj clinical evidence • diseasedex • dynamed • first consult • fpin clinical inquiries • infopoems/inforetriever (now essential evidence plus) • pepid • uptodate the survey was sent to all members of the canmedlib mail list. questions were designed to determine why libraries subscribe to or promote the use of certain products, and the library staff members’ satisfaction with the products in terms of ease of use, efficiency and information provided. after identifying the five most subscribed to or promoted resources, the project developed a test to determine the comprehensiveness of each product. twenty questions were randomly chosen from the u.k. national health service’s primary care question answering service. the primary care question answering service is a compilation of clinical questions submitted by u.k. health practitioners that have been answered and summarized. this resource was selected because of the range of clinical questions and because the questions had been submitted by practicing clinicians, thus increasing the likelihood of their applicability for this project. the questions were selected with no knowledge of the answers. the librarian researcher searched for the answers to these questions in each of the five most used resources, seeking answers to each question and determining whether the answers included levels of evidence based library and information practice 2008, 3:2 7 evidence. all searches were performed by the same librarian. the study did not seek to determine the accuracy or usefulness of the answer provided (e.g., whether it was not detailed enough, too detailed, too long, or too short). results survey participation a survey was distributed in march 2007 to the canmedlib mail group, and a reminder was sent the following week. respondents had a two-week period to complete the survey. as an incentive to participate in the survey, respondents had the opportunity to compete in a draw for one of two ipod shuffles. to be entered in the draw, the participant had to include a name and email address, but the inclusion of the name and email address was not necessary to complete the survey. a total of 52 surveys were returned, for a return rate of approximately 10-11%. no statistical analysis was performed on the results, because the low number of surveys returned meant the analysis could not be statistically significant. in light of this, the information gained from this study provides only an indication of trends, rather than a statistically accurate picture. the membership of canmedlib is not limited to librarians, but the majority of respondents (86.3%, n=44) identified themselves as librarians. the majority of respondents (84.3%, n= 43) worked in hospital libraries, academic libraries, or in academic libraries that also served hospitals. resources used the evidence based bedside information tools most often subscribed to or promoted by the canadian health libraries participating in the survey were bmj clinical evidence, uptodate, acp pier, bandolier, and first consult. (figure 1.) most used evidence based bedside information tools 0 10 20 30 40 50 bm j clinical evidence up todate acp pier bandolier first consult resource us ed % l ib r a r ie s s u b sc r ib e d t o o r p r o m o ti n g r e so u r c e figure 1: five most-used evidence based bedside information tools in participating canadian health libraries. evidence based library and information practice 2008, 3:2 8 other resources reported by canadian libraries responding to the survey included attract, bestbets, diseasedex, dynamed, fpin clinical inquiries, infopoems/inforetriever, and pepid. of these, only bestbets, infopoems/inforetriever, and dynamed were used by three or more survey respondents. other products were suggested that did not meet the inclusion criteria outlined in the project methodology, such as emedicine, epocrates, md consult, trip, and e-therapeutics. it is noteworthy that 22.9% of the respondents were not using any of the listed resources, although it was not clear if these participants used other products they considered to be evidence based bedside information tools. when asked the importance of considerations for purchasing or promoting the selected evidence based bedside information tools, the most common responses were comprehensiveness of the resource, cost, and inclusion of levels of evidence. actual decisions to purchase or promote products were driven by cost (n=10), demand from physicians or administration (n=14), and the inclusion of products in consortial deals or packages (n=9). survey participants rated the ease of use, efficiency, and the informative nature of three evidence based bedside information tools used in their institutions. ratings used a likert scale, where the possible answers were “strongly agree,” “agree,” “disagree.” and “strongly disagree.” the results showed that librarians were generally satisfied with the evidence based bedside information tools being used in their libraries. figures 2, 3, and 4 show the results for the five resources that were most used in these canadian health libraries. is the resource easy to use? 0 2 4 6 8 10 12 uptodate bmj clinical evidence acp pier bandolier first consult resource strongly agree agree disagree strongly disagree figure 2: ease of use of five evidence based bedside information tools. evidence based library and information practice 2008, 3:2 9 uptodate, bmj clinical evidence and bandolier were perceived as easy to use by all the respondents who used them (n=20, n=16, and n=4 respectively). acp pier was the only resource that garnered any negative responses regarding ease of use, with three of twelve respondents disagreeing with the statement that acp pier is easy to use. of all respondents, only one indicated that any of the top five tools was not informative. one of twelve respondents disagreed that acp pier is informative. one of four respondents strongly agreed that bandolier is an informative tool, while half (n=10) of respondents indicated that uptodate is informative. bmj clinical evidence had the highest percentage of respondents (62.5%, n=10) strongly agreeing with the statement that the resource is informative. although acp pier was the only tool to elicit negative responses to the questions regarding ease of use and informative nature, all tools received at least one negative response regarding efficiency. no respondents strongly disagreed with the statement that the tools were efficient. one out of sixteen respondents disagreed that bmj clinical evidence is an efficient resource, but four out of twelve respondents disagreed that acp pier is an efficient resource. is this resource informative? 0 2 4 6 8 10 12 uptodate bmj clinical evidence acp pier bandolier first consult resource strongly agree agree disagree strongly disagree figure 3: informative nature of five evidence based bedside information tools. evidence based library and information practice 2008, 3:2 10 comprehensiveness testing no resource was able to answer all twenty of the randomly chosen questions from the primary care question answering service. uptodate provided answers for the most questions, answering a total of fourteen. however, only one of those fourteen answers provided a level of evidence. acp pier and first consult each had answers for ten of the twenty questions. acp pier provided the levels of evidence for ten questions, and first consult had levels of evidence for five questions. bandolier performed most poorly with a total of only four answers for the twenty questions, none of which offered levels of evidence. table 1 shows the complete results for the test of the comprehensiveness of the resources. is the resource efficient? 0 2 4 6 8 10 12 uptodate bmj clinical evidence acp pier bandolier first consult resource strongly agree agree disagree strongly disagree figure 4: efficiency of five evidence based bedside information tools. evidence based library and information practice 2008, 3:2 11 discussion because the membership of canmedlib varies as new members subscribe and others unsubscribe, the exact number of people who received the survey is not known. the list moderator indicated that at any given time there are approximately 450 to 500 members of canmedlib (fahey). fiftytwo responses to the survey were received, giving an approximate response rate of 10% to 11%. due to the low response rate, results of the survey are not statistically significant, although qualitative information can be gleaned from the responses. three resources were identified as being in common use in these canadian health libraries: uptodate, bmj clinical evidence, and acp pier. the number of libraries subscribing to or promoting the use of first consult and bandolier, the next most used resources, was low. there is an abundance of evidence based bedside information tools on the market competing for a library’s attention and funding. at present, purchasing decisions in these libraries are based on cost, consortial deals, requests from practitioners, and top down decisions from administrators. however, evaluation of these resources is essential prior to making purchasing decisions. uptodate is as an easy to use and informative resource, according to results of this study and other user satisfaction surveys (campbell and ash “comparing”). however, the testing completed in this study showed that levels of evidence accompanied only seven percent of the questions answered. this indicates that practitioners could not be confident that their answers were based on good evidence. in contrast to this, although 25% of respondents who listed acp pier as one of their choices for evaluating the resources did not agree that it is easy to use, levels of evidence accompanied 100% of the answers found in acp pier. this means that if practitioners were to use acp pier, they could avoid taking additional time for critical appraisal of the evidence. they could still be confident in applying information from pier in their practice while decreasing the time needed to progress from forming a clinical question to implementation of the information in their practice. since survey participants believed acp pier to be not as easy to use as uptodate, practitioners may be less likely to use it. respondents did, however, rate acp pier as an informative resource, with only one respondent disagreeing with this statement. these results show a need for resources to be both easy to use and comprehensive. no matter how comprehensive it may be, a resource is resource number (percent) questions answered number (percent) questions answered and providing levels of evidence uptodate 14 (70%) 1 (7%) acp pier 10 (50%) 10 (100%) first consult 10 (50%) 5 (50%) bmj clinical evidence 6 (30%) 2 (33%) bandolier 4 (20%) 0 (0%) table 1: test results for comprehensiveness of resources evidence based library and information practice 2008, 3:2 12 not useful if it is not easy to use. conversely, a resource that is easy to use but not comprehensive, may not be as valid for evidence based practice, but it is likely to get used more often due to the high usability. it is clear that there is a high degree of variability in evidence based bedside information tools. evaluation is difficult due to the varied factors that need to be considered when making purchasing decisions. this was a survey of librarians, not health care practitioners. practitioners’ views on these resources may be quite different. previous studies have demonstrated that librarians and health care practitioners place importance on different criteria when evaluating evidence based bedside information tools. (kupferberg and jones hartel) it is not enough to use librarians’ attitudes when judging the appropriateness of a resource that will be used by both librarians and health practitioners. purchasing decisions should be made within the context of how practitioners will use the resources. it is important to have input from all patrons who may use a resource when deciding on what to purchase. this study provides evidence from a librarians’ perspective, but including the physicians’ viewpoint would have resulted in a more balanced view. future research is required to facilitate evaluation of these resources. a similar study using practitioners as searchers, rather than librarians, would provide evidence of the use of the resources in practice and more information on their impact and usefulness. practitioners would also be better qualified to evaluate the quality of the answers provided. this study was limited in that it measured only whether or not the resource gave an answer, rather than the quality or accuracy of the answer. the response rate of 10%-11% is an approximation, not only due to the varied number of canmedlib subscribers, but also due to the fact that people may have forwarded the survey to other colleagues. some institutions may have more representation than others, depending on how participants interpreted the survey. due to the small sample size, the results can give only general indications. this was a study of canadian librarians, thus the results are not representative of all countries. it would be interesting to survey librarians from other countries to identify similarities and differences at an international level. other potential areas for further study include how to make a comprehensive resource such as acp pier more userfriendly and an examination of the usefulness of resources in the pda environment as meaningful bedside information tools. conclusion it is evident that both uptodate and bmj clinical evidence are easy to use and informative. however, the lack of levels of evidence means that these products do not necessarily reduce the time needed to practice evidence based medicine. in contrast, acp pier was identified as being the most comprehensive in terms of inclusion of levels of evidence, although respondents did not believe it to be as easy to use. all three resources have significant weaknesses that need to be addressed in terms of providing comprehensive, reliable and user-friendly evidence to practitioners. although this is a small study with some limitations, librarians making collections development decisions should find it informative. there are different ways to evaluate a product, and all of these must be evidence based library and information practice 2008, 3:2 13 acknowledged before making a confident decision about products. this study is a step towards providing evidence for health librarians involved in collection development and acquisition of evidence based bedside information tools. taking the views of clinicians into consideration when deciding on which products to purchase would further improve the evidence base. works cited alper, brian s., james j. stevermer, david s. white, and bernard g. ewigman. "answering family physicians' clinical questions using electronic medical databases." the journal of family practice 50.11 (nov. 2001): 960-5. alper, brian s., david s. white, and bin ge. "physicians answer more clinical questions and change clinical decisions more often with synthesized evidence: a randomized trial in primary care." annals of family medicine 3.6 (nov. – dec. 2005): 507-13. campbell, rose, and joan ash. "comparing bedside information tools: a usercentered, task-oriented approach." amia annual symposium proceedings (2005): 101-5. campbell, rose, and joan ash. "an evaluation of five bedside information products using a user-centered, taskoriented approach." journal of the medical library association 94.4 (oct. 2006): 435-41, e206-7. fahey, sue. personal interview. 3 mar. 2007. green, michael l., and tanya r. ruff. "why do residents fail to answer their clinical questions? a qualitative study of barriers to practicing evidence-based medicine." academic medicine 80.2 (feb. 2005): 176-82. ketchell, debra s., leilani st. anna, david kauff, barak gaster, and diane timberlake. "primeanswers: a practical interface for answering primary care questions." journal of the american medical informatics association 12.5 (sept.-oct. 2005): 53745. koonce, taneya y., nunzia bettinsoli giuse, and pauline todd. "evidence-based databases versus primary medical literature: an in-house investigation on their optimal use." journal of the medical library association 92.4 (oct. 2004): 407-11. kupferberg, natalie, and lynda jones hartel "evaluation of five full-text drug databases by pharmacy students, faculty, and librarians: do the groups agree?" journal of the medical library association 92.1 (jan. 2004): 66-71. magrabi, farah, enrico w. coiera, johanna i. westbrook, a. sophie gosling, and victor vickland. "general practitioners' use of online evidence during consultations." international journal of medical informatics 74.1 (jan. 2005): 112. magrabi, farah, johanna i. westbrook, enrico w. coiera, and a. sophie gosling. "clinicians' assessments of the usefulness of online evidence to answer clinical questions." medinfo 11.pt.1 (2004): 297-300. rosenberg, william, and anna donald. "evidence based medicine: an approach to clinical problem-solving." bmj 310.6987 (29 apr. 1995): 1122-6. evidence based library and information practice 2008, 3:2 14 sackett, david l., william m. c. rosenberg, j. a. muir gray, r. brian haynes, and w. scott richardson. "evidence based medicine: what it is and what it isn't." bmj 312.7023 (13 jan. 1996): 71-2. schwartz, kendra, justin northrup, nejla israel, karen crowell, nehman lauder, and anne victoria neale. "use of online evidence-based resources at the point of care." family medicine 35.4 (apr. 2003): 251-6. westbrook, johanna i., a. sophie gosling, and enrico coiera. "do clinicians use online evidence to support patient care? a study of 55,000 clinicians." journal of the american medical informatics association 11.2 (mar.-apr. 2004): 113-20. evidence based library and information practice 2008, 3:2 15 appendix 1 survey the purpose of this survey is to gather information on the selection and use of evidence based bedside information tools. for the purpose of this project i am using the following definition to identify these tools: evidence based bedside information tools: web-based, searchable, patient oriented, predigested forms of information. they offer evidence summaries of relevant research that has been critically appraised by scholars in the field and are peer reviewed. please take a few minutes to answer the following questions. thank you very much for your participation. survey: 1. are you a(n): a. librarian b. library staff c. archivist d. other (please specify) _______________________________ 2. in what type of library do you work? a. academic b. hospital c. academic library which also serves a hospital d. government e. public f. special g. other (please specify) _______________________________ 3. does your library subscribe to any of the following evidence based bedside information tools? please check all that apply. a. acp pier b. attract c. bandolier d. best bets e. clinical evidence f. diseasedex g. dynamed h. first consult i. fpin clinical inquiries j. inforetriever k. pepid l. uptodate m. other (please specify) _______________________________ n. not applicable – go to question 17. evidence based library and information practice 2008, 3:2 16 4. what was the decision process used by your library to decide to purchase this (these) products? (open ended answer) 5. what was most important to you when deciding on a product? a. easy to use interface b. levels of evidence c. cost d. comprehensiveness e. pda compatibility f. other (please specify)__________________________________ 6. do you perceive these products to be well used? (open ended) 7. who tends to use these products most often? a. librarians b. physicians c. students d. other (please specify) ___________________________________ for questions 8-16, please use the scale to indicate your agreement with the following statements: (please fill in the name of the product you are using to answer each set of questions. use up to three products (if you have three).) product one __________________ 8. the product is easy to use. 1 2 3 4 9. the product is informative 1 2 3 4 10. the product is efficient (it can answer clinical questions in under 5 minutes) 1 2 3 4 product two __________________ 11. the product is easy to use. 1 2 3 4 12. the product is informative 1 2 3 4 13. the product is efficient (it can answer clinical questions in under 5 minutes) 1 2 3 4 product three __________________ 14. the product is easy to use. 1 2 3 4 15. the product is informative 1 2 3 4 16. the product is efficient (it can answer clinical questions in under 5 minutes) 1 2 3 4 evidence based library and information practice 2008, 3:2 17 for questions 8-16, the following four point scale is used 1 – strongly agree 2 – agree 3 – disagree 4 – strongly disagree 17. if you wish to be included in a random draw to win one of two ipod shuffles, please leave your name and email address. when analyzing responses, names will be omitted. again, thank you for taking the time to respond to this survey. evidence based library and information practice 2012, 7.4 82 evidence based library and information practice article key performance indicators in irish hospital libraries: developing outcome-based metrics to support advocacy and service delivery michelle dalton librarian, hse mid-west library & information services university of limerick limerick, ireland email: michelledalton@gmail.com received: 6 june 2012 accepted: 10 sept. 2012 2012 dalton. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to develop a set of generic outcome-based performance measures for irish hospital libraries. methods – various models and frameworks of performance measurement were used as a theoretical paradigm to link the impact of library services directly with measurable healthcare objectives and outcomes. strategic objectives were identified, mapped to performance indicators, and finally translated into response choices to a single-question online survey for distribution via email. results – the set of performance indicators represents an impact assessment tool which is easy to administer across a variety of healthcare settings. in using a model directly aligned with the mission and goals of the organization, and linked to core activities and operations in an accountable way, the indicators can also be used as a channel through which to implement action, change, and improvement. conclusion – the indicators can be adopted at a local and potentially a national level, as both a tool for advocacy and to assess and improve service delivery at a macro level. to overcome the constraints posed by necessary simplifications, substantial further research mailto:michelledalton@gmail.com evidence based library and information practice 2012, 7.4 83 is needed by hospital libraries to develop more sophisticated and meaningful measures of impact to further aid decision making at a micro level. introduction quantitative measures of performance are an essential management tool in any organization. key performance indicators (kpis) help them meet key strategic objectives, drive and deliver change, and assess the impact and effectiveness of services. appropriate metrics not only provide a high-level snapshot of service levels at any given point in time, but also help to inform the operational activities and tasks that contribute to achieving the key strategic goals of the organization. within the health science library sector in ireland, the primary performance measures are typically input-based metrics, usage statistics, and other operational measures (harrison, creaser, & greenwood, 2011). these statistics typically include gate counts, borrowing totals, the number of books held per staff member, the cost per use of electronic resources, the number of reference queries answered, or the number of information literacy sessions delivered. as largely inputor usage-focused indicators, these measures capture activity levels effectively but represent extremely blunt tools for assessing real effectiveness and impact. in contrast, outcome measures capture the “impact or effects of library services on a specific individual and ultimately on the library’s community” (matthews, 2008, p. xiv). this very evidence is becoming increasingly important in order to promote, and advocate for, the value of health science libraries, and in particular hospital libraries – or as ritchie (2010, p. 1) succinctly advises: “knowing why you exist (not simply what you do).” in this respect hospital libraries have a unique raison d’être. they are required to support a number of mission critical goals within the institution from “saving hospitals thousands of dollars per year to saving patients’ lives” (holst et al., 2009, p. 290). the value chain within which hospital libraries must position themselves requires: providing the right information at the right time to enhance medical staff effectiveness, optimize patient care, and improve patient outcomes … save clinicians time, thereby saving institutions money… provide an excellent return on investment for the hospital, playing a vital role on the health care team from a patient’s diagnosis to recovery. (holst et al., 2009, p. 290) however, given the absence of any robust quantitative evidence regarding the value contributed by hospital libraries in ireland, such claims remain largely unsubstantiated, and may even appear merely aspirational to some. the use of performance indicators is now commonplace across nearly all aspects of the irish healthcare sector, including public health services which are administered by the health service executive (hse). in recognition of the need for an effective assessment tool, the hse designed and implemented the healthstat system. the indicators incorporated in healthstat provide an overview of how services are delivered using a broad range of various performance measures. notably, however, there are currently no library-related service indicators included within the system, or indeed within any of the systematic or standardized assessment frameworks which are implemented by the hse (health service executive, 2011). indeed, the report on the status of health librarianship & libraries in ireland (shelli ) articulated the pressing need for health science libraries in ireland to establish “a body of evidence based library and information practice 2012, 7.4 84 evidence, with performance indicators, available at the level of individual libraries and nationally, used for service promotion and advocacy” (emphasis added) (harrison et al., 2011, p. 42). in this context, the aim of this study was to develop a potential set of performance measures sufficiently general to be applied to other libraries in broadly similar settings nationally, whilst still retaining some value at a local level (in this case, a library based within an acute hospital). literature review an outcome based approach to measurement effective performance measurement intrinsically requires measuring the “right” things in the “right” way. it is a complex task, however, to distill a library’s core activities, functions, and goals into a narrowly defined, yet sufficiently powerful, set of indicators. kpis can be viewed through a variety of lenses, including:  the goal attainment model driven by strategic objectives  the systems resource model of input measures  the internal systems model derived from workflows and communications processes  the multiple constituencies model based on the extent to which different stakeholders’ needs are met (cameron, 1986) in the context of impact assessment, the goal attainment model offers a particularly good fit. within this framework, the inputs (e.g., operational activities and decisions) that drive performance indicators should also impact on the organization’s strategic objectives and desired outcomes (hauser & katz, 1998). for instance, a desired objective of a hospital may be to deliver efficient and timely patient care, and a relevant performance indicator for the library could be to save clinicians’ time as a result of using library services. corresponding inputs may include reconfiguring or streamlining workflows and staffing arrangements in order to reduce the response times for clinical information queries. these same inputs should also impact on the overarching objective of efficient patient care, which in this case is a reasonably logical hypothesis, a priori. these relationships and interdependencies also mirror boekhorst’s (1995) model of performance measurement, which emphasises the direct links between goals and objectives, and performance measurement and activities, allowing operational tasks to be consistently aligned with strategic aims. figure 1 boekhorst’s model of performance measurement (md ishak & sahak, 2011, p.5) evidence based library and information practice 2012, 7.4 85 matthews’s (2008) balanced scorecard model adapted from kaplan and norton (1992) also adopts a strategic focus with respect to measurement. performance indicators should reflect the organization’s strategy across four perspectives: financial, customer, internal business processes, and learning and growth. outcome measures that help to assess key strategic objectives can play an important role as part of this approach. matthews breaks the concept of outcomes into immediate, intermediate, and ultimate outcomes, shifting in focus from creating value at the individual level to the overall impact on the organization. effectively measuring these outcomes can help the organization to communicate its long-term or strategic value to users. donabedian’s (1966) seminal work on evaluating medical care frames the concept of quality assessment within a model of structures, processes, and outcomes – in other words, the resources used by the organization, the activities carried out in healthcare delivery, and the outcomes on patient care. appropriate performance indicators can be used within the framework to capture and measure key elements in this chain, therefore helping to assess the overall quality and performance of the healthcare system. however, libraries are perhaps guilty of an overreliance on a structural approach in this regard, by focusing on measuring the inputs and resources used to deliver desired outcomes, in spite of “the major limitation that the relationship between structure and process or structure and outcomes, is often not well established” (p. 695). this argument reinforces the need for objective outcome-based indicators in order to assess the performance of hospital libraries in a valid and meaningful way. “ultimately, the goal of health care is better health, but there are many intermediate measures of both process and outcome” (who, 2003, p. 5). holst et al. (2009) identify three core channels through which hospital libraries can potentially add real value: patient outcomes, time savings, and cost savings. these variables are not exogenously determined, and indeed saving the time of clinical staff will also, all other things being equal, reduce costs and improve patient outcomes as staff can treat more patients in the same amount of time. by focusing on these three channels, this study is limited to an attempt to capture and assess the value of library and information services to clinical practice and outcomes, rather than the contribution which hospital libraries also make to research output. the latter is obviously another channel through which hospital libraries add value, but constructing a suitable indicator to measure this variable is outside the scope of this study. this also largely reflects the core mission of the hse in its focus on patient care (health act, 2004). developing well-designed and actionable indicators loeb contends that “the central issue in performance measurement remains the absence of agreement with respect to what should be measured” (2004, p. i7). the health information and quality authority of ireland (hiqa) recommends that performance indicators used to measure healthcare quality should exhibit certain properties:  provide a comprehensive view of the service without placing an undue or excessive burden on organizations to collect data.  be explicitly defined and based on highquality and accurate data.  measure outcomes which are relevant and attributable to the performance of the healthcare system in which they are employed.  not be selected based solely on the availability of data.  be supported by local measures in order to inform practice and operations at a local level. (hiqa, 2010, p. 20-1) evidence based library and information practice 2012, 7.4 86 these principles provide a baseline standard for performance measures in this study. parmenter (2010) extends these properties further, outlining the typical attributes of effective kpis as measures that are non-financial, frequently measured, acted on by senior management, indicating the necessary action required by staff, tying responsibility down to a team or individual, having a significant impact, and encouraging appropriate action. here the emphasis on action is notable. frequently data may be collected as a matter of routine or obligation but not effectively utilized or acted upon. however, in contrast with usage statistics or input-based metrics, outcome-based indicators are, by definition, directly driven by core strategic objectives, and therefore are inextricably linked with the actions supporting these goals. consequently, outcome-derived kpis can be channelled more easily into concrete, actionable insights, resulting in real changes in systems, processes, and services. supporting good governance through assessment and accountability quantitative performance indicators also have a role to play in providing an objective assessment of services and in both internal and external consistency in the decision-making process. external reporting, transparency, and compliance are critical dimensions of good governance. the organizational structure of hospital libraries is also changing (harrison et al., 2011). staffing pressures dictate that an increasing number of healthcare libraries in ireland are likely to be run by solo librarians in the future, with a single individual having responsibility for managing all aspects of library services. this further increases the need for external and objective measures to serve as a verifiable cross-check on services. appropriate outcome-based kpis stimulate action in a way which also attributes responsibility. as it must be made clear who “owns” each indicator, this increases accountability within the organization. achieving this buy-in successfully in practice requires building an environment centred on trust, whereby performance measures and targets are clearly communicated, understood, and accepted as fair by all staff and stakeholders. however, if kpis are derived directly from strategic objectives and outcomes, it is often easier for the individuals concerned to see the relevance of and need for such measures, and staff are therefore more likely to view assessment in a positive way. evidence based advocacy a significant body of literature already exists on the importance of impact assessment as a tool for advocacy in hospital libraries outside ireland. weightman and williamson’s systematic review of library impact (2005) appraises 28 studies which each assess at least one direct clinical outcome. survey instruments are the most frequently used method of data collection, but in most cases the limitation of “desirability bias” (p. 6) arising from selfselection is highlighted as a weakness. twenty different impact measures are recorded from the studies based in the traditional library setting, indicating that some level of variation exists as to which outcomes are perceived as the most critical in influencing patient care. the landmark rochester study (marshall, 1992) is included in weightman and williamson’s (1995) review. the historical context, marked by a change to u.s. federal requirements regarding hospital library provision in 1986, sparked the need for potent advocacy tools to improve the “visibility and status of the library” by expressing value in “the bottom line,” that is, the impact on clinical decision making (p. 170). the study adopted a relatively detailed approach in measuring the impact of library use specifically on physicians’ practice, including aspects such as the choice of tests, drug treatment, and patient advice. as it is hoped that the indicators generated by this research can be used to demonstrate the value of services to a broader range of health and social care professionals evidence based library and information practice 2012, 7.4 87 (including management and administrative staff) reflecting a multidisciplinary approach to healthcare, this level of detail was rejected in favour of higher level indicators to avoid causing confusion by presenting respondents with irrelevant or excessive choices. moreover, as the shelli report (2010) indicates, there is a need for a national measure, and as significant heterogeneity exists across regions and local healthcare facilities across ireland, some simplification is unavoidable. further qualitative research, such as structured interviews which could be tailored to a specific discipline or local context, could help to pinpoint and elucidate concrete or specific examples of impact, but this falls outside the scope of the present study. in some healthcare institutions there has been an increased shift towards outsourcing, redeployment, and the use of “shared services” models in recent years, precipitated by the economic and political landscape (harrison et al., 2011). libraries have not been immune to such developments, and indeed have even been seen by some as easy targets in the potential for cost savings (geier, 2007). however, ritchie contends that in many cases such decisions are in fact based on clear economic arguments that stand up to valid and rigorous cost-effectiveness analysis. these evidence based financial rationales pose “very real threats to our survival, and serious challenges to our ability to develop and thrive; and we have to be able to justify our existence in their terms” (2001, p. 1). but the shelli report notes that “there is currently little, if any, evidence of the impact of health information services – how the use of library services and/or resources feeds into direct patient outcomes or financial benefits” (harrison et al., 2011, p. 7). it is unclear why there is a lack of such measures within the irish hospital library environment. it can perhaps be in partly attributed to the lack of data and integrated evidence base within the healthcare sector generally. indeed levis, brady, and helfert (2008) draw attention to this problem, noting that: computerised information systems have not as yet achieved the same level of penetration in healthcare as in manufacturing and retail industries. in ireland many serious errors and adverse incidences occur in our healthcare system as a result of poor quality information. (p. 1) however, initiating a culture of accountability and outcome-based performance assessment can and should be a positive development for libraries, and one which provides a rare and valuable opportunity to leverage evidence based advocacy. librarians as a profession may understand the benefits that effective information services can offer to an organization, but this is not enough. hospital libraries must articulate and verifiably demonstrate the value of their services in the language which is understood by the commercial and corporate world: that is, by expressing their services as strategic objectives, outcomes, and value for money. methods the study is underpinned by a broadly positivist approach, and various models and frameworks are used as a theoretical paradigm to inform the development of a potential set of quantitative performance indicators. as the aim of this research specifically relates to the impact assessment of library services on measurable healthcare outcomes and objectives, cameron’s (1986) goal attainment model was selected as the most appropriate framework within which to place the analysis. boekhorst’s (1995) model of performance measurement was used as a lens through which to identify and analyze the relationships and links between the hse’s mission and strategic objectives, and library performance and activities. this reflected the need to look beyond the mission and goals of the library itself towards those of the parent evidence based library and information practice 2012, 7.4 88 organization. for the purpose of this study, the model was simplified slightly by combining the goals and objectives into a single element. in order to construct outcome-based indicators, a clear picture was needed of the organization’s mission, and the strategic objectives and desired outcomes of the acute hospital sector in ireland. organizational mission the hse was established under the health act 2004 as the single body with statutory responsibility for the management and delivery of health and personal social services in the republic of ireland. as outlined in the act, “the objective of the executive is to use the resources available to it in the most beneficial, effective and efficient manner to improve, promote and protect the health and welfare of the public” (2004, pt. 2, s.7). this statement was selected as the overall organizational mission for the model. strategic objectives and outcomes given the aim of developing a set of performance indicators which are sufficiently broad to be applied across multiple hospital settings, the key strategic objectives were identified from the report of the national acute medicine programme (2010), a framework document for the delivery of acute medical services to improve patient care. the report highlights eight overarching aims of the programme as follows: 1. safe, quality care. 2. expedited diagnosis. 3. the correct treatment. 4. an appropriate environment. 5. respect of their [patients’] autonomy and privacy. 6. timely care from a senior medical doctor working within a dedicated-multidisciplinary team. 7. improved communication. 8. a better patient experience. (health service executive & royal college of surgeons ireland, 2010, p. 1) these objectives clearly do not operate exogenously, and there is likely to be some correlation between them. therefore, in view of the need to develop a set of pragmatic, measurable and high-level indicators, these eight individual objectives were assessed and grouped together based on commonality. from this process, three primary objectives emerged: quality of patient care, safety of patient care, and efficiency/speed of patient care. these objectives are also congruent with holst et al.’s (2009) analysis of the channels through which hospital libraries can deliver value: improved patient outcomes, saving clinician’s time, and reducing costs. furthermore, they also directly mirror three of the five core domains of healthcare quality which are identified by hiqa (2010). the two remaining dimensions, equality of care and person-centredness, were not included, as library services were viewed to have limited, if any, influence over these aspects. key performance indicators the final stage required mapping these conceptual objectives into a set of explicit indicators. as well as being directly linked to the organizational mission and objectives, it was critical that the indicators should also be consistent with the recommendations outlined in hiqa’s guidance on developing key performance indicators and minimum data sets to monitor healthcare quality (2010). a comprehensive literature review was undertaken to identify the primary operational factors that influence the quality, safety, and efficiency/cost of healthcare, which library services can also support. however, these three concepts are broad and complex variables that can be measured and assessed through myriad evidence based library and information practice 2012, 7.4 89 different indicators. even 50 years on, klein’s (1961, p. 144) conclusion that “there will never be a single comprehensive criterion by which to measure the quality of patient care” still holds some degree of weight. for this reason, broad indicators relating to improvement in patient care or practice were chosen as proxies, rather than drilling down into more specific diagnostic or therapeutic outcomes – reflective of donabedian’s general “yardstick” of specificity rather than a “watertight, logic-system” (1966, p. 703). whilst this may represent a somewhat vague and normative standard open to an element of ambiguity as to what constitutes improvement or reduction, it was viewed as a necessary compromise, given the need to apply the indicators across disparate health and social care contexts to reflect a multidisciplinary approach, and indeed varying hospital environments. data collection and administration of survey a survey questionnaire was selected as the data collection instrument to measure the indicators due to the simplicity and cost of administration. one of the key aims of the questionnaire design process was to ensure that the burden on respondents was minimized. this is particularly germane to the healthcare setting, as doctors typically exhibit a low to moderate response rate to survey questionnaires (olmsted, murphy, mcfarlane, & hill, 2005). indeed, poor response rates to previous surveys required an innovative approach as to how the instrument could be packaged effectively to busy clinical and management staff to best encourage response. for this reason, the survey was deliberately branded as “one question” rather than as a survey, to highlight the simplicity and minimal time commitment involved on the part of the respondent. to produce the final survey, the five performance indicators were incorporated as possible responses to the question: “how did the information provided by the library help?” hospital staff are also free to indicate that the information provided had no impact or effect. in phrasing both the question and responses, the plain language style guide for documents (hse & nala, 2009) was consulted to ensure clarity of expression. the survey was also piloted with a number of clinical and library staff to ensure that it was easy to interpret and understand. the online survey tool surveymonkey was used to administer the question, and a survey link was included with the responses to any clinical information or reference queries of substance. it is difficult to classify “substance” in an objective way across all local contexts, however it is generally assumed to refer to strategyor consultation-based queries, as clarified by warner (2001). in practice, this refers to complex mediated literature searches, where a full search report and supporting documentation are returned to the user. such queries are received from healthcare (physicians, nurses, and allied health professionals) and health management staff, and thus the potential survey population is relatively disparate. this in turn necessitates the need for the survey responses to be phrased using terminology sufficiently general to be applicable across a range of hospital contexts. as the link is included in response to a specific, individual query or transaction, it is clear to the user that the survey relates explicitly to this particular interaction rather than to library or information services in a more general sense. no explicit incentive is offered to encourage completion of the survey, but as it is included within personalized correspondence and search results, rather than as a generic promotional email, this may in itself prompt users to respond after they have assessed the information. all responses received are anonymous and subjects are made aware of this. the survey instrument has been designed to be replicated in other similar hospital libraries, so that data can be pooled in order to generate significant sample sizes for future analysis and interpretation. we plan to analyze and report survey results every six months, with the provision that sample sizes are sufficient to generate meaningful insight. evidence based library and information practice 2012, 7.4 90 results informed by the organizational mission and three core strategic objectives listed above, the final performance indicators selected were: 1. influence on patient care or guiding of clinical practice/policy. 2. length of hospital stay. 3. referral to another department. 4. staff time. 5. risk/error reduction. at the local level, these indicators can subsequently be mapped to a corresponding set of core library activities and tasks that influence each measure, that is, library-specific inputs such as staff workflows, resources, and local systems to help support decision making. when translated into survey responses, these indicators were presented to staff through the surveymonkey interface as illustrated in figure 3. results from the responses received during the six-month period since the survey was initially introduced are illustrated below. a total of 93% of staff stated that the information provided by the library had saved them time, and 86% claimed it had influenced their decision on patient care, clinical practice, or policy – a broadly similar proportion to that estimated in the rochester study with significantly larger sample sizes (marshall, 1992). with over half of respondents indicating that risk or errors had also been reduced, these results suggest at least some positive impact of library services on key strategic objectives. no respondents indicated figure 2 flowchart of strategic objectives and indicators (adapted from boekhorst, 1995) evidence based library and information practice 2012, 7.4 91 that the information failed to have any effect or impact; however it is likely that this is in part due to self-selection bias from the nature of the survey – a common limitation, as highlighted by weightman and williamson (2005). further research would be needed to estimate the extent to which this factor influences the overall results. as a tool for advocacy, a single snapshot of data offers some value, but as performance indicators, survey results are only really meaningful when compared over time or in a cross-sectional context, and so these initial results are of limited value in assessing library services in relative terms. to date the survey has been rolled out in only one regional area. thus, while of value at a local level, the real potential for a national-level indicator remains untapped. discussion application as a tool in practice the need for objective and quantitative performance measures in hospital library settings is clear (harrison et al., 2011; ritchie, 2010). outcome-based measures that reflect critical outputs and outcomes are invariably more visible than demand-derived metrics, which offer little or nothing from a marketing and advocacy point of view (chan & chan, 2004). if an instrument such as this survey could be applied in a standardized way to produce a national measure of performance, benefits would likely accrue, not only in fostering a culture of objective and continuous assessment to drive local service improvements, but also as a valuable tool for evidence based advocacy. furthermore, extending the survey more widely would also offer the potential to obtain larger sample sizes for increased reliability, precision, and statistical power. more sophisticated analysis may also be possible. data could be used to identify any statistically significant differences across hospitals or regions. in addition, results could be used to estimate the correlation, if any, between the outcomes achieved and levels of inputs (for example, library budgets or staff numbers) either on a cross-sectional or time series basis. notwithstanding these advantages, there is no guarantee that using these performance indicators will deliver a real change in hospital or clinical practice. improvement requires more than tracking and monitoring data and identifying problems. it must be accompanied by real action, buy-in, and commitment from stakeholders in a visible and accountable way. in essence, “providing managers and staff with accurate, intuitive, and easily interpretable data figure 3 presentation of online survey question to users evidence based library and information practice 2012, 7.4 92 is one third of the recipe for improvement. the other ingredients are alignment with strategic objectives and a system for accountability” (wadsworth, 2009, p. 69). in view of this, it is hoped that by developing the indicators using a model directly aligned with the strategic mission and goals of the organization and linked to core activities and operations in an accountable way, the performance measures documented above will also facilitate real and meaningful followthrough on change. limitations the questionnaire developed for this article represents a substantial simplification in assessing the efficiency and quality of patient care, and intrinsically represents a selfassessment by the user. firstly, positive selfselection (or desirability bias as discussed in weightman & williamson (1995)), whereby those who find the service of greater value are more motivated to respond to the survey question, may introduce bias into the results. indeed it is a more-than-plausible hypothesis that those who do not value the library’s services will simply not respond to the questionnaire. it is likely that this is in part responsible for the fact that no respondents indicated that the information provided to them had no effect or impact. as the survey is directly linked to a specific transaction, it also excludes non-users by definition – a further limitation. moreover, whilst staff may claim that the research and information support provided by the library saved them time or reduced the risk of errors in their practice, this assessment may be subject to bias or variance in interpretation among the respondents. there is something of a catch-22 at play in this respect. the need for widely applicable indicators for the reasons outlined above necessitates a significant degree of generality in specification. however, this same generality leads to an increased dependence on “the interpretations and norms of the person entrusted with the actual assessment” (donabedian, 1966, p. 704). striking the right balance between both needs is a challenge. the value of the results generated by the survey could be significantly enriched by additional qualitative data obtained through interviews or focus groups. a mixed methods approach such figure 4 responses received from online survey evidence based library and information practice 2012, 7.4 93 as this would help to capture the story behind the quantitative headlines, and also yield greater insight into why, when, and how hospital staff use the library. as survey responses are anonymous, a separate recruitment process would have to be undertaken to identify potential interviewees, and to include non-users also to help address the aforementioned limitations. conclusion in spite of the limitations outlined above, the absence of any real outcome-based measure within the irish hospital library sector is simply too pervasive to ignore. whilst the indicators and data collection framework proposed in this study may be formative and incipient at best, there is a clear need for evidence of impact to help fill the gap which exists at present between library services and hospital outputs, outcomes, and objectives. perhaps it is time for irish hospital librarians to redirect some of their time and effort away from collecting solely input and usage focused metrics, and towards developing meaningful outcome-based measures? given that efficacy is such a key driver in healthcare, the former are of limited insight, whilst “the validity of outcome as a dimension of quality is seldom questioned” (donabedian, 1966, p. 693). instead of focusing on measuring activities and inputs in isolation as libraries have often done in the past, adopting an outcome-based model allows key objectives and the need for accountability to drive service delivery, ultimately ensuring that library services remain relevant to, consistent with, and of direct value to the organization. traditional measures of activity can still tell a valuable story, but an alternative narrative is also required. given the aim of creating broadly generic indicators which are measureable in practice and transferable across a variety of contexts (internal and external), simplification is a pragmatic and necessary constraint, but as tukey argues: “far better an approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise” (1962, p. 13). it is hoped, therefore, that this initial framework can provide a platform for irish hospital libraries to assess performance at a macro level. performance indicators should allow us to answer two critical questions: “are we still relevant to the organization? and if not, why not?” until we can answer these questions objectively, we must continue the search for valid and meaningful measures of performance. references boekhorst, p. (1995). measuring quality: the ifla guidelines for performance measurement in academic libraries. ifla journal, 21(4), 278-281. cameron, k. s. (1986). effectiveness as paradox: consensus and conflict in conceptions of organizational effectiveness. management science 32(5): 539-553. doi: 10.1287/mnsc.32.5.539 chan, a. p. c., & chan, a. p. l. (2004). key performance indicators for measuring construction success. benchmarking: an international journal, 11(2), 203-221. doi: 10.1108/14635770410532624 donabedian, a. (1966, 2005) evaluating the quality of medical care. milbank quarterly, 44: 166-203. reprinted in milbank quarterly, 83(4), 691–729. doi: 10.1111/j.1468-0009.2005.00397.x geier, d. b. (2007). prevent a disaster in your library: advertise. library media connection, 25(4), 32-33. harrison, j., creaser, c., & greenwood, h. (2011). irish health libraries: new directions: report on the status of health librarianship & libraries in ireland (shelli evidence based library and information practice 2012, 7.4 94 ). retrieved 13 oct. 2012 from http://hdl.handle.net/10147/205016 hauser, j. r., katz, g. m., & international center for research on the management of technology. (1998). metrics: you are what you measure! working papers 172-98, massachusetts institute of technology (mit), sloan school of management. retrieved 13 oct. 2012 from http://web.mit.edu/~hauser/www/paper s/hauser-katz%20measure%2004-98.pdf health act 2004 (ireland). (2004). in irish statute book. retrieved 13 oct. 2012 from http://www.irishstatutebook.ie/pdf/2004 /en.act.2004.0042.pdf health information and quality authority (2010). guidance on developing key performance indicators and minimum data sets to monitor healthcare quality. dublin: hiqa. retrieved 13 oct. 2012 from http://www.hiqa.ie/system/files/hi_kpi _guidelines.pdf health service executive (2012). hse west service plan 2012. ireland: hse west. retrieved 13 oct. 2012 from http://www.hse.ie/eng/services/publicati ons/corporate/westserviceplan2012.pdf health service executive (2011). healthstat – metrics and targets. retrieved 13 oct. 2012 from hse website http://www.hse.ie/eng/staff/healthstat/ metrics/ health service executive health promotion unit & national adult literacy agency (2010). plain language style guide for documents. dublin: health service executive. retrieved 13 oct. 2012 from http://hdl.handle.net/10147/98048 health service executive & royal college of surgeons ireland. (2010). report of the national acute medicine programme. dublin: health service executive. retrieved 13 oct. 2012 from http://www.hse.ie/eng/services/publicati ons/services/hospitals/amp.pdf holst, r., funk, j. c., adams, h. s., bandy, m., boss, c. m., hill, b., joseph, c. b., & lett, r. k. (2009). vital pathways for hospital librarians: present and future roles. journal of the medical library association, 97(4), 285–292. doi: 10.3163/15365050.97.4.013 kaplan, r. s., & norton d. p. (1992). the balanced scorecard: measures that drive performance. harvard business review, 69(1), 71–79. klein, m. w., malone, m. f., bennis w. g., & berkowitz, n. h. (1961). problems of measuring patient care in the outpatient-department. journal of health and human behavior, 2(2), 138–144. levis, m., brady, m., & helfert, m. (2008). information quality issues highlighted by deming’s fourteen points on quality management. international conference on business innovation and information technology (icbiit), 24 january 2008, dublin city university, ireland. retrieved 13 oct. 2012 from http://www.tara.tcd.ie/bitstream/2262/40 479/1/icbiic_v2.pdf loeb, j. m. (2004). the current state of performance measurement in health care. international journal for quality in health care, 16(suppl.1): i5-i9. marshall, j. g. (1992). the impact of the hospital library on clinical decision making: the rochester study. bulletin of the medical library association, 80(2): 169-78. retrieved 13 oct. 2012 from http://www.ncbi.nlm.nih.gov/pmc/articl es/pmc225641/pdf/mlab00115-0079.pdf evidence based library and information practice 2012, 7.4 95 matthews, j. r. (2008). scorecards for results: a guide for developing a library balanced scorecard. westport, ct: libraries unlimited. md ishak, a. h., & sahak, m. d. (2011). discovering the right key performance indicators in libraries: a review of literatures. paper presented at the 1st perpun international conference and workshop on key performance indicators for libraries 2011, 17-19 october 2011, universiti teknologi malaysia, johor bahru. retrieved 13 oct. 2012 from http://eprints.ptar.uitm.edu.my/4120/1/ d1_s2_p3_330_400_amir_hussain__ 6.pdf olmsted, m. g., murphy, j., mcfarlane, e., & hill, c. a. (2005). evaluating methods for increasing physician survey cooperation. paper presented at the 60th annual conference of the american association for public opinion research (aapor), miami beach, fl, may 2005. retrieved 13 oct. 2012 from http://www.rti.org/pubs/olmstedpresent ation.pdf parmenter, d. (2010). key performance indicators: developing, implementing, and using winning kpis. new jersey: wiley. ritchie, a. (2010, march 29-april 1). thriving not just surviving: resilience in a special library is dependent on knowing why you exist (not simply what you do). paper presented at alies 2010 conference: “resilience” 29 mar-1 apr 2010. retrieved 13 oct. 2012 from http://www.em.gov.au/documents/ann %20ritchie%20-%20nt.pdf tukey, j. w. (1962). the future of data analysis. annals of mathematical statistics, 33(1), p. 1-67. wadsworth, t., graves, b., glass, s., harrison, a.m., donovan, c., & proctor, a. (2009). using business intelligence to improve performance. hfm (healthcare financial management), 63(10), 68-72. warner, d. g. (2001). a new classification for reference statistics. reference & user services quarterly, 41(1): 51-55. weightman, a. l., & williamson, j. (2005). the value and impact of information provided through library services for patient care: a systematic review. health information & libraries journal, 22(1): 425. doi: 10.1111/j.1471-1842.2005.00549.x. who regional office for europe’s health evidence network (hen) (2003). how can hospital performance be measured and monitored? copenhagen: who. retrieved 13 oct. 2012 from http://www.euro.who.int/__data/assets/ pdf_file/0009/74718/e82975.pdf evidence summary   vanished open access journals; why preservation is needed   a review of: laakso, m., matthias, l., & jahn, n. (2021). open is not forever: a study of vanished open access journals. journal of the association for information science & technology, 72(9), 1099–1112. https://doi.org/10.1002/asi.24460    reviewed by: kathy grams associate professor of pharmacy practice massachusetts college of pharmacy and health sciences boston, massachusetts, united states of america email: kathy.grams@mcphs.edu   received: 26 feb. 2023                                                                  accepted:  6 apr. 2023      2023 grams. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30337     abstract   objective – to determine the number of open access journals that have vanished from the web and to summarize their publishing lifespan, geographical and disciplinary characteristics.   design – a descriptive research study.   setting – the internet and internet archive.   subjects – open access journals.   methods – to identify vanished open access (oa) journals, vanished was defined by the authors as “a journal that published at least one volume as immediate oa after which production ceased, and the journal, together with the published full-text documents, disappeared from the web.” if the journal content partially existed, it would be considered as vanished if <50% was available during 12 months of data collection which occurred september 2019–september 3, 2020. in 2020, the oa journal list was created by searching directory of open access journals (doaj), ulrichsweb global serials directory, and scopus title lists. the list was cross-referenced with database records from doaj from 2010–2012, 2012–2014, and 2014–2019; ulrichsweb title lists from may 24, 2012, and july 3, 2018; and scopus title lists from february 2014 and april 2018, to determine the missing titles. previous research by the primary author and two peers, and previous publications, also contributed to the list of vanished journals. data was collected manually, and duplicates were removed. authors searched the keepers registry to be sure that the journal content was not preserved or accessible. only titles with an issn number were kept in the final list. the authors then searched indexing databases and google to find the vanished journal’s website, then accessed the website through the internet archive's wayback machine to gather the information such as the year founded, last year of publication, last year available online, language, country, affiliation, and academic discipline.   main results – authors identified 154 completely vanished journal titles and 20 partially vanished journal titles, to total 174 verified titles. journals originated from 47 countries; the majority were published in english (n=137), and most were from north america, europe and central asia (n=109). social sciences and humanities domain represented 52.3% or 91 titles, and the last publication year of most titles occurred between 2010 and 2014 (n=110). the authors estimated the average time of the last published issue to the last available time on the internet to be within 1 year for 68 titles and within 5 years for 144 titles.   conclusion – although the results represent a small number of the available oa journals at the time of the study (1.2%), it reinforces the authors’ theme that “open is not forever” and raises concern of the potential loss of scholarly work.   commentary   the cat generic critical appraisal tool created by perryman & rathbun-grubb (2014) was used to assess the quality of this study.   laasko et al. conducted a valuable literature review in addition to their exploratory study. their methods were detailed, and their results were presented in a clear and understandable way. the only downgrade was the limitation of tracking something that is not there. the authors admit that this was a challenge.   because there is no general agreement on the party or parties responsible for preserving scholarly oa articles and they are at risk of loss if they are discontinued, laasko et al. set out to establish the number and characteristics of oa journals that have ceased to be published and have not been preserved. they stated that a data source that tracks the availability of journals over time does not exist. they used indexing and abstracting services to create a dataset of active records of journals and compared them to historical datasets from doaj and ulrichsweb, the earliest set dated 2010–2012. this was not a small task as more than 14,000 oa journals were included on doaj in 2020. however, the doaj service started with a list of 300 oa journals in 2003 (doaj, 2023). it is unclear if all journals that ceased to be published between 2003 and 2010 would have been present on the dataset from 2010-2012. previous research projects conducted by the authors and previous published studies were also consulted and resulted in identifying a small number (n=37) of vanished journal titles prior to 2010.   authors mention that only titles that included an issn or e-issn number were kept and this number was used to determine if the journal was participating in digital preservation. the keepers registry only lists titles by issn number. the authors do not mention how many titles from their dataset did not have an issn number and could not be searched.   the limitations of searching for something that is not there did not prevent the authors from identifying 174 oa scholarly journals that have vanished from the web. they used reputable resources and valuable tools to classify journals in the social sciences, health sciences, life sciences, and physical sciences; sciences that are still at risk of losing scholarly articles. in their research, laasko et al. identified 900 journal titles that were inactive and not preserved, thus at risk for vanishing.   the authors do well in describing the need for a system of tracking active and inactive journals over time. they also emphasize that “open is not forever” and that there is a need to preserve scholarly oa journals before their content vanishes completely.   academic libraries, already committed to preserving the content of purchased or subscription-based content, are in an ideal position to identify and include oa journals relevant to their institution for digital preservation. however, there are questions whether preservation is considered the function of the journal publisher. protecting scholarly records is complex, requires time, finances, collaboration, and standards to govern and support the process, therefore it is important for both librarians and publishers to investigate methods to improve the lifecycle of oa scholarly journals that are not already archived using a preservation service.   references   doaj. (directory of open access journals). (2023, february 20). doaj at 20 – open, global, and trusted since 2003. doaj news service. https://blog.doaj.org/2023/02/20/doaj20-open-global-and-trusted-since-2003/   laakso, m., matthias, l., & jahn, n. (2021). open is not forever: a study of vanished open access journals. journal of the association for information science & technology, 72(9), 1099–1112. https://doi.org/10.1002/asi.24460    perryman, c. & rathbun-grubb, s. (2014). the cat: a generic critical appraisal tool. http://www.jotform.us/cp1757/thecat   evidence summary   planning, coordinating, and managing off-site storage is an area of increasing professional responsibility for special collections departments   a review of: priddle, c., & mccann, l. (2015). off-site storage and special collections: a study in use and impact in arl libraries in the united states. college & research libraries, 76(5), 652-670. doi:10.5860/crl.76.5.652   reviewed by: melissa goertzen collection development analysis & support librarian columbia university libraries new york, new york, united states of america email: mjg2227@columbia.edu   received: 6 nov. 2015    accepted: 1 feb. 2016      2016 goertzen. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to measure the use of off-site storage for special collections materials and to examine how this use impacts core special collections activities.   design – survey questionnaire containing both structured and open ended questions. follow-up interviews were also conducted.   setting – association of research libraries (arl) member institutions in the united states of america.   subjects – 108 directors of special collections.   methods – participants were recruited via email; contact information was compiled through professional directories, web searches, and referrals from professionals at arl member libraries. the survey was sent out on october 31, 2013, and two reminder emails were distributed before it closed three weeks later. the survey was created and distributed using qualtrics, a research software that supports online data collection and analysis. all results were analyzed using microsoft excel and qualtrics.   main results – the final response rate was 58% (63 out of 108). the majority (51 participants, or 81%) reported use of off-site storage for library collections. of this group, 91% (47 out of 51) house a variety of special collections in off-site storage. the criteria most frequently utilized to designate these materials to off-site storage are use (87%), size (66%), format (60%), and value (57%). the authors found that special collections directors are most likely to send materials to off-site storage facilities that are established and in use by other departments at their home institution; access to established workflows, especially those linked to transit and delivery, and space for expanding collections are benefits.   in regard to core special collections activities, results indicated that public service was most impacted by off-site storage. the authors discussed challenges related to patron use and satisfaction. in regard to management and processing, directors faced challenges using the same level of staff to maintain two locations instead of one. also, the integration of new workflows required additional oversight to ensure adequate control at all points of process. static staffing levels and increased levels of responsibility impacted preservation and conservation activities as well. a central concern was the handling of materials by facility staff not trained as special collections professionals. in regard to the facilities themselves, a general concern was that commercial warehouses do not always provide the kind of environmental control systems recommended for storage of special collections materials.    of the total sample group, 12 participants (19%) said their institution does not use off-site storage for special collections. when asked if this may occur in the future, four directors (33%) said they anticipate off-site storage use within the next five years. lack of space was listed as the primary motivation.   conclusion – study findings provide evidence for what was previously known anecdotally: planning, coordinating, and managing off-site storage is a significant professional responsibility that will only grow in the future. as primary resources are integrated into research, teaching, and learning activities, the acquisition of special collections materials will continue to grow. discussions regarding off-site storage workflows and strategic planning will continue as professionals seek compromises that meet the unique needs of acquisition, preservation, and public service.   commentary   every day library professionals consider how to make the most of a precious resource: library space. one strategy is investment in off-site storage facilities. the authors stated that the implementation of off-site storage by arl member libraries increased during the last three decades. benefits of off-site storage include preservation-quality environmental conditions and convenient storage of materials; challenges are linked to a lack of direct patron access and removal of collections from library stacks (deardorff & aamot, 2006).   despite a large body of professional literature that addresses advantages and challenges connected to off-site storage, few studies explore its impact on special collections. two notable exceptions are papers by lafogg and weideman (2001) and sundstrand (2008, 2011), which examine the preparation and planning required when relocating archival materials. the study at hand provides evidence for the impact, both positive and negative, of off-site storage on core special collection activities.     the strengths of the study include the suitability of the methodology to the central research question, well-defined criteria for the selection of participants, and the clear presentation of data collection strategies and study findings. the value of the study lies in its uniqueness: through the survey tool, the authors capture observations, thoughts, and opinions regarding the impact of off-site storage on acrl competencies such as public service, management, preservation, and processing. the findings provide evidence for what was previously known anecdotally and provide a baseline for future studies.   one limitation the reviewer found is that the authors did not provide an operational definition of the term “off-site storage” to survey participants. as this term conjures up varying connotations, a definition may have provided greater clarity. also, future research including the point of view of staff that do not hold administrative positions would provide insight into the practical aspects of integrating off-site storage into daily responsibilities. however, the authors acknowledge these limitations and neither impacts the importance of the research findings to the professional community.      as the authors observe, the high response rate indicates that further studies exploring retrieval methods, collection management, and integration of off-site workflows and services are a logical next step. research projects like these would assist in the development of strategies surrounding retrieval time and delivery, distance, and the perceived loss of browsability (barclay, 2010). as demands on library space increase, documentation of best practices and strategies linked to off-site storage for special collections is beneficial to both the professional and research communities.   references   barclay, d. (2010). the myth of browsing: academic library space in the age of facebook. american libraries, 41(6). retrieved from http://americanlibrariesmagazine.org/2010/05/19/the-myth-of-browsing/   deardorff, t. c., & aamot, g. j. (2006). spec kit 295: remote shelving services. retrieved from http://publications.arl.org/remote-shelving-services-spec-kit-295/     lafogg, m. c., & weideman, c. (2001). special collections. in a. n. nitecki and c. l. kendricks (eds.), library off-site shelving: a guide for high-density facilities (pp. 205-218). englewood: libraries unlimited inc.   sundstrand, j. k. (2008). placing manuscript and archival collections into an automated storage and retrieval system at the university of nevada, reno. journal of archival organization, 6(1), 71-80. doi: 10.1080/15332740802235380   sundstrand, j. k. (2011). getting to mars: working with an automated retrieval system in the special collections department at the university of nevada, reno. journal of archival orgainzation, 9(2), 105-117. doi: 10.1080/15332748.2011.602604 ebl 101   conducting your own research: something to consider   virginia wilson shirp coordinator health sciences library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   originally published in: evidence based library and information practice, 4(4), 95–98. https://ejournals.library.ualberta.ca/index.php/eblip/article/view/6819/5839     received: 01 nov. 2009 accepted: 11 nov. 2009      2016 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   in the last issue we looked at a variety of free and open access lis resources that you can use to find evidence pertaining to your practice question. consulting the evidence is obviously the cornerstone of evidence based practice and evidence based librarianship (ebl). there are occasions, though, when a search of the available evidence just isn’t fruitful. the articles found may not be directly applicable to your situation. the articles might increase your body of knowledge about a particular subject, but perhaps are not practical enough. or, you might not be able to find anything at all. this ebl 101 column asks you to consider collecting your own evidence in order to inform your decision-making process, i.e., undertake some research of your own.   of the three definitions proposed for ebl early in the millennium, only one specifically mentions the notion of librarians conducting research. crumley and koufogiannakis’ definition describes ebl as including “encouraging librarians to conduct high quality qualitative and quantitative research” (62). for academic librarians on the tenure track, such research is often essential for professional movement, and will most likely (although not always) be tied to professional practice. however, there are more benefits to be derived from conducting research than getting tenure or promotion, and ideally more than tenure-track, academic librarians will be conducting it.   what are the benefits of practitioners conducting research? the benefits are many, and i would suggest that they outweigh the barriers that might hamper librarians and information professionals in their quest to conduct research:   research aids in the improvement of problem solving and decision making in the workplace. research by library and information professionals creates new knowledge and contributes to the growth of lis as a profession and a discipline. undertaking research can help make library professionals critical consumers of the research literature. librarian-researchers bring an additional set of skills to the table in order to provide optimal information services to researchers in other fields. conducting a research project can improve an individual’s ability to think critically and analytically. (adapted from versoza)   it’s important not to let the thought of a research project strike fear into your heart. one of the barriers to the practice of ebl is the real or perceived notion held by librarians that they do not possess the necessary skills to undertake research. however, projects can be simple and practical. the process of ebl is intended to help improve your decision making and problem solving at work. thus, research questions that emerge from practice can be powerful motivating factors in the conducting of research because they are found in your everyday work and can aid you with it.   work has been done of compiling research questions that need to be answered. in 2001, jonathan eldredge published an article entitled “the most relevant and answerable research questions facing the practice of health sciences librarianship.” in 2006, lewis and cotter looked at that issue again in their article, published in evidence based library and information practice, entitled “have the most relevant and answerable research questions facing librarians changed between 2001 and 2006?” both articles are useful in terms of getting an idea of what research topics are important to librarians, health or otherwise, and can help set the stage for thinking about your own practice questions.   professional associations can be a good source of both research topics and research funding. often, associations will put together a research agenda, a way to focus resources on research topics that would benefit members.   cla/acb (canadian library association) library research and development grants http://www.cla.ca/am/template.cfm?section=grants&template=/cm/htmldisplay.cfm&contentid=5758 be sure to check with your cla division or interest group for research funding opportunities as well. acrl (association of college and research libraries) research and scholarship committee http://www.ala.org/ala/mgrps/divs/acrl/about/sections/is/committees/research.cfm this page contains the acrl research agenda for library instruction and information literacy, a bibliography of research methodology resources, and much more for the librarian-researcher. mla (medical library association) research http://www.mlanet.org/research/ this page contains grant information, research resources, and projects and studies. sla (special libraries association) research grant http://www.sla.org/content/resources/scholargrant/resgrant/index.cfm this grant description suggests research topics and the focus is on evidence-based practice.   there are a variety of books and websites that can assist you in becoming familiar with and more adept at various research methods. workshops, webinars, and online courses can help you spruce up your skills as well.   a few resources for the librarian-researcher online activities:   the education institute often has sessions that deal with research methods. http://www.thepartnership.ca/partnership/bins/index_ei.asp some books to get you started: leedy, paul d., and jeanne ellis ormrod. practical research: planning and design. 8th ed. upper saddle river, nj: prentice hall, 2005. powell, ronald r., and lynn silipigni connaway. basic research methods for librarians, 4th ed. westport, ct: libraries unlimited, 2004. online tools for research activities: survey monkey http://www.surveymonkey.com/ sample size calculator: http://www.surveysystem.com/sscalc.htm webpages that perform statistical calculations: http://statpages.org/ bank of canada inflation calculator: http://www.bankofcanada.ca/en/rates/inflation_calc.html outcomes toolkit: http://ibec.ischool.washington.edu/static/ibeccat.aspx@subcat=outcome%20toolkit&cat=tools%20and%20resources&tri=toolkitabout.htm (these links are from the evidence based toolkit for public libraries created by steph hall and me: http://ebltoolkit.pbworks.com/.) conducting your own research from a front line perspective will assist you in your own decision-making process. but make sure you give thought to broadcasting the results beyond your own library. by disseminating your results via scholarly journal, conference presentation, personal communication, or other dissemination mode, you will be making it possible for other practitioners to learn from and build upon your work. a barrier to the practice of ebl is lack of buy in by managers. the research process, including the publishing of results, is not seen as important or worthwhile in some instances. however, genoni, haddow, and ritchie state that “practitioners must increase their contributions to research literature. … the writing-up of research needs to be recognized and rewarded by employers as a legitimate professional task” (57). research projects can enhance the library’s status within its community, both locally and professionally, which will help to generate the buy in. the dissemination process helps build and strengthen the evidence base, and helps to build a culture of research amongst librarian and information professionals.   i’m always on the lookout for good quality books/websites/online sources and courses dealing with research, and i know i’ve only scratched the surface in this column. please pass along your favourites to me and i will compile them for a future column (virginia.wilson@usask.ca). coming up next time in ebl 101, an introduction to critical appraisal.   works cited   crumley, ellen, and denise koufogiannakis. “developing evidence-based librarianship: practical steps for implementation.” health information and libraries journal 19.2 (2002): 61-70.   eldredge, jon. “the most relevant and answerable research questions facing the practice of health sciences librarianship.” hypothesis 15.1 (2001): 9-17.   genoni, paul, gaby haddow, and ann ritchie. “why don’t librarians use research?” evidence based practice for information professionals: a handbook. eds. andrew booth and anne brice. london: facet, 2004. 49-60.   lewis, suzanne, and lisa cotter. “have the most relevant and answerable research questions facing librarians changed between 2001 and 2006?” evidence based library and information practice 2.1 (2007): 107-20.   verzosa, fe angela m. “motivating librarians to conduct research.” philippine association of academic and research librarians (paarl) national seminar-workshop on empowering information professionals to make a difference. ateneo de naga university, bicol, philippines, 24 october 2006. 1 nov. 2009 <http://www.slideshare.net/verzosaf/motivating-librarians-to-conduct-research>.   evidence summary   diversity, equity, and inclusion professional development events in academic libraries have minimal impact on knowledge, behavior, and organizational change without meaningful design and participant intentionality   a review of: dali, k., bell, n., & valdes, z. (2021). learning and change through diversity, equity, and inclusion professional development: academic librarians’ perspectives. journal of academic librarianship, 47(6), 102448. https://doi.org/10.1016/j.acalib.2021.102448   reviewed by: eugenia opuda health & human services librarian assistant professor dimond library university of new hampshire durham, new hampshire, united states of america email: eugenia.opuda@unh.edu   received: 7 mar. 2023                                                                    accepted:  25 apr. 2023      2023 opuda. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30344     abstract   objective – to explore experiences of library diversity, equity, and inclusion (dei) professional development events (pdes) and to examine the long-term impacts on knowledge and personal practices of librarians and organizational change   design – exploratory, qualitative survey, hermeneutic phenomenology, retrospective reflection on a critical incident   setting – academic libraries across the united states and canada   subjects – 141 survey responses from academic librarians in public and private institutions who attended a dei pdes   methods – the authors distributed a survey in late 2020 to librarians in public and private academic institutions across the us and canada. participants were asked to reflect retrospectively on memorable and impactful dei pdes they attended in the last five years and to focus on events at least a year before the survey was conducted. participants were also asked to describe if the pde changed their own learning and practice, as well as their organization’s policies and practices. the authors used hermeneutic phenomenology as a framework for their research and data analysis, which they describe as an approach to phenomenology that emphasizes the range and diversity of experiences. the survey questions were designed to illicit retrospective reflection and critical incident technique in order to capture the most impactful and memorable experiences with pdes from respondents. survey responses were coded, categorized, interpreted, and then mapped to the transtheoretical model (ttm) of change, a principle that breaks the phases of change into six processes: precontemplation, contemplation, preparation, action, maintenance, and termination. additionally, the authors also ensured they represented diverse backgrounds as a method for addressing and reducing bias.   main results – the authors uncovered a trove of varied experiences and perspectives related to personal as well as organizational knowledge and change. however, they identified five broader categories of learning from the study data: cognitive learning, behavioral learning and change, personal learning and change, social learning and change, and emotional learning and change. the majority of study participants (n=91) chose to attend dei pdes rather than being required by their organizations to attend. many came into the pde with goals to attain new knowledge (n=39) and to change their behavior (n=30). similarly, respondents noted that the pde impacted them most by giving them new awareness around dei topics and systemic inequities (n=51) and inspired behavioral changes and a commitment to taking action (n=28). pdes also impacted some participants’ self-awareness around their own biases and privileges (n=22). however, responses included skepticism around the impact of pdes on any real and meaningful change, particularly related to organizational culture and action. respondents emphasized concern about performative allyship and underscored the challenges and barriers to making dei a meaningful component of many institutional practices and policies. while respondents noted increased awareness around systemic inequities, these responses stood in stark contrast to several comments expressing frustration at the lack of meaningful organizational change and demoralization felt at the fruitlessness of dei pdes. the authors’ initial data analysis revealed that pdes most impacted cognitive awareness and behavioral action. however, when they mapped the data to the ttm, results emphasized the impact of pdes on behavioral learning and action.   conclusion – academic librarians who participate in dei-related pdes experience a wide array of emotional responses to the training and leave with a broad range of cognitive, behavioral, and affective impacts. while data suggests that these pdes increased awareness and knowledge and behavioral action the most, there is little that suggests that meaningful organizational change follows afterwards. another challenge is the gap between having awareness of dei topics and taking steps toward meaningful self-improvement. the authors note that it takes time for knowledge to translate into action and highlight the importance of post-pde check ins by pde facilitators and library administration. academic librarians should approach dei pdes with intentionality by challenging themselves to set goals and use their new dei knowledge to create actionable change both personally and organizationally. on the other hand, creators of dei pdes should take time to understand the participants and organizational culture in order to design trainings intentionally, using the ttm as a guide to identify how the pde maps to the stages of change leading to meaningful action and follow up. without intentionality, follow up, and goal setting, dei-related pdes may be ineffective, performative, and demoralizing.   commentary   this research expands on the authors’ previous publication on recent improvements made to dei pdes and the impact in library settings. previous research by leung (2022) and ely (2021) critically examines the authenticity and impact of one-time pdes and dei statements of university libraries and note the ineffectiveness and harm of performative dei work. cruz’s (2019) literature review explores dei in academic libraries and similarly notes that true and meaningful commitment to dei requires thoughtful and intentional action. research by geiger and colleagues (2023) indicates that academic library workers feel more dissatisfied than satisfied around dei initiatives and building dei into the work culture. dali et al. (2021) add to the current literature on dei in academic libraries and bring a unique approach by examining long-term impacts of dei pdes and including a rich set of qualitative data.   this article was evaluated using the adapted russell and gregory’s (2003) qualitative research appraisal questions (suarez, 2010). the authors identify a clear and specific research question, though their discussion includes an additional component of addressing recommendations and suggestions for designing effective dei pdes. the qualitative survey design guided by hermeneutic phenomenology, combined with retrospective reflection and mapping to the transtheoretical model appropriately matches the exploratory research question. the authors account for their varied backgrounds and experiences and utilize those varied experiences as a method of bias control.   study methods lack some details including survey question creation, validity testing, distribution, duration of the study, and data management. inclusion criteria only required that respondents were people employed at academic libraries at the time the research was being conducted. the data analysis techniques are appropriate to the study and richly described. additionally, the authors are up front about gaps in their own research and note the lack of responses focused on accessibility, immigration, religion, gender, and sexual identity. however, the authors do not examine the harms that dei pde experience may cause to participants from historically marginalized backgrounds, nor the credibility of memory recall of events that occurred in the past. lastly, the authors also do not identify whether there was any example of pde that had a meaningful, positive impact.   findings from this study are richly described and the authors take a holistic approach to examining the many facets of dei pde experiences. they highlight several optimistic comments about dei awareness and commitment to action, as well as deep cynicism about the impact of pdes on organizational change and the intentions of participants in attending pdes. this study seems to validate some of the cynicism and skepticism felt around the lack of impact of pdes. the authors note that any small, positive change in dei action must be acknowledged and celebrated, but that there is a scarcity of examples of organizations that have made these changes in meaningful ways.   this study is a call to action for librarians and library organizations to approach pdes with more intentionality, to make commitments for personal and organizational change, and to follow up on those goals. as a predominately white profession, meaningful progress around dei in librarianship is of vital importance to truly uphold our professional values of diversity and social inclusion. this study underscores the ineffectiveness and performative aspects of some of these initiatives. the authors highlight actional steps for designing meaningful learning activities for pdes which include gauging participant levels of readiness and interest, customizing the training to organizational needs, and following up with participants on their commitments to change. the authors also call for a focus on intersectionality in dei pde design and note the lack of pdes related to certain areas. participants must explore their reasons for attending pdes to have meaningful personal and organizational impacts.   references   cruz, a. m. (2019). intentional integration of diversity ideals in academic libraries: a literature review. the journal of academic librarianship, 45(3), 220–227. https://doi.org/10.1016/j.acalib.2019.02.011   dali, k., bell, n., & valdes, z. (2021). learning and change through diversity, equity, and inclusion professional development: academic librarians’ perspectives. journal of academic librarianship, 47(6), 102448. https://doi.org/10.1016/j.acalib.2021.102448   ely, e. (2021). diversity, equity & inclusion statements on academic library websites. information technology & libraries, 40(4), 1–22. https://doi.org/10.6017/ital.v40i4.13353   geiger, l., mastley, c. p., thomas, m., & rangel, e. (2023). academic libraries and dei initiatives: a quantitative study of employee satisfaction. the journal of academic librarianship, 49(1), 102627. https://doi.org/10.1016/j.acalib.2022.102627   leung, s. (2022). the futility of information literacy & edi: toward what? college & research libraries, 83(5), 751–764. https://doi.org/10.5860/crl.83.5.751   suarez, d. (2010). evaluating qualitative research studies for evidence based library and information practice. evidence based library and information practice, 5(2), 75–85. https://doi.org/10.18438/b8v90m     using bibliometrics to illustrate the impact of the library on military medical research using evidence in practice   using bibliometrics to illustrate the impact of the library on military medical research   holly rose beverley medical librarian cadence group contract at the walter reed army institute of research silver spring, maryland, united states of america email: holly.r.beverley.ctr@health.mil   maureen humphrey-shelton command librarian united states army medical research and development command and united states army medical research institute of infectious diseases frederick, maryland, united states of america email: maureen.p.humphrey-shelton.civ@health.mil elisia george medical librarian united states army medical research institute of chemical defense aberdeen proving ground, maryland, united states of america email: elisia.a.george.civ@health.mil   olivia briere librarian cadence group contract at the department of justice washington, d.c., united states of america email: okbriere1207@gmail.com   received: 9 mar. 2023                                                                  accepted: 17 apr. 2023      2023 beverley, humphrey-shelton, george, and briere. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   data availability: data associated with this article cannot be made publicly available because they are proprietary information to the united states army.     doi: 10.18438/eblip30345     setting   in this article, the authors describe how bibliometrics were used as a novel method of demonstrating use and impact of specialized biomedical library collections. by exploring an approach used previously for collection development, the authors were able to illustrate library impact on military medical research beyond the traditional approach of return-on-investment investigations and statistical presentations.   the u.s. army medical research and development command (mrdc) libraries consist of the headquarters virtual library and five physical subcommand libraries, all staffed by master’s degree-holding librarians. information professionals in the mrdc libraries consortia are often solo librarians or small library teams who assist highly experienced researchers with electronic access and literature searching. mrdc libraries are present throughout the research workflow providing effective virtual libraries and end-user training.   mrdc libraries support basic science and translational research. the mrdc pursues medical research and testing that pertains to human treatments, like neurological effects of exposure and how illnesses or trauma manifest. animal and human research conducted at our organizations frequently translates into military products and devices, as well as treatments for military and civilians. mrdc libraries facilitate the research of illnesses and their causation while also focusing on global military health. the focus on military health encompasses both physical and psychological causes and prevention.   problem   the mrdc library program is a multifaceted consortium overseen by a command librarian in the headquarters library and five subcommand libraries located at the united states army medical research institute of chemical defense (usamricd), walter reed army institute of research (wrair), united states army medical research institute of infectious diseases (usamriid), united states army institute of surgical research (usaisr), and the united states army aeromedical research laboratory (usaarl). besides providing an onsite location for usage, each subcommand library offers additional resources and services specifically tailored to their users. the goal of the library program is to keep the service member at the forefront of our collections, services, and trainings to contribute to the mrdc mission of ensuring that the u.s. armed forces are equipped to protect themselves from disease and injury.   historically, mrdc was a subordinate organization under the u.s. army medical command and had access to the army medical department (amedd) virtual library resources (figure 1) as well as maintaining their local physical resources. recently, the defense health agency (dha) was established to bring together all the medical departments of all military branches under one umbrella to consolidate resources when possible. as mrdc transitions to dha, those clinical library resources are now accessible through the mrdc virtual library. however, as mrdc librarians know from first-hand experience, dha clinical library resources alone would not be sufficient for mrdc researchers because of differing usage needs from their clinical colleagues.   dha clinical library resources are designed to meet the needs of medical professionals working in military treatment facilities (e.g., clinicians, nurses, physician assistants, and other clinical staff directly responsible for patient care). it is more likely that clinical library users would need evidence-based treatment summaries, patient care guidelines, anatomy or procedure textbooks, clinical decision support, manuals or electronic tools to check for contraindications or drug interactions. maintaining access to older years of scientific literature coverage is rarely considered best practice in clinical libraries. this is especially true if clinical information services are implemented in a point-of-care setting where clinicians only need access to the most recent research as it relates to patient treatment. information from the last five years is often sufficient for these purposes.   military medical research library collections support scientific staff in the discovery and testing of novel therapeutics, preventative measures, or treatments for diseases and conditions that affect humans. the mrdc libraries provide access to the full body of knowledge within a given discipline, in applicable subject areas since such content might contribute to growth in the field of military medical research. some of these subject areas encompass infectious diseases, combat casualty care, military operational medicine, chemical defense, clinical and rehabilitative medicine, and entomology, among others because of their prevalence in the field (u.s. army medical research and development command, n.d.). because the researchers frequently read and cite content from older scientific journal literature, the mrdc librarians also purchase backfiles to maintain journal access to many years of coverage. lercher and smolinsky (2016) extrapolate, “if the use of older scientific literature is increasing, then one might argue that the value of archives of such literature is likewise increasing” (pp. 1219–1220). analysis of the increased citations and usage trends for older scientific literature supports the value of more extensive archives in research libraries, though the frequency of citing older articles can vary by discipline as can the definition of “older” articles (verstak et al., 2014).   though the librarians maintain an ongoing awareness of army medical researchers’ needs and advocate for the resources that these specialized library users require, new leadership or new funding agencies would also benefit from this information especially since dha is largely composed of clinical military treatment facilities, whereas mrdc is composed of military medical research facilities. as a result, this project emerged to simultaneously advocate for the resources needed by mrdc researchers and staff while illustrating the differing subject expertise among researchers supported by the mrdc library program in preparation for the transition to dha.   evidence   federal medical library surveys are conducted regularly by clinical libraries, but using a survey as the research instrument was not sufficient due to the clinical library-oriented questions in the survey instrument and the lack of statistical proof of library impact desired by the mrdc librarians. though mrdc was invited to participate in a federal library survey initiated by clinical libraries, the points of data collection did not align with the mission and services provided by the medical research libraries. a working group consisting of four of the mrdc librarians developed questions for a survey in an earlier phase of this project, but ultimately chose not to use it.   several different approaches to demonstrating value were explored through a literature review. literature regarding “return on investment” in library settings uses different conceptual models like social value analysis, value co-creation, perceived value, long term impact of library service use, and economic impact (urquhart, 2020). literature on topics surrounding library value and approaches to impact studies often come from academic or clinical settings (de groote et al., 2020; de groote & scoulas, 2022). additionally, literature which addresses “value studies” focuses heavily on financial data and justifying ever-increasing costs of operation rather than focusing on deliverables and the greater impact of library services. none of these options were found to be suitable for this project.   very limited research was discovered on the topic of using bibliometric data in federal research libraries. though identifying literature that would illuminate an applicable strategy for mrdc libraries proved challenging, belter and kaske’s 2016 article in college & research libraries provided an example of how citation metadata could be used in federal research libraries for collection development. after reviewing the article, the working group determined that using the citation metadata of research publications authored by mrdc researchers could also be used to investigate whether mrdc or dha provided the most access to cited articles in mrdc researchers’ publications. cited references would provide a compelling foundation for why specific resources are required and demonstrate the library’s impactful role in the research process.   though belter and kaske used web of science, the mrdc librarians used scopus for their analysis. scopus is an abstract and citation database that covers thousands of book and journal titles. the majority of the journals are peer-reviewed and cover such fields as life sciences, social sciences, physical sciences, and health sciences. scopus also enables affiliation-based searching, which helps identify publications within a database that were written by individuals employed by or associated with a particular agency. from professional experience, the mrdc librarians knew that scopus content coverage was more comprehensive and captured more research authored by mrdc affiliated researchers than web of science.   after determining how to proceed, the working group utilized the bibliometric tools in scopus to analyze the citation metadata of all publications written by mrdc researchers from 2017–2021. all mrdc locations were included in the analysis of 3,631 total scientific journal articles (figure 1). the librarians analyzed 130,256 cited references used to create the published articles. mrdc libraries currently receive subscription resources funded by dha, mrdc, and subcommand libraries. free or open access resources and interlibrary loan (ill) services were evaluated as well. further analysis determined the library that most likely provided the cited source. coverage of cited references from the articles retrieved in the affiliation searches are presented in figures 2 and 3, using one subcommand’s affiliation search as an example. these data were compared to the mrdc virtual library and subcommand library holdings.     figure 1 coverage of all citations from mrdc researcher-authored publications between 2017–2021.       figure 2 example of affiliation search for one of the five subcommand libraries (wrair) evaluated in this study: what was published by wrair-affiliated authors between 2017–2021?     figure 3 example of publications' cited references for one of the five subcommand libraries evaluated in this study: what was cited by wrair-affiliated authors in their written publications between 2017–2021?     to identify the library that provided each cited source, librarians evaluated their holdings for each cited reference listed in mrdc-affiliated publications to determine the level of access provided. for every resource considered, current coverage was required at a minimum, and the archival coverage was also evaluated. in the instance that a resource was covered by mrdc and dha, credit was given to the organization that provided the most extensive full text access and the most extensive archive (figure 4). for example, each author went through the list of individual journal titles with articles that were cited by mrdc researchers between 2017–2021 and searched their online library catalog to confirm whether the mrdc command or their subcommand library provided current access to each journal’s content. if a journal could be found in both a dha-provided resource and an mrdc-provided resource, and current access was provided by both, the distinguishing criteria would be the extensiveness of the archive or the years of coverage that were provided for that journal within that resource (e.g., clinicalkey or sciencedirect). so, when comparing the level of access provided for a particular journal in the dha-provided resource clinicalkey versus the mrdc-consortium provided sciencedirect, both dha and mrdc had current coverage for the journal. however, mrdc was given credit because that agency had a more extensive archive for that journal.    ill statistics were also incorporated to account for gaps in coverage and to highlight the mrdc subcommand libraries’ contributions to connecting researchers with resources. ill services facilitate unique literature requests from researchers and scientists to enable the work they do advancing their field of research and creating deliverables for military service members. though outside libraries were used to fill ill requests during this timeframe, most ill requests were filled by other research libraries within the mrdc consortia, thus reinforcing the fact that even if one of our libraries does not have access to a particular resource, local subscriptions throughout the consortia can often facilitate the request.     figure 4 example of evaluating one subcommand library’s holdings: which organization provided access (mrdc, dha, free = open access, or blank if covered by mrdc and dha without a difference in archival coverage) to journals cited by researchers at this subcommand?     implementation   this project determined that the resource collections put together by the mrdc library program and subcommand libraries are extremely comprehensive and effective at fulfilling the needs of researchers. analysis of citation metadata can help justify continued funding for research-specific resources as libraries continue to partner with military medical researchers to fulfill the department of defense (dod) needs for experimental research. loss of resources will greatly impact the ability of military medical researchers to obtain research needed to further the missions of the dod and could have potentially grave impacts for warfighters down the road.   these data have been presented in annual reports and multiple presentations discussing the importance of libraries to headquarters leadership.   outcome   the majority of the topics covered in mrdc-authored publications are not clinical in nature and are clearly bench or experimental research; therefore, clinical-leaning resources would not satisfy the needs of the mrdc researchers. each location determined that most of the references needed to generate their research were available through mrdc-provided subscriptions, either by the command-wide subscriptions or through locally subscribed resources. further determination found that ill provided by the subcommand libraries (figure 1) was also used a small percentage of the time.   powerpoint presentation slides were used to highlight the purchasing agency with the most expansive resource coverage at each mrdc institution based on citations from the last five years. the presentation was shared with mrdc leadership and local libraries to demonstrate library impact and advocate for comprehensive resource funding. initial slides provided a description of how the data were gathered, the years included in our publication set, and significant revelations from the data. namely, that most publications generated by mrdc organizations referenced materials provided by the mrdc virtual library or subcommand libraries. mrdc library subscriptions encompass a range of medical research topics designed to align with the organizational mission. in no instance did any library rely primarily on the clinical resources provided by dha.   conveying budgetary requirements to non-library stakeholders called for rethinking ways to distinguish clinical resources and mrdc research resources, and which subscriptions best support the needs of our users. specialized backgrounds of non-library stakeholders ought to be considered when communicating with leadership, especially when contemplating how to best present library information and metrics that will most effectively illustrate the role of the library in the mission. the most effective metrics for each library may vary depending on the type of library and its deliverables.   reflection   analysis of citation metadata can help libraries justify budgetary costs, make collection development decisions, and affirm the libraries’ alignment with critical research activities. bibliometrics should be used to advocate the impact of libraries within their organizations. although mrdc libraries differ from academic and clinical libraries, this evidence-based methodology is applicable in other types of library settings. while surveys can be beneficial for gathering feedback from the patron, information professionals should also consider using citation metadata to illustrate the role that the library plays in research, publication, and medical product development.   many stakeholders are concerned with the cost of library operations, leading to an increase in library literature focused on value studies and return on investment. however, librarians in the mrdc library consortia recognize that the most important “currency” in our libraries is research. the bibliographies of mrdc publications prove that mrdc libraries provide access to the primary cited resources in our researchers’ publications. libraries must consider the backgrounds of library stakeholders and leadership when evaluating what evidence would most effectively communicate their impact.   author contributions   holly rose beverley: data curation (equal), formal analysis (equal), project administration (lead), supervision (lead), writing – original draft (lead), writing – review & editing (lead) olivia briere: data curation (equal), formal analysis (equal) elisia george: data curation (equal), formal analysis (equal), writing – review & editing (equal) maureen humphrey-shelton: conceptualization (lead), data curation (equal), formal analysis (equal), methodology (lead), supervision (equal), writing – review & editing (equal)   disclaimer   material has been reviewed by the walter reed army institute of research and the united states army medical research and development command. there is no objection to its presentation and/or publication. the opinions or assertions contained herein are the private views of the authors and are not to be construed as official or as reflecting true views of the department of the army or the department of defense.   references   belter, c. w., & kaske, n. k. (2016). using bibliometrics to demonstrate the value of library journal collections. college and research libraries, 77(4), 410–422. https://doi.org/10.5860/crl.77.4.410   de groote, s. l., akusa dunya, b., scoulas, j. m., & case, m. m. (2020). research productivity and its relationship to library collections. evidence based library and information practice, 15(4), 16–32. https://doi.org/10.18438/eblip29736    de groote, s. l., & scoulas, j. m. (2022). the impact of the academic library on students' success, in their own words. portal: libraries and the academy, 22(2), 355–374. https://doi.org/10.1353/pla.2022.0021   lercher, a., & smolinsky, l. (2016). persistent value of older scientific journal articles. scientometrics, 108(3), 1205–1220. https://doi.org/10.1007/s11192-016-2011-5    urquhart, c. (2020). assessing return on investment in health libraries requires lateral thinking. health information and libraries journal, 37(1), 1–4. https://doi.org/10.1111/hir.12298    u.s. army medical research and development command. (n.d.). program areas: medical research & development. retrieved november 15, 2022 from https://mrdc.health.mil/index.cfm/program_areas/medical_research_and_development   verstak, a., acharya, a., suzuki, h., henderson, s., iakhiaev, m., lin, c. c. y., & shetty, n. (2014). on the shoulders of giants: the growing impact of older articles. arxiv. https://doi.org/10.48550/arxiv.1411.0275  evidence based library and information practice 2012, 7.1 53 evidence based library and information practice article the impact and effect of learning 2.0 programs in australian public libraries michael stephens assistant professor school of library and information science, san jose state university san jose, california, united states of america email: mstephens7@mac.com warren cheetham coordinator of information and digital library services, city libraries townsville, queensland, australia email: warren.cheetham@townsville.qld.gov.au received: 16 sept. 2011 accepted: 7 feb. 2012 2012 stephens and cheetham. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/bync-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – with adoption of the program world-wide, the learning 2.0 model has been lauded by library professionals as a mechanism to educate library staff and transform libraries. this study, part of the 2009 caval visiting scholar project, seeks to measure the impact and legacy of the model within australian public libraries to understand what benefits, changes and effects occur. methods – a national web-based survey for those who had participated in a learning 2.0 program. results – the national survey had 384 respondents, and a total of 64 respondents were identified as the public library staff data set for this article. public library staff reported success in the program and described feelings of increased confidence, inclusivity, and a move to use emerging technologies as part of library service. conclusion – the analysis yields the following thematic areas of impact and effect: personal practice is enhanced with knowledge and confidence; impact is mainly personal, but organisational changes may follow; the library is using the tools to varying degrees of success, and organizational blocks prevent use of tools. these mailto:mstephens7@mac.com� mailto:warren.cheetham@townsville.qld.gov.au� evidence based library and information practice 2012, 7.1 54 finding offer evidence that learning 2.0 programs can have a positive effect on library staff and subsequently on the organization itself. introduction august 2011 marks the five year anniversary of the debut of the learning 2.0 program created by helene blowers of the public library of charlotte mecklenberg county (plcmc). blowers utilized the article “things you (or i) might want to do this year” by stephen abram and “distilled it down to 23 things that she wanted her staff to understand through hands-on experience” (hastings, 2007). the “things” included blogging, subscribing to rich site summary (rss) feeds, and exploring sites such as youtube and flickr. hanly (2007) reported the plan was to include all staff in learning: “blowers challenged her 550 staffers to become more web savvy.” the program was based around a series of weekly online learning activities. participants would spend a few minutes each day exploring an emerging web technology and compose a reflective blog post about what they learned. some staff might work alone while others would work in groups or meet to discuss progress. blowers encouraged group discovery within the program (2008). the original learning 2.0 with the initial “23 things” is still available on the web at http://plcmcl2-about.blogspot.com /. plcmc and blowers offered the program online for use by any interested library via a creative commons license. because of this move to share the content freely, the model struck a chord with the library community. learning 2.0 goes global since 2006, libraries all over the world have offered variations of the “23 things” program for their staff. blowers (2009) estimated close to 1,000 libraries and organizations have used the program. as more emerging technologies appeared, the programs have evolved as new tools are introduced. recent programs have included tools such as twitter, facebook and mobile library applications. blowers (2008) reported that libraries around the world were accessing the learning modules to educate staff about web 2.0 tools and increase their interest and confidence levels. one of the first institutions to replicate blowers’ model was yarra plenty library in victoria, australia, where library leaders used the program to educate staff about emerging social tools that would be included in that public library’s strategic plan (lewis, 2008). yarra plenty library ceo christine mackenzie praised the program as a means of educating library staff, in a presentation at the public libraries: building balance conference in august 2007, noting that “it has received wide recognition” and favorable coverage in resources such as wired.com. transformational learning? some have called the program transformational (abram, 2008); while others have noted its ability to bring staff together in a common goal: learning emerging technologies. lewis (2008) reported “the learning 2.0 program had a great impact on staff, who now know they are capable of learning new technologies.” gross and leslie (2008) reported success with the program in an academic library setting but stated “to our knowledge, no formal evaluation of learning 2.0 has been conducted. however, the take-up rate among libraries worldwide has been impressive and stands as an endorsement of the program” (p. 796). a later case study by gross and leslie (2010) detailed the program’s implementation and offered insights to make it more effective. there is a lack of actual evidence of the program’s impact in the scholarly literature. some questions emerge: what is the true impact of the program on the library and library staff? what does it mean to say that the program “has fundamentally changed the staff's way of thinking and working in the 21st century” (titangos and mason, 2009)? how evidence based library and information practice 2012, 7.1 55 have the tools been adopted? does the culture of learning continue? this study, launched as part of the caval visiting scholar project in 2009, utilizes web survey methodology to gauge the impact and efficacy of the program in australia’s public libraries. literature review to frame the impact of a learning program such as “23 things” in a public library setting, a survey of applicable adult learning theory and other research concerning learning environments helps to set the stage. adult learning knowles (1970) proposed concept of andragogy, focused on adult learning, includes these assumptions, as summarized by merriam and caffarella (1999): we grow to be more self-directed in thought and action throughout our lives. we gather ongoing experiences that influence and illuminate our learning. we are ready to learn tasks or concepts directly related to our roles and lives. we move from future-oriented learning to problem solving focused learning. we are motivated to learn by internal factors over external ones. (p. 272) these concepts comprise a model of adult learning based on the characteristics of adult learners and their life experiences. task-based and focused on learning at the point of need, knowles’ model can serve as a roadmap for designing learning experiences for adults. learning 2.0 was designed as an exercise in self-direction for library staff that needed to learn about technologies that were impacting the way people consume and interact with information, as noted by hanly (2007). lifelong learning, self-directed learning and “learner control” also foundational to the original learning 2.0 program was an emphasis on lifelong learning. hiemstra (1976) argued that three forces are at work to propel ongoing interest and need for lifelong learning: constant change, occupational obsolescence and an individual’s desire for self-actualization. many learning 2.0 programs began with an emphasis on “lifelong learning” to set the stage, the original program utilizing a resource called “7 1/2 habits of highly successful online learners.” the original program was also based on the concept of self-directed learning. candy (1991) offered various definitions of self-directed learning (sdl) as learner-created, learnermanaged, and self-motivated. candy also provided a summary of several decades of findings concerning sdl (1991) that include a social component or interaction with others: • interaction with other people usually motivates sdl. • sdl is non-linear in nature and relies on serendipity. • sdl is rarely a solitary activity; it often occurs within a social grouping. (p.199) acknowledging that sdl is truly not a solitary activity, candy later argued (2004) that a more fitting description for sdl would be “learner control,” in which the learner can “take control over a narrow range of choices” (p. 50). he also recognized the potential for guided online education to free the learner to explore beyond specified course material. hough (2006), a librarian and trainer, recognized the possibility for learner control or sdl in the “23 things” program. according to hough, the program emphasizes a shift from providing step-by-step directions to approaching professional development as an opportunity to increase independence, confidence, and awareness of the potential of emerging technologies. she encouraged library leaders to embrace this new paradigm for learning as they adopt a service ethic that stresses interaction with users. another description of the program noted it is a process “that all librarians should follow, even though we doubtless will end up in different places” (maxymuk, 2008, p. 66). evidence based library and information practice 2012, 7.1 56 transformative learning within the literature on adult learning, mezirow’s transformative learning theory (1991) offers a theoretical lens through which to better understand the impact of learning 2.0. the process in which adults respond to events or new experiences and how it changes or enhances their frame of reference and subsequent approaches to other events/experience is the basis for this theory. transformative learners re-align their viewpoints as more knowledge is obtained. the process is “more inclusive, discriminating, self-reflective, and integrative of experience” (mezirow, 1997, p. 5). cranton (2006) offered this further definition: “when people critically examine their habitual expectations, revise them, and act on the revised point of view, transformative learning occurs” (p.19) play and exploration another important component of the learning 2.0 professional development model is an emphasis on play (blowers, 2008). jenkins (2006) defined play as “the capacity to experiment with one’s surroundings as a form of problem-solving,” and asserted that play is one of the emerging social literacies or skills for education. the learning 2.0 model combines play and opportunities to explore new spaces into a unique approach to selfdirected professional development. aims with adoption of the program world-wide, the learning 2.0 model has been lauded by library professionals as a mechanism to educate library staff and transform libraries. developed for adult learners, the program includes an emphasis on self-direction, group discovery, and exploration and was intended to inform participants about the potential of emerging technologies to enhance library service and inspire confidence. as noted above, the purpose of the research project was to investigate if the program has “fundamentally changed the staff's way of thinking and working in the 21st century” (titangos & mason, 2009). the following research questions frame the research project to measure the impact and legacy of the model within australian public libraries: • to what extent have learning 2.0 programs enhanced library staff’s confidence and ability to explore and utilize emerging technologies? • to what extent have learning 2.0 programs had an impact on library services? • what are staff perceptions of the efficacy of learning 2.0 programs in libraries? • what are exemplary practices for the program? methods web survey a web-based survey tool, designed to address the research questions, contained two areas of focus, reflecting a research design that is both quantitative and qualitative. question types included demographic queries, likert scale and open-ended. the combination of quantitative and qualitative data, including five open-ended questions, provided an indepth examination of learning 2.0 participants’ perceptions of the impact of the program. the survey questions are reproduced in appendix a. sample the target population for the large-scale national survey was determined to be those who are employed in the libraries and information professions in australia and had participated in a learning 2.0 program offered locally or at the state level. because the program is intended to be inclusive for all types of library staff, library employees at all levels as well as students and consultants were invited to participate. the researchers understood the sample would be broad and varied, similar to the original focus of the inclusive learning 2.0 program. the institutional review board of the dominican university, river forest, illinois verified the survey questions in the spring of 2009. the evidence based library and information practice 2012, 7.1 57 web survey was announced on various australian library-related mailing lists, libraryrelated blogs, and other channels of communication and remained open for three weeks. a total of 384 valid responses were collected. responses deemed incomplete were those in which no answers were given beyond the initial demographic questions. these responses were removed from the initial data set. results survey data analysis analyzing the national survey data included a combination of quantitative analysis for the demographic data and descriptive content analysis for the open-ended questions. both researchers shared preliminary and final coding duties to ensure inter-coder agreement for the category responses. code-books were created for each question type and shared between the researchers and utilized for both the academic library subset and the public library subset. public library staff demographics the national survey had 384 respondents, 86% female and 14% male. those aged 45 to 54 were the highest number reporting, at 29%, followed by 28% in the 35 to 44 age range. the largest group was that of academic library staff at 18%; while 15% of respondents identified themselves as working in a public library setting. a total of 64 respondents were identified as the public library data set for this article. survey respondents who selected the following job categories were included: reference librarian (public library), children and youth services librarian and those respondents who chose to enter a response in the “other” field that indicated they worked in a public library setting. these included “local studies librarian (public library),” “mobile library operator,” “local history librarian,” and “library technician, public library.” these selections were verified by evaluating the open-ended responses for mention of the public library setting. any respondent that could not be verified as a public library employee was removed from the data set. of this selected data set, 87% were female. the most frequent age category was 35 to 44 at 30% followed by 45 to 54 at 28%. completion of the program completion rates of the program and the open-ended answers to the question “if you didn’t complete the program, why not?” showed a high percentage of the public library data set did complete the program (94%), and only three individuals reported noncompletion. those three respondents answered the follow up open-ended question, including these coded responses “no time / too busy” and “lack of interest / content not relevant.” one respondent noted that the main barrier to completing the program was too little time and difficulty concentrating: “same old problem the world over so many pressures doing the course in bites, phones ringing, workroom activities happening around you it was snatch and grab learning. many, like myself, did the course from home in our own time and were able to enjoy it that way.” success of the program one section of the survey asked participants to rate a series of statements via a likert scale exploring support by administrators, and administrator/management participation, as well as a rating of the success of the program. the majority of respondents agreed or strongly agreed with the statement “my library’s manager/supervisor backed the program.” administrative and management participation in the programs drops however, with 61% of respondents to the statement “my library’s manager/supervisor participated in the program”, strongly agreeing (36%) or agreeing (25%) that their manager took part. this did not appear to drastically affect the perception that “the program was a success,” with 64% strongly agreeing (39%) or agreeing (25%) with the statement. evidence based library and information practice 2012, 7.1 58 continuing to explore for the question “are you continuing to explore emerging technologies online?” the majority of respondents reported in the affirmative (94%) with the remainder (3%) reporting they were not continuing exploration. three of the public library subset of respondents did not answer the question. impact of the program: perception statements the final portion of the survey explored changes and impact of learning 2.0 on the library and on the individual’s professional practice via a series of open-ended questions. utilizing descriptive content analysis, the creation of codebooks and inter-coder checks, the category responses offer insights into the impact of the program, as perceived by participants. the following are the primary perception statements that account for the majority of responses for the impact section: • we’re implementing or using the tools to varying degrees of success • library staff is more comfortable learning about new technologies • there is better/increased awareness of 2.0 tools • library staff use the tools discovered to enhance work the following sections detail the category responses from each section of the open-ended portion of the survey as well as a related likert scale section exploring confidence and other topics. organizational changes for perceived organizational changes, 48 respondents in the public library data set answered the question, with the majority (44%) noting that their public library has adopted the tools with varying degrees of success. the results of content analysis on this section are represented in table 1. the majority of coded responses aligned with the “vary degree of success” category. one respondent noted: “the library has introduced an online social network for readers and the staff who have done the course are more likely than others to add blog posts, images, book reviews and comments to the site.” another category response focused on staff perception and interest in 2.0 tools. “our library staff are now more willing to embrace new technologies and ways of getting our message out there,” noted one respondent. another response “generally a greater knowledge of web 2.0 tools. we can assume knowledge in some areas as most of our staff have gone through the programme. some staff also became more confident in exploring new technology (realising they weren't going to “break” anything by playing with it!)” further illustrates this point. for those who noted there had been no impact some identified blocks put in place by information technology departments or city government which were a hindrance. one respondent reported “blocked by archaic it and communications section,” while another table 1 organizational changes after the program “what organizational changes have you noticed as a result of the program?” no. % we’re implementing or using the tools to vary degrees of success 21 43.75% no impact 10 20.8% better understanding/more openness to/increased perception of 2.0 tools 10 20.8% improved communication & sharing between staff 4 8.33% restrictions and blocks prevent us from using the tools effectively 3 6.25% evidence based library and information practice 2012, 7.1 59 stated “we are trying to do more 2.0 but often bureaucracy and it issues are stopping us.” comfort, confidence, committee structures and encouragement the next section was a set of statements exploring comfort levels and abilities as well as changes at the organisational level rated by respondents via a likert response scale. the public library staff subset results are represented in table 2. the selected responses for this section are positive when addressing any of the statements concerning personal proficiencies. the responses of “agree” or “strongly agree” are the most frequent for statements such as table 2 comfort, confidence, committee structures and encouragement after the program for public library data set strongly disagree disagree sometimes agree / sometimes disagree agree strongly agree response count skipped question total no. & (%) no. & (%) no. & (%) no. & (%) no. & (%) no. & (%) no. no. i’m comfortable learning about new technologies. 0 (0%) 0 (0%) 3 (5.26%) 18 (31.58%) 36 (63.16%) 57 (100%) 7 64 i’m confident i can learn new technologies. 0 (0%) 0 (0%) 2 (3.57%) 20 (35.71%) 34 (60.71%) 56 (100%) 8 64 i like to explore technology on my own: 0 (0%) 2 (3.57%) 10 (17.86%) 18 (32.14%) 26 (46.43%) 56 (100%) 8 64 i’m prepared to help our library users with emerging technologies. 0 (0%) 0 (0%) 4 (7.02%) 29 (50.88%) 24 (42.11%) 57 (100%) 7 64 the team/committee structures at my library have improved because of this training. 2 (3.57%) 12 (21.43%) 20 (35.71%) 15 (26.79%) 7 (12.50%) 56 (100%) 8 64 i’m encouraged to try new things at my job. 1 (1.79%) 3 (5.36%) 9 (16.07%) 24 (42.86%) 19 (33.93%) 56 (100%) 8 64 evidence based library and information practice 2012, 7.1 60 “i’m comfortable learning about new technologies” and “i’m prepared to help our library users with emerging technologies.” for the statement “the team/committee structures at my library have improved because of this training” the frequency of response shifts to “sometimes agree”/”sometimes disagree” and “disagree”, but returns to a more positive frequency for the statement “i am encouraged to try new things at my job.” lasting impact of the program responses to the open-ended question “what has been the lasting impact on your library after learning 2.0,” are reproduced in table 3. selected responses for the category of “better/increased awareness of 2.0 tools/inclusive feeling for staff” include: “the fact that it was a team effort and we all learnt together regardless of status, age etc. being exposed to new things was wonderful” and “staff are aware that emerging technologies will influence patron requests for information delivery.” for the category response confidence/helping library users, the next most prominent response in the public library subset, respondents noted such statements as “greater confidence in using web 2.0 technologies and talking to patrons about them (we work in a public library and get many older people wanting clarifications about web 2.0).” those who reported no impact used such statements as “nothing discernible” and, perhaps reflecting the influences of governing departments, “not muchhave only been allowed to implement a delicious account everything else is blocked by it or communications.” for “what changes have you made to your personal professional practice because of the program?” the most prominent coded response of the public library subset was “i use the tools i discovered to enhance my work.” selected responses include “higher proportion of my role now utilising these technologies where possible rather than older options” and “i am attempting to keep an open mind about information delivery methods for the future. thus, i am reading professional material/articles, but also playing with various web 2.0 technologies to ensure my skills are relevant to any library/information service workplace.” the full data analysis is reproduced in table 4. for the category response “i feel generally more in the know/comfortable/current “one respondent noted: “i feel empowered to keep up with change, and enthused about the possibilities,” while another noted the program’s self-directed nature “i am willing to jump in and work things out there was a bit of problem solving in the 23 things program it wasn't all spoon fed.” table 3 lasting impact of the program “what has been the lasting impact on your library after learning 2.0” no. % better/increased awareness of 2.0 tools/ inclusive feeling for staff 21 36% confidence/helping library user 12 21% improvements to staff communication 8 14% increased use of tools in personal life 1 2% increased use of tools in library 9 15.5% management/it more open to tools 1 2% no impact 6 10% evidence based library and information practice 2012, 7.1 61 discussion this study identifies the perceptions of the impact of learning 2.0 programs in australian public libraries. the analysis of multiple choice and open-ended questions yields the following thematic areas of impact and effect as well as suggestions for exemplary practice for learning 2.0: • personal practice is enhanced with knowledge and confidence, and exploration continues after the program. • impact is mainly personal, but organizational changes may follow. • the library is using the tools to varying degrees of success. • organizational blocks prevent use of the tools. the following sections examine each of these thematic areas more closely. personal practice is enhanced with knowledge and confidence, and exploration continues after the program a significant benefit or impact of the program is the increased knowledge and confidence for those who have participated. when asked to gauge impact on the organization, survey respondents noted organizational change is not as prevalent, but staff feel more comfortable and “in the know.” for libraries that have offered the program, the results of this study point to a library staff that is “more confident in exploring new technology” and “staff who were scared of technology feel more confident.” these findings support mezi row’s transformative learning theory (1991). cranton’s previously cited definition of transformative learning highlights the gaps of library staff exploring tools such as blogging, twitter and facebook as a way to understand emerging technology and what role the library might play with users: “when people critically examine their habitual expectations, revise them, and act on the revised point of view, transformative learning occurs” (2006, p. 19). the findings of this study also support candy’s (2004) designation that “learner control” allows the learner to “take control over a narrow range of choices” (p. 50) and that online education can free the learner to explore beyond specified course material. impact is mainly personal, but organizational changes may follow many of the answers for the question concerning organizational change highlighted impact at the individual level but some also pointed to the potential changes that could occur as staff continue to utilize technologies covered in the program. these changes center table 4 changes to personal professional practice after the program “what changes have you made to your personal professional practice because of the program?” no. % i use the tools i discovered to enhance my work 20 46.5 i share/collaborate more because of the tools 3 6.98% i use rss to read feeds/keep current 6 13.95% i feel generally more in the know/comfortable/current 8 18.60% i communicate with my coworkers and colleagues using the tools 6 13.95% evidence based library and information practice 2012, 7.1 62 on improved library service and better interaction with library users. one respondent noted “our library staff are now more willing to embrace new technologies and ways of ‘getting our message out there’” while another responded: “staff are talking about how to incorporate these technologies into customer service offerings.” one respondent noted the program lead to “awareness and ability to assist the customers.” further evidence of institutional impact is found in responses such as this: “quiet staff who were nonetheless willing to play within the learning 2.0 program have now become key proponents in moving our library service and its web applications forward they have found their voice and are respected for using it.” promotion of these services and approaches toward emerging technologies, despite the varying degrees of success, benefits the library as a whole. this response supports this thinking about the future: the program provided “an awareness that things are changing and the need to learn and address the issues with this so we can stay relevant & receive the funding we need.” the library is using the tools to varying degrees of success one respondent stated: “not a lot has changed i have more access to info & ideas that i wouldn't have had before but time and resources are very limited and it's a struggle to get a lot of our basic duties done.” this response is indicative of the third thematic area of impact and effect. some libraries are implementing emerging technologies, but the success of many initiatives remains unclear. despite this, concrete results did arise from this program: for example, one respondent described the “appointment of a new technologies librarian.” organizational blocks prevent use of tools culled from various sections of the survey, this thematic area highlights the fact that in some instances blocks and prohibitive policies inhibit use of the tools. this may be within the library or from external governing entities. some respondents expressed frustration at a lack of “progressive” and “archaic” approaches to emerging technologies by it departments and local government. statements reported general blocks, “we are unable to do most things covered by web2,” while others reported that managers did not approve of use of the tools: “participants continue to lobby supervisors for wider use of learning 2.0 technologies.” highlighting the personal impact of the program on individuals but a lack of impact organizationally, one respondent reported: “many staff have moved along with technology though it is the one that provides the most obstructions be it from a control / security / non progressive point of view.” outside of the library, one respondent noted that the “it department at council has not encouraged use of web 2.0.” exemplary practice one goal of this research, as supported by caval in 2009, is the development of a list of exemplary practice for learning 2.0. analysis of the key questions from the survey leads to a preliminary list of exemplary practice to ensure success for libraries launching learning 2.0 programs as an extension of professional development (pd) activities. the list, originally published by stephens and cheetham (2011), is further supported by the findings of this area of the project. it includes: • understand the program yields the personal benefits of confidence and a willingness to explore first, organizational benefits may follow. • the program can promote the library and its services to other departments and increase credibility and visibility. • allow staff time to work on the program and make it a firm commitment. • break down any barriers on the tools put in place by it departments making sure access is possible from employee computers. evidence based library and information practice 2012, 7.1 63 limitations and future research limitations of this study include questions not utilized in the analysis and lack of supporting focus group data. two questions (q10 and q11) in the survey were removed from the data set because of an error with the survey software. due to scheduling issues at the queensland public library association conference, only one focus group of public library staff was conducted with a low number of participants. those results are not reported in this article. other public library focus groups conducted at city libraries townsville could not be used because the program was ongoing during the time of the research visit. these limitations, however, will influence future study of the learning 2.0 phenomenon. more focus groups and surveys in other areas and within other types of libraries wield more data and support for the findings. analysis of the generational questions from the survey is planned for future publication as well as an examination or respondents views of management support and communication during and after the program. future research of the learning 2.0 program should further explore the impact of the program and should focus on institutional adoption of the tools, planning and evaluation. this would allow for an even clearer picture of impact on library services. further research will include fine tuning the survey instruments and sampling other groups of library staff in areas where learning 2.0 has been frequently adopted, such as the united states and the netherlands. conclusion the results of this study provide evidence that the adoption of learning 2.0 as a professional development program for emerging technologies yields personal benefits for public library staff that include increased confidence, a willingness to explore and increased comfort with technology. most participants in the survey reported success in the program, with only a small number of the public library data subset reporting inability to complete the program due to lack of time or support. success from the program is described in various ways, including incorporating the tools in the library setting, feeling more comfortable and confident exploring new technologies, adopting the tools into personal practice and better awareness of emerging technologies overall. a useful benefit for those launching such professional development programs are suggestions for exemplary practice based on this evidence. these ideas and the results of this study offer support for professional development librarians and administrators to utilize this free, open program for staff training. references abram, s. (2008). the 23 things learning 2.0. stephen’s lighthouse. retrieved 26 feb. 2012 from http://stephenslighthouse.com/2008/02 /05/the-23-things-learning-2-0/ blowers, h. (2008). ten tips about 23 things. school library journal, 54(10): 53-57. retrieved 26 feb. 2012 from http://www.schoollibraryjournal.com/ article/ca6600689.html blowers, h. (2009). wj hosts 23 things summit. retrieved from http://www.librarybytes.com/2009/02/ wj-hosts-23-things-summit.html [no longer accessible] candy, p. c. (1991). self-direction for lifelong learning: a comprehensive guide to theory and practice. san francisco, ca: jossey-bass. candy, p. c. (2004). linking thinking: selfdirected learning in the digital age. canberra: dept. of education, science and training. retrieved 26 feb. 2012 from http://www.dest.gov.au/sectors/trainin g_skills/publications_resources/profile s/linking_thinking.htm#versionavaila ble http://stephenslighthouse.com/2008/02/05/the-23-things-learning-2-0/� http://stephenslighthouse.com/2008/02/05/the-23-things-learning-2-0/� http://www.schoollibraryjournal.com/article/ca6600689.html� http://www.schoollibraryjournal.com/article/ca6600689.html� http://www.dest.gov.au/sectors/training_skills/publications_resources/profiles/linking_thinking.htm#versionavailable� http://www.dest.gov.au/sectors/training_skills/publications_resources/profiles/linking_thinking.htm#versionavailable� http://www.dest.gov.au/sectors/training_skills/publications_resources/profiles/linking_thinking.htm#versionavailable� http://www.dest.gov.au/sectors/training_skills/publications_resources/profiles/linking_thinking.htm#versionavailable� evidence based library and information practice 2012, 7.1 64 cranton, p. (2006). understanding and promoting transformative learning (2nd ed.). san francisco, ca: josseybass. gross, j. & leslie, l. (2010). learning 2.0: a catalyst for library organizational change. the electronic library, 28(50): 657 – 668. doi: 10.1108/02640471011081942 hanly, b. (2007). public library geeks take web 2.0 to the stacks. retrieved 26 feb. 2012 from http://www.wired.com/culture/educati on/news/2007/03/learning2_0 hastings, r. (2007). journey to library 2.0. library journal. retrieved 26 feb. 2012 from http://www.libraryjournal.com/article/ ca6431957.html hiemstra, r. (1976). lifelong learning: the professional education series. lincoln, ne: professional educators publications. hough, b. (2006). teaching people to be savvy travelers in a technological world. computers in libraries, 26(5): 8-12. jenkins, h. (2006). confronting the challenges of participatory culture: media education for the 21st century. macarthur. retrieved 26 feb. 2012 from http://digitallearning.macfound.org/atf /cf/%7b7e45c7e0-a3e0-4b89-ac9ce807e1b0ae4e%7d/jenkins_whit e_paper.pdf knowles, m. s. (1970). the modern practice of adult education: andragogy versus pedagogy. new york, ny: cambridge. krueger, r. a., and casey, m. a. (2009). focus groups: a practical guide to applied research (4th ed.). thousand oaks, ca: sage. lewis, l. (2008). library 2.0: taking it to the street. vala 2008: libraries/changing spaces, virtual places: 14th biennial conference and exhibition, melbourne, australia. retrieved 26 feb. 2012 from http://www.vala.org.au/vala2008proceedings/vala2008-session-1-lewis mackenzie, c. (2007). proceedings from the pla 2007 conference: creating our future workforce: planning for library 2.0 and beyond. retrieved from http://www.pla.org.au/conf2007/pape rs/creating_our_future.pdf maxymuk, j. (2008). 23 things and more. the bottom line: managing library finances, 21(2): 64-66. doi:10.1108/08880450810898337 merriam, s., and caffarella, r. s. (1999). learning in adulthood. san francisco, ca: jossey-bass. mezirow, j. (1991). transformative dimensions of adult learning. san francisco, ca: jossey-bass. mezirow, j. (1997). transformative learning: theory to practice. new directions for adult and continuing education, 74: 5-12. doi:10.1002/ace.7401 stephens, m., and cheetham, w. (2011). the impact and effect of learning 2.0 programs in australian academic libraries. new review of academic librarianship, 17(1): 31-63. doi:10.1080/13614533.2011.547414 titangos, h-l., and mason, g. l. (2009). learning library 2.0: 23 things @scpl. library management, 30(1/2): 44-56. doi:10.1108/01435120910927510 http://www.wired.com/culture/education/news/2007/03/learning2_0� http://www.wired.com/culture/education/news/2007/03/learning2_0� http://www.libraryjournal.com/article/ca6431957.html� http://www.libraryjournal.com/article/ca6431957.html� http://digitallearning.macfound.org/atf/cf/%7b7e45c7e0-a3e0-4b89-ac9c-e807e1b0ae4e%7d/jenkins_white_paper.pdf� http://digitallearning.macfound.org/atf/cf/%7b7e45c7e0-a3e0-4b89-ac9c-e807e1b0ae4e%7d/jenkins_white_paper.pdf� http://digitallearning.macfound.org/atf/cf/%7b7e45c7e0-a3e0-4b89-ac9c-e807e1b0ae4e%7d/jenkins_white_paper.pdf� http://digitallearning.macfound.org/atf/cf/%7b7e45c7e0-a3e0-4b89-ac9c-e807e1b0ae4e%7d/jenkins_white_paper.pdf� http://www.vala.org.au/vala2008-proceedings/vala2008-session-1-lewis� http://www.vala.org.au/vala2008-proceedings/vala2008-session-1-lewis� http://www.pla.org.au/conf2007/papers/creating_our_future.pdf� http://www.pla.org.au/conf2007/papers/creating_our_future.pdf� / evidence based library and information practice/ / review article   textbook alternative incentive programs at u.s. universities: a review of the literature   ashley lierman instruction & education librarian campbell library rowan university glassboro, new jersey, united states of america email: lierman@rowan.edu   received: 27 mar. 2020                                                              accepted: 9 sept. 2020      2020 lierman. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29758     abstract   objective – this article reviews current literature on incentive grant programs for textbook alternatives at universities and their libraries. of particular interest in this review are common patterns and factors in the design, development, and implementation of these initiatives at the programmatic level, trends in the results of assessment of programs, and unique elements of certain institutions’ programs.   methods – the review was limited in scope to studies in scholarly and professional publications of textbook alternative incentive programs at universities within the united states of america, published within ten years prior to the investigation. a comprehensive literature search was conducted and then subjected to analysis for trends and patterns.   results – studies of these types of programs have reported substantial total cost savings to affected students compared to the relatively small financial investments that are required to establish them. the majority of incentive programs were led by university libraries, although the most successful efforts appear to have been broadly collaborative in nature. programs are well-regarded by students and faculty, with benefits to pedagogy and access to materials beyond the cost savings to students. the field of replacing textbooks with alternatives is still evolving, however, and the required investment of faculty time and effort is still a barrier, while inconsistent approaches to impact measurement make it difficult to compare programs or establish best practices.   conclusion – overall, the literature shows evidence of significant benefits from incentive programs at a relatively low cost. furthermore, these programs are opportunities to establish cross-campus partnerships and collaborations, and collaboration seems to be effective at helping to reduce barriers and increase impact. further research is needed on similar programs at community colleges and at higher education institutions internationally.     introduction   the cost of textbooks is prohibitive for many postsecondary students. the national center for education statistics found that for the 2016–17 academic year, the average cost of books and supplies for entering full-time undergraduate students at four-year institutions was $1,263, almost 10% of the average cost of tuition (national center for education statistics, 2017). while textbook prices are no longer rising as quickly as they were in the earliest part of the 21st century, in part due to institutional efforts to make lower-cost options available (levitan, 2018), providing relief from textbook costs is still a major concern for student success and college affordability. this is particularly true since high textbook costs have been shown to prevent students from acquiring needed materials for the academic curriculum (senack, 2014).   one of the ways that colleges and universities have responded to this issue is by encouraging faculty to replace traditional course textbooks with materials that are available to students at no additional cost. these may include resources that are owned by the institution’s library or open educational resources (oer). oer are commonly defined as educational resources, most often but not always available digitally and online, that are both free of cost and freely available for use, adaptation, and redistribution (wiley et al., 2014). both of these types of resources, however, tend to be less centralized and marketed to faculty than traditional textbooks, and faculty feedback has indicated that the cost of time and labour associated with creating, adapting, implementing, or even simply locating those that are appropriate can be substantial (bell, 2012; delimont et al., 2016; batchelor, 2018). belikov & bodily (2016) have identified other significant barriers to faculty adoption of oer specifically, most notably lack of information on oer, lack of discoverability of oer, and confusion over the distinctions between oer and other types of resources.   to overcome these barriers, over the past decade, a growing number of postsecondary institutions have begun to offer faculty small financial incentives to encourage the use or development of textbook alternatives. typically, these initiatives take the form of a small grant program where faculty apply and agree to certain requirements, a body within the institution evaluates their proposals, and a certain number of applicants are awarded some type of financial remuneration for the effort that their implementation of an alternative will entail. after their courses have been taught, participating faculty may be required to report out on their experiences, participate in later assessment efforts, or do both.   aims   the aims of the present work are to conduct a narrative review of the professional and scholarly literature specifically on incentive grant programs for textbook alternatives and to seek out themes and commonalities in the experiences of the authors and other participants. key items to investigate include common patterns and factors in the design, development, and implementation of these initiatives at the programmatic level, trends in the results of assessment of programs, and unique elements of certain institutions’ programs and what impact they appear to have had. finally, conclusions and recommendations are drawn from the literature that may be used to inform developers and maintainers of similar programs in the future.   methods   the first step in the process of gathering literature for this review was establishing a general scope for inclusion. while community colleges have developed a significant number of textbook alternative incentive programs, these programs differ significantly in implementation from those at four-year colleges and universities, and the latter is the focus of this present study. similarly, many more incentive programs exist or have existed than those discussed here, but the set of cases to follow includes only those that have been presented in the literature to maximize the narrative description, reflection, and assessment data that are available. the scope of this work is also limited to studies of full incentive programs not individual course implementations or specific resources. additionally, in recognition of the significant international differences around issues of oer and college affordability, i have considered here only studies of institutions from the united states. a comparison of global trends in oer implementations and incentives in higher education would be a valuable direction for future study but is not the aim of the present work.   with these foci in mind, i constructed and ran a search in five databases: educational resources information center, or eric; education source (ebsco); educational administration abstracts; library literature & information science full text (h.w. wilson); and library, information science & technology abstracts, or lista. i selected this set of databases because it comprises those available to me with the most significant literature coverage in education and in library and information science, with multiple databases included in each area to increase comprehensiveness. the search included results published since 2010 that contained variants of the terms "affordable," "alternative," or "replace" anywhere, contained variants of the terms "program" or "initiative" or "fund" or "grant" anywhere, and contained variants of the term "textbook" specifically in the title or subject terms. the term "open" was originally included in the first set of terms, but it returned too many irrelevant results simply concerning the development of individual open textbooks and did not significantly increase the relevant results over the other terms and as such was ultimately eliminated.   when initially conducted, the search retrieved 152 results, of which approximately 30 were selected. i reviewed and weeded these initial 30 results for their relevance to the scope stated previously and reviewed the bibliographic references of each work for additional relevant publications that the original search might have missed. the search also included using the directory of institutions available from sparc's connect oer site to identify institutions with financial incentive programs for materials replacement and searching the names of these individual institutions along with variants of the word "textbook" across multiple library and information science databases. in this stage, the search specifically focused on library and information science databases because of sparc's large academic library membership. after this stage, the results underwent a similar process of review, weeding, and citation mining.   additional criteria for inclusion and exclusion emerged and were applied after this process to further narrow the results. sources were included only if they described the development of the program, the assessment of the program, or both in enough detail to answer most of the following questions:   who on campus led or leads the program? when and how did it begin? what were the steps taken to begin the program? who were the partners onand off-campus? from where did funding come? what was the total funding and how much was awarded to each recipient? what were the application requirements? how many applications have there been? what was assessed? what were the results? what strategies have been considered to increase the impact of the program?   i selected these questions for their particular importance to campus stakeholders who would be responsible for the creation of an incentive program and who would be most interested in knowing what results such a program might produce. sources were excluded if insufficient detail about the formation or assessment of the program was provided to answer most of these questions. for example, cases where an incentive program was discussed briefly as part of a broad description of campus oer efforts were omitted. some sources were also excluded because their primary focus appeared to be the creation of an oer publishing platform that happened to be incentivized by grants rather than focusing on replacing textbooks and reducing course costs through the incentive program. studies focusing more on the content, delivery systems, or pedagogical value of textbook alternatives than on the development and functioning of an incentive program were likewise considered to be out of scope, unless they were connected to a program on which other, more general studies were available. after establishing a final list of remaining studies had been, i took extensive notes on any description of the development and assessment of programs that had been included and analyzed the results for trends and recurring themes across institutions. specifically, the answers to each question identified previously were compared across institutions and coded into commonly recurring categories or noted as unique. where no answer was found in the literature to a question for a given institution, more information was sought on the website and other publicly available materials of the institution's textbook alternative program. if no information could be found by consulting these materials, the answer to that question was noted as "not stated."   results   table 1 provides the u.s. institutions included in this review, with the years that their textbook alternative incentive programs began. (some of the institutions listed established these initiatives as part of a larger program for textbook affordability, but the date provided is the date that the incentive program, specifically, began.)   leaders and partners   in the studies examined, program leaders have overwhelmingly been university libraries or library systems. where libraries were not the sole program leaders, programs were instead led by campus-wide committees that included library representatives; libraries were represented in the leadership of all programs considered. in most cases, however, onor off-campus partners have also supported programs in conjunction with libraries. table 2 identifies the leadership and additional partners of each included institution.   university bookstores were, by a narrow margin, the most common partner on textbook alternative incentive programs. this seems surprising given bookstores' presumed interest in the continued sale of traditional textbooks. while agee and mune (2014) note the apparent strangeness of such partnerships, they claim that most university bookstores now rely on the revenue streams from other merchandise more than that of textbooks and tend to find that the goodwill generated by collaborating on textbook affordability outweighs the revenue lost by decreasing textbook sales (p. 18).   table 1 institutions and start dates of their textbook alternative incentive programs start date institution(s) 2010 temple university 2011 university of massachusetts-amherst 2013 north carolina state university kansas state university university of california, los angeles san jose state university 2014 university of oklahoma 2015 university system of georgia east carolina university & university of north carolina-greensboro (joint collaboration) university of texas at san antonio 2016 rutgers university university of washington florida state university university of north dakota   table 2 participants in textbook alternative incentive programs by institution institution program leader(s) other partners on program temple university library none university of massachusetts-amherst library faculty centre openstax open textbook network provost's office north carolina state university library university bookstore kansas state university collaborative university-wide faculty team, including library representatives student government university administration university senate ucla library california digital library student government university bookstore university senate university system administration san jose state university library faculty centre university bookstore university system administration university of oklahoma library college of arts and sciences college of business faculty centre openstax open textbook network university system of georgia libraries network galileo (virtual library project) online core curriculum leadership openstax state of georgia university presses east carolina university & university of north carolina-greensboro libraries provost's office university bookstore university of texas at san antonio library faculty centre openstax registrar student government university bookstore rutgers university library student section of njpirg (public interest research group) university of washington library friends of the uw library organization open textbook network rebus foundation florida state university library none university of north dakota university-wide committee chaired by library and provost representatives faculty student government technology and instruction centres   funding and awards   numerical comparisons of incentive programs based on the literature are not necessarily definitive due to differences in measurement strategies, lengths of assessment periods, and other factors between studies. nonetheless, a few rough patterns do emerge on comparison of funding sources, total amounts, and amounts per award by program. table 3 shows this information (where available) for the represented institutions.   library budgets were the most common source of funds (where stated) by a significant margin, while various grant sources from within or without the university system were also relatively common. no individual program described investing more than $60,000 total in incentive grants, and most total funding pools were somewhere between $10,000 and $40,000, with a substantial number also totaling less than $10,000. some institutions opted for a flat amount for individual awards, while others used tiered funding distributions that provided larger incentives to faculty teaching higher-enrolment or higher-impact courses. in either case, only one program offered award amounts of less than $500 and only one offered amounts of more than $5000, and for the minimum award amount was most commonly between $500 and $1000.   table 3 funding and awards for textbook alternative incentive programs by institution institution funding source total funding pool amount per award temple university library budget not stated $1000 university of massachusetts-amherst library budget provost $10,000 $1000 for smaller classes $2500 for larger classes north carolina state university not stated not stated between $500 and $2000 kansas state university university grant library budget administration (later) $60,000 (first round) $40,000 (second round) $50,000 (administration funding) not stated ucla library budget university system grant campus partners $27,500 $1000 for courses under 200 enrolment $2500 for courses over 200 enrolment san jose state university university system grant $20,000 (first round) $49,000 (second round) $500–$2000 (first round) $1000 (second round) $1500 (final) university of oklahoma   not stated $9600 (pilot) $1200–$2500 (pilot) $250–$2500 (second year) university system of georgia state budget not stated up to $10,800 for courses under 500 enrolment up to $30,000 for courses over 500 enrolment east carolina university & university of north carolina-greensboro library budget provost state grant $10,000 (pilot) not stated for grant phase $1000 university of texas at san antonio library budget $7500 $1500 rutgers university library budget donor funding not stated $500 $1000 university of washington   friends of the library grant $4500 $1500 florida state university library budget $6000 $1000 university of north dakota   local foundation library donor fund $25,000 (partially for non-incentive costs) $3000   table 4 applicants and requirements for textbook alternative incentive programs by institution institution total applicants accepted applicants grant requirements temple university 11 11 proposal only university of massachusetts-amherst   8 8 workshop attendance assessment syllabus submission repository deposit of materials final report north carolina state university not stated not stated; 20 total courses application only kansas state university 14 12 application only ucla 27 not stated workshop attendance san jose state university 23 not stated; 25 total sections in first round workshop attendance syllabus submission university of oklahoma not stated 5 application only university system of georgia not stated 29+ in round 1 assessment sustainability measures open access to materials final report peer review (highest level) east carolina university & university of north carolina-greensboro 22 (pilot) not stated (grant phase) 10 (pilot) 38 (grant phase) meet with librarian university of texas at san antonio 11 (first round) 33 (second round) 5 (first round) not stated (second round) application only rutgers university not stated 32 (first round) 57 (by time of writing) assessment syllabus submission university of washington 3 2 proposal only florida state university 7 6 memo. of understanding workshop attendance meet with librarian university of north dakota   2 2 workshop attendance meet with librarian   applicants and requirements   most programs appear to have had relatively few applicants and awardees, where numbers were provided, with almost all having fewer than 30 total applicants and several having fewer than 10. a majority of programs also accepted a relatively high percentage of their applicants, with a significant majority either accepting all applicants or accepting more than two-thirds of the total pool. only two programs that provided application and acceptance statistics accepted fewer than 50% of those who applied. by a narrow margin, the majority of programs also required only an application or proposal, but other common requirements were for faculty to attend a workshop on implementing textbook alternatives, to submit a proposed syllabus revisions incorporating the replacement materials, to participate in some form of assessment of the program, to meet with a librarian for support, to submit a final report on their project, or to make any modified or created materials available openly in kind either via the institutional repository or otherwise. applications for the university system of georgia’s grants were also required to describe the measures they intended to take for sustainability, and applications for the largest and most far-reaching award type (aimed at textbook replacements affecting entire departments or institutions) were required to undergo double-blind peer review (gallant, 2015).   table 4 provides the number of applicants to each program, the number that were accepted and funded, and what was required of faculty to apply for a grant where each was stated.   student impact and cost savings   as mentioned in the section on funding and awards, comparing the numbers of students impacted or their cost savings across multiple institutions is difficult since not all institutions measured comparable spans to one another, and the time frame or calculation formula used for a reported figure is not always clear. rough categories of impact do emerge from the literature, but these are not necessarily accurate representations of the current state of these programs. table 5 provides the reported estimates of students impacted and cost savings by programs where these were given.   with regard to the figure stated for the university system of georgia, it should be noted that croteau (2017) gives the figure for the first round of grants as $760,000, which does not seem compatible with the $9 million figure provided by gallant (2015). this may be due to a difference in calculation methods between the two authors, as the formulae in use are unclear. given the amount of the system’s grants, its state support, and affiliated efforts to eliminate materials costs for online courses across the system, however, it is also not impossible for the cost savings to have increased to this degree over time.   most studies that reported numbers of students impacted indicated that their programs had affected fewer than 2000 students—relatively few, given the enrolment numbers for most of these institutions. many of these studies were, however, reporting on pilot programs, and presumably future efforts would seek to expand their scope of impact. moreover, in those instances where both an initial total investment amount and an estimation of cost savings were included, the difference of the two was generally substantial. figure 1 shows the initial investment and cost savings by institution where stated.   an important caveat when comparing these numbers is the point that student cost savings are not calculated identically by each institution. the most commonly cited method of calculating cost savings was to multiply the cost of course materials before and after program participation by the number of enrolled students and subtract the latter from the former, but precise applications of this formula varied. at florida state, for example, this formula was used with an estimated average of students enrolled annually in the courses in question (soper et al., 2018), while ucla and umass-amherst used actual observed enrolment numbers but estimated the costs of course materials (smith, 2018; farb & grappone, 2014). reporting on the program at san jose state included both estimated and actual savings. the former was calculated based on the estimated number of students or the enrolment cap and the list price of previously used materials and the latter based on the actual observed total cost of course materials and number of students who actually enrolled (bailey & poo, 2018). at the university of oklahoma, potential cost savings were calculated in advance for purposes of evaluating proposals, using a similar formula of projected enrolment multiplied by original and reduced costs of materials, but it unclear whether this was also how the final cost savings were calculated (waller et al., 2018). a number of other studies provided no formula for their cost estimates at all, and descriptions of the programs at ut san antonio and the university of washington stated only that cost savings were calculated by their partners at openstax or the open textbook network (ivie & ellis, 2018; batchelor, 2018).   table 5 students impacted and cost savings from textbook alternative programs by institution institution estimated students impacted estimated cost savings temple university not stated not stated university of massachusetts-amherst 1600 (2011–2015) $101,632 (2011–2015) north carolina state university not stated $250,000 kansas state university 10,941 (2015–16) 17,963 (first two years total) $921,000 (2015–16) $1.61 million (first two years total) ucla over 1000 $112,000 san jose state university 777 (first round) $117,739 (first round) university of oklahoma 420 (pilot) $116,000 (pilot) $274,000 (second year, first semester) university system of georgia not stated $9 million (first two rounds) east carolina university & university of north carolina-greensboro not stated (pilot) 3300 total (grant phase) $150,120 (pilot) $547,000 total (grant phase) university of texas at san antonio 568 $94,000 rutgers university 9000 over $2 million university of washington 180 $27,000 florida state university not stated $56,000 university of north dakota not stated $3.7 million maximum (two years)   figure 1 initial investment versus cost savings (where stated).   further complicating the matter, several authors suggested that student cost savings may be lower than the estimations because of methods that students commonly use to acquire textbooks for less than what the texts would cost if purchased new, such as rentals, buying used texts, using older editions, and similar methods (lashley et al., 2017; walker, 2018; todorinova & wilkinson, 2019). kansas state program evaluators even attempted to compensate for this consideration in their calculation formula for student cost savings by using the actual cost of textbooks in their calculations only if they would have cost under $100 and using $100 as a flat cost for any texts costing $100 or more (lashley et al., 2017). the study at the university of north dakota simply acknowledged that its calculation using original new textbook costs multiplied by enrolment numbers represents a maximum possible cost savings to students from the program and that the real impact was most likely lower (walker, 2018).   other trends in assessment   student impact and cost savings were the most commonly assessed data from incentive programs, but a number of programs also included assessment of other factors. in several studies, student academic performance was measured before and after the implementation of textbook alternatives, and in all cases performance was found to be the same or better afterward (smith, 2018; croteau, 2017; thomas & bernhardt, 2018). furthermore, grimaldi, mallick, waters, and baraniuk (2019) have pointed out that measures of textbook alternatives’ impact on student learning to date have probably underrepresented the benefits because the measures examine the difference in performance of all students in the course and not only the students who could not otherwise have afforded access to the textbooks, which does not accurately represent where the impact on learning should be expected.   table 6 results of student feedback on textbook alternative programs by institution institution positives noted by students negatives noted by students temple university cost savings preference for print kansas state university cost savings ease of access customization preference for print dislike of specific replacements used ucla cost savings not stated university of texas at san antonio accessibility ease of use not stated   rutgers university cost savings ease of use difficulty in notetaking and collaboration   table 7 results of faculty feedback on textbook alternative programs by institution institution positives noted by faculty negatives noted by faculty temple university   cost savings to students increased student access ability to customize & update time investment   north carolina state university improved teaching not stated kansas state university cost savings to students improved teaching ability to customize & update perceived student satisfaction time investment technological issues copyright challenges ucla improved teaching not stated university system of georgia improved teaching not stated university of texas at san antonio improved teaching quality concerns university of washington improved teaching ability to customize & update time investment quality concerns   beyond performance measures, both faculty and student feedback on textbook alternative incentive programs was overwhelmingly positive at all institutions where it was collected. tables 6 and 7 detail specific positives and negatives noted in student and faculty feedback, respectively, for those programs where it was collected.   it is also of note that when students were asked to evaluate the quality of the materials they were provided in lieu of textbooks at university system of georgia institutions, the principal finding was that students were not effective evaluators of resource quality, and their estimations were disproportionately swayed by superficial factors like visual appearance (croteau, 2017).   identified challenges   some authors identified major challenges in implementing their institutions’ programs. as suggested previously, one of the most commonly recurring challenges was the amount of time and effort that implementation required for participating faculty, and some program organizers observed a need for increased recognition of faculty efforts in this area with regard to tenure and other professional advancement decisions (agee & mune, 2014; delimont et al., 2016; bazeley et al., 2019). the need for relevant faculty training and support was also widely recognized, and faculty feedback at some institutions indicated that more support was needed than had been provided (bailey & poo, 2018; young, 2016; delimont et al., 2016; subramony, 2018).   strategies for sustainability and increasing impact   many of the programs discussed in the literature were in early stages or pilot versions at the time of writing, and few were in a position to discuss any sustainability planning or outcomes specifically. most authors, however, at least discussed future directions for the program in question, the majority of which focused on increasing the program's impact. these strategies could be said to be a means of planning for sustainability in themselves, as the greater the program's apparent success the greater the likelihood of continued funding and labor to support it.   the most common planned strategies for ensuring sustainability and increasing the impact of programs were targeting courses with particularly high enrolment or with high course costs or both and working to increase collaboration with additional partners across campus, particularly faculty and other units. table 8 lists the planned strategies for increasing impact for the institutional programs where they were given.   unique program elements   while some common trends can be observed across multiple institutions, there are a few programs with unique and notable elements in their design, implementation, or context. the university system of georgia’s textbook transformation grants program, for example, clearly represents something of a standout case among those described as it spans a full system of state institutions, is funded at the state level, and provides awards that are closer to full grants than the micro-grant models used at other universities. it is also unusual because, while other programs offer tiered awards based on enrolment numbers, the textbook transformation grants program actually offers four different grant types based on type of alternative implementation: one for faculty simply using oer or other resources with no cost to students; one for faculty using open textbooks produced by the initiative openstax with whom the program is partnered; one for faculty creating course packs sourced from library resources in partnership with a librarian; and one for large-scale transformations of multiple courses, a department, an institution, or multiple institutions (croteau, 2017). at the same time, the last funding level would surely not be possible without state-level support for the program and the possibility of relatively large awards. similarly, ucla and san jose state were both able to develop relatively large funding pools and disburse relatively high numbers of awards in large part due to investment in their programs from the state level (farb & grappone, 2014; bailey & poo, 2018). when local governments invest in the affordability of higher education in this manner, there does seem to be an impact on the relevant programs.   internally, however, other institutions have been able to use innovative approaches to improve the effectiveness of their incentive programs. ncsu’s program, one of the oldest and most influential, stands out for its use of data from its parallel textbook lending program to inform choices of target for the textbook alternative incentive program (thompson et al., 2017). this hybridization shows the value of taking a multivalent approach to textbook affordability and how one initiative at an institution can be used to improve another. kansas state’s program is remarkable for its unusual level of success and penetration into the culture of the university, with support from across the administration and multiple units of the institution, and a funding pipeline directly from university-level student fees and donations (lashley et al., 2017). the secret to its success may be in its origin as a multi-departmental faculty collaboration, which was effective and timely enough to attract the interest and support of the administration. finally, the program at east carolina university and unc-greensboro is unique in being a partnership between two universities to create a communal incentive program and thus maximize their resources and return. even where other institutions are within the same state or even system, most have tended to maintain their own individual programs. the case of east carolina university and unc-greensboro, however, alongside that of the university system of georgia shows that cross-institutional collaboration has the potential to make universities more successful in their efforts than they could be alone.   table 8 planned strategies for ensuring sustainability and increasing impact by institution institution sustainability and impact strategies planned university of massachusetts-amherst targeting high-enrolment/high-cost courses increasing collaboration across campus moving to a tiered funding structure increasing overall funding providing release time for participating faculty north carolina state university targeting high-enrolment/high-cost courses providing greater support to participants moving to a tiered funding structure seeking support from student government departmentor curriculum-level replacement kansas state university increasing collaboration across campus departmentor curriculum-level replacement funding program from student tuition ucla targeting high-enrolment/high-cost courses seeking more applicants assessment and program improvement san jose state university targeting high-enrolment/high-cost courses departmentor curriculum-level replacement seeking more applicants university of oklahoma targeting high-enrolment/high-cost courses increasing collaboration across campus providing greater support to participants pushing open sharing of adapted/created materials east carolina university & university of north carolina-greensboro targeting high-enrolment/high-cost courses providing greater support to participants university of texas at san antonio increasing collaboration across institutions rutgers university increasing student awareness of affordability initiatives university of washington using rebus foundation partnership to distribute labor florida state university   targeting high-enrolment/high-cost courses increasing collaboration across campus increasing collaboration across institutions university of north dakota increasing collaboration across campus seeking support from student government   discussion   limitations   by its nature, the present review is limited in its representation of textbook alternative incentive programs. as a narrative review of the literature, it is bound by the acknowledged limitations of such reviews, specifically a lack of critical appraisal of the evidence found in the literature and strict evaluative criteria for inclusion. given the relative newness of these types of programs and the scarcity of the available literature, all relevant studies were included to maximize the size of the data pool without regard for methodological rigour by individual authors. this uncritical approach and the inconsistencies in available data from the studies that were included may ultimately skew the perceived results.   furthermore, for the reasons that were discussed in the methods section, i consulted only published literature (and primarily peer-reviewed scholarly and professional literature). this decision conflicts, however, with the fact that even the oldest programs of this type are less than a decade old, and many programs are likely not yet at a stage to yield publishable results. programs not represented in this review may eventually yield significantly different results than those that have been discussed. many programs are also likely still not in their final forms and may continue to change over time given the relative newness of these types of intervention. there is a need for ongoing investigation and review of incentive programs like those discussed here as well as similar discussion of programs at community colleges and outside the u.s.   libraries as collaborative leaders   it is fair to say that libraries provide the leadership for the majority of incentive programs discussed here. equally apparent, however, is that in each of these cases partnerships with other bodies across campus, and even outside of it, have been vital.  involving administrators, faculty, and students in the process of managing incentive programs and other textbook affordability measures has been a key component of the success of all of these programs and has allowed the library to build buy-in across communities, share leadership with other stakeholders, and learn more about their needs and perspectives on the issues. working with broader oer organizations and communities also provides leadership support for librarians in working with these programs and in many cases has helped to source the resources that faculty use when replacing their textbooks (such as in the cases of partnerships with openstax and the open textbook network). the program descriptions indicate that support from the state government can increase what an incentive program is capable of offering and accomplishing—but it is quite possible for a program to be very extensive, well-funded, and successful without the support of the state, such as in the case of kansas state. funds can be drawn from a variety of sources, and strong collaborations within campus seem from the literature to be a more reliable predictor of success than support from without.   benefits of incentive programs   another strong indication of the literature is that the return on investment of incentive programs is very high, both in terms of numbers of students impacted and the textbook cost savings effected. none of the programs examined seem to have invested much more than $50,000 total in their incentives and most much less than that. yet student cost savings have been reported in the hundreds of thousands or even millions for the same programs with impacts on hundreds, thousands, or tens of thousands of students. as previously mentioned, establishing direct connections between the inputs and outputs of various programs is difficult due to their differences in measurement approaches, but the total funding amount for a program does not always seem to be closely related to its eventual impact. the number of students that each individual faculty recipient is able to reach by replacing textbooks may be a more significant factor than how many faculty receive awards or the size of the awards they receive. in any case, it is impressive to produce these kinds of results by distributing micro-grants of only $500 to $2500 to only 10 to 30 faculty members. tiered awards by enrolment numbers may be an effective approach to targeting higher-impact courses, although enough assessment data of such structures is not yet available to make a determination.   beyond cost savings, faculty and student responses to these programs have been reported as highly positive across all studies where they have been collected, with some notable minor drawbacks failing to outweigh the overall benefits. not only do students and faculty both value the financial savings for students in these cases, but faculty at some institutions have reported feeling that they have become better, more thoughtful, and more innovative educators as a direct result of implementing textbook alternatives. using oer or strategic selections from the library collection appears to help faculty think more critically and more deeply about their subject matter than does simply using a preset commercial textbook, and developing new oer can be seen as a valuable scholarly pursuit that deepens disciplinary knowledge and pedagogical deliberation. a vitally important next step, however, will be the appropriate recognition of this work with respect to faculty tenure and professional advancement decisions. not only is it vital to acknowledge faculty efforts toward creating open resources as the scholarly participation that these resources represent, but also it is necessary for faculty to be supported in this way if they are to make time for participation in the oer world amid their already busy schedules. the studies also indicate the vital importance of providing support in the form of training, professional development, and guidance as faculty take on these new challenges so that their efforts are successful and their participation in the program continues. encouragingly, there is mounting evidence that fears about textbook replacement’s negative impacts on student performance have been unfounded, as the majority of cases have seen unchanged or improved academic achievement with the implementation of new resources. the positive impact is likely even greater than has been reported. the points made by grimaldi et al. (2019) about the insufficiency of statistical approaches in this area are well taken.   conclusions   the emerging literature on textbook alternative incentive programs indicates that these programs have a significant positive impact. primarily, the studies considered here have found these programs to greatly benefit students financially and to inspire improvements to faculty pedagogy. furthermore, the programs are relatively affordable to begin and maintain, especially compared to the returns on the investment that have been reported. there are still significant barriers to entry associated with these initiatives, particularly faculty time and training and buy-in from both faculty and students, but cross-campus collaborations and expanding the types of incentives offered to faculty may help to increase participation. it is also worth noting, however, that there is a great deal of diversity in institutional approaches to these types of programs. the literature shows no standardization to speak of nor even sufficient evidence for a set of best practices or recommendations to emerge. while enough prior examples exist that each new institution initiating an incentive program need not reinvent the wheel, program developers at each institution will have to carefully consider their institution's individual needs and characteristics to develop the approach to funding, leadership, number of awardees, implementation, and assessment practices that will be most effective locally.   a major concern for the future of several programs considered here is increasing student impact and the resulting cost savings. this is not surprising given that these are the primary criteria by which these types of programs have tended to be assessed. a number of potential strategies for accomplishing this have been discussed, but it may be that the most effective way to increase impact is simply to find ways to develop buy-in and investment from more and more units across campus as evidenced by the extraordinary success of the program at kansas state. indeed, the most important factor in these programs so far may also be the one that holds the key to growth and success in their future: partnerships. collaborations within and between universities between different fronts in the fight for college affordability and across systems and consortia all seem to hold the most promise in terms of improving and expanding textbook alternative incentive programs and other efforts to improve educational access and success. the strength of communities and organizations working together is clearly felt in all the success stories that have been recounted here, and if that lesson is taken to heart, even greater successes may lie ahead.   as research in this area is still limited, a number of possible directions exist for future studies to pursue. a review of the literature (and possibly other documentation) on programs at community colleges would be of value for comparison to these findings and analysis of the similarities and differences in approach between different institutional types. studies of the practices of institutions outside the u.s. would also be of significant interest. a more comprehensive review of data on all existing incentive programs, including those without associated publications, would be a daunting task but also potentially of substantial value. furthermore, as indicated by grimaldi et al. (2019), there is a need for more rigorous and more nuanced analysis of the impact of implementing alternatives on students' academic performance because the results in this area have thus far been inconclusive. similarly, moving toward standardization of institutional formulae for calculating student cost savings would be tremendously beneficial as future researchers seek to more accurately understand the impacts of these programs.   references   agee, a., & mune, c. (2014). getting faculty into the fight: the battle against high textbook costs. against the grain, 26(5), article 9. https://doi.org/10.7771/2380-176x.6843   bailey, c., & poo, a. (2018). teaming up with faculty: a new tactic in the textbook battle. against the grain, 30(5), article 53. https://doi.org/10.7771/2380-176x.8160   batchelor, c. transforming publishing with a little help from our friends: supporting an open textbook pilot project with friends of the libraries grant funding. in a. wesolek, j. lashley, & a. langley (eds.), oer: a field guide for academic librarians (pp. 415–432). https://scholarworks.umass.edu/librarian_pubs/71   belikov, o. m., & bodily, r. (2016). incentives and barriers to oer adoption: a qualitative analysis of faculty perceptions. open praxis, 8(3), 235–246. http://dx.doi.org/10.5944/openpraxis.8.3.308   bell, s. j. (2012). coming in the back door: leveraging open textbooks to promote scholarly communications on campus. journal of librarianship and scholarly communication, 1(1), ep1040. https://doi.org/10.7710/2162-3309.1040   croteau, e. (2017). measures of student success with textbook transformations: the affordable learning georgia initiative. open praxis, 9(1), 93–108. https://doi.org/10.5944/openpraxis.9.1.505   delimont, n., turtle, e., bennett, a., adhikari, k., & lindshield, b. (2016). university students and faculty have positive perceptions of open/alternative resources and their utilization in a textbook replacement initiative. research in learning technology, 24(1), 1–13. https://doi.org/10.3402/rlt.v24.29920   farb, s., glushko, r., orfano, s., & smith, k. (2017). reducing the costs of course materials. serials review, 43(2), 158–162. https://doi.org/10.1080/00987913.2017.1316628   farb, s., & grappone, t. (2014). the ucla libraries affordable course materials initiative: expanding access, use, and affordability of course materials. against the grain, 26(5), article 14. https://doi.org/10.7771/2380-176x.6848   gallant, j. (2015). librarians transforming textbooks: the past, present, and future of the affordable learning georgia initiative. georgia library quarterly, 52(2), 12–17. https://digitalcommons.kennesaw.edu/glq/vol52/iss2/8   grimaldi, p., waters, a., & baraniuk, r. (2019). do open educational resources improve student learning? implications of the access hypothesis. plos one, 14(3), 1–14. https://doi.org/10.1371/journal.pone.0212508   ivie, d., & ellis, c. advancing access for first-generation college students: oer advocacy at ut san antonio. in a. wesolek, j. lashley, & a. langley (eds.), oer: a field guide for academic librarians (213–238). https://scholarworks.umass.edu/librarian_pubs/71   lashley, j., cummings-sauls, r., bennett, a., & lindshield, b. (2017). cultivating textbook alternatives from the ground up: one public university’s sustainable model for open and alternative educational resource proliferation. international review of research in open and distributed learning, 18(4), 212–230. https://doi.org/10.19173/irrodl.v18i4.3010   levitan, m. (2018, august 6). textbook costs drop as nearly half of colleges use openstax. diverse issues in higher education. https://diverseeducation.com/article/121872/    national center for education statistics. (2017). average total cost of attendance for first-time, full-time undergraduate students in degree-granting postsecondary institutions, by control and level of institution, living arrangement, and component of student costs: selected years, 2010-11 through 2016-17 [data set]. retrieved from https://nces.ed.gov/programs/digest/d17/tables/dt17_330.40.asp?current=yes   senack, e. (2014). fixing the broken textbook market. u.s. pirg education fund and the student pirgs. https://uspirg.org/reports/usp/fixing-broken-textbook-market   smith, j. (2018). seeking alternatives to high-cost textbooks: six years of the open education initiative at the university of massachusetts amherst. in a. wesolek, j. lashley, & a. langley (eds.), oer: a field guide for academic librarians (333–350). https://scholarworks.umass.edu/librarian_pubs/71   soper, d., wharton, l., & phillips, j. (2018). expediting oer on campus: a multifaceted approach. in k. jensen & s. nackerud (eds.), the evolution of affordable content efforts in the higher education environment: programs, case studies, and examples (135–149). https://open.lib.umn.edu/affordablecontent/chapter/expediting-oer-on-campus-a-multifaceted-approach/   subramony, d. (2018). instructors’ perceptions and experiences re: creating and implementing customized e-texts in education courses. educational considerations, 44(1), 13. https://doi.org/10.4148/0146-9282.1692     thomas, w.j., & bernhardt, b.r. (2018). helping keep the costs of textbooks for students down: two approaches. technical services quarterly, 35(3), 257–268. https://doi.org/10.1080/07317131.2018.1456844   thompson, s., cross, w., rigling, l., & vickery, j. (2017). data-informed open education advocacy: a new approach to saving students money and backaches. journal of access services, 14(3), 118–125. https://doi.org/10.1080/15367967.2017.1333911   todorinova, l., & wilkinson, z. (2019). closing the loop: students, academic libraries, and textbook affordability. the journal of academic librarianship, 45(3), 268–277. https://doi.org/10.1016/j.acalib.2019.03.010   walker, s. (2018). facilitating culture change to boost adoption and creation of open educational resources at the university of north dakota. in k. jensen & s. nackerud (eds.), the evolution of affordable content efforts in the higher education environment: programs, case studies, and examples (150–161). https://open.lib.umn.edu/affordablecontent/chapter/facilitating-culture-change-to-boost-adoption-and-creation-of-open-educational-resources-at-the-university-of-north-dakota/   waller, j., taylor, c., & zemke, s. (2018). from start-up to adolescence: university of oklahoma's oer efforts. in a. wesolek, j. lashley, & a. langley (eds.), oer: a field guide for academic librarians (351–380). https://scholarworks.umass.edu/librarian_pubs/71   wiley, d., bliss, t., & mcewen, m. (2014). open educational resources: a review of the literature. in j. m. spector, m. d. merrill, j. elen, & m.j. bishop (eds.), handbook of research on educational communications and technology (4th ed., pp. 781–789).   evidence summary   use of esbco discovery tool at one university reveals increased use of electronic collections but decreased use in circulation of print collections   a review of: calvert, k. (2015). maximizing academic library collections: measuring changes in use patterns owing to ebsco discovery service. college & research libraries, 76(1), 81-99. http://dx.doi.org/10.5860/crl.76.1.81       reviewed by: aoife lawton systems librarian health service executive dr. steevens’ library, dr. steevens’ hospital dublin 8, ireland email: aoife.lawton@hse.ie   received: 01 sep. 2015   accepted: 04 nov. 2015      2015 lawton. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to find out what the effects of a discovery tool are in relation to usage of print and electronic library collections, and with the aim to measure the effects in three specific areas: circulation numbers, use of electronic resources, and interlibrary loan requests.   design – comparative quantitative analysis of usage statistics and data sets.   setting – a regional comprehensive university in the united states of america.   subjects – usage data from a university library.   methods – the methods used were informed by three hypotheses stated at the beginning of the study. first, an analysis of usage data of e-resources tested the hypothesis that the introduction of a discovery tool would increase use of e-resources. second, to test whether the use of print collections increased, circulation statistics including items borrowed via consortia and in-house use statistics were measured. finally, interlibrary loan statistics from 2010 to 2013 were collated to test if the ebsco discovery service (eds) led to a decrease in interlibrary loan requests.     main results – the introduction of the ebsco discovery tool resulted in increased use of ebscohost and other databases at the library in question. however, the library's circulation statistics decreased, with a drop of 28% of checkouts compared to the previous year. the drop is more pronounced with undergraduates, who checked out 39% fewer items after the eds was introduced. there was a 30% decrease in requests for borrowing items from a consortia. there was insufficient data to support or refute the third hypothesis.   conclusion – the implementation of a discovery tool at one library has had both postive and negative outcomes. an increase in the use of electronic collections was observed as a positive outcome, whereas a decrease in the use of print collections was a negative outcome. due to the findings of the study, the library revised its policy on content inclusion to the eds. any new content is now screened for suitability before it is included. as a changing student demographic evolves at the library, with an increase in distance and online learners, the library will grow its collection in line with their needs. the author notes that a further study is needed to examine ebook usage, and recommends that the library consider a move towards ebooks for all collections.   commentary   discovery tools have become commonplace in academic libraries, yet a paucity of research addresses their impact on usage of electronic and print collections. this research addresses this deficit. while the details of the collections available at the primary library and other libraries included in the research are unique to those libraries, the methodology may prove beneficial to other libraries, especially those that have adopted ebsco's discovery service. the findings are relevant for collection development librarians and systems librarians.    the limitations of the study are noted as being quantitative in nature. the usage statistics, as well as circulation statistics and interlibrary loans, are the main source of data. overall, this study adheres to a fair level of validity when checked against glynn’s (2006) critical appraisal checklist. however, there is little information provided about the study population. the author notes a changing student demographic, but there is no detail provided about the ratio of undergraduates to postgraduates, or distance learners to those on-site. the methods used to collect the data are described, including the use of standard counter jr1 reports. the data is captured at a valid time both prior and post intervention of the eds. it is unclear, however, whether all of the library’s database subscriptions remained consistent for the duration of that time. the usage data was compared with similar libraries without discovery systems, which adds validity to the findings.   the aims are clearly outlined and reiterated with a discussion of the results. the results are presented using seven tables which assist the reader with visualisation of the data. it may have been helpful to include a screenshot of the discovery tool with a description of access points to the library’s collections, since it is unclear from the study what the access points to the library collections are. for example, it is not stated whether the library has links enabled in google scholar, which may increase full-text usage of materials.   the author supplied sufficient detail about data collection to enable other libraries using eds and ebscohost databases to replicate the study. for libraries with other discovery tools in place, modification to the data collection methods would be required.   as more libraries adopt discovery tools to enhance reader discovery of library collections, the research presented here may be useful for other librarians considering adopting a discovery system or evaluating their current system. the research methods are practical in nature and are of particular use to those libraries using eds as their choice of discovery tool. any library with a discovery tool in place may wish to examine its impact upon electronic and print collections and use this research as a reference. collection development policies could also be informed by this research.    reference   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154     evidence based library and information practice 2011, 6.3 1 evidence based library and information practice editorial should we value knowledge and expertise? denise koufogiannakis editor-in-chief collections & acquisitions coordinator, university of alberta libraries edmonton, alberta, canada email: denise.koufogiannakis@ualberta.ca © 2011 koufogiannakis. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. as i write this editorial, i am on an airplane, reflecting on the eblip6 conference, held june 27-30, 2011 in salford, u.k. in my personal opinion, the conference was a great success. there were a wide variety of concurrent paper sessions from an international group of delegates, thought provoking keynotes, and just the right amount of social activity, including the main conference dinner at the manchester united football club! this journal will have a feature section in our next issue (december) that highlights the conference, including keynote presentations, some of the papers that were presented, and commentaries from attendees about the conference itself. so for now, i’ll just offer my warmest congratulations to the organizers. as i left salford and tried to reflect on what i had learned and discussed with others, there were many things that came to mind. immediate things that stood out for me had to do with impact, reflection, and the complexity of decision making. the theme of eblip6 was “valuing knowledge and expertise”. this is a somewhat controversial theme for an evidence based practice conference, where research evidence and its implementation are the focus, and expert opinion is not generally held in high regard. none of the keynote speakers’ presentations spoke directly to the theme, however several paper presentations did include some reference to the importance of professional knowledge. expertise is a loaded word, filled with notions of snobbery and over-confidence, even closemindedness. if anything, those involved with eblip remove themselves as far from the notion of “expert” as they can. but if we consider an expert to be someone who has built up a significant amount of professional knowledge (both through experience and research) on a topic, then the eblip movement should not simply dismiss this notion of “expert”. perhaps, we more appropriately should consider expert mailto:denise.koufogiannakis@ualberta.ca� evidence based library and information practice 2011, 6.3 2 voices (knowledgeable, reasonable, experienced voices) as ones that we need to listen to as another source of evidence within our profession. reflecting on this made me realise that in my own professional approach, i often seek out different experts depending upon the questions i am trying to answer. when i was a newer librarian, my more senior colleagues were the experts i learned from and they gave me guidance at that point in my career. now, when starting new projects, i may consult experts from other institutions to help me by passing on their knowledge of something that is new to me; and i, in turn, do the same for other librarians. that consultation is a starting point, a guidepost, and yes, i certainly value it. without it, where does one even begin? as someone who tries to be evidence based in my practice, i think the key is to not stop with that expert opinion, but to build upon it, test it, ensure that it works in a given context. that process usually involves research evidence and local data, and a quest to build upon previous professional knowledge. the theme of eblip6 was perfect. it came at a time when evidence based practice in lis is looking inward. there were many people presenting similar thoughts – about just how complex decision making is, how there are other forms of evidence besides research, that practice is messy, and that the eblip model is perhaps a bit too simple. we need to think about values, about what we already know, and how to build on that knowledge without simply being dismissive. we also need to look at the model we hold up as an evidence based one, and test it – there needs to be more research on evidence based practice itself. the conference reinforced for me that eblip is at an exciting point where we have moved into a reflective phase that is sure to result in growth and change. the conference taught me that we must value knowledge and expertise, but at the same time, also question knowledge and expertise. we should acknowledge and embrace the complexity of the work we do as lis professionals, and be glad it never fits in a tidy box. each and every one of us uses our own knowledge and expertise to assist with deciding what to do in a situation, and if we are active, conscious professionals, our knowledge continues to grow and our minds remain open to new possibilities. valuing our profession and why we do this work, requires such openness and above all else drives us to make the best decisions we can at a particular point in time. i think of eblip as a movement within lis that strengthens our knowledge and expertise as a profession by reminding our professional community of the importance of both doing and using research. the scientific aspect of our work continually needs to be reinforced and built upon. paired with professional knowledge and expertise, we may have a winning combination! / evidence based library and information practice evidence based library and information practice 2011, 6.4 3 evidence based library and information practice editorial responsibilities 2011. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. editor‐in‐chief: denise koufogiannakis associate editor (articles): alison brettle associate editor (evidence summaries): lorie kloda associate editor (classics): jonathan eldredge associate editor (features, commentaries, using evidence in practice, ebl101, news): denise koufogiannakis production editor: katrine mallan editorial intern: michelle dunaway copyeditors: heather pretty (lead copyeditor), georgianne bordner, marcy brown, molly des jardin, linda ferguson, richard hayman, mary virginia taylor indexing support: pam morgan writing assistance: hope leman, tracy powell iwaskow, carol waseleski / evidence based library and information practice microsoft word comm_brophy.doc evidence based library and information practice 2007, 2:1  149 evidence based library and information practice     commentary    narrative based practice      peter brophy  professor of information management   centre for research in library & information management  manchester metropolitan university  united kingdom  e‐mail: p.brophy@mmu.ac.uk      © 2007 brophy. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.        introduction    the telling of stories lies at the heart of  human communication. if we go back to the  earliest human societies, we find that it is  through story that knowledge, information,  meaning, and wisdom are passed from  generation to generation. take the hebrew  bible, texts of critical importance to judaism,  islam, and christianity, as an example. it  opens with stories: the creation of the world  in six “days” (with the important, if now  neglected, seventh day of rest!) and the story  of the garden of eden, used to provide an  explanation of why there is evil in the  world, setting out a belief that this was not  how things were meant to be. continue  through the narrative and we find story  piled on story, from noah and the flood,  through abraham’s journey to the negev,  the slavery of the chosen people in egypt,  their wanderings in the desert and their  eventual establishment in the promised     land. all of these are stories and all are  stories with a point. they are certainly not  presented for mere entertainment and  neither are they intended as a  straightforward record of historical events.   stories, quite simply, are one way of  depicting reality and of revealing what lies  beneath the surface of events. they are  interested in meaning rather than the  recitation of “facts.” they help us to explore  what is significant. they take full account of  the human dimension. they are concerned  with interpretation.  in everyday life, stories are used to offer a  more graphic description of unfamiliar  scenes than a straightforward factual report  might provide. take this description of  medical practice among the ancient  babylonians:  they have no physicians, but when a  man is ill, they lay him in the public  square, and the passers‐by come up to  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:1  150 him, and if they have ever had his  disease themselves or have known  any one who has suffered from it,  they give him advice, recommending  him to do whatever they found good  in their own case, or in the case  known to them; and no one is  allowed to pass the sick man in  silence without asking him what his  ailment is (herodotus).  the story of how medicine was practised  gives a much better insight into the society  of those times than would a table of  mortality rates – and itself recommends the  use of personal narrative (“whatever they  found good in their own case”) as a means  of communicating advice.  throughout medieval europe the art of  storytelling, in prose and verse, was much  practised. in england we had chaucer’s  canterbury tales and malory’s enduring  fable le morte d’arthur. in wales there was  the mabinogion; in italy boccacio enthralled  his readers with the decameron; in poland  jan kochanowski is credited with writing  the first polish drama, the dismissal of the  greek envoys; and so we could go on. at  this distance in time it is hard for us to  realise just how powerful such narratives  could be.  britain, the world’s first industrial nation,  allowed almost unbelievable poverty to  flourish during the nineteenth century. peter  ackroyd has written, “london was indeed  becoming what the lancet described as a  ‘doomed city.’ the average age of mortality  in the capital was 27, while that for the  working classes was 22, and in 1839 almost  half the funerals in london were of children  under the age of ten” (ackroyd 218). yet  many would argue that it was charles  dickens’ stories of the poor and destitute of  london, in novels like oliver twist and  little dorrit, which did more to change  public opinion than any number of fact‐ laden official reports. of course, those  stories had to be supported by hard,  quantitative evidence if they were to lead to  purposive action. so, alongside dickens, we  had reports such as that by b. seebohm  rowntree who calculated that of those  living in “primary” poverty (defined as  “families whose total earnings are  insufficient to obtain the minimum  necessaries for the maintenance of merely  physical efficiency”), 16 per cent had  suffered the loss by death of the chief wage‐ earner, 22 per cent had families with more  than four children, and 52 per cent simply  did not earn enough to feed and house  themselves properly (119‐145),. it was the  coincidence of story and hard data which  led to action. neither would have been as  effective on its own.  the fascination of stories lies in their  connectedness to our own lives. they appeal  to experience. further, they offer an holistic  analysis – they consider not just the  “simple” fact, but draw in context and  culture, and unashamedly offer a point of  view.  some would distinguish “story” from  “narrative,” where the former is simply one  type of the latter, being characterised by an  appeal to emotional effect (see, for example,  gabriel 5). others would dispute such a  distinction, although recognising that the  telling is as important as the tale. and  linking the two, of course, is the storyteller.  the question for us is whether and how the  power of narrative can be harnessed by and  for professional disciplines in general and  our own profession in particular. evidence  based practice provides a starting‐point for  such a debate.    evidence based practice  it is hardly necessary to review the  development of ebp when writing for  evidence based library and information  practice, although a brief summary is  perhaps in order. it is generally accepted  that this approach was first developed at  mcmaster university in canada in the early  evidence based library and information practice 2007, 2:1  151 1990s, broadening out in the mid‐1990s to  “evidence based healthcare” and then, by  the late 1990s, emerging into other fields  such as social work, education, and human  resource management.  as booth (6), in a comprehensive review of  the literature has remarked, it was natural  for librarians – who were becoming  involved in assessing medical evidence in  their role as information and knowledge  managers – to seek to apply this paradigm  to their own profession. however, booth  also points out that as yet there is no  consensual definition of evidence based  librarianship. for our present purposes,  eldredge’s definition is attractive,  emphasising that the evidence sought comes  from both quantitative and qualitative  sources and is derived from the experience  of work itself:   evidence based librarianship (ebl)  seeks to improve library practice by  utilising the best available evidence in  conjunction with a pragmatic  perspective developed from working  experiences in librarianship. the best  available evidence might be produced  from quantitative or qualitative  research designs, although ebl  encourages more rigorous forms over  less rigorous forms of evidence when  making decisions (72).  booth’s own definition also contains an all‐ important emphasis on the observations and  preferences of clients and users:  evidence based librarianship (ebl) is  an approach to information science  that promotes the collection,  interpretation and integration of  valid, important and applicable user‐ reported, librarian‐observed and  research‐derived evidence. the best  available evidence, moderated by  user needs and preferences, is applied  to improve the quality of professional  judgements (6).  there remain questions over how well the  evidence based practice paradigm fits  professional disciplines like librarianship.  apart from considerations of management  practice, various observers have asked just  where the evidence base is to be found. the  healthcare professions have a well  established infrastructure dedicated to the  collection of data on every possible medical  intervention and a rigorous procedure of  quality assurance, including peer review,  dedicated to ensuring that knowledge about  drugs, treatments, and disease management  is widely shared. librarianship is not like  that, though the efforts of eblip to correct  this should be recognised and applauded.  two specific issues need to be addressed  with some urgency if the ebp paradigm is to  be extended successfully to professional  disciplines beyond those engaged in health  care. the first is the nature of the evidence  that is collected and used. the emphasis on  randomised controlled trials (rcts) in  medicine is less obviously applicable in any  discipline which operates primarily through  its social context – and the emphasis in  librarianship on user‐centred approaches  certainly operates this way. secondly, an  issue which has arisen in medicine becomes  even more pressing in such socially focused  disciplines, namely that the communication  of evidence is at least as important as the  evidence itself. it is worth looking at these  two issues in rather more detail.    the nature of evidence  while it would be wrong to claim that  quantitative evidence is always required by  proponents of ebp, the emphasis placed on  the rct as a “gold standard” is revealing of  a view of evidence based on “scientific,”  positivist paradigms. in this view, the  scientific method, based on the testing of  hypotheses through experimental methods,  dominates. the rct is in a real sense the  purest form of this approach, with its  requirement for double‐blind trials using  evidence based library and information practice 2007, 2:1  152 properly constituted control groups, with  the data gathered being used to disprove  (but, as popper pointed out, never prove)  the original hypothesis. eldredge appears to  accept this hierarchy of evidence when he  refers to a preference for “more rigorous  forms over less rigorous forms of evidence”  and goes on to state that “adherents to both  evidence‐based medicine (ebm) and ebl  consider the rct to be the highest level of  evidence found in a single research study”  (eldredge 72). while there has been some  recognition that rcts are not always the  most appropriate methods in eblip – brice  et al.  suggest that “randomised controlled  trials, although offering the most reliable  results for effectiveness questions, are not  always the most appropriate study designs  for other types of questions” (8) – there is  little evidence that qualitative approaches  are given anything like equal prominence to  quantitative ones in eblip. eldredge simply  suggests that they are useful to “generate  valuable exploratory hypotheses” which can  “[be] subsequently confirmed through  larger, quantitative based research designsʺ  (72).  however, the positivist approach has an  irreconcilable difficulty when applied to  real‐world social systems, namely that it is  literally impossible to control all the  variables. a scientific experiment relies on  the ability of the researcher to carry out the  experiment within a closed system, or as  close to that as is possible to achieve. thus  medical researchers go to enormous lengths  to minimise any bias introduced by external  factors – even in the laboratory this is a  major issue. but social systems, which  include libraries, do not and cannot operate  in such conditions.  in response to this dilemma, researchers  have developed methods based on the  relativist paradigm. as i have written  elsewhere:  [s]uch approaches accept that reality  can only be described through the  eyes and understanding of the  observer – which includes the  researcher. to add to the complexity,  all we have to describe the world is  language, which itself introduces  ambiguity, bias and difference. none  of this denies the major achievements  of scientific method and that  approach’s ongoing importance.  however, when we seek to  understand and describe complex  systems, we should not expect the  positivist tradition to supply all the  answers. (brophy, “measuring  library performance” 21)  although “pure” relativists would reject the  positivist paradigm, the most common  research perspective is generally termed  “post‐positivist.” this approach recognises  that while the goal of achieving “certain”  knowledge through scientific experiment is  flawed, objectivity is possible provided that  it is seen as a shared characteristic of society.  understanding arises as we make  observations and communicate with one  another, including the criticism of each  others’ work, recognising the social contexts  within which we operate and from which  we cannot divorce ourselves. it is also worth  adding that post‐positivism is open to  “critical approaches,” such as the feminist  paradigm, which attempt to counter the  prevailing assumptions which so easily slip  into so‐called objective research. given these  perspectives, any reference to the rct as the  “gold standard” merely provides evidence  of an attitude which sees research and  evidence based practice from a positivist  standpoint. this in turn suggests a view that  organisations are mechanistic systems  which perform well if the right inputs are  provided and the right processes put in  place. the question is whether that is an  adequate stance within a professional  discipline which claims to be focused on  human beings (“users”) and which relies on  evidence based library and information practice 2007, 2:1  153 individual human judgement for so much of  its operational delivery.  within the health disciplines there is  increasing recognition that qualitative  approaches have a major role to play.  indeed, it is interesting to note that research  has been undertaken on the issue of how  well the secondary literature and current  indexing practice supports the retrieval of  such studies. shaw et al. comment that  “better indexing by databases and explicit  qualitative methodology descriptors from  authors are required to make searching for  qualitative evidence more efficient and  effective” (4) , and undoubtedly this is one  of the reasons for under‐utilisation of such  studies.    communication  it sometimes seems to be assumed by  proponents of ebp that provided “hard”  evidence is available and provided that  practitioners are convinced that an evidence  based approach is appropriate, then the  problem has been solved. this makes  unwarranted assumptions about how we  communicate, how we learn and how we  absorb and utilise new knowledge. even in  the health services, this is a serious problem.  if it were not, the drug companies would  not spend such huge sums on ill‐disguised  promotion of their products.     there is a vast literature on communication  in organisations, among which the  overriding emphasis is on communication  as a characteristic of social systems – exactly  the point raised by the relativists. this can  be linked to the emerging consensus among  researchers with an interest in pedagogy  (the theory of teaching and learning) on  constructivist approaches. these approaches  suggest that rather than emphasising the  transmission of “facts” (accepted knowledge  about the world), modern societies need to  encourage learning which encompasses both  openness to differing world views and the  ability to relate new ideas to existing  knowledge in meaningful ways, so that each  of us is continually constructing, sharing,  and reconstructing our understanding of the  world in all its complexity.  to communicate knowledge in a form  which enables it to be used constructively  (in the above sense) therefore becomes a  critical issue. it is not surprising in these  circumstances that the attention of many  management theorists and practitioners is  turning to how communication takes place  in broader society, which soon suggests that  narrative may have a much larger part to  play than has previously been  acknowledged.  it is this twin requirement – to broaden our  understanding of the nature of evidence and  to focus more on how evidence is  communicated effectively – that has led to a  growing interest in the use of narrative in  organisations. a number of management  “gurus” and others have drawn attention to  the power of narrative in decision making.  for example, stephen denning has written:  steadily increasing recognition of the  importance of narrative in  mainstream management is now  inevitable. . . . narrative thinking is  contributing to an emerging view of  organizations that more accurately  reflects not only the traditional  structural, process‐oriented, control‐ based aspects of an organization but  also the living, flowing aspects of  organizations – where talking,  thinking, dreaming, feeling human  beings work and play and talk and  laugh and cry with each other, in a  way that is organic and self‐adjusting  and naturally innovative. (denning in  brown et al. 176)    ethnography and the place of stories in  research  evidence based library and information practice 2007, 2:1  154 stories have always been important to the  practice of ethnographic research, which  derives principally from anthropology and  the study of the relationships between  people and investigates the cultural, social,  personal, physical and other aspects of their  lives. sarantakos suggests that ethnographic  research is based on four principles which  have informed anthropological studies:    culture. the focus of concern is on the  shared culture of groups of people,  examining and seeking to understand the  shared patterns of behaviour, values, norms  and standards.    holism. the focus is on the whole system  being observed, rather than on an isolated  part. meaning and purpose can only be  discerned within the context of the whole  system.    in‐depth studies. ethnography is not  interested in the superficial data collected by  counting or even by questioning, but relies  on “living in” the group that is being  investigated.    chronology. originally, anthropologists  concentrated on primitive cultures, although  ethnographic approaches are now used by a  wide range of social scientists to study both  primitive and modern societies over time.  ethnography is interested in change, in how  people and societies alter and reinvent  themselves over time. (196‐7)   ethnographic research is sometimes  criticised as being too subjective – because  the researcher is immersed in the culture  being studied, he or she cannot stand back  and be fully objective. hannabuss (“being  there” 100) comments that this criticism  seems to be particularly sharp where “the  evidence takes the form of narratives or  stories, possibly communicated  spontaneously by the respondents.” again,  one suspects that eldredge, whose definition  was quoted earlier, might be hostile to story  as a “less rigorous form of evidence,” as  might many of the published proponents of  ebp.  it is easy to see why this criticism is levelled,  but the danger is that if it is allowed to  prevail, a great richness of knowledge and  understanding will be lost. linde (161)  provides a taxonomy of knowledge in which  she points out that “tacit knowledge” needs  to be considered alongside “explicit  knowledge,” the latter so often being the  “stuff” in which knowledge management  and similar approaches are interested. she  observes that tacit knowledge “is commonly  and easily conveyed by narrative.” she  continues, “knowledge about identity, who  one is and what one’s history has been, is a  very important part of tacit knowledge.”     social constructivism  we can link the ethnographic approach to a  number of significant aspects of postmodern  thought and particularly to the idea of social  constructivism. gergen wrote  authoritatively about this concept, pointing  out that our discourse about the world – the  ways in which we think and speak about it,  and the ways we apprehend “reality” – is  the product of communal interchange.  because human beings continually interpret  and reinterpret our understanding of the  world in a social setting, that understanding  is not, and cannot be, a fixed map of “how  things are,” as an objectivist stance would  claim.    it is within this notion of social  constructivism that the importance of story  or narrative becomes clear. stories enable us  to communicate within groups – social,  family, work or whatever – because they  provide context and interpretation. bates  (20) suggests that in analysing narratives we  can identify:  evidence based library and information practice 2007, 2:1  155 • sequential and temporal structures,  which help us to understand the  sequencing of actions which make  up the story. traditionally, for  example, a story might start off:  “once upon a time . . . ,” thus  marking its beginning. it is  important to note that this reflects  the generally teleological structure  of stories – they are about means  and ends.  • a specific focus or orientation of the  narrative, which is an expression of  the views and perceptions of the  storyteller and how they  understand the situation they are  describing    • the way in which the story is told,  including the use of language, tone  of voice, expressiveness and so on.  hannabuss comments on the use of  narrative in organisations and the way in  which members of the organisation use the  narrative in order to deduce more general  lessons:  one of the most frequent uses of  narrative in organisations [is to  generalise from them]. they are  unique and idiosyncratic, but they are  used as a key part of sensemaking  because they tell us something  important about the circumstances  now, and they can also be used as  pointers or lessons for the future (e.g.  if the company were to do this, then  similar or comparable outcomes  would occur). they can also be  generalised from and add to the store  of exemplars of successful and  unsuccessful management practice  which becomes the folklore of  management itself. (“narrative  knowledge,” 412)  in a separate paper, hannabuss also  comments that “narrative allows for . . .  reflexivity” (“telling tales” 222); that is, it  encourages reflection on the outcomes of  decisions, and so is a highly suitable vehicle  for the reflective practice which is so widely  regarded as essential to professionalism (see  schön). “knowledge inherent in practice is  to be understood as artful doing”  (hannabuss, “telling tales” 222). thus  narrative leads to action.  it is also worth adding here that the nature  of constructivist understandings of learning  – and learning is at the heart of management  and decision making – is one that is highly  conducive to the sharing of experience by  means of story. we learn from one another  by telling the stories of our experiences and  listening to the experiences of others.  constructivism places considerable  stress on the sharing of perspectives –  in other words learners participate in  the process by revealing their  interpretations to others and  receiving feedback and new insights.  the aim of this activity is not  primarily to learn new “things,” to  acquire “factoids” to slot into a  mental filing cabinet, but to develop  ways of interpreting the world which  are flexible enough to be applied  successfully in new situations.  (brophy, “networked learning” 136)  the argument advanced here is that in order  to improve our decision‐making we need to  go beyond the “hard” sources that are  normally the focus of demands for  “evidence” to encompass the much “softer”  and admittedly less objective stories which  people – including professionals – tell each  other. it is possible to discern this movement  in the very same health professions from  which evidence based practice first sprang.  in 1998 a collection of essays was published  under the title narrative based medicine  (greenhalgh and hurwitz, 4‐5). they write:  evidence based library and information practice 2007, 2:1  156 you could make an objective list of  the actions you performed over the  last week, but if it were simply a  “factual” account, it would not mean  anything. but if you told us what you  had done in the last week, not only  would your story acquire meaning,  but in telling it, both you the narrator  and we the listeners would be  compelled to reflect on it in order to  gain a greater understanding of what  had gone on. . . . just as history does  not exist in nature, but is created in  the telling, so, too autobiography and  the medical case history emerge out  of transactions which mean they are  at the same time both less and more  than the “facts” of the case. [original  emphasis]  the same is true of many other professions,  and can be true of librarianship. we are  more likely to find meaning in the telling of  how things have been experienced by others  than in the formality of arid statistics and  measures. or, at the very least, meaning is  more likely to be discerned when the two  are used in harmony.    narrative based practice  what does all this mean for the professional  practice, including the practice of  librarianship? well, perhaps it suggests that  in order to improve our management  practices and the delivery of services to our  clients, we need to take much more  seriously the role of narrative and find ways  to capture narrative systematically as part of  our evidence base. we need to acknowledge  explicitly that, in fact, good managers have  always relied on story – the anecdote shared  in the corridor or the “war stories” swapped  over a drink in the conference bar. but  sharing anecdotes in a haphazard and  random fashion is by no means an adequate  response to the challenge of professional  practice in the modern world. rather we  need to develop new ways of capturing,  sharing and using narrative as a systematic  part of service delivery and management.  we need to develop narrative based practice  (nbp).   using narrative is not an easy option. very  few people are natural storytellers and the  art, for surely that is what it is, needs to be  learned and practised. educators need to  include both the construction and the use of  narrative alongside other research  techniques, giving due prominence to issues  such as what makes an effective narrative,  how narratives should be selected and  different ways of presenting them to  different audiences. continuing professional  development (cpd) needs to offer courses  in the use of narrative for practising  professionals. librarians and other  professionals need to learn the art not just of  telling but of listening to story.  there are examples of the use of narrative in  the literature of librarianship. the uk’s  people’s network, for example, has  demonstrated its impact in part through the  use of story. alongside the statistics of use  and the analysis of demographic  characteristics of users, there are accounts  like the following:  i obtained a place at college on their  basic ecdl (european computer  driving licence) course and used the  library to practise some of the things i  learned at the college. if it hadn’t been  for the initial use of the library  computer i don’t think i would have  considered the college course. . . . i  now have a new job in which i need  computer skills. so from playing  about with the library computer i  now have a successful career, all  because computers were installed in  local libraries. (brophy, “the people’s  network,” 8‐9)  stories can be powerful advocacy tools for  libraries as well as tools for their  evidence based library and information practice 2007, 2:1  157 management. the same is true in many  other professions.     conclusions  this paper has suggested that narratives can  be a powerful addition to the evidence base  upon which professionals rely when making  decisions, that they can illuminate complex  situations, can be effective tools for  advocacy, and can form the basis for more  meaningful communication within and  between organisations. none of this should  be taken to suggest that the ebp approach is  ill‐advised, nor that the idea of basing  decision‐making on good evidence is  flawed. but a paradigm which is  enthusiastic to go beyond positivist  perspectives and is hospitable to narrative  based practice offers opportunities for  professionals to improve their decision  making. it also challenges them to approach  professional practice in new ways,  exploiting the riches that qualitative sources  of evidence and narrative forms of  communication have to offer.    works cited  ackroyd, p. dickens. abridged ed. london:  vintage, 2002.  bates, j.a. “use of narrative interviewing in  everyday information behavior  research.” library and information  science research 26.1 (2004): 15‐28.    booth, a. “bridging the research‐practice  gap? the role of evidence based  librarianship.” the new review of  information and library research 9.1  (2003): 3‐23.    brice, a., a. booth and n. bexon. “evidence  based librarianship: a case study in  the social sciences.” world library and  information congress: 71st ifla  general conference and council. oslo,  norway. august 2005. 12 february  2007  <http://www.ifla.org/iv/ifla71/papers/1 11e‐brice_booth_bexon.pdf>.    brophy, p. “networked learning.” journal  of documentation 57.1 (2001): 130‐156.    brophy, p. the people’s network: a turning  point for public libraries: first findings.  london: resource, 2003.    brophy, p. measuring library performance.   london: facet, 2006.    brown, j. s. et al. storytelling in  organizations: why storytelling is  transforming 21st century  organizations and management.  oxford: elsevier butterworth‐ heinemann, 2005.    denning, s. ʺstephen denning: some  thoughts in 2004ʺ. brown, j. s. et al.  storytelling in organizations: why  storytelling is transforming 21st  century organizations and  management. oxford: elsevier  butterworth‐heinemann, 2005. 165‐179.    eldredge, j. d. “evidence‐based  librarianship: what might we expect  in the years ahead?” health  information and libraries journal, 19.2  (2002): 71‐77.    gabriel, y. storytelling in organizations:  facts, fictions and fantasies. oxford:  oxford u.p., 2000.    gergen, k.j. towards transformation in  social knowledge. 2nd ed. london:  sage, 1994.    greenhalgh, t. and hurwitz, b. narrative  based medicine: dialogue and  discourse in clinical practice. london:  bmj books, 1998.    http://www.ifla.org/iv/ifla71/papers/1 evidence based library and information practice 2007, 2:1  158 hannabuss, s. “being there: ethnographic  research and autobiography.” library  management 21.2 (2000): 99‐106.    hannabuss, s. “narrative knowledge:  eliciting organisational knowledge  from storytelling.” aslib proceedings  52.10 (2000): 402‐13.    herodotus. the history of the persian wars.  trans. p. halsall. (430 bce). 12  february 2007  <http://www.fordham.edu/halsall/anci ent/greek‐babylon.html>.    linde, c. “narrative and social tacit  knowledge.” journal of knowledge  management 5.2 (2001): 160‐170.  rowntree, b.s. poverty: a study of town   life. london: macmillan, 1901.  sarantakos, s. social research. 2nd ed.   london: macmillan, 1998.    schön, d. the reflective practitioner. how  professionals think in action. london:  temple smith, 1983.    shaw, r.l. et al. “finding qualitative  research: an evaluation of search  strategies.” bmc medical research  methodology 4/5 (2004). 12 february  2007 <http://hdl.handle.net/2381/59>              http://www.fordham.edu/halsall/anci http://hdl.handle.net/2381/59 research article   teens’ vision of an ideal library space: insights from a small rural public library in the united states   xiaofeng li assistant professor department of library and information science pennsylvania western university clarion, pennsylvania, united states of america email: xli@pennwest.edu   yoojin ha professor department of library and information science pennsylvania western university clarion, pennsylvania, united states of america email: yha@pennwest.edu   simon aristeguieta assistant professor department of library and information science pennsylvania western university clarion, pennsylvania, united states of america email: saristeguiet@pennwest.edu   received: 31 july 2023                                                               accepted: 10 oct. 2023      2023 li, ha, and aristeguieta. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30410     abstract   objective – this study delves into the perspectives of teenagers regarding their desired teen space within a small rural public library in the united states.   methods – to capture the richness of their thoughts, a visual data collection method was employed, wherein 27 8th-grade participants engaged in a drawing activity during an art class at a local middle school. two additional teens were recruited for individual semi-structured interviews.   results – through this creative exercise, the study unveiled the various library activities, amenities, books, and visual designs that resonated with the teens, as they envisioned their ideal teen space.   conclusion – the study’s findings hold practical implications for librarians working with this population, offering valuable insights to enhance and optimize teen services at the library. by aligning the library’s offerings with the desires of the young patrons, the potential for a thriving and engaging teen community within the library is enhanced.     introduction   public libraries have long been essential providers of youth services, offering not only access to information and fostering multiple literacies, but also cultivating vital 21st-century competencies among young individuals (abbas & koh, 2015). while public libraries are prevalent across the united states, they exhibit notable variations in capacity and resources. rural communities, in particular, confront distinct challenges, including poverty, digital divides, and resource limitations (meyer, 2018; perryman & jeng, 2020; real et al., 2014). consequently, small rural libraries often grapple with reduced funding, limited collections, staffing, space, services, and programs.   given these constraints, it becomes crucial for small rural public libraries to identify the unique needs of their community members, enabling them to provide services efficiently and effectively. unfortunately, there has been a dearth of attention directed toward understanding the youth in rural areas and their utilization, or lack thereof, of public libraries. in light of this, in the present study, the researchers endeavored to explore the perspectives of teens regarding their aspirations for public libraries. by understanding what teens desire to see in these institutions, the researchers aimed to provide valuable guidance to practitioners and researchers seeking to build a resilient future for youth services in libraries, particularly in small rural settings. through such understanding, libraries can adapt and thrive amidst the challenges, fostering an environment that caters to the evolving needs of the young generation.   literature review   teens and public libraries   public libraries have a long history of serving teens, dating back to the 1800s (bernier, 2020). the young adult library services association (yalsa), a division of the american library association (ala), was established in the 1960s to provide resources and support for librarians who work with teens. public libraries serve as key players in advancing teens’ educational and well-being interests, as evidenced in research showing libraries’ support for contemporary youths’ connected learning experiences (subramaniam et al., 2018), promotion of digital and data literacy (bowler et al., 2019; kelly et al., 2023), and librarians addressing the health information needs of teens (knapp et al., 2023; powell et al., 2023).   research in library and information science (lis) has shed light on the multifaceted ways in which teenagers utilize public libraries, especially for the social opportunities afforded by public libraries. surveys conducted in both urban and suburban areas of the united states have revealed that teens frequently turned to public libraries as invaluable hubs for accessing necessary information resources, fostering social connections within a safe space, and having a positive environment for other personal activities (agosto, 2007). teens in a semi-rural area in the united states reported that they use a public library makerspace to tinker, learn, socialize, and pursue their personal interests (li & todd, 2019). furthermore, studies conducted in rural areas in canada have demonstrated that teens visited public libraries for attending programs, hanging out, and participating in collaborative learning opportunities (kelly et al., 2023; reid & howard, 2016).   to enhance the services for teens, researchers have delved into the various features that teens desire in public library spaces. in studying 25 newly constructed and renovated public libraries in the united states, agosto et al. (2015) highlighted the importance of providing comfortable and inviting physical spaces, meeting teens’ information needs, and offering many opportunities for both leisure and academic activities. similarly, cook et al. (2005) found that factors such as hosting teen-only events, providing food options, and offering general amenities were positively correlated with urban teens’ positive perceptions of libraries. teen librarians also shared a set of practices that contributed to a positive teen library experience, including building a welcoming teen space that fosters ownership and social interactions, treating teens with respect, and focusing on teens’ interests (ornstein & reid, 2022).   while many studies have reported positive library experiences among teens, research also shows conflicting results. for instance, howard (2011) showed that while teens were generally satisfied with their local public libraries, the focus group discussions among teens revealed some dissatisfaction. howard argued that the status quo in teen spaces might meet what teens considered normal, but it may not be ideal for them. multiple factors contributed to teen dissatisfaction with libraries. abbas et al. (2008) surveyed over 4,000 teens in western new york state, revealing that one of the major contributing factors to library non-use was the lack of convenience. outdated or irrelevant technology can be off-putting for teens in urban areas (agosto et al., 2016; meyers, 1999). library staff who are distant, strict, or impatient with teens can negatively impact their impressions of the library and deter them from using it (agosto & hughes-hassell, 2010; howard, 2011). in addition, library collections that do not address issues relevant to specific teen demographics and cultural identities can make teens feel disconnected from their libraries (agosto & hughes-hassell, 2010; meyers, 1999). teens and tweens who are black, indigenous, and people of color chose not to use public libraries because of perceived risks of not meeting institutional policies, rules, and behavioural expectations (gibson et al., 2023).   the current literature further identifies a significant contributing factor to teens’ non-use of libraries – library spaces. meyers (1999) found that teens perceived library spaces as “dull”, morgue-like, “boring”, and not designed for teens’ needs. two decades later, research findings concerning teens’ perceptions of library spaces remain consistent. inadequate library spaces and equipment for teens continue to negatively affect teens’ library use (howard, 2011). bishop and bauer (2002) found that young adults in both urban and rural areas considered an attractive teen library space as the most important factor in their library use. cook et al. (2005) found that early teens viewed inviting and teen-only areas as positive indicators of libraries. this result was further supported by bernier et al.’s (2014) survey of 411 libraries, which found a positive correlation between the amount of library space dedicated to young adults and the level of library participation from teens. bernier (2010) emphasized the importance of youth input and participation in space design to meet their aesthetic needs. therefore, it is crucial to consider teens’ perspectives when designing library spaces to enhance their library use.   rural public libraries   public libraries are important institutions in rural communities, playing crucial roles beyond book repositories. rural libraries are trusted resource providers for various information needs, assist patrons in finding print and digital resources, and serve as community centers where people gather and meet others (grove & brasher, 2020). public libraries in rural areas also have the capacity to contribute to local economic growth by supporting job skills training and small businesses development (hughes & boss, 2021; mehra et al., 2017; real & rose, 2017). additionally, public libraries may be the only institutions in rural areas that provide free access to computers and the internet, and support for technology skills (real & rose, 2017). rural libraries offer health and wellness programs that make positive impacts on rural residents (flaherty & miller, 2016; lenstra et al., 2022).   however, research has shown that rural libraries in the united states face many challenges. staffing and funding are among the most often reported challenges. fischer (2015) reported staffing challenges and limited funding in rural and small libraries, including a lack of librarians with master’s degrees, even though the survey findings showed that there has been some improvement in conditions in these rural and small libraries. access to technologies and the internet is limited in rural communities. according to real and rose (2017), rural libraries offered the fewest public access computers overall, and internet speed was often inadequate to meet the needs of patrons. situated in this challenging environment, services to children and teens suffer tremendously. real and rose noted that rural libraries tended to have fewer programs related to science, technology, engineering, arts, and mathematics (steam), and fewer formal after-school programs, like homework help, compared to their urban counterparts.   to tackle the challenges encountered by rural libraries, researchers underscore the importance of community engagement and the implementation of innovative outreach strategies. reid and howard (2016) highlighted the deployment of mobile library services, book delivery services, and the establishment of satellite library branches in serving community members in rural areas. additionally, kelly et al. (2023) demonstrated that a partnership between rural libraries and local community organizations provided local youth access to laptops, enabling their participation in coding clubs hosted by the libraries. despite the longstanding tradition of public libraries serving teenagers, research focused on teen services is clearly limited, with an even greater scarcity of studies centered on rural teen populations. to address the identified research gap in the current literature in lis, this study aims to explore the following research question:   rq: how would teens design their desired teen space in a small rural public library?   methods   the use of surveys, focus groups, and interviews has been common in lis research to explore the opinions and preferences of teenagers regarding libraries. however, visual participatory research, which involves gathering visual data like photographs and drawings created by the participants, has been an underutilized method (weber, 2008). in lis, there has been an increasing interest in employing photography as a visual method to enhance qualitative data collection (e.g., agosto & hughes-hassell, 2005; barriage, 2021; li & todd, 2019). however, the application of drawings as a data collection method has received limited attention, with hartel’s (2014) research being an exception, wherein college students’ drawings were used to illustrate their conceptualizations of information. the use of visual data collection methods has proven to be a valuable tool in research as it enables participants to engage more deeply with research questions (gauntlett, 2005). in particular, young people tend to be more attentive and involved when visual activities are included (hartel, 2014; subramaniam, 2016). according to weber (2008), images encourage participants to consider research questions from diverse perspectives, beyond what is possible with writing or speech. researchers can gain a more comprehensive understanding of participants' viewpoints through visual data (weber, 2008; woodgate et al., 2017). in this study, we aimed to broaden previous research by employing this method to gain insight into the library images that young people aspire to have in their communities.   data collection   data were collected in a small rural area in the united states. the researchers collaborated with the principal and art teacher at a local middle school to gather data. consent forms were distributed to each homeroom in the middle school. in all, 27 8th graders (ages 13 to 14) were recruited to participate in a drawing activity during art class, during which the researchers were not present. participants were asked to design their ideal public library space for teens and write responses to three prompts, which were to 1) describe the library as it was, 2) describe what you wished the public library to be, and 3) describe their drawings. these written responses were requested to help us understand their perceptions of the public library and what they wanted in their public library. all the drawings were submitted anonymously. furthermore, to complement the visual and written data collected from the drawing activity, two additional teens (ages 13 to 14) were recruited through convenience sampling, which involved inviting individuals that the researchers had known, to participate in individual semi-structured interviews. these interviews allowed researchers to ask follow-up questions, providing further clarification and validation of emerging themes derived from the drawing activities. each interview was conducted virtually and lasted approximately 30 minutes. see appendix for the interview questions. this study was approved by the university’s institutional review board.   data analysis   to analyze the data collected from the drawings, written responses, and interview transcripts, we imported them into dedoose, a qualitative data analysis software. we used the constant comparison technique in the initial rounds of open coding and axial coding (charmaz, 2006). initially, we independently coded the same set of four participants’ design activity sheets to examine the differences and similarities of our interpretations of the data. we discussed each code together and came up with 50 initial codes inductively (e.g., feeling irrelevant, not having many books, freedom of doing whatever they want, hanging out, gaming, a colorful place, etc.). these 50 codes were further grouped into 7 categories, including social interactions, being kids/freedom to express themselves, relevance, visual appeals, books/information, learning, and emotions. then we used another set of six participants’ data, which were randomly selected from the data, to test the shared understanding of these emergent themes. using the dedoose training center, the code application test results showed that the inter-coder reliability arrived at an overall cohen’s kappa value of 0.46, indicating a fair agreement (fleiss, 1971). codes with a cohen’s kappa value less than 0.65 were discussed to reach an agreement. during the discussion of different interpretations of the excerpts and codes, if a code was updated, the researchers went through all the data to compare them to the newly modified code. when a new code was generated, comparisons between codes were also made. through meetings and discussions, the researchers identified a total of four categories – activities, amenities, books, and visual design – that were collectively agreed upon to answer the research question explored in this study. when comparing the generated codes from drawings and written responses to those from the interviews, no discernible differences in results were observed between these two data collection methods. all names reported here are pseudonyms that were assigned by the researchers. all the quotations are verbatim.   findings   overall, the data analysis revealed predominantly negative perceptions of the local public library among teens. for instance, jordan succinctly stated, “not fun,” while taylor commented on its “gloomy and boreing [sic]” atmosphere. when prompted to share their ideas on redesigning the library to appeal to individuals like themselves, their suggestions were categorized into four themes: activities, amenities, books, and visual design. each of these primary themes is described below, accompanied by illustrative examples.   desired library activities   the data analysis highlighted a major concern among teens regarding the availability of engaging activities at the library. when the library failed to offer compelling programs, teens perceived the library as irrelevant to their interests and needs. for instance, one participant, avery, described libraries as “a big buiding [sic] with really nothing to do for kids.” on the other hand, participants expressed their desires for a library that catered to their preferences, indicating a need for a space that allowed for free-choice fun activities, learning opportunities, and social interactions. for example, morgan noted, “my drawing is a trendy library for teens 13-19 to socialize, study, and have fun.” see figure 1 below. similarly, alex shared: “a fun place where kids can go to hang have fun and learn.”   figure 1 design for teen space (by morgan).   twelve teens emphasized the importance of having a dedicated study area in their ideal library, expressing a desire for study rooms with tutors and spaces suitable for schoolwork and group activities. quinn, for instance, wrote: “i also thought a study room would be nice for doing schoolwork/group activities.”   in their designs, participants emphasized the importance of offering spaces for creative expression and entertainment. the appeal of video games was evident among seven teens, who expressed a desire for a dedicated game room in the library. cameron, for example, wrote: “also a lot of kids my age like playing video games so i thought a game room would attract a lot of people.” charlie in the interview explained: “i think people would probably mostly play games. maybe someone plays educational games… that’ll be fun, like people can learn something and they can play games at the same time.” intriguingly, two teens included music rooms in their ideal library design. cameron highlighted the popularity of music among young people, stating: “i know a lot of people (including myself) that enjoy listening/making music, so i thought a music room would be nice to have.” rowan expressed a specific desire for “rated-r music.”   social interactions were also crucial for eight teens who participated in the drawing activity and two interviewees. finley expressed a wish for the library to be “1 cool place where i can go hang out with friends,” while skylar indicated that the library could serve as a place to “make new friends.” in addition to having fun with friends, the teens also sought support from their peers, as blake wrote: “somewhere for teens can go to get support from other teens.” this desire for a space to hang out and socialize was shown in their designs where they drew curved couches at the center of the teen space, for example, the design from dakota in figure 2 below.   figure 2 design for teen space (by dakota).   desired library amenities   data analysis revealed that teens expressed a strong desire for convenient and appealing amenities in the library. these amenities encompassed a range of facilities, including vending machines, food services, arts and craft tables, ping-pong tables, large tvs, and comfortable furniture. the teens’ input offered valuable insights into their vision of the ideal library space, where they could enjoy various activities and feel comfortable spending time.   five participants specifically mentioned the inclusion of food services, such as a “snack bar” or a “café,” in their designs for the ideal library. for instance, emerson creatively envisioned a “secret door” leading to a pantry filled with an abundant variety of snacks from around the world, highlighting the appeal of convenient food options within the library setting. the participants’ feedback underscores the importance of offering diverse amenities that cater to their interests and needs. as exemplified below in figure 3, the amenities featured a café in the top left corner, a flat screen tv and bean bags in the middle right-hand section, and lockers for instruments in the bottom right corner of the dedicated teen space.   figure 3 design for teen space (by cameron).   desired books   teens’ perceptions of the books housed in the library were diverse, with some expressing satisfaction with the library’s book collection, while others voiced concerns about the relevance and variety of books available. the majority of teens described libraries as having an adequate number of books, with tom mentioning, “there are a lot of bookshelves everywhere.” however, one teen pointed out the limitations, stating, “the teen space is very small...there aren’t many books in it.” sage further elaborated that the library should strive to be “a fun place with better books.” this desire for “better books” might explain the conflicting perceptions, as teens recognized that libraries contained books, but these books did not always align with their interests and preferences. in the interview, charlie explained: “people still like non-fiction books more...so informational books might be good.” hayden expressed an interest in books related to debate and cooking skills in their ideal library design (see figure 4 below), highlighting the importance of relevant and engaging content.   figure 4 design for teen space (by hayden).   it also became apparent that teens’ interests in reading influenced their perception of libraries. quinn, for instance, stated: “i thought the library was only a place for reading and at the time i didn’t like to read at all.” this showed how a lack of interest in reading during a particular period led them to perceive the library solely as a place for reading, reflecting a limited perspective at the time.   desired visual design   five teens expressed the view that libraries were perceived as outdated. samuel succinctly described libraries as “bland” and “old,” while sage shared a similar sentiment, stating, “in my opinion, it’s kinda vintage and bland.” conversely, when prompted to envision their ideal libraries, six teens expressed a strong desire for libraries to be “trendy” and modern. blake mentioned the inclusion of “nice big cool lighting,” while quinn specifically highlighted a wish for a “colorful carpet.” similarly, morgan wrote “libraries could be boring to the eyes; you should also add some color so it can pop eyes.” see figure 5 below for an example design that highlighted the desire for colors in the teen space.   figure 5 design for teen space (by alan).   discussion   the present study aimed to understand teenagers’ views on designing the teen space in a small rural public library in the united states. the desired activities reported by the teens in this study emphasized the importance of creating a library space that allows them the freedom to engage in activities that interest them, such as studying, socializing with friends, playing video games, and enjoying music. teens’ insights suggested that the availability of books is only one aspect influencing teens’ perceptions of libraries. the content of these books and how well they cater to the diverse interests and passions of young patrons were equally crucial. the sentiments expressed among teen participants emphasized the importance of creating visually appealing and dynamic library spaces for teens. by incorporating engaging activities, well-thought-out amenities, diverse book collections, and visually appealing designs, libraries have the potential to attract and resonate with the younger generation. particularly in rural libraries that commonly face challenges such as limited staff and budgeting, knowing teens’ preferences becomes critical for libraries to prioritize their budget and services to create an inclusive and appealing environment that meets the needs and desires of their teen patrons.   overall, the findings of this study showed that teens in rural areas shared many commonalities with teens in urban and suburban areas regarding their desired library space and library uses (agosto, 2007). teens’ desire for an attractive teen space reported in this study aligns with the insights presented by bishop and bauer (2002). additionally, the findings underscored the importance of providing a more relevant collection for teens, echoing previous research that highlighted the disconnect between teens and library collections that fail to represent their interests and identities (agosto & hughes-hassell, 2010; meyers, 1999).   while this study confirmed many findings from previous research, the researchers noted that teens in this present study did not strongly express their desires for libraries to meet their information needs. this finding is different from previous research in which teens used libraries for information resources (agosto, 2007; agosto et al., 2015). one possible explanation for this difference could be the limited availability of public transportation options for teens in rural areas, making it challenging for them to physically visit libraries, especially when considering the impractical walking distances involved. another possible explanation could be rural teens might have other readily accessible information sources such as online websites, local community centers, and school libraries. further research is needed to understand rural teens’ information practices in their everyday life, so that rural libraries can provide services that cater to the needs of rural communities.   the findings of this study have practical implications for youth services librarians to consider when developing teen spaces and engaging with rural teens. given the common challenges of limited resources faced by small and rural libraries, it’s crucial for librarians to understand the interests and needs of their local teens, so that tailored programs and services for teens can be provided. forming a teen advisory council where teens have a platform to share their interests and needs can be helpful, and also encourages teens to take on leadership roles in the ownership of their libraries. low-cost assessment methods, such as informal interviews and drawing activities as utilized in this present study, can also help librarians understand teens’ preferences of library spaces (e.g., preferences of snack areas and aesthetic design elements), desired activities (e.g., studying, socializing, gaming, and relaxation), and reading preferences. additionally, recognizing the limited public transportation options in rural communities, libraries can implement bookmobiles and outreach programs to bring library resources directly to rural teens and leverage social media sites and virtual programs to connect with teens. lastly, as suggested in previous research (kelly et al., 2023; reid & howard, 2016), rural libraries should build partnerships with local schools, homeschoolers, and local community organizations to promote library programs and services for teens, and seek additional resources, funding, and support for teen programs and services.   limitations   the researchers acknowledge several limitations that may impact the scope and depth of the findings. first, it is important to note that the overall number of drawings collected from a single grade within a rural school was small and limited, which may restrict the transferability of the findings to other age groups and geographic areas. future research should consider a more diverse sample by encompassing young people from different age groups and various rural areas.   second, while the two interview participants were asked about their use of the local library and possession of a library card, these questions were not presented to the participants in the drawing activity. future research could investigate whether existing library usage is related to the themes identified in this study’s findings.   third, the study relied mainly on data collected from drawings and accompanying texts, which may offer valuable information but may lack the richness of in-depth interviews. visual data, as noted by literat (2013), can be highly interpretable, and understanding the context surrounding the creation of the drawings becomes crucial. gauntlett (2005) highlighted the importance of participants interpreting and sharing the meanings of their drawings to avoid overinterpretation or incorrect interpretation by researchers. future research could address this limitation by engaging in more comprehensive interviews to encourage participants to elaborate on their designs and provide deeper insights into their perspectives. for instance, understanding what teens specifically mean by “better collection” would provide a more nuanced understanding of their desires for library resources.   last, the study did not clearly identify which of the four identified categories mattered the most to the teens. while it is evident that teens desired a variety of amenities, activities, visually appealing elements, and relevant collections, the study did not prioritize or rank these preferences. future research could invite teens to rank these categories according to their importance, which would aid librarians in efficiently prioritizing their efforts when developing teen spaces. given the potential challenges of limited budgets in small rural libraries, such rankings would offer valuable guidance for decision making and resource allocation.   conclusion   in this exploratory study, we aimed to uncover the preferences of teens in a rural community regarding their desired features for the teen space in their local public library. by engaging the teens in designing their ideal library space, the study identified four major categories that hold significance for librarians when developing teen spaces. these categories encompassed activities, amenities, books, and visual designs, offering valuable insights into teens’ needs and expectations.   the study revealed that teens expressed a strong desire for a versatile space that can accommodate various activities, including studying, socializing, gaming, and relaxation. they sought amenities like food and comfortable furniture to enhance their overall experience. teens also yearned for library collections that better align with their interests and needs, reflecting a desire for resources that resonate with their preferences. teens emphasized the importance of a bright and visually appealing space within the public library. this aesthetic consideration is seen as pivotal in creating an inviting and attractive environment for teen patrons.   youth services librarians can utilize these findings to guide the development of teen spaces that effectively cater to the preferences of their young patrons. by incorporating the desired activities, amenities, books, and visually appealing elements, librarians can create an engaging and relevant library environment that speaks directly to the needs of teens. ultimately, the researchers aimed to ensure that teens find their local public library appealing, inclusive, and meaningful, a space that enhances their connection with the library and fosters a sense of belonging within the community.   funding   this work was made possible through the support of pennsylvania western university (formerly clarion university of pennsylvania) under the university community fellow program grant.   author contributions   xiaofeng li: conceptualization, formal analysis (equal), funding acquisition (equal), methodology (equal), writing – original draft, writing – review & editing (lead) yoojin ha: data curation, formal analysis (equal), funding acquisition (equal), methodology (equal), writing – review & editing simon aristeguieta: formal analysis (equal), funding acquisition (equal), methodology (equal), writing – review & editing   references   abbas, j., kimball, m., bishop, k., & d’elia, g. (2008). why youth do not use the public library. public libraries, 47(1), 80–86.   abbas, j., & koh, k. (2015). future of library and museum services supporting teen learning: perceptions of professionals in learning labs and makerspaces. the journal of research on libraries and young adults, 6, 1–24.   agosto, d. e. (2007). why do teens use libraries? results of a public library use survey. public libraries, 46(3), 55–62.   agosto, d. e., bell, j. p., bernier, a., & kuhlmann, m. (2015). “this is our library, and it’s a pretty cool place”: a user-centered study of public library ya spaces. public library quarterly, 34(1), 23–43. https://doi.org/10.1080/01616846.2015.1000777   agosto, d. e., & hughes-hassell, s. (2005). people, places, and questions: an investigation of the everyday life information-seeking behaviors of urban young adults. library & information science research, 27(2), 141–163. https://doi.org/10.1016/j.lisr.2005.01.002   agosto, d. e., & hughes-hassell, s. (2010). revamping library services to meet urban teens’ everyday life information needs and preferences. in d. e. agosto & s. hughes-hassell (eds.), urban teens in the library: research and practice (pp. 23–40). american library association.   agosto, d. e., magee, r. m., dickard, m., & forte, a. (2016). teens, technology, and libraries: an uncertain relationship. the library quarterly, 86(3), 248–269. https://doi.org/10.1086/686673   barriage, s. (2021). examining young children’s information practices and experiences: a child-centered methodological approach. library & information science research, 43(3), 101106. https://doi.org/10.1016/j.lisr.2021.101106   bernier, a. (2010). spacing out with young adults translating ya space concepts into practice. in d. e. agosto & s. hughes-hassell (eds.), urban teens in the library: research and practice (pp. 113–126). american library association.   bernier, a. (ed.). (2020). transforming young adult services (2nd ed.). ala neal-schuman.   bernier, a., males, m., & rickman, c. (2014). “it is silly to hide your most active patrons”: exploring user participation of library space designs for young adults in the united states. the library quarterly: information, community, policy, 84(2), 165–182. https://doi.org/10.1086/675330 bishop, k., & bauer, p. (2002). attracting young adults to public libraries: frances henne/yalsa/voya research grant results. journal of youth services in libraries, 15(2), 36–44.   bowler, l., acker, a., & chi, y. (2019). perspectives on youth data literacy at the public library: teen services staff speak out. the journal of research on libraries and young adults, 10(2), 1–21.   charmaz, k. (2006). constructing grounded theory: a practical guide through qualitative analysis. sage publications.   cook, s. j., parker, r. s., & pettijohn, c. e. (2005). the public library: an early teen’s perspective. public libraries, 44(3), 157–161.   fischer, r. k. (2015). rural and small town library management challenges. public library quarterly, 34(4), 354–371. https://doi.org/10.1080/01616846.2015.1106899   flaherty, m. g., & miller, d. (2016). rural public libraries as community change agents: opportunities for health promotion. journal of education for library & information science, 57(2), 143–150. https://doi.org/10.12783/issn.2328-2967/57/2/6   fleiss, j. l. (1971). measuring nominal scale agreement among many raters. psychological bulletin, 76(5), 378–382. https://doi.org/10.1037/h0031619   gauntlett, d. (2005). using creative visual research methods to understand media audiences. medienpädagogik: zeitschrift für theorie und praxis der medienbildung, 9, 1–32. https://doi.org/10.21240/mpaed/09/2005.03.29.x   gibson, a. n., hughes-hassell, s., & bowen, k. (2023). navigating ‘danger zones’: social geographies of risk and safety in teens and tweens of color information seeking. information, communication & society, 26(8), 1513–1530. https://doi.org/10.1080/1369118x.2021.2013920   grove, s. a., & brasher, n. (2020). the role of rural public libraries in providing access to online government services. center for rural pennsylvania. https://www.rural.pa.gov/getfile.cfm?file=resources/pdfs/research-report/rural-libraries-exec-sum-2020.pdf&view=true   hartel, j. (2014). drawing information in the classroom. journal of education for library and information science, 55(1), 83–85.   howard, v. (2011). what do young teens think about the public library? the library quarterly, 81(3), 321–344. https://doi.org/10.1086/660134   hughes, c., & boss, s. (2021). how rural public libraries support local economic development in the mountain plains. public library quarterly, 40(3), 258–281. https://doi.org/10.1080/01616846.2020.1776554   kelly, w., mcgrath, b., & hubbard, d. (2023). starting from ‘scratch’: building young people’s digital skills through a coding club collaboration with rural public libraries. journal of librarianship and information science, 55(2), 487–499. https://doi.org/10.1177/09610006221090953   knapp, a. a., hersch, e., wijaya, c., herrera, m. a., kruzan, k. p., carroll, a. j., lee, s., baker, a., gray, a., harris, v., simmons, r., kour sodhi, d., hannah, n., reddy, m., karnik, n. s., smith, j. d., brown, c. h., & mohr, d. c. (2023). “the library is so much more than books”: considerations for the design and implementation of teen digital mental health services in public libraries. frontiers in digital health, 5, 1183319. https://doi.org/10.3389/fdgth.2023.1183319   lenstra, n., slater, s., pollack porter, k. m., & umstattd meyer, m. r. (2022). rural libraries as resources and partners for outside active play streets. health promotion practice, 15248399211073602. https://doi.org/10.1177/15248399211073602   li, x., & todd, r. j. (2019). makerspace opportunities and desired outcomes: voices from young people. the library quarterly, 89(4), 316–332. https://doi.org/10.1086/704964   literat, i. (2013). “a pencil for your thoughts”: participatory drawing as a visual research method with children and youth. international journal of qualitative methods, 12(1), 84–98. https://doi.org/10.1177/160940691301200143   mehra, b., bishop, b. w., & partee ii, r. p. (2017). small business perspectives on the role of rural libraries in economic development. library quarterly, 87(1), 17–35. https://doi.org/10.1086/689312   meyer, j. (2018). poverty and public library usage in iowa. public library quarterly, 37(1), 53–60. https://doi.org/10.1080/01616846.2017.1312193   meyers, e. (1999). the coolness factor: ten libraries listen to youth. american libraries, 30(10), 42–45. ornstein, e., & reid, p. h. (2022). ‘talk to them like they’re people’: a cross-cultural comparison of teen-centered approaches in public library services. journal of librarianship and information science, 54(3), 451–468. https://doi.org/10.1177/09610006211020090   perryman, c. l., & jeng, l. h. (2020). changing models of library education to benefit rural communities. public library quarterly, 39(2), 102–114. https://doi.org/10.1080/01616846.2019.1621736   powell, t. w., smith, b. d., offiong, a., lewis, q., kachingwe, o., lovette, a., & hwang, a. (2023). public librarians: partners in adolescent health promotion. public library quarterly, 42(4), 361–372. https://doi.org/10.1080/01616846.2022.2107349   real, b., bertot, j. c., & jaeger, p. t. (2014). rural public libraries and digital inclusion: issues and challenges. information technology & libraries, 33(1), 6–24. https://doi.org/10.6017/ital.v33i1.5141   real, b., & rose, r. n. (2017). rural libraries in the united states: recent strides, future possibilities, and meeting community needs. ala office for information technology policy. https://www.ala.org/advocacy/sites/ala.org.advocacy/files/content/pdfs/rural%20paper%2007-31-2017.pdf   reid, h., & howard, v. (2016). connecting with community: the importance of community engagement in rural public library systems. public library quarterly, 35(3), 188–202. https://doi.org/10.1080/01616846.2016.1210443   subramaniam, m. (2016). designing the library of the future for and with teens: librarians as the “connector” in connected learning. journal of research on libraries & young adults. http://www.yalsa.ala.org/jrlya/2016/06/designing-the-library-of-the-future-for-and-with-teens-librarians-as-the-connector-in-connected-learning/#_edn7   subramaniam, m., scaff, l., kawas, s., hoffman, k. m., & davis, k. (2018). using technology to support equity and inclusion in youth library programming: current practices and future opportunities. the library quarterly, 88(4), 315–331. https://doi.org/10.1086/699267   weber, s. (2008). visual images in research. in j. knowles & a. cole, handbook of the arts in qualitative research: perspectives, methodologies, examples, and issues (pp. 42–54). sage. https://doi.org/10.4135/9781452226545.n4   woodgate, r. l., zurba, m., & tennent, p. (2017). worth a thousand words? advantages, challenges and opportunities in working with photovoice as a qualitative research method with youth and their families. forum: qualitative social research, 18(1), 126–148. https://doi.org/10.17169/fqs-18.1.2659   appendix interview questions   tell us a little bit about yourself. what grade are you in? how do you usually use your local public library? and why? do you have a library card? show and tell us what you drew. let the participants lead the talk. the interviewer will follow up with the why questions. pay attention to the space and design elements what materials and furniture do they want to see in the space? what kind of space design and arrangement? pay attention to the service/programming elements what kinds of services and programs do they mention when they talk about their pictures? tell us one thing that you want to see or want to do in our public library.     research article   identifying and classifying user typologies within a united kingdom hospital library setting: a case study   lynn easton library manager (south) nhs greater glasgow and clyde library network queen elizabeth university hospital glasgow, united kingdom email: lynn.easton2@ggc.scot.nhs.uk   scott adam assistant librarian nhs greater glasgow and clyde library network queen elizabeth university hospital glasgow, united kingdom scott.adam@ggc.scot.nhs.uk   trish durnan senior library assistant nhs greater glasgow and clyde library network maria henderson library, gartnavel royal hospital glasgow, united kingdom email: trish.durnan@ggc.scot.nhs.uk   lorraine mcleod assistant librarian nhs greater glasgow and clyde library network beatson west of scotland cancer library glasgow, united kingdom email: lorraine.mcleod@ggc.scot.nhs.uk   received: 18 dec. 2015   accepted: 22 oct. 2016         2016 easton, adam, durnan, and mcleod. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to identify available health library user typology classifications and, if none were suitable, to create our own classification system.  this is to inform effective future library user engagement and service development due to changes in working styles, information sources and technology.   methods – no relevant existing user typology classification systems were identified; therefore, we were required to create our own typology classification system.  the team used mixed methods research, which included literature analysis, mass observation, visualization tools, and anthropological research.  in this case study, we mapped data across eleven library sites within nhs greater glasgow and clyde library network, a united kingdom (u.k.) hospital library service.   results – the findings from each of the nhs greater glasgow and clyde library network’s eleven library sites resulted in six user typology categories: e-ninjas, social scholars, peace seekers, classic clickers, page turners and knowledge tappers.   each physical library site has different profiles for each user typology.  the predominant typology across the whole service is the e-ninjas (28%) with typology characteristics of being technically shrewd, it literate and agile – using the library space as a touch down base for learning and working.   conclusions – we identified six distinct user types who utilize hospital library services with distinct attributes based on different combinations of library activity and medium of information exchange.  the typologies are used to identify the proportional share and specific requirements, within the library, of each user type to provide tailored services and resources to meet their different needs.     introduction   several factors contribute to different types of users accessing hospital library services.  whilst some users continue to utilize the library for what could be considered traditional reasons (book borrowing and book based studying) changing work patterns and space restrictions mean that u.k. hospital libraries are also being used for non-traditional purposes (work, online study and leisure).  in addition, different learning styles and technological competencies mean that library users now prefer to access information through a variety of media such as smart devices.   health and hospital libraries are unique within the library sector with a very time-limited user base due to clinical demands.  the users are primarily busy clinicians and nurses who have patient care responsibilities, who demand instant access to information and who have no available workstations within their workplace (thomas & preston, 2016)  a large proportion of our users can be students or student doctors who appear to be using mobile technologies on the wards (chamberlain, elcock & puligari, 2015). to satisfy the demands of these different users and engage with their current and future service needs, a short-life working party was set up within the national health service greater glasgow and clyde (nhsggc) library network.  service provision within nhsggc libraries has been based on assumptions that professional role predicts style of library use e.g. busy nurses would focus on paper textbooks rather than electronic resources, with no further investigation to corroborate these assumptions.  however, more recent day-to-day anecdotal observations led us to suspect that this was no longer the case and that library use is based on characteristics other than professional role.   we chose to investigate if any relevant user typology classification scheme already existed that could be used, or adapted, by hospital libraries to identify the distinct differing classes of user that we encounter.  the definition of typology namely “classification of human behaviour characteristics according to type” (“typology”, 2007) was used to focus our project.   from the literature review, we identified that there are no existing typologies that match our particular needs to classify our users.  existing typologies originating from other sectors did not apply to our unique status as a provider of library services to busy clinicians, nurses, students and student doctors.  as a result of identifying this gap within the research literature, we created a unique user typology classification scheme specifically for nhsggc hospital libraries, but that could be used by other health and hospital libraries. we used mixed methods research to uncover relevant typologies and explored methods of visualizing our results.    literature review   to gain a better understanding and knowledge of nhsggc library users we undertook a literature analysis, based on themes, to identify the literature.  within the thematic literature analysis we looked for information on the following themes: changing library space and environments, what typologies have been used before, physical typologies, virtual or online typologies, health library specific typologies, methods of identifying the typologies and recommendations for use from these typologies.  the literature review identified that library environments are changing and that one way to identify our users’ current requirements is to place our users into a classification scheme.  following the analysis of the literature, we excluded out-of-hours, virtual and non-users from our project as out of scope.   we searched the following sources: embase, emerald, health management online, hmic, lista, medline and psycinfo, and online library catalogues: olib and shelcat for english language literature published since 2008.  the major search terms included the following: library*, knowledge, information, typolog*, behavio*, characterist* and millennial* (appendix one).  we kept up to date with any literature found during our project and added it into our knowledge base.   changing library environments   analyzing the changing library environment is identified as a theme by holder and lange (2014), talvé (2011) and todd (2009).  holder and lange (2014) state how they used mixed methods to identify space use and user satisfaction in canada.  they used observation, as a technique, eleven times and showed individual versus group study preferences and how this fed into service development within the library physical environment.   talvé (2011) plots the changing use of libraries across the decades and identifies the future of the library environment in australia.  she identifies that “the more virtual we become, the more we seek tactile, earthy, soft nesting spaces”.  she notes that “people like to be with other people in neutral spaces” and the library has a physical role in this.  she goes on to suggest that libraries are “places for collaboration” where library users gather together to solve issues creatively.   also in australia todd (2009) identifies that different library areas suit different types, for example “introverts” may prefer seating that faces into the wall and “extroverts” may prefer wide open comfortable seating.  she made use of surveys and observations and discovered discrepancies between what students planned to do and what they actually did.  for example, 32% of students planned to work on individual assignments, but under observation only 25% were observed doing this.  this study identified that using observation and surveys improved library performance.  the use of typologies in this article led us to acknowledge that we required to audit our own users to identify their use for library service development.   libraries are evolving, matching user learning styles to physical and virtual library space.  our study acknowledged that we needed a tool to measure the classification of library users within a u.k. hospital library environment to inform suitable changes to the library environment to match our users’ working styles, medium of exchange and activities.   library typologies   several general non-library specific typologies were identified by greene and myerson (2011), who noted that the world is changing to become more focused on the economy of knowledge.  these typologies were identified via ethnographic study, interview and visual tools around how people used their office space.  this london study suggested several typology classifications including anchors and connectors.  greene and myerson noted generational typologies such as generation x, millennials and baby boomers.  these typologies have a 20-year age span therefore we discounted these typologies as too broad for the purpose of our study, e.g. baby boomers will be retired or nearing retirement.   library specific typologies were identified by bilandzic and foth (2013) and zickuhr, purcell and rainie (2014).  within a wide-ranging study of american public libraries, zickuhr et al. (2014) identified typologies including “library lovers” and “distant admirers”.  bilandzic and foth (2013) analyzed library use within a learning context in australia; using ethnographic techniques several typologies were identified e.g. “learning freak fred” and “what can i do here sophia”.  these typologies are close to what we were looking to identify within the nhsggc library hospital context, but were rejected early on because the library context within this paper did not fit our own research context as it is from a “digital cultural centre” context.  our literature review did not identify any health library specific typologies.   we therefore discarded the use of existing library typologies within our research topic.  the use of existing typologies would have been time saving and would have created comparable results to study within published papers.  in reality we did not feel that the typologies presented to us within the literature could be transferred to the one situation with nhsggc hospital libraries because our study is aimed at classifying users within the physical use of space only, and in a professional nhs health service hospital setting.  we expected our small data sample size would not cover more than two generations.  this meant that we rejected the use of known typologies due to the differences in scale and limited transferability of results from our u.k. hospital setting compared to the large-scale users and resources of public library or higher education library settings.   virtual and online typologies   virtual or online typologies were identified by lawrence and weber (2012), brandtzaeg and heim (2011) and nicholas, rowlands, clark and williams (2011).  in a study in the united states of america, lawrence and weber (2012), observed higher education students late at night and generated amusement from students about the “diligence” of the librarians observing them out of hours.  however, this is in a higher education setting which would result in high footfall, particularly at exam times, and would not be comparable to the nhs greater and glasgow out of hours setting.  within nhsggc libraries there is out of hours use, particularly for on-call staff, rather than students.  this study also concentrated more on use of the library and activities rather than typologies.   brandtzaeg and heim (2011) from norway identified online social networking typologies such as “sporadics” and “lurkers” using an online questionnaire, whilst nicholas et al. (2011) identified web information seeking behaviour in the united kingdom.  they classified the behaviours into various animal typologies such as “web hedgehog” and “web ostrich”.   due to the relatively low numbers of out of hours library users within the nhsggc context and the challenges of identifying virtual typologies, the typologies identified in these papers were rejected as methods for our research.  there is a noted potential to study these at a future date if resources such as new technologies e.g. tracking via mobile apps became cheaper and more widely available.   typology methodologies   typology methodologies are discussed by urquhart (2015), kline (2013), gajendragadkar et al. (2013) and lawrence and weber (2012).  observation can be a very useful research tool and urquhart (2015) noted that as yet “little research discussed observation as a major part of the research methodology”, whilst suggesting that it can be a time and labour intensive process.  she notes that modern digital tools such as “phones and digital recorders” make observation a relatively easier method to use than in the past.   kline (2013) interviewed david green from the erial (ethnographic research in illinois academic libraries) project (erial, 2015), and he confirms that even a relatively tiny study can identify a lot about your library users.  the negative side of this is that it can take a great deal of staff time to run such a study.  he also identifies that using ethnography puts librarians into the “users’ world” thus motivating change, and this matched the service improvement goals of our project.   gajendragadkar et al. (2013) undertook a covert observational study in an nhs hospital setting proving that such ethnographic techniques could be used within an nhs setting.  similarly, lawrence and weber (2012) noted that their research took in a variety of styles “written surveys, interviews, observation, mapping and statistics” and this encouraged us to proceed with mixed methods research within our own project.   online tools such as blogs and social media (#ukanthrolib, 2014 and lanclos, 2015) are used to identify anthropological and ethnographic methods within library settings and these tools informed our small-scale project.   ethnography has historically been linked to both anthropology and sociology.  reeves, peller, goldman and kitto (2013) in their paper on ethnography, within educational research, state that “the ethnographer goes into the field to study a cultural group”.  they also go on to note that small groups have been studied and documented since the early twentieth century.  our study aimed to identify a small group, namely u.k. hospital library users, and this fits in with the ethnographic methodology.    brewer (2000) states “ethnography is not one particular method of data collection but a style of research” and its ability to mix and match research methods such as observation, personal diaries and interviews gives credence to the fact that ethnography has become an evolving and increasingly used tool within libraries within the last ten years as can be seen from the popularity of the ux (user experience) in libraries concept (uxlibs, 2016).   typologies use in practice   the literature also identified recommendations for how typology classifications could be used.  bilandzic and foth (2013) suggest that new mobile device technologies allow a more fluid and non-owned space.  this resonated with our project as this matches the increased number of agile workers using library space within nhsggc.  their research is not directly replicable with our users as the observation took place over five months within a large library and we were unable to devote similar timescales to our project.   difficulties in finding out what non-users think and do in the library were identified in the united kingdom by booth (2008).  he categorised people into typologies such as “non-seekers” or “confident collectors”.  he described how typologies can help influence the design for library space around the various different wants and demands of users.  this specifically fits in with the demands and requirements for service improvements due to changes of working styles, technologies and information resources within nhsggc.  we rejected this typology because the research is not set in a u.k. hospital library setting.   we identified that nhsggc libraries have been evolving with the change of use both traditionally and technologically and from solitary to group learning to virtual.  we identified that we need to observe user typologies that, once diagnosed, can be used as a tool to develop library services.  we searched the literature and identified that typologies have been classified within the library and digital contexts but that the previous research did not drill down specifically enough for the purposes of our research within the u.k. hospital library context.   methods   the short life working group did not identify a relevant user typology classification tool within the literature, suitable for a u.k. hospital library setting, therefore we created our own typology classification system.  the team used mixed methods research including literature analysis, mass observation, visualization tools and ethnographic research.  we tabulated data across eleven library sites within the nhsggc library network.   initial scoping of methods   initial discussions using smart board® technology enabled the working group to model, and have interactive discussions, around the definitions of users’ activities and how to collect the data.   analysis of the literature noted that mixed methods research methodology such as observation is frequently used with typology work (bilandzic & foth, 2013).  observation includes the use of qualitative and quantitative data.  we decided, using this evidence, to create an observational method that would suit our small-scale library setting but that would be generic enough to be used in any hospital library.   at this stage, we devised an initial prototype three-dimensional activity axis grid (figure 1) based on our knowledge of nhs u.k. hospital libraries, and the review of literature around changing library environments (holder & lange, 2014). we came up with a three-dimensional cube, with gridlines, as we identified three important dimensions of knowledge behaviour.   the first dimension is the method of use, namely “traditional” use (e.g. reading a book) or “virtual” (e.g. searching a database), giving the potential for “mixed” use (e.g. reading a book whilst utilising a laptop).  our second dimension is whether the activity is undertaken alone (solitary) or within a group.   the third dimension is the activity itself within the library setting which, we identified, could consist of study (e.g. reading a textbook), information seeking (e.g. asking library staff for help with finding an electronic journal article), or reflection/learning (e.g. writing up an audit).     figure 1 prototype three-dimensional activity axis grid.     test observation   observation includes the use of qualitative (e.g. asking library users what space they use within the library for what purpose) and quantitative data (e.g. numbers of people sitting at a particular seat within the library over a given period).  urquart (2015) defines this type of observation as “simple observation” which enables you to watch what is happening but not intervene or change the activity.  we felt that simple observation would avoid the need to request ethics approval, and cause less disruption to our end-users as frequent interruptions to question them would have disrupted their library activity and studies.   in october 2013, the group tested an initial observational tool on five nhsggc library sites.  one hundred individual bits of test data were collected.  after collection, we discussed our methods and any problems that had arisen, such as being unsure which box to tick for various activities.  this test also identified that our data collection did not capture the three-dimensional activity that we had sought to identify with the help of our prototype three-dimensional activity axis grid (figure 1).   using these data we redesigned the observation sheets several times, utilising test data, until we finalized our mass observation grid design (figure 2).    following on from our prototype three-dimensional activity axis grid (figure 1), we refined the observation grid (figure 2) into two separate grids each featuring an axis of medium of information exchange versus an axis of library activity.  this created a two-dimensional approach but across two separate facets of use, in theory the three axes we originally worked to (figure 1).  the first facet focused on solitary behaviour (individual people working alone) and the second facet on groups (two or more people working together).  this captured all the data we required but created a more logical measurement.   we defined the activities into learning, study, information seeking, working and social (figure 3).  we also identified and defined the resources utilized within the library space as interpersonal, book/paper, bring your own device, pc/it equipment and library staff.   live mass observation   the live observation ran over one week in march 2014 on eleven sites within nhsggc.  the observation took the form of a paper grid (figure 2), which was marked up by local library staff doing the observation at each location.  the library sites varied from larger multi-disciplinary nhs libraries with large footfall to smaller nhs libraries with part-time staffing and limited space.     figure 2 blank observation grid.   figure 3 observation grid guide.     library staff observed all use and footfall activity within the library setting – and marked one score mark on the grid for every new activity versus medium of activity.  if users changed what they were doing, or whom they were doing it with, this was noted on the grid as a simple score.  the record of activity could be fluid e.g. one person could enter the library and take part in different activities with different resources.  this did mean that the observation was open to a certain level of subjectivity and therefore the working group offered an online webex® conference to all library site staff to attempt to minimize potential inconsistencies, and to explain the observation methods and techniques.    to ensure consistency amongst all library sites participating in the live observation we created an observation grid guide (figure 3) that identified the initial classification of users’ use of physical library space that we were aiming to identify.  these instructions and examples of activities and resources formed the backbone of the observation.  this was backed up with the working group acting as mentors during the week, who were able to intervene if there were any questions whilst the observation was ongoing.   as nhs library sites can be busy at different times, due to clinical requirements, plus one library was moving location during this time, we were not prescriptive about when sites would observe their users, just that they would observe within the timeframe of that week.  we also knew that as sites are different sizes we would get different sample sizes from each site.  therefore, we allowed library sites the freedom to choose their sampling times and amounts, which in retrospect may have affected our study sample size for some sites.   visualization   once the data were returned from the eleven library sites, the working group recorded, analyzed and tested the results of these data.  the review of the literature had identified that visualization of the data is the key to analyzing separate classes of data (urquhart, 2015).   we analyzed the test data using microsoft excel® charts to enable visualization.  a promising output at this stage was their surface contour charts.  our initial thoughts were to produce some form of three-dimensional visualization, as it was hoped that distinct typologies would jump out as peaks or hot spots.  the surface contour chart (figure 4) created the three-dimensional element we had used with our prototype three-dimensional activity axis grid (figure 1).  ultimately, this approach failed as the imagery failed to produce the clear results for which we had hoped.  whilst the surface contour option (figure 4) enabled us to pinpoint accurately the specific cross sectional areas of high activity of our library users, the contour chart did not provide a suitable visualization of the axis between activity and resources that we were seeking.  the surface charts did help towards us identifying the categorizations that we were interested in establishing to enhance data analysis, at the intersections.      figure 4 surface contour map.     we tested the microsoft visio® software package (figure 5) which shows an alternative visualization of the data.  this visualization software was rejected because it offered no relevant graphical interpretation suitable for our needs as it did not show the axis of information exchange versus an axis of library activity in enough detail.   figure 5 microsoft visio ® visualization.     we re-analyzed the data and identified that to create typologies relevant to the uk hospital library setting we needed to match the intersection of the observed activity along one axis with the observed medium along the other.  during this re-analysis it was identified (figure 6) that the chosen composite data of activity type (traditional or non-traditional) intersecting with medium of information exchange (traditional, technical or human) gave us the closest match to the number of user typologies found in other papers e.g. bilandzic and foth (2013) and brandtzaeg and heim (2011) who classified into five typologies.  given the relatively small amount of data collected in our project, we decided that six user typologies was the maximum number of classification types into which the data could be split.  we therefore annotated our observational grid and mapped the data, where they intersected, to our six typologies (figure 7).   we focused our typologies research on the medium of information exchange, plus the actual library activity, rather than actual professional health service staff or undergraduate students.  we did this to ensure that we captured actual activity of library users rather than assuming that because you were e.g. a doctor that you would automatically have the same user typology as all other doctors.  the same applied to us identifying and classifying use by undergraduate students on placement, as from the literature (nicholas et al., 2011), we had already noted that not all users within the same generation used resources in the same way e.g. we are aware in our day to day library role of undergraduate students who prefer physical books and older doctors who prefer to use e-books for their work.  we were interested in what our users used the library for, how this use is changing and not in who they were professionally.   results   collation of the results of the two axes, firstly of traditional or non–traditional activity intersecting with, secondly, traditional, technical or human medium of information exchange led to the six user typology definitions: page turners, classic clickers, knowledge tappers, peace seekers, e-ninjas and social scholars (figure 6).   individual and group results were generated for each library site and for the nhsggc library network as a whole.  we found that user typologies were consistent across all eleven library network sites, as we provide the same services to the same users, the only difference usually being the size and scale of the library resources, library space and library users on site.  overall results were collated and e-ninjas made up 28% of the individual user typologies identified during this project (figure 7).     figure 6 user typology grid.     figure 7 mapping of observation grid data to user typologies categories.     figure 8 mass observation results.     the e-ninja typology is most prevalent across the library network (28%) (figure 8), which reflects the move within nhsggc organizational culture to agile working.  this type brings their own device into the library and tends to be technologically competent.  they use the library space as a buffer zone between work and personal space.   the second most popular typology, at 27%, is the knowledge tapper, who have excellent interpersonal skills; they rely on knowledge from library staff and can be seen as organizational knowledge brokers.  the knowledge tapper requires a space to communicate.   an interesting typology, at 19% of those observed, are the social scholars, who are also the typology most likely to operate in a group.  this is due to their characteristics of being more non-traditional users.  they see the library space as somewhere to learn from other people in a more informal manner than previously seen within nhsggc library space.  they see the library as a third place.    a steady number of users were identified as classic clickers (13%).  this is the type of person who comes into the library space just to use the pcs.  they use the pcs to learn and work, and use library staff for minor technical it issues.  we felt that, over time, these users may become e-ninjas with encouragement.   page turners were observed less frequently (at 9%) within the library space.  this typology is traditional users, those who come into the space and enjoy learning from books and paper.  they come into the library to browse the stock, will sometimes sit and study, but often take their books and papers to their home or workplace.   the lowest number of user typologies observed within the library setting is the peace seeker, at just 4% of all observations, and this is a library user who is looking for quiet and silence to work.  they are a solitary worker and see the library as a neutral space that does not hold the distractions of work or home.  peace seekers need to concentrate and use the library as reflective space.   quick quiz   once we had identified our typologies, we wanted to test our hypothesis about how we had classified library users.  the working group created a quick fun quiz using questback (www.questback.com/uk) to allow users to find out what typology they might be.  we emailed out this link to library network users and we got a return of over 350 user results.  the results fundamentally differed from our observation (figure 9).  the reasons for this could include the fact that it was an online quiz and therefore attracted a different typology.  it may also mean that virtual or online users, whom we did not capture in our physical library observations, participated in the survey as it was emailed out to all library network members.  it may also have meant our questions in the quiz needed recalibrating.  this is an interesting adjunct to the main research and allowed us to question the validity of the main results of our research.   figure 9 user typologies quiz results.   discussion   within nhsggc, each physical library site is shown to have different proportions and profiles for each of our uniquely identified health library user typologies.  although typology methodologies were discovered in the literature review we felt that none of these would fit the specific requirements of our project, e.g. web technology typologies would not reflect our users’ physical footfall.  we recognised through our observation that users can have multiple typologies and that these can change over time.   many recent articles have focused more on virtual typologies, which we felt would be hard to capture, within our nhsggc context given our limited project timescale.  we also rejected millennials and generation x style typologies at this stage, as they are wider generational typologies and too broad for the purposes of this case study.   the results of our typologies research in 2014 enabled us to forecast changing typology use for a new library site that opened in 2015.  through utilizing the data from this project, we identified that a new-build u.k. hospital library would require more space for e-ninjas and group learning types such as social scholars, than page turners or peace seekers.  we input this research into the architect plans and enabled zoning more space for e-ninjas (agile, fluid laptop users) e.g. creating adaptable power points and wi-fi across the library to enable rapid access to information.  we required three separate rooms within the library space, which are used flexibly to suit different typologies at different times.  one of the spaces is bookable as a group space for e.g. social scholars, but when the room is not booked it creates more individual silent study space for typologies such as the page turners and peace seekers.   we utilized the typologies as a promotional tool when this new u.k. hospital library opened.  we used six specially designed bookmarks and posters (figures 10 and 11), one for each typology, with information about that specific typology and identified what library services could be best suited to them.  the bookmarks grabbed attention and encouraged dialogue between library staff and users.     figure 10 user typology definitions for bookmarks.     figure 11 bookmarks and poster.     future work   the project took a lot longer to scope, plan and action than anticipated.  the mass observation was run over one week.  there is potential to run it again in the future to see if the proportions of typologies within the library network change as library environments develop.    we hope this study has added to the literature on user classification tools within libraries.  informal feedback from other health sector library staff has been positive.  they recognized these typologies within their own user base and indicated that they are keen to use this classification system in their own libraries.  ideas that could be explored in the future, that were beyond the scope of this project, include the potential to capture more closely multiple typologies of individuals or groups over time.  virtual and out of hours typologies were also beyond the scope of the current project but would be an interesting project to pursue in the future.   conclusions     currently there is a lack of studies relating specifically to user typologies within the uk hospital library sector.  our case study enabled us to create a bespoke user typology classification system that, when used in conjunction with a programme of structured observation, could be utilized by other u.k. hospital libraries to gain an understanding of how their users utilize physical library services and space.  consequently, user engagement and service development could be more effective as services, resources and physical design will be based on health-specific user typologies.   references   #ukanthrolib. (2014, mar.). spaces, places and practices: ucl-ioe joint library anthropology seminar. retrieved from https://storify.com/senorcthulhu/spaces-places-and-practices-ucl-ioe-joint-library   bilandzic, m., & foth, m. (2013). libraries as coworking spaces: understanding user motivations and perceived barriers to social learning. library hi tech, 31(2), 254-273.   http://dx.doi.org/10.1108/07378831311329040   booth, a. (2008). in search of the mythical "typical library user". health information and libraries journal, 25(3), 233-236. https://dx.doi.org/10.1111/j.1471-1842.2008.00780.x   brandtzaeg., p. b., & heim, j. (2011). a typology of social networking sites users. international journal of web based communities, 7(1), 28-51. https://dx.doi.org/10.1504/ijwbc.2011.038124   brewer, j. d. (2000). ethnography. buckingham: open university press.   chamberlain, d., elcock, m., & puligari, p. (2015). the use of mobile technology in health libraries: a summary of a uk-based survey. health information & libraries journal, 32(4), 265-275. https://dx.doi.org/10.1111/hir.12116   ethnographic research in illinois academic libraries. (2015). erial project. retrieved from http://www.erialproject.org/   gajendragadkar, p.r., moualed, d.j., nicolson, p.l., adjei, f.d., cakebread, h.e., duehmke, r.m., & martin, c.a. (2013). the survival time of chocolates on hospital wards: covert observational study.  bmj, 347, f7198. http://dx.doi.org/10.1136/bmj.f7198   greene, c., & myerson, j. (2011). space for thought: designing for knowledge workers. facilities, 29(1/2), 19-30. http://dx.doi.org/10.1108/02632771111101304   holder, s., & lange, j. (2014). looking and listening: a mixed-methods study of space use and user satisfaction. evidence based library and information practice, 9(3), 4-27. http://dx.doi.org/10.18438/b8303t   kline, s. (2013). the librarian as ethnographer: an interview with david green. college & research libraries news, 74(9), 488-491. http://crln.acrl.org/content/74/9/488.full   lanclos, d. (2015). donna lanclos the anthropologist in the stacks. retrieved from  http://www.donnalanclos.com/   lawrence, p., & weber, l. (2012). midnight-2.00 a.m.: what goes on at the library? new library world, 113(11/12), 528-548. http://dx.doi.org/10.1108/03074801211282911.    nicholas, d., rowlands, i., clark, d., & williams, p. (2011). google generation ii: web behaviour experiments with the bbc. aslib proceedings, 63(1), 28-45. http://dx.doi.org/10.1108/00012531111103768   reeves, s., peller, j., goldman, j., & kitto, s. (2013). ethnography in qualitative educational research: amee guide no. 80. medical teacher, 35(8), e1365-e1379. http://dx.doi.org/10.3109/0142159x.2013.804977   talvé, a. (2011). libraries as places of invention. library management, 32(8/9), 493-504. http://dx.doi.org/10.1108/01435121111187860   thomas, g., & preston, h. (2016). barriers to the use of the library service amongst clinical staff in an acute hospital setting: an evaluation. health information & libraries journal, 33(2), 150-155. http://dx.doi.org/10.1111/hir.12141   todd, h. (2009). library spaces new theatres of learning: a case study. journal of the european association for health information and libraries, 5(4), 6-12.   typology.  shorter oxford english dictionary (6th ed.). (2007) oxford: oxford university press.   urquhart, c. (2015). observation research techniques. journal of eahil, 11(3), 29-31. http://eahil.eu/wp-content/uploads/2015/09/29-31-urquhart.pdf   uxlibs. (2016). exploring ethnography, usability and design in libraries. retrieved from http://uxlib.org/   zickuhr, k., purcell, k., & rainie, l. (2014). from distant admirers to library lovers and beyond: a typology of public library engagement in america. in pew research center. retrieved 6 nov. 2016 from http://www.pewinternet.org/2014/03/13/library-engagement-typology/     appendix search terms used to identify relevant articles   librar* and (typolog* or behav* or characteris*) ((information or knowledge) adj/n1 seeking) librar* and ((information or knowledge) adj/n1 seeking) ((information or knowledge) seeking) and (typolog* or behav* or characteri* or style*) (librar* n1 behav*) n1 use* librar* and typolog*   librar* adj3 space library* adj3 chang* librar* adj3 enviro*   user adj3 behav$ user$ adj3 typ$ user$ adj3 group$   millennial*1 google generation generation x1 generation y1 digital native* i-generation i-gen generation-i gen-i net generation net gen   mesh terms: exp libraries/ exp information theory/ exp information seeking behavior/   key 1 term used alone and also in combined with and (librar* or typolog* or behav* or characteri*) adj= adjacent n1= within one word $ or *= truncation exp= explode research article   information literacy beyond librarians: a data/methods triangulation approach to investigating disciplinary il teaching practices   britt foster public services librarian henry madden library california state university, fresno fresno, california, united states of america brfoster@csufresno.edu   received: 30 aug. 2019                                                                  accepted: 3 jan. 2020      2020 foster. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29635     abstract objective – while library literature contains many studies examining faculty perceptions of the value of librarian-led information literacy (il) instruction, there is little evidence regarding il instruction practices of disciplinary faculty independent of librarians. in a climate of uncertain budgets, increasing student enrollment, and increased conversation around the need for il, media, and digital literacy skills, this study aimed to investigate a little-researched area of the il instruction, learning, and development milieu.   methods – in collaboration with the institutional research office, a data and methods triangulation approach was used. a survey of disciplinary faculty was administered and disciplinary faculty focus groups were also conducted. student outcomes and annual assessment reports, documents that describe teaching and assessment methods for courses across the university, were analyzed. voyant, a text-mining tool, was also used to determine key phrases and terms related to il in these documents.   results – results revealed that disciplinary faculty highly value skills and understandings affiliated with il competency. faculty provide the majority of il learning opportunities independent of librarians, although these learning opportunities are generally provided through implicit, rather than explicit, methods. pedagogical methods that may enable explicit practices, such as the use of standards and competencies, are infrequently used.   conclusion – evidence and findings from this study are being used to inform several initiatives to work with disciplinary faculty for il instruction, including new services, resources, and instruction models to support il development in students.     introduction   the california state university (csu) is a large public university, serving students seeking a professional, applied, comprehensive education (csu, n.d.). as a member of the csu system, california state university, fresno (fresno state) fulfills this function for the fresno region of the central valley of california, serving four counties in the heart of the state. fresno state is a large campus of 25,000 students and 3,000 employees, with degrees offered in the arts and humanities, agricultural sciences, business, engineering, health and human services, social sciences, and sciences. fresno state also offers 43 masters degrees and three doctoral degrees (csu, 2016).   as a member of the western association of schools and colleges senior college and universities section (wscuc), fresno state is evaluated for accreditation on five core competencies: critical thinking, quantitative reasoning, written communication, oral communication, and information literacy (il) (wscuc, 2016). as part of the assessment work related to accreditation, the il assessment committee has administered the standardized assessment of information literacy skills (sails) test to incoming freshmen (fall) and graduating seniors (spring) every year for several years. the results have consistently demonstrated that fresno state incoming students score far below average in il skills, and have not yet developed proficiency in research and information skills essential to their academic and professional success.   the teaching and learning librarians at henry madden library (madden library) primarily provide il instruction through the traditional one-shot, in sessions lasting approximately 30-90 minutes. in the context of sails scores, as well as other il assessments, madden library librarians have often discussed the effectiveness of the one-shot to provide students with foundational il skills, as well as the more advanced skills necessary for their undergraduate and graduate degrees.   in exploring ways to innovate in developing higher-order il skills, the opportunity to partner with the institutional research office, the office of institutional effectiveness (oie), in their inaugural oie faculty fellows program was presented. designed to familiarize faculty with institutional research services and resources, this program provided faculty researchers a stipend to explore questions relevant to university interests, including accreditation. the library researcher chose to explore the pedagogical practices of disciplinary faculty related to il, focusing specifically on how disciplinary faculty teach il to their students. in choosing to examine il instruction and learning outside the context of library teaching and learning work, the goal of this study is to gain an understanding of how to partner with disciplinary faculty to develop il skills in students.   literature review   many studies have examined il within the university ecosystem. these include efforts to integrate il into the curriculum, both at the institutional and departmental levels; examining collaborative il instruction between librarians and disciplinary faculty; and, to a smaller degree, the role of disciplinary faculty in teaching il independent of the library or their librarian colleagues. much of this literature is written by and for librarians (bury, 2016; dacosta, 2010), and as such, the independent teaching of il by disciplinary faculty has not been fully explored.   beyond the library: il across the institution   examining il in the context of wider implementation has been motivated by several factors. amongst these is the desire to increase the impact of il instruction beyond the one-shot (smith, 2006, the inclusion of il within university accreditation standards (which necessarily broadens the responsibility for il teaching and assessment) (owusu-ansah, 2004; thompson, 2002), and to examine or emphasize unique aspects of il teaching within the disciplines (brems, 1994; detlor, julien, willson, serenko, & lavallee, 2011; emmons et al., 2009; gonzales, mcmillen, & fabbi, 2009; lwoga, 2013; mounce, 2010). exploring il outcomes, conceptualizations, and practices beyond librarian instructional design can inform methods for successful implementation of il beyond library curriculums and services (mackey & jacobson, 2005).   a necessary component of this work is collaboration between disciplinary faculty and librarians. the literature around these collaborations is extensive (there is in fact a journal dedicated to documenting and communicating this work, collaborative librarianship). these studies generally consist of reporting on collaborative projects, or, exploring the theory and approach to the collaborative process itself, as in gardner and white-farnham (2013). mounce (2010), in their review of librarian/disciplinary faculty collaborations from 2000-2009, provides an in-depth look at the nature of some of these reported collaborations.   the work described above, however, generally centers on the librarian/library perspective. one reason for this is that il research remains largely within the domain of librarianship (bruce, somerville, stoodley, & partridge, 2014). the development of il as a concept and as a literacy has also generated from within the library and information sciences (leaning, 2017). the result is that disciplinary faculty are less aware of methods to integrate il effectively into their teaching practice (bury, 2016).   while the above-mentioned studies on collaboration have examined some methods to increase the capacity of il teaching for non-librarian instructors, again, the literacy expertise lies with the librarians. this is demonstrated through “teach-the-teacher” approaches, when librarians use this expertise to provide their disciplinary colleagues with the skills to integrate il into their own practice (bury & sheese, 2016; everett, 2010; smith, 2006; veach, 2009).   faculty perspectives on il   beyond methods to integrate il more broadly into the institution, there has also been an examination of disciplinary faculty and their perspectives on or valuing of il concepts and competencies. this is central to this study, which seeks to extend knowledge not just of what faculty think of il, but how they “do” il, integrating it into their teaching work. an additional component of faculty perspectives on il is their perspective on the responsibility for teaching il—do they view that responsibility as theirs, belonging to the librarian, or elsewhere?   recent studies on disciplinary faculty perceptions of il generally measure perceptions in three areas: 1) perceptions of the library, librarians, and library il instruction; 2) general perceptions of il skills and concepts and their value to postsecondary students, and 3) perceptions of students’ il skills.   this first consideration is interesting in that it emphasizes the central nature of libraries and librarians in research around il. the tension between librarians and disciplinary faculty, and the sense of librarians as “minor faculty” (or without faculty status at all), has proven to be an obstacle for some librarians in their il work (gardner & white-farnham, 2013; julien & given, 2002; nilsen, 2012). it is necessary, therefore, to understand how disciplinary faculty perceive librarians in order to be successful in outreach and collaborations.   an element of the perceptions of disciplinary faculty of librarians is their role in the responsibility for il instruction. several studies have found varying perspectives on this role. within the context of accreditation, thompson (2002) discusses the role that both librarians and disciplinary faculty have in teaching il. several studies have found that disciplinary faculty agree, with some caveats: nilsen (2012) found that faculty believe in a shared responsibility, but with a larger role for librarians. saunders (2012) also found that faculty believe in a shared responsibility, but with a larger role for disciplinary faculty. bury’s (2011) findings indicated that faculty believe that both librarians and faculty should have a shared role, but that ultimately, faculty do the work of il instruction themselves. weiner (2014) also found that faculty do the work of il instruction, in a study of 299 faculty at purdue university. these findings confirm that faculty, either with or without librarians, view themselves as having a role in il instruction.   multiple studies have found that faculty highly value il skills and competencies for their students (bury, 2011, 2016; dacosta, 2010; lwoga, 2013; nilsen, 2012; saunders, 2012; weetman, 2005). dacosta (2010) evaluated several key il skills, and found disciplinary faculty wished students had developed all of the listed skills by the time they graduate: no skill scored below 80% of responding faculty. bury (2016) discovered that faculty strongly desire their students to have the ability to develop a topic, as well as evaluate and synthesize found information. however, multiple studies have also found that disciplinary faculty perceive their students to be weak in il understandings (leckie & fullerton, 1999; lwoga, 2013; nilsen, 2012; saunders, 2012).   if disciplinary faculty view il as their responsibility, and value il as a skill necessary for students to learn, how then, do faculty do the work of il teaching? there have been limited studies of il pedagogy outside of librarianship, the gap which this study aims to address. in the few studies found, il teaching is often tacit, and assumed to be learned through “osmosis” (gardner & white-farnham, 2013; lwoga, 2013; weetman, 2005). leckie and fullerton (1999) examined engineering and science faculties’ il practices, and found only half incorporated some aspect of il all the time. the most consistent practice was assignments designed to develop critical thinking (leckie & fullerton, 1999); bury (2016) also found that faculty highly value pedagogy that develops critical thinking, and higher-order cognitive skills. the integrated nature of il, critical thinking, and other academic literacies was also found to be central to faculty’s teaching practices, and they view these literacies as intricately connected to disciplinary knowledge (bury & sheese, 2016).   the previous emphasis on librarianship as the lens through which faculty il teaching practices are viewed, and the evidence that faculty view il as valuable and necessary for their students, provides the framework for a deeper look at faculty il pedagogy, independent of librarians.   aims   to enable deeper il learning for fresno state students, an investigation of faculty il teaching practices was conducted. through an understanding of these practices, it is hoped that librarians will be able to better work with their disciplinary faculty to create and develop further a rich il teaching culture across campus. to interrogate the existing il teaching culture, the following questions guided the study:   1.       what il teaching practices are departmental faculty currently implementing? 2.       what il standards, resources, and concepts are departmental faculty currently using to inform their il teaching practice? 3.       what il skills and concepts are valued within a particular discipline, as an academic literacy and/or as a professional skill?   methods   to investigate these questions, a data and methods triangulation approach was used. a survey of disciplinary faculty was administered, and disciplinary faculty focus groups were conducted as a follow-up. in addition, student outcomes and annual assessment reports, documents that describe teaching and assessment methods for courses across the university, were analyzed. voyant, a text-mining tool, was used to determine key phrases and terms related to il in these documents. this method was selected to account for the complexity of capturing teaching practices, and to be able to capture data through multiple facets.   survey   the survey consisted of six sections:   1)       demographic information. 2)       a list of il skills, concepts, and understandings, and if and how instructors include these in their teaching practice, on their syllabus, and who teaches the conceptdepartmental faculty, a librarian, or both. 3)       the same list of il skills, concepts, and understandings, and how important these skills are to their students as students, and, how important these skills are to their students post-graduation. 4)       key il resources, including global resources, such as the framework for information literacy in higher education (known as the framework) (association of college and research libraries, 2016), and local resources, such as library tutorials and research guides. 5)       and an open-ended comment box, where instructors could share any additional il-related thoughts.   the survey was sent electronically to a representative sample of disciplinary faculty of all ranks, including non-tenure track adjunct faculty. a total of 602 faculty received the survey, and 122 responses were received. incomplete responses were removed, for an n=91 (15% response rate). survey participant responses are outlined in table 1.   focus groups   two separate focus groups were conducted, for a total of 9 faculty participants (7 tenure-track, 2 adjunct) participants self-selected from survey respondents. lasting an hour each, participants were asked seven questions in a semi-structured interview format.   faculty il practices focus group: questions   1.       how would you describe or define information literacy? 2.       do you see a distinction between information literacy and other concepts such as digital literacy, media literacy, or critical thinking? do you see areas of overlap? 3.       in your teaching work, what information literacy practices are you currently engaging in? 4.       what do you feel is your role in teaching information literacy? 5.       what role does information literacy play in your discipline? 6.       do you see a distinction between general information literacy, and the information culture and practices of your discipline? what role do different partners (k-12, gen ed courses, w courses, etc.) play in developing these skills? 7.       what resources do you need in order to incorporate or further develop instruction and assessment of information literacy into your teaching practice?   responses were recorded and transcribed, and then independently coded by the researcher and a graduate research assistant who was also present during the focus groups. results were then analyzed and discussed together.   table 1 demographics of disciplinary faculty survey respondents by percentage characteristic percentage gender female male prefer not to state   62 36 2 age 19 years of age or younger 20-30 years of age 31-40 years of age 4150 years of age 51-60 years of age 6170 years of age prefer not to state   2 6 36 20 23 11 2 years teaching at the college level first year 2-4 years 5-7 years 8-10 years 11-15 years 16 years or more   6 25 23 8 23 23 rank temporary/lecturer: part-time temporary/lecturer: full-time assistant associate full   24 13 36 16 11 college arts & humanities health & human services science & math social sciences business agricultural sciences & technology education engineering     18 19 13 12 8 11 13 5   assessment reports   at fresno state, assessment coordinators work with the faculty in their departments to gather data on specific learning outcomes assessed on a schedule that has been reported to the university assessment coordinator. the results of these assessment activities are then stored on the fresno state website. these assessment reports were used as a third data source. for this study, the most recent three years of assessment reports (15/16, 16/17, and 17/18) were used, because departments only assess a select number of outcomes a year, and so a broader sample was needed to capture any recent il teaching and assessment work (i.e., if il wasn’t assessed in 15/16, it may have been in 16/17). a total of 172 reports (1731 pages) were analyzed.   the reports were initially analyzed using voyant tools. the 172 reports were uploaded to voyant as a master pdf. using “document trends,” a feature of voyant tools, a list of terms and phrases generated from the survey and focus groups were used to target relevant areas from within the larger corpus. the terms used were creat*; source*; citation|cite*; search*; information literacy; synthe*; database*; library; plagiar*; database; evaluate information~5; article peer~5. the ‘*’ was used to capture all forms of a term (e.g., plagiar* to return plagiarism, plagiarize, etc.). the ‘|’ was used to count terms as a single count (e.g., citation|cite as a single term). the ‘~’ was used to capture “near” results, where terms occurred within a certain number of words of each other. these results were then set to capture the nearest 50 words on either side of these terms, exported to excel by term/phrase, and qualitatively assessed. results were also used to follow up directly in the assessment report text, if phrase results indicated deeper results would be found through further reading.   results   results of the survey, focus groups, and assessment report analyses are reported in this section. results from each individual section are reported first, followed by a triangulation analysis of major themes and discoveries based on the combination of results.   survey results   results of the survey are reported below. survey sections 2 and 3 (il skills, concepts, and understandings and inclusion in teaching practice, on syllabi, and teaching responsibility; as well as the same list and academic and professional value) are reported in table 2. survey sections 4 and 5 (awareness and use of il teaching resources) are reported in table 3.   table 2 il skills, concepts, and understandings in teaching practice, by percentage, as well as academic professional value, on a point value of 1-4 il skill, understanding, or concept inclusion in teaching practice inclusion as a syllabus outcome teaching responsibility academic/ professional value   not relevant introduced/developed but not explicitly taught already utilized/understood by my students explicitly taught and assessed in my courses yes no i teach this skill/concept a librarian teaches this skill/concept both a librarian and i teach this skill/concept academic value professional value   percent average value selecting a topic with appropriate scope and according to available information 11 13 36 40 34 66 80 4 16 3.05 3.50 using search strategies, including keyword searching and advanced search features/search construction 12 27 36 25 18 82 53 19 28 3.11 3.56 knowing key databases, government websites, or other sources of information for the course/discipline 21 13 43 23 23 77 51 17 32 2.74 3.11 using a range of appropriate sources according to the topic and or/discipline 5 18 44 34 44 56 74 6 19 3.16 3.39 evaluating information 5 19 48 29 58 42 85 4 11 3.26 3.78 determining if a source is a peer-reviewed, scholarly article 25 9 34 32 25 75 62 2 36 2.68 3.22 reading a scholarly article (including the parts of a scholarly article) 18 11 35 36 42 58 83 4 13 3.00 3.35 using information ethically (appropriate citing, avoiding plagiarism, etc.) 2 24 31 43 51 49 85 4 11 3.63 3.82 protecting and valuing personal information (including issues of privacy, copyright and intellectual property, and ethically sharing and disseminating information) 17 21 39 23 31 69 88 3 9 3.37 3.65 synthesizing existing information resources to create a new information product (e.g., a literature review, a research paper, a class presentation, etc.) 10 12 38 40 59 41 92 0 8 3.05 3.29 utilizing different information creation formats such as audio, visual, and text, as well as use different platforms for information sharing including papers, presentations, social media, websites, or other information tools 14 19 41 25 33 67 89 3 8 2.89 3.00 understanding and participating in the information culture of the discipline or profession (i.e., how information is created, shared, and valued) 21 15 40 23 35 65 97 0 3 2.90 3.25 understanding the political, cultural, and economic aspects of information creation, access, and dissemination 37 12 36 14 23 77 100 0 0 2.74 2.88 reflecting on personal research and information creation habits and making changes based on those reflections 27 20 31 22 29 71 97 3 0 2.95 3.06   table 3 il learning resource awareness and use, by percentage il resource or service awareness use   not aware of this resource somewhat familiar with this resource very familiar with this resource i have used this resource in my teaching practice at fresno state   i have not used this resource in my teaching practice at fresno state     percent information literacy competency standards for higher education, association of college and research libraries (acrl) 65 34 1 8 92 framework for information literacy for higher education, acrl 68 28 4 5 95 information literacy value rubric, association of american colleges & universities (aac&u) 69 27 5 9 91 research/subject guides, madden library 14 49 38 59 41 library diy, madden library 35 42 23 39 61 information literacy modules in blackboard, madden library 41 37 22 29 71 assignment research calculator (arc), madden library 74 20 6 9 91 video tutorials, madden library 26 58 16 39 61 reducing plagiarism on campus workshop, ida m. jones & judith c. scott 23 45 33 33 68 midterm and finals write-in, writing center and madden library 37 44 20 27 73 academic success workshops: library skills, learning center 30 46 25 32 68 information literacy instruction/librarian-led instruction session, madden library 42 38 21 28 72 general library workshop (refer students to workshop not designed specifically for my course) 36 48 16 24 76 consultation with librarian subject liaisonassignment/syllabus design, madden library 26 44 30 35 65   focus group results   major themes discovered through analyses of focus groups transcripts include il skills, understandings, and beliefs; disciplinary differences in il and related pedagogical practices; and il practices. il skills, understandings, and beliefs includes the il attributes disciplinary faculty include within their teaching practice. disciplinary differences in il addresses areas where faculty identified how il differs within disciplines, and how it impacts il teaching and learning. il practices refers to specific methods, approaches, or learning theories faculty use within their il teaching practice. these major themes, with sub-themes, are reported in table 4.   table 4 il teaching practices focus groups themes and sub-themes theme sub-themes il skills, understandings, and beliefs evaluating information synthesizing information ethical use of information finding information general literacy (reading, writing) tools (e.g., research tools) using information like a scholar/professional fluency across platforms and with alternative formats/multiple formats social media/web 2.0 technologies navigating overabundance of information/overwhelming amounts of information developing agency through information 21st century skills critical thinking (critical reading) disciplinary differences in il the role of peer-reviewed literature authoritative knowledge il practices document/artifact (besides a research/term paper) oral presentation collaboration with librarians bloom’s taxonomy/scaffolding implicit, integrated methods   assessment report results   assessment reports were initially analyzed using voyant, and then qualitatively coded for il teaching practice insights. il-related terms and total counts are reported in table 5. note that these figures are for total mentions of these terms, and includes some noise discovered through the second phase of analysis (e.g., the term “database” generally referred to library resources, but a reference to a department contact information database was also included).   table 5 assessment report il-related terms and total count term count creat* 440 source* 241 citation|cite* 168 search* 93 information literacy 62 synthe* 45 database* 36 library 36 plagiar* 30 evaluate information~5 19 database 19 article peer~5 4   activities related to creating or synthesizing information into a new product (creat*, synthe*) were strongly represented in the assessment reports. information evaluation and the need for authoritative sources of information were also strongly represented. concerns about ethical use of information, such as appropriate citation practices and avoidance of plagiarism, were also well-represented.   overall results   combining these results reveals that disciplinary faculty highly value skills and understandings affiliated with il competency, and incorporate il into their teaching practice. disciplinary faculty consider il necessary for self-empowerment and agency in the information age, and are concerned about their students’ ability to navigate a complex information environment. faculty provide the majority of il learning opportunities independent of librarians, and are infrequently using broadly-available il teaching tools (such as national standards or rubrics) or locally-available il teaching tools (such as library research guides). il learning is generally provided through implicit, rather than explicit, methods. pedagogical methods that may enable explicit practices, such as the use of standards and competencies, il outcomes, and transparent lessons/assignments, are infrequently used.   discussion   from these results, several possibilities emerge around faculty il teaching practices.   value of il to faculty   the high value fresno state faculty place on il and associated skills and understandings confirms earlier work in this area, as mentioned in the literature review section above. faculty in particular value the higher order il skills, and view them as necessary for their students’ success not just in the classroom, but post-graduation, as well—in fact, they value il even higher post-graduation.   as bury (2016) and bury & sheese (2016) discuss, faculty are also questioning the ways in which other academic literacies interact with these higher-order il skills, and how they impact disciplinary understandings. in the focus groups, digital literacy, media literacy, critical thinking, and print literacy were discussed as being integral to il, overlapping with il, and even confusing in the distinction between these literacies. in terms of teaching practice, faculty expressed concern about how to develop exercises that can develop these multiple literacies, when the lines between the two are blurry: for example, being able to adeptly generate high quality and engaging content on social media (media and digital literacy), but needing more understanding about the social, political, and economic implications of the platform (il).   extremely important to il and the teaching practices of faculty is working to develop the ability to synthesize information into new works. several faculty referenced bloom’s taxonomy (anderson, krathwohl, & bloom, 2001) and the highest-level skill of creation: in the assessment reports, there were many assignments and exercises centered around the ability to work with existing literature to support new ideas and to be adept in creating new information products in multiple formats. in a study of faculty perceptions of il, bury (2016) discusses the potential for faculty engagement with the framework (association of college and research libraries [acrl], 2016) as a pedagogical approach. this interest in creation, which is emphasized in the framework (acrl, 2016), may signal an area of alignment between disciplinary faculty and librarians’ view of il.   faculty view il teaching as their responsibility   faculty very strongly identified with the teaching of il skills. in the survey and in the assessment reports, it became clear that faculty work with librarians to teach things like database demonstrations, citation style, and the basics of search, but all other il skills are viewed as their teaching responsibility. this might be viewed as a challenge: how can librarians be involved with the faculty’s work to develop higher order il skills in students, if disciplinary faculty view this mainly as their domain?   bury and sheese (2016) draw on the work of tamsin haggis (2006) and an approach to academic literacies that rejects a deficit approach in order to embrace a systemic one, in which the whole of an institution works together to develop student literacy through the core curriculum. this is an approach that has been used successfully for writing (bury & sheese, 2016). at fresno state, “writing across the disciplines” learning is formalized at all levels of the institution, including lower-division, upper division, disciplinary, and graduate writing requirements for all majors. this could serve as a potential model for il, where librarians serve as the pedagogical experts to help faculty develop methods for instruction in il at multiple places within the curriculum.   the implicit nature of faculty il teaching   faculty reported rarely including il outcomes in either their syllabi or their assignment descriptions. they were also surprisingly infrequent in assessment reports, where il is a core competency to be assessed with regularity. in instances where il was discussed, it was often listed as a singular outcome, i.e., “students will demonstrate information literacy.”  in the focus groups, they often talked of il learning as a process that occurs through osmosis, an underlying skill woven into writing or critical thinking assignments.   the tacit nature of expert practice is central to the threshold concepts behind the framework (acrl, 2016; hofer, townsend, & brunetti, 2012). there is perhaps potential for librarians to work with disciplinary faculty to identify these tacit practices and concepts, and use the tools of il to begin to name and identify these understandings in order to teach them to novice learners (townsend, hofer, hanick, & brunetti, 2016). in an institutional ethnography of il instruction, lafrance (2016) states, “librarians and [first year writing] faculty alike may willingly embrace a key term [information literacy] to demonstrate their desire to serve students and a campus community but may do so in ways that diverge from the pedagogical currents of national statements and more recent research-driven findings about effective practice” (p. 119). because il research has been generally siloed within library/information science research, these national standards and best practices may be exciting tools for professional development for disciplinary faculty. concepts taught, or desired to be taught, by fresno state faculty include evaluation, synthesis, and ethical use of information, all of which have rich pedagogical practice behind them within the library community. carrying these practices beyond the library may yield impactful results in il instruction outside the library.   implications for practice   evidence and findings from this study are being used to inform several initiatives. collaborations with the institutional research office, members of the library subcommittee of the academic senate, instruction librarians, and the center for faculty excellence are resulting in the development of new services, resources, and instruction models to support il competency in students. an il breakout session has been offered for two years at the new faculty orientation, and the lesson plan from this session is being used to design a mobile professional development course that can be offered to departments, colleges, and other units.   other methods to formalize and institutionalize il are also being explored, including a credit-bearing il course and inclusion of il in the general education curriculum.   limitations   while the design of this study attempted to address challenges related to self-reported data (survey and focus group results) through a triangulation approach also including assessment reports, limitations remain. access to additional documents related to faculty teaching and learning would have been ideal. this includes the use of syllabi, but syllabi are not systematically collected at fresno state. observational data and data from students would also be useful, and may lessen social desirability bias concerns for self-reporting of teaching practices (kopcha & sullivan, 2007), but privacy and academic freedom concerns limit access to these data sources.   while an aspect of this study focuses on the implicit il teaching practices of faculty, it must be acknowledged that il, writing, critical thinking, and many other literacies or academic competencies are sometimes difficult to delineate. for example, when reviewing assessment reports, the act of creating a “new” information product, such as a term paper or slide presentation, may be considered an information practice. however, the information skills necessary to successfully complete these assignments may also be treated as writing, oral communication, or visual literacy skills. while it is useful for librarians to view these assignments through an il lens, disciplinary faculty may view these from a more holistic perspective, and use tools, services, and resources outside the il domain to teach these skills. this may also limit what pedagogical practices faculty view as il practices. further research into this area may be particularly rich for librarians in understanding faculty il practices.   this study was undertaken as institutional research, and as such, the results are limited to a small sample size at a single institution, limiting the generalizability of the findings. the tools used to conduct this survey are free to use, however, and it is hoped that further research at other institutions will lead towards additional findings that can inform librarians’ support of their disciplinary faculty’s il teaching practices.   conclusion   investigating methods to increase librarians’ ability to support student development of higher order il skills, the il teaching practices of disciplinary faculty were investigated. using a data/methods triangulation approach, a survey of faculty was administered; focus groups were held; and assessment reports were analyzed. the goal of this approach was to discover how faculty are currently implementing il into their teaching practice; what standards and other il resources they are using to do so; and how they value il for their students, both within the classroom and post-graduation. while many studies have investigated disciplinary faculty perceptions of il, this study adds to a very small body of literature by situating these perceptions not in the value of librarians teaching il, but through faculty’s own perceptions of il within their teaching work. this is particularly relevant to the goals of the framework, which emphasizes collaboration and the transdisciplinary nature of il (acrl, 2016). to do this work, librarians must understand how the framework and il fits into the nature of their partner faculty’s existing teaching practices. an even smaller body of literature has attempted this investigation within the framework context (see dawes, 2019 and dubicki, 2019).   results revealed that faculty value il highly, particularly for students post-graduation, and that disciplinary faculty view il as within their teaching domain. this confirms prior research into the value and teaching responsibility of il (bury, 2011; saunders, 2012; weiner, 2014). additional findings extend this work by discovering that faculty may not be aware of or use il standards or resources to help make il skills and concepts explicit. in addition, while previous studies confirmed that faculty view il instruction as their domain, this study provides new information into how disciplinary faculty do this work, through acceptance and application of existing il resources, il learning outcomes, and assessment activities of il. as much of this work is revealed to be implicit, disciplinary faculty may be intrigued by the potential of threshold concepts within il as well as their discipline, towards developing expertise in research and information use for their students.   these findings provide evidence for several ways forward for librarians at fresno state to support disciplinary colleagues’ il teaching practices, including the creation of an il-credit bearing course; workshops and other services and resources to develop disciplinary faculty’s tools for il teaching; and working with disciplinary faculty to formalize il teaching towards promoting student il development.   acknowledgements   this study was funded by the office of institutional effectiveness faculty fellows program, initiated by dr. angel sanchez. several oie staff were essential in conducting this study, including matthew zivot, chris hernandez, marie tongson-fernandez, and azucena rodriguez. graduate student shauna dauderman was a key collaborator in designing and co-administering the faculty focus groups, transcribing focus group interviews, cleaning up initial survey data, and conducting initial analyses, including creating tableau visualizations. other members of the 2017/2018 oie faculty fellows cohort are also much appreciated for their feedback and recommendations which strengthened this study. the research services unit of henry madden library also contributed useful feedback in the design of the faculty il practices survey. amanda dinscore of henry madden library is also acknowledged for her support of the administration of the faculty il practices survey and study.   references   anderson, l. w., krathwohl, d. r., & bloom, b. s. (eds.) (2001). a taxonomy for learning, teaching, and assessing: a revision of bloom's taxonomy of educational objectives (complete ed.). new york: longman.   association of college and research libraries. (2016). framework for information literacy for higher education. retrieved from http://www.ala.org/acrl/standards/ilframework   brems, c. (1994). taking the fear out of research: a gentle approach to teaching an appreciation for research. teaching of psychology, 21(4), 241-243. https://doi.org/10.1207/s15328023top2104_10   bruce, c., somerville, m. m., stoodley, i., & partridge, h. (2014). diversifying information literacy research: an informed learning perspective. in c. bruce, k. davis, h. hughes, h. partridge, & i. stoodley (eds.), information experience: approaches to theory and practice (pp. 169-186). bingley, uk: emerald group publishing limited.   bury, s. (2011). faculty attitudes, perceptions and experiences of information literacy: a study across multiple disciplines at york university, canada. journal of information literacy, 5(1), 45-64. https://doi.org/10.11645/5.1.1513   bury, s. (2016). learning from faculty voices on information literacy: opportunities and challenges for undergraduate information literacy education. reference services review, 44(3), 237-252. https://doi.org/10.1108/rsr-11-2015-0047   bury, s., & sheese, r. (2016). academic literacies as cornerstones in course design: a partnership to develop programming for faculty and teaching assistants. journal of university teaching and learning practice, 13(3), 1-18. https://ro.uow.edu.au/jutlp/vol13/ iss3/3   california state university. (2016). csu local admission and service areas. retrieved from https://www2.calstate.edu/apply/freshman/documents/csulocaladmission-serviceareas.pdf   california state university. (n.d.). history. retrieved from https://www2.calstate.edu/csu-system/about-the-csu/pages/history.aspx dacosta, j. w. (2010). is there an information literacy skills gap to be bridged? an examination of faculty perceptions and activities relating to information literacy in the united states and england. college & research libraries, 71(3), 203-222.    https://doi.org/10.5860/0710203   dawes, l. (2019). through faculty's eyes: teaching threshold concepts and the framework. portal: libraries and the academy, 19(1), 127-153. https://doi.org/10.1353/pla.2019.0007   detlor, b., julien, h., willson, r., serenko, a., & lavallee, m. (2011). learning outcomes of information literacy instruction at business schools. journal of the american society for information science & technology, 62(3), 572-585. https://doi.org/10.1002/asi.21474   dubicki, e. (2019). mapping curriculum learning outcomes to acrl's framework threshold concepts: a syllabus study. the journal of academic librarianship, 45(3), 288-298. https://doi.org/10.1016/j.acalib.2019.04.003   emmons, m., keefe, e. b., moore, v. m., sánchez, r. m., mals, m. m., & neely, t. y. (2009). teaching information literacy skills to prepare teachers who can bridge the research-to-practice gap. reference & user services quarterly, 49(2), 140-150. https://doi.org/10.5860/rusq.49n2.140   everett, j. b. (2010). a study of faculty teaching of information literacy in alabama's public associate's colleges (unpublished doctoral dissertation). university of alabama, tuscaloosa, alabama.   gardner, c. c., & white-farnham, j. (2013). “she has a vocabulary i just don’t have”: faculty culture and information literacy collaboration. collaborative librarianship, 5(4), 235-242. https://digitalcommons.du.edu/collaborativelibrarianship/vol5/iss4/3   gonzales, a. h., mcmillen, p., & fabbi, j. l. (2009). new avenues for integrating information literacy into the curriculum. in i. gibson, r. weber, k. mcferrin, r. carlsen, & d. a. willis (eds.), proceedings of society for information technology & teacher education international conference 2009 (pp. 1639-1644). chesapeake, va: association for the advancement of computing in education. retrieved from https://digitalscholarship.unlv.edu/lib_articles/301/   haggis, t. (2006). pedagogies for diversity: retaining critical challenge amidst fears of ‘dumbing down.’ studies in higher education, 31(5), 521-535. https://doi.org/10.1080/03075070600922709   hofer, a. r., townsend, l., & brunetti, k. (2012). troublesome concepts and information literacy: investigating threshold concepts for il instruction. portal: libraries & the academy, 12(4), 387-405. https://doi.org/10.1353/pla.2012.0039   julien, h., & given, l. m. (2002). faculty-librarian relationships in the information literacy context: a content analysis of librarians' expressed attitudes and experiences. canadian journal of information & library sciences, 27(3), 65-87.   kopcha, t., & sullivan, h. (2007). self-presentation bias in surveys of teachers’ educational technology practices. educational technology research and development, 55, 627-646. https://doi.org/10.1007/s11423-006-9011-8   lafrance, m. (2016). an institutional ethnography of information literacy instruction: key terms, local/material contexts, and instructional practice. wpa: writing program administration, 39(2), 105-123. https://associationdatabase.co/archives/39n2/ 39n2lafrance.pdf   leaning, m. (2017). media and information literacy (pp. 33-51). cambridge, massachussets: chandos publishing.   leckie, g. j., & fullerton, a. (1999). information literacy in science and engineering undergraduate education: faculty attitudes and pedagogical practices. college & research libraries, 60(1), 9-29. https://doi.org/10.5860/crl.60.1.9   lwoga, e. t. (2013). faculty perceptions and practices in health sciences information literacy instruction in tanzania. library philosophy and practice. https://digitalcommons.unl.edu/libphilprac/1017   mackey, t. p., & jacobson, t. e. (2005). information literacy: a collaborative endeavor. college teaching, 53(4), 140-144. https://doi.org/10.3200/ctch.53.4.140-144   mounce, m. (2010). working together: academic librarians and faculty collaborating to improve students' information literacy skills: a literature review 2000-2009. the reference librarian, 51(4), 300-320. https://doi.org/10.1080/02763877.2010.501420   nilsen, c. (2012). faculty perceptions of librarian-led information literacy instruction in postsecondary education. paper presented at the world library and information congress: 78th ifla general conference and assembly, helsinki. retrieved from https://www.ifla.org/past-wlic/2012/105-nilsen-en.pdf   owusu-ansah, e. k. (2004). information literacy and higher education: placing the academic library in the center of a comprehensive solution. the journal of academic librarianship, 30(1), 3-16. https://doi.org/10.1016/j.jal.2003.11.002   saunders, l. (2012). faculty perspectives on information literacy as a student learning outcome. the journal of academic librarianship, 38(4), 226-236. https://doi.org/10.1016/j.acalib.2012.06.001   smith, r. l. (2006). philosophical shift: teach the faculty to teach information literacy. american library association association for college and research libraries. retrieved from http://www.ala.org/acrl/publications/whitepapers/nashville/smith   thompson, g. b. (2002). information literacy accreditation mandates: what they mean for faculty and librarians. library trends, 51(2), 218-241. retrieved from https://www.ideals.illinois.edu/bitstream/handle/2142/8462/librarytrendsv51i2h_opt.pdf   townsend, l., hofer, a. r., hanick, s. l., & brunetti, k. (2016). identifying threshold concepts for information literacy: a delphi study. communications in information literacy, 10(1), 23-49. https://doi.org/10.15760/comminfolit.2016.10.1.13   veach, g. l. (2009). teaching information literacy to faculty: an experiment. college & undergraduate libraries, 16(1), 58-70. https://doi.org/10.1080/10691310902753983   weetman, j. (2005). osmosis—does it work for the development of information literacy? journal of academic librarianship, 31(5), 456-460. https://doi.org/10.1016/j.acalib.2005.05.007   weiner, s. a. (2014). who teaches information literacy competencies? report of a study of faculty. college teaching, 62(1), 5-12. https://doi.org/10.1080/87567555.2013.803949   wscuc. (2016). educational quality: student learning, core competencies, and standards of performance at graduation. handbook of accreditation 2013 revised. retrieved from https://www.wscuc.org/resources/handbook-accreditation-2013/part-iii-wasc-quality-assurance/institutional-report/components-institutional-report/4-educational-quality-student-learning-core-competencies-and-standards-performance   microsoft word es_bogel.doc evidence based library and information practice 2006, 1:4      49 evidence based library and information practice       evidence summary    school libraries play an active, transformational role in student learning and  achievement      a review of:  todd, ross j. “student learning through ohio school libraries: a summary of the ohio     research study.” ohio educational library media association    15 dec. 2003. ohio educational library media association (oelma), 2004. 15   nov. 2006  <http://www.oelma.org/studentlearning/documents/oelmaresearchstudy8page.pdf>.    reviewed by:  gayle bogel  director, learning resources and information technology, john read middle school  redding, connecticut  united states of america  e‐mail: gbogel@gmail.com      received: 21 august 2006            accepted: 15 november 2006      © 2006 bogel. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – this study explored links  between school libraries and student  learning outcomes that were defined in a  multidimensional context, using data  provided by the students themselves.  the  researchers examined learning outcomes  that reached beyond the existing  correlations of school library services and  standardized test scores.  insight was  provided into the interactions between  students and school libraries that affect  student learning. an overarching goal of the  study was to establish ongoing dialogue to  focus on evidence based practices that may  lead to continuous improvement in school  library services and to provide the basis for  further research.    design – web based survey.    subjects – participants were 13,123 students  in grades 3‐12 and 879 faculty at 39 schools  across the state.    setting – ohio public school libraries.     methods – thirty‐nine effective school  libraries, staffed by credentialed school  librarians, were chosen through a judgment  sampling process, using criteria based on  http://www.oelma.org/studentlearning/documents/oelmaresearchstudy8page.pdf http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:4      50 ohio guidelines for effective school library  media programs. the guidelines are aligned  to academic content standards, assessments,  resources, and professional development.    two web based surveys were used to collect  quantitative and qualitative data from  students and faculty:   1. the impacts on learning survey,  composed of likert scale responses  to 48 statements and an open‐ended  critical incident question for  students.  2. the perceptions of learning  impacts survey was a similar  survey for faculty.       survey questions were based on dervin’s  theory of information seeking that advances  the idea of ‘helps’ as the constructive  process of bridging gaps in information use  that lead to new knowledge or making sense  (sense‐making) in relation to a perceived  information need (todd and kuhlthau). the  term ‘helps’ includes both inputs (help that  the school library provides in engaging  students in learning) and outputs (learning  outcomes of academic achievement and  active agency in the learning process).   the survey statements included a   combination of conclusions based on  selections from school library research  studies, and the information literacy  standards for student learning from the  american association  of school librarians’  information power: building partnerships  for learning.      the two surveys were used to triangulate  multiple sources of data to illustrate the  “helps” provided by the school library to  student learning. students were also given  the opportunity to describe “helps” in their  own words in an open‐ended critical  incident question.    main results – the data showed that the  selected effective school libraries were  perceived as providing ‘helps’ in dynamic  ways that appeared to have a transformative  effect on student learning.  school libraries  and librarians were viewed as having an  active role in the learning process.  of the  students surveyed, 99.4 % believed that  school libraries helped them become better  learners.  the results were grouped into  seven blocks of ‘help’ concepts that frame  the contributions of the school library and  the school librarian to student learning  (table 1).       help concept  summary of  survey statements  block 1  find and locate information.  “how helpful the school library is with getting  information you need.”   block 2  use information to complete  school work.  “how helpful the school library is with using the  information to complete your school work.”  block 3  school work in general.  “how helpful the school library is with your  school work in general.”    block 4  using computers in the library,  at school, and at home.  “how helpful the school library is with using  computers in the library, at school, and at home.”  block 5  general reading interests.  “how helpful the school library is to you with  your general reading interests.”  block 6  outside school  “how helpful the school library is to you when  you are not at school.”       block 7   general school aspects/reactions  “now, some general things.”    table 1. todd’s help concepts  evidence based library and information practice 2006, 1:4      51 the study noted that perceptions of the  effect of school libraries are strongest for  elementary students, and perceptions of the  effect decrease as students move through  middle and high school. comments from  students indicate that mastery of  information skills that lead to independent  learning may contribute to the perception  that the library is not as strong a ‘help’ in  later school years.      in ranking the mean scores of the block  concepts, the effective school library ranked  strongest as a resource agent and technical  agent, to support student research and  projects with both print and non‐print  resources. the qualitative data further  clarified student perceptions that the library  contributed to individualized learning,  knowledge construction, and academic  achievement.  instructional interventions  that benefited from contributions by the  librarian included conducting research  effectively; identifying key ideas; analyzing,  synthesizing, and evaluating information;  and developing personal conclusions.      in comparing student and faculty data, there  was consistency in the perception of value in  the top three conceptual groups of “helps”:  libraries as resource agents, as agents for  information literacy development, and as  agents for knowledge construction.      conclusion – the data analysis illustrated  that school libraries were actively involved  in student learning and were perceived as  important factors in student learning and  achievement by both students and faculty.   consistency throughout the sample showed  perceptions of multiple effects of school   libraries in facilitating student learning for  building knowledge.      student comments and survey results  showed that students perceived the library  as providing strong support for reading for  curriculum and informational needs and as  less helpful with regard to individual  reading for pleasure or personal pursuits.  the study speculates that perhaps the  emphasis on academics and test‐oriented  schooling may leave students little time to  pursue independent reading interests  during school hours.    the study identified factors for effective  school libraries: informational,  transformational, and formational elements.  these factors may be used as building  blocks for shaping practices that help  effective school libraries bring about student  achievement.     • informational: resources,  technological infrastructure, and  reading resources.  • transformational: information  literacy, technological literacy, and  reading engagement.  • formational: knowledge creation,  use, production, dissemination,  values, and reading literacy.     the visual model of the factors for effective  practice and their relationship to student  outcomes will be of particular help to  practitioners. (todd and kuhlthau 23)    commentary      the methodology of this study was a  departure from earlier research that used  statistical modeling to draw correlations  between student achievement and the  presence of school libraries (lance, proof of  the power).  the qualitative survey was  used to gather data from the students  themselves and to define and illustrate  interactions between students and school  librarians, providing a starting point for  future studies.       in an interview with lance for school library  media research, callison noted that the  surveys used in this study assessed only  evidence based library and information practice 2006, 1:4      52 positive responses to the perceptions of help.   todd and kuhlthau had earlier discussed  their choices in constructing the likert scale  survey in the report findings (4), clarifying  that although they considered using a  negative response (rather than the “”does  not apply” option), they ultimately decided  that their study was not about whether  school libraries help students, which had  already been established by repeated  statewide studies that had determined a  positive correlation between the presence of  school libraries and student achievement  (lance, proof of the power), but was rather  an attempt to look more deeply at how  school librarians actively affect the learning  process. they chose to use the surveys to  focus closely on best practices by school  librarians to “understand much more richly  how effective school librarians do help  students by identifying and elaborating the  “helps” construct” (todd and kuhlthau 4),  and illustrate the multidimensional effects  of school librarians as active agents in the  learning process.    although the researchers described the  student sample as being large in comparison  to other school library research studies, they  did not provide details about how the data  was collected, other than to say that the  survey was available online.  other data not  given in the report was the overall size of  the student population in the 39 schools, the  percentage of total students who responded  to the survey, and the number of student  respondents who were regular library users.     the enormous strength of the study is that it  very effectively achieved its stated objective  to provide qualitative, descriptive evidence  of the multidimensional effects of  interventions by school librarians.  it offered  insight into the activities that build  knowledge, contribute to student learning,  and affect student achievement. the results  provide scaffolding for further research.   devising methods to empirically measure  the effect on student outcomes is a next  logical step.  research to quantify the  measurable impact of the ‘help’ constructs  may provide data that encompasses the  broader dimensions explored by the  researchers, and may at the same time  provide data that aligns and substantiates  the narrower measures of standardized tests.      the identification of a model to use in the  pursuit of continuous improvement based  on best practices is a useful tool for  practitioners.  the visual representation of  the essential learning foundations (todd  and kuhlthau 23) for informational,  formational, and transformational building  blocks is an effective way to help translate  the research results into daily practice.  the  model also serves as a framework for  practitioners and future researchers to  design investigations into the effect of  school libraries on student learning  outcomes.     defining the specific interactions between  students and librarians that promote  knowledge building provides a perspective  for the learning potential in school libraries,  and a broader, substantial definition of  school librarians as partners in the  educational process.      works cited    american association of school  librarians/association for educational  communications and technology.   information power: building  partnerships for learning. chicago:  american library association, 1998.    dervin, brenda. “from the mind’s eye of  the user: the sense‐making  qualitative‐quantitative  methodology.” qualitative research in  information management.    eds. jack  d. glazier and ronald r. powell.  evidence based library and information practice 2006, 1:4      53 englewood, co: libraries unlimited,  1992. 61‐84.    lance, keith. proof of the power: recent  research on the impact of school  library media programs on the  academic achievement of u.s. public  school students. eric clearinghouse  on information and technology  syracuse, ny. eric digest. 2001‐00‐00.  eric document reproduction service.  ed456861. 15 nov. 2006   <http://www.eric.ed.gov/ericdocs/dat a/ericdocs2/content_storage_01/0000000 b/80/2a/33/ac.pdf>.  lance, keith curry, and daniel callison.  “enough already?: blazing new trails  for school library research: an  interview with keith curry lance.”  school library media research. 2005.  19 nov. 2006.  <http://www.ala.org/ala/aasl/aaslpubsa ndjournals/slmrb/editorschoiceb/lancei nterviewlane.htm>.   “ohio guidelines for effective school  library media programs.” ohio  department of education. 17 july 2006  <http://www.ode.state.oh.us/gd/templ ates/pages/ode/odedetail.aspx?page= 3&topicrelationid=340&content=1726 7>.                            todd, ross j. “how effective are effective  school libraries?: students’  perspectives from ohio.” orana 40.1  (mar. 2004): 9‐21. 19 nov. 2006  <http://www.alia.org.au/publishing/ora na/40.1/ross.todd.html>.    todd, ross j., and carol c. kuhlthau.  “student learning through ohio  school libraries: background,  methodology, and report of findings.”  ohio educational library media  association. 2004. ohio educational  library media association. 2 dec. 2006      <http://www.oelma.org/studentlearnin g/documents/oelmareportoffindings .pdf>.        http://www.eric.ed.gov/ericdocs/dat http://www.ala.org/ala/aasl/aaslpubsa http://www.ode.state.oh.us/gd/templ http://www.alia.org.au/publishing/ora http://www.oelma.org/studentlearnin microsoft word es_loy.doc evidence based library and information practice 2007, 2:1  140 evidence based library and information practice       evidence summary    electronic journals appear to reduce interlibrary lending in academic libraries      a review of:  wiley, lynn and tina e. chrzastowski.  a. “the impact of electronic journals on interlibrary  lending: a longitudinal study of state‐wide interlibrary loan article sharing in  illinois.” library collections, acquisitions & technical services 29.4 (dec. 2005): 364‐81.    reviewed by:   john loy   learning resources manager  avon & wiltshire mental health partnership nhs trust, callington road hospital  brislington, bristol, uk  e‐mail: john.loy@awp.nhs.uk      received: 28 november 2006    accepted: 10 january 2007      © 2007 loy. this is an open access article distributed under the terms of the creative commons attribution  license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and  reproduction in any medium, provided the original work is properly cited.      abstract    objective – to determine the impact of  electronic journals on interlibrary loan (ill)  activity. the hypothesis predicted that ill  requests would fall by approximately 10%  during a four‐year period, that e‐journal use  would increase by 10% per year and that  there would be a correlation between the  two.    design – longitudinal data analysis of  interlibrary loans over an eight year period  from 1995 to 2003. the second part of the  study is a retrospective data analysis of e‐ journal use from 2001‐2005.    setting – the 26 largest libraries in the state  of illinois, usa; all but the chicago public  library are academic institutions.  subjects –  1. journal article photocopy requests  originating in the 26 libraries divided  into three data sets: 1995/96, 1999/00 and  2002/03.   2. electronic journal usage statistics from  25 libraries subscribing to packages  within the ebscohost database for the  fiscal years 2001‐2005.    methods –a retrospective analysis was  conducted using interlibrary loan data for  journal article photocopy requests either  originating from or being satisfied by the 26  libraries in the study. it examined the data  in three ways: the 26 libraries together,  requests sent to libraries in the state of  illinois excluding the 26, and requests using  libraries outside the state. the second part  of the study examines usage data of  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:1  141 electronic journals available in 25 of the 26  libraries.    main results – in the period from 1999 to  2003 a reduction in ill requests of nearly  26% was observed within the participating  26 libraries.     analysis by broad subject discipline  demonstrates that social sciences and  sciences show the largest drop in requests –  a 25% decrease from 1995‐2003. the number  of requests from an individual journal title  drops significantly in science by 34% within  the state and by 37% for out‐of‐state  requests.    while the humanities actually showed an  increase in the number of requests, the large  increase in out‐of‐state requests (20.6%  overall between 1995 and 2003) slowed  significantly with an increase of only 2.6%  from 1999‐2003 indicating that sources other  than ill are providing articles to this field.    nearly identical peaks and troughs in ill  requests over the three study periods  demonstrate predictably consistent high and  low use subject areas.    use of the e‐journals collection was shown  to increase at well over 10% per year.    of the most highly requested ill titles, 46%  were available as e‐journals, indicating a  significant lack of awareness or inability to  access electronic resources among some  library users.    conclusion – the hypothesis that state‐wide  ill requests would decline by 10% was far  surpassed. libraries most frequently  borrowed titles that were low‐use and  outside the scope of their collections. titles  requested more than 20 times in each study  period were those least frequently borrowed,  as well as least requested from outside the  state, which demonstrates a cost‐effective  use of library resources. this indicates that  libraries are judiciously providing access to  high‐use titles locally. all three data sets  included in‐state titles requested more than  20 times, as well as 18 titles requested from  out of state, suggesting that they should be  considered for purchase within illinois.  while access to e‐journals appears to have  reduced the number of ills, there is clearly  a need for some libraries to improve the way  in which they help their users access the  collection.    commentary    the authors take a popular view – the more  e‐journals, the fewer the ill requests – and  place it under a bright spotlight.  this is the  third article the authors have published on  the subject, all of which examine the  developing picture in illinois, so they are  clearly very familiar with the territory.  broadly speaking the hypothesis is upheld  and ill requests do drop significantly. this  is in line with similar, previous studies. in  miami, the louise calder memorial library  experienced a drop in interlibrary loan  activity of over 30% in a five year period  following the increase in availability of e‐ journals (burrows).      while it is reassuring when research  upholds our professional instincts and  observations, does this study actually  provide the evidence to uphold its  conclusions? while a great many of the  authors’ conclusions are likely, they  themselves state “it is not possible to  definitively correlate the predicted  statewide reduction of ill requests to  specific holdings of electronic full‐text  journals” (367). certainly the two coincide,  and are probably related, but we are never  presented with hard and fast evidence to  that effect.     it is important to remember that much of the  research into collection management is very  evidence based library and information practice 2007, 2:1  142 context‐specific, and to date the majority of  the evidence comes from case studies. as  with this article many of the case studies are  university‐based, and if approaching the  article from outside academia it will  probably not take long for the “yes, but…”  factor to come into play. the users of  university libraries are likely to be engaged  in active research with a clear goal and to be  reasonably familiar with electronic  resources. yet if 46% of the most highly  requested journals are available as e‐ journals, and they are not being accessed  electronically by university users in this  study, what are the chances of the more  casual library user getting to grips with the  demands of electronic access?    the reason interlibrary loans are dropping  in this study is likely that the users are  finding the full‐text for themselves. the  importance of links between bibliographic  databases and full‐text e‐journal collections  is also emphasised here and the authors  rightly call for us to provide a more  connected service to our users in terms of e‐ journals and databases: “libraries often  represent aggregations of journals together  as a package, not splitting out individual  titles and therefore making them nearly  impossible for patrons to find” (378).  research has shown that online journal  collections without links from bibliographic  databases demonstrate lower levels of use  (de groote).     there are a great many charts, figures and  tables, and graph fatigue does set in after a  while as the brain struggles to interpret the  information being presented. at times the  results are confusingly laid out and you are  obliged to flick back and forth through the  pages, jumping from table to chart to graph  and attempt to marry them to the text in  order to keep up. time and again the  authors present these statistics suggesting  that there is a link between the two, but that  evidence is not present.    what i continue to struggle with, even after  repeated reading, is section 6 detailing the  analysis of the electronic journal access. are  the e‐journal titles selected by the libraries  or a pre‐packaged bundle provided by  esbcohost? while the latter seems likely  this is not explicit in the article. several  readings in and i am still not clear quite how  the authors arrive at some of the figures in  this section on e‐journals. what exactly are  the 7300 access points they refer to here? i  longed to be part of an audience to ask  questions for clarity.     a recently published study provides useful  complementary reading, exploring similar  themes around the impact of e‐journals, but  this time within the healthcare setting of the  uk’s national heath service (nhs) and  well away from universities (crudge).  among the conclusions is that at present the  national nhs e‐journal collection – a pre‐ packaged bundle – does not meet the  current awareness reading needs of  healthcare staff. while the e‐journal  collection does include many high‐impact  factor titles, there is a very real need for the  local print holdings in libraries to  supplement this in order to meet the needs  of staff.     so while the authors present a great deal of  food for thought, do they actually present us  with evidence? well, it does seem…very  likely…probably…?    works cited    burrows suzetta. “a review of electronic  journal acquisition, management, and  use in health sciences libraries.”  journal of the medical library  association 94.1 (jan. 2006): 67‐74.    crudge, sarah e., and mary l. hill.  “electronic journal provision in a  health‐care library: insights from a  consultation with nhs workers.”  evidence based library and information practice 2007, 2:1  143 health information & libraries journal  23.2 (jun. 2006): 87‐94.    de groote, sandra l. and josephine l.  dorsch. “measuring use patterns of  online journals and databases.”  journal of the medical library  association 91.2 (apr. 2003): 231‐40.      microsoft word es_1782 evidence based library and information practice 2008, 3:3 73 evidence based library and information practice evidence summary academic librarians have concerns about their role as teachers a review of: walter, scott. “librarians as teachers: a qualitative inquiry into professional identity.” college and research libraries 69.1 (2008): 51-71. reviewed by: virginia wilson shirp coordinator, health sciences library, university of saskatchewan saskatoon, saskatchewan, canada e-mail: virginia.wilson@usask.ca received: 30 may 2008 accepted: 4 july 2008 © 2008 wilson. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – this study explores how academic librarians are introduced to teaching, the degree to which they think of themselves as teachers, the ways in which being a teacher has become a significant feature of their professional identity, and the factors that may influence academic librarians to adopt a “teacher identity.” design – a literature review extended by qualitative semi-structured interviews. setting – the research took place at an american university with the carnegie foundation for the advancement of teaching designation of “doctoral/research — extensive.” subjects – six academic librarians. methods – the main feature of the article is an extensive literature review around the themes of lis, teaching, and qualitative research methodologies. the literature review is supplemented by qualitative research consisting of semi-structured interviews of between 45 and 90 minutes each, which were conducted during spring 2004 with six librarians (five women and one man), whose length of professional service ranged from 2 to 32 years. all of the participants worked at the same institution. the data collected were reviewed throughout the process using field memos and a research log. the data were analyzed using a coding process where discrete ideas evidence based library and information practice 2008, 3:3 74 that emerged from the data were used to identify a small number of themes. the initial conclusions in the study were validated through member checking during the writing phase. “member checking involves sharing draft study findings with the participants, to inquire whether their viewpoints were faithfully interpreted, whether there are gross errors of fact, and whether the account makes sense to participants with different perspectives” (centre for health evidence). main results – five themes around teaching and teacher identity as they pertain to academic librarians emerged from the data. the first theme was the centrality of teaching. each participant sought out a position where the teaching role was valued. the role of teacher spilled over into the other roles of the librarian, i.e., reference service, collection development, etc. the next theme was the importance of collegial and administrative support, which is critical to the ability to focus on work as a teacher. the stress of multiple demands emerged as a theme, as time dedicated to teaching was often at the expense of something else. another theme was the problems with professional education around teaching. instruction course offerings in library schools were reported to be meagre, and some were badly planned and executed. the fifth theme involved stereotypes and misperceptions. studies have shown that the academic library profession has been poorly understood by students and faculty. study participants believed that many of their campus colleagues were either unaware of what they did, or were misinformed by popular culture stereotypes of librarians. conclusions – the small sample size precluded the making of any definite conclusions based on the study results. other limitations of the study include the relatively short amount of time spent in the interview process and the narrow range of librarians chosen to participate. the author notes that a subject pool more representative of academic librarians’ full range of opinions regarding the importance of teaching as a professional responsibility would have resulted in more complex themes emerging. while the author is aware of the study’s limitations, he feels there is value in the qualitative research design, in giving voice to individual librarians, and in the provision of insight into some of the research questions found in the literature of learning to teach and of teacher identity. given the limitations, walter makes three conclusions about his findings. he points out the lack of a formal introduction to teaching in many library programs which has been explored by other studies and concludes that his study “suggests that continuing lack of attention to this issue results in a difficult introduction into the profession for new academic librarians” (64). regarding continuing and professional education, walter concludes that “this study suggests that there are a number of important questions about the content and conduct of these opportunities for instruction librarians that have not been explored in the literature” (64). finally, walter concludes that “this study suggests that there is an important connection between research on student perceptions of academic librarians, the study of teacher identity, and the future of the profession” (64). commentary the author is forthcoming about the limitations of his small sample size and defines his study as exploratory in nature. he also notes that the participants were chosen for their strong commitment to their role as teachers. the value of the themes that emerged from the interviews with study participants lies largely in the notion that they are a starting point to further research evidence based library and information practice 2008, 3:3 75 into the area of teacher identity among librarians. what is perhaps more valuable in this paper is the extensive use of the research literature to inform the various ideas throughout. the literature review is robust, and the author includes results from previous studies all through the paper to strengthen his statements and conclusions. the author examined the library literature as well as the literature around teaching and education. for anyone interested in this topic, the notes section alone is an excellent resource. additionally, the article includes the interview questions used. the author talks about the value of qualitative research design and analysis for the field of librarianship. he cites a study that noted that academic librarians’ voices are more likely to be aggregated in study results, and individual voices are often not heard. the qualitative methodology can alleviate this issue by giving individuals a chance to speak and be heard while still offering conclusions that can be informative in a broader context. that said, the author himself only uses the voices of his study participants five times throughout the paper. using more quotes from the research results, quoting the librarians’ own voices, would have strengthened the rather tentative conclusions given by the author, and would have reinforced the value of the qualitative research design. this article is useful for academic librarians interested in exploring the development of a teacher identity as part of their professional practice. the article also suggests further research avenues that might be explored, such as the perceived lack of a formal introduction to teaching in lis education programs, an analysis of the content areas included as part of formal coursework on instruction, and the impact on professional development among teaching librarians of more focused attention to the notion of teacher identity. evidence based library and information practice evidence based library and information practice 2011, 6.4 137 evidence based library and information practice evidence summary collaborative learning is an effective method for improving the e-health literacy of older adults in the community a review of: xie, b. (2011). older adults, e-health literacy, and collaborative learning: an experimental study. journal of the american society for information science and technology, 62(5), 933-946. doi: 10.1002/asi.21507 reviewed by: theresa s. arndt associate director for library resources & administration waidner-spahr library, dickinson college carlisle, pennsylvania, united states of america email: arndtt@dickinson.edu received: 01 sept. 2011 accepted: 31 oct. 2011 2011 arndt. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to determine whether collaborative learning strategies in an informal class setting can improve electronic health literacy skills of older adults. design – preand post-test instruments used to measure effects of an educational intervention. setting – small group classes offered at two branches of a large, publicly funded, urban public library in maryland. subjects – a total of 111 adults aged 52 to 91, mean age 70.4 (sd 8.0), completed the study. the majority of participants were from minority populations (66% african american, 3% latino, 3% asian). thirty three percent of participants reported an annual household income below $20,000. eight percent were non-native english speakers. the majority of participants had low-level or no computer/internet experience prior to the study. methods – collaborative learning strategies were used in small group hands-on computer classes to deliver a standardized curriculum (helping older adults search for health information online: a toolkit for trainers from the national institute on aging). strategies employed were: explicit statement of group/participatory nature of class, periodic peer shared reflection times during class, evidence based library and information practice 2011, 6.4 138 active encouragement of discussion between peers, hands-on work with partners, group discussion of real-life questions from participants, and structured shared reflection time at the close of each session. participants were recruited through local advertisements. no incentive other than the free classes was offered. groups met for two hours, twice a week for four weeks. assessment was via pre and post-tests. general computing knowledge/skills were measured using objective tests of abilities. questions from several established scales were adapted for additional assessment. e-health literacy was measured using questions of perceived skill and comfort in finding health information online; perceived usefulness of the internet for help making health decisions; and perceived importance of the internet for obtaining health information. subjects were also asked to report on changes to their health behaviour/decision making post intervention, and learning effort expended during the study. additional questions measured psychological adjustment to later life, attitudes toward computers, attitude toward the aging experience, and attitude toward the collaborative learning method and the class. main results – a dependent t-test analysis indicated strong significant gains post-test in computing/internet knowledge and skills, and in e-health literacy efficacy (perceived skills/comfort with using the internet for health information and decision-making). pre-test results showed participants to be low on these measures, leaving much room for improvement. perception of the usefulness and importance of the internet for health decision-making also showed strong gains. significant positive changes were also found in these areas of attitude: reduction of computer anxiety, attitude toward physical changes associated with aging, and improvement in attitude toward the collaborative learning method. a majority of participants indicated altering health-related behaviours as a result of the class, including changing the way they think about diet or exercise, changing the way they cope with a condition, and changing their approach to maintaining health. the results showed no significant change in self-esteem, self-efficacy, and psychological attitudes toward aging. computer interest and efficacy also showed no significant change, perhaps because participants already measured high in these prior to the intervention. the amount of time participants spent preparing for class correlated significantly to e-health literacy efficacy and perceived importance of internet health information, but not to other outcome measures. group composition (gender, peer familiarity, prior computer experience) did not affect outcomes; however composition was uneven, as groups were small and variable in size, and 71% of participants were female. conclusion – the study supports the use of a collaborative learning approach to effectively deliver e-health literacy instruction to older adults in a community setting. commentary the author has made a valuable contribution to the literature on methods for educating older adults for improved e-health literacy. briefly, e-health literacy refers to the effective retrieval and appropriate application of electronically delivered health information for personal use. the introduction and literature review give a concise summary of the research on collaborative learning techniques. the author highlights significant gaps in the literature, particularly in regards to collaborative learning efficacy with older adult populations outside a formal educational setting. this study begins to address that gap by testing the efficacy of incorporating collaborative learning techniques into public library classes for older adults. evidence based library and information practice 2011, 6.4 139 methodological limitations (which the author notes) include the lack of randomization in subject selection and lack of a control group. additional weaknesses are that 4.8% of participants admitted to the study were younger than the “60 and above” age specified in the study plan, and the study had a dropout rate of 35%, though the author notes that the demographics of the group as a whole remained fairly consistent. statistical analysis using a dependent t-test is appropriate for a study without a control group. the improvements in both general computer literacy and e-health literacy measures as a result of the intervention were quite strong. participant self-reports of the impact on their health related behaviours and decision-making are also compelling. noting the low amount of preparation outside of class, the author speculates that participants may have had limited access to computers outside of class time, and recommends provision of extra-curricular access for future interventions. this study has particularly valuable implications for outreach librarians and public health educators. the author provides a useful introduction to the collaborative learning method, with which educators of adult populations may not be familiar. it is notable that the standardized toolkit for trainers from the national institute on aging, while using other active learning techniques, incorporates very little true collaborative learning as detailed in this article. busy practitioners may undertake educational efforts without any rigorous assessment and with impact taken largely on faith. this study supports the value of providing and funding such efforts, particularly for low-income and minority older adults, groups especially at risk of having low health literacy. / evidence based library and information practice evidence based library and information practice 2012, 7.4 114 evidence based library and information practice evidence summary interlibrary loan rates for academic libraries in the united states of america have increased despite the availability of electronic databases, but fulfilment rates have decreased a review of: williams, j. a., & woolwine, d. e. (2011). interlibrary loan in the united states: an analysis of academic libraries in a digital age. journal of interlibrary loan, document delivery & electronic reserve, 21(4), 165-183. doi: 10.1080/1072303x.2011.602945 reviewed by: kathryn oxborrow team leader hutt city libraries lower hutt, new zealand email: kathryn.oxborrow@huttcity.govt.nz received: 1 june 2011 accepted: 5 oct. 2011 2012 oxborrow. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objectives – to determine the number of interlibrary loan (ill) requests in academic libraries in the united states of america over the period 1997-2008, and how various factors have influenced these rates. these factors included electronic database subscriptions, size of print journal and monograph collections, and the presence of link resolvers. data were collected from libraries as both lenders and borrowers. the study also looked at whether the number of professional staff in an ill department had changed during the period studied, and whether ill departments led by a professional librarian correlated positively with rates of ill. design – online questionnaire. setting – academic library members of the online computer library center (oclc) ill scheme in the united states of america. subjects – a total of 442 academic library members of the oclc ill scheme. methods – an electronic questionnaire was sent to 1433 academic library member institutions of the oclc ill scheme. data were collected for libraries as both lending and borrowing institutions. data were analyzed using a statistical software package, specifically to calculate spearman’s rank mailto:kathryn.oxborrow@huttcity.govt.nz http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/ evidence based library and information practice 2012, 7.4 115 correlations between the variables and rates of ill. main results – responses to the electronic questionnaire were received from 442 (31%) academic libraries. there was an overall increase in the number of ill requests in the period 1997-2008. the number of ill requests which were unfulfilled also increased during this period. there was a positive correlation between rates of ill and all of the variables investigated, with the strongest correlations with size of print monograph collections and size of print journal collections. the numbers of staff in ill departments remained relatively static during the period covered by the study, although the majority of staff working in ill was composed of paraprofessionals. there was a weak positive correlation between numbers of ill requests and whether ill departments were headed by a professional librarian. conclusions – access to full text electronic databases has not decreased the numbers of ill requests in academic libraries in the united states of america. in fact, ill requests have increased, probably due to the fact that students and staff of academic libraries now have access to a larger number of citations through online databases and other information sources. the authors suggest that the increase in unfulfilled ill requests is also due to this increased access. libraries with large print collections are more likely to receive ill requests precisely because they have more material to lend out, and may make more ill requests due to the research output of their presumably larger institutions. there may be a higher number of ill requests fulfilled by departments headed by a professional librarian because a librarian has more knowledge of sources to fulfil requests. commentary this study was done as part of a previously conducted, larger scale study in the area of ill research, and the authors give a good summary of earlier literature. it aimed to discover whether a series of factors correlated positively with numbers of ill requests conducted over a number of years in the united states of america. the data covers a wide range of scenarios: separate datasets were recorded for ill requests made and fulfilled, and for libraries as borrowers and as lenders. the presentation of the data makes it difficult for the reader to interpret the results. the authors use a coding system to represent ranges of numbers of ill requests. all of the figures and tables are presented in this way, so the reader must continually refer back to the coding, and in some cases the code numbers are divided into decimal points, meaning the reader must calculate for themselves what the actual figures are. the coding is also misleading as the numerical range covered by each code number varies widely. the scales used on the graphical representations of the results also vary, which makes it very difficult to compare among them. the results of this study do not give the reader the full story. although a great number of institutions were surveyed, every institution did not answer every question, and there is no explanation of this in the text. furthermore, information is not given about the size and distribution of the institutions surveyed. this makes some of the conclusions which the researchers draw somewhat shaky, as they do not discuss how these factors may also have a bearing on the results. there are other aspects which the researchers did not cover in their conclusions, such as the fact that the presence of link resolvers may facilitate the ill application process, and that non-fulfilment in many cases may be due to library users’ inability to find items in the collection. this is an interesting research area, and the authors suggest that further study could be undertaken into the declining numbers of ill requests being fulfilled. further studies could also look at the staffing question touched upon by the authors, such as a smaller scale comparison of the success rates of professionally qualified staff and paraprofessionals in the fulfilment of ill requests. a study similar to that carried out by bernardini & mangiaracina (2011), who evidence based library and information practice 2012, 7.4 116 investigated the types of items that are being requested by ill after the introduction of subscription bundles in italy, would also be an interesting addition to the literature. ill is a rapidly changing area of librarianship, particularly with continuing technological advances such as e-books, so it is an area which requires continual research. implications for practice include the importance of providing information literacy training to library users to ensure they can access the available material. references bernardini, e., & mangiaracina, s. (2011). the relationship between ill/document supply and journal subscriptions. interlending and document supply, 39(1), 9-25. doi: 10.1108/02641611111112101 news   call for applicants for eblip journal: evidence summaries writers    2023. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30364     eblip seeks to add several writers to the evidence summaries team. evidence summaries are critical appraisal syntheses, which provide analysis regarding the validity and reliability of the methodology used in an original research article. as such, they are a key component of eblip to aid readers in making informed decisions in their local practice. evidence summaries team members are required to write two evidence summaries per year, with a two-year commitment to the journal. evidence summaries cover all areas of library and information studies, and we encourage applications from information professionals in areas such as school, public, and special libraries, as well as academic settings. interested persons should send a cover letter, indicating areas of strength they would bring to the role, and resume/cv as a single pdf file to fiona inglis (associate editor, evidence summaries) at finglis@wlu.ca by july 15, 2023. applicants who are shortlisted will be asked to submit a sample evidence summary. *please note that evidence based library and information practice is a non-profit, open access journal and all positions are voluntary and unpaid. the positions are an excellent opportunity for continuing professional development and gaining experience in reviewing and critically appraising library-related research. **only those applicants who are selected or shortlisted will be contacted by the editors. about the journal:   published quarterly and hosted by the university of alberta, this peer-reviewed, open access journal is targeted at all library and information professionals interested in an evidence based model of practice. by facilitating access to librarianship research via original research articles and evidence summaries of relevant research from the library literature, evidence based library and information practice enables librarians to practice their profession in an evidence based manner. please visit the evidence based library and information practice web site (https://journals.library.ualberta.ca/eblip/index.php/eblip) for further information about the journal.   evidence based library and information practice 2012, 7.2 76 evidence based library and information practice evidence summary undergraduate information literacy instruction is not enough to prepare junior doctors for evidence based practice a review of: cullen, r., clark, m., & esson, r. (2011). evidence-based information-seeking skills of junior doctors entering the workforce: an evaluation of the impact of information literacy training during pre-clinical years. health information & libraries journal, 28(2), 119-129. doi:10.1111/j.14711842.2011.00933.x reviewed by: carol d. howe reference librarian/associate professor gabriele library, immaculata university immaculata, pennsylvania, united states of america email: chowe@immaculata.edu received: 9 feb. 2012 accepted: 1 may 2012 2012 howe. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to determine if junior doctors entering the workforce retain the information literacy skills they learned as undergraduates. design – structured interviews and observations. setting – wellington medical school of the university of otago in new zealand. medicine in new zealand is an undergraduate program. subjects – thirty-eight university of otago students who were starting their fourth year of undergraduate medical training between 1994 and 2004. at the time of this study, the students had graduated and were a number of years into advanced training for their speciality, i.e., junior doctors. the participants represented five cohorts, each having received a different level of information literacy instruction as undergraduates. cohort 1, with the most years in clinical practice at the time of the study, received no formal information literacy instruction as undergraduates. cohorts 2 to 5 received information literacy instruction in their fourth undergraduate year. the focus of instruction for cohorts 2 and 3 was on developing an effective search strategy, whereas the instruction for cohorts 4 and 5 focused more on the critical appraisal of articles. mailto:chowe@immaculata.edu evidence based library and information practice 2012, 7.2 77 methods – in 2008 and 2009, the authors contacted cohort graduates. two medical librarians from the wellington medical library interviewed and observed participants to establish their level of information literacy. the librarians asked an initial six questions to determine how much participants remembered of their undergraduate information literacy instruction, how they search for clinical information, what databases they use, how they evaluate information, and if they have had any formal or informal information literacy instruction since graduating. for question seven, participants described a recent situation in which they searched for clinical information relating to a given patient. for question eight, participants rated their own skill level as “no skills”, “some skills”, or “highly skilled” on the following seven parameters: choosing a source to search, brainstorming search terms, using boolean operators, using database limits, finding randomized controlled trials (rcts) and systematic reviews, using “explode” and “focus”, and evaluating articles. for the last question, the librarians observed and rated participants as they conducted a search in realtime. the librarians rated participants’ performance on the same parameters as question eight using the same scale of “no skills”, “some skills”, or highly skilled”. main results – most participants said they remembered or at least vaguely remembered the information literacy instruction they received in their fourth year. the authors noted that most participants seemed to have expanded on the skills they originally learned as undergraduates. participants reported using a variety of information sources such as pubmed and cochrane but were often unsure about what constitutes a database. a large number indicated that they use google to find information. rarely had participants asked a medical librarian for help with online searching. the authors define evaluation as either intrinsic (based on information contained in the article itself) or extrinsic (based on such criteria as a journal’s reputation or its publisher). all the cohorts, even cohorts 4 and 5 who received the most instruction on critical appraisal, relied more or equally on extrinsic factors than on intrinsic factors. when asked if they had received further formal or informal information literacy instruction since their fourth undergraduate year, most participants in cohorts 1 and 2 said they had. fewer participants in cohorts 3, 4, and 5 indicated they had received further instruction. the participants on average rated themselves highest on using database limits and lowest on using “explode” and “focus”. the observers on average rated the participants highest on choosing a source to search and lowest on finding rcts and systematic reviews as well as using “explode” and “focus”, which tied for the lowest rating. the observed searches on average were rated lower than the selfassessments on all but one parameter. none of the average scores for either the self-rating or the observer-rating approached “highly skilled”. conclusion – the authors concluded that the information literacy instruction the participants received as undergraduates did not prepare them adequately for evidencebased practice. even though most participants said they remembered their undergraduate information literacy instruction, neither the average scores for the self-rating nor the observer-rating approached “highly skilled”. from that they could surmise that the attainment of information literacy should be a career-long learning process, beginning with undergraduate instruction and extending throughout one’s clinical practice. the authors also found that the level of instruction cohorts received as undergraduates did not seem to correspond to their current ability. cohort 1, who received no information literacy instruction as undergraduates, scored higher on average than cohorts 3 and 4 on the self-assessment and higher than cohorts 3, 4, and 5 on the observer assessment. cohort 1 also used more evidence based sources than did cohort 4, who received the most training on evidence-based medicine. evidence based library and information practice 2012, 7.2 78 cohorts 1 and 2 reported the most postgraduate information literacy instruction, leading the authors to postulate that the further along one is in his medical career, the more important evidence based practice, and thus information literacy instruction, becomes. even with additional instruction, however, the participants did not seem prepared for evidence-based practice. the authors concluded that information literacy instruction during postgraduate training and clinical practice—possibly giving the doctor’s specialty consideration when designing instruction— might be more important than undergraduate instruction. they also concluded that information literacy instruction might be more effective when its importance is emphasized by senior clinicians. commentary the ability to make well-informed clinical decisions is an indispensable skill for doctors to have. the importance of this study is highlighted when one considers how lack of training on how to practice evidence based medicine can affect patient outcomes. by comparing the participants’ self-ratings with the observers’ ratings, the authors found that the participants thought more highly of their skills than was warranted. this could create a hazardous situation in which doctors do not fully explore the body of evidence available to them. the more one uses a skill, the more one perfects it. this supports the notion that, not just medical students, but also practicing doctors, should regularly exercise their information literacy skills. many participants indicated that they had had further information literacy instruction since they were undergraduates, which is promising. this might explain the authors’ observation that most participants seemed to have expanded on the skills they were originally taught as undergraduates and also the inconsistency in skill level among cohorts. the validity of this study is weakened by several flaws. first, the sample size was small, and all participants came from the same university. secondly, the ten-year time span between cohorts made comparison among them questionable, especially considering that the web was new in 1995 and commonplace by 2004. moreover, the content and search interfaces of the tools participants reported using had probably changed quite a bit over ten years. the participants were asked how much they remembered of their undergraduate information literacy instruction, but they were asked anywhere from 4 to 14 years after the fact. this introduced the potential for inaccurate recollections. finally, there are many discrepancies between data reported in the text of the article and its tabular representation. for example, the findings section states that 27 participants consulted a librarian ‘occasionally’ or ‘rarely’, whereas table 3 indicates that only two participants asked a librarian for help. despite its weaknesses, this study is important because it underscores the need for future research of this type. it would be interesting to see results from medical students in countries other than new zealand, for example. also, since there is a growing emphasis on and new resources for evidence-based medicine, it would be helpful to see the results from a more current study. given the authors’ finding that junior doctors are insufficiently prepared for evidence based practice, further research is critical to highlight the importance of careerlong information literacy learning. research in practice   evidence, local context, and the hierarchy   virginia wilson director, centre for evidence based library and information practice (c-eblip) university library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   received: 11 nov. 2015 accepted: 18 nov. 2015      2015 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   a key piece of evidence based library and information practice (eblip) is evidence (obviously!). it’s a word that is a bit loaded or has been in the past. what counts as evidence? does the term “evidence” automatically suggest quantitative research? in early days, as eblip emerged from evidence based medicine, some purveyors and users of research evidence privileged quantitative research—hard numbers generalizable across large populations. the research evidence hierarchy was espoused as the model to follow, with randomized controlled trials (rcts) at the peak, the pinnacle of the pyramid, and case studies often forming the large lower layer or base of the pyramid. in later years, systematic reviews and meta-analyses deposed rcts from the top of the hierarchy.   the idea of a hierarchy of evidence as koufogiannakis outlined in 2010 is problematic for various reasons as it pertains to eblip. i believe a hierarchy of evidence doesn’t make sense for one particular reason: local context. eblip consists of four components that must be present: research evidence, professional expertise and knowledge, user preference, and the local context. each component should be explored, examined, and acknowledged when approaching a practice problem in an evidence based manner. i suggest envisioning these components as something like figure 1.   if you do not acknowledge all of the elements, it’s not truly eblip, and you won’t have utilized everything at your disposal in order to make a decision or solve a problem. if everything is placed against the backdrop of the local context, as in figure 1, then how can a hierarchy of evidence be effective? the best systematic review to be found pertaining to your particular question might not be applicable if it does not resonate with the way things are currently configured in your local setting. that’s why critical thinking and critical appraisal of whatever evidence is found is so important to the eblip process.     figure 1 components of eblip     in eblip, moving away from the rigid early hierarchy of evidence, the idea of research evidence has broadened to include and value all types of research: qualitative, quantitative, mixed methods, practical, theoretical, participatory—you name it; it’s included. additionally, evidence arising from professional knowledge and expertise and from user preference has also come to the fore in terms of acceptability. of course, it must be acceptable if you refer to the venn diagram in figure 1.   research conducted by koufogiannakis looks at this very issue, and she reveals that librarians are using a wide variety of evidence to inform their practice. koufogiannakis contends that “the focus of eblip over the past 15 years has neglected to incorporate . . . parts of what the movement in fact defines itself to include, namely the user-reported and librarian observed forms of evidence” (2011, p. 42). the evidence used in pursuit of eblip when approaching a problem or a decision in practice depends upon the nature of the problem and the question you need to answer. this question arises from where you work, your local setting. if you leave out your own expertise developed over time and the needs, wants, or desires of your users when approaching your question, you’ve left out two-thirds of eblip. and then if you apply some kind of arbitrary hierarchy to the research evidence you’ve found, the focus becomes the hierarchy itself rather than the local.   so, if you’re going to look through a lens when practicing eblip (and i would argue that lenses are ubiquitous no matter where we go and what we do), choose the local context as your lens. if we can keep the idea front and centre that we must not let our own settings slip out of sight when approaching a practice question, we will realize that the best evidence is the evidence that supports and informs our practice.   references   koufogiannakis, d. (2010). the appropriateness of hierarchies. evidence based library and information practice 5(3): 1-3. retrieved from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/8853/7348   koufogiannakis, d. (2011). considering the place of practice-based evidence within evidence based library and information practice (eblip). library and information research 35(111): 41-58. retrieved from http://www.lirgjournal.org.uk/lir/ojs/index.php/lir/article/view/486/527     evidence based library and information practice 2012, 7.3 32 evidence based library and information practice article exploring publishing patterns at a large research university: implications for library practice kathleen amos project manager, council on linkages between academia and public health practice public health foundation washington, district of columbia, united states of america email: kamos@phf.org allyson mower scholarly communications & copyright librarian marriott library university of utah salt lake city, utah, united states of america email: allyson.mower@utah.edu mary ann james electronic resources manager marriott library university of utah salt lake city, utah, united states of america email: maryann.james@utah.edu alice weber interprofessional education librarian eccles health sciences library university of utah salt lake city, utah, united states of america email: alice.weber@utah.edu joanne yaffe associate professor of social work college of social work university of utah salt lake city, utah, united states of america email: joanne.yaffe@utah.edu mary youngkin librarian emerita eccles health sciences library university of utah salt lake city, utah, united states of america email: mary.youngkin@utah.edu mailto:kamos@phf.org mailto:allyson.mower@utah.edu mailto:maryann.james@utah.edu mailto:alice.weber@utah.edu mailto:joanne.yaffe@utah.edu mailto:mary.youngkin@utah.edu evidence based library and information practice 2012, 7.3 33 received: 17 april 2012 accepted: 8 aug. 2012 2012 amos, mower, james, weber, yaffe, and youngkin. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – the research project sought to explore the value of data on publication patterns for decision-making regarding scholarly communications and collection development programs at a research-intensive post-secondary institution, the university of utah in the united states. methods – publication data for prolific university of utah authors were gathered from scopus for the year 2009. the availability to university of utah faculty, staff, and students of the journals in which university of utah authors published was determined using the university of utah libraries’ catalogue; usage was estimated based on publisher-provided download statistics and requests through interlibrary loan; and costs were calculated from invoices, a periodicals directory, and publisher websites and communications. indicators of value included the cost-per-use of journals to which the university of utah libraries subscribed, a comparison of interlibrary loan costs to subscription costs for journals to which the university of utah libraries did not subscribe, the relationship between publishing venue and usage, and the relationship between publishing venue and cost-per-use. results – there were 22 university of utah authors who published 10 or more articles in 2009. collectively, these authors produced 275 articles in 162 journals. the university of utah provided access through library subscriptions to 83% of the journals for which access, usage, and cost data were available, with widely varying usage and at widely varying costs. cost-per-use and a comparison of interlibrary loan to subscription costs provided evidence of the effectiveness of collection development practices. however, at the individual journal title level, there was little overlap between the various indicators of journal value, with the highest ranked, or most valuable, journals differing depending on the indicator considered. few of the articles studied appeared in open access journals, suggesting a possible focus area for the scholarly communications program. conclusions – knowledge of publication patterns provides an additional source of data to support collection development decisions and scholarly communications programming. as the estimated value of a journal is dependent on the factor being studied, gathering knowledge on a number of factors and from a variety of sources can lead to more informed decision-making. efforts should be made to expand data considered in areas of scholarly communications and collection development beyond usage to incorporate publishing activities of institutionally affiliated authors. evidence based library and information practice 2012, 7.3 34 introduction several librarians and researchers have written about scholarly communications and the current state of flux as digital technology proliferates (battin, 1980; byrd, 1990; odlyzko, 1997; shaughnessy, 1989; tenopir & king, 1997, 2000). in response to these changes, many libraries have implemented scholarly communications programs to raise awareness, provide services, and advocate for changes in costs of and access to scientific, technical, and medical (stm) journals (bergman, 2006; malenfant, 2010). the term “scholarly communications” encompasses the myriad ways in which scholars communicate their research findings and original creations to their peers and others. a library scholarly communications program must respond to a range of disciplines and their communication media and formats: journal articles, books, paintings, music, etc. knowing what programming, services, content, and collections a library needs to provide requires a deep understanding of unique, discipline-specific scholarly communication and publishing patterns. publishing choices can indicate potential journal value. as schauder (1994) has said, “to be a scientist or scholar it is necessary to write and to publish: it is not as essential to command a wide readership, or a contemporary readership” (p. 82). such publishing choices can also demonstrate attitudes towards non-traditional approaches to journal article dissemination such as selfarchiving and open access publishing. attitudes regarding these venues play a primary role in the development of a scholarly communications program. if scholars, as one study has found, do not see or remain unaware of pricing or access problems in the journal publishing system, library programs and services need to respond accordingly to those realities (rowlands, nicholas, & huntington, 2004). this paper reports on initial research regarding one mode of scholarly communication – the journal article – and explores journal publishing patterns, rates of access, and costs at a research-intensive university for the purposes of informing a scholarly communications program. this type of research can also inform collection development decisions. as library journal budgets continue to constrict, collection development practices may require rethinking, especially regarding evidence based decisionmaking. collection development, “the process of planning and building a useful and balanced collection of library materials” (reitz, 2004-2012), is a core function of librarians in many professional settings. a number of factors may contribute to the decision to select or deselect a particular journal (dhawan, phull, & jain, 1980) and a variety of sources are potentially useful for guiding this type of activity, including knowledge of the journal publishing practices of faculty and research staff at the institution, journal download statistics, and interlibrary loan (ill) requests. this paper demonstrates the potential value of these approaches and provides usage and cost analyses for the university of utah (uu). literature review a limited number of studies have looked at publishing patterns as important data for evaluating library collections, but no literature was located that reflects the use of publishing pattern data to develop scholarly communications programs. some authors have demonstrated the use of citation analysis to evaluate how libraries meet the information needs of a local campus. for example, dykeman (1994) determined citation studies to be “one indicator of a collection’s strength or weakness” after looking at the publishing output of science and engineering faculty at a technical institute (p. 145). labonte (2005) analyzed publishing patterns as well as citations to determine the library’s journal collection in response to a newly created research institute, starting with faculty publication decisions, but focusing mostly on journals cited in these publications. lascar and mendelsohn (2001) identified 12 key researchers, searched web of science to determine where they published, and then reviewed the references cited in these evidence based library and information practice 2012, 7.3 35 publications. the authors discovered only a few newer journals cited by researchers that the library did not hold and used these data to justify adding them to the collection, arguing that “collections must be focused on the needs of the researchers as indicated by their citation and publication patterns” (p. 432). other authors have suggested that while citation analysis provides important information for collection decisions, publication patterns should be given greater weight. for example, salisbury and smith (2010) concluded that usage statistics alone do not always reflect the true use and impact of a journal and that different types of data greatly improve the ability to assess a library’s collection, suggesting that libraries “at a minimum … provide access to the periodicals in which … researchers published more than five times” in addition to considering the number of citations (p. 76). hughes (1995) obtained bibliographies from an academic department, developed a list of journals, and searched citations by faculty in scisearch and journal impact factor ranking in journal citation reports (jcr). hughes gave faculty publication patterns the greatest weight in journal subscription decisions, arguing that publication patterns indicated the journal’s importance to faculty for both quality and readership. wilson and tenopir (2008) suggested that libraries should consider using a combination of publishing patterns and local citation analysis, as well as survey data on reading and writing productivity, in order to analyze the quality of a journal. these studies indicate a range of decision criteria used by libraries to inform journal collection development decisions. while many libraries rely primarily on citation analysis for decisions, some libraries utilize usage and ill statistics, and a few incorporate publishing patterns into decision-making. emerging social tools such as altmetrics – metrics for measuring the impact of scholarship based on the social web (priem, taraborelli, groth, & neylon, 2010) – could provide another source of information beyond citation analysis. none of the articles located focused on evaluating costs in relation to the usage of the journals or on potential ways such metrics can influence a scholarly communications program. the present study aims to contribute to this gap in knowledge related to libraries’ potential use of faculty publishing patterns and data about the costs associated with accessing journals in which faculty publish, in order to inform collection development decisions and scholarly communications efforts. aims this research builds on that of previous authors by considering a university’s scholarly communications program and library journal subscriptions from the perspective of the publication patterns of university faculty and staff. the study explores the availability of journal articles published by prolific authors at the uu, as well as the associated usage and costs of access. specifically, the research addresses the following seven questions with respect to the journals in which prolific authors publish: 1. what proportion of subscription journals is accessible to researchers at the uu through institutional holdings? 2. how often are these journals used at the uu? 3. what are the costs to the uu of providing access to these journals? 4. how do the costs of these journals relate to their usage at the uu? 5. do the journals in which authors publish most frequently have the highest usage? 6. do the journals in which authors publish most frequently provide the best value for the money spent? 7. what proportion of these journals is published under an open access model? methods this study was based on a list of journals in which prolific uu authors published in 2009. to identify journals in which uu authors published, the scopus database was used. scopus was selected for its size and coverage of multiple disciplines; the other widely used, evidence based library and information practice 2012, 7.3 36 multidisciplinary citation database, web of science, was not available at the uu at the time this research began. using the affiliation search function, a search was conducted for “university of utah” and the results were manually reviewed to identify entities affiliated with the uu. the database was searched for all articles published by authors listing one of these entities in their affiliation (for the full search string, see appendix a). search results were limited to items published in 2009, as this was the most current year for which access and cost data were available. prolific authors were defined as those publishing 10 or more articles in a given year, in this case 2009. the threshold of 10 or more articles helped create a manageable sample size for an initial exploration. scopus allows search results to be filtered based on a variety of factors, including author, and displays the number of items that will be retrieved if a filter is applied. search results were filtered to identify uu-affiliated authors who had published 10 or more articles in 2009. the list of publications for each of these authors was exported, and the journals in which the publications appeared were identified. all scopus searches were conducted in february 2010. access access to journals was assessed based on holdings listed in the uu libraries’ online catalogue. a catalogue search was conducted in early 2010 for each journal to determine whether the uu held an active subscription to the title during the 2009 calendar year. journals publishing under an open access model were identified from entries in the uu libraries’ sfx link resolver. usage to determine usage of the journals to which the uu libraries subscribed, journal download statistics for 2009 were collected, either directly from publishers or from aggregators. usage for the uu is determined by internet protocol (ip) address ranges and includes on-campus access as well as remote access by authenticated users. both pdf and html downloads were counted in the usage statistics. for journals to which the uu libraries did not subscribe, usage was estimated based on ill requests. the number of ill requests processed for university-affiliated faculty, staff, or students in 2009 for each journal was obtained from statistics maintained by the ill departments of the uu libraries. costs for each journal, the annual subscription cost for 2009 was determined. several sources for establishing costs were available, and the source used depended on whether the uu libraries subscribed to a journal. for journals to which the uu libraries subscribed, invoices were reviewed to identify the cost paid. approximately 70% of the journals in this sample were obtained through journal packages or consortium-based purchases, and individual prices could not be identified by means of invoices. invoices for packages often do not itemize costs by journal title, and title prices within a package can vary significantly, limiting the utility of dividing the total package cost by the total number of journals. if it was not possible to identify the cost of a journal through invoices, the data were collected using, in order of preference, ulrich’s periodicals directory, journal publishers’ websites, or personal communications with publishers or vendors. ulrich’s served as a single source and efficient means of checking prices for several journal titles. when ulrich’s did not list a journal or its price, the publisher’s website was consulted. if pricing information was not included on the publisher’s site, an email was sent to a publisher representative or vendor requesting the information. estimations of annual subscription costs for journals to which the uu libraries did not subscribe were based on the same sources, consulted in the same order, with the exception of uu libraries’ invoices. the costs of providing journal access through ill were estimated using a standard cost of $30 per ill request (jackson, 2003). for each evidence based library and information practice 2012, 7.3 37 journal, the number of ills processed for university-affiliated faculty, staff, or students in 2009 was multiplied by this standard cost estimate to provide an approximate ill cost. value after estimating journal costs and usage, these two factors were compared. a cost-per-use value was approximated by dividing the estimated subscription cost by the number of downloads for each journal to which the uu libraries subscribed. for journals to which the uu libraries did not subscribe, the estimated ill cost was compared to the subscription price of the journal to identify journals for which purchasing a subscription might be more cost-effective than providing access through ill. in addition, choice of publishing venue was compared to journal usage and to the estimated cost-per-use values. results this exploratory study focused on the most prolific uu authors in 2009, as defined by number of published articles. the “university of utah” affiliation search in scopus returned 59,807 publications, 3,275 of which were published in 2009. there were 22 universityaffiliated authors who met the criterion for inclusion, which was defined as publication of 10 or more articles in 2009. collectively, these authors published 275 articles in 162 journals. a complete list of journals, along with the number of articles published by prolific uu authors in each journal, appears in appendix b. access, usage, and costs of these journals were investigated. complete data on these three factors were available for 150 of the journals, containing 254 of the articles identified. the following analysis is based on those 150 journals. access of the 150 journals studied, 125 (83%) were accessible to university-affiliated researchers through library subscriptions (figure 1). the 125 journals that were accessible to university-affiliated researchers through subscription contained 217 (85%) of the 254 articles published by the university’s most prolific authors in 2009 (figure 2). the remaining 37 (15%) of these articles appeared in journals that were not accessible through the uu libraries. usage usage of the 125 journals to which the uu libraries subscribed varied widely, from a high of 42,208 article downloads to a low of figure 1 access to journals in which prolific university of utah authors published in 2009; n = 150 evidence based library and information practice 2012, 7.3 38 zero downloads during 2009 (table 1). mean usage was 2,157 downloads per journal, while median usage was 365 downloads per journal. the most frequently used journal in this sample was nature, followed by the journal of the american chemical society with 32,792 downloads, science with 29,943 downloads, and proceedings of the national academy of sciences of the united states of america (pnas) with 28,061 downloads. articles from two journals were not downloaded at all during 2009, and four additional journals were each downloaded only once. the majority (89) of the journals in this sample were downloaded less than 1,000 times each, while only six journals had more than 10,000 downloads. ill statistics were collected for the 25 journals to which the uu libraries did not subscribe (table 2). of these, 19 (76%) were requested at least once during 2009, while the remaining 6 (24%) were not requested. there were 184 ill requests processed for the 25 journals, for an average of 7 requests per journal (mdn = 5 requests per journal). there were 10 or more requests each for 7 (28%) of these journals. of the journals that were requested, the number of requests per journal ranged from a high of 44 requests for toxicon to a low of 2 requests each for the american journal of therapeutics, computing and visualization in science, and congenital heart disease. figure 2 access to articles published by prolific university of utah authors in 2009; n = 254 table 1 number of article downloads for subscribed journals; n = 125 number of article downloads maximum 42,208 mean 2,157 median 365 minimum 0 evidence based library and information practice 2012, 7.3 39 costs subscription costs were estimated for the 125 journals to which the uu libraries subscribed. cost estimates varied significantly from a high of $12,370 to a low of $107 (table 3). the cost of 1 (1%) of the journals was more than $10,000, while 66 (53%) journals cost more than $1,000 each. in total, the cost paid to access these journals was valued at approximately $270,250, an average of $2,162 per journal (mdn = $1,160 per journal). for the 25 subscription journals not purchased by the uu libraries, the subscription cost was estimated to total approximately $31,300, an average of $1,252 per journal (mdn = $653 per journal). costs again varied, from $62 to $5,039 (table 4). the subscription costs of 9 (36%) of the journals were estimated at more than $1,000 each. for the subscription journals to which the uu libraries did not subscribe, ill costs were also estimated (table 4). the cost of ill in 2009 for these 25 journals amounted to approximately $5,520, with individual journal costs ranging from $0 to $1,320 depending on the number of requests processed. the average ill cost per journal was estimated at approximately $221 (mdn = $150). value cost and usage a comparison of cost to usage was used to assess value for the journals made accessible to the uu research community through library subscriptions. as was the case with cost and downloads, the calculated cost-per-use values varied widely for 123 of the 125 subscribed journals that were used, from a high of $6,713 to a low of $0.01 (m = $1.00) (table 5). the costs-per-use of 3 of the journals were greater than $1,000 each, while the majority of journals (85) had costs-per-use of less than $10 each. the cost-per-use of 31 (25%) of the journals was estimated at $1 or less. cost-per-use was not calculated for the two journals that were not downloaded at all during 2009. a comparison of ill costs, which reflect usage, to subscription costs was used to assess value for journals to which the uu libraries did not subscribe. it was estimated that the ill costs paid by the uu libraries in 2009 were higher than the costs of the corresponding journal table 2 number of ill requests for non-subscribed journals; n = 125 number of requests maximum 44 mean 7 median 5 minimum 0 table 3 estimated costs for subscribed journals; n = 125 estimated subscription costs maximum $12,370 mean $2,162 median $1,160 minimum $107 total $270,250 evidence based library and information practice 2012, 7.3 40 subscriptions for two (8%) journals. for an additional three (12%) journals, the equivalent of more than half of the subscription costs was spent in providing ill. the ill costs of 18 (72%) of the journals were equal to less than 25% of the costs for those journals, had they instead been purchased by subscription. for the journal requested most frequently, toxicon, the ill cost was estimated to be $1,320, approximately 42% of the estimated subscription cost. publishing venue and usage of the journals to which the uu libraries subscribed, the largest number of articles was table 4 comparison of estimated subscription and ill costs for nonsubscribed journals; n = 25 estimated subscription costs estimated ill costs maximum $5,039 $1,320 mean $1,252 $221 median $653 $150 minimum $62 $0 total $31,300 $5,520 table 5 estimated cost-per-use for subscribed journals; n = 123 cost-per-use maximum $6,713 mean $1 minimum $0.01 table 6 top 10 subscribed journals ranked by number of articles by prolific uu authors and rank by usage in 2009; n = 125 title number of articles rank by usage (high to low) astrophysical journal 13 51st aip conference proceedings 7 86th journal of physical chemistry b 7 9th american journal of obstetrics and gynecology 6 19th journal of the american chemical society 6 2nd biochemistry 5 10th inorganic chemistry 5 20th pediatric critical care medicine 4 75th physical review b condensed matter and materials physics 4 16th physical review d particles, fields, gravitation and cosmology 4 85th evidence based library and information practice 2012, 7.3 41 published in astrophysical journal, followed by aip conference proceedings, and the journal of physical chemistry b. astrophysical journal ranked 51st out of 125 journals in usage by uu faculty, staff, and students, with 738 downloads in 2009, while aip conference proceedings ranked 86th with 140 downloads (table 6). three journals were highly ranked in both number of uu articles published and number of downloads: the journal of the american chemical society (ranked 2nd in downloads), the journal of physical chemistry b (ranked 9th), and biochemistry (ranked 10th). the most frequently used journal, nature, contained two articles written by prolific uu authors. publishing venue and cost-per-use with respect to cost-per-use, astrophysical journal ranked 68th out of 123 journals, with an estimated cost of $3.75 per download in 2009. aip conference proceedings ranked 103rd at $38.57 per download, and journal of physical chemistry b ranked 33rd, with a cost-per-use of $1.15. the journal of the american chemical society was the only journal ranked highly in both number of uu articles and cost-per-use, at $0.13 per download, the fifth-lowest cost overall (table 7). the lancet, the journal with the lowest cost-per-use, published one article by a prolific uu author. open access two (1%) of the journals in the full sample of 162 journals were open access. these journals contained 3 (1%) of the 275 articles by prolific uu authors published in the sample journals. for these journals, access was available to all university-affiliated researchers at no subscription cost to the uu libraries. download statistics of open access journals for uu affiliates were not available, so the value of these journals was not calculated. discussion a review of the literature identified a potential gap in knowledge regarding libraries’ use of data about institutionally affiliated authors’ choices of publishing venues to inform table 7 top 10 subscribed journals ranked by number of articles by prolific uu authors and rank by costper-use in 2009; n = 123 title number of articles rank by cost-per-use (low to high) astrophysical journal 13 68th aip conference proceedings 7 103rd journal of physical chemistry b 7 33rd american journal of obstetrics and gynecology 6 11th journal of the american chemical society 6 5th biochemistry 5 25th inorganic chemistry 5 38th pediatric critical care medicine 4 48th physical review b condensed matter and materials physics 4 47th physical review d particles, fields, gravitation and cosmology 4 100th evidence based library and information practice 2012, 7.3 42 scholarly communications and collection development programs. this case study illustrates how knowledge of such publishing decisions and various associated factors can inform decision-making at a large research institution. factors considered included the accessibility, usage, costs, and value of the journals in which the most prolific university authors in 2009 published. access to the journals in which these authors published in 2009 was high, with more than 80% of the published articles available through uu libraries subscriptions. percentages of access to the journals in which universityaffiliated authors published and to the individual articles published were similar – 83% and 85%, respectively. this alignment between publication patterns and subscription status reflects positively on the success of the collection development program in selecting resources relevant to the fields in which university research is occurring and providing the results of that research to users. that much of the university’s research is available through library subscriptions could have implications for the perceived value of an institutional repository (ir), in that researchers may not feel as strongly about the necessity of an ir if most university research is already easily accessible. ir advocates may find more success focusing their efforts initially on the articles to which the libraries are unable to provide access through subscriptions. usage of the journals studied revealed a small number of very highly used journals, with most journals used only occasionally. four high-profile journals – nature, the journal of the american chemical society, science, and pnas – were used more than 25,000 times each in 2009, more than double the number of downloads for the fifth-ranked journal. with median usage at 365 downloads per journal, half of the journals studied were used an average of once per day or less. most journals to which the uu libraries did not subscribe were accessed through ill. demand exists for these journals, but is not extreme; only slightly more than one quarter of journals were requested 10 or more times. as might be expected, a large difference in usage was shown between journals to which university researchers had direct access and those to which they did not. the average usage of journals to which the uu libraries subscribed was more than 300 times higher than for those available through ill. in general, providing access to university research through journal subscriptions is expensive. no journal to which the uu libraries subscribed could be purchased for less than $100, more than half cost more than $1,000, and one journal subscription cost more than $10,000. the cost of journals to which the uu libraries did not subscribe was generally less than those to which subscriptions were purchased. however, the cost of providing ill access to those journals was also generally less than that of purchasing subscription access. because of this, ill was an economical way to provide access to rarely used journals. several indicators of value were considered in this study: the relationships between cost and usage of journals, between publishing venue and usage, and between publishing venue and cost-per-use. a comparison of cost to usage demonstrated the need to consider this value independently for each journal studied. a simple average calculated by dividing the total cost of the journals analyzed by the total number of downloads would produce a costper-use of $1; however, only 25% of the journals had individual costs-per-use of $1 or less. the costs-per-use calculated for the majority of journals were less than the typical costs of providing access to an article through ill or pay-per-view methods, a desirable measure for collection development. ill was shown to be an effective practice for the uu libraries, with the costs of ill rarely exceeding the costs of subscriptions to the journals requested. even for the journal requested most frequently, the ill cost was estimated at only 42% of the subscription cost. journals with high costs-per-use and ill costs approaching subscription costs should receive special attention in collection development decisions, as they present opportunities in which changes in purchasing practice have the potential to significantly, and positively, impact the journal budget. evidence based library and information practice 2012, 7.3 43 results from this study have shown that the journals in which prolific uu authors publish are not necessarily those that uu faculty, staff, and students use frequently, nor those that might be considered the best value based on cost-per-use calculations. this finding is worthy of further study and merely scratches the surface of the complex topic of the reading habits of those active in the research enterprise. some of the journals in which prolific uu authors published are used, while others are not, even though research in that area occurs on campus. while nature had the highest number of downloads and the lancet had the best cost-per-use, neither accounted for more than 1% of the papers published by the authors studied. conversely, astrophysical journal contained the largest number of uuauthored articles, yet had less than 1,000 downloads and a cost-per-use in approximately the middle of the rankings. that rankings of journals by these three factors – number of university-affiliated publications, usage, and cost-per-use – differ highlights the complexity involved in selecting journals for purchase and illustrates the value added by considering multiple factors when making collection development decisions. in isolation, each factor could offer a very different perspective on value to the university. the journal of the american chemical society represents the rare journal that performs well on all three factors, making it an ideal subscription candidate. finally, the study results indicated that very few prolific authors published in open access journals, perhaps showing that campus support of this model as an access mechanism to research is low. open access journals offer benefits for readers in terms of access and cost, but their usage is more difficult to evaluate. low levels of open access publishing among prolific uu authors have implications for a scholarly communications program, particularly in relation to increasing awareness of open access as a viable publishing option and considering the value for university funding of open access. limitations this research offers a case study example of the journal publishing patterns at a large, research-intensive post-secondary institution in the us; results are reflective only of the institution and time period studied and may not be generalizable to other institutions or across time. as a pilot exploration, this research considered only journal articles by those authors who published 10 or more articles in 2009 and identified authors using only a single database, scopus. works by authors publishing in non-journal venues were not included in this analysis, and the choice of scopus for identifying authors may have privileged those publishing in stm journals over the social sciences and humanities due to its disciplinary coverage. the publication patterns of less prolific authors may differ, and different authors may be identified as prolific using other definitions, data sources, or years of analysis. analysis was limited by the inability to obtain complete pricing and usage data for all journals in the sample and by the need to use a variety of sources to gather this information. furthermore, costs for journals purchased as part of packages represented estimates rather than actual prices paid, as these costs were derived from sources other than uu libraries’ invoices. purchasing journals through packages rather than standalone subscriptions tends to reduce the subscription costs; therefore, determining individual journal costs from sources other than library invoices may have over-estimated the true costs paid to access the journals and inflated the calculated costs-per-use for journals obtained through package-based subscriptions. the use of download statistics to represent usage ignores use of other mechanisms for accessing literature and cannot provide article-level data. similarly, the use of ill requests ignores the use of journals accessed through pay-per-view or other means, and cost estimates may not be directly comparable to estimates based on subscription prices. evidence based library and information practice 2012, 7.3 44 conclusions decisions regarding a university’s scholarly communications program or collection development activities can be based on a variety of factors. as illustrated in the literature, citation analysis is a popular strategy for gathering data; this research reflects on the value of a different type of analysis – publication analysis. an exploration of publishing patterns of university-affiliated authors and related data has been shown to provide additional information, which can help inform better decision-making. in the case of the uu, access was available through the uu libraries to the majority of journal articles published by the most prolific university-affiliated authors in 2009. cost and usage data allowed for a consideration of the effectiveness of current journal selection decisions, and a comparison of the “best” journals when ranked by number of university publications, usage, and cost-per-use highlighted differences among analyses based on different factors. articles published in open access journals accounted for only a very small portion of publishing in this study, suggesting a continued need for additional investigation and education in this area. the results of this study have implications for both collection development and scholarly communication programming. collection development decisions are routinely based on the usage of journals as measured by citations, download statistics, or ill requests. however, more comprehensive decisions may be possible through the consideration of additional factors, including the publishing patterns of university researchers. a list of journals that are widely used is not necessarily identical to a list of journals in which researchers publish, even when those researchers are among the most prolific authors on campus. the consideration of usage or publishing patterns may paint a different picture of the value of journal subscriptions, and multiple types of data allow for improved decision-making when shaping the journal collection of a research library. data about such patterns can also provide needed input when developing the activities, events, and educational offerings of a scholarly communications program. in making scholarly communications and collection development decisions, efforts should be made to consider data related to as many potentially relevant factors as is practical given the constraints of time and other resources. further research is needed to explore the implications of these results on a broader scale and determine whether they are generalizable beyond the population and time studied. a more comprehensive illustration of university publishing could be obtained with a larger and more inclusive sample, including all university authors rather than limiting to the most prolific and using a more complete list of university publications. to further inform scholarly communications, additional study concerning the number of open access compared to subscription journals published in the areas in which research occurs at the university, reasons for choosing subscription rather than open access venues for publication, and rates of archiving in the ir could shed additional light on these findings. in addition, research to investigate the relationship between the journals in which authors publish and the journals authors cite in their work would provide further value for collection development decisions. acknowledgement this research was supported in part by an appointment to the nlm associate fellowship program sponsored by the national library of medicine and administered by the oak ridge institute for science and education. references battin, p. (1980). research libraries in the network environment: the case for cooperation. journal of academic librarianship, 6(2), 68-73. retrieved 17 aug. 2012 from http://www.eric.ed.gov/ericwebport al/detail?accno=ej225890 bergman, s. s. (2006). the scholarly evidence based library and information practice 2012, 7.3 45 communication movement: highlights and recent developments. collection building, 25(4), 108-128. doi:10.1108/01604950610705989 borgman, c. l. (ed.). (1990). scholarly communication and bibliometrics. newbury park, ca: sage publications. byrd, g. d. (1990). an economic “commons” tragedy for research libraries: scholarly journal publishing and pricing trends. college & research libraries, 51(3), 184-195. retrieved 17 aug. 2012 from http://www.eric.ed.gov/ericwebport al/detail?accno=ej412093 byrd, g. d. (1999). medical faculty use of the journal literature, publishing productivity and the size of health sciences library journal collections. bulletin of the medical library association, 87(3), 312–321. retrieved 17 aug. 2012 from http://www.ncbi.nlm.nih.gov/pmc/arti cles/pmc226591/ dhawan, s. m., phull, s.k., & jain, p, (1980). documentation notes: selection of scientific journals: a model. journal of documentation, 36(1), 24-32. doi:10.1108/eb026689 dykeman, a. (1994). faculty citations: an approach to assessing the impact of diminishing resources on scientific research. library acquisitions: practice & theory, 18(2), 137-146. doi:10.1016/0364-6408(94)90021-3 hughes, j. (1995). use of faculty publication lists and isi citation data to identify a core list of journals with local importance. library acquisitions: practice & theory, 19(4), 403-413. doi:10.1016/0364-6408(95)00055-e jackson, m. e. (2003). assessing ill/dd services study: initial observations. arl: a bimonthly report, 230/231, 21-22. retrieved 17 aug. 2012 from http://www.arl.org/bm~doc/illdd.pdf labonte, k. b. (2005). citation analysis: a method for collection development for a rapidly developing field. issues in science and technology librarianship, 43. retrieved 17 aug. 2012 from http://www.istl.org/05summer/refereed.html lascar, c., & mendelsohn, l. d. (2001). an analysis of journal use by structural biologists with applications for journal collection development decisions. college & research libraries, 62(5), 422433. retrieved 20 aug. 2012 from http://crl.acrl.org/content/62/5/422.shor t malenfant, k. j. (2010). leading change in the system of scholarly communication: a case study of engaging liaison librarians for outreach to faculty. college & research libraries, 71(1), 6376. retrieved 20 aug. 2012 from http://crl.acrl.org/content/71/1/63.short odlyzko, a. (1997). the economics of electronic journals. first monday, 2(8). retrieved 20 aug. 2012 from http://firstmonday.org/article/view/542 /463 priem, j., taraborelli, d., groth, p., & neylon, c. (2010). altmetrics: a manifesto. retrieved 20 aug. 2012 from http://altmetrics.org reitz, j. m. (2004-2012). odlis: online dictionary for library and information science. retrieved 20 aug. 2012 from http://www.abcclio.com/odlis/odlis_c.aspx rowlands, i., nicholas, d., & huntington, p. (2004). scholarly communication in the digital environment: what do authors want? learned publishing, 17, 261-273. doi: 10.1087/0953151042321680 salisbury, l., & bajwa, v. (2004). faculty evidence based library and information practice 2012, 7.3 46 publications as a source of information for identifying and satisfying users’ needs: a case study at the university of arkansas, fayetteville. journal of agricultural & food information, 5(3), 11-25. doi:10.1300/j108v05n03_04 salisbury, l., & smith, j. s. (2010). the use of web of knowledge to study publishing and citation use for local researchers at the campus level. collection management, 35(2), 69-82. doi:10.1080/01462671003597959 satariano, w. a. (1978). journal use in sociology: citation analysis versus readership patterns. library quarterly, 48(3), 293-300. retrieved 20 aug. 2012 from http://www.eric.ed.gov/ericwebport al/detail?accno=ej194533 schauder, d. (1994). electronic publishing of professional articles: attitudes of academics and implications for the scholarly communication industry. journal of the american society for information science, 45(2), 73-100. doi:10.1002/(sici)10974571(199403)45:2<73::aidasi2>3.0.co;2-5 shaughnessy, t. w. (1989). scholarly communication: the need for an agenda for action – a symposium. journal of academic librarianship, 15(2), 68-78. retrieved 20 aug. 2012 from http://www.eric.ed.gov/ericwebport al/detail?accno=ej394039 tenopir, c., & king, d. w. (1997). trends in scientific scholarly journal publishing in the united states. journal of scholarly publishing, 28(3), 135-170. doi:10.3138/jsp-028-03-135 tenopir, c., & king, d. (2000). towards electronic journals: realities for scientists, librarians, and publishers. washington, d.c.: special libraries association. vieira, d., & faraino, r. (1997). analyzing the research record of an institution’s list of faculty publications. bulletin of the medical library association, 85(2), 154157. retrieved 20 aug. 2012 from http://www.ncbi.nlm.nih.gov/pmc/arti cles/pmc226243/ wilson, c. s. & tenopir, c. (2008). local citation analysis, publishing and reading patterns: using multiple methods to evaluate faculty use of an academic library’s research collection. journal of the american society for information science and technology, 59(9), 1393-1408. doi:10.1002/asi.20812 evidence based library and information practice 2012, 7.3 47 appendix a full “university of utah” affiliation search string in scopus (af-id("university of utah" 60025488) or af-id("university of utah school of medicine" 60024978) or af-id("university of utah health sciences center" 60019012) or af-id("university of utah health care" 60012445) or af-id("university of utah hospital and clinics" 60031018) or afid("intermountain injury control research center" 60016976) or af-id("university of utah research park" 60011197) or af-id("utah poison control center" 60014876) or af-id("university of utah orthopaedic center" 60008753) or af-id("utah cardiac transplant program" 60020272) or afid("university of utah neuropsychiatric institute" 60013699) or af-id("utah cancer registry" 60021500) or af-id("utah autism research program" 60024406) or af-id("university of utah institute for biomedical engineering" 60010083) or af-id("the brain institute at the university of utah" 60011781) or af-id("university of utah college of pharmacy, pharmacotherapy outcomes research center" 60026907)). appendix b journals ranked by number of articles by prolific university of utah authors in 2009; n = 162 title number of articles astrophysical journal 13 aip conference proceedings 7 journal of physical chemistry b 7 pathology case reviews 7 american journal of obstetrics and gynecology 6 journal of the american chemical society 6 biochemistry 5 inorganic chemistry 5 lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics) 5 pediatric critical care medicine 4 physical review b condensed matter and materials physics 4 physical review d particles, fields, gravitation and cosmology 4 american journal of cardiology 3 american journal of kidney diseases 3 angewandte chemie international edition 3 chemmedchem 3 concepts in magnetic resonance part b: magnetic resonance engineering 3 hypertension 3 journal of cosmology and astroparticle physics 3 journal of organic chemistry 3 new england journal of medicine 3 sensors and actuators, b: chemical 3 accounts of chemical research 2 acta neurochirurgica 2 biometrics 2 biophysical journal 2 cancer epidemiology biomarkers and prevention 2 channels 2 evidence based library and information practice 2012, 7.3 48 title number of articles chemistry a european journal 2 clinical cardiology 2 crystengcomm 2 current opinion in drug discovery and development 2 econometric theory 2 ieee transactions on neural systems and rehabilitation engineering 2 inorganica chimica acta 2 international journal of modern physics d 2 journal of clinical neuroscience 2 journal of medicinal chemistry 2 journal of molecular and cellular cardiology 2 journal of multivariate analysis 2 journal of neurosurgery 2 journal of neurosurgery: pediatrics 2 journal of physical chemistry a 2 magnetic resonance imaging 2 magnetic resonance in medicine 2 nature 2 nephrology dialysis transplantation 2 obstetrics and gynecology 2 pace pacing and clinical electrophysiology 2 plos one 2 polyhedron 2 postgraduate medicine 2 science 2 toxicon 2 academic emergency medicine 1 academic radiology 1 acm sigplan notices 1 acs chemical biology 1 acta diabetologica 1 american heart journal 1 american journal of clinical pathology 1 american journal of emergency medicine 1 american journal of hypertension 1 american journal of medicine 1 american journal of nephrology 1 american journal of perinatology 1 american journal of surgical pathology 1 american journal of therapeutics 1 annals of human genetics 1 annals of noninvasive electrocardiology 1 annals of statistics 1 annals of thoracic surgery 1 atherosclerosis 1 biomedical microdevices 1 breast cancer research and treatment 1 canadian journal of statistics 1 cancer causes and control 1 cancer prevention research (philadelphia, pa.) 1 evidence based library and information practice 2012, 7.3 49 title number of articles cardiology clinics 1 cardiovascular research 1 catheterization and cardiovascular interventions 1 chemical communications 1 circulation 1 circulation research 1 communications in statistics theory and methods 1 computing and visualization in science 1 congenital heart disease 1 congestive heart failure 1 critical care medicine 1 critical pathways in cardiology 1 current biology 1 current opinion in structural biology 1 diabetes 1 diseases of the colon and rectum 1 dmm disease models and mechanisms 1 endocrinology 1 european journal of heart failure 1 expert review of ophthalmology 1 faraday discussions 1 fertility and sterility 1 ieee transactions on biomedical engineering 1 ieee transactions on visualization and computer graphics 1 information processing in medical imaging: proceedings of the ... conference 1 international journal of cancer 1 international journal on software tools for technology transfer 1 jacc: cardiovascular interventions 1 journal of biomedical informatics 1 journal of cardiac failure 1 journal of cardiovascular pharmacology and therapeutics 1 journal of chemical physics 1 journal of chemical theory and computation 1 journal of clinical endocrinology and metabolism 1 journal of econometrics 1 journal of electrocardiology 1 journal of health care for the poor and underserved 1 journal of heart and lung transplantation 1 journal of magnetic resonance imaging 1 journal of medical genetics 1 journal of micromechanics and microengineering 1 journal of molecular biology 1 journal of neuro-oncology 1 journal of neuroscience 1 journal of nutrition 1 journal of parenteral and enteral nutrition 1 journal of pediatrics 1 journal of physiology 1 journal of rheumatology 1 journal of scientific computing 1 evidence based library and information practice 2012, 7.3 50 title number of articles journal of statistical planning and inference 1 journal of the american college of cardiology 1 journal of the american society of nephrology 1 journal of the electrochemical society 1 journal of the royal statistical society. series b: statistical methodology 1 journal of time series analysis 1 journal of women's health 1 the lancet 1 microsystem technologies 1 molecular biology and evolution 1 molecular carcinogenesis 1 molecular phylogenetics and evolution 1 neurosurgical focus 1 neurotherapeutics 1 nmr in biomedicine 1 nutrition and metabolism 1 obesity 1 orthopedics 1 pediatric and developmental pathology 1 pediatrics 1 peptides 1 physical chemistry chemical physics 1 pituitary 1 probability theory and related fields 1 proceedings electronic components and technology conference 1 proceedings of spie the international society for optical engineering 1 proceedings of the acm sigplan symposium on principles and practice of parallel programming, ppopp 1 proceedings of the national academy of sciences of the united states of america 1 radiology 1 seminars in cell and developmental biology 1 surgery for obesity and related diseases 1 surgical neurology 1 test 1 theoretical chemistry accounts 1 editorial evidence based library and information practice 2009, 4:2 37 evidence based library and information practice article school library media specialist collaboration with special education personnel in support of student learning lesley s. j. farmer professor california state university, long beach long beach, california, united states of america e-mail: lfarmer@csulb.edu received: 18 december 2008 accepted: 09 may 2009 © 2009 farmer. this is an open access article distributed under the terms of the creative commons attribution license, <http://creativecommons.org/licenses/by/2.0>, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – the objective of this study was to identify factors for effective collaboration between school library media specialists and special education personnel in support of student learning. methods – a review method was used to examine illustrative studies of collaboration. results – the analysis revealed studies that represented a variety of methodologies: survey, observation, interview, action research, and participatory ethnography. the review identified cross-study factors that facilitate collaboration between school library media specialists and special educators: shared knowledge via cross-training and regular professional interaction, effective communication skills, and effective educational team planning and co-teaching of meaningful learning activities. conclusion – the study concluded that school library media specialists and special education personnel need to share their knowledge and expertise about the effective use of appropriate resources and services for students with special needs. mailto:lfarmer@csulb.edu evidence based library and information practice 2009, 4:2 38 introduction collaboration is a central value in school librarianship. numerous studies have addressed collaboration between school library media specialists and classroom teachers as well as between school library media specialists and site administrators. school library media specialist collaboration with special education personnel has received less attention, although these staff members can contribute significantly to student success, particularly as students with disabilities are mainstreamed into the typical classroom. the unique qualities of special educators make them valuable partners for school library media specialists. the impact of these partnerships is best measured based on the evidence of the results of those collaborative efforts to impact student learning. this article examines the field of special education and its relationship to educational librarianship. a review of studies about school library media specialist practices relative to special education personnel identified cross-study factors that facilitate collaboration between school library media specialists and special educators in support of student achievement. school library media specialists and special education legislation what constitutes special education? according to the individuals with disabilities education improvement act of 2004 (ideia), the u.s. federal government defines ‚special education‛ as ‚specially designed instruction, at no cost to parents, to meet the unique needs of a child with a disability‛ (11). what then constitutes a disability? ideia defines a child with a disability as one: (i) with mental retardation, hearing impairments (including deafness), speech or language impairments, visual impairments (including blindness), serious emotional disturbance (referred to in this title as ‘emotional disturbance’), orthopedic impairments, autism, traumatic brain injury, other health impairments, or specific learning disabilities; and (ii) who, by reason thereof, needs special education and related services. (idei act 6) the law further stipulates that for children between the ages of three and nine, the term ‚disability‛ may be expanded to include a child: (i) experiencing developmental delays, as defined by the state and as measured by appropriate diagnostic instruments and procedures, in one or more of the following areas: physical development; cognitive development; communication development; social or emotional development; or adaptive development; and (ii) who, by reason thereof, needs special education and related services. (idei act 6-7) in either case, the child’s disability must adversely affect his or her educational performance. while a student with a temporary disability, such as a broken leg, might require short-term accommodations while the bone is healing, the intent of special education is to help individuals with chronic or acute disabilities participate and perform to the fullest reasonable extent in academic settings. under this definition, about 9% of children ages 3 to 21 are served through special education efforts. evidence based library and information practice 2009, 4:2 39 school library programs provide resources and services for all students, which includes students with special needs. not only is this philosophy socially responsible, but it is also a legal necessity. several federal laws in the united states impact school library programs for students with special needs: americans with disabilities act of 1990 (ada). public services must provide reasonable accommodations for all individuals with a disability. the law provides protection from discrimination. individuals with disabilities education act of 1990, 1997, and 2004 (idea). this act asserts that all eligible students are guaranteed a free public education, regardless of disability. the act provides federal financial assistance to state and local education agencies to support that mandate. assistive technology act of 1998. federal grants provide assistive technologies and associated services for persons with disabilities. the act also supports related programs and research. section 508 of the rehabilitation act of 1973. this set of standards and guidelines for information technology accessibility applies to some library programs that receive state funding. individuals with disabilities education improvement act of 2004 (ideia). this act encourages the incorporation of universaldesign based technology as part of the method to meet the needs of individuals with disabilities. gibson identified four categories of library services that need to be addressed when supporting academic achievement of students with disabilities: 1) policies and procedures. school library policies must comply with government laws and regulations, such as those mentioned above. school library media specialists also need to make sure that the entire school community, particularly families with affected students, know how the library implements those policies through appropriate accommodations. sample policies include selection policies that address needs of students with special needs, differentiated circulation periods, differentiated class management procedures, and web page development. 2) access to facilities and equipment. shelving heights, aisle widths, traffic flow, signage, furniture, and lighting all need to accommodate students with physical challenges. adaptive technologies need to be available so all students can have physical access to information, e.g., larger monitors for computers, keyboards with track balls, optical scanners, and reading software. 3) specific services. information needs to be available in a variety of formats, such as braille, audio, captioned videos, and electronic files. student-specific services may also be called for, such as extended lending periods, delivery of materials to students’ homes, customized picture books (e.g., texturized), customized teaching aids (e.g., task cards), and individual instruction. 4) staff development. library staff usually need training to enable them to interact successfully with students with special needs. some training can be general, such as universal design. other training would be disability-specific (e.g., evidence based library and information practice 2009, 4:2 40 autism or vision impairment) and student-specific, based on each student’s individual education plan. who are special education personnel? special education personnel address the academic needs of students with mild to severe disabilities: sensory, mobile, developmental, or cognitive. personnel work with students in self-contained and mainstreamed classrooms, and they collaborate with other specialists as well as classroom teachers and community agencies. the educational preparation of special education personnel varies from a high school diploma to advanced degrees. special education professionals (typically called special educators, special education instructors, or education specialists) may have a basic teaching credential and an added credential for teaching special education, or they may have a credential that enables them to teach only in special education areas. in any case, their academic preparation includes special education theory and practice, learner characteristics and development, language development, instruction, collaboration, and assessment (council for exceptional children). according to the national board for professional teaching standards, special education instructors must have the following competencies: knowledge of learners and social development knowledge of curriculum assessment and intervention ability use of various resources ability to provide a safe, caring, and stimulating learning environment communication and collaboration skills (1). according to gibson’s clusters of services, special educators can offer valuable expertise for school library media specialists. special educators know the relevant laws, and can help school library media specialists comply with those laws (e.g., web page accessibility). special educators can help school library media specialists select and arrange furniture and equipment to facilitate physical access for all students. special educators can also recommend appropriate resources – books, electronic resources, and adaptive technology – to match the needs of individual students. special educators can provide formal training and just-in-time aid. several characteristics of special educators resemble those of school library media specialists: specialized knowledge and preparation, often they are the sole site experts in a given facility; a dual focus on resources and services; formal and informal instructional roles; and their role as student support professionals. other similarities between special educators and school librarians include having no standard curriculum; often relying on just-in-time information or skills; scheduling and planning time constraints; and the potential to work with all students and other school personnel (mcgrath). what do school library media specialists contribute to special education? to collaborate effectively requires that both parties contribute to the effort. school library media specialists have a broad and deep knowledge about resources across the curriculum and in different formats, which can help special educators match materials with individual students. while special educators might know more about adaptive technologies, school library media specialists are likely to know internet and other online resources that could be useful for students with special needs. evidence based library and information practice 2009, 4:2 41 furthermore, school libraries are likely to have more current technology than special education rooms. since library use should support the curriculum, school library media specialists can link learning activities with special educators’ strategies. even more than special educators, school library media specialists work with the entire school community, and can introduce special educators to teachers who might not otherwise come into contact with them. as part of their standards for special educators, the council for exceptional children identified several collaboration factors that apply to work with school library media specialists: knowledge of the school library media specialist’s role in individual education plans knowledge of learners and learning assessment skills instruction and accommodation skills communication skills (28). hopkins offers several ideas for ways that school library media specialists and special educators can collaborate. conduct literature reviews on special education issues select resources that meet students’ needs assess physical access to information assess intellectual access to information develop learning activities that accommodate and address students’ needs assess and address professional development needs be involved in educational policies that support the needs of students with special needs. (‚accessibility‛ 18 ). jackson asserted that school library media specialists should know about various disabilities and the specific needs of students with disabilities, particularly since each type of disability may require unique resources and strategies. even within each type of disability many variations may exist. for example, autism, more accurately called autism spectrum disorder, includes five major disorders, and even one disorder may be manifested in substantially different behaviors by children of the same age. literature review: aspects of school librarianship and special education little current research exists in the professional literature regarding the collaboration of school library media specialists and special educators, particularly in terms of evidence based practice. historically, studies about library services for youth with special needs have been written soon after the enactment of federal laws pertaining to this student population. an example is the cluster of publications from the early 1990s after the americans with disabilities act and the individuals with disabilities education act were enacted (american library association; walling and karrenbrock; wright and davis). because newer legislation about supporting students with special needs has been enacted, and advances have been made in dealing with this population, the literature review for this article was limited to the past ten years. the following databases were examined: dissertations abstracts international; eric; education index; academic search elite; library literature and information science; library, information science & technology abstracts; psycinfo; socindex; sociological abstracts; family and society studies worldwide; cinahl; and the u.s. department of education’s web site, ‚what evidence based library and information practice 2009, 4:2 42 works clearinghouse‛ <http://ies.ed.gov/ncee/wwc/>. the most fruitful search terms were ‚school librar*‛ and ‚special educat*‛. when the term ‚collaboration‛ or ‚cooperative‛ was included, results were meager. the goal was to find studies that used solid research data (evidence) to develop recommendations for best practices to impact student achievement. several studies surveyed school library media specialists about accommodations for students with special needs, including collaboration with special education teachers. these studies evidenced some effort to provide appropriate resources, but in most cases, collaboration was missing or spasmodic; furthermore, library-specific and disabilityspecific training was needed in order to collaborate successfully (lani; loomos). a few studies analysed current practices, but did not apply the findings to develop interventions and test for their impact (allen; cox; murray ‚implications‛). other studies described successful collaborations, but did not provide preand post-test evidence to document the impact of the efforts (appignani and lawton; blaum and bryant). still other articles gave recommendations without rigorous research to back their claims (noonan and harada; hopkins ‚extending;‛ jurkowski). the most comprehensive review of existing studies on collaboration between school librarians and special educators was published in a 2006 issue of intervention in school & clinic. like other edited collections, the papers are uneven in quality, but the fact of the existence of this collection is encouraging as a signpost of beginning research efforts. many of the articles in this issue were anecdotal and prescriptive, rather than evidence based, and they were not included in this review. sadly, the amount of research from special education that speaks to collaboration with school library media specialists is even sparser. the most germane research was williams’ study linking special education teachers with information technology, although he did not refer to school library media specialists. the seminal researcher in the field appears to be the australian librarian jane murray. she laid the foundation in australia for identifying contributing factors for strong collaboration between school library media specialists and special educators. evidence based practice the following studies represent a range of research as noted above. a few studies explicitly provide evidence based examples of leveraging collaboration between school library media specialists and special educators, focusing on student learning and access to information, and drawing upon each stakeholder’s expertise. other studies highlight conditions for collaboration. collaboration studies in australia and new zealand the most significant and broad-based studies in educational library and disabilities services collaboration seems to emerge from australia and new zealand. because new zealand post-secondary and tertiary education can serve students as young as sixteen, and because the collaborative nature of disabilities services and educational libraries is well codified, their code development is included in this literature review. new zealand now has a national code of practice for creating inclusive environments (achieve kia orite), which further guides libraries and disability support services in their collaboration. the code was developed following wide consultation across new zealand, including the new zealand ministry of health and the new zealand ministry of education. the consultation included a survey to guide the evidence based library and information practice 2009, 4:2 43 writing of the code. the survey was administered to 95 students with impairments, members of disability groups, academicians, and disability support staff, including librarians. the survey listed 13 action plans. based on the responses, a draft code was developed and reviewed by disability staff and focus groups in seven cities, involving 165 people. the code explicitly addresses libraries in terms of providing accessible technology and computer facilities; accessible library electronic resources; assistance cards that facilitate students asking for librarian help both in person and by telephone; library workshops delivered in the students’ preferred learning modality; and other library disability support services. the code more generally discusses the need for staff training and positive interaction with these students. new zealand’s most well-known and respected example of library and special education best practices is at victoria university. the university’s office of disability support services has developed a strong professional relationship with university librarians to support student learning <http://www.victoria.ac.nz/library/services/ disabilitysupport.aspx> (gibson 60). when students first enter the university, they are introduced to both the library and special education centers. in 1992 the library was granted funding to assist students with disabilities, and a 2000 grant supported the center’s expansion. in collaboration with can-do, a support group for students with disabilities, librarians created a separate room with adaptive technology to aid students. in developing services, a consultation process between disability support services and library staff revealed that personnel attitudes about disabilities were the most significant barriers for students with disabilities. therefore, disability support services trained library staff in the experiences of people with disabilities and the implications for library services, best practices in library services for this population, and communication and information strategies. in the process, librarians reviewed their personal practices to insure that they provided inclusive service. the library and disability support services regularly monitored library services to students with disabilities through an annual survey as well as through direct student feedback received in group meetings, individual appointments and via a feedback box. changes were made, either in response to specific situations or as part of long-term planning. for example, a kitchenette was installed as a social space in response to student feedback. as a result of this initiative, students had a safe supportive place to study and social networks formed, although there was no formal assessment of improved student learning as a direct result of library services. nevertheless, over time the new zealand code mentioned above was incorporated into the assessment process (gibson 65). australian librarian janet murray conducted several studies focusing on the relationship between school library media specialists and special education teachers. her research, begun in 1994, involved sending surveys to 1,450 public, independent, and catholic primary and secondary school library media specialists in victoria and new south wales (493 responded) to assess their awareness of the needs of students with disabilities. she sent a second survey to the respondents eighteen months later to collect follow-up data. fifty-two percent of the school library media specialists had received training about disabilities (mainly in public schools), but none of the training was library-specific (and no training was given to paraprofessional library staff). furthermore, only 57% of the respondents were aware of recent legislation that might impact library services for students with disabilities, and http://www.victoria.ac.nz/library/services/disabilitysupport.aspx http://www.victoria.ac.nz/library/services/disabilitysupport.aspx http://www.victoria.ac.nz/library/services/disabilitysupport.aspx evidence based library and information practice 2009, 4:2 44 very few sites had policies dealing with library services for this population. murray concluded that in-house and local training specifically for school library media specialists was sorely needed (‚enhancing‛ 1). from this population, murray identified fifteen schools (five primary, seven secondary, and three k-12) for her next study, focusing on communication and collaboration between school library media specialists and special educators as assessed by observation and interviews (‚implications‛). a constant comparative case study methodology was used to identify salient factors for impactful collaboration. the mode of delivery (i.e., separate or inclusive classroom) did not impact learning significantly, but the provision of disability-friendly library facilities was positively significant for student learning. in combination with weekly learning activities supervised by the school library media specialist and special education staff, this welcoming atmosphere resulted in students being able to successfully perform library skills. further, students were able to perform those skills not only in the school library, but they were also able to transfer the skills to the public library setting (murray noted that during the second year of implementing this strategy at one location, the special education teacher did not accompany the students, and the school library media specialist had to spend significant time supervising circulation functions rather than teaching). school library media specialists who were aware of disability resources and who knew how to interact with students with disabilities, contributed significantly to student success; this knowledge was gained either from personal experience or work with special educators. having a special education aide available also facilitated student use of resources. the study recommended that school library media specialists take a more proactive role in telling special educators about available resources, including the use of technology. school library media specialists can conduct literature reviews for their special educator colleagues. on their part, special educators need to inform school library media specialists about specific students’ needs and disabilities in general. with this shared knowledge, school library media specialists and special educators could co-teach this population. on a site-wide basis, a school culture of collaboration also facilitated partnerships between school library media specialists and special educators. murray found that effective communication and management skills were also necessary for effective collaboration (‚implications‛ 20). site status surveys about student disabilities and services cox’s 2004 survey of rural missouri elementary school library media specialists aimed to assess accessibility to library resources and services for students with sensory or mobile impairments. from the 783 sites surveyed, 387 school library media specialists responded to questions about physical accessibility of their library facilities and about their work with impaired students, collaboration with special education teachers, and professional development about disabilities. of the respondents, 79% reported they were encouraged to collaborate with teachers about the needs of students with impairments and that student achievement improved because of accessibility to the library. nevertheless, school library media specialists indicated that they needed disability-specific training. almost half had general training about disabilities, such as information about relevant legislation, but only a third received any training about barriers that specific students might experience. most school library media specialists tried to address the needs of evidence based library and information practice 2009, 4:2 45 individual students through the choice of appropriate resources, differentiated circulation policies, and differentiated instruction such as multi-sensory resources. a majority incorporated assistive technology such as scanners, text readers, and amplifiers. however, only a third had a library website, and of those, only 3% of those websites were in compliance with the americans with disabilities act. fewer than 20% of school library media specialists in this study had acquired large-print books and books in braille, fewer than 4% had a captioned television set in the library, and only one school had a telephone device for the deaf in the library (fewer than 5% of schools had such a device for their disabled students). similarly, lani surveyed 78 elementary school library media specialists in eight southern new jersey counties (59% of the total school library media specialist population of 133) about their training experiences and information needs related to educating learning disabled students. few reported they had received training, and most had received little information about individual students and their specific needs. school library media specialists stated they had to proactively seek this data. nevertheless, most respondents had at least minimal contact with classes having learning disabled students; 45% of the responding school library media specialists worked with classroom teachers of disabled students weekly, and 10% worked with them daily. on the other hand, only 6% of school library media specialists occasionally collaborated with special education teachers. typical school library accommodations included visual aids, specialized computer programs, use of peer study buddies, and trained adult aides. school library media specialists indicated that successful inclusion required time for collaboration and strong staff support. studies about interventions to improve conditions for effective collaboration several studies focused on the conditions that foster effective collaboration between school library media specialists and special education teachers. these studies described specific resources and actions that might have potential for improved student achievement. technically, school library media specialists and special educators are support service personnel, so while the library program can improve with the collaboration of special educators, even more significant results can be obtained using a collaborative team approach incorporating classroom teachers. in examining ways to help students with literacy, farr developed second-grade and fifth-grade collaborative education teams of classroom teachers, reading specialists, special educators, and school library media specialists. in both cases, the teams met bimonthly to discuss the curriculum with the goal of improving student achievement, sometimes focusing on individual student work. the school library media specialists influenced decisions about which library material resources would be acquired for use by students in each grade. the special educators suggested modifications for instructing students with special needs. classroom observation notes verified the actions taken as a result of the teams’ collaboration, and team members selfreported student improvement (although no formal test scores were examined by the researcher). the team members improved their own practices as a result of their collaboration. factors that contributed to the effectiveness of the collaborative process included mutual respect, equity, active listening, open questioning, flexibility, trust, and understanding of the bigger picture. the percentage of students with special needs impacted discussion and results; the teams not only implemented appropriate evidence based library and information practice 2009, 4:2 46 learning activities, but they also addressed larger curriculum issues. this result confirmed murray’s earlier findings, although quantitative statistics about student performance were not collected. emphasis was placed on inter-faculty learning processes. heeks and kinnell examined how school library media specialists support special education in terms of resources, instruction, and staffing. five school districts comprising a total of ten libraries in england participated in the study, which involved document analysis, observation, and interviews with staff working with year seven special needs students. the researchers found that school library media specialists’ collaboration depended on their knowledge about special needs and the school’s operations. awareness of learning difficulties helped school library media specialists choose appropriate materials for this population, and they generally included special educators in the selection process. when special educators let school library media specialists know about learning needs, acquisition decisions improved. while improved student performance was not addressed explicitly, the researchers noted that the project’s school library media specialists increased the length of their instruction sessions, and made greater attempts to tie the sessions to classroom subject matter. in one school, book circulation increased fourfold. project libraries were refurbished to accommodate students with special needs, and surveys indicated this effort was appreciated by both staff and students. matsudo examined how school library media specialists could be involved in special education. to that end, she conducted semi-structured interviews with five special needs education coordinators in japan about their perceptions of library programs. libraries were seen as places that provided information resources and offered comfortable spaces to connect with others. school library media specialists were considered material specialists with spontaneous associations with students, and considerate of student progress. based on these perceptions, special educators and school library media specialists collaborated to provide suitable materials to address individual students’ needs, to address students’ affective needs, and to provide educational support to foster student socialization. allen examined the practices of 65 school library media specialists in a large public school district in north carolina. her online survey collected data about: 1) how school library media specialists gain information regarding best practices in special education and how they learn about students with special needs; 2) library services and instructional accommodations for special needs students; and 3) school library media specialists’ collaboration with special education teachers. most respondents (particularly high school library media specialists) reported low self-confidence about special education best practices; 78% of respondents received no relevant information about special needs instructional techniques. respondents stated they would prefer to get information from special education teachers, particularly as part of a professional development activity. of those who did receive such information, the special education teacher most frequently provided the training. only 22% had seen a student’s individual education plan (mainly at the elementary school level), nevertheless, 59% reported that they collaborated with special education teachers (least frequently at the elementary school level). at all levels, school library media specialists tried to connect library resources with classroom units, and made efforts to provide differentiated instruction (e.g., scaffolded worksheets, different texts, evidence based library and information practice 2009, 4:2 47 materials with more visuals, audio texts, hands-on manipulatives, or interactive web tutorials). at the middle school level, some school library media specialists worked with special education teachers to select appropriate resources for their students. unfortunately, no attempt was made to determine whether the collaborations impacted student learning. the main recommendation was the identified interest and need for professional development conducted by special education teachers for school library media specialists. focusing on ways to scaffold learning via the library website, jackson conducted research on collaborating with special education teachers to design effective library web pages at a suburban high school near chicago. she identified four types of scaffolding: cognitive (providing information), meta-cognitive (facilitating self-regulation), procedural (giving instruction) and strategic (suggesting alternatives). she asked special education teachers to identify their students’ research needs and issues, and created web pages to address their specific needs. then she had both special education teachers and their students’ pilot-test and evaluate the scaffolding web pages. as a result of this collaboration, jackson was able to produce a valuable library website to support students’ independent learning: <http://whs.d214.org/results/whslibspecial/> . on the other hand, no follow-up analysis was conducted to find out if the academic needs of students with special needs were actually met through the website content. the strength of this research lies, rather, on the process of working with special education teachers to design a solution that has the potential of impacting student achievement. studies about special education library aides building on the importance of transition training for high school students with disabilities, jilbert asserted that school libraries constitute a valuable vocational education venue, because on-site employment is convenient for students, there is a guarantee of adequate supervision and support, and the libraries facilitate collaboration with special education teachers. the library and special education staff participating in jilbert’s study collaboratively analysed library aide tasks according to vocational-technical education job descriptions. as students began their work, the school library media specialists and special education teachers conducted situated vocational assessments, comparing the students’ work with the actual job skills needed, so that diagnostic data could be gathered, and specific training could be provided to optimize student performance. jilbert noted that these student aides improved their communication skills and also became more productive, with some obtaining jobs as library assistants after graduation. appignani and lawton described their successful special training and employment program (step) at the south brunswick (new jersey) high school library. the school library media specialists worked with the school’s special education staff to determine life work skills that would be useful for students with special needs. the team then identified which tasks could be done in the library, and structured a student work plan to provide real-life experiences for these students, from preparation of job applications and practice for interviews to on-the-job training and supervision. the program has been popular, and some graduates have gone on to be successfully employed in local businesses. http://whs.d214.org/results/whslibspecial evidence based library and information practice 2009, 4:2 48 site-level action research efforts focused on impactful collaboration moyer and farmer are both school library media specialists who have provided reports of their action research projects focusing on collaboration with special education teachers to impact student achievement. moyer investigated the use of the accelerated reader computerized reading management program, with 69 special education ninthand tenth-grade students. the objectives of the project were to help these students become more aware of their reading habits, to increase their library reading, and to improve their reading scores. students received preand postintervention surveys to assess their attitudes about reading and their habits of sharing reading with their families and peers. in collaboration with the special education teachers, the school library media specialists developed a general reading rubric, and they worked with the students to create individual reading goals. those students who met their goals could attend an award event and receive a $20 gift certificate to spend in a local bookstore. the school library media specialists reinforced student reading by presenting them certificates when they passed accelerated reader tests, regardless of the goal. as a result, students borrowed more books (an increase of 5 to 32 books per student), visited the library weekly by themselves, connected reading with academic success, and improved both their reading and grammar scores. the school library media specialists and special education teachers reviewed and modified their action plans so that students had more choices in how they demonstrated knowledge about the books they read. students could suggest alternative accelerated reader questions, and they could request books to be added to the accelerated reader database of tests. by the end of the second year, students expressed personal enthusiasm for reading, independent of any tests or incentives. farmer’s action research project on information literacy infusion incorporated special education expertise and training. her study site was actively engaged in whole school reform, and information literacy constituted one initiative under the umbrella of reading improvement across the curriculum. one aspect of her study involved a participatory ethnographic approach to collaboration between school library media specialists and special education instructors. prior to the study, special educators tended to focus on pullout instruction, and interacted little with the rest of the teaching faculty. moreover, the self-contained special education class, located across the hall from the library, seldom visited the library or used its facilities. in an effort to address the information literacy needs of this population, the school library media specialist spoke with the special education coordinator and visited the special education learning center. the school library media specialist found few current resources or technology for the students, yet she did observe good instructional practices such as the use of visuals and inspiration software (<http://www.inspiration.com>) to capture student knowledge. the school library media specialist exchanged ideas with the special education coordinator about reading support, both in terms of learning activities and resources. she learned about relevant publishers for this population and ordered print and audio books for the library, and she gave books to the special education learning center, based on the recommendations of the special education teacher. the school library media specialist modified four library computer stations to enable students with visual and motion limitations evidence based library and information practice 2009, 4:2 49 to access needed information. the school library media specialist met with the selfcontained classroom special education teachers and aides to show them the adapted computers and new, targeted books (displayed together for easy access); as a result, special education personnel felt more comfortable with the library program, and they brought their students to the library almost daily. based on the special educators’ needs assessment and suggestions for modifying instruction, the school library media specialists gave special education students targeted instruction on finding and using library resources, particularly to help them locate articles and visual materials. the students began checking out library materials for the first time, and some students visited and used the library productively on an independent level. the school library media specialist also encouraged the special educator to give a faculty in-service on reading interventions, and used grant money to pay the special educator to attend after-school department chair meetings (a cost that was picked up by the site the following year and made into a permanent position). these venues enabled classroom teachers to get to know the special education coordinator and her areas of expertise, leading to increased collaboration between the special education personnel and the school library media specialist (e.g., providing more differentiated instruction, using alternative reading resources, modifying student projects and assessment instruments. the focused collaboration of the school library media specialist, special education staff, and classroom teachers resulted in more on-task academic behavior and higher graduation rates among this student population. discussion while several studies indicated that school library media specialists and special education personnel want to collaborate (allen; cox; lani), few studies provided evidence of an impact on student learning. nevertheless, the studies in this report do reveal some needs and possible factors for effective collaboration. most of the studies described here based their conclusions on perceptions. such selfreporting may be skewed; however, school librarianship research often relies on surveys for data collection. surveys were the prevalent data gathering method used in these studies (allen; cox; gibson; lani; matsudo; moyer). heeks and kinnell and matsudo conducted interviews, and murray led focus group discussions (‚implications‛). several others observed behaviors, mainly for case study research (farmer; farr; heeks and kinnell; murray, ‚implications‛). heeks and kinnell and gibson analysed documents to ascertain significant differences. gibson noted library skills gains, and the two studies about vocational aides (appignani and lawton; jilbert ) assessed students’ improved library work competency. the reported prevailing status of collaboration between school library media specialists and special educators is uneven, largely due to the underlying conditions for such collaboration. on the positive side, matsudo found that special educators generally held high opinions about school library media specialists, which facilitated their collaboration. cox reported that the majority of school library media specialists used assistive technology, incorporating it into the library program. allen and cox also noted that school library media specialists helped students with special needs by choosing appropriate resources, having different circulation policies, and providing differentiated instruction. lani mentioned that school library media specialists used visual learning aids and special computer programs to aid special needs students. evidence based library and information practice 2009, 4:2 50 cox and murray (‚enhancing‛) reported that roughly half of responding school library media specialists had training about disabilities in general, and most respondents stated that they needed disability-specific or library-targeted training. allen asserted that most school library media specialists had low self-confidence dealing with students having special needs, and she also found that fewer than a quarter of the respondents had even seen an individual education plan. in terms of resources to support special education, cox discovered that only a third of the schools had library websites, and that only 3 % of them complied with the americans with disabilities act guidelines. cox also found that only a fifth of the respondents had large-print or braille books, fewer than 4% had captioned television, and only one library had a telephone with hearing-impairment options. not surprisingly, then, several needs emerged: training about disabilities and learning needs of this student population (allen; cox; jackson; lani; murray ‚implications‛), policies that addressed disabilities (murray ‚enhancing‛), staff support (lani), and time to collaborate (lani). the majority of studies analysed for this article were descriptive in nature, identifying existing factors and making recommendations for improvement. however, several studies did develop interventions to improve collaboration and its impact on student learning. farmer, farr, heeks and kinnell, and murray (‚implications‛) asserted that training, provided either by the special educator or by the librarian, had several benefits: a greater frequency and depth of collaboration, better acquisition decisions, and improved lessons and instruction. moyer explained that the school library media specialists and the special educators jointly supported an accelerated reader program by developing a reading rubric and reading goals, and by providing incentives, which resulted in more independent student visits to the library, increased book circulation, and improved reading and grammar scores. likewise, library aide programs included jointly developed training plans and assessments, resulting in improved library and communication skills (appignani and lawton; jilbert). studies by gibson and by heeks and kinnell addressed the affective domain and found that providing a disabilities-friendly library facility resulted in more student visits and in improved student socialization. these studies were not highly empirical in nature; even the case studies reporting specific student learning gains (appignani and lawton; farmer; jilbert; moyer; murray ‚implications‛) did not always generate specific statistics, nor can their results be generalized. jackson’s incorporation of input from special educators and students into web page design did not include follow-up to determine if the improved website impacted student learning. with the current emphasis on data-driven decision-making, future studies might be more inclined to gather baseline data, develop and implement interventions, and then analyse the results to determine significant student learning effects due to collaborative efforts. conclusions these research studies demonstrate how collaboration between school library media specialists and special education personnel can impact student learning. even if uneven in methodology, these studies provide first steps in investigating such collaborative efforts, and suggest areas for further investigations. several findings cut across the studies, regardless of population or approach. one evidence based library and information practice 2009, 4:2 51 of the main strands was the identification of factors that contributed significantly to collaborative efforts. the most important factor that emerged was shared knowledge between school library media specialists and special education personnel: about special education learners and learning (allen; cox; gibson; heeks and kinnell; murray ‚implications‛) about relevant resources in a variety of formats (cox; farmer; murray ‚implications‛) about positive, safe learning environments (gibson; heeks and kinnell; murray ‚implications‛) about formative assessment and its analysis and application (jackson; moyer) about appropriate accommodations and interventions for specific students (appignani and lawton; farmer; jilbert) such mutual information requires crosstraining and regular professional interaction (appignani and lawton; farmer; farr; jilbert; moyer; murray ‚implications‛). other contributing factors to impactful collaboration included effective communication and management skills (gibson; murray ‚implications‛), time for collaboration (lani), and staff support (lani; murray ‚implications‛). with such collaboration in place, students are more likely to achieve because of effective planning and co-teaching of meaningful learning activities in consort with classroom teachers (farmer; farr; murray ‚implications‛). noting the number of studies that developed assessment instruments but did not develop interventions or measures of their effectiveness, school library media specialists have many opportunities to conduct evidence based research to measure their impact on student achievement. specific areas of potential research might include: empirical data about the impact of paraprofessional library staff and special education staff patterns on school library media specialists’ efforts in support of special education and the academic achievement of students with special needs empirical data about the impact of disability-friendly library facilities on the achievement of students with special needs empirical data about the impact of library assistive technology and other resources on the academic achievement of students with special needs identification of conditions or factors that facilitate collaboration between school library media specialists and special needs educators longitudinal data about the nature and impact of collaboration between school library media specialists and special education instructors: frequency of communication, extent of co-planning and implementation, partnership role, extent of interdependence impact of disability-specific interventions developed through collaboration efforts of school library media specialists and special needs educators on the academic and social development of disabled students data about special education content in school library media academic preparation program curricula and school library services content in special education academic preparation program curriculum; impact of incorporating this content evidence based library and information practice 2009, 4:2 52 on the collaborative efforts of school library media specialists and special educators identification of optimum roles for school library media specialists and special educators in conjunction with classroom teacher efforts for students with special needs. one promising suggestion for evidence based practice that might build on collaboration efforts between school library media specialists and special education instructors is the incorporation of responseto-intervention. this federally mandated strategy, grounded in the individuals with disabilities education improvement act of 2004, focuses on inclusive classroom instruction and assessment. the teacher identifies those subgroups that do not meet standards and provides needed targeted interventions. the response to those interventions is then assessed to determine if additional help is needed. while the classroom teacher serves as the point person, response-to-intervention strongly encourages a team approach (mellard and johnson). with their combined knowledge base, school library media specialists and special educators can offer significant support to students with special needs as well as their classroom teachers. by documenting their strategies and student learning assessments, school library media specialists could build a persuasive case that they impact classroom instruction. more generally, school library media specialists and special education personnel have much expertise to share with each other and with others in the school community. increased research and documentation of those efforts can strengthen their collaboration and contribute significantly to student achievement. works cited achieve: the national post secondary education disability network incorporated. (2003). new zealand code of practice consultation survey. wellington, nz: achieve. achieve: the national post secondary education disability network incorporated. kia orite -achieving equity: the new zealand code of practice for creating an inclusive environment for students with impairments. wellington, nz: new zealand ministry of education, 2004. allen, kendra. ‚the school library media program and special education programs.‛ ms thesis. university of north carolina at chapel hill, chapel hill, nc, nov. 2008. 18 may 2009 <http://etd.ils.unc.edu/dspace/bitstr eam/1901/570/1/kendraallen.pdf>. american library association. programming for serving children with special needs. chicago: american library association, 1994. appignani, londa, and deborah lawton. ‚partnerships at work in the library.‛ book report 18.3 (nov./dec.1999): 10-11. blaum, dawn, and bryant, cathy. ‚happily ever after: a tale of collaboration between the library media specialist and special education.‛ knowledge quest 33.2 (nov/dec 2004): 34–5. http://etd.ils.unc.edu/dspace/bitstream/1901/570/1/kendraallen.pdf http://etd.ils.unc.edu/dspace/bitstream/1901/570/1/kendraallen.pdf http://etd.ils.unc.edu/dspace/bitstream/1901/570/1/kendraallen.pdf evidence based library and information practice 2009, 4:2 53 council for exceptional children. what every special educator must know: ethics, standards, and guidelines for special educators. 5th ed. arlington, va: council for exceptional children, 2003. cox, john e. ‚accessibility of rural missouri library media centers.‛ diss. central missouri state university, warrensburg, mo, 2004. downing, joyce a. ‚media centers and special education: introduction to the special issue.‛ intervention in school & clinic 42.2 (nov. 2006): 6777. downing, joyce a. students with emotional and behavioral problems: assessment, management, and intervention strategies. saddle river, nj: merrill/prentice hall, 1997. farmer, lesley. building information literacy through a whole school reform approach.‛ knowledge quest 29.4 (2001): 20-4. farr, william joseph ‚collaboration among teachers of literacy.‛ diss. university of connecticut, 2004. gibson, ava. ‚disability support services, victoria university of wellington.‛ improving the quality of library services for students with disabilities. eds. peter hernon and philip calvert. westport, ct: libraries unlimited, 2006. 57-66. heeks, peggy, and margaret kinnell. ‚learning support for all? special needs pupils and learning resources.‛ journal of librarianship and information science 29.3 (sept. 1997): 143–53. hopkins, janet. ‚extending inclusive learning: library and special education collaboration.‛ library media connection 23.6 (mar. 2005): 17–9. hopkins, janet. ‚school library accessibility: the role of assistive technology.‛ teacher librarian 31.3 (feb. 2004): 15-8. hutchinson, nancy g. ‚beyond ada compliance: redefining accessibility.‛ american libraries 32.6 (2001): 76-8. jackson, carolyn m. ‚the high school library web site: scaffolding information literacy skills.‛ diss. illinois state university, bloomington, il, 2006. jilbert, deborah. ‚vocational instruction in the library media center.‛ in serving special needs students in the school library media center. ed. carten wesson and margaret keefe. westport, ct: greenwood, 1995. 67-78. jurkowski, odin. ‚the library as a support system for students.‛ intervention in school & clinic 42.2 (2006): 78-83. lani, janet w. ‚the role of the library media specialist in the education of learning disabled students in southern new jersey.‛ ma thesis. rowan university, glassboro, nj, 1999. 15 may 2009 <http://www.rowan.edu/library/row an_theses/ru1999/0199ther.pdf>. loomos, kathleen. an investigation of urban teachers’ attitudes toward students with disabilities in the least restrictive environment. diss. loyola university of chicago, 2001. http://www.rowan.edu/library/rowan_theses/ru1999/0199ther.pdf http://www.rowan.edu/library/rowan_theses/ru1999/0199ther.pdf http://www.rowan.edu/library/rowan_theses/ru1999/0199ther.pdf evidence based library and information practice 2009, 4:2 54 lucas, jessica. ‚stretching and expanding: the changing role of special educators.‛ missouri innovations in education 32.1 (2004). matsudo, hiroyo. ‚school librarians’ anticipated support for students with sen: using a modified grounded theory approach.‛ proceedings of the iasl conference, berkeley, 2-5 august 2008. mcgrath, p.j. (1990). ‚proactive and reactive responses to support teaching-learning difficulties.‛ cooperative planning and teaching. ed. james henri. wagga wagga, nsw: centre for information studies, charles sturt universityriverina. 92-7. mellard, daryl, and evelyn johnson. rti: a practitioner’s guide to implementing response to intervention. thousand oaks, ca: corwin press, 2008. moyer, mary. ‚accelerated reader sparks high school reading excitement.‛ knowledge quest 35.1 (sept./oct. 2006): 34-9. murray, janet. ‚enhancing the skills of school library staff to cater for individual student needs.‛ proceedings of the ifla conference, bangkok, 20-28 august, 1999. murray, janet. ‚the implications of inclusive schooling for school libraries.‛ international journal of disability, development & education 49.3 (sept. 2002): 301-22. national board for professional teaching standards. early children though young adulthood/exceptional needs specialist. arlington, va: national board for professional teaching standards, 2003. new zealand ministry of health. the new zealand disability strategy: making a world of difference. wellington, nz: ministry of health, 2001. new zealand ministry of education. tertiary education strategy 2002-07. wellington, nz: ministry of education, 2002. noonan, mary, and violet harada. ‚special education and inclusion: opportunities for collaboration.‛ school reform and the school library media specialists. ed. sandra hughes-hassell and violet harada. westport, ct: libraries unlimited, 2007. 131-44. u.s. government. office of special education programs. individuals with disabilities education improvement act of 2004. washington, dc: government printing office, 2004. public law 108–446; 118 stat. 2647. walling, linda, and marilyn karrenbrock. disabilities, children, and libraries: mainstreaming services in public libraries and school library media centers. englewood, co: libraries unlimited, 1993. williams, peter. ‚using information and communication technology with special educational needs students: the views of frontline professionals.‛ aslib proceedings 57.6 (2005): 539–53. evidence based library and information practice 2009, 4:2 55 wright, keith, and judith davis. serving the disabled: a how-to-do-it manual for librarians. new york: neal schuman, 1991. evidence based library and information practice evidence based library and information practice 2011, 6.4 107 evidence based library and information practice article activity-based costing (abc) and time-driven activity-based costing (tdabc): applicable methods for university libraries? kate-riin kont phd student, tallinn university researcher, academic library of tallinn university head of acquisitions, tallinn university of technology library tallinn, estonia email: kont@lib.ttu.ee signe jantson phd student, tallinn university of technology researcher, academic library of tallinn university head of bibliographic department, tallinn university of technology library tallinn, estonia email: jantson@lib.ttu.ee received: 08 apr. 2011 accepted: 26 aug. 2011 2011 kont and jantson. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – this article provides an overview of how university libraries research and adapt new cost accounting models, such as “activity-based costing” (abc) and “timedriven activity-based costing” (tdabc), focusing on the strengths and weaknesses of both methods to determine which of these two is suitable for application in university libraries. methods – this paper reviews and summarizes the literature on cost accounting and costing practices of university libraries. a brief overview of the history of cost accounting, costing, and time and motion studies in libraries is also provided. the abc and the tdabc method, designed as a revised and easier version of the abc by kaplan and anderson (kaplan & anderson 2004) at the beginning of the 21st century, as well as the adoption and adaptation of these methods by university libraries are described, and their strengths and weaknesses, as well as their suitability for university libraries, are analyzed. mailto:kont@lib.ttu.ee� mailto:jantson@lib.ttu.ee� evidence based library and information practice 2011, 6.4 108 results – cost accounting and costing studies in libraries have a long history, the first of these dating back to 1877. the development of cost accounting and time and motion studies can be seen as a natural evolution of techniques which were created to solve management problems. the abc method is the best-known management accounting innovation of the last 20 years, and is already widely used in university libraries around the world. however, setting up an abc system can be very costly, and the system needs to be regularly updated, which further increases its costs. the tdabc system can not only be implemented more quickly (and thus more cheaply), but also can be updated more easily than the traditional abc, which makes the tdabc the more suitable method for university libraries. conclusion – both methods are suitable for university libraries. however, the abc method can only be implemented in collaboration with an accounting department. the tdabc method can be tested and implemented by separate departments, and thus can contribute to the provision of better and more effective library services at lower costs. however, the involvement of experts in costing and accounting is recommended. introduction in economically stable times, little attention is paid to effectiveness, but in hard times, it becomes essential. therefore, reassessment of activities and reforms, caused by the recent financial crises, aim to save money and improve work performance in the public, as well as in the private sector. public sector organizations may have less incentive for efficiency, and this is related to the principle of budgetary control. indeed, a public organization’s budget does not depend on the efficiency and performance of the organization. because of this lack of control, public sector organizations were seldom interested in saving their budgetary funds. if an organization or its department strove to be financially effective and economize, this may well have resulted in a lower budget for the next year. in the current socioeconomic situation, efficiency and performance have become imperative. the challenge is to cope with the same or reduced resources in managing the same processes and activities, ensuring that the quality of the service and its results are not affected. according to iso 2789:2006, the operating costs of libraries should be classified as follows: personnel costs, acquisition costs, interlibrary loan costs, collection maintenance costs, administrative costs, and other costs. personnel costs are considered to be the largest item of expenditure for libraries, followed by acquisition costs, administrative costs and other costs. libraries, therefore, have known precisely into which categories the costs belong, but lack a specific overview of the activities between which these costs are divided. even if they have had precise knowledge of how much money was spent, for example, on each acquired publication, it has been very difficult to determine the entire cost of acquisition and cataloguing. personnel costs related to the processing of an acquired publication have not been added to the purchase price of the publication. libraries today must respond to the general demand for cost transparency and effective cost management. libraries can assess in detail the cost of collection building, relying on the data they have traditionally collected. what they need now is reliable data about the cost of their services and products. in order to obtain precise information about the actual expenses related to different work processes in libraries, managers have looked towards evidence based library and information practice 2011, 6.4 109 methods that were originally developed for industry and private sector organizations. literature review in the past, library managers paid little attention to such areas of librarianship as the development and effective management of book collections, classification, cataloguing, and the like. this inattention was not important; the small size of collections, staff, buildings, and clientele made for simplicity of operation and did not demand a very sophisticated approach to the ways of doing things (coney, 1952, p. 83). by the end of the 19th century, libraries had become service-providing institutions, whose task was to collect, store, preserve, and make available books for users. simultaneously, libraries developed a need to justify their budgets and costs to their parent organizations, be it a university (in the case of university libraries) or a local government (in the case of public libraries). library managers were ready to start to apply new management techniques in their libraries, many libraries being large enough to apply the ideas of cost accounting. librarians in these libraries were interested in achieving maximum efficiency at minimum cost. they accumulated data on unit costs, particularly costs associated with the cataloguing and processing of materials (which amounts to a large part of a library’s budget), in order to identify ways to reduce these costs. cost accounting studies, as well as time and motion studies, were undertaken on a regular basis so as to create efficiencies in library operations through time reductions (lynch, 1979, p. 262). early studies and reports of library cost accounting (cutter, 1877; whitney, 1885; bishop, 1905) indicate that one of the main reasons why cost accounting reached libraries was the need of library managers to justify their costs to the public, as well as to their parent organizations. however, this was seldom easy. critics seemed to think that investment in the cataloguing system was wasteful. in addition to cataloguing costs, the work such as helping readers to find necessary books, keeping shelves in order so that every book could be found at its designated spot, and replying to written enquiries, which did not seem to involve costs in the eyes of the public, had to be justified. the first library institution in which cataloguing was measured in terms of time spent was the grand rapids public library (usa) in 1914 (reichmann, 1953). the organizational committee of the study reported: “today the library must emulate the business organization in employing the cheapest grade of labor where it can be used and using its highest priced labor only for strictly professional work.” it added: “each member of the staff should be doing the most advanced work for which she is equipped” (morsch, 1954). this study marked the beginning of a new stage in the history of cost accounting research. the librarian was no longer a scholar with independent time use, but was transformed into an employee performing routine work, from whom, in addition to accuracy and thoroughness, speed and productivity in performing tasks was required. only cataloguing costs are ever mentioned in all these early references, probably because it has always been the most costly part of library work, and thus, library managers have been constantly looking for ways to cut these costs. lucile m. morsch (1954) has said: “economy in cataloguing is economy that actually saves expense in money or time on the library budget as a whole, and does not merely save this expense in the catalogue department to transfer it to another department or to some future time” (p. 479). a study carried out by fremont rider in wesleyan university, middleton, connecticut in 1935 focused on the idea that administration and overhead, by which rider meant the rent or cost of housing the catalogue department, heating, lighting, water, telephone costs, printing, stationery and postage, depreciation, insurance, janitorial services, and building repairs, should be calculated as a part of evidence based library and information practice 2011, 6.4 110 cataloguing costs. the author of the study warns that no cost system itself can cut costs. all it can do is to show the administrator where the costs may and should be cut (rider, 1936; harris, 1989). in the 1940s, libraries began to adapt the time and motion studies method, originally developed by frederick w. taylor and further developed by frank and lillian gilbreth. while library literature contains many examples of cost studies and reports of time devoted to the different phases of library operation, there have been only a few applications of time and motion study technique in the formal sense. in fact, many of these studies actually exist only as manuscript materials (for example, jewel c. hardkopf’s and watson o'd. pierce’s studies from 1949, referred to by logsdon (1954)). time and motion studies in libraries did not only measure the performance of individual workers, but also encompassed such matters as work simplification, salary standardization, determination of the standards of performance for specific library operations, improvement of working conditions (in regard to light, noise, and fatigue), systematic in-service training, and employee turnover. another characteristic of the use of time and motion studies in libraries was a careful definition and assignment of work in each department. work definitions were expected to facilitate the measurement of performance. they fixed responsibility of the performance and influenced the hiring and assignment of personnel (lynch, 1979, p. 261). in the 1960s and 1970s, social indicators emerged in public sector management, including libraries. this movement is closely related to human resource management. besides staff management and its intraorganizational aspects, human resource management deals with the general issues of human management, including those related to the labour market and job performance. job performance evaluation makes it possible to assess positions and employees’ work, that is, work performance (türk, 2005). although a number of theories and paradigms were developed to manage, analyze, and study organizations and their activities, the principles devised by taylor appealed to many library managers. random time sampling for work and cost analysis became popular among library managers and researchers (spencer, 1971; masterson, 1976; divilbiss & self, 1978; mick, 1979; mosborg, 1980). however, the first studies that took into account the employee perspective – rest periods, staff meetings and inevitable interruptions – did not appear until the 1980s1990s and in the 2000s, with the development of new public management (npm) and evidence-based policy (ebp). at that point, new cost accounting methods, such as activitybased costing (abc), which was designed in the united states during the 1980s by cooper and kaplan (1988), and time-driven activitybased costing (tdabc), which was designed as a revised and easier version of the abc by kaplan and anderson at the beginning of the 21st century (2004; 2007), emerged and were adopted and adapted by university libraries. abc, tdabc and university libraries the dependence on local government and state finances, as well as the increasing prices of publications, electronic databases, and other materials, have influenced the library and information sector in both positive and negative ways since the 1980s (roberts, 2003). libraries have traditionally offered their services free of charge. new services based on expensive licence fees make it economically difficult for libraries to serve with limited and shrinking resources, when price increases exceed the annual increase of library budgets (haarala, 2004). in the 1990s, savings in staff and in information resources became common. downsizing staff was, and still is, a painful operation – civil servants were laid off, and voluntary vacation was recommended. the lack of funding for staff was evident, and action was needed (haarala, 2004). to get a better picture of the activities that libraries are actually engaged in and their cost, studies using new cost accounting/costing models in libraries around the world have become evidence based library and information practice 2011, 6.4 111 common. constraint on public funds and a shift to an enterprise culture have undermined the tradition of the free provision of services. however, this has also encouraged a climate of innovation and so gained new potential and momentum (roberts, 2003). the activity-based costing (abc) method is the best-known management accounting innovation of the last twenty years (wegman, 2009). it was originally used in the manufacturing sector in response to dissatisfaction with traditional management accounting techniques that rely on volumebased methods for allocating overheads to products. according to t. a. spedding and g. q. sun (1999), these methods are highly inaccurate in the modern manufacturing environment. today, much of the significant cost in producing an item is not volume related (for example, the cost of engineering, order processing, planning, and quality control for high technology, made-to-order products, or just-in-time delivery). the abc, however, takes into account the cost incurred at the activity level and then attributes the cost to products according to the activities that a product goes through (spedding & sun, 1999, p. 291). the abc is a logical approach to the management of the organization, which helps to clarify and to assess an organization's processes and identify their cost (ellisnewman, 2003). the idea behind activitybased costing is simple and logical – costs and expenses do not arise automatically; charges and expenses are incurred as a result of management activities. there is no cost inside the organization; there are only activities carried out for customer service, which in turn lead to charges. on the basis of the literature review, a diagram (figure 1) can be drawn to illustrate the structure as well as the key concepts of the abc system (ellis-newman, 2003; karu, 2008). resources – an economic element that is required in the performance of activities. cost – the monetary value of resources used or sacrificed or liabilities incurred to achieve an objective, such as to acquire or produce goods or to perform an activity or service. cost driver – factors that cause changes in the cost of an activity. resource cost driver – an indicator that helps to associate the costs of resources to corresponding activities and to distribute the costs of different resources between activities. activity – what we do in an organization. activity cost driver – a measure of the consumption of an activity by products, customers or services. used as a basis of assigning activities to cost objects. cost object – an activity, output, or item, of which cost is to be measured. in a broad sense, the cost object can be an organizational division, a function, task, product, service, or a customer. there is a four-step approach to implementing the abc system, involving the following steps (ellis-newman, 2003): • identify the key activities and relevant cost drivers, • allocate staff time to activities, • attribute staff salaries and other costs to activity cost pools, • determine the cost per cost driver. the testing and implementation of the abc method is already very common in university libraries around the world (goddard & ooi, 1998; ceynowa, 2000; poll, 2001; ellis figure 1 the structure of the activity-based costing system. evidence based library and information practice 2011, 6.4 112 newman, 2003; heaney, 2004; ching, leung, fidow, & huang, 2008). many library managers have decided that the activity-based costing method is the best of the existing cost analysis methods adapted for evaluating library products. some examples of studies in university libraries using the abc model: • the university of southampton applied the abc methodology to its library services in 1991. the results showed substantial differences in the allocation of the central overhead costs between academic faculties using the model and the existing system. the abc approach resulted in significant improvements in several aspects. it: 1) created incentives so as not to overconsume library services, 2) provided more equitable overhead allocation than traditional systems, and 3) resulted in verified and refuted allocated costs (goddard & ooi, 1998). • in 1996, edith cowan university (ecu) in australia adopted the abc and undertook practical steps towards implementing the system. the feasibility of the activity-based costing method in the context of various library operations in the university library was studied. the research mostly focused on activities related to reader services and interlibrary loan. the results enabled library management to gain information about activity costs that the traditional university accounting system did not provide. the study allowed library managers to determine necessary and unnecessary activities. only those services that added value were retained, whereas services with no added value were discontinued, thus cutting costs for the university (ellisnewman & robinson, 1998). • the german research association carried out a project “cost management for academic libraries.” the aim was to test activity-based costing and develop a method of cost management appropriate to the financial management framework of public sector provision for academic libraries. the project started in april 1997 and resulted in the publication of a handbook that included the necessary software in july 1999. fiftyfive library “products” were defined. the project results showed that in view of the fact that budgets have become ever tighter, activity cost accounting helps to preserve and ensure the financial scope of academic information provision (ceynowa, 2000; poll, 2001). • oxford university library services (ouls) started to deploy the abc system in 2001 to identify work efficiencies and calculate the actual costs of materials and services. like most academic libraries, the income for ouls comes from multiple sources, including government funding, donations, and project funding. by implementing the abc system, ouls was more accountable to the university for effective budget management than departments that have not yet analyzed their activities in such a way (heaney, 2004). • three academic libraries – the library of the university of utrecht in the netherlands, the library of the university of luleå in sweden, and the library of the university of bremen in germany – used the abc analysis in the development of a current and future cost/benefit financial model to support the library management’s decision-making and the library budgeting process by eliminating uncertainty. they compared the costs of current and future digitised library services. future costs were defined as the costs of the desired and expected library service with a horizon of five to six years ahead. it appeared that, for example, the utrecht university library expects personnel costs to decrease by 12 percent, because the evidence based library and information practice 2011, 6.4 113 number of personnel will decrease by 20 percent and the average wage per employee will increase by 10 percent, as the personnel will need to be more highly qualified, whereas luleå university library expects a minor decrease in personnel costs by three percent and bremen university library estimates the number of personnel to increase by five percent, so that personnel costs should also increase by five percent (kollöffel & kaandorp, 2003). • in 2008, the abc method was used to analyze and enhance the activities of the super e-book consortium in taiwan and hong kong. the consortium includes 42 members from 160 universities and colleges. all consortium members pay an annual membership fee of 50,000 usd to the e-book fund. since the number of fulltime employees (fte) in the member libraries varied from 10 to 350, those member library employees involved in the activities of the consortium were interviewed to determine key activities so that the abc method could be implemented in the consortium. each interviewee’s role in certain activities was determined alongside the time (in hours) that they spent on these activities. thereafter, the salaries of employees and other costs related to each activity were distinguished and the total cost of each activity was calculated, as well as the cost of the cost driver (ching, leung, fidow, & huang, 2008). the abc appeared to possess two significant flaws. firstly, setting up the abc system can be very costly, especially if the current accounting system of the university does not support the collection of the abc information. secondly, the system needs to be regularly updated, which further increases its costs. moreover, it also became clear from many case studies in university libraries that the implementation problems of the abc render it less efficient than theory would suggest. there are significant costs associated with developing and maintaining such a system relative to traditional methods. the library system was time-consuming to produce, taking one person almost three months of effort. to maintain such a system would also be relatively expensive, as the cost driver rates would have to be periodically recalculated. producing similar systems for all central overheads would be very expensive. these limitations motivated kaplan and anderson to develop the time-driven activitybased costing (tdabc) method, a revised version of the abc, to solve these problems without losing the benefits (2004; 2007). the tdabc model can be estimated and installed quickly as only two parameters are required: 1) the number of time units (e.g., minutes) consumed by the activities related to the cost objects (the activities the organization performs for products, services, and customers), and 2) the cost per time unit. hence, the tdabc systems can be implemented more quickly (and thus more cheaply), as well as updated more easily than the traditional abc (pernot, roodhooft, & van den abbeele, 2007). in other words, it is necessary to determine the capacity cost rate and the use of capacity of the implemented activities carried out by each subunit. both parameters are easily identifiable. the capacity cost rate is determined as in figure 2 (karu, 2008; kaplan & anderson, 2007). figure 2 the capacity cost rate formula. evidence based library and information practice 2011, 6.4 114 practical capacity is often estimated as a percentage, for instance, 80% or 85% of theoretical capacity. that is, if an employee or machine can normally work 40 hours per week, practical capacity could be assumed to be 32 hours per week. this estimate allows for 20% of personnel time for breaks, arrival and departure, and communication and reading unrelated to actual work performance, and 20% of machine time for downtime due to maintenance, repair, and scheduling fluctuations (karu, 2008; kaplan & anderson, 2007). it is important to stress, though, that the question is not about the percentage of time an employee spends doing an activity, but how long it takes to complete one unit of that activity (the time required to process one order: for example, how much time it takes to deal with one interlibrary loan request – order reception, request handling, and transmission of orders) (kaplan & anderson, 2004). knowing the real (practical) capacity of the resources used and the time spent on activities, it is possible to determine the cost of each activity by multiplying the time spent on activities by the practical capacity of the resources (karu, 2008). in the context of the university library, the tdabc method has been tested twice in belgium, at the arenberg library of the catholic university of leuven. • the first study concentrated on interlibrary loan (ill). the data was collected via direct observations and by conducting interviews with the head of the library and employees involved in interlibrary loan. data on the costs was obtained from accounting and the rest of the data retrieved from archival documents and annual reports of the library. the observations and the • interviews were conducted in 2006 and the financial data dates back to 2005. the authors note that direct observation was carried out by the ill staff member responsible: the researchers explained to each individual what to observe (i.e. all different activities concerning ill) and what to measure (i.e. the time needed to perform the activities). the interviews were open, without a structured questionnaire. the ill activities were identified by separating incoming and outgoing requests as well as the originals and copies. after this, the cost per minute was identified, and based on this, the costs related to the ill activities were determined. the results indicated, for example, that the scanning of articles to be sent out to the recipients took around 3.4 to 7.4 minutes, due to outdated scanning equipment, which also reduced the quality of the result. readers often preferred articles that were scanned and forwarded via email as pdf files to copies of articles sent by regular mail. this gives an opportunity to preserve the article for future reuse for the library that performed the scanning. thus, an investment in new scanning devices would certainly pay off (pernot, et al, 2007). • the second study concentrated on acquisitions. the researchers set up a time-driven activity-based costing system for the ill service and for the activities of acquisitions. data gathering consisted of the following three steps: 1) the authors had a general conversation with the head of the library and some employees to grasp the scope and complexity of the acquisition process. it was important to know whether or not the scope should be limited (e.g., only the purchase of domestic books) or expanded (e.g., including cataloguing) to match the complexity to their expectations regarding the work and the time available. 2) the most important aspect of this step was that the researchers explicitly informed the staff about exceptions and alternatives. since the researchers had only limited time and a limited sample for the measurements, they had to make sure that the sample was evidence based library and information practice 2011, 6.4 115 representative for the general activity of the library. the staff walked them through different situations and possibilities, so that the authors of the study were able to measure each possibility, or at the very least mention that it is in theory a possibility, which is very rarely used. 3) the authors timed the staff during the performance of those tasks. they made measurements at specific times over a two month period, and noted some representative cases, in addition to the regular workload on that day. the authors also “replicated some information of a certain measurement.” for instance, they timed each e-mail that was opened, using the same time for the action “opening and reading mail” in different sections and phases. the measurement was continuously done for several cases, while the researchers kept track of the time of each phase. not stopping after each case and starting again at the next assured a more normal environment for the employees (stouthuysen, swiggers, reheul, & roodhooft, 2010). the authors of both studies conclude that the tdabc can contribute to the provision of better library services at lower costs. moreover, according to the authors, the tdabc approach looks promising for other library services, such as “cataloguing” or “reference services,” which have similar characteristics to ill and acquisitions (complex, time consuming and entailing a good deal of variation) (pernot et al, 2007; stouthuysen et al, 2010). discussion by weighing the advantages and disadvantages of tdabc and abc methods, it can be concluded that the tdabc method is better suited to university libraries. since the university libraries usually belong to the central accounting system of the university, the exploitation of the abc model would be more time-consuming, laborious, and costly, and also assumes investment by university management. the following ideas are presented as potential areas where the tdabc model could be applied in a university library. 1. many leading university libraries (especially in the baltic and nordic countries) are the subjects of the legal deposit copy act, which, on one hand, provides university libraries with means to enhance their collections; yet, in library work, it is not acceptable to consider deposit copies as sources of acquisition free of charge. on the other hand, the acquisition of deposit copies does not allow the library to suspend further acquisition, due to the strict rules imposed on the usage of deposit copies stemming from the obligation of permanent preservation – deposit copies are not subject to home lending and can be used only in the reading rooms of libraries. thus, the registering, processing, additional acquisition, and preservation of deposit copies result in large costs to libraries. 2. the growing number of new publications (books and periodicals) that need to be registered and placed on reading room shelves leads to the need for an increased number of personnel and time spent, which enhances the costs of these activities. a comparison between the in-library use, home lending, and interlibrary loan suggests that the in-library use of publications by experienced readers is the cheapest of the three, as it saves the time of the library personnel. in the case of home lending, additional time is used for documenting the loan, returning the publication, or extending the loan. however, most research libraries have acquired selfservice machines for returning and borrowing publications, which enables them to reduce the workload of librarians and cut queues in rush hours. the self-service returning machine enables patrons to return books 24 hours per day, including sundays and public holidays, when the library is closed. 3. interlibrary loan is a widely used service by libraries, which enables librarians to order publications missing in one library from other evidence based library and information practice 2011, 6.4 116 libraries, thereby providing users access to the resources of other libraries. an exact cost accounting is especially important with interlibrary loan, since the operating costs of this service and the qualification requirements of the personnel are higher than those of regular loans. in addition, interlibrary loan is a service requiring considerable amounts of time. costs of interlibrary loan enquiries differ substantially – there is a large difference between simple and more complex enquiries. the processing of an interlibrary loan enquiry might take from five minutes up to one hour, depending on how difficult it is to find the desired publication in a domestic or foreign database. thus, placing all interlibrary loan activities under the same total expenditure, without considering different types of interlibrary loan enquiries, would be inaccurate. 4. the number of enquiries submitted during periods of academic work is considerably higher compared to, for instance, the summer period, when no substantial academic work is being carried out. in order to respond to an increased need during busy periods, more staff from other departments are relocated to work at the information desk, which increases the costs in the cost pool of personnel costs. if additional workforce was not used and resources were not decreased, the quality of service would drop at busy periods. if staff was not relocated to perform other tasks during periods of low academic work, when the number of enquiries is small, the costs of processing enquiries would become unacceptably large. 5. the tdabc model provides considerable potential for exploitation in electronic libraries. once the processing of printed publications has achieved almost maximum efficiency, then attention can be focused on digital materials. for instance, negotiations regarding the purchase of licences for electronic materials can be much more timeconsuming than those for printed publications or journals. the use of some electronic materials requires specific equipment and computer workstations. the acquisition of such an infrastructure and its maintenance is obligatory for the library, and these expenses must also be considered. furthermore, the services related to electronic materials, in turn, require more highly skilled and higher paid personnel. although not trained as accountants, library managers rely on accounting information for strategic planning and operational decisionmaking. increased demands for institutional accountability with university performance and costs under increased scrutiny place library managers under increased pressure to maintain quality services while faced with decreased funding and tighter budgets. a commitment to greater efficiency requires an understanding of cost behaviour. conclusions the abc and tdabc models are both suitable for university libraries, particularly when managers are seeking the answers to specific questions, such as the costs • to acquire (order and purchase) a book, • to catalogue a book, • to provide an interlibrary loan service, • to provide reference services, • to acquire an electronic database, • and other processes. the implementation of cost accounting systems in libraries has historically been treated as a technical innovation rather than an organizational or management innovation. this means that the results of such research are largely affected by: • what methods are used to identify the time spent on activities and • how well the management of libraries or researchers were able to explain to the staff the necessity for such research. the most important consideration is that librarians are not machines that can be set at a given speed and expected to produce a uniform product. for better management of public sector organizations, it is vital to cut costs and evidence based library and information practice 2011, 6.4 117 achieve efficiency and proficiency in both the shortand long-term. the potential to activate better management practices certainly exists in almost every public sector organization, including libraries. as is evident from the studies described, the testing and implementation of the abc model is already widespread across university libraries. many library managers have decided that the activity-based costing method is the best of the existing cost analysis methods adapted for evaluating library products. however, the fact is that many managers who have tried to implement the abc in their organizations have abandoned the attempt in the face of rising costs and employees’ irritation. and last, but not least, this system can only be implemented in collaboration with the accounting department of the wider organization. in contrast, the tdabc model can be tested and implemented by departmental managers for each separate library department. the tdabc considers many aspects that affect employees’ efficiency and performance, e.g., rest periods, personal time for breaks, arrival and departure, and communication and reading unrelated to actual work performance. however, the involvement of experts in costing and economics is recommended. acknowledgements this paper is based on a paper first presented at the 6th international evidence based library and information practice conference eblip6, 27-30 june 2011, university of salford, salford, uk. the authors would like to acknowledge the work of hope lehmen for assisting with the english writing of this paper. references bishop, w. w. (1905). some considerations on the cost of cataloging. library journal, 30(1), 10-14. retrieved 20 oct. 2011 from http://babel.hathitrust.org/cgi/pt?view =image;size=100;id=mdp.390150369078 25;page=root;seq=27;num=10 ceynowa, k. (2000). activity-based cost management in academic libraries: a project of the german research association. performance measurement and metrics, 1(2), 99-114. ching, s. h., leung, m. w., fidow, m., & huang, h. l. (2008). allocating costs in the business operation of library consortium: the case study of super e book consortium. library collections, acquisitions and technical services, 32(2), 97-103. coney, d. (1952). management in college and university libraries. library trends, 1(1), 83-94. cooper, r. & kaplan, r. s. (1988). measure costs right: make the right decisions. harvard business review, 66(5), 96-103. cutter, c. a. (1877). dr. hagen’s letter on cataloguing. library journal. 1(6), 216 220. retrieved 19 oct. 2011 from: http://books.google.com/ books?id=0r0daaaayaaj&printsec =frontcover#v=onepage&q&f=false divilbiss, j. l. & self, p. c. (1978). work analysis by random sampling. bulletin of the medical library association, 66(1), 19-23. retrieved 19 oct. 2011 from: http://www.ncbi.nlm.nih.gov/pmc/arti cles/pmc225293/pdf/mlab00085 0043.pdf ellis-newman, j. (2003). activity-based costing in user services of an academic library. library trends, 51(3), 333–348. ellis-newman, j. & robinson, p. (1998). the cost of library services: activity-based costing in an australian academic library. journal of academic librarianship, 24(5), 373379. evidence based library and information practice 2011, 6.4 118 goddard, a. & ooi, k. (1998). activity-based costing and central overhead cost allocation in universities. public money and management, 18(3), 31–38. haarala, a. r. (2004, may-june). the economics of services in a changing environment. iatul annual conference, kraków, poland. retrieved 19 oct 2011 from: http://www.iatul.org/doclibrary/publi c/conf_proceedings/2004/ arja-riitta20haarala.pdf heaney, m. (2004). easy as abc? activity based costing in oxford university library services. the bottom line, 17(3), 93–97. harris, g. (1989). historic cataloging costs, issues and trends. the library quarterly, 59(1), 1-21. international organization for standardization. (2006). iso 2789:2006: information and documentation— international library statistics. geneva, switzerland: international organization for standardization. kaplan, r. s. & anderson, s. r. (2004). time driven activity-based costing. harvard business review, 82(11), 131-138. kaplan, r. s. & anderson, s. r. (2007). time driven activity-based costing: a simpler and more powerful path to higher profits. boston: harvard business school press. karu, s. (2008). kulude juhtimine ja arvestus tulemuslikkusele suunatud organisatsioonis (cost management and accounting in performance-oriented organization). tartu: rafiko. kollöffel, j. & kaandorp, a. (2003). developing a cost/benefit financial model for hybrid libraries. serials: the journal for the serials community, 16(1), 41-49. retrieved 20 oct. 2011 from:http://uksg.metapress.com/medi a/d62yfjyvmr7yu4kmexej/contribution s/f/y/7/v/fy7vucccdhk2w4gp.pdf logsdon, r. h. (1954). time and motion studies in libraries. library trends, 2(3), 401-409. lynch, b. p. (1979). libraries as bureaucracies. library trends, 27(3), 259-268. masterson, w. a. j. (1976). work study in a polytechnic library. aslib proceedings, 28(9), 288 – 304. mick, c. k. (1979). cost analysis of information systems and services. in m. e. williams, (ed.), annual review of information science and technology. (pp. 37-64). white plains, ny: knowledge industry publications. morsch, l. m. (1954). scientific management in cataloging. library trends, 2(3), 470 483. mosborg, s. f. (1980). measuring circulation desk activities using random alarm mechanism. college and research libraries, 41(5), 437-444. pernot, e., roodhooft, f. & van den abbeele, a. (2007). time-driven activity-based costing for interlibrary services: a case study in a university. the journal of academic librarianship, 33(5), 551 560. poll, r. (2001). dollars with sense: analysing costs in libraries. the bottom line: managing library finances, 14(3), 184-191. reichmann, f. (1953). costs of cataloging. library trends, 2(2), 290-317. rider, f. (1936). library cost accounting. the library quarterly, 6(4), 331-381. roberts, s. a. (2003). financial management of libraries: past trends and future evidence based library and information practice 2011, 6.4 119 prospects. library trends, 51(3), 462 493. spedding, t. a. & sun, g. q. (1999). application of discrete event simulation to the activity based costing of manufacturing systems. international journal of production economics, 58(3), 289-301. spencer, c. c. (1971). random time sampling with self-observation for library cost studies: unit costs of interlibrary loans and photocopies at a regional medical library. journal of american society for information science, 22(3), 153-160. stouthuysen, k., swiggers, m., reheul, a.-m. & roodhooft, f. (2010). time-driven activity-based costing for a library acquisition process: a case study in a belgian university. library collections, acquisitions, and technical services, 34(2-3), 83-91. türk, k. (2005). inimressursi juhtimine (human resource management). tartu : tartu ülikooli kirjastus [tartu: tartu university publishing]. http://tallinn.ester.ee/record=b2082082 ~s1*est wegmann, g. (2009). the activity-based costing method: development and applications. the icfai journal of accounting research, 8(1), 7-22. whitney, j. l. (1885). on the cost of cataloging. library journal 10(jan-dec), 214-216. retrieved 20 july 2010 from library journal digital archive: http://babel.hathitrust.org/cgi/pt?u=1& num=214&seq=2&view=image&size=1 00&id=mdp.39015036908682 / evidence based library and information practice/ / conclusions microsoft word es_lewis.doc evidence based library and information practice 2007, 2:1  131 evidence based library and information practice       evidence summary    critical care nurses on duty: information‐rich but time‐poor    a review of:  mcknight, michelynn. “the information seeking of on‐duty critical care nurses: evidence  from participant observation and in‐context interviews.” journal of the medical library  association 94.2 (apr. 2006): 145‐51.    reviewed by:   suzanne lewis   library manager, central coast sector, northern sydney central coast health   gosford, new south wales, australia  e‐mail: slewis@nsccahs.health.nsw.gov.au      received: 30 november 2006    accepted: 29 january 2007      © 2007 lewis. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to describe critical care nurses’  on‐duty information‐seeking behavior.    design – participatory action research using  ethnographic methods.    setting – a twenty‐bed critical care unit in a  275‐bed community (non‐teaching) hospital.    subjects – a purposive sample of six  registered nurses (rns) working shifts in  the critical care unit.    methods – the researcher accompanied six  rns on various shifts (weekdays and  weekends, day and night shifts) in the  critical care unit and used participant  observation and in‐context interviews to  record fifty hours of the subjects’  information‐seeking behavior. transcripts  were written up and checked by the subjects  for accuracy and validity. the resulting rich  data was analyzed using open coding  (concepts which emerged during data  gathering, for example “nurse’s personal  notes”); in vivo coding (participant‐supplied  concepts, for example “reading on duty”);  and axial coding (hierarchical, researcher‐ developed concepts such as “information  behaviors, information sources, information  uses, and information kinds”) (147).    main results – the critical care nurses  constantly sought information from people  (patients, family members, other health care  workers), patient records, monitors, and  other computer systems and noticeboards,  but very rarely from published sources such  as books or online databases. barriers to  information acquisition included equipment  failure, illegible handwriting, unavailable  people, social protocols (for example  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:1  132 physician – nurse interaction), difficult  navigation of computer systems, and  mistakes caused by simultaneously using  multiple complex systems.    conclusion – critical care nurses’  information behavior is strongly patient‐ centric. knowledge‐based information  sources are rarely consulted on duty due to  time constraints and the perception that this  would take time away from patient care. in  seeking to meet the knowledge‐based  information needs of this group, librarians  should be wary of traditional, academic  models of information delivery. instead,  they should consider a tailored ready  reference service incorporating quality and  quantity filtering.    commentary    hospital librarians are often frustrated that  their information literacy training programs  and library marketing strategies fail to bring  nurses into the library, either physically or  online. instead of sending out yet another  survey to clinical staff to find out why they  did not use the library or its resources, the  author of this article spent time in the  critical care unit of a community hospital  observing the information‐seeking behavior  of nursing staff. the answer was obvious:  nurses do not have time to search for,  retrieve, or review health literature while on  duty.    knowledge‐based information with which  librarians are mainly concerned is only one  small part of the information spectrum for  the subjects of this study. the on‐duty  critical care nurses’ observed information  behavior was patient‐centric and sought  information from patients, patients’ families,  charts, computer systems, monitoring  equipment and other health workers.  information sought was also patient‐specific  (blood tests, vital signs, medications  administered), social, and logistic. only  once, during the fifty hours of observation,  did the researcher see one of the subjects  seeking knowledge‐based information from  the reference books and internet access  available in the unit.    the author has chosen a focused  ethnographic methodology which is  appropriate to the aim of the research.  ethnography has been defined as “a set of  research methods and an associated  conceptual stance developed and used by  anthropologists for investigating  uncontrolled real‐world settings” (forsythe,  402). focused ethnography is usually  conducted with a cultural sub‐group (in this  case critical care nurses in a community  hospital) and is used to obtain information  on a specific topic (information‐seeking  behavior). the ethnographic researcher is  often regarded as a participant observer,  conducting research with the subjects rather  than on the subjects. this is certainly the  case here where the author often helped the  nurses with small tasks and has,  appropriately, written the report in the first  person. however, the article’s voice is  highly subjective and would have benefited  from a more objective tone and omission of  phrases such as “my experience” and  “surprised by this.”     strategies used to ensure reliability of the  recorded data included transcription within  24 hours, review of transcripts by the  participants, and coding of the data by the  researcher herself to preserve in‐context  understanding. using participant  observation as well as unstructured  interviews offsets the potential unreliability  of self‐reported data alone. it would have  been helpful to have the codes used to  analyze the data included in an appendix.  the software used in the study is identified  as non‐numerical unstructured data  indexing searching and theorizing  (nudist) qualitative research software and  its later version n6.  evidence based library and information practice 2007, 2:1  133 possible sources of bias affecting the validity  of the study derive from both the researcher  and the subjects. the researcher declares her  conceptual stance – “a belief in the  importance of published literature for health  care providers” (147). the fact that she is not  a nurse was both a limitation of the study (a  nurse may have had more insight into other  nurses’ behavior) and a strength, as she was  not distracted by issues of patient care. the  sample size was small (only six subjects)  and was purposive, that is, selected by the  investigator herself. the sample was  additionally biased due to self‐selection,  with at least one nurse volunteering for the  study after seeing the researcher observing  another nurse. the researcher was initially  aware of some “performance” behavior due  to the presence of an observer. however this  behavior was not sustainable because of the  intense and demanding workload of the  nurses.    the ethnographic research methodology  used in this study highlights the importance  of context for understanding the  information needs of particular groups of  potential or actual library patrons. in the  sense that this study describes the behavior  of one group of people at one time in one  setting, the results cannot be widely applied.  however the study’s value lies partly in the  fact that the researcher has chosen nurses as  subjects and a community hospital as the  setting rather than following previous  studies identified in the literature review,  which tend to focus on the information‐ seeking behavior of doctors or students in  an academic (teaching hospital) setting. this  article is thus likely to be of interest to  health librarians in smaller, community  (non‐teaching) hospitals and health centers.    this study demonstrates that “no one can  retrieve reliable literature and systematically  review it while watching monitors, checking  on patients, administering and verifying  therapies, and answering telephone calls”  (150). it is a reminder to all librarians that  the reason their patrons (or potential  patrons) are not using the library may be  that they are just too busy, and that  traditional models of information delivery  may not “fit” real‐life conditions.    the author suggests a few strategies to  deliver information in clinical settings but  the discussion could usefully have  expanded on two points mentioned briefly  in the article. the first is the finding that  “the critical care nurses’ information seeking  often did not take the form of a syntactic  question or an articulated query” (148). this  suggests a role for librarians in training  nurses to formulate answerable clinical  questions. the second is the observation that  nurses’ “typical conversations with doctors  of pharmacy were longer than their  conversations with doctors and doctors of  osteopathy” (148). more detailed  information on these particular information‐ seeking interactions would have been  valuable in light of the author’s claim that  librarians are better able to provide the  knowledge‐based information services that  hospital‐based pharmacists are already  providing (150).    nevertheless, this study is valuable in  highlighting the challenge facing many  health librarians to develop innovative  strategies for delivery of focused, filtered,  high quality information to their time‐poor  patrons.     works cited    forsythe, diana e. “using ethnography  to investigate life scientists’  information needs.” bulletin of the  medical library association 86.3  (jul. 1998): 402‐9.  ebl 101   an introduction to critical appraisal   virginia wilson shirp coordinator health sciences library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   originally published in: evidence based library and information practice, 5(1), 155–157. https://ejournals.library.ualberta.ca/index.php/eblip/article/view/7458/6438     received: 30 jan. 2010   accepted: 05 feb. 2010      2016 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   last time, this column focused on librarians conducting research as part of step 2 – find the evidence – in evidence based library and information practice, and what that might mean for the individual and the profession. it’s time to move to step 3 – appraise the evidence. when you’ve looked to the literature and have found some research articles that just might inform your practice, the next step is to look critically at these articles to assess whether or not the research could be used to improve practice or help inform decision making. critical appraisal is the process of assessing and interpreting evidence by systematically considering its validity, reliability, and applicability for quantitative articles, and its credibility, transferability, dependability, and confirmability for qualitative articles. critical appraisal is a developable skill that uses explicit methods.   this process may sound daunting, but i would argue that we do this in our everyday lives to some extent. reading newspapers, watching television, scanning our rss feeds…we process what we read, see, and learn through our own experiences, and we don’t take everything at face value. critical thinking is a necessity in a world where “information overload” is not just a catch phrase. there has to be some way to filter what is out there so that parts of it can be useful to the individual and not just so much white noise. this process, with added rigor, is the next step in the practice of ebl.   there are a few criteria that we either implicitly or explicitly use to determine what is worth reading. there’s the interest factor: are we interested in the topic of the article as outlined by the abstract or the title? then there are extrinsic factors: have we heard of the article? is it published in a peer reviewed journal? is the author from a reputable institution? then there are the intrinsic factors, and these are what the critical appraisal process can tease out. these factors “relate to research design and aspects of methodology” (booth & brice, 2004, p. 105).   so to that end, let’s look at some term definitions:   ·         validity: “the extent to which the results of the research are likely to be free from bias” (booth & brice, 2004, p. 105).   ·         reliability: “the likelihood that this study could be reproducible, rather than the results just being a ‘fluke’ or chance result” (booth & brice, 2004, p. 105) ·         applicability: “the extent to which the results are likely to impact on practice” or be able to effectively inform decision making (booth & brice, 2004, p. 105).   a good round of critical appraisal can help to determine the presence (or not) of these terms, and thus the usability of the research article. qualitative studies are no less relevant to libraries. trisha greenhalgh and rod taylor have an excellent article entitled “how to read a paper: papers that go beyond numbers (qualitative research).” in it, they outline the questions to ask when reading a paper based on qualitative analysis.   so luckily, the tools are out there to aid in the critical appraisal process. in addition to the greenhalgh & taylor article, there have been several checklists developed to guide the reader through research articles in a systematic way.   ·         lindsay glynn created a useful checklist especially for library and information science. this checklist can be downloaded here: http://ebltoolkit.pbworks.com/f/eblcriticalappraisalchecklist.pdf . ·         the cristal checklists for various types of studies (appraising a user study: http://www.shef.ac.uk/scharr/eblib/use.htm and appraising an information needs analysis: http://www.shef.ac.uk/scharr/eblib/use.htm). ·         reliant is the readers’ guide to the literature on interventions addressing the need for education and training: http://eprints.rclis.org/7163/1/reliant__final_.pdf ·         tools were developed through the critical appraisal skills program (casp) in the uk: http://www.phru.nhs.uk/pages/phd/resources.htm   it might seem like there are just too many checklists and tools to wade through to get started. however, i suggest that you choose one checklist to start with. personally, i recommend the one created by glynn to get you going. then sit down with the list and the article and work through them both just to get a feel for the process. you might not be able to answer some of the questions on the checklist, which is alright. just look at the article as objectively as you can, and a useful assessment of the article will start to emerge.   as i mentioned before, critical appraisal is a skill, and it can be honed through practice. there are a couple of things that you can do to increase your skill level. journal clubs are a good way to practice critical appraisal in a collegial setting.  a journal club is a group of individuals who meet regularly to critically evaluate articles in particular practice settings. a journal club was recently started in the health sciences library at the university of saskatchewan. health sciences librarians gather to discuss a previously chosen research article in a critical fashion. this process increases our understanding of research methods, keeps us abreast of research in our field, and provides a comfortable atmosphere in which to expand our critical appraisal skills.   as well, the evidence summaries found in each issue of evidence based library and information practice are all examples of the critical appraisal process. koufogiannakis (2006) states that “reading critical appraisals informs us all of the questions we should be asking when we approach a research article and allows us to become more familiar with a critical approach to reading the literature of our field” (p. 81).   undertaken as a step in the process of evidence based library and information practice, or as a stand alone endeavor, critical appraisal is a worthwhile activity for information professionals. bringing your own knowledge and experience to the table in combination with research in our field creates a bridge that spans the research/practice gap.   next time, a look at applying research evidence to practice, and in particular, applicability: what is it? how do you find it?   references   booth, a. & brice, a. (2004). appraising the evidence. in a. booth & a. brice (eds.) evidence-based practice for information professionals: a handbook (pp. 104-118). london: facet.   booth, a. & brice, a. (n.d.).  appraising a user study. in cristal online. retrieved from http://www.shef.ac.uk/scharr/eblib/use.htm   booth, a. & brice, a. (n.d.).  appraising an information needs analysis. in cristal online. retrieved 20 feb. 2010 from http://www.shef.ac.uk/scharr/eblib/needs.htm   critical appraisal skills program. (2007). appraisal tools. in public health resource unit. retrieved 20 feb. 2010 from http://www.phru.nhs.uk/pages/phd/resources.htm   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399.   greenhalgh, t. & taylor, r. (1997). how to read a paper: papers that go beyond numbers (qualitative research). bmj, 315(7110), 740-743.   koufogiannakis, d. (2006). small steps forward through critical appraisal. evidence based library and information practice, 1(1), 81-82.   koufogiannakis, d., booth, a., & brettle, a. (2005). reliant: reader’s guide to the literature on interventions addressing the need for education and training. lir, 30(94), 44-51.   evidence summary   engineering students and professionals report different levels of information literacy needs and challenges   a review of: phillips, m., fosmire, m., turner, l., petersheim, k., & lu, j. (2019). comparing the information needs and experiences of undergraduate students and practicing engineers. the journal of academic librarianship, 45(1), 39-49. https://doi.org/10.1016/j.acalib.2018.12.004   reviewed by: kimberly mackenzie research data and scholarly communications librarian university of massachusetts medical school worcester, massachusetts, united states of america email: kimberly.mackenzie@umassmed.edu   received: 16 sept. 2019                                                                 accepted:  17 jan. 2020      2020 mackenzie. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29654     abstract   objective – to compare the levels of information literacy, needs, and challenges of undergraduate engineering students with those of practising engineers.   design – electronic survey.   setting – large land grant university in the midwestern united states and multiple locations of a global construction machinery manufacturing company (locations in asia pacific, europe, north america).   subjects – engineering undergraduates and full-time engineers.   methods – two voluntary online surveys distributed to (a) students in two undergraduate engineering technology classes and one mechanical engineering class; and (b) to engineers in an online newsletter. none of the questions on the survey were mandatory. because the call for practising engineers generated a low response rate, direct invitations were sent in batches of 100 to randomly selected engineers from a list provided by the human resources department of the company participating in the study. the surveys were similar but not identical and included multiple choice, likert scale, and short answer questions. data analysis included two-sided unpaired sample t-tests (quantitative data) and deductive and inductive content analysis (qualitative data).   main results – there were 63 students and 134 professional engineers among the respondents. survey response rates were relatively low (24.3% for students; approximately 4.5% for employees). students rated themselves higher overall and significantly higher than did engineers on the questions “know where to look for information” (students m = 5.3; engineers m = 4.2) and “identifying the most needed information” (students m = 5.5; engineers m = 4.8) (mean values reported on a 7-point scale). neither group rated themselves highly on “reflecting on how to improve their performance next time” or “having a highly effective structure for organizing information,” though engineers in north america rated themselves significantly higher than those in asia pacific on organizing information, knowing where to look for information, and using information to make decisions.   both students and engineers reported often using google to find information. the library was mentioned by one-half of engineers and one-third of students. engineers reported consulting with peers for information and making more use of propriety information from within their companies, while students reported using youtube videos and online forums, as well as news and social media. more than half of students (57%) reported having enough access to information resources, while 67% of engineers felt that they lacked sufficient access. the most common frustration for both groups was locating the information (45% of student responses; 71% of engineer responses). students reported more frustration with evaluating information (17%) compared to engineers (9%).   conclusion – engineering students and professional engineers report differences in their levels of confidence in finding information and differences in the complexity of the information landscape. engineering librarians at the university level can incorporate this knowledge into information literacy courses to help prepare undergraduates for industry. corporate librarians can use this information to improve methods to support the needs of engineers at all levels of employment.   commentary   information literacy education for undergraduate engineering students does not necessarily prepare them with the information gathering skills they will need as professional engineers because the academic environment differs from the corporate environment, particularly in terms of complexity and faster pace (leiss & ludwig, 2018). while academic librarians may have limited opportunities to educate engineering students in information literacy, recent research, including this study, suggests that time may be best spent focusing on literacy skills that will be needed in their post-university careers, such as accessing and evaluating a variety of information (i.e., grey literature and standards). this survey adds to the growing body of literature on this topic by analyzing information literacy skills in order to understand how students could be better prepared for professional challenges as well as to improve information and resource access at the professional level.   boynton and greenhalgh’s (2004) critical survey appraisal tool was consulted for this review. the surveys used in this study included variations on questions from the self-directed information literacy scale (fosmire, douglas, van epps, purzer, & fernandez, 2018). respondents in the reviewed study were similarly asked to consider their responses in relation to a recent engineering project they had undertaken. based on the boynton and greenhalgh (2004) criteria, the likert scale questions were appropriately phrased (phillips, fosmire, petersheim, & turner, 2016). however, there were some differences between the two surveys that makes direct comparison for certain questions difficult. for example, both surveys included the question, “how did you go about acquiring the information, skills, or abilities you needed to complete the project?” but the choices for answers were different for the two groups studied. only students were given a multiple choice answer including youtube/videos and online forums, while engineers would have had to comment on them in short answers.   one potential limitation of the study, in addition to low response rates, is the ability of results to be generalized. the sampling method included student respondents attending one university and engineers employed by one corporation, albeit in various locations. additional demographic details might be included in future studies; here, while years of employment for engineers was reported, the age of the employee, or years since finishing university, were not. therefore, it is difficult to judge whether changes over time in technology and online information gathering were a source of challenge or frustration. for students, 97% were male. it is not clear whether these students had prior information literacy training.   students tended overall to rate themselves higher than professional engineers on most questions, but they could be overestimating their abilities, as noted by the authors (p. 46). however, there is no direct measure of their abilities or the success of the projects they were reporting on, therefore it is unclear whether their estimations were justified. bandyopadhyay (2013) found a similar overestimation of abilities in undergraduate biology students, but also included a measure of actual skill level (which was lower than perceived skill level).   nevertheless, these results can be useful for any engineering librarian. at the university level, these results can help librarians consider ways to improve information literacy curricula, particularly the complexity of information needs undergraduates may experience as practising engineers. it is clear from the differences in survey responses that undergraduates may not have a clear understanding of the types of resources a professional engineer may need, such as internal documentation. corporate engineering librarians can use these survey results when designing programs for new employees, as well as an impetus for increasing the number of tutorials and help aids in order to help users locate the information they need. despite the shortcomings mentioned above, the survey itself could be a useful tool for librarians wishing to design a similar study to determine the needs of their users. however, particularly when surveying undergraduates, a measure of actual skill level in conjunction with self-perceptions may be more useful than the survey alone. these results have meaning beyond engineering librarians, and similar surveys could be used regardless of the nature of the corporate library or academic specialty.   references   bandyopadhyay, a. (2013). measuring the disparities between biology undergraduates’ perceptions and their actual knowledge of scientific literature with clickers. journal of academic librarianship, 39(2), 194–201. https://doi.org/10.1016/j.acalib.2012.10.006   boynton, p. m., & greenhalgh, t. (2004). hands-on guide to questionnaire research: selecting, designing, and developing your questionnaire. bmj, 328(7451), 1312-1315. https://doi.org/10.1136/bmj.328.7451.1312   fosmire, m., douglas, k. a., van epps, a. s., purzer, s., & fernandez, t. m. (2018). self-directed information literacy (sil) scale. retrieved from purdue university research repository: https://doi.org/10.4231/r790221g   leiss, c., & ludwig, p. (2018, june). engineering graduates at work: reality check for information literacy. libraries for the future: inspiring spaces to open science. paper presented at the meeting of iatul, oslo, norway. retrieved from https://docs.lib.purdue.edu/iatul/2018/infolit/3/   phillips, m., fosmire, m., petersheim, k., & turner, l. (2016). survey protocols to investigate the information habits and needs of engineering and engineering technology students and practicing engineers. libraries faculty and staff creative materials, paper 17. retrieved from https://docs.lib.purdue.edu/lib_fscm/17/ research article   interiors, affect, and use: how does an academic library’s learning commons support students’ needs?   erica defrain assistant professor, social sciences librarian university of nebraska-lincoln nebraska, united states of america email: edefrain2@unl.edu   miyoung hong senior lecturer, interior design indiana university bloomington indiana, united states of america email: miyhong@iu.edu   received: 14 nov. 2019                                                             accepted: 31 mar. 2020      2020 defrain and hong. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29677     abstract   objective – this study sought to identify the learning needs, satisfaction levels, and preferences of students using an academic library’s learning commons. a particular focus was understanding whether the socio-collaborative environment facilitated by the learning commons was aligned with the institutional objectives of supporting intensive study and scholarly work.   methods – a mixed methods sequential explanatory study design was used, in which quantitative findings were supplemented by qualitative findings. data for the study were drawn from 59 hours of observations documenting behaviors of 9,249 individuals, as well as survey responses from 302 students. three semi-structured focus groups with 10 students were held to discuss and clarify findings.   results – behavior mapping and survey data showed that students were largely satisfied with the learning commons and that it was considered a supportive environment for them to complete their stated tasks. incongruity was observed between the learning commons’ intended and actual use; although 75% of spaces were designated for collaboration, 50% of survey respondents identified independent work as their primary task and 76% of individuals were observed working independently. in focus group discussions, students praised the space for its vibrant ambiance and facilitation of social connections, but acknowledged that more serious study required retreat into quieter spaces found elsewhere in the library.   conclusion – the learning commons is an important and desirable space for students, providing a safe and community-oriented environment that is located in the center of campus. while students deemed the atmosphere successful for fostering social relationships and creating an overall sense of belonging, care needs to be taken to maintain a proper balance between quiet and collaborative spaces. the methods used in this study underscore the importance of gathering data from multiple sources, offering guidance to other libraries seeking to create, re-envision, and assess their learning spaces.     introduction   when foster and gibbons (2007) completed their seminal ethnographic study of students at the university of rochester’s river campus, they uncovered a rich dataset of undergraduates’ work processes that could be used to improve their library’s physical spaces and services. their report concluded by providing a broad appeal for user-centered design based on evidence interpreted in a “relevant context”:   we are designing technology, spaces, and services for an academic library, not a summer camp, a fitness center, or an airport. students may want to eat in the library, socialize in the library, and sleep in the library, and we may want to make that possible. but they can do those things elsewhere. there are somethings they can only do in the library; those things must have priority. (foster & gibbons, 2007, p. 82)   since that time, libraries have transitioned away from building collection-oriented spaces in favor of the more user-oriented learning commons, intended for collaborative social learning and overseen by a blend of campus partners offering broad student services (bailey & tierney, 2008). this model has become so common that bennett (2015, p. 215) proclaimed, “no one now plans an academic library without a learning commons.”   in december 2016, the university of nebraska-lincoln (unl), a large public university in the midwest, opened the doors to the adele hall learning commons (lc), transforming approximately 30,000 square feet of the main library’s ground floor from shelving space for over 300,000 books to seating and group study rooms for nearly 500 students. immediately becoming a bustling hub of activity located well-within the “heart” of campus, the lc borrowed heavily from what has since come to be known as the archetypal learning commons design, an open-concept, mixed-use environment, with a fireplace and coffee shop in its center. while the space’s popularity was one indicator of the positive impact of the project having been consecutively voted students’ favorite study spot in its first two years (unl libraries, 2018), it was uncertain how successful the project was in meeting the university’s goal that it “accommodate and promote intensive study and scholarly work” (fedderson, 2014).   in order to understand the lc’s success as an informal learning space, this study sought to answer three foundational research questions that aimed to 1) identify the student populations using the lc, 2) understand what students use it for, and 3) evaluate its interiors, atmosphere, and services according to the stated needs of the students. secondary data informed study design and primary data was gathered through surveys, behaviour mapping, and focus group conversations. the research findings contribute to a broader understanding of the impact of library learning spaces on learning behaviour, and how student-centered spaces and their usage can influence the academic success of students. thus, this research enables others to better anticipate the needs of their users when designing similar spaces, to evaluate the efficacy of their own library’s learning commons, and to assess how well their learning commons are fulfilling their intended purpose as a learning space.   literature review   in 2009, bennett (p. 190), a prominent voice in the literature surrounding physical library spaces, outlined four millennia of library building designs culminating in the current “learning-centered paradigm” that situates academic libraries prominently in two of four identified stages of students’ intentional learning. more recently, bennett (2015, p. 220) provided a concise conceptual framework for integrating learning into library space design to help “ensure that the things of learning, the affordances we create, such as the learning commons actually foster learning in a way that we might assess.”   while learning commons might be commonplace, meaningful evaluation according to bennett’s recommendations present a challenge. for example, during interviews with 41 lead architects, head librarians, and library consultants for 22 recent library renovation projects, head (2016, p.14) discovered that none of those involved had established success metrics during planning stages. most interviewees referenced using preand post-project occupancy levels as an indication of positive impact and overarching goal, with one library consultant asserting “we knew doubling the amount of seats in the new building would have an immediate impact on student success” (head, 2016, p. 14). in-depth post-occupancy evaluations were broadly viewed as too complicated, with one lead architect explaining:   unless you’re trying to write a paper and need some data, we don’t look at these measures. we always say we’d love to do more assessments, but the reality is they take more time and effort and by then you’ve already moved on to the next project. (head, 2016, p. 25)   despite such complications, a small body of assessment research has steadily accumulated over the past decade. empirical studies examining library learning spaces have largely been case studies relying on a combination of qualitative and quantitative methods, often employing ethnographic and anthropological tools and techniques for studying students’ behaviors (andrews, wright, & raskin, 2016; archambault & justice, 2017; asher, 2017; thomas, van horne, jacobson, & anson, 2015; trembach, blodgett, epperson, & floersch, 2019). many of these studies operated under the unconfirmed assumption of a relationship between learning commons’ users’ satisfaction and learning outcomes. this assumption was not rigorously investigated until recently, when woo, serenko, and chu (2019) identified a strong positive association between these factors. the authors tested a model tying satisfaction, information literacy instruction, and expectation disconfirmation theory (edt) to the chi wah learning commons environment at the university of hong kong, concluding that student satisfaction was indeed an appropriate measure of evaluation by stating: “psychological outcomes affected behavioral outcomes, which in turn produced a number of benefits. this shows that students’ cognitive changes could alter their behaviors, resulting in positive consequences, which is the goal of the learning commons” (woo, serenko, & chu, 2019, p. 416).   a frequent theme found in research about learning commons is a documented tension between users’ stated and observed desires for quiet and the collaborative activities emphasized by the built environment (archambault & justice, 2017; asher, 2017; jaskowiak, garman, frazier, & spires, 2019; james, 2013; walton & cunningham, 2016; whitchurch, 2009). some authors cite an inevitable evolution in learning behaviors as rationale for defending the learning commons’ collaborative environs. in opposition to yoo-lee, lee, and velez’s (2013, p. 499) own findings, in which 58 of 100 students surveyed said there were no disadvantages to having quiet spaces in a library, the authors asserted that “libraries need to understand the learning style of new generations and provide spaces like information commons or learning commons to reinforce the social aspects of learning...to create a dynamic, comfortable, and collaborative environment.” similarly, despite staff’s ample anecdotal evidence to the contrary, whitchurch (2009, p. 71) was so comfortable in the assumption that the commons must “provide a space for the new generation of college students and the manner in which they study” that he elected not to include individual users in his assessment. in another study conducted at the university of iowa, a week of observational data revealed that students used the group spaces for independent study 47% of the time; however the authors discarded the finding remarking that “this shift in pedagogies may not have yet permeated into students’ study habits” (thomas et al., 2015, p. 808).   in contrast, many authors do seem to consider students’ preference for quiet as evidence that there is an overall imbalance between individual and collaborative study spaces, particularly pertaining to libraries. while looking at libraries and student’s feelings of “homeness”, mehta and cox (2019, p. 27) remarked that individual study spaces were “highly valued” and that “academic atmospheres” should be prioritized. oliveira (2016, p. 356) summarized 12 recent studies showing strong evidence of students’ preference for quiet study spaces, reflecting that  “quietness is still highly valued by students and that individual study spaces (communal or isolated) are still being heavily used in academic libraries today.” james (2013, p. 6) documented that 78% of 6,846 users were working independently in a library’s “collaborative learning center” and as such suggested that a name change more reflective of actual use might be in order.   aims   the past decade has brought us closer to understanding how and why today’s students use library spaces, however still very little is known about how successful the learning commons model is, both for the students who inhabit these spaces and for the libraries in which they are built (asher, 2017; head, 2016). the learning commons model is nearly synonymous with collaborative learning space, yet numerous studies have recorded students’ use of the spaces as more reflective of traditional academic library atmospheres. therefore, a key aim of this study is to explicitly investigate this prevalent schism between solitary and collaborative study, and how a learning commons contributes to the student learning experience, guided by the following research questions:   1)      what populations of students are using the learning commons? 2)      what are students using the learning commons for? 3)      what are the needs and preferences of the students using the learning commons, according to a) atmosphere and environment, b) workspace features, and c) help and learning services?   methods   a mixed methods sequential explanatory study design was used, in which qualitative data was collected subsequently to quantitative, with the goal of more robustly understanding students’ behaviors and needs within the lc (creswell, plano clark, gutmann, & hanson, 2003, p. 178). this approach was selected so that the primarily quantitative findings could be refined and explained via follow-up interviews that were conducted with a purposefully selected subset of students (mccrudden & sparks, 2018). additionally, the “robust methodology seeking multiple data sets to establish a clear evidence-based assessment” employed in this study adheres to deed and alterator’s (2017, p. 56) conceptual model for including the lived experience when conducting occupancy assessments of informal learning spaces.   this study was guided by nitecki and simpson’s (2016) theoretical framework regarding library spaces, which asserts that the individual student is influenced by different layers of the environment, from the higher education context (e.g., culture and policies) to physical design (e.g., furnishings and materials), which impact individual behaviors and cognitive functions (e.g., study habits, attention, and motivation). the relationship between affect and emotion is apparent between the built environment and cognitive functioning (amedeo, golledge, & stimson, 2008; cranz & pavlides, 2013; woo, serenko, & chu, 2019), suggesting that when physical and psychological needs are satisfied, learners are more productive, focused, and able to learn. thus, for the learning commons model to fulfill its learning-centered mission, it must satisfy the multifaceted needs of its learners.   quantitative data collection and analysis   in january 2018, quantitative data was gathered through observation and an online survey distributed to students inhabiting the lc. field data collection was carried out under supervision by a team of four undergraduate student research assistants from unl’s college of architecture. all research assistants were trained in study procedures and design, and received certification for conducting human subjects research. the university’s human research and protection program reviewed and approved all instruments and procedures for this study.   fifty-nine hours of field observations were documented using a visual traffic sweep technique similar to that of given and archibald (2015), capturing the behaviors of 9,249 individuals in the lc. five distinct seating layouts in the lc were identified using autocad software, showing zone boundaries, square footage, layout, and seating capacity. ensuring a representative sample of students’ behaviours in a learning commons is complex, as space use differs dramatically throughout the day, week, and semester (asher, 2017, p. 72). as such, we identified hour-long data collection periods over a span of three weeks, in which research assistants would “sweep” the entire lc to capture students’ behaviors at peak, mid, and low-level occupancy rates. research assistants were instructed in unobtrusive observation techniques, in which the observer does not intentionally make their presence known to those being observed (given & leckie, 2003). observers used a paper-based template to physically document the locations of individuals, sizes of active collaborations, and if furniture had been rearranged. inter-reliability testing was conducted to ensure behavior mapping was consistently carried out. to delineate findings from the data, behavior maps were overlaid to visually identify activity patterns and space use (figure 1) and numeric count data was transferred into a spreadsheet for quantitative analysis.   figure 1 example overlay of zone 1 behavior observations of 2,774 individuals during 59 mid-morning visual sweeps.   the survey instrument (appendix a) was self-developed based on bennett’s (2015) conceptual model of student learning needs and post-occupancy evaluation best practices (preiser, rabinowitz, & white, 2015). the instrument included 16 7-point likert-type scale items asking students to assess the physical design aspects (spatial design, furniture layout, and indoor environmental quality), technology, services, and their own productivity within the lc. an additional six items gathered demographic data, including gender identity, major, age, and race. the instrument was developed iteratively. it was piloted with six students and a panel of faculty librarians prior to dissemination to secure content validity of the items, to establish clarity and comprehension, and to verify the time required for completion.   the web-based cross-sectional survey data were gathered using random probability sampling (hall, 2008). the lc was divided into five discrete zones based on institutionally designated use (figure 2). minimum survey quotas were set for each zone based on total seating capacity.   figure 2 floor plan of the learning commons showing furnishing and zones included in this study.   to be eligible, survey participants needed to be current students and at least 19 years of age, due to nebraska’s age of majority designation (nebraska legislature, 2018). employing the same hour-long schedules used for the observations, research assistants recruited participants by approaching individuals in each zone and asking them to complete a survey, making them aware that participants could be entered into a drawing for 1 of 3 $50 gift cards. if an individual agreed and met the criteria for participation, they were immediately asked to complete an online survey using a tablet provided by the recruiter. if an individual dissented, the assistant noted the occurrence and location before moving on. using this method, survey data was gathered from 356 students. after non-eligible participants and incompletes were removed, there was a total of 302 fully completed surveys.   for data analysis, responses were considered for the lc as a whole (n = 302) and then tested for between-group differences of respondents grouped according to zone location in the lc: 1) collaborative and relaxing space (n = 95), 2) collaborative space with mobile furniture (n = 77), 3) quiet reading room (n = 37), 4) quiet study room (n = 15), and 5) reservable group study rooms (n = 78). the survey data was analyzed using univariate statistical procedures and item responses and demographic trends were examined using cross-tabulation and frequency counts.   qualitative data collection and analysis   the survey included 1 open-response item intended to gather limited qualitative data through asking students for their input on any aspect of the lc (n = 110). after preliminary analyses of survey and observation data were conducted, three areas were identified as needing supplementary discussion; a semi-structured focus group format was identified as being the most time sensitive method for obtaining this insight. survey respondents were not identifiable, therefore a convenience sample was used in which students seated in the lc were randomly approached and asked if they would be willing to participate in the hour-long focus group. a representative sample of the overall lc population was sought during recruitment, to reflect diversity of gender identity, major, and age, again with 19 being the minimum age of participation. in total, 10 students (3 male and 7 female, comprised of 6 undergraduates and 4 graduates from a range of disciplines) participated in a series of 3 semi-structured, 1-hour focus groups, held in a private study room at the lc.   during the focus groups participants were prompted to discuss a series of open questions that were emailed in advance (appendix b). the questions focused on students’ study patterns, perceptions and opinions of physical and environmental features at the lc, and the efficacy of the learning commons model including personal use of help services. the focus groups had 1 moderator, 1 note-taker, were audio-recorded, and all participants received a $20 gift card for their time. thematic analysis was conducted, both on the survey’s open-response item, and on the notes and audio recordings taken during the focus group sessions (liamputtong, 2011).   results   rq1) what population of students are using the learning commons?   the results left no doubt that the lc is a popular space for students. throughout the 59 hours of observational data, 9,249 individuals were identified in the lc, with an average actual occupancy of 48.5% of 323 total seats. if using perceived occupancy, as preferred by other studies (foster & gibbons, 2007; james, 2013; khoo, rozaklis, hall, & kusunoki, 2016) in which spaces are considered at capacity with only 50% occupancy, the average total building perceived occupancy rate increases dramatically to over 89%, which is consistent with other studies on learning commons (archambault & justice, 2017; cha & kim, 2015; jaskowiak et al., 2019).   survey data illuminated the demographics of the students using the lc (table 1). participation largely reflected university demographics, with the majority of respondents being white (72.2%), domestic (88.08%), undergraduate (94.7%), and between 19 and 24 (95.36%) years of age. college enrollment of survey respondents was also reflective of overall university enrollment, with the greatest percentage of responses coming from students enrolled in the colleges of arts & sciences (28.15%) and business (24.5%).   survey data reflected a gender dynamic reported in similar studies (khoo et al., 2016; thomas et al., 2015), with females returning a larger number of surveys (63.25%) than males (35.76%). this is in contrast to the total student population, in which 51.8% of 26,079 enrolled students are male and 48.2% female (office of institutional effectiveness and analytics, 2018).   table 1 survey sample demographics, n = 302 demographic category n % class standing freshman 71 23.51 sophomore 91 30.13 junior 71 23.51 senior 53 17.55 graduate 15 4.97 no response 1 .33 gender female 191 63.25 male 108 35.76 no response 3 .99 age 19-20 185 61.26 21-24 103 34.11 25-30 11 3.64 31-34 2 .66 no response 1 .33 major a agricultural sciences 27 8.94 architecture 14 4.64 arts & sciences 85 28.15 business 74 24.5 college of nursing 2 .66 education & human sciences 48 15.89 engineering 30 9.93 fine & performing arts 9 2.98 journalism 13 4.30 public affairs 5 1.66 undeclared 12 3.97 unsure 3 .99 international student yes 36 11.92 no 266 88.08 a participants are allowed to select multiple majors, thus the sum exceeds sample size; percent calculated by number of respondents.   rq2) what are students using the learning commons for?   the lc was built to emphasize collaborative social interaction, consequently the dedicated quiet study space comprises less than a quarter of the total seating capacity. coursework was the key focus for nearly three quarters of survey respondents (74.9%), who reported being nearly equally divided between working independently (49.9%) or collaborating with a group (50.1%). the observational data indicates that while many students were inhabiting tables with at least one other individual, active collaborations in which students were either talking or focusing on a shared document were infrequently observed. of the 9,249 total lc inhabitants documented during the 59 hours of observations, only 23.9% of individuals were observed actively collaborating.   the lc was constructed with five distinct design typologies aiming to support a diverse array of uses and activities. while not overtly defined, the grid layout results in a unique environment for each corner, with group study rooms placed throughout the core. descriptive analyses of the survey items shows that 89.4% of respondents were moderately to extremely satisfied with the lc according to a 7-point scale (m = 6.27, sd = 0.70), and 83.5% of students reported that choice of space supported their overall productivity needs from very to extremely well on a 5-point scale (m = 4.19, sd = 0.77).   a two-way contingency table analysis of stated task and zone location revealed a significant relationship and relatively strong effect size, pearson χ2(4, n = 302) = 55.61, p < .001, cramer’s v = .43.   according to both observational and survey data, students’ behaviours and stated tasks differed by zone location. the survey results aligned with expectations (table 2); the majority of students in zones designated for collaboration indicated that they were working collaboratively (zones 2 and 5) and the majority of students in zones designated for quiet study were working independently (zones 3 and 4). despite zone 1 being designed to support more collaborative activities, the majority of students (61.2%) reported working independently within this space.   observational data revealed that less than a quarter of total individuals were seated in groups of two or more (table 3). zone 5’s group study rooms were the only zone in which a majority of students (77.6%) were observed in groups that were actively engaging in shared tasks such as conversing or sharing documents.   table 2 survey respondents’ stated tasks according to zone location, n = 302 a   zone 1 collaborative & relaxing space zone 2 collaborative space with mobile furniture zone 3 quiet reading room zone 4 quiet study room zone 5 group study rooms total n % n % n % n % n % n %   independent tasks coursework alone 56 48.3 29 32.6 28 66.7 9 52.9 14 16.9 136 39.2   non-coursework alone 7 6.0 3 3.4 5 11.9 1 5.9 2 2.4 18 5.2   relaxing alone 8 6.9 1 1.1 3 7.1 3 17.7 4 4.8 19 6.0   total 71 61.2 33 37.1 36 85.7 13 76.5 20 24.1 173 49.9   group tasks coursework with a group 30 25.9 37 41.6 6 14.3 3 17.7 48 57.8 124 35.7   non-coursework with a group 3 2.6 6 6.7 0 0 1 5.9 11 13.3 21 6.1   socializing 12 10.3 13 14.6 0 0 0 0 4 4.8 29 8.4   total 45 38.8 56 63.0 6 14.3 4 23.5 63 75.9 174 50.1   total   116   89   42   17   83   347     a participants were allowed to select multiple tasks, 3.7% selected > 1 task; percent calculated on number of respondents. row and column totals might exceed 100% due to rounding.   table 3 independent versus collaborative activities observed by zone over 59 total observations a   zone 1 collaborative & relaxing space zone 2 collaborative space with mobile furniture zone 3 quiet reading room zone 4 quiet study room zone 5 group study rooms total size n % n % n % n % n % n % 1 2252 81.2 1850 80.4 2355 97.8 237 98.3 342 22.4 7036 76.1 2 406 14.6 308 13.4 40 1.7 4 1.7 550 36.0 1308 14.4 3 84 3.0 99 4.3 12 0.5 0 0 318 20.8 513 5.6 4+ 32 1.2 44 1.9 0 0 0 0 316 20.7 392 4.2 total 2774   2301   2407   241   1526   9249   a greatest percentage of collaboration size per zone in bold. row and column totals might exceed 100% due to rounding.   rq3) what are the needs and preferences of the students in the learning commons?   atmosphere and environment   when asked to identify their most important environmental needs for productivity that day, findings aligned with cha and kim’s (2015) study. most students overwhelmingly identified amount of spaces (56.6%) and noise level (49.7%) as their top choices (figure 3).   figure 3 top environmental needs of students according to survey results (n = 302).   in focus group conversations, nearly all of the students referenced using the lc for less intensive work, during which social distractions were more welcome. the four graduate students stated that they only used the lc for academic work when in need of a group study room or when other campus buildings were closed and would otherwise move to quieter locations within the main library. of the survey respondents, 10.2% were somewhat to extremely dissatisfied with noise, and all focus group participants agreed when 1 student stated “i think all students have an understanding if you need a quiet place to get work done, then you need to not be in the learning commons. i think it’s just understood amongst everybody.” for those times when they had nowhere else to go, most focus group participants referenced using noise-blocking ear buds to help mitigate distractions in the lc. in the words of one female graduate student, “i have to use ear plugs, because otherwise i can hear everyone talking and coming in and out and everything.”   workspace features   although survey items asked participants to state their satisfaction regarding wall color, flooring, workspace, layout, and furnishing, all zones except the group study rooms include multiple interior types making it difficult to identify the most and least successful interiors from survey results. some survey respondents expressed dissatisfaction regarding each of the categories, however the overall average satisfaction rankings were high (table 4). comfort of furnishings received the most critical assessment, with 6.2% of respondents expressing slight to extreme dissatisfaction.   table 4 survey respondents’ satisfaction with interiors, n = 302 satisfaction category feature n minimum maximum mean sd layout size of workspace 302 2 7 6.14 0.94 different spaces 302 1 7 6.11 0.94 supports task 302 2 7 6.19 0.98 furniture overall comfort 302 2 7 6.03 1.04 adjust furniture 301 1 7 5.97 1.05 materials/fabric 302 3 7 5.87 1.06 colors 302 2 7 5.99 1.14 interiors wall colors 302 1 7 6.02 1.09 flooring materials 302 1 7 6.05 1.07 surface finishings 302 1 7 6.18 0.98 environment temperature 301 1 7 5.89 1.19 air quality 302 3 7 6.08 1.01 lighting 301 2 7 6.18 1.01 views to outside 301 2 7 6.4 1.02 noise level 300 1 7 5.55 1.28 amenities beverage/snack options 302 1 7 5.7 1.20 whiteboard availability 302 1 7 5.53 1.24 wifi connectivity 302 2 7 6.16 1.08 outlets/power 302 1 7 6.01 1.19 in-house tech 302 1 7 6.02 1.05 printing services 302 1 7 5.86 1.23 overall overall satisfaction 302 3 7 6.27 0.70   the ability to adjust workspaces within the lc was an important feature for students. overall, most (91.4%) were satisfied with their ability to adjust the furniture, however, 23.7% of respondents in zone 3 (quiet reading room) were slightly to extremely dissatisfied with this feature. the quiet reading room’s tables and chairs are immobile, while other zones have casters aiding the rearrangement of most tables and chairs.     figure 4 zone 2: collaborative area with mobile furniture showing furniture rearrangements throughout observations (n = 59).   when space reconfigurations were simplified via movable furniture, students took advantage of this flexibility. in zone 2, reconfiguration of furnishings was documented throughout all 59 observations (figure 4). both tables and chairs were moved frequently to enable larger collaboration sizes to create a more suitable space for inhabitants. this agency was an important feature for at least one of the male undergraduate focus group students who stated, “i’ve moved desks and chairs and all that kind of stuff and it’s very helpful. if i need a bigger table we can switch, and it works out well. i’d rather have that than them stuck in the ground.”   help and learning services   the lc, in adherence to the learning commons model, strives to be more than simply an unmediated space to study. a defining feature of a learning commons is the integration of a network of campus support services into the space with a multitude of objectives, including raising awareness of services, encouraging help-seeking behaviors, and providing barrier-free access to departments and resources aimed at enhancing student success (blummer & kenton, 2017). at the center of the lc are two connected service desks featuring the only permanently situated building partners: a library service desk, occupied mostly by library student workers, and a technology help desk staffed by campus it specialists. in addition to housing the digital learning center for test proctoring, the lc also has a mixed-use space in which a multitude of other campus services are periodically stationed such as writing tutors, career services staff, and a tutoring service called “study stop.” the students’ awareness and use of these services were measured (figure 5), with the understanding that this would indicate how relevant they felt the services were to their learning needs. as expected, students’ awareness and use of services were higher for the permanently stationed services.   figure 5 students' awareness and previous use of service desks according to survey respondents (n = 302).   in focus group conversations, students stated they had limited experience using any of the onsite support services, with the library’s service desk accounting for the lowest amount of use and familiarity among participants. one female undergraduate student who reported using the lc an average of 15 hours a week admitted, “i actually have no idea what [the library help desk] is.” another female undergraduate was enthusiastic regarding services she had received from the library help desk, but her interactions had only involved directional or operations-related requests: “i have used the [the library help desk] because i just have general questions about the layout...[or] like a marker runs out, and then temperature in the rooms, and then bringing down the blinds.” a male undergraduate student who was aware of the help desk but had never used it explained his reluctance to ask for research help by saying, “i would go ask for help from my professor, but i would not ask anybody here.” when asked to clarify, the student continued:   i would say maybe part of the reason i don't go to the [library help desk] people is because it is also another undergrad student sitting there, looking very disinterested with their job. not that i wouldn't look disinterested, but i feel like it's just another person that wouldn't really understand my problem like a professor would. so i might have to sit there and explain it to them. it would just be a lot more work talking to someone else who's sitting at that desk.   when asked to identify the three most important amenities or services for their learning needs (figure 6), few students selected any of the service desks. rather, wifi, power outlets, and food options received the highest scores, suggesting that the space itself and not the service desks is a key draw to the lc. the physical proximity to the library and its collections was a top need for nearly a quarter of survey respondents (23.8%).   figure 6 student's top services (n = 302).   discussion   the collaborative nature of a learning commons disrupts the traditional image of the quiet confines of academic library spaces. for many campuses, this has been a welcome opportunity to create a centralized communal space that has the overwhelming approval of the academic community (head, 2017). for many libraries, this has resulted in the funding of long overdue aesthetic updates, dramatically increased gate counts, and changes in perceptions of what the library’s relationship to campus is (blummer & kenton, 2017). for some, the popularity of active learning pedagogies results in a disruptive and costly imbalance between quiet and social spaces in the learning commons (andrews, wright, & raskin, 2016; james, 2013). the many positive effects of a learning commons should not be taken to mean that quiet atmospheres are no longer relevant to twenty-first century learners.   while this study’s findings detect an imbalance between the original intent of the designers and the needs of the eventual inhabitants, our students expressed overwhelming satisfaction with the lc. this satisfaction is largely attributable to the flexibility of the design, as well as the library’s willingness to allow the users to dictate the environment and establish the desirable balance between quiet and collaborative. unlike some learning commons, the lc includes a large quiet reading room (zone 3) and quiet study rooms (zone 4), resulting in approximately one quarter of seating being officially reserved for quiet study. these quiet spaces showed the highest average occupancy rates, and were identified as being intentionally sought out by students at significantly higher rates than the more social spaces. when lc inhabitants needed additional quiet study spaces, they were afforded the agency necessary to create this in zone 1. this flexibility and student-centered approach has translated into the lc’s overall popularity amongst all students; the lc is not necessarily favored by any one discipline or user type. this contrasts with asher’s (2017) discovery that humanities students accounted for only 1% of learning commons, with the key difference between the two sites being the amount of formally designated quiet study space.   libraries hoping to create a learning commons or redesign existing spaces should be very careful to identify and protect this proper balance between quiet and collaborative. while visibly active students filling academic library spaces does help to provide evidence of a library’s importance to a modern campus, students still need quiet spaces with minimal distractions to focus on their academic work. if other locations within the library exist to which students needing these spaces can retreat, emphasizing socio-collaborative environments can be a focal point. if, however, truly quiet spaces are being subsumed in the process, the ability to support the full spectrum of learning needs will be at risk.   students’ underuse of the help services in the lc needs to be better understood, not only locally but in the broader context of the learning commons model. a previous study examining gate count records confirmed that while foot traffic at the main library increased by 80% immediately following the opening of the lc, there was no discernible increase in help services (allison, defrain, hitt, & tyler, 2019, p. 309). great effort and careful consideration goes into identifying and staffing the service desks to be aligned with students’ needs. while there are numerous psychosocial reasons attributable to students’ reluctance to seek help (black, 2016), academic libraries can still play a role in not only encouraging but increasing user engagement. as found by others (asher, 2017; thomas et. al, 2015), most survey respondents expressed an awareness of the different help services available in the lc, yet only one third had voluntarily used any of them. the testing center is the university’s designated space for year-round proctored exams, making it understandable that nearly 80% of survey respondents had used the center before. with a prominent shared service desk situated in the middle of the lc that is staffed during all hours of operation, underuse cannot be attributed to poor visibility. when paired with the focus group students’ limited understanding of the purpose of the library’s service desk and their reluctance to ask a peer for help, identifying where the library’s services can be most impactful needs to be more thoroughly investigated.   limitations   a limitation for this study was the minimum age of participation for both survey and focus groups. the university is located in 1 of only 2 states in which the age of majority is 19, meaning that any research involving younger participants would require parental consent. it is unknown what percentage of students who use the lc are 18 or younger, but this age range comprised 17.5% of total enrollment for fall 2017 (office of institutional effectiveness and analytics, 2018, p. 67). turn-down data captured by our research team showed that only 12 (3.4%) of the 356 students approached were ineligible to participate due to being below the age of majority. it is unknowable, therefore, what effect the responses from this age group would have had on aggregate survey data.   an additional limitation was the small window during which the observational and survey data were collected. the ebb and flow of semester patterns and the perpetual shift in visitors cannot be precisely monitored or captured in a three-week time period. data collection occurred towards the middle of the spring semester, greatly curbing generalizability of findings.   finally, this study considered only those using and inhabiting the spaces of the lc. the most dramatic impact of this limitation is that participants were most likely students with positive views of the learning commons, as evidenced by the high overall satisfaction rates. broadening the scope of the study to identify and include the students who do not feel well-served by the learning commons would be an important step towards more critically understanding the benefits and challenges afforded by this space.   further research   the amount of data gathered throughout this study was substantial, and there are a multitude of remaining research questions that should be explored. this manuscript focused only on a small number of questions that were felt to be the most essential to initially answer. the intent is to continue the investigation through additional, more complex analyses, with a particular focus on understanding students’ satisfaction, tasks, and preferences according to various demographic variables, such as age, class standing, and major.   how the learning commons model contributes to students’ relationship with academic libraries more broadly is also of great interest. given that the learning commons aesthetic is being emulated in campus spaces outside of academic libraries (walton & matthews, 2013), continuing to investigate the variant needs of learners in these spaces across campuses could reveal the essential features and aspects of libraries’ spaces that will ensure their continued success. deed and alterator (2017) outlined a model of participatory analysis for such a complex comparative study; this will inform the next stage of this study in which four additional informal learning spaces located in close proximity to the lc will be considered in order to assist with identifying the “things” that students can only accomplish in a library, as proposed by foster and gibbons (2007, p. 82).   conclusion   understanding the learning needs of students in a learning commons is a complex, multifaceted task. through combining multiple data points, this study identified why the learning commons is such a popular space and which features are especially attractive for students. the combination of unobtrusive observation, surveying student preferences, and discussing patterns and findings in focus groups revealed enlightening insights that were critical to understanding the value of the learning commons. the results underscore the importance of enabling students to personally decide the appropriate balance between quiet and collaborative spaces, in addition to identifying the functions considered most essential for students’ needs fulfillment. proper evaluation of informal learning spaces does require considerable time and effort; however, it should become standard practice in academic libraries because it plays such an essential role in illuminating patron needs and increasing understanding of how to better engage with them.   acknowledgements   this study was funded by a grant from the maude hammond fling faculty research fellowship through the university of nebraska-lincoln’s research council. the authors would like to thank research assistants sheila cobb, lindsay meyer, nicole phinney, and savannah scoville for their assistance in data collection, and the undergraduate creative activities and research experience program.   references   allison, d., defrain, e., hitt, b. d., & tyler, d. c. (2019). academic library as learning space and as collection: a learning commons’ effects on collections and related resources and services. the journal of academic librarianship, 45(3), 305–314. https://doi.org/10.1016/j.acalib.2019.04.004   amedeo, d., golledge, r. g., & stimson, r. j. (2008). person-environment-behavior research: investigating activities and experiences in spaces and environments. new york, ny: guilford press.   andrews, c., wright, s. e., & raskin, h. (2016). library learning spaces: investigating libraries and investing in student feedback. journal of library administration, 56(6), 647–672. https://doi.org/10.1080/01930826.2015.1105556   archambault, s. g., & justice, a. (2017). student use of the information commons: an exploration through mixed methods. evidence based library & information practice, 12(4), 13–40. https://doi.org/10.18438/b8vd45   asher, a. d. (2017). space use in the commons: evaluating a flexible library environment. evidence based library & information practice, 12(2), 68–89. https://doi.org/10.18438/b8m659   bailey, d. r., & tierney, b. (2008). transforming library service through information commons: case studies for the digital age. chicago, il: american library association.   bennett, s. (2009). libraries and learning: a history of paradigm change. portal: libraries and the academy, 9(2), 181–197. http://doi.org/10.1353/pla.0.0049   bennett, s. (2015). putting learning into library planning. portal: libraries and the academy, 15(2), 215–231. https://doi.org/10.1353/pla.2015.0014   black, s. (2016). psychosocial reasons why patrons avoid seeking help from librarians: a literature review. the reference librarian, 57(1), 35-56. https://doi.org/10.1080/02763877.2015.1096227   blummer, b., & kenton, j. m. (2017). learning commons in academic libraries: discussing themes in the literature from 2001 to the present. new review of academic librarianship, 23(4), 329–352. https://doi.org/10.1080/13614533.2017.1366925   cha, s. h., & kim, t. w. (2015). what matters for students’ use of physical library space? the journal of academic librarianship, 41(3), 274–279. https://doi.org/10.1016/j.acalib.2015.03.014   cranz, g., & pavlides, e. (2013). environmental design research: the body, the city, and the buildings in between (3rd ed.). san diego, ca: cognella academic publishing.   creswell, j. w., plano clark, v. l., gutmann, m. l., & hanson, w. e. (2003). advanced mixed methods research designs. in a. tashakkori & c. teddlie (eds.), handbook on mixed methods in the behavioral and social sciences. (pp. 209-240). thousand oaks, ca: sage publications.   deed, c., & alterator, s. (2017). informal learning spaces and their impact on learning in higher education: framing new narratives of participation. journal of learning spaces, 6(3), 54-58. retrieved from http://libjournal.uncg.edu/jls/article/view/1439/1165   fedderson, t. (2014, july 10). learning commons project begins. in nebraska today; university communications. retrieved from https://news.unl.edu/newsrooms/today/article/learning-commons-project-begins/   foster, n. f., & gibbons, s. (eds.). (2007). studying students: the undergraduate research project at the university of rochester. chicago, il: association of college and research libraries.   given, l. m., & archibald, h. (2015). visual traffic sweeps (vts): a research method for mapping user activities in the library space. library & information science research, 37(2), 100–108. https://doi.org/10.1016/j.lisr.2015.02.005   given, l. m., & leckie, g. j. (2003). “sweeping” the library: mapping the social activity space of the public library. library & information science research, 25(4), 365–385. https://doi.org/10.1016/s0740-8188(03)00049-5   hall, j. (2008). cross-sectional survey design. in p. lavrakas (ed.), encyclopedia of survey research methods. (p. 173). thousand oaks, ca: sage publications. https://doi.org/10.4135/9781412963947.n120   head, a. j. (2016). planning and designing academic library learning spaces: expert perspectives of architects, librarians, and library consultants. in project information literacy research report: the practitioner series. retrieved from https://eric.ed.gov/?id=ed571288   institutional effectiveness and analytics. (2018). in unl graduation and retention rates. retrieved from https://iea.unl.edu/   jaskowiak, m. a., garman, k., frazier, m., & spires, t. (2019). we’re all in this together: an examination of seating and space usage in a renovated academic library. library philosophy and practice, 1-17. retrieved from https://digitalcommons.unl.edu/libphilprac/2645/   james, r. m. (2013). culture war in the collaborative learning center. journal of learning spaces, 2(1). retrieved from http://libjournal.uncg.edu/jls/article/view/502/377   khoo, m. j., rozaklis, l., hall, c., & kusunoki, d. (2016). “a really nice spot”: evaluating place, space, and technology in academic libraries. college & research libraries, 77(1), 51–70. https://doi.org/10.5860/crl.77.1.51   liamputtong, p. (2011). focus group methodology: principles and practice. london: sage publications ltd. https://doi.org/10.4135/9781473957657   mccrudden, m. t., & sparks, p. c. (2018). an explanatory sequential mixed methods investigation: how do pre-reading instructions affect adolescents’ beliefs and belief justifications about a controversial issue? sage research methods cases. https://doi.org/10.4135/9781526431912   mehta, p., & cox, a. (2019). at home in the academic library? a study of student feelings of “homeness”. new review of academic librarianship, 1-34. https://doi.org/10.1080/13614533.2018.1547774   nebraska legislature. (2018). nebraska revised statute 43-2101, 43–2101 § 2101. retrieved from https://nebraskalegislature.gov/laws/statutes.php?statute=43-2101   nitecki, d. a., & simpson, k. (2016). communicating the library as a learning environment. journal of learning spaces, 5(2), 14. retrieved from http://libjournal.uncg.edu/jls/article/view/1318   oliveira, s. m. (2016). space preference at james white library: what students really want. the journal of academic librarianship, 42(4), 355–367. https://doi.org/10.1016/j.acalib.2016.05.009   preiser, w. f. e., rabinowitz, h. z., & white, e. t. (2015). post-occupancy evaluation. new york, ny: routledge.   thomas, b., van horne, s., jacobson, w., & anson, m. (2015). the design and assessment of the learning commons at the university of iowa. the journal of academic librarianship, 41(6), 804–813. https://doi.org/10.1016/j.acalib.2015.09.005   trembach, s., blodgett, j., epperson, a., & floersch, n. (2019). the whys and hows of academic library space assessment: a case study. library management, 41(1), 28-38. https://doi.org/10.1108/lm-04-2019-0024   office of institutional effectiveness and analytics (2018). university of nebraska-lincoln fact book 2017-2018. retrieved from https://iea.unl.edu/publications/fb17_18.pdf   unl libraries. (2018). libraries blog: the adele coryell hall learning commons is transforming teaching and learning. libraries blog. retrieved from http://unllibraries.blogspot.com/2018/03/the-adele-coryell-hall-learning-commons.html   walton, g., & cunningham, m. (2016). informal learning spaces (ils) in university libraries and their campuses: a loughborough university case study. new library world, 117(1/2), 49–62. https://doi.org/10.1108/nlw-04-2015-0031   walton, g., & matthews, g. (2013). evaluating university’s informal learning spaces: role of the university library? new review of academic librarianship, 19(1), 1-4. https://doi.org/10.1080/13614533.2013.755026   whitchurch, m. j. (2009). evaluating group use of the information commons. college & undergraduate libraries, 16(1), 71–82.  https://doi.org/10.1080/10691310902754130   woo, e. m., serenko, a., & chu, s. k. (2019). an exploratory study of the relationship between the use of the learning commons and students' perceived learning outcomes. the journal of academic librarianship, 45(4), 413-419. https://doi.org/10.1016/j.acalib.2019.05.007   yoo‐lee, e., heon lee, t., & velez, l. (2013). planning library spaces and services for millennials: an evidence‐based approach. library management, 34(6/7), 498–511. https://doi.org/10.1108/lm-08-2012-0049     appendix a survey instrument   q1. what are you primarily here for today? (click all that apply) 1.      i’m working on coursework alone 2.      i’m working on coursework with a group 3.      i’m working on non-course related activities alone 4.      i’m working on non-course related activities with a group 5.      i’m mostly socializing 6.      i’m mostly relaxing 7.      something else: ________     q2. why did you choose this particular location in the learning commons? (click all that apply) 1.      it was the only available space 2.      i specifically wanted a seat in this area 3.      it was the first available space i saw 4.      someone else chose it 5.      something else: __________     q3. how long are you planning on staying during today's visit? 1.      less than 30 minutes 2.      30 minutes to less than 1 hour 3.      1 to less than 2 hours 4.      2 to less than 3 hours 5.      more than 3 hours     q4. on average, how often have you come to the learning commons this semester? 1.      7 days a week 2.      4 6 days a week 3.      2 3 days a week 4.      1 day a week 5.      less than 1 day a week 6.      this is my first visit     q5. refer to the map to answer the following: how familiar are you with the following features of the different zones in the learning commons?     before this survey, i was aware of this i have used this zone before   yes no yes no zones 1 & 2 are intended for collaborative work and social interaction o   o   o   o   zones 3 & 4 are intended for individual quiet study o   o   o   o   zone 5 is intended for groups of two or more students o   o   o   o       q6. how familiar are you with the following services in the learning commons?       q7. thinking about the space you are currently in, how satisfied are you regarding the layout of each of the following:     extremely satisfied moderately satisfied slightly satisfied neither satisfied nor dissatisfied slightly dissatisfied moderately dissatisfied extremely dissatisfied size of my personal workspace o   o   o   o   o   o   o   different types of spaces available for use o   o   o   o   o   o   o   how well the layout supports my task for today o   o   o   o   o   o   o       q8. thinking about the space you are currently in, how satisfied are you regarding the following aspects related to the furniture?     extremely satisfied moderately satisfied slightly satisfied neither satisfied nor dissatisfied slightly dissatisfied moderately dissatisfied extremely dissatisfied overall comfort o   o   o   o   o   o   o   ability to adjust the furniture to meet my needs o   o   o   o   o   o   o   material/fabric o   o   o   o   o   o   o   colors o   o   o   o   o   o   o       q9. thinking about the space you are currently in, how satisfied are you regarding the following features of the interiors?     extremely satisfied moderately satisfied slightly satisfied neither satisfied nor dissatisfied slightly dissatisfied moderately dissatisfied extremely dissatisfied wall colors o   o   o   o   o   o   o   flooring materials o   o   o   o   o   o   o   workspace surface finishes(e.g., desktop, table) o   o   o   o   o   o   o       q10. thinking about the space you are currently in, how satisfied are you regarding each aspect of your surrounding environment?     extremely satisfied moderately satisfied slightly satisfied neither satisfied nor dissatisfied slightly dissatisfied moderately dissatisfied extremely dissatisfied temperature o   o   o   o   o   o   o   air quality (e.g., stuffy/stale air, cleanliness, odors) o   o   o   o   o   o   o   lighting o   o   o   o   o   o   o   views to the outside/windows o   o   o   o   o   o   o   noise level o   o   o   o   o   o   o       q11. thinking about the learning commons in general, how satisfied are you regarding each of the following amenities?     extremely satisfied moderately satisfied slightly satisfied neither satisfied nor dissatisfied slightly dissatisfied moderately dissatisfied extremely dissatisfied beverage and snack options o   o   o   o   o   o   o   white board availability o   o   o   o   o   o   o   wireless connectivity o   o   o   o   o   o   o   access to outlets/power o   o   o   o   o   o   o   computers and other technology resources o   o   o   o   o   o   o   print, scan, and copy services o   o   o   o   o   o   o       q12. from the list below, drag the top 3 most important environmental factors contributing to your productivity today in the learning commons into the box below:   top 3 most important environmental factors 1.      amount of the spaces 2.      different types of spaces 3.      comfort of furniture 4.      ability to adjust furniture 5.      colors and textures 6.      thermal comfort (temperature) 7.      indoor air quality (odors) 8.      visual comfort (lighting quality) 9.      views to the outside/windows 10.   noise level 11.   other:     q13. from the list below, drag the top 3 most important services or amenities contributing to your productivity today in the learning commons into the box below:   top 3 most important services or amenities 1.      access to the library and library resources 2.      computers and other technology resources 3.      white boards 4.      beverage and snack options 5.      wireless connectivity 6.      access to outlets/power 7.      library help services 8.      it help services 9.      writing center services 10.   exam commons 11.   career services 12.   study stop 13.   other:   q14. how well does your choice of space in the learning commons support your ability to get your job done today?   1.      extremely well 2.      very well 3.      moderately well 4.      slightly well 5.      not well at all q15. overall, how satisfied are you with the learning commons? 1.      extremely satisfied 2.      moderately satisfied 3.      slightly satisfied 4.      neither satisfied nor dissatisfied 5.      slightly dissatisfied 6.      moderately dissatisfied 7.      extremely dissatisfied   q16. finally, do you have any comments, suggestions, or feedback you would like to share with us about the learning commons?     appendix b focus group questions and prompts   1.      talk about your life as a student here: a.      what is your major? do you live on or off-campus? how many classes are you taking? do you work? how much time do you spend studying? b.      is there a place in your college life—outside of the classroom—where most of your academic learning occurs, e.g., library, dorm, home, coffee house, online? why is this your “go-to learning place”?   2.      what do you like best about the general layout of the learning commons? how does it help you do your job? what more would you like to see in the design? if you had that, what would that allow you to do?   3.      tell us about individual work at the learning commons. how does the physical layout impact individual work? how would any improvements in design help with individual work? which design and environmental factors interfere with individual work?   4.      tell us about the level of collaboration. is it easy to collaborate with others when you work at the learning commons? which design and environmental factors interfere with group work?   5.      how important do you feel a learning commons is for today’s college students? how important is the lc to you in regard to successful assignment completion, study habits, performance in courses, learning and acquisition of knowledge, or educational goals?   6.      do you use any of the services available at the lc? why or why not? are there other services that might be more helpful? when you are studying or working on assignments, how do you most often get help?   7.      finally, are there any additional comments or observations you would like to make about the learning commons?   evidence based library and information practice evidence based library and information practice 2011, 6.4 88 evidence based library and information practice article library instruction for freshman english: a multi-year assessment of student learning susan gardner archambault head, reference & instructional services loyola marymount university los angeles, california, united states of america email: susan.gardner@lmu.edu received: 16 aug. 2011 accepted: 28 oct. 2011 2011 gardner archambault. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/bync-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – the objective of this study was twofold: 1) to assess the effectiveness of curriculum changes made from the 2009 freshman english library instruction curriculum to the 2010 curriculum at loyola marymount university (lmu); and 2) to evaluate the effectiveness of library instruction delivered via a “blended” combination of face-to-face and online instruction versus online instruction alone. methods – an experimental design compared random samples of student scores from 2009 and 2010 worksheets to determine the effects of a new curriculum on student learning. a second experiment examined the effect of delivery method on student learning by comparing scores from a group of students receiving only online instruction against a group receiving blended instruction. results – the first component of the study, which compared scores between 2009 and 2010 to examine the effects of the curriculum revisions, had mixed results. students scored a significantly higher mean in 2010 on completing and correctly listing book citation components than in 2009, but a significantly lower mean on constructing a research question. there was a significant difference in the distribution of scores for understanding differences between information found on the internet versus through the library that was better in 2010 than 2009, but worse for narrowing a broad research topic. for the study that examined computer aided instruction, the group of students receiving only computer-assisted instruction did significantly better overall than the group receiving blended instruction. when separate tests were run for each mailto:susan.gardner@lmu.edu� evidence based library and information practice 2011, 6.4 89 skill, two particular skills, generating keywords and completing book citation and location elements, resulted in a significantly higher mean. conclusions – the comparison of scores between 2009 and 2010 were mixed, but the evaluation process helped us identify continued problems in the teaching materials to address in the next cycle of revisions. the second part of the study supports the idea that computer-assisted instruction is equally or more effective than blended instruction. introduction information literacy in higher education is defined as a set of abilities requiring individuals to "recognize when information is needed and have the ability to locate, evaluate, and use effectively the needed information” (acrl 2000). lawson (1999) wrote “although bibliographic instruction has been ongoing in libraries for many years… no single method has been established as the best” (p. 77). the association of college & research libraries’ professional standards on information literacy, introduced in 2000, have helped guide the practice of library instruction, but there is still no established “best” method. anderson & may (2010) state “essentially, one class period is inadequate to provide the necessary information to gain the il skills set forth by the acrl” (p. 496). houlson (2007) notes “one-shot workshops and orientations typically cover too much information and rely on passive learning” (p. 104). according to dewald (1999) the traditional “best practices” of good class-based library instruction dictate that it be assignment-related; include active learning exercises; accommodate more than one learning style (auditory and visual); and have clear educational objectives (pp. 26-27). she advocates that these same components be used as a guide in the design of web based instruction. dewald cautioned that online tutorials “cannot completely substitute for a human connection in learning” and they are “best used in connection with academic classes rather than in isolation” (p. 31). online or computer-assisted library instruction has several benefits over face-toface instruction, including greater immediacy, greater flexibility and convenience, instant feedback, self-pacing, and greater consistency, since every student receives the same information (holman, 2000, p.54). in most cases, the instruction remains accessible for repeat learning, if desired. however questions remain regarding the effectiveness of computer-assisted library instruction compared to face-to-face instruction at teaching undergraduates basic library skills. there is significant pressure for libraries to adopt more evidence-based practices and to measure the ways those practices are contributing to student learning within the framework of the university. with regard to library instruction, the library “must move from a content view (books, subject knowledge) to a competency view (what students will be able to do)” (smith, 2001, p. 32). oakleaf (2009) describes an information literacy instruction assessment cycle (iliac) framework for libraries to follow that consists of seven steps: identifying learning goals and their outcomes; creating and enacting learning activities; gathering and interpreting data to measure learning; and enacting decisions and changes based on that data. after step seven, the cycle starts again with new objectives. this assessment cycle provides feedback that librarians can use to improve their own skills, reflect on their teaching, and examine their assumptions about learning (oakleaf, 2009, p.541). samson notes “an assessment is only valuable when the analyses are used to augment or change the program being assessed” (p. 341). oakleaf (2006) adds “to close the loop, educators finish the assessment cycle by using data to improve teaching and learning programs (p. 51). evidence based library and information practice 2011, 6.4 90 librarians at loyola marymount university (lmu) have been examining the ways they teach information literacy to undergraduate students. following revisions to the curriculum, they were keen to establish whether the revisions made a difference as well as the methods in which the instruction was provided. this paper describes a study comprising two experiments that examines these issues. literature review using assessment to improve teaching walsh (2009) discusses the following nine methods used in case studies to assess student learning and information literacy skills following library instruction: multiple choice questionnaires, quizzes/tests, bibliographies, essays, portfolios, self-assessment, observation, simulation, and final grades. knight (2006) advocates using “authentic assessment” methods that measure how students apply their knowledge to real-time tasks and incorporate that knowledge into academic work, rather than the artificiality of traditional standardized tests that fail to measure higher order thinking (45). although many case studies describe the results of testing information literacy skills during one year, not many discuss how they used the data to improve their instruction programs or compare the results of using the same assessment technique across multiple years. an exception is scharf, elliot, huey, briller, & joshi (2007), who assessed information literacy through the writing portfolios of seniors in humanities classes. the authors used previous writing portfolios to develop information literacy variables, and then analyzed the results of that assessment to address instructional issues raised by the assessment and ways to improve the teaching of those skills. warner (2003) piloted formal assessment in 2002 on a group of pre-freshmen doing three assignments and then changed and improved teaching methods to address the learning problems she discovered for the following year. fain (2011) examined the results of using a pre-/post-test library skills assessment over a five-year period to evaluate the effectiveness of an information literacy program and make changes. burkhardt (2007) discussed results of pre-/post-tests over a fiveyear period on a three-credit undergraduate class and how improvements would be made to the teaching of low-scoring areas. finally, oakleaf (2009) discusses two rounds of an assessment cycle using a rubric and how the results were used to improve an online information literacy tutorial for undergraduates taking a required first-year writing course. face-to-face versus online learning many articles have compared the effects of face-to-face instructional delivery versus online instructional delivery on student learning, but the aggregate results are inconclusive. some researchers found no significant difference in student scores for the same teaching material delivered in traditional face-to-face format versus a computer-assisted format (germain et al., 2000; kaplowitz & contini, 1998; vander meer & rike, 1996; zhang, watson, & banfield, 2007; holman, 2000; koufogiannakis and wiebe, 2006). contrary to this, lawson (1989) found students using a tutorial performed better than face-to-face instruction, while anderson & may (2010) found online instruction yielded higher scores than blended or face-to-face instruction. madland & smith (1988) and churkovich & oughtred (2002) reported that students receiving face-to-face library instruction performed slightly better than computer-assisted learners; and kraemer, lombardo, & lepkowski (2007) reported students receiving online-only instruction scored lower than both face-to-face or blended instruction. context loyola marymount university (lmu) is a private, jesuit university in los angeles, california. the lmu freshman english program (consisting of english 110: introduction to college writing classes, required of all freshmen) is the primary evidence based library and information practice 2011, 6.4 91 avenue through which students are introduced to the library. the director of the freshman english program requires all english 110 instructors to bring their classes to the library once during the fall semester for a one-shot instruction sessions lasting 50 or 75 minutes. the reference department designed an “english 110 library worksheet” and an accompanying online five-module introduction to the research process promoted via a libguides-based guide. the first two modules of the worksheet are completed prior to the face-to-face library session as an independent homework assignment and using the libguide as a support resource. during the face-to-face session, students progress through the last three modules while completing the related sections of the worksheet. the original five student learning outcomes in the lmu freshman english library instruction program developed for 2009 are: 1. given a broad research topic, use the 4w questions (who, what, where, when) to write a research question (ability to define a research question that is significantly narrower than the original topic). 2. given a research topic, pick out the key concepts and compile a list of search terms or keywords (ability to compile five relevant keywords for the research topic that would yield useful results if typed into the library catalog or an article index). 3. given background information about google and the library, list two differences between the two related to content, organization, quality, or access (specific teaching points are authority; “invisible web;” free versus fee-based; quality control; personal assistance; pagerank technology; and scholarly versus popular). 4. given a research topic and access to the library's catalog, find 2 relevant books on the topic and record all relevant citation information (citation elements are title, author, location, call number, subject heading, availability, and floor). 5. given a research topic and access to a general article index database, find 2 relevant articles on the topic and record all relevant citation information (citation elements are author, title, publication name, volume/issue, date, and pages). after scoring the worksheets from 2009, librarians made the following changes to the curriculum to address lower-scoring areas: • in an effort to help students learn how to define a research question that is narrow, staff inserted five additional topic examples and resulting research questions into the libguide (module 1). • to address difficulties in selecting key concepts and generating keywords, we added five additional examples to the libguide. we also added a video showing a student brainstorming for keywords, and two interactive concept mapping tools. finally, we changed the worksheet to allow the topic and keywords to be updated at the end of the worksheet as they evolved (module 2). • due to large numbers of blank answers in the section on articulating key differences between information found on the internet versus the library, our librarians reduced the number of teaching points to give students more time. to address answers showing misunderstandings, a visual image on the libguide shows a person “fishing” for information in an ocean with information from both the visible and the invisible web (module 3). • in response to a large number of blank answers for book floor location and book availability, our staff modified the worksheet to include designated evidence based library and information practice 2011, 6.4 92 boxes for those answers. because we could still assess what we needed by decreasing the quantity, we reduced the number of required books from two to one, and reduced the required citation elements to conform more strictly to mla style citation components (we removed subject headings and availability status, but added location-city, publisher, year, and medium) (module 4). • since we could still assess what we needed by decreasing the quantity of articles discovered, we reduced the required number of articles from two to one. this gave students more time to focus on relevancy. we modified the required citation elements to conform to mla style (we added elements for database, medium, and date of access) (module 5). the literature provides few examples of the use of assessment to improve the curriculum at a basic undergraduate level, and provides conflicting evidence on face-to-face versus online learning. the purpose of this study was to analyze the effect of the revisions made to freshman english library instructional materials on student learning. specifically, to compare student scores between 2009 and 2010 and to compare blended versus computer-aided learning. the final objective was to identify ongoing or new problem areas to feed into the next cycle of revisions. methods assessing the effectiveness of revisions to the curriculum between 2009 and 2010 (blended learning instruction models) librarians made revisions, described above, to the 2009 curriculum to develop the 2010 curriculum. more specifically, librarians targeted the lower-scoring modules from 2009 and made the changes outlined in the “context” section. to determine the effects of a new curriculum on student learning, staff used an experimental design to compare student scores from 2009 and 2010, and used a random number table to sample 100 worksheets from the total number collected from both the fall 2009 (755) and the fall 2010 (587) cohorts. all eight members of the reference & instruction department, consisting of seven librarians and one library assistant, graded the worksheets using the scoring rubric described below. each grader scored their assigned worksheets and recorded the scores in a google form using an analytic scoring rubric (see appendix a). staff then calculated averages for all modules and subsections for the fall 2009 and fall 2010 cohorts. a t-test was used to determine significant differences in means for each subsection in 2009 and 2010, along with a chisquare test of independence to see if the number of scores in each scoring group in 2009 and 2010 was different or equal. comparing blended learning to online learning a second experiment examined the effect of delivery method on student learning by comparing scores from a group of students in 2010 receiving only online instruction through the libguide against the 2010 group receiving blended instruction. the students receiving online-only instruction completed all five modules online; the students receiving blended instruction completed the first two modules online and the final three modules in class during face-to-face instruction. in total, 46 worksheets (the entire sample available) from the online-only group were singlegraded by library staff, who independently recorded the scores in a google form using the same analytic scoring rubric mentioned above. staff calculated averages for all modules and compared the averages against the 100 sample worksheet scores from 2010’s blended instruction group in part 1. the t-test was used to determine significant differences in means for each subsection. scoring rubric librarians employed an analytic scoring rubric as the primary means of assessment to measure student learning in both experimental studies. an analytic rubric divides performance into “separate facets and each evidence based library and information practice 2011, 6.4 93 facet is evaluated using a separate scale” (moskal, 2003),.but each facet can also be summed to form a total score. our rubric (see appendix a) separates each of the five learning modules into subsections with stated student learning outcomes and corresponding acrl information literacy standards, then lists overall evaluation criteria for each subsection as well as specific evaluation criteria for scoring each task. the rubric ranks each student on a 1-3 point scale of beginning, developing, or proficient for all areas. the rubric was developed in-house; for more on that development process see gardner and acosta (2010). librarians established interrater reliability of the rubric prior to the study through a twenty worksheet sample in which the percent-agreement across multiple judges was at least 92% for each worksheet. results assessing the effectiveness of changes to the curriculum between 2009 and 2010 (blended learning instruction models) to assess the effectiveness of the curriculum changes made between 2009 and 2010, librarians compared student scores for both years to see if there were significant improvement in scores for each module in 2010. overall, the results were mixed. when averaging scores across all five modules, student worksheet totals were relatively the same in 2010 (2.49) compared to 2009 (2.48). when considering the mean aggregates for each module separately, students did worse on module 1 (narrowing a topic and defining a research question) and module 2 (picking out key concepts and listing keywords) in 2010 than 2009, but they did better on module 3 (listing two differences between information found on google versus through the library), module 4 (finding books), and module 5 (finding articles). when each sub-module was tested for statistical significance, students did significantly better in 2010 on module 3 (listing two differences between information found on google versus through the library) and parts of module 4 (completion and accuracy of book citation elements). however, they did significantly worse on module 1 (completion of “who, what, where, when” questions about a topic and developing a research question from the topic). module 1: narrowing a topic module 1a measures completion of answering four “who, what, where, when” questions about the topic on the worksheet. students receive a proficient (3) score for answering all four questions; a developing (2) score for answering two or three of the questions; and a beginning (1) score for answering one or none of the questions. the mean score for the 2009 students (m=2.75, sd=.54) was not significantly larger than the scores for the 2010 students (m=2.68, sd=.70) using the twosample t-test for equal variances, t(198)=.80, p=(.43). the chi-square test for independence indicates a difference between the distribution of scores for 2009 and 2010, χ2(2)=7.42, p=.02 (table 1). the higher number of “beginning” scores in 2010 reflects a higher number of students who left that section blank. more than half of the “beginning” scores from 2010 did not do their assigned homework and left all sections of modules 1 and 2 blank on the worksheet. reasons for this difference are table 1 module 1a: distribution of student scores evaluation criteria beginning (1) developing (2) proficient (3) narrows topic (completion) 2009 2010 2009 2010 2009 2010 5 13 15 6 80 81 evidence based library and information practice 2011, 6.4 94 unknown, but might include less conscientious students in 2010, or a higher rate of student absences in 2010. since the scoring rubric did not have a separate category for blank answers, staff re-examined the worksheets to discover this problem. students receiving a “developing “score in both groups did not answer all four questions about their topic, and it appears to be because they did not think all the questions applied to their topic. module 1b measures the ability to construct a research question. students receive a proficient (3) score for a question that is narrower than the original topic and specific enough for a research question; a developing (2) score for a question that is still too broad for a research question; and a beginning (1) score for a question that is not much narrower than the original topic. in the final overall worksheet score this section counts double since it is considered a higher-level skill. the mean score for the 2009 students (m=2.55, sd=.63) was significantly larger than the scores for the 2010 students (m=2.34, sd=.68) using the two-sample t-test for equal variances, t(198)= 2.26, p=(.02). the chi-square test for independence indicates no significant difference between the distribution of scores for 2009 and 2010, χ2(2)=5.34, p=.07). the main reason for the difference in “beginning” scores between 2009 and 2010 is because seven students from 2010 left it blank, failing to do this part of their homework. students also lost points for failing to narrow the topic at all (such as “who is tom cruise?”) or choosing an unrealistic research question (such as “how many writings does robert frost have?”). more students from 2009 received a “proficient” score, while more students from 2010 received a “developing” score for questions that were still too broad for a research question. a typical example of a question receiving a “developing” score is “what characteristics determine whether or not students will succeed in college?” this question is narrower than a question about general college success, but still not specific about what characteristics to determine success they want to research. to receive a “proficient” score, the question could be modified to “how good of a predictor is standardized testing for college success?” because the scores in this section significantly decreased in 2010, we continue to seek ways to improve the teaching material. we will take the six research question examples on the libguide and illustrate what would constitute a “beginning” question, a “developing” question, and a “proficient” question to further illustrate the refining process. also, we can encourage faculty to assign a general research topic to the class. module 2: key concepts module 2a measures the ability to list three key concepts within a research question. students receive a proficient (3) score for listing all three of the most important concepts from their research question; a developing (2) score for listing two of the three; and a beginning (1) score for listing one or none. the mean score for the 2009 students (m=2.47, sd=.67) was not significantly larger than the scores for the 2010 students (m=2.41, sd=.78) using the two-sample t-test for equal variances, t(198)=.58, p=(.56). the chi-square test for independence indicates no significant difference between the distribution of scores for 2009 and 2010, χ2(2)=4.11, p=.13 (table 3). in the 2010 group, twelve students left this table 2 module 1b: distribution of student scores evaluation criteria beginning (1) developing (2) proficient (3) defines research question 2009 2010 2009 2010 2009 2010 7 12 31 42 62 46 evidence based library and information practice 2011, 6.4 95 section blank compared to only four in the 2009 group. only six students in each group received a “beginning” score because they tried and failed to list more than one of the key concepts from their research question. common mistakes for students in each group receiving a “developing” score were adding an extra term that was related to but not part of the original research question; or forgetting to include the “who” part of the research question or the “what” part in favor of a less important adjective (for a topic about “modern undergraduates,” including both ‘modern’ and ‘undergraduates’ as key terms in lieu of the main verb). module 2b measures the ability to list keywords related to a research topic that would lead to search results in a library catalogue or article index. students receive a proficient (3) score for listing a total of five or more relevant keywords related to their research question; a developing (2) score for listing three or four keywords; and a beginning (1) score for listing two or less keywords. the mean score for the 2009 students (m=2.47, sd=.78) was not significantly larger than the scores for the 2010 students (m=2.32, sd=.84) using the twosample t-test for equal variances, t(198)=1.31, p=(.19). the chi-square test for independence indicates no significant difference between the distribution of scores for 2009 and 2010, χ2(2)=1.77, p=.41 (table 4). students who received a “beginning” or a “developing” score often had trouble with the concept of a synonym, either conceptually or technically (picking a word not recognized as a linguistic unit, such as “automotist” as a synonym for driver). it was also common for students to select descriptive phrases rather than a singular synonym or keyword (e.g, using “family falls apart” instead of “divorce”). as the additional examples and tools added to the libguide in 2010 for this section did not increase scores, and the number of hits for this portion of the libguide was only 632, we will add a mandatory interactive keywords exercise to help students understand the concept of a keyword and gain more practice. module 3: information found on the internet versus the library module 3 measures the ability to list differences between information found on the internet and the library related to our teaching points. students receive a proficient (3) score for listing a total of two or more differences; a developing (2) score for listing one difference; and a beginning (1) score for listing no differences. the mean score for the 2009 students (m=2.06, sd=.92) was not significantly smaller than the scores for the 2010 students (m=2.19, sd=.83) using the twosample t-test for equal variances, t(198)=–1.05, p= .29). the chi-square test for independence indicates a significant difference between the distribution of scores for 2009 and 2010, χ2(2)=6.36, p=.04 (figure 5). there were less “beginning” scores in 2010, and a primary reason was because fewer students in 2010 left this section blank than in 2009 (18 compared to 35) we shortened the module in 2010 to contain fewer and more standardized teaching points, which may have allowed more time for table 3 module 2a: distribution of student scores evaluation criteria beginning (1) developing (2) proficient (3) lists key concepts 2009 2010 2009 2010 2009 2010 10 18 33 23 57 59 table 4 module 2b: distribution of student scores evaluation criteria beginning (1) developing (2) proficient (3) compiles keywords 2009 2010 2009 2010 2009 2010 18 24 17 20 65 56 evidence based library and information practice 2011, 6.4 96 completion. we also added a visual image to the libguide of a person “fishing” for information in an ocean containing information from both the visible and the invisible web in an attempt to help illustrate differences in information types. since more students received a “developing” score in 2010 than in 2009, and since more students were able to list one difference related to our teaching points rather than no differences, the image might have been partially effective. the same number of students received a “proficient” score both years, however, so we cannot be sure of the extent of our changes. we will independently test module 3 on a group of students both with and without the visual image. also, we will do a peer review observation of module 3 to check for standardization. students lost points for factually incorrect answers (such as “the internet gives you web sites with false information”) or answers that were not one of our teaching points (such as “libraries can give you what you need”). module 3 was consistently the lowest scoring module among students from both years. module 4: finding books module 4 measures the ability to find one book in the library catalogue (4a: quantity), with separate scores for completeness (4b) and the accuracy (4c) of the following citation elements: location, subject heading, availability, call number, floor, author, title, location/city, publisher, year, and medium. the relevancy of the book to the research topic (4d) illustrates a more sophisticated level of development, therefore this score counts double in the final overall calculated worksheet score. the mean scores for 2009 and 2010 were not significantly different for the section relating to quantity (4a) and the section relating to relevancy (4d); almost all students in both groups could find one relevant book using the library catalogue. for the section relating to completeness (4b), the mean score for the 2009 students (m=2.43, sd=.57) was significantly smaller than the scores for the 2010 students (m=2.71, sd=.54) using the two-sample t-test for equal variances, t(198)=–3.56, p=(.00). the chi-square table 5 module 3: distribution of student scores evaluation criteria beginning (1) developing (2) proficient (3) lists differences or characteristics: lib & internet 2009 2010 2009 2010 2009 2010 39 26 16 29 45 45 table 6 module 4: distribution of student scores module evaluation criteria beginning (1) developing (2) proficient (3) 4a (quantity) lists 1 book 2009 2010 2009 2010 2009 2010 1 2 99 98 4b (completion) citation elements 4 4 49 21 47 75 4c (accuracy) citation elements 5 4 51 27 44 69 4d (relevancy) links book to topic 11 10 89 90 evidence based library and information practice 2011, 6.4 97 test for independence indicates a difference between the distribution of scores for 2009 and 2010, χ2(2)=17.63, p=.00. the main reason for the increase in scores between 2009 and 2010 is likely due to a lack of designated answer boxes on the 2009 worksheet for the book floor and availability questions. in 2009, students left blank the element for “floor” 38 times, followed by “availability” 20 times, resulting in lower scores. after we modified the worksheet to have separate boxes for each individual citation element, the 2010 students only left the “floor” element blank eight times. also, the changes made to citation elements collected makes this section now match its learning outcome of recording “all pertinent” citation information needed for modern language association (mla) citation style. for the section on accuracy (4c), the mean score for the 2009 students (m=2.39, sd=.58) was significantly smaller than the scores for the 2010 students (m=2.65, sd=.56) using the two-sample t-test for equal variances, t(198)= 3.22, p= (.00). the chi-square test for independence indicates a difference between the distribution of scores for 2009 and 2010, χ2(2)=13.03, p=.00. in 2009, a lot of answers counted as incorrect were blank for “floor” and “availability.” when looking to see how many of the answers were incorrect rather than merely blank, the results are “subject” (5 times) and “floor” (5 times). in 2010, the only element that was attempted (not blank) but incorrect was “floor” (15 times). this shows that “floor” is still problematic, and students either fail to locate the floor directory or do not know how to interpret it. we have requested that the floor location of a book appear as a popup over the call number in the next upgrade of our catalogue. module 5: finding articles module 5 measures the ability to find one article in the article index (5a: quantity), with separate scores for completeness (5b) and the accuracy (5c) of the following citation elements: author, article title, source, volume, issue, date, pages, database, medium, and date of access. there is also a separate score for relevancy to the research topic (5d), which counts double on the overall worksheet score since it illustrates a higher lever of development. the mean scores for 2009 and 2010 were not significantly different for any of the sections in module 5; almost all students in both years could find one article using an article index and complete the citation elements correctly. despite the fact that the curriculum was changed in 2010 to require only one article to give students more time to focus on relevancy (section 5d), students did not do significantly better on that section. the fact that students in both years scored low on finding keywords (module 2b) also probably influenced the relevancy score for articles, since they were instructed to type these same keywords into the article index. table 7 module 5: distribution of student scores module evaluation criteria beginning (1) developing (2) proficient (3) 5a (quantity) lists 1 article 2009 2010 2009 2010 2009 2010 4 7 96 93 5b (completion) citation elements 7 7 18 18 75 75 5c (accuracy) citation elements 8 8 20 18 72 74 5d (relevancy) links article to topic 12 12 21 15 67 73 evidence based library and information practice 2011, 6.4 98 overall, the average student scores were better in 2010 than 2009 on module 3 (listing differences between sources found on google versus the library), module 4 (finding books), and module 5 (finding articles). when each sub-module was tested for statistical significance, though, the significant results were module 3 (listing two differences between information found on google versus through the library) using the chi-square test for independence; and parts of module 4 (completion and accuracy of book citation elements) using the two-sample t-test for equal variances. unfortunately, the average student scores were worse in 2010 than 2009 on module 1 (narrowing a topic and defining a research question) and module 2 (picking out key concepts and listing keywords). when sub-modules were tested for statistical significance, module 1a (narrowing a topic by completing “who, what, where, and when” questions) was significant using the chi-square test for independence, and module 1b (developing a research question for the topic) was significant using the two-sample t-test for equal variances. comparing blended learning to online learning the second part of the study involved assessing the effectiveness of delivery method on students receiving the 2010 curriculum. staff compared scores from students receiving a blended combination of face-to-face and online instruction through the libguide against the scores of students receiving only online instruction. the mean worksheet scores across all five modules for the blended instruction group of students (m=2.49, sd=.39) was significantly smaller than the scores for the students receiving only online instruction (m=2.63, sd=.25) using the two-sample t-test for equal variances, t(198)= 2.11, p=(.04). looking at each module individually, students receiving online-only instruction did slightly better on module 1 (defining a research question), module 2 (picking out the key concepts of the research question and compiling keywords), module 3 (listing two differences between information found on google versus through the library), and module 4 (finding books). they scored exactly the same on module 5 (finding articles). however, when each subsection was tested for statistical significance only two of them were statistically significant: module 2b (listing keywords) and module 4b (completion of book citation elements). module 2b: listing keywords the mean score for the 2010 online-only group on module 2b, listing keywords (m=2.67, sd= .52) was significantly larger than the scores for the 2010 blended group (m=2.32, sd=.84) using the t-test for equal variances, t(144) =– 2.64, p=(.01). the 2010 blended group was negatively affected by seventeen blank answers. the group doing the entire worksheet as a homework assignment using only the libguide had almost no blank answers. it is possible that the online-only group paid more attention to the supplemental materials listed on the libguide, such as the interactive concept map tools and the brainstorming video, leading to this difference. these students may have read more carefully in the absence of any personal assistance. the blended group had the benefit of seeing the keywords module demonstrated by a librarian during the face-to-face session, which could have led to higher scores but did not. a higher number of students in the online-only group indicated at the end of the worksheet that they went back and added keywords to module 2b, which probably increased their scores. it is also possible that students who completed the worksheet during a face-to-face instruction session had limited time and may not have seen the prompt at the end of the worksheet to add more keywords. module 4b: completing citations the mean score for the online-only group on module 4b, completing book citation elements, (m=2.91, sd=.28) was significantly larger than the scores for the 2010 students receiving blended instruction (m=2.71, sd=.54) using the t-test for equal variances, t(144)=– 2.41, p=(.02). key differences centered on the “floor” element and “medium” element. evidence based library and information practice 2011, 6.4 99 “floor” was left blank eight times for the blended group but only two times for the online-only group. the floor element was not on the book record in the library catalogue, but could only be learned by consulting a separate directory. the “medium” element was left blank twelve times by the blended group but only one time by the online-only group. again, time constraints may have played a factor for the blended instruction group, since they had to complete module 4b during a 50-minute class period and the online-only group had time for repeated readings or more reflection. discussion this two-part study has examined the effects of curriculum changes on student learning by comparing differences in student scores in 2009 and 2010. furthermore, it has compared the scores of students who received instruction by blended learning against those who received an online tutorial. the average worksheet scores across all five modules were almost the same when comparing the effects of changes to the curriculum in two different years (2.49 in 2009 versus 2.48 in 2010), but there were significant differences across individual subsections. students in 2010 showed significant improvement in module 3, listing two differences between information found on the internet versus through the library. fewer students left this section blank in 2010, probably because the module contained fewer and more standardized teaching points, and students had more time to complete that section of the worksheet in class. the impact of adding a visual image of a person “fishing” for information to illustrate the visible web versus invisible web is unknown; we plan to test the image against a control group. since this was the lowest scoring module for both years, we will also undergo a peer review check for consistency of the teaching points amongst all librarians. the other result implying positive and effective changes made to the curriculum in 2010 was module 4 (completion and accuracy of book citation elements). the improvement in student scores was largely due to less blank answers, and therefore having separate, designated fields appears to encourage responses to individual components of a question on a worksheet. it was disappointing that students scored significantly worse in 2010 on module 1, narrowing a broad research topic and constructing a research question. more students in 2010 left this section blank, effectively failing to do their homework. since this segment was completed independently as a homework assignment, we can only speculate on the cause but will work with instructors to ensure greater accountability. because of the lower scores due to blank answers, we will differentiate between a “blank” answer and an “incorrect” answer in future rubrics to separate “time” versus “learning” problems. for module 1b (developing a research question), the six topic examples on the libguide will extend to illustrate for each a “beginning” research question, a “developing” question, and a “proficient” question. despite these mixed results, the feedback loop continues to give us valuable information to improve the teaching materials for the next cycle. for a complete list of changes we will make next year, see appendix b. the results for the part of the study that compared blended learning against online learning showed that online learning students scored significantly better on average across all five modules than the group receiving blended instruction (2.63 versus 2.49). when subsections were broken down and tested for statistical significance, listing keywords (2b) and completion of book citation elements (4b) tested positive using the t-test for equal variances. for both sections, more students in the blended instruction model had blank answers. this suggests that the time constraints of a 50-minute class period played a negative role, but it also highlights the fact that research process is cyclical and it is important to simulate this by allowing students the flexibility to go back and change their topic or to add keywords as the research evidence based library and information practice 2011, 6.4 100 evolves. the self-paced nature and flexibility of online instruction better accommodates this. this part of the study adds to the research on the effects of instructional delivery on student learning. because the online-only group scored significantly higher than the blended instruction group overall, and two individual skill elements also scored significantly higher when isolated, it supports the findings of anderson & may (2010), who found that students receiving online instruction scored better when tested on their ability to find sources than students receiving blended instruction. it contrasts with the findings of kraemer, lombardo, & lepkowski (2007), who found that students receiving online-only instruction scored lower than those receiving blended instruction. it indirectly supports lawson (1989), who found an online tutorial more effective than face-to-face instruction. more research needs to be conducted comparing blended instruction to other models. as highlighted in the literature review, the majority of studies that compare online instruction to face-to-face instruction find no significant differences in the two delivery methods. in this study, the blended instruction group did not appear to benefit significantly from the face-to-face demonstrations of each module during class time. the ability to work at their own pace and complete the worksheet when they wanted, rather than during a compressed class period, appears to have a greater positive impact than live modeling from a librarian, leading to higher scores for the online-only group. the major limitation of this study is its methodology. since the design was a field experiment, many different classes were included in the random sample. further, the attitude of the instructor in each class may have impacted how seriously students took the worksheet assignment. similarly, seven different librarians taught the face-to-face segments, and while standardization was attempted, there may have been some inconsistencies in delivery. no control group was used to test how well a student could complete the worksheet skills on their own without any aid, either from the libguide or blended instruction. another limitation was that there was no pre-test, and so it is unknown whether the students in each year started from a different baseline of knowledge. also, no student demographic data was collected to check whether the students in each group were similar. finally, the in-house rubric used to score student worksheets only has three possible scores, so parts of the high inter-rater reliability achieved at calibration might have been reached by chance and not because the judges truly scored the same way. nevertheless, the study highlights issues involved in conducting experiments in practice as well as the limitations of using experimental methods for changing practice. conclusion the purpose of this study was to assess the effectiveness of curriculum changes made from the 2009 lmu freshman english library instruction curriculum to the 2010 curriculum, and also to compare the effectiveness of library instruction delivered online-only versus a “blended” combination of face-to-face and online instruction. results were mixed; students scored significantly higher in 2010 on comparing internet and library resources and accurately completing book citation components, but scored lower on narrowing a topic and defining a research question than students taking the course in 2009. despite these mixed results, the evaluation process helped us identify continued problems in the teaching materials for the next revision cycle. results of the instructional delivery method comparison revealed students receiving only computer-assisted instruction did better overall, and significantly better on the individual skill elements of generating keywords and completing book citation and location elements than students receiving blended instruction. this suggests that computer-assisted instruction through a libguide holds promise as an alternative to face-to-face or blended instruction. the teaching materials from this study can be adapted for a variety of instructional settings using any combination of delivery method. evidence based library and information practice 2011, 6.4 101 acknowledgements a version of this paper was presented at the 6th international evidence based library and information practice (eblip) conference, held in salford, uk (june 2011). the author would like to thank the lmu instruction coordinator, elisa slater acosta; all members of the lmu reference & instruction department; laura massa in the lmu office of assessment; and the william h. hannon library for a 2010/11 research incentive travel grant. references anderson, k. & may, f.a. (2010). does the method of instruction matter? an experimental examination of information literacy instruction in the online, blended, and face-to-face classrooms. journal of academic librarianship, 36(6), 495-500. association of college and research libraries. (2000). information literacy competency standards for higher education. retrieved from http://www.ala.org/ala/mgrps/divs/ac rl/standards/informationliteracycomp etency.cfm burkhardt, j.m. (2007). assessing library skills: a first step to information literacy. portal: libraries and the academy, 7(1), 25-49. churkovich, m. & oughtred, c. (2002). can an online tutorial pass the test for library instruction? an evaluation and comparison of library skills instruction methods for first year students at deakin university. australian academic and research libraries, 33(1), 25–38. dewald, n.h. (1999). transporting good library instruction practices into the web environment: an analysis of online tutorials. journal of academic librarianship, 25(1), 26-31. fain, m. (2011). assessing information literacy skills development in first year students: a multi-year study. journal of academic librarianship, 37(2), 109119. gardner, s. & acosta, e.s. (2010). using a rubric to assess freshman english library instruction. in s. hiller, k. justh, m. kyrillidou, & j. self (eds.), proceedings of the 2010 library assessment conference: building effective, sustainable, practical assessment (pp. 159-173). washington, dc: association of research libraries. germain, c.a., jacobson, t.e., & kaczor, s.a. (2000). a comparison of the effectiveness of presentation formats for instruction: teaching first-year students. college & research libraries, 61(1), 65-72. holman, l. (2000). a comparison of computerassisted instruction and classroom bibliographic instruction. reference & user services quarterly, 40(1), 53-60. houlson, v. (2007) getting results from oneshot instruction. college & undergraduate libraries, 14(1), 89-108. kaplowitz, j. & contini, j. (1998). computerassisted instruction: is it an option for bibliographic instruction in large undergraduate survey classes? college & research libraries, 59(1), 19-27. knight, l.a. (2006). using rubrics to assess information literacy. reference services review, 34(1), 43-55. koufogiannakis, d. & wiebe, n. (2006). effective methods for teaching information literacy skills to undergraduate students: a systematic review and meta-analysis. evidence based library and information practice, evidence based library and information practice 2011, 6.4 102 1(3). retrieved from http://ejournals.library.ualberta.ca/ind ex.php/eblip/article/view/76/153. kraemer, e.w., lombardo, s.v., & lepkowski, f.j. (2007). the librarian, the machine, or a little of both: a comparative study of three information literacy pedagogies at oakland university. college & research libraries, 68(4), 330342 lawson, m.d. (1999). assessment of a college freshman course in information resources. library review, 48(2), 73-78. lawson, v.l. (1989). using a computerassisted instruction program to replace the traditional library tour: an experimental study. rq, 29(1), 71-79. madland, d., & smith, m.a. (1988). computerassisted instruction for teaching conceptual library skills to remedial students. research strategies, 6(2), 5264. moskal, b.m. (2003). recommendations for developing classroom performance assessments and scoring rubrics. practical assessment, research & evaluation, 8(14). retrieved from http://pareonline.net/getvn.asp?v=8 &n=14 oakleaf, m. (2006). assessing information literacy skills: a rubric approach. (doctoral dissertation). retrieved from proquest dissertations and theses database. (umi no. 3207346) oakleaf, m. (2009). the information literacy instruction assessment cycle: a guide for increasing student learning and improving librarian instructional skills. journal of documentation, 65(4), 539-560. samson, s. (2000). what and when do they know? web based assessment. reference services review, 28(4), 335342. scharf, d., elliot, n., huey, h., briller, v., & joshi, k. (1997). direct assessment of information literacy using writing portfolios. journal of academic librarianship, 33(4), 462-477. smith, k.r. (2001). new roles and responsibilities for the university library: advancing student research through outcomes assessment. journal of library administration, 35(4), 29-36. vander meer, p.f., & rike, g.e. (1996) multimedia: meeting the demand for user education with a selfinstructional tutorial. research strategies, 14, 145-158. walsh, a. (2009). information literacy assessment: where do we start? journal of librarianship and information science, 41(1), 19-28. warner, d.a. (2003). programmatic assessment: turning process into practice by teaching for learning. journal of academic librarianship, 29(3): 169-176. zhang, l., watson, e.m., & banfield, l. (2007). the efficacy of computer-assisted instruction versus face-to-face instruction in academic libraries: a systematic review. journal of academic librarianship, 33(4), 478-484. http://www.google.com/url?q=http%3a%2f%2fpareonline.net%2fgetvn.asp%3fv%3d8%26n%3d14&sa=d&sntz=1&usg=afqjcnf881gah9urtqima3ftigb8c8jbna� http://www.google.com/url?q=http%3a%2f%2fpareonline.net%2fgetvn.asp%3fv%3d8%26n%3d14&sa=d&sntz=1&usg=afqjcnf881gah9urtqima3ftigb8c8jbna� http://www.google.com/url?q=http%3a%2f%2fpareonline.net%2fgetvn.asp%3fv%3d8%26n%3d14&sa=d&sntz=1&usg=afqjcnf881gah9urtqima3ftigb8c8jbna� http://www.google.com/url?q=http%3a%2f%2fpareonline.net%2fgetvn.asp%3fv%3d8%26n%3d14&sa=d&sntz=1&usg=afqjcnf881gah9urtqima3ftigb8c8jbna� http://www.google.com/url?q=http%3a%2f%2fpareonline.net%2fgetvn.asp%3fv%3d8%26n%3d14&sa=d&sntz=1&usg=afqjcnf881gah9urtqima3ftigb8c8jbna� http://www.google.com/url?q=http%3a%2f%2fpareonline.net%2fgetvn.asp%3fv%3d8%26n%3d14&sa=d&sntz=1&usg=afqjcnf881gah9urtqima3ftigb8c8jbna� http://www.google.com/url?q=http%3a%2f%2fpareonline.net%2fgetvn.asp%3fv%3d8%26n%3d14&sa=d&sntz=1&usg=afqjcnf881gah9urtqima3ftigb8c8jbna� http://www.google.com/url?q=http%3a%2f%2fpareonline.net%2fgetvn.asp%3fv%3d8%26n%3d14&sa=d&sntz=1&usg=afqjcnf881gah9urtqima3ftigb8c8jbna� http://www.google.com/url?q=http%3a%2f%2fpareonline.net%2fgetvn.asp%3fv%3d8%26n%3d14&sa=d&sntz=1&usg=afqjcnf881gah9urtqima3ftigb8c8jbna� http://www.google.com/url?q=http%3a%2f%2fpareonline.net%2fgetvn.asp%3fv%3d8%26n%3d14&sa=d&sntz=1&usg=afqjcnf881gah9urtqima3ftigb8c8jbna� http://www.google.com/url?q=http%3a%2f%2fpareonline.net%2fgetvn.asp%3fv%3d8%26n%3d14&sa=d&sntz=1&usg=afqjcnf881gah9urtqima3ftigb8c8jbna� http://www.google.com/url?q=http%3a%2f%2fpareonline.net%2fgetvn.asp%3fv%3d8%26n%3d14&sa=d&sntz=1&usg=afqjcnf881gah9urtqima3ftigb8c8jbna� http://www.google.com/url?q=http%3a%2f%2fpareonline.net%2fgetvn.asp%3fv%3d8%26n%3d14&sa=d&sntz=1&usg=afqjcnf881gah9urtqima3ftigb8c8jbna� evidence based library and information practice 2011, 6.4 103 appendix a ***title here beneath the label*** student learning outcomes evaluation criteria beginning = 1 developing = 2 proficient = 3 module 1 1.a. defines or modifies information need to achieve manageable focus through the dissection of a broad topic [acrl standard 1, indicator 1.d] narrows topic (completion) answers zero or 1 of the "when, where, who, what" questions about the topic in the box answers 2 or 3 of the "when, where, who, what" questions about the topic in the box answers all 4 of the "when, where, who, what" questions about the topic in the box (x1) 1.b. defines or modifies information need to achieve manageable focus through the construction of a specific research question defines research question constructs no question or a question that is not much narrower than the original topic constructs a question that is narrower than the original topic, but still too broad for a research question constructs a question that is narrower than the original topic and specific enough for a research question (x2) module 2 student learning outcomes evaluation criteria beginning = 1 developing = 2 proficient = 3 2.a. identifies key concepts and terms that describe the information needed [acrl standard 1, indicator 1.e] lists key concepts lists 1 or less of the most important concepts from the research question lists 2 out of the 3 most important concepts from the research question lists all 3 of the most important concepts from the research question (x1) 2.b. identifies keywords, synonyms and related terms for the information needed [acrl standard 2, indicator 2.b] compiles keywords lists a total of 2 or less relevant keywords for the research topic. if in doubt, type the keywords into the library catalog or article index to test for relevancy lists a total of 3-4 relevant keywords for the research topic. if in doubt, type the keywords into the library catalog or article index to test for relevancy lists a total of 5 or more relevant keywords for the research topic. if in doubt, type the keywords into the library catalog or article index to test for relevancy (x1) module 3 student learning outcomes evaluation criteria beginning = 1 developing = 2 proficient = 3 investigates the scope, content, or organization of two information retrieval systems [acrl standard 2, indicator 1.c] lists differences or characteristics lists no differences between or characteristics of google, the internet, or the library related to authority; invisible web; free versus feebased; quality control; personal assistance; pagerank technology; popularity; or scholarly lists 1 difference between or characteristic of google, the internet, or the library related to authority; invisible web; free versus feebased; quality control; personal assistance; pagerank technology; popularity; or scholarly lists 2 or more differences between or characteristics of google, the internet, or the library related to authority; invisible web; free versus feebased; quality control; personal assistance; pagerank technology; popularity; or scholarly (x1) module 4 student learning outcomes evaluation criteria beginning = 1 developing = 2 proficient = 3 4.a. recognizes relevant information sources using the library catalog and records all pertinent citation information for future reference [acrl standard 2, indicators 5.c and 5.d] a. (quantity) locates 1 book finds no books in the library catalog and doesn't write down any citation information no "2" value for this x finds 1 book in the library catalog and writes down the citation information (x1) evidence based library and information practice 2011, 6.4 104 4.b. b. (completion) lists location and citation components location (1 point) call number (1point) floor (1 point) author (1 point) title (1 point) location/city (1 point) publisher (1 point) year (1 point) medium (1 point) location (1 point) call number (1 point) floor (1 point) author (1 point) title (1 point) location/city (1 point) publisher (1 point) year (1 point) medium (1 point) location (1 point) call number (1 point) floor (1 point) author (1 point) title (1 point) location/city (1 point) publisher (1 point) year (1 point) medium (1 point) (x1) total is 4 or less t total is 5-8 total is 9 4.c. c. (accuracy) lists location and citation elements in correct fields location (1 point) call number (1 point) floor (2 points) author (1 point) title (1 point) location/city (1 point) publisher (1 point) year (1 point) medium (1 point) location (1 point) call number (1 point) floor (2 points) author (1 point) title (1 point) location/city (1 point) publisher (1 point) year (1 point) medium (1 point) location (1 point) call number (1 point) floor (2 points) author (1 point) title (1 point) location/city (1 point) publisher (1 point) year (1 point) medium (1 point) (x1) total is 3 or less total is 4-8 total is 9-10 4.d. d. (relevancy) links book to research topic finds a book that is not relevant to the research topic. if in doubt, look at the subject field, title field, call number field, or table of contents to make the determination no "2" value for this x finds a book that is relevant to the research topic. if in doubt, look at the subject field, title field, call number field, or table of contents to make the determination (x2) module 5 student learning outcomes evaluation criteria beginning = 1 developing = 2 proficient = 3 5.a. recognizes relevant information sources using an article index; understands the elements and correct syntax of an article citation; and records complete citation information from the index for future reference [acrl standard 2, indicators 5.c and 5.d] a. (quantity) locates 1 article finds no articles in the article index and doesn't write down any citation information no "2" value for this x finds 1 article in the article index and writes down the citation information (x1) 5.b. b. (completion) lists citation components author (1 point) article title (1 point) source/pub (1 point) *volume (1 point) *issue (1 point) date (1 point) pages (1 point) database (1 point) medium (1 point) date/access (1 point) *if newspaper, no vol/issue so give 1 point author (1 point) article title (1 point) source/pub (1 point) *volume (1 point) *issue (1 point) date (1 point) pages (1 point) database (1 point) medium (1 point) date/access (1 point) *if newspaper, no vol/issue so give 1 point author (1 point) article title (1 point) source/pub (1 point) *volume (1 point) *issue (1 point) date (1 point) pages (1 point) database (1 point) medium (1 point) date/access (1 point) *if newspaper, no vol/issue so give 1 point (x1) total is 5 or less total is 6-9 total is 10 evidence based library and information practice 2011, 6.4 105 5.c. c. (accuracy) lists citation elements in correct fields author (1 point) article title (1 point) source/pub (2 points) *volume (1 point) *issue (1 point) date (1 point) pages (2 point) database (1 point) medium (1 point) date/access (1 point) *if newspaper, no vol/issue so give 1 point author (1 point) article title (1 point) source/pub (2 points) *volume (1 point) *issue (1 point) date (1 point) pages (2 point) database (1 point) medium (1 point) date/access (1 point) *if newspaper, no vol/issue so give 1 point author (1 point) article title (1 point) source/pub (2 points) *volume (1 point) *issue (1 point) date (1 point) pages (2 point) database (1 point) medium (1 point) date/access (1 point) *if newspaper, no vol/issue so give 1 point (x1) total is 4 or less total is 5-10 total is 11-12 5.d. d. (relevancy) links article to research topic finds an article that is not relevant to the research topic. if in doubt, look at the abstract or full text of the article finds an article that is somewhat relevant to the research topic, but is general or broad. if in doubt, look at the abstract or full text of the article finds an article that is relevant to the research topic. if in doubt, look at the abstract or full text of the article (x2) evidence based library and information practice 2011, 6.4 106 appendix b future changes to teaching materials • for module 1b (developing a research question), the six topic examples on the libguide will extend to illustrate for each a “beginning” research question, a “developing” question, and a “proficient” question. a dartboard analogy getting closer to its target will be visually created. also, we will push for the instructor to assign a research topic. • for module 2b (keywords) we will add a mandatory keywords exercise for extra practice. • for module 3 (internet resource versus library resource), we will test the visual “fishing” image used and also do a peer review check for consistency of the teaching points. • the rubric will be modified across all modules to include a “0” score for blank answers, to differentiate a blank answer from an incorrect one. • the entire worksheet with all five modules will be assigned as homework using only the libguide prior to the face-to-face library session rather than only modules 1 and 2. / evidence based library and information practice/ / module 2: key concepts research article   small library research: using qualitative and user-oriented research to transform a traditional library into an information commons   quincy d. mccrary associate librarian the carl gellert and celia berta gellert library notre dame de namur university belmont, california, united states of america email: qmccrary@ndnu.edu   received: 1 dec. 2016     accepted: 6 feb. 2017          2017 mccrary. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective the project team investigated the changes necessary to transform the original library into an information commons. the researchers sought to drive the project by asking for patrons’ input, rather than rely on the vision of administrators or librarians.   methods the project team used four techniques to gather data. they recorded patron use patterns, administered surveys, conducted formal interviews, and facilitated comment boards.   results each of the four methods used in this research delivered similar conclusions. patrons used the library as a study hall, but the space did not facilitate collaboration. patrons requested more group study spaces, more access to power, and a quieter environment. patrons identified the value of developing a learning community in the library. finally, patrons advocated for the retention of physical collections in the library building.   conclusion the present library building, designed to facilitate individual, quiet, textual based learning, no longer serves the needs of its patrons. analysis of this project’s data supports the need to develop an information commons. the gellert library is not just a place to store books and study. rather, it is a place where meaning and learning emerges from access to knowledge.     introduction   the educational mission of notre dame de namur university (ndnu) embraces the idea of holistic learning communities. at ndnu, learning communities develop when incoming classes of students engage in pod learning environments. in this model, sections of the same class come together periodically across a semester for large group learning activities like community engagement, special topics, and speaker series. an information commons model for the library that embraces collaborative information seeking would enhance pod instruction and help build learning communities. the university provost, along with the library director, articulated a clear need for the library to transform into an information commons. the library director formed a team including a project leader (a formally trained anthropologist), the library director, and three student assistants to complete this project. the project team attempted to discover what modifications to the library’s space could transform it into a modern information commons.   literature review   librarian donald beagle best described the notion of an information commons as a facility designed to organize workspace and service delivery around the integrated digital environment (beagle, 1999). it includes the physical commons where open floors and browsable stacks allow quick access to information and collaboration, the virtual commons where users access the vast digital content of the library, and the cultural commons of research collaborations, workshops, and tutorial programs (beagle, 2011). originally developed in the 1980s, the information commons concept emerged in different forms in the 1990s (for example as an information hub, media union, or a learning commons). in 2010, steven johnson presented a ted talk titled “where good ideas come from: the natural history of innovation.” here johnson explored the role of the coffeehouse in the enlightenment, arguing it provided "a space where people would get together from different backgrounds, different fields of expertise, and share" (as cited in holland, 2015). in many ways, this mirrors what an information commons is attempting to create in libraries today. today, an information commons fosters an environment centred on the creation of knowledge and self-directed learning rather than an isolated user accessing information (rawal, 2014). the earlier reader-centred paradigms led to spaces that championed collections and a “well lit area for reading” (bennett, 2009, pp. 181-182). technological changes over the last few decades have resulted in a substantial move of information from print to digital. this allowed libraries to re-appropriate areas once dedicated to bookshelves for more user-oriented spaces (heitsch & holly, 2001). sarah hutton in the “final report of the learning commons assessment task force for the  university of massachusetts (amherst)” notes that “a space has evolved from a combined library and computer lab into a full-service learning, support, research, and project space” (hutton, 2015, p. 10).    the turn to qualitative studies of libraries is a relatively new practice. sandstrom and sandstrom (1995) were some of the first researchers to identify a need for qualitative research in libraries. ethnographic studies of university students in general are also limited, with the exception of michael moffatt’s (1989) study of students at rutgers university titled “coming of age in new jersey: college and american culture.” susan blum’s research published as “my word!: plagiarism and college culture” (2009) is an ethnographic examination of plagiarism in student assignments. cathy small’s “my freshman year: what a professor learned by becoming a student” gives an account of student life at northern arizona university based on her own experience enrolling as a “returning” student (published as nathan, 2005). in the mid-2000s several projects using ethnography to understand library users’ needs and behaviors resulted in very good projects such as bryant, 2007, 2009; foster and gibbons, 2005, 2007; jahn, 2008; ostrander, 2008; othman, 2004; and suarez, 2007. nancy fried foster and susan gibbons at the university of rochester (foster & gibbons, 2007) conducted one of the first large-scale ethnographic studies of how students utilize the library in 2004–2006. the tremendous success of this study in uncovering the details of student life drove many librarians to conduct similar studies. fresno state university conducted an excellent ethnographic study (delcore, mullooly, & scroggins, 2009). smale and regalado (2010) have begun publishing work conducted in the cuny libraries in the undergraduate scholarly habits ethnography project. head and eisenberg in “lessons learned: how college students seek information in the digital age” (2009) seek to understand student information-seeking behaviors at many colleges and universities across the united states. the influential “so you want to do anthropology in your library?: a practical guide to ethnographic research in academic libraries” (asher & miller, 2010) provided a benchmark and toolkit for further ethnographic research of libraries. lastly, khoo et al. (2012) do an excellent job summarizing the current state of qualitative research used in the study of libraries.   aims   the notre dame de namur university library is a single 40,000 square foot room. a second story balcony over three quarters of the floor houses the book collection. prior to the alterations brought about by this project, students using the library tended to work individually. group work was conducted at large tables in hushed whispers that often carried throughout the building. the noise from older keyboards in the computer lab area could dominate the building with frenzied typing. instruction sessions that promoted active learning disrupted the entire building. as a result, speaker sessions, presentations, open microphone nights, etc., were rarely scheduled. the library building, due to its structure and technology, did not promote a collaborative information seeking and learning environment. the research team for this project sought to discover, using a four technique method, how to create such an environment. the primary research question was, what changes could convert the library into an information commons?   methods   in 2014, the internal review board for ndnu approved this research and any publication of the work. utilizing a qualitative approach, the project team employed four techniques to build a holistic snapshot of user needs. these four techniques were   ·         recording patron use patterns logging of users’ place and activity in the building. ·         surveys measurements of what services are being used and ranking satisfaction with them. ·         formal interviews following an interview guide and used to illicit a broader response. ·         comment boards self-reported responses to questions and prompts.   the data collected included how patrons use the library, the ways they seek help, and their interactions with library spaces. participation in this project was voluntary. the project leader informed respondents about their right of consent. over 300 respondents participated in this project.   the research spanned the 2015-2016 academic year. patrons completed surveys advertised through the library’s website and through signage in the library. staff requested that patrons who completed the survey take part in a formal interview. the project leader conducted 24 formal interviews in a small office in the main library building. the interviews followed a guide (see appendix c). the project leader then transcribed the interviews. the project leader and two designated assistants recorded patrons’ location, study type (individual or group), and technology use every hour the building was open. at different periods during the 2015-2016 school year, large comment boards (located in three key points in the library) displayed alternating questions. patrons self-reported directly on the comment boards.   results   each of the four methods used in this research resulted in similar conclusions. patrons used the library as a study hall, but the space did not facilitate collaboration. patrons requested more group study spaces, more access to power, and a quieter environment.   recording patron use patterns   who uses the library, how do they use it, and why? over the fall semester in 2015, the project leader and two assistants observed and recorded patrons in the building at one-hour intervals. the project leader developed four categories for recording use patterns.   ·         students working individually ·         students working at library computers ·         total number of students working in groups (and total groups) ·         total number of laptops in use   library staff generated a map of the floor and used various symbols to describe the categories outlined above (see appendix a). data from this recording process showed that patrons use the library as their main study hall and collaboration space. within 15 minutes of opening and until closing, patrons used the library to work independently and in groups.  students working on personal laptops, who did not use a computer terminal at the time of observation, made up 74% of patrons using the library. students working in groups located at large tables made up 37% of patrons, while 29% used library computer terminals. students working alone made up 62% of patrons, and 58% percent used a library terminal. the library space includes large tables, small tables, and individual carrels. at intervals throughout the day/evening, patrons occupied all locations.   survey   who uses the library, how do they use it, and why? a survey of library users provided a range of information about user preferences and behaviors. staff administered the survey virtually, via the library website and the campus digital news source “ndnu pulse.” the student body at notre dame de namur is relatively small at just under 2000 students. patrons completed over 300 surveys, representing nearly 15% of possible respondents. a copy of the survey is included in appendix b.    use the survey measured how often patrons reported using specific features and services of the library. meeting with a librarian was the most frequent service used, followed by using the book collection, using a computer for academic work, getting research help, using wifi, using copiers, using a power outlet, and studying alone. patrons reported finding a reserve book, studying in a group, printing, and scanning less frequently, followed by meeting with a tutor/professor, using a table, using a computer for non-academic reasons, and meeting with friends. as should be expected, the microfilm/microfiche collection showed the least amount of use. the high frequency of “meeting with a librarian” speaks well of the library’s integration into the curriculum, and to the value of patron-oriented service to the library’s users.       importance the survey attempted to measure how important specific features and services provided by the library are to its users. survey respondents ranked the need for quiet study spaces as very important, followed by meeting with a librarian and attending a library class. next in importance were a desire for longer hours, power outlets, group study spaces, access to desktop computers, more comfortable furniture, a browsable journal collection, printing services, access to software, moveable furniture, scanning, photocopying, and an oral presentation practice space.  browsing the book collection ranked lowest, yet it was still a 29% favorability ranking. respondents commenting on open discussion boards also ranked the need for quiet study space as the highest priority for users.      satisfaction the survey prompted users to identify how satisfied they are with specific features and services of the library. respondents ranked satisfaction with librarians the highest. users were also very satisfied with the library databases and the borrowing desk. users were unsatisfied with the quality of wifi, the quality of the library collection, access to power outlets, and lastly they were not happy with their access to the reference desk. high satisfaction rankings for librarians speaks well of the library’s mission to provide excellent, hands-on, patron-oriented service.   interviews   who uses the library, how do they use it, and why? included in the online survey was an option to conduct an in-person interview. of the 300 surveys completed, 24 interviewees were identified. interviewees were mostly upper-level students: 11 fourth-year, 6 third-year, 3 second-year, 1 first-year, and 3 graduate students. all participants were from the social sciences and the humanities, including the graduate students. interviewees reflected the demographics at ndnu, with a majority of white females participating. for more demographic information on ndnu, please see https://www.niche.com/colleges/notre-dame-de-namur-university/. interviews were completed in an informal setting (a faculty member’s office) following a pre-arranged interview guide (appendix c). the lead researcher, a formally trained anthropologist, conducted the interviews. as was uncovered using other methods of inquiry, most students described the same conditions, needs, and desires about the library. students noted:   ·         the library is loud ·         there are not enough group study spaces ·         there are not enough power outlets ·         the wifi is poor   however, one broad-based question (why is there a library on campus?) elicited many interesting responses. a key narrative in these comments that did not emerge from the other data centres on the idea of a learning environment. for example, respondents noted that libraries are on campus to intrigue and encourage students. they saw the library as a community centre for learning. one respondent noted the library is here to “foster the idea of a community of students who are very into their studies.” many libraries are adapting from housing collections to an information commons model, and our users seem eager for this change. not only do students want new technologies, they also want to “enjoy the library as a contemplative oasis” (freeman, 2005, p. 6). the library at ndnu is a space where a learning commons is prospering, even within the constraints of its current physical structure. students identify with the idea of a shared community even if they do not always articulate how the library fosters this concept. students desire a space they can claim as their own for the making, creating, learning, and exploring that happens outside of the classroom. this freedom to create such an academic space makes the library special, and central, to student life. in many ways, the library acts as a middle ground between social space, private space, and academic space. many interviewees noted how they prefer to come to the library late, after classes and dinner, and even after socializing. other interviewees see the library as a collaborative space, even with the current structure somewhat limiting group learning.  of the 24 interviewees, 18 commented that the library is a social space in some context, either as a place to discuss questions raised in the classroom or as a destination to meet other students and plan activities. each of these activities helps to build a community and a cultural space, creating a learning environment that is unique to ndnu.   comment boards (flip charts)   who uses the library, how do they use it, and why? at strategic locations throughout the library, library staff placed large paper flip charts with attached markers, and wrote questions for patrons to answer at the top of each (appendix d). flip charts were accessible for two weeks at the beginning of each month. users self-responded directly on the comment boards. overwhelmingly, the comment boards revealed four primary issues:   ·         noise – 28 comments noted a need for noise reduction/quiet space ·         wifi – 20 comments noted a need for better wifi ·         group space – 11 comments noted the need for group study space ·         power access – 9 comments noted a need for more electrical outlets   looking at the total comments noting a need for quiet space, four comments isolated “social noise” as the primary sound issue (example: “the library is not a hangout it is a place to read, study, and get work done”). commenters also requested that library staff “respect the need for quiet” and that they “enforced less talking.” the issue of “social noise” is a challenge for the development of an information commons at ndnu.   commenters also made specific requests:   ·         a multi-media lab (in process, 2017) ·         a research lab (in process, 2017) ·         white boards (added, spring 2016) ·         glass boards (test board added, spring 2016) ·         more large tables (added, spring 2016) ·         more stuffed chairs (added, spring 2016) ·         add tvs with beds and pillows (monitor  with streaming content added, spring 2016) ·         add inspirational quotes (new mural about diversity added, spring 2016) ·         bluetooth printers (in process, 2017) ·         get rid of the smelly carpet by the printers (additional steam cleaning performed, spring 2016) ·         provide better air conditioning (ac replaced, summer 2016) ·         a snack bar (altered policy to allow food from the cafeteria, spring 2016) ·         a coffee cart (altered policy to allow drinks from the cafeteria, spring 2016) ·         more single use desks (added, spring 2016) ·         more computer stations (additional laptops and ipads added, spring 2016) ·         more computers just for printing (in process, 2017) ·         more comfy chairs (added, spring 2016) ·         hooks in the bathrooms for book-bags (in process, 2017) ·         more light, stay open later (added additional hours, spring 2016) ·         open the library earlier (added additional hours, spring 2016) ·         unlimited printing (students now receive 500 pages free) ·         “bathrooms that don’t look like insane asylums” (in process, 2017) ·         “please remove the gum from the walls where the individual desks are” (completed, spring 2016)   discussion   what modifications to the library’s space could transform it into a modern information commons? determining how often patrons use library facilities is critical to envisioning the library’s future. realizing that this is no small task, the methods proposed by the project leader provided a viable alternative to simple daily data collection (i.e. door counts, etc.). the use of libraries has changed over time from primarily textual study to collaboration and digital information seeking. performing patron use studies provides the evidence necessary to make effectual decisions about how facilities should be changed or modified to meet the needs of an ever-changing patronage.   from early on in the process, library staff conducting the patron use recording found that the library is a highly used study space, especially by individual students. these data enabled the library to justify the need for additional group study spaces and, we hope, will lead to a major renovation to facilitate active, collaborative learning. the library is the only space on campus dedicated to studying. when asked what other spaces were available for studying, residential students (those who live on campus) chose their apartments. however, students who commute to campus dismissed alternatives to the library such as the commuter lounge or the writing center. for example, one interviewee noted “i would probably try and study in the [writing] center, but i find it is really too noisy in there sometimes, it is not like the library because it is too enclosed.  i can’t even take tests in there either…i mean the new building is nice but i would rather be in the library.”   scholarly evidence notes how physical book circulation has declined over the years (allison, 2015). when students responded about the value of having book stacks in the library, a wide range of discussions emerged that centred on the idea that the presence of books helps students feel like the library is a place of knowledge and learning. one student noted during an interview, “it makes me feel like i am being productive, you know, that’s why i like being in the library…you are surrounded by a lot of knowledge, so it makes me feel more motivated; it motivates you.” while today's academic library users browse books less, they still value the possibility of doing so.   outcomes the findings from this study resulted in many improvements for patrons at the gellert library. responses provided by students on the comment boards gave an excellent list of minor and major problems. the survey’s results showed what services are valued, and how satisfied users are with the library building. following users’ suggestions, library staff relocated the information desk to a more central area in the building. the reference print collection, substantially reduced and merged into the main circulating collection, is now nearly non-existent. its removal created a lot of space around the information desk. this allowed for the relocation of more comfortable seating, taken from a “reading room” in the rear of the building, to the reference area. large tables are on one side of the building, with smaller round tables located around the reference desk. these few changes have substantially altered the way that students use the reference area. increased reference desk use statistics, including more one-on-one collaborations, proves this renovation was useful to patrons. these changes helped to create a physical information commons in the gellert library. capital improvements on campus resulted in the library having improved access to the campus electrical grid and internet. a fibre optic backbone, completed over the summer of 2015, dramatically increased the quality of the campus network. library staff installed three 885-joule surge-suppressing power strips to a central, curved partition called the “art wall.” this provided power access to an area of the library that previously had none. each wall outlet positioned adjacent to a study area had surge-protecting wall taps added. not only did this add power outlets to the floor, but each also included multiple usb ports for peripheral charging. improved dedicated carrels in individual study areas have had charging stations added to the desks. library staff installed a multi-device charging station in the library foyer, as well as a monitor for streaming information content in the same area.  a fleet of 20 laptops and 15 ipads are now circulating to library users. lastly, facilities replaced the old air conditioning/heating system in the summer of 2016. the combination of increased power access, increased network quality, better quality environmental control, and additional technology shows substantial moves toward a more robust virtual information commons.   library staff created a dedicated quiet study area complete with additional carpeting, indoor plants, new artwork, individual carrels with lamps, and multi-port surge-protecting power strips. re-positioning of large tables to one end of the main floor, and grouping smaller tables together on the other end allowed for some sense of separate study spaces. this has been successful in reducing “social noise” complaints. two further areas have been designated “study lounges,” complete with overstuffed chairs, lap-rest boards, and coffee tables. an improved classroom was created with space made available by substantially reducing the print journal collection, and has a wall-mounted smart board, modular furniture, multiple mobile white boards, and a mobile smart board. this dramatically improved instruction and collaboration in the library. these changes are facilitating a cultural information commons at the gellert library.   a complete inventory of the collection, with an orientation towards refurbishment, was finished in spring 2016. the inventory will help librarians and faculty work through a thorough weeding process, making the physical collection more current, browsable, and complete. the inventory project will also allow library staff to consider ways to highlight the collection in the building, thus creating an environment such as those described by patrons in the formal interviews. lastly, dedicated group study rooms (presently labeled a learning commons in the architectural renderings) will be included in the coming renovation of the ralston manson, a beautiful historic structure located on campus. the addition of some enclosed study rooms, even if they are not in the library itself, will complete the list of priority changes elicited via this research.   conclusions students today do not require the services once demanded by previous generations of library users. the present library building, designed to facilitate individual, quiet, textual based learning, no longer serves the needs of its patrons. analysis of this project’s data supports the need to develop quiet study spaces, to increase access to power outlets, and to develop group study spaces. patrons are satisfied with access to computers, librarians, the library collection, and even to some extent the current building. when asked to envision a new library building, respondents instead discussed alterations to the present one. many respondents described the value of the building as a marker of community for users, especially for students living on campus. building a learning community was especially important to students as they envisioned what a library “is.” not only did students identify the library as a place where knowledge is stored and accessed, as a place of active learning, but also as a place of knowledge sharing between individuals. in essence, they described an information commons.   today, most students at ndnu can access tremendous amounts of information using their personal devices. yet the role of the physical library on campus is even more important than ever before. the gellert library is not just a place to store books. rather, it is a place where meaning and learning emerges from access to knowledge. as the library continues its transformation into an information commons, it has become a welcoming space that encourages exploration, creation, and collaboration between students, teachers, and the broader community. we hope that our library will continue to inspire users to construct new knowledge and meaning.   references allison, d. (2015). measuring the academic impact of libraries. portal: libraries and the academy, 15(1), 29-40.   asher, a., & miller, s. (2010). so you want to do anthropology in your library?: a practical guide to ethnographic research in academic libraries. illinois, the ethnographic research in illinois academic libraries (erial) project. retrieved from http://www.erialproject.org/wp-content/uploads/2011/03/toolkit-3.22.11.pdf   beagle, d. (1999). conceptualizing an information commons. journal of academic  librarianship, 25(2), 82-89. http://dx.doi.org/10.1016/s0099-1333(99)80003-2   beagle, d. (2006). the information commons handbook. new york: neal-schuman.   beagle, d. (2011). from learning commons to learning outcomes. educause center for applied research bulletin, fall, 1-11. retrieved from https://net.educause.edu/ir/library/pdf/erb1114.pdf   bennett, s. (2008). the information or the learning commons: which will we have? the journal of academic librarianship, 34(3), 183-185.   bennett, s. (2009).  libraries and learning: a history of paradigm change. portal: libraries and the academy, 9(2), 181-197.   blum, s. d. (2009).  my word! plagiarism and college culture. ithaca, ny: cornell university press.   bryant, j. (2007). an ethnographic study of user behavior in open3 at the pilkington library, loughborough university (master’s dissertation). retrieved from https://dspace.lboro.ac.uk/2134/3136     bryant, j. (2009). what are students doing in our library? ethnography as a method of exploring library user behavior. library & information research, 33(103), 3–9.   delcore, h. d., mullooly, j., & scroggins, m. (2009). the library study at fresno state. fresno, ca: institute of public anthropology, california state university. retrieved from http://fresnostate.edu/socialsciences/anthropology/documents/ipa/thelibrarystudy(delcoremulloolyscroggins).pdf   duke, l. m., & asher, a. d. (eds.). (2011). college libraries and student culture: what we now know. chicago: american library association.   foster, n. f., & gibbons, s. (2005). understanding faculty to improve content recruitment for institutional repositories. d-lib magazine, 11(1). retrieved from http://dlib.org/dlib/january05/foster/01foster.html   foster, n. f., & gibbons, s. (eds.). (2007). studying students: the undergraduate research project at the university of rochester. chicago: association of college and research libraries.   head, a. j., & eisenberg, m. b. (2009). lessons learned: how college students seek information in the digital age. washington: information school, university of washington. retrieved from http://ctl.yale.edu/sites/default/files/basic-page-supplementary-materialsfiles/how_students_seek_information_in_the_digital_age.pdf   heitsch, e. k., & holley, r. p. (2011). the information and learning commons: some reflections. new review of academic librarianship, 17(1), 64-77. doi:10.1080/13614533.2011.547416   holland, b. (2015). 21st-century libraries: the learning commons. retrieved october 02, 2016, from http://www.edutopia.org/blog/21st-century-libraries-learning-commons-beth-holland   jahn, n. (2008). anthropological motivated usability evaluation: an exploration of ireon – international relations and area studies gateway. library hi tech, 26(4), 606–621. doi 10.1108/07378830810920932   khoo, m., rozaklis, l., & hall, c. (2012). a survey of the use of ethnographic methods in the study of libraries and library users. library and information science research, 34(2), 82-91. doi:10.1016/j.lisr.2011.07.010   moffatt, m. (1989).  coming of age in new jersey: college and american culture. new brunswick, nj: rutgers university press.   nathan, r. (2005).  my freshman year: what a professor learned by becoming a student. ithaca, ny: cornell university press.   ostrander, m. (2008).  talking, looking, flying, searching: information seeking behavior in second life. library hi tech, 26(4), 512–524. doi 10.1108/07378830810920860   othman, r. (2004). an applied ethnographic method for evaluating retrieval features. electronic library, 22(5), 425–432.   rawal, j. (2014). libraries of the future: learning commons a case study of a state university in california (master’s thesis). retrieved from http://hdl.handle.net/10211.3/134872   sandstrom, a. r., & sandstrom, p. e. (1995). the use and misuse of anthropological methods in library and information science research. library quarterly, 65(2), 161-99.   smale, m., & regalado, m. (2010). undergraduate scholarly habits ethnography project.  grace-ellen mccrann memorial lecture, lacuny spring membership meeting, cuny graduate center, june 11, 2010.   suarez, d. (2007). what students do when they study in the library: using ethnographic methods to observe student behavior. electronic journal of academic and special librarianship, 8(3). retrieved from http://southernlibrarianship.icaap.org/content/v08n03/suarez_d01.html   appendix a patron recording map staff used this map to record where patrons were sitting, if they were using a laptop, and if they were studying individually or in groups.   appendix b library survey staff used this survey to measure the frequency of use and patron satisfaction with the library space and services.     appendix c space assessment interview guide the project leader used formal interviews to gain insight into who used the library and why, and to gain an understanding of how users viewed the library building.   what academic year are you? do you live on campus or commute? what days are you on campus?  when you are on campus, where do you study and why? where else is there to study besides the library? what is the ideal setting for you when studying? what kinds of academic activities do you do when you are in the library? when in the semester do you use the library the most?     why is there a library on campus?            what do librarians do? describe the importance of having a library to you/your major have you gotten help from library staff? tell me about your experience… have you used the paper book collection? tell me why and how… do you use the databases? tell me why and how… do you use the library website? tell me why and how… if you imagined the perfect group study space, what would it look like? if you imagined the perfect individual study space, what would it look like? when you see a library with shelves of books, what does it make you think about and how does it make you feel? when you see a row of computer terminals in a library, what does that make you think about and how does that make you feel? have you ever been in the library and not had access to a computer? what do you do? what would your ideal library look like? what other uses could the library fulfill?   appendix d comment board prompts staff used comment boards to give students a venue for providing suggestions and comments about issues important to the research.   what would make the library instruction space a better learning environment? in a couple words, describe the perfect individual study space in a couple words, describe the perfect group study space what do you like about the library space? what would you like to see different? in a couple of words, tell us all the reasons you use the library what matters the most to you about the library?     evidence summary   age and context sensitivity associated with reduced success in finding health information online   a review of: agree, e. m., king, a. c., castro, c. m., wiley, a., & borzekowski, d. l. g. (2015). “it’s got to be on this page”: age and cognitive style in a study of online health information seeking. journal of medical internet research, 17(3), e79. http://dx.doi.org/10.2196/jmir.3352   reviewed by: cari merkley associate professor mount royal university library calgary, alberta, canada email: cmerkley@mtroyal.ca   received: 01 sep. 2015   accepted: 01 sep. 2015      2015 merkley. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine the extent to which age and cognitive style influence an individual’s ability to successfully locate health information online.   design – quantitative study employing scales and regression analysis.   setting – a school of public health and a school of medicine at two universities in the united states of america.   subjects – 346 men and women 35 years or older.   methods – participants for the online health study (ohs) were recruited from the community at both study sites using stratified sampling and screened with a web-based tool to ensure they had the necessary level of digital literacy to complete the study tasks. once enrolled, participants completed the rapid estimate of adult literacy in medicine (realm) to measure their health literacy and the witkin group embedded figures test (geft) to determine their cognitive style (labelled by researchers as context sensitive or context independent). participants were asked to search online for answers to six specific questions on heart-healthy diets, flu vaccinations, alternative medicine and memory, genetic testing, assistive medical technology, and skin cancer, with 15 minutes of search time allowed for each question. participants reported their answers after each search, which were later assigned scores for accuracy and for specificity. when combined, these two scores were used as a measure of success. researchers used stata 11 statistical software to run logistic regression, ordinal logistic regression, and generalized linear models on the data in order to predict which variables were associated with success on the search tasks.   main results – only 323 of the 346 participants completed all study tasks, and their data formed the basis of the analysis. on average, participants correctly answered 4.1 out of 6 questions. participants provided the most accurate and successful answers for the question on heart-healthy foods, and the least accurate answers on the question about seasonal flu shots. they were the least successful in answering the question about herbal supplements for memory. across all models, older participants were less likely to be successful in locating the answers to the questions than younger participants, even controlling for the other variables measured in the study. in particular, older participants had the most difficulty with the question on medical technology, which required the use of mapping. overall, the models suggest that higher levels of education, greater daily internet use, and higher health literacy were associated with greater success on the search tasks, the extent to which varied from question to question. the exception in the case of education was the question relating to herbal supplements and memory, as participants with higher levels of education were more likely to score poorly in their responses. participants whose cognitive style was found to be context sensitive were less likely to find the information needed in their online searches than those who were context independent, particularly on the questions relating to a heart-healthy diet, skin cancer, and medical technology.   conclusion – the study suggests that age, cognitive style, level of health literacy, daily internet use, and prior education are all important variables in determining whether an individual can successfully take advantage of the increasing amount of health information available on the internet. specific approaches to web design could be used to improve the success rate of those who are context sensitive, and greater support and direction to reputable online health sources from medical and information professionals could assist those who are less health literate.   commentary   for those working to address disparities in health literacy, the results of the online health study suggests that there is a tough road ahead. even among a non-representative sample characterized by familiarity with the internet, high levels of health literacy, and in many cases a college education, only 9% of the study participants were able to correctly answer all 6 health related questions (p. 10). older adults struggled more than their younger counterparts, even when their computer skills were not in question. while the study acknowledges the potential role of librarians in supporting health literacy, they may face the same struggles with having adequate time, resources, and training to do so as the authors note for medical professionals (luo & park, 2013). studies such as this demonstrate that this work, while challenging, is necessary for public health.   while cognitive styles and their relationship to search behaviour have been studied extensively, their specific impact on electronic health literacy appears to be relatively unexplored. the decision to use the terms “context sensitive” and “context independent” to describe participants’ cognitive styles throughout the article is curious. the geft test itself specifically measures the concepts of field independence and field dependence (witkin, 1971), and those are the terms used in the studies cited by the researchers to support their discussion of cognitive styles. employing this more common terminology would be helpful to readers wishing to explore the concepts further in the literature, and not contribute to the already confusing array of potentially synonymous terms used to describe cognitive (or learning or intellectual) styles. there is ongoing debate among scholars about how field independence-dependence should be best addressed, or whether it needs to be addressed at all, in the design of online resources and learning environments (evans, richardson, & waring 2013), which may not be clear from the limited literature review provided.   the literature review also references only a fraction of the extensive research that has taken place on older adults’ use of the internet. variables such as income, health status, race/ethnicity, and access to broadband internet have been shown to impact internet use among this population, but their influence was not investigated in this study (e.g., choi, 2011; flynn, smith, & freese, 2006; smith, 2014). further detail on the regression analysis conducted on the included variables should have been reported, such as the goodness of fit of the models or the p-values (lang & altman, 2013). it would also have been helpful to know if the computer labs where participants completed their exercises at each institution were comparable in terms of available equipment and systems.   the findings of this study may be of interest to information professionals who work with the public on questions of consumer health, as well as those involved in the development or selection of online tools addressing this need. those interested in cognitive styles and the design of online resources will wish to explore the literature further to find the evidence needed to support their decision making.   references   choi, n. (2011). relationship between health service use and health information technology use among older adults: analysis of the us national health interview survey. journal of medical internet research, 13(2), e33. http://dx.doi.org/10.2196/jmir.1753   evans, c., richardson, j. t. e., & waring, m. (2013). field independence: reviewing the evidence. british journal of educational psychology, 83(2), 210-224. http://dx.doi.org/10.1111/bjep.12015   flynn, k. e., smith, m. a., & freese, j. (2006). when do older adults turn to the internet for health information? findings from the wisconsin longitudinal study. journal of general internal medicine, 21(12), 1295-1301. http://dx.doi.org/10.1111/j.1525-1497.2006.00622.x   lang, t. a, & altman, d. g. (2015). basic statistical reporting for articles published in biomedical journals: the “statistical analyses and methods in the published literature” or the sampl guidelines international journal of nursing studies, 52(1), 5-9. http://dx.doi.org/10.1016/j.ijnurstu.2014.09.006   luo, l., & park, v. t. (2013). preparing public librarians for consumer health information service: a nationwide study. library & information science research, 35(4), 310-317. http://dx.doi.org/10.1016/j.lisr.2013.06.002   smith, a. (2014). older adults and technology use. retrieved from the pew research centre website: http://www.pewinternet.org/2014/04/03/older-adults-and-technology-use/   witkin, h. a. (1971). group embedded figures test [database record]. retrieved from psyctests, http://dx.doi.org/10.1037/t06471-000       evidence based library and information practice 2011, 7.2 101 evidence based library and information practice news/announcements congratulations to the uk north west clinical librarian systematic review and evaluation group 2012. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. congratulations to the uk north west clinical librarian systematic review and evaluation group who have won the practitioner researcher excellence award which has been awarded by the uk lis research coalition. the group comprises a 15 member team of clinical librarians who work at a range of organizations across the uk nhs. in the first instance the librarians conducted a systematic review to determine how best to evaluate clinical librarian services. in a follow up project, they conducted a region-wide evaluation which put the recommendations of the review into practice to demonstrate how clinical librarians make an impact on a wide range of organizational outcomes. interim results of the project were presented at eblip6. the group was mentored by alison brettle of salford university and the project is an excellent example of evidence based library and information practice. the award will be presented at the dream conference on 9th july 2012. further information about the prizewinners and the project is available at http://lisresearch.org/2012/05/31/north-westclinical-librarian-systematic-review-andevaluation-group-wins-practitionerresearcher-excellence-award/ and http://www.salford.ac.uk/nmsw/research/healt h-technologies-and-evidence further information about the dream conference, which may well be of interest to eblip readers, is available at http://lisresearch.org/dream-project/dreamevent-5-conference-monday-9-july-2012/ http://lisresearch.org/2012/05/31/north-west-clinical-librarian-systematic-review-and-evaluation-group-wins-practitioner-researcher-excellence-award/ http://lisresearch.org/2012/05/31/north-west-clinical-librarian-systematic-review-and-evaluation-group-wins-practitioner-researcher-excellence-award/ http://lisresearch.org/2012/05/31/north-west-clinical-librarian-systematic-review-and-evaluation-group-wins-practitioner-researcher-excellence-award/ http://lisresearch.org/2012/05/31/north-west-clinical-librarian-systematic-review-and-evaluation-group-wins-practitioner-researcher-excellence-award/ http://www.salford.ac.uk/nmsw/research/health-technologies-and-evidence http://www.salford.ac.uk/nmsw/research/health-technologies-and-evidence http://lisresearch.org/dream-project/dream-event-5-conference-monday-9-july-2012/ http://lisresearch.org/dream-project/dream-event-5-conference-monday-9-july-2012/ research article   do institutional repository deposit guidelines deter data discovery?   shawn w. nicholson associate dean for digital initiatives michigan state university libraries east lansing, michigan, united states of america email: nicho147@msu.edu https://orcid.org/0000-0002-2144-3578   terrence b. bennett business / economics librarian r. barbara gitenstein library the college of new jersey ewing, new jersey, united states of america email: tbennett@tcnj.edu https://orcid.org/0000-0002-1469-0271   received: 20 jan. 2021                                                               accepted: 5 july 2021          2021 nicholson and bennett. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   doi: 10.18438/eblip29913     abstract   objective – this study uses quantitative methods to determine if the metadata requirements of institutional repositories (irs) promote data discovery. this question is addressed through an exploration of an international sample of university irs, including an analysis of the required metadata elements for data deposit, with a particular focus on how these metadata support discovery of research data objects.   methods – the researchers worked with an international universe of 243 irs. a codebook of 10 variables was developed to enable analysis of the eventual randomly derived sample of 40 institutions.   results – the analysis of our sample irs revealed that most had metadata standards that offered weak support for data discovery—an unsurprising revelation in view of the fact that university irs are meant to accommodate deposit and storage of all types of scholarly outputs, only a small percentage of which are research data objects. most irs seem to have adopted metadata standards based on the dublin core schema, while none of the irs in our sample used the data documentation initiative metadata that is better suited for deposit and discovery of research datasets.   conclusion – the study demonstrates that while data deposit can be accommodated by the existing metadata requirements of multi-purpose irs, their metadata practices do little to prioritize data deposit or to promote data discovery. evidence indicates that data discovery will benefit from additional metadata elements.     introduction   just as every designer knows that form follows function, data professionals adhere to the dictum that documentation drives discovery. university based institutional repositories (irs) continue to play an evolving and expanding role in the scholarly communication ecosystem, including the collection, organization, and dissemination of digital data objects. to remain relevant within this evolving ecosystem, university irs need to support a common language that advances data discovery—not only across academic institutions, but throughout the wider research data network. a first and crucial step in promoting this common language is the design of deposit forms and guidelines for the metadata that accompanies digital data deposit, which is essential for discovery, reuse, and interoperability—the fundamental elements of the fair (findable, accessible, interoperable and reusable) guiding principles first articulated by wilkinson et al. (2016). while these principles advocate machine action, human readability and full understanding of means of access remains important.   this paper reports on the results of our empirical exploration across an international sample of university irs to analyze the required metadata elements for data deposit. specifically, we examined ir deposit forms and guidelines to determine comparable fields as mapped against the dublin core schema, with a particular focus on how these guidelines support the requirements and expectations for data discovery within and across diverse academic disciplines.   literature review   a diverse and rich body of literature exists across the three lines of inquiry—institutional repositories, open access movements, and metadata requirements—that wonderfully intertwine to undergird our motivations for this research. lynch (2003) got right to the heart of the matter by observing, “the development of institutional repositories emerged as a new strategy that allows universities to apply serious, systematic leverage to accelerate changes taking place in scholarship and scholarly communication” (p. 327). the article clearly and importantly framed the ir as a service that is greater than the sum of its software and hardware parts. additional points of emphasis included the importance of standards development and targeted metadata handling, with the latter clearly signaling to ir developers then, as now, the need for broad organizational commitment. writing only a few years after lynch, green and guttmann (2007) reminded readers of the relatively long history of discipline/domain specific digital repositories emanating from the social science data community. they promoted the idea that the ir and discipline/domain repository developers, despite different missions and roles, must find ways to work in concert so as to fully support the research community while concomitantly optimizing data stewardship.   shortly thereafter, salo (2008) caused a stir with a frank assessment of irs by plainly stating, “most repositories languished understaffed and poorly-supported, abandoned by library and institutional administrators, scoffed at by publishers, librarians, and open-access ideologues” (p. 99). salo painted a grim picture of self-archiving practices, and dimmed the bright promise of authors agreeably inputting needed metadata. the article helpfully and hopefully concluded with a series of ideas to advance ir goals that would engender success. chief among these was advice for easing deposit through simplifying the input forms and metadata requirements.   when considered as a whole, this early ir literature might easily be mapped to the hype cycle made popular by the information technology firm gartner, specifically tracking the curve that begins with the “innovation trigger” and continues through toward the end of 2010 when the idea of irs seemed to rest in the “trough of disillusionment” (gartner, n.d.).    in parallel to and at times clearly intertwined with the ir discussions are the arguments for and against open access (oa). oa not only promotes unfettered access to content, but also offers broad benefits to scholarly practice by enabling replication, thus reducing duplicate data collection efforts, and accelerating scientific progress. the early 2000s witnessed a series of oa statements and pronouncements from broad coalitions; notable among these are the budapest open access initiative and the organisation for economic co-operation and development (oecd) final communique. the budapest statement (2002) advocated for “free and unrestricted online availability” (para. 1) to peer reviewed journal literature and, importantly, pointed up the value of self-deposit. the oecd communique (2004) illuminated the importance of documentation to make data available and accessible internationally. investigating the development of open access journal publishing, laakso et al. (2011) documented a rapid growth in article output from the early 1990s through 2009 while also observing a demonstrable increase in the number of journals providing oa.   with oa journals filling up a growing number of irs, an emerging need arose to facilitate efficient discovery and dissemination of the content. the open archives initiative (oai) gained prominence (lagoze & van de sompel, 2003). the oai’s chief aim was to promote interoperability through standards-based exposure and exchange of metadata. while libraries have had a long history of generating metadata records for catalogues and indexes, it was becoming clear that the role of researchers and practitioners also had to be taken into account for the growing number of digital repositories to be successful. robertson (2005) noted the importance of including “skills from computer science and learning technology as well as lis, together with enthusiasts and from a great diversity of other disciplines as well” (p. 296).   moulaison sandy and dykas (2016) surveyed a random sample of administrators of us-based irs included in the opendoar (directory of open access repositories) registry. the authors concluded that staffing, standards, and systems combined to enable quality metadata. above all, qualified staff proved to be most crucial. many authors were beginning to investigate the impact of quality metadata upon discovery across a broad range of fields. giuliani et al. (2016) declared that “metadata production is still perceived as a complex, tedious and time-consuming task. this typically results in little metadata production and can seriously hinder the objective of facilitating data discovery” (p. 239). amplifying this observation, radio et al. (2017) noted that   the proliferation of research datasets and their availability in various repositories require metadata that provides sufficient context and organizational clarity to enable their use. however, datasets come in myriad forms, structures, and relationships. as characteristics of datasets vary across disciplines, it is reasonable to suggest that the methods by which they are discoverable by metadata should be informed by the considerations unique to differing research areas. (p. 161)   recent years have not witnessed any dwindling of scholars, funders, and national policy-making bodies investigating how open access literature, irs, and interoperable metadata interplay. plan s, the europe-backed program, is a set of principles that ensure open and immediate access to funded research publications. it was first launched by coalition-s in 2018 (coalition-s, n.d.). the us office of science and technology policy (2020) call for characteristics of data repositories “ensures datasets are accompanied by metadata sufficient to enable discovery, reuse, and citation of datasets, using a schema that is standard to the community the repository serves” (p. 3086).   as higher education institutions, and especially libraries, continue to explore ways to best provide research data services to their research communities, the use of digital repositories for storing and providing access to datasets continues to evolve. with a focus only on institutions deploying the digital commons ir software, manninen (2018) sought to assess the ease of locating datasets in these irs, and the metadata standards employed in depositing the datasets. finding no universally applied standard and noting that “56 unique fields were identified from the 15 example data items,” the study concludes by reminding the reader that through “robust metadata, curated research data repositories will be discoverable, usable, and interoperable into the future independent of the repository platform” (p. 10). the study by kim et al. (2019) of metadata practices across 20 repositories in three academic disciplines provided the initial impetus for our present study of academic irs. we borrow heavily from their framework and methodology, particularly, the idea that documentation—data deposit forms and attendant guidelines—“performs the dual purposes of defining a contract between depositors and repositories and gathering information about the deposited data” (p. 843). we are especially drawn to the manner in which these authors promote the concept of data reuse. we want to further the conversation by exploring aspects of discovery—a necessary precursor to reuse.   in sum, the rich and varied contemporary scholarly conversation around open access, irs, and metadata suggests that these ideas have moved along gartner’s hype cycle, through the “slope of enlightenment” and heading toward the “plateau of productivity” (gartner, n.d.). the purpose of our project is to advance this conversation with the addition of an empirically-based descriptive component.   aims   this paper reports on the results of our exploration across an international sample of university-based irs to analyze the required metadata elements for data deposit. specifically, we examine ir deposit forms and guidelines to determine comparable fields as mapped against the dublin core schema, with a particular focus on how these guidelines support the requirements and expectations for data discovery within and across diverse academic disciplines. our aim was to explore the following questions:    1.      is a deposit form that allows or enables author-supplied metadata present, and how does the deposit guidance describe metadata requirements? 2.      what is the prevalence and uniformity of metadata standards? 3.      is there use of controlled vocabulary, and evidence of mediated deposit? 4.      is there evidence of deposited research data?   methods   the study sought to analyze irs from an international cohort. to generate a universe of irs, we first made use of the opendoar api to target singapore, australia, the united kingdom, new zealand, and hong kong—an english-language cohort. opendoar is a directory of open access repositories that adhere to specific criteria for inclusion, such as comprising academic content and being freely available. the results (institution name, ir url, country, and software name) were exported as a json structured file, and then the openrefine tool was used to format as a comma separated value (csv) for ready analysis. the us universe was drawn directly from the moulaison sandy and dykas (2016) article. those 50 us-based irs were simply copied and pasted into the master csv file. the resulting universe of irs by country of origin is shown in table 1.   table 1 universe, institutional repositories (irs) by country country number of irs percentage of irs australia 83 34% hong kong 7 3% new zealand 17 7% singapore 6 2% united kingdom 80 33% united states 50 21% totals 243 100%   after using ms excel to assign a random number to each row of the master csv, we sorted the universe in ascending numerical order. using the methodology from kim et al. (2019) as broad inspiration as well as some targeted elements specific to our inquiry, we developed a coding scheme. beginning with two irs randomly selected from the universe, we performed time trial and simplified interrater agreement testing to determine variance in code assignments. our aim was to arrive at appreciable consistency, not an absolute correct code for each case. only minor modifications were necessary to finalize the coding scheme (see appendix). using the randomized numbers, we identified the first 40 results to arrive at a 16.5% sample. the sample tracks reasonably well to the originating universe, as shown by comparing table 1 to the sample 40 repositories by country in table 2.   table 2 sample, institutional repositories (irs) by country country number of irs percentage of irs australia 9 23% hong kong 2 5% new zealand 3 7% singapore 2 5% united kingdom 14 35% united states 10 25% totals 40 100%   once the sample was created, including a direct url associated with each ir, we systematically browsed each ir’s author and user guidelines, depositor instructions, faqs and other supporting documentation in search of key terms, recording coded variables in a shared google sheet. google sheets was chosen for ease of use and shared quality control capabilities. author one coded irs 1-20 and author two coded irs 21-40, with regular discrepancy checks for discussion and resolution throughout the coding period.   results   metadata standards in deposit forms, guidelines and sample records   we first looked for the presence of a deposit form (or self-deposit form) that allows or enables author-supplied metadata to accompany deposits into an ir. such a form was easily discoverable in only four of the irs in our sample, while more than half did not include a form. the remaining irs required potential depositors to log in with a username and password, so if a deposit form was available, we could not view it. however, supplementary guidelines for depositors (apart from a form) were found for more than half of the irs (including half of those with password-protected access for depositors), thereby providing us with a fuller picture of the deposit process. among these guidelines, more than two-thirds discussed or described metadata requirements for materials submitted to the ir. ultimately, we discovered some mention of metadata requirements, either directly within a deposit form or described in supplemental guidelines, for just under half of the 40 irs examined. finally, we looked at one or more sample records from each ir to glean additional information about metadata standards. from irs for most of the institutions at which deposit forms or guidelines yielded no metadata information, we were able to discern something about metadata standards by examining the sample record. and among the institutions whose ir deposit forms or guidelines provided some initial guidance about metadata requirements, our examination of the sample record offered additional clarification of metadata standards at nearly half of these.   institution size and inclusion of datasets in ir content   while our focus was specifically on the presence of research data in irs, such data storage is unlikely to be the primary focus of an ir. with the exception of repositories that are created for the specific purpose of research data storage and management (one of which was included in our sample), irs tend to emphasize deposit of articles or preprints, working papers, book chapters, reports, and other text-based scholarly outputs. an examination of deposit forms or guidelines (or other descriptive information) from the irs in our sample revealed that just over half included any mention of data or datasets among the types of material to be stored in the ir.   to complement table 2, we gathered enrollment information for the hosting institution of each of the irs in our sample, as shown in table 3. we used this information to examine the association (if any) between the size of the institution and the apparent acknowledgment of research data as a type of record that could be found in the institution’s ir. we discovered that among the 21 institutions in our sample whose irs noted the specific presence of datasets (about half of the total sample), less than one-third of these were smallor medium-sized institutions, while the remaining two-thirds were large institutions with enrollments greater than 15,000.   table 3 sponsoring institution enrollment size total enrollment category number of institutions <5000 small 6 5001-15,000 medium 7 >15,001 large 27   metadata schema   among the irs in our sample, nearly all had records that clearly reflected dublin core (dc) metadata. of these, only a small number reflected weak or moderate use of dc, represented by six or fewer identifiable dc elements; the remaining majority represented full adoption of dc, as evidenced by the presence of more than six dc elements (and more than half of these had ten or more identifiable dc elements).   it is noteworthy to mention that from our examination of deposit forms, guidelines, and sample item records, we found no evidence that any of the irs in our sample had adopted the standards for describing research data specified by the data documentation initiative (ddi), “an international standard for describing the data produced by surveys and other observational methods in the social, behavioral, economic, and health sciences” (ddi alliance, n.d.).   from our analysis, we were able to determine that just over half of the irs in our sample maintained uniformity across item records by imposing some sort of controlled vocabulary onto their metadata elements. of the remaining irs, a few allowed for metadata tagging that did not adhere to a controlled list of elements, while there was insufficient evidence from the remaining irs [n=15] for us to determine if a controlled vocabulary was in use.   mediated deposit   deposit of items into the ir was either fully controlled by librarians or other ir staff, or partially mediated (self-deposit with administrator review) at more than half of the irs in our sample. not surprisingly, two-thirds of the institutions that had a mediated deposit process were large institutions (table 3) where, presumably, ir staff was likely to be larger and, therefore, able to take on the responsibility of overseeing the deposit of records into the ir. only three of the irs in our sample appeared to support total self-deposit with no administrative review, while we were unable to determine if mediated deposit was present at nearly one-third of the irs in our sample.   discussion   as evidenced from the results of the studies cited above, the consensus conclusion is that irs are doing an inadequate job of promoting and enabling data discoverability; our study offers additional support for this conclusion. integral to discoverability is the presence of descriptions and metadata. metadata matters, and as meadows asserts in a 2019 scholarly kitchen post, it could very well help save the world! while it is easy to delight in this hyperbolic blog post title, the importance of descriptive elements that enable findability and interoperability should be evident. meadows’ crucial contention that metadata creates efficiencies is illuminated through four key actors: creators, curators, custodians, and consumers. in figure 1, we provide a graphic representation of meadows’ actors and their roles, and find resonance with meadows’ behavioral nudge that each actor shares “collective responsibility to ensure that [metadata] is the best it can be” (2019).   figure 1 four cs of metadata, adapted from meadows (2019).   whereas kim et al. (2019) focused on the role of metadata in data-deposit requirements to enable reuse of data from discipline-based repositories, we sought to build upon their study by examining the role of metadata to promote data discovery in multipurpose (and multidisciplinary) academic irs. in their analysis of data deposit requirements compiled from previous studies, kim et al. noted that “methodology” is included among the five most common data deposit metadata elements in discipline-specific repositories (p. 845). since methodology is not a dc metadata element, it is reasonable to intuit that the metadata employed by the non-discipline-repositories in our study is even less effective in enabling data discovery.   from their study of the data curation practices in irs, lee and stvilia (2017) noted “a dearth of research, on identifier metadata quality, uses and practices for research data in the context of irs” (p. 2), as these irs relied too heavily on simple dc metadata elements, which fail to accommodate the complexity and diversity of datasets. the lack of dataset-specific metadata is connected to issues with reusing, sharing, and searching the data. our observations reinforced these findings, particularly, as noted, none of the irs in our sample used a more granular schema such as ddi. as garnett et al. (2017) state, dc is in wide use because "by default, most oai portals serve dublin core metadata" (p. 208). the open archives initiative protocol for metadata harvesting standard's chief aim is to promote interoperability through standards-based exposure and exchange of metadata (open archives initiative, n.d.). the reliance on dc elements for discovery, as our results suggest, omits important data-specific metadata. garnett et al. (2017) refer to these as "structural metadata elements that describe dataset variables, questions, concepts, categories, values, etc. that accompany the physical dataset" (p. 206).   from the results of our study, it seems unlikely that data-seeking researchers will develop any reliance on serendipitous searching for data discovery. using the definition of discovery put forth by schonfeld (2014) as “the process and infrastructure required for a user to find an appropriate item” (p. 3), we would have to concede that while irs may support limited discoverability of text-based academic outputs (depending on how well they are configured for search-engine optimization), scholars in pursuit of existing datasets to support their research are likely to be disappointed by irrelevant or nonexistent search results within irs. this would perpetuate the practice of general data-seeking only within discipline-based data repositories, with researchers only occasionally turning to an academic ir in pursuit of a known item (or at least an expected-to-exist item).   and it is probably unfair to consider this an unacceptable result. to remain relevant, an academic ir also has to remain current, so the first priority of its administrators is to ensure ongoing deposit (with a focus on more commonly occurring text-based scholarly outputs) in order to avoid the dormancy and abandonment predicted by salo (2008) and subsequently reinforced by others. for example, in 2011, giesecke observed that irs still can appear to be "a solution in search of a problem," and that continued evidence of nonparticipation by faculty scholars "would make one ask why should institutions reallocate resources to create and maintain an institutional repository" (p. 540). and as recently as 2017, tillman observed that "salo's assessment of the flaws in repository strategies as then practiced ... and her recommendations for next steps ... remain relevant nine years later" (p. 2). with support devoted to storage and access, there may be few resources (including professional staff) remaining to focus on other considerations such as detailed metadata mark-up. here again we are girded by meadows' contention that good metadata requires actors in multiple roles; the custodians and curators might, ideally, effectively engage data creators (and possibly consumers) towards the goal of improving metadata. except for irs at some large and well-resourced institutions, there will always be constraints from resource limitations. thus, for most irs, there remains little expectation that staff expertise will be devoted to the creation of a robust discovery overlay—especially for data, which, from what we’ve observed, might only comprise a handful of total records in the ir.   while it would require a separate study to gain evidence of researcher behavior, it nevertheless seems unreasonable to infer that these circumstances are likely to provoke potential data discoverers into rattling the ir administrators with demands for improved metadata to support data discovery, as most researchers probably know that they can turn to their discipline-based data repositories when discovery needs to happen. as these disciplinary repositories have already established themselves as a primary venue for data storage, it would be difficult (and probably pointless) for academic irs to compete in that space. as pirolli (2016) notes, “information foraging theory assumes that people adapt their information-seeking behavior to maximize their rate of gaining useful information to meet their ongoing goals” (para. 1). thus, the existing low expectations for data discovery in irs is likely to create a self-perpetuating cycle of inaction: if there is no expectation that researchers will turn to irs for data discovery, then there is no impetus for ir administrators to make enhancements to metadata standards and practices that will enable data discovery. the small population of ir data depositors (compared with depositors of text-based research outputs) are also not poised to raise an outcry, because they’re likely only using the irs for data deposit when it is a convenient way to meet publishers’ data management requirements. so the only remaining cage-rattlers are the authors of studies such as ours—and we are unlikely to capture the attention of ir administrators in a way that would effect significant changes in policy or practice.   notwithstanding this state of current practice, the community continues to grapple with roles and goals, though not without divergent opinion. writing in late 2020 as we were completing this manuscript, shearer and kingsley, representing the confederation of open access repositories, addressed the evolving discussion around criteria for selecting data repositories. their blog post was in direct response to the data repository selection document produced by representatives from journals and publishers who were already working as part of the fairsharing community (fairsharing.org, n.d.). a crucial point of contention rests on the degree to which the community can accept concentration of repository services. it is our hope that all of the key actors mentioned by meadows (2019)—creators, curators, custodians, and consumers—will continue to engage in this discussion.   limitations   while our sample was large enough to afford meaningful observations, a larger sample size is always desirable. our results show, for example, a strong relationship between institution size and a propensity to acknowledge directly that research data may be stored (if not discovered) in an ir. specifically: the larger the institution, the more likely the mention of datasets. while further study would be necessary before we could suggest that these results reflect causation, it is nonetheless reasonable to infer that research institutions with higher enrollments also have more faculty (and others who conduct research), which increases the likelihood that any type of material (including data) makes its way into the ir. a related unsurprising observation is that mediated deposit is more likely to occur at larger institutions. again, an interrelationship does not prove causation; further study would be needed to arrive at any stronger conclusion.   of necessity, we had to limit our analysis to irs at academic institutions in english-speaking countries. it’s possible that the most innovative and forward-thinking advances in ir development are happening at non-english-language institutions, or that these institutions have developed a more advanced culture of data deposit for reuse (and therefore greater advances in data discovery). moreover, if we had used different criteria to generate a sample—such as, for example, starting with a universe of irs in which we had first identified existing records for datasets—then our observations would have focused on practices among irs that are known to accommodate data deposit (and may therefore be expected to enable data discovery by employing data-specific metadata).   conclusions   there is an extensive body of literature around academic irs as enablers of open access, including access to datasets for replication and reuse. the purpose of our study is to present an empirical analysis of the role of metadata in promoting data discoverability, a necessary precursor to reuse. by examining the deposit forms, guidelines, and output of an international sample of academic irs, our observations reinforce the findings of others: while data deposit can be accommodated by the existing metadata requirements of these multi-purpose irs, their metadata practices do little to prioritize data deposit or to promote data discovery.   it is reasonable to expect that this status quo will perdure. academic irs have to address many competing priorities, most of which are skewed toward deposit, storage, discovery and retrieval of text-based scholarly outputs. therefore, the effort and expertise that must be expended to maintain an ir will inevitably continue to favor articles and other texts. within this reality, academic irs will perpetuate their established, and expected, function as convenient containers for a limited subset of research-related datasets.   author contributions   shawn nicholson: conceptualization, data curation, investigation, methodology, software, writing original draft, writing review & editing terrence bennett: formal analysis, investigation, writing original draft, writing review & editing   acknowledgments   the authors wish to thank heather moulaison sandy and felicity dykas for sharing their previously compiled list of us-based irs, which proved to be a time-saving starting point for deriving our sample for this study. we are also very grateful to the two anonymous reviewers for their thoughtful and thorough comments on an earlier draft of this article.   references   budapest open access initiative. (2002, february 14). budapest open access initiative. https://www.budapestopenaccessinitiative.org/read   coalition-s. (n.d.). plan s: making full and immediate open access a reality. https://www.coalition-s.org/   ddi alliance. (n.d.). data documentation initiative (ddi). https://ddialliance.org/   fairsharing.org. (n.d.). fairsharing.org. https://fairsharing.org/   garnett, a., leahey, a., savard, d., towell, b., & wilson, l. (2017). open metadata for research data discovery in canada. journal of library metadata, 17(3-4), 201-217. https://doi.org/10.1080/19386389.2018.1443698   gartner, inc. (n.d.). gartner hype cycle. https://www.gartner.com/en/research/methodologies/gartner-hype-cycle   giesecke, j. (2011). institutional repositories: keys to success. journal of library administration, 51(5-6), 529-542. https://doi.org/10.1080/01930826.2011.589340   giuliani, g., guigoz, y., lacroix, p., ray, n., & lehmann, a. (2016). facilitating the production of iso-compliant metadata of geospatial datasets. international journal of applied earth observation and geoinformation, 44, 239-243. https://doi.org/10.1016/j.jag.2015.08.010   green, a. g., & gutmann, m. p. (2007). building partnerships among social science researchers, institution-based repositories and domain specific data archives. oclc systems & services: international digital library perspectives, 23(1), 35-53. https://doi.org/10.1108/10650750710720757   kim, j., yakel, e., & faniel, i. m. (2019). exposing standardization and consistency issues in repository metadata requirements for data deposition. college & research libraries, 80(6), 843-875. https://doi.org/10.5860/crl.80.6.843   laakso, m., welling, p., bukvova, h., nyman, l., björk, b.-c., & hedlund, t. (2011). the development of open access journal publishing from 1993 to 2009. plos one, 6(6), e20961. https://doi.org/10.1371/journal.pone.0020961   lagoze, c., & van de sompel, h. (2003). the making of the open archives initiative protocol for metadata harvesting. library hi tech, 21(2), 118-128. https://doi.org/10.1108/07378830310479776   lee, d. j., & stvilia, b. (2017). practices of research data curation in institutional repositories: a qualitative view from repository staff. plos one, 12(3), e0173987. https://doi.org/10.1371/journal.pone.0173987   lynch, c. a. (2003). institutional repositories: essential infrastructure for scholarship in the digital age. portal: libraries and the academy, 3(2), 327-336. https://doi.org/10.1353/pla.2003.0039   manninen, l. (2018). describing data: a review of metadata for datasets in the digital commons institutional repository platform: problems and recommendations. journal of library metadata, 18(1), 1-11. https://doi.org/10.1080/19386389.2018.1454379   meadows, a. (2019, june 11). better metadata could help save the world! the scholarly kitchen. https://scholarlykitchen.sspnet.org/2019/06/11/better-metadata-could-help-save-the-world/   moulaison sandy, h., & dykas, f. (2016). high-quality metadata and repository staffing: perceptions of united states-based opendoar participants. cataloging & classification quarterly, 54(2), 101-116. https://doi.org/10.1080/01639374.2015.1116480   office of science and technology policy. (2020). draft desirable characteristics of repositories for managing and sharing data resulting from federally funded or supported research. federal register, 85(12), 3086-3087. https://www.govinfo.gov/content/pkg/fr-2020-01-17/pdf/2020-00689.pdf   open archives initiative. (n.d.). protocol for metadata harvesting (pmh). https://www.openarchives.org/pmh/   organisation for economic co-operation and development. (2004, january 30). science, technology and innovation for the 21st century. meeting of the oecd committee for scientific and technological policy at ministerial level, 29-30 january 2004—final communique. https://www.oecd.org/sti/sciencetechnologyandinnovationforthe21stcenturymeetingoftheoecdcommitteeforscientificandtechnologicalpolicyatministeriallevel29-30january2004-finalcommunique.htm   pirolli, p. (2016). information foraging. in l. liu & m. t. özsu (eds.), encyclopedia of database systems. springer. https://doi.org/10.1007/978-1-4899-7993-3_205-2   radio, e., rios, f., oliver, j. c., hickson, b., & wallace, n. (2017). manifestations of metadata structures in research datasets and their ontic implications. journal of library metadata, 17(3-4), 161-182. https://doi.org/10.1080/19386389.2018.1439278   robertson, r. j. (2005). metadata quality: implications for library and information science professionals. library review, 54(5), 295-300. https://doi.org/10.1108/00242530510600543   salo, d. (2008). innkeeper at the roach motel. library trends, 57(2), 98-123. https://doi.org/10.1353/lib.0.0031   schonfeld, r. c. (2014). does discovery still happen in the library? roles and strategies for a shifting reality. ithaka s+r briefing. https://sr.ithaka.org/wp-content/uploads/2014/10/sr_briefing_discovery_20140924_0.pdf shearer, k., & kingsley, d. (2020, november 24). input to “data repository selection: criteria that matter.” confederation of open access repositories (coar) news-updates. https://www.coar-repositories.org/news-updates/input-to-data-repository-selection-criteria-that-matter/   tillman, r. k. (2017). where are we now? survey on rates of faculty self-deposit in institutional repositories. journal of librarianship and scholarly communication, 5(1), article ep2203. https://doi.org/10.7710/2162-3309.2203   wilkinson, m. d., dumontier, m., aalbersberg, ij. j., appleton, g., axton, m., baak, a., blomberg, n., boiten, j.-w., bonino da silva santos, l., bourne, p. e., bouwman, j., brookes, a. j., clark, t., crosas, m., dillo, i., dumon, o., edmunds, s., evelo, c. t., finkers, r., … mons, b. (2016). the fair guiding principles for scientific data management and stewardship. scientific data, 3, article 160018. https://doi.org/10.1038/sdata.2016.18     appendix   code category label/description 1.0 current status of ir is directory info up to date? 1.1 yes -ir found if opendoar url (especially from moulaison sandy & dykas list) is broken/outdated; check opendoar for updated url 1.2 no ir found use this code only if search/browse yields no ir       2.0 deposit form where user inputs metadata 2.1 y   2.2 n   2.3 dk / can't determine likely password protected/controlled         3.0   guidelines doc text (separate from deposit form) describing self-deposit process and needed metadata 3.1 y   3.2 n   3.3 y, but does not specifically describe metadata fields 3.4 dk / can't determine don't know / password protected           4.0   deposit form or guidelines mention data specifically mentions dataset (as standalone item, or as supplement to article, book chapter, working paper, etc.) 4.1 y   4.2 n   4.3 n, but data deposit noted elsewhere   4.4 dk / can't determine no sample "data" record       5.0 sample record use of sample output to determine metadata fields 5.1 still nothing nothing in form or guidelines and record provides inadequate additional detail 5.2 at least something nothing in form or guidelines yet record provides a modicum additional detail 5.3 adds nothing additional metadata adequately described elsewhere, sample output added nothing new 5.4 adds more some metadata described elsewhere, sample output offers more detail 5.5 muddies metadata adequately described elsewhere, but sample output muddled/contradicted guidelines           6.0     dc elements explicit dublin core https://www.dublincore.org/specifications/dublin-core/dces/ 6.1 title   6.2 creator   6.3 subject   6.4 description   6.5 publisher   6.6 contributor   6.7 date   6.8 type   6.9 format   6.10 identifier (doi, orcid)   6.11 source   6.12 language   6.13 relation   6.14 coverage   6.15 rights   6.16 no apparent dc element   6.17 unknown can't see form, no documentation       7.0 ddi mentioned is ddi specifically noted for data description? 7.1 y   7.2 n         8.0 controlled vocabulary is controlled vocabulary used for some descriptors? 8.1 y   8.2 n   8.3 unknown           9.0 institution student enrollment   9.1 <5000 small 9.2 5,000-15,000 medium 9.3 >15,000 large         10.0   metadata mediated are descriptors vetted, corrected before ir record is published? 10.1 no totally self-deposit 10.2 sort of self-deposit with administrator review 10.3 totally the entire deposit process is handled by librarians or other ir staff 10.4 dk can't tell / unknown     evidence based library and information practice evidence based library and information practice 2011, 6.4 134 evidence based library and information practice evidence summary the library of congress, dewey decimal, and universal decimal classification systems are incomplete and unsystematic a review of: zins, c., & santos, p. l. v. a. c. (2011). mapping the knowledge covered by library classification systems. journal of the american society for information science and technology, 62(5), 877-901. doi:10.1002/asi.21481 reviewed by: cari merkley associate professor library, mount royal university calgary, alberta, canada email: cmerkley@mtroyal.ca received: 02 sept. 2011 accepted: 31 oct. 2011 2011 merkley. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to determine the extent to which knowledge is currently addressed by the library of congress (lcc), dewey decimal (ddc), and universal decimal (udc) classification systems. design – comparative analysis of the lcc, ddc, and udc systems using zin’s 10 pillars of knowledge. setting – the faculty of philosophy and science at a brazilian university. subjects – forty one subject-related classes and 386 subclasses from the first two levels of the lcc, ddc, and udc systems. methods – to evaluate the lcc, ddc, and udc systems, the researchers employed the 10 pillars of knowledge, a “hierarchical knowledge tree” developed by the lead author of this study (p. 878). according to the authors, the 10 pillars of knowledge seek to illustrate relationships between fields of knowledge while capturing their breadth. the first level of the pillars consists of the following categories: knowledge, supernatural, matter and energy, space and earth, nonhuman organizations, body and mind, society, thought and art, technology, and history. each of the 10 pillars is further subdivided, resulting in a four level hierarchical structure of 76 categories. of the 76 categories, 55 are unique subject areas. a selection of subject-based classes and mailto:cmerkley@mtroyal.ca� evidence based library and information practice 2011, 6.4 135 subclasses from the first two levels of the lcc, ddc, and udc systems were then mapped to the relevant subclasses within the pillars. analysis was limited to the first two levels of lcc, ddc, and udc, except for the lcc categories of bf and bl where further subclasses were analyzed. classes or subclasses in lcc, ddc, or udc that were not subject based (for example, those based on publication type) were excluded from the study. in total, 41 main classes and 386 subclasses from llc, ddc, and udc were categorized using the 10 pillars. main results – the llc, ddc, and udc systems were deemed to be complete and systematic in their coverage of only three of the 10 pillars: matter and energy, thought and art, and history. this means that there was at least one class or subclass in each of the three systems that corresponded to the subclasses in these pillars. the remaining seven pillars were only partially covered by the three systems to varying degrees. for example, the coverage of religion in lcc and ddc show evidence of a bias towards christianity and incomplete coverage of other faiths. in addition to the lack of completeness in terms of subject coverage, the researchers found inconsistencies and problems with how relationships between subjects were illustrated by the systems. for example, botany should be a subclass of biology, but the subjects occupy the same level in the lcc, ddc, and udc systems. researchers also noted cases where subclasses on the same level were not mutually exclusive e.g., the br (christianity) and bs (the bible) subclasses in lcc. overall, llc performed slightly better than ddc or udc, covering 47 of the 55 unique subject categories in the 10 pillars. it was followed by udc with 44 out of 55, and ddc with 43 out of 55. some of the 55 unique subject categories in the 10 pillars system were not represented by any of the systems: 3 subclasses under society (society at large – area based, social groups – age, and social groups – ethnicity), 2 under technology (technologies – materials and technologies – processes), and 1 under foundations (methodology). conclusion – the researchers conclude that none of the three major classification systems analyzed provides complete and systematic coverage of the world of knowledge, and call for the library community to move to new systems, such as the 10 pillars of knowledge. commentary those of us who work with lcc, ddc, or udc on a daily basis can certainly speak to some of the inconsistencies or limitations of these systems. zins and santos’ work provides a spotlight on many of the weaknesses of these systems individually and collectively. in fact, the strength of the study is that it allows for comparison between the systems. however, zins and santos clearly anticipate criticism of the methodology they employed. in the paper, they acknowledge concerns expressed by some scholars around the fact that they limited their analysis to the top two levels of lcc, ddc, and udc, when a deeper analysis of further levels would have revealed greater coverage of the 10 pillars subclasses. zins and santos respond, “we decided to focus on the first two levels of the hierarchical structures because these levels embody the essence of the classification system” (p. 896). that said, it should be noted that the researchers used the third and fourth levels of the evaluation tool, the 10 pillars, when making judgments about the completeness of the lcc, ddc, and udc systems. the comparisons being made are further undermined by the researchers’ occasional deviation from their stated focus on the first two levels of the systems under review. in several instances, the third level of lcc subclasses were analyzed (e.g., bf and bl), without clear justification for why lower levels were included in the study in these circumstances and not others. the 10 pillars of knowledge system itself provides potential researchers with an area of future study, as it does not appear to have yet been the focus of review by other scholars. the reference to the primary author’s own work in evidence based library and information practice 2011, 6.4 136 lieu of a robust literature review also isolates the research being presented from pre-existing debates in the literature. zins and santos lead us to question the systems currently employed in many libraries, but further debate and discussion is necessary before it can be definitively said that the 10 pillars of knowledge is a worthy successor. / evidence based library and information practice microsoft word es 1486_haddow_final evidence based library and information practice 2008, 3:2 52 evidence based library and information practice evidence summary self-archiving to institutional repositories is improved by assisted and mandated deposit; disciplinary culture is not a factor a review of: xia, jingfeng. “assessment of self-archiving in institutional repositories: across disciplines.” the journal of academic librarianship 33.6 (dec. 2007): 647-54. reviewed by: gaby haddow faculty librarian, humanities, curtin university library perth, western australia email: g.haddow@curtin.edu.au received: 01 march 2008 accepted: 25 april 2008 © 2008 haddow. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – to test the assumption that authors familiar with subject-based repositories are more likely to self-archive to institutional repositories. design – comparative content analysis. setting – institutional repositories (irs) from the following seven universities: queensland university of technology (qut), university of melbourne, university of queensland, lund university, university of glasgow, university of southampton, and university of strathclyde. the irs included in the study were selected on the basis of repository size and use of eprints software. faculty size data and ir deposit policies were drawn from universities’ web sites. methods – each ir was searched to determine the number of deposits in the disciplines of chemistry, physics, economics and sociology. physics and economics were selected because these disciplines have established internationally renowned subject-based repositories, in contrast to chemistry and sociology, which have not. deposits from the disciplines were identified from subject terms, keywords and departmental names in metadata records. a “deposit rate” for the four disciplines in each ir was calculated. the metadata records were examined for name of the depositor, date of deposit, full-text availability, item type, and format. information in the field “deposited by” was 55 used to identify the extent of self-archiving (that is, deposited by the author). faculty size for the four disciplines at the seven universities was established from departmental web site information. for the purposes of making comparisons between the irs, these data were converted into “rates of faculty” size by dividing the number of faculty in the department by the total number of faculty at the institution. a weighted rate of deposits by discipline was calculated by dividing the rate of faculty size by the deposit rates. to take into account disciplinary differences in publication productivity, these rates were subjected to further analysis. using an “average publications per year” calculation for each discipline (from a 1977 paper), a final weighted rate of depositing was calculated for the four disciplines in the seven irs. main results – without weighting for faculty size, deposit rates vary greatly between disciplines. in most institutions, deposit rates for chemistry and sociology were higher than rates for physics and economics. when faculty size is controlled for, the highest deposit rates in five irs were for chemistry and sociology. only two irs were found to have the highest deposit rates for physics and economics. these results did not change overall when the weighting for publishing productivity was applied: the same five irs had highest deposit rates for chemistry and sociology. exceptions to these findings were the irs at university of melbourne and university of queensland, where the highest deposit rates were for economics and physics. on examination of depositor information, it was found that only 2.3% of economics deposits in the melbourne ir were self-archived. administrative assistants and other staff were responsible for depositing 97.7% of the ir’s economics holdings. self-archiving of physics items to the melbourne ir was 90%; however, these deposits comprised student theses and dissertations only. self-archiving practices were examined for: chemistry, physics and economics deposits at the university of melbourne; chemistry and economics at the university of queensland; and chemistry, physics and sociology at queensland university of technology (the only ir in the sample with a mandatory deposit policy). like melbourne, self-archiving of economics deposits at the university of queensland was also low, at 17%. of the remaining economics deposits, a librarian was responsible for depositing 68%. chemistry deposits at both melbourne and queensland had much higher self-archiving rates, 76.2% and 100% respectively, than those found for physics and economics. at qut, where deposit into the ir is mandatory, self-archiving rates are high for the three disciplines for which findings are reported. the self-archiving rate for chemistry was 68.3%, sociology 46.3%, and physics 42.9%. a librarian was responsible for the majority of the remaining deposits. conclusion – this research tested the proposition that disciplines familiar with subject-based open access repositories, such as physics and economics, are more likely to contribute to irs. its findings did not support this view. instead, the study found no particular pattern of deposit rate across the four disciplines of chemistry, physics, economics and sociology in the seven irs. operational aspects of irs, such as assisted and mandated deposit, appear to have a more significant effect on deposit rates. assisted deposit, either through departmental administrative staff or 56 librarians, accounted for relatively high deposit rates for economics in the queensland and melbourne irs. deposit date information in the queensland ir suggests administrative staff of the economics department deposit to the ir on an ongoing basis. students showed a high rate of self-archiving for theses and dissertations. it might be speculated that a mandate policy at queensland university of technology is responsible for the high self-archiving rates seen for economics, chemistry and sociology. however, librarians have assisted in the process, depositing over half the items for physics and sociology. the author recognises the value of both assisted and mandated deposit, but raises questions about how this will affect faculty use of irs. for example, in cases where faculty have no role in contributing to an ir and therefore no familiarity with it, will they in fact use it? another important consideration is the policy approach taken to temporary faculty and a mobile academic workforce. in conclusion, the author states, “institutional repositories need a mandate policy to ensure success”. commentary increasingly, research is indicating that placing a publication in an open access environment will increase its impact in the scholarly community. for this reason, it is important for those involved in developing open access systems to have as much information as possible to ensure success in their endeavours. xia’s research contributes to the growing number of studies with this aim. see, for example, a recent paper by davis and connelly (2007), summarised in this journal. xia’s sample for analysis was limited to seven irs using the same software, eprints, which includes a field for depositor name. this meant the researcher was able to gather data about archiving practices without the necessity of conducting a survey of academics at the institutions. in terms of reliability of raw data, it is difficult to imagine a better method for gathering quantitative information about archiving practices. by checking this data against departmental web site information, the study produced useful findings about those responsible for depositing to irs. however, the questionable reliability of web site information relating to faculty and discipline areas, acknowledged by xia, affects the confidence with which we can view the findings for deposit rates by discipline. exacerbating this concern is the lack of information about how deposit rates were calculated. it might be assumed the numbers presented are a percentage of the total holdings of the ir, but the reader is faced with a three-digit decimal with no raw data to provide context for the results. furthermore, the calculation for final weighted deposit rates for each discipline relies on publications productivity estimates published in a 1977 paper. ultimately, these deficiencies in the study, as it is reported, do not diminish the most interesting aspect of the study, which is the results relating to archiving practices. the snapshot of archiving practices at queensland, melbourne and qut provides ir managers with important information with which to develop policy. in essence, self-archiving to irs is not widely practised by academics unless mandated, as at qut. even with mandated deposit, a large proportion of items were deposited by people other than the author. no explanation is given in the paper as to why an analysis of strathclyde’s self-archiving practices was excluded, despite very high deposit rates for chemistry in the ir. 57 with some clarification from the researcher, this study could be expanded to explore deposit rates in a larger sample of irs and discipline groups. it is an area of increasing interest as institutions establish and develop open access systems to promote research undertaken by their staff and students. while quantitative studies of self-archiving practices raise some difficulties due to the depositor name data required, reliable findings about effective strategies to encourage archiving to irs (other than mandated deposit) would be most welcome. and if mandated deposit policies are widely implemented, research will be needed to identify effective processes to monitor policy and appropriate methods to impose in cases of non-compliance. as this is one of three papers by xia stemming from research on self-archiving, details of the companion papers are listed below for interested readers. works cited davis, phillip m., and matthew j.l. connolly. “institutional repositories: evaluating the reasons for non-use of cornell university’s installation of dspace.” d-lib magazine 13.3/4 (2007). 2 may 2008 <http://www.dlib.org/dlib/march07/ davis/03davis.html>. xia, jingfeng, and li sun. “factors to assess self-archiving in institutional repositories.” serials review 33.2 (june 2007): 73-80. xia, jingfeng, and li sun. “assessment of self-archiving in institutional repositories: depositorship and full-text availability.” serials review 33.1 (march 2007): 14-21. evidence based library and information practice evidence based library and information practice 2011, 6.4 15 evidence based library and information practice commentary from solving puzzles to designing solutions: integrating design thinking into evidence based practice zaana howard phd candidate, faculty of design swinburne university of technology melbourne, australia email: zhoward@swin.edu.au kate davis lecturer, information studies group, information systems discipline, faculty of science and technology queensland university of technology brisbane, australia email: k3.davis@qut.edu.au received: 31 oct. 2011 accepted: 03 nov. 2011 2011 howard and davis. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. introduction with the rapid evolution of technology, coupled with changing customer needs and expectations, businesses, communities and even society as a whole, are facing increasingly complex challenges. many of these challenges did not exist five – or even two – years ago. libraries, with their reliance on technology and focus on customer service, are by no means immune to these complex challenges. retrofitting existing solutions not designed to handle such complexity is no longer sufficient. the design literature calls these “wicked problems” (rittel & webber, 1973) which require new solutions be designed to suit their “wicked” nature. for example, to date the purchase and circulation models for ebooks has failed significantly as libraries and ebook vendors have focused on traditional circulation models and attempted to retrofit these models to materials that are, by their very nature, different from print collections. as a result, the features of electronic books – such as, for example, the ability of numerous patrons to borrow them concurrently – have not been exploited. as libraries of all types grapple with increasing proportions of their collections moving to electronic format, a new approach to the complex problem of ebook management is needed. this requires an agile, flexible and mailto:zhoward@swin.edu.au� mailto:k3.davis@qut.edu.au� evidence based library and information practice 2011, 6.4 16 human centred approach to create sustainable and scalable solutions adaptable to the rapid pace of change. evidence based practice (ebp) has been touted as one method for problem solving, particularly within health care and libraries. it focuses on using evidence for decision‐ making, and locates the evidence base within literature (eldredge, 2004). while literature can provide grounding, it is not sufficient to give insights or determine solutions to wicked problems where radical innovation and agility is required. ebp focuses on solving “tame” problems ‐ problems that can be analyzed and understood in order to devise an appropriate solution, such as solving puzzles or algebra (stompff, 2010). in relation to libraries, an example of a tame problem would be the introduction of a new physical format such as dvds. the problem could be understood and analysed within the library context and an appropriate solution devised. this new format still fit within the existing system; however, it required some thoughtfulness regarding policies and processes. it was akin to puzzle solving rather than problem solving. in contrast to ebp with its focus on tame problems, design thinking focuses on solving complex and wicked problems, where there is often little or no precedence. design thinking, in its simplest form, is an approach for solving business problems, similar to the way designers approach design problems (dunne & martin, 2006). it is a human centred rather than literature focused approach that is collaborative and participatory by nature. this enables it to be more agile and produce more innovative, scalable, and future focused results than ebp. incorporating design thinking principles and tools into ebp has the potential to move its applicability beyond tame problems and continuous improvement, toward wicked problem solving and innovation. this paper proposes a hybrid approach to maximise the strengths of the two methods for designing solutions to wicked problems. what are wicked problems? the term “wicked problems” was coined by rittel & webber (1973) in reference to the complexity of problems in social planning. over time, it has been adapted to fit within a wider perspective to more generally address problems that are both ill‐structured and ill‐ defined. wicked problems are a class of complex social system problems, which are “ill formulated where the information is confusing, where there are many clients and decision makers with conflicting values, and where the ramifications in the whole system are thoroughly confusing” (churchman, 1967, 141). they are unique, with complex interrelationships and interdependencies because the clients, stakeholders and context are different for each problem and subject to continuous change. if we consider wicked problems within a library context, we start to see each book or resource as part of a much more complex and macro system – one resource as part of one collection within a library that is situated within a community, which in turn is situated within a city, and so on. each level of the system has varying interrelationships and interdependencies between objects, spaces, environments, and people. the notion of wicked problems acknowledges the unique context of the problem and takes a holistic view to develop a customized solution (rittel & webber, 1973). where problems are wicked, new solutions appropriate for the context need to be developed. ebp is insufficient to allow for designing appropriate solutions. ebp is akin to solving a puzzle, with the assumption that there is just one solution, the pieces of which can be found in the current narrowly defined literature‐focused “evidence base”. this evidence base needs to be diversified in order to design new solutions for the wicked problems faced by libraries (partridge, 2011; koufogiannakis 2011; pan and howard, 2009). this represents a shift from finding a solution to designing solutions. within this concept, evidence based library and information practice 2011, 6.4 17 design thinking brings a unique way of looking at problems and designing solutions (stompff, 2010). design thinking for problem solving design thinking is an approach for problem framing and solving which can be applied to tangible products as well as intangible services and systems (buchanan, 1992; kimbell, 2009). it denotes a collaborative and human centred problem solving approach using a design mindset to solve wicked problems. a design mindset refers to the perspectives and mental processes designers move through during the act of designing (cross, 2006). brown’s (2008; brown & wyatt, 2010) design thinking process consists of three stages: inspiration, ideation, and implementation (see figure 1). it is a non‐linear approach so, while there are definite stages, there is no predetermined manner in which these should be navigated. instead, brown (2008) suggests considering the phases as a system of spaces to move through, back, and between for iterative purposes throughout the process. rather than steps, these spaces demarcate related activities. for brown (2008), inspiration considers the context and circumstances that motivate the search for solutions. this involves spending considerable time on problem finding, leading to the iterative design of a solution that takes into account the complex systems in which the problem resides. ideation is the process of generating, developing, and testing ideas that may lead to new solutions; and implementation is the development of the product to market. this approach provides a valuable framework for considering design from problem acknowledgement and definition through to final problem resolution. over the past decade design thinking has been increasingly used as an innovation approach across a number of industries including business and management (brown, 2008; dunne & martin, 2006), healthcare (brown, 2008; duncan & breslin, 2006), library and community services (bell, 2008) and social innovation (brown & wyatt, 2010). design thinking has evolved to be considered a purposeful human centred design approach for innovation and change making for individual, community, and societal benefit, which can be applied within any number of contexts. due to its social and participatory nature, it has great potential for agility and to produce radical innovation rather than incremental improvement as per ebp. comparing design thinking and evidence based practice both ebp and design thinking can be imagined in comparative stages (see table 1). there are similarities in the purpose of the stages, however differences lie in the approach and activities of each. for example, within the first phase, both methods emphasize the importance of asking or addressing the right question. the manner in which this is achieved differs remarkably between the two. for ebp the problem is first constructed into a question followed by searching for the “best available evidence”, which may be published or unpublished literature or other “authoritative resources” (eldredge, 2000, 291). the evidence is then appraised for relevance to the problem. for design thinking, the inspiration phase draws evidence from both primary and secondary sources and is grounded in the contextual local data. this involves observing figure 1 design thinking process (adapted from brown, 2008). evidence based library and information practice 2011, 6.4 18 customers, interviewing and conducting workshops with stakeholders, understanding the organization’s vision, locating the problem within industry trends, and may include consulting the literature. the focus is on understanding human behavior, needs, and values. all of this data is then synthesized in order to understand the problem from a holistic perspective and construct a question. in this way, design thinking is heavily human centred, and uses participatory methods to gather and understand data. where ebp determines a solution based on the appraisal of the evidence, implements it, and then evaluates the result, design thinking uses the results of the inspiration phase to brainstorm and ideate multiple futures and solutions. a selection of these are prototyped and tested with customers and stakeholders for evaluation and feedback. solutions are then iterated and refined as needed prior to implementation. this ensures the solution is viable, feasible, caters to the required human needs and ultimately resolves the problem. an understanding of the two methods determines that ebp focuses heavily on deductive thinking for decision making, whereas design thinking uses inductive, deductive and abductive, “leap of faith” (martin, 2009) thinking to generate solutions. design thinking is a holistic, whole‐brained approach that seeks to understand the many interrelationships as well as the qualitative and quantitative aspects involved, whereas ebp takes a dominantly logical, analytical, left brained approach to decision making. a hybrid model this paper proposes a hybrid model (figure 2) integrating the most valuable aspects of both ebp and design thinking to provide an agile and rigorous approach for wicked problem solving. there are six stages in this hybrid model: define the problem, research, prototype and test, implement, evaluate, and storytelling. as with design thinking, the phases of the model are not intended to be linear. in particular, the phases of problem definition, research, and prototyping and testing are spaces to be moved in and out of as needed. define the problem defining the problem in this model focuses on curiosity and asking “why” questions such as, “why is it the way it is currently? why is this table 1 phases of evidence based practice and design thinking evidence based practice (booth, 2004) design thinking (brown, 2008) define the problem inspiration (problem finding, research, insights) find evidence appraise the evidence apply results of appraisal ideation (brainstorming, prototyping, testing) evaluate change implementation (execution, evaluation, evolution) redefine the problem figure 2 hybrid ebp and design thinking model evidence based library and information practice 2011, 6.4 19 a problem? why does it matter? what are the business constraints?” the answers may be found through documents, reports, conversations, workshops, or experiences. through curiosity, developing an understanding of the current context enables the crafting of a question that defines the problem accurately. research research acknowledges the need to collect and consider primary and secondary research. in this phase, design thinking and ebp converge to bring together information from and about the past through the literature of relevant disciplines as well as personal stories; the present through observation, interviews, workshops and industry trends; and the future through identification of aspirations. appraisal is absorbed into this research process. in this model, appraisal is focused on synthesis and sense making, where research is evaluated and insights synthesized. connections between data are identified and organized into meaningful and valuable findings for action. the question is then reviewed and iterated based on this research as necessary. prototype and test the inclusion of prototyping and hypothesizing allows the use of abductive thinking and creativity to identify possible solutions. it involves generating multiple solutions through structured ideation and then prototyping the solutions most likely to provide significant benefits. these prototypes are then tested directly with customers and stakeholders. the feedback from testing, as well as personal reflections, allows further understanding of the problem and allows the iteration of solutions in a meaningful way. this new information can be combined with the synthesized information from the research phase to inform further hypothesizing and prototyping. implement and evaluate implementation and evaluation take on rollout and review roles in this model. due to the significant effort in research, prototyping and testing, and the commitment to collaboration and engagement throughout the process, implementation becomes a routine, minimal risk process as the majority of the barriers and issues have already been addressed. evaluation is also routine, gathering feedback and reviewing the success of the solution after a determined period of time. this feedback can be used for further iteration or evolution of the solution. storytelling problem redefinition is replaced by storytelling, a process to close the loop and contribute to the evidence base. when solutions have been implemented and evaluated, it is important to tell the story – informally and formally. this adds to the evidence base that can be drawn from in the research phase of future ebp processes. design thinking acknowledges that every problem is unique, so the purpose of feeding back into the evidence base is not presenting a solution that can be picked up and implemented elsewhere. it is about documenting the process, the inputs, and the learnings. a hybrid approach maximizes the strengths of the two methods for designing solutions to wicked problems. while design thinking brings new methods and tools to ebp, the primary benefit of merging the two approaches is the new mindset that design thinking brings to ebp. this mindset focuses on human centredness rather than literature; it redefines what we might consider to be “evidence”, and involves collaboration and engagement of customers and stakeholders throughout the whole process to ensure human needs are met. it is also future oriented, looking forward rather than looking back, and enabling a mindset of reframing problems to support solving those with little or no precedence. evidence based library and information practice 2011, 6.4 20 conclusion there are significant benefits to be gained from adopting a human centred rather than literature focused foundation for ebp. the proposed hybrid approach of integrating ebp with the mindset, tools, and methods of design thinking is one possible model that could move ebp forward. the model is exploratory, with the recommendation that it be tested. it brings together the most rigorous aspects of ebp and the human centredness of design thinking to create a model that allows for creativity and innovation while also allowing for solutions grounded in evidence. it has the potential to move ebp’s applicability beyond tame problems and continuous improvement, toward wicked problem solving and innovation. the authors welcome approaches from organizations that would be willing to test the hybrid model in practice. references booth, a. (2004). formulating answerable questions. in a. booth & a. brice (eds.), evidence based practice for information professionals: a handbook (pp. 61‐70). london: facet publishing. bell, s. j. (2008). design thinking. american libraries, 39(1/2), 44‐49. brown, t. (2008). design thinking. harvard business review, 86(6), 84‐92. brown, t. & wyatt, j. (2010). design thinking for social innovation. stanford social innovation review, winter, 30‐35. buchanan, r. (1992). wicked problems in design thinking. design issues, 8(2), 5‐ 21. churchman, c. w. (1967). wicked problems. management science, 14(4), b141‐b142. coyne, r. (2005). wicked problems revisited. design studies, 26(1), 5‐17. cross, n. (2006). designerly ways of knowing. london: springer. duncan, a. k., & breslin, m. a. (2009). innovating health care delivery: the design of health services. journal of business strategy, 30(2/3), 13‐20. dunne, d. & martin, r. (2006). design thinking and how it will change management education: an interview and discussion. academy of management learning & education, 5(4), 512‐523. eldredge, j. (2000). evidence based librarianship: an overview. bull medical library association, 88(4), 289‐ 302. eldredge j. (2004). how good is the evidence base? in a. booth & a. brice (eds.) evidence based practice for information professionals: a handbook (pp. 36‐48). london: facet publishing. kimbell, l. (2009). beyond design thinking: design‐as‐practice and designs‐in‐ practice. centre for research on sociocultural change (cresc) annual conference. manchester: university of manchester. koufogiannakis, d. (2011, june). considering the place of practice‐based evidence within eblip. presentation delivered at eblip6: valuing knowledge and expertise, salford, england. martin, r.l. (2009). the design of business: why design thinking is the next competitive advantage. boston, mass.: harvard business press. pan, d. & howard, z. (2009). reorganizing a technical services division using collaborative evidence based information practice at auraria evidence based library and information practice 2011, 6.4 21 library, evidence based library and information practice, 4(4), 88‐94. partridge, h. (2011). moving forward: defining and conceptualizing evidence based practice for the library and information profession. presentation delivered at eblip6: valuing knowledge and expertise, salford, england. rittel, h. & webber, m. (1973). dilemmas in a general theory of planning. policy sciences, 4, 155‐169. stompff, g. (2010). don’t call me wicked! klatergoud. accessed 31 may 2011: http://www.klatergoud.com/2010/12/d on%e2%80%99t‐you‐call‐me‐wicked/ / evidence based library and information practice evidence based library and information practice evidence based library and information practice 2010, 5.4 87 evidence based library and information practice evidence summary academics are reading more electronic journal articles in more subjects, using varying strategies to find and manage them a review of: ollé, c., & borrego, á. (2010). a qualitative study of the impact of electronic journals on scholarly information behavior. library & information science research, 32(3), 221-228. reviewed by: christina e. carter associate professor consortium library, university of alaska anchorage anchorage, alaska, united states email: trina@uaa.alaska.edu received: 17 sept. 2010 accepted: 26 oct. 2010 2010 carter. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to assess how the increase in number of electronic journals available to academic scholars has changed their information-seeking or consulting behaviour, with respect to 1) the amount and diversity of sources they read; 2) strategies they use to keep up-to-date in their fields; 3) use of personalized information services; 4) determining the value and relevance of articles; and 5) personal management of scientific information. this study is a followup to an earlier quantitative study (borrego, anglada, barrios, & comellas, 2007) in the same setting. design – qualitative, using an open-ended questionnaire, followed by personal interviews of a small group of the respondents. setting – universities that are members of the consortium of academic libraries of catalonia (cbuc), which is made up of the eight public catalan universities and the national library of catalonia, spain. subjects – one hundred thirty-seven scholars from the member universities of diverse ages and disciplines. eleven of these academics were selected for personal interviews. methods – the authors used a two-staged approach to gather comments from researchers on their use of electronic journals. first, an open-ended, self-administered questionnaire (with some pre-testing done) mailto:trina@uaa.alaska.edu� evidence based library and information practice 2010, 5.4 88 was sent by e-mail to some 490 academics who had provided e-mail addresses in the quantitative study; 137 were returned and analyzed. personal interviews were then conducted with 11 scholars who had given the most detailed answers in the questionnaire. their ages ranged from 28 to 67; each was from a different discipline, and from six of the universities. informed consent sheets (describing the study and guaranteeing anonymity) were given to the 11 interviewees. personal interviews were conducted in the subjects’ offices by one of the authors, and lasted between 45-60 minutes. in the interview stage, the authors wanted to examine: use and assessment of the library, access to electronic information, and impact of e-resources on information behaviour. subjects were also asked to provide three suggestions on improving access to scientific information. main results – the amount of reading and number of electronic journals and other sources consulted among the scholars who participated in this study has increased. threequarters of survey respondents consult more journals and read more articles. the scholars reported that they are grateful for the increase in electronic information and its enhanced ease of access, and are not overwhelmed by it. their reading has become more discriminatory, though, with many reporting “skimming” much of what they read to save time. scholars keep up-to-date in three main ways: web browsing of journal issues, library database searches, and toc e-mail alerts. more than 90% of survey respondents reported conducting database searches. google and google scholar were often mentioned ahead of specific library database names. in determining value and relevance of an article, its author and abstract are key for scholars. in addition, personal information management techniques used by scholars were all over the board. the three main methods were use of print or electronic folders, reference management software, and no system. many of the academics felt their information management systems were “rudimentary” (p. 225). the request for suggestions and comments on the questionnaire was not answered by “most of the sample” (p. 226). those who did respond to this request asked for more library resources. the main complaint expressed by scholars concerned the difficulty and complexity of finding journal article content using the library website (e.g., varying databases, difficulty of interpreting what journal electronic and print holdings are available). because of this, a number of scholars used google to find librarysubscribed content. conclusion – by having greater and easier access to e-journals, scholars accessing the cbuc read more articles from more disciplines. scholars would prefer a simpler library interface to search for online content. due to the complexity of finding article content, they use web search tools like google and google scholar to get to what they need faster. the authors of this study believe research should be conducted on the use of the consortium’s metasearch tool to reduce the complexity. research should also be conducted on value-added features of search interfaces for particular disciplines. commentary as the authors note, in qualitative research, a representative sample is more important than a large response rate. the relatively low return rate of 28% of the 490 scholars who actually received the survey is mitigated by the fact that the disciplines and ages of the academics appear to reflect the population at large. the exact time frame for the end of collection of surveys is not stated; collection of questionnaires appeared to last a month and a half (mid november through december). neither the self-administered questionnaire nor the interview questions are explicitly included in the article; even providing a link to the survey and interview questions or responses would have been helpful. individual responses are important to study because of the effort involved in writing them, evidence based library and information practice 2010, 5.4 89 as well as the richness and context they provide, but it is difficult to glean if there may have been additional insights provided by the scholars if these responses are not summarized or included in some manner. the respondents’ ages and disciplines are represented in tables, but the responses are not similarly represented for the reader. further, there is sometimes confusion in the article about where responses were captured: the questionnaires or the interviews? the authors state they wanted to find out how a scholar’s age might be related to his/her information-seeking behaviour. however, they do not address this in the discussion or conclusion. the authors also include several non-sequitur statements that have a tendency to detract from their main points. for instance, they state in their problem statement that “. . . the question of whether sociocognitive factors and technical barriers may be affecting the adoption of electronic resources at catalan academic libraries has not been addressed” (p. 221). the authors never return to this issue. the research of tenopir and her colleagues (tenopir, king, spencer, & wu, 2009) has further indicated that issues such as purpose, work assignment, and productivity of the scholar are also related to reading patterns. the authors state that the results of their survey bear on library allocation of resources, in light of increased spending on e-resources and tighter academic library budgets. the results do not really elucidate, though, where money might specifically be better allocated (e.g., more in a specific discipline) for the scholars at the catalan universities, nor are there stated plans by the authors to study this in more depth. from this reviewer’s perspective, one of the most interesting results from this study is the range of personal strategies that academics reported using in an attempt to manage journal content/references they read and collect. some scholars had faulty beliefs about the capabilities of bibliographic software. this could bear on library education efforts. according to the authors, the cbuc provides a consortial license to refworks, although there is no explicit mention of training on it provided by library staff. in this spanish university setting, could there be cultural aspects to faculty adoption of reference management software? or, are there commonalities in the use (or not) of such software by faculty, regardless of the national/cultural setting? at any rate, studying information management and organization strategies is an area that needs further investigation. much will also likely need to be studied at levels of learning and cognition, and personal traits of scholars (doong & wang, 2009). references borrego, a., anglada, l., barrios, m., & comellas, n. (2007). use and users of electronic journals at catalan universities: the results of a survey. journal of academic librarianship, 33(1), 67-75. doong, h., & wang, h. (2009). predictors of diverse usage behavior towards personal knowledge management systems. online information review, 33(2), 316-328. tenopir, c., king, d. w., spencer, j., & wu, l. (2009). variations in article seeking and reading patterns of academics: what makes a difference? library & information science research, 31(3), 139148. / evidence based library and information practice evidence summary evidence summary   insufficient understanding of user benefits impedes open data initiatives at museums   a review of: booth, p., navarrete, t., & ogundipe, a. (2022). museum open data ecosystems: a comparative study. journal of documentation 78(4), 761-779. https://doi.org/10.1108/jd-05-2021-0102   reviewed by: jordan patterson associate librarian a. p. mahoney library st. peter’s seminary london, ontario, canada email: jpatte46@uwo.ca   received: 24 may 2023                                                             accepted:  28 june 2023      2023 patterson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30372     abstract   objective – using nardi and o’day’s (1999) definition of ecosystem as “a system of people, practices, values, and technologies in a particular local environment,” to understand how art museums form their policy to interact with and respond to the various open data (od) ecosystems in which they operate.   design – multiple case study consisting of interviews and subsequent qualitative analysis, as well as document analysis.   setting – european art museum od ecosystems.   subjects – subjects included 7 management staff members at 3 separate mid-size, art-based museums located in norway, the netherlands, and spain; an unspecified number of representatives from a cultural-policy agency in each of those countries; an unspecified number of government, museum, and research documents from within each museum’s od ecosystem.   methods – the researchers identified 3 museums with od initiatives and conducted in-depth interviews with relevant staff members at each institution. the researchers also interviewed representatives from relevant national od policy-related agencies. the researchers coded their data and developed a list of five key od “ecosystem components,” which they used to analyze the 3 specific museum ecosystems under consideration.   main results – open data initiatives at cultural heritage institutions are subject to a number of internal and external pressures. museums are typically responsive to their environments, and top-down policy requirements appear to be an effective means of advancing open data initiatives. nevertheless, the value proposition of open data appears to be insufficiently understood by museum staff and other stakeholders. as a result, museums participate in od initiatives even when the benefit remains undemonstrated and the use of od—how and by whom—remains unclear.   conclusion – the needs and wants of od end-users remain ill-defined and poorly understood. as a result, museums expend resources and effort to supply od, while remaining uncertain about the return on their investment. attention to users could result in “more robust information flows between ecosystem components.”   commentary   “if you build it, they will come.” it is easy to invoke this inspirational slogan to supply an impetus for an open data project before a real use case has been established. but is it true? despite widespread adoption of od practices and policies, it appears that without a proper understanding of the end-user’s role in the od ecosystem, the benefits of od are often more theoretical than actual, more pie-in-the-sky than a real return on investment. the experiences—and the challenges—documented in this study will likely sound familiar to anyone who has invested significant time and energy in an open data project at a cultural heritage institution. in this paper, the researchers bring into focus a common problem for museums and libraries.   assessed with perryman and rathbun-grubb’s “the cat: a generic critical appraisal tool” (2014), this research meets a strong standard of validity. recognizing that their work is among the first to study museum od ecosystems qualitatively, the researchers clearly state a number of limits to their paper. in particular, they note concerns about the case study method’s adequacy vis-à-vis the complex ecosystem paradigm, the lack of methodological rigour in ecosystems analysis, and a certain arbitrariness in the definition of ecosystem. with these cautions in mind, and still perceiving the academic value of inquiry into od ecosystems, the researchers rightly frame the study as exploratory and suitable for the development of theoretical propositions, if not hard and quantified conclusions. the researchers also include a word of caution about the generalizability of the study, given that it only considers three cases sharing the same general characteristics.   nevertheless, the exploratory nature of the study should not dissuade readers from taking its findings seriously. the extensive literature review demonstrates how the work builds on established concepts, and the researchers’ discussion ties their analysis back into and confirms findings from other research. for instance, though the case studies focus on medium-sized institutions, by relating the present study to previous research, the authors make informed judgments about od activity at smaller and larger institutions, and therefore gesture toward a fuller conception of od initiatives at museums, whatever the size. this study probes the world of museum od ecosystems, and future researchers will find herein a useful model for further study of this area.   the emphatic takeaway from this study, however, is a point the researchers are at pains to repeat throughout: museums lack a clear knowledge of user od needs. the directness of this statement is an open invitation to further research. it is also an invitation to professionals at cultural heritage institutions to revise and rethink their od practices. if museums sense their data has tremendous potential, yet remain underwhelmed with its use once it is made open (i.e., their return on investment), they should actively pursue an understanding of the data’s ultimate users and demonstrate the successful reuse of their data. if museums do not take this step, they will not see the maturation of od ecosystems, and their projects will fail to justify further investment.   references   booth, p., navarrete, t., & ogundipe, a. (2022). museum open data ecosystems: a comparative study. journal of documentation 78(4), 761-779. https://doi.org/10.1108/jd-05-2021-0102   nardi, b., & o'day, v. (1999). information ecologies: using technology with heart. mit press.   perryman, c., & rathbun-grubb, s. (2014). the cat: a generic critical appraisal tool. http://www.jotform.us/cp1757/thecat   evidence based library and information practice 2012, 7.3 92 evidence based library and information practice evidence summary younger adults derive pleasure and utilitarian benefits from browsing for music information seeking in physical and digital spaces a review of: laplante, a., & downie, j. s. (2011). the utilitarian and hedonic outcomes of music informationseeking in everyday life. library & information science research, 33, 202-210. doi:10.1016/j.lisr.2010.11.002 reviewed by: diana k. wakimoto online literacy librarian, california state university, east bay doctoral student, san jose-qut gateway program hayward, california, united states of america email: diana.wakimoto@csueastbay.edu received: 18 apr. 2012 accepted: 11 june 2012 2012 wakimoto. this is an open access article distributed under the terms of the creative commons‐ attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – this study’s objective was to identify the utilitarian and hedonic features of satisfying music information seeking experiences from the perspective of younger adults when using physical and digital music information retrieval (mir) systems in their daily lives. design – in-depth, semi-structured interviews. setting – large public library in montreal, canada. subjects – 15 french-speaking younger adults, 10 males and 5 females (aged 18 to 29 years, mean age of 24 years). methods – a pre-test was completed to test the interview guide. the guide was divided into five sections asking the participants questions about their music tastes, how music fit into their daily lives, how they discovered music, what music information sources were used and how they were used, what made their experiences satisfying, and their biographical information. participants were recruited between april 1, 2006 and august 8, 2007 following maximum variation sampling for the main study. recruitment stopped when data saturation was reached and no new themes arose during analysis. interviews were recorded and the transcripts were analyzed via constant comparative method (ccm) to determine themes and patterns. mailto:diana.wakimoto@csueastbay.edu evidence based library and information practice 2012, 7.3 93 main results – the researchers found that both utilitarian and hedonic factors contributed to satisfaction with music information seeking experiences for the young adults. utilitarian factors were divided between two main categories: finding music and finding information about music. finding information about music could be further divided into three sub-categories: increasing cultural knowledge and social acceptance through increased knowledge about music, enriching the listening experience by finding information about the artist and the music, and gathering information to help with future music purchases including information that would help the participants recommend music to others. hedonic outcomes that contributed to satisfying information seeking experiences included deriving pleasure and feeling engaged while searching or browsing for music. especially satisfying experiences were those where the participants felt highly engaged in the process and found new, independent, non-mainstream music. not finding new music did not automatically lead to an unsatisfying experience for the participants; however, technology malfunctions in digital mir systems and unpleasant environments such as those with unfriendly staff in physical music spaces (libraries and stores), led to unsatisfying experiences for the participants. conclusions – as the results show that the hedonic aspects of music information seeking are very important, designers of mir systems must take into account the hedonic as well as utilitarian outcomes when creating user interfaces. mir systems should be designed with browsing as well as searching capabilities so searchers can make serendipitous discoveries of new music and information about music. in other words, mir systems need to be engaging to ensure satisfying interactions for searchers. commentary this study is well positioned in the larger body of research literature on consumer behaviour, information seeking, and music seeking. the researchers’ study extends this literature by focusing on the little studied area of music information seeking in everyday life where a person may have no defined information need. this study is well designed with a clearly explained research problem, theoretical framework, methodology, and results. the researchers presented well-reasoned conclusions connecting the study to the existing literature base as well as discussing how the study extended the understanding of how younger adults interact with mir systems. the researchers did not overgeneralize their conclusions and clearly stated the limitations of their research. they also suggested avenues for future research including studying other populations of individuals in different geographical locations and with other information needs in order to more fully understand music informationseeking in multiple contexts. this is a very well written and strong study, appraised using the critical appraisal checklist created by glynn (2006) and found to be valid. it is especially strong in relating the new findings to the theoretical framework and to relevant literature and there are very few weaknesses. the inclusion of the interview guide would have been appreciated and would allow for others to replicate the research. the reporting of the results overall is very organized and answers the research questions posed at the beginning of the article; however, there is no differentiation between the participants’ experiences using digital mir systems and physical mir systems. this differentiation would be useful for designers of mir systems who may want to use the study’s results to improve the mir systems. furthermore, while the article reported the gender of the participants, it did not comment on differences or similarities in experiences or perspectives of information seeking based on gender. future research into mir systems could use gender as a lens of analysis to determine if and how gender impacts user satisfaction with mir systems. this study extends our understanding of how younger people perceive satisfying music information-seeking experiences and reevidence based library and information practice 2012, 7.3 94 emphasizes the need to take into account hedonic factors as well as utilitarian factors when designing mir systems. for designers of mir systems, these results demonstrate the necessity of modifying user interfaces so they facilitate browsing for information and music as the researchers noted that “searching for music for everyday-life purposes are often motivated by a vague or ill-defined need” (p. 209). the study should lead to more research on the music information seeking which will further help in designing mir systems and allow librarians to better engage their users in satisfying information-seeking experiences. references glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi:10.1108/07378830610692154 classics   richard trueswell's contribution to collection evaluation and management: a review   a review of: trueswell, r. l. (1969). some behavioral patterns of library users: the 80/20 rule. wilson library bulletin, 43(5), 458-461.   reviewed by: jacob l. nash resource management librarian health sciences library and informatics center university of new mexico albuquerque, new mexico, united states of america email: jlnash@salud.unm.edu    received: 7 mar. 2016    accepted: 11 may 2016      2016 nash. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to demonstrate the relationship between library circulation and the percent of a library's holdings satisfying circulation.   design – retrospective cohort study of library circulation data.   setting – cambridge and northampton, massachusetts, united states of america.   subjects – the users of the monographic holdings of the air force cambridge research laboratory library and the forbes public library.   methods – trueswell compiled circulation data from the air force cambridge research laboratory library over a period of 5 years, ostensibly from 1964-1969. additionally, he gathered circulation data from forbes public library. trueswell compared each respective collection of data against the entire holdings of each library, after which he converted each measure to a percentage of the whole.   main result – based on the collected data, trueswell found that the percentages of both libraries’ holdings that satisfy circulation follow a power law distribution. he compared this with a previous study measuring journal circulation at a health sciences library that exhibited the same pattern. he stated that these similar distributions demonstrated the "80/20 rule." the distribution is such that any given percent of circulation will provide the percent of a library’s circulating holdings necessary to satisfy it. additionally, trueswell found that 75 percent of current circulation had circulated at least once within the preceding year.   conclusion – the findings have implications for core collection development, purchasing multiple copies of a given title, determining the optimal size of a library's collection, and weeding. trueswell also submits the idea of developing regional interlibrary loan centers for books that do not circulate often, as a cost saving measure for most libraries.   commentary   richard trueswell introduced a practice-changing methodology of evaluating library material usage and making collection management decisions based on data with the publication of “some behavioral patterns of library users: the 80/20 rule.” in doing so, he also introduced concepts derived from business inventory and return-on-investment approaches to selection.   trueswell’s was not the first paper that investigated core titles which garner the majority of use within libraries. postell (1946) studied journal check outs and requests at the louisiana state university medical library. he found that local use patterns of library material provide more accurate data for evaluating journals, supporting the conclusion proposed by estelle brodman (1944) that the prevailing method of journal evaluation promulgated by gross and gross (1927) was unscientific. for further review of these studies, see eldredge (2008).   fleming and kilgour (1964) reported on a core collection of biomedical journal titles that satisfy the majority of usage at both columbia and yale medical libraries. the authors confirmed that such a subset of journals exists at their respective libraries. trueswell repurposed part of their data in his classic article to demonstrate the principle that about 20% of the entire holdings for these libraries satisfied about 80% of the use of journals. fleming and kilgour noted that their results may generalize to other scientific libraries, but did not claim their findings constituted a rule or natural law.   trueswell’s methodology of studying book circulation and non-circulation, and the postulation of the 80/20 rule, marked the beginning of a large body of material use studies in the library literature. the famous, or infamous, “pittsburgh study” found that 40% of monographs acquired by the university library at the university of pittsburgh in 1969 did not circulate in the following 6 years (kent, 1979). this study formed the basis for the maxim that only 40% of books will circulate. this was followed up with a study by hardesty (1981) who attempted to replicate the pitt study at small liberal arts college with similar findings. another study in an academic health sciences library reported similar results (fenske, 1994).   eldredge (1998) conducted a circulation study of monographs acquired during a 5-year period at an under-resourced and “relatively ‘young’” health sciences library and found that the majority (84%) of monographs added to the collection during the period of january 1 – december 31, 1993 circulated at least once within 4 years of purchase. in a similar study, blecic (2000) found that roughly 81% of monographs added to the collection during the time between mid-august, 1994 and mid-august, 1995 circulated at least once. blecic noted that significantly more monographs were required to meet the 80% circulation mark: 38% of monographs rather than 20%, a number that closely resembles eldredge’s findings of 36%.   eldredge (1998) described a variant of the 80/20 rule (which was, and still often is, inaccurately termed “the pareto principle”) by detailing joseph juran’s “vital few and the trivial many” principle, which states, briefly, that a small group of a total population exerts disproportionately larger effect than the rest(as cited in eldredge, 1998).  based on the evidence reported by eldredge and blecic (2000), the vital few explanation seems more likely over the stricter 80/20 quantity, at least for medical libraries.   trueswell garnered attention during the 1970s, perhaps because of the 80/20 rule or the pittsburgh study, and this may have been the impetus behind studies critical of his methodology. in one slightly inflammatory paper, trueswell’s techniques were criticized as “either meaningless or fallacious” (sargent, 1979). trueswell responded to this claim in an invited    letter that his methods were often misunderstood, and attempted to show where sargent misapplied his ideas (trueswell, 1979). later, turner (1980) detailed misapprehensions of trueswell’s methodology in an attempt to point out where his techniques have value, noting that “[s]tatistical techniques are meant to be filtering systems” and “are not ends in themselves as trueswell has mentioned in much of his published work” (emphasis in original, p. 137).   the 80/20 rule seems to have taken on a life of its own, and the intention behind it got lost in the shuffle. trueswell (1969a) himself noted in his classic that “[t]he rule is sometimes expressed as the 75/25 rule with the same interpretation” (p. 458) indicating it was never meant to be an exact formula. in a study examining the proportion of library users and total circulation, trueswell (1968) noted that “the salient point of the study is the fact that a very small proportion of the active borrowers accounted for a very large segment of the circulation” (p. 493). furthermore, the literature of the 80/20 rule may suffer from varying levels of bias. many of the studies reporting on the 80/20 rule use diverse parameters for data collection, which can lead to information bias. trueswell (1969) examined the percent of circulated titles satisfying circulation, whereas eldredge (1998) and blecic (2000) used monographs acquired during a specific period, for example. fenske (1994) seemed to intuit that perhaps loose parameters would bias her sample (arguably, they did), but she did not make the appropriate adjustments to her data. eldredge (1998) likely designed his study to mirror kent’s (1979) study to discern whether those findings applied to health sciences libraries. likewise, eldredge (1998) incorporated in-house use with circulation data to show that even more books were used when other forms of “use” were tallied than just circulation. trueswell’s and kent’s methods relied solely on circulation data.   reporting bias may also exist. trueswell (1969b) noted that heavily used books would need new book cards more frequently, perhaps leading to the destruction of the full cards prior to tabulation, and many librarians are familiar with “helpful” patrons who reshelve their own materials after using them in-house, resulting in under-reporting this use.   nisonger (2008) summarized an extensive, but not comprehensive, selection of studies that reported on the 80/20 rule. in a subset of 19 articles reporting on the concentration of titles in print serials usage, nisonger (2008) showed that most approximated the 80/20 rule but noted that “the 80/20 numbers are not expected to work out precisely” (p. 65). however, averaging the ratio of the percent of titles that account for a percent of use from table 1 (pg. 66) yields a proportion of 0.316, notably higher than the 0.25 expected from the 80/20 rule. this means that roughly 25% of material would be required to satisfy 80% of usage (80×0.316 = 25.28), based on those 19 studies.   this may seem like splitting hairs, especially if the 80/20 rule is not meant to work out precisely. yet, if the rule is applied to a large collection, say the university of illinois at urbana-champaign, who recently surpassed their 13 millionth volume (“13-millionth volume acquired,” 2012), that extra 5% can add up to a very large number. even if only 40% of volumes circulated within the past five years, the unaccounted for 5% works out to 260,000 extra volumes ((13,000,000 × 0.4) × 0.05 = 260,000). determining material to weed or relocate based on the 80/20 rule for an institution this size could lead to costly, negative results. it is thus misleading to represent library material usage as always fitting an 80/20 rule, and there is no compelling reason to abide the imprecision of such a generalization. any given library can expect variation in the usage of materials, the main point being that there will be a disproportionate pattern in that usage.   despite the conflicting evidence, the 80/20 rule pattern has been observed generally in or applied to many circumstances surrounding libraries and usage of library materials, a few of which follow by way of example. in addition to trueswell’s classic, britten (1990) applied the 80/20 rule to usage amongst specific lc classes within an academic library to help determine where and how much to allocate to new purchases. another study observed that cumulative use across an academic consortium in the north eastern united states exhibited a near 80/20 trend (davis, 2002). more recently, singson and hangsing (2015) observed the same trend in usage of electronic journals across an academic consortium in india. nisonger (2008) provides many more examples.   it is worth mentioning just how tricky it is to measure use, both of print as well as electronic resources. as noted above, book cards get lost, patrons reshelve their books, or statistics from the previous year accidentally get expunged. with the advent of e-books, a one-to-one comparison between the formats becomes nearly impossible to make. each time any part of an e-book is accessed, a use is recorded, whereas each time a print book is checked out, a single use is recorded. someone who checks out a print book is likely to open the book, or “access it,” more than once during the checkout period, skewing any such usage studies, similar to a situation where a study does not account for in-house use in the final tally. while use data can be helpful in demonstrating trends, caution must be exercised in decision making and when interpreting study results.   regardless, analyzing the usage statistics to determine which resources are being used is a common tactic in library collection management (blake & schleper, 2004). depending on the mission, ensuring that the library sees a return on investment for journals and books is important due to limited financial resources. additionally, as many journals and books are bundled together, it can be instructive to examine which titles in a bundle are more heavily used than others in the same bundle. this type of analysis may help in determining whether to “break up the big deal” (blecic, wiberley, fiscella, bahnmaier-blaszczak, & lowery, 2013)   mostly absent from trueswell’s work is why such disproportionate patterns are visible in collections usage. eldredge (1998) included potential explanations by comparing student to book ratios and overall customer to book ratios between his library and libraries from other studies. in addition, he drew from other research to explain why health sciences students engaged in problem based learning might compete more heavily for library resources, which may help to explain higher use. among the libraries he studied, burell (1985) reported that lower than average borrowing rates constituted the need for a larger proportion of the collection to make the 80% figure.   current research into book use and selection is promising. investigation into e-book usage at a large university library revealed that catalogue records that include table of contents and summary notes have a much higher probability, over 80%, of being used than those without these features (harker & sassen, 2015). these results may imply usage trends are neither static, nor can they be generalized across platforms. fry (2015) provides a critical look at many claims about apparent lack of book circulation made over the past decade that have been used as the basis to shift collecting practices or claim that collection development conducted by librarians is inherently faulty. fry (2015) concludes that such claims are not made based on data, but made by “persuasive proponents and a publishing industry eager to sell e-books and control content” (p. 19).   while his classic article is perhaps most famous for the 80/20 rule, trueswell also introduced ideas about joint library facilities thatare being actualized. for example, texas a&m university system and the university of texas system have developed a shared space as a cost and space saving measure (“new texas a&m-university of texas systems joint library facility proving broadly successful,” 2014). as print continually goes out of vogue, particularly for journals in science and medical libraries, repurposing space within the building may take priority. a collaboration exists between the national library of medicine and participating medical libraries to preserve a core of print journal titles, allowing participating libraries to keep an agreed upon selection of journal runs and to discard or otherwise remove from the building the rest of their journal volumes (fishel & collins, 2011). this approach ensures preservation as well as potential ill availability for titles in this program. while trueswell may not have foreseen the almost wholesale transition from print to electronic in many library resource domains (and many libraries), his techniques may prove useful in making evidence based decisions on which volumes to keep, and which to send to the print repository or discard altogether.   demand driven acquisitions (dda) is essentially a digital version of the regional ill centers that trueswell also proposed in his classic article. trueswell’s (1969a) idea for the ill center was to house the trivial many offsite, with a “built-in procedure that any item requested from the lesser-used area will … enter the core collection. thus, the core collection would continue to reflect user-circulation requirements regardless of the source of the items” (p. 461). dda is quite similar in function, where a library adds a pool of electronic titles to its holdings and only pays for these titles as they are used by patrons. when an item from the pool triggers a purchase, it is added to the library’s permanent holdings.   while not the first to propose a systematic and mathematical approach to determine which library materials are useful, trueswell was the first to introduce business and inventory concepts to the library literature and, further, to propose a method to determine what comprises a core library collection, based on use. despite methodological limitations, richard trueswell therefore is a founding thinker and practitioner in evidence based practice in library collection development and management. trueswell’s research, for better or worse, is still being cited in 2015, almost 40 years after he introduced his ideas to the library literature.   references   13-millionth volume acquired. (2012, september 7). retrieved 21 april 21 2016, from http://www.library.illinois.edu/news/thirteenth_millionth_volume.html   blake, j. c., & schleper, s. p. (2004). from data to decisions: using surveys and statistics to make collection management decisions. library collections, acquisitions, and technical services, 28(4), 460–464. http://dx.doi.org/10.1016/j.lcats.2004.09.002   blecic, d. d. (2000). monograph use at an academic health sciences library: the first three years of shelf life. bulletin of the medical library association, 88(2), 145.   blecic, d. d., wiberley, s. e., fiscella, j. b., bahnmaier-blaszczak, s., & lowery, r. (2013). deal or no deal? evaluating big deals and their journals. college & research libraries, 74(2), 178–194. http://dx.doi.org/10.5860/crl-300   britten, w. a. (1990). a use statistic for collection management: the 80/20 rule revisited. library acquisitions: practice & theory, 14(2), 183–189.   brodman, e. (1944). choosing physiology journals. bulletin of the medical library association, 32(4), 479–483.   burrell, q. l. (1985). the 80/20 rule: library lore or statistical law? journal of documentation, 41(1), 24–39. http://dx.doi.org/10.1108/eb026772   davis, p. m. (2002). patterns in electronic journal usage: challenging the composition of geographic consortia. college & research libraries, 63(6), 484–497. http://dx.doi.org/10.5860/crl.63.6.484   eldredge, j. d. (1998). the vital few meet the trivial many: unexpected use patterns in a monographs collection. bulletin of the medical library association, 86(4), 496–503.   eldredge, j. d. (2008). predicting future information resource utilization under conditions of scarcity: the first cohort study in health sciences librarianship. evidence based library and information practice, 3(4), 82–88. http://dx.doi.org/10.18438/b8gp7n   fenske, r. e. (1994). evaluation of monograph selection in a health sciences library. bulletin of the medical library association, 82(3), 265–270.   fishel, m., & collins, m. e. (2011). nlm and nn/lm national cooperative medical lournals print retention program: medprint. nlm technical bulletin, (381). retrieved from https://www.nlm.nih.gov/pubs/techbull/ja11/ja11_nnlm_print_retention.html   fry, a. (2015). conventional wisdom or faulty logic? the recent literature on monograph use and e-book acquisition. library philosophy and practice (ejournal). retrieved from http://digitalcommons.unl.edu/libphilprac/1307/   gross, p. l., & gross, e. m. (1927). college libraries and chemical education. science (new york, n.y.), 66(1713), 385–389. http://dx.doi.org/10.1126/science.66.1713.385   hardesty, l. (1981). use of library materials at a small liberal arts college. library research, 3(3), 261–82.   harker, k., & sassen, c. (2015). enhancing access to e-books. evidence based library and information practice, 10(1), 5–19. http://dx.doi.org/10.18438/b8g02v   kent, a. (1979). use of library materials: the university of pittsburgh study. m. dekker.   new texas a&m-university of texas systems joint library facility proving broadly successful. (2014, september 22). retrieved from http://today.tamu.edu/2014/09/22/new-texas-am-university-of-texas-systems-joint-library-facility-proving-broadly-successful/   nisonger, t. e. (2008). the “80/20 rule” and core journals. the serials librarian, 55(1-2), 62–84. http://dx.doi.org/10.1080/03615260801970774   postell, w. d. (1946). further comments on the mathematical analysis of evaluating scientific journals. bulletin of the medical library association, 34(2), 107–109.   sargent, s. h. (1979). the uses and limitations of trueswell. college and research libraries, 40(5), 416–23.   singson, m., & hangsing, p. (2015). implication of 80/20 rule in electronic journal usage of ugc-infonet consortia. the journal of academic librarianship, 41(2), 207–219. http://dx.doi.org/10.1016/j.acalib.2014.12.002   trueswell, r. w. (1968). some circulation data from a research library. college & research libraries, 29(6), 493–495. http://dx.doi.org/10.5860/crl_29_06_493   trueswell, r. w. (1969a). some behavioral patterns of library uuers: the 80/20 rule. wilson library bulletin, 43(5), 458–461.   trueswell, r. w. (1969b). user circulation satisfaction vs. size of holdings at three academic libraries. college & research libraries, 30(3), 204–213. http://dx.doi.org/10.5860/crl_30_03_204   trueswell, r. w. (1979). “the uses and limitations of trueswell”: a comment. college and research libraries, 40(5), 424–425.   turner, s. j. (1980). trueswell’s weeding technique: the facts. college and research libraries, 41(2), 134–138.   ebl 101   applicability: what is it? how do you find it?   virginia wilson shirp coordinator health sciences library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   originally published in: evidence based library and information practice, 5(2), 111–113. https://ejournals.library.ualberta.ca/index.php/eblip/article/view/8091/6970     received: 21 apr. 2010  accepted: 05 may 2010      2016 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   we’ve come a long way over the past year in this column. there’s been the formulation of the well-built question, the seeking and finding of evidence in the published literature, the consideration of conducting research yourself, and the appraisal of research evidence. now that you have some valid evidence, you need to determine its applicability to the situation at hand.   applicability “relates to the extent to which the results are likely to impact on practice” (booth & brice, 2004). whether or not a particular research study is applicable to your situation is subjective. what works for one person’s situation, may not work for another because “libraries show considerable variation with regard to environment, context and institutional values” (booth, 2009). determining applicability is an essential step in evidence based practice, and it is important to note that there are different levels of applicability to look for.   the evidence you find that is relevant to your situation will usually be one of four things:   1.       directly applicable 2.       needs to be locally validated (i.e. replicate the study at the local level) 3.       needs to be adapted (derivation) 4.       improves your understanding of the situation (koufogiannakis & crumley, 2004 (1, 2 and 4), and booth, 2004 (3))   ideally, a piece of research evidence you find will match all or many of the particulars of your situation, so you can apply it directly and move on to evaluating the results. how do you decide if evidence is directly applicable? there are several variables to consider:   user group: does the user group in the study compare to your user group? timeliness: is the research current enough? technology research, for example, should be the most current research available, while research on collections can be somewhat older. cost: is it fiscally feasible to apply the evidence in your current environment? politics: is there resistance to change in your work environment? will the new concept be accepted? will an adversarial relationship be created? severity: how critical is it that a solution to your situation be found, and quickly? (koufogiannakis and crumley, 2004)   with a bit of thought, you can determine whether or not the research evidence you have found can be directly applied to the decision you need to make or the problem that needs to be solved in your practice.   because, at present, the body of evidence for library and information studies is smaller than, for example, medicine, finding research that is directly applicable can be difficult. you will more likely find evidence that resembles your situation, but that needs to be replicated and validated at the local level. koufogiannakis and crumley state that “when librarians locally validate existing evidence, they are building the evidence base” (2004). it is worthwhile keeping in mind that if you go the route of validating the evidence you have found by replicating it at your level, the greater lis community will benefit if you write up your efforts and find a way to disseminate the information. similarly, taking evidence that illustrates larger issues and applying it at a local level can result in rich and useful material with which to work and can be used as an example by other libraries.   additionally, part of the evidence you find could be adaptable to your own situation. booth (2004) calls this derivation. derivation is possible when “some aspect of the methodology or perhaps the instrument (questionnaire or interview schedule) can be adapted to your own practice, though different” (booth, 2004). in other words, to avoid reinventing the wheel you can borrow a part or parts of a research study to construct your own. as an added benefit, “deviation from a reported intervention, tailoring it according to detailed knowledge of the library’s clientele, may lead to an enhanced chance of success” (booth, 2009).   the most common form of evidence found is evidence that helps to improve a librarian’s understanding of a situation. while not directly applicable to what is going on, this kind of evidence can be applied in ways that increase knowledge and provide a larger context. keeping up with issues important to your practice is a central part of evidence based practice, and reading widely is a good habit to develop.   evidence-based practice for information professionals: a handbook contains an excellent chapter on applicability (chapter 10). the book is currently out of print but a pdf of the pre-publication manuscript has recent been made available at http://ebliptext.pbworks.com/. also useful is an applicability checklist compiled by librarians at the university of newcastle in australia. the checklist contains questions about user group, timeliness, cost, politics, and severity that will help to determine the applicability of research evidence. you can find the checklist here: http://www.newcastle.edu.au/service/library/gosford/ebl/toolkit   only you can decide if a piece of research is applicable to your situation. with these tools, categorizing the research should be straightforward, and you will be well on your way to applying the evidence you have found. in the next issue, ebl 101 will focus on evaluating the results of the research application.   references   booth, a. (2004). what research studies do practitioners actually find useful? health information and libraries journal, 21, 197-200.   booth, a. (2009). eblip five-point-zero: towards a collaborative model of evidence based practice. health information and libraries journal, 26, 341-344.   booth, a., & brice a. (2004). appraising the evidence. in a. booth & a. brice (eds.), evidence-based practice for information professionals: a handbook (pp. 104 118). london: facet.   koufogiannakis, d., & crumley, e. (2004). applying evidence to your everyday practice. in a. booth & a. brice (eds.) evidence-based practice for information professionals: a handbook (pp. 119‐126). london: facet.   microsoft word es_hook.doc evidence based library and information practice 2007, 2:1  137 evidence based library and information practice       evidence summary    web‐based portal for impact evaluation reveals information needs for museums,  libraries and archives      a review of:  williams, dorothy a., caroline wavell, graeme baxter, alan maclennan, and debbie jobson.  “implementing impact evaluation in professional practice: a study of support needs  within the museum, archive and library sector.” international journal of information  management 25.6 (dec. 2005): 533‐48.    reviewed by:   david hook  manager, operations information and configuration management, mda  brampton, ontario, canada  e‐mail: david.hook@mdacorporation.com        received: 11 december 2006  accepted: 11 january 2007      © 2007 hook. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – this study reports on research  into the information and support needs of  practitioners in the museum, archive, and  library sectors, who are undergoing an  impact evaluation.    design – qualitative survey.    setting – web‐based questionnaire.    subjects – twenty‐one practitioners in the  fields of museums, archives, and libraries.    methods – the study made use of a small‐ scale web portal that provides impact  evaluation research findings, toolkits, and  examples of methods. the portal’s intent  was to present to the users multiple views of  the available information in order to  overcome the problem of users not being  able to identify their needs. a purposive  sample group consisting of 50 practitioners  from the museum, library, and archive fields  was invited to participate in a questionnaire  evaluating the website.    main results – despite a fairly low  response rate (49%) and poor distribution  among the three sectors (museums, libraries,  and archives), the results indicated a  significant difference in the levels of  knowledge and understanding of impact  evaluation. over half of the organizations  surveyed had done some assessment of their  evidence based library and information practice 2007, 2:1  138 institution’s economic impact, and there  appears to be a rising trend towards doing  impact studies for specific projects and  developments. nearly a quarter of the  organizations had not undertaken any  impact evaluation study previously.  practitioners already familiar with impact  evaluation tended to look at broader range  of fields for expertise, whereas those with  less familiarity remained within their own  sector. practitioners with less experience  preferred tools, guidance, and examples of  methodologies as opposed to actual  evidence of impact. the results also  provided the authors with feedback on their  web portal and how to organize the  information therein.    conclusions – one of the findings of the  study was that the overall reaction to impact  evaluation support through research  evidence, guidance, and other mechanisms  was positive. for most practitioners,  evaluation itself and the level of  understanding of impact evaluation are at  early stages. the primary goals for those  undertaking impact evaluation were found  to be professional and organizational  learning, thus there is a need for practical  help and guidance in these areas. time  limitation appeared to be a significant factor  in the responses – particularly with smaller  organizations – suggesting that their portal  material would provide much‐needed  assistance to such organizations. finally, it  was concluded that future emphasis should  be placed on developing practical  applications rather than pure research.    commentary   this study reports on research into the  professional needs of practitioners in the  museum, archive, and library sectors who  are undergoing an impact evaluation.  previous studies (wavell, baxter, johnson &  williams) have shown that impact  evaluation can pose many challenges to  practitioners, due to the lack of available  mechanisms to accurately assess impact. the  authors provide a detailed literature review  to further explain the current climate of  evaluation.    this study undertook an interesting  methodology to avoid some of the problems  encountered in typical questionnaires that  ask about information needs. without  having proper knowledge of available  solutions, questionnaire respondents may  have difficulty identifying their actual  information needs. by combining the survey  with a web portal that presents available  solutions as the authors have done, the  respondents can develop an appreciation for  their actual information needs. the web  portal used for this study was built from  knowledge gained on a previous study by  the authors.    one the flip side, by piggybacking this  study on the back of a web portal, this  methodology leads to some confusion. it  appears that the survey was designed to  both evaluate a web portal and to determine  support needs for practitioners for impact  evaluation. the study findings seem to  address the former issue while the  conclusions, the latter. was it worthwhile to  combine these two objectives? was the  survey instrument effective in addressing  the two needs of the study? it is difficult to  say because the survey questions were  unfortunately not provided, so one cannot  assess whether the questions fairly  addressed both issues or if they were more  geared toward one or the other. as well,  without the survey questions, it is not  possible to replicate the study based on the  information provided.     similarly, screen shots of the web portal  would also have been a useful aid to  understand the feedback responses on the  portal itself. as with the survey questions,  no images of the web portal were included  with the study.  evidence based library and information practice 2007, 2:1  139 one of the concerns about the study is the  sample size. of 50 people invited to  participate, only 21 actually completed the  survey – 10, 6, and 4 from the museum,  archive, and library sectors respectively.  one participant was from all sectors.  unfortunately, the low participation rate  does not give sufficient numbers to draw  conclusions that can be applied across the  professions. the authors, however, do  acknowledge this limitation. another  concern is that considering there are as  many participants from the museum sector  as the other two combined, some of the  findings might well be skewed towards the  needs of museum practitioners.    the study is not particularly clear on how  the participants were chosen. there is some  indication that they may have been previous  users of the web portal, but there is also  indication that participants were recruited  via cold calling.    there are some inaccuracies in table 1 in the  number and percentage of questionnaires  sent. the percentages total 117% and the  numbers total 60, rather than 51 as shown in  the table. this is an error in the number of  archives questionnaires sent; table 1 shows  19 whereas the text mentions ten.    unfortunately, with the low response rate,  the results must be regarded as anecdotal.  that being said, the results do give some  insight into the current state of research and  support for impact evaluations. as well, the  authors did receive some valuable feedback  on the available features, and the  organization and presentation of content on  their web portal.    works cited    wavell, caroline, graeme baxter, ian   johnson, and dorothy williams.  “impact evaluation of museums,  archives and libraries: available  evidence project.” 2002. the robert  gordon university. 4 feb. 2007  <http://www.rgu.ac.uk/files/imreport.p df>.                      microsoft word es_haigh.doc evidence based library and information practice 2006, 1:1  86 evidence based library and information practice       evidence summary    children’s input is vital to creating an online library that meets children’s  information needs     a review of:   druin, allison. “what children can teach us: developing digital libraries for children with  children.” the library quarterly 75.1 (january 2005): 20‐41.    reviewed by:   susan haigh   senior policy officer, library and archives canada  ottawa, ontario, canada  e‐mail: susan.haigh@lac‐bac.gc.ca    received: 3 january 2006  accepted: 15 february 2006      © 2006 haigh. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – through use of an  interdisciplinary research team that  included children, the study aimed to  demonstrate that including children in the  design of a digital library for children would  result in some new approaches that would  improve the site’s usability for the target  user group.    design – case study.    setting – the research was conducted at  university of maryland over a four‐year  period  and involved an interdisciplinary  research team of adult researchers from  information studies, computer science,  education, art, and psychology as well as  seven children aged 7‐11.      subjects – seven children participated in  the design team over two years; 153 children  were observed and interviewed in the  design phase; and the resulting new  approaches were validated post‐launch by  analysis of international children’s digital  library (icdl)  (http://www.icdlbooks.org)  users and usage patterns from november  2002‐november 2003 (over 90,000 unique  users and 19,000 optional questionnaire  respondents).      method – the study included seven  children in the design team for a digital  library of international children’s books,  which resulted in new approaches to  collection development, cataloguing, and  the search interface.  in the design phase,  research methods involving the seven  children included brainstorming techniques,   “cooperative inquiry”, low‐tech prototyping;  and lab use studies. the team also  http://creativecommons.org/licenses/by/2.0 http://www.icdlbooks.org evidence based library and information practice 2006, 1:1  87 undertook observation and interviews of  153 children engaged in searching and  selecting books from public library  catalogues. in validating the new  approaches that resulted from the design  research, the team employed web log  analysis, a voluntary online survey, and  working with children in local schools to  understand their use of icdl.      main results – the inclusion of children’s  viewpoints in the design stage of the icdl  had an impact in three areas: collection  development, metadata, and interface  design.     for collection development, the research  showed that kids were interested in books  about children from other cultures and other  times in history; in animals, both real and  make‐believe; in books that are sensitive to  other cultures; and in books that are in good  condition.   for metadata, the research  showed that children do not distinguish  ‘fiction’ and ‘non‐fiction’;  look for ‘scary  stuff’ or ‘gross stuff’; are often seeking books  that make them feel a certain way;  care  about the look of book covers and may  recall books by jacket colours; and use free  vocabulary like ‘princesses’ and ‘jokes’. for  interface design, the children’s involvement  led to more search options (utilizing the new  categories of metadata that were created),  and customization options such as ability to  choose different forms and colour palettes  for book readers (e.g. the comic book reader,  the spiral book reader).    web log and survey data, as well as lab tests,  showed that the innovations resulting from  the children’s design input were used. of  the over 90,000 unique users who visited the  site in its first year, “genre” and “color”  were statistically the fourth and fifth most  popular search categories.  in lab tests, girls  used “color” twice as often as boys, and  older boys preferred “genre” while younger  children did not pay attention to that  category.          conclusions – a first conclusion is that  children’s input is vital to creating an online  library that meets children’s information  needs, tendencies and preferences.     also, seven design principles emerged:     1. children’s input is invaluable and  they should be involved in the  design of their libraries.  2. digital collections for children  should consider works both  contemporary and historical, and in  different languages and  representative of different cultures.  3. a variety of search interfaces are  needed and it is particularly  important to express categories with  visual icons.  4. additional metadata can be needed  to reflect children’s views of  relevant search criteria.  5. interfaces should be customizable,  such as providing various formats  of reader that could themselves be  customized in colour.  6. tools should be suitable for use  from the home and for collaborative  use, such as use by a parent with a  child.  7. innovation requiring high  bandwidth must be balanced with a  low bandwidth version to assure  broadest possible use.     lastly, the researchers concluded that more  research is merited to assess the broader  impact of digital libraries on children as  searchers and readers.     commentary    involving users in site design and analysing  usage post‐launch to improve design, are  frequently neglected activities, but are well  evidence based library and information practice 2006, 1:1  88 known to be vital to a site’s usability and  ultimate success.  as the author notes, this is  even rarer when the users are children; too  often information professionals “talk about  children but rarely talk to them.”  gaining  insights into children’s search behaviours  and preferences before, during and after  designing a searchable digital library for  them would seem self‐evident, but how  many of us have been involved in  developing web sites or even library  services for kids where our only concession  to their viewpoint was the use of brighter  colours and (possibly) less and simpler text?      achieving universal usability is difficult.  but it is important to carefully question how  to interpret the observed evidence. for  example, does the evidence that book colour  is one of the most frequently chosen search  parameters indicate a true user preference,  or is it interface‐determined (i.e. by the  prominent screen placement of those icons)?  are users using it to re‐find a book (which is  appropriate, as based on memory) or to  initially find a book (which is less  appropriate)?  while the article does give  brief summaries of web log, survey, and lab  test findings, it does not provide extensive  statistical analysis focused on the specific  innovations brought about by the children’s  design involvement.       i was left wondering whether the voluntary  questionnaire filled out by 19,000  respondents had provided any user opinion  data about the overall success of the library,  by having asked questions such as whether  they had found what they were looking for,  whether they were satisfied with their  experience, and whether they would  recommend the site to others. the article  also did not indicate future plans for the site,  and whether works will continue to be  added to achieve a critical mass of content  for all search category results.       the extensive user research that has gone  into this intriguing site has resulted in some  important innovations that should continue  to be honed. the findings should become  part of our collective understanding of the  preferences of young library users. but we  can’t rest there: we need to begin to adapt  our vocabularies, innovate in our interfaces,  and become more rigorously (and  vigorously) user‐centric. these research  findings need to be built into the  development of children’s libraries in the  future.      evidence based library and information practice 2012, 7.4 68 evidence based library and information practice article implementing the critical friend method for peer feedback among teaching librarians in an academic setting yvonne hultman özek senior librarian library & ict unit faculty of medicine lund university, lund, sweden email: yvonne.hultman_ozek@med.lu.se gudrun edgren professor centre for teaching & learning faculty of medicine lund university, lund, sweden email: gudrun.edgren@med.lu.se katarina jandér librarian library & ict unit faculty of medicine, lund university, lund, sweden email: katarina.jander@med.lu.se received: 9 mar. 2012 accepted: 4 nov. 2012 2012 özek, edgren, and jandér. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/bync-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – the role of the academic librarian has become increasingly educative in nature. in this study, the critical friend method was introduced among teaching librarians in an academic setting of medicine and health sciences to ascertain whether this approach could be implemented for feedback on teaching of these librarians as part of their professional development. mailto:yvonne.hultman_ozek@med.lu.se mailto:gudrun.edgren@med.lu.se mailto:katarina.jander@med.lu.se evidence based library and information practice 2012, 7.4 69 methods – we used a single intrinsic case study. seven teaching librarians and one educator from the faculty of medicine participated, and they all provided and received feedback. these eight teachers worked in pairs, and each of them gave at least one lecture or seminar during the study period. the performance of one teacher and the associated classroom activities were observed by the critical friend and then evaluated and discussed. the outcome and effects of critical friendship were assessed by use of a questionnaire. results – the present results suggest that use of the critical friend method among teaching academic librarians can have a positive impact by achieving the following: strengthening shared values concerning teaching issues; promoting self-reflection, which can improve teaching; facilitating communication with colleagues; and reducing the sense of “loneliness” in teaching. this conclusion is also supported by the findings of previous studies. conclusion – the critical friend method described in this study can easily be implemented and developed among teaching librarians, provided that there is support from the organization. this will benefit the individual teaching librarian, as well as the organization at large. introduction the role of the academic librarian has changed rapidly over the past decade. an important issue in this context is that these professionals are now being required to play an increasingly educative role in institutions of higher education. this can be described as a transition from being in charge of bibliographic instruction and sessions aimed at teaching skills in locating resources (expert-oriented tasks) to being educators in information retrieval and reference management with a context-based, learner-centred focus. information literacy as a subject has gradually become embedded in the curricula of undergraduate, masters, and postgraduate programmes, as well as in the continuing work of researchers. consequently, teaching in the area of information literacy today is associated with a context-based situation in the higher education environment (biggs &tang, 2007). handal (1999) has pointed out that university lecturers have extensive expertise in their own fields, but they are not always trained as professional teachers. he suggests that applying the critical friend method during peer observation is one of many approaches that can improve teaching skills among faculty. a similar situation exists for professional librarians, who do not undergo teacher training, because their education is focused on library and information science. thus the evolving complex role of the academic librarian also requires new competencies and should include continued professional development in teaching and learning on the same level as that provided to the teaching faculty at the institution. the recent shift towards learning-centred teaching in higher education has prompted universities in sweden to introduce incentives to encourage teaching staff to improve and raise their awareness of their teaching skills. these incentives include: formal teaching awards nominated by students or colleagues, nomination to academies of educators, and teaching portfolios for documentation and reflection on one’s own teaching skills. it is equally important for the academic librarian to understand and use learning theories, and to become engaged in the teaching role (giustini, 2008, 2009; handal, 1999; peacock 2001). in sum, fundamental requirements for academic librarians as teachers include having knowledge of theories and different learning styles, as well as the ability to communicate with academics (peacock, 2001). with the emergence of this new role, redefinitions of the academic librarian have been offered, which emphasize that it is essential both to take evidence based library and information practice 2012, 7.4 70 issues further in teaching and learning, and to encourage a culture of reflective practice as a teacher (lupton, 2002). for example, by attending faculty development courses related to teaching in higher education, an academic librarian will receive an orientation in learning theories in a wider context and will also have the opportunity to create equal partnerships with faculty members (peacock, 2001). peer feedback for improving teaching skills for professional development is equally essential in the academic librarian’s teaching role (castle, 2009; norbury 2001; samson & mccrea, 2008; snavely & dewald, 2011). the critical friend method, which entails being a friend as well as having one, has proven to be a powerful tool to facilitate the process of continuous improvement in teaching (baskerville & goldblatt, 2009; costa & kallick, 1993; dahlgren et al., 2006; handal, 1999). this method involves observing and giving friendly criticism on a colleague’s teaching, and it is based on integrity and mutual trust between colleagues (costa & kallick, 1993; dahlgren et al., 2006; handal, 1999; o’keefe, lecouteur, miller, & mcgowan, 2009). this makes it possible to gain immediate access to a colleague’s expertise and feedback, which can facilitate continued professional development. for the academic librarian as a teacher, the critical friend method can aid self-reflection and help improve teaching skills. the aim of this study was to implement the critical friend method and explore the perceptions of the participants in an academic library setting. other methods of peer review are not explored, as the intention of the research was not to compare the various methods, but instead to implement the critical friend method. literature review in the literature search, information was retrieved from these databases: web of science (sci, ssci), eric, scopus, pubmed, cinahl, lista, and socindex. the search engine google scholar was also used. the years 1980 to 2010 were included in the search strategies, and the following keywords were employed: critical friend, critical friendship, peer observation, academic librarians, professional development, teaching librarians, constructive criticism, structured feedback, and selfassessment. when searching on terms related to “critical friend,” we also excluded terms related to “academic librarian” in order to limit the results to information focused on professional development in teaching. use of the keywords “critical friend” and “peer observation” frequently identified articles in the school improvement literature, higher education, and more recently in the lis literature. the search term “critical friend” was extended to also encompass “peer observation,” because it seems that these two terms are used interchangeably in the literature. citation tracking in web of science and scopus was performed for some key articles on the topic “critical friend.” searching with the google scholar engine also identified papers presented at conferences. a final literature search for 2011 was performed. the literature thus far has indicated that implementation of the critical friend method in various projects involving teachers has mainly yielded positive results that have contributed to effective faculty development. most studies used qualitative methodology; some were quantitative. we found examples of beneficial outcomes for the individual teacher, including increased confidence, confirmation of good practice, a sense of belonging to a collegial community, and breaking the sense of isolation in teaching assignments (dahlgren et al., 2006; o’keefe et al., 2009). it has also been observed that more extensive improvement of teaching methods in the educational organization has led to increased networking and team building that creates a trusting working environment (rosario, lourdes, & de juana, 2003). in the research report of a school reform project, towndrow (2008) states that colleagues alone could help each other by using their reflective voices as a critical lens, concluding that “this strategic move has the power to put teachers, not policy makers, in the driving seat of educational transformation” (p. 919). an early study (achinstein & meyer, 1997), on the other hand, demonstrated the difficulty of combining criticism with evidence based library and information practice 2012, 7.4 71 friendship, as tension arose between participants of the study. the study ran for three years and participation was voluntary. the authors state in their conclusion that the ability to critically challenge teacher colleagues comes with time and practice. thus, considering the different uses of the critical friend method, it is clear that this approach has mainly provided positive outcomes, despite the use of variables reflecting disparate roles and purposes of the critical friend at the various levels of educational organizations. however, the positive outcomes in the literature may be due to publication bias, as negative results are rarely published. despite these positive results, it has also been pointed out that formalizing and regulating a critical friendship can have a negative impact if “critical friend” is used as a concept in official documents, thereby distorting the original meaning (swaffield, 2007). however, it is not within the scope of this study to analyse the suitability of using the critical friendship method. nevertheless, some clarification is needed, and hence these variables are discussed briefly below in the sections concerning the definition of critical friendship. the “friend” in critical friendship as mentioned in the introduction, there are various contexts and formats in which the critical friendship approach has been implemented in the educational settings of schools and universities. but what is a critical friend, and how can we explain critical friendship to create a universal definition for this method? the literature on critical friendship often quotes this statement made by costa and kallick (1993): “a trusted person who asks provocative questions, provides data to be examined through another lens, and offers critiques of a person’s work as a friend” (p. 50). the term “critical friend” has been discussed further by researchers, which has resulted in a variety of definitions or statements. according to swaffield (2007), critical friend refers to a supportive yet challenging relationship between professionals. furthermore, the relationship between critical friends is one that encourages and cultivates constructive critique. the main conditions must exist; besides trust and commitment, it is necessary to have knowledge of the context of the teaching environment. for example, considering school improvement, swaffield and macbeath (2005) disapprove of large-scale use of critical friendship in the self-evaluation and policymaking of schools, and they have raised the question of whether “it may be difficult for the [consultant or leader] to stray too far from policy agendas and political objectives. the freedom to be intellectually subversive and challenging of received wisdom lies close to the heart of the critical friend’s value and purpose” (p. 251). thus swaffield (2007) warns that the lack of knowledge of the local settings by outside experts can have negative consequences, because the critical friend model will become distorted and lose its original powerful form of support. in a research action study by baskerville and goldblatt, a critical friend has been defined “as a capable reflective practitioner … who establishes safe ways of working and negotiates shared understandings to support and challenge a colleague in the de-privatising of their practice” (2009, p. 218). these studies indicate that implementing a more formalized version of the critical friend concept can constitute one of many assessment methods when performed as part of career steps or tenure track (summative purposes). therefore, using the critical friend method for formal peer observation (summative purposes) rather than for individual professional development (formative purposes) in a non-hierarchical setting can distort the original meaning of having a critical friend for feedback. thus, the critical friend concept should not be confused with peer review of teaching for a formal reward system (summative purposes). biggs and tang (2007) also stress that peer review of teaching – having a critical friend – should be for formative purposes for reflection and improvement, and should be separated from summative assessment. in this regard, we can ask what is the meaning and definition of “friend” in critical friendship, if it is used in connection with broader goals evidence based library and information practice 2012, 7.4 72 rather than the individual’s professional and personal development towards becoming a reflective teacher. thus, it has been pointed out that we need to extend our understanding of the role of the critical friend, because little research has been done in this area (swaffield, 2004). gibbs and angelides (2008), shed some light on the term “friendship” within the role and context of the critical friend and take swaffield’s criticism further for clarification. in their extensive conceptual analysis, gibbs and angelides conclude that “critical friends do not report their judgements to others” (p. 223) and then leave. they state that “critical friends visiting schools, giving feedback and then leaving can hardly be considered friendship, although it might well be critical!” (p. 223), indicating that using “friendship” to describe an evaluating process made by external experts (summative purposes) is not justified here. in short, these authors aim to give a more precise definition of “friendship.” they base their discussion on historical writings such as those of aristotle, immanuel kant, francis bacon, and others in an attempt to depict the central characteristics of relationships between friends, including mutuality, awareness of care, engagement, and trust. thus, gibbs and angelides argue that the notion of trust is an essential aspect of critical friendship that introduces “the competence of trust [which] functions as a means to an end [and] is the adequate fulfilment of commitments and functions at the centre of social structures. this is the trust in critical friends” (p. 219). the “critical” in critical friendship having established a definition of the friend in a critical friendship, we turn our attention to the built-in force in the critical friendship – the constructive criticism – that aids the development and improvement needed to become a reflective teacher. handal (1999) has pointed out that, in academia and research, advancement of knowledge requires the critical tradition of quality assurance, including aspects such as the peer-review process for publication, the thesis defence, and journal clubs. here, the tradition of constructive criticism is an accepted and essential activity to ensure the progression of research. in contrast, a similar tradition of collegial feedback and constructive criticism in teaching is not as prevalent in the higher education environment. the literature on assessment and evaluation has shown that teaching can be a lonely endeavour, and at the same time there are thresholds that must be crossed before allowing a peer to enter the sphere of teaching to make observations (dahlgren et al., 2006; handal, 1999). feedback on teaching is gained primarily through evaluations provided by course participants (handal, 1999), but relying solely on such assessments can have a built-in bias and may not deliver responses regarding feedback from all aspects of the teaching and learning situation (o’keefe et al., 2009; wellein, ragucci, & lapointe, 2009). furthermore, selfassessment as a complementary tool for reflection and improvement may not be effective enough, if it is performed in isolation and without communication. investigations of feedback and self-assessment have suggested that a more successful approach is to recognize that there is a need for an external source in goal-setting and continuous structured feedback on the learner’s specific goals for improvement (archer, 2010; hattie & timperley, 2007; kevin & regehr, 2005). so the question is: how can learning aimed at improving teaching skills be accomplished? to fill the void of managing reflection in isolation, the teaching community in higher education has used the critical friendship method for some time as a complementary tool in the regular evaluations and assessments that are conducted to improve teaching methods and skills (wellein et al., 2009). it has been shown that the built-in structured feedback in the context of critical friendship is essential for reflective thinking in the professional development of the individual. moreover, as gibbs and angelides (2008) argue, it is the trust in the relationship that renders the giving and taking of criticism effective. in critical friendship, once mutual trust and engagement have been established, the next step is to provide and receive structured feedback, and then use the benefits attained to develop teaching. evidence based library and information practice 2012, 7.4 73 however, structured feedback in constructive criticism can have different approaches in this communication process and may or may not be as helpful as expected. in an extensive literature review on assessment for learning, wiliam (2011) asserted that “information does not become feedback unless it is provided with a system that can use that information to affect future performance” (p. 4). this statement is derived from a conceptual analysis study in which hattie and timperley (2007) describe the benefits of feedback as a tool and present a model based on three questions: what are the goals? what progress is being made towards the goals? what activities are needed to progress? after those questions are answered, this model also discriminates between four different levels of feedback: the task, the processing, the regulatory level, and the self level. hattie and timperley suggest that feedback seems to be most effective when it provides cues to directions on the processing level, and this in turn appears to help the receiver to pursue more challenging goals. they also conclude that giving feedback as “praise” or by offering comments like “that was good” is ineffective for professional improvement and development since comments made by participants in various studies remained on a general level that gave no specific cues about how to handle the task (i.e., the self level of giving feedback). general statements about “self” may even lead to negative effects such as insecurity. furthermore, wiliam (2011) has stated that feedback must do more than simply provide information: it must be domain-specific, which means that the information given must be generated within a particular system and for a particular purpose. in this context, information should alter the gap between the feedback and the goals set by the learner. when the gap is closed, a learning process directed towards improvement and reflection is begun, as has also been outlined in a review article by norcini (2010) and in a critique of the literature by archer (2010). to summarize, in contrast to self-assessment and evaluation by course participants, the essence of critical friendship consists of structured feedback and constructive criticism that can encourage improvement and selfreflection. this means that having a critical friend involves doing the following in order to become a reflective teacher: setting goals, embracing the challenge of receiving feedback from a trusted colleague, and reflecting on the goals that have been set and the work that is intended to achieve improvement (handal, 1999; towndrow, 2008). our study focused on a critical friend who is an equal in a non-hierarchal setting for formative purposes only, and who shares the context and values with the one to be “criticized” and does not report observations to authorities or any official records, nor to other people (formative purposes). the next section explains the process and outcomes of critical friendship applied in the context of the teaching environment of academic librarians. the critical friendship method used in this study emanated from the fundamental principles of mutual trust, engagement, and commitment in a nonhierarchal setting (costa & kallick, 1993; dahlgren et al., 2006; handal, 1999). methods context our chosen method was a single, intrinsic case study (stake, 1995). a case study is suitable when contemporary events are the focus and behavioural events are not controlled (yin, 2009). the project was carried out within the library unit of the faculty of medicine at lund university in sweden, which is a medium-sized integrated support and service resource centre for students, researchers, and faculty members. the library unit has facilities in two cities, lund and malmö, and it serves 2,500 full-time students enrolled in nine undergraduate degree programmes in medicine and health sciences, as well as 1,000 post-graduate students and 1,200 employees. some of the education provided by the faculty of medicine library unit is conducted at the regional skåne university hospital. at the time this study was carried out, the library unit had 11 librarians who were in charge of evidence based library and information practice 2012, 7.4 74 operating the educational and research resource centres of the faculty of medicine in both cities, and 4 of the 11 also served as contact librarians for all the educational programmes, including phd studies. in swedish universities, librarians do not belong to the academic staff, but rather to administrative staff, and their positions are not tenured but permanent. thus formative assessment is most appropriate for the teaching librarians. teaching performed by the librarians at that time consisted primarily of seminars and workshops, all related to information literacy and scientific communication. in 2002, the management of the library unit implemented a professional development policy for the staff with the aim of requiring teaching librarians to at least take the basic course, “introduction to teaching and learning in higher education,” which is offered by the centre for teaching and learning at the faculty of medicine. giving and taking feedback are topics included in the course. the course program on teaching methods and learning is also part of the required continued education for faculty development. the idea of also making this course mandatory for teaching librarians was a first step in introducing these professionals to the teaching and learning methods and values applied in the parent institution. participation in such courses is one of many strategies that are intended to maintain continued professional development in teaching and learning issues and to improve the teaching methods. the case study in 2009, a project using the critical friend method was initiated at the library unit. the centre for teaching and learning was contacted for support, and the project was started at the beginning of the academic year and continued for 10 months. participation was voluntary, and eight librarians signed up, none of whom had any previous experience in using the critical friendship method. the peer observation was to be used for formative purposes only. the participants had been asked to sign up in pairs if possible, but they did not. accordingly, they were assigned to pairs by the teacher from the centre for teaching and learning. participants with similar prior experience and who had previously worked together and knew each other were assigned to the same pair. the first group meeting took place in a neutral location, and the teacher from the centre for teaching and learning had prepared readings and activities to be done. it was considered important to create a climate of trust. the intention was to establish a common understanding of the concept of critical friend and to discuss the participants’ expectations and perceptions of potential difficulties that could emerge. after the first meeting, it was up to the members of the individual critical friend pairs to decide how to observe each other’s teaching and how to give feedback. this opportunity to decide feedback criteria within the pairs was chosen since teaching forms could vary between giving lectures, facilitating group work, and leading seminars. to guide the process, the participants could use articles published by handal (1999) and dahlgren et al. (2006) and a critical friend protocol including an extensive collection of different questions for the peer observation. one participant dropped out at an early stage due to limited teaching during the project time and difficulty finding the time to participate. the teacher from the centre for teaching and learning thus entered the project as a participant, instead of being just an observer as had been planned. two additional meetings were held during the project period: one of these, the second meeting, was conducted after five months to present a report from the critical friend pairs and to discuss difficulties that had been experienced; the other (third) meeting took place at the end of the project and concerned the final reports from each pair. after the last meeting, a questionnaire (see appendix) with seven open-ended questions was distributed to the participants. two of the authors (ge and kj) worked independently to evaluate the questionnaire responses by content analysis (graneheim & lundman, 2004). an overview of the steps in the project is illustrated in figure 1. evidence based library and information practice 2012, 7.4 75 results the project was completed by all eight participants (after replacement of one initial drop-out). after two preparatory meetings, at the third and final meeting, all pairs were to report what they had done and how they perceived the critical friend process and the results. the statements made at this meeting clearly showed that the participants had taken the opportunity to tailor the process to suit their individual needs. overall, it seemed that their impressions of critical friendship were positive, but there were negative impressions as well. some participants found it difficult to find the time for discussions and agreements, and in particular to be present at scheduled teaching. some also would have preferred to select their own partner. after the final meeting, all eight participants were asked to complete a questionnaire consisting of open-ended items, and seven responded. the participants were guaranteed confidentiality, but in such a small group they were not anonymous to the authors. four categories emerged from the descriptions of their critical friend experiences, and we chose to call these “peer learning,” “change and innovation,” “professional development,” and “prerequisites”(table 1). peer learning the participants discovered new ways of learning from the following: receiving feedback, observing someone else teaching, and giving feedback. several pointed out that discussions with a colleague in an informal setting were important for learning. “we had some very fruitful discussions when meeting afterwards for penetrating the structured evaluation outcome.” learning included increased awareness of positive aspects of their teaching, as well as things that could be improved. in particular, the participants stressed the realization of both their own strengths and those of the colleagues they were paired with. “it was also very nice and uplifting getting constructive feedback on my work from a peer, particularly when the feedback highlighted things that i previously was unaware of.” an important issue here was that learning with a partner reduced the feeling of being alone in the teaching situation. “having a critical friend gave me a sense of not being entirely on my own.” change and innovation several participants mentioned that observing and being observed by a colleague had given them the opportunity to see things from a new perspective. “it was very interesting to observe the participants from behind and see what they were doing at different stages of the lesson.” figure 1 overview of the project steps evidence based library and information practice 2012, 7.4 76 in the discussions that took place both before and after teaching sessions, the participants had opportunities to exchange knowledge and ideas about teaching. “very fruitful as a forum for exchange of ideas and ‘tips and tricks.’” several participants were convinced that the critical friend project had had immediate outcomes in improved teaching. “an immediate impact of my teaching has been an improved ‘student focus.’” “the students have benefited from the project, though in an indirect way.” professional development the project gave the participants opportunities to develop as professionals, not only through the activities in the critical friend pairs, but also as a result of the two project report meetings, once an open climate had been established at the first meeting. “the meetings on two occasions with the whole group involved in the project gave us a sense of coming closer together and may be a suitable forum for educational matters in the future.” a confirmation of the value of the new role of academic librarians as teachers was also seen as an outcome of the project. “it has brought us together to reinforce our identity as teachers.” the realization that it is possible to learn from colleagues can reinforce the development of the professional community. “cf can open up new perspectives and identify ‘hidden jewels’ in the other person’s teaching.” some of the participants pointed out that an increased awareness of personal strengths can augment professional development. “i became more confident in what i was doing.” an important aspect of professional development is reflection, and this was mentioned by some of the participants. “gives you opportunity to reflect on your own teaching.” prerequisites the advice or recommendations that emerged from the informants’ responses indicated that table 1 categories and codes in the answers to the open-ended items in the questionnaire peer learning change/innovation professional development prerequisites informal communication about teaching new perspectives teaching partnership mutual trust exchange of ideas and experiences creative discussions confidence building open-mindedness learning from observing and from being observed improvement of teaching librarian as a teacher willingness to change new interaction with colleagues on an informal, “friendlier” basis utilize and apply colleagues’ experiences and knowledge time and supportive management reduced feeling of loneliness reflection evidence based library and information practice 2012, 7.4 77 the prerequisites for a successful critical friend process include aspects such as the importance of mutual trust and respect between the members of the friend pair. “you must have an open mind, be yourself, humble and open.” an attitude of openness and willingness to change is also a prerequisite for the success of a critical friend project. the external conditions mentioned most often concerned having the following: the time and the opportunity to engage in the project; management that was explicitly supportive and willing to understand the purpose in a broader perspective and context; the ability to recognize the positive contribution of professional improvement to the organization at large. “calculate the time carefully – it takes more time than you might think.” we also found that the working process differed among the pairs. some worked in a more structured manner, using a common preformed evaluation protocol and pre-meetings before each teaching session, whereas others worked in an unstructured fashion without prior discussion about what to evaluate or to select any common evaluation criteria. the most important thing that people learned about their own teaching was an improved awareness of their own actions (such as moving around, turning away, or not speaking loudly enough), and of what their students did during teaching and how easily they became distracted. they also learned to make more use of peer learning for the students by facilitating the exchange of prior knowledge and experiences between them. discussion in this study, we explored the critical friend method. we found two articles, a qualitative method analysis by dahlgren et al. (2006) and a conceptual study by handal (1999), which evoked interest in the concept. using our professional experience we decided to implement a project. we also wanted to find out what the results in the literature showed. as the project was running, we searched the literature further for deeper knowledge of the critical friend method to enhance the professional development of academic librarians as teachers, which suited our context. the findings in our literature search and our case study showed some positive outcomes. the participants who responded to the questionnaire were favourable to giving and receiving structured feedback in peer observations performed with a colleague. one participant stressed that getting structured feedback was more useful than hearing the general statement “that was good.” this agrees with results reported in a meta-analysis by hattie and timperley (2007), indicating that professional development may be promoted by feedback given on the task level, that is, during setting up of goals, identifying needs, and reflecting on the challenge. some participants experienced giving feedback as more of a challenge than they had expected. another positive perception was that the project decreased the feeling of being alone in the teaching situation, and this beneficial outcome has also been highlighted in previous peer observation projects (baskerville & goldblatt, 2009; costa & kallick, 1993; dahlgren et al., 2006; handal, 1999). using a critical friend can fill the void when results of course assessment are seldom or never delivered to the teaching librarian, a situation that often arises when teaching in undergraduate programmes because they are not always compulsory. furthermore, in independent workshops and courses given for faculty members, whether or not course assessments are performed depends on how the teacher perceives the value of such evaluation. in cases like that, it appears that the critical friend method can be a useful way to fill the gap. one of the outcomes of the present critical friend project suggests that this method has strengthened a number of values shared by the teaching librarians, such as self-reflection and evidence based library and information practice 2012, 7.4 78 learning, as well as better communication with colleagues. the respondents pointed out that such an undertaking takes time (like any professional development) and that management should understand the value of the activity and allow it to take the time that is needed. nonetheless, even though the participants had allocated time for the activity, they still had to consider schedules and other tasks concurrently and this was often perceived as a distraction. thus it is clear that shared values that include a clear vision aligned with the organizational values and beliefs are needed to support and allow manifestation of an activity that is intended to promote learning and professional development through teamwork (senge, 2006). the strengths of our study include the fact that we searched the literature for possibilities of professional development of academic librarians as teachers, and then we applied our findings in a realistic setting with no special resources allocated, except for the teacher from the centre for teaching and learning. another advantage was the high degree of completion of the project, with only one drop-out at an early stage. hattie (2009) has identified themes that are important for professional development of teachers that can have an effect on the outcomes of students. we applied most of those themes in our intervention: involvement of an external expert; engagement; teachers talking to teachers; and opportunities to challenge pre-existing beliefs about learning. the weaknesses of our investigation were the low number of participants, the lack of anonymity, and the fact that participation was voluntary. some participants might thus have been unwilling to report negative perceptions. it is plausible that individuals who take part voluntarily in a project will be inclined to perceive outcomes as positive. another limitation was that the authors were also participants, and so other participants may have been less inclined to state negative feelings and outcomes. however, hattie (2009) has reported that voluntary involvement did not affect the outcomes of various studies described in the literature. another limitation of the current work was that the only outcomes considered were the perceptions of the participants; we were not able to study actual results as improved teaching and learning. conclusion the critical friend method described in this study can easily be implemented and developed among teaching librarians, which may benefit the individual librarian as well as the organization at large. the shared values of working in a feedback context to achieve reflection, improvement, and development have the potential to create continuity on the level of the individual professional and to contribute to the entire organization. in this approach, it also seems to be an advantage to have a coach who is a teacher that is familiar with the organization. since this is just a single case study, further studies are needed, preferably using and comparing other methods. however, as one of the participants in the present study stated: “be yourself, humble and open. this is not a competition. it is a great opportunity for your professional development to utilize and apply the experiences and knowledge of your colleagues. there is no course fee, and there are no travel expenses!” acknowledgements the authors express their gratitude to all our colleagues who participated in and contributed to the project. this project was first presented at the eahil conference, ucl, brussels, belgium, 4-6 july, 2012. all three authors participated in the project as librarians (yho, kj) and as a teacher (ge). all authors took part in drafting the manuscript and performing the critical reviews, and approved the final version of the manuscript. evidence based library and information practice 2012, 7.4 79 references achinstein, b., & meyer, t. (1997, mar. 24-27). the uneasy marriage between friendship and critique: dilemmas of fostering critical friendship in a novice teacher learning community. paper presented at the annual meeting of the american educational research association chicago, il. archer, j. c. (2010). state of the science in health professional education: effective feedback. medical education, 44(1), 101-108. baskerville, d., & goldblatt, h. (2009). learning to be a critical friend: from professional indifference through challenge to unguarded conversations. cambridge journal of education, 39(2), 205-222. doi: 10.1080/03057640902902260 biggs, j. b., tang, c. s. (2007). teaching for quality learning at university: what the student does (3rd ed.). maidenhead, uk: mcgraw-hill/society for research into higher education & open university press. castle, s. (2009). peer observation and information skills teaching: feel the fear and do it anyway!: the introduction of peer observation at the university of east london sconul focus, 72-75. costa, a. l., & kallick, b. (1993). through the lens of a critical friend. educational leadership, 51(2), 49-51. dahlgren, l. o., eriksson, b. e., gyllenhammar, h., korkeila, m., sääfrothoff, a., wernerson, a., & seeberger, a. (2006). to be and to have a critical friend in medical teaching. medical education, 40(1), 72-78. doi: 10.1111/j.1365-2929.2005.02349.x gibbs, p., & angelides, p. (2008). understanding friendship between critical friends. improving schools, 11(3), 213-225. doi: 10.1177/1365480208097002 giustini, d. (2008). utilizing learning theories in the digital age: an introduction for health librarians. journal of the canadian health libraries association 29(3), 109-115. giustini, d. (2009). utilizing learning theories in the digital age: from theory to practice. journal of the canadian health libraries association, 30(1), 19-25. graneheim, u. h., & lundman, b. (2004). qualitative content analysis in nursing research: concepts, procedures and measures to achieve trustworthiness. nurse education today, 24(2), 105-112. doi: 10.1016/j.nedt.2003.10.001 handal, g. (1999). consultation using critical friends. new directions for teaching & learning(79), 59-70. hattie, j., & timperley, h. (2007). the power of feedback. review of educational research, 77(1), 81-112. hattie, j. a. c. (2009). visible learning: a synthesis of over 800 meta-analyses relating to achievement. london: routledge. kevin, w. e., & regehr, g. (2005). selfassessment in the health professions: a reformulation and research agenda. [review]. academic medicine, 80(10), s46-s54. lupton, m. (2002). the getting of wisdom: reflections of a teaching librarian. australian academic research libraries, 32(33.2). norbury, l. (2001). peer observation of teaching: a method for improving teaching quality. new review of academic librarianship, 7(1), 87-99 evidence based library and information practice 2012, 7.4 80 norcini, j. (2010). the power of feedback. medical education, 44(1), 16-17. doi: 10.1111/j.1365-2923.2009.03542.x o’keefe, m., lecouteur, a., miller, j., & mcgowan, u. (2009). the colleague development program: a multidisciplinary program of peer observation partnerships. med teach, 31(12), 1060-1065. doi: 10.3109/01421590903154424 peacock, j. (2001). teaching skills for teaching librarians: postcards from the edge of the educational paradigm. australian academic and research libraries, 32(1), 26-40. rosario, a., lourdes, c. s., & de juana, s. (2003). critical friends: a tool for quality improvement in universities. quality assurance in education: an international perspective, 11(1), 31-36. samson, s., & mccrea, d. e. (2008). using peer review to foster good teaching. reference services review, 36(1), 61-70. senge, p. m. (2006). the fifth discipline : the art and practice of the learning organization (rev. and updated ed.). london: random house business. snavely, l., & dewald, n. (2011). developing and implementing peer review of academic librarians’ teaching: an overview and case report. the journal of academic librarianship, 37(4), 343351. stake, r. e. (1995). the art of case study research. thousand oaks, ca: sage publications. swaffield, s. (2004). critical friends: supporting leadership, improving learning. improving schools, 7(3), 267278. doi: 10.1177/1365480204049340 swaffield, s. (2007). light touch critical friendship. improving schools, 10(3), 205-219. doi: 10.1177/1365480207077845 swaffield, s., & macbeath, j. (2005). school self-evaluation and the role of a critical friend. cambridge journal of education, 35(2), 239-252. doi: 10.1080/03057640500147037 towndrow, p. a. (2008). critical reflective practice as a pivot in transforming science education: a report of teacherresearcher collaborative interactions in response to assessment reforms. international journal of science education, 30(7), 903-922. doi: 10.1080/09500690701279014 wellein, m. g., ragucci, k. r., & lapointe, m. (2009). a peer review process for classroom teaching. american journal of pharmaceutical education, 73(5), 79. wiliam, d. (2011). what is assessment for learning? studies in educational evaluation, 37(1), 3-14. doi: 10.1016/j.stueduc.2011.03.001 yin, r. k. (2009). case study research. design and methods. thousand oaks, ca: sage publications. evidence based library and information practice 2012, 7.4 81 appendix critical friend: the project questionnaire briefly describe how you worked together with your critical friend, in particular considering how you reached consensus on the criteria for evaluation! what aspects of your participation in the critical friend project were positive? what was the most important thing you learned by taking part in the critical friend project? do you think you will change anything in your teaching as a result of the critical friend project? besides professional development of teaching, do you feel that work in the critical friend project has had any other effects? will you continue to use the critical friend method even if the organized activities with a group leader are no longer conducted? why or why not? do you have any advice or recommendations you would like to give to others who want to test the critical friend method? evidence based library and information practice editorial responsibilities      2015. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     editor-in-chief: lorie kloda   associate editor (evidence summaries): heather pretty   associate editor (articles): rebekah (becky) willson, lisl zach   associate editor (classics): jonathan eldredge   associate editor (reviews): denise koufogiannakis   associate editor (features, commentaries, using evidence in practice, research in practice, news): lorie kloda   guest editor (feature): helen partridge   production editor: michelle dunaway   editorial intern: melissa griffiths   editorial advisors: lindsay alcock, alison brettle, katrine mallan, pam ryan   copyeditors: richard hayman (lead copyeditor), georgianne bordner, bryan chan, julie evener, heather healy, stacey penney, maria tan, mary virginia taylor, nikki tummon, alison yeoman   indexing support: meral alakus   microsoft word es_brown.doc evidence based library and information practice 2006, 1:2      23 evidence based library and information practice       evidence summary    new search strategies optimize medline retrieval of sound studies on treatment or  prevention of health disorders    a review of:  haynes, r. brian, k. ann mckibbon, nancy l. wilczynski, stephen d. walter, and stephen r.  were. “optimal search strategies for retrieving scientifically strong studies of treatment  from medline: analytical survey.” bmj 330.7501 (21 may 2005): 1179.    reviewed by:  marcy l. brown  clinical medical librarian  the western pennsylvania hospital, forbes regional campus  monroeville, pennsylvania, united states of america    received: 10 march 2006            accepted: 21 april 2006      © 2006 brown. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to develop and test search  strategies for retrieving clinically sound  studies from the medline database on the  prevention or treatment of health disorders.     design – analytical survey.    subjects – the data sources were articles  about treatment studies selected from 161  journal titles indexed for medline in the  year 2000.    setting – medline database searches  performed at the health information  research unit, mcmaster university, in  ontario, canada.     methods – researchers hand searched each  issue of 161 journal titles indexed in  medline in the year 2000 to find treatment  studies. journal content included internal  medicine, family practice, nursing, and  mental health titles. selected studies met the  following criteria: randomisation of subjects,  outcome assessment for at least 80% of who  entered the study, and an analysis consistent  with study design. of 49,028 potential  articles, 6,568 were identified as being  treatment or prevention related, and 1,587  met the evaluation criteria. the study  authors then created search strategies  designed to retrieve articles in medline  that met the same criteria, while excluding  articles that did not. they compiled a list of  4,862 unique terms related to study criteria,  and tested them using the ovid  technologies search platform. overall,  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:2      24 18,404 multiple‐term search strategies were  tested. single terms with specificity greater  than 75% and sensitivity greater than 25%  were combined into strategies with two or  more terms. these multiple term strategies  were tested if they yielded sensitivity or  accuracy greater than 75% and specificity of  at least 50%.    main results – of the 4,862 unique terms,  3,807 retrieved citations from medline  that researchers used to assess sensitivity,  specificity, precision, and accuracy. the  single term that yielded the best accuracy  while keeping sensitivity greater than 50%  was ‘randomized controlled trial.pt.’.the  single term that yielded the best precision  while keeping sensitivity greater than 50%  was also ‘randomized controlled  trial.pt.’.this term also gave the greatest  balance of sensitivity and specificity.  combination strategies varied. some two‐ term combinations outperformed single  term strategies and three‐term combinations.  tables in the article provide the top three  search strategies yielding the highest  sensitivity, specificity, accuracy, and balance  between sensitivity and specificity.     search strategies with sensitivity greater  than 50% and specificity greater than 95%  were evaluated further by adding search  terms using logistic regression techniques.  the best strategies for maximizing  sensitivity had sensitivity greater than 99%  and specificity higher than 70%. the best  strategies for maximizing specificity had  sensitivity greater than 93% and specificity  more than 97%.     conclusion – in addition to providing the  best strategies developed in this study,  authors compared their results with the  results from 19 other published strategies.  the published strategies had a sensitivity  range of 1.3% to 98.8% on the basis of the  hand searched articles. these were all lower  than this study’s best sensitivity of 99.3%.  two strategies published by dumbrigue,  with specificities of 98.1% and 97.6%,  outperformed this study’s most specific  strategy of 97.4%.    commentary    the authors clearly state the purpose of the  study, and their choice of an analytical  survey is the appropriate methodology for  this research question. while the methods  are described in detail, the description is  somewhat disjointed and extends into the  section discussing results. readers might  have been better served by a chronological  discussion of study methods. the authors  provide a straightforward diagram detailing  their data collection procedures.    the authors selected 161 clinical journals to  hand search. although they do not detail  the process by which these titles were  selected, they direct readers to two prior  studies containing specific methods (haynes  and wilczynski, montori et al.). the journal  list evolved from an iterative process over  several years, based on a hand search  examination of more than 400 journals  recommended by physicians, librarians, and  editors. other factors used to determine the  journal list included science citation index  impact factors and the authors’ assessment  of the journals’ scientific merit.     the top strategies are presented in a series  of seven tables that are visually streamlined  and easy to read and interpret. formulas for  calculating sensitivity, specificity, precision,  and accuracy are presented in another table.  many searchers are familiar with the  concepts of sensitivity and specificity, but  less familiar with precision and accuracy, so  the inclusion of these formulas is important.  data in table 4 indicate that the top three  search strategies yielding the highest  specificity had sensitivity greater than 93%,  while the text of the results section indicates  evidence based library and information practice 2006, 1:2      25 this was 94%. this is a minor error and does  not affect the subsequent discussion.    also curious is why the dumbrigue  strategies mentioned in the results section  were not included in the specificity range of  the 19 other published strategies. the  authors write, “the specificities for the  published strategies ranged from 63.3% to  96.6%.” yet in the next sentence, dumbrigue  (who is included in the list of the 19  strategies) is credited for having two  strategies with specificity figures of 98.1%  and 97.6%. it is not clear why the one with  the sensitivity greater than 50% was not  included in the range.    the authors point to the reasons why no one  search strategy is perfect. factors such as  indexing inconsistency, indexing rule  changes over time, and the inclusion of  many non‐clinical journals in medline  will affect retrieval.    while searching for research evidence is  only one part of the complete skill set  needed to practice evidence based medicine,  it is the one area for which many clinicians  have little or no training. most clinicians  will not have the time or the patience to  learn to construct multiple‐term search  strategies using controlled vocabulary,  which means that tested and predefined  search strategies become doubly important.  strategies that have been submitted to  rigorous statistical testing and that have  demonstrated high sensitivity and/or  specificity, can be incorporated into clinical  literature databases to assist novice  searchers in retrieving relevant and  scientifically strong studies.    this is exactly what happened with the  results of this study. the most sensitive and  the most specific search strategies  developed by the study authors have been  incorporated into the pubmed clinical  queries search filter and into the search  features of the ovid technologies  medline interface. the single most  optimal strategy was added to skolar search  options. a busy clinician looking for only  one or two papers most relevant to a  particular clinical question, can use the  specificity filter. a researcher more  interested in the universe of evidence for a  clinical question can use the sensitivity filter  to retrieve a larger number of potentially  useful articles.    one problem with the focus of these search  strategies being limited to randomised  controlled trials is that they exclude studies  where randomisation is not possible. critics  of evidence based practice have previously  written of this problem (ray, schelling,  vincent). population‐based studies, certain  treatment modalities, cohort studies, and  other study designs are excluded from the  search, potentially leading searchers to  incorrectly conclude that these other papers  are not rigorous or scientifically sound.    this shortcoming aside, this is evidence  based practice at its best. even librarians  with years of search experience can, and do,  benefit from the work done by this health  information research team to develop and  test high quality search strategies.    works cited    dumbrigue, herman b., et al. “assessment  of medline search strategies for  randomized controlled trials in  prosthodontics.” journal of  prosthodontics 9.1 (march 2000): 8‐13.    haynes, r. brian, and nancy l. wilczynski  for the hedges team. “optimal search  strategies for retrieving scientifically  strong studies of diagnosis from  medline: analytical survey.” bmj  328.7447 (1 may 2004): 1024.    evidence based library and information practice 2006, 1:2      26 montori, victor m., et al. “optimal search  strategies for retrieving systematic  reviews from medline: analytical  survey.” bmj 330.7482 (8 jan 2005): 68.    ray, joel g. “evidence in upheaval:  incorporating observational data into  clinical practice.” archives of internal  medicine 162.3 (11 feb 2002): 249‐54.    schelling, franz a. “clinical trials:  deliberations on their essence and  value.” journal of evaluation in clinical  practice 10.2 (may 2004): 291‐96.    vincent, jean‐louis. “evidence‐based  medicine in the icu: important  advances and limitations. chest 126.2  (august 2004): 592‐600.  evidence based library and information practice 2011, 6.3 98 evidence based library and information practice news/announcements evidence based library and information practice preserves with the clockss archive © 2011. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. the clockss archives is pleased to announce that it has partnered with evidence based library and information practice (eblip) to preserve the e‐journal in clockss's geographically and geopolitically distributed network of redundant archive nodes, located at 12 major research libraries around the world. by archiving with clockss, evidence based library and information practice has committed to the preservation of their e‐journal. this action provides for content to be freely available to everyone after a "trigger event" and ensures an author's work will be maximally accessible and useful over time. denise koufogiannakis, editor‐in‐chief of the journal notes, "eblip is pleased to partner with clockss in order to ensure long‐term preservation and access to our journal's content. as an open access and online‐only publication, we know that this partnership will strengthen the trust that authors place in eblip when choosing to publish with us. we value the clockss archive's commitment to the preservation of scholarly information, and its collaborative and distributed model for doing so." clockss executive director randy s. kiefer adds, "the clockss archive welcomes evidence based library and information practice into the community's archive. as an open access publication, by archiving with clockss, they have ensured that the original research and commentary in their publication will remain available to all, now and in the future. clockss has achieved a rare consensus among libraries and publishers, and we are grateful, as evidence based library and information practice joins the clockss archive, for its generous willingness to preserve the publication in a way that secures it for the long‐term good of scholars worldwide." about clockss: the clockss (controlled lockss) archive is a not‐for‐profit joint venture between the world's leading scholarly publishers and research libraries whose mission is to build a sustainable, geographically distributed dark archive with which to ensure the long‐term survival of web‐based scholarly publications for the benefit of the greater global research community. http://www.clockss.org http://www.clockss.org/� / evidence based library and information practice microsoft word es_stephens.doc evidence based library and information practice 2006, 1:4      67 evidence based library and information practice       evidence summary    elementary, middle, and high school students vary in frequency and purpose when  using online digital references    a review of:  silverstein, joanne. “just curious: children’s use of digital reference for unimposed queries  and its importance in informal education.” library trends 54.2 (fall 2005): 228‐44.      reviewed by:  julie stephens  media specialist, calhoun educational complex   calhoun, georgia, united states of america  e‐mail: stephensj@calhounschools.org      received: 5 september 2006            accepted: 26 october 2006      © 2006 brown. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to determine 1) how and with  what frequency children use digital  references to answer their own unimposed  questions; 2) whether digital reference  services support their self‐initiated learning;  3) whether digital reference services support  the transfer of student motivation and  curiosity from the formal to the informal;  and 4) what instructional and software  designers should consider in creating tools  that support learning.    design – inductive analysis.    setting – virtual reference desk’s (vrd)  learning center (http://vrd.askvrd.org/) and  the national science foundation’s (nsf)  digital reference service  (http://www.esteme.org) during excellence  in science, technology, engineering, and  mathematics education week (esteme),  april 11‐16, 2005.    subjects – elementary (k‐5), middle (6‐8),  and high school (9‐12) students from the  general public. one hundred fourteen  questions were analyzed, however there is  no indication of the number of different  students who submitted the questions.    methods – this study was conducted using  a pool of 600 questions from students,  teachers, parents, and the general public  that were submitted to two digital reference  services intended for students. three  hundred experts in the fields of math and  http://creativecommons.org/licenses/by/2.0 http://vrd.askvrd.org/ http://www.esteme.org evidence based library and information practice 2006, 1:4      68 science volunteered to answer the  submitted questions during excellence in  science, technology, engineering, and  mathematics education week. because the  digital services employed a pull‐down  menu to describe the user as a student,  teacher, parent, etc., the questions could be  narrowed to those submitted by students.  the questions were also narrowed to those  marked as “just curious” from a question  purpose menu that contained categories  including “written report,” “science fair  project,” and “just curious.” a total of 114  unique questions from elementary, middle,  and high school students were analyzed to  determine the study objectives. the 114  questions were loaded into a qualitative  software application (hyperresearch) for  inductive analysis. questions from students  were coded as elementary, middle, or high  school and only those questions derived  from students’ self‐initiated interests were  analyzed.    main results – analysis revealed that  elementary students submitted a large  portion of the questions. middle school  students asked the most questions, of which  some questions were compound (more than  one question in a given query). older  students submitted the least amount of  questions. an unanticipated finding was that  students’ grade levels correlated to the foci  of their queries, which regarded “my life,”  “my stuff,” “other people,” “the world,”  “the universe,” or “abstract thought.”  high school students were interested in the  narrowest foci pertaining to their immediate  circumstance rather than the larger topics of  other people, the world, and the universe.  the majority of middle school students were  interested in abstract concepts, and 45% of  elementary school students’ queries were  about how the world works. although this  study was not longitudinal, results suggest  that student curiosity may shift over time.  results also indicated that younger children  demonstrated interests outside the  classroom that were related to formal  learning previously introduced within the  classroom. this carry over of interest  declined in upper grades. queries that were  unimposed but related to an academic  subject such as science or social studies were  most evident in questions submitted by  elementary students, while questions  dealing more with career planning, health,  death, and anxiety were most frequently  addressed by middle school students. the  findings also indicated that the use of digital  reference services is at its highest in  elementary school, peaks in middle school,  and falls drastically in high school.    conclusion – 1) how and with what  frequency do children use digital reference  services to answer their own unimposed  questions? the results of this study revealed  a trend on the frequency and purpose of use  of digital references when seeking answers  to self‐initiated questions. elementary  students tend to use the digital reference  services more often and for answers to  questions that may be related to classroom  curriculum. middle school students utilize  digital references to look for information  pertaining to careers, health and welfare,  death and anxiety. high school students  submitted questions pertaining to their  immediate circumstances (“my life and my  world”) rather than focusing on others.   2)  do digital reference services support self‐ initiated learning? of the original 2,258  questions submitted, 13% were considered  “informal.” these findings indicate that  digital reference services support self‐ initiated learning. 3) could digital reference  services support the transfer of student  motivation and curiosity from formal to  informal education and learning? the  frequency of questions from elementary  students that were coded as “curriculum  related interest” leads one to conclude that  digital reference services can indeed support  the transfer of student interest from formal  to informal education. 4) what should  evidence based library and information practice 2006, 1:4      69 instructional and software designers  consider when creating tools that support  the notion of transformed education and  learning? although it is impossible to  actually know the nature of the difficulties  experienced by users, duplicate questions  from the same user were received by the  reference services, which suggests that the  user may be experiencing difficulty with the  software or that the software may actually  be malfunctioning during usage. compound  questions were also frequently submitted.  software designers should take note of these  findings to design services that are age‐ appropriate, especially regarding the type of  questions each age group tends to ask, and  that support the needs of elementary,  middle, and high school students. software  designers should perhaps even consult with  students who use these services during the  design process.     commentary    the results of this research were based on  only two digital reference services.  each  was designed for a different purpose: one  for academic subjects and one for only  science‐related questions. this circumstance  could, by nature, influence the type of  queries that were submitted and the validity  of the results.  after weeding non‐ appropriate questions and users, the pool of  subjects was small; however enough data  was collected to observe a trend relevant to  digital reference librarians and software  designers.  some data from this study may  have been invalid due to the fact that                       teachers and parents often register as  “students” when using the services. even  with these limitations, this research  contributes to our understanding of the  nature of unimposed questions different age  groups seek to answer via digital reference  services.  conclusions from this study could  be used to improve the structure and  services of such tools.  a close look at the  type of questions and questioning  techniques used by students could also  prove helpful to reference librarians and  software designers.    works cited    esteme: excellence in science, technology,  engineering, and mathematics education.  department of education. 4 nov. 2006  <http://www.esteme.org/>.    virtual reference desk aska service. 2006.  drei learning center. 4 nov. 2006  <http://vrd.askvrd.org/default.aspx>.   http://www.esteme.org/ http://vrd.askvrd.org/default.aspx microsoft word news_studyday.doc evidence based library and information practice 2006, 1:3  102 evidence based library and information practice     news      study day: evidence based librarianship and information practice in uk health  libraries        © 2006 eblip. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    evidence based librarianship and  information practice in uk health  libraries    an opportunity to hear about eblip in  action! this study day will provide  examples of eblip in practice, and give you  some ideas as to how eblip can be used to  improve your library and information  service.     for those of you who couldn’t make it out to  australia in 2005, this study day is an  opportunity to hear 6 of the british speakers  who presented at the evidence based  librarianship conference in brisbane last  year.  presentation topics range from  providing information for patient care to the  issues and problems experienced by  librarians’ conducting research, as well as  exploring why mental health staff don’t use  library services.      in the afternoon you’ll be able to choose  between one of the following workshops:    • how to conduct and interpret  evaluations of information skills training   • how to conduct and interpret user  surveys/information needs analyses      • how to conduct and interpret  evaluations of library and information  services       date: monday 20th november 2006    venue: central london, uk.    cost:    hlg member: £65.00 + vat (£11.38) =  £76.38  non‐member: = £80.00 + vat (£14.00) =  £94.00      to book a place, or for further details and a  full programme, please contact marishona  ortega: mortega@lincoln.ac.uk or (01522)  837350          http://creativecommons.org/licenses/by/2.0 evidence summary   library and information science research literature is chiefly descriptive and relies heavily on survey and content analysis methods   a review of: aytac, s. & slutsky, b. (2014). published librarian research, 2008 through 2012: analyses and perspectives. collaborative librarianship, 6(4), 147-159.   reviewed by: heather coates digital scholarship & data management librarian university library indiana university-purdue university indianapolis (iupui) indianapolis, indiana, united states of america email: hcoates@iupui.edu   received: 01 sept. 2015 accepted: 23 oct. 2015      2015 coates. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to compare the research articles produced by library and information science (lis) practitioners, lis academics, and collaborations between practitioners and academics.   design – content analysis.   setting – english-language lis literature from 2008 through 2012.   subjects – research articles published in 13 library and information science journals.   methods – using a purposive sample of 769 articles from selected journals, the authors used content analysis to characterize the mix of authorship models, author status (practitioner, academic, or student), topic, research approach and methods, and data analysis techniques used by lis practitioners and academics.   main results – the authors screened 1,778 articles, 769 (43%) of which were determined to be research articles. of these, 438 (57%) were written solely by practitioners, 110 (14%) collaboratively by practitioners and academics, 205 (27%) solely by academics, and 16 (2%) by others. the majority of the articles were descriptive (74%) and gathered quantitative data (69%). the range of topics was more varied; the most popular topics were libraries and librarianship (19%), library users/information seeking (13%), medical information/research (13%), and reference services (12%). pearson’s chi-squared tests detected significant differences in research and statistical approaches by authorship groups.   conclusion – further examination of practitioner research is a worthwhile effort as is establishing new funding to support practitioner and academic collaborations. the use of purposive sampling limits the generalizability of the results, particularly to international and non-english lis literature. future studies could explore motivators for practitioner-academic collaborations as well as the skills necessary for successful collaboration. additional support for practitioner research could include mentorship for early career librarians to facilitate more rapid maturation of collaborative research skills and increase the methodological quality of published research.   commentary   librarianship includes practitioners serving a range of clients in widely different settings. as such, librarianship includes individuals with diverse backgrounds, expertise, and research interests. the goal of this study was to characterize the portion of lis literature describing the research conducted by practitioners and academics engaged in research. the results confirm some common knowledge about lis research; namely, that librarians rely heavily on surveys and relatively simple statistical techniques. it also provides new insights about the possible differences in the types of research conducted by practitioners and academics.   unfortunately, the validity of the results remains unclear due to the study’s incomplete reporting of the rationale and method for coding the research articles (glynn, 2006). two issues are of primary concern. first is the lack of information provided about the inclusion criteria for the sample of research articles. it is difficult to understand how the results relate to the broader body of lis research literature without knowing how the authors defined and bounded their research. the second issue is the use of a journal sample that may not equally represent all librarians, specifically corporate, special, and school librarians. the authors do not provide selection criteria or address this limitation. although the authors claim the results are representative of practitioner-researchers, purposive sampling does not support this generalization.   determining face validity of the categories used for analysis of authorship, topics, research methods, and research techniques requires more information than is provided. the article would benefit from further details about category descriptions and the process for developing and assigning these codes. in particular, collapsing authorship into single author and collaborative authorship groups may hide possible differences between co-authorship and larger collaborative research partnerships. another concern regarding authorship roles arises from the use of author affiliation and title as proxies for classifying authors into practitioners, academics, students, or other. finally, the study excludes collaborations between librarians and academics beyond those in lis.   the disconnect between the introduction, results, and conclusion makes it difficult for the reader to identify implications for practice. the authors do not clearly relate the results to the literature cited (e.g., watson-boone, 2000), nor do they provide meaningful conclusions about the utility of the results for library practice or future practitioner research.   future research in this area could identify professional trends in collaborative research and the use of specific research methods and statistical techniques. such studies should carefully consider choices about defining authorship models and use standard definitions for research approaches, methods, and statistical techniques. extrapolating librarian professional development needs from this limited sample of journals excludes librarian research published in interdisciplinary and topical journals. any future studies should include a wider sample of journals or use a stratified sample of practicing librarians and lis researchers from a range of settings.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi: http://dx.doi.org/10.1108/07378830610692154      watson-boone, r. (2000). academic librarians as practitioner-researchers. journal of academic librarianship, 26(2), 85-93. http://dx.doi.org/10.1016/s0099-1333(99)00144-5     evidence based library and information practice commentary   systematic searching in a post-pandemic world: new directions for methods, technology, and people   paul levay information specialist national institute for health and care excellence manchester, united kingdom email: paul.levay@nice.org.uk   jenny craven information specialist manchester, united kingdom email: cravenj@btinternet.com   received: 5 aug. 2023                                                              accepted: 13 oct. 2023      2023 levay and craven. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30415     introduction   in january 2019 we concluded our book, systematic searching: practical ideas for improving results, by asking the question, where do we go from here?” we would like to begin to answer this question by assessing how the search landscape has changed since our book was published (levay & craven, 2019). the covid-19 pandemic has accelerated change and led us in new directions, as well as confirmed some of the issues that we already anticipated.   this commentary focuses on the challenges facing information specialists and others who are engaged in searching to support systematic reviews and other evidence syntheses in healthcare, briefly reviewing the main developments that have occurred in relation to methods, technology, and people. throughout this commentary, we refer to information specialists as a general term to describe any information professional involved in systematic searching. this article is intended to prompt discussion about how the information profession might need to adapt, and it is not a comprehensive summary of the recent literature. in addition, we have not covered information about how to conduct systematic searching, as a new guide to that topic is available (foster & jewell, 2022).   methods   we have to ensure that we keep our methods up to date to meet new challenges so that we continue delivering the right evidence at the right time.   living reviews and living guidelines   the importance of keeping systematic reviews and guidelines up to date became vitally important during a global pandemic caused by a new disease. living reviews, which are continually updated to incorporate relevant new evidence as it becomes available, had been established several years before the pandemic (elliott et al., 2017). covid-19 meant we had to accelerate the process of developing living guidelines to incorporate the findings of living reviews into evidence-based recommendations. a living guideline contains one or more recommendations that are “kept current by an optimized guideline-updating process that accounts for potentially consequential evidence as soon as or shortly after it becomes available” (el mikati et al., 2022, p. 1155).   a key question when developing living guidelines is whether to search broadly to cover a whole topic or to run several, targeted searches (mcdonald et al., 2023). the decision is affected by how often the evidence is likely to change, the number of databases or other search techniques required, the frequency of the searching, the current level of uncertainty in the evidence, and the time available for processing the search results. reporting the searches for a living guideline is challenging as the recommendations could be based on different strategies, date limits, and sources (metzendorf et al., 2022).   covid-19 has shown that it is feasible to establish living reviews that support living guidelines. the challenge now is to extend the living approach in topics other than covid-19. this will require thinking about the issues we explore in the following sections relating to types of evidence, using technology efficiently, and developing new skills.   searching for the right types of evidence   as information specialists, we had to identify the appropriate types of evidence for the covid-19 pandemic. there is an ongoing debate about how systematic reviews can incorporate mechanistic evidence, which is derived from studies that explain the factors, interactions, and other mechanisms that are responsible for a phenomenon (greenhalgh et al., 2022). for example, reviews on the effectiveness of face masks for stopping the spread of covid-19 might consider a wide range of study designs from various disciplines, including in vitro experiments, imaging data, aerosol science, and engineering research (greenhalgh et al., 2022). search strategies focusing on data from clinical trials would miss significant areas of this evidence. these are new subject areas for many information specialists in healthcare settings, requiring us to explore the appropriate sources and search techniques to retrieve mechanistic evidence.   the international focus on mechanistic evidence highlighted a long-standing issue on how we identify and synthesize evidence to understand how complex interventions operate (greenhalgh & peacock, 2005). it was even more important than ever that we discussed with review teams the types of evidence required instead of relying on our familiar processes. we need to apply this approach in all systematic reviews rather than relying on standardized methods. we must choose the appropriate search approaches (cooper et al., 2022) and sources (levay et al., 2022a) for the topic and the type of evidence required. we know that using different search approaches and a wide range of sources takes longer and requires more resources (briscoe et al., 2022). we have to resolve these logistical challenges with review teams or else we risk missing relevant evidence.   incorporating preprints   accessing the latest evidence was crucial when millions of lives were potentially at risk from covid-19. preprints, which are full manuscripts of papers made available before or in parallel with the peer review process (clyne et al., 2021), were required when our core sources could not provide the up-to-date studies needed to understand covid-19.   we have to devise methods to deal with the challenges that we face when incorporating preprints into evidence synthesis (khalil et al., 2021). it is important to update processes so that we can identify whether the preprints included in a systematic review changed or were retracted after they had been peer reviewed (brierley et al., 2022). our reference management practices need updating, as a preprint and the linked final article are not strictly duplicates, although they must not be double counted in a meta-analysis. preprint servers are also of varying scope and quality (kirkham et al., 2020), often lacking the sophisticated interfaces needed to write precise search strategies. some of these technical issues have been resolved since preprints became available on europe pubmed central (rosonovski et al., 2023).   technology   we must deploy technology effectively to be able to handle increasing complexity, higher volumes, and different types of evidence.   study-based registers   study-based registers (databases in which all references to a study are available in a single record) have been available for several years to make evidence easier to find (shokraneh & adams, 2019). for example, a study-based register covering clinical trials might bring together references to the protocol, main results, and sub-group analyses into a single record. covid-19 gave fresh impetus to these registers, as they facilitated rapid searches, while reducing research waste and duplication between teams.   a number of open access covid-19 study-based registers have been established, such as the cochrane covid-19 study register (metzendorf & featherstone, 2021) and epistemonikos covid-19 l·ove (verdugo-paiva et al., 2022). they have been reviewed favourably in terms of completeness and timeliness (butcher et al., 2022; pierre et al., 2021).   we would benefit from having guidance to help us identify when to use registers in place of separate databases. barriers to uptake include technical ones (such as how easy it is to export the results) or personal ones relating to confidence with using unfamiliar databases and other search tools, such as study-based registers. registers are time consuming and expensive to maintain so extending this approach would require the major producers of systematic reviews to invest in the infrastructure.   machine learning classifiers   a feature in facilitating up to date and comprehensive study-based registers and living systematic reviews was the widespread adoption of machine-learning classifiers. machine learning deploys algorithms that learn to perform a specific task in order to make predictions based on the training data that has been provided (thomas et al., 2021). in the context of systematic reviews, the training data is often the decisions on which papers to include or exclude. the data enables the algorithm to “learn” which words and phrases are more likely to lead to a paper being included, in comparison to those indicating that the paper should be excluded (o’mara-eves et al., 2015).   machine learning requires large quantities of training data, and this takes time to acquire, validate, and process (stansfield et al., 2022). it would be fruitful to share machine-learning algorithms across topics or domains so that each review team does not have to start afresh. machine learning has been used to identify randomized controlled trials (thomas et al., 2021) and to populate the cochrane covid-19 study register (shemilt et al., 2022). as machine learning becomes more fully incorporated into workflows for screening search results (chappell et al., 2023), we may be able to provide broader, less precise, strategies for some reviews. information specialists can advise review teams, and we should be involved in decisions about how and when to use machine learning.   developing search strategies   technological developments are driving change even in areas where we have well-developed practice. numerous tools are available to help us design and deliver searches, as listed on the systematic review toolbox website (johnson et al., 2022). more changes are coming that will affect how we do this work.   automated indexing will affect the sensitivity and precision of our strategies, encouraging us to review how we develop and test searches. to aid selection of controlled vocabulary terms, the national library of medicine (nlm) has had a fully automated process for indexing medline records with medical subject headings (mesh) since april 2022 using the medical text indexer (mti) (national library of medicine, 2022). there are initial indications that mti, compared to human indexers, will be responsible for applying more mesh terms to each record, omitting age-related check tags, and choosing headings from different levels in the hierarchy (chen et al., 2023). this increasing automation might influence the effectiveness of particular search strategies. as a result, we might need to review any search strategies, including validated search filters, written before fully automated indexing was implemented.   another area worth exploring is the potential of using search visualization to replace the familiar form-based method of inputting queries into services such as pubmed. for example, there has been some success in testing a visual interface for creating and editing searches, such as the one provided by 2dsearch (svarre & russell-rose, 2022). we will benefit from collaborating with computer scientists and software engineers to develop the tools we need. these conversations can be facilitated by using design principles relevant to systematic searching (macfarlane et al., 2022).   can we use artificial intelligence (ai) to generate search strategies? text-generation systems are already being rolled out to question-answering services in familiar search engines, such as bing and google. we are now seeing attempts to apply generative ai to evidence synthesis with mixed results (qureshi et al., 2023). chatgpt-3.5, launched in november 2022, can generate seemingly plausible pubmed strategies, if prompted with the right question (wang et al., 2023). these strategies would not currently pass through our peer-review checklists, as they can contain serious errors, such as subject headings that do not actually exist in mesh (wang et al., 2023).   we should not, however, over-emphasize the fact that strategies generated by chatgpt-3.5 are currently “unusable” (qureshi et al., 2023, p. 2). the use of third-party plug-ins will improve the accuracy of chatgpt-4.0, which is currently available to subscribers. generative ai will probably be incorporated into bibliographic databases, with trials imminent in scopus and web of science, among others (van noorden, 2023). the technology is going to improve massively and very quickly!   as large language models are fundamentally based on prediction, the quality of their training data is vitally important. we know, however, that many published systematic reviews are based on low-quality searches (de kock et al., 2020). any strategy generated from this poor data is likely to be flawed. longer term, we should take steps to ensure that ai systems are learning from high-quality training data. the most effective ai systems for use in evidence synthesis will be those that incorporate the recommendations from the international collaboration for the automation of systematic reviews (icasr) (beller et al., 2018).   ai cannot be ignored and so information specialists must be ready to lead the transformation of working practices. it is probable that we will see human-in-the-loop systems develop, rather than purely automated evidence syntheses. we should grasp the opportunity to expand our roles into troubleshooting, user education, evaluation of sources, and validation of results.   people   several chapters in our book explored the benefits of effective communication and collaboration, and these skills have never been as important as they were during the covid-19 pandemic.   collaboration   effective collaboration might be between information specialists and the wider team, or it might feature groups of information professionals in local, national, and international networks (waffenschmidt & hausner, 2019). collaboration between information specialists was particularly valuable during covid-19, which caroline de brún (2022) has helpfully summarized as involving:   ·        supporting other librarians ·        reducing duplication of effort ·        sharing best practices ·        problem solving and local support   these principles were demonstrated by the librarian reserve corps (lrc), a voluntary network of medical, health sciences, and public health librarians, who came together to provide an evidence-based response to the international emergency (callaway, 2021). the lrc has published a valuable guide that draws on the lessons of covid-19 to guide searching during future emergencies (brody et al., 2023). as another example, the european association for health information and libraries (eahil) evidence based information special interest group (ebi-sig) is working on a project to create a living open access library of search strategy resources (ebi-sig, 2023). we have also seen the launch of the searchrxiv website for sharing, archiving and identifying search strategies (cabi digital library, 2023), which demonstrates the value of large-scale collaborative efforts.   networking and developing relationships are vital for keeping up to date with other information specialists and organizations.   communication   communication is a key skill for facilitating collaboration. many of us have been using zoom, microsoft teams, and other platforms far more than we had ever envisaged five years ago. this certainly helped to develop international links. we ought not overlook how these networks helped to overcome isolation and promote wellbeing in a time of lockdowns and other pandemic restrictions (de brún, 2022).   communication skills are as important as technical knowledge in training to become an information specialist (levay et al., 2022b). we must be able to demonstrate that our searches are reliable, transparent, and reusable (sampson, 2019). the preferred reporting items for systematic reviews and meta-analyses literature search extension (prisma-s) is a guide to reporting search histories in a replicable and transparent way (rethlefsen et al., 2021). the checklist is also a useful communication tool, as it encourages us to share our work clearly and completely. we would encourage all information specialists to integrate prisma-s into their processes.   skills and training   skills and training were a central theme in our book to ensure information specialists could lead change and drive improvements. the evidence on the value of expert searchers has continued to accumulate since then (ramirez et al., 2022).   information specialists need to develop the skills to promote equality, diversity, and inclusion through our work in systematic reviewing. for example, with appropriate experience, writing a search protocol is an opportunity to help tackle health inequalities, if we select appropriate data sources, ensure the search strategies cover diverse populations, and deal with underrepresentation in the literature (naicker, 2022). important work has been done to show how we might discuss outdated, discriminatory, and other potentially inappropriate terminology with review teams, and then consider how we should report these search terms carefully when they are required in a strategy (townsend et al., 2022).   there are opportunities to learn from the experiences of data librarians, who are involved with managing the research lifecycle, data curation, data analysis, and visualization (ashiq & warraich, 2022). data management skills would be useful for managing data generated during a systematic review, incorporating real-world evidence, advocating for open data, and gaining a better understanding of technical challenges.   in addition, all information specialists would benefit from acquiring data literacy and computational-thinking skills to help us to solve problems with technology. having this awareness opens up new opportunities to introduce technology into our work, through learning how to code, understanding programming languages, developing apps, or using packages from github. bibliometrix (aria & cuccurullo, 2017) and similar packages perform well-established tasks quickly and effectively with powerful results, such as identifying phrases in a set of text, checking citation networks, and topic modelling. evaluating these tools and knowing how and when to use them appropriately will be essential for everyone, including those who do not learn to code themselves.   the purpose of developing these skills is to improve systematic reviews through the value information specialists add to teams. it is not just about doing the searches, it is about educating review teams on automation, showing them how to deploy technology and improving their processes. we have the skills and experience to be agents of cultural change as we help teams to integrate ai and other technology into existing processes. we will do this effectively where we can show that the new ways of working uphold the values of accuracy, transparency, and accountability (arno et al., 2021).   conclusion   the covid-19 pandemic has clearly had a significant impact on the methods and technology we use for systematic searching, accelerating some trends and introducing new challenges. in terms of methods, searching for evidence on covid-19 has focussed on living reviews and guidelines, the use of mechanistic evidence, and new sources, such as preprints. the increasing volume and complexity of evidence necessitates better use of technology, such as study-based registers, machine-learning classifiers, and visualization software. international collaboration was valuable during the pandemic, and it was facilitated through good communication. we can promote equality, diversity, and inclusion through our searches. new skills, such as data management and coding, will become increasingly valuable. automation will not result in our spending less time on systematic searching, but it may change how we focus our efforts.   fundamentally, we stand by the conclusion we drew in our book, systematic searching: practical ideas for improving results, that these methods and technologies will only be deployed effectively if information specialists are involved and set the agenda. the five years since 2019 have shown that systematic searchers need to be flexible, creative, and at the forefront of innovation: we expect these trends will intensify in the coming years.   author contributions   paul levay: conceptualization (equal), writing – original draft (lead), writing – review and editing (equal) jenny craven: conceptualization (equal), writing – original draft (supporting),writing – review and editing (equal)   acknowledgements   the authors would like to thank amy finnegan, tom hudson, catherine jacob, caroline miller, marion spring, nicola walsh, and riz zafar for their comments on earlier versions of this paper.   disclaimer   the views expressed in this paper are those of the authors and not necessarily those of the national institute for health and care excellence (nice).   references   aria, m., & cuccurullo, c. (2017). bibliometrix: an r-tool for comprehensive science mapping analysis. journal of informetrics, 11(4), 959–975. https://doi.org/10.1016/j.joi.2017.08.007   arno, a., elliott, j., wallace, b., turner, t., & thomas, j. (2021). the views of health guideline developers on the use of automation in health evidence synthesis. systematic reviews, 10(1), 16. https://doi.org/10.1186/s13643-020-01569-2   ashiq, m., & warraich, n. f. (2022). a systematized review on data librarianship literature: current services, challenges, skills, and motivational factors. journal of librarianship and information science, 55(2), 414–433. https://doi.org/10.1177/09610006221083675   beller, e., clark, j., tsafnat, g., adams, c., diehl, h., lund, h., ouzzani, m., thayer, k., thomas, j., turner, t., xia, j., robinson, k., & glasziou, p. (2018). making progress with the automation of systematic reviews: principles of the international collaboration for the automation of systematic reviews (icasr). systematic reviews, 7(1), 77. https://doi.org/10.1186/s13643-018-0740-7   brierley, l., nanni, f., polka, j. k., dey, g., pálfy, m., fraser, n., & coates, j. a. (2022). tracking changes between preprint posting and journal publication during a pandemic. plos biology, 20(2), e3001285. https://www.ncbi.nlm.nih.gov/pmc/articles/pmc8806067/   briscoe, s., abbott, r., & melendez‐torres, g. j. (2022). expert searchers identified time, team, technology and tension as challenges when carrying out supplementary searches for systematic reviews: a thematic network analysis. health information & libraries journal. advance online publication. https://doi.org/10.1111/hir.12468   brody, s., loree, s., sampson, m., mensinkai, s., coffman, j., mueller, m., askin, n., hamill, c., wilson, e., mcateer, m. b., & staines, h. (2023). searching for evidence in public health emergencies: a white paper of best practices. journal of the medical library association, 111(1), 566–578. https://doi.org/10.5195/jmla.2023.1530   butcher, r., sampson, m., couban, r. j., malin, j. e., loree, s., & brody, s. (2022). the currency and completeness of specialized databases of covid-19 publications. journal of clinical epidemiology, 147, 52–59. https://doi.org/10.1016/j.jclinepi.2022.03.006   cabi digital library. (2023). searchrxiv. retrieved october 12, 2023, from https://www.cabidigitallibrary.org/journal/searchrxiv   callaway, j. (2021). the librarian reserve corps: an emergency response. medical reference services quarterly, 40(1), 90–102. https://doi.org/10.1080/02763869.2021.1873627   chappell, m., edwards, m., watkins, d., marshall, c., & graziadio, s. (2023). machine learning for accelerating screening in evidence reviews. cochrane evidence synthesis and methods, 1(5), e12021. https://doi.org/10.1002/cesm.12021   chen, y. y., bullard, j., & giustini, d. (2023). automated indexing using nlm's medical text indexer (mti) compared to human indexing in medline: a pilot study. journal of the medical library association, 111(3), 684–695. https://doi.org/10.5195/jmla.2023.1588   clyne, b., walsh, k. a., o'murchu, e., sharp, m. k., comber, l., o’ brien, k. k., smith, s. m., harrington, p., o'neill, m., teljeur, c., & ryan, m. (2021). using preprints in evidence synthesis: commentary on experience during the covid-19 pandemic. journal of clinical epidemiology, 138, 203–210. https://doi.org/10.1016/j.jclinepi.2021.05.010   cooper, c., booth, a., husk, k., lovell, r., frost, j., schauberger, u., britten, n., & garside, r. (2022). a tailored approach: a model for literature searching in complex systematic reviews. journal of information science. advance online publication. https://doi.org/10.1177/01655515221114452   de brún, c. (2022, june 1–3). knowledge makes the world go round: librarians working together to fight the covid infodemic [poster session]. european association for health information and libraries, rotterdam, netherlands.   de kock, s., stirk, l., ross, j., duffy, s., noake, c., & misso, k. (2020). systematic review search methods evaluated using the preferred reporting of items for systematic reviews and meta-analyses and the risk of bias in systematic reviews tool. international journal of technology assessment in health care, 37(1), e18. https://doi.org/10.1017/s0266462320002135   ebi-sig. (2023). library of search strategy resources. retrieved october 12, 2023, from https://sites.google.com/view/searchresourceslib/home   el mikati, i. k., khabsa, j., harb, t., khamis, m., agarwal, a., pardo-hernandez, h., farran, s., khamis, a. m., el zein, o., el-khoury, r., schünemann, h. j., & akl, e. a. (2022). a framework for the development of living practice guidelines in health care. annals of internal medicine, 175(8), 1154-1160. https://doi.org/10.7326/m22-0514   elliott, j. h., synnot, a., turner, t., simmonds, m., akl, e. a., mcdonald, s., salanti, g., meerpohl, j., maclehose, h., hilton, j., tovey, d., shemilt, i., & thomas, j. (2017). living systematic review: 1. introduction—the why, what, when, and how. journal of clinical epidemiology, 91, 23–30. https://doi.org/10.1016/j.jclinepi.2017.08.010   foster, m. j., & jewell, s. t. (2022). piecing together systematic reviews and other evidence syntheses. rowman & littlefield.   greenhalgh, t., fisman, d., cane, d. j., oliver, m., & macintyre, c. r. (2022). adapt or die: how the pandemic made the shift from ebm to ebm+ more urgent. bmj evidence-based medicine, 27(5), 253–260. https://doi.org/10.1136/bmjebm-2022-111952   greenhalgh, t., & peacock, r. (2005). effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. bmj, 331(7524), 1064–1065. https://doi.org/10.1136/bmj.38636.593461.68   johnson, e. e., o’keefe, h., sutton, a., & marshall, c. (2022). the systematic review toolbox: keeping up to date with tools to support evidence synthesis. systematic reviews, 11(1), 258. https://doi.org/10.1186/s13643-022-02122-z   khalil, h., tamara, l., rada, g., & akl, e. a. (2021). challenges of evidence synthesis during the 2020 covid pandemic: a scoping review. journal of clinical epidemiology, 142, 10–18. https://doi.org/10.1016/j.jclinepi.2021.10.017   kirkham, j. j., penfold, n. c., murphy, f., boutron, i., ioannidis, j. p., polka, j., & moher, d. (2020). systematic examination of preprint platforms for use in the medical and biomedical sciences setting. bmj open, 10(12), e041849. https://doi.org/10.1136/bmjopen-2020-041849   levay, p., & craven, j. (2019). conclusion: where do we go from here? in p. levay, & j. craven (eds.), systematic searching: practical ideas for improving results (pp. 289–292). facet publishing.   levay, p., heath, a., & tuvey, d. (2022a). efficient searching for nice public health guidelines: would using fewer sources still find the evidence? research synthesis methods, 13(6), 760–789. https://doi.org/10.1002/jrsm.1577   levay, p., walsh, n., & foster, l. (2022b). the national institute for health and care excellence information specialist development pathway: developing the skills, knowledge and confidence to quality assure search strategies. health information and libraries journal, 39(4), 392–399. https://doi.org/10.1111/hir.12460   macfarlane, a., russell-rose, t., & shokraneh, f. (2022). search strategy formulation for systematic reviews: issues, challenges and opportunities. intelligent systems with applications, 15, 200091. https://doi.org/10.1016/j.iswa.2022.200091   mcdonald, s., sharp, s., morgan, r. l., murad, m. h., & fraile navarro, d. (2023). methods for living guidelines: early guidance based on practical experience. paper 4: search methods and approaches for living guidelines. journal of clinical epidemiology, 155, 108–117. https://doi.org/10.1016/j.jclinepi.2022.12.023   metzendorf, m., & featherstone, r. m. (2021). evaluation of the comprehensiveness, accuracy and currency of the cochrane covid-19 study register for supporting rapid evidence synthesis production. research synthesis methods, 12(5), 607–617. https://doi.org/10.1002/jrsm.1501   metzendorf, m., weibel, s., reis, s., & mcdonald, s. (2022). pragmatic and open science-based solution to a current problem in the reporting of living systematic reviews. bmj evidence-based medicine, 8(4), 267–272. https://doi.org/10.1136/bmjebm-2022-112019   naicker, r. (2022). critically appraising for antiracism. education for information, 38(4), 291–308. https://doi.org/10.3233/efi-220052   national library of medicine. (2022). indexing faqs. retrieved october 12, 2023, from https://support.nlm.nih.gov/knowledgebase/article/ka-05326/en-us   o’mara-eves, a., thomas, j., mcnaught, j., miwa, m., & ananiadou, s. (2015). using text mining for study identification in systematic reviews: a systematic review of current approaches. systematic reviews, 4(1), 5. https://doi.org/10.1186/2046-4053-4-5   pierre, o., riveros, c., charpy, s., & boutron, i. (2021). secondary electronic sources demonstrated very good sensitivity for identifying studies evaluating interventions for covid-19. journal of clinical epidemiology, 141, 46–53. https://doi.org/10.1016/j.jclinepi.2021.09.022   qureshi, r., shaughnessy, d., gill, k. a. r., robinson, k. a., li, t., & agai, e. (2023). are chatgpt and large language models “the answer” to bringing us closer to systematic review automation? systematic reviews, 12(1), 72. https://doi.org/10.1186/s13643-023-02243-z   ramirez, d., foster, m. j., kogut, a., & xiao, d. (2022). adherence to systematic review standards: impact of librarian involvement in campbell collaboration's education reviews. the journal of academic librarianship, 48(5), 102567. https://doi.org/10.1016/j.acalib.2022.102567   rethlefsen, m. l., kirtley, s., waffenschmidt, s., ayala, a. p., moher, d., page, m. j., & koffel, j. b. (2021). prisma-s: an extension to the prisma statement for reporting literature searches in systematic reviews. systematic reviews, 10(1), 39. https://doi.org/10.1186/s13643-020-01542-z   rosonovski, s., levchenko, m., ide‐smith, m., faulk, l., harrison, m., & mcentyre, j. (2023). searching and evaluating publications and preprints using europe pmc. current protocols, 3(3), e694. https://doi.org/10.1002/cpz1.694   sampson, m. (2019). communication for information specialists. in p. levay, & j. craven (eds.), systematic searching: practical ideas for improving results (pp. 249–268). facet publishing.   shemilt, i., noel-storr, a., thomas, j., featherstone, r., & mavergames, c. (2022). machine learning reduced workload for the cochrane covid-19 study register: development and evaluation of the cochrane covid-19 study classifier. systematic reviews, 11(1), 15. https://doi.org/10.1186/s13643-021-01880-6   shokraneh, f., & adams, c. e. (2019). study-based registers reduce waste in systematic reviewing: discussion and case report. systematic reviews, 8(1), 129. https://doi.org/10.1186/s13643-019-1035-3   stansfield, c., stokes, g., & thomas, j. (2022). applying machine classifiers to update searches: analysis from two case studies. research synthesis methods, 13(1), 121–133. https://doi.org/10.1002/jrsm.1537   svarre, t., & russell-rose, t. (2022). think outside the search box: a comparative study of visual and form-based query builders. journal of information science. advance online publication. https://doi.org/10.1177/01655515221138536   thomas, j., mcdonald, s., noel-storr, a., shemilt, i., elliott, j., mavergames, c., & marshall, i. j. (2021). machine learning reduced workload with minimal risk of missing studies: development and evaluation of a randomized controlled trial classifier for cochrane reviews. journal of clinical epidemiology, 133, 140–151. https://doi.org/10.1016/j.jclinepi.2020.11.003   townsend, w., anderson, p., haines, k., hansen, s., james, l., maceachern, m., rana, g., saylor, k. & saylor, k. (2022). addressing antiquated, non-standard, exclusionary, and potentially offensive terms in evidence syntheses and systematic searches. retrieved october 12, 2023, from https://doi.org/10.7302/6408   van noorden, r. (2023). chatgpt-like ais are coming to major science search engines. nature, 620(7973), 258. retrieved from https://www.nature.com/articles/d41586-023-02470-3   verdugo-paiva, f., vergara, c., ávila, c., castro-guevara, j., cid, j., contreras, v., jara, i., jiménez, v., lee, m. h., muñoz, m., rojas-gómez, a. m., rosón-rodríguez, p., serrano-arévalo, k., silva-ruz, i., vásquez-laval, j., zambrano-achig, p., zavadzki, g., & rada, g. (2022). covid-19 living overview of evidence repository is highly comprehensive and can be used as a single source for covid-19 studies. journal of clinical epidemiology, 149, 195–202. https://doi.org/10.1016/j.jclinepi.2022.05.001   waffenschmidt, s., & hausner, e. (2019). collaborative working to improve searching. in p. levay, & j. craven (eds.), systematic searching: practical ideas for improving results (pp. 229–248). facet publishing.   wang, s., scells, h., koopman, b., & zuccon, g. (2023). can chatgpt write a good boolean query for systematic review literature search? sigir '23: proceedings of the 46th international acm sigir conference on research and development in information retrieval, taipei. 1426–1436. https://doi.org/10.1145/3539618.3591703     editorial editorial   evidence summary theme: professional issues   heather macdonald associate editor (evidence summaries) health and biosciences librarian carleton university ottawa, ontario, canada email: heather.macdonald@carleton.ca    2023 macdonald. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30376     the evidence summaries (es) in this issue focus on the domain of professional issues from koufogiannakis et al.’s (2004) content analysis of publications in librarianship. as librarians, we need to be aware of many aspects of the profession, even if our own positions have a more limited scope. selecting just six articles for es for this issue was both a pleasure (exploring so many areas) and a challenge (limiting to only six).   just as this domain can include a variety of topics, so too are the es diverse in their content and setting. the settings cover public and academic libraries, and hail from the americas to asia. they include critical appraisals of articles that look at diversity, equity, and inclusion in professional development, patron privacy, discussions of librarianship status as a profession, social responsibility and accountability, open access and preservation, and predatory publishing. i hope you enjoy reading and learning from them.   starting with eblip 17(1) we repackaged the es by theme. having covered the six domains from koufogiannakis et al. (2004), we will continue with thematic packaging of the es but will be exploring new themes. stay tuned for more es!   reference   koufogiannakis, d., slater, l., & crumley, e. (2004). a content analysis of librarianship research. journal of information science, 30(3), 227–239. https://doi.org/10.1177/0165551504044668 review article   individualized research consultations in academic libraries: a scoping review of practice and evaluation methods   karine fournier head, reference services health sciences library university of ottawa ottawa, ontario, canada email: karine.fournier@uottawa.ca   lindsey sikora health sciences research liaison librarian health sciences library university of ottawa ottawa, ontario, canada email: lindsey.sikora@uottawa.ca   received: 28 july 2015    accepted: 20 oct. 2015         2015 fournier and sikora. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   introduction – librarians in academic institutions have been providing personalized services to the student population by offering individualized research consultations (irc) for decades. these consultations usually consume many hours of librarians’ busy schedules, and yet the impact of these consultations is unknown. therefore, it’s worth asking the question: what assessment methods have been used in academic libraries to evaluate the impact of irc?   methods – a retrospective scoping review of the literature was performed using the following databases: library and information science abstracts (lisa), educational resources information center (eric), library and information technology abstracts (lista), scopus, and web of science. additionally, a manual search of the included papers reference lists was conducted to locate additional relevant papers. articles that mentioned a format of evaluation or assessment and were based within a library setting were included. articles that discussed group instruction that were not in a library setting, or that did not include any form of evaluation or assessment, were excluded.   results – researchers located 578 articles and reviewed titles and abstracts. 523 titles were eliminated, while full text sources of the remaining 55 were examined to check inclusion and exclusion criteria. 20 articles remained for qualitative synthesis. specific methods of assessment were reviewed and three overall assessment methods were identified: 1) usage statistics, 2) survey, and 3) objective quantitative methods.   conclusion – many articles using a usage statistics method stated that they wanted to further their assessment of individual consultations. several authors using a survey method described the value of the information gathered by surveying their users for improving their service, but also mentioned that this method is subjective in nature. they mentioned that objective assessment methods would provide a better understanding of the impact of ircs. the few articles using objective quantitative methods obtained mixed results. overall, more research in the assessment of ircs is needed, particularly those with objective quantitative methods. introduction   the purpose of reference services in academic libraries has always been to help users with their research endeavors. the manner in which help is provided differs from institution to institution and has evolved over time. literature shows that in order to provide library patrons with help and guidance, librarians in most academic institutions have been transitioning away from a service point, such as the reference desk, into more specialized and advanced assistance through referrals made by library support staff (arndt, 2010), or by the delivery of individualized consultation services to users. studies have shown that staffing reference desks or one-point service desks with library support staff has been efficient; one study determined that 89% of questions could be efficiently answered by non-librarians (ryan, 2008).   individualized research consultation (irc) services have had many names over the years: “term paper clinics”, “term paper counseling”, “research sessions”, “term paper advisory service”, “personalized research clinics”, “research assistance programs”, “individualized instruction” and so on. essentially, an irc is a one-on-one instructional session between a librarian and a user in order to assess the user’s specific research needs and help them find information. while group instruction is a great way to introduce students to various library skills, individual research consultations allow for more in-depth questions that are specific to a student’s information needs. one advantage that this type of service provides over traditional reference services is that it gives “students the individualized attention and serves them at their points of need” (yi, 2003, p. 343).   aim   academic librarians can spend many hours helping individuals with their research projects. while research has examined the ways in which information literacy (il) skills have been taught in the classroom, research conducted for one-on-one consultations is reported less frequently. with this observation, a scoping review seemed appropriate to further enhance our knowledge of irc assessment methods. as arksey and o'malley (2005) stated:   scoping studies might aim to map rapidly the key concepts underpinning a research area and the main sources and types of evidence available, and can be undertaken as stand-alone projects in their own right, especially where an area is complex or has not been reviewed comprehensively before. (p. 21)   colquhoun et al. (2014) further formalize the definition of scoping reviews:   a scoping review or scoping study is a form of knowledge synthesis that addresses an exploratory research question aimed at mapping key concepts, types of evidence, and gaps in research related to a defined area or field by systematically searching, selecting, and synthesizing existing knowledge. this definition builds on the descriptions of arksey and o’malley, and daudt to provide a clear definition of the methodology while describing the key characteristics that make scoping reviews distinct from other forms of syntheses. (p.1292) this scoping review attempts to answer the following question: which evaluation methods are used to measure impact and improve individualized research consultations in academic libraries?    method   researchers conducted a systematic search of the following databases for the years 1990-2013: library and information science abstracts (lisa), educational resources information center (eric), library and information technology abstracts (lista), scopus, and web of science. search terms included: individual consultation, research consultation, one-on-one consultations, research clinics, personalized research, and evaluation, assessment, impact, with combinations of these terms. researchers used the thesauri of individual databases alongside keyword searching.   using a manual search of the included articles’ reference lists, the authors located additional relevant articles. some articles were found while searching the reference lists, dating back further than our original search date range. as they were key first articles on the topic and answered our inclusion criteria, we decided to keep them. due to constraints with acquiring proper translation, we only included articles written in english or french, with english abstracts of articles in other languages assessed, if available. we searched google scholar for online grey literature in hopes of locating unpublished studies and other reports exploring individualized research consultations, with little additional information found.   we included descriptive articles, qualitative and quantitative studies, single case studies, and review articles if they discussed evaluating or assessing individual consultations. we excluded book chapters, policy papers and documents, commentaries, essays, and non-published theses, as these types of documents did not address evaluating/assessing ircs as primary studies. we included articles that discussed individualized research consultations at the undergraduate or graduate level, included some form of evaluation or assessment, and were based within a library setting. we excluded articles that discussed group instruction, were not included in a library setting (such as consultation for profit), and which did not include any form of evaluation or assessment.   both authors conducted data collection and synthesis, and collectively wrote the background, conceptualized the review, undertook the searches, and screened the articles. we each screened the articles, and then compared. we discussed disagreements between the inclusion and exclusion of the articles, and reached a decision for each situation. we both synthesized the data and crafted the findings.   data collected from the reviewed articles included aims of the studies, type of evaluation/assessment involved, procedures and methods used, audience level (e.g., undergraduate, graduate, faculty, etc.), and main findings.   results   the modified prisma flow chart (moher, liberati, tetzlaff, & altman, 2009) in figure 1 demonstrates the number of articles and results from the selection and screening process. researchers identified 543 potential articles through database searching (after duplicates removed), and found an additional 35 articles through cited reference searching of the references lists of the included articles. all titles and abstracts were reviewed, and 523 of the articles did not include any form of evaluation, or mention individual consultations, and were therefore excluded. we examined full text sources of the remaining 55 articles against inclusion and exclusion criteria, leaving 20 articles for our qualitative synthesis (figure 1).       figure 1 modified prisma flow chart     qualitative thematic analysis   this section presents the analysis of the 20 included articles’ extracted data, which are compiled and organized in table 1 for rapid review. an in-depth overview of each study is available in the appendix.     table 1 assessment methods and included articles assessment methods usage statistics survey objective quantitative methods included articles attebury, sprague, & young, 2009 auster, devakos, & meikle, 1994 erickson & warner, 1998 becker, 1993 bean, 1995 donegan, domas, & deosdade, 1989 hoskisson & wentz, 2001 cardwell, furlong, & o'keeffe, 2001 reinsfelder, 2012 lee, 2004 coniglio, 1984   meyer, forbes, & bowers, 2010 debreczeny, 1985 yi, 2003 gale & evans, 2007   gratch & york, 1991 imamoto, 2006 magi & mardeusz, 2013 rothstein, 1990 schobert, 1982 total 6 11 3     we grouped articles into three overall types of assessment methods: usage statistics, survey, or objective quantitative methods. table 1 lists the assessment methods with their affiliated articles. these categories were inspired by attebury, sprague and young (2009), who previously classified such articles in two main categories: “surveys as evaluation tools have been a popular means of assessment […] while for other authors analysis of statistics and writing about their program has served as a useful mean of evaluation” (p. 209).   table 2 summarizes the specific methods of assessment, which articles they are correlated with, and the number of students surveyed/tested, if applicable. where usage statistics were used, the type of statistics gathered include number of students encountered, number of hours or sessions provided, librarian’s preparation time, length of the meeting, student’s affiliation (e.g., department or program), reason for requesting an appointment (e.g., course-related, paper, dissertation, etc.), and student’s gender. when a survey was used, methods included the use of an evaluation form completed by patrons following the appointment, surveys sent to users of the service weeks or months later, and use of an evaluation form completed by the service’s provider (e.g., librarian, mls student). when the assessment was via an objective quantitative method, all three articles developed their own unique assessment methods to evaluate individual research consultations. in these cases, the methods all include a certain level of objectivity, as opposed to the more subjective nature found with assessment using surveys. the methods included assessment of assigned database searches performed by the patron, information literacy skills test (multiple choice questions), and citation analysis of the draft and final papers’ bibliographies from the students.     table 2 specific methods of assessment used assessment methods specific methods of assessment articles no. of students surveyed or tested usage statistics 1.        usage statistics compilation stats acquired: n/a no. of students seen attebury, sprague, & young, 2009 yi, 2003 no. of hours or sessions provided attebury, sprague, & young, 2009 becker, 1993 hoskisson & wentz, 2001 lee, 2004 meyer, forbes, & bowers, 2010 yi, 2003 librarian’s preparation time attebury, sprague, & young, 2009 lee, 2004 length of meeting attebury, sprague, & young, 2009 yi, 2003 student affiliation (dept., or program) lee, 2004 reason for request (course-related, paper, dissertation, etc.) attebury, sprague, & young, 2009 yi, 2003 student’s gender attebury, sprague, & young, 2009 survey 1.        evaluation form filled by patrons after individual consultation auster, devakos, & meikle, 1994 39 bean, 1995 27 gale & evans, 2007 23 magi & mardeusz, 2013 52 imamoto, 2006 95 rothstein, 1990 77 2.        surveys sent to users of the service weeks or months later cardwell, furlong, & o'keeffe, 2001 16 25 coniglio, 1984 57 debreczeny, 1985 60 gratch & york, 1991 17 shobert, 1982 19 3.        evaluation form to be filled by the service’s provider (librarian, mls student) auster, devakos, & meikle, 1994 n/a bean, 1995 45 gale & evans, 2007 n/a rothstein, 1990 n/a objective quantitative methods 1.        assessment of assigned database searches performed by the patron erickson & warner, 1998 31 2.        information literacy skills test (multiple choice questions) donegan, domas, & deosdade, 1989 156 3.        citation analysis of students draft and final papers’ bibliographies reinsfelder, 2012 103     table 3 populations that used irc services per assessment method populations no. of articles using usage statistics no. of articles using a survey no. of articles using objective quantitative methods total no. of articles undergraduate students only 1 5 2 8 graduate students only   1 1 2 undergraduate et graduate students 5 5   10 faculty members or researchers (as an additional population) 3 1   4     additionally, we have determined how many papers provided irc services to undergraduate students, graduate students, or both, along with faculty members or researchers, as shown in table 3. many articles mentioned having a specific population in mind when starting a service, but ended up serving additional populations. ten articles described serving both the undergraduate and graduate population, and eight articles the undergraduate students only. two articles were evaluating a service offered to graduate students only.   most of the articles included in the detailed analysis are not “studies” per se, but rather a description of library services. therefore, many of those articles do not use the intervention/comparison/outcome format. table 4 is an attempt to categorize the extracted data using these categories, with the presumption that this is an interpretation exercise. the intervention is the assessment method used in the selected articles, the comparison is listed if used in the included articles, and the outcome is an overall summary of the benefits and outcomes for each assessment method.     table 4 interventions, comparisons, and outcomes intervention comparison outcome usage statistics n/a usage statistics method allows for an in-depth analysis of how the service is used and can contribute to decision-making for the future or the modification of the service. anecdotal comments are heavily used throughout the included articles and were a large part of the service’ performance analysis. survey n/a surveys were used to mainly acquire information on users’ satisfaction. other information of interest for survey’s creators related to the service’s marketing, and users’ affiliation. all of the included articles had positive feedback (satisfaction level) from their users. objective quantitative methods: specific interventions individualized medline tutorial (erickson & warner, 1998) 1) group tutorial 2) control group: no tutorial no statistically significance differences were found for the search duration, the quantity of articles retrieved, the recall, and the precision rate. there were many limitations to the study, such as a small group of participants, low compliance rates, and a change in the database platform at the study’s mid-point. participants felt satisfied with their searches, and were interested in improving their medline search skills. the authors concluded that time constraints is a major obstacle for information professionals to provide individual tutorials to hundreds of residents, who themselves struggle to free some time from their busy hospital schedule to receive adequate database search skills training. term paper counselling (tpc) (donegan, domas, & deosdade, 1989) 1) group instruction session 2) control group: no instruction statistically significant differences appeared between tpc and the control group, and between group instructions and the control group. no statically significant differences were found between tpc and group instruction. the authors concluded that either type of intervention (group or individual) is appropriate when teaching basic library search skills. individualized consultation (reinsfelder, 2012) 1) control group: no consultation using citation analysis and comparing students’ draft and final papers using a rating scale allowed the author to run nonparametric statistical tests. statistically significant differences were found between draft and final papers for the experimental group, but no significant difference were found for the control group. the author concluded that students benefited from an individualized consultation and showed an improvement in their sources’ quality, relevance, currency, and scope.     discussion   several articles relied heavily on usage statistics to assess their individualized research consultation (irc) services. whether the number of students seen, or the number of hours librarians spent preparing or providing the consultations, these statistics can tell us how this service is used by the student population, but they do not describe the impact of such services except when anecdotal comments from users are recorded. in addition to usage statistics, attebury, sprague, & young (2009), as well as yi (2003), gathered and analyzed information about the content of ircs. yi noted that the most frequent themes discussed during an irc were “topic assistance”, “search skills”, and “database selection”, and these are just some of the elements covered in class presentations. this suggests that irc could benefit from a better alignment with information literacy standards to develop students’ information literacy skills. overall, many articles using this method mentioned the need to further the assessment of irc beyond usage statistical analysis. attebury, sprague & young (2009), mention their intention of collecting information on student satisfaction to help evaluate and improve their service on a continuous basis.   the set of articles relying on survey methods is large, and some date back to the 1980’s. most surveys or evaluation forms focus on user satisfaction, and many authors suggest this gives an indication of the success of the service provided and helps to adjust the service delivery if needed. however, this is still based on user sense of satisfaction and not on actual performance outcomes. cardwell, furlong & o’keeffe (2001) indicate that “because of the nature of [personalized research clinics], it is difficult to truly assess student learning and isolate the long-term impact that an individual session has on a student’s knowledge and skills” (p. 108). also, as stated by several authors, but well-phrased by shobert (1984), “evaluating a project like this objectively is nearly impossible. there is a built-in bias in its favor: where there was nothing, suddenly there is an individualized instruction program; the responses are bound to be positive” (p.149). whether it is a new service or not, providing tailored individual help to students will always be appreciated, which skews user satisfaction in survey results. recently, magi and mardeusz (2013) surveyed students on their feelings before and after the individual consultation, and the comments they received demonstrate the value of individual consultation. it relieves students’ anxiety, and instead of feeling overwhelmed, students felt encouraged and more focused. the psychological well-being of students is less frequently studied in relation to the impact of individual consultations, but this study demonstrated a less obvious impact, one certainly worth mentioning when it comes to the value of the time spent with students individually.   as stated earlier, shobert (1984) mentioned that it would be nearly impossible to objectively evaluate an individual research consultation service. the three articles using objective quantitative methods have attempted to do just that by measuring, in an objective manner, the impact individualized research consultations have on student’s information literacy skills. they all have taken different paths to evaluate this impact. results were unsuccessful in demonstrating a statistically significant difference on the impact of individual consultation between group instruction and term paper counselling (tpc) (donegan, domas, & deosdade, 1989), and between getting an individual tutorial or not (erickson & warner, 1998). these authors explained that many reasons could account for these results, such as low compliance at performing the tasks requested, and test validity and reliability. in the donegan, domas, and deosdade (1989) study, results showed no statistically significant difference between group instruction and term paper clinics. this study focused on introductory material, such as that usually taught in a first year undergraduate class. one could venture to say that basic library skills can easily be provided to students in a group setting, and that perhaps individual consultations are more appropriate for advance skills development. reinsfelder (2012) found a statistically significant difference in his study, which he concludes “[provides] some quantitative evidence demonstrating the positive impact of individual research consultation” (p. 263). he also stated that “librarians were frequently able to make more meaningful connections with students by addressing the specific needs of each individual” (p. 275), which speaks to the very nature of individual research consultations.   our scoping review was not without its limitations. firstly, our review is only descriptive, with limited information to quantify our findings. further research would be required to assess the impact of individualized research consultations to correctly identify specific methods that increase the searcher’s success. secondly, none of the articles included in our study were critically appraised, limiting the reproducibility, completeness, and transparency of reporting the methods and results of our scoping review. however, as there is already limited information available regarding ircs, we did not want to exclude any articles on the topic. future research should focus on quantifying the impact of individualized research consultations. as our scoping review demonstrates, we were only able to find three studies that used objective quantitative assessment methods. not only will gathering more quantitative information further inform ircs’ practices, but it will also complement the descriptive information obtained from surveys and usage statistics. it should be noted that there are different methods that may need to be considered when examining ircs between disciplines. further research should also examine these differences, attempting to find the best methods for individual disciplines. lastly, a more in-depth examination of the evaluation of the quality of the studies that we found should be undertaken.   conclusion   our research question asked, which evaluation methods are used to measure impact and improve ircs in academic libraries? we were able to identify usage statistics and surveys as the main methods of assessment used to evaluate ircs. in addition, three articles attempted to objectively and quantitatively measure the impact of individual consultations. this amounts to very few studies compared to the wealth of articles on the assessment of group instruction.   individual research consultations have been around for decades and help students at various stages of their research activities. providing this personalized service one-on-one is time consuming for librarians, and should be better acknowledged and assessed.   future research should address the need for more objective assessment methods of studies on ircs. in combination with usage statistics and surveys, objective quantitative studies would yield a greater quality of evaluation for ircs. furthermore, as these evaluation methods become more valid, a closer inspection of ircs across disciplines could be explored with greater success.   references   arksey, h., & o'malley, l. (2005). scoping studies: towards a methodological framework. international journal of social research methodology, 8(1), 19-32. http://dx.doi.org/10.1080/1364557032000119616   arndt, t. s. (2010). reference service without the desk. reference services review, 38(1), 71-80. http://dx.doi.org/10.1108/00907321011020734   attebury, r., sprague, n., & young, n. j. (2009). a decade of personalized research assistance. reference services review, 37(2), 207-220. http://dx.doi.org/10.1108/00907320910957233   auster, e., devakos, r., & meikle, s. (1994). individualized instruction for undergraduates: term paper clinic staffed by mls students. college & research libraries, 55(6), 550-561. http://dx.doi.org/10.5860/crl_55_06_550   bean, r. (1995). two heads are better than one: depaul university's research consultation service. in c. j. jacob (ed.), the seventh off-campus library services conference proceedings, (pp. 5-16). mount pleasant, mi: central michigan university. http://gcls2.cmich.edu/conference/past_proceedings/7thoclscp.pdf   becker, k. a. (1993). individual library research clinics for college freshmen. research strategies, 11(4), 202-210.   cardwell, c., furlong, k., & o'keeffe, j. (2001). my librarian: personalized research clinics and the academic library. research strategies, 18(2), 97-111. http://dx.doi.org/10.1016/s0734-3310(02)00072-1 colquhoun, h. l., levac, d., o'brien, k. k., straus, s., tricco, a. c., perrier, l., … moher, d. (2014). scoping reviews: time for clarity in definition, methods, and reporting. journal of clinical epidemiology, 67(12), 1291-1294. http://dx.doi.org/10.1016/j.jclinepi.2014.03.013   coniglio, j. w. (1984). bibliographic counseling: the term paper advisory service. show-me libraries, 36(12), 79-82.   debreczeny, g. (1985). coping with numbers: undergraduates and individualized term paper consultations. research strategies, 3(4), 156-163.   donegan, p. m, domas, r. e., & deosdade, j. r. (1989). the comparable effects of term paper counseling and group instruction sessions. college & research libraries, 50(2), 195-205. http://dx.doi.org/10.5860/crl_50_02_195   erickson, s. s., & warner, e. r. (1998). the impact of an individual tutorial session on medline use among obstetrics and gynaecology residents in an academic training programme: a randomized trial. medical education, 32(3), 269-273. http://dx.doi.org/10.1046/j.1365-2923.1998.00229.x   gale, c. d., & evans, b. s. (2007). face-to-face: the implementation and analysis of a research consultation service. college & undergraduate libraries, 14(3), 85-101. http://dx.doi.org/10.1300/j106v14n03_06   gratch, b. g., & york, c. c. (1991). personalized research consultation service for graduate students: building a program based on research findings. research strategies, 9(1), 4-15.   haynes, r. b., mckibbon, k. a., walker, c. j., ryan, n., fitzgerald, d., & ramsden, m. f. (1990). online access to medline in clinical settings. a study of use and usefulness. annals of internal medicine, 112(1), 78-84. http://dx.doi.org/10.7326/0003-4819-112-1-78   hoskisson, t., & wentz, d. (2001). simplifying electronic reference: a hybrid approach to one-on-one consultation. college and undergraduate libraries, 8(2), 89-102. http://dx.doi.org/10.1300/j106v08n02_09   imamoto, b. (2006). evaluating the drop-in center: a service for undergraduates. public services quarterly, 2(2), 3-18. http://dx.doi.org/10.1300/j295v02n02_02   lee, d. (2004). research consultations: enhancing library research skills. reference librarian, 41(85), 169-180. http://dx.doi.org/10.1300/j120v41n85_13   magi, t. j., & mardeusz, p. e. (2013). why some students continue to value individual, face-to-face research consultations in a technology-rich world. college & research libraries, 74(6), 605-618. http://dx.doi.org/10.5860/crl12-363   meyer, e., forbes, c., & bowers, j. (2010). the research center: creating an environment for interactive research consultations. reference services review, 38(1), 57-70. http://dx.doi.org/10.1108/00907321011020725   moher, d., liberati, a., tetzlaff, j., & altman, d. g. (2009). preferred reporting items for systematic reviews and metaanalyses: the prisma statement. journal of clinical epidemiology, 62(10), 1006-1012. http://dx.doi.org/10.1016/j.jclinepi.2009.06.005   reinsfelder, t. l. (2012). citation analysis as a tool to measure the impact of individual research consultations. college & research libraries, 73(3), 263-277. http://dx.doi.org/10.5860/crl-261   rothstein, s. (1990). point of need/maximum service: an experiment in library instruction. reference librarian, 11(25/26), 253-284. http://dx.doi.org/10.1300/j120v11n25_11   ryan, s. m. (2008). reference transactions analysis: the cost-effectiveness of staffing a traditional academic reference desk. journal of academic librarianship, 34(5), 389-399. http://dx.doi.org/10.1016/j.acalib.2008.06.002   schobert, t. (1982). term-paper counseling: individualized bibliographic instruction. rq, 22(2), 146-151.   yi, h. (2003). individual research consultation service: an important part of an information literacy program. reference services review, 31(4), 342-350. http://dx.doi.org/10.1108/00907320310505636     appendix descriptive summary analysis of included articles   this appendix presents the included articles’ extracted data as an in-depth review of each study, organized by the three categories used in this review: usage statistics, survey, and objective quantitative methods.   usage statistics   attebury, sprague, & young, 2009; university of idaho library   the university of idaho library created a “research assistance program” (rap) for its patrons in 1998. usage statistics compiled over 10 years were used to determine if the service still effectively met the needs of its users. using quantitative and qualitative data, the authors examined consistencies in usage patterns (i.e., male to female ratio, on-campus vs. remote users, undergraduate vs. graduate students), average amount of time librarians spent preparing for the individual consultation, the length of the consultation, how advanced in the research process students were, types of assignments and sources, along with challenges encountered (e.g., “no shows” and communication issues). students were required to fill out a form, either online or at the information desk, requesting information on their topic, a description of their assignment, the due date, and the number and types of sources required. once the session was completed, the librarian who met with the student completed a brief form indicating how much time was spent preparing for the session, along with the actual meeting time and follow-up (if any), how they communicated with the student and any other problems that were encountered. the service was offered to both undergraduate and graduate students, but the majority of students using the service were undergraduate students. the authors concluded that the service’s assessment process helped to better understand what direction to take for the future of the service. becker, 1993; northern illinois university (niu)   the library offered research clinics for a large-scale first year english program, with two groups targeted: honour students and educationally disadvantaged students. librarians met with one or two students simultaneously, using their term papers to introduce them to basic reference sources for locating books and articles pertinent to their topic. it was noted that disadvantaged students appeared to respond best to the sessions. while the honour students also benefited from the sessions, authors noted that a different skill set were required for these groups. honour students had more sophisticated library research skills, so their instruction needs were different than those of the educationally disadvantaged students. both the instructors and student feedback were given anecdotally. while the librarians met and reviewed the program after the first semester of implementation, the article provide a narrative evaluation of the program using anecdotal feedback obtained from librarians and students, along with the statistics acquired on attendance and hours. the author concluded that the research clinics provided the needed follow-up to library labs already offered to freshman students. it is also noted that staffing is the major challenge identified in the pursuit of this particular service.   hoskisson & wentz, 2001; utah state university   a formal program was created to address an increased need for individualized attention for students needing assistance from a librarian at utah state university. authors noted that demanding users complete a detailed information form may act as a barrier to this service, so they avoided these forms and direct email queries were used via web form. unsolicited feedback from students came mostly in the form of appreciation but also some mentioned they actually learned library skills that they would reuse. librarians provided anecdotal feedback, indicating that if the “student did not respond with feedback, they could not gauge how helpful they had been” (p. 99). while the article reported statistics regarding the number of users per month, as well as librarian participation (number of students met per month), no formal statistics were available regarding the number of appointments versus email transactions per librarian. in order to strengthen their students’ information literacy skills, the authors mentioned that:   an accurate, formal evaluation system is always difficult to implement. perhaps a class will be taught and term paper consultations set up with an equal number of students in order to make a comparison of the two groups’ abilities to obtain pertinent research articles. preand post-tests would lend data with which to judge the value of the program (p.100-101).   the authors concluded that this new hybrid service exceeded their expectations since it held a large number of queries, many positive comments from students.   lee, 2004; mississippi state university libraries   the mississippi state university libraries provided individualized instructional sessions to undergraduates, graduates, and faculty members. these sessions were managed via an official service, where users must complete a form to request an appointment. librarians are expected to respond within 24 hours and to prepare before the appointment (as per the institution’s reference department’s performance standards). the form included information about referral source and user’s department affiliation, which allowed the author to analyze one year of usage statistics. the library was interested in the referral source in order to evaluate their marketing strategy. librarians (40.5%), faculty members (23%), and outreach programs (16.2%) represent the most frequent referral methods. the findings showed that users of the service were mostly graduate students (64.9%), represented a variety of departments, with the department of education being the most represented (32.4%). the author concluded that a different promotion approach is needed for the undergraduate clientele, but the service’s overall assessment is positive, since it provided extended research support to its users.   meyer, forbes, & bowers, 2010; penrose library, university of denver   authors described the implementation of a reinvented reference service divided in two distinctive services: a research desk staffed with trained graduates students, and a dedicated space with dual computers for one-on-one consultation, where students can book an appointment with a subject-specialist librarian. the service was inspired by the general design of writing centres, where students can get help privately but yet in a visible open-space, which can ease students’ anxiety when in need of help. usage statistics show that both undergraduate and graduate students were users of the service, as well as faculty members. social sciences and business students were the heaviest users of the service. anecdotal comments from users were recorded, which showed a positive reaction to the new service. librarians were also satisfied with this new model, as it allowed them to better use their expertise rather than simply staffing the reference desk. this unique model, with dual computers, allows students to lead the interaction and to go at their own pace which helps develop their research skills. the authors concluded “the consultation model is more effective for student learning, more fulfilling for librarians, and more efficient use of time for both” (p. 66).   yi, 2003; california state university san marcos (csusm) library   the author described this article as case study of the csusm library’s individual research consultation service (ircs). eight advantages of ircs over traditional reference service are summarized. the author states that the ircs provided a channel for students to get in-depth individualized research assistance for their projects with a subject information specialist. the ircs was part of the library reference service for several years, and it was offered to both undergraduates and graduates. students completed an individual appointment form, where they indicated their research question and the nature of assistance requested, then the student was matched with a subject librarian. the librarians had to record the main topics covered on the appointment form, and these forms were archived since 1996. the author analyzed two years of these archived forms, coded the data and added it to a database. direct observations of ircs, and interviews with three librarians, were additional methods used. usage statistics recorded the number of sessions, the hours provided, and the number of students seen. the author also gathered the number of hours librarians taught information literacy sessions, and determined that librarians spent 32% of their teaching time, with ircs being a type of teaching, doing ircs. additional information extracted from the performed data analysis indicated that 87% of ircs sessions were course-project related, which, the author emphasized, allowed for teachable moments as students had an immediate need to be filled. also, 31% of students had attended a library class previous to requesting an ircs, and 77% of these sessions were for students requesting help for 300-level courses or above. the most frequent topics covered during an ircs were: “topic assistance”, “search skills”, “database selection”. the author suggested that ircs sessions have the potential to be a teaching medium where information literacy goals could be better addressed if the librarians involved are conscious of their role in that matter. the author concluded that the ircs could be developed as a multi-level, multi-phased il program, instead of an extension of the reference service.   survey   auster, e., devakos, & meikle, 1994; university of toronto   the authors outlined the planning, implementation, and assessment of a “term paper clinic” (tpc) for undergraduate students only. the tpc was the result of collaboration between the faculty of library and information science and the sigmund samuel library, the main undergraduate library at the university of toronto. mls students were the sole providers of individual consultations to undergraduate students. the project was interesting for the library as it provided individual consultations to undergraduate students as an extension of the existing consultation service in place for graduate students and faculty members. mls students received a three-hour orientation (two hours in-class, and one hour in the library). the tpc was scheduled for two-hour periods over three weeks during the spring semester in 1993, where two mls students were scheduled to work each period. a designated desk near the reference desk was dedicated to tpc, and the service was provided on a walk-in basis. the mls student first spent approximately twenty minutes with the undergraduate student and filled in the tpc library research guide form to record the needed information. the mls student then created a tailored tpc library research guide, and met with the undergraduate student usually within twenty-four hours of the initial meeting to provide the guide. to assess this new service, every undergraduate student received a survey at the end of the second meeting, and asked to return it to the reference desk. the survey had a 49% response rate. the service was originally designed for first and second year students, but other academic levels used the service as well, which showed that many students needed individual in-depth assistance, and should not be denied that specific type of service. additional information extracted from the survey included: how students learned about the service (77% from posters), their satisfaction level (68% assessed the service as being very useful or somewhat useful), and what skills were learned from the clinic (e.g. the need to focus, using different research approaches, using keywords and subject headings). mls students were also asked to provide feedback on their experience. a content analysis of their reports described the experience as a success and the mls students commented that they would like to see this service continued, they found it rewarding as it provided them with practical experience. the overall analysis of the experience also underlined some problems. mainly, the mls students’ inexperience was a barrier to provide an adequate service. also, some undergraduate students misunderstood the tpc’s goal and believed it would provide essay-writing assistance. the author concluded that both the mls students and the librarians benefitted from this experience.   bean, 1995; depaul university.   the author describes the implementation and evaluation of depaul’s research consultation service, offered to both the undergraduate and graduate student population. the implementation process included the creation of a “research consultation appointment request form”, and the development of procedures, such as length of sessions to be provided. an evaluation process was put in place after the service had been running for one year. goals were set before the start of the evaluation process. the method used was in two parts: 1) the librarian would complete an evaluation form, then 2) the patron would fill out a separate evaluation form. the response rate was of 91% for librarians, 55% for students. results from the librarians’ forms revealed that 86% of the time librarians rated their sessions either “excellent” or “good”. other information gathered was “preparation time” and “sources used”. students’ forms showed that 100% of students surveyed rated sessions either “very helpful” or “helpful”. other information included the student’s program or department, and where they learned about the service.   cardwell, furlong, & o'keeffe, 2001; gettysburg college, marquette university, and bowling green state university (bgsu)   the authors analysed the personalized research clinics (prcs) program offered at three different institutions, and address logistics, assessment methods, and publicity. at bgsu, prcs are offered to undergraduate students. a similar service is offered to graduate students and faculty members, but is managed differently. prcs are most utilized during a four-week period, but are also offered throughout the semester. for that four-week period, a schedule is organized with librarians’ availabilities. students booked their appointment at the reference desk and provided information about their research project. evaluation forms were given to students since the implementation of the service. to strengthen the evaluation process, evaluation forms to be filled by librarians were added to the mix, and paired with the student’s evaluation forms. data from this comparison exercise is not provided in the article. a general comment is mentioned, where the authors state that after the implementation of the two-part evaluation form, students’ comments were “strong” (i.e., they were satisfied by the service).   the marquette university libraries provided prcs to undergraduate students, graduate students, and faculty members. students booked appointments either on the library website, by phone, or in-person. the requests were sent to either the contact person for the humanities and social sciences, or for the sciences. the contact person decided which subject librarian was the most appropriate for each request. the librarian and the requestor then set up a meeting time. usage statistics were collected over the years including gender, affiliation, academic level, and field of study. an eighteen-question survey was sent by mail to all prc attendees for one calendar year (2000). twenty-five attendees answered the survey, of which 70% were graduates students, and 30% were undergraduate students. results showed that 24 out of 25 respondents indicated the session was “definitely” worth their time, while 22 indicated they would “definitely” use the service again. the authors also stated that the service seemed useful for students and seemed to meet their expectations. they concluded that marquette’s libraries would continue to provide the service.   the gettysburg college library underwent reorganization, thus readjusted how prcs were provided. different assessment methods were used to evaluate the service. printed surveys were used. details about this assessment are not available in this article.   coniglio, 1984; iowa state university   the author described the staffing, scheduling, publicity and evaluation of the “term paper advisory service” (tpas) at the iowa state university library. this service was designed for undergraduate students, but graduate students and faculty members used the service as well. a steering committee was formed to plan the tpas’s structure. fourteen librarians offered the service, from which some were non-reference librarians. a training session was held specifically for them before tpas started. their procedure stated that there would be no effort to match student’s topic to a librarian’s specialty, as the author pointed out, in order to mimic common reference desk interactions. tpas were scheduled for two two-week periods, around mid-terms and finals. students were requested to fill a worksheet asking information about their topic. the appointment was then booked the next day, for fifteen minutes. the librarian created a customized pathfinder before the appointment, identifying relevant sources for the student. the meeting started at the reference desk, and the librarian then took the student to the physical location of the relevant sources listed. to assess the service, an open-ended questionnaire was sent to the one hundred students who participated to the first two-week period. fifty-seven completed questionnaires were returned. the author summarized the results, saying that students were very favorable to the tpas service. students would recommend the service to a friend. after the service’s revision, tpas was to be offered all semester long, and four additional librarians joined the team. meeting length was also adjusted to thirty minutes, and librarians’ preparation time was increased to 48 hours. the author concludes “tpas complements and supplements the basic work done with students in library instruction classes and lectures” (coniglio, 1984, p. 82).   debreczeny, 1985; university of north carolina   the author describes the development of the “term paper consultations” (tpc) at the university of north carolina’s undergraduate library. the tpc program did not provide a bibliography to students, but rather a search strategy, along with reference material. students were also shown how to use the lc subject heading guide, and are shown potentially relevant periodical and newspaper indexes. students booked a thirty-minute appointment on a sign-up sheet, and each day librarians selected appointments that worked with their schedule. the service was offered year long, with busy periods of four weeks each semester. four and a half professional librarians, library school assistants, and one graduate student staffed the tpc. with increased demands, the staff decided to record the tpc appointments’ content on a form, which was designed to describe each step of the research process. eventually, an index of those tpc files was produced, and was used both for tpc and for the reference desk. to assess the service, a survey was conducted, which 60 students answered. results show that 100% of respondents would have recommended the service to someone else, 90% said they would have used tpc for another assignment, and 92% mentioned that it fulfilled their expectations. the author pointed out that the tpc subject files and the indexes developed over the years were extremely useful not only for the tpc service, but also for everyday reference questions.   gale and evans, 2007; missouri state university   the meyer library’s research consultation service was offered to all undergraduate and graduate students, staff, and faculty. requests were made through an electronic form on the library website. these requests were routed to the appropriate librarian according to subject expertise and availability. the form asked specific questions about the student’s topic, resources already consulted, and so forth, which allowed librarians to prepare before meeting the student. two surveys were designed to assess the research consultation service. the patron’s survey, which consisted of both open-ended and likert scale questions, was sent to all of the service’s participants during one year. results from 23 students who answered the survey (31% response rate) showed that 52% of the respondents strongly agreed that the library’s material selection met their research needs, while 88% of respondents strongly agreed the consultation helped them with their research. in addition, 60% of respondents strongly agreed they felt more confident in their ability to use the library’s resources. the second survey, consisting of six open-ended questions, was distributed to librarians providing the service. the main results showed that all librarians spent at least 30 minutes preparing, and all respondents felt satisfied to have helped the majority of the students. librarians also commented that the service was beneficial to the university community, and a valuable use of faculty time. the authors concluded, in light of both surveys, that this kind of tailored one-on-one service was worth continuing.   gratch & york, 1991; bowling green state university   the bowling green state university (bgsu) libraries were offering individual consultations to all students for many years. these consultations were not specifically tailored to graduate students’ needs, and were not highly publicized. a pilot project, the personalized research consultation service (percs), provided individual consultation to graduate students specifically. four departments were included in the pilot project, and 30 students used the percs in the first year. a survey was sent to the participating students. additionally, phone interviews were carried out with faculty advisors to ask their opinion of the service. the survey produced a 56.7% response rate. results show that all respondents found the consultation helpful, and they would recommend the service to a friend, and 76.5% of the respondents used percs to get help with a thesis or dissertation. once the pilot project was over, it was decided to continue the percs and to make it available to all graduate students.   imamoto, 2006; university of colorado   the boulder library at the university of colorado embarked on a partnership with the university’s program for writing and rhetoric (pwr) to integrate information literacy concepts in the first-year course, which consisted of four parts: 1) online tutorial on basic library research, 2) course-integrated library seminar, 3) theme-based course reader, and 4) drop-in “research center”. the research center is different from the library’s usual individual consultation services because no appointments are needed, only graduate students staff the center, and it is available only to undergraduate students registered in a specific writing course. the graduate students are provided with a comprehensive training at the beginning of the school year. to assess the research center, an evaluation form with three open-ended questions was given to each student at the end of the interaction. two questions were added the following semester. completed forms were to be dropped off at the research center. in total 95 students filled out the evaluation form, for a response rate of 23%. results show that 95% of respondents felt the graduate student providing the consultation was helpful, 15 respondents requested more hours as an area of improvement, and 6 respondents asked for more tutors (graduate students). students’ experience of the research center scored 4 or 5 out of 5 for 83% of respondents. in conclusion, the author articulates additional information that would be helpful to gather in a future survey, such as students’ backgrounds, which could help understand better what students need in order to improve the service.   magi & mardeusz, 2013; university of vermont   this study used a qualitative approach to investigate students’ views on individual research consultation value, and what motivates students to request this particular type of assistance. both undergraduate and graduate students requesting research help were included in the study. moments after the consultation was completed, students were invited to answer an open-ended question survey. the authors expressed how this study is not about the “effectiveness of consultations in terms of student learning outcomes” (p. 608), but rather why individual research consultations are valuable to students. in total 52 students responded to the survey. results show that students learned about the research individual consultation service mostly through professors, and from in-class library presentations. all respondents said they would use this service again. more than one-third of respondents said that their motivation for booking a consultation “was the need for help finding information and choosing and using resources” (p. 610). when the students were asked about the type of assistance that the librarians provided during research consultation, three-quarters of respondents answered: “by selecting and recommending sources, including databases and reference books, and brainstorming about places to search” (p. 611). the authors also asked: what do students who use individual consultations find valuable about face-to-face interaction with librarians, even with the availability of online help? the authors summarized results in this way: “a face-to-face interaction allows for clear, quick, efficient, and helpful dialogue; can ask questions and get immediate responses” (p. 612). students also mentioned how a face-to-face meeting allows for a replication of the steps taken by the librarians in the resources navigation. lastly, the authors asked the students to describe their feelings before and after the consultation. before the appointment, one-third of the respondents used the words “overwhelmed”, “stressed” and “concerned”. “relieved” is the word most frequently used by students to describe their feeling after the consultation, and “confident” and “excited” were also popular expressions. the authors concluded that reference librarians, who care deeply about students’ information literacy competency development, should consider making individual research consultation part of their reference service.   rothstein, 1990; university of british columbia   this article is a reproduction of a presentation the author offered in 1979 about a “term paper clinic” (tpc) conducted at the ubc sedgewick library for a number of years, with mls students staffing the clinic. the tpc was offered twice a year for a two-week period to undergraduate students only. mls students had a five-week preparation period that comprised of instructions in reference sources and strategies, a lecture by the professor, an accompanying written guide called the tcp literature search: approaches and sources, a library tour, in-class sample question practices, and access to tcp guides previously produced. during the tpc’s operation period, students (called “recipients” by the author) would meet twice with the mls student; they would first register and provide information about their needs, then a second meeting would be scheduled to provide the recipient with the desired information in form of a search guide. to assess the tcp’s success, three feedback methods were used. first, the mls students submitted a report on their experience. second, the sedgewick librarians’ anecdotal comments were captured. third, recipients were asked to fill out a questionnaire. mls students’ comments were summarized as followed: “tpc gave them a sense of confidence as they realized that they did indeed have a great deal of knowledge which laymen did not possess; they began to think of themselves as professionals” (p. 263). one downside of this project for mls students was time, as it was more work than anticipated for them to produce the search guides. sedgewick librarians, on the other hand, had almost all favourable comments for the tpc. recipients’ feedback was taken through a questionnaire that was sent out to all tpc’s participants every semester. the author analyzed the results of one particular semester (fall of 1976). the evaluation form held a 30% response rate (77 students). main results showed that 90% of respondents answered that the service provided was either extremely or very useful, that 94% of respondents said that the tpc helped them improve their knowledge of library resources, and that 92% of respondents mentioned that they feel better prepared to use the library on their own. as an additional evaluation, in 1976 an mls student conducted 60 interviews with recipients of the service. her findings support the results obtained by the recipients’ evaluation forms, mls students’, and librarians’ feedback. the author concludes that a “personalized, extensive reference service provided at the point of need is a very effective method of teaching the use of the library” (p. 269).   schobert, 1982; university of ottawa   the author describes the planning, execution, and evaluation of a pilot project held at the university of ottawa’s morisset library called “term paper counselling” (tpc). the tpc was held once during the academic year, for a two-week period in the winter semester, and was to be provided only to undergraduate students. students had to first book an appointment and fill out a form providing information about their topic. librarians would then prepare a search guide that provided a selection of indexes, bibliographies, etc. during the appointment, the librarian would go through the guide showing the student how to use the various suggested bibliographies and indexes. to evaluate this new program, the author sent a questionnaire to all participants one month after the tpc was over. only 19, students answered, though almost all respondents were enthusiastic about the new service. the author concluded that tpc is a worthwhile service, and will continue providing it.   objective quantitative methods   erickson & warner, 1998; thomas jefferson university   the authors conducted a one-of-a-kind study, where the impact of a one-hour medline individual tutorial was assessed with specific outcomes measured (i.e., search frequency, duration, recall, precision, and students’ satisfaction level). these individual sessions were specifically designed for obstetrics and gynaecology residents. this was a randomized, controlled, blinded study, conducted with 31 residents. these students were divided in three groups. group a was the control group that received no formal medline tutorial. group b and group c received one-hour individual tutorials, including advanced medline search features, such as mesh searching, focus and explode functions, and so forth. group b had their tutorial in a hands-on format, where the residents performed the search themselves. group c received a tutorial where the instructor performed the searches. all participants answered a survey before and after their searches, asking them about their computer experience, what they thought was a reasonable number of articles retrieved when searching medline, and how long a search should take. no statistically significance differences were detected among participants. all residents had to perform four assigned searches; two before the tutorial, and two after. three faculty members independently rated the citations retrieved for relevance. a seven-point relevance scale was developed for this purpose at mcmaster university (haynes et al., 1990). the primary investigator rated recall and precision. results show that there were no statistically significant differences between the pre-tutorial assigned searches and the after-tutorial assigned searches for the search duration, the number of articles retrieved, the recall rates, the precision rates, or the searcher’s satisfaction level. limits to the study included the small group of participants, low compliance rates, and a change in the database platform at the study’s mid-point. participants felt satisfied with their searches both assigned (85%) and unassigned (64%), and were interested in improving their medline search skills (60% wanted further formal training). the authors concluded that time constraints is a major obstacle for information professionals to provide individual tutorials, especially since there were 700 residents and fellows at this institution that particular year, and for residents who struggle to free some time from their busy hospital schedule to receive adequate database search skills training.                 donegan, domas, & deosdade, 1989; san antonio college   authors describe a bibliographic instruction experiment comparing two instructional methods: group instruction sessions vs, individual instruction sessions called “term paper counselling” (tpc). participants included 156 students enrolled in an introductory management. the authors first developed learning objectives that would be used to measure students information literacy skills for both instructional methods. then, they created and tested two versions of multiple choices questions, which they trialed with two groups of students (one having had library instruction, and the other did not). data from the testing was compiled, and no difference appeared between the two versions. however, a difference was noted between the two groups regarding the students’ il knowledge, which was expected since one group had not received a library instruction course yet. in the fall semester, students from the management course were divided in three groups. group 1 received group instruction, group 2 received tpc, and group 3 received no instruction as the control group. all students were informed that a library skills test would be administered and it would be worth 5% of their grade. for group 1, the test was administered right after the library instruction. for group 2, librarians had to prepare a pathfinder first on each student’s topic, followed by a meeting with the student (individually), then students would be given 25 minutes (same length of time for all groups) at the end of the meeting to answer the test. group 3 was given the test in the classroom. once their test was completed, the librarian would inform them that they were part of an experiment, and they would be allowed to retake the test after they were provided with a library instruction session. using tukey’s hsd (honestly significant difference) test, results show that a significant difference existed between tpc and the control group, as well as between group instruction and the control group; but no significant difference was found between tpc sessions and group instruction sessions. the authors conclude that “term paper counselling and group instruction are comparably effective techniques for teaching basic library search strategy” (p. 201).   reinsfelder, 2012; penn state mont alto   this author used citation analysis to evaluate the quality of students’ sources included in draft papers before meeting a librarian, and again with the final paper after the meeting with a librarian. criteria used were currency, authority, relevance and scope. faculty members teaching various undergraduate courses were invited to participate in the study by inviting their students to book an appointment with a librarian, and by sharing their students’ drafts and final papers for citation analysis. in total 10 classes were included in the study, 3 of which were part of the control group, where students’ draft and final papers would be assessed, but students would not meet with a librarian. additionally, faculty members were asked three open-ended questions to provide their observation and perception of the process. nonparametric statistical tests were used for data analysis. for the experimental group, those who met with a librarian), a significant difference between draft and final papers was found in all criteria except for authority. no significant difference was found for the control group. faculty commented that this approach was worthwhile. the author indicated that using a rating scale is useful to measure objectively students’ sources’ quality, but there is room for subjective interpretation. the author concluded that students who partook in an individual research consultation with a librarian showed an improvement in their sources’ quality, relevance, currency and scope.     research article   dissertations and theses in top nursing publications: a bibliometric study   stephen woods social science librarian, social science data and government information the pennsylvania state university libraries university park, pennsylvania, united states of america email: swoods@psu.edu orcid id: 0000-0003-2102-1890   kathleen phillips nursing and allied health librarian the pennsylvania state university libraries university park, pennsylvania, united states of america email: kec5013@psu.edu orcid id: 0000-0002-0098-6354   andrew dudash librarian for political science, policy studies, and government information the pennsylvania state university libraries university park, pennsylvania, united states of america email: amd846@psu.edu orcid: 0000-0002-9546-3440   received: 16 apr. 2020                                                              accepted: 28 sept. 2020      2020 woods, phillips, and dudash. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29764     abstract   objective to investigate the overall prevalence of citations to theses and dissertations, as well as their prevalence in feature articles, editorials, and review articles in top research nursing journals. to evaluate differences between journals and to determine whether there was a change in use over time.   methods journals were selected from the medical library association’s nursing and allied health resource section’s 2012 selected list of nursing journals. an evaluation was conducted of citations from 3,711 articles published in 2011 and 2018 in 7 top nursing journals. thesis and dissertation citations were identified and categorized by type of scholarly communication: feature articles, reviews, and editorials. analysis was conducted for the prevalence of citations for theses and dissertations based on percentage of overall citations and the percentage of articles with a thesis and dissertation citation.   results thesis and dissertation citations accounted for 0.41% of all citations. however, 9.43% of the articles contained at least one thesis and dissertation citation. feature articles contained more thesis and dissertation citations than review articles and editorials. the journal of advanced nursing, journal of clinical nursing, and the scandinavian journal of caring sciences published a higher percentage of articles with at least one thesis and dissertation citation.   conclusion the overall use of theses and dissertations in nursing scholarship is comparatively low compared to other forms of scholarly communication. however, this unique form of scholarship viewed from its impact on the percentage of scholarly articles in nursing demonstrates that theses and dissertations have made more of a contribution than previously reported. our research provides libraries and the nursing academy with empirical evidence for the value of theses and dissertations. it provides librarians and the nursing academy justification for continuing efforts to preserve, enhance access through digital repositories, and to continue to explore strategies to promote the use of theses and dissertations in research.     introduction   theses and dissertations have a long-established history in nursing scholarship (yam, 2005). the emergence of the professional doctorate in nursing in the 1990s, with its policy-driven emphasis on the practical application of research, expanded the scope and output of theses and dissertations as a form of scholarly communication in nursing (yam, 2005; smith, 2013). furthermore, the development of electronic theses and dissertations (etd) and the growth of repositories and archives in 2005 generated considerable discussion around the use and access of theses and dissertations to further scholarly research.   macduff (2009) provides a case study concluding that web-based repositories have the potential to impact the nature, use, and access of theses and dissertations. goodfellow (2009) explores the impact of etd repositories by reviewing the networked digital library of theses and dissertations, australasian digital theses program, and the commercial publisher proquest’s dissertations and theses. goodfellow and co-authors (2012) conducted a survey to understand the knowledge and use of etd by faculty, students, and alumni from nursing schools in australia, new zealand, united kingdom, and the us. only 44% of the 209 participants understood how to access etds at their institutions and only 27% cited an etd in a publication. mcduff and co-authors (2016) conducted a qualitative study of 14 nursing scholars in order to understand how nursing scholars engage with etds. they identified six themes from the interviews: importance of initial exposure, process of searching for etds, access issues from multiple disciplines, handling and reading etds, application of etds, cultural and institutional changes in etds.   literature review   bibliometric studies offer a unique lens for exploring the use of theses and dissertations in nursing research. traynor (2011) argues that bibliometrics offers insight into the state and status of research in nursing, the value of disciplinary activity, and patterns of publishing within the profession. this bibliometric study focuses on the use of theses and dissertations in nursing scholarship and patterns of use in nursing journals.   bibliometric studies of grey literature from various disciplines have been conducted to measure the prevalence of thesis and dissertation citations in research (anderson & thiery, 2005). scholars define grey literature as material published by governments, academics, businesses, and industries not controlled by commercial publishers (farace, 1997). while the categorization of grey literature is not an exact science, there is a consensus in grey literature studies identifying theses and dissertations as one of the grey literature categories measured (see examples in table 1).   two larger bibliometric studies unrelated to grey literature research also examined the scientific impact of theses and dissertations using bibliometrics. larivière, zuccala, and archambault (2008) measured the instances of dissertations and theses cited in 266 journals from 1945 to 2004 within web of science (thompson scientific). it was determined that dissertations and theses comprised less than 1% of the references in journals from both the social science and humanities and the natural sciences. furthermore, the authors identified a leveling off in the 1980s regarding the number of theses and dissertations cited in the scholarly material evaluated. the study also demonstrated a decline in the percentage of thesis and dissertation references compared to the total number of references. the authors examined the percentage of thesis and dissertation references in smaller specific subject domains within the natural sciences and the social sciences and humanities. clinical medicine and biomedical research had the lowest percentage at around 0.2%. approximately 25% of the dissertation and thesis citations in the natural sciences were self-citations, while 15% in the social science and humanities were self-citations.     table 1 theses and dissertations in grey literature studies grey literature studies # thesis & dissertation (td) citations # articles # citations journal type/year (alberani, pietrangeli, & mazza, 1990) 390 1,398 22,072 science/1990 (pelzer & wiese, 2000) 259 2,159 55,823 veterinary/2000 (woods, phillips, & dudash, 2020) 277 1,467 52,116 nursing/2011     rasuli, schopfel, and prost (2018) conducted a similar study using scopus (elsevier) data from 1970 to 2017, concluding that 1.5% of publications contained a citation for a thesis or dissertation. the data demonstrated an increase in the percentage of publications citing theses and dissertations in all fields of study, led by publications in the arts and humanities. the authors also examined the contributions of etds, by evaluating a representative sample of theses and dissertations to determine whether they were available in an institutional repository through searches in google scholar and bielefeld academic search engine. they found that 83% were available in some type of electronic repository.   aims   to establish the prevalence of thesis and dissertation citations in core scholarly and research nursing journals, the following are examined: percentage of thesis and dissertation citations found in the total number of citations, types of articles citing theses and dissertations, and number of articles citing theses and dissertations. differing from previous studies, the prevalence of thesis and dissertation citations based on the percentage of articles that included a thesis and dissertation citation are also examined. the authors argue that this approach provides a more meaningful way to measure prevalence for unique types of scholarly communication like theses and dissertations. this research explores the following questions:   1.      what is a more accurate bibliometric method to determine the use of theses and dissertations in the scholarship of nursing? 2.      does the use of thesis and dissertation citations change over time? 3.      what types of articles cite theses and dissertations? 4.      are articles that cite theses and dissertations evenly distributed among the journals?   methods   the authors used citation analysis to determine the incidence of thesis and dissertation citations appearing in the bibliographies of articles published in 2011 and 2018 from 7 top nursing research journals.   journal selection   journals selected for analysis are based on evaluation of the research impact of journals from the 2012 “selected list of nursing journals” published by the nursing and allied health resource section (nahrs) of the medical library association (sherwill-navarro & allen, 2012). criteria for the selection of journals is based on the combined rank, from a rank based on the number of research articles and a rank based on the percentage of research (table 2).   the top six journals with the highest combined ranks were selected for this study. the journal of women’s health was also included as the highest-ranked independent publisher, providing seven total journals. the authors’ decision to include a title from an independent publisher recognizes the scholarly contributions of journals outside standard academic publishers. citation data were collected from articles published in 2011, because the 2012 nahrs list (sherwill-navarro & allen, 2012) is based on 2007-2011 data. articles from 2018 were selected because they offer the most current complete data to evaluate change in the use of td citations over time. the analysis is therefore focused on the following journals:   1.      journal of clinical nursing (jcn) 2.      patient education and counseling (pec) 3.      infection control & hospital epidemiology (iche) 4.      journal of advanced nursing (jan) 5.      scandinavian journal of caring sciences (sjcs) 6.      maternal and child health journal (mchj) 7.      journal of women’s health (jwh)     table 2 journal selection rank nahrs journals 2012 # research article rank research % rank combine rank publisher journal of clinical nursing 1 1 2 wiley patient education and counseling 4 5 9 elsevier infection control & hospital epidemiology 5 4 9 cambridge press journal of advanced nursing 3 13 16 blackwell maternal and child health journal 11 6 17 springer scandinavian journal of caring sciences 14 3 17 wiley international journal of nursing studies 7 14 21 elsevier qualitative health research 9 17 26 sage cancer nursing 20 9 29 kluwer journal of women's health 6 26 32 mary ann liebert midwifery 17 19 36 elsevier journal of nursing management 13 27 40 wiley nursing research 28 12 40 lww international journal of nursing practice 23 18 41 wiley journal of school health 18 24 42 wiley health care for women international 27 16 43 taylor & francis nurse education today 12 36 48 elsevier journal of pain and symptom management 10 40 50 elsevier american journal of public health 2 49 51 apha     data selection   meta-data from articles published in 2011 and 2018 from these seven journals were extracted from web of science to create a parent article data set. a comparison of articles listed on the journals’ official websites identified missing articles along with appropriate meta-data, which were added to this study. the authors analyzed article types occurring in all of the journals selected, in order to make comparisons between journals. these article types included feature articles, editorials, and reviews. this study evaluated 3,113 articles (table 3); 297 articles identified as correspondence, letters, retractions, proceedings, and biographies were not included in the study. a separate citation data set for citations was created using the articles’ digital object identifier (doi) which functioned as the key to map each citation to the parent article data (pad) set. the pad included the dois, authors, authors’ institutions, titles of articles, article types, volume numbers, page numbers, and years published. citations for articles found in web of science were pulled from the database. the citation count for each article was confirmed through the journals’ websites. missing citations were added as needed. citations for articles missing from web of science were added through the journals’ websites.   the authors analyzed 76,566 citations from 2018 and 61,072 citations from 2011 (table 4). the data were organized to identify the authors, year, sources, and corresponding dois with the pad set. however, most citation records also included other useful information such as: volume, page number, uniform resource locator, institutions, publisher, or book chapter titles.     table 3 number of articles analyzed by type of article   feature article editorial review total iche 2011 165 20 6 191 iche 2018 194 36 19 249 jan 2011 214 16 28 258 jan 2018 198 40 47 285 jcn 2011 306 60 40 406 jcn 2018 489 21 63 573 jwh 2011 213 9 4 226 jwh 18 223 24 13 260 mchj 2011 171 3 0 174 mchj 2018 209 2 5 216 pec 2011 290 24 23 337 pec 2018 210 27 42 279 sjcs 2011 95 4 2 101 sjcs 2018 136 6 14 156           total 2011 1,454 136 281 1,693 total 2018 1,659 156 203 2,018 total 3,113 292 306 3,711     table 4 number of citations analyzed by type of article   feature article editorial review total iche 2011 3,537 520 450 4,507 iche 2018 3,869 615 902 5,386 jan 2011 8,930 166 1,674 10,770 jan 2018 9,412 618 2,640 12,670 jcn 2011 11,607 573 2,134 14,314 jcn 2018 20,807 277 3,158 24,242 jwh 2011 8,622 153 181 8,956 jwh 2018 8,390 362 739 9,491 mchj 2011 6,153 26 232 6,411 mchj 2018 6,244 8 234 6,486 pec 2011 10,739 254 1260 12,253 pec 2018 7,749 722 3,219 11,690 sjcs 2011 3,757 20 84 3,861 sjcs 2018 5,811 15 775 6,601           total 2011 53,345 1,712 6,015 61,072 total 2018 62,282 2,617 11,667 76,566 total 115,627 4,329 17,682 137,638     an initial review of the citations identified common serial titles and publications by news sources, conference proceedings, government, and corporate authors. these were coded as “not tds” with no further evaluation. citations from web of science imported with the code “thesis” were initially coded as “td”, but were reviewed later using worldcat or the library catalogues of the issuing institutions. often, web of science did not import crucial citation information or formats that would assist with determining the thesis and dissertation status. in such cases, the original article was consulted to determine if there were further clues. if the title remained undetermined, then a search was conducted in worldcat or google scholar for the citation in question. only titles that were confirmed as theses and dissertations were included in this study.   the citation data set included 564 citations identified as theses and dissertations. this study is an aggregation of all thesis and dissertation types within the nursing profession, allowing the avoidance of international nuances between theses and dissertations. for example, the data set includes licentiate theses, medical specialty theses, postgraduate theses, doctor of philosophy dissertations, doctor of professional studies dissertations, and one undergraduate senior honors thesis. coding for each thesis or dissertation citation included: doi corresponding to the article in the pad set, td author name, issuing year, title of the td, title of the td in the original language if available, type of td, and name of the issuing institution.     table 5a total prevalence of thesis and dissertation citations   td citations articles with td % articles with td % td citations iche 2011 2 2 1.05% 0.04% iche 2018 1 1 0.40% 0.03% jan 2011 77 52 20.16% 0.71% jan 2018 72 39 13.68% 0.57% jcn 2011 102 69 17.00% 0.71% jcn 2018 127 77 13.44% 0.52% jwh 2011 3 3 1.33% 0.03% jwh 2018 2 1 0.38% 0.02% mchj 2011 11 9 5.17% 0.17% mchj 2018 5 5 2.31% 0.08% pec 2011 26 21 6.23% 0.21% pec 2018 16 13 4.66% 0.14% sjcs 2011 50 26 25.74% 1.30% sjcs 2018 70 32 20.51% 1.06%           total 2011 271 182 10.75% 0.44% total 2018 293 168 8.33% 0.38% total 564 350 9.43% 0.41%     table 5b prevalence of thesis and dissertation citations without iche and jwh   td citations articles with td % articles with td % citations with td total 2011 266 177 13.87% 0.56% total 2018 290 166 11.00% 0.47% total 556 343 12.32% 0.51%     results   analysis by articles   percentage of citations   seven journals were selected for analysis from 2011 and 2018, comprising six top-ranked research journals in nursing and the top-ranked journal from an independent publisher. a total of 3,711 articles generated 137,638 citations, of which 564 citations were identified as theses and dissertations. combining both years, theses and dissertations represent 0.41% of the citations from the articles evaluated in these journals. the data shows a decrease from 0.44% in 2011 to 0.38% in 2018 (table 5a).   percentage of articles   this study shows that 9.43% of the combined articles from 2011 and 2018 contain at least one thesis and dissertation citation (table 5a). this approach also shows a decrease in the percentage of articles citing one or more thesis or dissertation from 2011 (10.75%) to 2018 (8.33%). the number of thesis and dissertation citations increased from 2011 to 2018, but there was also an increase in total articles (table 3) and subsequently the total numbers of citations (table 4).   journal comparison   the scandinavian journal of caring science (sjcs) had the highest percentage of thesis and dissertation citations (1%). sjcs had the highest percentage of articles with at least one thesis and dissertation citation in sjcs for both of the years evaluated 2011 (25.74%) and 2018 (20.51%). the journal of advanced nursing (jan) and the journal of clinical nursing (jcn) had similar percentages (0.71%) for the number of thesis and dissertation citations to overall citations in 2011. jan (20.16%) and jcn (17%) had a higher percentage of articles with thesis and dissertation citations in 2011 in comparison to the other journals. the percentage of articles with a thesis and dissertation citation in jan and jcn decreased in 2018. patient education and counseling (pec) (0.21%) and maternal child health journal (mchj) (0.17%), in comparison to other journals, had a smaller percentage of thesis and dissertation citations. both demonstrated a percentage decrease in thesis and dissertation citations from 2011 to 2018. the percentage of articles with a thesis and dissertation citation in pec and mchj fell below the average (10.58%) for all 6 journals. journal of women’s health (jwh) and infection control & hospital epidemiology (iche) had the smallest percentage of thesis and dissertation citations and the lowest percentage of articles with a thesis and dissertation citation.   through looking at the prevalence of articles with a thesis and dissertation citation, based on the overall number of articles with a thesis and dissertation citation, it was found that jcn (146), jan (91), and scjs (58) published 85% (295) of the articles identified in this study. pec (34), mchj (14), and jwh (4) follow in rank. sjcs (26, 31) and jcn (69, 77) increased the number of articles with a thesis and dissertation citation from 2011 to 2018. the other journals demonstrate a decrease in this same time frame. removing the journals jwh and iche as potential outliers increases the overall prevalence of thesis and dissertation citations in the remaining journals (table 5b). overall, 12.32% of articles not including jwh and iche contain a thesis and dissertation citation, compared to 9.43% for all the journals in the study. the percentage of thesis and dissertation citations increased from 0.41% to 0.51% when jwh and iche were removed from the analysis.   analysis by article type   the prevalence of thesis and dissertation citations by the types of articles found in these selected publications was also explored. as stated in the methods, this analysis was limited to feature articles, editorials, and review articles.   feature articles   the identification of 475 thesis and dissertation citations occurred, which represents 0.41% of the total citations for feature articles (table 6). sjcs had the highest percentage of thesis and dissertation citations to overall citations in feature articles, but this percentage decreased between 2011 (1.3%) and 2018 (1.06%). jan had the next highest percentage of thesis and dissertation citations to overall citations with 0.85% in 2011 and 0.47% in 2018. jcn was comparable with the journals selected at 0.73% in 2011 and at 0.55% in 2018. the percentage of thesis and dissertation citations to overall citations in pec and mchj were noticeably smaller, while jwh and iche had the smallest percentage of thesis and dissertation citations in feature articles.     table 6 feature articles: prevalence of theses and dissertations journal td citations articles with td % articles with td % td citations iche 2011 0 0 0.00% 0.00% iche 2018 1 1 0.52% 0.03% jan 2011 76 51 23.83% 0.85% jan 2018 44 25 12.63% 0.47% jcn 2011 85 56 18.30% 0.73% jcn 2018 114 70 14.31% 0.55% jwh 2011 3 3 1.41% 0.03% jwh 2018 2 1 0.45% 0.02% mchj 2011 11 9 5.26% 0.18% mchj 2018 4 4 1.91% 0.06% pec 2011 24 19 6.55% 0.22% pec 2018 12 10 4.76% 0.15% sjcs 2011 49 25 26.32% 1.30% sjcs 2018 50 29 21.32% 0.86%           total 2011 248 163 11.21% 0.46% total 2018 227 140 8.44% 0.36% total 475 303 9.73% 0.41%     table 7 review articles: prevalence of theses and dissertations journals td citations review with td % articles with td % td citations iche 2011 2 2 33.33% 0.44% iche 2018 0 0 0.00% 0.00% jan 2011 1 1 3.57% 0.06% jan 2018 25 11 23.40% 0.95% jcn 2011 13 9 22.50% 0.61% jcn 2018 13 7 11.11% 0.41% jwh 2011 0 0 0.00% 0.00% jwh 2018 0 0 0.00% 0.00% mchj 2011 0 0 0.00% 0.00% mchj 2018 1 1 20.00% 0.43% pec 2011 2 2 8.70% 0.16% pec 2018 4 3 7.14% 0.12% sjcs 2011 1 1 50.00% 1.19% sjcs 2018 20 3 21.43% 2.58%           total 2011 19 15 14.56% 0.32% total 2018 63 25 12.32% 0.54% total 82 40 13.07% 0.46%     regarding feature articles, 303 were identified with td citations (9.73%) from the 3,113 feature articles evaluated in this study. for feature articles, sjcs, jcn, and jan had the highest percentages of articles with a thesis and dissertation citation. jcn had the highest overall total number of articles with a td citation (n=126). the percentage of articles with a thesis and dissertation citation in sjcs, jcn, and jan decreased from 2011 to 2018. however, the number of articles with a thesis and dissertation citation for jcn increased from 56 in 2011 to 70 in 2018. while the percentage of feature articles with a thesis and dissertation citation in pec was lower than sjcs, jcn, and jan, the number of articles in pec (n=19) was higher than mchj (n=13), jwh (n=4), and iche (n=1). mchj, jwh, and iche had the lowest percentage of feature articles with a thesis and dissertation citation.   review articles   the review articles evaluated provided the next highest percentages of thesis and dissertation citations by citation counts. there were 82 thesis and dissertation citations representing (0.46%) of citations found in review articles (table 7). overall, the percentage of thesis and dissertation citations in review articles increased from 0.32% in 2011 to 0.54% in 2018. the percentage of thesis and dissertation citations in sjcs increased in from 1.19% in 2011 to 2.58% in 2018. jan increased from 0.06% to 0.95% in that same timeframe. pec, mchj, and iche had the fewest number of review articles with thesis and dissertation citations, and jwh had none.   of the 306 review articles, 40 contained a thesis and dissertation citation (13.07%). from the journals selected, jcn had the highest number of review articles (n=16) with a thesis and dissertation citation followed by jan (n=12). jcn experienced a decrease in the percentage of articles with a thesis and dissertation citation from 22.5% in 2011 to 11.11% in 2018, while jan had an increase from 3.57% in 2011 to 23.4% in 2018. the percentage of review articles with a thesis and dissertation citation from pec was consistent from 2011 (8.7%) to 2018 (7.14%). jwh had no review articles with a thesis and dissertation citation. the low number of review articles published in iche 2011 (6), sjcs 2011 (2), mchj 2011(0), mchj 2018 (5), jhw 2011 (4), minimize the effectiveness of evaluating the percentages of review articles with a td and the percentage of td citations in those particular journals (table 7).     table 8 editorial articles: prevalence of theses and dissertations   td citations editorial with td % articles with td % td citations jan 2018 3 3 7.50% 0.49% jcn 2011 4 4 6.67% 0.70%           total 2011 4 4 2.94% 0.23% total 2018 3 3 1.92% 0.11% total 7 7 2.40% 0.16%     editorials   only seven out of 292 editorials contained thesis and dissertation citations, representing (0.16%) of the citations found for this article type. only 2 journals had thesis and dissertation citations for editorials: jan 2018 (0.49%) and jcn 2011 (0.7%). editorial articles with thesis and dissertation citations were only found in jan 2018 and jcn 2011, representing 7.5% and 6.67% of the editorials found in these journals respectively.   discussion   this study confirms the low percentage of thesis and dissertation citations compared to overall citation counts found in earlier studies (larivière, zuccala, & archambault, 2008; rasuli, schöpfel, & prost, 2018). however, the approach to evaluate prevalence based on percentage of articles with a thesis and dissertation citation indicates that around 1 out of every 10 articles published in a top scholarly nursing journal cites a thesis and dissertation.   this study supports the argument that the percentage of articles with a thesis and dissertation is a better representation of the usage of theses and dissertations in scholarly communication. traditional weight given to scholarly journal articles in scholarship skews the overall percentage of citations, thus minimizing the contributions of unique scholarly materials such as theses and dissertations.   it is clear from this study that the use of theses and dissertations in the nursing journals evaluated decreased from 2011 to 2018. although this finding is consistent with earlier research (larivière, zuccala, & archambault, 2008; rasuli, schöpfel, & prost, 2018), it would be inappropriate to assert that this decrease represents an ongoing trend. further study would need to be conducted for the intervening years.   the contributions of theses and dissertations to scholarly communication in nursing is further supported in this study by the overwhelming percentage of theses and dissertations found in feature articles, rather than other types of articles. previous research by larivière, zuccala, & archambault (2008) and rasuli, schöpfel, & prost (2018) argues that thesis and dissertation citations were most likely to be found in review articles, but this study contradicts these findings. the overall percentage of citations in feature articles is 0.41% and 0.46% for review articles. the percentage of articles containing a thesis and dissertation citation in feature articles is 9.73% and 13.07% for review articles. in sum, the use of theses and dissertations in nursing feature articles is comparable to use in review articles.   finally, the comparison of journals that publish articles with theses and dissertations provides new insights for researchers looking for publishing venues, as well as opportunities for further research. based on this study, research articles citing theses and dissertations are more likely to be published in jcn, jan, and sjcs.   limitations   journal selection   the journals selected had an english language bias. the authors did not have access to articles from the journal site for iche 2018, so were not able to verify the accuracy of all the article citation data obtained from web of science. only two thesis and dissertation citations were confirmed from iche citations for 2011 from web of science and the journals website. the data extracted from web of science for 2018 not confirmed from the journals site contained one thesis and dissertation citation. extrapolating from the low use of theses and dissertations citations in iche in 2011 and the low occurrence of thesis and dissertation citations in the 2018 data from web of science data, there is a high probability that the percentage of thesis and dissertation citations from iche in 2018 remained consistent.   analytics   the authors recognize that theses and dissertations published in some nursing programs were compilations of three to four articles published in scholarly journals. such scholarly articles were not identified as theses and dissertations for analysis. the prevalence of theses and dissertations by the percentage of review articles for sjcs, jwh, and mchj was problematic due to the small n in the total number of review articles published by these journals. consequently, the percentage of review articles for these journals was skewed.   generalizability   this study focused on the prevalence and impact of thesis and dissertation citations by type of article from a limited core of nursing journals. consequently, it does not address why a thesis or dissertation was cited or why a journal published an article with a thesis or dissertation. to explore causality, a deeper analysis of the theses and dissertations would need to be conducted, including the article citing the thesis and dissertation and the nature of the journal. this would include such elements as the subject matter of the thesis and dissertation, thesis and dissertation age, number of unique theses and dissertations, thesis and dissertation institutions, thesis and dissertation committee members, number of times the thesis and dissertation was cited overall, the type of thesis and dissertation, and whether or not there is electronic access. a deeper analysis of the citing article would also be needed, such as authors of the article (to determine whether or not a thesis and dissertation is self-cited), institutions of the article’s authors, and subject matter of the article.   conclusion   the percentage of articles citing a thesis or dissertation provides clear empirical evidence of the impact of theses and dissertations on nursing scholarship. this study shows that 1 out of 10 articles contained a thesis and dissertation citation; in the case of jcn, jan, and sjcs this number is closer to 1 out of 5 articles. the implications of these findings for librarianship and nursing scholarship is worth consideration.   collection building   the findings suggest that theses and dissertations are an important format for building collections for nursing. collection development strategies should include the selection of theses and dissertations, along with other traditional materials such as journals and monographs. academic libraries have traditionally played a key role in the preservation of their institutions’ theses and dissertations in hardcopy as well as digital formats. the scholarly use of theses and dissertations in nursing research provides justification for continued financial support for the preservation of theses and dissertations. furthermore, this study supports library initiatives to enhance access and discoverability of theses and dissertations through etd repositories and distributed networked bibliographic databases, as discussed in the introduction.   bibliographic instruction and reference   this study reinforces the importance of including discussions of theses and dissertations in bibliographic instruction and reference consultations. the prevalence of theses and dissertations in both review articles and feature articles in nursing scholarship necessitates a closer look at pedagogical approaches for instruction, as well as developing strategies for research consultations. although bibliometric research methodology is helpful for identifying key usage trends, it is not necessarily useful for identifying pedagogical approaches for communicating these concepts to users. in sum, this study reinforces the need for future research to assist librarians in communicating the benefits and use of theses and dissertations as unique contributions in advancing research.   nursing scholars and nursing academy   nursing scholars and the nursing academy should seriously consider the potential contribution of theses and dissertations to their scholarship. a number of nursing scholars and a core set of journals already cite theses and dissertations in nursing research, demonstrating that at least some nursing scholars recognize that there is empirically sound research being conducted by graduate students in the nursing academy. the affirmation of scholarship between established and future nursing researchers will only strengthen the community of nursing scholars.   furthermore, the nursing academy needs to consider the promotional value of its theses and dissertations. as a published work, the student, academic institution, and program all contribute to the scholarship of every thesis and dissertation. the nursing academy needs to work in partnership with the academic libraries to promote and provide access to theses and dissertations, in order to advance nursing scholarship, enhance the reputation of the nursing academy, and attract future students.   future research   further investigation of academic nursing programs producing theses and dissertations cited in core nursing journals would assist librarians in the development of focused collection strategies. it would also be useful to look at the lifespan of cited theses and dissertations to determine whether digitization of older hardcopy materials would benefit from access in etd repositories. a closer evaluation of theses and dissertation citations could help determine the impact and use of etd repositories. the authors argue that it is disingenuous to determine the impact of etd by simply looking at the increase or decrease of theses and dissertations.   closer evaluation of thesis and dissertation citations provides further insight for librarians providing instruction and reference. further research needs to be done focusing on how researchers use theses and dissertations, to help librarians develop instructional and consultative approaches. for example, are theses and dissertations simply used to enhance the introduction, support the literature review, or used to develop research methodology? clinical and medical librarians need to publish systematic learning strategies for finding and accessing theses and dissertations. this aggregation of theses and dissertations did not explore the idiosyncrasies of the various types of theses and dissertations. a further examination could provide librarians a better understanding of the use of masters’ theses, doctoral dissertations, professional dissertations, and international equivalents. looking at the academic programs of the cited theses and dissertations could offer librarians a better understanding of the interdisciplinary contributions of theses and dissertations.   an exploration of the international representation of theses and dissertations also offers some intriguing research opportunities. which countries or regions are producing more theses and dissertations cited in nursing scholarship? which nursing programs within these countries are better represented and why? are there regional networked bibliographic databases that contain bibliographic information about these cited theses and dissertations? do they offer multiple language searching? is a digital version available from the academic institution? is there a difference in how countries offer digital versions or discoverability?   references   alberani, v., de castro pietrangeli, p., & mazza, a. m. (1990). the use of grey literature in health sciences: a preliminary survey. bulletin of the medical library association, 78(4), 358-363. retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/pmc225438/   anderson, k.l. & c. thiery (eds.). (2005, oct. 10-14). information for responsible fisheries: libraries as mediators. proceedings of the 31st annual conference. rome, italy.   farace, dj. (1997, nov. 13-14). perspectives on the design and transfer of scientific and technical information. third international conference on grey proceedings. amsterdam, netherlands: transatlantic greynet.   goodfellow, l. m., macduff, c., leslie, g., copeland, s., nolfi, d., & blackwood, d. (2012). nurse scholars' knowledge and use of electronic theses and dissertations. international nursing review, 59(4), 511-518. https://doi.org/10.1111/j.1466-7657.2012.01008.x   goodfellow, l. m. (2009). electronic theses and dissertations: a review of this valuable resource for nurse scholars worldwide. international nursing review, 56(2), 159-165. https://doi.org/10.1111/j.1466-7657.2008.00703.x   larivière, v., zuccala, a., & archambault, é. (2008). the declining scientific impact of theses: implications for electronic thesis and dissertation repositories and graduate studies. scientometrics, 74(1), 109-121. https://doi.org/10.1007/s11192-008-0106-3   macduff, c. (2009). an evaluation of the process and initial impact of disseminating a nursing e-thesis. journal of advanced nursing, 65(5), 1010-1018. https://doi.org/10.1111/j.1365-2648.2008.04937.x   macduff, c., goodfellow, l. m., nolfi, d., copeland, s., leslie, g. d., & blackwood, d. (2016). slipping through the net: the paradox of nursing's electronic theses and dissertations. international nursing review, 63(2), 267-276. https://doi.org/10.1111/inr.12256   sherwill-navarro, pamela & allen, margaret (eds.). (2012). 2012 nahrs selected list of nursing journals. retrieved 22 october 2020 from https://sites.google.com/site/nahrsnursingresources/selected-journals-list   pelzer, n. l. & wiese, w. h. (2003). bibliometric study of grey literature in core veterinary medical journals. journal of the medical library association, 91(4), 434-441. retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/pmc209509/   rasuli, b., schopfel, j., & prost, h. (2018, sept.). edt programs after two decades: exploring impact. the 21st international symposium on electronic theses and dissertations, taipei, taiwan. retrieved 22 october 2020 from https://hal.archives-ouvertes.fr/hal-01885481/document   smith, n. (2013). professional doctorates and nursing practice contribution: a systematic literature search and descriptive synthesis. journal of nursing management, 21(2), 314-326. https://doi.org/10.1111/j.1365-2834.2012.01446.x   traynor, m. (2011). bibliometrics as politics: the case of emerging disciplines. international nursing review, 58(1), 26-27. https://doi.org/10.1111/j.1466-7657.2010.00874.x   woods, s., phillips, k., & dudash, a. (2020). grey literature citations in top nursing journals: a bibliometric study. journal of the medical library association, 108(2), 262-269. https://dx.doi.org/10.5195%2fjmla.2020.760   yam, b. m. c. (2005). professional doctorate and professional nursing practice. nurse education today, 25(7), 564-572. https://doi.org/10.1016/j.nedt.2005.05.012   evidence based library and information practice using evidence in practice   reimagining research guidance: using a comprehensive literature review to establish best practices for developing libguides   mandi goodsett performing arts & humanities librarian, oer & copyright advisor the michael schwartz library cleveland state university cleveland, ohio, united states of america email: a.goodsett@csuohio.edu   marsha miles head, collections and digital initiatives / art librarian the michael schwartz library cleveland state university cleveland, ohio, united states of america email: m.a.miles24@csuohio.edu   theresa nawalaniec sciences & engineering / nursing librarian the michael schwartz library cleveland state university cleveland, ohio, united states of america email: t.nawalaniec@csuohio.edu   received: 17 nov. 2019                                                                  accepted: 31 jan. 2020      2020 goodsett, miles, and nawalaniec. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29679 setting   located in downtown cleveland, ohio, cleveland state university (csu) is comprised of 10 colleges and schools, which offer over   175 academic programs, including several doctoral programs. the university, which has a current enrollment of more than 17,000 students, is highly diverse with regard to age, ethnicity, and country of origin. the michael schwartz library (msl) supports this diverse community with a collection of over 1 million titles, nearly 400,000 of which are electronic resources. the msl subject librarians create and maintain 340 publicly viewable libguides, both general and course-specific, spanning 64 subjects. research guides are online reference tools that librarians create to help students and faculty conduct research. research guides can include lists of relevant sources, instructional content related to the research process, and contact information for library staff. libguides are research guides built on a web publishing and content management platform offered by springshare and used by libraries throughout the world. the msl libguides are the focus of our research, which relies on a variety of evidence, including an extensive literature review of libguide design and user experience, data from our own users, and our librarians’ professional knowledge and experience.   problem   to help frame the research and decide what evidence to obtain, the researchers used the pico (problem, intervention, comparison, outcome) model to develop our research question. first, the problem was articulated: msl librarians were unsure how useful our libguides were to csu faculty, staff, and students on campus, and what impact the design of the guides had on their usefulness. we had encountered literature that suggested switching the layout of our guides from top to side navigation. when proposing this suggestion to our colleagues, it was met with some resistance, which was an additional problem. without evidence, we didn’t know which design would be most effective. the intervention we chose was to develop research guide usability best practices using relevant literature, present the results to our colleagues, and observe if the presentation of evidence improved the librarians’ receptivity of our recommendations.   evidence   evidence based library and information practice (eblip), which relies on evidence rather than theory or previous precedent as a basis for practice (hjorland, 2011), was used to structure the design of this study. we followed the eblip model of koufogiannakis and brettle (2016): we articulated our problem (described above), assembled relevant evidence, assessed the evidence for quality, and agreed to a course of action as a research team and department. as of this writing, we are still in the implementation stage, and will adapt our approach based on the outcomes of our intervention. in our case, the evidence assembled included local data from a community usability survey, the impressions and experiences of the researchers, and a thorough review of the relevant literature.   to determine the best intervention to address the problem, we conducted a thorough literature review. we gathered evidence by searching a variety of databases and platforms including academic research complete; acrl techconnect; c&rl news; digital commons network; education research complete; eric; google; google scholar; library, information science & technology abstracts with full text; and web of science.   results were excluded if they were published before 2013, not related to research guides (instead focusing on library websites or other online portals), or not related to user design. a variety of terms were considered acceptable to refer to user design, including design, layout, user experience, and others. since there were too many libguides that describe best practices to make including them practical (over 2,000 in a libguide community search), and because most of their evidence was anecdotal, these were also eliminated from the review results. we also investigated the citations in remaining resources and included them if they did not meet the exclusion criteria. two articles were included, despite falling outside of the date parameters of the review, because they were cited so heavily in the literature and clearly remained relevant to the design of research guides.   table 1 search strings included "best practices for libguides" (libguide or "subject guide" or "research guide") and use "libguides best practices" filetype:pdf alao and libguides "research guide" and "user experience" libguide "research guide" and "user experience" libguide and "best practices" "research guides" and "best practices" research guide best practices (libguide or "subject guide" or "research guide") and (evidence based or best practice)     we assessed the gathered evidence by creating a list of codes for user experience and design best practices. to reduce bias in code creation, each of the three researchers developed codes separately and then the codes were compared and assembled into a master list. the literature sources were then coded by the researchers independently and results were analyzed and synthesized to create a list of best practices. each best practice was accompanied by a list of all the relevant supporting literature, and the literature was color-coded to show what kind of evidence contributed to the authors’ conclusions (e.g., qualitative, quantitative, mixed methods, and anecdotal). the full color-coded list of best practices may be found here: https://researchguides.csuohio.edu/ld.php?content_id=47624389. best practices from the literature that appeared to contradict one another were retained to reveal areas where more research is necessary.   table 2 provides a summary of the suggestions we found via iterative literature searches, which was the primary basis of our research. however, in order to collect additional, local evidence and establish a benchmark for student, faculty, and staff satisfaction with the msl’s libguides in our specific context, we also conducted a usability survey using limesurvey in february 2019. undergraduate and graduate students, faculty, adjuncts, librarians, and library staff were included in the survey whether or not they had used research guides. the survey was available in the library’s voting booth (a publicly-accessible computer set up in a prominent location in the library lobby), as a link on the library website, and emailed directly to faculty by subject librarians. it was confidential, incentivized by a raffle, and solicited information such as what college the participant was from, whether they had used libguides before, what goals they had when visiting the site, and whether their goals were met. the survey was made available for two weeks and had 114 responses. the data from this survey were to be used to compare user satisfaction before and after the implementation of the literature-supported best practices to the library’s guides.    in the meantime, the researchers updated an existing research guide using the literature-based best practices list to demonstrate to our librarian colleagues how a guide might be modified to better match user experience standards. these guides and other relevant documents can be found here: https://researchguides.csuohio.edu/bestpractices.    table 2 literature-based best practices with conflicting evidence in brackets category best practice details design / organization / layout template provide a guide template for all librarians [a template is only so useful guides should be customized to their unique audiences in some cases, and authors should retain freedom over guide content and design] policy create standards based on best practices or other criteria uniformity / consistency follow a unified, consistent format and design (fonts, background, color scheme) for subject guides and their content make sure labels and language are consistent across guides consistently name a core set of tabs by subject or format key resources / best bets box provide a “key resources” or “best bets” box in a prominent location on the guide use a large enough text size (larger than default for libguides 1.0) hierarchy list resources strategically or by importance, rather than alphabetically sequence content in the order students would likely need to encounter it to accomplish their tasks put the most important content on the left and/or top of the page in an f-pattern integration use the main library or university website “frame” to visually integrate the guide with the rest of the website personal presence include a professional photo of one or more librarians on the guide make guides more personal by providing librarian contact information and option to chat chunking content split up content into meaningful chunks number of columns use a two column layout [use a three column layout] don’t include important content in right column (users ignore this as it is commonly ad space on websites) navigation top vs. side navigation use side/left navigation to make menu more visible tabs tabs tend to be unnoticed and large numbers of them confuse users and cause clutter, so use only most relevant ones, usually all in a single row search box include a search box as students prefer to be able to search the guide for content rather than browse/read [don’t include a libguides search box on guides, as students often treat it as a discovery or google search. if a search box is included, include a description of what can be searched.] provide embedded search boxes for research tools (i.e. databases, catalog, etc.) table of contents do not provide a box on the guide that outlines its contents, while also providing tabs, as this is considered redundant by users [provide a table of contents box on the homepage of each guide because students often overlook tabs, and/or to prevent users from having to scroll down] content jargon avoid the use of jargon throughout the guide or, if it’s necessary, provide clear explanations of unfamiliar language labeling use short, clear, meaningful titles for guide names, boxes, menus, pages, and tabs if possible, include a description (annotations) for tools provided in the guide, especially if their titles are not self-explanatory or use jargon name guides, tabs, and boxes the way students would search for them writing for the web write content using best practices for web writing use bullet points and bolded or varied text sizes to make pages easier to read content maintenance regularly check for broken links, perhaps with a link check tool make sure videos and screenshots are up-to-date make sure terminology and content is current develop a maintenance plan for guides use the libguides asset manager to efficiently update links and reuse content across all guides friendly tone use a conversational tone in the text of guides audio/visual material incorporate interactive and visual content to engage students [use images sparingly, as they often add more clutter and waste space] widgets include a chat widget allowing users to chat with the subject librarian when they are online less text / content ensure amount of information on pages and in boxes is appropriate include less content/fewer pages to avoid cognitive overload and encourage more usage avoid long lists; if lists are used, create them such that users can skip to sections/content of interest accessibility   ensure guide can be easily read by a screen reader ensure all videos on guides are captioned ensure all images have alt tags all “click here” links should instead have descriptive text for the link location ensure the color of text and other elements contrasts enough avoid relying solely on color for meaning make guides ada accessible (or meet other accessibility standard) purpose instruction vs. reference consider the purpose of the guide (to teach or to provide curated resource lists) when designing it provide instructional content in the guide that will help students complete the tasks that likely brought them there build the guide around one or more student learning outcomes or other pedagogical goals create course specific guides rather than broad subject guides considering audience think about how users will search for content in the guide, and in accessing the guide; let that govern your design the purpose of the guide should be made explicit to students connect to class / assignment tie the content of guides to specific course research and assignments external factors guides menu organize guides by how users would likely require access to meet an information need promotion & marketing librarians and, especially, instructors should promote guides link to guides in the learning management system email a link to the guide to students, provide the link in an in-class handout, and/or demonstrate how to access the guide in class guide access / discovery provide a link to guides on the library’s homepage provide links to guides in the learning management system consider ways of improving findability of guides in an organic search reduce duplication / stale guides remove unused or stale guides guide assessment / maintenance use guide usage data to regularly assess guides use usability testing (focus groups, surveys, etc.) and outreach to regularly assess guides guide authors should review guides regularly guides team / administrator assemble an administrative team to maintain upkeep of guides and set guide standards for the institution   implementation   after the evidence was assembled and analyzed, an intervention took place to apply the best practices for usability and improved design to our libguides. a 90-minute session was scheduled with guide creators to present the evidence, best practices, demo guide, and checklist (https://researchguides.csuohio.edu/ld.php?content_id=50666759) and to discuss implementation. six out of twelve guide creators attended. reception was much more favorable compared with previous discussions. indeed, sharing our research encouraged guide creators to adapt the best practices where practical. it was determined that application of the best practices should be flexible to allow for different disciplines and specific guide uses. an optional follow-up meeting to work on the research guides (a hack-a-thon) was scheduled for about a month later. four guide creators participated in the hack-a-thon, and others worked at their own desks.   reflection   one thing we learned while working on our literature review was that there is still not enough rigorous evidence about best usability design practices for research guides, and much of what does exist is specific to one institution. we also found that some of the evidence was conflicting, so more research into those specific areas would be helpful.    an additional challenge we faced in gathering evidence was soliciting usable results to our survey. we lesarned that many participants in the survey did not know what a research guide was, or had never used one. these participants gave responses to the survey that did not provide relevant information about our research guides and, for this reason, many had to be removed from our analysis. we also found flaws in our survey questions. rather than asking patrons how they used a research guide, we discovered that it would perhaps be more useful to ask patrons to show us in real-time how they would fulfill a need using a research guide.    finally, we learned a great deal from the process of using evidence to recommend department-wide change in the library. we cannot force our library colleagues to change their user design decisions, nor would we necessarily want to. we found that doing the research and presenting a well-founded set of recommendations resulted in our colleagues sometimes choosing to make changes to their guides based on our best practices investigation. however, the process also helped us become aware of unique circumstances that may warrant ignoring our recommendations, and the discussion that this engendered helped us all feel more comfortable with the resulting decisions. we hope to conduct additional usability studies in the future to make a stronger case for applying research guide design best practices in a way that best helps our local community of library users.   references   hjørland, b. (2011). evidence‐based practice: an analysis based on the philosophy of science. journal of the american society for information science and technology, 62(7), 1301-1310. https://doi.org/10.1002/asi.21523   koufogiannkis, d. & brettle, a. (eds). (2016). being evidence based in library and information practice. london, uk: facet publishing.   microsoft word es 1487_mckenna_final evidence based library and information practice 2008, 3:2 32 evidence based library and information practice evidence summary best reference practices are not observed in telephone ready reference services a review of: agosto, denise a. and holly anderton. “whatever happened to ‘always cite the source?’” reference & user services quarterly 47.1 (2007): 44-54. reviewed by: julie mckenna deputy library director, regina public library regina, saskatchewan, canada e-mail: jmckenna@reginalibrary.ca received : 03 march 2008 accepted : 06 april 2008 © 2008 mckenna. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – to study source citing practice in telephone reference service in large public libraries in the united states and canada. design – field simulation (unobtrusive testing). setting – large public libraries in the united states and canada. subjects – telephone reference staff of the 25 largest public libraries in the united states and canada. methods – the 2005 world book almanac was used to select the 25 largest (in terms of population served) public libraries in canada and the united states. each system’s web site was checked to locate the telephone number for reference service. for some systems it was necessary to call the general telephone number for the main library or the first branch listed on the web site. five ready reference test questions were developed from a list of questions that students in a graduate library and information science course had previously asked of public library telephone reference services. the selected questions in the order that they were asked were: 1. can you tell me when valentine’s day is? 2. who is the current governor/premier (of the state/province where the library is located)? 3. what is the population of montana? 33 4. in which state is the southern poverty law center (splc) located? 5. what is the french word for “chiropractor”? the authors called each of the 25 libraries during five consecutive weeks at different times of the business day. each week, one question was asked; once an answer was received, no clarification was requested and the call was ended. the study reports the results of 125 reference transactions. for this study, the following definitions were used to assess complete citation for each type of information resource: • for a web site – the complete url (title and sponsor of the site not required). • for a digital database – the database title and the title and year of the specific item (author, publisher, page number not required). • for a print resource – the title and year (author, edition, page number, publisher and place of publication not required). each reference transaction was noted to record whether the answer was correct and to define the nature of source citing that occurred. other notes were kept to describe other respondent behaviors and attitudes demonstrated during the transaction. main results – 93.6% of the answers to the 125 reference questions were correct. complete citations were provided seven times (5.6%) and partial citations were provided an additional thirty-one times (24.8%). in 68% of the 125 transactions, no source citation information was provided. there was a corresponding relationship between the difficulty of the reference question and the respondent’s provision of any citation source (either a complete or incomplete citation source). sources were generally not provided for simple questions even though the practice of citing is expected for all levels of questions. the practice of citing in order to reveal the path to the answer so that the user may become independent in the future was not observed. in addition, five “negative closure” techniques were employed by respondents. these included unmonitored referral; immediate referral away from the service; articulating that the encounter would not be successful at the start; shutting down the transaction either by tone of voice or by use of phrase that precluded any further interaction with the user; or claiming that the information did not exist or was not available. a reliance on digital formats rather than print sources was found. conclusion – the accuracy rate of the answers to the questions was very high (93.6%), but other aspects of the service were considered to be less than satisfactory. the reference and user services association (rusa) guidelines, considered the best practices for reference service, were not observed and in particular, the source citation rule was not followed. commentary this article provides some useful recommendations on how to improve telephone reference service that could be of benefit for libraries of all types and could also inform the delivery of chat and e-mail reference services. the percentage of correct answers (93.6%) was much higher than previous studies of telephone reference services which have found accuracy in the 50-70% range. this was attributed to the less difficult questions that were asked in this study than in those undertaken previously. it would be 34 interesting to determine whether more difficult questions would have led to greater frequency of source citing though it does not lessen the finding that for simple questions, rusa guidelines are generally not followed. the study authors acknowledged that although respondents were described as librarians throughout the report, only 24.8% of respondents identified themselves as librarians during the reference encounters. it would be surprising to find that support staff was not the level of staff most frequently employed for telephone ready reference services. although this study did not undertake to reveal the level of staff, it would be interesting to learn what level of staff is generally employed in the fulfillment of this service. it would also be interesting to determine how many of the 25 libraries believed that they were delivering the standard of service described in the rusa guidelines. there was no indication as to how many calls were necessary to achieve the 125 reference transactions that were reported as the study results. both authors undertook the calling, but there was no reference to a standard script or any standardized record keeping instrument that was used to track response activity. the authors referred to respondent attitude as demonstrated through verbal intonation and other behaviors, but not to how that evidence was captured in a systematic manner. the authors’ initial frustration with finding the reference service telephone number extended to the challenge of the labyrinth that constituted the automated telephone directory system. this experience is generally reflective of the approach in many libraries and worthy of note and attention. the repeated respondent use of “negative closure” strategies and the isolated incidences of disrespectful behavior serve as a reminder to all libraries that assurance of the quality of the user experience may require staff training in both reference and customer service standards. the authors noted the reliance on digital formats to answer questions. it would be interesting to discover the extensiveness of the print collections available to the staff in the telephone reference services in these 25 libraries. the findings of this study will be of interest to all types of libraries who provide telephone reference services. work cited reference and user services association. “guidelines for behavioral performance of reference and information service providers.” june 2004. american library association. 29 february 2008. <http://www.ala.org/ala/rusa/protoc ols/referenceguide/guidelinesbehavi oral.cfm>. evidence based library and information practice 2011, 6.4 4 evidence based library and information practice acknowledgement of editorial advisors 2011. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. in 2011, the following people volunteered their time as editorial advisors, providing blind peer review of articles and evidence summaries that were submitted to evidence based library and information practice. the editorial team thanks the editorial advisors for their time, knowledge, and dedication which have contributed to the success of this publication. erin alcock, memorial university of newfoundland, canada kalyani ankem, emporia state university, united states of america andrea baer, united states of america greg bak, library and archives canada, canada anthony bernier, san jose state university, united states of america alissa black‐dorward, fordham university school of law, united states of america kate boddy, peninsula college of medicine and dentistry universities of exeter and plymouth, united kingdom andrew booth, university of sheffield, united kingdom sian brannon, university of north texas, united states of america jeanette buckingham, university of alberta, canada deborah charbonneau, wayne state university, united states of america trish chatterley, university of alberta, canada evidence based library and information practice 2011, 6.4 5 sara chiessi, italy amanda click, american university in cairo, egypt chris cooper, peninsula technology assessment group, united kingdom lisa cotter, newcastle university, australia james eric davies, loughborough university, united kingdom kathleen de long, university of alberta, canada su epstein, saxton b. little free library, united states of america alison farrell, memorial university of newfoundland, canada nancy fawley, virginia commonwealth university in qatar, united arab emirates bill fisher, san jose state university, united states of america carol gordon, rutgers school of communication & information, united states of america k. alix hayden, university of calgary, canada tony horava, university of ottawa, canada zaana howard, swinburne university / fusion consulting, australia joanne jordan, keele university, united kingdom anthi katsirikou, university of piraeus, greece catherine king, national center for immunisation research and vaccine preventable diseases, australia laura kuo, hunter college, united states of america vincent larivière, l'université du québec à montréal (uqam), canada suzanne lewis, gosford hospital library, australia michael lines, university of victoria, canada yazdan mansourian, tarbiat moallem university, iran sara marcus, queens college, cuny, united states of america marcia mardis, wayne state university, united states of america evidence based library and information practice 2011, 6.4 6 peter marques, ontario workplace tribunals library, canada christine marton, university of toronto, canada russell mccaskie, department of the treasury, australia paula mcmillen, university of nevada las vegas, united states of america misa mi, oakland university william beaumont school of medicine, united states of america dan mirau, concordia university college of alberta, canada obianuju mollel, alberta health services libraries, canada athulang mutshewa, university of botswana, botswana cleo pappas, university of illinois at chicago, united states of america françoise pasleau, university of liege, belgium margo pickworth, shore preparatory school, australia t. scott plutchak, university of alabama at birmingham, united states of america asim qayyum, charles sturt university, australia helen robertson, university of calgary, canada ann roselle, phoenix college, united states of america robert russell, northern state university, united states of america shawky salem, alex centre for multimedia and libraries, egypt alvin schrader, university of alberta, canada allison sivak, university of alberta, canada becky skidmore, society of obstetricians and gynaecologists of canada, canada daniella smith, university of north texas, united states of america yoo‐seong song, university of illinois, united states of america mark spasser, palmetto health, united states of america emily symonds, university of louisville, united states of america evidence based library and information practice 2011, 6.4 7 sharon tabachnick, southern college of optometry, united states of america donna timm, louisiana state university, united states of america ingrid tonnison, northern sydney central coast health, australia barbara wildemuth, university of north carolina, united states of america leslie williams, university of colarado, united states of america alison yeoman, aberystwyth university, united kingdom li zhang, university of saskatchewan, canada / evidence based library and information practice ebl 101   evaluating the results of evidence application, part one   virginia wilson client services librarian murray library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   originally published in: evidence based library and information practice, 5(3), 73–74. https://ejournals.library.ualberta.ca/index.php/eblip/article/view/8973/7352     received: 03 aug. 2010 accepted: 06 aug. 2010      2016 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   after the question has been formulated and the evidence has been found, appraised, and applied, the next step is evaluation. it is easy to move on to something else and skip the evaluation. lack of time and the desire to get on with what needs to be done next can thwart our best intentions. but evaluation is an important step in evidence based library and information practice (eblip) and it should be carried out on two levels: the practitioner level, where the person undertaking eblip looks at his or her own performance in the process; and the practice level, where what has been implemented is assessed. part one of “evaluating the results of evidence application” will deal with evaluation at the practitioner level.   in the past few years, the idea of reflective practice has been written about in conjunction with eblip. cultivating the practice of reflection is helpful in all realms of professional practice, and especially helpful in terms of eblip, because it helps practitioners continue to learn and grow in daily practice. booth (2004) writes about reflection in evidence-based practice for information professionals: a handbook. there, he talks about the notion of reflective practice as first discussed by donald a. schön in the reflective practitioner: how professionals think in action. schön (1983) differentiates between “reflection-on-action” and “reflection-in-action”: essentially reflecting after the fact and reflecting as you go.   grant (2007) published a systematic review which found that published reflections among library and information professionals have shifted from “reminiscence and retrospective accounts of careers and organizational development” lacking much analytical insight to “analytical accounts of reflection both ‘on’ and ‘in’ action” (p. 164). and in the last issue of evidence based library and information practice, koufogiannakis (2010) offers some thoughts on reflection in practice where she states that “research knowledge only takes us so far” and that through reflection, we can gain “a better understanding of how and why we do what we do as librarians and information professionals” (p. 2).   evaluating your own performance as an evidence based practitioner involves contemplating and analyzing what you did, or in other words, reflecting on action. booth (2004) suggests some questions to ask yourself:   ·         did i ask a specific focused question? ·         did i find efficiently the best evidence to answer my questions? ·         did i evaluate the evidence reliably according to validity and usefulness? ·         did i apply the results of the research appropriately to a specific user or group of users? (p. 127)   of course, you can reflect in more (or less) detail on your own performance. booth (2004) claims that “[reflective practice] can be as simple, or as complicated, as you want it to be” (p. 130).   reflection should also take place throughout the eblip process – reflection-in-action. while it may take awhile to remember to stop and reflect, especially if the project has momentum and you are worried that stopping to ponder might jeopardize it, getting into the habit of thinking about daily practice as it happens will help strengthen reflection as a continuous process in practice. thoughtfully considering what is going on as it happens brings an element of consciousness to professional practice, and can help get away from the “we’ve always done it this way” trap.   so how might an evidence based practitioner reflect in action? booth (2004) suggests several ways to get started. solo reflection might take place in a diary or with a mentor on an ongoing basis. group reflection can be looked at as action learning, where the group collaboratively, self-reflectively, and critically documents it at all states of the process. in the era of web 2.0, reflection can take place on a blog or a wiki, on a social networking site such as facebook with a group of like-minded professionals, or however else you might currently utilize 2.0 technologies. it would be beneficial to create communities of reflective practice where experiences and knowledge could be shared.   try to make time for reflecting upon and evaluating your own performance as an evidence based librarian or information professional. the more you self-assess, the easier the process will become and the more you will learn about using evidence in practice. next time, i’ll look at evaluating the results of evidence application at the practice level.   references   booth, a. (2004). evaluating your performance. in a. booth & a. brice (eds.), evidence‐based practice for information professionals: a handbook (pp. 127-137). london: facet.   grant, m.j. (2007). the role of reflection in the library and information sector: a systematic review. health information and libraries journal, 24(3), 155-166.   koufogiannakis, d. (2010). thoughts on reflection. evidence based library and information practice, 5(2), 1-3.   schön, d.a. (1983). the reflective practitioner: how professionals think in action. new york: basic books. evidence based library and information practice 2012, 7.3 51 evidence based library and information practice article mining the cultural evidence: situating planning and leadership within the academic library culture lyn currie head, education & music library university of saskatchewan saskatoon, saskatchewan, canada email: lyn.currie@usask.ca carol shepstone university librarian mount royal university calgary, alberta, canada email: cshepstone@mtroyal.ca received: 5 dec. 2011 accepted: 2 july 2012 2012 currie and shepstone. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/bync-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – this study investigated organizational culture in two academic libraries in order to propose culturally responsive strategies for developing planning and leadership initiatives. a case study conducted at the university of saskatchewan library (shepstone & currie, 2008) was replicated at two other canadian academic libraries to generate some comparative data on organizational culture in canadian academic libraries. methods – the competing values framework (cameron & quinn, 1999, 2006) provided the theoretical framework and the methodology for diagnosing and understanding organizational culture. the organizational culture assessment instrument (ocai) was administered by questionnaire to all library staff at mount royal university and carleton university libraries. results – scores on the ocai were used to graphically plot and describe the current and preferred culture profiles for each library. we compared the cultures at the three libraries and proposed strategies for initiating planning and developing leadership that were appropriate for the preferred cultures. mailto:lyn.currie@usask.ca mailto:cshepstone@mtroyal.ca evidence based library and information practice 2012, 7.3 52 conclusions – this research demonstrates that academic library culture can be diagnosed, understood, and changed in order to enhance organizational performance. examining organizational culture provides evidence to guide strategy development, priority setting and planning, and the development of key leadership abilities and skills. creating culturally appropriate support mechanisms, opportunities for learning and growth, and a clear plan of action for change and improvement are critical. introduction there is a growing interest within the library sector in the role organizational culture plays in shaping the workplace and contributing to the effectiveness and success of the organization. an analysis of organizational culture provides a context and starting point for creating a road map for change and continued organizational development. a clear understanding of the organizational cultures can help libraries to grow and thrive, and help determine the right pathways for organizational change (roberts, 2009). culture is often defined as the sum of activities – symbolic and instrumental – that exist in the organization and create shared meaning. socialization is the process through which individuals acquire and incorporate an understanding of those activities. organizational culture gives identity, provides collective commitment, builds social system stability, and allows people to make sense of the organization (sannwald, 2000). mining the cultural evidence provides rich organizational data to inform planning. assessing the organizational culture provides evidence of the collective will and the norms at play within an organization at a particular point in time, how the members of the organization might want to change and reshape these norms, and how these patterns might influence future success of the organization. studying the cultural dynamics of an organization also enables us to recognize the shared goals and actions that are most likely to succeed and how they can be best implemented. a research study in 2006 at the university of saskatchewan (u of s) library explored the organizational cultures of the library and proposed actions to implement culture change and achieve organizational transformation and renewal (shepstone & currie, 2008). at the time of the study, 15 of 38 librarians were new to the library and addressing their socialization and acculturation (black & leysen, 2002) raised questions concerning the impact of the library’s culture on their work. in addition, analyzing the library’s culture would also inform the strategic planning process and contribute to the transformation and renewal initiated by the new dean of the library. having completed that study we were interested in comparing our findings at the u of s with other canadian academic libraries. an opportunity to do this was pursued in 2009 when two of the researchers, recently appointed as senior administrators at carleton university and mount royal university, replicated the u of s study. the u of s study had focused on identifying culture preferences and proposing strategies to achieve a culture change. in the new studies the researchers examined culture preferences in order to focus on planning and leadership, key elements for change that had been identified in the 8rs study of human resource trends in canadian cultural industries (8rs research team, 2005). (note that mount royal university officially moved from college to university status in september 2009. the data for this article were gathered prior to this name change.) the researchers also hoped that generating some comparative data on organizational culture in canadian academic libraries would provide a basis for further research on the academic library culture in canada. evidence based library and information practice 2012, 7.3 53 literature review organizational culture and change we have reported previously on the research that demonstrates the importance of assessing culture in order to achieve significant and lasting change in an organization (shepstone & currie, 2006, 2008). understanding an organization’s culture is essential for managing change and improving institutional performance (gregory, 2008; quinn, 1988; schein, 2004). tierney (2008) comments that understanding organizational culture is critical for those who recognize that academe must change but are unsure how to make that change happen. an understanding of culture enables an organization’s participants to interpret the institution to themselves and others, and in consequence to propel the institution forward. for any organizational change to be sustainable there need to be changes to perceptions, beliefs, patterns of behaviour and norms, and ways of sense-making that have developed over long periods of time. the culture of an organization creates behavioural expectations that direct employees to act in ways that are consistent with its culture. behaviour change then is critical to the success of any culture change. institutionalizing change in an organizational culture requires a conscious attempt to show people how the new approaches, behaviours, and attitudes have helped improve performance, and taking sufficient time to ensure the next generation of leaders and managers personify the new approach (kotter, 1996). the “seven s” model of waterman, peters, and phillips (1980) recognized that successful culture change may require a change in structure, symbols, systems, staff, strategy, style of leaders, and skills of managers. as learning and knowledge-creating organizations, academic libraries are places where people continually expand their capacity to create the results they desire and where new and expansive patterns of thinking are nurtured (senge, 1990). garvin (1993) describes a learning organization as skilled at creating, acquiring, and transferring knowledge and at modifying its behaviour to reflect new knowledge and insights. an organization that modifies rather than reinforces behaviour needs a schema of socialization that allows for creativity and difference to flourish, and encourages new members to participate in the re-creation rather than merely the discovery of a culture. organizational culture and leadership a full, nuanced understanding of an organization’s culture assists leaders in articulating decisions in a way that speaks to the needs of members of the organization and marshals their support. when we use a cultural perspective we have a better understanding of how seemingly unconnected acts and events fall into place and how to help the organization’s members move forward. an awareness of organizational culture encourages leaders to consider real and potential conflicts within the organization, to recognize structural or operational contradictions that suggest tensions in the organization, to implement and evaluate everyday decisions with a keen awareness of their role and influence on organizational culture, to understand the symbolic dimensions of ostensibly instrumental decisions and actions, and to consider why different groups in the organization have varying perceptions about institutional performance (tierney, 2008). numerous theoretical frameworks for studying leadership in higher education institutions have been proposed, such as baldridge’s (1971) tripartite model of academic governance, which characterizes organizational types and how leadership manifests its character in each. studies of leadership in the postsecondary sector, as well as the public, business, and military sectors, have given rise to the emergence of organizational theories of ambiguity, organized anarchy, garbage can processes, and loose coupling (cohen & march, 1974; march evidence based library and information practice 2012, 7.3 54 & olsen, 1986; mohr, 1982; scott, 1981; weick, 1979). recent research has focussed on shared governance as a form of collaborative leadership which incorporates the specialized knowledge and experience from all staff and increases the effectiveness of policy-making, to bring a broader range of experience and knowledge to weigh on decision-making than traditional hierarchical leadership (escover, 2008; hansen, 2009). gobillot (2009) argues that “connected leadership” involves leaders engaging with employees, improving performance by building trust, and giving meaning to workplace relationships. the aim of leadership is to secure engagement, alignment, accountability, and commitment. researchers have also investigated leadership, change, and institutional effectiveness within postsecondary institutions (kezar & lester, 2009; kuh & whitt, 1988; tierney, 2008). bergquist and pawlak’s (1992, 2008) analysis of the interaction of academic cultures and the leadership practices needed to engage all six cultures has contributed to our understanding of organizational behaviour in higher education. the six cultures operating in the academy – the collegial, managerial, developmental, advocacy, virtual, and tangible – are what make higher education institutions so challenging to learn in, work in, administer, and lead. cameron and quinn (1999, 2006) explored the relationship between leadership roles and managerial skills, and personal and organizational effectiveness, in order to identify the leadership competencies most needed to support an organizational culture change process. they found that in organizations with a dominant culture type, the most effective managers and highperforming leaders demonstrate a matching leadership style while parenthetically developing capabilities and skills that allow them to succeed in other culture types. leaders operate both within the context of the culture and as change agents upon the culture. pors (2008) explored the relationship between library directors’ behaviour, style, and propensity to acquire information, and the direction and change processes in libraries. he argued that leadership is an important element in the configuration of organizational culture, and both leadership styles and the leader’s approach to innovation, change, and competency development are important in relation to the directions of the organization. bolman and deal (2008) developed four perspectives or frames for understanding organizational leadership (structural, human resource, political, and symbolic), and described the leadership values evident in each. they concluded that for leaders to be successful, they need the ability to see organizations as organic forms in which needs, roles, power, and symbols must be integrated to provide direction and shape behaviour. the literature on library leadership (garvin, edmonson, & gino, 2008; hernon & rossiter, 2007; hernon & schwartz, 2008; mathews, 2002; mech & mccabe, 1998; riggs, 1999) discusses the emergence of leadership theories and styles, such as situational, distributed, authentic, transactional, and transformational leadership, and focuses on examining leadership competencies and effectiveness. for maloney, antelman, arlitsch, and butler (2010), organizational culture defines and creates leaders – those who have the ability to recognize changes in the external environment that necessitate internal change and are able to lead an adaptation of their own organization’s culture to meet new challenges. some researchers have been critical of the lack of evidence-based research on library leadership. weiner (2003) claims that many aspects of leadership have not been addressed and a comprehensive body of cohesive, evidence based research is needed. lakos (2007) supports creating a culture of assessment and argues for leadership that enables a library to accept evidence based management based on the use of data in planning and decision-making. any discussion of leadership attributes appropriate to the culture of the organization also needs to account for the diversity in the evidence based library and information practice 2012, 7.3 55 workforce, particularly along generational lines. the extensive literature on the influence of generational perspective includes descriptions of the perceptions of desired leadership traits as evidenced by traditionalists, baby boomers, gen-xers, generation jones, and millennials, to name a few (beck, 2001; howe & strauss, 2000; lancaster & stillman, 2002; martin, 2006; ulrich & harris, 2003; wellner, 2000; young, hernon, & powell, 2006). researchers emphasize the need for a creative and constructively engaged workforce, and an environment that accommodates the needs and wants of each generation, and acknowledges that the workplace will progress only when an intergenerational dialogue is encouraged, nurtured, and becomes a seamless part of the operating environment. organizational culture and planning identifying an organization’s culture plays an important role in implementing a successful planning process. mcclure (1978) claims that revealing the current dominant values and beliefs of an organization is a critical foundational step in developing a planning process. planning to plan is where organizational culture plays its most vital role and where cultural norms will either facilitate or impede further planning decisions. planning based on shared outcomes, vision, and mission, and a discussion of past success and future milestones, is a key component of any effort to change a library’s culture (russell, 2008). identifying organizational culture norms and aspirations is helpful in determining the most advantageous planning processes for a particular organization. exploring organizational culture can also be instrumental in determining an organization’s readiness for change. for schein (2004) it is a question of whether the organization is “unfrozen” and ready for change or suffering from inertia and unwillingness to consider change (p. 325). strategic planning, when grounded in organizational culture awareness, provides guidance in how to balance potentially quick wins with those areas that may take more patience and effort to come to fruition. in all planning activities and processes, engagement and readiness are perhaps the most critical factors in the ultimate success of the plan. the best-constructed planning processes, with the most creative or tested methods, may not come to a successful and workable plan if an organization’s culture is not fully and actively considered. bolman and deal (2008) observe that organizational structures and processes such as planning, evaluation, and decision-making are often more important for what they express than for what they accomplish. an organization’s culture is revealed and communicated through its symbols, myths, vision, and values. at harvard university, for example, professors are bound less by structural constraints than by rituals of teaching, values of scholarship, and the myths and mystique of harvard. leaders who understand the significance of symbols can shape more cohesive and effective organizations so long as the cultural patterns are aligned with the challenges of the marketplace. there is a substantial body of research that also offers longitudinal evidence linking culture to organizational effectiveness and success (baker, riesing, johnson, stewart, & day baker, 1997; cameron, 1986; collins, 2001; collins & porras, 1994; kotter & heskett, 1992; linn, 2008; lysons, hatherly, & mitchell, 1988; quinn & cameron, 1983; quinn & rohrbaugh, 1983). assessing organizational culture we have discussed elsewhere (shepstone & currie, 2006) the value of assessing culture as a necessary first step when undertaking organizational change, renewal, and improvement. change involves changes to fundamental perceptions, beliefs, patterns of behaviour, and norms and ways of sensemaking that have developed over long periods of time. plans for change must be carefully integrated into existing culture, recognizing the potential points of resistance and finding opportunities to build on existing strengths. evidence based library and information practice 2012, 7.3 56 the research on organizational culture and change and the research frameworks and methodologies that have been developed, in particular the extensive application of cameron and quinn’s (1999, 2006) competing values framework (cvf) to assess culture, has been well documented (giek & lees, 1993; gregory, 2008; lamond, 2009; paulin, ferguson, & payaud, 2000; sendelbach,1993; stevens, 1996; thompson, 1993). much of the literature that analyzes library culture draws on the cvf to investigate the question of culture and subcultures (faerman, 1993; kaarst-brown, 2004; lakos & phipps, 2004; maloney et al., 2010; salanki, 2010; shepstone & currie, 2008; varner, 1996). aims our review of the literature revealed three areas that we wanted to address in framing our research:  there were no studies of organizational culture in canadian academic libraries. we wanted to produce a study that could generate some interest in comparative research on academic library culture in canada.  as bergquist and pawlak (2008) observe, cultural analyses yield important insights into the life and dynamics of an organization but they often provide little guidance to the organizational leader for engaging those cultures. it was our intention to provide such guidance to the senior leadership by identifying specific strategies appropriate to the cultures of the libraries under investigation.  there is a need for applied research that leads to practical actions. lowry (2011) is critical of the cvf model, claiming it “leads to assessments that find all four archetypes at work in a library and, thus, lead to generalizations without much precision that may not lead to effective action” (p. 3). in undertaking this research we were primarily interested in producing an action plan or set of strategies for developing leadership and planning processes that would be effective in the desired culture. the new case studies therefore set out to explore three questions: 1. what is the current as opposed to the preferred culture of each library? 2. what strategies are appropriate for initiating planning and developing leadership in the preferred culture of each library? 3. what comparisons of the current and preferred cultures can be drawn from the three libraries? methods we defined organizational culture as a collective understanding, a shared and integrated set of perceptions, memories, values, attitudes, and definitions that have been learned over time and which determine expectations (implicit and explicit) of behaviour that are taught to new members in their socialization into the organization (shepstone & currie, 2008). the case study method was used to undertake this site-specific exploration of organizational culture. by delineating and describing key dimensions of culture via case study, a more intense analysis and specific understanding of organizational culture are possible (tierney, 2008). the case study method is useful as an exploratory technique when applied to investigations of organizational performance, structure, and functions (hernon & schwartz, 2008). we chose mount royal university and carleton university libraries because two of the three researchers worked at those institutions and could provide local oversight of the study. applying the competing values framework cameron and quinn’s competing values framework (cvf) provided a theoretical framework for understanding organizational culture. it offered a process for identifying what needs to change in an organization’s evidence based library and information practice 2012, 7.3 57 culture and for developing a strategy to initiate a culture change process. the cvf also employs a reliable and validated instrument, the organizational culture assessment instrument (ocai), for diagnosing culture. cameron and quinn (2006) collected cultural profiles using the ocai from more than 3,000 organizations to develop “typical” dominant culture types for organizations from a number of sectors. the instrument has been used in numerous organizational studies that have all tested the reliability and validity of both the instrument and the approach (kalliath, bluedorn, & gillespie, 1999; peterson, cameron, spencer, & white, 1991; quinn & spreitzer, 1991; yeung, brockbank, & ulrich, 1991; zammuto & krakower, 1991). cameron and freeman (1991) produced evidence for the validity of the ocai in their study of organizational culture in 334 institutions of higher education. zammuto and krakower (1991) used this instrument to investigate the culture of higher education institutions. using the cvf offered an opportunity to compare the library findings to these “average” dominant cultures in other higher education organizations, thus providing benchmark data. as we noted elsewhere (shepstone & currie, 2006) the methodology is appealing in its simplicity both in application and interpretation. the ocai is easy for participants to complete and straightforward for researchers to score and analyze. the ability to graphically represent or plot the scores helps to describe and communicate the findings in a meaningful way and stimulates a high level of interest and engagement in the organizational assessment (varner, 1996). a description of the cvf used in this study is provided in the appendix. the ocai was administered by questionnaire to all staff of each library inviting their participation in the study. part one of the questionnaire gathered participant data for each institution. part two, the current organizational culture assessment, required responses to six questions on the ocai to reflect perceptions of the current state of the library. the questions contained four descriptions of academic libraries and respondents were to distribute 100 points among the four descriptions depending on how similar the descriptions were to their library. part three required responses to the same six questions scored according to how the library should be in five years in order to be highly successful, thus identifying the preferred organizational culture. in gathering participant data we were interested in identifying possible subcultures among different groupings of staff, such as by functional area, level of administrative responsibility, years of service, age range, and generational “group.” an assurance of anonymity for all respondents was issued. the questionnaire was distributed giving participants two weeks to respond. two subsequent follow-up notices were distributed a week apart in an effort to increase the number of participants. we hired a graduate student to score the responses and plot the culture profiles for the u of s study, using the instructions provided by cameron and quinn (2006). this work was completed by the two researchers administering the study at mount royal and carleton. results in reporting on the results we have included the u of s data from the 2006 study for comparison purposes. details of the responses received and the response rates for each institution are provided in table 1. while librarian responses at the three institutions (67%, 62%, and 73%) were statistically significant, the response rates for the support staff were considered too low to be statistically significant. we therefore limited our analysis of the data to the librarian responses. we were unable to account for the low response rate for support staff across the three institutions except to note that at the u of s the administration of the questionnaire to support evidence based library and information practice 2012, 7.3 58 staff followed two other major staff surveys both on campus and within the library, which suggests the low response rate might in part be attributed to survey fatigue. in order to identify possible subcultures among different groupings of staff, we collected participant data on functional unit, level of administrative responsibility, public versus technical services affiliation, years of service, etc. however, given the small subpopulation sizes involved and a requirement by the research ethics review boards at each institution to guarantee anonymity of respondents, we were not able to report these results. this is one of the unfortunate limitations of case study research involving small populations. using the librarians’ scores on the ocai, the current and preferred organizational culture profiles for each library were constructed by plotting the average scores for each alternative (clan, adhocracy, market, and hierarchy) on the diagonal lines in each quadrant. we drew culture profiles for each library to compare the current and preferred cultures across the three libraries. (see figure 1.) when interpreting the culture plots, an analysis of scoring should be sensitive to differences of 10 points or more, according to cameron and quinn (2006). the plots revealed three academic libraries with distinctly different current cultures as perceived by the librarians in each, and three similar preferred culture profiles. current cultures at the u of s library, librarians scored the library highest in the market culture, indicating a focus on productivity, external positioning, competitive actions, market leadership, achievement of measurable goals and targets, and a prevailing concern with stability and control. at carleton university library, the library scored highest in the hierarchy culture, indicating a formalized and structured workplace where rules and policies hold the organization together, procedures govern what people do, leaders are coordinators and organizers, and maintenance of a smoothrunning organization, stability, predictability, and efficiency prevail. mount royal college library scored highest in the clan culture, characterized by a focus on people and relationships, a sense of cohesion, participation, and belonging, and an organization held together by loyalty and high commitment where long-term goals, teamwork, consensus, and individual development are valued and emphasized. table 1 survey responses university of saskatchewan mount royal college carleton university surveys distributed librarians 36 13 30 support staff 109 45 76 total 145 58 106 responses received librarians 24 8 22 support staff 32 16 25 total 56 24 57 response rate librarians 67% 62% 73% support staff 29% 36% 33% total 39% 41% 54% evidence based library and information practice 2012, 7.3 59 preferred cultures a comparison of the preferred culture profiles for the three libraries revealed a common desire for a transition to an adhocracy culture (mount royal by 10 points, u of s by 27 points, and carleton by 15 points). there was also a preference for stronger elements of a clan culture at carleton university (by 10 points) and the u of s (by 11 points). the u of s librarians preferred a culture with a reduced market orientation and increased adhocracy elements such as innovation and autonomy, along with increased clan characteristics such as a focus on the individual and a more personalized workplace. for mount royal library the preference was for a significant increase in innovation and autonomy of an adhocracy culture with maintenance of the existing clan elements. carleton university librarians demonstrated a preference for increasing both adhocracy and clan elements and significantly decreasing the prevailing hierarchy culture. discussion organizational culture in the higher education sector movement toward a preferred organizational culture must consider the larger cultural and political context in order to have success. it is instructive to consider the organizational culture characteristics of the university within which the library operates. cameron and quinn have mapped composite or common cultural characteristics based on organizational type or sector. academic libraries, as integral parts of much larger organizations, are influenced by and reflective of the cultural characteristics of their parent institution. research that has explored organizational culture within academic settings (baker et al., 1997; lysons, hatherty, & mitchell, 1998; pors, 2008) has derived a common cultural profile of academic institutions. post-secondary educational organizations typically exhibit organizational cultures that are strong in adhocracy with an emphasis on hierarchy characteristics (cameron & quinn, 2006). these competing values are a logical finding as post-secondary institutions have extremely entrenched structures of hierarchy and rank while engaged in the business of creating new knowledge and ideas through research and teaching. the pursuit of simultaneous contradiction has been found to be highly successful in colleges and universities in coping with conditions of uncertainty, complexity, and turbulence. a desire for a stronger adhocracy culture aligns with learning organizations. all three of the libraries in this study, however, spoke to a desire to enhance or maintain significantly figure 1 library culture profiles – librarians evidence based library and information practice 2012, 7.3 60 high clan cultural characteristics. this finding raises questions about the ability to achieve this within a library in a parent institution where clan qualities might not be as valued or visible. more precisely, to what degree are these three libraries congruent with their own parent-institutions’ cultural characteristics? although this question was not explored in this study, it may influence how the library participates in and supports the mission of the institution, as well as how successfully the library adapts, interacts, and works with other campus units, or how it supports and engages with the students, faculty, and staff. organizational effectiveness organizations tend to develop a dominant organizational culture over time as they adapt and respond to challenges and changes in the environment. paradoxically, organizational culture creates both stability, by reinforcing continuity and consistency through adherence to a set of consensual values, as well as adaptability, by providing a set of principles to follow when designing strategies to cope with new circumstances (cameron & quinn, 2006). cameron and quinn’s research emphasizes the need for organizational flexibility and adaptability in order to draw on all four cultural quadrant skills and values, and argues that it is in the tension and balance of competing values that organizations are best able to maintain effectiveness and organizational health. while there may be dominant cultural characteristics more appropriate to an individual organization or particular type of institution, it is important for organizations to be able to draw on the full range of resources and competing characteristics, depending on the situation and need. organizational effectiveness is inherently paradoxical. to be effective, an organization must possess attributes that are simultaneously contradictory, even mutually exclusive (cameron, 1986). it follows then that those in leadership positions must be able to draw upon skills and strategies from a similar range of competing perspectives. understanding when to shift foci from internal to external, from process-based to creative, are important competencies and abilities for leaders to exercise. conclusions our research was undertaken to identify the current and preferred organizational cultures of three canadian academic libraries, and to suggest strategies appropriate for initiating planning and developing leadership skills and attributes aligned with the preferred culture of each library. understanding the existing culture, and identifying the type of culture preferred by library staff, is a first step in achieving a culture change. by focusing on the area of incongruence between the current and preferred cultures, the changes that are desired can be identified. the evidence gathered about existing and preferred cultural traits can be used to guide strategy development, priority setting and planning, and the development of key leadership abilities and skills for libraries. developing institution-specific and culturally responsive strategies. it is important to identify the behaviours and competencies that are needed to reflect the new culture. for the u of s study we mapped the leadership roles and managerial competencies to the quadrants of the cvf to illustrate the behaviours leaders and managers at all levels should adopt and where to focus their skill development (shepstone & currie, 2008). given the similar cultural findings at carleton and mount royal libraries we believe this list of competencies and attributes would be relevant in these libraries. for the u of s library we developed an action plan with strategies that address innovation, continuous improvement, teamwork, interpersonal relationships, and staff development – all characteristics of the adhocracy and clan cultures desired by the u of s librarians. in order to develop the desired cultural characteristics of adhocracy and clan cultures at the mount royal and carleton evidence based library and information practice 2012, 7.3 61 libraries, we propose the following key strategies to help this culture change process unfold (table 2). the evidence gathered from our research has confirmed and informed strategic planning and implementation at mount royal and table 2 key strategies for building clan and adhocracy cultures a building clan – “collaborate” building adhocracy – “create” focus on teams, relationship building, and staff development: teams:  build cross-functional teamwork opportunities  develop programs to increase teambuilding skill  emphasize inter-unit mobility and crossfunctional communication relationship building:  improve relations between front-line and support operations  build inter-unit staff relationships and develop expectations for working together  improve communication and reduce “silos” between faculty/staff and unit/area staff  identify items needing coordination and collaboration between units staff development:  expand staff involvement in planning, decision making & problem solving  establish operational and strategic planning groups & opportunities – communicate to leaders how strategic pressures are impacting the library and how this might impact their roles  empower front-line staff and supervisors to make key decisions and react quickly to emerging needs  provide an employee recognition system that recognizes contributions and commitment focus on the future, innovation, and continuous improvement: future:  revisit organizational values and vision to encourage a focus on the future  appoint champions/leads responsible for monitoring /tracking major issues and identifying most advantageous areas for growth and development  focus on forecasting/anticipating and exceeding client needs and new expectations  plan for long and short term and ensure the process stretches current assumptions innovation:  ensure vision statement inspires creative initiative  develop ways to encourage, measure, and reward innovative behaviour of individuals and teams  recognize those activities that help ideas get developed and adopted  provide opportunities for staff to share new and experimental ideas. celebrate trial-and-error learning and take opportunities to learn from failure continuous improvement:  encourage discussion on creating and implementing change, and implement process improvement  move to flexible structures that emphasize adaptability, agility, and creativity  focus on the library as a learning organization and make changes to increase the capacity to learn more effectively  task front-line staff with conceptualizing new strategies for expanding/improving services a based on: cameron, k., and quinn, r. (2006). diagnosing and changing organizational culture based on the competing values framework. rev. ed. san francisco: jossey-bass. evidence based library and information practice 2012, 7.3 62 provided a visual representation and reminder for what is needed to successfully move closer to an adhocracy culture while maintaining and fostering the existing clan elements in the face of rapid growth, diversification, and transformation as the institution undergoes the transition from college to university status. this has translated into placing a greater emphasis on and support for individually focused professional and skill development and for an expansion of continuous learning and leadership opportunities. faculty are expanding their academic autonomy through newly formalized programs of scholarship, and expanding opportunities for teaching and for participation in shared, acting, or rotating leadership roles. project-based opportunities have been encouraged and new committee chairing opportunities have been developed. for support staff the emphasis has been on increasing staff engagement in planning and creating new ways to ensure meaningful participation at the library and unit levels. support staff have been encouraged to accept roles on task forces and projects, chair committees, and use opportunities for job enrichment and project work to increase skills, flexibility, experience, and job satisfaction. at carleton university the results of the study have contributed to discussions on strategic planning and organizational restructuring. increasing the clan culture required expanding staff participation in planning, reviewing the most significant gaps between the preferred culture and existing leadership styles, and ensuring transparency in decisionmaking and use of feedback. this has involved articulating what is currently done well, focusing on interrelationships and building collaboration between departments, and adopting more responsive and user-focused approaches. supporting research and innovation to build the desired adhocracy culture has required moving from a focus on boundaries and delineation of responsibilities to an articulation of big-picture goals, clarification of leadership roles, and a reexamination of resource allocation within the library. we undertook this research to generate a sampling of comparative data on organizational culture in canadian academic libraries. our findings, based on the perceptions of librarians, revealed different current cultural characteristics but similar preferred cultural characteristics for three academic libraries in canada. differences in institutional size, mandate, and age did not seem to impact librarians’ cultural preferences among these three libraries. further research to analyze current and preferred cultures in other canadian academic libraries would be interesting to determine if the preference for a shift to organizational cultures with a dominant adhocracy culture supported by strong clan elements found in these three libraries, applies more broadly and could be considered a national or sector-based trend. acknowledgements the authors would like to acknowledge the contribution of pat moore for her questionnaire administration and data analysis at carleton university library. references baldridge, j. (1971). academic governance. berkley, ca: mccutchan publishing co. baker, c. m., riesing, d.l., johnson, d. r., steward, r. l., & day baker, s. (1997). organizational effectiveness: toward an integrated model of schools of nursing. journal of professional nursing, 13(4), 246-255. barker, j. (1995). triggering constructive change by managing organizational culture in an academic library. library acquisitions: practice and theory, 19, 919. beck, m. e. (2001). the abcs of gen x for librarians. information outlook, 5, 11-20. evidence based library and information practice 2012, 7.3 63 bergquist, w. h., & pawlak, k. (2008). engaging the six cultures of the academy. san francisco, ca: jossey-bass. black, w. k., & leysen, j. m. (2002). fostering success: the socialization of entry-level librarians in arl libraries. journal of library administration, 36(4), 3-26. bolman, l. g., & deal, t. e. (1992). leading and managing: effects of context, culture and gender. educational administration quarterly, 28(3), 314320. bolman, l. g., & deal, t. e. (2008). reframing organizations : artistry, choice and leadership. san francisco, ca: josseybass. cameron, k. s. (1986). effectiveness as paradox: consensus and conflict in the conceptions of organizational effectiveness. management science, 32(5) (organization design), 539-553. cameron, k. s., & freeman, s. j. (1991). cultural congruence, strength and type: relationships to effectiveness. research in organizational change and development, 5, 23-59. cameron, k. s., & quinn, r. e. (2006). diagnosing and changing organizational culture: based on the competing values framework (rev. ed.). san francisco, ca: jossey-bass. cameron, k. s., quinn, r. e., degraff, j., & thakor, a. v. (2006). competing values leadership: creating value in organizations. cheltenham,uk: edgar elgin publishing. cohen, m. d., & march, j. g. (1974). leadership and ambiguity. boston, ma: harvard business school press. collins, j. c. (2001). good to great: why some companies make the leap and others don’t. new york, ny: harpercollins. collins, j. c., & porras, j. i. (1994). built to last: successful habits of visionary companies. new york, ny: harperbusiness. 8rs research team. (2005). the future of human resources in canadian libraries. edmonton, ab: 8rs canadian library human resources study. escover, m. (2008). creating collaborative leadership and shared governance at a california community university. new york, ny: edwin mellen press. faerman, s. (1993). organizational change and leadership styles. journal of library administration, 19(3/4), 55-79. garvin, d. a. (1993). building a learning organization. harvard business review, 71, 28-91. garvin, d. a., edmonson, a.c., & gino, f. (2008). is yours a learning organization? harvard business review, 86(3), 109-116. giek, d. g., & lees, p. l. (1993). on massive change: using the competing values framework to organize the educational efforts of the human resource function in new york state government. human resource management, 32, 9-28. gobillot, e. (2009). leadershift: reinventing leadership for the age of mass collaboration. london, england: kogan page. gregory, b. t., harris, s. g., & armenakis, a. a. (2009). organizational culture and effectiveness: a study of values, attitudes and organizational outcomes. journal of business research, 62(7), july, 673-879. hansen, m. t. (2009). collaboration: how leaders avoid the traps, create unity, and reap big results. boston, ma: harvard business press. evidence based library and information practice 2012, 7.3 64 hernon, p. & schwartz, c. (2008). leadership: developing a research agenda for academic libraries. library & information science research, 30, 243 249. hernon, p. & rossiter, n. (eds.). (2007) making a difference: leadership and academic libraries. westport, ct: libraries unlimited. hernon, p., powell, r. r., & young, a. p. (2003). the next library leadership: attributes of academic and public library directors. westport, ct: libraries unlimited. honea, s. m. (1997). transforming administration in academic libraries. journal of academic librarianship, 23, 183-190. howe, n., & strauss, w. (2000). millennials rising: the next great generation. new york, ny: vintage books. kaarst-brown, m. l., nicholson, s., von dran, g. m., & stanton, j. m. (2004). organizational cultures of libraries as a strategic resource. library trends, 53(1), 33-53. kalliath, t. j., bluedorn, a. c., & gillespie, d. f. (1999). a confirmatory factor analysis of the competing values instrument. educational psychological measurement, 59(1), 143-158. kezar, a. j., & lester, j. (2009). organizing higher education for collaboration: a guide for campus leaders. san francisco, ca: jossey-bass. kirchner, t. (2007). applying organizational theory to interlibrary loan and document delivery department. journal of interlibrary loan document delivery and electronic reserve,17(4), 7786. kotter, j. p. (1996). leading change. boston, ma: harvard business school press. kotter, j. p., & heskett, j. l. (1992). corporate culture and performance. new york, ny: free press. kreitz, p. a. (2009). leadership and emotional intelligence: a study of university library directors and their senior management teams. college and research libraries, 70(6), 531-54. kuh, g. d., & whitt, e. j. (1988). invisible tapestry: culture in american colleges and universities. washington, dc: ashe-eric higher education report series vol.1. lakos, a. (2007). evidence-based library management: the leadership challenge. portal: libraries and the academy, 7(4), 431-450. lakos, a., & phipps, s. ( 2004). creating a culture of assessment: a catalyst for organizational change. portal: libraries and the academy, 4(3), 345-361. lamond, d. (2003). the value of quinn’s competing values model in an australian context. journal of managerial psychology, 18, 46-59. lancaster, l.. & stillman, d. (2002). when generations collide. new york, ny: harper collins. langfred, c., & shanley, m. (1997). the importance of organizational context: a conceptual model of cohesiveness and effectiveness in work groups. public administration quarterly, 21(3), 349-369. linn, m. (2008). organizational culture: an important factor to consider. the bottom line: managing library finances, 21(3), 88-93. lowry, c. b. (2011). subcultures and values in academic libraries – what does climatequal research tell us? in 9th northumbria international conference on performance measurement in libraries evidence based library and information practice 2012, 7.3 65 and information services: proving value in challenging times. august 22-25, 110. lysons, a., hatherly, d., & mitchell, d. a. (1998). comparison of measures of organisational effectiveness in u.k. higher education. higher education, 36(1), 1-19. maloney, k., antelman, k., arlitsch, k., & butler, j. (2010). future leaders’ views on organizational culture. college and research libraries, july, 322-344. march, j. g., & olsen, j.p. (1986). garbage can models of decision making in organizations. in j.g. march & r.weissinger-baylon (eds.). ambiguity and command: organizational perspectives on military decision making. marshfield, ma: pitman. martin, j. (2006). i have shoes older than you: generational diversity in the library. the southeastern librarian, 54, 4-11. matthews, c.j. (2002). becoming a chief librarian: an analysis of transition stages in academic library leadership. library trends, 50(4), 578-604. mcclure, c. r. (1978). the planning process and strategies for action. college and research libraries, 39(6), 456-66. mech, t. f., & mccabe, g. b. (eds.). (1998). leadership and academic libraries. westport, ct: greenwood press. mohr, l. b. (1982). explaining organizational behavior. san francisco, ca: josseybass. neilson, g., pasternack, b. a., &van nuy, k. e. (2005). the passive aggressive organization. harvard business review, october, 82-92. patterson, m. g., west, m. a., shackleton, v. j., dawson, j. f., lawthom, r., maitlis, s., robinson, d. l., & wallace, a. (2005). validating the organizational climate measures: links to managerial practices, productivity and innovation. journal of organizational behaviour, 26(4), 379-408. paulin, m., ferguson, r. j., & payaud, m. (2000). effectiveness of relational and transactional cultures in commercial banking: putting client-value into the competing values model. international journal of bank marketing, 18, 328-337. peterson, m., cameron, k. s., spencer, m., & white, t. (1991). assessing the organizational and administrative context for teaching and learning. ann arbor, mi: national center for research to improve post-secondary teaching and learning. ed 338121. pors, n. o. (2008). management tools, organizational culture and leadership: an explorative study. performance management and metrics, 9(2), 138-152. quinn, r. e. (1984). applying the competing values approach to leadership: toward an integrative model. in j. g. hunt, r. stewart, c. schriesheim, & d. hosking, (eds.), managers and leaders: an international perspective. new york, ny: pergamon. quinn, r. e. (1988). beyond rational management. san francisco, ca: jossey-bass. quinn, r. e., & cameron, k. (1983). organizational life cycles and shifting criteria of effectiveness: some preliminary evidence. management science, 29(1), 33-51. quinn, r. e., & cameron, k. (1988). paradox and transformation: toward a theory of change in organization and management. cambridge, ma: ballinger publishing. quinn, r. e.. & rohrbaugh, j. (1981). a competing values approach to organizational effectiveness. public productivity review, 5(2) (a evidence based library and information practice 2012, 7.3 66 symposium on the competing values approach to organizational effectiveness), 122-140. quinn, r. e., & rohrbaugh, j. (1983). a spatial model of effectiveness criteria: towards a competing values approach to organizational analysis. management science, 29(3), 363-377. quinn, r. e., & spreitzer, b. m. (1991). the psychometrics of the competing values culture instrument and an analysis of the impact of organizational culture on quality of life. in r.w. woodman & w.a. pasmore (eds.), research in organizational change and development, 5. greenwich, ct: jai press. riggs, d. e. (1999). library leadership: observations and questions. college and research libraries, 60(1), 6-7. roberts, k. (2009). from outgrown and over managed to resilient organizations. feliciter, 55, 37-38. russell, k. (2008). evidence-based practice and organizational development in libraries. library trends, 56(4), 910-930. sannwald, w. (2000). understanding organizational culture. library administration and management, 14, 814. schein, e. (2004). organizational culture and leadership. san francisco, ca: jossey bass. scott, w. r. (1981). organizations: rational natural and open systems. englewood cliffs, nj: prentice hall. sendelbach, n. b. (1993). the competing values framework for management training and development: a tool for understanding complex issues and tasks. human resources management, 32(1), 75-99. senge, p. (1990). the fifth discipline. new york, ny: doubleday. shepstone, c., & currie, l. (2006). assessing organizational culture: moving towards organizational change and renewal. in proceedings of the library assessment conference, building effective, sustainable, practical assessment, september 25-27, 369-380. charlottesville, va. shepstone, c. & currie, l. ( 2008). transforming the academic library: creating an organizational culture that fosters staff success. journal of academic librarianship, 34, 358-368. smart, j. c. (2003). organizational effectiveness of 2-year colleges: the centrality of cultural and leadership complexity. research in higher education, 44(6), 673703. stevens, b. (1996). using the competing values framework to assess corporate ethical codes. journal of business communication, 33, 71-84. salanki, v. (2010). organizational culture and communication in the library: a study on organizational culture in the lucian blaga central university library. cluj philobiblom vool, xv, 455-523. thompson, m. p. (1993). using the competing values framework in the classroom. human resources management, 32, 101119. tierney, w. g. (2008). the impact of culture on organizational decision making: theory and practice in higher education. sterling, ma: stylus publishing. ulrich, j. m. & harris, a. l. (2003). genxegesis: essays on “alternative” youth (sub)culture. madison, wi: university of wisconsin press. varner, c. h. (1996). an examination of an academic library culture using a evidence based library and information practice 2012, 7.3 67 competing values framework. phd dissertation, illinois state university. waterman, r. h., peters, t. j., & phillips, j. r. (1980). structure is not organization. business horizons, june, 50-63. weick, k. e. (1979). the social psychology of organizing. (2nd ed.). reading, ma: addison-wesley. weick, k. e., & quinn, r. e. (1999). organizational change and development. annual review of psychology, 50(1), 361-386. wellner, a. s. (2000). generational divide. american demographics, 22, 52-58. yeung, a. k. o., brockbank, j. w., & ulrich, d. o. (1991). organizational culture and human resources practices: an empirical assessment. in r. w. woodman & w. a. pasmore, (eds.), research in organizational change and development, 5, greenwich, ct: jai press. young, a. p., hernon, p., & powell, r. r. (2006). attributes of academic library leadership: an exploratory study of some gen-xers. journal of academic librarianship, 32, 489-502. zammuto, r. f., & krakower, j. y. (1991). quantitative and qualitative studies of organizational culture. in r. r. woodman & w. a. pasmore. (eds.), research in organizational change and development, 5. greenwich, ct: jai press. evidence based library and information practice 2012, 7.3 68 appendix the competing values framework (cvf) the cvf distinguishes between two major intersecting dimensions in organizations. the horizontal dimension reflects the extent to which an organization has a control orientation – the degree of emphasis on flexibility, discretion, and dynamism as opposed to stability, order, and control. the vertical dimension reflects the extent to which the organization is focused on its internal or external functioning – the degree of internal orientation, integration, and unity as opposed to an external orientation, differentiation, and rivalry. these two dimensions form four quadrants that represent distinct organizational perspectives. each quadrant is identified as a cultural type representing opposite or competing assumptions, orientations, and values. thus four dominant culture types emerge from the framework. this is represented in figure 2. figure 2 competing values framework based on: cameron, k., and quinn, r. (2006). diagnosing and changing organizational culture based on the competing values framework. rev. ed. san francisco: jossey-bass. the clan culture is typified by a friendly place to work where people share a lot of themselves, leaders serve as mentors, the organization is held together by loyalty and tradition, commitment is high, the evidence based library and information practice 2012, 7.3 69 emphasis is on the long-term benefit of individual development, high cohesion and morale, and a premium is placed on teamwork, participation, and consensus. the adhocracy culture is characterized by a dynamic, entrepreneurial, and creative workplace where people take risks, leaders are visionary and innovative, the commitment to experimentation and innovation holds the organization together, readiness for change and meeting new challenges is important, and the emphasis is on being at the leading edge of new knowledge, services, and products. a market culture is a results-oriented workplace where leaders drive the organization toward productivity, results, and profits, an emphasis on winning holds the organization together, and the prevailing concern is on competitive actions and achieving goals, targets, and increasing its competitive position. the hierarchy culture is a formalized and structured place to work where formal rules and policies bind the organization, procedures govern what people do, effective leaders are good coordinators and organizers, maintenance of a smooth-running organization is all important, and the long-term concerns are stability, predictability, and efficiency. the survey instrument the organizational culture assessment instrument (ocai) is a data-gathering instrument based on six content dimensions which reflect the fundamental cultural values and implicit assumptions about the way an organization functions:  the dominant characteristics of the organization  leadership style  management of employees  organizational glue or bonding mechanisms  strategic emphases  criteria of success the ocai poses a series of statements that reflect the key elements in describing organizational culture. when organizational members respond to questions about these dimensions, the values underlying organizational culture can be uncovered. figure 3 provides a sample of the ocai as used in the study. evidence based library and information practice 2012, 7.3 70 figure 3 sample of ocai from cameron and quinn (2006), diagnosing and changing organizational culture: based on the competing values framework. rev .ed. jossey-bass: san francisco. part 2 organizational profile of the library please answer the following six questions to reflect your perception of the current state of the u of s library system. each of the questions contains four descriptions of academic libraries. please distribute 100 points among the four descriptions a, b, c, d depending on how similar the description is to the u of s library. none of the descriptions is any better than the others; they are just different. for each question, please use 100 points. 2.1 dominant characteristics (divide 100 points) a. ____________library a is a very personal place. it is like an extended family. people seem to share a lot of themselves. b. ____________library b is a very dynamic and entrepreneurial place. people are willing to stick their necks out and take risks. c. ____________library c is a very formalized and structured place. policies and procedures generally govern what people do. d. ____________library d is very competitive in orientation. a major concern is with getting the job done. people are very production oriented. evidence based library and information practice evidence based library and information practice 2011, 6.4 57 evidence based library and information practice article employers’ perspectives on future roles and skills requirements for australian health librarians suzanne lewis manager, library services central coast local health district new south wales, australia email: slewis@nsccahs.health.nsw.gov.au gillian hallam adjunct professor, information science discipline faculty of science and technology queensland university of technology queensland, australia email: g.hallam@qut.edu.au ann ritchie editor, australian library journal australian library and information association canberra, australia email: ann.ritchie@alia.org.au catherine clark assistant director, research and learning support (medicine and dentistry) information services the university of western australia perth, western australia email: catherine.clark@uwa.edu.au cheryl hamill librarian in charge fremantle hospital and health service fremantle, western australia email: cheryl.hamill@health.wa.gov.au melanie kammermann editor, hla news health libraries australia australian library and information association canberra, australia email: melkam88@yahoo.com.au mailto:slewis@nsccahs.health.nsw� mailto:ann.ritchie@alia.org.au� evidence based library and information practice 2011, 6.4 58 patrick o'connor librarian, toowoomba clinical library toowoomba, queensland, australia email: patrick_o'connor@health.qld.gov.au received: 15 may 2011 accepted: 15 oct 2011 2011 lewis, hallam, ritchie, clark, hamill, foti and o’connor. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – this study, which comprises one stage of a larger project (alia/hla workforce and education research project), aimed to discover employers’ views on how (or whether) health librarians assist in achieving the mission-critical goals of their organizations; how health librarians contribute to the organization now and into the future; and what are the current and future skills requirements of health librarians. methods – each member of the project group approached between one and five individuals known to them to generate a convenience sample of 22 employers of health librarians. there were 15 semi-structured interviews conducted between october and november 2010 with employers in the hospital, academic, government, private, consumer health and not-for-profit sectors. the interview schedule was sent to each interviewee prior to the interview so that they had time to consider their responses. the researchers wrote up the interview notes using the interview schedule and submitted them to the principal researcher, who combined the data into one document. content analysis of the data was used to identify major themes. results – employers expressed a clear sense of respect for the roles and responsibilities of library staff in their organizations. areas of practice such as education and training, scientific research and clinical support were highlighted as critical for the future. current areas of practice such as using technology and systems to manage information, providing information services to meet user needs and management of health information resources in a range of formats were identified as remaining highly relevant for the future. there was potential for health librarians to play a more active and strategic role in their organizations, and to repackage their traditional skill sets for anticipated future roles. interpersonal skills and the role of health librarians as the interface between clinicians and information technology were also identified as critical for the future. conclusions – interviews with employers provided valuable insights into the current and future roles and skills requirements of health librarians in australia, enriching the findings of the earlier stages of the research project. the next step is to work with the stakeholder groups in this project and use the research project’s findings as the evidence base on which to develop a structured, modular education framework comprising a postgraduate qualification in health librarianship and a continuing evidence based library and information practice 2011, 6.4 59 professional development structure supporting a three-year cycle of certification and revalidation. introduction in australia, health librarians work in many different environments, including hospitals, research institutes, pharmaceutical companies, government departments, regional health services, professional colleges, universities, not-for-profit and community organizations, and parts of public library services and others. entry to the profession is via completion of a course of study accredited by the australian library and information association (alia) (http://www.alia.org.au/education/courses/rec ognition.html). currently, there is no australian health library specialization and no mandatory requirement for professional registration or for the maintenance of professional skills. therefore, health specific skills and knowledge are principally acquired informally in the workplace. recent national health reforms in australia have included the creation of a single national compulsory registration body, the australian health practitioner regulation agency (ahpra), (http://www.ahpra.gov.au). from july 2010 this has assumed responsibility for the registration and regulation of 10 health professions, with four more to be added in 2012. health librarians are not among these 14 professional groups. as health librarians are at present excluded from national health workforce registration and are thus without recognition as a health profession, they risk being classified in the clerical or administration streams. the risk of not having nationally recognised qualifications and registration for health librarians with ongoing continuing professional development (cpd) requirements was articulated by ritchie in 2008: in the context of the australian health workforce, in which national level registration with requirements for regular cpd are increasingly the norm, health librarians will lose credibility and status if they don’t have a structured and regulated cpd system. in addition, and perhaps more importantly, they risk losing competitiveness in the health information professional market. (ritchie, 2008, p.103) advances in e-health in australia are another important environmental driver for workforce planning and developing the skills of the health information professions as they will “precipitate the integration of patient care systems, such as the shared electronic health record, with clinical decision-support information tools, consumer health information and other knowledge resources, all requiring customisation at point-of-care. implementation requires skills to consult with and train clinicians; information professionals will need to know how to manage the content as well as the technology which runs the systems” (ritchie, 2008, p. 103f). the research currently being conducted by the health libraries australia (hla) group of the australian library and information association (alia) is in response to growing awareness of the need to operate nationally as part of the e-health and health professional workforce initiatives discussed above. hla is conducting the alia/hla workforce and education research project, which takes an evidence based approach to identifying future skills requirements for health librarians in australia, and to developing a structured, modular education framework to meet these requirements. the first part of the project has been to establish future roles and skills requirements for health librarians in australia. a literature review and environmental scan were carried out that identified the following main trends: current and future implementation of a national electronic health record; development evidence based library and information practice 2011, 6.4 60 of web portals for consumers and health providers including purchasing and managing resources; national registration and compulsory ongoing professional development requirements for health professionals; emerging specialist roles for health librarians; expanding roles for librarians in user education including elearning; and greater involvement in research, as part of a multi-disciplinary team. these trends informed subsequent online surveys of australian health librarians and health library managers. finally, semi-structured interviews with employers of health librarians across a range of sectors were conducted. as they comprise one of the main stakeholder groups, with a unique perspective on health workforce planning and future needs for the health professional workforce, it was considered critical to the research that this group was consulted. this paper reports on the third and final research phase of the project – the interviews with health librarian employers. the earlier stages of the research have been reported elsewhere (hallam et al, 2010; ritchie et al., 2011). objective the objective of the interviews was to discover employers’ views on how (or whether) librarians assist in achieving the missioncritical goals of their organizations, and how health librarians contribute to the organization now and into the future. the research team wished to explore the perspective of employers regarding the current and future skills requirements of health librarians, and specifically how these might be considered relevant to the anticipated directions of the parent organization. methods an interview schedule, titled future roles for health librarians, was developed to form the basis of the semi-structured interviews. the schedule consisted of a brief introduction to the project, followed by open-ended questions in two parts: section one – ‘”how health librarians assist you in doing your job and achieving your goals”; and section two – “how health librarians contribute to your organization now and possibilities for the future”. the schedule also contained prompts for several of the questions for the interviewers to probe or extend the subject’s response if appropriate. see “appendix a” for the full interview schedule. the interview schedule was based on two pieces of research from the united states. the first was vital pathways: the hospital libraries project established by the 2005/06 medical library association under the leadership of mary joan tooey, to review the status of hospital librarians and develop strategies to support the profession. the project culminated in a symposium reported in the journal of the medical library association (tooey, 2009). vital pathways, in turn, built on the results of an earlier project which identified five “missioncritical” goals of hospital administrators, and related these to the librarian’s role in helping to achieve these goals. these mission-critical goals were: clinical care; management of operations; education; innovation and research; and customer service (holst et al., 2009, p. 285). it is important to note that these were the “mission-critical” goals as articulated by the hospital administrators, and in this way the study looked at how organizational needs could inform future development of the roles of hospital librarians. the second piece of research used in developing the interview schedule was the value of the hospital library study, funded by the national network of libraries of medicine, middle atlantic region (dunn, brewer, marshall, & sollenberger, 2009). the two objectives of this study were: “to investigate the views of hospital administrators about librarians and library services in their institutions and how they make decisions around what services are provided and funded in their hospital”; and “to explore the views of health sciences librarians, informed by interviews with hospital administrators on the value of the hospital library” (martin, 2008, p. 1). accordingly, the librarians participating in evidence based library and information practice 2011, 6.4 61 this project interviewed hospital administrators at their institutions and then participated in focus groups to explore their experiences of interviewing their employers and hearing their views on the value of the library to the organization. the alia/hla project reference group drew on the value of the hospital library study for several of the questions asked in the interviews with australian health librarian employers. the alia/hla project reference group consists of six health librarians and a principal researcher. each member of the project group approached between one and five individuals known to them to generate a convenience sample of 22 employers of health librarians. health librarians work in a broad range of contexts and the reference group aimed to reflect this range in the interview sample. there were 15 semi-structured interviews conducted between october and november 2010 with employers in the hospital, academic, government, private, consumer health and not-for-profit fields. the interview schedule was sent to each interviewee prior to the interview so that they had time to consider their responses. interviews were conducted face-to-face or by telephone, and each lasted around one hour. by agreeing to be interviewed, the subjects consented to participate in the research. the researchers wrote up the interview notes using the interview schedule and submitted them to the principal researcher, who combined the data into one document and removed all information that could potentially identify the participants. the interviews generated a considerable amount of qualitative data, which needed to be summarized to reveal the key ideas expressed by the interviewees. the process of content analysis was used to rigorously analyze, examine and verify the textual content of the interview data. content analysis has been defined as “a detailed and systematic examination of the contents of a particular body of material for the purpose of identifying patterns, themes or biases” (leedy & ormrod, 2001, p. 155). it is commonly used as “a research technique for making replicable and valid inferences from texts (or other meaningful matter) to the contexts of their use” (krippendorff, 2004, p. 18). content analysis involves identifying the characteristics or qualities of the text to be examined in precise terms. specific codes are developed which are used as the units of analysis to facilitate the categorisation of the textual content. the qualitative dimensions of the process allow the categories of data to be compared and their interconnectedness examined, enabling key themes to emerge. when the analytical judgments are entirely objective, such as locating the appearance of certain words in the text, a single ‘rater’ can perform the content analysis. however, in many situations the analytical judgments are likely to be more subjective, requiring two or three ‘raters’ to be involved, and a composite of their judgments used. in the present study, as the researchers anticipated that the textual data collected through the interviews would potentially have multiple meanings and interpretations, three raters undertook the categorisation of the interviews and the principal researcher critically reviewed and collated the coded interview data. as a first step in the content analysis, one of the researchers undertook the initial review of the data and developed a non-hierarchical coding scheme. suggestions for coding were based on three main sources. the first source was the project groups’ initial environmental scan and literature review (hallam, et al., 2010). this work examined the australian health care system and the health library sector, international trends in health libraries, and education and credentialing of health librarians. the second source was the medical library association’s competencies for lifelong learning and professional success (medical library association, 2007). the seven mla competencies, plus an additional competency on continuing professional development, formed the basis of questions about current and future responsibilities and the knowledge evidence based library and information practice 2011, 6.4 62 and skills required of health librarians in the two surveys conducted prior to the interviews. much of the coding scheme for analysis of the interviews was based on these eight competencies (adapted as competency 1 to competency 8) which cover the following areas: c1 understanding the health sciences and health care environment; c2 providing information services to meet user needs; c3 managing health information resources; c4 leadership, finance, communication and management; c5 using technology and systems to manage information; c6 curricular design and instruction, including information literacy training; c7 understanding scientific research methods; and c8 maintaining currency of professional knowledge and practice. the third source used in developing the coding scheme was the two pieces of research from the united states discussed above vital pathways: the hospital libraries project and the value of the hospital library study. the vital pathways project used results from a previous research study by abels, cogdill and zach (2002) who developed a taxonomy of five mission-critical goals and 15 organizational goals to which hospital librarians contributed. this taxonomy was a valuable resource in developing the coding scheme used for the alia/hla study. once the first researcher had developed the initial coding scheme and categorised the data, two more researchers categorised the data independently. as a result, a number of additional codes were identified directly from the data in an iterative process. the three coders and the principal researcher met by teleconference to clarify definitions of existing codes and suggest new codes where thematic gaps were identified, and this process continued by email until no further new codes or amendments to existing codes were identified. the final coding scheme consisted of 52 codes. consistency of coding was an important aspect of the process to ensure a strong degree of inter-rater reliability. as the three researchers were employed in diverse health library contexts (hospital, academic and government department), it was feasible that there could be some degree of subjectivity in the interpretation of the textual material, resulting in potential inconsistencies in the allocation of the codes. results and discussion the findings are reported through a discussion of the content analysis process in general and thematically through the particular perspectives of the interviewees’ mission-critical goals, the librarians’ existing and anticipated contributions to the organization, and the future roles and skills that might be required by health librarians. characteristics of the interview subjects the interview subjects represented a wide spectrum of the health sector from major urban, regional and rural areas of australia, and a range of employer positions including medical directors, chief knowledge/information officers and senior academics. content analysis of the interview data the process of coding revealed that the views expressed by the interviewees were closely aligned with the conceptual ideas explored in the earlier stages of the research project, validating the selection of the sources used for the development of codes. while no statistical examination of the allocated codes was undertaken, all codes were utilised in the analysis of the interview data. overall it was found that the direct involvement of the three raters over the life of the research project and their collaborative development of the coding schema resulted in a sound degree of consistency in the content analysis. evidence based library and information practice 2011, 6.4 63 it was apparent that one rater brought a strong managerial perspective to the analysis, evident for example in the interpretation of the statement “developing a corporate taxonomy for the organization”, which two raters coded as metadata (‘manage cataloguing, classification, abstracting of resources’), but which the third allocated the code knowman to represent the knowledge management process. a venn diagram illustrating the notion of “librarians intersecting with all” drafted by one interviewee was analysed by two raters as network, defined as ‘develop and maintain networks to meet users’ information needs’, but again as knowman by the third, implying that health librarians can play a role as knowledge managers to link the different elements of a health service. beyond this, some discrepancies tended to be conceptual in nature, with raters showing their preferences for specific groups of codes. there was also some overlap between the codes infoaccess, ‘identify appropriate methods of information delivery and access, including for diverse populations’ and eaccess, ‘use technology to enable permanent access to electronic information’, which highlights the digital nature of contemporary health information. mission-critical goals the interview subjects were asked about their role in their organizations and the missioncritical goals associated with it. the range of responses to this question was broad, but still fitted within the five mission-critical goals identified by abels et al. (2002) and adopted by the vital pathways project, namely: clinical care; management of operations; education; innovation and research; and customer service (holst et al., 2009, p. 286). this was despite the fact that in the abels study only administrators from hospitals and academic health sciences centres were interviewed, whereas the alia/hla study involved interviews with a much wider range of health librarian employers. it is worth noting that for the alia/hla study, the definition of the fifth mission-critical goal – customer service – was expanded to include not only patients, carers and their families, but also students and academics in a university setting, staff of a government department, a non-government organization (ngo) or a private company. librarians: contributions to the organization and future roles and skills the open-ended questions posed in the interviews provided the interviewees with the scope to “demonstrate their unique way of looking at the world” (cohen, manion, & morrison, 2007, p. 151), so as to provide very individual perspectives about the roles and skills of health librarians employed in a range of contexts, and to allow unanticipated issues to be raised. it was valuable to note, therefore, that the content analysis revealed that the employers, while distant from the operational dimension of the library itself, could confidently discuss the roles and skills of their staff in ways that reflected the mla’s competency framework and the issues and challenges facing the profession that were identified in the literature review and environmental scan. coding of the data revealed groupings of commonly allocated codes, particularly in relation to the following questions: • 1.3 do your librarians assist you in achieving any of [your] goals? • 2.1 what do you consider are the main ways that health librarians contribute to your organization now? • 2.3 are there one or two specific things your librarians offer that are especially useful to this organization? • 2.4 are there challenges or opportunities for your organization where your librarians could be involved in the future? and • 2.6 what skills and qualities do you think health librarians will need in order to be able to contribute effectively to your organization in the future? evidence based library and information practice 2011, 6.4 64 these groupings aligned closely with the mla competencies for health librarians on which part of the coding scheme was based, and which is used as a framework to present the results of the data analysis and discuss major themes related to each group of codes (c1 to c8). in the following discussion, reference is made to the main findings of the earlier stages of the research, particularly the online surveys. figure 1 below shows the individual librarians’ and library managers’ perspectives of the areas of professional knowledge and responsibilities reported as currently applied ‘often’ or ‘very often’; and figure 2 shows the two groups’ perspectives on the professional knowledge and responsibilities regarded as likely to ‘increase to some extent’ or ‘increase significantly’ in the next three to five years. figure 1 current application of professional knowledge and responsibilities (competencies 1-8): individual librarian respondents and library manager (institutional) respondents. figure 2 future increase in professional knowledge and responsibilities: individual librarian respondents and library manager (institutional) respondents. evidence based library and information practice 2011, 6.4 65 c1: understanding the health sciences and health care environment one group of codes centered on knowledge of the library’s parent organization and included understanding organizational policies, the clinical care / education / research environment, the economic and legal environment and the organization’s role in the broader health environment. the interview subjects were specifically asked “does your organization involve your librarians in strategic planning and / or organization-wide, mission-critical committees?” (question 2.5). responses ranged from enthusiasm for librarians to become more involved with organizational strategic planning, to “it would depend on capacity of individuals”, to “the library needs to become involved in such committees or be left behind”, to concern about finding the balance between the health librarian’s strategic role and immediate operational needs. the overall perception from responses to this question was that librarians are currently more involved in dayto-day operations rather than in strategic planning, but that this could change in the future. the theme of proactivity versus reactivity emerged strongly from the data in response to a number of current and potential roles for health librarians. there was a keen sense that health librarians could make a greater strategic contribution to their organizations by becoming involved in strategic planning, more embedded in teaching and research, or integrated into clinical systems. concerns were expressed that library and information professionals were often more reactive than proactive, so there was considerable work to be done to ensure that they were not overlooked or sidelined in a fast-changing environment. many respondents commented on the need for the librarians to be more proactive, “pushing out to survive”. c2: providing information services to meet user needs not surprisingly, much of the interview data reflected the core role of health librarians in providing services to meet users’ information needs. in the surveys conducted prior to the interviews, this was the area of professional knowledge and responsibility cited as most often needed by both individual respondents and library managers (see figure 1). it was also among the top four areas of professional knowledge and responsibilities regarded as likely to increase over the next three to five years (see figure 2). interview participants stressed the importance of this role for health librarians, particularly their skills in searching and locating information resources which saved their clients’ time, their provision of alerting services to keep their clients up to date with the latest information in their specialty area, and support of evidence-based practice. comments included: “supporting clinical staff [to] access knowledge and information are important and critical”; librarians are “the navigators”, “compilers of information for busy clinicians”; and “their searching skills are critical”. most interview subjects regarded these roles as likely to increase in importance in the future, with the exception of one person who commented: “if it’s just about accessing information, everyone will know how to do that, my five-year-old grabs my iphone to find information…” (however the implication here may have been that it is more than ‘just about accessing information’ and librarians need to refine, extend and market their intermediary role in ‘providing information services to meet user needs’, as suggested by some of the other comments.) generally, employers saw these roles transforming, with health librarians moving from finding knowledge to facilitating application of knowledge by integrating knowledge resources with clinical systems, and analysing and synthesising information to meet and anticipate particular user information needs. transformation of current skills to fill new roles was a theme that emerged strongly from the data analysis in relation to this and many of the other competency areas. evidence based library and information practice 2011, 6.4 66 c3: managing health information resources another core role of health librarians identified during the data analysis was managing information resources, including negotiating with vendors; selection, purchase and licensing of resources; managing copyright; managing cataloguing, classification and abstracting of resources; and managing conservation and archiving of resources. in the surveys, 81% of health librarians and 69% of health library managers responded that they or their staff often or very often needed this area of professional knowledge and responsibility (see figure 1); in terms of the future, this area was second only to knowledge of technology and systems in terms of competencies predicted to increase (see figure 2). the interview subjects also viewed the management of information resources as an important current and future role for health librarians. several mentioned the need for skills in this area in a constrained fiscal environment: “opportunities to consolidate and share purchases need to be maximised”; “some of the national procurement, packaging up of resources, getting some harmonisation of ‘best buys’, work with national librarians is important, better value for money”. in response to “turmoil in the publishing arena” (escalating costs, publishers’ “rearguard action … in response to open access initiatives” and administrators’ push to cut back on subscriptions), one subject wanted to see “preemptive strikes from librarians, not to accept the status quo, to challenge and push for new models of access to information”. the same individual also commented that it is “essential to have seamless access to information whether at uni or in the hospital – licensing provisions are stupid”. two recurring themes can be identified from these comments: the need for health librarians to be proactive, and the repackaging of their traditional skills to adapt to a rapidly changing information environment. c4: leadership, finance, communication and management analysis of the interview data relating to management roles for health librarians highlighted the theme of personal relationships. respondents tended to focus on the marketing, public relations and interpersonal skills required of health librarians now and into the future. one employer commented: “a lot of the service’s success is about personal relationships and visibility”. another noted: “we need to get closer to our academic or clinical role to ‘sell’ our skills to them in a better way”. and another observed: “technology has taken that personal interface away – a lot of it – but still, that face of the library, that personal touch .... there’s plenty of opportunities”. despite many comments on the importance of online access to information resources, managing a physical library facility was also regarded as important: “i think the physical space has an important role in the hospital as a place where people can go and have a bit of time out and do some reading, a quiet place to work”. c5: using technology and systems to manage information much of the interview data related to health librarians’ knowledge of technology, not just in the traditional areas of information access and dissemination, but also in relation to elearning, e-health, integration of clinical decision support tools with the electronic health record, website development and management, and use of web 2.0 technology. in the surveys that preceded the interviews, 81% of individual respondents and 67% of library managers indicated that they or their staff ‘often’ or ‘very often’ needed to understand and use technology and systems to manage all forms of information (see figure 1). in terms of the future, individual respondents expressed a stronger belief that the requirement for technological competencies would increase, with 82% indicating that there would be an increase ‘to some extent’ or ‘significantly’, compared with 69% of library managers (see figure 2). the evidence based library and information practice 2011, 6.4 67 comments provided stressed that it was critical for librarians to keep up with new technologies, especially mobile technologies. these comments were echoed in the interviews with the employers, many of whom conceptualised health librarians as the interface between it and clinicians. one interview subject predicted: “i can see a role with mobile technologies – bedside use in clinical environments…. [the librarians will be the] interface between the it people, the clinicians and health informaticians. they can interpret the two worlds. i am thinking about the use of ipads, search engines, access to health records.” another commented: “yes i really like the idea that the librarians are like warmware – able to help get the most out of our computer hardware and software”. and a third observed: “librarians understand how people interact with systems and language”. clinical decision support was cited by several respondents as an area where health librarians could potentially have a much greater role: “the library has a big role to play because to be effective [decision support] needs to be targeted, relevant and how you integrate some of the decision support tools with the clinical apps is a big challenge, and we’re not even in that space yet”. the rate of change in this area was predicted to increase, and librarians should be aware that new roles could emerge very quickly: “the speed of change means great opportunities in the next 1-2 years. information convergence – getting information rapidly and then incorporating patient data into the mix. on the fly with mobile devices”. c6: curricular design and instruction, including information literacy training analysis of the interview data also highlighted the health librarian’s role in education and training, including curriculum design, educational needs assessment, user education, evaluation of learning outcomes and use of technology in design and delivery of training. the one off/orientation role for new staff was also common. in the surveys, this was one of the top four areas of professional knowledge and responsibility regarded as likely to increase over the next three to five years (see figure 2). the interview subjects also saw this as an area in which health librarians are contributing now and where there is an opportunity for their contribution to increase in the future, particularly in the area of elearning. comments on librarians’ roles in this area included: “training medical staff and students to be effective searchers”; “supporting the … teaching and learning core business of the university”; “support[ing] continuing professional development for existing clinical staff”; and “pre-vocational training support for overseas trained doctors”. interestingly, codes which the researchers interpreted as relating to the additional competency added by the project group (“maintaining currency of professional knowledge and practice”), were in practice applied to the interview data describing librarians’ role in supporting professional development of staff within the organizations. the employers interviewed did not refer to professional development specifically for health librarians, but when prompted by the interviewers, several respondents were enthusiastic about the idea of formal qualifications in health librarianship. c7: understanding scientific research methods closely related to health librarians’ role in education is their involvement in research. survey respondents were fairly conservative in their thinking about this role, with only 43% of individual respondents and 49% of managers regarding this as an area where librarians often or very often required skills and knowledge (see figure 1). with regard to the future, 60% of individuals and 51% of managers regarded this as a role likely to increase (see figure 2). many of the interview subjects were similarly conservative, describing the current and future role of health librarians in research mainly in terms of providing expertise at the literature review stage of research projects. however several of the employers interviewed predicted that evidence based library and information practice 2011, 6.4 68 health librarian involvement in this area would and should increase substantially. one respondent commented at some length, citing “research skills [and] academic writing skills” being required, as well as “a new structure to move librarians out of the service/support roles and into academic pathways”, similar to the canadian model of academic librarians with tenure. librarians are becoming research partners, they are coauthors of academic papers and reports. this is becoming an increasingly important role that they play. it’s no longer just about searching for the information (librarian as handmaiden, unacknowledged service provider), it is about becoming a partner in the team. there’s an expansion of their roles; a blurring of the margins. two interview subjects mentioned the importance of health librarians having skills in bibliometrics or citation analysis and predicted that this was a role that would increase in importance as librarians became more closely involved in research. health librarians would need “excellent knowledge of the various journal ranking systems and ability to compile data from these for a particular research area”. future roles and skills in the interviews it appeared that health librarian employers were generally more ambitious than the librarians themselves when it came to envisaging future roles, skills and qualities for the profession. in the online surveys, respondents generally envisaged “more of the same”. in contrast, health librarian employers talked about “repackaging traditional librarian skills – data management and curation, web development, citations and bibliometrics”, “extending skill sets” and “becoming partners”. one interview subject observed that “the important skill sets are any of the hybrid of roles about how you apply information management in the health system, integrating the knowledge with the clinical systems so clinicians use it”. degree of satisfaction the responses to question 2.2 (“are you satisfied with the role that your health librarians play in your organization now?”) reflected a high level of satisfaction with the services currently provided by health librarians – (“with cuts around here… if i had to die in a ditch [the library is] one service that i would do it for”) but also some uneasiness about how to evaluate satisfaction with library services. one respondent commented: “well i’m not dissatisfied, but i don’t know how to measure the level of satisfaction…. i don’t know what the utilisation is of the library [by] any of our staff to be honest with you.” overall there was a strong perception that the library service needed to be more visible, to create and maintain a higher profile within the organization; “a sense that librarians are too passive – like to see them more active and more proactive, less reactionary. would like librarians to be more revolutionary”. limitations of the research the alia/hla research team was sensitive to the risk of bias that could creep into the interview stage of the investigation. in the context of research, bias has been defined as “any influence, condition or set of conditions that singly or together distort the data” (leedy & ormrod, 2001, p. 221). one possible source of bias is the researchers’ objective in conducting the alia/hla workforce and education research project – to develop a postgraduate qualification and continuing professional development structure for health librarians in australia. mardis’ warning about evidence-based library and information practice in school librarianship is relevant to all lis researchers: “the use of ebl in school librarianship has been labeled as blurring the evidence based paradigm because research pursuits are often a response to a perceived evidence based library and information practice 2011, 6.4 69 threat to the profession or a need to advocate for an aspect of practice” (mardis, 2011, p. 7). as lyons reminds eblip practitioners, “advocacy has no place in ebp…. in fact, avoiding bias and prejudiced agendas is the main reason these practices developed in the first place” (lyons, 2009, p. 65). the research group acknowledges that advocacy for the future of health librarianship in australia may have influenced the conduct and results of this research. however the project group has made every attempt to conduct the research with rigor and objectivity, and has identified the limitations of the research. the librarians who acted as interviewers knew their subjects and in many cases the relationship was that of employee/employer, which can arguably introduce the possibility of bias. the research team was also conscious of the fact that there were multiple interviewers, rather than a single interviewer, which meant it was not truly possible to control the potential bias that could emerge through the expectations, attitudes, preconceptions or opinions of multiple interviewers. the interviewers were all instructed to pose the questions directly as written on the interview schedule, with further guidance given through the stated examples for prompting the interviewee in order to elicit a more detailed response. the sample size itself in the alia/hla study (15 subjects) is equivalent to the sample size in both the abels study (abels, et al., 2002), and the value of the hospital library study (martin, 2008). the alia/hla sample is weighted towards health librarian employers in hospitals and universities. ideally, the sample could have been larger and included more than one representative from pharmaceutical companies, professional colleges, government departments and community organizations. the focus of the interview subjects was almost entirely medical, with nursing and allied health barely mentioned. this was an omission on the part of the research team and should be addressed if a second round of interviews is conducted. in terms of the content analysis, it is acknowledged that distinctions in the coding of textual data by different raters are inevitable. increased inter-rater reliability could potentially have been achieved if additional time had been spent on developing the skills of the three raters, to ensure that there was no confusion about the scope of the operational definitions and that reliable levels of objectivity were applied in the interpretation of the data. however, as a nonhealth librarian, the principal researcher played a role in ensuring that there was a strong degree of impartiality in reviewing and reporting the findings. conclusion the interviews with employers provided further insights into the current and future skills of health librarians in australia, augmenting the findings from the literature review, environmental scan and surveys of health librarians and health library managers (hallam, et al., 2010; ritchie, et al., 2011). not all the interviewees were directly familiar with the library operations, but there was a clear sense of respect for the roles and responsibilities of library staff. it was found that, in the rapidly changing environment of health services, there were opportunities – indeed expectations – for health librarians to play a more active and strategic role in supporting and even driving the change process in their institutions, to be proactive rather than reactive. the employers’ own areas of responsibility reflected the five mission-critical goals identified in earlier research in the united states and their understanding of the skills the librarians required could be mapped directly to the seven competencies promulgated by the mla. to remain viable in the future, however, health librarians need to extend their skill sets and become more embedded in their organizations and more directly aligned with their strategic goals. employers were generally more adventurous than the librarians themselves when envisioning repackaging and extending the traditional skill sets of health evidence based library and information practice 2011, 6.4 70 librarians. areas of practice such as education and training, scientific research and clinical support were highlighted as critical domains of future activity, all requiring a high level of understanding and experience of ict. these findings correspond to the new and emerging roles discussed in the environmental scan (hallam, et al., 2010; ritchie, et al., 2011), with an anticipated increase in knowledge and responsibilities in the areas of tailored reference services (such as the role of clinical librarians), advances in technology and systems (health informatics), and the teaching role (understanding curricula design and instruction). employers tended to conceptualise health librarians in terms of the interface between clinicians and information technology, with librarians possessing the interpersonal skills to negotiate between people and technology in a rapidly changing health environment. the alia/hla workforce and education research project has adopted an evidence based approach to determining the future skills requirements for the health library workforce in australia. now that the research phase of the project is complete, the next step is to develop a structured, modular education framework to meet these requirements. analysis of the information gathered through the project will be used to inform the development of education and training pathways. alia has approved the establishment of a working party that will work towards implementing a professional development scheme for australian health librarians with a three-year cycle of certification and revalidation (ritchie & hallam, 2011). the working party will also explore with education providers the development of tailored professional development opportunities that may articulate into a postgraduate qualification in health librarianship. these initiatives will not only help ensure a strong future for those already employed in the sector, but also encourage a new generation of information professionals to consider a career in health librarianship. acknowledgements: the authors acknowledge and thank their colleagues who conducted interviews – laura foley, jill buckley-smith and prue deacon. this paper is based on a paper first presented at the 6th international evidence based library and information practice conference eblip6, 27-30 june 2011, university of salford, salford, uk. references abels, e. g., cogdill, k. w., & zach, l. (2002). the contributions of library and information services to hospitals and academic health sciences centers: a preliminary taxonomy. journal of the medical library association, 90(3), 276284. cohen, l., manion, l., & morrison, k. (2007). research methods in education (6th ed.). abingdon, oxon.: routledge. dunn, k., brewer, k., marshall, j. g., & sollenberger, j. (2009). measuring the value and impact of health sciences libraries: planning an update and replication of the rochester study. journal of the medical library association, 97(4), 308-312. doi: 10.3163/1536-5050.97.4.016 hallam, g., ritchie, a., hamill, c., lewis, s., newton-smith, c., kammermann, m., & o'connor, p. (2010). australia's health libraries: a research-directed future. library trends, 59(1), 350-372. holst, r., funk, c. j., adams, h. s., bandy, m., boss, c. m., hill, b., joseph, c.b. & lett, r. k. (2009). vital pathways for hospital librarians: present and future roles. journal of the medical library association, 97(4), 285-292. doi: 10.3163/1536-5050.97.4.013 evidence based library and information practice 2011, 6.4 71 krippendorff, k. (2004). content analysis: an introduction to its methodology (2nd ed.). thousand oaks, ca: sage. leedy, p.d., & ormrod, j.e. (2001). practical research: planning and design (7th ed.). upper saddle river, nj: merrill prentice hall. lyons, r. (2009). commentary. the call for evidence based practice: speaking louder than words. evidence based library and information practice, 4(3), 63-67. mardis, m. (2011). evidence or evidence based practice? an analysis of iasl research forum papers, 1998-2009. evidence based library and information practice, 6(1), 4-23. martin, e. r. (2008). value of the hospital library: focus group report. new york: national network of libraries of medicine, middle atlantic region. retrieved 8 nov. 2011 from http://www.nnlm.gov/mar/about/pdf/ valuereport012208.pdf medical library association. (2007). the educational policy statement of the medical library association: competencies for lifelong learning and professional success. retrieved 10 may 2009, from www.mlanet.org/pdf/ce/200705_edu_ policy.pdf ritchie, a. (2008). future perspectives. future visions for continuing professional development in health librarianship: an australian perspective. health information and libraries journal, 25(suppl. 1), 103-105. doi: 10.1111/j.1471-1842.2008.00819.x ritchie, a., & hallam, g. (2011). certification and revalidation – the way forward for australian health librarians. hla news, march, 1, 17-20. retrieved 8 nov. 2011 from http://www.alia.org.au/groups/health nat/hla/hla-news-march11.pdf ritchie, a., hallam, g., hamill, c., lewis, s., foti, m., o'connor, p., & clark, c. (2010). designing a specialist postgraduate qualification and continuing professional development structure for the health librarian workforce of the future. australian academic and research libraries, 41(4), 276-299. tooey, m. j. (2009). a pathway for hospital librarians: why is it vital? journal of the medical library association, 97(4), 268272. doi: 10.3163/1536-5050.97.4.010 / evidence based library and information practice/ / microsoft word art_abbott.doc evidence based library and information practice 2006, 1:1    58 evidence based library and information practice       article    persuasive evidence: improving customer service through evidence based  librarianship      wendy a. abbott  associate director, customer services, bond university  gold coast, queensland, australia  e‐mail: wendy_abbott@bond.edu.au      received: 10 december 2005    accepted: 24 february 2006      © 2006 abbott. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective ‐ to demonstrate how evidence based practice has contributed to informing  decisions and resolving issues of concern in service delivery at bond university library.     methods ‐ the paper critically analyses three evidence based research projects  conducted at bond university library. each project combined a range of research  methods including surveys, literature reviews and the analysis of internal performance  data to find solutions to problems in library service delivery. the first research project  investigated library opening hours and the feasibility of twenty‐four hour opening.  another project researched questions about the management of a collection of feature  films on dvd and video. the third project investigated issues surrounding the teaching  of endnote to undergraduate students.      results ‐ despite some deficiencies in the methodologies used, each evidence based  research project had positive outcomes. one of the highlights and an essential feature of  the process at bond university library was the involvement of stakeholders. the ability  to build consensus and agree action plans with stakeholders was an important outcome  of that process.    conclusion ‐ drawing on the experience of these research projects, the paper illustrates  the benefits of evidence based information practice to stimulate innovation and  improve library services. librarians, like most professionals, need to continue to  develop the skills and a culture to effectively carry out evidence based practice.                       http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:1    59 introduction    like most australian university libraries,  bond university library employs a range   of quality processes and gathers vast  quantities of data to measure performance  and ensure students and staff receive a   high standard of library service. for  example, performance indicators such as  processing turnaround times are used to   verify that resources are available for  customers in a timely manner. usage  surveys inform decisions about resource  selection and retention and customer  surveys objectively measure levels of  customer satisfaction and indicate the  service areas where further development  is needed. these performance indicators  and survey results are benchmarked  internally over time and externally against  other libraries and organisations in the  wider community.    a great deal of qualitative data is also  gathered to measure the library’s  performance including free comments in  surveys, feedback received via suggestion  boxes (physical and virtual), focus groups,  roundtable forums and so on.    while some of these measures are  designed to diagnose problems and  prescribe the appropriate response or  remedial action required, others raise  issues for which there is no ready‐made  solution. at bond university library there  has been a natural progression from using  these systematic quality measures to  applying evidence based practice as a  rigorous methodology for solving  problems in service delivery.    evidence based librarianship in action    despite the absence of an agreed upon  definition for evidence based librarianship  there is general consensus that evidence  based practice in library and information  science means applying a sequence of  information management processes (brice,  booth and bexon 3):        specifying the problem  the crucial first stage in evidence  based practice is to formulate a  focussed, structured, answerable  question. models such as pico (for  evidence based medicine) and spice  (which is more relevant for  information practice) have been  developed to help structure the  question (brice, booth and bexon 5).    finding the evidence to answer the  problem  the second stage in evidence based  practice is to conduct a comprehensive  search of the literature to find  evidence relevant to the question  posed. qualitative and quantitative  research methods may also be used to  locally validate the published  evidence (koufogiannakis and  crumley 124).    appraising the evidence   the validity, reliability and  applicability of the evidence must  then be appraised using critical  evaluation techniques (brice, booth  and bexon 9).     applying the results  the results of evidence based research  may be directly applicable or may  simply improve an understanding of  the issues. variables such as the  nature of the user group, timeliness of  the evidence, costs, politics and the  severity of the problem influence how  the results are applied  (koufogiannakis and crumley 120‐  126).    evaluating the outcome   the outcome of evidence based  practice can be evaluated on two  levels. firstly on a technical level:  have the stages of evidence based  practice been successfully followed?  secondly, has the intervention that  resulted from the evidence based  process actually made the anticipated  difference (booth 127)?      evidence based library and information practice 2006, 1:1    60 redefining the problem   through evaluating the outcome of  the evidence based practice, the  original problem may be redefined or  a new question can emerge to guide  future research (booth 128). this last  step in the process is sometimes  combined with the evaluation stage  above.    this paper will describe the application of  evidence based practice in solving a  number of practical problems that arose at  bond university. two of the issues  emerged from customer feedback and the  other evolved through a collaborative  project between an academic group and  library staff.     case study on library opening hours    questions: should bond university library  increase opening hours to meet student  demand? how realistic is twenty‐four  hour opening?    while bond university library opening  hours are good by australian standards,  in 2004 students were increasing pressure  on the library to extend its opening hours  including requests for twenty‐four hour  opening. requests were received through  the library suggestion box, the rodski  customer survey and from the student  council and other student associations.     the decision was made to use an evidence  based approach when the issue was  escalated to the senior university  executive level. library management  decided that a comprehensive report with  recommendations based on best practice  and other relevant evidence would inform  all stakeholders and help to resolve the  issue.     the evidence was gathered in a variety of  ways:    the library’s opening hours were  compared to those of other australian  university libraries using the council     of australian university librarians  (caul) annual statistics. this  revealed that bond library’s opening  hours were the fourth best in  australia. only one other institution’s  opening hours were significantly  better.    a survey of the twenty‐four hour  facilities of other australian university  libraries was conducted using caul  to distribute a questionnaire. this  showed that six australian  universities had a twenty‐four hour  facility associated with the library  ranging from a traditional computing  laboratory to specially designed and  constructed areas incorporating  individual and collaborative  learning/study facilities. the survey  results are available on the caul  website (“24x7 library facilities”).    a review of the literature illustrated  that internationally, an increasing  number of institutions are opening  their libraries for twenty‐four hours  for some days of the week and new  libraries are being built to incorporate  a twenty‐four hour facility. research  conducted in the united states  indicates that students’ requests for  extended library opening hours are  perennial and they vary from requests  to open until 2.00 am during exam  periods to demands to open twenty‐ four hours per day seven days of the  week, most days of the year (steele  and walters; curry; engel, womack  and ellis).    a quantitative analysis of the  feedback about opening hours  received from bond university  customers in 2004 indicated a low but  persistent level of complaints (see  appendix a for more details on how  this analysis was performed).    a quantitative analysis of the usage  patterns gathered in the library’s  facilities use survey was used to  show that the current spread of hours  approximated closely to usage  evidence based library and information practice 2006, 1:1    61 patterns (see appendix a for more  details on the methodology for the  quantitative analysis based on the  facilities use survey).    a detailed costing was prepared for  four different scenarios ranging from  longer opening hours for the duration  of the semester to a short period of   twenty‐four hour opening leading up  to the examination period (see  appendix b for details on how the  costing was done).    the evidence was appraised and  summarised to produce a report  incorporating a short term  recommendation to further increase  opening hours in the weeks leading up to  the examination period. the  recommendation for the longer term was  to incorporate a twenty‐four hour study  facility when and if the library is  refurbished. based on the caul survey,  such a facility should provide individual  and group study spaces, computers,  wireless access, laptop ports, photocopier  and printing equipment, vending  machines, lounges and appropriate  security arrangements.    the report was discussed with the student  council and the senior university  executive and while the recommendation  for the long term was accepted, the short  term proposal was not adopted. through  the process, all stakeholders gained a good  understanding of the issues and agreed  with the decision that was made.     in the period since the research was  undertaken, the concept of refurbishing  the library has progressed and the  university has invited proposals from  architects to develop a brief for the project.  the evidence gathered to resolve the  opening hours question has contributed to  a better understanding in the wider  university community of the library’s  role in providing a learning environment   in addition to its traditional role as a  repository of books and provider of  electronic resources. this is helping to  convince the senior executive that a  refurbishment is necessary to meet student  needs.    the refurbishment proposal has defined a  new question: “what facilities and   services would students like to see  included in a refurbished library?” and   stakeholder feedback is being gathered  using an online survey and focus groups.    case study on management of access to  feature films    questions: how can the feature films on  dvds and videos be organised and  managed to maximise access for the bond  community? should these resources be  more tightly controlled to improve access?  what is the most appropriate shelf  arrangement?    to support the bond university school of  film and television (ftv) programs, the  library has an extensive collection of  feature films on dvd and video. the  collection benefits the entire bond  community and is heavily used. however,  academic staff and students in the ftv  school had a number of issues concerning  their access to the feature films:    due to the popularity of the films,  ftv staff and students felt that the  dvds and videos were often not  available for their primary purpose  (i.e. the teaching and learning needs of  ftv courses). they felt that the dvds  and videos they wished to access were  often damaged, lost or loaned to  borrowers who were not undertaking  ftv courses.    the arrangement of dvds and videos  using the library of congress  classification and a genre designation  was too difficult to use and a title  order arrangement was requested.    evidence to resolve the questions posed  was gathered using a survey, analysis of  reports of damaged and lost items,  analysis of loan patterns and perusal of  the policies applied by other university  evidence based library and information practice 2006, 1:1    62 libraries. a literature review was also  undertaken.    the evidence gathered is summarised as  follows:    a survey of the management practices  used in australian university libraries  was conducted using caul to  distribute a questionnaire. this   confirmed that most universities have  relatively open access polices for their  films on dvd and video; i.e. loans  available to all borrowers, material on  open access, standard loan periods,  multiple items allowed for borrowing.    the survey revealed libraries were  using a variety of security  arrangements including traditional  library security strips as well as  locked security cases. according to the  survey results, most libraries were not  experiencing a great deal of theft or  loss of feature films. videos were  found to be the most prone to damage  due to their aging format.    the survey showed there were  significant differences in how libraries  shelve their feature films although  about a third of those surveyed use a  system that effectively organises them  in title order.    according to the survey, 70% of  caul libraries use some form of  booking system to ensure films are  available for class purposes.    the results of the caul survey are  available on the caul website  (“feature film collection  management”).    the literature review indicated that  management of audiovisual  collections and especially feature films  in academic libraries is an evolving  situation for all aspects of their  collection management and access.  the trend is strongly towards  integrating audiovisual collections  into the mainstream of library  collections, minimising specialist  treatment due to the medium (merry;  brancolini).    analysis of bond library’s loan  statistics confirmed that usage of the  feature films was high compared with  other parts of the collection but that  most borrowers were respecting the  loan period.     analysis of lost and missing reports  has been ongoing and this has  highlighted the titles that are most  susceptible to theft, namely dvds of  popular television series.     based on these findings ftv academic  staff were consulted and a number of  steps could be taken immediately. for  example, a manual booking system for  staff to reserve their titles for class  teaching was promoted and an automated  system is under consideration. secondly,  feature films were re‐shelved in title order.  lastly, security was increased for the  popular television series that were found  to be susceptible to theft by relocating  them to closed access at the loans desk.  lockable cases are being investigated in  order to be able to return them to open  browsing access.    a report with recommendations is being  prepared to vary the loans policy,  including proposals to limit both the  number and length of time that feature  films can be borrowed. the new policy is  to be discussed and agreed with  stakeholders and would incorporate  maximum flexibility for academic staff  and students in the ftv school to facilitate  their borrowing for teaching, learning and  research purposes.      case study on teaching undergraduate  students endnote      this case study is at an early stage and  thus is less complete than the two case  studies described above.    evidence based library and information practice 2006, 1:1    63 question: how can undergraduate  students be supported to learn the basics  of using endnote?    at bond university all undergraduate  students are required to undertake a series  of core courses including a course  delivered by the school of information  technology (sit) that is designed to   ensure that students understand the key  concepts of information technology. the  course aims to enable all students to gain  competency in using a range of  productivity packages. in 2005, it was  decided to add endnote to the list of  applications that students would master in  this course.    since the acquisition of a university site  licence for endnote in 2002, the library has  been responsible for support and teaching  of the program within the university  community. acknowledging this expertise,  sit staff called on the library’s support in  the new initiative.     the literature was reviewed and a survey  of practice in australian universities was  conducted via caul as summarised  below:    the literature review revealed a  plethora of articles about the technical  aspects of endnote including reviews  of new versions of the software but  relatively few articles on support and  teaching endnote (east; siegler and  simboli).    the caul survey showed that all but  one of the thirty‐four universities that  responded had a site licence for  endnote and those thirty‐three  provide support and training in some  form. the institutions surveyed  indicated that they focus their endnote  training on staff, researchers and  postgraduates; however, sixteen  provide endnote training for  undergraduates, usually at the request  of academic staff.     in terms of evaluating training,  seventeen of the institutions surveyed  via caul ask participants in  workshops to evaluate content and  delivery to inform course  development but none appeared to  assess the learning of participants or  use exams to test competency. a few  are using quizzes and exercises for  learners to check their own progress  as they work through endnote  tutorials.    the results of the survey are available  on the caul website. (“endnote”)     as the evidence gathered did not reveal  any assessment techniques that could be  used to evaluate competency in using  endnote, further discussions were held  with sit academic staff and it was agreed  that liaison librarians would give input  to the development of the endnote  mastery tests that would be produced by  sit staff. library staff conducted endnote  training for the sit tutors who would  deliver the practical tutorials for  undergraduates. it was also agreed that a  liaison librarian would be coopted to  introduce endnote in the lecture program.    at the end of the semester, students’  results on the endnote mastery test will be  analysed. a broader question about the  impact of teaching endnote still needs to  be researched and answered: how will  teaching undergraduate students endnote  impact on their overall information skills?    evaluating our performance    using evidence based practice at bond  university library constitutes a deliberate  strategy for introducing change and  continuous improvement. while  professional experience and judgement  are not discounted, using evidence based  practice is an attempt to move away from  relying solely on anecdote and personal  experience to solve important problems in  service delivery.     analysing our performance in applying  evidence based practice at bond  university illustrates our successes as well  evidence based library and information practice 2006, 1:1    64 as our limitations. one of the highlights  and an essential feature of the process at   bond library is the involvement of  stakeholders. this varies from seeking  their input to partnering with them to  learn from their expertise. the ability to  build consensus and agree upon action  plans with stakeholders is an important  outcome of that process. stakeholder  involvement is akin to the patient centred  approach that is a hallmark of evidence  based medicine (sackett et al.). in the bond  context, it continues a strong tradition of  developing and maintaining good  relationships with customers.     in critically analysing each step of the  evidence based approach we used, the  three case studies have been a valuable  learning experience:     specifying the question: in the case of the  opening hours issue and the access to  feature films problem, the questions were  focussed by analysing the feedback from  customers. in the case of support for  endnote the question was not as easy to  define and as it transpired, the broader  question that could have been asked still  needs to be addressed. in all the projects a  structured process such as pico or spice  could have helped focus the questions and  in turn would have aided the process of  choosing the research design (brice, booth  and bexon 5‐6).    finding the evidence to answer the problem: in  each case, a range of techniques was used  to gather evidence, including literature  searches, surveys and other data gathering  methods. as reported by brice, booth and  bexon (7), the literature search process  was complicated due to the range of  information sources that comprise the  evidence base. the databases we searched  included ebsco’s academic search premier,  business source premier, library information  science and technology abstracts and  professional development collection. infotrac  expanded academic and eric were also  searched. in each case, structured search  methods were used across the target  databases. searching a wider range of  sources could have produced better  results but this was not practical within  the time available.    in each case, a survey was conducted of  caul members (40 australian university  libraries) and the council of new zealand  university librarians members (6 new  zealand university libraries). the surveys  gathered a broad base of valid and reliable  evidence about current practice in the  sector. however, the survey methodology  we used could have been more rigorous.  for example, the questions could have  been more thoroughly tested to obtain  more accurate and comprehensive  information and more thought could have  been given to how the responses would be  analysed. if the survey methodology had  been automated, the results would have  been easier to analyse. web‐based survey  applications with inbuilt analysis  methodologies can save a lot of time for  those responding to surveys as well as  those analysing the responses. while  using surveys as the predominant research  method may not meet the gold standard  for research rigor (west 12–13), in the case  of the questions raised at bond university,  they satisfied the need to gather evidence  quickly so that pressing service problems  could be resolved in a timely manner.    appraising the evidence: the evidence base  generated by the literature searches was  informative and contributed to an  improved understanding of the issues. but  as found by booth and brice (10), the  results for the literature searches  conducted for all three case studies  showed few studies with findings that  could be transferred to the current  situations.  this may be inevitable given  the practical nature of the questions being  posed. fortunately, the lack of transferable  results in published studies was partly  overcome by the broad base of evidence  generated through the caul surveys. the  survey information was analysed and the  major trends for the sector could be  readily identified and used as a guide for  practice at bond university. another  valuable evidence base in two of the three   cases described was the analysis of local  usage statistics.   evidence based library and information practice 2006, 1:1    65  applying the results: in determining the  applicability of the evidence assembled,  questions about costs, severity, timeliness  and politics, as suggested by  koufogiannakis and crumley (121–123) all  came into play. in each case feedback from  stakeholders had been significant in  raising the issues in the first instance.  their involvement was equally important  in deciding how the results of the research  would be applied to address those issues.  in each case, library staff discussed the  evidence with stakeholders and through  the process a consensus developed on how  best to proceed. both library staff and  stakeholders were informed by the  research and assured that plans to take  action were soundly based.    evaluating the outcome and redefining the  problem: the opening hours case was  concluded to the satisfaction of both  library staff and stakeholders and is now  pending a refurbishment of the library.  this has effectively defined a new  direction for our research. the other two  cases have not yet been concluded  although following initial research and  consultation with stakeholders some  interim actions have been undertaken and  this has allayed some of the biggest  concerns of stakeholders especially in the  case of managing access to feature films.   in evaluating the endnote case study, the  broader question about the impact of  teaching endnote on students’ information  skills has emerged to define a new  research question.    conclusion    the three case studies illustrate bond  library’s first intentional foray into using  evidence based practice to capitalise on  well established quality processes and to  stimulate innovation and change. the  concept of “professional practice built on  the explicit and judicious use of current  best research findings in making decisions  about the performance of the day‐to‐day  role” seems hard to fault (todd 9).  however, as the bond university  experience has shown, there are barriers to  overcome even when the work  environment is conducive. like most  professionals, librarians at bond  university need to develop the skills and a  culture to effectively carry out evidence  based practice. these include the skills to  articulate questions, undertake research,  appraise research findings and implement  a course of action. above all it requires  librarians to develop a culture of  questioning and reflecting on what we do.  despite the limitations in some aspects of  the evidence based practice used at bond  library, the success of the three research  projects described in this paper amply  demonstrates the benefits of an evidence  based approach. persuasive evidence!      works cited    “24x7 library facilities.”  surveys/questionnaires distributed to  caul no. 2004/27. caul. 17  february 2006  <http://www.caul.edu.au/surveys/>.     booth, andrew. “evaluating your  performance.” evidence‐based  practice for information professionals:  a handbook. ed. andrew booth and  anne brice. london: facet publishing,  2004. 127‐37.    booth, andrew, and anne brice. “why  evidence‐based information  practice?” evidence‐based practice for  information professionals: a  handbook. ed. andrew booth and  anne brice. london: facet publishing,  2004. 1‐12.    brancolini, kristine r. “video collections  in academic libraries.” video  collection development in multi‐ type libraries: a handbook. ed. gary  p. handman. westport, ct:  greenwood press, 2002. 47‐75.    brice, anne, andrew booth and nicola  bexon. “evidence based librarianship:  a case study in the social sciences.”  proceedings of the 71st ifla general  congress and council, august 14‐18,  2005, oslo, norway. 19p. ifla. 1 sept.  2005,  http://www.caul.edu.au/surveys/ evidence based library and information practice 2006, 1:1    66 <http://www.ifla.org/iv/ifla71/papers  111e‐brice_booth_bexon.pdf>.    curry, ann. “opening hours: the contest  between diminishing resources and a  24/7 world.” journal of academic  librarianship 29 (2003): 375‐85.    east, john w. “academic libraries and the  provision of support for users of  personal bibliographic software: a  survey of australian experience with  endnote.” lasie 32.1 (2001): 64‐70.    “endnote.” surveys/questionnaires  distributed to caul no. 2005/4.  caul. 17 february 2006  <http://www.caul.edu.au/surveys/>.     engel, debra, kay womack and ursula  ellis. “opening a library twenty‐four  hours.” journal of library  administration 36 (2002): 95‐109.    “feature film collection management”  surveys/questionnaires distributed to  caul no.2005/8. caul. 17 february  2006  <http://www.caul.edu.au/surveys/>.     koufogiannakis, denise, and ellen  crumley. “applying evidence to your  everyday practice.” evidence‐based  practice for information professionals:  a handbook. ed. andrew booth and  anne brice. london: facet publishing,  2004. 119‐126.    merry, lois k. “the devil in the details:  an academic library acquires a  video collection.” library collections,  acquisitions and technical services 28  (2004): 298‐311.    sackett, david l., et al. evidence‐based  medicine: what it is and what it isn’t.  1996. centre for evidence‐based  medicine. 7 sept. 2005  <http://www.cebm.net/ebm_is_isnt.as p>.     siegler, sharon, and brian simboli.  “endnote at lehigh.” issues in science  and technology librarianship 34  (2002). association of college and  research libraries, science and  technology section. 1 sept. 2005  <http://www.istl.org/02‐ spring/article4.html>.    steele, patricia ann, and carolyn walters.  extended library hours, spec kit no.  264. washington, d.c.: association of  research libraries, 2001.    todd, ross. “ross todd’s letter from  america.” orana 39.3 (2003): 9‐10.    west, kathy. “the librarianship report:  convincing evidence; seeking out the  best available evidence.” information  outlook 7 (2003): 12‐15.     http://www.ifla.org/iv/ifla71/papers http://www.caul.edu.au/surveys/ http://www.caul.edu.au/surveys/ http://www.cebm.net/ebm_is_isnt.as http://www.istl.org/02%e2%80%90 evidence based library and information practice 2006, 1:1    67   appendix a: details on quantitative analyses used    quantitative analysis of feedback about opening hours    to assess the level of feedback about opening hours, unsolicited feedback received in the  library’s suggestion box and “free” comments recorded in the library’s 2004 customer  survey (conducted by the rodski behavioural research group) were measured as a  percentage of the total number of comments received.     the analysis showed that in 2004, 7% of suggestion box comments and almost 4% of free  comments in the customer satisfaction survey were requests for longer opening hours. in the  first half of 2005 the number of suggestion box requests for longer opening hours grew to  10%.     usage patterns identified in the facilities use survey    bond university library regularly conducts a facilities use survey by counting the number  of users occupying the library’s various facilities (study carrels, open tables, lounge chairs,  library catalogues, student use computers, laptop ports, group study rooms, dvd and video  players, etc.). the survey also includes a total head count. a precise count is taken four times  daily throughout the library’s opening hours during the survey period (two weeks each  semester).     in 2004, the facilities use survey illustrated that patronage was heaviest at mid afternoon and  showed that patronage grew and declined on either side of the peak time of day; building  gradually from opening time and tailing off towards closing time.      evidence based library and information practice 2006, 1:1    68 appendix b: detailed costing to extend library opening hours    the major expense in extending the library’s opening hours is the cost of additional staffing  and to illustrate the scale of funding that would be required, the costs of four scenarios were  developed. the scenarios are listed below from the most expensive (1.) to the least expensive  (4.) diminishing by approximately 50% for each option down the list.     1. open 24x7 for 2 weeks per semester during exam periods: cost per annum for 2 libraries  (main and law)                 2. open extended hours for friday, saturday and sunday (until 9.00pm) throughout the  semester: cost per annum for 2 libraries (main and law)     3. open until midnight (monday to thursday) for second half of semester (7 weeks): cost  per annum for 2 libraries (main and law)       4. extend opening hours for friday, saturday and sunday until 11.00pm for the last three  weeks of the semester: cost per annum for 2 libraries (main and law).    option 1 would also require additional security arrangements including cctv and close  surveillance by security staff.      evidence based library and information practice 2012, 7.1 1 evidence based library and information practice editorial looking forwards and looking back alison brettle editor-in-chief senior lecturer, school of nursing, midwifery and social work, university of salford salford, united kingdom email: a.brettle@salford.ac.uk 2012 brettle. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. welcome to the seventh volume of evidence based library and information practice. this is my first issue in my new role of editor-inchief, and i’m honoured and delighted to take over the position. as many of you may know, i’ve been involved with the journal almost since its inception, primarily as associate editor (articles). during that time, i’ve sincerely enjoyed working with the supportive eblip team, as well as many authors throughout the world, and i’m looking forward to continuing that work in my new role. spring is traditionally a time of new growth and change, and the eblip journal is no different. in the last issue, denise koufogiannakis reflected on how the journal had grown and developed during its first 6 years, and i look forward to contributing to its continued success. there have been a number of changes in the editorial team. i welcome wayne jones from carleton university, canada, who has taken over as associate editor (articles) and brings a wealth of editorial experience. heather pretty has taken over as lead copyeditor and is joining our editorial meetings to help ensure the continued quality and consistency of the journal. as part of quality assurance and development, the editorial team is examining and revising our journal guidelines, so look for those in forthcoming months. our former editor-inchief, denise koufogiannakis, is developing a new role as associate editor (reviews). as a keen advocate of reviews for providing evidence, developing skills for research and evidence based practice, and documenting and establishing an evidence base for our profession, i’m looking forward to the first review which is likely to be published in the next issue. too much change can be disruptive, so i’m pleased that both lorie kloda and jonathan eldredge are maintaining their positions as associate editors for evidence summaries and classics, respectively. when starting something new, it is often useful to look back (to build on experience or make sure we don’t make the same mistakes!). when i looked back on my own evidence based library and information practice journey, i realized it began in the mid 1990’s, certainly before i was even aware that the phrase had mailto:a.brettle@salford.ac.uk� http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/� evidence based library and information practice 2012, 7.1 2 been coined. i’ve never worked in a library, and my first professional post was as an information specialist within a research unit that supported evidence based health care. i knew little about research and even less about evidence based practice, i clung to the hope that i knew something about being an information professional! a long time academic, my manager championed library and information professionals believing they had a key role to play in the evidence based practice movement – whether by finding information, developing information products, or helping others to find information for practice. furthermore, he encouraged me to examine my own practice, and when there were no answers to some of the problems we came across, to research, evaluate, and write about it. this encouragement resulted in my first forays into being an evidence based library and information practitioner (brettle, long, grant, & greenhalgh, 1998; brettle & long, 2001). i challenge you to do the same; you never know where it may lead. much of the debate about evidence based library and information practice has been about its name (e.g., booth, 2003), its definition (e.g., booth, 2003), the evidence (e.g., crumley and koufogiannakis, 2003 and eldredge, 2004), and most recently, whether it has a future or whether we should all retire (booth, 2011). for me, eblip is not about the name; it is about what we do, how we behave, and how the library and information profession can have a continued future. (and in case you’re wondering, i’m not about to retire). our professional roles are all about evidence (in its various forms). the contexts in which we work are changing, but as a profession we have skills that make us good at dealing with research, information, knowledge, or evidence (for example, organizing, writing, or searching). it therefore makes sense that we build evidence based approaches into our working lives. we can use these approaches to transfer our knowledge to different contexts and demonstrate our value and worth to employers. this may take various forms – critically reading a paper to get new ideas, seeking out like minded colleagues to discuss their approach to a problem, evaluating a service or testing alternative approaches so that we work in the most effective and efficient way, and so on. the purpose of the journal is to “provide a forum for librarians and other information professionals to discover research that may contribute to decision making in professional practice.” it does this in a number of ways which i hope encourage you, our readers, to incorporate evidence based approaches into your working lives. evidence summaries seek to provide practitioners with a user friendly overview of research that they can use in their decision making, articles focus on generating eblip related research, the using evidence in practice section focuses on the practical use of evidence, and finally, the commentaries provide a forum for debate. over the past 6 years i have watched and contributed to the journal’s development and seen its impact on professional practice. i look forward to contributing to its future and hope that you will continue to give it your support. references booth, a. (2003). bridging the research practice gap? the role of evidence based librarianship. new review of information & library research, 9(1), 323. doi:10.1080/13614550410001687909 booth, a. (2011) is there a future for evidence based library and information practice? evidence based library and information practice, 6(4), 22-27. brettle, a. j., & long, a. f. (2001). comparison of bibliographic databases for information on the rehabilitation of people with severe mental illness. bulletin of the medical library association, 89(4), 353-362. brettle, a. j., long, a. f., grant, m. j., & greenhalgh, j. (1998). searching for information on outcomes: do you need to be comprehensive? quality in health care, 7(3), 163-167. evidence based library and information practice 2012, 7.1 3 crumley, e., & koufogiannakis, d. (2002). developing evidence-based librarianship: practical steps for implementation. health information & libraries journal, 19(2), 61. doi:10.1046/j.1471-1842.2002.00372.x eldredge, j. d. (2004). inventory of research methods for librarianship and informatics. journal of the medical library association, 92(1), 83-90. / evidence based library and information practice evidence based library and information practice evidence based library and information practice 2009, 4:3  evidence based library and information practice       article    access all areas: exploring the use of library and it facilities by university of salford  pre‐registration diploma nurses during periods of clinical practice placement    mike raynor  information specialist, national institute for health and clinical excellence (nice)  manchester, united kingdom  email: michael.raynor@nice.org.u.k.     received: 01 december 2008        accepted: 10 july 2009       © 2009 raynor. this is an open access article distributed under the terms of the creative commons attribution  license, http://creativecommons.org/licenses/by/2.0>, which permits unrestricted use, distribution, and  reproduction in any medium, provided the original work is properly cited.    abstract    objective ‐ to assess the scale of library and it resource access issues reported  anecdotally by nursing students on clinical placement. to map patterns of it and  library usage behaviour to inform future service plans for this population of students.    methods ‐ a multi‐response self‐completion questionnaire administered on a group  basis to 78 pre‐registration diploma nursing students.    results ‐ the survey group were heavy users of library and it facilities while on  placement (87% of respondents accessed library and it facilities at least once per  week).  survey respondents encountered barriers to accessing these resources,  including lack of awareness of local facilities, lack of time to access facilities,  supervisors’ attitudes towards use of library and it facilities by placement students,  and feeling inadequately prepared for accessing and using resources at a distance  from the university. respondents relied heavily on facilities in the placement location  and university facilities and responses suggest that use by location was strongly  linked to resource format. use of public library facilities to support study while on  placement was revealed by the survey. respondents’ main source of help for library  and it issues was clinical mentors.  the survey indicated that the experience of library  and it access varied greatly with location of placement venue. the study was limited  in eliciting more detail about location‐specific experiences and findings need to be  explored further with respondents using focus group methodology.    conclusions – library and it access problems encountered by the pre‐registration  nursing students included in this survey were not large scale. although there is no  evidence of behaviour to circumvent systemic problems, the study identified small but  4 mailto:michael.raynor@nice.org.uk evidence based library and information practice 2009, 4:3  5 significant barriers.  five recommendations addressing these barriers will require  collaborative service planning between university of salford and national health  service (nhs) learning resource staff to implement. this collaboration needs to  include service users and the practice mentors who support them on clinical  placement.    introduction and background    pre‐registration nursing programmes in the  united kingdom (u.k.) are required by the  nursing and midwifery council (nmc) to  have a balance of learning that is “50%  practice and 50% theory, in all nursing  programmes” (17). the nmc was established  in 2001 by the government of the u.k. and has  a statutory responsibility to ensure all nurses  and midwives are competent to practice in the  u.k. the nmc sets the standards for nurse  education and, in its guidance to course  planners, states: “the practice part of the  programme should provide opportunities to  experience 24‐hour/7 day care” (17).    in their studies ward and moule (63) and  norton et al. (6) report students as frequently  experiencing feelings of isolation while on  placement and turning to information and  communication technologies (ict) (e.g. e‐mail,  virtual learning environments (vle)) as a  means of communication. these studies also  present evidence of barriers to access that not  only undermine the students’ educational  experience, but also have serious implications  for student retention.     middleton et al., in a study of pre‐registration  nursing students and teacher training  students’ access to learning resources, found  that during periods of placement access can be  far from equitable (25). this was particularly  marked for health students working in  community and primary care trust (pct)  settings who often had no access to either a  library or a computer terminal.  mailer (114)  reports competition for access to networked  computers with clinical staff as being a  problem while ward and moule report access  to computers as being “strictly hierarchical”  (63).      respondents in the majority of these studies  rarely explicitly differentiate between library  and it activities. this reflects not only an  acceptance of the convergence of modern  library and it services, but also the degree to  which students have routinely assimilated  technology‐driven learning activities into their  day‐to‐day studies. this assimilation poses  problems for students who have no central  library and it facilities on site, as access to  computers in the immediate area of work  (generally a ward setting) is sometimes  problematic for technical and organisational  reasons. marriott reports anecdotal evidence  of computers on wards being “unavailable for  studying and therefore cannot be considered  to meet the needs of students” (262).  middleton et al. reported that placement  students were viewed as “very low priority”  (9) with respect to it access while norton et al.  report “provision of library and it services is  still rigidly segregated along professional lines  in many national health service (nhs)  organisations” (8).      the practice placement context    the university of salford, u.k., has  approximately 1500 pre‐registration nursing  students, approximately 1000 of whom study  at diploma level. the majority of these  students are sent on placement in the greater  manchester area to 15 national health service  (nhs) trust organisations covering acute and  primary care settings. in addition to this there  are non‐nhs venues including prison health  care facilities, hospices and day care units  (university of salford, school of nursing)      <www.nursing.salford.ac.u.k./cpp/index.php  >.      during placement, students are supported,  supervised, and assessed on their skills and  http://www.nursing.salford.ac.uk/cpp/index.php evidence based library and information practice 2009, 4:3  performance by practice mentors who are  nhs staff. these staff form an integral part of  a pre‐registration nursing student’s teaching  team and there are nearly 5000 registered  practice mentors supporting pre‐registration  nursing students studying in the greater  manchester area. greater manchester covers a  geographical area of approximately 500 square  miles and has a population of some 2.24  million (office for national statistics, 2001  census).     all staff within nhs trusts have access to  library and it facilities provided by nhs  trust library services. these services, where  possible, are located on the same physical site  as the clinical facilities or at a geographically  centralized site that may service several trust  locations. while on practice placement, pre‐ registration nursing students are registered as  nhs staff and therefore have full access to  trust library and it services. the trust  libraries operate as a network of specialized  health libraries at a local, regional and national  level.     information and learning services (ils) at the  university of salford comprises it and  telephone services, information  communication and technology (ict)  training services, library services, and  performance and compliance support to all  staff and students. ils has formed robust  working partnerships with nhs library staff  in greater manchester and the north‐west  region at both the operational and strategic  levels via the library and information health  network north west (lihnn) and health  care libraries unit (hclu)  <www.lihnn.nhs.u.k.>. the partnership  includes meeting regularly to receive updates  on service developments, to resolve any  immediate operational issues and to identify  areas of mutually beneficial service  development. nhs trust and university of  salford library staff also provide reciprocal  training and university library staff are  represented on standing nhs trust library  committees. at a strategic level, university of  salford staff are routinely invited to briefings  and consulted on regional and national nhs  trust library developments.    library management system (lms) borrowing  statistics and electronic resource log‐in data at  greater manchester nhs trust libraries reveal  university of salford nursing students to be  heavy users of trust library facilities and  services.      the university of salford has purchased key  reading materials for some of the trust  libraries (termed the ‘books in trust’ initiative)  in an attempt to ease some of the pressure on  nhs trust library print collections. the work  and support of nhs trust library staff is a  major element in ils’s goal to provide  responsive library and it services to nursing  students at the university of salford.    objectives of the study    a small but significant volume of anecdotal  evidence has been gathered from pre‐ registration nursing students at the university  of salford regarding negative experiences with  library and it resources while on clinical  placement.      for example, pre‐registration nursing students  often complain about the inaccessibility of the  university’s vle from within nhs  organisations or the lack of access to local  library resources while on placement. there is  often confusion regarding the existence of  dual university and nhs electronic resource  accounts and the differing access routes for  electronic resources.      the study has two objectives:    1. to look for evidence of the issues  highlighted above in order to assess  their scale and explore underlying  causes.    2. to map patterns of it and library  usage behaviour to inform future  service planning for this population of  students.      6 http://www.lihnn.nhs.uk/ evidence based library and information practice 2009, 4:3                                                methods  a self‐completion questionnaire (appendix 1)  was administered to a convenience sample of  pre‐registration diploma nursing students  who had recently returned from clinical  practice placement (no later than one month  prior to the administration of the survey).   permission from module leaders was gained  to enter classrooms to present a brief  description of the survey before offering  students the opportunity to complete it.  completed surveys were collected and any  questions regarding the survey were  answered.        six venues were visited over a period of two  weeks during november and december 2006.  this yielded 78 useable surveys (100%  response), which represented approximately  7% of the pre‐registration diploma nursing  population at the university of salford at that  point in time (this percentage is based on  approximate recruitment numbers of 1172  students).      this method allowed groups recently  returning from placement with their  experiences relatively fresh in their minds to  be targeted and promoted a greater response                                         frequency of library visit by location (multi-response) 27 34 3 23 17 26 3 12 0 10 20 30 40 50 60 70 nhs library university of salford library other university library public library % o r re sp on de nt s monthly weekly   fig. 1. frequency of library visit by location by diploma nursing students on clinical placement  rate. ethical approval was not required to  conduct this survey.   results    overview    the 78 respondents in this survey comprised  42 students from year 1, 33 students from year  2, and 4 students from year 3.  the nhs  organisations from which these students had  returned were distributed across 20 sites in the  greater manchester area.  the majority of  respondents (64%) were based in two  organisations and 88% of respondents had  internet access at home.       furthermore, the majority of respondents  (66%) considered access to library and it  facilities while on clinical placement to be an  important issue and 87% accessed library and   it facilities at least once/week while on  placement.    location preference    figure 1 shows the majority of respondents  preferred to use the university of salford’s  library facilities (60%) or nhs trust facilities  (44%).      7 evidence based library and information practice 2009, 4:3                                                    a noteworthy minority (35%) also made use of  public library facilities, however a similar   number (33%) of respondents used both  university of salford and nhs library and it   facilities at least once/month while on  placement.    further examination of the data showed that  the location preferences in figure 1 exhibited  no trends associated with placement location  or year of study.    reasons for library visits while on placement    figure 2 shows that the most frequent reasons  for visiting a library/it facility during  placement were: to ‘use books’ 58%, ‘access  blackboard’ (the university of salford’s vle)  53%; ‘use the internet’ and ‘use journals’ were  roughly comparable at 46% and 45%  respectively. it is noteworthy that almost a  third (31%) of respondents still viewed  libraries as a place to study.                                                           during your placement for what purpose did you use the library? multi-response 0 10 20 30 40 50 60 70 us e bo ok s bl ac kb oa rd us e the in ter ne t us e j ou rn als pl ac e t o stu dy ema il ph oto co py ing w or d p ro ce ss ing ot he r no n-r es po ns e % o f r es po nd en ts   fig. 2. reasons for a library visit by diploma nursing students on clinical placement in the nhs  use by location    analysis of the responses to the questions  related to use of library and it resources by  location revealed a distinction between   location and resource type. nhs trust  facilities were clearly identified with  providing access to print books (37% of   respondents used nhs facilities for this  reason) and print journals (19% of  respondents) but less so with provision of e‐ books (3% of respondents) and e‐journals (4%  of respondents). use of the same resources in a  university of salford location revealed less  drop‐off in the numbers using electronic  equivalents: 54% of respondents used print  books while 26% used e‐books; 37% used print  journals compared with 35% who used e‐ journals.      usage of the university’s vle was  predominantly from home (54% of  respondents) or from within the university  (35% of respondents). use of academic  databases such as cinahl was mainly from     8 evidence based library and information practice 2009, 4:3                                            home (49% of respondents) or from within the  university (42% of respondents).    problems accessing library and it facilities    while on placement 33% of respondents (26  people) reported experiencing problems  accessing library resources (see appendix 1,  q.11). further analysis of the data showed that  these respondents were not linked to a  particular trust location or trust type (i.e.  primary care or acute). neither were  problems linked to year of study (12 first year  students, 13 second year students, and 1 third  year student).    some respondents (n=15) elaborated on these  issues. these responses fell into 7 pre‐defined  categories (appendix 1, q12):    1. unable to access e‐books/e‐journals  because athens passwords did not  work (4 comments)  2. no free time/not given time to access  library and it facilities (3 comments)  3. no library at placement (3 comments)  4. no internet access available at  placement venue (3 comments)  5. no study space/study space too noisy  (2 comments)  6. could not access e‐mail (1 comment)                                    staff w ho w ere consulted for he lp by place m ent students at le ast once/w ee k 0 6 5 23 0 5 10 15 20 25 school of nursing teaching staf f university of salf ord library staf f nhs library staff practice mentor % o f re sp o n d en ts   fig. 3. staff consulted for help by diploma nursing students at least once/week  7. access to recommended texts (1  comment)    training and help with library and it resources  while on placement    respondents were asked if they believed they  had received appropriate training on how to  access library resources from off‐campus  before going on placement. this elicited a  mixed response (50% yes 50% no). further  analysis provided no evidence to suggest this  split was linked to specific placement locations  or respondents’ year of study.    when students required help with library  resources during periods of practice  placement their most frequent source of  information was the practice mentor (figure  3).    issues raised by students    the final question of the survey provided an  opportunity for respondents to “list one  change you would like to make regarding  access to the library resources while on  placement”. the majority (60%) of  respondents offered at least one response  each.      three themes or main areas of concern were  highlighted:  9 evidence based library and information practice 2009, 4:3  1. have time allocated to use the library  and access the resources  (11  comments)  2. have computer/internet access on  placement (9 comments)  3. have library facilities/study areas  when on placement (6 comments)    a further 8 comments provided other  suggestions for library/resource changes  which included:    • “a larger e‐book selection.”  • “a helpline to access, for example a  telephone number (sic).”  • “make the location of library clearer.”  • “difficult to access  internet/blackboard ‐ useful to  immediately support learning on  placement.  shown the library maybe  on induction.”  • “making it easier to access hospital  library.”  • “to enter into internet on my own.”  • “be given nhs login prior to  placement to use pcʹs on placement.”  • “ease of use of passwords off  campus.”    discussion    in a recent study of the needs of health and  social care placement students, callaghan et  al. identified trends in students’ behaviour  that enabled them to “circumvent the  difficulties they encountered with resource  access and use” (258). the main findings from  this sample of students reveal that there are  some small scale issues with it and library  resource access directly attributable to being in  a placement situation. however these issues  appear reasonably small scale with only 33%  (28 people) saying that they had encountered a  problem while on placement. there is no  evidence to suggest trends in behaviour are  associated with systemic difficulties.    one of the themes that repeatedly arose was a  lack of time or not being given time by  practice mentors to access library and it  resources. this needs to be viewed in light of  the result that of the 34% who asked for help  with library and it issues on a weekly basis,  23% asked their practice mentor.      resources need to be put in place by the  university to support practice mentors in this  area of their work. clinical mentors have a  role to play in supporting students with  library and it activities, however studies  suggest that the mentors themselves require  support to do this effectively. king and  mackenzie point to the role of clinical mentors  in raising students’ awareness of library and  it facilities, but trust librarians interviewed as  part of the study report “that clinical mentors  did not always seem to have information  about trust library resources” (19).       an unpublished survey by the university of  salford suggests that there is demand by this  population for support from nhs and  university of salford library and it staff.  howard (2006) in a survey of 277 clinical  mentors in greater manchester found that 56%  would find training in the use of university  resources useful and 57% would find training  in equivalent nhs resources useful. almost a  third (30%) of respondents to howard’s  survey said that they frequently advised  students on finding learning resources. it  would also be appropriate to include in any  such intervention, elements to raise practice  mentors awareness of the pervasiveness of  electronic resources in nursing education.      conversely a relatively low frequency of use  of either salford or nhs library staff is clearly  highlighted and perhaps reveals ineffective  publicity of contact mechanisms for help with  library and it issues. this is highlighted in  comments such as “(i would like) a helpline  to access, for example a telephone number“  from respondents, when asked what library  and it service improvements they would like  to see.      recommendations    many of the nhs trust librarians in greater  manchester have begun to build strong  10 evidence based library and information practice 2009, 4:3  working relationships with practice mentors,  providing training and advice on services.  university of salford library and it services  need to work with nhs library staff to utilize  their existing relationships and expand  contacts in order to establish a presence in the  day‐to‐day working lives of this community of  associate educators.     recommendation 1: university of salford  and nhs library and it staff to develop a  strategy to raise the awareness library  and it resources targeted at the practice  mentor community.    a small number of students suggested that  one change they would like to make (in  response to q 16) would be to “have  computer/internet access on placement” (9  comments). analysis showed that these  students were based at relatively small  locations away from large hospital centres and  had no central it or library facilities. these  sites comprised pct facilities such as day  hospitals and community drop‐in centres  linked to specific mental health conditions. it  is important that these students be made  aware of their nearest accessible centre (nhs,  university of salford, other university library).      recommendation 2: university of salford  library staff should develop a marketing  campaign to run in the weeks leading up  to periods of placement aimed at nursing  students giving details of nhs trust  library locations and alternative library  and it access rights.     this would benefit all health and social care  students going out on placement and would  act to support and reinforce recommendation  3 (see below).    training    half the respondents believed that the training  they had received prior to placement did not  appropriately equip them with the skills to  effectively access resources off‐campus. the  main issues related to access (“unable to access  e‐books/e‐journals because athens passwords did  not work”, “have computer/internet access on  placement”) and may in part be as a result of  inadequate training, rather than for any  technical or organisational reasons. university  of salford training for this group focuses  exclusively on university of salford resources.  also the timing of this training in relation to  periods of clinical practice can be ad‐hoc and  frequently results in large periods of time  elapsing between students receiving training  and going on clinical placement.     although nhs library and it staff provide  introductory library and it training as part of  clinical placement induction days, the survey  strongly suggests that nhs electronic  resources (e.g. e‐books, e‐journals) are  underused by this group of students. a  partnership approach between nhs and  university of salford library staff to ensure  mutually complementary content and suitably  timed sessions is required to address these  issues.      recommendation 3: greater manchester  and university of salford library and it  staff should plan collaborative library and  it training sessions tailored for students  starting clinical practice placements.    it is interesting to note that a significant  minority (35%) of respondents used public  libraries to access resources. the most  common reason given by respondents for  using public libraries was to access books. of  all respondents who borrowed books while on  placement, 16% borrowed from a public  library. this is an area that requires further  exploration. for example, borrowing statistics  by this population and data on the type and  quality of queries they generate would enable  university and nhs library and it services to  offer the appropriate support and develop  working relationships across sectors.    recommendation 4: use of public library  facilities by placement students should to  be investigated further with a view to  developing cross‐sector working  relationships.     11 evidence based library and information practice 2009, 4:3    limitations    one of the weaknesses of the convenience  sampling used in this study was that time  constraints rendered bsc groups inaccessible  to the author. this somewhat reduced the  scope of what was originally planned as a  survey representative of the entire pre‐ registration population (i.e diploma and bsc).  the sample in this study is approximately 5%  of the entire pre‐registration population and  represents a relatively small proportion of the  diploma population at approximately 7%.     further, ready access to the pre‐registration  diploma population also resulted in a sample  population dominated by first year and  second year students. third year diploma  nursing students spend up to 14 weeks on  practice placement (the longest period in their  academic career) and data from these students  would arguably have revealed library and it  issues specifically associated with large  periods of time away from university  facilities.    gathering in‐depth detail about individual  issues was also difficult via a survey.  descriptions of problems were limited to one  or two sentences due to the format of the  questionnaire and details as to why students  were using specific resources were difficult to  capture. further it was often difficult to  establish the cause of a problem. for example  the comment “e‐books & e‐journals ‐ couldnʹt  access” made by a second year student based  at a large acute trust well served by central  library and it facilities, could be attributed to  inadequate training, technical problems or  organisational issues.    it is therefore recommended that rich data  from focus groups needs to be gathered from  this student population to enable a more in‐ depth exploration of individual access issues  in order to better establish causality.        recommendation 5: focus group  methodology should be deployed with this  population of students to enable more in‐ depth data to be gathered to further inform  recommendations 1 to 4.     black and bury discuss the potential of  working collaboratively to provide a  “considerably enhanced service” (45) if  stakeholders are fully involved in planning  and ownership of any projects. all of the  recommendations made by this survey require  greater stakeholder collaboration. the  university of salford and the greater  manchester nhs trust librarians should use  their strong working alliance to address the  issues outlined by this study.    conclusions    this study set out to assess the scale of library  and it resource access issues reported  anecdotally by nursing students on clinical  placement. it was found that library and it  access problems encountered by the pre‐ registration nursing students included in this  survey were not large scale and there is no  evidence of behaviour to circumvent systemic  problems. however, the study has identified  small but significant barriers related to: being  given time to access resources; feeling  inadequately trained to deal with electronic  resource usage off‐campus; and being  unaware of how to ask for help while off‐ campus. there is no evidence to suggest that  these problems are associated with specific  placement locations or respondents’ year of  study.    having no access to library or it facilities was  raised as an issue with students located at  nhs sites distant from central library and it  facilities.    furthermore, the study sought to map  patterns of it and library usage behaviour to  inform future service planning for this  population of students. the study  demonstrated that this population of students  prefer to use library and it facilities in  university of salford and nhs locations. a  small but significant minority use public  library facilities for the purposes of study  during periods of clinical placement. they  12 evidence based library and information practice 2009, 4:3  prefer to use university of salford electronic  library resources in preference to nhs  equivalents. there is a large proportion of  home internet access and there is significant  use of the university’s vle and electronic  databases from home.    further work needs to be carried out with this  body of students using focus group methods  to elicit more detail about the barriers  highlighted by this survey. reasons for use of  public library facilities also needs to be  explored to enable university and nhs staff  to offer appropriate support to public libraries  and facilitate collaboration.       acknowledgements     mike raynor was liaison librarian,  information and library services, university  of salford, salford, u.k. until august 2009.    this article was first presented as a paper at  the 5th international evidence based library  and information practice (eblip5) conference,  stockholm, sweden, july 2009.      works cited    baird, iain, david peacock, sharon dobbins,  and graham walton. “exploring  health pre‐registration students use  of learning resources while on  clinical placement: replication of  case studies at northumbria and  teesside universities, u.k.” health  information and libraries journal 23.4  (2006): 286‐9.    callaghan, lynne, alan doherty, susan j. lea,  and daniel webster. “understanding  the information and resource needs  of u.k. health and social care  placement students.” health  information and libraries journal 25.4  (2008): 253‐260.     black, coral, and rachel bury. “all for one,   one for all: collaboration between  nhs and higher education in  establishing provision of a multi‐ disciplinary, hospital‐based library  and information service.” health  information and libraries journal 21.1  (2004): 39‐45.    howard, roz. a report of the findings of the   survey of information training and  support needs of clinical mentors,  educators and supervisors.  unpublished report, june 2006. for  details contact roz howard,  information and learning services,  health and social care faculty,  liaison manager  r.m.c.howard@salford.ac.u.k..     king, colette, alison mackenzie, and simon   speight. an investigation into the use  of learning resources by mmu  students on placement or work based  learning in the nhs. (december  2004). unpublished report. contact  manchester metropolitan university’s  elizabeth gaskell library gaskell‐lib‐ enq@mmu.ac.u.k..     mailer, liz. “the u.k.’s smartal project: st   martin’s college health  studentsaccess to learning resources  while on placement.” health  information and libraries journal   23.2 (2006): 110‐7.    marriott, richard. “access to learning   resources for students on placement  in the u.k.: what are the issues and  how can we resolve them?” health  information and   libraries journal 22.4 (2005): 262‐6.    middleton, anne, gill rowell, and graham   walton. health and education  northumbria students access to  learning resources (hensal)  project: case studies of nhs   pre‐registration students and school‐ centred initial teacher training  (scitt) students. northumbria  13 mailto:r.m.c.howard@salford.ac.uk mailto:gaskell-lib-enq@mmu.ac.uk mailto:gaskell-lib-enq@mmu.ac.uk evidence based library and information practice 2009, 4:3  university: information management  research institute, school of  informatics, 2002.    norton, frances, pat spoor, martin gill, susan   mottram, and mark clowes. the  liquid library project: accessing  information while on clinical  placement: the experience   of leeds university health students.  leeds: leeds university library, 2003.    gazetteer of the old and new geographies of  the united kingdom. london: office  for national statistics, 1999. 6 march  2009  <http://www.statistics.gov.u.k./downl oads/ons_geography/gazetteer_v3.pdf >.      standards of proficiency for pre‐registration   nursing education: standard 6 –  structure and nature of educational  programmes. london: the nursing  and midwifery council,   2004. 21 august 2008.  <http://www.nmc‐ u.k..org/aframedisplay.aspx?docum entid=328>.    ward, rod and pam moule. “supporting pre‐  registration students in practice: a  review of current ict use.” nurse  education today 27.1 (2006): 60‐7.    14 http://www.statistics.gov.uk/downloads/ons_geography/gazetteer_v3.pdf http://www.statistics.gov.uk/downloads/ons_geography/gazetteer_v3.pdf http://www.nmc-u.k..org/aframedisplay.aspx?documentid=328 http://www.nmc-u.k..org/aframedisplay.aspx?documentid=328 http://www.nmc-u.k..org/aframedisplay.aspx?documentid=328 evidence based library and information practice 2009, 4:3  15 information services division www.isd.salford.ac.u.k.          appendix 1      a survey of library use by nursing students whilst on practice placement    section a: about you  1. what programme of study are you currently following at the university of salford?  please indicate:    diploma      bsc        2. what is your current year of study?  please indicate:    1st      2nd      3rd        3. at which trust/hospital/site was your last placement?      ……………………………………………………………    4. whilst on placement do you usually have internet access?    yes      no      5. do you have internet access at home?  yes    no        section b:  about your library usage  6.  how important do you think access to library resources is whilst on placement?  important      useful but not essential   unimportant      7.  whilst on placement did you have access to an nhs library facility?  yes    no      don’t know      8. during your placement for what purpose did you use a library?  please tick all that apply.  place to study        use the internet          use books     use journals      word processing            e‐mail      blackboard      photocopying                other              please specify ………………………..    9. whilst on placement on average how often did you use the library facilities in the following  locations?  please tick all that apply    http://www.isd.salford.uk/ evidence based library and information practice 2009, 4:3    once per week   once per month  less than once  per month  nhs library        university of  salford library        other university  library        public library          section c: about your use of library resources whilst on placement  10.  during your placement and for your studies, from which locations did you use the following  library resources?  please tick all that apply      nhs library  university of  salford  library  other  university  library  public  library   home  books            electronic  books            journals            electronic  journals            databases  e.g. cinahl            study space            blackboard              11. whilst on placement did you experience any problems accessing library resources?  yes       no     please go to q12 please go to q13    12. if you answered “yes” to q11 please briefly indicate the nature of the difficulty at each location  nhs library  books………………………………………………electronic  books………………………………………………  journals……………………………………………electronic  journals…………………………………………….   databases…………………………………………study  space……………………………………………………  blackboard…………………………………………other……………………………………………………… ……    university of salford library  books………………………………………………electronic  books………………………………………………   journals……………………………………………electronic  journals…………………………………………….   databases…………………………………………study  space……………………………………………………  blackboard…………………………………………other……………………………………………………… …..  16 evidence based library and information practice 2009, 4:3    other university library  books………………………………………………electronic  books………………………………………………   journals……………………………………………electronic  journals…………………………………………….   databases…………………………………………study  space…………………………………………………….  blackboard…………………………………………other……………………………………………………… ……    public library  books………………………………………………electronic  books………………………………………………   journals……………………………………………electronic  journals…………………………………………….   databases………………………………………….study  space…………………………………………………..  blackboard…………………………………………other……………………………………………………… …..    home  books………………………………………………electronic  books………………………………………………   journals……………………………………………electronic  journals…………………………………………….   databases…………………………………………study  space…………………………………………………..  blackboard…………………………………………other……………………………………………………… ….    section d: about getting help and training   13. before going on placement do you feel you received appropriate training on how to access library  resources from off campus?    please indicate:        yes        no      14. whilst on placement on average how often did you consult each of the following for help with  using library resources?  please tick all that apply      school of nursing  teaching staff  university of  salford library  staff  nhs library staff  practice mentor  one per week          once per month          never              15. did the help and advice you received enable you to resolve your problem?  please indicate by  circling the appropriate response(s) in the table below:    17 evidence based library and information practice 2009, 4:3  18 school of nursing teaching staff  yes        no       don’t know  university of salford library staff  yes        no       don’t know  nhs library staff  yes        no       don’t know  practice mentor  yes        no       don’t know    16. finally reflecting on your placement please list one change you would like to make regarding  access to library resources whilst on clinical placement          many thanks for your help with completing this survey    if you would like a response to any of the information you have given on this survey please  provide contact details below    name:……………………………………………………………………………………………..    preferred means of contact:  ……………………………………………………………………………………………  ………………………………………………………………………………………………………    alternatively if you have a query about any of the issues raised on this survey please contact  mike raynor, liaison officer, school of nursing:    e‐mail: m.j.raynor@salford.ac.u.k.    tel: 0161 295 2445  mailto:m.j.raynor@salford.ac.uk evidence based library and information practice 2011, 7.1 134 evidence based library and information practice news/announcements phd studentship in information science available in scotland 2012. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. http://ilab.cis.strath.ac.uk/esrc/studentships.h tml the scottish dtc pathway in information science is pleased to announce the availability of an esrc‐funded phd studentship in information science beginning in september 2012. this studentship is offered as part of the scottish graduate school of social science. an esrc studentship is offered each year, and details on the latest call are provided below. deadline for applications: 30th march 2012 eligibility: uk and eu nationals the information science pathway is a collaborative activity involving the following institutes: • edinburgh napier university, institute for informatics and digital innovation • robert gordon university, department of information management • university of glasgow, humanities advanced technology and information institute • university of strathclyde, department of computer and information sciences interested applicants should apply directly to the university where they believe their research interests are best represented by staff research activity, signalling their interest in being considered for an esrc studentship. as applications need to be checked for eligibility and supported by a potential supervisor, you are advised to make contact with your chosen department as soon as possible, and well in advance of the final deadline. by 30 march 2012, you should provide the following application materials to your university contact: • a completed sgs dtc studentship coversheet • a research proposal (maximum of 1200 words) • a cv • a statement of support from your potential supervisor(s) • a completed equal opportunities monitoring form • additional information: check with the institution to which you are applying to establish whether any additional application material is required for information on information science research activities at specific institutes applicants are advised to visit the following web pages or contact the person indicated below: http://ilab.cis.strath.ac.uk/esrc/studentships.html� http://ilab.cis.strath.ac.uk/esrc/studentships.html� http://www.esrc.ac.uk/funding-and-guidance/guidance/postgraduates/� http://www.socsciscotland.ac.uk/__data/assets/word_doc/0009/76905/sgs_dtc_pathway_studentship_coversheet.doc� http://www.socsciscotland.ac.uk/__data/assets/word_doc/0009/76905/sgs_dtc_pathway_studentship_coversheet.doc� http://www.socsciscotland.ac.uk/__data/assets/word_doc/0010/76906/dtc_scotland_eo_monitoring_form.doc� http://www.socsciscotland.ac.uk/__data/assets/word_doc/0010/76906/dtc_scotland_eo_monitoring_form.doc� evidence based library and information practice 2011, 7.1 135 edinburgh napier university prof hazel hall h.hall@napier.ac.uk university of glasgow dr ian anderson i.anderson@hatii.arts.gla.ac.uk robert gordon university dr simon burnett s.burnett@rgu.ac.uk university of strathclyde prof forbes gibb forbes.gibb@cis.strath.ac.uk the allocation of the award will be determined by the governance committee established for the pathway. general information on esrc studentships can be found at the esrc website library and information science practitioner researcher excellence award the library and information science research coalition has announced a library and information science (lis) practitioner researcher excellence award for lis professionals based in the uk and republic of ireland. the award will be presented in july 2012 to either (a) an individual librarian or information scientist, or (b) a team that has made a substantial contribution to lis research since 2009. the award winner will receive £500 and a plaque. the presentation to the winner will be made next year on monday 9th july 2012 at the british library at the last of the five dream project events. nominations should take the form of a 500‐ word statement that demonstrates the research excellence of the candidate (or candidate team) and refers to evidence of their research output (e.g. published articles, blogs, conference papers, presentations on slideshare). the deadline for nominations is 30th april 2012. nominations should be sent to hazel.hall@lisresearch.org . members of the lis research coalition board of directors will form the judging panel for the award. notes 1. in this context “practitioner researcher” means anyone who works delivering a library or information service. academics and consultants are not eligible for nomination. 2. self‐nomination is not permitted. 3. see also the main dream project web page. http://www.csi.napier.ac.uk/� http://www.soc.napier.ac.uk/~hazelh/esis/hazel.html� mailto:h.hall@napier.ac.uk� http://www.gla.ac.uk/departments/hatii/research/� http://www.gla.ac.uk/schools/humanities/staff/iananderson/� mailto:i.anderson@hatii.arts.gla.ac.uk� http://www4.rgu.ac.uk/abs/research/page.cfm?pge=5244� http://www4.rgu.ac.uk/abs/staff/page.cfm?pge=5660� mailto:s.burnett@rgu.ac.uk� http://www.ilab.cis.strath.ac.uk/� http://www.cis.strath.ac.uk/cis/staff/index.php?uid=27413� mailto:forbes.gibb@cis.strath.ac.uk� http://www.esrc.ac.uk/funding-and-guidance/guidance/postgraduates/index.aspx� http://lisresearch.org/dream-project/� http://lisresearch.org/dream-project/� mailto:hazel.hall@lisresearch.org� http://lisresearch.org/dream-project/� http://lisresearch.org/dream-project/� / evidence based library and information practice evidence based library and information practice evidence based library and information practice 2011, 6.4 41 evidence based library and information practice article barriers and facilitators to research use among allied health practitioners: a mixedmethod approach to assessment mary dunne information officer national documentation centre on drug use health information & evidence directorate health research board dublin, ireland email: mdunne@hrb.ie received: 20 sept. 2011 accepted: 13 nov. 2011 2011 dunne. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objectives – the disparity between what is known to be effective and what is done in practice points to barriers to research use among health practitioners. library and information services (lis) collect, organize and disseminate published research findings so they may be uniquely positioned to be of influence. this study aimed to identify barriers and facilitators to research use among allied health practitioners working in the alcohol and other drugs (aod) field in ireland, and to explore the services, strategies, and resources that may help alleviate these issues. methods – three focus groups were held with aod practitioners. a survey questionnaire was then sent by post to 175 counsellors. the survey included the barriers to research utilization scale (barriers scale) (funk et al. 1991), which assessed potential barriers from four factors: practitioner, setting, qualities of the research, and communication. results – the number of responses was 71 (41%). all communication-related barriers scale items, and some items associated with the setting and practitioner, were perceived to be a moderate or great barrier by the majority of survey respondents. similar issues were also raised in focus groups, where language, presentation, and time to engage with research were considered significant influences. qualitative aspects of the study also revealed scepticism about research application and relevance. mailto:mdunne@hrb.ie� evidence based library and information practice 2011, 6.4 42 all proposed lis were rated as moderate or great facilitators by the majority of respondents who expressed an opinion (those who choose “no opinion” or did not respond, 6–8%, were excluded). conclusions – the high incidence of communication-related issues among top barriers and the enthusiasm expressed about proposed library services and training reveals the key role that lis personnel can play in enabling practitioners to use research in practice. the addition of setting and practitioner factors indicates that a holistic, collaborative approach to promoting the effective use of research collections and resources is required. mixed-method data collection (focus group and survey) provided a rich source of information, and may offer a useful approach for future study. introduction “librarianship has had a long preoccupation with the research-practice gap” (booth, 2003, p.3). not only must lis be informed by evidence, there is interest in how the research supplied to clients impacts on practice (and subsequent client/patient care) (marshall, 2007). the difference between what is known to be effective and what is done in practice points to barriers to research use among frontline practitioners (hutchinson & johnston, 2006). most studies examining this gap have involved health and medical professionals. less is known about the needs of allied health practitioners, such as outreach, community, and project workers; counsellors; and education officers. workers in the aod area exemplify this grouping. this study aimed to identify barriers to using research experienced by allied health practitioners working in the aod area in ireland, and to explore the information services, strategies and resources that might facilitate better use. literature review most modern conceptions of evidence based practice recognise the importance of integrating knowledge from “scientific” research sources with knowledge from practice experience, colleagues, training, organisational policies, and clients (eccles, 2009; walter, nutley, percy-smith, mcneish, & frost, 2004). that is, balancing explicit science with the tacit “art” of practice (gioia, 2007). similarly, there are many ways in which research evidence is used. research use not only applies where there is a direct impact on decision making (instrumental use), it also includes raising awareness of research findings, changing attitudes and beliefs, increasing knowledge and understanding (conceptual use), and justifying a position, action, or inaction (symbolic use) (arney, bromfield, lewig, & holzer, 2009). diffusion and research use theory the transfer of research into practice is a complex process, though there have been recent attempts to define relevant concepts (addiction technology transfer center network technology transfer workgroup, 2011). there appears to be some consensus that there is no “one size fits all” approach to effective research transfer, and that numerous interactive factors must be considered, such as the setting, the individual, the communication channels, and the evidence itself (eccles, 2009; greenhalgh, robert, macfarlane, bate, & kyriakidou, 2004). theories, models, and frameworks in this field range from those that focus on individuals (intrapersonal and interpersonal change) to evidence based library and information practice 2011, 6.4 43 those that explain or predict action at higher levels (ecological/organisational change) (bywood, lunnay, & roche, 2008b). these may guide lis personnel when selecting types of assessment tools and interventions. for example, the parihs (promoting action on research implementation in health services) framework reveals the interdependent relationship between the nature of the evidence, the context, and the facilitation method (kitson et al., 2008). and the tcu (texas christian university) programme change model developed in the aod field, outlines the factors that might affect the transfer of research into practice (simpson & flynn, 2007). both of these emphasize careful diagnosis and planning within client contexts, and underpin useful assessment tools (bartholomew, joe, rowanszal, & simpson, 2007; helfrich, li, sharp, & sales, 2009; mccormack, mccarthy, wrigh,t & coffey, 2009). rogers’s diffusion of innovations (1962; 2003) theory has also influenced much study of the research-practice gap. in an extensive review, bywood et al. (2008b) placed this multidimensional, behavioural, stage theory among 20 theories that appeared to have the greatest potential for use in dissemination and implementation of innovations in the aod area. however, diffusion of innovations has been subject to criticism. for example, it does not address self-efficacy (people's beliefs about their performance capabilities, which determine how they feel, think, motivate themselves, and behave (bandura, 1994)), and the need to develop change skills (bywood et al., 2008b). the theory has also been faulted for appearing to divide the population into progressive insiders (innovators) and stagnant outsiders (imitators or “laggards”) (mcmaster & wastell, 2005). this theory must therefore be applied with care, with practitioners as partners rather than “recipients” of knowledge. despite these reservations, the theory appears to be a sound basis from which to explore research use in practice. assessment tools there are at least 25 instruments designed to measure nurses’ attitudes towards research utilisation (frasure, 2008). this proliferation of instruments demonstrates the difficulty in choosing methods that can capture the multiple factors that influence research use in practice. in a review of studies examining barriers and integration of evidence into practice between 1998 and 2007, cochrane and colleagues remarked on how few surveys were based on existing theoretical frameworks, though stated that “the exception is the comprehensive barrier assessment, barriers” (2007, p.97). based on the theory of diffusion of innovations, the barriers to research utilization scale tool was devised by funk, champagne, wiese, and tornquist (1991) to examine the barriers and facilitators to using research in nursing practice (http://barriers.web.unc.edu/). the barriers scale has been used extensively in the us, australia, the uk, and other european countries. a 2010 review by kajermo et al. included 65 studies using the barriers scale. it forms a large part of the questionnaire used in the present study. the barriers scale has been described as the most frequently used method to determine barriers in research use (hutchinson & johnston, 2004). it comprises 29 items related to four factors: the practitioner, the setting, qualities of research, and communication. reliability tests have shown that the items fit well within these factors (funk et al., 1991; yava et al., 2009), though other factor solutions have been proposed (closs & bryar, 2001; retsas, 2000). the barriers scale survey also provides an opportunity for respondents to add barriers and suggest up to three facilitators of research use. in this study the first person pronoun (“i”) and terms more familiar to irish workers were used. similar modifications have been made in previous studies (kajermo et al., 2010), however no assessment of the effect to validity was made in this case. http://barriers.web.unc.edu/� evidence based library and information practice 2011, 6.4 44 the barriers scale has been criticised for ignoring wider environmental, political, and technical factors, and not always translating well to european contexts (boström, kagermo, nordström, & wallin, 2008; closs & bryar, 2001). however the barriers scale has been used in numerous countries around the world, including ireland (glacken & chaney, 2004; parahoo, 2000). it includes multiple factors, and the content and face validity have been widely tested and supported (hutchinson & johnston, 2006). although titles may differ, similar factors have also been found using other assessment tools, for example, the parihs-based orca survey (helfrich et al., 2009). facilitators of research use successful implementation of research is affected by the type of evidence, the qualities of the context and the way the process is facilitated (kitson et al., 2008). examining all of these elements is therefore important. the most successful implementation strategies address change at multiple levels and are sustainable over a prolonged period (bywood, lunnay, & roche, 2008a). they also include particular features (table 1). table 1 features of successful implementation strategies and related factors strategies for overcoming barriersa features of successful strategiesb corresponding barriers scale factorsc • address factors relating to the individual • build relationships between researchers, practitioners and policymakers • assessment of, and focus on, barriers to change practitioner • understand and accommodate the influence of environmental context • organisational changes that require practitioners to respond or take action • clear identification of roles and activities setting • address the nature of research evidence and build agreement around evidence • reliable and credible source, with accurate, evidence based information research • improve the presentation and dissemination of research findings • tailored information that is personalised and modified to the local setting • information relevant to the practitioner and their client needs • clear and succinct message, with simple, focused objectives that require small practical changes • interactive format that is appealing, persuasive and encourages participation • reinforced messages, with additional materials and support • systems or procedures that are accessible and easy to use, with little effort required to comply communication a lewig, arney & scott, 2006; b bywood et al., 2008a; c funk et al., 1991 evidence based library and information practice 2011, 6.4 45 methods cochrane and colleagues’ (2007) review of 256 journal articles examining barriers and integration of evidence into practice, found that surveys were the most popular measure (70% of studies), but that use of qualitative (focus groups (6%); interviews (7%)) and mixed-method approaches (17%) was increasing. the three methods of data collection used in this study gave comprehensive and complementary information. the literature review provided context and guidance. the qualitative, focus group aspect allowed participants to raise their own issues in an open forum. the quantitative, survey approach then facilitated taking these themes to a larger population; enabled (some) comparison with other studies; and will facilitate future replication. focus groups three focus groups were conducted with students from an addiction studies course during a day-long visit to the national documentation centre on drug use in dublin (http://www.drugsandalcohol.ie). students were randomly assigned in groups of six or seven to sessions lasting an average of 40 minutes. the students’ occupations and backgrounds mirrored the diversity found in aod practice. the schedule was guided by existing research (callaghan, doherty, lea, & webster, 2008), and began with an introduction followed by an opening question (about research information sources) designed to promote thinking around research use. to encourage open dialogue, just two core questions (about experiences of barriers and facilitators) were asked. a round-up and brief summary were made in conclusion. all issues raised in the focus groups could be aligned with the four barriers scale factors, underlining the barriers scale’s suitability as a measurement tool. survey following a pilot study, and some adjustments to language and length, a survey questionnaire was sent to all 175 counsellors who provided alcohol, drug, or addiction counselling listed in a national online directory (32 organisations and 143 individual counsellors). the questionnaire was sent by post with an explanatory cover letter and freepost return envelope. no personal details were obtained. reminders and letters of thanks were sent ten days later. completion and return was taken as implied consent to use the data, which was collected over a five week period. the questionnaire had four sections: demographic profiles, current research use and potentially associated variables, potential library service facilitators, and the barriers scale. a final open-ended question invited comments. results response rate the response rate for this survey was 41% (71 responses from 175). this low rate, although consistent with other barriers scale studies, may suggest response bias (those most concerned with the subject are the most likely to respond). although the low response means that generalisations cannot be easily made, there were a sufficient number of respondents to indicate the challenges faced by the survey population in using research. survey respondent variables there was a good mix of rural (40%) and urban-based (60%) respondents. a high percentage of survey respondents have thirdlevel qualifications (83%), and are currently engaged in study (24%). a larger study would be able to assess whether these and other variables are correlated with barriers and facilitators of research use. http://www.drugsandalcohol.ie/� evidence based library and information practice 2011, 6.4 46 leadership support has been found to be an important facilitator of research use (yava et al., 2009). of the survey respondents who had managers (n=39), the majority were positive about the support they received from their managers in finding evidence (59%) or engaging in training courses (64%) during work hours. most respondents (76%) had access to the internet at work, which suggests this is a useful medium for practitioners to access the aod literature. more than half searched for evidence from published research sources at work at least once a week/month (59%). an even higher percentage searched for evidence from home (70%). only one respondent never searched for aod research. these responses suggest an active and motivated group of aod practitioners that requires access to resources at various times and locations, including outside of office hours. although numbers were small, some tests of association between variables were performed, for example, between level of management support and frequency of searching; between type of setting (rural and urban) and how frequently new research is discussed with colleagues/peers; and between level of qualification and frequency of searching for evidence. there was just one statistically significant relationship. higher qualifications were associated with more frequent searching for evidence at home (p=0.001). barriers scale the focus group discussions were not intended to provide a representative list of barriers, but rather a general impression of what influenced participants’ use of published research. emerging themes were compared against the four factors of the barriers scale. analysis showed that key themes were repeated in each session. survey respondents scored each barriers scale item from 1 (not a barrier) to 4 (a great barrier). there was also a “no opinion” option. the average score across all items was 2.37. the top 13 items were rated as a moderate or great barrier by the majority of respondents who expressed an opinion (table 2). twelve survey respondents added items that they perceived as barriers. some of these related to existing barriers scale items (lack of time, not user friendly, poor presentation, too complex) or emphasized unease with the type of research being produced: research results often tend to dehumanize the client. (survey respondent 4) not focused enough on 'social psychology' of addictions. (survey respondent 5) barriers that could not be easily associated with existing items related to lack of information sharing across groups or organisations, funding constraints, and cultural differences when integrating research into practice. characteristics of the communication in this study, issues associated with communication were seen as the most significant barriers to research use. for focus group participants, issues of presentation and language were highlighted in all sessions. many practitioners had no training in statistical analysis or were unfamiliar with the scientific terminology used in articles related to practice. this made relevant publications incomprehensible. the language is very important. if the language looks like it’s too technical then it’s not as useful as if it’s easy to understand. (participant 2, group 1) issues of clarity, language, and relevance were considered among the greatest barriers to research use by survey respondents also. of note, seven of the top eight barriers related to research communication. evidence based library and information practice 2011, 6.4 47 table 2 mean score for barriers scale items communication setting research practitioner moderate/ great barrier % 1 statistical analyses are not understandable 3.19 78 2 implications for practice are not made clear 2.86 71 3 the amount of research information is overwhelminga 2.75 66 4 the relevant literature is not compiled in one place 2.75 65 5 the research is not reported clearly and readably 2.73 60 6 i do not have time to read research 2.63 54 7 research is not readily available 2.60 55 8 the research is not relevant to my practice 2.56 59 9 i feel results are not generalizable to my setting 2.54 55 10 i am unaware of the research 2.54 55 11 i do not feel capable of evaluating research quality 2.51 62 12 facilities are inadequate for implementation 2.50 53 13 medical staff will not cooperate with implementation 2.50 53 14 there is insufficient time on the job to implement new ideas 2.40 43 15 the literature reports conflicting results 2.35 42 16 i am isolated from knowledgeable colleagues 2.30 41 17 i am uncertain whether to believe research results 2.26 42 18 the research has not been replicated 2.24 32 19 administrators/managers will not allow implementation 2.24 41 20 other staff are not supportive of implementation 2.22 40 21 the research has methodological inadequacies 2.20 38 22 i feel benefits of changing practice will be minimal 2.13 33 23 i do not feel i have enough authority to change 2.10 34 24 research reports/articles are not published fast enough 2.05 36 25 there is not a documented need to change practice 1.98 35 26 the conclusions drawn from the research are not justified 1.92 21 27 i see little benefit for myself 1.81 23 28 i do not see the value of research for practice 1.51 12 29 i am unwilling to change/try new ideas 1.48 12 total mean for factors 2.78 2.39 2.17 2.03 aitem 3 does not belong to a specific factor but has been added here under communication. this does not affect the mean score. evidence based library and information practice 2011, 6.4 48 characteristics of the setting many allied health practitioners do not work in traditional organisations or in one single location. for this reason, this study uses the term “setting” instead of “organisation.” focus group participants raised issues of support, authority, and capacity. for example, lack of time was raised in all sessions. this was also a key theme for survey respondents. characteristics of research evidence focus group participants wanted more local research, done in a way they thought better reflected the people with whom they worked. it is necessary to change policy but we need research done by grass roots – putting it out there so that people can get it. (participant 3, group 3) barriers scale results for this factor did not show that the qualities of research were an impediment to research use. however, a number of comments mirrored the concerns voiced in the focus groups about the type of research being produced. the stats are useful to plot social trends, but not for the micro-level i work at. we need a more human faced and centred scientific approach. (survey respondent 4) characteristics of the practitioner in terms of practitioners’ views about research, two contrasting themes arose during the focus groups: a general appreciation of the need for research, and scepticism about the motivation behind some research. research is done just to say to the public that they are doing something. (participant 2, group 3) some saw it as the responsibility of policy makers rather than of frontline practitioners to implement findings. i wouldn’t have a whole lot of faith in research…it’s great to have reports, but nothing getting done. (participant 2, group 1) not all practitioners felt capable of engaging in, or using, research. this view may come, in part, from how they saw their own role: i think that people in the caring business don’t have the scientific mindset and shy away from research. (participant 3, group 3) in the survey, the lowest ranked barriers scale items were associated with the practitioner factor. the low mean scores for these items suggest that respondents did not dispute the value of research or see lack of personal willingness to use research as a significant barrier. the three highest rated practitioner items may be associated with having motivation but lacking the skills and resources to use research effectively. this is borne out by results from other parts of the survey (for example, high levels of current research use, and high demand for facilitators). in light of the scepticism around research use raised in focus group discussions, the option “these skills are not relevant to my work” was added to the training preferences section of the survey. thirteen percent (n=9) chose this option. although not obviously reflected in the barriers scale results, this view was further supported by survey comments. a lot of expensive research produces results that many people know already. (survey respondent 45) the direct and potentially critical nature of barriers scale practitioner items may make them unsuitable for measuring barriers associated with attitude and competence. qualitative measures can give greater insight into these components. in this study, more cautious attitudes to research use were revealed in comments by some practitioners. evidence based library and information practice 2011, 6.4 49 facilitators one might reasonably assume that the resolution of barriers from every factor would facilitate research use. as part of the barriers scale, 46 respondents (65%) proposed general facilitators. these items (roughly) mirrored barrier items, and once again emphasized research communication as the primary factor. in the light of the focus group discussions and previous research (bertulis & lord, 2005), survey respondents were also asked to rate a list of potential lis facilitators, scoring from 1 (“not a facilitator”) to 4 (“a great facilitator”). all options were rated as a moderate or a great facilitator by the majority of respondents who expressed an opinion (table 3) (those who choose “no opinion” or did not respond, 6–8%, were excluded from analysis). training enabling continuing education is consistent with rogers’s diffusion of innovations theory, that a knowledge stage is a prerequisite for research implementation. research-related training has relevance to all four barriers scale factors. survey respondents were asked to choose their preferred type, format and setting for training (table 4, figures 1 & 2). respondents favoured practical workshops (44%), though blended (24%) and virtual (19%) formats were also popular. only two respondents (3%) preferred one-to-one sessions, and only five (8%) favoured having sessions in their own settings. table 3 proposed library and information service facilitators mean score scored as moderate/great facilitator (%) a list of evidence based websites 3.62 89.4 an email newsletter summarising research news 3.60 87.9 “plain english” summaries of new research 3.54 91.0 the facility to email an information professional with a question 3.28 81.8 organised meetings with others from your field 3.14 72.2 current awareness service 3.09 80.0 help with literature searching 3.03 67.7 an online discussion forum with others from your area of practice 2.79 53.0 note. for the open option “other facilitator” no lis suggestions were made. table 4 preferred type of training on understanding and using research type of training no. % advanced information-searching skills 35 49.3 critical analysis of research 32 45.1 research and evaluation methods 23 32.4 basic information-searching skills 14 19.7 basic computer skills (including email and internet) 4 5.6 these skills are not relevant to my work 9 12.7 note. for the open option “other training” – one suggestion, “experiential training,” was made. aparticipants could choose more than one type of training evidence based library and information practice 2011, 6.4 50 discussion the data collection methods used in this study revealed numerous issues to be addressed and ways to overcome them. the four-factor approach of the barriers scale provided a useful way of organizing themes from both focus groups and survey. communication for the communication factor there was clear agreement between focus group and survey respondents. the prominence of this factor may be partly an artefact of the lis focus of this study. however, it is likely that a diverse population without specified research oriented qualifications or professional resource support systems do encounter these issues. this prominence provides both a challenge and opportunity for lis in this area. setting according to rogers, organisational acceptance is a prerequisite to an individual’s ability to adopt an innovation, and a system’s norms can be a barrier to change (2003). focus on this factor is in line with recommendations from aod literature that the scope of research transfer should be broader than training; it should also involve figure 1 preferred format for research-related training figure 2 preferred setting for research-related training one to one s es s i ons 3% group l ecture s es s i ons 9% practi cal works hops 45% vi rtual l earni ng envi ronment, vi a the i nternet 19% bl ended l earni ng (mi xed face-face & i nternet) 24% preferred training format , n=67 ons i te (own organi s ati on) 8% offs i te (e.g. heal th l i brary) 17% onl i ne 17% combi nati on of s etti ngs 58% preferred training setting, n=65 evidence based library and information practice 2011, 6.4 51 workforce development, so that change is accepted and reinforced at all levels in an organisation (addiction technology transfer center network, 2004). barriers scale scoring by nurses has shown remarkable consistency, with “insufficient time for nurses to implement new ideas” as the most frequently cited barrier in 73% of studies (carlson & plonczynski, 2008). it is interesting to note that the item related to “lack of authority to change” has been rated as the greatest barrier in a number of studies (glacken & chaney, 2004; parahoo, 2000). perhaps because counsellors have more clinical control than other practitioners, this item scored lowest of the “setting” barriers in the present study. this demonstrates that results cannot necessarily be transferred across client populations. research qualities in this study, survey items associated with the qualities of research received the second lowest total average score, indicating that participants generally trusted research methods and outputs. however, the comments made by focus group and survey respondents revealed that the type of research being conducted was often not relevant to practice. whereas other researchers have attributed “relevance” to the research factor (osterling & austin, 2008), in the barriers scale relevance barriers are ascribed to the communication and setting factors. this might explain why, despite the reservations about research found in comments, there was a low barrier score for the research qualities factor. practitioner according to rogers (2003), the first step in the innovation-decision process is being aware of relevant research. but having sufficient information about what research does, its effects, and how to use it, is also a crucial early influence. although a positive attitude alone is not necessarily sufficient to bring about the use of research, it has been found to influence and be a strong predictor of use in practice (hutchinson & johnston, 2006). the practitioner factor had the lowest average barrier score, though two items (being unaware of research and not capable of evaluation) were rated as moderate/great barriers by the majority of respondents. as in other studies, items related to personal motivation were rated at the bottom of the barriers scale (glacken & chaney, 2004). there may be some bias in a question that asks practitioners whether their views or behaviour interfere with perceived competence, and comments suggested both positive and negative attitudes to literature-based research in this population. although most barriers scale items associated with practitioners were not rated as significant barriers, comments showed that some counsellors believed that treatment should stem almost exclusively from experience and evolving client need, rather than from literature-based research. other studies have found social care cultures to be experiential rather than empirically based (callaghan et al., 2008; lewig et al., 2006). training careful planning is needed in order to offer learning opportunities and pathways that capture various skill levels, and to overcome geographical and temporal constraints. for allied health practitioners, training that incorporates experiential learning is more likely to be effective (skinner, roche, freeman & mckinnon, 2009). counsellors who feel isolated in work may not commit to using training interventions effectively (joe, broome, simpson & rowan-szal, 2007). additional support and management “buy-in” may therefore be required. these factors must be built into needs and readiness assessments for the design of training programmes. evidence based library and information practice 2011, 6.4 52 study limitations and lessons learned although self-report data may not perfectly match reality, a mixed-method approach to data collection provided a rich source of information. focus groups participating groups in this study were from a convenience sample, with attendance at the national documentation centre on drug use providing a valuable opportunity to obtain their views. participants were from a rather specific cohort – those undertaking education – and may therefore have had an awareness of research that did not reflect that of the wider practitioner population. however, for these same reasons, they were likely to be key informants and provide important insights into this area. survey in the absence of a national registry of allied health practitioners in ireland, the best available source for one discipline (counsellors) was used as a sampling frame. this online directory was free, national, and up-to-date, but did not have complete coverage. counsellors were chosen because of their important frontline role and the lack of knowledge around their needs. however, the availability of a comprehensive contact listing was also a factor. one may surmise that the harder it is to locate other practitioners, the more they need to be located. further research is required to carry out needs assessments with these hard-to-reach groups. engaging directly with health and community services should yield better access to practitioners for future studies. a partnership approach may also improve response rates, and identify the particular needs of client subgroups (such as those without internet access or third-level qualifications). although care was taken to provide open options, social desirability bias (respondents answering in a way they think is expected) may have focused responses towards lisrelated issues. a key lesson was the importance of inviting comments. this option was used by 21 respondents (30%) and provided insights that were not captured elsewhere. including a validated standard measurement instrument in the survey design is time-saving and provides a strong basis for enquiry and comparison. however, it also imposes restrictions in terms of content, language, and length. the barriers scale is quite long and, even though some adaptations were made, the language remained difficult to interpret in places. although creation and testing of a new tool was beyond the scope of this study, a simpler, plain language, facilitator based scale would be preferable for future studies. indepth qualitative methods, such as interview, would provide greater insight into practitioner and research factor elements. conclusion in order to ensure that research resources are used effectively, lis personnel should understand potential barriers to use, and develop services and strategies to address these issues. a mixed-method approach to data collection offers the broad scope required to capture the numerous issues involved in research use in practice. quantitative methods offer large-scale feedback. qualitative assessment methods reveal insight into personal attitudes and concerns. the barriers and facilitators to research use identified in this study could be placed within four interactive and overlapping factors: communication, setting, research, and practitioner. the importance of communication barriers in particular, and the enthusiasm shown for facilitators in this study, suggest that library and information services have a key role to play in making evidence based library and information practice 2011, 6.4 53 research resources more compatible with practitioner need. it is hoped that these findings will assist lis personnel to ascertain and understand the issues that clients face in accessing and making use of their resources. acknowledgements for their support and guidance, i would like to thank my colleagues brian galvin, louise farragher, and mariea nelson in the national documentation centre on drug use. i would also like to thank hugh preston from aberystwyth university for his astute and helpful comments. finally, and most importantly, i would like to express my gratitude to all participants, whose valuable opinions and insight formed the core of this study. it is my hope that these contributions will have an impact on the way library and information services are provided to allied health practitioners in the future. initial results from this paper were presented in the form of a poster at the 6th international evidence based library and information practice conference, salford, uk, june 2011. the poster won the audience nominated “best poster” award. references arney, f. m., bromfield, l. m., lewig, k., & holzer, p. (2009). integrating strategies for delivering evidence informed practice. evidence & policy, 5(2), 179-91. doi:10.1332/174426409x437928 addiction technology transfer center (attc) network technology transfer workgroup. (2011). research to practice in addiction treatment: key terms and a field-driven model of technology transfer. journal of substance abuse treatment, 41(2), 169 178. doi:10.1016/j.jsat.2011.02.006 addiction technology transfer center network (2004). the change book: a blueprint for technology transfer (2nd ed.). in addiction technology transfer center network. retrieved 16 nov. 2011 from http://www.nattc.org/pdf/the_chang e_book_2nd_edition.pdf bandura, a. (1994). self-efficacy. in v. s. ramachaudran (ed.), encyclopedia of human behavior (vol. 4, pp. 71-81). new york: academic press. (reprinted in friedman h. (ed.), (1998). encyclopedia of mental health. san diego: academic press). retrieved 16 nov. 2011 from http://des.emory.edu/mfp/banency.ht ml bartholomew, n. g., joe, g. w., rowan-szal, g. a., & simpson, d. d. (2007). counselor assessments of training and adoption barriers. journal of substance abuse treatment, 33(2), 193-199. doi:10.1016/j.jsat.2007.01.005 bertulis, r., & lord, j. (2005). report of key findings of rcn’s survey of the information needs of nurses, health care assistants, midwives and health visitors. in royal college of nursing. retrieved 16 nov. 2011 from http://www.rcn.org.uk/__data/assets/p df_file/0014/112082/003216.pdf booth, a. (2003). bridging the research practice gap? the role of evidence based librarianship. new review of information and library research, 9(1), 3-23. doi:10.1080/13614550410001687909 boström, a.-m., kajermo, k. n., nordström, g., & wallin, l. (2008). barriers to research utilization and research use among registered nurses working in the care of older people: does the barriers scale discriminate between research users and non-research users on perceptions of barriers? evidence based library and information practice 2011, 6.4 54 implementation science, 2008, 3(24). doi:10.1186/1748-5908-3-24 bywood, p., lunnay, b., & roche, a. (2008a). effective dissemination: a systematic review of implementation strategies for the aod field. in national centre for education and training on addiction. retrieved 16 nov. 2011 from http://nceta.flinders.edu.au/download _file/-/view/378 bywood, p. t., lunnay, b., & roche, a. m. (2008b). effective dissemination: an examination of theories and models of change for research dissemination in the aod field. in national centre for education and training on addiction. retrieved 16 nov. 2011 from http://nceta.flinders.edu.au/download _file/-/view/385 callaghan, l., doherty, a., lea, s. j., & webster, d. (2008). understanding the information and resource needs of uk health and social care placement students. health information and libraries journal, 25(4), 253-260. doi:10.1111/j.1471-1842.2008.00769.x carlson, c. l., & plonczynski, d. j. (2008). has the barriers scale changed nursing practice? an integrative review. journal of advanced nursing, 63(4), 322 333. doi:10.1111/j.1365 2648.2008.04705.x closs, s. j., & bryar, r. m. (2001). the barriers scale: does it 'fit' the current nhs research culture? journal of research in nursing; 6(5), 853-865. doi:10.1177/136140960100600509 cochrane, l. j., olson, c. a., murray, s., dupuis, m., tooman, t., & hayes, s. (2007). gaps between knowing and doing: understanding and assessing the barriers to optimal health gain. journal of continuing education in the health-professions, 27(2), 94-102. doi:10.1002/chp.106 eccles, c. (2009). effective delivery of evidence-informed practice. evidence & policy, 5(2), 193-206. doi:10.1332/174426409x437937 frasure, j. (2008). analysis of instruments measuring nurses’ attitudes towards research utilization: a systematic review. journal of advanced nursing, 61(1), 5-18. doi:10.1111/j.1365 2648.2007.04525.x funk, s. g., champagne, m. t., wiese, r. a., & tornquist, e. m. (1991). barriers: the barriers to research utilization scale. applied nursing research, 4(1), 39–45. retrieved 16 nov. 2011 from http://www.sciencedirect.com/science ?_ob=miamiimageurl&_cid=272378 &_user=1069227&_pii=s089718970580 0527&_check=y&_origin=browse&_zo ne=rslt_list_item&_coverdate=1991 02-28&wchp=dglzvlk zskwz&md5=059f32cdf665ad8ab4a6e 2bc96163583/1-s2.0 s0897189705800527-main.pdf gioia, d. (2007). using an organizational change model to qualitatively understand practitioner adoption of evidence-based practice in community mental health. best practices in mental health, 3(1), 1-15. glacken, m., & chaney, d. (2004). perceived barriers and facilitators to implementing research findings in the irish practice setting. journal of clinical nursing, 13(6), 731-740. doi:10.1111/j.1365-2702.2004.00941.x greenhalgh, t., robert, g., macfarlane, f., bate, p., & kyriakidou, o. (2004). diffusion of innovations in service organizations: systematic review and recommendations. the milbank quarterly, 82(4), 581-629. retrieved 16 nov. 2011 from http://www.milbank.org/quarterly/820 4feat.html evidence based library and information practice 2011, 6.4 55 helfrich, c. d., li y.-f., sharp, n. d., & sales, a. e. (2009). organizational readiness to change assessment (orca): development of an instrument based on the promoting action on research in health services (parihs) framework. implementation science, 4(38). doi:10.1186/1748-5908-4-38 hutchinson, a. m., & johnston, l. (2004). bridging the divide: a survey of nurses’ opinions regarding barriers to, and facilitators of, research utilization in the practice setting. journal of clinical nursing, 13(3), 304-315. doi: 10.1046/j.1365-2702.2003.00865.x hutchinson, a. m., & johnston, l. (2006). beyond the barriers scale: commonly reported barriers to research use. journal of nursing administration, 36(4), 189-199. retrieved 16 nov. 2011 from http://journals.lww.com/jonajournal/a bstract/2006/04000/beyond_the_barr iers_scale__commonly_reported.8.a spx joe, g. w., broome, k. m., simpson, d. d., & rowan-szal, g. a. (2007). counselor perceptions of organizational factors and innovations training experiences. journal of substance abuse treatment, 33(2), 171-182. doi:10.1016/j.jsat.2006.12.027 kitson, a. l., rycroft-malone, j., harvey, g., mccormack, b., seers, k., & titchen, a. (2008). evaluating the successful implementation of evidence into practice using the parihs framework: theoretical and practical challenges. implementation science, 3(1). doi:10.1186/1748-5908-3-1 lewig, k. arney, f., & scott, d. (2006). closing the research-policy and research practice gaps: ideas for child and family services. family matters, 74, 12 19. retrieved 16 nov. 2011 from http://www.aifs.gov.au/institute/pubs/ fm2006/fm74/kl.pdf marshall, j. g. (2007). measuring the value and impact of health library and information services: past reflections, future possibilities. health information and libraries journal, 24(supp. 1), 4-17. doi:10.1111/j.1471-1842.2007.00750.x mccormack, b., mccarthy, g., wright, j., slater, p., & coffey, a. (2009). development and testing of the context assessment index (cai). worldviews on evidence-based nursing, 6(1), 27-35. doi:10.1111/j.1741 6787.2008.00130.x mcmaster, t., & wastell, d. (2005). diffusion – or delusion? challenging an is research tradition. information technology & people; 18(4), 383-404. doi:10.1108/09593840510633851 kajermo, k. n., boströlm, a.-m., thompson, d. s., hutchinson, a. m., estabrooks c. a., & wallin, l. (2010). the barriers scale -the barriers to research utilization scale: a systematic review. implementation science, 5(32). doi:10.1186/1748-5908-5-32 osterling, k. l., & austin, m. j. (2008). the dissemination and utilization of research for promoting evidence based practice, journal of evidence based social work, 5(1/2), 295-319. doi:10.1300/j394v05n01_11 parahoo, k. (2000). barriers to, and facilitators of, research utilization among nurses in northern ireland. journal of advanced nursing, 31(1), 89-98. doi:10.1046/j.1365-2648.2000.01256.x retsas, a. (2000). barriers to using research evidence in nursing practice. journal of advanced nursing, 31(3), 599-606. doi:10.1046/j.1365-2648.2000.01315.x evidence based library and information practice 2011, 6.4 56 rogers, e. m. (1962). diffusion of innovations. illinois: free press of glencoe. rogers, e. m. (2003). diffusion of innovations. (5th ed.). new york: free press. simpson, d. d., & flynn, p. m. (2007). moving innovations into treatment: a stage based approach to program change. journal of substance abuse treatment, 33(2), 111-120. doi: 10.1016/j.jsat.2006.12.023 skinner, n., roche, a. m., freeman, t., & mckinnon, a. (2009). health professionals' attitudes towards aod related work: moving the traditional focus from education and training to organizational culture, drugs: education, prevention and policy, 16(3), 232-249. doi:10.1080/09687630902876338 walter, i., nutley, s., percy-smith, j., mcneish, d., & frost, s. (2004). improving the use of research in social care practice. in social care institute for excellence. retrieved 16 nov. 2011 from http://www.scie.org.uk/publications/k nowledgereviews/kr07.pdf yava, a., tosun, n., çiçek, h., yavan, t., terakye, g., & hatipoğlu, s. (2009). nurses' perceptions of the barriers to and the facilitators of research utilization in turkey. applied nursing research, 22(3), 166-175. doi:10.1016/j.apnr.2007.11.003 / evidence based library and information practice/ / abstract introduction literature review diffusion and research use theory assessment tools facilitators of research use focus groups survey table 1 features of successful implementation strategies and related factors results response rate survey respondent variables characteristics of research evidence characteristics of the practitioner facilitators table 2 mean score for barriers scale items table 4 preferred type of training on understanding and using research table 3 proposed library and information service facilitators discussion communication research qualities practitioner focus groups survey figure 1 preferred format for research-related training figure 2 preferred setting for research-related training conclusion acknowledgements references evidence summary   cataloguers may tend to have learning styles different from other library job responsibilities   a review of: rinehart, a., sharkey, j., & kahl, c. (2015). learning style dimensions and professional characteristics of academic librarians. college & research libraries, 76(4),450-468. doi: 10.5860/crl.76.4.450   reviewed by: eamon c. tewell reference & instruction librarian long island university, brooklyn campus brooklyn, new york, united states of america email: eamon.tewell@liu.edu   received: 23 dec. 2015  accepted: 8 apr. 2016      2016 tewell. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine whether relationships exist between academic librarians’ learning styles and their professional work responsibilities.   design – self-selecting survey.   setting – email listservs.   subjects – 1579 academic librarians.   methods – the authors used the index of learning styles questionnaire, based on the felder-silverman learning styles model consisting of eight dimensions on four scales: active/reflective, sensing/intuitive, visual/verbal, and sequential/global. the multiple choice survey was distributed online to 23 email listservs for academic librarians in 2011, and to 14 additional listservs in 2013 targeting technical services librarians. 1579 responses were received in total, which were analyzed using anova with a tukey-kramer post-hoc mean separation, and descriptively using observed frequencies.   main results – in examining the relationship between positions and learning styles, the study revealed there to be five statistically significant p-values when the data were analyzed. catalogers (n=145) were found to be more reflective learners compared to administrative (n=321) and instruction librarians (n=228) at the p = 0.009 level. administrative, instruction, and “other” librarians were found to be more intuitive learners than catalogers, who are more likely to be sensing learners, at the p = 0.0004 level. digital librarians (n=40) are more likely to be visual learners and catalogers more likely to be sequential learners when compared to several other librarian categories, at the p = 0.020 and p = 0.001 levels respectively.   conclusions – the authors concluded that there were some statistically significant differences between librarians’ learning styles scores according to job responsibilities. catalogers were found to have different learning styles than other types of librarians for three out of four scales. based on these findings, the authors indicate that further research into how librarians’ work responsibilities impact learning styles is justified.   commentary   there are many ways to understand the complex process of learning, and learning styles have received a great deal of attention in the educational realm. many studies have appeared both supporting and criticizing learning styles theories and instruments, including a body of research critical of learning styles that begins largely with curry (1990) and extends to pashler et al. (2008), who conducted a review of the literature and found an inadequate evidence base to justify incorporating learning styles into educational practice. a number of studies have explored learning styles as applied to library instruction. relevant to the goals of the study at hand, two articles have considered librarians’ learning styles, including choi (1989), and squires, hoopes, & gillum (1992).   this investigation includes a number of factors that contribute positively to its validity. the historical development of learning styles is well documented and summarized, and the model used for the study is clearly described for those unfamiliar with learning styles. the authors acknowledge other viewpoints in the learning styles debate, and include a discussion of the validity of the index for learning styles used in the study. additionally, the reasons for selecting the instrument and the ways it is scored are useful information in weighing the evidence presented.   some of the research’s limitations are worth noting. the survey response rate was not calculated, and as the authors observe, the respondents were non-random and self-selected. the possibility of receiving duplicate responses appears to be significant, as the survey was sent to a large number of listservs two years apart. it was not mentioned whether the research received irb approval. as noted in the study, several of the instrument’s questions were more appropriate for students in a classroom environment than for professionals. the authors describe several practice implications regarding learning styles, including matching one’s learning style to professional development opportunities and the possibility of managers in library settings being mindful of employees’ preferences, which would require learning styles activities to be conducted in one’s workplace.   the study contributes evidence that some academic librarian position types may potentially be more inclined toward certain learning styles. because of the study’s correlational nature, it cannot be ascertained whether people with a certain learning style are more likely to choose cataloging, or whether the work of cataloging could cultivate a particular learning style. given that one’s work duties are not necessarily equivalent to oneself in regards to learning preferences, additional avenues could be explored. for example, librarians might work across different departments, feel that their current position is not a good fit for their abilities, or delineate between personal and professional learning. as such, the qualitative investigation of learning among librarians might be pursued by considering what compels librarians to learn, how professional development interests change over the course of one’s career, or if there are links to one’s teaching style, including what librarians themselves identify as key factors influencing their learning.   references   choi, j. n. (1989). learning styles of academic librarians. college & research libraries, 50(6), 691-99. http://eric.ed.gov/?id=ej405682   curry, l. (1990). a critique of the research on learning styles. educational leadership, 48(2), 50-56. http://eric.ed.gov/?id=ej416434   pashler, h., mcdaniel, m., rohrer, d., & bjork, r. (2008). learning styles: concepts and evidence. psychological science in the public interest, 9(3), 105-119. doi:10.1111/j.1539-6053.2009.01038.x    squires, d. a., hoopes, h. k., & gillum, g. p. (1992). librarians: a thinking and learning styles portrait. library administration and management, 6(4), 173-79. http://eric.ed.gov/?id=ej451933 evidence based library and information practice 2012, 7.3 86 evidence based library and information practice evidence summary library and information science doctoral research appears to be showing less and less interest in library topics, and concern among practitioners may be justified a review of: finlay, c. s., sugimoto, c. r., li, d., & russell, t. g. (2012). lis dissertation titles and abstracts (1930– 2009): where have all the librar* gone? the library quarterly, 82(1), 29-46. doi:10.1086/662945 reviewed by: r. laval hunsucker information and collection specialist emeritus university libraries, universiteit van amsterdam amsterdam, the netherlands email: amoinsde@yahoo.com received: 2 june 2012 accepted: 27 july 2012 2012 hunsucker. this is an open access article distributed under the terms of the creative commons‐ attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to determine whether library and information science (lis) doctoral research at north american institutions has, over the last eighty or so years, displayed a clear trend toward addressing topics other than those associated with librarianship and traditional library functions; and whether one can discern, in this regard, any significant differences among those institutions. design – conceptual content analysis of dissertation titles and abstracts. setting – north american universities with american library association accredited lis programs in the period 1930 through 2009. subjects – the titles and, to the extent available, the abstracts of 3,230 lis doctoral dissertations completed at these institutions during this period. methods – having opted for a directed, singlecategory type content analysis, the researchers began by pre-establishing a group of terms which they assumed could “represent the core curriculum of the master’s in library science”: terms which they surmised would therefore be able to function, where they appeared in “the records of doctoral output”, as good indicators that that output itself can rightly be judged to have had “an explicit focus on libraries/librarianship” (pp. 36, 44). the terms selected were: “librar*”, “catalog*”, “circulat*”, “collection develop*”, “collection manag*”, “school media”, and “reference” (where “*” mailto:amoinsde@yahoo.com evidence based library and information practice 2012, 7.3 87 indicates truncation, and that any term beginning with the respective letter string was acceptable). the researchers then simply tallied for each of the 3,230 dissertations under investigation how many times one or more of the pre-chosen terms occurred in its title and in its abstract, not recording which term or terms that occurred. (they do not make entirely clear to what extent data collection was computerized.) they subsequently analyzed the data longitudinally and by institution, with only one, nominal and dichotomous, variable for the title as well as for the abstract: whether or not any of the pre-chosen terms occurred at least once. multiple occurrences, whether of the same term or of varying terms, played no role. their analysis for the entire period of 1930 through 2009 was based on title data only, and did not take doctorate-granting institution into account. the separate analysis (n=2,305) for the period 1980 through 2009 excluded the thirty cases in which one or more of the terms occurred in the title but none of them occurred in the abstract. main results – one occurrence of any of the specified terms in the title was, for the overall period of 1930-2009, enough for any given dissertation to be qualified as having an explicit focus on libraries/librarianship. the percentage of such dissertations remained fairly stable from the 1930s through the 1980s, at between 56% and 62%, with the exception of an unexplained dip for the 1950s to 44.1%. then, for the 1990s, the researchers discovered a fall-off from 57.9% to 36.0%, and in the following decade a further decrease, down to a level of 21.5%. during the separately-analyzed period 19802009, the percentage of dissertations with at least one of the specified terms in the title as well as in the abstract diminished steadily from well over half (58.4%) for 1980-1984 to less than 1 in 5 (19.8%) for 2005-2009. a chisquare test revealed that the relationship between year of dissertation and term occurrence is statistically significant. by far the greatest decrease, of 15 percentage points, was that between the first half and the second half of the 1990s. interestingly, during the whole thirty-year period, the percentage where a term appeared not in the title but only in the abstract remained fairly constant, at around 20%, give or take about 2.5 percentage points. yet when one looks at how many of the dissertations displayed none of the terms in the title and none in the abstract, one sees a continuous increase starting at 20.7% for 19801984 all the way up to 61.0% for 2005-2009, with the sharpest climb, of more than 17 percentage points, occurring around the mid1990s. the distinction between the year 1980 and the year 2009 is even greater: from just over 1 in 7 (14.7%) to more than 3 out of 5 (62.2%). the analysis by institution revealed a statistically significant relationship for the period 1980-2009 between institution at which the dissertation was written and the occurrence of any of the terms at least once in both title and abstract. certain institutions (most notably suny-albany, syracuse, missouri, hawaii, montréal, and long island) showed a much higher than average overall level of no occurrence, and some (michigan in particular, but also, for example, florida state and the university of north carolina) displayed a remarkably consistent decline in occurrence. conclusion – the researchers conclude that their study, insofar as north america is concerned, “has provided empirical evidence for . . . the lessening focus in lis dissertations on topics commonly associated with librarianship” and that it “supports the assertion that this focus varies significantly between schools—with some schools demonstrating a more explicit connection to library-related topics than other schools” (p. 43). they are unsure how best to interpret these findings or how they could be applied, but they do offer certain suggestions for possible interpretations and pose a few questions regarding what those interpretations might imply (p. 44). one could, they suggest, argue that the terms employed in the study “are themselves antiquated, and dissertations evidence based library and information practice 2012, 7.3 88 are charting new territory, pushing the boundaries of both research and practice.” another possibility is that “while the dissertations may not be immediately applied work, the work could be utilized for application.” on the other hand, it may simply be the case that the selected terms indeed remain trustworthy indicators, and that doctoral candidates “are no longer studying topics that are relevant to the practical field” of librarianship. one could perhaps even justifiably assert that lis is in effect no longer a single unified discipline but, rather, has split into a library field and an information field, whereby the latter has been steadily gaining the upper hand in lis programs, albeit less so at some institutions than at others. in pondering the above alternatives for interpretation of this study’s results, the library practitioner will probably also be inclined to reflect, the authors suggest, on the prospects for adequate academic research support of actual library practice, while keeping in mind, furthermore, that the formal education of future practitioners will largely remain in the hands of those trained as lis doctoral students. to what extent will these educators feel an affinity with, and possess an understanding of, the world of practical librarianship? commentary this article is the most recent contribution to a literature, extending back roughly half a century but of very modest extent, dealing with the topical orientation of north american lis dissertations. the study’s importance and uniqueness lie in the attempt to reveal significant trend data over an exceptionally long period, indeed starting with the year in which the first lis doctoral degree was granted, to less than three years ago. the data offered are nevertheless less ample and expressive, and their analysis less refined, than in the cases of those previous, diachronically more restricted studies. there is to my knowledge no critical appraisal tool available that has been expressly designed for the evaluation of content analysis studies, in lis or elsewhere. however, generic checklists, and even many of those created for other fields or for other study types, can certainly be of use. one major stipulation in many such checklists is that a research study publication should begin by unambiguously indicating what the reason for the study was: why was it carried out, and why is that important? (e.g., connaway & powell, 2010, pp. 310, 314-317). ideally, this research purpose, along with a clear “analytic story” and an evident “underlying logic”, should inform the entire research presentation, through to the discussion and the conclusions (treloar, champness, simpson, & higginbotham, 2000), and it should furthermore be sufficiently placed in the context of related previous scholarship (connaway & powell, 2010, pp. 314-317). the researchers should be sure to “distinguish adequately between the problem and the purpose of the study”, that is, between what was studied, and why (p. 316). the article under review unfortunately fails to meet these basic requirements. in their introduction, and throughout the greater part of the text, the authors repeatedly suggest that the aim of the research was to investigate the evolution and boundaries of lis’s disciplinary identity, and to clarify its present status. however, the specific conclusions that they ultimately draw imply, very strongly, that their principal goal was in fact to establish that there is a growing disconnect between doctoral research and the actual concerns of library practice. in their final paragraph, nonetheless, they play down any question of a gap between research and practice and again emphasize the issue of how “to more precisely map the evolution of our discipline” (p. 44). thus, they leave the reader in complete uncertainty as to the actual purpose and import of their study. the reliability of the study design is open to little question. its validity, however, is quite another matter. how appropriate and trustworthy should we consider a measurement approach that depends on, respectively, the minimal and isolated occurrence, or the absence, of an arbitrarily limited number of words or two-word evidence based library and information practice 2012, 7.3 89 combinations in document surrogates, without regard to syntax or context or to frequency of occurrence, for peremptorily establishing whether the documents themselves do or do not qualify as dealing significantly with a topic of interest to a given field of professional activity? (janis, 1949; krippendorff, 2004, pp. 22-25, 213) and that while at least some of those terms in themselves can have (widely) varying meanings and usages in the language, by no means limited to the domain of librarianship? face validity (krippendorff, 2004, pp. 313-314) is, therefore, an obvious concern here. what lies behind that is a fundamental problem of empirical validity: in this case, more specifically, of semantic validity (pp. 319, 323-324), one that renders the findings considerably less illuminating and useful than they otherwise could have been. remarkable is that the researchers apparently neglected to subject their method to any kind of validity test. krippendorff (1980) already long ago suggested ways in which content analysts can carry out semantic validation of their research designs even without expending the extra time and costs required for pilots or for preor post-testing. and had the researchers indeed chosen not a conceptual, but for example a relational or a contrastive or a contextual variety of directed content analysis, their results would surely have proven considerably more enlightening. the inclusion of additional terms could have broadened their results to encompass further important aspects of contemporary library practice, such as instruction, liaison, management of electronic resources, scholarly communication issues, special collections, etc. the original researchers do admit (p. 43) that they were using a “blunt instrument”, but apparently were content with achieving, rather easily and quickly in this way, at least a modicum of empirical evidence for what they saw as the broadly shared anecdotal impression that north american lis education at its highest level has in the last decade and a half been more and more abandoning its interest in traditional library functions and, possibly, in “librarianship” as a professional or occupational identity. whether this abandonment should be seen as a bad thing, or indeed as a good thing, for lis practitioners will depend entirely on how one feels about, and perhaps on how dependent one perceives oneself to be on, that occupational identity and that traditional library world. in any case, what would now be particularly useful, and what this study unfortunately did not even attempt to provide, is some good trend data not on what lis doctoral research is focusing less and less on, but on what things it is in fact focusing more and more on. data that preferably go this time beyond just north america. only then shall we be in a position to decide whether there exists a significant gap between the kinds of research that contemporary lis practice could benefit from, and the kinds of research that doctoral students, and therefore our future lis educators, are actually carrying out. references connaway, l. s., & powell, r. r. (2010). basic research methods for librarians (5th ed.). santa barbara, ca: libraries unlimited. janis, i. l. (1949). the problem of validating content analysis. in h. d. lasswell, n. leites, and associates, language of politics: studies in quantitative semantics (pp. 55-82). new york: stewart. krippendorff, k. (1980). validity in content analysis. in e. mochmann (ed.), computerstrategien für die kommunikationsanalyse (pp. 69-112). frankfurt/main; new york: campus. krippendorff, k. (2004). content analysis: an introduction to its methodology (2nd ed.). thousand oaks, ca: sage. treloar, c., champness, s., simpson, p. l., & higginbotham, n. (2000). critical appraisal checklist for qualitative research studies. indian journal of pediatrics, 67(5), 347-351. doi: 10.1007/bf02820685 article   “ask, acquire, appraise”: a study of lis practitioners participating in an eblip continuing education course   anthea sutton information specialist information resources group health economics and decision science school of health and related research the university of sheffield sheffield, united kingdom email: a.sutton@sheffield.ac.uk   andrew booth reader in evidence based practice and director of information health economics and decision science school of health and related research the university of sheffield sheffield, united kingdom email: a.booth@sheffield.ac.uk   pippa evans information specialist information resources group health economics and decision science school of health and related research the university of sheffield sheffield, united kingdom email: p.evans@sheffield.ac.uk   received: 13 july 2012    accepted: 03 apr. 2013      2013 sutton, booth, and evans. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – the project sought to examine the aspects of the question answering process in an evidence based library and information practice (eblip) context by presenting the questions asked, articles selected, and checklists used by an opportunistic sample of australian and new zealand library and information professionals from multiple library and information sectors participating in the “evidence based library and information practice: delivering services that shine” (eblip-gloss) folioz e-learning course.   methods – the researchers analyzed the “ask,” “acquire,” and “appraise” tasks completed by twenty-nine library and information professionals working in australia or new zealand. questions were categorized by eblip domain, articles were examined to identify any comparisons, and checklists were collated by frequency.   results – questions fell within each of the six eblip domains, with management being the most common. timeliness, relevance, and accessibility were stronger determinants of article selection than rigour or study design. relevance, domain, and applicability were the key determinants in selecting a checklist.   conclusion – this small-scale study exemplifies the eblip process for a self-selecting group of library and information professionals working in australia and new zealand. it provides a snapshot of the types of questions that library and information practitioners ask, and the types of articles and checklists found to be useful. participants demonstrated a preference for literature and checklists originating from within the library and information science (lis) field, reinforcing the imperative for lis professionals to contribute to eblip research.   introduction   questions asked by practitioners, whether they be health professionals, teachers, social work practitioners, or librarians, contribute greatly to our understanding of evidence based practice (booth, 2006). they can provide a valuable insight into the nature of uncertainties encountered in day-to-day practice and the questions that such uncertainties provoke (chalmers, 2004). they also permit researchers to calculate approximations for the frequency with which such questions arise (ely et al., 1999). taken further, studies of questioning behaviour frequently result in the production of classifications or taxonomies that allow examination of the characteristics of particular question types (ely et al., 1999, 2002). at a practical level, real-life questions provide a basis for evaluating the coverage and fitness-for-purpose of information resources (ely et al., 1999). they also allow identification of barriers encountered when attempting to address an outstanding question (ely et al., 2002). finally, where such questions are pursued to eventual resolution, they can yield a pragmatic glimpse of the relative value of the evidence base, and of specific study types (glasziou, vandenbroucke, & chalmers, 2004), in answering real-life concerns of practitioners (ely, osheroff, chambliss, ebell, & rosenbaum, 2005). outside of this practical context, practitioner questions that are either unanswered or inadequately and incompletely answered provide a rich vein for the generation of future research priorities.   this study sought to examine practitioner questions in lis, in order to present a snapshot of the current concerns for library and information staff working in australia and new zealand at the given time.   literature review   the field of medicine has been very active in examining the characteristics of questions generated by healthcare professionals, particularly in the course of delivering clinical care. the research literature makes the frequent assumption that such studies successfully reflect the information needs of those being studied (smith, 1996). however, several commentators observe a significant attrition in the numbers and types of questions in the stages that precede articulation, and then pursuit of an information need generated from a patient encounter (gorman & helfand, 1995; booth, 2005; glasziou & haynes, 2005; wimpenny, johnson, walter, & wilkinson, 2008). within an educational context, where practitioners identify, prioritise, and select from a range of questions that have occurred during their recent practice, the likelihood that the resultant questions are representative is further compromised (hersh et al., 2002). nevertheless, such questions have particular value in modelling the technical aspects of the evidence based practice process (grefsheim & rankin, 2007; grefsheim,  rankin, & whitmore, 2007), particularly in focusing the question, identifying the source of a potential answer, matching that source article to a suitable appraisal checklist, and then conducting a structured appraisal of the retrieved study (gray, 2010).   evidence based library and information practice (eblip) similarly recognizes the importance of a well-formulated practitioner-led question as the stimulus for subsequent inquiry (eldredge, 2000; kloda, 2008). it has witnessed several noteworthy attempts to capture the questions asked by practising librarians and information specialists. typically such studies focus only on the point of question generation and have not pursued the likelihood of finding a satisfactory answer. for example, eldredge (2001) conducted an opportunistic international survey of the “most relevant and answerable research questions” facing the health library profession. however, close examination of survey results reveals an emphasis on their relevance with no formal criteria used to identify the degree to which they were answerable. lewis and cotter (2007) revisited questions identified by this 2001 survey (eldredge) and compared them with those asked at an educational eblip workshop 5 years later. they identified a gap between those questions being asked by library and information practitioners and those being addressed by researchers. a 2011 study (eldredge, ascher, holmes, & harris, 2012) identified the top-ranked research questions specifically for the medical library profession building on a previous study (eldredge, harris, & ascher, 2009) but upgrading the methodology to improve answerability. a previous study has taken the “demand-supply chain” for eblip question-answering further by asking “what studies do practitioners actually find useful?” (booth, 2004) an alternative approach is to work from the opposite (i.e. the supply) end and to examine the characteristics of the literature in connection with its question-answering potential. crumley and koufogiannakis (2002) created a taxonomy of six domains (i.e. broad subject areas) within which library and information practitioner questions might be framed. they subsequently revised this taxonomy in the light of the characteristics identified from a significant sample of the library literature (koufogiannakis, slater, & crumley, 2004).   aims and objectives   the objective of this study is to extend previous research by examining five interlinked aspects of the question answering process, namely:   1)       the questions posed by library and information practitioners (booth, 2006; kloda, 2008); 2)       the assignment of questions to domains (wilson, 2009a); 3)       the articles retrieved to attempt to answer such questions (wilson, 2009b); 4)       the study designs of such articles (lorenzetti, 2007; wilson, 2009c); and,  5)       the selection of appraisal tools used to scrutinise such studies (booth, 2007; wilson, 2010).   this study aims to provide valuable insights into the practical realities of attempting to pursue evidence based practice in a library setting. it will present the questions asked by a small sample of library and information professionals working in australia or new zealand and undertaking an online continuing professional development (cpd) course in 2010. in addition, the articles selected to answer these questions and the checklists used to appraise the articles will be presented. the data will be collated in order to reflect on the types of questions asked by library and information practitioners, the articles selected, and the most commonly used checklists.   methods   the study draws upon responses from an opportunistic sample of australian and new zealand library and information professionals from multiple sectors involved in an eblip educational opportunity. it therefore does not claim to be representative of the wider lis population. an opportunistic sample is not random; respondents are selected based on convenience (mcgraw-hill higher education, 2001).   scharr information resources group (school of health and related research, the university of sheffield) designs and delivers a program of continuing professional development online courses for library and information professionals in australia and new zealand in association with the australian library and information association (alia). in 2010, the “evidence based library and information practice: delivering services that shine” (eblip-gloss) course was delivered to twenty-nine library and information professionals working in those countries. the course was designed and delivered by the authors of this study.   of the twenty-nine library and information practitioners included in this study, fifteen were drawn from the academic sector, six worked within health services, four were public librarians, two were employed in government, and one identified him or herself as “technical.” one practitioner did not specify the library and information sector he worked in.   the eblip-gloss course consisted of readings, podcasts, tasks, and exercises relating to the evidence-based library and information practice process. the course was structured around the five eblip elements: “ask,” “acquire,” “appraise,” “apply,” and “assess.” this article focuses on the first three elements. the course tasks relating to these were as follows:   1)  ask – course participants identified a “burning question” relating to their own library and information service. participants were asked to focus their question using the “spice” framework (booth, 2006), that is, to identify the setting-perspective-interest (phenomenon of)-comparison-evaluation for their specific topic in order to facilitate identification of relevant evidence. participants were also required to locate their question within a specific eblip domain (management, information access and retrieval, professional, collections, reference enquiries, education) (koufogiannakis et al., 2004). 2)  acquire – course participants were asked to identify an appropriate article to help answer their burning question. 3)  appraise – course participants were asked to identify a suitable critical appraisal checklist in order to appraise their chosen article.    course participants collated their work into a portfolio for submission at the end of the course, and the information for the ask, acquire, and appraise tasks was later extracted into tables by the researchers. the questions and articles selected by participants were categorized by the domain allocated by the lis practitioners. one of the researchers (pe) did this by grouping the questions by domain and collating them into table 1 (see appendix). the researchers then analyzed the questions and articles to see if any recurrent themes were present.   course participants were asked to indicate whether they “agreed to the folioz team using material from my portfolio for training, sharing good practice, course evaluation or publicity.” participants who had previously withheld their consent for generic use of their portfolio received a follow-up email outlining the purposes of this article and seeking consent for their anonymous contributions to be included in the analysis. all participants subsequently gave their consent.    results   the questions were divided into six domains (management, information access and retrieval, professional, collections, reference enquiries, and education) and categorized by sector to identify trends. there were no clear patterns linking sector and domain. however, all sectors reflected the wider recurring themes of innovation and efficiency.   the “burning questions”   the questions harvested by this eblip course reflect the significant variety of issues being faced by library and information practitioners. recurring themes include efficiency (especially of staff time), the development of innovative services, and improving existing services. participants were asked to categorize their question into one of the six eblip domains (crumley & koufogiannakis, 2002; koufogiannakis et al., 2004): management, information access and retrieval, professional, collections, reference and enquiries, and education. examples from each are included below in table 1. the full list of burning questions categorised by domain and sector can be found in the appendix. the questions are presented exactly as the participants posed them and with the domain(s) the participants assigned to their own question. some participants assigned more than one domain to their question if they felt it crossed domains. one question was not categorized into a specific eblip domain, but the participant defined it as “marketing and library promotion.” as the management domain includes marketing, the question has been included as a management question for subsequent analysis.   “management” was the domain that contained the largest number of questions. questions concerned staffing, customer services, and the use of library spaces. another issue emerging from the management domain was difficulties in trying to engage users with online resources through marketing. engaging users with online resources also figured prominently in the information access and retrieval domain, alongside more traditional questions relating to classification schemes.   two questions fell within the professional issues domain. the first concerned the use of dedicated software for capturing data on librarian workflows. the second was specific to participation in the folioz eblip-gloss course.   the collections domain encapsulated some wider questions facing libraries: outsourcing expertise, e-books and budgets. two question whether employing these services is wise, while another is posed at a more operational level.   five questions fell within the reference and enquiries domain; again, the quest for improved efficiency was an underpinning theme.   the questions that fell within the education domain could be divided into three categories: online resources, information literacy, and referencing and plagiarism. whilst effectiveness is an important issue, being able to prove this effectiveness is key in justifying support for library services.   table 1 examples of burning questions categorized by domain domain number of questions sample question management 10 “what evidence is there that pre-tertiary student conduct and learning improves with the provision of social networking spaces and areas to assist the use of personal digital equipment within the library?” education 6 “are the students transferring the skills learnt in that [library training] unit to the other units they are in enrolled in 1) in the same semester? 2) are the skills used in the second and following semesters?” reference and enquiries 5 “is the information desk at [the] library meeting its objectives in providing a service to students that helps them use the library and its resources more effectively when looking for information for their assignments?”     information access and retrieval 4 “what are alternative options in making use of technologies and/or web-based platforms to use for presenting, organising and facilitating access to technical data, manuals and other documentation for users?” collections 4 “what evidence is there that the breakdown of the collection budget is allocated to various collections appropriately?”   professional issues 2 “what terms/terminology other than ‘evidence-based’ can the librarians look for when researching and looking at articles to determine whether they are evidence based?”       articles chosen to answer the questions   a further point of interest was the nature of the articles chosen to address the original burning questions. participants were given a briefing to read that provided guidance on acquiring evidence and listed suggested resources (folioz, 2010). participants then searched for and selected their own articles, although advice was given from the course facilitators if the course participants emailed with queries.    this study confirmed the characteristics of the library and information literature in that the majority of studies (n=19) used by course participants were either surveys or case studies. qualitative methods were generally well-represented although the abstracts for such studies generally revealed pragmatic use of a qualitative methodology rather than existence of an underpinning paradigm. two literature reviews and two conceptual/theoretical papers revealed that background questions can be advanced by more overarching discursive works. only a small number of studies were comparative, either using a case-control retrospective design (2 studies) or internal comparison (before-after, 1 study; interrupted time series, 1 study). there were no randomized controlled trials or systematic reviews present in the articles used.   generally, articles were of recent origin, with the year of the course (2010) being well represented in the chosen selection. the oldest article was dated 1996. a large majority of the articles were subsequently found to be available as free full-text via either online journals or article repositories. australasian journals and authors were also well represented. only one study figured in more than one response (korah & cassidy, 2010).   checklists chosen for critical appraisal of the articles   course participants were given a list of suggested checklists (see table 2) in order to appraise their chosen “burning question” article. they were also given a worked example using the reliant checklist (koufogiannakis, booth, & brettle, 2006) and asked to read a book chapter on appraising the evidence (booth & brice, 2004). alternatively, participants could identify a checklist for themselves if they felt none of the checklists listed were appropriate to appraise their chosen article. most participants (n=28) chose one of the suggested checklists, with only one participant identifying his or her own checklist. if participants contacted the course facilitators with queries, advice on selecting a checklist was given.     table 2 critical appraisal checklists chosen checklist reference number of course participants cristal checklist for appraising a user study cristal (2010b) 10 reliant koufogiannakis et al., (2006) 5 critical appraisal checklist for a questionnaire study boynton & greenhalgh (2004) 4 ‘a critical appraisal tool for library and information research’ glynn, l. (2006) 3 casp appraisal tool for systematic reviews critical appraisal skills programme (2006) 3 cristal checklist for appraising an information needs analysis cristal (2010a) 2 critical appraisal checklist for an article on an educational intervention university of glasgow dept. of general practice (n.d.) 1 casp qualitative appraisal tool critical appraisal skills programme (2006) 0   the most commonly used checklists were those produced for the cristal initiative, with 12 participants choosing either the information needs analysis/audit (cristal, 2010a) or user study checklist (2010b). these checklists were employed for articles derived from multiple domains – marketing, information access and retrieval, collections management, and reference. many of these questions focused on user aspects and services or on some form of published standards (e.g., classification).   the reliant checklist (koufogiannakis et al., 2006) was used by 5 participants, mostly in the domain of education (n=4), with one in the domain of management. the bmj questionnaire checklist (boynton & greenhalgh, 2004) was used by 4 participants, mostly in the domain of management (n=3), with one in collections management. the ebl (glynn, 2006) and casp (systematic reviews) (critical appraisal skills programme, 2006) checklists were each used by 3 participants. one participant used the checklist on educational interventions and one used an alternative published checklist (markville secondary school, 2009) due to their article being a literature review, which was not specifically one of the studies covered by the suggested checklists.   in most cases, the choice of checklist was heavily influenced by its relevance and appropriateness to the article chosen to address the “burning question.” it was further affected by the domain within which the question appeared – e.g., the educational intervention checklist (university of glasgow department of general practice, n.d.) matched questions appearing under the education domain. the format also influenced some participants, with reference to checklists being clear and logical. the fact that checklists such as those from casp use screening questions to check whether it is worth continuing with the appraisal was also noted as a positive. a further positive feature was inclusion within a checklist of a section on applying the article to one’s own setting (applicability), a key facet in evidence based practice.   choice of checklist was also influenced by the checklist authors’ affiliation, for example, if they were affiliated with an organization in the same sector as the participant it was more likely to be used. one participant also noted that the reliant checklist was specifically designed for use by library and information professionals, considering this a positive in terms of its appropriateness.   most participants found that the checklist that they had selected met their requirements, but some made adjustments where necessary, using the published checklist as a guideline. it was noted that sometimes it was difficult to assess the appropriateness of a checklist, so some participants trialled several checklists on their chosen article before making a final decision.   discussion   course participants generated a wide range of questions, demonstrating the numerous areas facing the modern library and information professional. the questions, although grounded in the context of the service where the library and information practitioner works, cross contexts and provide a snapshot of the current issues facing the lis profession. this contrasts with a volume of library trends journal which asked researchers to identify questions that could and should be answered (lynch, 2007).   questions fell within each of the six eblip domains, with management being the most common. similarly, research on questions asked by librarians found that the most commonly asked were in the domain of management in both 2001 and 2006 (lewis & cotter, 2007). this could be due to the testing times we find ourselves in: library and information managers have to prove the worth of their services more than ever, and come up with the evidence to support existing services and to make bids for new developments. eldredge et al. (2012) also found that the fifteen top-ranked research questions for the medical library profession reflected “a high level of anxiety with respect to the financial future of health sciences libraries.”   the variety of articles selected reflects the variety of the questions asked. there appears to be a preference for direct applicability (booth, 2004): for example, most of the literature fell within the library and information science domain. the choice of study design depended both on the type of question (for example, background questions tended to be answered by overview type articles such as literature reviews), and on the availability of types of study design within a given field. for example, no rcts were selected, but this is more telling about the types of study designs published rather than the selection choices of the course participants. eldredge et al. (2012) recommend the types of research designs which could answer the top ranked research questions in the medical libraries field, the majority being cohort study. the medical library association research agenda committee is co-ordinating systematic review teams to identify the current evidence on these questions as a first stage (eldredge, ascher, holmes, & harris, 2013).   participants demonstrated a preference for recent literature, again affirming applicability, e.g., the article can be applied to the current context. similarly, local articles within australasia were well-represented, again demonstrating a preference for applicability. convenience also figured prominently: articles tended to be those where the course participants could access the full-text immediately, at no extra cost. this is to be understood in terms of the short timescale available in the constraints of the course.   no single checklist was appropriate to the needs of all the library and information practitioners participating in the course. the wide range of questions was reflected in a correspondingly wide range of checklists used. a key consideration in selecting a checklist was relevance to the article, which participants identified as their main reason for selecting the checklist. the lis professionals studied proved themselves to be resourceful, flexible, and adaptable, in editing existing checklists to suit their needs, or sourcing their own checklists in addition to those suggested by the course team. this is an important skill to have when appraising the evidence.   limitations of the study   as stated, the study was a “small-scale” study of a specific group of lis practitioners, and generalizations are not possible. ideally, such a study would capture a representative sample of questions asked by library and information practitioners. participants in the folioz course were self-selecting, indicating at least that they possessed a motivation for trialling the steps of evidence based practice and may well have already had an issue in mind to work with during the course. the political, economic, social and technological context of australasia at the time of the study will likely have shaped pervasive themes encountered within and across sectors. nevertheless, an increasingly global economy, common flows of professional knowledge across boundaries, and, above all, a shared evidence base are reasons to expect the presence of common concerns engaging the profession more widely.   furthermore, questions were identified, or even generated, in response to a specific educational task instruction. we cannot ascertain whether an individual participant selected their question because of its priority for that participant or because of its viability for subsequent stages of the process. such uncertainty is shared with other educational assignments where integrity in pursuing genuine questions may be challenged by a desire to perform well in the assessment. it would be particularly interesting to examine the frequency with which participants changed their original question by the time they had to complete their subsequent tasks – although in fairness, some participants did qualitatively describe this in their portfolio.   selection of articles may also be determined by factors other than their genuine suitability for addressing the original “burning question.” for example, under time pressures a course participant may settle for an article that superficially meets the question topic but which may not represent the best available article or the best answer to the question. indeed, prescribed evidence based practice procedures such as working down a hierarchy of evidence until one finds the highest study type relating to the question may similarly be compromised by time pressures. a possibility even exists, although more likely in a credit-bearing rather than vocational course, of selecting a suitable article for appraisal and then working backwards to generate an appropriate question.   ideally, a future study would explore the progress of eblip questions, not simply as far as a potential answer, as in this case, but in reaching an actual resolution of the originating problem. nevertheless, this study advances understanding of the links between question generation and the subsequent stages of the eblip process: for example, how an initial question is translated into an information need and the extent to which that information need is subsequently met by a retrieved article. we can similarly identify the extent to which the type and study design of the identified article is accommodated by the available checklists, whether generic or library-specific. the study also alerts us to some of the difficulties encountered in identifying or obtaining a relevant article and in selecting and locating an appropriate checklist. in reality, however, as the eblip process was undertaken in an educational situation, it is likely to be extensively “sanitized.” although course participants are encouraged to be reflective, they will not necessarily describe “false starts,” in formulating and then changing their question, or “false hits,” in identifying an article that subsequently fails to address their question. it would be interesting for future research to investigate the same process in a practical rather than theoretical context in order to draw comparisons and contrasts.   conclusions   this study demonstrates the  ask, acquire, and appraise components of the eblip process for a self-selecting group of library and information professionals working in australia and new zealand, by presenting the five inter-linked aspects of the question answering process. it provides a snapshot of the variety of challenges the lis profession faces both day-to-day and from a future planning perspective. it has built on previous research, by analyzing eblip questions by domain and identifying that the area of management in lis is still a key concern. there is no link between lis sector and eblip domain; themes are cross-cutting across the domains. the recurrent themes of the eblip questions are efficiency, innovation, and service improvement. within the theme of efficiency, proving this to justify support for library services is important. in the analysis of articles and checklists, the course participants demonstrated a preference for literature and checklists originating from within the lis field, as opposed to seeking transferable research from other disciplines. the articles chosen were mostly surveys or case studies, but this is to be expected as it is characteristic of the lis literature.    the study found that convenience and applicability are key issues for lis professionals wanting to employ eblip. in terms of article selection, lis domain, locality, and currency were key determinants. affiliation of the article author was also a deciding factor. positive elements of checklists were noted as domain (again demonstrating a need for direct applicability), clear and logical structure, the existence of screening questions, and the presence of a section on applicability. there is an ongoing need for the library profession to explore these areas, particularly in conducting research with rigorous study design where appropriate. in the short term, there could also be some mileage in demonstrating how generic checklists and research in other areas could be adapted to the eblip process.     references   booth, a. (2001). turning research priorities into answerable questions. health information and libraries journal, 18(2), 130-132.   booth, a. (2004). what research studies do practitioners actually find useful? health information & libraries journal, 21(3), 197-200. doi: 10.1111/j.1471-1842.2004.00527.x   booth, a. & brice, a. (2004). appraising the evidence. in booth, a. & brice, a. (eds.) evidence-based practice for information professionals: a handbook (pp.104-118). london: facet publishing.   booth, a. (2005). the body in questions. health information & libraries journal, 22(2), 150-155. doi: 10.1111/j.1471-1842.2005.00566.x   booth, a. (2006). clear and present questions: formulating questions for evidence based practice. library hi-tech, 24(3), 355-368.   booth, a. (2007). who will appraise the appraisers? the paper, the instrument and the user. health information & libraries journal, 24(1), 72-76. doi: 10.1111/j.14711842.2007.00703.x   boynton, p. m. & greenhalgh, t. (2004). hands-on guide to questionnaire research: selecting, designing, and developing your questionnaire. bmj, 328(7451), 1312.  doi: 10.1136/bmj.328.7451.1312   chalmers, i. (2004). well informed uncertainties about the effects of treatments. bmj, 328(7438), 475–476. doi: 10.1136/bmj.328.7438.475   cristal (2010a). appraising an information needs analysis: 12 questions to help you make sense of an information needs analysis/information audit. netting the evidence. retrieved 19 apr., 2013, from http://nettingtheevidence.pbwiki.com/f/needs.doc   cristal (2010b). appraising a user study: key questions to help you make sense of a user study. netting the evidence. retrieved 19 apr., 2013, from http://nettingtheevidence.pbwiki.com/f/use.doc     critical appraisal skills programme (2006). 10 questions to help you make sense of a review. retrieved 19 apr., 2013, from http://www.casp-uk.net/wp-content/uploads/2011/11/casp_systematic_review_appraisal_checklist_14oct10.pdf   crumley, e. & koufogiannakis, d. (2002). developing evidence-based librarianship: practical steps for implementation, health information & libraries journal, 19(2), 61-70.   eldredge, j. d. (2000). evidence based librarianship: formulating ebl questions. bibliotheca medica canadiana, 22(2), 74-77.   eldredge, j. d. (2001). the most relevant and answerable research questions facing the practice of health sciences librarianship, hypothesis, 15(1), 3-5.   eldredge, j. d., ascher, m. t., holmes, h. n., & harris, m. r. (2012). the new medical library association research agenda: final results from a three-phase delphi study. journal of the medical library association, 100(3), 214-218.   eldredge, j. d., ascher, m. t., holmes, h. n., & harris, m. r. (2013). top-ranked research questions and systematic reviews. hypothesis, 24(2), 19-20.   eldredge, j. d., harris, m. r., ascher, m. t. (2009). defining the medical library association research agenda: methodology and final results from a consensus process. journal of the medical library association, 97(3), 178-185.   ely, j. w., osheroff, j. a., ebell, m. h., bergus, g. r., levy, b. t., chambliss, m. l., & evans, e. r. (1999). analysis of questions asked by family doctors regarding patient care. bmj, 319(7206), 358-361.   ely, j. w., osheroff, j. a., ebell, m. h., chambliss, m. l., vinson, d. c., stevermer, j. j., & pifer, e. a. (2002). obstacles to answering doctors' questions about patient care with evidence: qualitative study. bmj, 324(7339), 710.   ely, j. w., osheroff, j. a., chambliss, m. l., ebell, m. h., & rosenbaum, m. e. (2005). answering physicians' clinical questions: obstacles and potential solutions. journal of the american medical informatics association, 12(2), 217-224. doi:10.1197/jamia.m1608   folioz (2010). evidence based library and information practice (eblip-gloss): acquire. retrieved 22 apr., 2013, from http://foliozeblipgloss.pbworks.com/f/acquire.doc     glasziou, p. & haynes, b. (2005). the paths from research to improved health outcomes. acp journal club, 142(2), a8-a10.   glasziou, p., vandenbroucke, j. p., & chalmers, i. (2004). assessing the quality of research. bmj, 328(7430), 39-41.   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech 24(3), 387-399.   gorman, p. n. & helfand, m. (1995). information seeking in primary care: how physicians choose which clinical questions to pursue and which to leave unanswered. medical decision making, 15(2), 113-119. doi:10.1177/0272989x9501500203   gray, g. e. (2010). the 5-step evidence-based medicine model. in c. barr-taylor (ed.), how to practice evidence-based psychiatry: basic principles and case studies (pp. 13-15). arlington va, usa: american psychiatric publishing inc.    green, m. l. (2006). evaluating evidence-based practice performance. evidence based medicine, 11(4), 99-101. doi:10.1136/ebm.11.4.99   grefshiem, s. & rankin, j. a. (2007). making a commitment to eblip: the role of library leadership. paper presented at 4th international evidence based library and information practice conference, north carolina. retrieved 22 apr., 2013, from http://www.eblip4.unc.edu/papers/grefsheim.pdf   grefsheim, s. f., rankin, j. a., & whitmore, s. c. (2007). making a commitment to eblip: the role of library leadership. evidence based library and information practice, 2(3), 123-129. retrieved 22 apr., 2013, from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/447   hersh, w. r., crabtree, m. k., hickam, d. h., sacherek, l., friedman, c.p., tidmarsh, p., mosbaek, c., & kraemer, d. (2002). factors associated with success in searching medline and applying evidence to answer clinical questions. journal of the american medical informatics association, 9(3), 283-93. doi:10.1197/jamia.m0996   kloda, l. (2008). asking the right question. evidence based library and information practice, 3(4), 79-81.   korah, a. & cassidy, e. d. (2010). students and federated searching: a survey of use and satisfaction. reference & user services quarterly, 49(4), 325–332.   koufogiannakis, d., booth, a., & brettle, a. (2006). reliant: reader’s guide to the literature on interventions addressing the need for education and training. retrieved 22 apr., 2013, from http://eprints.rclis.org/handle/10760/8082   koufogiannakis, d., slater, l., & crumley, e. (2004). a content analysis of librarianship research, journal of information science, 30(3), 227-239.   lewis, s. & cotter, l. (2007). have the most relevant and answerable research questions facing librarians changed between 2001 and 2006? evidence based library and          information practice, 2(1), 107-120.   lorenzetti, d. (2007). identifying appropriate quantitative study designs for library research. evidence based library and information practice, 2(1), 3-14. retrieved 22 apr., 2013, from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/157/236    lynch, m. j. (2003). research questions for the twenty-first century. library trends, 51(4), 499-502. retrieved 22 apr., 2013, from https://ideals.illinois.edu/bitstream/handle/2142/8492/librarytrendsv51i4_opt.pdf?sequence=3   markville secondary school (2009). checklist for analysing a literature review. retrieved 22 apr., 2013, from http://www.markville.ss.yrdsb.edu.on.ca/history/honours/lit_review_checklist.doc   mcgraw-hill higher education (2001). communication research: glossary. retrieved 22 apr., 2013, from http://highered.mcgrawhill.com/sites/0767412176/student_view0/glossary.html   smith, r. (1996). what clinical information do doctors need? bmj, 313(7064), 1062-1068. doi: 10.1136/bmj.313.7064.1062   university of glasgow department of general practice (n.d.). critical appraisal checklist for an article on an educational intervention. retrieved 22 apr., 2013, from http://www.gla.ac.uk/media/media_64040_en.pdf   wilson, v. (2009a). looking to the literature: domains to help determine where to look. evidence based library and information practice, 4(2), 182-184. retrieved 22 apr., 2013, from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/6174/5359   wilson, v. (2009b). looking to the literature: open access and free sources of lis evidence. evidence based library and information practice, 4(3), 75-77. retrieved 22 apr., 2013, from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/6521/5536   wilson, v. (2009c). matching question types to study designs. evidence based library and information practice, 4(1), 51-52. retrieved 22 apr., 2013, from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/5063/5115   wilson, v. (2010). an introduction to critical appraisal. evidence based library and information practice, 5(1), 155-157. retrieved 22 apr., 2013, from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/7458   wimpenny, p., johnson, n., walter, i., & wilkinson, j. e. (2008). tracing and identifying the impact of evidence: use of a modified pipeline model. worldviews on evidence based nursing, 5(1), 3-12.   evidence based library and information practice editorial responsibilities      2015. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     editor-in-chief: lorie kloda   associate editor (evidence summaries): heather pretty   associate editor (articles): rebekah (becky) willson, lisl zach   associate editor (classics): jonathan eldredge   associate editor (reviews): denise koufogiannakis   associate editor (features, commentaries, using evidence in practice, research in practice, news): lorie kloda   production editor: michelle dunaway   editorial intern: melissa griffiths   editorial advisors: lindsay alcock, alison brettle, katrine mallan, pam ryan   copyeditors: richard hayman (lead copyeditor), georgianne bordner, julie evener, linda ferguson, heather healy, stacey penney, maria tan, mary virginia taylor, alison yeoman   indexing support: meral alakus   microsoft word art_ed_3.doc evidence based library and information practice       editorial    inspiration, anyone?        lindsay glynn  editor‐in‐chief    public services librarian and instruction coordinator, health sciences library   memorial university of newfoundland   st. johnʹs, newfoundland, canada  e‐mail: lglynn@mun.ca      © 2006 glynn. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      i have to admit that writing an editorial for  this issue was a struggle. trying to sit down  and write when the sun was shining outside  and most of my colleagues were on vacation  was, to say the least, difficult.  add to that  research projects and conferences…let’s just  say that i found myself less than inspired.  a  pitiful plea for ideas to a colleague resulted  in the reintroduction to a few recent  evidence based papers and resources which  inspired further searching and reading.    though i generally find myself surrounded  (more like buried) in research papers and  eblip literature, somehow i had missed the  great strides that have been made of late in  the world of evidence based library and  information practice.  i realize now that i am  inspired by the researchers, authors and  innovators who are putting eblip on the  proverbial map.  my biggest beef with  library literature in general has been the  plethora of articles highlighting what we  should be doing.  take a close look at the  evidence based practitioners in the  information professions: these are some of  the people who are actively practicing what  has been preached for the past few years.    take, for example, the about‐to‐be released  libraries using evidence toolkit by northern  sydney central coast health and the  university of newcastle, australia (see their  announcement in this issue).   an  impressive advisory group is responsible for  maintaining the currency and relevancy of  the site as well as promoting the site and  acting as a steering committee for related  projects.  this group is certainly doing more  than “talking the talk”: they took their  experience at the 3rd international evidence  based librarianship conference and did  something with the information they  obtained by implementing solutions that  worked in their environment.  the result?  the creation of a collection of tools for all of  us to use.  this toolkit is just what eblip  needs: a portal to resources aimed at  supporting the information specialists who  want to adopt the evidence based model of  practice.  i have already got it bookmarked  and set up an rss feed.  even before the  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:3    2 official toolkit launch, a wealth of  information is available on the website  including presentations, project and events  information, and a blog containing site  updates.      there has been much discussion on  increasing the knowledge base from which  to draw evidence for library and  information practitioners.  original research  needs to be published so that we can use it  as evidence to support our decision making.   the literature is lacking the significant  numbers of publication types widely  considered to be the highest level of  evidence: systematic reviews, meta‐analyses  and randomized controlled trials.  this issue  of evidence based library and information  practice proudly boasts both a systematic  review/meta‐analysis and a randomized  controlled trial.  denise koufogiannakis and  natasha wiebe, in their systematic  review/meta‐analysis, provide evidence that  shows that computer assisted instruction is  as effective as traditional instruction for  students at an introductory, undergraduate  level.  nicola pearce‐smith compared the  effectiveness of self‐directed, web‐based  learning with a classroom‐based, interactive  workshop in her randomized controlled trial.    i am looking forward to reading and  utilizing more and more of these in the  future.                                another example is even closer to home.  i  recently attended a strategic planning  meeting at my institution where new  services and procedures were discussed.   unlike the past, when new initiatives were  implemented and later evaluated, all  librarians instinctively indicated that the  literature should be searched first to see if  there was any evidence either for or against  changing or adding new services or  procedures.  the evidence based model of  practice is catching on.      i know that there are numerous examples of  information practitioners taking a proactive  role in putting research into practice, and  many research papers that are worth  mentioning.  to highlight them all would be  a task much too grand for an editorial.  kudos to all of you, and thanks for the  inspiration.     evidence summary   faculty decisions on serials subscriptions differ significantly from decisions predicted by a bibliometric tool   a review of: knowlton, s. a., sales, a. c., & merriman, k. w. (2014). a comparison of faculty and bibliometric valuation of serials subscriptions at an academic research library. serials review, 40(1), 28-39. http://dx.doi.org/10.1080/00987913.2014.897174   reviewed by: sue f. phelps reference librarian washington state university vancouver library vancouver, washington, united states of america email: asphelps@vancouver.wsu.edu   received: 3 dec. 2015     accepted: 2 feb. 2015      2016 phelps. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to compare faculty choices of serials subscription cancellations to the scores of a bibliometric tool.   design – natural experiment. data was collected about faculty valuations of serials. the california digital library weighted value algorithm (cdl-wva) was used to measure the value of journals to a particular library. these two sets of scores were then compared.   setting – a public research university in the united states of america.   subjects – teaching and research faculty, as well as serials data.   methods – experimental methodology was used to compare faculty valuations of serials (based on their journal cancellation choices) to bibliometric valuations of the same journal titles (determined by cdl-wva scores) to identify the match rate between the faculty choices and the bibliographic data. faculty were asked to select titles to cancel that totaled approximately 30% of the budget for their disciplinary fund code. this “keep” or “cancel” choice was the binary variable for the study. usage data was gathered for articles downloaded through the link resolver for titles in each disciplinary dataset, and the cdl-wva scores were determined for each journal title based on utility, quality, and cost effectiveness.   titles within each dataset were ranked highest to lowest using the cdl-wva scores within each fund code, and then by subscription cost for titles with the same cdl-wva score. the journal titles selected for comparison were those that ranked above the approximate 30% of titles chosen for cancellation by faculty and cdl-wva scores.   researchers estimated an odds ratio of faculty choosing to keep a title and a cdl-wva score that indicated the title should be kept. the p-value for that result was less than 0.0001, indicating that there was a negligible probability that the results were by chance. they also applied logistic regression to quantify the association between the numeric score of cdl-wva and the binary variable of the faculty choices. the p-value for this relationship was less than 0.0001, also indicating that the result was not by chance. a quadratic model plotted alongside the previous linear model follows a similar pattern. the p-value of the comparison is 0.0002, which indicates the quadratic model’s fit cannot be explained by random chance.   main results – the authors point out three outstanding findings. first, the match rate between faculty valuations and bibliometric scores for serials is 65%. this exceeds the 50% rate that would indicate random association, but also indicates a statistically significant difference between faculty and bibliometric valuations. secondly, the match rate with the bibliometric scores for titles that faculty chose to keep (73%) was higher than those they chose to cancel (54%). thirdly, the match rate increased with higher bibliometric scores.   conclusions – though the authors identify only a modest degree of similarity between faculty and bibliometric valuations of serials, it is noted that there is more agreement in the higher valued serials than the lower valued serials. with that in mind, librarians might focus faculty review on the lower scoring titles in the future, taking into consideration that unique faculty interests may drive selection at that level and would need to be balanced with the mission of the library.   commentary   with the rising cost of serials and a repeated need to make choices about what to keep and what to cut, the authors of this study present a unique process to determine how to involve faculty in the decision making process. they state that faculty selector models for monographs have historically been “conceptual rather than data driven” (p. 29); however, librarians have designed data-driven tests to compare library selections to those of faculty. though there have been reports in the literature about how to integrate faculty choices into serials decisions, there have not been any experiments into how faculty valuations compare to bibliometric valuations. it was with this intention that the authors set about designing this study, which was set in a local context and could possibly be replicated at any other location.   the researchers chose the california digital library weighted value algorithm (cdl-wva) to assess the value of the journals for this study because it integrates multiple datasets – including local usage, local citations, journal ranking measures and cost effectiveness. because it is designed to measure the value of journals for a specific library, and not value in general, it was determined to be an accurate comparator for this study.   the study was evaluated using glynn’s critical appraisal for library and information research checklist (glynn, 2006). the overall score was 84%, indicating that the study is valid. the population, data collection, study design, and results sections rated 75%, 80%, 100% and 83% respectively, all within the range of validity. the complex nature of the methodology weighed heavily in scoring results.   though the researchers express regret that the study lacked actionable conclusions, they present an interesting idea for serials valuation and faculty participation in the serials selection. they made a compelling case for the use of the cdl-wva as a bibliometric tool, and how to use that data to make a fair comparison with the faculty valuations.   there are some minor concerns, however. one is the two year difference between data collected from faculty and data from the link resolver. faculty choices would heavily depend on those specific faculty members who responded to the library request for cancelation choices, which may change within two years based on faculty turnover, research interests and courses they were teaching at the time. the rationale that the faculty choices were predictive is reasonable under these circumstances. though the article was detailed in its description of their methodology, some readers may want additional details regarding the dispersal of journal funds across disciplines at the institution studied, and how varied usage of journals by discipline factors into the analysis of the data.   these concerns in no way detract from the value of the methodology. overall, this study offers a model for serials evaluation that could be replicated by other libraries, as more serials cuts will likely be in our future.   reference   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154 microsoft word news2_1810_final_final_final evidence based library and information practice 2008, 3:2 77 evidence based library and information practice news cilip training and development courses © 2008. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. develop new skills and expertise through cilip training and development. courses cover such topics as cat and class, copyright, library and information management skills, marketing and management and personal development. courses running soon are: reflective practice for library and information staff identify the personal and organizational benefits to be gained from reflective practice. <http://www.cilip.org.uk/training/training/2 008/persdev/reflectivepractice.htm> how to construct a thesaurus make informed decisions about thesaurus designs that will enhance the quality of your subject data. <http://www.cilip.org.uk/training/training/2 008/catandclass/howtoconstructathesaurus.h tm> librarian as consultant understand the high level aspects of your role and develop the skills to improve confidence and credibility. <http://www.cilip.org.uk/training/training/2 008/mp/librarianasaconsultant.htm> supporting looked after children explore the reading and information needs of looked-after children and the role of libraries in supporting them. <http://www.cilip.org.uk/training/training/2 008/libinfo/supportinglookedafterchildren.h tm> moving on in marc 21: computer and web resources focus on bibliographic description and develop a greater depth of knowledge by exploring aacr2 alongside marc format requirements. <http://www.cilip.org.uk/training/training/2 008/catandclass/movingonmarccomputerwe b.htm> copyright compliance: policies and risk management identify high, medium and low risk activities and learn the practical steps to stay within the law. full details: <http://www.cilip.org.uk/training/training/2 008/copylic/copyrightcompliance.htm> evidence based library and information practice 2008, 3:2 78 dewey: a beginner’s guide covers all aspects of dewey, from basic navigation of the scheme and simple number allocation to the building of complex notation. <http://www.cilip.org.uk/training/training/2 008/catandclass/dewey.htm> introduction to competitive intelligence gain a good overview of ci including business frameworks, non-published sources and legal/ethical issues. <http://www.cilip.org.uk/training/training/2 008/mp/introductioncompetitiveintelligence. htm> leading others through change join other senior professionals and managers who are planning or implementing change. <http://www.cilip.org.uk/training/training/2 008/md/leadingothersthroughchange.htm> evidence based healthcare on the web cut through the maze of medical/healthcare information and focus on the solid evidencebased sources freely available on the web. <http://www.cilip.org.uk/training/training/2 008/libinfo/evidencebasedhealthcare.htm> handling enquiries remotely: effective enquiry answering by phone and e-mail manage communications effectively and minimize the risk of misunderstandings that happen when an enquirer can't see or hear you. <http://www.cilip.org.uk/training/training/2 008/libinfo/evidencebasedhealthcare.htm> web 2.0: building on the basics explore ways in which web 2.0 applications not only promote your information centre, but provide a richness and depth of information to your users. <http://www.cilip.org.uk/training/training/2 008/ict/web20buildingonthebasics.htm> finding and understanding market research information identify and evaluate major market research publishers and discover how to use government, trade and other organizations as a source. <http://www.cilip.org.uk/training/training/2 008/libinfo/findingmarketinformation.htm> teaching legal research skills explore the different evaluation methods to improve the effectiveness of your teaching sessions. <http://www.cilip.org.uk/training/training/2 008/tl/teachinglegalresearch.htm> negotiating licenses gain a better understanding of licence agreements, the terms and conditions and the challenges involved in contract negotiation. <http://www.cilip.org.uk/training/training/2 008/copylic/negotiatinglicences.htm> metadata in an e-learning environment improve your management of e-learning resources and learn how to create strategies for improving retrieval. <http://www.cilip.org.uk/training/training/2 008/catandclass/metadatainanelearningenvir onment.htm> furthering your management skills identify your strengths for teamwork and leadership and broaden your understanding of team dynamics. <http://www.cilip.org.uk/training/training/2 008/md/furtheringmanagementskills.htm> going beyond google understand google's limitations and explore the 'invisible web', subject gateways and virtual libraries. <http://www.cilip.org.uk/training/training/2 008/ict/goingbeyondgoogle.htm> evidence based library and information practice 2008, 3:2 79 repackaging your research: adding value to your answers learn how to present visually attractive and well expressed answers that add real value to your service. <http://www.cilip.org.uk/training/training/2 008/libinfo/repackagingresearch.htm> cataloguing and classification an introduction to the skills of information organization, understanding the principles involved in aacr2, dewey and marc. <http://www.cilip.org.uk/training/training/2 008/catandclass/catandclass.htm> critical appraisal skills for healthcare librarians: the basics designed to give you the skills to critically evaluate research papers and interpret their findings. <http://www.cilip.org.uk/training/training/2 008/libinfo/crticalappraisalskills.htm> essential skills for library and information staff ideal for those who are new to working in a library or information service, covering the core skills needed in your role. <http://www.cilip.org.uk/training/training/2 008/libinfo/essentialskills.htm> all cilip members receive up to 40% discount on all cilip training courses. you can get in touch with training team by calling 020 7255 0560 or e-mail: <training@cilip.org.uk >. review article   undercover feminist pedagogy in information literacy: a literature review   emily kingsland liaison librarian mcgill university montreal, quebec, canada email: emily.kingsland@mcgill.ca   received: 31 aug. 2019                                                             accepted: 22 jan. 2020      2020 kingsland. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29636     abstract   objective – feminist pedagogy in library instruction presents a new approach to actively engaging students in the research process. while feminist pedagogy in universities found early adoption in the 1970s, it is a newer phenomenon in library instruction, finding its early roots in works by ladenson (2010), accardi (2010), and accardi (2013).   by fostering active engagement and critical thinking skills, feminist library instruction sessions encourage students to question authority, actively participate in the knowledge production process, and become aware of their power and information privilege as they navigate increasingly complex information environments. at its core, this specific pedagogical approach subverts traditional classroom dynamics by focusing on diversity and inclusion. this literature review demonstrates how feminist pedagogy is currently being practiced in academic library information literacy sessions and how students can be assessed in a feminist manner.   methods – practitioners of feminist pedagogy draw on techniques and methodologies designed to emphasize and value different experiences, such as cooperative learning, collaborative learning, inquiry-based learning, and inquiry-guided learning. these techniques and methodologies are used to develop students’ information literacy skills, to take ownership of the research process, and to stimulate critical inquiry.   for the literature review, the following databases were searched: education resources information center (eric) on the proquest platform; library & information science abstracts (lisa); library, information science & technology abstracts (lista); scopus; and web of science core collection. hand searching in worldcat, as well as cited reference searching and bibliography mining, were also conducted. the searches were run between november 2018 and april 2019, followed by a second round in july 2019 based on participant feedback from the 2019 eblip10 conference. case studies, books, book chapters, literature reviews, research papers, interviews, surveys, and papers based on statistical and qualitative analysis were consulted.   results – while some librarians may lack familiarity with feminist theory, feminism writ large influences academic librarians’ professional practice (schroeder & hollister, 2014). librarians can incorporate feminist pedagogy into their practice and assessment in many concrete ways. however, librarians who focus on feminist pedagogy may face obstacles in their teaching, which may explain why publications on feminist pedagogical discourse within library and information studies have emerged only within the last decade (fritch, 2018; hackney et al., 2018). the most common challenge feminist librarians face is the restrictive nature of the standalone, one-shot information literacy session. moreover, there is much room for improvement in library and information studies programs to introduce students to the theory and practice of feminist pedagogy.   conclusion – this paper highlights examples of feminist methods librarians can put into practice in their information literacy sessions and ways in which students can be assessed in a feminist manner. the literature demonstrates that feminist pedagogy has been successfully implemented for decades in universities. by comparison, practicing feminist pedagogy at the library instruction level is a relatively new area of focus within the profession. hopefully, this growing trend will lead to more evidence based literature in the near future.      introduction   feminist pedagogy in library instruction presents a new approach to actively engaging students in the research process. i am a liaison librarian at mcgill university, which is a large research institution in montreal, quebec, canada. as is the case with many librarians, i do not have a formal background in feminist studies. instead, i discovered the concept of feminist pedagogy after the completion of my library and information studies (lis) degree, through a position as a subject librarian for the feminist and gender studies department at the university of ottawa. through work organizing feminist wikipedia edit-a-thons on international women’s day and on ada lovelace day, i expanded on my foundational knowledge of feminist theory. while my work at these standalone one-shot workshops proved to be professionally and personally rewarding, i was aware that the audience was self-selected, and i wanted to expand the reach of my practice of feminist pedagogy.   a trend i observed when teaching standalone or one-shots to undergraduate psychology students was that while the vast majority of these students were women, the first students to participate were usually without fail the male minority. i wished to foster an environment where everyone felt welcome and encouraged to participate and therefore turned to literature on feminist pedagogy. i decided to conduct a literature review to determine how feminist pedagogy is being practiced in library information literacy sessions.   definitions    before examining the literature surrounding feminist pedagogy in library information literacy sessions, it is important to define key terms to ensure readers are on the same page. librarian maria accardi is the author of feminist pedagogy for library instruction. published in 2013, it is a seminal work. she explains that the feminist approach is “broadly concerned with social justice and sees education as a site for social change and transformation, exposing and ending oppression against women and all other kinds of marginalization: racism, xenophobia, classism, ableism, and so on” (p. 39). she goes on to explain, “beyond simply understanding how knowledge is produced, feminist library instructors encourage students to be agents of change, thus transforming the dominant culture of knowledge production” (p. 39).   critical information literacy was popularized by lis professor james elmborg in 2006, and it asks librarians to encourage students to think critically about the information they encounter, be it academic or otherwise, and to develop what can be understood as a “critical consciousness” (p. 192). accardi acknowledges that feminist pedagogy is a form of critical pedagogy, noting that feminist educators are concerned with learner-centered, anti-hierarchical, collaborative, and participatory learning environments that value personal narratives. accardi states:   critical library pedagogy asks educators to consider the context in which students are situated in their everyday lives and consider these contexts as they plan library instruction. feminist library pedagogy goes further, by encouraging instructors to include the lives and knowledge of students in their lesson planning and to teach from the standpoint that all knowing is partial. (pp. 13-14)   the theory of intersectionality was developed in 1989 by feminist scholar kimberlé crenshaw. the term gained visibility in academic writing when feminist activists such as author and professor bell hooks (1994) described the multiple oppressions faced by black women to be intersectional oppressions. intersectionality attempts to identify how various forms of social stratification (such as race, class, religion, sexual orientation, gender, age, and disability) interlock with systems of power to impact those who are most marginalized in society. as articulated by hooks (2015), the emergence of intersectionality “challenged the notion that ‘gender’ was the primary factor determining a woman's fate” (p. xiii).   history   grounded in feminist theory, feminist pedagogy in higher education emerged in the 1970s, born out of three dramatic upheavals: the women’s movement, rapidly changing student demographics, and a demand for more inclusive and egalitarian knowledge across all scholarly disciplines. the first instance of the concept of feminist pedagogy appeared in 1981 in an essay by dr. berenice m. fisher, titled “what is feminist pedagogy?” its aims, broadly speaking, are to encourage students “to gain an education that would be relevant to their concerns, to create their own meanings, and to find their own voices in relation to the material” (maher & tetreault, 2001, p. 4). for their part, maher and tetreault’s pioneering 1994 work the feminist classroom (later updated in 2011) draws on in-class observations and in-depth interviews with both professors and students at six american colleges and universities. it examines the feminist pedagogical approach of professors over a period of two decades. their work, which also examined the dynamics of gender, race, and privilege, demonstrates that feminist pedagogy has the potential to transform any classroom. it is about the way teachers teach, regardless of the classroom’s subject matter.   with regards to feminist pedagogy within the field of lis, the earliest seminal work of note is librarian sharon ladenson’s book chapter “paradigm shift: utilizing critical feminist pedagogy in library instruction,” published in 2010. ladenson includes an overview of the core tenets of feminist pedagogy, including its resistance to passive student behaviour and dedication to active learning, critical thinking, cooperation, collaboration, difference, and diversity. ladenson’s chapter went on to inform accardi’s aforementioned 2013 work, feminist pedagogy for library instruction: “feminism already is, in a way, informing our teaching practices, in that library instruction favors active learning, a nurturing environment, and learner-centered pedagogy. we are already doing this. why make the politics more explicit? because this is how social change happens” (pp. 57-58). in her 2014 article “cyborgs in the academic library: a cyberfeminist approach to information literacy instruction,” librarian gina schlesselman-tarango points out that we in libraries are sadly behind the times when it comes to practicing feminist pedagogy. schlesselman-tarango writes: “feminist pedagogy and library instruction are relatively new bedfellows, and while feminist techniques have likely been employed in library instruction for some time, this approach to information literacy has only recently been explored in detail” (p. 38).   aims   this literature review seeks to examine how feminist pedagogy is being practiced in academic library information literacy sessions and whether students can be assessed in a feminist manner. it aims to provide concrete examples of how to put feminist pedagogy into practice. it also seeks to highlight challenges that librarians might face when practicing feminist pedagogy and provide, where possible, suggestions for addressing these challenges. furthermore, it will discuss how students may be evaluated in a feminist manner and provide cases taken from the literature.    methods   the following databases were searched: education resources information center (eric) on the proquest platform; library & information science abstracts (lisa); library, information science & technology abstracts (lista); scopus; and web of science core collection. the author also performed hand searching in worldcat for books and book chapters. search terms included feminis* and (pedagog* or teach* or educ* or instruct*) and librar*. the study benefited from cited reference searching and bibliography mining to identify related articles that may not have explicitly referred to feminist pedagogy, but drew on many of its principles through the practice of critical information literacy. this practice also helped to identify seminal, foundational works. the “librar*” portion of the search string was removed in order to find materials that addressed feminist pedagogy in the university classroom, rather than simply restricting its practice to academic libraries. the searches were conducted between november 2018 and april 2019, and a second round was run in july 2019, based on participant feedback from the 2019 eblip10 conference. the latter search broadly examined feminism in librarianship, rather than narrowing it strictly to feminist pedagogy. case studies, books, book chapters, literature reviews, research papers, interviews, surveys, and papers based on statistical and qualitative analysis were consulted.   literature review    how can library and information professionals go about practicing feminist pedagogy? can students be assessed in a feminist manner? the literature revealed several concrete examples of how to do so. the author consulted literature both within and outside of the field of lis and included the broader topic of critical information literacy in addition to feminist pedagogy.   how to put feminist pedagogy into practice   dialogue facilitation   at its core, practicing feminist pedagogy boils down to being an excellent facilitator of dialogue and group discussion (accardi, 2013; couture & ladenson, 2017; grimm & meeks, 2017; hackney et al., 2018; lai & lu, 2009; maher and tetreault, 2001; tewell, 2018; wallis, 2016). the ideal feminist pedagogue steers the conversation with a gentle touch, asking prompting questions when necessary, to move the dialogue along without implicating oneself too much in it. “creating opportunities for dialogue and discussion was central to the instructional practice” of many of tewell’s (2018, p. 19) study participants who were practicing critical information literacy. since feminist pedagogy falls under the umbrella of critical information literacy, much can be gleaned from librarian eamon tewell’s work interviewing librarians. those who practice feminist pedagogy are open to different ideas, even those that do not mirror their own, and openly encourage different voices to be heard. facilitators should aim to “foster a feminist learning space that privileges dialogue, collaboration, experience-based knowledge, and gender-centeredness – all of which resonates with feminist pedagogy” (lai & lu, 2009, p. 65). while not librarians, lai and lu are professors who taught an undergraduate online course, titled images of women in western civilization. they used it as a case study and examined how employing feminist pedagogy facilitated student asynchronous online discussions (p. 58).   while the term “intersectionality” had yet to be coined, maher and tetreault’s research dating back to the late 1980s indicates that intersectional feminist pedagogy was already being practiced in some academic institutions. they recorded a discussion in 1987 around white feminist theory, which focuses on “gender as the major issue and [subordinates] race, class, and sexual orientation to the primacy of gender oppression (while ignoring white women’s skin privilege on other accounts)” (p. 170). their research demonstrated that when the majority of a classroom is composed of white students, it enables “racial insulation,” which means that gender is discussed in a bubble, and disregards race, class, and sexuality (p. 170). to counteract racial insulation, they observed the successful intersectional feminist facilitation technique of dr. gloria wade-gayles, who taught images of women in literature at emory university. the class was made up of a mix of black and white, and male and female students. wade-gayles noted that students tended to cluster together based on race. during one class, she twice asked students to change where they sat. maher and tetreault observed that as a result, white female students who were sitting on the perimeter of the room moved closer to the centre and intermingled with their black classmates (p. 173). the physical space that students occupied influenced the dynamics and discussions that took place in the classroom. “as we compared the classes we observed, we saw that dynamics of position shaped the particular forms that mastery, voice, and authority took in each classroom” (p. 173).   nearly half of the librarians interviewed by librarian dr. annie downey (2016) “centered their classes around dialogue or discussion, and for many of those, it was the chief device they used to teach critical information literacy concepts” (pp. 91). while downey’s work was framed within the context of critical information literacy, its findings, much like tewell’s, can help inform feminist pedagogy. a good instructor will find ways to encourage shyer participants to join in through various means. this could be through a think-pair-share activity, for example, which is a collaborative learning strategy in which participants are given time to think about a topic or question, are then paired with another student for discussion, and then may share their findings with the larger group. this technique ensures that individual students are not put on the spot, as it were, and has been adopted by many feminist pedagogy practitioners, such as librarians wallis (2016) and couture and ladenson (2017). these subtle, undercover facilitation strategies will help librarians and professors alike “move discussions towards deeper, richer, and critical directions” (lai & lu, 2009, p. 63). ultimately, creative dialogue facilitation is key to practicing feminist pedagogy in information literacy.   feminist search examples    perhaps the most common recurring practice was simply using examples in one-shots that are linked in some way, shape, or form to feminism. for example, subject librarians in science, technology, engineering, and mathematics (stem) fields can demonstrate database searches by looking up “women in science” or “gender and the informal economy” or notable women in the field, such as mathematician ada lovelace. librarian ellen broidy (2007) explains how lovelace was used as a search example:   this iconic figure from the first half of the nineteenth century drew the students’ attention to the fact that women had been part of the technological revolution from the very beginning. at the same time, ada’s life, work, and untimely death from uterine cancer vividly illustrated the intersecting themes of gender, technology, and the politics of information. (p. 501)   ladenson (2010) incorporated intersectional feminist examples into her gender studies classes by including active learning techniques to stimulate critical inquiry:   at the beginning of each instruction session, the librarian shows students colorful pictures of a prominent woman in popular culture and/or public life (such as oprah winfrey). she also shows students pictures of another politically, socially and/or historically significant woman who is less ubiquitous in contemporary culture (such as angela davis). after briefly identifying each woman and naming some of her significant accomplishments, the librarian asks students to write down at least one question they would like to ask about each woman, and list at least one information source in which they would expect to find the answer. after generating their ideas, the students share their questions and information sources with the class. next, the librarian lists the information sources on the board, and engages students in further discussion by asking them to identify which ones are primary, and which ones are secondary. (pp. 109-110)   by highlighting the experiences and contributions of women of colour in the classroom, and subverting traditional classroom dynamics, ladenson demonstrates an appreciation for intersectional feminist pedagogy.   these simple practices enable librarians to surreptitiously incorporate feminist pedagogy into the classroom. students who are following along will find articles related to women practicing in these heretofore male-dominated professions and perhaps glean a bit of information about a topic they would otherwise not have been exposed to during their studies. a biological literature professor observed by maher and tetreault (2001) had students do labs, where they learned that in past scientific tests the behaviours of female fish were not examined. at the time of publication, maher was a professor of education at wheaton college in norton, massachusetts, and tetreault was provost and vice president for academic affairs at portland state university. their book, the feminist classroom, is a seminal work in the field of academic feminist pedagogy. while the students “did not see gender as a central issue of the course” (maher & tetreault, 2001, pp. 137), this simple act of feminist pedagogy highlights simply one example of the ultimate undercover feminist. this applies to the lis classroom, as well. noble, austin, sweeney, mckeever, and sullivan (2014) developed a course on race, gender, and sexuality in the information professions. they made “a concerted effort to use scholars of color and women in lis to draw attention to the ways that lack of representation in the field directly impacts what can be studied, and from whose vantage points and epistemologies” (p. 218). using feminist search examples is one of the simplest ways to incorporate feminist pedagogy into one’s practice, and an effective strategy if subtlety is required.   faculty collaboration    a common difficulty for academic librarians who teach is finding ways to address broader conceptions of information within the constraints of a brief standalone session. reaching out to faculty before a one-shot and asking for topics or problems the class could work on together can provide several positive outcomes. downey (2016) argues that “building relationships with the teaching faculty is of the utmost importance to librarians and they are very careful with the relationships they develop” (p. 132). some librarians in the literature found this to be a helpful way to ensure the students had topic examples that were relevant to their assignments and lead to large gains in authentic student involvement and learning (couture & ladenson, 2017; downey, 2016). establishing relationships with faculty may lead to trust and open doors to conversations around pedagogical theory and practice.   some librarians found faculty outreach to be a potential entry point into discussing feminist pedagogy, depending on how receptive they were to librarian topic pitches and how the conversation flowed. for example, couture and ladenson (2017) met with faculty and had positive discussions around course objectives, type and number of sources, and the assignment’s relation to feminist theory. “eventually, we decided to not prescribe a set number or type of sources but instead use the session to reinforce and expand on concepts addressed in earlier in the semester, such as synthesizing information from numerous disciplines” (couture & ladenson, 2017, pp. 182-183).   potential downsides to this could be the faculty member being too prescriptive in deciding the content of the one-shot. this is not a perfect practice, but it can help identify faculty allies, open doors for future collaboration, and aid in practicing feminist pedagogy in future library information sessions.   student consultation    if the librarian is fortunate enough to be embedded in a class, invited to multiple sessions throughout the semester, or solicited to create an assignment and grading rubric, a recommended practice is to involve the students in the creation of the evaluation criteria and/or portions of the syllabus. accardi (2013) writes:   not only is feminist pedagogy concerned with subverting patriarchal subject matter, but it also is concerned with the way any subject matter is taught. . . . instead of the teacher serving as the ultimate authority on all knowledge and information, knowledge is collaboratively discussed and created by the students and the teacher together. (p. 25)   this empowers students and gives them agency over their education. rather than being passive receivers of knowledge (freire, 1970) students become actively engaged in the creation of their own assessment. a librarian does not need to be embedded in a class to consult students. this could also be as simple as beginning a library workshop by asking what students’ goals are for the session, or asking them to name one thing they want to learn that day, and then tailoring the information literacy session to suit their needs accordingly.   student consultations can also take the form of critiquing one another’s work. librarian ellen broidy (2007) taught a semester-long course and had students send their project proposals to the entire class two days ahead of their meeting. when the class met, the students discussed and dissected the topics and made suggestions for overcoming challenges. this method adhered to the “basic tenets of feminist pedagogy and [broidy’s] desire for a high degree of participation” (p. 499). to the causal student observer, nothing about this practice is blatantly feminist. this technique may therefore help those who wish to incorporate feminist pedagogy in the classroom, but who may not want to make their agenda explicit.   student consultation is also worth considering for lis faculty. many lis students come to library school with diverse work and academic backgrounds, as well as life experiences and intellectual tools that align with the principles of critical, as well as feminist, pedagogy (pawley, 2006, p. 165). faculty, librarian dr. christine pawley argues, can “take another step in this direction by setting aside hierarchical models of curriculum development in which they themselves constitute the main source of expertise and curricular knowledge” (p. 164) by partnering with students on developing the lis curriculum.   meaningful for librarians and information professionals   a major recurring theme was that almost all librarians, information professionals, and professors who engage in feminist pedagogy find the practice to be extremely meaningful (accardi, 2013; couture & ladenson, 2017; lai & lu, 2009; tewell, 2018). for example, one-shots over time can become rote and repetitive. by continually seeking new material, new feminist examples, and new ways to teach, those who practice feminist pedagogy are keeping the material fresh for themselves. there is also great satisfaction derived from engaging students in feminist debates, watching them delve into feminist literature, enabling conversations and discussions around feminist topics, and learning how much they took away from the session.   librarians and information professionals can also exert their feminist agenda outside of the classroom. this method may be particularly helpful for those who have little face-to-face time with their students. it can be as simple as creating a book display tied to feminism, social justice, or other progressive topics. the information professional who, as librarian baharak yousefi (2017) puts it, “co-opts and subverts existing structures” with the long-term goal of “progressive change or equitable access in mind” is a classic undercover feminist (p. 101).   practicing feminist pedagogy is not without its challenges. this portion of the paper will highlight major issues librarians have encountered and how they have counteracted this resistance.    challenges with feminist pedagogy   one-shot instruction    standalone, one-shot information literacy sessions are the most common form of teaching by academic librarians (downey, 2016). one survey conducted by downey looked at librarians from a variety of institutional types and found that 94% of teaching librarians teach one-shots (p. 82). the nature of the one-shot makes meaningful, in-depth feminist pedagogy difficult to put into practice outside of very basic, surface-level examples. wallis (2016) writes that “librarians using critical and feminist pedagogies are ultimately stunted by the one-shot” (p. 5). research by tewell (2018) confirmed, “even the most sophisticated pedagogy is really, really limited in [the one-shot] format” (p. 21). yet even when presented with these challenges, feminist pedagogy is “still important, still possible, still worth pursuing” (accardi, 2013, p. 69). accardi (2013) goes on to explain that:   the marginal status of librarians gives us more freedom to experiment with our pedagogy than regular teaching faculty have, especially if we are not bound by the strictures of the credit-bearing information literacy course. while the one-shot class has its own set of challenges, it also has more flexibility that progressive librarians can take advantage of and subvert for progressive purposes. (p. 69)   while librarians cannot observe the long-term effects of this instruction, they can still covertly introduce feminist concepts into the classroom.   faculty and administrative pushback    librarians can encounter pushback from faculty or university administration when incorporating overtly feminist examples in their teaching. when librarians reach out to faculty for collaboration and make their feminist agenda clear, this presents a risk. unfortunately, librarians must be careful in declaring their intention to practice feminist pedagogy, depending on the faculty member or lecturer. at some institutions, there is a power imbalance between the subject librarians and their respective faculty members. as downey (2016) explains, “librarians often feel indebted to any professor who allows them to teach for their classes and are overly careful to make sure they do not upset them in any way by treading too heavily in their territory, making too many suggestions for content, or making suggestions too strongly” (pp. 149-150). one librarian interviewed by tewell (2018) highlighted lack of faculty understanding as a major obstacle to putting critical information literacy into practice: “faculty don’t know that librarians actually think about these things and have a pedagogy and theory that drive our work. they think we just show databases and that is the extent of our value. changing that perception is very hard and often demeaning” (p. 22).   the literature indicates that expanding or building on critical information literacy, including feminist pedagogy, is heavily dependent on library administrative support (downey, 2016; maher & tetreault, 2001). this can help negate what maher and tetreault dub “turf issues” (p. 128) that arise when faculty and librarians have different visions for information literacy instruction. downey explains that library administrators should “lay the groundwork and strive to make information literacy more of a priority at the institutional level” (pp. 149-150).   subject matter   certain courses lend themselves more easily to feminist pedagogy. obvious examples include feminist theory and feminist literature classes, as the students are already primed. other courses, particularly those in stem fields, may present larger challenges in finding natural ways to incorporate this. nevertheless, librarians can become creative in how they choose to include feminist examples in their work. as previously mentioned, maher and tetrault (2011) observed a professor teach a course on biological literature. the professor had students do labs, where they learned that past scientific tests did not examine the behaviours of female fish – only male. the professor also asked students to do a “free-write” on women in science. the professor “seemed to be saying simply that if women – and their differences – were included in scientific thinking and practice, then science would be expanded and improved” (p. 138). proof that this subtle tactic was yet another example of undercover feminism, maher and tetreault found that students viewed the course pragmatically and “did not see gender as a central issue of the course” (p. 137). and yet when maher and tetreault talked to each student individually, they shared that they found the class empowering.   student resistance    feminist pedagogy has as one of its main tenets to hold students’ individual experiences as equally valid as the academic information discussed in the class. how students apply their own lived experiences as a means of understanding and interacting with the literature is a major first step towards the students engaging with the literature in a feminist manner. in many instances, the instructor takes a back seat to these discussions, and merely acts as a guiding light. unfortunately, there can be student resistance to this method of instruction. as hooks (1994) notes, “this type of learning process is very hard; it’s painful and troubling” (p. 153). some students perceive it as laziness on the instructor’s part or are conditioned to perceive their classmates’ life experiences as being of lesser import than the literature under review.   about one-third of the librarians interviewed by downey in 2016 described instances where students were resistant to critical information literacy methods (p. 99). some students expect only to be filled with knowledge, as outlined by freire’s (1970) seminal work pedagogy of the oppressed. others expect to be taught practical skills “that they could apply on the job, leaving theory by the wayside” (noble et al., 2014, p. 216).   one strategy, developed by couture and ladenson (2017), was to “ease students into this uncomfortable territory by demystifying the session” (p. 187). they recommend beginning the session by explaining one’s approach and outlining the benefits, and then moving on engaging the students in a group conversation on how “the process of raising critical questions can lead you to different directions that may create a more focused area of inquiry” (p. 187). clearly sharing one’s agenda is a solid technique if it is available to the librarian. however, this method is only available to those who do not fear potential repercussions.   authority   one of the disadvantages to being either a tenure-track librarian or a precariously employed contract librarian is that one may find oneself in a vulnerable place, with a desire not to “rock the boat.” while feminism, if not the theory and practice then the word itself, has increasingly become more mainstream in popular culture, there may be hesitation amongst librarians to openly declare their intentions or agenda until they have obtained tenure or a permanent position. as one librarian interviewed by tewell (2018) explained: “it is not always clear what their attitudes/approach to pedagogy are and as a young, tenure-track librarian i often feel like i need to ‘play it safe’” (p. 22).   some early career, tenure-track, or contract librarians may feel conflicted when tackling the contradictory emotions of wanting to be seen as an expert in the field, and yet simultaneously wishing to create an open and anti-hierarchical atmosphere in the classroom. how can librarians balance these seemingly opposed issues? noble et al. (2014) argue that while dialogue facilitation places the librarian in a position of authority, it prevents the discussion from deteriorating and ensures students deeply engage with the material (p. 215). maher and tetreault recommend illuminating the “constantly shifting context of professional authority” by making one’s authority “positional, rather than externally imposed, by grounding it in personal experience, knowledge, and situation” (p. 165). gender also plays an important role in how authority is perceived. research has shown that female professors receive poorer student evaluations than their male counterparts (peterson, biederman, andersen, ditonto, & roe, 2019). maher and tetreault’s (2001) found that that female professors practicing feminist pedagogy will deliberately maintain authoritative presences, particularly in large classrooms, to counteract this unconscious bias (p. 139).   tenured librarians, or those who have secured a permanent position within their institution, should use their authority to critically engage, ask difficult questions, and use their influence for good. by not doing so, librarian jennifer vinopal (2016) argues these librarians are not staying neutral – rather, they are “reinforcing systems of domination and oppression that need, instead, to be dismantled.” intersectional feminism provides librarians with both a theory and a practice for addressing oppression.   assessment   in seeking to answer the question “can students be assessed in a feminist manner?” the author uncovered several successful methods in the literature. it should be noted that many in the profession struggle to “meaningfully examine learning in feminist classrooms while not reinforcing power structures inherent within assessment mechanisms” (couture & ladenson, 2017, p. 186). however, the literature points towards several ways in which librarians can assess their students – and themselves – in a feminist manner, which is to say in a way that is “learner-centered and diverse and validates differing perspectives and voices (accardi, 2013, p. 112).   librarian lauren wallis (2016) had students work in pairs on shared google docs, where they had to create article maps. this enabled wallis, from her podium, to track their work in real-time and allowed her to identify students who were struggling and needed some additional help, which she provided by simply checking in during the activity. likewise, when she noticed groups making interesting observations, she could encourage students to share their ideas during the class discussion. in both cases, the technology – and the assessment it enabled – supported the session’s emphasis on dialogue between students and teacher. she was later able to review their work after class and email comments to the students. this, wallis states, allowed her to “engage in feminist assessment that thwarts the notion that there is only one answer or one way of knowing and experiencing the world” (p. 4). wallis confirmed that observationally her assessment interventions were successful, as the students were “more willing to question the established system of scholarly conversation than students in a traditional one-shot class” (p. 5).    a strong example for both practicing and assessing the effectiveness of feminist pedagogy is the one-minute essay (couture & ladenson, 2017). this technique is either used mid-session or at the end of a session to learn how much the students have taken away from the instruction. it asks the student to take a minute and write down what they learned and what is still unclear. this is a particularly useful tool during one-shots, given the paucity of valuable feedback one tends to receive, and can be tailored to align with instructor’s learning objectives (pp. 186-87). the one-minute essay also provides a moment of self-reflection for the students, and this reflective practice aims to give students agency in the learning process, which is another essential facet of feminist pedagogy.   self-assessment for the instructor is just as important as assessing students’ learning. librarian dory cochran (2016) recommends practicing our own self-assessment following the class and reflect on student reactions to the lesson. cochran provides possible questions for the librarian to consider: how much did i talk in comparison to students? when should i have talked more or less in order to draw out students’ ideas? what types of perspectives or viewpoints did students discuss? what was everyone’s participation level like? what might have influenced some to participate less and others more? (p. 113).    librarians teaching in semester-long lis programs could practice feminist pedagogy in one of the following ways: including students in the development of their assessment; featuring feminist lis content on the syllabus; and using any of the assessment methods included in this paper. work by librarians dr. bharat mehra, hope a. olson, and suzana ahmad (2011) has shown that the top five ways to include diversity in courses is through course readings, class discussion, assignment topics, and case studies. pawley (2006) argues that critical information literacy in the lis classroom should not simply be restricted to issues-based courses. foundational courses in lis, as well as research methods courses, can also provide opportunities for this critical reflection (p. 164). librarians offering credit-based workshops, or workshops that appear on a student’s co-curricular record, could work alongside the credit-issuing body to assess the student throughout their time at their institution. this would allow the lis profession to track the long-term benefits of feminist pedagogy.   by introducing feminist pedagogical practices in the lis classroom, and either covertly or overtly sharing one’s agenda with the students, librarians help to shape classroom conversations. these conversations are the first step towards encouraging students, future lis professionals, to critically examine lis literature, to enact positive change in the field by testing out feminist pedagogical approaches, and perhaps later to evaluate said approaches, with the eventual hope of later producing evidence based lis literature. currently the lis field is lacking vis-à-vis publications that examine intersectional identities. a survey of lis journals published from 1975 to 2013, conducted by information professionals hackney et al. (2018), shows that less than 1% of the literature “is concerned with questions of identity, and of that fraction, the majority does not consider intersectional identities or attempt praxis in significant ways” (p. 29). it is imperative that these practices be employed in a timely manner. as jaeger, subramaniam, jones, and bertot (2011) warn, “unless meaningful action occurs soon, lis as a profession and libraries as a societal institution risk becoming exclusive rather than inclusive” (p. 177).    conclusion   feminist pedagogy is an important, useful, and effective pedagogical tool in higher education. it can be achieved in myriad ways: shrewd class facilitation; building relationships with faculty and lecturers to discuss pedagogical practices; consulting students on class content, syllabus creation, and evaluation criteria; and using search examples that highlight the work of individuals who identify as women. these methods can be deployed surreptitiously or overtly and ensure a meaningful and empowering experience for students and information professionals alike.   while lis literature on feminist pedagogy emerged only a decade ago, this paper demonstrates that through creative use of free technology and one-minute essays and by practicing self-assessment, librarians and information professionals can assess their pedagogical practices in a feminist manner and contribute to future literature on the topic. the major challenge librarians are continually up against is the restrictive nature of the standalone, one-shot information literacy session, since tracking the long-term effects of feminist pedagogy on its learners is difficult. however, this also provides librarians with more flexibility than instructors and grants us the freedom to test out different pedagogical practices.   as a profession, librarians have a role to play in enacting social change, fostering active student engagement, and promoting critical thinking skills. through feminist pedagogy, librarians subvert traditional classroom dynamics, making us undercover feminists. this paper provides librarians with examples to experiment with feminist pedagogy and feminist assessment in the classroom, with the hope that it will lead to more evidence based literature in the near future.    references   accardi, m. t. (2010). teaching against the grain: critical assessment in the library classroom. in m. t. accardi, e. drabinski, & a. kumbier (eds.), critical library instruction: theories and methods (pp. 251-264). duluth, mn.: library juice press.   accardi, m. t. (2013). feminist pedagogy for library instruction. sacramento, ca: library juice press.   broidy, e. (2007). gender and the politics of information: reflections on bringing the library into the classroom. library trends, 56(2), 494-508. https://doi.org/10.1353/lib.2008.0004   crenshaw, k. (1989). demarginalizing the intersection of race and sex: a black feminist critique of antidiscrimination doctrine, feminist theory and antiracist politics. the university of chicago legal forum, 139-167.   cochran, d. (2016). using pop culture, feminist pedagogy, and current events to help students explore multiple sides of an argument. in n. pagowsky & k. mcelroy (eds.), critical library pedagogy handbook (vol. 2, pp. 109-115). chicago, il: association of college and research libraries.   couture, j., & ladenson, s. (2017). empowering, enlightening, and energizing: research as inquiry in women’s and gender studies. in s. godbey, s. b. wainscott, & x. goodman (eds.), disciplinary applications of information literacy threshold concepts (pp. 177-189). chicago, il: association of college and research libraries.   downey, a. (2016). critical information literacy: foundations, inspiration, and ideas. sacramento, ca: library juice press.   fisher, b. m. (1981). what is feminist pedagogy? the radical teacher, 18, 20-24.   freire, p. (1970). pedagogy of the oppressed (m. b. ramos, trans.). new york, ny: herder and herder.   fritch, m. (2018). teaching as a political act: critical pedagogy in library instruction. educational considerations, 44(1), 1-11. https://doi.org/10.4148/0146-9282.1868   grimm, s., & meeks, a. (2017). break the stereotype! critical visual literacy in art and design librarianship. art documentation, 36(2), 173-190. https://doi.org/10.1086/694238   hackney, s. e., handel, d., hezekiah, b., hochman, j., lau, a., & sula, c. a. (2018). visualizing identities in lis literature. journal of education for library and information science, 59(1-2), 10-32. https://doi.org/10.3138/jelis.59.1-2.04   hooks, b. (1994). teaching to transgress: education as the practice of freedom. new york, ny: routledge.   hooks, b. (2015). feminist theory: from margin to center. new york, ny: routledge.   jaeger, p. t., subramaniam, m. m., jones, c. b., & bertot, j. c. (2011). diversity and lis education: inclusion and the age of information. journal of education for library and information science, 52(2), 166-183.   ladenson, s. (2010). paradigm shift: utilizing critical feminist pedagogy in library instruction. in m. t. accardi, e. drabinski, & a. kumbier (eds.), critical library instruction: theories and methods (pp. 105-112). duluth, mn: library juice press.   lai, a., & lu, l. (2009). integrating feminist pedagogy with online teaching: facilitating critiques of patriarchal visual culture. visual culture & gender, 4, 58-68.   maher, f. a., & tetreault, m. k. t. (2001). the feminist classroom: dynamics of gender, race, and privilege. lanham, md: rowman & littlefield.   mehra, b., olson, h., & ahmad, s. (2011). integrating diversity across the lis curriculum: an exploratory study of instructors’ perceptions and practices online. international federation of library associations and institutions, 37(1), 39-51. https://doi.org/10.1177/0340035210396781   noble, s. u., austin, j., sweeney, m. e., mckeever, l., & sullivan, e. (2014). changing course: collaborative reflections of teaching/taking ‘race, gender, and sexuality in the information professions’. journal of education for library and information science, 55(3), 212-222.   pawley, c. (2006). unequal legacies: race and multiculturalism in the lis curriculum. library quarterly, 76(2), 149-168.   peterson, d. a. m., biederman, l. a., andersen, d., ditonto, t. m., & roe, k. (2019). mitigating gender bias in student evaluations of teaching. plos one, 14(5), https://doi.org/10.1371/journal.pone.0216241   schlesselman-tarango, g. (2014). cyborgs in the academic library: a cyberfeminist approach to information literacy instruction. behavioral and social sciences librarian, 33(1), 29-46. https://doi.org/10.1080/01639269.2014.872529   schroeder, r., & hollister, c. (2014). librarians’ views on critical theories and critical practices. behavioral and social sciences librarian, 33(2), 91-119. https://doi.org/10.1080/01639269.2014.912104     tewell, e. c. (2018). the practice and promise of critical information literacy: academic librarians’ involvement in critical library instruction. college & research libraries, 79(1), 10-34. https://doi.org/10.5860/crl.79.1.10   vinopal, j. (2016, june 11). tech-ing to transgress: putting values into library practice [keynote]. retrieved from http://hdl.handle.net/2451/34801   wallis, l. (2016). mapping power and privilege in scholarly conversations. in n. pagowsky & k. mcelroy (eds.), critical library pedagogy handbook (vol. 2, pp. 1-7). chicago, il: association of college and research libraries.   yousefi, b. (2017). on the disparity between what we say and what we do in libraries. in s. lew & b. yousefi (eds.), feminists among us: resistance and advocacy in library leadership (pp. 91-105). sacramento, ca: library juice press.   bibliography   accardi, m. t. (ed.) (2017). the feminist reference desk: concepts, critiques, and conversations. sacramento, ca: library juice press.   angell, k., & tewell, e. (2017). teaching and un-teaching source evaluation: questioning authority in information literacy instruction. communications in information literacy, 11(1). https://doi.org/10.15760/comminfolit.2017.11.1.37   booth, c. (2014, december 1). on information privilege [blog post]. retrieved from http://infomational.com/2014/12/01/on-information-privilege/   bowers, j., crowe, k., & keeran, p. (2017). “if you want the history of a white man, you go to the library”: critiquing our legacy, addressing our library collections gaps. collection management, 42(3-4), 159-179. https://doi.org/10.1080/01462679.2017.1329104   byrd, d. (2010). teaching the ‘isms’: feminist pedagogy across the disciplines. towson, md: institute for teaching and research on women.   cooke, n. a. (2016). information services to diverse populations: developing culturally competent library professionals. santa barbara, ca: libraries unlimited.   creasap, k. (2014). zine-making as feminist pedagogy. feminist teacher, 24(3), 155-168. https://doi.org/10.5406/femteacher.24.3.0155   darder, a. baltodano, m., & torres, r. (2009). the critical pedagogy reader (2nd ed.). new york, ny: routledge.   gore, j. (1993). the struggle for pedagogies: critical and feminist discourses as regimes of truth. new york, ny: routledge.   jacobs, h. l., & jacobs, d. (2009). transforming the one-shot library session into pedagogical collaboration: information literacy and the english composition class. reference and user services quarterly, 49(1), 72-82. https://doi.org/10.5860/rusq.49n1.72     keilty, p., & dean, r. (2013). feminist and queer information studies reader. sacramento, ca: litwin books.   lew, s., & yousefi, b. (2017). feminists among us: resistance and advocacy in library leadership. sacramento, ca: library juice press.   megwalu, a. (2014). practicing learner-centered teaching. the reference librarian, 55(3), 252-255. https://doi.org/10.1080/02763877.2014.910438   nicholson, k. (2018, june 8). information into action? reflections on (critical) practice [keynote]. retrieved from https://ir.lib.uwo.ca/fimspres/51/   schlesselman-tarango, g. (2016). the legacy of lady bountiful: white women in the library. library trends, 64(4), 667-686. https://doi.org/10.1353/lib.2016.0015   schlesselman-tarango, g., & suderman, f. (2016). critical pedagogy and the information cycle: a practical application. in n. pagowsky & k. mcelroy (eds.), critical library pedagogy handbook (vol. 2, pp. 59–69). chicago, il: association of college and research libraries.   seale, m. (2010). information literacy standards and the politics of knowledge production: using user-generated content to incorporate critical pedagogy. in m. t. accardi & e. drabinski (eds.), critical library instruction: theories and methods (pp. 221–235). duluth, mn: library juice press.   shrewsbury, c. m. (1993). what is feminist pedagogy? women’s studies quarterly, 21(3/4), 8-16.   tewell, e. (2015). a decade of critical information literacy: a review of the literature. communications in information literacy, 9(1), 24-43. https://doi.org/10.15760/comminfolit.2015.9.1.174   tewell, e., & angell, k. (2016). authority and source evaluation in the critical library classroom. in n. pagowsky & k. mcelroy (eds.), critical library pedagogy handbook (vol. 2, pp. 49-57). chicago, il: association of college and research libraries.   wilkinson, c. w. (2004). stronger students, better research: information literacy in the women’s studies classroom. feminist collections, 25(4), 1-5.   wilkinson, c. w. (2006). learning from student learning: a librarian-instructor’s view of her information literacy class. feminist collections, 28(1), 9-16.     evidence based library and information practice 2009, 4:2 174 evidence based library and information practice evidence summary clinical informatics consult service positively affects some clinical decisions in the icu a review of: mulvaney, shelagh a., leonard bickman, nunzia b. giuse, warren e. lambert, nila a. sathe, and rebecca n. jerome." a randomized effectiveness trial of a clinical informatics consult service: impact on evidence-based decision-making and knowledge implementation." journal of the american medical informatics association 15.2 (2008): 203-11. reviewed by: jennifer kelson buckinghamshire hospitals nhs trust amersham, united kingdom email: jennie.kelson@buckshosp.nhs.uk received: 20 december 2008 accepted: 06 may 2009 © 2009 kelson. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. objective – to determine whether the provision of synthesized research evidence provided by the clinical informatics consult service (cics) affects the clinical decision-making of clinicians working in intensive care units (icus). design – non-blinded randomized control effectiveness trial. setting – icus in united states-based 658 bed university hospital providing tertiary care for adults and children. subjects – clinical staff working within one of four icus who submitted a request for clinical information during the study period. methods – valid requests submitted by clinical staff from the four clinical icus (medical, paediatric, trauma, or neonatal) were randomly allocated to receive information from the cics (cics provided) or no information (no cics provided). pre-consult forms, completed at the time of the request, examined reasons for the request and the clinical actions clinicians thought might be influenced by the search results. requestors could opt out of the no cics provided group either before or after the randomization of their request. responses to requests, supplied within 0.5 to 7 days as agreed with the requestor, included a search strategy and bibliographic references, a targeted list of full-text articles, and a written synthesis and critique of the relevant research. mailto:jennie.kelson@buckshosp.nhs.uk http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2009, 4:2 175 clinicians within both groups were free to conduct their own searches and reviews. an online evaluation form, emailed to recipients, was used to assess the impact of the information supplied. the evaluation form asked clinicians to record the time spent on their own searches, sources of information consulted including colleagues, the immediate and future impact of the information provided (either from the cics or their own searches), what influence the information had on their clinical actions, whether there were any barriers to using the information, and quality and overall satisfaction with the results provided by the cics. data was analyzed according to the randomized group assignment using standard intention-totreat analysis for the main outcomes between the two groups. statistical adjustments were made to control for possible clustering of responses or multiple ratings from individual clinicians. the data was also analyzed on an efficacy basis depending on who provided the search results. the groups were clinician only, cics librarianonly, or both clinician and cics librarian. results from the clinician only search group were used as a comparison to the remaining groups. this assessment did not take account of the randomization and therefore constitutes a cohort analysis. results were analysed by one of two methods using statistical software sas proc mixed (v9) for multi-level quantitative data analysis, e.g., analysis of variance, and spss (v14) for all other quantitative data analysis, including descriptive statistics. main results – the study period was conducted over 19 months: august 2004 to march 2006. during this time, 299 valid requests were received and 226 post consult evaluation forms were returned giving a response rate of 76%. post consult forms were returned for 108/146 of the cics provided group and 118/153 of the no cics provided group. the 24% of requests that had no post consult evaluation were excluded from further analysis. statistical tests, conducted to check for potential bias relating to missing data, suggested that the missing data had little impact on the findings. medical and neonatal icus accounted for the majority of completed forms (40.3% and 38.1% respectively). the majority of opt-outs (10.2% overall) were from the medical icu. no significant difference in outcome variables was found between opt-out and other requests when tested using analysis of variance (anova). evaluation forms were completed by 89 unique clinicians and over half (49) submitted more than one request. the average requests per clinician was 2.96, sd 3.17, range 1-15, and the average number of requests per clinician who submitted more than one request was 4.57, sd 3.55, range 2-15. total number of responses, mean, standard deviation, and cohen’s d effect size were reported for the outcome variables based on intention to treat analysis. results showed no significant difference between the groups on the immediate impact of the information provided, the number of articles read or the frequency with which clinicians consulted colleagues, either formally or informally. the potential future impact of the information was rated higher in the cics group (p=<0.01) and clinician’s satisfaction levels in the cics group were also significantly higher (p=<0.001). only the specific action “different or new treatment” option showed a statistically significant difference between the cics provided and no cics provided groups. a significantly greater percentage of clinicians in the cics provided group reported conducting their own searches (70.2%) compared to the no cics provided group (36.8%). where clinicians reported that the information provided, either from their own searches or evidence based library and information practice 2009, 4:2 176 cics, had no impact on patient care decisions, the reasons most commonly cited were that they were already doing what was recommended, there was insufficient evidence to guide their decision, or that the available evidence did not apply to their patient. analysis of the results on the basis of the intervention actually received showed both potential future impact and satisfaction were significantly higher for both the librarian-only (p<0.001 for both outcomes) and librarian plus clinician search groups (p<0.001 for both outcomes) compared to the clinician only group. in addition, there was a statistically significant difference (p=0.02) of future impact in favour of the librarian plus clinician search group over the librarian-only group. time taken to complete searches also showed a statistically significant difference between the groups, which is not surprising given that time reported by the librarians included time spent searching and summarizing results, whilst the clinicians recorded time spent searching only. conclusion – clinicians reported a higher level of satisfaction with search results provided by the cics and rated the future impact of the information more highly. the cics showed a statistically significant impact on some aspects of clinical decision-making, particularly with regard to treatment decisions. provision of information by the cics also appeared to encourage clinicians to undertake their own searches. however, the reasons for this are unclear. commentary in general, the methodology section is thorough and explains the approach taken. potential biases in the study design and in the analysis of the results were identified and the authors endeavoured to mitigate these by using appropriate statistical techniques (e.g., checking for differences between the groups and using techniques to control for multiple ratings by clinicians). both sas and spss were used to analyze the results (sas for multi-level analyses and spss for the others). however, the authors have provided little explanation as to why both programs were necessary or where the various packages were used. as this study was a randomized trial, it is a major omission that a discussion of the sample size or power calculation for the study was not included. underpowered studies, where insufficient subjects are recruited to a study in order to detect a difference between the intervention and control groups, may lead to the erroneous conclusion that the intervention had no effect (type ii error). the study reported a number of instances, particularly with regard to actions taken as a result of receiving information, where no statistical difference was found between the cics provided and no cics provided groups. based on the information presented, it is unclear whether this is a valid result, or due to the sample size being too small. clinicians were asked to self-report on their behaviour and it is conceivable that they under or over-estimated the impact of the information provided. a more objective measurement of clinical actions would improve the reliability and validity of the study. the forms used to record information requests and postconsultation evaluations were also omitted and it is not possible to judge whether, as the authors speculate, question ordering influenced clinician’s responses regarding overall impact of search results. clinicians’ existing knowledge of and prior training in using health sciences journal literature and databases may have influenced the likelihood they would undertake their own searches and the quality of their retrieved results. however, these aspects were not assessed prior to the study. in addition, follow-up interviews or focus groups could be used to investigate reasons why some clinicians do not use the cics and to ascertain the reasons why clinicians felt the cics was useful or not useful. clinicians who conducted their own searches in addition to those conducted by cics evidence based library and information practice 2009, 4:2 177 may exhibit specific characteristics or personality traits, and this may be worth exploring. the authors identified other limitations of the study, including the randomization of requests by priority level rather than by clinical unit (although this is aspect of the randomization process was not made clear in the methods section) and the impact the expertise of the librarians may have had in relation to the outcomes. a number of previous studies have attempted to ascertain the effect of clinical librarian services; however, this is the first to use a randomized study design. this research is especially timely given the emphasis on impact assessment of healthcare library services within the u.k. however, the results of this study may have limited applicability owing to the icus setting and omission of the pre and post evaluation forms. the study results are also potentially inconclusive owing to a possible lack of a power. evidence based library and information practice editorial responsibilities      2021. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     editor-in-chief: ann medaille   associate editor (evidence summaries): stephanie krueger, christina wissinger   associate editor (research articles): erin owens, lisl zach   associate editor (classics & reviews): jane schmidt   associate editor (commentaries, using evidence in practice): ann medaille   communications officer (news): kim mackenzie   production editor: rachel hinrichs   editorial intern: melissa cober   editorial advisors: lindsay alcock, alison brettle, michelle dunaway, lorie kloda, denise koufogiannakis, heather pretty   copyeditors: heather healy (lead copyeditor), katelyn baroody, kirstin duffin, julie evener, emily kingsland, alison moore, jane morgan-daniel, elizabeth stregger, nikki tummon, elaina vitale, ross wilson   editorial assistant: samantha sheplawy   evidence based library and information practice 2012, 7.1 4 evidence based library and information practice editorial responsibilities 2012. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. editor‐in‐chief: alison brettle associate editor (articles): wayne jones associate editor (evidence summaries): lorie kloda associate editor (classics): jonathan eldredge associate editor (feature): denise koufogiannakis associate editor (reviews): denise koufogiannakis associate editor (commentaries, using evidence in practice, ebl101, news): alison brettle feature editor: pam ryan production editor: katrine mallan editorial intern: michelle dunaway copyeditors: heather pretty (lead copyeditor), georgianne bordner, molly des jardin, linda ferguson, richard hayman, mary virginia taylor indexing support: pam morgan http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/� / evidence based library and information practice evidence summary   the value of information in professional settings is experienced through relationships and networks   a review of: sharun, s. (2019). exploring value as a dimension of professional information literacy. journal of information literacy, 13(2), 26–40. https://doi.org/10.11645/13.2.2627   reviewed by: rachel e. scott associate dean for information assets illinois state university normal, illinois, united states of america email: rescot2@ilstu.edu   received: 21 apr. 2020                                                             accepted:  8 july 2020      2020 scott. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29766     abstract   objective – to critically explore the frame “information has value” in a workplace setting.   design – semi-structured interviews.   setting – community health centre in canada.   subjects – seven health and human services staff members serving vulnerable, urban youth ages 12 to 24.   methods – the researcher employed phenomenography to analyze interviews and to identify categories of information practice.   main results – four categories of information practice emerged: resourcing, referring, outsourcing, and advocating. the researcher identified the value of information as central to participants’ experience of information practice in the workplace. subjects’ understanding of the nature and significance of value was situated within personal relationships and professional networks.   conclusion – the study demonstrated that a specific aspect of information literacy can be successfully investigated to highlight its complexity and to show how it is experienced in a specific setting. a second conclusion was the centrality of interpersonal relationships to how value is experienced in professional information practice. the researcher recommends further study exploring relational value and in the sociocultural practice of information literacy.   commentary   much of the scholarship on information literacy focuses on students and information professionals and is conducted within educational institutions. by conducting this study in a community health centre, sharun continues the work of lloyd (2009) and hicks (2015) to investigate information practices within a professional context in order to contribute to the development of a more holistic sociocultural theory of information literacy.   koufogiannakis, booth, and brettle’s reliant (reader’s guide to the literature on interventions addressing the need for education and training) instrument supports the critical assessment of interventions in library and information science (2006). i employed reliant to evaluate the design, results, and relevance of sharun’s study. the researcher explained the appropriateness of phenomenography as a research methodology within lis and cited relevant studies to convey how it has been effectively employed. sharun took important steps to confirm the validity of the coding and results. the researcher recorded and transcribed the interviews before sending transcripts to participants for verification. additionally, the category names came from participant language, and the categories were subsequently validated when the author asked participants to provide definitions and examples. the author does not indicate if any software was employed to facilitate coding the transcripts or analyzing the qualitative data.   the results section presents a definition and discussion of each of the four categories of information practice identified. long quotes from participants expound on how they experience each of these categories in their daily work and highlight the relational aspect of value in their information practices. by making explicit a connection between the study’s findings and the acrl framework, the author makes the article more immediately relevant to information professionals interested in that document.   this study meets its stated objective of critically exploring value in the information practice of health and human services employees. by investigating a specific aspect of information practice in the workplace, this study adds to the literature by contributing to the development of a multifaceted sociocultural theory of information literacy. a more specific and robust methods section may have encouraged readers to adopt the study to local settings. although the study is unlikely to be reproducible in its specific context, the richness of the qualitative data and the author’s work to confirm the validity of the responses and analysis add to the value of this study.   references   hicks, a. (2015). drinking on the job: integrating workplace information literacy into the curriculum. loex quarterly, 41(4), 9–15. https://commons.emich.edu/loexquarterly/vol41/iss4/4   koufogiannakis, d., booth, a., & brettle, a. (2006). reliant: reader’s guide to the literature on interventions addressing the need for education and training. library & information research, 30(94), 44–51. https://www.lirgjournal.org.uk/index.php/lir/article/view/271    lloyd, a. (2009). informing practice: information experiences of ambulance officers in training and on-road practice. journal of documentation, 65(3), 396–419. https://doi.org/10.1108/00220410910952401   evidence summary   digital object identifiers (dois) prove highly effective for long-term data availability in plos one   a review of: federer, l. m. (2022). long-term availability of data associated with articles in plos one. plos one 17(8), article e0272845. https://doi.org/10.1371/journal.pone.0272845   reviewed by: hilary jasmin research and learning services librarian health sciences library the university of tennessee health science center memphis, tennessee, united states of america email: hjasmin@uthsc.edu   received: 30 may 2023                                                             accepted:  20 july 2023      2023 jasmin. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30378     abstract   objective – to retrieve a range of plos one data availability statements and quantify their ability to point to the study data efficiently and accurately. research questions focused on availability over time, availability of urls versus dois, the ability to locate resources using the data availability statement and availability based on data sharing method.   design – observational study.   setting – plos one archive.   subjects – a corpus of 47,593 data availability statements from research articles in plos one between march 1, 2014, and may 31, 2016.   methods – use of custom r scripts to retrieve 47,593 data availability statements; of these, 6,912 (14.5%) contained at least one url or doi. once these links were extracted, r scripts were run to fetch the resources and record http status codes to determine if the resource was discoverable. to address the potential for the doi or url to fetch but not actually contain the appropriate data, the researchers selected at random and manually retrieved the data for 350 urls and 350 dois.   main results – of the unique urls, 75% were able to be automatically retrieved by custom r scripts. in the manual sample of 350 urls, which was used to test for accuracy of the urls in containing the data, there was a 78% retrieval rate. of the unique dois, 90% were able to be automatically retrieved by custom r scripts. the manual sample of 350 dois had a 98% retrieval rate.   conclusion – dois, especially those linked with a repository, had the highest rate of success in retrieving the data attached to the article. while urls were better than no link at all, urls are susceptible to content drift and need more management for long-term data availability.   commentary   the study contributes value to a body of literature surrounding data availability statements that has been established in several disciplines, including another publication by the author (federer, 2018). the author’s prior publication in this area notes a sharp increase in compliance since the 2014 plos one requirement of data availability statements but only 20% of complying publications use a repository to store their data. plos one has recently worked to incentivize use of repositories, creating an “accessible data” feature for articles using open science framework (osf), figshare, or dryad repositories (plos one, 2019). this incentive to brand work as accessible is further supported by the current study, which boasts 84.3% of resources in a repository available in comparison to 72% shared via other means.   the ebl critical appraisal checklist was used to measure validity of the study (glynn, 2006). overall, the study is sufficiently strong, with a 93.75% validity calculation. because the study used custom scripts, the only item from the checklist not accounted for is a validated data collection instrument. however, all scripts are available in the open science framework and can be found in the study’s data availability statement. this study tackled clear and concise research questions that it then answered with continued clarity, and the study methods are easy to follow and replicate for future research.   the information provided by the author has valuable implications for scholarly practice. as requirements for transparency grow, data availability statements may become the norm across academia. the use of dois, particularly in repositories, can save time for both readers and authors. for readers, the doi/repository route takes the least steps to reach the data; for authors, they will be spared emails from readers requesting the data if they cannot find it through the data availability statement. this may also be a valuable opportunity for libraries to build institutional repositories and incentivize faculty to input their data, as mounting proof indicates the necessity of transparency and replicability. if construction of a repository is outside the scope of a library’s time and budget allotment, librarians and informationists may benefit their users by sharing information about existing repositories available to them.   there are implications for future research, as this study solely measures two years of plos one’s data availability statements. this design should be replicated to measure these differences in different disciplines, in different journals, and in more recent years because the requirement for data availability has only grown.   references   data availability. (2019, december 5). plos one.  retrieved from https://journals.plos.org/plosone/s/data-availability   federer, l. m. (2022). long-term availability of data associated with articles in plos one. plos one 17(8), article e0272845. https://doi.org/10.1371/journal.pone.0272845 federer, l. m., belter, c. w., joubert, d. j., livinski, a., lu, y-l., snyders, l. n., & thompson, h. (2018). data sharing in plos one: an analysis of data availability statements. plos one 13(5), article e0194768. https://doi.org/10.1371/journal.pone.0194768 glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387–399. https://doi.org/10.1108/07378830610692154   evidence based library and information practice 2012, 7.1 65 evidence based library and information practice using evidence in practice floating collections at edmonton public library adrienne brown canty (now with the bc ministry of agriculture) manager, circulation procedures edmonton public library edmonton, alberta, canada email: acanty@epl.ca louise c. frolek (now retired) director, collection management and access edmonton public library edmonton, alberta, canada richard p. thornley manager, idylwylde branch edmonton public library edmonton, alberta, canada email: rthornley@epl.ca colleen j. andriats community librarian, londonderry branch edmonton public library edmonton, alberta, canada email: candriats@epl.ca linda k. bombak assistant manager, capilano branch edmonton public library edmonton, alberta, canada email: lbombak@epl.ca christalene r. lay membership services supervisor, whitemud crossing library edmonton public library edmonton, alberta, canada email: clay@epl.ca michael dell (now retired) manager, branch consulting edmonton public library edmonton, alberta, canada mailto:acanty@epl.ca� mailto:rthornley@epl.ca� mailto:candriats@epl.ca� mailto:lbombak@epl.ca� evidence based library and information practice 2012, 7.1 66 received: 2 sept. 11 accepted: 1 dec. 11 2012 canty, frolek, thornley, andriats, bombak, lay, and dell. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. setting the edmonton public library (epl) is a 17‐ branch urban public library system serving 782,439 residents (edmonton, 2009) in the city of edmonton, alberta, canada, which has a land area of 684.37 square kilometres (canada, 2006). edmonton has experienced strong growth in both population and physical size since 2001. both annual circulation and in‐person visits at epl are increasing, with a 23% jump in circulation from 2008 to 2009 alone. epl’s circulation in 2010 topped 13.3 million items, and in‐person visits that year exceeded 5.6 million. epl’s service model is, “we are one library, with one staff and one collection. every customer is my customer.” the library follows a community‐led service philosophy, collaborating to understand and respond to individual and community needs for collections and services. problem edmonton public library customers may borrow items from, and return them to, any service point in the system. epl historically employed a static “home library” model, in which all circulating items were returned to an assigned home branch upon check‐in. in this model, items sent between service points required processing by staff at the receiving location before they were available for use. with static collections, an overwhelming volume of items in transit frequently delayed customers’ receipt of materials. epl’s growing circulation led to increased staff workload associated with sorting, transferring, receiving, and shelving items. frequent physical handling of materials caused items to wear out prematurely and caused ergonomic issues for staff. inconsistent labeling practices at different locations detracted from customers’ “one‐ library” experience. a comprehensive process change was necessary to manage rising demands without compromising well‐established public service levels and expectations. epl began considering a floating collections model in 2004. under this model, an item is shelved where it is returned: its home branch is the location where it is checked in. through floating, epl hoped to: • reduce the handling of materials; • get materials to customers more quickly; • standardize labelling of materials; • give customers a consistent experience from branch‐to‐branch; and • promote and increase the use of the holds services. evidence in considering the need for floating collections and the potential impacts on services, staff, and customers, epl drew heavily upon the experience of other library systems and examinations of its own data. other library systems had reported achieving desirable outcomes such as substantial reductions in “in transit” materials through implementing floating collections. for example, jefferson county public library reduced the volume of material moving among http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/� evidence based library and information practice 2012, 7.1 67 its branches by 67 percent after floating its collections (cress, 2004), and sarasota county library system reported close to a 50 percent reduction in material moving between branches a year after implementing floating (sarasota county libraries, 2009). epl generated data using sirsidynix’s director’s station product, which allows customized analysis of customer, circulation, and collection data. the amount of material in transit, circulation figures, and patterns in customer holds were analyzed, as was the time required to fulfill customer holds of “on order” items and to process new acquisitions. customer feedback was also closely monitored for comments about the changes, as well as the occurrence of word fragments like “float” and “collect,” during and after the transition. implementation epl assembled a central collection standards team to manage the transition from static to floating collections, comprised of the director of the collection management and access division, three branch managers, two branch assistant managers, and one branch librarian. home locations are the divisions of a service point’s own collection, e.g., audio‐visual, fiction, picture books, etc. proceeding with floating collections required standardization both of home locations in use and physical labeling practices to ensure that all circulating items could float and be shelved anywhere in the system. the team reviewed each home location and the number of items and service points using it to determine whether it should be retained, eliminated, or merged with another home location. changes resulting from the review of home locations included creating modifiers for genre fiction (e.g., ficromance and ficmystery) and merging the english as a second language and literacy collections into a new home location, literacy and english language learning. standardizing children’s collections was challenging due both to collection size and the number of home locations in use. major changes in children’s home locations included renaming the i‐can‐read collection to the more inclusive easy readers, and eliminating the fairytales location to shelve these items with children’s picture books. new home locations were created for collections such as graphic novels for adults and children which had previously been included in adult fiction and juvenile fiction, respectively. the new home locations distinguished the graphic novels as discrete collections that could be shelved separately. epl implemented floating in stages. while the examination of the collection was underway, epl began floating small collections of unique materials, initially bestsellers, then videocassettes, through a range of formats until all adult and juvenile collections were floating. as new collections were added they floated immediately; video games launched as a floating collection when epl began circulating them in december 2008. over a five–year period, epl implemented floating across its entire circulating collection of 1.6 million items. the only circulating items that do not float are a local history collection, periodicals and some government publications, and a small number of reference titles. outcome overall, epl’s implementation of floating has been successful. floating elicited little direct response from customers, and did not result in an appreciable number of floating‐related customer concerns. more specifically, several desired outcomes were achieved, including: 1. a marked reduction of material in transit at the same time as epl was experiencing a steady increase in circulation (see figure 1); 2. a 68% increase in customer holds since 2008 (although floating collections is likely only one contributor to this trend) (see figure 1); evidence based library and information practice 2012, 7.1 68 3. greatly standardized and centralized acquisitions and processing (necessitated by floating); and 4. more immediate delivery of new acquisitions to customers. epl continues to address a number of more challenging outcomes that have also resulted from floating collections, including: 1. uneven distribution of materials among and within branches (e.g., a small branch might have a disproportionate number of mystery novels because of one customer’s borrowing pattern); 2. variation in the application of weeding guidelines between branches; and 3. the implications for staff of the shift to floating, particularly changes to workload and workflow. epl has adopted collectionhq software (http://www.collectionhq.com) as one tool to help with the first two challenges. prior to collectionhq epl used a locally developed web tool, the “floating dating service”, to manage the redistribution of items from crowded locations to locations that lacked certain types of materials. with floating collections, however, using this manual system proved untenable and adopting collection hq’s automated, evidence‐based approach became necessary. collectionhq allows intelligent redistribution of materials across the epl system. associated issues revealed by monitoring floating include: 1. determining the number of copies of a single title that are required/acceptable for a system of epl’s size; 2. using book‐lease programs as an alternative to purchase (which epl is now using for some bestseller titles); and 3. educating staff of the importance of not adjusting assigned home locations. in terms of the cultural change inherent in floating, some staff perceived floating as a threat to the careful development of local collections. while most staff have embraced the “one library” philosophy embodied in floating, others remain tied to the older, more established vision of branch collections and retain a strong sense of ownership of “their” collections. 50.000 250.000 450.000 650.000 850.000 1.050.000 1.250.000 ja n 0 6 a p r 06 ju l 0 6 o kt 0 6 ja n 0 7 a p r 07 ju l 0 7 o kt 0 7 ja n 0 8 a p r 08 ju l 0 8 o kt 0 8 ja n 0 9 a p r 09 ju l 0 9 o kt 0 9 ja n 1 0 a p r 10 ju l 1 0 o kt 1 0 circulations transits holds figure 1 circulation, transited items, and holds, january 2004‐october 2010. evidence based library and information practice 2012, 7.1 69 reflection it was immediately apparent that floating would reduce materials handling. as noted previously, almost immediately upon implementation of floating came the realization that rebalancing collections among service points would be an issue. director’s station helped us understand the extent of each service point’s collection. however, having collection hq in place prior to floating would have been helpful in identifying items for withdrawal through its reporting capabilities. performing an inventory of the entire collection prior to floating would also have been helpful, giving us a better understanding of our “true” collection. a clean‐up was essential prior to implementation to rid the collection of damaged, outdated, “grubby,” and duplicate items. because floating began with smaller collections, the process of preparing collections for floating became more routine by the time larger collections were floated. epl proceeded with floating despite obstacles and delayed implementation of floating for some collections, choosing not to wait until everything was perfect before launching. although the transition to floating took place gradually, the collection standards team found communicating the concepts and intentions of floating to staff to be more challenging than anticipated. even now staff struggle with inconsistencies relating to floating, weeding, and other aspects of collection management at a branch level. an established rebalancing plan at the outset and greater time preparing collections prior to implementation would have made the transition easier. the collection standards team continues to monitor, refine, and simplify (where possible) these processes, and address inconsistencies when they are identified; we expect this to be an ongoing team focus. locally‐relevant collections now take shape more organically at each service point; content, allocated shelf space, and shelf location change according to local use. collections now strongly support the community‐led service philosophy by reflecting individual community needs; items float to the service points with the greatest demand for them. for example, materials in russian gravitate to libraries serving communities with higher proportions of customers using russian materials. epl also integrated roving customer service at approximately the same time as floating in order to assist customers at their points of need, i.e., in the stacks, and can explain the changes associated with floating collections to customers, help them to find their desired materials, and provide guidance and instruction in using the catalogue to locate and place holds on items. floating has proved a successful initiative at epl, and the results, both expected and unexpected, have been worth the work involved. references canada, statistics canada. (2006). community profiles: city of edmonton. retrieved february 10, 2012 from http://www12.statcan.gc.ca/census‐ recensement/2006/dp‐pd/prof/92‐ 591/index.cfm?lang=e. cress, a. (2004). the latest wave. library journal, 129(16), 48‐50. edmonton, election and census services. (2009). municipal census. retrieved february 10, 2012 from http://www.edmonton.ca/city_govern ment/municipal‐census.aspx. sarasota county libraries. (2009). floating collection information. retrieved february 10, 2012 from http://www.sclibs.net/floating.aspx. http://www12.statcan.gc.ca/census-recensement/2006/dp-pd/prof/92-591/index.cfm?lang=e� http://www12.statcan.gc.ca/census-recensement/2006/dp-pd/prof/92-591/index.cfm?lang=e� http://www12.statcan.gc.ca/census-recensement/2006/dp-pd/prof/92-591/index.cfm?lang=e� http://www.edmonton.ca/city_government/municipal-census.aspx� http://www.edmonton.ca/city_government/municipal-census.aspx� http://www.sclibs.net/floating.aspx� / evidence based library and information practice microsoft word ed_4457 evidence based library and information practice 2008, 3:4 1 evidence based library and information practice editorial moving on lindsay glynn editor-in-chief acting head, public services, health sciences library memorial university of newfoundland st. john’s, newfoundland, canada e-mail: lglynn@mun.ca © 2008 glynn. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. well, here we are with another full volume published. i have watched this journal grow from an idea to a sustainable, reputable journal with readers and contributors from all over the world. it has grown from a small editorial group to a collection of peer reviewers, evidence summary writers, copyeditors, etc., that is eighty-four strong (and counting). i have wavered between wondering whether promoting evidence based librarianship is akin to flogging a dead horse to feeling secure in knowing that we are making a difference. being involved with this journal has made me look at what i do in a different light and i now approach decisions and change with what i refer to as “structured flexibility”. following the ebl framework is the structured part, and when it works, it works well. but we all know that there is not always an answer in the literature nor is there a guarantee that implemented evidence based change will work similarly in different environments. that’s where the flexibility comes in. three years as editor-in-chief has been a challenging, enjoyable, time-consuming, and fascinating learning experience. it has provided me with numerous opportunities in terms of speaking engagements, workshop offerings, and valuable discussion and discourse. it has also provided me with research and project ideas that i have had to place on the back burner until a time when i have enough hours in the day. recognizing that adding additional hours to the day is, well, impossible, i have decided to step down from my editorial role to pursue other activities. this was a bittersweet decision, but a necessary one. i am pleased to announce that denise koufogiannakis will be taking on the role of editor-in-chief for the next term. denise, as many of you know, has also been involved with this journal since its inception and, after a brief period of reduced involvement, has eagerly stepped up to the plate. this journal would not be what it is today without denise’s continued dedication and passion. you’re in good hands. evidence based library and information practice 2008, 3:4 2 i look forward to watching this journal continue to grow and to contributing to it as a peer reviewer. thank you to the entire editorial team for providing me with three years of collaborative teamwork and a most memorable, meaningful professional experience. i am grateful for all your valuable time, effort, and expertise. best of luck to you all, and i’ll see you at eblip5. (you didn’t think i’d miss sweden, did you…?) microsoft word comm1_1491_final evidence based library and information practice 2008, 3:1  82 evidence based library and information practice       commentary     reality through evidence      lorraine busby  university librarian  memorial university of newfoundland  st. john’s, newfoundland  canada  e‐mail: lbusby@mun.ca      received: 03 march 2008    accepted: 03 march 2008       © 2008 busby. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.       “perception is reality” ... or is it?  this famous  saying frequently is used to justify whatever  course of action we undertake, will  undertake, or have already undertaken.   when perception is reality, “i” am never  wrong, “i” am always right.  never doubt  the importance of the “i”.  or to phrase it  another way, “it’s all about me”.  behind the  importance of being “right” are a range of  emotions, facts, uncertainties, views,  evidence, history and analysis.  for many  library staff, being “right” is a passion.  on  issues that seemingly should have any  number of acceptable answers, pitched  battles are waged to ensure that all are  swayed to the obvious correct answer.    the motivations are the best!  the banner of  the user is waved high ‐‐ our users are  always first and foremost in guiding our  decisions!  yet our users’ perception of a  google searchable free web crash everyday  into the library’s efforts to spread  information literacy throughout the campus  and the community.  a librarian’s  background, experience, training, and  knowledge of controlled and structured  information come second to the uninformed  but free and easy alternative.  perhaps  reality is more accurately reflected in a  different saying: “ignorance is bliss”.      in the academic world we participate in an  environment that elevates research  methodology and processes to a pedestal  that is under‐pinned with peer‐reviewed  validation using universally accepted  standards.  librarians are, by training and  inclination, service oriented rather than  research oriented.  those of us truly  interested in exploring the profession from a  research‐based perspective usually register  for and complete a phd program.  there is  evidence based library and information practice 2008, 3:1  83 no better preparation for research than the  phd process.  most librarians, however,  have not chosen to follow that route.   without the credibility of a phd we need to  ensure that our professional practice  activities are based on more than just  perceptions and assumptions.  just as  clinicians ensure that appropriate patient  care follows from evidence based decisions,  so too must those working in the  information world ensure that professional  practice activities are grounded in  appropriate evidence‐based decisions.    yet, in many ways academic librarians  operate at a daily disadvantage.  we are  partners in the educational process but  fulfill an intermediary role.  this role is  critical to educational successes but it is a  role that is anchored in the environment in  which we operate, and therefore dependent  upon others.  between students’ desire for  information that is “good enough” to meet  the requirements of an assignment and our  faculty colleagues’ concern for pedagogical  control we must insert ourselves to achieve  both groups’ interests while upholding  library outcomes.  although our profession  is not regulated, and there are few standards  and benchmarks, we must somehow achieve  academic credibility.  within the profession  we bemoan the undervaluing of our skills  and expertise.  for our contributions to the  educational process to be recognized, we  must approach our work with an intellectual  rigor that will allow our efforts to be viewed  and appreciated on their inherent merits.  embracing an evidence based approach will  change our environment for the better.  by  adopting and promoting practices that  derive from thoughtful evaluation of the  evidence before us we move beyond  perceptions.  by consciously structuring our  efforts to identify needed inputs for  decisions we set the stage to achieve  meaningful outcomes.  if we believe we are  “right” then we should welcome the  evidence that will prove it so.  then it is our  responsibility to give back to the profession  by both creating data for others to utilize  and by reporting on our efforts to build  upon the contributions of colleagues.  perception can legitimately be a starting  point; reality is where we land. if we want  to control our destinies we have to not only  be open to the evidence before us but we  must seek out new interpretations of that  evidence.  a good starting point is to  recognize that “i” am correct only when the  evidence substantiates my position.  we  need to consciously seek to inform our  decisions with the best possible information  and analysis available.  “it is not hard to  learn more.  what is hard is to unlearn when  you discover yourself wrong.”i  as we  undertake a journey through the profession  let us do it with a willingness to learn from  evidence that is suitable for the issue.  let us  embrace an evidence based approach to  serving our users and learning more about  the work we do.     i attributed to martin h. fischer in numerous sources, including the february 2008 ama management  update (newsletter for ama members) and the croydon u3a inc sharing knowledge & skills webpage at  http://home.vicnet.net.au/~unita/timetable.htm (accessed february 25, 2008).  evidence based library and information practice 2012, 7.1 122 evidence based library and information practice evidence summary lack of training and a self-service environment leaves staff and users uncertain about health information in a public library setting a review of: harris, r., henwood, f., marshall, a., & burdett, s. (2010). “i'm not sure if that's what their job is." consumer health information and emerging "healthwork" roles in the public library. reference and user services quarterly, 49(3), 239-252. reviewed by: kate kelly librarian royal college of surgeons in ireland dublin 2, ireland email: katekelly@rcsi.ie received: 1 june 2011 accepted: 5 oct. 2011 2012 kelly. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to explore the role and expectations of patrons and staff about the role of a public library in enabling citizens’ “health information work.” this involves helping citizens take responsibility for their own health care by finding and using health information. design – case study. setting – a single, uk public library with a self-service delivery model based in a city centre in the spring of 2006. self-service also applies to reference services and is designed to “empower users to locate and use information on their own.” subjects – 202 library visitors who came to the library specifically to find health information completed a questionnaire, 15 of these visitors were later interviewed; 19 library staff (10 librarians, 6 library officers and 3 senior managers). methods – mixed quantitative and qualitative methods. a print questionnaire was administered to adult library users (age 18 and over). semi-structured interviews were conducted with fifteen randomly selected library users who completed the questionnaire, sixteen library staff who worked directly with library users, and with three librarians in senior management positions in the library. descriptive statistics were calculated from the questionnaire, recorded interviews were http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/� evidence based library and information practice 2012, 7.1 123 transcribed, and the text analyzed to identify recurring themes. main results – while all respondents came to the library to seek health information results from the questionnaire found that half (50%) of respondents came to the library to look for information on a specific health problem; 37% of respondents had tried finding information elsewhere before coming to the library; 40% usually searched the internet when they needed health information or advice although only 32% reported trusting the internet somewhat or completely; 67% intended to borrow books; only 4% indicated that they had planned to ask library staff for help; and 59% reported finding what they were looking for by themselves. results from the interviews found users, front line staff, and managers in general agreement about the role of the library as a starting point for health information, and that the library was a neutral and non-threatening environment. there was also agreement among the three groups interviewed that the public library fills a gap when health care providers, particularly doctors, are unable to meet the information needs of some of their patients. library staff were concerned about interpreting information as well as the impact of a self-service philosophy on the quality and length of interactions with users, and seemed unclear about their role in relation to health information provision. library staff had no training in supporting health information and limited or no knowledge of authoritative online health resources and how to use them, and their approach to internet searching was similar to users. this lack of training and expertise appeared obvious to library users. users did not identify interpretation of information by librarians as an issue but did reference the impact of self-service and the internet on the role and morale of the library staff. neither library users nor library staff identified librarians as a resource to be used when seeking health information. the value of the library for users was the book collection and they saw the library as second only to physicians as a source of trustworthy information. conclusion – uncertainty about the role of librarians in health information provision was evinced by both librarians and library users. both groups were also uncertain about the relationship between self-service and technology, and the way in which librarians and their work are almost invisible. health policies emphasize personal responsibility for health yet individuals are not enabled to find answers to their questions. the absence of health knowledgeable front line staff in public libraries is “worrisome.” the obvious trust users have in the library suggests that efforts to develop consumer health information in these settings continue to be a worthwhile response to the “pressures on citizens to take responsibility for their health”. commentary as this is an exploratory case study the findings cannot be generalized to all public libraries. the policy context of increasing personal responsibility for health care is well made as is the difficulty lay searchers experience using the internet to find health information. the interim between when the research was conducted and its publication has seen pertinent publications on the topic of consumer health and public libraries published both in the u.s. and the u.k. it was surprising, though, not to see reference in the discussion or literature review to older studies and guidelines such as the health link feasibility study (2005) which specifically references the potential of uk public libraries in health information provision or the 2001 ala reference service guidelines for handling medical queries. the lack of knowledge of staff and management in the library under study about issues surrounding the provision of healthcare information in public libraries is indeed “worrisome.” consumer health information delivered via public libraries or in partnership with health sciences libraries is not a new idea and there are resources targeted at librarians available through professional associations evidence based library and information practice 2012, 7.1 124 and publications (see reference list.) in highlighting the uncertainty of staff and their concerns around interpreting health information, the article mostly reinforces what is already known about health information delivery in a public library setting. the most striking theme in this article is the invisibility of librarians in the library studied and the effect of pursuing a self-service delivery model. in this reviewer’s opinion it is the uncovering of the impact of the self-service model on the quality of interactions which gives value to this article and which may give pause for thought if the public library is to play a role in enabling citizens to take responsibility for their own health care. references american association for the advancement of science, aaas. (2010). the challenge of providing consumer health information services in public libraries. retrieved 13 feb. 2012 from http://www.healthlit.org/pdfs/aaasfi nal.pdf baker, l. m., & manbeck, v. (2002). consumer health information for public librarians. lanham, md: scarecrow press. business reference and services section, reference and user services association. (2001). guidelines for medical, legal, and business responses. retrieved 13 feb. 2012 from http://www.ala.org/rusa/resources/gui delines/guidelinesmedical casini, b., & kenyon, a. (2002). the public librarian's guide to providing consumer health information. chicago: public library association. consumer and patient health information section (caphis) of the medical library association. consumer health information: a selected bibliography. retrieved 15 feb. 2012 from http://caphis.mlanet.org/chis/bibliogra phy.html health link. (2005). feasibility study information and support for patient choice and the public library service. retrieved 13 feb. 2012 from http://www.healthlinklibraries.co.uk/r esources_references.asp smith, s., & duman, m. (2009). the state of consumer health information: an overview. health information & libraries journal, 26(4), 260–278. / evidence based library and information practice baker, l. m., & manbeck, v. (2002). consumer health information for public librarians. lanham, md: scarecrow press. business reference and services section, reference and user services association. (2001). guidelines for medical, legal, and business responses. retrieved 13 feb. 2012 from http://www.ala.org/rusa/resources/guidelines/guidelinesmedical casini, b., & kenyon, a. (2002). the public librarian's guide to providing consumer health information. chicago: public library association. smith, s., & duman, m. (2009). the state of consumer health information: an overview. health information & libraries journal, 26(4), 260–278. evidence summary   conceptualizing practical aspects of public library initiatives provides a useful model for future research   a review of: sung, h. y., hepworth, m., & ragsdell, g. (2013). investigating essential elements of community engagement in public libraries: an exploratory qualitative study. journal of librarianship and information science 45(3), 206-218. doi: 10.1177/0961000612448205   reviewed by: sara sharun campus librarian, penticton okanagan college library penticton, british columbia, canada email: ssharun@okanagan.bc.ca   received: 01 sep. 2015   accepted: 2 nov. 2015      2015 sharun. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   objective – to examine characteristics of a community engagement (ce) initiative in a public library in order to identify and describe essential elements of ce and develop a model for ce in public libraries.   design – case study.   setting – a public library in a mid-sized city in england, united kingdom.   subjects – an unspecified number of community members, library staff, and external agency staff participating in a community news program.   methods – there were 12 semi-structured interviews conducted with library staff, community members, and participants from 12 community news agencies operating under the umbrella of a larger community news organization. the authors directly observed an unknown number of undescribed program meetings and events. they also performed document analysis on unspecified government policies, media reports, and program publications to corroborate the information gathered from their interviews and observations.   main results – the data were analyzed in an inductive manner using atlas qualitative data analysis software. results are described in a qualitative manner and do not correspond directly to the individual methods used. the coding of data from interviews and observations (which are not analyzed separately) resulted in the identification of seven themes related to community engagement in libraries: belonging, commitment, communication, flexibility, genuineness, relevance, and sustainability.   conclusion – the “essential elements” model that was developed from this case study can be applied by other public libraries in their processes and practices, and can contribute to the creation of a more genuinely community-driven approach to service.   commentary   this study addresses a gap in the literature on libraries and community engagement (ce), a concept for community development that has been emphasized by government agencies in the united kingdom. in its attempt to more clearly define ce in a library context and develop a useful model for understanding ce, the authors contribute to the future development of “wider, deeper and stronger levels of ce in library services” (p. 215). while this is an exploratory case study that does not present measurable outputs for ce that would qualify as evidence, the authors provide a means of gathering future evidence by identifying measurable elements for library practitioners to apply to programs and services. future studies may use this “essential element” model, further define these elements and develop measurements for them, which would allow librarians to gather evidence of the value and significance of ce in their libraries.   this study meets many of the criteria described in greenhalgh’s (1997) critical appraisal checklist for evaluation of qualitative research. in particular, the authors are clear about the study’s purpose and the need to address a significant gap in the lis literature on conceptualization of community engagement practices. the results of this qualitative study are credible and significant for library and information practice. the authors have clearly identified the fundamental characteristics that impact ce, setting the stage for future studies to work more closely with those characteristics.   a qualitative approach was appropriate for the question the authors are trying to answer, which requires definition and description rather than measurement. the authors made efforts to ensure quality control and maintain reliability and validity, and one of the study’s strengths is in its attempt to triangulate multiple research methods and data collection methods. however, its usefulness for library practitioners and researchers is limited by the lack of information presented about study design, data collection, and population, and by the presentation of information in tables and figures that are difficult to interpret. a more detailed description of the population, study design, and results, as well as a more explicit connection between the methods used and the findings would have improved the authors’ argument for their study’s validity and reliability, and made it possible for others to replicate and build upon this study.   the model presented in this article provides a framework for gathering evidence related to ce and making arguments for the value of authentic community engagement in libraries. qualitative studies like this one remind lis researchers that conceptualization of practices and theoretical approaches to research contribute to the field and provide a strong foundation for future research, both qualitative and quantitative.   the authors of this case study offer a suggestion that libraries are part of a larger community and a larger information ecosystem. this study suggests that positive reception among community participants and collaborators results when libraries take a grassroots, community-led approach, rather than a top-down, library-centric approach to engagement.   reference   greenhalgh, t. & taylor, r. (1997). how to read a paper: papers that go beyond numbers (quantitative research). bmj, 315(7110), 740-743.   evidence based library and information practice 2012, 7.3 80 evidence based library and information practice evidence summary relationships between librarians and faculty still need further investigation a review of: phelps, s. f., & campbell, n. (2012). commitment and trust in librarian-faculty relationships: a systematic review of the literature. the journal of academic librarianship, 38(1), 13-19. doi:10.1016/j.acalib.2011.11.003 reviewed by: giovanna badia liaison librarian mcgill university, schulich library of science and engineering montreal, quebec, canada email: giovanna.badia@mcgill.ca received: 3 june 2012 accepted: 27 july 2012 2012 badia. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. objective – to examine how the key mediating variable (kmv) model of morgan and hunt’s commitment-trust theory of relationship marketing can be used to look at the relationships between librarians and faculty as reported in the literature. relationship marketing stresses customer retention and long-term customer relationships, rather than focusing on the product. to also identify: 1) the methods reported in the literature to evaluate relationships between librarians and faculty; 2) the elements reported in the literature that lead to commitment and trust in librarian-faculty relationships; and 3) the elements reported in the literature that prevent commitment and trust in librarianfaculty relationships. design – a systematic review. setting – a university in the united states. subjects – 304 journal articles on librarianfaculty relationships were read and analyzed for variables included in the kmv model of relationship marketing. methods – the authors searched 20 databases to find publications in various disciplines. their search strategy included, but was not limited, to the following keywords: faculty, librarian*, relationships, library users, information professionals, liaisons, academic, university, college*, collaboration, and perceptions. they initially selected 389 references based on the occurrence of search terms in the title or abstract, as well as the mailto:giovanna.badia@mcgill.ca evidence based library and information practice 2012, 7.3 81 presence of related subject headings. the authors then read the abstracts and included/excluded references based on the following criteria: inclusion criteria: academic libraries or special libraries. english language, any instance of collaboration or cooperation, subject term or mention of relationship, the words trust or commitment or antecedents or outcomes from the model included in the abstract. exclusion criteria: blogs, books, emails, or any article that could not meet the subject inclusion criteria (p. 14). additional articles were identified by scanning the bibliographies of the articles selected at the abstract stage, searching the directory of open access journals (doaj) and google scholar, as well as conducting a cited reference search in web of science and google scholar. among the 304 journal articles that the authors selected, read and analyzed, only 13 of these satisfied the last inclusion criteria of the systematic review in that they contained “a high presence of the kmv model’s antecedents and outcomes” (p. 15). many articles concentrated on a service or project that librarians worked on with faculty and did not discuss the librarian-faculty relationship. main results – 77 out of the 304 analyzed articles discussed research methodologies. the methods used in these articles to evaluate relationships between librarians and faculty were: surveys (53%); literature reviews (26%); interviews (18%); and focus groups (5%). the 13 articles containing variables from the kmv model indicated the following positive antecedents as leading to commitment and trust in librarian-faculty relationships: communication (7/13 articles); shared values (7/13 articles); and relationship benefits (7/13 articles). the negative antecedent that hindered commitment and trust in librarianfaculty relationships was reported as opportunistic behavior in 4 articles (e.g., librarians seen as having an ulterior motive when they market their services to faculty). cooperation (12/13 articles); functional conflict (2/13 articles); and uncertainty, i.e., faculty uncertain about the teaching ability of librarians (2/13 articles), were found to be the outcomes of relationships between librarians and faculty. conclusion – the authors found that “a focus on communication, shared values and benefits of the relationship would build stronger ties and foster commitment and trust with teaching faculty” (p. 17). whereas the literature shows that collaborations between librarians and faculty are important to librarians’ work, very few studies have actually examined the librarian-faculty relationship. future studies should explore in-depth the basics of relationship building between librarians and faculty. commentary this study is unique in that it combined the kmv model of relationship marketing and the research methodology of systematic reviews to answer questions about librarian-faculty relationships. the authors described relationship marketing and summarize literature that deems it appropriate for use in libraries. they used the kmv model to look at the relationships between librarians and faculty since it provided a previously tested framework for their investigation. questions from lindsay glynn’s eblip critical appraisal checklist (2006) were used to help determine the strengths and weaknesses of this study. the strengths of this study lie in its systematic review of the published literature to identify studies about librarian-faculty relationships, and the authors’ analysis of their combined use of the systematic review process and kmv model to answer their research questions. a weakness of this study is the brief description of the search strategy used for the systematic review. the authors provide some of the terms they used in their search strategy, but they do not provide their complete strategy and indicate how the search terms evidence based library and information practice 2012, 7.3 82 were combined. this makes it difficult to duplicate the search, thus contradicting the authors’ statement that “systematic reviews … use a replicable search strategy” (p. 14). another weakness of the study, which the authors mention, is their combined use of the systematic review process and kmv model. journal articles identified from the literature search were coded using variables from the kmv model, thereby excluding from the authors’ analysis the elements from librarianfaculty relationships that did not fit into this model. the published literature also does not contain all antecedents and outcomes of librarian-faculty relationships, since some of these “are expected social norms of academia and therefore not always written about explicitly” (p. 17). notwithstanding its weaknesses, this study describes characteristics that librarians can adopt in their relationships, such as communicating regularly with faculty and concentrating on common values rather than on marketing library services, if they wish to establish new relations or strengthen existing ones with faculty. it also provides a lesson to librarians considering the use of a theoretical model to analyze data from a systematic review, i.e., refrain from using only a preexisting model for data analysis since it can potentially exclude findings that do not fit into the model, thereby biasing the conclusions of the systematic review. references glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi:10.1108/07378830610692154 evidence based library and information practice 2012, 7.1 1 evidence based library and information practice classics gertrude lamb’s pioneering concept of the clinical medical librarian a review of: lamb, g., jefferson, a., & white, c. (1975). and now, ‘clinical librarians’ on rounds. hartford hospital bulletin 30(2), 77-86. reviewed by: karla van kessel manager health sciences library, london health sciences centre london, ontario, canada email: karla.vankessel@lhsc.on.ca received: 9 sept. 2011 accepted: 1 feb 2012 2012 van kessel. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to determine if “the medical librarian with special skills and training in tested methods for approaching medical literature serve a valuable interface between the professional who is taking care of patients and the knowledge explosion in medicine wherein lies the key to better patient care” (p. 78). design – qualitative study involving the participant librarians keeping a reflective journal of all interactions with the subjects involved in the first 6 months of the study (september 1974 – march 1975). setting – hartford hospital, connecticut. subjects – teaching physicians, house staff, and medical students at hartford hospital. methods – this pilot project, funded by a twoyear grant from the u.s. public health service and the national library of medicine, placed three medical librarians (two full-time and one part-time) on rounds with pediatrics, medicine, and surgery teams. the librarians kept diaries to record “critical incidents” (p. 86), including the “acceptance of the program, its impact on patient care, its potential for changing the information seeking behavior of health professionals, and its usefulness for developing a core collection of clinical readings” (p. 86). main results – despite a few physicians’ initial apprehension, each of the three clinical librarians recorded indications of acceptance mailto:karla.vankessel@lhsc.on.ca� evidence based library and information practice 2012, 7.1 126 by clinical staff, including a dramatic increase in literature search requests; increased phone calls, drop-ins, pages, and requests for research assistance; and gestures of acceptance from house staff and students. more broadly, the literature searches in lamb’s report identifies direct patient care (including to “resolve a debate” (p. 84)), medical teaching/education, and searching techniques for clinicians. it is implied that these interactions resulted in a higher profile of the resources and services offered through the library; as one patron queried, “would you show me how to find articles and where everything is in the library sometime?” (p. 83). conclusions – the authors state that while their conclusions are only preliminary and no firm conclusions can be drawn, there are four observations of note: 1. the clinical librarian can be accepted as part of a patient care teaching team by contributing to educational activities. 2. the clinical librarian provides quick and useful information to assist in the decisions and management of patient problems. 3. there is an opportunity to strengthen and modify the information seeking behavior of the health professional. 4. as patient care questions recur, there is a need for a “patient care information system” which can be initiated and supported through the provision of photocopied articles (p. 86). commentary dr. gertrude lamb is credited with originating the concept of the “clinical librarian” (cimpl, 1985, p. 21). by identifying a gap between what medicine as a discipline knew about good patient care and the knowledge that was actually applied to the care of patients (arcari, 1977, p. 18), lamb saw an opportunity for librarians to be the connection during a time of “information explosion”(lamb, jefferson & white, 1975, p. 79). prior to this preliminary report, jama: journal of the american medical association announced the funding and goals for the project, entitling the short column “and now, ‘clinical librarians’ on rounds”. lamb uses the same title for this article crediting jama in her opening paragraph (lamb, jefferson & white, 1975, p. 77). lamb’s anecdotal, story-like style was a means of providing an update for the project in its early months. despite being written in 1976, many of the issues reported through the clinical librarians’ diaries resonate with hospital librarians today, such as the physician transition from being highly skeptical of a librarian’s value to realizing the value and benefit of having a librarian on the clinical team. the following excerpts capture the team dynamic often evident between physician and clinical librarian in today’s health care environment. dr. ___ made it clear that … this project would not be high on his list. he felt particularly strongly that the house staff should get most of their information from textbooks. if they were desperate, then they might ask the clinical librarian to help solve their problem. (lamb, jefferson & white, 1975, p. 79) i got a frantic call from dr. ___ just before 9 a.m. he had a very sick patient and wanted to know what i could find on current treatment for lymphangitic cancer of the lung. i did a medline search, reviewed the citations, and selected four good articles. i photocopied these articles and delivered them to dr. ___ on my way to 10:00 a.m. rounds. (lamb, jefferson & white, 1975, p. 79) another clinical librarian reports her experience with acceptance on the clinical teams: evidence based library and information practice 2012, 7.1 127 a librarian working in the wards is a novelty. people accepted me at first because i was such a novelty. the first few days they made a point of explaining things to me – what rounds are, what terms means, etc. after the first week they stopped explaining and assumed i knew. i suppose that is a measure of acceptance. (lamb, jefferson & white, 1975, p. 80) it is of interest to note that the uptake of clinical librarian services greatly impacted the workload of the librarians: “at the end of the first month, the clinical librarian was working a six-day sixty-hour week. to save her health and sanity, she was assigned to a four-day, forty-hour week” (p. 81). this article is later followed-up with another report, “bridging the information gap” (1976), in which lamb reviews the hurdles and successes of the project, including a budgetary decision to forego the addition of another surgical resident in favor of keeping the clinical librarian. lamb did not capture or present any empirical data on this project. in one of lamb’s later articles, lamb (1982), compares the use of surveys to “beauty contests,” measuring popularity rather than actual impact (p. 4). lamb was directly involved in several early adopters of the clinical medical librarian (cml) design (algermissen, 1974, p. 358) and paved the way for other clinical librarian projects for decades to come (scura, 1981, pp. 50-52). other, more quantitative studies were later performed as a derivative of the work done by lamb and others (scura, 1981, pp. 5052) showing the efficiency and effectiveness of clinical librarian programs (davidoff, 2000, p. 996). despite many reviews, reports, and articles highlighting the benefits of clinical librarians as part of hospital patient care teams (scura, 1981, p. 50; barbour, 1986, p. 1921), lamb’s cml concept is often criticized for being too labor-intensive, expensive (demas, 1991, p. 17) and lacking sufficient evidence of impact on patient care (veenstra, 1992, p. 21). more recent research on cmls employ far more rigorous methods to examine cml value and effectiveness. one such systematic review on the effectiveness of clinical librarianship concludes that “there is some relatively strong evidence that [cml] programs have been well accepted and liked by most” (cimpl wagner, 2004, p. 31), while another systematic review challenges this notion by stating, “although it is widely accepted that c[m]ls are effective, this review has identified little evidence to support this”(winning, 2003, p. 19). three systematic reviews (brettle, 2010; winning, 2003; cimpl wagner, 2004) examining the effectiveness of cml programs all conclude that further, high quality research is required in order to more fully understand the impact of such services. criticism of the cml concept does not change the fact that lamb’s innovative project not only inspired further research on the topic, but also fundamentally changed the way many hospital librarians viewed their services and value in the health care setting. references algermissen, v. (1974). biomedical librarians in a patient care setting at the university of missouri-kansas city school of medicine. bulletin of the medical library association 62(4), 354358. and now, ‘clinical librarians’ on rounds. (1974). jama: the journal of the american medical association 230(4), 521. arcari, r., & lamb, g. (1977). the librarian in clinical care. the hospital medical staff 6(12), 18-23. barbour, g. l., & young, m.n. (1986). morning evidence based library and information practice 2012, 7.1 128 report. role of the clinical librarian. jama: the journal of the american medical association 255(14), 1921-2. brettle, a., maden-jenkins, m., anderson, l., mcnally, r., pratchett, t., tancock, j., webb, a. (2010). evaluating clinical librarian services: a systematic review. health information and libraries journal 28 3–22. cimpl, k. (1985). clinical medical librarianship: a review of the literature. bulletin of the medical library association 73(1), 21-8. cimpl wagner, k., & byrd, g.d. (2004). evaluating the effectiveness of clinical medical librarian programs: a systematic review of the literature. jama: the journal of the medical library association 92(1), 14-33. davidoff, f., & florance, v. (2000). the informationist: a new health profession? annals of internal medicine 132(12), 996-998. demas, j. m., & ludwig, l.t. (1991). clinical medical librarian: the last unicorn? bulletin of the medical library association 79(1), 17-27. lamb, g., jefferson, a., & white, c. (1975). and now, ‘clinical librarians’ on rounds. hartford hospital bulletin 30(2), 77-86. lamb, g. (1976). bridging the information gap. hospital libraries 1(10), 2-4. lamb, g. (1982). a decade of clinical librarianship. clinical librarian quarterly 1(1), 2-4. scura, g., & davidoff, f. (1981). case-related use of the medical literature. clinical librarian services for improving patient care. jama : the journal of the american medical association 245(1), 502. veenstra, r. j. (1992). clinical medical librarian impact on patient care: a one-year analysis. bulletin of the medical library association 80(1), 19-22. winning, m.a., & beverley, c.a. (2003). clinical librarianship: a systematic review of the literature. health information & libraries journal 20(s1), 10-21. / evidence based library and information practice evidence based library and information practice 2012, 7.1 5 evidence based library and information practice guest editorial eblip and public libraries pam ryan director, library services edmonton public library edmonton, alberta, canada email: pryan@epl.ca 2012 ryan. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. kudos to the evidence based library and information practice editorial team for planning this special issue focused on public libraries. this issue features research articles from lis faculty and public library practitioners on timely and important topics: the use of volunteers in public libraries; using customer experience data to inform service practice; the efficacy of a web‐based staff training program; a study on the contributions and value of public libraries; and an evidence based collection analysis process. all of the evidence summaries in this issue also focus on public library issues. public librarians may be less familiar with these critical appraisals of research articles, their use as another tool to facilitate knowledge translation from research, and as another source of evidence with which to make informed decisions. eblip is one area where librarians from every sector can work together, sharing a common interest in evidence based professional practice. eblip at its best puts aside sector silos and offers a broad perspective for our work in all library types. the eblip conference is inclusive of all library types and evidence based library and information practice has always included articles and evidence summaries from across the spectrum of academic, public, school, health, and other special libraries in its mission to provide a forum for librarians and information professionals to discover research that may contribute to decision making in professional practice. despite this welcome inclusion in eblip, public librarian participation is notably low. this mirrors the grim reality of low public librarian research and publication rates, as well as the small overall percentage of lis research articles about public library practice. the results of a content analysis study (penta, 2005) showed that over a four year period just 3% of article authors in north american lis journals were employed in public libraries. even in public library quarterly, only 14% of the authors were public librarians (penta, 2005). an earlier study that reviewed the state of research in north american lis journals about public libraries over a five year period showed that only 7% of lis research articles were public library oriented (hersberger, 2001). it isn’t surprising that public librarian research and publication rates are lower than that of mailto:pryan@epl.ca� http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/� evidence based library and information practice 2012, 7.1 6 their academic librarian and lis faculty colleagues. public librarians do not share the same research tradition, mandate, or requirement as our academic colleagues. however, the lis literature and overall evidence base of lis suffers from this lack of contribution from our public library practitioners and lack of information about our public library practice. eblip practitioners should be concerned about the overall quality and representativeness of the lis literature and should consider strategies that can help balance sector specific research and publication efforts. while public librarians do make significant use of local data to inform organizational decision making, further encouragement and opportunity is needed to take the additional step of publishing these evidence based approaches with the lis community. the “using evidence in practice” section of evidence based library and information practice is a good example of a forum that public librarians should consider for these types of publications. while the barriers to eblip are applicable across all sector types and are well documented (booth, 2011), specific attention is required to engage public librarians. orientation to eblip as a model of practice is needed first, and secondly, support and attention is needed to assist in overcoming the barriers to participation. now, more than ever, with fiscal pressures and societal changes challenging the value of our public libraries, we need a strong base of evidence upon which to draw support and inform evidence based practice and advocacy efforts. the evidence base needs increased contributions about public library practice and value from both lis faculty and practitioner‐researchers to ensure balance and relevance. while attention and commitment from organizations, associations, and practitioners are all required to implement change, the expertise and leadership from within our eblip community could also be harnessed to provide an engagement and support framework to further engage public librarians to our community. further opportunity and welcome into the eblip community could be encouraged through even small steps, such as ensuring there are public librarians on the conference organizing and local arrangement committees, and an eblip editorial position dedicated to seeking out and working with public librarians to encourage and support publication. the eblip community can further support our public library colleagues by working collaboratively on projects, such as helping to define the research questions of most pressing concern for public libraries. in my former position as an academic librarian, i often heard the opinion that eblip was perceived to be just for and about health librarians and health libraries. eblip has grown so much that we know this is no longer true, but there is still work to be done. the number of public librarians in our ranks is still discouraging. we need to work together, reach out and provide opportunities, and share our eblip experience to engage more of our public library colleagues. special thanks to the editorial team of evidence based library and information practice for providing this forum for publication and discussion. references booth, a. (2011). barriers and facilitators to evidence‐based library and information practice: an international perspective. perspectives in international librarianship, 2011(1). doi:10.5339/pil.2011.1 hersberger, j., & demas, c. (2001). the current state of public library research in select peer‐reviewed journals: 1996‐2000. north carolina libraries, 59(1), 10‐14. retrieved 28 feb. 2012 from http://www.ncl.ecu.edu/index.php/nc l/article/viewfile/288/306 penta, m., & mckenzie, p. (2005). the big gap remains: public librarians as authors in lis journals, 1999‐2003. public library quarterly, 24(1), 33‐46. doi:10.1300/j118v24n01_04 / evidence based library and information practice microsoft word es_1789 evidence based library and information practice 2008, 3:3 76 evidence based library and information practice evidence summary morning report presentation with literature search associated with decreased length of hospital stay a review of: banks, daniel e., runhus shi, donna f. timm, kerri ann christopher, david charles duggar, marianne comegys, and jerry mclarty. “decreased hospital length of stay associated with presentation of cases at morning report with librarian support.” journal of the medical library association 95.4 (oct. 2007): 381-87. reviewed by: jennie kelson library services manager buckinghamshire hospitals nhs trust amersham, united kingdom e-mail: jennie.kelson@buckshosp.nhs.uk received: 02 june 2008 accepted: 21 july 2008 © 2008 kelson. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – to determine whether hospital charges, length of stay or 30-day readmission rates were affected by case discussion at residents’ morning report (mr), accompanied by librarian-provided literature search results, within 24 hours of admission. design – case-control study and survey. setting – louisiana state university school of medicine. subjects – mr cases presented during the study period august 2004 to march 2005 matched with one to three cases (controls) of patients who were hospitalised in the period january 2000 to july 2005. house officers who presented the mr cases during the study period were the subjects for the survey. methods – mr cases were presented between 8:00am and 9:00am, five days a week, and only one case was presented at each mr. during the study period, august 2004 to march 2005, the presenting house officer selected independently a case for evidence based library and information practice 2008, 3:3 77 presentation at mr from patients admitted during the previous 24 hours. the selection was made without guidance as to which cases or illnesses to choose and without the knowledge or involvement of the attending physician or investigators. the term “house officer” is not defined in the article so it is not clear whether this refers to medical residents only, or if it includes interns. the faculty librarian, with clinical input from the chief resident or the chair of the department of medicine, conducted a search of the medical literature immediately following the mr to identify articles or citations that would answer two questions posed at the end of the presentation. the chair of the department or chief resident selected the articles that provided the most clinically sound answers to the questions. by 10:00 am, copies of the selected articles were hand-delivered to the presenting house officer and shared with members of the ward team. for the case-control component, the mr cases presented during the study period were matched against a comparison group of patients who were hospitalised in the period january 2000 to july 2005. matching of controls to the mr cases was made on the basis of the primary international classification of diseases (icd-9) diagnostic code, patient age and concomitant secondary diagnosis. mr cases with no matching control case were excluded from analysis of length of stay, costs or readmission rates. a maximum of three control cases were randomly selected where a mr case matched with more than three control cases. data regarding demographic information of the patient (age, sex, race, insurance coverage, marital status and number of diseases diagnosed), the length of stay, total hospitalisation charges and readmission rates within 30 days of initial discharge were extracted from the medical records of matched mr and control cases. statistical tests (student t or chi-squared tests) were used to compare differences between the demographic details of patients in the mr group and the control cases. wilcoxon signed rank test and sign test were used to analyse nonparametric data such as length of stay and hospital charges. median values, rather than the mean, were used for the outcome measures to reduce the influence of any extreme or outlier values. the researchers considered a p value less than p = 0.05 to be statistically significant. for the survey component of the study, each presenting house officer was asked to complete a questionnaire for each mr case they presented during the study period, which asked them to comment on the quality of the articles located in response to the questions posed at the mr and to say whether the information would influence their treatment of the patient. a copy of the questionnaire is accessible online via a link within the article; however it is not stated who analyzed the results or how this was done. main results – of the 105 cased presented at mr during the study period, 55 cases could be matched with at least one control up to a maximum of three cases, resulting in a total of 136 control cases. statistical analysis of the mr cases and the control cases showed no significant difference in the demographic details between the groups. mr cases had a median length of stay of three days compared to five days for control cases. this difference was statistically significant (p = 0.0238). a logarithmic plot comparing length of stay between mr cases and controls showed a positive association evidence based library and information practice 2008, 3:3 78 that was statistically significant (p = 0.012) between presentation at mr and a reduced median length of stay. median hospitalisation charges were $7,045 for the mr cases and $10,663 for the control cases; however, the difference was not statistically significant (p = 0.24). a logarithmic plot for total charges showed that, in most cases, charges for mr cases were lower than controls; however, the differences did not reach statistical significance (p = 0.18). readmission rates within thirty days of initial discharge were 16.4% for mr cases and 16.8% for controls. there was no statistically significant difference in readmission rates between the groups (p > 0.88). analysis of the survey responses found that the house officers commented on the quality of the articles for 60 of the 105 mr cases presented. in 43 cases, the house officer commented that the articles had a positive influence on patient management. comments for a further ten cases indicated that, although they did not alter the management of mr patient cases, the house officers believed that the articles provided good background information which may be helpful in the future. seven other comments indicated that the articles had not influenced their patient management in any respect. conclusion – presentation of cases at mr accompanied by dissemination of literature search results resulted in a statistically significant shortened median length of stay and lower hospital charges compared to matched control cases. readmission rates within 30 days of first diagnosis showed no differences between mr cases and control cases. supplementary survey results found that the 41% of the presenting house officers believed that the literature search information provided following the mr presentation positively influenced patient management. commentary this is an interesting study that attempts to quantify an issue of importance to all clinical librarians. one of the strengths of this study is the use of objective outcome measures, such as length of stay and readmission rates, which help demonstrate the contribution that clinical librarians can make to patient care. the authors chose to use a case-control study design. they did not discuss why they used this methodology in preference to alternatives, such as a prospective study design. a common problem with casecontrol studies is the difficulty in trying to account for all possible areas of variation between the control and intervention groups. as the control cases were drawn from in-patients admitted over a five-year period, there were many aspects that were difficult to define and outside the researchers’ control that, as they said, could have accounted for the variation between mr and control cases. these included the introduction of new therapies, more effective interventions or changes in medical practice. there may have been differences in the quality of care provided by the house officers, and year to year variability or seasonal differences in the date of admission could also have generated variability in patients and diseases. the researchers have made every effort to limit any differences. selection of mr cases was made independently thereby limiting selection bias, and mr and control cases were closely matched by diagnosed conditions and demographic details. evidence based library and information practice 2008, 3:3 79 established statistical techniques were used to compare the outcomes between the groups and also for checking that the demographic characteristics of the groups were similar. it is therefore more likely that the outcomes reported are valid and that they are a result of the mr intervention. owing to the complexity of the cases and differing co-morbidities, the researchers had difficulty matching controls to the mr cases and were unable to match 50 of the 105 mr cases despite having access to over 19,000 potential control cases. the researchers do suggest that a greater statistical power (i.e., more matched cases) may have allowed them to detect a significant effect on charges, although no power calculation was included in the paper. the method of identifying the articles was not described in detail, only that they were “identified as providing the most clinically sound answers to the questions” (382) by the chair of the department of medicine or the chief resident. no criteria were given on which the choice of articles was based. it is helpful that a copy of the questionnaire is included in the supplementary online content to the article. however it is not clear who created or distributed the questionnaire, although it seems to be implied that it was the librarians based at the school of medicine where the study took place. it is also unclear who analysed the survey data or the basis on how this was done. as the researchers have pointed out, the house officers may not have acted upon nor even read the literature research results and there was no compulsion for them to do so. comments on the quality of the literature results were made for 57% of the mr cases; however, we do not know if it was the same house officers who responded or indeed, how many house officers were surveyed. the researchers have used an interesting approach (case control and survey) to investigate whether mr with librarian support makes a significant difference to patient care. as such, it may be worth continuing the study using a longitudinal or prospective study design. the intervention (mr with librarian support) was studied only as to whether it affects clinical outcomes for those presented at mr, which is a relatively small proportion of cases, and these might not be representative of the population. as suggested by the survey results, this type of intervention may have more overall effect on physicians’ knowledge and skills, but the article does not assess this. it is disappointing that the use of the literature search results were not reported more fully in the article. although outside the remit of this study, a further study to investigate the relative contributions of the mr presentation and the literature search conducted by the librarian in affecting clinical outcomes would be of interest. evidence summary   evaluation of self-ratings for health information behaviour skills requires more heterogeneous sample, but finds that public library print collections and health information literacy of librarians needs improvement   a review of: yi, y. j. (2015). consumer health information behavior in public libraries: a qualitative study. the library quarterly: information, community, policy, 85(1), 45-63. http://dx.doi.org/10.1086/679025   reviewed by: carol perryman assistant professor texas woman’s university denton, texas, united states of america email: cp1757@gmail.com   received: 4 dec. 2015     accepted: 3 feb. 2016      2016 perryman. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to understand public library users’ perceptions of ability to locate, evaluate, and use health information; to identify barriers experienced in finding and using health information; and to compare self-ratings of skills to an administered instrument.   design – mixed methods.   setting – main library and two branches of one public library system in florida.   subjects – 20 adult library users purposively selected from 131 voluntary respondents to a previously conducted survey (yi, 2014) based on age range, ethnicity, gender, and educational level. of the 20, 13 were female; 11 white, 8 black, 1 native american; most had attained college or graduate school education levels (9 each), with 2 having graduated from high school. 15 respondents were aged 45 or older.   methods – intensive interviews conducted between april and may 2011 used critical incident technique to inquire about a recalled health situation. participants responded to questions about skill self-appraisal, health situation severity, information seeking and assessment behaviour, use of information, barriers, and outcome. responses were compared to results of the short form of the test of functional health literacy in adults (s-tofhla) test, administered to participants.   main results – on a scale of 100, participants’ s-tofhla scores measured at high levels of proficiency, with 90% rating 90 points or above. self-ratings of ability to find health information related to recalled need were ”excellent” (12 participants) or “good” (8 participants). fourteen participants did not seek library assistance; 12 began their search on the internet, 5 searched the library catalogue, and 3 reported going directly to the collection. resource preferences were discussed, although no frequency descriptions were provided. 90% of participants self-rated their ability to evaluate the quality of health information as “good” or “excellent.” participants selected authority, accuracy, and currency as the most important criteria of quality evaluation; however, other important criteria such as editorial review of content were not mentioned. participants rated their ability to use health information as either “excellent” (17) or “good” (3).   conclusion – use of health information enabled health behaviour change for participants, although conflicting information tended to increase anxiety. barriers to success in all areas of inquiry include difficulties with terminology, collection limitations, asking a librarian for assistance, and lack of awareness of resources. librarians should improve their health literacy skills in order to advise on all aspects of health information seeking, evaluation, and use. collaborative efforts are suggested, such as special libraries and public library efforts, and health professional workshops or seminars offered to public library patrons.   commentary   health information behaviour including seeking, appraisal, and use by consumers has long been a topic of study. the author places the current work within this body of knowledge, identifying gaps with regard to public library patrons, their own self-appraisal of skills and reported experience.   two instruments aided critique of the work (glynn, 2006; perryman, 2009).   use of the s-tofhla test and critical incident technique were appropriate as methods to examine the phenomena in question. reporting of more affective responses to the search, appraisal, and use process helped to illustrate and refine findings.   while the author provides research context, no mention is made of the dissertation from which this work is derived. pertinent research not mentioned includes vakkari, savolainen, and dervin (1997), whose findings on trust, source preference, and individual context informed the dissertation. as well, huber, boorkman, and blackwell (2008) mention health information seeker preference for information from peers with similar conditions, echoing yi’s finding.   in several instances readers are not informed sufficiently to completely evaluate the method in use or the analysis of results. interview questions are missing, and no time frame for participants’ recalled situation is provided. recall discrepancies could limit the precision of detail, a concern recognized by the author.   participants self-rated their abilities to find, evaluate, and use health information. with regard to finding information, none rated their skills as poor or even average, and all considered themselves good or excellent. no definition of these terms was provided. descriptive statistics about source preferences and frustrations or limitations would have also aided understanding and lent strength to recommendations.   participants felt librarians lacked health information skills, and felt their information needs were more specific than those public librarians were prepared to support. yet the author concludes with a suggestion that support groups and workshops would help. these may be of limited value to the respondents in this sample.     the author recognized the limitations of the research due to uniformly high performance on the s-tofhla test, the self-selected, relatively homogeneous group, and issues of limits in recall. the study would have benefitted from inclusion of more extensive interview responses, particularly to illustrate findings about barriers and perceptions about the usefulness of librarians. with regard to the organization of the work, the author introduced new findings about participants’ reliance on information obtained from others with similar health conditions in the conclusion rather than earlier in the results section.   overall, the study offers a good example to follow in expanding future inquiries to larger samples and to participants whose skills are self-assessed or tested at lower levels. suggestions for improved health information literacy skills and awareness will be of interest to public librarians and those in other settings where the provision of health information to consumers is a primary focus.                                                                                                                                                                             references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154   huber, j., boorkman, j., & blackwell, j. (eds). (2008). introduction to reference sources in the health sciences. new york: neal-schuman.   perryman, c. (2009). critical appraisal tool for bibliometric studies. retrieved from https://www.dropbox.com/s/8wwl8waj7mqv6n2/critical%20appraisal%20tool%20-%20bibliometric%20study-1.doc?dl=0      vakkari, r., savolainen, r., & dervin, b. (eds.). (1997). information seeking in context: proceedings of an international conference on research in information needs, seeking and use in different contexts, 14-16 august, 1996, tampere, finland.  london: taylor graham.   yi, y. j. (2014). health literacy and health information behavior of florida public library users: a mixed methods study. journal of librarianship and information science, 47(1) 17–29. http://dx.doi.org/10.1177/0961000614531159       evidence summary   there is a relationship between resource expenditures and reference transactions in academic libraries   a review of: dubnjakovic, a. (2012). electronic resource expenditure and the decline in reference transaction statistics in academic libraries. journal of academic librarianship, 38(2), 94-100. doi: 10.1016/j.acalib.2012.01.001   reviewed by: annie m. hughes reference librarian wilson dental library, university of southern california los angeles, ca, united states of america email: amhughes@usc.edu   received: 29 nov. 2012 accepted: 8 feb. 2013      2013 hughes. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to provide an analysis of the impact of expenditures on electronic resources and gate counts on the increase or decrease in reference transactions.   design – analysis of results of existing survey data from the national center for educational statistics (nces) 2006 academic library survey (als).   setting – academic libraries in the united states.   subjects – 3925 academic library respondents.   methods – the author chose to use survey data collected from the 2006 als conducted by the nces. the survey included data on various topics related to academic libraries, but in the case of this study, the author chose to analyze three of the 193 variables included. the three variables: electronic books expenditure, computer hardware and software, and expenditures on bibliographic utilities, were combined into one variable called electronic resource expenditure. gate counts were also considered as a variable. electronic resource expenditure was also split as a variable into three groups: low, medium, and high. multiple regression analysis and general linear modeling, along with tests of reliability, were employed.   main results – the author determined that low, medium, and high spenders with regard to electronic resources exhibited differences in gate counts, and gate counts have an effect on reference transactions in any given week. gate counts tend to not have much of an effect on reference transactions for the higher spenders, and higher spenders tend to have a higher number of reference transactions overall. low spenders have lower gate counts and also a lower amount of reference transactions.   conclusion – the findings from this study show that academic libraries spending more on electronic resources also tend to have an increase with regard to reference transactions. the author also concludes that library spaces are no longer the determining factor with regard to number of reference transactions. spending more on electronic resources is also important to increase both in-person and electronic reference transactions.      commentary   in this study, the author chose to address a gap in the current research regarding electronic expenditure and its relation to an increase or decrease in reference transactions. according to the author, multivariate analysis is a rare choice with regard to studying this topic, and often, the sample size chosen is small. the article analyzes previously published survey data that include a larger sample size than most studies on the topic choose to address.   data from the als 2006 were utilized for this study, and variables related to electronic expenditure were analyzed. however, there is uncertainty with regard to how data were collected in the original study, and the author does not go into great detail or include the original survey instrument or the portions of the instrument used. total gate counts and total reference transactions used in the calculations were not included. more information is needed on how the variables from the als study were used to create the one encompassing variable called electronic resources expenditure.    the author split the gate count and expenditure variables into three categories (low, medium, and high) which allowed for a depiction that higher spenders tended toward more reference transactions and higher gate count also tended toward higher reference transactions, but low spenders and low gate count equated to lower rate of reference transactions. one problem is that it does not look as if the author considered confounding variables affecting reference transactions, such as outreach to the user population, level of instruction provided on use of electronic resources, and the physical space itself. level of expenditure may be correlated with a higher level of spending on librarian resources and therefore outreach to the community. resources might be more abundant in the higher spending category and therefore instruction may be more available to those using electronic resources. physical space may be more appealing as well in the higher spender category.    electronic resources play an important role with regard to a collection and its use. however, it is not clear from this study that electronic resource expenditures are the real reason for increase or decrease in reference transactions. perhaps if the author coded the original data provided by the als survey as to what type of reference transaction occurred (for example, are the questions regarding use of electronic resources?), then the impact of electronic resources on reference transactions could be accurately calculated. the split variables included in the study do provide an interesting analysis with regard to higher spenders versus lower spenders, concluding that higher spenders do tend to experience more volume with regard to transactions; however, it is difficult to conclude that an increase or decrease in transactions is wholly due to these factors.   a final limitation with regard to the study is that it does not include a discussion section where the author could comment on methodological issues or address any bias of the survey conducted.     references    glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi: 10.1108/07378830610692154   evidence based library and information practice evidence based library and information practice 2011, 6.4 158 evidence based library and information practice evidence summary assessing vocational development in prospective school librarians a review of: jones, s. (2010). the occupational choice of school librarians. library trends, 59(1-2), 166-187. reviewed by: gayle bogel assistant professor graduate school of education and allied professions, fairfield university fairfield, connecticut, united states of america email: gbogel@fairfield.edu received: 29 june 2011 accepted: 15 oct. 2011 2011 bogel. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to explore the motivations for choosing school librarianship as a career. design – narrative research, qualitative interviews. setting – the school media program specialization in a master of library science program at a large research university. subjects – five graduate school students seeking initial certification as school library media specialists. methods – the researcher employed narrative research, based on career construction theory and the life story interview, (mcadams, 1995). this methodology is a set of looselystructured, open-ended questions designed to encourage detailed, in-depth responses from the participant teller, combined with a more structured career style interview (savickas, 2005) designed to elicit self-defining stories. data was collected through qualitative interviews, using personal narrative interviews that focused on the entire life of each person from birth to present (life story interviews) and additional questions in an interview format (career style interview.) data was collected and analyzed in two stages. the analysis first examined participants as individuals then tried to identify commonalities among the stories. each life story was examined to determine a career style on the basis of vocational personality, career adaptability and life theme. the interview data was then analyzed for thematic connections and occupational choice. mailto:gbogel@fairfield.edu� evidence based library and information practice 2011, 6.4 159 main results – the final analysis identified the following themes as relevant to the study, and to the career choices of graduate students planning to be school librarians: parental expectations; career changers; librarian mentors; prior library work experiences; reading; library experiences; altruism-service; desire to work with children; financial stability and security; flexibility of work schedule; emotional distance; and vocational personality. in general, the participants appear to value safe, traditional career choices that allow them to put family first and maintain a flexible work schedule, while also providing financial stability and security. those who are already teachers noted interest in a job that provides relief and emotional distance from the everyday social problems of students. all five noted the “love of reading” as a motivation for choosing this career path. the vocational personality of each subject was determined through interpreting their answers on the career interest surveys, using holland occupational theme riasec codes: realistic, investigative, artistic, social, enterprising, and conventional. the vocational personality of all five students did not match the predominant code for school librarians. none exhibited a resemblance to the “enterprising” type, the most prominent personality for successful school librarians. all five did exhibit “artistic” vocational personality, and the researcher notes that this may be an explanation for the choice of librarianship as a second career, as artistic types take longer to find a compatible career match. the researcher also notes that the final code is a “best estimate” for each subject. conclusion – examining the vocational development of the five graduate students through vocational personality, career adaption and life theme afforded a broader viewpoint than traditional survey studies. the determining of vocational personality type may be helpful to recruitment efforts to the profession, and help make the career choice more visible. commentary the sample size and demographic composition of this study have a great effect on the validity of the results put forward by the researcher. while the limitation in sample is acknowledged, the researcher does not acknowledge that the bias in the results gives an inaccurate picture of those who are entering the field. although the researcher notes that the vocational personality of the five participants did not match the most prominent type for school librarians, she states that “every occupation has room for a wide range of personalities” (p. 176). she does not resolve that these participants are not representative of successful school librarians in the larger professional community as identified through interpretations in other studies and the rating scale. the rating scale itself clearly notes the enterprising personality type as the predominant trait for school librarians. there is a striking contrast between the graduate students’ perceptions of the field and the reality of current working situations in school libraries. none of the participants mentioned motivation related to interest or expertise in implementing technology, and seemed unaware that their career choice would demand rigorous training in technology skills. for library educators, this finding has enormous implications for helping pre-service librarians gain a clear understanding of the demands in the field, and ensuring that those who graduate are well equipped to implement current best practices that rely heavily on integrating technology and content in the curriculum. the “love of reading” that was prominently noted as a motivation for entering the field needs to be firmly linked to “love of technology” as consistent theme for pre-service school librarians. the participants did not view their career choice as a challenging or forward-thinking role in school reform (american association of school libraries [aasl], 2010), but relied on stereotypical views of libraries and educational environments as “safe” traditional evidence based library and information practice 2011, 6.4 160 choices. the finding that emotional distance was a motivating factor for classroom teachers who doubted their ability to maintain relationships with students is a troubling observation, and illustrates the misperceptions common to many educators that the work of school librarians is less demanding than classroom teaching. the implications for practice stated by the researcher for recruiting school librarians are based on the assumption that the personality traits of the study participants lead to successful careers. given the current emphasis on advocacy, leadership, and finely honed collaborative and technological skills (aasl, 2011; zmuda and harada, 2008), these graduate students did not appear to understand or possess the necessary dispositions for satisfaction with their career choices. a larger implication for library educators is that preparing new professionals may require significant focus on personal development and dispositions. while the overall approach of this research may yield helpful results for school library educators as they help pre-service school librarians develop the skills and dispositions to be successful, a follow up study with a closer look at successful school librarians would be a welcome balance to this otherwise well-designed study. references american association of school librarians. (2010). standards for the 21st century learner. retrieved june 4, 2011, from http://www.ala.org/ala/mgrps/divs/aa sl/guidelinesandstandards/learningsta ndards/standards.cfm american association of school librarians. (2011). a planning guide for empowering learners. retrieved june 4, 2011, from http://www.ala.org/ala/ mgrps/divs/aasl/guidelinesandstandar ds/planningguide/planningguide.cfm mcadams, d. p. (1995). the life story interview. retrieved may 29, 2007, from northwestern university school of education and social policy, foley center: http://www.sesp.northwestern.ed u/foley/instruments/interview/ savickas, m. l. (2005). the theory and practice of career construction. in s. brown & r. lent. (eds.), career development and counseling: putting theory and research to work (pp. 42–70). hoboken, nj: john wiley zmuda, a., & harada, v. h. (2008). librarians as learning specialists: meeting the learning imperative for the 21st century. westport, ct: libraries unlimited. / evidence based library and information practice microsoft word es_brown.doc evidence based library and information practice 2007, 2:1  144 evidence based library and information practice       evidence summary    interactive, web‐based information skills tutorial well received by graduate  students in health and social care research      a review of:  grant, maria j., and alison j. brettle. “developing and evaluating an interactive information  skills tutorial.” health information and libraries journal 23.2 (june 2006): 79‐86.    reviewed by:   marcy l. brown  clinical medical librarian  the western pennsylvania hospital, forbes regional campus  monroeville, pennsylvania, united states of america  e‐mail: wordsmith@alltel.net      received: 28 november 2006  accepted: 23 january 2007      © 2007 brown. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to determine whether a newly  developed interactive, web‐based tutorial  on ovid medline was acceptable to  students, and to identify whether the  tutorial improved students’ information  skills.    design – objective and subjective  assessment within a small cohort study.    setting – an evidence based practice  module within a masterʹs in research (mres)  program at the university of salford, uk.    subjects – a total of 13 usable evaluations  were received from graduate students who  took an evidence based practice module as  part of their mres coursework.  methods – information skills (is) were  taught in weeks two and three of a 12‐week  module on evidence based practice. each of  the two is sessions lasted approximately  three hours. at the beginning of the first  session, baseline skills were assessed by  asking the students to perform a literature  search on either the effectiveness of nursing  interventions for smoking cessation, or the  effectiveness of rehabilitation after stroke.  the ovid medline tutorial was  introduced at the first session, and guided  hands‐on practice was offered. homework  was given, and between‐session use of the  tutorial was encouraged. at the end of the  second session, students were asked to  complete another search in order to assess  short‐term impact of the tutorial. both sets  of search results were scored using a  checklist rubric that looked for boolean  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:1  145 operators, use of mesh terms, use of limits,  number and relevance of references, and  other assessment criteria. the rubric was a  modified version of a tool published by  rosenberg et al.     the tutorial remained available throughout  the 12‐week module, at which time a  systematic literature review was assigned in  order to measure longer‐term impact. as an  additional subjective measurement, a  questionnaire regarding the information  skills sessions and tutorial was given at the  end of the second is session (week 3).      main results – thirteen objective  assessments (literature search results) were  returned and usable. according to the  scored pre‐training search, two students  could use multiple search techniques  correctly and in a systematic manner. the  post‐training search results indicated that  six students could systematically search,  which is triple the original number. at the  end of the 12‐week ebp module, that  number had increased to seven students.  this demonstrated a significant difference  between pre‐training and post‐training  scores (p = 0.040), as well as a significant  difference between post‐training and post‐ module scores (p = 0.008).     eight of the subjective questionnaires, which  measured perceptions on a five‐point scale,  were returned. all responses indicated that  “the sessions were useful, well structured  and interesting” (83). seven of the eight  were entirely positive, either agreeing or  strongly agreeing with each of the eleven  questions about things such as search skill  improvement, information skills knowledge,  and confidence in searching. the small  sample size made it difficult to generalise  these results. ad hoc comments varied and  sometimes contradicted each other, such as  one request for simpler tutorial instructions  in contrast with the comment that the  “tutorial ‘couldn’t be simpler’” (84).   conclusions – students rated the is sessions  positively, including the web‐based  medline tutorial. search skills improved,  as was demonstrated by comparing pre‐ training search results with post‐training  and end‐of‐module searches. continuing  feedback indicates that the tutorial is used  within other departments and programs as a  standalone tutorial.    commentary    this study was evaluated using the  university of glasgow critical appraisal  checklist for educational intervention  studies. the study authors provided two  focused questions they intended to answer.  they also did a good job of describing the  learning need for an intervention:  “information skills (is) are essential for  healthcare students if they are to…become  qualified evidence based practitioners” (79).  the intervention and assessment measures  were explained in sufficient detail.     the biggest problem with the study is that  the research questions were specific to the  interactive web‐based tutorial, but it was  not possible to separate the effects of the  tutorial from the effects of the face‐to‐face  instructional sessions offered in conjunction  with the tutorial. even the subjective  questionnaire, a copy of which was offered  as an appendix to the published study,  asked most of its questions about the is  training in general rather than about the  web tutorial.     this shows that the single cohort study was  not the best study design to measure the  effects of the tutorial. comparing an  experimental group – one offered the is  sessions with the tutorial – with a control  group taking the is sessions without the  tutorial would have been a more effective  way to measure chosen outcomes. the  authors acknowledged the following  limitations: the limited ability to generalise  evidence based library and information practice 2007, 2:1  146 results due to the small number of students;  the lack of a control group; and the need to  test the tutorial with a larger group to see if  facilitator‐to‐student ratio affects results.    the authors stated, “findings indicate that  the use of this tutorial to complement taught  sessions … in conjunction with the  availability of guided feedback have a  greater impact on [information skills]  development” (85). it is unclear what  intervention has the lesser impact if this one  has the greater. unfortunately, without a  comparison intervention this statement  lacks credibility.    there is a preponderance of existing  literature documenting the “no significant  difference” phenomenon. a 2001 book by  thomas russell, and its ongoing companion  web site, provide an evidence based look at  outcomes of face‐to‐face versus distance  instruction and find no significant difference  in learner outcomes. this is not to say that  there are no studies which document a  difference in outcomes, but that factors  beyond the delivery medium usually  influence whether or not face‐to‐face or  distance delivery is more effective.     instructional librarians and developers of  information literacy tutorials need future  research identifying those characteristics  that make web‐based information literacy  instruction more or less effective. do  student characteristics come into play?  should researchers examine educational  background, gender or age in relation to  outcomes in web‐based instruction? do the  multimedia elements affect outcomes? how  should information literacy instruction be  “chunked” for maximum learning  effectiveness? stronger randomised  controlled trials, rather than single case  studies, are needed to fully explore these  and other related research questions.    works cited    department of general practice. critical  appraisal checklist for an article on  an educational intervention.”  university of glasgow, scotland. 8 feb.  2007  <http://www.gla.ac.uk/departments/ge neralpractice/educational.pdf>.    rosenberg, william m., jon deeks, anne  lusher, robin snowball, gordon  dooley, and david sackett.  “improving searching skills and  evidence retrieval.” journal of the  royal college of physicians of london  32.6 (nov. 1998): 557‐63.    russell, thomas. the no significant  difference phenomenon: a  comparative research annotated  bibliography on technology for  distance education. 5th ed.  montgomery, al: idecc, 2001.     http://www.gla.ac.uk/departments/ge evidence summary   seven years of noise reduction strategies in an academic library improve students’ perceptions of quiet space, especially among graduate students   a review of: mccaffrey, c. & breen, m. (2016). quiet in the library: an evidence-based approach to improving the student experience. portal: libraries and the academy, 16(4), 775-791.  http://doi.org/10.1353/pla.2016.0052       reviewed by: elaine sullo head, reference and instruction himmelfarb health sciences library the george washington university washington, district of columbia, united states of america email: elainej@gwu.edu   received: 31 aug. 2019                                                                 accepted:  18 oct. 2019      2019 sullo. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29637     abstract   objective – to examine the interventions implemented by an academic library for noise management, and their impact on library users, over a seven-year period.    design – retrospective data analysis.   setting – university library in ireland.   subjects – libqual data from 2007, 2009, 2012, and 2014.   methods – the researchers analyzed data from the 22 core libqual questions and the three dimensions of library as place, information control, and effect of service. the study focused specifically on libqual question lp2 in the library as place dimension: quiet space for individual work. qualitative free text comments in the surveys related to noise or quiet issues were also analyzed. the adequacy mean was used to determine improvement in scores; this metric is calculated by subtracting the minimum mean score from the perceived mean score.   main results – libqual scores related to the quiet space question steadily improved over the seven-year period studied. the adequacy mean went from -1.2 to -0.13, representing a 1.07 degree of improvement. for all 22 questions, the adequacy mean increased from 0.02 to 0.38, showing overall improvement of 0.36. researchers reviewed the data for all individual questions to measure the degree of change over the seven years; the quiet space question had the highest level of improvement of all of the questions. considering user groups’ perceptions, there was a 2.03 degree of improvement for graduate students, while there was a 0.82 degree of improvement for undergraduates.   the researchers wanted to know if the noise interventions had a specific impact on the quiet space question compared to a more general impact on the “library as place” dimension. none of the other “library as place” questions improved to the degree of the quiet space question. of the “library as place” questions, question lp5, the group space question, was the only one where the adequacy mean dropped, with an adequacy mean difference of -0.23.   external benchmarking conducted by the researchers put these results in an international context, using consortium data from arl in north america and the society of college, national and university libraries (sconul) in the united kingdom (u.k.).   conclusion – based on the study findings, the long-term noise management program implemented from 2007 to 2014 at the university library had a measurable impact, and users’ perceptions of the quiet space in the library improved.  because perceptions improved most among graduate students, researchers concluded that future efforts for noise management strategies should consider focusing on this group.   commentary    library spaces, in recent years, have transformed from quiet study spaces to spaces that have encouraged collaboration, group learning, social interactions, and technological advancement. however, along with these newly purposed spaces comes growing noise and lack of a quiet environment. according to the authors, the literature on noise management in libraries has grown in recent years but mostly consists of opinion-style articles with only a small number of articles that include evidence based research (mccaffrey & breen, 2016). the lack of research studies, along with the small number of published studies focusing longitudinally on a single libqual question, prompted the authors to conduct this study, which analyzes data across a seven-year period, and examines the impact of noise interventions on library users.   the authors used a critical appraisal tool developed by glynn (2006) to evaluate the study. the data included responses from undergraduates, graduates, faculty, and staff, and was representative of the user group populations. the response rate was 9% in 2007 and increased to 17 % in 2014. the survey transitioned from libqual in 2009 to libqual lite in 2012. in 2012, instead of answering all 22 questions, users answered three core questions and a randomly selected subset of users answered the remaining questions, meaning that not all respondents answered the quiet space question. as such, there were fewer responses to this question, particularly among graduate students and faculty. rather than solely focus on libqual quantitative data, the researchers could have more deeply explored the libqual free text comments related to noise and space. more of a focus could have been placed on the graduate student population, given they were most affected by library noise. supplementary qualitative inquiries, in addition to the libqual data, could have been used to identify which noise reduction strategies would be most effective, instead of using trial and error to attempt to satisfy this population.   the study findings, while specific to an academic library in ireland, may be applicable to academic libraries in the u.k. and worldwide, as many libraries seem to be grappling with the same noise issues. however, because not all libraries have the same needs in terms of group collaborative space and quiet study space, noise reduction strategies will differ based on the desires of the local library user population as well as the makeup of the library user groups. some of the noise management interventions described in this study are affordable and can be easily implemented in other libraries, while other interventions might be cost prohibitive and difficult to put into practice in some library environments due to space constraints. the library director was able to secure capital funding for significant library renovations; this solution is not one that many libraries are afforded. the article includes a table that displays specific interventions and the period in which they were implemented. however, because multiple interventions were put into place between surveys, the effectiveness of individual interventions could not be determined. libraries wishing to use the evidence provided here may want to consider these interventions as a starting point but more fully investigate which would be most appropriate for their student population.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154   mccaffrey, c. & breen, m. (2016). quiet in the library: an evidence-based approach to improving the student experience. portal: libraries and the academy, 16(4), 775-791.  https://doi.org/10.1353/pla.2016.0052   evidence based library and information practice evidence based library and information practice 2011, 6.3 16 evidence based library and information practice article an examination of library anxiety at cape breton university lenard j. lawless access services librarian cape breton university sydney, nova scotia, canada email: lenard_lawless@cbu.ca received: 20 apr. 2011 accepted: 20 july 2011 2011 lawless. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – library anxiety as a phenomenon has been discussed for decades. while it is generally recognized, seeing its effects within a specific institution can often be difficult. this study examined the presence and degree of library anxiety among students at cape breton university in canada. methods – a modified version of the library anxiety scale (las) was provided to the students via an online survey. invitations to take the survey were sent to students by email and via the student union’s social networking site. results – the average score on the las showed only mild anxiety levels among all the respondents. when categorizing the results by either program of study or program year, the results vary from no to mild anxiety. little variation was seen between the sexes, with both males and females scoring close to the overall average of mild anxiety. conclusions – with no segments of the student body scoring in the moderate to severe levels, the overall las scores for cape breton university’s students appear to be in a range that could be considered “normal.” introduction anxiety is defined as a “mood or state characterized by apprehension and somatic symptoms of tension in which an individual anticipates impending danger, catastrophe, or misfortune” (vandenbos, 2007). most librarians have been in and around libraries for so long that it can be difficult to grasp the concept of library anxiety. it is, however, very mailto:lenard_lawless@cbu.ca� evidence based library and information practice 2011, 6.3 17 real. while one would hope that patrons coming to our libraries do not experience a sense of danger or catastrophe, the sense of misfortune may well be common. casual discussions with some students at cape breton university (cbu), an undergraduate institution located in sydney, nova scotia, canada, provided anecdotal evidence of potentially high levels of library anxiety among the student body. these conversations covered many aspects of the library, from perceived lack of customer service to confusion regarding the layout of the library and the terminology used by staff. the possibility that these isolated conversations were representative of a large number of students was the impetus for a research project examining the level of library anxiety among students at cbu. literature review history of library anxiety in 1986, constance mellon became the first to recognize library anxiety as a real phenomenon. she started out examining library instruction and its helpfulness; her investigation found that students held more fear about beginning their research than specific problems with conducting the research. mellon found that: [t]he library phobia that [students] described seemed to tie in loosely with the work being done on math and test anxiety. it thus seemed logical to describe students’ fear in the library as library anxiety and to consider treating it within the anxiety framework (mellon, 1986, p. 163). mellon discovered that students were overwhelmed by the size of the library, did not know where to begin their research, and did not know how to proceed once they began. she felt that library instruction should be broadened to provide comfort and ease, rather than trying to teach the specifics of research too quickly. while mellon’s theory of library anxiety seemed sound, there existed no scale by which to measure it. bostick (1992) created just such a rubric when she created the library anxiety scale (las). bostik in the course of validating and testing the las, bostik (1992) found that the causes of library anxiety could be summed up in five categories. these causal categories include: barriers with staff (perceptions that librarians and staff are unapproachable or preoccupied); affective barriers (stemming from a belief that the student holds inadequate skills); comfort with the library (concerning the general safety and welcoming nature of the library space); knowledge of the library (familiarity with the layout and policies); and mechanical barriers (ability to use and the operational condition of various mechanical equipment). by using the las to measure each of these categories, it may be possible to determine any library’s anxiety inducing components (bostik, 1992). two researchers have come to the fore studying library anxiety, often employing the las. jiao and onwuegbuzie have found that library anxiety can be related to personal characteristics such as age and nationality (jiao, onwuegbuzie, & lichtenstein, 1996; jiao, onwuegbuzie, & bostick, 2006); a desire for structure and perfection (jiao & onwuegbuzie, 1998); a poor sense of determination (jiao & onwuegbuzie, 1999c); and perceived low levels of academic self-competence (jiao & onwuegbuzie, 1999a). the research partners went on to discover a social context to library anxiety, finding that cooperative students often had the lowest levels of library anxiety (jiao & onwuegbuzie, 2002). causes of library anxiety specific studies have been conducted in an attempt to identify the direct causes of perceived library anxiety. research focusing on student interviews attempted to identify exact difficulties regarding library use evidence based library and information practice 2011, 6.3 18 (andrews, 1993). this work uncovered a number of problems specific to using the library itself, including those with “the catalogue, with locating books, the classification scheme and library layout” (andrews, 1993). the results also indicated that feelings of inadequacy among the students interviewed led them to consult their peers as sources of assistance, rather than seek out the expert librarians readily available to assist them. excessive jargon in libraries, even when simply describing the physical library (e.g., aisles, rows, stacks) was termed “library noise” (keefer, 1993). students’ paths to success are so narrow and specific that a walk in the library itself can prove to be “noise” by distracting students with physical details (keefer, 1993). moreover, while librarians are able to reduce library anxiety levels through instruction, students’ worries about deadlines led them to become anxious about their research despite the instruction sessions (keefer, 1993). there have also been attempts to determine the reasons why students make use of the library in the first place. a study conducted by jiao and onwuegbuzie (2004) found the top three reasons were to obtain a book; to study for a test; and to use the computerized indexes and online resources. students who came in for the computerized indexes and online resources exhibited the highest level of library anxiety. the continued expansion of electronic resources and services compounded this problem, further increasing the prevalence of library anxiety. they concluded that “negative computer attitudes may inhibit student’s interests in learning to use the library resources and thereby weaken academic performances while elevating library anxiety” (jiao & onwuegbuzie, 2004). some college and university students turn to public libraries rather than the libraries within their own institutions. a study consisting of student interviews found students reported convenience, ease of use and familiarity as well as subjective appeal among their reasons for choosing the public library (antell, 2004). the students felt that the smaller size of local public libraries created a more manageable atmosphere and that the staff were friendlier. costs of library anxiety concerning the real world effects upon students, jiao and onwuegbuzie (1997) found that: library-anxious students experience more interfering responses during various stages of the information search process and, as such, tend to focus less of their energy and attention on the task itself, thus impeding their learning process (jiao & onwuegbuzie, 1997, p. 373). while the library strives to present itself as an ally of the students during the research and learning process, it can actually serve as an impediment to those students suffering from library anxiety. even when students decide to go to the library, their attempts to minimize their exposure to what they perceive to be an unfriendly environment can often lead to failure to find the proper material. they may not stop to ask for assistance and may even simply give up and leaving the library without the resources they need to conduct their research (keefer, 1993). jiao and onweugbuzie found that students with high levels of library anxiety tend to produce research proposals of lower quality (jiao & onwuegbuzie 1999b). the fact that that librarians were not consulted or sought out was identified as a reason for the low quality of the students’ work. feelings of anxiety, lower quality work, and the tendency to give up and leave the library could even culminate in students, especially graduate students, failing to complete their programs. as many as 95% of students engage in procrastination of writing research papers and other academic tasks (jiao & onwuegbuzie, 2007). jiao and onweugbuzie found that a relationship exists evidence based library and information practice 2011, 6.3 19 between this procrastination and library anxiety, especially in the areas of affective barriers, comfort with the library and mechanical barriers. while they were unable to determine if library anxiety initially caused procrastination or the procrastination triggered library anxiety, they concluded that the two were linked in a feedback loop where each continued to exacerbate the other (2007). measuring library anxiety anwar, al-kandari and al-qallaf (2004) used a modified version of the las to determine library anxiety levels among students at kuwait university. upon collecting the results, the authors found that the numerical representation of library anxiety levels was not useful. to help categorize their results, they proposed the following levels of anxiety: table 1 proposed categories for las scores no anxiety 1.00 – 2.21* low anxiety 2.22 – 2.65 mild anxiety 2.66 – 3.54 moderate anxiety 3.55 – 3.98 severe anxiety 3.99 – 5.00 *the “no anxiety” level was listed in the article as “0.00 – 2.21” but the lowest achievable score on the las is actually 1.00. methods purpose of study the research question that drove this project was whether cbu students suffered high levels of library anxiety, as suggested by anecdotal evidence. the literature indicates that traditional academic library aspects such as the online catalogue, classification scheme, and excessive use of jargon increases students’ library anxiety. it is important to analyse library anxiety as studies have shown that high levels of anxiety among students can lead to procrastination, library avoidance, reduced quality of deliverables, and even increased drop-out rates. participants an email was sent to all cbu students, inviting them to participate in an online survey. in addition, a link to the online survey was posted on the student union’s social networking page. with a student body of roughly 3,400, it was expected that roughly 10% (340) would respond, but only 162 took the modified library anxiety scale (las). still, this was enough to produce a 95% confidence level with only a 7.5% error level. the various schools of the university were well represented, with the school of arts and social sciences (sass) comprising 37.7% (61); school of science and technology (sst) 29.0% (47); school of graduate and professional studies (g&p) 21.0% (34); and school of business (buss) 12.3% (20). instruments and procedures the las (bostick, 1992), consists of 43 simple statements, with students responding by using a five-point likert scale. the survey contains statements from five areas: barriers with staff; affective barriers; comfort with the library; knowledge of the library; and mechanical barriers. for each statement, a response of “1”denotes strong disagreement, whereas a response of “5” denotes strong agreement with the statement. the average of all the response numbers provides a generalized anxiety level for the respondent. the statements on the las are quite simple, for example: “i feel comfortable using the library.” a high score on any subscale represents high anxiety in this area. this instrument has been found to generate both reliable and valid scores (bostick, 1992; onwuegbuzie, jiao, & bostick, 2004). questions regarding demographics (e.g., gender, program of study, program year) were added to the standard forty-three questions. it was hoped that these demographic questions could be used to help target marketing of the library to particular programs and possibly to specific years within those programs. with evidence based library and information practice 2011, 6.3 20 one question reading, “the librarians are unapproachable” and another reading, “the reference librarians are unapproachable”, the former was changed to read, “the library staff are unapproachable” to eliminate such close duplication. since no change machine is provided at the library, “the change machines are usually out of order” was replaced with “the microfilm machines are usually out of order.” a “comments” box was included at the end of the survey to allow respondents to provide feedback. results the results of the survey show low to moderate levels of anxiety overall. with possible final averages ranging from 1 to 5, actual scores by category were: barriers with staff (2.53); affective barriers (2.90); comfort with the library (2.48); knowledge of the library (2.25); and mechanical barriers (2.57). the results of the respondents from sass exhibited the lowest overall average score. they showed the lowest levels of anxiety concerning barriers with staff (2.05); affective barriers (2.40); comfort with the library (2.00); and knowledge of the library (1.70 the lowest of any school in any category). mechanical barriers, fell in the middle of the range (2.58). business students scored the highest level of anxiety when dealing with affective barriers (3.21) and comfort with the library (2.88), and the lowest level when dealing with mechanical barriers (2.32). barriers with staff and knowledge of the library found business students closer to the average (2.20 and 2.19, respectively). students from sst showed the highest level of anxiety in the category of mechanical barriers (2.73). they were mid-range for the categories of knowledge of the library (2.35); barriers with staff (2.50); affective barriers (2.83); and comfort with the library (2.83). g&p students had the highest overall las score (2.73). they were highest in the categories of barriers with staff (2.71) and knowledge of the library (2.55). they were slightly below the category average for affective barriers (2.88) and mechanical barriers (2.49). they were slightly above the averages for comfort with the library (2.73) and knowledge of the library (2.55). overall average las scores showed very little variation by gender (female = 2.57, male = 2.38). female students exhibited a range from the lowest value of 2.15 (knowledge of the library) up to the highest of 2.85 (affective table 2 average scores on las by program of study sass buss sst g&p average barriers with staff 2.05 2.20 2.50 2.71 2.53 affective barriers 2.40 3.21 2.83 2.88 2.90 comfort with the library 2.00 2.88 2.83 2.73 2.48 knowledge of the library 1.70 2.19 2.35 2.55 2.25 mechanical barriers 2.58 2.32 2.73 2.49 2.57 average 2.13 2.62 2.65 2.73 2.45 number 61 20 47 34 162 table 3 average scores on las by gender female male barriers with staff 2.47 2.22 affective barriers 2.85 2.63 comfort with the library 2.65 2.39 knowledge of the library 2.15 2.11 mechanical barriers 2.46 2.67 average 2.57 2.38 number 73 89 evidence based library and information practice 2011, 6.3 21 barriers) for a difference of 0.70. male students ranged from 2.11 (knowledge of the library) to 2.67 (mechanical barriers) for a difference of 0.56. the category with the largest separation between genders was comfort with the library where female respondents scored a high of 2.65 and male respondents a 2.39 for a difference of 0.6. knowledge of the library, the lowest for both female and male respondents, showed a difference of only 0.04. when arranging the responses according to the respondent’s year in program, a clear pattern can be seen. while mechanical barriers saw higher scores among second year students (2.90) compared with first year students (2.49), all other categories saw drops in scores (average drop of 0.25). barriers with staff, knowledge of the library and mechanical barriers were lower among third year students (2.24, 1.93, and 2.32) than they had been among second year students (2.29, 2.04 and 2.90). fourth year students scored lower on affective barriers than third year students (2.67 vs. 2.81) but scored higher on all of the other categories (average increase of 0.21). students beyond their fourth year scored the lowest in all five categories (average drop of 0.78). discussion the cbu library provides many of the traditional sources of library anxiety. the use of library jargon to describe a confusing layout following a seemingly obscure classification scheme could understandably lead to higher values on the las. anecdotal evidence at cbu indicated that the student body may feel relatively high levels of anxiety regarding the library. this survey found that the level of library anxiety among cbu students was lower than expected; it is closer to what could be called “normal.” making use of proposed library anxiety categories (anwar, al-kandari and alqallaf, 2004), it can be seen that, overall, students at cbu overall exhibit low anxiety (overall average was 2.45). within these results, however, certain patterns were observed. students in sass scored in the no anxiety level for the categories of barriers with staff, comfort with the library and knowledge of the library and low anxiety with affective barriers and mechanical barriers. this contrasts with students from sst scoring low anxiety for barriers with staff and knowledge of the library and mild anxiety for affective barriers, comfort with the library, and mechanical barriers. traditionally, arts students make use of libraries early and often throughout their degree (head, 2008). early year science students tend to rely on their lab instructors and basic internet searches while upper year science students engage in traditional library research (haines, light, o’malley, & delwiche, 2010). perhaps the difference in exposure level between the two schools can account for the reported difference in these categories. while there was very little variation in anxiety levels observed between the sexes, a clear pattern was seen when arranging the results by year in program. in four of the five las categories, scores were seen to peak in early year students, drop down for mid range students before climbing to another peak in fourth year students. those beyond their fourth year saw another drop in levels. higher table 4 average scores on las by year in program 1 2 3 4 4+ barriers with staff 2.52 2.29 2.24 2.51 1.75 affective barriers 2.91 2.77 2.81 2.67 1.93 comfort with the library 2.60 2.51 2.58 2.64 1.68 knowledge of the library 2.56 2.04 1.93 2.19 1.35 mechanical barriers 2.49 2.90 2.32 2.57 2.21 average 2.65 2.47 2.43 2.55 1.77 number 42 48 30 29 13 evidence based library and information practice 2011, 6.3 22 anxiety in the first two years of university may be explained by the novelty and complexity of academic research. university level research is very different from that conducted at the high school level. the lower level of anxiety in the middle years likely represents a growing level of information literacy that would lead to more familiarity with the library’s resources and less anxiety (whitmire, 2007). oddly, the consistent nature of the las scores in the affective barriers category seems to indicate that the students do not get an increased sense of confidence in their own abilities as they progress through their degrees. the assignment of major research projects in the final year of most programs could account for the spike in anxiety. most students beyond their fourth year of study have either started a second degree or have slowed their progression through their first. for those in a second degree, they may have overcome the anxiety of major research projects, allowing them to proceed with confidence and comfort in the library. for those that have had a slower path to degree completion, their lower course loads may lend them more time to work on each assignment, which, in turn, could decrease the anxiety associated with library research. most g&p students identified as being in their first year of study. it is possible that they are new to cbu and, as such, were adapting to an environment that may have been quite different from what they had become accustomed, leading to relatively high levels of library anxiety. the few g&p students that identified as being in their fifth year exhibited low levels of library anxiety that were similar to the rest of the students beyond their fourth year of study. it is possible that these g&p students had completed their undergraduate degrees at cbu and, as such, have become very familiar with this particular library, leading to relatively low levels of library anxiety. implications students from sass are required to use library resources at every step through their degree. this consistent exposure to a potentially stressful environment seems to acclimatize them to the library, as seen in their consistent low scores on the las. the findings of this research suggest that this model of consistent library usage should be adopted in other schools as well. library orientations in first year with little library research required until fourth year results in a punctuated equilibrium of information literacy learning for students in the sst. were they to engage in a consistent regimen of library research projects throughout their program, science students would reap the same benefits of lower library anxiety demonstrated by arts students. students in g&p demonstrated relatively high degrees of library anxiety. many of them have not studied at cbu prior to their current program and, as such, are not familiar with the library. while some may already be information literate from their time at other institutions, others may not have learned how to do research or may have been away from the academic environment long enough to forget much of what they did learn. while more orientations and library instruction could help some of these students, many are enrolled as distance students and would not benefit from on-site sessions. for the benefit of all students in general, and these students in particular, the library’s web presence must also be re-examined. terminology should be simplified, navigation made easier, and online tutorials created to provide similar learning environments to the on-site sessions currently provided. suggestions for further research while the las was useful in determining overall levels of anxiety among students, a shorter survey with more open-ended questions may provide more insight into the real causes of anxiety among the students. a “comments” box was included at the end of this survey, but of the 162 respondents, only 4 chose to leave a comment. these comments (reproduced in full in appendix b) seem to focus on the hours that the library is open, evidence based library and information practice 2011, 6.3 23 printing and photocopying costs, and noise levels due to other students engaging in “loud conversations in the study areas.” since none of these areas of concern were covered by the las, it is difficult to conclude whether these understandable concerns are universal among students, or limited to a small fraction. perhaps future iterations of the las could include questions about hours, copying and printing costs, and noise levels. the view expressed by at least one survey respondent that the las is “remarkably repetitive” could be addressed by removing questions that are too similar to one another. the invitations for the survey were sent only via email and a social networking web site. the results of this survey may speak only to the students who are comfortable in an electronic environment. repeating the survey in print form with copies handed directly to students in person could provide a fuller picture of the entire student body. if faculty cooperation can be obtained, print copies of the survey could also be distributed in classrooms. an ethnographic student study observing their actual usage of the library may help provide a better understanding of the reported low levels of library anxiety. examining student research and learning habits, as well as their study and work behaviours in the library’s physical space, could help inform issues of information literacy instruction and reallocation of space. further research should also include a web site usability study to ensure that the library is providing the best possible service to its students regardless across all access points. conclusion obscure classification schemes, confusing layouts, unfamiliar jargon and imposing online catalogue systems can all contribute to increased levels of library anxiety among university students. according to the literature, increased library anxiety can lead to procrastination, library avoidance, poor quality deliverables, and even increased dropout rates. with this in mind, it is very important that academic libraries examine the level of anxiety felt by their student patrons. discussions with individual students indicated the possibility of high levels of library anxiety at cbu, but a survey of the student body resulted in lower than expected levels. while gender did not affect library anxiety levels, program differences followed previously observed trends. arts students build confidence by completing research projects every year, while science students experience a spike in anxiety as they proceed through their programs and move from practical lab reports to research essays. overall, students experienced relatively high anxiety levels in their first two years and lower levels in their third year, followed by increased levels again in their final year. the adjustment to the rigors of university research may explain the decrease seen among third year students, while more significant research assignments in the final year renewed student anxiety. even with the peaks and valleys of the las scores at cbu, the overall numbers were quite low. the low levels reported in this survey indicate that the cbu library is not seen as a potential “source of misfortune” and does not contribute to delinquency among students. while the results of this survey will not lead to an immediate change in practice at the cbu library, it will lead to further study based on the suggestions for further research. references andrews, j. (1993). an exploration of students' library use problems. library review, 40(1), 5-14. antell, k. (2004). why do college students use public libraries? reference & user services quarterly, 43(3), 227-236. anwar, m.a., al-kandari, n.m., & al-qallaf, c.l. (2004). use of bostick’s library anxiety scale on undergraduate evidence based library and information practice 2011, 6.3 24 biological sciences students of kuwait university. library & information science research, 26(6), 266-283. bostick, s. l. (1992). the development and validation of the library anxiety scale. (ph.d., wayne state university). (etd collection for wayne state university) retrieved from http://digitalcommons.wayne.edu/diss ertations/aai9310624. (aai9310624) haines, l.l., light, j., o'malley, d., & delwiche, f.a. (2010). informationseeking behavior of basic science researchers: implications for library services. journal of the medical library association, 98(1), 73-81. head, a.j. (2008). information literacy from the trenches: how do humanities and social science majors conduct academic research? college & research libraries, 69(5), 427-445. jiao, q. g., & onwuegbuzie, a. j. (2004). the impact of information technology on library anxiety: the role of computer attitudes. information technology & libraries, 23(4), 138-144. jiao, q. g., & onwuegbuzie, a. j. (2002). dimensions of library anxiety and social interdependence: implications for library services. library review, 51(2), 71-78. jiao, q. g., & onwuegbuzie, a. j. (1999a). selfperception and library anxiety: an empirical study. library review, 48(3), 140. jiao, q. g., & onwuegbuzie, a. j. (1999b). identifying library anxiety through students' learning-modality preferences. the library quarterly, 69(2), 202-216. jiao, q. g., & onwuegbuzie, a. j. (1999c). is library anxiety important? library review, 48(6), 278-282. jiao, q. g., & onwuegbuzie, a. j. (1998). perfectionism and library anxiety among graduate students. the journal of academic librarianship, 24(5), 365371. jiao, q. g., & onwuegbuzie, a. j. (1997). antecedents of library anxiety. the library quarterly, 67(4), 372-389. jiao, q. g., onwuegbuzie, a. j., & bostick, s. l. (2006). the relationship between race and library anxiety among graduate students. information processing and management, 42(3), 843-851. jiao, q. g., onwuegbuzie, a. j., & lichtenstein, a. a. (1996). library anxiety: characteristics of 'at-risk' college students. library and information science, 18(2), 151-163. keefer, j. (1993). the hungry rat syndrome: library anxiety, information literacy, and the academic reference process. rq, 32(3), 333-340. mellon, c. a. (1986). library anxiety: a grounded theory and its development. college & research libraries, 47(2), 160165. onwuegbuzie, a. j., jiao, q. g. (2000). i’ll go to the library later: the relationship between academic procrastination and library anxiety. college & research libraries, 61(1), 45-54. onwuegbuzie, a. j., jiao, q. g., & bostick, s. l. (2004). library anxiety: theory, research, and applications. lanham, md.: scarecrow press. vandenbos, g.r. (2007). a.p.a. dictionary of psychology. washington, dc: american psychological association. whitmire, e. (2007). a longitudinal study of undergraduates' academic library experiences. journal of academic librarianship, 27(5), 379-385. evidence based library and information practice 2011, 6.3 25 appendix a modified library anxiety survey instrument demographic questions sex: area of study: years in university: please mark the number which most closely matches your feelings about the statement. the numbers range from: 1=strongly disagree 2=disagree 3=undecided 4=agree 5=strongly agree 1 i am embarrassed that i don't know how to use the library. 1 2 3 4 5 2 a lot of the university is confusing to me. 1 2 3 4 5 3 the library staff are unapproachable. 1 2 3 4 5 4 the reference librarians are unhelpful. 1 2 3 4 5 5 the librarians don't have time to help me because they're always too busy on the telephone. 1 2 3 4 5 6 i can't get help in the library at the times i need it. 1 2 3 4 5 7 library clerks don't have time to help me. 1 2 3 4 5 8 the librarians don't have time to help me because they're always busy doing something else. 1 2 3 4 5 9 i am unsure how to begin my research. 1 2 3 4 5 10 i get confused trying to find my way around the library. 1 2 3 4 5 11 i don't know what to do next when the book i need is not on the shelf. 1 2 3 4 5 12 the reference librarians are unapproachable. 1 2 3 4 5 13 i enjoy learning new things about the library. 1 2 3 4 5 14 if i can't find a book on the shelf, the library staff will help me. 1 2 3 4 5 15 there is often no one available in the library to help me. 1 2 3 4 5 16 i feel comfortable using the library. 1 2 3 4 5 17 i feel like i am bothering the reference librarian if i ask a question. 1 2 3 4 5 18 i feel safe in the library. 1 2 3 4 5 19 i feel comfortable in the library. 1 2 3 4 5 20 the reference librarians are unfriendly. 1 2 3 4 5 21 i can always ask the librarian if i don't know how to use a piece of equipment in the library. 1 2 3 4 5 22 the library is a comfortable place to study. 1 2 3 4 5 23 the library never has the materials i need. 1 2 3 4 5 24 i can never find things in the library. 1 2 3 4 5 25 there is too much crime in the library. 1 2 3 4 5 26 the people who work at the circulation desk are helpful. 1 2 3 4 5 27 the library staff doesn't care about students. 1 2 3 4 5 28 the library is an important part of my school. 1 2 3 4 5 29 i want to learn how to do my own research. 1 2 3 4 5 30 the copy machines are usually out of order. 1 2 3 4 5 31 i don't understand the library's overdue fines. 1 2 3 4 5 evidence based library and information practice 2011, 6.3 26 32 good instructions for using the library's computers are available. 1 2 3 4 5 33 librarians don't have time to help me. 1 2 3 4 5 34 the library's rules are too restrictive. 1 2 3 4 5 35 i don't feel physically safe in the library. 1 2 3 4 5 36 the computer printers are often out of paper. 1 2 3 4 5 37 the directions for using the computers are not clear. 1 2 3 4 5 38 i don't know what resources are available in the library. 1 2 3 4 5 39 the library staff doesn't listen to students. 1 2 3 4 5 40 the microfilm machines are usually out of order. 1 2 3 4 5 41 the library is a safe place. 1 2 3 4 5 42 the library won't let me check out as many items as i need. 1 2 3 4 5 43 i can't find enough space in the library to study. 1 2 3 4 5 comments: evidence based library and information practice 2011, 6.3 27 appendix b full text of respondent comments the following comments are reproduced here, verbatim, from the four surveys that contained entries in the “comments” box. • this survey was remarkably repetitive [sic]. there were multiple questions about safety and "useless" librarians. i realize this might be a potential issue, but i would also like to raise points as far as the physical environment of the library. it is not confusing to use for me because i understand it, however in comparison to other university libraries it is quite confusing and ineffectual. library hours was another question which was not brought up. being unable to go to the library on a friday or saturday evening has caused me much anxiety and many times resulted in me postponing work to sunday night (often the night before an assignment is due) that could have easily been completed prior if the library had been open friday or saturday evening. finally, printing and photocopying fees are another issue that has caused me anxiety. i have been known to adjust essay margins so that i can save on paper due to its outrageous printing cost. again, in comparison to other university libraries, cbu has remarkably high printing and photocopying fees. these are just a few issues which have caused me anxiety that are not even breached in this anxiety survey. • the one problem i have are the in considerate people who are on thier [sic] phones or have loud music playing and those who are carrying on loud conversations in the study areas. • i understand the library is going through some renovations at the moment; however, i believe a good idea would be to install more pc's upstairs where there is less traffic. also, i do not like the idea of the printer charge being 15 cents per page either [sic]. i understand that the library has to ultimately buy printer paper and ink cartriges [sic]; however, we pay a lot [sic] of tuition to go to the university. maybe the library should think about taking on a printer charge with tuition so when we go to print papers we do not have to worry about paying the circulation desk. the previous idea should see the students get 100 dollars in free printings or something along those lines. • the library staff are top notch, what i don't like about the library is that it is not a warm, comfortable space to study. / evidence based library and information practice/ / abstract introduction literature review history of library anxiety causes of library anxiety costs of library anxiety purpose of study an email was sent to all cbu students, inviting them to participate in an online survey. in addition, a link to the online survey was posted on the student union’s social networking page. with a student body of roughly 3,400, it was expected that rou... instruments and procedures results discussion suggestions for further research appendix a modified library anxiety survey instrument appendix b full text of respondent comments an assessment of information control: understanding library service quality from users’ perspectives research article   an assessment of information control: understanding library service quality from users’ perspectives   retno sayekti senior lecturer at the department of library and information sciences universitas islam negeri sumatera utara medan, indonesia email: retnosayekti69@uinsu.ac.id   muhammad aditya librarian universitas islam negeri sumatera utara medan, indonesia email: muhammad.aditya@uinsu.ac.id nurhayani lecturer at the department of islamic guidance and counselling universitas islam negeri sumatera utara medan, indonesia email: nurhayani@uinsu.ac.id tessa simahate chief librarian universitas negeri medan medan, indonesia email: echa@unimed.ac.id   yusniah lecturer at the department of library and information sciences universitas islam negeri sumatera utara medan, indonesia email: yusniah93@uinsu.ac.id   rina devianty lecturer at the department of indonesian language universitas islam negeri sumatera utara medan, indonesia email: rinadevianty@uinsu.ac.id   received: 13 jan. 2021                                                               accepted: 20 mar. 2022      2022 sayekti, aditya, nurhayani, simahate, yusniah, and devianty. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29916     abstract   objective – this study aims to measure library users’ perceptions of the quality of information control using libqual, a survey instrument that measures library users’ minimum perceived and desired levels of service quality across three dimensions: effect of service, library as place, and information control. numerous studies using libqual have emphasized the service aspect, while quality of information control has received less attention. previous studies have reported low quality of information control in academic libraries.   methods – a descriptive survey was conducted at the library of the universitas islam negeri sumatera utara (uinsu), medan, indonesia, where active members of the library total 49,892. using proportional random sampling, 100 completed surveys were obtained from a total population of 49,892.   results – this study shows that the quality of information control in the library of uinsu medan does not meet minimum user expectations. nevertheless, ease of navigation of information was perceived as acceptable. the study also reveals that the library has promoted information services through exhibition activities, user education activities, and social media.   conclusion – the findings suggest the need for libraries to improve the quality of information services, including content of information, access protocols, search time, ease of navigation, interface, and access from outside the campus. further, libraries need to conduct continuous service quality evaluation on a regular basis (using tools such as libqual) to understand the needs of users in terms of information control better. the results from the present study provide strong evidence to support a recommendation that, in general, universities should provide required resources and funding for libraries to improve information services to ensure that the libraries meet quality standards.     introduction   this research was conducted at the state islamic university of sumatera utara medan (later called uinsu), indonesia. the university has a total of 49,892 students consisting of undergraduate and graduate levels spreading to several faculties such as islamic education, social sciences, communication and islamic preaching, islamic jurisprudence and law, islamic economics and business, science and technology, public health, and islamic thoughts. student enrolment in the university automatically enables them to become members of the library. however, all enrolled students do not, in fact, become users of the library. from our preliminary observation, we learned that students’ attendance at the library is very low. student attendance statistics drawn from the library system show that the average student physically visits the library once per year. meanwhile, the average number of daily access to the library is about 250 students. this number is not comparable with the current number of library members.       in terms of its collection, the library has 114,345 volumes of books and subscribes to electronic journal databases. nevertheless, the use of these databases is still limited. despite students’ lack of knowledge in using them, we observed that librarians do not provide consistent instruction or tutoring on the use of digital library services. training on the use of the databases for the users was only conducted when the library first subscribed to those databases. therefore, it is not uncommon to encounter students who cannot open opac or access the journal databases. there are no library guide or protocols provided for users to access electronic resources. students are left confused not knowing how to operate the computer to access the journal databases. meanwhile, access to electronic resources is only made possible by visiting the library and is impossible from outside the campus.     this study presents a rigorous method of analyzing students’ perceptions of information access provided by the uinsu library. with this work, we intend to assess the perception of students regarding information service quality provided by the uinsu library. the study is based on the assumption that the alignment of students’ perceptions, may, in our views, enhance the quality of information services.   this study aims to measure the quality of academic library service based on the information control dimension from the perspectives of the users' perceptions. the hypothesis is that there is a difference between the users’ perceptions and minimum expectations, as well as between the users’ ideal expectations and perceptions of the information control dimension.   literature review   the quality of higher education’s library services in the digital era is dependent on the librarian (nur & seran, 2019). today’s young internet users form the majority of future library users, thus libraries are expected to become agents of information dissemination and exchange, and to provide users assistance with accessing, converting, analyzing, and evaluating information effectively and efficiently. gardner and eng (2005) claim that quality of service is dependent on whether the library can serve as a source of information for the younger generation.   patil and sawant (2017) hold that library services are said to be of quality only in cases where user expectations are satisfied. khadka and maharjan (2017) define customer satisfaction as an overall evaluation based on total service, over time. however, huang et al. (2017) emphasize that the user focuses more on the concrete knowledge desired from the library, while librarians are more focused on how users learn to discover things with a variety of resources and technical assistance, and whether they become independent in fulfilling their information retrieval abilities.   plum (1994) has long understood the need for academic libraries to integrate technological advances into direct social relations with visitors. library services demand responsiveness and speedy information delivery balanced with self-reliance i.e., enabling user independence in exploring and utilizing library information resources. libqual is able to measure the quality of library services because self-reliance and responsiveness are part of the indicators in the libqual dimension (association of research libraries, n.d.). this instrument, however, has received criticism (edgar, 2006; mccaffrey, 2019; roszkowski et al., 2005) despite its wide usage as a measure of library user satisfaction and improving library service quality. mccaffrey (2019), for instance, argues that libqual only measures one-sided concepts in the field of library operations. the instrument also emphasizes gradual user success and the importance of independence, but does not explicitly conceptualize the user's experience in library quality services or immediate need for information, education, persuasion, or professional assistance.   service quality measurements aim to distinguish between the actions and the service of a library. thus, the measurement of service quality is often in a user-based survey form. conversely, outcome measurements help to plan and assess programs from a user perspective and are more specifically oriented towards certain programs. often this is performed by interviewing users shortly after participating in a library program to ascertain whether the program objectives are being achieved, and the results can be very helpful for outreach programs. the outcome-based libqual evaluates the professionalism of librarians (mccaffrey, 2019). thus, libqual also serves as a tool for confirming service excellence, thereby measuring service quality.   some studies discovered low information control dimension in gap analysis and ease of access indicators (islamy et al., 2016; kayongo & jones, 2008). thus, there is a need to improve services related to easy access, especially in terms of the suitability of the information searching system in the library.   information control is a dimension used for measuring service quality from a content perspective, for measuring access to information resources, and for measuring the scope of content offered by libraries, as well as convenience, navigation ease, timeliness, equipment availability, and user independence in resource use (thompson et al., 2009). this includes the collection strength and availability, information coverage, speed and convenience in accessing information from within or outside the library, and the speed with which users are able to access information. the availability of equipment and independence are also part of this dimension. furthermore, computer equipment availability, search access, and wi-fi hotspot facilities are important factors in encouraging the academic community to be independent in utilizing library services.   access to information resources   the various forms and information content provided are part of the ease of access to information in libraries, while the use of electronic and online media such as vle (virtual learning environment), social media, and websites is part of measuring library information content and services, determining user perceptions and expectations (atkinson, 2016).   currently, there is no definition simple enough to explain the accessibility of information (foley & helfert, 2010), however, some experts define accessibility as “easy access to a large number of information resources within the system” or “data acquisition or achievement in information searching” (blixrud, 2002, p. 158; nwachukwu et al., 2014, p.1). a study by mudarri and abdo al-rabeei (2015) shows numerous models developed in measuring online access to information. these include the access control matrix (lampson, 1974), mandatory access control (mac) (nunes leal franqueira, 2006), discretionary access control (dac) (verification and test methods for access control policies / models, 2017), bell-lapadula (bell & lapadula, 1973), role-based access control (ferraiolo & kuhn, 1992), clark and wilson models (ge et al., 2004). therefore, institutions or organizations, particularly libraries, must consider security when granting access to online information resources and must establish information access security policies, i.e., rules and guidelines governing the protection and distribution of information resources, as well as descriptions of the resources protected and how the resources are protected from misuse and unauthorized use.   furthermore, convenience in searching for information is an important factor in attracting and creating a relationship between customers and the services rendered by a company or organization. this relationship determines the achievement of the organization's competitive goals (salehi et al., 2012). convenience refers to the ability to use technology independently, and ease in searching for information, especially on websites. library users as consumers have to have preferences and controls (nykiel, 2001). the convenience aspect of information control also means allowing users to bring food and drink into the library (gardner & eng, 2005).   another important aspect in information searching, both in local databases and web-based systems, is the ease of navigation. the navigation system includes the menu, sitemap, navigation trail, and page view (interface). a good system eases the difficulties faced by users, preventing them from getting lost while searching for information and increasing the effectiveness of information searching (zheng, 2015). thus, the information presented through the web or library portal must consider the users’ ease of access, as an unattractive interface or complicated menu is tedious and causes users to get lost while searching. the web navigation design also contributes directly to the general user experience in browsing websites and related applications, and there are numerous possible web portal navigation systems and designs, each with corresponding advantages and disadvantages. libraries must therefore develop web navigation systems based on user behavior in searching for information, not based on content, because the web navigation system helps to increase the number of visits in accordance with user behavior. this also provides convenience for users searching for the information resources presented (ferguson et al., 2015). consequently, librarians ought to involve users by conducting a need analysis using observation and survey techniques before designing the website, such as the one conducted by the california state university (ochoa, 2020).   speed (timeliness) in information searching is the time required to access available information, measured by comparing the expected time with the actual time expended before information is made available or presented for use (loshin, 2009). this time is important, especially in the use of a database system, because timeliness in searching contributes significantly to service quality from the user's perspective, and not only applies to manual information searching, but to digital, electronic or automation-based information services (foley & helfert, 2010; li et al., 2021). therefore, measuring the library service quality in the dimension of information control, must also consider the accessibility of the information provided by the library. the speed of access to information during searching is crucial, depending on the environment or circumstances in the organization in which the database is being used. this aspect is even more crucial in cases where the system is unable to present the information in a timely manner (omeluzor, 2020), for instance in traffic control, industrial automation, court systems, and programmed stock trading. in database systems, the timeliness aspect is often sacrificed to increase security in information searching. however, research (ahmed & vrbsky, 2002) proves that the two aspects are able to run simultaneously without anything being sacrificed. this means that increasing the timeliness of information access while maintaining the security system is possible.   the rapid development of mobile technology has dramatically changed information searching. this is due, in part, to the portability of smartphones enabling ease of information access, irrespective of time and place (ferguson et al., 2015). a new habit of “accessing information through fingertips” has developed in the community (zhang & liao, 2015, p. 62). these changes in lifestyle have implications for libraries. information service providers must now package and provide easily accessible information services, as well as simple and uncomplicated navigation, thus allowing users to search for information independently, without the help of librarians (self-reliance). this independence in accessing information is an indicator of the user’s control over the unfettered access to information. to maintain independence in seeking information that meets their needs, users need to have the skills to search and use information (stanziano, 2016).   self-reliance and self-determination in the search and use of information are crucial, particularly for people from disadvantaged developing countries, including nigeria, sri lanka, uganda, and tanzania. tanzania has developed the principles of education for self-reliance (esr) and adopted them as the nation’s educational philosophy (sanga, 2016). in esr, educated people must be self-reliant, responsible, and democratic. this also applies to independent learning, relying on one's own ability while searching for information and knowledge.   according to tuckett and stoffle (2016), self-reliant library users are able to conceptualize the type of information and scope needed, solve the problems at hand, define the required steps to find this information, determine the appropriate reference sources to meet information needs, ensure the reference source is available, as well as to determine the quality and reliability of the information obtained through library searches.   aims   this study aimed to investigate the service quality of the library of uinsu medan, in terms of information control. chen (2015) shows that students perceive the quality of information provided by librarians to be so low that they prefer using google to a library web portal, while ihejirika et.al. (2021) reveal that users visit the library website so infrequently that the library needs to adopt social media to promote the websites. therefore, the objectives of this study were to:   1.      evaluate the difference between users’ perceptions and minimum expectations of the information quality services of uinsu library 2.      evaluate the difference between users' perceptions and ideal expectations of the information quality services of uinsu library   methods   research approach   this study was carried out in the library of uinsu using a quantitative descriptive method, where the independent variable of information control was measured to recognize service quality. the service quality in this information control dimension was measured using the libqual instrument.   population   the population of this study was currently active registered members of the library, with a total number of 49,892 members distributed across various faculties in uinsu (table 1, appendix b). subsequently, 100 completed surveys were obtained from a total population of 49,892.   research instrument development   based on the libqual framework in the dimension of information control, the researcher developed a structured questionnaire (appendix a). the indicators of information control in this study refer to libqual covering: a) scope of content, including the collection availability and scope offered; b) convenience, including comfort and absence of obstacles in accessing information; c) ease of navigation or ease of access to information resources; d) timeliness, the access time required to find relevant and accurate information; e) equipment or modern devices; f) self-reliance, ensuring the users' independence in using access facilities and various information search instructions in the library. meanwhile, a semantic differential scale of nine levels was used for grading, with nine being the highest and one being the lowest. table 3 (appendix b) shows the indicators distributed into a series of instrument items.   distribution and data collection   the distribution and collection of the questionnaire was conducted using two techniques: visiting classrooms to get representatives from each faculty according to the number of samples from the proportionate random sampling technique (table 2) and also by visiting the library during its access hours when questionnaires were distributed to undergraduate and graduate students who were using the library. since the uinsu has two different locations, the researcher visited each library at a different time. the participants were recruited through approaching library visitors and asking whether they had time to answer the questionnaire voluntarily, with no incentives or inducements.   results   hypothesis testing was conducted using spss software version 25, to determine the mean difference between perceptions, minimum expectations, and ideal expectations, in terms of the dimension of information control within libqual. subsequently, the parametric statistical technique paired t-test was used, because the hypothesis testing was performed on two variables (expectations and perceptions) on the same sample continuum. the results of the hypothesis test on information control are shown in table 4 (appendix b).   based on the results of hypothesis testing, the mean of information control perception was 5.43 and the mean of minimum expectation of information control was 5.85. the results of the paired t-test showed a significant difference of 0.003 between the information control perception and the minimum expectation of information control. furthermore, the means of information control perception and desired information control were discovered to be 5.43 and 7.91, respectively, while the paired t-test showed a significant result of 0.000. based on the basic provisions of the paired t-test, the perception of information control was concluded to be different from the desired information control.   a subsequent gap analysis was carried out between the perceptions, minimum expectations, and desired information control of the users. this analysis included the adequacy gap between the users’ perceptions and minimum expectations, as well as the superiority gap between the users’ perceptions and ideal expectations.   differences in users’ perceptions and minimum expectations   the adequacy gap analysis value was reviewed to determine the adequacy quality of the library service in the dimension of information control. meanwhile, the adequacy gap analysis obtained a significant difference of -0.42, between the average perception (5.43) and minimum expectations (5.85) of users in the dimension of information control (table 5, appendix b). thus, the performance of the information quality and access in the uinsu library has not exceeded the users' minimum expectations. the negative adequacy gap score for the information quality dimension shows that performance has not satisfied users, and indicates the areas for improvement regarding information service quality.   the uinsu library service quality based on the six indicators of the information control dimension is presented in table 6 (appendix b). according to table 6, only the ease of navigation has a perceived value of 6.00, above the users' minimum expectations (5.94). however, this is below the ideal expectation of 8.01. therefore, of the six information control dimensions, only the perception of ease of navigation indicator is included in the tolerance zone. thus, the adequacy gap analysis on the ease of navigation indicator obtained a positive score of 0.06. this shows that the service quality of uinsu library regarding easy access to information resources is considered adequate by users.   meanwhile, the adequacy gap analysis on the other information control indicators all showed negative results. the adequacy gap scores on the scope of content indicator, convenience, timeliness, equipment, and self-reliance were -0.43, -0.77, -0.36, -0.73, and -0.26, respectively. these negative results indicate that the quality of library services regarding the collection availability, convenience in accessing the information, time to find information, modern equipment, and independence of the users, are currently considered inadequate.   table 7 (appendix b) shows one item with a perception in the tolerance zone and seven items below the tolerance zone. this produces seven items with negative adequacy gap values and one item with a positive adequacy gap score, namely ease of navigation. per table 7, the information quality and access in the uinsu library is considered adequate by users, but only in terms of the library's ability to facilitate information and to ensure users are able to access things independently.   in terms of the adequacy gap, there are seven question items from the information control dimension with negative results: one item in the area of convenience, one item in the area of timeliness, three items in the area of scope of content, one item in the area of modern equipment, and one item in the area of self-reliance. based on the superiority gap of the 7 question items, the most negative score was obtained by the convenience item (-0.77), followed closely by the equipment category (0.73). this finding suggests that the seven negative question items are areas where the uinsu library services are not acceptable to users and require improvement.   differences in users' perceptions and ideal expectations   the results of the service superiority gap analysis were reviewed to determine the difference in the mean value of users' perceptions and ideal expectations, in order to calculate the ideal service quality gap of the uinsu library. the service gap score measurement is presented in table 8 (appendix b). the score of the gap analysis on the information control dimension was discovered to be -2.48. this score was obtained from the difference in the mean values of perceptual information control (5.43) and the ideal information control (7.91). the negative results of the service gap analysis on the information control dimension show that the performance of information quality and access in uinsu library is currently unable to exceed the users' ideal expectations. this value does not automatically make service quality in this dimension acceptable, because the perceived value of the information control dimension is outside the tolerance zone, between the minimum (5.85) and ideal (7.91) expectations. table 9 shows only one of six existing indicators in the dimension of information control, the ease of navigation indicator has a perceived value above the users' minimum expectations (6.00). this score is between the minimum (5.94) and the ideal expectation (8.01). thus, of all the information control dimensions, only the perception in the ease of navigation indicator is within the tolerance zone. according to the gap analysis, the ease of navigation indicator obtained a negative value of -2.01, indicating that users’ ideal expectations have not been fulfilled. however, this value is acceptable, because the perceived value is in the tolerance zone.   the service quality gap analysis on the scope of content, convenience, timeliness, equipment, and self-reliance indicators also produced negative values. the most negative superiority value (-2.74) was obtained for the convenience indicator, followed by the equipment (-2.66), scope of content (-2.54), self-reliance (-2.48), and timeliness (-2.37) indicators. these negative values show that users' ideal expectations of the five indicators have not been fulfilled and are below the tolerance zone. this also shows the room for improvement in service quality. the quality of service at uinsu library regarding eight items of the dimension of information control is presented in table 10 (appendix b). the negative results of the gap analysis on all of the information control dimension items generally indicate that the service quality of the uinsu library, on the aspects of information quality and access to information, does not currently meet the ideal expectations of users. however, the ease of navigation item is acceptable, despite the negative superiority value. this is because the four items are in the tolerance zone (the perception is above the minimum expectation).   the negative score indicating the ease of independent access to information in the uinsu library is currently below users' ideal expectations. meanwhile, the results of the gap analysis of the other items produced negative scores outside of the tolerance zone. these seven areas of library service therefore require improvement and are not acceptable according to uinsu users. this statement was obtained based on the perceived value of the seven items, and shows the results below the ideal expectations of the users. the convenience aspect was found to obtain the lowest superiority gap score, indicating that users have high expectations for the quality of access to electronic resources, and expect easy access from homes or offices.   discussion   the quality of library services is assessable from the dimension of information control. according to users, there are at least five aspects that need improvement including convenience, equipment, the scope of content, timeliness, and self-reliance. these aspects show the major problems faced by the uinsu library service. meanwhile, less severe problems include the library’s ability to provide remotely accessible electronic information and provision of equipment that support independent searches in the library to access printed and electronic information sources. users perceive that the library provides good speed of access time, enabling them to find relevant and accurate information through the library website (https://library.uinsu.ac.id/).   the ease of navigation indicator was concluded to be the only acceptable poor service performance in the information control dimension. this is related to the ease of independent information access in the uinsu library.   meanwhile, the most unacceptable indicator was convenience, meaning users expect high-quality access to library electronic resources, irrespective of location. the fact that students prefer using google search engine to library databases when seeking information indicates that the library does not provide enough convenient access to the information needed by the users. for this case, badke (2014) suggests three ways of designing a library portal to be convenient for the users; first, simplify the tools without abandoning the features that make it better than google and second, help users find convenience in the proprietary academic search tools by teaching the structure of databases. since convenience can be regarded as multiple dimensions of the time cost, energy cost, and psychological cost, simplifying the search tool will help improve the quality of information. convenience of information seeking includes ease of use, ease of access, and immediacy of access to information resources.   in this study, the users were also found to be intolerant of the low performance in other categories. subsequently, the service aspects regarding the library website, the scope of printed and electronic collections in accordance with user needs, as well as the availability of modern information access equipment, also contribute to the fulfillment of users' expectations in the information control dimension.   the weaknesses of library services in the aspects described above, do not necessarily indicate unavailability of services; for example, the uinsu library was found to have subscribed to various electronic journal databases including emerald insight, ebscohost, and cambridge core, in addition to making computers available. furthermore, the website https://library.uinsu.ac.id/ was made available to provide access to various information sources. however, the low scores given by users on the aspects above indicate problems of information quality and access to library services, and the need for increased promotional activities.   promotion is an important factor for the use of information sources owned by libraries, as it serves as an effort to socialize the library's services and numerous information sources, and to educate users (patil & pradhan, 2014). also, exhibitions have had a significant impact on the use of information sources (delawska-elliott et al., 2015). library users additionally reported that the promotion of library information sources through social media including twitter, facebook, and instagram, triggered them to pay more attention. several studies have shown that the use of social media networks is effective in promoting activities or services (islam & habiba, 2015; quadri & adebayo idowu, 2016). a study by salisbury et al. (2012) shows that students also use social media to follow professors and library services. the ineffective use of social media as a means of engaging library users was discovered to be due to inadequate purpose and planning in the social media marketing approach (ihejirika et al., 2021). in addition, since september 2018, the uinsu library has made efforts to reach the public, especially the university’s students, by using the library’s instagram account. research shows that the use of social media for library promotion has proven to be effective to build trust and relationships with library users (lund & wang, 2021). also, a library festival is held at the end of each year, with various promotional activities and library value campaigns targeted at users. furthermore, training activities for journal databases are conducted, in relation to efforts for promoting electronic information sources. these databases are introduced to various departments in the uinsu medan, and some of the activities are conducted for postgraduate students at both masters and doctoral levels.   the library needs to manage marketing and promotional activities by formally setting a plan at the beginning of each year and conducting program evaluation at the end (delawska-elliott et al., 2015). richardson and kennedy (2014) suggest nine components of a marketing cycle, they are: project description, current market, swot analysis (strengths, weaknesses, opportunities, and threats), target market, marketing goals and objectives, marketing strategies, action plan, management, and assessment. in summary, librarians need to be aware that marketing library information services is an integral part of librarians’ activities (bedenbaugh, 2016).   conclusion   this study suggests that there is a need for the improvement of information service quality. therefore, the uinsu library should consider improving convenience, equipment, the scope of content, timeliness, and self-reliance aspects in information services. academic libraries serve students at different level of studies, as well as lecturers, and researchers who might have different information needs. therefore, providing information to these user groups also needs to be taken into account. each user group, including lecturers, undergraduate, postgraduate, and doctorate students, as well as researchers ought to be represented in the library’s information provision. however, kayongo and jones (2008) showed that satisfying each user group is a challenge for libraries. nevertheless, today’s generation y requires libraries to be accessible 24/7, which means that libraries must provide facilities that ensure accessibility at all times.   although incorporating social media to promote information access has proven effective (islam & habiba, 2015), it is not sufficient to just teach users information skills. further, if social media is used to promote information access, it must be well-organized (ihejirika et al., 2021), planned carefully, and executed regularly (patil & pradhan, 2014). in addition, academic libraries must conduct library instruction programs on a regular basis as part of their outreach, to provide assistance to those users needing to improve their skills   libraries should consider using tools such as libqual to analyze user information needs and continuously improve the quality of information control services. ignoring these aspects of managing and presenting information will produce lower levels of user satisfaction and increase the gap between perception and ideal expectations. the library needs to conduct well-planned library promotional activities to market its digital products (including journal subscriptions) to users, keeping in mind that social media platforms, such as facebook, have proven to be effective for promoting library information content. libraries can employ mixed methods approaches, formally and informally, to educate users about information literacy skills so that they can be independent in searching for and using information to solve problems.   the present study was conducted at an islamic university in southeast asia, but the findings are consistent with studies from other countries. thus, the results from the present study provide strong evidence to suggest that, in general, universities should provide required resources and funding for libraries to improve information services to ensure that libraries meet quality standards. the present study also provides insight into how universities, in general, might allocate funds for collection development and modern equipment, as well as resources for user education. the study contributes to the extant literature by revisiting the libqual model (association of research libraries, n.d.)   this study is limited in that this work is preliminary and further research on how to implement information literacy courses into the curriculum needs to be conducted. in-depth research regarding strategies for academic libraries to enhance library quality is also required.   author contributions   retno sayekti: conceptualization, software, writing – review & editing muhammad aditya: methodology, data curation and analysis nurhayani: writing – original draft tessa simahate: visualization, investigation yusniah: supervision rina devianty: validation   references   ahmed, q. n., & vrbsky, s. v. (2002). maintaining security and timeliness in real-time database system. journal of systems and software, 61(1), 15–29. https://doi.org/10.1016/s0164-1212(01)00111-x   association of research libraries. (n.d.). libqual+ charting library service quality. http://www.libqual.org   atkinson, j. (2016). reviews of library services: an overview. in quality and the academic library: reviewing, assessing and enhancing service provision. (pp. 57–60). chandos publishing. https://doi.org/10.1016/b978-0-12-802105-7.00006-3   badke, w. (2014). the convenience factor in information seeking. online searcher, 38(6). https://www.infotoday.com/onlinesearcher/articles/infolit-land/the-convenience-factor-in-information-seeking-100274.shtml?   bedenbaugh, r. a. (2016). marketing is our game: tackling the library awareness gap. public services quarterly, 12(4), 321–328. https://doi.org/10.1080/15228959.2016.1231603   bell, d., & lapadula, l. j. (1973). secure computer systems: mathematical foundations. in mitre technical report 2547 (vol. 1, issue mtr-2547). http://www-personal.umich.edu/~cja/lps12b/refs/belllapadula1.pdf   blixrud, j. c. (2002). library quality assessment: libqual+tm. statistics in practice – measuring & managing, 155–161. https://www.lboro.ac.uk/microsites/infosci/lisu/downloads/statsinpractice-pdfs/blixrudposter.pdf   chen, y. h. (2015). testing the impact of an information literacy course: undergraduates’ perceptions and use of the university libraries’ web portal. library & information science research, 37(3), 263–274. https://doi.org/10.1016/j.lisr.2015.04.002   delawska-elliott, b., grinstead, c., & martin, h. j. (2015). developing a marketing orientation in hospital library services: a case report. medical reference services quarterly, 34(4), 481–489. https://doi.org/10.1080/02763869.2015.1082390   edgar, b. (2006). questioning libqual+: critiquing its assessment of academic library effectiveness. proceedings of the american society for information science and technology, 43(1), 1-17. https://doi.org/10.1002/meet.14504301112   ferguson, a. m., mclean, d., & risko, e. f. (2015). answers at your fingertips: access to the internet influences willingness to answer questions. consciousness and cognition, 37, 91–102. https://doi.org/10.1016/j.concog.2015.08.008   foley, o., & helfert, m. (2010). information quality and accessibility. in t. sobh (ed.), innovations and advances in computer sciences and engineering. (pp. 477–481). springer. https://doi.org/10.1007/978-90-481-3658-2   gardner, s., & eng, s. (2005). what students want: generation y and the changing function of the academic library. portal: libraries and the academy, 5(3), 405–420. https://doi.org/10.1353/pla.2005.0034   ge, x., polack, f., & laleau, r. (2004). secure databases: an analysis of clark-wilson model in a database environment. lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics), 3084, 234–247. https://doi.org/10.1007/978-3-540-25975-6_18   huang, h., kai wah chu, s., yuyang liu, l., & yi zheng, p. (2017). understanding user-librarian interaction types in academic library microblogging: a comparison study in twitter and weibo. the journal of academic librarianship, 43(4), 329–336. https://doi.org/10.1016/j.acalib.2017.06.002   ihejirika, k. t., goulding, a., & calvert, p. (2021). rethinking academic library use of social media for marketing: management strategies for sustainable user engagement. journal of library administration, 61(1), 58-85. https://doi.org/10.1080/01930826.2020.1845547   islam, m. m., & habiba, u. (2015). use of social media in marketing of library and information services in bangladesh. desidoc journal of library and information technology, 35(4), 299–303. https://publications.drdo.gov.in/ojs/index.php/djlit/article/view/8455/4982   islamy, m. a., wahyudin, d., & margana, h. h. (2016). analisis tingkat kepuasan pemustaka tentang kualitas layanan perpustakaan dengan menggunakan metode libqual+tm : studi deskriptif di perpustakaan pusat institut teknologi bandung. edulib info, 3(1), 1–11. https://ejournal.upi.edu/index.php/edulibinfo/article/view/9042/5634   kayongo, j., & jones, s. (2008). faculty perception of information control using libqual+tm indicators. the journal of academic librarianship, 34(2), 130–138. https://doi.org/10.1016/j.acalib.2007.12.002   khadka, k., & maharjan, s. (2017). customer satisfaction and customer loyalty (thesis, centria university of applied sciences). https://www.theseus.fi/bitstream/handle/10024/139650/khadka_kabu  and  maharjan_ soniya.pdf?sequence=1&isallowed=y    lampson, b. w. (1974). protection. information sciences, 18–24.   li, h., chen, z., & guo, q. (2021). the evolution and approaches of information analysis service of university libraries in china. science & technology libraries, 40(1), 52-64. https://doi.org/10.1080/0194262x.2020.1830921   loshin, d. (2009). data quality and mdm. in d. loshin (ed.), master data management. (pp. 87–103). morgan kaufmann. https://doi.org/10.1016/b978-0-12-374225-4.00005-9   lund, b. d., & wang, t. (2021). information dissemination and interactions in higher education social media posts. journal of promotion management, 27(4), 547-561. https://doi.org/10.1080/10496491.2020.1851848   mccaffrey, c. (2019). transforming the university library one step at a time: a ten year libqual + review. new review of academic librarianship, 25(1), 59–75. https://doi.org/10.1080/13614533.2018.1511438   mudarri, t., & abdo al-rabeei, s. (2015). security fundamentals : access control models. international journal of interdisciplinarity in theory and practice, 7, 259–262. http://www.itpb.eu/index.php/ct-menu-item-3/15-economics/193-7-cislo-48-clanok   nunes leal franqueira, v. (2006). access control from an intrusion detection perspective. (ctit technical report series; no. 06-10). centre for telematics and information technology (ctit)   nur, m., & seran, m. s. b. (2020). service quality of border region higher education libraries. jurnal ad’ministrare, 6(2), 145. https://doi.org/10.26858/ja.v6i2.12429   nwachukwu, v. n., lucky, a. t., & salami, p. f. (2014). availability, accessibility and use of information resources and services among information seekers of lafia public library in nasarawa state. information and knowledge management, 4(10), 1–12. https://core.ac.uk/download/pdf/234671811.pdf   nykiel, r. a. (2001). technology, convenience and consumption. journal of hospitality & leisure marketing, 7(4), 79-84. https://doi.org/10.1300/j150v07n04_07   ochoa, i. v. (2020). navigation design and library terminology: findings from a user-centered usability study on a library website. information technology and libraries, 39(4). https://doi.org/10.6017/ital.v39i4.12123   omeluzor, s. u. (2020). evaluation of integrated library system (ils) use in university libraries in nigeria: an empirical study of adoption, performance, achievements, and shortcomings. evidence based library and information practice, 15(4), 49–67. https://doi.org/10.18438/eblip29604   patil, s. k., & pradhan, p. (2014). library promotion practices and marketing of library services: a role of library professionals. procedia social and behavioral sciences, 133, 249–254. https://doi.org/10.1016/j.sbspro.2014.04.191   patil, s., & sawant, s. (2017, january 24-25). service quality expectations of academic library users [paper presentation]. national conference on enhancing the role of the library in teaching and learning, pune, maharashtra, india.  http://eprints.rclis.org/32356/   plum, t. (1994). academic libraries and the rituals of knowledge. rq, 33(4), 496–508. https://www.jstor.org/stable/20862529   porat, l. (2013). marketing and assessment in academic libraries: a marriage of convenience or true love? evidence based library and information practice, 8(2), 60–67. https://doi.org/10.18438/b8fs5m   quadri, g. o., & adebayo idowu, o. (2016). social media use by librarians for information dissemination in three federal university libraries in southwest nigeria. journal of library & information services in distance learning, 10(1–2), 30–40. https://doi.org/10.1080/1533290x.2016.1156597   richardson, h. a. h., & kennedy, m. r. (2014). how to market your library’s electronic resources. the serials librarian, 67(1), 42–47. https://doi.org/10.1080/0361526x.2014.899289   roszkowski, m. j., baky, j. s., & jones, d. b. (2005). so which score on the libqual+tm tells me if library users are satisfied? library & information science research, 27(4), 424–439. https://doi.org/10.1016/j.lisr.2005.08.002   salehi, f., abdollahbeigi, b., langroudi, a. c., & salehi, f. (2012). the impact of website information convenience on e-commerce success of companies. procedia social and behavioral sciences, 57, 381–387. https://doi.org/10.1016/j.sbspro.2012.09.1201   salisbury, l., laincz, j., & smith, j. j. (2012). science and technology undergraduate students’ use of the internet, cell phones and social networking sites to access library information. issues in science and technology librarianship, 69. https://doi.org/10.5062/f4sb43pd   sanga, i. (2016). education for self reliance: nyerere’s policy recommendations in the context of tanzania. african research journal of education and social sciences, 3. http://arjess.org/education-for-self-reliance-nyereres-policy-recommendations-in-the-context-of-tanzania/   stanziano, s. (2016). information seeking behavior of older adults. the serials librarian, 71(3–4), 221–230. https://doi.org/10.1080/0361526x.2016.1230532   thompson, b., kyrillidou, m., & cook, c. (2009). item sampling in service quality assessment surveys to improve response rates and reduce respondent burden: the “libqual+® lite” example. performance measurement and metrics, 10(1), 6–16. https://doi.org/10.1108/14678040910949657   tuckett, h. w., & stoffle, c. j. (2016). learning theory and the self-reliant library user. reference & user services quarterly, 24(1), 58–66. http://www.jstor.org/stable/25827283   verification and test methods for access control policies / models, nist special publication 800-192 68 (2017). https://doi.org/10.6028/nist.sp.800-192   zhang, j., & liao, b. (2015). learning on the fingertips: the opportunities and challenges of educational apps. journal of education and practice, 6(20), 62–67. https://iiste.org/journals/index.php/jep/article/view/24198/24771   zheng, g. (2015). web navigation systems for information seeking. in m. khosrow-pour (ed.), encyclopedia of information science and technology. (pp. 7693–7701). igi global. https://doi.org/10.4018/978-1-4666-5888-2.ch758   appendix a survey   respondent identity name                                                                : ……………………………                                         student id number                                      :……………………………            gender                                                             : ……………………………           major/study program                                  : ……………………………                          level (bachelor/magister/doctor)             : ……………………………             part i: activities at the library in this section, please put a cross (x) on one of the items you choose. during your time as a student, did you visit and take advantage of the library service more than once? a.      yes (please go to number 2) b.      no (you do not need to answer the next question. thank you) how often do you take advantage of library services and facilities? a.      rarely b.      once a week c.      twice a week d.      more than twice a week   give your reasons for visiting the library. .............................................................................................................................................................................................................................................................................................................................................................................................................................. how often do you access library information sources through the web? e.      rarely f.       once a week g.      twice a week a.      more than twice a week   give your reasons ..............................................................................................................................................................................................................................................................................................................................................................................................................................   part ii. perceptions on the quality of information control of uinsu library services   direction: you are asked to provide a perception regarding the quality of library services, specifically the quality you know and feel. there are no wrong answers. put a checkmark (ü) on one of the numbers you choose and give your reasons. the answer option is determined as follows. a.      number (1) means low perception. b.      number (9) means your perception is high. the higher the score, the higher the grade of the aspect in question. no. library service as far as i know and feel try not to choose neutral answers.       1. the library's electronic resources can be accessed from my home or office   1 2 3 4 5 6 7 8 9     strongly disagree       strongly agree 2. the library website allows me to search for information independently   1 2 3 4 5 6 7 8 9     strongly disagree       strongly agree 3. the library has the printed materials i need for my work   1 2 3 4 5 6 7 8 9     very minimal       very large 4. the library has the electronic resources i need   1 2 3 4 5 6 7 8 9   very minimal       strongly agree         5. the library has modern equipment that helps access the information i need   1 2 3 4 5 6 7 8 9   strongly disagree       strongly agree 6. the library has access tools that are easy to use and helps find things independently   1 2 3 4 5 6 7 8 9   strongly disagree       strongly agree 7. the library makes information easy to be accessed independently   1 2 3 4 5 6 7 8 9   strongly disagree       strongly agree 8. the library has printed and electronic collections needed for my work   1 2 3 4 5 6 7 8 9                                                                             appendix b   table 1 active user of north sumatra state islamic university (uinsu) medan library faculty/department affiliation active library members da'wah and communication 3,692 islamic economics and business 8,458 social sciences 2,032 tarbiyah and teacher training 18,532 public health 1,469 science and technology 3,099 sharia and law 5,178 ushuluddin and islamic studies 3,016 postgraduate 4,416 total 49,892   table 2 sample based on proportional random sampling technique faculty/department affiliation sub population sample sum   of rounding results da'wah and communication 3,692 3692 x 100 49892 7 islamic economics and business 8,458 8458 x 100 49892 17 social sciences 2,032 2032 x 100 49892 4 tarbiyah and teacher training 18,532 18532 x 100 49892 38 public health 1,469 1469 x 100 49892 3 science and technology 3,099 3099 x 100 49892 6 sharia and law 5,178 5178 x 100 49892 10 ushuluddin and islamic studies 3,016 3016 x 100 49892 6 postgraduate 4,416 4416 x 100 49892 9 total   49892 100   table 3 item distribution variable indicator the number of item (s) information control scope of content convenience ease of navigation timeliness modern equipment self-reliance 3 1 1 1 1 1   table 4 the result of hypothesis testing dimension score t mean significance description icp – meic 5,43 5,85 -0,42 0,003 significant different icp – dic 5,43 7,91 -2,48 0,000 significant different   table 5 adequacy gap score in the information control dimension description p me ag information control (information quality and access)   5,43 5,85 -0,42   table 6 adequacy gap score of minimum expectation indicator p me ag a. scope of content 5.42 5.85 -0.43 b. convenience 4.95 5.72 -0.77 c. ease of navigation 6.00 5.94 0.06 d. timeliness 5.49 5.85 -0.36 e. equipment 5.20 5.93 -0.73 f. self-reliance 5.53 5.79 -0.26   table 7 adequacy gap score per questions item indicator question p  me ag convenience the electronic resources in uinsu library can be accessed from my home or office 4.95 5.72     -0.77 timeliness the website of uinsu library allows me to search for information independently 5.49 5.85     -0.36 scope of content the library has the printed materials i need for my work 5.51 5.77     -0.26 scope of content the library has the electronic resources i need 5.38 5.89     -0.51 scope of content the library has print and/or electronic collections for my work 5.37 5.90     -0.53 modern equipment the library has modern equipment facilitating me to access the information i need 5.20 5.93     -0.73 self-reliance the library has access tools that are easy to use and allow me to find things independently 5.53 5.79   -0.26 ease of navigation the library makes information easy to be accessed independently 6.00 5.94     0.06   table 8 the superiority gap score description p dic sg information control (information quality and access)   5,43 7,91 -2,48   table 9 the superiority gap score of each indicator indicator p   dic sg a. scope of content 5.42   7.96 -2.54 b. convenience 4.95   7.69 -2.74 c. ease of navigation 6.00   8.01 -2.01 d. timeliness 5.49   7.86 -2.37 e. modern equipment 5.20   7.86 -2.66 f. self-reliance 5.53   8.01 -2.48   table 10 the superiority gap score per question item indicator question p dic sg convenience the electronic resources in uinsu library can be accessed from my home or office 4.95 7,69     -2.74 timeliness the website of uinsu library allows me to search for information independently 5.49 7.86     -2.37 scope of content the library has the printed materials i need for my work 5.51 7.88     -2.37 scope of content the library has the electronic resources i need 5.38 7.92     -2.54 scope of content the library has print and/or electronic collections for my work 5.37 8.07     -2.70 modern equipment the library has modern equipment facilitating me to access the information i need 5.20 7.86     -2.66 self reliance the library has access tools that are easy to use and allow me to find things independently 5.53 8.01         -2.48 ease of navigation the library makes information easy to be accessed independently 6.00 8.01     -2.01     microsoft word news7_1497_final evidence based library and information practice 2008, 3:1  92 evidence based library and information practice       news     eblip: call for classic research studies      © 2008.  this is an open access article distributed under the terms of the creative commons attribution  license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and  reproduction in any medium, provided the original work is properly cited.       can you identify a research study that has  had a major influence on your practice or  library and information practice in general?  are you keen to make more people aware of  this research and its value? are you willing  to write a summary and appraisal of this  research?    evidence based library and information  practice (eblip)  http://ejournals.library.ualberta.ca/index.ph p/eblip is seeking nominations of classic  research studies that have impacted  practice, had an influence on lis  researchers, and stood the test of time. we  need your help to identify these classic  studies in our field and commit to writing a  summary of that research.  volume 2, issue  4 (december 2007) of eblip published six  such classics, featuring research from carol  kuhlthau, joanne gard marshall, robert s.  taylor, and more. we want to continue to  highlight past research that is important and  bring that research to the attention of new  readers.  please consider nominating a great  research article to be featured in eblip.    for more information, or to nominate a  research article, please contact denise  koufogiannakis  denise.koufogiannakis@ualberta.ca.  nominations should be accompanied by a  full bibliographic citation and an  explanation of the contribution of the  research to the field of library and  information practice.  if the article is  selected, a schedule for publication and  deadlines will be arranged with the editor.    thanks,   denise   associate editor (classics)   evidence based library and information  practice    research article   assessment of information and communication technology for selective dissemination of information and current awareness services: a case study of university libraries in the south-west zone of nigeria   saturday u. omeluzor, ph.d. university librarian clifford university owerrinta, abia state, nigeria email: someluzor@yahoo.com, omeluzors@clifforduni.edu.ng   gloria o. oyovwe-tinuoye readers’ services librarian university library federal university of petroleum resources effurun delta state, nigeria email: gloriatinuoye@yahoo.com   received: 17 nov. 2015  accepted: 10 sept. 2017       2017 omeluzor and oyovwe-tinuoye. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to assess the use of information and communication technology (ict) for selective dissemination of information (sdi) and current awareness services (cas) in university libraries in the south-west zone of nigeria.   methods – a descriptive research design was adopted. the instrument used for data collection was a structured questionnaire administered to a population consisting of 379 librarians, with 353 usable questionnaires retrieved.   results – findings revealed that most university libraries in the south-west zone of nigeria do not use ict in delivery of sdi and cas. it is evident in this study that despite the known positive effects of ict in library services, traditional methods were predominantly used for sdi and cas to the library users. the study revealed that erratic internet services, insufficient training, inadequate ict skills, and low support for ict were hindrances towards ict use for sdi and cas.   conclusions – the integration of ict features in library services for the delivery of cas and sdi has been a challenge in university libraries in south-west nigeria. only a few libraries and a low percentage of librarians had adopted the use of ict in the delivery of cas and sdi, while a larger number of libraries resorted to the use of traditional methods. the level of ict literacy among the librarians in this study is low, as a higher percentage of librarians did not have adequate ict skill to use available online resources on the internet and other ict tools to deliver sdi and cas in south-west, nigeria. this is not unconnected to the fact that the training and technical support received by the librarians is inadequate, and the level of support that academic libraries received from their university managements in south-west nigeria in terms of funding for ict development is inadequate, which led to low internet services.     introduction   in recent times, information and communication technology (ict) has been deployed in university libraries to help library users gain access to information, as well as to become up-to-date with recent developments in their areas of specialization. selective dissemination of information (sdi) is a conscious attempt by librarians to search databases in order to find relevant information for each library user or group of library users to fit their information needs, while current awareness services (cas) keep users up to date with the latest professional literature in their fields of interest, and inform library users about new acquisitions in the library. sdi is tailored to a particular group of library users, while cas is for every library user. this study considers the extent to which ict is incorporated in library services in the delivery of sdi and cas in university libraries in south-west nigeria and if, like most university libraries in developing countries, nigeria has yet to use ict in providing cas and sdi services to the library users.   university libraries in nigeria   nigeria’s first university (and first university library) was established in 1948 at the university college ibadan, now the university of ibadan. the earlier universities established in nigeria in the late 1940s and early 1950s were mainly public universities with functional libraries. the purpose of establishing them was to meet the educational need of nigerians before and after independence, to train government workers, train professional workforce, and to help achieve rapid industrialization and development after independence (buzz nigeria, n.d.).   in 1999, the federal government of nigeria granted charters to five private universities (babcock, bowen, covenant, igbinedion, and madonna) which were owned and operated by christian denominations and had served as missionary schools where church workers were trained. babcock university, for example, was owned and operated by the seventh-day adventist church in nigeria and had started in 1959 as adventist college of west africa.   in addition to the earlier universities, nigeria recognized the need to train workers in order to develop the gas and oil industry. therefore, in march 2007, the federal government of nigeria established the federal university of petroleum resources, effurun (fupre), located in delta state in south-south zone of nigeria. it is the first in africa and one of only a few in the world. the broad aim for establishing fupre was “to have a specialized university that will produce a unique high level personnel and relevant expertise for the oil and gas industry in nigeria and worldwide” (the federal university of petroleum resources library guide, 2014). the university offers courses in petroleum engineering, mechanical engineering, electrical and electronic engineering, chemical engineering, marine engineering, geology, environmental science, computer science/mathematics, physics, and chemistry, with the university library providing the information, materials, and ict infrastructure that helps prepare the students to face the challenges posed by the evolving technological advancement in the oil and gas industry. at the time of opening, the fupre library had a stock of 3,000 books, 85 journal titles, over 2,000 electronic books, and journals in oil and gas, general sciences, engineering, ict, and geology on its database. presently, the fupre collection has grown to 6,662 print books and 1,702 journal titles. it also has an e-library with 140 computer systems used by both staff and students (see appendix b, figures 3 and 4) to browse the internet and backed up with an inverter to provide uninterrupted power supply. the upgrading of the fupre library was achieved through an intervention from the tertiary education trust fund (tetfund), a special fund set aside by the federal government of nigeria for library development.   with regard to ict, abubakar (2011) has noted that university libraries in nigeria are at a crossroads because they are operating in an era of dwindling finances. he added that the constraints of university libraries in providing sdi and cas services include erratic internet services and inadequate hardware and software. similarly, fowowe (2017) claimed that funding of university libraries in nigeria is inadequate, affecting collection development and hiring of adequately skilled workers. ogunsola (2004) stated that the diffusion of ict into africa in general, and into nigeria specifically, has been at a snail's pace, such that the gap between information-rich developed countries and african countries continues to widen. while some university libraries in nigeria have made tremendous efforts in the area of ict adoption and use, others have yet to use icts fully due to inadequate funding, inadequate staff, lack of support by university management, interruption in internet access, and unreliable power supply.   statement of the problem   selective dissemination of information (sdi) and current awareness services (cas) are well known for delivering prompt and up-to-date information services to the library users. the effect of these services has been felt in most libraries across the world. some libraries display ict features and tools on their webpage for the purpose of disseminating information, while in practice they are not used. this study investigates the extent to which university libraries in south-west nigeria have adopted and used ict for sdi and cas, with a view to identifying the challenges and making suggestions on the way forward to enhance quality service delivery to the library users.   literature review   the need for libraries to respond quickly to the information needs of users through sdi and cas has repeatedly been mentioned in dated and contemporary literature (broady-preston & barnes, 2002; fourie, 2003; uzohue & yaya, 2016). shultz and de groote (2003) stated that the use of cas and sdi services would benefit librarians in developing professional relationships with patrons while increasing their professional expertise and subject knowledge. similarly, domini, goh, wong and chen (2010) asserted that cas and sdi are considered essential for success and survival in today’s environment. prompt responses as a result of using ict will ensure that the library patrons choose to use library services, rather than selecting from a range of alternatives such as vendors and the internet.   according to uzohue and yaya (2016), cas is beneficial in keeping users informed and providing access to needed documents and information in a preferred format. on the other hand, sdi services support users in bringing the right information closer to them, enabling the users to use time efficiently to attend to other needs instead of searching for information resources. uzohue and yaya (2016) noted that display boards and shelves, displays of tables of contents, newspaper cuttings, alerting of heads of departments, compilations of bibliographies, reading lists, and indexing and abstracting, among others, were traditional methods of cas and sdi. these have changed due to advancement in ict. advancement in ict can make sdi achievable using services such as emailing, whatsapp, short message service (sms), and rich site summary (rss) feeds, among others, to communicate with library users.   some university libraries in nigeria, however, have yet to take advantage of ict in delivering sdi and cas. nkiko and iroaganachi’s (2015) study on the need to provide community-focused sdi service revealed that among the academic libraries in ogun state, nigeria, only 3 out of the 17 use sdi for community development programs. equally, oguonu’s (2013) investigation of reference and information services in state teaching hospital medical libraries in south-east nigeria showed that sdi was not provided by the libraries, while online catalogue and ict facilities were also not available for reference and information services due to inadequate funding and lack of ict facilities in reference services.   role of cas in the university library   in some university libraries in the world, ict infusion into library services has given unprecedented access to information. for instance, the university of sheffield library uses their webpage driven by ict to deliver cas to the library users (university of sheffield, 2017). similarly, the university of illinois library uses rss feeds to keep up with publication in users’ fields, while email alerts are used for the electronic bibliographic database (university of illinois, 2017). cornell university library also uses ict tools such as blog, rss feed, and citation alerts to notify their library users about new books, journal tables of contents, and databases subscribed to by the library, among other activities (cornell university, 2017). fourie (2003) noted that since acquisition librarians operate in a dynamic environment that requires constant alertness to new developments, cas, especially those available for free on the world wide web (www), would be helpful in keeping up with trends in information processing and service delivery.   role of sdi in library services   many studies have emphasized the importance of sdi in library services in easing access to information. nkiko and iroaganachi (2015) emphasized the need for sdi in university libraries for dissemination of community-based information and services to different categories of library users. uzohue and yaya (2016) in their study expressed the need for medical librarians to use cas and sdi to deliver library services to their users. madukoma (2015) investigated the perception of users on electronic reference services at babcock university library, nigeria, in order to identify their needs, and concluded that electronic reference services were not adequately used due to lack of awareness of the availability of the services in the library. she recommended the use of memos and posters to create awareness of reference services. effective use of ict in library services, especially for sdi and cas, will enable library users to promptly access quality information that will support their learning, teaching, research, and decision making (oyewusi & oyeboade, 2009; uzohue & yaya, 2016).   challenges of using ict for sdi and cas in university libraries   the implementation of ict in university libraries in developing countries, and especially in nigeria, has faced several challenges (ayo, 2001). a study by issa, ayodele, abubakar and aliyu (2011) on the application of information technology to library services at the federal university of technology, akure library, ondo state, nigeria, showed that the only facilities available for services in the library are internet and computer. similarly, haliso and ogungbemi’s (2014) study showed a disparity in the use of library software, library website, and electronic library services in the six academic libraries in lagos state. ayo (2001) stated that most university libraries in nigeria were yet to fully implement ict due to lack of funds, erratic power supply, and lack of qualified personnel. omeluzor and oyovwe-tinuoye (2016) noted that a general non-use of integrated library systems (ils) in academic libraries in delta state, nigeria was due to inadequate ict infrastructure and funding. in addition, onyeonoru (2001) saw the issue of inadequate funding for development of education in africa as a serious challenge. there is a decline in funding of nigerian universities due to fallen crude oil prices in the international market (mitchell, 2016). similarly, omeluzor, madukoma, bamidele, and ogbuiyi (2012) concluded that the decline in funding of nigerian universities contributed to low research output of academic staff.   skill and knowledge are relevant for the manipulation of icts in university libraries. adeyoyin (2005) portrayed poor ict skills among library staff as a problem hindering the university libraries’ services. inadequate skilled personnel in the area of ict, indifference of administrative bodies, and insufficient support groups left many ict-based functions in libraries in bangladesh dormant (uddin & hasan, 2012). a major challenge in ict use in most university libraries is the lack of technical support and training, resulting in major setbacks (arachchige, 2002; hasan, 2009; & haliso, 2011).   aims   this study aims to:   1.       identify university libraries in south-west nigeria that have ict features on their webpage for delivery of cas and sdi. 2.       determine the media used in delivery of sdi and cas services to the library users 3.       identify the challenges of using ict in delivering sdi and cas services in university libraries in south-west nigeria.   methods    this study investigated the application of information and communication technology (ict) in providing selective dissemination of information (sdi) and current awareness services (cas) to library users in universities in south-west nigeria. the study adopted a survey research design. the population is made up of the entire population of 379 librarians in the 37 federal, state, and private university libraries in south-west nigeria. further demographic details are shown in appendix a, table 1. the data collection instrument used was a structured questionnaire (see appendix c). the instrument was subjected to a reliability test using cronbach alpha reliability coefficient, and produced a result of 0.72. this means that the instrument is reliable, since the test result is above the acceptance point of 0.50. the questionnaire was administered directly to the respondents with the support of six research assistants. of the 379 questionnaires returned, 353, or 93%, were found usable. the questionnaire was analyzed, and the results are presented in frequency table, percentage, and chart in appendix a, tables 2 to 4 and figure 2.     figure 1. showing the number of universities and those with ict features on their webpage for delivery of sdi and cas in south-west nigeria.     results   research objective 1 was to identify libraries with ict features on their webpage for delivery of cas and sdi in south-west (sw) nigeria. appendix a, table 2 shows that out of the  37 universities in sw, 3 federal university libraries and 3 private university libraries use features such as blogs, rss feed, ask a librarian, twitter, google+, email, opac, facebook, and youtube for delivery of sdi and cas to their patrons. none of the state university libraries have ict features on their webpages. appendix a, table 3 shows that 4 federal university libraries, 8 state university libraries, and 19 private university libraries in south-west nigeria do not have ict features on their library webpage. (see figure 1.)   a majority of the libraries (31) do not have ict features on their library webpages (see appendix a, table 2). the research also shows that five university libraries have two ict features or more, while crawford university library has only rss feed. the use of only one ict tool may be insufficient to deliver cas and sdi to library users, since some library users may not use that particular tool. however, findings reveal that babcock university library and university of ibadan library provide online assistance to library patrons using opacs which are not linked to their library webpages on the university website.   research objective 2 was to determine the media used in delivering sdi and cas services to the library users. among the respondents in this study, 98 (28%) mostly use the internet to deliver sdi and cas, 64 (18%) rarely use the internet, and 191 (54%) of the respondents do not use the internet at all to deliver sdi and cas services to library users. (see appendix a, table 3) this implies that since the internet is not consistently used, dissemination of information to the users may be limited to only those who access information via notice boards and memos. this finding is in variance with aina (2014) who stated that since the advent of the internet, libraries in developed and some developing countries have risen to the challenge by ensuring that the internet is used to enhance the provision of information services in order to retain the patronage of the library. in addition, 85 (24%) of the respondents use computers to deliver cas and sdi, 67 (19%) of the respondents rarely use computers, and 201 (57%) do not use computers at all to deliver cas and sdi. (see appendix a, table 3.)   findings also show that 151 (43%) of the respondents use telephones to deliver cas and sdi, 102 (29%) of the respondents rarely use it, and 100 (28%) of the respondents do not use telephones at all for cas and sdi (see appendix a, table 3.) use of telephones in university libraries for sdi and cas is slightly adopted, which means the users may be more inclined to use other sources of information. non-use of telephones, of course, also prevents users from receiving reference services via text message. this finding supports madukoma (2015), who found that inadequate ict facilities in academic libraries hindered users’ access to reference services. her findings revealed that the internet, cellphone, sms, and email were slightly adequate. this finding also supports capron (2000), who identified mail, telephone, tv, and radio as the traditional ways users send and receive information.   results also indicate that 223 (63%) of the respondents use notice boards and 230 (65%) of the respondents use memos to deliver cas and sdi to the library users. this finding agrees with madukoma (2015), who recommended that memos be sent to deans and department heads to inform staff about recent additions to the library collection, and that posters should be designed and pasted on strategic notice boards in the university premises. it also shows that 39 (11%) of the respondents mostly use ask a librarian, blogs, and rss feeds, while 314 (89%) of the respondents did not use the tools for sdi and cas. it is evident here that relevant tools such as ask a librarian, blogs, rss feeds, and email have not been adopted for the delivery of sdi and cas. this may be connected with lack of skill, as stated by adeyoyin (2005) and uddin and hasan (2012) as a major factor hindering ict adoption in academic libraries. this result clearly shows that the use of traditional methods such as notice boards and memos still dominates other methods of information dissemination in university libraries in south-west nigeria.   research objective 3 was to identify the challenges facing university libraries in using ict to deliver sdi and cas services in south-west nigeria. this study considered four factors that other studies have determined can affect the use of ict in service delivery in university libraries in nigeria. these include internet connection, training and technical support in the use of ict, librarians’ level of ict skill, and university managements’ support.   internet connection   results show that 84% of the respondents indicate that erratic internet services hinder cas and sdi in university libraries (see appendix a, figure 2). this agrees with studies on library infrastructure and facilities that have shown that the problem of internet access in nigeria is still unabated (nkiko & iroaganachi, 2015). this result also agrees with abubakar (2011) who stated that most university libraries in nigeria are severely constrained by erratic internet services, and with ezeani and igwesi (2012), who report that bandwidth and internet facilities in university libraries in nigeria are outdated and cannot support remote access to information.   training, technical support, and skill level   appendix a, figure 2 also shows that 69% of the respondents indicated that insufficient training and technical support hinders the delivery of cas and sdi in university libraries. this substantiates the findings of ayo (2001) that most university libraries in nigeria have not yet fully implemented ict as a result of inadequate qualified personnel. it also supports arachchige (2002) and haliso (2011), who found the lack of technical support as a hindrance towards ict use in university libraries. it further supports adeyoyin (2005) and okiy (2012), who described the poor ict infrastructural development, poor ict skills, and the poor training of librarians in university libraries in nigeria.   appendix a, figure 2 further shows that 81% of the respondents agreed that inadequate ict skill among librarians was a hindrance in the use of ict for sdi and cas services. inadequate skilled personnel for the provision of ict service to the library users is a prevalent factor (mathew & baby, 2012; quadri, 2012).   support from university management   the studies of hasan (2009) and osaniyi (2010) revealed that lack of support by university managements also militate against ict use in university libraries. the implication of this finding is that access to relevant information sources will be hindered; hence, libraries will not be able to provide sdi and cas to the library users.   limitations and opportunities for further study   this study was limited to 37 university libraries in south-western nigeria. the method of study was limited to a structured questionnaire without an “other” option provided in any of the questions. a study of more libraries using a questionnaire with more options might produce different results. future studies might also investigate the characteristics and practices of universities that this paper revealed are able to offer ict despite a nation-wide funding crisis.    conclusions   the integration of ict features in the library services for the delivery of cas and sdi has been a challenge in university libraries in south-west nigeria. in an era of advancement in ict in all sectors of human endeavour, most university libraries, especially in nigeria, have yet to adopt ict in the delivery of sdi and cas to its patrons. among the university libraries in south-west nigeria, few libraries and a low percentage of librarians have adopted ict in the delivery of cas and sdi, while a larger number of libraries have yet to do so, but rather resorted to the use of notice boards and memos. the level of ict literacy among the librarians in this study is low, as a large percentage of them did not have adequate ict skill to use available online resources on the internet and other ict tools to deliver sdi and cas in south-west nigeria. this is not unconnected to the fact that the training and technical support received by the librarians is inadequate and the level of support that academic libraries received from their university managements in south-west nigeria in terms of funding for ict development, internet services, and training is insufficient. it is obvious that non-use of ict hinders access and use of library resources and services. therefore, it is imperative to use icts in university libraries for the delivery of sdi, cas, and other services to the library clients.   acknowledgement   the authors sincerely acknowledge the assistance of carol waseleski and the editorial support from the reviewers and editors of eplip in helping to prepare this manuscript for publication.   references   abubakar, b. m. (2011). academic libraries in nigeria in the 21st century. library philosophy and practice (e-journal), 446. retrieved from http://digitalcommons.unl.edu/libphilprac/446/   academic staff training and development (ast&d). (2016). in tertiary education trust fund (tetfund), university of nigeria, nsukka. retrieved from http://tetfund.unn.edu.ng/staff/   adeyoyin, s. o. (2005). information and communication technology (ict) literacy among the staff of nigerian university libraries. library review, 54(4), 257-266. https://doi.org/10.1108/00242530510593443   aina, l. o. (2014). the current practice of librarianship: a journey to extinction of the profession in nigeria? being a paper presented at the 8th jire olanlokun memorial lecture at the julius berger hall, university of lagos, p. 12. arachchige, j. j. g. (2002). an approach to marketing in special and academic libraries of sri lanka: a survey with emphasis on services provided to the clientele. retrieved from http://eprints.rclis.org/6731/1/artpradeepa.pdf   ayo, t. a. (2001). information and communication technologies and the information professionals in the information age: the nigerian perspectives. information management, 6(1), 33-43.   brief history of fupre library. (2014). in the federal university of petroleum resources library guide (p. 1).   broady-preston, t. and barnes, e. (2002). creating and sustaining competitive advantage in libraries: wales, a case study. in r savard (ed.), education and research for marketing and quality management in libraries: ifla satellite meeting, quebec, august 14-16 2001, vol. 99. munchen: ifla publications, k. g. saur, pp. 309-316.   buzz nigeria (n.d.). top 10 oldest universities in nigeria – when and how they were established. retrieved from http://buzznigeria.com/top-10-oldest-universities-in-nigeria-when-and-how-they-where-established/   capron, h. l. (2000). computers: tools for an information age. new jersey: prentice hall.   cornell university (2017). current awareness (rss). retrieved from https://www.library.cornell.edu/research/current-awareness   dominic, p. d. d, goh, k. n., wong, d. & chen, y. y. (2010). importance of service quality for competitive advantage with special reference to industrial product.  international journal of business information systems, 6 (3), 378-397.   ezeani, c. n. & igwesi, u. (2012). using social media for dynamic library service delivery: the nigeria experience. library philosophy and practice (e-journal), 814. retrieved from http://digitalcommons.unl.edu/libphilprac/814/   fourie, i. (2003). how can current awareness services (cas) be used in the world of library acquisitions? online information review, 27(3), 183195. https://doi.org/10.1108/14684520310481409   fowowe, s. o. (2017). funding academic libraries in nigeria: a survey of some nigerian university libraries. retrieved from https://www.researchgate.net/publication/237775531_funding_academic_libraries_in_nigeria_a_survey_of_some_nigerian_university_libraries   haliso, y. & ogungbemi, j. i. (2014). internet vs library: coping strategies for academic librarians in lagos state, nigeria. information and knowledge management, 4(1), 58-64.   haliso, y. (2011). factors affecting information and communication technologies (icts) use by academic librarians in southwestern nigeria. library philosophy and practice (e-journal) 571. retrieved from http://digitalcommons.unl.edu/libphilprac/571/   hasan, n. (2009, oct.). issues and challenges in open source software environment with special reference to india. paper presented at international conference on academic libraries (ical 2009), university of delhi, delhi, india. retrieved from http://crl.du.ac.in/ical09/papers/index_files/ical-43_144_317_1_rv.pdf   issa, a. o., ayodele, a. e., abubakar, u. & aliyu, m. b. (2011). application of information technology to library services at the federal university of technology, akure library, ondo state, nigeria. library philosophy and practice. retrieved 16 november 2017 from http://www.webpages.uidaho.edu/~mbolin/issa-ayodele-abubakar-bola.htm   madukoma, e. (2015, aug.). users’ perception of electronic reference services in babcock university library, ilishan-remo, ogun state, nigeria. paper presented at ifla world library and information congress, cape town, south africa. retrieved from http://www.ifla.org/files/assets/reference-and-information-services/publications/2015_ifla_reference_section-madukoma-en2.pdf   mathew, s. k. & baby, m. d. (2012). developing technology skills for academic librarians: a study based on the universities in kerala, india. library philosophy and practice (e-journal), 702.  retrieved from http://digitalcommons.unl.edu/libphilprac/702/   mitchell, c. (2016). oil price fall is main reason for tough times in nigeria. financial times. retrieved from https://www.ft.com/content/3a47381a-7371-11e6-bf48-b372cdb1043a   nkiko, c. & iroaganachi, m. a. (2015, aug.). community-focused selective dissemination of information services for empowering women through information provision and utilization: center for learning resources as a catalyst for social change. paper presented at ifla world library and information congress, cape town, south africa. retrieved from http://www.ifla.org/files/assets/reference-and-information-services/publications/005_iroaganachi_en.pdf   ogunsola, l. a. (2004). nigerian university libraries and the challenges of globalization: the way forward. electronic journal of academic and special librarianship, 5(2-3).   oguonu, o. c. (2013). assessment of reference and information services in state teaching hospital medical libraries in south east nigeria. retrieved from http://repository.unn.edu.ng:8080/xmlui/handle/123456789/2189   okiy, r. b. (2012). towards accelerated development of academic library services in nigeria for national development in the 21st century. library philosophy and practice. retrieved from http://www.webpages.uidaho.edu/~mbolin/okiy2.htm   omeluzor, s. u. & oyovwe-tinuoye, g. o. (2016). assessing the adoption and use of integrated library systems (ils) for library service provision in academic libraries in edo and delta states, nigeria, library review, 65(8/9), 578-592. https://doi.org/10.1108/lr-01-2016-0005   omeluzor, s. u., madukoma, e., bamidele, i., & ogbuiyi, s. u. (2012). use of electronic information resources and research output by academic staff in private universities in ogun state, nigeria. canadian social science, 8(3), 8-15. https://doi.org/10.3968/j.css.1923669720120803.1895   onyeonoru, i. p. (2001). civil society and corruption: state policing or policing the state? nigerian journal of psychology, 18(2), 46-62.   osaniyi, l. (2010). evaluating the x-lib library automation system at babcock university, nigeria: a case study. information development, 26(1), 87-97. https://doi.org/10.1177/0266666909358306   oyewusi, f. o. & oyeboade, s. a. (2009). an empirical study of accessibility and use of library resources by undergraduates in a nigerian state university of technology. library philosophy and practice. retrieved from http://www.webpages.uidaho.edu/~mbolin/oyewusi-oyeboade.htm   quadri, g. o. (2012). impact of ict skills on the use of e-resources by information professionals: a review of related literature. library philosophy and practice (e-journal), 762. retrieved from http://digitalcommons.unl.edu/libphilprac/762   shultz, m. & de groote, s. l. (2003). medline sdi services: how do they compare? journal of the medical library association, 91(4), 460–467. retrieved from http://www.ncbi.nlm.nih.gov/pmc/articles/pmc209512/   uddin, j. & hasan, n. (2012). use of information technology in library service: a study on some selected libraries in northern part of bangladesh. international journal of library and information science, 4(3), 33-44.   university of illinois (2017) current awareness (rss). retrieved from http://guides.library.illinois.edu/c.php?g=347966&p=2345651   university of sheffield (2017). news (rss). retrieved from https://www.sheffield.ac.uk/news/rss-feeds   uzohue, c. e. & yaya, j. a. (2016). provision of current awareness services and selective dissemination of information by medical librarians in technological era. american journal of information science and computer engineering, 2(2), 8-14.   appendix a tables 1­–4, figure 2   table 1 designation and academic qualification of the respondents designation frequency percentage university librarian 56 16 principal/senior librarian 63 18 librarian i 87 24 librarian ii 92 26 assistant librarian 55 16 total 353 100 academic qualification of respondents doctor of philosophy (ph.d.) 79 22 masters degree (m.sc.) 180 51 bachelor degree (ba/b.sc.) 94 27 total 353 100     table 2 university libraries in south-west nigeria with ict features available on their webpages s/n name of university ownership availability of ict feature on library webpage ict feature available 1. federal university of technology, akure library federal available blog, rss feed, ask a librarian, email 2. university of agriculture library, abeokuta federal available rss feed, twitter, google+ and facebook 3. university of lagos library federal available ask a librarian, rss feed 4. bowen university library, iwo. private available opac, rss feed 5. covenant university library, ota. private available rss feed, facebook, youtube 6. crawford university library, igbesa, lagos. private available rss feed     table 3 media used in delivering sdi and cas services to library patrons media mostly used rarely used not used internet 98 (28) 64 (18) 191 (54) computer 85 (24) 67 (19) 201 (57) telephone 151 (43) 102 (29) 100 (28) notice board 223 (63) 130 (37) memo 230 (65) 123 (35) ask a librarian 39 (11) 314 (89) blog 39 (11) 314 (89) rss feed 39 (11) 314 (89) email 39 (11) 86 (24) 228 (65)     table 4 university libraries in south-west nigeria without ict features available on their webpages 1. federal university, oye-ekiti library federal nil nil 2. obafemi awolowo university library federal nil nil 3. university of ibadan library federal nil nil 4. national open university library, lagos federal nil nil 5. adekunle ajasin university library, akungba. state nil nil 6. ekiti state university library, ado ekiti. state nil nil 7. ladoke akintola university of technology library, ogbomoso. state nil nil 8. olabisi onabanjo university library, agoiwoye. state nil nil 9. ondo state university of science and technology library, okitipupa. state nil nil 10. osun state university library, oshogbo state nil nil 11. tai solarin university of education library, ijebuode. state nil nil 12. lagos state university library, ojo, lagos. state nil nil 13. achievers university library, owo, ondo. private nil nil 14. adeleke university library, ede. private nil nil 15. afe babalola university library, ado ekiti. private nil nil 16. ajayi crowther university library, ibadan. private nil nil 17. babcock university library, ilishan-remo. private nil nil 18. bells university of technology library, otta. private nil nil 19. caleb university library. private nil nil 20. cetep university library, lagos. private nil nil 21. crescent university library. private nil nil 22. elizade university library, ilara-mokin. private nil nil 23. fountain university library, oshogbo. private nil nil 24. joseph ayo babalola university library, ikeji-arakeji. private nil nil 25. oduduwa university library, ipetumodu. private nil nil 26. pan african university library, lagos. private nil nil 27. redeemer’s university library, mowe, ogun. private nil nil 28. southwestern university library, oku owa, lagos. private nil nil 29. wesley university of science and technology library. private nil nil 30. mcpherson university library, seriki sotayo, ajebo. private nil nil 31. leed city university library, oyo. private nil nil       figure 2 factors hindering the use of ict in delivering sdi and cas in south-west nigeria.     appendix b staff and students browsing the internet at the fupre e-library   figure 3 staff and students accessing the internet at the fupre e-library.     figure 4 students browsing the internet at the fupre e-library.     appendix c   questionnaire on assessment of information and communication technology (ict) in selective dissemination of information (sdi) and current awareness services (cas) in university libraries in south-west nigeria   dear respondent,   this questionnaire is on assessment of information and communication technology (ict) in selective dissemination of information (sdi) and current awareness services (cas) in university libraries in south-west nigeria. it is solely to gather data for the completion of the research. we will appreciate it if you would participate and provide relevant information to accomplish this research on schedule. all information provided will be treated with utmost confidentiality and will only be used for the purpose of this research.   the researchers.   section a: demographic information of respondent   1.             what is the name of your university library? ......................................................................... 2.             what is your designation? (a) university librarian      (b) principal/senior librarian       (c) librarian 1     d) librarian ii     (e) assistant librarian 3.             what academic qualification did you attain? (a) doctoral degree             (b)   masters degree              (c) bachelor degree   section b:  identify libraries with ict features on their webpage for dissemination of cas and sdi   4.      kindly indicate the available ict features on your library webpage for dissemination of cas and sdi. ict features available not available not sure blog       rss feed       ask a librarian       email       opac           section c: find out the media used in dissemination of sdi and cas services to the library users.   5.         kindly indicate by ticking   your use of the following media for dissemination of sdi and cas services to your library users. media mostly used rarely used not used internet computer telephone (text messaging) notice board memo ask a librarian blog rss feed email   section d: identify the challenges of using ict in delivering sdi and cas services in south-west nigeria.   6.      kindly indicate by ticking  the challenges facing use of ict in delivery of sdi and cas services in your library.   challenges tick as it apply erratic internet services   insufficient training/technical support   inadequate ict skill among librarians   low support on ict       thank you.     evidence based library and information practice evidence based library and information practice evidence summary point-of-care healthcare databases are an overall asset to clinicians, but different databases may vary in usefulness based on personal preferences a review of: chan, r. & stieda, v. (2011). evaluation of three point-of-care healthcare databases: bmj point-of-care, clin-eguide and nursing reference centre. health and information libraries journal, 28(1), 50-58. doi: 10.1111/j.1471-1842.2010.00920.x reviewed by: carol d. howe reference librarian/assistant professor gabriele library, immaculata university immaculata, pennsylvania, united states of america email: chowe@immaculata.edu received: 02 aug. 2011 accepted: 15 oct. 2011 2011 howe. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. evidence based library and information practice 2011, 6.4 152 abstract objective – to evaluate the usefulness of three point-of-care healthcare databases (bmj point-of-care, clin-eguide, and nursing reference centre) in clinical practice. design – a descriptive study analyzing questionnaire results. setting – hospitals within alberta, canada’s two largest health regions (at the time of this study), with a third health region submitting a small number of responses. subjects – a total of 46 alberta hospital personnel answered the questionnaire, including 19 clinicians, 7 administrators, 6 nurses, 1 librarian, 1 preceptor, and “some” project coordinators. subjects were chosen using a non-probability sampling method. methods – the researchers developed an online questionnaire consisting of 17 questions and posted it on the university of calgary’s health sciences library and the health knowledge network websites. the questions, in general, asked respondents how easy the databases were to search and use, whether the database content answered their clinical questions, and whether they would recommend the databases for future purchase. most questions required a response for each of the three databases. the researchers collected quantitative data by using a likert scale from 1 to 5, with 5 being the most positive answer and 1 being the most negative. they collected qualitative data by asking open-ended questions. main results – with regard to ease of searching, bmj point-of-care (bmj) received the greatest number of responses (71%) at level 5. a smaller number of respondents (56%) rated nursing reference centre (nrc) at level 5. clin-eguide received 59% of the responses at level 5, but it also received the greatest number of responses at the next highest level (level 4). respondents rated all three databases similarly with regard to levels 1 and 2. regarding how easy the resources were to learn, most respondents rated all three databases as easy to learn (bmj, 77%; clin-eguide, 72%; and nrc, 68%). very few respondents thought any of the databases were difficult to learn. the researchers gleaned from open-ended questions that the respondents generally thought all three databases were faster and easier to use than the conventional databases they had used. respondents did not always agree with one another, however, about which features they liked or why. with regard to content, most respondents agreed that the information in all three databases was relevant to their needs (94.6% for clin-eguide and 87.9% for bmj and nrc). respondents also generally agreed that all three databases answered their questions to a high degree. clin-eguide had the highest percentage of answers at levels 4 and 5 and the lowest percentage of answers at level 2. nrc was the reverse, with the lowest percentage of answers at levels 4 and 5 and the highest percentage of answers at level 2. still, the researchers felt that all three databases answered respondents’ questions to a similar degree. in the open-ended questions, respondents voiced additional likes and dislikes about content, but again, answers among respondents were not consistent with one another. respondents were asked how often they would use the resource if it were available though their library. the majority of bmj users reported that they would use it extensively or moderately. about 36% and 39% of nrc users reported they would use it extensively or moderately, respectively; while 43.5% and 34.8% of clin-eguide users reported they would use it extensively or moderately, respectively. when asked if they would recommend the resource for the library, 84.8% would recommend clin-eguide, 75% would recommend bmj, and 67.6% would recommend nrc. the open-ended questions generally indicated that respondents would recommend all three databases. regarding how respondents preferred training on these resources, users preferred online tutorials to learn clin-eguide and nrc. users preferred website tips and instruction to learn bmj. the least preferred methods of training for all three databases were live demonstration and classroom training. conclusion – none of the databases particularly stood out with regard to usability and content. the respondents generally liked all three databases. it is important to note, however, that detailed comparisons among the databases were difficult to make. first, respondents did not always give an answer for all three databases for a given question. because of this, and to present a more meaningful analysis, the researchers often reported the number of respondents who answered a certain way as a percentage rather than a number. second, although the respondents generally liked all three databases, opinions about likes and dislikes were not consistent among respondents. for example, one respondent thought the nrc and clin-eguide interfaces were more difficult to navigate than bmj, while another respondent thought bmj had the harder-to-navigate interface. the researchers felt that respondents’ prior experience with the databases may have influenced their preferences. they were unable to determine if the respondents’ professional interests had any influence on their preferences. inconsistent responses made it difficult for researchers to assign an overall value to a given database. therefore, this survey did not help to make definitive purchasing decisions. the researchers felt they would have to look at each resource much more carefully to make such a decision. the researchers noted several ideas for future research of this sort. they acknowledged that the sample size was not big enough to determine statistical significance and thought that better marketing of the questionnaire may have increased the numbers. they also thought that it would be interesting to observe the respondents using the databases in real-time to find out such things as: what information they require in their daily work, how long it takes them to find it, and what they do with it once they find it. commentary point-of-care databases are a relatively new addition to the clinicians’ toolbox. the purpose of such databases is to give clinicians quick access to reliable information that can be used to treat their patients in real time. as with any database, librarians need to evaluate point-of-care databases before making expensive purchasing decisions. the objective of this study was to do just that. while the researchers met this objective, the results were less than satisfying. out of the three databases, there was no clear winner that would lead the researchers to purchase one database over the others. thus, the value of this study is not necessarily the evaluation of the databases at hand but the implications for further research of this type. one problem was the small sample size, a problem further exacerbated by the fact that respondents did not always give an answer for each of the three databases. a larger number of respondents could have alleviated this. in order to get more respondents, the researchers thought in hindsight that they could have posted a note about the survey prior to users entering the databases, and then directed users to the survey when they were finished searching. additional marketing efforts could also have helped increase the number of respondents. judging from the results of this study, the researchers could hardly make a sound purchasing decision. it is not enough that the respondents would probably be happy with any of the three databases. it was clear from the open-ended questions that users had definite opinions about the features and content of the different databases. what is missing from this research is perhaps a connection between the users’ comments and the type of task they were trying to accomplish at the time of comment. is one database better than the others for point-of-care versus post-care versus non-care information research? the researchers also could not determine if the clinicians’ professional interests influenced their preferences. do oncology nurses prefer one database while plastic surgeons prefer another? such information would help librarians to make purchasing decisions within the framework of clinicians’ work and interests. overall, this study is a valuable contribution to the evaluative process of point-of-care databases. the researchers have provided a good basic analysis of three important point-of-care databases: bmj point-of-care, clin-eguide, and nursing reference centre. more importantly, however, they have provided valuable insight for future research of this type. evidence based library and information practice evidence based library and information practice 2011, 6.4 169 evidence based library and information practice classic salton and buckley’s landmark research in experimental text information retrieval a review of: salton, g., & buckley, c. (1990). improving retrieval performance by relevance feedback. journal of the american society for information science, 41(4), 288–297. reviewed by: christine f. marton adjunct instructor faculty of information, university of toronto toronto, ontario, canada email: christine.marton@utoronto.ca received: 08 may 2011 accepted: 02 nov. 2011 © 2011 marton. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objectives – to compare the performance of the vector space model and the probabilistic weighting model of relevance feedback for the overall purpose of determining the most useful relevance feedback procedures. the amount of improvement that can be obtained from searching several test document collections with only one feedback iteration of each relevance feedback model was measured. design – the experimental design consisted of 72 different tests: 2 different relevance feedback methods, each with 6 permutations, on 6 test document collections of various sizes. a residual collection method was utilized to ascertain the “true advantage provided by the relevance feedback process.” (salton & buckley, 1990, p. 293) setting – department of computer science at cornell university. subjects – six test document collections. methods – relevance feedback is an effective technique for query modification that provides significant improvement in search performance. relevance feedback entails both “term reweighting,” the modification of term weights based on term use in retrieved relevant and non‐relevant documents, and “query expansion,” which is the addition of new terms from relevant documents retrieved (harman, 1992). salton and buckley (1990) evaluated two established relevance feedback models based on the vector space model (a spatial model) and the probabilistic model, respectively. mailto:christine.marton@utoronto.ca� evidence based library and information practice 2011, 6.4 170 harman (1992) describes the two key differences between these competing models of relevance feedback. [the vector space model merges] document vectors and original query vectors. this automatically reweights query terms by adding the weights from the actual occurrence of those query terms in the relevant documents, and subtracting the weights of those terms occurring in the non‐relevant documents. queries are automatically expanded by adding all the terms not in the original query that are in the relevant documents and non‐relevant documents. they are expanded using both positive and negative weights based on whether the terms are coming from relevant or non‐relevant documents. yet, no new terms are actually added with negative weights; the contribution of non‐relevant document terms is to modify the weighting of new terms coming from relevant documents. . . . the probabilistic model . . . is based on the distribution of query terms in relevant and non‐relevant documents, this is expressed as a term weight, with the rank of each retrieved document then being the sum of the term weights for terms contained in the document that match query terms. (pp. 1‐2) second, while the vector space model “has an inherent relationship between term reweighting and query expansion” (p. 2), the probabilistic model does not. thus, query expansion is optional, but given its usefulness, various schemes have been proposed for expanding queries using terms from retrieved relevant documents. in the salton and buckley study 3 versions of each of the two relevance feedback methods were utilized, with two different levels of query expansion, and run on 6 different test collections. more specifically, they queried test collections that ranged in size from small to large, and that represented different domains of knowledge, including medicine and engineering with 72 experimental runs in total. salton and buckley examined 3 variants of the vector space model, the second and third of which were based on the first. the first model was the classic rocchio algorithm (1971), which uses reduced document weights to modify the queries. the second model was the “ide regular” algorithm, which reweights both relevant and non‐relevant query terms (ide, 1971). and the third model was the “ide dec‐ hi” algorithm, which reweights all identified relevant items but only one retrieved nonrelevant item, the one retrieved first in the initial set of search results (ide & salton, 1971). as well, 3 variants of the probabilistic model developed by s.e. robertson (robertson, 1986; robertson & spark jones, 1976; robertson, van rijsbergen, & porter, 1981; yu, buckley, lam, & salton, 1983) were examined: the conventional probabilistic approach with a 0.5 adjustment factor, the adjusted probabilistic derivation with a different adjustment factor, and finally an adjusted derivation with enhanced query term weights. the 6 vector space model and probabilistic model relevance feedback techniques are described in table 3 (p. 293). the performance of the first iteration feedback searches were compared solely with the results of the initial searches performed with the original query statements. the first 15 documents retrieved from the initial searches were judged for relevance by the researchers and the terms contained in these relevant and non‐relevant retrieved items were used to construct the feedback queries. the authors utilized the residual collection system, which entails the removal of all items previously seen by the searcher (whether relevant or not), and to evaluate both the initial and any subsequent queries for the reduced collection only. both multi‐valued (partial) and binary weights (1=relevant, 0=non‐relevant) were used on the document terms (table 6, p. 296). also, two types of query expansion method were evidence based library and information practice 2011, 6.4 171 applied: expanded by the most common terms and expanded by all terms (table 4, p. 294). while not using any query expansion and relying solely on reweighting relevant and non‐relevant query terms is possible, this option was not examined. three measures were calculated to assess relative relevance feedback performance, the rank order (recall‐ precision value); search precision (with respect to the average precision at 3 particular recall points of 0.75, 0.50, and 0.25), and the percentage improvement in the 3‐point precision feedback and original searches. main results – the best results are produced by the same relevance feedback models for all test collections examined, and conversely, the poorest results are produced by the same relevance feedback models, (tables 4, 5, and 6, pp. 294‐296). in other words, all 3 relevance feedback algorithms based on the vector space retrieval model outperformed the 3 relevance feedback algorithms based on the probabilistic retrieval model, with the best relevance feedback results obtained for the “ide dec hi” model. this finding suggests that improvements in relevance from term reweighting are attributable primarily to reweighting relevant terms. however, the probabilistic method with adjusted derivation, specifically considering the extra weight assignments for query terms, was almost as effective as the vector space model relevance feedback algorithms. paired comparisons between full query expansion (all terms from the initial search are utilized in the feedback query) and partial query expansion by the most common terms from the relevant items, demonstrate that full expansion is better, however, the difference between expansion methods is small. conclusions – relevance feedback methods that reformulate the initial query by reweighting existing query terms and adding new terms (query expansion) can greatly improve the relevance of search results after only one feedback iteration. the amount of improvement achieved was highly variable across the 6 test collections, from 50% to 150% in the 3‐point precision. other variables thought to influence relevance feedback performance were initial query length, characteristics of the collection, including the specificity of the terms in the collection, the size of the collection (number of documents), and average term frequency in documents. the authors recommend that the relevance feedback process be incorporated into operational text retrieval systems. commentary although not widely stated, it is implicitly understood that information retrieval is the foundation of evidence based practice. evidence is obtained by searching one or more text collections of peer‐reviewed journal literature and obtaining relevant articles. the goal of conducting a search is to retrieve relevant documents from a text collection. a searcher enters a query, also commonly referred to as a search statement, into the search interface for an information retrieval system (search engine), and a ranked list of search results is retrieved and presented to the searcher. the search results should meet the user’s specific information need. thus, relevance is a key concept in information retrieval. relevance refers to the match based on topicality between the query terms entered into the search interface of the information retrieval system and the items retrieved. there must be a match between the terms in the query and the terms in the documents retrieved, with documents of highest relevance retrieved first. techniques that improve the effectiveness of the search process are those that increase relevance (croft, metzler, & strohman, 2010; manning, raghavan, & schütze, 2008; meadow, 1992; salton & buckley, 1990; salton & mcgill, 1986). relevance feedback is a query reformulation technique invented in the 1960s that has demonstrated effectiveness in improving search performance by improving the correspondence between query terms and document terms. relevance feedback algorithms or formulas are associated with evidence based library and information practice 2011, 6.4 172 retrieval models. today, 3 retrieval models dominate: the vector space model, probabilistic models, and the widely used boolean model. each retrieval model is characterized by a unique ranking algorithm to produce a list of documents that have been scored in order of highest to lowest relevance. relevance feedback was originally developed from the vector space model created by gerard salton in the 1960s. all relevance feedback algorithms, irrespective of the underlying retrieval model, make use of query term reweighting and query expansion (the addition of new terms to the revised query). both relevance feedback processes can be manual (user‐driven) or automated (computer‐based), sometimes referred to as “pseudo‐relevance,” or “partially automated.” also, all relevance feedback algorithms rely on text statistics, generally the frequency of query term occurrences in individual documents in a document collection and also the frequency of query terms in the document collection overall (salton, 1968; salton, 1971; salton & buckley, 1990; croft, metzler, & strohman, 2010). in their seminal article, “improving retrieval performance by relevance feedback”, salton and buckley (1990) conducted empirical research on the relative performance of two relevance feedback processes based on the vector space model and the probabilistic model, respectively. since its publication in jasis twenty years ago, their work has been cited over 400 times in the web of science’s citation databases, which include the following citation indexes: science citation index expanded (sci‐expanded); social sciences citation index (ssci); arts & humanities citation index (a&hci); conference proceedings citation index‐ science (cpci‐s), and conference proceedings citation index‐social science & humanities (cpci‐ssh). there are many recent citations in the academic literature for salton and buckley’s article, which demonstrates the ongoing importance of their research in several specialized areas of information retrieval (ir) research and practice. for medical librarians, novel ranking tools for pubmed, such as ranksvm (yu et al., 2010) and misearch (states, ade, wright, bookvich, & athey, 2009), which reference salton and buckley’s work, hold promise for improving the relevance of pubmed search results. other areas of ir research indebted to salton and buckley’s research include: image ir (rahman, antani, & thoma, 2011; su, huang, yu, & tseng, 2011; arevalillo‐ herráez, ferri, & moreno‐picto, 2011; setchi, tang, & stankov, 2011); setchi & bouchard, 2010; kwan, gao, guo, & kameyama, 2010; setchi, tang, & bouchard, 2009); video ir (vallet, hopfgartner, jose, castells, 2011; yadav & aygun, 2009); web ir (kaptein & kamps, 2011; xu, luo, yu, & xu, 2011; hamdi, 2011; li, otsuka, & kitsuregawa, 2010; fu, 2010; gabrilovich et al., 2009; nauer & toussaint, 2009; yumoto, mori, & sumiya, 2009; kuppusamy & aghila, 2009), web commerce (verma, tiwari, & mishra, 2011), web 2.0 rss feed content (teng, liu, & ren, 2010), and multilingual ir (he & wu, 2011; he, tu, luo, & li, 2009; tu, he, & luo, 2009). more broadly, salton’s vector space model continues to influence current ir research, including patent ir (chen & chiu, 2011); image ir (martinet, chiaramella, & mulhern, 2011; berber & alpkrocack, 2010); tv content ir (yu & zhou, 2009); multilingual ir (chew, bader, helmreich, abdelali, & verzi, 2011; rajan, ramalingam, ganesan, palanivel, & palaniappan, 2009); web search (wang & bai, 2009), and web 2.0 blog posts of fiction reviews in particular (chen, lee, huang, & kuo, 2010). overall, their experimental information retrieval research paper represents a classic both because of its findings and its rigorous study design that utilized the residual collection system (salton, 1968; salton, 1971; salton & mcgill, 1986; salton & buckley, 1990). while salton and buckley’s empirical study of relevance feedback processes is regarded as a classic in the field of experimental information retrieval, several issues are evident. first, although the authors assert that the initial search statement should be a tentative query evidence based library and information practice 2011, 6.4 173 or trial run, conducted solely for the purpose of retrieving several relevant documents from a document collection (p. 288), their study design utilizes “high‐quality initial searches . . . for experimental purposes.” (p. 291). second, the focus is on topical relevance (the match between query terms and terms in documents), whereas the authors do not examine user relevance, which includes a consideration of socio‐cognitive factors. third, test collections are utilized instead of actual information retrieval systems. all of these characteristics of the study design point to the controlled environment of experimental information retrieval research, which limits its generalizability to pragmatic, real‐life searches. the study design utilized by salton and buckley (1990) has several shortcomings, foremost of which is a bias in favor of relevance feedback processes derived from the vector space model. this bias is manifested in several ways. first, although 6 relevance feedback processes are compared, 3 based on the vector space model and 3 based on the probabilistic model (a seemingly balanced approach to the evaluation of relevance feedback performance), results from the experimental runs for the probabilistic adjusted derivation are not presented in tables 4 and 5 – a curious omission. of greater importance is the supremacy of all 3 vector space model‐based relevance feedback processes over all 3 probabilistic models of relevance feedback examined. the authors attribute the poorer performance of the probabilistic model‐based relevance feedback processes to the indirect method of reweighting terms and the greater emphasis on non‐relevant terms in the probabilistic methods. yet, another plausible explanation points to the methods used by the authors to revise or adjust the derivation in the probabilistic model relevance feedback algorithms. it is plausible that other methods of derivation for term reweighting could result in greater success in relevance feedback performance for the probabilistic relevance feedback processes. another contentious issue concerns the methods used for query term expansion and the use of only one feedback iteration. harman (1992) examined different query term expansion methods with probabilistic relevance feedback processes and many feedback iterations to determine optimal relevance feedback. her rigorous approach to experimental text retrieval using the large‐ scale nist collection demonstrated that multiple feedback iterations and different query term expansion methods with relevance feedback processes based on the probabilistic model can lead to substantial query improvement, thus refuting to some extent the findings reported in salton and buckley’s paper. references arevalillo‐herráez, m., ferri, f. j., & moreno‐ picot, s. (2011). distance‐based relevance feedback using a hybrid interactive genetic algorithm for image retrieval. applied soft computing journal, 11(2), 1782‐1791. doi:10.1016/j.asoc.2010.05.022 berber, t., & alpkrocak, a. (2010). an extended vector space model for content‐based image retrieval. in c. peters, b. caputo, j. gonzalo, g. h. f. jones, j. kalpathy‐cramer, h. müller, & t. tsikrika (eds.). multilingual information access evaluation ii: multimedia experiments, lecture notes in computer science, 6242, 219‐222. chen, h. w., lee, k. r., huang, h. h., & kuo, y. h. (2010). unsupervised subjectivity‐lexicon generation based on vector space model for multi‐ dimensional opinion analysis in blogosphere. in d.‐s. huang, z. zhao, v. bevilacqua, j. c. figueroa (eds.). advanced and intelligent computing theories and applications. lecture notes in computer science, 6215, 372‐379. chen, y. l., & chiu, y. t. (2011). an ipc‐based vector space model for patent retrieval. information processing & evidence based library and information practice 2011, 6.4 174 management, 47(3), 309‐322. doi:10.1016/j.ipm.2010.06.001 chew, p. a., bader, b. w., helmreich, s., abdelali, a., & verzi, s. j.. (2011). an information‐theoretic, vector‐space model approach to cross‐language information retrieval. natural language engineering, 17(1), 37‐70. doi:10.1017/s1351324910000185 croft, w. b., metzler, d., & strohman, t. (2010). search engines: information retrieval in practice. boston: addison‐ wesley. fu, x. (2010). towards a model of implicit feedback for web search. journal of the american society for information science and technology, 61(1), 30‐49. doi:10.1002/asi.21198 gabrilovich, e., broder, a., fontoura, m., joshi, a., josifovski, v., riedel, l., zhang, t. (2009). classifying search queries using the web as a source of knowledge. acm transactions on the web, 3(2), 1‐28. doi:10.1145/1513876.1513877 hamdi, m. s. (2011). somse: a semantic map based meta‐search engine for the purpose of web information customization. applied soft computing journal, 11(1), 1310‐1321. doi:10.1016/j.asoc.2010.04.004 harman, d.k. (1992). relevance feedback revisited. in n. j. belkin, p. ingwersen, & a. m. pejtersen (eds.). proceedings of the 15th annual international acm sigir conference on research and development in information retrieval. new york: acm. he, d., & wu, d. (2011). enhancing query translation with relevance feedback in translingual information retrieval. information processing and management, 47(1), 1‐17. doi:10.1016/j.ipm.2009.09.008 he, t. t., tu, x. h., luo, j. & li, f. (2009). chinese query expansion based on topic‐relevant terms. information: an international interdisciplinary journal, 12(2), 369‐376. ide, e. (1971). new experiments in relevance feedback. in g. salton (ed.). the smart retrieval system: experiments in automatic document processing. (pp. 337‐ 354). englewood cliffs, nj: prentice hall inc. ide, e., & salton, g. (1971). interactive search strategies and dynamic file organization in information retrieval. in g. salton (ed.). the smart retrieval system: experiments in automatic document processing. (pp. 373‐393). englewood cliffs, nj: prentice hall, inc. kaptein, r., & kamps, j. (2011). explicit extraction of topical context. journal of the american society for information science and technology, 62(8), 1548‐ 1563. doi: 10.1002/asi.21563 kuppusamy, k. s., & aghila, g. (2009). feast ‐ a multistep, feedback centric, freshness oriented search engine. in 2009 ieee international advance computing conference. (pp. 997‐1001). doi:10.1109/iadcc.2009.4809151 kwan, p. w., gao, j., guo, y., & kameyama, k. (2010). a learning framework for adaptive fingerprint identification using relevance feedback. international journal of pattern recognition and artificial intelligence, 24(1), 15‐38. doi:10.1142/s0218001410007841 li, l., otsuka, s., & kitsuregawa, m. (2010). finding related search engine queries by web community based query enrichment. world wide web, 13(1‐2), 121‐142. doi:10.1007/s11280‐009‐0077‐1 manning, c. d., raghavan, p., & schütze, h. (2008). introduction to information evidence based library and information practice 2011, 6.4 175 retrieval. new york: cambridge university press. martinet, j., chiaramella, y., and mulhern, p. (2011). a relational vector space model using an advanced weighting scheme for image retrieval. information processing & management, 47(3), 391‐ 414. doi:10.1016/j.ipm.2010.10.003 meadow, c. t. (1992). text information retrieval systems. san diego: academic press. nauer, e., & toussaint, y. (2009). crechaindo: an iterative and interactive web information retrieval system based on lattices. international journal of general systems, 38(4), 363‐378. doi:10.1080/03081070902857613 rajan, k., ramalingam, v., ganesan, m., palanivel, s., and palaniappan, b. (2009). automatic classification of tamil documents using vector space model and artificial neural network. expert systems with applications, 36(8), 10914‐10918. doi:10.1016/j.eswa.2009.02.010 rahman, m. m., antani, s. k., & thoma, g. r. (2011). a query expansion framework in image retrieval domain based on local and global analysis. information processing and management, 47(5), 676‐ 691. doi:10.1016/j.ipm.2010.12.001 robertson, s. e. (1986). on relevance weight estimation and query expansion. journal of documentation, 42(3), 182‐ 188. doi:10.1108/eb026793 robertson, s. e., & sparck jones, k. (1976). relevance weighting of search terms. journal of the american society for information science, 27(3), 129‐146. doi:10.1002/asi.4630270302 robertson, s. e., van rijsbergen, c. j., & porter, m. f. (1981). probabilistic models of indexing and searching. in r. n. oddy, s. e. robertson, c. j. van rijsbergen, & p. w. williams (eds.) information retrieval research (pp. 35‐56) london: butterworths. rocchio, jr., j. j. (1971). relevance feedback in information retrieval. in salton, g. (ed.). the smart retrieval system: experiments in automatic document processing, pp. 313‐323. englewood cliffs, nj: prentice hall inc. salton, g. (1968). automatic organization and retrieval. new york: mcgraw‐hill. salton, g. (1971). relevance feedback and the optimization of retrieval effectiveness. the smart retrieval system: experiments in automatic document processing, pp. 324‐336. englewood cliffs, nj: prentice‐hall inc. salton, g., & mcgill, m. j. (1983). introduction to modern information retrieval. new york: mcgraw‐hill. salton, g., & buckley, c. (1990). improving retrieval performance by relevance feedback. journal of the american society for information science, 41(4), 288–297. setchi, r., tang, q., & stankov, i. (2011). semantic‐based information retrieval in support of concept design. advanced engineering informatics, 25(2), 131‐146. doi:10.1016/j.aei.2010.07.006 setchi, r., & bouchard, c. (2010). in search of design inspiration: a semantic‐based approach. journal of computing and information science in engineering, 10(3), 031006. retrieved 22 nov. 2011 from http://scitation.aip.org/getpdf/servlet/ getpdfservlet?filetype=pdf&id=jcisb 6000010000003031006000001&idtype=c vips&prog=normal setchi, r., tang, q., & bouchard, c. (2009). ontology‐based concept indexing of images. in j. d. velásquez & s. a. ríos evidence based library and information practice 2011, 6.4 176 (eds.). knowledge-based and intelligent information and engineering systems: lecture notes in artificial intelligence, 5711, 293‐300. states, d. j., ade, a. s., wright, z. c., bookvich, a. v., & athey, b. d. (2009). misearch adaptive pubmed search tool. bioinformatics, 25(7), 974‐976. doi:10.1093/bioinformatics/btn033 su, j.‐h., huang, w.‐.j, yu, p. s., & tseng, v. s. (2011). efficient relevance feedback for content‐based image retrieval by mining user navigation patterns. ieee transactions on knowledge and data engineering, 23(3), 360‐372. doi:10.1109/tkde.2010.124 teng, z., liu, y., & ren, f. (2010). create special domain news collections through summarization and classification. ieej transactions on electrical and electronic engineering, 5(1), 56‐61. doi:10.1002/tee.20493 tu, x., he, t., & luo, j. (2009). term relevance estimation for chinese query expansion. in ieee 2009 international conference on natural language processing and knowledge engineering. (139‐145) piscataway, nj: ieee. vallet, d., hopfgartner, f., jose, j. m., & castells, p. (2011). effects of usage‐ based feedback on video retrieval: a simulation‐based study. acm transactions on information systems, 29(2). doi:10.1145/1961209.1961214 verma, a., tiwari, m. k., & mishra, n. (2011). minimizing time risk in on‐line bidding: an adaptive information retrieval based approach. expert systems with applications, 38(4), 3679‐ 3689. doi:10.1016/j.eswa.2010.09.025 wang, f.j., & bai, z.y. (2009). the web software mining based on vector space model. in h. zhang (ed.). icfin 2009: 2009 first international conference on future information networks. (pp. 275‐279) piscataway, nj: ieee. xu, z., luo, x., yu, j., & xu, w. (2011). mining web search engines for query suggestion. concurrency computation: practice and experience, 23(10), 1101‐ 1113. doi:10.1002/cpe.1689 yadav, t., & aygün, r. s. (2009). i‐quest: an intelligent query structuring based on user browsing feedback for semantic retrieval of video data. multimedia tools and applications, 43(2), 145‐178. doi:10.1007/s11042‐009‐0262‐3 yu, c. t., buckley, c., lam, k., & salton, g. (1983). a generalized term dependence model in information retrieval. information technology: research and development, 2, 129‐154. yu, h., kim, t., oh, j., ko, i., kim, s., & han, w.‐s. (2010). enabling multi‐level relevance feedback on pubmed by integrating rank learning into dbms. bmc bioinformatics, 11(suppl. 2), 1‐10. doi:10.1186/1471‐2105‐11‐s2‐s6 yu, z. w., & zhou, x. s. (2009). combining vector space model and category hierarchy model for tv content similarity measure. in w. jiang (ed.). mue 2009: third international conference on multimedia and ubiquitous engineering. (pp. 130‐136) los alamitos, ca: ieee computer society. yumoto, t., mori, y., & sumiya, k. (2009). converting topics of user query sequences for cooperative web search. in k. rose (ed.). proceedings: the seventh international conference on creating, connecting and collaborating through computing. (pp. 121‐127) los alamitos, ca: ieee computer society. / evidence based library and information practice wang, f.j., & bai, z.y. (2009). the web software mining based on vector space model. in h. zhang (ed.). icfin 2009: 2009 first international conference on future information networks. (pp. 275-279) piscataway, nj: ieee. microsoft word es 4460 evidence based library and information practice 2009, 4:1 32 evidence based library and information practice evidence summary online programs and geographic proximity are key determinants of information professionals’ interest in pursuing post-master’s education at the doctoral level a review of: powell, ronald r. and susan e. boling. “post-master’s educational needs of information professionals.” journal of access services 3.4 (2005): 29-43. reviewed by: martha ingrid preddie school of information and library science, university of north carolina at chapel hill chapel hill, nc, united states of america e-mail: mipreddie@gmail.com received: 01 december 2008 accepted: 22 january 2009 © 2009. preddie. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. objective – to investigate post-master’s educational needs and interests of information professionals. design – survey research using print and electronic questionnaires. setting – the geographic area surrounding wayne state university in michigan, united states of america. subjects – members of the library associations of michigan, ohio, indiana, and southwestern ontario, canada. methods – systematic random samples were derived from the membership lists of the library associations in ohio, indiana, and southwestern ontario. paper questionnaires were mailed to those selected. michigan library association’s official policy barred the release of its membership list to researchers. consequently, announcements of the survey were placed in three successive issues of the association’s electronic newsletter. interested members were directed to a web site to complete an electronic version of the questionnaire. this option was also extended to members of the other three library associations. the overall research question was investigated through specific questions that sought to ascertain the overall level of interest in professional library and information studies (lis) education, levels of interest in specific types of programs, factors that favoured or deterred enrolment in doctoral programs, as well as the fields of 33 study that were preferred for combination with lis in doctoral programs. with the exception of demographic type questions (e.g., place of residence and educational qualifications) and two questions that required open ended responses, the questionnaire design encompassed questions with likert scale type responses. analysis of the responses included descriptive statistics, the use of pearson chisquare to determine statistically significant relationships, and, to a lesser extent, content analysis. main results – a total of 270 questionnaires (33%) were returned from three populations: ohio, indiana, and ontario. a self-selected sample of 101 members (6%) of the michigan library association responded. in general, almost 80% of the respondents admitted some importance to adding to their lis qualification. however, only 41% felt that this was important or extremely important. from a choice of six educational offerings, namely, continuing education activities, non-degree master’s coursework, a second master’s degree, post-master’s certification, doctoral programs, and other, continuing education activities was the most valued, by 65.5% of the respondents. participants were asked about their reasons, and the importance of these, for considering or deciding to enrol in an lis doctoral program. the yearning to acquire knowledge was reported by 69.7% as the major reason, followed by 45.8% of the respondents who cited the wish to increase their income potential. in terms of major factors, prestige received the lowest rating, 21.1%. the time involved (73.8%), cost (66.3%), and distance from the program (63.2%) were cited as the major deterrents to enrolling in doctoral programs. when asked about the likelihood of pursuing a doctoral lis program in combination with business administration, computer science, or without any combination, “not likely” was the most popular choice. those who were very likely or likely to pursue a joint program totalled approximately 30% of the respondents, while 37% indicated an interest in undertaking a doctorate in lis only. the most frequent reasons proffered by those who selected “not likely” or “definitely not” for any of the three doctoral offerings included lack of interest, mitigating factors (e.g. time, cost, age, and program location), unfavourable cost/benefit analysis, preference for another area of knowledge, and the view that ph.d.s were only useful for university faculty. given the option to name subject areas that they desired to see combined with lis in a doctoral program, 23 of 101 respondents proposed education, while 19 opted for public administration. a question inviting any other comments on the issue of post-master’s education yielded a predominant desire for “non traditional instruction,” particularly online courses as well as classes held in convenient locations. in terms of relationships between geographic location and factors that influenced interest in enrolling in a doctoral program, the desire “to become a more effective manager,” and “other” were statistically significant. the analysis also revealed a strong positive relationship between willingness to pursue an lis-only doctoral program and the availability of such a program in geographic proximity. similarly, there was a strong relationship between willingness to pursue an lis program in close vicinity and the importance that was given to further lis education. conclusion – online instruction and geographic proximity are key determinants of information professionals’ interest in pursuing post-master’s education at the doctoral level. continuing education activities, non-degree coursework, and 34 certificate programs are preferred over doctoral lis programs, despite the finding that the major reason for enrolling in doctoral lis programs is to fulfill the desire for knowledge acquisition. schools offering library and information science studies need to explore options for providing distance-education doctoral lis programs as a means of reversing the current shortage of lis faculty. commentary this research was critically appraised using the eblip critical appraisal checklist developed by lindsay glynn. the study investigated the post-master’s education needs of information professionals. the methodology employed was a questionnaire survey administered to representative samples of members of three of the four library associations in the geographic area of choice. the survey methodology and sampling technique were suitable for the research question. the researchers admitted, however, that the self-selected nature and small size of the sample of respondents from the michigan library association “limited the generalizability of the results for that group” (32). with regard to this group, the use of an additional method of data collection (e.g. focus group) could have served to increase the validity of the findings, since it is possible that the postmaster’s educational needs of those who did not volunteer to respond, could be different. additionally, while the overall response rate was 33%, the response rates for indiana and ontario were 29% and 13% respectively. these response rates suggest the possibility of some degree of non-response bias. the study stated that 43 respondents possessed qualifications that were not an mlis. it would have been useful for an analysis to be conducted in order to ascertain whether their responses differed significantly from persons with the mlis qualification. nevertheless, despite these limitations the conclusions accurately reflect the analysis. no suggestions were provided for further areas of research. nonetheless, future research of this nature would benefit from an examination of information professionals’ post-master’s educational needs in the context of the practice specializations of respondents (e.g., special, academic, public, and school librarianship). such a study would expand the knowledge base pertinent to the post-master’s educational needs of informational professionals beyond current literature, which the researchers themselves acknowledged as having focused predominantly on the field of academic librarianship. the researchers chose to focus their study on the geographic area close to wayne state university. replication of this research in other geographic areas would be useful for comparative purposes and generalizability. this article is significant to the field of lis since it provides insight into the wide spectrum of interests that practitioners have in the sphere of post-mlis education. in terms of library and information practice, lis schools as well as professional associations need to find ways to provide the variety of desired post-mlis educational programs, and in the case of doctoral studies, in online mode. ph.d. qualified librarians are needed to serve as faculty for mlis programs. this need was underscored by the researchers who alluded to looming shortage of teaching faculty in lis schools (berry 20). moreover, through the process of obtaining a ph.d., librarians will build competencies in undertaking high quality quantitative and qualitative research, including longitudinal studies. such research is requisite not only to increasing the literature in the lis field, but also to improving the knowledge base of the profession, towards facilitating 35 evidence-based library and information practice. works cited berry, john. “lis boom spurs new faculty push.” library journal 124.4 (2002): 20. glynn, lindsay. “a critical appraisal tool for library and information research.” library hi tech 24.3 (2006): 387-99. microsoft word es_1660 evidence based library and information practice 2008, 3:3 57 evidence based library and information practice evidence summary the information practices of physical science librarians differ from those of the scientific community: more research is needed to characterize specific information seeking and use a review of: brown, cecilia m. and lina ortega. “information-seeking behavior of physical science librarians: does research inform practice?” college & research libraries 66.3 (2005): 23147. reviewed by: carol perryman trln doctoral fellow school of information & library science, university of north carolina at chapel hill chapel hill, nc, united states of america e-mail: cp1757@gmail.com received: 13 april 2008 accepted: 17 june 2008 © 2008 perryman. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – as part of a larger study exploring the information environments of physical science librarians (ortega & brown), the authors’ overall objective for this study is to profile physical science librarians’ information behaviours. the authors’ two-part hypothesis was that first, peer-reviewed journals would be preferred over all other sources for research dissemination, resembling the preferences of scientists, and second, that peer-to-peer consultation would predominate for practice-oriented decisions. design – mixed methods: survey questionnaire followed by citation and content analysis. setting – five internationally disseminated professional association electronic mailing lists whose readership comprised those with interests in science librarianship: the american library association (ala) science and technology section; the american society for information science & evidence based library and information practice 2008, 3:3 58 technology (asist) science and technology information special interest group; the special library association (sla) chemistry division and its physicsastronomy-mathematics division; and the american geological institute geoscience information society. subjects – seventy-two physical science librarians voluntarily responding to an online survey. methods – a questionnaire was distributed to inquire about physical science librarians’ professional reading practices as well as their perceptions about the applicability of research to their work. participants were asked to rank preferences among 11 resource types as sources supporting daily business, including personal communication, conference attendance, electronic mailing lists, and scholarly journals. differences between the mean rankings of preferences were tested for significance by applying the friedman test with p>0.0005. journals identified most frequently were analyzed using the institute for scientific information’s (isi) web of science index and ulrich’s periodical index to measure proportions of research and nonresearch citations, as well as the general topic areas covered by the journals. next, content analysis was performed for the years 1995, 1997, and 2000 in order to characterize research methodologies used in the previously identified journals according to a previously tested schema (buscha & harter). results from this portion of the study were compared with participants’ responses about journal usage. main results – librarians reported using personal communication (both face-to-face and electronic mailing lists) more frequently as a means of information gathering than professional journals, web sites, conferences, trade publications, monographs, or ‘other’ resources. variations in responses appeared to correlate with years in the profession and in the respondents’ time in their current positions, although there are indications that the importance of all information resources to practice and research declines over time. the relative importance of resources is also shown in time spent reading journal literature, less than 5 hours per week for 86% of participants. conclusion – for the first hypothesis, the authors found that unlike scientists, survey participants did not prefer research publications as vehicles for dissemination of their research results. for the second, librarians ranked peer-reviewed journals third in preference after personal communication and electronic mailing lists as sources of information supporting daily practice, supporting the second hypothesis that respondents would emulate the information use practices of mathematicians. commentary critical evaluation tools used for this evidence summary include the worksheet for critical appraisal of qualitative research, a checklist provided by mcmaster university, canada, and the evaluation tool for mixed method studies, from the university of salford, uk. the authors perform a valuable service to the profession in this inquiry, which is without precedent for physical science librarians. the methods used appear to be appropriate to the research questions, although there is no evidence of instrument validation. the authors address some of the limitations of the study: response rates are small, and estimating the generalizability of findings is not possible due to incomplete knowledge about the electronic mailing list participants. based upon the numbers evidence based library and information practice 2008, 3:3 59 provided, the survey participants (n=72) represent just 2% of subscribers to the electronic mailing lists named (n=3,560), and there are no data available to enable us to know what percentage of these subscribers are physical science librarians, or how representative the respondents might be of the entire population of physical science librarians. although the study invited international participation from physical science librarians, there is no apparent attempt to examine geographical differences: all findings are generic to the sample. an additional weakness with regard to participant demographics is the apparent lack of information about academic background. inquiry about coursework or experiences in the sciences and the participants’ valuation of the rigour of the research obtained in the highly cited lis journals would have enriched this limited feedback considerably. with these limitations the authors nonetheless claim to have captured a snapshot of the information-seeking behaviours of the population of physical science librarians, but due to the limitations mentioned, and the voluntary nature of the questionnaire, this claim seems questionable: although only a small fraction of the total number of the list subscribers responded to the questionnaire, the consistency of the answers received suggest that those responding provide a representative snapshot of the current research practices of physical science librarians. (235) unlike the authors’ larger study (ortega & brown), the survey questions were not included, which constitutes a considerable barrier to critical evaluation of the research. questions for analysis in both studies are drawn from the same survey, though the survey questions listed in that work are selective, and include no questions about reading habits or preferences. understanding space limitations of paper publications, the authors would be well advised to provide access to the entire set of research questions as a way to encourage replication and to assist with improvement of practitioner research and critical evaluation skills. one of the problems wrought by absence of the survey questions for critical evaluation (and for replication) is that the reader can have no clear understanding whether the journals identified were most read, or most consulted for decision or research support. it also remains unclear whether participants read research articles published in the journals named, as opposed to the nonresearch content measured by the authors (regardless of study type), although the authors appear to have assumed this was the case (241). participants were asked to “describe their most recent application of the research literature,” (236) and reported using it to learn about technology, to prepare for teaching, and to make collection decisions, with 11% using lis research literature for their own research, although here a follow-up question about which journals provided literature for that purpose would have helped with a linkage between resource and application in practice. questions about the librarians’ assessment of the quality of research literature they have ranked would also be useful additions that would improve the applicability of the findings. the study under examination poses questions to be examined by further research, perhaps using the same methods, strengthened by access to the survey tools. physical science librarians might be identified through association membership lists so that sample sizes and response rates evidence based library and information practice 2008, 3:3 60 could be shown. requests for more detail in the survey questions, such as career and educational history, might also have improved the usefulness of this study. for example, participants were asked to name most often-read professional journals, and then to value their applicability in practice – but the question was recall-based as opposed to situational (e.g., what journals do you find most useful in making practicerelated decisions, or conducting practicerelated research?): having ranked the resources [in terms of importance to their daily practices], the librarians then were asked to gauge the importance of these resources overall, as well as the lis literatures, to their practice (235). there is a gap between the reported application for lis research literature, and the most frequently read literature, so that using citation rankings and content analysis to characterize the journals named is not as pertinent as it might have been had the study authors linked questions about journals read to journals (and specifically, research articles within the journals) used to support everyday patron support and administrative, continuing education, or research activities. further, in addressing use of the isi web of science index as source of impact factors, the authors did not address the sparse coverage of lis titles or any concerns about limitations of impact factor analysis, which may have affected their findings. while the authors state that “journals designated as highly read by the subjects surveyed were analyzed for the number of citations to both researchoriented as well as ‘other,’ non-research articles using the institute for scientific information’s (isi’s) online citation index” (233), there is no more detailed description of this process that would enable replication. overall, the contribution of this work to our understanding of librarian information behaviours is unquestioned, but it serves best as an intriguing suggestion of where further inquiry might benefit the profession. works cited busha, charles h. and stephen p. harter. research methods in librarianship: techniques and interpretation. new york: academic press inc., 1980. health sciences library, mcmaster university. “worksheet for critical appraisal of qualitative research.” 4 july 2008 <http://hsl.mcmaster.ca/documents/ed ucation/nursing/qualitativerev.doc>. ortega, lina and cecilia m. brown. “the face of 21st century physical science librarianship.” science & technology libraries 26.2 (2005): 71-90. school of nursing, university of salford, salford, greater manchester, uk. “evaluation tool for mixed method studies.” 4 july, 2008 <http://www.fhsc.salford.ac.uk/hcprd u/mixed.htm>. classic   professionalism reconsidered   a review of: bundy, m. l., & wasserman, p. (1968). professionalism reconsidered. college & research libraries, 29(1), 5-26. https://doi.org/10.5860/crl_29_01_5   reviewed by: emily drabinski interim chief librarian the graduate center, city university of new york new york, new york, united states of america email: edrabinski@gc.cuny.edu   received: 1 may 2020                                                               accepted:  7 july 2020      2020 drabinski. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29772     abstract objective – in their 1968 editorial for college & research libraries, mary lee bundy and paul wasserman interrogated the nature of librarianship as a profession. they describe what they see as the limits of contemporary practice and offer ways forward for those concerned with the status of librarians.   design – the article offers an analysis of the question, making use of selected contemporary literature on american librarianship, rather than empirical research or a literature review.   setting – bundy and wasserman locate their critique in the daily work of academic librarians. their descriptions are based on their own observations.   subjects – the authors focus on “the real world in which librarians practice” rather than “abstract academic terms” (p. 7). their subjects are library workers who, by virtue of the mls, are identified as professionals in the library workplace. bundy and wasserman note that these library workers “often spend considerable time being concerned about whether or not they are truly professional” and go on to take up these concerns themselves (p. 5).   methods – bundy and wasserman compare librarianship to “what is customarily considered to constitute professional behavior” (p. 7). their comparison is structured through an analysis of three categories of professional relationships: librarian to client, librarian to institution, and librarian to professional association. this taxonomy of relationships is their own; the authors do not refer to analyses of professionalism in other disciplines such as nursing, social work, or education, fields where similar questions have arisen. the authors describe each of these professional relationships in turn through their own observations as a professor and dean of the library program at the university of maryland.   main results – bundy and wasserman argue that librarianship does not meet the threshold for professional behaviour in any of these three categories of practice. the relationship between the client and the professional requires expertise: “the professional knows” (p. 8). according to the authors, most reference transactions involve questions that “would not overtax the capacity of any reasonably intelligent college graduate after a minimum period of on-the-job training” while an “essential timidity” prevents them from clearly stating what they do know (p. 8). given this, the relationship with the client can never be professional: the client knows as much as or more than the librarian. bundy and wasserman make an exception for children’s librarians, arguing that their clientele benefits from the “close control of the content of collections to reflect excellence” (p. 9). otherwise, librarians are “in awe” of both the expanding bibliographic universe and the “growing sophistication of middle-class readers” (p. 9). unless librarians understand themselves to be experts, and engage as experts with their clients, they cannot be professionals.   professionals also see themselves as superior to their institution, struggling against “institutional authority which attempts to influence [their] behavior and performance norms” (p. 14). the professional resists disciplinary mechanisms that force workers to conform to institutional norms, maintaining authority over their own work.  in bundy and wasserman’s view, librarians instead display “rigid adherence to bureaucratic ritual” where “the intellectual and professional design is sacrificed upon the altar of economic and efficient work procedures” (p. 15). librarians focus on the efficient completion of narrowly defined tasks that enable compliance with institutional demands instead of placing their relationships with clients at the center of their professional life. library administrators encourage this restriction on the status of their employees. the authors argue that the librarian who attempts to maintain a professional relationship “is seen as a prima donna, impatient with necessary work routines, unwilling to help out in emergencies, a waster of time spent in idle conversation with his clientele about their work--renegade and spoiled” (p. 16). acting “like a professional” is incompatible with the ways librarians normally relate within the larger institution.   finally, professional status requires professional associations. these associations should ensure the quality of education in professional programs while facilitating the growth of connections between professional librarians. again, librarianship fails: its professional association is guilty of “accrediting and re-accrediting programs of doubtful merit thereby giving its imprimatur to schools very distant from any ideal or even advanced attainment” (p. 21).  when it gathers librarians together at annual meetings, those committees “consist of members explaining why they have failed to complete assignments or committees which deliberate weightily the means for perpetuating themselves instead of considering the purpose or program, or still others which consume hour after hour preoccupied with minutiae” in organizations that are reduced to “the associational excesses of the ritual, the routine, and the social” (p. 23).   conclusion – for bundy and wasserman, librarianship fails to qualify as a profession because the field cannot lay claim to a particular area of expertise, slavishly follows the rules of the institutions in which it is embedded, and is governed by professional associations that fail to ensure the rigor of professional education while reducing relationship-building to the reproduction of the association itself. unless the field works to become more thoroughly professional, they argue, librarianship cannot advance or innovate, doomed to “not only decline rapidly, but ultimately face obsolescence” (p. 25).   commentary   bundy and wasserman, american professors of librarianship writing in a distinctly american context, published their editorial at the same time that the united states saw both an increase in the social wage and the intensification of struggles over control of it. in colleges and universities, lyndon johnson’s higher education act of 1965 had authorized increased federal funding for colleges and universities in the form of direct aid to institutions and student financial aid programs like federal work-study and subsidized loans for students and their families (hegji, 2018). this infusion of cash led to a boom in enrollment as the number of students in higher education institutions rose over the course of the following decades (snyder, 1993). academic libraries were serving more students with larger budgets than ever before.   at the same time, higher education proved fertile ground for the growth and expansion of social movements. the black panther party was founded by bobby seale and huey p. newton in 1966; the two met as students at merritt college in oakland, california. the weather underground emerged in the same decade at the university of michigan while the university of california, berkeley was host to the free speech movement that would spread to campuses nationwide. just as resources infused the system, struggles for the more equitable distribution of political and social power intensified. librarianship was not exempt from these forces.   librarianship was caught in a familiar tension: should librarians focus on elevating the status of the field by professionalizing like our colleagues in medicine and the law, or should the role we can play in fights for social justice take precedence? during this same decade, the latter impulse made significant headway in the field. the office for intellectual freedom was founded in 1967 and the freedom to read foundation followed in 1969. within the american library association, progressive movements took root as organized entities, including the social responsibility round table (1969), the task force on gay liberation (1971), and the committee on the status of women in librarianship (1976). ethnic affiliates began to be established at the start of the 1970s and included the black caucus of the american library association (1970), the national association of spanish speaking librarians in the united states (1971, now reforma), the chinese american librarians association (1973), the american indian library association (1979), and the asian/pacific librarians association (1980). the late 1960s and 1970s were a golden time for progressive political movements in librarianship.   bundy and wasserman staked their claim in this debate squarely on the side of professionalism as a bid for primacy and position in broader social contexts, including higher education. rather than place efforts into “a wide range of national, international, research, and societal responsibilities for which it is less than ideally equipped” (p. 25), the field ought instead to focus on the substance of librarianship itself. the work of the field should not be about finding ways to participate in or find common cause with broader social movements. instead, the field should focus on boosting professional status in order to be “in the vanguard of new or imaginative directions for librarianship” (p. 25). librarians needed to act more like doctors and lawyers and less like activists or functionaries in order to survive.   writing in 1968, bundy and wasserman’s push for a more robustly professional librarianship can be seen as a gambit for a larger slice of the expanding institutional pie. as budgets and student bodies grew, more resources were up for grabs, and librarians competed with other campus entities for their share. indeed, bundy and wasserman saw professionalization as essential if librarians were to continue to dominate their field: “in order to fulfill their original mandate of serving as guardian of society’s information needs and in order to influence positively the forward motion of progressive information development in a time of competition with other emergent information-oriented disciplines” (p. 6). unless librarians made a strong case for themselves as the true guardians of human knowledge, they were at risk of being replaced by other academic entities on campus. we hear echoes of this in today’s anxieties around the replacement of our reference desks by google, a tool that has essentially replaced the ready reference collections of bundy and wasserman’s day.   bundy and wasserman point to library training programs as a root cause of a library field they saw as essentially clerical. transforming the degree program could also be a solution. instead of focusing library training on “memorizing names of famous modern librarians, committing to memory large sections of classification schedules, cluttering their minds with details of whether certain books have an index and table of contents or not,”  library education should engage broader questions, “studying the reasons for contemporary trends in societal information developments, the logic of comparative systems of classification, the structure of bibliography and information agencies as resources for problem solving, or the personal, organizational, and social group determinants of information need” (p. 20). their argument anticipates the contemporary focus in lis programs on information behavior and social information practices, as well as pointing to critical librarianship as an emerging discourse. debates about what constitutes the best curriculum in lis programs continue along lines similar to those outlined by the authors in 1968 as librarians demand a more rigorous intellectual engagement with information and society, considered essential if librarians are to be more than simply enforcers of narrowly defined bureaucratic norms.   in a short but provocative paragraph, the authors ask whether collective bargaining might offer a straighter route to professional status for american librarians, a group for whom unionization and professionalization might be seen as in conflict. such a suggestion runs counter to many contemporary libraries where union/non-union traces precisely the border of the paraprofessional/professional divide. collective bargaining, bundy and wasserman suggest, is a superior method of producing the “militant group solidarity” they see as necessary for professionalization (p. 23). indeed, as they say in the union movement, management is the best organizer: pulling together as workers around shared grievances and enemies in order to struggle for better wages and working conditions can cohere a group of individuals like little else. the authors stop short of advocating for unions for librarians. like other institutions, they claim, union bureaucracy can be stultifying, “a reinforcement of the very rigid authority structure of libraries which serves now as an impediment to innovation and furtherance of service commitments” (p. 24). in many cases, professional librarians still see unions this way: mechanisms for the production of staff and the rules that govern them that hobble the innovations a more “entrepreneurial” workforce would otherwise produce.   concerns about whether or not librarianship is a profession continue to animate the field, discussed “endlessly” (p. 5) just as bundy and wasserman complained fifty years ago. worry that librarians are too servile, too docile, and too narrow to survive a changing technological and economic landscape continue in the guise of “future-proofing” and appeals to entrepreneurial and other business values. the authors’ complaint that “innovation remains on trial when it should be encouraged” reads as fresh as if it were written today (p. 25). as investment in higher education shrinks, librarians turn to learning analytics and efforts to quantify library value as strategies to ensure their continued existence. associations and institutions steer clear of political conflict by hewing closely to what are described as professional values around free speech and academic freedom.   read in the context of the present, bundy and wasserman’s editorial serves as a warning against too narrow a focus on professional status as the means to the end of a robust and well-resourced academic librarianship. in 1968, just as today, the call to professionalize or face replacement or obsolescence puts the emphasis on the wrong analytic frame. attacks on librarianship must be met on a different terrain. we might instead conceive of disinvestment in higher education and the demands of capital that all units on campus generate profit as the problem. in this case, the solution to our always already impending demise lies not in transforming ourselves, but in transforming the social and economic formations that directly attack librarianship and so many other necessary social goods.   references   hegji, alexandra. (2018). the higher education act: a primer. retrieved from https://fas.org/sgp/crs/misc/r43351.pdf   snyder, thomas d., ed. (1993). 120 years of american education: a statistical portrait. retrieved from https://nces.ed.gov/pubs93/93442.pdf     evidence based library and information practice 2012, 7.3 1 evidence based library and information practice editorial are we bridging the research practice gap? alison brettle editor-in-chief senior lecturer, school of nursing midwifery and social work university of salford salford, united kingdom email: a.brettle@salford.ac.uk 2012 brettle. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. one of the key aims of evidence based library and information practice is to bridge the research practice gap and make the findings of lis research more accessible to library and information practitioners. i’ve therefore been keenly following the uk rilies project (research in librarianship – impact evaluation study; http://lisresearch.org/rilies-project/), which has been looking at ways to increase the impact of library related research for practitioners. the project culminated in a resources briefing (http://lisresearch.org/2012/07/10/research-intopractice-lis-research-resources-briefing/) which i attended, and was thrilled to hear the project team report that the journal was one of the most appreciated sources of lis research for practitioners. my self-congratulation was a little short lived, however, when the next set of findings presented was a range of resources that practitioners had heard of, but had yet to use – and sure enough, eblip was among them. furthermore, other findings of the project included practitioners reporting a need for accessible summaries of research evidence!! the project team concluded that there was no shortage of research resources available to practitioners, but the challenge was finding the best way to make them available and easily accessible. as an open access journal, therefore, we need to work harder on publicizing the work we do. i’ve thus taken on board the recommendation that “here lies an opportunity for those with responsibility for freely available open access repositories of lis research materials to raise awareness of their resources amongst the practitioner communities” (hall, 2012). it is really important that as a journal we do take this message on board, as we have begun to find that the evidence summaries in eblip do make a difference. over the past year, supported by a grant from the canadian association of research libraries, and led by our associate editor for evidence summaries, lorie kloda, we have been conducting a research project into the impact of evidence summaries. the project will be written up in full and the results published elsewhere, but in brief we validated a tool to assess the impact of the summaries on practitioners, used the tool mailto:a.brettle@salford.ac.uk http://lisresearch.org/rilies-project/ http://lisresearch.org/2012/07/10/research-into-practice-lis-research-resources-briefing/ http://lisresearch.org/2012/07/10/research-into-practice-lis-research-resources-briefing/ http://lisresearch.org/2012/03/21/rilies-report-dissemination-for-impact/ evidence based library and information practice 2012, 7.3 2 to survey a number of evidence summary readers, and followed some of these up with more in-depth interviews. initial results are promising, and we have found that evidence summaries impact on knowledge, individual practice, and more widely in the workplace of evidence summary readers. earlier in the summer, we presented the results at a range of national (canada and uk) and international conferences (in europe and the us). hopefully, these presentations (e.g., http://www.slideshare.net/lkloda/kloda-mla2012-impact), will begin to further increase the awareness of evidence summaries – and perhaps turn some of that awareness into action. this issue sees a slight change in the evidence summaries, as described in lorie’s editorial at the beginning of the evidence summary section. so, if you haven’t read an evidence summary before – i challenge you to read one today – and see if it makes a difference to your practice. references hall, h. (2012). rilies report highlights 2: dissemination strategies for impact. in library and information science research coalition. retrieved 21 aug. 2012 from http://lisresearch.org/2012/03/21/riliesreport-dissemination-for-impact/ http://www.slideshare.net/lkloda/kloda-mla-2012-impact http://www.slideshare.net/lkloda/kloda-mla-2012-impact microsoft word art2_920_final evidence based library and information practice 2008, 3:1  33    evidence based library and information practice         article    improving customer satisfaction: changes as a result of customer value discovery      susan mcknight  director of libraries and knowledge resources  nottingham trent university  nottingham, united kingdom  email: sue.mcknight@ntu.ac.uk    mike berrington  deputy university librarian, customer services  nottingham trent university  nottingham, united kingdom  email: mike.berrington@ntu.ac.uk      received: 13 december 2007      accepted: 10 february 2008      © 2008 mcknight and berrington.  this is an open access article distributed under the terms of the creative  commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted  use, distribution, and reproduction in any medium, provided the original work is properly cited.    abstract    objective: to identify gold standard services for customers in an academic library  and determine whether interventions following the identification of customer value  increased student satisfaction.     methods: “customer discovery workshops” were undertaken with academic staff  and undergraduate on‐campus students to provide managers and library staff with  information on the services and resources that customers valued, and what irritated  them about existing services and resources. the impact of interventions was  assessed two years after the research using a university student satisfaction survey  and an independent national student satisfaction survey.    results: the findings resulted in significant changes to the way forward‐facing  customer services were delivered. a number of value adding services were  introduced for the customer. overall customer satisfaction was improved.  evidence based library and information practice 2008, 3:1  34   conclusions: the customer value discovery research has created a culture of  innovation and continuous improvement. an operational plan was introduced to  track activity and performance against the objectives identified in the customer  value research. however, there is a constant need to innovate.    introduction    in february and march 2005, libraries and  learning resources (llr) at nottingham  trent university (ntu), a large multi‐ campus teaching and research university in  the united kingdom, undertook a series of  workshops to ascertain customer values and  irritants. the purpose of this research was to  provide detailed information, from the  customers’ perspective, as to what an  excellent library service would look like.  through the customer value discovery  process, irritations caused by the current  library service were also ascertained, as  were perceptions of current performance  against the valued services and attributes of  the ideal library.    this paper focuses on a case study that uses  customer feedback to change the delivery of  forward‐facing library services in a multi‐ campus library service, with the aim of  improving customer satisfaction. it  complements another research paper  (mcknight “acquisition”) that focused on  changes to acquisition and cataloguing  processes as a result of customer value  discovery research.     measuring customer satisfaction is not new  in a library environment.  brophy cites a  number of methodologies that can be used  for this purpose: user surveys and  questionnaires; the libqual+™  benchmarking instrument; customer  feedback analysis; and focus group  interviews. customer value discovery is  another methodology and has been  primarily used in the for‐profit sector,  especially in retail, finance, and insurance.   some performance measuring techniques  operate by benchmarking against other  similar service providers. the customer  value discovery methodology does not lend  itself to such comparisons, as it is about the  service being researched today with current  customers and current service providers  engaged in the identification of values,  irritations, and perceptions of performance.    a new director of library services  commenced employment at ntu in august  2004. it was a time of significant  organisational change across the university.  an academic restructure, created four  academic colleges with nine schools that  replaced ten faculties. it was necessary for  llr to realign its forward‐facing  organisational structure, especially its  subject librarians/information specialists  who were organised along faculty and  campus‐specific lines, to support the new  academic arrangements. at the same time, a  new university‐wide strategic plan was  published that included, for the first time, a  key platform of ‘gold standard customer  service.’    objectives    a primary objective of this research was to  establish library customers’ perceptions of a  ‘gold standard.’ the ntu strategic plan  does not define ‘gold standard’ but rather  challenges all parts of the university to  strive towards delivering the best possible  customer experience.  a related objective was to improve customer  satisfaction. prior to undertaking the  evidence based library and information practice 2008, 3:1  35 research in 2005, there was no definitive  data on how students judged the library and  it resource room services. however,  students are only one customer segment of  an academic library. academic staff  members are also a significant customer  group. therefore, the research was aimed at  the two largest customer segments:  academic staff and on‐campus  undergraduate students.    the final objective was to actively engage  library staff in the discovery process. this  was to ensure that staff clearly understood  customers’ perceptions of the current  services so that the library could see why  changes may be necessary. active  engagement in deciding what actions should  be implemented as a result of the research  creates ‘buy‐in’ to the changes, some of  which could be quite significant. this was  particularly important where changes to  work practices, reporting relationships, and  job descriptions were necessary.    methods    the new library director had previous  experience in australia using customer  value modelling, which uses a combination  of focus group techniques and sophisticated  technology, to identify customer values. dr.  karl albrecht (“northbound train”) and dr.  kevin austin of enzyme international  (aust.) have developed this concept over  time (albrecht ”corporate radar”). because  of the positive outcomes of using the process  in australia, the same process was adopted  for this research at ntu.    initial planning     a consultative approach was used to engage  managerial staff in initial positioning and  planning decisions. a number of events  were conducted prior to the customer value  discovery workshops:    • in october 2004, there was an initial  planning session with llr heads to  reach agreement on an overall  restructure. up to this time, the  following reported to the director:  head, eservices; head, lending  services and administration;  collection development manager;  acquisitions manager; e‐services  manager; and three information  managers representing the campus  academic liaison teams. the new  structure established three new  business units: customer services  (responsible for all student facing  library services, including liaison  with academic staff), information  resources (responsible for library  technical services work) and the  business support unit (responsible  for administrative functions like  human resource management,  finance, and purchasing). later in  2005, the educational development  unit was created with responsibility  for developing and managing the  university’s new e‐learning  programmes.  • in december 2004, a planning  forum with key llr managers was  conducted to identify barriers to  success and agree top‐level  objectives for an llr operational  plan. each llr unit then wrote its  own action plan to meet the  objectives (and overcome barriers)  in the operational plan.  • in january 2005, an llr customer  facing teams workshop was held  to identify the changes required,  especially in a structural sense, to  align liaison librarian functions with  the new academic structure of the  university.          evidence based library and information practice 2008, 3:1  36 customer value discovery workshops    these events were followed in february and  march 2005 with facilitated customer value  discovery workshops. the process of  customer value discovery research  involves holding workshops with the  respective customer segments (in llr’s  case: academic staff and undergraduate on‐ campus students) observed by library staff;  a consolidation workshop to bring together  the outcomes of the customer value  discovery workshops; and a process of  interactive value modelling that enables  ‘what if?’ questions to be modelled using  software that plots the impact of the  proposed changes upon customer value and  satisfaction. in all, eight customer discovery  workshops were held (five student and  three staff workshops over three different  university campuses).  because of the small  sample size at one campus, a number of one‐ on‐one interviews were also held. in all,  about 90 students and 35 academic staff  were involved, with approximately 40  library staff acting as observers during the  workshops.    further information on the customer value  discovery workshop process can be found  in mcknight (”customer value” and  “acquisition”).    a diagram of the process follows:  customer value discovery preparation segmentation & study design customer discovery workshops per segment analysis workshops build value model consolidation interactive value modelling & action planning “customer” refers to a target group quick wins ongoing measurement     figure 1. customer value discovery process © enzyme international (aust.)      evidence based library and information practice 2008, 3:1  37 hierarchy of value unanticipated desired expected basic frustrated irritated angry     figure 2. hierarchy of value © enzyme international (aust.)    customer value discovery is based on a  very simple model, the hierarchy of value  (figure 2). on its continuum, it describes the  potential experiences that any interaction  between a customer and service provider  can provide. in a service industry, such as a  library, all interactions, whether with a staff  member or a resource (e.g. book) or service  (e.g. web page) can both satisfy and irritate a  customer at the same time.     the aim is to consistently deliver on the  basic and expected services, and to deliver  desired and unexpected services (but not at  the expense of neglecting the basic/expected  services) and doing these without causing  irritation to the customer.     the hierarchy of value (figure 2), as  designed by austin/enzyme international is  similar to kano’s model of customer  satisfaction (as described by 12manage  <http://www.12manage.com/methods_kano _customer_satisfaction_model.html>). kano  describes six factors or dimensions: basic  factors, excitement factors, performance  factors, indifferent attributes, questionable  attributes and reverse attributes. the  following table shows how the kano  dimensions compare to the hierarchy of  value model.  evidence based library and information practice 2008, 3:1  38   kano model attributes  hierarchy of value & irritant model  basic factors  basic and expected  excitement factors  desired and unexpected  performance factors  plotted on a performance graph for each  value/irritant factor  indifferent attributes  not identified by the customers in this model.  only important values and irritations are  described  questionable attributes  not valued, so not identified by the customers  reverse attributes  irritants: angry, irritated, frustrated    table 1. kano model and customer value model comparison    despite some similarities, there are marked  differences:  • in the kano model, pre‐defined  pairs of questions are prepared for  each service/product feature being  scrutinised. in the customer value  discovery model, the customers  describe their perceptions of  excellence/value, and rate these  against all other identified value  (and irritant) factors for importance  and performance.  • the kano model provides customer  satisfaction and dissatisfaction upon  a complex diagram supported by  complex tables. the customer value  discovery methodology produces  multiple easy to interpret graphs  depicting hierarchies of value and  irritation.  • a further significant difference is  the notion of service/product  providers actively participating in  the process of discovery. in the  customer value discovery process,  service/product providers vote as  they assume the customer will vote,  thus providing a gap analysis of  customer and provider perceptions.  while the kano instrument could be  provided to the service/product  providers to complete as they expect  the customer will complete the  survey and the two diagrams and  tables compared, it would not  provide the emotional involvement  of staff that the customer value  discovery process engenders.    the customer discovery workshops  provide a wealth of information: a  prioritised list of irritants, with the  frequency of irritation and the severity of  the irritation identified, as well as prioritised  values, with the importance and perception  of current performance against these value  factors. the process utilises a mix of high  and low technology: individual workbooks,  sticky notes to capture individual priorities,  affinity diagrams to develop thematic sets of  customer priorities, wireless voting  technology, and forced pairing software to  ascertain the hierarchy of values and  irritants.     an important and unique feature of the  process is that library staff members observe  the workshops in silence and vote as they  think the customers will vote during the  different stages (priority/importance and  performance) of the workshop. this  challenges staff assumptions and highlights  just how important it is to really ask and  listen to customers, as there are often  significant gaps in perceptions. this  personal involvement in the process and  evidence based library and information practice 2008, 3:1  39 active listening to the customers’ comments  helps staff to understand why changes are  necessary if the organisation is to deliver the  customer values and reduce the irritants.     student satisfaction surveys    in 2005 and 2007, ntu conducted student  satisfaction surveys. in 2005, 3400 students  completed an online survey; in 2007, more  than 5500 students undertook the survey.  questions relating to the services of llr  were included in both surveys and the  changes in satisfaction level are shown in  table 6.    a separate government survey administered  by the higher education funding council  for england (hefce) was conducted in  2005, 2006 and 2007. these surveys,  completed by graduating students, ascertain  the level of satisfaction in a full range of  university services, including library and it  resources. the survey questions on library‐ related services were comparable in 2006  and 2007 providing changes in satisfaction  level for the graduating cohorts.  the results  are included in table 7.      results    values and irritations    the customer value discovery research  provides hierarchies of values and irritations  in graphical format. the highest value or  irritation is given a value of 100% and the  others are expressed as a percentage of the  highest ranked item based on the voting  results using wireless keypads that force  tradeoffs of pairs of values and pairs of  irritants. sample graphical representations  are provided below.    overall hierarchy of value – comparative low relative importance high 0% 50% 100% note: value factors are ranked in order of importance for all segments combined easy access to materials where & when i need them inspiring environment which supports diverse needs comprehensive available relevant resources knowledgeable friendly accessible staff who help me good quality cheap photocopying & printing avail’ity of reliable up to date technologies & facilities timely targeted training proactive partnerships b/w academic staff & library opening hours which meet user needs user friendly loans policies & procedures managing multi media & curriculum content services clearly communicated to users academics students     figure 3.  hierarchy of value for academic staff and students      evidence based library and information practice 2008, 3:1  40 low relative severity high 0% 50% 100% hierarchy of irritation – comparative academics students note: irritants are ranked in order of severity for all segments combined inadequacy of collection & its management noisy & inappropriate study environment inadequate expensive photocopying / printing restrictive & difficult loans policies & practices can't find materials i need unhelpful uninterested staff unreliable & limited it / av inadequate academic liaison & communication inadequate opening hours problems with catalogue queuing for service lack of training & support when i need it don't like card access policies     figure 4.  hierarchy of irritation for academic staff and students    there is a direct correlation between many  of the irritants and values.  therefore, by  focusing on reducing irritation, there is a  corresponding improvement in value for the  customer.  action taken following the customer  value discovery process    the values and irritants obtained via the  cvd process and the corresponding action  taken are presented in the tables below. the  values and irritants were actively used to  inform operational planning, staff  development activity and budget setting  within llr. prioritising activity and  expenditure in this way made it possible to  secure sufficient funding or redirect funding  from non‐value adding services to deliver  an ambitious portfolio of improvements at a  time of constrained budgets within the  university, safe in the knowledge that the  time, effort, and funding was being  effectively targeted. many of the  improvements were straightforward and  directly addressed customer concerns –  unreliable photocopiers replaced were with  new equipment for example.  however the  customer value discovery process also  provided additional insight into customer  perceptions of llr and its services as well  as encouraging and challenging library staff  to think more creatively about service  delivery.  they were also able to use their  expertise to ‘make the connections’ between  customer responses and the subsequent  development of new services: the innovation  in information skills provision is a good  example of this. the customer value  discovery process was a public statement to  customers that llr was committed to  improving services, therefore, it was  important that feedback was provided about  how it had responded. hence, besides the  usual promotion of individual service  developments, a poster campaign, using a  ‘you asked for …’ and ‘we responded by …’  approach was used to demonstrate llr had  listened and acted in a timely manner.  evidence based library and information practice 2008, 3:1  41   library collections  consolidated values and  irritants  action  related values  • easy access to materials where and when  i need them  • comprehensive, available, relevant  resources    related irritants  • inadequacy of the collection and its  management  • cannot find materials i need  • problems with the catalogue    • developed a library information  resources policy to guide future  collection development and collection  management (approved by academic  board after wide consultation)  • digitised short loan items, where  possible, to maximize access to these in‐ demand resources  • purchased additional copies of core  reading list resources  • established projects to improve reading  list management  • secured additional funding for books  and journals  • licensed additional ejournals so as to  enable 24 x 7 access  • licensed more ebooks (e.g. from 200  titles to 34,000)  • reclassified the entire library collection  to one scheme (from six)  • installed a new library management  system to provide a much more user‐ friendly user interface to the catalogue  • enhanced the federated search service to  enable easy access to digital databases  and ejournals  • redesigned the llr web site to make it  more accessible (disabled users) and with  richer in content  • implementing an institutional repository  to archive research and scholarly  publications and to make these publicly  discoverable  • postal loans services available for  students not regularly on campus  • implementation of secure electronic  delivery and online requests for inter‐ library loans  table 2: correlation of values, irritations and actions – library collections    evidence based library and information practice 2008, 3:1  42 library collections    it was no surprise that ‘easy access to  comprehensive, relevant resources which were  available when needed’ would be highly  valued. however, the significance placed on  this relative to all other areas of service  delivery demonstrated to the university that  improvements to the collection should be  the highest priority. this was addressed in  two ways. first, an increase in the budget  for the collection (particularly electronic  resources) was agreed to address historical  under‐funding relative to other similar  universities and to provide the means by  which several new collection development  initiatives could be funded. secondly, a  library information resources policy to  govern and direct collection development  and provide guidance to the liaison  librarians over priority areas for expenditure  was developed in consultation with the  academic community. this policy  framework was used to guide projects  relating to targeted collection developments,  digitisation of local content, introduction of  new material formats such as ebooks, and  management of reading list materials.     for a more complete account of changes in  the information resources area, see  mcknight (“acquisitions”). the  combination of increased funding and  improved management of spending plans  was expected to realise considerable benefits  in this key area of concern to llr’s  customers.  evidence based library and information practice 2008, 3:1  43   the library environment  consolidated values and  irritants  action  related values  • inspiring environment that supports  diverse needs  related irritants  • noisy inappropriate study environment                        related value  • good quality cheap photocopying and  printing  related irritant  • inadequate and expensive photocopying  and printing            related value  • user friendly loans policies and  procedures  related irritant  • restrictive and difficult loans policies  and practices        related value  • opening hours which meet user needs  related irritant  • inadequate opening hours    • identified silent, quiet and ‘time out’  zones in each library. silent areas are  patrolled to maintain a conducive study  environment   • improved signage in all libraries  • submissions to estates for major capital  works approved resulting in  refurbishment of library and it spaces  (new furniture, lighting etc)  • student and staff artwork used to  decorate library public spaces  • ‘funky’ furniture installed in student  ‘time out’ rooms; vending machines also  installed        • new copiers introduced to improve  reliability   • reduced costs of photocopying  • colour copying facilities provided at  each campus  • introduced self‐payment kiosks for  printing and copying          • revised loans policy and increased limit  of items allowed for borrowing  • revised inter‐library loans policy with  consistent implementation  • consistent implementation of loans  policies on all campuses      • extended summer opening hours  • review of evening and weekend opening  hours with the intention of extending  hours at key times during the academic  year  evidence based library and information practice 2008, 3:1  44     related value  • availability of reliable up‐to‐date  technology and facilities  related irritant  • unreliable and limited it / av  equipment          • upgraded it resource room equipment  • established additional resource rooms  (more pcs)  • introduced more consistent support for  students in other university it resource  rooms  • wireless access implemented in all  libraries  table 3. correlation of values, irritations and actions – the library environment    the library environment    the customer value discovery research  highlighted several factors concerning the  day‐to‐day use of the libraries and resource  rooms (noise, reliability of equipment,  opening hours, attractiveness of the  buildings, etc) that were a source of ongoing  irritation to customers. in response to this a  number of initiatives, including building  refurbishment and equipment replacement  programmes, were initiated resulting in a  much improved customer experience. while  staff already recognized many of the  required improvements, the customer  value discovery process helped with the  prioritisation and scheduling of  improvements and securing funding to  carry out the changes.     a particularly important irritant was the  view that noise levels were excessive in the  libraries and impacted negatively on their  value as study locations. this was a complex  issue to address: the libraries needed to  support a variety of different study activities  including group work as well as individual  silent study, and all within a constrained  space envelope. after considering a range of  options, the view was taken that the mix of  group, quiet and silent study areas was  essentially right, but that they were  inadequately promoted and managed.  consequently, a small library staff working  group was formed, led by a senior library  assistant, to develop new approaches to the  management of the various study zones.   this resulted in the novel and successful  introduction of colour zoning across the  libraries and it resource rooms (red for  silent, amber for quiet, and green for social  areas), a development that was well  received by the student users. this working  group was a good example of teamwork and  empowerment, as the staff involved in  maintaining an appropriate study  environment was directly engaged in  defining the solutions.    evidence based library and information practice 2008, 3:1  45 front‐line staffing  consolidated values and  irritants  action  related value  • knowledgeable, friendly, accessible staff  who help me    related irritant  • unhelpful, uninterested staff    • multiple front‐line service teams  restructured into a new single customer  facing team to improve responsiveness   • all staff reminded of importance of a  welcoming environment; customer  service training provided and included  in annual staff training schedule   • increased budget for staff development  with major focus on development of  customer service skills  • introduced performance development  and review for all staff to help capture  skill development needs and monitor  performance  • revised the student charter to reflect the  values identified by students, and  established key performance indicators.  created a new service charter for  directed at academic staff    table 4. correlation of values, irritations and actions – front‐line staffing    front‐line staffing    before restructuring, provision of first‐line  student services in the libraries had been  provided by multiple teams, organized  according to functional areas with separate  inquiry points reflecting a support model  unchanged from the late 1990s. however,  over recent years the demands on services  from students and other customer groups  has increased significantly with blurring and  overlap among the teams. the customer  value discovery process identified that the  current team structures led to “confusing  help points” with library customers facing a  number of desks depending on their needs.  this confirmed assumptions that the  customers of llr view it as a single entity  rather than discrete teams and thus expect a  seamless service provision.     as a result of this, an llr working group  was established which recommended a new  organisational model based on the merger of  the teams with the new team working from  a new single information desk. from an  organisational point of view, this provided  increased flexibility in the staffing resource  to improve responsiveness to service  demand. from a customer point of view, a  simplified arrangement for obtaining  support was the result, as an inquirer no  longer had to know which desk to approach  for support. for the university, this was a  major organisational change, affecting  almost 100 members of staff, and required  extensive negotiations with the staff  evidence based library and information practice 2008, 3:1  46 concerned and trade unions as it required  major retraining/reskilling for staff. a few  months into the new organisational  arrangements, staff were acknowledging  that while the change had been challenging  for them, it had helped improve inter‐team  working and improved their understanding  of the importance of delivering joined‐up  services to the customer.     academic liaison  consolidated values and  irritants  action  related value  • pro‐active partnership between academic  staff and library    related irritant  • inadequate academic liaison and  communication          related value  • timely targeted training    related irritant  • lack of training and support when             i need it  • restructured information team to  provide support for university strategic  priorities  • increased emphasis of sharing good  practice across all schools (e.g.  information literacy training, research  support)  • increased emphasis on support for e‐ learning; trained librarians to support  roll‐out of new virtual learning  environment    • included library information in ‘getting  started,’ a six‐week online induction  programme for new students  • introduction of ‘welcome desks’ at the  beginning of the academic year to help  new students settle in  • significantly improved induction and  information skills provision, with an  increased emphasis on information skills  training for all students and staff   • introduced online tutorials for just‐in‐ time learning  • embedding information skills into first  year programmes included in ntu  institutional learning & teaching  enhancement strategy, and being  progressively introduced  • introduced drop‐in sessions for  information skills development  • introduced self‐directed library  induction tours using mp3 players    table 5.  correlation of values, irritations and actions – academic liaison  evidence based library and information practice 2008, 3:1  47 academic liaison    nottingham trent university had always  placed great importance on the close     involvement of its professional librarians  with the teaching, learning and research  activities in the university.  the customer  value discovery process emphasised the  importance of this with a clear demand for  more joint working with academics and  skills development work with students.  prompted by a major restructuring of the  university’s academic departments, which  would in itself have forced a matching  realignment of the responsibilities of the  professional librarians, llr used this as an  opportunity to fundamentally restructure  the academic liaison team into a new single  unit. this provided the flexibility to respond  to unmet demand, and provide the  librarians in the new structure with a  coherent strategy for liaison work. the  ‘reinvented’ team provided the impetus and  resources to introduce a number of new  developments: collaboration in elearning  projects, the use of web 2.0 type  technologies to improve relevance and take  up of student induction and skills sessions,  development of additional online self‐help  information tutorials, introduction of a  virtual reference service, support and  involvement in the university’s ‘welcome’  programme for new students, and  introduction of drop‐in sessions to  complement the existing programmed  information skills sessions.     the hierarchy of values has also informed  the revision of llr’s service charter,  creating a new charter for services to  academic staff. (see  <http://www.ntu.ac.uk/llr/about_us/servic e_charter_values/index.html>.)    ntu student satisfaction survey    the university’s bi‐annual student  satisfaction survey demonstrates that the  actions taken in response to the customer  value discovery research have had a  positive impact on student perceptions of  service.              evidence based library and information practice 2008, 3:1  48    survey area  2005 % actively  satisfied  2007 % actively satisfied  change library website  73%  81%  +8%  opening hours  70%  76%  +6%  helpful and accessible staff   67%  74%  +7%  access to printers/ photocopiers  55%  73%  +18%  availability of study places  64%  73%  +9%  availability of e‐journals and databases  66%  70%  +4%  range of e‐journals and database  61%  69%  +8%  availability of group study rooms  n/a  68%  n/a  range of books  61%  65%  +4%  range of printed journals  61%  64%  +3%  availability of pcs in libraries  48%  63%  +15%  noise levels  52%  62%  +10%  training and support for students in using  library facilities  57%  61%  +4%  range of audio and visual materials  61%  59%  ‐2%  availability of books and materials from  reading list  47%  53%  +6%  loan periods  52%  57%  +5%  loan of laptops for in‐library use  n/a  51%  n/a  table 6. nottingham trent university library student satisfaction survey comparison, 2005‐ 2007    this ntu survey, which only canvasses  students, does not replace the customer  value discovery process. while there is a  similarity between the needs of students and  those of academic staff, the customer value  discovery research provided evidence about  what academic staff valued, as well as the  priority of services valued by on‐campus  undergraduate students.    national student satisfaction survey    although there is no comparable data  available for 2005, the improvement in the  three library‐related questions in the uk  hefce national survey of students from  2006 to 2007 shows a marked improvement  in rating over the 12 month period and also  satisfaction ratings that are above the sector  average on all cases.                  evidence based library and information practice 2008, 3:1  49   q. 16 ‐ the library’s resources and services are good enough for my needs  – 82nd in 2006  – 59th in 2007 with a score of 81% (average sector score 77% satisfaction)  q.17 – i have been able to access general it resources when i need to  – 44th in 2006  – joint 19th in 2007 with a score of 91% (average sector score 84% satisfaction)  q.18 – i have been able to access specialised equipment, facilities or rooms …  – joint 66th in 2006  – joint 30th in 2007 with a score of 78% (average sector score 73% satisfaction)    table 7. national survey of student satisfaction results for library‐managed services,  nottingham trent university      discussion    gold standard customer service    the first objective of the customer value  discovery research was to be able to  articulate what ‘gold standard’ meant for  the library. the hierarchy of value  provides the evidence regarding which  services are most important, enabling focus  on those areas that will make the greatest  impact on student and academic staff  satisfaction.     within llr, the notion of ‘gold standard’ is  to consistently deliver the basic and  expected services; to strive to provide the  desired and unexpected services that would  delight the customer; and, at all times,  reduce the potential for irritating the  customer in the course of service delivery.  this implies continuous improvement as  unexpected services become the norm.    customer satisfaction    the second objective of the customer value  discovery exercise was to improve customer  satisfaction. the evidence provided by the  two student satisfaction surveys attest to the  improvement in student satisfaction over a  two year period. it is argued that the  improvement in all aspects (except ‘range of  audio and video materials’) is due to the  initiatives that were put in place as a result  of the initial research in early 2005. the one  criterion that did not improve was a service  that is being deliberately phased out,  involving the move from analogue resources  (videos, slides, compact discs) to digital  resources.    engagement of llr staff    the customer value discovery process  requires engagement of the organisation’s  staff. the observer status of staff is  fundamental to the workshops with  customers. the observers are required to  listen and to vote as they think the  customers will vote. this enables a greater  understanding of customers’ problems,  values and expectations. the personal  experience engenders greater commitment  from staff to act on the feedback that they  receive. this is supported by social science  research on persuasion that identified that  “public commitments, even seemingly  minor ones, direct future action” (cialdini  64). the very fact that the organisation is  undertaking the research and has staff  listening at the workshops demonstrates  that commitment.     evidence based library and information practice 2008, 3:1  50 through the forced‐pair voting process  during the customer value discovery  workshops, assumptions of staff are  challenged.  the staff observers vote as they  expect the customer to vote. the gap  analysis on performance and priorities can  reveal of a lack of understanding of  customer perceptions and needs.     management commitment to act upon the  evidence presented by the customer value  discovery process is vital to its success. to  ignore the results would be to raise  inappropriately expectations of both staff  and customers.  therefore, the engagement  of staff at all levels in the organisation (not  just the observers) with the research  outcomes and the identification of actions to  improve value and reduce irritations is  important.    in llr, this engagement was achieved in a  number of ways:  • the values identified by customers  have been reflected in the llr  operational plan.  the majority of  objectives in the customer  perspective come directly from the  customer value discovery research.  the others come from emerging  requirements and the university’s  strategic plan directions.  o c1 objective: facilitating  easy access to a  comprehensive, relevant  range of information  resources  o c2 objective: facilitating  the development of  informed and independent  learners  o c3 objective: creating an  inspiring environment that  meets the diverse needs of  learners, teachers and  researchers  o c4 objective: enhancing  support for e‐learning  o c5 objective: enhancing  support for researchers  o c6 objective: strengthening  our support for the  widening participation  agenda  o c7 objective: developing  and sustaining a proactive  partnership between llr  and academic staff  • managers within llr engage their  staff in the definition of actions  under these objectives to identify  what is to be undertaken in the  planning period.  • individual performance planning  and review (pdr) objectives are  aligned with the operational plan so  as to link individual performance  with the overarching customer  values.  • teamwork and task groups are used  extensively to deliver on the plan.  from a customer’s perspective, the  organisational unit is ‘just the  library’ or ‘the university’; they do  not see the different units that have  to work together to deliver on the  customer values. this has resulted  in a reduction in the sense of  organisational silos that existed  before the customer value  discovery research.  • monthly managers’ forum  meetings monitor and review  actions within the operational plan,  ensuring that there is broad  understanding of all the initiatives  and the related project  dependencies.      conclusion and future research    austin said:  “if you’re going to get everyone in  the organisation to concentrate on  delivering customer value, you’ve  evidence based library and information practice 2008, 3:1  51 got to give them a concrete model of  what that value is, as defined by the  customers themselves. it’s not  enough to ask employees to smile  and be nice. you need to give them  a workable definition of the value  they’re expected to deliver, and then  help them learn and use the critical  work practices that deliver the  value.” (albrecht “corporate  radar” 91)    this is the concept that is driving change in  llr. the annual planning objectives for the  customer perspective are couched in the  terms of the customers’ values; the pdrs are  aligned so every individual can see how  their actions link in to the achievement of  the operation plan, which is aimed at  delivering value to customers.    in an interview with the australian institute  of management, albrecht said that  organisational intelligence is ʺthe capacity of  an enterprise to mobilise all of its  brainpower, and to focus that brainpower  on accomplishing its mission.” llr is  endeavouring to be an intelligent  organisation. by understanding precisely  what customers value; by striving for “gold  standard” customer service; and by  engaging staff at all levels to implement the  necessary initiatives to deliver added value  and to reduce irritation, llr is using its  collective intelligence.    there is much still to do in imbedding the  new ways of working, with customer values  at the centre of all decision‐making.  however, the framework has been  established and there is significant buy‐in by  staff. the constant need to innovate is  challenging, as staff grapple with finite  budgets and increasing expectations of fee‐ paying customers. what was new yesterday  will be an expected service tomorrow. so the  process of consolidating new practices and  services must be carried out in tandem with  designing and delivering new services to  meet emerging customer expectations.     “value targeting” refers to aiming for  whatever matters most to individual  consumers. (albrecht ”supertrends” 26).  library services in the past have been aimed  principally at large cohorts of customers.  personalisation of services is an emerging  issue to be addressed, especially when the  millennials (those born between 1982 and  2000) attend university. their expectations  of personalised services, their extensive use  of portable it devices and services, and their  preferences to work and study in teams will  require new interventions by higher  education library services. library  management and staff cannot afford to be  complacent, regardless of how satisfied  today’s customers are.    further research is planned. a comparison  of values and irritants between australian  and uk academic libraries will be  undertaken to ascertain whether there are  ‘core library values’ that transcend national  barriers. the relationship of irritants with  values will be examined and english and  australian irritations compared.  should  core values be identified, these will be  compared to the benchmarking areas and  questions in the libqual+™ instrument.    the customer value discovery  methodology is robust and well tested.  however, limited library budgets may not  extend to contracting the facilitators to  undertake the research, although the cost, as  a percentage of total budget, is very small  and the benefits of targeting funding to  services that add value is considered  justification. further work will be  undertaken to see if any refinements of the  process could be made to reduce the costs  but still deliver useful results and thus make  the model more accessible to library  management teams.    evidence based library and information practice 2008, 3:1  52 works cited  12manage management communities.  customer satisfaction model  (kano), 24 february 2008. 2 mar.  2008  <http://www.12manage.com/metho ds_kano_customer_satisfaction_mo del.html>    albrecht, k. corporate radar: tracking the  forces that are shaping your  business. new york: american  management association, 2000.  ‐‐‐. “eight supertrends shaping the future  of business.” the futurist. (sept‐oct  2006): 25‐9.  ‐‐‐. the northbound train: finding the  purpose; setting the direction;  shaping the destiny of your  organization.  new york: american  management association, 1994.  ‐‐‐. interview with australian institute of  management. karl albrecht:  organisational intelligence. 2003. 2  mar 2008  <http://www.aim.com.au/resources/ article_kalbrecht.html>.   brophy, p. measuring library performance:  principles and techniques. london:  facet publishing, 2006.    cialdini, robert b. “the science of  persuasion.” scientific american  (february 2001): 62‐7.    kano, n., seraku, n., takahashi, f. and  tsuji, s. “attractive quality and  must‐be quality.” hinshitsu  (quality, the journal of japanese  society for quality control) 14.2  (1984): 39‐48.     mcknight, s. “customer value research.”  management, marketing and  promotion of library services based  on statistics, analyses and  evaluation. eds. trine kolderup  flaten and k.g. saur. munchen:  ifla publications, 2006. 206‐216,  2006.    ‐‐‐. “acquisition and cataloguing processes:  changes as a result of customer  value discovery.” evidence based  library and information practice  (eblip) 2.4 (2007):22‐35. 2 mar 2008  <http://ejournals.library.ualberta.ca/i ndex.php/eblip/article/view/521/66 7>.     nottingham trent university. the ntu  strategic plan 2004‐2010.  nottingham trent university, uk. 2  mar 2008  <http://www.ntu.ac.uk/about_ntu/st rategic_plan/12910.pdf>.    evidence based library and information practice 2012, 7.1 110 evidence based library and information practice evidence summary email reference transactions reveal unique patterns about end-user information seeking behaviour and librarians’ responses in academic and public libraries outside the u.s. and canada a review of: olszewski, l., & rumbaugh, p. (2010). an international comparison of virtual reference services. reference & user services quarterly, 49(4), 360-368. reviewed by: giovanna badia liaison librarian schulich library of science and engineering, mcgill university montreal, quebec, canada email: giovanna.badia@mcgill.ca received: 10 dec. 2011 accepted: 31 jan. 2012 © 2012 badia. this is an open access article distributed under the terms of the creative commonsattributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to investigate and compare the nature of e-mail reference services in academic and public libraries outside the united states. design – longitudinal comparative study. setting – a total of 23 academic and public libraries in ten countries: australia, belgium, france, germany, mexico, the netherlands, slovenia, south africa, sweden, and the united kingdom. subjects – the authors collected reference questions that were e-mailed to the 23 libraries for the weeks of april 3, 2006 and april 7, 2008. questions were sent from the libraries’ websites to questionpoint, a collaborative, online reference service that was used to answer the questions received. methods – the authors randomly selected 25 questions for each library for the weeks under investigation. if a library did not receive 25 email reference questions that week, then they collected transactions from subsequent weeks until the quota was met or until the end of the month. the authors examined transactions from a total of 919 questions – 515 questions in 2006 and 476 in 2008. all identifying information about the user was stripped from each transaction collected. each transaction was labeled according to the following categories: mailto:giovanna.badia@mcgill.ca� http://creativecommons.org/licenses/by-nc-sa/2.5/ca/� evidence based library and information practice 2012, 7.1 111 • type of institution, i.e., whether the question was sent to an academic or public library • language of the question • question type, i.e., whether the question was about library policy or access to electronic resources (labeled “access” questions), about library holdings (labeled “bibliographic” questions), or about finding specific information on a topic (labeled “subject” questions) • answer type, i.e., whether the response consisted of: a confirmation, clarification, fact, instructions, referral to a pathfinder/bibliography, referral to another library/person/place, or no answer. • user status, i.e., whether the person asking the question was an undergrad, a graduate student, or a staff/faculty member • subject classification of the questions using the dewey decimal classification system • response time main results – the e-mail transactions that were examined revealed a wide range of enduser and librarian behaviors. english, followed by dutch, german, and french, were the languages most frequently used by library users. countries also varied in terms of the types of questions received. for example, more than 75% of the email queries in belgium (which only had academic libraries participate in this study) were “access” questions, while mexico (which also consisted of all academic libraries) only received 6% “access” questions, france (all public libraries) had relatively few access questions, and sweden (also all public libraries) had none. public libraries received the most “subject” questions (75%) compared to academic libraries (28%). public libraries answered “subject” questions with facts over a third of the time, while academic libraries responded with instructions close to half of the time. among the academic libraries, graduate students asked slightly more “access” questions than undergraduates (62% versus 56%), and undergraduates asked more “subject” questions than graduate students (26% versus 13%). the “subject” questions submitted to academic libraries were divided almost equally among topics in the humanities (36%), the sciences (32%), and the social sciences (32%). this differed from public libraries; the latter received mostly questions about humanities topics (65%). the time taken to respond to users’ reference questions ranged from a few minutes to a few weeks between libraries. some libraries set the response times on their websites. those libraries that indicated longer response times on their sites met the users’ expectations more often, up to a maximum of 100 percent of the time. most of the characteristics of email reference services that are listed above remained consistent from 2006 to 2008. the two areas that changed over two years were the libraries’ response time and the types of questions asked by university students. “access questions increased (by 14 percent among graduates and by 4 percent among undergraduates), and bibliographic and subject questions decreased in both groups” (p. 364). response time improved overall from 2006 to 2008. conclusion – the authors’ analysis of the 919 transactions of e-mail reference questions revealed unique patterns about end-user information seeking behavior and librarians’ responses in academic and public libraries outside the united states and canada. one of these patterns is that the public libraries participating in the study received the highest percentage of “subject” questions. the authors state that “the pattern of a much higher percentage of subject-related questions in public libraries contrasts with the general virtual reference trend in academic libraries, which shows a much higher percentage of access questions. since many of the access questions concerned connection problems or logging on to databases, the relatively fewer number may indicate that the arts and humanities disciplines require less database searching and that the users need specific answers instead” (p. 367). evidence based library and information practice 2012, 7.1 112 the data also revealed significant differences between the types of questions asked by undergraduates versus graduate students. undergraduates asked two thirds of the subject questions submitted to academic libraries and graduate students asked just over a fourth. the authors assume that this finding indicates that graduate students do more of their own research than undergraduates. the authors were concerned by the increase in the number of access questions posed by undergrads and graduate students from 2006 to 2008. they suggested that websites, databases, and other resources might have become more difficult to use over the years. they also noted that questions in technology almost doubled from 2006 to 2008. one of the patterns that were revealed contradicted the authors’ assumption that libraries with slow response times in 2006 would improve in 2008 as they became more proficient in providing virtual reference services. the majority of libraries in the study improved their turnaround time from 2006 to 2008, but the two slowest libraries took even longer to respond to their users. commentary this study reveals interesting patterns about the behavior of users of email reference services in academic and public libraries in australia, belgium, france, germany, mexico, the netherlands, slovenia, south africa, sweden, and the u.k. the authors summarize the existing literature on virtual reference services in non-us countries at the beginning of the article; however, they do not link their findings back to their literature review. this missing link makes it hard to determine how this study adds to, or fits in, our present body of knowledge on the subject. 23 libraries participated from ten countries, which comes out to approximately 2 libraries participating per country. the small number of participating institutions from each country would make it difficult to generalize the findings from the participating libraries to all academic and public libraries in that specific country. an explanation about how the libraries were selected might have nullified this comment. the number of questions examined was also small; 25 questions were randomly selected per library per week under investigation. this study would need to be expanded to include more libraries and more questions to validate the authors’ conclusions. finally, this reviewer would have liked to see canada and the united states included in the study. if the latter was beyond the scope of their study, then the authors could have made a comparison to the published literature on virtual references services in canada and the u.s. do canadian and american libraries show similar patterns? if not, how are they different? the authors mention that “questions as [to] whether the success of virtual reference services relies on cultural attributes are important to answer when developing and implementing reference services in countries of widely divergent cultures” (p. 367). however, they do not take the opportunity to discuss the cultural context of their findings. this study presents fascinating patterns of virtual reference services in 10 countries, although the authors fail to place their findings in a wider context. a discussion of the wider context would have added value to this article by making it clear to the reader how these findings can influence the practice of libraries participating in collaborative virtual reference services via questionpoint. / evidence based library and information practice evidence based library and information practice 2012, 7.4 117 evidence based library and information practice evidence summary courses studied by new york librarianship students affect their perceived readiness for employment a review of: creel, s. l., & pollicino, e. b. (2012). practitioners’ & lis students’ perceptions on preparedness in the new york metropolitan area. education for information, 29, 53-69. doi: 10.3233/efi-2010-0911 reviewed by: kirsty thomson subject librarian heriot-watt university edinburgh, united kingdom email: k.s.thomson@hw.ac.uk received: 1 sept. 2012 accepted: 13 nov. 2012 2012 thomson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to investigate whether librarianship students felt ready to enter the workforce, and whether practitioners felt recent graduates were suitably prepared. design – survey. setting – a university in new york city, and school and public librarians working in the new york metropolitan area. subjects – 55 mls students, 167 school library practitioners, and 181 public library practitioners. methods – students surveyed practitioners about new graduates’ readiness to work as librarians. the students also assessed their own readiness. main results – detailed analysis of differences between the responses of the four subject groups – public librarians, school librarians, public library students, and school library students – for each of six survey statements is provided. practitioners and students felt that school librarianship graduates were more prepared for work than public librarianship graduates. this may have been due to differences in the practical components of their courses. conclusion – preparedness for library employment is related to the courses studied by librarianship students. mailto:k.s.thomson@hw.ac.uk http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/ evidence based library and information practice 2012, 7.4 118 commentary many degrees approved by the american library association (ala) do not require students to take courses covering all of the ala final core competencies statement (american library association, 2009) there is a disconnect between employers’ and educators’ expectations of what college courses should teach, particularly in relation to the practical components of the courses. this paper outlines previously published work in this field, and reports on a survey investigating new graduates’ preparedness for work in a library. the opinions of librarians were gathered and students assessed their own readiness. the authors acknowledge problems in their survey design, such as the small sample size and limited geographical area. they also draw attention to a significant flaw: they asked librarians about the preparedness of graduates, but did not check if these librarians had recently worked with a new graduate. the cristal checklist for appraising a user study (n.d.) was used as a guide to further assess the research methods described in this paper. the paper lacks information about how participants were recruited, and the response rate is not given. a large number of students were involved in conducting the surveys (n=55); it would have been useful to include more detail about how the authors checked for consistency across surveys conducted by different students. the authors compared public and school librarianship students but there was not a control group who had not attended mls training, so the impact of the courses cannot be fully evaluated. future studies could compare the readiness of students at the start and end of their degrees to see if attending library school changes their capabilities. students were asked if they agreed with statements about their own preparedness, but the discussion sometimes interprets this as an opinion on the readiness of all students (e.g., “think recently graduated mls students” (p. 59)). some students might assess their own readiness differently to their classmates’ readiness; this purpose of this question needs to be clarified in future iterations of the student survey. in the analysis of the survey, the data is often presented as bar charts and percentages. the inclusion of numerical data tables would have been useful (if permitted by the journal), as this would enable readers to carry out their own analysis. demographic information about the student participants is provided, but demographics of the employed librarians are not given. authors writing about countryspecific education practices should include definitions of terminology (e.g., “service learning”) for international readers. this study should influence the design of librarianship courses as it identifies areas where students and employers feel graduates are not well prepared for the workplace. further research is needed to investigate why employers feel students are underprepared, and to explore whether revising the content of librarianship courses would have an impact on preparedness. references cristal checklist for appraising a user study. (n.d.). retrieved november 28, 2012, from http://nettingtheevidence.pbwiki.com/ f/use.doc american library association. (2009). ala’s core competences of librarianship. retrieved november 28, 2012, from http://www.ala.org/educationcareers/fi les/careers/corecomp/corecompetences /finalcorecompstat09.pdf evidence based library and information practice using evidence in practice   rethinking staff development needs during covid-19   michael priehs organizational development coordinator wayne state university libraries detroit, michigan, united states of america email: mpriehs@wayne.edu   received: 24 feb. 2021                                                               accepted: 15 apr. 2021      2021 priehs. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29934   setting   wayne state university (wsu) is an urban, public research university located in the midwestern united states. wsu has 13 schools and colleges offering approximately 350 academic programs and serving over 26,000 students. the wayne state university library system (wsuls) includes the david adamany undergraduate library, the arthur neef law library, the purdy/kresge library, the vera p. shiffman medical library and its learning resource center at the eugene applebaum college of pharmacy and health sciences, and the walter p. reuther library of labor and urban affairs, and university archives. additionally, the school of information sciences and the detroit area library network are under the library system’s charge. the library system employs around 120 full-time and 60 part-time staff across all units.   problem   beginning in 2017 under the direction of a new dean of libraries, the wsuls adopted a new strategic focus and reframed our work around four visionary pillars: student success, community engagement, scholarship, and organization and culture. to support our focus on organization and culture, the library system began a concerted effort to focus on organizational development (od) initiatives and positive culture change within the libraries. od strategies can be classified under the following areas: organizational effectiveness, organizational structure, ongoing performance and productivity initiatives, and organizational learning (society for human resource management, n.d.). in full support of this initiative, the dean of libraries enabled the transition of an existing librarian to a new od coordinator role and the hiring of an od consultant. from 2017 to early 2020, this new od team led several system-wide initiatives. both the od coordinator and consultant were gallup-certified strengths coaches (gallup, 2021). accordingly, the cliftonstrengths assessment (gallup, 1999) and one-on-one coaching were offered to every staff member. in 2019, the od team facilitated a year-long in-house leadership series that covered a variety of topics, including effective communication, building cohesive teams, managing conflict productively, emotional intelligence, building networks, presentation skills, and leading effective meetings. separate sessions were held for existing leaders (those with direct reports) and influential leaders (those without direct reports) with the intent of providing a safe space for staff to share.   in order to meet the needs of our users in an ever-changing information environment, the od team, including the dean of libraries, focused on senge’s model of learning organizations, which are described as “organizations that discover how to tap people’s commitment and capacity to learn at all levels in an organization” (1990, p. 4). these efforts are intended to support everyone from student assistants to senior administration and require a wide range of topics and strategies based on the needs of the individual staff members. many libraries have utilized the learning organization model including, but not limited to, the university of arizona, illinois’ north suburban library system, the university of california, stanford university, and the university of nebraska-lincoln (giesecke & mcneil, 2004). senge’s (1990) model is based on five characteristics:   systems thinking: seeing the connections and interrelationships within an organization.   personal mastery: “the discipline of continually clarifying and deepening our personal vision, of focusing our energies, of developing patience, and of seeing reality objectively” (senge, 1990, p. 7). this represents a personal commitment to lifelong learning and achieving a high level of proficiency.   mental models: our assumptions and generalizations that influence how we understand or interpret our organizations.   shared vision: members of the organization creating a picture of how the organization will look in the future together.   team learning: setting aside assumptions and creating an open environment for dialogue and discussion. the ways in which a team interacts can be a barrier to learning. recognizing these patterns can help a team grow and adapt, which is fundamental to organizational success. this process increases the capacity for a team to create its desired results.   from the beginning of this new initiative at wsuls, the entire design centered on the in-person working environment, as the library system did not have a culture of remote work. in early 2020, the covid-19 pandemic began to impact higher education and in march, the library system was forced to transition to a primarily remote work environment. as with most of the world, we suddenly found ourselves needing to transition our work to this new reality.   the od team quickly realized we would need to rethink our ongoing work, as priorities in the library system were upended and we needed to address an entirely new set of needs. od work can have a direct impact on an organization’s agility. according to aghina and de smet (2015), “agility is the ability of an organization to renew itself, adapt, change quickly, and succeed in a rapidly changing, ambiguous, turbulent environment” (para. 2). having an existing od team allowed us to address these new needs immediately, helping the library system to quickly adapt and meet the unexpected challenges we faced and helping us to thrive during a global pandemic.   evidence   the od team regularly seeks staff input on completed sessions, desires of staff, and overall perceptions of the system’s culture. a desire for staff development initiatives had long been expressed and supported by several internal staff surveys. upon completion of the 2019 leadership series, all staff were invited to participate in a survey intended to better understand the perceptions of the series, as well as inform our planning for future initiatives. the survey was completed by 58 participants; 69.6% indicated that they had attended at least one of the sessions. additionally, we found that 34.4% attended 1-2 sessions, 12.5% attended 3-4 sessions, 25% attended 5-6 sessions, 18.8% attended 7-8 sessions, and 9.4% attended 9 or more sessions. of the participants who shared their reasons for not attending, most indicated that they had scheduling conflicts that prevented them from attending. the most beneficial sessions, according to survey participants, were those on communication, managing conflict, and the introductory session on defining leadership. the least beneficial sessions were on diversity and inclusion, and presentation skills. the vast majority of respondents indicated a strong desire for the team to continue hosting a similar series. the most common suggestions for new topics centered on self-development and professional development opportunities.   with the sudden transition to remote work during the pandemic, staff at all levels were expressing concern and uncertainty about this new way of operating. not only were staff concerned with finding ways to do their work remotely, but they were also concerned with childcare and schooling issues, the health and wellbeing of family members, and the overall stress associated with living through a pandemic. managers and the od team were often contacted with requests for support and resources, ranging from scheduling and shifting work priorities to technological barriers and stress management assistance.   it was clear from the vocal staff, as well as from library administration, that we needed to transition our od work to focus on these new pressing issues. we needed to support staff in transitioning to remote work, maintaining their well-being, and socially connecting now that they were unable to meet face-to-face, eat lunch together, or simply cross paths.   implementation   in response to the pandemic and our new remote work environment, the od team quickly implemented several new initiatives. first, with support from our marketing team and administration, we began hosting weekly all-staff town hall meetings. these town halls served as a vehicle for library leadership to share information and address staff concerns. in the early days of the pandemic, we invited a human resources leader to dialogue with staff and three library leaders from china to share their experiences reopening their libraries during the covid-19 pandemic. these town halls continue to be hosted, although now with less frequency.   in the first week of april 2020, the od team began a brief series of virtual sessions called “managing our work during covid-19” to share best practices and our individual experiences of working remotely. the od team began each session sharing the most current updates on the pandemic, as well as several strategies and resources for staff wellbeing and productivity, including tips on how to structure their day and strategies for eliminating distractions. staff were also encouraged to share their experiences.   in june 2020, we hosted our first in-house virtual conference. we invited proposals from all staff members and accepted 24 sessions that lasted an entire week. there was no specified theme for the conference in order to encourage participation from all staff, with topics ranging from mindfulness and imposter syndrome to library publishing and linking to electronic resources. in several cases we had first-time presenters, and the od team worked with them to develop their sessions.   outcome   prior to the pandemic, attendance to our in-person sessions varied greatly. depending on the topic of the session, we could have as many as 30 attendees or as few as three. in contrast, our new virtual sessions typically have at least 30 attendees. through a follow-up survey and personal correspondence, staff expressed that these sessions were beneficial in content as well as in providing a venue in which to connect with colleagues. we believe that the increase in attendance can also be attributed to the sessions’ accessibility since staff no longer have to travel across campus to attend them in person. from the earliest sessions, the od team recognized the opportunity that exists in hosting virtual sessions. while we are still primarily operating remotely, we anticipate continuing to offer virtual sessions when we return to campus.   for several of our staff members, the virtual conference was their first experience presenting and for others it was their first time attending a conference. unlike the librarians, many of our support staff do not have travel budgets, so the cost of attending a typical in-person conference presents a significant barrier. this barrier becomes increasingly problematic for support staff who are interested in furthering their career in the information professions or who are also working on their mlis. by offering an in-house free virtual conference, we removed these barriers and provided new opportunities for staff.   several staff members have expressed in meetings and personal correspondence that they feel a more “personal” connection with their colleagues. in addition to remote work often giving you a literal view into a coworker’s home, several of our activities have included sharing personal stories and interests.   reflection   when developing our od efforts, we considered how they fit into the learning organization model. for example, our leadership series had sessions to help staff improve skills and progress their own personal mastery. the “managing our work during covid-19” series helped our transition to a remote work environment by allowing staff to think about our mental models and our shared vision for the organization during the pandemic. the virtual conference provided us an opportunity to focus on team learning in a new way for our organization.   while we believe that these efforts to adjust to the pandemic and the new remote work environment were successful, there are still further adjustments and improvements that can be made. culture change can be a slow process and each successful project moves the organization closer to our shared vision. another important point is that staff development initiatives are never a one-size-fits-all approach; what works for one institution may not work for another and what impacts one individual may not impact another.   an additional challenge we faced was that, despite repeated calls from staff for more opportunities to gather and share information, few individuals suggested topics or were willing to facilitate these meetings. this may impact the sustainability of these initiatives over time. this has been less of an issue to date, as our needs changed fairly often as we came to understand the realities of the pandemic and remote work. as staff have become more comfortable with the realities of the pandemic and remote work, there is less demand for sessions related to this transition. staff have also begun developing their own remote networks, both informally and related to their job functions.   technological barriers are another challenge. some staff members have poor or nonexistent, internet service at home and other staff do not have the correct devices to work remotely. we were able to provide devices for these staff members, although it took some time to ensure that all employees were equipped as necessary. while the wsu has offered microsoft office 365 for several years, which includes the communication platform teams, it was not fully embraced as a productivity tool within the libraries. there was a learning curve for many of our staff in fully understanding and utilizing this tool. an added complication was that the wsu began offering zoom several months into the pandemic. while there are benefits to having several tools to choose from based on need, this created an additional learning requirement for staff who were unfamiliar with this new software.   based on the continued attendance and positive feedback we have received from our od efforts during the pandemic, we believe our transition to remote sessions and the associated rapid redesign of our efforts were successful. we have decided to make the virtual conference an annual event; planning for 2021 has already begun. the success of our “managing our work during covid-19” sessions has led to the creation of several other online events, ranging from social gatherings, such as a lunchtime dance party with a soul dj and a social event where the crafters and artists on staff shared their recent works, to well-being sessions targeting remote work and the isolation and stress of living and working through a pandemic.   references   aghina, w., & de smet, a. (2015, december 1). the keys to organizational agility. https://www.mckinsey.com/business-functions/organization/our-insights/the-keys-to-organizational-agility   gallup. (1999). cliftonstrengths assessment. gallup.   gallup. (2021). gallup certification. https://www.gallup.com/learning/309920/gallup-certification.aspx   giesecke, j., & mcneil, b. (2004). transitioning to the learning organization. library trends, 53(1), 54-67.   senge, p.m. (1990). the fifth discipline: the art and practice of the learning organization. doubleday.   society for human resource management. (n.d.). introduction to the human resources discipline of organizational and employee development. https://www.shrm.org/resourcesandtools/tools-and-samples/toolkits/pages/default.aspx   evidence based library and information practice 2011, 6.4 30 evidence based library and information practice news/announcements the 6th international evidence based library and information practice conference (eblip6): conference report and reflections 2011 brettle, grant, fraser, dalziel, gardois, lewis, kaalvik, and eldredge. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. conference report alison brettle senior lecturer/information specialist salford centre for nursing midwifery and collaborative research university of salford salford, great britain, united kingdom email: a.brettle@salford.ac.uk maria grant research fellow (information) salford centre for nursing midwifery and collaborative research university of salford salford, great britain, united kingdom email: m.j.grant@salford.ac.uk the 6th international evidence based library and information practice conference (eblip6) was hosted by the school of nursing and midwifery at the university of salford, in salford greater manchester, uk from june 27‐ 30, 2011, and was chaired by alison brettle and maria grant. planning for the conference took almost two years following a successful bid to host the conference in september 2009. the conference was attended in person by 170 delegates from 22 countries and 22 delegates registered to attend online using elluminate software. not surprisingly, the highest number of delegates came from the uk (71), although countries which had previously hosted eblip conferences were also well represented including canada (17), australia (7), usa (21), and sweden (17). delegates also traveled from qatar, saudi arabia, india, japan, nigeria, turkey, and estonia, which suggests that the interest in eblip is spreading far and wide. the conference planning extended as far as the weather, with the end of june seeing the best days of the whole british summer! the conference began with a series of pre‐ conference workshops, attended by over 40 delegates and facilitated by international experts. the workshops included: an introduction to evidence based library and information practice (andrew booth), reflective practice (barbara sen), critical appraisal (lorie kloda), and an introduction to meta‐synthesis (christine urquhart). all the workshop sessions and the facilitators were rated as good or excellent by the majority of participants. mailto:a.brettle@salford.ac.uk� mailto:m.j.grant@salford.ac.uk� evidence based library and information practice 2011, 6.4 31 the main conference was opened by professor tony warne, head of school of nursing and midwifery, who briefly offered his perspectives on evidence based practice. peter brophy followed with an engaging keynote “why marvin fell out of the top floor window last week: why narrative based practice matters.” keynotes on the remaining days were from martin hall, vice chancellor of the university of salford on “openness: the essential quality of knowledge” (see commentary in this issue); ross todd, rutgers university, “from information literacy to enquiry: implementing a holistic model of evidence based practice in school libraries”; and hazel hall, napier university and the library and information science research coalition, “project output versus influence in practice: impact as a dimension of research quality” (see commentary in this issue). the volume of submissions was high, as well as excellent, therefore the main programme was very full and organized into seven parallel sessions representing the conference themes of “innovation, education and research,” “practicality and applicability,” “outcomes, impact and value,” and “theory and reflection.” this translated into 59 presentations which were rated good or excellent by 89% of those who completed the evaluation. a selection of these presentations are summarized and published in this issue and we hope to include more in the future. the 20 poster presenters were given an opportunity to describe their posters in two “minute madness” sessions which were enjoyed greatly and highly rated by over three quarters of those who completed the evaluation. conferences are not just about presentations, they offer an opportunity to network and socialize with like‐minded professionals from a range of backgrounds. often the favourite part of conferences, we were keen to make sure that the social programme offered plenty of opportunities for this as well as highlighting local history and culture. time to show that the north of england is not just flat caps, whippets, and fish and chips! pre‐conference, there were visits to two historical libraries: chetham library, the oldest lending library in the world; and the working class library, which offers an insight into the history from the industrial revolution. the pre‐conference day ended with a pub quiz at the “ape and apple” where a joint team of eblip6 chairs and eblip journal editors managed a pretty good showing! at the end of the first day, the welcome reception held in salford art gallery and museum provided opportunities to meet the mayor of salford whilst enjoying local delicacies of meat and potato pie and red cabbage followed by manchester tart. the conference dinner held at manchester united’s famous football ground “old trafford” was for some “one of the most memorable evenings of their lives” and even for supporters of rival teams a “great evening despite the venue.” the evening began with a chance to wander round the museum, admire the trophies (although the cabinet was a bit emptier than the previous season), and drink wine. the evening continued with jazz music, a brief history of the club, and a chance to go into the stadium during the dinner itself. at the end of the final day, delegates attended a historical pub crawl (which comprised far more history than local ales) and an evening meal in manchester’s china town. andrew booth closed the conference with his personal reflections on the conference and the status of evidence based library and information practice (see commentary in this issue). andrew also presented the awards for best presentations and posters. the winners of the award for best presentation (by ipc and audience vote) were kate davis, queensland university of technology, and zaana howard, swineburn university of technology for their “redesigning evidence based practice for wicked problem solving.” winners of the ipc best poster were philip kroth, holly philips, and jonathan eldredge, university of new mexico for their “evaluation of an evidence based evidence based library and information practice 2011, 6.4 32 scholarly communication conference focused on support of translational investigators using a prospective longitudinal cohort design.” the winner of best poster as voted for by conference delegates was mary dunne, health research board, ireland for her “barriers and facilitators to research use: the role” (this has been written up as a paper and is included in this issue). a full list of winners (including those who were highly commended) was given in issue 6(3): 94‐95 of this journal. presentations from the conference are available from the conference website (www.salford.eblip6.ac.uk) and will ultimately be hosted on the university of salford repository along with podcasts of the presentations which took place in the main hall. watch this space for details of when these become available. report from day 1, tuesday, june 28, 2011 (used with permission from the library and information science research coalition blog) katie fraser information librarian leicester university leicester, great britain, united kingdom email: kcf5@le.ac.uk the day kicked off with a quick welcome from tony warne, head of school of nursing at the university of salford, talking about information literacy and the teacher‐student relationship. he was followed by the first of the conference keynotes, peter brophy. peter took us through the role of narrative in evidence based practice, from the stories that our users tell about our information services, to its underlying importance in capturing the complexity of our own everyday practice. it was a great start to the day, and by the end of the session i’d already had a conversation on twitter channel with an envious follower of the feed wishing that they were here! i attended parallel sessions on academic libraries and information literacy, both everyday strands of my own work. several of the talks picked up on peter’s ideas about the complexity inherent in library work, particularly those talks focusing on assessment and the challenges of reducing complex information behaviours to a mark scheme! i’ll definitely be considering some of these ideas in my own teaching. however, my favourite talk looked outside the world of library instruction. allyson washburn and sheila bibb – who teach an applied anthropology course – had asked anthropology students to conduct a series of ethnographic studies on student use of the library as coursework. it was fascinating to hear how the students had investigated the same topic from a variety of different angles, and there was also food for thought about the untapped opportunities academic librarians have to collaborate with departments in order to gather evidence: recruiting social scientists to help us gather evidence, computer scientists to develop our online services, and so on. the session i enjoyed most, however, was a little bit more outside my professional comfort zone, the post‐lunch discussion on ”theory and models of eblip.” i was hoping to get an overview of how researchers and practitioners see evidence based practice from this conference, and the three talks harmonized perfectly to answer this question. first helen partridge asked us to challenge what constitutes evidence in library and information practice. she suggested that most of our ideas about evidence based practice were inherited, and that we need to consider what constitutes “good” evidence in our own profession, and demonstrate that its use can transform practice. denise koufogiannakis followed this up with a discussion about the non‐traditional types of evidence that library and information professionals use: “local evidence” like user feedback, usage data, and observations gained in context; and “professional knowledge,” which is often tacit (highly contextual and mailto:kcf5@le.ac.uk� evidence based library and information practice 2011, 6.4 33 difficult to explain) or produced by reflection on our own practice. finally, barbara sen and chris lee spoke about evidence and reflection. both, they emphasized, are about critically examining everyday practice. each uses a different approach to examine that practice, but in the end they’re highly complementary. no research could begin without reflection on potential explanations and approaches to studying a problem. overall, it was interesting to hear that the library and information community is only just starting to reach an overview of how it sees and uses evidence based practice itself. report from day 2, wednesday, june 29, 2011 (used with permission from the library and information science research coalition blog) katrina dalziel deputy subject librarian (medicine) swansea university swansea, wales, united kingdom email: k.dalziel@swansea.ac.uk there was rain in manchester this morning but by the time the conference got under way on wednesday the skies were already brightening! the morning keynote session was delivered by professor martin hall, vice chancellor of the university of salford, our conference host. in “openness: the essential quality of knowledge” martin gave a thought‐provoking and entertaining talk focused on the question “what should a fully open‐access university, structured around an open access repository for publications, resources, and data sets, look like?” he argued that openness drives the knowledge economy. the closure of knowledge restricts innovation and is alien to the essential qualities of a university, especially in the context of how and why universities were established and evolved: to allow experts or academics in a field to share their knowledge in exchange for academic reputation. he asked publishers to look at their current business models and develop tools that could aid open access publishing and encourage a new era of openness in the world of academia. one comment made by martin that was particularly popular with the audience was that he considers all people involved in populating and promoting open access repositories as heroes. it was good to hear that librarians and information professionals are appreciated in this role! this first keynote session was followed by “poster madness.” this was a one minute madness presentation opportunity for those delegates presenting posters at the conference. i hadn’t experienced the one minute format before and was thoroughly impressed by the participants’ ability to provide coherent outlines of their research in such a short period of time. there was a second madness session in the afternoon, so i review both together here. i feel that i need to make special mention of those presenters that particularly impressed me. mary dunne from the health research board in ireland presented one minute on her “barriers and facilitators to research use: the role of library and information services” in poetry form. i also enjoyed emma thompson’s minute. she remarked on her position as one of just a couple of business librarians attending the conference. she questioned how to encourage all subject librarians to get involved in eblip. “ebp by stealth” was her recommendation! after the morning coffee break i attended the “innovation, education and research: theory and searching” strand. jason eyre of de montfort university discussed an alternative outcome to “the pitstop project (supporting students on placement using social media).” in “learning by example: developing evidence based library and information practice through supporting academic programmes with a culture of evidence based practice,” his main message was that social workers work in a “real” world environment where peer approval, time constraints and other issues mean that ebp mailto:k.dalziel@swansea.ac.uk� evidence based library and information practice 2011, 6.4 34 is not practiced. we need to remember that our students will be moving into real world situations and careers, where they may not even have access to the suite of resources that their academic library provides. our teaching of information literacy (il) needs to fit with this experience, and empathy is key. this presentation made me reflect on how i approach il instruction with nursing and medical students and how we need to offer advice and training that can fit into the real world of our students. the afternoon keynote session of the day was delivered by dr. ross j. todd, associate professor in the school of communication and information at rutgers, the state university of new jersey, and director of the center for international scholarship in school libraries (cissl) at rutgers university. this was another inspirational keynote address, this time reflecting on ebp from the school librarianship perspective. the whole day was informative, engaging, and thought provoking. i have a lot to reflect on in my own practice as an academic library professional and many issues to consider relating to how our profession can embrace ebp more fully. i think perhaps a good starting point would be the adoption of the evidence for practice, evidence in practice, and evidence of practice holistic model in both our practice and research activities. report from day 3, thursday, june 30, 2011 (used with permission from the library and information science research coalition blog) paolo gardois phd student school of health and related research. sheffield university sheffield, great britain, united kingdom email: paolo.gardois@sheffield.ac.uk professor hazel hall opened the final day of eblip6 in salford with a thought‐provoking keynote speech on “impact.” both patrons and managers demand services that really make a difference, and impact may take different forms: from changing users’ information behaviours, to assessing academic impact through bibliometric measures, or evaluating services based on specific outcome measures – especially in the academic sector. also, impact is very difficult to measure and evaluate. the impact of research on practice, for example, is often dependent on the cumulative and indirect effect of practitioners’ exposure to research output. impact counts, anyway! in the current economic climate research must demonstrate that it actually has an impact on practice, and that the research–practice gap should be bridged or at least reduced. hazel then shared with the audience evidence emerging from the lis research coalition’s rilies project which is due to report later this year. several factors play a key role in increasing research uptake by practitioners: quality, scale, and applicability of research itself; means of face‐to‐face dissemination; availability of accessible textual sources to be used as a reference in daily practice; high profile dissemination partners; and – last but not least – individuals who act as research connectors, as well as social media. hazel finished her presentation by referring to the question “what difference does it make?” appropriately citing the smiths, whose salford lads’ club photograph is now one of the most iconic in british music history. later in the morning parallel section 6 focused on a range of topics: web‐based services to enhance users’ experience of library services; analysis of electronic resources usage by patrons as a key indicator of value generated by academic library services; the development of evidence based services in academic and health libraries, and their impact on quality improvement. as budgets shrink and patrons’ expectations rise, all three sessions offered really mailto:paolo.gardois@sheffield.ac.uk� evidence based library and information practice 2011, 6.4 35 useful tools to improve service provision and demonstrate value for money. the session before lunch showed an innovative and interactive format: the lis research coalition organized a panel session involving lis practitioners and journal editors. the session aimed to improve communication between the two parties and help information professionals plan the publication of their work with a better understanding of the goals and practical steps involved in editorial processes. for example, the editors advised the careful project‐management of any potential publication, paying close attention to the information needs of the target journal’s audience, and not to underestimate the value of what professionals have to say to their colleagues and peers. aiming for a high standard of work is important, but the editors encouraged members of the audience not to be obsessed with perfection. peer reviewers can help improve the quality of work submitted with their feedback. importantly, the peer review process should be viewed as a dialogue during which both parties have a potential to learn. also worth emphasizing was the difference between research and practice based articles, for both of which there are specific lis journals. even negative results, which are rarely published, are of great interest to audiences. after a refreshing lunch and a final look at the posters (of amazing variety and really high quality), delegates were ready for the last two sessions of the conference. parallel session 7 engaged the audience on a wide array of issues related to innovation and development of services, including the role of libraries in the management of scientific datasets, performance measurement techniques such as activities‐ based costing, methodological reflections on best practices and the uptake of an evidence based approach in library services, and the available evidence base for evaluating the effectiveness of web 2.0 services. a specific session gauged the progress of evidence based practice in the health sector. here topics included the value of services offered by nhs libraries, the efficient use of bibliographic databases, and the impact of clinical librarianship on patient care and organizational objectives. then the time came for the closing address by andrew booth, who underlined the multidimensional and complex nature of “evidence based library and information practice.” virtually all the vocabulary used in the label can be discussed and modified, and the eblip6 conference had proved a valuable forum for the concepts to be discussed. andrew also pondered the future of eblip. one key development resides in focusing less on research and randomized controlled trials and more and more on what really needs to be done to improve users’ experience in a really messy world. andrew referred to the concept of “knowledge interaction,” which accounts for the need for genuine partnership between actors. picking up on previous speakers’ references to music (keynotes dr. ross todd and professor hazel hall had cited bjork and the smiths respectively), andrew recited his own version of the lyrics of the go‐go’s “my lips are sealed” to close the formal programme. then awards were conferred and votes of thanks given. finally it was “goodbye salford” after a very interesting and stimulating three days. conference reflection suzanne lewis manager, central coast health service library northern sydney central coast health gosford, new south wales, australia email: slewis@nsccahs.health.nsw.gov.au the themes of eblip6 were theory and reflection; outcomes, impact and value; practicality and applicability; and innovation, education and research, with the concurrent sessions organized according to the four themes. the theme i found most interesting was theory and reflection, as the speakers in these sessions mailto:slewis@nsccahs.health.nsw.gov.au� evidence based library and information practice 2011, 6.4 36 challenged their audiences to think about what comprises the evidence on which we claim to base our practice. this was appropriate for the 6th biennial eblip conference as it signals that over the last 12 years the paradigm has developed and matured. lis practitioners are no longer solely concerned with demonstrating eblip in practice, but are now (re)examining the theoretical basis of eblip and pausing to reflect on how the model might be developed and improved by recognizing points of convergence and synergies with other disciplines and theoretical models. helen partridge from queensland university of technology, australia, challenged the assumption that evidence in lis is derived solely or even mainly from research. this idea was expanded by denise koufogiannakis from the university of alberta, canada, who considered the place of practice based evidence in eblip. evidence apart from formal, published research has a valid place in eblip and includes local, user‐centred evidence such as usage statistics, user feedback, librarian observation, reports from colleagues, evaluation of progress, plus professional knowledge which includes informal and formal learning, tacit knowledge, and reflection. the challenge, of course, is how to capture practice based evidence, particularly tacit and corporate knowledge, which may explain why this kind of evidence has, to date, taken second place to research based evidence in the eblip paradigm. barbara sen and chris lee from the university of sheffield, uk, examined the commonality and divergence between the eblip model of library and information practice and the sea‐change (situation, evidence, action) reflective model developed by barbara. they acknowledged being inspired by denise’s editorial on reflective practice in the eblip journal (koufogiannakis, 2010), and also andrew booth’s “5 mirrors” model of reflection. all three papers challenged some of my assumptions about eblip and prompted me to consider aspects of my own professional practice relating to practice based knowledge and the value of reflection in informing practice. continuing with the stream of theory and reflection, the awards for best paper (delegates’ choice) and best paper (judged by the conference committee) both went to kate davis from queensland university of technology, australia, for her presentation entitled “redesigning evidence based practice for wicked problem solving.” this was an innovative and challenging conceptual paper which claimed that ebp focuses on answering the “easy” questions which have already been addressed by research. but what about complex, “wicked” problems for which there is no published evidence and which require agile, innovative thinking? kate and her co‐author zaana howard (swinburne university of technology, australia) proposed incorporating elements of design thinking and ebp into a hybrid approach to complex problems. it’s a fascinating idea and you can find out more at kate’s website (http://katedavis.info). apart from the formal program, two highlights of eblip6 for me were the conference dinner and poster madness! in the madness session, poster presenters had one minute to “sell” their poster to the delegates. a clock counted down on the screen behind them as they spoke and a siren sounded when time was up. despite the pressure, all the participants performed very well but the prize for the most entertaining “madness” presentation went to mary dunne of the health research board, ireland, who delivered an overview of “barriers and facilitators to research use: the role of library and information services” entirely in rhyming verse! finally, conference delegates were given the opportunity to visit what some might argue is the heart of manchester – the old trafford football stadium, home of the manchester united soccer club. pre‐dinner drinks were served in the old trafford museum and trophy room, where we browsed the glittering prizes http://katedavis.info/� evidence based library and information practice 2011, 6.4 37 and memorabilia of the club. we enjoyed dinner overlooking the famous pitch. the eblip6 conference brought together delegates from the uk, the usa, canada, australia, new zealand, scandinavia, india, nigeria, the west indies, japan, belgium, taiwan, the netherlands, and more. i look forward to continuing my involvement with eblip which is now a truly international community going from strength to strength. conference reflection hilde kaalvik sør trøndelag university college trondheim, norway email: hilde.kaalvik@hist.no comment on article presentation by ib lundgren: “the student and the information search process: library development using student voices” first, i would like to start by congratulating the author for his interesting contribution to the conference. ib lundgren works as a librarian at malmø university library. he primarily works with user services. his study is part of a two year ongoing project at malmø university. one of the main objectives of the project is to contribute to student retention. to succeed on that, i guess a good start is letting the starting point be ‐ the students! and that’s exactly what this study does. lundgren’s project attempts to reveal new knowledge about students’ information search process, from the student point of view. important questions on this topic are: “how do the students formulate their problems concerning information literacy?”; “what is the students’ knowledge?”; and, “what exactly are their problems?” to find out more about these issues lundgren and his team analyzed the book‐a‐librarian‐service. students have the possibility to book a librarian for an hour session of guided information searching. they book the session online by filling out a form where they have to answer some questions. these questions actually reveal their problems when it comes to information literacy. it appears that the so‐called ‘helpless student’ feels completely lost, asking questions she thinks are just silly. lundgren’s example is a student writing: “i’m lost. i want to know how you search a subject. how can i know what has already been done and what kind of literature i have to read? it might be silly to ask questions like this but as i said i’m lost.” the quite opposite student writes: “having trouble finding scientific articles, i’m mostly finding reviews.” the first question reveals that the student hardly knows anything about searching databases for literature. then you have her opposite – the student who knows much about information literacy, therefore asking questions revealing good knowledge about her search strategy. lundgren and his team are doing this content analysis because they want to develop their services for the students and, in the long run, become an inclusive learning environment at the university. after finishing the analysis of the book‐a‐ librarian‐service they are able to draw some conclusions. most importantly, they find that there is a great range between the ‘driven’ student and the much more helpless student when it comes to the knowledge of information literacy and the search process. it’s also important to notice that both kinds of students co‐exist in the same classroom, and of course in the same library. lundgren discusses that this diversity among students poses a pedagogical challenge concerning how the library should meet the different needs of a more and more heterogeneous student population. based on the results of the book‐a‐librarian analyses, lundgren concludes that the library should offer a diversity of services and solutions. with this in mind, they have developed their practices regarding il learning activities as well as the design of the physical library environment and web services. mailto:hilde.kaalvik@hist.no� evidence based library and information practice 2011, 6.4 38 why do i find lundgren’s study interesting? it is well known that as a teacher you have to be aware of the differences in knowledge amongst students. level of differentiation and individualization of teaching is a discussion we’ve had for a long time in education and pedagogy in norway and many other countries. lundgren’s study shows that librarians should or could do something more than just adjust teaching in information literacy (il). the study shows that it’s obviously desirable amongst students to make changes that adjust library services to the great differences in student’s il knowledge. what i find interesting is the reflection and that lundgren and his team collected information before implementing changes to improve service. i think it’s important to let student voices be the starting point for changes in the library. the students are our customers and that means we should be aware of how they think and what their problems might be. if you don’t reflect, don’t do some work to find out what your customer really needs and wants from your services, i’m afraid you might tend to just go on doing things like you always have done. that might be easy and comfortable, but it just can’t be smart thinking in the long run. the information crisis we are watching nowadays on the internet shows me that the librarian should be a guide for the students and help them to help themselves. in the article “eleven steps to eblip service” (2009), andrew booth concludes by saying: “above all, our mantra should be ‘don’t work harder, work smarter’‐ where smart means using your time and other resources more efficiently, more effectively and more economically.” i think that’s what lundgren is trying to achieve by letting the students’ voices be in the center of the study. i work as a librarian at the university college in sør‐trøndelag. my education is both teacher and librarian. i teach students il and i always try to reflect by asking myself: “do i do the right thing when it comes to how i teach?” i often discuss with my colleagues who also teach if we ought to make some changes. we’ve thought about the challenge because of the differences in the students’ knowledge gaps. in fact, they are all different, and i think you have to teach with this as a starting point. you should ask yourself: “how can i succeed teaching il?” what we have done recently is to have different kinds of tasks. some are easy, some are more difficult. in this way the more clever student doesn’t just sit and wait, updating her facebook profile because she’s already done with the exercises waiting for the more lost ones who really need time to finish. this is an easy thing to do, but not too obvious when you haven’t done it before or if you don’t know how different the knowledge amongst students is when it comes to il. another really important issue is how to teach so it works. that’s another question i think the librarians should be aware of and research. are we all good teachers? how should you teach to be able to give the students new knowledge? do they really learn or do you just teach without thinking about the learning process? do you have in mind that learning by doing is important? if you find they seem to be bored or don’t listen to what you have to say – could it perhaps have to do with your way of teaching? you have to teach how to teach. to reflect on why, what, and how to do things both in the classroom and in the library is the most important thing in my job as a librarian. my conference experience was entirely positive. both the social aspect and the academic content had a very high standard. i was a first time delegate and would really love to join eblip7. on the plane from norway to manchester i was in such a great conference mood and looking forward to meeting the rest of the delegates. attending a conference is a bit like going into a new dimension. it’s like we’re all thinking: “hey, we’re all in the same boat hoping to get to know each other and gain some new knowledge on our way!” when i arrived in manchester, the sun was shining and i had some fish and chips and a local beer. piccadilly garden was crowded and children were jumping in and out of the big water fountain. when i went on a double‐decker evidence based library and information practice 2011, 6.4 39 bus the next morning, after a very tasteful english breakfast at ramada hotel, i thought to myself: “this start is so good, will the conference meet my expectations?” my answer is: “yes, it did!” conference reflection jonathan eldredge associate professor health sciences library and informatics center the university of new mexico albuquerque, new mexico, united states of america email: jeldredge@salud.unm.edu as if following an unconscious symmetry, eblip6 circled back to the u.k. on the tenth anniversary of the first international evidence based librarianship conference (eblip1) held in sheffield during 2001. this writer keynoted at eblip1 during 2001so the occasion of his attending eblip6 during 2011 prompted him to reflect on the comparisons and contrasts of the two conferences. this reflection piece will focus upon geographic and sectorial representation of attendees, the worldwide eblip community, and the research methods referenced at the two conferences. sheffield university’s andrew booth and his colleagues sponsored eblip1 amidst growing excitement about the relatively new concept of evidence based librarianship. eblip1 included participants from non‐u.k. countries such as one each from australia, canada, denmark, norway, and the us. two attendees hailed from sweden. otherwise, eblip1 was largely a u.k. centred event. eblip1 still featured remarkable uk‐origin presentations originating in the u.k. such as catherine beverley and m. alison winning’s “clinical librarianship: a systematic review” (2003) and alison brettle’s “information skills training,” which later was published as a systematic review (2003). ellen crumley and denise koufogiannakis presented their first effort to define the subject domains of librarianship, which still inform current research (eldredge, harris, & ascher, 2009). other presenters included anne brice, maria grant, and margaret haines who all continue to be luminaries in librarianship. eblip6, in contrast, could truly claim international attendance. as noted elsewhere in the compilation of reports, 22 countries were represented at eblip6. u.k. health sciences librarians were the principal attendees at eblip1 so conference themes and perceptions of eblip concomitantly followed these attendance patterns. at that time eblip (or, “ebl” as it was then called) was having difficulty distinguishing itself from evidence based medicine despite the strenuous efforts of eblip pioneers to establish a distinct identity. perhaps health sciences librarians were so steeped in their own collaborative roles in evidence based medicine that this distinction inevitably took a couple of more years to take hold for all health sciences librarians. eblip6 during june 27‐30, 2011still attracted a contingent of librarians and other information professionals from the health sciences. the decided shift of eblip toward a multi‐sectorial movement was reflected in the attendance by academic, corporate, government agency, non‐ governmental organization, public, research institute, school, and special librarians. as proved to be the case in 2001, however, academic and health sciences librarians presented most research paper or posters. could this pattern reflect the fact that these presenters have far more institutional incentives to research and present their findings at professional conferences? this writer has distinguished between eblip producers and consumers for a number of years (eldredge, 2008). eblip6 confirmed this pattern since, while all attendees were interested in applying evidence in their decision‐making, only a minority of librarians produce applied research evidence. producers, it turns out, originate from institutions offering incentives to conduct that research. this pattern suggests the need for librarians and other information professionals who apply evidence in their practices to articulate their most mailto:jeldredge@salud.unm.edu� evidence based library and information practice 2011, 6.4 40 important research questions so that evidence producers’ research will be aligned with these professionals’ needs. eblip1 provided plentiful tea and meal time opportunities for face‐to‐face social networking among attendees. eblip1 focused primarily on such conference‐bound socializing venues. eblip6 provided a contrast, beginning with a pub quiz at the ape and ale pub in manchester the night before the eblip6 opened. teams formed quickly and competed readily amidst much merriment. during eblip6 groups of attendees continued their discussions following the last official conference events of the evening by migrating to pubs for conversations, some extending into the late night. overall, eblip6 displayed the characteristics of a more mature social movement with recognizable interlocking informal social networks. over the years a number of eblip conference attendees have developed close professional and even personal friendships. the aforementioned eblip1 paper and poster presentations were based upon both quantitative and qualitative research methods. eblip6 similarly offered both quantitative and qualitative research methods, just in far greater volume of papers (n = 57) and posters (n = 23) and far greater diversity of methods represented. abstracts of papers and posters can be accessed at the eblip6 website http://www.eblip6.salford.ac.uk/abstracts.php. this writer has attended all six eblip conferences. the many changes in the conferences over the years prompted this writer to wonder about the next 10 to 20 years. past eblip conferences have been hosted in the u.k., canada, the u.s.a., australia, and sweden. in what exciting new venues will future eblip conferences occur? what future developments await us at future eblip conferences? references beverley, c. a. & winning, m. a. (2003). clinical librarianship: a systematic review of the literature. health information and libraries journal, 20(suppl. 1), 10‐21. doi:10.1046/j.1365‐2532.20.s1.2.x booth, a. (2009). eleven steps to eblip service. health information and libraries journal, 26(1), 81‐84. doi:10.1111/j.1471‐ 1842.2008.00836.x brettle, a. (2003). information skills training: a systematic review of the literature. health information and libraries journal, 20(suppl. 1), 3‐9. doi:10.1046/j.1365‐ 2532.20.s1.3.x eldredge, j. d. (2008). evidence‐based practice. in m. s. wood (ed.). introduction to health sciences librarianship, (pp. 241‐ 269). new york: the haworth press. eldredge j. d., harris, m. r., & ascher, m. t. (2009). defining the medical library association research agenda: methodology and final results from a consensus process. journal of the medical library association, 97(3), 178‐185. doi:10.3163/1536‐5050.97.3.006 koufogiannakis, d. (2010). reflections on practice. evidence based library and information practice, 5(2), 1‐3. retrieved 29 nov. 2011 from http://ejournals.library.ualberta.ca/index .php/eblip/article/view/8266/6971 http://www.eblip6.salford.ac.uk/abstracts.php� / evidence based library and information practice research article   teaming up to teach teamwork in an lis master’s degree program   lauren h. mandel, phd associate professor graduate school of library and information studies, university of rhode island kingston, rhode island, united states of america email: lauren_mandel@uri.edu   mary h. moen, phd assistant professor graduate school of library and information studies, university of rhode island kingston, rhode island, united states of america email: mary_moen@uri.edu   valerie karno, phd, jd associate professor and director graduate school of library and information studies, university of rhode island kingston, rhode island, united states of america email: vkarno@uri.edu   received: 27 nov. 2019                                                             accepted: 10 apr. 2020      2020 mandel, moen, and karno. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29684     abstract   objective – collaboration and working in teams are key aspects of all types of librarianship, but library and information studies (lis) students often perceive teamwork and group work negatively. lis schools have a responsibility to prepare graduates with the skills and experiences to be successful working in teams in the field. through a grant from the university office of assessment, the assessment committee at the university of rhode island graduate school of library and information studies explored their department’s programmatic approach to teaching teamwork in the mlis curriculum.   methods – this research followed a multi-method design including content analysis of syllabi, secondary analysis of student evaluation of teaching (set) data, and interviews with alumni. syllabi were analyzed for all semesters from fall 2010 to spring 2016 (n = 210), with 81 syllabi further analyzed for details about their team assignments. some data was missing from the dataset of sets purchased from the vendor, resulting in a dataset of 39 courses with set data available. interviews were conducted with a convenience sample of alumni about their experiences with teamwork in the lis program and their view of how well the lis curriculum prepared them for teamwork in their careers (n = 22).   results – findings indicate that, although alumni remembered teamwork happening too often, it was required in just over one-third of courses in the sample period (fall 2010 to spring 2016), and teamwork accounted for about one-fifth of assignments in each of these courses. alumni reported mostly positive experiences with teamwork, reflecting that teamwork assignments are necessary for the mlis program because teamwork is a critical skill for librarianship. three themes emerged from the findings: alumni perceived teamwork to be important for librarians and therefore for the mlis program, despite this perception there is also a perception that the program has teamwork in too many courses, and questions remain about whether faculty perceive teaching teamwork as important and how to teach teamwork skills in the mlis curriculum.   conclusions – librarians need to be able to collaborate internally and externally, but assigning team projects does not guarantee students will develop the teamwork skills they need. an lis program should be proactive in teaching skills in scheduling, time management, personal accountability, and peer evaluation to prepare students to be effective collaborators in their careers.     introduction   while not all library and information studies (lis) courses emphasize teamwork, it is a crucial skill for students to be successful in the field (evans & alire, 2013; henricks & henricks-lepp, 2014). yet, how is teamwork taught and evaluated as a learning objective in a graduate library school program? the assessment committee at the university of rhode island graduate school of library and information studies (uri gslis) conducted a review of aggregated mean scores on the 12 learning objectives from the idea student evaluation of teaching (set) instrument, which includes learning how to work with others on a team. the idea student ratings of instruction are a proprietary set sold by campus labs; the instrument measures student self-reported perceptions of their learning on 12 idea learning objectives. the university administers the idea survey each semester, asking students to self-report their perceived learning for each of the 12 idea learning objectives, regardless of whether those objectives are relevant to the course. the assessment committee discovered that the mean score on objective 5, “acquiring skills in working with others as a member of a team,” was the lowest of all 12 objectives across all courses for which an idea survey was administered, 2010 to 2016. while this is a self-assessment of learning, instructors at uri gslis had informally discussed their observation of students’ negativity concerning group work, and the review brought to light the omission of teamwork or collaboration from the department learning outcomes. the committee determined that improving teamwork skills for lis students should be a department priority.   the terms collaboration, group work, and teamwork are often used interchangeably. the term used in the idea objective is “team,” which was the inspiration behind the title of this project. for purposes of this paper, teamwork refers to any assignment in a course that requires two or more students to work together to produce an output, whether this was labeled as group work, teamwork, partner work, or collaboration. there might have been one grade assigned to the group, or students might have been assigned grades individually.   teamwork assignments in lis education allow students to assess and build team skills for future use in the workplace (rafferty, 2013). working collaboratively in libraries is increasingly necessary as problems become more complex and resources become scarcer (calvert, 2018; laddusaw & wulhelm, 2018; marcum, 2014). collaborative projects help library staff develop relationship building skills that can be rewarding professionally. collaborating within a library can increase communications by breaking down silos, building trust among staff, leveraging skill sets that complement each other, and allowing all involved to contribute to projects and learn from colleagues (bello et al., 2017; calvert, 2018; cole, 2017). collaboration between libraries and other like-minded institutions can improve the visibility of library services by increasing the use of library resources and attendance at programs (laddusaw & wilhelm, 2018), raising public awareness of libraries (marcum, 2014), and increasing patron learning of information literacy skills (laddusaw & wilhelm, 2018; saines et al., 2019). based on the importance and benefits of collaboration for libraries, lis schools have a responsibility to prepare graduates with the skills and experiences to be successful working collaboratively in the field.   through a grant from the uri office of assessment, the committee designed this study to explore how teamwork was being taught across the curriculum and how alumni perceived their experiences working in teams both in the mlis program and their careers in order to identify possible interventions to improve the department’s approach to teaching teamwork and collaboration skills to mlis students. researchers examined artifacts of teaching (course syllabi and scores on the idea teamwork objective) and interviewed alumni about their experiences working in teams during the mlis program and in their careers. this study raised questions about what the skills of teamwork are, how important teamwork is perceived to be for lis careers, and how teamwork skills can be taught effectively in an mlis program. teamwork is a crucial skillset for lis students to learn as it is a requirement of most library jobs, but assigning team projects in courses is not enough; students need to be actively taught teamwork skills to prepare them for library jobs in which they will be asked to collaborate with colleagues inside and outside their libraries.   literature review   benefits of teamwork   teamwork is commonly utilized in higher education to develop students’ collaboration and teamwork skills (o’farrell & bates, 2009; rafferty, 2013; snyder, 2009). teamwork provides students the opportunity for peer-to-peer interactions that support learning and building one’s network (roy & williams, 2014). it leverages the strengths of team members and provides opportunities to explore their abilities in a safe educational setting. collaborative learning is particularly beneficial in professional master’s degree programs because of the positive aspect of sharing life experiences (oliveira et al., 2011).   student perceptions of teamwork   students report that they like teamwork because they can learn from peers and develop ongoing relationships (roy & williams, 2014) and that teamwork was effective at generating ideas (mckinney & cook, 2018). yet, they often see teamwork as a negative aspect of courses that utilize it (bernier & stenstrom, 2016). students do not enjoy having to depend on their peers who may have different objectives and levels of commitment from them (bernier & stenstrom, 2016; capdeferro & romero, 2012), they perceive there is an unfair system of reward and punishment for teamwork and that students get away with doing little or nothing (bernier & stenstrom, 2016; capdeferro & romero, 2012; mckinney & cook, 2018; roy & williams, 2014), they identify problems with logistics (bernier & stenstrom, 2016; capdeferro & romero, 2012), and they fear being stuck with all the work due to unbalanced workload among a team (capdeferro & romero, 2012; mckinney & cook, 2018; roy & williams, 2014). issues in communicating (o’farrell & bates, 2009; shah & leeder, 2016) and team dynamics (calvert, 2018) are also commonly cited challenges. students also perceived that the lack of instructor input, either guidance at beginning or assistance during a project, contributed negatively to teamwork experiences (capdeferro & romero, 2012). student learning style also can affect how students perceive teamwork; students who had negative perceptions of teamwork tend to prefer working alone (shah & leeder, 2016).   collaborative learning in lis education   since collaboration is an “essential skill for students to acquire and practise, as many real-world problems require us to work together” (shah & leeder, 2016, p. 609), then it is important for lis schools to teach students how to collaborate (bernier & stenstrom, 2016; roy & williams, 2014; shah & leeder, 2016). although students’ knowledge can increase during the teamwork process, so might their stress level (kim & lee, 2014). communicating remains a challenge even when students used a variety of electronic or digital resources during the teamwork process to share work (o’farrell & bates, 2009). structures such as a designated team leader, scheduled meetings, and clear and regular communication positively affect the team experience while perceived laziness of members does not (mckinney & cook, 2018). interventions such as a video on how to work successfully in small teams and explicit guidelines to enhance teamwork do not substantially lessen the negative attitudes students held about teamwork (bernier & stenstrom, 2016). how to teach teamwork in a way that students both learn from and enjoy it remains an area in need of further investigation.   methods   the uri gslis assessment committee conducted an assessment research project, funded by a university grant, to inform pedagogical improvement with regard to teamwork across the entirety of the lis curriculum, guided by three research questions:   what is the average idea score on objective 5 in lis courses that require teamwork, and how does this compare to the overall mean score across all lis courses? how is teamwork taught in the lis courses that require it? how effective do students perceive the curriculum to be in preparing them for teamwork in their careers?   this multi-method research included content analysis of syllabi, secondary analysis of set data, and interviews with alumni.   content analysis   the department had 210 syllabi from fall 2010 to spring 2016. the sample included courses delivered online, face to face, and in hybrid formats. a graduate assistant (ga) working on the research project analyzed the syllabi to identify which courses required team assignments. to ensure the most comprehensive dataset, all assignments that required two or more students to work collaboratively to produce a shared output were classified as teamwork for this study. the ga tabulated the number of both required and optional team assignments, the total number of assignments, and the percentage team assignments comprised of the total grade. syllabi were further coded for assignment type; inclusion of assignment descriptions and rubrics that detailed teamwork expectations, learning outcomes, or best practices/additional resources; and keywords used in teamwork expectations or learning outcomes.   secondary analysis   in the 2016-17 academic year, the department, with the support of the university provost’s office, purchased scores on the 12 idea learning objectives for all lis courses from fall 2010 to spring 2016 from campus labs (n=39). preliminary analysis focused on mean scores for the objectives across all courses (mandel, 2017). the secondary analysis dug deeper into the scores for individual courses on objective 5, comparing courses identified in the content analysis as requiring and not requiring teamwork. some data was missing from the dataset due to courses not having received an idea evaluation because they were taught by adjuncts or faculty nearing retirement, had low enrollment, or were taught in summer (uri had not been conducting idea evaluations on summer courses). other data was missing the course code on the faculty information form, so the courses could not be easily identified as lis courses by campus labs.   interviews   the project pi and ga conducted telephone interviews with a convenience sample of alumni about their experiences with teamwork in the lis program and their view of how well the lis curriculum prepared them for teamwork in their careers. alumni were asked first about their experiences with teamwork in the mlis program. they were asked to describe one or two specific assignments they did as part of a group, how the group coordinated the work and brainstormed, what they liked and disliked about group work, whether an instructor ever did anything to make their experience with group work easier or better, positive experiences working in groups and what made these experiences positive, and challenging experiences working in groups as well as strategies to mitigate or overcome those challenges. alumni were then asked about teamwork experiences in their careers. they were asked to describe their experience with group work in their career, how their group work experiences in the mlis program influenced their ability to work in groups on the job, what they like and dislike about group work on the job, and what recommendations they had for mlis instructors to prepare students for professional group work.   researchers used the department constant contact account to recruit alumni who attended the program between fall 2010 and spring 2016 to participate in the interviews. alumni were not asked about demographic data such as their gender, year of graduation, or the specific breakdown of the formats of the courses they had taken, but during the time they attended the program, 42.6% of program courses were offered in the hybrid format, 43.8% were offered online, and 13.6% were offered face to face. one interviewee stated during the first question that they were not really able to comment on the topic so that interview was not utilized, leaving 22 completed interviews, at which point the researchers were no longer learning anything new about alumni experiences with teamwork in the program and had reached saturation. both the pi and ga took notes during the interviews and then analyzed their notes thematically. their analyses were collated to produce one set of emergent themes.   results   lis courses that require teamwork   content analysis revealed that 81 courses in the sample required teamwork (38.6%). teamwork assignments were most frequently required in courses on management, reference, information science and technology, community relations, school library media, information literacy instruction, and research methods. this represents a mix of required and elective courses. other courses that required teamwork once in the sample period were collection management, academic libraries, instructional design, children’s literature, youth services, social science reference, government publications, archives and preservation, leadership, and internship. courses in instructional technology and social networking required teamwork twice during the sample period. optional teamwork assignments were found in courses on collection management, information science and technology, special libraries, and research methods.   the average number of teamwork assignments used in courses that require teamwork is 2.3. (the averages were 0.14 for courses with optional teamwork and 2.5 for all courses with teamwork assignments). the average number of total assignments per course is 13.4, meaning that required teamwork assignments comprised 19.0% of total assignments, on average (1.8% for courses with optional teamwork and 20.8% for all courses with teamwork assignments). assignment types were categorized as written, presentation, peer evaluation, discussion (either live in class or asynchronous via online discussion board), interview, project, or role play. the majority of teamwork assignments were written (n = 75; 87.2%), with the next most popular assignments being presentations (n = 50; 58.1%) and role play (n = 21; 24.4%); see table 1.   table 1 types of teamwork assignments used in lis coursesa assignment type total classes using % classes using written 75 87.2 presentation 50 58.1 role play 21 24.4 peer evaluation 13 15.1 discussion (live or online forums) 10 11.6 interview 1 1.2 project 1 1.2  asome courses had multiple types of teamwork assignments, so percentages exceed 100%.   forty-five syllabi included teamwork expectations or learning outcomes (52.3%), 14 included teamwork best practices or additional resources (16.3%), and 13 included peer evaluation assignments (15.1%). the most frequently mentioned topic in teamwork expectations or learning outcomes was collaboration (n = 60), followed by respect (n = 35) and functionality (n = 32); see table 2. best practices and additional resources included quotes, instructors’ advice on being a good member of a team, and a chart comparing teams versus groups referenced from a management textbook.   table 2 frequency of topics in teamwork expectations or learning outcomesa category n collaboration (including networks, partnerships, cooperation) 60 respect (including appreciate, recognize) 35 functionality (including evaluation, effectiveness, efficiency, practical) 32 communication (including synthesizing ideas, openness) 21 equitable workload 17 support (including coach, help, support, mentor) 14 professionalism (including collegiality) 13 decision-making (including democratic) 9 role-play 8 problem solving 5 trust (including rely on) 4 athree terms did not fit any categories: find inspiration, important, and wisdom (which appeared twice).   on average, teamwork comprises 29.3% of the total grade, ranging from 5% to 70%. most commonly, teamwork comprised 30% of the grade (n = 33; 38.4%). eleven course syllabi did not specify the percentage of the total course grade that teamwork assignments comprised. teamwork comprised a larger percentage of total course grades than it comprised of the total number of assignments (see figure 1).   figure 1 comparison of the percentage that teamwork comprised of total course assignments to total grade.   the dataset from campus labs included idea scores for 39 of the 81 courses identified as requiring teamwork (48.2%). while this is a smaller portion of the courses requiring teamwork than the researchers were hoping to analyze, analysis was still conducted. the aggregated mean idea score on objective 5 for these courses is 3.96. this is higher than the aggregated mean idea score on objective 5 for all courses in the time period, which was 3.34. given the size and nature of the sample (i.e., not random), the statistical significance of this difference could not be tested.   when instructors complete the faculty information form prior to administering the idea evaluation, they are asked to rate the 12 idea objectives as essential, important, or minor to the course. for the 39 courses in the dataset that required teamwork, 19 instructors selected objective 5 as essential or important (48.7%), and 17 instructors (43.6%) selected objective 5 as minor or no importance. the highest aggregated mean score on objective 5 was for classes in which instructor selected objective 5 as "important" (4.04), with next highest for instructors who selected "minor/no importance" (3.97), followed by instructors who selected "essential" (3.88); see table 3.   table 3 aggregated mean score for courses requiring teamwork by instructor-selected importance of objective 5 aggregated mean score importance selected 4.04 important 3.97 minor/no importance 3.88 essential 3.84 default-impa athis category indicates the instructor did not identify the objective as essential, important, or minor (i.e., left the selection blank).   alumni perceptions of teamwork   while the interview questions specified “group” and “group work,” alumni responded using teamwork, group work, collaboration, and other terms interchangeably. a few interviewees shared very bad experiences in courses with team members who did not pull their weight, professors who did not help them make a bad situation better, or where they felt the professor did not want to hear complaints. most interviewees reported positive experiences with teamwork in the program, but they also remembered teamwork happening too often, and one reported feeling “wow, we’re in a group again. we’re always in a group.”   the majority of interviewees recalled enjoying the social aspects of working in teams the most: meeting new people, forming lasting personal and professional relationships, collaborating, sharing ideas and perspectives, and appreciating others’ strengths. they enjoyed learning how to work with other people, improving their communication skills, and learning from other students’ experiences at other libraries or other library types. working in a team also afforded greater support when one person was struggling. it also helped in brainstorming ideas and in accomplishing more than the team members could alone. on the job, interviewees reported they enjoy the opportunities they have to collaborate, share ideas and perspectives, motivate, and inspire each other. they perceive that teamwork on the job helps to promote productivity and gain a better understanding of their institution or organization as a whole.   the biggest issue mentioned about teamwork in classes was scheduling, especially for teams of more than three people and when one or more members wanted to meet in person and the others did not want to or could not do that. the second biggest issue is dealing with the student who does not pull their weight or drops off the radar. interviewees wanted to make sure everyone had equal parts and did their share. when a teammate did not contribute, interviewees indicated they wanted or needed the professor to get involved or suggested that instructors have a process and policy set out in advance to handle those situations. during challenging team dynamics or experiences, they appreciated having a written team contract to clearly state team expectations and provide a process for resolving the issues. in addition, peer evaluations eased the tension when team members were not pulling their own weight and ensured accountability.   other challenges reported by interviewees included stress from not being able to reach a team member, unclear roles and lack of leadership in a team, assignments that did not lend themselves to teamwork or that did not have a clear relevance for the job, and having to trust other people to do their part of an assignment. regarding leadership, one person noted the challenge could be especially high in a program with many introverts who do not want to take on a leadership role. there were also concerns about how to call out people for not doing their share when you do not know them well and may never have met in person (interviewees did not specify whether they were recalling face-to-face, hybrid, or online courses).   challenges to working in teams on the job include inability or lack of desire to compromise or give up control when one has a particularly vivid idea or vision and frustration when each step needs approval from someone higher up. interviewees also dislike difficult power dynamics and confrontation when working in groups on the job. one said, “there is discord in groups,” so you have to know how to deal with it.   interviewees concur that using teamwork is an everyday part of work in libraries. they said things like, “pretty much every library you work at, you're working with a team of people” and “group work is a huge part of my career. if you are not able to do group work as a librarian, you are not going to be happy, build strong professional connections, or get much done.” only one interviewee said they never work in teams, but they had graduated less than a year prior the interview and had sought committee work to obtain teamwork experience. interviewees said teamwork assignments are necessary for the mlis program but that the department should take care to actually teach how to work in teams, use teamwork when appropriate for assignments, and not assign teamwork to decrease instructors’ grading responsibilities.   the majority of interviewees believed that teamwork experiences during their mlis program influenced their ability to work in teams on the job; only five were not sure or did not feel that it directly influenced their real world experiences. interviewees felt that they were better prepared for real world experiences; they were able to identify personal strengths and weaknesses, knew when to take the lead and when to step back, and understood warning signs of team conflict; they knew how to listen and communicate respectfully, the importance of laying out expectations, how to use new communication technology, and how to be flexible.   interviewees reported that the program stressed that being a librarian means constantly sharing and improving on ideas through being an open community. librarians can always tap into their networks. student work in the mlis program helped formed the idea that “we’re all in this together towards a common goal” and librarianship is less competitive than other industries. no matter how annoying teamwork may be in school, interviewees reported that it is necessary because it is part of the job. a few disputed this, but mostly they agreed that, “good or bad, it’s an extremely valuable learning experience.”   discussion   answering the research questions   the average idea score on objective 5 in lis courses that require teamwork (rq1) is higher than the overall mean score on that objective across all lis courses. however, the difference is less than one point, and the significance cannot be measured given the limitations of the sample size and quality. the average score on this objective is higher for instructors who indicate this objective is important than for instructors who indicate this objective is essential (the highest-level priority). follow-up research should investigate instructors’ perceptions of the relationship between the teamwork they assign and their selection of important and essential objectives.   in lis courses that require teamwork (rq2), teamwork comprises less than three assignments, about 20% of the total class assignments and about 30% of the total class grade, and it is primarily focused on written and presentation assignments. only slightly more than half of courses that use teamwork give any sort of expectations or learning outcomes in the syllabus, and less than a quarter include best practices, additional resources, or peer evaluation assignments. it seems that, in this program, teamwork is utilized but not necessarily taught. the most commonly mentioned topic in teamwork expectations and learning outcomes is collaboration, which reflects the focus in the literature on the importance of collaboration in libraries. here too, future research should look at instructor perceptions of teaching teamwork, such as the instructor’s purpose or goal in assigning teamwork.   over three-quarters of the alumni interviewees reported that teamwork experiences during their mlis program had a positive influence on their ability to work in teams in their careers (rq3). while they find compromise, ceding control, and office politics to be frustrating, they reported that what they learned in the mlis program prepared them to identify their own strengths and weaknesses as a team member, when to step up or step back, and warning signs of impending conflict. they also learned communication and technology skills that made them better able to negotiate teamwork in their careers. critically, alumni reported that the program helped them see that librarians are constantly collaborating, preparing them for the realities of their day-to-day work.   perceived importance of teamwork for librarians   both the literature and our alumni report that being able to work in teams, groups, committees, or other multi-person arrangements is a critical skillset for librarianship. a key aspect of this is collaboration, which is seen as an “essential skill” (shah & leeder, 2016, p. 609) that is necessary for library work (calvert, 2018; laddusaw & wulhelm, 2018; marcum, 2014). collaboration is the most frequently used term in teamwork expectations or learning outcomes in the syllabi analyzed for this study, and it is mentioned in the ala and llama competencies (ala, 2008; llama, 2016), along with other teamwork skills: emotional intelligence, conflict resolution, and problem solving (llama, 2016).   all but one of the alumni interviewees reported working in teams on the job. they perceive teamwork as an essential component of librarianship and library school as a crucial place to learn how to work with others to achieve a common goal. alumni perceive that the program should teach self-assessment, conflict management, respectful communication, setting expectations, collaborative technology tools, flexibility, and knowing when to lead or when to go with the flow of the team.   perception of “too much” teamwork   even though alumni perceive teamwork as essential to librarianship and a crucial skillset for the mlis program to teach, they also perceive the program as having teamwork in too many courses. the reality is that teamwork was required in a little over one-third of the courses in the sample set. the program requires 36 credits (i.e., 12 courses), suggesting that most students would experience 3 to 4 courses with teamwork. however, because many of the required courses (management, reference, information science and technology, research methods, and internship) required teamwork, students may have taken even more courses with teamwork than that.   there are three tracks in the program: school library media (slm); libraries, leadership, and transforming communities (lltc); and organization of digital media (dm). about 25 to 30% of students are on the slm track with 5 to 10% of students on the other tracks at any given time. the majority of students are not on a track. depending on the track, students may have actually taken half or more of their credits in courses that used teamwork:   ·        slm track. students are required to complete management, reference, information science and technology (or research methods as the requirements shifted from one to the other during the sample period), school library media, information literacy instruction, and children’s literature. ·        lltc track. students are required to complete management, reference, information science and technology or research methods, internship, community relations, and leadership, and many students on this track elect to take collection management. ·        dm track. students are required to complete management, reference, information science and technology or research methods, internship, and many students on this track elect to take collection management and information literacy instruction. ·        general track. students are required to take management, reference, information science and technology, and internship, and many elect to take collection management.   for a student attending full time (three courses per semester), this could mean one or two courses requiring teamwork every semester they are in the program. for part-time students, it could be they are assigned teamwork every other semester or more often, and any student could be in two courses requiring teamwork concurrently.   one way the department might tackle this perception of too much teamwork is to tie teamwork to two required courses to ensure all students have to learn the skills at both an introductory and reinforcement level, but then strongly suggest it be avoided in electives. teamwork could be added to the catalog descriptions of the two courses so students would know which courses require teamwork and arrange their schedules accordingly. the department could review the idea objective 5 scores only for the two designated “teamwork” courses to track any changes on this objective over time.   another approach is to change students’ perceptions of teamwork, so they look forward to, or at least do not dread, teamwork assignments. improving how teamwork is taught can help with this (see next section), but the department may need to undertake a pr campaign as well. the department could record short videos of students and alumni reflecting on the positive aspects of teamwork in the program and their careers and show these videos at new student orientation and the beginning of courses requiring teamwork. instructors could also ask students at the beginning of the term to reflect on positive experiences they have had with teamwork in the past and consider what made those positive and how they can work with their teammates to replicate what worked previously.   implications for lis curriculum   there is an issue about the degree to which faculty perceive teaching teamwork as important. three of the full-time faculty in the program are the investigators on this project, but it gives us pause that, even in classes that require teamwork, faculty do not identify teamwork as an essential learning objective for the course either on the idea instrument or their syllabus. might that be due to the fact they are not explicitly teaching teamwork skills or due to the low percentage teamwork assignments comprise of total course assignments and grades? how can we garner faculty buy-in for a focused effort on teaching teamwork?   our alumni tell us that teamwork is a critical skill for librarianship and that our students need to be prepared to be effective members of teams when they graduate, and the literature supports this. but how do we teach the soft skills of teamwork? it is clear from this research that we have considerable room for growth in this area. for example, peer evaluation assignments are considered a teamwork best practice (capdeferro & romero, 2012; roy & williams, 2014; xu et al., 2013), but they were used in only about 15% of courses that employed teamwork assignments. none of the syllabi indicated that the courses are actively teaching the specific teamwork skills alumni identify having learned. the required management course did cover the topic of managing teams for one week, but are we truly expecting our students to learn how to communicate, negotiate, and lead in teams without formal training? also, alumni report the biggest issues of teamwork are scheduling and managing teammates who do not do their fair share of work; yet these topics are rarely covered in teamwork expectations and learning outcomes in course syllabi.   based on the findings, the investigators in this study are designing a teamwork instructional module that can be utilized in any course in the program. the goal of this module is to make it easy for faculty to teach teamwork without adding the burden of an additional topic to their teaching load and to provide a consistent teamwork language and approach across the mlis curriculum. the module includes a lesson on teamwork covering definitions and benefits of teamwork, what kind of teammate you are, and strategies for working as part of a team; a quiz faculty can adopt as either a formative or summative assessment; a sample team contract template; and a sample peer evaluation instrument. one of the members of the research team implemented team contracts in spring 2016, and some of the alumni who were interviewed referred to that document as smoothing over a lot of potential areas of conflict among team members. other faculty have since adopted a team contract and anecdotally report fewer instances of needing to step in to help a team resolve conflict. the module is being piloted, and results will be reported in future publications.   limitations   this study focused on the perceptions of alumni from one mlis program so the results cannot necessarily be generalized beyond our own students and alumni. however, the make-up of the student body at most u.s. lis schools is similar, and it is likely that the learning styles of students in one program mirror the learning styles of students in other programs. there is some question about why our alumni reported such positive experiences with teamwork in their program when the literature indicates one should expect otherwise. it is possible that the gap in time between being a student and working in the professional world could have mitigated feelings of stress and frustration. also, alumni who volunteered to be interviewed may be more likely to work better in teams, work well with others, and feel comfortable taking on responsibility than the student who goes missing during an assignment or drops out of the program.   conclusion   teamwork is prevalent in all aspects of the library field. it is critical for students in lis programs to develop teamwork skills so they can be successful in their jobs. librarians need to be able to collaborate internally within their libraries and forge external collaborations beyond their libraries to secure grant funding, develop partnerships, and promote advocacy. assigning team projects does not guarantee students will develop the teamwork skills they need. lis schools can follow the lead of the business management field that has specifically researched how to teach teamwork (rafferty, 2013; snyder, 2009; yazici, 2005). taking an active role in teaching skills in scheduling, time management, personal accountability, and peer evaluation may help overcome the limited way this lis school is currently teaching teamwork. other questions still need to be investigated, such as instructors’ perceptions of teamwork as an essential learning objective and ways to make teamwork assignments more successful for students. this assessment project is a first step in the direction of developing a program-wide curriculum that prepares lis students to be productive and effective members of teams, groups, committees, collaborations, and partnerships in their careers.   references   american library association. (2008). core competencies. retrieved from http://www.ala.org/educationcareers/careers/corecomp/corecompetences   bello, l., dickerson, m., hogarth, m., and sanders, a. (2017). librarians doing dh: a team and project based approach to digital humanities in the library. collaborative librarianship, 9(2), 97–103. retrieved from https://digitalcommons.du.edu/collaborativelibrarianship/vol9/iss2/6/   bernier, a., & stenstrom, c. (2016). moving from chance and “chemistry” to skills: improving online student learning outcomes in small group collaboration. education for information, 32, 55–69. https://doi.org/10.3233/efi-150960   calvert, k. (2018). collaborative leadership: cultivating and environment for success. collaborative librarianship, 10(2), 79–90. retrieved from https://digitalcommons.du.edu/collaborativelibrarianship/vol10/iss2/4/   capdeferro, n., & romero, m. (2012). are online learners frustrated with collaborative learning experiences? the international review of research in open and distance learning, 13(2), 26–44. https://doi.org/10.19173/irrodl.v13i2.1127   cole, m. (2017). what collaboration means to me: collaboration as a cocktail. collaborative librarianship, 9(2), 74–76. retrieved from https://digitalcommons.du.edu/collaborativelibrarianship/vol9/iss2/2/   evans, g. e., & alire, c. a. (2013). management basics for information professionals (3rd ed.). chicago, il: neal-schuman.   henricks, s. a., & henricks-lepp, g. m. (2014). desired characteristics of management and leadership for public library directors as expressed in job advertisements. journal of library administration, 54(4), 277–290. https://doi.org/10.1080/01930826.2014.924310   kim, j., & lee, j. (2014). knowledge construction and information seeking in collaborative learning. canadian journal of information and library science, 38(1), 1–21. https://doi.org/10.1353/ils.2014.0005   laddusaw, s., & wilhelm, j. (2018). yours, mine and ours: a study of successful academic & public library collaboration. collaborative librarianship, 10(1), 30–46. retrieved from https://digitalcommons.du.edu/collaborativelibrarianship/vol10/iss1/6/   library leadership and management association (llama). (2016). leadership and management competencies. http://www.ala.org/llama/leadership-and-management-competencies   mandel, l. h. (2017). aggregated idea data for lsc courses, fall 2010-spring 2016. retrieved from https://harrington.uri.edu/wp-content/uploads/2018/04/aggregateideadata-report_revised.pdf   marcum, d. (2014). archives, libraries, museums: coming back together? information & culture, 49(1), 74–89. https://doi.org/10.7560/ic49105   mckinney, p., & cook, c. (2018). student conceptions of group work: visual research into lis student group work using the draw-and-write technique. journal of education for library and information science, 59(4), 206–227. https://doi.org/10.3138/jelis.59.4.2018-0011   o’farrell, m., & bates, j. (2009). student information behaviours during group projects: a study of lis students in university college dublin, ireland. aslib proceedings, 61(3), 302–315. https://doi.org/10.1108/00012530910959835   oliveira, i., tinoca, l., & pereira, a. (2011). online group work patterns: how to promote a successful collaboration. computers & education, 57, 1348–1357. https://doi.org/10.1016/j.compedu.2011.01.017   rafferty, p. d. (2013) group work experiences: domestic mba student experiences and outcomes when working with international students. journal of further and higher education, 37(6), 737–749. https://doi.org/10.1080/0309877x.2011.645470   roy, l., & williams, s. (2014). reference education: a test bed for collaborative learning. the reference librarian, 55(4), 368–374. https://doi.org/10.1080/02763877.2014.929923   saines, s., harrington, s., boehringer, c., campbell, p., canter, j., & mcgreary, b. (2019). reimaging the research assignment: faculty-librarian collaborations that increase student learning. college and research libraries news, 80(1), 14–17, 41. https://doi.org/10.5860/crln.80.1.14   shah, c., & leeder, c. (2016). exploring collaborative work among graduate students through the c5 model of collaboration: a diary study. journal of information science, 42(5), 609-629. https://doi.org/10.1177/0165551515603322   snyder, l. g. (2009). teaching teams about teamwork: preparation, practice and performance review. business communication quarterly, 72(1), 74-79, https://doi.org/10.1177/1080569908330372    xu, j., du, j., & fan, x. (2013). individual and group-level factors for students' emotion management in online collaborative groupwork. internet and higher education, 19, 1–9. https://doi.org/10.1016/j.iheduc.2013.03.001   yazici, h.j. (2005). a study of collaborative learning style and team learning performance. education & training, 47(2/3), 216–229. https://doi.org/10.1108/00400910510592257       conference paper   informed systems: enabling collaborative evidence based organizational learning   mary m. somerville university librarian and library director university of colorado denver denver, colorado, united states of america email: mary.somerville@ucdenver.edu   niki chatzipanagiotou phd candidate linnaeus university växjö, sweden email: niki.chatzipanagiotou@lnu.se    received: 3 aug. 2015     accepted: 28 oct. 2015      2015 somerville and chatzipanagiotou. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – in response to unrelenting disruptions in academic publishing and higher education ecosystems, the informed systems approach supports evidence based professional activities to make decisions and take actions. this conceptual paper presents two core models, informed systems leadership model and collaborative evidence-based information process model, whereby co-workers learn to make informed decisions by identifying the decisions to be made and the information required for those decisions. this is accomplished through collaborative design and iterative evaluation of workplace systems, relationships, and practices. over time, increasingly effective and efficient structures and processes for using information to learn further organizational renewal and advance nimble responsiveness amidst dynamically changing circumstances.   methods – the integrated informed systems approach to fostering persistent workplace inquiry has its genesis in three theories that together activate and enable robust information usage and organizational learning. the informationand learning-intensive theories of peter checkland in england, which advance systems design, stimulate participants’ appreciation during the design process of the potential for using information to learn. within a co-designed environment, intentional social practices continue workplace learning, described by christine bruce in australia as informed learning enacted through information experiences. in addition, in japan, ikujiro nonaka’s theories foster information exchange processes and knowledge creation activities within and across organizational units. in combination, these theories promote the kind of learning made possible through evolving and transferable capacity to use information to learn through design and usage of collaborative communication systems with associated professional practices. informed systems therein draws from three antecedent theories to create an original theoretical approach.   results – over time and with practice, as co-workers design and enact information-focused and evidence based learning experiences, they learn the way to decision-making and action-taking. increasingly more complex experiences of information exchange, sense making, and knowledge creation, well supported by workplace communication systems and professional practices, further dialogue and reflection and thereby enrich analysis and interpretation of complexities and interdependencies.   conclusions research projects and evaluation studies conducted since 2003 demonstrate the transformative potential of the holistic informed systems approach to creating robust workplace learning environments. leaders are responsible for design of workplace environments supportive of well contextualized, information-rich conversations. co-workers revisit both the nature of organizational information and the purpose of organizational work. as colleagues better understand the complexities of the organization and its situation, they learn to diagnose problems and identify consequences, guided by informed systems models. systemic activity and process models activate collaborative evidence based information processes within enabling conditions for thought leadership and workplace learning that recognize learning is social. enabling communication systems and professional practices therefore intentionally catalyze and support collegial inquiry to co-create information experiences and organizational knowledge through evidence based practice to enliven capacity, inform decisions, produce improvements, and sustain relationships. the informed systems approach is thereby a contribution to professional practice and workplace renewal through evidence based decision-making and action-taking in contemporary organizations. contextual introduction   the search for a robust approach for catalyzing organizational learning experiences arose in 2003 within a north american academic library experiencing unprecedented changes and persistent uncertainty. volatile forces within the scholarly ecosystem had irrevocably altered traditional relationships among researchers, librarians, publishers, and vendors (somerville, schader, & sack, 2012; somerville & conrad, 2013; 2014), requiring new workflows and workplace competencies. in addition, changing   pedagogical practices and new business models in higher education (e.g., coaldrake & stedman, 2013; crow & dabars, 2015) necessitated redesigning facilities, reconsidering collections, and reinventing services. these converging forces required that staff members learn to see their organizations and understand their roles in new ways because “library services in higher education will continue to be crucial to the core processes of learning, teaching, and research as long as the key library structures, processes, services, and staff roles evolve to accommodate epochal changes occurring in publishing and communications” (wawrzaszek & wedaman, 2008, p. 2).   more than a decade later, unrelenting disruptions in both higher education and scholarly communication ecosystems continue, fundamentally challenging traditional assumptions about academic library roles, responsibilities, services, and facilities. as a consequence, academic librarians around the globe are asking:   how could the library organization better reflect the vision of the institution of which it is a part? how could the library and its collections, services, and spaces best serve the institution? how do library outcomes add value to the academic experiences of students and faculty? how might the library function more interdependently with other campus learning and teaching activities? what programs not in the library at present should be in the facility in the future? (lippincott, 2014; hemmasi, lefebvre, lippincott, murray-rust, & somerville, 2015).   such enterprise level questions hold considerable promise for catalyzing constituent engagement, creating shared vision, and building stakeholder partnerships. their profound importance in forging vital new directions underscores the inadequacy of reliance on mere ‘busyness’ statistics, such as gate counts and pdf downloads, for evidence. rather, “systemic changes require systemic responses because a case-by-case or incident-by-incident response was inadequate, given the magnitude of transformation underway” (somerville, 2015, p. 45). in response, informed systems – which integrates complementary informationand learning-focused theories – addresses a research-in-practice problem – i.e., the lack of an integrated model to inform workplace learning in contemporary information and knowledge organizations.   the informed systems approach supports evidence based professional activities to make decisions and take actions. it enables co-workers to make informed decisions by identifying the decisions to be made and the information required for those decisions. this is accomplished through collaborative design and iterative evaluation of workplace systems, relationships, and practices. over time and with experience, increasingly effective and efficient structures and processes for using information to learn advance organizational renewal and nimble responsiveness amidst dynamically changing circumstances.   informed systems principles and practices exercise and enable participatory design, action learning, and perpetual inquiry through “using information to learn” (bruce, 2008) in ever expanding professional situations. a persistent focus on cultivating rich information experiences through information-centered and action-oriented dialogue and reflection serves to advance information exchange and knowledge creation, through which transferable learning occurs and organizational capacity builds (somerville, mirijamdotter, bruce, & farner, 2014). this conceptual paper presents systemic activity and process models that activate collaborative evidence based information processes within enabling conditions for thought leadership and workplace learning.   antecedent thought   this integrated approach to fostering persistent workplace inquiry has its genesis in three theories that together activate and enable robust information usage and organizational learning. the informationand learning-intensive theories of peter checkland in england, which advance systems design, stimulate participants’ appreciation during the design process of the potential for using information to learn (checkland & holwell, 1998). within a co-designed environment, intentional social practices continue workplace learning, described by christine bruce in australia as informed learning (bruce, 2008) enacted through information experiences (bruce, davis, hughes, partridge, & stoodley, 2014). in combination, these theorists promote the kind of learning made possible through evolving and transferable capacity to use information to learn through design and usage of collaborative communication systems with associated professional practices.   in addition, in japan, ikujiro nonaka’s theories foster information exchange processes and knowledge creation activities within and across organizational units. an organization is thereby considered a knowledge ecosystem consisting of a complex set of interactions between people, process, technology, and content. knowledge emerges through exchange of resources, ideas, and experiences through which individual knowledge becomes corporate knowledge (nonaka, 1994).  this “knowledge-related work requires thinking – not only monitoring, browsing, searching, selecting, finding, recognizing, sifting, sorting and manipulating but also being creative, always questioning, interpreting, understanding situations…with particular focus on how to put questions, draw inferences, give explanations and conclusions, prioritize” (materska, 2013, p. 231) within increasingly complex and ever-changing environments.   stated differently, informed systems learning outcomes emerge through integration of multi-disciplinary theory from around the world. according to checkland’s soft systems methodology (ssm), learning emerges through collaborative design of organizational systems and professional practices (checkland & scholes, 1990; checkland & poulter, 2010). in a complementary fashion, bruce recognizes that collective understanding advances through intentional use of information to learn in the workplace (i.e., bruce, 1997; 1998; 1999; 2008; 2015), while nonaka emphasizes the possibilities for social knowledge creation within workplace environments (nonaka, 1994; nonaka & takeuchi, 1995; von krogh, ichijo, & nonaka, 2000; nonaka, konno, & toyama, 2000; nonaka & toyama, 2007). in a highly synergistic fashion, these antecedent ideas have, in combination, informed the evolution of models for enabling and enacting collaborative evidence based decision-making, both creating requisite conditions and guiding learning processes. at its essence, informed systems recognizes that when information is managed effectively, it facilitates collaboration among co-workers that furthers decision-making and advances organizational learning based on that information (chatzipanagiotou, 2015).   since 2003, informed systems evolved to foster information exchange, reflective dialogue, knowledge creation, and conceptual change. results from evaluative studies (e.g., somerville, schader, & huston, 2005; somerville, rogers, mirijamdotter, & partridge, 2007; somerville, 2009; mirijamdotter & somerville, 2009; somerville, 2015) reveal that, over time and with practice, this collaborative learning approach progresses co-workers’ capacity for creating systems and producing knowledge, activated by participatory design, amplified by systems thinking, and exercised by informed learning. in “working together” (somerville, 2009) to generate knowledge, colleagues contribute complementary knowledge skills, work responsibilities, and social statuses which advance social, relational, and interactive aspects of work life (townsend, 2014). capacity builds through using information to learn in ever expanding professional contexts that exercise evidence based decision-making and action-taking.   approach fundamentals   research-in-practice project results from 2003 to 2006 at california polytechnic state university in san luis obispo (e.g., mirijamdotter & somerville, 2009; somerville, 2009) and at the university of colorado in denver from 2008 through 2015 (e.g., somerville & howard, 2010; somerville & mirijamdotter, 2014; somerville, 2015) demonstrate the efficacy of cultivating informed learning experiences within enabling, co-designed workplace systems. after considerable dialogue and reflection among the international research team, somerville, mirijadmotter, and bruce, the approach was named informed systems in 2012 and introduced in a multi-author book on international information experience in 2014 (somerville & mirijamdotter, 2014). in california, some early principles for workplace leadership emerged from pilot projects. these elements recognize the recursive nature of systems perspectives and knowledge practices for workplace leadership that aims to further organizational learning.   integral to creation of a robust learning organization, leaders are responsible for design of workplace environments supportive of information-rich conversations. systems thinking can be used to contextualize workplace issues in terms that revisit both the nature of organizational information and the purpose of organizational work. it follows that as leaders apply systems thinking methodologies and tools to understand the complexities of the organization and its situation, staff members learn to diagnose problems, identify consequences, and make informed responses within a holistic context (somerville, schader, & huston, 2005, pp. 222-223).   evaluative results from this early development work demonstrate that application of these principles changes how co-workers think and what they think about.   more specifically, individuals see the underlying context and assumptions for their decision. this new relational understanding predisposes them to adjust their assumptions and strategies as they learn – in other words, as they change appreciative settings. over time and with practice, individuals’ adoption of systems thinking and thinking tools provides a collective strategy for successfully responding to new information and unique situations. and, finally, sustained conversations rich in relational context provide the substance of a robust organizational learning environment. this dialogue has transformative potential when it activates and extends prior learning (somerville, schader, & huston, 2005, p. 223).   building upon this foundation, university of colorado denver leadership activities focused on exercising and elaborating informed learning capacities as transferable outcomes of “using information to learn” (bruce, 2008) within informed systems. these capabilities were catalyzed during organizational systems design and extended through professional workplace practices, and include:   information and communication technologies to harness technology for information and knowledge retrieval, communication, and management, information sources and information experiences to use information sources (including people) for workplace learning and action-taking, information and knowledge generation processes to develop personal practices for finding and using information for novel situations, information curation and knowledge management to organize and manage data, information, and knowledge for future professional needs, knowledge construction and worldview transformation to build new knowledge through discovery, evaluation, discernment, and application, collegial sharing and knowledge extension to exercise and extend professional practices and knowledge bases which generate workplace insights and informed decisions, and professional wisdom and workplace learning to contribute to collegial learning, using information to learn to better take action to improve (bruce, hughes, & somerville, 2012).   in recognition of the requisite conditions for furthering these essential elements, informed systems models foster boundary-crossing knowledge creation and systems-enabled knowledge management in the workplace.   knowledge processes assume that people can learn to create knowledge on the basis of their concrete experiences, through observing and reflecting on that experience, by forming abstract concepts and generalizations, and by testing the implications of these concepts in new situations. process-based learning activities lead to new concrete experience that initiates a new cycle. it follows that reflective practitioners learn through critical (and self-critical) collaborative inquiry processes that foster individual self-evaluation, collective problem-formulation, and inclusive active inquiry (somerville & mirijamdotter, 2014, p. 206).   learning the way to action-taking thereby advances when participants have increasingly more complex experiences of information exchange, sense making, and knowledge creation, well supported by workplace communication systems and professional practices, further dialogue and reflection and thereby enrich analysis and interpretation of complexities and interdependencies. it naturally follows that learning is a socio-cultural process that cultivates “resilient workers” (lloyd, 2013) as, over time and with practice, co-workers design and enact information-focused and evidence based learning experiences.   learning essentials   within informed systems, the working definition for a learning organization is “a purposeful social interaction system in which collective information experiences are fostered by professional information practices to bring about change in organizational awareness and behavior and thereby further knowledge creation processes” (somerville, 2015, p. 49). within such a ‘whole systems’ framework, organizational leadership must establish and embed sustainable social interactions and enabling workplace systems that can successfully determine: “what information…experiences do we want to facilitate or make possible? what information and learning experiences are vital to further our…professional work?” (bruce, 2013, p. 20).   within this framework, co-workers gain progressive insight into nuanced dimensions of using information to learn through exploring such questions as these: “what constitutes information?…what is being learned? how is understanding/experience of the world changing? what can we do to enrich informed learning experiences?…to introduce new experiences? how would…range of experiences, and awareness of these experiences, be demonstrated?” (bruce, 2012, n.p.).   in addition to consideration of experiential dimensions of workplace information, the informed systems learning approach recognizes that assumptions and conclusions, including norms and values on which collective judgements are based, is the result of previous individual, group, and organizational experiences and history. so explicit reflective practices are designed to promote individual and group awareness of tacit thinking and reasoning. questions for making thinking visible include: “what is the observable data behind that statement? does everyone agree on what the data is?…how did you get from that data to these abstract assumptions? when you said ‘[your inference]’, did you mean ‘[my interpretation of it]?” (senge, 1994, p. 245). such workplace practices encourage individuals and groups to reconsider and reframe thinking, feeling, and responding.   improved understanding occurs because “the knowledge that individuals and organizations have of themselves provides the framework in which they choose alternatives from among a huge, often unaccountable, range of possibilities” (leonard, 1999, n.p.). self-knowledge is also mediated by the culture and language in which discussions take place and the extent to which it is possible to integrate various perspectives. informed systems models, therefore, guide participants in moving beyond surface topics to explore deeper issues through reflective inquiry and collaborative action (somerville, 2015). taking action to improve then produces changes in the ways of perceiving and of becoming newly aware and thereby learning.   enactment of workplace learning requires an enabling environment for information exchange, sense making, and knowledge creation activities that advance information use and learning relationships through socio-cultural processes and practices co-designed by co-workers. collective capacity for discussion and analysis of complexities and interdependencies grows through intentional construction and reconstruction of the learner during interactive relationships and sustainable networks comprised of information, technology, and people. such “construction of learning, of learners and of the environments in which they operate” (hager, 2004, p. 12) evolve to adopt and adapt, create and recreate, contextualize and re-contextualize through wider and wider circles of consultation, cooperation, and collaboration.   viewed through an information experience lens, colleagues collectively expand the information horizons of their work environments through wider and wider circles of consultation, cooperation, and collaboration. while engaging with new information types and communication processes, they establish productive information-sharing relationships which extend beyond team boundaries through critical and creative information use and through generation and sharing of new knowledge necessary to taking purposeful action (somerville & mirjamdotter, 2014). informed systems thereby offers models for (re)learning processes, conducted within enabling systems infrastructure for collaborative evidence based information practice.   collaborative evidence-based information process model   an inquiry-intensive and evidence based informed systems workplace requires significant attention to both process and content. while exploration of peer-reviewed publications oftentimes initiates evidence based practices, authoritative evidence may include a wide range of information sources and professional knowledge. quantitative and qualitative research results, local statistics, open access data, and even accumulated knowledge, opinion, relationships, and instinct may prove useful, depending on local circumstances (koufogiannakis, 2011; 2012; 2013a; 2013b; 2015).   understanding that librarians use evidence to convince, allows an entire organisation to proceed with this as a known entity, and should enable that organisation to look more completely at what the pertinent forms of evidence contribute to the decision, to weigh those pieces of evidence, and to make a decision that is more transparent. the use of evidence for convincing illustrates the complexity of decision-making, particularly within academic libraries, and points to the fact that evidence sources do not stand alone, and are not enough in and of themselves. the eblip process must account for the human interactions, and organisational complexity within which decisions are being made (koufogiannakis, 2013a, p. 172).   a holistic workplace approach therefore requires consideration of elements of organizational design and professional practice essential to collaborative decision-making. this includes fostering a culture of well elaborated organizational processes and knowledge practices (somerville, rogers, mirijamdotter, & partridge, 2007; pan & howard, 2009; 2010; mirjamdotter, 2010; somerville & howard, 2010; somerville & farner, 2012; somerville, 2013; howard & somerville, 2014). evidence based learning processes are also necessarily collegial, conducted within a positive work environment, and enabled by appropriate processes for open discussions for decision-making and action-taking. “knowledge and understanding are thereby learned through active…practice by an individual, within the larger body of practice” (schön, 1983, p. 50), which situates and contextualizes intersubjectively created meaning and changes over time through renegotiation.   the collaborative evidence-based information process model (figure 1) delineates these collaborative processes that advance using information to learn through interactive relationships between the organizational context (elements 1-5), in which individuals and groups create meanings and intentions, which leads to purposeful action (element 6) being taken, with the support of information transfer and knowledge generation systems (element 7).         figure 1 collaborative evidence-based information process model note. adapted from: checkland, p., & holwell, s. (1998). information, systems, and information systems: making sense of the field, p. 106. chichester, england: john wiley & sons. reprinted with permission from john wiley & sons. published in: somerville, m. m. (2015). informed systems: organizational design for learning in action, p. 52. oxford, england: chandos publishing. reprinted with permission from chandos, an imprint of elsevier.     the model recognizes that individuals select information from the workplace (and extended) environment based upon a worldview consisting of existing interests, experience, and values. in other words, unless purposeful intervention occurs, individual perception is highly selective and tends to reinforce existing assumptions. so the first step in designing a sense making process for organizational (re)learning is to initiate conscious reconsideration. raising awareness to stimulate re-thinking requires catalyzing the innate mental processes that are performed tacitly, without individuals making conscious decisions about what is being admitted for consideration, and can eventually widen consideration about what assumptions to make or which data to select.   elements 1 and 2 and the interaction between them involve selectively perceiving reality and making judgments about it through filtering processes that influence what individuals choose to mind and, consequently, use as perception and interpretation filters. these dimensions of information experience are negotiated through sense making processes, including dialogue and reflections (element 3). learning thereby emerges within the context of workplace vision and shared assumptions, including cultural beliefs and associated interpretations and workplace practices, as depicted in element 4.   organized information systems (is) and appropriate information technology (it), together with information and information technology skills (element 7), further inform, enrich, and enable learning. in this way, tacit assumptions represented in a worldview are explicitly reconsidered in the light of emergent new norms and values. judgments evolve and are explicated among employees through dialogue, which then become the bases for forming intentions (element 5) towards particular actions to be carried out (element 6). as is characteristic in systems models, the seven elements are seen as interacting, i.e., element 7 informs and enriches element 4, and it enables and supports element 5, even as it helps to create the perceived world (element 2), including vision, values, and practices (somerville, mirijamdotter, bruce, & farner, 2014).   within this systemic context, thought leaders and knowledge activists offer filters to select what is important from available information models to expand individuals’ ability to understand and use information to learn (nonaka, 1994). these interventions are challenging because tacit knowledge “consists of mental models, beliefs, and perspectives so ingrained that we take them for granted and therefore cannot easily articulate them” (nonaka, 2007, p. 165). however, as “new explicit knowledge is shared throughout an organization, other employees begin to internalize it – that is, they use it to broaden, extend, and reframe their own tacit knowledge” (nonaka, 2007, p. 166) through “purposeful discourse focused on exploring, constructing meaning and validating understanding” (garrison, 2014, p. 147).   informed systems leadership model   the informed systems leadership model identifies essential elements for such organizational leadership, supported by collaborative learning relationships that catalyze systemic outcome and process evaluation cycles. this systems model visually represents purposeful activities necessary to construct and sustain an environment that enables informed learning experiences through informed leadership. the model presents activities that together comprise processes for action and, ideally, for transformation through high-level leadership activities.     figure 2 informed systems leadership model note. originally published in m. m. somerville. (2009). working together: collaborative information practices for organizational learning, chicago, il: the association of college & research libraries/american library association. used with permission from acrl. re-published in: m. m. somerville (2015). informed systems: organizational design for learning in action, p. 55. oxford, england: chandos publishing. used with permission from chandos, an imprint of elsevier.     the activities in purposeful activity models are expressed as verbs in imperative form and are linked in sequence, illustrated by arrows – which denote communication. additionally, when there are arrows in two directions between activities, this illustrates two way communication and interaction. for example, in figure 2, activity 1 represents the initiating activity. however, activities 2, 3, and 4 also contribute to activity 1 and thus must also be carried out to complete the full cycle. additionally, activities can be ordered in layers to connote that they form a grouping. activities outside the layered group, but with an arrow pointing to or from a boundary line, illustrate interaction and communication with all activities inside the layered boundary. for example, activity 5 may lead to insights that promote modifications and improvements in any of the activities in the “core grouping” of activities 1 through 4. finally, feedback processes are illustrated, as are related activities such as monitoring the performance of all activities so that pro-active decisions can be made about changes needed to adapt to changing internal or external conditions, rather than passively reacting to the inevitable.   in this spirit, the model illustrates essential aspects of workplace learning, enabled by design thinking. activity 1 encourages collective exploration and, thereby, fosters robust learning. its centrality in the model reflects the conviction that contemporary organizations cannot be managed in the traditional sense. rather, co-workers should be encouraged to actively engage in information exchange and knowledge creation through using information to learn within enabling co-designed systems.   activity 2 recommends appreciative inquiry and systems thinking to advance understanding of organizational parts, their interrelations, and their synergies. emphasis on big picture and life affirming understanding crosses organizational boundaries and bridges individual silos. in the informed systems leadership model, this concept is reflected in organizational vision, mission, values, and goals, which constitute activity 3.   activity 4 recognizes the critical importance of enabling the expression and extension of thinking through purposefully designed systems that connect people with ideas, oftentimes with technologies. such workplace infrastructure facilitates using information to learn and to share, with the aspiration to generate collective knowledge reflective of improved understanding.   activity 5 acknowledges the significance of engaging in collegial activities to improve professional practices and local situations. therefore, activity 6 represents the importance of ongoing reflection and dialogue to create continuous improvements in using information to learn how to take action to improve situations. activity 7 indicates that sustained movement forward depends upon establishing strong learning relationships inside and outside the organization. organizational leaders are responsible for coordinating and resourcing outcomes of activities 1 through 7, as indicated in activity 8.    in order to nourish learning experiences and support worldview maturation, activity 9 recommends using interactive evaluation to ensure responsive adaption. in this way, activity 9 initiates a feedback cycle, where performance can be monitored to inform modifications that anticipate changes. in addition, activity 10 acknowledges the importance of high-level alignment of mission and vision with human and fiscal resources, negotiated within learning relationships exercised through action-oriented inquiry and inclusive decision-making (somerville, mirijamdotter, bruce, & farner, 2014).   in combination, informed systems leadership and collaboration models design enabling systems and informing activities that cross professional and organizational boundaries through a strong “people oriented” approach, customizable to local circumstances. it recognizes that workplace learning originates from interactions and relationships among organizational members, which enable investigation and negotiation of diverse interests, judgments, and decisions. reflection and dialogue processes promote learning through critical (and self-critical) inquiry experiences that foster individual self-evaluation, collective problem-formulation, and nuanced professional development (somerville & mirijamdotter, 2014). informed systems thereby promotes transformation in organizational awareness and workplace behavior through intentional design that nurtures engagement among individuals and with information.   concluding reflections   contemporary organizations must develop workplace environments that enable nimble decision-making and action-taking. in response, at the macro level, informed systems models guide how and why organizations build knowledge bases. at the micro level, design methodologies and learning theories guide how and why co-workers use information to learn to co-create enabling systems and evidence practices. along the way, attention moves from transaction based activities to organizational transformation outcomes enacted through intuiting, interpreting, integrating, and institutionalizing knowledge together.   in response, informed systems appreciatively explores the intersection of information, technology, and learning experiences in organizational knowledge creation. thought leaders create and refine information activities that produce learning experiences and, over time and with experience, advance integration of evidence based practice into workplace culture, as detailed in the informed systems leadership model. within this enabling framework, a companion collaboration evidence-based information process model guides collective decision-making and action-taking to ensure perpetual learning and continuous improvement. as detailed in this conceptual paper, these models illustrate the efficacy of integrating the work of three theorists, bruce, checkland, and nonaka, into a hybrid theory with an associated methodology for workplace transformation.   informed systems results since 2003 demonstrate that change, and ultimately transformation, occurs through using information to learn. this depends on learning-centered and information-focused workplace relationships fortified by professional practices that amplify evidence based collaborative processes for decision-making and action-taking. within this organizational environment, colleagues learn to initiate inquiries and to design experiences that are information-centered, evidence-grounded, action-oriented, and learning-focused. mental models and collective conceptions change. co-workers reinvent roles, responsibilities, processes, and relationships, as they co-design potential futures.   references   bruce, c. (1997). the seven faces of information literacy. blackwood, south australia: auslib press.   bruce, c. s. (1998). the phenomenon of information literacy. higher education research & development, 17(1), 25–43.   bruce, c. s. (1999). workplace experiences of information literacy. international journal of information management, 19(1), 33–47.   bruce, c. s. (2008). informed learning. chicago, il: association of college & research libraries/american library association.   bruce, c. s. (2012). education, research, and information services workshop presentation, auraria library, denver, colorado. (unpublished.)   bruce, c. s. (2013). keynote address. information literacy research and practice, an experiential perspective. in s. kurbanoglu, e. grassian, d. mizrachi, r. catts, & s. spiranec (eds.), communications in computer and information science series: 397. worldwide commonalities and challenges in information literacy research and practices: european conference on information literacy. revised selected papers, (pp. 11–30). heidelberg, ny: springer.   bruce, c. s. (2015). information literacy: understanding people’s information and learning experiences. in western balkan information literacy: proceedings of the 12th international scientific conference (pp. 11-16). bihac, bosnia: cantonal and university library bihac.   bruce, c., partridge, h., hughes, h., davis, k., & stoodley, i. (eds.). (2014). information experience: approaches to theory and practice (pp. 203-220). (library and information science, vol. 9). bingley, england: emerald.   bruce, c. s., hughes, h., & somerville, m. m.  (2012). supporting informed learners in the 21st century. library trends, 60(3), 522-545.   chatzipanagiotou, n. (2015). toward an integrated approach to information management: a literature review. 5th international conference on integrated information, mykonos, greece. retrieved from http://www.icininfo.net/index.php/program/submissions   checkland, p. (2011). autobiographical retrospections: learning your way to ‘action to improve’ the development of soft systems thinking and soft systems methodology. international journal of general systems, 40(5), 487–512.   checkland, p., & holwell, s. (1998). information, systems, and information systems: making sense of the field. chichester, england: john wiley & sons ltd.   checkland, p., & poulter, p. (2010). soft systems methodology. in m. reynolds, & s. holwell (eds.), systems approaches to managing change: a practical guide (pp. 191–242). london, england: the open university, published in association with springer-verlag london limited.   checkland, p., & scholes, j. (1990).  soft systems methodology in action. chichester, england: john wiley & sons ltd.   coaldrake, p., & stedman, l. (2013). raising the stakes: gambling with the future of universities. st. lucia, queensland, australia: university of queensland press.   crow, m. m., & dabars, w. b. (2015). designing the new american university. baltimore, md: johns hopkins university press.   garrison, d. r. (2014). community of inquiry. in d. coghlan & m. brydon-miller (eds.), the sage encyclopedia of action research. (vol. 1, pp. 147-150). los angeles, ca: sage reference.   hager, p. (2004). conceptions of learning and understanding learning at work. studies in continuing education, 26(1), 3–177.   hemmasi, h., lefebvre, m., lippincott, j., murray-rust, c., & somerville, m. m. (2015). organizational and service models. designing libraries for the 21st century iv, north carolina state university, raleigh, north carolina, usa.   howard, z., & somerville, m. m. (2014). a comparative study of two design charrettes: implications for codesign and participatory action research. co-design: international journal of cocreation in design and the arts, 10(1), 46-62.   hughes, h.  (2014). researching information experience: methodological snapshots. in c. bruce, h. partridge, h. hughes, k. davis, & i. stoodley (eds.), information experience: approaches to theory and practice. (pp. 33-50). bingley, england: emerald.   koufogiannakis, d. (2011). considering the place of practice-based evidence within evidence based library and information practice (eblip). library and information research, 35(111), 41-58.   koufogiannakis, d. (2012). academic librarians’ conception and use of evidence sources in practice. evidence based library and information practice, 7(4), 5-24. retrieved from https://ejournals.library.ualberta.ca/index.php/eblip   koufogiannakis, d. (2013a). how academic librarians use evidence in their decision making: reconsidering the evidence based practice model (doctoral dissertation). retrieved from http://cadair.aber.ac.uk/dspace/   koufogiannakis, d. (2013b). academic librarians use evidence for convincing: a qualitative study. sage open, 3(april-june), 1-12. http://dx.doi.org/10.1177/2158244013490708   koufogiannakis, d. (2015). determinants of evidence use in academic librarian decision making. college & research libraries, 76(1), 100-114. http://dx.doi.org/10.5860/crl.76.1.100   leonard, a. (1999). viable system model: consideration of knowledge management. journal of knowledge management practice. retrieved from http://www.tlainc.com/jkmp.htm   lippincott, j. (2014). designing library spaces. designing libraries for the 21st century iii, university of calgary, alberta, canada.   lloyd, a. (2013). building information resilient workers: the critical ground of workplace information literacy. what have we learnt? in s. kurbanoglu, e. grassian, d. mizrachi, r. catts, & s. spiranec (eds.), communications in computer and information science series: vol. 397. worldwide commonalities and challenges in information literacy research and practices: european conference on information literacy. revised selected papers. (pp. 219–228). heidelberg, ny: springer.   materska, k. (2013). is information literacy enough for a knowledge worker? in s. kurbanoglu, e. grassian, d. mizrachi, r. catts, & s. spiranec (eds.), communications in computer and information science series: vol. 397. worldwide commonalities and challenges in information literacy research and practices: european conference on information literacy. revised selected papers. (pp. 229–235). heidelberg, ny: springer.   mirijamdotter, a. (2010). toward collaborative evidence based information practices: organisation and leadership essentials. evidence based library and information practice, 5(1), 17-25. retrieved from http://ejournals.library.ualberta.ca/index.php/eblip     mirijamdotter, a., & somerville, m. m. (2009). collaborative design: an ssm-enabled organizational learning approach. international journal of information technologies and systems approach, 2(1), 48-69.   nonaka, i. (1994). a dynamic theory of organizational knowledge creation. organization science, 5(1), 14-37.   nonaka, i. (2007). the knowledge-creating company. harvard business review, 85, 162-171.   nonaka, i., konno, n., & toyama, r. (2000). seci, ba and leadership: a unified model of dynamic knowledge creation. long range planning, 33, 5-34.   nonaka, i., & takeuchi, h. (1995). the knowledge-creating company: how japanese companies create the dynamics of innovation. new york, ny: oxford university press.   nonaka, i., & toyama, r. (2007). why do firms differ? the theory of the knowledge-creating firm. in k. ichijo & i. nonaka (eds.), knowledge creation and management: new challenges for managers. oxford, england: oxford university press.   pan, d., & howard, z. (2009). reorganizing a technical services division using collaborative evidence-based information practice at auraria library. evidence based library and information practice, 4(4), 88-94. retrieved from http://ejournals.library.ualberta.ca/index.php/eblip   pan, d., & howard, z. (2010). distributing leadership and cultivating dialogue with collaborative ebip. library management, 31(7), 494-504.   schön, d. (1983). the reflective practitioner—how professionals think in action. new york, ny: basic books.   senge, p. (1994). the fifth discipline fieldbook: strategies and tools for building a learning organization. new york, ny: doubleday/currency.   somerville, m. m. (2009). working together: collaborative information practices for organizational learning. chicago, il: association of college & research libraries/american library association.   somerville, m. m. (2013). digital age discoverability: a collaborative organizational approach. serials review, 39(4), 234-239.   somerville, m. m. (2015). informed systems: organizational design for learning in action. oxford, england: chandos publishing.   somerville, m. m., & conrad, l. y. (2014). collaborative improvements in the discoverability of scholarly content: accomplishments, aspirations, and opportunities. a sage white paper. los angeles, ca: sage publications ltd. retrieved from http://www.sagepub.com/repository/binaries/pdf/improvementsindiscoverability.pdf   somerville, m. m., & conrad, l. y. (2013). discoverability challenges and collaboration opportunities within the scholarly communications ecosystem: a sage white paper update. collaborative librarianship, 5(1), 29–41. retrieved from http://www.collaborativelibrarianship.org/index.php/jocl   somerville, m. m., & farner, m. (2012). appreciative inquiry:  a transformative approach for initiating shared leadership and organizational learning. revista de cercetare si interventie sociala [review of research and social intervention], 38(september), 7-24. retrieved from http://www.rcis.ro/   somerville, m. m., & howard, z. (2010). information in context: co-designing workplace structures and systems for organizational learning. information research, 15(4). retrieved from http://www.informationr.net/ir/    somerville, m. m., howard, z., & mirijamdotter, a. (2009). workplace information literacy: cultivation strategies for working smarter in 21st century libraries. in d. m. mueller (ed.), pushing the edge: explore, engage, extend proceedings of the 14th association of college & research libraries national conference, seattle, washington (pp. 119-126). chicago: association of college & research libraries.   somerville, m. m., & mirijamdotter, a. (2014). information experiences in the workplace: foundations for an informed systems approach. in c. bruce, h. partridge, h. hughes, k. davis, & i. stoodley (eds.), information experience: approaches to theory and practice (pp. 203-220). (library and information science, vol. 9) bingley, england: emerald.   somerville, m. m., mirijamdotter, a., bruce, c. s., & farner, m. (2014). informed systems approach: new directions for organizational learning. in m. jaabæk & h. gjengstø (eds.), international conference on organizational learning, knowledge, and capabilities (olkc 2014: circuits of knowledge). oslo, norway: bi norwegian business school. retrieved from http://www.diva-portal.org/smash/get/diva2:714317/fulltext02.pdf   somerville, m. m., rogers, e., mirijamdotter, a., & partridge, h. (2007). collaborative evidence-based information practice: the cal poly digital learning initiative. in e. connor (ed.), evidence-based librarianship: case studies and active learning exercises (pp. 141-161). oxford, england: chandos publishing.    somerville, m. m., schader, b., & huston, m. e. (2005). rethinking what we do and how we do it: systems thinking strategies for library leadership. australian academic and research libraries, 36(4), 214–227.   somerville, m. m., schader, b. j., & sack, j. r. (2012). improving the discoverability of scholarly content in the twenty-first century: collaboration opportunities for librarians, publishers, and vendors. a sage white paper. los angeles, ca: sage publications ltd. retrieved from http://www.sagepub.com/repository/binaries/librarian/discoverabilitywhitepaper/   townsend, a. (2014). collaborative action research. in d. coghlan & mary brydon-miller (eds.), the sage encyclopedia of action research (vol. 1, pp. 116-119). thousand oaks, ca: sage publications ltd.   von krogh, g., ichijo, k., & nonaka, i. (2000). enabling knowledge creation: how to unlock the mystery of tacit knowledge and release the power of innovation. london: oxford university press.   wawrzaszek, s.v., & wedaman, d.g. (2008). the academic library in a 2.0 world. research bulletin, issue 19, boulder, co: educause center for applied research. evidence summary   bangladesh public libraries' response to covid-19 pandemic   a review of: begum, d., roknuzzaman, m., & shobhanee, m. e. (2022). public libraries’ responses to a global pandemic: bangladesh perspectives. ifla journal, 48(1), 174–188. https://doi.org/10.1177/03400352211041138     reviewed by: david dettman associate professor and library instruction program coordinator university of wisconsin-stevens point stevens point, wisconsin, united states of america email: ddettman@uwsp.edu   received: 31 aug. 2023                                                             accepted: 17 oct. 2023      2023 dettman. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30438     abstract   objective – examine quantitative and qualitative data to determine the level of preparedness and capacity to provide key resources during the recent covid-19 global pandemic with the goal of recommending measures to better cope with future similar situations.   design – online questionnaire and semi-structured interviews. descriptive statistics were used to analyze quantitative data, followed by a narrative analysis of the qualitative data.   setting – public libraries in bangladesh.   subjects – heads of the libraries such as directors, librarians, or librarians-in-charge.   methods – a survey was distributed to all 71 government public libraries in bangladesh administered and governed by the department of public libraries under the ministry of cultural affairs. the survey elicited 50 valid responses, resulting in a response rate of 70%. a structured form questionnaire was prepared using google forms and included 12 closed-ended questions with multiple options and one open-ended question to collect librarian views on handling services and key functions post pandemic. telephone interviews were conducted with eight librarians from the district public libraries.   main results – the data revealed that all the libraries reported having some level of preparedness for pandemics and disasters. however, few libraries reported high levels of confidence in this regard. for example, 82% responded that they either had a “low” or a “minimum” preparedness for dealing with a pandemic. the survey revealed that none of the responding libraries were fully open to provide user services during the covid-19 pandemic while 100% were open for administrative functions. additionally, 72% reported having some sections open but others closed, while 22% indicated that the library only provided specified services, most of these in an online environment. only 12% indicated that they provided access to mobile library services.   the libraries responses to crisis management included creating public awareness through social media campaigns (50%), supporting other government bodies to mitigate the pandemic (46%), providing counseling services for particular users (40%), promoting authoritative and authentic national and global strategies on the pandemic for the public (34%), providing pandemic-related information via phone or email on demand (32%), and distributing relief to affected people (30%). other activities included organizing online social and cultural programs to reduce public anxiety with special reference to vulnerable groups (24%), online staff training (22%), monitoring and supervising government programs (20%), and providing research data on the pandemic (10%).   the response to needs regarding redesigning library programs and services for the post-pandemic world revealed that 90% of libraries identified a need for a dedicated online service platform, 80% saw the need to create policies related to health and safety, while 72% indicated a need for more emphasis on digital content. the responses indicated 56% were in favor of introducing innovative online services. conclusion – in response to the results of their study, the authors make several recommendations regarding helping the libraries of bangladesh to cope with consequences of future pandemics similar in scale to covid-19. the first recommendation is to move away from “providing services in a traditional manner,” while developing dedicated online platforms running on high-bandwidth internet connections (begum et al., 2022, p. 183). these include introducing a secure cloud-based library automation system to provide a robust and stable information gateway. the development of digital resources should also be a priority. this includes both the digitization of library resources and more widespread access to an increasing number of subscription databases.   the data indicated a strong desire among librarians to introduce innovative library services which the authors identify as digitized content delivery, remote access to online resources, online chat reference, current awareness services, online programming mirroring face-to-face programming, home delivery service for books, strategic disaster policy development, online information literacy programs, online career counseling, and leadership along with online training, workshops, and seminars. the authors conclude that:   libraries face challenges, such as transitioning to virtual services, breaking long-established service models, funding for digital resources and its impacts on library operations in the future, enforcing social distancing measures, creating social awareness, and the lack of formal pandemic plans and policies …. (begum et al., 2022, p. 179)   commentary   in the article introduction the authors illustrate the catastrophic impact that covid-19 had on poverty-stricken citizens. according to a brac economic report, this segment of the population experienced an 80% drop in daily income. public libraries both in the united states and globally are important resources for career training and intellectual self-improvement during the best of economic times, and they become invaluable in times of disasters like the global pandemic. the authors rightly echo this sentiment stating that “as community-based organizations, public libraries have the potential to make contributions during any natural or man-made crisis” (begum et al., 2022, p. 185). the authors suggest moving to an online model that will allow for the electronic dissemination of this kind of information. having said that, the authors do not indicate to what degree the poor have access to technology in the first place. if the situation mirrors that in the united states, recommendations related to shrinking the digital divide precipitated by economic stratification would be most welcome. if this is the case, one recommendation might be to dedicate resources for ensuring there are access points (e.g., local “hot spots”) in these communities for the technology “have-nots” in times of disaster (and more generally) to share the kinds of information that can help those struggling with poverty learn about and apply for available social services and programs designed to provide aid, discover opportunities for employment, and receive health and hygiene information to help slow the spread of a virus like covid-19. after assessing the quality of the study using the cat: a generic critical appraisal tool created by perryman and rathbun-grubb (2014), i found the quality to be high. the two main authors are both professors in the department of information science and library management, university of dhaka, bangladesh, and the third is listed as an undergraduate student of information science. both primary authors have published extensively on public libraries and other issues related to library and information science. an extensive literature review is included to help give context to the study, and the results of the survey and interview are clearly communicated both textually and visually. the methods employed are also clear, and the conclusion rests firmly on the analysis of the collected data. what is lacking is any mention of the limitations of the study and the direction further research might take. one thing to consider would be an environmental scan of the communities dominated by those in poverty to determine the best means for providing access to technological devices for the purpose of connecting to library resources that have been moved online.   references   begum, d., roknuzzaman, m., & shobhanee, m. e. (2022). public libraries’ responses to a global pandemic: bangladesh perspectives. ifla journal, 48(1), 174–188. https://doi.org/10.1177/03400352211041138   perryman, c., & rathbun-grubb, s. (2014). the cat: a generic critical appraisal tool. http://www.jotform.us/cp1757/thecat   microsoft word es 1459_loy evidence based library and information practice 2008, 3:2 58 evidence based library and information practice evidence summary better collaborative working is likely to increase uptake of library resources in a university setting a review of: hightower, barbara, carolyn rawl, and michelle schutt. “collaborations for delivering the library to students through webct.” reference services review 35.4 (2007): 541-51. reviewed by: john loy learning resources manager, avon & wiltshire mental health partnership nhs trust callington road hospital brislington, bristol, united kingdom e-mail: john.loy@awp.nhs.uk received : 24 february 2008 accepted : 13 april 2008 © 2008 loy. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objectives – to ascertain the extent to which university faculty members are integrating library resources within the webct course management system / managed learning environment. also, to identify the reasons why faculty members are not integrating library resources within webct, and to explore their willingness to do so in future. design – case study with survey questionnaire and selected interview follow up. setting – urban campus of auburn university in montgomery, alabama, united states of america. subjects – one hundred and two members of faculty received a questionnaire. five follow-up interviews were conducted. methods – one hundred and two members of faculty with webct-supported courses received a questionnaire. they were asked to indicate if they linked to library resources, and if so which specific ones (e.g. library homepage, online catalogue, etc.). if they did not link to library resources they were asked to indicate the reason why not, either by selecting one or more of five pre-selected reasons, or by detailing their own. follow-up interviews were conducted with five respondents, two of whom (education and nursing) linked to library resources, and three of whom (from business, education and science) did not. evidence based library and information practice 2008, 3:2 59 server log reports were also examined to identify how many students were entering library resources from webct. main results – the survey response rate was 28% (29 individuals), and of these nursing and sciences were the highest users with 7 individuals in each faculty. of the 29 respondents only 7 (24%) currently linked to library resources. nursing were the biggest group with 10 links (38% of the total), science with 6 (23%) and 5 each from liberal arts and education (19%). the resources chosen to link to were • article databases (6) • library homepage (5) • ask a librarian e-mail service(3) • net library e-book collection (3) six other library resources scored lower and two were not linked to at all. while few respondents actually provided links at present, 77% of those did express an interest in doing so in the future. almost a third of respondents asked to be contacted to learn more about how to provide links and to learn more about library services. only 57 incidents of students linking through to library resources from webct occurred in the study period. conclusion – numbers of faculty currently providing links from webct to the library is very small, and few students find their way to library resources via this route. however, interest generated by the survey and follow up indicates that this may be a valuable means of promoting library resources. commentary this is a curious hybrid study. from the outset the main purpose of the article is identified as three-fold: to present survey results, discuss collaborative efforts, and consider methods to get faculty on board. this three-pronged approach does unfortunately mean that the piece emerges as something of a curate’s egg: good in parts but lacking overall. due to the case-study elements there does seem to be a great deal of scene setting, but then not a lot of meat on the bones for an eleven page article. while the research element is only one part of the piece, the authors don’t help their case by the way in which they present their results. we never do find out how the survey was distributed e-mail, paper, or some other way? likewise selection criteria for the five respondents chosen for follow-up interviews is never explained, and we are not given any detail whatsoever about the format of the interviews. these gaps in describing the methodology are frustrating, and detract from the value of the article. qualitative data can be wonderfully rich and informative, but here the reporting of the data from the interviews is cursory at best, with disappointingly little space devoted to it. i expected the interviews to follow-up such areas as why the nursing faulty were the keenest to signpost library resources. while there is some hypothesis regarding nursing students’ lifestyle and use of webct earlier in the article, we learn next to nothing from the interviews about the nursing faculty’s relationship with the library. the article is also at times poorly set out on the page. the survey is helpfully reproduced in full, but it leads immediately into a figure detailing results, and it is easy to be confused on a first reading. parts of the paper are devoted to describing events which happened after the study had finished, and again it can be difficult to distinguish results of the research led by the authors from observations following the evidence based library and information practice 2008, 3:2 60 study. we learn that the numbers of students accessing library resources from webct rises dramatically when the library is involved in developing a module for the nursing faculty. it is not made clear whether that is actively intended as part of this study, or coincidentally happened soon after. this somewhat muddled reporting does the authors a disservice. one really useful message which does come through loud and clear is the reminder that libraries cannot work in isolation and that “librarians should be more proactive in promoting our services.” (548) it is worrying, and perhaps at the same time strangely reassuring, to read of the familiarity of the challenges faced by our colleagues. it is good to hear of the ways in which they are striving to improve the services they provide, and the authors clearly remind us of the importance of collaborative working with the services libraries are there to support. i did struggle at times with the authors’ scattergun approach and found myself wondering quite what readers might take away from this. anyone with a few years’ experience behind them will find themselves facing something of an anticlimax and have an overwhelming feeling of, “well, no real surprises there.” while this article does certainly contribute to the evidence base, it is perhaps best viewed as an interesting case study rather than a robust piece of research. evidence based library and information practice evidence based library and information practice 2011, 6.2 4 evidence based library and information practice editorial responsibilities © 2011. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. editor‐in‐chief: denise koufogiannakis associate editor (articles): alison brettle associate editor (evidence summaries): lorie kloda associate editor (classics): jonathan eldredge associate editor (features, commentaries, ebl101, news): denise koufogiannakis production editor: katrine mallan editorial interns: andrea baer, michelle dunaway copyeditors: priscilla stephenson (lead copyeditor), georgianne bordner, marcy brown, molly des jardin, richard hayman, dale storie, mary virginia taylor indexing support: pam morgan writing assistance: hope leman, tracy powell iwaskow, carol waseleski / evidence based library and information practice research article   an exploratory study of accomplished librarian-researchers   marie r. kennedy serials & electronic resources librarian loyola marymount university los angeles, california, united states of america email: marie.kennedy@lmu.edu   kristine r. brancolini dean of the library loyola marymount university los angeles, california, united states of america email: brancoli@lmu.edu   david p. kennedy senior behavioral and social scientist rand corporation santa monica, california, united states of america email: davidk@rand.org   received: 16 sept. 2019                                                                  accepted: 3 jan. 2020      2020 kennedy, brancolini, and kennedy. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29655     abstract   objective – this work explores potential factors that may contribute to a librarian becoming a highly productive researcher. an understanding of the factors can provide evidence based guidance to those at the beginning of their research careers in designing their own trajectories and to library administrators who seek to create work conditions that contribute to librarian research productivity. the current study is the first to explore the factors from the perspective of the profession’s most accomplished librarian-researchers.   methods – this exploratory and descriptive study recruited 78 academic librarians identified as highly productive researchers; 46 librarians participated in a survey about their professional training and research environments, research networks, and beliefs about the research process. respondents supplied a recent cv which was coded to produce a research output score for the past 10 years. in addition to fixed-response questions, there were five open-ended questions about possible success factors. all data were analyzed with descriptive statistics and tests of significance correlations.   results – accomplished librarian-researchers have professional training backgrounds and research environments that vary widely. none is statistically associated with research output. those with densely connected networks of research colleagues who both know each other and do research together is significantly related to research output. a large group of those identified in the research networks are “both friend and colleague” and offer each other reciprocal support. in open-ended questions, respondents mentioned factors that equally span the three categories of research success: individual attributes, peers and community, and institutional structures.   conclusion – the authors found that that there are many paths to becoming an accomplished librarian-researcher and numerous factors are conducive to achieving this distinction. a positive research environment includes high institutional expectations; a variety of institutional supports for research; and extrinsic rewards, such as salary increases, tenure, promotion, and opportunities for advancement. the authors further conclude that a librarian’s research network may be an important factor in becoming an accomplished librarian-researcher. this finding is supported by both the research network analysis and responses to open-ended questions in which collaboration was a frequent theme.     introduction the authors of this article investigated the professional training and environment, research networks, and attitudes about research of accomplished librarian-researchers. the authors consulted a group of librarian-researchers who represent the high end of research productivity to explore potential contributors. this is the first study that examines these possible contributors for the population of the most productive librarian-researchers. in academia, the proxy for productivity is publication activity, so the authors identified academic librarian-researchers who have written the highest number of library and information science (lis) publications over the past 10 years. the authors analyzed the resulting data to learn if there are commonalities among these librarian-researchers.   problem statement librarians at the outset of their research careers can benefit from understanding factors that contributed to the productivity of accomplished librarian-researchers, such as professional training and research environment, social supports in the research network, and beliefs about and the practice of the research process. insight into these factors can help them to imagine their own career trajectories. to that end, this study is guided by two research questions: 1.       what are the factors that accomplished librarian-researchers identify as having contributed to their becoming a productive researcher? 2.       what are the compositional commonalities of the research networks of these librarian-researchers with a high level of research output? literature review in the lis literature there has been a recent focus on research productivity among librarian-researchers, including the factors that may be related to the successful completion of research projects. there are three areas of concern in this study related to factors that may align with productivity of librarian-researchers in an academic setting: professional training and research environment; research network; and beliefs about and the practice of the research process. this section addresses literature in those areas. professional training and research environment many academic librarians are actively conducting and disseminating the results of their original research. librarians author the majority of articles in lis journals (chang, 2016), including the profession’s most highly-regarded journals (galbraith, smart, smith, & reed, 2014). for example, they account for the majority of the authors in the journal of academic librarianship (luo & mckinney, 2015). despite this success, an often-cited barrier to librarian research productivity is the lack of research training in the lis master’s curriculum. lili luo found in her 2010 review of the degree requirements for the 49 american library association-accredited lis programs that 61% list research methods as a required course (luo, 2011). however, there is not a standard research methods curriculum at the master’s degree level, so the training offered across the programs varies in content and depth. in addition to lack of research training, librarians cite other barriers, including lack of research confidence, lack of a research community, lack of institutional support, and lack of time (kennedy & brancolini, 2018). despite these barriers, researchers have found that some academic librarians are intrinsically motivated to move forward with a research agenda, noting reasons such as personal satisfaction, intellectual curiosity, and the desire to contribute to the profession (fennewald, 2008; hollister, 2016; perkins & slowik, 2013). related to those intrinsic motivators, watson-boone (2000) noted that academic librarian authors’ efforts “improve their own practice and further develop their own levels of expertise” (p. 91). the employment environment may also contribute to the productivity of librarian-researchers. in a study of the relationship between, faculty status and research productivity, galbraith et al. (2014) examined the authorship of articles published in the top 23 lis journals. they found that 42% of the articles were written by academic librarians. of those, 65% worked at libraries with faculty status and tenure. hoffmann, berg, and koufogiannakis (2014) identified 42 empirical research articles on productivity for librarian and non-librarian practitioner-researchers – such as doctors, nurses, and social workers. based on a definition of research productivity as “completion of research activities and subsequent dissemination of research findings” (p. 15), the authors conducted a content analysis of these articles and identified 16 factors that they believe contribute to research productivity, which cluster into 3 broad categories: individual attributes, peers and community, and institutional structures and supports. they used these categories to develop a survey administered to 1,653 librarians who worked at the 75 canadian research knowledge network and were likely conducting research as part of their job responsibilities (hoffmann, berg, & koufogiannakis, 2017). in the study, hoffmann et al. calculated a research productivity score and looked for statistical correlations with specific success factors. they did not identify a single factor within the three categories as the main statistical contributor to research productivity, leading them to conclude that “an environment that embraces all three areas, by encouraging individual attributes, foster peer and community interaction, and providing institutional supports, will be likely to promote research productivity among librarians” (p. 116). research networks research “peers and community” networks are important contributors to research productivity. a successful librarian-researcher has built his or her own personal network of social contacts over the course of a career. these networks can be measured using the method of social network analysis which is designed to describe the relationships between those social contacts (borgatti, everett, & johnson, 2013; wasserman & faust, 1994). social network analysis provides a conceptual structure and measures for examining the relationships in a research network. the authors identified only one publication that uses social network analysis to measure the research networks of librarian-researchers. that singular study investigated the personal research networks of novice librarian-researchers (kennedy, kennedy, & brancolini, 2017). the approach to measurement used in that analysis is the personal or egocentric network approach, since the focus of study was to understand the social ties surrounding an independently sampled set of focal individuals (mccarty, 2002). in that research the authors relied on those focal individuals (referred to as egos) to report on the relationships with those they identified as being part of their research network (referred to as alters) and their perceptions of the relationships between all possible pairs of those in the network, including themselves (krackhardt, 1987). network composition (the types of people in the network) was measured by calculating the proportion of network ties with a certain characteristic, such as the proportion of network members who offer research assistance (as described in crossley et al., 2015). network structure (interconnections among network members) was assessed by calculating the ratio measure of the number of connections among network members compared to the total possible number of ties (called density) (as described in crossley et al., 2015; mccarty, 2005; wasserman & faust, 1994). the authors did not find any examples in the literature of investigations of the networks of accomplished librarian-researchers.  beliefs about and the practice of the research process as the literature in the field of library and information science (lis) has now well described the barriers to conducting research, researchers have turned their attention to the influences on research success. the recent literature has focused on supportive structures from an administrator viewpoint (berg, jacobs, & cornwall, 2013; perkins & slowik, 2013; sassen & wahl, 2014; smigielski, laning, & daniels, 2014) as well as from a practitioner-researcher perspective (fiawotoafor, dadzie, & adams, 2019; meadows, berg, hoffmann, gardiner, & torabi, 2013; vilz & poremski, 2015). methods the authors developed a survey to elicit information about the respondents’ professional training, their research networks, their beliefs about research, and their research practices. the survey included questions about research beliefs and practice adapted from hoffmann et al. (2017). the survey also included five open-ended questions about factors contributing to research productivity designed to elicit a more comprehensive understanding of the factors directly from the accomplished librarian-researchers. the authors also measured productivity of participants directly by collecting and coding full and recent cvs of each participant rather than self-reports, which can be unreliable (hoffmann et al., 2017). study population a requirement of this research is identifying the most productive librarian-researchers. there is no database kept in the united states at the national level of academics classified by their field of specialty, as in italy (abramo, d’angelo, & di costa, 2019), so the authors needed to create a list of those accomplished librarian-researchers for this work. the list was formed from two sources of data. the first source of data was drawn from clarivate’s web of science (clarivate, 2018) and focused on librarians working at public and private university libraries in the united states of america that are members of the association of research libraries (arl) (2019), which are research-intensive institutions. using the web of science social science citation index, the authors conducted advanced searches for each of the 99 arl libraries (using the organization enhanced field tag), combined with the topic of “library” and the web of science category, “information science library science,” including all document types, and published from the time span of 2007 to 2018. from each library the authors of those publications were ranked by number of items published. those with five or more items published were highlighted. the researchers conducted an internet search for each of those authors, to verify if the person was a practicing librarian; if so, they were included in the set, resulting in 39 librarians. the authors supplemented this list with a second source of data: a list of researchers not necessarily affiliated with arl libraries provided by the first author of an article about worldwide contributors to the literature of library and information science (walters & wilder, 2015). this study identified the top librarian authors in the field (based on a harmonic weight of authors publishing in 31 lis journals). from this data the authors selected the top 50 from the united states and merged them with their web of science set. there were 10 names included on both lists, producing a total of 79 unique librarians, 60 from arl member libraries and 19 from other academic libraries.  recruitment and survey dissemination after receiving approval of the protocol from the institutional review board, the authors sent an initial email with the request for participation with a link to a personalized survey. recruitment emails were successfully sent to 74 of the 78 librarian-researchers. three were not able to be contacted because their emails were returned as undeliverable and one was unintentionally omitted from recruitment. one follow-up email was sent to those who did not respond to the initial request. the recruitment email may be found as appendix a. a $100 usd gift card was offered to each respondent who completed the survey and supplied their cv. survey design and measures the authors designed the survey around three areas of concern related to research output: professional training research environment; the research network of the respondent; and beliefs about and the practice of the research process. the survey was constructed using egoweb 2.0 (2015), the freely-available open source tool for network data collection. the survey was administered using a personalized url. professional training and research environment respondents were asked a series of questions that assessed their graduate-level educational background, including the year in which their lis degree was completed and if they wrote a thesis while completing their lis degree or another master’s degree (yes/no). respondents were also asked if they believed their lis degree prepared them to read and understand research-based literature (yes/no) and if they believe it prepared them to conduct original research (yes/no). to assess experience with research method training, respondents were presented with a list of educational activities about research methods and asked to mark all in which they have ever participated. the list includes: formal master’s degree lis course; formal master’s degree non-lis course; formal doctoral lis course; formal doctoral non-lis course; continuing education program; staff development program; self-education, and; none of these. to assess respondents’ early and current research support, respondents were presented with a list of support options and asked to identify which were available them and which they had used. the options include: release time; short-term pre-tenure research leave; sabbaticals for librarians; travel funds (full); travel funds (partial); research grants; formal mentorship; informal mentorship; research design consultant; workshops. to measure mentoring experiences, respondents were asked if they had ever participated in any formal or informal mentorship programs. respondents were also asked if they had achieved tenure at a previous institution and/or at their current institution and their rank. the respondents were then asked one question to assess if they conducted their early research either (1) on their own, with partners who were (2) more, (3) less or (4) equally experienced, or with research teams that were composed of (5) mostly novice researchers or (6) mixed novice and experienced researchers. they were also asked this same question about their current research coupled with an open-ended question asking them to describe their current research. finally, respondents were asked two open-ended questions, one prompting the participant to note anything else about their professional training over the last 10 years that they believe may have contributed to their productivity, and the other prompting to note anything about their research environment over the last 10 years. research network after answering questions about their own research experiences, respondents were asked about their research networks using standard ego-centered network data collection procedures (crossley et al., 2015; mccarty et al., 2019). the first step, network elicitation, prompted respondents to name the people (up to 40) with whom the respondents have research interactions (their “alters”). next, “name interpreter” questions were asked about each alter to produce measures of network composition. questions included how often the respondent interacted with alters over the past 30 days, and how often they discussed research during those interactions. respondents also classified each alter as a personal friend, professional colleague, or both friend and colleague and reported on their advice/help relationship with each alter (the respondent usually asks for advice/help, usually offers advice/help, or the research interactions include asking for and giving help in equal amounts). respondents reported if alters were local to the respondents’ workplace, and their mode of usual communication with each alter (in person, online, phone, etc.). finally, respondents were asked if they had a formal mentoring relationship with each alter and if they mentored the alter or the alter mentored the respondent. after each name interpreter question, respondents were asked one question to measure network structure. respondents were asked to evaluate the relationship between each unique pair of alters: if they know each other and, if yes, if they had research interactions. the section of the survey ends with an open-ended question to discover if there is anything else about the people in the current research network that may have contributed to the productivity of the respondent. beliefs about and the practice of the research process the last section asked respondents to evaluate twenty-eight statements regarding beliefs about the research process with a yes or no response to report whether it generally applies to them or not. the statements are a subset from the survey administered by hoffmann et al. (2017) to academic librarians employed by canadian research libraries. to facilitate comparisons with this previous study, the statements used as much of the verbatim language as the original question as possible. the final question prompted the respondent to think back on their entire career and list the three factors that have been the most significant to them becoming a productive librarian-researcher (open-ended text response). the informed consent and full questionnaire may be found as appendix b. research productivity research productivity was measured based on a review of each participant’s current cv. at the completion of the survey, participants were asked to forward their cv to the authors, who reviewed the research output over the last 10 years. the authors used the counting and scoring scheme developed by hoffmann et al. (2017, p. 107), outlined in table 1. the score was not adjusted for multi-authored pieces; if the output was listed on a cv, it was counted as one item, regardless of author position.   table 1 scores used for research output output type score poster 0.5 presentation 1 conference proceeding 1 non-peer-reviewed article 3 book chapter 5 edited book 6 peer-reviewed article 9 authored book 10   items such as book reviews, creative writing, teaching a class, moderating a conference panel, editing a journal, or writing an evidence summary were not included as research output. although these works are scholarly in nature, they were excluded because they are not dissemination of original research. for this analysis, if a presentation was determined to be part of a participant’s job performance (for example, a webinar about how to use a library resource), it was not scored. the authors do include the following, as done by hoffmann et al. (2017): poster; presentation; conference proceeding; non-peer-review article; book chapter; edited book; peer-reviewed article; authored book. analysis descriptive statistics (counts and percentages for categorical / nominal responses, means and standard deviations for continuous measures) were calculated for each individual survey item. the final research output score for each participant was calculated by multiplying the number of types of output and their related scores, then adding all scores together. for questions about the research network, descriptive measures of network composition were calculated from the raw responses about alter characteristics and relationships with alters provided by respondents. first, network size was calculated at the participant level by counting the total number of alters provided by each participant and then averaged across all participants. measures of network composition were produced at the respondent level as well as across all respondents’ networks. counts of different types of network members were produced for each respondent (e.g. professional colleagues, mentees, etc.). also, measures of percent of different types of network members were produced for the entire sample of alters by counting the total number of network members with the characteristic divided by the total number of alters named by participants. the measure “density” was produced to measure the network structure of each respondent’s ego-centric network data using statistical software r’s “igraph” package (borgatti, everett, & johnson, 2013; csárdi, 2019). density is the ratio of observed relationships in a network to the total number of possible network ties and rages from zero (no observed ties) to one (all possible ties exist). a density measure was produced for the tie between alters who knew each other and did research together. the authors conducted bivariate correlation tests to test the association between survey responses and research output. first, a shapiro-wilk test was conducted in spss (version 24) to test for data normality for the research output scores and it was determined that the distribution of research output scores is not normal (p = .00). the histogram for the research output scores is included in appendix c. the following findings, then, use the non-parametric tests mann-whitney u and spearman’s rho, depending on the nature of the variables tested. the significance of correlations was evaluated at the 95% confidence level (p < .05). coding of open-ended questions the authors coded responses the four open-ended questions, using codes initially informed by the research success factors identified by hoffmann et al. (2014, 2017). the initial code definitions were iteratively modified and refined to fit the data, including differentiating factors that are close to one another. for example, the authors split education from experience, to create two codes; and they wrote definitions for intrinsic motivations to differentiate them from personality traits. the authors also created a new code for job-related characteristics or opportunities, to account for respondents’ comments about the nature of their work and its contribution to their research. they also eliminated one of the factors, departmental/institutional qualities, as it was impossible to differentiate it from organizational climate. the research success factors provided a useful framework for coding the respondents’ answers to open-ended questions and validated the categories and success factors identified by hoffmann et al. (2017). new codes were easily placed within the three categories: individual attributes, peers and community, and institutional structures and supports. the codebook is appendix d, with example text from the survey respondents. results a total of 46 participants completed the survey and provided their cvs, for a 58.97% completion rate. of the 46 respondents, 70% currently work in arl member libraries and 30% in other academic libraries. survey professional training and research environment other than holding a lis master’s degree (held by all but one of the 46 respondents), there was diversity in professional training and research environment among respondents. there was a range of types of graduate degrees and a mix of degree types. there were 19 respondents who hold no additional degree beyond the lis. another 15 respondents hold a second master’s degree, while 12 respondents hold a doctoral-level degree, 9 of those with a second master’s degree and 3 without an additional master’s degree. the professional age of the group varies, with degree completion ranging from 1970 to 2015. of the responses received, 1 respondent completed the lis degree in the 1970s, 5 completed it in the 1980s, 17 completed it in the 1990s, 19 completed the degree from 2000-2010, and 3 completed it since 2011. on average, this group has held their professional lis credentials for about 20 years (sd = 8.62). there is strong agreement in the group related to their belief that their lis master’s degree did not adequately prepare them to conduct original research; 38 of the 46 do not believe their degree provided research-readiness. on the whole, the group participates minimally in educational activities about research methods, reporting about three activities, with self-education being the most popular, noted by 41 of the 46 respondents. only 1 of the 45 respondents with a lis master’s degree wrote a thesis while completing the lis degree. of the 26 who reported holding an additional master’s degree, 12 wrote a thesis while completing that degree (46.15%). of the 45 respondents, 26 believe that their lis master’s degree adequately prepared them to read and understand research-based literature, but only 8 believe that their lis master’s degree adequately prepared them to conduct original research. the group notes the availability and use of partial travel funds from their institutions or libraries, with that option present for 39 of the 46 respondents. the support option least offered was short-term pre-tenure research leave, with only 11 respondents reporting it; 5 of those 11 had taken advantage of that support. the group has participated more often in informal mentoring opportunities, both as mentor (33 of 46) and mentee (30 of 46), than formal mentoring opportunities. it was reported that 27 had participated in a formal program, as a mentor and 11 had participated in a formal program, as a mentee. in total, 35 respondents (76%) replied that they had achieved tenure either at their previous institution, at their current institution, or both. at their current institutions 33 of the respondents are currently at the rank of associate librarian or librarian. there were 11 respondents who skipped past this question, which did not have a required response; it is unclear if the respondent refused to answer this question, accidentally skipped answering, or if they did not achieve tenure at their previous or current institution. early successful efforts in conducting research were mainly conducted as solo endeavors, noted by 25 respondents. similarly, 22 responded that they currently mainly conduct their research alone. research network of the respondents, 43 provided complete network data. the number of people in each research network ranged from 1 to 32, with the most frequently reported as 4 (with 8 respondents reporting this number). the average number of people in the research networks is 8.09 (sd = 7.03). see figure 1 for the range of network sizes reported by the respondents. of the 348 total people mentioned, the respondents reported having been in contact over 3 or more times, for any reason over the past 30 days, with 170 (48%). the respondents had research interactions with 82 (34%) of the 348 mentioned. figure 1 numbers of people in the research network.   on average, the respondents offered help to 1.75 other people, with even support on average of 4.90. as shown in table 2, “professional colleague” is the group with the largest relationship type reported, with 172 people in the category. it is interesting to note that in the relationship type of “both friend and colleague,” reciprocal support is the highest, with 124 people fitting those two criteria.   table 2 category of relationship by type of support (n = 348)   personal friend professional colleague both friend and colleague number reported % number reported % number reported % i’m usually asking this person for advice or help 10 43.48 47 27.33 16 10.46 i’m usually giving this person advice or help 3 13.04 56 32.56 13 8.50 it’s pretty even; i ask for help but also give help in equal amounts 10 43.48 69 40.12 124 81.05 total 23 100 172 100 153 100   of the 179 people (of 348 total) identified as working at the same institution, in the same library, the majority of communications are done in person, with 158 reported as such. of the 35 identified as working at the same institution but not in the library, half of communications are done in person, followed by email. of the 134 who do not work at the same institution, 43% (57) of those communications are conducted via email. the majority of the 348 people mentioned as part of the respondents’ research networks are not involved in any mentor relationship, with 219 being identified as having “no mentor relationship between us.” of the 348, 67 are reported as being the mentor in the relationship, and 62 as being the mentee. of the 2,273 possible relationships between the persons named, the average density is 54% (sd = 0.49). only 361 (or about 16%) do research together. beliefs about and the practice of the research process all 46 respondents answered each research process question (listed in appendix e). among the four lowest scoring questions were three questions designed to measure peer support and one related to extrinsic motivation. two of the three peer support questions ask about participation in a writing group and a journal club, both support activities that are relatively recent activities and focused on the needs of novice researchers. the lowest scoring question, with only seven yes responses, states “i do research only because it is a requirement of my job.” the 12 highest scoring questions, with scores at or above 40, were designed to measure personal commitment to research, institutional support, extrinsic motivations, and personality traits. of the 46 respondents 45 answered yes to: “i can achieve my research goals”; and “publishing gives me a personal sense of satisfaction”.  research productivity as shown in table 3, presentations are the most recorded research output, accounting for 49.91% of total research output (802 presentations). the least recorded research output, accounting for just 0.50%, are edited books. the one output type which all participants had used is peer-reviewed articles, with one as the minimum recorded; all other output types have zero recorded as the minimum. the research output scores for each participant were calculated according to the weights noted in table 1 and ranged from 32.5 to 307. correlations between research output and professional training, research environment there were no significant correlations between research output and professional training or research environment. completing a thesis for an additional master’s degree (other than the lis) was also not significantly correlated with research output (u = 78.5, p = .838). belief that one’s lis degree had prepared one to read and understand research-based literature and research output was also found to be not statistically significant, as well as belief that one’s lis degree had prepared one to conduct original research (u = 244.5, p = .954 and u = 126.5, p = .530, respectively). the authors divided into one group those respondents who reported participating in four or more educational activities about research methods and into another group those who reported three or fewer, of seven possible activities listed in the survey, but found no statistically significant difference between the groups, related to research output (u = 236.5, p = .727). the authors found no statistical significance between the number of research support options provided by the institution or library, and research output (rs(46) = .181, p = .228). there is no statistically significant difference in the distributions of those who are currently tenured and research output (u = 207.5, p = .864).   table 3 participant research output, 2008-2018 output type min max mean median sd total number reported % of output reported poster 0 68 3.7 1 10.3 169 10.52% presentation 0 92 17.4 11 21.5 802 49.91% conference proceeding 0 7 1.2 0 1.9 55 3.42% non-peer-reviewed article 0 14 1.6 1 2.5 74 4.60% book chapter 0 12 1.9 1 2.4 86 5.35% edited book 0 2 0.2 0 0.4 8 0.50% peer-reviewed article 1 27 8.6 8 5.4 397 24.70% authored book 0 3 0.35 0 0.7 16 1.00%   research output, related to the research network ego-centric network size was not significantly correlated with research output (rs(46) = -.061, p = .687). also, having any type of mentor relationship with an alter (as formal mentor, as informal mentor, as formal mentee, as informal mentee) was not significantly correlated with research output (u = 216.5, p = .372; u = 193.5, p = .608; u = 147, p = .251; u = 222.5, p = .686). being above average in giving help or above average in giving/taking an equal amount of help from those in the research network was also not significantly correlated with research output (rs(11) = .318, p = .341 and rs(15) = -.190, p = .498, respectively). there was no significant correlation between the number of alters in the research network who either worked in the library or at the university and research output (rs(43) = -.098, p = .530). the authors did find that the density of egocentric networks defined by ties between alters who both knew each other and did research together was significantly correlated with research output (rs(41) = .398, p = .010). the denser the research collaboration network, the higher the research output. figure 2 shows an example of a network of a low research output respondent that also has a low-density network and a network of a high research output respondent, with a high-density network. it is interesting to note that in the low research output network, most of the alters are categorized as colleagues only, with a few friends/colleagues but in the high research output network all of the alters are categorized as both friends/colleagues. figure 2 research output and research tie density.   research output, related to statements about the research process the authors tested for associations between each of the 28 statements and research output and found 1 significant association. the distribution of the statement, “i have space where i am able to work effectively on my research” is similar to the distribution of research output (u = 10, p = .009). the authors replicated the analysis completed by hoffmann et al. (2017), using the number of peer-reviewed articles as the outcome variable, with no resulting statistical significance. responses to open-ended questions the responses to the open-ended questions revealed important factors in the librarians’ process in becoming accomplished researchers. respondents noted that the research environment can be a major contributor to research productivity, or it can be a hindrance. a positive environment has many components, including institutional supports, collaboration, and community. the environment that many respondents found most conducive to developing as a researcher includes high expectations; a variety of supports for research; and extrinsic rewards, such as salary increases, tenure, promotion, and opportunities for advancement. this is an illustrative comment: “my library is very supportive of research and scholarship, and librarians are expected to publish and work on scholarly projects.” many librarians noted that being on the tenure-track provided an extrinsic motivation to develop research skills and become a productive researcher. one librarian wrote: “there is both pressure and support in tenure track positions for conducting research.” another theme was that often librarians conduct research in order to fulfill the requirements of tenure but develop a personal commitment to research and intrinsic motivation. this is a representative comment: “the tenure track and the focus on writing was a big element that got me started. once i became comfortable, i realized how much i enjoyed writing.”     another important factor is supportive colleagues and a community of researchers. this is an illustrative comment: “librarians have created networks of support such as writing sessions and research forums, in which we share our projects with our colleagues and possibly find opportunities to cooperate.” some of the comments reveal personality traits that contribute to research success. this is representative: “i believe in working for shared good, which is truly collaborative. i base my work on mutual aid and tend and befriend (not competitive, pushing ourselves to be our best together, inclusivity and diversity that grows and improves from our differences), and my collaborators work in the same manner. this makes collaborative work more productive, better, and more joyous.” discussion the most unique finding of this work relates to the research network. the authors found that a high number of persons in the networks who both know each other and do research together is significantly related to research output. the denser the research network is in terms of research collaborations, the higher the research output. another finding of this work is that a productive grouping of network members is those who are both friends and colleagues and also give/take research help in equal amounts. these findings align well with the responses to the open-ended question about the people in the current research network that may have contributed to the productivity of the respondent, which are mainly on the theme of collaboration. as might be expected, there was little variation in responses to yes and no statements designed to differentiate successful librarian-researchers from other librarians. the librarians in the present study were chosen because they are all accomplished researchers. of the 28 statements about the research process, the strongest group response was to the statement, “i can achieve my research goals”; 45 of 46 respondents agreed in the affirmative that this statement does generally apply to them. of the respondents, 44 have “participated in activities that support lis research,” and 43 “have space where i am able to work effectively on my research.” the authors’ statistical tests of associations between research output and all other variables from the survey overwhelmingly did not show significant findings. in the area of professional training and research environment, there were no significant correlations between those variables and research output. fiawotoafor et al., as did the authors, found no positive correlation between number of years in the profession and research output (2018). in the area of research networks, the one meaningful significant finding is that those who have networks with high density of research collaborators was significantly related to research output. in the area of the statements about the research process there was one significant association, with those who said that they had space where they can work effectively on their research tended to have higher research output. the authors wish to be clear that this finding does not imply a causality; it may be that having space helps productivity or that productive researchers make sure to have space to work productively. the open-ended questions revealed both commonalities and differences among the respondents. they offer important insights into the individual motivations of librarian-researchers. in addition to the many positive factors in their lives and professional environments, the open-ended questions also provided respondents with an opportunity to mention negative factors in research productivity, including the loss of a supportive supervisor or administrator, the demands of a new administrative position, family pressures, and anxiety over the need to publish and achieve tenure. the impact of assuming an administrative position is especially interesting; respondents noted that this changed their research or hampered their ability to conduct and disseminate research. limitations of this study and future research this study is the first to examine the research experiences and beliefs of accomplished librarian-researchers. one limitation of the study is the difficulty in defining the population. although all respondents are among the most productive librarian-researchers in the u.s., many equally productive researchers may have been missed by the methodology used to identify them. the respondents vary in the volume of their research output and the types of their research output. hoffmann et al. (2017) noted that librarians often disseminate the results of their research via conference presentation rather than publication. the authors found this to be true as well and considered that non-research presentations might skew the research totals. in order to reduce this effect, the authors counted only presentations that are scholarly in nature. the point system also strongly favors publication over presentation. in a future study, the authors would like to explore the phenomenon of librarians presenting about their research rather than publishing their findings. another interesting finding is that while most of the respondents have a positive attitude toward research and feel confident in their research abilities, some expressed a high level of anxiety regarding research and do not enjoy research. the authors plan to explore these factors in follow-up interviews with the respondents. what is the source of this anxiety and lack of confidence?  the authors do not report in this study on the hierarchical academic rank of the people the librarian-researchers identify in their networks (as described in fu, velema, & hwang, 2018), though this may be a fruitful topic of conversation in follow-up interviews. the authors could discover if the choices the librarian-researchers made about the people in the development of their research networks over the course of their career were decided based on “reaching up” in the hierarchy, to gain a research-related benefit (fu et al., 2018, p. 266). this line of inquiry would expand on a narrow area of focus in this research, that of mentor relationships, and whether the respondents act as mentees (those in a lower rank gain an advantage from a mentor in a higher rank), mentors, or have no mentor relationship with those identified in their research networks (abramo, d’angelo & murgia, 2017; hollingsworth & fassinger, 2002). for this work the authors do not focus on collaborations leading to co-authorship, though that is well addressed in the literature (see lee & bozeman, 2005, and xia, chen, wang, li, & yang, 2014) and may be an area the authors identify as a possible future area of inquiry with this data set. abramo et al. (2019) describe in their work the advantage of scientific collaboration may have, especially related to attracting different resources and perspectives which result in a wider audience for the research. the authors did notice during their review of the cvs that some of those in the population were co-authors, so in future work the group may examine the co-authorship network to look for associations between centrality and research output (as in abbasi, altmann & hossain, 2011, and abbasi, jalili, & sadeghi-niaraki, 2018). finally, the authors did not address the concept of gender and research productivity. research is being conducted in this area (see mayer & rathmann, 2018) and is a topic of concern, given that the field of librarianship is dominated by women. hoffmann et al. (2017) found that gender did not have a significant effect on research productivity and so for this work the authors decided not to pursue that as a variable. however, for librarian-researchers at the highest levels of accomplishment, there may be gender differences. this may be an interesting area of inquiry for follow-up in interviews with respondents. summary this work explores the factors that may contribute to a librarian becoming an accomplished researcher. an understanding of these factors can provide evidence based guidance to those at the beginning of their research careers in designing their own trajectories. it may also aid library administrators in creating a supportive environment for researchers.   the population studied is the group of librarians identified as accomplished researchers. they were identified through 2 means: employed at association of research libraries institutions who published more than 5 items indexed in the social science citation index in the last 10 years and the top 50 most published librarian-researchers for 2007-2012 (walters & wilder, 2015). this population was recruited into the study that included both a survey and cv component. analyses of the resulting survey data and cv data show that this population has professional training backgrounds and current environments that vary widely and are not statistically associated with research output. those with a high number of persons in their networks who both know each other and do research together is significantly related to research output, a unique finding for the profession of library science. a large group of those identified in the research networks are “both friend and colleague” and offer each other reciprocal support. those who agree with the statement, “i have space where i am able to work effectively on my research” is also associated with research output. the statistical data do not tell the entire story, however. in open-ended questions, the respondents cited numerous factors over their careers that led to their research success. these factors span the three categories identified and studied by hoffmann et al. (2014, 2017). the results of this study support their finding that becoming a productive researcher is the product of individual attributes, peers and community, and institutional structures. from these categories, the three most frequently-mentioned factors from the open-ended questions were developing a personal commitment to research, collaboration, and positive organizational climate. furthermore, the open-ended questions allowed the respondents to elaborate on positive and negative influences in their educational background; previous work; and professional and personal environments. acknowledgments this project was made possible in part by the institute of museum and library services grant re-40-16-0120-16.   the authors thank scelc (https://scelc.org/) for providing a research incentive grant to supply some of the monetary incentives for this research.   references   abbasi, a., altmann, j., & hossain, l. (2011). identifying the effects of co-authorship networks on the performance of scholars: a correlation and regression analysis of performance measures and social network analysis measures. journal of informetrics, 5(4), 594-607. https://doi.org/10.1016/j.joi.2011.05.007 abbasi, a., jalili, m., & sadeghi-niaraki, a. (2018). influence of network-based structural and power diversity on research performance. scientometrics, 117, 579-590. https://doi.org/10.1007/s11192-018-2879-3 abramo, g., d’angelo, c. a., & di costa, f. (2019). the collaboration behavior of top scientists. scientometrics, 118, 215-232. https://doi.org/10.1007/s11192-018-2970-9 abramo, g., d’angelo, a. c., & murgia, g. (2017). the relationship among productivity, research collaboration, and their determinants. journal of informetrics, 11(4), 1016-1030. https://doi.org/10.1016/j.joi.2017.09.007 association of research libraries. (2019). list of arl members. retrieved from https://www.arl.org/membership/list-of-arl-members   berg, s. a., jacobs, h.l.m., & cornwall, d. (2013). academic librarians and research: a study of canadian library administrator perspectives. college and research libraries 74(6), 560-572. https://doi.org/10.5860/crl12-366   borgatti, s. p., everett, m. g., & johnson, j. c. (2013). analyzing social networks. thousand oaks: sage publications limited.   chang, y-w. (2016). characteristics of articles co-authored by researchers and practitioners in library and information science journals. the journal of academic librarianship 42(5), 535-541. https://doi.org/10.1016/j.acalib.2016.06.021   clarivate. (2018). web of science. retrieved from http://webofknowledge.com/wos   crossley, n., bellotti, e., edwards, g., everett, m. g., koskinen, j., & tranmer, m. (2015). social network analysis for ego-nets. thousand oaks: sage publications limited.   csárdi, g. (2019). package ‘igraph.’ retrieved from https://cran.r-project.org/web/packages/igraph/igraph.pdf   egoweb 2.0 [computer software]. (2015). retrieved from https://github.com/qualintitative/egoweb    fennewald, j. (2008). research productivity among librarians: factors leading to publications at penn state. college & research libraries, 69(2), 104-116. https://doi.org/10.5860/crl.69.2.104    fiawotoafor, t., dadzie, p. s., & adams, m. (2019, january). publication output of professional librarians in public university libraries in ghana. library philosophy and practice (e-journal) 1-40. retrieved from https://digitalcommons.unl.edu/libphilprac/2303/   fu, y.-c., velema, t. a., & hwang, j.-s. (2018). upward contacts in everyday life: benefits of reaching hierarchical relations in ego-centered networks. social networks, 54, 266-278. https://doi.org/10.1016/j.socnet.2018.03.002    galbraith, q., smart, e., smith, s. d., & reed, m. (2014). who publishes in top-tier library science journals? an analysis by faculty status and tenure. college & research libraries, 75(5), 724-735. https://doi.org/10.5860/crl.75.5.724   hoffmann, k., berg, s. a., & koufogiannakis, d. (2014). examining success: identifying factors that contribute to research productivity across librarianship and other disciplines. library and information research, 38(119), 13-28. https://doi.org/10.29173/lirg639   hoffmann, k., berg, s., & koufogiannakis, d. (2017). understanding factors that encourage research productivity for academic librarians. evidence based library and information practice, 12(4). https://doi.org/10.18438/b8g66f   hollingsworth, m. a., & fassinger, r. e. (2002). the role of faculty mentors in the research training of counseling psychology doctoral students. journal of counseling psychology, 49(3), 324-330. https://doi.org/10.1037/0022-0167.49.3.324   hollister, c. v. (2016). an exploratory study on post-tenure research productivity among academic librarians. the journal of academic librarianship, 42(4), 368-381. https://doi.org/10.1016/j.acalib.2016.04.021   kennedy, m. r., & brancolini, k. r. (2018). academic librarian research: an update to a survey of attitudes, involvement, and perceived capabilities. college & research libraries, 79(6), 822. https://doi.org/10.5860/crl.79.6.822   kennedy, m. r., kennedy, d. p., & brancolini, k. r. (2017). the evolution of the personal networks of novice librarian researchers. portal: libraries and the academy, 17(1), 71-89. https://doi.org/10.1353/pla.2017.0005      krackhardt, d. (1987). cognitive social structures. social networks, 9(2), 109-134. https://doi.org/10.1016/0378-8733(87)90009-8   lee, s., & bozeman, b. (2005). the impact of research collaboration on scientific productivity. social studies of science, 35(5), 673-702. https://doi.org/10.1177/0306312705052359   luo, l. (2011). fusing research into practice: the role of research methods education. library & information science research, 33(3), 191-201. https://doi.org/10.1016/j.lisr.2010.12.001   luo, l., & mckinney, m. (2015). jal in the past decade: a comprehensive analysis of academic library research. the journal of academic librarianship 41(2): 123-129. https://doi.org/10.1016/j.acalib.2015.01.003   mayer, s. j., & rathmann, j. m. k. (2018). how does research productivity relate to gender? analyzing gender differences for multiple publication dimensions. scientometrics, 117, 1663-1693. https://doi.org/10.1007/s11192-018-2933-1   mccarty, c. (2002). structure in personal networks. journal of social structure, 3(1). retrieved from https://www.cmu.edu/joss/content/articles/volume3/mccarty.html   mccarty, c., lubbers, m., vacca, r., & molina, j. l. (2019). conducting personal network research: a practical guide. guilford press.   meadows, k. n., berg, s. a., hoffmann, k., gardiner, m. m., & torabi, n. (2013). a needs-driven and responsive approach to supporting the research endeavours of academic librarians. partnership: the canadian journal of library and information practice and research 8(2), 1-32. https://doi.org/10.21083/partnership.v8i2.2776   perkins, g. h., & slowik, a. j. w. (2013). the value of research in academic libraries. college & research libraries, 74(2), 143-158. https://doi.org/10.5860/crl-308   sassen, c., & wahl, d. (2014). fostering research and publication in academic libraries. college & research libraries 75(4), 458-49. https://doi.org/10.5860/crl.75.4.458   smigielski, e. m., laning, m. a., & daniels, c. m. (2014). funding, time, and mentoring: a study of research and publication support practices of arl member libraries. journal of library administration, 54(4), 261-276.  https://doi.org/10.1080/01930826.2014.924309   vilz, a. j., & poremski, m. d. (2015). perceptions of support systems for tenure-track librarians. college & undergraduate libraries, 22(2), 149-166. https://doi.org/10.1080/10691316.2014.924845   walters, w. h., & wilder, e. i. (2015). worldwide contributors to the literature of library and information science: top authors, 2007-2012. scientometrics, 103, 301-327. https://doi.org/10.1007/s11192-014-1519-9   wasserman, s., & faust, k. (1994). social network analysis: methods and applications. new york: cambridge university press.   watson-boone, r. (2000). academic librarians as practitioner-researchers. the journal of academic librarianship, 26(2), 85-93. https://doi.org/10.1016/s0099-1333(99)00144-5   xia, f., chen, z., wang, w., li, j., & yang, l. t. (2014). mvcwalker: random walk-based most valuable collaborators recommendation exploiting academic factors. ieee transactions on emerging topics computing, 2(3), 364-375. https://doi.org/10.1109/tetc.2014.2356505   appendix a recruitment email   email subject: personal invitation, survey for accomplished librarian-researchers   email body:   greetings! we invite you to participate in a study of accomplished librarian-researchers. we selected you for this study based on your high number of publications in recent years. as part of a small group of productive librarian-researchers, we hope you will agree to participate in the study. the purpose of the study is to understand the factors that contributed to your productivity.   for your participation, we are pleased to offer you a $100.00 gift card (via https://www.giftcards.com/virtual-gift-cards). your participation includes two actions: 1.       complete a web-based survey. in the survey, we will ask you to click through a series of questions with options for response. the survey can take up to 30 minutes to complete. 2.       send your current cv to marie kennedy at marie.kennedy@lmu.edu, so that we may examine the last ten years of your scholarly productivity and professional experience. we plan to publish and present the results of this study. at the time of publishing and presenting, the data will be anonymized. there are no expected risks for you in participating in this research.   to initiate your participation, please complete the survey at [personalized survey url]. use the email address [participant’s email address] at the prompt in the survey. please plan to complete the survey by july 1, 2018.   your voice is an important one in this research. thank you for your consideration.   kind regards,   marie r. kennedy, loyola marymount university kristine r. brancolini, loyola marymount university david p. kennedy, rand corporation   appendix b survey   page 1: accomplished librarian-researchers -informed consent   introduction to the study   we invite you to participate in a study of accomplished librarian-researchers. you have been selected for this study based on your high number of publications in peer-reviewed journals in recent years. as part of a small group of productive librarian-researchers, we hope you will agree to participate in the study.   purpose   the purpose of this study is to understand the factors that contributed to the productivity of accomplished librarian-researchers. we hope to publish and present the results of this study. at the time of publishing and presenting the data will be anonymized.   what will happen during the study   we will ask you to take two actions: 1. complete this web-based survey. in the survey we will ask you to click through a series of questions with options for response. the survey is likely to take up to 30 minutes to complete. 2. send your current cv to marie kennedy at marie.kennedy@lmu.edu, so that we may examine the last ten years of your scholarly productivity and professional experience.   your privacy is important   we will make every effort to protect your privacy. no sensitive information will be gathered as part of this survey. any information you provide will remain confidential. only the co-investigators will view the results of the survey in their raw form.   your rights   your participation in this study is completely voluntary and no risks are anticipated for you as a result of participating. if you decide to be in the study, you will have the right to stop participating at any time.   incentive   when the co-investigator has confirmed that your cv has been received and the survey completed you will be sent a $100.00 gift card from any egift card brand listed at https://www.giftcards.com/virtual-gift-cards.   institutional review board approval this study has been reviewed by the office of research and sponsored projects at loyola marymount university. if you have any questions about your rights as a research participant in this study, please contact the david a. moffet, ph.d., chair, institutional review board, loyola marymount university, one lmu drive, suite 47000, los angeles, ca 90045 at (310) 338-4400 or at david.moffet@lmu.edu.   if you agree with all of the above statements, provide your electronic signature by clicking on "next" below.   page 2: introduction to the survey there are three sections to this survey. it begins with a series of questions about your research training and work environment, continues with a section about your research network, and finishes with a series of statements about the research process.   question 1. please select the response that best describes your graduate-level educational background. i have an lis master’s degree i have an lis and another master’s degree i have a non-lis master’s degree i have a doctoral degree   question 2. in what year did you complete your lis degree? enter the four-digit year. text entry for response (question 2 appears if question 1 response is i have an lis master’s degree or i have an lis and another master’s degree)   question 3. did you write a thesis in completing your lis degree? yes no i don’t remember (question 3 appears if question 1 response is i have an lis master’s degree or i have an lis and another master’s degree)   question 4. did you write a thesis in completing another master's degree? yes no i don’t remember (question 4 appears if question 1 response is i have an lis and another master’s degree or i have a non-lis master’s degree)   question 5. do you believe that your lis master's degree adequately prepared you to read and understand research-based literature? yes no (question 5 appears if question 1 response is i have an lis and another master’s degree or i have an lis and another master’s degree)   question 6. do you believe that your lis master's degree adequately prepared you to conduct original research? yes no (question 6 appears if question 1 response is i have an lis and another master’s degree or i have an lis and another master’s degree)   question 7. in which of the following educational activities about research methods have you ever participated? check all that apply. formal master's degree lis course(s) (e.g., research methods, statistics) formal master's degree non-lis course(s) (e.g., courses in other departments) formal doctoral degree lis course(s) (e.g., research methods, statistics) formal doctoral degree non-lis course(s) (e.g., courses in other departments) continuing education program(s): workshops, conferences, or other continuing education activities outside the library/your institution staff development program(s) provided by your library or university self-education activities (e.g., professional reading, online tutorial) none of these   question 8. did you take advantage of any of the following research support options provided by your institution or library when you were early in your research career? check all that apply. release time during the work week short-term pre-tenure research leave sabbaticals for librarians travel funds (full reimbursement) travel funds (partial reimbursement) research grants formal mentorship (experienced librarian researcher partners with novice researcher) informal mentorship (journal club discussions or article/proposal feedback sessions) research design or statistical consultant workshops or other forms of continuing education no research support was available to me   question 9.1. which of the following research support options does your current institution or library provide for librarians, and which have you taken advantage of? release time during the work week it is offered and i have taken advantage of it it is offered but i have not taken advantage of it it is not offered   question 9.2. which of the following research support options does your current institution or library provide for librarians, and which have you taken advantage of? short-term pre-tenure research leave it is offered and i have taken advantage of it it is offered but i have not taken advantage of it it is not offered   question 9.3. which of the following research support options does your current institution or library provide for librarians, and which have you taken advantage of? sabbaticals for librarians it is offered and i have taken advantage of it it is offered but i have not taken advantage of it it is not offered   question 9.4. which of the following research support options does your current institution or library provide for librarians, and which have you taken advantage of? travel funds (full reimbursement) it is offered and i have taken advantage of it it is offered but i have not taken advantage of it it is not offered   question 9.5. which of the following research support options does your current institution or library provide for librarians, and which have you taken advantage of? travel funds (partial reimbursement) it is offered and i have taken advantage of it it is offered but i have not taken advantage of it it is not offered   question 9.6. which of the following research support options does your current institution or library provide for librarians, and which have you taken advantage of? research grants it is offered and i have taken advantage of it it is offered but i have not taken advantage of it it is not offered   question 9.7. which of the following research support options does your current institution or library provide for librarians, and which have you taken advantage of? formal mentorship (experienced librarian-researcher with an agreement to advise a less experienced librarian-researcher – one-on-one) it is offered and i have taken advantage of it it is offered but i have not taken advantage of it it is not offered   question 9.8. which of the following research support options does your current institution or library provide for librarians, and which have you taken advantage of? informal mentorship (more casual one-on-one consultation about research or peer mentoring, such as journal clubs discussions or article/proposal feedback sessions) it is offered and i have taken advantage of it it is offered but i have not taken advantage of it it is not offered   question 9.9. which of the following research support options does your current institution or library provide for librarians, and which have you taken advantage of? research design or statistical consultant it is offered and i have taken advantage of it it is offered but i have not taken advantage of it it is not offered   question 9.10. which of the following research support options does your current institution or library provide for librarians, and which have you taken advantage of? workshops or other forms of continuing education it is offered and i have taken advantage of it it is offered but i have not taken advantage of it it is not offered   question 10.1. have you ever participated in any of the following types of mentorship program? formal mentorship (experienced librarian-researcher with an agreement to advise a less experienced librarian-researcher – one-on-one) in which you are the mentor yes no   question 10.2. have you ever participated in any of the following types of mentorship program? formal mentorship (experienced librarian-researcher with an agreement to advise a less experienced librarian-researcher – one-on-one) in which you are the mentee yes no   question 10.3. have you ever participated in any of the following types of mentorship program? informal mentorship (more casual one-on-one consultation about research or peer mentoring, such as journal clubs discussions or article/proposal feedback sessions) in which you are the mentor yes no   question 10.4. have you ever participated in any of the following types of mentorship program? informal mentorship (more casual one-on-one consultation about research or peer mentoring, such as journal clubs discussions or article/proposal feedback sessions) in which you are the mentee yes no   question 11. have you attained tenure? at a previous institution at my current institution   question 12. what is the highest rank you attained at a previous institution? assistant librarian/professor associate librarian/professor librarian/professor n/a librarians do not have academic rank (question 12 appears if question 11 response is at a previous institution)   question 13. what is the highest rank you attained at your current institution? assistant librarian/professor associate librarian/professor librarian/professor n/a librarians do not have academic rank (question 13 appears if question 11 response is at my current institution)   question 14. in what year did you complete your highest academic rank? enter the four-digit year. text entry response (question 14 appears if question 13 response is assistant librarian/professor, associate librarian/professor, or librarian/professor)   question 15. think back to your earliest successful efforts in conducting research. how did you mainly conduct it? mainly solo mainly with a partner who was more experienced than i was mainly with a partner who was less experienced than i was mainly with a partner who was equally experienced as i was mainly on a team of novice researchers mainly on a team with both novice and more experienced researchers   question 16. describe the type of research you are currently conducting. what methods are you using? what research questions are you exploring? text entry response   question 17. how do you mainly conduct your current research? check all that apply. mainly solo mainly with a partner who is more experienced than i am mainly with a partner who is less experienced than i am mainly with a partner who is equally experienced as i am mainly on a team of novice researchers mainly on a team with both novice and more experienced researchers   question 18. is there anything else about your professional training over the last ten years that you believe may have contributed to your productivity? text entry response   question 19. is there anything else about your research environment over the last ten years that you believe may have contributed to your productivity? text entry response   question 20. please name here the people with whom you have research interactions. you may just bounce ideas off of some of these people and with others you may work more closely and often. these may or may not be people you communicate with on a regular basis and may be professional colleagues, personal friends, and family. you may name up to 40 people. you may list just their first names, if you are able to distinguish between them that way; neither your name nor their names will be shared with anyone outside the survey. text entry response   question 21. during the past 30 days, how often have you had contact for any reason with each of the following people? [respondent is presented with the names of the people entered in question 20, with matrix response option] not at all once or twice three or more times   question 22. during the past 30 days, how often have you talked about research (help, advice, bounce ideas off of) with each of the following people? [respondent is presented with the names of the people entered in question 20, with matrix response option] not at all once or twice three or more times   question 23. how would you mainly describe your relationship with each person? select one type of relationship for each person. [respondent is presented with the names of the people entered in question 20, with matrix response option] personal friend professional colleague both friend and colleague   question 24. how would you characterize the majority of your research interactions with each of the following people? [respondent is presented with the names of the people entered in question 20, with matrix response option] i’m usually asking this person for advice or help i’m usually giving this person advice or help it’s pretty even; i ask for help but also give help in equal amounts   question 25. are these people at your current institution? [respondent is presented with the names of the people entered in question 20, with matrix response option] at my institution, in my library at my institution, not in my library not at my institution   question 26. how do you mainly interact with these people? select one mode of interaction that you usually use with each person. [respondent is presented with the names of the people entered in question 20, with matrix response option] in person (face to face) online forum (chat room, facebook, twitter) phone calls texts or personal messages video conference (skype, gotomeeting) email   question 27. how would you characterize the type of interactions you mainly have with each of these people? [respondent is presented with the names of the people entered in question 20, with matrix response option] we usually just chat about research, or we may give/get some brief help from each other on a project. we've done a/some small projects together. (example: sat on a conference panel together; collaborated on a conference poster) we've worked on a/some major projects together. (example: put together a conference session, published an article together)   question 28. do you have a mentor relationship with any of these people? [respondent is presented with the names of the people entered in question 20, with matrix response option] i am a formal or informal mentor to this person this person is a formal or informal mentor to me we do not have any kind of mentor relationship   question 29. does [person 1 entered at question 20] know each of the other people in your list, and do they interact for research-related purposes? [respondent is presented with the names of the people entered in question 20, with matrix response option] they know each other and have research interactions they know each other but i don’t know if they talk about or do research together they know each other but don’t talk about or do research together i don’t know if they know each other they don’t know each other   question 30. is there anything else about the people in your current research network over the last ten years that you believe may have contributed to your productivity? text entry response   question 31. indicate whether or not each statement, in general, applies to you. i consider research to be a priority. yes no   question 32. indicate whether or not each statement, in general, applies to you. i am currently working on a research project. yes no   question 33. indicate whether or not each statement, in general, applies to you. i usually have a research project that i'm working on. yes no   question 34. indicate whether or not each statement, in general, applies to you. i believe it is important for librarians to contribute to the profession via research. yes no   question 35. indicate whether or not each statement, in general, applies to you. i work on research outside of regular work hours. yes no   question 36. indicate whether or not each statement, in general, applies to you. i schedule dedicated time for research. yes no   question 37. indicate whether or not each statement, in general, applies to you. i have participated in activities that support lis research (e.g., peer review, editor of a journal, providing writing assistance to a colleague, etc.). yes no   question 38. indicate whether or not each statement, in general, applies to you. i have time to do research within my job. yes no   question 39. indicate whether or not each statement, in general, applies to you. i have space where i am able to work effectively on my research. yes no   question 40. indicate whether or not each statement, in general, applies to you. professional associations are a source of research community for me. yes no   question 41. indicate whether or not each statement, in general, applies to you. i attend conferences in order to connect with others who have similar research interests. yes no   question 42. indicate whether or not each statement, in general, applies to you. i feel like i belong to a research community. yes no   question 43. indicate whether or not each statement, in general, applies to you. i have participated in a peer support group related to research. yes no   question 44. indicate whether or not each statement, in general, applies to you. i have participated in a writing group. yes no   question 45. indicate whether or not each statement, in general, applies to you. i have participated in a journal club. yes no   question 46. indicate whether or not each statement, in general, applies to you. i ask my colleagues for feedback on my research. yes no   question 47. indicate whether or not each statement, in general, applies to you. i have received merit increments or promotion due to my research activities. yes no   question 48. indicate whether or not each statement, in general, applies to you. i am (formally or informally) expected to participate in research and scholarship. yes no   question 49. indicate whether or not each statement, in general, applies to you. i do research only because it is a requirement of my job. yes no   question 50. indicate whether or not each statement, in general, applies to you. i enjoy doing research. yes no   question 51. indicate whether or not each statement, in general, applies to you. i enjoy writing for publication. yes no   question 52. indicate whether or not each statement, in general, applies to you. i am confident that i have the ability to do research. yes no   question 53. indicate whether or not each statement, in general, applies to you. i can achieve my research goals. yes no   question 54. indicate whether or not each statement, in general, applies to you. i enjoy presenting at conferences. yes no   question 55. indicate whether or not each statement, in general, applies to you. i enjoy speaking with colleagues about my research. yes no   question 56. indicate whether or not each statement, in general, applies to you. publishing gives me a personal sense of satisfaction. yes no   question 57. indicate whether or not each statement, in general, applies to you. i can easily identify questions that could be answered through research. yes no   question 58. indicate whether or not each statement, in general, applies to you. i do research to satisfy my curiosity. yes no   question 59. thinking back on your entire research career, please list the three factors that have been the most significant in you becoming an accomplished librarian-researcher. text entry response   question 60. would you be interested in being contacted about participating in a possible follow-up study about accomplished librarian-researchers? yes no   final page. clicking the "finish" button below completes the survey. thank you for your participation. marie will be in touch with you via email to confirm the completion of the survey. in the meantime, if you have any questions or feedback about this survey please contact us at marie.kennedy@lmu.edu.   appendix c. histogram (with normal curve) of research output scores     appendix d codebook – accomplished librarian-researchers success factors   code definition when to use when not to use how to use and examples individual attributes: education formal education; continuing education; research training respondent mentions the impact of education on their research if an educational program is not mentioned specifically, use: experience example:  pursuing a ph.d. (both in terms of coursework and mentorship from my advisor and peers). example:  getting an undergraduate degree that required i learned research methodologies and statistics and apply them in a thesis. experience previous research experience; previous job experience with a research component respondent mentions a previous job that had a research component or requirement; or ambiguously uses the word “background.” if an educational experience is mentioned specifically – courses, academic degree program, continuing education, use: education example:  on-the-job experience in my former career… example:  having a background in psychology. intrinsic motivations personal and professional reasons for conducting research respondent gives responses that answer the “why” question with an internal motivation. respondent uses verbs rather than emotion words to describe motivations. when respondent gives responses that answer the “why” question with external, tangible rewards – or to avoid punishment.  see “extrinsic motivations.” code when respondent write about a research-oriented personality trait.  example:  having a strong desire to write and solve problems. or more general motivation without specific rewards or punishment. example:  to enhance my career. personality traits self-efficacy; innate drive to achieve or excel; creativity; leadership; positive attitude respondent expresses positive feelings in the context of conducting research or any of its components   when the respondent specifically mentions activities or behaviors related to research, rather than emotions, use: personal commitment to research. code when respondent expresses “an interest in” or uses positive emotional language: “confident,” “achieve,” “enjoy,” “sense of satisfaction.”  example:  i love questions and finding the answers. example:  i have an unrelenting desire to do more, better. personal commitment to research making research and writing a priority; participation in research-related activities, including serving as a peer reviewer; reading research; use of productivity tools. respondent mentions conducting research despite lack of monetary or other tangible rewards – or that these rewards are not their motivation. respondent mentions behaviors and activities, not emotions. research has become part of the respondent’s routine.  a habit of research.    example:  i’m continually thinking about current and future projects. example:  selection of topics that can be addressed using the existing data from publicly available sources… i spend less time administering surveys and more time analyzing the results. job-related characteristics or opportunities   the nature of the respondent’s job leads to opportunities for research, stimulates research ideas, or provides access to data. opportunity to positively affect practice; connection to teaching. job reveals relevant and interesting research topics. respondent mentions job characteristics or research contributing to work success – opportunity to link research to practice. when respondent mentions rewards or punishments related to the job, use: extrinsic motivations. example:  the kind of work i do (electronic resources) often has new things to write on. example:  constant change in technology and standards relating to cataloging and metadata provide many opportunities for research. peers and community: collaboration composition and practices of research teams; collaborations between lis faculty and librarians respondent mentions the positive impact of collaborators, co-authors, or research partners if respondent mentions community of researchers, use: community example:  colleagues willing to work with me. community professional associations; research networks; socialization respondent mentions interactions with other researchers as a motivating factor or desire to contribute to librarianship. if respondent mentions actually working with others on research, use: collaboration example:  i love learning from others – especially those with more experience and expertise. example:  to nourish the profession. guidance and support of editors   respondent specifically mentions editors respondent mentions someone else who played a role in their research success   family and personal relationships   respondent mentions the positive influence of a family member or a friend who is a researcher or encouraged their research respondent mentions positive impact of mentors or professional colleagues. see “mentoring” or “peer support.” example:  my brother who is a prolific scholar and role model. mentoring informal and formal mentoring; supervising students; being a mentor; being mentored respondent mentions mentoring or being mentored; or working with students on research; or working with an individual. the respondent mentions “peer mentoring” specifically, use: peer support example:  working with a library dean who helped me see how research questions present themselves in the daily responsibilities of librarianship. peer support peer mentoring; writing support groups; seminar series respondent mentioned the positive influence of peers who were also conducting research; specifically mentions “peer mentoring.” the respondent mentions specific programs design for and with peers to support one another’s research.   example:  early peer-mentoring group at my current institution (1984-1994)   peers and community barriers lack of any of the supports noted above. respondent mention unsupportive peers or lack of a research community.     institutional structures and supports: extrinsic motivation monetary rewards; required for promotion and/or tenure; requirement for employment; desire to build resume respondent mentions tangible rewards for research or punitive measures if they had not conducted research   example:  i am on the tenure track (and after achieving tenure, advancing to full professor is the only clear way to get a raise) positive organizational climate supportive leadership; research valued by the organization; culture of research respondent mentions encouragement or environmental support factors; research as an “expectation.”  “rewards” are unspecified.   example:  research is encouraged. example: working in a library that rewards it. institutional resources equipment; funding; staff support respondent mentions monetary or human resources provided by the library or the institution respondent mentions other types of support. see “institutional supports.” example:  our library administration funds librarians to hire research assistants. institutional supports   support for research that is not monetary in nature, either funding or staff resources.  general statement of support. respondent mentions “institutional support” – without specifying the type of support. or mentions institutional supports like a statistical consultant or help from irb. respondent mentions a support that has a monetary value.  see “institutional resources.”   example:  support from my library for research. time autonomy over work schedule; balance between responsibilities; release time; sabbatical leave respondent mentions time-related factors – such as release time or sabbatical – work schedule flexibility.   example:  research time off has been helpful. institutional barriers the absence of the one or more of the institutional or organizational support noted above. respondent mentions the negative influence of organizational or institutional conditions   example: i accepted a position at an institution in which librarians do not hold faculty status and so there was no imperative to contribute to the scholarly literature in my field.   appendix e section 3 of the survey, statements about the research process   the respondents were asked to indicate whether or not the following statements, in general, applied to them.   statement yes no i consider research to be a priority. 37 9 i am currently working on a research project. 41 5 i usually have a research project that i'm working on. 38 8 i believe it is important for librarians to contribute to the profession via research. 41 5 i work on research outside of regular work hours. 36 10 i schedule dedicated time for research. 29 17 i have participated in activities that support lis research (e.g., peer review, editor of a journal, providing writing assistance to a colleague, etc.). 44 2 i have time to do research within my job. 35 11 i have space where i am able to work effectively on my research. 43 3 professional associations are a source of research community for me. 29 17 i attend conferences in order to connect with others who have similar research interests. 31 15 i feel like i belong to a research community. 26 20 i have participated in a peer support group related to research. 20 26 i have participated in a writing group. 12 34 i have participated in a journal club. 10 36 i ask my colleagues for feedback on my research. 38 8 i have received merit increments or promotion due to my research activities. 35 11 i am (formally or informally) expected to participate in research and scholarship. 41 5 i do research only because it is a requirement of my job. 7 39 i enjoy doing research. 42 4 i enjoy writing for publication. 37 9 i am confident that i have the ability to do research. 42 4 i can achieve my research goals. 45 1 i enjoy presenting at conferences. 32 14 i enjoy speaking with colleagues about my research. 41 5 publishing gives me a personal sense of satisfaction. 15 1 i can easily identify questions that could be answered through research. 40 6 i do research to satisfy my curiosity. 40 6     top of form   evidence based library and information practice 2011, 6.3 100 evidence based library and information practice news/announcements call for proposals: 4th qualitative and quantitative methods in libraries international conference (qqml2012) © 2011. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. it is our great pleasure to announce the 4th qualitative and quantitative methods in libraries international conference (qqml2012), 22–25 may 2012, in limerick, ireland. we invite you to submit a paper, abstract, poster, or workshop proposal for the conference. the proposal abstract submission deadline is october 30, 2011. since 2009 qqml has provided an excellent framework for the presentation of new trends and developments in every aspect of library and information science, technology, applications and research. the conference will consider, but not be limited to, the following indicative themes: advocacy, networking, and influencing balanced scorecard tools in libraries bibliometrics conceptual and organizational perspectives of knowledge communication copyright and licensing data mining development and assessment of digital repositories development of new metrics digital archives digitization distance learning and the role of the library e-books e-learning and the contribution of the libraries, archives and museums e-research e-science electronic publishing human resources management information and knowledge services information literacy information retrieval innovative management institutional repositories intercultural management knowledge based systems and their applications knowledge management concept and technology libraries and shared services library cooperation: problems and challenges at the beginning of the 21st century library management and marketing library statistics measuring information literacy effectiveness metadata creation new means of selecting, collecting, organizing, and distributing digital content open access and open source operational information systems evidence based library and information practice 2011, 6.3 101 performance measurement and competitiveness publishing models, processes and systems qualitative and quantitative methodologies resource development policy scholarly information and new communication technologies semantics software strategic management team building and management technology transfer and innovation in library management theoretical models of information media user education you may also send proposals for special sessions (4‐6 papers) or workshops (more than 2 sessions), including the title and a brief description, to secretariat@isast.org, or via the electronic submission form at http://www.isast.org/abstractpaperregister.html. contributions may be realized through one of the following ways: o structured abstracts (not exceeding 500 words) and presentation; o full papers (not exceeding 7,000 words); o posters (not exceeding 2,500 words); o visual presentations (pecha kucha format). these presentations consist of exactly 20 slides, each of which is displayed for 20 seconds. total presentation time is precisely 6 minutes 40 seconds and so it is important to use the transition feature in powerpoint to time your presentation exactly. in all of the above cases at least one of the authors must be registered for the conference. abstracts and full papers should be submitted electronically within the timetable provided on the web page at http://www.isast.org/importantdates.html, and must be in compliance with the author guidelines found at http://www.isast.org/abstractpaperregister.html. all abstracts will be published in the conference book of abstracts and on the conference website. papers will be published on the conference website with permission of the author(s). student competition the conference offers postgraduate students and phd candidates a competition opportunity. students who submit to the conference may take part by submitting one paper to the competition. students who enter the competition must participate in the conference. the competition papers will be judged by a committee, and final selection will be based on the significance of the research theme, the research methodology, the description of the results, and the organization of the presentation. professors and supervisors are also encouraged to organize conference sessions of postgraduate theses and dissertations. please direct any questions regarding the qqml 2012 conference and student research competition to the secretariat of the conference at secretariat@isast.org. on behalf of the conference committee: dr. anthi katsirikou, conference co‐chair library director university of piraeus piraeus, greece head, european documentation center board member of the greek association of librarians and information professionals anthi@asmda.com jerald cavanagh bsc econ, msc, ma, local committee co‐chair institute librarian limerick institute of technology limerick, republic of ireland jerald.cavanagh@lit.ie mailto:secretariat@isast.org� http://www.isast.org/abstractpaperregister.html� http://www.isast.org/importantdates.html� http://www.isast.org/abstractpaperregister.html� mailto:secretariat@isast.org� mailto:anthi@asmda.com� mailto:jerald.cavanagh@lit.ie� evidence based library and information practice 2011, 6.3 102 padraig kirby ba (hons) hdiplis, local committee co‐chair acting serior library assistant limerick institute of technology limerick, republic of ireland padraig.kirby@lit.ie mailto:padraig.kirby@lit.ie� / evidence based library and information practice microsoft word art_goddard2.doc evidence based library and information practice 2007, 2:1  68 evidence based library and information practice     feature article    getting to the source: a survey of quantitative data sources available to the everyday  librarian:  part ii: data sources from specific library applications       lisa goddard  emerging services librarian/division head for systems  memorial university of newfoundland libraries  st. john’s, newfoundland, canada  email: lgoddard@mun.ca    received: 01 december 2006    accepted: 07 february 2007      © 2007 goddard. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    abstract    this is the second part of a two‐part article that provides a survey of data sources which are  likely to be immediately available to the typical practitioner who wishes to engage in  statistical analysis of collections and services within his or her own library.  part i outlines  the data elements which can be extracted from web server logs, and discusses web log  analysis tools. part ii looks at logs, reports, and data sources from proxy servers, resource  vendors, link resolvers, federated search engines, institutional repositories, electronic  reference services, and the integrated library system.     introduction    this article is the second in a two‐part guide  intended to help practitioners identify and  access web server log data that may be  useful for quantitative analysis of resource  and service use. part i introduced readers to  the standard data elements that appear in  web server logs and provided practical  pointers for aggregating and interpreting  this log data. part ii explores logs and  reports generated by library‐specific  applications such as proxy servers,  electronic index and journal sites, link  resolvers, federated search tools, virtual  reference software, and integrated library  systems. the reader should be aware that  this information may change over time as  applications evolve and mature. this article  is intended as a practical guide to help  researchers undertake quantitative analysis  in the current environment.   http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:1  69 host  created  accessed  counts  www.ncbi.nlm.nih.gov:80  10/31/2002 14:39  11/4/2006 21:41  3029226  www.sciencedirect.com:80  10/31/2002 15:26  11/4/2006 21:47  2282226  www.blackwell‐synergy.com:80  10/31/2002 15:12  11/4/2006 21:29  2279443  info.library.mun.ca:80  10/9/2001 13:20  11/4/2006 21:39  1798910  spweb.silverplatter.com:80  10/31/2002 13:55  11/3/2006 19:41  1483264  134.153.184.164:80  10/31/2002 13:55  11/4/2006 8:42  1469768  www.jstor.org:80  10/31/2002 15:28  11/4/2006 21:48  803051  www3.interscience.wiley.com:80  10/31/2002 15:28  11/4/2006 21:48  515526  online.statref.com:80  10/31/2002 14:39  11/4/2006 21:47  425362  serials.abc‐clio.com:80  10/31/2002 15:08  11/4/2006 20:01  387078  muse.jhu.edu:80  10/31/2002 15:28  11/4/2006 19:20  237479  www.csa.com:80  10/31/2002 13:55  11/4/2006 21:24  148247    table 1. ezproxy hosts report. from ezproxy administration interface      remote patrons: measuring proxy server  use    a proxy server is used to authenticate off‐ site users prior to their access to licensed  electronic products. the proxy server logs  hold a host of information about remote use  of  e‐books, e‐journals, and e‐index sites.  because proxy servers are basically web  servers, they log much of the standard data  available in web server logs, including the  urls of the remote patrons who make  requests, date/time stamps, and the urls of  the licensed resources requested.     142.162.51.235 ‐ ‐ [10/nov/2006:13:52:57 ‐ 0700] ʺget  http://www.somedb.com:80/index.html  http/1.0ʺ 200 1234    as in web logs the query portion of a url  appears in the proxy log:     &title=journal+of+transcultural+nursing& atitle=the+effect+of+social+support+and+a cculturation+on+postpartum+depression+i n+mexican+american+women        proxy server logs can also be configured to  record more or less information, depending  on the logging format chosen.  like  standard web logs, they can be run through  the same analysis tools that are used to  analyze library web site logs. proxy logs are  mined for information about the frequency  with which off‐site patrons access licensed  resources held on other servers.     some proxy server packages provide  summary reports of activity within their  administration interfaces. at memorial  university of newfoundland (mun)  libraries we have used ezproxy from useful  utilities (<http://www.ezproxy.com>) since  2002. the ‘hosts’ report within the  administration interface provides a  summary of use for each vendor server  configured in the log from the time that  entry was first made. in table 2, the date  ‘created’ represents a patron’s first access of  that server via the library proxy. ‘accessed’  refers to the last time that a remote patron  accessed that resource.     when reading the ezproxy hosts report, it is  important to remember that it is arranged  according to server name, rather than by  specific services offered by vendors. because  http://www.somedb.com:80/index.html http://www.ezproxy.com evidence based library and information practice 2007, 2:1  70 major indexes or e‐journal packages may be  hosted across multiple servers, all of the  host entries for a given domain (e.g.,  <ejournals.ebsco.com:80> and <www‐ ca.ebsco.com:80> will need to be identified.  it is therefore necessary to know which  server names and domains are registered to  a given service provider. aggregating all  hits on a particular domain may give you  more useful data about comparative use of  resources according to the vendor or  provider.    if urls to restricted resources have been  wrapped with the ezproxy starting point  url, then it is possible to use proxy logs to  generate ‘gate‐count’ information about  both local and off‐site database access.  “wrapping” a url simply means to put the  ezproxy login link at the beginning of the  url. ezproxy starting point urls take the  following form:    http://qe2a‐ proxy.mun.ca/login?url=http://restricted.re source.com/     useful utilities, the manufacturer of the  ezproxy software, offers a proxy directive  called ‘logspu’ which creates a separate log  file for starting point urls. both local and  remote users must pass through the proxy  url in order to access the destination url,  http://restricted.resource.com . when the  server administrator has configured logspu,  ezproxy creates a separate log file that  records the url of the requested resource, a  session identifier, and an indicator of  whether the requesting user ip was local or  remote. unlike the main proxy log, the spu  log does not record every subsequent hit on  the remote server, but provides a simplified  dataset which reveals how often each entry  point to a restricted resource was chosen by  a user, and indicates whether that user was  on‐site or off‐site. this is a very useful  mechanism for measuring the number of  times each index or aggregated e‐journal  package was accessed by a patron. it does  not, however, provide information about the  searches executed from within that index or  the full‐text resources selected from within  the vendor gateway.     use data provided by vendors    most database and e‐journal vendors make  usage reports available to their clients. in  order to access these reports it is necessary  to know the administrative url and login  for each separate vendor interface. this can  be a major data collection project, if the  library subscribes to many different  resources, and if several different  departments administer them.      typically, vendors will offer reports that  summarize all site activity for a selected  period of time. table 2 is a summary report  from e‐book vendor ebrary that provides  title level activity for a particular e‐book  package.    summaries for database activity are usually  also available. the example in table 3  provides a monthly summary of the total  number of searches performed compared  against each database from a particular  provider (in this case, proquest) and the  formats provided.    metasearch engines provide some  additional challenges for database use  analysis, because they initiate searches  against many products at once, driving up  the number of searches reported by  database vendors. in order to interpret the  statistics in a meaningful way, libraries will  need a way to distinguish the number of  search sessions initiated from the native  interface from those initiated from  metasearch interfaces.       http://qe2a%e2%80%90proxy%00 http://restricted.re http://restricted.resource.com evidence based library and information practice 2007, 2:1  71 title  user  sessions  pages  viewed  pages  copied  pages  printed  survival analysis  157  3992  16  1337  antigone  33  440  0  0  zen & the brain: toward an understand  of meditation & consciousness  30  315  25  27  making social science matter : why  social inquiry fails and how it can  succeed again  28  467  3  38  complete critical guide to geoffrey  chaucer  20  322  0  8  microsoft excel vba programming for  the absolute beginner  19  613  10  0  witchcraft continued : popular magic in  modern europe  16  89  0  0  fundamentals of cellular network  planning and optimisation: 2g/2. 5g/3g‐  evolution to 4g  15  138  11  0  racial theories in fascist italy  15  36  0  0  visual basic 2005 express : now playing  15  686  1  0  7 hidden reasons employees leave :  how to recognize the subtle signs and  act before itʹs too lat  14  131  5  0    table 2. ebrary title level activity statistics report. generated from ebrary administrative interface.    more granular database usage reports are  available to provide information on the  number of times specific journal titles are  viewed in each database. the following  report from proquest (table 4) also  provides information on the kinds of  document formats provided.    because librarians will not have access to the  vendor’s raw log data, and because one  doesn’t necessarily know the criteria upon  which reports are based, it can be a  challenge to interpret and compare the data.  in recent years libraries and vendors have  worked to develop a standard means to  record and exchange online usage data.     one of the most successful initiatives has  been project counter (counting online  usage of networked electronic resources).  counter has developed and published  codes of practice for the dissemination of  statistics related to electronic journal,  database, book, and reference resource use.  vendors who have implemented  counter‐compliant reports are required  to clean, sort, and organize usage data  according to the methods prescribed in the  relevant code of practice. libraries using  counter‐compliant vendor reports can be  assured that data is generated in a reliable  and consistent way across different vendor  services and products. there are a number  of counter‐compliant reports, and a  vendor may implement one or many,  depending on the services offered. the  available usage reports include:    journal report 1: number of  successful full‐text article  requests by month and journal  evidence based library and information practice 2007, 2:1  72   database  journal name  citation  abstract  text  total  cbca education  physical & health education  journal  33  8  274  315  proquest historical  newspapers nyt  new york times  (1857‐ current file)  196  185  0  381  cbca current  events  canadian press newswire  7  2  420  429  dissertations &  theses: a&i  proquest dissertations and  theses  14  315  0  329  cbca reference  the canadian nurse  11  10  106  127  cbca reference  canadian journal of public  health  25  4  33  62  cbca education  education canada  8  18  134  160  cbca current  events  canwest news  2  5  202  209  cbca education  canadian journal of  education  15  10  132  157  cbca current  events  the globe and mail  175  0  0  175  cbca business  canadian hr reporter  13  20  58  91  cbca education  alberta journal of  educational research  2  9  119  130    table 4. derived from document usage by database/journal report. proquest administrative interface. journal report 2: turnaways by  month and journal   book report 1: number of  successful title requests by month  and title   book report 2: number of  successful section requests by  month and title   database report 1: total searches  and sessions by month and  database   database report 2: turnaways by  month and database   database report 3: total searches  and sessions by month and service   total searches run by month and service a  full list of counter usage reports,  descriptions, and examples are available in  the counter code of practice for journals  and databases:  <http://www.projectcounter.org/r2/count er_cop_release_2.pdf> and in the  counter code of practice for books and  reference works:                 <http://www.projectcounter.org/cop/books/c op_books_ref.pdf>.  counter provides a standard set of  methods and reports to measure the use of  electronic resources, but it does not help to  solve the problem that vendor‐delivered  reports are scattered all over the web in a  variety of password‐protected interfaces. it  is time‐consuming to collect and aggregate  reports from different vendors, even when  http://www.projectcounter.org/r2/count http://www.projectcounter.org/cop/books/c evidence based library and information practice 2007, 2:1  73 comparable data is available. enter sushi,  niso’s standardized usage statistics  harvesting initiative.     sushi is a protocol that automates the  retrieval of vendor‐generated use reports.  the protocol is designed to support the  transfer of counter‐compliant reports. it  allows libraries to schedule programs that  automatically harvest statistical reports from  vendor web sites on a regular basis. sushi  delivers usage reports as xml documents  structured according to the counter xml  schema. an application that understands  how to interpret and parse counter xml  is needed to display these reports in a user‐ friendly fashion. this could be a custom xsl  style sheet written by a programmer, or it  could be part of a commercial electronic  resource management system (erms).    many libraries have implemented ermss to  keep track of electronic subscriptions, terms  of use, vendor contact information,  administration interface urls, passwords,  and usage data. verde, from ex libris,  (<http://www.exlibrisgroup.com/verde.htm>)  is an example of an erms that includes  support for the implementation of sushi to  automatically gather usage reports from  many vendors. this is an ideal model, as it  not only streamlines the process of  gathering data, but it also integrates  electronic resource usage information into  the larger management system without a  need for staff intervention or re‐keying.     link resolver data    link resolvers are poised to provide a  critical piece of the data puzzle for libraries.  like all online resources, link resolvers run  on web server technology and generate  standard web logs of transaction  information. link resolver logs have several  advantages over other library data sources.  firstly, the link resolver provides usage  information that cuts across many different  resources and services offered within the  library’s digital environment. secondly, link  resolver hits indicate that a patron has  found a record for a resource that interests  him sufficiently to request the full text.  thirdly, link resolvers provide usage  information at the article‐ or book‐chapter‐ level because of the granularity of the data  contained within openurl requests.  openurl provides a standard way to  encode a full article‐level citation into the  query portion of a url. web logs are  capable of recording the query section from  each url (the portion following the ?).  following is an example of an openurl  query:    http://mun‐resolver.asin‐ risa.ca:8080/?genre=article&isbn=&issn=027 8193x&title=working+mother&volume=29 &issue=8&date=20061001&atitle=building+ the+new+stereotype.&aulast=gunn%2c+eil een+p.&spage=92&sid=ebsco:academic+ search+premier     it encodes the full article citation:    author: gunn, eileen p.,   title: building the new  stereotype.  journal title: working  mother   issn: 0278‐193x  volume: 29   issue: 8   date:  2006‐10‐01   pages: 92    citation information is valuable for serials  collection analysis. it can identify heavily  used electronic journal titles as well as those  rarely accessed. the citation information can  also help to identify journal titles requested  from indexes, but not owned by the library.  this is possible because the resolver records  every request it receives, regardless of  whether or not the resource is available in  the library. most link resolvers will provide  http://www.exlibrisgroup.com/verde.htm http://mun%e2%80%90resolver.asin%e2%80%90risa%00 evidence based library and information practice 2007, 2:1  74 title  count  science  465  journal of advanced nursing  383  nursing times  332  nature  262  nursing science quarterly  208  nursing  204  pediatrics  202  journal of nursing education  197  new england journal of medicine  187  journal of college student development  184  canadian journal of zoology  180    table 5. top n requests by journal title report. generated from sirsidynix single search administration  interface    issn  title  count  no issn    65898  0954‐7762  nursing times  323  0730‐3084  journal of physical education, recreation and dance  277  0309‐2402  journal of advanced nursing  272  0002‐936x  american journal of nursing  206  0008‐4263  canadian journal of public health  195  0036‐8075  science (washington d c)  192  0362‐4331  new york times  188  0419‐4209  dissertation abstracts international section a:  humanities and social sciences  182  0148‐4834  journal of nursing education  173  0008‐4301  canadian journal of zoology  172  0028‐0836  nature  170    table 6. top n requests by issn report. generated from sirsidynix resolver administration interface.    online reporting tools for commonly sought  metrics, including the number of requests  for a specific journal title (table 5) or issn  (table 6) within a given time period.    openurl links also provide information  about the referring index. in the openurl  example above, the user used ebsco’s  academic search premier database to locate  the referenced article citation. referrer  identities specify which indexes generate  frequent requests for full‐text material. in  addition to title‐level reports, standard  reports are also usually available that tally  requests according to the index used.     the data collected in link resolver logs has  obvious implications for local collection  development, as it contains a great deal of  information about e‐index, e‐book, and e‐ journal usage.  some have explored the use  of link resolver data to create new indicators  of scholarly quality at a global level. the bx  project, sponsored jointly by ex libris  (developers of the widely used sfx link  resolver) and the los alamos national  laboratory, seeks to aggregate link resolver   evidence based library and information practice 2007, 2:1  75 rfr_ids request counter:    memorial university of newfoundland  referrer  count  epnet.com:cinahl  9560  ncbi.nlm.nih.gov:  4598  epnet.com:academic search premier  4093  epnet.com:psycinfo  2865  hwwilson.com:omnift  2711  silverplatter.com:bxcd  1962  abc‐clio.com:xdb  1893  pqil:mla  1667  csa.com:socioabs‐set‐c  1438  epnet.com:eric  1328  abc‐clio.com:ahl  913  abc‐clio.com:ha  820  google.com:  806      table 7. top n databases referring users to link resolvers. generated from sirsidynix resolver  administration interface.    data from many institutions and subject it to  data‐mining techniques. the goals of the  project are to reveal large‐scale trends in  scholarly communication, to provide new  measures of the impact and prestige of  publications, and to provide insight into use  patterns by diverse scholarly communities  (bollen et. al, 2005).    link resolvers and overlap analysis    the back‐end of every link resolver is a  large vendor‐maintained knowledge base  that maps periodical holdings to each of the  full‐text indexes and e‐journal packages  containing that particular journal title. we  can see, for instance, all of the full‐text  journals available within academic search  premiere, and the years for which those  titles are available. as well, we can see every  package that offers full or partial access to a  specific journal title. this information can be  leveraged to compare the holdings and  coverage of two or more databases or e‐ journal packages being considered for  purchase. cufts, a link resolver developed  at simon fraser university library for the  council of prairie and pacific university  libraries (coppul), makes a holdings  comparison tool freely available over the  internet. the cufts product allows  comparisons of up to four aggregated  resource packages and identifies those  journal holdings that may be duplicated  across the packages. a development version  of cufts can be found at  <http://cufts2.lib.sfu.ca/mainttool/public/co mpare>.    it is also possible to perform collection  overlap analysis through the use of the  library’s link resolver. this is because link  resolvers not only track the partial or full  title runs available within each aggregated  resource, but they also contain information  about each of the packages to which the  library subscribes.’ the sfx resolver from  ex libris ‐  <http://www.exlibrisgroup.com/sfx.htm> ‐  offers overlap analysis tools for subscribers,  as does serial solutions’ article linker ‐  http://cufts2.lib.sfu.ca/mainttool/public/co http://www.exlibrisgroup.com/sfx.htm evidence based library and information practice 2007, 2:1  76 <http://www.serialssolutions.com/overlap.a sp.>.     the library can also take advantage of  serials solutions’ overlap analysis tools if  it  subscribes to the e‐journal full marc  records service, whether or not article  linker has also been implemented. overlap  analysis reports provide data on the  duplication of titles in subscribed packages  and help determine which packages may be  de‐selected.    another important feature of these tools is  the ability to evaluate new aggregated  packages being considered for purchase.  the overlap analysis tool makes it possible  to compare the holdings of any available  package with those to which the library has  current subscriptions. it is able to generate  reports that indicate the percentage of  partial and full overlap with current  electronic subscriptions, as well as the  percentage of unique holdings within the  package under consideration. it is possible  to drill down to the title‐level to see which  titles are completely unique and which will  be duplicated in full or in part.     federated search    federated search (or metasearch) tools have  been adopted by many libraries as part of  their resource discovery arsenals. some  popular metasearch tools for libraries  include endeavor’s encompass, webfeat’s  prism, and ex libris’ metalib. all of these  tools send simultaneous searches to multiple  distributed database targets and return  results to the user in a single, consistent  results interface. these tools provide single  search access to groups of databases  clustered around specific subjects or  resource types, and they may also offer  sorting and de‐duping functions.     metasearch applications generate two types  of logs: web server logs and metasearch  application logs. both may hold valuable  information about the ways in which the  tools are used. web server logs, as we know,  can provide information about the times and  dates of searches, user ips, the number and  length of sessions, the types of queries, and  the items that are viewed. the metasearch  application’s internal logs may also contain  some or all of this information, but are more  likely to organize data collection around  metrics specific to the metasearch process:    • number of targets searched  simultaneously  • number of simultaneous sessions   • speed with which results are  returned from different targets  • user profile information from the  authentication database.    most metasearch engines provide some kind  of reporting interface, although the types of  reports available vary from product to  product and depend on the way the  application logs have been configured.  webfeat’s smart reporting interface,  <http://www.webfeat.org/products/smart.ht m>, offers one of the most full‐featured  reporting tools available for metasearch  applications. smart is counter‐ compliant, and provides database activity  reports that include searches conducted  from the federated search interface, as well  as those executed in the native interfaces of  the databases. information available from  metasearch logs includes information on  user sessions, search activity, and database  activity:    user session information:   • number of successful and failed  logins   • session length   • user ip address  • browser and operating system  • average number of simultaneous  active sessions (by date and time)  • number of session timeouts  http://www.serialssolutions.com/overlap.a http://www.webfeat.org/products/smart.ht evidence based library and information practice 2007, 2:1  77 search activity:   • total number of searches over a  period   • number of targets searched  simultaneously per session   • search queries     database activity:   • number of searches conducted  against each specific target   • speed with which results are  returned from each database  • number of database turnaways or  failed searches   • number of hits returned from each  target database  • number of times full record requests  were made from each target  database  • number of times a ‘next set’ request  was made from a database  (database result sets are usually  truncated in metasearch result sets,  so users can request ‘next set of n  results’ from a specific database.)  • number of times users clicked  through to the native database  interface  • number of full record requests made  for a specific journal title, article title,  author, database, or issn.    institutional repositories and use data    most of the data sources discussed so far are  those that help evaluate the use of electronic  resources hosted remotely and delivered by  third‐party vendors. many libraries have  electronic collections created and stored  locally. digital archives and institutional  repositories (irs) are two ways that libraries  have adopted an expanded role as the  custodians and gatekeepers of local  electronic collections with unique or rare  materials. these collections may be the  result of large and time‐consuming  digitization efforts within the library. they  may contain theses, dissertations, pre‐prints,  presentations, and other scholarly materials  produced within the larger institution. some  of these collections provide a means for  institutions to organize, search, and access  administrative documents, annual reports,  policy papers, and the results of internal  studies. all of these local collections  represent a significant investment of time  and money, and librarians must determine  the value of these investments in terms of  increased visibility and use.     some commonly used digital repository  platforms are:    • dspace, an open source platform  originally developed by mit  libraries and hewlett‐packard  <http://www.dspace.org>  • gnu eprints, open source  repository software  <http://www.eprints.org>  • contentdm, a commercial  software package offered by oclc.  <http://www.oclc.org/contentd m/>    activity data on these systems is held in  both web server logs (for end‐user access  data) and system logs (to record transaction  information about staff workflows).      reporting tools are still under development  for each of these systems, as are metrics to  assess the use and effectiveness of digital  repositories. because search and access  information can be captured in standard  web logs, web analysis tools can be  implemented to uncover standard measures  of end‐user activity. this includes data  about the most heavily used collections,  frequently accessed objects, number of items  viewed during a typical session, activity by  time and date, search strategies, referring  urls and search engines, geographic  location of user‐based ip addresses, and file  downloads.  http://www.dspace.org http://www.eprints.org http://www.oclc.org/contentd evidence based library and information practice 2007, 2:1  78 contentdm server report  collections accessed from 2006‐09‐01  to  2006‐10‐31  collection  image  times  viewed  centre for newfoundland studies  ‐ digitized maps  a general chart of the island of  newfoundland with the rocks...  36  centre for newfoundland studies  ‐ digitized maps  tierra nueva (b) [cartographic  material].  26  centre for newfoundland studies  ‐ digitized maps  a chart of the banks and harbours of  newfoundland. ‐‐  21  maritime history archive ‐ forbes  family fonds  unidentified fraternal group standing  in front of a large …  19  maritime history archive ‐ forbes  family fonds  ʹʹaugust 1925ʹʹ  19  maritime history archive ‐  grenfell photograph collection  a grenfell mission nurse standing  outside in a garden  18  maritime history archive ‐  grenfell photograph collection  a nurse and three injured children  sitting on the front step...  18  maritime history archive ‐  international grenfell  association...  grenfell mission building with the  sign ʹʹfaith hope and l...  17  centre for newfoundland studies  ‐ digitized maps  a new map of the only useful and  frequented part of new foun...  16    table 8. collections and objects accessed report. generated from contentdm 4.0 administration interface    contentdm server report top search terms  from 2006‐06‐01 to 2006‐08‐30                  search terms                                 times searched  grenfell mission st. anthony  57  bell island  43  st anthony husky  29  tilley  24  st anthony  17  harbour grace  15  eilley  14  avila  12    table 9. top search terms report. generated from contentdm 4.0 administrative interface      the current version of dspace uses a java‐ based logging utility called log4j to generate  system activity logs as plain text files,  although add‐on tools to generate sql and  xml versions of the logs are under     discussion and development by members of  the dspace community. contentdm  administrative activity is logged in a sql  database, and custom administrative  activity reports can be created through the  evidence based library and information practice 2007, 2:1  79 use of crystal reports or other sql query  software. the contentdm 4.0  administrative interface provides canned  reports for some of the more common usage  measures, such as those in tables 6 and 7,  identifying which hits on collections /  objects and user search strategies  respectively.    collection metadata is held in the repository  database and can generate statistics about  the character of the library’s digital  collections, such as the total number of  objects, the size of each individual collection,  the number and extent of compound objects,  dates when each of those objects was  created, the number of null items (items  with no associated file), and breakdowns of  the numbers of items by file type (e.g., jpg,  pdf, tiff, mp3, or wmv).    some libraries have created custom tools in  order to perform more detailed analysis. an  example is oregon state university’s  contentdm controlled vocabulary  analyzer,   <http://oregonstate.edu/~reeset/contentd m/downloads.html>.  the tool generates  reports on controlled vocabulary usage  across all of a library’s contentdm  collections. resulting data can be used to  identify conflicting terms within a  controlled vocabulary or to understand the  scope of different collections by indicating  the frequency with which certain terms have  been applied.     end‐user searches and views are contained  in web server log data, but other logs exist  that describe staff‐side activity on the  system. this data quantifies the amount of  time documents spend at each phase of the  submission and staging process, the number  of times each document is touched by a staff  member during the process, staff login  counts by date/time and session duration,  and other workflow information that can be  analysed to identify bottlenecks affecting  document turn‐around time.  one of the interesting features of irs is that  libraries can attach administrative and  statistical information about each digital  object as a linked part of the record for that  object. in this way statistical and processing  information can become part of the  metadata for a specific object. a good  example of this can be found in the eprints  repositories of the university of melbourne  (<http://eprints.unimelb.edu.au/>) and the  university of tasmania  (<http://eprints.utas.edu.au/>). these  institutions have developed a joint program  to provide links to statistical information  from each eprints record, allowing users to  view the download statistics for that  particular item.     a second interesting attribute of irs is the  ability to store entire reports within the  digital repository system as discrete objects  with their own metadata. the digital  repository can act as the repository for log  and report information. if these logs have a  common xml format, they can also be  exposed for  harvesting by other ir systems  using a protocol developed by the open  archives initiative (oai).  by harvesting log  data from various ir systems, libraries will  be able to compare ir content and usage  data across different institutions. the  interoperable repository statistics (irs)  project, <http://irs.eprints.org/about.html>,  is an initiative that seeks to promote data  sharing through the use of oai harvesting.     electronic reference services    there are several different approaches to  electronic reference services:    • e‐mail/web‐based asynchronous  reference  • chat/synchronous reference  through the use of instant  http://oregonstate.edu/~reeset/contentd http://eprints.unimelb.edu.au/ http://eprints.utas.edu.au/ http://irs.eprints.org/about.html evidence based library and information practice 2007, 2:1  80 messaging clients, e.g., aim, msn  messenger, and meebo  • full‐featured co‐browsing systems,  e.g., oclc’s questionpoint or ask  a librarian from tutor.com    the data available from virtual reference  software depends on the type of service and  the data the library requests from users  before allowing them to initiate a virtual  reference session. identifiers for patron and  responder are available in transcripts, logs,  or e‐mail headers. these may take the form  of a login name, an e‐mail address, or a chat  display name. because reference chat logs  and transcripts often contain personally  identifying data, librarians need to strip this  kind of information before beginning data  analysis. randomizing programs can be  written to change meaningful strings such  as e‐mail addresses to meaningless series of  characters, even while preserving a unique  identifier for each string. in this way repeat  visits can be tracked without exposing a  patron’s identity. through the collection of  other information, (e. g., patron‐level,  location, and institutional affiliation), patron  types can be correlated with query types  and subjects.    date/time information is also usually  available. for e‐mail‐based reference, this  data would be included in the message  header; in chat services the date, hour,  minute, and second of each line in the  correspondence is logged in the session  transcript. librarians can use this data to  determine the frequency of electronic  reference queries, and the distribution of  queries over the days of the week and hours  of the day. date/time stamps can also be  used to evaluate turn‐around times for  electronic reference services.     all types of electronic reference products  have the capability of saving a complete  transcript of the conversation between the  librarian and the patron. while email and  virtual reference software keep a transcript  automatically,  most instant messenger (im)  clients have to be specifically configured to  log session transcripts. librarians may use  transcripts to evaluate the type of questions  received (e.g., hours, services, known‐item  queries, technical help, or research help), the  types of sources to which users have been  referred (e.g., web pages, indexes, guides,  or reference sources), as well as a subject  breakdown of research questions received  (e.g., business, engineering, or medicine).      because these transcripts are manifest as  unstructured natural language questions  and answers, human mediation is usually  required to evaluate and characterize the  interactions they record. qualitative analysis  software such as atlas.ti,  <http://www.atlasti.com/>, or qsr nvivo,  <http://www.qsrinternational.com/products/ productoverview/nvivo_7.htm>, is  designed to analyse ‘soft’ data (i.e.,  unstructured text files) that lack a controlled  vocabulary. this type of software can be  useful to librarians who wish to undertake  in‐depth analyses of large numbers of  virtual reference transcripts.     full‐featured virtual reference applications  also offer the option for patron exit surveys  at the end of each session, where user  feedback can be solicited regarding the  usefulness of the answer received and the  software’s ease of use.     the digital reference electronic warehouse  (drew), under development by scott  nicholson and r. david lankes from the  university of syracuse, is a project that aims  to create an archive of cross‐institutional  digital reference transactions. drew  architects are developing an xml schema to  represent transaction information from e‐ mail, im, and commercial virtual reference  products. by bringing many similarly  structured transactions together into a single  space, nicholson and lankes hope to  http://www.atlasti.com/ http://www.qsrinternational.com/products/ evidence based library and information practice 2007, 2:1  81 produce a rich dataset to which online  analytical processing (olap) and data‐ mining tools can be applied to “discover  patterns in users, experts, question metadata,  and works referred to by experts in the  answer” (lankes and nicholson, 2005). the  purpose of drew is to provide a  collaborative research space, management  information system, and toolset, which can  be used to understand and improve digital  reference services. by amalgamating virtual  reference data from many different libraries,  drew’s developers aim to provide a tool  that allows us to better understand the  character of electronic reference and to tailor  our services more closely to patron needs.     ils data sources     the integrated library system (ils) holds a  great deal of rich and detailed data about  many of the library’s key operations.  information about funds, vendors, spending  trends, number of items catalogued within  specific call‐number ranges, number of  items added to the collection each month,  turn‐around times for item processing, and  a great deal of other administrative data is  held in the ils database. ils logs are even  more useful, as they contain details about  every transaction that has ever transpired in  the system, whether or not that data exists  in the current production database. a  patron or vendor record may have been  removed from the database, but the  existence of that account and all of the  activity attached to it will still be contained  in the server history logs.     the ils circulation module links to patron  records that may contain:    • zip or postal code  • primary branch affiliation  • departmental affiliation  • major area of study  • status as undergraduate, graduate,  faculty, or alumni  • date of birth    this data from the ils circulation records  allows the study of patron activity in a  much more granular fashion than might be  possible in web server logs, which  frequently record no identifying information  about a user.     ils logs, like most other types of logs, are  held in dense, largely inaccessible text files  on the ils server. the raw log data is  encoded through the use of proprietary  vendor data codes, unlike web logs that  follow internationally recognized standards  for encoding data. this means that a  vendor‐provided translation tool or  datacode manual will be necessary in order  to make sense of the log data. the following  is a sample of log data from the mun  libraries’ sirsidynix unicorn ils:    e200510280830240058r^s36ivfflgodda rd^fcnone^feqeii^iqxx(1727366.1)^ni 31^nz33^dalt^nd32^nh245^ng10^nea rchâeologie du vin et de lʹhuile dans  lʹempire romain /|cjean‐pierre  brun.^om^^o  e200510280830250058r^s37ivfflgodda rd^fcnone^feqeii^iqxx(1727366.1)^ni 32^nz34^dalt^nd33^nh260^ng^nepari s :|berrance,|cc2004.^om^^o  e200312121523590020i ^@38hdiqt 174.5  c64 1983^i4t 174.5 c64  1983^is1^nq31162004594801^nsqeii^ilc heckedout^ltnqcharged^^z  e200312121523590020f  ^5nrn^jxn^cklisa^cllisaʹs test  account^ca12/12/2003,15:23^ci10/4/2004,23: 59^cx4mur$2^cgcheckedout^iacol lingridge, david.^ibtechnology in the  policy process : controlling nuclear power /  david collingridge. ‐‐^^o  e200312121523590020r^s39znffsirsi^fc none^feqeii^mi1^mo2^mlitem_check ed_out,due^^o  e200312121523590020f  ^@39znmi1^mqenglish^mpy^mm002 evidence based library and information practice 2007, 2:1  82 c~0 item_checked_out||item checked  out|due||due|^^o  there is an ethical concern about accessing  raw ils log files. as evident in the example  above, the circulation record in the raw log  file contains elements that disclose the  identity of individuals and the particular  items they borrow. because librarians are  concerned about privacy issues, many ils  vendors provide tools to strip logs of patron  barcodes and other data that might allow  the identification of specific users while  retaining broader profile information for the  purpose of statistical analysis. ethical  concerns dictate that librarians undertaking  ils log analyses should ensure that user ids,  barcodes, and personal information are  removed from the raw data before making  logs available for study of any kind. this is  even more critical if the logs are to be  duplicated in more than one location or  shared with several people for the purpose  of analysis.      beyond ils history logs there are web logs,  a second‐level of logging for the web‐based  opac interface. web logs, like all of the  other web server logs discussed here, record  the query portion of searches executed and  may also indicate the frequency with which  patrons choose to browse or keyword search  a particular field.     198.165.141.237 ‐ ‐ [31/may/2003:10:23:19 ‐ 0230] ʺget  /uhtbin/cgisirsi/x/0/5?user_id=remotewe b&searchdata1=remote+sensing+of+sea+i ce http/1.1ʺ 200 69189    134.153.164.143 ‐ ‐ [31/may/2003:10:46:19 ‐ 0230] ʺget  /uhtbin/cgisirsi/x/0/5?user_id=remotewe b&searchdata1=lazarus+and+folkman  http/1.1ʺ 200 28582    142.163.11.5 ‐ ‐ [31/may/2003:11:10:30 ‐0230]  ʺget  /uhtbin/cgisirsi/tyqtircyga/127540009/8/10 4295/american+journal+of+diseas es+of+children http/1.1ʺ 200 29928  this information can be used to analyse the  catalogue search behaviour of patrons.  because these are standard web server logs,  commercial or freely‐available web log  analysis software can draw out information  about dates and times of heavy opac usage  and help to isolate and cluster search  strategies.      the ils has a number of canned reports that  can be used to gather data about the number  of classroom reserves that are created, items  catalogued, items circulated, and other  baseline activity. ils reporting systems are  useful to gather aggregated data, such as the  number of av material circulations that  occurred in a given month, but they are not  optimised to allow detailed examination of  that data. the limitations of ils reporting  tools have made the extraction and  manipulation of the huge data stores  contained in ils logs very difficult.    a number of strategies have emerged in  recent years to help librarians access and use  ils data in new ways. one approach has  been the development of new reporting  tools, modelled on the kinds of business  intelligence systems widely adopted in the  corporate world. olap reporting tools, such  as director’s station from sirsidynix,  provide multi‐dimensional views of library  activity. these tools allow librarians to find  relationships between activities that occur in  different library units such as selection,  acquisitions, cataloguing, and circulation.  these tools are intended to help librarians  identify use patterns and trends over time.  olap reporting tools allow librarians to  view aggregated activity data and then drill  down into that data for more specific  information, adding and removing different  dimensions to create custom views of the  data. other features include the ability to  perform custom calculations on the fly,  create natural language queries (“which  evidence based library and information practice 2007, 2:1  83 hour of the day and day of the week had the  highest circulation?”), sort data within and  across hierarchies, export report data to  spreadsheets for further manipulation, and  create alerts for certain thresholds of activity,  e.g. create an alert when the total amount of  fines owed at my branch reaches $1000.     the kinds of questions that can be answered  easily using an olap reporting tool  includes a wide range of topics:    • what is the average age of my  collection of chemistry books?   • what is the distribution of  publication dates in that collection?   • what is the average cost per‐item in  my collection of chemistry books?   • what is the distribution of  chemistry book purchasing across  vendors and funds?   • how many chemistry books with  publication dates before 1995  circulated this year?   • what has been the spending trend  over the last ten years for chemistry  books?   • which are the top one hundred  circulating chemistry books over the  past three years?   • what percentage of my chemistry  collection circulated to  undergraduates, graduates, or  faculty members over the past five  years?   • what is the distribution of those  borrowers across university  departments?   • how does circulation per‐item in  the chemistry collection compare  with that of the general collection?     ils data & cross‐library comparison  tools     the tools identified above are those that  help evaluate collection use in a single  library or library system. vendors are also  developing tools, such as the normative  data project (ndp) or oclc’s worldmap  that allow ils data to be integrated with gis  programs for demographic analysis. oclc  has several data mining projects that  attempt to leverage its worldcat data for  cross‐library comparison. these include the  systemwide print book collection study, the  comparative collection assessment project,  and the oclc mining for digital resources  study    the normative data project (ndp),  <http://www.libraryndp.info/>, combines  gis‐based demographic information with  ils data to aid library planners and  decision‐makers. this project combines  transaction and bibliographic data extracted  from the ils logs of a number of different  libraries, with census data, and u.s. library  data from the national center for education  statistics.  the ndp project seeks to extract  and normalize ils data from many different  vendor platforms to provide a standard  view of data for cross‐library data sharing  and comparison. understanding which titles,  formats, and languages are in heavy  demand by different categories of patrons in  comparable libraries can help predict which  resources will be used by your own patrons.     oclc’s gis‐based information system,  worldmap, seeks to gather key statistical  information from libraries around the world,  including holdings and titles by place of  publication, types of libraries, number of  libraries, librarians, users, volumes, and  annual expenditures. worldmap provides  an intuitive graphical entry point for  comparing titles and holdings at national  and regional levels. it also permits regional  comparisons on staffing, expenditures, and  accessibility of library services.   oclc’s worldcat database contains more  than 65 million bibliographic records  representing over 950 million holdings from  http://www.libraryndp.info/ evidence based library and information practice 2007, 2:1  84 more than 20 million libraries around the  world. this creates an invaluable data  source by which libraries can evaluate their  own collections in a larger context, and  opens up opportunities for cross‐library  collection analysis, and cooperative regional  collection development. oclc’s worldcat  collection analysis service allows libraries  to compare their own holdings against those  of peer‐group libraries to identify subject‐ matter strengths, gaps, and overlaps.   oclc has several other current ils data‐ mining projects. the systemwide print book  collection study analyses aggregate print  book holdings from worldcat to identify  rare or unique materials in individual  library collections, as well as titles that are  widely available within a particular region.  this information can help libraries target  titles that are widely available in other  collections and therefore easily procured  through inter‐library loan. oclc’s  comparative collection assessment project  incorporates holdings information with ill  usage information to help libraries identify  collection strategies for print and e‐book  collections. the oclc mining for digital  resources study aims to analyse holdings  patterns, material types, and cataloguing  activity for digital resources across library  collections to identify trends and best  practices for collecting and organizing  digital materials.   conclusion    this article has focused on data sources that  are likely to be immediately available to the  typical practitioner who wishes to engage in  statistical analysis of collections and services  within his or her own library. there is an  enormous amount of quantitative data  available to support evidence based studies,  and a surprising amount of that data is  either collected within the library, or readily  accessible through standard reporting tools  offered by service and resource providers.  this data can help to identify problems with  existing interfaces and tools, to identify and  analyze search strategies across different  types of resources, to evaluate the frequency  and type of use of existing services and tools,  to evaluate service turn‐around times, to  identify high‐demand resources, to identify  infrequently used products and services, to  identify new services and resources which  may be in demand, to identify the ebb and  flow of demand by month, day or hour, to  discover more about the behaviour of  specific patron groups, to analyze  information requests according to subject  groupings, and to make cross‐branch or  cross‐library comparisons on all of those  measures.     beyond the need to locate and access data  sources, a number of further challenges face  libraries in the area of quantitative analysis.  libraries require well‐developed standards  for reporting use data to allow for more  accurate comparisons across similar vendor  products, services and resources. libraries  need to aggregate data that currently tends  to be available in silos according to specific  applications or providers. we must combine  different kinds of use data into common  spaces that will allow us to find correlations  between evidence from a number of sources,  and to identify trends that cut across  different services and applications. it is  fairly easy to extract data to track a single  patron’s behaviour during a single session  and across a single type of resource or  service. it is extremely difficult to get a  holistic view of how a single patron  interacted with all of the possible resources  during a single information‐seeking session.  that patron may have started at the library’s  web site, conducted an opac search, linked  to the metasearch interface from within the  opac, linked to a record in a vendor’s  native interface via the proxy server, from  there clicked the resolver button to find local  holdings for an article of interest, and finally  evidence based library and information practice 2007, 2:1  85 viewed the full‐text of the resource on the  site of an e‐journal vendor. ideally, we will  develop tools to track, analyse and correlate  all of this activity in a single data space (a  data farm or data warehouse) that contains  views of log data from all of the available  products and services. the scope of work is  even more daunting when one imagines  trying to create a standardised warehouse  for data sources across a number of different  libraries. lastly, of course, metrics and key  performance indicators have not been  clearly defined for many of our electronic  resources, so even with accessible data, it is  difficult to know exactly how to best  measure certain activities, and what  inferences confidently be made.     resources for log analysis in libraries    proxy servers:     ezproxy discussion list. 2000‐2006. state  university of new york. 14 nov. 2006.  <http://ls.suny.edu/read/search/results? forum=ezproxy&words=logspu>.     zagar, chris. “ezproxy support: log  format.” 2006. useful utilities. 14 nov.  2006.   <http://www.ezproxy.com/support/cfg /logformat/>.     use data from vendors     counter. counting online usage of  networked electronic resources. 2006.  counter. 17 nov. 2006  <http://www.projectcounter.org/>     ex libris. “ex libris verde erm system  successfully tests sushi to deliver  raw usage statistics,” library  technology guides. 23 nov. 2005. 27  feb. 2007  <http://www.librarytechnology.org/ltg‐ displaytext.pl?rc=11684>.     hiott, judith. “collecting and using  networked statistics: current status,  future goals,” library quarterly 74.4  (oct. 2004): 441‐54.     nicholas, david, paul huntington, hamid r.  jamali, and anthony watkinson. “the  information seeking behaviour of the  users of digital scholarly journals.”  information processing & management  42.5 (2006): 1345‐65.      niso standardized usage statistics  harvesting initiative (sushi). 2005.  niso. 17 nov. 2006  <http://www.niso.org/committees/sus hi/sushi_comm.html>.     link resolvers:     bollen, johan, oren beit‐arie, and herbert  van de sompel. “the bx project:  federating and mining usage logs  from linking servers.” cni fall 2005,  5‐6 dec. 2005, phoenix, az. 27 feb.  2007 <http://library.lanl.gov/cgi‐ bin/getfile?la‐ur‐05‐9439.pdf>.     cufts overlap analysis tool. 27 feb.  2007     <http://cufts2.lib.sfu.ca/mainttool/publ ic/compare>.     federated search:     chandler, sarah, and nan hyland.  “federated searching and academic  libraries: one size fits all?” 226th  american chemical society (acs)  national meeting, new york city. 8  sept. 2003. 27 feb.  2007  <http://hdl.handle.net/1813/3040>.     webfeat. smart. 27 feb.  2007  <http://www.webfeat.org/products/sm art.htm>.     webfeat. smart: statistical measures  available real time. april 2005. 27 feb.  http://ls.suny.edu/read/search/results? http://www.ezproxy.com/support/cfg http://www.projectcounter.org/ http://www.librarytechnology.org/ltg%e2%80%90displaytext%00 http://www.niso.org/committees/sus http://library.lanl.gov/cgi%e2%80%90bin/ http://cufts2.lib.sfu.ca/mainttool/publ http://hdl.handle.net/1813/3040 http://www.webfeat.org/products/sm evidence based library and information practice 2007, 2:1  86 2007   <http://www.carli.illinois.edu/uploads/ file/wf/webfeat‐smart‐detail.pdf>.     institutional repositories:     dimema. contentdm online help,  version 4. “reports.” 2006. 27 feb. 2007  <http://www.dimema.com/help4/serve r‐admin/reports.html>.     dspace federation. dspace wiki:  dspacestatisics. 16 nov. 2006. 27 feb.  2007  <http://wiki.dspace.org/index.php/dspacest atistics>.     frost, jim, and henry pisciotta. penn state  university libraries. “visual image  user study (vius): vius reports  8.7‐ authentication logs.” oct. 2003. 27  feb. 2007  <http://www.libraries.psu.edu/vius/8.7. pdf>.     organ, michael. “download statistics ‐  what do they tell us? the example of      research online, the open access  institutional repository at the  university of wollongong, australia,”  d‐lib magazine 12.11 (2006). 27 feb.  2007 <http://www.dlib.org/dlib/novem ber06/organ/11organ.html>.     reese, terry. oregon state university  contentdm development  projects/resource page: oregon state  university contentdm controlled  vocabulary analyzer (aug. 2006). 27  feb. 2007  <http://oregonstate.edu/~reeset/cont entdm/downloads.html>.     sale, arthur, and christian mcgee.  tasmania statistics software  download  (2006). 27 feb. 2007  <http://eprints.utas.edu.au/262/>.  university of southampton, uk, and key   perspectives ltd., uk. “irs:  interoperable repository statistics.” 3  mar. 2005. 27 feb. 2007  <http://irs.eprints.org/about.html>.    electronic reference services:     lankes, david l. and nicholson, scott.  “creating the infrastructure for  collaboration between digital  reference services and researchers:  the digital reference electronic  warehouse (drew) project,” jcdl ’05,  7‐11 june 2005, denver, co.  <http://portal.acm.org/citation.cfm?id= 1065497&dl=acm&coll=&cfid=151515 15&cftoken=6184618>  marsteller, matt, and paul neuhaus. “the  chat reference experience at carnegie  mellon university.” poster  presentation. annual meeting,  american library association. 16 june  2001. san francisco, ca. 27 feb. 2007  <http://www.contrib.andrew.cmu.edu/ ~matthewm/ala_2001_chat.html>.  ils data sources:     blecic, deborah d., nirmala s. bangalore,  josephine l. dorsch, cynthia l.  henderson, melissa h. koenig, and  ann c. weller. “using transaction log  analysis to improve opac retrieval  results,” college & research libraries  59.1(jan.1998): 39‐50.      cooper, michael d. “usage patterns of a  web‐based library catalog,” journal  of the american society for  information science and technology  52.2 (jan. 2001): 137‐48.   cummins, caroline. “below the surface,”  library journal net connect 15 jan.  2006. 27 feb. 2007  http://www.carli.illinois.edu/uploads/ http://www.dimema.com/help4/serve http://wiki.dspace.org/index.php/dspacest http://www.libraries.psu.edu/vius/8.7 http://www.dlib.org/dlib/novem http://oregonstate.edu/~reeset/cont http://eprints.utas.edu.au/262/ http://irs.eprints.org/about.html http://portal.acm.org/citation.cfm?id= http://www.contrib.andrew.cmu.edu/ evidence based library and information practice 2007, 2:1  87 <http://www.libraryjournal.com/article /ca6298559.html>.  hamilton, scott, and helen thurlow.  “transaction log analysis @ state  library of queensland,” 3rd  international evidence based  librarianship conference. brisbane,  queensland, 17 oct. 2005. 27 feb. 2007  <http://conferences.alia.org.au/ebl2005/ hamilton.pdf>.  lavoie, brian, and roger schonfeld. “a  systemwide view of library  collections.” oclc online computer  library center and ithaka. cni spring  task force meeting. 5 april 2005. 27  feb. 2007  <http://www.oclc.org/research/present ations/lavoie/cni2005.ppt>.  lavoie, brian, lorcan dempsey, and lynn  silipigni connaway. “making data  work harder,” library journal net  connect. 15 jan. 2006. 27 feb. 2007  <http://www.libraryjournal.com/article /ca6298444.html>.   nicholson, scott. “proof in the pattern,”  library journal net connect 131(jan.  2006): 2‐6. 27 feb. 2007  <http://www.libraryjournal.com/article /ca6298562.html>.     normative data project for libraries  (ndp).  27 feb. 2007    <http://www.libraryndp.info/>.                         oclc. data mining research area. 2007. 27  feb. 2007  <http://www.oclc.org/research/projects /mining/>.  tlc. the library corporation. “press  release: tlc partners with civic  technologies, inc.” 4 may 2004. 27 feb.  2007  <http://www.tlcdelivers.com/tlc/press/ pr050404.asp>.    general resources for quantitative  analysis:    association of research libraries.  association of research libraries  statistics and measurement  department. mines for libraries:  measuring the impact of networked  electronic services. 2006.   <http://www.minesforlibraries.org/min es/index.cfm>.  borgman, christine l., and sandra g. hirsh.  “rethinking online monitoring  methods for information retrieval  systems: from search product to  search process,” journal of the  american society for information  science 47.7 (jul. 1996): 568‐83.     jansen, bernard j. “search log analysis:  what is it, whatʹs been done, how to  do it,” library & information science  research 28.3 (june 2006): 407‐32.     nicholson, scott. “approaching  librarianship from the data: using  bibliomining for evidence‐based  librarianship,” library hi‐tech  24.3(2006): 369‐75.      nicholson, scott. “the basis for  bibliomining: frameworks for bringing  together usage‐based data mining  and bibliometrics through data  warehousing in digital library  services,” information processing and  management 42.3 (may 2006): 785–804.     zucca, joe. “traces in the clickstream: early  work on a management information  repository at the university of  http://www.libraryjournal.com/article http://conferences.alia.org.au/ebl2005/ http://www.oclc.org/research/present http://www.libraryjournal.com/article http://www.libraryjournal.com/article http://www.libraryndp.info/ http://www.oclc.org/research/projects http://www.tlcdelivers.com/tlc/press/ http://www.minesforlibraries.org/min evidence based library and information practice 2007, 2:1  88 pennsylvania.” information  technology and libraries 22 .4(dec.  2003): 175‐9.    software    article linker c. 2003. serial solutions.  seattle, wa. feb 28, 2007.  <http://www.serialssolutions.com/over lap.asp>    ask a librarian 2.2 c. 2007 tutor.com. new  york, ny. feb 28, 2007.  <http://www.tutor.com/products/aal.as px>    atlas.ti 6 c.2007. berlin, germany. feb. 28,   2007 <http://www.atlasti.com/>    contentdm 4 c. 2006. oclc. dublin, oh.  feb. 28, 2007.  <http://www.oclc.org/contentdm/>    cufts 2 c. 2007. council of prairie and  pacific university libraries (coppul).  victoria, bc. feb. 28, 2007.  <http://cufts2.lib.sfu.ca/mainttool/publ ic/compare>.    director’s station 3.8 c. 2006.  sirsidynix.  hunstville, al. feb. 28, 2007.  <http://www.sirsidynix.com/solutions/ products/analytical.php>    dspace 1.4 c. 2006. originally developed by  mit libraries and hewlett‐packard.  released under the gnu general  public license. feb. 28, 2007.  <http://www.dspace.org>    ezproxy 4.0 c. 2006. useful utilities. peoria,  az. feb 28, 2007.  <http://www.ezproxy.com>    eprints 3 c. 2006.  lead developer,  christopher gutteridge. released  under the gnu general public  license.feb. 28, 2007.  <http://www.eprints.org>    metalib 4 c. 2007. ex libris group. des  plaines, il. feb. 28, 2007.  <http://www.exlibrisgroup.com/metali b.htm>    webfeat 3 c. 2005. old brookville, ny. feb.  28, 2007.  <http://www.webfeat.org/products/pri sm.htm>    nvivo 7 c. 2006.  qsr international pty ltd.  australia. feb. 23, 2007.  <http://www.qsrinternational.com/pro ducts/productoverview/nvivo_7.htm>,    questionpoint c. 2006. oclc. dublin, oh.  feb. 28, 2007.  <http://www.oclc.org/questionpoint/de fault.htm>    sfx 3 c. 2005. ex libris group. des plaines,  il. feb. 28, 2007.  <http://www.exlibrisgroup.com/sfx.ht m>     verde 2.0 c. 2006. ex libris group. des  plaines, il. feb. 28, 2007.  <http://www.exlibrisgroup.com/verde. htm>    http://www.serialssolutions.com/over http://www.tutor.com/products/aal.as http://www.atlasti.com/ http://www.oclc.org/contentdm/ http://cufts2.lib.sfu.ca/mainttool/publ http://www.sirsidynix.com/solutions/ http://www.dspace.org http://www.ezproxy.com http://www.eprints.org http://www.exlibrisgroup.com/metali http://www.webfeat.org/products/pri http://www.qsrinternational.com/pro http://www.oclc.org/questionpoint/de http://www.exlibrisgroup.com/sfx.ht http://www.exlibrisgroup.com/verde evidence summary   librarian-led information literacy training delivered in small groups improved medical students’ confidence in their ability to use evidence based resources effectively   a review of: mcclurg, c., powelson, s., lang, e., aghajafari, f., & edworthy, s. (2015). evaluating effectiveness of small group information literacy instruction for undergraduate medical education students using a preand post-survey study design. health information & libraries journal. 32(2), 120-130. http://doi.org/10.1111/hir.12098   reviewed by: elizabeth stovold information specialist, cochrane airways group st george’s, university of london tooting, london, united kingdom email: estovold@sgul.ac.uk   received: 19 nov. 2015 accepted: 18 jan. 2016      2016 stovold. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to assess the effectiveness of librarian-led small group information literacy sessions, which were integrated into the second year of a three-year undergraduate medical course. design – a preand post-intervention survey questionnaire. setting – a large university in canada. subjects – a cohort of 160 second year undergraduate medical students enrolled in the three-year programme of a large university in canada. methods – as part of the redevelopment of the undergraduate three-year medical course, information literacy skills in evidence based medicine were integrated into the seminar and small group teaching programme. every week for five weeks, 3 librarians each visited 4 small groups of 15 students to deliver a 15-minute session as part of a 2-hour long seminar led by practising physicians. the sessions did not include a formal hands-on component, however, students were encouraged to try out searches on their own devices. each 15-minute session covered 3 learning objectives, including how to use pubmed clinical queries, how to use mesh, and how to search for systematic reviews and guidelines.   a preand post-intervention survey design was used to assess students’ perceptions of the impact of these sessions. the students were asked to complete an online survey monkey survey before and after the five week lecture block. the questions covered resource selection, perception of barriers to finding evidence based information, and the students’ confidence in using evidence based resources. the data were analysed descriptively. main results – the pre-survey achieved a 90% (144/160) response rate while the post-survey achieved a 75% (112/160) response rate. the post-survey indicated an increase in the likelihood that students would use ovid medline, carry out a literature search, and consult a librarian, with a decrease in those who would consult a print or online textbook. there was limited change in the students’ confidence that they could find answers quickly, but more of an increase in the proportion of students who were confident they could find systematic reviews and guidelines, and use search limits, pico, and mesh. before the intervention, “knowing where to search,” devising a search strategy, and retrieving too many results were all thought to be obstacles by the students. after the small group training, students considered these issues less of a problem.   the post-survey also included an opportunity for the students to comment on their experience with the programme overall. of the 54 responses received, 34 identified the library component as being the most important thing they had learned in the small group part of the course. conclusion – the authors conclude that integrating information literacy into the undergraduate curriculum as part of the small group seminar series is effective. they suggest future directions for research, such as a study to assess the impact of the training on specific skills rather than student confidence and evaluations of other teaching methods.   commentary   teaching and training are key elements of the role of a librarian working in the health sector (sen, chapman, & villa, 2014). this study of an alternative method to a traditional library-based session for delivering an information literacy program to medical students was appraised using the reliant tool (koufogiannakis, booth, & brettle, 2006). the study’s strength is its rationale and reporting. the objective was explicitly stated and the study population was clearly defined. each small group contained the same number of participants and they each received the same number of training sessions with a faculty-supported librarian. the mode of delivery was described as engagement and discussion. the learning objectives of each training session were presented and were the same for each group. the authors used a preand post-intervention survey design, which was appropriate to assess the short term outcomes that were measured. the survey questions were based on previously published work, and the full questionnaire was appended to the report to allow for reproducibility.   results were described narratively and presented graphically as percentages to show the preand post-intervention comparison. when describing the results, the authors used language such as “significant change” and “significant increase,” although no statistical significance tests were reported. the study aimed to assess the effectiveness of the intervention, however it was the students’ confidence in their ability to find and use resources that was measured, rather than any quantifiable measure of the effect of this method on skills. the authors of the paper recognise this as a limitation, as well as noting that this is a three-year course therefore the results may not be generalizable to a four-year medical program.   the study does not to attempt to compare the integrated teaching method with a different teaching model or control group, as ilic, tepper, and misso (2012) did in their randomised controlled trial comparing the effect of a formal literature searching workshop with a control group who did not attend a workshop. as there is no control and the outcomes measured are subjective rather than an objective assessment of information literacy skills, the results should be interpreted with caution. however, the integrated method did appear to be successful in improving the students’ confidence with using evidence based resources, so other institutions may wish to incorporate this approach into their own programmes as an alternative to the traditional stand-alone library teaching session. they should evaluate the effect of the intervention on students’ skills, and directly compare it with other methods.   references   ilic, d., tepper, k., & misso, m. (2012). teaching evidence-based medicine literature searching skills to medical students during the clinical years: a randomized controlled trial. journal of the medical library association : jmla, 100(3), 190–196. http://doi.org/10.3163/1536-5050.100.3.009   koufogiannakis, d., booth, a., & brettle, a. (2006). reliant: reader’s guide to the literature on interventions addressing the need of education and training. library and information research, 30(94), 44-51. retrieved from http://www.lirgjournal.org.uk/lir/ojs/index.php/lir/article/view/271   sen, b. a., chapman, e., & villa, r. (2014). working in the health information profession: perspectives, experiences and trends: the results of an eahil-funded 25th anniversary project. in: proceedings of the 14th eahil 2014 conference, rome, italy. retrieved from http://eprints.whiterose.ac.uk/id/eprint/78521   evidence based library and information practice 2011, 6.4 22 evidence based library and information practice commentary is there a future for evidence based library and information practice? andrew booth reader in evidence based information practice school of health and related research (scharr) university of sheffield sheffield, united kingdom email: a.booth@sheffield.ac.uk received: 10 nov. 2011 accepted: 10 nov. 2011 2011 booth. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. the above title may seem a pessimistic note on which to open the concluding address for a conference on evidence based library and information practice, particularly one that has been as stimulating, thought‐provoking, and successful as the eblip6 in salford, uk. in defence of my jeremiad i can offer the following exhortation, albeit one offered in a slightly different context. “being willing to put everything we have back into the 'central pot' (i.e. everything is up for evaluation) . . . is actually the first ‐ and arguably the most important ‐ step of eblip service” (booth, 2009a). so following my own mantra i am going to question each element embodied in the phrase, “the future of evidence based library & information practice.” the library is dead, long live the librarian first target for the iconoclast is the concept of “the library.” while the “l word” has been a useful coat‐hanger, upon which to communicate to the external world what exactly librarians do, i am not alone in feeling that the inherent disadvantages of this label now outweigh the advantages. in short, it constrains rather than elucidates. focusing our question using the spice mnemonic (booth, 2006) we must recognise that our interventions, to range against the comparator of “standard library service,” may involve locations or services positioned outside the library. indeed, in our increasingly virtual world, our interventions, programmes, and services may involve no physical location at all! furthermore, any ongoing dialogue regarding eblip and its mailto:a.booth@sheffield.ac.uk� evidence based library and information practice 2011, 6.4 23 future needs to access evidence informed, or even derived, from far beyond the library walls. finally the context, or setting to continue with the spice terminology, for such a dialogue must not be within the self‐absorbed arena of the library and its staff. instead it must involve the wider, and distinctly less comfortable environs of the organisation and its objectives as a whole. in short the information function, regardless of label, must be able to survive amid the cross‐ currents of organisational and environmental change. evidence based ? having already pruned our starting phrase to “the future of evidence based library & information practice” it is timely to consider what exactly we mean by evidence based. for many within the profession this expression feels far too deterministic and mechanistic for what actually occurs in practice. intuitively “evidence based practice” feels like it belongs in arenas, such as medicine, where the quality and consequent certainty of the evidence is much more robust than in our own social science domain. additionally, as i commented following the eblip5 conference in stockholm, the eblip model seems to be much more dependent on persuasion and negotiation with colleagues (booth, 2009b) than the unilateral action of an autonomous clinician. similar reservations hold for the alternative expression “evidence led ” which almost seems to invite rebellion. i do have some sympathy for the ideas behind evidence supported or evidence informed practice. however the latter loses the sense of imperative too much, appearing optional and even wishy‐washy – as in “thank you for informing me how much the public park means to you. now i am going to concrete it into a parking lot.” personally i find it difficult to conceive of a movement that will be able to sustain its impetus behind the rallying banner of “informed” – but then it may just be that i am ill‐informed! a further complication arises from the recent popularity of the formulation “practice based evidence.” while inverting the ebp concept in such manner may provide a refreshingly new perspective it is certainly not a position that one can adopt for very long ‐ not, that is, without causing a sudden rush of blood to the head! whilst venturing into this territory can i take the opportunity to stick the knife into my personal non‐favourite, “evidence based research”? can there ever be such a thing as “non‐evidence based research” – apart from, that is, bad information or knowledge? research. and isn’t research one form of evidence anyway? – in which case are we not in danger of creating the mobius strip of “evidence based evidence” and consequently disappearing within our own fundamentals? having arrived at “the future of evidence <something> based library & information practice” we can consider whether “information” is in fact the commodity in which we should be dealing. increasingly, given the societal information explosion and individual information overload, emphasis is re‐centering on adding value. information, like data, is seen as a raw material most organisations are interested in the refined product. in short, our skills of drilling for previously uncharted information, hidden beneath seemingly impermeable surfaces, are less in demand than the ability to turn this into an endless variety of knowledge products. in trading in the “futures” market our attention as a profession should focus on “knowledge” in preference to “information.” one advantage of such a switch comes in providing access to the useful, although, in my mind, overused concepts of tacit and explicit knowledge. such concepts, arguably, bring us closest to the intent behind the conference theme of “valuing knowledge and expertise” (koufogiannakis, 2011a). evidence based library and information practice 2011, 6.4 24 practice makes perfect? having reduced our original phrase to “the future of evidence <something> based library & information practice” we turn our attention to one of the foremost challenges of the whole “evidence <something>” movement as articulated by the statement of the medical library association: "individual . . . librarians must apply the results of research routinely to library and information service practice, to the development of information policy, and to other information issues important to . . . institutions” [italics added] (medical library association, 1995). do we yet see much evidence of the routine adoption of evidence into our professional practice? yes, the eblip conferences admirably showcase and profile exciting initiatives, pilot projects, and stimulating ideas. but how does our own version of the movement perform in terms of sustainability and spread? in short, how many examples exist where the influence of these initiatives extends beyond awareness ‐ listening to presentations at conferences or reading articles in the eblip journal? clearly the road to practice, as opposed to ideas and hypotheses, is a long way from being well‐ travelled!: “the future of evidence based library & <something> information practice” what is evidence? one of the distinctive contributions of the spread of evidence based practice across multiple domains and sectors is that each domain has shaped and moulded its own conception of what is meant by “evidence.” so, for example, progression of the movement into the domain of social care and social work has resulted in enhanced recognition of the importance of client or user perspectives. while wide‐encompassing and culturally sensitive conceptions of evidence are much to be welcomed we have to recognise that an undesirable consequence of this tendency has been the distillation of the significance of the term. if, as in alice in wonderland, we can choose to make evidence mean whatever we want it to mean then literally everything becomes “evidence” (koufogiannakis, 2011b). from here it is not too far to a point where nothing becomes evidence – in fact we start to wheel huge quantities of evidence around in wheelbarrows, like some devalued currency not even worth a loaf of bread. as a consequence we now hurl “evidence” unceremoniously out into the streets: “the future of evidence based library & <something> information practice” what of “the future”? listening to at least two plenary keynotes from the eblip6 conference, particularly those from individuals based within the academic sector, one could be forgiven for wondering whether libraries will have any future at all! a serious point from such an unwelcome conclusion is that other library sectors, including academic, public and health, may well have to learn lessons from the school library sector where eblip is less a strategy for continuous quality improvement and more a survival strategy – a tactic for fighting your corner and defending your territory in times of economic constraint and genuine recession. we can only speculate on the future of eblip amidst the double dip. “the future of evidence based library & <something> information practice” those of you who are still following this brutal “concept amputation” will have realised that with the above‐heralded demise of “the future” we are now at the minimalist position of being left with the word “of.” now admittedly the word “of” has made a significant contribution to popular culture – the bodies of both fiction and non‐fiction would no doubt be immeasurably the poorer for its absence. one need only think of “of mice and men” or “the joy of sex” for example. however, forgive me if i am being evidence based library and information practice 2011, 6.4 25 unduly pessimistic, but i cannot conceive of the word “of” as able to sustain an international movement! towards an alternative? if the darkest hour is indeed before dawn then now is a timely point within my commentary to introduce a concept that offers a realistic alternative to evidence based practice – namely, knowledge interaction! knowledge interaction is a recently proposed and real‐life variant of knowledge translation (kt). over recent years kt itself has been assuming increasing importance and use in public health, medicine, and health services research (straus et al., 2009). kt is a relatively new term used to describe a relatively old problem – the underutilization of evidence based research (there, my bete noir phrase, again!), often described as a gap between "what is known" and "what is currently done" in practice settings (booth, 2011; lavis et al., 2003). as with evidence based practice, the popularity of knowledge translation originates from canada. the canadian institutes for health research (cihr) define kt as “the exchange, synthesis, and ethically‐sound application of knowledge – within a complex set of interactions among researchers and users – to accelerate the capture of the benefits of research . . . through . . . more effective services and products . . .” (cihr, 2004). rather than, i hope, representing a cynical attempt by the aforementioned country to perpetuate a lumber industry that has already benefited economically from the widespread deforestation associated with evidence based practice, kt differs from the traditional process of diffusion. it is primarily an active and manipulated process that involves "all steps between the creation of new knowledge and its application and use to yield beneficial outcomes for society"(cihr, 2004, p. 4). why a variant of knowledge translation? compelling reasons exist to recommend knowledge interaction as a variant of knowledge translation. • it doesn’t rely on rcts • it doesn’t even rely on research • it focuses on what needs to be done and on how to do it • it can be applied to any intervention/programme/practice • within knowledge translation there is a considerable evidence base around what works, and • knowledge translation is the information specialist’s core business nevertheless, despite its apparent novelty, the term knowledge translation has already been superseded because, along with knowledge transfer, they are seen to “misrepresent the tasks that they seek to support. by articulating the complex and contested nature of applied social research, and then highlighting the social and contextual complexities of its use, we can see that other terms may serve us better” (davies et al., 2008). the same authors therefore recommend the term “knowledge interaction” suggesting that “'knowledge interaction' might more appropriately describe the messy engagement of multiple players with diverse sources of knowledge . . .” (davies et al., 2008). herein lies the principal reason for my support, namely that this very term captures the intent behind the revised model of eblip that i first suggested after the eblip5 conference – messy, iterative, and multiplayer. not that i would dare suggest that eblip6 in salford will be the last conference to bear that particular label for, as the authors recognise: “while it might be hard to shift the terminology of knowledge transfer in the short term, awareness of its shortcomings can enhance understanding about how social research can have wider impacts” (davies et al., 2008). evidence based library and information practice 2011, 6.4 26 in conclusion for the reasons outlined above knowledge interaction conveys more accurately my personal conception of what should be signified by evidence based library and information practice. in short, evidence based practice (and indeed knowledge translation) suggests an orderliness that it is neither present, feasible, nor desirable. in contrast knowledge interaction best reflects our messy real world and conveys genuine partnership. nevertheless i should conclude my “expert” commentary by evoking the warning from david sackett, a founding father of evidence based medicine, that we should “never trust experts”. indeed rather than spend the intervening years between subsequent eblip conferences trying to hammer home the dogmatic, and essentially gloomy, pronouncements afforded me in both my closing address and this associated commentary, i plan to take to heart sackett’s own caution: is redemption possible for the sins of expertness? the only one i know that works requires the systematic retirement of experts . . . surely a lot more people could retire from their fields and turn their intelligence, imagination, and methodological acumen to new problem areas where, having shed most of their prestige and with no prior personal pronouncements to defend, they could enjoy the liberty to argue new evidence and ideas on the latter's merits. (sackett, 2000) acknowledgement this paper is based on a closing address given at the 6th international evidence based library and information practice conference (eblip6), held at the university of salford, june 27‐30, 2011. references booth, a. (2006). clear and present questions: formulating questions for evidence based practice. library hi tech, 24(3), 355‐368. doi:10.1108/07378830610692127 booth, a. (2009a). eleven steps to eblip service. health information & libraries journal., 26(1), 81‐84. doi:10.1111/j.1471‐ 1842.2008.00836.x booth, a. (2009b). eblip five‐point‐zero: towards a collaborative model of evidence‐based practice. health information & libraries journal., 26(4), 341‐344. doi:10.1111/j.1471‐ 1842.2009.00867.x booth, a. (2011). barriers and facilitators to evidence‐based library and information practice: an international perspective. perspectives in international librarianship,1 doi:10.5339/pil.2011.1 canadian institutes for health research. (2004). knowledge translation strategy 2004‐ 2009: innovation in action. in canadian institutes of health research. retrieved 21 nov. 2011 from http://www.cihr‐ irsc.gc.ca/e/documents/kt_strategy_2004 ‐2009_e.pdf davies h., nutley s., & walter i. (2008). why 'knowledge transfer' is misconceived for applied social research. journal of health services research & policy, 13(3), 188‐190. doi:10.1258/jhsrp.2008.008055 koufogiannakis, d. (2011a). should we value knowledge and expertise? evidence based library and information practice, 6(3), 1‐2. retrieved 21 nov. 2011 from http://ejournals.library.ualberta.ca/index .php/eblip/article/view/11163 evidence based library and information practice 2011, 6.4 27 koufogiannakis, d. (2011b). what is evidence? evidence based library and information practice, 6(2), 1‐3. retrieved 215 nov. 2011 from http://ejournals.library.ualberta.ca/index .php/eblip/article/view/10245 lavis, j. n., robertson, d., woodside, j. m., mcleod, c. b., & abelson, j. (2003). how can research organizations more effectively transfer research knowledge to decision makers? milbank quarterly, 81(2), 221—248. medical library association. (1995). using scientific evidence to improve information practice: the research policy statement of the medical library association. chicago: medical library association research task force. sackett, d. l. (2000). the sins of expertness and a proposal for redemption. bmj, 320 (7244):1283. doi:10.1136/bmj.320.7244.1283 straus s., tetroe j. m, & graham i. d., editors (2009). knowledge translation in health care: moving from evidence to practice. london: wiley‐blackwell. / evidence based library and information practice is there a future for evidence based library and information practice? the library is dead, long live the librarian evidence ubasedu? information or knowledge? what is evidence? towards an alternative? why a variant of knowledge translation? in conclusion acknowledgement references evidence based library and information practice using evidence in practice   gathering evidence for sustainable development goals: an alignment perspective   paula ochôa integrated researcher and assistant professor centro de humanidades (cham-nova fcsh) universidade nova de lisboa lisbon, portugal email: paulatelo@fcsh.unl.pt   leonor gaspar pinto integrated research and invited assistant professor centro de humanidades (cham-nova fcsh) universidade nova de lisboa lisbon, portugal email: lgpinto@sapo.pt   received: 1 sept. 2019                                                                    accepted: 11 nov. 2019      2020 ochôa and pinto. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29638 setting   in september 2015, the united nations (un) member states adopted the 2030 agenda for sustainable development and set the global, national, and local framework for putting that responsibility into action (united nations, 2015). the new global agenda includes 17 sustainable development goals (sdgs) that must be met by all countries by 2030 so that "no one will be left behind" (united nations, 2015, p. 6). the 169 associated targets are monitored and reviewed using a set of global   indicators (sustainable development solutions network, 2015).   sectorial organizations have worked to bring their potential contributions toward sustainable development to the attention of member states and the un. in the post-2015 process, the international federation of library associations and institutions (ifla) committed to promoting libraries within the the 2030 agenda for sustainable development as a key initiative under the strategic directions set for 2016-2021 (ifla, 2015). access to information has been recognized as a target under sdg16 (target 16.10), culture and information and communication technology have also been included in the sdgs (target 11.4 and targets 5b, 9c, 17.8, respectively), and universal literacy is recognized in the vision for the 2030 agenda for sustainable development (ifla, 2018).   problem   in portugal, initiatives to promote library and information services within the 2030 agenda are still modest. research intersecting information science with performance evaluation and sustainable development is not yet sufficiently developed (ochôa & pinto, 2019).   since 2012, researchers at the centre for the humanities (cham) in the faculty of social sciences and humanities of universidade nova de lisboa have been pursuing a line of research that intersects performance evaluation of library and information services (lis) with sustainability, transition management, and competence development. in 2013-2014, this intersection was put into practice through in co-creationworkshops directed at students of lis courses that focus on building a sustainability assessment framework. these interdisciplinary experiences led to workshop participants outlining a draft of a conceptual framework for assessing the sustainability and impacts of lis. the conceptual framework was further consolidated (ochôa & pinto, 2014) and expanded by introducing the evaluation perspectives provided by: 1) iso 16439 standard (2014), namely impact evaluation areas/focus and methods for collecting evidence of impact; 2) media and information literacy indicators (ochôa & pinto, 2015).   in november 2016, a team of researchers at cham were willing to think globally but act locally looked for an answer to the question how can public libraries gather evidence and evaluate their contribution to the sdgs? the researchers started an action research project named bibliotecas públicas e sustentabilidade: recolha de evidências da contribuição para os ods (public libraries and sustainability: gathering evidence of contribution to sdgs, aka project pls). project pls aims to develop a framework (model) for evaluating public libraries’ contribution to the sdgs, and to tailor the model to portuguese public libraries (pinto & ochôa, 2018a).   evidence   evaluation research and evidence based theory provided the general framework for project pls. the research question was addressed through an qualitative methodology supported by the combination of two principal methods: 1) literature review and 2) construction and application of conceptual models as well as tools for analysis. the literature review process was complemented by a questionnaire that analyzed portuguese public libraries’ involvement in sdgs international, national, and local initiatives.   an integrated and holistic approach was used to co-create the model for the alignment and evidence gathering of libraries’ contribution to sustainable development (see figure 1). for the model, the concept of evidence can be any type of information or data used to help answer a question. the information or data collected becomes evidence when it is used to demonstrate a library’s contribution(s) to sustainable development, when it is used to determine goal or target achievement, when it is used to show the alignment initiatives adopted, or when it is used to understand various impacts on stakeholders (bill & melinda gates foundation, 2015).   in the alignment process, the use and/or adaptation of library assessment data is valued using any or a combination of the following: ·         un global indicators ·         portuguese national indicators (whenever possible) that correspond to the un global indicators ·         iso 11620 – library performance indicators ·         iso 16439 – methods and procedures for assessing the impact of libraries   any of these may be complemented by the global libraries impact planning and assessment guide (bill & melinda gates foundation, 2015) or the generic learning outcomes and generic social outcomes (arts council england).                                                           figure 1 model for the alignment and evidence gathering of libraries’ contribution to sustainable development (pinto & ochôa, 2018).   implementation   the cornerstone of the model is a roadmap that is anchored in the identification of potential (inter)relations between the dimensions that frame the impacts of library activities or projects and the sdgs. the roadmap supports the alignment cycles in four steps:   1.       step 1 – preplanning and learning. this step creates the conditions for the implementation of the process that allows libraries to obtain evidence of their contribution to drive forward the un 2030 agenda. library and information professionals need to develop basic competences (e.g., to understand the architecture and main concepts of the model). 2.       step 2 – planning in the face of alignment. based on the information collected in step 1, decisions must be made that will determine what evidence will come out of the alignment process. this step requires several actions such as defining smart indicators and choosing adequate methods to gather data or evidence. evidence can be structured according to five levels. level 1, the most basic, requires only a clear description of the project or activity and its importance to the community and sdgs. in level 2, libraries gather data that shows some positive change (alignment), but they may not be able to confirm what caused the modification. at level 3, libraries demonstrate causality between results and the alignment dimensions. in level 4, evidence is confirmed by independent evaluators, and a determination is made as to whether the project shows potential for replication in other libraries. finally, level 5, the highest level, implies the recognition of libraries’ contributions to sdg by national and international entities. 3.       step 3 – implementing. the plans from prior steps need to be put into practice. data is gathered and analyzed, and then the information that will be communicated is prepared. 4.       step 4 – using evidence to advocate for libraries. this step focuses on the effective use of a narrative tool that values the use of anecdotal evidence for impact assessment obtained from users, staff, and other stakeholders. the sdgs story must provide answers to three questions: why? who did what, how, and when? so what?   there have been several experiences of implementation focused on developing sharing skills through participation at libraries’ workshops, at academic and professional conferences, at academic classes, and through promotional incentives, such as national awards.   outcomes   the model for the alignment and evidence gathering of libraries’ contribution to sustainable development was consolidated and expanded within another project. under ifla’s international advocacy programme, the portuguese association of librarians, archivists and documentalists implemented the project libraries for development and the un 2030 agenda (april to july 2018). to accelerate library and information professionals’ involvement in the systematic process of evaluating and promoting libraries’ contribution to sdgs, a step-by-step guide for libraries (pinto & ochôa, 2018) was built on this model and made available through the website of the portuguese association of librarians, archivists and documentalists.   one of the model’s tools, the roadmap for the alignment and evidence gathering of libraries’ contributions to the 2030 agenda, was tested in two workshops targeted at portuguese public libraries professionals in july 2018. it also provided the framework for creating a national award in october 2018 related to best practices in sustainable development in libraries of all types.   one of the most lasting and consequential effects of the model may be the impact it is having on lis student engagement, especially among those who use it most as a tool to reach out and identify new assessment profiles and roles in libraries. the ability to extend evidence based librarianship into assessment competencies demands the exploration of new possibilities for understanding and sharing within curricular content.   reflection   the relationships between the 2030 agenda for sustainable development and public libraries’ outcomes can be relevant at different points to evidence based librarianship. one of the early difficulties in exploring the phenomenon of strategic alignment lies in the ability to capture that dimension in a meaningful way. social aspects, such as immediate personal outcomes, health, and well-being, have been acknowledged as important traits for acquiring new understanding and developing new knowledge through the model of implementation.   further, evidence alignment can be significantly associated with other relevant assessment constructs and tools as a management tool and as an agenda for joint research with several stakeholders. in a future research agenda, it is our belief that epistemic communities will develop transdisciplinary and evaluative practices. in this scenario, co-evaluation models will have a pivotal role. different alignment strategies and evidence will also be valued to deal with the complexity of new open evaluation processes.   the implementation in various contexts of the model for the alignment and evidence gathering of libraries’ contribution to sustainable development fosters a set of evidence that values strategic alignment as well as impactful evaluation practices and competencies that can be used toward libraries’ contribution to the un 2030 agenda. the roadmap proved to be particularly useful in developing library and information students’ and professionals’ alignment and assessment competencies. it also supports libraries’ advocacy efforts toward their recognition as active and valuable partners in the development and implementation of national, regional, and local strategies that will help deliver on the sdgs.   references   arts council england. generic learning outcomes. retrieved 10 june 2019 from https://www.artscouncil.org.uk/sites/default/files/s2d12_detailed_framework.pdf arts council england (2016). generic social outcomes: indicator bank for museums, libraries and archives. retrieved 10 june 2019 from https://www.artscouncil.org.uk/sites/default/files/s3d29_gso_indicator_bank.pdf bill & melinda gates foundation (2015). global libraries impact planning and assessment guide. http://www.publiclibraryadvocacy.org/wpcontent/uploads/2015/06/ipa-guide-2015.pdf ifla (2015). ifla strategic plan 2016-2021. https://www.ifla.org/files/assets/hq/gb/strategic-plan/2016-2021.pdf ifla (2018). libraries, development and the united nations 2030 agenda. https://www.ifla.org/libraries-development iso 11620 (2014). information and documentation: library performance indicators. geneva: international organization for standardization. iso 16439 (2014). information and documentation: methods and procedures for assessing the impact of libraries. geneva: international organization for standardization. ochôa, p., & pinto, l.g (2014). sustainability metrics in library and information services: a quality management framework. in proceedings of the 2014 conference, helsinki, finland. second plenary session (2014, june 3). retrieved 30 may 2019 from http://docs.lib.purdue.edu/iatul/2014/plenaries/5 ochôa, p., & pinto, l.g. (2015). sustentabilidade e medição de impactos em organizações: o papel dosindicadores de literacia mediática, comunicação e cidadania. in s. pereira, & m. toscano (eds.), literacia, media e cidadania. livro de atas do 3. congresso, (pp. 408-420). braga: cecs. retrieved from http://www.lasics.uminho.pt/ojs/index.php/cecs_ebooks/article/viewfile/2252/2169 ochôa, p., & pinto, l.g. (2019). agenda 2030 e o campo de intervenção da ciência de informação: dinâmicas de aprendizagem, envolvimento e desenvolvimento de competências. edicic 2019. paper presented at edicic, barcelona. retrieved from http://eprints.rclis.org/38546/ pinto, l.g., & ochôa, p. (2018). alinhamento e evidências do contributo para o desenvolvimento sustentável: guia para bibliotecas. june 2018. retrieved from https://www.researchgate.net/publication/326581158_alinhamento_e_evidencias_do_contributo_para_o_desenvolvimento_sustentavel_guia_para_bibliotecas pinto, l. g., & ochôa, p. (2018a). public libraries' contribution to sustainable development goals: gathering evidences and evaluating practices. in p. hauke, m. charney, & h. sahavirta (eds.), going green: implementing sustainable strategies in libraries around the world (pp.46-59). berlin/boston: de gruyer. sustainable development solutions network (2015). indicators and a monitoring framework for the sustainable development goals: launching a data revolution for the sdgs. https://irp-cdn.multiscreensite.com/be6d1d56/files/uploaded/150612-final-sdsn-indicator-report1   united nations (2015). transforming our world: the 2030 agenda for sustainable development. https://sustainabledevelopment.un.org/content/documents/21252030%20agenda%20for%20sustainable%20development%20web.pdf news   library service & social wellbeing data release    2020. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29778     how do people describe their ideal community? in what ways do public library directors create a sense of belonging through service?   qualitative data, as well as complete transcripts from over two hundred community residents interviewed in eight remote rural us communities are now available for use by researchers, policy makers, and librarians.   the field data were collected as part of the rural library service and social wellbeing project (https://rurallibraries.org/). this three-year research and application work is funded, in part, by the institute of museum and library services and is led by the southern tier library system (ny) in partnership with pioneer library system (ny), new mexico state library (nm), association of rural and small libraries, and reforma. through mixed methods research conducted by public library workers, the project answers:   ·        are public libraries a component of social wellbeing in resource poor geographies? ·        if so, what practices do they employ?   public library practitioner/researchers margo gustina, hope decker, and eli guinnee worked with local library directors in remote rural localities who had a public library in town, but no other formal community anchor (including schools, major medical facilities, or large-scale employer), to set up interviews with community residents. through 114 interviews, the team was able to record conversations with hundreds of people on their feelings about and experiences of community.   these transcripts were coded and analyzed and in concert with survey and statistical data inform libraries build pathways to wellbeing findings and associated evidence-based resources. the research team has continued to work with local library partners to translate these findings into evidence-based resources for public library practice which leads to positive social wellbeing outcomes. these resources are concurrently under development and testing for usability and relevance.   more information on the rural library service & social wellbeing project, findings, resources, and open data for use can be found on our project site: https://rurallibraries.org, through our project registration on open science framework, or by emailing principal investigator margo gustina: research@rurallibraries.org.   use the data! make new discoveries!     microsoft word news2_1811 evidence based library and information practice 2008, 3:2 80 evidence based library and information practice news the 2007 acrl environmental scan: a lirg seminar © 2008. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. this exciting seminar, organized by the library and information research group (lirg), is an exploration of some of the major assumptions shaping practice in academic library and information services based on the environmental scan published by the ala's association of college and research libraries (acrl) research committee in 2008. the seminar features a panel of uk-based expert speakers sheila cannell, (director of library and collections (university of edinburgh), chris powis (director of academic services university of northampton) and gary horrocks (deputy director, customer services king's college london) who will each give their reaction to the 2007 acrl scan. then there will be a planned video conference to involve the chair of the acrl. a general discussion will follow. for the scan itself see <http://www.acrl.org/ala/acrl/acrlpubs/whit epapers/environmental_scan_2.pdf>. as a pre-cursor to the seminar, lirg will hold its agm, with presentations of the lirg research award and the student prize for 2008. the 2007 lirg elsevier research award winner jacqueline chelin (university of the west of england) will also talk about her research on "e-book usage". the seminar/agm will run from 1:00-4:00 on 29 july 2008 at the council room, strand campus, king’s college london. attendance costs £30 (free of charge to members of lirg). to book a place at the seminar/agm please contact: alan poulter hon. lirg secretary/policy forum representative department of computer and information sciences university of strathclyde livingstone tower, 26 richmond street glasgow g1 1xh e-mail: alan.poulter@cis.strath.ac.uk telephone: 0141 548 3911 evidence based library and information practice 2012, 7.3 98 evidence based library and information practice review article evidence based medicine teaching in undergraduate medical education: a literature review misa mi medical librarian oakland university william beaumont school of medicine library rochester, michigan, united states of america email: mi@oakland.edu received: 22 apr. 2012 accepted: 21 june 2012 2012 mi. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objectives – to determine the year when evidence based medicine (ebm) was introduced and the extent to which medical students were exposed to ebm in undergraduate medical education and to investigate how ebm interventions were designed, developed, implemented, and evaluated in the medical curriculum. methods – a qualitative review of the literature on ebm interventions was conducted to synthesize results of studies published from january 1997 to december 2011. a comprehensive search was performed on pubmed, cinahl, web of science, cochrane library, proquest dissertations & theses, psycinfo, and eric. articles were selected if the studies involved some form of quantitative and qualitative research design. articles were excluded if they studied ebm interventions in medical schools outside the united states or if they examined ebm interventions for allied health profession education or at the levels of graduate medical education and continuing medical education. thirteen studies which met the selection criteria were identified and reviewed. information was abstracted including study design, year and setting of ebm intervention, instructional method, instruction delivery format, outcome measured, and evaluation method. results – ebm was introduced to preclinical years in three studies, integrated into clinical clerkship rotations in primary care settings in eight studies, and spanned preclinical and clinical curricula in two studies. the duration of ebm interventions differed, ranging from a workshop of three student contact hours to a curriculum of mailto:mi@oakland.edu evidence based library and information practice 2012, 7.3 99 30 student contact hours. five studies incorporated interactive and clinically integrated teaching and learning activities to support student learning. diverse research designs, ebm interventions, and evaluation methods resulted in heterogeneity in results across the 13 studies. conclusions – the review reveals wide variations in duration of ebm interventions, instructional methods, delivery formats for ebm instruction, implementation of an ebm intervention, outcomes measured, and evaluation methods, all of which remain relevant issues for further research. it is important for medical educators and health sciences librarians to attend to these issues in designing and delivering a successful ebm intervention in the undergraduate medical curriculum. introduction evidence based medicine (ebm) is healthcare practice based on integrating knowledge gained from the best available research evidence, clinical expertise, and patients’ values and circumstances (dickersin, straus, & bero, 2007). evidence based practice emphasizes new skills that learners must acquire and use: question formation, search and retrieval of the best available evidence, critical appraisal of the evidence, and application of the evidence to an individual or a group of patients (finkel, brown, gerber, & supino, 2003; montori & guyatt, 2008). ebm serves as a powerful educational tool or strategy that allows students and clinicians to become lifelong self-directed learners and information masters able to fill their knowledge gaps and practice high quality medicine (barnett, smith, & swartz, 1999; bordley, fagan, & theige, 1997). ebm as an approach to practicing medicine has spawned widespread attention from medical educators who consider ebm as an important content area to be addressed in the undergraduate medical education curriculum. the association of american medical colleges (aamc) established the medical school graduation questionnaire (gq), a national questionnaire, for medical schools to use in program evaluation and to improve the medical student experience (association of american medical colleges, 2012). the questionnaire contains specific items asking graduating medical students to self-report the level of adequacy in ebm instruction in their medical education curriculum. the liaison committee on medical education (lcme) that accredits medical education programs in the united states and canada considers gq data as part of important evidence demonstrative of the educational quality of new and established medical educational programs (melnyk, 2006). the lcme also includes education standards for ebm related skills being appropriately addressed in a medical education program (liaison committee on medical education, 2012). to develop and implement innovations to medical schools across the united states, the undergraduate medical education for the 21st century project identified the practice of ebm and population-based medicine as one of nine key content areas to be integrated into medical school curricula in the clinical years (o'connell & pascoe, 2004). at the level of graduate medical education, the accreditation body for residency programs in the united states – the accreditation council for graduate medical education (acgme) – endorsed six core competencies for residents in residency programs, one of which is the competency in practice based learning and improvement (steward, 2001). the competency requirement stipulates that residents demonstrate the ability to locate, appraise, and apply evidence from scientific studies to their patient health problem. to better prepare medical students for beginning a medical residency program, a medical education program is expected to incorporate ebm into its curriculum to teach students basic skills in clinical decision making and application of evidence based information to medical practice. evidence based library and information practice 2012, 7.3 100 as consumers of health/medical information, physicians are inundated with diverse and abundant information resources with variable quality and clinical relevance. they face a formidable challenge of staying current with biomedical knowledge and applying the growing knowledge to specific patients. “the skills needed to find potentially relevant studies quickly and reliably, to separate the wheat from the chaff, and to apply sound research findings to patient care have today become as essential as skills with a stethoscope” (glasziou, burls, & gilbert, 2008, p. 704). given the health care environment that is changing constantly and becoming increasingly sensitive to optimizing the quality and cost of patient care, medical educators face an important task of developing curricula that provide a sound scientific foundation for clinical practice and scholarly investigation and that prepare medical students to become physicians to practice medicine competently in the 21st century (mahoney et al., 2004). medical students need to master an enormous volume of medical knowledge in their medical education from various sources. they may be very comfortable in using google or wikipedia to search for information. however, they lack proficiency in identifying, searching, and filtering information appropriate to address different types of clinical questions. thus, it is essential for them to develop high level information literacy and acquire fundamental, important skills for lifelong, self-directed, problem based learning from the early years of their medical education. basic ebm skills of searching, appraising, and applying research evidence to individual patients should be taught early and applied as an integral part of learning throughout the four years of the medical education (glasziou et al., 2008). ebm integrated into the curriculum will be able to educate students who will become more effective residents and, ultimately, practicing physicians best able to make informed clinical decisions about the care of patients and to handle the exponential growth of biomedical knowledge. it is important for medical educators and health sciences librarians to become cognizant of the available education research evidence and current practice in teaching ebm in undergraduate medical education. the knowledge will aid them as they launch their efforts of designing, developing, and implementing an ebm intervention, whether it is in the format of a curriculum, course, or series of workshops in preclinical or clinical years. a comprehensive literature search yielded no systematic or other types of reviews on ebm in the undergraduate medical education. this qualitative review of studies on ebm was conducted to determine the year when ebm was introduced and the extent to which medical students were exposed to ebm in undergraduate medical education. the review also attempted to examine how ebm interventions were designed, developed, implemented, and evaluated in the medical curriculum. methods data sources the review of the literature covered the period from 1997, the publication year for david sackett’s seminal work evidence-based medicine: how to practice and teach ebm, to 2011, when the literature review for this article was performed. since 1997, sackett’s work has helped drive the paradigm shift in the practice of medicine and has had a far reaching impact on the evolution of ebm teaching across the continuum of medical education. the literature search included pubmed, cinahl, web of science, cochrane library, proquest dissertations & thesis, psycinfo, and eric. pubmed was searched by subject with mesh terms (medical subject headings): education, medical, undergraduate; students, medical; evidence-based practice; and evidence-based medicine. the search results were limited to the language of english. pubmed and other databases were searched with the text words of evidence-based medicine, evidence-based practice, and evidence. these words were combined with medical education, undergraduate medical education, medical school, clerkship, medical students, as well as variations of the following evidence based library and information practice 2012, 7.3 101 text words: teach*, learn*, train*, educat*, instruct*, curricul*, and program. the search of these databases was supplemented by hand searching of reference lists of all included articles. cited references were also searched in web of science to identify relevant articles that cited those studies selected for the review. selection criteria studies were included in the review if they evaluated the effect of formal ebm teaching in undergraduate medical education programs leading to a medical degree accredited by the lcme. formal ebm teaching was defined as any intervention in the form of a curriculum, required or elective course, ebm components integrated into a preclinical course, curriculum, or clinical clerkship rotation. the types of research design utilized in clinical research may be inadequate in researching, investigating, and reporting educational phenomena with complexities of educational programs and wide variations in instructional methods, learning outcomes, and differences in learner characteristics. qualitative research methods or alternative educational research methods are underutilized and valuable research tools in medical education research. as hatala and guyatt pointed out, welldesigned qualitative studies using these methods are equally valuable in contributing evidence to the research of teaching evidence based medicine (2002). therefore, studies that utilized the following types of research design (table 1) commonly employed in education research were selected for the review (campbell & stanley, 1963; gall, gall, & borg, 2007). further information on each research design type can be found in appendix a. articles were excluded from the review if they were studies on:  ebm teaching at postgraduate and continuing education levels  ebm teaching that is not part of, or not integrated into a curriculum, an existing/required course, program, or clinical clerkship rotation in undergraduate medical education  ebm in medical schools not accredited by the lcme  effect of ebm teaching in osteopathic medical education or other allied health professional education table 1 types of research design control-group designs with random assignment 1. pretest-posttest control-group design r o x o r o o 2. posttest-only control-group design r x o r o quasi-experimental designs 3. static-group comparison design x o o 4. nonequivalent control-group design o x o o o single-group designs 5. one-short case study x o 6. one-group pretest-posttest design o x o note. r = randomization; o = pretest or posttest; x = experimental treatment/intervention. evidence based library and information practice 2012, 7.3 102 data extraction data were extracted from each eligible study on study design, year of ebm introduced in the curriculum, setting, ebm intervention, outcome measured, instructional method, and evaluation method. furthermore, information on instructional strategies was examined using the hierarchy of evidence based medicine teaching and learning activities (table 2) as described by khan and coomarasamy (2006). kirkpatrick’s four levels of evaluation (table 3) were adapted to determine the extent to which ebm outcomes were measured (kirkpatrick, 1994, 1996). the variation in research designs, interventions, and outcome measures in the selected studies precludes quantitatively combining results with a meta-analysis technique. results literature search the search strategies identified 13 studies on ebm in undergraduate medical education which met the inclusion criteria. these articles were derived from a large pool of 1,279 articles through a process of title scanning, abstract reading, hand searching, and elimination of duplicate articles from multiple databases. the review of the studies was conducted within the framework of the instructional design model addie (the five phases or steps in designing effective instruction): analysis, design, development, implementation, and evaluation in instructional design (gustafson & branch, 2007). the phase of analysis is beyond the scope of this review. table 4 provides a summary of the studies selected for the review. table 2 hierarchy of evidence based medicine teaching and learning activities level 1 interactive, and clinically integrated teaching and learning activities level 2 a) interactive, classroom based teaching and learning activities b) didactic, but clinically integrated teaching and learning activities level 3 didactic, and classroom or standalone teaching and learning activities table 3 four levels of evaluation level 1 learner satisfaction (reaction) perceptions, opinions, satisfaction level 2 learning outcomes (learning) change in knowledge, skills, attitudes level 3 performance improvement (behaviour) change in behaviours, transfer of learning to a patient care setting, performance in a practice setting level 4 patient/health outcomes (results) tangible results--improvement in the health and wellbeing of patients evidence based library and information practice 2012, 7.3 103 table 4 summary of study design, year of ebm in the curriculum, and settings author (year) study design year of ebm in the curriculum and setting wadland, barry, farquhar, holzman, and white (1999) static-group comparison design year 3: clinical campuses in 6 communities barnett et al. (2000) static-group comparison design year 1 year 3 and year 4: clinical clerkships in medicine, pediatrics, psychiatry, surgery, obstetrics and gynecology, neurology, community medicine, geriatrics ghali et al. (2000) nonequivalent control-group design year 3: ambulatory care sites during internal medicine clerkship thomas and cofrancesco (2001) one-shot case study year 3 and year 4: ambulatory medicine clerkship rotation srinivasan et al. (2002) one-shot case study year 1 holloway, nesbit, bordley, and noyes (2004) one-shot case study year 1 year 2 (same cohorts of first year medical students followed up throughout year 2) dorsch, aiyer, and meyer (2004) one-shot case study year 3: internal medicine clerkship cayley (2005) one-group pretest-posttest design year 3: family medicine clerkship schilling, wiecha, polineni, and khalil (2006) pretest-posttest control-group design year 3: family medicine clerkship nieman,cheng, and foxhall (2009) one-shot case study year 1 aronoff et al. (2010) one-group pretest-posttest design year 3: core clinical clerkships west, jaeger, and mcdonald (2011) one group pretest-posttest design year 2 year 3: clinical rotations of internal medicine, surgery, pediatrics, obstetrics and gynecology, neurology, and psychiatry sastre, denny, mccoy, mccoy, and spickard (2011) one group pretest-posttest design year 3: inpatient portion of internal medicine clerkship blocks evidence based library and information practice 2012, 7.3 104 description of studies the 13 studies demonstrated variability in methodology. one study had the pretestposttest control group design with random assignment; three studies had the quasiexperimental design; four studies used one group pretest-posttest design; and five studies utilized one shot case study design. participants of the studies ranged from firstyear to fourth-year medical students. three studies reported the integration of ebm into preclinical education curricula. ebm was introduced to third-year clinical clerkships in eight studies. two studies reported a longitudinal ebm intervention that spanned preclinical and clinical years. barnett et al. (2000) evaluated ebm instruction integrated into the first-year medical education curriculum and clinical education; while the study by west et al. (2011) reported the outcomes of a longitudinal ebm into the second-year preclinical education and third-year core clinical rotations. studies on ebm during clinical rotations were conducted at various settings of clerkships of internal medicine, family medicine, and pediatrics on community campuses (wadland et al., 1999), outpatient internal medicine clerkship (ghali et al., 2000; thomas & cofrancesco, 2001), inpatient portion of internal medicine clerkship blocks (sastre et al., 2011), family medicine clerkship (cayley, 2005; schilling et al., 2006), or core clinical clerkship rotations such as internal medicine, surgery, pediatrics, obstetrics and gynecology, neurology, and psychiatry (aronoff et al., 2010; barnett et al., 2000; west et al., 2011). design and development of ebm interventions the primary goal of ebm instruction in the 13 studies was to develop medical students’ essential skills for evidence based practice and evidence based problem solving; other goals included developing lifelong learners (barnett et al., 2000; dorsch et al., 2004), introducing basic concepts of epidemiology and biostatistics (west et al., 2011), and developing and presenting a research proposal (wadland et al., 1999). ebm skills were the main focus of the ebm interventions in the studies; however, the ebm interventions reported in three studies included no content on the second step of the ebm process – acquiring the evidence (srinivasan et al., 2002; wadland et al., 1999; west et al., 2011). there were a varied number of contact hours devoted to the ebm interventions in the 13 studies. the time ranged from three student contact hours in a workshop (sastre et al., 2011) to 30 contact hours in an ebm curriculum (holloway et al., 2004). ebm components were introduced into preclinical years in various ways: offered as a short ebm course of eight contact hours enhanced with problem based learning small group discussion sessions facilitated by clinicians (srinivasan et al., 2002); integrated into courses such as library science and medical informatics, epidemiology, microbiology (barnett et al., 2000), and mastering medical information (holloway et al., 2004); taught as part of a series of educational interventions in an elective family medicine preceptorship program for first and second-year medical students (nieman, cheng, & foxhall, 2009); and offered as a course of 22 contact hours at the end of year 1 (west et al., 2011). in clinical years, ebm was integrated with clinical educational experiences through inpatient and outpatient primary care clerkship rotations of internal medicine, family medicine, and several other core clinical clerkships (aronoff et al., 2010; cayley, 2005; dorsch et al., 2004; ghali et al., 2000; sastre et al., 2011; schilling et al., 2006; thomas & cofrancesco, 2001; wadland et al., 1999). among the 13 studies, only two studies, one by barnett et al. (2000) and the other by west et al. (2011), had a longitudinal ebm curriculum that spanned the preclinical curriculum and clinical core clerkships. the content domain of the ebm interventions in the 13 studies was derived from multiple sources. they included users’ guides to the medical literature (guyatt & rennie, 2002), a series of articles on “harnessing medline” (mckibbon, walker-dilks, haynes, & evidence based library and information practice 2012, 7.3 105 wilczynski, 1995; mckibbon & walker-dilks, 1994a, 1994b), ebm steps as outlined by sackett, rosenberg, gray, haynes, and richardson (1996), and journal articles on ebm teaching. four studies failed to report how ebm content was developed, although the ebm interventions in these studies reflected some basic ebm principles and processes widely recognized in the medical literature on ebm teaching and practice. the ebm curriculum in one study comprised only online content: a literature searching tutorial, the website of the national guideline clearinghouse (www.guideline.gov/), and the website on the calculation of nnt (number needed to treat). three out of the 13 studies included instruction on using pre-appraised ebm resources such as dynamed, acp journal club, cochrane library, and the national guideline clearinghouse (nieman et al., 2009; sastre et al., 2011; schilling et al., 2006). ebm knowledge and component skills in the ebm interventions were taught with various methods such as didactic lecture, live demonstration, hands-on practicum, small group discussion, and case based presentation. in addition, journal club (barnett et al., 2000), problem based small group discussion (srinivasan et al., 2002), and worksheets (dorsch et al., 2004; nieman et al., 2009; srinivasan et al., 2002) were also reported as interactive teaching and learning activities in the ebm interventions. to categorize the ebm interventions in the 13 studies with the hierarchy of effective teaching and learning activities proposed by khan and coomarasamy (2006), five studies incorporated ebm teaching and learning activities of level 1, which represents the most effective practice of teaching and learning of ebm; one study used activities of level 2a; six studies integrated activities of level 2b; and one study fell under level 3 activities. out of the 13 studies, three explored the alternative instruction delivery format to provide ebm learning experience for students on the internet or through a web content management system (webct or blackboard). given the variability in faculty’s availability in teaching ebm, online ebm instruction could provide consistent and equivalent learning experiences for students rotating across multiple clinical training sites or offer students a tool to enhance their learning in a traditional classroom setting. srinivasan et al. (2002) used a 20 page web-based ebm curriculum and an online practice examination developed in webct to supplement a short ebm course made of components of didactic lectures and interactive, problem based small group discussion sessions. first-year medical students used the online ebm curriculum as an independent study tool to reinforce their ebm learning (srinivasan et al., 2002). schilling et al. (2006) investigated the use of a webbased curriculum to teach ebm. the entire curriculum encompassed an online tutorial and information from two other websites. in the study, an asynchronous discussion board moderated by faculty was also used as a learning activity to encourage students’ discussion of their patient case encounter, clinical questions, search process, and findings. although the curriculum was intended as a web-based ebm instruction tool, ebm content covered was incomplete since essential ebm skills of critical appraisal and evidence application were not addressed in the ebm curriculum. another study reported an ebm intervention featuring a two part ebm course for third-year medical students who went through core clinical clerkship rotations (aronoff et al., 2010). the first part was delivered in six didactic online modules of materials via blackboard (blackboard academic suite; blackboard, inc., washington, dc), each of which was followed by a focused practicum that students completed and submitted to an online faculty mentor for review and feedback. the second part of the course required that each student generate a clinical question on a patient seen during each of four clinical rotations and complete four evidence summaries using the critically appraised topic format (cat) developed by sackett, richardson, rosenberg, and haynes (1997). the students sent the evidence summaries to their mentor for review and correction, if necessary. evidence based library and information practice 2012, 7.3 106 implementation of ebm interventions nine out of the 13 studies did not provide any information on any faculty development opportunities nor delineate the preparation or qualification of any faculty involved in teaching ebm. only four studies explained the process of implementing their ebm intervention, particularly faculty development efforts, on varying levels of detail. the first of the four studies was conducted at the college of human medicine at michigan state university, which is a community-based medical school without a central university hospital. the ebm intervention of a nine session curriculum was delivered on clinical campuses across six communities. a course director and course coordinator provided central management for faculty development and course implementation. each community campus had a research director who served as the community course coordinator and participated in the course content and format selection. a two day faculty development event was held in which four mcmaster university faculty members led intensive training seminars to train and prepare 50 faculty members to teach ebm locally (wadland et al., 1999). the development of a four year longitudinal ebm curriculum reported by barnett et al. (2000) started with the establishment of an evidence based medicine working group at the mount sinai school of medicine. its multidisciplinary team members were charged with the task of revamping the ebm component in the traditional medical curriculum in collaboration with all preclinical course and clinical clerkship directors. faculty development opportunities were provided on different levels, including a full day retreat for course and clerkship directors to receive ebm training provided by faculty from mcmaster university. these trained course and clerkship directors in turn provided training for other faculty in their own department. srinivasan et al. (2002) investigated the effect of a one month ebm course integrated into the preclinical medical curriculum at the indiana university school of medicine. the course combined traditional didactic lectures with interactive small group, problem based learning sessions to teach 138 first-year medical students ebm principles and skills. the small group, problem based sessions were facilitated by 16 faculty members who represented four departments of emergency medicine, pediatrics, internal medicine, and library sciences at the medical school. to prepare faculty for the small group sessions, a facilitator’s handbook was developed to provide faculty with consistent small group experiences, detailed objectives, timelines, commonly asked questions and answers, sample dialog, completed users’ guides to the medical literature worksheets (guyatt & rennie, 2002), critical concept summaries, background reading material, and small group teaching strategies and references. furthermore, dedicated secretarial support was available to faculty in the early introduction of the ebm course to preclinical medical students. all faculty facilitators who participated in the small group sessions had ebm experience through taking formal ebm courses, teaching ebm, holding a mph (master of public health) degree, or writing about ebm for peer reviewed journals. in another study on an ebm longitudinal curriculum reported by west et al. (2011), ebm faculty were given supported time for each graded assignment and administrative time for direct student contact during the period of teaching and maintaining the short course in the curriculum. each instructor involved in ebm teaching had extensive ebm knowledge and skills through their advanced training in biostatistics and epidemiology and participation in the workshop offered at mcmaster university. these instructors also had experience in teaching basic and advanced ebm topics to residents at mayo clinic. the ebm curriculum in the study by wadland et al. (1999) was standardized to ensure the equivalent ebm learning experience in three primary care clerkships of internal medicine, family medicine, and pediatrics at multiple locations. srinivasan et al. (2002) used didactic lectures in conjunction with the problem based learning approach which demanded a great evidence based library and information practice 2012, 7.3 107 deal of investment in resources (e.g., faculty and their time in facilitating small group sessions). two other studies, one by barnett et al. (2000) and the other by west et al. (2011), were characterized by a longitudinal curriculum integrated into both preclinical and clinical years of medical education. these four studies demonstrate that the successful implementation of an ebm intervention across multiple sites, with a problem based small group discussion component, or integrated into both preclinical and clinical education curricula, are contingent on important factors such as provision of faculty development opportunities, resources coordination, and instructional and administrative support for ebm faculty. evaluation of ebm outcomes variations in evaluation methods used to assess the effect of the ebm interventions in the studies preclude any quantitative analysis of a pooled effect size of the results from the studies. therefore, the outcomes reported in the studies were examined using kirkpatrick’s four level evaluation model: satisfaction with a learning experience (level 1), learning in terms of any change in knowledge, skills, or attitudes (level 2), performance improvement or behavioural changes in a patient care setting (level 3), and impact on patient health outcomes (level 4). five studies incorporated level 1 evaluation; all 13 studies examined changes in knowledge, skills, or attitudes at level 2; and three studies had level 3 evaluation of the outcomes of behavioural changes in a patient care setting. none of the studies went beyond level 3 evaluation. a summary of ebm interventions, outcomes measured, evaluation methods, and levels of evaluation is presented in appendix b. level 1 evaluation: reaction and satisfaction five out of the 13 studies evaluated learners’ satisfaction and reaction to the ebm intervention. wadland et al. (1999) found that second-year medical students’ rating of an ebm course was not significantly different from that of other courses offered in the medical curriculum and that there were no significant variations in responses across multiple community clinical sites. srinivasan et al. (2002) evaluated both students and faculty’s satisfaction with an ebm course integrated into the first-year medical curriculum. the evaluative results showed that a majority of students reported enjoying the ebm course and felt that the course material was appropriate for their learning level and related to clinical practice. srinivasan et al. (2002) also sought faculty’s feedback as an indicator of the effect of ebm instruction. in faculty’s opinions, medical students could perform well in ebm-related areas and use ebm concepts as well as or better than residents whom the faculty had supervised in the past year. both faculty and student respondents agreed that the course material was appropriate for first-year medical students. in the study by sastre et al. (2011), third-year medical students also reacted to their ebm learning positively. they felt that ebm learning was useful and that they would incorporate their acquired skills into clinical care of patients. holloway et al. (2004), however, reported that a majority of student respondents (58 out of 67 respondents) expressed negative comments about ebm instruction and evaluation that spanned the preclinical education. the students felt that there was too much emphasis on ebm (43-page syllabus) in their preclinical curriculum. they also commented that the test module with a 15-20 page evaluation packet (a five step ebm evaluation module) took too much time to complete and was of little value to their learning of good ebm skills. thomas and cofrancesco (2001) found that thirdand fourth-year medical students rated the usefulness of their ebm learning through an ebm report less favorably than their clinical sessions with their preceptor. the researchers also found that the students perceived their clinical competence in ebm with a lower value than diagnostic decision making and selfdirected learning. evidence based library and information practice 2012, 7.3 108 level 2 of evaluation: change in skill and knowledge level 2 evaluation was conducted in all 13 studies to measure any possible change in learners’ knowledge, skills, or attitudes as a result of ebm instruction. learning outcomes in these studies were measured with a wide array of evaluation methods such as the aamc medical school graduation questionnaire (wadland et al., 1999), critique of a relevant article (barnett et al., 2000; schilling et al., 2006), self-administered ebm skill survey (dorsch et al., 2004; ghali et al., 2000; schilling et al., 2006; west et al., 2011), self-administered examination of knowledge and skills (aronoff et al., 2010; dorsch et al., 2004; srinivasan et al., 2002; west et al., 2011), self-efficacy questionnaire (cayley, 2005; nieman et al., 2009), a five step ebm evaluation module (holloway et al., 2004), pico case summary (nieman et al., 2009), and cat (critically appraised topic) summary (aronoff et al., 2010). one study reported an improved response rate for appropriate training in literature analysis and research skills on the aamc medical school graduation questionnaire since one of the objectives for the ebm curriculum in the study was to develop students’ research skills (wadland et al., 1999). five studies reported improvement in literature searching and other library skills (barnett et al., 2000; dorsch et al., 2004; ghali et al., 2000; holloway et al., 2004; schilling et al., 2006). these skills were measured with various instruments such as a rating scale (barnett et al., 2000; schilling et al., 2006), evaluation rubric (holloway et al., 2004), case scenario based skill test (dorsch et al., 2004), and self-reported survey (ghali et al., 2000). ebm skill in critical appraisal was evaluated as an outcome in six studies: four reported significant improvement on objective measures (aronoff et al., 2010; dorsch et al., 2004; srinivasan et al., 2002; west et al., 2011); one reported improvement on self-reported skill test (ghali et al., 2000); and one failed to report any evaluative results through a critical appraisal measure – part of two evaluation modules that tested the five steps of the ebm process (holloway et al., 2004). the results of most studies reveal improvement of varying degrees in terms of learners’ knowledge and skills in ebm. level 3 of evaluation: changes in behaviour three studies incorporated level 3 evaluation measures in evaluating the extent to which any change was made in students’ behaviour in a clinical setting. two of the three studies reported students’ successful use of ebm in actual clinical work through application of the five steps of the ebm process in an ebm report (aronoff et al., 2010; thomas & cofrancesco, 2001). students in the study by thomas and cofrancesco (2001) wrote an ebm report based on a real patient case. students in the study by aronoff et al. (2010) formulated a clinical question generated by a patient seen during each of four clinical rotations. a complete cat form was required of students and reviewed but not scored by faculty mentors. in the study by sastre et al. (2011), level 3 evaluation was conducted through evaluation of citations of ebm resources in students’ inpatient admission notes and the quality of the ebm component of the discussion portion in the notes. another measure to evaluate students’ use of ebm resources in the same study was computer log recordings of students’ online use of various ebm resources via an electronic medical resource system. analysis of students’ patient notes reveals a significant improvement on the overall quality of ebm incorporation into the discussion of a patient’s problem as a result of ebm instruction. the computer log data showed a significant increase in students’ usage for all electronic resources. ebm resources such as systematic peer reviewed resources (e.g., cochrane databases, clinical guidelines) represented 59% of all resources accessed following the workshop. however, there was no significant improvement in the number of citations per patient note before and after the ebm intervention. discussion the review was conducted to summarize the findings of the studies retrieved from comprehensive literature searching of relevant evidence based library and information practice 2012, 7.3 109 resources. the analysis of 13 studies reveals assorted approaches to instructional design, development, implementation, and evaluation of ebm interventions in undergraduate medical education. the definition of ebm and component skills required to practice ebm are not debatable in the current medical literature. however, there is no standard or model for how ebm should be designed and developed and how it should be effectively taught and evaluated in undergraduate medical education. the studies show wide variations in the use of instructional strategies, delivery formats for ebm instruction, implementation of an ebm intervention, and outcome measures, all of which remain important issues for further research. effectiveness of an educational intervention is inescapably linked to the outcomes of educational activities (belfield, thomas, bullock, eynon, & wall, 2001). interactive and clinically integrated activities support active learning and could lead to medical students’ deep understanding of content material and superior experience in classroom and clinical settings. the review shows that several studies made efforts to incorporate clinically integrated ebm learning experiences and clinically relevant outcome measures into ebm instruction in the medical curriculum. however, more than half of the studies reviewed fell short of integrating interactive teaching and learning activities into the ebm interventions. future research is required to determine the effect of ebm instruction on learning outcomes as a result of sound instructional design principles and active learning strategies employed in the design and development of an ebm intervention in undergraduate medical education. delivery of ebm instruction with information technology holds promise in affecting ebm learning outcomes for medical students. a study by davis, crabb, rogers, zamora, and khan (2008) demonstrated that first-year medical students in a computer-based ebm session had gains in knowledge and attitude similar to those in the lecture-based ebm session. cook et al. (2008) conducted a metaanalysis of internet-based learning in the health professions. their findings provided further evidence for the effectiveness of internet-based instruction similar to traditional instructional methods. three of the 13 studies utilized online learning as an approach to teaching ebm or supplementing the traditional method of teaching ebm. the results from these studies suggest implications for medical educators and health science librarians in developing and teaching ebm in the medical curriculum. online/internet-based ebm instruction that is properly designed, developed, and implemented could support students’ self-directed learning, reinforce ebm learning in a traditional classroom setting, or present great potential for standardizing the quality of ebm teaching to achieve equivalent learning experience across multiple clinical teaching sites or clerkship rotations. to a great extent, effective delivery of an ebm intervention hinges on a good implementation plan addressing issues related to faculty recruitment for ebm teaching, faculty training in ebm, administrative support, time and space allocated to instruction, and procedures for recording and collecting data pertaining to course and student evaluation (gustafson & branch, 2007). nine of the 13 studies failed to provide any information on the implementation process of an ebm intervention. implementation is one of six important steps in curriculum development for medical education (kern, thomas, & hughes, 2009), as well as in the process of instructional design (gustafson & branch, 2007). successful integration of ebm into the medical curriculum necessitates ebm training for busy primary care clinicians or preceptors in the clinical teaching and application of ebm (cayley, 2005; nicholson, warde, & boker, 2007) and for teaching ebm in a longitudinal fashion rather than as a single innovation (nieman et al., 2009). given that faculty development is likely more important than any other step in implementation (bordley et al., 1997), faculty development efforts are pivotal to any effective ebm instruction for medical students in either preclinical or clinical years, as evidenced by higher levels of learner satisfaction and knowledge gains found in the studies by srinivasan et al. (2002) and west et evidence based library and information practice 2012, 7.3 110 al. (2011). “teaching ebm may need to focus as much on teachers as on students” (del mar, glasziou, & mayer, 2004, p. 990). implementation of a successful ebm intervention requires investment of adequate faculty time, resources (mi, moseley, & green, 2012), and provision of faculty development opportunities (green, 2000; nicholson et al., 2007). the analysis of the studies reveals a wide variation in ebm learning outcomes due to study heterogeneity in research designs, which did not lend to an estimation of a combined effect from these studies. the adapted 4 level evaluation model by kirkpatrick was applied to examine the extent to which learning outcomes were evaluated. the results showed that the majority of the studies focused on level 1 and level 2 evaluation to measure learning that occurs separated from the realtime setting of practice. few studies went beyond level 3 evaluation, which is more difficult to measure because it requires assessment in the practice setting (straus et al., 2004). however, evaluation of the effectiveness of ebm as an educational intervention should aim to incorporate level 3 and level 4 evaluation into curriculum development to measure the long-term impact of ebm instruction on sustained behavioural changes in a patient care setting and possibly, patient health outcomes. west et al. (2011) stressed the need for research on the sustained increases in perceived and measured ebm knowledge and benefits beyond medical school into postgraduate medical education and practice. clearly, students’ independent application of ebm skills in a clinical setting and into their residency warrants further research efforts. the article only reviewed studies on ebm interventions in the medical curriculum of u.s. medical schools given the differences that exist between medical schools in the us and other countries in terms of duration of undergraduate medical education programs, requirement in competence-based learning outcomes, and program accreditation standards. further effort could be made to systematically review the literature including international studies, related to specific topics such as the use of different instructional methods in teaching ebm (e.g, team based learning, problem based learning) and different instruction delivery formats (e.g., online instruction of ebm). such review would afford additional insights into ebm teaching in undergraduate medical education. the qualitative review was limited to full research reports published in peer review journals. the selected reports addressed formal ebm instruction as part of or integrated into preclinical or clinical curricula, which is the focus of the review. future research could be conducted to examine how each ebm step or component (e.g., clinical question formation, searching the literature for the evidence, or critical appraisal) is instructed and evaluated in undergraduate medical education. systematic reviews could be carried out through exhaustive searching of literature including conference proceedings and other grey literature to shed further light on teaching practices and innovations in ebm instruction in undergraduate medical education. the inclusion criteria specifically established in this review circumscribed the number of studies selected. it should be pointed out that there is a view frequently held in medical education that “all that is needed in medical education is commonsense” (harden, 1986, p. 522). the view may partly account for the level of evidence based educational research on ebm and other educational phenomena. clearly, there is a need to employ different approaches to research and to break with or question existing teaching practices in medical education in general and ebm teaching in particular. relevant research on the use of these approaches to research and creative and innovative methods of teaching ebm would add significantly to the literature in this field. conclusion the review was conducted as an attempt to present research evidence on how ebm was designed, developed, implemented, and evaluated in undergraduate medical education and to help inform medical educators and health sciences librarians in their efforts to evidence based library and information practice 2012, 7.3 111 practice evidence based teaching of ebm. the review also pointed out a number of areas that warrant further research on ebm in undergraduate medical education. khan and coomarasamy (2006) comment that the substantial heterogeneity in the types of educational interventions and their effects necessitate an analysis beyond simply focusing on the overall results of studies on ebm teaching. the article analyzed and discussed ebm interventions in educational research from the perspective of instructional design. it examined some relevant issues related to the treatment of ebm content in the undergraduate medical curriculum. these issues included ebm instructional design, implementation, faculty development, and levels of evaluation of learning outcomes. failure to attend to these issues could undermine the effectiveness of an ebm intervention in the medical education curriculum. references aronoff, s. c., evans, b., fleece, d., lyons, p., kaplan, l., & rojas, r. (2010). integrating evidence based medicine into undergraduate medical education: combining online instruction with clinical clerkships. teaching & learning in medicine, 22(3), 219-223. doi:10.1080/10401334.2010.488460 association of american medical colleges. (2012). medical school graduation questionnaire: 2012 all schools summary report. in association of american medical colleges. retrieved 4 aug. 2012 from https://www.aamc.org/download/3004 48/data/2012gqallschoolssummaryrepo rt.pdf barnett, s. h., kaiser, s., morgan, l. k., sullivant, j., siu, a., rose, d., rico, m., smith, l., schechter, c., miller, m., stagnaro-green, a. (2000). an integrated program for evidence-based medicine in medical school. mount sinai journal of medicine, 67(2), 163-168. barnett, s. h., smith, l. g., & swartz, m. h. (1999). teaching evidence-based medicine skills to medical students and residents. international journal of dermatology, 38(12), 893-894. doi:10.1046/j.1365-4362.1999.00844.x belfield, c., thomas, h., bullock, a., eynon, r., & wall, d. (2001). measuring effectiveness for best evidence medical education: a discussion. medical teacher, 23(2), 164-170. doi:10.1080/0142150020031084 bordley, d. r., fagan, m., & theige, d. (1997). evidence-based medicine: a powerful educational tool for clerkship education. the american journal of medicine, 102(5), 427-432. campbell, d. t., & stanley, j. c. (1963). experimental and quasi-experimental designs for research. boston, ma: houghton mifflin company. cayley, w. e., jr. (2005). evidence-based medicine for medical students: introducing ebm in a primary care rotation. wisconsin medical journal, 104(3), 34-37. retrieved 18 jul. 2012 from http://www.wisconsinmedicalsociety.o rg/_wms/publications/wmj/issues/wm j_v104n3/cayley.pdf cook, d. a., levinson, a. j., garside, s., dupras, d. m., erwin, p. j., & montori, v. m. (2008). internet-based learning in the health professions: a metaanalysis. journal of american medical association, 300(10), 1181-1196. doi:10.1001/jama.300.10.1181 davis, j., crabb, s., rogers, e., zamora, j., & khan, k. (2008). computer-based teaching is as good as face to face lecture-based teaching of evidence based medicine: a randomized controlled trial. medical teacher, 30(3), https://www.aamc.org/download/300448/data/2012gqallschoolssummaryreport.pdf https://www.aamc.org/download/300448/data/2012gqallschoolssummaryreport.pdf https://www.aamc.org/download/300448/data/2012gqallschoolssummaryreport.pdf http://www.wisconsinmedicalsociety.org/_wms/publications/wmj/issues/wmj_v104n3/cayley.pdf http://www.wisconsinmedicalsociety.org/_wms/publications/wmj/issues/wmj_v104n3/cayley.pdf http://www.wisconsinmedicalsociety.org/_wms/publications/wmj/issues/wmj_v104n3/cayley.pdf evidence based library and information practice 2012, 7.3 112 302-307. doi:10.1080/01421590701784349 del mar, c., glasziou, p., & mayer, d. (2004). teaching evidence based medicine. british medical journal, 329(7473), 990. doi:10.1136/bmj.329.7473.989 dickersin, k., straus, s. e., & bero, l. a. (2007). evidence based medicine: increasing, not dictating, choice. british medical journal, 334(suppl 1), s10. doi:10.1136/bmj.39062.639444.94 dorsch, j. l., aiyer, m. k., & meyer, l. e. (2004). impact of an evidence-based medicine curriculum on medical students' attitudes and skills. journal of the medical library association, 92(4), 397-406. retrieved 18 jul. 2012 from http://www.ncbi.nlm.nih.gov/pmc/arti cles/pmc521510/pdf/i0025-7338-09204-0397.pdf finkel, m. l., brown, h. a., gerber, l. m., & supino, p. g. (2003). teaching evidence-based medicine to medical students. medical teacher, 25(2), 202204. doi:10.1080/0142159031000092634 gall, m. d., gall, j. p., & borg, w. r. (2007). educational research: an introduction (8th ed.). boston: pearson/allyn & bacon. ghali, w. a., saitz, r., eskew, a. h., gupta, m., quan, h., & hershman, w. y. (2000). successful teaching in evidence-based medicine. medical education, 34(1), 18-22. doi:10.1046/j.1365-2923.2000.00402.x glasziou, p., burls, a., & gilbert, r. (2008). evidence based medicine and the medical curriculum. british medical journal, 337, a1253. doi:10.1136/bmj.a1253 green, m. l. (2000). evidence-based medicine training in internal medicine residency programs: a national survey. journal of general internal medicine, 15(2), 129133. retrieved 17 jul. 2012 from http://www.springerlink.com/content/ r351055rj56h1130/fulltext.pdf gustafson, k. l., & branch, r. m. (2007). what is instructional design? in r. a. reiser & j. v. dempsey (eds.), trends and issues in instructional design and technology (2nd ed., pp. 10-16). upper saddle river, nj: pearson/merrill prentice hall. guyatt, g., & rennie, d. (eds.) (2002). users' guides to the medical literature: essentials of evidence-based clinical practice. chicago, il: american medical association. harden, r. m. (1986). approaches to research in medical education. medical education, 20(6), 521-531. doi:10.1111/j.1365-2923.1986.tb01394.x hatala, r., & guyatt, g. (2002). evaluating the teaching of evidence-based medicine. jama: journal of the american medical association, 288(9), 1110-1112. doi:10.1001/jama.288.9.1110 holloway, r., nesbit, k., bordley, d., & noyes, k. (2004). teaching and evaluating first and second year medical students' practice of evidence-based medicine. medical education, 38(8), 868-878. doi:10.1111/j.1365-2929.2004.01817.x kern, d. e., thomas, p. a., & hughes, m. t. (2009). curriculum development for medical education: a six-step approach (2nd ed.). baltimore, md: johns hopkins university press. khan, k. s., & coomarasamy, a. (2006). a hierarchy of effective teaching and learning to acquire competence in evidenced-based medicine. bmc medical education, 6, 59. doi:10.1186/1472-6920-6-59 kirkpatrick, d. l. (1994). evaluating training programs: the four levels. san francisco, ca: berrett-koehler. http://www.ncbi.nlm.nih.gov/pmc/articles/pmc521510/pdf/i0025-7338-092-04-0397.pdf http://www.ncbi.nlm.nih.gov/pmc/articles/pmc521510/pdf/i0025-7338-092-04-0397.pdf http://www.ncbi.nlm.nih.gov/pmc/articles/pmc521510/pdf/i0025-7338-092-04-0397.pdf evidence based library and information practice 2012, 7.3 113 kirkpatrick, d. (1996). great ideas revisited. training & development, 50(1), 54-59. liaison committee on medical education. (2012). functions and structure of a medical school: standards for accreditation of medical education programs leading to the m.d. degree. in liaison committee on medicual education. retrieved 4 aug. 2012 from http://www.lcme.org/functions.pdf mahoney, j. f., cox, m., gwyther, r. e., o'dell, d. v., paulman, p. m., & kowlowitz, v. (2004). evidence-based and population-based medicine: national implementation under the ume-21 project. family medicine, 36(jan. suppl.), s31-35. retrieved 17 jul. 2012 from http://www.stfm.org/fmhub/fm2004/ja nuary/johns31.pdf mckibbon, k. a., walker-dilks, c., haynes, r. b., & wilczynski, n. (1995). beyond acp journal club: how to harness medline for prognosis problems. acp journal club, 123(1), a12-a14. mckibbon, k. a., & walker-dilks, c. j. (1994a). beyond acp journal club: how to harness medline for diagnostic problems. acp journal club, 121(suppl. 2), a10-12. mckibbon, k. a., & walker-dilks, c. j. (1994b). beyond acp journal club: how to harness medline for therapy problems. acp journal club, 121(suppl. 1), a10-12. melnyk, b. m. (2006). calling all educators to teach and model evidence-based practice in academic settings. worldviews on evidence-based nursing, 3(3), 93-94. doi:10.1111/j.17416787.2006.00061.x mi, m., moseley, j. l., & green, m. l. (2012). an instrument to characterize the environment for residents' evidencebased medicine learning and practice. family medicine, 44(2), 98-104. retrieved 17 jul. 2012 from http://www.stfm.org/fmhub/fm2012/f ebruary/misa98.pdf montori, v. m., & guyatt, g. h. (2008). progress in evidence-based medicine. journal of american medical association, 300(15), 1814-1816. doi:10.1001/jama.300.15.1814 nicholson, l. j., warde, c. m., & boker, j. r. (2007). faculty training in evidencebased medicine: improving evidence acquisition and critical appraisal. journal of continuing education in the health professions, 27(1), 28-33. doi:10.1002/chp.090 nieman, l. z., cheng, l., & foxhall, l. e. (2009). teaching first-year medical students to apply evidence-based practices to patient care. family medicine, 41(5), 332-336. retrieved 17 jul. 2012 from http://www.stfm.org/fmhub/fm2009/m ay/linda332.pdf o'connell, m. t., & pascoe, j. m. (2004). undergraduate medical education for the 21st century: leadership and teamwork. family medicine, 36 (suppl. january), s51-56. retrieved 4 aug. 2012 from http://www.stfm.org/fmhub/fm2004/ja nuary/marks51.pdf sackett, d. l., richardson, w. s., rosenberg, w., & haynes, r. b. (1997). evidencedbased medicine: how to practice and teach ebm. new york, ny: churchill livingstone. sackett, d. l., rosenberg, w. m. c., gray, j. a. m., haynes, r. b., & richardson, w. s. (1996). evidence-based medicine: what it is and what it isn't. british medical journal, 312, 71. doi:10.1136/bmj.312.7023.71 http://www.lcme.org/functions.pdf http://www.stfm.org/fmhub/fm2004/january/johns31.pdf http://www.stfm.org/fmhub/fm2004/january/johns31.pdf http://www.stfm.org/fmhub/fm2012/february/misa98.pdf http://www.stfm.org/fmhub/fm2012/february/misa98.pdf http://www.stfm.org/fmhub/fm2009/may/linda332.pdf http://www.stfm.org/fmhub/fm2009/may/linda332.pdf http://www.stfm.org/fmhub/fm2004/january/marks51.pdf http://www.stfm.org/fmhub/fm2004/january/marks51.pdf evidence based library and information practice 2012, 7.3 114 sastre, e. a., denny, j. c., mccoy, j. a., mccoy, a. b., & spickard, a. (2011). teaching evidence-based medicine: impact on students' literature use and inpatient clinical documentation. medical teacher, 33(6), e306-312. doi:10.3109/0142159x.2011.565827 schilling, k., wiecha, j., polineni, d., & khalil, s. (2006). an interactive web-based curriculum on evidence-based medicine: design and effectiveness. family medicine, 38(2), 126-132. retrieved 17 jul. 2012 from https://www.stfm.org/fmhub/fm2006/f ebruary/katherine126.pdf slawson, d. c., & shaughnessy, a. f. (1999). teaching information mastery: creating informed consumers of medical information. journal of american board of family practice, 12(6), 444-449. srinivasan, m., weiner, m., breitfeld, p. p., brahmi, f., dickerson, k. l., & weiner, g. (2002). early introduction of an evidence-based medicine course to preclinical medical students. journal of general internal medicine, 17(1), 58-65. retrieved 17 jul. from http://www.springerlink.com/content/ u42tnw482l38441r/fulltext.pdf steward, m. g. (2001). core competencies. in accreditation council for graduate medical education. retrieved 4 aug. 2012 from http://www.acgme.org/acwebsite/rrc _280/280_corecomp.asp straus, s. e., green, m. l., bell, d. s., badgett, r., davis, d., gerrity, m., ortiz, e., shaneyfelt, t. m., whalen, c. mangrulkar, r. (2004). evaluating the teaching of evidence based medicine: conceptual framework. british medical journal, 329(7473), 1029-1032. doi:10.1136/bmj.329.7473.1029 thomas, p. a., & cofrancesco, j., jr. (2001). introduction of evidence-based medicine into an ambulatory clinical clerkship. journal of general internal medicine, 16(4), 244-249. retrieved 17 jul. 2012 from http://www.springerlink.com/content/ b7k81391322u64n1/fulltext.pdf wadland, w. c., barry, h. c., farquhar, l., holzman, c., & white, a. (1999). training medical students in evidencebased medicine: a community campus approach. family medicine, 31(10), 703708. retrieved 17 jul. 2012 from http://stfm.org/fmhub/fullpdf/novd ec99/mse.pdf west, c. p., jaeger, t. m., & mcdonald, f. s. (2011). extended evaluation of a longitudinal medical school evidencebased medicine curriculum. journal of general internal medicine, 26(6), 611615. doi:10.1007/s11606-011-1642-8 https://www.stfm.org/fmhub/fm2006/february/katherine126.pdf https://www.stfm.org/fmhub/fm2006/february/katherine126.pdf http://www.springerlink.com/content/u42tnw482l38441r/fulltext.pdf http://www.springerlink.com/content/u42tnw482l38441r/fulltext.pdf http://www.acgme.org/acwebsite/rrc_280/280_corecomp.asp http://www.acgme.org/acwebsite/rrc_280/280_corecomp.asp http://www.springerlink.com/content/b7k81391322u64n1/fulltext.pdf http://www.springerlink.com/content/b7k81391322u64n1/fulltext.pdf http://stfm.org/fmhub/fullpdf/novdec99/mse.pdf http://stfm.org/fmhub/fullpdf/novdec99/mse.pdf evidence based library and information practice 2012, 7.3 115 appendix a types of research design 1. control group designs with random assignment  pretest-posttest control group design o random assignment of research participants to experimental and control groups o administration of a pretest to both groups o implementation of an intervention to the experimental group but not to the control group o administration of a posttest to both groups  posttest-only control group design o random assignment of participants to experimental and control groups o implementation of an intervention to the experimental group but not to the control group o administration of a posttest to both groups 2. quasi-experimental designs  static-group comparison design o participants not randomly assigned to the experimental and control groups o implementation of an intervention to the experimental group but not to the control group o a posttest, but no pretest administered to both groups  nonequivalent control group design o participants not randomly assigned to the experimental and control groups o implementation of an intervention to the experimental group but not to the control group o both groups taking a pretest and posttest 3. single group designs  one short case study o implementation of an intervention for participants (no control group) o administration of a posttest  one group pretest-posttest design o administration of a pretest to research participants (no control group) o implementation of an intervention o administration of a posttest evidence based library and information practice 2012, 7.3 116 appendix b summary of ebm interventions, outcomes measured, evaluation methods, and levels of evaluation author (year) ebm interventions outcomes measured evaluation methods levels of evaluation wadland et al. (1999) year 1 and year 2:  first component (8 hours) prior to formal clinical clerkships, focusing on critiquing articles and answering questions about study quality and applicability to patient care year 3 and year 4:  second component (12 hours) during clerkships of internal medicine, family practice, and pediatrics, focusing on evaluating guidelines relevant to primary care and developing clinically relevant research proposals  computerized literature search  understanding of structure of medical research articles  critical analysis of medical research articles  evaluation of practice guidelines and their development  development and oral presentation of a research proposal  aamc medical school graduation questionnaire  course evaluation questions  survey at two points of assessment (at graduation and at the end of first-year residency) 1, 2 barnett et al. (2000) year 1 and year 2:  medical informatics integrated into the library science and medical informatics course (7½ hours)  critical appraisal of articles integrated into the environmental medicine module of the epidemiology course  use of literature to identify unknown laboratory specimens in the microbiology course year 3:  one hour session during orientation on definition of goals and objectives of curriculum and description of components in each rotation  ebm clerkships of medicine, pediatrics, psychiatry, surgery, obstetrics and gynecology, neurology, community medicine, geriatrics, with each clerkship teaching one of the mcmaster modules such as appraising overview, prognosis, therapy, harm, and diagnostic test articles year 4  ebm added to the community and  exposure to medical informatics both prior to and during medical school  information retrieval  critical reasoning  medical informatics questionnaires  fourth-year ebm exercise with four cases: literature search strategy, 5 relevant citations, one paragraph critique of the single most relevant article 2 evidence based library and information practice 2012, 7.3 117 preventive medicine course  two hour instruction in medline  geriatrics clerskship: questions from course director for smallgroup discussion ghali et al. (2000) four 90 minute sessions offered throughout year 3, covering:  ebm steps: developing focused clinical questions from patient care problems encountered in students’ clinical rotations, group discussion of actual clinical scenarios, formulating a specific question, conducting evidence based literature searching for articles, and selecting one article for critical appraisal  reading/library behaviours  skills and attitudes on issues relating to ebm  self-reported skill survey 2 thomas et al. (2001) year 3 and year 4:  instructions in the structure of an ebm report during clerkship orientation  ninety minute small group session modeling a five step approach with a clinical case  selecting a patient case with a diagnostic or therapeutic dilemma from clerkship experience  ninety minute small group session with students searching for the best evidence, submitting report, and presenting findings to peers  formulation of questions  application of evidence to specific situations  written report graded with a 12-item checklist  program evaluation including an end-ofclerkship assessment and self-assessment 1, 3 srinivasan et al. (2002) short ebm course of eight student contact hours in year 1:  two 1-hour lectures: 1-hour introductory lecture reviewing standard biostatistical concepts and construction of clinical questions and 1-hour introduction of clinical questions  three 2-hour small group sessions on evaluation skills frequently used by clinicians: assessment of risks/benefits of therapeutic interventions and diagnostic test, and of causation of harm; discussing a clinical vignette, developing a relevant question, and evaluating a corresponding article  supplemental online ebm curriculum: 20 page web-based ebm curriculum and a practice  student preparation, performance, participation  utilization of supplemental web curriculum  student/facilitator satisfaction  small-group attendance and facilitator questionnaire  online practice examination and written final examination  usage of online supplemental ebm curriculum  student satisfaction online questionnaire  facilitator satisfaction and experience 1, 2 evidence based library and information practice 2012, 7.3 118 examination questionnaire holloway et al. (2004) twenty to 30 student contact hours of ebm instruction integrated into basic science curricula of preclinical years:  introduction to ebm during first four weeks of curricula: eight to ten student contact hours in the mastering medical information course: an ebm overview, medical informatics, ebm databases, framing a well built searchable question, and critical appraisal of the literature  a series of ebm reinforcements during the reminder of year 1 as part of an ambulatory clerkship course: five student contact hours, including an advanced skills workshop, a patient-centered ebm exercise with individualized feedback, a peer comparison report, and an ebm review  ebm reinforcement continued in year 2: a series of seven monthly, 2hour small group ebm tutorials led by local ebm experts to review clinical cases and practice building clinical questions, searching and appraising the literature  generation of pico questions  medline searching  critical appraisal skills  application of results of appraisal to patients  self-assessment  two test modules evaluating students’ ability to carry out five ebm steps  self-assessment questions  student satisfaction survey and written comments 1, 2 dorsch et al. (2004) eight 1-hour weekly seminars during a 12 week internal medicine clerkship in year 3:  two sessions: defining ebm, formulating clinical questions based on a standardized case scenario, identifying and reviewing ebm search strategies and resources  three sessions: developing critical appraisal skills for therapy, diagnosis, and meta-analysis articles  three sessions: presenting critically appraised topics (cat) based on typical patient care problems encountered during clerkship to consolidate and demonstrate learned skills  performance in applying ebm skills  preand postclerkship survey  preand posttest skills assessment 2 cayley (2005) year 3:  six session curriculum combining exploration of basic principles of  understanding the use of ebm  questionnaire developed by slawson and 2 evidence based library and information practice 2012, 7.3 119 ebm with application of these principles to real-life cases from students’ clinical experience shaughnessy (1999) schilling et al. (2006) year 3: week 1:  two course integrated, web-based learning tutorials on medline and ebm database information retrieval skills, requiring 40-60 minutes to complete week 4  introduction to the national guideline clearinghouse (www.guideline.gov/) from the us department of health and human services, agency for healthcare research and quality (ahrq) week 5  learning how to calculate and interpret nnt (number need to treat) statistic from michigan state university department of family practice’s tutorial introduction to information mastery (www.poems.msu.edu/infomastery /)  information retrieval skills  ebm practice skills  perception about clerkship experience  medline literature search strategies  articles identified by students as providing the best evidence to address a clinical case  post-clerkship nnt test (calculating a nnt from a hypothetical clinical trial)  post-clerkship survey 1, 2 nieman et al. (2009) year 1:  mandatory 2-hour orientation workshop: asking a clinical question about a sample case, accessing databases (e.g., dynamed), and discussing appraisals of relevant medical literature  elective family medicine preceptorship: completing ebm case summaries for four patients whom students would select with the agreement of their preceptor  documentation of ebm process  self-efficacy  level of ebm learning  preceptors’ attitudes toward using an ebm project as a focus of their feedback  four pico case summaries analyzed with bloom’s taxonomies of cognitive and affective domains  student selfefficacy questionnaire  preceptor questionnaire 2 aronoff et al. (2010) year 3: two parts of an ebm course:  part 1 (18 weeks to complete): six online modules, each of which contained didactic material and a focused practicum/assignment  part 2 (24 weeks to complete): formulating a clinical question generated by a patient seen during each of four clerkship rotations; completing four evidence summaries using critically competence in ebm  a practicum/ assignment each student completed and submitted to an assigned online faculty mentor for review and feedback  completing a cat form  fresno test 2, 3 evidence based library and information practice 2012, 7.3 120 appraised topic (cat) format developed by sackett et al. (1997) administered before and after the ebm course west et al. (2011) end of year 2:  short course of 22 contact hours over a 2-week period: didactic and small-group sessions on full range of ebm skills adapted from users’ guide to the medical literature year 3:  ebm integrated with clinical experiences in each clinical rotation with each student generating a clinical question from a patient encounter, searching for an article addressing the question, critically appraising the article, and producing a brief summary of the evidence and its application to the patient from whom the clinical question arose  ebm skills and knowledge  self assessment of the importance of ebm for medical education and clinical practice  berlin questionnaire  fresno test  self-rated ebm knowledge and assessment of importance of ebm for medical education 2 sastre et al. (2011) year 3:  a single, physician-led, hands on 3hour workshop teaching clinical question formation, locating and using pre-appraised resources, discussing strengths and weakness of available pre-appraised resources  attitudes and knowledge about literature searching, prior use of ebm resources, and knowledge of clinical question formation and searching techniques  impact on patient care  preand postsurveys  inpatient admission notes as a surrogate marker of impact on patient care  computer log data of students’ searching of various ebm resources using hyperlinks and search tools integrated within an electronic medical resource system 1, 2, 3 microsoft word es 4461 evidence based library and information practice 2009, 4:1 17 evidence based library and information practice evidence summary female public library patrons value the library for services, programs, and technology a review of: fidishun, dolores. “women and the public library: using technology, using the library.” library trends 56.2 (2007): 328-43. reviewed by: virginia wilson shirp coordinator, health sciences library, university of saskatchewan saskatoon, saskatchewan, canada e-mail: virginia.wilson@usask.ca received: 01 december 2008 accepted: 19 january 2009 © 2009. wilson. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. objective – this study attempts to give insight into why and how women use the public library and information technology, and how they learned to use the technology. design – qualitative survey. setting – the research took place at the chester county library in exton, pennsylvania, usa. subjects – one hundred and eighty-four female library patrons 18 years and older. methods – an anonymous qualitative survey was handed out to all patrons at the chester county library 18 years of age and older who came into the library on four separate days and times. times were chosen to obtain a good representation of library patrons, and included daytime, evening, and weekend hours. the survey consisted of questions about library use, information sought, information seeking behaviour, technology used, and how the respondents learned to use the technology. the surveys were collated and spreadsheets were created that reported answers to yes/no and other data questions. word documents facilitated the listing of more qualitative answers. the data were analyzed using a thematic content analysis to find themes and patterns that emerged to create grounded theory. in thematic content analysis, “the coding 18 scheme is based on categories designed to capture the dominant themes in a text (franzosi 184). there is no universal coding scheme, and this method requires extensive pre-testing of the scheme (franzosi 184). grounded theory “uses a prescribed set of procedures for analyzing data and constructing a theoretical model” from the data (leedy and ormrod 154). main results – the survey asked questions about library use, reasons for library use, using technology, finding information, and learning to use online resources. a total of 465 surveys were distributed and 329 were returned. from the surveys returned, 184 were from female patrons, 127 from male patrons, and 18 did not report gender. the data for this article are primarily taken from the 184 female respondents who reported ages between 18 and 79 years. seventy-one percent of these reported having a bachelor’s degree or higher. the study uses some contrasting data from the men’s responses where appropriate. in terms of library use, out of the 184 respondents, 42% came to the library monthly, while 36% visited the library weekly. sixty-two percent of respondents knew they could email the library and 72% knew that they could call the library with questions. as for reasons for library use, the most prominent response was to borrow books rather than buying them. the second most common reason for using the library related to children’s books and programming for children. other common reasons for library use included research activities, using public computers, reading, use of services such as photocopying and tax forms, and to volunteer or tutor. the library was also used as a place of solitude, where women could find a place and time for themselves. the author compared the men’s results to the women’s responses, and found that coming to the library for books was lower on the list, and very few men mentioned children’s library services. men came to the library more often than women to study or read. in terms of using technology, the female respondents were fairly tech-savvy. seventyfour percent of respondents felt comfortable using computers. only 5% replied that using computers meant more work for them. eighty-two percent said they used a computer on a regular basis, and 98% reported that they had used the internet. out of those who use the internet, 91% used it at home, 64% used it at work, and 34% used it at the public library. ninety-eight percent of women who used the internet used a search engine such as google or yahoo to find information. topics frequently mentioned were medical and travel information, information for their children, and shopping. men, by contrast, listed shopping and finding medical information as their second reason for using the internet. general research topics were most frequently cited by men. seven survey questions focused on finding information. the internet was the number one choice for finding health information, sports scores, the date of thanksgiving, and the phone number of their state senator. the library was the first place to find a good book. results indicated that although women use libraries to find information, they use the internet more, as libraries were at least third on the list of places women looked for most of the topics inquired about. when asked about their computer use, 71% of respondents said they used a computer to gain information for work, 74% said they used it for hobbies, and 81% used it to access medical information on the internet. sixtyfive percent of respondents used email and chat to keep in touch with family and friends. 30% of the women asked felt that books were more valuable than using a computer. forty-six percent reported that being able to ask a librarian for help was an 19 appreciated service. the use of library technology figured in the survey. seventytwo percent of respondents reported that they were comfortable using the online catalogue and 53% said they used the library’s webpage. only 19% said they used the library’s databases. the comments section of the survey included evidence that the women either did not know these electronic resources existed, or they did not understand what databases are for. however, 47% said they had access to online databases from other sources, for example, higher education institutions, public schools, businesses. those who did use online databases were asked how they learned to use them. sixteen percent were self-taught. only a few had formal training, including 3% who were taught by a public library staff member. sixty percent of respondents indicated they would like formal training: 23% preferred individual training, while 77% preferred training in a class setting. the survey attempted to discern the value of participants’ library experience by using positive and negative critical incidents. the participants responded to questions about their best and worst experiences using the library. best experiences included those involving books; children’s literature, programs, and family projects; library technology; access to non-print materials; the library as a place for solitude; other library services; and library staff. the negative experiences included library issues such as having to return books on time, getting an overdue notice or fine on an item already returned, and desired books being out of the library, noise in the library. the number of positive experiences reported was higher than the number of negative experiences. conclusions – although definitive conclusions are difficult to make using qualitative analysis, fidishun summarizes her findings by reporting that her study of women public library patrons found that technology features prominently in women’s lives, and that they regularly use the internet to find information. however, many women were not aware of the databases available at the public library. books were an important part of the library experience for these women, as were traditional library services, such as asking a librarian for help. women often are the ones who bring children to the library and seek information for them. and the women surveyed valued the library as place. commentary this is an interesting study with a robust sample size that reveals some useful information about women and how they use the public library and technology. the author discusses the trustworthiness of her study. trustworthiness in qualitative research is akin to validity, reliability, and credibility in quantitative research. in order to attempt to ensure the trustworthiness of the data, the author employed triangulation. as she states herself, triangulation involves “multiple forms of evaluation,” that is, more than one data collection method being used in a study (331). it appears that the only method used in this study was the survey method. the survey included questions posed in more than one way, which is not how triangulation is usually applied. postsurvey, the author discussed the results with a number of public librarians to obtain a verification of the information, but she does not suggest that talking with the librarians after the survey was completed was actually a separate data collection method. more elaboration on this process would have been helpful and the researcher should have described her use of triangulation in more detail. it also would have been useful to know if she pre-tested her instrument to 20 ascertain that her safeguards for data trustworthiness were on the mark. the researcher mentions the use of grounded theory as part of her methodology. there is a lack of discussion around grounded theory in the article, and it is unclear if the proper grounded theory approach has been used. although grounded theory requires extensive pre-testing of the data analysis scheme, there is no evidence of pre-testing in the article. the author acknowledges that as a qualitative study, the results cannot be generalizable to the entire public, but are reflective of the patrons who were surveyed. there are a few other points to make note of in this research article. the researcher, admittedly not employed as a public librarian, attempted to put herself into a frame of mind that reflected the mission and clientele of the public library. the literature review is not especially robust, and it is quite outdated. it is unknown whether there were no previous studies to be found, or whether the reviewer did not flesh out the literature review piece of the article, but some mention should have been made about previous studies: their availability or lack thereof. the researcher mentions the diversity of public library patrons. however, 71% of the sample had bachelor’s degrees or higher. there should be some discussion on how this influences the findings, or at the very least, an acknowledgement of this fact. the delivery of the results of the survey was problematic in that some of the data were reported in percentages and some were described in more nebulous terms such as “a few,” “some,” “several,” “a fairly large number,” and “the most prominent response.” as well, five tables were provided in the article, reporting on questionnaire results. it is unclear whether these tables are meant to encompass the totality of the survey, or if they just include a few key details. it would have been helpful had the author included the survey instrument as an appendix to the paper, not only for possible replication purposes, but so the reader could perhaps have a clearer understanding of the data analysis. the discussion and conclusion sections of the paper were light, consisting mainly of paraphrasing and further summarization of the data analysis section. the author concludes that libraries and librarians are still viable and necessary; women are driven by their traditional familial roles to visit and use the library; and women use technology in variety of settings, including the public library. more discussion on the setting of the library would have been helpful in providing more context for the women surveyed. although stronger conclusions were not made, this paper can be useful for public libraries and librarians who are interested in knowing more about how their female patrons are using their services. in particular, results around how female patrons would like training delivered, their lack of understanding of databases, and their use of the library as it reflects their familial role, can all be helpful when planning public library programming. works cited franzosi, roberto. “thematic content analysis.” the sage encyclopedia of socialscience research methods. eds. michael lewis-beck, alan e. bryman, and tim futing liao. thousand oaks, ca: sage, 2004. leedy, paul d. and jeanne ellis ormrod. practical research planning and design. 8th ed. upper saddle river, nj: merrill prentice hall, 2005. evidence based library and information practice 2012, 7.4 111 evidence based library and information practice evidence summary evidence from students’ information seeking diaries underscores the importance of including librarians in undergraduate education a review of: lee, j. y., paik, w., & joo, s. (2012). information resource selection of undergraduate students in academic search tasks. information research, 17(1), paper511. retrieved 8 aug., 2012 from http://informationr.net/ir/17-1/paper511.html reviewed by: maria melssen medical librarian, independent contractor port clinton, ohio, united states of america email: mariamelssen@gmail.com received: 14 aug. 2012 accepted: 27 sept. 2012 2012 melssen. this is an open access article distributed under the terms of the creative commons‐ attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐ sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to determine what information resources undergraduate students choose to complete assignments for their courses, why they choose those resources, the process of selecting those resources and the factors that contributed to selecting the resources, and their perceptions of those resources. design – semi-structured information seeking diary. setting – private university in seoul, korea. subjects – 233 undergraduate students from all majors and all years. methods – students selected one assignment from their elective course and recorded the following in a diary: what the assignment was, the topic they needed to research to complete the assignment, resources used, the factors that contributed to choosing the resources, and perceptions of those resources. main results – data were analyzed both qualitatively and quantitatively. the factors that affected the students’ resource selection were analyzed qualitatively using an open coding method created by the researchers. the factors were not predetermined by the researchers, but were selected based on the factors identified by the students. online resources (67.1%) were the most frequently selected resources by the students compared to human resources (11.5%), print materials mailto:mariamelssen@gmail.com http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/ http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/ evidence based library and information practice 2012, 7.4 112 (11.5%), and mass media (3%). students used an average of 5.28 resources to complete one assignment. factors that affected the students’ selection of resources were the type of information provided by the resource, the features of the resource, the search strategy used when searching in the resource, and the students’ interaction with other people when selecting and using the resource. more than one factor typically contributed to the students’ selection of the resource. the students’ perceptions of the resources they selected were analyzed quantitatively: perceptions were analyzed in six content areas using a five point scale. correlations and similarities across the six content areas were also analyzed. perceptions of resources were broken down into six categories and the resources were rated on a five point scale. librarians (4.50) were the most useful resource and lecture notes (5.0) were the most credible. family (3.29) was the least useful, and social question and answer services, such as yahoo! answers, (2.62) were the least credible. family was the most accessible and familiar resource (4.90 and 4.95 respectively). experts (2.25) were least accessible and librarians (1.50) were the least familiar. students were most satisfied with google scholar (4.33) and were most likely to use an online database (4.52) again. they were least satisfied with social question and answer services (3.05) and least likely to use a report sharing site (2.93) again. the usefulness and credibility of the resource contributed most to the undergraduates’ satisfaction with the resource, while accessibility and usefulness were the major contributing factors to users intended continued use of a resource. conclusion – there are multiple reasons that support further information literacy education. information literacy courses would encourage and teach students how to effectively use resources that they found credible and reliable, but considered inaccessible and unfamiliar. information literacy education would also help educate students on how to best formulate their search strategies and how to select the best resource to use based on that strategy. students also highlighted the importance of human interaction in resource selection and utilization. this is an opportunity for librarians and professors to play a more active role in assisting students in selecting and using the best resources to complete course assignments. due to the limitations of this study, further research is needed to investigate the factors affecting the exclusion of resources, not only the inclusion. future study designs should address the characteristics of the study participants themselves, such as the age and gender. the impact of the research topic and subject on resource selection, as well as what resources are selected for non-academic information needs, should also be investigated. commentary this study supports and complements the current body of literature regarding information resource selection of undergraduate students. the findings are similar to other studies in regard to student resource preferences; however, the study design is unique and examines student resource selection, usage, and perceptions in real time. critical appraisal of this study was completed using the evidence based library and information practice critical appraisal checklist (glynn, 2006). the validity of the study was analyzed in four content areas: population, data collection, study design, and results. validity of the data collection, study design, and results are strong, while validity for population selection is questionable. though the sample size and response rate are appropriate, and according to the authors is representative of the population being studied, the selection of study participants is problematic. selection of the participants was not randomized, nor were inclusion and exclusion criteria clearly outlined. gender, age, and confounding variables, such as year of study, were not taken into account. data collection is clearly defined and the timing of data collection is appropriate; however, the study design was not validated and neither informed consent nor ethics approval is mentioned in the study. evidence based library and information practice 2012, 7.4 113 despite the concerns regarding population selection, this study has many strengths. all who volunteered for the study completed the study. the study design is simple and appropriate for the objectives. in regard to the collection of the data, the researchers validated their content analysis with inter-coder reliability to ensure the reliability of the coding method they used. of most interest is the use of diaries and allowing participants to selfselect their own topics. unlike other studies that collect data after someone has done a search or relying on the participants to remember why and what they chose to search for, this study collected the data in real time. also, the results of the study were clearly explained and could be applied at similar institutions. this study provides evidence of the importance of involving librarians in undergraduate education and the necessity of information literary instruction for undergraduates. results from this study could help justify having librarians as members of the university’s faculty and librarians being included in the development of undergraduate curriculum. references glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi: 10.1108/07378830610692154 evidence summary   manuscripts published in a specific chemistry journal must be both important and suitable according to peer reviewers   a review of: bornmann, l., & daniel, h.-d. (2010). the manuscript reviewing process: empirical research on review requests, review sequences, and decision rules in peer review. library & information science research, 32(1), 5-12. https://doi.org/10.1016/j.lisr.2009.07.010   reviewed by: michelle dubroy discipline librarian, researcher services griffith university southport, queensland, australia   received: 25 feb. 2021                                                              accepted:  9 apr. 2021      2021 dubroy. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29936     abstract   objective – to examine the peer review process at a single journal.   design – analysis of business records.   setting – peer review system of a single journal.   subjects – documents produced when reviewing manuscripts submitted for publication to journal angewandte chemie international edition and reviewed in the year 2000.   methods – peer review process information was extracted from the journal’s archives. various aspects, such as review sequences and decision rules, were analysed and summarised in tables.   main results – of the 1899 manuscripts reviewed in the year 2000, 46% (n = 878) were accepted for publication and 54% (n = 1021) were rejected. on average, a manuscript received 2.6 reviews before an editor made a publication decision. just over half (n = 962, approx. 51%) of manuscripts were subject to two review steps. a small number of manuscripts (n = 104, approx. 5.5%) were subject to 5, 6 or 7 review steps. the more steps an article was subject to, the greater likelihood it would be accepted. editors “generally follow a so-called clear-cut rule” (p.11) in which manuscripts accepted for publication must be considered both important and suitable for publication by at least two peer reviewers.   conclusion – the results “give a sense of commitment [and care] ...probably typical of most prestigious journals” (p.11).   commentary   peer review is a fundamental part of the research process. despite its importance, traditional peer review is said to lack sufficient transparency, accountability, and consistency (ross-hellauer, 2017).   this study was likely the first to examine the inner workings of the peer review system. years later, our understanding of peer review has greatly increased, yet many challenges remain (tennant, 2018).   the study was evaluated using a critical appraisal tool (perryman & rathbun-grubb, 2014). while the study was successful in opening the “black box” (p.5) of peer review at a single journal, it also suffered from a few weaknesses.   a concise literature review gives adequate background and context. the methods generally appear logical for addressing the stated objectives. further, the authors reported their findings thoroughly and made good use of tables.   reproducibility is unfortunately limited. many details regarding the research process are unknown. did the authors code the information in some way? did they use software for the analysis? readers can only guess.   further, the intended audience and utility of this study is unclear. the authors refer to library “collection managers” (p. 11), but they do not suggest how this group might apply the results of the study to their practice. additionally, the authors do not discuss how other users, such as researchers, might make practical use of the study’s findings. clearer research questions, focused on supplying readers with meaningful answers (doolan & froelicher, 2009), may have given the analysis needed direction.   overall, this notable study revealed details regarding an important, yet relatively opaque, part of the research ecosystem. it may be of some interest to academic librarians who support researchers in their scholarly publishing. however, clearer, more audience-focused research questions may have helped increase the study’s usefulness. additionally, more transparent research processes would have enabled other researchers to productively build on their work.   references   doolan, d. m., & froelicher, e. s. (2009). using an existing data set to answer new research questions: a methodological review. research and theory for nursing practice, 23(3), 203-215. https://doi.org/10.1891/1541-6577.23.3.203   perryman, c., & rathbun-grubb, s. (2014). the cat: a generic critical appraisal tool. https://www.jotform.us/cp1757/thecat   ross-hellauer, t. (2017). what is open peer review? a systematic review [version 2; peer review: 4 approved]. f1000research, 6(588). https://doi.org/10.12688/f1000research.11369.2   tennant, j. p. (2018). the state of the art in peer review. fems microbiology letters, 365(19). https://doi.org/10.1093/femsle/fny204   evidence based library and information practice 2011, 6.4 1 evidence based library and information practice editorial renewal and change denise koufogiannakis editor-in-chief collections & acquisitions coordinator, university of alberta libraries edmonton, alberta, canada email: denise.koufogiannakis@ualberta.ca 2011 koufogiannakis. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. the end of yet another calendar year leads to a time of new beginnings and new challenges. for me, this is the end of my three year term as editor-in-chief of eblip, which has been a very rewarding challenge personally and professionally. i would like to thank each and every person who has contributed to the success of the journal. eblip relies on the talented and professional people who regularly give of their time and expertise. i particularly want to thank the editorial team who has worked so closely with me over the past three years: alison brettle, lorie kloda, katrine mallan, jonathan eldredge, michelle dunaway, and our former intern andrea baer. thank you! in the past three years, eblip has continued to grow and thrive. we currently have more than 3400 registered readers, and an editorial advisory board comprised of 70 people from 11 countries. several of our papers have had more than 5000 pdf downloads, and the average number of downloads is 1308. i am also pleased to let you know that we have recently been accepted for inclusion in scopus. in looking back and reflecting on the past three years, we have continued to build a strong open access journal that is relevant to lis practitioners. we have increased the number of articles being published, and the overall number of submissions. content is wide ranging, including multiple sectors within library and information studies. our evidence summaries continue to be the heart of our journal, with more and more research from our field being critically appraised in this way. we have also added new sections such as using evidence in practice, in which authors reflect on incorporating evidence into practice. and now it is time yet again for renewal and change. i am very pleased to welcome alison brettle of the university of salford, as she begins her three year term as editor-in-chief. alison has been the associate editor (articles) since the mailto:denise.koufogiannakis@ualberta.ca� evidence based library and information practice 2011, 6.4 2 very early days of publication and brings extensive knowledge of evidence based practice to the role. alison was recently named as one of four “information hubs” by participants in a study by the u.k.’s lis research coalition’s rilies project which is investigating the impact of lis research on lis practice – a great honour! eblip readers are lucky to have alison take over as editor-in-chief! as for me, i will be sticking around as associate editor for a new section we are adding to eblip – reviews. review articles provide a way for librarians to obtain an overview of the evidence on a particular topic and stay current with the literature. reviews may take on several different forms, including an overview, systematic review, meta-analysis, literature review, or stateof-the-art review, among others. for a detailed explanation of the types of reviews possible, please consult grant and booth (2009). we are particularly interested in reviews which contain information on the state of research on a particular topic, so if you are interested in exploring a topic and writing about it, please be in touch. this issue of eblip contains several written versions of presentations given at the eblip6 conference in june of this year. in the feature section, we have keynote addresses, a winning presentation, and reflections on the conference from participants. as well, all of the papers in the peer-reviewed articles section are based on presentations given at the conference. in addition to the conference content, lorie kloda has put together a record number of evidence summaries for you. enjoy! references grant, m. j., & booth, a. (2009). a typology of reviews: an analysis of 14 review types and associated methodologies. health information and libraries journal 26(2): 91108. doi: 10.1111/j.14711842.2009.00848.x eblip editors at the eblip6 conference: alison brettle, lorie kloda, denise koufogiannakis, jonathan eldredge / evidence based library and information practice microsoft word comm_hallam.doc evidence based library and information practice 2006, 1:3  88 evidence based library and information practice     commentary    evidence based library and information practice: whose responsibility is it anyway?      gillian hallam   senior lecturer  school of information systems  faculty of information technology  queensland university of technology, brisbane, australia  e‐mail: g.hallam@qut.edu.au      helen partridge  lecturer  school of information systems  faculty of information technology  queensland university of technology, brisbane, australia  e‐mail: h.partridge@qut.edu.au      © 2006 hallam and partridge. this is an open access article distributed under the terms of the creative  commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      increasingly, the discipline of library and  information science (lis) is challenged to  engage in research in order to “create new  knowledge and thereby contribute to the  growth of lis as a profession or discipline.  if research is absent, non existent or even  scarce, there is no profession, but only an  occupation grounded in techniques, routine  and common sense” (juznic and urbanija  325).  in the united kingdom in 2001, the  centre for information research was  commissioned by the chartered institute of  library and information professionals  (cilip) to conduct an examination into the  research landscape of library and  information science.  the examination  concluded that the value of research in the  lis discipline can be experienced on both  the professional and personal levels.   at the  professional level, research can inform  practice, assist in the future planning of the  profession, and raise the profile of both the  discipline and library and information  service itself.  at the personal level, research  can “broaden horizons and offer individuals  development opportunities” (mcnicol and  nankivell 77).  it seems, however, that  evidence based library and information  practice is currently a minority interest,  rather than a part of the mainstream.  mcnicol and nankivell wrote, “work is  needed to stimulate greater interest in, and  respect for, research within the lis  community” (82).  they recommended that  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:3  89 “research should be promoted as a valuable  professional activity for practitioners to  engage in” (82). the study by cilip  concluded that for this to occur, there must  be an “obligation on the part of library  schools, employers, and professional bodies  to ensure that practitioners have the  necessary skills to be able to conduct  research effectively” (mcnicol & nankivell  82).     as lis educators we can focus on the role  that library schools can play to foster a  research culture within the profession.  we  can introduce specific teaching and learning  models to help develop student awareness,  understanding, and skills in evidence based  information practice.  but our big challenge  is to encourage the other stakeholders –  individual professionals, employers, and  professional associations – to play a role in  working collaboratively with us to develop  the culture of evidence based practice (ebp)  that, ideally, should pervade our profession.    in this article we review some of our ideas  and experiences that have contributed to  shift the professional horizons of our own  goals.      the need for engagement    there is a certain degree of irony inherent in  our professional shortcomings, our lack of  focus on research within the discipline.   crumley and koufogiannakis observed that  “in our profession we help our patrons  make decisions by leading them to research  evidence.  it is vital that we follow the same  model: we should consult our own  literature when we have questions about  best practices in our field” (112).  ritchie  also noted that given our role as managers  of the literature of research, library and  information professionals are uniquely  placed “to model the principles of evidence  based practice, not only as they apply to  other disciplines which we serve, but also as  they apply to our own professional  practice. . . . ’if you are not modelling what  you are teaching, you are teaching  something else’” (6).  library and  information professionals should indeed  ‘practice what they preach’.  in the context  of teaching, library schools undoubtedly  have a critical role to play to foster an  appreciation of research and its relationship  to the growth of the profession as a whole.   brice et al. propose that “lobbying  educational institutions to increase research  and appraisal skills teaching in the  curriculum” (289) is one of several short  term priorities required to both sustain the  current interest in and to engage new  activists in the evidence based phenomenon.     at the 2nd international evidence based  librarianship conference held in 2003,  schrader commented on the incongruity  between the prominence of research in the  ala standards for accreditation of masters  programs in library and information  studies and the reality of current library  education in the u.s.  schrader lamented,  “why research training is not a mandatory  component of every accredited program  remains a mystery to me, and why deficient  programs are able to get and continue their  accredited status is an even bigger mystery”  (171).  these concerns may indeed echo  across the educational institutions in other  countries; both lis educators and the  accrediting bodies should heed the need to  take action.  in addition, as the call for  certification of practicing librarians becomes  more strident, the potential for professionals  to demonstrate knowledge and skills in  research moves beyond the campus and into  the workplace.    identifying the stakeholders    booth has observed that one of the main  assumptions in fostering a culture of  evidence based practice is that “practitioners  are enlightened enough to want to practice  evidence based practice” (“mirage” 57).  for  evidence based library and information practice 2006, 1:3  90 evidence based library and information  practice to succeed, library and information  professionals need to be willing to develop  the skills and knowledge essential for ebp  and to be motivated to use these within the  context of their day‐to‐day work.  whilst  current lis education can help to foster the  birth of a new breed of evidence based  information professionals, ultimately it is up  to the new professionals working in  industry to put into place the skills they  have learned during their academic studies.   it is also the responsibility of those  professionals who have been in the field for  some time, and who have not had the  opportunity to learn about ebp via their  formal library studies, to develop the  necessary skills, knowledge, and  understanding of ebp by undertaking  professional development activities.      however, individual professionals clearly  cannot succeed in developing a culture of  ebp without the support of both their  employers and their professional  associations.  employers have a  responsibility to provide an environment  that encourages and rewards ebp and to  provide access to the resources and training  needed to implement ebp within the work  context.  professional associations have a  responsibility to encourage, enable and  reward the development of ebp skills and  knowledge within the profession as well as  developing and articulating the profession’s  ebp policies and future directions.    these views suggest that, whilst formal lis  education can commence the process,  success will only be achieved when all  stakeholders in the profession actively work  together to build the desired culture.  for  the lis profession this means cooperation  and collaboration between professionals,  educators, employers and our professional  associations. we have developed a matrix  (table 1) that seeks to identify the  contribution that could feasibly be made by  each of the different stakeholders in  pursuing the ebp goal.           stakeholders  roles or responsibilities    professionals    individuals have a responsibility to:  • have the motivation to be evidence based information professionals.  • ensure that they acquire and maintain the knowledge and skills  necessary for evidence based practice by completing professional  development programs.   • undertake work‐based or higher degree research and actively take a  role in establishing a clearer and stronger link between theory and  practice within the profession.      educators    educators have a responsibility to:  • provide and promote educational courses and qualifications necessary  for developing the skills required of evidence based information  professionals.  • seek opportunities for updating their own practical skills and  knowledge to ensure relevancy and currency of curriculum.  • provide and promote research degrees such as masters by research and  evidence based library and information practice 2006, 1:3  91 phds to the profession.  • develop and deliver professional development courses in the area of  evidence based practice.  • actively pursue research activities to generate new theoretical  knowledge.  • seek opportunities for cooperative research and development activities.    employers    employers have a responsibility to:  • provide the opportunities and resources for professionals to engage in  evidence based practice, including dissemination of findings to the  profession.  • encourage and allow their staff to engage in ongoing learning and  professional development necessary for maintaining skills and  knowledge in evidence based practice.  • provide opportunities for lis students to learn about evidence based  practice in situ through fieldwork and project work.  • participate in lis course advisory work.  • serve as guest lecturers and tutors within lis education programs.  • include evidence based practice activities as part of the staff appraisal  program.  • encourage mentoring of evidence based practice for novice or  inexperienced research‐professionals.  • provide opportunities for collaboration with academic researchers and  other industry employers on research project and funding.      professional  bodies    the professional association has a responsibility to:  • encourage, enable, and reward the learning and development of skills  and knowledge in evidence based practice within the profession.   • determine and articulate a policy position on the role of evidence based  practice for the profession as a whole and in the education for library  and information professionals specifically.  • encourage employers to support evidence based information practice  by recognising institutional members’ involvement in, and  dissemination of, evidence based projects as eligible for an individual’s  professional development points.  • ensure that the profession’s research agenda, priorities, values, and  expectations are heard in government and by other key policy and  decision makers.  • offer research awards and funding opportunities.  • facilitate forums for sharing and dissemination of evidence based  practice knowledge.  • arrange partnerships with training providers to provide training  courses in evidence based practice tailored to the specific needs of  diverse contexts (e.g., public libraries or law libraries).       table 1: stakeholders’ roles and responsibilities  evidence based library and information practice 2006, 1:3  92   strategies to engage all stakeholders    as lis educators in the masters of information  management program at the queensland  university of technology (qut), we feel that we  can encourage professional discussion at the local  level through our involvement with industry  professionals, employers and the professional  association, both in and beyond the classroom.   central to our teaching philosophy is the belief in a  holistic approach to student learning where the  personal and professional dimensions are  intertwined (partridge and hallam).  we propose  that this holistic focus in lis education can help the  twenty‐first‐century library and information  professional learn about the diverse research  methodologies that can be applied in the workplace  to gather, review, evaluate, and disseminate key  research data.  the student is given the opportunity  to develop into a ‘reflective practitioner’, as  proposed by booth, with the ability to “critically  analyse [and] make informed judgements”  (“systems” 70), drawing on a toolbox of skills and  knowledge which will include evidence based  practice.  to train the new evidence based  information professional, lis education must  therefore focus on developing students’ skills as  reflective practitioners who are not only discipline  savvy but are also equipped with a rich portfolio of  generic capabilities, including communication skills,  teamwork, problem solving, and critical thinking.    critical reflection is increasingly identified as a core  skill required by all professionals.  for many  students entering the masters course, reflective  practice is a novel and unexplored – and often  challenging – domain.  taking this into  consideration, the program has been designed to  provide students the opportunity, over the three  semesters of study, to progressively gain both  theoretical understanding and practical experience  in reflective practice.  four units over the three  semester program have been developed to provide  the context for the students to learn about and to  experience reflective practice firsthand.  students  are encouraged to use reflective practice as a tool  both for self evaluation and for the review of team  processes, to compose a reflective learning journal  to support the construction of their own     understanding and knowledge development, and to  conclude their  studies with the creation of a  professional portfolio which presents their  reflections on their own role as an information  professional.     in addition to the blend of discipline knowledge  and generic capabilities, the masters program  includes authentic learning activities and the direct  involvement of industry professionals.  the  involvement of industry professionals contributes  significantly to the success of our holistic model of  lis education.   industry professionals provide the  authentic context from which students can directly  learn about discipline knowledge and generic  capabilities within the profession. they contribute  to the lis program through three vital roles: as  guest speakers, as mentors, and as industry  partners in the learning activities.  it is through the  direct involvement of professionals that students  grasp the amazing diversity of information  problems and the range of possible approaches to  solve them. through the learning tasks developed  collaboratively with industry professionals and  employers, students are offered a fertile context to  apply the skills and knowledge they acquire within  the academic confines of their studies.      fieldwork placements and industry based projects  are an invaluable source of linking theory to  practice within an authentic learning environment.   increasingly, libraries and information agencies in  the region are contacting us to initiate interesting  and challenging project work that not only enables  students to become directly involved in authentic  research activities, but it also encourages the cross‐ fertilisation of ideas and skills between the students  and the professionals.  such opportunities help  students – and professionals – begin to think  critically about the skills, knowledge, attributes,  conceptual structures, and thinking processes that  they need as working information professionals.  ideally, it will also see industry professionals  inspired to enrol in masters by research courses, to  help build a research culture and to increase the  capacity for knowledge development with the lis  discipline.      evidence based library and information practice 2006, 1:3  93 in our roles as joint‐chairs of the 3rd evidence  based librarianship conference held in brisbane,  australia, in october 2005, we were able to involve  the national professional association as  the host for  the event.  the australian library and information  association (alia) has consequently   played an important role in encouraging  professional debate, resulting in a better  understanding of the issues, thereby inviting a  stronger degree of commitment on the part of both  professionals and employers.  whilst lis education  can adopt a proactive stance in fostering ebp within  the profession by including relevant and necessary  skills within the lis curriculum, an evidence based  culture within the library and information  profession will not be achieved unless all  stakeholders are actively and proactively engaged  with the process.    conclusion    our own teaching and learning model is just one  possible approach to embedding ebp within lis  education. we do believe that fostering a culture of  ebp requires lis education to serve as the  cornerstone to equip library and information  professionals with the necessary skills and  understanding to become reflective practitioners in  an evidence based industry, with the aim of  progressing professional knowledge.  accordingly,  we strive to develop a curriculum which focuses on  content that is topical and relevant, but which at the  same time helps develop the individual dimensions  of the students so that they graduate with a sound  understanding of the full spectrum of professional  and personal attributes that they will need for  successful careers as lis professionals. it would be  beneficial for lis education, and for the future of  the library and information profession in general, if  other teaching and learning models were presented  and critically discussed. this profession requires an  informed discussion at both the national and  international levels about the specific skills and  knowledge required by the evidence based library  professional, about the best teaching and learning  approaches that will help to prepare and educate  future evidence based professionals, and about  ways to firmly establish an evidence based culture       within the profession, so that the profession itself  truly has a future.      works cited    booth, andrew. “mirage or reality?” health  information and libraries journal 19.1(mar.  2002): 56‐58.    ‐‐‐.  “where systems meet services: towards  evidence based information practice.” vine  33.2 (2003): 65‐71.    brice, anne, andrew booth, ellen crumley, denise  koufogiannakis, and jon eldredge. “a future  for evidence based information practice?”  evidence based practice for information  professionals: a handbook.  eds. andrew  booth and anne brice. london: facet, 2004.  279‐92.    crumley, ellen, and denise koufogiannakis.  “developing evidence based librarianship:  practical steps for implementation.” health  information and libraries journal 19.2 (2002):  61‐70.    day, j.  “curriculum change and development,”  the education of library and information  professionals in the united kingdom.  eds.  judith elkin and tom wilson, [london]  continuum international publishing group‐ mansell, 1997. 31‐52.    juznic, primoz, and jose urbanija. “developing  research skills in library and information  studies.” library management 24.6‐7 (2003):  324‐31.    mcnicol, sarah, and clare nankivell.  “lis   research landscape: a review and  prognosis.” cirt online project and  publications archive. 2003. cirt / centre for  information research. 1 may 2005  <http://www.ebase.uce.ac.uk/cirtarchive/>.                                            http://www.ebase.uce.ac.uk/cirtarchive/ evidence based library and information practice 2006, 1:3  94 partridge, helen, and gillian hallam. ”the double  helix: a personal account of the discovery of  the structure of [the information professional’s]  dna.”  australian library and information  association (alia) biennial conference, gold  coast, australia, 21‐24 september 2004. 10  may 2005     <http://conferences.alia.org.au/alia2004/pdfs/p artridge.h.paper.pdf>.                                                                                     ritchie, ann.  “evidence based decision making.”  incite 20.12 (1999). 10 may 2005     <http://www.alia.org.au/publishing/incite/199 9/12/appraisal.html>.    schrader, alvin. m. “the three cultures of  librarianship: a personal odyssey through  education, research and service.”  bibliotheca  medica canadiana 24.4 (2003): 169‐74. http://conferences.alia.org.au/alia2004/pdfs/p http://www.alia.org.au/publishing/incite/199 evidence summary   university instructors use of, and satisfaction with, library services in the year following the covid-19 outbreak   a review of: mcclure, j. (2023). the covid-19 pandemic and the rapid shift to an exclusively online format: tracking online instructors’ utilization of library services over a year of virtual learning at the university of memphis. college & research libraries, 84(1), 100–120. https://doi.org/10.5860/crl.84.1.100    reviewed by: andrea miller-nesbitt associate librarian schulich library of physical sciences, life sciences, and engineering mcgill university montreal, quebec, canada email: andrea.miller-nesbitt@mcgill.ca   received: 7 sept. 2023                                                               accepted: 13 oct. 2023      2023 miller-nesbitt. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30440     abstract   objective – to determine online instructors' satisfaction with and level of use of library services during the year following the onset of the covid-19 pandemic.   design – survey questionnaire and follow-up interview.   setting – the university of memphis (uofm) is a medium-sized, public, urban, r2 (doctoral university with high research activity) university. at the time of publication, uofm employed 930 full-time faculty and—through uofm global—the option to earn degrees entirely online.   subjects – survey respondents (n = 56) were online instructors at the university of memphis.   methods – a confidential survey was distributed to all deans and department chairs at the uofm with instructions to disseminate the survey to all instructors teaching online course(s). respondents were invited to participate in a follow-up interview.   main results – three common themes identified from the data were 1) respondents would have used the enhanced library services but were not aware of them; 2) respondents were very grateful for the services offered, in particular library instruction, kanopy, and interlibrary loan; and 3) respondents did not feel like their courses would benefit from the library or its offered services.   conclusion – based on the research results, the author concludes that the uofm library must focus efforts on increasing visibility and communication, embedding the library in course design and assessment, as well as improving hybrid library instruction and offering purchase-on-demand collection development. the author has begun work on a follow-up study looking at ways to enhance the embedded librarianship service and increase communication between the uofm librarians and online teaching faculty.   commentary   the author was able to draw on research related to a range of topics adjacent to the one addressed in this study, including library use and undergraduate success (mayer et al., 2020), online library services (murray, 2020), and disaster preparedness in academic libraries (mcguire, 2007). there are several relevant publications that discuss online library services in the early days of the pandemic, such as atkinson (2021) and shoaib (2022), that are missing from the literature review, but it is possible that they were published after this article was submitted for review.   the study was appraised using the cat critical appraisal tool by perryman and rathbun-grubbs (2014). the research questions are explicitly stated, and the literature review provides relevant background given the availability of literature on the topic at the time of submission. the chosen methods are appropriate. however, as the author points out, the data may not be representative of all departments at the uofm due to the unknown extent of survey distribution and low response rate from research-intensive departments. both the survey tool and interview questions are included in the study. very few participants (n = 2) agreed to a follow-up interview; however, the survey instrument included several open-ended questions, so the author was able to collect qualitative data.   the quantitative results are presented clearly in the text as well as graphically. the selected quotes from the open ended responses provide insight into the breadth of responses received. these data illustrate the uniqueness of each instructor's perspective on library services. the author helpfully divides the implications of the research into three main sections: marketing and outreach, faculty involvement and training, and enhancement of existing services.   the covid-19 pandemic has had a dramatic and lasting impact on libraries around the world. this study provides insight into how a library at a research-intensive university in the united states responded to the changes required in the early months of the pandemic and how those changes were received by the instructors teaching online. now that the effects of the pandemic have lessened, it would be interesting to determine which services initiated in the spring and summer of 2020 have been maintained and which have been retired. the data suggest that the awareness, perceptions, and practices of the respondents are quite variable. this bolsters the value of the liaison model, whereby academic librarians know the needs and expectations of their departments well and can tailor their offerings accordingly.    references   atkinson, j. (2021). the times they are a-changin’: but how fundamentally and how rapidly? academic library services post-pandemic. in d. baker & l. ellis (eds.), libraries, digital information, and covid (pp. 303–315). chandos publishing. https://doi.org/10.1016/b978-0-323-88493-8.00019-7   mayer, j., dineen, r., rockwell, a., & blodgett, j. (2020). undergraduate student success and library use: a multimethod approach. college & research libraries, 81(3), 378–398. https://doi.org/10.5860/crl.81.3.378   mcclure, j. (2023). the covid-19 pandemic and the rapid shift to an exclusively online format: tracking online instructors’ utilization of library services over a year of virtual learning at the university of memphis. college & research libraries, 84(1), 100–120. https://doi.org/10.5860/crl.84.1.100    mcguire, l. (2007). planning for a pandemic influenza outbreak: roles for librarian liaisons in emergency delivery of educational programs. medical reference services quarterly, 26(4), 1–13. https://doi.org/10.1300/j115v26n04_01   murray, j., & feinberg, d. (2020). collaboration and integration: embedding library resources in canvas. information technology and libraries, 39(2). https://doi.org/10.6017/ital.v39i2.11863   perryman, c., & rathbun-grubb, s. (2014). the cat: a generic critical appraisal tool. http://www.jotform.us/cp1757/thecat   shoaib, m., ali, n., anwar, b., & abdullah, f. (2022). library services and facilities in higher education institutions during coronavirus disease (covid-19) in pakistan. journal of information science. advance online publication. https://doi.org/10.1177/01655515221141035     microsoft word art_oakleaf.doc evidence based library and information practice 2007, 2:3  27 evidence based library and information practice     article    using rubrics to collect evidence for decision‐making: what do librarians need to  learn?      megan oakleaf  assistant professor  school of information studies  syracuse university  syracuse, new york  usa  e‐mail: moakleaf@syr.edu        received: 25 april 2007     accepted: 6 august 2007    © 2007 oakleaf. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    abstract    objective ‐ every day librarians make decisions that impact on the provision of library  products and services.  to formulate good decisions librarians must be equipped with  reliable and valid data.  unfortunately, many library processes generate vast quantities of  unwieldy information that is ill suited for the evidence based decision‐making (ebdm)  practices librarians strive to employ.  librarians require tools to facilitate the translation of  unmanageable facts and figures into data that can be used to support decision‐making.  one  such tool is a rubric.  rubrics provide benefits to librarians seeking to use ebdm strategies.   this study examined librarians’ abilities to use rubrics as a decision facilitation tool,  explored barriers that might prevent effective rubric usage, and suggested training topics  that address potential barriers.    methods ‐ the data for the study came from student responses to open‐ended questions  embedded in an online information literacy tutorial, lobo, used by first‐year students in  english 101 at north carolina state university (ncsu).  fifteen academic librarians, five  instructors, and five students applied rubrics to transform students’ textual responses into  quantitative data; this data was statistically analyzed for reliability and validity using  cohen’s kappa.  participant comment sheets were also examined to reveal potential hurdles  to effective rubric use.    mailto:moakleaf@syr.edu http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:3  28 results ‐ statistical analysis revealed that a subset of participants included in this study  were able to use a rubric to achieve substantially valid results.  on the other hand, some  participants were unable to achieve an expert level of validity and alluded to roadblocks  that interfered with their ability to provide quality data using rubrics.      conclusion ‐ participant feedback can be categorized into six barriers that may explain why  some participants could not attain expert status: 1) difficulty understanding an outcomes‐ based approach, 2) tension between analytic and holistic rubric structures, 3) failure to  comprehend rubric terms, 4) disagreement with rubric assumptions, 5) difficulties with data  artifacts, and 6) difficulties understanding local library context and culture.  fortunately,  each of these barriers can be addressed through training topics that maximize the usefulness  of a rubric approach to ebdm.     introduction    every day librarians make decisions that  impact on the provision of library products  and services.  to formulate good decisions  librarians must be equipped with reliable  and valid data.  unfortunately, many library  processes generate vast quantities of  unwieldy information that is ill‐suited for  the evidence based decision‐making (ebdm)  practices librarians strive to employ.   librarians require tools that facilitate the  translation of unmanageable facts and  figures into data that can be used to support  decision‐making.  one such tool is a rubric.      rubrics are “descriptive scoring schemes”  used to analyze (moskal) and judge the  quality of services, products, or  performances (popham 95).   in libraries,  rubrics can be employed to examine and  evaluate a multitude of library products and  services.  for example, rubrics can capture  useable data about information‐seeking  behavior, customer service skills, marketing  and outreach efforts, collection strengths,  and information commons effectiveness.   this article briefly reviews the benefits of  using rubrics to facilitate ebdm and  outlines a study in which librarians applied  a rubric to capture and understand evidence  of student information literacy skills— evidence that led to the improvement of a  library instruction tutorial.  primarily, this  article focuses on identifying difficulties  librarians may encounter when using  rubrics, and it recommends training topics  to maximize the usefulness of a rubric  approach to ebdm.  additional elements of  the study, including methodology, are  described in other documents and  presentations (oakleaf, “assessing  information literacy”; oakleaf “assessment  of student”).      rubrics defined   rubrics are tools that describe the parts and  levels of performance of a particular task,  product, or service (hafner 1509).  rubrics  are often employed to judge quality  (popham 95), and they can be used across a  broad range of subjects (moskal).  “full  model rubrics” are the most descriptive type  of rubric.  formatted in a chart or table, full  model rubrics include target indicators or  “criteria” in the left column and levels of  performance across the top (callison 34).   the first of two components that comprise a  full model rubric is criteria.  criteria are the  essential tasks or hallmarks that indicate a  successful performance (wiggins v‐6:2).   performance descriptors are the second  component of rubrics.  performance  descriptors “spell out what is needed, with  respect to each evaluative criterion . . . [for]  a high rating versus a low rating” (popham  96).      evidence based library and information practice 2007, 2:3  29 rubrics can be described as holistic or  analytic.  a holistic rubric “score[s] the  overall process or product as a whole,  without judging the component parts  separately” (nitko 226).  holistic rubrics  provide one score for a whole product or  performance based on an overall impression.   analytic rubrics “divide . . . a product or  performance into essential traits or  dimensions so that they can be judged  separately—one analyzes a product or  performance for essential traits.  a separate  score is provided for each trait” (arter and  mctighe 18).  individual scores can be  summed to form a total score from an  analytic rubric (nitko 226).      a number of information literacy rubrics  exist in the library and information science  literature.  the following authors report the  use of rubrics to assess information literacy  in higher education: d’angelo, merz and  mark, rockman, emmons and martin,  buchanan, franks, gauss and kinkema,  hutchins, kivel, kobritz, warmkessel,  smalley, knight, and choinsky, mark, and  murphey.  while authors used rubrics to  evaluate artifacts of student learning in  library instruction, additional research is  merited.   this study sought to examine the  statistical reliability and validity of rubrics  used by multiple raters and to investigate  barriers that might limit the effective use of  rubrics in library decision‐making.      benefits of rubrics  as a tool for ebdm, rubrics offer a number  of benefits.  first, rubrics provide librarians  the opportunity to discuss, determine, and  communicate agreed upon values (callison  36).  librarians who create rubrics must  agree upon the criteria by which a library  product or service will be analyzed and  evaluated.  they must also come to  consensus about what the different  performance levels of each criterion “look  like.”  through the rubric development  process, librarians engage in meaningful  discussions that result in shared views of the  library product or service under scrutiny.   librarians who use existing rubrics as tools  for ebdm can obtain similar benefits by  engaging in activities and discussions to  adapt a rubric to local needs.  once created  or adapted, rubrics not only represent  consensus views of librarians, they also  communicate shared values to others,  including new librarians and external  stakeholders (stevens and levi 23).  finally,  they help “combat . . . accusations that  evaluators do not know what they are  looking for” (bresciani, zelna, and  anderson 30).      rubrics offer a second important benefit for  ebdm: descriptive, yet easily digestible data  (bresciani, zelna, and anderson 30).   because full model rubrics include  descriptions of the key components of a  library product or service at a number of  performance levels, they provide detailed  information for decision‐making.  at the  same time, rubric data is easily simplified.   for example, one might report that a library  service operated at a “good” level during  80% of observed performances.  if more  detail is required, the rubric definition of  “good” can be added to the percentage to  make it more meaningful.  in this way,  rubric data can be reported in simple  percentages or with details described in the  rubric as the situation and audience require.     rubrics offer librarians engaged in ebdm a  third significant benefit.  because rubrics  analyze agreed upon, detailed descriptions  of library activities, they prevent inaccuracy  of scoring (popham 95) and bias (bresciani,  zelna, and anderson 31).  rubrics clarify  schemes for evaluation ahead of time, and  therefore reduce subjectivity (moskal).   since rubrics guide librarians to focus on  essential criteria (callison 35), they can  assess products and services more easily  and objectively (bernier 25).  even in team  or collaborative environments, rubric  evidence based library and information practice 2007, 2:3  30       figure 1.  study rubric assessing student evaluation of web site authority      evaluations are “likely to be reasonably  objective and consistent” (callison 35).        rubrics offer a fourth benefit.  when rubrics  are employed to make evidence based  decisions about library instructional  programs, as in the study described below,  students are significantly impacted.  because  rubrics reveal the expectations of instructors  and librarians, students can focus on  achieving, rather than deciphering, learning  goals.  because full model rubrics offer both  numerical scores and descriptions of  performance levels, ratings are more  meaningful to students than letter or  numerical scores alone (bresciani, zelna,  and anderson 35).  they also provide  students with feedback about what they  have already learned and what they have  yet to learn.  as a result, rubrics support  student learning, self‐evaluation, and  metacognition.  methodology  this study investigated librarians’ use of  rubrics as an ebdm tool to improve an  online information literacy tutorial.  the  data for the study came from student  responses to open‐ended questions  embedded in an online information literacy  tutorial, lobo, used by first‐year students  in english 101 at north carolina state  university (ncsu), raleigh, north carolina.   this study focused on one open‐ended  tutorial question that required students to  analyze and evaluate the authority of a  web‐site they intended to use as support for  an academic paper.    study participants applied a full‐model,  analytic rubric (figure 1) to transform  students’ textual responses into quantitative  data. they then completed an open‐ended  comment sheet designed to capture their  opinions of the process.  to replicate  conditions encountered in real life, the  evidence based library and information practice 2007, 2:3  31 internal (ncsu) participants in this study  took part in a training session.  in contrast,  the external (non‐ncsu) participants were  provided with a substantial amount of  background material, directions, and  examples to familiarize them with the  campus culture and study context—content  that might be found in an article or written  report, but they did not receive face‐to‐face  training.      after both groups applied the rubric to the  student responses, the data was statistically  analyzed for reliability and validity.  data  that was determined to be reliable  (consistent) and valid (accurate) was put to  use for instructional decision‐making.  data  that was not found to be both reliable and  valid was further analyzed to identify  barriers that impede librarians’ successful  use of rubrics.      study participants  there were twenty‐five participants in this  study evenly divided into five groups:     • ncsu librarians   • ncsu english 101 instructors   • ncsu english 101 students   • association of research libraries  (arl) instruction librarians    • arl reference librarians with  limited instruction responsibilities.      selected participants represented a cross  section in gender, race, and areas of  expertise.      the fifteen internal (ncsu) participants  (librarians, instructors, and students) took  part in a rubric training session.  because  most of the participants had little prior  experience with rubrics, the researcher  introduced rubrics by providing a definition,  describing the component parts of rubrics,  and providing the study rubric.  the  researcher followed a multi‐step process to  familiarize the participants with the task of  scoring student responses.  the researcher  began by sharing five “anchor” responses to  demonstrate the range of student responses  with the participants and to model the  scoring process by “thinking aloud.”   working in small groups the participants  scored five more anchor responses  independently and discussed the scores they  assigned.  during these sessions discussion  groups focused on inconsistent scores and  attempted to reconcile them.  following the  discussion session, groups reported their  scores to the full group, and the full group  discussed the remaining inconsistencies and  reconciled them.  this process was repeated  twice.  afterwards, participants scored  seventy‐five student responses.  finally,  they completed an open‐ended comment  sheet and exited the scoring session.    the ten external (non‐ncsu) participants  did not experience a training session.   instead, they were supplied with study  materials, background information, and  directions delivered via the mail.  the  mailing included study background  material, the seventy‐five study responses,  and the open‐ended comment sheet.  the  participants returned their completed study  materials by mail.    statistical analysis  to prepare the materials for statistical  analysis, the researcher assigned each cell of  the study rubric a point value.  the point  values were subjected to quantitative  analysis to describe student performance,  test for interrater reliability, and explore the  validity of participants’ rubric use.  of the  three statistical analyses, the last is most  significant for demonstrating the utility of  rubrics as tools for ebdm.    to establish that rubrics can produce valid  analysis and evaluation of library products  and services, this study used cohen’s kappa  statistic and a “gold standard” approach.      evidence based library and information practice 2007, 2:3  32 kappa statistic  strength of agreement  <0.00  poor  0.00‐0.20  slight  0.21‐0.40  fair  0.41‐0.60  moderate  0.61‐0.80  substantial  0.81‐1.00  almost perfect    figure 2.  kappa statistics and strength of agreement       average kappa  rank  participant group  status  0.72  1  ncsu librarian  expert  0.69  2  instructor  expert  0.67  3  instructor  expert  0.66  4  instructor  expert  0.62  5  ncsu librarian  expert  0.61  6  instructor  non‐expert  0.59  7  instructor  non‐expert  0.58  8  student  non‐expert  0.56  9  student  non‐expert  0.55  10  ncsu librarian  non‐expert  .055  11  student  non‐expert  0.54  12  student  non‐expert  0.52  13  student  non‐expert  0.52  14  ncsu librarian  non‐expert  0.43  15  external instruction librarian  non‐expert  0.32  16  external reference librarian  non‐expert  0.31  17  external instruction librarian  non‐expert  0.31  18  ncsu librarian  non‐expert  0.30  19  external reference librarian  non‐expert  0.30  20  external instruction librarian  non‐expert  0.27  21  external reference librarian  non‐expert  0.21  22  external instruction librarian  non‐expert  0.19  23  external reference librarian  non‐expert  0.14  24  external instruction librarian  non‐expert  0.13  25  external reference librarian  non‐expert    figure 3.  rank order of participants by average kappa      typically used as a measurement of  interrater reliability, cohen’s kappa can also  be used to compare a group of raters to a  “gold standard” to check for validity (gwet  202).  gwet explains that the gold standard  is the “correct classification of subjects made  evidence based library and information practice 2007, 2:3  33 by an experienced observer” (202).  when a  gold standard approach is used, it is  assumed that “the researcher knows the  ‘correct classification’ that may be due to an  expert judgment” (223).  gwet explains,  “the question that the researcher wants to  answer is whether the . . . raters agree with  the standard.  instead of evaluating the  extent of agreement between raters, the  researcher wants to know the truthfulness of  the observers’ ratings.”  this approach is  also known as a “rater‐to‐standard  reliability” or “rater‐to‐expert reliability”  (223).  using cohen’s kappa, this study  compared each participant’s rubric scores to  the gold standard set by the researcher.   then participants were ranked according to  their rater‐to‐standard reliability.  for  ebdm, only scores from the most “expert”  or valid rubric users are fit for use.     a major benefit of this method for  establishing valid rubric data is that cohen’s  kappa statistic can be easily interpreted  using an index (figure 2) established by  landis and koch (landis and koch 165).  in  this study, kappas were calculated for each  participant on the four rubric criteria as well  as the summary score assigned to student  responses (figure 3).    expert rubric users  statistical analysis revealed that a subset of  participants included in this study were able  to achieve substantially valid results.  the  top five participants—two internal (ncsu)  librarians and three instructors—formed an  expert group of raters.  the expert  participants’ rubric data contributed to  improved online instruction and allowed for  comparative statistical analysis with the  other participant five‐member sub‐groups.   the emergence of an expert participant  group is a promising sign for the use of  rubrics to support ebdm.    on the other hand, three internal librarians  and all of the external librarians included in  the study were unable to achieve an expert  level of validity (figure 3).  an analysis of  the interrater reliability among the expert  participants (figure 4) and non‐expert  participants (figure 5) demonstrates the  distinction between the two groups.  expert  participants demonstrated moderate and  substantial agreement across all criteria of  the study rubric as well as the overall  “grade” assigned to the student  performance.  non‐expert participants  showed only fair or slight agreement across  nearly all areas of the rubric.  the  application of two‐sided t tests (alpha level  of .05) to this data shows that these  differences in reliability levels are  statistically significant; t values of  individual rubric criteria range from 4.2 to  16.3.     this statistical analysis revealed two  additional items worth noting.  first, and  perhaps not surprisingly, the external  librarians provided the least valid rubric  results of all participants.  this may be  attributable to their lack of training or lack  of familiarity with ncsu library contexts.   the second remarkable result is the wide  variation of the internal librarians’ validity  rankings.  although the highest validity  rank was achieved by a ncsu librarian, a  second ncsu librarian earned scores that  were no more accurate than external  librarians.  this extreme variation across  ncsu librarians demonstrates that training  and familiarity with library culture are not  enough to ensure valid rubric usage.  this  finding also highlights the importance of  conducting basic statistical tests to confirm  the validity of data used to make decisions  regarding library products and services.      not all rubric users are experts  one of the most important findings of this  study is that not all rubric users are experts.   while this might seem obvious, in practice  many rubrics are used by people who have      evidence based library and information practice 2007, 2:3  34 expert rat ers 0.77 0.74 0.48 0.6 0.52 -0.2 0 0.2 0.4 0.6 0.8 art iculat es crit eria cit es indicat ors p rovides examples judges use grade p oor slight fair moderat e subst ant ial almost p erfect figure 4.  kappa statistics for expert rubric users, standard error =.03      non-expert raters 0.29 0.24 0.17 0.47 0.27 -0.2 0 0.2 0.4 0.6 0.8 articulates criteria cites indicators provides examples judges use grade poor slight fair moderate substantial almost perfect   figure 5.  kappa statistics for non‐expert rubric users, standard error =.006    not been verified as expert in the use of such  tools.  often, expert ability is assumed based  on an individual’s educational background,  experience, or position in an educational  institution.  however, in this     study the group of expert participants  crossed divisions of background,  experience, and institutional position.  thus,  the belief that rubrics can be used reliably  and validly by those who have a particular  evidence based library and information practice 2007, 2:3  35 degree, a specific type of experience, or a  certain position within the institution is a  faulty assumption.  if rubric users are  selected because of their education,  experience, or position, rather than  demonstrated ability to provide reliable and  valid scores, library processes may be  evaluated inconsistently, inaccurately, or  unfairly.  this study supports the idea that  only the rubric results of demonstrated  experts should be used to make decisions  that impact library products and services.      characteristics of expert rubric users  while this study offers a statistical process  for identifying expert rubric users, it is also  wise to consider the factors that make an  “expert” an expert.  why do some rubric  users achieve expert status while others do  not?  some possible answers to this question  emerge both from the literature and from  the comments of the participants in this  study.      some rubric users might be naturally  proficient scorers.  wolfe, kao, and ranney  note that the most proficient scorers tend to  focus on the general features of a product or  performance and “adopt values espoused  by the scoring rubric” more so than less  proficient scorers.  throughout the scoring  process, proficient rubric users revisit the  established criteria in order to ensure that  consistency is maintained (moskal).   however, less proficient rubric users tend to  interrupt their observation of a product or  performance to monitor how well it satisfies  the rubric, rather than focusing on the  product or performance and then reviewing  it against the rubric (wolfe, kao, and  ranney).  the literature also reports that  non‐expert rubric users may have diverse  outlooks, perspectives, and experiences that  need to be taken into account (colton et al.  9).  tired rubric users may not score  students evenly over a time period.  the  mood of the rater can affect scores, as can  prior knowledge (moskal).  finally, training  can impact the proficiency of rubric users,  and this is born out by the significant  differences between internal and external  participants in this study.      in addition, there may be other barriers that  keep rubric users from becoming experts.  in  this study, participants completed an open‐ ended comment sheet after scoring student  responses.  non‐expert participants alluded  to some of these roadblocks in their  comments.  their comments can be  categorized into six barriers that may  explain why some participants could not  attain expert status.      barrier 1:  difficulty understanding an  outcomes‐based approach   libraries have a strong tradition of using  input and output measures to evaluate  products and services, and sometimes they  are slow to adopt outcomes‐based  approaches.  as a result, librarians may  struggle with outcomes‐based tools such as  rubrics.  the rubric used in this study was  based on outcomes included in the  association of college and research  libraries’ information literacy competency  standards for higher education, a document  well known among academic reference and  instruction librarians.  despite the  popularity of these standards and the recent  push for outcomes‐based assessment of  information literacy skills in academic  libraries, some participants of the study  appeared to be unfamiliar with key concepts  of outcomes‐based methods.  a few  participants voiced concerns that an  outcomes‐based approach to the assessment  of information literacy instruction may fail  to measure what they termed student  “understanding” or “ability.”  these  participants felt that using measurable  outcomes to assess student learning focused  too much on specific skills—too much  “science” and not enough “art.”  one  participant wrote, “while the rubric  measures the presence of concepts . . . , it  evidence based library and information practice 2007, 2:3  36 doesn’t check to see if students understand  [the] issues.”  another participant stated,  “this rubric tests skills, not . . . real  learning.”  these comments indicate that  librarians need to learn more about the  values and principles of outcomes‐based  assessment before they can make  meaningful progress in the assessment of  information literacy skills.      the need for improvement in this area has  also been noted in the literature.  despite  reports that librarians are increasingly asked  to join other educators in justifying their  programs using learning outcomes  (lichtenstein 28), iannuzzi reports, “we  have yet to see widespread implementation  of outcomes assessment methodologies in  terms of student learning in our academic  libraries” (iannuzzi 304).  despite the slow  diffusion of such measures, outcomes‐based  approaches offer real benefits to librarians  striving to use ebdm practices to advance  library programs.  smith states, “it is  important for libraries to understand the  processes that are used to define learning  outcomes, to select measures, to collaborate  with other academic departments, and to  use the results to improve their programs.”      barrier 2:  tension between analytic and  holistic approaches   a second barrier that may have prevented  some librarians in this study from attaining  expert status is a lack of comfort with  analytical evaluation.  one participant  worried that parsing user behavior into  components may result in  oversimplification, commenting that using  the rubric “was really simple.  but i worried  that i was being too simplistic . . . and not  rating [student work] holistically.”  another  participant seemed to prefer assigning an  “overall” score, stating “the rubric is a good  and a solid way to measure knowledge of a  process, but it does not allow for raters to  assess the response as a whole.”      these comments reveal tension between  analytic and holistic approaches to  evaluation.  both approaches have  advantages and disadvantages.  arter and  mctighe recommend holistic rubrics for  simple products or performances,  particularly ones with only one important  criterion to assess.  they also note that  holistic rubrics are useful for “getting a  quick snapshot of overall quality or  achievement” (arter and mctighe 21).   unfortunately, holistic rubrics provide only  limited feedback (mertler) and provide “no  detailed analysis of the strengths and  weaknesses of a product or performance.  so,  they’re not as useful diagnostically to help  plan instruction.  nor do they provide  students with detailed feedback to guide  their improvement” (arter and mctighe 21).      because analytic rubrics “divide . . . a  product or performance into essential traits  or dimensions so that they can be judged  separately” (arter and mctighe 18), they  allow for separate evaluations of each factor  along a different descriptive scale (moskal).   this part‐to‐whole approach makes analytic  rubrics better suited for “judging complex  performances (e.g., research process)  involving several significant dimensions”  (arter and mctighe 22).  according to  mertler, analytic rubrics are preferred when  a focused response to stakeholders is  required.  however, these advantages come  at a cost.  analytical rubrics take more time  to create and to use.  arter and mctighe  write, “after all, you have more to discern”  (23).      the decision to use a holistic or analytic  rubric is one that should be based on the  product or performance to be assessed, the  criteria to be observed, and the purpose of  the assessment.  mertler states that the most  important factor to consider is how the  results of the assessment will be used.  he  writes, “if an overall, summative score is  desired, a holistic scoring process would be  evidence based library and information practice 2007, 2:3  37 more desirable.  in contrast, if formative  feedback is the goal, an analytic scoring  rubric should be used.”  in the case of the  study rubric, formative feedback was  required, not only for students, but also for  librarians.  without formative feedback,  librarians could not make decisions for  improvements to the tutorial.  in a different  ebdm scenario, a holistic rubric might be  more appropriate.    barrier 3:  failure to comprehend the  rubric  a third issue that may prevent rubric users  from becoming “experts” is simple:  a failure  to understand some aspect of the rubric.   nearly all the participants used in this study  claimed that they understood the terms  used in the rubric.  still, one important  comprehension problem surfaced during  this study.  a few external librarian  participants failed to realize that rubrics are  intended to judge performances on quality,  not quantity, or how many times a  performance is observed (callison 36).  one  participant wrote, “the student might cite  one example, . . . but not . . . enough for me  to consider it exemplary.”  this focus on  quantity rather than quality reveals a lack of  experience with rubric design that might be  easily addressed through training.    barrier 4:  disagreement with assumptions  of the rubric  the content of any rubric is based on certain  assumptions.  in this study, the major  assumption is that the capacity to use the  criterion of authority for web site evaluation  is comprised of the ability to use criterion  terminology, cite examples of indicators of  the criterion, identify those indicators in an  example web site, and make a reasoned  decision about the use of the example web  site.  this assumption is based on two  documents from the association of college  and research libraries: the information  literacy competency standards for higher  education and the objectives for information  literacy instruction: a model statement for  academic librarians.  perhaps the biggest  hurdle for some study participants was their  disagreement with the standards on which  the rubric was based; as a result, they could  not fully internalize the assumptions of the  rubric.  the standard that posed the most  difficulty for participants was standard 3.2,  which reads, “the information literate  student articulates and applies initial criteria  for evaluating both the information and its  sources.”  several participants argued that  students should not have to articulate web  evaluation criteria—in this case, authority.   these participants felt that assessing  students’ ability to use criteria terminology  resulted in “looking for specific terms and  missing the overall point of the responses.”   one participant commented that by  expecting students to articulate criteria  terminology, the rubric “valued students’  ability to use particular words but does not  measure their understanding of concepts.”   this comment indicates that disagreement  with one part of the rubric impacted the  participant’s ability to adopt the rubric in its  entirety.      it is important to note that conflicts with the  assumptions of a rubric might be avoided if  participants are included in the rubric  development process.  for this study, such  an approach was not feasible, but research  confirms the value of allowing stakeholders  to discuss and determine agreed upon  values of student learning.      barrier 5:  difficulties with artifacts   artifacts of library processes present a fifth  potential barrier to participants’ expert  status.  in this study, several participants  commented on the difficulty of interpreting  artifacts of student learning because some  student responses were cryptic, incomplete,  vague, or incorrect.  one external  participant said that she found herself  “giving the more cryptic answers the benefit  of the doubt.”  another complained that  evidence based library and information practice 2007, 2:3  38 student responses were sometimes  incomplete.  she questioned, “if a student  answer consists of a bulleted list of  responses to the prompt, but no discussion  or elaboration, does that fulfill the  requirement?”  another lamented, “it’s  really hard . . . when students are asked to  describe, explain, draw conclusions, etc. and  some answer with one word.”  a fourth  asked, “should the rubric be used on ‘text’  that isn’t in complete sentence form?  how  much should one use the lobo prompt to  interpret student answers?”  some  participants were stymied by incorrect or  partially incorrect answers.  one wrote, “i  suspect my own perceptions of the  ‘correctness’ of the answers affected me.”   another stated, “it killed me that i couldn’t  take points off for incorrect information.”   these comments indicate that difficulties  with artifacts can affect participants’ ability  to produce reliable (consistent) and valid  (accurate) results.    barrier 6:  difficulties understanding  library context and culture  in this study, external librarian participants  achieved the lowest levels of validity, a  result that could be due to lack of training or  a lack of familiarity with ncsu library  conventions and culture. it appears that the  typical model of exporting a tool that works  at one library for use in another library via  professional listservs or journals may not  produce valid data for ebdm.  this is an  important finding with implications for  librarians’ professional practice.  if tools are  to be shared among libraries, perhaps local  training and adaptation are necessary, and  the results should be statistically analyzed to  confirm data quality.    the need for training  in the hands of experts, the rubrics  employed in this study yielded detailed and  comprehensible data that facilitated  instructional decision‐making. this study  also reveals that non‐expert rubric users  may be unable to produce valid evaluations  of library products and services without  additional local training.      to overcome barriers to successful rubric  usage, training should teach librarians the  value and principles of outcomes‐based  approaches to library analysis and  evaluation.  training should incorporate the  theories that underlie rubrics as well as the  advantages and disadvantages of analytic  and holistic rubric models.  in addition,  training should cover the structural issues  that can limit the reliability, validity, and  overall usefulness of rubrics: some rubrics  are not well written (popham 95); some use  wording that is too general or too specific  (tierney and simon); some are too long  (popham 98); some include inconsistencies  (tierney and simon); and some emphasize  quantity rather than quality (callison 36).   furthermore, training should address  methods for eliminating disagreement with  assumptions of a rubric.  finally, because  rubrics may be used to analyze and evaluate  library products and services that generate  “messy” data, training should review the  difficulties librarians are likely to encounter  in the data and methods for handling them.    conclusion  this study demonstrated that not all  librarians are proficient or “expert” in the  use of rubrics.  it also revealed that six  barriers may impact librarians’ ability to  apply rubrics for ebdm: 1) difficulty  understanding outcomes‐based assessment;  2) tension between analytic and holistic  rubric approaches; 3) failure to comprehend  rubric content; 4) disagreement with rubric  assumptions; 5) difficulties with artifacts of  library processes; and 6) difficulty  understanding non‐local library context and  culture.  all these barriers can be addressed  by training.  however, if training is required  to help librarians use rubrics to produce  reliable and valid results, skeptics might ask  if the benefits are worth the time and energy  evidence based library and information practice 2007, 2:3  39 required.  certainly, there are costs  associated with training, but the advantages  of rubrics outweigh the costs (prus and  johnson 25).  the participants of this study  confirmed the value of rubrics—nearly all  participants stated that they could envision  using rubrics to improve library  instructional services.  indeed, results from  the study were used to enhance tutorial  content and questions.  both participant  feedback and tutorial improvements attest  to the merit of rubrics as tools for effective  ebdm practice.  future research efforts  should focus on the attributes of expert  raters and the effects of different types and  levels of rater training; these research foci  will illuminate best practices in the use of  rubrics for ebdm.  in short, librarians eager  to use ebdm strategies should invest in  rubric training and pursue research in this  area.  to do otherwise is to discard a viable  approach to ebdm that offers librarians a  wealth of benefits.     acknowledgement    a version of this paper was presented at the  4th international evidence based  librarianship conference, held in chapel  hill, north carolina, usa (may 2007) and  awarded best paper.  see  http://www.eblip4.unc.edu/papers/oakleaf. pdf      works cited    arter, judith, and jay mctighe. scoring  rubrics in the classroom:  using  performance criteria for assessing and  improving student performance.  thousand oaks, california: corwin  press, 2000.    association of college and research  libraries. information literacy  competency standards for higher  education. 2000. 22 april 2005  <http://www.ala.org/ala/acrl/acrlstanda rds/informationliteracycompetency.ht m>.    association of college and research  libraries.  objectives for information  literacy instruction: a model  statement for academic librarians.  2001. 25 july 2007  <http://www.ala.org/ala/acrl/acrlstanda rds/objectivesinformation.htm>.    bernier, rosemarie. “making yourself  indispensable by helping teachers  create rubrics.” csla journal 27.2  (2004): 24‐25.    bresciani, marilee j., carrie l. zelna, and  james a. anderson.  assessing student  learning and development:  a  handbook for practitioners.  washington: national association of  student personnel administrators,  2004.      buchanan, lori e. ʺassessing liberal arts  classes.ʺ  assessing student learning  outcomes for information literacy  instruction in academic institutions.  ed. elizabeth fuseler avery. chicago:  association of college and research  libraries, 2003. 68‐73.     callison, daniel. “rubrics.” school library  media activities monthly 17.2 (oct  2000): 34.    choinski, elizabeth, amy e. mark, and  missy murphey. ʺassessment with  rubrics:  an efficient and objective  means of assessing student outcomes  in an information resources class.ʺ  portal:  libraries and the academy 3.4  (2003): 563‐75.    colton, dean a., xiaohong gao, deborah j.  harris, michael j. kolen, dara  http://www.eblip4.unc.edu/papers/oakleaf http://www.ala.org/ala/acrl/acrlstandards/informationliteracycompetency.htm http://www.ala.org/ala/acrl/acrlstandards/informationliteracycompetency.htm http://www.ala.org/ala/acrl/acrlstandards/informationliteracycompetency.htm http://www.ala.org/ala/acrl/acrlstandards/objectivesinformation.htm http://www.ala.org/ala/acrl/acrlstandards/objectivesinformation.htm evidence based library and information practice 2007, 2:3  40 martinovich‐barhite, tianyou wang,  and catherine j. welch. “reliability  issues with performance assessments:  a collection of papers.” act research  report series 97.3 (1997). 28 aug. 2007  <http://www.act.org/research/reports/p df/act_rr97‐03.pdf>.    dʹangelo, barbara j. ʺintegrating and  assessing information  competencies in a gateway  course.ʺ reference services review  29.4 (2001): 282‐93.    emmons, mark, and wanda martin.  ʺengaging conversation: evaluating  the contribution of library  instruction to the quality of student  research.ʺ college & research  libraries 63.6 (2002): 545‐60.    franks, dana. ʺusing rubrics to assess  information literacy attainment in  a community college education  class.ʺ  assessing student learning  outcomes for information literacy  instruction in academic institutions.  ed. elizabeth fuseler avery.  chicago: association of college and  research libraries, 2003. 132‐47.     gauss, nancy, and kathleen kinkema.  ʺwebliography assignment for a  lifetime wellness class.ʺ  assessing  student learning outcomes for  information literacy instruction in  academic institutions. ed. elizabeth  fuseler avery. chicago: association  of college and research libraries,  2003. 161‐71.     gwet, kilem. handbook of inter‐rater  reliability: how to estimate the  level of agreement between two or  multiple raters. gaithersburg, md:  stataxis, 2001.    hafner, john c. “quantitative analysis of  the rubric as an assessment tool:   an empirical study of student peer‐ group rating.” international  journal of science education 25.12  (2003).    hutchins, elizabeth o. ʺassessing  student learning outcomes in  political science classes.ʺ  assessing  student learning outcomes for  information literacy instruction in  academic institutions. ed. elizabeth  fuseler avery. chicago: association  of college and research libraries,  2003. 172‐84.     iannuzzi, patricia. “we are teaching, but  are they learning:  accountability,  productivity, and assessment.”  journal of academic librarianship  25.4 (1999): 263‐266.    kivel, andy. ʺinstitutionalizing a  graduation requirement.ʺ   assessing student learning  outcomes for information literacy  instruction in academic institutions.  ed. elizabeth fuseler avery.  chicago: association of college and  research libraries, 2003. 192‐200.     knight, lorrie a. ʺusing rubrics to  assess information literacy.ʺ  reference services review 34.1  (2006): 43‐55. 25 july 2007  <http://www.emeraldinsight.com/in sight/viewcontentitem.do?contentt ype=article&contentid=1541934>.    kobritz, barbara. ʺinformation literacy in  community college  communications courses.ʺ   assessing student learning  outcomes for information literacy  instruction in academic institutions.  ed. elizabeth fuseler avery.  http://www.act.org/research/reports/pdf/act_rr97%e2%80%9003.pdf%00%00 http://www.act.org/research/reports/pdf/act_rr97%e2%80%9003.pdf%00%00 http://www.emeraldinsight.com/insight/viewcontentitem.do?contenttype=article&contentid=1541934 http://www.emeraldinsight.com/insight/viewcontentitem.do?contenttype=article&contentid=1541934 http://www.emeraldinsight.com/insight/viewcontentitem.do?contenttype=article&contentid=1541934 evidence based library and information practice 2007, 2:3  41 chicago: association of college and  research, 2003. 207‐15.     landis, j. richard and gary g. koch.  “the measure of observer  agreement for categorical data.”  biometrics 33 (1977): 159‐174.    lichtenstein, art a.  “informed  instruction:  learning theory and  information literacy.” journal of  educational media and library  sciences 38.1 (2000): 22‐31.    mertler, craig a. “designing scoring  rubrics for your classroom.”  practical assessment, research and  evaluation 7.25 (2001). 27 july 2007  <http://pareonline.net/getvn.asp?v= 7&n=25>.    merz, lawrie h., and beth l. mark. clip  note #32: assessment in college  library instruction programs.  chicago: association of college and  research libraries, 2002.    moskal, barbara m. “scoring rubrics:   what, when, and how?” practical  assessment, research, and  evaluation 7.3 (2000). 27 july 2007  <http://pareonline.net/getvn.asp?v= 7&n=3>.    nitko, anthony j. educational  assessment of students. englewood  cliffs, nj: prentice hall, 1996.    oakleaf, megan.  ʺassessment of student  information literacy skills:  an  investigation of a rubric  approach.ʺ  undergraduate  assessment symposium, north  carolina state university, raleigh,  nc, usa, 7 apr. 2006.    oakleaf, megan.  “assessing information  literacy skills: a rubric approach.”   diss. u. of north carolina – chapel  hill. 2006.    popham, w. james. test better, teach  better: the instructional role of  assessment. alexandria, va:  association for supervision and  curriculum development, 2003.    prus, joseph and reid johnson. “a  critical review of student  assessment options.” new  directions for community colleges  88 (1994).    rockman, ilene f. rubrics for assessing  information competence in the  california state university.  2002. 10  apr. 2005.  <http://www.calstate.edu/ls/1_rubr ic.doc>.    smalley, topsy n. bay area community  colleges information competency  assessment project.  2003. 17 nov.  2003.  <http://www.topsy.org/icap/icapr oject.html>.    smith, kenneth r. new roles and  responsibilities for the university  library:  advancing student  learning through outcomes  assessment. association of  research libraries, 2000. 24 july  2007  <http://www.arl.org/newsltr/213/ass ess.html>.    stevens, dannielle d. and antonia levi.  introduction to rubrics: an  assessment tool to save grading  time, convey effective feedback,  and promote student learning.  sterling, virginia: stylus, 2005.    tierney, robin and marielle simon.  “whatʹs still wrong with rubrics:   http://pareonline.net/getvn.asp?v=7&n=25 http://pareonline.net/getvn.asp?v=7&n=25 http://pareonline.net/getvn.asp?v=7&n=3 http://pareonline.net/getvn.asp?v=7&n=3 http://www.calstate.edu/ls/1_rubric.doc http://www.calstate.edu/ls/1_rubric.doc http://www.topsy.org/icap/icaproject.html http://www.topsy.org/icap/icaproject.html http://www.arl.org/newsltr/213/assess.html http://www.arl.org/newsltr/213/assess.html evidence based library and information practice 2007, 2:3  42 focusing on the consistency of  performance criteria across scale  levels.” practical assessment,  research, and evaluation 9.2 (2004). 27  july 2007  <http://pareonline.net/getvn.asp?v= 9&n=2>/    warmkessel, marjorie m. ʺassessing  abilities of freshmen to reconcile  new knowledge with prior  knowledge.ʺ  assessing student  learning outcomes for information  literacy instruction in academic  institutions. ed. elizabeth fuseler  avery. chicago: association of  college and research libraries,  2003. 249‐56.     wiggins, grant. “creating tests worth  taking.” a handbook for student  performance in an era of  restructuring. eds. r. e. blum and  judith arter. alexandria, va:  association for supervision and  curriculum development, 1996.    wolfe, edward w., chi‐wen kao, and  michael ranney. “cognitive  differences in proficient and  nonproficient essay scorers.”  written communication 15.4 (1998).   25 july 2007.   <http://wcx.sagepub.com/cgi/conten t/abstract/15/4/465>.        http://pareonline.net/getvn.asp?v=9&n=2 http://pareonline.net/getvn.asp?v=9&n=2 http://wcx.sagepub.com/cgi/content/abstract/15/4/465 http://wcx.sagepub.com/cgi/content/abstract/15/4/465 microsoft word art_boock.doc evidence based library and information practice 2007, 2:3  76 evidence based library and information practice     article    the use of value engineering in the evaluation and selection of digitization projects      michael h boock  assistant professor  head of technical services  oregon state university libraries  121 the valley library  corvallis, or 97331, usa  email: michael.boock@oregonstate.edu    may chau  associate professor  agricultural librarian  oregon state university libraries  121 the valley library  corvallis, or 97331, usa  email: may.chau@oregonstate.edu      received:  6 april 2007        accepted: 22 june 2007      © 2007 boock and chau. this is an open access article distributed under the terms of the creative  commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    abstract    objective ‐ the authors describe a simple and effective tool for selecting digitization  projects from competing alternatives, providing decision makers with objective,  quantitative data.    methods ‐ the paper adopts the value engineering methodology for the selection,  evaluation and ranking of digitization project proposals. project selection steps are  described. selection criteria are developed. digitization costs are presented as an equation.  project value is determined by calculating projected performance of digital collections based  on the established criteria over cost.     results ‐ scenarios are presented that evaluate and rank projects based on an evaluation of  performance criteria and cost. the communication and use of rating criteria provides  mailto:boock@oregonstate.edu mailto:chau@oregonstate.edu http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:3  77 selectors with information about how proposed collections are evaluated. the transparency  of the process output is easily communicated to stakeholders.    conclusion ‐ value engineering methodology provides a tool and a process that gives  decision makers a set of objective, quantitative data upon which selection of digitization  projects is based. this approach simplifies the selection process, and creates transparency so  that all stakeholders are able to see how a decision was made.     introduction and background    value engineering (ve), sometimes referred  to as value methodology, provides business  and other organizations with a structured  methodology for evaluating product or  strategic options and a formula that can be  used to measure, rank and compare options.  the methodology is taught in project  management and industrial engineering  programs and used within total quality  management (tqm) and business process  reengineering (value engineering). the  concept calls for several steps of project  planning such as information gathering,  determination of performance measurement,  analysis of other options (brainstorming),  ranking of ideas, and a comparison of  options. the key steps of quantitative  ranking and comparison of options is  accomplished through the development of  performance criteria measures, ranking  performance according to those established  measures and determining project costs.  value is then defined according to the ratio  of function or performance over cost  (pearsall & eng). this is mathematically  displayed as:    value = performance / cost    performance has a different meaning in  different contexts. in this article,  performance refers to the relative  significance of digitized information to  internal and external users, the uniqueness  of the digitized information and the relative  exposure that results. costs include staffing,  copyright permissions and equipment  expenditures. value increases as a result of  performance improvements or cost  decreases ‐ often accomplished through  elimination of wasteful or unnecessary  practices.     measuring the ratio of performance to cost  results in a quantitative representation of  the estimated value of potential digitization  projects. quantitative representation is  useful in a setting in which resources are  limited and there are competing collections  proposed for digitization. using the  quantitative measures that result from the  calculation of the equation, each project’s  value is ranked and compared against  others. the highest‐ranking project has  higher priority over those ranked lower. the  communication of the results to interested  parties ensures that communication barriers  are decreased and the selection process is  more transparent. hays finds that using  quantitative measures can increase  communication (hays 125).     the effectiveness of applying the ve  methodology in academic libraries is  sparsely documented. pershing and  krutulis adopt the mudge matrix ve model  (mudge 174‐183) to determine salary equity  in academic libraries. they find that the  methodology “worked extremely well for  indiana university even though it was not  designed for that particular use or  environment.” bick and sinha apply ve in  the evaluation of journals and find that it  provides an objective measure for  evidence based library and information practice 2007, 2:3  78 identifying journal value in support of  subscription and cancellation decisions.     the methodology is also applied in  curriculum development and review and  provides “a formal way of closing the  feedback loops at all programmatic levels  from the course level to the objective level”  (gershenson, hailey, batty and phillips 140).  other uses of ve are recorded in the health‐ care field. for example, jakhanwal & singh  describe the use of ve to justify the cost of  cataract surgery.    this article presents a stepped approach that  libraries can utilize when selecting projects  for digitization. it discusses the creation of  performance and cost criteria that can be  used to determine the overall value of  competing digitization projects. the model  provides transparency to the selection  process and more fluid communication  within libraries regarding the selection of  digitization projects. example scenarios  illustrate the entire process from the  calculation of performance for sample  projects to the ranking of the sample  projects.     steps in the selection of digitization  projects    selection steps are helpful for administrators  to consistently apply standards and select  appropriate groups of materials for cost‐ effective digitization. in ve, project  participants and decision‐makers engage in  a process that is broken into a structured  sequence of steps (value engineering).  project selection steps, adapted from the  formal ve process, include:    a.  examination and initial review of  information provided in digitization project  proposals. in order for the project proposal  to be effective, it must be accurate and  informative particularly in regard to the  criteria upon which it will be judged. project  proposals must be a collective effort and  written in consultation with all stakeholders,  including subject experts. proposals that  lack information may be sent back with  suggested improvements.     b.  determination of digitization costs. the  project proposal must include a realistic  estimation of production costs. cost  determination must be coordinated with  appropriate departments within the  institution to ensure accuracy.     c.  measurement of expected performance.  the project proposal must demonstrate its  capability to reflect and support the  academic standards of the institution  according to teaching and research trends.  the performance should be in line with the  library’s collection development policies,  library, departmental and institutional  strategic plans and mission statements.  performance and cost are the two factors  that determine the value of a digitization  project.    d.  presentation of scenarios. project  proposals should be compared with each  other using written scenarios that illustrate  the potential value of the projects.     e. final measurement and ranking of  projects. discussion of proposed projects  may precede measurement and ranking by  each member of the administrative team.  the persons who proposed the projects are  informed of the decision and given copies of  the measures and rankings.    determination of digitization costs    in order to justify the expenditure of fiscal  and human resources on digitization  activities, many libraries engage in cost  analyses to determine how much it costs to  digitize materials. libraries also determine  costs to enable more accurate customer  charging for digitization work. the  evidence based library and information practice 2007, 2:3  79 michigan university library documented  the full range of digitization costs in “the  making of america iv: the american voice,  1850‐1877” project. they aggregated costs  into broad categories according to the steps  undertaken in the digitization process  (university of michigan digital library  services 8).    the british library identified costs  associated with item digitization in terms of  a “life‐cycle model” that also identified the  outlay of all costs within the digitization  workflow” (shenton 11). to determine the  total cost to complete the digitization of an  item or a collection of items, they proposed  the use of an equation that adds the cost of  specific tasks, from selection through  preservation.     this article further develops the cost  variables provided by the british library by  considering additional tasks required within  the digitization workflow including the cost  of server space and obtaining copyright  permissions. costs unique to  digitization  projects such as equipment that is not likely  to be used for anything other than the  particular project have also been added.     a sample of the project may be digitized to  determine the amount of time required to  complete the project. this also helps to  ensure that all aspects of the digitization,  including all of the preparatory work, is  accounted for and that the cost analysis is  accurate. based on the sampling and time  estimates, total salaries and benefits for each  person with any responsibility for the  completion of the work can be considered  and accounted for. other infrastructure  costs such as staff training, software or  equipment are not considered as project  costs because of their anticipated utility for  other projects. adding all of the variables  together enables an institution to compute  the total cost of digitization and making a  collection available.   cost of digitizing a collection of materials  and making the collection available is  expressed as follows:    c = sel + ipr + cop + cons + ret + cap + qc +  met + pres + web + mgt + stor + x    c  total cost of digitizing a collection of  materials and making it available  online  sel  selection  ipr  checking of intellectual property  rights  cop  obtaining copyright permissions  cons  conservation check and remedial  conservation  ret  retrieval, reshelving, debinding,  rebinding, material preparation  cap  capture of digitized items  qc  quality assurance of digitized items  met  metadata creation  pres  preservation of digitized items  web  website creation and graphical  design  mgt  oversight, supervision, management  stor  storage of digitized items  x  additional costs unique to the project  not otherwise represented    measurement of expected performance    performance criteria must measure the  degree to which the digitized collection  meets an institution’s strategic goals,  reflecting the academic standards and  subject interests of the institution. the  criteria must also reflect the expected impact,  accessibility and usage of the materials to be  digitized and measure the degree to which  the collection is used in the teaching and  research of the institution. the national  information standards organization  recommends guidelines for building quality  digital collections (national information  standards organization, a framework),  many of which are included here. for  evidence based library and information practice 2007, 2:3  80 example, niso recommends that a digital  collection needs to follow an existing  documented collection development policy.    the criteria listed below serve as an example  of performance criteria. individual  institutions may revise these to their own  specifications or devise their own to align  with their collection development policies.  the criteria must represent what is  important to the respective institution. the  performance criteria are offered here as an  example: significance of content to internal  stakeholders (is); significance of content to  external stakeholders (es); uniqueness of  content (uniq); exposure (exp).     total expected project performance is  expressed as follows:     p = ((2*is) + es + uniq + exp)*100  significance of content to internal  stakeholders is multiplied by 2 to reflect the  relative importance of this criterion at most  institutions. the total of each criterion is  multiplied by 100 in order to normalize the  data. other institutions should consider the  importance of each criterion to their  institution and weight them accordingly.     significance of content to internal  stakeholders    clearly, if a collection lacks value to the  university’s stakeholders, digitization  should not be pursued. significance of  content to internal stakeholders (is) is the  degree to which a collection, once digitized,  supports the immediate and long term  research and teaching needs of the  institution, the expected usage of the  collection by university students, faculty  and staff, and the degree to which the  digitized collection supports the university’s  strategic plan. (table 1)  significance of content to external  stakeholders    a highly successful digital collection is of  interest to researchers and users outside of  the university, as a greater good in support  of life‐long learning, information discovery  and research. this criterion measures the  expected usage of the collection outside of  the university and the degree to which the  collection supports the university’s  particular subject niche within the global,  digital learning environment.      subject specialist librarians are pivotal to the  preparation of digitization proposals not  only because of their familiarity with the  research that is conducted in their colleges  but also because of their broader knowledge  of the content of their subject areas, with  their own library’s collection and other  collections. the subject specialist librarian  should be able to determine the significance  of the collection to internal and external  users. in order to identify what content  would be most useful to have digitized, the  subject specialist librarian may also choose  to conduct focus groups and surveys prior  to the writing of the proposal.    the degree to which a digital collection  contributes to the codification of a  discipline’s knowledge is measured within  this criterion. (table 2)  uniqueness    in 2004, google launched its ambitious  project to digitize 30 million books,  partnering with prestigious academic  institutions (quint). the us government’s  printing office is in the process of digitizing  all federal government documents (gpoʹs  digitization and preservation initiatives).  these large scale digitization efforts, along     evidence based library and information practice 2007, 2:3  81 criterion  definition  rating  scale   unit of measure/quantification  significance  of content  (internal)  a measure of the  significance of knowledge  encoded in the resources  to the institution.  5  significant information related to the  institution’s areas of focus. meets the  current and long‐term research and  teaching needs of the institution. has  great current and historical importance.  supports the university’s strategic plan.      4  significant information related to general  teaching and research trends of the  institution. has current and historical  importance.      3  significant information related to general  teaching and research trends of the  institution. lacks either current or  historical significance.      2  not directly related to the teaching and  research of the institution. lacks either  current or historical significance to the  institution.      1  not related to the teaching and research  of the institution. lacks both current and  historical significance to the institution.  table 1. significance of content to internal stakeholders  criterion  definition  rating  scale   unit of measure/quantification  significance  of content  (external)  a measure of the  significance of knowledge  encoded in the document  and its related subject  disciplines known to be of  interest to external  researchers.  5  significant information of great current  and historical interest to researchers  outside of the institution.       4  significant information of current and  historical interest to researchers outside of  the institution.      3  significant information lacking either  current or historical interest to researchers  outside of the institution.      2  minor significance to researchers outside  of the institution.       1  lacks significance to researchers outside  of the institution.    table 2. significance of content to external stakeholders  evidence based library and information practice 2007, 2:3  82 criterion  definition  rating  scale   unit of measure/quantification  uniqueness  of content  a measure of the  uniqueness of the material  included in the collection to  be digitized and the  likelihood of it being  digitized elsewhere.  5  content is held uniquely by institution  and extremely unlikely to be digitized by  another institution or government agency.      4  content is rare and unlikely to be  digitized by another institution or  government agency.      3  content is held by less than a dozen  libraries and is unlikely to be digitized by  another holding library, institution or  government agency.      2  content is held by hundreds of libraries  and is likely to be digitized by another  institution eventually.      1  content is held by hundreds or more  libraries and is already in the process of  being digitized.    table 3. uniqueness      with those of microsoft live, yahoo!,  internet archive, the million books project  and others, all suggest that a significant  percentage of the world’s literature written  in english will be publicly available  electronically in some form within ten years.    despite these tremendous collective  undertakings, many unique institutional  resources such as original photographs,  archival materials, gray literature such as  university technical reports and conference  proceedings, and other unique or rare  institutional resources remain to be  digitized. this criterion measures the  relative uniqueness of a collection and the  likelihood of it being digitized elsewhere.  (table 3)    exposure    the exposure criterion measures the degree   to which the digital collection garners the   institution positive recognition and press.  the criterion also assesses the potential for  the digital availability of the collection to  result in grants and other funding.  (table 4)  example scenarios     the following four digital collection project  scenarios and rankings illustrate how the ve  process helps measure, rank and prioritize  digitization projects. all costs presented are  in us dollars.   in order for proposals to be  ranked consistently, the person who  proposes a project for digitization should be  familiar with the criteria. the exercise is  most useful when comparing projects of a  similar size with similar costs.   because of the use of cost as a divisor,  projects that are smaller and that cost  significantly less money than other projects   evidence based library and information practice 2007, 2:3  83   criterion  definition  rating  scale   unit of measure/quantification  exposure  a measure of the amount of  positive press and exposure  that is likely to result from  digital availability of  collection.  5  stature of university greatly enhanced.  likely to receive widespread national and  international attention upon project  completion. likely to result in future  grants and collaborative opportunities.       4  likely to receive regional and national  attention. likely to result in future grants  and collaborative opportunities.      3  likely to receive some regional and  limited national attention. likely to result  in smaller regional grants.      2  likely to receive limited regional and no  national attention. no potential for  receiving future grants based on collection  digitization.      1  likely to receive no attention outside of  the institution.  table 4. exposure  under consideration will have a higher  ranking bias. therefore, when projects of  disparate size are compared, institutions  may wish to calculate cost as a separate  consideration and rank projects according to  the performance criteria alone. to rank  projects independent of cost, in the example  scenarios listed below, institutions would  compare the performance totals for each  project and costs separately.     project a is a collection of 1500 original  plant specimens and the accompanying  original descriptions of the specimens. the  specimens are essential to the study of the  botanical history of the region. the  specimen collectors, many of whom are  prominent botanists in the region, authored  the accompanying descriptions that  appeared in research journals. the  specimens are entirely unique. bringing the  type specimens together with the  descriptions is unique among type specimen  digitization projects from around the  country. the institution’s strategic plan  identifies the management of natural  resources as a focus and strength.    project b is a proposal to digitize 100  documents pertaining to a variety of cherry  developed at and marketed by a department  of the university since the 1980s. there is  minimal research available pertaining to this  cherry variety owing to the fact that it is a  patent‐protected crop. little is known about  it outside of the institution. the cherry is  potentially important to the state economy,  and the department plans to export the  cherry variety nationally and  globally. digitizing the documents could  assist in marketing the cherry and provide  exposure for the work of the developers of  the crop. the documents include technical  reports produced by the department; and,  while they are well known and widely used  within the department, they are largely  unknown outside the department.      evidence based library and information practice 2007, 2:3  84   performance criteria  performance (p)  cost (c) in  thousands   value  (v)  significance (internal)*2  8  25      significance (external)  5        uniqueness  5        exposure  5        total performance  23        total performance score  (multiplied by 100)  2300          2300  25  p/c=v  92    project a      performance criteria  performance (p)  cost (c) in  thousands   value  (v)  significance (internal)*2  4  20      significance (external)  5        uniqueness  5        exposure  3        total performance  17        total performance score  (multiplied by 100)  1700          1700  20  p/c=v  85     project b      performance criteria  performance (p)  cost (c) in  thousands   value  (v)  significance (internal)*2  2  15      significance (external)  5        uniqueness  1        exposure  3        total performance  11        total performance score  (multiplied by 100)  1100          1100  15  p/c=v  73    project c          evidence based library and information practice 2007, 2:3  85 performance criteria  performance (p)  cost (c) in  thousands   value (v)  significance (internal)*2  10  20      significance (external)  5        uniqueness  5        exposure  3        total performance  23        total performance score  (multiplied by 100)  2300          2300  20  p/c=v  115    project d      project c is a proposal to digitize a  collection of classic ghost stories authored  by prominent authors such as joseph  sheridan, le fanu, henry james and edgar  allan poe. while most of the stories have  already been digitized, this project also  digitizes literary criticism of the stories. the  primary users are students in the english  department. the university offers a  graduate degree in english literature that  graduates five to ten master’s degree  students per year. the literary criticisms will  be widely used by scholars around the  world.    project d is a proposal to digitize 1100  special reports published by the university’s  agricultural experiment station from the  mid 1950s to the present. beginning in the  2000s the reports are available online. the  library is discussing a project with the  department to house these reports in the  university’s institutional repository.  digitizing the older documents provides  access to the entire run of the series online.  most of the publications are cataloged but  remain difficult to find. scanning and  providing keyword searching of the  documents dramatically increases their  availability and strengthens the library’s  collections in this area.  conclusion    as libraries increase their digitization efforts,  many of the same selection principles  applied in the selection of print materials are  used to select digitization projects. other  considerations, such as campus politics and  budgetary issues, may also enter into project  selection decisions. because of the  sometimes large‐scale and high cost of  digitization projects, library administrators  should select projects carefully according to  established criteria.     value engineering is a structured thought  process that assists administrators in the  selection of significant and unique content  for digitization that enhances the stature of  the university, using established criteria in  the ranking of projects. the use of the value  engineering methodology as a tool and  process provides decision‐makers with  objective, quantitative data upon which  selection decisions can be based. the use of  established rating criteria provides selectors  with information about how their proposed  collection will be evaluated.  finally, the  transparency of the process output is easily  communicated with stakeholders.          evidence based library and information practice 2007, 2:3  86 works cited    bick, dawn and sinha reeta. “maintaining a  high‐quality, cost effective journal  collection”. college & research  libraries news 52.88 (1991):  485‐49.    gershenson, john k., christine e. hailey, j.  clair batty and warren f. phillips.  “application of value engineering  techniques in curriculum  development and review.”  international journal of engineering  education. 18.2 (2002): 140‐150.    government printing office. “gpoʹs  digitization and preservation  initiatives.” 27 sept 2006. 21 mar 2007  <http://www.gpoaccess.gov/legacy/ind ex.html>.    hays, robert h. “qualitative insights from  quantitative methods.” using logical  techniques for making better  decisions. ed. douglas dickson. new  york: harvard business review, 1984.  124‐139.    jakhanwal, s.p. and singh, g.b.  “application of “value engineering”  to rationalize the cost of cataract  surgery in eye clinic at tata main  hospital.” journal of the academy of  hospital administration 13.2 (2001). 29  may 2007  <http://www.indmedica.com/journals. php?journalid=6&issueid=20&articleid =164&action=article>.    mudge, arthur e. value engineering: a  systematic approach. new york:  mcgraw‐hill book company, 1971.    national information standards  organization. “a framework of  guidance for building good digital  collections.” 2004. 12 aug 2006  <http://www.niso.org/framework/fram ework2.html>.    pearsall, mike and p. eng. project  performance measurement. 2004.  value engineering, ministry of  transportation. 11 nov 2006  <http://www.scav‐ csva.org/postconf2004‐ mp/csva%20track3/track%203‐ 4%20oct%2026%20pearsall.pdf>.   available from author of this paper.      pershing, gwendolyn and mary krutulis.  “using a value engineering technique  to determine salary equity among  academic libraries.” library  administration & management 10.4  (1996): 240‐245.    quint, barbara. google and research  libraries launch massive digitization  project 2004. 10 oct 2006.  <http://newsbreaks.infotoday.com/nbre ader.asp?articleid=16307>.    shenton, helen. “developing life cycle  models at the british library – work in  progress.” digital preservation forum.  2002. 8 oct 2006 <http://  www.dpconline.org/graphics/events/p resentations/pdf/lifecycledpc.pdf>.    the university of michigan digital library  services. “assessing the costs of  conversion: making of america iv:  the american voice 1850‐1876.” july  2001. 25 may 2007  <http://www.umdl.umich.edu/pubs/m oa4_costs.pdf>.    ʺvalue engineering.ʺ wikipedia. 11 nov.  2006. answers.com 19 mar. 2007.  <http://www.answers.com/topic/value‐ engineering>.  http://www.gpoaccess.gov/legacy/index.html http://www.gpoaccess.gov/legacy/index.html http://www.indmedica.com/journals.php?journalid=6&issueid=20&articleid=164&action=article http://www.indmedica.com/journals.php?journalid=6&issueid=20&articleid=164&action=article http://www.indmedica.com/journals.php?journalid=6&issueid=20&articleid=164&action=article http://www.niso.org/framework/framework2.html http://www.niso.org/framework/framework2.html http://www.scav%e2%80%90csva.org/postconf2004%e2%80%90mp/csva%20track3/track%203%e2%80%904%20oct%2026%20pearsall.pdf%00%00 http://www.scav%e2%80%90csva.org/postconf2004%e2%80%90mp/csva%20track3/track%203%e2%80%904%20oct%2026%20pearsall.pdf%00%00 http://www.scav%e2%80%90csva.org/postconf2004%e2%80%90mp/csva%20track3/track%203%e2%80%904%20oct%2026%20pearsall.pdf%00%00 http://www.scav%e2%80%90csva.org/postconf2004%e2%80%90mp/csva%20track3/track%203%e2%80%904%20oct%2026%20pearsall.pdf%00%00 http://www.scav%e2%80%90csva.org/postconf2004%e2%80%90mp/csva%20track3/track%203%e2%80%904%20oct%2026%20pearsall.pdf%00%00 http://newsbreaks.infotoday.com/nbreader.asp?articleid=16307 http://newsbreaks.infotoday.com/nbreader.asp?articleid=16307 http://www.dpconline.org/graphics/events/presentations/pdf/lifecycledpc.pdf http://www.dpconline.org/graphics/events/presentations/pdf/lifecycledpc.pdf http://www.dpconline.org/graphics/events/presentations/pdf/lifecycledpc.pdf http://www.umdl.umich.edu/pubs/moa4_costs.pdf http://www.umdl.umich.edu/pubs/moa4_costs.pdf http://www.answers.com/topic/value%e2%80%90engineering%00 http://www.answers.com/topic/value%e2%80%90engineering%00 evidence based library and information practice 2011, 7.4 150 evidence based library and information practice news/announcements beta phi mu/lrrt research paper award for 2013 2012. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. this award is being jointly presented by the beta phi mu international honor society (http://www.beta-phi-mu.org/) and the american library association’s library research round table (http://www.ala.org/lrrt/) to recognize excellent research into problems related to the profession of librarianship. any ala member is eligible for this $500 award, and all methodologies and research topics/questions are eligible for consideration. the criteria to be followed for the selection of an award winner are:  importance of the research question or problem  adequacy of the review of relevant literature  appropriateness of the methodology used  effectiveness of the application of the methodology  addition of the findings to the knowledge and/or praxis in the field of librarianship  articulation of the conclusions emanating from the study  clarity and completeness the page limit for submissions will be thirty (30) double-spaced pages plus bibliography. only complete papers will be considered and submissions should be made electronically to the contact person below. the submissions must not have been published prior to march 1, 2013 and should follow apa style. individuals may submit only one paper. jointly authored papers are acceptable, but all authors must be ala members, and will split the award of $500. the deadline for submission is march 1, 2013. all submissions that meet the deadline and the criteria (including length of paper) will be considered. the papers will undergo a blindreview process by a joint bpm/lrrt award committee and the winner will be notified by may 1, 2012. please include a title page with title of paper and author contact information including name, institutional affiliation, mailing address and email address. the award will be presented during one of lrrt’s research programs at the ala annual meeting in chicago, il, june 27-july 2, 2013. email submissions as word documents only to: john m. budd beta phi mu-lrrt research paper chair school of information science & learning technologies 303 townsend university of missouri columbia, mo 65211 buddj@missouri.edu https://uos-portal.salford.ac.uk/owa/redir.aspx?c=40f39f4996424bc1bf62a2b121712034&url=http%3a%2f%2fwww.beta-phi-mu.org%2f https://uos-portal.salford.ac.uk/owa/redir.aspx?c=40f39f4996424bc1bf62a2b121712034&url=http%3a%2f%2fwww.ala.org%2flrrt%2f https://uos-portal.salford.ac.uk/owa/redir.aspx?c=40f39f4996424bc1bf62a2b121712034&url=mailto%3abuddj%40missouri.edu evidence based library and information practice 2011, 7.4 151 evidence summary   user education and file standards best options to ensure open educational resources are truly open   a review of: ovadia, s. (2019). addressing the technical challenges of open educational resources. portal: libraries and the academy, 19(1), 79-93. https://doi.org/10.1353/pla.2019.0005   reviewed by: jordan patterson cataloguing and metadata librarian memorial university of newfoundland st. john’s, newfoundland and labrador, canada email: jpatterson13@mun.ca   received: 29 nov. 2019                                                                 accepted:  28 jan. 2020      2020 patterson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29685     abstract   objective – to describe common technical challenges of open educational resources (oers) and recommend solutions.   design – descriptive study.   setting – online open educational resources in higher education.   subjects – open educational resources.   methods – drawing from the literature and his own experiences, the author explains the necessity of accepted standards of “openness” and describes the many ways oers fail to meet these standards. the author also describes common technical challenges that impede openness, then proposes solutions to address these challenges.   main results – technical limitations often prohibit oers from being truly open. providers can design their resources to encourage reuse, redistribution, revision, and remixing. three strategies for addressing technical challenges in oers are user education, open file standards, and using git to facilitate distributed version control.   conclusion – git is a compelling option for distributed version control, but entails its own technical challenges. user education and established open file standards are the best strategies to ensure that oers are open in both a legal and a technical sense. the article concludes with the author’s opinions about how oer directors may most realistically implement these solutions.   commentary   libraries have embraced openness as a guiding ideal in the internet age, but in 2020, what openness looks like is far from a closed question. today, work toward a positive open paradigm continues on many fronts: linked data technologies and bibframe seek to liberate bibliographic data from the rigid shackles of marc; research data management aims at greater transparency for published studies in all disciplines; scholars remain cautious of predatory open access publishing. as the author demonstrates, there is a gap to bridge between openness in theory and openness in practice in many areas of librarianship, and oers are no exception.   the article is not a systematic study of the technical challenges of oers per se, and did not lend itself to evaluation using a typical critical appraisal checklist, such as glynn’s (2006). instead, the article addresses, in broader strokes, the incongruities between an accepted conceptual standard of openness and the realities of implementation, where technology creates barriers in spite of our goodwill and desire to maximize access to materials. the author introduces and describes this standard—the four r’s: reuse, redistribute, revise, and remix—and dedicates roughly a third of the article to demonstrating how oers can, and do, fall short of it. while, as mentioned, the article is not a systematic study, the problems described will be so familiar to the average internet user, and the paper is so well referenced with citations to open access authorities such as david wiley and michael w. carroll, that there is little reason to seriously doubt the article’s claims.   the article’s solutions remain speculative as they are not, strictly speaking, tested. nevertheless, the author’s propositions are well-informed, based on previous research, and offer a clear starting point for a researcher or oer director looking to practically address certain issues. user education and document formatting standards may prove to be relatively easy to test and implement. the other solution, “a graphical tool that harnesses the complexity of git while shielding users from that complexity,” may prove more difficult.   while libraries will certainly find specific points about oers relevant, there are more general lessons to heed. the article illustrates the necessity of perpetual realignment with ideas as they evolve. despite our best intentions, materials will remain “open in theory but closed in practice” if we do not work to keep pace with technological and theoretical developments. we might find, for instance, that what passed for open in 2010 doesn’t pass at the decade’s end. we are reminded, too, that this upkeep is not simply in adherence to some arbitrary ideal, but is in the service of our patrons.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154   microsoft word es 1460_haglund evidence based library and information practice 2008, 3:2 48 evidence based library and information practice evidence summary students with non-proficient information seeking skills greatly over-estimate their abilities a review of: gross, melissa, and don latham. "attaining information literacy: an investigation of the relationship between skill level, self-estimates of skill, and library anxiety." library & information science research 29.3 (2007): 332-53. reviewed by: lotta haglund head of information and public relations karolinska institutet, university library, stockholm, sweden email: lotta.haglund@ki.se david herron scholarly developer karolinska institutet, university library, stockholm, sweden email: david.heron@ki.se received: 25 february 2008 accepted: 23 april 2008 © 2008 haglund and herron. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – the objective of this study is an investigation of the relationship between students’ self-assessment of their information literacy skills and their actual skill level, as well as an analysis of whether library anxiety is related to information skill attainment. design – quantitative research design (information literacy test (ilt), library anxiety scale (las), pre and post surveys). setting – florida state university, united states. subjects – students, incoming freshmen. methods – information literacy skills were measured using the information literacy test (ilt), presenting subjects with 65 multiple choice items designed around four of the five acrl information literacy standards, in which students were expected to: 1) determine the nature and extent of the information needed; 2) access needed information effectively and efficiently; 3) 49 evaluate information and its sources critically and incorporates selected information into his/her knowledge base system; 4) understand many of the economic, legal and social issues surrounding the use of information and accesses and uses information ethically and legally. the ilt categorized participant scores as nonproficient (<39), proficient (30-53), or advanced (54 or higher). the test had earlier been validated and tested for reliability (cameron, wise & lottridge). library anxiety was measured using the library anxiety scale (las). the las test asks participants to rate 43 statements using a likert-type scale, with ratings from 1 to 5. the scale can also be used to produce five subscales, identified as 1) barriers with staff; 2) affective barriers; 3) comfort with the library; 4) knowledge of the library; 5) mechanical barriers. fifty percent of the students were given the ilt first, and the other half of the respondents was given the las first, in order to neutralize the effect of the ilt itself. the two tests were sandwiched between preand post-surveys that were designed to collect demographic information, measure self-assessment of skills, and gather information about exposure to information literacy skills instruction. the students were asked to estimate their performance on the ilt both before and after taking the test, in terms of the expected percent of questions they would be able to answer correctly, the number of questions they would answer correctly, and how their performance would compare to the scores of the other incoming freshmen (in percentages). the students participating in the study represented the top and the bottom 25 % of the incoming class (as assessed by gpa and sat/act scores) who were participating in summer session. they were contacted via email and asked to participate in the study. there were 51 participants in the study, 33 in the top quartile and 18 in the bottom quartile. they were given the incentive of a gift certificate for the university bookstore, and that those scoring in the top 15 % of all participants would be included in a raffle for an additional 4 gift certificates. also, a response time effort analysis was performed, data produced being consistent with times suggesting all participants were engaged in the test. all data were coded and analyzed using spss, for all tests alpha was set at p=0.05. main results – the main aim of the study was to test the hypothesis that students who test non-proficient on an information literacy test tend to overestimate their competency to a higher degree than proficient and advanced students. in the preand post-surveys, the students were asked to estimate their performance on the ilt in terms of the expected percentage of questions they would answer correctly, the number of questions they expected to answer correctly, and how their performance on the ilt would compare to others taking the test (in percentage). the results of the study show that all students overestimate their abilities, both in terms of performance and relative performance, in the pre-survey. the estimated percentage correct answers for the whole group was 75%, but the actual percentage correct was 65%. the estimated score was 50 and the actual 39, and the estimated comparison with their peers was 77% and the actual 53%. all three measures demonstrated a significant difference between estimated and actual values. on the ilt, the mean score for the bottom tier of the students was 34, and the mean score for the top tier was 42, showing a significant difference between the top and 50 bottom tier in a t-test. on the ilt, 23 students scored less than 39 (= nonproficient), 27 scored between 39-53 (proficient). only one student, a top quartile participant, showed advanced information literacy skills (a score above 53). in the post-survey, the students still overestimated their performance, but to a lesser degree. all three groups adjusted their self-estimates in the post-survey in response to information skills testing, but the non-proficient group overestimated their skills to a higher degree, on both pre-and post-surveys. the estimated percentage correct answers for the whole group was 69%, but the actual was 65%, the estimated score was 44 and the actual 39, and the estimated comparison with their peers was 70% and the actual 53%. all three measures demonstrated a significant difference. all results show that the original hypothesis holds: non-proficient students overestimate their abilities to a greater degree, both in terms of score and performance relative to their peers. the las was used to see if there was a relationship between student scores on the ilt and library anxiety. bi-variate analysis was performed on the ilt scores and student total scores on the las, and the results show that library anxiety tended to decrease with higher scores on the ilt. this result was not expected from the theory. in the pre-survey, the students were asked how they had obtained their information literacy skills. the top tier indicated a reliance on more formal sources (e.g. school library media center, classroom, and/or public library), while the bottom tier relied on less formal sources (e.g. friends and selfteaching). this suggested that library instruction was not effective reaching the weakest group of students. in addition, nearly 75% of all participants responded that they were largely self-taught. conclusions – the results show a difference between the level of information literacy skills between lower-performing and higher-performing freshmen, but the study as a whole gives some evidence that many students could be information illiterate when entering university (45% tested nonproficient). there is an association between scores on an information literacy test and students’ estimates of their information literacy skills. students who demonstrate low-level skills hold even more inflated views of their abilities, and the very competent may tend to underestimate their performance. however, the limited number of subjects in this study makes further analysis impossible. no correlation was found between performance on the ilt and the experience of library anxiety as measured by the las, although the subscale “knowledge of the library” demonstrated a negative relationship with information literacy skills. that is, the higher the score, the less the anxiety. there are several limitations of the study (which the authors acknowledge). one limitation that is important to note the small sample size, which limits the possibility to generalize the findings to the broader population of incoming students. more research is needed on how to best design, market, and deliver information literacy education to students with nonproficient information literacy skills since they seem to be unaware of their deficiencies. commentary this article was exceptionally well written making it a pleasure for the reviewers to read. the study design seemed to be very rigorous. however, there are nevertheless 51 some minor shortcomings of the study, described in the following sections. a good case for the design of the study is made, providing good evidence for the validity of the study. although the study can be judged as reliable, the assessment of reliability would benefit from additional information about the total student population. one also wonders on which grounds the authors have decided on the preferred 60 respondents (n=51). to enhance the understanding of the reader, the tables and figures of a text should complement each other, and this is not always the case in this article. the titles to the figures should be more descriptive, making them easier to interpret. the presentation of the study results is sometimes repetitive, in text and figures, especially noted in relation to figure 1 and 2 and tables 4-6. this somewhat confusing presentation is a pity since here lies the climax of the results, and the conclusion of the study would have been clearer had the results been presented in a less confusing way. another confusing factor is that the authors discuss the figures in the text, jumping from one to the other without referring to them directly by name. in relation to figure 2, one wishes that the authors had explained the calculations behind it, since this would have made it easier to understand. on page 345, the mean estimate comparing performance to other incoming freshmen by percentile is given as 73 (sd=11.853, n=21). this must be a mistake since the number cannot be found in any of the tables, and neither 73 nor n=21 can be found in table 6. on a deeper pedagogical level, one can wonder about the usefulness of testing information literacy skills out of subject context, since this study describes assessment of cognitive skills in a decontextualized and non-subject specific environment. it would also have been interesting to see how the ilt relates to a taxonomy of knowledge, for example, bloom’s taxonomy, or the now popular (in europe) solo taxonomy. however, these issues are probably beyond the scope of the study as well as this evidence summary. some of the authors’ conclusions also seem somewhat outside of the span of the article. for example, the authors suggest that the study “presents a new way of thinking about how to improve student learning in both traditional and distance learning environments,” and this assertion is unsupported. however, this does not influence the overall assessment of the article. the authors also conclude that this study, together with other similar recent studies, might form a basis for additional research on how to set up information literacy education in innovative ways. this may well be the case, and library practice would benefit from this; hitherto, no studies have been able to give an answer to the question of which method of user education works best and for which user group (brettle). works cited brettle, alison. “information skills training: a systematic review of the literature.” health information and libraries journal 20.s1 (2003): 3-9 cameron, l. and s.l. wise and s.m. lottridge. “the development and validation of the information literacy test.” college and research libraries 68 (2007): 22936. microsoft word es3_927_final evidence based library and information practice 2008, 3:1  61 evidence based library and information practice       evidence summary    post‐secondary students prefer im to e‐mail for personal and social communication    a review of:  lancaster, sean, david c. yen, albert h. huang, and shin‐yuan hung. “the selection     of instant messaging or e‐mail: college students’ perspective for computer    communication.” information management & computer security 15.1 (2007):    5‐22.      reviewed by:  virginia wilson  shirp coordinator, health sciences library, university of saskatchewan  saskatoon, saskatchewan, canada  e‐mail: virginia.wilson@usask.ca      received: 30 november 2007    accepted: 20 january 2008      © 2008 wilson. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – this study investigates college  students’ perceptions of instant messaging  (im) and e‐mail for conveying emotions,  aiding in relationship building, ease of use,  and reliability.    design – a survey consisting of 59  questions was administered to 1,000 college  students, resulting in 545 usable responses.    setting – the research took place at a  midwestern university in the united states.    subjects – 1,000 management information  systems (mis) college students.    methods – a 59‐question survey was  distributed to 1,000 mis students during the  2005 academic year. 545 usable responses  were returned. participation was voluntary.  during the pre‐phase of the research, four  categories were determined: emotion,  relationship, usage, and reliability.  questions were then written for a pilot  study using likert scaling. the post‐research  phase involved evaluating the questions  linguistically to ensure proper word usage,  comprehension, and lack of bias.    evidence based library and information practice 2008, 3:1  62 main results – the questions in the section  on conveying emotion dealt with how well  the two technologies (e‐mail and im)  communicated feelings and emotions. while  both technologies were acknowledged as  being able to communicate more than  merely text, im was clearly preferred for  expressing emotion. fifty‐two percent of the  respondents strongly agreed or agreed that  they used emoticons (originally symbols  created with letters and special characters;  later evolving into graphical images  produced and made available by im  companies) to express emotion in im, while  fewer than 11% agreed or strongly agreed  that they did so in e‐mail. more than 70% of  the respondents strongly agreed or agreed  that their friends used emoticons in im,  while fewer than 14% strongly agreed or  agreed that their friends used emoticons in  e‐mails. more than 75% of respondents  agreed that it is easier to convey emotions in  im than via e‐mail.     analysis on the questions that dealt with the  technologies as useful relationship builders  again showed a preference for im. im was  preferred by a greater number of  respondents for fostering friendships,  improving relationships with friends or  team members, building relationships, social  interaction, and social networking. this  section also found that more than 75% of the  respondents felt that im was more useful  than e‐mail when interacting with friends.  however, when asked about interacting  with co‐workers specifically, only 44% were  in agreement. nearly 32% were neutral on  this point, while 24% were in disagreement.  it appears that im is preferred over e‐mail  for fostering social and friendly  relationships, but is not preferred when the  interaction is work‐related and more  professional.    in terms of use, both technologies were  considered easy to learn and use by more  than 90% of respondents. in terms of  preferred use, more than 60% of  respondents use im for personal and social  interaction, while less than 1% uses im for  work‐related activities. nearly 80% of  respondents preferred using e‐mail for  work, and nearly 75% preferred using im  for social interaction. sixty‐three percent of  respondents did not agree that im is just as  effective as face‐to‐face meetings, while  nearly 75% did not agree that e‐mail can be  as effective as meeting face‐to‐face.    in the areas of reliability, security, and  privacy of im and e‐mail, respondents  perceived e‐mail more favourably. sixty‐ four percent of respondents agreed or  strongly agreed that im is very reliable,  while 80% of respondents felt the same way  about e‐mail. just over 30% of those  questioned agreed that im is very secure,  while more than 50% felt that way about e‐ mail. however, the gap is narrower when  users were queried as to whether or not they  worry about security: 13% strongly agreed  with the statement that users do not worry  about their security when using im, while  15% strongly agreed with the statement as it  pertained to e‐mail. the same types of  questions were asked about perceived  privacy. nearly 50% agreed that their  privacy is protected with e‐mail, while just  under 35% had the same belief about im.  more than 60% of respondents reported that  they do not worry about privacy when e‐ mailing, while about 53% of respondents do  not worry about privacy with im.    conclusion – im was perceived more  favourably than e‐mail in 3 of the 4  categories: conveying emotions, building  relationships, and ease of use. in the  reliability category, e‐mail was perceived  more favourably. however, there is a  division between social interaction and  professional communication. im is preferred  for personal and social relationships, while  e‐mail is preferred for work‐related  evidence based library and information practice 2008, 3:1  63 communication. neither of the technologies  is as desirable as face‐to‐face meetings.     commentary    this research project was fairly  straightforward in its execution. the 59‐ question survey was ambitiously  constructed to cover a wide range of topics:  use, reliability, building relationships, and  conveying emotions. the survey was  designed using likert scaling, but it was not  made clear what the likert scale range was  for this particular survey. additionally, the  last four questions in the survey (56‐59  inclusive) appear to be open ended  questions where the likert scaling would  not be applicable. it is unclear as to how  these questions were handled. the results of  questions 56‐59 are reported in the text of  the article as well as via figure 7. the two  methods of reporting the results do not  seem to coincide. the figure is somewhat  confusing. the authors also fail to mention  how they established the validity and  reliability of the survey instrument.    as for the results of the survey, the analysis  was merely descriptive and not inferential,  so no conclusions may be drawn. there is  one discrepancy in the article, which was  cleared up with an e‐mail to the  corresponding author: the text of the article  states that the survey consisted of 69  questions, while the list of questions (table  1) numbered 59. the author confirmed that  it was a 59‐question survey.     the paper includes a brief section on the  evolution of im as well as a broad literature  review on eleven different topics:  information richness, ease of use, use of  emoticons, multimedia, playfulness, flow,  cognitive fit, bounded rationality, media  selection, perceived commitment, and user  satisfaction. all eleven dimensions inform  the construction of the survey questions to  seemingly variable degrees. although the  literature review is comprehensive in terms  of items touched upon, some dimensions are  merely given cursory treatment by the  authors, with one or perhaps two articles  pertaining to the topic. it is a broad  literature review, but not a deep one. and  while the literature review is lacking in  depth in some areas, the authors are clear  and succinct in their provision of a  theoretical background.    the authors recognize the limitations of  their study, which chiefly involve the  selection of the study subjects. college  students of approximately the same age are  a narrow group, so it is somewhat unlikely  that the results of this study could be  applicable to other user groups.  additionally, it is likely that among the  chosen respondents, any work‐related  conclusions would be drawn from course  and group work rather than actual work  experience. a further limitation, and one not  stated in the paper, is that the students are  all from the management information  systems program. at least, it is assumed  that this is the program, based on the  primary authors’ affiliation and the  information found at the university website.  the article does not explicitly define the mis  acronym. this program places “an emphasis  on state‐of‐the‐art computing and it skills”  (decision sciences). not only are the  subjects college students of a similar age,  they are also all in an academic program  dealing with high level computer  knowledge. there is the assumption that  they are atypical users of technology. these  factors taken together suggest that applying  the study results to other user groups  should be done with caution.    despite the limitations, the results of this  research would be beneficial if applied in a  college or university setting in decision  making around the area of reference  services. many institutions offer chat  reference or reference services by e‐mail.  evidence based library and information practice 2008, 3:1  64 having a greater understanding of how  college students perceive these technologies  could help in deciding which kind of service  to offer.       works cited    decision sciences & management   information systems. 2007. farmer  school of business, miami  university. 27 nov. 2007  <http://www.fsb.muohio.edu/  departments/dsc‐mis >.  evidence based library and information practice 2011, 6.3 56 evidence based library and information practice evidence summary retention initiatives are employed in academic libraries, although not necessarily for this purpose a review of: strothmann, m., & ohler, l. a. (2011). retaining academic librarians: by chance or by design? library management, 32(3), 191-208. doi: 10.1108/01435121111112907 reviewed by: laura newton miller science & engineering librarian carleton university ottawa, ontario, canada email: laura_newtonmiller@carleton.ca received: 18 may 2011 accepted: 2 aug. 2011 © 2011 newton miller. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to study methods that support retention of academic librarians. design – exploratory research using an online survey; non-random sample. setting – academic libraries, nearly all located within the u.s. (97.2%). subjects – a total of 895 professional academic librarians. methods – the researchers sent an online survey link to professional electronic mail lists and directly to heads of association of research libraries (arl) member libraries. the 23-item survey was available from february 19, 2007, through march 9, 2007, and contained questions about the professional experience of respondents, their libraries, and their universities. subjects were asked to identify retention activities that were currently offered at their workplaces (both library-specific and university-wide) and to rate their satisfaction for each available initiative. the list contained fifteen initiatives based on the researchers’ literature review. http://dx.doi.org/10.1108/01435121111112907� mailto:laura_newtonmiller@carleton.ca� evidence based library and information practice 2011, 6.3 57 main results – almost half (46.3%) of respondents were 50 or older and 7.5% under 30 years old, leaving 46.2% between the ages of 3050 years old (although this percentage is not explicitly stated in the paper except in a table). nearly half of the subjects were in the first ten years of their careers. 80.2% had held between one and four professional positions in their careers, and even when length of professional experience was factored out, age had no effect on the number of positions held. most job turnover within the past three years (3 or fewer open positions) was in public service, while other areas of the library (i.e., technical services, systems, and administration) reported zero open positions. only 11.3% of respondents noted that their libraries have deliberate, formal retention programs in place. despite this, there are several libraryand university-based initiatives that can be considered to help with retention. the most reported available library-based retention initiative was the provision of funding to attend conferences (86.8%). librarians also frequently reported flexible schedules, support and funding for professional development and access to leadership programs. university-based retention programs included continuing education funding, new employee orientations, faculty status, and the chance to teach creditbearing courses. only 22. 2% of subjects reported formal mentoring programs as a retention strategy. librarians were very or somewhat satisfied with schedule flexibility (79.6%). they were generally satisfied with other initiatives reported. in response to 22 five-point likert scale descriptions of positive library work environments, subjects most agreed with statements that allowed librarians to have control of their professional duties, that allowed for personal or family obligations, and that supported professional development. librarians agreed less often regarding statements about salaries, research support, and opportunities for advancement. conclusion – academic librarians are involved in and are benefitting from some library and university-based retention initiatives, even though retention may not be the primary strategic goal. commentary this study examines librarians’ views on retention efforts in academia. as an exploratory study, the authors should be commended for attempting to fill a gap in the evidence concerning retention of academic librarians. the researchers make limitations clear to the reader, including the fact that as an exploratory study, the potential list of strategies may not be exhaustive. although choosing specific heads of libraries to fill out the form was important in order to get some management perspective, they cautioned readers against generalizing from the results to the entire library population. however, it is interesting that the authors decided to survey heads of arl libraries while invitations to librarians were from academic libraries of all sizes. the authors note that there were substantial differences between the responses of administrators compared with librarians, but it is difficult to know if these differences were more between larger research libraries and smaller libraries, as opposed to differences between administrators and staff. although the methods are clearly described and easy to follow, inclusion of the actual survey used would have enhanced transparency and reproducibility. calculation of a response rate is not possible because there is a lack of information about how many surveys were actually distributed. although sometimes difficult with web-based surveys, information about how many library directors were approached or how many librarians subscribed to the professional electronic mail lists could provide some context around response rate. there is a potential problem that is not mentioned in the paper that could skew results. 895 professional academic librarians answered evidence based library and information practice 2011, 6.3 58 the survey, but it is unclear how many were from the same libraries. in other words, if more than one librarian answered questions about retention strategies employed in their libraries, it could make one particular retention strategy seem to be more popular than another, even though it is only one library’s activity. it does not affect data regarding respondents’ opinions about a particular retention strategy, but readers should be aware that the most reported retention initiative numbers may be distorted. it would have been interesting to read more about generational issues and retention. the authors discussed that almost half of the respondents were 50 or older (“baby boomer”) and that only 7.5% were under 30 (“millennial”). however, the authors do not consider that 46.2% were between the ages of 30 and 50 (“generation x”). perhaps it’s this reviewer’s particular bias toward that generation, but it would have been beneficial to see more discussion on these issues. it would be interesting to conduct the survey again, since economic conditions were arguably not as bad in 2007 as those to follow in 2008/09 onwards. “millennials” are now in similar, if not worse, economic conditions than “gen xers” were when first looking for professional employment. because some of the literature included in this section is somewhat dated, it would be a valuable study to explore in further detail. some degree of turnover is healthy in academic environments in order to bring about innovation and change. limited turnover can also negatively affect opportunities for promotion. the challenge for administrators is to establish a balance between staff retention and turnover. this paper is a good first effort in understanding retention initiatives, and should be read by academic librarians who hold management or administrative positions in order to better inform themselves of important human resource issues. / evidence based library and information practice news   call for applicants for eblip journal: associate editor (research articles)    2020. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29781     evidence based library and information practice (eblip), a peer reviewed open access journal published since 2006 seeks to fill a position of associate editor (research articles). the successful candidate will be expected to begin their role by october 2020 and serve a 3 year term (with possibility of renewal). this person will share the workload with a second associate editor.   the associate editor (research articles) will be responsible for:   overseeing the complete editorial process for items submitted to the research articles section (this includes assignment of peer reviewers, monitoring the peer review process, communicating with authors and peer reviewers, making acceptance decisions, ensuring required changes to manuscripts are made, communicating with copyeditors and the production editor to ensure the final copy is as expected). communicating and consulting with the editor-in-chief on a regular basis. attending editorial team meetings on a regular basis and participating in the overall governance of the journal. ensuring that all necessary deadlines are met. communicating with potential authors and responding to queries.   for more information about research articles and other types of submissions accepted by eblip see: https://journals.library.ualberta.ca/eblip/index.php/eblip/about/submissions#authorguidelines   the ideal candidate for associate editor (research articles) will be well-versed in evidence based practice and research methods. this position requires dedicated time on a regular basis, and it is therefore essential that interested persons ensure available time to devote to this position prior to applying. it is estimated that the workload is approximately 10-15 hours per month.   interested persons should send a cover letter, indicating areas of strength they would bring to the role, and resume/cv as a single pdf file to lorie kloda, editor-in-chief, at lorie.kloda@concordia.ca by july 15, 2020. the current associate editors (research articles) are happy to answer specific queries about the role on request. please email ann medaille (amedaille@unr.edu) and lisl zach (lisl@marsez.com).   **please note that evidence based library and information practice is a non-profit, open access journal and all positions are voluntary and unpaid.   about the journal:   published quarterly, and hosted by the university of alberta, this peer-reviewed, open access journal is targeted at all library and information professionals interested in an evidence based model of practice. by facilitating access to librarianship research via original research articles and evidence summaries of relevant research from the library literature, evidence based library and information practice will enable librarians to practice their profession in an evidence based manner.   please visit the evidence based library and information practice web site (http://journals.library.ualberta.ca/index.php/eblip) for further information about the journal.         evidence based library and information practice 2011, 6.3 90 evidence based library and information practice ebl 101 research methods: design, methods, case study…oh my! virginia wilson liaison librarian, murray library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca received: 13 aug. 2011 accepted: 16 aug. 2011 © 2011 wilson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. as this column transitions into looking at various research methods, some preliminary topics must be explored in order to put research methods into context. last time i looked at qualitative and quantitative research in general, and discussed the background of both types of research and the tension between researchers invested in each of these methods. lately, the mixed methods approach has gained traction, and i will discuss that in forthcoming columns as well. but first, in order to situate research methods, this column will explore research design. before undertaking research, there must be a plan, and that is where design comes in. but research design is not merely a work plan. research design is “the entire process of research from conceptualizing a problem to writing research questions, and on to data collection, analysis, interpretation, and report writing” (bodgan & taylor, 1975, p. 5, as cited in creswell, 2007). it is the “logical sequence that connects the empirical data to a study’s initial research questions, and ultimately, to its conclusions” (yin, 2003, p. 5, as cited in in creswell, 2007). the confusing part is that, while reviewing research literature as well as websites focusing on research, i have found that quite often the terms research design and research methods are used inconsistently. for example, the terms research design and research method are both used to refer to a case study, a third, and different, entity. let’s think about “case study” for a minute. the chapter on qualitative case studies in the sage handbook of qualitative research states that a “case study is not a methodological choice but a choice of what to be studied,” and goes on to say that “by whatever methods, we choose to study the case” (stake, 2005, p. 443). the book case mailto:virginia.wilson@usask.ca� evidence based library and information practice 2011, 6.3 91 study research: design and methods “covers the distinctive characteristics of the case study as a research method” (yin, 2003, p. 2). so is the case study both a method and a design? is it a method sometimes and a design at others? and where is the definitive answer? there really isn’t one. just as i was about to throw my hands up in despair, i came across an article in the information systems field which puts forth a straightforward delineation between research design and research method that i am going to use for this column: “research strategy [or research design] is defined as ‘a way of going about one’s research, embodying a particular style and employing different methods.’ research method is defined as ‘a way to systemise [sic] observation, describing ways of collecting evidence and indicating the type of tools and techniques to be used during data collection’” (cavaye, 1996, p. 227). based on this delineation, a case study is first and foremost a research design, and it makes sense that a variety of methods could be used to collect the evidence, depending upon which angle one is taking in approaching the research question. cavaye (1996) reminds us that case study research is multi‐faceted and can be undertaken by using “a positivist or an interpretivist stance, can take a deductive or an inductive approach, can use qualitative and quantitative methods, [and] can investigate one or multiple cases” (p. 227). while cavaye does describe some characteristics of a case study method that are employed in case study research—it does not seek to control variables; it looks at a case in its natural context; focuses on (generally) one site; and it uses qualitative, in addition to quantitative, tools and techniques— even these characteristics seem more akin to design than to method (p. 229). so for the purposes of this ongoing column, in which i will examine various research methods and provide some sources for further exploration, i will adopt the initial definitions from cavaye seen above. i may focus a later column on case study research in more detail. but as it is, i am ready to move on to a specific, hopefully more straightforward, method. next time around, i will look at content analysis. references cavaye, a.l.m. (1996). case study research: a multi‐faceted research approach for is. information systems journal 6(3): 227‐ 242. creswell, j.w. (2007). qualitative inquiry & research design (2nd ed.). london: sage. stake, r.e. (2005). qualitative case studies. in n.k denzin and y.s. lincoln (eds.), the sage handbook of qualitative research (3rd ed.). (pp.443‐446), thousand oaks, ca: sage. yin, r.k. (2003). case study research: desing and methods. thousand oaks, ca: sage. / evidence based library and information practice article   impact of the reading buddies program on reading level and attitude towards reading   hayley dolman youth services librarian grande prairie public library grande prairie, alberta, canada email: hdolman@gppl.ab.ca   serena boyte-hawryluk head of children’s and youth services grande prairie public library grande prairie, alberta, canada email: sboyte@gppl.ab.ca   received: 26 oct. 2012    accepted: 9 feb. 2013      2013 dolman and boyte-hawryluk. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – this research examines the reading buddies program at the grande prairie public library, which took place in july and august of 2011 and 2012.  the reading buddies program pairs lower elementary students with teen volunteers for reading practice over the summer.  the aim of the study was to discover how much impact the program would have on participating children’s reading levels and attitudes towards reading.   methods – during the first and last sessions of the reading buddies program, the participants completed the elementary reading attitudes survey (eras) and the graded word recognition lists from the bader reading and language inventory (6th ed., 2008).  participants were also asked for their grade and sex, and the program coordinator kept track of attendance.   results – there were 37 reading buddies participants who completed both the preand post-tests for the study.  on average, the program had a small positive effect on participants’ reading levels and a small negative effect on their attitudes towards reading.  there was a larger range of changes to the eras scores than to the reading test scores, but most participants’ scores did not change dramatically on either measure.   conclusions – although findings are limited by the small size of the data-set, results indicate that many of the reading buddies participants maintained their reading level over the summer and had a similar attitude towards reading at the end of the program.  on average, reading levels increased slightly and attitudes towards reading were slightly more negative.  many factors could not be taken into account during the study (e.g., the amount of reading done at home).  a study with a control group that did not participate in the program could help to assess whether the program helped to combat summer learning loss.     introduction   the reading buddies program was a new program in 2011 at grande prairie public library.  this program is modelled on the partners in reading program that took place at this library from 1990 to 2008. in 2011, the program was adapted to reflect the current needs of the community.   the new program was intended to pair teen volunteers with lower elementary students for reading practice and fun activities over the summer.  in 2011, grade 1 to 4 students were invited to participate in the program.  in 2012, this was changed to grades 1 to 3, as there was greater demand for the program from families of younger students in 2011.  the large age range also made it difficult to plan developmentally appropriate group activities. the program was marketed towards struggling readers, but children at any reading level could participate in the program.   volunteer recruitment expanded to include college students and some adults when it became clear that we would have far more child participants than teenage volunteers.  in 2011 there were 19 teen and 9 adult volunteers. (as some of the teens volunteered for more than one session, 28 of the 37 pairs had teen volunteers.)  in 2012, there were 29 teen and 5 adult volunteers and of the 44 pairs, 39 had teen volunteers. in 2011, volunteers attended an hour-long training session before the start of the program, in which they learned ways to facilitate the reading process.  in 2012, we extended the training session to one and a half hours to accommodate activities and discussion about strategies for reading with their partners, rather than the simple presentation we had done the year before.   each year, the program ran for seven weeks during the summer.  each session of reading buddies was an hour and a half long.  approximately one hour of this time was spent in one-on-one reading.  the pairs also had the option of using literacy-based games and activities during this time.  the other half hour was spent in group activities, including storytimes, puppet shows, and interactive story-based activities.   reading buddies gives children the opportunity to practice reading throughout the summer, a time when many children fall behind in reading fluency.  in order to be successful, reading buddies should have an impact on the children who participate.  the study was designed to assess the program’s impact on the children’s reading abilities and attitudes towards reading.   literature review   the summer reading gap   there are few who doubt the importance of the ability to read. reading is necessary for success in a world in which text is a major medium for communication.  children who are fluent readers will be more successful in school and as adults, but attaining that level of reading ability requires practice (ross, 2006).  as elementary students, children will naturally learn at different rates and be subject to outside influences such as socio-economic status and family literacy.   research in education has identified what is known as the “summer reading gap.”  this is a phenomenon in which some children maintain or increase their reading level over the summer holiday, whereas other students seem to go backwards in development (roman, carran, & fiore, 2010).  this effect is cumulative, leading to greater and greater discrepancies in ability as children progress through school.  the summer reading gap has also been linked to socio-economic status, as students from higher income families tend to have greater access to libraries and other learning opportunities during the summer months (mcgill-franzen & allington, 2003).   as heyns (1978) initially pointed out, public libraries are in a unique position to address the summer reading gap.  not only are they open during the summer, but libraries have been offering variations of the summer reading program for over a century (roman et al., 2010).  today, almost all libraries offer free, structured reading programs for children of all ages.  this programming serves to motivate children to continue reading while they are out of school, and thus may serve to prevent or limit summer learning loss.   although there is a field of research addressing the summer reading gap from the education perspective, relatively little literature directly examines how summer reading programs in libraries impact student achievement.  heyns’s (1978) study found that children who participated in summer reading programs gained more vocabulary than children who did not, regardless of socioeconomic status, gender, or number of books read.  roman et al. (2010) recently conducted a large-scale longitudinal study, comparing students who participated in summer reading programs at libraries with students who did not.  overall, this study showed that children who participated in voluntary summer reading programs increased their reading levels more than children who did not.  in the research that does exist, it seems that voluntary participation in a reading program has more impact than forced reading, whether at home, summer school, or the library.  it appears that the greatest factor in summer reading achievement may be access to and regular use of library materials and programs.   reading partner programs   there have been a number of studies on tutoring programs for reading skills.  many of these programs took place in schools and run throughout the school year (block & dellamura, 2001; burns, senesac, & silberglitt, 2008; fitzgerald, 2001; gattis et al., 2001; lague & wilson, 2010; marious, 2000; paterson & elliott, 2006; theurer & schmidt, 2008; vadasy, jenkins, antil, wayne, & o’connor, 1997).  several programs specifically targeted students at risk of reading failure (burns et al., 2008; fitzgerald, 2001; gattis et al., 2001; lague & wilson, 2010; paterson & elliott, 2006; vadasy et al., 1997).   all of these programs showed an improvement in the students’ reading abilities.  burns et al. (2008) studied the long-term effects of a reading program, and found that two years after the help one student to succeed (hosts) program, hosts students had higher fluency, comprehension, and reading progress scores than non-hosts students.    the length of the program is an important factor.  fitzgerald’s (2001) study of a tutoring program compared a group of students who received tutoring for a full term and students who were tutored for less than the full term.  the students who were tutored for the full term showed higher gains in reading ability.  fitzgerald also noted that students showed greater growth in the second half of the program, and that different skills improved at different points in the program: during the first half, students showed more improvement in phonological awareness, whereas in the second half there was greater improvement in reading words.   the tutors also impacted the effectiveness of the programs.  the age of tutors does not appear to be an important factor: programs with volunteers who were peers (lague & wilson, 2010), older students (block & dellamura, 2001; marious, 2000; paterson & elliot, 2006; theurer & schmidt, 2008), college students (fitzgerald, 2001), adults (jalongo, 2005), or a mix of community volunteers (gattis et al., 2010; vadasy et al., 1997), all showed improvements in students’ reading.  in all of these studies, tutors received some form of training.  vadasy et al. (1997) studied a program with very structured lesson plans and found that the “children whose tutors implemented the lessons as designed demonstrated significantly higher reading and spelling achievement” (lesson content section, para. 2). though not studied in depth, theurer and schmidt (2008) noted that while some of the “fifth-grade buddies were naturals and interacted comfortably with the first graders, others seemed uncertain and tentative, not quite knowing what was expected of them” (p. 261). they integrated training on choosing books, reading strategies, and interpersonal skills into the program.   because these studies look at programs that are based in schools and run throughout the school year, the programs are longer than our summer reading buddies program, which runs for seven weeks.  as shown in fitzgerald’s (2001) study, the length of the program can impact the students’ gains in reading.   the structure of the programs studied varied, and it is difficult to compare the effects of each program.  vadasy et al.’s (1997) conclusions support a more structured program.  our reading buddies program was loosely structured, with the majority of the time spent reading one on one with the volunteers, so it is important to have a closer look at the effects of a loosely structured program on students’ reading abilities.   reading abilities and attitudes   reading buddies aims to improve children’s reading abilities, but also to instill a positive attitude about reading.  the two factors are intricately related.  it seems that students who have a negative attitude about reading are less likely to read voluntarily and will read less overall than their reading-positive companions (sainsbury & schagen, 2004).  over time, this leads to larger and larger gaps in ability between students.  research has indicated that reading achievement and attitudes about reading are related among elementary students (diamond & onwuegbuzie, 2001).  indeed, mckenna and kear (1990) developed the elementary reading attitudes survey (eras) as another way (besides reading tests) for teachers to assess their students.    logan and johnston (2009) studied over 200 students in order to compare reading abilities and attitudes between boys and girls.  they found that girls had more positive attitudes towards reading overall, and that this was correlated with their reading ability.  interestingly, the relationship between reading attitude and ability was found to be weaker in boys than in girls.   the dominican study (roman et al., 2010) revealed that most librarians perceived that their programs had a positive effect on students’ reading levels and attitudes about reading.  block and dellamura (2001) also observed that children placed a higher value on reading at the end of their tutoring program.  however, the students’ attitudes about reading were never directly tested in either program.    aims   the goal of the study was to test two hypotheses:   hypothesis 1: children enrolled in the reading buddies program will have better reading skills at the end of the program than at the start of the program. hypothesis 2: children enrolled in the reading buddies program will have a more positive attitude towards reading at the end of the program than at the start of the program.   methods   reading test   we used the graded word recognition lists from the bader reading and language inventory (6th ed., 2008) to test the participants’ reading skills.  the graded word recognition lists “can serve as a quick check of the student’s word recognition and word analysis abilities” (bader & pearce, 2008, p. 4).  they do not measure other reading skills such as comprehension.   the test consists of several lists of progressively more difficult words.  this test was chosen because it covered a wide range of reading levels (preschool to high school), had been updated recently, and could be easily administered within the limited time we had available.  while the test is american, the words chosen did not reflect any regional spelling variations.  differences in the american and canadian school systems may have made the grade level results inaccurate; however, we were interested only in the change in reading level, not the grade levels themselves. the test was administered one on one during the first and last sessions of the reading buddies program.   elementary reading attitudes survey   we used a modified version of the eras, or elementary reading attitudes survey (mckenna & kear, 1990), to evaluate how participants’ attitudes about reading changed over the duration of the reading buddies program.  this survey was originally developed as a way for teachers to determine how their students felt about reading.  it has also been used for research studies about reading attitudes (black, 2006; martinez, aricak, & jewell, 2008; worrel, roth, & gabelko, 2002), mostly in school settings.    the eras uses images of the popular comic book character, garfield, to elicit participants’ emotional responses about reading.  questions ask “how do you feel …?” about a reading-related activity and participants circle one of four images of garfield that corresponds with their feeling.   the eras was extensively tested during its development to determine its validity and reliability.  after the format and items had been decided upon, the researchers administered the test to over 18,000 firstto sixth-grade students across the united states. calculation of cronbach’s alpha revealed high internal consistency of items within each sub-scale.  to determine the validity of the survey, participants were asked directly about their reading habits and other activities.  high scores on the survey, indicating a very positive attitude towards reading, were correlated with literary activities such as good access to school and public libraries.  low scores on the survey were correlated with non-literary activities such as large amounts of television-watching.   the survey contains two sub-scales, one measuring recreational reading and one measuring academic reading.  for the purposes of this research, we only used the first sub-scale. we chose to eliminate the second sub-scale because of frequent references to the school context, which are not suited to our purpose.   in each year of the study, the eras survey was administered to the groups of reading buddies participants during the first and last sessions of the program.  the 10 questions of the first sub-scale were read aloud to the participants, who completed their own paper copy of the survey.   demographics and program participation   as part of the eras, participants were also asked for their grade and sex.  during the program, attendance records were kept, so there was a record of how many sessions each child attended.   results   in 2011, 19 out of the 37 children participating in the program completed both the preand post-tests.  in 2012, there were 18 reading buddies participants who took part in the study (although only 17 completed both the preand post-test of the eras), for a total of 37 study participants over two years.   nineteen of the study participants were boys and 18 were girls.  the breakdown of grades they had just completed was as follows:     figure 1 grades completed by reading buddies participants     during registration, we asked that parents register their children in reading buddies only if they expected to be able to attend at least five of the seven sessions.  figure 2 shows the number of participants grouped by the number of sessions they attended.     figure 2 number of sessions attended by reading buddies participants     reading test   participants were given a score on the reading test between -1 (preschool) and 9 (high school).  the score is intended to reflect a normal reading level for a student’s grade (e.g., a score of 2 is a second-grade reading level).  half scores could also be given (e.g., 1.5).  we subtracted the participants’ pre-test reading scores from their post-test reading scores to determine the change in reading level.   on average, there was a small increase in the participants’ reading levels over the course of the program.  the average change in reading test scores was 0.08.  the range for the change in reading test scores was from -1.5 to 2.  ten participants showed an increase in reading score, 8 showed a decrease, and 19 showed no change.  as figure 3 shows, few children’s reading levels changed by more than 0.5 in either direction.     figure 3 changes in reading test scores between the first and last sessions of reading buddies     the number of sessions may have had an impact on the changes in reading levels, with a correlation coefficient of 0.13.  the average change in reading score increased with the number of sessions attended, up to six sessions. see table 1.   few children attended fewer than five sessions (over half the study participants came to six sessions), so results here are also not conclusive.     table 1 average change in reading level by number of sessions attended number of sessions attended average change in reading level 3 -0.5 4 -0.17 5 0.06 6 0.18 7 0     grade level also appeared to make a difference to changes in reading levels. between kindergarten and grade 2, the change in reading level became more positive as the grade level increased.  however, the correlation coefficient was not significant at -0.01.  while it appears that the program’s positive effects peak around grade 2, it is important to keep in mind that the majority of the study participants were in first and second grade (only two third-grade students participated in the study).   the program also had a bigger impact on girls’ reading scores than on boys’, though overall it did have a small positive impact on both.  the average change in score for girls was 0.14 and for boys was 0.03.   reading attitudes survey   all participants were given a reading attitudes score between 10 and 40, with higher scores indicating a more positive attitude towards reading.  contrary to our expectations, the average change in eras scores between the first and last sessions was a decrease of 1.17 points.  the change in eras scores ranged from -14 to 15.  sixteen participants showed an increase in their eras score, 19 showed a decrease, and 1 showed no change.  while there was a wide range in changes to the eras scores, large changes in eras scores were uncommon: the majority or participants remained within 5 points of their pre-test score. see figure 4.     figure 4 changes in eras scores between the first and last session of reading buddies     there was no correlation between the number of sessions attended and changes in eras scores. there appeared to be a relationship between the grade level of the child and the changes in their attitude toward reading in the 2011 group — the positive effects of the program increased up until the third grade — however, this was not so evident once the 2012 data was added.  the correlation coefficient for last completed grade and change in eras score was 0.29.  as there were only two third-grade students and three kindergarten students who participated in the study, it is difficult to draw any conclusions about these results.   the program had slightly less impact on girls’ eras scores than on boys’, although both sexes show a slight decrease in attitude towards reading over the summer.  on average, the female participants’ scores decreased by 0.35 points and the male participants’ scores decreased by 1.89 points. see figure 5.     figure 5 sex differences between eras scores     in general, the boys also had lower raw scores on the eras test than the girls.  in the pre-test, girls scored an average of 35.88 versus boys’ average scores of 30.42.  the post-test revealed similar results, with girls scoring 35.53 and boys scoring 28.53.   discussion   the small number of participants in this study makes drawing any strong conclusions difficult.  our results show some interesting trends with regard to what effect the reading buddies program has had on its participants, but it is difficult to declare whether the program was successful or not.  on average, the participants showed a slight increase in their reading level over the summer and their attitudes about reading became slightly more negative; however, the changes were very small.  many participants maintained the same reading level and a similar attitude towards reading.   although the average change in score for the reading test was slightly positive, there were some participants whose reading levels decreased between the preand post-test.  this may be a symptom of the overall learning loss that occurs during the summer.  since we have no control group for comparison, it is difficult to evaluate whether our program made a significant difference in combating summer learning loss.    the research demonstrates that the more sessions the children attended, they more likely it was that their reading abilities would increase.  this should be emphasized to parents, so that fewer sessions are missed during the summer.  it is also possible that a longer program would have a more positive impact (e.g., a program run during the school year).  we suspect that the short duration of the program will prohibit it from ever causing large increases in reading ability; however, the number of sessions seems to be sufficient to help maintain reading levels.   for several of the participants, the program had a negative impact on their attitude towards reading.  though it is impossible to say why this was the case, the child’s attitude towards participating in the program may have been a factor.  participants may have attended the program at the behest of parents or teachers, rather than of their own volition.  selection bias may also have been a factor, as the program was marketed towards struggling readers, who may have a more negative attitude towards reading than the general population.   however scant the data may be, this information may point in the direction of potential changes to the program.  on average, the boys entering the program had less positive attitudes towards reading than the girls, and also saw less positive effects from the program on both measures.  this is consistent with research indicating that boys generally fall behind girls in reading level as they progress through school (taylor, 2005).  better results for boys might be achieved if more attention were paid to their particular needs and interests.   there were many factors that could not be measured in this study.  the reading buddies sessions were loosely structured, and the pairs had choices with regards to how much time they would spend reading, discussing the books, and playing literacy-based games.  the impact of supplementary activities versus time spent in one-on-one reading during the program was not measured. the task of keeping a record of the time spent on various activities may have distracted volunteers from their most important task: engaging with their younger partners. additionally, some activities (e.g., reading and discussion) are so intertwined that measurements of time spent on them were unlikely to be accurate.   the volunteers’ skill as reading partners was also not taken into account. volunteers all received the same training; however, many other factors affected their performance, such as personality, previous experience in similar programs, comfort levels with children, willingness to ask for help, and improvement over the course of the program.  quantifying the volunteers’ skill as reading partners was impractical given the number of factors involved.   there were also factors outside of the program that we were unable to measure.  as discussed, voluntary reading is more effective than forced reading at reducing the summer reading gap (roman et al., 2010).  it stands to reason that participants who were motivated to read on their own may have had more success in the program than those who did not read voluntarily.  unfortunately, we had no way of accurately measuring how much voluntary reading participants were doing outside of the program.   during the program, it was casually observed that some of the participants’ parents were more enthusiastic about reading than others.  this behaviour included making an effort to attend every session, encouraging children to check out books, bringing the family to other reading programs at the library, and reading books themselves while waiting for their children.  it would be very interesting to see if this parental influence was related to improvements in reading level and attitudes, however we had no way of determining this during the first two years of the program.  for future years, we hope to provide parents with information or training at the start of the program to emphasize the importance of modelling reading behaviour within the family.   conclusions   this study was undertaken to determine how our library’s summer reading mentor program would influence the participants’ reading abilities and attitudes about reading.   our first hypothesis about the reading buddies program was supported: on average, the effects of the program on reading skills were positive.  however, due to the small number of participants, further study will be needed to confirm these results.  it is also clear that, while reading levels may improve slightly during reading buddies, maintaining children’s reading levels is a more realistic goal for this program.    the second hypothesis, which postulated that the program would lead to an increase in positive attitudes about reading, was not supported by the data gathered.  some participants did demonstrate a higher score on the post-test, as compared to the pre-test, but on average the study showed a small negative impact on attitude towards reading.  due to the small number of participants, further study will be needed to confirm these results.   it appears that reading buddies helps to combat summer learning loss, both reading abilities and attitudes; however a study with a control group would provide stronger evidence for this finding.      references   bader, l. a., & pearce, d. l. (2008). bader reading and language inventory. 6th ed. pearson.   black, a.-m. l. (2006). attitudes to reading: an investigation across the primary years. virginia, qld: australian catholic university.   block, c. c., & dellamura, r. j. (2000/2001). better book buddies. the reading teacher, 54(4), 364-70.   burns, m. k., senesac, b. j., & silberglitt, b. (2008). longitudinal effect of a volunteer tutoring program on reading skills of students identified as at-risk for reading failure: a two-year follow-up study. literacy research and instruction, 47(1), 27-37. doi:10.1080/19388070701750171   diamond, p. j., & onwuegbuzie, a. j. (2001).  factors associated with reading achievement and attitudes among elementary-aged students. research in the schools, 8(1), 1-11.   fitzgerald, j. (2001). can minimally trained college student volunteers help young at-risk children to read better. reading research quarterly, 36(1), 28-46. doi:10.1598/rrq.36.1.2   gattis, m. n., morrow-howell, n., mccrary, s., lee, m., jonson-reid, m., mccoy, h., tamar, k., molina, a., & invernizzi, m. (2010). examining the effects of new york experience corps® program on young readers. literacy research and instruction, 49(4), 299-314. doi:10.1598/rrq.36.1.2   heyns, b. (1978). summer learning and the effects of schooling. new york: academic press.   jalongo, m. r. (2005). tutoring young children’s reading or, how i spent my summer vacation. early childhood education journal, 33(3), 121-123.   lague, k. m., & wilson, k. (2010). using peer tutors to improve reading comprehension. kappa delta pi record, 46(4), 182-186.   logan, s., & johnston, r. (2009).  gender differences in reading abilities and attitudes: examining where these differences lie.  journal of research in reading, 32(2), 199-214.doi:10.1111/j.1467-9817.2008.01389.x   marious, s. e., jr. (2000). mix and match: the effects of cross-age tutoring on literacy. reading improvement, 37(3), 126-130.   martinez, r. s., aricak, o. t., & jewell, j. (2008). influence of reading attitude on reading achievement: a test of the temporal-interaction model. psychology in the schools, 45(10), 1010-1022. doi:10.1002/pits.20348   mcgill-franzen, a., & allington, r. (2003). bridging the summer reading gap. instructor, 112(8), 17-20.   mckenna, m. c., & kear, d. j. (1990). measuring attitude toward reading: a new tool for teachers. the reading teacher, 43(8), 626-639.   paterson, p. o., & elliott, l. n. (2006). struggling reader to struggling reader: high school students’ responses to a cross-age tutoring program. journal of adolescent & adult literacy, 49(5), 378-389.  doi:10.1598/jaal.49.5.2   roman, s., carran, d. t., & fiore, c. d. (2010). the dominican study: public library summer reading programs close the reading gap. river forest, il: dominican university. retrieved 22 feb. 2013 from http://www.dom.edu/gslis/downloads/dom_imls_book_2010_final_web.pdf   ross, c.s. (2006). the company of readers. in c.s. ross, l. mckechnie, & p.m. rothbauer, reading matters: what the research reveals about reading, libraries, and community. (pp. 1-62). westport, connecticut: libraries unlimited.   sainsbury, m., & schagen, i. (2004).  attitudes to reading at ages nine and eleven.  journal of research in reading, 27(4), 373-386. doi:10.1111/j.1467-9817.2004.00240.x   taylor, d. l. (2005). “not just boring stories”: reconsidering the gender gap for boys. journal of adolescent & adult literacy, 48(4), 290-298. doi:10.1598/jaal.48.4.2   theurer, j. l., & schmidt, k. b. (2008). coaching reading buddies for success. the reading teacher, 62(3), 261-64. doi:10.1598/rt.62.3.8   vadasy, p. f., jenkins, j. r., antil, l. r., wayne, s. k., & o'connor, r. e. (1997). the effectiveness of one-to-one tutoring by community tutors for at-risk beginning readers. learning disability quarterly, 20(2), 126-39. doi:10.2307/1511219   evidence based library and information practice 2012, 7.1 108 evidence based library and information practice evidence summary teenagers’ public library needs are difficult to determine a review of: howard, v. (2011). what do young teens think about the public library? the library quarterly, 81(3), 321-344. url: http://www.jstor.org/stable/10.1086/660134 reviewed by: jason martin associate librarian university of central florida libraries orlando, florida, united states of america email: jason.martin@ucf.edu received: 3 oct. 2011 accepted: 14 dec. 2011 2012 martin. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to discover the attitudes of twelve to fifteen year-olds toward the public library. design – mixed methodology consisting of a survey and focus groups. setting – an eastern canadian regional municipality. subjects – twelve to fifteen year-old middle school students. methods – using a disproportionate stratified sample and multistage clustering, the author mailed 900 surveys to middle school students; 249 surveys were completed and usable. those students who completed the survey and who also indicated they would be willing to participate in a focus group were randomly selected to participate in nine focus groups with between 7 to 12 students in each group. main results – discrepancies exist between the teens’ level of satisfaction with the library indicated on the survey (high) and expressed in the focus groups (low). teens seldom use the public library due to: their non-existent relationship with library staff, although teens who were “active readers” used the library more; lack of appealing programs and program promotion; no teen-focused website; poor teen facilities within the library; and an overall failure of the public libraries to include teenagers. conclusion – public libraries need to be more responsive to teen needs to attract teens to use http://www.jstor.org/stable/10.1086/660134� mailto:jason.martin@ucf.edu� http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/� evidence based library and information practice 2012, 7.1 109 the library. to uncover these needs, libraries should use mixed methods of discovery. commentary teenagers inhabit a world of their own; they are too old to be a child and too young to be an adult. being caught between these two periods of life can be stressful and frustrating for teenagers. this “unbelonging” extends to public libraries, which focus almost exclusively either on pre-teen children or on adult users. public libraries must do a better job of meeting the needs of teen users to encourage lifelong library use. however, little research has been done examining the library needs of teenagers. vivian howard attempts to add to the small quantity of research on teen user needs with her mixed-methods study of middle school students aged 12 to 15 in an eastern canadian municipality. the survey results indicate teens use the library very little, but that they are highly satisfied with the level of service they receive. the results from the focus groups, however, show a much lower level of satisfaction with the public library. howard uses the findings of several previous studies of teens and libraries to frame her study and to determine what areas of library services should be targeted. the focus groups were used to enhance the survey findings and appeared to focus solely on the previously identified areas of library service. qualitative research should be a time for exploration that may very well lead to insights and discoveries outside the original frame. a less rigid approach to the focus groups may have yielded service problems or praise which lay outside the previous studies’ findings. howard does not indicate whether she used spss or another statistical software package to analyze the survey results, or whether the focus group results were analyzed by hand or through the use of a program like nvivo. nor does she provide the full survey, focus group question guide, or a code sheet used for analysis of the focus group results. the study is well-designed and well-written, and adds to the scant literature on teen public library needs. however, howard only presents her findings and offers no suggestions for ways in which public libraries may improve their practices. her assertion libraries must use mixed-methods to uncover teen needs seems flawed considering the wide differences between her survey results and focus group findings. perhaps adding a third method, such as observation, would help to triangulate the results and thus provide more concrete discoveries. / evidence based library and information practice evidence based library and information practice 2012, 7.3 90 evidence based library and information practice evidence summary research quality and newsworthiness of published articles are partial predictors of journal impact factors a review of: lokker, c., haynes, r. b., chu, r., mckibbon, k. a., wilczynski, n. l., & walter, s. d. (2012). how well are journal and clinical article characteristics associated with the journal impact factor? a retrospective cohort study. journal of the medical library association, 100(1), 28-33. doi:10.3163/1536-5050.100.1.006 reviewed by: jason martin head of public services dupont-ball library, stetson university deland, florida, united states of america email: jmartin2@stetson.edu received: 3 may 2012 accepted: 31 july 2012 2012 martin. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – determine what characteristics of a journal’s published articles can be used to predict the journal impact factor (jif). design – a retrospective cohort study. setting – the researchers are located at mcmaster university, hamilton, ontario, canada. subjects – the sample consisted of 1,267 clinical research articles from 103 evidence based and clinical journals which were published in 2005 and indexed in the mcmaster university premium literature service (plus) database and those same journals’ jif from 2007. method – the articles were divided 60:40 into a derivation set (760 articles and 99 journals) and a validation set (507 articles and 88 journals). ten variables which could influence jif were developed and a multiple linear regression was run on the derivation set and then applied to the validation set. main results – the four variables found to be significant were the number of databases which indexed the journal, the number of authors, the quality of research, and the “newsworthiness” of the journal’s published articles. mailto:jmartin2@stetson.edu evidence based library and information practice 2012, 7.3 91 conclusion – the quality of research and newsworthiness at time of publication of a journal’s articles can predict the journal impact factor with 60% accuracy. commentary journal impact factors (jifs) are calculated over a two-year period by dividing the total number of the journal’s citations by the total number of “citable” articles published in that journal. such a seemingly simple idea has led to a great deal of discussion and controversy. a journal’s impact factor is what helps establish it as a “core journal,” a label which carries a great deal of prestige. critics of jifs are quick to argue editorial policies and other influences can manipulate impact factors. the authors of this study set about to determine what, if any, facets of a journal’s articles are associated with jifs. their sample consisted of articles indexed in the mcmaster university plus database. they developed 10 variables they thought would predict jif with 4 of those variables proving significant. while the authors mainly describe how the research quality and newsworthiness of a journal’s published studies can predict jif thereby making an impact factor an indicator of worth and value, they do admit the highest predictor of jif was the number of authors and the amount of databases which indexed the journal. the methodology of the article is well-defined and strictly followed. however, the authors admit several major limitations of the study exist. the first limitation is the plus database uses an extensive screening system whereby only evidence based articles receiving high scores from a trained research reviewer and a group of physicians representing various fields are indexed in the database. (the definitions of research quality and newsworthiness used by the authors are the same ones the raters use for plus.) this creates a population of articles which can be described as the best of the best. the second limitation is the study’s use of such a small sample: 103 journals. in addition to being small, the sample did not include online journals which studies have shown are typically more often cited, although it is unclear what the researchers meant by “online journals.” the authors encourage future research on the prediction of jifs using a random sample from a larger population of both lowand high-quality studies. the authors state this study can lead to higher jifs for journals if editors were to include practicing clinicians in the peer review process and make sure their journals are indexed in an abundance of databases. (this is somewhat out of the hands of the journal editors since the database editors make the final decisions on what journals to index.) the authors also posit jifs can help direct clinicians to higher quality studies. all-in-all, this study, while finely executed, does little to clear the murky waters which surround the use of impact factors. evidence based library and information practice 2012, 7.1 22 evidence based library and information practice article analyzing public library service interactions to improve public library customer service and technology systems holly arnason digital literacy librarian, digital literacy initiatives edmonton public library edmonton, alberta, canada email: harnason@epl.ca louise reimer director, library services edmonton public library edmonton, alberta, canada email: lreimer@epl.ca received: 2 sept. 2011 accepted: 13 jan. 2012 2012 arnason and reimer. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/bync-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to explore the types and nature of assistance library customers are asking library staff for in a large canadian urban public library system. methods – a qualitative study employing transaction logging combined with embedded observation occurred for three-day sample periods at a selection of nine branches over the course of eight months. staff recorded questions and interactions at service desks (in person, by phone, and electronically), as well as questions received during scheduled and non-scheduled provision of mobile reference service. in addition to recording interaction details and interaction medium, staff members were also asked to indicate briefly the process or resources used to resolve the interaction. survey data were entered and coded through thematic analysis. results – the survey collected 6,099 interactions between staff and library customers. of those 6,099 interactions, 1,920 (31.48%) were coded as pertaining to technology help. further analysis revealed significant library customer need for help with internet workstations and printing. mailto:harnason@epl.ca� mailto:lreimer@epl.ca� evidence based library and information practice 2012, 7.1 23 conclusions – technology help is a core customer need for edmonton public library, with requests varying in complexity and sometimes resolved with instruction. the library’s internet workstations and printing system presented critical usability challenges that drove technology help requests. introduction public libraries across north america face a great challenge and opportunity in redesigning public services to align with diverse and evolving customer needs – increasingly utilizing digital information sources and personal and social technologies. in may 2010, edmonton public library, an urban public library located in western canada, launched a study into the nature and types of questions received at service desks (in person, by phone, electronically) and while providing mobile reference service (“roving”). the principal and sole researcher for this study was a recently graduated intern librarian hired by edmonton public library to explore future service directions for reference and readers services. the internship project was supervised by one of edmonton public library’s directors of public services. edmonton public library is a seventeen branch library system located in alberta, canada and serving a population of 728,349 (city of edmonton, 2009). the purpose of this study was to discover and explore detailed information about the nature and types of questions customers were asking, with the goal of informing the redesign of public services to broadly improve customer service. this article describes methods, results, and conclusions of the study, with focus set tightly on the technology help data subset of the larger study. literature review public libraries and technology the literature speaks widely to the potential impact, challenges, and opportunity for technology use and technology learning in public libraries for library customers and library staff. scholars such as aabø (2005) and fox, horne, king, seely, and walsh (2008) broadly address the potential for public libraries to use information communication technologies to impact community and culture. based on the results of a survey of finnish library users and non-users about the value of public libraries in digital society, aabø locates the public library’s points of technological access as possible sites for customers facing barriers, in terms of both physical access and a range of fluencies, to participate democratically and inclusively in the maintenance and formation of social and cultural identities. fox et al. speak to the potential of emerging technologies to create rich opportunities for instruction that strengthen and bridge communities (including customers and for the library profession itself), while also providing a context where libraries can remain strategically innovative among customers and community stakeholders. lai (2011) used document analysis, observation, and focus group interviews to identify various methods and strategies in use by canadian public libraries to improve the information literacy skills of both library customers and staff. importantly, lai identified that in a large urban public library an effective training program incorporated four structural characteristics: advanced information and communications technology (ict) structure, provision of formal training, improvement of staff information literacy (il) and instructional skills, and partnerships with local organizations. a key finding reported was that a significant barrier staff faced as providers of formal and informal information literacy instruction was a lack of “appropriate guidance in understanding the theories of il and adult learning while teaching library patrons” (p. 87). bertot (2009) used a case study of 25 public libraries in the united states (combining interview, observation, and inventory) to explore the staffing, budgetary, and infrastructural requirements for supporting evidence based library and information practice 2012, 7.1 24 public access technologies, and to identify factors underlying successful provision of public access technologies. technologies that bertot looked at included public workstations; wireless access; online resources, such as databases and downloadable content; as well as the library’s more structural services – integrated library systems and digital reference services. bertot found that by providing a combination of public access through hardware and software, public libraries became points of access supporting use of digital technology by the public. as nagy (2011) illustrates in “next-generation service in the library,” this use varies in focus and intensity, and is influenced by expectations and functions present in external web services. supporting this range of use is crucially challenging when trying to accommodate the customer use of rapidly evolving personal devices within the library: usb sticks, mp3 players, mobile phones, and so on. bertot found that public libraries in his case study experienced budgetary and planning pressures when addressing needs such the replacement and addition of equipment or systems, the lack of integration between existing and new systems, or the need for upgrading buildings and networks to support increasing public demand for power and bandwidth. bertot noted that public libraries experienced success when library staff possessed the ability to lead by forming partnerships to extend access within the community and through the community’s support. it was also beneficial when library staff could articulate a strong rationale for why the library was engaged in provision of these services and why it should be supported financially and politically. in looking at the provision of public access technologies from a public services perspective, bertot found that a major factor in success was to have staff who could understand the spectrum of technologies involved and think creatively about problem solving across library and technology services. of particular interest within the literature of librarianship and educational technology are those studies that attempt to analyze customer experience with library services and spaces, whether through survey, interview, interaction logging, or other methods of examining usage and experience. from the academic library context one of the most interesting studies that encompasses, but also looks beyond technology services, is documented in the book studying students: the undergraduate research project at the university of rochester. focusing on undergraduate users, this research team applied a wide range of ethnographic and anthropological techniques including interviews, a pilot service adjustment, diary mapping, photo surveys, and charrette-style workshops. in the process of conducting a diary mapping exercise with undergraduate students, the researchers found that students surveyed checked computers for communication and information several times throughout the day across campus and where computer access was provided. given that the study was carried out in 2005, students in the survey opted not to bring bulky laptops along, but preferred to utilize laptops when several hours of work in one location was anticipated (gibbons & foster, 2007b). a linked finding appeared in the research project’s charrette-style workshops conducted with a small group of students, where students were asked to furnish an ideal library space. students in this exercise frequently included workspace that would be suitable for extended work on a mobile device, with many power outlets available (gibbons & foster, 2007a). students in this exercise also revealed interesting perceptions in the staff support element of their designs: students rarely make distinctions between the types of staff needed in the library. instead, they include a generic staff person who is expected to provide reference assistance, check out materials, answer it questions, and brew a great latte. (gibbons & foster, 2007, p. 25) another interesting finding about student expectations in the use of library technology pertained to perceptions uncovered in the evidence based library and information practice 2012, 7.1 25 diary mapping exercise about library computers: it also was clear that students do not understand that the computer lab, which is housed in the physical library building, is not part of the library. it is obvious to library and computing staff that the two entities are different, but not to students. we now understand a little better why students are confused, surprised, and sometimes disappointed when the library computers do not have the same software and functionality as the workstations in the computer center. because of this project, providing access to an identical desktop and suite of services became a top priority for the library . . . (clark, 2007, p. 53) focusing on the library kiosk in the public library context, a study of usage and experience at thunder bay public library (tbpl) uncovered similar insights about customer confusion and frustration when trying to access technology services within the library (aegard, 2010). the 2009 study conducted by thunder bay public library aimed at understanding how library customers were using library kiosks and at improving that experience (aegard, 2010). during a ten-day period, aegard’s team utilized a combination of kiosk usage statistics that revealed the most frequently accessed resources, staff survey to collect perceptions and experiences, and customer survey to determine customer satisfaction with and perceptions of the library’s kiosks. the researchers found that the library’s opac was overwhelmingly accessed (64.65%) on the kiosk, and the ‘book a computer’ service was the next most frequently accessed at 9.81%. further, attempts to access unauthorized web sites (children’s and adult together) comprised 8.73%. the findings led the team to remove some poorly utilized resources (such as a mouse tutorial), while in general maintaining the basic functionality for accessing opac and service-related functions. while mostly positive, tbpl’s customer survey uncovered some customer frustration with the lack of functionality of the kiosks, for example, the inability to print and the lack of access to web even when linked through a catalogue record. the researchers similarly noted some tension in maintaining that functionality and excluding open internet access: “it has been challenging to communicate the function of the kiosks to customers. after all, people walk in, see a computer, and naturally assume they can use it to get on the web” (aegard, 2010, p. 18). library literature speaks widely to the changing nature of reference services, but some studies of note look specifically at reference through using the method of transaction logging and analysis. though small in numbers, these studies reveal interesting insights about how library customers and library staff interact around the delivery of technology services. cavanagh (2006) conducted an ethnographic study of four branches within an urban public library system to explore the library and reference activity as a knowledge-sharing and knowledge construction ecology. though this study was not explicitly directed at examining technology use, in the course of observing reference transactions cavanagh reported that the computer often “functions as an explicit knowledge container and as a prop in the interaction narratives” (p. 14) when customers ask questions such as “what does the computer say about this book?” cavanagh located technology help in the sphere of relationship building activities shared by staff and library customers, when she related a customer question about access to digital photos on library computers that ended with personal talk between a customer and a staff member. cavanagh’s study provides a unique perspective that considers the user experience of library information services as a knowledge sharing and knowledge construction site, rather than focusing solely on staffing or resource design. wong (2009) conducted a study of technology help transactions at an information commons (ic) at hong kong university with the stated purpose to inform training design for ic staff. evidence based library and information practice 2012, 7.1 26 by extracting a sample of questions from a larger dataset spanning four and a half years, wong categorized technology questions and then performed further analysis probing at transaction complexity. importantly, wong categorizes questions based on what services library users needed help with, rather than the resources employed by library staff. of 1,411 questions in the technology subset, wong found that printing and computer assistance questions dominated, followed by questions regarding scanning and copying functions. based on a system of tiering questions by complexity of staff skills required, wong found that 82.6% of the technology questions required help that was manageable by a person without a formal technical background. the remaining 17.6% of technology questions required help that went beyond general computer literacy and required a more formal background in information systems to address. at mcneese state university, finnell and fontaine (2010) used a reference transaction study to guide the development of subject study guides, instructional outreach efforts, and collection development. based on 6 months of data collection and preliminary analysis through subject classification coding, the researchers created 11 study guides, purchased new materials, and created 19 new bibliographic instruction courses. the researchers also refocused program efforts to liaise with faculty in key subject areas based on patterns of help requested from students. two transaction logging studies attempted to apply analysis of desk transactions to the challenge of staffing services with at an appropriate level for skill and cost effectiveness. meserve, belanger, bowlby, and rosenblum (2007) applied the warner model for the categorization of reference transaction into tiers of complexity to transactions logged in fall 2005 and fall 2006 with the purpose of determining appropriate staffing (professional vs. paraprofessional) for services (in person, telephone, virtual) of a merged academic and public library. the researchers were able to adjust staffing based on analysis of frequency of level 1 (non-resource) and level 2 (skillbased) questions in certain services in comparison with the frequency of level 3 (strategy-based) and level 4 (consultation) questions. the researchers also concluded that question classification and frequency had informed training design, as training was an important factor in maintaining the quality of service in a tiered environment. ryan (2008) applied content analysis to repurposed reference transaction data at stetson university library to determine the cost effectiveness of staffing a reference desk with library professionals. during 4 collection periods, 6,595 questions were captured and considered in the study. ryan found that 862 (12.4%) of the queries were informationoriented technology questions and estimated the dollar cost in staffing for each kind of question. ryan found that 89% of the total questions were answerable by non-librarians, and suggested that staffing adjustments to desk service or alternative service delivery models should be explored along with further study of student needs through varying community consultation techniques. the unresolved complexity in ryan’s study is based in how the questions were mapped from type to staffing level. the present study touched on areas examined by the literature: composition of transactions by type, the public library in relation to digital literacy and inclusion, user experience with library-provided technology, and service/staffing design implications. it approached the subject of technology interactions and service design implications by focusing on customer-staff interactions that center around technology within the public library’s space. aims this study attempted to address the following questions: 1. what is the nature and what are the types of help being requested by library customers? evidence based library and information practice 2012, 7.1 27 2. what changes to service design are identifiable and actionable based on these requests? methods a question logging form was used to capture the details of interactions between library staff members and customers. the principal researcher adapted the question logging form from one originally designed by edmonton public library’s acting manager of assessment and research for an internal question pattern study conducted within edmonton public library to determine staffing levels for a specialized, permanent, “storefront” service point. the form was restructured to capture the following details for each staff-customer interaction: • date of question (day, month) • question details . . . • answered with . . . (resource or process) • question location (on desk, roving, on phone, by chat) the form was initially tested for four hours at one service point on a high traffic desk, and minor adjustments were made based on cell space requirements for fields and in consultation with edmonton public library’s manager of assessment and research. the final version of the form retained all four data elements listed above and is included in appendix b. twelve service points (9 branches in total, with 1 branch containing 4 service points) were selected from the library system by the principle researcher and the project supervisor in an attempt to get a varied sample of branch sizes and locations within the city, as well as variation in the types of communities served. variety in size and location were the main factors by which the branches selected, but it was also decided that the study would include three service points that were heading into new building projects, the main branch (potentially heading to renovations), and one branch hypothesized to be impacted by the opening of a new service point in 2013. the researcher and supervisor decided on a threeday sample range for collecting interactions based on the volume of data collected during the four-hour test run and time considerations to control the scale of the overall project. branch visits for data collection were scheduled in advance, and visit dates were staggered for each location so the lead researcher could be physically present to observe interactions. dates were selected based on convenience and in consideration of an external requirement for all data collection and analysis to be completed within the oneyear timeline of the internship. in pre-visit staff meetings, the lead researcher briefed participating public services staff at each branch on the study’s purpose and provided instructions on how to record survey interactions. staff members were encouraged to share as much detail about each interaction as possible, but were asked to avoid applying any classification (e.g., “the customer asked a directional question”), and were instructed to exclude personally identifying information. the question logging form was distributed for staff use in paper format, primarily for the ease of use by staff during the flow of work. although this public library has used web forms for other internal surveys, it was anticipated that staff might provide more detailed information on paper due to comfort and presence in mind. the lead researcher was physically present in the branches during data collection to observe a sample of the interactions between staff and customers. once the data collection stage at each location was completed, the lead researcher entered the form data into an excel spreadsheet. coding was approached by adapting from the analytical framework of grounded theory, as described in charmaz’s constructing grounded theory: a practical guide through qualitative research (2006) and corbin and strauss’ basics of qualitative research: techniques and procedures for developing grounded theory (2008). the lead researcher coded the data to develop thematic descriptors (concepts), and sorted those thematic descriptors into larger categories as they emerged. thematic descriptors were permitted to emerge organically in the coding evidence based library and information practice 2012, 7.1 28 process, and were developed and applied by considering both what the customer was asking and how the interaction was apparently resolved, rather than using definitions from the literature.1 the code book, which was developed to maintain consistency, is included in appendix a. limitations the data collected in this study has significant limitations: • the data collection consisted of only three days at each location, and different days were sampled at each location. as a result, the findings are indicative of a snapshot understanding, rather than a comparative or longitudinal study of activity traffic patterns. • data elements were recorded by participating frontline staff. a followup interview was not conducted with customers, so there is some potential for the interactions as recorded to be an imperfect or incomplete reflection of the customer’s actual information need. • although participating staff recorded diligently, the data does not reflect all of the activity happening in a given branch on a given day. some interactions were not included due to factors such as desk volume, forgetfulness, and desk shift 1 in the katz (1997) classification scheme that was later modified by arnold & kaske (2005), as referenced by radford and connaway (2007), there seemed to be problematic assumptions, for instance with the classification of an interaction as ready reference. what assumptions are being made about the nature of customer information needs and the nature of (digital and print-based) information itself when it is proposed that there is a single, uncomplicated, and straightforward answer to the question “who is the prime minister of canada?” these assumptions potentially disrupt sensitivity to the customer’s unstated and stated information needs by creating mental models about service where library staff are observed to substitute listening and asking probing questions as the first steps of an interaction with verbatim keyword searching. transitions. the researcher observed that roving questions were misreported as desk questions on occasion. • chat data was found in two location datasets only, due to implementation of system-wide chat monitoring occurring after most of the selected locations had been surveyed. results from june 2010 to february 2011 a total of 6,099 interactions were recorded by participating staff from 12 service points during the survey. of those 6,099 interactions, 1,920 (31.48%) were coded in the technology help category. the technology help category became a strong focus of analysis in the study, and the key findings for this category are presented in these results. technology help in context as can be seen in figure 1 and table 1, the codes and concepts that grouped to form the technology help category formed the largest cluster of interactions in the study at 31.48%. this category was second only to finding books, multimedia, and information at 25.23%, which included reference and readers services interactions. defining technology help the technology help category included customer interactions involving computers, printing, photocopiers, wireless, library provided equipment (e.g., headphones), and mobile or personal devices. interactions involving the self checkout stations and self check-in autosortation units, though strongly related to circulation, were also included in this category. these interactions were included as technology help because both systems presented a computer driven interface to the customer, and because the problem solving strategies in this area drew heavily upon instruction and technical skills. evidence based library and information practice 2012, 7.1 29 0 5 10 15 20 25 30 35 about spaces about services circulation membership finding books, multimedia, information technology help percentage of interactions table 1 description of figure 1 category codes with breakdown by raw number and percentage code category description raw number percentage technology help help with computers, printing, wireless, photocopiers, website, devices, self checkout 1,920 31.48% finding books, multimedia, information help finding books, multimedia, and information (readers and information services) 1,539 25.23% circulation help checking materials in or out, checking item status, and using the holds system 1,159 19% memberships help with new cards, card renewals, fine payments, and account management 1,155 18.93% about services procedural or logical questions about how the library’s services work 1,000 16.39% about spaces help locating library services and spaces (non-collection related) 453 7.42% amultiple category codes have been applied to each interaction, resulting in percentages higher than the interaction total. figure 1 percentage of interactions by category. evidence based library and information practice 2012, 7.1 30 the technology help category is further broken down into subcategories coded for the object with which help was requested (figure 2). interactions involving computers (1,023), printing (383), and self checkout stations (224) were the most dominant topics in the technology help category. customers asked for help with connectivity and file management across a broad range of devices including laptops (21), usb (10), ipad (4), kindle (3), iphones (3), netbooks (2), sony psp (2), kobo (1), sony ereader (1), blackberry (1), an unidentified cellphone (1), palm device (1), and nintendo wii (1). defining help: instruction and intervention textual analysis of the “answered with…” portion of the question logging form provided an insight into how library staff members were resolving technology help interactions (figure 3). the researchers identified that 671 (35%) of the technology coded interactions were resolved using instruction, and 387 (20%) were resolved when staff intervened administratively using workstation management software. in 862 (45%) technology coded interactions, too little detail was provided for the researcher to categorize the kind of help provided to the customer. technology instruction interactions ranged in complexity from simple device troubleshooting (e.g., customer wants to find a place where laptop can be plugged in to charge) to complex instruction in areas such as file management concepts, networking, document composition, and online etiquette. a number of technology coded interactions were blended with informational needs. customers were in the library attempting to accomplish a set task layered with social meaning and technology instruction (e.g., applying for a job online). these interactions required staff to instruct in information literacy skills and to guide in reference mediation of the resource. help in three areas: computers, printing, and self checkout a major finding of the study was that of 1920 technology questions (and of 6,099 questions in 0 200 400 600 800 1000 1200 wireless mobile devices ebooks/ereading autosortation equipment telephone photocopiers website self checkout printing computers technology help figure 2 technology help subcategories by number. evidence based library and information practice 2012, 7.1 31 total), 1,032 pertained to assistance with computer workstations and a further 383 questions pertained to printing from the library’s public workstations. as well, 224 customers requested help with the library’s self checkout kiosks. it is important to examine why these three areas appeared so dominantly. textual analysis of the technology help data revealed that customers requested help that varied in complexity with computers, printing, and other key self service technologies such as the self checkout machines. some of this help was due to the interplay of task complexity and gaps in computer or information literacy on the part of customers (e.g., a customer needed to apply for a job online, but first needed to receive instruction on email), but help requests were also physically observed to arise due to critical usability issues with key systems. one of the most critical usability issues was located in the unfamiliar interface presented by software on the library’s internet workstations. this interface, while operating atop windows xp, disrupted customer expectations for the microsoft windows “desktop metaphor” by omitting the start bar and windows explorer. further, the interface forced customers through a third party “application launcher” that appears in the middle of the screen and involves a vertical scrollbar for selecting applications. the restrictions applied through the software also forced customers to access removable media (memory sticks) through the open file menu choice in common applications, a process that appeared to be idiosyncratic and unexpected for customers. textual analysis and physical observation revealed this interface to be problematic for computer savvy customers who appeared to be familiar with the windows environment. it was also observed to be problematic as a learning environment for customers who indicated, verbally or otherwise, inexperience or anxiety with computer use. this is particularly troubling to consider when it appeared that some of those customers were at the library for the very purpose of accessing digital literacy instruction in the form of one-on-one computer tutorials. although the computer workstations stood out in the survey results, the self checkout machines were the third most reported object in technology help requests and all branch locations surveyed included at least one self checkout machine. textual analysis and observation revealed two kinds of help requested with self checkout machines: the first type of help requested was when the self checkout would not perform a specific, expected, user-initiated task, such as checking 35% 20% 45% types of technology help provided tech instruction tech maintenance undefined figure 3 types of help provided for technology coded interactions. evidence based library and information practice 2012, 7.1 32 out an item or printing a receipt. textual analysis revealed that this failure sometimes arose due to a system generated condition (e.g., library policy blocks checkouts at a $10 fine count, with the idea that customers will approach a desk and resolve the bill). at other times, the interaction text spoke to problems where the failure was more mechanical in nature (e.g., a jammed receipt printer is easy for customers to diagnose and report to the desk). the second kind of help requested most often was when customers were unable to intuit the steps required – or the messages presented – by the self checkout kiosks. for example, the self checkout will inform customers that an expired library card or a card that has reached a numerical checkout limit is “invalid.” from observation, customers would approach the desk, alarmed, thinking the card information might be “missing” from the system, when in reality the severity of the problem was far less than signaled by the vague message. customers were similarly deprived of useful system messaging when attempting to check out an item with limitations imposed by type (e.g., reference) or items with an erroneous state (e.g., still discharged to another customer). a third possibility that was observed by the researcher, but did not appear substantively in the text of the interaction data, may exist in how customers conceptually link the self checkout kiosks to account information needs. in the data, 300 instances were recorded in which customers asked the library for information about the contents or status of library memberships. although question analysis revealed many possible causes for this question, it is notable that customers were, at times, observed to proceed directly from the self checkout to a service desk in order to obtain more detailed information about their accounts, or to confirm the status of an item when the machine messaging or functionality left that status in doubt. for example, at one branch two customers were observed asking staff members to verify that items were checked out properly when the number of items in hand did not match the number of items listed by receipt. the title and author details on circulated items are not currently presented in a list format through the self checkout interface, but are instead accessible through library service “catalogue stations,” which are restricted in function. the researchers speculate that catalogue stations have a stronger conceptual link with searching than with account management, but further study is required. textual analysis revealed that the printing system (383) presented very comparable challenges to the problem types that customers encountered with the self checkout kiosks. customers would approach when the document failed to print as expected, but this involved an added layer of complexity in troubleshooting for staff and customers because of the additional number of sources from which the issue could be arising. in some cases, customers struggled with the layer of print management software used by the library to release and coordinate payment for prints. the printing software appeared to disrupt the expectations of customers accustomed to home or office printing environments by presenting a series of vague messages and unclear confirmation steps in the form of pop-up dialog boxes. in some cases, customers approached the desk with a problem that turned out to be insufficient credit for printing (which is passively indicated by the software through an account balance or a pop-up dialog box, rather than as active item requiring immediate resolution and offering steps to proceed), but was sometimes misidentified by customers as a physical problem such as “the printer is out of paper.” in attempting to resolve this problem, some staff members indicated that they had checked the printer immediately, some indicated that they had accompanied the customer to his/her computer for a closer look at where the customer was in the process and possible instruction on the software, and some staff members indicated that they had performed administrative intervention by viewing the customer’s account details from a staff workstation. in some cases, the troubleshooting path involved more than one strategy and more than one trip evidence based library and information practice 2012, 7.1 33 to a customer or staff workstation to identify and resolve the issue. technology in library space survey data revealed a small class of interactions in which customers asked for help in spaces impacted by the use of technology, but not with technology itself. textual analysis and observation revealed that differing activities located in library computing spaces created tensions among customers with respect to noise, concentration, and privacy. there were 11 interactions across 5 locations throughout the data in which customers asked where they could find quiet space for themselves or for a study group. there were also 16 noise complaints across 5 locations in computing contexts where customers who were engaged in an individual task requiring concentration (e.g., writing an email or resume) objected to noise generated by groups of customers also on computers but engaged in social activities such as viewing social media sites, streaming media, or playing games. among the category of interactions that included codes and concepts about the library’s physical space, interactions in which customers asked about the location of internet workstations ranked third (56) behind questions about the washrooms (133) and questions about where to return materials (84). one potential reason why these interactions arose is because of the apparent visual similarity between the internet workstations and some of the catalogue computers (that are restricted to opac and database access) at some locations (figure 4). discussion the survey results indicated that for the technology help interaction category, customer requests for help with the library’s computer workstations dominated all other subcategories. based on three significant areas of analysis arising from the results, the researcher recommended that the library conduct a thorough usability review of the internet workstation interface. in particular, the researchers recommended that the library aim to address five common customer tasks (logging on, launching applications, locating and opening a document, printing, and downloading/attaching a file with email) recorded in the survey data, through a change in configuration of the current software or through a search and evaluation of new software to implement. further, the researchers recommended that the library, figure 4 at edmonton’s downtown stanley a. milner library, the stand up catalogue station (left) appears ambiguous in function, but is at least distinctive in orientation in comparison to the internet workstation (centre) and the sit down catalogue station (right). evidence based library and information practice 2012, 7.1 34 upon any changes, follow up with a user study – including a focus group of customers familiar with a windows operating environment and at least one webmail service – to test the changes. the researchers recommended that the library should explore implementation of a solution for printing that is easier for customers to use, which requires minimal steps, presents clear language, and allows for self service payment. the researchers also recommended that the library conduct an audit of service messages that are commonly presented to customers by key technology systems, similar to the focus presented by saarti and raivo (2011). in that study, the scholars read one public library’s opac as a text and analyzed the social and information literacies required to decode that text in the larger context of human-computer, human-information interactions. survey results, especially in regard to the confusing language presented by the self checkout stations, speak to the need for public libraries to present messages and language to customers that are friendly in that they are easy to understand, possibly graphically represented, concise and accurate, and instructive. these qualities serve to create a sense of choice and power for customers. public libraries not only need to present friendlier and clearer messages where possible, but must also communicate that this kind of user experience is a requirement when they select vendor software and provide feedback to vendors about existing products. where self service options necessarily eclipse in-person contact with staff (e.g., system generated holds and overdue notices), messages need to be evaluated with particular care. does the tone, language, and visual design of system notices reflect the library’s mission, vision, and values? or does the tone and language shut down communication? system messages – like advocacy and marketing messages – must facilitate meaningful conversations and action between public libraries and communities. the researchers additionally recommended that the library adjust desk staffing strategies and mobile reference practices to increase staff awareness to “rove” the self checkout machines and the autosortation units, which appeared extensively in the survey results. follow up study is recommended on the question of co-existing spaces and digital workspaces that support individual and shared tasks. interactions in this subcategory appeared in a limited way in the survey data, and appeared to require alternative methods of study – as in gibbons and foster’s studying students project – to be captured in a systematic manner. from a staffing perspective, the volume and complexity of technology instruction questions (671) and the prominence of technology questions as a whole (1,920) in the study, suggested that public library customers need public library staff who see technology as a foundational component of library work, who feel capable in attempting to help, and who have superb interpersonal skills. crucially, given the rate of technological change, library staff members also require the skills to co-learn and co-instruct with a variety of different systems, devices, and problem classes. colearning, interpersonal, rusa reference interview, and strong search skills remain key elements of any customer service strategy around technology training. conclusions the study found that technology help is a central customer need within edmonton public library, and that help requests vary in complexity and are frequently resolved with instruction. the study also found that technology help questions are sometimes blended with more complex information needs, and infused with evolving social meaning (e.g., job search). internet workstations, printing, and self checkout are asked about in significant numbers and present critical usability challenges for library customers due to issues with interface design and workflow. the researcher recommends that the library focus on improving ease of use in key systems, on auditing service messages for clarity and accuracy, and on re-designing some staffing practices and structures to support technology assistance as a foundational component of library work. evidence based library and information practice 2012, 7.1 35 given the strong appearance of technology help in this study, the edmonton public library has acted to address the need for digital literacy services at a strategic level in its business plan document: the way we share: epl business plan 2011 2013. the library has resolved to position digital literacy services with customers through programming and partnerships, and to position digital literacy services with library staff by implementing new positions and training delivery models, with the ultimate goal of developing the skills and confidence to help customers with technology. further, the library is resolved to address some user experience issues identified in the study (e,g., printing) by offering “online services [that] are easy to use – at home and on the go” and to become “[the] community’s digital workspace” (2011, pp. 14-15). references aabø, s. (2005). the role and value of public libraries in the age of digital technologies. journal of librarianship and information science, 37(4), 205-211. doi:10.1177/0961000605057855 aegard, j. (2010). library kiosks: a balancing act. computers in libraries, 30(8), 16-20. arnold, j., & kaske, n. (2005). evaluating the quality of a chat service. portal: libraries and the academy, 5(2), 177-193. doi:10.1353/pla.2005.0017 bertot, j. (2009). public access technologies in public libraries: effects and implications. information technology & libraries, 28(2), 81-92. cavanagh, m. (2006). re-conceptualizing the reference transaction: the case for interaction and information relationships at the public library reference desk. canadian journal of information & library sciences, 30(1/2), 1-19. charmaz, k. (2006). constructing grounded theory: a practical guide through qualitative research. thousand oaks, ca: sage publications, inc. city of edmonton. (2009). municipal census results. in city of edmonton. retrieved 1 feb. 2012 from http://www.edmonton.ca/city_govern ment/facts_figures/municipal-censusresults.aspx clark, k. (2007). mapping diaries, or where do they go all day? in n. f. foster & s. gibbons (eds.), studying students: the undergraduate research project at the university of rochester. (pp. 48-54). chicago: association of college and research libraries. retrieved 1 feb. 2012 from http://hdl.handle.net/1802/7520 corbin, j. m., & strauss, a. l. (2008). basics of qualitative research: techniques and procedures for developing grounded theory. thousand oaks, ca: sage publications, inc. gibbons, s., & foster, n. f. (2007a). library design and ethnography. in n. f. foster & s. gibbons (eds.), studying students: the undergraduate research project at the university of rochester. (pp. 20-29). chicago: association of college and research libraries. retrieved 1 feb. 2012 from http://hdl.handle.net/1802/7520 gibbons, s., & foster, n. f. (eds.). (2007b). studying students: the undergraduate research project at the university of rochester. chicago: association of college and research libraries. retrieved 1 feb. 2012 from http://hdl.handle.net/1802/7520 edmonton public library. (2011). the way we share: epl business plan 2011 2013. in edmonton public library. retrieved 1 feb. 2012 from http://www.epl.ca/sites/default/files/p df/businessplan_2011_final.pdf evidence based library and information practice 2012, 7.1 36 fox, k., horne, l. m., king, t., seely, s., & walsh, k. (2008). the librarian as bridge-builder. public services quarterly, 4(2), 177-185. doi: 10.1080/15228950802203281 finnell, j., & fontane, w. (2010). reference question data mining: a systematic approach to library outreach. reference & user services quarterly, 49(3), 278-286. retrieved 1 feb. 2012 from http://www.rusq.org/wpcontent/uploads/2010/10/rusq49n3_1 0_finnell.pdf katz, w. a. (1997). introduction to reference work. (7th ed.). new york: mcgrawhill. lai, h.-j. (2011). information literacy training in public libraries: a case from canada. educational technology & society, 14(2), 81-88. retrieved 1 feb. 2012 from http://www.ifets.info/journals/14_2/7.p df meserve, h. c., belanger, s. e., bowlby, j., & rosenblum, l. (2009). developing a model for reference research statistics: applying the “warner model” of reference question classification to streamline research services. reference & user services quarterly, 48(3), 247258. retrieved 1 feb. 2012 from http://www.rusq.org/wpcontent/uploads/2009/48n3/pdf/rusq 48n3_meserve.pdf nagy, a. (2011). next-generation service in the library. library technology reports, 47(7), 8-10. radford, m. l., & connaway, l. s. (2007, june). not dead yet! ready reference in live chat reference. 13th rusa new reference research forum, washington, dc, usa. retrieved 1 feb. 2012 from http://www.oclc.org/research/activities /synchronicity/resources/20070624-alarusa-handout.pdf ryan, s. m. (2008). reference transactions analysis: the cost-effectiveness of staffing a traditional academic reference desk. journal of academic librarianship, 34(5), 389-399. retrieved 1 feb. 2012 from http://pdn.sciencedirect.com/science?_ ob=miamiimageurl&_cid=272069&_ user=1069227&_pii=s009913330800100 6&_check=y&_origin=browse&_zone=r slt_list_item&_coverdate=2008-0930&wchp=dglbvlvzskzv&md5=d9492bcbf4fab85bc74d90 0ac137ef24/1-s2.0-s0099133308001006main.pdf saarti, j., & raivio, j. (2011). how to communicate with a machine: on reading a public library’s opac. journal of librarianship and information science, 43(1), 22-29. doi:10.1177/0961000611401258 wong, g. k. w. (2010). information commons help desk transactions study. journal of academic librarianship, 36(3), 235-241. retrieved 1 feb. 2012 from http://pdn.sciencedirect.com/science?_ ob=miamiimageurl&_cid=272069&_ user=1069227&_pii=s009913331000063 7&_check=y&_origin=browse&_zone=r slt_list_item&_coverdate=2010-0531&wchp=dglzvlszskzk&md5=812007525f9472d549c8352 e93a4daa3/1-s2.0-s0099133310000637main.pdf evidence based library and information practice 2012, 7.1 37 appendix a codes used for thematic analysis code definition aboutepl-services questions about how the library's services or procedures function aboutepl-spaces questions about where equipment, people, or areas are in the library accountlookup staff retrieve customer account details advisory an advisory interaction between staff and customers answered no declined a customer's request arts and literature reference in the area of arts and literature (700 800s dewey) blocked customer's account is blocked due to status (fines, etc) booking booking a computer btg request for best seller to go item card renewal privilege renewal cdrequest customer requests material (music, spoken) in cd format community reference question about edmonton people, places, things, or organizations complaint a customer complaint conversation staff and customer converse circulation pertaining to checkins, checkouts, item renewal, holds circ trouble customer encountered difficulty trying to circulate an item directional where is… dvdrequest request for material in a dvd format. ebooks/ereading questions about ebooks, eaudiobooks, or ereading equipment questions about headphones, disk drives, or other in-library equipment eresourcerequest request for material in a digital format. expiry a membership was expired family card indicates family management scenario a parent managing one or more cards fines an inquiry about fines or fine payment forgot card customer wants to use services or borrow, but lacks physical card forgot pin customer wants to use services, but needs pin formatting customer asks for assistance with document or text formatting on computer fttg request for a flicks n tunes to go item gamerequest request for material in game format genre asking for a genre or topic area of materials genealogy a genealogy reference question government and law a reference question pertaining to government or law, at all levels health reference a reference question about health information holds help locating holds that have already been placed information servicescatalogue staff use the catalogue with a customer information servicesreference a reference question internet pass request for a temporary internet pass interlibraryloans questions pertaining to or resolved with interlibrary loans evidence based library and information practice 2012, 7.1 38 item renewal customer asks for borrowed material to be renewed job seeking request for help searching or applying for a job juvenile request pertains to youth services language learning request for language learning material, resources, or programming library instruction staff instruct a customer on library functions lost card customer reports a lost card lost and found requests for lost effects or for the lost and found specifically membership pertaining to membership status or policy mobile technology request for help using a mobile device multimedia request for non-print material new card a new registration, new library card issued pholds customer request that a hold is placed printing help printing documents or using the print system programs questions about program offerings, registration, or attendance readers service questions about print materials readers servicemagazine questions about magazines readers servicenewspaper questions about newspaper recruitment questions about epl as an employer referral staff refer a customer's question to another desk, service point, or external organization resume help customer requests help composing a resume security staff refer to security staff science and technology a reference question in the science or technology area (dewey 500 600s) snag a missing disc or component of material is discovered or reported specific title customer request for a specific title or author stationery customer request for pens, pencils, papers, bags, staplers tech instruction customers request or staff provide technology instruction tech maintenance staff resolve a technical problem for a customer using admin software or other administrative intervention (reboot, etc) technology services-asu autosortation unit technology servicescomputers computer workstations technology servicesdatabases licensed databases technology servicesfaxing request for fax service technology serviceslaptops loanable laptops (spw) technology serviceslending century park lending machine technology servicesmicrofilm microfilm technology servicesnotifications ils generated notifications on holds or overdues technology servicesphones request to use the library's telephones evidence based library and information practice 2012, 7.1 39 technology servicesphotocopier photocopying help technology services-sco self checkout technology servicesviewing dvd viewing stations (cra) technology serviceswebsite website technology serviceswireless connecting to epl's wireless time extension request for more computer time evidence based library and information practice 2012, 7.1 40 appendix b “what customers want” survey form / evidence based library and information practice/ / analyzing public library service interactions to improve public library customer service and technology systems objective – to explore the types and nature of assistance library customers are asking library staff for in a large canadian urban public library system. methods – a qualitative study employing transaction logging combined with embedded observation occurred for three-day sample periods at a selection of nine branches over the course of eight months. staff recorded questions and interactions at service ... introduction public libraries and technology the literature speaks widely to the potential impact, challenges, and opportunity for technology use and technology learning in public libraries for library customers and library staff. technology help in context defining technology help defining help: instruction and intervention help in three areas: computers, printing, and self checkout technology in library space appendix a codes used for thematic analysis appendix b “what customers want” survey form exploring topics and genres in storytime books: a text mining approach research article   exploring topics and genres in storytime books: a text mining approach   soohyung joo associate professor, school of information science university of kentucky lexington, kentucky, united states of america email: soohyung.joo@uky.edu   erin ingram research assistant, school of information science university of kentucky lexington, kentucky, united states of america email: erin.ingram@uky.edu   maria cahill associate professor, school of information science university of kentucky lexington, kentucky, united states of america email: maria.cahill@uky.edu   received: 15 apr. 2021                                                                  accepted: 14 oct. 2021      2021 joo, ingram, and cahill. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29963     abstract   objective – while storytime programs for preschool children are offered in nearly all public libraries in the united states, little is known about the books librarians use in these programs. this study employed text analysis to explore topics and genres of books recommended for public library storytime programs.   methods – in the study, the researchers randomly selected 429 children books recommended for preschool storytime programs. two corpuses of text were extracted from the titles, abstracts, and subject terms from bibliographic data. multiple text mining methods were employed to investigate the content of the selected books, including term frequency, bi-gram analysis, topic modeling, and sentiment analysis.   results – the findings revealed popular topics in storytime books, including animals/creatures, color, alphabet, nature, movements, families, friends, and others. the analysis of bibliographic data described various genres and formats of storytime books, such as juvenile fiction, rhymes, board books, pictorial work, poetry, folklore, and nonfiction. sentiment analysis results reveal that storytime books included a variety of words representing various dimensions of sentiment.   conclusion – the findings suggested that books recommended for storytime programs are centered around topics of interest to children that also support school readiness. in addition to selecting fictionalized stories that will support children in developing the academic concepts and socio-emotional skills necessary for later success, librarians should also be mindful of integrating informational texts into storytime programs.     introduction   storytime programs for preschool children are a common provision of public libraries worldwide. with a long history (albright et al., 2009) and the highest rates of attendance among public library program offerings (miller et al., 2013), storytimes serve as the backbone of public library programming. preschool storytimes are typically designed to be highly interactive, learning-focused, thirty-minute enjoyable endeavors that incorporate shared book readings, props, songs, and activities intended to engage both children and their adult caregivers (diamant-cohen & hetrick, 2014; goulding et al., 2017; mills et al., 2018).   book selection is one task among many that storytime providers undertake when planning a public library storytime program. other planning tasks may include deciding on a theme, incorporating educational tips for caregivers, gathering supplies for an activity or craft, creating handouts for caregivers, integrating technology resources, and choosing songs, interactive rhymes, fingerplays, movement activities, or flannel board activities (diamont-cohen & hetrick, 2014; ghoting & martin-diaz, 2006). to aid in planning these tasks, storytime providers may rely on professional resources for storytime development such as conferences, books published by the american library association and other library publishers, trainings offered through state library agencies and consortia, and webinars or courses from commercial websites such as webjunction or library juice academy. however, these formal resources may not be accessible to all storytime providers due to financial costs or time constraints. therefore, storytime providers at a public library with a limited budget may prefer to plan with the aid of a free informal, online resource, such as a librarian’s blog about storytime programs or a library’s website with recommended reading lists.    in this study, we explored the subject matter of books recommended by online resources for public library preschool storytime programs. we extracted bibliographic records of books recommended for storytimes and analyzed them to explore topics, genres, and sentiment of those books. we applied text mining methods for data analysis, such as term frequency and network analysis, lda topic modeling, and sentiment analysis.   literature review   theoretical framework   picture books serve as unique multimodal sources of information in which verbal and visual elements combine and interact to convey meaning (martens et al., 2012). the tension between visual and verbal modes—the tendency to slowly gaze upon images and layout versus the forward momentum of reading text—make the picture book reading experience distinct and synergistic (sipe, 1998). serafini (2012) has posited four roles essential for picture book reading: 1) the navigator role entails proficiently employing conventions of print and cognitive strategies and skills to move through the written text while simultaneously making sense of the visual design and images; 2) the interpreter role consists of meaning making; 3) in acting as a designer, the reader determines the path through which a text is experienced by determining the order of attention to and importance of textual and visual elements; and 4) in enacting the interrogator role, the reader serves as a critical analyst acknowledging that messages and interpretations are socially constructed and politically, socially, and culturally powerful. using eisenberg and small’s (1993) framework for information-based education, campana (2018) demonstrated that information resources, information process, and individuals interact within the storytime context to provide learning opportunities for all involved. as a primary information resource within storytime programs, picture books are an important element of the storytime learning environment, and the storytime librarians’ processes facilitate children’s picture book reading and meaning making.   general content of children’s books   nearly half of all picture books published in english in the united states feature animals as leading characters (horning, 2016). preschool-aged children have shown an aesthetic preference for artwork, such as picture book covers, that feature animals (danko-mcghee & slutsky, 2011). markowsky (1975) pointed to four reasons why children’s book authors and illustrators might choose animal characters: a) void of the accoutrements associated with specific groups of people, animal characters are relatable to children of all stripes and circumstances; b) animal characters produce an element of whimsy, enabling escape, inspiration, and imagination; c) animal characters serve as a form of shorthand or symbolic representation; and d) animal characters with exaggerated features and characteristics lend an element of humor.   book choice in preschool education   several studies have examined characteristics of books read aloud in preschool classrooms and centers. mesmer (2018) interviewed the staff of 31 preschool centers in the southwestern united states who expressed that the characteristics they considered most frequently were illustrations, rhyme, length, simple content, and a topic relevant to children. when examining the genres of such books, researchers have noted a lack of informational books (pentimonti et al., 2011; thoren, 2016). this is concerning because sharing informational books provides many educational benefits to children, such as content knowledge and preparation for successfully reading this genre in school (lennox, 2013; neuman et al., 2016), and because children are highly interested in informational books (baldwin & morrow, 2019; price et al., 2012). in addition, studies of preschool classrooms revealed a lack of concept books, or books that focused on foundational academic topics such as counting and the alphabet (gou et al., 2013; pentimonti et al., 2011).   book choice in public library storytimes   diamant-cohen and hetrick (2014) contended that “the main goal of preschool storytime is to help children develop a positive connection with books and illustrations, which will later translate into a positive attitude toward books in general” (p. 4). to accomplish this goal, librarians typically incorporate three to four books in each storytime program (diamant-cohen & hetrick, 2014; goulding et al., 2017; kociubuk & campana, 2019). naturally, the topics and sentiment of books shared in public library storytime programs will affect the extent to which children develop a positive connection; thus, librarians’ book choices are important.   despite the importance of book choice, only a few studies have examined the characteristics of books chosen for storytime programs. one study examined the books shared in 69 baby, toddler, preschool, and family storytimes in public libraries in the state of washington during 2013 (kociubuk & campana, 2019). the book characteristics collected were genre (fiction or nonfiction) and publication date; thus, this study does not provide insight into the topic or sentiment of storytime books. in contrast, two surveys of professionals who choose storytime books revealed that they did take into account the topic and other aspects of a book’s content (carroll, 2015; fullerton et al., 2018). carroll (2015) identified seven influential factors in book selection: length and complexity; illustrations; subjects, concepts, and themes; use of language such as rhyme and repetition; how easily the book could be used to invite audience participation; elements such as suspense or humor that could emotionally engage the children; and personal preference. similarly, respondents to fullerton et al.’s (2018) survey chose language use and illustrator’s craft as top considerations when selecting books.   objectives of public library storytimes   one of the important program objectives of storytimes lies in learning and education (campana, 2018; fehrenbach et al., 1998; peterson, 2012). storytime providers have long emphasized early literacy development of young children when designing programs (albright et al, 2009), but scholars have recently noted the broader role libraries play in supporting school readiness (campana, 2018). storytime serves as a valuable opportunity for children to build knowledge structures of colors, numbers, singing, alphabet, and more (cahill et al., 2018). according to celano and neuman (2001), “by reading books, telling stories, and reciting rhymes, librarians offer children a ‘leg up’ in developing emergent reading skills” (p. 39).   while parents and caregivers certainly value storytimes for the learning opportunities they afford, many view these programs as family entertainment venues (khoir et al., 2017) and  worthy experiences simply because of the joy they bring to children (cahill et al., 2020). most storytime programs offer opportunities for children and adults to interact, play, and sing (celano & neuman, 2001). stories told in rhymes and picture book versions of songs, in addition to encouraging playful interaction, may aid children in developing early literacy skills such as phonological awareness (giles & fresne, 2015).   additionally, storytimes serve as opportunities to stimulate and extend children’s feelings and emotional experiences because reading children’s literature can be a source of emotional learning for children (short, 2018; thoren, 2016) and shared book reading has been found to be a viable activity to help children build socio-emotional competence (schapira & aram, 2019). further, a majority of literacy educators, including librarians, regard social emotional learning as a responsibility of literacy educators but one for which they need further support (international literacy association, 2020).   the aim and research questions of the study   storytime programs for young children are offered in nearly all public libraries. however, few researchers have investigated the topics and content of the books that librarians use in such programs. it is important to have a better understanding of the content of the books to produce best practices and suggestions for book selection in storytimes for young children. our overarching aim through this study was to explore topics and genres of the books that are read in storytimes for young children in public libraries. in addition, we aimed to examine the nature of sentiment represented in books widely used in storytimes for young children. the methodological contribution of this study is that it employed computational text analysis methods to investigate the content of a large sample size of books recommended for library storytime programs. the following two research questions guide the investigation of this study:   rq 1. what are the topics and genres of books recommended for public library storytime programs designed for preschool children?   rq 2. what is the nature of sentiment represented in books recommended for public library storytime programs designed for preschool children?   table 1 online resources for storytime chosen for this study resource number of total themes number of themes in stratified random sample esther storytimes 74 15 jbrary 19 4 mcls kids 158 32 silly librarian preschool 100 20 storytime katie 195 39 storytime secrets 70 14 total 616 124   methods   to identify informal sources that storytime providers were likely to encounter and use, we conducted a simple internet search for “storytime resources for librarians.” we then selected the first six sources that recommended books for storytime use based on theme: esther storytimes, jbrary, mcls kids, silly librarian, storytime katie, and storytime secrets. we next created a list of all preschool storytime themes shared on each site, which totaled 616 in all (table 1). because each theme contained multiple book recommendations, including the entire population would be laborious and unfeasible from a data preparation standpoint; thus, we randomly selected 20% of the themes, resulting in 124 themes in the sample. for each theme, we recorded the name and author of each book recommended as supporting the theme and appropriate for storytime.   in this study, we analyzed largely two types of textual information collected about each book from the worldcat database: “titles and abstracts” and “subject terms.” we excluded any books that did not have an abstract available from worldcat. after removing them from the book list, 429 books were used for analysis. two sets of text corpus data were made for text mining: (1) titles and abstracts and (2) subject terms.   multiple text mining techniques were employed to explore the content of storytime books. first, a term frequency analysis was conducted to identify the most frequent terms that occurred in each corpus. the collected text underwent a pre-processing process, including tokenization, stopword elimination, and stemming. in addition, bi-grams were further investigated to identify key concepts or popular genres in storytime books. second, we analyzed the relationships among the terms based on term co-occurrence analysis. a term co-occurrence map was created to identify key topics and genres in storytime books. third, latent dirichlet allocation (lda) topic modeling was applied to uncover prevailing topics underlying the books commonly recommended for storytimes. lda topic modeling is an unsupervised machine learning method to discover hidden themes or topics from unstructured text (blei, 2012). we extracted 20 topics from the corpus of titles and abstracts. fourth, we explored the nature of emotion reflected in storytime books by analyzing sentiment terms. the sentiment lexicon constructed by liu (2015) was adopted to investigate emotional aspects of the text from storytime books. the bing lexicon classifies selected words into the binary categories, i.e., positive and negative sentiment categories (https://www.cs.uic.edu/~liub/fbs/sentiment-analysis.html).   textual analysis was conducted using various r software packages. r is a software tool that can be used for statistical analysis and data science, and it includes a multitude of packages for natural language processing and text mining. for term frequency analysis, two packages were mainly used: r tm and r tidy. for the lda topic model, the r topic models package was employed.   results   rq 1. what are the topics and genres of books recommended for public library storytime programs designed for preschool children?   first, we calculated the most frequent terms from the corpus of titles and abstracts. it exhibited 2,289 unique terms and 7,718 tokens after removing stopwords. in textual analysis, a unique word (also called type) refers to a distinct term in a corpus while a token indicates an occurrence of a unique type (jackson & moulinier, 2007). we investigated the top 96 stemmed terms that appeared more than 15 times (appendix a). the two most frequent terms are “book” (1st, 1.35%) and “anim” (2nd, 1.17%). abstracts usually summarize the books, so it is not a surprise that the term “book” occurred most frequently from the corpus. interestingly, “anim,” which indicated the term of “animal(s),” was observed second most frequently. in addition, there were other animal-related terms observed among top ranked terms; for instance, “bear” (3rd), “cat” (15th), and “dog” (21st), among several others. the results also included several terms associated with early learning, such as “color” (8th), “rhyme” (12th), “alphabet” (26th), “read” (43rd), and “count” (82nd). another distinct group of frequent terms can be classified as book audiences or characters, such as “young” (7th), “babi” (9th), “children” (10th), “boy” (12th), and “reader” (23rd). in addition, several terms were related to actions or movements, such as “play” (3rd), “find” (3rd), and “danc” (37th). we also observed terms depicting emotion, such as “love” (3rd), “fun” (43rd), and “enjoy” (82nd).   we next turned attention to the “subject terms” corpus and tallied frequencies of the terms observed in it. the corpus consisted of 538 unique terms and 3,642 tokens. subject terms were much shorter than the combinations of titles and summaries. also, this corpus mostly consisted of nouns with only few adjectives or verbs observed. the top 82 most frequent terms that appeared more than five times are listed in appendix b. the analysis showed that subject terms are likely to provide genre information or more condensed topic terms. the two top-ranked terms indicated genres of books: “fiction” (1st, 25.95%) and “juvenil” (2nd, 13.40%). worldcat organizes books by genre using subject headings. for example, a fairy-tale fall by apple jordan has three subject terms: “autumn -juvenile fiction,” “princesses -juvenile fiction,” and “halloween -juvenile fiction,” and represents a typical format of worldcat subject terms, which is a combination of a topical term and a genre. thus, genre related terms were a frequent observation from the corpus. other genre classification terms that ranked highly included “stori” (3rd), “rhyme” (6th), “literatur” (7th), “pictori” (10th), “picture” (15th), “movabl” (17th), and several others. this finding implied that most storytime books could be categorized as juvenile fiction, stories in rhyme, pictorial works, picture books, or movable books. we can also infer that storytime books involved other genres, such as “folklore” (24th), “poetri” (26th), and “nonfict” (55th).   in addition, subject terms depicted prevalent topics in storytime books. interestingly, similar to the results from the titles and abstracts, the term “anim” is ranked highly at 5th, revealing the popularity of animal related topics in storytimes. animal related terms included “dog” (19th), “bear” (20th), “cat” (24th), “rabbit” (46th), and several others. several of the terms implied topics related to knowledge and skills important for young children; for example “count” (20th), “song” (28th), “alphabet” (34th), “sound” (46th), “scienc” (46th), “read” (68th), and “languag” (83rd). storytime books also reflected social and behavioural topics, such as “friendship” (46th), “behavior” (61st), and “social” (68th). other notable topics or concepts that children can learn included: family (e.g., “famili” (32nd), “mother” (34th), “parent” (46th), “son” (61st), and “father” (55st)), nature (e.g., “natur” (32nd), “snow” (46th), “moon” (61st), and “tree” (83rd)), settings (e.g., “farm” (39th), “garden” (46th), and “zoo” (83rd)), halloween (e.g., “halloween” (34th) and “pumpkin” (68th)), and other objects/characters (e.g., “monster” (55rd), “pirat” (61st), and “dinosaur” (68th)).   we further analyzed bi-gram terms, which indicated two adjacent tokens, from the titles and abstracts corpus. in total, 9,018 unique bi-grams were observed after removing stopwords. appendix c lists the top 44 bi-gram terms that were counted four times or more. the top two bi-grams are “picture book” and “rhyming text.” also, among the ranked bi-grams, there were bi-grams that indicated the types of books, including “board book,” “simple text,” and “illustrations rhyming.” this finding highlighted that the nature of storytime books related to pictures, rhymes, and simple text to be shared with young children. the bi-gram analysis result also showed popular topics and characters in storytime books. as shown in figure 1, a term co-occurrence network revealed relationships among key terms in storytime books. the term “book” showed close associations with “picture,” “children,” and “animal,” which reveals popular types of books in storytimes. another notable linking group is made of “text,” “illustrations,” and “rhyming.”   the same bi-gram analysis was conducted for the subject terms (appendix d). in total, 3,250 bi-grams (1,312 unique bi-gram types) were observed. the most frequent bi-grams indicated the genres or types of storytime books; for example “juvenile fiction,” “juvenile literature,” “fiction stories,” “board books,” “pictorial works,” and so on. the visualization of term co-occurrence analysis highlighted the genres of storytime books as it placed “juvenile” and “fiction” among other genre terms in the center of the network diagram (figure 2).   to explore popular topics in storytime books, we derived 20 topics from the titles and abstracts corpus using the lda topic model (table 2). not all topics exhibit distinct and coherent thematic terms, but we found that these topics can be classified largely into several categories. there are several learning components observed from the extracted topics: for example, t7 (e.g., alphabet, letter) and t4 (e.g., color, crayon, count). animals and creatures were another prominent topic in storytimes, such as t6 (e.g., cat, dog), t2 (e.g., mouse, spider, pigeon), and t11 (e.g., bear, elephant). the lda topic model also detected topic terms relevant to book types, such as t9 (e.g., book, illustration), t14 (e.g., rhyme, illustration, and interactive) and t20 (e.g., illustration, rhyme). the components of activities, actions, and movements were also reflected in storytime books: for example, t5 (e.g., dance, swim, discover) and t4 (e.g., play). moreover, topic terms obtained from the lda algorithm represented other aspects of topics covered in storytime books, ranging from audiences (e.g., children, young reader), friends and families (e.g., friend, family, mother), nature (e.g., tree, sky, flower), settings (e.g., farm, garden), and emotions (e.g., fun, love, happy).   figure 1 a network of term co-occurrence relationships: titles and abstracts.   figure 2 a network of term co-occurrence relationships: subject terms.   table 2 lda topic model (20 topics) t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 night fun stop home busi visit pig differ follow pictur mous spider includ pigeon back web mommi cooki long varieti hat magic text rabbit classic pop best name treasur chicken play color crayon children creat heart zoo bestsel count time danc pumpkin halloween swim discov five favorit board pirat grade cat love dog shoe imagin pete around white kitti sing alphabet letter featur differ time pictur world full pea everi grow follow mani plant turn tree eat forest celebr librari book illustr life old artist time cut journey color easi egg girl back salli bug butterfli ice big chick bring t11 t12 t13 t14 t15 t16 t17 t18 t19 t20 bear big boy brown bus eleph tree celebr around cake find boy tri song pictur school describ best everyth join farm bunni mother text dinosaur tale bed amaz world readi young reader rhyme garden illustr moon children interact reveal edit stori give introduc train blanket surpris children includ sit answer perfect famili crocodil keep three full live water know show anim tail page cloth enjoy everyon wear parent louis aloud book babi love penguin toe red along delight flap page friend read simpl snow tri sheep monster kiss text hous appl ten illustr tree rhyme leav gingerbread pie simpl trick   rq 2. what is the nature of sentiment represented in books recommended for public library storytime programs designed for preschool children?   sentiment analysis was conducted to explore the emotional aspects of storytime language. we identified the top 25 positive and negative terms respectively (figure 3). overall, there were more positive terms than negative terms observed from the titles and abstracts corpus. frequently observed positive terms include “like,” “love,” “perfect,” “fun,” “classic,” “favorite,” and others. these positive terms described storytime books as likable, easy, fun, enjoyable, playful, happy, and other upbeat descriptions. on the contrary, there were fewer negative terms observed. the top ranked negative term turned out to be “fall.” according to our further observation, however, there were more cases when the term “fall” was used to indicate the season, rather than an act of falling or moving down: for example, “a little girl spends a glorious fall day picking apples and searching for the perfect pumpkin in this edition of a timeless favorite” in the book apples and pumpkins by anne rockwell. other negative terms did not necessarily have any negative nuance in the context of children's stories. for instance, pigs and bugs are likely to be featured as friendly characters rather than unpleasant objects in storytime books. despite the nuance based on context, other terms appeared with obvious negative connotations, such as “die,” “tired,” “trouble,” “lonely,” “skeptical,” “fear,” and others.   figure 3 positive and negative terms in storytime books.   discussion   by analyzing the most frequent terms, we explored different aspects of storytime books, such as popular topics, audiences, and styles or techniques for sharing the books. not surprisingly, animals were a major topic in these recommended books just as they are in picture books in general (horning, 2016). in our results, various types of animals were ranked highly, such as bear, cat, dog, mouse, and rabbit. because preschool-age children have shown a preference for artwork that features animals (danko-mcghee & slutsky, 2011), our results indicated that sharing these recommended books would appeal to children’s interests.   the frequent terms also pointed to opportunities to integrate school readiness and other elements of early learning into storytime programs. several different concepts related to children’s learning were observed, such as color, rhymes, alphabet, read, song, and count. the use of concept books may be especially important in public library storytimes to give children additional support in these content areas because studies have found that preschool classrooms and centers may offer only small numbers of concept books (guo et al., 2013; pentimonti et al., 2011). additionally, the findings coincided with the storytime providers’ preference for concept books that may engage and empower children with familiar, repetitive content (carroll, 2015). the frequency of terms related to early learning concepts indicated that providers can use these recommended books to achieve the school readiness related objectives of storytimes.   the term frequency analysis also pointed to opportunities for librarians to structure storytimes as emotionally positive and fun. frequent terms included “play,” “love,” “fun,” “enjoy,” and “dance.” based on attendees’ expectations that storytimes are a place for entertainment (khoir et al., 2017) and joyful experiences (cahill et al., 2020), sharing these recommended books can be part of meeting attendees’ expectations and accomplishing storytime objectives.   the analysis of subject terms illuminated popular genres of storytime books as they tended to include controlled heading terms of book genres/categories. the top 10 terms identified the most common genres of books shared in storytimes: fiction, juvenile, story books, rhymes, children's literature, board books, and pictorial works. we found that storytime books also included poetry, folklore, and nonfiction, although to a much smaller extent. these findings were similar to those of kociubuk and campana (2019), who found that less than 10% of stories shared in storytimes were in the narrative informational or non-narrative informational genres.    the term-level analysis revealed both visual and sound modes of story delivery in storytimes. terms related to visual mode, such as pictures and illustrations, were highly ranked in both the corpuses. also, storytime books actively involved sound modes, in particular, rhymes and songs. that is, storytime books are not simply text of stories, they promote interactivity through multiple modes by utilizing both visual and sound sensory channels to deliver stories to children. the importance of visual elements as a rationale for choosing books for read-alouds is reflected in research with children, storytime providers, children’s literature experts, and early childhood education teachers (carroll, 2015; danko-mcghee & slutsky, 2011; fullerton et al., 2018; mesmer, 2018).   movements and actions are another mode to facilitate interactions and are often integrated into storytimes; however, not many action/movement terms were observed among the top terms. this may be because storytime providers plan to get the audience moving before or after reading books aloud through activities such as songs, rhymes, or fingerplays (giles & fresne, 2015; kociubuk & campana, 2019).   we also explored emotional aspects of storytime books based on sentiment analysis. overall, the sentiment nature of the storytime context is very positive, showing more positive vocabulary proportionately in this study. that is, the fundamental atmosphere of storytimes would be very positive and enjoyable. in addition, we observed that storytime books included various words representing emotion. they can be sources for children to learn and understand a diverse spectrum of emotion and sentiment, thus giving providers the opportunity to contribute to children’s social-emotional learning, another objective of storytime programs.   this study was not without limitations. first, the analysis was done with the bibliographic records from the worldcat database. the dataset consisted of titles, abstracts, and subject terms of the selected books, but it did not include full-text. second, the sample of 429 books might not represent the entire book selection in storytime practices. the books included are recommended widely across the storytime community, but there is no information on which books are actually adopted and read in the field. third, several of the terms were not interpreted correctly in the sentiment analysis as the computational tool did not catch the meaning or nuance currently in the context. for example, the word “fall” was categorized into the negative terms even though it indicated the season. these limitations indicated a need for further research, which can investigate full-text content of an enlarged sample of storytime books. future research can also develop a sophisticated method to better analyze topical terms and sentiment based on text mining. additionally, the complexity of language can be investigated to assess the levels of vocabulary exposed to children via storytimes.   conclusion   in this study, we employed a text mining approach to explore topics of storytime books. we identified 429 books recommended for use in public library storytime sessions designed for preschool age children, and we collected two corpuses of text from the worldcat database based on these books: a) titles and abstracts, and b) subject terms. to investigate the nature of theme and sentiment in storytime books, we applied multiple text mining techniques for the collected text, such as term frequency analysis, bi-grams analysis, term co-occurrences network analysis, lda topic modeling, and sentiment analysis. the findings revealed popular topics and genres as well as emotional terms that would likely be addressed in storytimes drawing from these sources.   so what? why does it matter what books librarians may choose to share with children and caregivers during storytime? according to sipe (2008), “literature … allows us to perceive our lives, the lives of others, and our society in new ways, expanding our view of what is possible, serving as a catalyst to ignite our capacity to imagine a more just and equitable world. to understand stories and how they work is thus to possess a cognitive tool that not only allows children to become comprehensively literate, but also to achieve their full human potential” (p. 247). our findings suggested that the books recommended for storytime programs hold the promise of preparing children for school and for life.   author contributions   soohyung joo: conceptualization, data collection, data analysis, investigation, writing – original draft, writing – revision and editing erin ingram: conceptualization, data collection, literature review, investigation, writing – original draft, writing – revision and editing maria cahill: conceptualization, data collection, literature review, investigation, discussion, writing – original draft, writing – revision and editing.   acknowledgements   this work was supported by the institute of museum and library services (federal award identification number: lg-96-17-0199-17). we are grateful to the anonymous reviewers for their constructive comments.   references   albright, m., delecki, k., & hinkle s. (2009). the evolution of early literacy. children & libraries, 7(1), 13–18.   baldwin, c. g., & morrow, l. m. (2019). interactive shared book reading with a narrative and an informational book: the effect of genre on parent-child reading. preschool & primary education, 7(2), 81–101. https://doi.org/10.12681/ppej.20909   blei, d. m. (2012). probabilistic topic models. communications of the acm, 55(4), 77–84. https://doi.org/10.1145/2133806.2133826     cahill, m., joo, s., & campana, k. (2018). analysis of language use in public library storytimes. journal of librarianship & information science, 52(2), 476–484. https://doi.org/10.1177/0961000618818886    cahill, m., joo, s., howard, m., & walker, s. (2020). we’ve been offering it for years, but why do they come? the reasons why adults bring young children to public library storytimes. libri, 70(4), 335–344. https://doi.org/10.1515/libri-2020-0047    campana, k. (2018). the multimodal power of storytime: exploring an information environment for young children (publication no. 10825573) [doctoral dissertation, university of washington]. proquest dissertations and theses global.   carroll, v. (2015). preschool storytime in auckland’s public libraries: a qualitative study of book selection practices [master’s thesis, victoria university of wellington]. victoria university of wellington research archive. https://researcharchive.vuw.ac.nz/xmlui/handle/10063/4690    celano, d., & neuman, s. b. (2001). the role of public libraries in children’s literacy development: an evaluation report. pennsylvania library association.   danko-mcghee, k., & slutsky, r. (2011). judging a book by its cover: preschool children’s aesthetic preferences for picture books. international journal of education through art, 7(2), 171–185. https://doi.org/10.1386/eta.7.2.171_1   diamant-cohen, b., & hetrick, m. a. (2014). transforming preschool storytime: a modern vision and a year of programs. american library association neal-schuman.   eisenberg, m. b., & small, r. v. (1993). information-based education: an investigation of the nature and role of information attributes in education. information processing and management, 29(2), 263–275. https://doi.org/10.1016/0306-4573(93)90010-b   fehrenbach, l. a., hurford, d. p., fehrenbach, c. r., & brannock, r. g. (1998). developing the emergent literacy of preschool children through a library outreach program. journal of youth services, 12(1), 40-45.   fullerton, s. k., schafer, g. j., hubbard, k., mcclure, e. l., salley, l., & ross, r. (2018). considering quality and diversity: an analysis of read-aloud recommendations and rationales from children’s literature experts. new review of children's literature and librarianship, 24(1), 76–95. https://doi.org/10.1080/13614541.2018.1433473   giles, r. m., & fresne, j. (2015). musical stories infusing your read-alouds with music, movement, and sound. public libraries, 54(5), 31–34. http://publiclibrariesonline.org/2015/11/musical-stories-infusing-your-read-alouds-with-music-movement-and-sound/   ghoting, s., & martin-diaz, p. (2006). early literacy storytimes @ your library: partnering with caregivers for success. american library association.   goulding, a., dickie, j., & shuker, m. j. (2017). observing preschool storytime practices in aotearoa new zealand's urban public libraries. library & information science research, 39(3), 199–212. https://doi.org/10.1016/j.lisr.2017.07.005   guo, y., sawyer, b. e., justice, l. m., & kaderavek, j. n. (2013). quality of the literacy environment in inclusive early childhood special education classrooms. journal of early intervention, 35(1), 40–60. https://doi.org/10.1177/1053815113500343   horning, k. t. (2016, july 28). drilling down on diversity in picture books. ccblogc. https://ccblogc.blogspot.com/2016/07/drilling-down-on-diversity-in-picture.html    international literacy association. (2020). what’s hot in literacy report. https://www.literacyworldwide.org/docs/default-source/resource-documents/whatshotreport_2020_final.pdf   jackson, p., & moulinier, i. (2007). natural language processing for online applications: text retrieval, extraction and categorization. john benjamins.   khoir, s., du, j. t., davison, r. m., & koronios, a. (2017). contributing to social capital: an investigation of asian immigrants’ use of public library services. library & information science research, 39(1), 34-45. https://doi.org/10.1016/j.lisr.2017.01.005   kociubuk, k. j., & campana, k. (2019). sharing stories: an exploration of genres in storytimes. journal of librarianship and information science, 52(3), 905–915. https://doi.org/10.1177/0961000619882751   lennox, s. (2013). interactive read-alouds – an avenue for enhancing children's language for thinking and understanding: a review of recent research. early childhood education journal, 41(5), 381–389. https://doi.org/10.1007/s10643-013-0578-5   liu, b. (2015). sentiment analysis: mining opinions, sentiments, and emotions. cambridge university press.   markowsky, j. k. (1975). why anthropomorphism in children’s literature? elementary english, 52(4), 460-462, 466.   martens, p., martens, r., doyle, m. h., loomis, j., & aghalarov, s. (2012). learning from picturebooks: reading and writing multimodally in first grade. the reading teacher, 66(4), 285-294. https://doi.org/10.1002/trtr.01099   mesmer, h. a. (2018). books, read-alouds, and voluntary book interactions: what do we know about centers serving three-year-olds? literacy research and instruction, 57(2), 158–182. https://doi.org/10.1080/19388071.2017.1347220    miller, c., zickuhr, k., rainie, h., & purcell, k. (2013). parents, children, libraries, and reading. pew research center. https://www.pewresearch.org/internet/2013/05/01/parents-children-libraries-and-reading-3/   mills, j. e., campana, k., carlyle, a., kotrla, b., dresang, e. t., urban, i. b., capps, j. l., metoyer, c., feldman, e. n., brouwer, m, & burnett, k. (2018). early literacy in library storytimes, part 2: a quasi-experimental study and intervention with children’s storytime providers. library quarterly: information, community, policy, 88(2), 160–176. https://doi.org/10.1086/696581   neuman, s. b., kaefer, t., &, pinkham, a. m. (2016). improving low-income preschoolers’ word and word knowledge: the effects of content-rich instruction. the elementary school journal, 116(4), 652–674. https://doi.org/10.1086/686463   pentimoti, j. m., zucker, t. a., & justice, l. m. (2011). what are preschool teachers reading in their classrooms? reading psychology, 32(3), 197–236. https://doi.org/10.1080/02702711003604484   peterson, s. s. (2012). preschool early literacy programs in ontario public libraries. partnership: the canadian journal of library and information practice and research, 7(2), 1–21. https://doi.org/10.21083/partnership.v7i2.1961   price, l. h., bradley, b. a., & smith, j. m. (2012). a comparison of preschool teachers’ talk during storybook and information book read-alouds. early childhood research quarterly, 27(3), 426–440. https://doi.org/10.1016/j.ecresq.2012.02.003   schapira, r., & aram, d. (2019). shared book reading at home and preschoolers’ socio-emotional competence. early education and development, 31(6), 819–837. https://doi.org/10.1080/10409289.2019.1692624   serafini, f. (2012). expanding the four resources model: reading visual and multi-modal texts. pedagogies: an international journal, 7(2), 150–164. https://doi.org/10.1080/1554480x.2012.656347   short, k. g. (2018). what’s trending in children’s literature and why it matters. language arts, 95(5), 287–298. https://old.coe.arizona.edu/sites/default/files/pages/files/whats-trending-childrens-literature.pdf   sipe, l. r. (1998). how picture books work: a semiotically framed theory of text-picture relationships. children's literature in education, 29(2), 97–108. https://doi.org/10.1023/a:1022459009182   sipe, l. (2008). storytime: young children’s literary understanding in the classroom. teachers college press.   thoren, j. e. (2016). exploring preschool teachers’ perceptions regarding the barriers to selecting literature genres and utilizing extension activities: a qualitative multiple case study (publication no. 10240705) [doctoral dissertation, northcentral university]. proquest dissertations and theses global.   appendix a most frequent terms from the titles and abstracts corpus    rank term freq. percent rank term freq. percent 1 book 104 1.35% 43 full 19 0.25% 2 anim 90 1.17% 50 back 18 0.23% 3 love 48 0.62% 50 bed 18 0.23% 3 bear 48 0.62% 50 everyon 18 0.23% 3 find 48 0.62% 50 magic 18 0.23% 3 play 48 0.62% 50 garden 18 0.23% 7 young 45 0.58% 50 old 18 0.23% 8 color 42 0.54% 56 follow 17 0.22% 9 babi 39 0.51% 56 celebr 17 0.22% 10 children 38 0.49% 56 toe 17 0.22% 11 illustr 37 0.48% 56 just 17 0.22% 12 boy 36 0.47% 56 farm 17 0.22% 12 rhyme 36 0.47% 56 egg 17 0.22% 14 text 35 0.45% 56 board 17 0.22% 15 tri 33 0.43% 56 monster 17 0.22% 15 cat 33 0.43% 56 home 17 0.22% 17 stori 32 0.41% 65 surpris 16 0.21% 18 tree 31 0.40% 65 girl 16 0.21% 19 friend 30 0.39% 65 busi 16 0.21% 19 pictur 30 0.39% 65 learn 16 0.21% 21 dog 29 0.38% 65 shoe 16 0.21% 21 time 29 0.38% 65 featur 16 0.21% 23 reader 28 0.36% 71 child 15 0.19% 23 mani 28 0.36% 71 crayon 15 0.19% 23 big 28 0.36% 71 goe 15 0.19% 26 alphabet 25 0.32% 71 keep 15 0.19% 26 page 25 0.32% 71 kiss 15 0.19% 28 simpl 24 0.31% 71 beauti 15 0.19% 28 night 24 0.31% 71 pumpkin 15 0.19% 28 world 24 0.31% 71 stop 15 0.19% 28 hat 24 0.31% 71 give 15 0.19% 32 grow 23 0.30% 71 school 15 0.19% 32 mother 23 0.30% 71 snow 15 0.19% 32 includ 23 0.30% 82 crocodil 14 0.18% 35 ten 22 0.29% 82 count 14 0.18% 35 appl 22 0.29% 82 best 14 0.18% 37 perfect 21 0.27% 82 around 14 0.18% 37 danc 21 0.27% 82 creat 14 0.18% 37 red 21 0.27% 82 enjoy 14 0.18% 40 penguin 20 0.26% 82 share 14 0.18% 40 know 20 0.26% 82 everyth 14 0.18% 40 famili 20 0.26% 82 introduc 14 0.18% 43 fun 19 0.25% 82 spider 14 0.18% 43 water 19 0.25% 82 imagin 14 0.18% 43 read 19 0.25% 82 heart 14 0.18% 43 discov 19 0.25% 82 bath 14 0.18% 43 differ 19 0.25% 82 song 14 0.18% 43 mous 19 0.25% 82 duck 14 0.18%   appendix b most frequent terms from the subject terms   rank term freq. percent rank term freq. percent 1 fiction 945 25.95% 39 appl 10 0.27% 2 juvenil 488 13.40% 39 domest 10 0.27% 3 stori 126 3.46% 39 farm 10 0.27% 4 book 110 3.02% 39 play 10 0.27% 5 anim 99 2.72% 46 scienc 9 0.25% 6 rhyme 81 2.22% 46 rabbit 9 0.25% 7 literatur 60 1.65% 46 american 9 0.25% 8 children 55 1.51% 46 parent 9 0.25% 9 board 50 1.37% 46 garden 9 0.25% 10 pictori 42 1.15% 46 sound 9 0.25% 11 work 41 1.13% 46 friendship 9 0.25% 12 humor 27 0.74% 46 text 9 0.25% 13 toy 23 0.63% 46 snow 9 0.25% 13 specimen 23 0.63% 55 penguin 8 0.22% 15 pictur 22 0.60% 55 fictiti 8 0.22% 16 color 21 0.58% 55 monster 8 0.22% 17 english 19 0.52% 55 nonfict 8 0.22% 17 movabl 19 0.52% 55 duck 8 0.22% 19 dog 18 0.49% 55 father 8 0.22% 20 count 17 0.47% 61 hat 7 0.19% 20 bear 17 0.47% 61 moon 7 0.19% 22 subject 16 0.44% 61 individu 7 0.19% 22 child 16 0.44% 61 behavior 7 0.19% 24 folklor 15 0.41% 61 pirat 7 0.19% 24 cat 15 0.41% 61 son 7 0.19% 26 bedtim 14 0.38% 61 magic 7 0.19% 26 poetri 14 0.38% 68 train 6 0.16% 28 life 13 0.36% 68 read 6 0.16% 28 imagin 13 0.36% 68 autumn 6 0.16% 28 school 13 0.36% 68 pumpkin 6 0.16% 28 song 13 0.36% 68 bath 6 0.16% 32 famili 12 0.33% 68 butterfli 6 0.16% 32 natur 12 0.33% 68 dinosaur 6 0.16% 34 halloween 11 0.30% 68 social 6 0.16% 34 chicken 11 0.30% 68 cloth 6 0.16% 34 alphabet 11 0.30% 68 danc 6 0.16% 34 mother 11 0.30% 68 dress 6 0.16% 34 infanc 11 0.30% 68 easter 6 0.16% 39 charact 10 0.27% 68 state 6 0.16% 39 babi 10 0.27% 68 unit 6 0.16% 39 mice 10 0.27% 68 emot 6 0.16%   appendix c top bi-grams: titles and abstracts   rank bi-gram freq. rank bi-gram freq. 1 picture book 24 18 illustrations rhyming 5 2 rhyming text 17 18 letter alphabet 5 3 young boy 14 18 grades k-3 5 4 board book 11 18 york times 5 5 full color 10 18 one day 5 5 pete cat 10 18 new york 5 7 simple text 9 18 one one 5 8 young readers 8 30 past four 4 9 many things 7 30 times bestseller 4 9 little girl 7 30 stop kissing 4 11 young children 6 30 brown bear 4 11 ten little 6 30 letters alphabet 4 11 definitely wear 6 30 heart like 4 11 babba zarrah 6 30 like zoo 4 11 ice cream 6 30 spin web 4 11 minerva louise 6 30 text reveal 4 11 wear clothing 6 30 new way 4 18 apple pie 5 30 animals definitely 4 18 magic hat 5 30 book celebrates 4 18 white shoes 5 30 minutes past 4 18 board pages 5 30 old lady 4 18 gingerbread man 5 30 old macdonald 4   appendix d top bi-grams: subject terms   rank bi-gram freq. rank bi-gram freq. 1 juvenile fiction 412 33 fictitious character 8 2 stories rhyme 76 33 fiction dogs 8 3 juvenile literature 59 33 juvenile nonfiction 8 4 fiction stories 51 33 fiction domestic 8 5 board books 50 38 mother child 7 6 fiction animals 42 38 apples juvenile 7 7 pictorial works 41 38 fiction color 7 8 animals fiction 38 38 fiction family 7 9 works juvenile 34 38 fiction imagination 7 10 animals juvenile 27 38 fiction bears 7 11 humorous stories 26 38 child juvenile 7 12 books specimens 21 38 dogs juvenile 7 13 picture books 20 38 fiction mother 7 14 children's stories 19 38 rabbits fiction 7 14 toy movable 19 38 mice fiction 7 14 fiction humorous 19 38 sons fiction 7 14 movable books 19 38 life fiction 7 14 fiction board 19 38 imagination fiction 7 14 fiction juvenile 19 38 color fiction 7 20 books children 14 53 clothing dress 6 21 fiction children's 13 53 united states 6 22 bedtime fiction 12 53 parent child 6 23 animals infancy 11 53 baby books 6 23 fiction picture 11 53 infancy juvenile 6 23 bears fiction 11 53 cats juvenile 6 26 domestic animals 10 53 fiction schools 6 27 counting juvenile 9 53 fiction alphabet 6 27 fiction counting 9 53 fiction friendship 6 27 chickens fiction 9 53 fiction halloween 6 27 child fiction 9 53 fiction social 6 27 dogs fiction 9 53 pirates fiction 6 27 rhyme juvenile 9 53 play fiction 6 33 children's songs 8 53 books juvenile 6   122 evidence based library and information practice review article embedded academic librarianship: a review of the literature stephanie j. schulte assistant professor and education and reference services coordinator health sciences library the ohio state university columbus, ohio, united states of america email: schulte.109@osu.edu received: 4 june 2012 accepted: 4 oct. 2012 2012 schulte. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objectives – the purpose of this review is to examine the development of embedded librarianship, its multiple meanings, and activities in practice. the review will also report on published outcomes and future research needs of embedded librarian programs. methods – a search of current literature was conducted and summarized searching pubmed, cinahl, library, information science & technology abstracts (ebsco), academic search complete, and eric (ebsco) through august 23, 2012. articles were selected for inclusion in the review if they reported research findings related to embedded librarianship, if they provided unique case reports about embedded librarian programs, or if they provided substantive editorial comments on the topic. relevant study findings were assessed for quality and presented in tabular and narrative form. results – currently, there is disparity in how embedded librarianship is being defined and used in common practice, ranging from embedding an online component into a single course to full physical and cultural integration into an academic college or business unit of an organization. activities of embedded librarians include creating course integrated instruction modules for either face-to-face or online courses, providing in depth research assistance to students or faculty, and co-locating within colleges or customer units via office hours for a few hours to all hours per week. several case reports mailto:schulte.109@osu.edu 123 exist in the recent literature. few high quality research studies reporting outcomes of librarians or library programs labeled as embedded exist at this point. some evidence suggests that embedded librarians are effective with regards to student learning of information literacy objectives. surveys suggest that both students and faculty appreciate embedded librarian services. conclusion – most published accounts discuss librarians embedding content and ready access to services in an online course management system. a few notable cases describe the physical and cultural integration of librarians into the library user environs. future research using valid quantitative methods is needed to explore the impact of large scale, customized, embedded programs. introduction since the first mention of the phrase in the library literature in 2004 (dewey), embedded librarianship has received much attention. an entire double issue of public services quarterly was devoted to the topic in 2010 and acrl recently published a book on the topic (kvenild & calkins, 2011). kesselman and watstein (2009) published a narrative review of various ways librarians have been embedded, including course integrated instruction, participation in research teams, collaborations in scholarly communication initiatives, and physical location of librarians in academic departments. additionally, in 2009 the results of an in depth study of embedded librarianship were published as a result of funding from the special libraries association (shumaker & talley, 2009). to date, this is the largest and most comprehensive look at embedded librarianship and factors associated with successful programs. still, individual libraries may question whether to institute an embedded librarian program as they envision their futures and consider the evolving roles of librarians. decisions about future directions are further complicated by the polysemantic nature of embedded librarianship as evidenced by the variety of activities and degrees of embeddedness described in the literature. objectives the purpose of this review is to examine the development of embedded librarianship, its multiple meanings, and activities in practice. the review will also report on published outcomes and future research needs of embedded librarian programs. methods a literature search of pubmed, cinahl, library, information science & technology abstracts (ebsco), academic search complete, and eric (ebsco) was conducted using search terms of “embedded librarian,” “embedded librarianship,” embedded and librarian and (embedded or embed or embed*) and librarian*. a general search of the web using the google search engine was also conducted with similar terminology. the latest search was conducted august 23, 2012. articles were selected for inclusion in the review if they reported research findings related to embedded librarianship, if they provided unique case reports about embedded librarian programs, or if they provided substantive editorial comments on the topic. due to the large number of case reports on the topic, this review does not seek to be comprehensive in its presentation of them, but rather to provide a succinct awareness of current practices. 124 results history and definitions historically, embedded librarianship can be traced back to departmental libraries, where librarians provided services and collections within the confines of the department itself. some of these libraries still exist today, but many have been assimilated into main campus libraries. this has primarily been due to the expansion of digital content, competition for space for non-library uses, and economic factors associated with the costs of maintaining collections (drewes & hoffman, 2010). however, these same factors have served to isolate the librarian from his customer, causing librarians to explore ways to interact and integrate with those they serve. more recently, the expression ”embedded librarian” takes root in the phrase embedded journalist, a concept connected to wartime media coverage in the last several years. in this situation, journalists become a part of their military unit, providing a perspective, “a slice of the war” (“pros and cons of embedded journalism,” 2003) from their vantage point. drewes and hoffman (2010) provide a good discussion of the branch library concept and its connection to modern day embedded librarianship. brower (2011) also provides a concise recent history of the concept that offers an overview and characteristics of such programs. the topic of embedded librarianship can be somewhat difficult to define because of the wide range of approaches and interpretations presented in the literature. dewey (2004) first described it as “a more comprehensive integration of one group with another to the extent that the group seeking to integrate is experiencing and observing, as nearly as possible, the daily life of the primary group” and states how “overt purposefulness” is key to this “comprehensive collaboration” (p. 6). her paper discusses embedding librarians at a grand scale, weaving librarians throughout the fabric of academe. shumaker and talley (2009) considered embedded librarians as those “who provide specialized services within their organizations” (p. 4). by itself this seems too broad; however, they further describe these librarians by their most common activities. these included collaborating or contributing to the customer’s work or electronic workspace and attending meetings and conferences that were connected to the customer’s discipline. they also determined that librarians who were physically located with their customer groups and were funded to some degree by their customers did more of these activities. shumaker and talley use language that shapes embedded librarianship well, including that they “provide complex and value-added services” (p. 5). they report that these services are focused on the customer not the library, provided to small groups in their environment, go beyond discovery and delivery of information, and are built on trusted relationships in the context of the customer. kesselman and watstein (2009) agree with this, but broaden the concept such that “collaboration and integration” are important factors to consider. their stance includes models that could arguably be considered liaison models though not necessarily embedded as defined by dewey or shumaker and talley’s more restrictive descriptions. likewise, recent usage of the phrase proffers it as the latest trend, resulting in its use where formerly liaison programs or course integrated instruction would have been used. in fact, many liaison librarians already do many of the embedded librarian activities noted by shumaker and talley (2009). much like embedded journalists, many recent reports of embedded librarians note librarians becoming a literal part of academic colleges or departments, business units, or medical teams. recent literature references various degrees of time spent outside the library ranging from a few office hours (covone & lamm, 2010; matava, coffey, & kushkowski, 2010; matos, matsuoka-motley, & mayer, 2010), to more than 50 percent of time spent in the customer’s 125 environment (bartnik, 2007; brown & leith, 2007; fitzgerald, anderson, & kula, 2010; freiburger & kramer, 2009; martin, 2010). in the past, similar initiatives have been referred to as mobile or onsite reference, not embedded librarianship (lee, hayden, & macmillan, 2004; tao, mccarthy, krieger, & webb, 2009). other recent case reports of embedded librarianship refer to embedding librarians into a single course or research project (berdish & seeman, 2010; chestnut, wesley, & zai iii, 2010; konieczny, 2010; mcmillen & fabbi, 2010; muir & heller-ross, 2010). characteristics of these cases include creating or participating in online subject guides and discussion forums within the course management system (cms) housing the course. online embedding provides easy access to librarians throughout the course, whether students are nearby or not. some embedded librarians assume substantial teaching responsibilities (manus, 2009). other programs utilize the cms approach in addition to face-toface instruction and assistance (pritchard, 2010). whether embedded in an online or traditional course, these librarians’ efforts are similar to those of liaison librarians. shumaker and talley (2009) address these disparate definitions through their research methodology. in their initial survey, they found few distinctions between embedded and nonembedded librarians. many traditional librarians and embedded librarians were actually participating in similar activities. after discovering this, they began using the “specialized services within their organization” (p. 4) idea to distinguish what they considered to be truly embedded. common activities identified in the literature the shumaker and talley (2009) study identified many activities common to embedded librarians, some of which are used above to help define the concept. these same activities are also found in recent case reports and research studies. table 1 summarizes common activities with their associated case reports. the majority of case reports describe the creation and integration of online learning objects of various types (e.g. tutorials, guides, and links) within the course management system for specific courses. some describe office hours or some other co-location of librarians within a customer group. a few case reports describe purposeful integration of the librarian into the daily life of the customer to which dewey (2004) spoke and are worth further discussion here. bartnik (2007) describes her embedded librarian role in the college of business and public affairs at western kentucky university. her experience allowed her to locate herself within the school full time and build close relationships as an ad hoc faculty member. among her varied activities, she attended faculty meetings, assisted with publications, participated in interviews with faculty candidates, and provided in class and online instruction. she advocates for negotiating for office space in a high traffic area, remaining on the library’s payroll, and taking advantage of every meeting of the customer group. bartnik’s experience speaks to a grand potential of embedded librarianship to build close customer relationships given the right circumstances and personalities. in a later article, bartnik, along with her colleagues farmer, ireland, murray and robinson (2010), speaks of how new administrative duties took her away from her ideal embedded situation. though her embeddedness did not end entirely, the overall experience changed, including the loss of ad hoc faculty member privileges, an overall decrease in connectivity with faculty, and fewer research consultations. 126 table 1 common activities in embedded librarian case reports activity case reports embedded in course management system discussion board/forum participation links to library resources online subject guides bennett and simning (2010) chestnut et al. (2009) clark and chinburg (2010) covone and lamm (2010) hoffman and ramin (2010) kealey (2011) konieczny (2010) matava et al. (2010) matos et al. (2010) mcmillen and fabbi (2010) muir and heller-ross (2010) sullo, harrod, butera, and gomes (2012) collaboration on course design/assignments kealey (2011) manus (2009) mcmillen and fabbi (2010) muir and heller-ross (2010) pritchard (2010) co-teaching course (face-to-face or online) bartnik (2007) covone and lamm (2010) freiburger and kramer (2009) manus (2009) matos et al. (2010) muir and heller-ross (2010) pritchard (2010) in depth research to support student research bartnik (2007) berdish and seeman (2010) in depth research to support customer work grant applications research projects competitive intelligence bartnik (2007) fitzgerald et al. (2010) freiburger and kramer (2009) 127 physical co-location with customers office hours (partor fulltime) permanent office with customers bartnik (2007) brown and leith (2007) clyde and lee (2011) covone and lamm (2010) freiburger and kramer (2009) martin (2010) matava et al. (2010) matos et al. (2010) searing and greenlee (2011) embedded via social media filgo (2011) freiburger and kramer (2009) discuss several librarians at the arizona health sciences library who spend at least half their time within their liaison colleges and whose activities span more traditional liaison work to newer roles such as providing literature searches to support grant applications and serving as co-principal investigators on grant applications. martin (2010) adds spending 80% of her time in the pharmacy school at the same institution. they refer to their model as “liaison librarian in context” (p. 140). librarians at arizona health sciences library have the flexibility to customize embedded services based on customer needs. fitzgerald et al. (2010) describe an innovative embedded librarian program where librarians provide onsite market intelligence service to the non-profit mars discovery district, which assists entrepreneurs in canada. these librarians are partially funded by the mars group and spend the majority of their time with the group, but retain all their university of toronto connections, privileges, and responsibilities. the types of services provided range from the licensing of appropriate resources to in depth market analysis. this program is worth closer inspection for other reasons aside from the embedded aspect, including how they value their worth and how university librarians can work within a growing trend of universities partnering with non-profits to commercialize research. brown and leith (2007) describe a somewhat similar situation where they are embedded in an australian newsroom environment and support editorial functions of the media groups. both of these embedded librarian programs demonstrate the use of information expertise in partnership with customer expertise to market a product. in many ways, these programs illustrate non-traditional librarian roles while still maintaining the identity and duties of a traditional librarian. similarly, berdish and seeman (2010) describe an embedded librarian program focused on providing in depth research assistance to students in graduate business programs at the university of michigan. specifically, they provide assistance in an action based learning environment dubbed map: multidisciplinary action program. mba students in this program are doing extensive research on a real project for a real company. librarians in this program provide research help by being assigned to specific small groups and providing overviews and recommendations about which resources to use. each team arms itself with targeted information that can be used as they travel to work on their projects. 128 librarians’ willingness to investigate their customers’ needs and tailor services is common among these cases. they are not passive bystanders, but rather, proactive partners filling information gaps. though service oriented, the librarians are gaining professional reputations for excellent work and are highly valued partners. research studies very few quality research studies using the conceptual phrase “embedded librarian” exist. the shumaker and talley study is an exception to this; however, it does not report outcomes from individual embedded librarian interventions. comprehensive retrieval of research studies on the topic is more difficult due to the broad definition of embedded librarianship that pervades the current literature. some case reports and studies reference studies that allude to embedded librarians within the full text of the article but not within the title, the index terms or the abstract, such as the study by figa, bone, and macpherson (2009). additionally, many studies evaluating the effects of embedding information literacy instruction of any kind within a face-toface or online course could be considered relevant, since these activities are common to embedded librarian initiatives. for the purposes of this review, only studies that overtly considered the effects of embedded librarian initiatives were included (table 2). using these criteria, only seven quantitative studies were identified. this review also includes three qualitative studies reporting useful information, including one case study reporting methods of calculating value for services. of the first seven studies, significant heterogeneity was found. one study was a pretest/posttest study, one was a citation analysis, one was a comparison of scores on writing assignments, two were analyses of reference questions, one was a postimplementation survey by an embedded program, and one was the descriptive shumaker and talley (2009) study. descriptive study the shumaker and talley study (2009) used two surveys and site visits to define embedded librarianship and identify qualities of successful programs. the first survey was meant to identify those who were involved with embedded librarian programs within the population of special libraries association members. there were 961 employed respondents to this survey. the second survey was longer and sent to 234 embedded librarians (defined as providing specialized services with their customer group) from the first survey who indicated willingness to participate. of those, 130 responded. some findings of this study have been mentioned previously, and this review does not aim to give a comprehensive summary of the study. however, measures of success were identified and are pertinent to this review. they defined success in three ways: an increase in the number of librarians offering embedded services within a program, an increase in the customer’s demand of the services, and an increase in the number of services provided by the librarians to the customers. only 11 participants met these criteria. they then categorized 22 factors that separated the 11 participants who met all the criteria and the 16 respondents who met none of them into 4 categories: marketing and promotion, service evaluation, services provided, and management support. in general, successful programs publicized themselves in a variety of ways, measured outcomes in financial terms to justify their services, counted everything they did, provided complex research services and data analysis, and had written agreements between library administration and customer administration. the full report is lengthy, but worth the effort for those considering embedded programs. shumaker and talley draw attention to the changing nature of librarian service, from production of a list of results to evaluation and synthesis of relevant information. 129 pretest/posttest study edwards, kumar, and ochoa (2010) used a pretest/posttest questionnaire to measure student self efficacy and skills gained from embedding several librarian-created video modules on various research skills, and also gathered additional qualitative information. librarians were available online two hours per week in addition to participating in online discussion forums. this study had a low return rate on the preand posttests, with only 9 of 31 participants responding in the pretest and 7 of 31 responding in the posttest. self rated experience, comfort with resources and confidence in search increased slightly. the posttest also demonstrated more refined search techniques that were assumed to be the result of learning that occurred during the course. feedback gathered from five students’ responses to a request on a discussion board also indicated students learned research techniques. in an interview, the course’s faculty member expressed valuing the collaboration and asked to use the videos in other courses. the findings are relatively weak, measuring a small sample of students’ perceptions and skills in one course. however, they do hint at positive effects, if not of embedded librarians, of online video modules embedded into a course. comparison of writing assignment scores bowler and street (2008) evaluated intermediate level undergraduate writing assignments in five history and two women’s studies courses to gauge effectiveness of instruction with varying levels of librarian embedment. levels ranged from a single information literacy (il) session plus collaboration on the assignment to coteaching courses with information literacy either overtly taught by the expert (the librarian) or threaded throughout the course and taught by both the librarian and subject faculty. they used rubric-based scores to compare papers written early in the courses to final papers to measure improvement. generally, greater levels of librarian embedment resulted in greater improvement in writing assignments. the largest gains were seen when 5 librarians worked with problem based teams, noting an improvement of 21% in their problem based assignment. however, they also note that the cost of sustaining this level of engagement is prohibitive. comparison of two co-teaching methods suggested librarians “obviously and conspicuously” (p. 443) embedded throughout the term was more effective than seamless threading of il instruction. the authors noted a research score increase of 18% in the section where the librarian purposefully taught il versus just a 0.5% increase in the section with il threaded throughout. through an exit survey in one course, they also determined even though students generally felt neutral about the librarian’s assistance, their self rating of il skills improved at the end of the course. the methods of this study were reasonable in theory; however, the authors fail to report the sample (class) sizes and the timing of the librarian instructional interventions with relation to the first and last writing assignments. they also do not include their grading rubric. analysis of reference transactions two studies evaluated reference transaction data related to embedded librarian interventions. bennett and simning (2010) conducted correlation and linear regression calculations to show a positive relationship between the number of librarian interactions in an online course (discussion board comments) and the number of reference transactions in an online-only university setting. sullo et al. (2012) evaluated 82 reference questions encountered from 16 nursing and health sciences online courses with an embedded librarian component gathered in an approximate 16 month period. more than a third of questions were general research guidance questions, while another 22% were related to citation management, followed by 20% related to identifying, locating or using a library resource. as a result of these findings, librarians planned to embed resources within 130 the course management system so students did not have to find them on the library’s web page. citation analysis clark and chinburg (2010) used citation analysis to assess the effects of embedded librarians in two online sections and one face-to-face section of the same course. the embedded component of the course consisted of a tutorial (powerpoint with audio), links to resources, and participation in discussion forums as appropriate. no statistically significant differences were found between the online and face-to-face sections’ bibliographies when evaluated for the types and numbers of citations. the nature of the course required students to use many more trade and technical journal or website citations, and the authors suggested this could have affected the results. they emphasize that at a minimum, the study illustrates that online and face-to-face instruction produced similar student bibliographies in this course. post-implementation survey following the closing of the library and information science library at the university of illinois at urbana-champaign, several service changes were made, including placing an embedded librarian within the graduate school of library and information science (gslis) building several hours per week (searing & greenlee, 2011). one year after the change, faculty and staff at the library and the gslis were surveyed, and 105 responded. with regards to the embedded librarian, respondents indicated appreciation for their presence within the building but noted not using their services often. based on this and other responses, the embedded librarian made changes to the program. qualitative studies hoffman (2011) used a mixed methods approach, conducting a survey first to build interview questions and then phone interviewing embedded librarians from five institutions about their experiences in online courses. the interviews presented contrasting experiences related to workload. three librarians reported not being overly busy while two reported being quite busy. one librarian embedded in up to 35 sections of courses reported that this number of courses did not take a lot time. this may suggest limited engagement on either the part of the librarian or the students in the class, though hoffman did not speculate about this. both librarians who reported significant workload increases also graded assignments. this study also found that students had a positive response to online embedded librarians. hoffman noted the phrase embedded librarian is being used to describe both embedding in an online course and physical embedding in colleges or departments. kealey (2011) examined student learning in an online graduate epidemiology and evidence based practice course by using an embedded quiz in the course management system over three years. the quiz evolved somewhat over the three years; however, students consistently scored well. despite excellent quiz scores, closer inspection of quizzes revealed student difficulties with clinical queries and mesh in pubmed. based on yearly reflections, kealey revised her online video lecture and saw learning improvements. in an effort to measure value to their organization, fitzgerald et al. (2010) developed a “valuation formula.” the formula was value equals time spent (at can$200 per hour) divided by cost of the resources delivered. the cost per hour was selected because it fell between typical fee-based library services and consulting mbas charges. based on this formula, these librarians calculated that they had provided can$4.5 million of resources and can$480,000 of service in 2008. this represented ten times the investment in databases licensed for the mars group. 131 table 2 summary of embedded librarian research studies author study type population findings bowler and street (2008) comparison of writing products students in five intermediate level undergraduate history courses and two undergraduate women’s studies courses; sample size not given 18% improvement in research scores of writing assignment when librarian co-taught course and presented as expert in the class and taught il content. almost no improvement (1%) seen when content was threaded throughout with both subject faculty and librarian teaching but librarian was not presented as il expert. shumaker and tally (2009) descriptive: two surveys combined with site visits first survey: special libraries association membership (n=961) second survey: respondents identified from first survey as providing specialized services with their customer group (n=130 of 234 identified) first survey: demonstrated substantial overlap in duties of non-embedded librarians and embedded librarians. determined specialized services within their customer group as essential factor. second survey: measures of successful programs identified as 1) increase in number of librarians offering embedded services, 2) increase in demand for services and 3) increase in the number of librarians providing services. only 11 respondents met all 3 measures. successful programs marketed well, provided complex research and analysis services, and had written agreements between library and customer. suggested a move away from producing lists of results to providing analysis and synthesis. bennett and simning (2010) correlational study (using observation) psychology graduate students at an online university number of embedded librarian postings in online course statistically significantly correlated to number of reference transactions (r =0.491; p=0.010), indicating a moderate correlation. 132 clark and chinburg (2010) citation analysis research paper citations from 3 sections of a state university’s upperdivision undergraduate management information systems course receiving librarian instructional session; 1 section taught face-to-face (130 citations), 2 sessions taught online (247 citations). student citation patterns nearly identical despite instructional method. no statistical differences in the distribution or frequency of sources between the two instructional methods. edwards et al. (2010) pretest/posttest questionnaire; analysis of post-course feedback via discussion forum and faculty interview. 31 students in an online, 8 week foundations of educational technology course. low response rate on both pre(9/31) and posttest (7/21) surveys; some increases seen in students’ perceptions of experiences and comfort with databases, as well as actual skills demonstrated; statistical comparison between preand posttest surveys not reported. discussion forum comments were positive in nature. subject faculty found collaboration successful, asked to use content in other courses, felt connecting librarian content and assignment in advance was key. fitzgerald et al. (2010) case study customers from the mars discovery district, a non-profit entrepreneurial incubator working in collaboration with university of toronto developed valuation formula. value = time spent (at can$200/hour) cost of delivered resources the amount of services and resources provided by the librarians was ten times the cost of licenses resources. 133 hoffman (2011) qualitative mixed methods: survey, phone interviews seven librarians from six institutions; reports information from interviews representing five of the six institutions time investment of embedded librarians varied; may be related to level and amount of teaching/grading responsibilities rather than number of courses alone. students tended to report positive experiences. kealey (2011) qualitative reflection of online quizzes physician assistant students in a required online graduate level epidemiology and evidence based medicine course over three years (45 students in year 1; 52 students in year 2, 53 students in year 3). librarian-led online modules included screen capture videos for instructional purposes and were modified as necessary. high averages for the 5-point and 10-point assessments used in the course: 4.94-4.98 out of 5 and 9.0-9.17 out of 10, respectively. analysis of actual responses revealed student weaknesses in understanding clinical queries and the use of medical subject headings (mesh) in medline. searing and greenlee (2011) survey, case study faculty and staff at the university of illinois at urbanachampaign university library and graduate school of library and information science (105 respondents) survey addressed many areas not directly related to an embedded librarian. with regards to a new embedded librarian model, gslis faculty noted enjoying easy access to librarian, but missed the physical collection they formerly had. respondents split opinions on the advantages of the new model, with several noting no benefits while several noted no drawbacks. sullo et al. (2012) analysis of reference transactions 82 reference transactions from discussion boards and emails of online courses with embedded librarian classification of questions: 34% general research guidance; 22% citation questions; 20% using library resources; 10% off campus access. 134 overall, there is a lack of formal, systematic processes to quantify outcomes demonstrating embedded librarian impact. only two studies analyzed artifacts of learning and another two studies attempted to directly measure practical skills through free responses or quizzes. no study evaluated an embedded librarian who was physically and culturally integrated into an academic or business unit. despite this, results suggest that librarians embedded in online and face-to-face course settings have positive effects on student learning. discussion this review aimed to examine the development of embedded librarianship, its multiple meanings and activities in practice, and report on published outcomes. librarians have sought to engage their customers – faculty, staff, students, business units, and the public – in a variety of ways for years. this review found that embedded librarianship is another attempt to do just that: engage. current literature illustrates that the phrase embedded librarianship is widely applied and could mean anything from having an online presence in a course to wholly working amid the end user group. commonly, embedded librarians are providing learning objects and a presence within the online course management system. while there are plentiful case reports describing embedded librarian work, there are few notable published cases where librarians have truly become part of “the daily life of the primary group” as dewey suggests. the notable cases of bartnik et al. (2010), freiburger and kramer (2009), fitzgerald et al. (2010), brown and leith (2007), and berdish and seeman (2010) provide rich examinations of experiences reaching out to library users, building close relationships, and customizing services. unfortunately, there are no high quality studies evaluating the effectiveness or value of embedded librarian programs like these. most research that does exist has sought to evaluate impact of instructional initiatives, primarily where librarians are embedded in online or face-to-face courses. these evaluations are similar to those seen in the liaison or instruction librarian literature. schilling and applegate (2012) note that “without access to individual learners and artifacts, rigorous research methodologies cannot be implemented” (p. 261). the very nature of embedded librarianship supports this level of access to both learners and artifacts. the studies in this review attempted to evaluate educational impact of embedded librarianship by using artifacts that were convenient to them and by reflecting on their experiences, yet rigorous methods were generally not applied. embedded librarians who are physically and culturally integrated within their customers are akin to clinical librarians. brettle et al. (2010) suggested clinical librarians utilize the critical incident technique (cit) to connect their work to important customer outcomes, which may also be appropriate for programs embedded to a lesser degree. cit studies can be conducted by collecting in depth customer stories about positive and negative incidents or by presenting specific critical incidents followed by questions about their perceptions and behaviors following the incidents (radford, 2006). while the stories and perceptions of customers may have enough influence on some institutional stakeholders, cit is primarily a qualitative methodology. as such, future cit research on embedded librarianship could provide a basis for forming hypotheses that could be systematically and quantitatively studied. clearly, future research is needed. as librarianship evolves from the production of lists of resources to the evaluation and synthesis of information, as noted by shumaker and tally (2009), questions arise regarding the time investment of individual librarians. future research should explore how diverse, customized, embedded programs can realistically be evaluated using valid methods. findings from in-progress research regarding library and librarian value may be beneficial to 135 inform future embedded librarian research. remaining relevant to customers is vital to the survival of librarianship. embedded librarianship, in its many forms and degrees of embeddedness, may prove to enhance the relevancy of librarians in the digital world. limitations of this review include inclusion of only studies that overtly used the concept of embedded librarianship to describe themselves. in doing so, many studies reporting outcomes related to integrating il instruction in a variety of ways were not included. for programs focused solely on embedding instruction online or in the classroom, these studies would hold great relevance and should be considered. additionally, the review did not include any programmatic evaluations of liaison librarian programs, though the author is not aware of any high quality reports. because individual embedded librarian programs can and do vary greatly, the review may not be applicable to all situations. conclusion a review of literature on the concept of embedded librarianship revealed multiple usages. most published accounts discuss librarians embedding content and ready access to services in the online course management system. a few notable cases describe the physical and cultural integration of librarians into the customer environs. no rigorous reports of outcomes to evaluate impact of embedded librarianship were found. some reports suggest that embedding librarians in online or face-toface courses has positive impact on student learning. future research using valid quantitative methods is needed to explore the impact of large scale, customized, embedded programs. references bartnik, l. (2007). the embedded academic librarian: the subject specialist moves into the discipline college. kentucky libraries, 71(3), 4-9. bartnik, l., farmer, k., ireland, a., murray, l., & robinson, j. (2010). we will be assimilated: five experiences in embedded librarianship. public services quarterly, 6(23), 150-164. doi:10.1080/15228959.2010.498772 bennett, e., & simning, j. (2010). embedded librarians and reference traffic: a quantitative analysis. journal of library administration, 50(5/6), 443-457. doi:10.1080/01930826.2010.491437 berdish, l., & seeman, c. (2010). a referenceintensive embedded librarian program: kresge business administration library's program to support action-based learning at the ross school of business. public services quarterly, 6(2-3), 208-224. doi:10.1080/15228959.2010.497462 bowler, m., & street, k. (2008). investigating the efficacy of embedment: experiments in information literacy integration. reference services review, 36(4), 438-449. doi:10.1108/00907320810920397 brettle, a., maden-jenkins, m., anderson, l., mcnally, r., pratchett, t., tancock, j., thornton, d. & webb, a. (2011). evaluating clinical librarian services: a systematic review. health information & libraries journal, 28(1), 3-22. doi:10.1111/j.14711842.2010.00925.x brower, m. (2011). a recent history of embedded librarianship: collaboration and partnership building with academics in learning and research environments. in c. kvenild & k. calkins (eds.), embedded 136 librarians: moving beyond one-shot instruction (pp. 3-16). chicago: association of college and research libraries. brown, d., & leith, d. (2007). integration of the research library service into the editorial process. aslib proceedings, 59(6), 539-549. doi:10.1108/00012530710839614 chestnut, m. t., wesley, t., & zai iii, r. (2009). adding an extra helping of service when you already have a full plate: building an embedded librarian program. public services quarterly 6( 2–3 ), 122-129. doi:10.1080/15228959.2010.498780 clark, s., & chinburg, s. (2010). research performance in undergraduates receiving face to face versus online library instruction: a citation analysis. journal of library administration, 50(5/6), 530-542. doi:10.1080/01930826.2010.488599 clyde, j., & lee, j. (2011). embedded reference to embedded librarianship: 6 years at the university of calgary. journal of library administration, 51(4), 389-402. doi:10.1080/01930826.2011.556963 covone, n., & lamm, m. (2010). just be there: campus, department, classroom . . . and kitchen? public services quarterly, 6(2-3), 198-207. doi:10.1080/15228959.2010.498768 dewey, b. i. (2004). the embedded librarian: strategic campus collaborations. resource sharing & information networks, 17(1/2), 517. doi:10.1300/j121v17n01_02 drewes, k., & hoffman, n. (2010). academic embedded librarianship: an introduction. public services quarterly, 6(2-3), 75-82. doi:10.1080/15228959.2010.498773 edwards, m., kumar, s., & ochoa, m. (2010). assessing the value of embedded librarians in an online graduate educational technology course. public services quarterly, 6(2), 271-291. doi:10.1080/15228959.2010.497447 figa, e., bone, t., & macpherson, j.r. (2009). faculty-librarian collaboration for library services in the online classroom: student evaluation results and recommended practices for implementation. journal of library & information services in distance learning, 3(2), 67-102. doi:10.1080/15332900902979119 filgo, e. h. (2011). #hashtag librarian: embedding myself into a class via twitter and blogs. computers in libraries, 31(6), 7880. fitzgerald, k., anderson, l., & kula, h. (2010). embedded librarians promote an innovation agenda: university of toronto libraries and the mars discovery district. journal of business & finance librarianship, 15(3/4), 188-196. doi:10.1080/08963568.2010.487689 freiburger, g., & kramer, s. (2009). embedded librarians: one library's model for decentralized service. journal of the medical library association, 97(2), 139-142. doi:10.3163/1536-5050.97.2.013 hoffman, s. (2011). embedded academic librarian experiences in online courses: roles, faculty collaboration, and opinion. library management, 32(6/7), 444-456. doi:10.1108/01435121111158583 hoffman, s., & ramin, l. (2010). best practices for librarians embedded in online courses. public services quarterly, 6(2-3), 292-305. doi:10.1080/15228959.2010.497743 kealey, s. (2011). continual evolution: the experience over three semesters of a librarian embedded in an online evidencebased medicine course for physician 137 assistant students. medical reference services quarterly, 30(4), 411-425. doi:10.1080/02763869.2011.609046 kesselman, m. a., & watstein, s. b. (2009). creating opportunities:embedded librarians. journal of library administration, 49(4), 383-400. doi:10.1080/01930820902832538 konieczny, a. (2010). experiences as an embedded librarian in online courses. medical reference services quarterly, 29(1), 47-57. doi:10.1080/02763860903485084 kvenild, c., & calkins, k. (eds.). (2011). embedded librarians: moving beyond one-shot instruction. chicago, il: association of college and research libraries. lee, j., hayden, k. a., & macmillan, d. (2004). “i wouldn't have asked for help if i had to go to the library": reference services on site. issues in science and technology librarianship 41. retrieved 12 oct. 2012 from http://www.istl.org/04fall/article2.html manus, s. j. b. (2009). librarian in the classroom: an embedded approach to music information literacy for first-year undergraduates. notes 66(2), 249-261.doi: 10.1353/not.0.0259 martin, j. r. (2010). the newly embedded librarian: pharmacy information liaison service. public services quarterly, 6(2-3), 187-197. doi:10.1080/15228959.2010.497908 matava, t., coffey, d., & kushkowski, j. (2010). beyond library walls: embedding librarians in academic departments. public services quarterly, 6(2-3), 165-173. doi:10.1080/15228959.2010.497835 matos, m. a., matsuoka-motley, n., & mayer, w. (2010). the embedded librarian online or face-to-face: american university's experiences. public services quarterly, 6(23), 130-139. doi:10.1080/15228959.2010.497907 mcmillen, p., & fabbi, j. (2010). how to be an e3 librarian. public services quarterly, 6(2-3), 174-186. doi:10.1080/15228959.2010.497454 muir, g., & heller-ross, h. (2010). is embedded librarianship right for your institution? public services quarterly, 6(2-3), 92-109. doi:10.1080/15228959.2010.497464 pritchard, p. a. (2010). the embedded science librarian: partner in curriculum design and delivery. journal of library administration, 50(4), 373-396. doi:10.1080/01930821003667054 pros and cons of embedded journalism (27 mar. 2003). in newshour extra. retrieved 12 oct. 2012 from http://www.pbs.org/newshour/extra/featur es/jan-june03/embed_3-27.html radford, m. l. (2006). the critical incident technique and the qualitative evaluation of the connecting libraries and schools project. library trends 55(1): 46-64. schilling, k., & applegate, r. (2012). best methods for evaluating educational impact: a comparison of the efficacy of commonly used measures of library instruction. journal of the medical library association, 100(4), 258-269. doi: 10.3163/15365050.100.4.007 searing, s. e., & greenlee, a. m. (2011). faculty responses to library service innovations: a case study. journal of education for library & information science, 52(4), 279-294. shumaker, d., & talley, m. (2009). models of embedded librarianship: final report. in special libraries association. retrieved 12 138 oct. 2012 from http://www.sla.org/pdfs/embeddedlibrari anshipfinalrptrev.pdf sullo, e., harrod, t., butera, g., & gomes, a. (2012). rethinking library service to distance education students: analyzing the embedded librarian model. medical reference services quarterly, 31(1), 25-33. doi:10.1080/02763869.2012.641822 tao, d., mccarthy, p., krieger, m., , & webb, a. (2009). the mobile reference service: a case study of an onsite reference service program at the school of public health. journal of the medical library association, 97(1), 34-40. doi:10.3163/15365050.97.1.00 editorial evidence based library and information practice 2009, 4:2 155 evidence based library and information practice evidence summary e-mailed evidence based summaries impact physician learning more than practice a review of: grad, roland m., pierre pluye, jay mercer, bernard marlow, marie eve beauchamp, michael shulha, janique johnson-lafleur and sharon wood-dauphinee. “impact of researchbased synopses delivered as daily e-mail: a prospective observational study.” journal of the american medical informatics association (2008)15.2: 240-5. reviewed by: gale g. hannigan professor texas a&m medical sciences library college station, texas, united states of america e-mail: g-hannigan@tamu.edu received: 02 december 2008 accepted : 18 april 2009 © 2009 hannigan. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. objective – to determine the use and construct validity of a method to assess the cognitive impact of information derived from daily e-mail evidence based summaries (infopoems), and to describe the self-reported impact of these infopoems. design – prospective, observational study over a period of 150 days employing a questionnaire and rating scale. setting – this study was conducted via the internet between september 8, 2006 and february 4, 2007. subjects – canadian medical association (cma) members who received infopoems via e-mail as of september 2006 were invited to participate. for inclusion in the analyses, a participant was defined as a practising family physician or general practitioner who submitted at least five ratings of infopoems during the study period (n=1,007). methods – volunteers completed a demographic questionnaire and provided informed consent online. each subsequent infopoem delivered included a link to a “ten-item impact assessment scale” (241). participants checked “all that apply” of descriptive statements such as: my practice mailto:g-hannigan@tamu.edu http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2009, 4:2 156 was (will be improved); i learned something new; i think this information is potentially harmful. each combination of selections made was considered a pattern of cognitive impact. college of family physicians of canada received continuing medical education (cme) credit for each infopoem rated. data were collected by the cma and forwarded weekly to the investigators who used descriptive statistics, principal component analysis, and multilevel factor analysis to analyze the data. main results – 1,007 participants rated an average of 61 infopoems (ranging from five to 111). a total of 61,493 patterns of cognitive impact were submitted. eightyfive different patterns were observed, i.e., there were 85 different combinations of the scale’s statements used. ten patterns accounted for 89.4% of the reports. the top five patterns were: i learned something new (35.2%); no impact (17.1%); this information confirmed i did (will do) the right thing (9.6%); i learned something new and my practice will be improved (9.4%); and, i was reassured (5.6%). i disagree with this information was checked at least once by 10.3% of the participants, and 8.0% checked i think this information is potentially harmful at least once. conclusion – the authors applied a cognitive assessment instrument to determine the impact of infopoems emailed to primary care physicians in canada and found that ten combinations of impact descriptors accounted for 89.4% of the total reports. most suggested a positive impact on knowledge or practice. of the total, 17.1% indicated no impact and 1.8% indicated the participant was frustrated as there was not enough information or nothing useful. commentary evidence based summaries are a new type of literature designed to assist physicians in their efforts to incorporate scientifically based information into patient care decision making. there needs to be good research into the impact of these summaries on clinical practice, and the authors are to be commended for taking this on and developing and testing an assessment tool. this study would have benefited from a study design flow chart. the response rate of the target population for the study cannot be calculated. as stated under methods, “all cma physicians who received infopoems via e-mail as of september 2006 were eligible to participate” (241). the authors do not provide that number, but do write that 12,800 cma members had started receiving infopoems by e-mail in 2005. a total of 1,007 physicians were counted as participants in the study, defined as having submitted at least five ratings. the number of physicians who originally agreed to participate but did not complete at least five ratings was not reported, either. this number would have been informative as a measure of use. did those physicians use the assessment less than five times because of lack of time or did many of them indicate frustrated…nothing useful or no impact on a few and then quit participating? knowing the reason for low use of the instrument by people who were initially interested seems key to the purpose of the study. the authors did compare the participants (those with five or more ratings) with canadian family physicians surveyed in 2004 and noted that “participants were more likely to report utilization of electronic reminder or warning systems in their practice” (241). very possibly, the study’s convenience sample was biased toward physicians more interested in using electronic updates and completing electronic assessments. also, family physicians who completed ratings received cme credits, which might indicate selection bias in favour of those who prefer this form of cme or those who participated evidence based library and information practice 2009, 4:2 157 primarily for the cme credit. the authors admit that “in the absence of cme credit, use of the method may be less impressive” (244). the study’s sample may or may not be a good indicator of use of evidence based summaries and the assessment tool; the results about impact can only be generalized to those who would use e-mail updates and the assessment tool five or more times. the authors acknowledge the limitations of self-report in determining changes in practice, i.e., did the physician who checked my practice will be improved actually experience an improvement in practice? readers may wish they had more explanation about the assessment tool itself. for example, why is it that the item no impact “could not be selected simultaneously with any other item” (241). couldn’t no impact be combined with i was frustrated as there was not enough information or nothing useful? that said, the authors are honing a set of descriptors for assessing the impact of evidence-based summaries and, perhaps, other clinical information. a major point the authors make relates to the value of “just in case” versus “just in time” information. as they point out, emailed summaries are a “push” communication, not targeted to individuals who might pay more attention to information that is personally relevant. databases, and specifically, point-of-care resources that enable one to search for evidence-based summaries at the time of need would most likely result in “greater attention to the message” (244). it is not surprising, then, that physicians reported that they learned something new more often than they reported that the e-mailed information changed practice. “pushed” as e-mails, evidence based summaries may be more sources of general knowledge than decision-support tools. there is value there, too. reading e-mailed evidence based summaries may be a more efficient lifelong learning strategy than browsing issues of current journals to keep up with the literature. as evidence based summaries proliferate and are shown, with studies such as this one, to be useful in and have an impact on clinicians’ knowledge – general or specific – perhaps medline will begin indexing these published summaries as a new publication type and even link them to the original article(s). that way, busy physicians (and librarians) using medline can find them more easily when patientspecific questions arise, too. research such as this can provide a valuable measurement tool for clinical librarians and others. evidence based library and information practice 2012, 7.4 102 evidence based library and information practice evidence summary traditional factors of fit, perceived quality, and speed of publication still outweigh open access in authors’ journal selection criteria a review of: solomon, d. j., & björk, b.-c. (2012). publication fees in open access publishing: sources of funding and factors influencing choice of journal. journal of the american society for information science and technology, 63(1), 98-107. doi: 10.1002/asi.21660 reviewed by: michelle dalton assistant librarian, hse mid-west library & information services university of limerick limerick, ireland email: michelle.dalton@ul.ie received: 16 aug. 2012 accepted: 9 oct. 2012 2012 dalton. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to determine the extent to which the open access (oa) status of a journal influences authors in their journal selection decisions and to analyze the sources of funding for the article-processing charges (apcs) applied in professional oa publishing. design – survey questionnaire. setting – the international open access scholarly publishing sector. subjects – 1,038 researchers across all academic disciplines who have recently published work in open access journals that charge apcs. methods – journals listed in the directory of open access journals were stratified into seven discipline clusters, and systematic random sampling was used where possible to collect a sample of up to 15 journals per cluster that levy apcs. for each individual journal, the authors of the 15 most recently published articles (working from 2010 backwards) were invited to complete a web-based questionnaire on the factors influencing their choice of journal and the source(s) used to fund processing charges. additional background information about the authors and journals was also collected and merged with the survey responses. mailto:michelle.dalton@ul.ie http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/ evidence based library and information practice 2012, 7.4 103 main results – the results of the survey identified the fit of the article with the journal’s subject area, the perceived quality or impact of the journal, and the speed of the peer-review and publishing process as the dominant factors in the journal selection decision of authors. all three aspects were judged as either “very important” or “important” by 80% or more of respondents – significantly higher than the corresponding figure of 60% in relation to the open access status of the journal. the analysis also indicated that two key elements appear to influence how apcs are funded: the research discipline and the country of origin of the author. the use of research grants to fund charges is more prevalent in scientific disciplines than in the humanities, whilst researchers based in lower-income countries more frequently identify apcs as a barrier than those in higher-income countries. grants and institutional funding tend to be the primary sources of funding for journals with higher apcs, whilst personal funding is utilised more often in cases where the fee is less than $500. conclusion – despite the increasing focus on the accessibility and visibility of research, academics still appear to place a greater value on ‘who’ rather than ‘how many’ readers access their research, and consequently traditional factors still persist as the main determinants in an author’s choice of journal. the future success of the apc model, compared with the traditional subscriptionbased or hybrid models, will ultimately depend on the ability of authors to obtain the necessary funding to pay such charges, combined with the extent to which the quality of services offered by open access publishers is perceived as being commensurate with the associated publishing fees. commentary the study deals with two emerging themes in scholarly publishing: how authors typically evaluate and select journals, and how scholars perceive the importance of a journal’s open access policy when submitting manuscripts. these results lend resonance to the view that journal rankings and impact factors (often used as a proxy for quality) remain highly influential in the scholarly publishing environment. while the intention of the study was to collect a representative sample from all disciplines, the difficulty encountered in obtaining sufficiently large samples outside the science technical and medical field, where apcs are less prevalent, highlights possible limitations in terms of the external validity and replicability of the results. this problem in itself flags opportunities for further research into why there is such a discrepancy in apc policies across disciplines, as well as the use of the apc as a proxy for the willingness to pay for publishing services, particularly with a view to estimating which services are valued most: visibility and dissemination, the peerreview process, reputation, or branding. the authors state that the survey was piloted before distribution with 123 authors across 4 journals, but that no subsequent changes were made to the instrument. however, the study acknowledges the lack of clarity in phrasing one of the questions (concerning the maximum charge authors would be willing to pay), which led to some respondents misinterpreting the intended context. this casts doubt over how effective this pilot testing actually was, as well as the consistency and reliability of the questionnaire. a more rigourous approach in developing and testing the survey instrument may have anticipated such problems and yielded more precise answers, thus increasing the validity of the survey. the concern that apcs are a more pervasive barrier for authors in lower-income countries and those working in certain disciplines where authors often pay charges from personal funds may ultimately lead to a bias in the volume or geographic distribution of the research published in open access journals if the apc model gains further traction. the overall awareness and recognition of these fees should ideally be made more explicit at funding agency and institutional levels, with more evidence based library and information practice 2012, 7.4 104 visible supports put in place by both publishers and institutions for those working in areas where processing charges act as a prohibitive barrier to publishing in oa journals. from the evidence presented, it appears that librarians still have an essential role to play in promoting the benefits of open access publishing to researchers, with almost 20% of authors indicating that the openness of a journal is of little or no influence when targeting a publication. furthermore, as the authors included in the sample exclusively comprise those recently published in oa journals rather than those published in both oa and subscription journals, this may be indicative, ceteris paribus, that the figure across researchers as a whole is potentially even higher. while the degree of openness may not currently be of intrinsic importance to authors, it is clear that those factors which are judged as critical are not exogenously determined. for instance, open access channels may increase the potential fit of an article, as niche publishing becomes more feasible in the context of a zero marginal cost model that is not dependent on a high volume of reader subscriptions to fund it. furthermore, a faster review and publication process is also an advantage offered by several open access journal publishers. promoting and highlighting these endogenous relationships to researchers could prove to be a valuable tool for librarians in further leveraging the support of authors for open access publishing. references boynton, p. m., & greenhalgh, t. (2004). hands-on guide to questionnaire research: selecting, designing, and developing your questionnaire. bmj, 328, 1312. doi: http://dx.doi.org/10.1136/bmj.328.7451. 1312 evidence summary   libraries assist disaster survivors with information needs and refuge but need to amplify their role and what they offer   a review of: braquet, d. m. (2010). library experiences of hurricane katrina and new orleans flood survivors. libres: library and information science research electronic journal, 20(1), 1. https://www.libres-ejournal.info/528/   reviewed by: samantha j. kaplan research & education librarian, liaison to the school of medicine duke university medical center library & archives durham, north carolina, united states of america email: samantha.kaplan@duke.edu   received: 28 jan. 2021                                                               accepted:  8 apr. 2021      2021 kaplan. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29938     abstract   objective – describe the experiences and library usage of patrons displaced by hurricane katrina and the new orleans flood.   design – a qualitative study with interview components and a questionnaire with open and closed-end questions   setting – new orleans, louisiana and surrounding area   subjects – 314 questionnaire respondents and 30 interview (24 face-to-face and 6 phone) participants with 5 individuals completing both   methods – the study consisted of an online questionnaire with open and closed-end questions occurring concurrently with semi-structured interviews conducted over the phone and in person. individuals were recruited via convenience sample by flyers at public locales in the new orleans area and electronic mailing lists, forums, blogs, and news sites that catered to the new orleans community.   main results – disaster survivors use libraries for internet access, information and technology assistance, mental relief, physical refuge, and also view them as symbols of both loss and hope. library resources (including the physical spaces) allowed survivors to regain a sense of control by helping patrons access local information and experience pre-disaster pastimes, such as leisure reading.   conclusion – the study provides rich description of how libraries can support people displaced by disaster, however just over half of participants did not consider the library a part of their disaster experience. future research should examine how libraries and library workers can amplify their impact during disasters and disaster recovery, as well as partner with disaster planning and response professionals.    commentary   libraries and library workers have often served patrons and their communities at critical junctures. this includes, but is in no way limited to, events such as the ferguson, mo protests (peet, 2015), providing ongoing support during the covid-19 pandemic (ali & gatiti, 2020), and natural disasters, which are the focus of this article.   this research did include some quantitative data but the findings were thematic and descriptive in nature, so it was appraised with the critical review form for qualitative studies (letts et al. 2007). while the author clearly articulated the need for the study, unreported methodological components undermine the rigor of the study. there was no mention of a theoretical or philosophical frame for the study and minimal description of how interview data was coded. the author also makes no acknowledgment of their assumptions or biases and how this could have shaped the data.   the known demographic data of the interviewees and survey respondents, by the author's own acknowledgment, is not representative of new orleans. further, 14 of 30 participants did not complete the demographic questionnaire sent to interviewees, creating a significant missing data problem, as from the tables we can determine that primarily white people with some higher education (community college, 4-year degree, graduate school) provided this information. this missing data issue was also observed, though to a lesser degree, in the survey respondent demographic tables. the survey was only distributed via internet, likely missing important perspectives and experiences of those without or with limited internet access. combined with all data being gathered one year after displacement, the author is wise to counsel against demographic generalizations or conclusions. library workers should contemplate the missing perspectives before considering how to apply the findings to their own setting.   despite these issues, this is a very valuable study as it centers the experience of the patron and disaster survivor, making them an agent and active participant rather than a passive recipient of the library's beneficence. the study's credibility, which would have been improved with an interview guide or list of interview questions, is strong as all of the themes are illustrated with participants' words. the manuscript would have benefitted greatly from the inclusion of the survey questions as well. without seeing the full survey, it is difficult to contextualize the limited findings reported.   the author's findings, while written over a decade ago, are underscored by the ongoing covid-19 pandemic which has highlighted the importance of the key themes identified by the author – information and technology assistance, internet access, and reading to escape. as flaherty (2016) asserted, library involvement in disaster planning "can pay dividends in the long run in terms of community stability, resiliency and recovery" (p. 3).   references   ali, m. y., & gatiti, p. (2020). the covid‐19 (coronavirus) pandemic: reflections on the roles of librarians and information professionals. health information & libraries journal, 37(2), 158-162. https://doi.org/10.1111/hir.12307   flaherty, m. g. (2016, august 15). here, there and everywhere: disasters and public libraries. in a sanctuary in times of need: the public library response [conference paper] ifla / wlic 2016, columbus, oh, united states. http://library.ifla.org/id/eprint/1375   letts, l., wilkins, s., law, m., stewart, d., bosch, j., & westmorland, m. (2007). critical review form: qualitative studies (version 2.0). http://www.peelregion.ca/health/library/eidmtools/qualreview_version2_0.pdf   peet, l. (2015). ferguson library: a community's refuge. library journal, 140(1), 12.   evidence based library and information practice evidence based library and information practice 2011, 6.4 180 evidence based library and information practice news/announcements evidence-based library and information practice network 2011. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. the canadian library association (cla) announces a new network. the evidencebased library and information practice network (eblipn), previously an interest group, was recently approved by the cla executive council. networks are established by and for cla members, and allow grass roots action in a particular focus area broadly related to library and information practice. the goals of eblipn are: 1. to represent the interests of librarians involved in evidence-based librarianship (ebl) and library related research. 2. to organize continuing education opportunities for librarians in this area. 3. to provide a means of communication between librarians involved in evidence-based librarianship. 4. to assist librarians with the dissemination of library research utilizing an evidence-based model. eblipn has an active email list which offers members the opportunity to communicate, share ideas, and disseminate knowledge regarding ebl and library related research. the list is a great way to keep current on ebl related events, workshops, sessions, and conferences. the network also plans an eblrelated wiki and a research award. eblipn is currently seeking members. if you are interested please contact the cla office at membership@cla.ca and keep watching for the new eblipn webpage. or you may contact network moderators at the addresses below. moderators: trina fyfe (fyfet@unbc.ca) and alison farrell (alisonr@mun.ca) mailto:membership@cla.ca evidence based library and information practice 70 evidence based library and information practice article navigating user feedback channels to chart an evidence based course for library redesign t. derek halling onsite services librarian texas a&m university medical sciences library college station, texas, united states of america email: dhalling@library.tamu.edu esther carrigan associate dean and director texas a&m university medical sciences library college station, texas, united states of america email: ecarrigan@library.tamu.edu received: 19 sept. 2011 accepted: 23 jan. 2012 2012 halling and carrigan. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/bync-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objectives – the objective of this project was to redesign library spaces based on the user feedback obtained from a broad complement of feedback channels. the overarching goal of this project was to develop an evidence based approach to the redesign of library spaces. methods – data from user-initiated and library-initiated feedback channels were collected and analyzed to determine priorities for library space changes. online/onsite suggestions, a library onsite census survey, the libqual+® survey, a whiteboard, ballot voting, and text voting were all used to gather input. a student advisory group was used as a sounding board for planned space changes before a final decision was made. results – data produced by different feedback channels varied both in the number of suggestions generated as well as the changes requested. composite data from all feedback channels resulted in a total of 687 suggestions identifying 17 different types of space changes. an onsite whiteboard, the libqual+® survey, and library census proved the most prolific in producing suggestions. mailto:dhalling@library.tamu.edu� mailto:ecarrigan@library.tamu.edu� 71 conclusion – priorities for space changes were best determined through a composite of suggestions received from all feedback channels. the number of suggestions and requests received that were initiated by users was so small that it had to be supplemented with library-initiated feedback requests. the use of multiple feedback channels enhanced the number, variety, and scope of the suggestions that were received. similar requests received through multiple feedback channels emphasized their importance to users. focused follow-up feedback channels were effective in clarifying user suggestions for specific changes. introduction the texas a&m university and texas a&m health sciences center (united states) have an enrollment of over 50,000 students. the medical sciences library (msl) at texas a&m university is charged with serving several diverse user groups within both of these institutions, including the colleges of agriculture and life sciences, medicine, nursing, pharmacy, veterinary medicine, and biomedical sciences, and the school of rural public health. as the number of visitors to the library continues to rise each year, msl has begun to redesign and renovate library spaces in response to demands. as electronic resources replace print collections, the collection-centric model of libraries is being replaced by the user-centred, user-experience model. in an effort to make certain that renovation funds are used most effectively and that redesigned spaces really work for library users, msl has been expanding its sources for user feedback and input into making space renovation decisions. organizational readiness at its essence, the ability to accept and act upon user feedback is a change-management challenge for the library organization. it requires moving library staff along a continuum which progresses from the vision of the academic library as a collections warehouse to a vision of the academic library as a composite of services (virtual and onsite) and a physical space that is an integral part of the campus learning environment. several steps were taken at msl to begin this effort. comments from libqual+® surveys prior to 2010 made clear the importance of the library as a place for undergraduate, graduate, and professional students. in an effort to provide the ongoing focus and accountability necessary to consistently respond to user feedback and implement change, an onsite services librarian position was created. this was a new position with primary responsibility for the total user experience in the library, which encompasses all physical spaces (user and collection spaces) and all services delivered. by position definition, the onsite services librarian was uniquely positioned to play a leadership role in library space redesign and to lead change efforts on a daily basis. to assist the onsite services librarian, and to build consensus around the issue of space, the technique of scenario planning was used to involve all library staff in imagining what msl would look like in 2015 (giesecke, 1998). library staff participated in a brainstorming session to identify the key forces in the environment of the library and its users which would be strong determinants of what msl would be in 2015. finally, as one of the key elements in the decision-making process concerning library redesign, the onsite services librarian created an msl student advisory council that consisted of student leaders from each of the primary user groups. the chief hope for the msl student advisory council was to channel student energy and promote their engagement in creating the future of library spaces. literature review numerous articles, book chapters, and books have explored library space planning in general and recent changes in the concept of 72 the library as a physical place (bennett, 2003; connor, 2008; council on library and information resources, 2005; ludwig, 2010; ludwig & starr, 2005; stewart, 2010). while there is no uniform vision for the future of the library, most of the authors fundamentally agree on several concepts that are illustrated in current design trends, although their works vary in emphasis of these concepts. all recognize that this is a time of great transition in libraries and that developments in information technology have been a major catalyst in this transition. there is growing recognition that libraries are an integral part of campus learning spaces (bennett, 2003; council on library and information resources, 2005). this leads to an acknowledgement that there continues to be a need for variety in library spaces to provide collaborative spaces, to meet the social dimensions of learning and of active learning, and to provide spaces for quiet study and contemplation (ludwig, 2010; ludwig & starr, 2005). these works approached the subject from a trends perspective, with little mention of user input into space decisions and no mention of the use of multiple feedback channels. another segment of the literature has focused on user-driven library design, with data collected chiefly through the use of surveys (antell & engel, 2006; vaska, chan, & powelson, 2009; walton, 2006). vaska et al. (2009) and walton (2006) collected feedback with a single user survey. hobbs and klare (2010) reported the use of ethnographic techniques to gather student input into library space decisions. the research focus of the hobbs study was student behaviour and use of campus spaces. the question at hand was the effect that having multiple feedback channels had on the results received. much has been written on the subject of changing library spaces and soliciting user feedback or gathering use data to guide decisions. several studies involved the use of multiple inputs in general space planning and redesign efforts. hiller (2001) focused primarily on the use of libqual+® survey data to guide library decisions concerning services, collections, and the library as place. he touched upon the use of additional locally based, large-scale surveys, but did not provide specific results or impacts of the multiple inputs. this confirmed the potential value of the libqual+® survey data in making space decisions. waxman, clemons, banning, and mckelfresh (2007) reported the use of questionnaires and field notes to document student behaviour and preferences for relaxation spaces to outline the specifications for the design of coffee shops within the library. although limited to space planning for an information commons, cataldo, freund, ochoa, and salcedo (2007) described the use of external site visits, surveys, focus groups, and interviews. the reports of both waxman et al. (2007) and cataldo et al. (2007) reinforced the local supposition that multiple data sources would enrich the results. dotson and garris (2008) documented the use of multiple data inputs of library-generated statistics from observational studies on the use of library computers, group tables, individual study desks, carrels, and stacks. this study design underscored the complexity of user preferences for differing spaces. moffat and anderson (2009) provided a brief report on the use of multiple user surveys, student focus groups, and postimplementation surveys in determining library services and spaces. although none of these provided data or an analysis of the impact or effectiveness of using multiple inputs, they also did not dismiss multiple user inputs as redundant or meaningless. since it was clear that there was little research in the library literature that analyzed the impact of multiple feedback channels in space planning, the project was begun with the intent to fill that gap. methods several different methods were used at msl to capture input from users concerning library spaces. these methods can be grouped into two categories based on whether the data collected is user-initiated or library-initiated. table 1 presents the data collection methods used during 2009 and 2010 to gather feedback concerning library space and the corresponding results. 73 suggestions received online and onsite at msl, examples of usually available userinitiated tools included an onsite suggestion box, online links for user comments, and an email address specifically created for user interaction. these feedback channels are all available to users whenever they are ready to provide feedback to msl. most submissions through the onsite suggestion box are often focused on improvements to the library, while comments received online through the website links and via email tend to be more openended communications. none of these is specifically focused on feedback for library spaces. occasionally, msl receives spontaneously organized and orchestrated feedback campaigns from a user group on a subject of particular interest to them, for example, the closing of a coffee shop. library census a library census, conducted every five years, is a library-initiated tool that consists of a series of questions given to every user who enters the msl on a single day. the focus is on collecting demographic data from the user, with additional questions aimed at identifying why the user has come to the library on that visit and what services they used. the most recent census was conducted on 13 april 2010. libqual+® survey created in 2000, the libqual+® survey is a user-centred tool that libraries use to solicit, track, and understand users’ opinions and insights about service needs and expectations. this web-based survey asks users to provide the minimal, optimal, and perceived ratings for library resources and services in three main facets: customer service (customer treatment, job knowledge of staff); information resources, their delivery, and accessibility; and the library as a place of study and learning (study environment, group and individual study needs) (association of research libraries, statistics and assessment program, 2010). the libqual+® survey has been used annually by the texas a&m university library since 2000. comments pertaining to the msl that were extracted from the general university library’s libqual+® survey and results from a 2009 pilot libqual+® survey (limited to the colleges in the health science center) indicated the growing importance of the library as a place of study. the usefulness of these comments led to the decision to conduct the 2010 libqual+® survey to target all msl user populations. an msl libqual+® survey was administered and available online from 16 february 2010 until 31 march 2010. survey invitations were sent to all students and faculty in the college of veterinary medicine and biomedical science and to all units within table 1 feedback channels feedback channels responses received responses with user comments comments regarding library space length of time of data collection u se rin it ia te d online/onsite suggestions 36 36 34 12 months l ib ra ry in it ia te d census 427 247 189 1 day libqual+® 714 368 133 6 weeks whiteboard 209 209 209 3 weeks ballot voting 115 115 115 3 weeks text voting 7 7 7 3 months 74 the texas a&m health science center. since the college of agriculture and life sciences had been included for 10 years in the general university library libqual+® survey, the decision was made to continue the established format for conducting the survey, following their method of using only a random sampling survey invitation for undergraduate and graduate students, but including all faculty of the college. follow-up and focused voting methods whiteboard voting, ballot voting, and text voting were all used during 2009-2010 to answer specific questions about re-engineering library spaces and replacing library furnishings. a large portable whiteboard asking what would help enhance the library space for their needs was used to gather ideas from users about what should be put into the large area freed up by the removal of the current journal display shelves. the whiteboard voting was used over a period of three weeks. ballot voting and cellphone text voting were both used to gather data on preferred choices for furniture selection. three sample chairs were brought into the library, allowing users the opportunity to try out the different styles and cast their vote for their top choice. ballot voting was open for three weeks and text voting for three months. student advisory council the msl student advisory council served as a sounding board for establishing priorities and finalizing space redesign decisions based on the information gathered through the various feedback channels described above. this 12-person council was comprised of student leaders from each of the primary colleges served by msl (college of veterinary medicine and biomedical sciences, texas a&m health science center, college of agriculture and life sciences) and includes undergraduate, graduate, and professional students. the council provided direct input as members had first-hand experience with the role the library played in the routine of the student. this input was gathered both for existing situations and, more importantly, before an expense was incurred in making a physical change. for larger or more expensive initiatives the onsite services librarian met with the library student advisory council to clarify intended activities and to open channels of communication and opportunities for customization before any expense was incurred. results during 2009-2010 there were 34 user-initiated suggestions that pertained to the library as a place, spanning 12 different space change areas. the top five requested changes from this feedback method were: additional power outlets, increased comfortable seating, more computers, relaxed food and drink policy, and improved lighting. there were also five compliments about the general msl environment. the msl census, conducted in april 2010, provided feedback about the library as place from responses to survey questions and from an open-ended comments opportunity provided on the form. although intended primarily as a demographic tool concerning library user populations, the census comments produced the largest variety of space improvement suggestions, and also helped underscore the importance of various library services and environmental features which were reported in other feedback channels. a total of 427 completed census forms were received. from the responses, 247 comments were provided; 189 of those comments concerned the library as a place and covered 14 different space change areas. the top five requested changes were: more quiet study, more computers, more comfortable seating, more study rooms/booths/carrels, and more power outlets. class work or class preparation was the most common reason users came to the library, followed by computer use, and the use of group study rooms or carrels. nearly half of all respondents considered the library as a get-away and as a place for relaxation. a total of 714 users responded to the 2010 msl libqual+® survey. of the 368 comments received, 133 dealt with the library as a place, representing 12 different space change areas. 75 of the library as place comments, 46% expressed compliments and 54% expressed concerns. the top five requested changes were: more comfortable seating, more quiet study, more group tables, more computers, and more study rooms/booths/carrels. the whiteboard was the most productive channel for user feedback, producing 209 suggestions that were focused on space, although most suggestions fell into only 8 different space change areas. the top five suggested uses for the newly opened space were: study rooms/booths/carrels, group tables, comfortable seating, computers, and a large-screen tv. there were three requests for the return of the current journal issues. a total of 115 ballots were cast in the chair voting, providing a clear preference for the study chair of choice. during the next opportunity to test out new chairs a system of cellphone text voting was used. considering the general activity level of students and their cellphones, the results were extremely surprising, with only seven votes cast. the final prioritization of space change areas was based on the cumulative totals for the change areas from all feedback channels, presented in figure 1. the following section describes six key priorities that emerged from the feedback vehicles and how they were addressed. seating several forces prompted the decision to purchase new study chairs. besides the need to replace broken study chairs, the client services desk staff noted increasing complaints from students that they could not find any seats in the library. comments from the msl census and whiteboard feedback also addressed the need for additional seating. based on a strong consensus of choice, 50 new fully adjustable student-selected chairs were purchased to increase or improve the seating options. as a second phase of this project, following many 2010 libqual+® comments requesting more of these chairs, an additional 50 were purchased during the summer of 2010. the use figure 1 space change area totals 0% 5% 10% 15% 20% e q u ip m e n t r o ll in g w h it e b o a r d s li g h ti n g fo o d /d r in k la r g e s c r e e n t v c o m p u te r s g r o u p t a b le s r o o m s /b o o th s /c a r r e ls s e a ti n g q u ie t s tu d y p o w e r h o u r s p a r k in g c li m a te c u r r e n t jo u r n a ls 76 of ballot voting in the choice of chair showed msl the importance and effectiveness of a focused follow-up to survey feedback. rooms/booths/carrels additional study rooms have been requested on every feedback channel employed by msl over the past several years (applegate, 2009). efforts have been underway to address this request. past efforts to increase group study rooms involved converting multiple small photocopy rooms into group study rooms. between 2009 and 2010 the decision to renovate and re-engineer little-used or unused office and work spaces into group study rooms resulted in a 50% net increase in study rooms at msl. monitoring of feedback channels and efforts to craft creative solutions for more group study rooms will continue. an architectural/interior design firm has been contracted to develop plans for renovation of msl first floor space, with actual redesign work to be completed during 2012. study booths were a relatively new request that came chiefly through the whiteboard feedback channel. this has been forwarded to the architectural/interior design team so that these can be incorporated into the redesign plan. carrel availability and functionality have been improved through re-arranging the carrels to take better advantage of power sources throughout msl, and to be certain that they are located in spaces that offer a quiet study atmosphere. group tables (collaborative areas) group study tables were one of the highest requested uses through the whiteboard feedback channel for the recovered current journal space. in addition, they were mentioned in msl census and libqual+® survey comments. there is an increasing expectation for group projects across the curricula of all student users of msl. as a result, the need for areas of collaborative study continues to rise (adamson & bunnett, 2002). msl took a dual-track approach to address this need. first, eight additional tables were placed in the space opened up by the removal of current print journal shelves as had been requested by users. second, an underutilized computer lab was refitted with furniture that allowed the computers and monitors to be stored below the desktop, transforming the “computer” desks into tables for collaborative work and study. these desks were also equipped with casters, allowing the tables to be rearranged as needed. quiet study one of the primary concerns that msl library users identified consistently as a priority in libqual+® surveys and through msl census was quiet study space. when viewed in conjunction with the increased emphasis on collaborative study areas discussed above, quiet study space can seem almost in direct conflict. msl is a two-storey building which allows for some separation between the collaborative first floor areas being developed and what has historically been considered a quiet study second floor. unfortunately, there is a large atrium opening to both floors which, although very pleasing aesthetically, also very effectively moves sound between the floors. msl client services staff often received spoken and written complaints about noise levels. a review of this situation by library leadership concluded that the best approach to meet the need for quiet, while maintaining collaborative spaces on the first floor, was to make an actual physical barrier between the two spaces. this barrier also needed to blend with the open, airy feeling which characterizes the library. with the library student advisory council in agreement, a decision was made to install a glass wall around the second floor study area that would provide a noise barrier between the atrium and the second floor. having msl users enter through a glass door to reach the “quiet study zone” has proved an effective reminder of the quiet study expectations for that area. the addition of the walls also created a small lounge area that made it possible for students to leave the quiet zone for phone calls or discussions that might disturb the quiet study environment. 77 computers improvements in this area were driven by requests from the onsite suggestion box, postings on the whiteboard, and libqual+® comments. public computers were relocated to make better use of power sources in columns throughout the library. as an added benefit, the move of these computers also improved user access. the move placed the computers closer to the single service point desk where staff would be more available for user assistance. it also became much easier for users to see where open seating could be found. finally, the move to the new location further away from large first floor windows eliminated a glare problem which had made using some of the computers undesirable. improvements in access to power also resulted in an increased demand for laptops available for checkout. between 2009 and 2010 msl increased the number of public computers (desktops and laptops) by 25%. in that same time frame the circulation of laptops increased by about 70%, and the higher usage of public laptops brought relief to the need for additional computer access. special efforts were also made to publicize and promote the use of 40 additional public computer workstations in the library’s computer lab. all of the above efforts resulted in an increase in computer access, although this request continues to be received from users. power libqual+® survey comments, whiteboard suggestions, and onsite suggestion box comments made clear the need for increased power in the library. this simple request was one of the most difficult and expensive for msl to address since it would require significant structural modifications to the building to add power outlets to the floor. in an effort to contain the costs of this enhancement, an inventory was conducted for all sources of power in public areas and a diagram was developed. through analysis of this diagram and building observations, solutions were developed to make much better use of the columns in library public spaces and the power source they provided. tables with public computers were rearranged to take advantage of outlets in nearby columns. this new arrangement freed up other power outlets that could be used for visitors with personal computers. as a temporary measure, several of the tables were equipped with power strips that could be shared by multiple users. a more permanent improvement in power availability was accomplished by retrofitting 23 study tables with pop-up power outlets at both ends of the table tops. this solution also allowed msl to include an led light bar as part of the retrofit package for the study tables. this change addressed requests for improved task lighting that had been voiced by users through online/onsite suggestions, msl census, and libqual+® survey comments. discussion there are advantages and disadvantages to both user-initiated and library-initiated approaches to feedback. user-initiated input has the benefit of identifying a specific concern that the user has at a particular point in time; it also offers the opportunity for immediate, personalized response from library staff. some of the challenges of relying on individual userinitiated input include the possibility of misinterpretation of user comments due to variation in language and differences in communication, the temptation to overgeneralize to the entire population, and difficulties in interpreting and analyzing freetext suggestions consistently. library-initiated input usually has the advantages of being more carefully planned, consistent, and based on a controlled vocabulary. this greater consistency improves data analysis, interpretation, and longitudinal trend analysis. the use of standard survey tools such as libqual+® also offers the opportunity for benchmarking against peer libraries. however, survey fatigue is a potential problem, as well as the challenge of finding a survey length that encourages participation and also results in significant amounts of valid data. limitations of each feedback channel the establishment of a broad complement of feedback channels for user input concerning 78 the redesign of library spaces contained challenges and limitations inherent in each individual channel. online/onsite suggestions, while providing specific concerns or requests, produced far too little input to be used as the sole feedback approach. the msl census, although one of the most productive channels for library space comments, was only a snapshot from users on one particular day. libqual+® offered the most statistically valid approach to gathering feedback, producing the third highest output of space comments and variety of space change requests. the whiteboard and ballot voting, while very productive and effective, were too focused on a particular space change or limited to a particular library space to supply ideas for changes to all library spaces. cellphone text voting did not produce enough responses to be useful. use of multiple feedback channels one objective of this project was to evaluate the use of multiple, broad-based feedback channels and whether it enhanced the feedback gained concerning changes in the library spaces. table 1 presents the cumulative amount of feedback received through each feedback channel and the length of time each feedback channel was available. all the library-initiated feedback channels produced many times more comments than the userinitiated suggestions. clearly, msl needed additional feedback channels beyond the online and onsite suggestion box. figure 1 presents a composite of all space-related suggestion areas received. the total number of comments for each space change request area listed in table 2 was compared to the total number of space-related comments, resulting in relative percentages for each space change area. table 2 space changes requested by feedback channels online/ onsite suggestions census libqual+® whiteboard ballot voting text voting totals more ergonomic seating 4 14 30 22 115 7 192 more group study spaces 2 12 9 73 0 0 96 general environment 5 53 30 0 0 0 88 more group study tables 1 7 14 49 0 0 71 less noise in quiet study 1 47 19 0 0 0 67 more computers 4 17 10 22 0 0 53 more power outlets 6 10 2 16 0 0 34 large screen tv 0 0 0 20 0 0 20 longer hours 2 6 4 0 0 0 12 relaxed food/drink policies 4 4 4 0 0 0 12 improved task lighting 3 6 2 0 0 0 11 more parking 0 3 7 0 0 0 10 user control of climate 1 5 0 0 0 0 6 more rolling whiteboards 1 0 0 4 0 0 5 more specialized equipment 0 2 2 0 0 0 4 return current journals area 0 0 0 3 0 0 3 cleaner bathrooms 0 3 0 0 0 0 3 totals 34 189 133 209 115 7 687 79 the question of the impact multiple feedback channels had on determining priorities seems best answered by a comparison of what would have been done if feedback came from only a single source, as well as what would not have been done, without the additional information from other feedback channels. if only the user-initiated feedback channels had been used, the priorities addressed would have been seating, computers, power, lighting, and food/drink policy. several of the top priorities from the census, libqual+® survey, and whiteboard feedback channels would not have been addressed. the areas of study rooms/booths/carrels, group tables, and quiet study would all have been missed. the library census identified four of the top priorities that were actually addressed. but the important area of group tables for collaborative work would have been missed. the libqual+® survey, taken as an individual source for changes in the library as place, was the most effective single route, but it would not have identified the request for added power outlets, which was included in three of the other feedback channels, as a priority. additionally, to truly add a higher level of specificity for the libqual+® user comments, future initiatives should include a categorization employed with comment analysis. the whiteboard, although focused on a specific area within the library, identified four of the top priorities addressed. the quiet study area was not mentioned, which was understandable, since this feedback channel focused on a first floor area of the library which is designated for collaborative space. a review of all feedback channels clearly indicated the need for library-initiated methods to supplement the relatively sparse suggestions provided by the user-initiated feedback channels. if any single libraryinitiated feedback channel had been used, from one to several important suggestions would have been missed. the use of multiple feedback channels uncovered several important priorities that were not strongly represented by any single feedback channel. while the libqual+® survey turned in a solid performance as a feedback channel for space redesign, the use of focused and followup feedback mechanisms (ballot voting, student advisory council) also proved very effective. conclusions msl accomplished the objective of this project, to use composite data from multiple user feedback channels in prioritizing space redesign. analysis of the feedback clearly indicated that the libqual+® survey and msl census are rich sources for user comments on the library as a place. this has led to a commitment to a regular schedule for the libqual+® survey and more frequent msl censuses. a review of the volume of suggestions received through the online and onsite suggestion avenues indicated the need for additional feedback channels. interpretation of feedback provided through all channels suggested the value of focused follow-up feedback techniques as appropriate. efforts to determine priorities for the numerous space change requests received resulted in the opinion that multiple feedback channels were very useful. the authors intend to continue research into the published evidence to validate this reported msl experience. msl will monitor the online/onsite suggestion box and the 2012 libqual+® survey to assess how users respond to the changes mentioned in this paper, will continue to solicit feedback from library users, and will carefully consider it in charting an evidence based course for library redesign. acknowledgements michael maciel, data analyst for the texas a&m university libraries, is to be recognized for his contribution to the libqual+® survey portion of this paper. his experience with the compilation and interpretation of the data results saved tremendous energies and provide added clarity to the process. references adamson, m. c., & bunnett, b. p. (2002). planning library spaces to encourage 80 collaboration. journal of the medical library association, 90(4), 437-441. retrieved 6 feb. 2012 from http://www.ncbi.nlm.nih.gov/pmc/arti cles/pmc128960/ antell, k., & engel, d. (2006). conduciveness to scholarship: the essence of academic library as place. college & research libraries, 67(6), 536-560. retrieved 6 feb. 2012 from http://crl.acrl.org/content/67/6/536.full. pdf+html applegate, r. (2009). the library is for studying: student preferences for study space. journal of academic librarianship, 35(4), 341-346. doi:10.1016/j.acalib.2009.04.004 association of research libraries. statistics and assessment program (2010). libqual+®: charting library service quality. retrieved 6 feb.2012 from http://www.libqual.org/home bennett, s. (2003). libraries designed for learning. washington, d.c.: council on library and information resources. retrieved 6 feb. 2012 from http://www.clir.org/pubs/reports/pub1 22/pub122web.pdf cataldo, t. t., freund, l., ochoa, m. n., & salcedo, m. (2007). the info commons concept: assessing user needs. public services quarterly, 2(4), 23-46. doi:10.1300/j295v02n04_02 connor, e. (2008). library space planning. in m. s. wood (ed.), introduction to health sciences librarianship (pp. 369-393). binghamton: routledge. council on library and information resources (2005). library as place: rethinking roles, rethinking space. washington, dc: clir. retrieved 6 feb. 2012 from http://www.clir.org/pubs/reports/pub1 29/pub129.pdf dotson, d. s., & garris, j. b. (2008). counting more than the gate: developing building use statistics to create better facilities for today’s academic library users. library philosophy & practice, 10(2), 1-13. retrieved 6 feb. 2012 from http://www.webpages.uidaho.edu/~m bolin/dotson-garris.pdf giesecke, j. (ed.) (1998). scenario planning for libraries. chicago: american library association. hiller, s. (2001). assessing user needs, satisfaction, and library performance at the university of washington libraries. library trends, 49(4), 605-625. retrieved 6 february 2012 from http://www.ideals.illinois.edu/bitstrea m/handle/2142/8375/librarytrendsv49i 4e_opt.pdf?sequence=1 hobbs, k., & klare, d. (2010). user driven design: using ethnographic techniques to plan student study space. technical services quarterly, 27(4), 347-363. doi:10.1080/07317131003766009 ludwig, l. (2010). health sciences libraries building survey, 1999-2009. journal of the medical library association, 98(2), 105-134. doi:10.3163/1536-5050.98.2.004 ludwig, l., & starr, s. (2005). library as place: results of a delphi study. journal of the medical library association, 93(3), 315326. retrieved 6 feb. 2012 from http://www.ncbi.nlm.nih.gov/pmc/arti cles/pmc1175798/ moffat, c., & anderson, g. (2009). library design: giving the students what they want. incite, 30(9), 28-29. stewart, c. (2010). the academic library building in the digital age: a study of construction, planning, and design of new library space. chicago: association of college and research libraries. vaska, m., chan, r., & powelson, s. (2009). results of a user survey to determine 81 needs for a health sciences library renovation. new review of academic librarianship, 15(2), 219-234. doi:10.1080/13614530903240635 waxman, l., clemons, s., banning, j., & mckelfresh, d. (2007). the library as place: providing students with opportunities for socialization, relaxation, and restoration. new library world, 108(9/10), 424-434. doi:10.1108/03074800710823953 walton, g. (2006). learners’ demands and expectations for space in a university library: outcomes from a survey at loughborough university. new review of academic librarianship, 12(2), 133149. doi:10.1080/13614530701330430 / evidence based library and information practice/ / article   formality in chat reference: perceptions of 17to 25-year-old university students   jennifer waugh instruction and reference librarian northern alberta institute of technology library edmonton, alberta, canada email: jewaugh@gmail.com   received: 18 july 2012    accepted: 18 feb. 2013      2013 waugh. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to examine the ways in which the formality of language used by librarians affects 17to 25-year-old university students’ perceptions of synchronous virtual reference interactions (chat reference), in particular, perceptions of answer accuracy, interpersonal connection, competency, professionalism, and overall satisfaction.   methods – this qualitative study used semi-structured interviews to examine the perceptions of participants. participants reviewed and responded to two virtual reference transcripts, portraying a librarian and student asking a simple question. one transcript portrayed a librarian using traditional, formal language while the other portrayed a librarian using informal language. five 17to 25-year-old university students were interviewed. data were analyzed using a phenomenological, qualitative approach to discover common themes.   results – analysis suggests that participants perceived the formal librarian as being “robotic” and impersonal while the informal librarian was thought to be more invested in the reference interaction. several participants viewed the formal librarian as more competent and trustworthy and questioned the effort put forth by the informal librarian, who was perceived as young and inexperienced. participants’ perceptions of professionalism were based on expectations of social distance and formality. satisfaction was based on content and relational factors. several participants preferred the formal interaction based on perceptions of competency, while others preferred the informal librarian due to perceived interpersonal connection.   conclusion – formality plays a key role in altering the perceptions of 17to 25-year-olds when viewing virtual reference interaction transcripts. both language styles had advantages and disadvantages, suggesting that librarians should become cognizant of manipulating their language to encourage user satisfaction.     introduction   libraries have undeniably been impacted by the invention of the internet. resources are increasingly made available in digital formats, sometimes exclusively, due to high demand for instantaneous access to information. in the face of these changes, virtual reference services (vrs) in their asynchronous (email reference) and synchronous (chat reference) forms have emerged as viable alternatives to traditional face-to-face (ftf) reference services in academic and public libraries (johnson, 2004).   synchronous vrs allow librarians and users to communicate in real time and users can connect wherever internet access is available. vrs offer many advantages, including speed, convenience, and the ability to save time through the use of pre-generated scripts (thompson, 2010). yet, chat can be a challenge for librarians who may find online communication difficult, lacking the visual and non-verbal cues that are central to ftf communication (fagan & desai, 2003; park, 2007).   in academic libraries, vrs are of particular importance for young students from the newly dubbed millennial generation (millennials), born 1979 to 1994 (sweeney, 2006). for this generation, technology is important: 90% of millennials use the internet compared with 79% of baby boomers, and instant messaging (im) services are frequently used for communication amongst millennials (pew internet and american life survey, 2004; pew internet and american life survey, 2010). accordingly, synchronous vrs are important for this user group and studies have found that undergraduates, who account for the majority of university-aged millennials, are the predominant user-group of synchronous vrs in academic libraries (arnold & kaske, 2005; houlson, mccready, & pfahl, 2006).   it is widely recognized that this age group has a unique online communication style. a librarian chatting with a 17to 25-year-old might note the frequent use of contractions (e.g., “btw”), emoticons (e.g., “:)”), and a lack of punctuation and capitalization (baron, 2004; haas, 2011; maness, 2008; park 2007; rourke & lupien, 2010), prompting some to suggest librarians mimic this informal style to appear more approachable (fagan & desai, 2003). however, though there is a wealth of literature exploring vrs, much is focused on the evaluation of the vrs of specific libraries or of question type and answer accuracy (arnold & kaske, 2005; cloughley, 2004; houlson, mccready, & pfahl, 2006; silverstein, 2006; white, abels, & kaske, 2003). few have explored the impact that formal or informal language may have on the virtual reference interaction.   radford (2006) and westbrook (2007) have identified the need for studies which examine formality in vrs, as formality is a critical component of relationship building and interpersonal communication when using computer-mediated communication (cmc). to date, studies of formality in cmc have largely used quantitative methods to identify broad trends (jessmer & anderson, 2001; walther & d’addario, 2001) or have qualitatively observed formality patterns (baron, 2004; haas, 2011; maness, 2008; park, 2007; radford, 2006; westbrook, 2007). this study uses a qualitative framework to more deeply investigate the ways in which the critical user group of 17to 25-year-old students perceive the use of formal and informal language in virtual reference to supplement previous quantitative and observational studies. the study addressed the following research questions:   does the formality of language used by librarians impact how 17to 25-year-olds perceive the credibility of answers provided to them during a virtual reference interaction? if so, how? does the formality of language used by librarians impact how 17to 25-year-olds perceive the competency or professionalism of the librarian in a virtual reference interaction? if so, how? does the formality of language used by librarians affect 17to 25-year-olds’ perceptions of interpersonal connection with the librarian during a virtual reference interaction? if so, how? does the formality of language used by librarians affect 17to 25-year-olds’ perceptions of satisfaction in virtual reference environments? if so, how?   literature review   interpersonal communication in the virtual reference interview   cmc is a unique form of communication. unlike ftf, it physically separates communicators from one another. as such, the ability to communicate through gesture, facial expression, or vocal qualities such as intonation is lost (park, 2007). this has prompted some, such as daft and lengel (1984), to suggest there is less potential to develop socioemotional relationships in this medium or that this reduced social presence makes cmc less friendly and personal (rice & love, 1987). however, a growing body of work suggests that cmc is as rich as ftf communication. rice and love (1987) discovered that over 30% of messages sent via cmc are socioemotional in content and more recent studies by park (2007), radford (2006), and walther and d’addario (2001) suggest that users of cmc have adapted the medium by developing textual cues to replace nonverbal and prosodic signals such as, intonation, accent, and vocal pitch. radford’s (2006) examination of vrs transcripts concluded that cmc is no less personal than ftf communication.   it is critical to consider interpersonal communication in the virtual reference interaction, as relational dimensions have been shown to greatly impact the ftf reference interaction (dewdney & ross, 1994; radford, 1998; ross & dewdney, 1998). durrance (1989) found that, in ftf reference interactions, users were unlikely to forgive negative interpersonal factors. dewdney and ross (1994) discovered that users who perceived librarians as friendly were more likely to express overall satisfaction. in fact, many of the guidelines dictating best practice for ftf reference interactions focus on interpersonal skills and building rapport (reference and user services association [rusa], 2004).   interpersonal dimensions have been found to be equally important in virtual reference interactions (connaway & radford, 2010; maness, 2008; mon, 2006; mon & james, 2007; nilsen, 2004; radford, 2006). mon (2006) found that users responded positively to librarians perceived as friendly and polite while impolite librarians were deemed unhelpful. mon and james (2007) examined virtual reference interactions which had received unsolicited “thank you” messages and discovered that satisfaction was determined both by content and relational dimensions. nilsen (2004) concluded that users respond similarly to virtual and ftf reference interactions and that interpersonal factors are important in both.   this body of work suggests that interpersonal communication is possible in cmc and plays a significant role in determining whether users perceive a virtual reference interaction as successful. yet few have examined how interpersonal information is communicated in the virtual reference interaction. radford (2006) identified “relational facilitators” and “relational barriers” in virtual reference interactions by qualitatively evaluating 285 virtual reference transcripts. her results suggest that actions such as using emoticons or abbreviations to compensate for nonverbal cues facilitate relationship building, while relational barriers include ending the interaction without an exchange of farewell. walther and d’addario (2001) found that emoticons could successfully convey interpersonal information, and recent studies have suggested that informal language such as abbreviations (e.g., “ttyl”), contractions (e.g., “gunna”), and emoticons and excessive punctuation (e.g., “thanks!!!!”) are associated with greater interpersonal connection in various cmc environments (park, 2007; park, 2008a; radford, 2006; westbrook, 2007).   politeness, formality, and cmc   several cmc studies integrate brown and levinson’s (1987) politeness theory to explain interpersonal and relational aspects in cmc. politeness theory is based on the assumption that individuals have a social “face” with a negative and positive aspect (arundale, 2005; brown & levinson, 1987). positive face involves the desire to be approved of and receive appreciation whereas negative face involves a person’s desire to be unimpeded in their actions (brown & levinson, 1987). being “polite” requires individuals to affirm the positive face of others without undermining their negative face. communication ultimately breaks down when persons are “impolite” or make face-threatening acts (fta) to either the negative or positive face (arundale, 2005; morand & ocker, 2002).   in cmc, formality indicators are a key component of politeness. formality establishes the boundaries and nature of the relationship between communicators (morand & ocker, 2002; westbrook, 2007). formal language is normally used among relative strangers and indicates social distance and seriousness. it can affirm the negative face by showing respect for a person’s autonomy and expertise (park, 2008b), whereas informal language is often used between those with less social distance between them and can affirm the positive face (morand & ocker, 2002).   politeness is complicated in cmc due to the lack of nonverbal and prosodic cues that are used in ftf interactions to clarify meaning (morand & ocker, 2002; westbrook, 2007). without such cues, speakers in cmc are prone to misinterpret messages, as was seen in nilsen’s (2004) study of virtual reference interactions where users often perceived librarians’ comments negatively while the same statements spoken ftf would likely not have elicited comment.   studies of politeness theory in cmc suggest that formality is a critical factor to consider in virtual reference interactions. jessmer and anderson (2001) suggest that those who send polite, grammatically correct emails are seen as more competent than those who send informal emails. yet in the virtual reference interaction, mon (2006) and thompson (2010) found that users may respond negatively to formal language which is perceived as “robotic.”  on the other hand, informality can encourage positive interpersonal relationships but may also imply that the sender is uneducated or of low status (jessmer & anderson, 2001). in order to facilitate successful communication in virtual reference interactions it is critical that librarians understand how different styles of communication are perceived by users of vrs.   summary   interpersonal communication has long been identified as critical to the ftf reference interaction, yet researchers are only beginning to understand its role in the virtual environment. previous studies have established patterns, yet burke and kraut (2008) note a major downfall of current cmc politeness or interpersonal communication studies in lis is that most are descriptive in nature. few have included users’ perceptions in their findings, in particular the perceptions of 17to 25-year-old university students. this study addresses these gaps in the literature and builds upon previous, descriptive studies by directly assessing users’ perceptions  of formality and informality in virtual reference and its impact on perceptions of professionalism, competency, credibility, interpersonal connection, and satisfaction with virtual reference interactions.   methods   inquiry and design   this exploratory study was deeply rooted in human context and personal perceptions, and so the researcher deemed qualitative research methods appropriate. a phenomenological approach was chosen, as the research questions focus on understanding the lived experience of participants and their perceptions of formality (kvale, 1996; leedy & ormrod, 2009; patton, 2002). semi-structured interviews were used to facilitate the discovery of common themes, while allowing participants to fully articulate their unique perspectives (seidman, 2005). during interviews, participants were asked to share their perceptions about two virtual reference transcripts, each portraying a librarian and undergraduate student asking a question. one transcript portrayed a formal librarian; the other, a librarian using informal language. see appendices a and b to view the transcripts that were used for all interviews.   sampling/participants   phenomenological studies seek to understand the perspectives of individuals with direct lived experience with the phenomenon under study (leedy & ormrod, 2009; patton, 2002). in this study, “lived experience” was supplied in the form of transcripts, however; it was essential that participants could relate to the user shown, who was designed to resemble a 17to 25-year-old student. thus, purposive sampling strategies were used to recruit participants similar to the user in the transcripts. participants were required to be between the ages of 17 and 25, native english speakers, registered as students at the university of alberta, and have had experience using instant messaging.   recruitment posters were placed at the university of alberta and electronic advertisements posted on student mailing lists. participants were self-selected by contacting the researcher and volunteering to participate. five participants from the university of alberta took part, including three graduate students (participants 3, 4, and 5) and two undergraduate students (participants 1 and 2). all were female and within the age range of 18 to 24. all gave their informed consent before participating and were assigned pseudonyms to preserve anonymity.   data collection   during interviews, participants responded to two virtual reference transcripts, each portraying a librarian and a student asking a question. one portrayed a librarian using formal language, while the other portrayed a librarian using informal language (see appendices a and b). participants were asked to read and comment on both transcripts, alternatively reading either the formal or informal transcript to begin. participants were given pens, highlighters, and paper to record their thoughts on each transcript. after allowing for careful reading and scrutinizing of the transcripts, the researcher posed a series of open-ended questions centred on themes of professionalism, competency, interpersonal connection, and satisfaction. transcripts acted as a lived experience eliciting more revealing interview data than a focus on abstract experience or opinions would (gubrium, 2001; leedy & ormrod, 2009). transcripts were carefully constructed based on previous research. the structure of each was based on published examples of virtual reference transcripts (fagan & desai, 2003; radford, 2006; westbrook, 2007), surveys of vrs (houlson, mccready, & pfahl, 2006; rourke & lupien, 2010), and the rusa (2004) guidelines for behavioral performance of reference and information service providers. the formal librarian’s language was modelled on grammatically correct english such as is seen in a business letter and lacked informality markers such as linguistic contractions (e.g., “it is” instead of “it’s”). the informal librarian’s language used informality markers such as abbreviations, emoticons, colloquial grammar, informal punctuation, linguistic contractions, and prosodic features (e.g., “. . .” for time passage), based on research by haas (2011), park (2008a, 2008b), radford (2006), and westbrook (2007). finally, the language of the user was modelled on studies of 17to 25-year-olds’ cmc communication, including linguistic analysis conducted by baron (2004), haas (2011), and maness (2008).   data analysis   transcribed participant interviews were subjected to in-depth qualitative analysis using an approach similar to groenewald’s (2004), involving phenomenological reduction, delineation of units of meaning, individual theme formation, and the extraction of general and unique themes for all interviews.  each individual interview was analyzed through multiple readings. units of meaning were delineated by considering the literal content of the transcribed interviews, the number of times a meaning was mentioned, and how the meaning was stated through paralinguistic and non-verbal cues recorded during interviews (groenewald, 2004). major units of meaning were tabulated using microsoft excel along with representative quotes to identify.   following this, commonalities amongst participants were identified by comparing individual themes,while ensuring that minority voices were not lost amongst the majority themes (groenewald, 2004).throughout the data analysis process, the researcher was conscious of bracketing presuppositions. bracketing, or phenomenological reduction, refers to the deliberate and purposeful opening up of the researcher to the phenomenon (groenewald, 2004). the researcher avoided projecting interpretations and meanings onto the participants by frequently returning to the original transcripts and audio-recorded interviews to avoid becoming too removed from the data (forde, 2011).   results and discussion   authenticity   to address the impact of formality on interpersonal connection, participants were asked to share their perceptions of the librarian’s mood and whether the librarian cared about the student’s question. a strong theme which emerged was the concept of authenticity.   the librarian as a person   four of the five participants perceived the formal librarian as being robotic or machine-like. participant 4’s initial reaction upon reading the formal transcript was to exclaim, “this librarian sounds like a robot!” participant 2 explained, “it’s robotic in the way she’s answering the question. she might have added some other extra stuff.” participant 3 easily articulated her perception of the formal librarian: “it’s almost like in some ways speaking to a machine. it’s very . . . action-reaction.” paralinguistic cues further suggested the participants had difficulty injecting “life” into the formal librarian’s speech, as all of the participants, when mimicking the formal librarian, used a monotone lacking in pitch and tone variation.   previous studies have similarly found an association between formal speech and perceptions of a “robotic” interaction (mon, 2006; thompson, 2010). however, though mon’s findings suggest “robotic” librarians are perceived negatively, this was not the case for all participants in this study. participant 3 perceived the “robotic” formal librarian as calming, stating, “it’s like talking to this very lovely artificial intelligence . . . with a very persuasive voice. very calming.” and though many participants mentioned the formal librarian’s “robotic” nature, most perceived it as acceptable (see theme 4: professionalism).   in contrast, the informal librarian was often associated with human characteristics. while explaining her perception of the informal librarian’s grammar, participant 1 stated, “i mean it shows that they’re a real person,” and later referred to the informal librarian as “very much an authentic person.” further evidence of this theme was seen in the frequencies with which participants attributed humanizing characteristics such as gender, emotion, and age to the informal librarian versus the formal. participants 2 and 5 did not associate mood, gender, or age with the formal librarian. participant 5 stated, “it’s formal, impersonal, it gives nothing.” when asked about the formal librarian’s mood, participant 2 stated: “the librarian is . . . nothing really.” on the other hand, most participants associated happiness, femininity, and youth with the informal librarian, suggesting the informal librarian was more easily humanized. this is an interesting expansion of mon’s (2006) discovery that participants often positively mention interacting with librarians who are identified as “real people” based on the provision of a name, and supports others who have concluded that informal language can increase interpersonal connection between communicators (park, 2007; radford, 2006; westbrook, 2007).   authenticity of emotion   the informal librarian’s emotional engagement was seen as more authentic than the formal in three key areas of the interaction: the greeting, expression of interest, and closing. four participants perceived the use of informality in the greeting as engaging and inviting. participant 3 stated, “when i first started reading it, i’m like oh that’s awesome, they’re really into this . . . you can feel when the enthusiasm comes through.” again, paralinguistic cues supported this theme. when mimicking the informal greeting, the participants injected enthusiasm and excitement into their tone, while the formal greeting was interpreted as monotonous and “flat,” according to participant 5. although kwon and gregory (2007) found approachability difficult to observe in the virtual reference interaction, this finding supports fagan and desai’s (2003) assertion that informal greetings may be perceived as more approachable than formal greetings.   in addition, the informal librarian was seen as being more invested in the interaction by four participants. in reference to the informal librarian, participant 1 stated, “they are a little bit more invested,” echoed by participant 3’s statement that the informal librarian was “more invested somehow.” conversely, the formal librarian’s expression of interest, “that sounds interesting. have you tried searching in eric?” (line 10, appendix b), was perceived as inauthentic. when asked if she felt the formal librarian cared about the student’s question, participant 2 stated that “she pretended to care about it” and “it’s not like she really cares, it’s just her job.” participant 3 commented:   . . . the librarian says, “that sounds interesting,” that’s almost so formal you can’t really tell if they mean it. if they were more casual almost like, “oh my god that’s really cool,” and then you can feel that enthusiasm whereas this feels almost like here’s my token enthusiasm.   showing interest builds rapport during the reference interaction (radford, 2006; rusa, 2004) and the formal librarian’s lack of interest was negatively perceived by participants 1 and 2, who preferred the informal interaction due to perceived interpersonal connection with that librarian.   finally, several participants viewed the informal librarian’s invitation to return for help as more authentic. participant 3, referring to line 28 of the informal transcript (“np j . . . do you need help w. anything else?”), stated, “no problem, happy face is like really, no problem. it’s awesome that you wanted help,” while participant 2 injected an inviting, excited tone to the informal librarian’s closure but a monotonous tone to mimic the formal.   competency   formality builds trust   participants were asked what factors suggested competence or incompetence in both transcripts. competence was associated with content-based factors, such as providing instructions and links. in addition, proper grammar was often connected to feelings of trust and three participants perceived the formal librarian as more competent than the informal, regardless of other content-based factors.   feelings of trust were related to the perception that the informal librarian did not put forth the same effort as the formal. participant 3 stated, regarding the informal transcript, “i realize it’s the exact same content and even though one of the things i found really professional (in the formal transcript) is the thoroughness, it doesn’t feel thorough because of the way they’ve conveyed it,” and, “i think i’d walk away feeling kind of like i’m not sure they’ve done everything they possibly could to get me the answer to this question.” participant 4 echoed this, flatly stating, “i wouldn’t trust this.” participant 5 stated, “i don’t understand why they aren’t using proper grammar. . . . i don’t understand what they’re going to do for me. if they don’t use proper grammar, what else are they not doing?” participant 5 perceived the informal librarian as so untrustworthy that she would have left, had she been involved in the interaction.   perceptions of trust may be explained by politeness theory, as greater formality tends to be used among relative strangers to establish a basic level of trust and can also suggest that the librarian shoulders responsibility for problem solving (morand & ocker, 2002; westbrook, 2007). kim (2005) found that formal language is perceived as coming from an expert. in addition, jessmer and anderson (2001) found that the senders of grammatically correct emails are seen as being more invested in the editing of their messages and thus more competent.   though trustworthiness was strongly tied to formality for three participants, it is important to note exceptions. participants 1 and 2 did not perceive the use of informal grammar as being indicative of incompetence. participant 1 made some connection, stating, “it’s nice to see the quotation marks around ‘search.’ i don’t know, it’s just like a, ‘oh yeah, like you’re smart, you type properly.’” however, when asked about the competency of the formal librarian, she stated that they were “maybe not more competent but maybe a little bit more experienced.” participant 2 made no connection between formality and trustworthiness. when asked how she perceived the competence of the informal librarian, participant 2 stated, “she actually knows as much as the other person but the way she’s presenting it is better,” suggesting that the informal librarian’s language actually made her more competent in the chat environment. these findings suggest it may be an overgeneralization to presume that formality is automatically associated with higher competency for this user group.   power dynamics   feeling patronized   participant 1 perceived the formal librarian’s statement, “then press the search button” (line 26, appendix b), as “patronizing” and felt that the formal librarian was “bossy,” whereas she perceived the same statement in the informal transcript as lighthearted and explained that the informal librarian was “more of a guide than a boss.” in cmc, the act of giving advice (to press the search button) could be interpreted as a face-threatening act which threatens the positive face by suggesting the recipient is flawed and the negative by appearing to constrain choices (westbrook, 2007). perceptions of bossiness may also relate to a lack of prosodic and non-verbal cues which some have suggested leads cmc communicators to infer rudeness from relatively small indicators (morand & ocker, 2002; nilsen, 2004; westbrook, 2007). this finding suggests that informality may mitigate some of the face-threat associated with giving advice in cmc.   in addition, two participants associated feelings of inferiority with formality. participant 2 preferred the informal librarian because, “she’s in your language, the way you are speaking and it’s . . . not like you’re feeling different or you’re feeling inferior because you can’t use the language.” participant 3, reflecting on previous virtual reference interactions, mentioned feeling “pressure” from formal librarians, and that, “through the use of their extremely great grammar that they’re almost a little bit condescending.”   speaking my language   the importance of individual themes should not be discounted in the wake of shared commonalities (hyener, 1999). a unique theme was participant 2’s association of informal language with identity. participant 2 frequently referred to the informal librarian as speaking “her language” while the formal librarian’s grammar was perceived as out of place:   she’s writing full sentences and using punctuation which you never use in chat . . . she never used any kind of words like those small acronyms we use for like “thanks” even we don’t put all the – we just go “thx.” she doesn’t do that. she used a full sentence for that!   ultimately, participant 2’s choice of librarian was based on the use of informal language which she associated with her age group. this may be based on the positive politeness associated with using “in-group” speak, which can suggest camaraderie and common ground (morand & ocker, 2002).   professionalism   rusa (2004) guidelines were associated with professionalism by all of the participants, including asking open questions and follow-up questions. four participants associated formality more strongly with professionalism than with content-related factors.   professional tone   three participants perceived the formal librarian’s “tone” as professional. participant 5 stated, “they maintain a sort of good professional tone. very even, it doesn’t seem overly on either side, not overly friendly or overly unfriendly.” when asked what was professional in the formal transcript, participant 4 explained, “the librarian isn’t overly friendly in this one . . . it’s not like this person’s asking for advice. they’re asking for research help.” conversely, these participants felt the informal librarian was overly friendly or eager, which was perceived as unprofessional. the lack of formality and consequent lack of negative politeness in the informal interaction may have caused a perception of “coming too close” or intrusion for these participants (morand & ocker, 2002; park, 2007).   expectations of formality   often, perceptions of professionalism were tied to expectations of formality and by extension to asymmetrical interactions. goffman (1956) suggests that many interpersonal interactions are asymmetrical in nature and that people engage in “status rituals” to define superiority in relationships. both radford (2006) and westbrook (2007) observed that the librarian-user interaction is fundamentally asymmetrical, with librarians taking on a superior status. in this study, the relationship between the student and formal librarian was asymmetrical since formality typically establishes high social distance (morand & ocker, 2002). yet, the informal transcript was relatively symmetrical since the librarian and student use similar language, suggesting less social separation (park, 2007).   three participants had explicit expectations of formality, which led to a perception of unprofessionalism when reading the informal transcript. participant 5 felt that, “librarians should care about grammar,” and participant 3, referring to the use of informal abbreviations, said they gave “such a flavour that you don’t really expect from a librarian. you know, you expect some level of formal[ity].”   informality sometimes meant participants were unable to clearly define the roles of the student and librarian. in regards to the informal transcript, participant 4 felt that, “this could be me talking to my friends,” whereas in the formal transcript the roles were “very clearly . . . student, librarian.” participant 3, referring to the informal transcript, stated there was “nothing in it that makes me feel like they’re behind a desk in a library.” these findings extend those of radford (2006) and westbrook (2007), who suggest that users may perceive asymmetry in vrs interactions as the norm and feel uncertain if the boundaries of the librarian-user relationship are less clearly defined.   “chat” professionalism   two participants were hesitant to label the informal librarian as unprofessional. participant 1 perceived informal grammar as “not unprofessional” but rather showing the librarian was a “real person.” participant 2 perceived informal language as appropriate to the chat environment, stating, “the language that she’s using is not professional but that’s what you expect from chatting . . . it’s better than writing full sentences and taking so much time in writing them.” these findings suggest that members of this user-group may be more accepting of informality in the virtual reference interaction and may not immediately associate informality with unprofessionalism.   satisfaction   durrance (1989) defines satisfaction as the willingness to return to or work with a librarian in the future. thus, to gauge satisfaction, participants were asked whether they would be willing to ask their own question of each librarian or if there was a librarian they would prefer to interact with. willingness to visit the librarians was notably divided. participants 3, 4, and 5 stated that they would not ask a question of the informal librarian whom they perceived as untrustworthy. these participants preferred the formal librarian, based on perceived authority:   i would definitely prefer the [formal] one . . . like i said i love the enthusiasm that comes with [the informal one] and the initial approachability that i felt, but i don’t feel like it was as good quality of an interaction as the [formal one]. (participant 3)   participants 1 and 2 were willing to visit both, yet preferred the informal librarian due to perceived interpersonal connection:   if it was me, i’d probably go to the happy one [informal] because . . . let’s say i’m panicking and i’m finishing up a last minute assignment . . . i wanna talk to someone who makes me feel just a little bit more encouraged. (participant 1)   participant 2 echoed this, stating, “i would prefer to go to the one that’s actually using the language . . .  of our age group cause you’re more comfortable with the person.”   limitations   participants in the study were not personally invested in the reference interactions presented to them. contextual markers of formality, such as humour, self-deprecation, and hedging (brennan & ohaeri, 1999), have been shown to impact the reference interaction (radford, 2006; westbrook, 2007), but because participants were not involved in the interactions, contextual markers could not be studied. in addition, due to the limited sample size of this pilot study, the responses of graduate and undergraduate students could not be reliably compared. future research may expand upon this study by analyzing the differences in the responses of graduate and undergraduate students, who may have different expectations regarding virtual reference interactions. finally, though formality levels in this study were purposefully static, in the typical reference exchange formality rises and falls during different stages of the reference interview (westbrook, 2007). future studies may seek to determine whether perceptions differ when formality levels fluctuate during the reference interaction.   conclusion   the findings of this study suggest that formality plays a key role in altering the perceptions of 17to 25-year-olds in relation to virtual reference interactions. both communication styles appeared to have benefits and drawbacks. whereas participants often perceived formal language as suggesting competency and trustworthiness, it also was interpreted as robotic, impersonal, and at times condescending. conversely, informal language suggested approachability, enthusiasm, and interpersonal connection, but also youth and inexperience. future research might examine how perceptions differ for different user groups and in different environments. would users of a public library’s vrs have similar expectations of formality and social distance? or would older users be as comfortable with informal language?   future studies may build upon the findings by investigating whether formality levels may be altered at points in the reference interaction to facilitate satisfaction. would using more informal language to greet, express interest, or invite users to return while using higher formality to provide answers build rapport while encouraging trust? might librarians be trained to understand politeness theory and respond to users with the appropriate cmc communication style?   what is clear from these findings is that formality plays a pivotal role in cmc and that librarians communicate far more than information in the virtual reference interaction. building upon these findings may help librarians develop a repertoire of communication tools, increasing their ability to better communicate in the virtual reference environment, and thereby increasing the effectiveness of this essential service.     references   arnold, j., & kaske, n. (2005). evaluating the quality of a chat service. portal: libraries and the academy, 5(2), 177-193.   arundale, r. b. (2005). pragmatics, conversational implicature and conversation.  in k. l. fitch & r. e. sanders (eds.), handbook of language and social interaction (pp. 41-63). mahwah, nj: lawrence erlbaum associates.   baron, n. s. (2004). see you online: gender issues in college student use of instant messaging. journal of language and social psychology, 23, 397-423.   brennan, s. e., & ohaeri, j. o. (1999). why do electronic conversations seem less polite? the costs and benefits of hedging. proceedings, international joint conference on work activities, coordination, and collaboration (wacc ’99) (pp. 227-235).  san francisco, ca.   brown, p., & levinson, s. c. (1987). politeness: some universals in language usage. cambridge, ny: cambridge university press.   burke, m., & kraut, r. (2008). mind your p’s and q’s: when politeness helps and hurts in online communities. proceedings from the acm chi conference. florence, italy.   cloughley, k. (2004). digital reference services: how do the library-based services compare with the expert services? library review, 53(1), 17-23.   connaway, l. s., & radford, m. l. (2010). virtual reference service quality: critical components for adults and the net-generation. libri, 60(2), 165-180.   daft, r. l., & lengel, r. h. (1984). information richness: a new approach to managerial behavior and organization design. research in organizational behaviour, 6, 191-233.   dewdney, p., & ross, c. s. (1994). flying a light aircraft: reference service evaluation from a user’s viewpoint. rq, 34(2), 217-231.   durrance, j. c. (1989). reference success: does the 55 percent rule tell the whole story? library journal, 114(7), 31-36.   fagan, j. c., & desai, c. m. (2003). communication strategies for instant messaging and chat reference services. reference librarian, 38(79/80), 121-155.   forde, a. (2011). evolutionary theory of mate selection and partners of trans people: a qualitative study using interpretative phenomenological analysis. the qualitative report, 16(5), 1407-1434.   goffman, e. (1956). the nature of deference and demeanor. american anthropologist, 58(3), 473-502.   groenewald, t. (2004). a phenomenological research design illustrated. international journal of qualitative methods, 3(1). retrieved 28 oct. 2011 from http://www.ualberta.ca/~iiqm/backissues/3_1/pdf/groenewald.pdf   gubrium, j. f. (2001). handbook of interview research: context & method. thousand oaks, ca: sage publications.   haas, c., & takayoshi, p. (2011). young people’s everyday literacies: the language features of instant messaging. research in the teaching of english, 45(4), 378-404.   houlson, v., mccready, k., & pfahl, c. s. (2006). a window into our patron’s needs: analyzing data from chat transcripts. internet reference services quarterly, 11(4), 19-39.   hyener, r. h. (1999). some guidelines for the phenomenological analysis on interview data. in a. bryman & r. g. burgess (eds.), qualitative research (vol. 3, pp. 143-164). london: sage.   jessmer, s. l., & anderson, d. (2001). the effect of politeness and grammar on user perceptions of electronic mail. north american journal of psychology, 3(2), 331-346.   johnson, c. m. (2004). online chat reference: survey results from affiliates of two universities. reference & user services quarterly, 43(3), 243-247.   kim, s. j. (2005). collaborative interaction behaviors in an information technology problem-solving context: cognitive movements of the helper and the helped. journal of information science, 31(6), 483-495.   kwon, n., & gregory, v. l. (2007). the effects of librarians’ behavioral performance on user satisfaction in chat reference services. reference & user services quarterly, 47(2), 137-148.   kvale, s. (1996). interviews: an introduction to qualitative research interviewing. thousand oaks, ca: sage publications, inc.   leedy, p. d., & ormrod, j. e. (2009). practical research: planning and design (9th ed.). upper saddle river, nj: pearson.   maness, j. m. (2008). a linguistic analysis of chat reference conversations with 18-24 year-old college students. journal of academic librarianship, 34(1), 31-38.   mon, l. m. (2006). user perceptions of digital reference services. (unpublished doctoral dissertation). university of washington.   mon, l., & janes, j. w. (2007). the thank you study. reference & user services quarterly, 46(4), 53-59.   morand, d. a., & ocker, r. j. (2002). politeness theory and computer-mediated communication: a sociolinguistic approach to analyzing relational messages. proceedings from the 36th hawaii international conference on system sciences.   nilsen, k. (2004). the library visit study: user experiences at the virtual reference desk. information research, 9(2). retrieved 1 dec. 2011 from http://informationr.net/ir/92/paper171.html   park, j. (2008a). linguistic politeness and face-work in computer-mediated communication, part 1: a theoretical framework. journal of the american society for information science and technology, 59(13), 2051-2059.   park, j. (2008b). linguistic politeness and face-work in computer-mediated communication, part 2: an application of the theoretical framework. journal of the american society for information science and technology, 59(14), 2199-2209.   park, j. (2007). interpersonal and affective communication in synchronous online discourse. library quarterly, 77 (2), 133-155.   patton, m. q. (2001). qualitative research & evaluation methods (3rd ed.). thousand oaks, ca: sage publications, inc.   pew internet and american life survey. (2010). millennials: a portrait of generation next. retrieved 1 dec. 2011 from http://www.pewsocialtrends.org/files/2010/10/millennials-confident-connected-open-to-change.pdf   pew internet and american life survey. (2004). how americans use instant messaging. retrieved 1 dec. 2011 from http://www.pewinternet.org/~/media//files/reports/2004/pip_instantmessage_report.pdf.pdf   radford, m. l. (2006). encountering virtual users: a qualitative investigation of interpersonal communication in chat reference. journal of the american society for information science and technology, 57(8), 1046-1059.   radford, m. l. (1998). approach or avoidance? the role of nonverbal communication in the academic library user’s decision to initiate a reference encounter. library trends, 46(4), 699-717.   reference and user services association [rusa]. (2004). guidelines for behavioral performance of reference and information service providers. retrieved 9 mar. 2013 from http://www.ala.org/rusa/resources/guidelines/guidelinesbehavioral   rice, r. e., & love, g. (1987). electronic emotion: socioemotional content in a computer-mediated communication network. communication research, 14(1), 85-108.   ross, c. s., & dewdney, p. (1998). negative closure: strategies and counter-strategies in the reference transaction. reference & user services quarterly, 38(2), 151-163.   rourke, l., & lupien, p. (2010). learning from chatting: how our virtual reference questions are giving us answers. evidence based library and information practice, 5(2), 63-74. retrieved 1 nov. 2011 from http://ejournals.library.ualberta.ca/index.php/eblip/article/viewfile/6927/6961   seidman, i. (2005). interviewing as qualitative research: a guide for researchers in education and the social sciences (3rd ed.). new york, ny: teachers college press.   silverstein, j. (2006). digital reference services: recommendations for supporting children’s informal learning. in m. k. chelton & c. cool (eds.), youth information-seeking behaviour ii (pp. 257-277). lanham, md: scarecrow press inc.   sweeney, r. t. (2006). millennial behaviors & demographics. retrieved 8 mar. 2013 from http://certi.mst.edu/media/administrative/certi/documents/article-millennial-behaviors.pdf   thompson, s. (2010). to script or not to script: or, the challenge of being both efficient and personal when interacting with virtual reference patrons. reference librarian, 51(4), 363-366.   walther, j. b., & d’addario, k. p. (2001). the impacts of emoticons on message interpretation in computer mediated communication. social science computer review, 19(3), 324-347.   westbrook, l. (2007). chat reference communication patterns and implications: applying politeness theory. journal of documentation, 63(5), 638-658.   white, m. d., abels, e. g., & kaske, n. (2003). evaluation of chat reference service quality: pilot study. d-lib magazine, 9(2). doi: 10.1045/february2003-white         appendix a informal transcript   line numbers added after data collection.   scenario: an undergraduate student completing their education degree requests information on how to find a journal article that is about motivating high school students to learn about science. the student is talking to a librarian, in real time, using chat (“ask a question”).   1 librarian: hi! i can help with that! 2 librarian: what do you need it for? 3 patron: it’s for a class 4 librarian: ok . . . can you tell me more about your topic? 5 patron: i have to research how to motivate students to learn about science 6 librarian: are you looking for a specific age range? or type of science . . . chem? bio?? 7 patron: high school students . . . it can be about any kind of science. 8 librarian: ok cool . . . have u tried searching in eric? 9 patron: no . . . what’s that? 10 librarian: it’s a database for education . . . it will probably have articles on your topic 11 patron: ok, how do i get there? 12 librarian: go to the library site: [url]. 13 librarian: and click on the databases link . . . it’s on the left-hand side. 14 patron: ok 15 librarian: let me know when you’re there! 16 patron: i’m there now 17 librarian: if you click on the e and scroll down u will find eric . . . 18 librarian: when you find it click on the link 19 patron: ok, it’s open 20 librarian: in eric we can try searching “motivation and high school science” 21 patron: k 22 patron: do i just type it where it says keyword search? 23 librarian: yep. then click search!! 24 librarian: do any of the articles look useful? 25 patron: yeah, some of them look good. 26 patron: thanks!! 27 librarian: np j . . .  do you need help w. anything else 28 patron: nope 29 librarian: please come back if you need more help!     appendix b formal transcript   line numbers added after data collection.   scenario: an undergraduate student completing their education degree requests information on how to find a journal article that is about motivating high school students to learn about science. the student is talking to a librarian, in real time, using chat (“ask a question”).   1 patron: i need to find an article on teaching high school science 2 librarian: hello. i can help you. 3 librarian: what do you need this information for? 4 patron: it’s for a class 5 librarian: okay. would you tell me more about your topic? 6 patron: i have to research how to motivate students to learn about science 7 librarian: what age of students are researching? are you interested in a particular field of science, such as chemistry or biology? 8 patron: high school students . . . it can be about any kind of science. 9 librarian: that sounds interesting. have you tried searching in eric? 10 patron: no . . . what’s that? 11 librarian: it is an educational database that will probably have articles on your topic. 12 patron: ok, how do i get there? 13 librarian: go to the library website: [url] 14 librarian: and click on “databases” link. it’s on the left-hand side. 15 patron: ok 16 librarian: let me know when you are there . . . 17 patron: i’m there now 18 librarian: if you click on “e” and scroll down you’ll find eric. 19 librarian: when you find it, click on the link. 20 patron: k, it’s open 21 librarian: now that we are in the database, let’s try searching “motivation and high school science” 22 patron: k 23 patron: do i just type it where it says keyword search? 24 librarian: yes. then press the “search” button. 25 librarian: do any of these articles look useful? 26 patron: yeah, some of them look good. 27 patron: thanks!! 28 librarian: you are welcome. do you need help with anything else? 29 patron: nope 30 librarian: please feel free to contact us again if you require additional assistance.   evidence based library and information practice 2012, 7.4 105 evidence based library and information practice evidence summary public library training program for older adults addresses their computer and health literacy needs a review of: xie, b. (2011). improving older adults’ e-health literacy through computer training using nih online resources. library & information science research, 34, 63-71. doi: /10.1016/j.lisr.2011.07.006 reviewed by: cari merkley associate professor mount royal university library calgary, alberta, canada email: cmerkley@mtroyal.ca received: 1 sept. 2012 accepted: 26 oct. 2012 2012 merkley. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to evaluate the efficacy of an ehealth literacy educational intervention aimed at older adults. design – pre and post intervention questionnaires administered in an experimental study. setting – two public library branches in maryland. subjects – 218 adults between 60 and 89 years of age. methods – a convenience sample of older adults was recruited to participate in a four week training program structured around the national institutes of health toolkit helping older adults search for health information online. during the program, classes met at the participating libraries twice a week. sessions were two hours in length, and employed hands on exercises led by master of library science students. the training included an introduction to the internet, as well as in depth training in the use of the nihseniorhealth and medlineplus websites. in the first class, participants were asked to complete a pretraining questionnaire that included questions relating to demographics and previous computer and internet experience, as well as measures from the computer anxiety scale and two subscales of the attitudes toward computers questionnaire. participants between september 2008 and june 2009 also completed pre-training computer and web mailto:cmerkley@mtroyal.ca http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/ evidence based library and information practice 2012, 7.4 106 knowledge tests that asked individuals to label the parts of a computer and of a website using a provided list of terms. at the end of the program, participants were asked to complete post-training questionnaires that included the previously employed questions from the computer anxiety scale and attitudes towards computer questionnaire. new questions were added relating to the participants’ satisfaction with the training, its impact on their health decision making, their perceptions of public libraries, and the perceived usability and utility of the two websites highlighted during the training program. those who completed pre-training knowledge tests were also asked to complete the same exercises at the end of the program. main results – participants showed significant decreases in their levels of computer anxiety, and significant increases in their interest in computers at the end of the program (p>0.01). computer and web knowledge also increased among those completing the knowledge tests. most participants (78%) indicated that something they had learned in the program impacted their health decision making, and just over half of respondents (55%) changed how they took medication as a result of the program. participants were also very satisfied with the program’s delivery and format, with 97% indicating that they had learned a lot from the course. most (68%) participants said that they wished the class had been longer, and there was full support for similar programming to be offered at public libraries. participants also reported that they found the nihseniorhealth website more useful, but not significantly more usable, than medlineplus. conclusion – the intervention as designed successfully addressed issues of computer and health literacy with older adult participants. by using existing resources, such as public library computer facilities and curricula developed by the national institutes of health, the intervention also provides a model that could be easily replicated in other locations without the need for significant financial resources. commentary this work is part of larger study entitled “electronic health information for lifelong learners,” and preliminary results were published in same journal in 2009. in this article, the researcher makes a strong case for greater involvement of the library and information community in addressing issues of health and e-health literacy. while the stated overall objective is to look at the effectiveness of the intervention studied in addressing e-health literacy among older adults, many of the measures employed focus exclusively on computer literacy skills or the usability of particular websites that focus on health information, with a small number of subjective questions relating to participants’ use of health information in their own lives. ehealth literacy is defined by norman and skinner (2006) as “the ability to seek, find, understand, and appraise health information from electronic sources and apply the knowledge gained to addressing or solving a health problem” (the lily model) (¶abstract section). it is apparent from this definition that e-health literacy is too multifaceted to be addressed by a single intervention, even one that spans eight classes. while the researcher is very clear about which aspects of this concept she seeks to address with her study design, it is hoped that more emphasis in future research will be placed on investigating the impact of training on the understanding, appraisal, and application of health information by older adults. in the study, the module relating to appraising health information was only presented to classes who finished the other eight modules earlier than expected. it is not clear if this module was sacrificed in the interests of time, but it could be argued such a skill has a longer potential shelf life than a particular resource. an evaluation of the study using the eblip critical appraisal checklist (glynn, 2006) highlighted a few areas where the reporting of results could be improved. one, providing the number of responses received for questions relating to participant satisfaction with the training, use of information in decision evidence based library and information practice 2012, 7.4 107 making, and views of public libraries would help place the percentages presented into context. a very small number of individuals completed both the pre and post knowledge tests (20 for computer skills, 15 for web skills), limiting the utility of the data collected through the sole objective measure employed in the study. this should be highlighted, although the researcher acknowledges the study would be strengthened with the inclusion of more objective measures of search skills. more information on the course dropouts mentioned briefly by the researcher would also provide a more inclusive portrait of the population under study. the inclusion of the questionnaires and knowledge tests as appendices would have been useful for future researchers hoping to replicate the study design. xie acknowledges a number of the study’s limitations. a convenience sample was employed, which significantly impacts the external validity of the study. having 13 different instructors deliver the workshop also likely impacted participant responses. what is not fully explained was the decision to recruit from library branches that serve large number of patrons from ethnic minority groups. xie briefly suggests a particular need among minority populations for health literacy interventions, but the literature review focuses on older adults as a whole and the study results are not parsed by the ethnic background of participants. further research on this issue may be needed. the study’s value to the library community lies in several areas. it outlines a successful training program that uses existing infrastructure and materials to deliver computer literacy instruction that also serves to address some health literacy needs. from its use of existing learning materials to the recruitment of instructors from the nearby library school, it is clear that the sustainability and transferability of the program to other locations was considered. the intentionality of the course design and its use of literature on learning theory and pedagogy is also a good model for others to follow in future. references glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi: 10.1108/07378830610692154 norman, c. d., & skinner, h. a. (2006). ehealth literacy: essential skills for consumer health in a networked world. journal of medical internet research, 8(2), e9. retrieved from http://www.jmir.org/2006/2/e9/ editorial   waiting for evidence   alison brettle editor-in-chief senior lecturer, school of nursing midwifery and social work university of salford salford, united kingdom email: a.brettle@salford.ac.uk      2013 brettle. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     as library practitioners we may often want evidence to help us make decisions or to provide a rationale for what we do.  we may then get frustrated that the research evidence doesn’t exist or doesn’t quite match our needs. this occurred to me when reading recent articles about searching in relation to systematic reviews (e.g., gehanno, et al., 2013; nourbakhsh, et al., 2012) which in turn reminded me of one of my first forays into evidence based librarianship. i  was working on a project investigating the feasibility of undertaking systematic reviews in social care (long et al., 2002a; 2002b) and one of my roles was to identify a set of “best” databases for searching in this field (brettle & long, 2001).  i soon realised not only that this was a difficult task, but that one of the differences between evidence based practice in social care and evidence based practice in medicine (whose practice we were emulating) is that the questions, the answers, and the evidence needed to obtain those answers are all more messy or fuzzy than a clinical question which can be broken down by pico (richardson et al., 1995). the overall conclusion of the social care project was that, despite this messiness and fuzziness, it was still possible to adopt a systematic approach and to identify “best” evidence (long et al, 2002a) and thus to undertake evidence based social care.    the same can be said about evidence based library and information practice; the interventions that we are involved in and the decisions we make often don’t involve cause and effect, and because of this there won’t be a clear-cut answer. this doesn’t mean there is no evidence or that we can’t be evidence based; it’s just that the medical hierarchy of evidence (guyatt et al., 1995) doesn’t fit, a point also made by crumley and koufogiannakis (2002). as in social care, we need to ensure that our view of evidence is a broad one but as my recent reading on systematic reviews suggests, we also need to be patient in waiting for answers. we need to think about building up a picture of evidence for our practice rather than hoping (or expecting) that one piece of research will provide the answers we need. my research into databases, mentioned earlier, seemed to throw up more research questions than i answered. i was able to provide an answer for a very specific topic but this couldn’t be generalised for all topics or all databases, so it didn’t help me a great deal in further searches or in teaching information literacy. my recent reading provides additional pieces of evidence about when certain resources are more appropriate than others, as well as generating a number of questions regarding methodology. this incomplete picture of evidence is good news – it gives practitioner researchers working in library and information practice plenty of questions to investigate, which will ultimately generate a better overall picture of evidence.   this march issue of eblip contains a wide variety of research articles, evidence summaries, reviews and commentaries. i hope it helps you build up a useful picture of the evidence you need for your practice.     references   brettle, a. j., & long, a. f. (2001). comparison of bibliographic databases for information on the rehabilitation of people with severe mental illness. bulletin of the medical library association, 89(4), 353-362.   crumley, e. & koufogiannakis, d. (2002). developing evidence-based librarianship: practical steps for implementation. health and information libraries journal, 19(2), 61-70. doi:10.1046/j.1471-1842.2002.00372.x   gehanno, j.-f., rollin, l. & darmoni, s. (2013). is the coverage of google scholar enough to be used alone for systematic reviews? bmc medical informatics and decision making, 13(7), doi:10.1186/1472-6947-13-7   guyatt, g.h., sackett, d.l., sinclair, j.c., hayward, r., cook, d.j., cook, r.j., bass, e., gerstein, h., haynes, b., & holbrook, a. et al. (1995). users' guides to the medical literature: ix. a method for grading health care recommendations. jama, 274(22), 1800-1804. doi:10.1001/jama.1995.03530220066035   long, a.f., godfrey, m., randall, t., brettle, a., & grant, m. j. (2002a). developing evidence based social care policy and practice. part 3: feasibility of undertaking systematic reviews in social care. leeds: university of leeds, nuffield institute for health. university of salford, health care practice r&d unit.   long, a., godfrey, m., randall, t., brettle, a., & wistow, g. (2002b). developing evidence based social care policy and practice. part 1: effectiveness and outcomes of rehabilitation for people with severe and enduring mental illness. leeds: university of leeds, nuffield institute for health. university of salford, health care practice r&d unit.   nourbakhsh, e. f., nugent, r. f., wang, h. f., cevik, c. f., & nugent, k. medical literature searches: a comparison of pubmed and google scholar. (2012). health information and libraries journal, 29(3), 214-222. doi: 10.1111/j.1471-1842.2012.00992.x   richardson, w.s., wilson, m.c., nishikawa, j. & hayward. r.s. (1995). the well built clinical question: a key to evidence based decisions [editorial], acp journal club, 123(2), a12-13.   evidence summary   survey applies public collection development librarians' support for intellectual freedom to collection process   a review of: oltmann, s. m. (2019). important factors in midwestern public librarians’ views on intellectual freedom and collection development: part 2. the library quarterly, 89(2), 156-172. https://doi.org/10.1086/702203   reviewed by: laura costello virtual reference librarian rutgers university libraries rutgers, the state university of new jersey new brunswick, new jersey, united states of america email: laura.costello@rutgers.edu   received: 1 dec. 2019                                                                    accepted:  20 jan. 2020      2020 costello. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29686     abstract   objective – to explore how librarian attitudes regarding intellectual freedom and demographic factors influence collection development decisions.    design – online survey.   setting – public libraries in the midwestern united states.   subjects – 645 collection development library professionals employed in public libraries.   methods – an electronic survey was distributed to 3,018 public library directors in nine midwestern states and completed by the library professional primarily responsible for collection development (oltmann, 2019, p. 6). the survey had a 21.37% response rate. the survey focused on intellectual freedom in the management of collections and probed the participants for their experiences and influences in making collection development decisions. the survey also asked participants to make hypothetical purchasing and holdings decisions for library materials based on a short description of the material.   main results – participants indicated that they used a variety of different tools for the selection of materials including patron requests. of the participants, 45.7% indicated that their library had a policy, practice, or metric to assess the balance of their collections, while 54.3% indicated that their libraries did not have policy or method in place for ensuring that their collection was balanced. of the respondents, 73.4% felt that local community values should be considered in collection development decision, but 62.3% said that this should not be the most important factor in decisions. overall, the political leaning of the community did not have an impact on participants’ alignment with the ala's stances on intellectual freedom. most respondents (73.4%) felt that government library funding bodies should have an influence over collection development decisions. some respondents indicated they felt internal pressure from other library staff or the library board to purchase particular materials (28.1%) or relocate materials (14.1%). respondents also indicated that they felt external pressure from their communities to purchase (32%) or restrict or withdraw (19.1%) materials. in the hypothetical purchasing scenario, most librarians indicated that they would purchase the majority of items. some participants (39.8%) felt tension between their personal and professional views on intellectual freedom.   conclusion – the first part of this article found that holding an mls degree had a significant impact on participants' stance on intellectual freedom and alignment with the american library association (ala) principles. this part indicated that they also felt greater pressure to withdraw, acquire, and manage particular materials in their collections and felt more tension between their personal and professional stances on intellectual freedom. age, gender, duration of work, and community political affiliations significantly impacted only some of the participants' responses. overall, there was general support for intellectual freedom and alignment with the ala principles; however, 40% of respondents indicated tension between their personal and professional beliefs about intellectual freedom.    commentary   this article is the second part in a pair of articles focusing on the same data. an evidence summary for the first part was published in evidence based library and information practice (costello, 2019). in the first part, oltmann (2019) examined the perspectives of librarians on intellectual freedom while the second part focuses on the collection development implications of those views. this article demonstrates that the strong support for principles of intellectual freedom participants demonstrated in the first part of the article bears out in their hypothetical collection development actions.   the hypothetical purchasing scenarios featured in the survey aligned with the stated views of the participants, but some of the comments seemed to indicate that the participants wanted more information about materials to be able to make their collection development decisions. the application portion of this survey may have worked better qualitatively. participants largely approved of theoretical books that had the potential to attract challenges, but it may have been more informative to hear about particular real purchases or selections that had been challenged and the actual actions participants took to negotiate those challenges. because the survey addressed hypothetical rather than real collection development choices and the librarians were not able to use the strategies they would ordinarily use to address intellectual freedom concerns, the responses may not be representative.   the author found that most participants did not have a specific plan to ensure balanced collection development and comments from respondents in this area indicated that some librarians relied on the contributions of patrons to fully develop parts of their collections. balance-finding is a particularly important strategy in these new acquisitions strategies since patron requests can overdevelop areas of interest (tyler, hitt, nterful, & mettling, 2019; blume, 2019). without a structured development plan in place, building collections from patron interests may stray from the guidelines promoted by the ala.   this article reconfirms the commitment of public collection development librarians to the intellectual freedom standards of the ala. though librarians support these standards in their ideas and actions surrounding collection development, a strong minority felt tension between their professional and personal feelings on intellectual freedom. in the limitations section of the article, the author draws attention to the fact that terms were not defined in the survey, so interpretations of terms like "tension" and "pressure" may have differed between participants. in the discussion, oltmann also suggests that a tension between personal and professional views might require further education of librarians or a revision of the standards, but it may just as easily be a natural part of working in intellectual freedom and a span of interpretations of the meaning of tension. in light of the finding that the political stances of communities do not have a significant impact on the intellectual freedom support of participants, it seems plausible that the tension might only represent a professional standard that is upheld despite a diversity of personal ethical frameworks. as the author suggests, a deeper qualitative look into the perspectives of collection development librarians could help augment the findings of this study.   references   blume, r. (2019). balance in demand driven acquisitions: the importance of mindfulness and moderation when utilizing just in time collection development. collection management, 44(2-4), 105-116. https://doi.org/10.1080/01462679.2019.1593908   costello, l. (2019). survey confirms strong support for intellectual freedom in public collection development librarians. evidence based library and information practice,14(3), 135-137. https://doi.org/10.18438/eblip29577   oltmann, s. m. (2019). important factors in midwestern public librarians’ views on intellectual freedom and collection development: part 1. the library quarterly, 89(1), 2-15. https://doi.org/10.1086/700659   tyler, d. c., hitt, b. d., nterful, f. a., & mettling, m. r. (2019). the scholarly impact of books acquired via approval plan selection, librarian orders, and patron-driven acquisitions as measured by citation counts. college & research libraries, 80(4), 525. https://doi.org/10.5860/crl.80.4.525   3 evidence based library and information practice editorial responsibilities © 2011. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. editor‐in‐chief: denise koufogiannakis associate editor (articles): alison brettle associate editor (evidence summaries): lorie kloda associate editor (classics): jonathan eldredge associate editor (features, commentaries, using evidence in practice, ebl101, news): denise koufogiannakis production editor: katrine mallan editorial intern: michelle dunaway copyeditors: priscilla stephenson (lead copyeditor), georgianne bordner, marcy brown, molly des jardin, richard hayman, dale storie, mary virginia taylor indexing support: pam morgan writing assistance: hope leman, tracy powell iwaskow, carol waseleski / evidence based library and information practice editorial evidence based library and information practice 2009, 4:2 97 evidence based library and information practice article librarian-teacher partnerships for inquiry learning: measures of effectiveness for a practice-based model of professional development joyce yukawa assistant professor graduate program, library and information science, college of st. catherine st. paul, minnesota, united states of america e-mail: jyukawa@stkate.edu violet h. harada professor library and information science program, university of hawaii honolulu, hawaii, united states of america e-mail: vharada@hawaii.edu received: 01 january 2009 accepted: 18 march 2009 © 2009 yukawa and harada. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – this study analyzed the effects of a practice-based model of professional development on the teaching and collaborative practices of 9 teams of librarians and teachers, who created and implemented units of inquiry-focused study with k-12 students during a yearlong course. the authors describe how the collection and analysis of evidence guided the development team in the formative and summative evaluations of the outcomes of the professional development, as well as the long-term results of participation in this initiative. methods – the authors used an interpretive, participative approach. the first author was the external reviewer for the project; the second author headed the development team and served as a participant-observer. triangulated data were collected from participants in the form of learning logs, discussion board postings, interviews, questionnaires, and learning portfolios consisting of unit and lesson plans and student work samples with critiques. data were also collected from the http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2009, 4:2 98 professional development designers in the form of meeting notes, responses to participants, interviews, and course documents. for two years following the end of the formal course, the authors also conducted follow-up email correspondence with all teams and site visits with six teams to determine sustained or expanded implementation of inquiry-focused, collaborative curriculum development. results – the practice-based approach to professional development required continual modification of the course design and timely, individualized mentoring and feedback, based on analysis and co-reflection by the developers on the evidence gathered through participant logs, reports, and school site visits. modeling the inquiry process in their own course development work and making this process transparent to the participating community were essential to improvement. course participants reported beneficial results in both immediate and long-term changes in practice. the summative evaluation identified significant changes in practice in three areas: (1) the design of inquiry-focused learning, (2) the roles of the teacher and librarian in collaborative development of instruction, and (3) the impact on student performance. two years after the yearlong professional development course, most participants indicated that they continued to incorporate inquiry-based approaches, and over half of the participants were involving other colleagues at their schools in inquiry-focused practices. six of the librarians assumed major curricular roles in their respective schools. conclusion – the practice-based model of professional development appears to be effective and sustainable. it has been tested and modified by other development teams in the last two years. more extensive use of the model in other contexts with further testing and refinement by other developers is needed to ensure that the model is robust and widely applicable. introduction professional development alternatives in hawaii have historically been restricted to ineffective, short-term workshops delivered away from actual school sites. moreover, these workshops have traditionally focused on the classroom and ignored the potential of collaborative teacher and librarian teaming on instruction. the purpose of this study was to refine a practice-based model of professional development and investigate its effectiveness through formative and summative assessments of a yearlong course for teams of librarians and teachers collaborating on curriculum development at their schools. our past experiences and those of others (e.g., marzano 66) have shown that typical approaches to professional development, such as one-day workshops and formal courses without opportunities to test strategies learned, are insufficient to affect significant changes in practice. our goal was to refine, implement, and disseminate a practice-based model for more effective professional development through librarian-teacher collaboration. we describe how the collection and analysis of evidence guided the development team toward ensuring that learning outcomes were achieved. modeling the inquiry process in their own course development work and making this process transparent to the participating community were essential to improvement. we present our experiences with the aim of encouraging evidence based library and information practice 2009, 4:2 99 further testing and refinement by other developers in other contexts to ensure that the model is robust and widely applicable. literature review professional development programs are systematic efforts to bring about change in teaching practices. high quality professional development is a central component in nearly every current proposal for improving education (guskey 381). though initiatives vary widely in their content and format, the desired outcome is student learning. researchers have increasingly focused on what makes professional development effective, exploring the complex links between the design of professional development, teachers’ learning during planned activities, and subsequent changes in classroom practice (penuel et al. 923; sato, wei, and darling-hammond 669). the view of the learning process that emerges across different theories of adult learning is a constructivist perspective with the following propositions (smylie 94). first, adults enter learning situations with accumulated knowledge, skills, and beliefs from past experiences. these may affect current learning by serving as cognitive and normative schemata through which individuals perceive and interpret their situations, new information, and themselves as learners. second, adult learning is problem-oriented and occurs when problems relate in meaningful ways to life situations. third, adults play an active role in their own learning and are not merely passive recipients of information. reflective practice occupies a position at the confluence of a number of ways of knowing for adult learners (ghaye and ghaye 18). a reflective educator is one who ‚is committed to continuous improvement in practice; assumes responsibility for his or her own learning; demonstrates awareness of self, others, and the surrounding context; develops the thinking skills for effective inquiry; and takes action that aligns with new understandings‛ (york-barr et al. 10). social constructivism also contributes to the foundation upon which current professional development is shaped. fenwick and tennant define adult learning as engaging in changing processes of human participation in a particular community of practice (62-63). a community of practice is any group of individuals who work together over time developing particular ways of doing things and talking about things that their members come to learn. wenger declares that individuals learn as they participate by interacting with the community (its history, assumptions, rules and patterns of relationship), the tools at hand, and the moment’s activity (11-15). a practice-based approach that embraces reflective practice offers a potentially effective means to achieve professional growth. the practice-based approach to inservice training (ball and cohen 3-32) uses authentic records and tools for teaching and learning to create a common ground for individuals and teams to work, jointly reflect, explore alternatives, and support each other. the curriculum centers on the tasks, questions, and problems faced in ongoing efforts to design units of instruction and strategies for assessing student learning. this approach recognizes that curriculum reform involves just-in-time, situated learning and acknowledges that the processes of teaching and learning are ambiguous, complicated, and nonlinear. instead of definitive answers and preordained solutions, participants focus on possibilities, methods of reasoning, and alternative conjectures as they learn from one another (yukawa, harada, and suthers 181). the consensus view posited by teacher evidence based library and information practice 2009, 4:2 100 learning researchers is that effective professional development should be designed to develop teachers’ capacity to work collectively on problems of practice within their own schools and with practitioners in other settings, as much as to support the knowledge and skill development in individual educators (elmore 96). key features identified in a review of the literature included the following: a well-articulated mission or purpose anchored in student learning and derived from analysis of student learning of specific content in a specific setting connection with issues of instruction and student learning of academic disciplines and skills in actual classrooms development and maintenance of collaborative practice within schools and across schools focus over time with the use of assessment and evaluation for continuous improvements timely supports for and feedback on teacher learning and practice (borko and putnam 38-51; elmore 96; nesbit, leach, and foley 85-90) in this article, we will describe design principles used in a yearlong practice-based professional development course, the specific objectives and interventions employed, and information on the development team and participants. design of the professional development course the professional development (pd) course, entitled ‚building inquiry partnerships,‛ employed a practice-based approach that was founded on principles of constructivist and adult learning, and was structured to support participants as they designed and implemented units of inquiry-focused study for k-12 students. development team the course development team consisted of three persons with dual roles as designers and learning facilitators/mentors. the lead designer and senior mentor was a professor in an lis program with extensive experience teaching and facilitating workshops. a second senior mentor was a retired library media specialist, experienced practitioner, and skilled facilitator. the third mentor was a staff member of the state’s school library services branch, skilled in the use of technology for learning. in addition, an external researcher attended all planning meetings and collected and analyzed data, as described more fully below. the external reviewer and the lead designer served as the investigators for this study. the population nine teacher-librarian teams from k-12 schools (a total of 21 female participants) were the focus of this study. there were four elementary school teams, one intermediate school team (with one librarian and two teachers), and four high school teams (one with two librarians and one teacher, and another with one librarian and two teachers). the team members had a wide range of school level experience, from a librarian in her second year of service to teachers nearing retirement. some teams had prior collaboration experiences, while others were collaborating for the first time. the teams were formed at the beginning of the pd course, when 10 librarians brought teacher partners with them at the start of the course in june 2005. participants in the pd course not included in this study were three librarians who began the course alone but later secured partners after the school year started. their teacher partners did not participate in the pd course. evidence based library and information practice 2009, 4:2 101 design principles the general design principles for the course were as follows (see also figure 1): embrace reflective practice for continuous improvement through awareness, deep thinking, and action that aligns with new understandings. incorporate authentic records and tools for teaching and learning that center on tasks, questions, and problems situated in practice. support a community of practice that provides common ground for work, collaboration, exploration, co-reflection, and mutual support. connect to patterns of practice in context, building from knowledge, skills, and beliefs from past experiences to develop effective practices in one’s classroom, school, and with other practitioners. promote formative and summative assessment in a continuous cycle of evidence-based practice. pd learning outcomes the designers identified the following desired outcomes for participants: fig. 1. practice-based pd model evidence based library and information practice 2009, 4:2 102 design an inquiry-based unit that connects to content standards and information literacy standards. determine essential questions through focusing on a generative theme and transforming standards into learning objectives. foster the inquiry process through student performance tasks that measure the learning goals and strategies that motivate curiosity, call for higher-level thinking, and support problem investigation. achieve assessment-driven decision making through collecting formative and summative assessment data and analyzing the data to inform instruction. these desired outcomes were specified as ten pd learning outcomes in the course syllabus (see also figure 1): 1. focus on a generative theme or problem. 2. identify one or more essential questions that drive the project. 3. transform standards into clearly stated learning criteria. 4. define performance tasks that clearly measure the learning goals. 5. incorporate strategies that motivate student curiosity. 6. incorporate strategies, which challenge students to generate higher level questions. 7. incorporate strategies that help students investigate their theme or problem effectively and efficiently. 8. incorporate tasks that assist students in creating personal knowledge from collected information and data. 9. require final products that challenge students to effectively communicate their knowledge. 10. integrate opportunities for students to (a) assess students’ own progress throughout their work, and (b) evaluate their final products. scaffolding strategies, support structures and tools scaffolding strategies, support structures, and tools were designed to promote and embody the design principles. the major strategies used by mentors to scaffold learning of new concepts and practices were as follows (see also figure 1): 1. engage the learner: tap prior knowledge, encourage questions. 2. introduce new learning: connect new concepts or skills to prior knowledge, demonstrate and model new concepts or skills. 3. provide for application and transfer: guide practice, provide timely feedback. 4. allow for independent practice: focus on site-based, work-embedded implementation of new learning. 5. promote interaction: encourage collaborative curriculum planning, cross-school dialogue, sharing of products. 6. provide continuous mentoring: encourage participation, show interest in progress, respond positively to inquiries, provide helpful feedback, extend the inquiry in key directions. 7. build critical reflection and coreflection: engage in ongoing assessment of progress and reflection on planning and implementation experiences. the support structure consisted of face-toface meetings and ongoing online dialogue. in june 2005, the course began with a threeday, face-to-face summer institute that was a critical engagement activity. the institute modeled an inquiry approach by challenging participants to think deeply about the following types of questions: evidence based library and information practice 2009, 4:2 103 what is worth learning? how do we guide students for this learning to occur? how might students demonstrate their learning? how do we know if this understanding has occurred? the mentors introduced the key concepts: (1) determining essential questions, (2) fostering the inquiry process, and (3) assessment-driven decision making. teams engaged in intraand inter-team discussions to share and reflect on their past experiences, thus connecting prior teaching activity with the key concepts. the teams also began the work of collaborative curriculum planning. the mentors introduced a general process for designing and implementing a unit, supported by monthly individual and team reflections and yearlong mentoring. each team was assigned a buddy team with whom to coreflect throughout the year. the teacher-librarian teams continued curriculum development and implementation on site. two other face-toface activities were a midpoint reunion and team presentations at an annual educational conference. the january 2006 midpoint reunion, which was also offered via videoconferencing, celebrated lessons learned, and identified areas of strength and areas for additional support. a critical component of the reunion was to revisit assessment issues. by this time, participants had started collecting assessment data and wanted more guidance in interpreting the results and brainstorming how best to use the findings in refining instruction. the designers invited participants to serve as critical friends to one another. in pairs, they shared their work and suggested ways to interpret the data and strategies for improving instruction. the designers also contributed to the discussion. in their reflections at the end of the day, many participants cited the benefits of collaboratively thinking about and discussing the complexities of assessment. one participant noted, ‚assessment is reflective learning.‛ the other major face-to-face event involved the teams presenting their units in progress at the february 2006 e conference, an annual educational conference sponsored by the state department of education. approximately a thousand teachers, school support personnel, and administrators as well as university faculty attended this conference. our teams were clustered in hour-long panels with each team given 10 to 15 minutes to share major learning goals, assessment strategies, collaborative teaching strategies, and student performances. two teams brought along students to talk about their learning experiences. the overwhelmingly positive responses they received from attendees served as a tremendous morale booster. one participant told the developers: i was very nervous about speaking for the first time at a state conference but what an experience! i had people coming up to me after the presentation to say how impressed they were with the team’s collaborative planning and the results we were getting from the students’ science investigations. i felt truly empowered. in addition to the face-to-face activities, the designers established an online support structure: (1) monthly individual reflection logs via email, (2) monthly team reports posted in workspaces on a university website <http://www.hnlc.org>, and (3) buddy responses in the workspaces to the monthly reports from other teams. the mentors also engaged in continuous online mentoring in hnlc.org and via email. the scaffolding tools included two texts written by the senior mentors that the teams used to help them integrate new concepts http://www.hnlc.org evidence based library and information practice 2009, 4:2 104 and to guide their work. participants were also provided with templates for the unit and lesson plans, a checklist for a generative topic, and a performance task template. supported by these strategies, structures, and tools, teams engaged in collaborative planning and implementation in situated learning through practice. the culminating product was a learning portfolio that included evidence of an inquiry-based unit and assessment-driven decision making (unit and lesson plans, rubrics, concept maps, worksheets, student self-assessment tools, formative and summative assessment tools, student work samples with teacher and librarian commentary at different levels of proficiency) and evidence of reflective practice (cumulative team reports, individual logs, and final reflections). methods of evidence collection and analysis the authors coordinated the collection and analysis of data for this study. the first author was the external reviewer for the project; the second author headed the development team and served as a participant-observer. findings were validated through data triangulation. for evidence of the design team’s work, the investigators collected minutes of planning and debriefing meetings, course agendas, and session plans. in addition, the external reviewer conducted interviews with each member of the design team. for evidence of participant practice, from june 2005 to may 2006, data were collected from individual reflection logs, monthly team reports and buddy responses, and final learning portfolios. a preand post-course questionnaire was administered to ascertain participants’ self-perceived gains on the ten pd learning outcomes (see table 1). five teams (11 participants) were selected for indepth, two-hour interviews from marchmay 2006, using open-ended questions to stimulate discussion (see appendix a). in addition, a questionnaire was administered in may 2006 to the 13 participants who were not interviewed (see tables 2 and 3). to examine longitudinal work beyond the year of the formal professional development, the investigators collected email responses from the teams in the second and third years (from 2006 through 2008) as well as anecdotal records on followup visits with six of the teams. data were coded for elements of the pd model, practice-based design principles, course components, interaction structure, and characteristics of the learning and collaboration processes specific to each participant and team. coder reliability was achieved by conducting member checks in the follow-up interviews. in the following five sections, we elaborate on the foci of the investigation: (1) the use of evidence-based practice to guide the formative development of the course, (2) the summative evaluation of changes in inquiry-focused teaching practices among the participants, (3) the implementation of course design principles, (4) use of technology, and (5) evidence of sustained and expanded practice. evidence based course implementation beyond attending the summer institute and the midpoint reunion, the participants posted over 500 messages during the eight months of the course. the development team met face-to-face at least once a month for approximately three hours and kept in continual contact by email to review the evidence and determine if the strategies, support structures, and tools were achieving the desired outcomes of the course. design modifications were made throughout implementation to better achieve the evidence based library and information practice 2009, 4:2 105 objectives. evidence of the design team’s work and participant practice were examined to document this iterative course design process. the following examples of the process are organized around several key pd learning outcomes for the participants. identifying a generative theme and developing essential questions the first day of the summer institute focused on major elements of an inquiryfocused approach. a key point was identifying a generative theme that is central to a discipline, invites student curiosity, is age-appropriate, offers opportunities for connections to students’ previous experiences, and leads to deeper investigation (perkins 93-95). reviewing the participant feedback, the development team realized that most participants were uncertain about how to develop generative themes. they revised the second day’s agenda to begin with discussing the differences between topics and generative themes and brainstorming strategies for developing generative themes. promoting student self-assessment from the start, participants acknowledged needing help with assessment. while all had conducted summative evaluations to analyze students’ final products, they were novices at formative assessments that involved student self-assessment. although these topics were addressed in the summer institute, the september team reports and logs indicated that this was an ongoing challenge. in response, the developers established an assessment helpline on the course website. they posted the first questions as identified from the reports and logs and welcomed input. while not everyone responded in the helpline discussion, participants viewed the discussion and shared in their subsequent logs and reports a range of assessment strategies including semantic webs, observation and performance checklists, rating scales, and rubrics. getting students to practice self-assessment was a challenge most teams had not faced before. the developers encouraged the use of ‚i can‛ statements with younger students and focused prompts for logs with older students. a number of teams subsequently used both strategies. one team videotaped their students as they rehearsed their oral presentations and encouraged self and peer critiquing of the videotapes. in addition, the developers redesigned the midpoint reunion in january to allow for hands-on time with assessment. participants brought drafts of their units and lessons and traded critiques on assessment strategies and tools. as noted in their subsequent logs, this helped teams adjust their teaching based on ongoing assessment of student performance. defining performance tasks to measure learning goals in early fall, several teams indicated that the plans for their inquiry projects were more complex than originally anticipated and requested additional consultations. in response, the developers conducted site visits with two high school teams. for one, re-examining the expectations for a history day project led to identifying more precise benchmarks for key tasks and devising checklists to allow students to more clearly see the sequence of assignments and associated deadlines. for the other team engaged in capstone projects, discussing how to incorporate quality service learning experiences resulted in the articulation of clearer expectations for both students and community mentors. in both cases, this individualized mentoring helped the teams evidence based library and information practice 2009, 4:2 106 to scaffold and scale the projects through more careful examination of students’ prior knowledge and the skills needed to execute the tasks. lessons learned as the developers read through the monthly reports and the logs, they realized anew the importance of the discourse occurring not only between developers and participants but also among the participants. they encouraged participants to look beyond their buddy teams to other team reports for more cross fertilization of ideas. they modeled the reflection process by sharing their collective reflections on the work being done across teams. each month, they posted a co-reflection on the major themes and topics evidenced in the current logs and reports, shared proposed action steps, posed more questions, and invited comments. these strategies resulted in critical exchanges among participants. an elementary team that had successfully used buddy pairing between 5th and 2nd grade students for note taking activities connected with another elementary team exploring grouping strategies for a multiage classroom to develop guidelines for cross-age pairings. in another example, two high school groups working on project-based learning posed critical questions for one another such as: how will the data collected from the research skills pretest help you to map out future lessons and activities? what skills will you assess? how do you sustain motivation in projects of this duration? what lessons were the developers learning? a practice-based approach to professional development required timely, individualized mentoring and feedback, based on careful analysis of the content and tone of online discourse and active listening during site visits and face-to-face meetings. collaborating as a development team necessitated co-reflecting on qualitative data being gathered through the logs, reports, and visits and using that evidence to work on modifications and introduce new interventions. in short, the developers discovered that modeling the inquiry process in their own course development work and making this process transparent to the participating community were essential to improvement. summative evaluation of course outcomes at the conclusion of the pd course, a summative evaluation was conducted to ascertain changes in inquiry-focused teaching practices among the participants. data were collected through two questionnaires and in-depth interviews with five teams (11 participants), using openended questions (see appendix a). questionnaire 1 was administered to all participants at three points in time: prior to the june 2005 summer institute, after the institute, and in march 2006. the questions addressed self-perceived gains on the ten pd learning outcomes (see table 1). questionnaire 2 was administered in may 2006 to participants who were not interviewed to ascertain any changes in inquiry teaching values (see table 2). additional data were collected through examinations of individual logs, team reports, and final portfolios (especially unit plans and critiqued student work samples). evidence based library and information practice 2009, 4:2 107 table 1: questionnaire 1 – self-perceived gains in knowledge and ability rating scale: 1 = i have limited or no knowledge about this. 2 = i know a little more about this but am not comfortable about applying this. 3 = i feel more comfortable about apply this. 4 = i feel quite comfortable about my knowledge and ability to apply this. key: shaded cells indicate gains over previous selfratings pre-institute post-institute end of course n=25* n=25* n=20 i am able to … 1 2 3 4 1 2 3 4 1 2 3 4 1. identify a theme or a problem that serves as a generative focus for study. 7 8 9 1 0 1 11 13 0 0 9 11 percent of total 28 32 36 4 0 4 44 52 0 0 45 55 2. develop one or more essential questions that relate to the theme or problem. 5 12 7 1 0 0 14 11 0 0 10 10 percent of total 20 48 28 4 0 0 56 44 0 0 50 50 3. transform standards into clearly stated criteria that demonstrate what is learned and how it is learned. 4 10 11 0 0 0 16 9 0 0 11 9 percent of total 16 40 44 0 0 0 64 36 0 0 55 45 4. create a performance task that measures achievement of learning goals. 6 9 10 0 0 1 14 10 0 0 14 6 percent of total 24 36 40 0 0 4 56 40 0 0 70 30 5. incorporate strategies that motivate student curiosity and interest. 3 10 12 0 0 1 15 9 0 1 14 5 percent of total 12 40 48 0 0 4 60 36 0 5 70 25 6. incorporate strategies that encourage students to generate higher-level questions. 4 13 8 0 0 1 14 10 0 1 14 5 percent of total 16 52 32 0 0 4 56 40 0 5 70 25 7. incorporate strategies that help students investigate their theme or problem effectively and efficiently. 4 15 6 0 0 0 17 8 0 1 12 7 percent of total 16 60 24 0 0 0 68 32 0 5 60 35 8. design tasks that assist students in creating personal knowledge from collected information and data. 3 16 6 0 0 2 15 8 0 0 15 5 percent of total 12 64 24 0 0 8 60 32 0 0 75 25 9. develop final products that challenge students to effectively communicate their knowledge. 3 13 9 0 0 0 15 10 0 0 14 6 percent of total 12 52 36 0 0 0 60 40 0 0 70 30 10. integrate opportunities for students to (a) assess students’ own progress throughout their work, and (b) evaluate their final products. 4 13 8 0 0 0 16 9 0 0 16 4 percent of total 16 52 32 0 0 0 64 36 0 0 70 30 * one teacher withdrew from the course, leaving 24 participants who completed. evidence based library and information practice 2009, 4:2 108 table 2: questionnaire 2 – changes in values of an inquiry-focused approach post-course change-in-importance indicators: = decreased in importance; 0 = no change; + = increased in importance n = 7. note: not all questions answered by every respondent indicators for an inquiry-focused class or library—the students . . . post-course change 0 + exhibit curiosity, ask meaningful questions. 0 1 6 collect and evaluate information. 0 1 6 actively engage in hands-on tasks and learning by doing. 0 2 5 express their ideas in a variety of ways (e.g., writing, graphing, mind mapping). 0 2 5 use performance indicators to assess their own work and/or critique their peer’s work. 0 3 4 exercise responsible and ethical use of resources and equipment. 0 3 4 remain on task. 0 3 4 articulate what they were doing and why they were doing it. 0 2 4 reflect on their strengths and weaknesses. 0 2 4 demonstrate performance of higher order thinking (e.g., comparing and contrasting, synthesizing, detecting patterns). 0 4 3 the findings are presented in three sections below: (1) design of inquiry-focused learning, (2) roles of the teacher and librarian in collaborative development of instruction, and (3) impact on student performance. design of inquiry-focused learning in interview and questionnaire responses, participants indicated changes in a range of areas, but particularly stressed inquiry learning, essential questions, and datadriven assessment. in the interviews, all five teams discussed how they have achieved a better understanding of inquiry learning. as a middle school teacher new to her subject noted, ‚inquiry < really opened my eyes. it helped me work with the standards and break them down. i never worked with essential questions before.‛ for mature partnerships, participation in the course led to deeper conversations about the nature of inquiry learning: ‚our big aha had to do with the inquiry unit plan. we addressed the essential questions as part of the background building, and the students’ own questions as the inquiry.‛ four of the five teams discussed the importance of essential questions. one middle school teacher clearly described the influence of the pd course on her understanding: ‚when we did our unit plans, we got students to do to their activities based on essential questions. our curriculum coordinator said, ‘this is what i’ve been telling you.’ it didn’t make sense until i took this class.‛ evidence based library and information practice 2009, 4:2 109 four of the five teams discussed the value of data-driven assessment. one new high school librarian using preand postassessment measures for the first time noted, ‚it gives us targets that we need to address, because we’re measuring how much learning it’s going to take. it helped me be more conscious of what was going on, of what we were doing and why we were doing it.‛ responses to questionnaire 1 (table 1) showed that self-perceived gains at the end of the course compared to those after the summer institute were highest in four areas: (1) identify a generative theme (from 52% to 55%); (2) develop essential questions (from 44% to 50%); (3) transform standards (from 36% to 45%); and (4) incorporate strategies for problem investigation (from 32% to 35%). for questionnaire 2, nearly all respondents indicated that they valued essential questions and data-driven assessment more highly at the end of the course: (1) ‚exhibit curiosity, ask meaningful questions‛ (6 of 7 respondents), and (2) ‚collect and evaluate information‛ (6 of 7 respondents) (see table 2). in addition, the individual logs for december/january indicated these changes in teaching practice due to an inquiry focus: (1) essential questions as the focus for student learning; (2) student-centered learning and independent inquiry; (3) datadriven decision making; (4) student selfassessment; (5) challenging students to achieve higher level thinking and problem solving; (6) more thoughtful planning; (7) designing units that are connected to the real world and relevant to students’ lives; and (8) more carefully incorporating standards. roles of the teacher and librarian in collaborative development of instruction one of the features of the pd course that had the greatest impact on teaching practices was the partnership between teachers and librarians. the value of partnering was one of the topics most often included in additional comments to questionnaire 1, and five of seven respondents to questionnaire 2 rated it ‚most helpful‛ for their learning. individual logs and interview data provided indicators of the roles of the teacher and librarian in the collaborative development of instruction. participants characterized the relationship as a partnership of equals, with teachers providing subject expertise and intimate knowledge of their students and librarians providing information literacy expertise, knowledge of resources, technology expertise, and guidance to students through the conceptual and emotional challenges of the research process. participants appreciated using each other as sounding boards in deepening conversations about unit and lesson planning, standards, essential questions, assessment tools, and information literacy instruction. a key change in roles was the degree to which librarians were integral to the entire process of planning, implementation, and assessment, with joint responsibility and accountability. a valuable theme was the way the partnerships extended to other faculty at the school. impact on student performance the developers examined the contents of the participants’ culminating portfolios to study the impact of the professional development on teaching practices. in particular, they analyzed the units of study and student work samples for indicators of influence on student learning and performance. here we evidence based library and information practice 2009, 4:2 110 provide several examples from different grade levels as suggestive of the impact on student performance that many participants observed. elementary school an elementary school librarian and a second grade teacher collaborated on and co-taught a yearlong unit on human migration, focusing on three essential questions: why do people move? how do people feel when they are moving? how do people acclimate? the lessons involved guidance on note taking, self-assessment of writing, selfassessment of information literacy, conducting interviews, data interpretation and evaluation, and cross-age tutoring by fifth graders. the individual logs and team reports documented the step-by-step process of planning, revising, implementing, and again revising based on evidence of student learning and student selfassessments. the team was particularly successful in using assessment data to scaffold and adapt learning tools. for example, they originally devised an ‚idea tree‛ for students to map the major concepts in their research. however, they discovered that use of the tree required that students have mastery of other critical sub-skills (e.g., taking notes of key ideas, webbing to connect these ideas with one another, transforming key words and phrases into meaningful sentences). this led the team to scaffold the learning experience by focusing on each of these sub-skills. in the final interview, both members of the partnership expressed a high degree of satisfaction with student learning and performance due to the course. according to the teacher partner: working together with my librarian partner, i’ve spiraled my kids far above the 2nd grade level standards. the students had to conduct an interview for the first time, applying their note taking skills. they had to organize their notes into a paragraph. they analyzed their results – they had to count, graph, and interpret their data. each student will complete a brochure, and the brochure will go to the new students and families coming into our community, so it has a practical focus. they do selfevaluation using checklists and revise their work using the checklists. a gifted and talented student that was out of focus is thriving on this experience, also the special education kids, just because we’re touching so many multiple intelligences through interviewing, writing, oral presentation. this unit has so much content in so many different areas. students don’t get that in our normal curriculum, which is scripted reading, scripted math, scripted social studies, scripted science. this was a regular second grade class at an average school. but this project is so different from our regular curriculum and offers so many skills and opportunities to apply those skills that the kids just thrived on it. middle school a special education (sped) teacher and her school librarian focused on a semester-long unit of study on polynesian migration. prior to this collaborative venture, the sped teacher indicated that she had used a ‚lot of paper and pencil worksheets‛ and ‚drill exercises‛ with her students. during the institute, the team began to explore alternative experiences for sped students and decided on having the students create first person narratives on a simulated voyage from tahiti to hawaii. to gain the necessary background knowledge, the evidence based library and information practice 2009, 4:2 111 students visited cultural centers and museums and combed through print and online resources with assistance from the teacher, librarian, and aides in the classroom. throughout the process, students worked on a group kwhl chart that served as a critical assessment tool documenting what they knew, what they wanted to learn, how they might find information, and what they had learned. beyond the disciplinary content, an important objective for this unit was to help sped students overcome a ‚learned helplessness.‛ as the teacher noted in a log: ‚these kids know they are sped kids and because we [teachers] expect so little from them, they also come to expect little of themselves. it is so heartbreaking and frustrating to see this. i wanted to do something that might raise the bar of learning.‛ the teacher and librarian worked with the students individually and collectively in helping them prepare their simple first person narratives. the students also created facsimiles of artifacts the polynesian travelers might have taken along on the voyages (e.g., baskets, rope, fishing equipment). finally, the students videotaped their presentations and shared their final products with their parents. in a culminating reflection, the teacher and librarian stated: ‚this has been a difficult but most exhilarating experience. our students have stepped outside their comfort zone to experience what inquiry learning is about.‛ the following snapshot from the same team report captures one student’s achievement: david [pseudonym] is very disruptive < does no homework or classwork < has many office referrals and suspensions. in this unit, he has been very involved < he created a star chamber to show how the navigators used the stars to travel. he struggled with his story and his penmanship was almost unreadable. so he used alpha smart and was motivated to keep going. his final story was exciting and funny < and he was so proud. high school a chemistry teacher collaborated with her school librarian to ‚make over‛ a unit of study on the chemical properties of matter. in previous years, she had required students to produce papers on different lab experiments and found that her students were largely unmotivated and copying from their friends’ work. the teacher and librarian decided to revamp the unit by incorporating a real-world application in it. they challenged the students to (1) explore the chemistry behind everyday objects, and (2) explain how these objects improved the quality of their lives. they designed the instruction to tap into students’ prior knowledge, broaden their exposure to the topic, assess progress at important checkpoints, and provide for continuous reflection and feedback. to guide students through the inquiry process, the team created a checklist that required students to regulate and assess their own progress and to reflect in writing about what they were learning and how they were learning it. students ultimately shared their findings in slide presentations before audiences of their peers and invited community guests. a critical component in the overall learning experience was the students’ ability to describe their growth as researchers. the following examples from different students’ logs submitted throughout the project reflected their grasp of the essential features of inquiry: effective search queries: ‚i have grown into a better researcher<my evidence based library and information practice 2009, 4:2 112 searches are now more reliable and effective.‛ iterative nature of the inquiry process: ‚you need to take a break and look through your work again and again and look for anything than can be improved<you edit multiple times.‛ evaluation of resources: ‚i felt that i was an average researcher before this project, but now i am much more critical on what information would be best to use<what’s more reliable.‛ collaborative knowledge construction: ‚you have to encourage everyone to input their ideas and try to integrate them so they will be more whole-heartedly willing to perform at their best.‛ implementation of course design principles as noted earlier, five course design principles guided the development of scaffolding strategies, support structures, and tools designed to support participants’ achievement of the pd learning outcomes (see figure 1). this section discusses the implementation of these principles under the occasionally limiting conditions of the course and the school environment, derived from participant comments in the final interviews and responses to the final questionnaire (see table 3). table 3: course assessment rating scale: 1 = least helpful; 2 = somewhat helpful; 3 = helpful; 4 = most helpful n = 7. note: not all questions answered by every respondent course feature 1 2 3 4 point value rank summer institute (face-to-face, 3 days) 0 0 1 6 27 1 partnering with a teacher/librarian 0 0 2 5 26 2 curriculum development and implementation at your school 0 1 3 3 23 3 learning portfolios 0 0 5 2 23 3 online mentoring 0 2 3 2 21 5 monthly team reports (posted online in hnlc.org) 0 2 3 2 21 5 monthly individual logs (email) 0 2 4 1 20 7 e conference presentations 0 2 2 2 18 8 midpoint reunion assessment (face-to-face & videoconferencing) 0 0 2 2 14 9 buddy team responses (posted online in hnlc.org) 0 5 1 0 13 10 evidence based library and information practice 2009, 4:2 113 reflective practice the first design principle was to embrace reflective practice for continuous improvement through awareness, deep thinking, and action that aligns with new understandings. aspects of the course design most relevant to implementing this principle were (in rank order by questionnaire responses) partnering, learning portfolios, online mentoring, monthly team reports, individual reports, and buddy team responses. for some participants, reflective practice was integrated with situated learning: ‚i got a lot out of this course because of the implementing part reflecting on everything, improving on it, and then doing another unit.‛ although partnering was highly valued, librarians recognized the limitations set by the school environment: at this point in time, there’s so much pressure for us to meet ayp [annual yearly progress]. teachers have so much on their plates. it’s hard to ask them to give more of their own time to collaborate, because they’re exhausted. [we have] a large campus with 1,800 students. the average class size in science is over 35. we have two librarians, and we’re fortunate not to be losing librarians. i can’t imagine teaming. it would definitely cut down on the kinds of instruction we do. our calendar is filled for the whole year. situated learning the second design principle was to incorporate authentic records and tools for teaching and learning that center on tasks, questions, and problems situated in practice. the aspect of the course design most relevant to implementing this principle was curriculum development and implementation at schools, which was ranked third in importance for learning by questionnaire respondents. one interviewee compared this course to her previous pd experiences: i had pd through the science teachers association or through uh *university of hawaii+, but it wasn’t where we had to take it back to school and implement something. my approach is always that i’m going to pick at least one thing to use in the classroom. for this course, we needed to implement the whole thing, not just one thing. we could decide what we were going to do, but we had a framework to work within. community of practice the third design principle was to support a community of practice that provides common ground for work, collaboration, exploration, co-reflection, and mutual support. aspects of the course design most relevant to implementing this principle were (in rank order by questionnaire responses) the summer institute, partnering, online mentoring, monthly team reports, conference presentations, midpoint reunion, and buddy team responses. one interviewee was positive about the benefits of the course community: it was good because there was such a positive tone. it felt good to have that kind of support. when you’re the only librarian on campus, you wonder, ‚am i doing it right?‛ to hear from other people that they’re going through the same thing, i realized i’m on track. when you send each other positive messages, it makes you want to share and evidence based library and information practice 2009, 4:2 114 want to contribute to a learning community. many participants felt that their strongest community bonds were with their partners: while i do feel some connection to the other participants in the course, it is my team partners who i feel represent the ‚community of practice.‛ these individuals made the learning experience rigorous, meaningful and knowledge filled. one barrier to forming stronger community ties among buddy teams was the diverse nature of their backgrounds: during the online portion, it would have been more effective if we were paired with a team at a similar level/ability. although we offered many solutions/feedback to our buddy team it was not reciprocated when asked for our own projects. our buddy team was science oriented, and i’m not science oriented. connect to practice in context the fourth design principle was to connect to patterns of practice in context, building from knowledge, skills, and beliefs from past experiences to develop effective practices in one’s classroom, school, and with other practitioners. all aspects of the course design were relevant to implementing this principle. a number of participants indicated that they would be extending their learning more broadly into the school, such as this librarian: because our library is currently involved with school-wide curriculum efforts, the course allowed us to create and formulate units that can be shared. the material created will be added to our existing materials and enhance the resource packet for teachers to use with their classes. the course allowed us to develop/test/improve instructional materials under the guidance and instruction of knowledgeable mentors. evidence-based practice the last design principle was to promote formative and summative assessment in a continuous cycle of evidence-based practice. aspects of the course design most relevant to implementing this principle were (in rank order by questionnaire responses) curriculum development and implementation at the school, learning portfolios, and online mentoring. while many participants found formative assessments that integrated student selfassessments to be challenging, some expressed comments such as this: ‚i feel lucky that i took this course, because it’s taught me be a better teacher. it's all evidence-based. i can understand better the learning process, targets, and assessment.‛ use of technology the use of technology enabled learning, communication, and the formation of community but also posed barriers at the same time. most participants valued the convenience of online communication and the open sharing in the workspaces: it’s really a good mode of communication. you can get to it anytime. it’s convenient. you can send long emails and not worry that you’re bothering someone. the advantages were that workspace gave everyone a common place to communicate. we evidence based library and information practice 2009, 4:2 115 could share files, engage in discussions, and post examples of works in progress. i found it to be very helpful when looking for immediate feedback. i would not only receive immediate feedback from my partner, but anyone else who wanted to share ideas with me. through the workspace, we could also share online resources with one another. however, the technology also posed barriers. on a technical level, school computers and networks sometimes failed for days at a time. participants also had varying levels of fluency and comfort with using technology for communication: it’s hard. you need to try to make sure you’re specific. i think, now how is it going to be received? it takes a longer time to compose a message. i have to learn to improve. it helped to meet them face-to-face. i would have felt uncomfortable not having met them before. i wouldn’t know how i need to communicate with them, how to choose my words. some felt that lack of time reduced the quality of online communication: ‚because it happened so frequently, i don’t think the quality of the online communication was so high. we were just trying to do it because it had to be done. i think if it was spaced out a little bit more, it would be better. it was crunch time every month.‛ others preferred face-to-face communication to online communication: it was difficult to write concisely about the project, the process, challenges and triumphs. it was easier for me to articulate about the project in person. face-to-face i can focus. my attention is all on that person, or group of people. if it’s online, the phone rings, kids are asking you questions, there’s something else to type, so i think i’ll get to it later. sometimes i close the window by mistake, or the computer freezes. there are all these distractions that make it not so efficient when it’s online.) on the value of building community online, responses were mixed: the workspace and email has made it easier to develop a sense of community. frequency of the encounters helps to build that feeling. without it, our relationships within our busy schedule would be out of sight out of mind. even though we didn’t get to hear from everyone all the time, we did know that we were all trying to accomplish the same thing and that is where the strength of technology helped to build community. i think the face-to-face would help to build a better community. you’re only getting information if you’re reading someone else’s piece, but if it’s face-to-face, you see everyone. longitudinal reports of sustained and expanded practice following the yearlong professional development experience, the authors conducted informal email follow-up with the participants in the 2006-2007 and 20072008 academic years. in all but one case, respondents indicated that they were continuing to incorporate inquiry-based approaches, e.g., using essential questions and assessment. in the single exception, the librarian had transferred to another school evidence based library and information practice 2009, 4:2 116 and both members of the original team had difficulties finding new partners. what also emerged was evidence that many participants were involving other colleagues in inquiry-focused practices. at least six school teams had connected their work to school priorities and reform efforts, such as reading and writing programs and standards-based curriculum initiatives. seven teams had extended their efforts to involve other teachers in their grade level or department. six had also reached out to faculty in other grade levels or departments, or collaborated with other support personnel such as the curriculum coordinator and technology resource teacher. from the start, librarians had taken the initiative to form the teams, maintain momentum, and contribute to a change in views about the role of the librarian as a curriculum partner. after the pd course, six of the librarians assumed major curricular roles in their respective schools. for example, two high school librarians were designated as leaders of professional learning communities that focused on improving teaching practices. in another instance, the librarian emerged as a key member of the school’s team to train all teachers in assessing students’ writing performance. the following e-mail comments from one respondent indicate how the training influenced her current leadership in curriculum and teaching: my deeper understanding of inquiry-based projects gained at our inquiry partnerships workshop has allowed me to transfer and apply my knowledge in our standardsbased efforts at my school. this year i co-facilitated a focus group with our curriculum coordinator. our group’s task (which was aligned to our school’s academic plan) was to identify five components of a standards-based classroom. the training i received inspired my selfconfidence and provided me with the necessary skills to step forward and become a leader in my school. conclusion a practice-based approach to professional development required continual modification of the course design and timely, individualized mentoring and feedback, based on analysis and coreflection by the developers on the evidence gathered through participant logs, reports, and school site visits. modeling the inquiry process in their own course development work and making this process transparent to the participating community were essential to improvement. course participants reported beneficial results in both immediate and long-term changes in practice. the summative evaluation identified significant changes in practice in three areas: (1) the design of inquiry-focused learning, (2) the roles of the teacher and librarian in collaborative development of instruction, and (3) the impact on student performance. two years after the yearlong professional development experience, most participants indicated that they were continuing to incorporate inquiry-based approaches, and over half of the participants were involving other colleagues at their schools in inquiry-focused practices. six of the librarians assumed major curricular roles in their respective schools. the practice-based model of professional development described in this study appears to be effective and sustainable. it has been tested and modified by other development teams in the last two years. in 2006-2007 and 2007-2008, the model was used in professional development focused on assessing for student learning. the training was conducted by two school evidence based library and information practice 2009, 4:2 117 librarians with consultative support from the original developers. using the model, the second author is currently working with a new development team on a proposed training initiative to assist high school teachers and librarians in facilitating senior capstone projects. other members of the new team include a university librarian, an administrator for the university’s college of education, an instructional technology specialist, and two high school librarians. more extensive use of the model in other contexts with further testing and refinement by other developers is needed to ensure that the model is robust and widely applicable. works cited ball, deborah l., and david k. cohen. ‚developing practice, developing practitioners: toward a practicebased theory of professional education.‛ teaching as the learning profession: handbook of policy and practice. eds. linda darling-hammond and gary sykes. san francisco: jossey-bass, 1999. 332. donnelly, roisin. ‚investigating the effectiveness of teaching ‘on-line learning’ in a problem-based learning on-line environment.‛ challenging research in problembased learning. eds. maggi savinbaden and kay wilkie. berkshire, england: society for research into higher education and open university press, 2004. 50-115. elmore, richard f. school reform from the inside out: policy, practice, and performance. cambridge: harvard education press, 2004. fenwick, tara, and mark tennant. ‚understanding adult learners.‛ dimensions of adult learning: adult learning: adult education and training in a global era. ed. griff foley. berkshire, england: open university press, 2004. 55-73. ghaye, anthony, and kay ghaye. teaching and learning through critical reflective practice. london: david fulton publishers, 1998. given, lisa. "qualitative research in evidence-based practice: a valuable partnership." library hi tech 24.3 (2006): 376-86. guskey, thomas r. ‚professional development and teacher change.‛ teachers and teaching: theory and practice 8.3/4 (2002): 381-91. harada, violet h., and joan m. yoshina. assessing learning: librarians and teachers as partners. westport: libraries unlimited, 2005. ---. inquiry learning through librarianteacher partnerships. worthington, oh: linworth, 2004. marzano, robert j. what works in schools: translating research into action. alexandria, va: association for supervision and curriculum development, 2003. nesbit, tom, linda leach, and griff foley. ‚teaching adults.‛ dimensions of adult learning: adult education and training in a global era. ed. griff foley. berkshire, england: open university press, 2004. 74-95. penuel, william r., barry j. fishman, ryoko yamaguchi, and lawrence p. gallagher. ‚what makes professional development effective? strategies that foster evidence based library and information practice 2009, 4:2 118 curriculum implementation.‛ american educational research journal 44.4 (2007): 921-58. perkins, david n. smart schools: from training memories to educating minds. new york: free press, 1992. sato, mistilina, ruth chung wei, and linda darling-hammond. ‚improving teachers’ assessment practices through professional development: the case of national board certification.‛ american educational research journal 45.3 (2008): 669-700. smylie, mark a. ‚teacher learning in the workplace: implications for school reform.‛ professional development in education: new paradigms and practices. eds. thomas r. guskey and michael huberman. new york: teachers college press, 1995. 92-113. todd, ross j. ‚evidence-based practice and school libraries: from advocacy to action.‛ the school library media specialist and education reform. eds. sandra hugheshassell and violet h. harada. westport: libraries unlimited, 2007. 57-78. wenger, etienne. communities of practice: learning, meaning, and identity. new york: cambridge university press, 1998. york-barr, jennifer, william a. sommers, gail s. ghere, and jo montie. reflective practice to improve schools: an action guide for educators. 2nd ed. thousand oaks: corwin, 2006. yukawa, joyce, violet h. harada, and daniel suthers. ‚professional development in communities of practice.‛ the school library media specialist and education reform. eds. sandra hugheshassell and violet h. harada. westport: libraries unlimited/teacher ideas press, 2007. 179-92. evidence based library and information practice 2009, 4:2 119 appendix a end-of-course team interview protocol these questions were used to stimulate discussion. participants were informed that their responses would remain anonymous. 1. how would you describe your partnership? how important has developing the collaborative relationship or partnership been in achieving your learning goals? how important has developing the collaborative relationship or partnership been in improving learning for your students? 2. has this pd course helped you improve student learning and student achievement? if so, in what ways? 3. how did the face-to-face overviews of strategies and tools (summer institute, january assessment session) affect your ability to incorporate the course learning outcomes as you did curriculum planning and teaching over the year? 4. what role did your team reports and individual logs play in your learning? 5. what achievements/difficulties did you encounter in the process of planning and implementation? 6. did you have a previous relationship with any of the mentors? how did the support from the three mentors influence your planning and implementation? did the support from the mentors affect your ability to communicate and form relationships, especially online? 7. did you have a previous relationship with any of the other participants? if so, how did this affect your ability to communicate and form relationships, especially online? how did the buddy team support influence your planning and implementation? 8. have you had previous professional development experiences? if yes, how did this pd course compare in terms of your professional growth? 9. did this pd experience give you a different perspective on the role of the library in your school? 10. how does the school environment support/not support your collaborative work? 11. what was easy and difficult about using the technology – in general, and for the course work and communication? what are the advantages and disadvantages of using hnlc.org and email for this pd? 12. what are your plans to build on your achievements from this course? microsoft word es 1466_ shulte evidence based library and information practice 2008, 3:2 35 evidence based library and information practice evidence summary high self-efficacy and high use of electronic information may predict improved academic performance a review of: tella, adeyinka, adedeji tella, c. o. ayeni, and r. o. omoba. “self-efficacy and use of electronic information as predictors of academic performance.” electronic journal of academic and special librarianship 8.2 (2007). 24 apr. 2008 <http://southernlibrarianship.icaap.org/content/v08n02/tella_a01.html> reviewed by: stephanie j. schulte assistant librarian, walter e. helmke library, indiana university-purdue university fort wayne fort wayne, indiana, united states e-mail: schultes@ipfw.edu received: 26 february 2008 accepted: 24 april 2008 © 2008 schulte. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – to determine if self-efficacy and use of electronic information jointly predicted academic performance and to determine what information sources students used most often. design – descriptive surveys (scales) for each of the three variables. setting – university of ibadan, nigeria, a metropolitan, government-supported university with approximately 18,000 students. subjects – seven hundred undergraduate and graduate students randomly chosen from 7 departments of the faculty (i.e., college) of education (100 students from each department). methods – students completed the morganjinks self-efficacy scale and the use of electronic information scale. academic performance was measured using a general aptitude test that covered general education, english language, and mathematics. the morgan-jinks scale consisted of 30 items, and the academic performance test consisted of 40 items. no instrument length was provided for the use of electronic information scale, and no details on the 36 actual content of the general aptitude test or the use of electronic information scale were provided. these surveys were completed at the university under conditions similar to that of a typical exam (i.e., no talking). all 700 subjects completed the surveys, and there was no evidence of participants providing informed consent or that they were given an opportunity to withdraw from the study. data was analyzed using multiple regression analysis, a suitable analysis for this type of data. main results – self-efficacy and use of electronic information together contributed to 9% (reported as 0.9% in the article) of the variance in academic performance, and each variable statistically significantly contributed to predicting academic performance (p<0.05). use of electronic information contributed more than did selfefficacy to the prediction of academic performance. the correlations of use of electronic information to high self-efficacy and academic performance to high selfefficacy were very slightly stronger than these variables to low self-efficacy. use of electronic info and self-efficacy were both statistically significantly correlated to academic performance (r = 0.2779 and r = 0.1559, respectively), though these correlations were modest. when asked what information source the students used most often, a little more than a third (35.42%) noted the internet, followed by cd-rom databases (20.43%), electronic journals (18.71%), and e-mail (18.29%). electronic books and bulletin boards were used least often (3.71% and 3.43%, respectively). conclusion – the original authors conclude that self-efficacy and use of electronic information “predict and influence academic performance” (discussion ¶ 6). since use of electronic information is related to greater academic achievement, academic institutions in africa should strive to provide internet access in all schools. information literacy instruction should become a required course for all students to promote appropriate use of electronic information. commentary self-efficacy refers to one’s perceptions of being successful at a specific task regardless of actual ability. it makes sense to think that an efficacious student might use electronic information more because they believe they can. in part, this study attempted to provide evidence to support the idea that greater use of electronic information leads to better academic outcomes by examining selfefficacy and use of electronic information as predictors of academic success. unfortunately, the study has several limitations not addressed by the authors, including the use of a self-efficacy scale originally geared to younger students that is not task specific; not providing or referencing the other two scales used in the study, or discussing validation of them; and not detailing the random selection of students. additionally, the conditions under which the students completed the instruments could have influenced measurements. the study reports several statistically significant results in relation to the research questions, yet the analyses seem misinterpreted. for example, self-efficacy and use of electronic information jointly contributed 9% of the variance of academic performance. a large amount of variance and thus other contributing factors (91%) remain unaccounted. both the r2 and the adjusted r2 (0.05531) indicate that these data do not represent a good statistical model and thus results should be questioned. further analyses from the multiple regression are also statistically significant, but they do not appear to be meaningful. correlation coefficients are in the range of 0.1559 to 0.385, values that do not indicate 37 strong correlations overall regardless of statistical significance. this study took place in nigeria. admission to institutions of higher education in nigeria is highly competitive and fraught with controversy (zurlo 981-86; okoroma 1-6). consequently, those students who are admitted could have higher self-efficacy and be higher performers in general, possibly skewing measurements of these variables. additionally, access to electronic information resources “is still considered a luxury” (discussion ¶ 5) in africa, which could potentially impact results on the use of electronic information scale. therefore, results from this study, if reliable, would only be generalizable to other institutions facing similar circumstances. data presented in this study do not robustly support the claims of the original authors. the authors have extrapolated modest correlations to behavior, and this may not be the case. with weak statistical data and a great deal of variance remaining unaccounted, the conclusions of the original article must be seriously questioned. because of this, this study is not recommended for making decisions related to library practice and serves as an example of the need to closely examine all variables, instruments, and analysis before incorporating study results into practice. works cited okoroma, n. s. “admission policies and the quality of university education in nigeria.” educational research quarterly 31.3 (2008): 3-24. zurlo, john a. “nigeria.” world education encyclopedia. ed. rebecca marlowferguson. 3 vols. detroit: gale group, 2002. evidence summary   faculty knowledge of information literacy standards has an impact in the classroom   a review of: saunders, l. (2012). faculty perspectives on information literacy as a student learning outcome. the journal of academic librarianship, 38(4), 226-236. doi: 10.1016/j.acalib.2012.06.001   reviewed by: giovanna badia liaison librarian, schulich library of science and engineering mcgill university montreal, quebec, canada email: giovanna.badia@mcgill.ca   received: 3 dec. 2012     accepted: 22 jan. 2013    2013 badia. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to discover how faculty perceives information literacy and examine whether professors in different disciplines view and approach information literacy differently.  particularly, the study seeks to address the following questions: “how do faculty members define or understand information literacy? are they familiar with existing standards such as [those from the association of college and research libraries] acrl? does the development of a local definition of information literacy impact faculty understanding? how important do instructors believe information literacy to be for their students? how do they address information literacy, or expect it to be addressed within the curriculum? are there disciplinary differences in faculty attitudes toward and approaches to information literacy?” (p. 227)   design – survey, i.e., an online questionnaire  followed by interviews. setting – colleges and universities in the united states. subjects – 834 faculty members in anthropology, the natural sciences, computer science, english literature, psychology, and political science from a sample of 50 american colleges and universities with undergraduate degree programs. methods – an email, containing a link to a brief online survey, was sent to 834 professors from academic institutions across the united states. three faculty members from each department in six different disciplines from each institution were contacted. the survey contained a mix of closed and open-ended questions and could be completed in less than 10 minutes. respondents were asked to supply their contact information if they agreed to be phoned for a follow-up interview. the interview consisted of six questions that were posed to all participants, with some changes depending on the answers given.   main results – regardless of discipline, the majority of faculty members who responded to the survey thought that information literacy competencies were important for their students to master. the majority also rated their students as only “somewhat strong” in “identifying scholarly materials, identifying reliable/authoritative information, finding relevant information, citing sources properly, synthesizing information, and searching databases” (p. 229). professors’ answers differed within different disciplines when it came to showing their own knowledge of information literacy standards, such as those of acrl, and assessing the abilities of their students. for example, biology students’ web searching skills were rated higher than students in english literature and anthropology. when faculty were asked their opinions about who should be responsible for information literacy instruction, there was no straight answer. many professors agreed that it is the responsibility of both faculty and librarians. those faculty members who were knowledgeable about information literacy standards were also among the ones who included information literacy instruction in their courses and thought it was important for their students to learn.   conclusion – according to the author, the study results show that possibilities continue to exist for librarians to be part of information literacy endeavours, but it is still up to the librarians to start and maintain conversations with faculty on this topic. because faculty members have not yet found systematic methods for integrating information literacy into the curriculum, they might be open to librarians’ suggestions and ideas on this topic. “perhaps the most important finding of this study is that knowledge of and familiarity with information literacy standards is more closely associated with whether faculty address information literacy in their courses than any other variable including disciplinary area” (p. 232). therefore, it is the librarian’s responsibility to engage in discussions with faculty about information literacy.     commentary   information literacy is a popular topic in the library science literature, but few studies have looked at what faculty members think about information literacy. the few studies that exist examined the attitudes of faculty within a single disciplinary area, institution, or geographic region (boon, johnston, & webber, 2007; dacosta, 2010; gullikson, 2006; hardesty, 1995; leckie & fullerton, 1999; mcguinness, 2006; nazari & webber, 2011; singh, 2005; wu & kendall, 2006). this study goes further by investigating the information literacy perceptions of professors in different disciplines from numerous academic institutions across the united states. the author states that “the results of this study provide academic librarians a broader insight into faculty understanding of information literacy and will help to advance the discourse of information literacy further into the disciplines” (p. 227).   this reviewer used glynn’s eblip critical appraisal checklist (2006) to help evaluate the study’s methodology. the author randomly selected a large sample of 50 colleges and universities to target. however, the author does not describe how these institutions were randomly selected, which makes it difficult to say whether the choice of population is unbiased and whether the results can be applied to a larger population. the 33.3% average response rate also prevents the results from being generalized. nonetheless, the study’s results provide some thought-provoking observations of faculty attitudes towards information literacy that librarians can use to engage in discussions with professors about incorporating information literacy into their courses and program curriculums.   this study encourages librarians to approach or keep reaching out to faculty about information literacy, since it “suggests that faculty might be receptive to approaches by librarians” and “that faculty have a lot of respect for librarians and their expertise” (p. 232). the professors surveyed appeared to welcome being contacted by librarians about information literacy and understood the challenges that librarians face in attracting students to training sessions. those faculty members who knew about information literacy standards tended to include information literacy instruction and assess these skills in their courses. after reading saunders’s article, this reviewer has been motivated to speak about information literacy standards when approaching faculty.     references   boon, s., johnston, b., & webber, s. (2007). a phenomenographic study of english faculty's conceptions of information literacy. journal of documentation, 63(2), 204-228. doi: 10.1108/00220410710737187 dacosta, j. w. (2010). is there an information literacy skills gap to be bridged? an examination of faculty perceptions and activities relating to information literacy in the united states and england. college and research libraries, 71(3), 203-222.   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399.   gullikson, s. (2006). faculty perceptions of acrl's information literacy competency standards for higher education. journal of academic librarianship, 32(6), 583-592. doi: 10.1016/j.acalib.2006.06.001 hardesty, l. (1995). faculty culture and bibliographic instruction: an exploratory analysis. library trends, 44(2), 339-367. leckie, g. j., & fullerton, a. (1999). information literacy in science and engineering undergraduate education: faculty attitudes and pedagogical practices. college and research libraries, 60(1), 9-29. mcguinness, c. (2006). what faculty think – exploring the barriers to information literacy development in undergraduate education. journal of academic librarianship, 32(6), 573-582. doi: 10.1016/j.acalib.2006.06.002 nazari, m., & webber, s. (2011). what do the conceptions of geo/spatial information tell us about information literacy? journal of documentation, 67(2), 334-354. doi: 10.1108/00220411111109502 singh, a. b. (2005). a report on faculty perceptions of students' information literacy competencies in journalism and mass communication programs: the acejmc survey. college and research libraries, 66(4), 294-310. wu, y. d., & kendall, s. l. (2006). teaching faculty's perspectives on business information literacy. reference services review, 34(1), 86-96. doi: 10.1108/00907320610648789   eblip seeks writing assistants news   eblip seeks writing assistants    2022. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30170     evidence based library and information practice (eblip) seeks volunteers to be writing assistants.  writing assistants provide guidance to non-native english speakers who submit promising papers to eblip but who need help in terms of organization, grammar, and general clarity of language to strengthen their submissions. this is a service that is provided free of charge by eblip as part of its commitment to support a greater global exchange of knowledge.   eblip editors refer authors to a writing assistant on a case-by-case basis. writing assistance volunteers act as advisors only. it is up to the author(s) of the manuscript to make all of the appropriate revisions and respond to the reviewers’ comments. working with an eblip writing assistance volunteer does not mean that the final revised manuscript will be accepted for publication; it may still need to be submitted for a further round of editorial/peer review. writing assistants may be acknowledged in published submissions.   ideal candidates will possess excellent written communication skills, attention to detail, and some experience in developmental copyediting and/or academic writing.   interested persons should send a statement of interest, indicating areas of strength they would bring to the role, as well as a brief resume to ann medaille, editor-in-chief, at amedaille@unr.edu by july 15, 2022. those invited to join the team will be appointed for a term (renewable) of two years. please email ann with any questions.   **please note that evidence based library and information practice is a non-profit, open access journal, and all positions are voluntary and unpaid.   about the journal:   published quarterly and hosted by the university of alberta, this peer-reviewed, open access journal is targeted at all library and information professionals interested in an evidence based model of practice. by facilitating access to librarianship research via original research articles and evidence summaries of relevant research from the library literature, evidence based library and information practice enables librarians to practice their profession in an evidence based manner. please visit the evidence based library and information practice web site (https://journals.library.ualberta.ca/eblip/index.php/eblip) for further information about the journal.     microsoft word es_mckenna.doc evidence based library and information practice 2007, 2:1  121 evidence based library and information practice       evidence summary    canadian library human resources short‐term supply and demand crisis is averted,  but a significant long‐term crisis must be addressed    a review of:  8rs research team. the future of human resources in canadian libraries february 2005.  edmonton, ab: university of alberta. 21 february 2007  <http://www.ls.ualberta.ca/8rs/8rsfutureofhrlibraries.pdf>.     reviewed by:   julie mckenna  services assessment librarian, university of regina  regina, saskatchewan, canada  e‐mail: julie.mckenna@uregina.ca      received: 1 december 2006    accepted: 29 january 2007      © 2007 mckenna. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to examine the human  resources environment in canadian libraries  in order to assess readiness to accommodate  change and to identify opportunities for  human resources planning. the “8rs” of the  study were defined as recruitment,  retirement, retention, remuneration,  repatriation, rejuvenation, re‐accreditation,  and restructuring.    design – this study was undertaken in  three phases over nearly three years through  the use a variety of methods including  literature review, analyses of existing data  (statistics canada and library school  graduate data), telephone interviews (with  senior library administrators), focus groups  (with representatives from canadian  association of research libraries, canadian  urban libraries council and alberta  association of library technicians), print  surveys (library institutions) and web‐based  surveys (of professional librarians and  paraprofessional library staff).    setting – canadian libraries that are not  component branches of a system, and that  employ professional librarians.    subjects – stage i:  17 senior library  administrators participated in telephone  interviews and three focus groups were  conducted.    stage ii:  surveyed library administrators  representing institutions. a multi‐stage  stratified random sampling technique was  used to ensure geographical representation  from each of canada’s provinces and  territories. full census participation was  http://www.ls.ualberta.ca/8rs/8rsfutureofhrlibraries.pdf http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:1  122 conducted for members of the canadian  association of research libraries and the  canadian urban libraries council. the  print survey instrument was distributed to  1,357 subjects; 461 completed surveys were  returned (response rate of 34% with results  for the total sample accurate within plus or  minus 3.8 per centage points, 95 times out of  100).    stage iii:  surveyed professional librarians  and paraprofessional staff. multi‐stage  random sampling was used to ensure  representation of library staff from all  library sectors and sufficient sub‐sample  sizes. of the 12,472 individuals in the  sampling frame, 8,626 were notified of their  selection to participate in the web‐based  survey. corrections were made to e‐mail  addresses and 7,569 e‐mail invitations with  the survey url were sent successfully. of  the 8,626 potential respondents, 3,148  librarians and paraprofessionals  participated (for a response rate of 37%). a  non‐random canada‐wide call for  participation was distributed to library staff  who had not been represented in the  sampling frame via the listservs of 56 library  associations. this provided an additional  1,545 respondents and the total sample size  increased to 4,693 for a confidence interval  of plus or minus 1.2%, 95 times out of 100.  the non‐random data from the canada‐ wide call was kept in a separate dataset file.    methods – stage i began with a literature  review and analysis of existing statistics  canada and library school graduate data.  three focus group sessions with  representatives from canadian association  of research libraries, canadian urban  libraries council and alberta association of  library technicians were held and in‐depth  telephone interviews were conducted in  may 2003 with 17 senior library  administrators. transcripts were  thematically coded and summarised. the  interview guide is available as appendix e  of the report. the results of stage i were  used to inform the design of the two survey  instruments (stages ii and iii).     stage ii was a 17‐page print survey  informed by insights gathered in stage i and  was sent to library directors in the summer  of 2003. the print mail‐out of the  institutional survey was sent to libraries that  employed at least one librarian and were not  a component branch of another system. a  copy of the institutional survey is available  as appendix c of the report.    stage iii was a web‐based survey of  librarians and paraprofessional library staff  undertaken in the summer of 2004 using  random and non‐random sampling methods.  this survey was developed from insights  gathered in stages i and ii. a copy of the  individual survey is available as appendix  d of the report.    main results – the analytical focus of the  275‐page report is on the broad canadian  library sector. data and analysis are  provided by type of library, type of staff,  and by geographical regions where  sufficient response rates have allowed  reporting. although the role of the  paraprofessional is examined in many  sections of the report, the principal focus is  on the professional librarian.    nine sections of the report present results,  analysis and strategic human resource  planning implications. highlights for the  broad canadian library environment are  briefly described below. each section of the  report provides further breakdown by  library sector, type of position, career stage  and other variables that provide significant  insight.    workplace demographic characteristics (section  c)  this section provides demographic  information about those in supervisory or  evidence based library and information practice 2007, 2:1  123 management roles (62% of librarians), union  density (67% of librarians and 79% of  paraprofessional staff), longevity in career,  part‐time employment, and gender, along  with other characteristics. results conclude  that visible minorities and aboriginal staff  are under‐represented across all types of  libraries and that few libraries recognise the  credentials of immigrant librarians.    recruitment to the profession and to the  organization (section d)  positive exposure to libraries and librarians  is the best predictor of librarianship as a  career choice and there were no significant  differences in the original motivations for  choosing the professional librarian career  between new professionals and senior  librarians. in response to the open‐ended  question about motivation for choosing  librarianship, none of the librarian  respondents (n=1,823) indicated leadership,  managing libraries or supervising others as  their reason (p. 52). many respondents  indicated reasons for choosing the  profession that are in alignment with the  values of librarianship, but few indicated  reasons that reflect the real nature of the  librarians’ role. the average age of new  librarian recruits is 37 (with little variation  between sectors).    thirty per cent of paraprofessionals are  interested in pursuing an mlis degree; 29  per cent are not interested because they are  satisfied with their current role. the major  barriers for paraprofessionals wishing to  pursue the mlis degree are inadequate or  unrecognised credentials (21% ‐ although  45% of paraprofessionals have an  undergraduate degree), geographic distance  (33%), lack of money (48%), and lack of time  (49%).    eighty per cent of libraries report that the  major barrier to recruiting is budget  constraints; other barriers include small size  of library (60%), organizational hiring freeze  (54%), inadequate librarian pay (54%),  geographic location (52%), inadequate pool  of qualified candidates (51%), and  inadequate pool of interested candidates  (50%).    the ten most important and difficult‐to‐fill  competencies when recruiting varied  significantly for each sector: leadership  potential, ability to respond flexibly to  change, and ability to handle high‐volume  workload were the three highest‐ranked  competencies across all libraries.    retirement (section e)  canadian libraries experienced librarian  retirements (11% of total current workforce)  and paraprofessional retirements (7% of  total current workforce) between 1997 and  2002. during this period, 79 per cent of  librarians retired before age 65. forty per  cent of librarians over age 50 estimate that  they will retire between 55 and 60. only 9  per cent of libraries have a succession plan.    staff retention:  inter‐ and intra‐organizational  mobility (section f)  librarians are satisfied with their work and  stay in their organization because they like  the job (85%), co‐workers (84%), and  workplace (79%). seventy‐seven per cent of  senior librarians and 87% of senior  paraprofessionals have been at their current  library for more than 10 years.     sixty‐nine per cent of librarians believe they  are qualified to move to higher level  positions, but 69% of institutions state that  limited librarian turnover contributes to a  lack of promotional opportunities.    education (section g)  seventy‐five per cent library administrators  agree that mlis programs equip graduates  with needed competencies, but 58%  recommended that the programs provide  more management, business and leadership  training.  seventy‐six per cent of  evidence based library and information practice 2007, 2:1  124 administrators believe that they have little  or no input into curriculum content of mlis  programs.    overall evaluation of mlis education by  recent librarian entrants is not positive. only  forty‐four per cent indicated that the  program provided a realistic depiction of  the job, while only 36% said the program  provided a realistic expectation of work in  their library sector. recent librarian entrants  (67%) were satisfied with the overall quality  of their mlis program, but few indicated  that their program provided them with the  necessary management skills (25%),  leadership skills (20%), or business skills  (12%) for their position. recent library  technician entrants were more satisfied (81%)  with their programs’ success in providing  general skills (87%), and providing a  realistic depiction of the job (72%).    continuing education (section h)  new librarians (72%) need a significant  amount of ongoing training, but only 56%  believe that their institution provides  sufficient training opportunities. only 30%  of libraries have a routine method for  determining training needs of librarians and  fewer (13 %) have an evaluation method for  training outcomes. in most cases, about half  of those who received training reported that  it improved their job performance.    quality of work and job satisfaction (section i)  librarians and paraprofessionals are  satisfied with their jobs (79% for each) and  librarians (72%) and paraprofessionals (61%)  agree that their salary is fair. most libraries  offer a wide range of benefits to their  employees, including life insurance (95%),  pension plan (92%), and medical benefits  (88%). librarians (80%) and  paraprofessionals (70%) are satisfied with  their benefits.    although a low percentage of librarians  agreed that they have little job stress (24%)  and only 39% found their workload to be  manageable, 62% of librarians agree that  their work allows work, family and personal  life balance. the statistics are slightly more  positive for paraprofessionals.    there is a gap between the desire to be  treated with respect (98% for all workers)  and the perception that respect is conveyed  (77% of librarians and 75% of  paraprofessionals). a similar gap exists  between desire to be involved in decision‐ making and actual involvement. the two  most important factors for job satisfaction  for all library workers are respectful  treatment and a job that allows them to  learn new skills and grow.    numerical librarian demand‐supply match  (section j)  libraries hired more librarians than they  lost in 2002, for a net three per cent increase.  many library administrators believe that  there will be a five‐year increased demand  for librarians (77%) and paraprofessionals  (81%).    the short‐term supply (next 5 years) of new  librarians to replace departures due to  retirements is predicted to have the capacity  to fill 98 per cent of the current librarian  positions; the capacity to replace library  technicians is 99 per cent. the long‐term  supply (next 10 years) of new librarians to  replace departures due to retirements is  predicted to have the capacity to fill 89 per  cent of the current librarian positions; the  capacity to fill technician positions is  identical. these predictions are based on no  growth in the number of positions in the  future.    match between organizational job function  demand and individual staff supply of skills,  abilities, talents, interests (section k)  libraries report that increased use of  information technologies (87%) and re‐ engineering (61%) have contributed the  evidence based library and information practice 2007, 2:1  125 greatest change in the roles of librarians.  libraries report that more librarians have  been required to perform a wider variety of  tasks in the past five years (93%) and that  this trend will continue over the coming five  years (94%).    a high percentage of libraries (86%)  reported that over the past five years  librarians have been expected to perform  more management functions and 56%  of  mid‐career and senior librarians believed  this had occurred. libraries (88%) believe  that this trend will continue; only 44% of  librarians indicate interest in performing  management functions.    mid‐career and senior librarians report that  job stress has increased over five years ago.  requirements to work harder (55%),  perform more difficult tasks (56%), perform  a wider variety of tasks (69%), and perform  more managerial functions (56%) are the  contributing factors. the performance of a  wider variety of tasks and more difficult  tasks was significantly related to the  assessment by librarians that their jobs were  more enjoyable, interesting, rewarding and  challenging.    institutions (78%) reported the increased  need for paraprofessionals to perform  librarian tasks over the past five years and  believe that this will continue (77%). only  28% of paraprofessionals believe they are  currently required to perform more librarian  tasks.    conclusion – the need to confirm the  existence and magnitude of the crisis that  will be created by upcoming retirements in  canadian libraries was a primary motivator  for this study. conclusive results were  obtained that should inform each sector and  geographic area in canada. the percentages  of staff over the age of 55 (librarians: 25%;  paraprofessionals: 21%) is much greater  than that of the canadian workforce (11%).  if there is no growth in the number of  positions needed, there will be no short‐ term supply‐demand crisis to fill the gaps  left by retirements. there will be a librarian  and technician shortage in ten years (a  shortfall of 11% of the current supply) and a  more significant crisis if the predicted  growth in staffing is factored in.  recruitment to the librarian and technician  professions is critical and the  paraprofessional staff may be a potential  pool of future mlis candidates if the  accessibility issues associated with the  programs are addressed. only nine per cent  of organizations have a succession plan in  place. there is great opportunity for the  development of strategic solutions.    in response to the open‐ended question  about motivation for choosing librarianship,  no respondent indicated leadership,  managing libraries or supervising others as  their reason. this is of concern when 62% of  librarians today work in a managerial role.  management and leadership skills are a  significant concern for recent graduates,  administrators, and librarians, with all  indicating that the workplace needs are  greater than the current preparedness. more  cooperation with mlis programs and  professional associations is essential to  ensure that leadership and management  skill development are supported through  the curricula and continuing education  planning. organizations must also develop  and support a culture where leadership is  encouraged and expected, and recognised.    there is a need for further development of  continuing education opportunities, and  training needs assessment and outcome  assessment programs may be beneficial.  paraprofessionals and new librarians are  less satisfied with the workplace training  opportunities available to them than  librarians in later stages of their careers.    evidence based library and information practice 2007, 2:1  126 role change will continue in libraries and  planning will be essential to ensure that  restructuring reflects the competencies that  will be needed in the new mix. workload  and job stress appear to be rising and will  require careful monitoring. there may be  opportunity to define roles for “other”  professionals in libraries.    library staff have a tendency to stay in their  institution for much of their career, making  decisions in the recruitment and hiring  processes of critical importance. loss of  employees due to turnover is not a problem  for most libraries, but the lack of turnover  has affected the promotional opportunities  for those who desire upward mobility.    an interesting recommendation was made  that two or more libraries may realise both  cost savings and benefits through the  sharing of staffing resources. if issues  surrounding credentials can be addressed,  there may be a potential pool of future  immigrant librarians.    commentary     the results of this study provide much‐ needed information about the general state  of human resources in canadian libraries.  the 275‐page document is filled with data,  analysis and insights that could never be  adequately represented in a short summary  such as this.     the report is a rich reference tool for  practitioners that can be used to examine  specific issues by type of library, type of  staff, occupational level, career stage, and by  geographic regions. it would greatly benefit  from the creation of an index to enable users  to search for common themes throughout  the document.    both the writing and the use of tables and  figures are presented in a clear and  accessible manner that will increase the  utility of this report for all levels of staff and  for public library trustees. this report is  written for the practitioner, with  information on trends and the strategic  planning implications of the findings  provided throughout.     the report’s authors acknowledge the  limitations imposed in presenting such  broad and large‐scale research results, but  the almost exclusive use of average  percentages without any details about the  standard deviations for these measures is a  barrier to the utility of some of the data. in  addition, there is an absence of other  descriptive statistics. the general purpose of  the study is clearly presented, and the  published study presents interesting  findings from the data.    the work of this study and the benchmarks  it has established through the data collected  are incomparable and will provide an ample  resource to inform strategic human resource  planning across canada. the researchers  indicate a willingness to share aspects of the  data set with others. the further analysis  and research that should be sparked from  this rich data are both exciting and urgent.  a call for submissions to a collection of  essays (to be published by the canadian  library association) was made in august  2006 (details at  http://www.ls.ualberta.ca/8rs/whatʹsnew.ht ml ) and we can expect further analysis and  insight based on the strong foundation that  this research project has provided.    there is a need for continual attention to  human resource issues in canada. on  december 12, 2006, ontario institutions no  longer have the right to impose mandatory  retirement at age 65. this could have a  significant impact on human resources in  canadian libraries, as 43% of librarians in  canada are employed in ontario. due to the  timing of this announcement, it was not  figured in the projections of the 8rs report.  http://www.ls.ualberta.ca/8rs/what%ca%b9snew.ht evidence based library and information practice 2007, 2:1  127   the 8rs research team was commissioned  by the cultural human resources council  to carry out a further study entitled training  gaps analysis for librarians and library  technicians (8rs research team 2006). this  study made use of the data collected for the  future of human resources in canadian  libraries in addition to further research to  resolve some of the gaps identified in the  original report. the executive summary can  be found online at  <http://www.culturalhrc.ca/competencies/li brarians/revised%20execsummary%20may %20rev.pdf> and the full report can be  purchased through the canadian library  association (orders@cla.ca).        http://www.culturalhrc.ca/competencies/li evidence based library and information practice 2012, 7.2 4 evidence based library and information practice editorial responsibilities 2012. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. editor-in-chief: alison brettle associate editor (articles): wayne jones associate editor (evidence summaries): lorie kloda associate editor (classics): jonathan eldredge associate editor (reviews): denise koufogiannakis associate editor (commentaries, using evidence in practice, ebl101, news): alison brettle production editor: katrine mallan editorial intern: michelle dunaway copyeditors: heather pretty (lead copyeditor), georgianne bordner, molly des jardin, linda ferguson, richard hayman, stacey penney, mary virginia taylor , alison yeoman indexing support: pam morgan http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/ conference paper   assessing the impact of embedding online academic and information literacy resources into a first year business course   sandy rae librarian (business) griffith university brisbane, queensland, australia email: s.rae@griffith.edu.au   margaret hunn learning adviser (business) griffith university brisbane, queensland, australia email: m.hunn@griffith.edu.au   received: 8 aug. 2015     accepted: 8 oct. 2015      2015 rae and hunn. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objectives – literature supports the concept that embedding academic and information literacy support into first year university courses enables students to proceed more confidently with researching and writing their assignments, and thus contributes to student success in their course. a need was identified for academic and information literacy support for a cohort of first year business students as part of the development of online course content for griffith online, the institution’s online study degree option. this led to a collaboration between information literacy librarians, learning skills advisers, educational designers, and academic course convenors to develop and implement online resources. this paper will present findings on the impact of these online resources.   methods – drawing on measures and methods identified in iso16439 “information and documentation: methods and procedures for assessing the impact of libraries” (international organisation for standardization, 2014), in conjunction with the indicators offered by lizzio’s (2006) five senses of success framework, evidence was collected and combined from a variety of sources over semester 2, 2014, and semester 1, 2015 to assess the impact of the online resource. inferred evidence was gathered from usage statistics (number of hits on the sites) and from performance measures (comparing student essay grade between those that did and did not use the resource). solicited evidence was gathered from a survey of students, students in focus groups, and interviews with course lecturers, tutors and other stakeholders.   results – the inferred evidence showed a positive impact on the student success indicators of the sense of resourcefulness, capability, connection, purpose and identity. the solicited evidence suggests that students saw the online resource in a positive light and that staff were happy with the impact it had on students’ work and learning. it is believed that the gathered evidence indicates the module did achieve the impact objective of a positive impact on the contribution to student success for these first year business students.   conclusions – the evidence has shown that this resource contributed to student success, and that staff and student satisfaction with the resource contributed to increased confidence with student academic skills and information literacy in respect to their assignment task. assessing the impact of the online resource on student success has helped to demonstrate the value of the library at griffith university to the wider community. the four-pronged collaboration relationship required for this approach was fostered with stakeholders outside of the library. introduction     one of the key future trends in higher education identified in both the “new media consortium (nmc) horizon report” (johnson, adams becker, estrada, & freeman, 2014) and the “top trends in academic libraries” (association of college and research libraries [acrl] research planning and review committee, 2014) is the integration of online, hybrid, and collaborative learning. like many academic institutions, griffith university is moving to online modes of course delivery. for learning advisers and information literacy librarians to address this shift, it is necessary to engage with the e-learning environment. a core first year bachelor of business course moving into the online environment presented the opportunity for collaboration between an information literacy librarian, a learning adviser, an academic, and an educational designer in the creation of an online resource for the teaching of research and writing skills in support of student assessment.   literature supports the concept that embedding academic and information literacy skills into first year university courses enables students to proceed more confidently with researching and writing their assignments, and thus contributes to student success in their course. the creation of online embedded resources represented a new direction for library teaching and learning at griffith university. therefore, it was necessary to assess the impact of the resources to clarify the library’s contribution to student success and academic library value.   drawing on measures and methods identified in information and documentation: methods and procedures for assessing the impact of libraries, iso16439” (international organisation for standardisation [iso], 2014), evidence was collected and combined from a variety of sources to assess the impact of the online resource. the evidence shows that this resource contributed to student success, and that staff and student satisfaction with the resource contributed to increased student confidence with academic and information literacy in respect to their essay assessment task. an integral part of this success was due to the collaboration between information literacy librarians and other stakeholders in providing academic and information literacy support to the first year business student experience and engagement.   background   griffith university offers a mixed mode method of delivery which consists of face to face and online offerings in courses and importantly, requires equity of access to services for both on-campus and online students. the university consists of five campuses over south east queensland, with a student body of over 43,000. the griffith business school, with a student population of over 11,000, delivers courses at all five campuses as well as online. historically, embedded information literacy skills and academic skills have been taught face to face in lecture time; however, due to the increasing amount of course content to be covered in lecture times, the opportunity to contribute has been severely reduced in the griffith business school. as more students move into the online method of course delivery, face to face delivery also represents a lack of equity in delivery for these students. embedding online literacy resources offered an opportunity to redress this issue for a compulsory first year bachelor of business course, which had over 1,000 students enrolled. the online resource “research and writing for business students” (the module) was created in collaboration with the course academic, the educational designer responsible for getting the course online, and business team library and learning staff, consisting of an information literacy librarian and a learning adviser.   eight topics covering researching, writing, and referencing were included in the module to support these students in their essay assignment task. the eight topics created consisted of:   navigating the library website unpacking the question scholarly and peer reviewed journal articles searching the library catalogue writing the plan searching google scholar writing the essay referencing   these topics covered the key academic and information literacy skills needed to scaffold the completion of the essay assignment task. the module was positioned in the course assessment folder, below the essay assignment task, in the learning management system, blackboard, in semester 2, 2014 and semester 1, 2015. it was utilised in several tutorial and workshop sessions by course tutors to explain key literacy skills needed to complete the essay assignment task, and so was highly embedded into the teaching of the course.   initial discussions about the creation of the resource highlighted the need for seamlessly embedding it in the course and for it not to appear as an add-on. to do this it was necessary to use the same interface and design established for the rest of the course and for the resource to be purposely built for the specific essay assignment task. each topic of the module included a short youtube video with additional information and links to further resources, and focussed on the specific essay assignment task. the topics were personalized as much as possible in order to engage with students in the online environment, as suggested in the nmc horizon report (johnson et al., 2014). for example, “searching the library catalogue” used keywords relevant to the essay assignment task, and “writing the essay” utilised exemplars provided by the academic.   the module was designed in collaboration with the educational designer to complement the overall course interface, and the content was created in collaboration with the library business team learning adviser and librarian and the course academic. importantly, it was strategically envisaged that the template for the module and some topics could also be repurposed in other courses.   literature review   collaboration   embedding information literacy and academic writing instruction into course curricula is not new. literature supports that a collaborative approach to the embedding of information literacy instruction in course curricula has positive outcomes for students (creaser et al., 2014; menchaca, 2014; nelson, 2014; pan, ferrer-vinent, & bruehl, 2014). a three-pronged collaboration model between an academic, a learning adviser, and an information literacy librarian has been suggested to overcome the often unrelated way that information literacy and academic skills have been presented in the past to university students (einfalt & turley, 2009, 2013; kokkinn & mahar, 2011; taib & holden, 2013). tinto (2005) suggests that any support given to students should be related to a specific course and a specific task in order to help students succeed in that course and actively involve them in learning. theis, wallis, turner, and wishart (2014) agree that the development of students’ academic literacies is enhanced through the use of curriculum embedded resources rather than add-on generic offerings from the library.   any support strategy must be contextualised and connected to the environment in which student learning takes place (nelson, 2014). the “nmc horizon report” (johnson et al., 2014) identifies the rise of online pedagogy at higher education institutions. the e-learning environment can provide a student-centred approach where students can proceed at their own pace and use different media types that suit their style of learning (lu & chiou, 2010). for information and academic literacy resources to be useful in an online environment, collaboration in creation should be widened from the three-pronged approach to include an educational designer in order to enhance the environment in which the resources are to be placed (gunn, hearne, & sibthorpe, 2011). as such, a four-pronged collaboration model between librarian, learning adviser, educational designer, and course academic was used in the development of the module.   evaluation   “the value of academic libraries: a comprehensive research review and report” (oakleaf, 2010) summarizes the importance for academic libraries to demonstrate their value, particularly in light of budgetary restraint and competing stakeholder interests. this importance is also emphasised in other studies (bausman, ward, & pell, 2014; brown & malenfant, 2012; creaser & spezi, 2012, 2014; gibson & dixon, 2011; tenopir, 2011). rather than just reporting on library achievements, kranich, lotts, and springs (2014) explore the notion of academic libraries turning outward so that library impact is measured in the contributions library achievements make to the broader community. whilst there are many ways of defining value, oakleaf (2010) identifies the two main approaches as financial value and impact value. menchaca (2014) argues that for measuring value in the academic library, impact is the more important measure as it relates to learning. as libraries engage with the online space and the embedding of seamless resources, they face new challenges as users may no longer identify that space with the library. consequently, the need to demonstrate impact becomes more crucial (sputore, humphries, & steiner, 2015).   studies support measuring impact that aligns with university outcomes (brown & malenfant, 2012; oakleaf, 2010; pan et al., 2014). library impact on institutional outcomes of “student success, student achievement, student learning, and student engagement” can be explored through evidence based practice (oakleaf, 2010, p. 12). as mentioned in the literature, links, although not always causal, have been examined between library usage and student outcomes such as attainment, recruitment, and retention (haddow, 2013; hubbard & loos, 2013; soria, fransen, & nackerud, 2013, 2014; stone & ramsden, 2013). gathering data, analyzing it, and presenting findings can demonstrate to academic faculty that collaborating with library staff is worthwhile and can contribute to student outcomes, thus creating library value (oakleaf, 2010). as impact and value are so closely linked, this allows establishment of not only value for the module but broader academic community library value (bausman et al., 2014; bonfield, 2014; brown & malenfant, 2012; creaser & spezi, 2012; menchaca, 2014; oakleaf, 2010; pan et al., 2014; tenopir, 2011).   “information and documentation: methods and procedures for assessing the impact of libraries, iso16439” (iso, 2014) provides an internationally recognised basis for assessing library impact (henczel, 2014). the standard describes effects such as “changes in skills and competence” and “higher success in research, study or career” (iso, 2014, p. 14) as demonstrating library impact. in addition, collaboration between library and academic staff for embedding library resources in courses can also affect library impact through changes in attitudes and behaviour (iso, 2014). combining methods can provide a fuller or richer story for assessing impact, but may also need more detailed analysis, as the findings from different source data may not be consistent (iso, 2014). henczel (2014, sept.) provided a diagrammatic interpretation of the standard methods and procedure for assessing the impact of libraries (figure 1).   at griffith university, lizzio’s (2006, 2011) five senses of success framework has been used as a predictor of student outcomes. this framework examines students’ success as depending on their sense of capability, connection, purpose, resourcefulness, and identity, and is particularly useful as it facilitates “conscious and reflective practice” and forms a basis for student engagement strategies for the broader griffith university community (wilson, 2009, p. 7). a sense of resourcefulness and capability can be promoted if students can find the information they need and are prepared for assignment tasks at university level (lizzio, 2011; wilson, 2009). a sense of connection is encouraged by the quality of relationships that are formed at university with peers, staff and the affiliation with their school (lizzio, 2011; wilson, 2009). as strengths and talents are developed and students learn how things are done at university, a sense of purpose and identity are fostered (lizzio, 2011; wilson, 2009). initially, the five senses of success framework was introduced to support student retention and engagement within the first year, but this has been expanded to incorporate the whole student lifecycle (lizzio, 2011). the use of the five senses of success framework to examine the student experience is supported in other studies that evaluate student support and engagement, and adds metrics that are meaningful outside of the library environment (burnett & larmar, 2011; chester, burton, xenos, & elgar, 2013; hutchinson, mitchell, & st john, 2011; sidebotham, fenwick, carter, & gamble, 2015). the five senses of success framework indicators can be aligned with those characteristics that have been previously used to evaluate e-learning programs, such as usability, content richness, flexibility, and learner community (chiu, hsu, sun, lin, & sun, 2005; lu & chiou, 2010; wang, 2003).     figure 1 henczel’s (2014, sept.) interpretation of impact assessment process based on iso16439.     figure 2 impact assessment process based on henzcel’s (2014, sept.) interpretation.     aim   the aim of this paper is to assess the impact of embedding an online academic and information literacy resource into a first year business course. measuring the impact will not only determine whether the resource created and provided to students made any difference to their success, but also demonstrate academic library value.   methods   drawing on measures and methods identified in iso16439 (2014), evidence was collected and combined from a variety of sources over semester 2, 2014, and semester 1, 2015 to assess the impact of the module on student success.   using iso16439 (2014) as interpreted by henczel (2014, sept.) (figure 2), an impact objective was established to discover if the module contributed to student success. this objective was aligned with impact indicators based on lizzio’s (2006, 2011) five senses of success framework of capability, connection, purpose, resourcefulness, and identity. inferred and solicited evidence was collected to support and explore those indicators.   inferred evidence was gathered from usage statistics (number of hits on the module), and from performance measures (comparing student essay grade between those that did and did not use the module). solicited evidence was gathered from a survey of students, students in focus groups, and interviews with other stakeholders such as course lecturers, tutors, and educational designers.   inferred evidence   statistics tracking in the learning management system was activated for the module for both semester 2, 2014 and semester 1, 2015. usage data for day and time of access to the module was also available from the learning management system, blackboard. usage data was matched to assessment grades from the grade center and the results analyzed using microsoft excel.   solicited evidence   following a pilot survey in semester 2, 2014, a student survey was conducted in semester 1, 2015 with an announcement and link to the survey placed in the learning management system, blackboard. the survey contained basic demographic questions, five response scale questions, and one open ended question for comments. an even number of options for the response scale questions ranging from strongly agree to strongly disagree was used to remove the undecided or neutral response (iso, 2014). each of the five response scale questions was designed to address one of lizzio’s (2011) five senses of success (table 1).     table 1 survey questions survey questions         five sense of success (lizzio, 2011) 1. the resource was easy to navigate and understand. resourcefulness; did the module help students to find information they need? 2. after using the resource, i was more confident to research and write academically. capability; did the module help students to prepare for tasks at university? 3. after using the resource, i was more confident in talking about the essay with other students and teaching staff. connection; did the module help students to engage with peers, academic staff and support staff? 4. after using the resource, i was better prepared to complete the essay. purpose; did the module help students to develop strengths and talents? 5. the resource helped to explain how assignments are done at university level. identity; did the module help students to learn how things are done as a business student at university?       two focus groups were held during face to face tutorial time: 15 students in the first focus group and 18 students in the second group. students were asked if they had used the module and what they found useful about it.   stakeholder interviews were conducted and an email was sent to the lecturer, tutors, and educational designer with the following five questions:   did you refer to any sections of the module in your tutorial sessions? what was your impression of it? did any of your students comment on it? if so, what did they say? do you have any suggestions as to how to improve on it?   results   inferred evidence   inferred evidence data collection was from spreadsheets within the learning management system which were collated with spreadsheets from the grade center. although a time consuming process, the online data collection resulted in a clearer picture of how students accessed and returned to the module, and matching usage with student essay assignment grades offered clearer information than could be gleaned from evaluating face to face teaching sessions.   usage data   as presented in table 2, the module was accessed 4,442 times in semester 2, 2014.     table 2 module usage statistics semester 2 2014 semester 1 2015 no. of students enrolled 1,023 784 hits to module 4,442 6,537 no. of students who accessed module (unique hits) 910 (89%) 750 (96%) average number of hits by students who used the module 4.88 8.72 % of students who used the module >once 90% 95%     for individual students, this varied from not accessing the resource at all to accessing the resource 29 times. in 2014, 89% of the students accessed the resource, increasing to 96% in 2015. the average number of hits to the resource per student was 4.88, indicating that students did find value in the resource, as they went back to it multiple times. in semester 1, 2015, the module was accessed 6,537 times, varying from accessing the resource once to accessing the resource 37 times. the average number of hits to the resource per student was 8.72.     usage data for day and time of access to the module (figure 3) highlights the 24/7 availability of the module. the module was used on all days of the week and at all hours of the day.     figure 3 module usage statistics: days of week and hour of day access.     performance results   comparing the average essay grade for those students who used and those students who did not use the module (table 3) indicates that use of the resource achieves a higher than average mark. in semester 2, 2014, the average class essay grade was 64%, those who used the module acquired a slightly higher than average grade of 65%, and for those who did not use the module the average grade was 47%. for semester 1, 2015, the average class grade was 58%, with those who used the module receiving 61% and those who did not use the module receiving on average 15%. this larger difference in 2015 between users and non-users of the module could be attributed to the much higher usage of the module in 2015. a high percentage of students (96%) accessed the module in 2015. the remaining 4% who did not may represent the lesser engaged students.     table 3 comparison of module usage and assignment grade semester 2, 2014 semester 1, 2015 average essay grade 16/25 (64%) 17.5/30 (58%) average essay grade those who used the module 16.25/25 (65%) 18.2/30  (61%) average essay grade those who did not use the module 11.7/25 (47%) 4.5/30 (15%)     solicited data   solicited evidence collection tended to be better facilitated by a face to face approach. the online survey did not reveal as much information as the face to face focus groups.   survey   the median survey response to all questions was between “slightly agree” and “agree,” indicating that the module contributed to student success, as shown in figure 4. this was supported by survey comments, in particular “i was feeling quite overwhelmed by the task of writing the essay, however after using the research and writing tool i feel a lot more confident and at ease as i have a better understanding of how to approach the task. thank you.”     figure 4 survey results.     however, the number of responses to the survey was low (42 responses, of which 35 used the module) and hampered by institutional policy on survey timing. this meant that the survey had to be concluded the same day as the essay assignment was due, limiting student reminders. it is worth noting that surveys as a method of gathering evidence in the academic or institutional environment for evaluation of assessment items needs to be carefully considered within the larger institutional environment due to conflicting survey priorities. even though response rates were low, the data from the survey adds to the overall picture of assessing the module and highlights the advantages of a combined methods analysis.   focus groups   overall, the focus group discussions were positive. for those students who did use the resource, they found the module easy to navigate, particularly with the table of contents, as students could easily select the topics that were most useful to them. of the eight topics, those rated as most useful were “writing the report” and “referencing,” although others found the searching topics useful as they were unsure of search terms. other comments included “had no idea what to do and resource gave me lots of ideas of what to do,” “helped clarify the questions,” and “video format easy to watch.” two students did not watch the videos as they preferred to use the transcript, which highlights the need to consider different learning styles.   the most frequent reason given for not using the module was that they did not know it was there, which highlights the importance of collaboration for support and promotion from academic staff. interestingly, students responded that they had not used the module even though topics had been shown during tutorial time. this highlighted the problem of assessing the impact of a highly embedded resource when students assumed that it was just another teaching tool of the course and did not associate it with being provided by the library.   stakeholder interviews   tutors reported via the course convenor that they received fewer than usual academic and information literacy questions about the essay assignment from students, and the “students said they found the videos helpful and came to see me to clarify points in relation to their essay.”   tutors’ comments below highlight the use of the module as a teaching tool:   “i can report that the research and writing for business students module on the course website were a valuable learning and teaching tool. i referred to every section of it during tutorials/workshops leading up to the due date for the submission of the essay.”    “the short video clips were generally very good and i received overwhelmingly positive feedback from my student cohort.”     some tutors mentioned that whilst the module was embedded into the tutorial and videos scheduled at various points in the semester, technical issues on numerous occasions prevented the videos from being played.   tutors’ comments also highlighted the timesaving benefits of the module for tutors:   “the fact that the resources are all together is handy for students and helpful to tutors who have limited time allocated in class to develop students "basic" writing skills. the students can choose to access the support tools/information whenever, wherever, however many times they like.”   “in responding to queries from students about an aspect of their research or writing process, i was able to direct them to the relevant resource in addition to providing my own guidance by e-mail or in person.”   one suggested improvement “would be to make the video clips more concise to hold students attention. this could perhaps be achieved by using more focused and direct language.”   the educational designer has since shown the module to other interested academics and commented that:   “the quality and value pretty much speaks for itself. academics like that it's co-located in the assessment folder so it's easy for students to find and it's contextual. they like that it's similar to what is taught into a course on-campus, but it's online... which means students can access it whenever they like, when they need it, as they are doing their assignments... they can see that it will lead to fewer questions for them!”   discussion   gathering the data to support the impact objective was made easier with the use of the framework offered by iso16439 (2014) for assessing library impact, in conjunction with the indicators offered by the five senses of success framework (lizzio, 2011). using this multifaceted approach to data collection, as recommended by the standard, allowed for a fuller picture to be drawn.    the inferred evidence showed a positive impact. the usage results indicate that the module added to the student sense of resourcefulness and capability; they were assisted in finding the right sorts of information they needed at the right time (lizzio, 2011). the high number of repeat visits to the module at various times of the day indicates that students found the module of assistance in writing and researching for their assignment. the increase in usage over the semesters may highlight the uptake of the module as a teaching tool by teaching staff. the high number of average hits to the module per student indicates the library’s engagement with the students enrolled in these courses. linking this back to student success, this high usage could be interpreted as the module contributing to the resourcefulness and capability of students in engaging them in the learning process in assignment preparation and research (lizzio, 2011).   the performance measures indicate that the module added to the students’ sense of capability; they were more able to complete the assignment to satisfactory levels if they had used the module (lizzio, 2011). matched with the high usage rates, these performance statistics could indicate that those students who used the module were more engaged with the course.   the solicited evidence suggests that students saw the module in a positive light and that staff were happy with the impact it had on students’ work and learning. the student survey and focus groups gave some indication that students found a sense of purpose and identity in their preparation (lizzio, 2011). their comments and survey responses supported that they were learning how to research and write for their assignment task, as well as how things were done at university (lizzio, 2011). the interviews with stakeholders gave a sense of promoting connection, that students were part of their learning process and were able to access help from the griffith university community (lizzio, 2011).   assessing the impact of the module provides the opportunity to reflect on practice. from feedback it was evident that some structural changes need to be made to the module to make it more targeted and direct. looking at the module from the student’s viewpoint and also the environment into which it has been embedded has made it clear that any topic which was not assessment focused needs to be re-examined. the necessity of collection and analysis of data is highlighted by the fact that module usage may not correspond with student identification of assistance from a learning adviser or librarian, as the resource for assignment assistance is so embedded into the course. this indicates an area for further study, as highlighted by sputore et al. (2015). however, the evidence collected does provide support for continuing collaboration using the four-pronged collaboration model between librarians, learning advisers, academics, and educational designers in the production of these embedded online assessment based resources (gunn et al., 2011).   the assessment of the module enabled the alignment of library practices to institutional strategic and operational plans through collaboration and building partnerships with academics, learning advisers, and educational designers. it has helped to demonstrate the library’s contribution to the achievement of griffith university’s strategic changes, such as meeting operational plans of a fully online, seamless student model and meeting opportunities presented by changes to teaching semesters.   using a combination of iso16439 (2014) with henczel’s (2014, sept.) diagrammatic interpretation of the standard and lizzio’s (2006, 2011) five senses of success framework may be beneficial to other academic libraries and the broader library community wishing to engage in evidence based practice to measure library impact that aligns to institutional outcomes. for other libraries, different impact objectives and indicators more relevant to their institutional outcomes may be more beneficial in assessing impact.   engaging in this research has provided the opportunity to document procedures and practices surrounding data gathering, analyzing, and reporting. documenting the process is valuable to establish a library connection to institutional outcomes, and worth considering for any libraries wishing to engage in evidence based practice.   conclusions   overall, the evidence showed that over 90% of students accessed the online resource “research and writing for business students,” and it was well received by both staff and students. students have stated that it gave them the confidence to get started on their assignments, and academic staff commented that it decreased the amount of generic questions they received about the assignment. using the criteria of the five senses of success (lizzio, 2006, 2011) as impact indicators, it is believed that the gathered evidence indicates the module did achieve the impact objective of a positive impact on the contribution to student success for these first year business students.   assessing the impact of the online resource “research and writing for business students” on student success has helped to demonstrate the value of the library at griffith university to the wider community. the business library and learning team at griffith university moved its teaching practice into the online environment, but did not lose relevance in supporting students. the four-pronged collaboration relationship required for this approach was fostered with stakeholders outside of the library.   references   association of college and research libraries (acrl) research planning and review committee. (2014). top trends in academic libraries: a review of the trends and issues affecting academic libraries in higher education. college & research libraries news, 75(6), 294-302. retrieved from http://crln.acrl.org/content/75/6/294.full.pdf+html   bausman, m., ward, s. l., & pell, j. (2014). beyond satisfaction: understanding and promoting the instructor-librarian relationship. new review of academic librarianship, 20(2), 117-136. http://dx.doi.org/10.1080/13614533.2014.911192   bonfield, b. (2014). how well are you doing your job? you don’t know. no one does. in the library with the lead pipe. retrieved from http://www.inthelibrarywiththeleadpipe.org/2014/how-well-are-you-doing-your-job-you-dont-know-no-one-does/   brown, k., & malenfant, k. j. (2012). connect, collaborate, and communicate: a report from the value of academic libraries summits. chicago, il: association of college and research libraries. retrieved from http://www.ala.org/acrl/sites/ala.org.acrl/files/content/issues/value/val_summit.pdf   burnett, l., & larmar, s. (2011). improving the first year through an institution-wide approach: the role of first year advisors. the international journal of the first year in higher education, 2(1), 21-35. http://dx.doi.org/10.5204/intjfyhe.v2i1.40   chester, a., burton, l. j., xenos, s., & elgar, k. (2013). peer mentoring: supporting successful transition for first year undergraduate psychology students. australian journal of psychology, 65(1), 30-37. http://onlinelibrary.wiley.com/doi/10.1111/ajpy.12006/epdf   chiu, c. m., hsu, m. h., sun, s. y., lin, t. c., & sun, p. c. (2005). usability, quality, value and e-learning continuance decisions. computers & education, 45(4), 399-416. http://dx.doi.org/10.1016/j.compedu.2004.06.001   creaser, c., cullen, s., curtis, r., darlington, n., maltby, j., newall, e., & spezi, v. (2014). working together: library value at the university of nottingham. performance measurement and metrics, 15(1/2), 41-49. http://dx.doi.org/10.1108/pmm-03-2014-0011   creaser, c., & spezi, v. (2012). working together: evolving value for academic libraries. retrieved from http://libraryvalue.files.wordpress.com/2012/06/ndm-5709-lisu-final-report_web.pdf   creaser, c., & spezi, v. (2014). improving perceptions of value to teaching and research staff: the next challenge for academic libraries. journal of librarianship and information science, 46(3), 191-206. http://dx.doi.org/10.1177/0961000613477678   einfalt, j., & turley, j. (2009). engaging first year students in skill development: a three-way collaborative model in action. journal of academic language and learning, 3(2), a105-a116. retrieved from http://journal.aall.org.au/index.php/jall/article/view/87/73   einfalt, j., & turley, j. (2013). partnerships for success: a collaborative support model to enhance the first year student experience. the international journal of the first year in higher education, 4(1), 73-84. http://dx.doi.org/10.5204/intjfyhe.v4i1.153   gibson, c., & dixon, c. (2011). new metrics for academic library engagement. in d. m. mueller (ed.), a declaration of interdependence acrl conference march 30–april 2, (pp. 340-351). chicago, il: association of college and research libraries. retrieved from http://www.ala.org/acrl/sites/ala.org.acrl/files/content/conferences/confsandpreconfs/national/2011/papers/new_metrics.pdf   gunn, c., hearne, s., & sibthorpe, j. (2011). right from the start: a rationale for embedding academic skills in university courses. journal of university teaching & learning practice, 8(1), 1-10. retrieved from http://ro.uow.edu.au/cgi/viewcontent.cgi?article=1159&context=jutlp   haddow, g. (2013). academic library use and student retention: a quantitative analysis. library & information science research, 35(2), 127-136. http://dx.doi.org/10.1016/j.lisr.2012.12.002   henczel, s. (2014). the impact of library associations: preliminary findings of a qualitative study. performance measurement and metrics, 15(3), 122-144. http://dx.doi.org/10.1108/pmm-07-2014-0025   henczel, s. (2014, sept.). library metrics workshop. presented at griffith university, brisbane, queensland, australia, september 12, 2014.   hubbard, m. a., & loos, a. t. (2013). academic library participation in recruitment and retention initiatives. reference services review, 41(2), 157-181. http://dx.doi.org/10.1108/00907321311326183   hutchinson, l., mitchell, c., & st john, w. (2011). the transition experience of enrolled nurses to a bachelor of nursing at an australian university. contemporary nurse, 38(1-2), 191-200. http://dx.doi.org/10.5172/conu.2011.38.1-2.191   international organisation for standardisation. (2014). information and documentation: methods and procedures for assessing the impact of libraries, iso16439. retrieved from http://saiglobal.com   johnson, l., adams becker, s., estrada, v., & freeman, a. (2014). nmc horizon report: 2014 higher education edition. austin, tx: the new media consortium. retrieved from http://cdn.nmc.org/media/2014-nmc-horizon-report-he-en-sc.pdf   kokkinn, b., & mahar, c. (2011). partnerships for student success: integrated development of academic and information literacies across disciplines. journal of academic language and learning, 5(2), a118-a130. retrieved from http://journal.aall.org.au/index.php/jall/article/view/170/115   kranich, n., lotts, m., & springs, g. (2014). the promise of academic libraries: turning outward to transform campus communities. college & research libraries news, 75(4), 182-186. retrieved from http://crln.acrl.org/content/75/4/182.short   lizzio, a. (2006). designing an orientation and transition strategy for commencing students: a conceptual summary of research and practice. griffith university first year experience project. retrieved from http://www.griffith.edu.au/__data/assets/pdf_file/0008/51875/alfs-5-senors-paper-fye-project,-2006.pdf   lizzio, a. (2011). succeeding@ griffith: next generation partnerships across the student lifecycle. in griffith university student lifecycle, transition and orientation. retrieved from http://www.griffith.edu.au/learning-teaching/student-success/first-year-experience/student-lifecycle-transition-orientation   lu, h. p., & chiou, m. j. (2010). the impact of individual differences on e-learning system satisfaction: a contingency approach. british journal of educational technology, 41(2), 307-323. http://dx.doi.org/10.1111/j.1467-8535.2009.00937.x   menchaca, f. (2014). start a new fire: measuring the value of academic libraries in undergraduate learning. portal: libraries and the academy, 14(3), 353-367. http://dx.doi.org/10.1353/pla.2014.0020   nelson, k. (2014). the first year in higher education-where to from here? the international journal of the first year in higher education, 5(2), 1-20. http://dx.doi.org/10.5204/intjfyhe.v5i2.243   oakleaf, m. (2010). the value of academic libraries: a comprehensive research review and report. chicago, il: association of college and research libraries. retrieved from http://www.ala.org/acrl/sites/ala.org.acrl/files/content/issues/value/val_report.pdf   pan, d., ferrer-vinent, i. j., & bruehl, m. (2014). library value in the classroom: assessing student learning outcomes from instruction and collections. the journal of academic librarianship, 40(3–4), 332-338. http://dx.doi.org/10.1016/j.acalib.2014.04.011     sidebotham, m., fenwick, j., carter, a., & gamble, j. (2015). using the five senses of success framework to understand the experiences of midwifery students enroled in an undergraduate degree program. midwifery, 31(1), 201-207. http://dx.doi.org/10.1016/j.midw.2014.08.007   soria, k. m., fransen, j., & nackerud, s. (2013). library use and undergraduate student outcomes: new evidence for students' retention and academic success. portal: libraries and the academy, 13(2), 147-164. http://dx.doi.org/10.1353/pla.2013.0010   soria, k. m., fransen, j., & nackerud, s. (2014). stacks, serials, search engines, and students' success: first-year undergraduate students' library use, academic achievement, and retention. the journal of academic librarianship, 40(1), 84-91. http://dx.doi.org/10.1016/j.acalib.2013.12.002   sputore, a., humphries, p., & steiner, n. (2015). sustainable academic libraries in australia: exploring ‘radical collaborations’ and implications for reference services. paper presented at ifla wlic, 15-21 august, 2015, cape town, south africa. retrieved from http://library.ifla.org/1078/1/190-sputore-en.pdf   stone, g., & ramsden, b. (2013). library impact data project: looking for the link between library usage and student attainment. college & research libraries, 74(6), 546-559. http://dx.doi.org/10.5860/crl12-406   taib, a., & holden, j. (2013). "third generation" conversations: a partnership approach to embedding research and learning skills development in the first year. a practice report. the international journal of the first year in higher education, 4(2), 131-136. http://dx.doi.org/10.5204/intjfyhe.v4i2.178   tenopir, c. (2011). beyond usage: measuring library outcomes and value. library management, 33(1/2), 5-13. http://dx.doi.org/10.1108/01435121211203275   thies, l., wallis, a., turner, a., & wishart, l. (2014). embedded academic literacies curricula: the challenges of measuring success. journal of academic language & learning, 8(2), 43-59. retrieved from http://dro.deakin.edu.au/eserv/du:30064956/thies-embeddedacademic-2014.pdf   tinto, v. (2005 january). taking student success seriously: rethinking the first year of college. in ninth annual intersession academic affairs forum, california state university, fullerton, 05-01. retrieved from http://www.hartnell.edu/sites/default/files/library_documents/bsi/v%20tinto%20retention.pdf   wang, y. s. (2003). assessment of learner satisfaction with asynchronous electronic learning systems. information & management, 41(1), 75-86. http://dx.doi.org/10.1016/s0378-7206(03)00028-4   wilson, k. (2009 june). the impact of institutional, programmatic and personal interventions on an effective and sustainable first-year student experience. presented at the 12th pacific rim first year in higher education conference, preparing for tomorrow today: the first year experience as foundation. townsville, australia. retrieved from https://www.griffith.edu.au/__data/assets/pdf_file/0007/409084/fyhe-2009-keynote-keithia-wilson.pdf evidence based library and information practice feature commentary   being a solo eblip practitioner   suzanne lewis manager, library services central coast local health district gosford, new south wales, australia email: suzanne.lewis@health.nsw.gov.au   virginia wilson director, centre for evidence based library and information practice (c-eblip) university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   received: 5 jan. 2014      accepted: 15 sept. 2015      2015 lewis and wilson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   one of the issues that emerged at the 8th international evidence based library and information practice conference (eblip8) held in brisbane, australia in july 2015 is the challenge of being a solo practitioner of evidence based library and information practice (eblip). this matter was raised in response to several papers and also during the ‘evidently practical’ panel discussion. this commentary explores the barriers encountered by solo eblip practitioners and also offers some possible solutions.   who are solo eblip practitioners? they are librarians who wish to make their professional practice evidence based but who work in isolation, without the immediate support of colleagues. the most obvious sub-group are librarians working in one-person libraries, but librarians who work in larger libraries, surrounded by colleagues, may also consider themselves to be solo eblip practitioners if evidence based practice is not encouraged, supported, or even recognized by peers and managers. in some environments eblip may receive token recognition or formal acknowledgment but may not be underpinned by practical, day-to-day cultural implementation. a number of eblip8 conference delegates identified themselves as working in either one-person libraries or libraries where eblip was not supported.   there are recognized barriers or challenges that arise with the practice of eblip and these are evident amongst solo practitioners as well as those working in a larger community. booth, in a thematic synthesis of 55 papers that reported at least one barrier to eblip, found that potential obstacles include the following:   lack of time lack of financial resources lack of infrastructure lack of organizational support lack of evidence base and/or poor access to evidence base lack of confidence real or perceived lack of research skills lack of research culture (booth, 2011, adapted from p. 1 of 15)   while lack of time, finances, confidence, or skills are common themes across the eblip community, a significant insufficiency for solo eblip practitioners is lack of support, organizational or otherwise. having someone to brainstorm with, collaborate with, and commiserate with helps to generate momentum and makes eblip more doable, especially in the early stages. but where can a solo practitioner find that support?   a solo eblip practitioner needs to find a tribe. as seth godin states, “[a] group needs only two things to be a tribe: a shared interest and a way to communicate” (2008, p. 21__). finding your “tribe”, a network of likeminded colleagues who can support you in your eblip endeavours, is essential. in these days of online interconnectedness this is easier than it has been in the past. your local, state/provincial, or national professional association is a good place to start. membership in a library and information science (lis) professional association has many benefits including putting you in touch with other professionals in your area who share your interests. professional associations also help keep you up to date with the latest news and trends, often provide access to lis journals, give members discounted registration for events, conferences, and training, and may operate professional development schemes leading to formal accreditation.   your network will also be found at specific eblip events such as the biennial international eblip conference. for many lis practitioners face to face conference attendance is difficult due to financial, personal, or workplace barriers. however virtual attendance is possible. for example, at the recent eblip8 conference in brisbane, australia there was a very active twitter feed (#eblip8), now available via storify (https://storify.com/eblip8/). audio of the panel discussion on day two of the conference was available live, with listeners able to submit questions remotely via twitter, and a recording of the discussion is also available on youtube (http://eblip8.info/2015/07/14/panel-session-recording-available/). it’s not the same as being there but it’s a great way to participate remotely.   members of your eblip group are reading, or even contributing to, the eblip journal (https://ejournals.library.ualberta.ca/index.php/eblip). this open access peer-reviewed journal is published quarterly and offers a range of content to support the eblip practitioner including original research articles, commentaries, and approximately ten evidence summaries per issue which critically appraise and synthesize previously published research and include practical applications for the research. you can register with the open journal system on the main page of the eblip journal to receive tables of contents of the latest issues via email. there are also opportunities to become involved with the journal as an evidence summary writer or peer reviewer.   if you would like to participate in reading and discussing the lis literature in a more informal setting an online journal club may be right for you. for example the health libraries australia group of alia, the australian library and information association, runs an online journal club using a wiki platform (http://hlajournalclub.pbworks.com/ ). members choose an article to discuss and the discussion remains open for a month. members can log into the wiki at any time to follow the discussion and/or post comments. at the end of the month the moderator calls for a volunteer to write up the discussion in the form of an evidence summary which is published in the open access hla news: national bulletin of health libraries australia (https://www.alia.org.au/enewsletters/alia-health-libraries-australia-news). membership of the journal club is open to any interested lis professionals.   if you can’t find a journal club that suits you, give some thought to starting one yourself. colleagues from other libraries or institutions may be interested in some continuing professional development. the centre for evidence based library and information practice (c-eblip) runs an in-house journal club at the university library, university of saskatchewan for the librarians there. every six weeks, rotating convenors choose an article and lead the discussion. convenors are responsible for writing up the journal club discussion for brain-work, the c-eblip blog. a list of articles and blog post links can be found here: http://library.usask.ca/ceblip/activities/c-eblip-journal-club.php   like-minded eblip others are definitely active on social media and e-lists. signing up to follow the eblip journal on twitter (https://twitter.com/eblip) is a good place to start. other twitter accounts that would be helpful to get to know folks and to keep you in the loop are @ceblip, @eblip8, or @ciliplirg. useful hashtags to follow include #lark, #eblip, #ebliprg, and #ceblipblog. in addition to following accounts and hashtags, consider becoming more active yourself on twitter. posting links to helpful articles, asking questions, and sharing your experiences can help to create a community around you. communities on facebook support the practicing eblipper, such as lark – library applied research kollective – (https://www.facebook.com/pages/lark-library-applied-research-kollektive/236960273098176?fref=ts) lark is open to librarians and professionals from all types of libraries who are interested in applied research.   once you start making connections through professional associations, e-lists, journal clubs, social media, and other avenues, you may be able to identify an eblip practitioner to approach to be your mentor. this could be someone geographically close to you, which would allow face to face meetings, or it may be someone you only ever meet online, via skype or even by telephone. regardless of the medium, a professional mentor can provide the support that eblip practitioners may feel is missing in their immediate workplace. professional associations are often a good place to start to find a mentor. for example, the medical library association runs a mentoring program including providing mentoring resources (https://www.mlanet.org/mentor/index.html).   so far we have discussed a range of practical strategies that solo eblip practitioners may find useful to put them in touch with the wider eblip community. finding your tribe is essential, but so is developing inner resources of attitude and resilience. much of the discussion about solo eblip practitioners at the eblip8 conference focused on the issue of individuals not being able to break through management or colleagues’ resistance to eblip, and the subsequent frustration they felt. resilience and resourcefulness were identified as critical to dealing with such situations. solo eblip practitioners were encouraged to develop boldness, to take risks, and to nurture resilience to cope with the inevitable knockbacks. diane coutu has identified one of the defining characteristics of resilient people as “ritualized ingenuity” also labeled by french psychologists as “bricolage”. she writes that “bricolage in the modern sense can be defined as a kind of inventiveness, an ability to improvise a solution to a problem without proper or obvious tools or materials” (coutu, 2002, pp.54-55). this is a useful concept for solo eblip practitioners who may find themselves piecing together a support network outside their workplace, creating possibilities for evidence-based practice where none are obvious, or even ‘flying under the radar’ to identify a project where they could apply eblip principles. mental toughness and emotional resilience are also attributes cultivated by resilient people. part of mental toughness is the ability to withstand setbacks and find ways to identify opportunities, learning, or personal growth in negative experiences (mcewen, 2011, p.37). of course this is easier said than done, but resilience is, at least in part, a learned skill.   so whether you are delving into eblip as a solo practitioner or you are working in a larger organization and want to get eblip off the ground, eblip8 panel member denise koufogiannakis offered useful advice: pick one thing that is important to you and work on that. whether it’s a small workplace project, an information-gathering exercise to take to management, or some other piece of work, start small, keep it local and grounded and, of course, evidence based. remember the three pieces of eblip that are crucial in order to work in this way: professional expertise, user preference, and the best available research evidence. set goals and make sure you stick to them; find what eblip8 keynote speaker virginia wilson referred to as an accountability buddy, someone who will help to keep you on time, on track, and on target.   even if you never break through organizational barriers to eblip, no one can stop you from being evidence based in your own professional practice. read widely and think critically. engage in active reflection on your actions and experiences as an information professional. keeping a journal or portfolio in which to record activities and events, not just what you did but how you responded and what you learned, will develop your reflective writing skills. finally, have confidence in yourself. you’ll never know what you can accomplish until you take that first step. and then once you do, the possibilities are endless.   references   booth a. (2011). barriers and facilitators to evidence-based library and information practice: an international perspective. perspectives in international librarianship, 1. http://dx.doi.org/10.5339/pil.2011.1   coutu, d. (2002). how resilience works. harvard business review, 80(5), 46-56.   godin, s. (2008). tribes : we need you to lead us. new york: portfolio.   mcewen, k. (2011). building resilience at work. bowen hills, qld: australian academic press.   news   apply for the 2023 research training institute fellowship program    2022. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30274      are you looking for a powerful way to increase your value to your library and develop your career and reputation? do you have a research idea or question that you want to pursue but do not know where to start?   the research training institute (rti) fellowship is a unique, highly effective, and collaborative online research and support program. rti assessment findings show the rti learning model is sound and effective, and increases the research skills, confidence, and productivity of participants. librarians of all levels of professional experience and types of work environments who provide health information services, collections, and support, and who want to contribute to improving library and health care outcomes are encouraged to apply to the rti fellowship program.   the institute is a one-year online program that consists of a series of online modules in advanced research methods, mentoring by faculty experts and peer coaches, preparing and implementing a research project, and an opportunity to present the research project at the mla ’24 conference & exhibits. rti ‘23 features an expanded research curriculum, schedule, and new support services. learn more about rti fellowship program details.   rti applications will be open until february 28, 2023. applications are now being accepted through february 28, 2023, for the 2023 cohort of rti research fellows. accepted applicants will be notified in april 2023. see the rti website for eligibility requirements and selection criteria. the institute also offers scholarship opportunities for professionals engaging in dei research, working in aahsl libraries, or possessing a financial need. rti ’23 submissions are not limited to u.s. librarians. canadian and international librarians are encouraged and welcomed to apply. discover and nurture your research talents! learn how to conduct and lead quality research projects. meet like-minded research colleagues, make lifelong friends, gain research confidence, form research collaborations, and help end users improve health and wellness! use your research to advocate your value and impact! apply today!   individuals who identify as being from underrepresented communities are encouraged to apply.   if you would like to learn more about the institute, please visit the rti website or contact susan lessick, ahip, fmla, at slessick@uci.edu.     evidence based library and information practice using evidence in practice   engaging with psychology students to find new ways of improving behaviour in libraries   louise dawson customer service assistant jb priestley library university of bradford united kingdom email: l.dawson@bradford.ac.uk   louise phelan academic support librarian for psychology jb priestley library university of bradford united kingdom email: l.phelan@bradford.ac.uk   received: 1 aug. 2023                                                              accepted: 11 oct. 2023      2023 dawson and phelan. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30446     setting   in the period following the pandemic, we observed that poor student behaviour was increasingly becoming a problem in our library. we decided to take a novel approach to try and discover new ways of encouraging positive use of the university library using existing resources and ensuring the inclusion of student voices to gather rich critically evaluated feedback to inform our service improvements.   the university of bradford is a medium-sized university in yorkshire, united kingdom. it is serviced by the jb priestley library, which is located centrally on campus. the library facilities offer a range of study spaces from silent study to group collaborative spaces. the library lacks a distinct separate entrance, as it is not in a separate building.   problem   following the full reopening of the library to normal service after the end of covid-19 pandemic restrictions, staff noticed a significant deterioration in student behaviour compared to pre-pandemic. problems ranged from disruptive and confrontational behaviour to vandalism of the building and contents. this had a significant negative impact on the student experience for legitimate library users and was unsustainable in terms of damage to facilities and on the wellbeing of students and staff. conversations with colleagues in other institutions confirmed that these problems were not bradford-specific.   existing ways of dealing with poor behaviour using signage about specific behaviours and policies (e.g., food and drink, vandalism, and alcohol) had little impact. a student behaviour group formed of library staff was created to consider solutions. one suggestion was taking the problem to psychology students within the university to gain a new perspective on understanding and addressing the roots of these behaviours. this combined student perspectives with a critically evaluated psychological grounding.   we wanted to find new ways of improving the experience of students using the library for legitimate study purposes in ways that could be implemented by staff and without the need for expensive improvements, furniture replacement, or building work. we also wanted to ensure that the student voice and perspective was included in a constructive and novel way.   evidence   we approached an associate professor of social psychology, about the possibility of involving students from the psychology department. the professor suggested that the problem be introduced as a case study as part of an established team-based learning session for a class of 25 master’s degree psychology students. team-based learning is an active learning method in which classroom sessions are devoted to team problem-solving rather than lectures (tweddell, 2020). both authors were then invited to attend the session to introduce the problem and assess the ideas presented.   we provided the following information and evidence to students:   a brief presentation with an overview of our issues in the library a spreadsheet of anonymous data detailing recent behavioural issues, including the date, time, location, number of people involved, and description of instances of poor behaviour, gathered over a month some recent comments from student users of the library from feedback volunteered by students through mechanisms such as meetings between academic librarians and student representatives from the departments they support some recent photographs of library vandalism copies of posters currently displayed in the library information on behaviour displayed on our website: https://www.bradford.ac.uk/library/contact-and-about-us/regulations-and-policies/     we asked the students to address these questions:   how can we encourage better use of our facilities? how can we better provide group working space without it becoming a social area?   the students were given three hours to consider the evidence and questions before presenting their work to the class. the students’ presentations included suggested actions as well as explanations from psychology theories that supported these actions. this allowed students to work on a real-world problem and enabled library staff to engage with students who may not have provided voluntary feedback otherwise. as this was part of students’ class module, their participation was marked and contributed to their qualification.   implementation   we compiled the suggestions from the students with their supporting psychological theories. this information was gathered from our notes in conjunction with the slides from the student presentations that were provided after the session. these were then analysed to find those that were already in place or were not implementable and those that could be taken forward and used to make improvements to library services.   the library behaviour group discussed the findings and helped plan and execute the implementation of student suggestions. all of this is detailed in table 1.   table 1 student suggestions and actions taken student suggestions from presentations supporting psychology theories actions taken by the library staff creating a library identity   ·        social identity theory ·        social learning theory ·        psychological frailty ·        we created a library mural depicting desirable library behaviour at the library entrance that was designed and executed by library staff.  unfortunately, students were not able to take part as initially hoped due to health and safety reasons. ·        we continued work on inclusive library initiatives that cut across subject, study level, and staff and student divides, such as a library book club and promotion of existing initiatives including the book exchange, calm space, and upcoming family space. recruiting student representatives to help with basic library tasks   ·        social learning theory   ·        this had already been identified as a need by the library and recruitment was in progress. two student champions roles helped extend available help beyond the library’s core staffed hours. modelling desired behaviours in specific areas ·        social learning theory ·        we created the library mural at the entrance and created study zone banners that were placed throughout the library. creating distinct and easily recognizable study areas for different purposes ·        social learning theory ·        the library banners include distinct colours, icons, inclusive language, and images modelling expected behaviour in each zone. use of inclusive and positive language in communications ·        social identity theory ·        inclusive, positive language was already in use across all communication mediums. increasing surveillance and security presence ·        positive and negative reinforcement ·        the 4 es ·        the library now has two designated security guards stationed at the information desk and roving throughout the library. each guard has been provided with bodycams. use punitive measures (e.g., loss of privileges for poor behaviour) and use of rewards for positive use of the library ·        positive and negative reinforcement ·        the 4 es ·        the library staff undertook an update existing policies with regards to expected behaviour and potential consequences. these are currently being finalized by senior university staff.  ·        we discussed rewarding good behaviour but decided that there were many problems with implementing this in a fair and meaningful way. reintroducing the anonymous text service used by students to report problem behaviour in the library ·        social learning theory   ·        the anonymous text service had already been replaced with the safezone app (https://safezoneapp.com/) available university wide instead. note: for more information on the psychological theories mentioned in the table above, see the appendix.   feasible changes to library service   of the actions we took in response the master’s students’ suggestions (as detailed in table 1), the two that were most novel and impactful were the library mural and the banners for each study zone.   library mural   the student suggestions covered by implementing the library mural include:   creating a library identity modelling desired behaviours in specific areas   we discussed the library identity with the student behaviour group. we came up with the idea of a mural (see figures 1 and 2) on the glass wall by the library entrance, as this is an area of high footfall. we required only a small budget for art supplies, and library staff undertook the creation of the artwork, led by aicha bahij, a professional artist who is also a staff member. the mural incorporated images of study activities in the library, modelling expected behaviour.   also depicted were iconic stages of the academic year, facilities offered by the library; books written by university staff; books on equality, diversity, and inclusion; and a welcoming skeleton near the entry gates to add a sense of whimsy and reflect that library users can borrow plastic skeletons.   figure 1 library mural with artists (left to right) aicha bahij (lead), louise dawson, emily cowler, and sean temple.   figure 2 library mural.   banners in each study zone   student suggestions covered by implementing the banners include:   ·        modelling desired behaviours in specific areas ·        creating distinct and easily recognizable study areas for different purposes ·        use of inclusive and positive language in communications   to clearly demark the silent, quiet, and group study areas and to make them easily recognizable, tall banners were put prominently in each study zone (see figure 3). each zone had a distinct colour, an image modelling the expected behaviour in that zone, information about the number that could study together, and acceptable noise levels in written and icon format. additional text listed other zones available and the consequences for not abiding by zone expectation. all text was written in positive and plain language.   figure 3 a quiet study zone banner.   unfeasible changes to library services   the use of punitive measures (e.g., loss of privileges for poor behaviour) and use of rewards for positive use of the library were discussed by the behaviour group and found to be unworkable within the library regulations. whilst the students were keen for punishment to be meaningful and act as a deterrent, the library has limited powers. however, updated policies and a new process for dealing with poor behaviour were already in progress. this will give a greater consistency to the management of student behaviour not just in the library but across the university, allowing the university to see and act upon patterns of poor behaviour. rewarding good behaviour was discussed, and whilst positive, the idea was impractical as it could not be implemented in a fair and consistent way.   outcome   behaviour in the library has improved. collaborating with students for suggestions for improvements has brought about tangible implementable ideas that would not have been gained otherwise. the mural in particular has led to greater student engagement as a unique feature on campus and made the entrance of the library a distinctive space. visiting international students have been keen to have it as the backdrop to their photographs with staff. the collaboration had many positive benefits such as strengthening relationships between the library and students as well as the library and an academic department. it brings ideas submitted in a classroom into real world existence, enabling students and staff to see their positive and tangible impact on a university service.   reflection   this project has provided a fascinating new perspective on tackling the perennial challenge of improving student experience and tackling poor behaviour. there are several advantages to this innovative way of gaining student feedback, which could encourage others to try something similar. the costs are few. the requirements were simply library staff time and a member of academic teaching staff willing to timetable a team-based learning session for us to present our problem to. the students produced rich feedback that was critically evaluated and, rather than being subjective or reactionary, was backed up by established psychological theories. it was also a unique opportunity to collaborate with an academic department in the university.   limitations to the project were that we consulted only one set of students in one course, so some student voices are missing. additionally, master’s degree students are often mature or may be international students so their perspectives may differ from other students. overall, however, this is a high impact, low cost, repeatable, and innovative way of gathering student input on improving student experience and managing poor behaviour.   acknowledgements   we would like to thank dr. peter branney without whom this research would not have been possible, aicha bahij for her original art and for being lead artist on the library mural, emily cowler and sean temple for their original art contribution to the library mural, sarah george for being an encouraging and guiding (tor)mentor on our first publication, the library customer services team for their collective contribution to the library mural and alison lahlafi for enabling us to undertake this research and feed into service improvements.   author contributions   louise dawson: conceptualization, investigation, methodology, project administration, resources, writing – original draft, writing – review and editing louise phelan: investigation, methodology, project administration, writing – review and editing   references   aitkenhead, e., clements, j., lumley, j., muir, r., redgrave, h., & skidmore, m. (2022). policing the pandemic. crest advisory. https://www.crestadvisory.com/post/report-policing-the-pandemic   bandura, a. (1977). social learning theory. prentice hall.   skinner, b. f. (1974). about behaviorism. cape.   stufano, a., lucchese, g., stahl, b., grattagliano, i., dassisti, l., lovreglio, p., flöel, a., & iavicoli, i. (2022). impact of covid-19 emergency on the psychological well-being of susceptible individuals. scientific reports, 12(1). https://doi.org/10.1038/s41598-022-15357-6   tajfel, h. (1982). social identity and intergroup relations. cambridge university press.   tweddell, s. (2020). evaluating the introduction of team-based learning in a pharmacy consultation skills module. pharmacy education, 20(1), 151–157. https://doi.org/10.46542/pe.2020.201.151157   appendix   psychology theories   social identity theory. this revolves around the creation of in-groups, characterised common expected behaviour, positive and favourable beliefs about the in-group and negative beliefs and opinions about those not part of the in-group (tajfel, 1982). social learning theory. this posits that people learn how to behave in any given situation or environment, based on observing and then mimicking behaviour in others that appears to be acceptable. this can be good or bad behaviour (bandura, 1977). positive and negative reinforcement. under this theory, behaviour is controlled or changed using the mechanism of offering reward and inflicting punishments (skinner, 1974). the 4 es (engage, explain, encourage, enforce). this is a framework for engaging with people behaving unacceptably and escalating if required. this is often employed by services such as the police (aitkenhead et al., 2022). psychological frailty (post pandemic). this theory posits that people have lost or did not gain key social skills that enable them to engage positively in different environments. this may also include high levels of mental ill-health in people (stufano et al., 2022).     microsoft word comm_dennis.doc evidence based library and information practice 2007, 2:2  116 evidence based library and information practice     commentary    the great debate: be it resolved that evidence based librarianship is a bunch of  hooey      chris dennis  head, collection development  queen elizabeth ii library  memorial university of newfoundland  st. johnʹs, nl  e‐mail: cdennis@mun.ca    danial duda  map librarian  queen elizabeth ii library  memorial university of newfoundland  st. johnʹs, nl   a1b 3y1  e‐mail: dduda@mun.ca    jessie mcgowan  senior information scientist  institute of population health  university of ottawa  ottawa, ontario, canada  e‐mail: jmcgowan@uottawa.ca    louise white  head of electronic resources and serials  queen elizabeth ii library  memorial university of newfoundland  st. johnʹs, nl  e‐mail: louisew@mun.ca      © 2007 dennis, duda, mcgowan and white. this is an open access article distributed under the terms of  the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits  unrestricted use, distribution, and reproduction in any medium, provided the original work is properly  cited.        mailto:cdennis@mun.ca mailto:dduda@mun.ca mailto:jmcgowan@uottawa.ca mailto:louisew@mun.ca http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:2  117 introduction    be it resolved that evidence based librarianship is a  bunch of hooey. this was the subject debated at  the memorial university libraries research fair,  held at the inco centre on the st. john’s  campus of memorial university, from  november 15‐17, 2006. the light‐hearted,  friendly and often tongue‐in‐cheek debate,   based upon the format of the great debate held  at canadian library association conferences,  closed out the three day fair. the affirmative  side was debated by louise white and chris  dennis, both from memorial university. the  negative side was debated by researcher‐in‐ residence jessie mcgowan from the university  of ottawa and danial duda from memorial  university. who won? to borrow from and to  manipulate the bard, “read on macduff” and  decide for yourself…better yet, what does the  evidence tell you?    the affirmative: louise white and chris  dennis    presenting the affirmative of this case requires  an examination of definitions.  the negative  meaning of ʺhooeyʺ can be analysed into three  components: it denotes talk without substance;  it connotes carelessness with reasoning and  inappropriate generalisation; and it suggests  that cover is being provided for ulterior motives  – that a resonant‐sounding aura of legitimacy  and originality is being conferred on a concept  that may well not be either, and which may  (mis)lead to malign consequences.    evidence based librarianship (ebl) seems to be  defined, in the words of a colleagueʹs  unpublished paper, as a ʺfocus on methods for  resolving daily problems in the profession  through the integration of experience and  research, involving asking questions, finding  information to answer them (or conducting  oneʹs own research) and applying that  knowledge to practice.ʺ the question that comes  to mind is, ʺdoes any of this mean anything  other than good well‐founded practice?ʺ if it  doesnʹt, it is a name without anything specific to  which to refer; talk without substance. one asks,  why give good well‐founded practice a new  name? is the concept simply an exhortation to  use research as a basis for making operational  decisions? whatʹs new about that, beyond the  label? is it really the case that librarians never  did this before? one might appeal to the  prescribed six‐step process as a way to  distinguish evidence‐based librarianship from  what went before, but in the same vein, one has  to ask whether that is any different from good  practice either, or calls into question all practice  before 1998, when the notion of ebl first had  currency. the first component of hooey,  denoting talk without substance, seems to apply.    we understand that the concept of ebl purports  to have broad application. depending on the  extent of the application claimed for it, this  seems an instance of the connotation we noted  in defining ʺhooey,ʺ of carelessness in reasoning  and inappropriate generalisation. one might ask  whether, to the extent that it does mean  anything other than good well‐founded practice,  it constitutes a general method to cover  everything we do. the answer should be no,  because there are questions and practical  decisions which in principle are not susceptible  of answer by evidence, or worse, for which  following ʺevidenceʺ (e.g., what everyone else  does or wants), can or will give the wrong  answer. community‐inspired censorship of  materials in public libraries is just one instance  where ʺevidenceʺ can be either beside the point  or actively deleterious to the mission of  librarianship. ours is an ethically founded  profession, not just a social science, and  deciding what we should do goes beyond  simple knowledge of facts. claims for the  completely general application of ebl thus seem  false.  alternatively, we could interpret it as a  suggestion that librarians should do research.  but that is motherhood: of course librarians  should do research. however, should we always  be seeking an ʺeblʺ way to address practical  questions?  the answer is ʺno,ʺ as we have seen;  evidence based library and information practice 2007, 2:2  118 there are questions to which that method is  inappropriate.      to affirm that ebl is hooey is also, as we noted  at the outset, to suggest that the concept  provides cover for ulterior motives, and may  lead to malign consequences. suspicion is  aroused by the fashionability of the term, which  tends to create a social compulsion to follow its  methodology against the ideal of independent  scholarly activity. such compulsion creates a  need to apply it to whatever work one does in  order to confer credibility. and this advances  the agenda of those who for various reasons  might wish to circumvent the application of  librariansʹ judgment to practical problems that  have ethical dimensions or to confine librariansʹ  research to prescribed areas for purposes of  evaluation or control. of course, they are  pleased to call it an ʺobjectiveʺ method,  forgetting that the purpose they intend is not  legitimate in the first place. in short, because it is  so vague and anodyne, the advocacy of ebl can  legitimise whatever one wants to do and deceive  with its imprecision. the danger is that  uncritical application of ebl can lead librarians  to suspend professional judgment in the name  of following evidence.    the negative: jessie mcgowan and danial  duda    hooey huh! is not the idea of evidence based  librarianship, or ebl, simply “good research”?  thus, why use a catch phrase or title like ebl?  advocates of this new and growing field of  research would explain that ebl has a rightful  place because the methods used in ebl help  decision makers at all levels of librarianship. yet,  there are those who think, and/or believe, that  the term ebl is simply a bunch of “hooey.”  they ask why give a process, or processes, that  have been used for decades a new “title”?  it is  interesting that librarians, just like physicians,  have felt “offended” by being asked to embrace  this philosophy. a few years after the  introduction of evidence based medicine in the  literature, sackett et al. replied to all criticisms  and verifying that the historical origins date  back to mid‐19th century paris. this article is  now over 20 years old and evidence based  medicine is now standard in most medical  schools – we need get on with things and catch  up! our question to proponents is, “what is the  alternative?” just make do with what feels good  or see what google has to say?  we think not.      a new term or title can breathe new life into an  established process. ebl is such a case. ebl  comes to the larger library community from one  of its specialized subfields: health librarianship.  health librarians, in turn, borrowed the process  from medical practitioners who asked clinical  questions to solve patient‐related problems. this  is what all fields in academia do, but each  discipline in academia is unique in nature,  especially in collecting evidence and how it is  used. what ebl does is combine the best  available external evidence with professional  librarian judgment to help decide how a  problem or question should be resolved.  good  professional judgment takes into account the  librarian’s work experience and knowledge of  the local factors of the problem so that the  solution can be implemented in a specific setting.   however, without current best library practice,  practice risks becoming outdated.      the ebl process can be explained in five steps.  the first step is to formulate the question which  can be seen as analogous to asking a reference  question but with specific criteria. sometimes,  new evidence does not need to be found, but a  new question needs to be formulated. next, you  need to find the evidence. a literature review  will help you determine if other people or  organizations have already dealt with a similar  situation and their findings or conclusions may  be all you need.  how deep do you need to go to  find the answers you need? once you find the  evidence, it needs to be appraised and evaluated  for its research value. does it meet the needs of  the situation? if the evidence does meet the  needs, then it has to be applied to your situation  taking into account the preferences of your user  audience. finally, the whole process has to be  evidence based library and information practice 2007, 2:2  119 evaluated to understand the impact the evidence  had in answering the question or to see if some  of the earlier steps need to be revisited (i.e.,  maybe your search finds no evidence and you  decide to ask another question). if you don’t  find sufficient evidence to answer your question,  you may also decide to contribute to the  literature and develop your own study.      no one is against the idea of “good research”  but to say a process like ebl is “hooey” simply  because that is what we have always done is  missing the point. what ebl does do is help you  think each step through. if a step is missed,  some decisions could be made that wastes time  in the long run. by following and understanding  these steps, librarians can be assured that they  are using the best practices to find a high quality  solution. this is not “hooey” but a tool that can  help us be better librarians.    works cited    guyatt, g.h.  “evidence‐based medicine.” acp  journal club suppl 114: suppl 2 (1991):    a‐16.    sackett, david l., william m. c. rosenberg, j. a.  muir gray, r. brian haynes, and w. scott  richardson. “evidence based medicine:  what it is and what it isnʹt.”  bmj 312:7023  (1996): 71‐2.      evidence summary   faculty in the applied and pure sciences may have limited experience with e-books   a review of: bierman, j., ortega, l., & rupp-serrano, k. (2010). e-book usage in pure and applied sciences. science & technology libraries, 29(1-2), 69-91. https://doi.org/10.1080/01942620903579393   reviewed by: jennifer kaari  librarian  east orange public library  east orange, new jersey, united states of america  email: jkaari@eopl.org    received: 1 mar. 2021                                                                    accepted:  15 apr. 2021      2021 kaari. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29939     abstract   objective – to determine the usage of and attitudes toward e-books among faculty in the applied and pure sciences.   design – online survey and in-person interviews.   setting – a large public university in the united states.   subjects – 11 faculty members.   methods – participants completed an 11-item survey covering demographic data and questions about electronic book experience and preferences. this was followed up by an in-person interview with the researchers. the interviews were structured into three sections: opening questions about e-book usage, an interactive demonstration and discussion of two preselected e-books, and final follow-up questions. interviews followed a general script of prepared questions, but also encouraged open discussion and dialogue.   main results – most participants in the study reported limited experience with e-books and only 3 of the 11 participants reported using library-purchased e-books in their research and instruction. participants noted ease of access and searchability as key advantages of e-books. concerns included the belief that reading and learning is more difficult on a desktop computer, as well as concerns about the stability and reliability of e-book access. participants also felt negatively about the necessity to create a new login profile and password to access e-books.  the study found no difference in the way faculty in pure and applied sciences approached e-books.   conclusion – the authors determine that e-books will likely become more commonly used in academia. users want e-books that are easy to use and customizable. in addition, the authors conclude that librarians need to understand their patrons’ needs as e-book users and proactively promote and market their e-book collections.   commentary   this paper begins with the question of whether e-books have reached the tipping point. nearly a decade after its original publication, e-books are well past that tipping point, but many of the questions being addressed in this paper remain relevant to academic librarians to this day. as illustrated in more recent reviews and studies on the topic, lack of awareness of e-books, issues about usability, and questions about e-book marketing and promotion in libraries remain open concerns (blummer & kenton, 2018; carroll, 2016).   based on glynn’s critical appraisal tool for library and information science, this study had a 77% validity (2006). the methodology is well-described and the authors include the full survey instrument as well as the questions used for the in-person interviews. the description of the results is robust. however, the study’s validity suffers from the lack of a clearly outlined research question and the conclusions are very general and are not well-linked to the results of the study itself. the authors note that their selection of participants was unscientific. this and the small number of participants represent limitations to the study and make the findings themselves not generalizable to other populations.   this paper highlights the value of utilizing multiple methodologies when tackling complex questions about user behavior and awareness. the authors note that by following up the online survey with in-depth interviews, they were able to discover discrepancies between the findings of the survey and the interviews, notably, that participants who claimed on the survey to have no experience with e-book usage did report experience when interviewed directly. incorporating a demonstration into the interview process also yielded interesting findings regarding users’ perspectives on specific features and functions that might not have been elicited from survey or simple interview questions, particularly given many of the participants' relative lack of experience with e-book interfaces.   despite its limitations, this paper has value for researchers and librarians who are interested in exploring how faculty attitudes toward and usage of e-books have or have not changed over time. in particular, it would be interesting to examine if some of the hesitations faculty held about e-book usage have been addressed by technological advances. the authors also note that while their study did not provide any evidence in differences between the pure and applied sciences in e-book utilization, it would be worth re-examining the possibility for differences between the fields now that e-book usage has become more commonplace.   references   blummer, b., & kenton, j. m. (2018). a systematic review of e-books in academic libraries: access, advantages, and usage. new review of academic librarianship, 26(1), 79-109. https://doi.org/10.1080/13614533.2018.1524390   carroll, a. j., corlett-rivera, k., hackman, t., & zou, j. (2016). e-book perceptions and use in stem and non-stem disciplines: a comparative follow-up study. portal: libraries and the academy, 16(1), 131-162. https://doi.org/10.1353/pla.2016.0002   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154 research article   acceptable and unacceptable uses of academic library search data: an interpretive description of undergraduate student perspectives   laura w. gariepy associate dean for research and learning virginia commonwealth university richmond, virginia, united states of america email: lwgariepy@vcu.edu   received: 14 feb. 2021                                                               accepted: 3 may 2021      2021 gariepy. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29923     abstract   objective – this article presents findings about undergraduate student attitudes regarding search data privacy in academic libraries. although the library literature includes many articles about librarian perceptions on this matter, this paper adds rich, qualitative evidence to the limited research available about student preferences for how libraries should handle information about what they search for, borrow, and download. this paper covers acceptable and unacceptable uses of student search data based on american undergraduate student perspectives. this is an important area of study due to the increasingly data-driven nature of evaluation, accountability, and improvement in higher education, which relies on individual-level student data for learning analytics. these practices are sometimes at odds with libraries’ longstanding commitment to user privacy, which has historically limited the amount of data collected about student use of materials. however, libraries’ use of student search data is increasing.   methods – this qualitative study was approached through interpretive description, a rigorous qualitative framework for answering practical research questions in an applied setting or discipline. i employed the constant comparative method of data collection and analysis to conduct semi-structured interviews with 27 undergraduate students at a large, american, urban public research institution. interviews included questions as well as vignettes: short scenarios designed to elicit response. through inductive coding, i organized the data into interpretive themes and subthemes to describe student attitudes.   results – participants viewed academic library search data as less personally revealing than internet search data. as a result, students were generally comfortable with libraries collecting search data so long as it is used for their benefit. they were comfortable with data being used to improve library collections and services, but were more ambivalent about use of search data for personalized search results and for learning analytics-based assessment. students had mixed feelings about using search data in investigations related to criminal activity or national security. most students expressed a desire for de-identification and user control of data. students who were not comfortable with their search data being collected or used often held their convictions more strongly than those who found the practice acceptable, and their concerns were often related to how data might be used in ways that harm members of vulnerable groups.   conclusion – the results of this study suggested that librarians should further explore student perspectives about search data collection in academic libraries to consider how and if they might adjust their data collection practices to be respectful of student preferences for privacy, while still meeting evaluation and improvement objectives. this study also introduces the qualitative framework of interpretive description to the library and information science literature, promoting use of this applied qualitative approach, which is well-suited to the practical questions often asked in library research studies.     introduction   in order to meet demands for accountability, demonstrate value, and effectively serve users, libraries must embrace assessment and evaluation (oakleaf, 2010, prindle & loos, 2017). data about individual students’ use of library collections and services can enable evidence based assessment techniques. however, librarians’ long-standing emphasis on user privacy has resulted in minimal collection of search data: information about what users search for, borrow, or download (malinconico, 2011; town & matthews, 2012; shuler, 2004). resistance to this type of data collection has limited the types of evaluation that libraries have used in the past. however, some libraries have begun to use student data in learning analytics models that more directly tie library use to measures of student success (jones, briney, et al., 2020; oakleaf, 2010, 2018b). learning analytics can be described as the use of student data to improve student learning, student success, or institutional effectiveness and efficiency (jones, briney, et al., 2020).   although many publications address librarian views on privacy, user perspectives are not well represented in the literature. a few studies examine student attitudes about search data privacy in libraries in the united states and the united kingdom (johns & lawson, 2005; jones et al., 2019; sturges et al., 2003; sutlieff & chelin, 2010), but most are limited in methodology or scope. in addition, the results paint a mixed picture of student perspectives and suggest the need for additional qualitative research to enrich the small body of extant literature on the topic.   literature review   statements from professional organizations have affirmed the importance of privacy in libraries (american library association, 1986, 2008, 2019a, 2019b; international federation of library associations and institutions, 2015; national information standards organization, 2015). historically, american librarians have espoused the belief that users cannot search freely for information if their searches are accessible to others. the majority of librarians consider the monitoring and collection of search data an invasion of library users’ privacy (zimmer, 2014). therefore, many libraries retain as little data as possible about what their users are searching for and reading in order to guarantee unfettered access to information, and to prevent the scrutiny of library users’ search habits by third parties (malinconico, 2011; town & matthews, 2012; shuler, 2004). in addition, the confidentiality of library records is protected by statutes, attorney general opinions, or state constitutions in all fifty states and the district of columbia (american library association, 2018), although law enforcement and other government agencies are able to obtain library records through a course of due process (american library association, 2019b).   however, librarian perspectives on how search data might be used are changing. some support de-identifying and protecting the privacy and confidentiality of student search data instead of deleting it, enabling evaluation approaches aligned with increasingly prominent learning analytics models on university campuses (brown & malenfant, 2015, 2016, 2017; davidson et al., 2013; oakleaf, 2010, 2018a, 2018b; town & matthews, 2012). similarly, standards documents from organizations outside of the american library association support the thoughtful collection, retention, and protection of library user data in order to improve services and collections (national information standards organization, 2015).   although the literature revealed historical and contemporary perspectives from the library profession about search data privacy, few studies addressed user perspectives. johns and lawson (2005) administered a survey to primarily undergraduate students in iowa regarding their awareness and attitudes about universities’ and libraries’ use of “online private information.” few respondents felt it was appropriate for university libraries to use students’ private online data to enhance library services. some indicated that it may be acceptable for libraries to view private online information, but only with informed consent, for a clearly stated purpose, and with the understanding that it would not be disseminated to third parties. sutlieff and chelin (2010) surveyed undergraduate students in the united kingdom (uk) and found that respondents trusted libraries to manage their private search data. nearly 60% were comfortable with the notion of libraries using their borrowing histories to make improvements to the library’s collection – a finding that contrasts johns and lawson’s results. sturges et al. (2003) also conducted a survey that sought uk users’ perspectives related to privacy concerns in libraries, and found that most respondents accepted that libraries should/could monitor use of electronic use of resources for misuse such as unauthorized access to materials, but felt that libraries should not pass along information about their activities in the library to commercial or official entities.   unfortunately, none of these studies provided evidence of methodological rigor or the psychometric properties of the instruments used. without evidence of reliability or validity, or clear definitions of key terms and constructs that the surveys purported to measure, the findings should be interpreted with caution.   in recent years, the data doubles (https://datadoubles.org/) research team published findings about student perspectives on privacy and learning analytics, including an emphasis on data collection in academic libraries (jones et al., 2019; jones, asher, et al., 2020). the authors conducted more than 100 interviews with undergraduate students in the united states, approximately one quarter of which focused specifically on libraries and learning analytics, while the other interviews focused on broader topics related to privacy and learning analytics in higher education.   although most participants were considering data privacy in higher education and libraries for the first time, the interviews still yielded useful data. students were generally accepting about data collection in academic libraries if it benefitted them, and saw potential advantages of using data to improve access to resources and provide personalized search results (jones et al., 2019; jones, asher, et al., 2020). similarly, they felt that learning analytics in higher education could be useful if the focus was on educational purposes and helping students. however, students were unable to detail specific practices that might achieve this purpose, given their limited familiarity with learning analytics.   many students expressed trust in libraries and universities and believed they were well-intentioned. they assumed that their institutions collected data about them, and expected that it would only be used within the institution in ways that would advance student success (jones et al., 2019; jones, asher, et al., 2020). however, some students stated that their relaxed privacy attitudes should not outweigh perspectives of peers who may feel differently, and acknowledged that students in vulnerable groups may have may have greater concerns about data collection. they opposed the idea of universities or libraries sharing any data about them with third parties with the exception of vendors like learning management systems or library databases. overall, students favored de-identifying data or using it in aggregate to protect privacy.   the data doubles (jones et al., 2019; jones, asher, et al., 2020) findings contributed the first in-depth understanding of student attitudes on search data privacy in academic libraries, especially as they pertained to learning analytics. otherwise, the literature pertaining to student perceptions of search data privacy in academic libraries provided few useful or reliable findings.   aims   the purpose of this study was to contribute to and build upon the small body of american and british research focused on user perspectives on search data privacy in academic libraries. specifically, this article presents findings related to two research questions: 1.      what are undergraduate student attitudes about whether academic libraries should collect and maintain user search data, and why? 2.      what are acceptable and unacceptable uses of student library search data according to undergraduate students, and why?   findings presented in this article are derived from a larger dissertation research study (gariepy, 2019), which examined other facets of student perceptions about search data privacy in academic libraries. additional findings will be shared in future publications, including articles about how student perspectives on search data privacy are formed; how students’ library search data privacy attitudes differ from their perspectives about internet search privacy, and an in-depth exploration of how student search data privacy attitudes are shaped by issues related to diversity, bias, and oppression.   methods   interpretive description   the scarcity of well-designed, rigorous research examining student attitudes about search data privacy in academic libraries affirms the need for an in-depth understanding of this issue and calls for a qualitative approach. questions well-suited for qualitative methods are those for which themes, patterns, and understandings have not been well documented or reported (thorne, 2016). this study was conducted using the qualitative approach of interpretive description, a methodology developed in the discipline of nursing by thorne (2016; see also thorne et al., 1997; thorne et al., 2004). interpretive description is a framework for gaining in-depth understanding of a phenomenon and subjective knowledge in clinical or applied disciplines. interpretive description’s practical focus prevents the need for researchers to engage in “methodological acrobatics” (sandelowski, 2000, p. 335), in which qualitative researchers try to fit their studies into established qualitative traditions, such as grounded theory, phenomenology, or ethnography, in an effort to signal rigor. because most of those traditions were born out of disciplines deeply rooted in theory such as anthropology and sociology, they are not a good fit for answering research questions intended to inform practice in applied settings, guiding disciplines toward practical action.   interpretive description provides a rigorous, epistemologically credible framework for research in applied and clinical disciplines that acknowledges the importance of subjective, experiential, and constructed knowledge. this aligns with the assumptions undergirding my motivation for this study: that different students experience the world differently, and that their diverse experiences, attitudes, and perspectives of the realities should be a critical component of how libraries approach the way we think about and handle search data privacy. figure 1 explicates the epistemological underpinnings of interpretive description.   interpretive description is not a discrete method, but rather an overall approach. it encourages the thoughtful utilization of methods from various qualitative traditions to answer specific research questions, which are posed in a way that allows answers to be resituated within the context of the applied field. interpretive description has potential to advance the quality and utility of qualitative research in librarianship, a discipline in which research tends to be highly practical and often informs practice. based on the publications located in the literature search, this article is the first introduction to the use of interpretive description in library research.     interpretive description studies: ·        are conducted in as naturalistic a context as possible in a manner that is respectful of the comfort and ethical rights of all participants, ·        explicitly attend to the value of subjective and experiential knowledge as one of the fundamental sources of applied practice insight, ·        capitalize on human commonalities as well as individual expressions of variance within a shared focus of interest, ·        reflect issues that are not bound by time and context, but attend carefully to the time and context within which human expressions are enacted, ·        acknowledge a social “constructed” element to human experience that cannot be meaningfully separated from its essential nature, ·        recognize that, in the world of human experience, “reality” involves multiple constructed realities that may well at times be contradictory, and ·        acknowledge an inseparable interaction between the knower and the known, such that the inquirer and the “object” of that inquiry influence one another in the production of the research outcomes. figure 1 epistemological underpinnings of interpretive description (thorne, 2016, p. 82).     within the framework of interpretive description, i identified the most effective data collection and analysis techniques to answer my research questions. i conducted in-depth, semi-structured interviews with undergraduate students at virginia commonwealth university (vcu), an urban, public, research university in richmond, virginia, united states of america, with more than 31,000 enrolled students. vcu is known for its racial and ethnic diversity: nearly half of the student body indicates that they are a member of an ethnic/racial minority group. the participants in this study were all currently enrolled undergraduate students at vcu, who had at least some experience using academic research libraries.   recruitment and sampling   before beginning recruitment, i obtained approval for the study from vcu’s institutional review board. the study was subject to expedited review given its low-risk nature. study participants were recruited through emails to faculty and students with whom i had a pre-existing relationship, posts in the vcu daily newsletter, social media posts, and flyers. a $15 amazon gift card incentivized participation. convenience sampling was the initial sampling method for the study (creswell, 2013), and 53 students expressed interest in the study. students were asked to complete a brief screening survey to ensure they had used academic libraries before and to provide demographic information.   i scheduled interviews on a rolling basis between march and may of 2019. because more students expressed interest in the study than i could practically interview, i used information provided in the screening survey to seek demographic diversity in terms of race, ethnicity, gender, major, and rank when selecting participants. i intended to seek participants who mirrored vcu’s rich diversity to the extent it was possible. despite efforts to increase diversity among interview participants, this qualitative study is not intended to be generalized. the goal of including heterogeneous students was to increase the richness of the data and findings.   this sampling approach was consistent with maxwell’s (2013) discussion of convenience sampling as a method of participant selection that can also be purposeful, especially when intended to increase the heterogeneity or richness of the participant pool. in addition, i used elements of purposeful, theoretical, and maximal variation sampling when selecting students to interview from the pool of those who expressed interest in the study (glaser & strauss, 1967; maxwell, 2013; thorne, 2016). glaser and strauss (1967) described theoretical sampling as “the process of data collection… whereby the analyst jointly collects, codes and analyses his data and decides what data to collect next and where to find them, in order to develop his theory as it emerges” (p. 45). an important component of theoretical sampling is maximal variation sampling, in which the researcher seeks participants who, based on the emerging themes and theory of the data, might illuminate a new angle of a particular concept or phenomenon (thorne, 2016).   after 27 interviews, i reached a point at which no new themes were emerging. thorne (2016) challenged the traditional notion of saturation in which a researcher can be confident that s/he has captured all variations in a subjective body of knowledge when one begins to hear the same information from different participants with no variation (sandelowski, 2008). thorne asserted that a lack of new information from study participants does not necessarily mean that all perspectives or manifestations of a phenomenon have been captured, and recommended that researchers acknowledge that other perspectives probably exist that will not or cannot be captured within the practical constraints of most studies. accordingly, i acknowledge that while no new themes were emerging after 27 interviews, i expect future studies to continue to reveal new themes, or delve deeper into specific themes that emerged in this study.   characteristics of the 27 students interviewed included:   ·        more than half of the students interviewed indicated that they were members of racial or ethnic minority groups. ·        most participants were women, but there were several men as well as two transgender/nonbinary students. ·        students from all undergraduate ranks were represented, from first-year students to seniors, but the highest proportion were first-years. ·        many participants were honors students. the high concentration of first-year students and honors students was largely a result of faculty members in the honors college enthusiastically encouraging participation in the study. ·        all participants were between the ages of 18 and 24.   data collection and analysis   data collection and analysis occurred simultaneously using the constant comparative method (glaser & strauss, 1967). thorne (2016) stated that “while straight description could occur in a study that gathers data first and thinks later, interpretive description will inevitably require that the ongoing engagement with data be strategically employed to confirm, test, explore, and expand on the conceptualizations that begin to form as you enter the field” (p. 109). interviews were held in person and audio-recorded, then professionally transcribed.  the average number of minutes per interview was 56. all participants provided informed consent. they were advised that their identities would be kept confidential and that no one except the primary researcher would have access to their interview recordings or transcripts in order to protect their privacy.   a semi-structured interview approach ensured that pertinent questions were asked in each interview, while still allowing flexibility in order to reveal information germane to the study as data collection and analysis progressed (guest et al., 2013; roulston & choi, 2018). the interviews were composed of both questions and vignettes (finch, 1987). the inclusion of vignettes, defined by finch (1987) as “short stories about hypothetical characters in specified circumstances, to whose situation the interviewee is invited to respond” (p. 105), enabled participants to respond to concrete situations in order to elicit more abstract ideas and attitudes (hazel, 1995). a domain-organized interview guide (appendix a) permitted flexibility to ask questions at the most logical time in the interview based on participants’ responses, as opposed to adhering to a strict order (guest et al., 2013).   i developed codes through inductive, emergent coding in atlas.ti (https://atlasti.com/). codes were developed without the aid of a coding schedule to ensure that they authentically reflected the attitudes of study participants. i engaged miles et al.’s (2014) approach of first cycle and second cycle coding to advance a thorough and reflective process. the final coding structure consisted of nearly 100 individual codes, grouped into 19 code families that i used to identify themes related to the research questions (appendix b).   evaluative criteria   to ensure integrity and rigor in the design, collection, and analysis of this study, i employed strategies described by thorne (2016) and lincoln and guba (1985), all of whom provided evaluative criteria for qualitative studies. thorne’s four criteria – epistemological credibility, representative credibility, analytic logic, and interpretive authority – have been developed specifically for the purposes of evaluating interpretive description studies. lincoln and guba developed their criteria – credibility, authenticity, transferability, and dependability – more generally for an array of qualitative studies, and remain prominent in the literature today. the primary strategies for meeting both sets of criteria were: ensuring alignment of the research questions with the purpose of the study, accounting scrupulously for decisions about sampling, data collection, and data analysis through analytic memos, and controlling for researcher bias through reflexive journaling. i also paid careful attention to extreme or negative cases whose perspectives represented significant differences of perspective from other participants, and clarified and confirmed findings during data collection with participants as appropriate.   findings   pseudonyms were assigned to all participants in order to share quotes that support themes. for clarity and readability, these themes are numbered, but the order and numbering does not reflect the significance of a theme in comparison to others.   foundational themes   the data revealed several themes about student awareness and assumptions related to privacy, academic libraries, and related topics. these foundational themes often played a pivotal role in shaping student thoughts about search data privacy in academic libraries and undergird other themes detailed in this article.   theme 1: first-time/evolving thoughts and limited awareness of library practices   although students were very much aware that their internet search habits were being tracked, most had not considered whether their library search data was being monitored. as one student said: “this is the first time that i’ve ever thought about it, if we’re being honest.” because students were considering issues related to privacy and academic libraries for the first time, the decision to use vignettes in the interviews proved to be prudent for eliciting rich responses. in some cases, student perspectives evolved over the course of the interview as they considered the vignettes.   theme 2: academic libraries are mostly used for academic assignments   many students thought of their academic library search data as impersonal because they typically used library resources for academic assignments. they typically did not see research associated with their assignments as reflective of their personal selves, and thought of library search data as “less sensitive” as a result:   …but i mean, libraries aren't getting a full picture of patrons just because our research is so skewed. like i feel like if you were to look up like what i like [at an academic library], i’d be weirdly into like whatever project i have rather than like who i am. (yoofi)   however, some students who were personally passionate about research in more controversial areas were more concerned about the privacy of library search records.   theme 3: acknowledgement of different privacy-related perspectives and experiences   as participants shared their own views on search data privacy in academic libraries, they also assumed that a plurality of viewpoints existed among fellow students. this expressed awareness was most prevalent when a student expressed low levels of concern about privacy themselves but acknowledged that others may have greater concerns. participants particularly noted that search data privacy may be more important for students who are members of vulnerable populations, or who are researching controversial or sensitive topics. some participants who were members of vulnerable or minoritized groups had firsthand experience with bias and described an increased need for privacy, and others acknowledged that data collection and use is often steeped in systemic bias. specific concerns about government access to search data was also raised, especially regarding vulnerable populations.   participant attitudes about library search data collection and privacy   themes presented in this section address student attitudes about search data privacy in academic libraries, as well as students’ nuanced views about acceptable and unacceptable uses of that data from their perspectives.   theme 4: comfort with libraries using search data to benefit students or improve services and collections   participants were largely comfortable with academic libraries collecting search data for purposes that benefitted students. this perspective was rooted in trust in libraries, combined with the fact that students reported they are largely desensitized to search data collection given their experiences on the internet and social media. as one participant put it, the library is “the least of my concerns” when it comes to data tracking.    in fact, a number of participants assumed that libraries were already collecting data about them. some were surprised or perplexed when they learned through vignettes that librarians often decouple search data from specific users, or even dispose of the data altogether. one participant described these practices as “a little bit drastic.” another indicated that “getting rid of it and not making use of it is a waste.” however, many students expressed a preference for deidentifying their library search data, and felt that libraries should be transparent about how they use it. some suggested ways for users to control their own data, such as opt-in or opt-out models. participants expected libraries to make reasonable efforts to create a secure information environment in order to protect student data from unauthorized parties.                although most students felt comfortable with the idea of academic libraries using search data if the intent was to benefit students, this was not universal. some students favored routine data purging – or never collecting it to begin with – in order to protect academic freedom and the ability to search without interference. participants who had the most fervent opinions about maintaining user privacy in libraries often spoke of their experiences as members of minoritized or oppressed groups, or similar experiences of others, which significantly contributed to their perspectives on search data privacy.   theme 5: views on uses of search data for individually tailored search results varies   students held varying attitudes about using library search data for individually tailored search results based on their previous search history. some thought it would be helpful, but some participants were skeptical about how much personalized results would actually increase convenience, particularly for undergraduate students. specifically, participants expressed that because individual undergraduate students’ research assignments vary widely due to general education courses or diversified interests, the type of research they do for one class differs from their needs in the next, which could result in unhelpful tailored search results. some participants also expressed concern that they would enter an “echo chamber” based on a system of tailored search results wherein they would only be exposed to information that aligned with their prior searches.   theme 6: use of library search data for learning analytics initiatives is controversial   students were mostly disapproving of learning analytics models as they related to library use, and found the learning analytics movement in general to be controversial. most participants expressed negative opinions about learning analytics approaches that treated low library use as a sign of potential academic issues, because they did not see failure to use the library as indicative of potential academic risk.   some participants were bothered by the idea of search data being used by academic advisors to flag students who may need extra support. they indicated that engaging in this practice of reporting “anonymous tips,” using library search data as an “academic issue detector,” or acting as the “gpa police” could erode the trust that students have in libraries. this may cause students to view a place they once perceived as helpful as a place engaged in “tattletaling,” instead. additionally, some found the learning analytics model to be generally patronizing, resembling a “helicopter parent”:   i get the intention but i don't feel like academic advisors or librarians should feel obligated to be responsible for the students … college is where you become more of yourself, where you figure yourself out. i feel like doing that kind of stuff to me would make me feel like i'm back in high school. (abeo)   on the other hand, some students felt that students who are coming from high school to college may benefit from the additional support of a learning analytics model in which the university used data to cue special outreach to students if there are signs of academic issues, including low library use.   students were not as negative about employing a research model that looked at data in aggregate as compared to the learning analytics model previously described, which hinged on individual level data and intervention. however, they questioned the notion of correlation versus causation. as one student said: “…i don't know, the relationship between use of library materials and gpa… i just don't think that's enough to… draw any sort of conclusions generally about either students or about the source” (kavya).   theme 7: varied and ambivalent views on search data for preventing bad behaviour   participants were asked to share their thoughts on library search data potentially being used by the government or law enforcement to prevent a variety of “bad behaviours” such as crime and terrorism. opinions varied significantly on the use of library search data in the course of criminal investigations or national security matters. some students felt that if lives could potentially be saved, then privacy should be sacrificed. others felt that privacy should be preserved, even if there is potential to use it to prevent undesirable behaviours and outcomes. both perspectives were sometimes held with strong conviction. some participants saw merit in arguments for and against using data this way, and were ultimately ambivalent about the right balance.   regardless of student perspectives on whether privacy or safety should be prioritized, a common theme emerged: students questioned the relevance of library search data in such investigations. because most participants did not feel that their academic library search data is personal or representative of their true selves, they felt that the information was unlikely to be useful in investigations about crime or terrorism:   i just don't feel like that would be effective at all. i feel like … monitoring google makes more sense or online video chats … that makes sense. but i really don't think there's anything in a library that's really going to help them that much. (clayton)   even if the usefulness of library search data in these circumstances was questionable from student perspectives, some still expressed concerns about how bias and stereotyping could present disproportionate risk to members of vulnerable groups if data was used for this purpose.   discussion   like jones et al. (2019) and jones, asher, et al. (2020), this study revealed that most students expressed trust in academic libraries. most participants indicated that they were comfortable with libraries using search data for certain purposes, and especially those that would benefit students or improve collections and services (jones et al., 2019; jones, asher, et al., 2020; sutlieff & chelin, 2010). however, not all participants felt this way. students who expressed concerns about how library search data might be collected and used often mentioned their own experiences related to bias, oppression, or stereotyping. many of those who were not concerned about their own search data privacy were attentive to the fact that others may be less comfortable or more vulnerable, depending on their race, religion, gender, sexual orientation, or abilities. like jones et al. (2019), jones, asher, et al. (2020), and johns and lawson (2005), this study revealed a want for transparency about how search data is collected and used, and many students supported models in which data is de-identified or anonymized.   in this study, i also presented findings not previously reported in the literature, and provided useful comparison to other studies. through the use of vignettes, i was able to elicit detailed, nuanced data from students. their complex and varied perspectives demonstrated that few types of search data use are entirely acceptable or unacceptable. overall, students were open to the use of search data for improving library services and collections, but had mixed feelings about whether or not tailored results would be beneficial to undergraduate students, given the variety of topic areas they pursued during their studies.   most students held fairly negative views about learning analytics scenarios, a finding somewhat different from that of jones et al. (2019) and jones, asher, et al. (2020), who found students to be conceptually positive about learning analytics. however, jones et al. acknowledged that student participants did not possess enough knowledge about learning analytics to imagine or provide specific examples of how data could be used, which may be partially responsible for the difference in findings. in this study, the use of a vignette about learning analytics in academic libraries provided an opportunity for concrete responses to specific scenarios. the concerns students expressed about learning analytics and libraries revolved mostly around their invasive, overbearing nature, and should be further researched and considered carefully as libraries increasingly embrace these approaches (oakleaf, 2010, 2018b).   finally, this is the first study that offered in-depth understanding of how students think about third-party access to academic library search data, including potential acquisition of search data by the government. this study revealed complex and nuanced views about the government’s right to use search data to protect public safety. although opinions varied about the extent to which government should have access to search data in academic libraries and under what circumstances, many participants felt that such data would not be useful, which reduced their conviction in the opinions they held about it. this sense of apathy was furthered because they viewed library data as neither reflective of their whole selves, nor likely to be of help in an investigation or screening for behaviours that could affect public safety. although there were exceptions, this contrasted significantly with many of the reasons that librarians emphasize the importance of deleting user search data (estabrook, 1996; harper & oltmann, 2017; zimmer, 2013), which is to protect users from third-party access to data, often referring to government entities.   like most qualitative studies, these findings are not intended to be generalized beyond the population of students in the sample, but can serve as a useful springboard for future research. areas of particular importance include more perspectives from members of minority groups and other vulnerable or underrepresented populations. in addition, the perspectives of other user groups beyond undergraduate students, such as graduate students and faculty, are likely different and important to understand. for example, undergraduates’ perspectives that their library search data is not representative of their true selves may be significantly different than a faculty member whose sustained research is focused on difficult social problems or controversial topics that are also personally important to them.   the use of the interpretive description framework, along with vignettes, was well-suited to understanding respondents’ complex views on privacy, and has potential for effectively exploring the perspectives of other groups, as well. finally, the findings of this study could play a role in developing a quantitative instrument to capture more generalizable findings about search data privacy perspectives, the findings of which could be used to inform libraries’ practices related to data privacy and assessment.   conclusion   this study makes an important contribution to a small body of literature about user perspectives on search data privacy in academic libraries. the findings added to the rigorous scholarship of the data doubles team (jones et al., 2019; jones, asher, et al., 2020), both by deepening the library profession’s nuanced knowledge about student perspectives through qualitative research, and by focusing specifically on data privacy matters as they pertain to academic libraries as opposed to higher education more broadly. it also introduces a new research methodology – interpretive description – to library and information science practitioner-researchers.   findings of this research suggested that while some students are comfortable with library search data collection and use, they are also concerned about equity, fairness, and bias. the fact that some members of underrepresented or marginalized groups from the participant pool felt threatened by the notion of their data being collected should compel librarians to reflect on ways to protect the privacy of those who may be most adversely affected if data is misused. this is especially important as the profession continues to consider new forms of data collection and assessment that rely on individual-level student data.   references   american library association. (1986). policy on confidentiality of library records. http://www.ala.org/advocacy/intfreedom/statementspols/otherpolicies/policyconfidentiality   american library association. (2008). code of ethics of the american library association. http://www.ala.org/tools/ethics   american library association. (2018). state privacy laws regarding library records. http://www.ala.org/advocacy/privacy/statelaws   american library association. (2019a). library bill of rights. http://www.ala.org/advocacy/intfreedom/librarybill     american library association. (2019b). privacy: an interpretation of the library bill of rights. http://www.ala.org/advocacy/intfreedom/librarybill/interpretations/privacy    brown, k., & malenfant, k. (2015). academic library contributions to student success: documented practices from the field. http://www.ala.org/acrl/sites/ala.org.acrl/files/content/issues/value/contributions_report.pdf   brown, k., & malenfant, k. (2016). documented library contributions to student learning and success: building evidence with team-based assessment in action campus projects. http://www.ala.org/acrl/sites/ala.org.acrl/files/content/issues/value/contributions_y2.pdf brown, k., & malenfant, k. (2017). academic library impact on student learning and success: findings from assessment in action team projects. http://www.ala.org/acrl/sites/ala.org.acrl/files/content/issues/value/findings_y3.pdf   creswell, j. w. (2013). qualitative inquiry and research design: choosing among five approaches (3rd ed.). sage. davidson, k. s., rollins, s. h., & cherry, e. (2013). demonstrating our value: tying use of electronic resources to academic success. serials librarian, 65(1), 74–79. https://doi.org/10.1080/0361526x.2013.800630   estabrook, l. s. (1996). sacred trust or competitive opportunity: using patron records. library journal, 121(2), 48–49.   finch, j. (1987). the vignette technique in survey research. sociology, 21(1), 105–114. https://doi.org/10.1177/0038038587021001008     gariepy, l. w. (2019). undergraduate students’ attitudes about the collection, use, and privacy of search data in academic libraries: an interpretive description [doctoral dissertation, virginia commonwealth university]. vcu scholars compass. https://doi.org/10.25772/vhtt-ph27    glaser, b. g., & strauss, a. l. (1967). the discovery of grounded theory: strategies for qualitative research. aldine. guest, g., namey, e. e., & mitchell, m. l. (2013). collecting qualitative data: a field manual for applied research. sage. harper, l. m., & oltmann, s. m. (2017). big data’s impact on privacy for librarians and information professionals. bulletin of the association for information science & technology, 43(4), 19–23. https://doi.org/10.1002/bul2.2017.1720430406   hazel, n. (1995). elicitation techniques with young people. social research update, 12. http://sru.soc.surrey.ac.uk/sru12.html international federation of library associations and institutions. (2017). ifla statement on privacy in library environments. https://www.ifla.org/publications/node/10056 johns, s., & lawson, k. g. (2005). university undergraduate students and library-related privacy issues. library & information science research, 27(4), 485–495. https://doi.org/10.1016/j.lisr.2005.08.006 jones, k. m., asher, a., goben, a., perry, m. r., salo, d., briney, k. a., & robertshaw, m. b. (2020). “we're being tracked at all times”: student perspectives of their privacy in relation to learning analytics in higher education. journal of the association for information science and technology, 71(9), 1044-1059. https://doi.org/10.1002/asi.24358 jones, k. m., briney, k. a., goben, a., salo, d., asher, a., & perry, m. r. (2020). a comprehensive primer to library learning analytics practices, initiatives, and privacy issues. college & research libraries, 81(3), 570-591. https://doi.org/10.5860/crl.81.3.570 jones, k. m., perry, m. r., goben, a., asher, a., briney, k. a., robertshaw, m. b., & salo, d. (2019). in their own words: student perspectives on library participation in learning analytics initiatives. proceedings of the annual meeting of the association of college and research libraries, usa, 262–274. http://datadoubles.org/wp-content/uploads/2019/04/acrl_2019.pdf lincoln, y. s. & guba, e. g. (1985). naturalistic inquiry. sage. malinconico, s. m. (2011). librarians and user privacy in the digital age. scires-it, 1(1), 159–172. http://dx.doi.org/10.2423/i22394303v1n1p159 maxwell, j. a. (2013). qualitative research design. sage. miles, m. b., huberman, a. m., & saldaña, j. (2014). qualitative data analysis: a methods sourcebook (3rd ed.). sage.   national information standards organization. (2015). niso consensus principles on user’s digital privacy in library, publisher, and software-provider systems (niso privacy principles).  https://groups.niso.org/apps/group_public/download.php/16064/niso%20privacy%20principles.pdf     oakleaf, m. (2010). the value of academic libraries: a comprehensive research review and report. association of college & research libraries. http://www.ala.org/acrl/sites/ala.org.acrl/files/content/issues/value/val_report.pdf   oakleaf, m. (2018a). library integration in institutional learning analytics. https://er.educause.edu/-/media/files/library/2018/11/liila.pdf?la=en&hash=4253e6937df10c850196a30066ff38e6c3b4f32d   oakleaf, m. (2018b). the problems and promise of learning analytics for increasing and demonstrating library value and impact. information and learning science, 119(1/2), 16-24. https://doi.org/10.1108/ils-08-2017-0080   prindle, s., & loos, a. (2017). information ethics and academic libraries: data privacy in the era of big data. journal of information ethics, 26(2), 22–33.   roulston, k., & choi, m. (2018). qualitative interviews. in u. flick (ed.), the sage handbook of qualitative data collection (pp. 233–249). sage.   sandelowski, m. (2000). focus on research methods: whatever happened to qualitative description? research in nursing and health, 23(4), 334–340. https://doi.org/10.1002/1098-240x(200008)23:4%3c334::aid-nur9%3e3.0.co;2-g   sandelowski, m. (2008). theoretical saturation. in l. m. given (ed.), the sage encyclopedia of qualitative research methods. sage. https://doi.org/10.4135/9781412963909.n456   shuler, j. a. (2004). privacy and academic libraries: widening the frame of discussion. the journal of academic librarianship, 30(2), 157–159. https://doi.org/10.1016/j.acalib.2004.01.008     sturges, p., davies, e., dearnley, j., iliffe, u., oppenheim, c., & hardy, r. (2003). user privacy in the digital library environment: an investigation of policies and preparedness. library management, 24(1/2), 44–50. https://doi.org/10.1108/01435120310454502   sutlieff, l., & chelin, j. (2010). 'an absolute prerequisite': the importance of user privacy and trust in maintaining academic freedom at the library. journal of librarianship & information science, 42(3), 163–177. https://doi.org/10.1177/0961000610368916   thorne, s. (2016). interpretive description: qualitative research for applied practice (2nd ed.). routledge.   thorne, s., reimer kirkham, s., & macdonald-emes, j. (1997). focus on qualitative methods. interpretive description: a noncategorical qualitative alternative for developing nursing knowledge. research in nursing and health, 20(2), 169–177. https://doi.org/10.1002/(sici)1098-240x(199704)20:2%3c169::aid-nur9%3e3.0.co;2-i   thorne, s., reimer kirkham, s., & o’flynn-magee, k.  (2004). the analytic challenge in interpretive description. international journal of qualitative methods, 3(2), 1–11. https://doi.org/10.1177%2f160940690400300101   town, s. j., & matthews, j. r. (2012). assessing library contributions to university outcomes: the need for individual student level data. library management, 33(6/7), 389–402. https://doi.org/10.1108/01435121211266203   zimmer, m. (2013). assessing the treatment of patron privacy in library 2.0 literature. information technology & libraries, 32(2), 29–41. https://doi.org/10.6017/ital.v32i2.3420   zimmer, m. (2014). librarians' attitudes regarding information and internet privacy. library quarterly, 84(2), 123–151. https://doi.org/10.1086/675329      appendix a interview guide since semi-structured interviews are intended to be flexible and evolving, the questions below are tentative. they exemplify the nature of questions that will be asked of study participants, but the questions themselves may change and evolve over the course of participant interviews. although questions are loosely ordered by domain, both the interviewer and the participants will be free to be responsive to the discussions the interview facilitates, and questions may be asked in a different order. throughout the interview, probing questions will be used as appropriate in which participants are invited to further explain their answers. frequently used follow-up questions will include: ·        could you tell me more about that? ·        why do you think you feel/think that way? introduction ·        introductions; small talk to establish rapport. ·        researcher seeks permission to record the interview. ·        “this study is about understanding students’ perceptions about privacy when it comes to searching for data and checking things out in academic libraries. you’ll hear me refer to that throughout the interview as “search data privacy” – the things you search for, download, or borrow from academic libraries. although the focus is on searching for information in an academic library environment, i might also ask some questions about your attitudes on searching for information in other environments, like on the internet, in order to contextualize the conversation.” ·        “there are no right or wrong answers to any of the questions – your perspective is what i’m interested in!  and there’s no such thing as talking too much – i’m interested to hear what you have to say.” ·        “i’m interested in this research because i think it will be helpful for libraries to understand student perspectives on this issue when developing policies on search data privacy, and to help us use data to improve our services appropriately.” ·        “throughout the interview, i will make reference to ‘using academic libraries’ and being ‘in academic libraries.’ however, academic libraries are not limited to physical locations, so experiences you have related to searching academic libraries’ websites, for example, are equally relevant.” ·        “i’ll also ask you to share some information about yourself with me, such as where you and your parents or family grew up. i’m interested in this because there’s some indication that people’s nationality or cultural background might help shape their views on privacy, and i’d like to better understand that.” ·        offer a brief overview of privacy and libraries, acknowledging that many students haven’t had a chance to think about this.   questions about the participant ·        what year are you at vcu?  ·        what’s your major? ·        where did you grow up? tell me a little bit about the place you lived. o   diversity o   political climate o   overall experience ·        where did your parents/family grow up? o   what brought you to [where they grew up]? o   did you visit there often?   domain 1: experiences with searching for information ·        tell me a little bit about your experiences using academic libraries. how have you used them? o   what kinds of information are you looking for when you search academic library resources? o   describe academic and/or personal uses of academic libraries ·        how do your experiences searching at an academic library differ from your experiences searching elsewhere, like on the internet? o   do you search for different types of information?   domain 2: perceptions of and expectations for privacy when searching for information ·        have you ever thought about whether your search habits were being monitored either in an academic library or in another search environment like the internet? if so, please describe how that made you feel. o   if you assume that your search habits are being monitored, does it affect the way you search? in what ways? o   do you use any other strategies to further protect privacy of your search activities? ·        who do you feel should or should not have access to data about what you search for, both on the internet and in academic libraries? ·        scenario a: for this question, i’m going to present a scenario, and then i would like you to share your reaction with me about how it makes you feel about privacy in that particular context. “an academic library wishes to improve its search features. to do so, they decide to collect and maintain data about what individuals search for, so that when that person logs into the library system, their results will be tailored based on their previous searches. an undergraduate student who uses the library regularly notices that when she searches for books and articles on the library website, that some of the results seem related to things she’s downloaded in the past.” o   how do you feel about this scenario? o   can you think of benefits or risks of this scenario? o   have you had any experiences that affect the way you think about this scenario? o   if you were to consider privacy and convenience on a spectrum of importance, with each at oppose ends, please talk about where you would fall on the spectrum. do you value privacy, convenience, or both? ·        scenario b: for this question, i’m going to present a scenario, and then i would like you to share your reaction with me about how it makes you feel about privacy in that particular context. “an academic library wishes to use data about what students search for, check out, and borrow to assess use of the collection and ways we might improve it. the library maintains a record of each student’s search data so that librarians can do data analysis by individual and group (for example, biology majors) about library use. this allows the library to make adjustments to the collection and to the services offered like teaching and outreach to serve students as effectively as possible.” o   how do you feel about this scenario? o   can you think of benefits or risks of this scenario? o   have you had any experiences that affect the way you think about this scenario? o   how would you feel if your search data were de-identified from your name and other identifying information? ·        scenario c: an academic library maintains a record of each student’s search data. the library uses the data to explore the relationship between use of library materials and academic success (like gpa and grades). when students have not used the library at all but are enrolled in courses that usually necessitate library use, librarians notify those students’ academic advisors as an early warning that the student could have academic issues. o   how do you feel about this scenario? o   can you think of benefits or risks of this scenario? o   have you had any experiences that affect the way you think about this scenario? ·        please describe feelings of trust or distrust you have for academic libraries, if any, and why you feel that way. ·        does the level of trust you have for libraries differ from the degree to which you trust google or other internet search engines? why? ·        scenario d: for this question, i’m going to present a scenario, and then i would like you to share your reaction with me about how it makes you feel about privacy in that particular context.  “an academic library elects to routinely purge any data about what library users search for, and what they check out, as soon as items are returned. the decision to do so was made because many librarians believe that people can only search freely for information if there is no possibility of someone else (be it the library or a third party) having access to what they search for. in routinely purging records, libraries forego data that could be useful in helping them design search tools and purchase collections that would serve library users’ needs.” o   how do you feel about this scenario? o   can you think of benefits or risks of this scenario? o   have you had any experiences that affect the way you think about this scenario? o   what do you think the right balance is between libraries collecting data about students’ search habits in order to improve services and protecting user privacy? domain 4: concerns about access to search data/borrowing histories from third parties ·        scenario e: for this question, i’m going to present a scenario, and then i would like you to share your reaction with me about how it makes you feel about privacy in that particular context.  “google maintains data about what people search for in order to better understand user search habits in order to improve the search experience and provide targeted advertisements. in an effort to prevent terrorism, the federal government begins routinely monitoring google search data to look for suspicious searching behavior.” o   how do you feel about this scenario? o   can you think of benefits or risks of this scenario? §  are there particular circumstances you can imagine in which it would be appropriate for third parties to access data about what people have searched for? o   have you had any experiences that affect the way you think about this scenario? o   would your perspective be different about this scenario if we replaced google search data with library search data/records?   closing questions ·        we’ve talked about a lot of things today. can you offer me a quick summary of your views on privacy of search data in academic libraries as they are right now? ·        do you think any of your life experiences or influences to date have shaped your views about how your search data should be handled when searching online or at the library? o   ask for expansion of previously mentioned influences o   are you a social media? do you feel that your use/non-use of social media has affected your views on privacy in general? ·        is there anything else you would like to share with me that you think would be important to this study?     appendix b codes organized by code families/pattern codes code family/pattern code individual codes academic and intellectual freedom academic/intellectual freedom and privacy: ambivalence/context/nuance academic/intellectual freedom and privacy: important academic/intellectual freedom and privacy: unconcerned data collection for safety/public good: limits intellectual/academic freedom internet: wary of filter bubbles libraries search data for safety/public good: limits intellectual/academic freedom monitoring changes behavior monitoring changes thought monitoring doesn't change behavior privacy more important for sensitive/controversial topics   academic library use academic library use blend of academic and personal use academic library use blend of academic, professional, and personal use academic library use focused on academic work context/nuance/ambivalence academic/intellectual freedom and privacy: ambivalence/context/nuance data collection for safety/public good: ambivalence/context/nuance data collection for safety/public good: context/nuance/ambivalence first time/evolving thoughts internet data collection: ambivalence/context/nuance internet tailoring: ambivalence/context/nuance learning analytics: ambivalence/context/nuance libraries search data for improvement: ambivalence/context/nuance libraries search data for safety/public good: ambivalence/context/nuance libraries tailoring: ambivalence/context/nuance library data collection: ambivalence/context/nuance privacy/convenience: ambivalence/context/nuance rationale behind searching behavior: ambivalence/context/nuance anonymization/de-identification anonymization is imperfect libraries: anonymization necessary libraries: anonymization not necessary awareness/assumptions acknowledges other perspectives assumes monitoring: general assumes monitoring: institutions/units/libs collect data aware of privacy issues/surveillance first time/evolving thoughts challenges with quantitative data academic variables more important than demographics alternate methods for learning about users anonymization is imperfect data collection can lead to bias/bad assumptions gpa correlation studies imperfect data library data collection: oversimplifies/disadvantages some groups/perspectives not counting findings for small cohorts data collection to prevent behavior data collection for safety/public good: ambivalence/context/nuance data collection for safety/public good: context/nuance/ambivalence data collection for safety/public good: limits intellectual/academic freedom data collection for safety/public good: negative feelings data collection for safety/public good: positive/okay growing up in 9/11 era libraries search data for safety/public good: acceptable/positive libraries search data for safety/public good: ambivalence/context/nuance libraries search data for safety/public good: limits intellectual/academic freedom libraries search data for safety/public good: negative fairness, bias, vulnerable populations data collection can lead to bias/bad assumptions library data collection: oversimplifies/disadvantages some groups/perspectives privacy and activism privacy more important for sensitive/controversial topics privacy more important to vulnerable populations general preferences/attitudes for library privacy controlling data/privacy intent/purpose/use is important library data collection: acceptable/positive library data collection: ambivalence/context/nuance library data collection: negative library data collection: oversimplifies/disadvantages some groups/perspectives library data collection: should benefit students nothing to hide relationship/use of entity changes expectations/behavior transparency uncomfortable checking things out in person general preferences/attitudes for privacy controlling data/privacy intent/purpose/use is important internet data collection: acceptable/positive internet data collection: ambivalence/context/nuance internet data collection: cynical/resigned internet data collection: negative internet data sharing/integration: acceptable internet data sharing/integration: negative nothing to hide privacy expectations have changed relationship/use of entity changes expectations/behavior transparency impact on behavior coping mechanisms monitoring changes behavior monitoring changes thought monitoring doesn't change behavior rationale behind searching behavior: ambivalence/context/nuance relationship/use of entity changes expectations/behavior influences accustomed to being tracked, monitored accustomed to privacy anxiety/paranoia assumes monitoring: institutions/units/libs collect data aware of privacy issues/surveillance close or invasive community/culture meant minimal privacy disabled/chronically ill family emphasized/discussed privacy and related issues growing up in 9/11 era immigrant family/participant negative privacy-related experience no negative privacy-related experiences nothing to hide political inclination privacy more important to vulnerable populations relationship/use of entity changes expectations/behavior religion/ethnicity sham use of social media and internet affects privacy perspectives learning analytics gpa correlation studies learning analytics: ambivalence/context/nuance learning analytics: negative learning analytics: neutral/positive privacy-convenience continuum privacy/convenience: ambivalence/context/nuance privacy/convenience: balance privacy/convenience: emphasis on convenience privacy/convenience: emphasis on privacy resignation/cynicism/acceptance accustomed to being tracked, monitored internet data collection: cynical/resigned tolerance for privacy invasions increased search data for library improvement libraries search data for improvement: acceptable/positive libraries search data for improvement: ambivalence/context/nuance libraries search data for improvement: negative tailoring controlling data/privacy internet tailoring: ambivalence/context/nuance internet tailoring: fine/good internet tailoring: negative internet: wary of filter bubbles libraries tailoring: acceptable/positive libraries tailoring: ambivalence/context/nuance libraries tailoring: control options libraries tailoring: negative libraries: wary of filter bubbles third party access/data sharing accustomed to being tracked, monitored data collection for safety/public good: ambivalence/context/nuance data collection for safety/public good: context/nuance/ambivalence data collection for safety/public good: limits intellectual/academic freedom data collection for safety/public good: negative feeling data collection for safety/public good: positive/okay distrust for government growing up in 9/11 era internet data sharing/integration: acceptable internet data sharing/integration: negative libraries search data for safety/public good: acceptable/positive libraries search data for safety/public good: ambivalence/context/nuance libraries search data for safety/public good: limits intellectual/academic freedom libraries search data for safety/public good: negative libraries: data access, sharing, third parties universities: data access, sharing, third parties trust distrust for google, internet, etc. distrust for government neutral about trust in libraries people and fines affect trust in libraries trust for google, internet, et al trust for institution trust libraries more than google, etc. trust/good feelings for libraries     evidence based library and information practice evidence based library and information practice 2012, 7.1 82 evidence based library and information practice article library instruction and academic success: a mixed-methods assessment of a library instruction program melissa bowles-terry assistant librarian university of wyoming libraries laramie, wyoming, united states of america email: mbowlest@uwyo.edu received: 15 dec. 2011 accepted: 29 jan. 2012 2012 bowles-terry. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objectives – this study examines the connection between student academic success and information literacy instruction. locally, it allowed librarians to ascertain the institution’s saturation rate for information literacy instruction and identify academic programs not utilizing library instruction services. in a broader application, it provides an argument for a tiered program of information literacy instruction and offers student perspectives on improving a library instruction program. methods – focus groups with 15 graduating seniors, all of whom had attended at least one library instruction session, discussed student experiences and preferences regarding library instruction. an analysis of 4,489 academic transcripts of graduating seniors identified differences in grade point average (gpa) between students with different levels of library instruction. results – students value library instruction for orientation purposes as beginning students, and specialized, discipline-specific library instruction in upper-level courses. there is a statistically significant difference in gpa between graduating seniors who had library instruction in upper-level courses (defined in this study as post-freshmanlevel) and those who did not. conclusions – library instruction seems to make the most difference to student success when it is repeated at different levels in the university curriculum, especially when it is offered in upper-level courses. instruction librarians should differentiate mailto:mbowlest@uwyo.edu� evidence based library and information practice 2012, 7.1 83 between lower-division and upper-division learning objectives for students in order to create a more cohesive and non-repetitive information literacy curriculum. introduction libraries are often called the “heart of the university,” and have long assumed a vital role in academic life. in recent years, however, libraries (along with many other university departments and programs) have increasingly been asked to prove their value to governing and funding bodies. the research report from the association of college and research libraries, value of academic libraries (2010), provides a research agenda for academic librarians who seek to demonstrate the value of library services. one part of the research agenda is to demonstrate the value that library services add to a university in the form of student learning and academic success. one way that librarians hope to affect student learning is by meeting students in the classroom. over the past decade, information literacy instruction has become a major part of the work of some academic librarians. information literacy is part of the general education program at the university of wyoming (united states): each student is required to take a course with an embedded information literacy component, and most of those classes are freshman-level classes that introduce students to study within their majors. since 2001 the library’s research and instruction services department has collected statistics regarding how many instruction sessions librarians have conducted. the number has risen from 127 in 2001 to 380 in 2010, and we currently teach about 7,500 students per year (or half of the student body). in the past three years, around 50% of those instruction sessions have been aimed at freshman-level classes. this represents a huge time commitment on the part of librarians, and it is important to consider whether or not the investment of time and resources makes a difference to students. this study attempts to see where librarians may have the most impact with face-to-face instruction, as well as to find out how the library instruction program is experienced from a student point of view. the general education program is undergoing revision, and we would like to have an argument for embedding information literacy learning outcomes at different points in the curriculum, rather than embedding them all in freshmanlevel courses. this study aims to establish the value of library instruction at various levels with a scaffolded approach. creating a plan for the incremental mastery of information literacy skills throughout the college curriculum is becoming a more prevalent concern in the library instruction community. a tiered approach to teaching information literacy is in line with the way many universities teach other literacies, such as writing and math, with introductory skills at the freshman level and then more advanced practice as students matriculate. in the 2011 instruction and assessment plan developed at university of wyoming libraries, we included a skills level table that suggests learning outcomes to be addressed at different points in the university curriculum (appendix a). this study deals with the experiences of students who graduated before the new instruction and assessment plan was in place, and so there were no common guidelines for librarians to teach information literacy with increasing complexity; however, in the future with this common set of practices, we may know with more certainty which skills students have learned and at what level of study. literature review much of the research literature related to assessment of library instruction is summarized in library assessment in higher education by joseph r. matthews (2007), in which multiple studies are categorized as either supportive of the idea that library instruction has a positive effect on student performance or non-supportive of that idea. results are fairly evenly split between studies that find a positive association and studies that evidence based library and information practice 2012, 7.1 84 find no association. these studies take the form of skills testing, academic performance, opinion surveys, and more. matthews’s review of the literature reflects the three major ways that librarians have established connections between library instruction and student academic success: surveys of student opinion or habits, examining student work or exams for specific skills, and analysis of grade point average (or another measure of academic performance) in relation to library instruction offered. all three of these methods have limitations: surveys provide an indirect and self-reported assessment of student success, examining student work places a limit on the sample size and is labour intensive for researchers, and while comparing grade point average with library instruction may suggest correlations between student success and library instruction, there are too many potentially confounding variables to claim that library instruction causes student achievement. student achievement is defined in value of academic libraries (2010) as one of several dimensions of student learning. it is often represented by gpa or scores on tests like the gre. a student’s gpa is an imperfect measure of learning and achievement, since grades cannot be directly mapped to learning outcomes like information literacy. there are factors besides learning or skill mastery that may be measured in a grade, such as attendance or participation. despite these confounding variables, gpa remains a widely accepted surrogate for student learning. this literature review will examine studies of the correlation between library instruction and grade point average as well as several studies that use focus groups to assess library instruction programs, because those are the two research methods used in this study. it will also discuss studies employing a mixedmethods approach to assess library instruction, which is an emerging area of research without many articles published to date. finally, one of the major research questions of this study is whether a tiered approach in a library instruction program is effective. while there is literature regarding tiered learning in information literacy within a class or within a major, there seem to be no such articles regarding a cross-curricular library instruction program, so although that topic is included in this study, it is not addressed in the literature review. moore, brewster, dooroh, and moreau (2002) at glendale community college in california published results from a project begun in 1999 that studied the impact of library classes and workshops on student success, with student success defined as gpa score in the following semester. their study showed a positive correlation between library classes and grade point averages, as compared with a group of students who did not receive library instruction. the study sample size, however, was quite small. still, the study is an early example of a positive correlation between library instruction and student achievement. kirk, vance, and gardner of middle tennessee state university (2010) collected data from their institution’s student database, including gpa, gender, act score, and retention, and matched that data to students who were enrolled in classes that received library instruction. the researchers hoped to demonstrate a relationship between library instruction and retention, but they found no measurable effect on freshman to sophomore retention, nor did they find an effect on gpa. there were important outcomes of the study, however. according to the authors, the study provided librarians with encouragement to seek access to student data for research purposes, which can help librarians not only to prove value, but to learn about saturation rates for library instruction and find out about gaps in instruction programs. the largest study of gpa and library instruction took place at hong kong baptist university library, and analyzed the library workshop attendance and graduation gpa of over 8,000 students (wong & cmor, 2011). the study found that if several workshops were offered to students, there was a higher tendency for library instruction to have a positive impact on grade point average. the evidence based library and information practice 2012, 7.1 85 authors suggested that multiple library workshops (as many as three or four) do have a positive correlation with greater academic success. the study marked a difference between undergraduate students who attend library workshops and graduate students who attend library workshops, but with no consideration of whether the undergraduates had library instruction in lower-division or upper-division classes. focus groups have been successfully used to assess various aspects of library services, including library instruction programs. academic librarians have utilized focus groups in order to learn about students’ perceptions of the role of the library and developing information literacy skills (morrison, 1997), to evaluate library services related to a problembased learning curriculum in a school of medicine (canning, edwards, & meadows, 1995), and to evaluate an information literacy program for a freshman-level biology course (spackman, 2007). the first two focus group studies were composed only of students, while the final study included teaching assistants as well as students. the benefits of using a focus group include the opportunity to get multiple perspectives at once (as opposed to a one-onone interview) as well as the spontaneous interactions between focus group participants, which can provide interesting avenues for conversation and for learning (morrison, 1997). mixed-methods research is a distinctive methodology that combines quantitative and qualitative research methods. combining research methods can offer a better understanding of the research problem as each approach adds an angle for analysis (ary, jacobs, & sorenson, 2010). some published library research projects have made use of this method in order to assess information literacy. one such study investigated the relationship between critical thinking and library anxiety among undergraduates in their information search process (kwon, 2008). kwon’s study used a survey instrument as the quantitative method and an examination of student essays as the qualitative method. by combining the two methods, the researcher found significant negative associations between critical thinking and library anxiety. another study utilizing mixed methods assessed a first-year information literacy course via preand posttests and focus group sessions (wakimoto, 2010). these examples of mixed-methods studies illustrate the value of combining qualitative and quantitative methods in order to understand complicated, multi-faceted issues like student learning. the mixed methods study described in the current article replicates elements of wong and cmor’s examination of connections between library workshop attendance and gpa, but adds the student perspective from focus groups to support and fill in gaps from the quantitative analysis. the utility of a tiered approach to information literacy instruction is not thoroughly addressed in the library literature, and this study of university of wyoming students seeks to fill that gap by providing an argument for a programmatic approach with library instruction at various levels in the curriculum. aims the purpose of this research is to learn about the relationship between students’ academic success and information literacy instruction. the author hypothesized that graduating seniors who had continuing library instruction in their sophomore, junior, or senior year would be more successful, as reflected by gpa, than students who had library instruction in their freshman year alone, due to repeated practice and reinforcement of library research skills. the author also sought to understand students’ perceptions of their own learning and their experience with library instruction. the study was undertaken with several research questions in mind: • what is the relationship between student academic success and information literacy instruction? • which students receive library instruction and which do not? evidence based library and information practice 2012, 7.1 86 • is there a good argument for creating a tiered program of information literacy instruction? • how can we improve our program of information literacy instruction? librarians often approach these questions from a librarian-centred perspective: we gather data on how many students attend information literacy instruction sessions, how many students successfully complete research assignments, and so on. at the university of wyoming all of our assessment data on this subject has been from the librarian perspective rather than from the student perspective. the author felt it was important to find out what the library instruction program looks like from a student perspective, which is one of the things this project attempts to do. methods assessment of student learning is bound to be imperfect, as so many factors lie outside of the instructor’s and the librarian’s control, but using a mixed-methods approach to gather both quantitative and qualitative data can give a more complete picture. the qualitative method for this study was a focus group discussion. in march 2011, with the assistance of another librarian, the researcher conducted two focus groups with graduating seniors to learn about their engagement with the library, and more specifically to find out what they learned from library instruction sessions. fifteen graduating seniors were recruited at the spring graduation fair. there were 10 women and 5 men, with majors in humanities, arts, sciences, applied sciences, and social sciences. this research method, of course, does not aim to be representative of the whole group of graduating seniors, but to learn more in depth about a few students’ experiences. the incentive for students to participate was a free meal and a usb drive. students in the groups gave their informed consent to participate and to be recorded. the facilitator had a script (appendix b) with questions that each student answered in turn. recordings of the two focus groups were transcribed and the author analyzed responses by organizing comments into six themes, which emerged from the participants’ comments: 1) suggestions for library instruction services; 2) anecdotes regarding the value of library instruction; 3) comments regarding the value of library sources, library space, and library staff; 4) suggested timing for library instruction; 5) barriers to asking for help; and 6) miscellaneous suggestions or requests. all of these themes proved useful for answering the questions addressed in this study. later in 2011, after spring graduation exercises, the author requested data from the registrar’s office for the quantitative element of the study. the institutional review board approved the transcript request, as well as the focus group element of the study. the author analyzed the academic transcripts of students who entered the university between 2005 and 2007 and who graduated between 2006 and 2011, excluding graduate and professional students. the dataset includes, for each student, a list of classes taken each year with grades for every class, major when the student entered the university, major when the student graduated, gpa at graduation (calculated on a four-point scale), and sex. a total of 4,489 student transcripts were involved. this analysis required a list of the classes that librarians have met with for the past several years. the research and instruction services department has kept records of this since 2001, but with varying levels of detail. a fairly comprehensive list of classes librarians taught from 2005 to the present was compiled, but in some cases it was impossible to find out if every section of the classes had library instruction or if it was just select sections visiting the library. the list of classes that received library instruction does not include individual student names or numbers, so when collating the list of classes that received library instruction with students’ academic transcripts, it was assumed that if a student completed a course with library instruction, then that student attended class on the day or days when library instruction was provided. the author created a database to compare the transcript data with records of library instruction sessions offered and sorted evidence based library and information practice 2012, 7.1 87 students into four groups: 1) those who received freshman-level library instruction and upper-level (post-freshman) library instruction, 2) those who received freshmanlevel instruction only, 3) those who received post-freshman library instruction only, and 4) those who received no library instruction. in the statistical analysis anova was used to compare the means of the four groups, which revealed a statistically significant difference (p<.0005). then a post hoc analysis was conducted to discover where the difference was found using the dunnett test, with the fourth group (which received no library instruction) as the control or baseline group. because there was no significant difference in gpa between groups one and three (both of which had upper-level instruction), it was appropriate to combine those groups. students who did not receive freshman-level instruction at university of wyoming (mostly transfer students) may have received freshman-level instruction at their previous institutions, which was not controlled for in the analysis. the resulting three comparison groups were: 1) those who received upper-level library instruction, 2) those who received freshmanlevel instruction only, and 3) those who received no library instruction. with these three groups the data was reanalyzed, using anova and a post hoc dunnett test once again. following common statistical practice, the level of significance was set at 0.05. results focus groups the 15 graduating seniors who participated in the focus groups reported between one and four visits to the library for instruction and said that such visits were generally useful. students expressed the need for two different types of instruction: an orientation in the first year followed by upper-division instruction in which students learn about resources in their majors. when discussing when, in the course of a college career, library visits are most useful, 12 of the 15 students suggested that a first-year visit plus later visits would be ideal. one student expressed it this way: “it would be cool if you had a freshman thing, then as you get more specialized in your field, more specific, scholarly instruction.” another suggestion on this topic was to create a library orientation (in-person or online) for transfer students or for review by upper-division students. course guides that provide an opportunity for later review are also valued by students. eight students talked about asking a librarian for help when they lacked information about how to use the library. others said they learned to use the library by asking friends or through trial and error. some expressed real barriers to asking for help: i think a lot of people – especially our generation – because we’ve grown up with the internet and computers and that’s the way we’re used to finding things out, so we’re not as inclined to ask people for help. we’re just like, “phhht, i can figure this out. give me three hours and a mouse and i’ll figure it out.” another student’s response suggests that an important aspect of library instruction is increasing awareness of library services: if you’re completely ignorant then you have no idea that you don’t even know. so unless somebody tells you that you don’t know you don’t know, then you’re not going to go look for that information. in general, as students learn more in college they become more aware of what they do not know (part of the educational process), and that includes library and information literacy topics. all of the students in the focus groups talked about research projects completed in specific classes and the databases, sources, and tools used for those projects that they learned about from library instruction sessions. students also said that the library enabled them to do other evidence based library and information practice 2012, 7.1 88 things, such as valuing scholarly research over basic web search results. academic transcript analysis analysis revealed a statistically significant relationship between students’ gpa at graduation and upper-division library instruction. the three comparison groups were: 1) students who received upper-level library instruction, 2) students who received only freshman-level instruction, and 3) students who received no library instruction at all. the mean gpa for each of the three groups is displayed in table 1; though the variance looks very small, statistical analysis reveals that there is a statistically significant difference. table 2 shows the results of anova: there is a statistically significant difference between the three groups, f(2,4486)=3.089, p<.0005. a post hoc analysis was conducted to find where the difference lies. the dunnett test was used and the “none” group was considered the control or baseline group, as seen in table 3. dunnett t-tests treat one group as a control, and compare all other groups against it. the dunnett test shows that the only group different from the control group is the upper-level instruction group with a mean difference of .0748, p<.0005. thus, students who receive upper-level instruction at the library also have higher gpas, while there is no significant difference in gpa for students who have only freshman-level library instruction. because this is ex post facto research, the author cannot claim that the instruction was the cause of the increase; there are too many confounding variables to claim causality in the relationship between information literacy instruction and gpa. perhaps most notably, there is probably an effect from the repetition of instruction, which was not analyzed in this study. but the analysis shows a statistically significant positive correlation between upper-level library instruction and a higher grade point average at graduation. table 1 mean gpa for three comparison groups mean gpa group 1: upper-level library instruction 3.289 group 2: freshman-level library instruction 3.247 group 3: no library instruction 3.214 table 2 anova to discover difference between groups with upper-level library instruction and groups without anova gpa sum of squares df mean square f sig. between groups 1.259 2 .630 3.089 .046 within groups 914.405 4486 .204 total 915.664 4486 table 2 anova to discover difference between groups with upper-level library instruction and groups without anova gpa sum of squares df mean square f sig. between groups 1.259 2 .630 3.089 .046 within groups 914.405 4486 .204 total 915.664 4486 evidence based library and information practice 2012, 7.1 89 discussion putting qualitative results together with quantitative results provides a well-rounded assessment of the instruction program. for each of the research questions listed, answers can be drawn from both the qualitative and quantitative elements of this study for a more complete picture. relationship between student academic success and information literacy instruction the students in the focus group all discussed specific skills or tools learned in library instruction sessions that they were able to use in research projects assigned for various classes, which points to the importance of library instruction in academic success. additionally, the academic transcript analysis shows a significant relationship between upper-division information literacy instruction and gpa at graduation, which is one standardized way of measuring academic success. clearly, the difference in gpa is very small (.075 between no library instruction and upper-level library instruction), but that difference can determine whether or not a student is accepted into a specific degree program and can also be an important factor for students applying to graduate schools. courses and programs that include or do not include library instruction all of the students in the focus groups received library instruction, but the students who had transferred to the university of wyoming after studying elsewhere expressed the need for a library orientation. an important consideration for librarians is to understand that transfer students are less likely to know the basics about the library. the analysis of academic transcripts also revealed degree programs in which students are less likely to receive library instruction. data gathered and analyzed in this study may be useful in marketing library instruction to those departments; this is an area for future research and action. the argument for creating a tiered program of information literacy instruction one of the major questions going into this project was whether there is a good argument for creating a tiered program of information literacy instruction. when students were asked when they would like to have received library instruction, 12 of the 15 in focus groups said they would like a freshman visit to introduce the library and its services plus a later, subjectspecific visit. the fact that there is a correlation between upper-division library instruction and higher gpa at graduation also suggests that information literacy instruction after the table 3 dunnett test to find the difference between groups multiple comparisons gap dunnet t(>control) (i) groups (j) groups mean difference (i-j) std. error sig. 95% confidence interval lower bound upper-level instruction no instruction .07480* .03607 .031 .0079 freshmanlevel instruction no instruction .03328 .04469 .310 -.0496 *the mean difference is significant at the 0.05 level. evidence based library and information practice 2012, 7.1 90 freshman year is important. additionally, learning theory argues that reinforcement and scaffolding are important to developing an understanding of concepts. the latest version of our departmental instruction and assessment plan includes a section that outlines appropriate learning outcomes for different levels, and we hope to make an argument that the new general education program should include information literacy learning outcomes at both the freshman-level and in an upper-division class in every major. improving the information literacy instruction program not every student takes the same path through college, but seeing the path that some take is enlightening for lesson planning and program development. information literacy is integrated in the university curriculum, but that curriculum is somewhat fragmented and experienced differently by each student. seeing how students actually experience the library instruction program is important to designing a cohesive (and non-repetitive) curriculum. in the focus groups, some students expressed irritation at hearing the same thing over again in library instruction sessions. it seems obvious, but instruction librarians should differentiate between lower-division and upper-division learning objectives. students suggested creating videos or short tutorials to cover the basic orientation information that upper-division students could review and that transfer students could use. one important thing learned from the analysis of academic transcripts is that three-quarters of uw students receive freshman-level instruction, so librarians should be aware when going in to upper-division classes that we are speaking to students who already know the basics. many of the students also said that they appreciated the availability of an online course guide to refer to after the library instruction session; students who had not found a guide for their class or major were jealous of students who had such guides. students’ comments about their willingness to talk to a librarian about research questions also suggest that meeting with them in their classes and inviting them to ask questions are an important part of library instruction. limitations the limitation of using focus groups is that results are not necessarily generalizable across a student body or between institutions. the analysis of academic transcripts was limited by the imperfect records kept regarding sections of courses that received library instruction. additionally, it is important to remember that in ex post facto research such as this, researchers cannot claim that the library instruction was the cause of the improved grade point averages among students who received instruction in upper-division classes; there is simply a correlation. there are numerous other variables that will have an effect on gpa and learning: student motivation and preparedness, research assignments that are engaging and challenging, level of course instructor engagement, and many more. this analysis can hardly take all of those factors into consideration. using gpa as a surrogate for student learning is not a direct measure of student learning, and there are differences in gpa that cannot be accounted for. average gpas differ between majors and colleges at university of wyoming and at other institutions, and grade inflation is a common concern in higher education. library instruction and its effects in various disciplines are another area for future study. also, this study focused on the value of library instruction embedded at different levels in the university curriculum, but did not account for the effect that repeated library instruction sessions may have. students in the transcript analysis comparison groups had between zero and six library instruction sessions and another analysis might look at the differences that emerge when students have repeated interactions with librarians in the classroom. recommendations this study helped the institution identify ways to improve instruction assessment practices, and other libraries can benefit from these observations as well. tracking which courses evidence based library and information practice 2012, 7.1 91 receive library instruction is vital. one recommendation that emerged from this study is to create or revise instruction reporting forms to collect data on the course, section, learning outcomes addressed, and assessment methods used. we must determine what coding is used in academic transcripts and use the same on reporting forms in order to facilitate data analysis. there are other measures of student academic success that may be more meaningful than gpa, such as subsequent employment rates, employer evaluations of former students, or percentage of students who go on to graduate school. those are valuable ways for librarians to assess the effectiveness of library instruction. correlating gpa, however, does provide a starting point for proving the value of library instruction. a mixed-methods study of this type can help an instruction program to plan for future assessment efforts. overall, a program-level survey from a student-centred perspective can give libraries a starting place for a longitudinal, coherent program of assessment as it offers a view on how library instruction touches students, and can help librarians to design a cohesive and effective library instruction program. asking students about their preferences, what they value, and how library instruction can be improved provides insight that librarians need. conclusions the focus groups and academic transcript analysis undertaken in this study demonstrated a positive correlation between higher gpa and information literacy instruction at university of wyoming, when the instruction was offered in upper-division courses rather than solely in freshman-level classes. this data provides an argument for creating a tiered program of information literacy, with information literacy learning outcomes embedded at different levels in the university curriculum. the study also provided librarians with a more complete picture of which students receive library instruction and which do not, along with data to provide to instructors and departments regarding the potential positive effects of library instruction. a library instruction program that has clearly defined goals for students at every level of university study and a scaffolded approach to student mastery of information literacy skills will have the greatest impact on student learning and student success. acknowledgements thanks to jenny weatherford for assistance with statistical analysis, david d. kruger for facilitating focus groups, lori terrill for a very helpful internal peer review of this article, and university of wyoming library administration for providing incentives for focus group participants. references ary, d., jacobs, l. c., & sorenson, c. (2010). introduction to research in education (8th ed.). belmont, ca: wadsworth cengage learning. canning, c. s., edwards, a. j., & meadows, s. e. (1995). using focus groups to evaluate library services in a problembased learning curriculum. medical reference services quarterly, 14, 75-81. kirk, r., vance, j., & gardner, j. (2010). the impact of library instruction on freshman performance & retention. proceedings from library assessment conference 2010: building effective, sustainable, practical assessment. baltimore, md: association of research libraries. retrieved 2 feb. 2012 from http://libraryassessment.org/bm~doc/k irk_rachel.pdf kwon, n. (2008). a mixed-methods investigation of the relationship between critical thinking and library anxiety among undergraduate students in their information search process. college & research libraries, 69(2), 117-131. evidence based library and information practice 2012, 7.1 92 matthews, j. r. (2007). library assessment in higher education. westport, ct: libraries unlimited. moore, d., brewster, s., dorroh, c., & moreau, m. (2002). information competency instruction in a two-year college: one size does not fit all. references services review 30(4), 300-306. doi:10.1108/00907320210451286 morrison, h. g. (1997). information literacy skills: an exploratory focus group study of student perceptions. research strategies, 15, 4-17. oakleaf, m. (2010). value of academic libraries: a comprehensive research review and report. chicago: association of college and research libraries. spackman, e. (2007). utilizing focus groups to evaluate an information literacy program in a general biology course. science & technology libraries, 27(3), 328. doi:10.1300/j122v27n03_02 wakimoto, d. k. (2010). information literacy instruction assessment and improvement through evidence based practice: a mixed method study. evidence based library and information practice, 5(1), 82-92. wong, s. h. r., & cmor, d. (2011). measuring association between library instruction and graduation gpa. college & research libraries, 72(5), 464-473. evidence based library and information practice 2012, 7.1 93 appendix a suggested learning outcomes skills taught at different levels in library instruction sessions, librarians can help students develop cognitively appropriate information literacy skills. the following are specific, discrete skills and concepts that we teach in information literacy instruction at university of wyoming libraries. these skills and concepts fit into the framework of the curriculum map, and identify additional library skills traditionally taught by librarians. first-year students and students in freshman-level classes work on developing general information literacy skills that will be applicable to research in their discipline and to lifelong learning: • inquiry & analysis o coming up with a researchable topic o articulating a research question o identifying useful keywords o finding known items by title or author o using advanced search tools on the web (google scholar, limiting searches by domain, etc.) o using library services and resources • think critically & creatively o evaluating sources for relevance and authority students at the sophomore and junior level develop the above skills and more subject-specific and advanced information literacy skills: • inquiry & analysis o using both primary and secondary sources o using some subject-specific databases o developing more sophisticated search strategies (i.e., boolean logic, truncation, and phrase searching) • ethical reasoning & action o understanding copyright and ethics o avoiding plagiarism and citing sources appropriately students at the senior level should become familiar with subject-specific resources, sophisticated search strategies, and should prepare for meeting their post-graduation information needs by developing the following skills: • inquiry & analysis o finding and using subject-specific information tools and databases o using controlled vocabulary o becoming familiar with important journals in their area of study o citation mapping and other advanced strategies for searching the literature in their area of study o finding and using resources to meet their professional information needs after leaving uw • thinking critically & creatively o evaluating the quality of information resources o understanding the information life cycle and where different information needs can be satisfied graduate students need the same skills as students at the senior level, but may also focus on: • inquiry & analysis evidence based library and information practice 2012, 7.1 94 o strategies for maintaining current awareness o citation management • synthesize & communicate o writing a review of the literature skill levels table freshman (general information literacy skills) sophomore and junior (emerging subject specialty) senior+ (subject specific) graduate+ (graduate student skills) coming up with a researchable topic copyright & ethics subject specific tools citation management articulating a research question avoiding plagiarism controlled vocabulary writing a literature review identifying useful keywords citation styles quality of information current awareness evaluating sources for relevance & authority primary and secondary sources journals in area finding known items search strategies: boolean, truncation, phrase information life cycle (where can your information need be met?) web-savvy (google tools) citation mapping and advanced strategies for literature searches using the library (orientation) lifelong professional resources (non-uw subscriptions) evidence based library and information practice 2012, 7.1 95 appendix b focus group script • welcome the students, thank everyone for coming, and introduce yourself. • invite everyone to get some food and drink. • provide consent forms for everyone to complete. we must have one on file for each participant. • give everyone a jump drive for participating. • make sure that everyone checks the “you may record this session” box and signs. • ask everyone to write their first name on a blank sheet of paper and make it into a table tent. names will be removed during transcription, but will be useful for identifying the participants during the session. go around and have everyone state her or his name. • after collecting consent forms, let everyone know that you’re going to start the conversation, which will last no more than 90 minutes. students may leave at any time and are not obligated to answer any of the questions. their participation will help us improve library instruction for those who follow them. request honesty, make sure they know that responses should be kept confidential, and invite students to ask any questions that they have during the session. • start the recording. • ask everyone these questions, and any follow-up questions that suggest themselves: o when have you visited the library with a class for a meeting with a librarian? (freshman, sophomore, junior, senior year) o how useful were those class visits? how could they have been more useful? o at what point in your college career would class visits to the library have been most helpful? o did you ever feel that you were lacking information about how to use the library or how to do research? when? what did you do? o what are some of the research projects you did while a student at uw? did you use anything you learned from a librarian to complete those research projects? how? o what has the library enabled you to do? o what do you value about the library? / evidence based library and information practice/ / skill levels table evidence based library and information practice 2012, 7.4 1 evidence based library and information practice editorial evidence and ethics alison brettle editor-in-chief senior lecturer, school of nursing midwifery and social work university of salford salford, united kingdom email: a.brettle@salford.ac.uk 2012 brettle. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. welcome to the december issue of eblip, the final issue of my first year as editor-in-chief. a year which i have thoroughly enjoyed and one where the fears over what to write in my editorials haven’t materialised. this quarter, ethics has featured quite heavily in my working life so i decided to make this the topic of the editorial, sharing some of my thoughts regarding evidence, ethics and how ethical principles are implemented within the eblip journal. ethics are “principles of conduct or standards of behaviour governing an individual or profession” (library and information science editorial committee, 2010), and as individuals or professionals we may be governed by various ethical codes. as i'm sure you know, eblip originated in the health domain, where ethical values and ethical research feature strongly. indeed, by its formal definition, research cannot take place unless “ethical approval” from an appropriate committee has been granted. the practicalities of taking research through the ethical approval process can often be time consuming, and those involved in research need to bear this in mind when planning a project. each committee will have a slightly different form and process (which can add to the frustration of the researcher), but basically will make their decision to approve on the basis that the research includes obtaining informed consent from participants (i.e., participants know what the research is about and what their involvement will mean); that the research will not cause harm to participants; that confidentiality will be maintained; and that the research undertaken is methodologically rigorous and worthwhile. preparing a proposal for ethical approval, whilst time consuming, makes the researcher think about all aspects of the research and how it is going to be operationalized, which can save lots of time and effort in the long run and may well also improve the research design. these principles are the same whatever discipline the research takes place in, and should be something that we are aware of as consumers of evidence. within lis in the uk, ethical principles have been put to the fore within a new professional framework (cilip chartered institute of library and information professionals, 2012a). the framework outlines the broad range of mailto:a.brettle@salford.ac.uk evidence based library and information practice 2012, 7.4 2 skills required by workers across the lis profession, placing these on a wheel with ethics and values in the centre, as they underpin the profession. placing ethics and values at the core in this way helps us set our knowledge into a wider context and, i believe, is one of the ways that we can make a difference as lis professionals. at the same time, our ethical values and principles help to differentiate us from other professions and help to define what we do as lis professionals. these ethical principles are outlined in a code (cilip chartered institute of library and information professionals, 2012b), which sets out professional responsibilities in relation to users, colleagues, and the information community and society. the elements which are particularly relevant to eblip, and which are espoused in the scope and mission of the journal, include maintaining and enhancing professional knowledge and competence, sharing results of research and development, encouraging best practice, and promoting equitable access to information. there are also ethical codes of practice for journal editors, these include one for lis editors (library and information science editorial committee, 2010) and a more general one which originated in the medical and health domain (committee on publication ethics: cope, 2011). both of these guide journal editors in relating to readers, authors, reviewers, and publishers, and both seek to establish best practice for journal publishing. for the eblip journal, these codes of practice provide a useful framework for ensuring the journal operates in a professional and ethical way. a recent example where the codes have been used in eblip is in dealing with a submission from one of the editorial team (koufogiannakis, 2012). strong ethical principles have been at the heart of the eblip journal since its inception, but decision making is quite informal and based on the agreement of the editorial team members who meet monthly and communicate regularly by email. in keeping with our informal decision making structure the editorial team discussed how to deal with the submission by an editorial team member and decided that it should be treated in exactly the same way as any other submission (i.e., subject to double blind peer review and would need to adhere to the decision of the appropriate editor). to do this we also needed to adjust the open journal system, so that the submitting editor did not have access to the area where submissions and reviews are stored. in line with the cope code (2011), we then made this procedure explicit on our web site (http://ejournals.library.ualberta.ca/index.php/ eblip/about/editorialpolicies#peerreviewproc ess), as well as documenting the decision on our internal wiki. as the journal grows and develops, it is important that our decision making and principles maintain consistency and are made explicit. we also need to plan for the future and ensure that the journal is open to as wide a range of people as possible. to this end all our positions are advertised widely, and each application is reviewed by at least two members of the team. terms of office and a policy for succession planning have also been established. the criteria and policies are stored on a wiki, which hosts a wide range of guidelines, procedures, and documents and is used by all members of the editorial team. as the team grows and similar issues are faced by new members of the editorial team, we have a reference point, policy, or guide to ensure that the decisions made are consistent. as appropriate these can be reflected in the policies which are stated on the journal website and are available to all our users. in this way, we hope that the journal maintains its strong professional and ethical ethos and continues to ensure that the best available evidence about lis research is made available to help practitioners in their decision making. references cilip chartered institute of library and information professionals. (2012a). your professional knowledge and skills base. in cilip. retrieved 27 nov. 2012 from http://www.cilip.org.uk/jobscareers/professional-knowledge-andskillshttp://ejournals.library.ualberta.ca/index.php/eblip/about/editorialpolicies#peerreviewprocess http://ejournals.library.ualberta.ca/index.php/eblip/about/editorialpolicies#peerreviewprocess http://ejournals.library.ualberta.ca/index.php/eblip/about/editorialpolicies#peerreviewprocess http://www.cilip.org.uk/jobs-careers/professional-knowledge-and-skills-base/pages/professional%20knowledge%20and%20skills%20base.aspx http://www.cilip.org.uk/jobs-careers/professional-knowledge-and-skills-base/pages/professional%20knowledge%20and%20skills%20base.aspx http://www.cilip.org.uk/jobs-careers/professional-knowledge-and-skills-base/pages/professional%20knowledge%20and%20skills%20base.aspx evidence based library and information practice 2012, 7.4 3 base/pages/professional%20knowledg e%20and%20skills%20base.aspx cilip chartered institute of library and information professionals. (2012b). code of professional practice for library and information professionals. in cilip. retrieved 27 nov. 2012 from http://www.cilip.org.uk/getinvolved/policy/ethics/documents/co de%20of%20professional%20practice% 20for%20library%20and%20informati on%20professionals%20oct%202012.p df committee on publication ethics. (2011). code of conduct and best practice guidelines for journal editors. in committee on publication ethics: cope. retrieved 27 nov. 2012 from http://publicationethics.org/files/code _of_conduct_for_journal_editors_mar1 1.pdf library and information science editorial committee. (2010). a statement of ethics for editors of library and information science journals. in library & information science editors. retrieved 27 nov. 2012 from http://www.liseditors.org/bm~doc/lis-editorstatement-ethics-2010-final.pdf koufogiannakis, d. (2012). academic librarians’ conception and use of evidence. evidence based library and information practice, 7(4), 5-24. http://www.cilip.org.uk/jobs-careers/professional-knowledge-and-skills-base/pages/professional%20knowledge%20and%20skills%20base.aspx http://www.cilip.org.uk/jobs-careers/professional-knowledge-and-skills-base/pages/professional%20knowledge%20and%20skills%20base.aspx http://www.cilip.org.uk/get-involved/policy/ethics/documents/code%20of%20professional%20practice%20for%20library%20and%20information%20professionals%20oct%202012.pdf http://www.cilip.org.uk/get-involved/policy/ethics/documents/code%20of%20professional%20practice%20for%20library%20and%20information%20professionals%20oct%202012.pdf http://www.cilip.org.uk/get-involved/policy/ethics/documents/code%20of%20professional%20practice%20for%20library%20and%20information%20professionals%20oct%202012.pdf http://www.cilip.org.uk/get-involved/policy/ethics/documents/code%20of%20professional%20practice%20for%20library%20and%20information%20professionals%20oct%202012.pdf http://www.cilip.org.uk/get-involved/policy/ethics/documents/code%20of%20professional%20practice%20for%20library%20and%20information%20professionals%20oct%202012.pdf http://www.cilip.org.uk/get-involved/policy/ethics/documents/code%20of%20professional%20practice%20for%20library%20and%20information%20professionals%20oct%202012.pdf http://publicationethics.org/files/code_of_conduct_for_journal_editors_mar11.pdf http://publicationethics.org/files/code_of_conduct_for_journal_editors_mar11.pdf http://publicationethics.org/files/code_of_conduct_for_journal_editors_mar11.pdf http://www.lis-editors.org/bm~doc/lis-editor-statement-ethics-2010-final.pdf http://www.lis-editors.org/bm~doc/lis-editor-statement-ethics-2010-final.pdf http://www.lis-editors.org/bm~doc/lis-editor-statement-ethics-2010-final.pdf evidence based library and information practice references cilip chartered institute of library and information professionals. (2012a). your professional knowledge and skills base. in cilip. retrieved 27 nov. 2012 from http://www.cilip.org.uk/jobs-careers/professional-knowledge-and-skills-base/pages/professio... cilip chartered institute of library and information professionals. (2012b). code of professional practice for library and information professionals. in cilip. retrieved 27 nov. 2012 from http://www.cilip.org.uk/get-involved/policy/ethics/documents/co... committee on publication ethics. (2011). code of conduct and best practice guidelines for journal editors. in committee on publication ethics: cope. retrieved 27 nov. 2012 from http://publicationethics.org/files/code_of_conduct_for_journal_editors_ma... library and information science editorial committee. (2010). a statement of ethics for editors of library and information science journals. in library & information science editors. retrieved 27 nov. 2012 from http://www.lis-editors.org/bm~doc/lis-edi... koufogiannakis, d. (2012). academic librarians’ conception and use of evidence. evidence based library and information practice, 7(4), 5-24. evidence based library and information practice 2012, 7.3 83 evidence based library and information practice evidence summary university engineering faculty depend on scholarly journals, web resources, and faceto-face consultations to help them with research a review of: engel, d., robbins, s., & kulp, c. (2011). the information-seeking habits of engineering faculty. college & research libraries, 72(6), 548-567. reviewed by: laura newton miller science and engineering reference librarian carleton university ottawa, ontario, canada email: laura_newtonmiller@carleton.ca received: 30 apr. 2012 accepted: 18 june 2012 2012 newton miller. this is an open access article distributed under the terms of the creative commons‐ attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to study the information-seeking behaviour of engineering faculty. design – online survey; purposive sample. setting – engineering departments of 20 large public universities in various regions of the united states. subjects – 903 engineering faculty members (including 35% professors; 24% associate professors, 23% assistant professors, and 17% ranked as adjunct faculty, instructors, lecturers, professors emeriti and “other”). methods – 4905 researchers were sent an email invitation to complete a 12-item survey with open and closed questions. email addresses were gathered from university websites. main results – 96% of those surveyed find access to online scholarly journals (current and backfiles) as very important or important. 71% believe access to the physical book collection is very important or important. 56% feel that access to electronic book collections is very important or important. (further analysis revealed a difference between newer and older faculty62% of newer faculty and 52% of faculty in field for 16 or more years think electronic book collections are important). print subscriptions to journals are important to only 37% of respondents, and providing space to conduct research is important to only 36% of those surveyed. besides attending conferences and scanning journals, face-to-face discussion mailto:laura_newtonmiller@carleton.ca evidence based library and information practice 2012, 7.3 84 with students and colleagues was a key resource for faculty for keeping current in the engineering field. 81% seek information at least weekly to prepare for lectures, about 74% at least monthly to conduct research or write publications, and 77% at least monthly to remain current in their field. 73% visited the physical library fewer than five times in the past year, but researchers were surprised that almost half (47%) rated assistance from library staff as important or very important. 70% see interlibrary loan services as important or very important. conclusion – engineering faculty rely on scholarly journals, internet, and other electronic resources for their research. they depend on face-to-face consultations with students and colleagues. the physical space of the library is less important. commentary research papers studying engineers usually focus on them as practitioners, as a comparison to scientists, or as a comparison between practitioners and faculty. this study focuses completely on the engineering faculty member’s information needs in a large university setting. using the eblip critical appraisal checklist, it was determined that a survey was an appropriate tool for this kind of research and there was face validity in the study design. the methods were clearly explained and the survey was included at the end of the paper, making it easily replicable for further study. a response rate of 18% is a little low (903 responses out of 4905 invitations). having a purposive sample of only large research institutions makes one wonder if results might have been different with a bigger response rate that included smaller institutions. a definition of “large” would have been beneficial for readers, and it would be interesting to discover if needs differ between faculty of smaller and larger universities. a description of trends by geographic area would have been useful. it is also unclear if more respondents were from one university than another. how evenly distributed were the results among institutions? could this have affected results? emails taken from institutional websites can sometimes be quite out-of-date. contacting administrators from engineering departments to forward the survey to faculty may have yielded results from a more up-to-date list. perhaps faculty may be more likely to respond to a survey sent from someone they know, as opposed to an email that may have gone straight to their junk folder. the results confirm findings from other studies that engineers rely heavily on online resources for their information needs. it was interesting to see that 71% of respondents feel the physical book collection is either important or very important, yet very few actually visit the physical library. faculty loan periods can vary among institutions, so perhaps they are coming in fewer times because of longer loan periods with online renewal options. depending on interlibrary loan options, it is possible that books are being delivered straight to their offices, negating the need to visit the physical library. there seems to be a difference between younger and older faculty’s views on e-books. future studies regarding how engineers use e-books, and whether there are differences between how various formats are used for everyday work needs will be important for librarians in understanding this user group. librarians face difficult financial decisions when purchasing resources in a variety of formats with shrinking budgets. this study reinforces the fact that engineering librarians need to figure out ways to stay relevant when dealing with patrons who rarely visit the physical library. providing online resources at point-of-need is one way to do that. this paper helps engineering librarians to better understand their faculty by knowing electronic access to journal articles, monographs and other online sources are important in research and teaching. evidence based library and information practice 2012, 7.3 85 references glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi:10.1108/07378830610692154 evidence based library and information practice evidence based library and information practice 2010, 5.4 115 evidence based library and information practice commentary evidence based cataloguing: moving beyond the rules kathy carter projects librarian, bibliographic and information technology services university of alberta libraries edmonton, alberta, canada email: kathy.carter@ualberta.ca received: 11 oct. 2010 accepted: 18 oct. 2010 2010 carter. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. cataloguing is sometimes regarded as a rulebound, production-based activity that offers little scope for professional judgement and decision-making. in reality, cataloguing involves challenging decisions that can have significant service and financial impacts. the current environment for cataloguing is a maelstrom of changing demands and competing visions for the future. with information-seekers turning en masse to google and their behaviour receiving greater attention, library vendors are offering “discovery layer” products to replace traditional opacs, and cataloguers are examining and debating a transformed version of their descriptive cataloguing rules (resource description and access or rda). in his “perceptions of the future of cataloging: is the sky really falling?” (2009), ivey provides a good summary of this environment. at the same time, myriad new metadata formats and schema are being developed and applied for digital collections in libraries and other institutions. in today’s libraries, cataloguing is no longer limited to management of traditional aacr and marc-based metadata for traditional library collections. and like their parent institutions, libraries cannot ignore growing pressures to demonstrate accountability and tangible value provided by their services. more than ever, research and an evidence based approach can help guide cataloguing decision-making. decisions librarians face a variety of cataloguing-related decisions that arise in various settings. these are only a few of the issues that are challenging us at the present time. at the level of international standards and practice, • can one set of descriptive rules serve for all types of information objects? • what metadata are required to meet the differing needs of different users? • will rda produce records that satisfy user needs? • when can marc be abandoned and what should replace it? mailto:kathy.carter@ualberta.ca� evidence based library and information practice 2010, 5.4 116 • how can libraries share metadata design and creation with other players such as publishers, indexers, archives, and authors? at the level of individual libraries, for traditional cataloguing, • which sources of metadata records are the most comprehensive, the best quality, the cheapest, the most suited to the library’s collection and policies? possibilities include oclc, book and other library vendors, skyriver, and other library catalogues. • what sort of record checking, upgrading and customization adds tangible value? • can savings be realized by outsourcing some work, such as copy cataloguing, original metadata creation, or authority control? and as other forms of metadata and discovery are added to traditional cataloguing, • how and when can dissimilar records be combined effectively in a single database or search? • what are the needs of our different user communities and how can we differentiate our metadata content, retrieval and display for each? • can a virtual international authority file (viaf) improve author disambiguation across multiple metadata sources, or would programmatic disambiguation solutions be more effective? • how important is consistency of data such as author, genre and geographic headings? • what software and system capabilities are most important for efficient, scalable operations and services? • what are the most effective ways to expose our metadata and integrate it into the web tools and venues where our users are? metadata over time, metadata is subject to change and reuse; it may be enhanced, corrected, indexed, extracted, merged, converted, crosswalked, and reformatted. but most importantly, metadata endures. worldcat contains records created centuries ago, cheek by jowl with millions of others created since, right up until just seconds ago. of equal importance, metadata is widely shared, reflecting the goal of minimizing redundant effort. decisions regarding metadata content and standards are made in this context of long-term and shared record use in aggregated environments. typically in the traditional cataloguing context, standards are developed by national and international bodies, changes are not made lightly or quickly, and practitioners tend to develop a habitual reliance on rules rather than evidence. conversely, creators of metadata for digital collections such as institutional repositories have often shunned traditional cataloguing structures such as aacr and marc, and developed standards that are better suited to individual collections and to networked resources and services. rda attempts to serve both these worlds, but has been the object of vigorous criticism from both.i management of cataloguing operations the organization, staffing, and supervision of cataloguing operations, and management of purchased or locally developed services and systems, are significant issues and cost centers for libraries. some of the issues are similar to management issues in other library and nonlibrary settings, and they are more amenable than metadata standards to local decisionmaking. yet operational decision-making cannot be separated from definition and assessment of desired outcomes. the difficulties in achieving this can be seen in the final report of the task force on cost/value assessment of bibliographic control of the ala heads of technical services in large research libraries interest group (2010). charged with “identifying measures of the cost, benefit, and value of bibliographic control for key stakeholder communities, and developing a plan for implementing these measures” (p.2), the task group noted that: evidence based library and information practice 2010, 5.4 117 the objective of this work was… to begin to identify sound measures that can inform decisions by those engaged in the creation, exchange, and use of bibliographic data. our ability to make sound decisions and mindful changes around bibliographic control is hindered by our lack of operational definitions of value and methodologies for assessing value within our institutions. (p.2) cataloguing research some aspects of cataloguing, such as the effectiveness of keywords vs. controlled vocabularies for subject searching, have been subject to detailed research.ii yet the scope, complexity, and uses of cataloguing present tremendous challenges to designing and conducting relevant research. in 1997 ling hwey jeng observed “the lack of empirical studies of many fundamental theories in cataloging” (p.124), and in 2008 janet swan hill noted “a persistent shortcoming in the decision-making process that needs to be addressed is the lack of serious research into user needs and benefits, and the actual impact on users of database quality decisions” (p.5). in recent years oclc has gone some distance to reduce this shortcoming, with research into metadata content and user behaviour and expectations. in their 2009 study “online catalogs: what users and librarians want” (oclc) provides a welcome well-grounded insight into users’ priorities, and a revealing comparison with how they differ from those of librarians. yet it is discouraging that the current extensive testing of rda has as its goal “to assure the operational, technical, and economic feasibility of rda” (library of congress, 2010, para. 1) with no attempt to include tests of its effectiveness in serving users. inspired by on the record: report of the library of congress working group on the future of bibliographic control (2008), the american library association’s association for library collections & technical services (alcts) declared 2010 as the year of cataloging research (news from alcts, 2009). this encouraging development should help build awareness of the value of research. barriers to evidence based practice in cataloguing in addition to the paucity of relevant cataloguing-related research, it can be difficult to find results of research that has been conducted. organisations and vendors who carry out and analyze research in support of their own product development often do not release it publicly, and research done by libraries to guide their own decisions is sometimes not publicised or distributed beyond their own institutions. relevant research, especially related to user interfaces and user behaviour, is done in non-library disciplines and may be overlooked. other barriers that can result in failure to seek and apply research findings to decision-making are lack of time or management support and a perceived disconnect between researchers and practitioners. in addition, the pace and variety of new technology and application developments creates pressure to implement novel services quickly and makes their assessment challenging. in cataloguing, there are powerful pressures to accept rules, records, and systems created by others. the dependencies resulting from this mode of practice can undermine cataloguers’ confidence in their own judgement and their ability to influence decisions about catalogue design and other uses of metadata. adopting an evidence based approach is one way to counteract and break out of this marginalization. learning from experience in 2006 the library of congress (lc) made a decision to cease creating series authority records and providing controlled series access points. many aspects of this decision and its ramifications do not offer a good model of evidence based practice. in announcing the change, lc representatives cited adequate series access provided by more powerful evidence based library and information practice 2010, 5.4 118 indexing and keyword access (“lc to cease providing,” 2006), and an analysis that indicated “a substantial amount of cataloging time could be saved in the area of series control” (conser, 2006). these reasons may have been valid, but no evidence for them was provided at the time or subsequently. in response to the change, oclc adjusted its record loading procedures to ensure that controlled series access would not be lost through overlay by an lc record, and the program for cooperative cataloging (pcc) created guidelines for member libraries who decided to continue series authority control in their records. both oclc and pcc allowed members the option of providing series control or not in their records. to the author’s knowledge, no further investigation has been done on the effects of these decisions. data about the following questions would provide valuable evidence for decision-making by individual libraries and national agencies. • what operational savings were realized by lc? • what were the costs for oclc in changing its record loading procedures and subsequent record maintenance? • what are the additional costs for libraries that continued to control series headings and maintain series authority records? • what was the impact, if any, on the operations of libraries that adopted lc’s policy? • what was the impact, if any, on catalogue users? • what is the cost difference between distributed series authority control and centralised series control by lc and other national libraries? would the same difference apply to authority control of other access points such as authors? development of the conser standard record for serials in 2007 demonstrates a more successful application of several principles of evidencebased practice. • objectives were clearly defined (functionality, cost-effectiveness, and conformity to current standards) • testing was carried out in 14 libraries to assess the success of proposed guidelines in meeting the objectives. results provided evidence of improved functionality and costeffectiveness. • changes to the standard record were made as a result of test findings. • methodologies and findings were well documented and disseminated (access level record for serials working group, 2006). • further testing is being done after implementation of the guidelines (terrill, 2009). with this information, a decision on whether or not to adopt the conser standard record is much easier and more defensible. using an evidence based approach in cataloguing research and evidence cannot provide all the answers for the difficult decisions faced by cataloguers, but our professional judgement and accountability are strengthened by a critical and evidence based approach in our practice. despite the many barriers, we can develop the habit of using research and evidence in everyday decision-making, and by example encourage colleagues to do likewise. here are some suggested ways to use and support evidence based cataloguing: • follow discussion lists, blogs, etc. to find out about articles, conference presentations and reports that describe research and findings. contact authors for further data if appropriate. don’t limit yourself to research by and for librarians. especially relevant is the growing body of information on information-seeking behaviour. iii • many decisions about metadata are appropriately made at the national and international level. as a practitioner, be informed. evidence based library and information practice 2010, 5.4 119 provide input to bodies that develop and decide on metadata standards, and encourage evidence based approaches in their work. if possible, volunteer to serve. participate in testing of proposed new rules and standards, and share methodologies and findings. examples are the test of the bibco standard record devised, carried out and reported by cataloguers at the university of washington libraries (brooking, 2010) and the rda testing currently underway in many libraries. be involved. • as a practitioner, collaborate with researchers in designing and conducting relevant research. use available data such as statistics from your ils and opac. replicate research done by others in order to verify (or not) their findings. follow up on their suggestions for further research and methodology improvements. report on unsuccessful and unexpected methods and findings; these can be just as useful as evidence of successes. disseminate your results so others can benefit from both your findings and your methods. this can be done in a variety of formal and informal ways, e.g. articles, conference presentations, posting reports on library and personal websites or blogs, or listserv discussions. be vocal. • use the opportunities provided by development or analysis of metadata schema for local digital collections to incorporate evidence based decisions. be resourceful. • when making decisions such as local variations in metadata standards, assess them for tangible evidence of improved outcomes. use research findings to demonstrate to others in your institution the value added by cataloguing. let them know how your decisions have been influenced by evidence. resist decisionmaking based on anecdote, assumption, and temporary expediency. encourage and model evidence based practice as an approach, not a rigid process that inhibits innovation. be persistent. the organization of information through cataloguing is a core activity of librarianship. at a time when the value and sustainability of library cataloguing is being seriously questioned within our profession, the pursuit and application of empirical evidence offer a way forward to a rejuvenated cataloguing culture and practice. references access level record for serials working group. (2006, july 24). final report. retrieved october 8, 2010, from http://www.loc.gov/acq/conser/alrfina lreport.html brooking, d. m., kiegel, j. a., & schiff, a. l. (2010, june 14). test of the bibco standard record. retrieved october 8, 2010, from http://staffweb.lib.washington.edu/co mmittees/cpc/bsr case, d. o. (2007). looking for information: a survey of research on information seeking, needs and behavior (2nd ed.). amsterdam: elsevier/academic press. conser. (2006, april 27). joint bibco/conser meeting. retrieved october 8, 2010, from http://www.loc.gov/acq/conser/conop 2006.html gross, t., & taylor, a. g. (2005). what have we got to lose? the effect of controlled vocabulary on keyword searching results. college & research libraries , 66, 212-230. hill, j. s. (2008). is it worth it? management issues related to database quality. cataloging & classification quarterly , 46 (1), 5-26. ivey, r. t. (2009). perceptions of the future of cataloging: is the sky really falling? cataloging & classification quarterly , 47 (5), 464-482. evidence based library and information practice 2010, 5.4 120 jeng, l. h. (1997). knowledge, technology, and research in cataloging. cataloging & classification quarterly , 24 (1/2), 113127. lc to cease providing controlled series access. (2006, april 21). posted to autocat discussion list. library of congress. (n.d.). testing resource description and access (rda). retrieved oct. 8, 2010, from www.loc.gov/bibliographicfuture/rda/ library of congress working group on the future of bibliographic control. (2008, january 9). on the record. retrieved october 27, 2010, from http://www.loc.gov/bibliographicfuture/news/lcwg-ontherecord-jan08final.pdf news from alcts. (2009). alcts newsletter online, ano , 20 (4). oclc. (2009). online catalogs: what users and librarians want. retrieved october 8, 2010, from http://www.oclc.org/reports/onlinecat alogs/fullreport.pdf task force on cost/value assessment of bibliographic control, heads of technical services of large research libraries interest group, american library association. (2010). final report. retrieved october 8, 2010, from http://connect.ala.org/files/7981/costva luetaskforcereport2010_06_18_pdf_47 879.pdf terrill, l. j. (2009). a snapshot of early acceptance of the conser standard record in local catalogs. serials review , 35 (1), 16-27. footnotes i see for example the archives of the ngc4lib discussion list (http://listserv.nd.edu/archives/ngc4lib.html) under subject “cooperative cataloging rules announcement” and of the autocat discussion list (https://listserv.syr.edu/scripts/wa.exe?a0=autocat) under subject “rda and the library discovery experience”. ii see for example t. gross & taylor, a.g. (2005) “what have we got to lose? the effect of controlled vocabulary on keyword searching results” college & research libraries. 66, 212-230. iii see for example case, d.o. (2007) looking for information: a survey of research on information seeking, needs, and behavior (2nd ed.) amsterdam: elsevier/academic press. http://listserv.nd.edu/archives/ngc4lib.html� https://listserv.syr.edu/scripts/wa.exe?a0=autocat� / evidence based library and information practice commentary evidence summary   most canadian universities and colleges outside of quebec rely on fair dealing rather than access copyright   a review of: henderson, s., mcgreal, r., & vladimirschi, v. (2018). access copyright and fair dealing guidelines in higher educational institutions in canada: a survey. partnership: the canadian journal of library and information practice and research, 13(2), 1-37. https://doi.org/10.21083/partnership.v13i2.4147   reviewed by: thomas rouleau officer and manager of copyright services university of ottawa ottawa, ontario, canada email: thomas.rouleau@uottawa.ca   received: 30 apr. 2021                                                                  accepted:  20 july 2021      2021 rouleau. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29965     abstract   objective – to investigate the interpretations of fair dealing applied across canadian post-secondary educational institutions outside of quebec and to determine whether such institutions have a licence with access copyright.   design – descriptive/quantitative study.   setting – canadian post-secondary education sector, excluding quebec. subjects – a total of 159 canadian post-secondary institutions outside of quebec, including 75 universities and 84 colleges.   methods – a list of canadian post-secondary educational institutions outside of quebec was compiled. data from participants relating to the research objective—reliance on an access copyright licence or use and interpretation of fair dealing—was collected via internet searches or, if unavailable online, via direct telephone communication with participants.   main results – a majority of canadian post-secondary educational institutions outside of quebec, approximately 78% (124 institutions), did not have a licence with access copyright. the smaller the institution, the likelier it was to have an access copyright licence. this was in part linked to the fact that smaller institutions typically do not have staff specializing in copyright; savings from terminating access copyright licences (charged on a per student basis) would not justify the creation of such positions. regarding fair dealing, 18% of study participants based their approach on the supreme court of canada’s six-factor test (29 institutions), while 53% applied the fair dealing guidelines created by universities canada (85 institutions).   conclusion – most of the institutions studied did not have access copyright licences and were relying on fair dealing instead, suggesting a bellwether for the copyright climate in the canadian higher education sector towards fair dealing. institutions may benefit from a future national consensus regarding interpretations of fair dealing concepts.   commentary   this research paper relates to a somewhat niche area. in part, as described by the researchers (p. 5), this may be due to the fact that the topic of the paper is relatively new and restricted to canada. trosow (2013) provides a snapshot of the sector as of 2013, shortly after the transition from access copyright to fair dealing licences in canada. in addition to the proposed increase in amounts charged for a licence in 2010, and the heavy-handed auditing and reporting requirements imposed by access copyright discussed by the researchers (pp. 2-3), trosow (2013) supports findings from this study. namely that this trend is linked to an expanded interpretation of fair dealing resulting from several supreme court of canada decisions, as well as a 2012 legislative change in the canadian copyright act. more recently, zerkee (2017) and patterson (2017) specifically analyzed the roles of those managing copyright within post-secondary educational institutions but did not include any evaluation of access copyright and fair dealing preferences.   the next portion of this commentary loosely applies the “quantitative research evaluation checklist” (qrec) described by efron and ravid (2018, p. 105). there were no major ethical considerations for this study, and the results were clearly expressed. the significance of the results from this study could perhaps have been described in greater detail, but the study did quantitatively establish what was previously anecdotally surmised. the researchers provided helpful avenues for potential future research, suggesting that a qualitative approach investigating context and motivations would supplement their findings and provide openings for further discussion. for institutions where information about potential participants was not publicly available, only ten were contacted by telephone. the researchers acknowledge this as a limitation of the study (p. 18). overall, this article should be of particular interest to librarians who work in the canadian post-secondary education sector, since they are often responsible for copyright licensing and fair dealing (zerkee, 2017). for institutions without an access copyright licence that make full use of fair dealing, it should be comforting to see these results. for those relying on making payments to access copyright rather than fair dealing, however, it suggests that they may wish to re-evaluate their approach.   references   efron, s. e., & ravid, r. (2018). writing the literature review: a practical guide. guilford press.   patterson, e. (2017). the canadian university copyright specialist: a cross-canada selfie. partnership: the canadian journal of library and information practice and research, 11(2). https://doi.org/10.21083/partnership.v11i2.3856   trosow, s. e. (2013). fair dealing practices in the post-secondary education sector after the pentalogy. in geist, m. (ed.). the copyright pentalogy: how the supreme court of canada shook the foundations of canadian copyright law (pp. 213-234). university of ottawa press.   zerkee, j. (2017). approaches to copyright education for faculty in canada. partnership: the canadian journal of library and information practice and research, 11(2). https://doi.org/10.21083/partnership.v11i2.3794   research article   understanding eblip at an organizational level: an initial maturity model   clare thorpe associate director (library experience) university of southern queensland springfield, queensland, australia email: clare.thorpe@usq.edu.au   alisa howlett coordinator (evidence based practice) university of southern queensland springfield, queensland, australia email: alisa.howlett@usq.edu.au   received: 1 sept. 2019                                                                    accepted: 4 jan. 2020      2020 thorpe and howlett. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29639     abstract   objective existing research around evidence based practice in the lis (library and information science) professional context over the past two decades has captured the experience of individual practitioners, rather than the organization as a whole. current models of evidence based library and information practice (eblip) relate to, and apply predominantly to, individuals or specific scenarios. yet despite a growing demand from institutional and library leaders for evidence to demonstrate why investments in libraries should continue, little is known about how an organization can enhance its maturity in evidence based practice. this paper addresses this gap by seeking to understand what an evidence based university library looks like and answering the questions: how does a university library leader know the library’s service and practice is evidence based? how can a university library measure and progress its maturity in evidence based practice?   methods sixteen semi-structured interviews were conducted with library professionals employed at australian and new zealand university libraries. transcripts were analyzed using thematic analysis techniques. the interviews, combined with a literature review and environmental scan of evidence based practices in university libraries, informed the development of a draft capability maturity model as a framework for developing evidence based practice in university libraries.   results the model identifies and describes characteristics at five different levels of evidence based practice maturity from least mature (ad hoc/sporadic) to most mature (transforming). three dimensions of experience help to define the characteristics at each level of maturity and provide a framework to understand how a university library might develop its organizational capacity in evidence based library and information practice.   conclusion library leaders and practitioners will benefit from the model as they seek to identify and build upon their evidence based practice maturity, enabling more robust decision-making, a deeper understanding of their clients and demonstration of value and impact to their stakeholders.     introduction   over the past two decades, an understanding about evidence based practice in the lis (library and information science) professional context has been informed by research focused on the experience of individual practitioners, rather than the organization as a whole (koufogiannakis, 2013). current models of evidence based library and information practice (eblip) relate to, and apply predominantly to, individuals or processes. with growing demand from institutional and library leaders for evidence to demonstrate why investments in libraries should continue (baker & allden, 2017; council of australian university librarians, 2016), developing the skills of individuals may be inadequate. this paper addresses that gap by proposing a model for eblip maturity in university libraries.   a qualitative approach was taken to establish an initial understanding of what evidence based practice looks like at a whole-of-library level. overarching themes in the data, together with existing eblip literature have informed the development of a capability maturity model, which can be used as a framework for evaluating and progressing evidence based practice in university libraries. this paper describes the study approach, the resulting themes and proposed model. limitations, as well as future research opportunities, are also noted. by extending the existing understanding of eblip from an individual to an organizational level, this study addresses a need to tackle organizational factors related to cultivating evidence based practice in library and information science.   literature review   library and information science professionals across a range of sectors have adopted eblip as a way of working or being because they are driven by curiosity, aspire to do better, feel a professional responsibility, and want to keep up-to-date (booth, 2002; eldredge, 2000; partridge, edwards & thorpe, 2010; gillespie, 2014). understanding how lis professionals, including library leaders, experience evidence based practice has been the focus of a number of eblip studies, providing insight into the various factors that influence the ways in which evidence is conceived, encountered, and used to improve service delivery and decision making (gillespie, miller, partridge, bruce & howlett, 2017; koufogiannakis, 2012; luo, 2018; partridge et al., 2010). however, existing evidence based practice models which focus on individual practitioners or processes potentially create a false impression that evidence based practice is an ad hoc process or the responsibility of only a handful of individuals dotted across the organization (howlett & thorpe, 2018). the existing understanding of evidence based practice in the lis profession therefore needs to be broadened in order to achieve its aims.   regardless of how eblip is experienced as an individual phenomenon (thorpe, partridge & edwards, 2008), the variation of individual experiences is highly influenced by the workplace. in a study that sought to understand how university librarians use evidence in their decision-making, koufogiannakis (2015) identified five determinants, each “largely tied to the environment or context” (pp. 102-103). each of the five elements (organizational dynamics, time, personal outlook, education and training and information needs) can be an enabler or obstacle to using evidence in professional practice (koufogiannakis, 2015). however, koufogiannakis (2015) found that organizational dynamics and the workplace context were the largest obstacle faced by lis practitioners in applying evidence based practice principles and processes. hiller, kyrillidou, and self (2008) identified that the adoption of evidence based practices in libraries occurred where the organizational infrastructure and culture was more receptive and supportive of the acquisition, analysis, presentation, and use of evidence. library leadership and a customer-centered organizational culture were two of the enablers that supported the development of evidence based practices in libraries (hiller et al., 2008). urquhart (2018) stressed that implementing a culture of assessment, or evidence based practice, requires all library staff, not just managers, to appreciate the importance of the qualitative and quantitative evidence that they gather and to understand how such evidence can help demonstrate the value of library services. gillespie, partridge, bruce, and howlett (2016) found that a shared organizational approach to evidence based practice, whether intentionally sought or not, guided and developed library services and collections, helped to build relationships with its clients, and ensured that services aligned with clients’ needs through an attitude and culture of continuous improvement.   aligned with the internal driver to build a collaborative, evidence based library culture, is the external demand from university leadership for evidence based arguments to demonstrate why investments in university libraries should continue (baker & allden, 2017; council of australian university librarians, 2016). library directors have been found to use a variety of evidence sources to demonstrate the library’s value within strategic planning activities (lembinen, 2018; newton miller, 2018). harland, stewart, and bruce (2017) reported that university library directors stressed the necessity of evidence in increasingly complex decision-making processes and in using evidence based value measurements to articulate the library's contribution to their university. the value of the library is increasingly judged by its demonstrated ability to help students learn and researchers research (madsen & hurst, 2018). the council of australian university librarians identified an aspirational long-term goal to nurture a culture of evidence based thinking and communication across university libraries in australia (owen, peasley & paton, 2017).  if being evidence based provides both intrinsic benefits to the library’s organizational culture and extrinsic benefits in demonstrating the library’s value to the university, then eblip adoption cannot be left up to individual practitioners. library leaders need to be able to evaluate the evidence based practice within their library to address both the internal and external drivers.   looking more broadly at the literature on library assessment (a term often used synonymously with eblip), many libraries have adopted tools, such as balanced scorecards, values scorecards, performance indicator frameworks, benchmarking, frameworks, and typologies, to demonstrate their value and impact and to assess cultural change (urquhart, 2018).  madsen and hurst (2018) proposed a library assessment success model that places the library as an organization within the eco-system of higher education. their model focused on how well the library aligns with the strategic goals of the parent entity and rated a library’s performance along an eight-point scale (madsen & hurst, 2018). maturity models have emerged in the information science literature and have been used as tools for internal and/or external benchmarking, identifying future improvement and providing guidelines for organizational development and growth (lasrado, andersen, & vatrapu, 2015). maturity models provide a framework for organization-wide action, implementation, and evaluation (nelson, clarke, stoodley, & creagh, 2015). maturity models facilitate the organization’s ability to actively and continuously identify its own priorities and quality standards, and to enact systems that support continuous improvement (marshall, 2010). at their simplest, maturity models describe a path from initial state to highest state of maturity in an organization’s culture and capabilities, usually focused on a specific set of competencies (wademan, spuches, & doughty, 2007). capability maturity models commonly have five levels and show a continuum of improvements around a particular competency (nelson et al., 2015). two recently published maturity models correlate to eblip, specifically the use of evidence and assessment processes. wilson’s quality maturity model (2015) provided a framework for assessing and progressing libraries on a journey towards a ubiquitous culture of quality across five levels of maturity (initial, repeatable, defined, managed, continuous) and included approaches and tools that are also applied in evidence based practice. the library assessment capability maturity model proposed by hart and amos (2018) can be used as a benchmarking or self-review tool to measure a library’s effectiveness in completing library assessment activities. this model adopted a similar five level scale of maturity, replacing continuous with optimised at its most mature stage (hart & amos, 2018).  while these existing models relate to aspects of evidence based practice, neither directly addresses the question of how a library can grow and mature eblip as a specific organizational competency within libraries.   aims   as an organization, libraries need to build robust capabilities and strategic approaches in evidence based practice to respond to a constantly changing environment and the demands and needs of stakeholders. this task cannot be left to individual practitioners. responsibility for implementing and developing evidence based practice capability needs to be shared and everyone has a role (hallam & partridge, 2006). in order for eblip to achieve its aims, this study broadens the existing understanding and perspective from individuals to whole library organizations.   what an evidence based library looks like is currently unknown. the aim of this study was to answer the question: how does a university library leader know that their library’s service and practice is evidence based? with a capability maturity model identified as our desired outcome, the authors sought to develop a framework that would help university library leaders measure and progress their organization’s maturity in evidence based practice.   methods   a qualitative approach with semi-structured interviews was used to explore the research question and develop a detailed, initial understanding of evidence based practice at the whole-of-library level (creswell, 2012). sixteen semi-structured interviews were conducted with library staff across australia and new zealand. participant recruitment used a combined convenience and purposive sampling approach. two criteria were considered when recruiting participants – they must be employed in a role at librarian (professional) level or above in a university library; and be available during the six-week data collection period. participants did not need to have a leadership role within their organization. full ethics clearance was obtained from the university of southern queensland ethics committee.   as active members of the australian university library community, the researchers drew upon their own networks to recruit participants. the researchers aimed to capture eblip experiences and perspectives from a range of library roles and functions to bring together a holistic, day-to-day understanding of what eblip looks like across a whole library organization. this was considered important in order to answer the research question and for university library leaders to understand eblip maturity and how eblip is integrated (or not integrated) in ways of working.   fourteen women and two men were interviewed. most participants were employed at australian universities with one person working at a new zealand institution. participants were employed as:   liaison/subject/outreach librarians (5) team leaders/managers (5) associate directors/deputy university librarians (4) directors/university librarians (2)   interviewees reflected on their own experiences with evidence based practice and the level of evidence based practice maturity demonstrated by their library. interviewees were also asked to describe the characteristics of an evidence based library. appendix a provides a list of sample interview questions that guided discussion.   interviews recordings were transcribed verbatim and analyzed using a thematic approach (braun & clarke, 2006). given that this study approached new ground in examining eblip from a whole-of-library perspective, the iterative approach described by braun and clarke (2006) offered the opportunity for the researchers to immerse in, and actively engage with, the data, relate the data to existing eblip research, and to develop an interpretation of the data through the six-step coding process. the analysis provided a rich description of the data about what evidence based practice might look like at a whole-of-library level. refining the codes enabled the identification of overarching themes that ultimately became the elements of the maturity model.   results   variation of ebp experience   variation in participant experiences and observations of evidence based practice within their libraries, as well as personal conceptions of evidence, evidence based practice, and what an evidence based library might look like, appeared along a spectrum. this spectrum captured varying extents to which participants’ libraries engaged, what evidence sources were valued (or not) and used, who is involved, and the methods and circumstances in which evidence is applied to practice and service delivery. how evidence and evidence based practice is conceptualized, the presence (or absence) of drivers for the evidence based practice process and evidence collection, and the circumstances in which evidence is applied to practice and service delivery contributed to the variation within the model. interviewee’s experiences of evidence based practice were influenced by the extent to which the whole library was involved or engaged in the approach. personal conceptions of what an evidence based library might look like also were on a continuum. variation and influences found in the interview data form the basis of the maturity model.   figure 1 the eblip capability maturity model.   the eblip capability maturity model   through the model, the researchers propose five levels or tiers of maturity from tier 1 (least mature) to tier 5 (most mature). the five tiers are: ad hoc/sporadic, justifying, emerging, experimenting, transforming. each tier is associated with awareness structures that involve different interpretations of the internal and external working environments, how evidence based approaches are planned and implemented, and how the category of experience perceives the role of evidence in decision-making. all libraries in the interview sample contributed data on collections, staffing, expenditures, library services, and library and university characteristics to the annual benchmarking activity facilitated by the council of australian university librarians (caul, 2018). therefore, as all the libraries were engaged in some form of evidence based practice, tier 0 (inactive state) was excluded from the model as not relevant to australian and new zealand university libraries.   tier 1: ad hoc/sporadic   evidence based practice activities are undertaken as part of the traditional collecting role of the library – in this case, collecting data. however, eblip processes do not move much beyond collecting statistics as the main source of evidence. there is little or no critical analysis, reflection or communication. eblip considered not relevant by staff or leadership. evidence may be overruled or sidelined by opinion during decision-making.   interview 10: “when i first came on board a lot of decisions were just made on people’s personal opinion, and that still happens to quite an extent, actually.”   tier 2: justifying   evidence based practice activities are used to justify actions taken and to demonstrate busyness across the organization. eblip processes are focused on collecting data or statistics with little consideration of alternative sources of evidence. there is a dependence on systems generated data to build a local data repository. eblip is acknowledged but only in the context of collecting statistics for reporting against metrics or to justify decisions already made.     interview 13: “i hear a lot of that justifying or ‘the evidence is telling us this, but that’s because blah, blah and blah’.  so then we don’t have to do anything about it, because we can sort of justify it.”   tier 3: emerging   evidence based practice is beginning to be applied; however, its application is limited to specific activities, such as projects. evidence is gathered, assessed, and applied to instigate a change or within the context of a project. the evidence collected may have limited future application due to the project outcomes focus of the eblip process.   interview 7: “every project, you’ve got to make sure that you have a sense of what is happening and possible solutions and how things have been applied and how you might apply them.”   tier 4: experimenting   evidence based practice seen as a desirable and attainable organizational goal. evidence informs decision making across a range of activities and functions. evidence is gathered to inform strategic planning activities or to improve the library’s services. staff are supported in developing their skills in evidence based practice and are encouraged to experiment with a range of approaches and methodologies.   interview 4: “that would mean taking kind of a pause and really thinking about what you were trying to solve, and thinking about the question or purpose of what you’re trying to do, and then what kind of evidence you could find. i feel like i use that process for most things because it makes me feel more confident to actually make a change or do something.”   tier 5: transforming   evidence based practice is underpins the day-to-day activity of the library. eblip is ubiquitously adopted as a way of working. evidence is gathered with a clear purpose that aligns with the university’s strategic goals. the library works together to develop an evidence base that is verified, trustworthy, contextualized, fit for purpose, available, and ready for use. evidence generates sophisticated insights that are used to communicate with influence. staff and leadership demonstrate an eblip mindset. evidence based practice empowers and transforms the library across all activities, enabling an agile, responsive, and creative organization.   interview 11: “i see it in terms of the library conducting itself, and being managed, and being driven forward, on the basis of evidence based practice. something approximating a kind of an ethos of the way people practice things and the way the library is managed.”   discussion   the proposed maturity model progresses existing eblip understanding by acknowledging this variation of experience, as well as putting workplace-related influences within the context of being evidence based as an organization. the eblip capability maturity model is the product of information from empirical and theoretical literature integrated with information from the semi-structured interviews. from the interview transcripts and existing eblip research, the differences between maturity at each tier emerged through the lens of three dimensions that identified how evidence based practice manifests in a university library setting. these dimensions have their foundations in existing eblip models. the dimensions are process, engagement, and evidence.   process   interviewees described the varying extents to which an evidence based practice process was applied within their libraries, whether this be not at all, on a one-off or semi-regular basis, or integrated into day-to-day ways of working. a process may be reactive or proactive, meaning that the purpose of gathering and applying evidence may or may not be thought about from the onset. interview data also revealed where the focus of an evidence based practice process might be at different maturity stages. for example, some participants described collection methods in detail. others focused on the process associated with making evidence usable or presenting it in a communicable format. experiences of, and exposure to, evidence based practice within their libraries indicated capability limitations to varying degrees in assessing and applying evidence to particular needs or practice questions. how the eblip process manifests correlated with the cyclical eblip model, sometimes referred to as the 5as model, that koufogiannakis and brettle (2016) proposed. a sixth element of communication was also included within the process dimension.   table 1 process dimension eblip capability maturity model process tier 1: ad hoc/ sporadic tier 2: justifying tier 3: emerging tier 4: experimenting tier 5: transforming articulate the need for evidence is not defined. evidence is collected to justify actions taken or to “show-off” the library’s activities. needs are defined within the context of a project or specific activity; or to instigate a change. eblip is used to identify gaps or problems in service delivery. proactive approaches are taken to understand and articulate problems/needs. assemble evidence may be collected due to tradition or to satisfy external demands. reliant on quantitative data available from library systems. evidence is collected for a specific task from local or research sources. evidence is collected to support service improvement. evidence is collected with clear strategic or operational purpose. assess if evidence is collected, it is not evaluated. evidence is manipulated to fit the required context and used to justify actions taken or not taken. evidence is assessed against its relevance and applicability to the local situation. evidence is assessed against the articulated need. sophisticated, contextual insights are drawn from evidence. apply/agree evidence is not used for decision-making or strategic planning. evidence has a limited role in informing decision making. evidence informs decisions for specific projects/activities. evidence is considered when making decisions and determining the strategic direction. evidence underpins how the library operates and determines its strategic direction. adapt evidence is left unused. it does not inform decisions made. evidence is used to justify the action (or lack of action) taken.  evidence gathered has limited future application or repeatability. starting to apply an iterative approach and to develop a local evidence base. continuous cycle of improvement. eblip enables agility, creativity, and responsiveness. communicate evidence is not used to communicate impact or value. raw/descriptive data is reported to demonstrate activity rather than value or impact. focus is on communicating outcomes from projects. focus is on communicating evidence to influence decision-making. evidence is used to demonstrate value and impact.   table 2 engagement dimension eblip capability maturity model engagement   tier 1: ad hoc/ sporadic tier 2: justifying tier 3: emerging tier 4: experimenting tier 5: transforming shared understanding eblip is seen as unrealistic or unattainable. it is not an organizational priority. eblip is synonymous with data collection, to demonstrate the “busyness” of the library. eblip is applied when implementing changes or projects. eblip is seen as a desirable goal to support service improvement and strategic planning cycles.   eblip is applied as a mindset that underpins the day-to-day activity of the library. responsibility no one/only as required or directed. individuals who are responsible for data collection. eblip is seen as an additional task to be completed by project managers and teams. sits with or is led by a dedicated staff member or team who have an eblip focus or role. shared across all staff and teams. “everybody would have it as part of their role.” role of leaders leaders do not see the benefit of eblip. it is experienced as “not relevant". leaders are reactive to the environment, as required by stakeholders. eblip is experienced as “a weapon". leaders require evidence to support project work and change proposals. eblip is experienced as “learning from research". leaders require evidence to support decision-making and service improvements.  eblip is experienced as “service improvement." leaders reinforce a culture of eblip. it is experienced as “a way of being". staff development not seen as a staff development priority. staff development priorities limited to data literacy. staff development as required to achieve project outcomes. focus on using evidence and developing research skills. staff are encouraged to experiment with different research methods. staff are supported to develop a deep understanding of eblip as it applies to their role.     table 3 evidence dimension eblip capability maturity model evidence tier 1: ad hoc/ sporadic tier 2: justifying tier 3: emerging tier 4: experimenting tier 5: transforming sources may occasionally recognize or acknowledge the value of local or research evidence. reliant on evidence (data) that is already accessible or routinely collected/system generated. local evidence is valued. considers best practice or benchmarking with local data to support a specific project or need, using a mix of research and local evidence. evidence from a range of sources is valued as a way to identify gaps and opportunities.   evidence is sourced with critical intent from a range of valid sources. evidence is verified, trustworthy, contextualized, fit for purpose, and ready for use. purpose opinion is valued more than evidence in decision-making. performance metrics focus. systems-generated data builds a database of statistics that is used infrequently in decision-making. project/activity based focus. evidence has a limited purpose, to inform decisions within the context of a project or activity. service improvement focus. evidence informs decision making across a range of activities and functions. holistic focus on building and maintaining an evidence base for the library. evidence empowers the library to make decisions across all activities and functions.   engagement   this element involves how the library as a whole engages with evidence based practice; promotes, prioritizes and enables evidence based practice approaches and capability. rather than focusing on organizational culture as a label for this element, engagement more accurately described participants’ experiences of how widely dispersed and focused their libraries were in using evidence based practice approaches. engaging in evidence based practice included the extent to which library staff were supported to develop their capabilities and skills; whether leaders explicitly prioritized evidence based practice as a way of working, and whether there was a shared understanding of how evidence based practice approaches were used to improve services and practice. some participants also commented on whose responsibility it was to drive engagement in evidence based practice. how leaders understand and sponsor eblip as way of working closely aligned with the five categories of experience identified by partridge, edwards, and thorpe (2010).   evidence   participants shared which evidence sources were used within their library to make decisions and improve practice the types of evidence involved in a library’s process, how evidence is identified and perceived, and an awareness of the limitations of evidence to different situations and contexts. though closely linked to process, this element describes an awareness of, and capability to identify, gather, and apply appropriate evidence to practice and service delivery. at the more mature end of the spectrum, a small number of participants were able to demonstrate an understanding of the limitations of applying and using different types of evidence. the types of evidence identified by participants indicated those most valued in making decisions within their libraries. the maturity model does not aim to prescribe a hierarchy of evidence, or to suggest higher levels of maturity use one source over another. rather, interview data indicated that an awareness and ability to use different types and combinations of evidence is of higher maturity. the variety of evidence sources (local, research, professional knowledge) that were identified from the interviews draws on koufogiannakis’ research (2011).   limitations and future research   the maturity model presented in this paper addresses a gap in the current understanding of evidence based library and information practice by broadening the perspective from individual practitioners to whole library organizations. in doing this, the scope of this study was refined to focus on university libraries, particularly in australia and new zealand. therefore, this maturity model may not be representative of evidence-base practice in other types of libraries such as public, school, or special libraries. it is anticipated that engagement with evidence-based practice is likely to differ between sectors, such as health and medical and academic libraries. further study in other library contexts and countries would help validate and strengthen the maturity model and its application to a range of library and information organizational contexts.   the challenges of developing an instrument to measure maturity in organizations include:   ·         how to measure the distance between maturity levels ·         what is the scale of measurement ·         how to calculate the overall maturity (lasrado et al., 2015).   at this stage of the research, the authors have yet to address these questions. validation of the maturity model is required. lasrado, andersen, and vatrapu (2015) noted that validation is usually undertaken following the publication of a maturity model. it is the authors’ intent to continue developing and validating the proposed model. implementation practices that can be repeated, measured, and continuously improved to create organizational change in eblip maturity are also yet to be developed (wademan et al., 2007). the authors intend to further develop the eblip capability maturity model to include a self-assessment matrix to help university library leaders measure their organization’s maturity status. resources, case studies, and recommended implementation and institutionalization practices for how to grow a library’s eblip maturity are also planned.   conclusion   as university libraries face increased scrutiny of their role and value to the institution, responsibility for evidence based practice and approaches to service delivery and communication lies with the whole library organization, not just individual practitioners. additionally, individual practitioners need organizational supports to enable evidence based practice and related capabilities. the eblip capability maturity model developed from this research responds to this challenge and promotes the importance of building evidence based practice capabilities at a broader, organizational scale. this is key to the ongoing sustainability of eblip, and the library itself, as it responds to its environment.   variation of experience, as originally found by partridge, edwards, and thorpe (2010), appear to still ring true to evidence based practice experiences at an organizational level, and therefore validates a capability maturity model as an appropriate framework for library leaders to evaluate evidence based practice within their libraries. this study also addresses what koufogiannakis (2015) found to be the largest obstacle to evidence based practice in academic libraries – the workplace context – and builds upon existing research and literature about the influencing factors and responsibilities that are involved in building a culture of evidence based practice (hallam & partridge, 2006; howlett, 2018). koufogiannakis’ (2012) five determinants of evidence use in academic libraries exist within this model, though elevated to an organizational level.   a culture of evidence based practice within an organizational or workplace context requires a shared approach and requires all library staff (hallam & partridge, 2006; urquhart, 2018). library leadership in evidence based practice is also essential to achieving its aims. by taking eblip a step forward in broadening the existing understanding and its models from the individual to organizational level, the maturity model developed from this research brings to light and captures how might a library leader know and measure the extent to which the library’s service and practice is evidence based. the model provides a framework by which library leaders can determine how mature their library is, or needs to be, and to identify characteristics of maturity for individuals, teams, and organizations to aspire to.   library leaders and practitioners will benefit from the model as they seek to identify and build upon their evidence based practice maturity, enabling more robust decision-making, a deeper understanding of their clients, and demonstration of value and impact to their stakeholders. future development and validation of the model will be undertaken to create tools which will provide practical application of the eblip capability maturity model so that libraries can grow and mature eblip as a specific organizational competency to the benefit of clients, staff, and stakeholders.   acknowledgements the authors would like to acknowledge the library professionals who willingly participated in this research. this research was conducted on the lands of the jagera, yuggera and ugarapul peoples of the springfield region, queensland, australia.   references   baker, d., & allden, a. (2017). leading libraries: the view from above. london: sconul. retrieved from https://www.sconul.ac.uk/publication/the-view-from-above    booth, a. (2002). from ebm to ebl: two steps forward or one step back? medical reference services quarterly, 21(3), 51-64. https://doi.org/10.1300/j115v21n03_04      braun, v., & clarke, v. (2006). using thematic analysis in psychology. qualitative research in psychology, 3(2), 77-101. https://doi.org/10.1191/1478088706qp063oa    council of australian university librarians. (2016). principles and guidelines for australian higher education libraries. retrieved from http://www.caul.edu.au/content/upload/files/best-practice/principlesguidelines2016public.pdf   council of australian university librarians. (2018). caul online statistics home. retrieved january 4, 2020 from https://statistics.caul.edu.au/index.php   creswell, j. w. (2012). educational research: planning, conducting and evaluating quantitative and qualitative research. 4th ed. boston: pearson.   eldredge, j. (2000). evidence-based librarianship: an overview. bulletin of the medical library association, 88(4), 289-302. retrieved from https://www.ncbi.nlm.nih.gov/pmc/journals/430/#mlab   gillespie, a. (2014). untangling the evidence: introducing an empirical model for evidence-based library and information practice. information research, 19(3). retrieved from http://www.informationr.net/ir/19-3/paper632   gillespie, a., miller, f., partridge, h., bruce, c., & howlett, a. (2017). what do australian library and information professionals experience as evidence? evidence based library & information practice, 12(41). https://doi.org/10.18438/b8r645   gillespie, a., partridge, h., bruce, c., & howlett, a. (2016). the experience of evidence-based practice in an australian public library: an ethnography. information research, 21(4). retrieved from http://www.informationr.net/ir/21-4/paper730.html    hallam, g., & partridge, h. (2006). evidence based library and information practice: whose responsibility is it anyway? evidence based library and information practice, 1(3), 88-94. https://doi.org/10.18438/b8t880   harland, f., stewart, g., & bruce, c. (2017). ensuring the academic library's relevance to stakeholders: the role of the library director. the journal of academic librarianship, 43(5), 397-408. https://doi.org/10.1016/j.acalib.2017.06.009     hart, s., & amos, h. (2018). the library assessment capability maturity model: a means of optimizing how libraries measure effectiveness. evidence based library & information practice, 13(4), 31-49. https://doi.org/10.18438/eblip29471     hiller, s., kyrillidou, m., & self, j. (2008). when the evidence is not enough. performance measurement and metrics, 9(3), 223-230.  https://doi.org/10.1108/14678040810928444   howlett, a. (2018). time to move eblip forward with an organizational lens. evidence based library & information practice, 13(3), 74-80. https://doi.org/10.18438/eblip29491   howlett, a., & thorpe, c. (2018, august). ‘it’s what we do here’: embedding evidence-based practice at usq library. paper presented at the asia-pacific library and information conference, gold coast, australia. retrieved from https://eprints.usq.edu.au/34729/   koufogiannakis, d. (2011). considering the place of practice-based evidence within evidence based library and information practice. library and information research, 35(111), 41-58. https://doi.org/10.29173/lirg486   koufogiannakis, d. (2012). academic librarians' conception and use of evidence sources in practice. evidence based library and information practice, 7(4), 5-24. https://doi.org/10.18438/b8jc8j   koufogiannakis, d. (2013). academic librarians use evidence for convincing: a qualitative study. sage open, 3(2). https://doi.org/10.1177/2158244013490708   koufogiannakis, d. (2015). determinants of evidence use in academic librarian decision making. college & research libraries, 76(1), 100-114. https://doi.org/10.5860/crl.76.1.100   koufogiannakis, d., & brettle, a. (eds.). (2016). being evidence based in library and information practice. london: facet publishing.   lasrado, l., andersen, k. n., & vatrapu, r. k. (2015). maturity models development in is research: a literature review. iris selected papers of the information systems research seminar in scandinavia, 6(6). retrieved from http://aisel.aisnet.org/cgi/viewcontent.cgi?article=1005&context=iris2015    lembinen, l. (2018). academic library directors’ strategic decision-making process. liber quarterly, 27(2), 1-21. https://doi.org/10.18352/lq.10224   luo, l. (2018). experiencing evidence-based library and information practice (eblip): academic librarians' perspective. college & research libraries, 79(4), 554-567. https://doi.org/10.5860/crl.79.4.554   madsen, c., & hurst, m. (2018). contextualizing library assessment within a broader ecosystem: proposed models for linking the strategic to the micro. performance measurement and metrics, 19(1), 18-29. https://doi.org/10.1108/pmm-09-2017-0042    marshall, s. (2010). a quality framework for continuous improvement of e-learning: the e-learning maturity model. journal of distance education, 24(1), 143-166. https://files.eric.ed.gov/fulltext/ej892382.pdf     miller, f., partridge, h., bruce, c., yates, c., & howlett, a. (2017). how academic librarians experience evidence-based practice: a grounded theory model. library & information science research, 39(2), 124-130. https://doi.org/10.1016/j.lisr.2017.04.003   nelson, k., clarke, j., stoodley, i., & creagh, t. (2015). using a capability maturity model to build on the generational approach to student engagement practices. higher education research & development, 34(2), 351-367. https://doi.org/10.1080/07294360.2014.956694   newton miller, l. (2018). university community engagement and the strategic planning process. evidence based library and information practice, 13(1), 4-17. https://doi.org/10.18438/eblip29351   owen, s., peasley, j., & paton, b. (2017). principles and guidelines for australian higher education libraries: capturing value. 2nd annual teqsa conference (p. 146-158). melbourne, australia. retrieved from https://www.hes.edu.au/sites/default/files/uploadedcontent/field_f_content_file/teqsa_2017_conference_proceedings.pdf    partridge, h., edwards, s. l., & thorpe, c. (2010). evidence-based practice: information professionals' experience of information literacy in the workplace. in a. lloyd & s. talja (eds.), practising information literacy: bringing theories of learning, practice and information literacy together. wagga wagga, new south wales: centre for information studies.   thorpe, c., partridge, h., & edwards, s. l. (2008). are library and information professionals ready for evidence based practice. paper presented at the alia biennial conference: dreaming 08, alice springs, australia. retrieved from https://eprints.qut.edu.au/28370/1/28370.pdf     urquhart, c. (2018). principles and practice in impact assessment for academic libraries. information and learning science, 119(1/2), 121-134. https://doi.org/10.1108/ils-06-2017-0053    wademan, m. r., spuches, c. m. & doughty, p. l. (2007). the people capability maturity model: its approach and potential to improve workforce performance. performance management quarterly, 20(1), 97-124. https://doi.org/10.1111/j.1937-8327.2007.tb00434.x   wilson, f. (2015). the quality maturity model: your roadmap to a culture of quality. library management, 36(3), 258-267. https://doi.org/10.1108/lm-09-2014-0102     appendix a sample interview questions   can you tell me about any challenges you or the library have faced in making decisions about the library’s services? can you identify any evidence that might have helped resolve the challenge? does the library regularly produce or revise a strategic plan, its goals and objectives? if so, can you tell how the library does this? if not, can you tell me why? can you tell me about any processes or evidence that is collected and analysed to evaluate: ·         the library’s services and programs? ·         the collections?  ·         the achievement of the library’s goals and objectives?   are there any routine or regular processes in place to collect evidence related to the library’s services and programs? if so, can you describe them? are there any routine or regular processes in place to analyse and report on the library’s services and programs? if so, can you describe them? (e.g. reporting schedule)  how does the library communicate its performance, value and impact to its stakeholders? do you think this is an area where the library could improve?   what does evidence based practice mean: ·         to you? ·         to the library?   what is ‘evidence’?   what potential benefits do you think evidence based practice has, or can have, to your library?   how confident do you feel the library can, or is operating in an evidence based way? how do you think the library can improve in being evidence based?       evidence summary   graduate assistants trained in reference may not consistently apply reference interview and instructional strategies in reference interactions   a review of: canuel, r., hervieux, s., bergsten, v., brault, a., & burke, r. (2019). developing and assessing a graduate student reference service. reference services review, 47(4), 527–543. https://doi.org/10.1108/rsr-06-2019-0041   reviewed by: sarah bartlett schroeder research & instruction librarian university of washington bothell/cascadia college campus library bothell, washington, united states of america email: sarahkb6@uw.edu   received: 5 mar. 2020                                                               accepted:  11 sept. 2020      2020 schroeder. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29750     abstract   objective – to evaluate the effectiveness of a reference training program for graduate student employees that seeks to encourage use of reference interview and instruction techniques in virtual and in-person reference interactions.   design – naturalistic observation with qualitative content analysis.   setting – a large, public research university in montreal, canada.   subjects – three graduate students in library and information science employed by the university library to provide virtual and in-person reference services.   methods – after completing a training program, the three participants provided virtual and in-person reference training for two consecutive semesters. they self-recorded their desk interactions in a google form. these self-reports, along with their online chat transcripts from questionpoint, were the subject of this study’s analysis. focusing on the questionpoint data, the authors coded the transcripts from these participants’ online reference interactions to reflect the presence or absence of a reference interview and various instructional techniques in their responses to patrons. also, all in-person and virtual questions were examined and categorized as being either transactional or reference questions. reference questions were further categorized as basic, intermediate, or advanced questions.   main results – of the chat transcripts analyzed, 49% were classified as containing reference questions rather than transactional questions. at the desk, 21.9% of interactions were coded as reference questions. taking the two semesters together, 232 of 282 virtual reference questions were considered basic, while 41 were labelled intermediate, and 9 classified as advanced. similarly, of 136 desk reference questions, 120 were classified as basic, 14 as intermediate, and 2 as advanced. in their coding of chat transcripts, researchers indicated whether the interaction contained no reference interview, a partial reference interview, or a complete reference interview. virtual chat transcripts from both fall and winter semesters showed that no reference interview took place in 77.3% of interactions. authors noted evidence of partial reference interviews in 19.3% of fall transcripts and 21.5% of winter transcripts. complete reference interviews took place in 3.4% of fall and 1.2% of winter transcripts. additionally, authors found that 65.5% of chat transcripts contained elements of instruction, with modelling and resource suggestion being the most prevalent forms.    conclusion – because the graduate students used complete or partial reference interviews in a small number of their virtual reference questions, the authors of this study determined that more emphasis ought to be placed on reference interviews, particularly virtual reference interactions, in future training programs. graduate students employed instructional strategies in observed virtual reference interactions, a promising trend.   commentary   over the past several years, a number of studies have considered how best to train students to provide online and in-person reference services. a university of michigan paper described an online and in-person hybrid training process (wetli, 2019). librarians at another canadian university completed a content analysis of online chat transcripts and patron surveys to determine the quality of student reference interactions (barrett & greenberg, 2018). in one study, researchers compared the quality of chat transcripts with student employees to those with librarians (lux & rich, 2016). this study uses a similar content analysis methodology.   this summary uses reliant, a tool designed to aid librarians in appraising evaluations of education and training programs in four areas: design, educational context, results, and relevance (koufogiannakis et al., 2006).   regarding educational context, it is unclear what prior relevant experience the three study participants brought to their roles, but it is clear what setting they are working in and that they received a mix of in-person instruction and observation hours with practicing librarians during the training program. in terms of results, the data do accurately show that the participants used reference interview and instructional tactics after training, if only for online interactions. while the authors based their coding techniques on previously published research and drew specifically from rusa and a 2008 study by desai and graves to inform their methodology, they did not specify whether they completed coding separately or together and, if the former, what level of overlap they achieved. this could influence the accuracy of the research design. adding such details together with comparison data, such as an analysis of librarian chat responses or questions participants answered before completing training, would have added richness to the analysis by offering a comparative framework for determining the significance of the data.   regarding relevance, the authors’ findings are interpretive and might not apply to all settings. they see a need to include greater emphasis on reference interview skills in future training, despite offering several reasons why a reference interview might not be appropriate for every question. they note briefly that some questions, such as known-item searches, may reasonably be answered without using reference interview techniques, but do not share how many interactions with advanced questions, for example, included partial or complete reference interviews, as compared to basic questions. future studies could incorporate and elaborate upon these topics.   as the authors note, libraries at colleges and universities with lis programs do commonly employ graduate students to provide reference services. thus, this study may offer librarians at such institutions ideas for improving their own training programs. more broadly, the training content pertaining to reference interviews and instructional techniques may be useful for training librarians who are recent graduates or have minimal reference experience, especially in online environments. the content analysis methodology is valuable to other researchers examining trends in reference services.   references   barrett, k. & greenberg, a. (2018). student-staffed virtual reference services: how to meet the training challenge. journal of library & information services in distance learning, 12(3–4), 101–119. https://doi.org/10.1080/1533290x.2018.1498620   desai, c. m., & graves, s. j. (2008). cyberspace or face-to-face: the teachable moment and changing reference mediums. reference & user services quarterly, 47(3), 242–255. https://opensiuc.lib.siu.edu/morris_articles/18/   koufogiannakis, d., booth, a., & brettle, a. (2006). reliant: reader’s guide to the literature on interventions addressing the need of education and training. library and information research, 30(94), 44–51. https://doi.org/10.29173/lirg271   lux, v. j., & rich, l. (2016). can student assistants effectively provide chat reference services? student transcripts vs. librarian transcripts. internet reference services quarterly, 21(3–4), 115–139. https://doi.org/10.1080/10875301.2016.1248585   wetli, a. (2019). training temporary reference staff for maximized learning: a case study. journal of academic librarianship, 45(5), article 102032. https://doi.org/10.1016/j.acalib.2019.04.009   microsoft word art_byrne.doc evidence based library and information practice 2007, 2:1  32 evidence based library and information practice     feature article    a statistical primer: understanding descriptive and inferential statistics      gillian byrne  information services librarian  queen elizabeth ii library  memorial university of newfoundland  st. john’s, nl , canada  email: gbyrne@mun.ca      received: 13 december 2006    accepted: 08 february 2007      © 2007 byrne. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    abstract    as libraries and librarians move more towards evidence‐based decision making, the data  being generated in libraries is growing. understanding the basics of statistical analysis is  crucial for evidence‐based practice (ebp), in order to correctly design and analyze research  as well as to evaluate the research of others. this article covers the fundamentals of  descriptive and inferential statistics, from hypothesis construction to sampling to common  statistical techniques including chi‐square, correlation, and analysis of variance (anova).      introduction  much of the research done by librarians,  from bibliometrics to surveys to usability  testing, requires the measurement of certain  factors.  this measurement results in  numbers, or data, being collected, which  must then be analyzed using quantitative  research methods. a basic understanding of  statistical techniques is essential to properly  designing research, as well as accurately  evaluating the research of others.   this paper will introduce basic statistical  principles, such as hypothesis construction  and sampling, as well as descriptive and  inferential statistical techniques. descriptive  statistics describe, or summarize, data, while  inferential statistics use methods to infer  conclusions about a population from a  sample.    in order to illustrate the techniques being  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:1  33                great job         lousy job                            if you accept the job    have a great experience  waste time & effort    if you decline the job  waste an opportunity  avoid wasting time & effort      figure 1. illustration of type i & ii errors.      described here, an example of a fictional  article will be used.  entitled perceptions of  evidence‐based practice: a survey of canadian  librarians, this article uses various  quantitative methods to determine how  canadian librarians feel about evidence‐ based practice (ebp).  it is important to note  that this article, and the statistics derived  from it, is entirely fictional.     hypothesis  hypotheses can be defined as “untested  statements that specify a relationship  between two or more variables” (nardi 36).  in social sciences research, hypotheses are  often phrased as research questions. in plain  language, hypotheses are statements of  what you want to prove (or disprove) in  your study.  many hypotheses can be  constructed for a single research study, as  you can see from the example in fig. 1.    in research, two hypotheses are constructed  for each research question. the first is the  null hypothesis.  the null hypothesis  (represented as h0) assumes no relationship  between variables; thus it is usually phrased  as “this has no affect on this”.  the  alternative hypothesis (represented as h1) is  simply stating the opposite, that “this has an  affect on this.” the null hypothesis is  generally the one constructed for scientific  research.    type i & ii errors  anytime you make a decision in life, there is  a possibility of two things going wrong.   take the example of a job offer. if you  decide to take the job and it turned out to be  lousy, you would have wasted a lot of time  and energy. however, if you decided to pass  on the job and it was great, you would have  wasted an opportunity.  it’s best illustrated  by a two by two box (fig. 1).     it is obvious that, despite thorough research  about the position (speaking to people that  work there, interview process, etc.), it is  possible to come to the wrong conclusion  about the job.  the same possibility occurs in  research. if your research concludes that  there is a relationship between variables  when in fact there is no relationship (i.e.,  you’ve incorrectly assumed the alterative  hypothesis is proven), this is a type i error.  if your research concludes that there is no  relationship between the variables when in  fact there is (i.e., you’ve incorrectly assumed  the null hypothesis is proven), this is a type   ii error. another way to think of type i & ii  errors is as false positives and false  negatives. type i error is a false positive,  like concluding the job is great when it’s  lousy.  a type ii error is a false negative;  concluding the job is lousy when it’s great.     type i errors are considered by researchers  to be more dangerous.  this is because  concluding there is a relationship between  variables when there is not can lead to more  extreme consequences.  a drug trial  illustrates this well.  concluding falsely that  a drug can help could lead to the drug being  put on the market without being beneficial  to the public.  a type ii error would lead to  a promising drug being left off the market,  evidence based library and information practice 2007, 2:1  34 which while serious, isn’t considered as dire.  to help remember this, think of the  conservative nature of science. inaction (and  possibly more testing) is less dangerous  than action.  thus, disproving the null  hypothesis, which supposes no relationship,  is preferred to proving the alternative  hypnosis.    there are many safety features built in to  research methodology which help minimize  the possibility of committing both errors,  including sampling techniques and  statistical significance, both of which you  will learn about later.    dependent and independent variables  understanding hypotheses help you  determine which variables are dependent  and which are independent (why this is  important will be revealed a bit later).   essentially it works like this:  the dependent  variable (dv) is what you are measuring,  while the independent variable (iv) is the  cause, or predictor, of what is being  measured.    in experimental research (research done in  controlled conditions like a lab), there is  usually only one hypothesis, and  determining the variables are relatively  simple. for example, in drug trials, the  dosage is the independent variable (what  the researcher is manipulating) while the  effects are dependent variables (what the  researcher is measuring).    in non‐experimental research (research  which takes place in the ‘real world’, such as  survey research), determining your  dependent variable(s) is less straightforward.   the same variable can be considered  independent for one hypothesis while  dependent for another. an example – you  might hypothesize that hours spent in the  library (independent variable) are a  predictor of grade point average (dependent  variable). you might also hypothesize that  major (independent variable) affects how  much time students spend in the library  (dependent variable). thus, your hypothesis  construction dictates your dependent and  independent variables.    a final variable to be aware of in  quantitative research is the confounding  variable (cv).  also know as lurking  variables, a confounding variable is an  unacknowledged factor in an experiment  which might affect the relationship between  the other variables.  the classic example of a  confounding example affecting an  assumption of a relationship is that murder  rates and ice cream purchased are highly  correlated (when murder rates go up, so  does the purchase of ice cream?). what is  the relationship?  there isn’t one; both  variables are affected by a third,  unacknowledged variable: hot weather.     population, samples & sampling  although it is possible to study an entire  population (censuses are examples of this),  in research samples are normally drawn  from the population to make experiments  feasible. the results of the study are then  generalized to the population.  obviously, it  is important to choose your sample wisely!    population  this might seem obvious, but the first step is  to carefully determine the characteristics of  the population about which you wish to  learn.  for example, if your research  involves your university, it is worthwhile to  investigate the basic demographic features  of the institution; i.e., what is the percentage  of undergraduate students vs. graduate  students?  males vs. females?  if you think  these are groups you would like to compare  in your study, you must ensure they are  properly represented in your sample.    sampling techniques  probability sampling  evidence based library and information practice 2007, 2:1  35 probability sampling means that each  member of the population has an equal  chance of being selected for the survey.   there are several flavors of probability  sampling; the common characteristic being  that in order to perform probability  sampling you must be able to identify all  members of your population     random sampling is the most basic form of  probability sampling. it involves identifying  every member of a population (often by  assigning each a number), and then  selecting sample subjects by randomly  choosing numbers. this is often done by  computer programs.    stratified random sampling ensures the  sample matches the population on  characteristics important to a study. using  the example of a university, you might  separate your population into graduate  students and undergraduate students, and  then randomly sample each group  separately. this will ensure that if your  university has 70% undergraduates and 30%  graduates, your sample will have a similar  ratio.    cluster sampling is used when a population  is spread over a large geographic region.   for example, if you are studying librarians  who work at public libraries in canada, you  might randomly sample 50 libraries, and  then randomly sample the librarians within  those libraries.    non‐probability sampling  simply put, this is any sampling technique  that does not involve random sampling.   often samples are not random because in  some research it is easier to perform  convenience sampling (surveying those who  volunteer, for example). also, sometimes the  population from which the sample is to be  taken cannot be easily identified.  a  common strategy employed by libraries is to  use patron records to derive random  samples. this is probability sampling only if  the population is library users; if the  population is an entire institution or city, it  is no longer random. with non‐probability  samples, you can only generalize to those  who participated, not to a population.    sample size  sample size is also extremely important to  be able to accurately generalize to a  population. generally, the bigger the sample,  the better. the central limit theorem states  that the larger the sample, the more likely  the distribution of the means will be normal,  and therefore population characteristics can  more accurately be predicted.  some other  things to keep in mind:    • if you want to compare groups with  each other (for example, majors),  you will need at least 5 subjects in  each group to do many statistical  analyses.    • poor response rate can severely  compromise a study, if surveys are  involved.  depending on the  distribution method, response rate  can be as low as 10% (ideally you  want a response rate over 70%)  (weisberg 119).ensure your sample  size is large enough to still provide  accurate results with a poor  response rate.    there is no magic formula to determine the  proper sample size – it depends on the  complexity of your research, how  homogenous the population is, and time  and human resources you have available to  compile and analyze data.    descriptive statistics  once you have performed your research  and gathered data, you need to perform  evidence based library and information practice 2007, 2:1  36   table 1. examples of hypotheses.      data analysis. choosing the appropriate  statistical method for the data is crucial.  the  bad news is, this means you have to know a  whole lot about your data – is it nominal,  ordinal or ratio? is it normally distributed?  let’s start from the very beginning.  a clear understanding of librarians’ perceptions of ebp is necessary to inform the development of  systems to support ebp in librarianship.    the following research questions were posed:  1. what are the perceptions of librarians of ebp?  2. does institution type the librarian works at affect perception?  3. does length of service of the librarian affect perception?    what are the hypotheses?    there are three being provided. here is a rephrasing of number 3:    h0 = “length of service of librarians has no affect on the perception of ebp”  h1 = “length of service of librarians affects the perception of ebp”    what are the type i & ii error possibilities?            the real situation (in the population)             h0 is true                     h1 is true    no error        type ii error        result of  research                  (from sample):        h0 is proven (length  of service doesn’t  affect perception)      h1 is proven (length  of service does affect  perception)    type i error        no error    what are the dependent and independent variables?    the researchers are attempting to determine whether length of service can predict perception of ebp,  or to rephrase, is perception of ebp dependant on length of service. therefore:    dependent variables: perception of ebp  independent variable: length of service  evidence based library and information practice 2007, 2:1  37 levels of measurement  nominal variables are measured at the most  basic level.  they are discrete levels of  measurement where a number represents a  category (i.e., 1 = male; 2 = female), but these  numbers do not imply order and  mathematical calculations cannot be  performed on them.  you could just as easily  say, 1 = male and 36,000 = female ‐ this  doesn’t mean that females are 35, 999 times  bigger or better than males! nominal  variables are of the least use statistically.    ordinal variables are also discrete categories,  but there is an order to the categories; they  increase and decrease at regular intervals.  a  good example is a likert scale:  1 = very  poor; 2 = poor; 3 = average, etc. in this  example, you can state 1 is ‘less’ or ‘smaller’  or ‘worse’ than 2.  the disadvantage of  ordinal variables is that you cannot measure  in between the values.  you do not know  how much worse 1 is than 2.    ratio (sometimes known as scale,  continuous or interval) variables are the  most robust, statistically, of variable types.   ratio variables have natural order, and the  distance between the points in the same.  think of pounds on a scale.  you know that      table 2. examples of sampling.    the sampling frame was the database of all librarians (defined as those who hold an mls)  who were members of the canadian library association in march 2005.  a total of 5,683  librarians were on the list. the list was divided up by type of library worked at (academic,  public, school, special, and other / not stated). a proportional random sample of 210 was then  selected. this ensured that even at a return rate of 40% a final sample size of 150 would be  achieved.    is this a random sample?  on first glance, yes.  however, this is only a true random sample if all librarians in canada  belonged to the canadian library association.  the design of this study means that the results  can only be generalized to canadian library association members, not to canadian librarians.    what sampling technique is used?  this survey used stratified random sampling to ensure that all types of librarians would be  represented, as illustrated in the chart below.  please remember that all values in this table are  for demonstration purposes and do not accurately reflect reality.    academic  librarians  public  librarians  school  librarians  special  librarians  other /  not stated  totals  real  proportion  1136 (20%)  2273 (40%)  568 (10%)  582   (15%)  582 (15%)  5683  sample size  42  (20%)  84  (40%)  21   (10%)  31   (15%)  31   (15%)  210  evidence based library and information practice 2007, 2:1  38   100 is lighter than 101. you also know that  101 is 1 pound heavier than 100.  finally the  scale is continuous; it is possible to weigh  100.58 pounds. the power of the ratio  variable is important to keep in mind for  your study.  for example, rather than asking  subjects to tick off an age category in a box,  you can ask them to fill in their age.  this  gives you the freedom to keep it as a ratio  variable, or to round the ages up into  appropriate ordinal values.      measures of central tendency  the theory of normal distribution tells us  that, if you tested an entire population, the  result (parameter) would look like a bell  curve, with the majority of values grouped  in the middle.  a good example of this  would be scores on test.      table 3. examples of variables.      selection of variables used in the study    variable name  variable label  values    type    type of library worked at  1 = academic, 2 = public…  length    length of service    income    income of respondent  1 = under 30,000, 2 = 31,000‐ 40,000…  age  age of respondent    ebp_aware  answer to the question i  have heard of ebp  1 = yes, 2 = no  ebp_score  score on the ebp  perceptions test      what level of measurement is type?  type is a nominal measurement. the numbers represent types of libraries, but no  mathematical calculations can be performed on them. ebp_aware is also a nominal  measurement.    what level of measurement is length?  because there are no values set for length it is a ratio variable. each librarian’s length of  service will be entered in years.  ebp_score and age are also ratio variables.    what level of measurement is income?  income is an ordinal variable. it has numbers representing categories, but there is a clear  ranking.  librarians in category one earn less money than librarians in category two.  evidence based library and information practice 2007, 2:1  39     figure 2. normal distribution of a bell curve.  however, when moving from parameters to  statistics, there is the probability that the  results will not reflect the population, and  thus not be normally distributed. measures  of central tendency provide you with  information about how your results are  grouped. there are three measures, and  which one to use depends on what level of  measurement the variable is.    mean (represented by m or μ) is the most  commonly referred to measure of central  tendency.  it is the average measure, where  each value is added, and then the sum  divided by the number of cases.  however,  it should be quite clear that the mean cannot  be used with nominal and ordinal variables.   imagine again a likert scale.  the mean  value might be 2.36, but what does that tell  you?  that the average respondent falls  somewhere closer to “i found this difficult”  than “i have no opinion”?    median (represented by mdn) is the measure  commonly used with ordinal data.  the  median is the halfway point of the data. to  calculate simply order your values from  lowest to highest and see at what value half  the data is below, and half is above.  the  median is also an extremely valuable  measure for ratio data when there are  outliers (think how the average income  variable would be skewed in a town with  one multimillionaire).  this is because  median is not affected by how far away  from the middle values are, just the quantity  of them.  the median for 2, 2, 3, 4, 4 is 3; the  median for 2, 2, 3, 4, 10 is also 3.      mode is often used with nominal data  (though it can also be calculated for other  variable types).  it is simply the most  frequently occurring value in a dataset.  an  example of when this would be an  appropriate measure is for major.  the  average major makes no sense, nor does the  halfway point major, but the most  frequently occurring major does.    measures of spread  measures of central tendency reveal much  about data, but not the whole story.  you  also need to know how the values are  spread across the spectrum.  measures of  spread will tell us whether the values are  clustered around the mean or more spread  out.  think of test scores; one group might  all score 70, while another group’s score  might range from 60‐90.  in this case, it is  possible that the mean, median and mode  would be the same, but we can see the  distribution is quite different. there are  three main statistical methods for  determining spread.    range is the most basic measure; it is  calculated simply by subtracting the lowest  score from the highest score.  however, this  is not the most accurate method as the range  can be skewed by outlier values (a very high  or very low score).    evidence based library and information practice 2007, 2:1  40 interquartile range is less likely to be  distorted by outliers, as it is calculated by  ordering the sample from highest to lowest,  then dividing the sample into four equal  quarters (percentiles). the median is then  calculated for each quartile.  subtracting the  median of the first quartile from the third  quartile obtains the interquartile range.    standard deviation (represented by sd or σ)  is the most sophisticated measure of spread,  and a widely used statistical concept.   statistical software will easily calculate  standard deviation, so the formula will not   be covered here.  because standard   deviation relies on calculations of the mean  it can only be used with continuous  variables.  a standard deviation score of 0  indicates that there is no variation of values.  the higher the standard deviation, the  larger the spread.   bivariate analysis  at heart of all research is an interest in  determining relationships between variables.      table 4. an example of measures of central tendency and measures of spread. characteristics of the variable age                                                           age of respondent    n        210  mean    44.05  median    43  mode    33  std deviation    12.77  range    38  percentiles  25  33    50  43    75  56.50      what does this tell us about the central tendencies of the data?  the average age of librarian respondent to this survey is 44.05.  half of the librarians were  over 43, while other half were under 43.  the most commonly occurring age was 33.      what does this tell about the spread of the data?  we can tell something about spread simply by looking at the difference between mean,  median and mode.  the fact that the mean is slightly higher than the median and much higher  than the mode indicates that there are some older respondents skewing the data.     the range indicates that there are 38 years between oldest and youngest respondent. this  large value could be due to the outliers at the upper end of the scale.  however, the large  standard deviation also indicates a wide spread of values.  this is not surprising, as logically  in any profession, there is likely to be a wide variety of ages.  evidence based library and information practice 2007, 2:1  41   there are many statistical methods for  exploring those relationships, which ones to  choose are often dependent on the type of  variables with which you are working  (nominal, ordinal or ratio). it is also  important to understand statistical  significance (the extent to which the  relationship can be generalized to the  population) and effect size (the strength of  the relationship) with bivariate analysis  techniques.    statistical significance     comprehending inferential statistics  requires a clear understanding of what is  meant by statistical significance.  for  something to be statistically significant, it is  unlikely to have occurred by chance  (remember that every time you are dealing  with a sample you are taking the chance that  your results will not reflect the population).  another way of putting it is that significance  tests denote how large the possibility is that  you are committing a type i error.   significance tests are affected by the  strength of relationship between variables  and the size of the sample. common levels  of significance (represented by alpha, or α)  are 5%, 1% and 0.1%; if α =.01, you are  stating that there is a one in one thousand  chance this happened by coincidence.      cross tabulation  what is a cross tabulation?  essentially a cross tabulation (cross tab) is a  table in which each cell represents a unique  combination of values.  this allows you to  visually analyze whether one variable’s  distribution is contingent on another’s.    when would you use a cross tabulation?  cross tabulations can be used to show  relationships between two nominal  variables, nominal and ordinal variables, or  two ordinal variables.  it can be used with  ratio data, as long as the variable has a  limited number of values. limitations of the  cross tabulation  cross tabulations provide you with a visual  view of comparative data, but because they  display simple values and percentages,  there is no way to gauge whether any  differences in the distribution are  statistically significant.    chi‐square  what is a chi‐square?  a chi‐square is a test which looks at each  cell in a cross tabulation and measures the  difference between what was observed and  what would be expected in the general  population. it is used to evaluate whether  there is a relationship between the values in  the rows and columns of a cross tab, and the  likeliness that any differences can be put  down to chance.     when would you use a chi‐square?  chi‐square is one of the most important  statistics when you are assessing the  relationship between ordinal and/or  nominal measures.    are there limitations of using chi‐square?  chi‐square cannot be used if any cell has an  expected frequency of zero, or a negative  integer.  it can be affected by low  frequencies in cells; if many of your cells  have a frequency of less than 5, the chi‐ square test might be compromised.    how do i know if the relationship is  statistically significant?  the chi‐square test provides a significance  value called a p‐value. the p‐value is  compared to α, which can be set at different  levels. if α = .05, then a p score less than .05  indicates statistical significant differences, a  p score greater than .05 means that there is  no statistical difference.  evidence based library and information practice 2007, 2:1  42     table 5. example of cross tabulation.        t‐test  what is a t‐test?  a t‐test compares the means between two  values. it tests whether any differences in  the means are statistically significant or can  be explained by chance.    when do you use a t‐test?  t‐tests are normally used when comparing  differences between two groups (i.e.,  undergraduates versus graduates) or in a  before and after situation (student  achievement before versus after library     instruction). a t‐test involves means,  therefore the dependent variable (the  variable you are attempting to measure)  must be a ratio variable.  the independent  variable is nominal or ordinal.    limitations of the t‐test  a t‐test can only be used to analyze the  means of two groups. for more than two  groups, use anova.      how do i know if the relationship is  statistically significant?  cross tabulation of type of library and i have heard the   term evidence‐based practice      yes  no  total  academic library           count  30  12  42     percentage  71.42%  28.58%  100%  public library           count  54  30  84     percentage  64.28%  35.72%  100%  school library           count  9  12  21     percentage  42.86%  57.14%  100%  special library           count  22  10  31     percentage  70.96%  29.04%  100%  other/not stated           count  20  11  31     percentage  64.51%  35.49%  100%  total count  100  110  210    what does this table tell us?  this table allows us to see the numbers of librarians who have heard of the term evidence‐ based practice broken down by type of library worked at. as you can see, there are some  differences between the groups; a smaller percentage of school librarians have heard of ebp  (42.86%, n = 9) than other type of librarians. there is no indication from this table, however, if  that difference is statistically significant.  evidence based library and information practice 2007, 2:1  43 like the chi‐square test, the t‐test provides a  significance value called a p‐value, and is  presented the same way.    correlation coefficients  what are correlation coefficients?  correlation coefficients measure the  strength of association between two  variables, and reveal whether the correlation  is negative or positive. a negative  relationship means that when one variable  increases the other decreases (e.g., drinking  alcohol and reaction time). a positive  relationship means that when one variable  increases so does the other (e.g.,study time  and test scores). correlation scores range  from ‐1 (strong negative correlation) to 1  (strong positive correlation). the closer the  figure is to zero, the weaker the association,  regardless whether it is a negative or  positive integer.        table 6. example of chi‐square.      a chi‐square statistic was then performed to determine if type of library worked at affected whether  librarians had heard the term evidence‐based practice. as you can see by the table below, p>.05,  therefore there is no statistical difference in distribution of awareness of ebp based on the type of  library worked at.      value  df  sig.  chi‐square  16.955  4  .990      why use a chi‐square?  a chi‐square is the statistic being used here because the relationship between two ordinal variables  (type of library worked at and awareness of the term ebp) is being explored.    what does value mean?  it is simply the mathematical calculation of the chi‐square.  it is used to then derive the p‐value, or  significance.     what does df mean?  df stands for degrees of freedom.  degrees of freedom is the number of values that can vary in the  estimation of a parameter. it is calculated for the chi‐square statistic by looking at the cross  tabulation and multiplying the number of rows minus one by the number of columns minus one (r‐ 1) x (c‐1). in this case, if we look back to fig. 4, we can see that we have a two by five table.  thus, (2‐ 1) x (5‐1) = 4.    what does sig. mean?  sig. stands for significance level, or p‐level. in this case p = .990.  as this number is larger than .05,  the null hypothesis is proven.  there is no statistically relationship between type of library and  awareness of ebp, despite the differences in percentages we saw in table 5.  evidence based library and information practice 2007, 2:1  44 when should you use correlation  coefficients?  correlation coefficients should be used  whenever you want to test the strength of a  relationship. there are many tests to  measure correlation; which one to use  depends on what variables you are  examining. a few are listed below:    nominal variables: phi, cramer’s v, lambda,  goodman and kruskal’s tau    ordinal variables: gamma, sommers d,  spearman’s rho  ratio variables: pearson r  limitations of correlation coefficients  correlation does not indicate causality.  simply because there is a relationship  between two variables does not mean that  one causes the other.  keep in mind  correlation only looks at the relationship  between two variables; there many be others  affecting the relationship (remember the  confounding variable!).  correlation  coefficients can also be skewed by outlier  values.     how do i know if the relationship is  statistically significant?  correlation scores range from ‐1 (strong  negative correlation) to 1 (strong positive  correlation). the closer the figure is to zero,  the weaker the association, regardless of  whether it is a negative or positive integer.    analysis of variance (anova)  what is anova?  like the t‐test, anova compares means,  but can be used to compare more than two  groups. anova looks at the differences  between categories to see if they are larger  or smaller than those within categories.    when should you use anova?  the dependent variable in anova must be  ratio.  the independent variable can be      table 7. example of a t‐test.   an independent samples t‐test was performed to determine if there was a statistical difference  between genders on the evidence‐based practice test. as the table below illustrates, there was  a significant difference in performance between males and females, t (19)=‐.398 p<.05      value  df  sig.  t‐test  ‐.398  19  .049      why use a t‐test?  a t‐test is used for these variables because we are comparing the mean of one variable (epb  test score, a ratio variable) between 2 groups (sex, a nominal variable).  an independent  samples t‐test is used here because the groups being compared are mutually exclusive ‐ male  and female.    how is the t‐test interpreted?  the t‐test value, degrees of freedom, and significance values can be interpreted in precisely  the same way as the chi‐square in fig. 5.  the significance value of .049 is less that .05,  therefore it can be stated that the null hypothesis is disproved; there is a statistical significant  difference between the performance of male librarians and the performance of female  librarians on the ebp perceptions test.  evidence based library and information practice 2007, 2:1  45   table 8. example of a pearson r correlation.      nominal or ordinal, but most be composed  of mutually exclusive groups     limitations of anova  anova measures whether there are  significant differences between three or  more groups, but it does not illustrate where  the significance lies – there could be  differences between all groups or only two.  there are tests called post hoc comparisons  which can be performed to determine where  significance lies, however.    how do i know if the relationship is  statistically significant?  an anova uses an f‐test to determine if  there is a difference between the means of  groups. the f‐test can be used to calculate a   p‐score, which is analyzed in the same way  as chi‐squares and t‐tests.     statistical significance and effect size  measures  significance tests have a couple of  weaknesses.  one is the fairly arbitrary  value at which statistical significance is said  to have occurred.  why is α = .051 not a  significant finding while α = .049 is?  the       second disadvantage is that significance  tests do not give an indication of the  strength of a relationship, merely that it  exists.  a smaller significance value could be  the result of a larger sample rather than a  strong relationship.  this is where effect  sizes come in. effect sizes are tests which  gauge the strength of a relationship. there  are many different effect size indices; which  to use depends on the statistical test being  performed.      multivariate analysis  any in‐depth discussion of multivariate  analysis is beyond the scope of a paper  entitled “statistical primer”; however, here  is a brief introduction.    multivariate analysis looks at the  relationship between more than two  variables, for example length of service and  type of librarian might together be  predictors of perception of ebp. using  bivariate statistical methods, it is not  possible to see the relationship between two  independent variables as well as their effect  on the dependent variable. there are several  multivariate statistical methods.  here are  two of the most common.   a pearson r correlation was performed to determine if there was a relationship between age  and score on the ebp test instrument. the correlation revealed that the two were significantly  related, r=+.638, n=210, p<.05.    why was a pearson r correlation performed?  a pearson r was done because both variables involved, age and ebp perceptions test score,  are ratio variables.    what does the r value tell us?  the r is correlation score.  remember that correlation scores range from +/‐1 to 0.  therefore, a  score of +.638 reveals that there is a strong positive correlation between age and ebp score.   the fact that it is positive means that when one variable increases so does the other – the older  the librarian, the higher they scored on the ebp test instrument.  evidence based library and information practice 2007, 2:1  46   table 9. example of anova.    statistical test  effect size measure  comments    chi‐square  phi  phi tests return a value between zero (no  relationship) and one (perfect relationship).    t‐test  cohen’s d    cohen’s d results are interpreted as 0.2 being a  small effect, 0.5 a medium and 0.8 a large effect  size. (cohen 157)  anova  eta squared  eta square values range between zero and one,  and can be interpreted like phi and cohen’s d.    table 10. statistical tests and effect size measures.      multivariate analysis of variance (manova)  is an anova which analyses several  dependent variables.  it can be interpreted  in much the same way as anova tests.   manova has advantages over doing  multiple anova tests, including reducing  the potential for type i errors (concluding  that there is a relationship when there is not).  conversely, manova tests can also reveal  relationships not apparent in anova tests.   multiple linear regression examines “the  relationship between one ‘effect’ variable,     called the dependent or outcome variable,  and one or more predictors, also called  independent variables” (muijs 168).  it is  designed to work with continuous variables,  though there are different techniques  available for analyzing other variable types.   while performing and analyzing regressions  are complicated, they are valuable tools for  examining the relationship between many  variables.  it is important to note that, like  other inferential statistical techniques,  values are created that provide the statistical  significance of the relationships.    for the ebp test instrument score, the analysis of variance (anova) revealed that there was  not a significant difference in performance f (3, 47)=3.43, p<.05 between types of librarians.  the critical value (.245) for the scores was obtained the f distribution table using dfbetween=4  and dfwithin=16.    why was an anova performed?  an anova was the appropriate statistical technique because the dependent variable (ebp  test score) is continuous, while the independent variable (type of library worked at) is  nominal and composed of several groups.    what does this tell us?  the f test score was calculated at 3.43.  this score was used in conjunction with the degrees of  freedom (because we are comparing several groups, there are two degrees of freedom scores,  one for between the groups (4) and one for within the groups (16) to calculate the p‐score.  p  = .245, which is greater than .05. therefore there is no difference in performance on the test  based on the type of library worked at.  evidence based library and information practice 2007, 2:1  47 conclusion  this paper is not intended to produce  statistical experts.  rather, it is a guide to  understanding the basic principles and  techniques common in library and related  research.  most statistical software packages,  such as spss or sas, will effortlessly  perform statistics, so it is far more important  that as a researcher you know a) how to  select an appropriate sample; b) know what  statistical technique is appropriate in which  situations; and c) be able to interpret results  correctly.  there are a few things you can do  to make yourself more comfortable with  statistics.  one is to purchase a basic  quantitative methods textbook. look for one  that comes with a cd of sample data sets.   running through the exercises in the  textbook will provide you with valuable  practice in performing and analyzing  statistics.  there are several textbooks  available in the library field, although any  social science quantitative methods texts  would be useful. the second thing you can  do is to read the research literature in your  field.  if you know the topic well, it is easier  to evaluate and interpret results.        works cited  cohen, j. “a power primer.” psychological   bulletin 112 (1992): 155‐159.    muijs, daniel.  doing quantitative research in   education with spss. london: sage,  2004.    nardi, peter m.  doing survey research: a  guide to quantitative methods. boston:  allyn and bacon, 2003.    weisberg, herbert f., jon a. krosnick, and  bruce d. bowen. an introduction to  survey research, polling, and data  analysis. 3rd ed. thousand oaks, calif.:  sage publications, 1996.      evidence based library and information practice evidence based library and information practice 2011, 6.2 85 evidence based library and information practice ebl 101 a new path: research methods virginia wilson client services librarian, murray library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca received: 26 apr. 2011 accepted: 28 apr. 2011 2011 wilson. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. for more than two years, this column has taken you through the steps of evidence based library and information practice (eblip). with the final step taken in the last issue (keeping in mind that eblip is an iterative process and that the notion of the final step is sometimes interpretive), it is time to choose a new path and take the first step on an exciting new journey. for the next while, i will explore the exhilarating world of research methods! do i sound invigorated? i am! i’m no expert, by any means. but, i am a lifelong learner; a practitioner-researcher with a strong interest in research methods, so we are going to learn together. and of course this column is ebl 101, so the information will be introductory and by no means exhaustive. as of right now, i have no set plan on the exact methods i will tackle, nor the order in which i will wrestle them to the ground. so if you have any needs or suggestions, please let me know. for this first column on our new path, i’m going to talk about qualitative and quantitative research in general. yes, that is a big topic for a small column, so let’s see how it goes. the rivalry between the toronto maple leafs and the montreal canadians (hockey for the non-canadians on board) has nothing on the rivalry between quantitative and qualitative research methods, or at least between the researchers devoted to them. qualitative scholars consigned quantitative research to the lower echelons of the scientific field because of its “subjective, interpretive approach” (denzin & lincoln, 2005, p. 2). qualitative researchers have defended the subjective, interpretive approach, stating that it does not reduce the subject to numbers, but rather uses a holistic approach to preserve the complexities of the subject (black, 1994, p. 425). granted, we are moving in the direction of being able to see the value in both types of methodology, and the rivalry is becoming a comfortable living arrangement (at least i think it is!). however, that was not always the case, and for a long time, only the seemingly solid numbers of quantitative data were seen to have value. mailto:virginia.wilson@usask.ca� evidence based library and information practice 2011, 6.2 86 in the early days of eblip, a hierarchy of evidence was put forward which borrowed heavily from evidence-based medicine, placing randomized controlled trials near the top and case studies down at the bottom. research methods were given a place in the hierarchy based on their perceived robustness as methods, not taking into account the quality of the research that comes out of that methodology, which is highly subjective depending on the individual study (booth, 2010, p. 84). qualitative methods were given a nod, but the impression was one of the privileging of quantitative methods. given writes that “qualitative researchers and the results of their work remain marginalized in eblip” (2007, p. 16). she explains this marginalization by looking at the history of eblip: “in understanding the historical path that this movement has taken in our field (i.e. moving through the ranks of health librarianship to try to find a stronger voice across library contexts), it is little wonder that the controversial baggage about the quality of qualitative evidence has been transported along the way” (given, 2006, p. 382). every movement has growing pains and i believe it is safe to say that qualitative methods are coming into their own in eblip. lis as a social science is well-suited to both realms of methodology. i refer you to an editorial written by denise koufogiannakis in evidence based library and information practice (2010, 5.3). in it, she addresses the evidence hierarchy and points out that the choice of research methods should be driven by the type of research question asked. this viewpoint about choice emerges elsewhere (newman & benz, 1998, p. 14; bell, 2010, p. 6; silverman, 2010, p. 9). the notion of research is often fraught, no matter from which discipline you look at it. and while i am going to focus the rest of the column on outlining some of the differences between quantitative and qualitative methodology, i urge you to explore the debate between the two. there seems to be as many opinions as there are researchers. as well, i should note that pitting one against the other via a laundry list of differences sets these two paradigms up as a dichotomy. many involved in the debate feel that a dichotomous relationship limits both methodologies (morgan & smircich, 1080, p. 491). we would be better served to realize that there are times when both types of methods can be used in the same research study (also known as mixed-methods research) for a depth and richness of perspectives and in order to triangulate findings. this is based on the premise that “multiple viewpoints allow for greater accuracy” (jick, 1979, p.602). in its simplest definition, quantitative research works with numbers and qualitative research works with people or text. quantitative methods attempt to answer “how many?” and qualitative methods attempt to answer “why is this happening?” quantitative research emerged from a positivistic paradigm, and the belief that there is one objective reality and it is our job to apprehend it. qualitative research follows the naturalistic paradigm, which posits that there are many realities, that the subject is involved in creating his/her own reality, and that it is our job to discover and explore those subjective realities. quantitative methods, using deductive processes and statistical analyses, attempt to confirm a hypothesis and test a theory, while qualitative methods, using inductive processes, explore a phenomenon and attempt to generate a theory. methods common to quantitative research include surveys, randomized controlled trials, and highly structured observation. methods common to qualitative research include in-depth interviews, focus groups, and participant observation. of course this only scratches the surface. the number of books and articles on the subject of research methodology is astounding. just three examples include research design: qualitative, quantitative, and mixed method approaches by john w. creswell, the sage handbook of qualitative research, and introduction to quantitative research methods: an investigative approach by mark balnaves and peter caputi. the ways in which researchers utilize research methods are varied. in upcoming columns, i will make my way from evidence based library and information practice 2011, 6.2 87 one method to another and i hope you will join me along the way. it strikes me that the features found in the open journal system used by eblip would be very handy for having an ongoing discussion on research methods. look for the article tools on the right side of the page when you have opened up an article in the journal. you will have the options, among others, to email the author or post a comment. you can also use the “add a comment” feature found underneath the pdf version of the article. i welcome such interaction, as i believe that a community of like-minded individuals could generate conversations that would enrich us all. references bell, j. (2010). doing your research project (5th ed.). berkshire, uk: mcgraw hill. black, n. (1994). why we need qualitative research. journal of epidemiology and community health, 48, 425-426. booth, a. (2010). on hierarchies, malarkeys and anarchies of evidence. health information and libraries journal, 27, 8488. denzin, n. k. & lincoln, y.s. (2005). the sage handbook of qualitative research (3rd ed.). thousand oaks, california: sage. given, l. (2007). evidence-based practice and qualitative research: a primer for library and information professionals. evidence based library and information practice, 2(1), 15-22. given, l. (2006). qualitative research in evidence-based practice: a valuable partnership. library hi tech, 24(3), 376-386. jick, t. d. (1979). mixing qualitative and quantitative methods: triangulation in action. administrative science quarterly, 24, 602-611. koufogiannakis, d. (2010). the appropriateness of hierarchies. evidence based library and information studies, 5(3), 1-3. morgan, g. & smircich, l. (1980). the case for qualitative research. academy of management review, 5(4), 491-500. newman, i. & benz, c.r. (1998). qualitative quantitative research methodology: exploring the interactive continuum. carbondale: southern illinois university press. silverman, d. (2010). doing qualitative research (3rd ed.). london, sage. / evidence based library and information practice bell, j. (2010). doing your research project (5th ed.). berkshire, uk: mcgraw hill. evidence summary   academic social networking sites are smaller, denser networks conducive to formal identity management, whereas academic twitter is larger, more diffuse, and affords more space for novel connections   a review of: jordan, k. (2019). separating and merging professional and personal selves online: the structure and processes that shape academics’ ego-networks on academic social networking sites and twitter. journal of the association for information science and technology, 70(8), 830-842. https://doi.org/10.1002/asi.24170   reviewed by: scott goldstein coordinator, web services & library technology mcgill university library montréal, québec, canada email: scott.goldstein@mcgill.ca   received: 1 dec. 2019                                                                    accepted:  20 jan. 2020      2020 goldstein. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29687     abstract   objective – to examine the structure of academics’ online social networks and how academics understand and interpret them.   design – mixed methods consisting of network analysis and semi-structured interviews.   setting – academics based in the united kingdom.   subjects – 55 u.k.-based academics who use an academic social networking site and twitter, of whom 18 were interviewed.   methods – for each subject, ego-networks were collected from twitter and either researchgate or academia.edu. twitter data were collected primarily via the twitter api, and the social networking site data were collected either manually or using a commercial web scraping program. edge tables were created in microsoft excel spreadsheets and imported into gephi for analysis and visualization. a purposive subsample of subjects was interviewed via skype using a semi-structured format intended to illuminate further the network analysis findings. transcripts were deductively coded using a grounded theory-based approach.   main results – network analysis replicated earlier findings in the literature. a large number of academics have relatively few connections to others in the network, while a small number have relatively many connections. in terms of reciprocity (the proportion of mutual ties or pairings out of all possible pairings that could exist in the network), arts and humanities disciplines were significantly more reciprocal. communities (measured using the modularity algorithm, which looks at the density of links within and between different subnetworks) are more frequently defined by institutions and research interests on academic social networking sites and by research interests and personal interests on twitter. the overall picture was reinforced by the qualitative analysis. according to interview participants, academic social networking sites reflect pre-existing professional relationships and do not foreground social interaction, serving instead as a kind of virtual cv. by contrast, twitter is analogized to a conference coffee break, where users can form new connections.   conclusion – academic social networking sites exhibit networks that are smaller, denser, more clustered around discrete modularity classes, and more reciprocal. twitter networks are larger and more diffuse, which is more conducive to fostering novel connections. the author makes suggestions for how academic social networking sites could encourage network building and rethink how academic reputation is measured.   commentary   this study, part of the author’s phd dissertation, looks at how academic social networking sites (snss) are used and conceptualized by a sample of users in the united kingdom. specifically, the author looks at academia.edu and researchgate, the two most well-known academic snss, as well as twitter, which is heavily used by academics despite being open to everyone. previous studies in this area have examined how academic sns uptake varies by discipline, which factors influence engagement (i.e., how many times a profile is viewed), and whom users choose to follow within the network. the author’s approach is novel in that she uses mixed methods, combining network analyses with semi-structured interviews of the networks’ egos. by contrast, the majority of studies to date rely on purely quantitative methods of data analysis.   the casp qualitative checklist (critical appraisal skills programme, 2018) was used for evaluation. the research questions are clearly articulated and the mixed methods methodology is appropriate, especially considering the need for qualitative analysis to illuminate why users choose the platforms they choose and how they think about what to share and to whom to link. the methods are clearly explained and sufficiently rigorous. findings are clearly explained and related back to earlier research. the author acknowledges limitations with the non-probability sampling strategy, although this is somewhat difficult to square with language earlier in the article that implies the research aim is to understand the structural characteristics of academics’ online social networks generally. since the sample was small and self-selecting, perhaps more care should have been taken to emphasize that the data cannot be used to make generalizations. in addition, for technical reasons, twitter network analyses were not performed for eight participants, and these eight were excluded from being interviewed for this reason. however, what made those eight participants special is they either followed, or were followed by, over 2,000 people. it is possible that these participants would have provided a novel perspective on how they use twitter compared to others who were interviewed, but this limitation was not considered in the qualitative analysis discussion.   this study, while largely replicating the findings of earlier work, has several implications for how academic librarians approach their liaison work with scholars. first, academics’ use of snss suggest their online “reach” is important to them, but the author highlights that twitter in particular “may offer more potential for novel connections and opportunities for academics” than academic snss (p. 839). perhaps twitter could be included as a normal part of the online identity management toolkit along with, for example, orcid registration, a personal website, and participation in an institutional repository. second, librarians should be aware of the research suggesting that academic snss metricize scholarly reputation and do so primarily based on traditional measures such as the journal impact factor. academic reputation, much like research quality, is a complex measure that is not yet reducible to a simple formula. users should understand this and not view where and how often they publish as the primary means to succeed in their field.   references   critical appraisal skills programme. (2018). casp qualitative checklist. retrieved from https://casp-uk.net/wp-content/uploads/2018/01/casp-qualitative-checklist-2018.pdf evidence based library and information practice 2011, 6.4 8 evidence based library and information practice commentary openness: the essential quality of knowledge martin hall vice-chancellor university of salford salford, united kingdom email: martin.hall@salford.ac.uk received: 23 aug. 2011 accepted: 25 aug. 2011 2011 hall. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. openness is an essential quality of knowledge, enabling academic practice and driving key aspects of the knowledge economy. in contrast, inappropriate restrictions on the distribution of knowledge damage innovation and discovery and have a direct effect on the quality of life. this has been appreciated for a very long time. thomas jefferson, for example, famously used the metaphor of the candle, pointing out how many flames could be ignited without extinguishing or diminishing the light from the original. however, the over-commercialization of intellectual property in the early years of the digital revolution has left a legacy that is both inappropriate and dangerous. the case for openness was made powerfully and eloquently by peter drahos and john braithwaite in their book information feudalism: who owns the knowledge economy, published in 2002. their case, still well worth reading today, was based on a careful analysis of the april 1994 agreement on trade-related aspects of intellectual property rights, or trips for short. this had been adopted by over one hundred countries meeting in marrakech; drahos and braithwaite (2002) showed how it had been put together by a remarkably small group of international interests, dominated by the entertainment industries. “trips”, they wrote, “was the first stage in the global recognition of an investment morality that sees knowledge as a private, rather than public, good” (p.198). the deleterious consequences were immediately apparent to, for example, a large number of small scale indian farmers, since it patented seed, preventing them from using their own seed stock rather than seeds licensed by multinational corporations such as monsanto. drahos and braithwaite show how the marrakech agreement was profoundly at odds with the development objectives of many of its mailto:martin.hall@salford.ac.uk� evidence based library and information practice 2011, 6.4 9 signatories. in contrast, the profound financial benefits to a narrow set of commercial interests were illustrated through a simple metaphor analogous to thomas jefferson’s candle, “if you came to own a patent in a genetically engineered cow that produces twice as much milk as existing cows, you had an asset that was equal in value to all the herds of all the world’s dairy farmers. and a more liquid asset than all that milk and all those cows” (drahos and braithwaite 2002, p. 198). given this, the release of digital opportunity: a review of intellectual property and growth in 2011, along with the quick announcement that the british government has accepted all its recommendations, is particularly to be welcomed. this review is sharply critical of the consequences of over-enforcement of intellectual property rights and the damage done by protectionism: “the copyright regime cannot be considered fit for the digital age when millions of citizens are in daily breach of copyright, simply for shifting a piece of music or video from one device to another. people are confused about what is allowed and what is not, with the risk that the law falls into disrepute.” digital material is “rotting away” because libraries are not allowed to archive it. reliance on enforcement is inappropriate; “instead, government should respond in four ways: by modernising copyright law; through education; through enforcement and by doing all it can to encourage open and competitive markets in licensed digital content, which will result in more legitimate digital content at prices which appeal to consumers” (hargreaves 2011, p. 6). the hargreaves review, and other reports and reviews like it, help us to regain the qualities and principles of openness that prevailed before trips and the commercial appropriation of new digital knowledge technologies some twenty years ago. this includes re-examining access to published research results; the search for new models of distribution, including “green” and “gold” systems; and the role of open access repositories at universities and other research institutions. a stable point is to see universities as organizations that are the opposite of the forprofit organizations that were so advantaged by trips and related policies. why? because the essence of academic life is to give intellectual property away rather than to set up secretive and legally-defended systems in order to extract maximum financial returns (hall, 2010). disciplines, and fields of study, are and always have been sophisticated global networks in which ideas and information circulate and are formalized. systems of circulation include flexible and open networks of collaboration, shared databases, conferences, workshops, and a wide variety of mechanisms for bringing people together to share their insights and information about commonly prioritized problems. systems of formalization include peer-reviewed academic journals, books by publishers with recognized academic credentials, edited collections of papers, and conference proceedings. taken together, this is a massive, open knowledge system that has been established over several centuries and which joins together some 10,000 institutions which are recognizable as universities, as well as hundreds of thousands of libraries and other forms of knowledge repositories (david, 2005). what drives this network? the fundamental imperative is maintaining and advancing the reputation of individual academics and research groups. we do this through well tested systems of recognition and authentication. at the heart of the system of recognition is citation, and citation is a sophisticated form of distributing intellectual capital. a major point of the work of universities is to have the scholarship of its academics cited with approval and respect by as many other academics as possible across the widest geographical span, forming what paul evidence based library and information practice 2011, 6.4 10 david has called “invisible colleges” (david, 1998). this system, of course, long predates the digital revolution of the early to mid-1990s although the digital revolution gives us immense opportunities to expand long established, open networked forms of academic knowledge distribution. and the university, too, is a resilient form of institution in its own right, despite the fact that every ten years or so its demise is predicted. why is this open academic network so resilient? it is as a result of the nature of knowledge itself. one of the beneficial consequences of the digital revolution of the mid-1990s has been the stimulation of research into the nature of knowledge. work in the field of the knowledge economy has shown how knowledge is best understood as a spectrum from tacit to highly codified forms. tacit knowledge is often shared by individuals on a face-to-face basis, circulating informally within groups. codified knowledge is expressed in ways that can be easily summarized, communicated, and distributed (see, for example, foray, 2004; collins, 2010). the work of a typical university science laboratory illustrates the spectrum. ideas originate in informal discussions and seminars and are tossed around until they have some valency and coherence. as this tacit knowledge takes shape, it begins to be codified, first as working papers and then as a formal publication. in its most advanced form, codified knowledge is expressed in the binary code that enables our digital world. the more knowledge is codified, the more it can be shared. in its codified forms, knowledge can be reproduced, potentially infinitely, without exhausting the original. the more knowledge is shared and reproduced, the more futile our attempts to contain, limit, or own its distribution. and the more that knowledge is distributed, the more likely new combinations with their own potential will be enabled and promoted (foray, 2004). it is clear that these particular and peculiar qualities of knowledge make it different from other categories of phenomena. the history of knowledge, and its exponential tendencies in explaining the world, can be mapped against the great inventions that facilitated communication of codified information, i.e., the printed book, the telegraph, and the internet. while we tend to think of the explosion of knowledge as a recent phenomenon, these essential qualities of knowledge have always been at the heart of the university and have been known for a long time. acknowledgement this paper is based on a keynote address given at the 6th international evidence based library and information practice conference (eblip6), held at the university of salford, june 27-30, 2011. references collins, h. (2010). tacit and explicit knowledge. chicago il: university of chicago press. david, p. (1998). communication norms and the collective cognitive performance of 'invisible colleges'. in g. b. navaretti, p. dasgupta, k.-g. maler and d. siniscalco, creation and transfer of knowledge: institutions and incentives. heidelberg: springer-verlag. david, p. (2005). innovation and universities’ role in commercializing research results: second thoughts about the bayh-dole experiment. palo alto, ca: stanford institute for economic policy research. drahos, p. and braithwaite, j. (2002). information feudalism: who owns the knowledge economy? new york, ny: new press. foray, d. (2004). economics of knowledge. cambridge, ma: mit press. evidence based library and information practice 2011, 6.4 11 hargreaves, i. (2011). digital opportunity: a review of intellectual property and growth. london, intellectual property office. hall, m. (2010). minerva's owl. a response to john houghton and charles oppenheim's 'the economic implications of alternative publishing models'. prometheus, 28(1), 61-71. doi: 10.1080/08109021003676375 / evidence based library and information practice given this, the release of digital opportunity: a review of intellectual property and growth in 2011, along with the quick announcement that the british government has accepted all its recommendations, is particularly to be welcomed. this review is sh... the hargreaves review, and other reports and reviews like it, help us to regain the qualities and principles of openness that prevailed before trips and the commercial appropriation of new digital knowledge technologies some twenty years ago. this inc... a stable point is to see universities as organizations that are the opposite of the for-profit organizations that were so advantaged by trips and related policies. why? because the essence of academic life is to give intellectual property away rather ... disciplines, and fields of study, are and always have been sophisticated global networks in which ideas and information circulate and are formalized. systems of circulation include flexible and open networks of collaboration, shared databases, confere... taken together, this is a massive, open knowledge system that has been established over several centuries and which joins together some 10,000 institutions which are recognizable as universities, as well as hundreds of thousands of libraries and other... what drives this network? the fundamental imperative is maintaining and advancing the reputation of individual academics and research groups. we do this through well tested systems of recognition and authentication. at the heart of the system of recog... this system, of course, long predates the digital revolution of the early to mid-1990s although the digital revolution gives us immense opportunities to expand long established, open networked forms of academic knowledge distribution. and the universi... why is this open academic network so resilient? it is as a result of the nature of knowledge itself. one of the beneficial consequences of the digital revolution of the mid-1990s has been the stimulation of research into the nature of knowledge. work ... the work of a typical university science laboratory illustrates the spectrum. ideas originate in informal discussions and seminars and are tossed around until they have some valency and coherence. as this tacit knowledge takes shape, it begins to be ... it is clear that these particular and peculiar qualities of knowledge make it different from other categories of phenomena. the history of knowledge, and its exponential tendencies in explaining the world, can be mapped against the great inventions th... hargreaves, i. (2011). digital opportunity: a review of intellectual property and growth. london, intellectual property office. hall, m. (2010). minerva's owl. a response to john houghton and charles oppenheim's 'the economic implications of alternative publishing models'. prometheus, 28(1), 61-71. doi: 10.1080/08109021003676375 news   call for applicants for eblip journal: production editor    2023. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30479     evidence based library and information practice (eblip), a peer-reviewed, open access journal published quarterly since 2006, seeks to fill the position of production editor. the successful candidate will be expected to begin their role by may 1, 2024, and serve a three-year term (with possibility of renewal).   the production editor will be responsible for:   layout of all journal content in pdf and html formats. working with the editorial team to ensure that the journal’s publication guidelines are followed and that the publication maintains a professional and consistent appearance. supervising the editorial intern to ensure proper formatting and accurate metadata for each article. scheduling all articles for publication on the website. communicating and consulting with the editor-in-chief on a regular basis. attending editorial team meetings via zoom on a quarterly basis and participating in the overall governance of the journal. ensuring that all necessary deadlines are met. communicating with potential authors and responding to queries.   the ideal candidate for production editor will be experienced in microsoft word, exhibit careful attention to detail, and have the ability to meet tight deadlines. familiarity with pkp’s online journal system is beneficial. knowledge of evidence based practice is also an asset.   this position requires dedicated time on a regular basis, and it is therefore essential that interested persons ensure available time to devote to this position prior to applying. the bulk of work occurs on a quarterly basis prior to journal publication, and the estimated workload is approximately 15-20 hours per issue.   interested persons should send a cover letter, indicating areas of strength they would bring to the role, and resume/cv as a single pdf file to ann medaille, editor-in-chief, at amedaille@unr.edu by january 26, 2024. for informal inquiries and more information about the role, contact the current production editor, rachel hinrichs, at rhinrich@iu.edu.   **please note that evidence based library and information practice is a non-profit, open access journal and all positions are voluntary and unpaid.   about the journal:   published quarterly and hosted by the university of alberta, this peer-reviewed, open access journal is targeted at all library and information professionals interested in an evidence based model of practice. by facilitating access to librarianship research via original research articles and evidence summaries of relevant research from the library literature, evidence based library and information practice enables librarians to practice their profession in an evidence based manner. please visit the evidence based library and information practice website for further information about the journal.     microsoft word ebl 101_10pt font evidence based library and information practice 2008, 3:3 91 evidence based library and information practice ebl 101 evidence based librarianship backgrounder su cleyle associate university librarian memorial university of newfoundland st. john’s, newfoundland, canada e-mail: scleyle@mun.ca julie mckenna deputy library director regina public library regina, saskatchewan, canada email: jmckenna@reginalibrary.ca received: 13 april 2008 accepted: 17 june 2008 © 2008 cleyle and mckenna. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. evidence based librarianship (ebl) is a means to improve the profession of librarianship by asking questions, finding, critically appraising and incorporating research evidence from library science (and other disciplines) into daily practice. it also involves encouraging librarians to conduct research (koufogiannakis and crumley, 112). welcome to ebl 101! this new column is designed to offer guidance into the workings of evidence based practice and answer that question: “how can i implement ebl in my library?” the intent is to offer short, simple columns on a variety of ebl topics allowing any librarian, regardless of library type or size, to practice evidence based librarianship. so let’s get to it, shall we? evidence based practice (ebp) is a term that we have all heard. usually it is associated with the health professions and originated the area of clinical medicine. the medical profession forged the way for many professions to embrace evidence based practice (ebm.) back in the 1990’s, canadian doctors sought to create an environment of lifelong learning and clinical practice that utilitized research to answer clinical questions. there are 5 steps to evidence based medicine: evidence based library and information practice 2008, 3:3 92 1. formulate an answerable question. 2. track down the best evidence 3. critically appraise the evidence (i.e. find out how good it is). 4. apply the evidence (integrate the results with clinical expertise and patient values). 5. evaluate the effectiveness and efficiency of the process (to improve next time) (glasziou 23). as you can see the concept of ebm is a simple one – gather evidence to help with answering questions and helping patients. for the discipline of medicine, the body of evidence is rich in these areas and it is conceivable that most questions can be answered through use of existing research evidence. but even though the steps seem simple enough, a variety of skills are needed to ensure each step is completed properly. for example, asking the right question is crucial to finding the best evidence; evaluating the evidence is pivotal to determining the best course of action, and so on. evidence based librarianship (ebl) evidence based practice is a practical approach to finding answers to questions and for professionals to stay abreast of current trends and research. it is also a useful model for contributing to the body of evidence. but will this model work in a discipline that is not grounded in the research practices associated with the collection and use of empirical data? the social sciences research base is very different from that in the sciences. it is possible to apply the evidence based practice model to social science disciplines, including librarianship? ebp, quite simply, can encompass original research and the evaluation and use of existing research. koufogiannakis, crumley, and slater reviewed several content analysis reports and note the “the variation in the interpretation of what constitutes a ‘research’ article…” (koufogiannakis, crumley, and slater 228). of the 2664 articles reviewed from the 2001 publishing year, 30.3% were identified as research articles. this is a rate similar to previous content reviews for librarianship, although there are variations in scope and definition of what “constitutes a ‘research’ article”. clearly, the higher percentage of articles appearing in our professional literature is not research oriented. librarianship is not primarily comprised of scholars or researchers. it is comprised of practitioners and administrators. thus, research has not necessarily made its way into our professional literature and our decision making processes. many of us in the profession now recognize the need to formalize our research and our decision making processes to ensure that we base our decisions on the best possible evidence. based on the ebm model, the steps for ebl are similar: 1. define problem 2. find evidence 3. appraise evidence 4. apply results of appraisal 5. evaluate change 6. redefine problem (booth and brice, 2003) the definition of the problem, or “the question”, is pivotal to the entire process. the next ebl 101 column will focus on asking the right question. works cited booth, andrew and anne brice: “evidence based librarianship: the first steps.” 2nd international evidence based librarianship conference, edmonton, alberta, evidence based library and information practice 2008, 3:3 93 canada, june 2003. april, 2005 <http://www.shef.ac.uk/scharr/eblib /conf2003.htm>. glasziou, paul. evidence based medicine workbook. london, gbr: bmj publishing group, 2003. august, 2008 <http://site.ebrary.com/lib/memorial /doc?id=10049688&ppg=29>. koufogiannakis, denise, and ellen crumley. “evidence based librarianship” feliciter, 3 (2002): 112-4. koufogiannakis, denise, ellen crumley, and linda slater. “a content analysis of librarianship research.” journal of information science 30.3 (2004): 227-39. microsoft word es 1467_protzko evidence based library and information practice 2008, 3:2 42 evidence based library and information practice evidence summary some in saskatchewan find the cochrane library useful after promotion, access and training efforts a review of: forbes, dorothy, christine neilson, janet bangma, jennifer forbes, daniel fuller, and shari furniss. “saskatchewan residents’ use of the cochrane library.” partnership: the canadian journal of library and information practice and research 2.2 (2007). reviewed by: shandra protzko organization information specialist, tucker memorial medical library, national jewish medical and research center denver, colorado, united states e-mail: protzkos@njc.org received: 26 february 2008 accepted: 10 april 2008 © 2008 protzko. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – to evaluate the use of the cochrane library by librarians, health care providers and consumers in the canadian province of saskatchewan. design – volunteer telephone interviews and surveys of training participants at multiple time points; usage statistics. setting – saskatchewan. subjects – ninety-four volunteers participated in the study. participants were self-selected from approximately 300 health practitioners and 100 public library staff attending training sessions, located primarily in rural areas. the majority of public library staff who attended training sessions were not professional librarians, although 31.5% of the study participants were librarians. nurses made up the next largest group (16.3%), followed by therapists (7.6%), library support staff (5.4%), pharmacists (4.3%), physicians (3.3%), other health care providers (20.7%), and other (9.8%). most were 40-65 years of age (71.6%) and female (92.4%). methods – forty-six training sessions were provided upon request between october 2004 and december 2006. attendees were invited to participate in the study. telephone interviews were conducted at three, six, nine, and twelve months 43 following training sessions. demographic information and data on the use of and satisfaction with the cochrane library were collected. additionally, monthly statistics were tracked by wiley-blackwell for user sessions, number of searches, and the number of full-text articles and abstracts visited. main results – telephone interviews revealed that 65.2% of participants had accessed the cochrane library at three months; 64.2% had at six months. at nine months access dropped to 45.2%. at twelve months only 27.4% of participants reported using the resource. of those who used the cochrane library, 16.4% reported at the threemonth interview that it was not helpful. this number decreased at six months (11.6%), nine months (7.7%) and twelve months (11.8%). 57.5% of respondents claimed to have learned something from the cochrane library, although a few (11.1%) reported that the information found had no impact. others reported that the knowledge gained confirmed their beliefs (26.1%) and/or helped in decision-making (32.6%). no time points were reported for the data collected about the use and helpfulness of information found in the cochrane library. three-year data from wiley-blackwell showed that the cochrane database of systematic reviews was most frequently accessed (abstracts=26,016; full texts=15,934). the cochrane central register was accessed 5,640 times and database of abstracts of reviews of effects was accessed 1,612 times. periods of low usage corresponded with summer and christmas breaks. the type of search strategy used was tracked; the authors note that an emphasis on mesh during training between october 2004 and december 2006 corresponded with the higher number of mesh searches during the same time period. participants reported using the cochrane library in response to patron requests, to prepare educational materials, and to support health care policy and practice changes. reasons for not using the cochrane library included lack of time, limited access to the internet, forgetting how to find and use the web site, and disappointment with the content. conclusion – since the fall of 2004, the cochrane library has been promoted and made available free of charge to all saskatchewan residents. usage fluctuates during the year, with less use during the summer and winter holidays; it is reasonable to presume that students use the cochrane library during the academic school year. most telephone interviewees who used the cochrane library reported that it was somewhat to very helpful; this number increased slightly over time while the number of respondents who used the resource fell measurably over twelve months. in other words, those who continued to use the cochrane library over time were more likely to report a higher level of satisfaction with the resource. interviews indicated how librarians used the cochrane library, why they do or do not use the resource, and their level of satisfaction. the study revealed less about how others, such as practitioners or consumers, use the resource. based on the limits of the telephone interviews, follow-up studies should try to capture more detailed usage data to describe the attributes of those who do and do not use the cochrane library. the authors note that additional data collected through online surveys or the wiley-blackwell website could help determine how to sustain use of the resource. commentary the study aimed to evaluate the use of the cochrane library, a premier information resource for evidence-based health care. the study will be of limited interest to those who provide access to evidence-based health care resources. this article thoroughly describes the context of the 44 study, detailing the process of access, promotion and training. the methodology employed included usage data and telephone interviews and surveys of volunteer participants. three years of quantitative data collected by wileyblackwell show a pattern of usage; the authors surmise that dips in usage may correlate to school and vacation calendars, but it is also possible that spikes correlate to training. the data on mesh searching do seem to suggest that training increases the use of mesh. the data would have been more interesting if the authors had compared usage before and after provincial access, promotion and training. a mixed-methods study was appropriate for the research question, however one point is confusing. as a whole, health care workers made up over 50% of respondents, while they made up about 75% of trainees; a table might have helped readers understand why the authors concluded that health care workers were not well represented in the qualitative data. librarians represented 31.5% of participants. public library staff (the majority of whom were not professional librarians) represented 25% of training attendees the data do provide a perspective of female librarians ages 40 to 65, in rural areas. however it would have been valuable to ascertain what kind of librarians had used the cochrane library. participants both requested training sessions and volunteered to be interviewed, reporting of telephone interview results was murky; it is difficult to extract discernible meaning from the numbers as reported, and the authors could have used a table for clarification. likewise, the authors recognise that a more detailed survey would have been useful. for instance, the question of helpfulness might have been better developed; helpfulness means different things to different users. the authors note that the plain-language summaries were well received while some reported that the reviews were more appropriate for researchers and health care workers than the general public. the questionnaire did not address plain-language summaries so it is unclear why the authors have included the commentary. as noted by the authors, saskatchewan’s health quality council provided funding for the licence, promotion, training, and evaluation. the study augments our awareness of why users, including librarians, do and do not use the cochrane library, and their level of satisfaction with the resource. unfortunately the study has flaws and leaves many questions unanswered. it would be interesting to explore the impact of universal access. what is the cost per use for universal access, and is it justified? how effective was promotion in reaching those who did not previously have access? the cochrane library is highly regarded resource for health care providers, librarians and consumers alike. this study should prompt further evaluation of user behavior and the benefits and limitations of free access. evidence based library and information practice evidence based library and information practice 2011, 6.4 161 evidence based library and information practice using evidence in practice problem based learning and evidence based medicine: utilizing the librarian trina m. fyfe northern health sciences librarian northern medical program, university of northern british columbia prince george, british columbia, canada email: fyfet@unbc.ca geoffrey w. payne regional associate dean (interim) & associate professor northern medical program, university of northern british columbia prince george, british columbia, canada email: payneg@unbc.ca received: 03 oct. 2011 accepted: 04 nov. 2011 2011 fyfe. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. setting in 2004 the university of british columbia (ubc) embarked on a distributed undergraduate medical education program (bates, 2008). the distribution of the undergraduate medical program involves a multi‐institutional partnership with four institutions across the province: the northern medical program (nmp) based at the university of northern british columbia (unbc) in prince george, the island medical program (imp) based at the university of victoria (uvic) on vancouver island, the vancouver fraser medical program (vfmp) based at ubc in vancouver, and as of fall 2011, the southern medical program (smp) based at ubc okanagan in kelowna. since the start of the program, the number of nmp students grew from 24 to 32 students in just 3 years (bates, 2008). the librarian for the nmp began shortly after the distribution of the program to unbc in prince george. over the years the nmp librarian has offered support to students in various ways through teaching, collection development, library advisory committee with faculty and student representation from each year, and traditional reference service offered both in the library and onsite in the medical building. the print collections are integrated into the main library at unbc, as there is not a separate medical library (fyfe, mcdavid, raworth, & snadden, 2009). faculty support for the nmp librarian in regards to professional development and the further growth of the librarian’s role in the program https://webmail.library.ualberta.ca/exchange/denise.koufogiannakis@ualberta.ca/fyfet/local%20settings/temporary%20internet%20files/content.outlook/qr8yyqyf/payneg@unbc.ca� mailto:payneg@unbc.ca� evidence based library and information practice 2011, 6.4 162 has always been encouraging; specifically, in regards to exploring methods of developing librarian‐student relationships. problem evidence based medicine (ebm) developed out of the fundamental need for clinicians to apply critical appraisal at the bedside, and thus became a philosophy of practice (guyatt & rennie, 2002) involving the skill of using best evidence to make patient care decisions (sackett, rosenberg, gray, haynes, & richardson, 1996). the term was first coined by the mcmaster university faculty in regard to medical practice and has since evolved beyond the scope of just medicine, now being used in other multiple health professions (guyatt & rennie, 2002). over the past two decades, the process for practicing ebm has undergone modifications but the fundamental principles of the five steps remain the same: 1) asking focused questions, 2) finding the evidence, 3) critical appraisal, 4) making a decision and 5) evaluating performance (”what is ebm?”, 2011). based on the amount of health information available, which has grown exponentially since the beginning of the ebm movement, ebm skills have become essential. this has proved challenging because the methods to obtain this information have evolved and changed in response to this growth (dawes, et al., 2005). not only have we seen the expansion of journals offering online full‐text of articles, one can now search specialized databases created specifically for certain specialties (dawes, et al., 2005) in response to evidence based medicine and “push” or “pull” knowledge dissemination and sharing (montori & guyatt, 2008). we have also seen attempts to guide clinicians in finding information in a strategic manner and consequently have seen the development of the hierarchy of evidence (guyatt & rennie, 2002), the “4s” model (haynes, 2001), the “5s” model (haynes, 2006) and now the “6s” model (dicenso, bayley, & haynes, 2009) of accessing pre‐appraised evidence. all of this is challenging to educators in undergraduate medical education and librarians attempting to teach students the skills required to practice ebm throughout their undergraduate education and beyond. as diao, galm, and shamon (2009) noted, “over the past decade, ebm has become increasingly integrated into the curricula of many medical schools. this change has been shown to improve students’ ability to develop clinical questions and perform effective literature searches” (p. 17). this resonates with one of guyatt’s (1991) originating ebm statements that “for the clinician, evidence based medicine requires skills of literature retrieval, critical appraisal, and information synthesis.” the struggle has been how to provide students with the skills necessary to become clinicians effective in practicing ebm. within the current university of british columbia undergraduate medical program (mdup), ebm and informatics are integrated themes in the four year curriculum. although there are pockets of each theme taught by faculty and librarians throughout the four year program it is difficult to ensure continuity. problem based learning (pbl) case developers and tutors are all encouraged to integrate ebm into cases and sessions (gill, bradley, & godolphin, 2009) but it remains unclear if this is being done in a consistent and regulated manner. according to the canmeds 2005 physician competency framework physicians should be able to “critically evaluate information and its sources, and apply this appropriately to practice decisions” (frank, 2005). could a librarian, in the role of a pbl tutor, have an effect on students’ ebm behaviour? would this be another teaching opportunity for a librarian in regards to ebm? another aspect of interest was the potential impact on librarian‐student relationships within the nmp. as spak and glover (2007) noted in their evaluation of the personal librarian program at the cushing/whitney medical library, librarians had been evidence based library and information practice 2011, 6.4 163 concerned about the declining contact with medical students due to the continued growth of the online world and the personal librarian program was the answer. the evaluation process revealed development of librarian‐ student relationships. these relationships became clearly defined, including what students could expect from a librarian, knowing who they can contact, the ability to obtain timely feedback from students, and, in some cases, the relationships continued after graduation. (spak & glover, 2007) at nmp it was felt that the librarian as a pbl tutor could have this same effect on the development of librarian‐student relationships; meaningful and sustainable relationships. could a librarian be an effective pbl tutor? what kind of tutor evaluations would the librarian receive from the students? would the librarian receive similar feedback from the students as other pbl tutors? it was decided to expand the librarian’s role to include pbl tutoring. evidence since the implementation of pbl into undergraduate medical school curricula across canada and abroad, librarians have responded to this curricular change by implementing creative and unique programs to ensure medical students are developing lifelong learning skills to be able to meet ebm competencies. pbl involves a process of inquiry that requires students to identify problems, develop hypotheses, describe mechanisms, gather information, acknowledge limits of knowledge, and create learning issues accordingly (office for faculty development and educational support, 2009). pbl cases progress throughout the week requiring students to utilize a systematic approach to generating hypotheses and applying information. lectures, labs, and learning issues provide the students with the knowledge required to approach each case. pbl has been supported by librarians in various ways including tutoring, co‐tutoring, collection development, the creation of toolkits, and liaison roles that involve student and tutor support (eldredge, 2004; eldredge, et al., 1998; fitzgerald, 1996; ispahany, et al., 2007; koufogiannakis, buckingham, alibhait, & rayner, 2005; satterthwaite, helms, nouravarsani, van antwerp, & woelfl, 1995). however, these early attempts to utilize the librarian within the pbl curriculum were not as focused as the one described in this paper. much of the literature was written in the mid‐ 1990s in response to the shift in integrating pbl into undergraduate medical education (rankin, 1996). eldredge’s (2004) article on tutoring in pbl is the only article found that depicts a librarian in a sole tutoring pbl role, similar to the one described in this paper. there is a great deal of literature looking at ebm and pbl, and the librarians’ role in teaching and support in ebm for undergraduate medical programs. based on the qualitative comments provided in the tutor evaluations, there is evidence that a librarian as a pbl tutor can have an effect on the students’ knowledge and demonstration of ebm behavior. a trial done at the university of alberta compared pbl groups with librarians as co‐tutors to groups without, in regard to knowledge about health information. based on a pre‐ and post‐test given to the students in each group, the trial found that the groups with librarian co‐tutors had a significant impact on the level of health information knowledge (koufogiannakis, et al., 2005). student evaluations of the librarian as a pbl tutor provide evidence that a librarian acting as a sole tutor can have an effect on the students’ engagement in ebm behaviour. in 1993, vernon (1995) conducted a survey of pbl tutors within 22 medical schools across north america in order to assess likes and dislikes about tutor pbl experiences compared to traditional medical education. he found one of the most rewarding aspects of tutoring pbl was building tutor‐student relationships (vernon, 1995). at nmp although relationships are made through traditional librarian‐student interactions, such as teaching and research consultation, a evidence based library and information practice 2011, 6.4 164 relationship built as a faculty member could have the potential for sustainable relationships, and to further develop our roles in ebm and as faculty members. eldredge’s (2004) experience certainly attests to this, as he describes how his role as a tutor increased the level of respect received from his fellow library and non‐library faculty members within the program. as a sign of this respect, elderdge received the designation of “master tutor” from his peers, as well as an acting role as ebm chairperson, and was an appointed member on a curriculum committee (eldredge, 2004). the same could be said for developing librarian‐student relationships. satterthwaite, et al. (1995), found that the employment of librarians as small group facilitators for the unique integrated clinical experience resulted in a higher number of students exploring the services and resources available at the library. building relationships early in a student’s education has the potential for expanding beyond the undergraduate program, into their residencies, and beyond. further, the previously mentioned personal library program at the cushing/whitney medical library found that the relationships the librarians built with students lasted beyond graduation, which is important for continued lifelong learning (spak & glover, 2007). it is, however, too early to tell if that will be the case in this study. in previous years, the librarian held regular reference hours in the health sciences centre which was anecdotally deemed unsuccessful, because students did not interact with the librarian in regard to asking the librarian questions, which was expected and is what the literature reveals (tao, mccarthy, krieger, & webb, 2009). although anecdotal, since the nmp librarian’s involvement as a pbl tutor began, the students have begun seeking her out for assistance with ebm and informatics questions. the literature on pbl tutors divulges a breadth of discussion about the type of background a tutor needs in order to facilitate pbl (schmidt & moust, 1995; schmidt, van der arend, moust, kokx, & boon, 1993). barrows (1992) maintained that an ideal tutor would be both an expert in the subject area and a good facilitator; however, barrows’ realized this is not always realistic and concludes that tutors without subject knowledge can be excellent facilitators. thus the nmp tutor’s role focused on facilitation of the pbl process, regardless of content expertise to highlight the following characteristics (office for faculty development and educational support, 2009): • ensuring all students are involved in the group process. • providing guidance to ensure that the learning process progresses and students meet case objectives. • monitoring the learning progress of the group and each student. • probing student knowledge using guiding questions. • managing group dynamics. • assisting students in developing independent, self‐direct learning skills. based on the existing evidence in the literature, it was hypothesized that a medical librarian would be an effective pbl tutor based on ebm expertise, critical thinking skills, previous experience in small group learning, communication skills, and the ability to negotiate different learning styles and abilities. implementation the nmp librarian became a pbl tutor in the fall of 2007, beginning with a second year endocrinology & metabolism five week block. in preparation for tutoring, the librarian took a two‐day pbl tutor training course offered by the faculty of medicine at ubc. this course provided the librarian with additional facilitation skills required for tutoring pbl. the librarian also engaged in tutor shadowing prior to the block in order to feel confident in a pbl setting. currently, the librarian has been a tutor for four first and second year pbl blocks: one second year endocrinology & metabolism block, two first year cardiology blocks, and one second year reproduction block. it must evidence based library and information practice 2011, 6.4 165 be noted that the librarian is assigned as a tutor to random blocks based on tutor scheduling and librarian availability. students evaluate tutors at the end of each block using an assessment tool prescribed by the program. each tutor is evaluated by each student in the respective pbl group based on the following criteria, plus any additional comments regarding strengths and suggestions for improvement: • ensured a safe learning environment and encouraged critical thinking (e.g., courteous, helped group adhere to ground rules, modeled constructive communication, used open‐ended questions, encouraged critical evaluation of evidence). • held students and the group accountable (e.g., recognized need for additional external information, encouraged accountability for information). • facilitated individual and whole group functioning (e.g., provided adequate direction, helped identify & deal with tutorial functioning, encouraged participation, provided constructive feedback, kept group activity flowing). the student ratings and comments for the librarian were compared to those of other tutors in the same block for each of the four blocks. outcome based on the student evaluations of the librarian for each of the four blocks, the librarian received comparable ratings to other tutors. in addition to comparable ratings, the librarian garnered significant qualitative results from the evaluations. ebm is an encouraged curricular theme throughout the four year program. in pbl, case writers and block chairs develop cases often with ebm integrated into the week’s content. tutors are encouraged to engage their students in ebm practices, often requesting that students cite their sources and challenge conflicting knowledge found in the literature. in the small groups that the librarian tutored, the students made the connection between the librarian and the librarian’s role in ebm without it being explicitly stated. for example, some of the feedback received over the four blocks included: • “as a librarian, she encouraged us to evaluate evidence and site [sic] our sources.” • “i liked that she asked us for our sources once in a while, especially if there was conflicting information.” • “she was a strong advocate for sighting [sic] sources of information and encouraged us to seek information from a variety of different sources.” reflection expanding the role of the librarian within the nmp has involved exposing the librarian to first and second year medical students as a pbl tutor. the ratings received by the librarian from the students have been comparable to that of other tutors in each block. in other instances of small group facilitation, librarians discovered that the communication skills used during the reference interview are similar skills used in the facilitation of small group learning (miller, 2001; satterthwaite, et al., 1995). in reflection, the probing aspect of pbl tutoring is quite similar to the probing of knowledge librarians engage in during a research consultation. pbl requires a great deal of dedicated time from tutors for preparation, weekly tutor meetings, pbl tutorials and assessments of the students. other attempts of librarian tutor and co‐tutors in pbl, or similar small group learning environments, have reflected time as a challenge and barriers for librarians to become tutors (koufogiannakis, et al., 2005). vernon (1995) found that tutors’ disliked the time requirements for pbl. in this scenario, the librarian was able to negotiate reduced reference hours in order to meet the time demands of tutoring pbl as it was deemed valuable for the librarian to continue in this role. at first the librarian spent many hours evidence based library and information practice 2011, 6.4 166 preparing for each block and each week’s case in order to overcome the lack of subject knowledge. it was soon realized that the facilitation skills involved in tutoring is what is key to a tutor’s success within pbl. once this was realized and embraced, preparing for pbl became less and less of an onerous task. outside of the comparable tutor evaluations, the librarian became a “known” figure within the nmp amongst the students and faculty. through her role as a tutor, the librarian built sustainable relationships with the other tutors and students. meeting with the other tutors for the block was really helpful in that the librarian could ask clarifying subject related questions, ask for tips on providing feedback for students, learn how others deal with difficult group dynamics and share success stories. having contact with the students outside of “traditional” librarian roles has provided the librarian with great insight into the process of pbl, the students’ use of resources in that environment, and has opened another venue for discussion of resources. the librarian also developed relationships with the students not often found in intermittent interactions within a library setting. based on the librarian’s experience described here, it is noted that future areas of exploration and research could be done relating to librarians as pbl tutors in regards to relationship building and its impact on ebm, since this case study is based on one librarian’s experience as a tutor. it would also be helpful to evaluate whether students’ perceptions of librarians change with the exposure of a librarian as a pbl tutor in addition to traditional roles. furthermore, as the landscape of the healthcare environment evolves, we as educators need to adapt to better prepare our students for the future. one future direction would be exploring if having a librarian as a pbl tutor changes medical students’ future scope of practice and how they interact with their future patients. conclusion a great deal of literature exists regarding the librarian’s role within pbl, which depicts librarians supporting both students and faculty. however, little research exists regarding librarians as pbl tutors as depicted in this paper, and the impact this exposure has on librarian‐student relationships and ebm. based on the canmeds competencies (frank, 2005) described previously, this study highlights for the first time how the librarian can play a critical role in the development of lifelong learning skills. based on the experiences at the nmp, these librarians suggest that further involvement of the librarian in medical education should be explored as they provide a valuable and still underutilized role in medical training. currently the librarian continues to tutor pbl, has maintained relationships with students and other tutors, and has taken on other roles within the program. acknowledgement excerpts of this manuscript were published in the icml 2009 conference proceedings: fyfe, t. m. & payne, g. w. (2009). undergraduate medical education: redefining the role of the librarian. paper presented at the meeting positioning the profession: the tenth international congress on medical librarianship, brisbane, australia. retrieved 8 nov. 2011 from http://espace.library.uq.edu.au/eserv/ uq:179933/n7_6_fri_fyfe_174.pdf references barrows, h. s. (1992). the tutorial process (revised ed.). springfield, il: southern illinois university school of medicine. bates, j. (2008). medical school expansion in bc. bcmj, 50(7), 368‐370. retrieved 7 nov. 2011 from http://www.bcmj.org/sites/default/file s/bcmj_50vol7_1_expansion.pdf dawes, m., summerskill, w., glasziou, p., carabellotta, a., martin, j., hopayian, k., porzsolt, f., burls, a., & osborne, j. evidence based library and information practice 2011, 6.4 167 (2005). sicily statement on evidence‐ based practice. bmc medical education, 5(1), 1‐7. doi:10.1186/1472‐6920‐5‐1 diao, d., galm, b., & shamon, s. (2009). evidence‐based medicine: an introduction for medical students. ubc medical journal, 1(1), 16‐18. retrieved 8 nov. 2011 from http://ubcmj.com/pdf/ubcmj_1_1_2009 _16‐18.pdf dicenso, a., bayley, l., & haynes, r. (2009). accessing pre‐appraised evidence: fine‐tuning the 5s model into a 6s model. evidence based nursing, 12(4), 99‐101. retrieved 8 nov. 2011 from http://ebn.bmj.com/content/12/4/99.2.f ull.pdf eldredge, j. (2004). the librarian as tutor/facilitator in a problem‐based learning (pbl) curriculum. reference services review, 32(1), 54‐59. doi:10.1108/00907320410519404 eldredge, j., tea, j., ducharme, j., harris, r., croghan, l., & perea, j. (1998). the roles of library liaisons in a problem‐ based learning (pbl) medical school curriculum: a case study from university of new mexico. health libraries review, 15(3), 185‐194. doi:10.1046/j.1365‐2532.1998.1530185.x fitzgerald, d. (1996). problem‐based learning and libraries: the canadian experience. health libraries review, 13(1), 13‐32. doi:10.1046/j.1365‐ 2532.1996.1310013.x frank, j. r. (ed.). (2005). canmeds 2005 physician competency framework: better standards, better physicians, better care. ottawa: royal college of physicians and surgeons of canada. fyfe, t., mcdavid, k., raworth, r., & snadden, d. (2009). medical education distribution in british columbia: a thriving partnership. journal of the canadian health libraries association, 30(2), 47‐49. retrieved 8 nov. 2011 from http://pubs.chla‐ absc.ca/doi/pdf/10.5596/c09‐015 gill, g., bradley, a., & godolphin, w. (2009). tips for pbl case developers: how to incorporate evidence‐based practice (ebp) into your pbl case. in university of british columbia faculty of medicine. retrieved 8 nov. 2011 from http://www.med.ubc.ca/__shared/assets/t ips_for_ebp_in_pbl_cases_20074432.d oc guyatt, g. h. (1991). evidence‐based medicine. acp journal club, 114(2), a‐ 16. guyatt, g. h., & rennie, d. (eds.). (2002). users’ guides to the medical literature: a manual for evidence‐based clinical practice. chicago: ama press. haynes, r. (2001). of studies, summaries, synopses, and systems: the “4s” evolution of services for finding current best evidence. evidence based medicine, 6(2), 36‐38. doi:10.1136/ebm.6.2.36 haynes, r. (2006). of studies, syntheses, synopses, summaries, and systems: the “5s” evolution of information services for evidence‐based healthcare decisions. evidence based medicine, 11(6), 162‐164. doi:10.1136/ebm.11.6.162‐a ispahany, n., torraca, k., chilov, m., zimbler, e. r., matsoukas, k., & allen, t. y. (2007). library support for problem‐ based learning: an algorithmic approach. medical reference services quarterly, 26(4), 45‐63. doi:10.1300/j115v26n04_04 koufogiannakis, d., buckingham, j., alibhait, a., & rayner, d. (2005). impact of librarians in first‐year medical and dental student problem‐based evidence based library and information practice 2011, 6.4 168 learning (pbl) groups: a controlled study. health information and libraries journal, 22(3), 189‐195. doi:10.1111/j.1471‐1842.2005.00559.x miller, j. m. (2001). a framework for the multiple roles of librarians in problem‐based learning. medical reference services quarterly, 20(3), 23‐ 30. doi:10.1300/j115v20n03_03 montori, v. m., & guyatt, g. h. (2008). progress in evidence‐based medicine. jama, 300(15), 1814‐1816. doi:10.1001/jama.300.15.1814 office for faculty development and educational support. (2008). pbl general tutor training package 2008‐ 2009. in university of british columbia faculty of medicine. retrieved 8 nov. 2011 from http://www.med.ubc.ca/__ shared/assets/2008‐2009_pbl_tutor _trainig_package7285.pdf rankin, j. a. (1996). problem‐based learning and libraries: a survey of the literature. health libraries review, 13(1), 33‐42. doi:10.1046/j.1365‐ 2532.1996.1310033.x sackett, d. l., rosenberg, w. m. c., gray, j. a. m., haynes, r. b., & richardson, w. s. (1996). evidence based medicine: what it is and what it isn’t. bmj, 312(7023), 71‐72. doi:10.1136/bmj.312.7023.71 satterthwaite, r. k., helms, m. e., nouravarsani, r., van antwerp, m., & woelfl, n. n. (1995). library faculty role in problem‐based learning: facilitating small groups. bulletin of the medical library association, 83(4), 465‐ 468. retrieved 8 nov. 2011 from http://www.ncbi.nlm.nih.gov/pmc/arti cles/pmc226066/pdf/mlab00105‐ 0082.pdf schmidt, h. g., & moust, j. h. (1995). what makes a tutor effective? a structural‐ equations modeling approach to learning in problem‐based curricula. academic medicine, 70(8), 708‐714. retrieved 8 nov. 2011 from http://journals.lww.com/academicmed icine/abstract/1995/08000/what_make s_a_tutor_effective__a.15.aspx schmidt, h. g., van der arend, a., moust, j. h., kokx, i., & boon, l. (1993). influence of tutors’ subject‐matter expertise on student effort and achievement in problem‐based learning. academic medicine, 68(10), 784‐791. retrieved 8 nov. 2011 from http://journals.lww.com/academicmed icine/abstract/1993/10000/influence_of _tutors__subject_matter_expertise_on. 18.aspx spak, j. m., & glover, j. g. (2007). the personal librarian program: an evaluation of a cushing/whitney medical library outreach initiative. medical reference services quarterly, 26(4), 15‐25. doi:10.1300/j115v26n04_02 tao, d., mccarthy, p. g., krieger, m. m., & webb, a. b. (2009). the mobile reference service: a case study of an onsite reference service program at the school of public health. journal of the medical library association, 97(1), 34‐ 40. doi:10.3163/1536‐5050.97.1.006 vernon, d. t. a. (1995). attitudes and opinions of faculty tutors about problem‐based learning. academic medicine, 70(3), 216‐223. retrieved 8 nov. 2011 from http://journals.lww.com/academicmed icine/abstract/1995/03000/attitudes_a nd_opinions_of_faculty_tutors_about. 13.aspx what is ebm? (2011). in cebm: centre for evidence based medicine. retrieved 7 nov. 2011 from http://www.cebm.net/index.aspx?o=19 14 / evidence based library and information practice microsoft word class1760 evidence based library and information practice 2008, 3:3 94 evidence based library and information practice classic constance mellon demonstrated that college freshmen are afraid of academic libraries a review of: mellon, constance a. “library anxiety: a grounded theory and its development.” college & research libraries 47 (1986): 160-65. reviewed by: edgar bailey reference/instruction librarian and associate professor phillips memorial library, providence college providence, rhode island, united states of america e-mail: ebailey@providence.edu received: 15 may 2008 accepted: 29 august 2008 © 2008 bailey. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – to better understand the feelings of college freshmen engaged in their first research project using an academic library. design – interpretive study involving analysis of personal writing describing the students’ research process and their reactions to it. setting – a medium-sized public university in the southeastern united states. subjects – students in freshman english courses. methods – english instructors assigned students to maintain search journals in which the students recorded a detailed description of their research process and the feelings they experienced while conducting research. in addition, students had to write an end-ofsemester, in-class essay in which they discussed their initial reactions to the research project and how their feelings evolved over the semester. the journals and essays were analyzed using the “constant comparative” method developed by glaser and strauss to identify “recurrent ‘themes’” (161). evidence based library and information practice 2008, 3:3 95 main results – 75 to 85 per cent of the students reported feelings of “fear or anxiety” when confronted with the research assignment. more specifically, they expressed a sense of being “lost”. this feeling derived from four causes: “(1) the size of the library; (2) a lack of knowledge about where things were located; (3) how to begin, and (4) what to do” (162). spurred by the question of why students did not seek help from their professors or a librarian, mellon re-examined the data and uncovered two additional prevalent feelings. most students tended to believe that their fellow students did not share their lack of library skills. they were ashamed of what they considered their own inadequacy and were, therefore, unwilling to reveal it by asking for assistance (162). conclusions – the original objective of mellon’s study was to gain information that would be useful in improving bibliographic instruction in her library. the discovery of the extent of students’ apprehension when confronted with a library research assignment came as something of a surprise. mellon later noted that, at the time she was conducting her research, she first became aware of the symptoms of math anxiety and realized that they closely resembled those she had identified in students confronting a library research assignment. at that point she coined the now widely used term “library anxiety” (mellon, “library anxiety and the non-traditional student” 79). she further realised that the research on math anxiety suggested the syndrome could be at least partially alleviated by simply acknowledging its existence to students. as a result, instruction librarians began openly discussing the affective aspects of library research in their classes, assuring students that their feelings of apprehension were both “common and reasonable” (164). they also devoted more conscious effort to presenting themselves as caring and approachable people who genuinely understood students’ feelings and wanted to help them. in addition, english faculty began devoting more class time to teaching the research process, even spending some out-of-class time in the library working with reference librarians to assist students. commentary mellon summarized a very substantial twoyear study in a remarkably brief article which failed to provide some important details. for example, it is unclear exactly how many students participated in the study or whether they all provided all requested data. mellon is also not very specific in describing how she applied the constant comparative method. she never actively pursued the findings summarized in her seminal article, noting that “my personal research moved in another direction” (onwuegbuzie, jiao, and bostick ix). nevertheless, her work caught the attention of other researchers who would seek ways to test her theory empirically. the first of these was sharon l. bostick, then a doctoral student at wayne state university, whose dissertation research focused on developing a reliable and valid quantitative measure of library anxiety applicable to undergraduate and graduate students at all levels. the result of her research was the library anxiety scale, a 43-question, 5-point likert scale questionnaire. this instrument was designed to measure what bostick’s research had discovered were the five major factors affecting library anxiety: barriers with staff; affective barriers; comfort with the library; knowledge of the library, and mechanical barriers. (onwuegbuzie, jiao, and bostick 311-315). most subsequent research on library anxiety has been based more on bostick’s work than on mellon’s original study; in fact, the library anxiety scale “has been used in virtually every quantitative study” (onwuegbuzie, jiao, and bostick 1). two evidence based library and information practice 2008, 3:3 96 particularly prolific scholars have done much of the work: anthony onwuegbuzie, a professor in the education counseling and leadership department at sam houston state university and qun g jiao, a librarian at baruch college. together with bostick, in 2004 they co-authored a book, library anxiety: theory, research and applications, which summarizes the research findings to that point. the book describes in detail the development of the library anxiety scale, discusses the research based on it and offers suggestions for reduction or prevention of library anxiety. onwuegbuzie, jiao and others have studied both the symptoms and the effects of library anxiety. these studies have confirmed not only that library anxiety is prevalent but that it can have a debilitating effect on the quality of student research. expanding on mellon’s original work with college freshmen, researchers have examined the incidence of the syndrome in minorities and foreign students and looked for differences based on gender, age, educational level or learning style. numerous studies have used other measurement scales to correlate library anxiety with various personality traits. the correlation between library anxiety and other anxieties such as math, statistics, writing and foreign language has been examined. of course, this research would have little practical value if it did not provide some guidance in eliminating or at least reducing the extent and intensity of library anxiety. onwuegbuzie, jiao and bostick’s book includes a chapter on “prevention, reduction and intervention”. suggestions are grouped in three main areas: improvements to the physical environment, library instruction, and effective reference service. many of the suggestions related to environment and reference do not appear to be based primarily on the library anxiety research. virtually all of them have appeared frequently in the literature of academic librarianship and most fail to distinguish between those interventions focused specifically on library anxiety and those designed simply to improve the library experience for all users. in the area of library instruction, the authors stress, as did mellon, the importance of recognizing and deliberately addressing the affective elements of the research process in all library instruction. they also argue that specific, targeted instruction is most effective in reducing anxiety because anxiety is situation specific (onwuegbuzie, jiao, and bostic 257-261). a 2007 literature review indicates that the pace of research on library anxiety may have slowed somewhat since the publication of library anxiety: theory, research and applications (carlile). however, as with so many areas of library research, the question most requiring examination today is the impact of technology. onwuegbuzie, jiao, and bostick readily acknowledge this, noting the change from “the location specific library environment to more open, virtual information settings” and asserting that the library anxiety scale needs to be modified to reflect this change (onwuegbuzie, jiao, and bostic 289). do today’s students, many of whom display considerable, if often misplaced, confidence in their own ability to locate information online and who may conduct most if not all of their research without even entering the library, experience the same anxiety when confronted with a research assignment as did mellon’s poor freshmen over twenty years ago? this question deserves an examination it has yet to receive. works cited evidence based library and information practice 2008, 3:3 97 carlile, heather. “the implications of library anxiety for academic reference services: a review of the literature.” australian academic and research libraries 38 (2007): 129-47. mellon, constance a. “library anxiety and the non-traditional student.” reaching and teaching diverse library user groups. ed. teresa b. mensching. ann arbor, mi: pierian press, 1989. 77-81. onwuegbuzie, anthony j., qun g. jiao, and sharon l. bostick. library anxiety: theory, research, and applications. lanham, md: scarecrow press, 2004. evidence based library and information practice editorial responsibilities      2021. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     editor-in-chief: ann medaille   associate editor (evidence summaries): stephanie krueger, christina wissinger   associate editor (research articles): erin owens, lisl zach   associate editor (classics & reviews): jane schmidt   associate editor (commentaries, using evidence in practice): ann medaille   communications officer (news): kim mackenzie   production editor: rachel hinrichs   editorial intern: melissa cober   editorial advisors: alison brettle, michelle dunaway, lorie kloda, denise lafitte, heather pretty   copyeditors: heather healy (lead copyeditor), katelyn baroody, kirstin duffin, julie evener, emily kingsland, jane morgan-daniel, elizabeth stregger, nikki tummon, elaina vitale, ross wilson   editorial assistant: samantha sheplawy   evidence summary   an online community of data enthusiasts collaborates to seek, share, and make sense of data   a review of: stvilia, b., & gibradze, l. (2022). seeking and sharing datasets in an online community of data enthusiasts. library & information science research 44(3). https://doi.org/10.1016/j.lisr.2022.101160   reviewed by: jordan patterson associate librarian a. p. mahoney library st. peter’s seminary london, ontario, canada email: jpatte46@uwo.ca   received: 16 nov. 2022                                                             accepted:  25 jan. 2023      2023 patterson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30280     abstract   objective – to understand the major activities, tools, sources, and challenges of online communities focused on datasets.   design – content analysis informed by activity theory.   setting – the r/datasets subreddit, a web forum for sharing, seeking, and discussing datasets.   subjects – 1232 “hot” or “top” discussion threads (1232 original posts and 6813 responding comments) first posted between 2010 and 2020.   methods – the researchers used reddit’s api to collect their sample of threads. using a random subset of the sample, the researchers developed a coding scheme for content analysis, which identified major themes in the data. through this process, they controlled for quality: each researcher coded half the subset independently, then together evaluated their intercoder reliability and discussed and resolved disagreements. the researchers also employed labelled latent dirchlet allocation to construct topic models corresponding to the theme’s manual content analysis, which produced profiles of the top 100 terms most likely to appear in that topic. finally, the researchers extracted urls from threads in the sample to ascertain types of information and data sources used by the community. presenting their findings, the researchers discussed notable themes and proposed a metadata model for describing datasets, the data q&a metadata (dqam) model.   main results – the r/datasets community engages in three distinct activities: asking and answering questions, disseminating information, and community building. the closely related q&a and dissemination activities shared themes of obtaining and aggregating data, sensemaking, collaborating and crowdsourcing, and data evaluation. community members frequently discussed tools, competencies, and sources for data work. major challenges for members of the community related to the general themes of data quality, accessibility, ethics, and legality. a proposed 16-element metadata schema should meet the needs of data enthusiasts.   conclusion – the content analysis reveals a dedicated community engaged in an array of data-seeking and data-sharing activities. data producers should be mindful of how their data can be accessed and used outside of their original professional or scholarly contexts.   commentary   where once datasets were the preserve of ivory-tower statisticians and scholars, the tech competencies of digital natives and a trend toward openness in the information age have made data wrangling a viable hobby, to which the existence of the subreddit r/datasets stands as a testament. the great proliferation of data has been attended by a movement to regulate and systematize the sharing and seeking of data. in canada, for instance, federal fund-granting agencies cihr, nserc, and sshrc (2021) require researchers to have research data management (rdm) plans in place in order to qualify for grants, hastening the development of data infrastructure online. the researchers ask how data enthusiasts operate in this landscape.   assessed with glynn’s (2006) critical appraisal tool, this study meets commonly accepted standards of validity. the researchers took evident care in their research design to ensure their content analysis coding scheme was reliable and free from bias; for instance, in developing their coding scheme, the researchers first worked independently to identify themes before coming together to compare their results and generate a final set of thematic categories.   the researchers’ dqam model is sure to be welcomed by casual and professional data enthusiasts alike, but it enters the study abruptly. a short literature review covering the state of dataset metadata schemas would have been a welcome addition. in writing about r/datasets, the researchers focus on a novel, non-academic context, but the problems they uncover in their study are not so novel—the issues identified by data enthusiasts plague data professionals as well, and have for some time. the researchers propose their own solution to these ills with the dqam model, but they could have fruitfully considered what other solutions were on offer first; for instance, they could have described how major players in other more typically academic areas such as rdm have approached the issue of dataset metadata.   for one example, the dataverse project (institute for quantitative social science [iqss], n.d.) “is an open source web application to share, preserve, cite, explore, and analyze research data” that has seen great adoption in the scholarly realm, with harvard at the forefront. when submitting their datasets to an instance of dataverse, researchers must input descriptive metadata, and for this purpose the project’s documentation includes a dataset citation metadata schema with 78 elements (iqss, 2019). how does the dqam model differ from this earlier dataverse schema? what are the merits and deficiencies of each compared to the other? what makes dqam stand out? the need for a metadata model custom-built for the non-academic context may exist, but it seems like an oversight to present the dqam model without discussing other dataset metadata schemas first.   looking to future studies, the researchers hope to “expand and triangulate” their findings by interviewing r/datasets community members. the demographic composition of this community would be a compelling question for further research. there seems to be an implicit assumption in this study that community members are just data enthusiasts, and it would be interesting to know how many also operate in a professional or scholarly context. the findings of this study demonstrate that r/datasets community members are highly capable, technically proficient, and ethically concerned -hallmarks of a commitment to data beyond the amateur. ultimately, this study underscores the need for researchers and other data-producers to handle their data with care; there is no telling where their data may end up.   references   canadian institutes of health research (cihr), natural sciences and engineering research council of canada (nserc), & social sciences and humanities research council of canada (sshrc). (2021, march 15). tri-agency research data management policy. government of canada. https://science.gc.ca/site/science/en/interagency-research-funding/policies-and-guidelines/research-data-management/tri-agency-research-data-management-policy   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154   institute for quantitative social science. (n.d.). about. the dataverse project. https://dataverse.org/about   institute for quantitative social science (2019, november 21). dataverse documentation v.4.18.1: user guide: appendix. the dataverse project. https://guides.dataverse.org/en/4.18.1/user/appendix.html   stvilia, b., & gibradze, l. (2022). seeking and sharing datasets in an online community of data enthusiasts. library & information science research 44(3). https://doi.org/10.1016/j.lisr.2022.101160   microsoft word comm_grefsheim.doc evidence based library and information practice 2007, 2:3  123 evidence based library and information practice     commentary    making a commitment to eblip: the role of library leadership      suzanne f. grefsheim, med, msls   director, nih library  national institutes of health  bethesda, maryland, united states  e‐mail: grefshes@mail.nih.gov    jocelyn a. rankin, phd, mln  chief, public health library and information center  centers for disease control and prevention  atlanta, georgia, united states   e‐mail: jrankin@cdc.gov    susan c. whitmore, ms, msls  chief, information and education services branch, nih library  national institutes of health  bethesda, maryland, united states  e‐mail: whitmors@nihrrlib.ncrr.nih.gov      received: 14 june 2007    accepted: 04 july 2007      © 2007 grefsheim, rankin, and whitmore. this is an open access article distributed under the terms of the  creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits  unrestricted use, distribution, and reproduction in any medium, provided the original work is properly  cited.      introduction    several commonly held assumptions form  the basis for the information profession’s  emerging commitment to advance evidence  based library and information practice  (eblip).  first, librarians should practise  according to the same principles advocated  for library users — that is, base their  information practice decisions on the best  possible evidence (dalrymple et al; sla  research committee and marshall; brice  and booth; eldredge).  second, the body of  evidence does not exist currently to make  finding evidence in the literature a realistic  possibility in most cases (booth; glynn).  third, many practising librarians are not  comfortable designing and conducting their  mailto:grefshes@mail.nih.gov mailto:jrankin@cdc.gov mailto:whitmors@nihrrlib.ncrr.nih.gov http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:3  124 own research (glynn).  and fourth,  employers need to create conditions that  encourage research in the workplace  (hallam and partridge; crumley and  koufogiannakis; cleyle).     we have focused on the fourth assumption,  that library management and administration  play a pivotal role in the success of  widespread adoption of eblip.  for  example, the association of research  libraries’ recent two‐year study on the  status of assessment in research libraries  found that certain organisational factors  appeared critical for effective, sustainable  library assessment projects. among these  factors were library leadership,  organisational culture, identifying  responsibility for assessment, library  priorities, sufficiency of resources, data  infrastructure, and assessment skills and  expertise (hiller et al).    in their frequently cited article, hallam and  partridge suggested that employers have the  following responsibilities for eblip:  providing opportunities and resources;  encouraging ongoing learning to maintain  or develop skills; providing training  opportunities for graduate students in  library and information science (lis);  advising graduate program lis faculty on  needs; serving as guest lecturers for lis  programs; including evidence based practice  as part of staff appraisal; encouraging more  experienced staff to mentor novices; and  providing opportunities for collaboration  with academic researchers (hallam and  partridge). we believe that, to control their  own destiny, library leaders must be much  more active in advancing eblip among  their staff.  on a broad range of issues, they  need to develop the evidence that supports  shifts in resources and services and justifies  emerging roles and innovative/experimental  services, or they risk having decisions about  their future made for them by others in their  organization.   employers taking a proactive role in  creating an eblip culture benefit at several  levels, both in more robust customer  support provided by staff members and also  in strengthened capacity to respond to  organizational requirements.  health science  librarians with working knowledge of  evidence based methods are well prepared  to support evidence based practice for  clinical and research staff in their larger  institution.  they are also inclined to engage  in evidence based information practice to  inform their own practice‐related decision‐ making.  at the organisational level, in  today’s rapidly evolving information  environment, libraries are routinely  required to conduct studies to assure they  are both meeting the needs of users and  achieving the expectations of their parent  organization in order to justify continued  funding.  typical studies include customer  needs assessments and satisfaction surveys;  usability studies; program evaluation,  outcomes and impact studies; systematic  performance measurement approaches such  as the “balanced scorecard”; cost/benefit  and trend analyses.  while hiring an outside  consultant is an option for large‐scale  studies, employers also need internal staff  with the knowledge and skills to design,  conduct and appraise studies for the wide  range of questions that routinely arise in the  practice of librarianship.      one library’s commitment to research    we offer the experience of the national  institutes of health (nih) library in  bethesda, md as an example of what  employers can do to better prepare their  staff to find the evidence to answer  questions that arise in practice and perhaps  even help assure the future of the library.   the nih library is a biomedical research  library with a staff of 56 full‐time employees  and 20 contractors that supports a major u.s.  government agency engaged in translational  bench‐to‐bedside research.  the nih itself  evidence based library and information practice 2007, 2:3  125 has over 20,000 employees, about half of  whom are in the scientific and clinical  positions that constitute the library’s  primary user group.  users include  researchers and fellows in the laboratories  and clinics as well as the science  administrators working in the various grant  administration programs.  the nih  library’s virtual services and collections are  comparable in size and scope to a large  academic biomedical library.     just as is frequently the case with busy  health care practitioners (coumou and  meijman), practice‐related questions  encountered by nih library staff also were  often left unanswered.   only a small  number were addressed each year, and  those only if they fit into a regular user  survey — either the large‐scale rigorous  biennial user study or smaller annual web  or print surveys targeted to assess specific  services.     crumley and koufogiannakis categorized  “librarianship questions” as falling into one  of the following six domains of practice  (crumley and koufogiannakis):  • reference/enquiries – providing  services and resources that meet the  needs of users  • education – finding the best  methods to educate users  • collections – building high‐quality  collections that meet the needs of  users  • management – managing people  and resources within the  organisation  • information access and retrieval –  creating better systems   • marketing and promotion –  promoting libraries to users and  non‐users    questions in all these domains had arisen at  the nih library in recent years.  the library  leadership team realized that informing  library practice by surveys alone would not  produce answers to many of the questions.   this recognition of the need to augment  practice research was reinforced by the  results of an environmental scan of the nih  that was commissioned as part of the  preparations for the revision of the library  strategic plan.  influential stakeholders at  the highest levels within nih were  interviewed as part of the scan. one of the  key findings noted by the consultant who  conducted the interviews was that the  library staff needed to be doing its own  research into user needs and information  seeking behavior. as a result, evidence  based information practice became a major  focus of the library’s vision statement in the  2005‐2009 strategic plan.      positive environmental factors     to create the conditions that would make  our vision a reality, the nih library  leadership team planned and implemented  a systematic approach to fostering eblip  that was compatible with the library’s long‐ standing commitment to be a learning  organization. the goal was to enable staff to  routinely apply evidence in decision‐making.   it was determined that 38 staff members  were in positions that had potential to  benefit from eblip training as follows: all 32  librarians, four of the 10 library technicians,  and two of the four it professionals. their  work raised answerable library research  questions and they were in positions best  suited to answer them.      a survey of these individuals revealed a  typical health sciences library staff — quite a  few library staff members had scientific  research experience and more than half had  working experience with research initiatives.  however, few if any had ever designed the  type of study required for “librarianship  questions”.        evidence based library and information practice 2007, 2:3  126   motivational factors for doing eblip research                                                                                                        percent                     overall                                                                                                  ranking as 4‐5          rank order                                                                                                        (n=33)                  training  64%  4.84  opportunity to collaborate with library staff  61%  4.60  opportunity to present at professional  meetings  58%  4.52  release time  45%  3.68  mentoring  39%  3.96  opportunity to present to other library staff  33%  3.84  inclusion in performance appraisal  33%  3.68  special funding  21%  2.80    table 1.  on a scale of 1‐5, with 5 as most important, staff rated these motivational factors.      we also wanted to understand what might  help inspire nih library staff to perform  eblip research.  the leadership team asked  the staff about the environmental and/or  administrative factors that were most  important to them.  of the eight factors  suggested, training was most valued: all the  staff participating in the eblip project  wanted a basic introduction to eblip, and  training in quantitative and qualitative  research methods that could be used to  answer practice questions.  also important  to these staff were opportunities to  collaborate with colleagues and to present at  professional meetings. release time was also  highly valued by some, although overall it  fell into the middle range of the rankings.   the least motivating factors at our library  were special funding, presentations to other  library staff, and adding research as a  performance appraisal metric [table 1].  we concluded that an environment  supportive of eblip would include formal  training in research skills that builds on  existing expertise as well as support and  release time for research projects, and team  mentoring as projects evolved.  recognition  for staff efforts and accomplishments was  considered as well. a celebratory library  research festival day was discussed and  librarians were encouraged to submit their  projects for presentation in the larger annual  nih research festival, an annual three‐day  event with poster and paper sessions that  showcase current research of nih staff in all  the institutes and centres.     eblip training for library staff    prior to developing the training plan, library  leadership worked with the various staff  teams to identify several of the librarianship  questions that had arisen in the previous six  months.  these questions [table 2] provided  a focus for exercises used in both the  introductory research class and the  qualitative research methods course.    the first class provided an overview and  rationale of the characteristics of evidence  based research for information practice.  the  research steps were outlined, including  posing the question, selecting the  methodology and study participants,  analyzing the data and reporting the results.   using a research jumpstart worksheet that  took them through the steps in the research    evidence based library and information practice 2007, 2:3  127       recent practice questions at the nih library    domain  question and research method  information  access & retrieval  1. what features and capabilities do nih researchers want in a federated  search engine? method: exploratory formative research using focus groups  and key informants  reference  2. did selective purchase of articles vs. borrowing from other libraries impact  customer satisfaction with document delivery services? method: program  evaluation with a comparative study of customer satisfaction as measured by  an online survey administered for 2 weeks in december 2005 and for 2 weeks  in february 2007.  management  3. did the transition to paraprofessional staffing at the information desk  affect user satisfaction and/or staffing efficiencies? methods: two‐part  program evaluation with a comparative study of customer satisfaction with  information desk services as measured by a survey in february 2006 and  february 2007; and a cost/time analysis by sampling paraprofessional and  librarian support of desk services for one week every other month during  2007.  reference  4. what is the value of an informationist assigned to a health sciences team?  methods: two‐part exploratory research using the diary method to record  critical incidents related to their work both retrospectively and prospectively;  and key informant interviews with customers to discover researchers’  perceptions of informationists’ roles, and their contributions and challenges  as members of research teams.  collections  5. which subject areas of print monographs are the most frequently used at  the nih library? method: a baseline retrospective cohort study of  monographic collection use by examining 2006 circulation and interlibrary  loan borrowing records in the library’s integrated library system.      table 2.  eblip questions categorized by crumley and koufogiannakis’ practice domains.      process, participants broke into small  groups to develop their research questions.   after the training, teams worked with the  instructor to further refine their questions  and explore possible methods.  the  instructor was available to these groups by  e‐mail, phone and in person in the months  following the class to respond to questions  and monitor progress.  as all the research  studies progressed, the teams maintained  contact with the instructor and also  conferred with other experienced  researchers on staff who volunteered to   serve as mentors.    in contrast to the quantitative studies that  are very common in the nih clinical  research setting, eblip questions best  answered using qualitative research  methods raised a level of discomfort among  staff participants.  we decided to provide in‐ depth training in qualitative methods for  those team members currently planning  qualitative studies as well as for others who  might use these methods in future studies.   the result was a formal, university‐affiliated  evidence based library and information practice 2007, 2:3  128 graduate course in qualitative methods with  both on‐site and distance learning  components.  the course included lectures;  extensive readings; opportunities to practise  qualitative methods; assignments to design  and implement practical studies; and  assistance and mentoring with study design  and data collection, organization, and  analysis.  in addition, for complex  quantitative studies, statistical support was  available from course instructors and other  experienced researchers at nih.      conclusion  employers can and should do more than  enable and encourage their staff to engage in  evidence based practice.  by proactively  providing a supportive framework for  eblip practice, library leadership can  successfully engage staff in eblip thinking  and small research studies.  it is to the  benefit of the employer as much as the  employee that valid answers are found to  the questions that arise in practice.      is the nih library’s eblip initiative  sustainable?  will the team study results  make a difference in practice?  based on our  experience, we believe library staff teams  can be successful with eblip research  projects.  those teams that have completed  their studies can point to findings that  support their current practice and/or  suggest new directions.  by implementing a  process to 1) identify existing skills, 2)  understand motivational factors and  respond to these, and 3) provide a  supportive training and mentoring  environment, the nih library experience is  an example of a staff development program  designed to provide practitioners with the  knowledge, skills and attitudes to design  and conduct a variety of quantitative and  qualitative research studies that will answer  “librarianship questions”.      our experience indicates that librarians with  some training in the research process and  ongoing mentoring can indeed design and  conduct studies to learn valuable  information that will help them introduce  new services; improve collection  development; and better understand the  information‐seeking behavior and customer  services needs of users in general as well as  specific target groups so the right  information reaches the right person at the  right time.    acknowledgement    based on a presentation at the evidence  based library & information practice 4th  international conference, durham, north  carolina, u.s.a., 7 may 2007.      works cited  brice, anne and andrew booth. “practical  issues in creating an evidence base for  library and information practice.”  library and information research  news 30.94 (2006):52‐60.   booth, andrew. “what research studies do  practitioners actually find useful?”  health information and libraries  journal 21.3 (2004):197‐200.  cleyle, su. “ebl is hot, help it sizzle.”  evidence based library and  information practice 1.3 (2006): 95‐97.     coumou, herma c. h.and frans j. meijman.   “how do primary care physicians  seek answers to clinical questions? a  literature review.”  journal of the  medical library association 94.1  (2006):55‐60.      crumley, ellen and denise koufogiannakis.  “developing evidence‐based  evidence based library and information practice 2007, 2:3  129 librarianship: practical steps for  implementation.” health information  and libraries journal 19.2 (2002):61‐70.    dalrymple, prudence w., jacqueline d.  bastille, jana bradley, et al. (1995).  research policy statement of the  medical library association: using  scientific evidence to improve  information practice.  9 july 2007  <http://www.mlanet.org/research/scien ce2.html>  eldredge, jonathan d. “the challenges  ahead for evidence‐based  librarianship.” bibliotheca medica  canadiana 23.2 (2001):57‐60.   glynn, lindsay. “the current state of ebl.”  evidence based library and  information practice 1.2 (2006):1‐2.    hallam, gillian clare and helen partridge.   “evidence based library and  information practice: whose  responsibility is it anyway?”  evidence based library and  information practice 1.3 (2006):88‐94.    hiller, steve, martha kyrillidou and jim self.   “when the evidence isn’t enough:  organizational factors that influence  effective and successful library  assessment.”  evidence‐based library  and information practice 4th  international conference, durham,  north carolina, united states.  7 may  2007.  sla research committee and joanne g  marshall. (2003). “influencing our  professional practice by putting our  knowledge to work.” information  outlook 7.1 (2003):40‐4.       http://www.mlanet.org/research/science2.html http://www.mlanet.org/research/science2.html evidence based library and information practice editorial responsibilities      2019. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     editor-in-chief: lorie kloda   associate editor (evidence summaries): stephanie krueger, christina wissinger   associate editor (research articles): ann medaille, lisl zach   associate editor (classics & reviews): jane schmidt, lorie kloda   associate editor (commentaries, using evidence in practice): lorie kloda   communications officer (news): tatiana bryant   production editor: rachel hinrichs   editorial intern: kimberly mackenzie   editorial advisors: lindsay alcock, alison brettle, michelle dunaway, denise koufogiannakis, heather pretty   copyeditors: heather healy (lead copyeditor), georgianne bordner, julie evener, julie james, emily kingsland, alison moore, jane morgan-daniel, stacey l. penney, elizabeth stregger, nikki tummon, elaina vitale   indexing support: kate shore   evidence summary   full-time faculty view information literacy as important but are unlikely to incorporate it into their teaching   a review of: bury, s. (2011). faculty attitudes, perceptions and experiences of information literacy: a study across multiple disciplines at york university, canada. journal of information literacy, 5(1). retrieved from http://ojs.lboro.ac.uk/ojs/index.php/jil/article/view/pra-v5-i1-2011-1   reviewed by: eamon c. tewell reference librarian sarah lawrence college bronxville, new york, united states of america email: etewell@sarahlawrence.edu   received: 24 nov. 2012 accepted: 22 jan. 2013      2013 tewell. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to explore faculty attitudes towards information literacy (il); in particular, faculty perception of student il competencies, importance of il skills and instruction, and ideal means of planning and delivering il instruction.   design – online survey questionnaire.   setting – large public research university located in toronto, canada.   subjects – 221 full-time faculty.   methods – the author designed and distributed an online survey to all full-time york university faculty (n=1,451) in march 2007 using zoomerang software. the survey consisted of between 26 and 36 questions depending on responses selected by respondents, and included both openand closed-ended questions. the author hand coded the qualitative data and used spss to analyze the quantitative data. the survey had 221 usable responses giving a response rate of 15.2%.   main results – the study revealed a high degree of concern among survey respondents regarding undergraduate students’ information literacy skills, accompanied by a perceived gradual increase in il abilities corresponding to student year. faculty ranked each of the association of college and research libraries’ (acrl) information literacy standards for higher education as being extremely important. no acrl standard ranked below 6 on a scale of 1 to 7, suggesting full agreement with the value of il proficiency. of the faculty 78.7% felt that il education should be a joint collaboration between faculty and librarians. a considerable majority of respondents (81.7%) answered that il instruction should be required for all students. far fewer faculty incorporated il teaching in practice, with 52.9% engaging in il instruction and 47.1% not incorporating il instruction at all. of the faculty who incorporated librarian-led il sessions into their courses, 85% of faculty perceived a “substantial impact” or “some impact” on their students’ il competencies.   conclusions – the author concludes that this study adds evidence to the claim that a disconnect exists between faculty beliefs about the importance of il and their teaching practices. faculty consistently express concern regarding student il abilities and support collaborative il instruction, yet the rate of il integration within their classes remains low. the results corroborate that faculty perceptions and attitudes towards il remain relatively consistent when compared with other studies. the author recommends that librarians be flexible regarding il instruction models and encourage further investigation of faculty development models to achieve wider il integration. a stronger advocacy role is also advised to increase instruction opportunities and the promotion of information literacy at the institutional level. the author identifies four areas for future research, including examining why faculty do not incorporate il instruction into their classes, disciplinary differences in il attitudes and adoption, which il instruction models faculty view as most effective, and replication of this study to test generalizability. as of the study’s publication, the author was conducting a qualitative follow-up study in the form of semi-structured interviews with faculty.     commentary   as equipping students with information literacy skills becomes increasingly essential to accomplishing the academic library’s mission, lis (library and information studies) practitioners encounter the difficult question of how to best provide this instruction. one approach to this issue is to measure faculty attitudes towards il, thereby gaining insight into opportunities for collaboration or effective promotion. numerous lis researchers have examined faculty responses to information literacy instruction, beginning with amstutz and whitson’s 1997 survey of faculty and academic professionals. in the past five years the pace of research in this area has slowed, with only one other major study completed on faculty attitudes towards il instruction (dacosta, 2010). without similar research being recently undertaken, this paper makes an important contribution to the literature. though this research was conducted in 2007 and thus results in a less timely study, the topic at hand remains highly relevant.   the author presents ample context for the study and compares the findings with other researchers’ work in each section. it is not stated whether the research instrument was validated or received ethics board approval, which is problematic if lis research is to realize a more thoroughly developed and validated assessment of faculty attitudes. the methodology is otherwise clearly described and allows for replication, and the author includes a link to the full survey. the results section is thorough, providing selected quotes from respondents and descriptive statistics including appropriate charts and graphs summarizing the data. the author correctly observes that the vast majority of information literacy research is published by librarians, for librarians, and within lis venues, but this observation does not appear to be acted upon with this particular research.   one issue regarding the study’s validity is the low response rate. the choice of a survey questionnaire necessitates a self-selected population sample, which may result in respondents who have a pre-existing bias regarding the survey’s subject. despite this weakness, an online survey was an appropriate methodology to address the author’s research questions. the response rate of 15.2% is low for this type of research, a point which the author acknowledges but notes as being only somewhat lower than similar studies conducted. additionally, this response was observed by the author as being too small to allow for statistically significant analysis of the results. the response rate coupled with a potential self-selection bias is important to consider when evaluating the study’s results, but does not ultimately render the conclusions invalid.   one area for further research, addressed in this study to a minor extent, is the disciplinary differences in faculty attitudes towards information literacy issues. understanding faculty attitudes in general may be useful, but librarians responsible for liaising with faculty and providing course-integrated il instruction could benefit from additional discipline-specific data. the survey results indicate a difference in responses according to respondent gender, and the gender dynamic of faculty attitudes towards il could be an interesting avenue for additional study. qualitative research could begin to address the major question of why faculty do not incorporate il into their classes, a step which the author has taken after completing this survey by conducting semi-structured interviews with instructors. most significantly, the author describes several initiatives at her home institution that resulted from the survey data, making explicit the potential practice implications for librarians seeking to develop a more vibrant culture of information literacy at their institution.     references   amstutz, a., & whitson, d. (1997). university faculty and information literacy: who teaches the students? research strategies, 15(1), 18-25. doi: 10.1016/s0734-3310(97)90004-5   dacosta, j. w. (2010). is there an information literacy skills gap to be bridged? an examination of faculty perceptions and activities relating to information literacy in the united states and england. college & research libraries, 71(3), 203-222. retrieved 5 february 2013 from http://crl.acrl.org/content/71/3/203.full.pdf+html   ebl 101   research methods: systematic reviews   virginia wilson director, centre for evidence based library and information practice (c-eblip) university library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   originally published in: evidence based library and information practice, 8(3), 83–84. https://ejournals.library.ualberta.ca/index.php/eblip/article/view/20437/15753     received: 12 aug. 2013 accepted: 17 aug. 2013      2016 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   a systematic review is not merely a literature review. while we can be systematic in preparing a literature review, a systematic review is a research method used to address a specific research question. systematic reviews “present a comprehensive summary of research based knowledge that can aid both practitioners and policy makers in decision making” (brettle, 2009, p. 43). meta-analysis, “the use of statistical methods to combine results of individual studies,” may or may not be conducted in a systematic review (the cochrane collaboration, 2002). systematic reviews in lis research were practically unheard of just over a decade ago. however, as denise koufogiannakis found, they “have become an important source of information because they both synthesize the existing research on a topic, as well as critically appraise it and try to draw conclusions from the total body of quality research evidence” (2012, p. 91).   the cochrane collaboration and the campbell collaboration are two organizations which oversee the conducting of systematic reviews based on specific guidelines. the cochrane collaboration focuses on the health sciences, while the campbell collaboration works in the areas of crime and justice, education, international development, and social welfare. additionally, an individual researcher or research team can undertake a systematic review to attempt to answer a research question. khan, kunz, kleijnen, andantes explain that a “review earns the adjective systematic if it is based on a clearly formulated question, identifies relevant studies, appraises their quality and summarizes the evidence by use of explicit methodology. it is the explicit and systematic approach that distinguishes systematic reviews from traditional reviews and commentaries” (2003, p. 118).   the process by which to undertake a systematic review has several steps:   1.            develop a research question   as with any research project, the development of a clear, explicit, and concise research question is the bedrock upon which the project rests. spend time thinking and planning.   2.            identify relevant work   do an extensive search for research studies. you may be looking at a particular date range. once you have exhausted the literature, decide which papers to include and exclude based on criteria that have come out of your research question. you will probably get a lot of results from your initial massive searches and the criteria will help to sort out which studies belong in your systematic review. a test run of the inclusion and exclusion criteria will show if more or is needed or changes need to be made. record the reasons for inclusion or exclusion.   3.            critically appraise the included studies   you need to look for quality and rigour. quality is difficult to define but thorough critical appraisal can help to determine if the study results are sound. critical appraisal will also help to determine whether or not meta-analysis will be a part of your systematic review.   4.            extract the data   your research question will guide the types of data you will want to extract from the included studies. create a standardized data extraction form to keep track of everything you pull from the studies. methodically review the articles in order to fill out the extraction form.   5.            synthesize and analyze the findings   “the goal of data synthesis is to go beyond simply summarizing but to also include ‘an analysis of the relationships within and between studies and an overall assessment of the robustness of the evidence’” (centre for reviews and dissemination, 2009, 48 as quoted in phelps & campbell, 2012, p. 13).   the final and best thing i can pass along is that denise koufogiannakis along with several contributors put together a comprehensive list of all the systematic reviews undertaken in lis to date. the lis systematic reviews wiki includes the reviews listed alphabetically by author and classified by topic. there is also a nice bibliography where you can find much more information on the systematic review.     references   brettle, a. (2009). systematic reviews and evidence based library and information practice. evidence based library and information practice, 4(1): 43-50.   khan, k.s., kunz, r., kleijnen, j., & antes, g. (2003). five steps to conducting a systematic review. journal of the royal society of medicine, 96: 118-121.   koufogiannakis, d. (2012). the lis systematic review wiki. retrieved 4 sept. 2013 from http://lis-systematic-reviews.wikispaces.com/welcome   koufogiannakis, d. (2012). the state of systematic reviews in library and information studies. evidence based library and information practice, 7(2): 91-95.   meta-analysis. (2002). in the cochrane collaboration open learning material. retrieved 4 sept. 2013 from http://www.cochrane-net.org/openlearning/html/mod12-2.htm      phelps, s. f., and campbell, n. (2012). systematic reviews in theory and practice for library and information studies. library & information research, 36(112): 6-15.   editorial   evidence based library and information practice adopts author contributions statements   erin owens associate editor (research articles) professor / access services coordinator & scholarly communications librarian sam houston state university, newton gresham library huntsville, texas, united states of america email: eowens@shsu.edu   ann medaille editor-in-chief director of research and instructional services university of nevada, reno libraries reno, nevada, united states of america email: amedaille@unr.edu    2021 owens and medaille. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29940     since its founding in 2006, evidence based library and information practice (eblip) has valued the contributions of authors who advance evidence based practice in the field of library and information science. to better recognize the important scholarly work of our authors, eblip’s editorial board believes that author contributions should be more clearly described in co-authored publications. beginning with the march 2021 issue, author contributions statements will be included with each co-authored research article, review article, or using evidence in practice article.   because authorship is the result of considerable effort and intellectual activity, authors deserve to be recognized for their unique contributions to scholarship. but while it is common to think of an author as being synonymous with a writer, authorship of scholarly works typically includes many more tasks than writing. in addition, authorship of scholarly works comes with responsibilities. authors must ensure that their works are truthful and have been produced in a manner that is consistent with ethical standards. they are to be held accountable if questions arise as to the integrity of their assertions.   because of the complexity of authoring scholarly works, collaborations are increasingly common. when collaboration occurs, authors frequently contribute to a work in different ways and in different amounts in accordance with the strengths they bring to the endeavor. this, after all, is the nature of collaboration. the production of scholarship often involves the execution of several different complex tasks that occur over months and years, including research design, data collection, and data analysis, as well as writing and editing, just to name a few. authors often take different roles in executing these tasks, which come with different expectations. at times, collaboration can be challenging, but it often enables the execution of more complex research projects which contribute to evidence based practice.   because of the demanding nature of scholarship, authors’ distinct contributions deserve recognition. at many libraries and related institutions, authorship of scholarly works is an expectation of employment, and authors may need to demonstrate the contributions they make. however, a simple author list lacks transparency about these contributions. it is often difficult for a reader to know an individual author’s contributions by looking at the list of authors beneath an article title. in addition, eblip does not designate corresponding authors in publications and instead publishes email contact information for all authors in order to facilitate conversations among scholars. scholarly communications may be enhanced through greater transparency about author roles. for these reasons, eblip will now be implementing author contributions statements.   to provide a framework for our implementation, eblip has adopted the contributor role taxonomy (credit) from casrai. the consortia advancing standards in research administration information (casrai) is an international non-profit initiative, founded in canada, with membership comprised of research institutions and their partners. casrai’s goal is to build networks among stakeholders in the research life cycle and publish guidelines to help funders, vendors, and research institutions improve consistency, utility, and comparability in data collected from researchers (https://casrai.org/about/).    credit is one of the resources provided by casrai to further this goal of data consistency; though credit is currently managed as an informal standard, a push is underway for formal standardization at the national information standards organization (niso). the taxonomy describes 14 roles commonly filled by scholarly contributors, ranging from conceptualization (formulating the research aims) and methodology (development or design of methods, creation of models) to validation (verification of overall reproducibility) and writing. each contributor can occupy one or multiple roles, and each role can be assigned to one or multiple contributors. optional modifiers can be added to a role to indicate a contributor’s degree of contribution: equal, lead, or supporting. roles are listed in full and defined at https://casrai.org/credit/.    eblip has chosen to adopt credit, as opposed to a set of customized roles or definitions, in order to maintain consistency across scholarly publishing for a global readership and authorship. since 2014, credit has been adopted by plos one, cell press, mit press, oxford university press, duke university press, elsevier, f1000 research, wolters kluwer, springer, sage publishing, and many other publishers and partners/vendors. the taxonomy has also been integrated and made machine-readable in various submission systems, such as aries’ editorial manager and river valley’s review. in the likely event that the open journals system (ojs) also integrates support for credit, eblip will be well served by adoption of this standard as compared to any customized approach.    the eblip implementation of credit will involve an author contributions statement at the end of the article text before the references; the statement will be formatted with each author name appearing in bold, followed by a list of the appropriate contributor roles, as illustrated in the following examples:   example 1    zhang lee: conceptualization, methodology, software aditi singh: data curation, writing original draft neeru acharya: visualization, investigation noel jenson: supervision vijay kumar: software, validation caryn dillon: writing review & editing    example 2 (illustrating optional degrees of contribution)    pierro correia: writing review & editing (equal) anna berkowitz: conceptualization (lead), writing original draft (lead), formal analysis (lead), writing review & editing (equal) yolanda roberto: software (lead), writing review & editing (equal) takaaki yamada: methodology (lead), writing review & editing (equal) qian wu: conceptualization (supporting), writing original draft (supporting), writing review & editing (equal)    authors will be asked to provide this author contributions statement after a co-authored research article, review article, or using evidence in practice article has been accepted for publication in eblip; single-author articles will not require an author contributions statement. all relevant contributions should be included, even if a specific contributor is not included in the authorship byline (for example, someone named in acknowledgements could also be included in contributions). in developing the author contributions statement, it is essential that all contributors share responsibility for assigning, reviewing, and confirming roles and degrees of contribution. this is necessary to honor the transparency of the research and authorship process, and reaching consensus on contributions prior to publication will reduce the potential for post-publication author disputes or the misuse of coauthored works, in terms of an individual contributor taking credit for more than their share in the work. the more that the scholarly publishing ecosystem moves to clarify contributions in a transparent and standardized manner, the more we will drive the visibility and recognition of the diverse skills and experiences that individual researchers bring to the table.    since its founding as an open access journal in 2006, eblip has served as an open forum for the sharing of evidence based practices across all library sectors and borders to support the greater exchange of knowledge. eblip has instituted several practices that support this philosophy of openness. in addition to making all articles freely available, eblip also promotes open data—that is, the sharing of data supporting research published in the journal. to support the discoverability of authors’ works, eblip includes digital object identifiers for all articles and is now including orcid numbers for authors in order to improve discoverability. the inclusion of orcid numbers helps to ensure that authors’ works are correctly and consistently identified across their body of scholarship. eblip’s editorial board believes that the inclusion of author contributions statements is a logical and important extension of the journal’s commitment to openness and transparency in all areas of its practice.       microsoft word es_needham.doc evidence based library and information practice 2006, 1:1  96 evidence based library and information practice       evidence summary    undergraduate and postgraduate students in a north american university are  choosing to use chat reference services for all kinds of reasons    a review of: ward, david. “why users choose chat: a survey of behavior and motivations.” internet  reference services quarterly 10.1 (2005): 29‐46.    reviewed by: gill needham  head of strategic and service development, the open university library and learning  resources centre  walton hall, milton keynes, united kingdom  e‐mail:  g.needham@open.ac.uk    received: 19 december 2005 accepted: 21 february 2006     © 2006 needham. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to investigate reasons why  undergraduate and postgraduate students  choose to use chat reference services and  their satisfaction with the experience.    design – user survey.    setting – large academic library in the  united states.    subjects – a total of 341 library users took  part in the survey. of these, 79 were  graduate students, 215 undergraduates, 21  staff, 5 alumni and 21 members of the public.    method – a user survey form was posted  on the library website for a period of one  month (march 2003). users of the chat  reference service received an automatic  closing message asking them to complete  the survey.  responses were fed into a  database to be analysed by the research staff.   additional data was also gathered about the  behaviour of all users of the chat service  during that period – both responders and  non‐responders.    main results – the most popular reason for  choosing to use a chat service was the desire  for a speedy response, followed by ‘distance  from the library’. responders chose to use  chat to ask a range of different kinds of  questions. nearly half (45%) were looking  for a specific resource and 23% were seeking  help at the beginning of their research.  there were notable differences between  undergraduate and graduate students.  twice as many undergraduates as graduates  used the service to help them get started on  their research. graduates were three times  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:1  97 as likely as undergraduates to be using the  service to address technical problems they  were having with the website. overall  satisfaction with the service was extremely  high, with 77.5% rating it 5 out of 5 and  94.5% giving it 4 or 5 out of 5.      conclusion – chat reference services are  popular with users primarily because they  are quick and convenient. however, users  expect that these services will be able to deal  with all types of questions, including help  with research. this challenges an  assumption by library staff that the medium  is only suitable for short factual enquiries  and general questions about library services.  it is suggested that a detailed analysis of the  chat transcripts from this period could  provide a valuable addendum to the survey  results.    commentary    librarians carrying out user surveys  sometimes make the mistake of trying to  collect too much information. in this study  the researchers have chosen to focus on  ‘three major research questions’ (although  question 2 appears to incorporate two  research questions):    1. what are patrons working on before they  go to chat?    2. why do they choose chat…as a method of  getting help, and what kinds of questions  do they think chat is best suited for?    3. how well do they think chat works for  answering their questions? (30).    the author designed the online survey to be  short and simple in order to encourage busy  online users to complete it. this paid off  with a respectable 54.9% response rate. the  survey questions are provided as an  appendix. there are seven questions, one of  which is open‐ended and one requires a  yes/no response. the other questions invite  participants to choose from a list of options  (e.g. types of questions posed to chat  services) and we are told that these have  been extracted from an analysis of previous  chat transcripts.      the pilot survey was conducted during  march 2003 and during the study period 628  chats were conducted (345 responded to the  survey). there is no reason to believe that  these would not be representative of the  general population of users of the chat  service although it would have been useful  to know at what times of day and days of  the week the survey was active, as this could  have been influential. similarly, the timing  of the survey (in the context of the academic  year) needs to be taken into account when  considering implications of the results.     this research is timely since the use of  instant messaging is becoming increasingly  popular and more and more libraries are  either using the technology to deliver  services or are planning to do so. it would  have been useful to know how many  reference enquiries the library received in  total during the study period, to see how  many patrons did not choose to use chat,  but this was clearly outside of the scope of  the study. it would have been helpful to test  the survey responses against the chat  transcript data and the author does point  this out as an area for further study.  from  this report, the main message for those who  are reviewing their chat reference services or  setting up from scratch is to avoid pre‐ defining how the service will be used. for  many students this technology is second  nature (far more so than for the majority of  librarians) – they themselves will define the  use and future development of the service.        evidence summary   first-year students’ understanding of research and their information literacy skills change over time and in four different ways   a review of: kirker, m. j., & stonebraker, i. (2019). architects, renovators, builders and fragmenters: a model for first year students’ self-perceptions and perceptions of information literacy. the journal of academic librarianship, 45(1), 1-8. https://doi.org/10.1016/j.acalib.2018.10.009   reviewed by: alisa howlett coordinator, evidence based practice university of southern queensland library springfield, queensland, australia email: alisa.howlett@usq.edu.au   received: 17 mar. 2020                                                             accepted:  22 apr. 2020      2020 howlett. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29752     abstract   objective – to explore students’ perceptions of their information literacy skills and how these change during the first-year experience.   design – a longitudinal qualitative study using cognitive dissonance theory.   setting – two large public universities in the united states of america.   subjects – students enrolled in research methods and information literacy-based courses in their first semester.   methods – students were required to submit two written self-reflections as part of their course; the first was completed in the first two weeks of the semester and the second at the end of the semester. informed consent was obtained for all reflections used for the study. the authors selected 12 students (6 from each institution) to participate in semi-structured interviews at the end of their first year. a total of 178 self-reflections were included in the analysis.   main results – the study found that students’ understanding of research changed during the first-year experience, and that students had four main journeys related to their information literacy skills and perceptions. instances of cognitive dissonance were observed. students can consider themselves both good and bad researchers at the same time. the study also revealed the research process as an emotional labour, not just an intellectual one.   conclusion – the study concluded that a shared understanding of “research” between librarians and students is needed in order to teach information literacy effectively. it is also important to recognise that students transform their information literacy over time (not just from a single class or program) and that teaching needs to meet students where they are on their journey, depending on their “developmental paths.”   commentary   librarians are well-placed to see and understand connections (and their importance) between students’ development as scholars and competencies such as information literacy (kirker & stonebraker, 2019, p. 1). existing literature often evaluates students’ information literacy skills as outcomes of a one-off class, pedagogical tool or program (karshmer & bryan, 2011; kim & shumaker, 2015). little is known about the students’ perspective on research and information literacy skills within their overall study experience. kirker and stonebraker’s study, which focuses on the first year experience, begins to fill this gap.   this study was appraised using the critical review form by letts et al. (2007). a thorough and logical review of the literature creates a compelling argument for the study and defines its scope and its contribution to existing knowledge about the impact of information literacy instruction in academic settings. using the cognitive dissonance framework as a lens through which to view information literacy development is an approach unlike other studies. research questions and aims prescribe the longitudinal qualitative design. students who did not complete both written self-reflections were excluded from the analysis process. participants who were selected for the semi-structured interviews were representative of their institutions’ student populations, however kirker and stonebraker are careful to note that findings of the study cannot be generalised and applied to all student populations.   kirker and stonebraker provide the self-reflection prompts that were given to students and state that similar questions were asked at the semi-structured interviews. this is particularly helpful in understanding more about the data that was collected and how data collection was linked to the research questions. although a comprehensive description of the data analysis process is provided, including quote samples and tables to illustrate findings, the actual process that the authors used to explore and identify patterns and themes in the data was not defined. given the qualitative design, thematic analysis and constant comparison is therefore assumed and should be considered a limitation of the study.   findings of this study present evidence that students’ understanding of research and perceived information literacy skills evolve over time and so librarians cannot simply assume that students arrive at information literacy classes and programs with a blank slate. information literacy skills are then exercised, developed, and integrated throughout the broader picture of the student learning experience. kirker and stonebraker suggest that librarians meet students where they are on their information literacy journey to ensure instruction is effective.   kirker and stonebraker also identify emotional labour as another factor involved in students’ development of information literacy skills. this is something librarians may need to keep in mind when designing and implementing information literacy programs. while the researchers don’t comment on differences between the two student cohorts involved in the study, this may be worth exploring with further research in order to develop more targeted and refined ways in engaging faculty and students in information literacy instruction or programs.   references   karshmer, e., & bryan, j. e. (2011). building a first-year information literacy experience: integrating best practices in education and acrl il competency standards for higher education. the journal of academic librarianship, 37(3), 255-266. https://doi.org/10.1016/j.acalib.2011.02.018   kim, s. u., & shumaker, d. (2015). student, librarian, and instructor perceptions of information literacy instruction and skills in a first year experience program: a case study. the journal of academic librarianship, 41(4), 449-456. https://doi.org/10.1016/j.acalib.2015.04.005   letts, l., wilkins, s., law, m., stewart, d., bosch, j., & westmorland, m. (2007) critical review form – qualitative studies (version 2.0). region of peel. http://www.peelregion.ca/health/library/eidmtools/qualreview_version2_0.pdf   microsoft word es4_929_final evidence based library and information practice 2008, 3:1  65 evidence based library and information practice       evidence summary    varied search protocols lead to clinically relevant results    a review of:  patel, manesh r., connie m. schardt, linda l. sanders, and sheri a. keitz. “randomized trial  for answers to clinical questions: evaluating a pre‐appraised versus a medline  search protocol.” journal of the medical library association 94.4 (2006): 382‐6.       reviewed by:  marcy l. brown  principal consultant, envision research  delmont, pennsylvania, united states of america  e‐mail: marcy@envisionresearch.info      received : 2 december 2007      accepted : 11 february 2008      © 2008 brown. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    abstract    objective – to determine the success rate of  electronic resources for answering clinical  questions by comparing speed, validity, and  applicability of two different protocols for  searching the medical literature.     design – randomized trial with results  judged by blinded panel.    setting – duke university medical center in  durham, north carolina, united states of  america.    subjects – thirty‐two 2nd and 3rd year  internal medicine residents on an eight‐ week general medicine rotation at the duke  university medical center.     methods – two search protocols were  developed:    protocol a: participants searched  medline first, and then searched  pre‐appraised resources if needed.    protocol b: participants searched  pre‐appraised resources first, which  included uptodate, acp journal  club, cochrane database of systematic  reviews, and dare. the residents  then searched medline if an  answer could not be found in the  evidence based library and information practice 2008, 3:1  66 initial group of pre‐appraised  resources.    residents were randomised by computer‐ assisted block order into four blocks of eight  residents each. two blocks were assigned to  protocol a, and two to protocol b. each day,  residents developed at least one clinical  question related to caring for patients. the  questions were transcribed onto pocket‐ sized cards, with the answer sought later  using the assigned protocol. if answers  weren’t found using either protocol,  searches were permitted in other available  resources.    when an article that answered a question  was found, the resident recorded basic  information about the question and the  answer as well as the time required to find  the answer (less than five minutes; between  five and ten minutes; or more than ten  minutes). residents were to select answers  that were “methodologically sound and  clinically important” (384). ten faculty  members formally trained in evidence‐based  medicine (ebm) reviewed a subset of  therapy‐related questions and answers. the  reviewers, who were blinded to the search  protocols, judged the applicability and  internal validity of the answers.     results – in total, 120 questions were  searched using protocol a and 133 using  protocol b; 104 answers were found by the  protocol a group and 117 by the protocol b  group. in protocol a, 97 answers were  found in medline (80.8%) and six answers  were found in pre‐appraised resources  (5.0%). in protocol b, 85 answers were found  in pre‐appraised resources (64.6%) and 31  were found in medline (23.3%). uptodate  was the major resource for answers in  protocol b.    a statistically greater number of answers  were found in less than five minutes in  protocol b (p<0.00004). however, a greater  number of questions went unanswered in  protocol b (23%).    the faculty reviewers determined that 75.8%  of the answers in protocol a were  applicable; 71.7% were judged applicable in  protocol b. eighty percent in protocol a  were internally valid, while 81.8% were  valid in protocol b.    conclusion – the study demonstrates that  to find answers to as many clinical questions  as possible, “residents and clinicians need  access to both pre‐appraised resources and  medline” (385). pre‐appraised resources  were faster to search, but their coverage was  not as comprehensive. medline is  comprehensive, but takes more time to  search effectively and requires that the  clinician or student have some knowledge of  critical appraisal in order to judge relevance  and applicability of found articles. protocol  b may be difficult to implement in many  institutions because of the high cost of pre‐ appraised resources.    commentary     this paper adds to a small body of literature  looking at the use of either pre‐appraised  resources or primary literature databases to  answer clinical questions. a 2004 study at  vanderbilt found that synthesized resources  either completely or partially answered only  60% of complex clinical questions and 68%  of general practice questions (koonce,  giuse, and todd 409). this current research  at duke is a welcome addition to the  literature on clinical information seeking  because it directly compares synthesized  resources to medline.     the study’s chief finding reinforces what  many medical librarians already know: that  a variety of resources is necessary to answer  the largest possible number of clinical  questions. it also reinforces the suspicion  that most busy clinicians will only search  evidence based library and information practice 2008, 3:1  67 synthesized resources because they are  faster and will answer almost as many  clinical questions as medline.     clinicians appear to prefer synthesized, pre‐ appraised resources. in 2005, brian alper,  david white, and bin ge investigated  clinical results when dynamed, a pre‐ appraised resource, was added to the  sources usually searched by physicians.  among those who added dynamed to their  search arsenals, “significantly more  participants found answers that changed  clinical decision making” (510). the study  did not attempt to rate the quality or  relevance of answers found, nor did it find  that using dynamed shortened a clinician’s  total search time. yet it did indicate that the  simple addition of an ebm resource led to  greater clinical reflection and subsequent  change in practice.     one minor problem with the current study  is that the authors reported their findings  inconsistently in table 2 and in the text  describing table 2. in the table, the p‐value  is noted as p=0.0004; in the text, the p‐value  is written as p<0.00004. without  recalculating this value it is impossible to  judge which is correct, but the significance is  likely not affected either way.    the researchers at duke had the foresight to  use an independent and blinded panel of  ebm teachers and practitioners to judge the  search results. the findings are encouraging,  particularly in light of the questions  unanswered by the 2005 research.  both  current search protocols generated a high  percentage of applicable and internally valid  clinical answers. admittedly, the small size  of the sample and the target population  (residents) make it difficult to generalize the  results to all clinician searchers. clearly,  some searchers can achieve good results  when they rely primarily on synthesized  ebm resources. larger and broader studies  are now needed to increase our comfort  level with these ebm resources, and to  ensure that they remain comprehensive,  reliable, and usable.      works cited    alper, brian s., david s. white, and bin ge.  “physicians answer more clinical  questions and change clinical  decisions more often with  synthesized evidence: a  randomized trial in primary care.”  annals of family medicine 3.6  (2005): 507‐13.    koonce, taneya y., nunzia bettinsoli giuse,  and pauline todd. “evidence‐based  databases versus primary medical  literature: an in‐house  investigation on their optimal  use.” journal of the medical library  association 92.4 (2004): 407‐11.  editorial   evidence summary theme: all things virtual   heather macdonald associate editor (evidence summaries) health and biosciences librarian carleton university ottawa, ontario, canada email: heather.macdonald@carleton.ca      2023 macdonald. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30483      this issue’s evidence summaries (es) takes a look at all things virtual; this includes the virtual library and virtual services.  with the recent global pandemic there has been a surge in virtual library services, however virtual services were well established in libraries before that.  the library as a virtual space is also not new but the pandemic brought the virtual library front and center for many.   a number of the es in this issue look at studies examining the impact of the pandemic on libraries.  this includes a study that looks at university instructors’ use of online library services, a study of virtual teamwork in public libraries in the united states, and a study investigating the response to the pandemic in public libraries in bangladesh.  in addition, one es analyzes a case study of technical services usability testing of the library website, and the final two es examine studies about virtual reference methods one looking at effectiveness of virtual services and the other at the difference between perception and usage of these services.   post-pandemic, libraries continue to provide virtual services and technologies.  this series of es highlights some of the most recent research in this area.  we hope that you enjoy learning from these all-things virtual es and can potentially apply them in your own work.    evidence based library and information practice evidence based library and information practice 2012, 7.4 139 evidence based library and information practice commentary the evolution of evidence based library and information practice, part i: defining eblip jonathan d. eldredge associate professor health sciences library and informatics center university of new mexico albuquerque, new mexico, united states email: jeldredge@salud.unm.edu received: 1 dec. 2012 accepted: 4 dec. 2012 2012 eldredge. this is an open access article distributed under the terms of the creative commons‐ attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. evidence based library and information practice (eblip) has achieved an impressive array of accomplishments during its brief lifespan. mysteriously, the recent 15th anniversary of eblip passed with little notice. if the past editor actually had not brought it to my attention, i might not have noticed this anniversary despite having had a direct role in its development (eldredge, 1997). eblip already has produced six international conferences, the establishment of this open access peer reviewed journal, continuing education courses based in the uk and us with broad international participation, representation in most types of libraries (academic, public, etc.), two special issues of peer reviewed journals, and two books (booth & brice, 2004; connor, 2007) devoted entirely to eblip. some of the most robust early eblip work originated in countries such as australia, canada, sweden, the uk, and the us. more recent eblip work has emerged from countries such as iran and japan (gavgani, 2009; yukiko, 2008). few phenomena in the library and information science (lis) world indeed can claim as many achievements within only 15 years. eblip provides a sequential, structured process for integrating the best available evidence into making important decisions. the practitioner applies this decision making process by using the best available evidence while informed by a pragmatic perspective developed from working in the field, critical thinking skills, and an awareness of different research designs, which is further modulated by knowledge of the affected user population’s values or preferences. eblip has evolved quickly during the past 15 years. it has managed this rapid pace due to a professional environment characterized by encouragement, inquiry, skepticism, dialogue, an openness to new information among participants, and a willingness on the part of lis professionals to change their own minds. the brief definition above retains my original mailto:jeldredge@salud.unm.edu evidence based library and information practice 2012, 7.4 140 ideas, while reflecting the further evolution of my thinking during the past 15 years (eldredge, 2000; eldredge, 2006; eldredge, 2008), as well as incorporating elements from others’ definitions within this dynamic eblip professional environment (booth, 2002 ; crumley & koufogiannakis, 2002). the eblip process the eblip process provides structure for reaching important decisions. the eblip process resembles evidence based processes in other professions such as education, health, management, or public policy analysis. the steps in the eblip process can be summarized as: 1. formulate an answerable question on an important issue 2. search for the best available evidence to answer the question 3. critically appraise the evidence 4. make a decision and apply it 5. evaluate one’s performance in the pages that follow i will use the structure and sequence of the eblip process to define more clearly and describe the five eblip steps. 1. formulate an answerable question davies (2011) has noted that “questions are the driving force behind evidence based practice (ebp).” early explorations with eblip questions focused upon formulation techniques and compiling lists of questions from colleagues around the world (booth, 2001; eldredge, 2001). lewis & cotter (2007) noted the relative stability of eblip questions by subject matter between 2001 and 2006, although their study pointed to persistent research-practice gaps. booth (2006) crystalized existing ideas about question formulation and provided perhaps the most pragmatic advice to date on how to formulate productive eblip questions. wilson (2009) recently provided a brief column that included key pointers on question formulation. a new development in 2008 marked an unexpected and significant turning point in eblip question formulation. completely separate teams in sweden and the us with no awareness of the other team, simultaneously conducted consensus building delphi studies to prioritize large numbers of eblip questions. interestingly, and reflecting similar synchronicity the same year, rossall, boyes, montacute, and doherty (2008) called for similar approaches via research networks in an effort unrelated to either the swedish or us projects. the swedish team led by maceviciute and wilson (2009) conducted a two round delphi study that surveyed librarians via email from different types of libraries (academic, public, etc.) about their top research concerns. the final phase of the project involved a face-toface workshop that consisted of leveraging nominal group exercises in an effort to reach consensus. the delphi study in the us during 2008 focused only upon the leaders and research oriented members of the medical library association (mla). mla leaders and the members of mla’s research section were queried through a two-phase series of surveys and voting on top ranked questions. this study produced 12 top priority research questions that became the mla research agenda. a slightly modified 2011 delphi study by the same team produced a list of top ranked research questions quite different from the 2008 study (eldredge, harris, & ascher, 2009; eldredge, ascher, holmes, & harris, 2012). harris, holmes, ascher, and eldredge (2012) conducted a subject analysis on the complete list of 140 questions submitted during the first phase of the 2011 delphi study. the short lists of high priority research questions generated by both the swedish and u.s. delphi studies allow the profession to target high priority research concerns with money and other resources. library administrators can use these short listed priority research questions to encourage their librarians to pursue studies in these important research areas. our profession will benefit most from devoting the greatest resources and incentives to answering the highest ranked eblip questions, although we should continue evidence based library and information practice 2012, 7.4 141 to encourage individual researchers to pursue alternative areas of applied research which are capable of improving our practice. 2. searching for the evidence members of our profession are viewed widely by others as the masters of organizing and searching for needed information. paradoxically, our own databases are poorly organized and our vast grey literature unsuitable for easy systematic inquiry. searching for the relevant evidence on the part of eblip practitioners, thus, poses considerable challenges. winning (2004) and beverley (2004) assessed the challenges and offered their practical solutions for finding the needed evidence. while the technical details of these approaches might need updating, the principles these authors offered for searching for eblip largely still work well. booth (2008) wrote a complementary column on tactics for searching that might further aid eblip practitioners. bradley (2007) has noted that we also often need to search other non-lis literatures to find potentially pertinent evidence. since many lis researchers receive incentives in the form of paid conference attendance when presenting papers or posters, but zero or even negative incentives to publish the same research results, our profession consequently deposits much of its intellectual capital in the extensive and not always easily accessed grey literature. the “evidence summaries” in this journal provide great assistance to busy practitioners who can neither search for the evidence nor appraise it critically. the evidence summaries represented a brilliant idea (koufogiannakis, 2006) that, like most of eblip, has used the best available evidence to foster improvement as it has evolved continuously (kloda, koufogiannakis, & mallan, 2011; kloda, 2012). the open access journal hypothesis supplements the evidence summaries in eblip through its regularly published literature reviews on recent research and its publication of expanded structured abstracts for the research awards granted annually by the medical library association. 3. critically appraise the evidence critical appraisal involves sifting through the best available evidence in order to make a sound decision. two core principles guide the critical appraisal process. first, the evidence must be appropriate for answering the specific eblip question. second, evidence can vary widely in its quality. making judgments on the appropriateness of evidence can be a challenge. ideally, every graduate of a library or information practice professional school would be equipped to meet this challenge with at least one semester length course on the strengths and weaknesses of the major qualitative and quantitative research methods. the absence of such coursework in the majority of graduates has led to a new role for librarians with such research methods training: “translator.” previously, i conceptualized the librarian roles of evidence “producers” and “consumers” (eldredge, 2008, p. 254), but the realities of eblip have led to a new third “translator” role. the aforementioned evidence summaries in this journal enlist the services of such translators to critically appraise research evidence, which includes gauging evidence appropriateness. the profession now has a far more robust evidence base than it did during the early years of eblip (dalrymple, 2010). systematic reviews are generally considered to be the highest form of evidence, regardless of eblip question type (eldredge, 2008). systematic reviews were once scarce in our profession. they are now far more common with 39 documented in a blog created by denise koufogiannakis (2012). we have seen a proliferation of rigorous quantitative and qualitative research studies in our literature (given, 2006), although much more research needs to be pursued (koufogiannakis & crumley, 2006; rossall et al., 2008). as koufogiannakis (2011) recently concluded, “the scientific aspect of our work continually needs to be reinforced and built upon” (p. 2). the same principle applies to developing our local sources of evidence. evidence based library and information practice 2012, 7.4 142 many times evidence summaries do not quite address an emerging eblip question so librarians must access the research literature. fortunately, there have been a number of critical appraisal checklists to guide their reviews of the research literature (booth & brice, 2003; glynn, 2006). evidence hierarchies are helpful tools to guide critical appraisal, but should not be utilized rigidly (eldredge, 2002; eldredge, 2008). brettle (2012) recently reminded us that additionally we need to be open to the possible utility of evidence that defies our present eblip categorizations. 4. make a decision and apply it eblip questions emerge within local contexts when the practitioner must make an important decision. some questions are more universally shared, as already noted in the discussion of delphi studies above. making decisions and applying them similarly occur in the specific local context. eblip practitioners must know their local users’ values and preferences in this fourth step to be successful. cognitive biases also present some of the most daunting challenges in making and applying a sound decision, regardless of local context. cognitive biases either interfere with our perception of situations or in our making decisions (eldredge, 2007). the decision making step in eblip includes so many potential pitfalls that it would require an entire commentary to begin to examine even the most fundamental issues. suffice it to state that some of the best minds in eblip have been grappling with these issues for years, and much more work remains. 5. evaluate one’s performance grant’s systematic review on reflective practice (2007) illustrates that since 1978 our profession has a scattered yet evolving history of incorporating self-evaluation into practice. she tracks a trend toward more sophisticated forms of reflection than the early forms that comprised mainly senior librarians’ reminiscences. others have noted that evaluating performance takes place at the individual, institution, and professional association levels, so it manifests itself in more than just a solitary form. conclusion and next steps anniversaries offer us a convenient chance to track progress, reassess, and reflect. eblip actually did not begin in 1997, only an articulation and initial definition that expanded into an impressive body of further work. much work preceded the beginning of eblip, too, although the origins are diffuse when viewed through the retrospective lens of much undocumented history (eldredge, 2004; russell, 2008). this commentary has described what, for many eblip readers, will be obvious. eblip has arrived on the lis scene, has been fairly well codified within an environment of skepticism and reflection over the past 15 years, and it certainly seems to be on the minds of many librarians and other information professionals these days. the new oxford american dictionary defines a “definition” as “an exact statement or description of the nature, scope, or meaning of something” (definition, 2010). this commentary addresses the nature and scope of eblip. aside from mentioning its role in decision making, however, this piece does not delve into the purpose(s) or meaning of eblip. part ii (in the march 2013 issue) will grapple with the function(s) that eblip serves within our profession. is eblip a social movement within our profession? a reformist movement? a new academic discipline? a paradigm shift? a diffusing innovation? the tentative answers to this functionalist or structural-functionist question in part ii might shed light on where we need to be heading over the next 15 years and how we might best get there. references beverley, c. (2004). searching the library and information science literature. in a. booth & a. brice (eds.), evidence based practice for information professionals: a handbook. (pp.89-103). london: facet. retrieved 6 dec. 2012 from evidence based library and information practice 2012, 7.4 143 http://ebliptext.pbworks.com/f/booth+ %26+brice+2004+ebp+for+info+profess ionals+-+a+handbook.pdf booth, a. (2001). turning research priorities into answerable questions. health information and libraries journal, 18(2), 130-132. doi:10.1046/j.14711842.2001.d01-3.x booth, a. (2002). from ebm to ebl: two steps forward or one step back? medical reference services quarterly, 21(3), 5164. doi:10.1300/j115v21n03_04 booth, a. (2006). clear and present questions: formulating questions for evidence based practice. library hi tech, 24(3), 355-368. doi:10.1108/07378830610692118 booth, a. (2008). unpacking your literature search toolbox: on search styles and tactics. health information and libraries journal, 25(4), 313-317. doi:0.1111/j.1471-1842.2008.00825.x booth, a., & brice, a. (2003). clear-cut?: facilitating health librarians to use information research in practice. health information and libraries journal, 20(suppl. 1), 45-52. doi:10.1046/j.13652532.20.s1.10.x booth, a., & brice, a. (eds.) (2004). evidence based practice for information professionals: a handbook. (pp.89103). london: facet. retrieved 6 dec. 2012 from http://ebliptext.pbworks.com/f/booth+ %26+brice+2004+ebp+for+info+profess ionals+-+a+handbook.pdf bradley, c. (2007). the e in ebl: finding the evidence to support your practice. feliciter, 53(1), 22-24. brettle, a. (2012). learning from others about research evidence. evidence based library and information practice, 7(2), 13. connor, e. (ed.) (2007). evidence-based librarianship: case studies and active learning exercises. oxford: chandos. crumley, e., & koufogiannakis, d. (2002). developing evidence-based librarianship: practical steps for implementation. health information and libraries journal, 19(2), 61-70. doi:10.1046/j.1471-1842.2002.00372.x dalrymple, p. w. (2010). applying evidence in practice: what we can learn from healthcare. evidence based library and information practice, 5(1), 43-47. davies, k. s. (2011). formulating the evidence based practice question: a review of the frameworks. evidence based library and information practice, 6(2), 75-80. definition. (2010). in a. stevenson & c. a. lindberg (eds.), new oxford american dictionary. (3rd ed., p. 455). new york, ny: oxford university press. eldredge, j. (1997). evidence-based librarianship: a commentary for hypothesis. hypothesis, 11(3), 4-7. retrieved 6 dec. 2012 from http://research.mlanet.org/hypothesis/ hypo11-3.pdf eldredge, j. (2001). the most relevant and answerable research questions facing the practice of health sciences librarianship. hypothesis, 15(1), 9-16. retrieved 6 dec. 2012 from http://research.mlanet.org/hypothesis/ hypo2001v.15%20no.1.pdf eldredge, j. (2002). evidence-based librarianship: levels of evidence. hypothesis, 16(3): 10-13. retrieved 6 dec. 2012 from http://research.mlanet.org/hypothesis/ hyp_v16n3.pdf eldredge, j. (2004). evidence-based information practice: a prehistory. in a. booth & a. brice (eds.), evidence evidence based library and information practice 2012, 7.4 144 based practice for information professionals: a handbook. (pp.2435). london: facet. retrieved 6 dec. 2012 from http://ebliptext.pbworks.com/f/booth+ %26+brice+2004+ebp+for+info+profess ionals+-+a+handbook.pdf eldredge, j. (2006). evidence-based librarianship: the ebl process. library hi tech, 24(3), 341-354. doi:10.1108/07378830610692118 eldredge, j. (2007). cognitive biases as obstacles to effective decision making. paper presented at the meeting of eblip4: transforming the profession: 4th international evidence based library and information practice conference, university of north carolina-chapel hill, durham, nc, usa. retrieved 6 dec. 2012 from http://www.eblip4.unc.edu/downloads /eblip4_final_revised_postconference.p df eldredge, j. d. (2000). evidence-based librarianship: an overview. bulletin of the medical library association, 88(4), 289-302. retrieved 6 dec. from http://www.ncbi.nlm.nih.gov/pmc/arti cles/pmc35250/pdf/i0025-7338-088-040289.pdf eldredge, j. d. (2008). evidence-based practice. in m. s. wood (ed.), introduction to health sciences librarianship. (pp. 241269). binghamton, ny: haworth. eldredge, j. d., ascher, m. t., holmes, h. n., & harris, m. r. (2012). the new medical library association research agenda: final results from a threephase delphi study. journal of the medical library association, 100(3), 214218. doi:10.3163/1536-5050.100.3.012 eldredge, j. d., harris, m. r., & ascher, m. t. (2009). defining the medical library association research agenda: methodology and final results from a consensus process. journal of the medical library association, 97(3), 178185. doi:10.3163/1536-5050.97.3.006 gavgani, v. z. (2009). the perception and practice of evidence based library and information practice among iranian medical librarians. evidence based library and information practice, 4(4), 37-57. given, l. (2006). qualitative research in evidence-based practice: a valuable partnership. library hi tech, 24(3), 376386. doi:10.1108/07378830610692145 glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi:10.1108/07378830610692154 grant, m. j. (2007). the role of reflection in the library and information sector: a systematic review. health information and libraries journal, 24(3), 155-166. doi:10.1111/j.1471-1842.2007.00731.x harris, m. r., holmes, h. n., ascher, m. t., & eldredge, j. d. (in press). inventory of research questions identified by the 2011 mla research agenda delphi study. hypothesis. kloda, l. (2012). improvements to evidence summaries: an evidence based approach. evidence based library and information practice, 7(3), 71-72. kloda, l. a., koufogiannakis, d., & mallan, k. (2011). transferring evidence into practice: what evidence summaries of library and information studies research tell practitioners. information research, 16(1), paper 465. retrieved 6 dec. 2012 from http://informationr.net/ir/161/paper465.html koufogiannakis, d. (2006). small steps forward through critical appraisal. evidence based library and information practice, 1(1), 81-82. evidence based library and information practice 2012, 7.4 145 koufogiannakis, d. (2011). should we value knowledge and expertise? evidence based library and information practice, 6(3), 1-2. koufogiannakis, d. (2012). lis systematic reviews. retrieved 6 dec. 2012 from http://lis-systematicreviews.wikispaces.com/welcome koufogiannakis, d., & crumley, e. (2006). research in librarianship: issues to consider. library hi tech, 24(3): 324340. doi:10.1108/07378830610692109 lewis, s., & cotter, l. (2007). have the most relevant and answerable research questions facing librarians changed between 2001 and 2006? evidence based library and information practice, 2(1), 107-120. maceviciute, e., & wilson, t. d. (2009). a delphi investigation into the research needs in swedish librarianship. information research, 14(4), paper 419. retrieved 6 dec. 2012 from http://informationr.net/ir/144/paper419.html rossall, h., boyes, c., montacute, k., & doherty, p. (2008). developing research capacity in health librarians: a review of the evidence. health information and libraries journal, 25(3), 159-174. doi:10.1111/j.14711842.2008.00788.x russell, k. (2008). evidence-based practice and organizational development in libraries. library trends, 56(4), 910-930. wilson, v. (2009). matching question types to study designs. evidence based library and information practice, 4(1), 51-52. winning, a. (2004). identifying sources of evidence. in a. booth & a. brice (eds.), evidence based practice for information professionals: a handbook. (pp.71-88). london: facet. retrieved 6 dec. 2012 from http://ebliptext.pbworks.com/f/booth+ %26+brice+2004+ebp+for+info+profess ionals+-+a+handbook.pdf yukiko, s. (2008). from ebm to ebl / eblip. pt.2: evidence-based practice for their own practice by medical librarians. journal of information processing and management, 51(2), 105-115. doi:10.1241/johokanri.51.105 research article   factors associated with the prevalence of precarious positions in canadian libraries: statistical analysis of a national job board   ean henninger liaison librarian simon fraser university library burnaby, british columbia, canada email: ean_henninger@sfu.ca   adena brons digital scholarship librarian & liaison librarian simon fraser university library burnaby, british columbia, canada email: adena_brons@sfu.ca   chloe riley library communications officer simon fraser university library burnaby, british columbia, canada email: chloe_riley@sfu.ca   crystal yin liaison librarian simon fraser university library burnaby, british columbia, canada email: yya192@sfu.ca   received: 2 june 2020                                                                accepted: 27 july 2020      2020 henninger, brons, riley, and yin. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29783     abstract   objective to collect and share information about the prevalence of precarious work in libraries and the factors associated with it.   methods the authors collected and coded job postings from a nationwide job board in canada for two years. descriptive and inferential statistics were used to explore the extent of precarity and its relationship with job characteristics such as job type, institution type, education level, and minimum required experience.   results the authors collected 1,968 postings, of which 842 (42.8%) were coded as precarious in some way. the most common types of precarious work were contracts (29.1% of all postings) and part-time work (22.7% of all postings). contracts were most prevalent in and significantly associated with academic libraries and librarian positions, and they were most often one year in length. both on-call and part-time work were most prevalent in school libraries and for library technicians and assistants, and they were significantly associated with all institution types either positively or negatively. meanwhile, precarious positions overall were least prevalent in government and managerial positions. in terms of education, jobs requiring a secondary diploma or library technician diploma were most likely to be precarious, while positions requiring an mlis were least likely. the mean minimum required experience was lower for all types of precarious positions than for stable positions, and the prevalence of precarity generally decreased as minimum required experience increased.   conclusion the proportion of precarious positions advertised in canada is substantial and seems to be growing over time. based on these postings, employees with less experience, without advanced degrees, or in library technician and assistant roles are more likely to be precarious, while those with managerial positions, advanced degrees, or more experience, are less likely to be precarious. variations in precarity based on factors such as job type, institution type, education level, and minimum required experience suggest that employees will experience precarity differently both within and across library systems.     introduction   precarious labour is an employment structure defined by the international labour organization as involving:   uncertainty as to the duration of employment, multiple possible employers or a disguised or ambiguous employment relationship, a lack of access to social protection and benefits usually associated with employment, low pay, and substantial legal and practical obstacles to joining a trade union and bargaining collectively. (2012, p. 27)   precarious labour takes many forms, all with the potential to produce material and psychological insecurity and vulnerability among workers. current examples of precarious labour include jobs associated with the gig economy, the trend towards adjunctification in higher education, and the use of temporary and poorly paid workers in farms and processing plants.   precarious labour also exists in all kinds of libraries and it affects workers at all levels. it can include workers in part-time or full-time positions, temporary or permanent positions, and on-call or auxiliary positions. although authors in recent years have begun to address the effects of precarious library work (henninger, brons, riley, & yin, 2019; lacey, 2019; skyrme & levesque, 2019), there is still very little scholarship documenting the prevalence of precarious work or describing the characteristics of precarious jobs. accordingly, this article focuses on examining the prevalence of precarious library jobs and the factors associated with them. it begins by situating itself relative to the literature on library job posting analyses and precarious employment. it continues by describing the methodology and the results of a study that involved collecting job postings from a nationwide job board over two years, coding the postings into various categories, and conducting descriptive and inferential statistical analyses. finally, it discusses the results and their implications for job searching, hiring, employment, and more.   one way of describing the differences between precarious and stable jobs is to establish the prevalence of precarious work, as well as associations within that prevalence, such as education required, years of experience, or job position. knowing how common precarity is and how it expresses itself within the profession will aid interested parties in imagining and enacting alternatives where desirable.   literature review   although literature on the prevalence and characteristics of precarity in libraries is limited, the research that does explore this topic centers on surveys and analyses of job postings. surveys are a common method of exploring the prevalence of certain characteristics in library jobs; however, there have been few surveys conducted and published specifically with precarity in mind. in canadian academic settings, there have been surveys describing the prevalence of precarious work and its negative effects on individuals as well as academic institutions (pasma & shaker, 2018; foster & birdsell bauer, 2019), but these surveys determine librarians in precarious contracts to be out of scope, despite the fact that many librarians are faculty members at such institutions. bladek (2019) pointed out that this omission is unfortunately common, with few reports or studies on precarity within academia including precariously employed librarians, and with lis (library and information studies) statistics rarely differentiating between full-time or part-time and temporary or permanent positions (p. 486). in the public context, a recent canadian union of public employees survey of over 800 public library employees in canada classified 28% of respondents as precarious and a further 24% as vulnerable to precarity, with 49% in stable or secure positions (cupe, 2017, p. 26).   in the united states, wilkinson (2015) surveyed 73 current and former part-time librarians who graduated from mlis (masters in library and information studies) programs between 2008-2012 and had held at least one part-time position following graduation (p. 348). for these part-time positions, the majority of respondents worked in academic and public libraries and over 55% worked concurrently in more than 1 position (wilkinson, 2015, p. 348 & p. 352).   another common means of exploring trends in library employment and characteristics of library-related jobs is through the analysis of job advertisements. studies have explored trends in advertisements for librarian positions in areas such as government documents (sproles & clemons, 2019), digital initiatives (skene, 2018), and electronic resources (ferguson, 2018). others have explored the relationship between posted qualifications and professional competencies or standards (gold & grotti, 2013; hartnett, 2014; henricks & henricks-lepp, 2014; maciel, kaspar, & vanduinkerken, 2018). additional studies have focused on assessing the professional skills required in postings for lis program curriculum development (messum, wilkes, peters, & jackson, 2016; wise, henninger, & kennan, 2011). however, such studies focus almost exclusively on positions requiring an mlis degree, and very few explore or note aspects related to precarity in their analyses.   one exception is a study by wilkinson (2016), which analyzes 56 part-time librarian positions in pennsylvania and new jersey. wilkinson (2016) found that the postings were primarily from academic libraries (48%) and public libraries (43%), with minimal postings from special libraries (7%) and school libraries (2%) (p. 74). in addition, she found that only 64% of the part-time postings included hours of work; of those that did indicate hours, the most common range was 16-20 hours (25%) (wilkinson, 2016, p. 75).   another exception is maccaferri and harhai’s (2019) study of public library job advertisements, which incorporated an analysis of both part-time postings and postings that did not require an mlis. their study covered 1 year’s worth of advertisements on a pennsylvania library email list and analyzed 124 public library postings. postings were fairly evenly divided between “professional” (mlis-holding) positions (52.42%) and “non-professional” positions (47.58%) (maccaferri & harhai, 2019, p. 12). the study found that 94.35% of all jobs posted were permanent positions (maccaferri & harhai, 2019, p. 12). however, “professional positions were predominantly full-time (80%) while non-professional positions were predominantly part-time (86.44%)”, representing a stark disparity based on educational level (maccaferri & harhai, 2019, p. 13). unfortunately, the authors did not break down the number of work hours within these part-time positions, nor did they identify on-call or auxiliary postings in the analysis.   reviewing the literature reveals a significant lack of information about the prevalence and characteristics of precarious library jobs. despite some studies touching on the issue, the extent of precarity remains under-examined, with most surveys and job advertisement analyses having minimal inclusion of precarious positions. as well, few studies use inferential analyses, which could enable authors to make generalizations or predictions about the broader population of actual jobs from job postings. according to harper’s (2012) review of 70 job advertisement analyses in lis, this minimal use of inferential statistics is one criticism of the genre.   the scholarship that does exist primarily focuses on part-time jobs and does not include contract or on-call jobs. in some cases, this limitation may be due to data collection methods, as job aggregators or national email lists may not include part-time or limited-term positions. for example, in a study of entry-level librarian positions, tewell (2012) captured 1385 postings over a year, of which only 78 (5.6%) were part-time (20 or fewer hours) or temporary (less than 1 year) (p. 414). wilkinson (2016) concurs that job advertisement analyses often exclude part-time positions, resulting “in a severe lack of reliable information about the duties, hours, and salaries of part-time professionals and paraprofessionals in libraries.” (p. 68). this exclusion may result in an overrepresentation of permanent full-time positions in analyses of job advertisements.   this article seeks to address some of these gaps through both descriptive and inferential analyses of a dataset representing two years’ worth of job postings from a canada-wide online job board.   aims   the aim of this research study is to better understand the prevalence of precarious library work and the factors associated with it, providing insight into the landscape of library employment trends. the research questions for this project are:   ·        what is the prevalence of precarious library job postings in canada? o   does the prevalence vary based on key characteristics of those postings? ·        to what extent are different characteristics of library job postings associated with precarity? o   do the characteristics of job postings change based on whether or not a job is precarious or based on the specific type of precarity (i.e., contract, on-call, or part-time)?   methods   the methodology for this study was initially informed by the authors’ status as precarious contract workers themselves. they determined that analyzing advertisements from a single website would be a means of collecting information that was within the scope of their shared capacity. the website chosen for analysis was the partnership job board, which is maintained by the british columbia library association to support members of the partnership, canada’s national network of provincial and territorial library associations.   the authors used a predetermined weekly schedule to review jobs posted on this site over the course of their assigned weeks, entering posting data into a shared spreadsheet, and saving copies of the postings to a shared drive. the authors assigned each posting a job id (identification) number and then entered additional identifying data consisting of date posted, date closed, job title, institution name, city, and province or territory. they also collected and coded data for aspects of job postings, listed with coding criteria in the appendix, that were decided a priori to be of potential interest in determining the prevalence of precarity and factors associated with it. finally, note fields were used to provide any necessary context for how the postings were coded. a total of 1,968 postings were collected over a period of 2 years, from november 15, 2017 to november 14, 2019.   after collecting postings, the authors reviewed the spreadsheet for consistency and recoded postings in two categories. institution types were recoded to split government positions into their own category, and a previously existing “special” category was collapsed into “other.” additionally, the majority of the postings coded as “other” under the education level were recoded into other categories. the resulting data set was cleaned to support legibility and data filtering.   the data analysis methods employed consisted of descriptive statistics using tableau, showing the frequencies and proportions of precarious jobs relative to non-precarious jobs, and inferential statistics using spss 25. the data used for inferential analysis consisted of two kinds of variables. there were seven nominal-level variables: three categories with multiple entries defining institution type, job level, or education level respectively, and four dichotomous categories defining whether or not a job was precarious, contract, on-call, or part-time, respectively. there were also two continuous, ratio-level variables, both expressed in months: contract duration and minimum required experience. due to a tendency in job postings to round both contract length and minimum required experience to the nearest year, these two variables were not normally distributed. given this, the broader population of actual jobs would likely replicate these non-normal distributions.   the authors performed pearson chi-square tests for independence to determine if significant differences existed among institution type, job level, and education level, and each of the four dichotomous variables describing whether or not a job was precarious, contract, on-call, or part-time. these tests were appropriate to compare two nominal-level variables consisting of categorical and independent groups.   the authors additionally performed independent-sample welch’s t-tests to look for significant associations between the continuous variable of minimum months of experience required and each of the four dichotomous variables describing whether or not a job was precarious, contract, on-call, or part-time. these tests were appropriate to compare differences in means between two independent samples where equal variance could not be assumed, and they remain robust for large and unequal sample sizes even when variables are not normally distributed. the authors also calculated confidence intervals for these tests.   in one instance, the authors calculated spearman’s rho to correlate the two ratio-level variables of contract length and minimum required experience. this non-parametric statistic using ranked data was appropriate given the non-normal distribution of these continuous variables.   for these analyses, the authors set the alpha level for statistical significance at α = 0.011 based on the equation in lakens (2018): α = 0.05/√(1968/100). although α is conventionally set to 0.05 in many settings, sample sizes in this study were easily large enough to make weak effects statistically significant for sufficiently high values of α, increasing the chances of observing an effect where none existed.   effect size is important to report along with statistical significance because it shows the magnitude of a change that one variable produces on another variable, allowing for more interpretation of that effect’s importance. accordingly, the authors calculated two measures of effect size: cramer’s v for chi-square tests, denoted as ϕc, and hedge’s g for t-tests, which was preferred to cohen’s d as it weights effect size based on sample sizes. differences between means and the sizes of test values (χ2 and t) relative to other values for the same kinds of tests also give indications of effect size. for chi-square tests, the authors also calculated standardized residuals, which measure the strength of the difference between observed and expected values and show how much each category in a chi-square test contributes to the overall association. at α = 0.011, a standardized residual contributes significantly if it lies outside of ± 2.54. as cohen (1988) discusses, the exact meaning of effect size depends in part on the context, content, and method of a given study. in the absence of any prior conventions for this kind of study, the authors used the conventions recommended by cohen for cramer’s v listed in table 1, and hedge’s g, where small = 0.2, medium = 0.5, and large = 0.8.   a note. df = degrees of freedom for contingency tables created for chi-square tests. adapted from a table and conventions by cohen (1988).   results   overall prevalence   over 2 years, the authors collected 1,968 job postings from the partnership job board and coded them according to the methodology. table 2 shows the overall prevalence of precarity and its subtypes. these subtypes were not mutually exclusive, as all on-call jobs were part-time, many contract jobs were also part-time, and some contract jobs were on-call.   figure 1 shows that the number of jobs posted by province was uneven, with 955 jobs based in ontario and 565 in british columbia, together comprising 77.2% of all jobs posted. postings from new brunswick had the highest prevalence of precarious employment (67.4%), followed by quebec (48.6%), british columbia (45.7%), and ontario (44.4%).   as seen in figure 2, the prevalence of precarity increased from the first year of data collection to the second. in year 1 (november 15, 2017 to november 14, 2018), precarious jobs made up 39.9% of all jobs posted. in year 2 (november 15, 2018 to november 14, 2019), precarious jobs made up 45.9% of all jobs posted. overall job postings were roughly equal in each year, with 998 jobs posted in year 1 and 970 jobs posted in year 2.         figure 1 job postings by precarity and province.     figure 2 job postings by precarity and year posted.     institution type   of all jobs posted in this period, the majority were from public libraries (55.8%), followed by academic libraries (32.6%). when stable and precarious postings were analyzed by type of institution, as seen in table 3, precarity was least prevalent among government library jobs (27.6%) and most prevalent among school library jobs (53.1%). the chi-square test showed a significant association between type of institution and whether or not a job was precarious χ2 (4, n = 1968) = 13.07, p = .011, and the effect size was small, ϕc = .08. no single category of institution significantly contributed to this association, meaning that no category had more or fewer precarious positions than expected.   limited term contracts were the most prevalent in academic libraries (34.6%), followed by public libraries (26.2%), as seen in table 4. they were least prevalent in school libraries (18.4%). there was a significant association between type of institution and whether or not a job was a contract χ2 (4, n = 1968) = 19.20, p = .001, also indicating a small effect size, ϕc = .10. academic libraries were the only significant driver of this association, with more contract postings than expected.   table 5 shows that on-call postings were most prevalent for school libraries (18.4%) and least prevalent in government jobs (1.1%). there was a significant association between institution type and whether or not a job was on-call χ2 (4, n = 1968) = 31.06, p < .001, again demonstrating a small effect size, ϕc = .13. school libraries contributed significantly to this association, with more on-call postings than expected, as did academic libraries with fewer than expected.                     finally, part-time postings were most prevalent in school library settings (46.9%) and least prevalent in government institutions (5.7%), as seen in table 6. there was a significant association between type of institution and whether or not a job was part-time χ2 (4, n = 1968) = 70.18, p < .001, indicating a medium effect size, ϕc = .19. all institution types significantly contributed to this association, with public and school library positions having more part-time positions than expected, and academic and government positions having fewer.   job type   postings for librarian jobs were the most prevalent type of position represented in the 2-year period (37.4%), followed by managers (23.4%), and technicians (18.4%). meanwhile, archivist postings were the least prevalent (1.8%). when analyzing the type of position for precarity, as seen in table 7, precarity was most prevalent among assistant positions (69.8%) and least prevalent among manager positions (14.1%). precarious manager positions were sometimes due to term limits for head or chief librarians, but the authors still coded these as precarious since they met the technical definition of a limited-term contract. there was a significant association between job type and whether or not a job was precarious χ2 (5, n = 1968) = 242.00, p < .001, representing a very large effect size, ϕc = .35. manager positions were a highly significant contributor to this association with far fewer precarious positions than expected, while assistant and technician positions also contributed with more than expected.   limited term contracts were most prevalent among archivist and librarian positions (38.9% and 38.7% respectively), as seen in table 8. there was a significant association between job type and whether or not a job was a contract χ2 (5, n = 1968) = 118.58, p < .001, and the effect size was large, ϕc = .25. manager and librarian positions were significant drivers of this association, with the former being more likely than expected to be contracts, and the latter being more likely than expected to be contracts.   meanwhile, table 9 demonstrates that on-call job postings were most prevalent among assistants (13.2%), and technicians (13.0%). they were least common for archivists (0.0%), and managers (0.4%), while librarians were close to the average at 6.5%. there was a significant association between job type and whether or not a job was on-call χ2 (5, n = 1968) = 66.18, p < .001, indicating a medium effect size, ϕc = .18. this association was significantly driven by manager jobs, which were much less likely to be on-call than expected, and by technician and assistant jobs, which were more likely to be on-call than expected. part-time job postings, as seen in table 10, were very prevalent among assistants (55.6%) and technicians (44.5%). they were least prevalent among managers (4.6%) and archivists (5.6%). there was a significant association between job type and whether or not a job was part-time χ2 (5, n = 1968) = 338.81, p < .001, indicating a very large effect size, ϕc = .42. this association was significantly driven by jobs of every type except for archivists, with manager and librarian jobs being less likely than expected to be part-time, and technician and assistant jobs more likely.                     education level   the authors excluded 75 postings from the analysis of education levels; 73 jobs that did not specify any educational qualifications and 2 postings that specified a minimum of grade 10 education. of the postings with required educational qualifications (n = 1893), jobs requiring a mlis or equivalent were the most common (58.6%) and jobs requiring a library technician diploma were the next most common (22.6%). when looking at precarity and education level as seen in table 11, precarious postings were most prevalent among jobs requiring some library coursework (90.6%) and jobs requiring a secondary diploma (85.7%). rates were substantially lower for all other categories, with the lowest rate among jobs requiring a mlis (35.3%). there was a significant association between educational level and whether or not a job was precarious χ2 (6, n = 1893) = 98.18, p < .001, and the effect size was large, ϕc = .23. jobs requiring some library coursework, secondary diplomas, library technician diplomas, or mlis degrees were all significant drivers of this association. jobs with mlis degrees were less likely to be precarious than expected, while the rest were more likely than expected.   limited term contracts were by far most prevalent among positions requiring some library coursework, comprising 78.1% of those positions as seen in table 12, and likely reflecting that many of these postings were meant to be completed during a library degree. they were least prevalent among jobs requiring other postsecondary degrees (22.0%) and library technician diplomas (22.7%). there was a significant association between educational level and whether or not a job was a contract χ2 (6, n = 1893) = 53.50, p < .001, representing a medium effect size, ϕc = .17. jobs requiring some library coursework were the only significant contributors to this association, being more likely than expected to be contracts.   table 13 demonstrates that the on-call employment structure was most prevalent among postings requiring secondary diplomas (17.1%), some library coursework (12.5%), and library technician diplomas (11.4%). meanwhile, no on-call jobs required a mas (master of archival studies) (0.0%). there was a significant association between educational level and whether or not a job was on-call χ2 (6, n = 1893) = 40.17, p < .001, and the effect size was medium, ϕc = .15. however, 4 cells in this test (28.6%) had an expected count of less than 5, resulting in a substantial loss of statistical power. jobs requiring library technician diplomas or mlis degrees were the only significant drivers of this association, with the former being more likely and the latter being less likely than expected to be on-call.   part-time jobs were extremely prevalent among postings that required a secondary diploma (80.0%), as seen in table 14. part-time postings were least prevalent when requiring a mas or mlis degree (7.7% and 10.4% respectively). there was also a significant association between educational level and whether or not a job was part-time χ2 (6, n = 1893) = 283.01, p < .001, indicating a very large effect size, ϕc = .39. postings requiring library technician diplomas and secondary diplomas significantly contributed to this association by being more likely than expected to be part-time, as did postings requiring an mlis, which were less likely than expected to be part-time.                     figure 3 job postings by precarity and minimum required experience.     minimum required experience   almost half of postings (n = 890, 45.2%) did not specify the minimum experience required for the position and were excluded from this analysis. of the remaining postings (n = 1078, 54.8%), the prevalence of precarity generally decreased as experience increased, as seen in figure 3. of the postings that required less than 1 year of experience (n = 88), 71.63% were precarious. of the postings requiring 1 year of experience (n = 162), 62.3% were precarious. for positions requiring more than 1 year of experience (n = 828), only 27.5% were precarious.   for the postings that listed a minimum required amount of experience, t-tests showed that on average the non-precarious jobs (m = 41.07, se = 0.91) required more months of experience than precarious jobs did (m = 21.48, se = 0.72). this difference of -19.59 months, 98.9% ci [-22.54, -16.63], was significant t(1073.58) = -16.88, p < .001, demonstrating a large effect size, g = 0.94. ongoing jobs (m = 36.92, se = 0.82) required more months of experience on average than contract jobs (m = 23.75, se = 1.02). the difference of -13.17 months, 98.9% ci [-16.52, -9.82], was also significant t(580.75) = -10.03, p < .001, and represented a medium effect size, g = 0.59. on average, jobs with stable hours (m = 34.95, se = 0.72) required more months of experience than on-call jobs (m = 15.71, se = 1.55). this difference of -19.24 months, 98.9% ci [-23.68, -14.80], was significant t(79.50) = -11.28, p < .001, showing a large effect size, g = 0.86. finally, full-time jobs (m = 37.85, se = 0.80) required more months of experience on average than part-time jobs (m = 18.54, se = 0.70). the difference of -19.31 months, 98.9% ci [-22.03, -16.60] was significant t(806.37) = -18.14, p < .001, and had a large effect size, g = 0.90.     figure 4 contract positions by contract duration.     contract length   temporary positions comprised 29.1% (n = 572) of the total postings. the authors coded these postings according to contract length as described in the appendix and as seen in figure 4. one-year contracts were by far the most common, comprising 38.1% of all temporary positions. an additional 30.4% of contracts were for less than 1 year. for job postings that reported both contract length and the minimum months of experience required (n = 214), spearman’s rho found a significant correlation between the 2 variables, p < .001, and a small effect size, rs = .25, meaning that contract length tended to increase along with minimum required experience.     figure 5 part-time postings by hours of work per week.     among part-time postings (n = 446) as seen in figure 5, the most common assignments of hours per week were 21-34 (33.6%) and 11-20 (24.6%). a substantial portion of postings (30.5%) had variable hours, indicating on-call work.   discussion   overview   the results show that precarious work is substantially and perhaps even increasingly prevalent in library job postings, with the percentage of precarious postings on the partnership job board rising from 39.9% in the first year of data collection to 45.9% in the second year. the landscape of precarious work varied, with important differences in prevalence and type of precarity based on the type of institution, type of position, and the educational or experiential requirements involved.   results from inferential statistics indicate that while precarious jobs were prevalent overall, they were not more likely to occur in one type of library over another. the results show that academic institutions were more likely to post contract positions than expected, corresponding with research conducted into sessional and adjunct labour in academia (pasma & shaker, 2018; foster & birdsell bauer, 2019) and showing that libraries are not immune to academic labour conditions, despite often being excluded from such studies. meanwhile, public libraries were more likely than expected to post for on-call and part-time positions. school libraries saw the highest prevalence of precarity, while government postings saw the least overall.   there were significant associations between whether a job was precarious and the type of position being advertised. library assistant and library technician postings were most likely to be precarious, while manager positions were least likely. these findings indicate that the prevalence of precarious employment in libraries overall is greater than suggested by previous research, which mainly focuses on librarian positions held by people with a mlis (mayo & whitehurst, 2012; wilkinson, 2016).   precarity was also strongly associated with the minimum level of education required for the position. for example, jobs requiring a secondary diploma or library technician diploma were much more likely than expected to be precarious than expected, especially in terms of on-call and part-time work, while jobs requiring mlis degrees were much less likely than expected to be precarious.   looking at minimum required experience, the results also show significant differences between precarious and non-precarious jobs. contract jobs had the highest mean minimum experience at about 24 months, the lowest mean difference relative to stable jobs at around 13 months, and the smallest effect size, suggesting that this form of precarity, involving regular working hours and in many cases full-time employment, requires more experience than others. by contrast, on-call jobs had the lowest mean minimum experience at about 16 months, suggesting that the least stable form of precarious work is also the easiest to get, at least based on experience.   the mean minimum required experience was significantly higher for stable jobs in all cases, and precarious work was less prevalent among positions requiring more experience, suggesting that available positions are less likely to be precarious as people gain more library experience. at the same time, the mean minimum experience was between one and two years for all categories of precarious jobs, suggesting that prior work experience is required even for precarious jobs. this lack of stable, entry-level positions combined with the amount of minimum experience typically required for all kinds of positions indicates that people can expect to be precariously employed for the first few years of their time in libraries.   the uneven distributions among these results suggest that workers will not experience precarity equally within institutions or across libraries as a whole. library employees who are early in their careers without advanced degrees, or in paraprofessional positions, are more likely to be working in precarious positions. these employees will therefore be the most likely to experience the stressors associated with precarity, such as financial instability, burnout, and poor mental health.   meanwhile, those in stable positions will be the most insulated from the effects of precarity, while also having the most power to affect policy, hiring, retention, and other factors relating to the wellbeing of precarious colleagues. these positions are most likely for staff in managerial positions with several years in the field, usually requiring a mlis or equivalent.   limitations   the results of this study may not be fully representative due to the limitations of job posting analyses as a method. although collected data should approach a representative distribution as the sample gets larger, it is possible that the actual population of jobs is more or less precarious than observed here. factors such as the authors’ definition of precarious work, their decision to code jobs as stable where their status was unclear, the fact that not all job postings are necessarily filled, and the fact that not all library jobs are posted to partnership may all affect the results’ generalizability. indeed, based on the high prevalence of librarian jobs and jobs requiring mlis degrees relative to other kinds of jobs, it is likely that partnership is primarily used for library jobs where organizations prefer having nationwide exposure and paying the listing fee. other jobs may be distributed internally, on library websites, or via municipal or provincial job boards, and job categories such as archivist jobs or government jobs may be posted in still other places. as a result, there may be a greater or lesser proportion of precarious jobs than shown in this dataset. comprehensive data on actual jobs from library systems, though it would be difficult to gather, could provide a useful contrast to the data represented here. finally, it is important to acknowledge the limitations inherent in a positivist approach. removing these postings from the contexts of their creation and circulation and reducing them to categories in a coding framework will necessarily produce a partial view of precarious work, with a limited ability to note anything about the material processes that produce precarious jobs or the people who hold them. other approaches may support a more holistic view of this topic. while these limitations should be taken into account, the existing data still points towards many significant differences and associations, as observed above, and can form a strong basis for future research.   future research   the authors did not conduct analyses combining three or more variable categories for this article, in order to maintain focus on the primary research questions and for the sake of brevity. however, further analysis could investigate specific aspects of precarity, such as differences in precarity between academic librarians and public librarians, or between managers with library technician diplomas and managers with mlis degrees. as well, researchers could apply methodologies such as content analysis to the postings collected for this study to determine, for instance, what proportion of contract positions list the rationales for the contracts, or whether the ways in which postings list salary ranges varies between precarious and stable jobs. the authors hope that making their dataset publicly available and archiving the original postings will help in this regard.   the current findings raise other issues for future inquiry as well. the distribution of different subtypes of precarity across institution types may result from different service models, and future research could seek to determine the causes of precarity within different institution types. meanwhile, looking at precarious jobs by education level reveals disparities based on educational qualifications. the issue of precarity and non-mlis positions remains understudied even in comparison to the scant research on library precarity overall, so further research is needed here too. this study focused on precarity within the canadian context, and additional research could compare levels and distribution of precarity with datasets from other geographic areas.   the prevalence of precarity among entry-level jobs and jobs requiring lower levels of education also raises questions about pipeline, hiring, and retention issues with implications for equity, diversity, and inclusion in libraries. it is already known that precarious workers are more likely to be racialized, women, lgbtq+, or have a disability (cranford & vosko, 2006; bernhardt, 2015; cupe, 2017). these results make clear that, whether through education or years of experience, the jobs that are the most accessible to the most people are also more likely to be precarious. in the quest for stable jobs, people from historically and presently marginalized groups must contend with racism, sexism, ableism, homophobia, and transphobia, in addition to the stresses of precarious work. given the barriers to equity, diversity, and inclusion before, during, and after hiring, processes in the predominantly white library profession’s (galvan, 2015) precarious employment structures deserve more attention in relation to these problems.   conclusion   this study aimed to establish a better understanding of the prevalence of precarious work and the factors associated with it in canadian libraries. the authors collected and coded job postings from a national job board over a period of two years and conducted statistical analyses that revealed significant differences in job precarity among different levels of experience and education, and different types of jobs and institutions. contracts and part-time work were the most common types of precarious employment, with a majority of contracts being for one year or less and about a third of part-time positions having variable hours. precarity was especially prevalent among school libraries, paraprofessional positions, positions requiring less education, and positions requiring two years of experience or less. by contrast, it was least evident in government libraries, managerial positions, positions requiring mlis or mas degrees, and positions requiring three years of experience or more. precarious jobs also required less experience on average than stable jobs. these findings show that precarious work is prevalent in canadian libraries and that this prevalence varies based on job characteristics.   references   bernhardt, n. s. (2015). racialized precarious employment and the inadequacies of the canadian welfare state. sage open, 5(2), 1-13. https://doi.org/10.1177/2158244015575639   bladek, m. (2019). contingent appointments in academic libraries: management challenges and opportunities. library management, 40(8/9), 485-495. https://doi.org/10.1108/lm-06-2019-0032   canadian union of public employees. (2017). employment increasingly precarious in public libraries, survey finds. in canadian union of public employees. retrieved from https://cupe.ca/employment-increasingly-precarious-public-libraries-survey-finds   cohen, j. (1988). statistical power analysis for the behavioral sciences. hillsdale, nj: lawrence erlbaum associates. https://doi.org/10.4324/9780203771587   cranford, c. j., & vosko, l. f. (2006). conceptualizing precarious employment: mapping wage work across social location and occupational context. in l. f. vosko (ed.), precarious employment: understanding labour market insecurity in canada (pp. 43-66). montreal and kingston: mcgill–queen’s university press.   ferguson, c. l. (2018). electronic resources and serials staffing challenges. serials review, 44(2), 122–130. https://doi.org/10.1080/00987913.2018.1471885   foster, k., & birdsell bauer, l. (2018). out of the shadows: experiences of contract academic staff. in canadian association of university teachers. retrieved from https://www.caut.ca/sites/default/files/cas_report.pdf   galvan, a. (2015). soliciting performance, hiding bias: whiteness and librarianship. in the library with the lead pipe. retrieved from http://www.inthelibrarywiththeleadpipe.org/2015/soliciting-performance-hiding-bias-whiteness-and-librarianship   gold, m. l., & grotti, m. g. (2013). do job advertisements reflect acrl’s standards for proficiencies for instruction librarians and coordinators? the journal of academic librarianship, 39(6), 558-565. https://doi.org/10.1016/j.acalib.2013.05.013   harper, r. (2012). the collection and analysis of job advertisements: a review of research methodology. library and information research, 36(112), 29-54. https://doi.org/10.29173/lirg499   hartnett, e. (2014). nasig’s core competencies for electronic resources librarians revisited: an analysis of job advertisement trends, 2000–2012. the journal of academic librarianship, 40(3-4), 247-258. https://doi.org/10.1016/j.acalib.2014.03.013   henninger, e., brons, a., riley, c., & yin, c. (2019). perceptions and experiences of precarious employment in canadian libraries: an exploratory study. partnership: the canadian journal of library and information practice and research, 14(2), 1-22. https://doi.org/10.21083/partnership.v14i2.5169   henninger, e., brons, a., riley, c., yin, c. (2020): precarity in libraries – partnership job postings. http://doi.org/10.25314/5089ab2f-5583-4675-8484-8446c0d9a701   henricks, s. a., & henricks-lepp, g. m. (2014). desired characteristics of management and leadership for public library directors as expressed in job advertisements. journal of library administration, 54(4), 277-290. https://doi.org/10.1080/01930826.2014.924310   lacey, s. (2019). job precarity, contract work, and self-care. partnership: the canadian journal of library and information practice and research, 14(1), 1-8. https://doi.org/10.21083/partnership.v14i1.5212   lakens, d. (2018). justify your alpha by decreasing alpha levels as a function of the sample size. in the 20% statistician. retrieved from https://daniellakens.blogspot.com/2018/12/testing-whether-observed-data-should.html   maccaferri, j. t., & harhai, m. k. (2019). what pennsylvania public libraries want: an analysis of pamailall job advertisements. pennsylvania libraries: research & practice, 7(1), 9-24. https://doi.org/10.5195/palrap.2019.201   maciel, m. l., kaspar, w. a., & vanduinkerken, w. (2018). (desperately) seeking service leadership in academic libraries: an analysis of dean and director position advertisements. journal of library administration, 58(1), 18-53. https://doi.org/10.1080/01930826.2017.1399711   messum, d., wilkes, l., peters, k., & jackson, d. (2016). content analysis of vacancy advertisements for employability skills: challenges and opportunities for informing curriculum development. journal of teaching and learning for graduate employability, 7(1), 72-86. https://doi.org/10.21153/jtlge2016vol7no1art582   pasma, c., & shaker, e. (2018). contract u: contract faculty appointments at canadian universities. in canadian centre for policy alternatives. retrieved from https://www.policyalternatives.ca/publications/reports/contract-u   skene, e. (2018). shooting for the moon: an analysis of digital initiatives librarian job advertisements. digital library perspectives, 34(2), 84-90. https://doi.org/10.1108/dlp-06-2017-0019   skyrme, a. e., & levesque, l. (2019). new librarians and the practice of everyday life. canadian journal of academic librarianship, 5, 1–24. https://doi.org/10.33137/cjal-rcbu.v5.29652   sproles, c., & clemons, a. (2019). the migration of government documents duties to public services: an analysis of recent trends in government documents librarian job advertisements. the reference librarian, 60(2), 83-92. https://doi.org/10.1080/02763877.2019.1570419   tewell, e. c. (2012). employment opportunities for new academic librarians: assessing the availability of entry level jobs. portal: libraries and the academy, 12(4), 407-423. https://doi.org/10.1353/pla.2012.0040   wilkinson, z. t. (2015). a human resources dilemma? emergent themes in the experiences of part-time librarians. journal of library administration, 55(5), 343-361. https://doi.org/10.1080/01930826.2015.1047253   wilkinson, z. (2016). a review of advertisements for part-time library positions in pennsylvania and new jersey. library management, 37(1/2), 68-80. https://doi.org/10.1108/lm-08-2015-0054   wise, s., henninger, m., & kennan, m. a. (2011). changing trends in lis job advertisements. australian academic & research libraries, 42(4), 268-295. https://doi.org/10.1080/00048623.2011.10722241     appendix coding fields, categories, and criteria for job postings   field categories notes job type archivist, assistant, librarian, manager, technician, other archivist = positions requiring a mas or equivalent assistant = positions using language such as assistant, associate, or clerk, typically not requiring library-specific credentials librarian = positions requiring an mlis or equivalent manager = positions with direct supervisory responsibilities requiring any kind of degree technician = positions requiring a library technician diploma or equivalent other = positions not fitting any of the above categories institution type academic, government, public, school, other positions were coded according to the kinds of institutions in which they were based. part-time full-time, part-time positions specifying 35 or more weekly hours were coded as ‘full-time,’ while those specifying fewer were coded as ‘part-time.’ positions that did not specify a number of hours and were not coded as on-call were assumed to be full-time. number of hours 1-10, 11-20, 21-30, 35+, variable, not specified. positions were further broken down based on ranges of hours worked. full-time jobs were assumed to be 35+ hours, and part-time jobs that did not specify hours were coded as ‘not specified.’ on-call regular, on-call positions that explicitly used language such as auxiliary, casual, on-call, and occasional, as well as postings which explicitly stated varying schedules and hours of work, were coded as ‘on-call.’ contract ongoing, contract positions that explicitly used language such as contract, term-limited, sessional, and temporary were coded as ‘contract.’ contract duration (months) [number of months], not specified coded based on the posting. duration was rounded to the nearest full month for durations expressed in weeks or specific date ranges. postings listing contracts as lasting ‘up to’ a period of time were coded as lasting the maximum duration. contracts that did not specify duration were coded as ‘not specified.’ precarious? yes, no any position coded as on-call, contract, or part-time was coded as ‘yes.’ education level library technician diploma or equivalent, mas or equivalent, mlis or equivalent, mlis or library technician diploma, other postsecondary degree, secondary diploma, some library coursework, not specified, other coded based on the minimum educational level required in the posting. postings that did not require a specific educational status were coded as ‘not specified.’ minimum experience (months) [number of months], not specified coded based on the posting. postings that required experience ‘up to’ a certain amount were coded as 0 months since there was explicitly no lower bound. postings that did not specify minimum required experience were coded as ‘not specified.’       microsoft word es 4062 evidence based library and information practice 2008, 3:4 69 evidence based library and information practice evidence summary positive perceptions of access to online library resources correlates with quality and quantity of scholarly publications among finnish academics a review of: vakkari, pertti. “perceived influence of the use of electronic information resources on scholarly work and publication productivity.” journal of the american society for information science and technology 59.4 (feb. 15, 2008): 602-12. reviewed by: scott marsalis social sciences librarian university of minnesota libraries minneapolis, mn, united states of america e-mail: marsa001@umn.edu received: 29 august 2008 accepted: 14 november 2008 © 2008 marsalis. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – to investigate the relationship between academics’ use of library electronic resources and their opinions regarding how these resources have impacted their work, and to investigate the association between this perceived influence and publication productivity during the previous two years. design – two specific questions added to an annual online user-survey questionnaire; additional data mined from survey setting – twenty-two finnish universities served by finelib, the finnish electronic library. subjects – seven hundred and sixty seven academic staff and full-time doctoral students. methods – a questionnaire was posted in april 2007 on finelib’s homepage and advertised on each university library’s main page, and focused on respondents’ experience in the previous two years. participants selected answers either from a list of category choices, or, when measuring evidence based library and information practice 2008, 3:4 70 perceptions, by rating agreement with statements along a four-point scale. controlled variables measured were the respondents’ academic position, their discipline, membership in a research group, whether their literature use was disciplinespecific or interdisciplinary, and their perception of the availability online of the relevant core literature. the independent variable measured was the scholars’ perception of the impact of the use of electronic library resources on their work. the dependent variable measured was the scholars’ self-reported publications in the two years preceding the survey. main results – participants reported a positive impact on the efficiency of their work, most strongly in areas of ease of access, with lesser impacts in the range of materials available to them and the ease with which they can keep up-to-date in their field. to a lesser extent, the scholars perceived a positive impact on the quality of their work. upon analysis, the study found that access to online library resources improved scholars’ work by the interconnected mechanisms of the ease of access and breadth of resources available positively impacting their ability to keep abreast of new developments and inspiring new ideas. the study found mixed results between perceived improved access and number of publications. although representation in national publications was not significantly impacted, there was a positive correlation with the number of international publications. there were interesting differences among disciplines and academic status, with a decreased impact among scholars in the humanities, and greater impact among lower-status or novice academics. conclusion – there are positive perceptions of the accessibility of online information and of its impact on the quality of work, and a correlation between these perceptions and the number of international publications, thus validating the investment in providing access to digital information resources to finnish academics. commentary this study seems to present significant evidence for positive results in scholarly output in an environment of easily accessible online library resources, but concerns about the methodology limit its usefulness. the setting for the study was finelib, a licensing consortium of finnish universities providing the major channel for accessing online journals and bibliographic databases to university faculty. two questions were added to an annual user survey, which were used in conjunction with data mined from the rest of the survey. the first question measured “how has the use of electronic resources affected your work/study/” by presenting eight statements to be ranked along a four-point scale. the second gathered information about publication productivity by asking for the number of peer-reviewed publications in the previous two years, categorized as national or international publications. the participants self-selected, the population was validated against the total finnish academic population as reported by kota online university statistics, and the author states the sample serves as relatively good model of finnish academia. however the sample is small, about five per cent of the population, and skewed, both in terms of status and discipline. thus, the statistical techniques used in the study, which assume normal distribution and equal variance evidence based library and information practice 2008, 3:4 71 among groups, are inappropriate to the sample. one area of concern is the decision to combine data elements which are inequivalent, e.g., the grouping of participants self-identified as full-time doctoral students with those identifying as assistant/researcher in order to better compare to a national data set. this action distorts the sample population. another instance of concern is the analysis of respondents’ perceived influence of eresources. the participants rated each of eight statements along a four-point scale: “considerably,” “to some extent,” “not at all,” or “don’t know.” because of the small number of responses in each of the latter two categories, responses of “not at all” and “don’t know” were collapsed into a single category; however these two response categories are too dissimilar to warrant this approach. there is also evidence of straining for statistical significance. at one point the author states, “the coefficients in natural sciences and engineering are nearly significant….” however, “nearly significant,” is an irrelevant statement; the values either are, or are not, significant. the questionable analyses of the data collected in this study unfortunately weakens both the validity of the author’s conclusions and the value of this study to evidence based practice. editorial   evidence summary theme: management   heather macdonald associate editor (evidence summaries) health and biosciences librarian carleton university ottawa, ontario, canada email: heather.macdonald@carleton.ca      2022 macdonald. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30275     the evidence summaries (ess) in this issue focus on the eblip domain of management. koufogiannakis et al. (2004) define this domain as “managing people and resources within an organization. this includes marketing and promotion as well as human resources” (p. 233).   much of the recent library management literature concentrates on the covid-19 context. this series of ess is no exception. they touch on flexible work arrangements (here to stay), resilience (challenges in achieving it), and impact of organization culture on staff morale (all about connection, respect, and value) during the pandemic. in addition, these ess critically appraise articles on agile project management (a how-to), managing change (a blueprint for tackling it), and again on staff morale (evidence of low morale in public libraries).    there are myriad challenges and opportunities in libraries for leaders, managers, and staff.  the ess in this issue give a taste of some of these. evidence of the nature of these challenges and theoretical and concrete examples of how to address them are highlighted. we hope you find helpful ideas and tools that you can bring to your own libraries.   references   koufogiannakis, d., slater, l., & crumley, e. (2004). a content analysis of librarianship research. journal of information science, 30(3), 227–239. https://doi.org/10.1177/0165551504044668 news   call for applicants for eblip journal: associate editor (research articles)    2021. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29968     evidence based library and information practice (eblip), a peer reviewed open access journal published since 2006, seeks to fill a position of associate editor (research articles). the successful candidate will be expected to begin their role by september 2021 and serve a three-year term (with possibility of renewal). this person will share the workload with a second associate editor.   the associate editor (research articles) will be responsible for:   overseeing the complete editorial process for items submitted to the research articles section (this includes assignment of peer reviewers, monitoring the peer review process, communicating with authors and peer reviewers, providing editing guidance to authors, ensuring required changes to manuscripts are made, making acceptance decisions, and communicating with copyeditors and the production editor to ensure the final copy is as expected). communicating and consulting with the editor-in-chief on a regular basis. attending editorial team meetings via zoom on a quarterly basis and participating in the overall governance of the journal. ensuring that all deadlines are met. communicating with potential authors and responding to queries.   for more information about research articles and other types of submissions accepted by eblip see: https://journals.library.ualberta.ca/eblip/index.php/eblip/guidelines   the ideal candidate for associate editor (research articles) will be well-versed in evidence based practice and research methods. this position requires dedicated time on a regular basis, and it is therefore essential that interested persons ensure available time to devote to this position prior to applying. it is estimated that the workload is approximately 15+ hours per month.   interested persons should send a resume/cv and cover letter indicating areas of strength they would bring to the role to ann medaille, editor-in-chief, at amedaille@unr.edu by july 16, 2021. the resume/cv and cover letter should be sent as a single pdf file. the current associate editors (research articles) are happy to answer specific queries about the role on request. please email erin owens (eowens@shsu.edu) and lisl zach (lisl@marsez.com).   **please note that evidence based library and information practice is a non-profit, open access journal, and all positions are voluntary and unpaid.   about the journal:   published quarterly and hosted by the university of alberta, this peer-reviewed, open access journal is targeted at all library and information professionals interested in an evidence based model of practice. by facilitating access to librarianship research via original research articles and evidence summaries of relevant research from the library literature, evidence based library and information practice enables librarians to practice their profession in an evidence based manner. please visit the evidence based library and information practice web site (https://journals.library.ualberta.ca/eblip/index.php/eblip) for further information about the journal.   evidence based library and information practice 2012, 7.2 5 evidence based library and information practice article interactions: a study of office reference statistics naomi lederer liberal arts librarian morgan library colorado state university fort collins, colorado, united states email: naomi.lederer@colostate.edu louise mort feldmann business and economics librarian morgan library colorado state university fort collins, colorado, united states email: louise.feldmann@colostate.edu received: 22 nov. 2011 accepted: 19 may 2012 2012 lederer and feldmann. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/bync-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – the purpose of this study was to analyze the data from a reference statistics-gathering mechanism at colorado state university (csu) libraries. it aimed primarily to better understand patron behaviours, particularly in an academic library with no reference desk. methods – the researchers examined data from 2007 to 2010 of college liaison librarians’ consultations with patrons. data were analyzed by various criteria, including patron type, contact method, and time spent with the patron. the information was examined in the aggregate, meaning all librarians combined, and then specifically from the liberal arts and business subject areas. results – the researchers found that the number of librarian reference consultations is substantial. referrals to librarians from csu’s morgan library’s one public service desk have declined over time. the researchers also found that graduate students are the primary patrons and email is the preferred contact method overall. mailto:naomi.lederer@colostate.edu mailto:louise.feldmann@colostate.edu evidence based library and information practice 2012, 7.2 6 conclusion – the researchers found that interactions with patrons in librarians’ offices – either in person or virtually – remain substantial even without a traditional reference desk. the data suggest that librarians’ efforts at marketing themselves to departments, colleges, and patrons have been successful. this study will be of value to reference, subject specialist, and public service librarians, and library administrators as they consider ways to quantify their work, not only for administrative purposes, but in order to follow trends and provide services and staffing accordingly. introduction reference services have traditionally been measured in some way in order to collect evidence, most commonly by a simple tick mark to indicate a transaction. in late 2006, colorado state university (csu) libraries moved from a traditional reference desk model to a referral system. staff and students working at a library information desk started to refer patrons to librarians for in-depth assistance, and the librarians wanted to collect data about their in-office reference consultations in order to capture information about this new service. csu is a land-grant institution located in fort collins, colorado, united states, with an fte of approximately 25,000 students. the libraries consist of a main library, morgan library, and a veterinary teaching branch. the csu libraries college liaison librarians unit consists of 10 librarians and 2 staff members. since 2007, these librarians have used a reference database developed in-house to record office research consultations. this database provides a place to input various data and to generate reports for librarians, the college liaison unit, and the libraries administration. administrators can use the database to see specific liaison workloads and which subjects have the most inquiries, and can then use this information for rebalancing of assignments (e.g., subjects reconfigured or other responsibilities reassigned to compensate for a heavier load) and justification of budgets for additional librarians and other relevant resources. at csu, college liaison librarians do not staff a public service desk, but provide reference assistance in their offices via drop-in and appointments. additionally, some librarians offer reference services in departments or colleges for two to four hours each week. csu libraries has a help desk at which staff and students may refer in-depth questions to librarians. the researchers were curious about how csu university library patrons are seeking information. claims that reference statistics are declining may refer only to data from the traditional reference desk. are patrons still seeking librarians for assistance? are trends at a national level, such as a decline in reference desk statistics, occurring locally? the data from the office statistics database provided an opportunity to identify patterns and to explore how patrons are seeking reference services, and in 2011 the database statistics were analyzed to answer these questions. the subject areas of the questions were also of interest because they might reflect success in outreach or areas that might be candidates for additional promotion of services. in this study, the researchers identified overall trends and looked specifically at the subject areas of liberal arts and business. literature review the broad topic of library statistics often encompasses collection holdings, staffing, and circulation data. in line with the focus of this article, only literature relating to library reference statistics was examined. only one article was found that discusses the collection of reference statistics resulting from transactions originating from multiple sources (reference desk, email, phone, instant messaging, etc.); the majority of articles focus on public service desk statistics, and those which have relevant ideas are discussed below. few articles consider how statistics are evidence based library and information practice 2012, 7.2 7 gathered, but rather focus on the results of the statistics gathering. furthermore, no close analyses of any particular librarians’ office interactions were found. novotny (2002) shows how some libraries collect reference statistics on paper, including example sheets with categories that in some cases are used away from services desks. examples include separate telephone and email reference question sheets, weekly summaries, and a question sheet with options for multiple types of contact with the patron available for each question. the summary of reference statistics covers public desks, not office numbers. measures for electronic resources (e-metrics) (2002) discusses digitally based reference (and other) transactions. possible statistics are provided for networked and electronic services and resources, but the emphasis is on electronic resources, not on the work that librarians might be doing somewhere other than at a reference or public service desk. electronic reference is just one aspect of the paper, and in any case it has changed substantially since 2002. in providing guidelines for gathering digital reference statistics, mcclure, lankes, gross, and choltco-delvin (2002) point out that “libraries have seriously underrepresented their services in terms of use of digital services being provided . . . by not counting and assessing these uses and users. as more users rely on digital library services—including digital library services—this undercount will continue to increase” (p. 8). the measures in these guidelines focus on digital reference, rather than on any kind of off-desk assistance. nevertheless, this type of statistical gathering could be a useful starting point for a library developing a statistics-gathering database. the majority of articles on digital reference services do not focus on the methods patrons use to contact librarians directly. instead they mention digital reference in passing or provide a careful analysis of where and how the services are available at specific locations (lederer, 2001; pomerantz, nicholson, belanger, & lankes, 2004; white, 2001), or focus on nonaffiliated users of the service (kibbee, 2006). articles on digital reference outside of north america and collaborative reference efforts are not closely enough related to the current topic to be included. some researchers have classified types of questions asked at reference or email reference services. henry and neville (2008) discuss how the katz classification, as detailed in introduction to reference work, and the warner classification, as detailed in his article “a new classification for reference statistics,” measured experiences at a small academic library. henry and neville include references to association of research libraries (arl) statistics and comparisons of newer types of access such as chat, email, and instant message services used in public as well as academic libraries. meserve, belanger, bowlby, and rosenblum (2009) applied the warner classification at their institution, evaluating it favourably and using it to support their tiered reference arrangement. greiner (2009) responds to the article by meserve et al. by questioning some of the conclusions and notes the decline in questions overall, attributing it partly to incorrectly interpreted questions, but also citing relationship building by librarians as a necessary component of good reference service. meserve (2009) replies with overall agreement, while emphasizing that at his library the paraprofessionals are well versed in their role, and reference services were in decline before paraprofessionals were put on a service desk. the evaluation of reference service is a frequent topic in the literature. logan (2009) provides a good overview in which he starts from the beginnings of reference services in 19th-century america, and points out that although the tools have certainly changed, the functions of reference have not. reference was not often discussed in publications until the 1970s, with an emphasis on assessment and evaluation of reference services in the 1990s, and more recently on “‘learning outcomes’ and ‘information literacy’” (p. 230). logan recommends the “establish[ment of] flexible criteria for good service,” which include components related to “behavioral characteristics . . . basic knowledge of resources and collections, subject knowledge, and reference skills” (p. 231). welch (2007) evidence based library and information practice 2012, 7.2 8 describes and discusses the national information standards organization’s (niso) z39.7-2004 standard, and highlights the importance of counting email, web page, and other reference transactions. library services have progressed beyond traditional desk transactions, and a method for tracking all reference transactions is necessary. library administrators need to be convinced of the relevance and importance of these new methods for providing research services. as welch (2007) writes, “including electronic reference transactions and visits to referencegenerated web pages in statistical reports are ways to demonstrate . . . our continuing usefulness to our patrons” (p. 103). the amount of effort expended for different types of questions is explored by gerlich and berard (2007, 2010). they outline 6 levels of effort and provide charts of questions by type for the 2003-2004 academic year (2007); and further broaden the collection of data to 15 libraries in 2010. one of the main points is that collection of statistics only from a traditional reference desk does not capture all reference transactions that are taking place – many transactions are via email and other methods. gerlich and berard (2010) argue that “reference transactions are on the decline as documented by librarians and their institutions, yet reference activities taking place beyond traditional service desks are on the rise” (p. 116). data collection techniques all too frequently do not take these additional assistance points into account, and “counting traffic numbers at the traditional reference desk is no longer sufficient as a measurement that reflects the effort, skill, and knowledge associated with this work” (p. 117). gerlich and berard discuss expended effort and difficulty in their larger reference effort assessment data (read) experiment (2007). murgai (2006) describes one library’s sampling of number and types of questions, which both notes some disadvantages of sampling, but also shows that the results of the sampling were within acceptable ranges of accuracy (though reference questions beyond the desk are not included). another statistic that is more difficult to collect is the often multiple and varied types of resources used by a librarian to answer a single question (tenopir, 1998). thomsett-scott and reese (2006) examine whether there is a relationship between changes in library technology and reference desk statistics. they note the changes in the number of questions when cd-roms and web-based resources were first introduced, and report that while reference statistics may be declining, the types of questions are “more intricate” or “complex” (p. 148): a review of the literature suggests that reference questions are taking longer to answer and are more extensive, yet the actual number of questions is declining. reference managers may need to reconsider how reference services are measured. statistics may be lower due to issues with the traditional recording method of “one patron, one tick” (p. 149). in other words, in the past a patron might come to the desk asking about books on a topic, then return to ask about articles, then return to ask for help with citations (three transactions). in the electronic world, this one patron is likely to be helped in a single transaction. additionally, the authors point out that “traditional statistical recording systems also may not include reference questions answered beyond the reference desk” (p. 162). they also examined gate counts for 1997-2004, circulation counts for 1998-2004, and various reference counts and types from 1989-2004 from their own library. not all types of statistics were gathered for all years, as email and chat started only in 1998. they conclude that “statistics should include online reference methods and possibly web page statistics as the proliferation of library-based web pages may . . . be answering many of the questions that face-to-face reference services answered in the past” (p. 163). some articles describe in-house databases created to collect reference statistics. aguilar, keating, and swanback (2010) describe the thinking behind their library’s in-house database as a “need to discover new ways to gauge the needs of our patrons and employ evidence based library and information practice 2012, 7.2 9 concrete data to make decisions” (p. 290). statistics are gathered at multiple service points – mostly reference desks, but also offices and remote locations as well – and used to justify collections purchases and increased staffing of their “ask a librarian” service during a specific time of day. data from arl and other reports are easily gathered from aguilar, keating, and swanback’s database. a second in-house database is described by feldmann (2009), which was created to capture the number of reference questions that were successfully referred from the new information desk after the reference desk was disbanded. the database evolved to be a useful tool for gathering information on librarians’ office transactions. the author cites articles that discuss referral services and various staffing models for tiered services. smith (2006) describes a web-based system for collecting statistics and discusses various reasons people have collected reference statistics, as well as the problems associated with collecting them, such as apathy and the wide variation in the parts and types of questions. the author describes how the database was developed and the types of information it collects, including screen shots and html coding of and for the database. the references and further reading are substantial. todorinova, huse, lewis, and torrence (2011) describe one university library’s choice of a commercial product, desk tracker, after using a system of clickers that did not record the time of transactions. the data collected included type of patron, form of the transaction (in person, email, or phone), and type of question, and it was used to assign appropriate staffing levels and to inform collection development decisions. some output weaknesses were found in the software, but the data have been proposed as potentially useful for decisionmaking and improving services and operations. although some of the literature examines reference statistics closely, it is in specific contexts such as health or medical libraries or gis systems (e.g., parrish, 2006), and has a more focused audience and set of questions. the literature still lacks a close examination of reference transactions away from the reference desk. this study looks closely at not only how the questions were asked, but how long it took to answer them, their subject areas (broadly and more specific, depending upon the topic), the status of the questioner, and whether or not the question was referred from someone else (e.g., via a service desk in the majority of cases). methods data were gathered from an office statistics database, which is a recording mechanism used to capture csu librarians’ reference transactions, both in-office and during office hours in a department or college on campus (see figure 1). the database was developed in late 2006, when a csu libraries business librarian, a staff member, and a member of the library’s technical services department created it using php scripting language and mysql. it was originally conceived as a method to track referrals from the newly implemented information desk (feldmann, 2009). starting in 2007, librarians no longer staffed a reference desk or any other public services desk, and staff and students working at the information desk (the now sole public service desk) would refer in-depth questions to librarians. the database initially provided a method for capturing the number of referrals received by librarians from the information desk, in addition to providing a place to record reference transactions. since librarians at this time placed a renewed emphasis on departmental outreach, it was also thought that the database would capture the impact of marketing their research consultation services to faculty. over the years, librarians have changed or modified input fields to reflect needs and improve the database. reports are easily generated in microsoft excel spreadsheets. the input form contains both required and voluntary fields. information collected in the required fields include name, contact type (email, drop-in, phone, appointment, office hours, or other), help desk referral, time spent, number of patrons assisted, and status of patron. voluntary information includes discipline area, course information, and comments. evidence based library and information practice 2012, 7.2 10 figure 1 csu libraries office statistics database entry form for this study, the researchers extracted numbers from the database in the aggregate (total from all librarians) for the years 20072010. additionally, data from the liberal arts and business librarians were extracted as samples to examine subject-specific data. the business librarian provides assistance for six departments: accounting, finance, marketing, management, computer information systems, and economics. the primary liberal arts librarian covers seven departments: english, history, art, communication studies, journalism and technical communication, ethnic studies, and design and merchandising, which is part of the college of applied human sciences. the database allows data to be pulled directly by various fields, date range, and by librarian. the researchers extracted data by contact type, number of patrons helped, time spent, patron status, and whether or not the question was a referral from the help desk for the years 2007-2010. this information was then examined to determine trends. results and discussion aggregate information table 1 shows that both the number of consultations and the numbers of librarians reporting have decreased between 2007 and 2010. while the total number of office consultations decreased by year, a corresponding drop in the number of librarians reporting also occurred, so that the mean (average per librarian) increased from 127 in 2007 to 154 in 2010. fewer librarians were employed in 2010 than in previous years due to attrition. table 2 shows that email was by far the most popular way that patrons received assistance, accounting for 50% (3,141 questions) of all transactions. evidence based library and information practice 2012, 7.2 11 table 1 2007-2010 office consultations by year year no. librarians reporting 2007 1,517 12 2008 1,856 12 2009 1,515 11 2010 1,395 9 table 2 2007-2010 office contact type contact type no. percent email 3,141 50% drop-in 1,214 19% phone 748 12% appointment 714 11% other 424 7% office hours 40 1% empty 2 0% total 6,283 the contact type of “office hours” refers to librarians providing dedicated office hours to answer questions from drop-in patrons, similar to traditional office hours that faculty provide. they were recorded only in january and february of 2007 as they were a short-term arrangement where librarians were assigned to be backups for the then information desk. referrals from the information desk were so rare that the concept was abandoned after a short run. “empty” indicates that no information was entered. “other” could mean helping someone in the library while en route to a meeting or returning to one’s office, service provided at a non-library location (for instance, the business librarian’s “librarian to go” reference in the college of business), instant messaging (im), and so on. the status of patrons who directly contacted librarians (table 3) shows that graduate students and undergraduate students are the heaviest users with faculty members in a solid third place. table 3 2007-2010 office patron status patron status no. percent graduate 2,030 32% undergraduate 1,969 31% faculty 1,156 18% community 557 9% staff 348 6% elsewhere 80 1% government 54 0.9% empty 51 0.8% visiting faculty 24 0.4% evidence based library and information practice 2012, 7.2 12 administrator 14 0.2% these figures show that graduate students visit their college liaison librarians in greater numbers than any other category, even though they are a much smaller percentage of the university’s overall student population. the majority of consultations are relatively short (table 4). table 4 shows that consultations with librarians are for the most part between 10 minutes and 1 hour 25 minutes. researchers who contact a librarian are more likely to have questions that require some research to answer, and talking with a student or faculty member in an office can often take longer than an interaction at a reference desk, as others in line at a desk can speed up a reply. a user who makes an appointment is not going to rush off. of course, there are questions that need only a brief answer; the 1 to 4 minute category includes any number of interactions that took no more than 10 seconds, but were recorded as one minute (see table 5). email dominates this (and all) categories, but drop-ins are also brief as a patron may have a quick question and thus stop by without an appointment (or is referred from the information or help desks). the number of referrals to librarians from the information or help desks is much lower than expected, as seen in table 6. additionally, referrals have been decreasing as time passes, as shown in table 7. table 4 2007-2010 office consultations – time spent with patron time spent no. patrons 15m-29m 1,667 30m-59m 1,321 10m-14m 853 1h-1h25 773 5m-9m 694 1m-4m 465 1h30m-1h59m 210 empty 143 2h+ 118 3h+ 24 4h+ 9 5h+ 6 note. m=minute; h=hour table 5 2007-2010 office consultations – short contacts time phone other office hours email drop in 1 minute 29 35 11 56 29 2 minutes 28 28 4 123 31 table 6 2007-2010 office consultations – referrals referral no. yes 1,041 no 5,242 evidence based library and information practice 2012, 7.2 13 librarians have made a push to directly promote themselves to students and faculty in order to provide the best possible service to their constituents. for example, flyers promoting the college liaison librarians by name and specialty have been distributed to faculty in departments. some librarians offer reference assistance for a few hours a week in departments or colleges on campus and this has increased the visibility of librarian services to faculty, staff, and students in these areas, and possibly resulted in direct contacts rather than referral from the reference desk. additionally, college liaison librarians are promoting their services directly to students in their library instruction sessions. it has been observed that faculty members who are familiar with the librarians’ services are more likely to refer their students directly to their college liaison librarian. the actual referrals from the service desk may be an even lower percentage than those recorded here; some librarians record a “referral” when a faculty member refers a student directly to a librarian. the decline in referrals from the information and help desks prompts many questions. do desk staff give patrons a librarian’s business card but the patron decides, for whatever reason, not to contact the librarian directly? are the desk personnel unfamiliar with the college liaison librarians, and therefore feel uncomfortable referring questions to them? are the desk staff and patron satisfied with the result of the transaction? has the nature of questions changed? do web pages and libguides play any role in filling research needs? these are all questions for further examination. subject-specific information: business csu libraries business office statistics consist of office research consultations, reference assistance during “office hours” held in the csu college of business, and assistance via instant messaging, email, phone, and referrals from the libraries’ ask-a-librarian email service. csu libraries has one business librarian who serves 5,800 students, including business majors and minors as well as onsite and distance graduate students. csu college liaison librarians enter their office statistics differently, and this impacts how the results can be analyzed. the business librarian designates all questions having to do with business as “business,” rather than parsing out further into such categories as finance or accounting. questions are often multi-disciplinary and it may be difficult to categorize the reference consultation topic into only one area. for example, students in csu’s college of business global social sustainable enterprise program often research a country’s social, political, and economic climate along with business logistics. business reference questions have generally been increasing, with a slight dip in 2010. total contacts in 2007 were 210, in 2008 were 356, in 2009 were 360, in 2010 were 298, and in 2011 were 342. taken alone it is difficult to explain the decrease in 2010 or why numbers are not continually increasing given that the number of business students is increasing, but it could be attributable to successful instruction sessions, students using the library’s libguides to find answers, more library-savvy table 7 2007-june 2011 office consultations – referrals year yes no. percent referred total no. questions 2007 478 1,040 32% 1,518 2008 325 1,530 18% 1,855 2009 139 1,376 9% 1,515 2010 99 1,296 8% 1,395 2011 (jan.-june) 32 729 4% 761 evidence based library and information practice 2012, 7.2 14 business students, and assignments requiring less or no library research. these results warrant further investigation, potentially through analysis of libguide and instruction statistics, or by more qualitative methods, particularly if the questions continue to show a decline in future years. contact types for business are similar to the aggregate data, with email being the primary contact type. table 8 shows contacts, percent of total, and a comparison with the aggregate (overall) percentages. undergraduates are the primary patrons for the years examined (see table 9). table 10 shows a comparison by year of patron status. in 2010, a trend change indicates that graduate more often than undergraduates are the more common contact type. further breakdowns were explored. graduate students contact the business librarian primarily by email (42%, 2007 through june 2011). the average time spent with a graduate student was 35 minutes. undergraduates contact the business librarian also primarily table 8 business librarian – contact type (january 2007-june 2011) contact type no. percent overall percent email 722 51% 50% drop-in 220 16% 19% other 215 15% 11% phone 134 9.6% 12% appointment 110 7.9% 7% table 9 business librarian – patron status (january 2007-june 2011) patron status no. percent overall percent undergraduates 486 40% 31% graduates 423 35% 32% faculty 145 12% 18% community 98 8% 9% staff 52 4% 6% visiting faculty 0 0% 0% elsewhere 22 2% 1% table 10 business librarian – patron status by year patron 2007 2008 2009 2010 2011 june undergraduate 53% 41% 40% 28% 28% graduate 28% 35% 33% 40% 39% faculty 8% 13% 11% 14% 20% community 9% 7% 8% 9% 6% staff 2% 2% 6% 6% 8% visiting faculty 0% 0% 0% 0% 0% elsewhere 0% 1% 2% 2% 0% evidence based library and information practice 2012, 7.2 15 by email (54%, 2007 through june 2011). average time spent with an undergraduate patron is similar to that spent with graduate students. while the aggregate data (table 4) show that most student office consultations are between 15 and 29 minutes, the business student data indicate that slightly more time is spent with them than the average of all patrons. subject-specific information: liberal arts the primary liberal arts librarian (responsible for 6 of the 13 departments in the college) answered 158 questions in 2007, 158 questions in 2008, 189 questions in 2009 (an increase of 31), 220 questions in 2010 (another increase of 31), and 251 questions in 2011. the trend has been higher numbers of questions after the second year. compared to the whole, the liberal arts librarian’s numbers have not always reflected the same trends, as seen in table 11. in the interactions of the liberal arts librarian, email, phone (by just 3%), and other were a smaller percentage of the total than for other librarians, with email showing a much smaller percentage; however, the liberal arts librarian had a higher percentage of drop-ins and appointments than other librarians. another difference from the whole was patron status (see table 12). undergraduates contacted the liberal arts librarian 22% more often than the overall population; however, graduate students made 11% fewer contacts (see table 12). faculty, community contacts, visiting faculty, and elsewhere were close to the overall picture. a possible explanation is that the liberal arts librarian teaches fewer graduate than undergraduate courses. moreover, 20% of the graduate student numbers (32 students) come from a non-liberal arts department, where she has taught the new graduate students in the library classroom every fall. table 11 liberal arts librarian – contact type (january 2007-june 2011) contact type no. percent overall percent email 309 38% 50% drop-in 246 31% 19% appointments 176 22% 11% phone 69 9% 12% other 1 0% 7% table 12 liberal arts librarian – patron status (january 2007-june 2011) patron status no. percent overall percent undergraduate 427 53% 31% graduate 166 21% 32% faculty 127 16% 18% community 55 7% 9% staff 17 2% 6% visiting faculty 5 0.6% 0.4% elsewhere 4 0.4% 1% evidence based library and information practice 2012, 7.2 16 as for the disciplines in which questions were asked, the top categories cover many subject areas (table 13). there were 35 areas represented, with the top 13 shown in table 13 (16 categories had 1 entry while 6 had 2-4, making up 5% of the total). there were many questions in design & merchandising, the non–liberal arts subject. subjects outside of liberal arts appear because the specialist for that area was not available that day, and because the liberal arts librarian’s second language is french. members of the french department are aware of her specialized knowledge from various interactions and ask questions specific to the french language of her, while in practice foreign literature research questions have been asked of the foreign languages librarian (who sometimes consults with the liberal arts librarian about these questions). comparisons across years show that history questions dominate; all but one of the four years examined had history in first place; in 2008 it was in second place and english language & literature had the most questions. english was in the top three of all years. design & merchandising was fourth in three of the years and third once (2008). of the most frequent areas, art had the most dramatic jump down from second in 2007 to fifth or seventh in the other three years. a possible explanation is a decrease in the number of library instruction sessions provided for art courses during the later years, thereby decreasing the number of students who meet their art librarian in person. conclusion this study examined patterns in patrons’ use of reference services in a library which no longer has a traditional reference desk. instead, a general help desk is used, among other methods, to refer patrons to subject-specific librarians for in-depth assistance. routinely collected data were examined to determine if patrons continue to seek librarian assistance without their presence at the reference desk. the data examined included the demographics of the primary patrons, how patrons contact librarians, and how much time librarians spend with them. these data show that from 2007 to 2010 the majority of patrons who contacted csu college liaison librarians were table 13 liberal arts librarian – office consults by discipline (january 2007-june 2011) discipline no. percent history 190 24% english language & literature 133 17% design & merchandising 100 12% speech 92 11% art 75 9% journalism 58 7% general 46 6% ethnic studies 21 3% bibliographic citation 16 2% other 11 1% education 6 1% foreign language & literature 6 1% library science 6 1% note. “bibliographic citation” is a newer entry; earlier entries were put into the “general” category. evidence based library and information practice 2012, 7.2 17 graduate students and their primary mode of contact was email. further examination of the statistics shows a marked decline over time in the number of referrals that librarians received from the information and help desks. over the same time period, there has not been a large increase in the number of office consultations, although contact numbers are fairly consistent and actually show an average increase per librarian given the decrease in number of librarians. similar trends were discovered for two subject librarians (business and liberal arts) whose data were examined separately. the database has proven to be useful for examining trends and plans for the future, including following the nature of questions (e.g., in-depth), or for using something similar to the reference effort assessment data (read) scale. college liaison librarians at csu are making efforts to promote their services on campus and these efforts may have contributed to increased awareness of librarian reference services by patrons. an in-depth examination of the direct impact of these promotion efforts would be worthwhile, although it must be noted that relying simply on statistical data may not provide a complete picture of how and why trends are occurring. at the same time, the tracking must not become so burdensome that it becomes a distraction from helping patrons. in some instances, students arrive back-to-back and asking them multiple questions takes from the time that is spent actually helping; moreover, remembering the details for later input into the database can be difficult when the patrons arrive in waves. a reference statistic was once satisfied with a quick tick mark, and while the data collected are useful, it must not end up overwhelming the people recording it. additionally, important soft data might be hard to quantify; for example, are the departments with which the librarians liaise satisfied with how their library is serving them? some subject areas/departments use the library and the librarian services more than others and this may simply be a disciplinespecific behaviour. further research to explore these patron behaviour patterns would be worthwhile. data gathering is useful for both library administrators and individual librarians as a means of quantifying their work. administrators may use this information to examine workloads and productivity, justify the need to hire new faculty, identify the need to purchase software to develop online tutorials, and identify overall trends. librarians may use the data to show their impact, see trends, and develop relevant online guides and tutorials. at csu libraries, the data revealed by the office statistics database can demonstrate which subject areas are using their college liaison (subject) librarians the most, and give guidance to the specialists as to which topical supplementary materials might be created to help serve their constituencies, such as web pages, libguides, tutorials, or handouts. it is important to remember that although data are useful, interpretation and presentation are important. quantifying librarians’ work can be difficult and may not always provide a complete picture of activity. acknowledgements parts of this article were reported at the workshop for instruction in library use (wilu) conference in regina, saskatchewan, canada, in june 2011. references aguilar, p., keating, k., & swanback, s. (2010). click it, no more tick it: online reference statistics. the reference librarian, 51(4), 290-299. doi:10.1080/02763877.2010.501421 feldmann, l. m. (2009). information desk referrals: implementing an office statistics database. college & research libraries, 70(2), 133-141. retrieved 21 may 2012 from http://crl.acrl.org/content/70/2/133.full. pdf gerlich, b. k., & berard, g. l. (2007). introducing the read scale: qualitative statistics for academic reference services. georgia library quarterly, 43(4), 7-13. retrieved 21 evidence based library and information practice 2012, 7.2 18 may 2012 from http://digitalcommons.kennesaw.edu/ glq/vol43/iss4/4 gerlich, b. k., & berard, g. l. (2010). testing the viability of the read scale (reference effort assessment data): qualitative statistics for academic reference services. college & research libraries, 71(2), 116-137. retrieved 21 may 2012 from http://crl.acrl.org/content/71/2/116.full. pdf greiner, t. (2009). letter to the editor. reference & user services quarterly, 49(2), 111. retrieved 21 may 2012 from http://rusa.metapress.com/content/k11 07l2616utk6q1/fulltext.pdf henry, d. b., & neville, t. m. (2008). testing classification systems for reference questions. reference & user services quarterly, 47(4), 364-373. retrieved 21 may 2012 from http://rusa.metapress.com/content/p80 16h827210644v/fulltext.pdf kibbee, j. (2006). librarians without borders? virtual reference service to unaffiliated users. journal of academic librarianship, 32(5), 467-473. retrieved 21 may 2012 from http://dx.doi.org/10.1016/j.acalib.2006.0 5.003 logan, f. f. (2009). a brief history of reference assessment: no easy solutions. the reference librarian, 50(3), 225-233. doi:10.1080/02763870902947133 lederer, n. (2001) e-mail reference: who, when, where, and what is asked. the reference librarian, 35(74), 55-73. doi:10.1300/j120v35n74_05 mcclure, c. r., lankes, r. d., gross, m., & choltco-devlin, b. (2002). statistics, measures, and quality standards for assessing digital reference library services: guidelines and procedures. retrieved 21 may 2012 from eric database (ed472588). measures for electronic resources (e-metrics): complete set. (2002). washington, dc: association of research libraries. retrieved 21 may 2012 from http://www.arl.org/bm~doc/emetrics.pdf.zip meserve, h. c. (2009). mr. meserve’s reply. reference & user services quarterly, 49(2), 111. retrieved 21 may 2012 from http://rusa.metapress.com/content/h07 1631726l1p5n8/fulltext.pdf meserve, h. c., belanger, s. e., bowlby, j., & rosenblum, l. (2009). developing a model for reference research statistics: applying the “warner model” of reference question classification to streamline research services. reference & user services quarterly, 48(3), 247258. retrieved 21 may 2012 from http://rusa.metapress.com/content/v8h 758vjlt275234/fulltext.pdf murgai, s. r. (2006). reference use statistics: statistical sampling method works. southeastern librarian, 54(1), 45-57. retrieved 21 may 2012 from http://selaonline.org/southeasternlibr arian/selnspring06.pdf novotny, e. (2002). reference service statistics & assessment: a spec kit. washington dc: association of research libraries. retrieved 21 may 2012 from http://www.arl.org/bm~doc/spec268w eb.pdf parrish, a. (2006). improving gis consultations: a case study at yale university library. library trends, 55(2), 327-339. doi:10.1353/lib.2006.0060 pomerantz, j., nicholson, s., belanger, y., & lankes r. d. (2004). the current state of digital reference: validation of a general digital reference model through a survey of digital reference evidence based library and information practice 2012, 7.2 19 services. information processing & management 40(2), 347-363. doi:10.1016/s0306-4573(02)00085-7 smith, m. m. (2006). a tool for all places: a web-based reference statistics system. reference services review, 34(2), 298315. doi:10.1108/00907320610669524 tenopir, c. (1998). online databases: reference use statistics. library journal 123(8), 3233. thomsett-scott, b., & reese, p. e. (2006). changes in library technology and reference desk statistics: is there a relationship? public services quarterly, 2(2/3), 143-165. doi:10.1300/j295v02n02_10 todorinova, l., huse, a., lewis, b. & torrence, m. (2011). making decisions: using electronic data collection to re-envision reference services at the usf tampa libraries. public services quarterly 7(1/2), 34-48. doi:10.1080/15228959.2011.572780 warner, d. g. (2001). a new classification for reference statistics. reference and user services quarterly, 41(1), 51-55. white, m. d. (2001). diffusion of an innovation: digital reference service in carnegie foundation master’s (comprehensive) academic institution libraries. journal of academic librarianship, 27(3), 173-187. retrieved 21 may 2012 from http://dx.doi.org/10.1016/s00991333(01)00179-3 commentary   library assessment and quality assurance creating a staff-driven and user-focused development process   håkan carlsson associate library director gothenburg university library gothenburg, sweden email: hakan.carlsson@ub.gu.se   received: 15 feb. 2016   accepted: 16 feb 2016         2016 carlsson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – gothenburg university library has implemented a process with the goal to combine quality assurance and strategic planning activities. the process has bottom-up and top-down features designed to generate strong staff-involvement and long-term strategic stability. methods – in 2008 the library started implementing a system in which each library team should state a number of improvement activities for the upcoming year. in order to focus the efforts, the system has gradually been improved by closely coupling a number of assessment activities, such as surveys and statistics, and connecting the activities to the long-term strategic plan of the library. results – the activities of the library are now more systematically guided by both library staff and users. the system has resulted in increased understanding within different staff groups of changing external and internal demands, as well as the need for continuous change to library activities. conclusion – library assessment and external intelligence are important for tracking and improving library activities. quality assurance and strategic planning are intricate parts in sustainable development of better and more effective services. the process becomes more effective when staff-driven and built upon systematic knowledge of present activities and users.   introduction   library development greatly benefits from continuous input of internal and external intelligence to be successful (davies, 2008). statistics are collected in all swedish libraries as part of the government-mandated official statistics of sweden (statistics sweden, 2016).  the impressive array of data, produced and communicated to national authorities every year, can also be of strategic use locally (høivik, 2008). in addition, qualitative data is collected regularly as patrons are surveyed, introducing ideas and user perspective into the organization.   quality assurance and strategic planning are vital parts of systematic public management (young, 2003). for libraries, it is not uncommon that these types of systems originate from the parent organization (broady-preston & lobo, 2011). the university of gothenburg implemented a university-wide quality assurance framework in 2001 (göteborgs universitet universitetsstyrelsen, 2001). a new strategic planning system has been in place at university of gothenburg since 2013 (göteborgs universitet, 2013). in addition, a number of departments (such as environment, worker’s, and fire safety) have their own systematic processes and auditing procedures to expand the picture.   all these tools are useful and valuable for improving and making our libraries more effective (dean & sharfman, 1996). however, they can feel unorganized and hard to communicate to the staff and stakeholders. gothenburg university library set out to condense the majority of these activities into one process. effort was put into building a user-focused and staff-centered bottom-up workflow. the system was further enhanced with a longer-term strategic cycle, which also relies on staff input and statistical intelligence.   methods and results   building the quality cycle the quality system framework, established by the university of gothenburg board, mandated all university schools and the library to set up quality systems. the nature of the local quality systems was not described in detail, but left to the different schools to formulate, based on individual circumstances. later, a system auditing procedure was put in place to give collegial advice on the development of the local systems.   the university library implemented its first quality system in 2003-4, when a set of goals was defined and assessed throughout the organization (götesborgs universitet biblioteksnämnden, 2003). the system was then developed and improved upon in 2008 and 2012.   describing the quality cycle the process, as it is in effect now, is based on a follow-up/planning period in january/february of each year (figure 1).   in the planning stage, each library team (7-15 staff) defines a few projects (activities) of varying size which should be completed before the end of the year. the resulting activity plans are gathered from throughout the organization and published on the library intranet. to strengthen the strategic relevance, a number of activities are selected from the current library strategic plan by the library leadership and assigned to the individual teams prior to the planning. this means that each team may have one or two activities which must be included in the activity plan in order to keep pace with strategic goals.     figure 1 quality cycle.   figure 2 input to follow-up activities.     to give the staff teams a current background for their work, the planning period is preceded by a follow-up and assessment routine (figure 2). during follow-up, the activity plans from the previous year are accounted for. in addition, the teams are asked to study and comment on data from a number of other sources. each year has a follow-up theme. centrally prepared reports are presented to the teams. the latest user survey may be distributed, or a report of improvement suggestions from staff or users can be presented. staff participation often leads to discussions of the current matter, for instance, why visitor numbers are down, or the nature of the enquiries at the front desk.   the follow-up period has proven to be a great inspiration for activity planning, which normally is scheduled for the following team meeting. it also allows follow-up activities from auxiliary management systems to be incorporated, making them more accessible to the staff.     building the strategic cycle some management activities are repeated less frequently than once a year. one of these is the development of the library strategic plan. a weakness in quality assurance work is that it does not necessarily make the library activities more focused or more effective. a long-term direction has to be established.    the way to handle this challenge was the creation of a strategic cycle (figure 3). early in the year before a new strategic plan is set to take effect, external intelligence is gathered in a more deliberate way than normal. vision statements and other long-term documents are revisited and revised. later that year, seminars and group discussions consolidate the material, and a new strategic plan is drafted. after approval, once the plan has taken effect, the quality cycle is used to drive the implementation.   collection of intelligence intelligence collection is key to the strategic cycle. one useful technique is benchmarking, which has been performed for a number of different themes using a method derived from siq (nilsson, örtelind, & östling, 2002). the ideal benchmark activity often starts with a specific need, and the definition of a number of themes of interest. libraries are then scanned to find peers with assets in the selected areas. identified libraries are contacted and invited to add themes of their own. the library leadership teams then meet to present their library’s activities in the selected areas of interest. this often gives a richer understanding of best practices within the selected themes.   the intelligence collection process preceding the latest completed strategic cycle produced a trend cloud, which then was used to develop and focus the plan (figure 4).     figure 3 strategic cycle.     figure 4 trend areas from intelligence collection 2012. main areas accented.     conclusions the implemented quality and strategic cycle at gothenburg university library has contributed to more user-focused and experience-driven library activities. staff ownership has facilitated collective involvement in addressing the toughest issues for our changing library environment. many more employees are now actively involved in change processes, and there is a wider understanding of developments as a result of allowing staff to systematically contribute to changes in their immediate environment.   there is great benefit from including assessment results in systematic change processes. many surveys and studies of statistical data have precipitated a large number of action plans throughout the years, but the follow-up and implementation steps have been more difficult. statistical data, especially data collected for national statistics, was rarely used for library development. the definition of a yearly process into which data and previous findings can be funneled has been shown to be a powerful driving force for implementing meaningful change.   references   broady-preston, j., & lobo, a. (2011). measuring the quality, value and impact of academic libraries: the role of external standards. performance measurement and metrics, 12(2), 122-135. http://dx.doi.org/10.1108/14678041111149327   davies, j. e. (2006). taking a measured approach to library management: performance evidence applications and culture. in t. kolderup flaten (ed.), management, marketing and promotion of library services based on statistics, analyses and evaluation (pp. 17-32). berlin, boston: k. g. saur. http://dx.doi.org/10.1515/9783598440229.17   dean, j. w., & sharfman, m. p. (1996). does decision process matter? a study of strategic decision-making effectiveness. academy of management jounal, 39(2), 368-392. http://dx.doi.org/10.2307/256784   høivik, t. (2006). comparing libraries: from official statistics to effective strategies. in t. kolderup flaten (ed.), management, marketing and promotion of library services based on statistics, analyses and evaluation (pp. 43-64). berlin, boston: k. g. saur. http://dx.doi.org/10.1515/9783598440229.43   nilsson, h. l., örtelind, a.-b., & östling, m. (2002). benchmarking : att lära av andra – en handbok i benchmarking [benchmarking: to learn from others – a handbook in benchmarking] göteborg: siq.   statistics sweden. (2016). official statistics of sweden. retrieved from http://www.scb.se/en_/about-us/official-statistics-of-sweden/   göteborgs universitet (2013). planering och uppföljning. [planning and assessment] retrieved from http://medarbetarportalen.gu.se/vision2020/planering-och-uppfoljning/   göteborgs universitet universitetsstyrelsen  (2001-06-11). göteborgs universitets kvalitetssystem.  [quality system for the university of gothenburg]   göteborgs universitet biblioteksnämnden. (2003). kvalitetssystem for göteborgs universitetsbibliotek. [quality system for gothenburg university library] retrieved from point 7 on http://www.ub.gu.se/info/organisation/biblnamnd/protokoll/arendelistor/bnarenden031119/bnarende031119.pdf   young, r. d. (2003). perspectives on strategic planning in the public sector. retrieved from http://www.ipspr.sc.edu/publication/perspectives%20on%20strategic%20planning.pdf   microsoft word es_hook.doc evidence based library and information practice 2006, 1:2    43 evidence based library and information practice       evidence summary    further research is required to determine which database products best support  research in public administration    a review of:  tucker, james, corey. “database support for research in public administration.” behavioral &  social sciences librarian 24.1 (2005): 47‐60.    reviewed by:  david hook  manager, operations information and configuration management  macdonald dettwiler and associates  brampton, ontario, canada  e‐mail: david.hook@mdacorporation.com    received: 1 march 2006           accepted: 6 april 2006      © 2006 hook. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to examine the extent to which  six commercial database products support  student and faculty research in the area of  public administration.    design – bibliometric study.    setting – academic library in the united  states.    subjects – six commercial business‐related  database products were examined:  proquest’s abi/inform global edition  (abi), ebsco’s business source premier  (bsp), gale’s general businessfile asap  (gbf), ebsco’s academic search premier  (asp), ebsco’s expanded academic index  (eai) and proquest’s international  academic research library (arl).  three of  the databases (abi, bsp, gbf) were chosen  because they address the management,  human resource, and financing elements of  public administration.  the other three (asp,  eai, arl) were included because of their  multidisciplinary coverage.    methods – a list of journal titles covering  public administration was assembled from  the institute of scientific information’s social  sciences citation index and previously  published lists of recommended journals in  the field.  the author then compared the  compiled list of journal titles against the  journal titles indexed by the six database  products.  he further analyzed the results  by level of journal coverage (abstract only,  full‐text, and full‐text with embargo) and  subject area based on categories described in  ulrich’s periodicals directory.    http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:2    44 main results – the study found that three  of the six database products ‐‐eai, bsp, and  arl ‐‐ provide indexing for the greatest  number of public administration journals  contained in the compiled list.  eia and  arl cover the greatest number of those that  are full‐text journals, while bsp and asp  cover the greatest number of those full‐text  journals limited by publisher embargoes.    conclusion – the author concludes that of  the six databases examined, eai, bsp, and  arl are the best for public administration  research, based on their strength in the  subject areas of public administration and  public finance.  the author also  recommends that librarians in the field of  public administration “carefully evaluate  each database to see which one best fits the  needs of the library and patrons” (56).    commentary     although this comparison study of six  commercial databases has potential, it does  not deliver strong enough supporting  evidence to advance our knowledge of the  extent to which databases support research  in public administration.    the biggest flaw with the study is that it  does not measure what it purports to  measure.  the study claims to examine “the  extent to which databases support student  and faculty research in the area of public  administration,” yet journals were not  selected based on any identified research  needs.  the author does not make a  connection between the databases and the  research needs of undergraduates,  graduates, doctoral students, or faculty,  despite discussion of the importance of  distinguishing between the needs of these  different groups.    in assessing the database products, the  author conducted only a quantitative  journal count and did not consider other  issues such as pricing, years of back file  coverage, user interface, or percentage of  peer‐review journals.  presumably, this  study was intended to aid librarians in  database product selection in the area of  public administration. without  consideration of these additional issues, the  study’s usefulness is limited, and the  proclamation that certain database products  are ‘the best’ cannot be supported.     using ulrich’s subject classification scheme  appears to have been an unfortunate choice  for this study, because it resulted in too  many journal titles falling into the  ‘other/miscellaneous’ category.  one  database, for example, had 80% of its full‐ text journal titles fall into that subject  category, making it very difficult to assess  its usefulness.    the study is at times inconsistent in the way  it refers to the various database products –  often switching between their abbreviations,  full names, product names, and vendor  names; sometimes even using different  acronyms for the same product.  there is  also some confusion between the terms  ‘criteria’ and ‘categories’; the study states  that the databases were analyzed for content  based on three criteria: ‘abstract only, full‐ text, and full‐text with embargo’.      although the author conducted a detailed  literature search, there is no clarification of  how his study differs from previous ones, or  even an explanation of why this study  needed to be done.  one of the study’s  conclusions was that public administration  librarians should undertake their own  evaluations of the databases. it would have  been a more effective study had one of its  goals been to reduce the need for individual  libraries to conduct their own independent  studies of the databases.  had the different research needs of the  various library client groups been first  identified and issues such as pricing, user  interface, and back file coverage, been  evidence based library and information practice 2006, 1:2    45 considered, this would have been a much  stronger study and would have been able to  aid librarians in selecting database products  in the area of public administration.  as it is,  the research from this study does not lend  itself to be used in decision making.    works cited    institute for scientific information. web of  knowledge. social sciences citation index.  journal citation reports. 15 feb. 2006  <http://scientific.thomson.com/products/jcr/ >.  http://scientific.thomson.com/products/jcr/ microsoft word art_given.doc evidence based library and information practice 2007, 2:1  15 evidence based library and information practice     featurearticle    evidence‐based practice and qualitative research: a primer for library and  information professionals      lisa given  associate professor  school of library & information studies  university of alberta  edmonton, alberta, canada  email: lisa.given@ualberta.ca      received: 01 december 2006    accepted: 06 february 2007      © 2007 given. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.   abstract    objectives ‐ this paper discusses the importance of qualitative research in evidence‐based  library and information practice (eblip), with a focus on practical tips for evaluating and  implementing effective qualitative research projects.     methods ‐ the paper provides a brief introduction to the nature of qualitative inquiry and  its status within current models of evidence assessment. three problems of excluding  qualitative research from the evidence‐base in library and information studies (lis) are  identified: 1) ignoring the social sciences and humanities traditions that inform research in  the field; 2) privileging of quantitative and experimental methods over others in evidence  assessment; and, 3) focusing attention away from the best evidence for lis research  problems.     results ‐ qualitative approaches commonly used in library and information contexts are  discussed, along with strategies for assessing quality in this work and some of the common  ethics‐related issues that researchers and professionals must consider.     conclusions ‐  lis professionals are encouraged to: 1) select research methods – including  qualitative approaches – that best suit lis questions; 2) design collaborative projects that  combine quantitative and qualitative approaches, that will address research questions in a  more complete way; 3) consider qualitative measures of rigor in assessing quality – rather  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:1  16 than imposing quantitative expectations; and 4) revise existing models of “evidence” to  recognize the value and rigor of qualitative research projects. objective: this paper  discusses the importance of qualitative research in evidence‐based library and information  practice (eblip), with a focus on practical tips for evaluating and implementing effective  qualitative research projects.       introduction  evidence‐based library and information  practice (eblip) seeks to formalize the links  between research and professional activities,  to enhance information services and  programs across library contexts. arising  out of health sciences librarianship and  following in the traditions of evidence‐based  practice in medicine and related fields, the  number of conferences and publications that  support this work have seen a dramatic rise  in the last few years.    however, qualitative researchers and the  results of their work remain marginalized in  eblip (see given 2006, for an in‐depth  discussion of these issues). just as  qualitative research in nursing is often given  less weight when compared to clinical trials  and other experimental methods in  biomedical practice (morse), so too is  qualitative research pushed to the edges of  eblip. unfortunately, eblip’s grounding of  “rigor” and “best evidence” in its historic  biomedical/experimental framework often  excludes the results of qualitative research  due to the imposition of inappropriate  expectations of how data will be gathered  and analysed (law). this approach raises a  number of problems for lis professionals  who wish to develop an appropriate  evidence‐base for lis contexts and problems:    1) it ignores the lis field’s grounding in a  social sciences and humanities tradition of  scholarship (as opposed to biomedical /  experimental frameworks), which  commonly includes qualitative research  approaches;    2) it privileges particular types of data and  research approaches (namely, those that are  quantitative or experimental in nature) over  other, equally valid research paradigms; and,    3) it focuses the lis field’s attention on the  wrong question by asking “what is the best  evidence?” based on models imported from  other (primarily health‐related) fields;  instead, lis professionals should ask “what  is the best evidence for the problems faced  in library and information science?” in the  context of research models appropriate to  the field.    qualitative research ‐ a central role in  eblip  although many lis problems may indeed  benefit from systematic reviews and other  experimental approaches (such as the use of  pre‐ and post‐testing in information literacy  instruction), there is a great deal of  qualitative evidence in the field that  practitioners cannot and should not ignore.  it is not only inappropriate to measure the  results of qualitative research using the  standards of clinical trials and other  biomedical and experimental models, but  doing so also negates one of the core tenets  of the lis profession – namely, selecting  appropriate, quality information to address  current problems or information gaps. as  lis professionals formalize strategies for  including research evidence in daily practice,  the strengths of our existing body of work  must be addressed. this paper, then, calls  on lis professionals to:    1) embrace those existing qualitative studies  which are rigorous and relevant to current  lis practice;  evidence based library and information practice 2007, 2:1  17 2) contribute to this literature by conducting  and publishing studies using qualitative  methods; and,    3) revise existing models of “best evidence”  in lis to include rigorous qualitative  research.    this paper is intended to guide lis  professionals in these endeavors, and to  serve as an introductory primer for  evaluating and implementing qualitative  research in eblip.    assessing the quality of qualitative  research  qualitative research in lis provides credible  evidence about patrons’ perceptions of  library services, the effectiveness of  information literacy strategies, the design of  web portals, and other service‐related  questions (e.g., saumure & given). other  projects use qualitative textual approaches  (such as discourse analysis) to assess library  policies and organizational structures, the  construction of classification systems, and  other questions that require textual forms of  data (e.g., hedemark, hedman & sundin).  increasingly, lis researchers are also  combining these approaches (i.e., using  multiple qualitative methods), or  completing studies that use both  quantitative and qualitative approaches to  obtain the best possible evidence (e.g.,  whitmire 2004). research in the field of lis  typically draws on the variety of methods  commonly used in social sciences and  humanities disciplines (including education,  sociology, anthropology, and other related  fields), with a strong emphasis on  qualitative and quantitative research  traditions. indeed, there are many research  methods texts that can guide lis  professionals in their assessment of  published qualitative research and in the  design and implementation of qualitative  research projects (e.g., shank).     to conduct research of high quality  (regardless of paradigm) researchers must  use appropriate methods to address the  research problems at hand. to assess patron  satisfaction with an information literacy  instruction session, for example, a  researcher should start by asking:    1. “what do i want to know?”    2. “what method is best to address this  problem?”    qualitative methods, by nature, can address  many of the “why” questions that librarians  and lis researchers have in mind. where  quantitative approaches are best for  addressing what has occurred or how many  events (e.g., counting the number of times a  patron used the library’s website), these  approaches cannot explain why these  behaviours happen. qualitative approaches  do just that; they are used to describe things  about which little is known, especially in  natural settings. they capture meaning (in  the form of individuals’ thoughts, feelings,  behaviours, etc.) instead of numbers, and  describe processes rather than outcomes  (mayan 5‐6). where quantitative methods  (in attempting to maintain objectivity and  reduce bias), strive to eliminate or control  for contextual elements (such as cultural  background or personal history) so they will  not ‘contaminate’ the variables under study,  qualitative researchers embrace these  elements and design studies that will  examine the relevant issues within a broader,  social context.     identifying qualitative studies  in compiling an evidence‐base for lis  practice, then, it is important to understand  that the intended goals of qualitative  research – and the types of data and analysis  used in these projects – are markedly  different from those of quantitative studies.  the data that result from qualitative work  tend to be very rich in scope, providing an  evidence based library and information practice 2007, 2:1  18 in‐depth glimpse of themes and issues. this  is quite different from the results of  quantitative studies, which typically  provide a broader, surface‐level perspective.  a few key features to note:    • hundreds of pages of transcripts  may result from a few in‐depth  interviews or dozens of field  journals may be documented in a  series of ethnographic observations.  the data are very rich in scope and  provide for detailed analysis.    • data collection and analysis may  occur over a period of months, even  within one setting or with only one  small group of participants.    • the sample sizes used in qualitative  research are typically small, due to  the rich data that are gathered and  the depth of analysis required.  researchers must balance financial  and time constraints against the  type of data required for  meaningful, transferable results  given the research questions being  addressed. typically, transferable  data (i.e., where themes can be  applied across a larger population)  will result with sample sizes of 15 to  18 people – but this will vary  depending on the scale and scope of  the research problem.    • as researchers may use multiple  methods to explore one research  problem the length of time to  complete a study may also be much  longer than with other research  approaches; even a small‐scale  qualitative project, using only one  method, can be “labor intensive”  due to the rich nature of qualitative  data (miles & huberman 46).    • results are typically published over  many months (or years) and in a  number of separate journal articles  or in a larger, book form. taken  together, these research reports can  provide a clear and in‐depth picture  of the topics under study.    qualitative methods in lis research  qualitative methods texts describe  numerous data collection strategies that can  be used in various settings and to address  different types of research problems (e.g.,  flick). a few methods that are commonly  used in library settings are detailed here,  with examples from the lis literature.  however, qualitative researchers are also  active in sociology, education, nursing, and  many other fields, and their results can  inform lis practice; these should be  included in an evidence‐base for eblip (e.g.,  education research can inform the design of  information literacy programs).    across all of these disciplines, models of  exemplary qualitative research have  emerged. individuals who wish to learn  more may find journals specializing in  qualitative methods (e.g., international  journal of qualitative methods; qualitative  inquiry; qualitative health research) to be  particularly helpful in providing an  overview of issues and guidelines for  implementing and evaluating qualitative  work.    methods involving human participants  in‐depth, qualitative interviews allow  researchers to examine issues from the  interviewee’s perspective, with a particular  focus on “why” an individual acts a  particular way or makes certain decisions.  qualitative interviews have been used in  many lis contexts to explore various topics  and patron behaviors (e.g., hart, henwood  & wyatt). in usability testing, for example,  qualitative interviews can be combined with  explorations of library websites and  evidence based library and information practice 2007, 2:1  19 databases to assess users’ preferences and  expectations regarding navigation and  information retrieval. a few key points to  note:   • interview data consist of verbatim  responses to an interviewer’s  questions, which are designed to  elicit feelings, attitudes, descriptions  of behaviors, and other elements  relevant to the research problem  (see seidman).    • interviews typically last 60 to 90  minutes, though this will vary  depending on the scope and goals of  the project.    • common themes and patterns  emerge from the data (especially  when more than 15 to 18 people are  interviewed).    • group interviews (often called  “focus groups”) are typically run  with one or more groups of  approximately five to eight people.    • focus groups are more challenging  to manage than individual  interviews, so should be conducted  by trained facilitators (see patton).    • ethics‐related issues will depend on  the nature of the interviews  (individual vs. group), the questions  asked (e.g., sensitive topics?), and  how data will be used (e.g., will  interviewees be identified – or will  pseudonyms be used?).    observing human behavior in libraries  (such as watching patrons as they use  library computers) can elicit insightful data  that cannot be captured using other  qualitative methods. observational methods  have been used in a variety of library  settings, to document the activities of  patrons and staff. in some cases,  ethnographic approaches have been used  (e.g., mckechnie), while other projects use  these methods in conjunction with  interviews, photographs, and other data  collection methods (e.g., leckie & hopkins).  some key points to note:    • in observational work, researchers  may document details about the  individuals in the setting (e.g., age,  gender, or behaviors), as well as  physical resources, layout, or other  elements of the organization under  study.    • observational data are typically  gathered over a long period of  ensure credibility. some qualitative  researchers may be on site for many  months (especially participant  observers, who may work as  librarians while gathering data);  others may gather data in intensive  observational sessions (to capture “a  day in the life” or a “week in the  life” of the library).    • observational methods can be overt  (e.g., where a reference librarian’s  colleagues know that s/he is  gathering data as a “participant  observer” during shifts on the  reference desk) or covert (e.g.,  where patrons are observed,  without their knowledge, while  searching for books in the stacks);  ethical issues will vary with the  types of data collected and analysis  strategies.    textual methods – with participants or  published texts  these approaches can take two forms –  asking individuals to create texts that will  later be analyzed (e.g., patron journals of  library search habits), or analyzing existing  texts (e.g., reviewing published library  policies on internet filtering). asking  evidence based library and information practice 2007, 2:1  20 patrons to document their activities can be  an effective way to examine patron behavior  without relying on individuals’ memories of  past events (e.g., julien & michels 2004).  examining existing texts can point to  debates in the field or policies that may  inadvertently exclude certain patrons (e.g.,  given & julien). some key points to note:    • participants typically need some  instruction about how much detail  to provide, how often, and on what  topics, but journals – whether  written or made using digital  audio/video recorders – can often  produce more detailed responses  than interviews or other methods  will allow, as individuals can take  more time to craft a response.    • journal entries can be combined  with other, complementary methods  (e.g., personal interviews), so that  the researcher can obtain a more  complete picture of the topics under  study.    • the proliferation of publicly  available annual reports, policy  documents, web portals, etc., makes  gathering textual data quick and  easy.    • as with other qualitative methods,  ethics‐related decisions with  participants will depend on the  nature of the investigation (e.g., will  participants be identified?) and the  type of data gathered.    • publicly available materials,  however, may be assessed without  formal ethics review – however,  many researchers choose to  anonymize data to protect  individuals’ identities (e.g., in  analyzing postings to public  listservs).  assessment criteria for qualitative  projects  the criteria used to assess the quality and  rigor of quantitative and qualitative studies  are vastly different, and grounded in each  paradigm’s specific (and often contradictory)  end‐goals. in quantitative research, rigor is  judged in terms of a study’s validity,  reliability, generalizability, and objectivity.  quantitative results are intended to be free  from bias, to be replicable across contexts,  and to generalize from the sample under  study to the full target population (e.g., to  all undergraduate students using canadian  academic libraries, or to all seniors using  public library websites). qualitative  research has its own, separate measures of  quality: credibility, transferability,  dependability, and confirmability (lincoln  & guba 301‐328).    these criteria are no less rigorous than those  used to assess quantitative data; they are  simply different, and require different steps  and measures to ensure quality data. these  steps may include: prolonged engagement  in the field; persistent observation;  triangulation of methods; negative case  analysis; peer debriefing; member checks;  and many other techniques that are often  used together. qualitative results are not  designed to generalize or be replicable as  with the results of quantitative studies.  rather, qualitative results often point to  areas where a single representation of  reality does not hold, or where specific sub‐ populations have particular needs that are  not reflected in broad generalizations (e.g.,  saumure & given’s finding that university  students with visual impairments require  special services in the academic library).    conclusion  as the eblip movement continues to gather  momentum in lis, it would be ideal to put  the controversies to rest regarding the value  of qualitative evidence to support change in  practice. there are a number of steps that  evidence based library and information practice 2007, 2:1  21 professionals can take to ensure that the  research they use – and conduct – is of the  highest caliber:    1. clearly articulate the research problem  and select methods that provide quality  evidence for that problem. methods  texts provide great advice for matching  problems to methods, and tips for  implementation and evaluation of  quality work.    2. design collaborative research projects,  where quantitative and qualitative  researchers – as well as lis scholars and  practitioners – can work together to  build a solid and highly relevant base of  evidence for lis practice.    3. review measures of trustworthiness  (credibility, confirmability,  dependability, and transferability) in  reports of qualitative research – while  recognizing that not all writers will  address these markers, explicitly. do not  presume a lack of rigor in these cases;  rather, examine sample size,  triangulation, etc. in assessing the  quality of the work.    4. revise existing measures of “evidence”  to best incorporate qualitative research  into eblip. rather than relying on  existing evidence‐base models and  hierarchies that are grounded in a  biomedical framework, develop new  measures that will reflect the social  sciences and humanities traditions of  the lis field.    in the health sciences, numerous  publications examine the problems and  limitations of not incorporating qualitative  research into systematic review protocols  and other measures of quality evidence (e.g.,  greenhalgh). it is important that eblip  follows this path and finds ways to include  qualitative research in the evidence‐base for  the lis. as qualitative and quantitative  approaches examine the world from two  very different vantage points, our “best  practice” is one that recognizes and values  both perspectives. existing eblip models  have created a valuable space for discussing  the relevance of research to practice; it is  now vital that those models evolve, to  ensure that all work of high quality will be  part of the eblip discourse.       works cited    flick, uwe. an introduction to qualitative  research. 3rd ed. london: sage, 2006.    given, lisa m. “qualitative research in  evidence‐based practice: a valuable  partnership. library hi‐tech: evidence‐ based librarianship 24.3 (2006): 376‐386.    given, lisa m. “the academic and the  everyday: investigating the overlap in  mature undergraduates’ information‐ seeking behaviors.” library &  information science research 24.1 (2002):  17‐29.    given, lisa m., and heidi julien. “finding  common ground: an analysis of  librarians’ expressed attitudes  towards faculty. the reference  librarian 43.89/90 (2005): 25‐38.     greenhalgh, trisha. “integrating qualitative  research into evidence based  practice.” endocrinology and metabolism  clinics of north america 31 (2002): 583‐ 601.    hart, angie, flis henwood and sally wyatt.  “the role of the internet in patient‐ practitioner relationships: findings  from a qualitative research study.”  journal of medical internet research 6.3  (2004). 30 november 2006  <http://www.jmir.org/2004/3/e36/>.    http://www.jmir.org/2004/3/e36/ evidence based library and information practice 2007, 2:1  22 hedemark, ase, jenny hedman and olof  sundin. “speaking of users: on user  discourses in the field of public  libraries.” information research 10.2  (2005). 30 november 2006  <http://informationr.net/ir/10‐ 2/paper218.html>.    julien, heidi & michels, david. “intra‐ individual information behaviour.”  information processing and management  40.3 (2004): 547‐62.    law, margaret. “the systematic review: a  potential tool for research‐grounded  library management.” proceedings of  the 33rd canadian association for  information science annual conference  (2005). 30 november 2006  <http://www.cais‐ acsi.ca/proceedings/2005/law_2005.pdf >.    leckie, gloria, & jeffrey hopkins. “the  public place of central libraries:  findings from toronto and  vancouver.” library quarterly 72.3  (2002): 326‐372.    lincoln, yvonna s., & guba, egan g.  naturalistic inquiry. newbury park, ca:  sage, 1985.    mayan, maria j. an introduction to  qualitative methods: a training module  for students and professionals. edmonton:  international institute for qualitative  methodology, 2001.    mckechnie, lynne (e.f.). “ethnographic  observation of preschool children.”  library & information science research  22.1 (2000): 61‐76.    miles, matthew b., & a. michael huberman.  qualitative data analysis: an expanded  sourcebook. thousand oaks, ca: sage,  1994.  morse, janice m. “beyond the clinical trial:  expanding criteria for evidence  [editorial].” qualitative health research  15.1 (2005): 3‐4.    palys, ted. research decisions: quantitative  and qualitative perspectives, 2nd ed.  toronto: harcourt brace, 1997.    patton, michael. qualitative evaluation and  research methods, 2nd ed. newbury  park, ca: sage, 1998.    saumure, kristie, and lisa m. given.  “digitally enhanced? an examination  of the information behaviours of  visually impaired postsecondary  students.” canadian journal of  information and library science 28.2  (2004): 25‐42.    seidman, irving. interviewing as qualitative  research:  a guide for researchers in  education and social sciences, 2nd ed.  new york: teachers college press,  1998.    shank, gary. qualitative research: a personal  skills approach. new jersey: prentice‐ hall, 2002.    whitmire, ethelene. “the relationship  between undergraduates’  epistemological beliefs, reflective  judgment, and their information‐ seeking behavior.” information  processing and management 40.1 (2004):  97‐111.  http://informationr.net/ir/10%e2%80%90 http://www.cais%e2%80%90acsi%00 commentary   the role of student advisory boards in assessment   ameet doshi director, service experience and program design georgia institute of technology atlanta, georgia, united states of america email: ameet.doshi@library.gatech.edu   meg scharf associate director, communication, assessment, and public relations university of central florida orlando, florida, united states of america email: meg@ucf.edu   robert fox dean, university libraries university of louisville louisville, kentucky, united states of america email: bob.fox@louisville.edu   received: 15 feb. 2016   accepted: 18 feb. 2016        2016 doshi, scharf, and fox. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – the objective for this commentary article is to assess and communicate the development, logistics, and overall value of student advisory boards for the libraries at three large research institutions. methods – the methods for developing and operating an advisory board vary between schools; however they share common approaches that could be viewed as "best practices" for sustainable and productive student advisory boards. results – our commentary aims to inspire libraries to invest in this value-added approach as part of a robust portfolio of assessment tools. the various practices outlined in the commentary could be helpful to librarians who seek to begin or further develop a student advisory board. conclusion – the unique relationship fostered by the advisory board enables libraries to use direct student feedback to confirm what is learned from surveys, focus groups, and observations. a strategic relationship with a student board can enable librarians to refine methods of obtaining information, or it can cause us to view information we have collected in a different way.     introduction   a student advisory board can be a unique addition to an academic library’s assessment program. boards give libraries valuable feedback. this paper addresses benefits of library student advisory boards, as well as procedures for maintaining successful boards to enhance or enrich information for assessment.   setting   practices for board management at georgia institute of technology, university of central florida (ucf), and the university of louisville have developed active, engaged advisory board members. members have contributed to outcomes such as service improvements, successful space redesign, and improved alignment between library strategic objectives and user needs. best practices identified by these institutions are applicable to many other contexts.   all three institutions have received valuable assessment information from student advisory boards. the input has triggered additional evaluation and assessment of programs resulting in improvements for services or resources in areas that needed addressing and focusing on campus life trends that affect library use. evidence demonstrates that the return on investment in student advisory boards has been a net positive for each institution.   problem    although research libraries have a relatively robust tradition of assessing and responding to student needs via survey tools such as libqual, concerted efforts to develop a long-standing, strategic relationship with students is a newer phenomenon (thompson, cook & heath, 2000). a student advisory board can help fill this void.  our experience suggests that student advisory boards give academic libraries feedback and information on user perceptions and experiences, strategic guidance, and suggestions/ideas for new or improved services, facilities, and resources. each of these three institutions has experienced the benefits of receiving information from a group of committed student board members.   the idea of advisory boards originated with corporations and non-profit organizations.  consumer panels or boards can react to products and services, as well as to the overall experience of shopping with or dealing with the organization. non-profits can get advice on the community environment, and get help with advocacy from board members. (carter, geehan, ross, 2005)   there are other groups that can give feedback to the library but are outside this paper's scope. many academic libraries have a group appointed by the provost or faculty senate to provide oversight on strategic directions and to be part of institutional governance. fundraising and friends’ groups can help the library tell its story to influential people. while the library can receive feedback from these groups, none of them is designed to give the kind of direct reports that users of the facilities, resources and services can.   evidence   the decision to form an advisory board can occur due to a variety of reasons. at georgia tech, an ad hoc group of honors students reached out with informal feedback to ensure that the students voice helped inform a library renovation project. the input that these students provided was valuable enough to encourage library administration to continue the board more formally. the dean at louisville had formerly worked with the student board while at georgia tech. based on the positive results there, he sought to recreate that type of experience after moving to louisville.   at ucf, the board was established with the broader goal of ensuring that library administrators had a direct line to student opinion and feedback when it was needed to evaluate services, resources and facilities. surveys, including libqual+, interviews, newsletters, blogs, suggestion boxes, mystery shoppers, and more have all generated feedback and facilitated communication with students. the board serves as a safety net to help answer the questions, “is the library’s story reaching students?  does the library know what students really think?” the development of this trusted channel for open communication has paid off in helping to keep the library’s planned building improvements on the university’s list of priorities despite funding challenges. the board’s role has been essential in smaller matters, such as board members reaching out to the student press to publicize new services, and confirming that students read and value the information disseminated in the "bathroom newsletter".   implementation   library decisions on the structure and operation of the board can determine the types of feedback that the board delivers. the purpose of its board, campus culture, tradition, and practices can dictate the formality of advisory board establishment, member recruitment, and composition. while different at each institution, these are vital pieces of the long-term success of these boards. boards present library leadership with the opportunity for direct interaction with an important group of stakeholders, but student members should not feel outnumbered by administrators at meetings. frequency of meetings and the meeting agenda are also crucial in sustaining member enthusiasm and in generating informative discussion.   one characteristic that makes student advisory boards unique is synergy. when the library surveys or interviews students, it receives individual feedback that is not influenced by the presence of others. even focus groups are individuals brought together on that occasion for that specific purpose. in contrast, student advisory board members become acquainted with each other over a period of time and develop an “esprit de corps.”  while they are individuals, they have common experiences as students of the same institution and users of the same library. in the best of circumstances, they can become a team that supports the library. demonstrating to the student board that they are important can help create the unique synergy that, combined with their being more comfortable with library administrators, allows for a more honest, open and caring conversation about the library.   to create an environment for this kind of feedback, examine the campus culture and determine how you will appoint members. receiving an appointment, and being able to put that on a resume, is important to students. student government and student organizations can help recruit or be solicited for top students as board members. faculty, administrators, and librarians can suggest student members. at georgia tech, particular attention is paid to the composition of the board because 70% of the student body are engineering majors. additionally, a unique relationship between the library and the institute’s top scholarship program ensures that high achieving, motivated students are appointed to the board. careful and conscientious recruitment is a key factor in ensuring a sustainable and active board. at ucf, students who contact the library with constructive criticism are recruited, often becoming the most enthusiastic board members.   logistical considerations for sustainable student boards are crucial. there should be some regularity regarding meeting frequency. among the institutions discussed here, three meetings per semester is common. it is also vital to distribute minutes and decisions from the previous meetings with the agenda for the next meeting. many board meetings occur during a lunch or dinner hour and a best practice is to provide a meal. finally, if available, meetings should be held in special or exclusive spaces such as an administrative boardroom.    results   examples of the role of board-gathered information and activities in enhancing the assessment process are included below. at our institutions, a key outcome for successful assessment initiatives has been that it closes the loop by soliciting information from users, analyzing it, acting upon it, and communicating the change back to users. the impact is felt by students who recognize that the library values their input. active student advisory boards produce comments and feedback, and with library decision-makers present, changes are made and reported back quickly. for example, at ucf, a board member complained about viewing the local news station in the coffee area, claiming it was nothing better than a crime blotter---a litany of robberies, car-jackings, and the like. the student initially suggested the bbc, but the board discussion led to displaying a national news station. favorable student commentary followed the change: the loop was closed. it demonstrated to the board that their opinions are important even on seemingly small matters.   at the university of louisville, the board has proven very useful in physical improvement projects and web usability studies. members serve as a sounding board for feedback on facilities changes. they have also been part of interactive design charrettes for planning construction.   at georgia tech, student board members have contributed in myriad ways from service design improvements, to being at the table with designers and architects putting comprehensive renovation plans together.   library administration can provide an overview on facilities and services, and student advocates can take it upon themselves to present that information to campus administrators. at georgia tech, student board members wrote a letter asking the institute’s administration to address the condition of the library building. however, if advocacy is not handled correctly it can become an ethical issue. for example, the library should not expressly ask board members to write a letter on its behalf. instead, advocacy should arise naturally as students become better informed about issues facing the library.   at ucf, advisory board members have tested survey questions and related assessment tools. reviewing survey results with students can generate further discussion and information-gathering, and can help translate survey commentary with more context. board members can make valuable suggestions to overcome the challenges of publicizing results and increasing participation in assessment efforts. they have also provided invaluable information for planning building projects. perhaps the most gratifying experience that each institution has had with its board is witnessing the conversion of board members to active advocates for the library. advocates help close the assessment loop by supporting the programs and enhancements suggested by assessment results, through invaluable word-of-mouth publicity and through support of funding initiatives that enable these improvements.   boards are not a shortcut to obtaining qualitative assessment information. if done well, everything takes time: recruiting the right students, preparing agendas, distributing minutes. student advisory boards are only one tool in an assessment portfolio.   conclusion   our students can inspire us. the unique relationship fostered by the advisory board environment enables libraries to use direct student feedback to confirm what we have learned from surveys, focus groups, and observations. a strategic relationship with a student board can enable us to refine our methods of obtaining information, or it can cause us to view information we have collected in a different way.   although managing a library student advisory board is time-consuming, it can be part of an effective and sustainable assessment plan. more importantly, the outcomes of this investment in leveraging student wisdom can pay dividends in a multitude of ways.   references   carter, t. (2003). customer advisory boards: a strategic tool for customer relationship building. new york: best business books.   geehan, s., & sheldon, s. (2005). connecting to customers. marketing management, 14(6), 36-42.   ross, j. a. (1997). why not a customer advisory board? harvard business review, 75(1), 12.   thompson, b., cook, c., & heath, f. (2000). the libqual+ gap measurement model: the bad, the ugly, and the good of gap measurement. performance measurement and metrics, 1(3), 165-178.       evidence summary   libraries support first-generation students through services and spaces, but can do more   a review of: arch, x., & gilman, i. (2019). first principles: designing services for first-generation students. college & research libraries, 80(7), 996–1012. https://doi.org/10.5860/crl.80.7.996   reviewed by: hilary bussell librarian for political science, economics, & communication the ohio state university libraries columbus, ohio, united states of america email: bussell.21@osu.edu   received: 3 dec. 2019                                                                    accepted:  17 jan. 2020      2020 bussell. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29689     abstract   objective – to provide recommendations for academic libraries in supporting first-generation college students, defined by the study’s authors “either as those whose parents have no college experience or did not graduate with a bachelor’s degree” (p. 996).   design – multidisciplinary literature review and two qualitative survey questionnaires.   setting – united states secondary and post-secondary schools.   subjects – college counsellor survey: 300 private/parochial, private/secular, public, and charter secondary/high schools in rural, suburban, and urban areas randomly selected from across all 50 states. academic library survey: academic libraries at 100 four-year institutions.   methods – college counsellor survey: an online survey was sent to potential subjects via email. the survey included basic demographic questions and open-ended questions about counsellors’ perceptions of the challenges faced by first-generation college students, the types of institutional support they need, and how academic libraries could offer support. the answers were analyzed in nvivo using a combination of inductive and axial coding in order to develop a set of themes reflecting the most commonly-mentioned challenges and support needs.   academic library survey: a survey invitation was sent directly to library deans and directors at 100 four-year institutions and to two academic library electronic mailing lists. the survey included questions about services academic libraries offer for first-generation students. the data were analyzed to determine whether academic libraries were offering services that matched the thematic areas identified in the college counsellor survey.   main results – college counselor survey: the response rate was 24% (n=78). there were six themes which were identified regarding challenges and support needs for first-generation students: academic and tutoring, social and cultural, home and family, navigating college, financial, and mentoring and advising. recommendations for library support included library participation in first-generation student orientations, library instruction in research methods, and conveying that librarians are welcoming and friendly.   academic library survey: out of the 59 responding libraries, 19 reported offering services designed for first-generation students, while 6 additional libraries reported offering services they believed would be useful for first-generation students, though they were not designed for them specifically. the library services addressed needs relating to the academic and tutoring, navigating college, and financial themes. the library services did not address the needs relating to the social and cultural, home and family, and mentoring and advising themes.   conclusion – based on the surveys and literature review, the authors provide recommendations for academic library services for first-generation students that address each of the six thematic areas of need. the recommendations include, but are not limited to, assigning liaisons for first-generation programs, working to create an inclusive and affirming library environment, designing library spaces that support the multiple social roles and identities of first-generation students, helping family members of first-generation students become aware of the library as a resource, providing training on applying for financial aid, and having librarians either provide outreach to mentorship programs or become mentors themselves.   commentary   research on first-generation college students has increased in library and information science over the past decade, as it has in higher education research as a whole (ilett, 2019). as the authors note, much of the literature on first-generation students relies on the deficit model, which ascribes achievement gaps in education to a problem on the part of individual students, rather than a failure of the educational system that is not set up to help all students succeed (ibid). this study contributes to a shift away from deficit thinking, towards an approach in which libraries and other educational institutions focus on how they can design services, spaces, and instruction that are more inclusive of all students.   this study was assessed using the cristal checklist for appraising an information needs analysis. the findings and recommendations for libraries are extensive and are based on triangulated data gathered through two surveys carried out with different populations and a multidisciplinary literature review. a number of examples and quotations are provided to illustrate the identified themes. the authors acknowledge as a limitation of the study the fact that their data are not based on input from first-generation students. in their next stage of research, the authors intend to gather data directly from first-generation students in order to test the validity of their recommendations and to explore their experiences with academic libraries.   the design and implementation of the two surveys are clearly described, and the instruments are provided as appendices to the article. the authors describe the inductive and axial coding approaches used to categorize survey responses. however, it is unclear whether the coding was done by one or both researchers. if the coding was done by both researchers, it would be helpful to know what steps were taken to ensure consistency between coders.   in addition to the surveys, the authors also base their recommendations for library services on a multidisciplinary literature review. one area in which the evidence presented could be stronger is if the authors had described how the multidisciplinary literature review was conducted (e.g.  information sources consulted and search terms used).   this study is significant in several respects. it sheds light on the needs of first-generation students to succeed in college by drawing on the expertise of college counsellors working in secondary schools. further, by using a multi-survey approach, it ties these findings to what academic libraries are doing to support first-generation students and makes concrete, detailed recommendations for ways that libraries could expand their support in areas such as mentoring and financial literacy. the reader is strongly encouraged to consult the original study for these recommendations. the study will be particularly useful in helping library practitioners support first-generation students in areas beyond library instruction, such as services, programming, and space design.                 references   cristal checklist for appraising an information needs analysis. (n.d.). nettingtheevidence.pbwiki.com. retrieved 22 november 2019 from http://nettingtheevidence.pbwiki.com/f/needs.doc   ilett, d. (2019). a critical review of lis literature on first-generation students. portal: libraries and the academy, 19(1), 177–196. https://doi.org/10.1353/pla.2019.0009 evidence based library and information practice editorial responsibilities      2020. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     editor-in-chief: lorie kloda   associate editor (evidence summaries): stephanie krueger, christina wissinger   associate editor (research articles): ann medaille, lisl zach   associate editor (classics & reviews): jane schmidt   associate editor (commentaries, using evidence in practice): lorie kloda   communications officer (news): tatiana bryant   production editor: rachel hinrichs   editorial intern: kimberly mackenzie   editorial advisors: lindsay alcock, alison brettle, michelle dunaway, denise koufogiannakis, heather pretty   copyeditors: heather healy (lead copyeditor), georgianne bordner, julie evener, emily kingsland, alison moore, jane morgan-daniel, stacey l. penney, elizabeth stregger, nikki tummon, elaina vitale   indexing support: kate shore   microsoft word es 4480 evidence based library and information practice 2009, 4:1 36 evidence based library and information practice evidence summary thematic categorization and analysis of peer reviewed articles in the lisa database, 2004-2005 a review of: gonzalez-alcaide, gregorio, lourdes castello-cogolles, carolina navarro-molina, et al. “library and information science research areas: analysis of journal articles in lisa.” journal of the american society for information science and technology 59.1 (2008): 150-4. reviewed by: carol perryman trln doctoral fellow, school of information & library science, university of north carolina at chapel hill chapel hill, north carolina, united states of america e-mail: cp1757@gmail.com received: 05 december 2008 accepted: 10 february 2009 © 2009. perryman. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. objective – to provide an updated categorization of library and information science (lis) publications and to identify trends in lis research. design – bibliometric study. setting – the library and information science abstracts (lisa) database via the csa illumina interface. subjects – 11,273 item records published from 2004-2005 and indexed in lisa. methods – first, a search was set up to retrieve all records from 2004-2005, limited to peer review items (called “arbitrated works” by the authors (150)) and excluding book reviews. second, thematic descriptor terms used for the records were identified. frequency counts for descriptor term occurrence were compiled using microsoft access and pajek software programs. from the results of this search, the top terms were analyzed using the kamada-kawai algorithm in order to eliminate descriptor term co-occurrence frequencies under 30. a cluster analysis was used to depict thematic foci for the remaining records, providing a co-word network that visually identified topic areas of most frequent publication. conclusions were drawn from these evidence based library and information practice 2009, 4:1 37 findings, and recommendations for further research were provided. main results – the authors identified 18 “thematic research core fields” (152) clustered around three large categories, “world wide web”, “education”, and “libraries”, plus 12 additional peripheral categories, and provided a schematic of field interrelationships. conclusion – domains of greatest focus for research “continue to be of practical and applied nature,” (153) but include increased emphasis on the world wide web and communications technologies, as well as on user studies. a table of the most frequently occurring areas of research along with their top three descriptor terms is provided (table 1, 152) (e.g., “world wide web” as the top area of research, with “online information retrieval” (268 occurrences), “searching” (132 occurrences), and “web sites” (115 occurrences)). commentary in this brief research report, the researchers used co-occurrence and cluster analysis methods to find emergent foci for research in the lis literature. the authors appear to have begun their research by making the assumption that all the articles indexed in lisa concern lis literature, a concept quickly disproved. among the article titles that comprise the dataset, this reviewer found the following: schoech, d., j.d. fluke, r. basham, d.j. baumann, and g. cochran. “visualizing multilevel agency data using olap technology: an illustration and lessons learned.” journal of technology in human services 22.4 (2004): 93-111. engin, m., o. cidam, and e.z. engin. “wavelet transformation based watermarking technique for human electrocardiogram (ecg).” journal of medical systems 29.6 (2005): 589-594. isik, h. “design and construction of thermoelectric footwear heating system for illness feet.” journal of medical systems 29.6 (2005): 627631. while it is possible that the authors restricted their analysis to lis areas of concern, it was not made explicit whether this was done. the query used in retrieval is documented as py (publication year) = 2004 or py = 2005, restricted to peer reviewed works (called “arbitrated works” (150)), and only book summaries (book reviews) were excluded by design. this reviewer cannot assess the percentage of materials that are not obviously related to lis research, but it seems likely that the findings are affected. additionally, although the authors describe their frequency analysis of ‘descriptors,’ in the connected table (table 1,151), the terms are called ‘keywords’ (“choose the keyword option to search the title, abstract, and descriptors simultaneously”)1. in the lisa database, key words are terms found in the title, abstract, descriptor, or identifier fields, while ‘descriptors’ are limited to controlled vocabularies (in this case the lisa thesaurus). 1 lisa csa illumina help and support: keyword search: descriptors, de = this field contains indexing terms taken from a thesaurus or controlled vocabulary. keyword, kw = this search strategy simultaneously searches the title (ti), abstract (ab), descriptor (de), and identifier (id) fields. [csa illumina help & support: fielded search evidence based library and information practice 2009, 4:1 38 in order to validate co-occurrence analysis, it would be helpful to consider how descriptor terms were assigned, in order to determine whether the terms are indicative of the research focus for the articles indexed. for example, a descriptor term such as ‘metaanalysis’ might equally be used to discuss the process for meta-analysis or to describe the research method used to conduct the indexed paper. here, examination of a random sample of articles would have improved the quality of this paper. there is also no description of the methods used to assign thematic areas, which should include a definition of the area as well as a description of the exclusion and inclusion criteria. the ability to replicate this search is made more problematic by the use of thematic areas assigned without explanation. although the authors’ uses of cluster analysis and co-occurrence frequency appear to be appropriate methods for examining a corpus of literature, the results are questionable due to the absence of consideration for database content coverage in the lisa database. future research using these replicable methods and with a more carefully selected initial dataset would constitute a substantial contribution to our understanding of lis research areas. this work was published as a brief communication, so severe space constraints likely barred inclusion of further discussion or detail. however, the authors failed to provide a literature review, only mentioning other research in discussing the findings from their analyses. some mention of the rational for methodologies used, especially in support of term occurrence as a justification for topicality (for example, zipf’s law of inverse proportion of term usage to topicality), as well as of the numerous previous attempts to characterize lis output and to evaluate the lisa database would have better informed their approach to this important topic. critical evaluation of this brief research report was done using the university of salford school of nursing hcprdu evaluation tool for quantitative analysis. works cited university of salford school of nursing. “evaluation tool for qualitative studies.” hcprdu. university of salford. 30 november 2008. <http://www.fhsc.salford.ac.uk/ hcprdu/tools/quantitative.htm>. zipf, george k. human behavior and the principle of least-effort. cambridge, ma: addison-wesley, 1949. ebl 101   research methods: action research   virginia wilson director, centre for evidence based library and information practice (c-eblip) university library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   originally published in: evidence based library and information practice, 8(4), 160–162. https://ejournals.library.ualberta.ca/index.php/eblip/article/view/20940/15998     received: 31 oct. 2013   accepted: 15 nov. 2013      2016 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   action research, also known as participatory action research, is a type of enquiry that emerged from the field of education and is primarily used by teachers. various other disciplines also use action research, such as public health and nursing (hannigan, 2008). however, any practitioner in a social practice who conducts research could find action research a useful tool. action research is undertaken at the local level and often involves the participation of those who might otherwise be considered subjects of the research. it’s a practical type of research designed to inform local issues of a practical nature. so, the teachers are looking at questions from their classrooms and librarians as researchers are looking at issues arising from their library. action research’s focal point is a problem to be solved and it is undertaken by practitioners.   a definition of action research:  “action research is the study of a social situation carried out by those involved in that situation in order to improve both their practice and the quality of their understanding” (munn-giddings & winter, 2002, p. 8). the notion of “social situation” here is a broad one, and while it may suggest mingling at a cocktail party, it should be taken to mean that any social interaction that a practitioner or a practitioner’s organization has with people (users, clients, students, patients, etc.) is a candidate for action research.   action research is a process that helps practitioners develop a better understanding about the particulars of a specific practice-based situation. it has a personal and a social aim: there is ideally improvement in the practitioner’s learning and an improvement in the situation that is being investigated. these two aims are interdependent. (mcniff, lomax, & whitehead, 2003).   in terms of the steps one takes for an action research project, there are a variety of ways in which the process is described. some say there are three steps although each of those steps has several sub-steps. sometimes the steps are referred to as phases with a variety of numbers there. i’m going to list five basic steps in action research with the caveat that this process is an iterative, cyclical, and reflexive process rather than a linear one.   action research steps   identify – problem, question, baseline data, and professional development plan – the intervention, assessment, and time to assess collect data analyze data reflect – what was learned, what will be improved, and making improvements (teachers’ pd inc, 2012-13)   critics of action research are bothered by the notion that the process involves the local level and input from stakeholders, claiming that these are case studies, or the “how i done it good in my library” study, and are most likely not generalizable to other situations or institutions (beck & manuel, 2008, p. 196). beck and manuel (2008) state that “while not claiming that its results are completely representative or generalizable, action research does include a number of methodological steps to ensure that it is rigorous in obtaining its results, and these steps help to ensure that results are at least somewhat representative or generalizable” (p. 196). as with any other methodology, care should be taken to ensure that it is the best methodology to deal with a particular research question.   action research has emerged in the library literature around the topic of information literacy and elsewhere. here are some examples.   greenan, e. (2002). walking the talk: a collaborative collection development project. school libraries in canada, 21(4), 12-14.   kendall, m. (2005). tackling student referencing errors through an online tutorial. aslib proceedings, 57(2), 131-145.   rious, k. (2013). teaching social justice in an information literacy course: an action research case study. catholic library world, 83(3), 191-195.   vezzosi, m. (2006). information literacy and action research: an overview and some reflections. new library world, 107(7-8), 286-301.   wilson, t.d. (2000). recent trends in user studies: action research and qualitative methods. information research 5(3). retrieved 31 oct. 2013 from http://informationr.net/ir/5-3/paper76.html   there are lots of books and articles available that describe the process of action research from various perspectives and in varying levels of detail. here are some examples:   chevalier, j.m. & buckles, d.j. (2013). participatory action research: theory and methods for engaged inquiry. new york : routledge.   cook, d. & farmer, l. (eds.). (2011). using qualitative methods in action research: how librarians can get to the why of data. chicago, il: association of college and research libraries.   howard, j.k. & eckhardt, s.a. (2005). action research: a guide for library media specialists. worthington, oh: linworth.   mcniff, j.  & whitehead, j. (2011). all you need to know about action research (2nd ed). los angeles, ca: sage.   mills, g.e. (2014). action research: a guide for the teacher researcher (5th ed). boston : pearson.   pavlish, c.p. & pharris, m.d. (2012). community-based collaborative action research: a nursing approach. sudbury, ma : jones & bartlett learning.   for a research topic with very practical, practice-based origins, action research may be the best way to explore it and obtain useful, reflective answers.   references   beck, s.e. & mauel, k. (2008) practical research methods for librarians and information professionals. new york: neal-schuman.   hannigan, g.g. (2008). action research: methods that make sense. medical reference services quarterly 16(1), 53-58. doi: 10.1300/j115v16n01_06   mcniff, j., lomax, p., & whitehead, j. (2003). you and your action research project. taylor & francis. retrieved 31 oct. 2013 from <http://www.myilibrary.com?id=3669>   winter, r. & munn-giddings, c. (2002). a handbook for action research in health and social care. routledge. retrieved 31 oct. 2013 from <http://www.myilibrary.com?id=6016>   teachers’ pd inc. (2012-13). action research model. retrieved 31 oct. 2013 from http://www.csuchico.edu/teachergrants/actionresearch/ar_model.shtml   evidence summary   students think science literacy is important and improves with lifelong learning   a review of: holden, i. i. (2010) science literacy and lifelong learning in the classroom: a measure of attitudes among university students. journal of library administration, 50:3, 265-282.  https://doi.org/10.1080/01930821003635002   reviewed by: heather macdonald health and biosciences librarian macodrum library carleton university ottawa, ontario, canada email: heather.macdonald@carleton.ca   received: 16 feb. 2021                                                              accepted:  22 mar. 2021      2021 macdonald. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29926     abstract   objective – investigate student attitudes to science literacy and lifelong learning as outlined in standard five of the information literacy standards for science and engineering/technology (ilste): the information literate student understands that information literacy is an ongoing process and an important component of lifelong learning and recognizes the need to keep current regarding new developments in his or her field (2006).   design – survey.   setting – a large public american r1 university.   subjects – undergraduate students in two classes: information literacy in the sciences (science group) and information literacy (non-science group).   methods – a 13 question survey was administered to students by a colleague of the author.  three categories of questions were asked: students' perceptions of the important of science literacy, students' assessment of their own science literacy skills and students' perceptions of lifelong learning in relation to standard five.   main results – survey participation included 13 students from the science group and 18 students from the non-science group. students agreed that science literacy is an important part of civic literacy and responsibility, and should be taught to undergraduates in the united states.  students from the science group frequently had more positive responses to statements than the students in the non-science group.  for example, 81% of science students either agreed or strongly agreed with the statement, “every responsible citizen should be aware of the latest scientific discoveries” compared to 61% of non-science students.  students felt that their science literacy skills had improved since high school.  most students were confident in their research skills including using emerging communication technologies. students believed that life-long learning and staying current contribute to good information literacy and science literacy.   conclusion – students think science literacy is an important part of being a responsible citizen. they also believe being a life-long learner improves science literacy.   commentary   this research article was written in 2010.  since that time the acrl framework for information literacy in higher education (acrl framework, 2016) was established.  there has been an explosion in new and emerging communication application technologies.  in addition, a considerable body of literature has been published on science literacy in the intervening years.   this commentary uses the cat critical appraisal tool (perryman & rathbun-grubb, 2014). the author, as an information literacy instructor, has demonstrated expertise in the field.  the objective and the rationale for the study were clearly outlined and the literature review provided definitions and background for science literacy, lifelong learning, and standard five of the ilste (2006).  the author stated that it was mostly science librarians who were bringing attention to science literacy.  in the intervening years, education researchers and scientists in various disciplines have also published on the topic of science literacy as evidenced by a quick search in the scopus database.  this demonstrates an ongoing interest in the topic.     overall this was a very well conducted study.  to address the research question, the author chose a survey design.  she received ethics approval and had a colleague administer the survey and send letters of consent to study participants, and only received the responses post grading.  the survey questions were provided in the results section of the article.    in the results, the author provided clear graphic, tabular, and narrative summaries for the various questions. figure 5 compared student literacy before college and at present. it would have been interesting to see the population broken down by science and non-science students, similar to the first four figures.  question 7 asked, “studies at the university have helped me to increase my level of science literacy,” with potential answers ranging from strongly disagree to strongly agree. although this question confirmed that students’ self-perceived science literacy improved since being at university it did not specifically ask about the impact of the il course.  a question getting at the impact of the il session itself could have been useful. overall the survey questions and the results helped provide a clear picture of students’ views. the author also addressed the study’s limitations acknowledging the small sample size and the reliance on self-assessment.    the author did an excellent job of making the connection between lifelong learning and science literacy. it is clear from the survey results that the students also saw this connection. in the conclusion the author asks why the general standards for information literacy do not include the same standard.  lifelong learning is now incorporated in the acrl framework.   the framework recognizes lifelong learning through the ongoing development of skills and knowledge as learners progress from novice to expert.  the author was prescient in asking the question in 2010.  this study remains relevant today.  recent articles (reed, hiles & tipton, 2019; sharon & baram-tsabari, 2020; vraga, tully & bode, 2020) propose better information and science literacy skills can help combat misinformation.  for science literacy instructors, these studies point to the need to make explicit the connection between lifelong learning, information literacy, and science literacy to help prepare their students to engage with the world.  this study also provides an excellent example of how to conduct research with student participants.    references          american library association, association of college and research libraries, and science and technology section’s task force on information literacy for science and technology. (2006). information literacy competency standards for science and engineering/technology. american library association. http://www.ala.org/ala/mgrps/divs/acrl/standards/infolitscitech.cfm   association of college and research libraries. (2016). framework for information literacy for higher education. http://www.ala.org/acrl/standards/ilframework   perryman, c. & rathbun-grubb, s. (2014). the cat: a generic critical appraisal tool. in jotform – form builder. http://www.jotform.us/cp1757/thecat   reed, k., hiles, s. s., & tipton, p. (2019). sense and nonsense: teaching journalism and science students to be advocates for science and information literacy. journalism and mass communication educator, 74(2), 212–226. https://doi.org/10.1177/1077695819834415   sharon, a. j., & baram-tsabari, a. (2020). can science literacy help individuals identify misinformation in everyday life? science education, 104(5), 873–894. https://doi.org/10.1002/sce.21581   vraga, e. k., tully, m., & bode, l. (2020). empowering users to respond to misinformation about covid-19. media and communication, 8(2), 475–479. https://doi.org/10.17645/mac.v8i2.3200     evidence summary   for non-expert clinical searches, google scholar results are older with higher impact while pubmed results offer more breadth   a review of: nourbakhsh, e. f., nugent, r. f., wang, h. f., cevik, c. f., & nugent, k. (2012). medical literature searches: a comparison of pubmed and google scholar. health information and libraries journal, 29(3), 214-222. doi: 10.1111/j.1471-1842.2012.00992.x   reviewed by: carol perryman assistant professor texas woman’s university denton, texas, united states of america email: cp1757@gmail.com   received: 25 nov. 2012 accepted: 22 jan. 2013      2013 perryman. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objectives – to compare pubmed and google scholar results for content relevance and article quality   design – bibliometric study.   setting – department of internal medicine at texas tech university health sciences center.   methods – four clinical searches were conducted in both pubmed and google scholar. search methods were described as “real world” (p. 216) behaviour, with the searchers familiar with content, though not expert at retrieval techniques. the first 20 results from each search were evaluated for relevance to the initial question, as well as for quality.   relevance was determined based on one author’s subjective assessment of information in the title and abstract, when available, and then tested by two other authors, with discrepancies discussed and resolved. items were assigned to one of three categories: relevant, possibly relevant, and not relevant to the question, with reviewer agreement measured using a weighted kappa statistic. the quality of items found to be ‘relevant’ and ‘possibly relevant’ was measured by impact factor ratings from thomsen reuters (isi) web of knowledge, when available, as well as information obtained by scopus on the number of times items were cited.   main results – google scholar results were judged to be more relevant and of higher quality than results obtained from pubmed. google scholar results are also older on average, while pubmed retrieved items from a larger number of unique journals.   conclusion – in agreement with earlier research, the authors recommended that searchers use both pubmed and google scholar to improve on the quality and relevance of results. searches in the two resources identify unique items based upon the ranking algorithms involved.     commentary   comparisons and tests of the utility, quality, and relevance of searching google scholar and pubmed for clinical questions in previous research (e.g., mastrangelo et al., 2010) have found that google scholar is a valuable adjunct to pubmed searching that may be easier for the non-expert searcher (shultz, 2007). as well, findings have shown that google scholar-retrieved items tend to be older and less specific due to filters and terminology affordances not provided in google scholar (anders & evans, 2010). this too is confirmed by the present study, as each resource examined contains unique materials not indexed by the other, including the gray literature accessible via google scholar (shultz, 2007). comparative measures of quality have included ranking position in results lists, presence of terms and related terms in abstracts and titles (tober, 2011), and measures of sensitivity and precision (anders & evans, 2010). using retrieval rankings to compare pubmed with google scholar is questionable at best, as search algorithms and objectives are quite different. the authors compared only the first twenty results from google scholar to those in pubmed, yet these resources rank results very differently. google scholar also indexes and retrieves items from a very broad spectrum of disciplines, while pubmed coverage, though still broad, is limited to biomedical publications. the relevance of retrieved items is assessed only through subjective examination of item titles and abstracts (though not all items had abstracts), and no further information, including the titles of items found relevant, were included.   the authors based their quality assessments on information from scopus about the citedness (tober, 2011) and overlap of results from related cochrane reviews (anders & evans, 2010). however, and without explanation, the authors have chosen to use web of knowledge for impact factor information rather than scopus, even though these databases do not provide the same publication coverage. this aspect of evaluation would have been improved by using just one of these resources. in addition, recognized problems with the use of impact factors and citation metrics to impute quality are not discussed as they relate to the present study.   this present study offers little new information to this still relatively sparse corpus. the authors conducted searches using a ‘real world’ level of search expertise, which is a departure from previous efforts, and of some value in that clinicians are known to employ a limited number of search terms and to examine only the first page of results. however, a lack of rigor and transparency in this study mars potential applicability.   from an initial set of four clinical questions, authors employed search strings in pubmed and google scholar, using different limiters in two of the four searches for both databases. the limiters for q1 and q2 in pubmed are reports, clinical series, and reviews (q2), but in google scholar, only a single limiter, randomized controlled trials, was used for q1. in both instances for google scholar searches, the authors set search limits to english language and to the then-available disciplinary set of medicine, pharmacology, and veterinary sciences (at the time of this review, disciplinary set limiters were no longer available in google scholar). as the authors limited their relevance and quality assessments to the first twenty results retrieved, the different search strings and filters may have radically altered findings, retrieval rankings, and evaluations of quality.   discrepancies between the initial question and the search strings used in the two search engines are not clarified or explained. while q2 and q4 each include a facet about outcomes, the search strings listed include no mention of this concept. the result is that readers cannot discern whether assessments of relevance were based on the complete initial questions using the information provided. for assessments of quality, the authors used web of knowledge to check journal title impact factor rankings, paired with statistics about how often the retrieved items were cited in ensuing literature. several problems are apparent in the use of this methodology. first, the authors state that not all journal titles for retrieved items are indexed in web of knowledge, while unlisted titles are not provided. second, while google scholar also retrieved non-article items, these are not likely to be indexed in web of knowledge. in both cases, this gap has undoubtedly affected quality assessments. the lack of data for impact factors or citedness is not addressed except as a brief footnote (p. 218). moreover, citedness has been disputed as a measure of quality, but the authors do not address this. while the authors employed solid and appropriate descriptive statistics to describe inter-rater reliability and correlations between impact factors and citedness, failure to address this issue affects the research rigour. finally, problems with google scholar reliability are recognized to limit a more rigorous and supported comparison between it and other, more conventional bibliographic databases, including pubmed. as jacsó (2012) has concluded, google scholar metadata is “substandard, neither reliable nor reproducible and it distorts the metric indicators at the individual, corporate and journal levels” (p. 462). considering his remarks, this reviewer can only speculate that the present research is one example of exactly what jacso warned against when he stated:   it is hoped that the wailing sound of air-raid sirens in this paper will act as an early warning for the tempting siren song in current papers about using google scholar to compute bibliometric data (publication and citation counts, the h-index and its variants) for ranking journals on a nationwide scale as part of assessing the scholarly productivity and impact of universities and colleges. (p. 463)   ultimately, the value of the study is limited by lack of transparency, making it difficult to evaluate or replicate the work. readers are asked to accept assessments of relevance without seeing the relevant/non-relevant citations, or even inclusion and exclusion criteria with which to deepen understanding and enable replication.   the perspectives of non-expert searchers in google scholar and pubmed comprise a valuable contribution to a scarce body of literature. awareness of a more naïve searcher’s perspective is needed to inform information professionals working with clinicians who have more advanced knowledge of subjects, but who are limited in their searching expertise. in addition, the research provides a basis for further study that may lead to improvement of retrieval mechanisms and techniques for both pubmed and google scholar.   this reviewer used a bibliometric tool (perryman, 2009) while evaluating this study, as no currently available tool would work to evaluate this research methodology. the question set is based upon existing published tools, with questions specific to bibliometric studies     references   anders, m. e., & evans, d. p. (2010). comparison of pubmed and google scholar literature searches. respiratory care, 55(5), 578-583.   cosijn, e., & ingwersen, p. (2000). dimensions of relevance. information processing & management, 36(4), 533-550.   jacsó, p. (2012). using google scholar for journal impact factors and the h-index in nationwide publishing assessments in academia–siren songs and air-raid sirens. online information review, 36(3), 462-478.   mastrangelo, g., fadda, e., rossi, c. r., zamprogno, e., buja, a., & cegolon, l. (2010). literature search on risk factors for sarcoma: pubmed and google scholar may be complementary sources. bmc research notes, 3(1), 131.   perryman, c. (2009). critical appraisal tool for bibliometric studies. retrieved from  http://evidence-based-librarian.blogspot.com/2013/03/another-update-bibliometric-study.html   shultz, m. (2007). comparing test searches in pubmed and google scholar. journal of the medical library association, 95(4), 442.   tober, m. (2011). pubmed, sciencedirect, scopus or google scholar: which is the best search engine for an effective literature research in laser medicine? medical laser application, 26(3), 139-144.   evidence summary   patron time-use may be an effective metric for presenting library value to policy makers   a review of: shepherd, j., vardy, k., & wilson, a. (2015). quantifying patron time-use of a public library. library management, 36(6/7), 448-461. doi: 10.1108/lm-09-2014-0110   reviewed by: ann glusker reference/consumer health librarian business, science and technology department the seattle public library seattle, washington, united states of america email: ann.glusker@spl.org   received: 1 mar. 2016    accepted: 15 apr. 2016      2016 glusker. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to test a metric for library use, that could be comparable to metrics used by competing government departments, for ease of understanding by policy makers.   design – four types of data were collected and used: time-diaries, exit surveys, gate counts, and circulation statistics.   setting – a large public library in british columbia, canada.   subjects – time-diary subjects were 445 patrons checking out materials; exit survey subjects were 185 patrons leaving the library.   methods – a paper-based time diary, prototypes of which were tested, was given to patrons who checked out library materials during a one-week period. these patrons were charged with recording the use of the checked-out items during the entire three-week loan period. from this information, the average number of hours spent with various types of loaned material (print and audio/dvd) was calculated. the average number of hours spent per item type was then applied to the circulation statistics for those items, across a month, to get a total of hours spent using all circulated material during that month. during the same one-week period of time-diary distribution, exit surveys were conducted by library staff with patrons leaving the library, asking them how long they had spent in the library during their current visit. the average number of minutes per visit was calculated and then applied to the gate count for the month, to get a total number of minutes/hours spent “resident” in the library that month. adding the totals, a grand total of patron time-use hours was calculated. a monetary value was applied per hour, using the results of a contingent valuation study from missoula, montana (dalenberg et al., 2004), in order to convert hours of library benefit into a dollar figure.   main results – there was a 24% response rate for the time diaries (106/445). the diary entries yielded an average of 3.5 hours of time-use per print item, and 1.9 hours per dvd. the range for audio materials was quite wide, and for all item types, a few heavy users skewed the averages. hours of secondary use (when people other than the original borrower read, listened to, or watched, the materials) were calculated, and represented 13% of the total hours. the average amount of time spent per visit was 42 minutes. applying these averages to one month of circulation figures and gate counts, respectively, the result was that patrons spent 182,000 hours using library services in one month. applying dollar amounts of benefit per hours spent, based on the missoula study, the result was that patrons had received $842,000 of benefit from their use of the library in that month.   conclusions – this study confirmed that the prototype performance measure of hours of patron use, and refinements in obtaining it, was a useful tool with which to present the case for the value of libraries to policy makers. the study estimates that 90% of library use occurs off-site, and that a surprising proportion of that use is by secondary users. future studies could refine the collection methodologies even more by collecting demographic information, by mapping user activities during branch visits, and by obtaining better information about secondary users of materials. future research should also take into account: seasonal effects on borrowing, reading level of borrowers, and possibilities for collecting information in online formats. with these developments, it might be possible to assign “enjoyment levels” to items in library catalogs.   commentary   proving a library’s value to those who make funding decisions is a crucial question for libraries everywhere. one common method is to present percentage returns on investment in libraries, by estimating cost savings and benefits, performing contingent valuation studies (figuring out how much patrons would be willing to pay out of pocket) or similar methodologies. contingent valuation studies are practical, and amenable to statistical analysis, but are extremely complex to develop and administer (hider, 2008; mcintosh, 2013). additionally, the methods and results are not easily understandable by policy makers, which can put libraries using them at a disadvantage when jockeying for funding against other equally deserving city departments. the authors of this study propose instead to present an easily understood measure, hours of patron time-use, obtaining data using the well-studied method of time-diaries. time-diaries are excellent for assessing daily and routine behaviors (paolisso & hames, 2010) and thus appear to be a good choice for studying library use.   for this evidence summary, methodologies were systematically assessed using the critical appraisal checklist by glynn (2006), which indicated issues of potential bias and lack of representativeness. it seems likely that the 24% of subjects who completed the diaries (a low response rate) were different in some important ways from those who did not; having more positive connections to the library and higher literacy levels are two possibilities. usable time diaries depend on diarists being reliable, motivated, accurate, candid, etc. (paolisso & hames, 2010), which could be a high bar for many patrons. also, for both the time-diaries and the exit surveys, the same staff who were serving patrons were acting as researchers, which could affect responses. the study authors themselves noted important limitations of their data, including the lack of demographic information. e-books and online resources were not examined, which would be a significant omission for many library systems. additionally, while there were many mentions of contingent valuation methodology, it only appeared in the final calculation of dollar valuation per hour, using a dated figure from a study that was not discussed in adequate proportion to its importance in deriving the final dollar figure.   while this study has many methodological issues (most of them freely acknowledged) and challenges, at the same time it makes compelling and novel points on the utility both of quantifying hours of library use, particularly those spent with materials outside the library, and of learning more about secondary users. perhaps most importantly, it shows how a simple measure, simply derived, could resonate with policy makers and lead to better-funded libraries. this study was in its essence a powerful political exercise, and it is likely that the city politicians to whom it was presented were not quibbling about methodology.   references   dalenberg, d., fitzgerald, d., schuck, e. & wicks, j. (2004). how much is leisure worth? direct measurement with contingent valuation. review of economics of the household, 2 (4), 351-265.   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi:10.1108/07378830610692154   hider, p. (2008). using the contingent valuation method for dollar valuations of library services. the library quarterly, 78 (4), 437-458.   mcintosh, c. (2013). library return on investment: defending the contingent valuation method for public benefits estimation. library & information science research, 35 (2), 117-126.   paolisso, m. & hames, r. (2010). time diary versus instantaneous sampling: a comparison of two behavioral research methods. university of nebraska-lincoln anthropology faculty publications, paper 87.     news   call for applicants for eblip journal: copyeditors    2022. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30277     evidence based library and information practice (eblip) is looking for two new copyeditors to join our copyediting team. ideally, candidates will be in place by march 2023, in time to begin work on the june issue.   the role of copyeditors is to:   communicate with the lead copyeditor regarding circumstances that might require temporary absences from editing availability   respond to editors in a timely manner to accept or reject copyediting requests meet copyediting deadlines set by editors utilize eblip copyediting guidelines alongside apa style guidelines ensure that proper grammar and spelling conventions are addressed verify citations and references to the extent possible with available bibliographic resources and advise editor of any not checked identify inconsistencies or changes in format or style as necessary   the ideal candidate will have copyediting or other detail-oriented editing experience and will possess a working knowledge of apa style and an interest in evidence based practice. the position requires dedicated time to ensure eblip publishing deadlines are met, and it is therefore essential that interested persons ensure available time to devote to this position prior to applying. the estimated workload is approximately 5 to 10 hours per issue, and much of this work falls over a one-month period, on a quarterly basis, prior to the publication of each issue.   interested persons should send a cover letter, indicating areas of strength they would bring to the role, and resume/cv as a single pdf file to heather healy, lead copyeditor, at heather-healy@uiowa.edu by january 20, 2023.    please note: evidence based library and information practice is a non-profit, open access journal and all positions are voluntary and unpaid. the positions are an excellent opportunity for continuing professional development and gaining experience in reviewing or critically appraising library-related research. about the journal:   published quarterly and hosted by the university of alberta, this peer-reviewed, open access journal is targeted at all library and information professionals interested in an evidence based model of practice. by facilitating access to librarianship research via original research articles and evidence summaries of relevant research from the library literature, evidence based library and information practice enables librarians to practice their profession in an evidence based manner. please visit the evidence based library and information practice web site (https://journals.library.ualberta.ca/eblip/index.php/eblip) for further information about the journal.   microsoft word es 3373 evidence based library and information practice 2008, 3:4 61 evidence based library and information practice evidence summary larger, higher-level academic institutions in the us do not necessarily have betterresourced library web teams a review of: connell, ruth sara. "survey of web developers in academic libraries." the journal of academic librarianship 34.2 (2008): 121-29. reviewed by: suzanne lewis librarian northern sydney central coast health gosford, new south wales, australia e-mail: slewis@nsccahs.health.nsw.gov.au received: 01 august 2008 accepted: 01 november 2008 © 2008 lewis. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. objective – to discover how library web teams’ staffing, backgrounds, tools, and professional development differ among various types of academic libraries. design – survey. setting – academic libraries in the united states. subjects – academic library web team members. methods – a systematic sample of every twelfth institution on the carnegie classification of institutions of higher education list was used to establish a sample group. a web search was carried out to identify each institution’s library web site and contact information for the web site designer or most appropriate alternative person. institutions were excluded from the sample if they had no web site at all, had no library web site, had a web site that did not mention a library, or had a spanishlanguage web site. in september 2006 an e-mail was sent to the contact for each institution in the sample group asking them to participate in an online survey. a follow up e-mail was sent two weeks later and the survey closed after one month. the survey respondents were asked to identify their institutions so that analysis of the results in relation to the size and type of institution could be carried out. evidence based library and information practice 2008, 3:4 62 the researchers used a simplified version of the carnegie classification to sort the responding institutions into five main groups. main results – the systematic sample consisted of 288 institutions (sample size 6.5%). the profile of the responding institutions was as follows: associate’s colleges (35.5%), baccalaureate colleges (18.2%), master’s colleges and universities (20.9%), doctorategranting universities (9%) and special focus institutions (15.5%). a total of 110 institutions completed the survey, yielding a response rate of 38.19%, although not all respondents answered all the survey questions. the final sample of 110 was 2.5% of the total 4384 institutions on the carnegie list. seventy-one per cent of institutions with multiple libraries shared web teams, with two-year colleges more likely (91.7%) to share a web team than four-year or above institutions (60.9%). the majority of responding institutions (94.4%) used inhouse library web site design, with only 5.6% of respondents outsourcing this task. nearly half (49%) of respondents indicated that library web design was done by one person and even the larger libraries did not necessarily have larger web teams. very few web team members (4.9%) had web design as their primary role; the majority (83.5%) indicated that it was just one component of their job. web team members from master’sand doctorategranting institutions were more likely to have taught themselves web design, while those from associate, baccalaureate and special focus colleges were more likely to have taken web design courses. for all respondents, the most commonly listed quality for selection to the web team was an interest in web design and the most valued skill for library web designers was the ability to organise information effectively. knowledge of web authoring software and basic html coding were the most commonly listed knowledge requirements for web team members. a significant number of respondents indicated that they or other web team members did not have access to web authoring (36.9%) and image editing (52%) software. generally (except for two very large institutions), the larger institutions were more likely to use database-driven systems for their library web sites and the smaller institutions were more likely to use content management systems. associate’s and special focus colleges were less likely than other types of institutions to use either database driven or content management systems. associate’s institutions were more likely to achieve ada (americans with disabilities act) web accessibility compliance. only 48.6% of respondents utilised usability testing during web site design. conclusion – the author expected that institutions providing higher levels of education would have better-resourced web design and training, but the results of the survey did not support this expectation. one reason why associate’s colleges performed better than other institutions in some areas of web design may be that these colleges tend to offer more web design and computer technology courses than baccalaureate, master’s and doctorategranting institutions. web site design and testing attracted fewer resources than might be expected in academic libraries. across all types of institutions, web design tended to be the responsibility of a small team or one person, with most web designers having other responsibilities apart from the library’s web site. just over half of the institutions evidence based library and information practice 2008, 3:4 63 surveyed did not implement usability testing of their library web sites. commentary the author sets the current study in context with a literature review describing previous interviews and surveys of library web designers. however the author has chosen not to use any of the survey instruments employed in these earlier studies, instead designing one which builds on questions asked in previous surveys. the survey instrument is not included with the article itself but can be accessed by following a link from the electronic version of the article. the questions are clearly posed, but the author does not state whether the instrument was validated before use. validation may have picked up one problem with the survey, which became clear as responses were received. the survey was designed on the assumption that most academic libraries had a web design team, but this was true for only 51% of participants and made it difficult for one-person web design “teams” to answer many of the questions. this survey attempts to relate information about academic library web design teams to the size and type of the parent institution. however, the unevenness of the final survey sample decreases the internal and external validity of the study, particularly the external validity. the unevenness was due to the way the sample was constructed, taking every twelfth institution from the carnegie classification list. the author acknowledges that a better method would have been to use a stratified sample, that is, first separate the academic institutions by type and then take a random sample from within each group. some questions are also raised by the exclusion criteria applied to the sample. in particular, institutions were excluded if no library web site could be found for them. this would have eliminated institutions whose library web sites sit behind a firewall and require validation to access. the author wisely chose not to use all thirtythree levels in the basic carnegie classification to sort responding institutions, instead opting for a simplified version containing five categories. it would have been helpful to include a glossary explaining the terminology used in these five categories, particularly for international readers unfamiliar with terms such as “associate’s colleges” and “baccalaureate colleges.” the survey was carried out in septemberoctober 2006, but this paper was not published until 2008. this time lag between research and publication makes the finding that only 7.4% of respondents use rss feeds to syndicate their library website content out-of-date. adoption of web 2.0 technologies by libraries is developing so quickly that by now it is likely that many more of the participating institutions are using either rss feeds or blogging tools with built-in rss to update their patrons on changes to the library’s web site. the most interesting finding of this survey was that academic libraries of all sizes and types are not allocating the staff resources to web design that might be expected or required. the library web site is a major point of access to the information resources and services provided by academic institutions for their students. indeed for many students it is the only point of access. an inviting, highly usable, well-designed evidence based library and information practice 2008, 3:4 64 and maintained web site should be a priority for academic libraries, and staff and budget allocations to web design should reflect this priority. the selected comments from survey participants quoted by the author reflect the frustration of librarians when this is not the case: i desperately wish that our web site could be tended regularly by someone with the skills and determination to make it extraordinary. as the portal to the library, it should be the most winning, winsome presence that we can possibly create. and it isn’t, simply because we don’t have the skills or staff to make it so (128). the findings of this study are most applicable to academic libraries within the united states, but enough information is given to enable the methodology to be repeated in other countries, with appropriate adjustment of the way academic institutions are classified and sampled. classification systems used by countries other than the united states are much less complex than the carnegie classification. this study will be of interest to the academic web designers, librarians and managers who are trying to juggle web design with their other responsibilities or secure funding to upgrade or redesign a library web site. this study has the potential to change practice by providing supporting evidence for web development funding, particularly in larger, higher-level institutions. conference paper   knowing how good our searches are: an approach derived from search filter development methodology   sarah hayman research fellow (evidence) flinders filters flinders university adelaide, south australia, australia email: sarah.hayman@flinders.edu.au   received: 10 aug. 2015  accepted: 30 oct. 2015      2015 hayman. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – effective literature searching is of paramount importance in supporting evidence based practice, research, and policy. missed references can have adverse effects on outcomes. this paper reports on the development and evaluation of an online learning resource, designed for librarians and other interested searchers, presenting an evidence based approach to enhancing and testing literature searches.   methods – we developed and evaluated the set of free online learning modules for librarians called smart searching, suggesting the use of techniques derived from search filter development undertaken by the caresearch palliative care knowledge network and its associated project flinders filters. the searching module content has been informed by the processes and principles used in search filter development. the self-paced modules are intended to help librarians and other interested searchers test the effectiveness of their literature searches, provide evidence of search performance that can be used to improve searches, as well as to evaluate and promote searching expertise. each module covers one of four techniques, or core principles, employed in search filter development: (1) collaboration with subject experts; (2) use of a reference sample set; (3) term identification through frequency analysis; and (4) iterative testing. evaluation of the resource comprised ongoing monitoring of web analytics to determine factors such as numbers of users and geographic origin; a user survey conducted online elicited qualitative information about the usefulness of the resource.   results – the resource was launched in may 2014. web analytics show over 6,000 unique users from 101 countries (at 9 august 2015). responses to the survey (n=50) indicated that 80% would recommend the resource to a colleague.   conclusions – an evidence based approach to searching, derived from search filter development methodology, has been shown to have value as an online learning resource. more information is needed about the reasons why people are using the resource beyond what could be ascertained by the survey results. introduction   effective searching is of central importance to the acquisition of published evidence across all disciplines of study and to informing practice and policy in diverse fields. evidence based practice has a strong presence in the health sciences (medicine, nursing, and allied health), and health is the sphere of activity within which the work in this paper is situated. however, evidence based approaches (such as the undertaking of systematic reviews) are now embedded in many other areas, including environmental science, engineering, and computer science. these approaches are also found in areas of policy, education, management, and social sciences (hayman and tieman, 2015b).   in health, decisions made about treatment of patients can have significantly different outcomes depending on the evidence on which those decisions are based. adverse effects can result from wrong or missing information in any field of endeavour. scientific development builds on research that has gone before and must be underpinned by accurate information. as librarians well understand, the key to the discovery of the best available evidence is a well-executed search.   together with the need to search and find the best available evidence, to underpin practice, research, and policy, is the challenge of searching effectively. databases of complex and differing structures hold a massive and increasing amount of bibliographic information. the quantity of published and indexed articles is vast, even without considering the “grey literature” that must also be searched for a comprehensive search, such as one undertaken for a systematic review. the scopus database contains 55 million records; web of science captures 65 million cited references annually; pubmed in june this year (2015) grew to 25 million records.   the technical challenges of searching are increasing, with a range of databases available in most fields of study, often using different thesauri and different search syntax. effective searching requires an understanding of boolean search techniques as well as knowledge of how they have been implemented in the particular search interface of each database. mcgowan and sampson (2005) have written of the need for expert searchers to understand “the specifics about data structure and functions of bibliographic and specialized databases, as well as the technical and methodological issues of searching.”   one tool available to enhance searching effectiveness is the search filter. we define the search filters created at caresearch and flinders filters as follows: a search filter is a validated search strategy built for a particular bibliographic database and with known performance effectiveness. each term in the strategy has been tested for its recall of references from a gold standard set. many search filters are now available from a range of different sources. they may be methodology-based search filters (designed to retrieve literature of a particular study type) or subject-based search filters (designed to retrieve literature on a particular subject). several useful websites provide information about where to find search filters and documentation about their development and validation, for example, the intertasc information specialists' sub-group search filter resource is an excellent source of information about methodological search filters (https://sites.google.com/a/york.ac.uk/issgsearch-filters-resource/home).   search filters are of variable quality and it is important to understand how to use them and how to judge them. not all are validated. there are useful appraisal tools for search filters, for example, the detailed issg search filter appraisal checklist (glanville, et al., 2008) and the cadth cai (bak, et al., 2009).   the search filters developed by caresearch palliative care knowledge network (http://www.caresearch.com.au) and its associated project flinders filters (http://www.flinders.edu.au/clinical-change/research/flinders-filters/) are topical (subject-based) search filters on topics including palliative care, heart failure, bereavement, dementia, primary health care, and australian indigenous health care. these filters were developed in ovidsp medline and translated for use in pubmed and are available online for use by anyone to conduct a search of tested and known reliability. we have also published articles on the search filter development and methodology employed for each filter listed above. (brown et al. 2014; damarell, tieman, sladek, and davidson, 2011; hayman and tieman, 2015, may 28; sladek et al., 2006. tieman, lawrence, damarell, sladek, and nikolof, 2014; tieman, hayman, and hall, 2015). an important element of search filter development is that the process of development is not only rigorous, but also documented and transparent.   the librarians working within these two projects to create search filters are part of a team developing an experimental research searching method. we have for some time discussed how the processes we use to develop the filters have caused us to re-examine the way we undertake general literature searching. the detailed technical bias minimisation approach we employ in search filter design and assessment offers opportunities to see how some of these conceptual approaches could be applied in the day-to-day literature searching undertaken by librarians and others. the receipt of the health informatics innovation award in 2012 from health libraries australia and medical director (then health communication network) provided an opportunity to create an online resource to capture elements of these processes and make them available for use by librarians and others who might find them useful.   investigation of existing online continuing professional development tools for librarians showed few resources available on expert searching. most guides to searching effectively in online bibliographic databases are user guides written by librarians for their patrons; these focus on using and understanding the different databases, and general searching principles. sampson and mcgowan (2005) wrote of librarians testing their retrieved sets, stating: ”the librarian must have the expertise to develop test strategies to verify the performance of terms and elements of the search, adjusting or abandoning nonperforming elements. often these tests rely on comparison against a strategy from a previously published review or the recall of a set of key references supplied by subject experts.” however, we found few tools available to teach how to do this. one example we found was the excellent online training on building search strategies provided by wichor bramer’s slide presentations (http://www.slideshare.net/wichor). another is dean giustini’s useful presentation on search techniques (http://www.slideshare.net/giustinid/expert-searching-for-health-librarians-2012). the press (peer review of electronic search strategies) tool is a validated tool for peer reviewing search strategies that, as well as providing quality assurance for the search itself, is likely to enhance searching skills through peer review and support, and use of the associated evidence based assessment checklist (mcgowan, sampson, and lefebvre, 2010). the chapter on designing search strategies in the cochrane collaboration handbook (higgins and green, 2011) is an indispensable guide to searching systematically, and we hope that the smart searching modules will provide some approaches to support the searching methods within that guide.   aims   we aimed to provide an online resource that would be self-paced, accessible and free to use, and that would introduce librarians (and other interested searchers) to techniques for applying an evidence based approach to their own searching practice. the module would utilise approaches used in research activities associated with search filter development adapted for individual and local searching contexts.   we also planned to undertake evaluation of the resource to gauge its usefulness. the intended audience is chiefly librarians, and the resource is likely to be of most use to those in the health sector. we expect that it will also be useful to librarians beyond health, as the principles are widely applicable to all searching. we hope that it may also be of use to anyone (librarian or not) with a keen interest in searching. a moderate level of searching expertise is desirable for those using the modules.   methods   development of the resource. we created an open-access website in google sites at https://sites.google.com/site/smartsearchinglogical/home. the website consists of four self-paced modules requiring no logon to use. all modules can be accessed at any time without the requirement to complete assessment first. simple quizzes are provided.   the methods suggested in the modules can be applied to sensitive or specific searches, as the need arises. they are likely to be principles and approaches already used by expert searchers; we hope that setting them out in this way will be useful and that elements of the approach can be used and adapted as necessary. the framework for searching in the modules reflects the stages used in search filter development and draws on some of the techniques used in their development.   our search filters are created using steps such as those set out in figure 1.   in drawing on this process to shape the learning modules, we focussed on the following key elements: (1) expert advisory group (eag); (2) gold standard set; (3) term identification; and (4) validation.   the eag ensures the clinical usefulness of the search filter and minimises bias that we (as searching experts but not necessarily subject experts) might bring to the search strategy. eag members provide advice on the scope of the filter, potential search terms, and possible sources of a representative gold standard set; they are also available to test draft search retrievals for relevance, as part of the validation process.   the gold standard set is a set of references representative of the entire scope of the topic to be retrieved by the search, and externally confirmed as relevant to the topic. this set is divided into three subsets so that term identification, creation, and validation can all be done within different sets of data; again, aiming to reduce any potential bias that could arise from building and testing within the same set.     figure 1 steps in search filter development at caresearch and flinders filters     term identification is the process of analysing the titles, abstracts, and subject headings of the references to identify text words (natural language terms) and controlled headings (usually mesh terms) to be tested for their recall effectiveness in the gold standard set.   validation includes the testing of the search strategy within a subset of the gold standard set, within the entire gold standard set, and often within an external validation set, to arrive at a percentage that is a measure of its retrieval performance. its ability to retrieve items known to be relevant (e.g., within the gold standard set) gives a sensitivity percentage rating; the number of relevant records retrieved out of a total set retrieved by the search strategy gives the precision percentage rating (using relevance assessment by external reviewers).   we drew from these four approaches as follows.   (1) eag became module 1: subject experts. the formal expert advisory group crucial to the search filter development process can be represented by seeking external advice from a subject-matter expert. this person may simply be the researcher or clinician who has requested the search, or may be a colleague in that field. advice they provide can help reduce bias that the librarian might bring to the search and can add a dimension to the search of external knowledge about the subject area. this knowledge can provide useful advice about appropriate scope for the search (e.g., dates when research in the subject changed significantly, or concepts that are uniquely associated with the topic), relevant terminology (e.g., synonyms in common use), key papers, journals, database, organisations, websites or authors in the field (they may even have a personal collection of papers to function as a potential sample reference set). they may also be able to undertake a relevance assessment of draft search retrievals, enabling adjustment of the search. while both librarians and health professionals are busy and always working under time constraints, it can nevertheless be extremely valuable to get some suggestions to inform the development of a search strategy before the search and some feedback after the search – both can supply useful information about the effectiveness of the search that will allow the librarian to analyse and tweak it. if it is not retrieving key papers that have been recommended in the field, why not? check the index terms and text words and see if any have been missed. if it is retrieving a large number of items that are not relevant, why is this happening? check the search terms that are retrieving the irrelevant items and see what happens if they are removed.   (2) gold standard set became module 2: sample set. the creation of a formal gold standard set, employed in the development of a search filter, is a major piece of work using an established methodology. without going to those lengths for a literature search, we nevertheless suggest that creating a sample set of references to guide a search can still be very useful. a sample set of references, known to be relevant to the search topic, provides a test set for (1) identifying terms used in the literature for the topic and (2) testing the effectiveness of the search in retrieving references known to be relevant. the contents and relevance of this sample set should be externally verified, not a set derived from the search that is being tested. possible sources of a sample set are: a collection of papers provided by an expert in the subject; a published database in the field; references from key papers known to be relevant (included studies in systematic reviews are an excellent source as they have been assessed as relevant within the systematic review process); articles from relevant and authoritative journals in the field.   (3) term identification became module 3: term identification. term identification is a standard process that already occurs in all literature searching to some extent. thorough analysing and testing of candidate terms for your search strategy is a very useful technique for ensuring a high performing search strategy that will capture a high proportion of relevant items and a low proportion of irrelevant ones. in the full search filter development model, we undertake extensive research, analysis, and testing of potential search terms for each subject. in general literature searching it is still possible to do some investigation and analysis to help identify the best terns for the search. sources for the terms will be: medical subject headings (mesh) or other database-specific thesauri, e.g., emtree, ieee thesaurus, cinahl subject headings, eric thesaurus; expert suggestions of relevant terms; analysis of key references (the sample set). it is useful to confirm with the subject expert that the candidate terms are correct and relevant. we suggest analysis of the frequency of text words (natural language terms) in searchable fields in the sample reference set, typically the title and abstract fields. this will give alternative candidate terms to test, i.e., those known to be associated with relevant references.   (4) validation became module 4: testing. testing can be done at a number of levels, from simple checks through to formal external validation. any element of testing introduced can result in an improved search. the search strategy is built by combining candidate terms and testing sequentially against the sample set to see how many references are retrieved. testing the terms and their performance in a set of known relevance is important, as it can assist in identifying what is not retrieved and why; it can identify terms that add nothing to the search results; and it will facilitate adjustment of the search to improve results. this type of test (assessing retrieval within a set of known relevant items) tests the sensitivity (or recall) of a search; that is, its ability to retrieve relevant items. testing for precision (i.e., how many of the retrieved citations are relevant) is also important. to assess this, we suggest external expert assessment of the relevance of number of relevant items the search has retrieved in a sample search in the open database. a comprehensive systematic review search requires maximum sensitivity and there is less concern with a high degree of precision. the searcher wishes to retrieve all relevant items and is willing to risk a large number of irrelevant retrievals. clinicians may however prefer that most items retrieved are relevant and not wish to wade through a large number of irrelevant items. it is possible and important to increase or decrease the sensitivity depending on the requirements of the end user. as sensitivity increases, precision will decrease, and vice versa. testing is an iterative process that feeds back into the development of the search strategy, improving it each time, and resulting in an enhanced search that is less likely to miss key references.   each module contains an explanation of the principle and why it is important. it also contains a worked scenario of a librarian undertaking a search for a clinician that goes across all four modules sequentially to illustrate the process as it might occur in practice.   the development of the modules was guided by an advisory group with expertise in searching, health librarianship, health informatics, and education drawn from organisations across australia (listed at https://docs.google.com/viewer?a=v&pid=sites&srcid=zgvmyxvsdgrvbwfpbnxzbwfydhnlyxjjagluz2xvz2ljywx8z3g6mjnmzwjkodjiowyznzcznw)   evaluation. detailed web usage statistics are available using google analytics, because the resource site is built in google sites. statistics available include: numbers of users by day since the site was launched, shown as total visits and unique visitors; total page views; time spent on site and bounce rate, as well as the geographic location of users. the web statistics are regularly measured and reported on the smart searching website itself using the programme seethestats (http://www.seethestats.com). feedback is sought directly from users on the site via email.   a survey of users worldwide was conducted in april 2015. the questions asked are provided in appendix a. the survey aimed to ascertain the occupations of the users, the nature of the organisations and disciplines where the users worked, and qualitative information about the usefulness of the resource. ethics approval was obtained from flinders university to conduct the survey. the user survey was pretested with colleagues for technical function and its content was reviewed by a senior colleague and with peers before it was disseminated. all questions were optional and no identifying data was obtained. notices informing users about the survey were put on the site itself and were sent to health librarian and searching email lists in australia and overseas. these were the same channels used to promote the site when it was launched.   a workshop presenting the smart searching resource was held as a satellite event in conjunction with the 8th international evidence based library and information practice conference (eblip8), in brisbane, queensland, australia, in july 2015. the workshop was attended by approximately 50 people over 2 sessions and was an opportunity to receive some direct feedback from participants about the content and usefulness of the resource.   results   web analytics. a summary of web statistics is presented in figure 2. they cover the period from launch on may 25, 2014 to the time of writing this article (august 9, 2015).   the figure shows the increase in usage of the site that occurred when the user survey was promoted in april 2015.   figure 3 shows the top 10 countries by visit. bounce rates are significantly lower (and session duration longer) for visitors from australia, the united kingdom, new zealand, canada, and ireland, suggesting these users are using the site more extensively. details for the top 20 countries visiting the site are shown in appendix b, and up-to-date statistics are made available on the smart searching website under usage and feedback.     figure 2 google analytics (overview) for the smart searching website (august 9, 2015)      figure 3 google analytics (geographic location for the smart searching website (august 9, 2015)     overall, these statistics show a total of 8,592 visits by 6,297 unique users (figure 2), from 101 countries (one country location being “not set”).   user survey. a total of 50 people responded to the survey. while this is a small percentage of the 3,855 unique users of the site at the end of april 2015, nevertheless it provides an indication of a range of views from users in different occupations and countries. table 1 summarises demographic responses.   table 2 shows responses relating to the use of the site and views about its usefulness.   finally, table 3 provides a selection of comments providing a representative overview of the responses to open-ended questions about the value of the resource. the selection of comments has been reviewed by members of the smart searching advisory group. a review of the comments identified three main themes: site function; time constraints; value.     table 1 smart searching user survey responses to demographic questions occupation librarian   41 (80.39%) other information professional   7 (13.73%) student   2 (3.92%) other   1 (1.96%) country australia   21 (42%) usa   16 (32%) uk 10   (20%) canada   1 (2%) cyprus   1 (2%) netherlands   1 (2%) discipline (self-described) health / health care / health sciences   34 (68%) medicine   2 (4 %) nursing   2 (4%) biomedicine, health   1 (2%) education   1 (2%) health / medicine   1 (2%) health and social care   1 (2%) health promotion   1 (2%) health, children, housing, aboriginal   1 (2%) health, psychology   1 (2%) medical school / allied health   1 (2%) public library   1 (2%) regulatory   1 (2%) social sciences   1 (2%) youth   1 (2%)   table 2 smart searching user survey responses to qualitative questions have you applied any of these techniques in your searching practice? yes, all or most 18 (36%) yes, a few   13 (26%) no, but i may do so   10 (20%) no, but it has made me think differently about my searching 5 (10%) no, and i am not likely to do so   1 (2%) no response 3 do you think you would use this approach for testing? not sure 23 (46%) yes 20 (40%) no 3 (6%) no response 4 would you recommend this site to a colleague? yes 40 (80%) no 0 (0%) not sure 9 (18%) no response 1     table 3 smart searching user survey responses to open-ended qualitative questions site function systematic and methodical approach clear and easy to use (13 other responses were similar to this) template applicable across disciplines while the content is usefulthe constant arrow moving would not be appealing to busy clinicians or medical librarians time constraints time is the major factor, followed closely by access to the subject experts it can possibly save me some time as i spend a lot of time in my job training and assisting health researchers in building effective literature searches time restraints, level of information need does not usually require that level of sensitivity (several like this) very useful, however not sure if i would have time to test every search in a real life work situation. value reassessing the way i approach things the more knowledge/ideas we share about improving search techniques the more beneficial it is to the profession i can see the value in being able to 'qualify' and measure my searching outcomes informative...i bet there are other librarians who, just like me, are not utilizing these techniques properly. adding those extra dimensions increases the robustness of our searching and helps to systematise the things we do i tend to be more intuitive than systematic with my searches […] reporting would force me to ensure consistency! seems great as a refresher for me but will also be really useful for staff training purposes i don't think that librarians test their search strategy and i feel it is an important tool to argue our competence and relevancy, especially in private enterprise it gives a measure of effectiveness that speaks for itself...numbers are extremely hard to dispute!       other feedback. there has been very little response to requests on the site for direct feedback, other than one detailed and useful response which led to some small adjustments, chief of which was the addition of a recommendation of the tool pubreminer (http://hgserver2.amc.nl/cgi-bin/miner/miner2.cgi). formal evaluation from the workshop following the eblip8 conference was undertaken. of those responses 22 of 26 rated the workshop as useful (4) or very useful (18). informal feedback on the day was positive, with one participant commenting that it “was a new way to think about approaching searching”.   discussion   the high rate of usage of the website smart searching internationally suggests that there is a desire for this type of information, and this is supported by many of the comments received in the user survey, some of which are shown above under value. there appears to be a gap in available resources providing instructions at an advanced level for developing and testing search strategies, especially free and online. some important guides have been cited in the introduction, but there appears to be an appetite for a step-by-step learning resource. such a resource could be useful for continuing professional development for librarians themselves, especially in, but not confined to, the health sector. one survey respondent commented that the approach is “applicable across disciplines” which has certainly been the intention, although the examples provided are from health. the general principles apply to all searching in any field. potentially such resources might also be useful to others (not only librarians) who are conducting sophisticated searches, such as researchers experienced in advanced searching in their own fields.   overall the responses were positive, with 80% responding “yes” to the key question “would you recommend this site to a colleague?” (table 2). of the 50 respondents 14 commented favourably on the clarity and logical approach of the site. there appeared to be general agreement amongst many of the survey respondents that there is a need both to improve and to measure our searching performance.   several respondents commented on the time-consuming nature of this approach (although one person believed it might save time, presumably if used to teach clients to conduct their own searches). we are very conscious that it may seem long-winded and cumbersome to test iteratively every term. it is suggested more as an overall way of thinking about searching than with an expectation that every search would require every step illustrated in the smart searching scenarios. the intention is for users to dip in and out as desired (and as appropriate for the particular search) and apply the elements they wish of this approach. we believe that any additional testing applied to any search has the potential to improve it. another aspect of this approach that has the potential to be time-consuming, both for the librarian and their “subject expert”, is the consultation between them, and the requirement for the subject expert to do some checking and verification during the process. this is something that can add enormous value but may be difficult to achieve in practice. it is worth remembering that the result of such investment of time and effort is a strong search with an ongoing value. it can be embedded on a website for reuse and can be used to set up a search alert; ultimately it may save time for the librarian and the client, and should have the immediate outcome of higher quality search results.   the question about using this method for testing one’s searches received a less clear positive response than other areas. of the respondents 46% were unsure whether they would use this method to do this (table 2). this may reflect that it is probably the most novel part of the approach and may be a more complex area of the site to follow. we believe that testing searches objectively provides an opportunity for librarians to provide some evidence of the effectiveness of their searching. to quote one respondent (table 3): “i don't think that librarians test their search strategy and i feel it is an important tool to argue our competence and relevancy”. it may be that there is agreement with this but that this particular method does not appeal, or is not clear. one response indicated that this section (4.2) was difficult to follow. in following up the survey results, and making adjustments to the site in response, we will review this section of the site and aim to clarify it with additional examples.   some respondents commented that they already apply these or similar techniques, and we expect that highly experienced searchers would not need to use these modules. one respondent did not like the quizzes and would like to see them removed while another respondent singled out the quizzes as a highlight. two respondents raised the issue of the nature and role of the subject experts, and whether this section is oversimplified or incorrect (librarians can also be subject experts). we believe that this is an important distinction: although librarians may indeed also be subject experts, it is a different role.   evaluation of the smart searching resource was limited by the difficulty of eliciting responses from users. the number of users (50) responding to the online survey is a small percentage of the over 6,000 people who have used it to date worldwide. we do not know how far they represent the users of the site as a whole. as far as geographic location is concerned, 42% of respondents were from australia, 32% from the united states, and 20% from the uk; this compares to the following percentages for website users at that time: australia 26%; united states 22%; united kingdom 12%.   while comments received were useful and informative, and overall positive, we would still like more information about why people are using it and whether they are finding what they are seeking. a future survey will be conducted attempting to find out more about people’s level of experience of this type of searching before they started the module and about the type of work they do, as well as more information about any differences use of the modules has made to their practice.   conclusions   we have developed a set of learning modules for librarians called smart searching, premised on the use of techniques undertaken by the caresearch palliative care knowledge network and flinders filters in development of search filters. we wish to emphasise that while this approach is derived from the search filter development model we use, it is a different process from the full development of a search filter using the methodology detailed in our published papers on the various search filters. it is a highly abbreviated and simplified approach, based nevertheless on the same principles of transparency, thoroughness, iteration, and minimisation of bias.   the self-paced modules are intended to help librarians (and others) test the effectiveness of their literature searches, providing evidence of search performance that can be used to improve searches, as well as evaluate and promote searching expertise. the four modules deal respectively with each of four techniques: collaboration with subject experts; use of a reference sample set; term identification through frequency analysis; and iterative testing.   the modules are provided free on the web and were launched on may 25, 2014. the resource appears to be well-used and valued. in the period from launch to the writing of this article (august 2015), web analytics show that 8,568 sessions worldwide were conducted on the modules, from 6,211 individual users in 101 countries. a user survey conducted in april 2015, while limited, provided an overall positive response from 50 survey participants across 6 countries, with 80% stating they would recommend it to a colleague. the survey also provided useful qualitative information which will guide further development of the resource.   we developed this resource because we believe that effective searching is of paramount importance and should be accorded the respect of a scientific approach. literature searching, as a key underpinning element of evidence based practice, must be able to be subjected to a scientific process of rigorous testing and falsifiability. search strategies should be documented, transparent, and reproducible. we should always ask: “what has my search missed and why?”   we will maintain and aim to improve the resource and welcome feedback, comments, and suggestions.   acknowledgements   caresearch palliative care knowledge network is funded by the australian department of health.   the author acknowledges with gratitude the work of her colleague yasmine shaheem and the contribution of the advisory group for the smart searching website.   acknowledgement is also warmly given to health libraries australia and medical director (formerly hcn) for the health informatics innovation award 2012 that supported the development of smart searching.   references   bak, g., mierzwinski-urban, m., fitzsimmons, h., morrison, a., & maden-jenkins, m. (2009). a pragmatic critical appraisal instrument for search filters: introducing the cadth cai. health information & libraries journal, 26(3), 211-219. http://dx.doi.org/10.1111/j.1471-1842.2008.00830.x   brown, l., carne, a., bywood, p., mcintyre, e., damarell, r., lawrence, m., & tieman, j. (2014). facilitating access to evidence: primary health care search filter. health information & libraries journal, 31(4), 293-302. http://dx.doi.org10.1111/hir.12087   damarell, r. a., tieman, j., sladek, r. m., & davidson, p. m. (2011). development of a heart failure filter for medline: an objective approach using evidence-based clinical practice guidelines as an alternative to hand searching. bmc medical research methodology, 11, 12. http://dx.doi.org/10.1186/1471-2288-11-12   glanville, j., bayliss, s., booth, a., dundar, y., fernandes, h., fleeman, n.d., foster, l., fraser, c., fry-smith, a., golder, s., lefebvre, c., miller, c., paisley, s., payne, l., price, a., & welch, k. (2008). so many filters, so little time: the development of a search filter appraisal checklist. journal of the medical library association, 96(4), 356-361. http://dx.doi.org/10.3163/1536-5050.96.4.011   hayman, s., & tieman, j. (2015, may 28). discovering the dementia evidence base: tools to support knowledge to action in dementia care (innovative practice). dementia (london). http://dx.doi.org/10.1177/1471301215587819   hayman, s., & tieman, j. (2015) finding the best available evidence: how can we know? ifla wlic 2015, cape town, south africa.session 141 science and technology. retrieved from: http://library.ifla.org/id/eprint/1138   higgins, j.p.t., green, s. (eds.) (2011). cochrane handbook for systematic reviews of interventions. [version 5.1.0, updated march 2011]. the cochrane collaboration. retrieved from www.cochrane-handbook.org   mcgowan, j., & sampson, m. (2005). systematic reviews need systematic searchers. journal of the medical library association, 93(1), 74-80.   mcgowan, j., sampson, m., & lefebvre, c. (2010) an evidence based checklist for the peer review of electronic search strategies (press ebc). evidence based library and information practice, 5(1), 149-54. retrieved from https://ejournals.library.ualberta.ca/index.php/eblip/article/view/7402   sladek, r., tieman, j., fazekas, b. s., abernethy, a. p., & currow, d. c. (2006). development of a subject search filter to find information relevant to palliative care in the general medical literature. journal of the medical library association, 94(4), 394-401.   tieman, j. j., lawrence, m. a., damarell, r. a., sladek, r. m., & nikolof, a. (2014). lit.search: fast tracking access to aboriginal and torres strait islander health literature. australian health review, 38(5), 541-545. http://dx.doi.org/10.1071/ah14019   tieman, j., hayman, s., & hall, c. (2015). find me the evidence: connecting the practitioner with the evidence on bereavement care. death studies, 39(5), 255-262. http://dx.doi.org/10.1080/07481187.2014.992498     appendix a smart searching user survey   question 1: country of residence   question 2: occupation librarian other information professional researcher student other (please describe):   question 3: organisation type (e.g. university, government department, hospital)   question 4: discipline/subject area (e.g. health, education, law)   question 5: how did you find out about this site? choose all that apply email list google search other search engine newsletter or journal article colleague website link other (please give details if you can):   we see you selected email list above. if you can, please supply the name of the list here   we see you selected newsletter or journal article above. if you can, please supply the title(s) here   we see you selected website link above. if you can, please supply any information about the website here   question 6: what did you find most useful about this site?   question 7: what did you find least useful about this site?   question 8: would you recommend this site to a colleague? yes no not sure   please give a reason for your answer to question 8   question 9: are there any changes you would like to see?   question 10: have you applied any of these techniques in your searching practice?   yes, all or most yes, a few no, and i am not likely to do so no, but i may do so no, but it has made me think differently about my searching   please give a reason for, and/or any comments about, your answer to question 10     question 11: do you think you would use this approach for testing or reporting on your searching strategy effectiveness (as described in the testing section of the site)? yes no not sure   please give a reason for, and/or any comments about, your answer to question 11   any other comments or suggestions?     appendix b location of visitors (top 20 countries) to the smart searching website (may 25, 2014 august 9, 2015)     country   acquisition     behaviour   sessions   % new sessions   new users   bounce rate   pages / session   avg. session duration     8,592 % of total: 100.00% (8,592) 73.29% avg for view: 72.52% (1.06%) 6,297 % of total: 101.06% (6,231) 53.83% avg for view: 53.83% (0.00%) 5.17 avg for view: 5.17 (0.00%) 00:03:29 avg for view: 00:03:29 (0.00%) 1.       australia 2,263 (26.34%) 62.17% 1,407 (22.34%) 38.22% 7.28 00:05:36 2.       united states 1,919 (22.33%) 80.88% 1,552 (24.65%) 64.72% 3.74 00:02:23 3.       united kingdom 1,000 (11.64%) 67.60% 676 (10.74%) 39.80% 6.96 00:04:23 4.       canada 828 (9.64%) 73.31% 607 (9.64%) 41.79% 6.06 00:03:19 5.       (not set) 587 (6.83%) 99.83% 586 (9.31%) 87.56% 1.10 00:00:28 6.       ireland 296 (3.45%) 69.26% 205 (3.26%) 44.59% 5.06 00:03:09 7.       new zealand 224 (2.61%) 69.20% 155 (2.46%) 39.29% 7.67 00:04:54 8.       spain 165 (1.92%) 22.42% 37 (.59%) 89.09% 1.55 00:00:51 9.       netherlands 123 (1.43%) 68.29% 84 (1.33%) 45.53% 6.72 00:03:34 10.    china 121 (1.41%) 100.00% 121 (1.92%) 88.43% .97 00:00:12 11.    japan 101 (1.18%) 98.02% 99 (1.57%) 80.20% 2.45 00:02:10 12.    sweden 85 (0.99%) 65.88% 56 (3.26%) 38.82% 7.25 00:04:58 13.    russia 84 (0.98%) 25.00% 21 (0.33%) 90.48% 1.13 00:00:14 14.    norway 82 (0.95%) 63.41% 52 (0.83%) 39.02% 8.11 00:07:42 15.    germany 79 (0.92%) 98.73% 78 (1.24%) 82.28% 1.41 00:00:39 16.    south korea 51 (0.59%) 100.00% 51 (0.81%) 84.31% 1.08 00:00:42 17.    brazil 41 (0.48%) 100.00% 41 (.65%) 85.37% 1.83 00:01:11 18.    france 41 (0.48%) 95.12% 39 (0.62%) 82.93% 1.46 00:00:25 19.    italy 41 (0.48%) 92.68% 38 (0.60%) 68.29% 4.37 00:01:13 20.    india 26 (0.30%) 96.15% 25 (0.40%) 61.54% 5.00 00:04:08     what do reference librarians do now? research article   what do reference librarians do now?   monty l. mcadoo student success librarian baron-forness library pennsylvania western university edinboro, pennsylvania, united states of america email: mmcadoo@pennwest.edu     received: 24 mar. 2022                                                              accepted: 14 july 2022      2022 mcadoo. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30129     abstract   objective – the primary purpose of this study was to better understand the nature of “reference” and reference transactions.   methods – this study looked at four years of reference transaction (rt) data recorded at a small, state-owned university.   results – the data clearly indicates that the overall number of rt continues to decline. it also reveals that, despite the use of student mentors, librarians are still involved with a majority of rt, regardless of whether or not they require the expertise of a librarian to resolve.   conclusion – continuing to be involved with rt which do not require the knowledge or training of a librarian (e.g., directional) can have a diminutive effect on the perceived role, work, and value of librarians. as such, it is suggested that these sorts of questions be addressed by student mentors or staff members. in turn, this will allow librarians to focus on those questions and activities which do require their unique knowledge and skills. along similar lines, it is also suggested that librarians explore and identify new, non-traditional ways of applying their expertise to student success initiatives and the overall academic life of the institution. with the merger of three libraries, data from this study has been and continues to be used to make informed decisions about the provision of reference services in a new, integrated library environment.     introduction   the perceived centrality of the library to higher education has a long history in the united states. in harvard college’s 1873-74 annual report, president charles eliot was one of the first to observe “the library is the heart of the university” (p. 39).  for nearly 150 years, this characterization by many of the library as the “academic hub” or “intellectual center” of the university has remained firmly entrenched in the hearts and minds of faculty and students alike.   by extension, it is not surprising that the perceived role and value of librarians to higher education have remained central. the many and varied ways librarians contribute to the information-seeking process are undeniable. thus, for many, the provision of “reference services” has continued to be the axle around which the work of librarianship revolves and is central to virtually every model of quality library service.   recently, however, the value of librarians is coming under increased scrutiny. for example, declining student enrollments and other factors have resulted in budgetary constraints. the latter is compelling librarians and campus administrators to identify and develop greater efficiencies, especially with regard to personnel costs (cottrell & bell, 2015). bandyopadhyay and boyd-byrnes (2016) make a similar observation stating “budget shortfalls and decreased demands for reference services have forced academic librarians and library administrators to rethink and redesign the reference service models to provide library users with a meaningful and efficient service to cater to their information needs” (p. 609).   as budgetary and other expectations for greater accountability grow, librarians are not alone in being asked to supply data which demonstrates the impact their work has on things such as graduation rates, retention, and various other measures of student and institutional success (goss, 2022). to that end, quantitative measures alone are no longer sufficient. that is, it is no longer enough to simply equate x number of reference transactions (rt) with value.   this study examines transaction data recorded over a four-year period in an attempt to better understand librarians’ roles in the provision of reference services.   literature review   for more than a century, much has been written about the nature of reference service. historically, as suggested above, helping to ensure the information needs of an individual are met has typically been seen as being a key (if not the primary) role of academic librarians. a cross-section of examples include:   ·        1876: samuel green, often referred to as the “father of reference,” observed among other things that “a librarian should be as unwilling to allow an inquirer to leave the library with his question unanswered as a shop-keeper is to have a customer go out of his store without making a purchase” (p. 79). ·        1994: summerhill defined “reference services” as simply “a facilitation of the connection between researchers and the information they desire or need” (p. 74).  ·        2008: reference and user services association (rusa), a division of the american library association (ala), articulated “reference work” and “reference transactions” as being “information consultations in which library staff recommend, interpret, evaluate, and/or use information resources to help others to meet particular information needs” (rusa). ·        2016: sosulski and tyckoson note that, “whether we call it reference or research or just plain help, librarians provide personal service to make sure that each person finds the information that best meets his or her needs” (p. 88).   the evolution of reference librarianship   whether via a virtual or a physical presence, the underlying notion of reference work has been that an individual approaches a librarian for assistance because the librarian possesses some expertise or knowledge that the individual does not. that is, it is believed a librarian has the specialized training needed to identify, locate, access, and retrieve information. historically, even if individuals were able to find information on their own, a librarian has typically continued to be perceived as being able to do so more effectively and more efficiently. as a direct result, the “reference desk” (i.e., where librarians have typically provided reference service) has long served as the focal point of the academic library. as stevens (2013) observes, the reference desk remains “a dominant symbol of the profession but also as an actual place where librarians “do” reference” (p. 205).   the library has long-served as centralized location for storing physical information (e.g., print, microfilm). until fairly recently, individuals would typically need to go to a library to get the information they needed because that was the only place the information could be accessed. librarians were trained and responsible for acquiring, organizing, and providing access to the library’s collection. because most individuals didn’t have access to or were unfamiliar with how to navigate the world of information, librarians served as essentially “mediators” and, as such, were a necessary and inextricable part of addressing information needs.   with the advent of technology, things began to change. in the last quarter of the twentieth century, librarians still served as mediators. however, this role was increasingly linked with helping individuals to access and navigate various technologies. for example, to access the desired information, individuals may have needed a librarian from this period to load a cd-rom or access an electronic information service (e.g., dialog). as resources became more user-friendly, however, librarians often found themselves less involved with the actual search and more involved with showing individuals how to navigate the interfaces of resources and software applications on their own.   in response, today, the look, feel, and function of reference desks continues to change. for example, many libraries now provide some form of virtual “desk” (e.g., chat, social media, email) to meet the demands of a growing number of remote learners. many now staff the latter or the physical reference desk with student workers, freeing-up librarians to perform “higher level” tasks (clark et al., 2020; keyes & dworak, 2017).   even so, the physical reference desk still plays a central role in many libraries. the continued significance attached to the physical reference desk is perhaps best illustrated by a 2008 study by banks and pracht which found that only 2 libraries out of 101 respondents did not have a physical reference desk. while speaking of the ongoing efficacy of the physical reference desk, freides’ (1983) observation can be applied to virtual desks as well:   the reference desk works best for directional questions and requests for specific factual information. it is not well designed for dealing with questions requiring interpretation or exploration, including what is probably the most common, and most important, type of reference inquiry in academic libraries, the open-ended, ‘information about’ request for assistance with term papers and other classroom assignments. (p. 467)   challenges to the librarian-as-mediator model   as noted above, providing research or “information assistance” has been a longstanding role of academic librarians. whether virtual or in-person, the predominant model of and justification for reference service typically continues to reflect the assumption of the need for a librarian-as-mediator in the information-seeking process.   but questions about whether or not this is still a valid assumption have been around for years (ford, 1986). numerous researchers note a decline in the number of questions actually requiring the expertise of a librarian (carlson, 2007; ewing & hauptman, 1995; maloney & kemp, 2015; stevens, 2013). in talking about the change in the type of questions to which librarians respond, bell (2007) states that “doling out software help and fixing paper jams, two frequent reference desk tasks these days squanders the talent and expertise of skilled librarians” (p. 1).   however, as miles (2013) observes, “the biggest problem with traditional reference service has to do not with the model, but with changes in the world around the reference librarian” (p. 321). that is, the value of one-to-one interactions with a librarian is not in dispute. but, because of the breadth and depth of changes to the information landscape, the nature of reference services, the emphasis we place on such, and the roles and responsibilities of reference librarians must necessarily change as well.   to better understand both the causes and nature of this shift, a number of changes resulting from today’s information rich, technology-laden world need to be acknowledged including the following.   researcher independence   at least part of the decline in and changing nature of rt can be attributed to researcher independence. as lewis (1995) observes, “most of the tools we currently use in the library were invented by librarians. the catalog, indexes, and abstracts, even in computerized versions, were developed by librarians. our new tools though come from outside the library world” (p. 12). today’s search interfaces are increasingly user-friendly and internet search algorithms are increasingly sophisticated. as a result, a growing number of individuals no longer need a librarian to explain things such as a “controlled vocabulary” or “proximity operators” to find the information they seek.   declining number of rt   over the past decade or so, some remain optimistic that virtual transactions will slow the decline or even result in increases in reference transactions. however, the accuracy, extent, and duration of these claims remain under study. for now, reports of consistent, continued, and often large decreases in reference transactions overall continue to emerge. among others (banks & pracht, 2008; carlson, 2007; stevens, 2013; weber & bowron, 2019), a study conducted by the american library association (2008) indicated a 50% decline in reference transactions between 1994 and 2008.   nature of questions   as with the declining number of transactions, there are a growing number of studies which find that the majority of questions typically being asked can be answered by staff members or even student workers. this is not a new observation. for example, more than 40 years ago, well before the “information age”, saint clair et al. (1977) reported that well-trained non-professionals could handle “at least 62.l percent of all questions posed at the reference desk” (p. 151). more recently, ryan’s (2008) study of nearly 7,000 transactions suggested that “89% could likely be answered by non-librarians” (p. 389). a study by lenkart and yu (2017) of more than 66,000 rt revealed the majority were directional (30.9%), with just under 9% characterized as about research assistance and ready-reference. in talking about claims that reference transactions are more complex, stevens (2013) says they are unsubstantiated and points to several studies which indicate that “only a small percentage [of research questions] are complex enough to require the help of a subject specialist” (p. 206).   response quality   even before the onset of technology, the quality of service provided by reference librarians had been questioned. ross (1998), for example, concluded that “the success rate for information service hovers in the 50 to 60 percent range” (p. 151). smart technologies (e.g., alexa, siri) and a growing number of online sites and services are often able to provide quicker responses of the same or even better quality than a librarian. shachaf (2009), for example, found that the quality of responses generated by wikipedia reference desk were generally comparable to those of library reference services. the risk of diminished response quality is potentially even greater for libraries which use students or untrained professionals to staff the reference desk.   reference service at the research site   in january, 2016, librarians at edinboro university of pa (eu) discontinued staffing the physical reference desk. instead, the physical desk was staffed with “student mentors.” there were three main reasons this change was affected:   decreased availability of librarians librarians at eu have faculty status. among other things, this means they are not required to work when classes are not in session (e.g., spring break). since 2000, the number of librarians declined from 11 to 8 (2016) to 6 (2019). because of the latter and based on identified times of need, librarians are not scheduled to work evenings or weekends. staffing the reference desk with student mentors was seen as a way to help to ensure broader, more continuous assistance throughout the year. approachability it was believed students might feel more comfortable approaching someone who was more “like them.” increased professional opportunities for librarians the change was seen as a way of freeing-up librarians, using the time they formerly would have spent staffing the physical desk to perform higher-level tasks requiring their unique skills and expertise.   during the 2016 to 2019 study period, student mentors staffed the physical desk a total of 84 hours per week during fall and spring semesters. some of those hours were scheduled with an on-call librarian (i.e., 27 hours per week) but the majority were scheduled when no librarian was available for the reasons outlined above. student mentors login to chat at the start of their shift under a generic, joint “reference desk” account. rt can be recorded automatically (chat) or manually as outlined below. student mentors are expected to forward all “research” and other questions requiring a librarian’s expertise to the on-call librarian. when no librarian is available, they submit questions to a common queue. when on-call, librarians are responsible for monitoring the queue and otherwise making themselves available for those needing assistance in-person or virtually.   research questions   historically, transaction log data has been used primarily by eu for determining periods of high usage in order to make informed decisions about providing reference services. to become more cost-effective and because of a significant decline in the number of librarians and rt, maximizing the use of librarians’ time and effort is crucial. with the merger of three libraries, this has become even more important and concerns over optimizing the use of librarians’ time and expertise remain central.   to that end, this study examines rt recorded by librarians compared to those recorded by student mentors along four dimensions:   patron type contact type time spent overall “information search” query type overall “information search”   data from this study were used and continues to be used to inform decisions and to make recommendations about the configuration of refanalytics (software used to log rt as described below) and the provision of reference services locally and in the new, integrated library environment.   data collection and analysis   setting   eu is one of 14 state-owned universities which comprise the pennsylvania state system of higher education (passhe). in fall of 2022, eu merged with two other passhe institutions to form pennsylvania western university (pennwest university).   data collection   as part of the evolution of reference services, eu librarians discontinued the use of a paper transaction log. starting in 2016, they began recording rt electronically via springshare’s refanalytics aspect of the libanswers platform.   both librarians and student mentors are responsible for logging transactions. refanalytics allows individuals to log transactions manually. transactions occurring via the integrated chat feature can be logged automatically. when no one is available, individuals needing assistance are able to submit their questions via email, text message, or a link on the library’s homepage. all submissions are automatically stored in a common queue and are responded to by the next available librarian. while monitoring the queue is primarily the responsibility of the on-call librarian, the queue can be checked at any time by any librarian on-campus or off-site. student mentors only to respond to items in the queue which are not research-oriented (e.g., directional, hours).   data challenges   in attempting to better understand the role of librarians in the provision of reference services, the researcher was limited to the available data and the way in which it was recorded. however, prior to data analysis, a cursory examination revealed a number of concerns associated with the data. the primary impact(s) of each concern on recorded rt is reflected parenthetically below as quantity (number of rt), type (type of rt), and time (time spent resolving rt).   inconsistent coding (quantity, type) the same query was often coded multiple ways. for example, despite having a designated “query type” (qt) of “printing issue,” examples of print-related rt were found in seven of the nine possible qt. incomplete data (type, time) many data fields were left blank making it difficult or even impossible to determine the actual subject of the rt. incorrect qt (type) some qt were applied incorrectly. for example, “looking for a dsm5 book in reference” was recorded as “archives” even though the query is clearly not related to archives. multiplicity (quantity, type) sometimes two or more unrelated questions posed by a single individual were recorded as a single transaction (e.g., “i’m having trouble printing and need a book on feminist art.”). this resulted in slightly reducing the overall number of actual rt. duplicate entries (quantity) duplicate entries typically occurred for one of two reasons. first, sometimes referrals from student mentors were logged by both the student and the librarian. other times, when a chat got disconnected, a new, distinct rt was sometimes created instead of the transaction being entered as a continuation of the initial rt. duplicate entries resulted in a slight inflation of the overall number of rt.   data homogenization   while chat transactions are recorded automatically, all other questions and responses must be entered manually. a time date stamp and the responder (i.e., student mentor or individual librarian) are supplied automatically based on login. refanalytics provides drop-down menu options for entering additional transaction data along three criteria described in table 1. the latter also provides the emergent codes used to homogenize the data for this study (described below).     table 1 refanalytics’ drop-down menu options’ homogenization criterion original options coded options patron type community, faculty, staff, student, n/a community, faculty, staff, student, unknown query type archives, catalog search, citations, computer issue, directional, other, printing issue, reference, remote login, scanning academic, access, building, circulation, citation, computer, directional, information search, instruction, other, printing time spent 1-5 minutes, 5-10, 10-15, 15-20, >20 1-5 minutes, 5-10, 10-15, 15-30, 30-60                     overall   in an attempt to generate a more consistent dataset which more closely reflected the actual nature and number of recorded rt, the researcher reviewed every rt and effected the following before analyzing the data.   for rt fields lacking data, sometimes it was impossible to know what transpired (e.g., time spent) and the field was left blank. however, for a significant number of entries, fields could be assigned by examining the actual query or other data. for example, the query “can you tell me where the writing center is located?” with no recorded qt was assigned a qt of “directional.” inconsistently applied and incorrectly assigned qt were corrected whenever possible. each aspect of multi-part qt was converted into an individual rt and coded accordingly. all rt for this study were then recoded and regrouped using the options noted in table 1.   information search (is)   in order to better understand the nature of those rt recorded with a qt of “information search” or is (i.e., those rt presumed to require a librarian), a similar review of both the actual questions recorded and the corresponding responses was conducted. the latter resulted in identifying seven common themes: research, circulation, access, assignment help (non-research), formatting, other, blank/unknown. as with many other rt, the lack of specificity in the way some rt were recorded sometimes made it difficult to determine which is transactions were “research” and which were not. for example, does a question recorded as “photography books” reflect someone looking for that section of the collection to browse or someone doing actual research on the topic who wanted multiple books? for the purposes of this study, when the nature of an is transaction involving searching was not clear, the quantity of sources requested was used to assign a theme. questions and answers which clearly indicated more than one source was desired were coded as “research” and those for which a single source was sought were coded as “circulation.”   dataset   the result of the above process was a dataset of 5,194 rt recorded from january 1, 2016 through december 31, 2019. due to covid and a building renovation project, data from march, 2020 until august, 2021 was purposely excluded to help ensure a more consistent dataset and analysis.   results   as a record of the work being performed by reference librarians, the library’s transaction log was examined along various dimensions, especially the nature of transactions recorded by student mentors versus those recorded by librarians.   total recorded transactions   table 2 provides a historical summary of transactions recorded from 1999-2019.   observations   except for slight increases from 2000 to 2001 and 2016 to 2017 and another from 2007 to 2009, the overall downward trend in recorded transactions over the past 20 years is consistent. since 1999, rt have declined more than 90%. during the 4-year study period, rt declined by approximately 26%.     table 2 recorded reference transactions (rt): 1999-2019 year recorded reference transactions (rt) 1999 11,593 2000 10,072 2001 10,223 2002 8,407 2003 7,485 2004 3,768 2005 1,705 2006 1,635 2007 1,249 2008 3,800 2009 8,382 2010 6,909 2011 5,533 2012 3,597 2013 3,505 2014 3,230 2015 1,673 2016 1,441 2017 1,447 2018 1,244 2019 1,062     patron type   table 3 summarizes the five patron types which could be assigned to a transaction.     table 3 patron type by responder a patron type (pt) student mentors student % of pt librarians librarian % of pt pt % of overall community 22 12.9 149 87.1 3.4 faculty 28 10.2 247 89.8 5.5 staff 5 15.6 27 84.4 < 1 student 1,507 33.7 2,966 66.3 88.9 unknown 31 37.3 52 62.7 1.6 total 1,593 31.6 3,441 68.4 5,034/100 a for 160 recorded transactions (40 by student mentors, 120 by librarians), no patron type was recorded. these transactions were not included as part of the above calculations or observations below.     observations   the overwhelming majority of rt (88.9%) were interactions with students. the second highest category (i.e., faculty) only represented just over 5% of all rt. librarians were involved with almost 90% of faculty rt and nearly two-thirds of student rt.   contact type   contact type (ct) refers to the form of interaction which took place and is summarized in table 4.     table 4 contact type by responder a contact type (ct) student mentors student % of ct librarians librarian % of ct ct % of overall in-person 1,488 36.6 2,577 63.4 78.3 all others b 143 12.7 986 87.3 21.7 total 1,631 31.4 3,563 68.6 5,194/100 a student mentor and librarian percentages differ from other tables because the latter are missing records. b includes chat, email, phone, and no determinable contact type.     observations   over 63% of in-person transactions and more than 87% of all other transactions were recorded by a librarian. despite a growing number of remote learners and online courses and programs, more than three-quarters of all rt took place in-person.   time spent   tables 5 and 6 present data about different aspects of time spent on transactions.   time spent: overall   the overall distribution of time spent recorded by responder is summarized in table 5.     table 5 overall time spent by responder a   time spent (ts) student mentors student % of ts librarians librarian % of ts ts % of overall   1-5 minutes 1,317 46.9 1,492 53.1 55.9   5-10 minutes 219 17.1 1,059 82.9 25.4   10-15 minutes 36 9.2 356 90.8 7.8   15-30 minutes 7 4.2 158 95.8 3.2   30-60 minutes 11 2.9 374 97.1 7.7   total 1,590 31.6 3,439 68.4 5,029/100   a for 165 recorded transactions (41 by student mentors, 124 by librarians), no time spent was recorded. these transactions were not included as part of the above calculations or observations below.     observations   55.9% of all transactions were resolved in < 5 minutes. 82.8% of rt recorded by student mentors were resolved in < 5 minutes. 43.4% of all rt recorded by librarians were resolved in < 5 minutes. 81.3% of all transactions were resolved in < 10 minutes. 96.6% of rt recorded by student mentors were resolved in < 10 minutes. 74.2% of all rt recorded by librarians were resolved in < 10 minutes. 89.1% of all transactions were resolved in < 15 minutes. 98.9% of rt recorded by student mentors were resolved in < 15 minutes. 84.5% of all rt recorded by librarians were resolved in < 15 minutes.   time spent: information search   table 6 summarizes time spent for transactions recorded as “information search.”     table 6 time spent by responder on information search time spent (ts) student mentors student % of ts librarians librarian % of ts ts % overall 1-5 minutes 45 60.8 321 20.7 22.6 5-10 minutes 18 24.3 541 35.0 34.5 10-15 minutes 7 9.5 243 15.7 15.4 15-30 minutes 2 2.7 128 8.3 8.0 30-60 minutes 2 < 1 309 20.0 19.2 none entered a 0 0 5 < 1 < 1 total 74 4.6 1,547 95.4 1,621/100 a because of the small number involved, the five transactions for which no data was entered for time spent were excluded from subsequent calculations.     observations   22.6% of all is transactions were resolved in < 5 minutes. 60.1% of is rt recorded by student mentors were resolved in < 5 minutes. 20.7% of all is rt recorded by librarians were resolved in < 5 minutes. 57.1% of all is transactions were resolved in < 10 minutes. 85.1% of is rt recorded by student mentors were resolved in < 10 minutes. 55.7% of all is rt recorded by librarians were resolved in < 5 minutes. 72.5% of all is transactions were resolved in < 15 minutes. 94.6% of is rt recorded by student mentors were resolved in < 15 minutes. 71.4% of all is rt recorded by librarians were resolved in < 15 minutes.   query type   query type (qt) reflects the nature of a given transaction. table 7 summarizes the distribution of the 11 possible qt which could be assigned to transactions.     table 7 query type by responder a query type (qt) student mentors student % of qt librarians librarian % of qt   qt % of overall academic 219 79.1 58 20.9   5.5 access 3 3.9 74 96.1   1.5 building 206 50.9 199 49.1   8.0 circulation 37 94.9 2 5.1   < 1 citation 14 16.5 71 83.5   1.7 computer 162 26.5 449 73.5   12.1 directional 291 56.4 225 43.6   10.3 information search 74 4.6 1,547 95.4   32.2 instruction 62 51.2 59 48.8   2.4 printing 519 40.7 755 59.3   25.3 other 1 16.7 5 83.3   < 1 total 1,588 31.6 3,444 68.4   5,032/100   a for 162 recorded transactions (43 by student mentors, 119 by librarians), no query type was recorded. these transactions were not included as part of the above calculations or observations below.     observations   two qt, “information search” and “printing”, accounted for 57.5% of all rt. librarians responded to more than 68% of all questions, regardless of qt.   information search (is)   just under a third of all rt (32.2%) were recorded as is. virtually all is (95.4%) were responded to by a librarian. an average of 405.25 is rt were recorded for each year of this study, representing an average of 7.8 per week or just over one per day.   characteristics of information search (is)   overall   table 8 summarizes the distribution of the seven characteristics assigned to is rt.     table 8 characteristics of is rt   identified theme # % research 856 52.8 circulation 398 24.6 access 141 8.7 assignment help (non-research) 82 5.1 formatting 66 4.1 other 44 2.7 blank or unknown 34 2.1 totals 1,621 100.0     observations   52.8% of all is rt were associated with research. 24.6% of all is rt were related to circulation (including single source and ready-reference sorts of questions).   is theme by responder   is themes were also examined in terms of responder as shown in table 9.     table 9 is theme by responder is theme student mentors student % of theme librarians librarian % of theme theme % overall research 25 2.9 831 97.1 52.8 circulation 26 6.5 372 93.5 24.6 access 8 5.7 133 94.3 8.7 assignment help (non-research) 0 0.0 82 100.0 5.1 formatting 2 3.0 64 97.0 4.1 other 4 9.1 40 90.9 2.7 blank or unknown 9 26.5 25 73.5 2.1 total 74 4.6 1,547 95.4 1,621/100     observations   librarians responded to more than 95% of all rt recorded as “information search.” librarians responded to more than 97% of all rt characterized as “research.” just over 90% of all “other” is rt were recorded by a librarian. librarians recorded nearly three-quarters of “blank or unknown” transactions.   discussion   total recorded rt   increases   the reason for the slight increases in overall rt recorded from 2000 to 2001 and 2016 to 2017 is unclear. however, in 2008, a new university president was inaugurated. at least in part, a new admissions initiative he implemented may explain both the increases and larger numbers from 2007 through 2011 as well as the gradual decline 2009 through 2014. it is suggested that the new policy resulted in a larger student population which, in turn, translated into a larger number of rt. conversely, when a new president arrived in 2010, the policy was revised. as a result, the decline from 2009 onward may simply reflect a smaller, more stable student population resulting from the new policy and a combination of both the graduation of and the attrition of students admitted under the previous one.   decreases   no single factor can explain the clear and dramatic decline in recorded transactions over the past 20 years. the reasons for such are likely many and varied. certainly, contributing to the decline is that the student population decreased from 7,079 or 6,251 full-time equivalent (fte) students in 1999 to 4,646 (3,858 fte) in 2019, which was a decrease of roughly one-third. another is the widespread growth of web-based forms of information. the proliferation of “smart devices” enabling individuals to search for information from virtually anywhere at any time, as well as increasingly user-friendly interfaces and more sophisticated search algorithms have certainly played a role as well. the role (if any) of the decline in the number of librarians and the use of student mentors is unclear. regardless of the reasons, a significant decline in transactions is undeniable.   patron type and contact type by responder   patron type   it is not surprising that the largest group of individuals seeking assistance was students. what is surprising is that, despite not being present at the physical desk, librarians were still involved with two-thirds of all student transactions. even though it was surmised that students would be more likely to approach a peer than a librarian, the data suggest this might not be the case.   contact type   with a growing number of off-site learners, online courses and programs, and remote access to library resources, it was surprising to learn that over three-quarters of all rt still occurred in-person. it was even more surprising that just over 63% of all in-person transactions were recorded by a librarian. both measures would seem to indicate that individuals still prefer a face-to-face interaction to a virtual one.   in both cases, particularly given that librarians no longer staff the physical desk, it is convenient to believe that students needing help recognize the value of librarians and consciously seek their assistance over that of their peers. however, while there may be some merit to this assumption, there are any of a number of other factors which might help to explain the significantly larger number of in-person interactions that librarians recorded. one of these is simple geography. two librarians’ offices are essentially adjacent to the reference desk and another librarian is in close proximity. this results in a lot of walk-ins when the student mentor is helping someone else or otherwise unavailable. the proximity of librarians also makes it easy for student mentors to refer patrons. at the time, there was no way to record interactions initiated by a student mentor and then transferred to a librarian. such transactions were generally recorded by the librarian.   two other factors which may contribute to the discrepancy revolve around the librarians themselves. first, librarians are more aware of the need for data collection and its various uses (e.g., scheduling). therefore, they may be more diligent than student mentors in terms of recording transactions. in addition, despite employing student mentors, at least two librarians at the time continued to advocate that all questions, regardless of the level, should be directed to and addressed by a librarian. as a result, those librarians may have recorded more transactions than the student mentors with whom they were paired.   time spent by responder: overall   in addition to research questions, time spent addressing an individual’s need is often seen as another measure of librarians’ worth. the underlying assumption is that questions which can be answered by students and staff typically don’t take as much time as those questions involving the expertise of a librarian. for example, informing someone of when the library’s open over the weekend takes far less time and expertise than helping someone working on a research paper to identify suitable resources and develop a research strategy for retrieving source material.   assuming the above assumption is valid, one would expect that most questions answered by student mentors (e.g., directional) would involve less time. the data from this study support this claim. nearly 83% of all rt recorded by student mentors were addressed in under 5 minutes and virtually all (i.e., 98.9%) were addressed in under 15 minutes.   though to a lesser extent, the time librarians spent answering questions is similar. more than 43% of all questions addressed by a librarian were resolved in under 5 minutes while more than 74% and nearly 85% were resolved in under 10 and under 15 minutes respectively. while librarians tended to spend more time answering a given question, the clear majority were addressed relatively quickly.   time spent by responder on information search   given the above, time spent addressing is questions was examined in greater detail. as noted earlier, such questions are often seen as being more involved and necessitating the expertise of a librarian. in turn, the implication is that they will involve more time to resolve.   whether or not the data from this study support this assumption or not is unclear. nearly one-fifth of all is questions were resolved by a librarian in under five minutes. nearly three-quarters were resolved in under 15 minutes. admittedly, there is no “magic number” or direct, inherent correlation between need and time spent. too, as observed above, many rt recorded as is are clearly “ready-reference,” quick answer sorts of questions. even so, it is difficult to argue that more in-depth needs and concerns typically associated with “research questions” and the need for a librarian can be sufficiently addressed in 15 minutes or less.   query type by responder   overall   even a cursory review of the data reveals that many of the transactions which occur can be addressed without the intervention of a librarian. the latter was one of the main arguments for hiring student mentors to staff the reference desk. thus, despite the fact that many of the questions clearly did not require the expertise of a librarian, it is somewhat surprising that librarians were involved with nearly 70% of all recorded rt.   information search (is)   of all the available qt, “information search” (is) represents the type of transaction typically seen as “professional” and as such requiring the training and expertise of a librarian. as maloney and kemp (2015) observe, such questions are seen as more complex in that they require a “deeper collection knowledge and more time to answer” (p. 961). that said, it is discouraging to learn that less than a third of all rt were actually characterized as is. even if all recorded is transactions required a more in-depth consultation with a librarian, the number of recorded is transactions only averaged slightly more than one per day of the period under study. this suggests that either not many students are conducting research or that they do not have a need for a librarian when they do.   at the same time, it is encouraging that the overwhelming majority of all recorded is transactions were addressed by a librarian. this would seem to reinforce the traditional role of librarians in identifying possible sources of information, developing search strategies, and providing other forms of information-seeking assistance.   characteristics of information search   theme and responder   as outlined above, the qt of is is supposed to be assigned to questions which are “professional,” requiring a librarian’s training and expertise. for the most part, these questions revolve around some aspect of the research process. despite the various challenges associated with recording transactions, it was somewhat surprising that only a little more than half (52.8%) of all transactions coded as “information search” actually revolved around research. at the same time, it was encouraging to note that just over 97% of “research” questions were addressed by a librarian. this suggests that student mentors have a good understanding of which questions should be directed to a librarian.   however, the lingering challenges moving forward are threefold. first, how are “professional questions” to be determined? where does the line get drawn between a student mentor answering a question and making a referral to a librarian? in terms of research transactions, that most “know one when they see one” is not sufficient. most seem to understand what a non-research question is. for example, asking the location of the restroom or how late the library will be open are questions student mentors are capable of addressing. despite some librarians’ aversion to such, most student mentors can be trained to answer many traditional ready-reference sorts of questions as well as to conduct basic searches for books and other sources of information. even so, despite the overwhelming majority of “research” questions being referred to a librarian, roughly 3% were still addressed by student mentors.   the reverse is even more problematic. that is, when a librarian is asked a “non-professional” question (e.g., how much does it cost to make a photocopy?), should these questions be referred to a student mentor? common sense and the strong service orientation of most librarians suggest not. and yet, it is clear that librarians are often asked and address these questions even though they do not require a librarian’s expertise or experience.   the third challenge to emerge from the data revolves around questions which fall outside the professional versus non-professional dichotomy. some examples from the data include being asked to proofread assignments, formatting citations and other aspects of an assignment, assisting with completing a graduate school application, and fixing problems with remote access to resources. admittedly, many librarians are fully capable of addressing these sorts of questions. the question is, though, should they do so and, if so, to what degree?   clearly articulating and otherwise distinguishing between professional and non-professional questions and who should address them remain key challenges moving forward.   conclusion   this study’s title is an intentional double-entendre. one aspect is descriptive. although limited to a single site, this study provides insight into the nature of rt and the “what” of reference services. historically, the longstanding tradition of providing “research assistance” is an inextricable element of most librarians’ identities. and yet, the data from this study more than suggest that not only are the number of transactions continuing to decline but the number actually requiring the expertise of a librarian are declining as well.   this brings-up the second, contemplative aspect of the title. looking ahead, what do we do now that our role as mediators in the information-seeking process continues to diminish? to answer this question we must determine 1) the best way to provide “reference service” and 2) other ways we can employ our training and expertise to contribute to the academic life of the campus.   moving forward, librarians must begin by taking a serious, candid look at “reference.” it is a given that librarians will continue to play a role in meeting individuals’ information needs. as buss (2016) observes, “a more reasonable approach is to evolve reference services, not to abandon them... more appropriately, librarians should adjust their hours, service points, and reference philosophies to meet the needs of specific constituents” (p. 268). for example, perhaps the role of ready-reference has forever been relegated to the online world. but, if so, this creates an opportunity for librarians to focus even greater time and effort on research questions.   however, it seems clear that we cannot continue sitting at a physical desk, waiting for someone to ask a question. as stevens (2013) points out, the physical desk is based on the reliance on printed information and that we must “dislodge our understanding of reference from the desk” (p. 205). at the very least, this approach is not cost-effective. worse, as friedes (1983) observed nearly 30 years ago and as reinforced by the data from this study, staffing a physical desk may actually be harming the profession in that librarians’ value risks being equated more with “tech support” or “general information” than with our expertise as information professionals.   to a large degree, the same can be said for virtual reference services (e.g., email, chat, social media). in part, the latter are needed to meet the growing population of remote learners, and yet, we need to be cautious that our virtual “desk” is not just a physical desk in disguise. that is, we have to ask if we have added a service which truly requires a librarian or have simply added another layer or responsibility which distracts librarians from engaging in and performing other professional activities.   adopting an appointment-based model is one alternative to explore. as summerhill (1994) notes, “an important benefit of a consultation model is that the level of service would be clearly and directly connected to staffing” (p. 82). inherent in scheduling an appointment is the assumption that an individual’s needs require the expertise of a librarian (i.e., otherwise, why schedule an appointment?). the model offers several benefits including:   walk-ins suppose the availability of a librarian and, necessarily, involve a lot of idle time. scheduling appointments ahead of time addresses these problems. librarians have the ability to prepare materials, examples, and so on. appointments provide an environment with limited distractions during which more time and focus can be devoted to the need in-depth. individuals may feel more comfortable discussing sensitive or controversial topics they would not discuss in an open, public space. not unlike doctor’s appointments, scheduling an appointment with a librarian reinforces the professional nature of the interaction. meeting one-to-one builds rapport which increases the likelihood of future interactions.   in addition to adopting a new perspective on the nature of reference service and how to provide it, librarians must also explore new ways of applying our education and expertise. accessing, storing, and organizing information remain central. but, as houston (2016) suggests, the skills of reference librarians “go beyond those included in traditional reference training: these include consulting and advising, teaching, interpreting, advocating, programming, and the ability to analyze the user experience and engage in design thinking” (p. 187). johnson (2018) also provides a good overview of some of the existing and emergent ways we can become involved including data curation, scholarly communication, and textbook cost reduction initiatives.   to truly be successful in this evolution will likely entail a less library-centric and more information-focused perspective. the fact is that, in an age of growing expectations for accountability, we need to identify new ways of making contributions to both the individual and the institution. in a 2012 study, detmering and sproles found that responsibilities for many entry-level library positions still reflected “traditional duties of reference, instruction, liaison, and collection development” (p. 553). however, they also observed that many responsibilities were being listed to accommodate emergent needs including duties such as instructional designer, marketing consultant, and web developer.   along those lines, some of the many opportunities to which librarians might apply their unique skills sets and expertise include the following:   using our organizational skills and technologies to create and maintain searchable databases of materials such as: approved course and program proposals for use by faculty and academic affairs personnel involved in curriculum development, accreditation, and assessment artwork and other special collections on-campus student newspapers, yearbooks, and scholarship (e.g., theses, capstone projects) participating in the development and (co-)teaching of first-year experience courses, research courses, and courses reflecting our interests and expertise with respect to information (e.g., information ethics, intellectual property, misand disinformation) serving on the campus’ institutional research board (irb) and offering to provide research assistance to faculty and students conducting research advising and otherwise emphasizing and engaging in our mentorship role with students forming collaborative, working partnerships with the campus’ writing center, tutoring services, and other student success-oriented departments and activities   unfortunately, since 2019, a number of local factors have limited librarians’ abilities to explore any of these opportunities in-depth. first, the number of librarians has been reduced from six (2019) to just three. this necessitated a realignment of duties and responsibilities. further, in 2020, it was announced that eu was to be merged with two sister universities in 2022. as a result, much of librarians’ “free time” over the past couple of years has been devoted to integrating three library operations into one.   still, there have been some successes. the author, for example, assumed the role of first-year experience coordinator and has conducted several internal assessments to assist with the integration process. this study is but one of the tangible outcomes of the latter as we begin to operate in the new, integrated environment.   in short, as data from this and other studies seems to suggest, the need for professional assistance will not disappear. in that respect, the traditional reference model is not the problem. rather, it is the changing nature of information and higher education which is forcing us to re-examine the provision of reference services and new ways of employing our skill sets. the degree to which we are successful with the latter will play a key role in determining our future and the value of the work we do, both perceived and actual. librarians should continue to engage with students in the research process. however, if librarians are to avoid increasing levels of marginalization to the point of irrelevance, it is imperative that we focus more time and effort on the unique types and quality of assistance only we can provide and that we identify new and better ways of doing so.   references   american library association. (2008). academic libraries in the united states statistical trends. ala office for research & statistics. http://www.ala.org/research/librarystats/academic/academiclibraries                                                                                                                                                                                      bandyopadhyay, a., & boyd-byrnes, m. k. (2016). is the need for mediated reference service in academic libraries fading away in the digital environment? reference services review, 44(4), 596-626. https://doi.org/10.1108/rsr-02-2016-0012   banks, j., & pracht, c. (2008). reference desk staffing trends. reference & user services quarterly, 48(1), 54-59. https://doi.org/10.5860/rusq.48n1.54   bell, s. j. (2007). who needs a reference desk? library issues: briefings for faculty and administrators, 27(6). http://dx.doi.org/10.34944/dspace/125   buss, s. p. (2016). do we still need reference services in the age of google and wikipedia? the reference librarian, 57(4), 265–271. https://doi.org/10.1080/02763877.2015.1134377   carlson, s. (2007, april 20). are reference desks dying out? the chronicle of higher education, 53(33), a37–a39. https://www.chronicle.com/article/are-reference-desks-dying-out/   clark, k. t., gabriel, h. m., & borysewicz, k. (2020). development, implementation and importance of an undergraduate peer research consultant program at the university of north dakota’s chester fritz library. reference services review, 48(4), 579–600. https://doi.org/10.1108/rsr-05-2020-0036   cottrell, t. l., & bell, b. (2015). library savings through student labor. the bottom line, 28(3), 82-86. https://doi.org/10.1108/bl-05-2015-0006   detmering, r., & sproles, c. (2012). forget the desk job: current roles and responsibilities in entry-level reference job advertisements. college & research libraries, 73(6), 543–555. https://doi.org/10.5860/crl-304   eliot, c. (n.d.). harvard-radcliffe online historical reference shelf. forty-ninth annual report of the president of harvard college: 1873–1874. https://iiif.lib.harvard.edu/manifests/view/drs:427074985$41i ewing, k., & hauptman, r. (1995). is traditional reference service obsolete? the journal of academic librarianship, 21(1), 3-6. https://doi.org/10.1016/0099-1333(95)90144-2   ford, b. j. (1986). reference beyond (and without) the reference desk. college & research libraries 47(5), 491-94. https://doi.org/10.5860/crl_47_05_491   freides, t. (1983). current trends in academic libraries. library trends, 31(3), 457-474.   goss, h. (2022). student learning outcomes assessment in higher education and in academic libraries: a review of the literature. the journal of academic librarianship, 48(2), 1-9. https://doi.org/10.1016/j.acalib.2021.102485   green, s. s. (1876, november 30). personal relations between librarians and readers. american library journal, 1(2-3), 74-81.   houston, a. (2016). what’s in a name? toward a new definition of reference. reference & user services quarterly, 55(3), 186–188. https://doi.org/10.5860/rusq.55n3.186   johnson, a. m. (2018). connections, conversations, and visibility: how the work of academic reference and liaison librarians is evolving. reference & user services quarterly, 58(2), 91-102. https://doi.org/10.5860/rusq.58.2.6929   keyes, k., & dworak, e. (2017). staffing chat reference with undergraduate student assistants at an academic library: a standards-based assessment. the journal of academic librarianship, 43(6), 469-478. https://doi.org/10.1016/j.acalib.2017.09.001   lenkart, j., & yu, j. (2017). specialized reference services at illinois: reference transactional analysis and its implications for service providers and administrators. reference & user services quarterly, 56(4), 268-276. https://doi.org/10.5860/rusq.56.4.268   lewis, d. w. (1995). traditional reference is dead, now let’s move on to important questions. the journal of academic librarianship, 21(1), 10-12. https://doi.org/10.1016/0099-1333(95)90146-9   maloney, k., & kemp, j. h. (2015). changes in reference question complexity following the implementation of a proactive chat system: implications for practice. college & research libraries, 76(7), 959–974. https://doi.org/10.5860/crl.76.7.959   miles, d. b. (2013). shall we get rid of the reference desk? reference & user services quarterly, 52(4), 320-333. https://doi.org/10.5860/rusq.52n4.320   reference and user services association: rusa. (2008, september 29). definitions of reference. http://rusa.ala.org/blog/2008/01/31/rusa-approves-new-definition-of-reference/documentid:d42fff7a-8841-1ad4-c188-ee0259d6e7ad    ross, c. s., & dewdney, p. (1998). negative closure: strategies and counter-strategies in the reference transaction. reference & user services quarterly, 38(2), 151-163.   ryan, s. m. (2008). reference transactions analysis: the cost-effectiveness of staffing a traditional academic reference desk, the journal of academic librarianship, 34(5), 389-399. https://doi.org/10.1016/j.acalib.2008.06.002   saint clair, j. w., aluri, r., & pastine, m. (1977). staffing the reference desk: professionals or nonprofessionals? the journal of academic librarianship, 3(3), 149-153.   shachaf, p. (2009). the paradox of expertise: is the wikipedia reference desk as good as your library? journal of documentation, 65(6), 977-996. https://doi.org/10.1108/00220410910998951   sosulski, n. w., & tyckoson, d. a. (2016). a reference for that: “what are we stopping?” and “what is shifting?” reference & user services quarterly, 56(2), 87-90. https://doi.org/10.5860/rusq.56n2.87   stevens, c. r. (2013). reference reviewed and re-envisioned: revamping librarian and desk-centric services with libstars and libanswers. the journal of academic librarianship, 39(2), 202-214. https://doi.org/10.1016/j.acalib.2012.11.006   summerhill, k. s. (1994). the high cost of reference: the need to reassess services and service delivery. the reference librarian, 20(43), 71-85. https://doi.org/10.1300/j120v20n43_07   weber, j. e., & bowron, c. r. (2019). the evolving reference desk: if we scrap it, then what? tennessee libraries, 69(2). https://www.tnla.org/page/69_2_weber   values-based practice in eblip: a review review article   values-based practice in eblip: a review   emilia c. bell coordinator (evidence based practice), library services university of southern queensland ipswich, queensland, australia email: emilia.bell@outlook.com   received: 23 may 2022                                                              accepted: 23 july 2022      2022 bell. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30176     abstract   objective – this narrative literature review examines how values and a values-based practice framework are positioned as significant to evidence based practice in libraries. this includes examining the partnership between values and evidence in decision making and reflective practice. the review responds to a gap in the literature on the origins and application of values-based practice in evidence based library and information practice (eblip).   methods – searches for this narrative review were conducted in library and information science databases, discovery tools, and individual journals. forward and backward citation searches were also undertaken. searches aimed to encompass both the eblip and library assessment literature. research and professional publications were considered for inclusion based on their engagement with values and values-based practice in eblip processes and decisions.   results – the findings highlight how values reflect positionality, driving action and decision making in all stages of evidence based practice in libraries. the literature emphasizes the role of values when practitioners engage with critical reflective practice or invite user voices in evidence. an explicit values-based practice approach was evident in the library assessment literature, though not explicitly addressed in the eblip literature or eblip models. this is despite a partnership between evidence based practice and values-based practice in the health sciences literature, with literature on person-centred approaches aiming to relate evidence to individuals.   conclusions – the eblip literature could further examine how values reflect positionality and drive action and decision making across all stages of evidence based practice. values-based practice offers an opportunity to critically reflect on whose voices, perspectives, and values are reflected in and contribute to the library and information science evidence base.     introduction   evidence based practice (ebp) has come to be a significant feature of library and information science (lis), at individual and organizational levels. it is used to engage in decision making and continuous service improvement and to demonstrate the reach, impact, and value of library services. existing approaches to evidence based library and information practice (eblip) provide a structured and rigorous framework to collecting, evaluating, and applying evidence to practice (thorpe & howlett, 2020). as libraries engage with evidence that captures user voices, it is paramount that there is an understanding of the values that drive approaches to eblip and the decisions around it. this includes recognizing the values that underlie motivations for undertaking specific ebp activities (doucette, 2016) and how they reflect positionality (magnus et al., 2018).   values-based practice emphasizes that decision making needs to consider “preferences, needs, hopes, [and] expectations” (fulford, 2008, p. 10) in hand with the best type of evidence. such considerations include values that should be integrated into and underpin the decision-making process. values are not only relevant to decision making but are “explicitly present” in decision making (fulford, 2008, p. 11). values are “action guiding” (hare, 1952, p. 163). they have an evaluative and prescriptive element that functions to guide choices (hancock, 1963). informed by this understanding, this review adopts doucette’s (2016) definition of values, in the eblip context, where they are the “beliefs and codes of behaviour that guide actions and decisions” (p. 289).   while there is an overlap and often synonymous use of “eblip” and “library assessment,” this review recognizes their potential differences in scope. this is seen in ryan’s (2006) evaluation that “in assessment work, evidence can only be local” and will largely centre on local user experiences and perceptions (p. 79). building on this, koufogiannakis and brettle (2016) describe assessment as being a component of eblip, whereby library professionals “gather and create local evidence” specifically for assessment (p. 35). considering this distinction, the review looks at how the eblip literature might engage with the broader lis literature on values and critical practice, as well as with the existing assessment literature on these areas. thus, this review draws from library assessment literature that is more overtly engaged with the topic of values-based practice. this is to then determine how a values-based practice approach might be engaged throughout eblip.   aim   this narrative review aims to present a synthesis of the literature on the role and application of values-based practice in eblip. it examines the role of values in eblip and examines the relevance and innateness of values-based practice to existing eblip models. while values are addressed in much of the literature on critical library assessment, eblip literature has not explicitly discussed values to the same extent. thus, this review is guided by the questions:   ·        how are values and a values-based approach significant to eblip? ·        how does values-based practice align with eblip processes?   the findings are intended to inform both library practice and future lis research in considering how a values-based approach might be embedded across all stages of eblip.   following this introduction, the method for undertaking this narrative literature review is described. the findings of this review are then presented, starting with an overview of values in lis and the origins of values-based practice. the article then looks at different themes that highlight the relationship between values and evidence. these themes include values and the eblip model, reflective practice, and critical assessment practices. finally, a discussion section explores the application of these findings to practice. it considers how values and evidence might work in partnership in practice and future research.   method   the search strategy adopted for this literature review was carried out across several databases and individual journals with relevant search techniques. it aimed to capture a contextual overview of values-based practice, in addition to evidence of its relevance to eblip. as a narrative review, the searches were not intended to be systematic and this paper does not aim to present an exhaustive or systematic review of values-based practice.   databases searched were primarily from the social sciences and included: ebscohost: academic search ultimate, library & information science abstracts (lisa), and library, information science & technology abstracts (lista). google scholar was also used as a discovery tool. the searches allowed for an interdisciplinary context that recognized the original health sciences context of values-based practice. while some of this literature is outside of, or peripheral to, the lis sector, the application of values-based practice to eblip remains the focus of this review. relevant non-scholarly literature (from professional publications in the lis field) was included in this review, as well as literature already known to the author or encountered through serendipitous discovery. serendipitous discovery refers to the “timely and useful, but unexpected, outcomes, discoveries, or even tangents which occurred while in quest of something else” (liestman, 1992, p. 525). in this review, serendipitous discovery was predominantly grounded in the author’s existing background knowledge (workman et al., 2014), which helped facilitate conditions for encountering relevant literature outside of active searching or browsing.   initial searches included variations of the search phrase ("values-based practice" or "values based practice" or "value based practice") and ("evidence-based practice" or "evidence based practice") and librar*. variations of (librar* or “information professional*” or “information science”) were also included to narrow searches to lis literature. searches in lis databases were widened by removing the term “evidence-based practice.” this recognized the instances where values-based practice might not be explicitly described as eblip and also captured library assessment literature. library assessment literature was selected for inclusion in this review but was not explicitly included as a search term, as there was a need to broaden, not narrow, searches in the lis literature.   searches in lisa and lista revealed a non-peer reviewed article by schachter (2007) on “value-based practice.” this article was excluded, however, as it refers to “value-based” with regards to the business management concept, focused on value creation for stakeholders. value-based management is recognized as distinct from the values-based practice identified in the health sciences alongside ebp. the latter is focused on the principles or standards that define judgements in decision making, rather than the generation of business value. while eblip outcomes are often used to demonstrate a library’s value to stakeholders, the focus of this review is on values as “action guiding” (hare, 1952, p. 163).   searches in lis journals were broader and less specific than in databases and discovery tools. search terms included either “values” or “values-based,” these being searched both alone and in combination with the term “evidence-based.” individual journals searched included evidence based library and information practice, in the library with the lead pipe, and portal: libraries and the academy. while synthesizing the literature, forward and backward citation searches were also undertaken, using reference lists, google scholar, and the citation database scopus.   findings   defining values-based practice   in its original health sciences context, ebp is positioned as complementary to values-based practice (fulford, 2008; stoyanov et al., 2020). values-based practice is considered a “primarily skills-based approach to working with complex and conflicting values” (fulford, 2008, p. 12). just as many practice-based disciplines beyond the library sector employ evidence based practice (wilson, 2017, p. 185), a range of professions and disciplines have engaged with values-based practice. it is found in literature across such areas as medical and health sciences (fulford, 2008), educational psychology (prendeville & kinsella, 2022), higher education (dai et al., 2021), and academic psychology with “mātauranga (māori knowledge, culture, values and worldview) and tikanga (māori practices)” (nutbrown et al., 2021, p. 21).   values-based practice takes a person-centred approach that relates evidence to individuals. it is intended to model and produce a partnership, despite the presence of conflicting values (fulford, 2008, p. 12). in health sciences, values are the “unique preferences, concerns and expectations each patient brings to a clinical encounter and which must be integrated into clinical decisions if they are to serve the patient” (straus et al., 2018, p. 1). they are considered an integrated and integral aspect of ebp (fulford, 2008). in a setting that requires engagement with service users, engaging with values can help to ensure a holistic service that considers what is important to the user (strudwick, 2021, p. 118). thus, values-based practice is considered not only complementary but “an essential partner” to ebp (fulford, 2008, p. 19). differences in values are made explicit and are recognized as an asset rather than a problem.   lis values and ethics in practice   values and ethics are established topics across the lis research literature (berg & jacobs, 2016; koehler, 2003; miller, 2007; young, 2020a) and are embedded in core values policy statements from lis professional associations (australian library and information association, 2018). rigling et al. (2018), however, describe how the focus of library research is on “demonstrating library value to external stakeholders as opposed to understanding library values” (para. 11). likewise, nicholson (2017) states that “as a profession, we’ve become veritably obsessed with value” (p. 2). this focus on generating “value” is largely reflected in the eblip and library assessment literature, where being able to demonstrate and provide evidence of value and impact is an expected outcome. for drabinski and walter (2016), “theory and practice should be mutually informative in our field, and inquiry into ‘values’ should occupy as privileged a place as inquiry into ‘value’” (p. 267). while the broader lis literature includes inquiry into values and is labelled “values-based” (miller, 2007), the concept of “values-based practice,” as an action-guiding process or framework, is not delineated in the eblip literature to the same extent.   the specific phrase “values-based practice” is not widely used in the lis literature, though various research articles, case studies, and assessment literature actively demonstrate the concept. rigling et al. (2018) examine developing a values-based open access statement and mavrinac (2005) explores values-based learning, leadership, and change. in the library assessment literature, young (2020b) examines professional values and explicitly addresses “values-based decision-making in the practice of library assessment” (slide 2). doucette (2016) also analyzes the values-based motives for undertaking assessment work. this body of research largely exists in the assessment literature, and work by nicholson (2017) and douglas (2020) actively engages with themes of critical librarianship or other critical theoretical approaches.   in the eblip research literature, values-based practice has not been explicitly positioned as a framework that can be applied or adapted by practitioners. brettle (2012) reflects on the ethical values and research that underpin eblip’s origins in the health sciences, and how core values should also position lis professionals’ knowledge and practice within a wider context. much of the eblip literature builds on this engagement with core values by encouraging critical reflective practice. miller et al. (2020) highlight the relevance of reflective practice models from other professions to eblip, such as kim’s critical reflective inquiry model, gibbs’ reflective cycle model, and a situation, evidence, action (sea) change model. such models can facilitate the application of ethics in practice by allowing an exploration of professional, personal, and organizational values. this is dependent, however, on library practitioners purposefully and consistently engaging with reflection throughout eblip processes (miller et al., 2020, p. 361).   in the library assessment literature, young (2020a) suggests that future research might actively apply a “lens of practical ethics” to assessment practice (p. 16). this lens positions ethics as the application of values and “prompts the practitioner to ask how one should behave in particular situations, with all of the attendant contextual factors and conflicts” (young, 2020a, p. 4).   given the extensive literature on developing an eblip model that offers a “realistic depiction of the eblip processes” and supports critical reflective practice (thorpe, 2021, p. 119), eblip literature might also pursue a complementary lens or framework that positions values-based practice alongside ebp in libraries.   values in the eblip model   the eblip model has evolved to become a “holistic and realistic depiction” of processes that apply to individual practitioners or groups and that help to “foster critical reflective practice” (thorpe, 2021, p. 119). the “5as” model of eblip proposed by koufogiannakis (2013a) has the stages: articulate, assemble, assess, agree, and adapt. thorpe also proposes an important sixth stage: announce or advocate. this sixth stage recognizes the significance of “communicating eblip outcomes and process to the local community and the professional evidence base” (thorpe, 2021, p. 117). the “5as” eblip model specifically encourages reflective practice as a crucial aspect of ebp (koufogiannakis, 2010). this involves asking and proposing questions, and eldredge (2000) claims that “questions drive the entire ebl [evidence based librarianship] process” (p. 292). koufogiannakis and brettle (2016, pp. 19–20) agree, but also highlight the many considerations that should occur when formulating questions so that they require reflection on “values in practice.” as such, the reflective questions suggested by koufogiannakis and brettle (2016, p. 15) point to the presence of underlying values that will inform and drive a library’s evidence needs and processes.   values inform the questions that are asked and the actions taken in decision making throughout the entire eblip process. values are explicitly identified by koufogiannakis (2013a, p. 14) in the agree stage. the agree stage is where evidence is used to determine a course of action and implement decision making, and koufogiannakis (2013a) suggests asking: “is the decision in keeping with our organization’s goals and values?” (p. 14). it is, however, from the very outset of the eblip process that values inform decision making. in the first articulate stage, where questions are formulated, koufogiannakis and brettle (2016) propose considering the goals, purpose, and assumptions that are present, and this requires considering the underlying values behind why something is significant to address. likewise, drabinski and walter (2016) recognize the need to not only ask questions on the what and how of library research but also the why. here, values inform not only the questions asked throughout ebp, but also how evidence is later assembled and assessed (koufogiannakis & brettle, 2016, p. 20).   values in reflective practice   reflective practice is recognized as critical to eblip (booth, 2003; koufogiannakis, 2010; miller et al., 2020). this is especially for aspects related to professional knowledge (hallam, 2018, p. 455). while reflection can take many forms, both formal and informal, in professional practice it is a process of thoughtfully considering actions and events, either as they happen or after the fact, and applying this knowledge to practice (miller, 2020). this is with the purpose of improving practice and building understanding and awareness around it. hallam (2018) maintains that “eblip represents the mind-set of a critically reflective practitioner” (p. 457) and booth (2003) also suggests that eblip’s future lies in “a more encompassing approach that embodies reflective practice” (p. 70). critical reflective practice requires challenging assumptions as part of reflective inquiry and includes recognizing power dynamics (brookfield, 1998). it extends reflection to include the “examination of one’s own assumptions about professional practice as well as assumptions of the profession and the broader culture” (miller, 2020, p. 18). the questions that koufogiannakis (2013a) poses within the eblip model are indicative of a reflective approach across all stages of eblip. these questions allow for underlying values to be considered at all stages, and they present values as inherent to any decisions that inform the ebp process. this includes values-based reflection on the actions taken for data creation and evidence collection, evaluation, and application.   in eblip, reflective practice and values culminate in actions. miller et al. (2020) determine that critical reflective practice helps not only to identify personal and professional values, but also to explore “where thoughts and actions diverge from these values” and the “courses of action that are consistent with these values” (p. 354). both miller et al. and thorpe (2021) draw a link between fostering critical reflective practice (that exposes bias, assumptions, structures, and values in decision making) and critical librarianship in eblip. considering eblip and critical librarianship, thorpe (2021) describes how “increasing the quality, quantity, and diversity of work contributed to the evidence base should also foster inclusion and diversity of opinion, inviting more voices and alternative perspectives into the profession” (p. 122).   this highlights how values are also present in the final advocate or announce stage of eblip, as the “critical nature of questioning that starts with the articulate stage should reach a logical conclusion with advocacy” (thorpe, 2021, p. 122). values-based practice offers an opportunity to critically reflect on whose voices, perspectives, and values are reflected in and contribute to the lis profession’s evidence base.   values are recognized as a “stabilizing element” in lis professional practice, and reflective practice is important for examining any bias or assumptions that underlie values (young, 2020a, p. 3). professional values, however, may appear as an aspirational ideal, remaining abstract rather than realized in practice. froehlich (2000) maintains that values are realised and measured through the choices of library professionals and library users. as such, ebp may present both a “measure” of the application of professional values, while also being an outcome of the values that drive it. froehlich (2000) explains how “this measure is drawn against the kinds of actions, choices and implementations that the ideal information professional or librarian would make in a specific context at a specific time” (p. 271).   further, when values are realized as contextually dynamic, a “professional may be the best gauge of how to embody a value or set of values … for a given situation” (froehlich, 2000, p. 271). similarly, jacobs and koufogiannakis (2014) argue that eblip does not provide a final solution or necessarily the “best” answer but will help to provide the best response or decision for a particular time and context. thus, identifying and actively engaging with values requires significant reflection on decision-making processes in a specific lis context.   values in critical assessment practices   an organizational culture of eblip can incorporate a culture of assessment through engagement with local evidence. regardless of the extent to which eblip and library assessment are considered synonymous, assessment practices remain relevant to ebp processes and both concepts overlap considerably. critical assessment sees practitioners explicitly considering matters of power, privilege, and positionality, recognizing library assessment as both a social and political act (magnus et al., 2018). this means examining and critically reflecting on one’s own underlying assumptions and choices, and on whom assessment is for. like values-based practice, critical approaches to library assessment respond to social context. critical approaches recognize that everyone involved in assessment are “individuals affected by social, political, and economic drivers” (benjes-small et al., 2019, para. 1). thorpe (2021) identifies how the “critical nature of questioning” that courses through all eblip stages sees eblip align with the critical librarianship movement (p. 122). from the outset, the use of evidence should reflect an awareness of organizational behaviour (and thus values) that is “named and professionally discussed” to counter bias in decision making (koufogiannakis, 2013b, p. 197).   library assessment literature has presented a focus on critical librarianship and critical practices, with a values and ethics orientation (benjes-small et al., 2019; douglas, 2020; fisher et al., 2019; magnus et al., 2018; nicholson, 2017; young, 2020a, 2020b). in work by young and brownotter (2018), critical and reflective approaches to assessment are implemented through participatory design practices, which provide a “values-driven approach to co-creation” (introduction). this repositioning of assessment practices is also seen in higher education more broadly, where wall et al. (2014) advocate for engagement with a social and values-driven context. they reflect on how:   by raising consciousness of the ethical and value-based decisions implicit in any assessment context, the practice of assessment truly becomes a complex social practice rather than a collection of technical data gathering approaches that might unwittingly serve power interests unintended by well-meaning individuals. (p. 13)   critical assessment requires recognizing social complexity and context. thus, individual library practitioners have a responsibility to engage in making “interpretive judgments” during the assessment process, especially in assessing evidence (wall et al., 2014, p. 12). since context defines evidence and requires interpretation, findings are not simply self-evident. it is through critical reflection of context that biases in assessment or ebp practices might be addressed (magnus et al., 2018). this includes reflection on the existing individual, organizational, and institutional values and norms brought into assessment practices.   extending beyond critical library assessment, assessment can also be positioned as a “practice of care,” which douglas (2020) advocates for. assessment as care aims to dismantle the dominant and accepted narrative of assessment as advocacy and reporting, and instead recognizes assessment as an “inherently relational act” that “prioritizes care over justification, connection over reporting, and people over products” (douglas, 2020, p. 47).  douglas (2020) concludes that this can be uncomfortable and that “it complicates our work of data collection, visualization, and comparison because feelings, care, and concern don’t fit within the confines of an assessment dashboard” (p. 61). assessment as care does not fit into existing practices of either teaching or assessment in libraries, especially academic libraries (douglas, 2020, p. 60). a critical reflective practice, however, means engaging with the discomfort of critical dialogue on our practices and biases (preater, 2020), which inevitably extends to our values. in a profession that has been warned of being “more interested in process and structure than people” (wiegand, 1999, as cited in samek, 2007, p. 4), values-based practice may help to realize person-centred eblip processes, with dialogue that elevates relationships and care in evidence.   values are often regarded as “aspirational” rather than “realized” in librarianship (young, 2020a, p. 3). thus, young (2020b) actively seeks to apply values-based practice to ethical dilemmas in librarianship, demonstrating a practical application in and beyond library assessment. young (2020c) makes three recommendations (or steps) for values-based assessment decisions, which are: value identification, ethical attunement, and critical self-reflection. ethical attunement refers to recognising, and being aware and responsive to, ethical dilemmas and choices.  together, these three steps set a practical lens to guide the application of ethics stemming from core professional values. in eblip, similar steps to those that young (2020b) suggests could be modelled, encompassing local evidence, professional knowledge, and research literature. presently, the eblip literature does not address values as overtly as young does in the library assessment literature. young’s (2020a) framework for values-based practice drives its practical application as “shared values then become the operational principles of an ethical practice” (p. 4). the “6as” eblip model (including thorpe’s [2021] addition) could see values-based practice applied across distinct stages of ebp, at both individual and organizational levels, with critical reflection and values embedded iteratively throughout the process.   discussion   values and evidence as a partnership   values-based practice is consistently considered in partnership with ebp in the health sciences literature (fulford, 2008; peile, 2013; stoyanov et al., 2020). fulford (2008) reflects that “values as well as evidence underpin all decisions” (p. 11). this is regardless of whether values are explicit or not. just as eblip helps to navigate evidence for decision making and continuous service improvement, values-based practice helps to navigate the values inherent to decision making in library services and eblip processes. looking to values and evidence together may help to ensure that services and practices are holistic and can engage with what is significant to users (strudwick, 2021, pp. 118–119).   the diverse range of values that might be considered in the eblip process invites competing values in decision making. values may reflect the experiences or positionality of a library’s users, individual professionals, the parent organization, or the wider lis sector. this can include competing values within the library’s own organizational culture (currie & shepstone, 2012), which may inform how evidence is applied to operational or strategic decision making. when working with conflicting values, developing a values-based approach may offer a process to resolve conflict or ethical dilemma (fulford, 2008). additionally, values-based practice should also be evidence based. that is, understanding and collecting information on the values involved in practice should not be informed by intuition alone. a skilled approach to ascertaining values (peile, 2013), especially around users and library communities, should be applied. this positions values-based practice as complementary to ebp, with a potential for alignment or partnership in approaches to either.   conflicting values may also extend to what evidence is valued in different contexts. where eblip should consider the “views, preferences and values” of its community (booth & brice, 2004, p. 5), different professional or disciplinary values (outside of lis) may require consideration. this can highlight differences in how a culture of ebp is developed and how the “philosophies and values on which the culture is based” are understood (hallam, 2018, p. 460). in libraries, this could mean inviting and valuing a humanities perspective of evidence, as jacobs and koufogiannakis (2014) suggest. adopting a humanities perspective in eblip means accepting that “theory and reflection are valid and reliable forms of evidence” (jacobs and koufogiannakis, 2014, p. 116). this means embracing uncertainty in research inquiry, to question, examine, and change, our decisions, practices, and core values.   looking to ebp collaboration outside of the library, adams et al. (2016) describe the need to reflect on professional differences between librarians and a university education faculty. in this instance, adams et al. explore how librarians might support teaching ebp as professional competency in education, rather than eblip in libraries. while not centred on eblip, the study still highlights the need for a nuanced understanding of different disciplinary values, including what is considered as evidence within different research paradigms and modes of inquiry. just as within the lis sector, adams et al. (2016) recognize that in education, evidence is to be “guided by the values of the client and the community” (p. 698). it is by identifying and understanding these disciplinary values that librarians might “initiate and extend collaborations with education faculty” (adams et al., 2016, p. 717). significantly, this requires negotiating “what ‘counts’ as knowledge” and what values are ascribed to evidence (adams et al., 2016, p. 718).   the role of values alongside evidence means recognizing that ebp is not in opposition to values, as it has sometimes been framed (fulford et al., 2013, p. 403). values are, then, not separate or contradictory to collecting empirical evidence. in the lis literature, booth and brice’s (2004) early definition of ebp calls for the “best available evidence”; however, this is to be “moderated by user needs and preferences” (p. 7). that is, practitioner research findings need to be mediated “by the views, preferences and values of the community within which the evidence is to be implemented” (booth & brice, 2004, p. 5). the social context that evidence sits within is recognized and library users are considered significant to understanding evidence.   user voices and values   in taking a person-centred approach, values-based practice can bring increased acknowledgement of user voices in libraries. in the health sciences, ebp’s “attention to clients’ values and expectations” aims to position clients as “active participants in the decision-making processes” (gambrill, 1999/2018, p. 289). definitions of eblip recognize the values and preferences of library users as a form of evidence to inform decision making (booth & brice, 2004, p. 7; eldredge, 2012, p. 139), and there is an increased interest from libraries in incorporating user experience as evidence (lewis, 2016, p. 107). such values are often considered “soft” evidence sources, as they relate to the user experience and preferences. soft evidence sources are those that are more informal or anecdotal and often tell a story or contribute to a larger narrative within a specific local context (koufogiannakis, 2013a; lewis, 2016).   in academic libraries (scoulas et al., 2021) and across higher education (mercer-mapstone et al., 2017), the concepts of “student voice” and “students as partners” have helped to guide the collection of userand person-centred evidence. these concepts are intended to drive decision making that positions students as partners, and they have continued to feature in inclusive library decision making during the covid-19 pandemic (appleton, 2020; milton & meade, 2018; scoulas et al., 2021). matthews (2019, p. 1), considering higher education more broadly, suggests that “students as partners” should be enacted as a values-based practice, one that actively represents the values of partnership. the values-based co-creation of partnership that matthews (2019) describes can extend to an evidence base that prioritizes libraries’ partnerships with their respective communities.   ebp in partnership with values-based practice can help to engage with library users’ values and to centre community. this may invite a more inclusive approach to data collection and build a diverse evidence base both locally and across the lis sector. for libraries with student communities, fisher et al. (2019) suggest several questions that guide a critical approach to library assessment practices, which include:   ·        “how do institutional assumptions and agendas shape the student data we collect?” ·        “where does our accountability lie in the creation and use of student data?” ·        “are we engaging students in the creation, analysis, and communication of data and insight?” (p. 29).   these questions direct attention to users’ values and expectations. they support building an evidence base that is contextualized by values and committed to engaging with values. such practice offers an opportunity to critically reflect on whose voices, perspectives, and values are reflected in and contribute to the lis profession’s evidence base.   a focus on users’ values relies on library professionals engaging in critical reflection on their own values, in addition to organizational and professional values. rigling et al. (2018) describe confronting an assumption that organizational values were “clear cut” and aligned with users’ needs around open access. instead, rigling et al. determine that individual staff values presented with more nuance than expected. it was “by turning the lens that we use to understand the needs of our users on ourselves [that] we were able to unpack our individual values and create a meaningful, grassroots policy statement on oa” (rigling et al., 2018, para. 2).   this values-based process resulted in challenging the idea that organizational policies and practices accurately reflected the values and perspectives of all library staff. a recognition of and engagement with individual staff values, even when conflicting, remains significant to how eblip and library assessment are pursued (magnus et al., 2018). trying to determine user values and perspectives, without critically reflecting on our own, can mean inadvertent consequences for users from otherwise well-intentioned questions and decisions in ebp.   limitations and future research   this paper aimed to offer a broad narrative overview and entry point for future research on values-based practice and eblip together. as it is not intended to be an exhaustive or systematic review, the broad scope and inclusion criteria represent a methodological limitation for its reproducibility and replicability. future research or practice could build on the findings of this review to overlay or adapt values-based practice to an eblip model, as young (2020b) does in library assessment. extending a values-based or ethical lens would see decisions that reflect professional and organizational values enacted at each stage of evidence based practice, as part of an iterative process.   conclusion   while values-based practice, as an established concept, is not explicitly addressed in the eblip literature, the significance of values to eblip processes is seen throughout the literature. the findings of this review affirm the relationship of values to reflective practice, decision making, and critical assessment practices in eblip. library decision making can employ both values and evidence in partnership to not only support decision-making processes, but also to extend a values-based lens to library users. this lens emphasizes integrating social context into library services and decisions. values-based practice presents opportunities to further a person-centred approach to eblip and further promote reflection in decision making. such reflection encourages consideration of the motives for engaging in eblip initiatives and the positionality of library professionals and organizations throughout the process.   references   adams, n. e., gaffney, m. a., & lynn, v. (2016). the role of evidence-based practice in collaborations between academic librarians and education faculty. portal: libraries and the academy, 16(4), 697–720. https://doi.org/10.1353/pla.2016.0048   appleton, l. (2020). academic libraries and student engagement: a literature review. new review of academic librarianship, 26(2-4), 189–213. https://doi.org/10.1080/13614533.2020.1784762   australian library and information association. (2018). alia core values policy statement. https://read.alia.org.au/alia-core-values-policy-statement   benjes-small, c., seale, m., hodges, a. r., & meiman, m. (2019, june 11). keeping up with... critical assessment. american library association. https://www.ala.org/acrl/publications/keeping_up_with/critical_assessment   berg, s. a., & jacobs, h. l. (2016). introduction: valuing librarianship: core values in theory and practice. library trends, 64(3), 459–467. https://doi.org/10.1353/lib.2016.0002   booth, a. (2003). where systems meet services: towards evidence‐based information practice. vine, 33(2), 65–71. https://doi.org/10.1108/03055720310509037   booth, a., & brice, a. (2004). why evidence-based information practice? in a. booth & a. brice (eds.), evidence-based practice for information professionals: a handbook (pp. 1–12). facet. https://doi.org/10.29085/9781856047852   brettle, a. (2012). evidence and ethics. evidence based library and information practice, 7(4), 1–3. https://doi.org/10.18438/b8p32r   brookfield, s. (1998). critically reflective practice. journal of continuing education in the health professions, 18(4), 197–205. https://doi.org/10.1002/chp.1340180402   currie, l., & shepstone, c. (2012). mining the cultural evidence: situating planning and leadership within the academic library culture. evidence based library and information practice, 7(3), 51–70. https://doi.org/10.18438/b8p02q   dai, k., matthews, k. e., & shen, w. (2021). ‘it is difficult for students to contribute’: investigating possibilities for pedagogical partnerships in chinese universities. teaching in higher education. advance online publication. https://doi.org/10.1080/13562517.2021.2015752   doucette, l. (2016). acknowledging the political, economic, and values-based motivators of assessment work: an analysis of publications on academic library assessment. in s. baughman, s. hiller, k. monroe, & a. pappalardo (eds.), proceedings of the 2016 library assessment conference: building effective, sustainable, practical assessment (pp. 288–298). association of research libraries. https://www.libraryassessment.org/wp-content/uploads/bm~doc/48-doucette-2016.pdf   douglas, v. a. (2020). moving from critical assessment to assessment as care. communications in information literacy, 14(1), 46–65. https://doi.org/10.15760/comminfolit.2020.14.1.4   drabinski, e., & walter, s. (2016). asking questions that matter. college & research libraries, 77(3), 264–268. https://doi.org/10.5860/crl.77.3.264   eldredge, j. d. (2000). evidence-based librarianship: an overview. bulletin of the medical library association, 88(4), 289–302. https://www.ncbi.nlm.nih.gov/pmc/articles/pmc35250/    eldredge, j. d. (2012). the evolution of evidence based library and information practice, part i: defining eblip. evidence based library and information practice, 7(4), 139–145. https://doi.org/10.18438/b8gc99   fisher, z., magnus, e., & branch, n. (2019, march 13). critical assessment practices: a discussion on when and how to use student learning data without doing harm [webinar]. association of college and research libraries instruction section – management & leadership committee. https://scholarcommons.scu.edu/library/205/   froehlich, t. j. (2000). intellectual freedom, ethical deliberation and codes of ethics. ifla journal, 26(4), 264–272. https://doi.org/10.1177/034003520002600405   fulford, k. w. m., van staden, c. w., & crisp, r. (2013). values-based practice: topsy-turvy take-home messages from ordinary language philosophy (and a few next steps). in k. w. m. fulford, m. davies, r. gipps, g. graham, j. z. sadler, g. stanghellini, & t. thornton (eds.), the oxford handbook of philosophy and psychiatry (pp. 385–410). oxford university press. https://doi.org/10.1093/oxfordhb/9780199579563.001.0001   fulford, k. w. m. (2008). values-based practice: a new partner to evidence-based practice and a first for psychiatry? mens sana monographs, 6(1), 10–21. https://www.ncbi.nlm.nih.gov/pmc/articles/pmc3190543/   gambrill, e. (2018). evidence-based practice: an alternative to authority-based practice (revisiting our heritage). families in society, 99(3), 283–294. https://doi.org/10.1177/1044389418786699 (reprinted from “evidence-based practice: an alternative to authority-based practice,” 1999, families in society, 80[4], 341–350, https://doi.org/10.1606/1044-3894.1214)   hallam, g. (2018). being evidence based makes sense!: an introduction to evidence based library and information practice (eblip). bibliothek forschung und praxis, 42(3), 453–462. https://doi.org/10.1515/bfp-2018-0067   hancock, r. (1963). a note on hare’s the language of morals. the philosophical quarterly (1950-), 13(50), 56–63. https://doi.org/10.2307/2217007   hare, r. m. (1952). the language of morals. oxford university press. https://doi.org/10.1093/0198810776.001.0001   jacobs, h., & koufogiannakis, d. (2014). counting what cannot be counted: bringing the humanities to eblip. evidence based library and information practice, 9(3), 110–120. https://doi.org/10.18438/b8pc8s   koehler, w. (2003). professional values and ethics as defined by “the lis discipline”. journal of education for library and information science, 44(2), 99–119. https://doi.org/10.2307/40323926   koufogiannakis, d. (2010). thoughts on reflection. evidence based library and information practice, 5(2), 1–3. https://doi.org/10.18438/b8z33m   koufogiannakis, d. (2013a). eblip7 keynote: what we talk about when we talk about evidence. evidence based library and information practice, 8(4), 6–17. https://doi.org/10.18438/b8659r   koufogiannakis, d. (2013b). how academic librarians use evidence in their decision making: reconsidering the evidence based practice model [doctoral dissertation, aberystwyth university]. university of alberta education and research archive. https://doi.org/10.7939/r39b7r   koufogiannakis, d., & brettle, a. (eds.). (2016). being evidence based in library and information practice. facet publishing. https://doi.org/10.29085/9781783301454   lewis, s. (2016). reflections on using patrons’ stories as practice-based evidence. evidence based library and information practice, 11(1), 107–110. https://doi.org/10.18438/b87k8t   liestman, d. (1992). chance in the midst of design: approaches to library research serendipity. rq, 31(4), 524–532. http://www.jstor.org/stable/25829128   magnus, e., belanger, j., & faber, m. (2018). towards a critical assessment practice. in the library with the lead pipe. https://www.inthelibrarywiththeleadpipe.org/2018/towards-critical-assessment-practice/   matthews, k. e. (2019). rethinking the problem of faculty resistance to engaging with students as partners in learning and teaching in higher education. international journal for the scholarship of teaching and learning, 13(2), article 2. https://doi.org/10.20429/ijsotl.2019.130202   mavrinac, m. a. (2005). transformational leadership: peer mentoring as a values-based learning process. portal: libraries and the academy, 5(3), 391–404.   mercer-mapstone, l., dvorakova, s. l., matthews, k. e., abbot, s., cheng, b., felten, p., knorr, k., marquis, e., shammas, r., & swaim, k. (2017). a systematic literature review of students as partners in higher education. international journal for students as partners, 1(1). https://doi.org/10.15173/ijsap.v1i1.3119   miller, j. m. (2020). reflective practice and health sciences librarians: engagement, benefits, and barriers. journal of the medical library association: jmla, 108(1), 17–28. https://doi.org/10.5195/jmla.2020.777   miller, j. m., ford, s. f., & yang, a. (2020). elevation through reflection: closing the circle to improve librarianship. journal of the medical library association: jmla, 108(3), 353–363. https://doi.org/10.5195/jmla.2020.938   miller, r. k. (2007). the value of values-based literature: an exploration of librarianship's professional discussion of core values [master’s thesis, university of north carolina]. unc carolina digital repository. https://doi.org/10.17615/8t9n-7d28   milton, s., & meade, q. h. (2018). more than just a student voice: facilitating student leadership development through the library student liaison program. journal of library administration, 58(4), 346–363. https://doi.org/10.1080/01930826.2018.1448650   nicholson, k. p. (2017, october 26). the “value agenda”: negotiating a path between compliance and critical practice [keynote address]. canadian libraries assessment workshop (claw), university of victoria, bc, canada. https://ir.lib.uwo.ca/cgi/viewcontent.cgi?article=1049&context=fimspres   nutbrown, s., murphy, b., demirbas, y., muñoz duran, f., hikuroa, d., & harré, n. (2021). investigating the call to kindness: a study with community participants in aotearoa new zealand. knowledge cultures, 9(3), 20–38. https://doi.org/10.22381/kc9320212   peile, e. (2013). evidence-based medicine and values-based medicine: partners in clinical education as well as in clinical practice. bmc medicine, 11, article 40. https://doi.org/10.1186/1741-7015-11-40   preater, a. (2020). widening participation praxis and library leadership. in s. s. hines & d. h. ketchum (eds.), critical librarianship (pp. 21–40). emerald publishing limited. https://doi.org/10.1108/s0732-0671202041   prendeville, p., & kinsella, w. (2022). ethics, values and values based practice in educational psychology. cambridge journal of education. advance online publication. https://doi.org/10.1080/0305764x.2022.2047893    rigling, l. s., carlisle, e., & waugh, c. l. (2018). in pursuit of equity: applying design thinking to develop a values-based open access statement. in the library with the lead pipe. https://www.inthelibrarywiththeleadpipe.org/2018/oa-statement/   ryan, p. (2006). ebl and library assessment: two solitudes? evidence based library and information practice, 1(4), 77–80. https://doi.org/10.18438/b8mk53   samek, t. (2007). librarianship and human rights: a twenty-first century guide. chandos publishing.   schachter, d. (2007). info pros can chart a path to excellence by adopting principles of management. information outlook, 11(11), 44–45.   scoulas, j. m., carrillo, e., & naru, l. (2021). assessing user experience: incorporating student voice in libraries' pandemic response. journal of library administration, 61(6), 686–703. https://doi.org/10.1080/01930826.2021.1947058   stoyanov, d., fulford, b., stanghellini, g., van staden, w., & wong, m. t. h. (eds.). (2020). international perspectives in values-based mental health practice: case studies and commentaries. springer international publishing. https://doi.org/10.1007/978-3-030-47852-0   straus, s. e., glasziou, p., richardson, w. s., & haynes, r. b. (2018). evidence-based medicine: how to practice and teach ebm (5th ed.). elsevier health sciences.   strudwick, r. m. (2021). the ethnographic radiographer. springer. https://doi.org/10.1007/978-981-16-7252-1   thorpe, c. (2021). announcing and advocating: the missing step in the eblip model. evidence based library and information practice, 16(4), 118–125. https://doi.org/10.18438/eblip30044   thorpe, c., & howlett, a. (2020). applied and conceptual approaches to evidence-based practice in research and academic libraries. liber quarterly, 30(1), 1–17. https://doi.org/10.18352/lq.10320   wall, a. f., hursh, d., & rodgers, j. w., iii. (2014). assessment for whom: repositioning higher education assessment as an ethical and value-focused social practice. research & practice in assessment, 9, 5–17. https://eric.ed.gov/?id=ej1062684   wilson, v. (2017). research in practice: mythbusting eblip. evidence based library and information practice, 12(2), 184–186. https://doi.org/10.18438/b8n37b   workman, t. e., fiszman, m., rindflesch, t. c., & nahl, d. (2014). framing serendipitous information‐seeking behavior for facilitating literature‐based discovery: a proposed model. journal of the association for information science and technology, 65(3), 501–512. https://doi.org/10.1002/asi.22999   young, s. w. h., & brownotter, c. (2018). toward a more just library: participatory design with native american students. weave: journal of library user experience, 1(9). https://doi.org/10.3998/weave.12535642.0001.901   young, s. w. h. (2020a, october 29). on ethical assessment: locating and applying the core values of library and information science [conference paper]. library assessment conference: diversity, equity, ethics, and privacy. https://www.libraryassessment.org/wp-content/uploads/2021/06/87-young-on-ethical-assessment.pdf   young, s. w. h. (2020b, october 29). on ethical assessment: locating and applying the core values of library and information science [conference presentation]. library assessment conference: diversity, equity, ethics, and privacy. https://scottwhyoung.com/talks/ethical-assessment-lac-2020   young, s. w. h. (2020c, october 19). what is the right thing to do? looking at lis practice through the lens of professional ethics. [conference presentation]. access conference. https://scottwhyoung.com/talks/lis-values-and-ethics-access-2020     research article   african american undergraduate students’ perceived welcomeness at a midsized university library   kirstin i. duffin research support librarian booth library eastern illinois university charleston, illinois, united states of america email: kduffin@eiu.edu   ellen k. corrigan cataloguing librarian booth library eastern illinois university charleston, illinois, united states of america email: ekcorrigan@eiu.edu   received: 23 jan. 2023                                                               accepted: 12 june 2023      2023 duffin and corrigan. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30312     abstract   objective – this project assessed african american students’ feelings of comfort and belonging about engaging with library resources and services at a public regional comprehensive university in the midwestern united states.   methods – this study used an explanatory sequential design. first, we surveyed degree-seeking african american undergraduates on their perceived welcomeness regarding the library’s collections and spaces, staff and users, and atmosphere and marketing. we then recruited focus group participants from the survey, and in focus group sessions, participants expanded on feedback provided in the survey, with particular emphasis on their feelings about their interactions and experiences with the library.   results – most students who participated indicated the library is a place where they felt safe and welcomed, although the library felt to some like a neutral space rather than a place that actively supported them. focus group participants shared several easily implementable suggestions for making the library a more attractive campus space for african american students.   conclusion – student recommendations will shape the services we provide for an increasingly diverse student body. changes to make the library as physical place more welcoming include exhibiting student artwork and featuring african american themes in displays. the library as a social space can become more welcoming in several ways. hiring a diverse staff and providing staff training on diversity and equity topics, offering engaging student opportunities for congregation in the library, and collaborating with african american student organizations will help to foster a sense of belonging among these students. facilitating opportunities for connection will contribute to african american undergraduates’ academic success.     introduction   eastern illinois university (eiu) booth library’s mission is to collaboratively empower the intellectual and creative growth of our diverse campus and community. to do so, we must seek input from voices representative of all our users. understanding what african american students need from their academic library is an understudied topic in the library literature. while some university libraries have utilized surveys or interviews to garner student input, far fewer have held focus groups to understand the needs of their underrepresented student populations (but see, for example: borrelli et al., 2019; schaller, 2011). the project summarized here builds on such work, with attention to our local african american undergraduate population. results from this project will help direct library efforts to support student engagement, retention, and graduation.   eiu is a public regional comprehensive university with a campus enrollment of just over 6,500, located in a semi-rural town with a population of 17,286 in which at least 80% of people identify as white (u.s. census bureau, 2021). booth library is eiu’s only library, situated centrally on campus. one of the seven core strategies of booth library’s 2020–2025 strategic plan is to “build a culture that supports diversity and inclusion,” with a goal of emphasizing intentionality in our efforts (booth library, 2020). the demographics of student enrollment at eiu have shifted over the past 15 years and are projected to continue becoming more diverse alongside national trends. in the fall of 2021, 35.4% of degree-seeking undergraduate students were from underrepresented racial groups, as compared to 11.9% of students in 2006 (eastern illinois university, 2021). eiu students identifying as african american have more than doubled in that time: 7.8% in 2006 to 21.0% in 2021 (eastern illinois university, 2021).   since african american students comprise our largest underrepresented racial group, we focused our study on their experiences with booth library. to narrow our scope, our study included only students of african descent from the united states rather than a more comprehensive range of black students from around the world, whose experiences vary due to differences among libraries internationally. similarly, we sought input only from undergraduates, whose needs differ from those of graduate students.   specifically, our study investigates these students’ perceptions of the library as a welcoming environment. we adopt the definition of welcomeness used by stewart et al. (2019), as drawn from the oxford english dictionary, in which a library user “is ‘gladly received’ and feels ‘allowed/invited to’ make use of academic library spaces, without microaggressive acts from staff and fellow library users” (p. 20). this study broadly examines several factors that may influence students’ sense of welcomeness, from providing resources to meet their information needs to offering a friendly and safe space for studying and congregating. insights gathered from this study will not only enable the library to improve services to this specific user base but also lay the groundwork for replicating the study with other defined populations.   literature review   meeting user needs has long been a focus of librarians, but understanding the needs of underrepresented populations has only been studied more recently. whitmire (1999, 2003) conducted some of the earlier research on the experiences of african american and other undergraduate students of color in academic libraries. she found that african american undergraduates use academic library services and resources more than white undergraduates, which may be facilitated by library support programs for minority students. external factors influenced student perceptions, as well. undergraduate students of color expressed, more so than white undergraduates, the value of increasing campus diversity both among employees and students as well as in the curriculum; however, students of color had a more neutral view of the academic library compared to their white peers (whitmire, 2004). whitmire (2004) encouraged further study on how libraries can provide a welcoming environment for underrepresented students. elteto et al. (2008) surveyed students’ perceptions of the physical library as a welcoming space at their urban university, with particular interest in gauging the feelings of students of color. they found that students of color felt less safe in the library than their white peers and were less likely than white students to ask for specific improvements to the library, though access to technology and technical and writing assistance were of more importance to students of color.   research is mixed in reporting african american students’ perceptions of welcomeness in academic libraries and may depend on the unique history and culture of a university. in a national survey, black students indicated a general sense of feeling welcome at their academic library (stewart et al., 2019), but campuses that are or have been predominantly white institutions, and libraries in which professional staff are primarily white, contribute to an atmosphere in which african american students may not feel as welcomed and supported as white students (chapman et al., 2020; folk & overbey, 2019; stewart et al., 2019). these researchers noted the historical framing of the academic library often as a white space (chapman et al., 2020) and that library employees may carry implicit biases that influence their interactions with students (folk & overbey, 2019).   libraries exist as a physical place and social space, so we must consider both as we assess students’ perceptions of welcomeness. wiegand (2011, 2015) has written about the historic role of the public library both as a civic institution and community space. wiegand (2005) encouraged librarians to analyze “the library in the life of our users” to inform the work we do (p. 61). undergraduate students have described the academic library first as a physical structure, perhaps initially imposing and overwhelming, but they also see it as a productive space to study and socialize (kracker & pollio, 2003; sare et al., 2021). the academic library might well do more to promote itself as a social space that fosters relationships and community (kim, 2017).   perceptions of support and belonging within an academic space and place influence a student’s collegiate experience. this feeling of connectedness may be more important for students of color, affecting their grades and selected major (murphy & zirkel, 2015; walton & cohen, 2007). those who do not feel welcome are at a disadvantage, academically and socially. campus initiatives that provide meaningful social connections have been shown to help students develop their sense of belonging and community, influencing academic persistence (bass, 2023; brooms, 2018; murphy et al., 2020). library services and programs that foster a welcoming environment can be part of the campus initiatives that contribute to students’ well-being and academic success.   while previous studies on the experiences of black students in academic libraries have had a methodological focus on surveying a broad range of students (elteto et al., 2008; stewart et al., 2019) or interviewing individual students to gain a more detailed understanding of the personalized experiences of students (folk & overbey, 2019), focus groups have been suggested as a useful approach to expand research on student affect of library services (elteto et al., 2008; whitmire, 2006). focus groups provide a synergy in which participants can build off of each others’ responses, and they allow the researcher the ability to clarify and probe responses (stewart et al., 2009). our research combines a survey with follow-up focus groups to explore the question of welcomeness.   aims   our research objective was to explore to what extent booth library is perceived as a welcoming place for our african american undergraduate students. through a survey and follow-up focus groups, booth library faculty solicited input from this underrepresented population regarding their use of library services. we inquired as to which services are valued by these students in order to ensure our resources are aligned with supporting their needs. we probed to understand our blind spots, where we could provide new or better services and resources to reach unmet student needs. by exploring and responding to the needs of our african american undergraduate students, booth library will be curating a culture of sustainability that supports the continuing success of students.   methods   we employed two forms of data collection for this study. our intent was to gather input from a wide swath of our campus population through a survey and conduct follow-up focus groups to sustain more in-depth conversations with students, providing a forum to expand on the thoughts they provided in the survey. focus groups encourage a dialogue among students, with participants able to share new thoughts and ideas after hearing from their peers, and they allow the researcher to have a more extended conversation and ask follow-up questions to facilitate researcher understanding of participant responses. a subset of students who completed the online survey were selected to participate in our focus groups. this research was approved by the eiu institutional review board in december 2021.   survey   initially, we planned to use libqual+ for our survey. upon closer investigation, we found that libqual+ does not assess user affect at the level of detail for which we were aiming and so would not meet our assessment needs. as has been previously argued, evaluation of library services is often a nuanced process that must take into consideration the context-dependent nature of the service being assessed (lilburn, 2017; seale, 2017). we instead adapted a survey that was developed by stewart et al. (2019), which was administered to black undergraduate students across the united states. for our study, we tailored questions to reflect the application of the survey within a single institution and centred questions around three themes: resources, interactions with people (employees and users), and atmosphere and outreach.   the survey was up to 33 questions in length and took about 10–15 minutes to complete (appendix a). we employed branch logic to create a survey that was responsive to students’ experiences. the first question was a screening question to ensure only african american undergraduate students from eiu completed the survey. from here, the first part opened with a screening question to gauge whether the respondent had used booth library resources. if so, they were directed to answer five likert scale questions (strongly agree to strongly disagree) about the library’s resources; if not, they were prompted to respond to an open-ended question about why they hadn’t used the library’s resources. the second part began with a screening question asking whether the respondent had interacted with people in booth library (employees or users). if yes, they were asked eight likert scale questions about their contact with people in the library; if not, they were asked an open-ended question about why they had not interacted with people in the library. the third part asked participants if they had visited booth library. if yes, they were asked seven likert scale questions about the library’s atmosphere and outreach; if not, they were asked an open-ended question about why they had not been to the library, along with four likert scale questions about the library’s atmosphere and outreach. all respondents were asked two questions about the amount of time they have used the library along with an open-ended question about their most memorable experience at booth library. the survey ended with six demographic questions and an invitation to sign up to participate in our focus groups. for completing the survey, students could opt in to be entered into a raffle for a $15 walmart gift card.   drafts of the survey were reviewed by a social science research expert and a college student affairs research expert, also coordinator of the making excellence inclusive initiative at eiu, and was pilot tested by three african american undergraduate students. the survey was sent in the spring of 2022 to 1,217 african american undergraduate students through an email from eiu’s office of inclusion and academic engagement, as well as emails sent by leaders of relevant student organizations (i.e., the black student union, national pan hellenic council, and eiu’s chapter of the n.a.a.c.p.). a promotional flyer inviting participation was posted through the black student union account on instagram and through booth library social media channels (instagram, facebook, and twitter), where relevant campus and student groups were tagged.   focus groups   focus group sessions expanded on feedback provided in the survey, with particular emphasis on participants’ feelings about their interactions and experiences with booth library. our focus group protocol is in appendix b.   in order to create a safe, welcoming environment for participants, previous researchers have recommended that focus group facilitators and notetakers be of the same race as the population being studied (chapman et al., 2020; folk & overbey, 2019). it can also be helpful for a non-library employee to facilitate focus groups, both to serve as a neutral figure with whom participants can feel open to discuss their positive and negative experiences and to minimize library employee biases from influencing outcomes (becher & flug, 2005; wahl et al., 2013).   understanding this as the ideal scenario, we had discussed employing two african american graduate assistants as our moderators with the director of our university’s office of inclusion and academic engagement. after receiving a solicitation from a professor seeking opportunities for graduate students from the college student affairs (csa) program’s research methods course to gain experience in applying methodology to a real project on campus, we switched gears. seeing a chance to directly advance students’ academic and professional goals, we offered to take on a team to assist with our study. we were then assigned a group consisting of one african american and two white students. because their course objective was to conduct a qualitative research project, these students developed the focus group protocol with our recommendations. in reviewing the survey results together, we collaboratively agreed to focus on students' socio-emotional experiences with booth library to elucidate participants' perceptions of the library as a welcoming place and how the library can be improved to be more welcoming.   the research team planned four 60-minute focus groups, with the aim to have 3–5 students in each group. potential participants were identified from self-selected volunteers in the pool of survey respondents. in an attempt to boost participation, sessions were offered mid-semester in spring of 2022 across multiple days at different times of day. after considering several spaces around campus, these sessions were held in the faculty reading room of booth library, because it provided a semi-private space with ample lighting. all participants received a $25 walmart gift card. demographic information was collected as a written questionnaire at the start of the focus group. to ensure confidentiality, all participants chose a pseudonym for themselves. focus group discussions began with questions about how participants currently use the library. questions then explored participants’ perceptions of representation within the library. the final questions sought recommendations for improvement of library services.   sessions were audio recorded with participant consent. a csa graduate student created full transcripts of the recordings using the free online transcription service temi and corrected by hand. csa graduate students independently reviewed the transcripts using in vivo coding, a method of qualitative data analysis using the actual language and terminology spoken by the participants as opposed to researcher-derived codes. with input from the graduate students, the librarians identified major themes based on the frequency with which concepts came up in discussion. repetition is a common theme-recognition technique (guest et al., 2012; ryan & bernard, 2003), and we used it to identify recurring topics that emerged between participants within a focus group and across focus group sessions.   results   survey   we had 70 respondents to our survey. of those, 52 were fully completed surveys; 11 were completed through either part i (library resources), part ii (interactions with people in the library), or part iii (library atmosphere and outreach); six were only completed through the initial qualifying question (are you an eiu undergraduate student who identifies as african american/black?); and one responded “no” to the initial qualifying question. we included in our analyses the 52 fully completed surveys and the 11 surveys that were fully completed through part i, part ii, or part iii, for a response rate of 5.2% (63 of 1,217 students to whom the survey was sent). responses overall were favorable, with a few areas noted for improvement.   demographics   all respondents were traditional-aged college students between the ages of 17 and 25, with a spread across student classification—freshmen (19%), sophomores (25%), juniors (25%), and seniors (31%)—and an almost even distribution among a range of majors (grouped by college and discipline), with slightly higher responses from stem majors and just one education major. by comparison, about 90% of all undergraduate degree-seeking students at eiu are of traditional college age. our participants were of slightly higher student classification than the average eiu undergraduate identifying as african american. participants in our study were slightly more likely to be an arts & humanities or business major and slightly less likely to be an education, social science, or stem major as compared to the overall population of eiu undergraduate students who identify as african american (eastern illinois university, 2021).   more women than men responded to the survey (83% vs. 17%), and just below half (46%) were first-generation college students. this compares to 58% women and 42% men overall at eiu who identify as african american undergraduate students (eastern illinois university, 2021). first-generation enrollment of african american undergraduates at eiu overall (52%) is slightly higher than our participant demographic (eastern illinois university, 2020). the respondents were frequent library users, with 92% visiting the library more than once a month (48% reported visiting weekly and another 25% visiting even more often), and the most common duration of visit being 1–2 hours (55%) followed by 3–4 hours (39%). most eiu students, faculty, and staff responding to our latest patron satisfaction survey reported physically visiting booth library once a week (54%, or 93 of 171 respondents), followed by once a semester (21%) and once a day (15%); “more than once a month” was not a response option (booth library, 2022).   library resources   for this set of questions, the library scored highest on meeting students’ needs for study spaces and course-related information (93% rating either strongly agree or somewhat agree on each question), closely followed by technology and non-course-related information (figure 1).   notably, in response to the open-ended question about their most memorable experience at booth library, 49% stated that studying in the library, individually or in a group, was particularly memorable for them. instruction programs (e.g., orientations, workshops, and class visits) received the lowest score, with 67% of those surveyed finding the library’s instructional offerings sufficient for effective use of its services.   figure 1 survey results for library resources. percent and number of responses strongly or somewhat agreeing that needs are met.   interactions with people in the library   responses to questions asking about interactions with people in the library were mostly positive. library employees were found to be welcoming (84%) and able to provide assistance (84%, figure 2). six of 47 students in the survey mentioned research and other help from staff as a memorable experience.   figure 2 survey results for interactions with people in the library (employees and users). percent and number of responses strongly or somewhat agreeing.   sixty percent of respondents also indicated that library employees did not express discomfort in their body language or treat them differently from other students. another question addressed microaggression, using the merriam-webster (2023) definition of “a comment or action that subtly and often unconsciously or unintentionally expresses a prejudiced attitude toward a member of a marginalized group, such as a racial minority.” twenty percent of those surveyed indicated that they had experienced microaggression from a library employee; 66% reported not experiencing microaggression. most interesting is the range of responses to the question of representation: while 36% agreed that people of their race held positions in the library, the same percentage of respondents disagreed with the statement, with the rest neither agreeing nor disagreeing.   library atmosphere and outreach   in terms of environment, respondents perceived the library as a welcoming place; between 80–90% reported feeling personally comfortable, safe, and that they have a sense of belonging when in the building (figure 3). sixty-seven percent viewed the library as being friendly to african american students as a whole.   figure 3 survey results for library atmosphere and outreach. percent and number of responses strongly or somewhat agreeing.   the library fared less well on outreach to the african american community on campus. more respondents agreed (67%) than disagreed (14%) that the library recognizes african american students through displays and programs, but only 40% saw themselves reflected in marketing efforts, with 23% finding the library less than adequate in this regard.   focus groups   of the 40 survey respondents who indicated willingness to participate in follow-up focus groups, a total of 9 self-selected students attended across 3 sessions. after gathering demographic data, participants were asked to discuss six questions probing their perceptions of welcomeness, delving deeper to illuminate responses from each of the three sections in the survey.   demographics   participants were traditional-aged college students, ranging between 18 and 24 years old. more seniors (4 of 9 participants) attended, along with 1 freshman, 3 sophomores, and 1 junior. focus group members represented all academic areas of our institution: business (3 participants), stem (2), allied health and human services (1), arts and humanities (1), education (1), and social sciences (1). eight women and one male student participated, and about half were first-generation students (5 of 9). our focus group members were frequent library users: one visited the library building more than once a week, five visited weekly, and three visited more than once a month. participants most commonly reported staying in the library for 1–2 hours each visit (8 of 9 students); one participant typically spent 3–4 hours.   themes   the findings from the focus group sessions largely supported and expanded on our survey results. from these interviews, five themes emerged.   (1)    use of booth library by participants was chiefly for studying and access to technology.   participants used the library primarily for studying and printing. five of the nine participants came to booth library for quiet, independent study. tiffany explained, “i like to sit [downstairs in the stacks] because i don’t like being seen in the library…. i just like to be [in] my own zone and, plus, distractions as well.” two participants indicated they studied in a group when they came to the library. one didn’t study in the library but used the library’s study rooms for student club meetings.                several of the participants who used booth library almost exclusively as a place for quiet, focused study occasionally experienced issues related to noise levels with other users. one student attempted to participate in an online class and realized her speaking up in class bothered another group studying nearby in the library. she has since joined class from the more social floor of the library and has found that to be a solution to her noise volume dilemma.   access to technology was another significant reason for coming to the library. participants used the computers and, even more so, the printers. some mentioned that they or a friend of theirs benefited from the laptop checkout service available through the library’s center for student innovation, especially when their personal laptop was malfunctioning and they had an assignment they needed to complete.   despite high ratings in the survey, focus group participants made little to no mention of information resources. only one participant talked about her experience with library instruction. two participants in separate focus groups mentioned regularly using the library’s book collection for non-academic reading. two participants across our focus groups were appreciative of the snacks that the office of civic engagement and volunteerism provided in the library and would like to see snacks continue to be offered here.   (2)    interactions with library employees were primarily positive, while feelings surrounding booth library were neutral.   participant comments were overwhelmingly positive regarding the assistance they received from library employees, including student workers. some mentioned instances of exceptional help from employees. michael stated, “the staff members went above and beyond. if they couldn’t find a book in the system, they would even go, still, check the shelves to see if the book was there.” suzy, another participant, mentioned library instruction:   one of the librarians for our class came in [and] gave us a tour and showed us how to use the resources in the library. and i’ve spoken to some other friends…and they said that never happened for them. so they didn’t really know how to use it, but that’s helped me personally.   while participants had generally positive experiences, their overall attitude was neutral in that they didn’t feel actively supported by the library. ophelia observed:   compared to my other experiences on campus, i would say that i feel safe in the library, cuz i’ve had bad experiences out there. but in here it’s just, it’s neutral i’d say. but it’s still better than bad experiences.   royal shared the sentiment, noting, “i haven’t experienced anything negative because of my race while i’m here. but it’s just like, there’s nothing really to show that support. there’s no actual, like anything to show appreciation.”   participants went on to share ideas for making the library feel more welcoming to african american students, and these suggestions are explored in the remaining themes from our focus groups.   (3)    seeing african american employees and students in booth library will attract more african american students to use the library.   the focus group discussions elucidated the mixed response to the question of representation on the survey: some see student workers of their race, but the absence of african american faculty and staff is noticeable. in one focus group, the participants speculated that the applicant pool might not be very racially diverse for full-time employees in this semi-rural midwestern community. they also commented on how seeing more african american students as users in the library would make the library feel more welcoming. alice said, “i would probably come here more, maybe, if i had seen more black staff here…. cuz most of the time when i come here, i don’t really see many black staff.” tiffany also commented:   …i know some people, they be intimidated to go to the libraries. when i meet some students, i mean us black students, it’s because they see there’s not people that look like them in the library…that’s studying and doing work, as well. so i feel like we just get that. we bring more of us in the library. and like i said before, it just make a more comfortable environment.   (4)    african american culture should be celebrated regularly with displays and programming throughout the year, not just during black history month.   displays for black history month were identified as the library’s chief or only acknowledgment of their race. one student commented on the survey that the scarcity of recognition aside from black history month is “another reminder that our history and culture is tolerated, not welcomed.” focus group participant alice added:   it would be nice if they would just do all through the year and not just only for black students, but everybody…. it doesn’t have to be a month in order for them to put out stuff and really show they support.   these students would like to see displays celebrating african american culture throughout the year, or a permanent collection of african american authors that students could browse. one participant suggested incorporating more modern, possibly student, artwork in the library. participants suggested several interactive social activities to make booth library feel more inviting, such as bringing in comfort animals, having competitions that utilize technology available in the library, hosting book clubs, partnering with african american student groups to facilitate study tables, or organizing events on african american culture for the greater campus community.   (5)    continual partnerships with african american student groups will help the library gain input on student needs and interests.   participants suggested collaborating with student organizations on promotional efforts and visiting them where they are, for example at residence hall meetings and student organization meetings. participants said they would like to see more advertising of library resources and events, especially via social media. snapchat and instagram were the most frequently mentioned by name as being regularly used by students. social media is preferred over email and flyers posted in the campus union and residence halls, although they recognize the usefulness of multiple modes of communication. they also recommended advertising on portable a-frame sidewalk signs outside the library’s entrances.   discussion   similar to the findings of recent studies (chapman et al., 2020; stewart et al., 2019), our participants reported generally positive experiences with their academic library. interestingly, despite our efforts to cast a wide net with recruitment for our study, participants were chiefly heavy library users, which aligns with participant profiles from previous research in this area (folk & overbey, 2019; stewart et al., 2019). due to a low response rate, our findings cannot be generalized to our campus population of african american undergraduates. we also were unable to explore causal connections between feelings of welcomeness and how those perceptions influence whether and how students use the library. instead, these findings can begin to inform future work in continuing to seek feedback from this student population, as well as shape outreach to additional underrepresented groups. participant suggestions—such as for the library to regularly seek student input, host engaging collaborative events, and diversify employees—will help to improve a sense of welcomeness for this subset of students in their use of the library.   predominant use of the library by our study’s participants was for meeting personal academic needs. a subset of participants did indicate, however, that they would benefit from more instruction on using library resources, a finding in line with previous research exploring the library needs of students of color (elteto et al., 2008). this outcome can serve to embolden our liaison librarians in their outreach to disciplinary faculty and student support groups, which will help with library messaging in communicating the value of and need for information literacy instruction. like shoge (2003), students in our study primarily used the library for studying and accessing technology. many came for quiet, focused study, but some felt safest when going to the library with other african american undergraduates, which mirrors whitmire’s (2006) finding that african american students will study with fellow african american students for companionship, but with white students primarily when they have a study group or study partner. while libraries have evolved in recent decades into more social spaces of engagement (seal, 2015), feedback from participants in our study underscored the value in continuing to maintain quiet spaces that students can use for focused study.   in addition to feeling safer when there was visible diversity among our library users, students in our research noted that the lack of diversity among library employees detracted from feelings of welcomeness. limited racial and ethnic diversity in the library profession has been reported in earlier studies (e.g., elteto et al., 2008; stanley, 2007; welburn, 2010). the whiteness of the library profession has implications for the quality of library services, such as with librarian approachability, responsiveness, and objectivity in reference transactions as we consider the needs of users of color (brook et al., 2015). despite a growing awareness of the problems associated with the embedded nature of race in our profession, even with aspirations by many to effect change, the lis field has much work to do to course-correct the impact of racism in libraries and library systems (crist & clark/keefe, 2022; schlesselman-tarango, 2017).   students mostly reported having affirming interactions with library employees, although a noticeable minority reported experiencing microaggression while at the library, a finding not unique to our study (folk & overbey, 2022; stewart et al., 2019). however, while participants in our study indicated their experience with microaggression in their interactions was with library employees, other researchers reported on microaggression stemming primarily from interactions with other library users. our study did not ask specifically about microaggression from library users, only by library employees. to address this feedback, our staff development committee organized a staff retreat on privilege. by providing such trainings within the library, rather than turning to opportunities open to the campus at large, the discussion can focus on the unique position of the academic library in students’ lives.   since the time of this study, our library hired a first-year experience/student success (fye/ss) librarian. this new position allows us to better leverage relationships with students beyond their academic pursuits, an area where the traditional subject liaison model can fall short. a significant role of the fye/ss librarian is to foster connections between student groups with an emphasis on promoting diversity and inclusion in our library services. our fye/ss librarian is collaborating with student groups to develop book displays that are relevant to our students. we have incorporated feedback from this study to develop library events that are more welcoming to african american students, such as creating events that are team-based and competitive, which have attracted racially diverse participation. as well, supported by the recommendation of our study’s participants, our fye/ss librarian has launched our first formal student advisory group, comprised of members representing our diverse student population, in order to seek continuous input from our stakeholders.   while library experiences were primarily positive, focus group participants shared that our library felt like a neutral place for them, and they expressed a desire for more proactive efforts to affirm the library’s support and appreciation of underrepresented students. this is in line with past research in which black students identified the academic library and its services as neutral territory (chapman et al., 2020; folk & overbey, 2022), an outcome that marks libraries as “complicit in their silence” (chapman et al., 2020, p. 12). displays and programming celebrating african american culture, along with more diverse representation in the library’s marketing materials, will foster efforts to make the library a “third place” in the lives of these students (oldenburg, 1997; whitmire, 2004). affirming this study’s results, the library has contributed to the development of, and now hosts, race chat events, which are open forums for the campus community that use reflective structured dialogue to discuss lived experiences and better understand participants’ views.   we thought we might hear more mixed feedback from participants, since it is common to hear both strong positive and strong negative feedback when soliciting suggestions for improvement, so we felt encouraged that participants had considerably more positive experiences to convey. our study did not have the reach we were hoping, and it is possible that students who have significant negative perceptions of the library did not contribute to our study. as we formulated this research project, we were eager to be able to implement any recommended changes, in order to instill trust between participants and their library. we were concerned that the suggestions we would receive would be resource prohibitive to implement. instead, students offered very achievable solutions to make the library more welcoming. future outreach to african american students who are non-library users, alongside continuing conversations with users, will enrich library progress toward serving as a welcoming campus resource for our african american students.   this study adds to the existing literature on african american students’ perceptions of feeling welcome in academic libraries by detailing the experiences of undergraduates at a semi-rural, regional comprehensive university. our use of focus groups as a methodology builds on previous studies that have explored african american student experiences with their academic libraries via surveys or one-on-one interviews (elteto et al., 2008; folk & overbey, 2019; stewart et al., 2019). focus groups allowed our participants to consider the experiences of their peers and develop shared recommendations to improve library services.   limitations   sample size was a limitation of both the survey and the focus groups. for the survey, only 63 complete or partial responses were received from soliciting a potential audience of more than 1,200 african american undergraduate students. of the 40 survey respondents who indicated willingness to participate in a focus group, only 18 volunteered when contacted by our csa graduate students, and of those, only nine attended the sessions. also, with only one survey respondent indicating they had not used booth library resources or services, we were unable to solicit representative feedback from non-library users in this study.   in all aspects of these limitations, a higher rate of participation might have been achieved by developing stronger interpersonal relationships between the study’s researchers (us) and campus leaders we identified to help recruit participants. our primary line of connection with many of our campus leaders was via email. while all were willing to share communications about our study, they may have taken a more active role in recruitment had we met each leader in person during the early stages of our outreach. this might assist, in particular, with recruiting non-library users. indeed, non-library users were a group from whom we had hoped the most to hear, to gain their valuable perspectives as to how the library can be made a more welcoming place for their academic and personal pursuits.   future research   the results from our study establish a starting point for future research. we recommend involving and collaborating with members of the relevant student organizations from the start and throughout the process to achieve a richer outcome. building relationships with these student groups would also help with promoting participation. as we largely relied on email and flyers to reach prospective participants, we further recommend making greater use of social media to publicize the study.   conclusion   this study contributes to the growing body of research on african american student experiences in the academic library. while the response rate to our survey was low, participants reported mostly positive feelings associated with their use of booth library. focus group discussions allowed participants to share their experiences and build off one another’s ideas in providing input. the academic library can become a more welcoming physical place for these undergraduates by featuring student artwork and spotlighting african american themes in displays throughout the year. the academic library as a social space can be more welcoming in several ways. hiring and retaining racially diverse employees will improve visible representation, and ensuring employees are trained on topics of diversity and equity will increase staff awareness. as well, the library should host events that celebrate african american culture and create opportunities for congregation such as through competitive and group activities. the library should be proactive in developing relationships with african american student organizations, collaborating with them in developing exhibits and events and continuously seeking their feedback. facilitating these opportunities for connection will improve the overall college experience for african american undergraduates and encourage their academic success. future studies should begin by building strong relationships with leaders of african american social groups on campus in order to increase the recruitment of research participants.   by involving students in the library’s planning process, academic libraries are better able to strategically address the voiced desires and unmet needs of their african american student users. while making strides to resolve the more systemic issues of racism in the library field, not the least of which is helping to create opportunities to diversify the profession, academic librarians can endeavor to make the library a more welcoming space for african american students.   acknowledgements   the authors wish to thank dr. catherine polydore for her feedback on survey design and dr. michael gillespie for his assistance with survey design and data analysis. the authors acknowledge dionne lipscomb, jacob mueller, and madeline reiher for their help in designing and leading the focus groups in this study. we thank don jason, sarah l. johnson, and amy odwarka for their valuable feedback on drafts of this article. this study was supported by the eiu foundation’s redden fund for the improvement of undergraduate instruction.   author contributions   kirstin duffin: conceptualization (equal), funding acquisition (lead), investigation (equal), methodology (equal), project administration, visualization (supporting), writing – original draft (lead), writing – review & editing (equal) ellen corrigan: conceptualization (equal), funding acquisition (supporting), investigation (equal), methodology (equal), visualization (lead), writing – original draft (supporting), writing – review & editing (equal)   references   bass, s. a. (2023). redesigning college for student success: holistic education, inclusive personalized support, and responsive initiatives for a digitally immersed, stressed, and diverse student body. change: the magazine of higher learning, 55(2), 4–13. https://doi.org/10.1080/00091383.2023.2180273   becher, m. l., & flug, j. l. (2005). using student focus groups to inform library planning and marketing. college & undergraduate libraries, 12(1–2), 1–18. https://doi.org/10.1300/j106v12n01_01   booth library. (2020). 2020–2025 strategic plan. eastern illinois university. https://thekeep.eiu.edu/lib_strategic_plan/1/   booth library. (2022). library patron satisfaction survey [unpublished data]. eastern illinois university.   borrelli, s., su, c., selden, s., & munip, l. (2019). investigating first-generation students’ perceptions of library personnel: a case study from the penn state university libraries. performance measurement and metrics, 20(1), 27–36. https://doi.org/10.1108/pmm-07-2018-0018   brook, f., ellenwood, d., & lazzaro, a. e. (2015). in pursuit of antiracist social justice: denaturalizing whiteness in the academic library. library trends, 64(2), 246–284. https://doi.org/10.1353/lib.2015.0048   brooms, d. r. (2018). exploring black male initiative programs: potential and possibilities for supporting black male success in college. journal of negro education, 87(1), 59–72. https://doi.org/10.7709/jnegroeducation.87.1.0059   chapman, j., daly, e., forte, a., king, i., yang, b. w., & zabala, p. (2020). understanding the experiences and needs of black students at duke. duke university libraries. https://hdl.handle.net/10161/20753   crist, e. a., & clark/keefe, k. (2022). a critical phenomenology of whiteness in academic libraries. the journal of academic librarianship, 48(4), 102557. https://doi.org/10.1016/j.acalib.2022.102557   eastern illinois university. (2020). eastern illinois university key publication facts. https://www.eiu.edu/ir/2020%20eiu%20key%20publication%20facts.pdf   eastern illinois university. (2021). fall enrollment tables. https://www.eiu.edu/ir/fall%202021%20fes%20part%20ii.pdf   elteto, s., jackson, r. m., & lim, a. (2008). is the library a “welcoming space”? an urban academic library and diverse student experiences. portal: libraries and the academy, 8(3), 325–337. https://doi.org/10.1353/pla.0.0008   folk, a. l., & overbey, t. (2019). narratives of (dis)engagement: exploring black/african-american undergraduate students’ experiences with libraries. in d. m. mueller (ed.), recasting the narrative: the proceedings of the acrl 2019 conference, april 10–13, 2019, cleveland, ohio (pp. 375–383). association of college and research libraries. http://www.ala.org/acrl/sites/ala.org.acrl/files/content/conferences/confsandpreconfs/2019/narrativesofdisengagement.pdf   folk, a. l., & overbey, t. (2022). narratives of (dis)engagement: exploring black and african american students’ experiences in libraries. ala editions.   guest, g., macqueen, k. m., & namey, e. e. (2012). applied thematic analysis. sage.   kim, j.-a. (2017). user perception and use of the academic library: a correlation analysis. the journal of academic librarianship, 43(3), 209–215. https://doi.org/10.1016/j.acalib.2017.03.002   kracker, j., & pollio, h. r. (2003). the experience of libraries across time: thematic analysis of undergraduate recollections of library experiences. journal of the american society for information science and technology, 54(12), 1104–1116. https://doi.org/10.1002/asi.10309   lilburn, j. (2017). ideology and audit culture: standardized service quality surveys in academic libraries. portal: libraries and the academy, 17(1), 91–110. https://doi.org/10.1353/pla.2017.0006   merriam-webster. (2023). microaggression. in merriam-webster.com dictionary. https://www.merriam-webster.com/dictionary/microaggression   murphy, m. c., gopalan, m., carter, e. r., emerson, k. t. u., bottoms, b. l., & walton, g. m. (2020). a customized belonging intervention improves retention of socially disadvantaged students at a broad-access university. science advances, 6(29), eaba4677. https://doi.org/10.1126/sciadv.aba4677   murphy, m. c., & zirkel, s. (2015). race and belonging in school: how anticipated and experienced belonging affect choice, persistence, and performance. teachers college record: the voice of scholarship in education, 117(12), 1–40. https://doi.org/10.1177/016146811511701204   oldenburg, r. (1997). the great good place: cafés, coffee shops, community centers, beauty parlors, general stores, bars, hangouts, and how they get you through the day (2nd ed.). marlowe & co.   ryan, g. w., & bernard, h. r. (2003). techniques to identify themes. field methods, 15(1), 85–109. https://doi.org/10.1177/1525822x02239569   sare, l., bales, s., & budzise-weaver, t. (2021). the quiet agora: undergraduate perceptions of the academic library. college & undergraduate libraries, 28(1), 18–36. https://doi.org/10.1080/10691316.2020.1850384   schaller, s. (2011). information needs of lgbtq college students. libri: international journal of libraries & information services, 61(2), 100–115. https://doi.org/10.1515/libr.2011.009   schlesselman-tarango, g. (ed.). (2017). topographies of whiteness: mapping whiteness in library and information science. library juice press.   seal, r. a. (2015). library spaces in the 21st century: meeting the challenges of user needs for information, technology, and expertise. library management, 36(8/9), 558–569. https://doi.org/10.1108/lm-11-2014-0136   seale, m. (2017). efficiency or jagged edges: resisting neoliberal logics of assessment. progressive librarian, 46, 140–145. http://www.progressivelibrariansguild.org/pl/pl46/140seale.pdf   shoge, r. c. (2003). the library as place in the lives of african americans. in learning to make a difference: the proceedings of the acrl 11th national conference, april 10–13, 2003, charlotte, north carolina. association of college and research libraries. http://hdl.handle.net/11213/17608   stanley, m. j. (2007). case study: where is the diversity? focus groups on how students view the face of librarianship. library administration & management, 21(2), 83–89. https://hdl.handle.net/1805/1440   stewart, b., ju, b., & kendrick, k. d. (2019). racial climate and inclusiveness in academic libraries: perceptions of welcomeness among black college students. the library quarterly, 89(1), 16–33. https://doi.org/10.1086/700661   stewart, d. w., shamdasani, p. n., & rook, d. w. (2009). group depth interviews: focus group research. in l. bickman & d. j. rog (eds.), the sage handbook of applied social research methods (2nd ed., pp. 589–616). sage.   u.s. census bureau. (2021). 2020 decennial census redistricting data (p.l. 94-171): table p1: race: charleston city, illinois. u.s. department of commerce. https://data.census.gov/table?q=population+in+charleston,+illinois+in+2020&tid=decennialpl2020.p1   wahl, d., avery, b., & henry, l. (2013). studying distance students: methods, findings, actions. journal of library & information services in distance learning, 7(1–2), 183–209. https://doi.org/10.1080/1533290x.2012.705656   walton, g. m., & cohen, g. l. (2007). a question of belonging: race, social fit, and achievement. journal of personality and social psychology, 92(1), 82–96. https://doi.org/10.1037/0022-3514.92.1.82   welburn, w. c. (2010). creating inclusive communities: diversity and the responses of academic libraries. portal: libraries and the academy, 10(3), 355–363. https://doi.org/10.1353/pla.0.0107   whitmire, e. (1999). racial differences in the academic library experiences of undergraduates. journal of academic librarianship, 25(1), 33–37. https://doi.org/10.1016/s0099-1333(99)80173-6   whitmire, e. (2003). cultural diversity and undergraduates’ academic library use. journal of academic librarianship, 29(3), 148–161. https://doi.org/10.1016/s0099-1333(03)00019-3   whitmire, e. (2004). the campus racial climate and undergraduates’ perceptions of the academic library. portal: libraries & the academy, 4(3), 363–378. https://doi.org/10.1353/pla.2004.0057   whitmire, e. (2006). african american undergraduates and the university academic library. journal of negro education, 75(1), 60–66. https://www.jstor.org/stable/40026504   wiegand, w. a. (2005). critiquing the curriculum. american libraries, 36(1), 58, 60–61.   wiegand, w. a. (2011). main street public library: community places and reading spaces in the rural heartland, 1876-1956. university of iowa press.   wiegand, w. a. (2015). part of our lives: a people’s history of the american public library. oxford university press.   appendix a survey questions   african american/black undergraduate perceptions of booth library as a welcoming place   are you an eiu undergraduate student who identifies as african american/black? “yes” – move on to survey. “no” – thank participant for their time and exit survey.   your responses will be kept anonymous. if you wish to participate in further research on this topic, you will have the option to add your name and e-mail at the end of this survey.   part i: have you used booth library resources? (for example: books, articles, technology, study spaces) “yes” – move on to question block. “no” – why have you not used booth library resources? [open text response]   for this set of questions, think about the resources available at booth library in relation to your race. response options: strongly agree / somewhat agree / neither agree nor disagree / somewhat disagree / strongly disagree / not applicable booth library provides information resources that meet my course-related needs (via books, journals, databases, etc.). booth library provides information resources that meet my non-course-related information needs (for example, news/current events, health, career, leisure). booth library provides sufficient instruction programs (for example class visits, workshops, orientations) for effective use of its services. booth library provides sufficient access to technology for my needs (for example computers, software programs, printers, etc.). booth library provides adequate study spaces.   part ii: have you interacted with people in booth library? (library employees or library users) “yes” – move on to question block. “no” – why have you not interacted with library employees or library users in booth library? [open text response]   for this set of questions, think about your interactions with people in booth library in relation to your race. response options: strongly agree / somewhat agree / neither agree nor disagree / somewhat disagree / strongly disagree / not applicable library employees provided sufficient assistance to meet my needs. library employees were welcoming toward me. library employees were unfriendly to me. library employees’ body language showed they felt uncomfortable with me. library employees treated me differently from other students. i have experienced microaggression* from library employees. *microaggression is defined as a comment or action that subtly and often unconsciously or unintentionally expresses a prejudiced attitude toward a member of a marginalized group (such as a racial minority). other library users’ body language and/or actions showed they were uncomfortable with me because of my race. people of my race hold positions in booth library.   part iii: have you visited booth library? “yes” – move on to question block. “no” – why have you not been to booth library? [open text response] and questions 17–20 (see below).   for this set of questions, think about the atmosphere at booth library in relation to your race. response options: strongly agree / somewhat agree / neither agree nor disagree / somewhat disagree / strongly disagree / not applicable i feel like i belong when i’m at booth library. when i walk into booth library, i feel comfortable. i feel safe when visiting booth library. i feel booth library, as a whole, is unwelcoming to african american/black students. i feel booth library, as a whole, is friendly to african american/black students. booth library’s marketing and outreach materials reflect african american/black students. booth library recognizes african american/black oriented events and activities (for example: library displays or exhibits around black history month, #blacklivesmatter, black authors and researchers, etc.).   for this set of questions, think about your use of booth library. how frequently, on average, do you visit booth library? daily more than once a week weekly more than once a month once or twice a semester never   what is the typical duration of your library visit? less than 1 hour 1-2 hours 3-4 hours 5-6 hours more than 6 hours not applicable   what is your most memorable experience at booth library? [open text response]   for this final set of questions, please tell us more about yourself. i identify as: female male non-binary i prefer to self-describe (please specify): [open text response] i prefer not to answer   my age is: [open text response]   i am a first-generation college student: yes no   based on my credit hours, my class rank is: freshman (0-29 credit hours) sophomore (30-59 credit hours) junior (60-89 credit hours) senior (90 or more credit hours)   my overall gpa is: [open text response]   broadly speaking, my intended major falls within (select all that apply): allied health & human services arts & humanities business education social sciences sciences, technology & math undecided other/unsure (please specify): [open text response]   if you are interested in participating in a 60-minute group discussion on this topic, please include your name and email below. if chosen, we will reach out to you later this month to schedule the session. discussion group participants will receive a $25 gift card as compensation for their time. your name: [open text response] your email: [open text response]   would you like to enter the raffle to win a $15 walmart gift card? “yes” – redirect to raffle survey. “no” – thank participant for their time and exit survey.   raffle survey:   if you would like to be entered into a drawing to win one of twenty $15 walmart gift cards, please include your contact information below. your name: [open text response] your email: [open text response]   end survey.     appendix b focus group protocol   participants arrive and are given a pseudonym, gift card, and directed into the interview room.   participants are welcomed and made aware that the focus group will be recorded and stored on an eiu password-protected server. informed consent paperwork is explained and participants are reminded of no penalty to withdraw. participants are informed of the process if they opt to withdraw from the focus group. last-minute concerns or questions are addressed, and the focus group begins.   pre-question: do we have your permission to record this interview?   demographic questions (asked via written questionnaire): what are your preferred pronouns? what is your self-identified gender? are you a first-generation student? what is your current academic classification based on credit hours? what is your current intended major? what is your self-identified ethnicity? why did you choose to attend this institution?   booth library perceived welcomeness questions: can you describe how you use booth library? can you describe your experiences with booth library? were they positive or negative? can you identify the causes for that outcome? can you describe your interactions with booth library staff? how did you feel leaving that interaction? can you describe how you feel when you are in booth library? do you feel represented? why or why not? can you describe one area of booth library you feel can be improved upon? why did you pick that area? do you feel booth library supports black students? why or why not? can you provide examples personal to you?   focus group concludes – participants are thanked. recording is saved and filed on password-protected server.     microsoft word news4416 evidence based library and information practice 2008, 3:4 93 evidence based library and information practice news call for papers: the second international m-libraries conference © 2008. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. following the successful first international m-libraries conference <http://library.open.ac.uk/mlibraries/2007/i ndex.html> in 2007 we are delighted to call for papers for the second international conference in vancouver in 2009. for more information visit the conference web site <http://library.open.ac.uk/mlibraries/>. the conference will be hosted by the university of british columbia in collaboration with athabasca university, the open university (uk) and thompson rivers university. the aim of the conference is to explore and share work carried out in libraries around the world to deliver services and resources to users ‘on the move’ via mobile or handheld devices, including mobile and smartphones, pdas, portable gaming devices, mp3 players and e-book readers. the conference will bring together researchers, technical developers, managers and library practitioners to exchange experience and expertise and generate ideas for future developments. if you would like to be involved in this exciting conference please submit your abstracts (up to 300 words) to <m-librariesconference@open.ac.uk> by 15 december 2008. these will be subject to peer review and authors will be notified by mid-january. we expect that papers may include research reports, demonstrations of technical developments, practical case studies or reviews. posters are also invited. themes will include: • service models for library services delivered to mobiles • cost and sustainability factors for mlibrary service development • the changing relationships between libraries and users resulting from innovation in ubiquitous computing • partnership projects for developing integrated services to mobiles • implications of mobile technologies on library space planning • reconfiguring library collection development to enhance ubiquitous access to resources • exploring methodologies for evaluating the impact of mobile and ubiquitous computing on library service development papers or presentations should last no longer than 20 minutes plus 10 minutes for evidence based library and information practice 2008, 3:4 94 discussions. poster sessions posters will be displayed throughout the conference and time will be allocated in the program for poster viewing. a prize will be awarded to the best poster. conference proceedings will be published online. the conference will be held at the university of british columbia campus in vancouver, british columbia. situated on canada's west coast, vancouver is a popular travel destination and is a host city for the 2010 olympic winter games. for more information see <http://hellobc.com/enca/regionscities/vancouver.htm>. contact information general enquiries: <leonora.crema@ubc.ca> program enquiries: <m-librariesconference@open.ac.uk> why not join the m-libraries facebook group? <http://www.facebook.com/group.php?gid= 7719700810> evidence based library and information practice evidence based library and information practice 2011, 6.3 4 evidence based library and information practice article situating student learning in rich contexts: a constructionist approach to digital archives education anthony cocciolo pratt institute school of information and library science new york, new york, united states of america email: acocciol@pratt.edu received: 28 apr. 2011 accepted: 09 may 2011 2011 cocciolo. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective this paper sought to determine whether a constructionist pedagogical approach to digital archives education could positively influence student perceptions of their learning. constructionism is a learning theory that places students in the role of designers and emphasizes creating tangible artifacts in a social environment. this theory was used in the instructional design of the digital archive creation project (dacp), a major component of a digital archives course offered to students enrolled in a master’s program in library science at pratt institute school of library and information science. methods participants were the 31 students enrolled in the dacp during the fall and spring semesters of 2010. they were surveyed as to their perceived learning outcomes as a result of their engagement with the dacp. results results indicated that students perceived strong increases in their learning following their engagement in the dacp, particularly in terms of their skills, confidence, understanding of topics covered in other courses, and overall understanding. factors that influenced these increases include the collaborative teamwork, the role of the facilitator or instructor, and individual effort. conclusion the project demonstrated that a constructionist pedagogical approach to digital archives education positively impacted students’ perceptions of their learning. mailto:acocciol@pratt.edu� evidence based library and information practice 2011, 6.3 5 his entrance is quite undramatic according to conventional standards. nevertheless, this is a subtly contrived, outrageously theatrical effect. no hush falls as george walks in. most of the students go right on talking. but they are all watching him, waiting for him to give some sign, no matter how slight, that the class is to begin. -christopher isherwood, a single man, pp. 56-57 introduction isherwood aptly captured the anticipation associated with the most common and wellknown form of college teaching--the lecture. this teaching method predominates in both popular conceptions of college instruction as well as in practice. despite its ubiquity, it has been criticized at best as a source of youthful boredom, and at worst as inhibiting individual thought and promoting rote learning (brookfield, 2006). to address the potential pitfalls of lecture-based instruction, a host of teaching methods have been developed to engage students more actively in the learning process. under such names as problem-based learning, active learning, and cooperative learning, students engage in instructional activities, rather than listening exclusively to the “sage on the stage” (king, 1993). lectures—when used in these contexts—are short in duration and used to clarify the problems at hand, rather than merely transmitting knowledge (hung, jonassen & liu, 2008). building further upon learner-centered teaching strategies, this project used a constructionist approach to digital archives education. constructionism places the learner in the role of designer. it emphasizes the importance of having a tangible artifact that learners can bring with them to discuss with fellow learners and pertinent others in their social networks (papert, 1980, 1991; kafai, 2006). constructionism should not be confused with constructivism, a related learning theory that describes more generally how learners construct new knowledge from their experiences (dewey, 1910). using constructionism as a theoretical model, the class was structured around the design of a tangible artifact—a digital archive—for a partnering institution. during the digital archive creation project (dacp), students developed a digital archive through engagement in a multitude of tasks, such as digitizing materials, designing and deploying a digital presence, creating metadata, and conducting research. in the final class, students presented the digital archive to the partner institutions for their continued use. this study investigated whether this pedagogical approach, where students solved authentic problems situated in meaningful activities, could positively impact student perceptions of their learning. the project was guided by the notion that librarians and library science educators should regularly be “finding and using the best research possible to aid in our decision making” (koufogiannakis & crumley, 2006). using this evidence-based library and information practice approach, the intent was to uncover the effectiveness of the dacp as an instructional method. theoretical background constructionism constructionism, as described by kafai (2006), “brings to mind the metaphor of learning by constructing one’s own knowledge, and is contrasted to the more traditional ‘instructionism,’ which favors the metaphor of learning by transmission of knowledge” (p. 38). seymour papert, the father of constructionism, described the artifacts that learners create as “objects-to-think-with,” where the designed artifacts “can become objects in the mind that help to construct, examine, and revise connections between old and new knowledge” (kafai, 2006, p. 39). the use of tangible objects that can become objects in the mind is thought to be particularly effective at promoting appropriation, or the process by which “learners make knowledge their own and begin to identify with it” (kafai, evidence based library and information practice 2011, 6.3 6 p. 39). for example, disparate artifacts such as pieces of metadata, digitized and un-digitized materials, content management system source code, metadata schemas, or visual designs, become units that the learner can arrange and re-arrange. understanding how each of these elements works, and assembling them into a coherent whole inspires a sense of “know how.” these real artifacts become objects in the mind, and new connections are formed by working with them. central to the theory of constructionism is the social nature of learning. rather than isolating the learner to construct his or her own knowledge, constructionism “focuses on the connected nature of knowledge with its personal and social dimensions” (kafai, 2006, p. 36). papert (1991) describes how learning “happens especially felicitously in a context where the learner is consciously engaged in constructing a public entity, whether it's a sand castle on the beach or a theory of the universe” (p. 1). rusk, resnick, and cooke (2007) concur, noting that learners construct new knowledge with particular effectiveness when they are “actively engaged in creating something that is meaningful to themselves or to others around them” (p. 19). under this theory, and in the case of the dacp, designing an artifact that was not perceived as socially relevant or personally meaningful to the students would be less efficacious at promoting learning. constructionist theory has been employed in a variety of learning contexts. papert (1980) initially employed constructionism in mathematics education by having students use the logo programming language to engage in creative experimentation and making computer-based artifacts. more recently, constructionism has been used to reframe media education and engage young people in creative production. peppler and kafai (2007) noted that “creative artifacts can then be used in the classroom for encouraging a critical discussion of new media and the transparency concerns of software interfaces” (p. 7). similarly, constructionist theory has been used to teach students computer programming by designing artifacts using scratch, a visual programming environment (maloney, peppler, kafai, resnick & rusk, 2008). it has also guided the underlying theory of “the computer clubhouse,” intel’s after-school program. in this program youth engage in creative technology production in a social environment (rusk, resnick & cooke, 2009). situated learning situated learning suggests that learning is best promoted by situating learners within a meaningful context where "'cognition' is constituted in dialectical relations among people acting, the contexts of their activity, and the activity itself” (lave, 1988, p. 148). like constructionism, situated learning implies that students should see their development as personally enriching and of value to the individuals with whom they are collaborating and the communities they are serving. for the educator, the role becomes more focused on creating an authentic context where learners can readily see how their work is having an impact on their environment (lave & wenger, 1991). lave and wenger (1991) drew attention to the collaborative aspects of work processes, particularly to legitimate peripheral participation, which describes a process by which a newcomer becomes a part of a community of practice by gradually taking on the role of the expert. this emphasis on collaborative work indicates that learners acquire new skills from each other as much as from experts, and in doing so they gradually become more adept. developing learning environments to support the formation of communities of practice has been used in a variety of contexts. for example, it has been used to reconceptualize teacher professional development to shift the focus from teachers acquiring new knowledge to focusing on teachers participating in shared practice (barab, barnett & squire, 2000). in library and information science education, the theory has been used to support adult learning related to core library science concepts, practices, values, and leadership skills (yukawa, 2010). evidence based library and information practice 2011, 6.3 7 problem-based learning problem-based learning (pbl) is an instructional method where students learn by solving authentic, ill-structured problems (barrows, 1985; hung, jonassen & liu, 2008). pbl is student-centered, and “knowledge building is stimulated by the problem and applied back to the problem” (hung, jonassen & liu, 2008, p. 488).instructors act as “facilitators (not knowledge disseminators) who support and model reasoning processes, facilitate group processes and interpersonal dynamics, probe students’ knowledge deeply, and never interject content or provide direct answers to questions” (hung, jonassen & liu, 2008, p. 489). pbl has been used in a variety of contexts, most notably and extensively in medical education (barrows, 1985). it has also been used in a variety of other contexts, such as teacher education and in developing problembased learning curricula for use with children. hung, jonassen, and liu (2008) reviewed the contexts and outcomes of problem-based learning implementations in k-12 education, noting that pbl has shown positive effects in a variety of content areas, including mathematics, science, literature, history, and microeconomics. they further noted that pbl has proved effective in urban, suburban, and rural communities, and with a variety of student groups, including both low-income and gifted students. in library and information science education, pbl has been used in the teaching of a basic reference course (baker, 1999) and in an online course curriculum (yukawa, 2007). instructional design the dacp was used in two 15-week semesters during the year 2010, each with a different partner institution. for the first semester (spring 2010), the partner organization was the dalton school, a private k-12 school based in manhattan, new york. the school had begun an oral history project to capture the school’s unique history and its grounding in progressive educational philosophy. the oral history project included 35 interviews from faculty, students, and staff recorded on audiocassette. the project goal was to make this material available to the dalton community via a web-based digital archive. in the second semester (fall 2010), the class partnered with the lesbian herstory archive (lha), the world’s largest and oldest archive of material about lesbians and lesbian communities, located in brooklyn, new york. again the goal was to digitize a collection of spoken word material available on 35 audiocassettes, and to make the content generally available on the internet. notable in this audio collection were oral history interviews with mabel hampton (1902-1989), a lesbian activist who was involved in the harlem renaissance and could speak of the lives of black women and lesbians during that period (nestle, 1997). the dacp employed each of the three theoretical commitments, primarily constructionism and secondarily situated learning and problem-based learning. situated-learning was used from the onset by creating a meaningful context for the students’ work. this was accomplished during the initial class session by clearly communicating to students the needs and interests of the partner organization for creating a digital archive. the use of context was further augmented in later weeks by a site visit to the archive. in the case of the lesbian herstory archive project, one of the class sessions was held at the archive and included a tour. similarly, a group of students visited the dalton school to do further research in the archives. in these site visits, students gained an understanding of the potential benefits of the digital archives. both organizations were concerned about the degradation of audiotape, especially as some of them were over thirty years old. additionally, students learned how the archives hoped to make the material more widely accessible by having it available digitally over a network. in sum, a meaningful context was created by making it clear to students that their work was needed evidence based library and information practice 2011, 6.3 8 and would be appreciated by a community of users. individual class sessions were organized around a problem-based learning approach, and each week students addressed a series of problems necessary to the continued development of the digital archive. the instructor would pose the problems or highlight what the students thought were problems, but the instructor would not give what could be perceived as “the right answer.” for both the students and the instructor, this was a challenging change of roles, especially in a college classroom where students are more familiar with “the sage on the stage,” and less so with the “guide on the side.” maintaining the facilitator role was essential, because it required students to make use of their research skills to find the most appropriate answers. for students the process required more cognitive and personal investment than simply internalizing correct answers. the course components, such as discussions, course readings and collaborative work, were oriented towards solving the week’s problems. for example, the course readings on topics such as digitizing audio material or creating metadata became crucial to completing the project in a way that coincided with professional practices. students could see how neglecting to keep up with course readings was not only a detriment to themselves, but could disappoint the collective effort and have a related impact on the quality of the project and the communities being served. students were divided into one of five groups (research, metadata, design, technology, or quality assurance) based on interests, and teams would have to address important questions on how to proceed with the project. for example, the metadata team would have to decide what metadata (if any) to create, what standard (if any) to use, and whether to organize items at the item or collection level. the technology team had to determine what content management system (if any) to use, how and where it would be hosted, how the technology supports the metadata schemes, among many other questions. the research team considered the state of the art in digital archives and how to make the archive more engaging to the community. the design team considered how to make the site useable and visually appealing. the quality assurance team devised a plan to ensure the quality of the recordings, metadata, and overall site usability. finally, the teams had to figure out how to work together to deliver a coherent product, as well as how to satisfy partner interests. each class session began with the students digitizing audiocassettes (five tape players were available in the classroom). the instruction lesson proceeded while the digitization process continued. by the end of the class instruction at each session, the digitization would be complete, and the students uploaded the files to a server. at the end of 12 weeks, all 35 tapes had been digitized. as suggested by constructionist theory, the students iteratively built the digital archive and had a tangible artifact to discuss and demonstrate to their greater social network. for example, students shared their work with others using social media (facebook and twitter) and their own personal websites. by the end of the semester, the students had produced a professional quality attractive digital archive. figure 1 is a screenshot of the dalton project, and figure 2 shows a screenshot of the lha project. both digital archives included .wav sound files for preservation, mp3 audio files for listening, and digitized photographs of the interviewees to visually augment the site. in addition, the students cataloged each oral history in detail to improve discovery efforts. for technology, the dalton group chose the omeka content management system, and the lha project chose the collectiveaccess content management system. evidence based library and information practice 2011, 6.3 9 figure 1 digital archive created by students for the dalton school. evidence based library and information practice 2011, 6.3 10 method research questions this study addressed the following research questions: 1. what gains in digital archive skills, confidence, understanding of topics covered in other courses, and overall understanding did the participants perceive as a result of engaging in the dacp? 2. what factors (i.e., collaborative team work, facilitator/instructor involvement, individual effort, or past knowledge) influenced any perceived learning outcomes? participants and method of analysis a t-test of independent samples was conducted to determine if there was a statistically significant difference between the two student groups. as there proved to be no difference, both groups were treated as a single group. figure 2 digital archive created by students for the lesbian herstory archive. evidence based library and information practice 2011, 6.3 11 the participants of this study were all graduate library science students, 15 females and 2 males from the dalton cohort and 13 females and 3 males from the lha cohort. all students received a voluntary survey (appendix) during the final session of the course, and the survey was completed and returned by all but one student in each cohort, yielding n=16 and n=15 responses. the goal of the survey was to collect student perceptions of their learning as a result of their engagement with the dacp. the instructor left the room while the students completed the paper survey. the survey results addressed the project’s research questions. the mean value of students’ perception of their learning outcomes addressed the first research question, and correlations between the perceived learning outcomes and the factors that influenced those outcomes focused on the second research question. the variable value for some categories was formed by student responses to multiple questions. in cases such as this, the value was computed by taking the average student responses across all questions within the category. for example, “skills” was the average value of student responses for questions 4-8 and 18-19. similarly, “overall confidence” was determined by responses to questions 1,16, and 17. “confidence” was evaluated by questions 2 and 3; and “teamwork” was determined by questions 812. the survey also allowed participants to add free-text comments. this qualitative data from these comments provided additional support for the quantitative analysis. results there was no statistically significant difference between the dalton and lha cohort in any of the response categories (p > .05). as a result, these groups could be treated as a single group of respondents who had received the same instructional treatment, but during different periods of time (n=31). research question 1: student perceived learning outcomes students strongly to moderately agreed that the dacp improved their skills, confidence, understanding of topics covered in other courses, and overall understanding of digital archives (table 1). table 1 student perceived learning outcomesa categories mean std. dev. confidence 3.75 0.45 skills 3.53 0.44 understanding of topics covered in other courses 3.52 0.81 overall understanding 3.84 0.28 a scale 1-4, with 4 being “strongly agree” with improvement. research question 2: factors influencing perceived learning outcomes participants strongly to moderately agreed that their perceived learning outcomes were influenced by their collaborative teamwork, the involvement of the facilitator, and the individual effort made (table 2). students moderately to strongly disagreed that they had prior experience creating a digital archive. table 2 student perceived factors influencing learning outcomesb categories mean std. dev. teamwork 3.59 0.47 facilitator 3.84 0.45 individual effort 3.77 0.43 prior knowledge 1.76 1.02 bscale 1-4, with 4 being “strongly agree” with the role the factor played. in addition, perceived increases in skills, confidence, understanding of topics covered in other courses, and overall confidence were positively correlated with teamwork, facilitator involvement, and individual effort evidence based library and information practice 2011, 6.3 12 (table 3). prior knowledge of creating digital archives was positively correlated with increased confidence, but not with the other perceived learning outcomes. the quantitative findings were complimented by the qualitative findings, where students affirmed the importance of the collaborative teamwork (e.g., “i’ve never had such a great group [. . . ] to work with,” increased confidence (e.g., “[it] will definitely help me in the future”), facilitator involvement (e.g.,“pratt needs more teachers like you”), the importance of producing a tangible artifact (e.g., “it’s also awesome to have a final realworld product”), and engaging in meaningful activity (e.g., “really felt like i was working towards something that actually matters”) (table 4). table 3 correlations between the perceived learning outcomes and factorsc perceived learning outcomes factors: confidence skills understanding of topics covered in other courses overall understanding teamwork 0.64** 0.48** 0.40* 0.53* facilitator 0.62** 0.64** 0.69** .83** individual effort 0.67** 0.69** 0.64** .70** prior knowledge 0.42* 0.34 0.15 0.26 cn = 31, * p < .05 **p < .01 (both two-tailed tests) table 4 complete qualitative feedback from student survey cohort comment dalton the digital archives creation project is one of the best things i’ve done at sils and you should definitely do one of these every semester—it was awesome and will definitely help me in the future. dalton great project – real world experience is always the best / even the problems we encountered were informative. dalton best, more applicable project ever. so well organized… it was great. i’ve never had such a great group (my team as well as the rest of the class) to work with. it’s also awesome to have a final real-world product. lha this was a wonderful classespecially having a real-world, group project! i can’t really think of any suggestions… keep up the good work – pratt needs more teachers like you. lha i think formatting this project as a class project with teams having different areas of focus really allowed for a higher quality end product. i’ve had classes where each team is responsible with all aspects and it can lead to patchy results. overall, this class was a success and i’m proud of the work we did. thanks! lha great project. overall thought it was wonderful + learned a lot. a few things i thought might help: more structured project management (timeline, who does what) – fewer people in research (it seemed like our bodies would have been helpful elsewhere, too) – not to say research wasn’t awesome, it just seemed like tech was struggling a bit at times – ability to present the final project to lha – ability to use collective access a bit more (just for experience) evidence based library and information practice 2011, 6.3 13 discussion the results indicate that a constructionist pedagogical approach to digital archives education, where students solve authentic problems situated in meaningful activity and produce tangible artifacts, can positively impact student reports of skills, confidence, understanding of topics covered in other courses, and overall understanding. these perceived learning increases are influenced by the collaborative teamwork, the facilitator, and the effort made by the individual. additionally, for students who already have experience creating digital archives, engaging in such a project was correlated with increased confidence. this finding makes practical sense: the more experience gained doing something, the greater the confidence. results also indicate that the content of the digital collection was less important than the process of engaging in the collective building effort. this was made evident by the finding that there was no statistical difference between the groups who engaged in the lha project and those who engaged in the dalton project. in this case, students did not need to have prior interests in lesbian activism or in private schooling to eventually become engaged in the purposes of the project. using the situated learning perspective, students become engaged in the purpose by engaging with the digitized content, collaborative teamwork, physically immersive site visits, and discussions with stakeholders from the partner institutions. limitations while this study had several strengths, there were also a number of limitations. the sample (n=31) was small, especially for a study using statistical analysis. however, the qualitative feedback affirmed the quantitative results, which further supported the validity. since the results were encouraging, the instructional method can be used again to generate additional data on how the instructional method impacts student perceptions of learning. a further limitation is that the learning outcomes were what the student perceived them to be; that is, they were not measured against an objective standard. further iterations of the study might include preand post-assessments of student understanding and skills. conclusion this study illustrated that a constructionist pedagogical approach, where students solve authentic problems in a meaningful context and produce tangible artifacts, holds great promise for library science education. the method can be used in other lis courses as well, such as ones dealing with digital libraries and digital curation. more generally, it points to the efficacy of learner-centered instruction and the production of artifacts as a way of engaging students. it also suggests a new role for instructors creating environments where meaningful activities can occur. in isherwood’s a single man, george surmised that his students’ curiosity about the subject matter was limited: “huxley, tennyson, tithonus. they’re prepared to go as far as tennyson, but not one step farther. there their curiosity ends. because, basically, they don’t give a . . . . ” (p. 63). he concluded that their lack of curiosity was a result of their not caring. the challenge for educators is to create a social and environmental context, using resources both inside and outside of the classroom, where it is nearly impossible not to care. references baker, l. m., & ling, h. j. (1999). use of problem-based learning in a basic reference course. journal of education for library and information science, 40(1), 66-69. barab, s. a., barnett, m. & squire, k. (2002). developing an empirical account of a community of practice: characterizing the essential tensions. journal of the learning sciences, 11(4), 489-542. evidence based library and information practice 2011, 6.3 14 barrows, h. s. (1985). how to design a problem-based curriculum for the preclinical years. new york: springer. brookfield, s. d. (2006). the skillful teacher: on technique, trust, and responsiveness in the classroom (2nd ed.). san francisco: jossey-bass. dewey, j. (1910). how we think. lexington, ma: d.c. heath. hung, w., jonassen, d. h., & liu, r. (2008). problem-based learning. in j. m. spector, j. g. van merriënboer, m. d., merrill, & m. driscoll (eds. kafai, y. b. (2006) constructionism. in r. k. sawyer (ed.), cambridge handbook of the learning sciences. cambridge: cambridge university press. koufogiannakis, d. & crumley, e. (2006). research in librarianship: issues to consider. library hi tech, 24(3), 324340. lave, j. (1988). cognition in practice: mind, mathematics and culture in everyday life. cambridge: cambridge university press. lave, j. & wenger, e. (1991). situated learning: legitimate peripheral participation. cambridge: cambridge university press. maloney, j., peppler, k., kafai, y. b., resnick, m. & rusk, n. (2008). programming by choice: urban youth learning programming with scratch. sigcse bulletin, 40(1): 367-371. nestle, j. (1997). i lift my eyes to the hill: the life of mabel hampton as told by a white woman. in m. b. duberman (ed.), queer representations: reading lives, reading cultures (pp. 258-277). new york: nyu press. papert, s. (1980). mindstorms: children, computers, and powerful ideas. new york: basic books. papert, s. (1991). situating constructionism. in i. harel & s. papert (eds.), constructionism (pp. 1-11). norwood, nj: ablex. peppler, k. a. & kafai, y. b. (2007). from supergoo to scratch: exploring creative digital media production in informal learning. learning, media and technology, 32(2), 149-166. rusk, n., resnick, m. & cooke, s. (2009). origins and guiding principles of the computer clubhouse. in y. a. kafai, k. a. peppler & r. n. chapman (eds.), the computer clubhouse: constructionism and creativity in youth communities. new york: teachers college press. yukawa, j. (2007). factors influencing online communication style in lis problembased learning. journal of education for library and information science, 48(1), 52-63. yukawa, j. (2010). communities of practice for blended learning: toward an integrated model for lis education. journal of education for library and information science, 51(2), 54-75. acknowledgements i would like to thank maxine wolfe and bronwynee pereira from the lesbian herstory archives, and adele bildersee and tobi fineber from the dalton school, for collaborating with my classes on these projects. i would also like to thank my colleagues at pratt institute for their help on various stages of this project, particularly tula giannini, debbie rabina, cristina pattuelli, david marcinkowski and jeffrey hiban. and lastly, thank you to the students of my courses for their hard work in making these digital archives a reality. evidence based library and information practice 2011, 6.3 15 appendix student final survey this anonymous and voluntary survey is intended to collect your feedback on the digital archive creation project (dacp). your feedback will be used to influence successive iterations of the course and to assess the efficacy of the dacp as an instructional method. scale 1-4 (4 = strongly agree, 1=strongly disagree, or n/a). please feel free to leave comments on the backside of this sheet. thank you for your participation. 1. the dacp improved my understanding of digital archives. 2. i feel more confident to create digital archives from having worked on the dacp. 3. i feel more confident to create an oral history project from having worked on the dacp. 4. the dacp improved my understanding of how to digitize materials. 5. the dacp improved my understanding of how to organize information within a digital archive. 6. the dacp improved my understanding of metadata standards. 7. the dacp improved my understanding of content management systems. 8. the dacp improved my understanding of web-based systems. 9. working with my team on the dacp improved my understanding of the subject matter. 10. i learned things from my dacp team members. 11. i feel that my dacp team members made an effort to advance the dacp. 12. i liked working with my dacp team. 13. working with classmates outside of my team on the dacp improved my understanding of the subject matter. 14. i made an effort to advance the dacp. 15. the instructor, in the context of the dacp, improved my understanding of the subject matter. 16. the dacp increased my understanding of the complexities of digital archiving projects. 17. the dacp increased my understanding of the complexities of working with information technology. 18. the dacp improved my understanding of digital rights management. 19. the dacp improved my understanding of digital preservation. 20. the dacp was too hard. 21. the dacp offered a real-world problem. 22. the course materials (readings) improved my ability to complete the dacp. 23. i have already created digital archives before taking the class. 24. the dacp improved my understanding of topics covered in other courses i have taken at sils (e.g., information technology, knowledge organization, etc.). / evidence based library and information practice/ / microsoft word es 4070 evidence based library and information practice 2008, 3:4 52 evidence based library and information practice evidence summary high school students struggle to find school-related information on the web a review of: shenton, andrew k. “the information-seeking problems of english high schoolers responding to academic information need.” library review 57.4 (2008): 276-88. reviewed by: cari merkley librarian, mount royal college calgary, ab, canada e-mail: cmerkley@mtroyal.ca received: 31 august 2008 accepted: 06 october 2008 © 2008 merkley. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – to investigate the informationseeking behaviour of high school students looking to meet school-related information needs. design – online questionnaire. setting – a comprehensive, publically funded high school in north-east england. subjects – seventy-seven high school students between the ages of 13 and 18 who responded to an online questionnaire that was distributed to the 900-1000 students enrolled at the institution. methods – an invitation to participate in an online questionnaire was sent to all students at the high school in october, 2006, via email. the total number of invitations sent was not indicated, although it is noted that current enrolment at the school is approximately 900-1000 students across years 9 to 13. in the e-mail, students were provided with a link to a questionnaire posted on the school’s intranet. the questionnaire consisted of six multiplechoice and three open-ended questions. qualitative data gathered through an openended question about problems encountered when seeking information for school was manually coded, and forms the focus of this article. evidence based library and information practice 2008, 3:4 53 main results – seventy-seven online questionnaires were completed by students between 31 october and 27 november 2006, when analysis of the data began. of the 77 respondents, only 35 provided data on problems encountered when seeking information for their assignments. most of the respondents in this group were in years nine, ten and eleven (ages 13-16), with only two in year 12 (16-17) and four in year 13 (17-18). over half (19/35) of respondents were female. forty remaining respondents either stated that they experienced no problems in finding the information they needed for school or did not answer the relevant question on the questionnaire. two participants indicated that they did not have the information they needed to complete their schoolwork because they did not look for it. over 20 distinct information-seeking problems were identified through inductive analysis of the qualitative data provided by 35 participants. difficulties encountered in the search for information largely fell into four major categories: problems determining an appropriate search strategy; barriers posed by limited school resources or internet filtering software; “process frustrations” (280) stemming from the perceived inadequacies of search engines, poorly designed web sites, and missing or broken web links; and, “shortcomings in the retrieved information” (281) in terms of relevance and accuracy. in addition, a small number of students either indicated that they had difficulty applying the information they found to the problem that prompted the search, or were concerned about copyright restrictions on how they could use the information. all but two of the problems reported by students related to information-seeking on the web. the web was the most popular source of information for students, with 71 out of 77 respondents listing it as one of the sources or the only source they consulted for school. conclusion – the results suggest a need for information literacy instruction among high school students, with a particular focus on effective use of the web. the author suggests that some of the students’ frustrations may have been due to an “overreliance” on web resources, and could have been avoided if they were educated in the use of additional types of tools (286). this reliance on web search engines proved problematic when web filters impeded the students’ academic research. some of the problems reported by students in 2006 in the search for academic information were similar to those recounted by students in 1999-2000 for the author’s earlier fieldwork in the same geographic area, including concerns about the accuracy or lack of detail of some web sources, difficulties identifying effective search terms, and barriers posed by internet filters. additional research is needed to determine whether students experience the same difficulties when searching for information to meet personal needs and interests as they do when they are searching for information at the behest of a teacher. commentary in his discussion of the practice of information-seeking behaviour research in schools, shenton outlines some of the benefits and drawbacks of conducting fieldwork in such an environment. while schools may provide researchers with a ready-made group of young people from diverse economic and social backgrounds, they also present additional challenges, such as the red tape involved in obtaining the requisite permissions from participants and their guardians, difficulties scheduling research into an already crowded school calendar, and finding space and privacy to evidence based library and information practice 2008, 3:4 54 work with students (shenton, “informationseeking research” 180-2). this article suggests that a school’s existing research culture may also pose problems for researchers looking to connect with students. shenton acknowledges that using an online questionnaire was not the best method for collecting qualitative data on information-seeking behaviour, but states that “the investigator had to temper his inclinations in order to conform to the expectations of the school’s senior managers” (shenton, “information-seeking problems” 277). the methodology employed was dictated by past practices at the school, and its use suggests a compromise made by the researcher in order to secure access to the subject pool. the reliance on this single source of self-reported data to create a snapshot of the difficulties encountered by students is problematic, as it relies on the students’ willingness to expend the time and energy to type out a considered response to an open-ended question. the use of an online questionnaire may have also lent itself to a higher response rate from students already predisposed to use the web first when looking for information. thirty-five students did provide valuable insight into the information-seeking process from the user’s perspective, making this article of interest not only those working in secondary education but also to the academic librarians who will inherit these students down the road. however, a combination of methodologies would have provided richer data and allow the researcher probe deeper into some of the scenarios recounted in the questionnaire. the potential strength of an online questionnaire – anonymity – was also compromised by the fact that the identity of respondents was logged by the school’s computer system and made available to the researcher. shenton suggests that some students may have lied about their experiences in order to save face, although several students’ willingness to admit to research inaction suggests that not all students were concerned about the researcher’s good opinion (“informationseeking problems” 279). one could infer from these statements that the students were aware that their responses were being tracked by the system, but shenton does not explicitly state that this was the case. the number of students who completed the questionnaire is described as “disappointingly small,” although the exact response rate to the questionnaire is not made available (shenton, “informationseeking problems” 279). based on the enrolment figures provided (900-1000 students), the total response rate for the survey may have ranged from 7.7 to 8.6%. of those who did provide data on problems encountered, most were between the ages of 13 and 16, with limited representation from upper year students. lack of data on the number of students enrolled in each year group makes it difficult to determine if this is simply a reflection of the proportion of students registered in years 12 and 13, or if there are other factors at work (e.g. more experienced students encountering fewer difficulties, less interest in participating among older students, etc.). other missing figures include how many respondents did not answer the question on problems encountered and how many indicated that they did not experience any problems at all. a combined total of both incidences is provided, but it may have been illuminating to examine them individually. the voices of students who felt they had been successful in their searching are missing from this article, and may have provided interesting examples of how students define a successful search or what information literacy skills they may already possess. evidence based library and information practice 2008, 3:4 55 the author was also interested in tracking changes in students’ information-seeking behaviour over time, particularly given the increasing importance of the web over the last decade. the links made between the findings of this study and the author’s 1999/2000 fieldwork would have carried more weight if data from the same school in two different time periods was being compared, rather than data collected at different institutions in the same geographic area. this more recent project also focused on students at a single school, whereas the earlier fieldwork collected data at six different institutions, lessening the potential impact of an individual school’s culture on the results collected. of interest to information professionals is the limited role of library resources in the students’ descriptions of their informationseeking behaviour. the one student who did talk about the library’s resources in geomorphology found them “restricted in depth of detail” (“information-seeking problems” 281). no mention is made of students consulting with information professionals when problems were encountered. the information provided about the school library focused on its computer lab, and it is not clear if it is staffed by information professionals. it would have also been interesting to know more about the physical and electronic resources made available to students at this high school through their library. for example, does the school currently subscribe to any electronic databases? if so, could the lines between subscription and freely available web content have been blurred in student responses? the few additional comments on resources provided by the school also talk about their inadequacy (not enough teacher-assigned texts and computers with web filters that impeded searching). the frustrations expressed by students who felt that their work had been obstructed by web filtering software are particularly important, given the debate around the use of such programs. the solution identified by shenton for the problems encountered by the young people in this study is formalised information literacy instruction, but the limitations of the study may make it difficult for others to draw firm conclusions about what direction such instruction should take. shenton’s emphasis on improving web searching does appear to be in keeping with the behaviour demonstrated by his respondents, although some of his additional suggestions may not prove to be as successful. students’ frustrations with the internet in this study did not necessarily lead them to try other sources. shenton himself highlighted this paradox in an earlier work: “young people are often highly critical of particular information resources, yet continue to use them habitually” (“paradoxical world” 4). one suspects that even with rigorous training in sources like paper indexes (which are also mentioned by shenton as a possible alternative for meeting some of the information needs identified by respondents), students would still turn to the web first for answers, not only for school, but also to meet personal needs. it also is not clear what level of information literacy instruction the students had already received, either from information professionals working in the school or from individual teachers. how did those students who experienced successful searches come by their skills? particularly intriguing are the three students who indicated concerns about copyright in their use of information found on the web. since all three students were completing the same assignment, one could speculate that the teacher of that course had addressed this issue in the classroom. the limitations of the data collection tool employed leave this and many other interesting questions about evidence based library and information practice 2008, 3:4 56 students’ information-seeking behaviour unanswered. works cited shenton, andrew k. “information-seeking research in schools: opportunities and pitfalls.” aslib proceedings 56.3 (2004): 180-86. ---. “the paradoxical world of young people’s information behavior.” school libraries worldwide 13.2 (2007): 1-17. editorial evidence based library and information practice 2009, 4:2 78 evidence based library and information practice article school librarianship and evidence based practice: progress, perspectives, and challenges ross j. todd associate professor school of communication, information and library studies rutgers, the state university of new jersey new brunswick, new jersey, united states of america email: rtodd@scils.rutgers.edu received: 05 january 2009 accepted: 27 april 2009 © 2009 todd. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – this paper provides an overview of progress and developments surrounding evidence based practice in school librarianship, and seeks to provide a picture of current thinking about evidence based practice as it relates to the field. it addresses current issues and challenges facing the adoption of evidence based practice in school librarianship. methods – the paper is based on a narrative review of a small but growing body of literature on evidence based practice in school librarianship, set within a broader perspective of evidence based education. in addition, it presents the outcomes of a collaborative process of input from 200 school libraries leaders collected at a school library summit in 2007 specifically to address the emerging arena of evidence based practice in this field. results – a holistic model of evidence based practice for school libraries is presented, centering on three integrated dimensions of evidence: evidence for practice, evidence in practice, and evidence of practice. conclusion – the paper identifies key challenges ahead if evidence based school librarianship is to develop further. these include: building research credibility mailto:rtodd@scils.rutgers.edu http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2009, 4:2 79 within the broader educational environment; the need for ongoing review and evaluation of the diverse body of research in education, librarianship and allied fields to make quality evidence available in ways that can enable practicing school librarians to build a culture of evidence based practice; development of tools, strategies, and exemplars to use to facilitate evidence based decision-making; and, ensuring that the many and diverse advances in education and librarianship become part of the practice of school librarianship. evidence based practice in the professional and research-based discourse of school librarianship has a short history and a limited documentary record, emerging in the school library arena in 2001 (todd ‚transition for preferred futures‛ 4). like its counterparts in education and librarianship, evidence based practice in school librarianship has emerged within the context of a developing evidence based paradigm of library and information practice (booth and brice 90), and changes in the school educational context, particularly with a new emphasis on evidence based decision making (marsh et al.). in the view of the author, its short history can be characterized by a limited sustained critical theoretical examination of the concept, with some development of central ideas about what evidence based practice is within school librarianship. this paper provides an overview of the background, progress, and development surrounding evidence based practice in school librarianship and seeks to provide a picture of current thinking about evidence based practice as it relates to the field. the context: the u.s.a. and international school library landscape two of the latest reports released by the u.s. federal government’s national center for education statistics provide a picture of the on the status of school libraries in the u.s.a. these are: status of public and private school library media centers in the united states: 1999–2000 (holton, et al.), and school library media centers: selected results from the education longitudinal study of 2002 (scott), a report providing an overview of the state of school libraries that serve 10th-graders based on a nationally representative sample of 15,525 10th-graders in 752 schools in the united states in 2002. in these reports, a school library media center is officially defined as: an organized collection of printed and/or audiovisual and/or computer resources which is administered as a unit, is located in a designated place or places, and makes resources and services available to students, teachers, and administrators. a library media center may also be called a library, media center, resource center, information center, instructional materials center, learning resource center, or some other name‛ (scott iv). key variables examined in these studies are staffing, library expenditures, and collection holdings. key findings include: 92% of all traditional public schools in 2000 have school libraries, (about 77,000 public school libraries) 97% of the 45 million students enrolled in public elementary or secondary schools in the u.s.a. attend schools with a school library 75% of public schools with a school library have a paid, state-certified library media specialist (compared with 20% of private schools) high schools with a school library are more likely than elementary or evidence based library and information practice 2009, 4:2 80 combined schools to employ a school librarian with an mls or related degree (52% of high schools compared to 39% of public elementary schools and 32% of combined schools) a larger proportion of school libraries in private schools than in public schools rely on adult volunteers (58% of private schools compared with 38% of public schools) 32% of public school libraries maintain a flexible schedule for class visits to the library public schools tend to provide greater access than private schools to the school library for students’ independent use before or after school (50% compared to 36%) females use the school library more often than males for assignments, in-school projects, homework, research papers, and to read books for fun students with different test scores use the library for different purposes students with high test scores are more likely than students with low or middle scores to use the library for assignments, in-school projects, and research papers. students with low test scores are more likely than students with high or middle test scores to use the library for homework, leisure reading, to read magazines or newspapers, to read books for fun, and for interests outside of school schools with larger student enrollments tend to have more types of equipment and technology than smaller schools 79% of students report that the library staff is very helpful or helpful with finding research resources; 65% of students report that the library staff is very helpful or helpful with using databases; 69% of students report that the library staff is very helpful or helpful with using the internet. (scott; holton) against this backdrop, the focus on an evidence based practice approach in school librarianship parallels developments in school education which are increasingly placing focus on measurable student attainment, measurable learning outcomes, continuous improvement, equity of educational opportunity, and accountability. at the same time, there is also considerable growth in school library research that specifically seeks to demonstrate the contribution of school libraries to student achievement (scholastic). in addition, the professional context of school librarianship over the last decade has seen challenges in terms of the viability of having school libraries and certified school librarians in every school and the marginalization of school libraries, in light of ongoing staffing and budgetary constraints (haycock; lonsdale). these two developments are elaborated here, and set the scene for elucidating key challenges facing evidence based school librarianship. evidence based education educational systems around the world are adopting orientations and practices that can be labeled as evidence based education. central characteristics include an emphasis on scientifically-based research to provide foundation for learning and instruction, and a focus on scientifically-based research as a framework for professional decision making and action. these are set within calls to make education less vulnerable to fads and untested interventions. this includes developing a culture of high expectations for optimal student learning outcomes, a focus on the continuous professional development of teachers, and building a evidence based library and information practice 2009, 4:2 81 shared understanding of the theory, research and practice of curriculum, pedagogy, and community. the institute of education science in the u.s. department of education defines evidence based education as the ‚integration of professional wisdom with the best available empirical evidence in making decisions about how to deliver instruction‛ (whitehurst 3). empirical evidence is conceptualized as ‚research that involves the application of rigorous, systematic, and objective procedures to obtain reliable and valid knowledge relevant to education activities and programs" (whitehurst 5). according to whitehurst, professional wisdom refers to the ‚judgment that individuals acquire through experience, consensus views reflected in numerous ways, including the effective identification and incorporation of local circumstances into instruction‛ (4). whitehurst believes that both are needed. he argues that the foundation of education practice is scientifically-based research from fields such as psychology, sociology, economics, and neuroscience, and especially from research in educational settings which generate cumulative knowledge and resolve competing approaches. he also argues that professional wisdom is an essential capacity to enable education to adapt to local circumstances and to operate intelligently in the many areas in which research evidence is absent or incomplete. he concludes "we need evidence-based education because current practice [in education] has failed. in no other field are personal experiences relied on to make policy choices and in no other field is the research base so inadequate‛ (qtd. in kersting 1). underpinning this focus on scientificallybased research as a framework for professional decision making and action is the need to avoid fad, fancy, and personal bias, and to advocate stances and positions, which whitehurst refers to as ‚strong calls to action‛ -without the supporting evidence derived from empirical research (1). davies, likewise argues that turning to evidence based education would make education less vulnerable to ‚political ideology, conventional wisdom, folklore, and wishful thinking,‛ not to mention ‚trendy teaching methods based on activity-based, studentcentered, self-directed learning and problem solving‛ (109). arising out of this focus has been the elucidation of the ‚gold standard‛ of educational research in the u.s.a, that is, scientifically-valid knowledge about what works generated in randomized controlled trials (rcts) (institute of education sciences). the gold standard of educational research is grounded in a concern for the plethora of educational interventions that are based on evidence from poorly-designed and/or advocacy-driven studies. it specifically requires research based on rcts that are well-designed and implemented, which demonstrate that there are no systematic differences between intervention and control groups before the intervention, and which employ measures and instruments of proven validity (institute of education sciences 1). according to the gold standard framework, if the intervention is not backed by ‚strong‛ evidence, comparison-group studies in which the intervention and comparison groups are very closely matched in academic achievement, demographics, and other characteristics may constitute possible evidence. types of studies that do not comprise ‚possible‛ evidence include prepost studies, comparison-group studies in which the intervention and comparison groups are not closely matched, and ‚metaanalyses‛ which utilize quantitative techniques for combining the results of individual studies, and which typically include the results of lower-quality studies (institute of education sciences 1-3). evidence based library and information practice 2009, 4:2 82 in the context of evidence based education, the federal education legislation in the u.s.a. centering on the no child left behind act of 2001 and the education sciences reform act (esra) of 2002 not only cite the importance of using evidence from scientifically-based research and its conscientious interpretation and integration to shape and direct professional practice, but also require state education systems to develop annual assessments to measure learning outcomes, school and student progress, and that educators use data to help improve the learning of all students. overall, this holistic approach identifies three central dimensions of evidence based education: evidence for practice (research foundation), evidence in practice (research integration), and evidence of practice (student learning impacts). mandinach et al. explain that: ‚school leaders are expected to chart the effectiveness of their strategies and use complex and often conflicting state, district, and local assessments to monitor and assure progress < to use data to improve school performance‛ (1). evidence based education not only calls for accountability of outcomes and student achievement results, but also calls for a more careful integration of system generated data in order to monitor outcomes and progress. data-driven decision making in education has become increasingly topical, placing emphasis also on organizational improvement being responsive to and enhanced through various kinds of data generated in schools and communities, such as expenditure on instructional materials and library resources. while the broader notion of datadriven decision making in not new, marsh et al. (2) emphasize that organizational improvement is enhanced by responsiveness to various types of data, including input data such as material costs, process data such as benchmark and performance data, outcome data such as achievement levels, and satisfaction data, including employee and customer opinions. the concept of data-driven decision making in education is not new and can be traced to decades of debates about measurementdriven instruction, state requirements to use site-based decision making and integrating outcome data into school improvement planning and site-based decision making processes. marsh, pane, and hamilton (3) present a conceptual framework of datadriven decision making in education (figure 1). fig. 1. conceptual framework of data-driven decision making in education (marsh, pane, and hamilton 3) evidence based library and information practice 2009, 4:2 83 the growing interest in evidence based education has not been without criticism. considerable debate for some decades now surrounds the appropriateness of rcts in education, and the counter voices raise a range of ethical and pragmatic concerns. key concerns center around the view that the model of causality underlying rcts is too simplistic to capture the complexity and multiplicity of teaching and learning in diverse educational settings, and that this simplicity puts meaningful learning at risk. given that rcts assume fidelity of intervention, it is argued that the reality of the intervention being faithfully implemented as intended in the rct is unlikely in the dynamics of an everyday classroom. it is argued that the reductionist approach of isolating the effect of specific factors assumes that it works in isolation to other factors. consequently, implementing tightly controlled instructional programs focusing on a single intervention runs the risk of building a rigid packaged educational approach, and denies the opportunity for diagnostic responses arising out of the dynamic interplay of a myriad of contextual factors and the interactions of teachers, students and environment (kerlinger; davies; morrison; willinsky; johnson; gordon). davies (108), for example, poses the question: can there be distinct and standardized ‘treatments’ in education? he argues that to establish fixed, universal causal patterns in teaching seems equally difficult, if not impossible: ‚unlike in most areas of medicine, in education the ‘treatments’ consist of symbolic interaction, with all the scope for multiple interpretations and responses which that implies‛ (109). calling for a valuing of craftbased practical judgment, davies is more permissive of a variety of procedures, thus voicing a broader conception of educational outcomes. in addition to the randomized controlled trial, he mentions survey and correlational methods, regression analysis, and analysis of variance. he allows for inquiries that seek to describe the meanings different people attach to different teaching activities, and the broader and long-term consequences of them, e.g. on ‚students’ and parents’ sense of self and their sense of social worth and identity‛ (115), and concludes that analyses of naturally occurring teaching interactions, conversation, and discourse are a part of quality teaching and learning. in a similar vein, willinsky argues that a reliance on the randomized clinical trial is difficult because experimental conditions are harder to maintain in schools, since the cost of undertaking such extensive studies is enormous; and because he considers it be a ‚disservice to the very goals of education to turn policies and programs – as well as the life of the classroom – over to the strict dictates of a statistically significant difference achieved in experimental trials‛ (6). this debate is further surrounded by claims that the evidence based practice movement is a passing trend (notwithstanding that the evidence based practice focus seeks to move professions away from fad or trendinfluenced decision making). additional concerns center on lack of expertise and low availability of evidence. it is argued that because the education profession is not steeped in the research traditions of the medical field out of which evidence based education emerged, it creates major gaps in expertise and professional discomfort with engaging in and utilizing such evidence. in addition, there are concerns that the what works clearinghouse, set up by the department of education to be a ‚central and trusted source of scientific evidence for what works in education‛ is so rigorous that few studies will meet the required level of scrutiny, resulting in the absence of enough evidence based library and information practice 2009, 4:2 84 evidence to make important instructional decisions. evidence based school librarianship evidence based practice in school librarianship is also grounded in the evidence based librarianship and information practice (eblip) movement, itself underpinned by the evidence based practice paradigm in the health care area which emerged in the early 1990s in the united kingdom in the fields of medicine and health care services (sackett et al. 71). booth posits that evidence based library and information practice: seeks to improve library and information services and practice by bringing together the best available evidence and insights derived from working experience, moderated by user needs and preferences. eblip involves asking answerable questions, finding, critically appraising and then utilizing research evidence from relevant disciplines in daily practice. it thus attempts to integrate user-reported, practitioner-observed and research derived evidence as an explicit basis for decision-making (booth and brice 92). consistent with booth, eldredge highlights its focus as employing the best available evidence based on library science research to arrive at sound decisions about solving practical problems in librarianship (eldredge 290). positioning evidence based librarianship as a dynamic and evolving approach to integrating research into practice, eldredge acknowledges that this does not take place in a ‚remote, ivory tower‛ microcosm; rather, it acknowledges that librarians ‚operate their libraries in the real world context of providing services and collections through managing budgets and other resources. thus, ebl constitutes an applied rather than theoretical science. ebl merges scientific research with the pressing need to solve practical problems‛ (290-1). one of the earliest elaborations and discussions of the concept of evidence based practice took place at the international association of school librarianship conference in auckland, new zealand in 2001. todd (‛transitions for preferred futures‛), in the conference keynote address presented the argument that in order for school libraries to play a key role in the information age schools and be perceived to add value to the learning goals and agendas, there needed to be a fundamental shift from the rhetorical, advocacy basis for ongoing practice, continuous improvement and development of school libraries, to an evidence based framework that focused on engaging with the research foundations of the profession to document evidence of contribution to curriculum outcomes and the learning goals of the school. this address posited that information is the foundation of meaningful learning in schools and forms the collection of objects around which the practice of school librarianship has been built. however, it argued that while its provision is ‚fundamental to functioning successfully in today's information and knowledge-based society‛ (ifla/unesco 1) information in and of itself is not the hallmark of the 21st century school library. rather, it posited that actions and evidences that show that school library inputs and processes, themselves based on sound decision making underpinned by available research, makes a real difference to student learning and enables the school to attain high standards of knowledge, skills and understanding of the curriculum, and to meet achievement and progress goals. since 2001, there appears to be some international effort in relation to explicating evidence based library and information practice 2009, 4:2 85 an evidence based framework for school librarianship, shaped by the dynamics of the professional context. to date, there has been considerable emphasis placed on the compilation and synthesis of the body of research evidence surrounding the field of school librarianship, particularly the impact of school libraries on student achievement. evidence based advocacy is clearly an intent of such documents, seeking to establish a strong argument for the support and continuous improvement for school libraries at national, state and local levels, as well as to draw attention to current issues facing school libraries. in addition, over the last eight years, some attention has been given to the development of frameworks for enabling school libraries at the local level to implement evidence based practices; and some, albeit limited take up by professional organizations (todd ‚evidence based manifesto‛ 5). haycock’s report of the crisis in canada’s school libraries documents the country’s ‚neglect‛ in investing in school libraries, particularly against a backdrop of ‚a mounting body of research evidence showing a strong and compelling link between student achievement and the presence of well-stocked, properly funded and professional-developed school library programs and services‛ (9). he cites specific evidence of neglect, for instance, that ‚10% of ontario elementary schools have a fulltime teacher-librarian, compared with 42% twenty-five years ago; alberta’s roster of teacher librarians half-time or more has dropped from 550 to 106 since 1978; and in british columbia, local school board funding levels now reveal dramatic inconsistencies in annual budgets for library resources, with the figures ranging from 80¢ to $35 per student per year, the latter providing for maintenance only, not growth‛ (11). roch carrier, the then national librarian of canada in 2002 concluded that the ‚state of our school libraries can only be described as desperate in almost every province‛ (haycock 13). against this backdrop, haycock provides an in-depth meta-analysis of the growing body of research evidence that shows the impact of quality school libraries on student achievement, reading and the growth of cultural identity in a case for revitalizing canada’s school libraries. in a similar vein, lonsdale’s review of the literature on the impact of school libraries on student achievement in an australian context published by the australian council of educational research for the australian school library association, builds on key concerns about the future of school librarianship in australia, and like haycock, seeks to construct an evidence based argument for the case for investing in school libraries across australia. lonsdale cites an ‚apparent decline in the numbers of qualified teacher librarians employed in school libraries in public schools in australia‛ (4), coupled with budget cutbacks and a focus on accountability (27). she argues that trends shown in several state-based studies indicate: a general shortage of teacher librarians (and other specialist teachers); the practice of schools using librarians rather than teacher librarians, or having staff with no library or teaching qualifications at all; teacher librarians being used in classrooms as subject teachers to fill gaps in staffing; an aging profession, with retirees not being replaced by sufficient numbers of graduates; and added responsibilities for teacher librarians in terms of technology maintenance and student use of technology (8). against this backdrop, lonsdale presents a synthesis of both australian and international research that presents an evidence based case for educational authorities at federal and state levels to invest in school libraries. as with haycock, lonsdale calls on local systems and individual schools to provide ‚local, evidence based library and information practice 2009, 4:2 86 evidence based practice if the roles of the school library and teacher librarian in student learning are to be valued in the way that the research suggests they should be valued‛ (2). school libraries work!, published by scholastic brings together both position statements from a range of library and educational organizations, and concise summaries of empirical studies that have been undertaken across the u.s.a. and canada to document the impact of school libraries on student achievement. these studies, involving approximately 10,000 elementary, middle and high school libraries serving an estimated 2.6 million students, and funded by diverse funding authorities such as state library and information agencies, education departments, and professional school library associations, employ a variety of research methods that the positive relationship between school libraries and student achievement. collectively, these studies show that in schools with quality school library facilities and programs, staffed by certified school librarians, students ‚learn more, get better grades, and score higher on standardized test scores than their peers in schools without libraries‛ (scholastic 4). while the focus of this compilation is on impact on student achievement, the research studies articulate the range of dimensions that underpin this impact, specifying an evidence based framework for decision making about school libraries and their continuous improvement. in addition to the employment of certified school librarians, such dimensions include: the provision of appropriate allocation of para-professional staff; actively supporting the curriculum through the provision of up-to-date adequate resources both print and electronic; the provision of an active instructional information literacy program integrated into curriculum content, and targeted towards meeting content standards; the provision of access and instruction based on flexible scheduling; the provision of a strong networked information technology infrastructure and instruction in its use; the provision of professional development on information literacy and technology literacy to the teaching faculty; the provision of appropriate budget allocation per student per year to ensure currency and vitality of the information base; the implementation of a vibrant reading program for academic achievement and personal enjoyment and enrichment; and the collaboration with other libraries and information agencies. there is some indication of evidence based practice philosophies being embedded in official documentation of professional school library associations around the world. the recent release of american association of school librarians (aasl) ‚standards for the 21st century learner‛ in 2007 articulate a move from guidelines for the development of school library programs, to standards that identify expected student outcomes as a result of school library interventions, and which provide a framework for identifying evidences of the professional work of school librarians. the new aasl standards marks a significant transition in the association’s provision of guidelines and standards over some 90 years; from quantitative measures of items held, to measuring school library programs in terms of program guidelines, to a current focus on learning outcomes that center on reading and literacy development; inquiry, critical thinking and knowledge development (including the ability to draw conclusions, make informed decisions, apply knowledge to new situations, and create new knowledge), sharing and using that knowledge productively and ethically, and pursuing personal and aesthetic growth. evidence based library and information practice 2009, 4:2 87 a more explicit example is the australian school library association (asla). in its ‚standards of professional excellence for teacher librarians‛ published in 2004, asla identifies the expectation that teacher librarians engage in evidence – that they are ‚well-informed about information literacy theory and practice,‛ ‚have a detailed knowledge of current educational pedagogy,‛ ‚evaluate student learning to provide evidence of progress in information literacy and reading,‛ and ‚use evidence to inform programs and services‛ (2-3). presumably, such evidence foundations provide a framework for shifting the basis on which value statements can be made about the professional practice of school librarianship, as well as about the role of the school library in student achievement. the asla advocacy kit published in 2006 titled ‚a teacher librarian advocate’s guide to building information literate school communities‛ was an adaption from the ala version which advocates that ‚teacher librarians and library advocates will speak out about the critical importance of information literacy skills and the key role of school libraries and teacher librarians‛ with a goal that ‚schools will expand their programs to include information literacy skills across the curriculum‛ (asla 5). yet, in constructing the argument underpinning the diverse range of advocacy strategies, there is almost no explicit reference to the use of evidence; rather it appears to give prominence to presenting a collection of rhetorical statements such as: ‚school libraries are changing and dynamic places, at the forefront of the information age‛(9); ‚teacher librarians have helped generations of australians to lead better, more satisfying lives‛ (9); ‚school libraries are part of the australian dream. they are places for education, enjoyment and self-help‛ (10), all with no focus on engaging with the evidence to support such claims. in elaborating ‚steps to success‛ (13) and constructing public service announcements (14), the only reference to engaging with evidence in any form – the practice of advocacy based on evidence comes in the form of a ‚quotable quote‛ of lonsdale: ‚existing research shows that school libraries can have a positive impact, whether measured in terms of reading scores, literacy or learning more generally, on student achievement. there is evidence to show that a strong library program that is adequately staff, resourced and funded can lead to higher student achievement regardless of the socioeconomic or educational levels of the adults in the community‛ (lonsdale 20). two key events contributing to the emerging discourse of evidence based practice in the school library sector have been the school library journal leadership summits held in chicago u.s.a. in 2006, and phoenix u.s.a. in 2007. during the chicago summit, 200 participants engaged in visioning school libraries of the future. through a collaborative process of short presentations, intensive discussion and debate, analysis and synthesis, the participants generated ten priorities / opportunities that were seen to be at the center of developing school libraries of the future. the list, with highest priorities first, strongly affirmed the emerging importance of evidence based practice and data-driven decision making to the professional outlook: 1. meshing the added value of libraries, derived from evidence, into the educational-learning environment 2. demonstrate through the use of data and evidence to various stakeholders that school librarians and library programs pay learning dividends and improve student achievement 3. develop and embrace new models for interacting with learners using 21st century technology evidence based library and information practice 2009, 4:2 88 4. provide stakeholders with instructional materials, information, and model school libraries to demonstrate excellence 5. technology 6. taking a leadership role in educational applications for emerging technology 7. librarians engaged as partners in the ongoing assessment of student learning 8. the inclusion of information literacy in teacher preparation curricula 9. target the have-nots – closing the information gaps 10. building trust and respect with library users; recruiting young school librarians the high priority given to evidence based practice played an important part in shaping the 2007 school library journal leadership summit that convened in phoenix, arizona in 2007. this summit, titled ‚where's the evidence? understanding the impact of school libraries,‛ also brought together a diverse mix of school library leaders, school administrators, and policy-makers, as well as experts from the medical field and education who engaged in intense and thoughtful presentations and discussions over the two days focusing on evidence based practice. the outcomes of this conference are reported in: ‚the evidencebased manifesto for school librarians: if school librarians can’t prove they make a difference, they may cease to exist‛ (todd). this document represents a current understanding of evidence based practice in the context of school librarianship. it sums up both the thinking and challenges as perceived by key leaders in the school library profession, particularly providing a conceptualization of evidence based practice in the school library arena, a practice that sits at the confluence of education and librarianship and adapts core ideas from the evidence based practice movement in education and librarianship. key aspects of this document are examined here. evidence based practice in school librarianship: current thinking a working conceptualization of evidence based practice in school librarianship to date is that it is about professional practice being informed and guided by best available evidence of what works, coupled with a focus on evidence of outcomes and impacts of services in relation to the goals of the educational environment in which it is situated. consistent with evidence based education and evidence based librarianship, it incorporates a decision-making framework based on the best available research evidence with professional knowing and experience to make professional decisions and take professional action, and to implement and continuously improve professional practice, as well as a framework for documenting evidence of outcomes. aligned with the student outcomes focus of many international educational systems, an integral component of evidence based school librarianship is the systematic collection, integration and dissemination of evidence of the tangible impacts, and outcomes of school library practices, with organizational goals and objectives including student achievement and the development of deep knowledge, deep understanding, and competencies and skills for thinking, living and working. this explicit focus on the collection of evidence is viewed as a local school responsibility, as well as regional, state, and national responsibilities. at a local school level, evidence based practice of school librarianship seeks to demonstrate the value-added role of a school library to the life and work of a school – outcomes that center on learning, literacy and living – and evidence based library and information practice 2009, 4:2 89 the development of students personally, socially, culturally, and globally. accordingly, current thinking on evidence based school librarianship seeks to establish a holistic approach that welds three dimensions of actionable evidence: evidence for practice, evidence in practice, and evidence of practice: evidence for practice focuses primarily on examining and using best available empirical research to form practices and inform current actions, and to identify best practices that have been tested and validated through empirical research. this is posited as the informational dimension of school library practice. evidence informs practice. evidence in practice focuses on reflective practitioners integrating available research evidence with deep knowledge and understanding derived from professional experience, as well as implementing measures to engage with local evidence to identify learning dilemmas, learning needs, and achievement gaps to make decisions about the continuous improvement of the school library practices to bring on optimal outcomes and actively contribute to school mission and goals. this is posited as the transformational dimension of school library practice. evidence of practice as the measured outcomes and impacts of practice, is derived from systematically measured, primarily user-based data. it focuses on the real results of what school librarians do, rather than on what school librarians do. it focuses on impacts, going beyond process and activities as outputs. it established what has changed for learners as a result of inputs, interventions, activities, processes, and charting the nature and extent and quality of effect. these dimensions or phases are not posited as linear and static. rather, they are presented as a dynamic, iterative and integrative process of welding evidence from multiple sources in a cycle of continuous transformation of data, information, knowledge, and wisdom to inform practice, to generate practice, and to demonstrate outcomes of practice (table 1). table 1 dimensions of actionable evidence: a holistic model of evidence based practice for school libraries evidence for practice foundation informational existing formal research provides the essential building blocks for professional practice: evidence in practice applications / actions process transformational locally produced evidence; data generated by practice is meshed with researchbased evidence to provide a dynamic decision-making environment: librarian-observed evidence evidence of practice results – impacts and outcomes; evidence of closing of gaps outcomes formational user-reported evidence learner changes as result of inputs, interventions, activities, processes evidence based library and information practice 2009, 4:2 90 in essence, the evidence based practice approach in school librarianship gives considerable explicit attention to ‚userreported evidence;‛ (booth and brice 92) that is, evidence of outcomes and impacts of professional practice based on studentgenerated evidence. in this respect, evidence based practice also serves as an important advocacy role through demonstrating the value-added role of school libraries to the learning goals of a school. this means that the day-by-day work of school librarians is directed towards demonstrating the tangible impact and outcomes of services and initiatives in relation to student learning outcomes. it involves critically analyzing accumulated data and making evidence based claims about student learning outcomes (loertscher 6). this explicit focus on student outcomes is somewhat driven by the current educational climate and increasing cuts and threats to the provision of school library services. this is coupled with a strong emphasis in education at the present time on student attainment, measurable learning outcomes, continuous improvement, and accountability. a focus on evidence based school librarianship shifts the basis on which value statements can be made about school libraries if they are to be perceived as playing a strong role in the school. by placing emphasis on measured outcomes, evidence based practice in school libraries shifts the focus from the medium to the message and articulating what school librarians do in their day-to-day work, to articulating what students become. by placing emphasis on systematically gathered evidence, it moves school library advocacy from a ‚tell me‛ framework to a ‚show me‛ framework. accordingly, evidence based practice first and foremost validates that quality learning outcomes can be achieved through the school library; secondly, and through this, validates the important role of the school librarian as an instructional partner in the school, and a key team member in achieving the schools mission and goals. common beliefs this articulation of the above conception of evidence based school librarianship is underpinned by a set of beliefs about school libraries. the first belief is that school libraries as schools’ information and knowledge commons are viewed as essential for addressing curriculum standards, the complexities of learning, and quality teaching in informationand technology-intensive 21st century schools (kuhlthau, maniotes, and caspari). identifying the evidence of the outcomes and positive relationship between school libraries and student achievement is viewed as a key to maintaining this role. the second belief is that school librarianship, as an applied science and profession, derives it practice mandate from a diverse body of theoretical and empirical knowledge; and active engagement with this body of knowledge enables the profession to continuously transform and improve. leading this transformation is the professional expertise of school librarians, certified through a program of university graduate education, who possess expertise, insights, and skills based on theoretical and empirical knowledge that they apply in practice. they continuously develop their knowledge and skills through professional development and ongoing engagement with constantly emerging body of research-based knowledge and its application to practice. thus the professional role of school libraries is founded on a strong evidence-base, welding together research, experience, insights, and systematic measures (aasl; scholastic; ifla /unesco). the third belief is that school libraries play a transformative role in the lives of students: evidence based library and information practice 2009, 4:2 91 the development of intellectual, social and cultural agency. school libraries, led by certified school librarians, provide the spark for students’ connections, interactions and use of information for developing deep knowledge and understanding, and achievement (haycock; lonsdale; scholastic). connected to this idea, the fourth belief is that the core work of school librarians centers on enabling the transformation of information to knowledge, and the development of attitudes, values, and beliefs through carefully designed instructional interventions and reading literacy programs that guide and engage students in their inquiry, learning, and reading development, and enable them to build new understandings and to develop personal viewpoints and perspectives (aasl). the fifth belief is that the value of a school library can be measured. the transformations as learning outcomes, as well as personal, social, and cultural growth can be documented, measured and disseminated, as evidenced in scholastic’s ‚school libraries work‛. this is the heart of evidence based school librarianship. interwoven with this notion, the sixth belief is that professional school library responsibility is an accountable responsibility, accounting for ability, not merely counting, and through the application of that ability, for meeting espoused goals. sustainable development through accountability has as a key characteristic a move from a rhetorical warrant to an evidential warrant for professional practice; from a persuasive / advocacy framework to a declarative / demonstrable framework; and from a process framework to an outcomes framework; from a ‚tell me‛ framework to a ‚show me‛ framework (todd ‚transitions for preferred futures‛; loertscher). key challenges ahead this paper concludes with an elaboration of some key challenges ahead if evidence based school librarianship is to develop further. first, the focus on research-based evidence, as defined by the u.s.a. department of education, presents a considerable challenge for school librarianship. as gordon points out, school library research studies are not among the current list of gold-standard studies published in the doe’s what works clearinghouse (1). despite the richness and diversity of school library research to date, there is no school library research that meets the gold standard requirement. for school librarians, evidence based education utilizing this approach would mean, for example, that they would select reading intervention and literacy programs for the school that led to significantly higher test scores than other programs, based on rigorous evidence of randomly assigned students to each program and to a control group. it would also mean that they would use information literacy interventions that consistently proved more effective than other methods with the very population that one was teaching through research evidence derived from randomized controlled trials. this signals an urgency for the whole school library research community to engage in some sustained and complex discussions on the future directions of school librarianship research, and what is needed to continue building a strong research base for the profession. it is the view of the author that there is considerable value in the school library community of researchers undertaking randomized trials as one of the broader range of research approaches it employs. despite the scalability and costs involved in undertaking such research, its representation in the body of gold standard research plays an important role in building credibility within educational circles and evidence based library and information practice 2009, 4:2 92 sustainability of school libraries in the future. the second key challenge for school librarians is to actively utilize the available research in their decision making, and to demonstrate in their school community that they are part of a sustained research culture, and to be supported in that endeavor by multiple stakeholders. lau’s survey of principals’ perceptions of school librarians found that while 80% of principals believe that the school library and school librarian play a role in the school, only 37% of principals said that the school librarian made them familiar with current research of library programs and student achievement, and 35% were made familiar with current research on library programs and reading development (53-4). there is a critical need for making available research-based evidence more accessible, interpretable and applicable to daily practice. some key research for the school library profession is locked up in membership-access-only providers and publishers. accessibility and utilization also require a much closer working relationship and greater levels of collaborations with school library researchers and educators, and school library practitioners in order to build a stronger community of participatory research. with a focus on local actions and local evidence that demonstrates the vital role that the local school library plays in the learning agenda of the local school, a third key challenge focuses on the development and provision of tools, processes and structures to accumulate locally generated evidence, and establishing claims derived from that data, as well as exemplars that showcase the claims. a widespread engagement in evidence based practice is likely to generate a vast and important amount of local data that are potentially informative for the profession at large. it is important for the profession to establish how this evidence can be accumulated across individual schools, districts, and shared. a fourth key challenge centers on the provision of systematic professional training for school librarians in interrogating, synthesizing and drawing conclusions and implications for action from research-based data, state data on achievement and progress, and locally collected data, and building evidence based advocacy approaches. this has broader implications for the formal and post-degree education of school librarians and their expertise with evidence based strategies. time is often presented as the key barrier to implementing approaches to evidence based practice, the belief being that it takes time to develop and implement measures, to analyze and synthesize data, and to disseminate findings. in meeting the time requirement, there is the perception that additional support staff is needed to enable the school librarian to undertake this ‚additional‛ work. this is about mainstreaming evidence based practice initiatives as a dimension of best practice rather than it being perceived as an addition to current practice, and conveying to school administrators that evidence based practice is a key component of the professional work of the school librarian and build their support to enable infrastructure and processes and collaborations on which evidence based practice is built to be put in place. conclusion this paper situates the emerging discourse on evidence based school librarianship at the confluence of evidence based education and evidence based library and information work. its development has been clearly grounded in the evidence based librarianship and information practice evidence based library and information practice 2009, 4:2 93 (eblip) movement, which provides the foundation for professional practice based on sound and careful decision making using available best evidence for solving professional problems, and providing a framework for continuous improvement. however, evidence based school librarianship is also situated within an educational context, with its emphasis on student outcomes, achievement, and optimal learning processes. accordingly, the evidence based practice movement in school librarianship gives explicit attention to evidence outputs as well as evidence inputs. it goes beyond accessing, appraising and utilizing research evidence in daily practice and solving professional problems, to measuring, demonstrating, and disseminating explicit evidence of outcomes. this is presented as a holistic approach to evidence based practice, encompassing the integration of three fundamental iterative dimensions: evidence for practice, evidence in practice, and evidence of practice. it represents an informational, transformational and formational approach to professional practice, where evidence informs and transforms practice, as well as signaling the measured outcomes and impacts of practice. this holistic approach to evidence clearly seeks to establish what has changed for learners as a result of inputs, interventions, activities, processes through the school library and work of school librarians, and charting the nature and extent and quality of effect. it is a timely approach, given the broader context of educational accountability and calls for quality measures and data-driven decision making in which school librarianship operates. this confluence also creates considerable tension for evidence based school librarianship. the current political climate, at least in the u.s., calls for an evidence foundation in random controlled trials that is non existent in school librarianship research. this is a matter for urgent discussion and action. it is a challenging requirement for the profession in a research environment that has no strong tradition for rct, and where the ethical and pragmatic considerations raise considerable issues for undertaking such studies. particularly from this standpoint, the future of evidence based practice for school librarianship hangs in the balance. coupled with this, and despite the absence of a body of ‚gold standard‛ research, there is the need for the ongoing review and evaluation of the diverse body of research in education, librarianship and allied fields, to make quality evidence available in ways that can serve the profession well, both physically and intellectually, and in ways that enable practicing school librarians who may not have substantive training in research methods, to take action. development and support are also required so that practitioners have access to evidence based practice tools, strategies, and exemplars to use in facilitating evidence based decisionmaking, and ensuring that the many and diverse advances in education and librarianship become part of the practice of school librarianship. in the spirit of school librarianship’s emerging framework of evidence for practice, evidence in practice and evidence of practice, the words of john f kennedy, 35th president of u.s., provide the challenge for moving on: ‚we set sail on this new sea because there is knowledge to be gained‛ (qtd. in moncur, ‚the quotation page‛). works cited american association of school librarians (aasl). standards for the 21st century learner. chicago: american library association, 2007. 1 may 2009 <http://www.ala.org/ala/mgrps/divs ../../../../dkoufogi/dkoufogi/local%20settings/temporary%20internet%20files/content.ie5/3gicym2a/qtd http://www.ala.org/ala/mgrps/divs/aasl/aaslproftools/learningstandards/standards.cfm http://www.ala.org/ala/mgrps/divs/aasl/aaslproftools/learningstandards/standards.cfm evidence based library and information practice 2009, 4:2 94 /aasl/aaslproftools/learningstandard s/standards.cfm>. american library association (ala) & association of educational and communication technology (aect). information power: partnerships for learning. chicago, il: american library association, 1998. australian school library association. ‚a teacher librarian advocate’s guide to building information literate school communities.‛ asla advocacy kit. 2006. zillmere, australia: australian school library association, 2006. 1 may 2009<http://www.asla.org.au/advoc acy/>. australian school library association. standards of professional excellence for teacher librarians. zillmere, australia: australian school library association, 2004. 1 may 2009 <http://www.asla.org.au/policy/stan dards.htm>. booth, andrew and anne rice. ‚prediction is difficult, especially the future: a progress report.‛ evidence based library and information practice 2.1 (2007): 89-106. 1 may 2009 <http://ejournals.library.ualberta.ca/ index.php/eblip/article/view/99/24 2>. davies, philip. ‚what is evidence-based education?‛ british journal of educational studies 47.2 (1999): 10821. eldredge, jonathan d. ‚evidence-based librarianship: an overview.‛ bulletin of the medical library association 88.4 (2000): 289–302. 1 may 2009 <http://www.pubmedcentral.nih.go v/picrender.fcgi?artid=35250&blobt ype=pdf>. gordon, carol a. ‚the trouble with the gold standard: school libraries & research: is educational research tougher than medical research?‛ school library journal (2007). 1 may 2009 <http://www.schoollibraryjournal.co m/article/ca6484334.html>. haycock, ken. the crisis in canada’s school libraries: the case for reform and re-investment. toronto: association of canadian publishers, 2003. 1 may 2009 <http://www.peopleforeducation.co m/librarycoalition/report03.pdf>. holton, barbara, yupin bae, susan baldridge, michelle brown, and dan heffron. the status of public and private school library media centers in the united states: 1999– 2000. washington, dc: u.s. department of education, institute of education sciences, 2004. 1 may 2009<http://nces.ed.gov/pubs2004/2 004313.pdf> ifla/unesco. school library manifesto: the school library in teaching and learning for all. the hauge. netherlands: ifla/unesco, 2006. 1 may 2009 <http://www.ifla.org/vii/s11/pubs/mani fest.htm>. institute of education sciences. identifying and implementing educational practices supported by rigorous evidence: a user friendly guide. washington dc.: u.s. department of education, institute of education sciences, national center for http://www.asla.org.au/advocacy/ http://www.asla.org.au/advocacy/ http://www.asla.org.au/advocacy/ http://www.asla.org.au/policy/standards.htm http://www.asla.org.au/policy/standards.htm http://www.asla.org.au/policy/standards.htm http://ejournals.library.ualberta.ca/index.php/eblip/article/view/99/242 http://ejournals.library.ualberta.ca/index.php/eblip/article/view/99/242 http://ejournals.library.ualberta.ca/index.php/eblip/article/view/99/242 http://ejournals.library.ualberta.ca/index.php/eblip/article/view/99/242 http://www.pubmedcentral.nih.gov/picrender.fcgi?artid=35250&blobtype=pdf http://www.pubmedcentral.nih.gov/picrender.fcgi?artid=35250&blobtype=pdf http://www.pubmedcentral.nih.gov/picrender.fcgi?artid=35250&blobtype=pdf http://www.pubmedcentral.nih.gov/picrender.fcgi?artid=35250&blobtype=pdf http://www.schoollibraryjournal.com/article/ca6484334.html http://www.schoollibraryjournal.com/article/ca6484334.html http://www.schoollibraryjournal.com/article/ca6484334.html http://www.peopleforeducation.com/librarycoalition/report03.pdf http://www.peopleforeducation.com/librarycoalition/report03.pdf http://nces.ed.gov/pubs2004/2004313.pdf http://nces.ed.gov/pubs2004/2004313.pdf http://nces.ed.gov/pubs2004/2004313.pdf http://www.ifla.org/vii/s11/pubs/manifest.htm http://www.ifla.org/vii/s11/pubs/manifest.htm http://www.ifla.org/vii/s11/pubs/manifest.htm evidence based library and information practice 2009, 4:2 95 education evaluation and regional assistance, 2006. johnson, eileen. parallel worlds of education and medicine: art, science, and evidence. connexions. 1 may 2009 <http://cnx.org/content/m15762/1.1/ >. kerlinger, fred n. foundations of behavioral research. orlando, fl: harcourt, brace jovanovich, 1986. kersting, karen. ‚bolstering evidencebased education‛. monitor on psychology 34.9 (2003). 1 may 2009 <http://www.apa.org/monitor/oct03/ bolstering.html>. kuhlthau, carol c., leslie k maniotes, and anne k. caspari. inquiry: learning in the 21st century. westport ct: libraries unlimited, 2007. lau whelan, debra. ‚what does your boss think about you?‛ school library journal 48.9 (2002): 52-5. loertscher, david v, with ross j todd. we boost achievement! evidence-based practice for school library media specialists. salt lake city: hi willow research & publishing, 2003. londsale, michele. impact of school libraries on student achievement: a review of the research: report for the australian school library association. melbourne: australian council for educational research, 2003. 1 may 2009 <http://www.asla.org.au/research/in dex.htm>. mandinach, ellen b., margaret honey, daniel light, cricket heinze, and luz rivas. creating an evaluation framework for data-driven. decision-making. new york: education development center's center for children & technology, 2005. 1 may 2009 <http://www.iste.org/content/navig ationmenu/research/ necc_research_paper_archives/n ecc_2005/mandinach-ellennecc05.pdf>. marsh, julie a., john f. pane, and laura s. hamilton. making sense of datadriven decision making in education: evidence from recent rand research. santa monica, ca: rand, 2006. 1 may 2009 <http://www.rand.org/pubs/occasio nal_papers/op170/>. moncur, michael. ‚the quotation page.‛ 1 may 2009 <http://www.quotationspage.com/q uote/39952.html>. morrison, keith. ‚randomized controlled trials for evidence-based education: some problems in judging ‘what works.’‛ evaluation and research in education 15.2 (2001): 69-83. sackett, david l., william m. c. rosenberg, j. a muir gray, r. brian haynes, and w. scott richardson. ‚evidence based medicine: what it is and what it isn't,‛ british medical journal 312 (1996): 71-2. 1 may 2009 <http://www.bmj.com/cgi/content/f ull/312/7023/71?view=full&pmid=85 55924>. scholastic library publishing. school libraries work!: research foundation paper. updated 2008 http://cnx.org/content/m15762/1.1/ http://cnx.org/content/m15762/1.1/ http://cnx.org/content/m15762/1.1/ http://www.apa.org/monitor/oct03/bolstering.html http://www.apa.org/monitor/oct03/bolstering.html http://www.apa.org/monitor/oct03/bolstering.html http://www.asla.org.au/research/index.htm http://www.asla.org.au/research/index.htm http://www.asla.org.au/research/index.htm http://www.iste.org/content/navigationmenu/research/%20necc_research_paper_archives/necc_2005/mandinach-ellen-necc05.pdf http://www.iste.org/content/navigationmenu/research/%20necc_research_paper_archives/necc_2005/mandinach-ellen-necc05.pdf http://www.iste.org/content/navigationmenu/research/%20necc_research_paper_archives/necc_2005/mandinach-ellen-necc05.pdf http://www.iste.org/content/navigationmenu/research/%20necc_research_paper_archives/necc_2005/mandinach-ellen-necc05.pdf http://www.iste.org/content/navigationmenu/research/%20necc_research_paper_archives/necc_2005/mandinach-ellen-necc05.pdf http://www.iste.org/content/navigationmenu/research/%20necc_research_paper_archives/necc_2005/mandinach-ellen-necc05.pdf http://www.rand.org/pubs/occasional_papers/op170/ http://www.rand.org/pubs/occasional_papers/op170/ http://www.rand.org/pubs/occasional_papers/op170/ http://www.quotationspage.com/quote/39952.html http://www.quotationspage.com/quote/39952.html http://www.quotationspage.com/quote/39952.html http://www.bmj.com/cgi/content/full/312/7023/71?view=full&pmid=8555924 http://www.bmj.com/cgi/content/full/312/7023/71?view=full&pmid=8555924 http://www.bmj.com/cgi/content/full/312/7023/71?view=full&pmid=8555924 http://www.bmj.com/cgi/content/full/312/7023/71?view=full&pmid=8555924 evidence based library and information practice 2009, 4:2 96 edition. new york, ny: scholastic library publishing, 2008. 1 may 2009 <http://www2.scholastic.com/conten t/collateral_resources/pdf/s/slw3_20 08.pdf>. scott, leslie. school library media centers: selected results from the education longitudinal study of 2002 (els:2002). washington, d.c.: u.s. department of education, institute of education sciences, 2002. 1 may 2009 <http://nces.ed.gov/pubs2005/20053 02.pdf>. todd, ross j. ‚building capacity and continuous improvement of school libraries: the delaware experience.‛ in world class learning and literacy through school libraries papers from the 37th annual conference of the international association of school librarianship, and the twelfth international forum on research in school librarianship, university of california at berkeley, california, usa, 3-7 august 2008. zilmere, australia: iasl, 2008. todd, ross j. ‚evidence-based practice and school libraries: from advocacy to action.‛ the school library media specialist and education reform. eds. sandra hugheshassell and violet h. harada. westport, ct: libraries unlimited, 2007, 57-78. todd, ross j. ‚the evidence-based manifesto for school librarians: if school librarians can’t prove they make a difference, they may cease to exist‛. school library journal april, 2008. 1 may 2009 <http://www.schoollibraryjournal.co m/article/ca6545434.html?nid=3362 >. todd, ross j. ‚transitions for preferred futures of school libraries: knowledge space, not information place; connections, not collections; actions, not positions; evidence, not advocacy‛. keynote address: international association of school libraries (iasl) conference, auckland, new zealand, 2001. 1 may 2009 <http://www.iaslslo.org/virtualpaper2001.html>. what works clearing house. ‚welcome to wwc.‛ u.s. department of education institute of education sciences. 1 may 2009<http://ies.ed.gov/ncee/wwc>. whitehurst, grover j. ‚evidence-based education (ebe).‛ powerpoint presentation at student achievement and school accountability conference october 2002. 1 may 2009 <http://www.ed.gov/nclb/methods/ whatworks/eb/evidencebased.pdf>. willinsky, john. ‚extending the prospects of evidence-based education.‛ insight, 1.1, (2001): 23-41. 1 may 2009 <http://pkp.sfu.ca/files/evidencebasededucation.pdf>. http://www2.scholastic.com/content/collateral_resources/pdf/s/slw3_2008.pdf http://www2.scholastic.com/content/collateral_resources/pdf/s/slw3_2008.pdf http://www2.scholastic.com/content/collateral_resources/pdf/s/slw3_2008.pdf http://www2.scholastic.com/content/collateral_resources/pdf/s/slw3_2008.pdf http://nces.ed.gov/pubs2005/2005302.pdf http://nces.ed.gov/pubs2005/2005302.pdf http://nces.ed.gov/pubs2005/2005302.pdf http://www.schoollibraryjournal.com/article/ca6545434.html?nid=3362 http://www.schoollibraryjournal.com/article/ca6545434.html?nid=3362 http://www.schoollibraryjournal.com/article/ca6545434.html?nid=3362 http://www.schoollibraryjournal.com/article/ca6545434.html?nid=3362 http://www.iasl-slo.org/virtualpaper2001.html http://www.iasl-slo.org/virtualpaper2001.html http://ies.ed.gov/ncee/wwc http://www.ed.gov/nclb/methods/whatworks/eb/evidencebased.pdf http://www.ed.gov/nclb/methods/whatworks/eb/evidencebased.pdf http://www.ed.gov/nclb/methods/whatworks/eb/evidencebased.pdf http://pkp.sfu.ca/files/evidence-basededucation.pdf http://pkp.sfu.ca/files/evidence-basededucation.pdf microsoft word es_kelson.doc evidence based library and information practice 2007, 2:3  91 evidence based library and information practice       evidence summary    uptodate rated highest in a combined task assessment/user‐satisfaction study of  five clinical information resources     a review of:  campbell, rose, and joan ash. “an evaluation of five bedside information products using a  user‐centered, task‐orientated approach.” journal of the medical library association  94.4 (oct. 2006): 435‐41.    reviewed by:   jennie kelson  library services manager, buckinghamshire hospitals nhs trust  amersham, united kingdom    e‐mail: jennie.kelson@buckshosp.nhs.uk        received: 01 june 2007    accepted: 08 july 2007      © 2007 kelson. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to compare users’ perceptions  of 5 clinical information resources, and to  assess the average number of questions  answered after attempting 3 randomly  assigned clinical questions on each resource.    design – a combined task assessment,  based on the design specification published  in the sixth text retrieval conference  (trec‐6) “interactive track,” and a user‐ satisfaction questionnaire developed from  previously published surveys.     setting – a health sciences library at a  university in the united states of america.    subjects – a convenience sample of 18   volunteers, who were either university   health care staff or students.     methods – a set of 15 clinical test questions  was developed from previous studies.  participants were randomly allocated 3 test  questions, which they then attempted to  answer using each of 5 commercially  available clinical information resources.  each participant was allocated a different set  of test questions for each resource and did  not attempt the same question on more than  one resource. as part of the overall study  design, the questions were randomised such  that each question was paired with each  resource at least once. the order in which  the resources were tested by participants  was also randomised. the resources tested  were acp’s pier, diseasedex,  firstconsult, inforetriever and uptodate.  training in use of the resources was not  mailto:kelson@buckshosp.nhs.uk http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:3  92 provided as part of the study; however,  participants were allowed to familiarise  themselves with each resource before  attempting the test questions. to simulate a  clinical situation, participants were asked to  spend a maximum of 3 minutes on each  question. the number of questions  successfully answered using each resource  was recorded. participants were also asked  to complete a user satisfaction questionnaire,  based on previously published  questionnaires, for each resource after  attempting the 3 questions allocated to that  resource. the questionnaire used a 5‐point  likert scale with participants asked to rate  attributes such as clarity, ease of use, speed  and accuracy of content. a final question  also asked participants to indicate which  resource they liked the best and which they  liked the least. participants also completed a  background questionnaire, again based on  previously published questionnaires,  covering aspects such as age, gender,  experience with searching and previous use  of various information resources, including  the 5 resources being tested.    main results –    characteristics of participants – participants  ranged in age from 28 to 49 years (mean 35  years), and were experienced computer  users with over 94% using a computer at  least once a day. the male (42%) and female  (58%) split of the group was roughly equal.  the participants’ occupations were  physician (44%), medical informatics  student with previous clinical experience  (28%), pharmacist (17%), nurse (6%) and  mri technologist (6%). participants had  been in their current profession for a mean  of 8 years (range 1 to 20 years). whilst 72%  of participants reported familiarity with  uptodate, no more than 12% of participants  reported familiarity with any one of the  other information resources tested.      clinical questions ‐ participants were able to  answer more questions with uptodate  (average 2.5 questions) compared to the  other resources, which ranged from an  average of 1.6 (acp’s pier) to 1.9  (diseasedex) questions answered. this  difference was found to be statistically  significant using the friedman test.      user satisfaction ‐ the user satisfaction  survey results showed no significant  differences in perceptions of the different  resources in relation to accuracy, currency of  content, speed or amount of information  provided. however, uptodate scored  significantly higher (friedman test) on ease  of use, clarity of screen layout and how well  it satisfiedparticipants’ needs.     overall, 13 participants (73%) rated  uptodate the best, 3 (18%) preferred  firstconsult and 1 (6%) rated acp’s pier  best. conversely, inforetriever was rated  worst by 6 participants (38%), 4 participants  (25%) each rated diseasedex and  firstconsult worst and 2 participants (13%)  rated acp’s pier worst. a chi‐squared  analysis found these rankings to be  significantly different.    conclusion – a number of commercial  information resources are now available that  aim to help clinical staff make treatment  decisions at the point of care. this study  evaluates 5 such resources by comparing  both success in answering typical clinical  questions and the results of a user  satisfaction survey.      the study indicates that participants were  able to find significantly more answers  when using uptodate compared to the other  resources tested. whilst there was no  statistically significant difference between  the user perception ratings assigned to each  resource with regard to speed, accuracy or  amount of information provided, participant  ratings for screen layout and ease of use  significantly favoured uptodate. in addition,  evidence based library and information practice 2007, 2:3  93 significantly more participants identified  uptodate as the best resource.    evaluations of clinical information resources  have traditionally focused on user ratings of  the content of these products. the findings  of this study suggest that this approach may  no longer be sufficient, and that evaluations  that address the user’s experience  (satisfaction concerning ease of use, speed,  etc.) are also needed.     commentary     the methodology used in this study is  clearly explained, and each stage of the  research uses, or is based on, previously  published techniques. the test clinical  questions and the questionnaires used in the  study provide a validated method that can  be used by others to evaluate similar  information resources.     a shortcoming of the study, which the  researchers have recognised, is the small  number of participants overall and the use  of a convenience sample of interested users.  as the participant group contained a  proportionally larger number of physicians,  it is possible that the overall results obtained  from this group may not be representative  of the wider potential user population. it  would be interesting to see whether similar  results were obtained if the study was  repeated with a larger group, incorporating  more variation in the clinical occupations  represented. the background questionnaire  also did not ask about the participant’s  race/ethnicity or level of education, and  these might be interesting characteristics to  capture in future studies.     participants were asked to complete the  tests in their own workplace and in their  own time; however, it is not clear whether  the authors had any checks in place to  ensure that participants adhered to the 3  minute limit per question or that the test  environments were comparable. whilst the  authors suggest that conducting the study  outside a laboratory setting enabled a higher  recruitment level, they also propose that  asking participants to complete the tests in  their own time may have accounted for the  high dropout rate (25%). conducting the  tests under standard laboratory conditions  or having a member of the research team  observe participants using the information  resources may have improved the reliability  of the results obtained.    the study results demonstrated that  participants were able to answer  significantly more questions using uptodate,  and that this resource was rated more highly  in the user satisfaction survey. caution is  advised with respect to this conclusion,  since 72% of participants were already  familiar with uptodate whilst very few had  previously used any of the other resources  being assessed. prior experience may have  biased participants in favour of uptodate.  unfortunately, the small number of  participants in this study who were  unfamiliar with uptodate precluded any  analysis of the results on the basis of  previous experience with the resources  tested.      overall, this is a valuable study indicating  that, of the 5 information resources assessed,  participants preferred uptodate and were  significantly more successful at answering  test clinical questions it. repeating the study  with a larger, more diverse clinical group  under more tightly controlled conditions  would improve confidence in the validity  and applicability of the current findings. microsoft word es 4072 evidence based library and information practice 2008, 3:4 72 evidence based library and information practice evidence summary teachers in the uk prefer research evidence that is synthesized, practical, and locally available a review of: williams, dorothy and louisa coles. “evidence-based practice in teaching: an information perspective.” journal of documentation 63.6 (2007): 812-35. reviewed by: stephanie schulte education and reference services coordinator john a. prior health sciences library the ohio state university columbus, ohio, united states of america email: schultes@ipfw.edu received: 01 september 2008 accepted: 07 october 2008 © 2008 schulte. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – the objective of the study was to explore how teachers in the united kingdom use research literature to inform their practice. special emphasis was given to the effect of the teachers’ information literacy and their level of ready access to research information. design – this study was primarily a qualitative study utilizing mixed methods, including individual interviews, focus groups (“literature review groups”), and an online discussion forum. qualitative data was supplemented by a questionnaire survey. setting – the qualitative portions of the study took place in the united kingdom across five education authorities; however, it is unclear where these authorities were geographically. the survey was distributed across scotland, england, and wales. the study was conducted during 2002 to 2003. subjects – nursery, primary, and secondary teachers, school librarians, school library systems, and education authority advisors (ea) in the united kingdom. evidence based library and information practice 2008, 3:4 73 methods – the 28 interviews (54% primary teachers, 39% secondary teachers, 7% special education) and four focus groups (each with three to five participants; 15 participants in total, including teachers at various levels from primary, secondary, and nursery schools) were conducted with volunteers from a random sample across five education authorities. recruitment was done by sending written materials to schools. those interviewed represented five authorities. focus groups were conducted in just four authorities. it is not clear when the individual and group interviews were conducted (time of day and year). volunteers were given the option to receive funding for a substitute teacher in order to participate, though none took advantage of this offer. the interview process used a vignette technique to elicit teachers’ attitudes to a situation in a non-threatening way. interviewees were asked to comment on two situations that might make them seek information. one was adding a new subject or new aspect of an existing subject into the classroom. the second dealt with a “new pedagogical challenge.” though the topics of the vignettes were provided, the exact way the topics were presented was not. they were asked how they would advise another teacher in these situations then relate their thoughts to a real situation they had faced. interviews were about one hour in length. twenty-five of the 28 interviews were taped and transcribed verbatim. extensive notes were made for the other three. transcripts and notes were analyzed using qsr n4 classic content analysis software. themes related to information literacy and information seeking, including barriers, were noted. four literature response groups were given examples of research information (journal articles, reports, etc) related to information computer technology (ict) and class size. each topic had information from various evidence levels and included both print and electronic format. again, the discussion topics were provided in the article, but the exact way the topics were presented was not. over 2 to 2.5 hours, teachers read the materials, made notes, and discussed the information as a group. for the ict information, groups were asked to discuss presentation of the information. for the class size information, groups were asked to discuss content of the materials. discussions were transcribed verbatim and analyzed with the content analysis software. a total of 3899 questionnaire surveys were distributed to teachers (3000), head teachers (500), school librarians (250), education authority advisors (100), and school library systems (49). response rate was exceptionally low in the teacher and head teacher categories, with only 10.9% of teachers and 15.6% of head teachers responding. response rates in other categories ranged from 31.2% to 55%, with the highest response rates from education authorities and school library systems. the survey served to gain an understanding of attitudes about using research in practice across a broad spectrum of stakeholders. the survey instrument itself, which was not provided in the article, was piloted prior to the study, but there is no mention of reliability or validity analysis. to supplement study data, participants who had stated they would be interested in participating in an online discussion were sent preliminary findings and asked to comment on themes using the online discussion forum. only 21 posts were gathered using this method. overall, the methods used in this study are appropriate for the questions that were posed in the article. qualitative studies are useful for gathering data where little is known and where more data would help evidence based library and information practice 2008, 3:4 74 identify possibly hypotheses for further study. main results – teachers in this study relied on a small set of resources and preferred what was readily available at their own school. these teachers most frequently used colleagues, in-service events, the internet, newspapers, and reports typically found in schools as sources of information. sources that information professionals would consider quality evidence were rarely mentioned. these teachers also tended to prefer sources that present information in a practical context. not surprisingly, time was identified as a major barrier to accessing research information in addition to limited access to resources. the internet was identified as the preferred point of access, citing ease of use, speed, and convenience as the reasons for this preference. comments suggested a preference for synthesized information sources. teachers indicated they felt the responsibility for disseminating research information fell on head teachers and ea. they also noted that access to information from their own school library was a problem. they felt librarians and libraries in schools were meant for students, not for teachers, and also expressed that school libraries typically did not provide access to research. comments suggested that the structure of teaching in a classroom itself does not allow teachers to use information as is needed for evidence-based practice. there were concerns expressed about lack of school library funding available to supply research evidence needed by teachers. school librarians and school library services respondents did indicate that they provide a broad range of services aimed at teachers, such as alerts to literature, information skills training, and advice on resources. however, their sources of research for teachers were limited in nature. librarians indicated that teachers did not ask for research information, but if demand were there, they could respond accordingly. teacher comments suggest that librarians might want to consider being more proactive in distributing information. though teachers expressed high confidence in their abilities to find information, comments about search habits and practices contradicted this. most remarks suggested searching techniques that were not sophisticated or showing higher levels of information literacy competency. the respondents were concerned with evaluating the quality of information sources but found this appraisal to be more difficult if the authors did not explain the impact or change in practice that should come from the findings. the participants in this study were likely biased toward using research information more than other teachers. this is an important limitation that the authors do address. additionally, the response rate for the survey was quite low for the teacher subset. despite this limitation, data from the questionnaire was used appropriately to confirm and clarify data from the qualitative portions of the study. conclusion – in order for teachers to appreciate the value of quality research evidence enough to regularly seek it out and place it into practice, a culture of evidence based practice must be embraced and supported by their school authorities. this study indicates significant potential for school librarians and systems to support this culture through proactive dissemination of research, provision of local access to materials, and education related to information literacy. school librarians must evolve from their typical student-centric role to accomplish this. evidence based library and information practice 2008, 3:4 75 commentary though this is a qualitative study conducted primarily with a small sample of nursery, primary, and secondary teachers in the united kingdom, the study data provide a wealth of information for all school governing bodies and librarians to consider. this article is quite long and provides ample comments and explanatory notes on the data gathered. this level of detail is excellent for those forming new hypotheses to test, which is one of the main reasons behind qualitative studies. however, one can get bogged down with so much detail. the authors could have done a better job summarizing the themes present in the data through additional tables and could have also expanded on the implications of the findings. just like the teachers in the study, it is likely that many practising librarians appreciate concise, synthesized information that shows practical implications. though we do know that teachers came from various levels across several authorities, we do not know which authorities they actually represent or the overall socioeconomic status of their school system. schools with more financial resources could possibly support evidence based practice in teaching more so than financially or socially disadvantaged schools. another concern about the data from this study is that the study took place over five years ago. it is possible that today’s teachers have improved access to research or improved support for incorporating research into practice than they did five years ago. as an exploratory study should do, this study brings out questions that librarians need to work to answer. what shortcuts could school librarians provide to busy teachers and education administrators? it is evident that research will not reach the classroom efficiently if local access to synthesized, practically written materials is not available. school librarians can work to improve this situation, customizing as needed for their target populations. furthermore, there appears to be a need for more active collaborations with teachers and school librarians to facilitate the information literacy competencies of both the teachers and their students. even with the methodological limitations considered, this article serves as another call to action to school librarians. evidence summary   while most information literacy research is included in the fields of library science and education, a considerable amount is found in medicine and health   a review of: aharony, n. (2010). information literacy in the professional literature: an exploratory analysis. aslib proceedings: new information perspectives, 62(3), 261-282. https://doi.org/10.1108/00012531011046907   reviewed by: jessica a. koos senior assistant librarian/health sciences librarian stony brook university stony brook, new york, united states of america email: jessica.koos@stonybrook.edu   received: 17 feb. 2021                                                              accepted:  30 mar. 2021      2021 koos. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29927     abstract   objective – to describe the published literature on information literacy from 1999-2009.   design – statistical descriptive analysis and content analysis.   setting – n/a   subjects – 1,970 publications from the web of science database.   methods – the web of science database was searched using the term “information literacy” in the advanced search under “topic,” and was limited to articles published from 1999-2009. next, information such as document type, subject areas, authors, source titles, publication years, languages, countries, keywords, and abstracts was collected from each document. a statistical descriptive analysis was conducted using the data. a content analysis was performed on the keywords and abstracts from a sampling of the results.   main results – information science/library science and education were the top subject areas of the identified articles, while the third largest subject area was “public, environmental and occupational health.” nine out of ten journal titles focused on library science, however the journal title containing the second largest number of articles was patient education and counseling. the content analysis revealed that the most common categories for keywords were “miscellaneous,” “health and medicine,” followed by “education.”   conclusion – the results indicated that information literacy research had been published mainly in journals associated with library science and education; however, a considerable amount of literature was published in health and medicine.   commentary   the quality of this study was appraised using the cat: a generic critical appraisal tool created by perryman & rathbun-grubb (2014). overall, the article was found to be of high quality based on this assessment. the research questions were clearly defined and matched the methods used. the methods were also explained with a sufficient amount of detail to allow for reproducibility. one limitation that was mentioned in the article was that since the database search was conducted in september 2009, any material published after that date was not included in the analysis. another limitation to this study was that only the web of science database was used to identify articles. although the author explained that bibliographic databases are the most effective way of searching the literature, there was no explanation as to why this specific database was chosen or why additional databases were not utilized in the study. searching additional databases would have yielded a more complete set of results. additionally, the analysis of keyword types in the article abstracts indicated that “miscellaneous” was the largest category. this is not very meaningful, and perhaps the coders could have provided a more thorough analysis.   it is interesting to note that the term “health literacy” is not mentioned in the article, as it is very closely related to information literacy. health literacy can be defined as “the degree to which individuals have the capacity to obtain, process, and understand basic health information and services needed to make appropriate health decisions” (ratzan and parker, 2000). a basic topic search in web of science revealed 13,191 results when the phrase “health literacy” was searched. when limited to the dates of the study (1999-2009), there were 1,376 results. there were only 556 publications retrieved when searching for the phrase “information literacy” and the terms “health” or “medicine,” and only 79 results when filtered for the study dates, indicating that the potential connection between information literacy and health/medicine may be greater than indicated in this article.   the major finding of this article was that although the topic of information literacy is largely dominated by articles from the fields of library science and education, the fields of health and medicine are major producers of information literacy-related research as well. the author explained that this could be indicative of the importance of being able to navigate health information, especially since there is so much available to consumers. the author also explained that this research is possibly produced by an increasing number of health sciences librarians, who publish on this topic. future research exploring the credentials of the authors of such articles is needed to verify this claim.   other findings of the research included that most of the publications identified were articles, published in the u.s., and written in english. it was also found that the number of publications on this topic increased over time, implying that it had been a topic of increasing interest during the time period studied. these results verified previous findings in the field.   a growing interest in the field of health literacy, especially due to the current pandemic, highlights the importance of practitioners being aware of their patients’ capability to manage their own health and medical issues it has been proven that poor health literacy leads to poor health outcomes (berkman et al, 2011), and initiatives to further the development of these skills are crucial. library science experts are in a unique position to use their skills to contribute to these efforts.   references   berkman, n. d., sheridan, s. l., donahue, k. e., halpern, d. j., & crotty, k. (2011). low health literacy and health outcomes: an updated systematic review. annals of internal medicine, 155(2), 97–107. https://doi.org/10.7326/0003-4819-155-2-201107190-00005   perryman, c. & rathbun-grubb, s. (2014). the cat: a generic critical appraisal tool. http://www.jotform.us/cp1757/thecat   ratzan, s. c., & parker, r. m. (2000). introduction. in c. r. selden, m. zorn, s. c. ratzan, & r. m. parker (eds.), national library of medicine current bibliographies in medicine: health literacy. national institutes of health, u.s. department of health and human services.   evidence based library and information practice 2012, 7.4 5 evidence based library and information practice article academic librarians’ conception and use of evidence sources in practice denise koufogiannakis collections and acquisitions coordinator university of alberta libraries edmonton, alberta, canada email: dak@ualberta.ca received: 20 aug. 2012 accepted: 15 nov. 2012 2012 koufogiannakis. this is an open access article distributed under the terms of the creative commons‐ attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – the objective of this study was to explore and understand how academic librarians use evidence in their professional decision making. the researcher aimed to gain insights on the relevance of the current eblip model to practice, and to understand the possible connections between scientific research and tacit knowledge within the practice of lis. methods – a grounded theory methodology was used, following the approach of charmaz (2006). participants were 19 academic librarians in canada. data was gathered via online diaries and semi-structured interviews over a six-month period in 2011. results – two broad types of evidence were identified (hard and soft), and are generally used in conjunction with one another. librarians examine all evidence sources with a critical eye, and try to determine a complete picture before reaching a conclusion. as well, librarians use a variety of proactive and passive approaches to find evidence. conclusions – these results provide a strong message that no single evidence source is perfect. consequently, librarians bring different types of evidence together in order to be as informed as possible before making a decision. using a combination of evidence sources, depending upon the problem, is the way academic librarians approach decision making. mailto:dak@ualberta.ca evidence based library and information practice 2012, 7.4 6 introduction evidence based practice (ebp) is a relatively young movement, which began in medicine (guyatt, 1991) and has since spread to other fields, including library and information studies (lis). in lis, very little research has been undertaken on the evidence based library and information practice (eblip) model that was directly adapted from medicine, despite the fact that lis is a social science discipline. this direct adaptation, without reflection on the differences between lis and medicine, has been a noted criticism of the current eblip model (given, 2006; hunsucker, 2007). with roots in evidence based medicine, does the focus on quantitative research evidence apply to librarians whose questions often demand explanations rather than judgments on the effectiveness of interventions? can the current model address academic librarians’ questions and assist with decision making in a meaningful way? the current model may be alienating some librarians who feel that the forms of evidence they are using are not being recognized as important. this research study examined the foundation of eblip by exploring how academic librarians use evidence in their practice. the definition of evidence used within this study was from the oxford dictionary – “the available body of facts or information indicating whether a belief or proposition is true or valid” (2010) – while keeping in mind that within ebp, evidence is generally considered to be research. the study sought to examine whether this was the case in lis practice or whether librarians have a broader interpretation of evidence. the research presented in this paper describes evidence sources used by academic librarians, as well as the reasons these sources are used. it also examines how academic librarians view different sources of evidence, and the differences between what is used in practice and what is conceptually considered to be evidence. literature review evidence sources in evidence based practice evidence based library and information practice is strongly modelled on the original evidence based medicine (ebm) process. the most widely cited and accepted definition of eblip was adapted from mckibbon, wilczynski, hayward, walker-dilks, and haynes’s (1995) definition of ebm, keeping all the same components and basic meaning, but inserting “user” in place of “patient” and “librarian” in place of “clinician”: an approach to information science that promotes the collection, interpretation and integration of valid, important and applicable userreported, librarian-observed, and research-derived evidence. the best available evidence, moderated by user needs and preferences, is applied to improve the quality of professional judgements. (booth, 2000) the ebm movement has generally focused on research studies as the primary source of evidence. ebm has produced many tools for practitioners, to assist them with critical appraisal of research evidence and with determining the strength of the research evidence. there has been criticism that evidence based models do not account for other forms of knowledge that are a vital part of professional practice (brophy, 2009; clark, 2011; davies, nutley, & walter, 2008). built into the ebm model is a hierarchy of evidence (howick et al., 2011; suny, 2004) which eblip has also mirrored (eldredge 2000a, 2000b, 2002). in the hierarchy of evidence, research methods such as randomized control trials are at the top of the hierarchy because they are more likely to be free of bias. while the levels of evidence are a well-known aspect of eblip, they are not something that the eblip community has wholeheartedly accepted. the application of such a hierarchy has been a concern for many within the field (banks, 2008; booth, 2010; evidence based library and information practice 2012, 7.4 7 crumley & koufogiannakis, 2002; given, 2006; koufogiannakis, 2010). beyond research evidence in the evidence based medicine model, scientific research is the main concept explored in relation to practice. however, there are other evidence sources beyond research that impact professional practice and decision making. in this study, practice theory was used as an alternative lens to view the eblip model. practice theory explores what people actually do in practice, and examines how the active doing of a practice leads to knowledge that is important to that practice. schatzki’s (1996) book, social practices, was the first to wholly focus on the practice concept. in that seminal work, schatzki outlines the theory of practices and the necessity of action within practice. a key element of practice theory is the concept of knowing in practice. in practice, knowing has two elements that cannot be separated; these are “knowing how” and “knowing that,” phrases first coined by ryle in 1945. knowing that relates to the mind, and how to do a particular thing, so that it is explainable. knowing how relates to doing the thing, or action, even if one does not know how to explain how one has done it (tacit knowledge). polanyi (1966) was the first to delve into tacit knowledge, explaining it as “we can know more than we can tell” (p. 4). schön, building upon the work of polanyi, writes in his influential 1983 work, the reflective practitioner: how professionals think in action, that “our knowing is in our action” (p. 49). for schön the work life of a professional depends on this tacit knowing in action. schön says: “even when [the practitioner] makes conscious use of research-based theories and techniques, he is dependent on tacit recognitions, judgements, and skilful performances” (p. 50). the two aspects, research and professional knowledge, must go hand in hand. looking beyond theory, several professions are beginning to embrace a practice-based evidence approach in addition to an evidence based practice one. in the fields of medicine and nursing, gabbay and le may (2011) have done ethnographic research to reveal how clinicians acquire and use their knowledge. they convey the importance of “knowledgein-practice-in-context” (p. 65), and note that medicine is an art in addition to a science. it requires judgment and decision-making skills in addition to scientific knowledge. many professional fields have also examined the importance to professional practice of evidence sources other than scientific research (clark, 2011; fox, 2003; rolfe, jasper, & freshwater, 2011; usher & bryant, 1989), not rejecting research but widening the conception of what is required to make good decisions in practice. for practitioners, learning occurs via doing (schön, 1983). within communities of practice (lave & wenger, 1991), the focus of this learning is on the social nature of the community. practitioners learn from others within their community, and likewise contribute to that learning. communities of practice occur whether one is conscious of them or not. in an unconscious format, practitioners rely on their internal networks to assist with learning tacit dimensions of their work, via conversations with colleagues, interactions in groups, and verification from peers. duguid (2005) explains that in becoming a practitioner, one needs to “learn to be,” which is part of ryle’s concept of “knowing how,” embodying the art of practice and tacit dimensions that are not easily made explicit. communities of practice have the potential to allow for individual practitioners to bring their practice-based knowledge to a conversation within their practicing community. practicebased knowledge is therefore made more explicit, and learning occurs within the group, ultimately influencing practice decisions. how academic librarians function within their communities of practice, and how these communities affect their knowledge and decision making, are of interest to this study because it is within such communities that tacit knowledge is formed. evidence based library and information practice 2012, 7.4 8 aims the aim of this study was to explore and better understand how academic librarians use evidence in their professional decision making. the purpose was to gain insights on the applicability of the current eblip model for lis practitioners, and to understand the possible connections between scientific research and tacit knowledge within the practice of lis. the following research questions were posed:  what forms of evidence do academic librarians use when making professional decisions? why do they use these types of evidence?  how do academic librarians incorporate research into their professional decision making? methods the study used a grounded theory methodology, following the approach of charmaz (2006). the methods used to collect data were online diaries (blogs) and semistructured interviews. ethical approval was received from both aberystwyth university, where the researcher was a student, and the university of alberta, where the researcher is employed as a librarian. the study used a purposeful sample of canadian academic librarians who had some interest in exploring the use of evidence in relation to their professional decision making. although the research was targeted at academic librarians, a wide variance was sought, and so an open invitation to participate was sent out on mailing lists that are used by academic librarians, such as the canadian association of college and university libraries mailing list, and the evidence based librarianship interest group of the canadian library association. the invitation was also sent out on twitter. twenty-one librarians initially agreed to participate in the study. two librarians later dropped out, due to time constraints, leaving a total of 19 participants. this number was sufficient to reach saturation of the data, and included variance amongst participants including demographics, work types within academic libraries, and knowledge of eblip. the 19 participants were geographically dispersed across canada and were all englishlanguage speakers. all worked in academic positions, identified themselves as academic librarians, and worked in a variety of roles and subject areas. the participants’ number of years of experience as librarians varied widely, ranging from less than two years to more than 30 years. they represented all levels of experience, from new librarians in their first job, to senior librarians nearing retirement. some librarians had many years of experience but had recently begun new positions, while others had been in the same position for many years. each participant’s familiarity with evidence based practice was assessed based on an analysis of comments in the diaries and interviews, and it was determined that eight participants were very familiar with ebp, three were moderately familiar, and eight had very little to no familiarity with ebp. the process of data collection occurred over a period of nearly six months, simultaneously in conjunction with data analysis. data collection occurred in a theoretical manner; as concepts emerged and patterns were discovered, the researcher followed up on those emerging concepts with the later participants. the study aimed for depth and richness of information rather than higher numbers of participants; the data is not meant to be generalized, but will be used to provide insights that may aid in the development of theory regarding evidence based approaches in librarianship. participants wrote in their online diaries for a period of one month. they were asked to note questions or problems that related to their professional practice and how they resolved those issues (see appendix a). participants used wordpress.com online blogging software, which allows for blogs to be kept private. all participants who completed the diary portion of the research agreed to a follow-up interview. the semi-structured interview process (see appendix b) allowed clarification evidence based library and information practice 2012, 7.4 9 and deeper analysis of specific aspects that participants may have noted in their diary entries, and allowed participants to look holistically at their experiences and to comment on the overall process. given the wide geographic distribution of participants across canada, most interviews were conducted via telephone or skype. all interviews were taped using a digital recorder. audio tapes were transcribed by a professional transcriptionist and checked for accuracy by the researcher. analysis of the diaries began as each was completed, using the constant comparison method to closely analyze the text and discover and group concepts related to the decision-making process of participants. this process of comparing each incident in the data with other incidents, and doing so continually as the data is gathered, allows the researcher to determine analytic similarities and differences (charmaz, 2006; corbin & strauss, 2008). as additional diaries and interviews were completed, the information gained from the earlier data was used to refine concepts and discover new ones, as is the norm within the grounded theory method. memo-writing was used to keep a reflective record of the approach to the research as well as emergent concepts. an open coding approach was used on a printed copy of the diary and interview transcripts, and later transferred into the nvivo software program, which was used to assist with the management of data analysis. very specific codes were later grouped into categories, as analysis was refined and a picture of the findings began to emerge. saturation of the data was reached by the 16th interview, when no new theoretical insights arose from the data and no new categories emerged when coding. findings the concept of evidence while the interviews in this study were semistructured with a focus on following up on situations that participants had raised in their blog diaries, participants were asked a direct question about what they considered to be evidence. other than one participant, all who responded to the question about what they considered to be evidence were very open to the possibility of what evidence could be within librarianship. responses that exemplified this outlook included “there are lots of things that are evidence” (librarian 10) and “i consider every information source to be evidence” (librarian 14). most participants named several sources of evidence, and usually put those in context. for example, they chose different evidence sources depending upon the problem faced. all participants noted research literature, or simply “literature,” as evidence, often qualifying this source in terms such as “obviously” or “of course.” however, there were some caveats put on the inclusion of published literature, due to the participants’ discomfort with the quality and relevance of literature they have found in the past. this is exemplified by librarian 10 who said “obviously research is another kind of evidence although it is not totally implacable” (librarian 10, interview). another concept mentioned very frequently as evidence was “looking at what other libraries do.” this evidence may come from the literature in the form of descriptive articles about an innovative service at a particular library, but may also be found by examining other libraries’ websites or catalogues, speaking with librarians at other institutions, or hearing about other library experiences via a conference presentation or an electronic mailing list. this type of evidence provides ideas and insights relating to a problem that a librarian may be working on. as librarian 20 noted: i do find that hearing the experience of other librarians, getting some of their ideas – maybe it’s not what you would term hardcore evidence, but i do find that that really just generates ideas, better ways of doing things or more interesting things. (librarian 20, interview) evidence based library and information practice 2012, 7.4 10 “what other librarians do” also provides a starting point and guidance when approaching a problem one has not encountered before, or when trying something new. there is a reassurance in knowing how things worked for someone else, particularly peer-sized institutions that have similar populations. such insights provide the level of detail that inquiring librarians need, as they are able to ask specific questions. data, or what is commonly referred to as statistics, was another key area mentioned by participants when asked to discuss what they thought of as evidence. keeping statistics on reference transactions, circulation of books, usage of electronic journals, interlibrary loan requests, and so on are very common in libraries. hence, it is not surprising that academic librarians are looking to those sources as evidence to help with their decision making. as librarian 11 pointed out: i think my gut reaction is that i want numbers of things. i want quantitative information. i want numbers of transactions, numbers of uses, and so on. i think that’s probably a fairly shallow interpretation of evidence, but that’s the kind i like. (librarian 11, interview) as with the literature, most librarians were also cautious about statistics and often qualified their statements by noting that there were problems with this type of evidence, and that it could not simply be viewed in isolation. very often, librarians referred to the need to look at many types of evidence, particularly depending upon the situation. this is exemplified by librarian 14, who stated: i consider every information source to be evidence. and i guess i mean that in the very broadest category, so it could be someone’s opinion or it could be a report. i feel confident in my ability to judge whether evidence is credible or not. so, i think i would look at everything. i wouldn’t discount anything. (librarian 14, interview) regardless of whether they felt certain that some sources really were “evidence” or not, participants did mention experience, opinion, and anecdote. these seem to fall into a grey area, as most people who mentioned them did not feel absolutely comfortable or certain that they were evidence sources. one person was very certain that they were not, and another that they were. but most were unclear about these sources, acknowledging that they were used, but uncertain about whether they could or should be considered evidence. academic librarians generally have a very wide view of evidence, while at the same time, they are for the most part unsure of what constitutes evidence. they want to consider evidence carefully and are willing to take into account whatever may help them with decision making. they also consciously weigh evidence in an effort to make a good decision with the available evidence. for this, they rely on their own professional judgment and knowledge of what is most important in a particular situation. evidence sources used the evidence sources used by academic librarians were numerous and detailed. in order to best convey this information, the evidence sources were grouped into two overarching types, hard evidence and soft evidence, at a final stage of the coding process in order to make a distinction between the types of evidence that were used or mentioned by participants. there were a total of nine categories of evidence, which are listed in table 1. “hard” evidence sources are usually more scientific in nature. ultimately, there is some written, concrete information tied to this type of evidence. a librarian can point to it and easily share it with colleagues. it is often vetted though an outside body (publisher or institution) and adheres to a set of rules. these sources are generally acknowledged as acceptable sources of evidence, and are what a librarian would normally think of as evidence in lis. evidence based library and information practice 2012, 7.4 11 the other type of evidence can be thought of as “soft” or non-scientific evidence. these evidence sources focus on experience and accumulated knowledge, opinion, instinct, and what other libraries or librarians do. this type of evidence focuses on a story, and how things fit in a particular context. soft evidence provides a real-life connection, insights, new ideas, and inspiration. these types of evidence are more informal and generally not seen as deserving of the label “evidence,” although they are used by academic librarians in their decision making. hard evidence sources published literature an important source of evidence consulted by academic librarians is the published literature. the published literature includes journal articles from both lis journals as well as nonlis journals, and can include both research articles and non-research articles, and quantitative and qualitative studies. it also includes books, databases, guidelines, bibliographies, and any other similar source table 1 sources of evidence used by academic librarians evidence source definition examples hard evidence published literature scholarly publications that have been vetted via a publication process journal articles (research and non-research), books, databases, conference papers, etc. statistics data pertaining to the use of a particular product or service usage statistics, reference statistics, circulation statistics, etc. local research and evaluation the evaluation and assessment of services course evaluations, surveys, focus groups, etc. other documents non-scholarly publications that provide information about a service, event, or person policies, web pages, blogs, course materials facts things that the majority of people agree to be true cost of a product, date of a publication soft evidence input from colleagues going to colleagues to ask their advice or feedback, or for information about a program or service that they may know about discussions, feedback, brainstorming, conference presentations tacit knowledge knowledge that is embodied by an individual and difficult to transfer to another person experience, intuition, “common sense” feedback from users individual feedback received from users on products or services comments, discussions, email anecdotal evidence “information obtained from personal accounts, examples, and observations. usually not considered scientifically valid but may indicate areas for further investigation and research” (jonas, 2005). stories, observation evidence based library and information practice 2012, 7.4 12 that has been published. participants noted that the literature provides them with a wider context, background information, and theoretical models. it also reinforces certain principles and reassures them of what they are doing. as the following comment illustrates, the literature reassures that one is on the right track: so, the lit search, i think it was useful, at least in terms of giving me confidence that i wasn’t overlooking anything major. that the stuff i had figured out was about right. (librarian 1, interview) the literature is rarely consulted in isolation. it is considered as just one piece of evidence in a decision and is often used for background information gathering when one is faced with a new problem. however, the literature does not always offer sufficient answers. librarians find the literature somewhat useful, but at the same time disappointing. they wish that the quality of the library literature was higher and that it was more relevant to their practice. sometimes, they do not find anything in the literature, or what they do find is not useful. however, no participant was ready to completely disregard the literature. while participants noted different types of literature and occasionally mentioned types of studies or the lack of good research, they detailed differences between specific types of research literature. statistics data in the form of library statistics is a very common source of evidence among academic librarians. participants frequently mentioned using information such as usage data, circulation statistics, reference statistics, interlibrary loan data, room bookings, and web usage data. this type of evidence is most common when problems arise relating to collection management, and also reference services. participants generally felt that such statistics provide an overall picture of the general situation as it pertains to use of a particular collection or service. for example, in comparing journals in a particular field, usage statistics would be looked at in order to determine what journals are being most heavily used by faculty and students. this would be considered very strong evidence when faced with decisions about possible cancellations. as librarian 8 commented: “i can’t quite think of a way to assess a resource without usage statistics” (librarian 8, diary). echoing this, librarian 5 noted: “from my perspective, i need to be able to support positions for or against purchases, cancellations, etc. i tend to base these on usage stats and acknowledge this” (librarian 5, diary). however, while participants used this type of evidence in their decision making and were frustrated if it was not easily available, they also pointed out that such information could not be used in isolation since there are limitations to relying on such data. participants emphasized that data and statistics were only one part of the story, and that context and other forms of evidence were also required before making a final decision. local research and evaluation academic librarians frequently incorporate evaluation and assessment of services into their work. many also take on research projects that are connected in some way to the work they do. while empirical research projects may be more scientifically rigorous, this type of work is usually not undertaken as frequently as local evaluations of projects or teaching. such evaluation is a source that academic librarians find useful in the ongoing improvement of their services. for example, when referring to instruction decisions, librarian 7 stated, “i find, probably, evaluations are the most – the best evidence that we have” (librarian 7, interview). sources in this category that were cited by participants include total market surveys such as libqual, university surveys that include the library, time audits to measure workload, staff surveys to generate feedback on workload, in-house surveys, testing how something works, evaluation of instruction, swot analysis, workplace climate surveys, individual research projects, preand post evidence based library and information practice 2012, 7.4 13 assessment instruction surveys, and web usability testing. such tools are useful to academic librarians who want input from the communities they serve, or from the staff that work at an institution. for example, librarian 8 had looked to the literature and discussed the situation with her colleagues, but still did not feel that she had all the evidence required to make her decision about a reference project. she concluded: “i’m convinced that i need to hear the voices of actual users. so, i’ve planned to undertake 3 focus groups next week” (librarian 8, diary) other documents this category includes non-scholarly sources that participants used, such as job postings, position descriptions, brochures, mandate documents, safety standards, collection policies, websites (particularly those of other libraries), collective agreements, internal procedure documents, blogs, twitter, and consultants’ reports. these types of documents are not scholarly or research based, but they provide pertinent information that may be useful in making decisions. for example, policy and procedure documents will guide what librarians decide in order to conform to the goals of the overall mission of the institution: “is the decision consistent with our policies and procedures?” (librarian 2, diary). overall, this category of evidence is a broad one, ranging from the official publications of a university, to those documents that are “on the fly” as pointers or tidbits of information, from sources such as twitter. despite this, all these types of “other documents” are a source that librarians draw upon, and are relevant depending upon the situation. facts facts are what the majority of people, if not all, agree to be true. in academic librarianship, some of the things that can be placed in this category include the cost of products, physical condition of materials, citation or publication information, what items are in the catalogue, license terms, the amount of physical space available, and hours of operation. facts are generally not disputed, although they may be occasionally. academic librarians use facts in their decision making in order to place certain realities around the decision, or to verify details before making a decision. for example, if a library has a $10,000 budget for a new resource but it costs $15,000, the fact of the budget amount in conjunction with the cost of the project may alone determine the decision (unless one or both are negotiable). another example would be deciding when to keep or cancel a subscription: checking the catalogue record confirmed: we have only a couple of issues of either publication – with so few issues, i questioned the usefulness of having them in the collection at all; they are not available electronically, they are not indexed, one of the titles appears to be the continuation of another title – which we do not have. (librarian 6, diary) soft evidence sources input from colleagues advice, feedback, and information from colleagues about a program or service are very common sources of evidence for academic librarians. almost all participants mentioned this as part of their decision making, whether they conceptualize it as evidence or not. “colleagues” were generally considered to be other librarians, but this was not always the case. getting input from colleagues, both from within and outside their institutions, provides academic librarians with a way to learn from others who have more experience in a particular area. it also provides confirmation of direction and support for the decision. this type of interaction combines the evidence of experience and knowledge with factors relating to the politics of the institution. it gives the librarian a sense of what other librarians do, and becomes a confirming experience. for many, it is also a way to obtain different viewpoints from one’s own, ensuring that the full picture is considered: evidence based library and information practice 2012, 7.4 14 i never want to sort of leave something with just my opinion. i want to see if i can find a couple of other varying opinions to inform what i’m doing. so, maybe it is evidence that informs me because at that point once there is an absence of anything that’s documented, i still think it’s valuable to then go and talk to peers or experts. (librarian 4, interview) ways of gaining such input from colleagues include one-on-one conversations, attending conference presentations, asking someone to critique teaching or writing, networking at group events (including conferences), corresponding via email or phone, and getting informal feedback from a number of people. this is usually undertaken in conjunction with other forms of evidence (hard sources), but this type of input is considered very valuable for providing insights and knowledge that cannot be gained from the more concrete sources of evidence. hence, combining what is found in the literature, or what statistics demonstrate, with the professional experience of colleagues puts other sources of evidence in context, provides insight, and highlights any potential problems. tacit knowledge as explained in the literature review, tacit knowledge refers to what we know but cannot easily explain (polanyi, 1966). participants mentioned their own professional knowledge acquired via experience and education, professional judgment, intuition, and reflection as elements they draw upon to guide their decision making. participants generally combined this type of evidence with other sources in order to strengthen and verify their decision making, but do not discount their knowledge and experience as irrelevant. as librarian 6 reflected in her diary: now i’m finding – as a result of more experience, confidence, knowledge, maturity – how important those initial gut reactions/instincts are and i’ve learned how to trust them and work with them and pay attention to them – however insignificant that may be. i’ve learned to bracket those instincts and look to the evidence – but in a way that is realistic and appropriate to the situation/question/issue. (librarian 6, diary) the academic librarians in this study used tacit knowledge very heavily in their decision making. this is evident in the number of references to tacit knowledge that arose in both the diaries and interviews. what is interesting is that tacit knowledge reveals itself when participants describe how they made decisions and the sources upon which they draw, but when they are directly asked what they consider to be evidence, tacit forms of knowledge are rarely mentioned. most librarians combine the tacit knowledge aspects of what they know as individual professionals and use it in conjunction with external evidence in order to make decisions. feedback from users obtaining feedback from library users arose in this study as a minor source of evidence. when it is more rigorous (as part of a study or planned evaluation), it can be placed in the category of local research and evaluation, which usually focuses on users of a service. however, it is included here as the individual feedback that librarians receive on products or services. this type of feedback is used most frequently in collections management, and also teaching and instruction activities. faculty feedback that is related to collections is most often looked favourably upon as a source that holds a great deal of weight in decision making. student feedback is also important to academic librarians, particularly as it relates to information literacy instruction, since librarians want to ensure they are helping the students be successful. in addition to formal evaluations, the informal feedback received following an instruction session is a valuable tool for reinforcement or as an indication that something needs to change. it may result in evidence based library and information practice 2012, 7.4 15 changes being made to a presentation or style of teaching for the following session. anecdotal evidence anecdotal evidence is “information obtained from personal accounts, examples, and observations. usually not considered scientifically valid but may indicate areas for further investigation and research” (jonas, 2005). most academic librarians would not include this in a conceptual discussion of what they consider to be evidence; however, it is a source of evidence that is often drawn upon when making decisions. librarian 16 mused about the usefulness of anecdotal evidence in relation to a collections and access issue: i guess even anecdotal evidence can be – to look at where it confirms or differs from available evidence and then go from there and try to figure out what’s happened and why; why all the librarians think everybody wants to have circulating current issues of journals and there’s no evidence showing that people are asking for this. (librarian 16, interview). anecdotal evidence may be the prompt that sets investigation of a potential problem into motion, and it is often used in group conversations when determining a course of action. this type of evidence is most frequently frowned upon as not being worthy, but in the absence of anything else, it is certainly used. most often, librarians will look to other sources of evidence to confirm or deny anecdotal evidence; as librarian 15 points out, “anecdotally i know about things like that. but you know, having some actual evidence would be helpful” (librarian 15, interview). how academic librarians find evidence data from the diaries and interviews was also coded according to how the participants obtained the evidence they used to make a decision. this coding resulted in five categories relating to how academic librarians find evidence when faced with a problem or question related to practice. the examples in table 2 come directly from the participants’ actions, and the grouping of these into broader methods of information finding was done by the researcher. table 2 how academic librarians find evidence method how examples pull proactive and specific literature search in databases; google (internet) search; gathering statistics for circulation or journal usage; looking up facts; asking colleagues questions related to their experience or sources of information push passive, general awareness notifications via toc services; twitter; rss feeds; attending conferences and listening to presentations; colleagues passing on information; getting feedback from users; anecdotal evidence (hearing stories) create proactive and specific including evaluation with instruction; doing a research project related to the problem; conducting in-house surveys or focus groups; keeping reference statistics reflect proactive examination of knowledge and experience carefully considering context and what is known about the situation; tacit knowledge (unique for each person) serendipitous discovery passive, by chance coming across an article or some other document or piece of evidence that is related to your decision, even though not directly looking for it (for example, picking up a journal and while flipping through it, finding something relevant); seeing something in the news that points to a source that is relevant evidence based library and information practice 2012, 7.4 16 the first and most obvious method of finding evidence to help with decision making is what is known as pulling the information required from various sources (“pull”) (cybenko & brewington, 1999). this is a very proactive way of obtaining information, and allows librarians to be specific about their needs. as librarian 4 commented: “i searched, i looked, i asked” in her quest to locate evidence. doing a literature search is a well-known way of pulling evidence on a particular topic. other ways of using the pull method would be searching google (internet), gathering statistics for circulation or journal usage at the point of need, looking up facts, and asking colleagues questions related to their experience. while discussing the management of approval plans with a monograph vendor, librarian 15 commented: i also use the vendor’s database site so i can see what the effects of adding a particular variable to a search would be. for example if i want to see how many slips would be received annually by our education selector in the lc section g73 (geography – study and teaching) i can run a search for that lc class, limiting it to appropriate readership levels and one calendar year. this way i can determine whether or not the slips are appropriate in content and if the number of slips is reasonable. (librarian 15, diary) a passive way of obtaining evidence is to have it pushed to you (“push”) (cybenko & brewington, 1999). setting up table of contents alerts or rss, following individuals or organizations on twitter, attending conferences, and listening to presentations are all ways in which evidence sources are pushed to librarians. since these sources are not the result of a specific search for information on a topic, much of what the librarian receives and filters through may not be directly relevant to the problem at hand, but often such sources provide an early indication of trends or aspects of practice that are changing, or new innovations. as one participant noted: “i have a lot of notifications coming over my desk so i see what sort of the trends are typically in the field so i feel like there are lots of things to learn” (librarian 10, interview). upon learning of new things via this method, an academic librarian may then further move to the pull method for more information. academic librarians also create their own evidence sources. this is very proactive and is usually in reaction to addressing a specific need. it includes situations where librarians conduct research or evaluation in relation to their work. some examples are including formal evaluation with instruction, designing a research project related to a problem, and keeping reference statistics so that trends in the use of reference service can be monitored over time. evidence sources that are created are generally used in-house for local decision making, but may also be published and fed back into the evidence base used by others: we – library administration – are looking for ways to improve productivity, efficiency and engagement within the unit, and are considering adding an additional layer of supervision to the existing structure. it has been challenging getting enough staff members to participate in frank discussion on the topic, and to articulate what they see as the major areas in need of improvement in the area. to help with this, we administered a survey to staff which yielded some helpful qualitative evidence with respect to how staff members view a variety of issues within the area and how they might be improved. opinions we suspected might be held broadly by staff members ended up not to be, and vice versa, which has helped to crystallize some of the planning initiatives we had in mind. (librarian 11, diary) “reflection” is another way that academic librarians find evidence, by taking time to carefully consider the problem at hand and draw upon their past experiences and knowledge in relation to the problem. considering the context of the problem, and what a librarian knows about the circumstances and people involved, is often very important for how to best approach a given situation. schön (1983) argues that such reflection allows practitioners to better deal evidence based library and information practice 2012, 7.4 17 with situations that are uncertain or unique. reflection on what is done, and how, strengthens the soft forms of evidence discussed earlier: i like to reflect, you know, when i’ve gathered the evidence i like to reflect, depending on how complex the situation is. but i’m finding more and more that taking some time to reflect is extremely useful and whether that’s – even if that’s half an hour or overnight, i like to give myself time to think about all the evidence that i’ve collected and let it ruminate, let it kind of come together and it helps me with seeing a direction. it helps me if i miss anything. you know, have i missed anything, or misread anything? because sometimes i’ll go back again to the evidence and look at it again and then i realize oh, actually this person said this and i took it to mean this, but actually now that i read it again i see that it means this. this changes things. so i’ve found that to be very useful, that reflection as part of the evidence. (librarian 6, interview) a final way that academic librarians find evidence is by obtaining it serendipitously. “serendipitous discovery” happens almost as if by accident, when librarians find something they weren’t expecting to find as a pleasant discovery. foster and ford (2003) conclude from their research that “serendipity would appear to be an important component of the complex phenomenon that is information seeking” (p. 337). in the case of academic librarians this may mean coming across an article or some other document or piece of evidence that is related to a decision, even though they were not directly looking for it. such discovery is passive, although subconsciously one may be looking for things that relate to the problem at hand. librarian 3 titled one of her blog posts “serendipity!” and went on to state: i knew that acrl had guidelines for instruction programs in academic libraries but i also knew that they are fairly out of date – 2003. i was just reading the latest issue of college and research libraries news (usually they sit for months on my desk before i have get to them but for some reason i opened the february 2011 issue) and i see that they have updated draft guidelines out! i looked at the acrl site, and they also have a new draft of characteristics of best practices of programs of information literacy! these are going to be very useful as we figure out what to do with our program. (librarian 3, diary) discussion evidence sources this study showed that there are benefits to both broad types of evidence that were identified. hard evidence sources are generally more scientifically rigorous; they confirm or add to what librarians may already know based on past experience and professional knowledge. they also increase confidence, and other people place more value in hard sources of evidence. hard evidence can be used for convincing purposes, and ultimately increases the depth of professional knowledge. soft evidence sources are also important; knowledge and experience allow librarians to judge situations and make quick decisions when necessary. soft evidence enables the necessary analysis and reflection on hard evidence sources, and facilitates putting problems into context. it is important to consider whether both types of evidence are equal and whether soft types of evidence should really be considered valid evidence. this study showed that both types of evidence were used and valued by academic librarians. however, it was only the hard evidence sources that were truly thought of as evidence by participants. this makes sense, as many of the soft sources of evidence stem from already-acquired internal knowledge; evidence is viewed as something that is external and gathered as proof to assist with solving problems and making decisions. for evidence based practice, which seeks to apply the best documented evidence, the evidence focus turns to the hard sources of evidence, which evidence based library and information practice 2012, 7.4 18 need to be gathered and critically evaluated. eblip must also remember the role of the soft evidence, however, and note its importance. evidence sources vary depending on the type of problem. for example, as agor (1989) and dane and pratt (2007) point out, there are situations when expert intuition is useful and best used. these include situations with significant time pressures and high uncertainty, in which a quick judgment needs to be made. in these situations, consulting an experienced practitioner (expert) in the field is best to make the decision, and intuition can be effective. such scenarios occur in libraries when there is an emergency situation, a problematic patron, or a difficult human resource issue, to name a few examples. decisions have to be made quickly and the soft sources of evidence very much come into play by helping librarians make good decisions in such circumstances. however, for decisions that are more planned and have time for investigation, the soft evidence offers a basis of knowledge from which to work and assist with the process of decision making. in these cases, the librarian would use the hard evidence sources to develop a more complete picture based on data, facts, and research in order to come to a logical conclusion about the best decision. the evidence sources used would be those that are most appropriate depending on the question. for example, in the case of designing an information literacy service for a university, the group working on the strategy would look to the research literature, seek out articles about what other institutions have done, examine any past information literacy evaluation that had taken place at the institution, consider learning outcomes tied to the curriculum, talk with faculty, and so on. many sources of evidence would be weighed to enable the team to come to a decision on the best way to provide service in that particular library. this study confirms that in academic librarianship, the forms of evidence are much broader than just research. both soft and hard evidence sources are used in conjunction, bringing together the science and the art of practice. the art of the craft allows librarians to embrace messy situations, find ways to be creative, and put professional judgments to use in order to find the best solutions to meet the needs of individual users. this is achieved by applying the best of what is found in the research literature together with the best of what practitioners know is likely to help a person. the science allows for certainty and confirmation, and builds the overall knowledge base. the findings show that research is valued by academic librarians and is used as an evidence source in decision making. however, academic librarians do not automatically assume that research is good or beneficial just because it has been published. they look at research with skepticism and want to ensure that the research is applicable to their own situations. the research literature alone rarely provides specific answers to the questions that practitioners have. it is almost always used in conjunction with other forms of evidence, including soft sources such as professional knowledge and intuition. librarians also incorporate other evidence sources such as statistics, local research and evaluation, and input from colleagues, in order to look at many variables prior to making a decision. implications for evidence based library and information practice while the definition of eblip noted earlier (booth, 2000) includes professional judgments, it does so only in a way that indicates that application of evidence to those professional judgments will improve them. it does not clearly account for the place of professional knowledge, nor is professional knowledge accounted for in the eblip model. lis professionals must reconsider this exclusion. based on the findings of this study, it is clear that professional knowledge and evidence sources are used together, and they are important aspects of the decision-making process. if broadly interpreted, the eblip definition covers much of what this study has found to be used by librarians in their decision making, but has a specific focus on research. the concept of “evidence” should be broadened to include more than the traditionally recognized research article evidence based library and information practice 2012, 7.4 19 (figure 1). eblip should include other types of data and recognize local circumstances. being “moderated by user preferences” is an important part of the definition, but is rarely explored in the eblip literature. user preferences are necessarily local and can be found through the evidence sources of usage statistics, feedback, local evaluation, research, and even anecdotal evidence. while the need to produce high-quality research that is applicable to practice remains (and this goal of the eblip movement should in no way be discouraged), this study shows that there are other forms of evidence beyond research that are also necessary for librarians to make decisions in their daily practice, regardless of the quality of the research literature. many professional librarians’ questions require local sources of evidence that cannot be obtained from the literature. for example, if the problem or question relates to reference service, then reference usage statistics should be considered, as should local feedback and potential local service evaluations. the eblip model should account for these as legitimate sources of evidence and should provide assistance for librarians in determining the best way to use these sources, similar to critical appraisal tools that have been developed for research articles. the eblip movement needs to discuss and debate the topic of what counts as evidence and how librarians can weigh different forms of evidence. in the future, eblip could focus on how to do better project evaluations, how to interpret user statistics, the best methods for collecting reference statistics, and so on. eblip was built on the ebm model, but in lis many different forms of evidence are used that also need to be considered. as noted, research found in the literature is often not directly relevant to the situation at hand. input from colleagues provides confirmation and support from those who know the local situation and the nuances of why things may or may not work within a specific context. hence, both aspects are important in academic librarians’ decision making. this is in keeping with the literature of practice-based evidence which stresses the importance of soft evidence sources. the same can be seen in other professions. in health care, for example, gabbay and lemay (2011) found similar results in their ethnographic study on the acquisition and use of knowledge by health care professionals. they developed the concept of “mindlines” and observed that judgment and “knowledge-in-practice-in-context” (p. 65) are essential. the mindlines concept demonstrates the importance of skills and knowledge beyond what is found in the research literature, and its contribution to decision making. figure 1 evidence sources in librarianship evidence based library and information practice 2012, 7.4 20 a model of eblip could take a holistic view of evidence, including that which is driven by practice as well as research. proponents of eblip should consider how evidence may be used in practice, and tie research and practice together rather than separating them. a first step is to recognize that what practitioners do is of utmost importance. obviously, without the practitioner, there is no practice, and practitioners are the ones who know what is happening within their contexts. practitioners use and create evidence through the very action of their practice. the local context of the practitioner is the key, and research cannot just be simply handed over for practitioners to implement. practitioners can use such research to inform their decisions but need to consider other components. the concepts found in practice theory, focusing on the practitioner and their knowing in practice – both local evidence and professional knowledge – help to provide a more complete picture of decision making within our profession. the importance that participants placed on learning about what other libraries do, and the high emphasis on gaining input from colleagues, show that practitioners are working within communities of practice for enhancement of their own knowledge and for reinforcement before moving ahead with new ideas. a community of practice may exist within the workplace, where local context is very important, or at a broader level amongst colleagues at other institutions. this broader community is built through conference attendance, as well as committee work on issues of shared interest, and references from colleagues. future research it would be beneficial for lis researchers or researcher-practitioners to explore and recommend the best evidence sources based on the type of question. this would not be a hierarchical list, but would serve as a guideline on what sources of evidence librarians should consider consulting for a given type of question. for example, for a collections problem, the research literature should be consulted, but other sources of evidence that would provide good information include usage statistics for e-products, circulation statistics, faculty priorities, tools such as oclc collection analysis, interlibrary loan and link resolver reports, and the publication patterns of faculty. researchers could determine the most relevant sources for each area of practice, and in what circumstances they are best used. it would also be very beneficial for practitioners if researchers would develop guidance on how to read the results of different evidence sources. this could include what practitioners need to consider when looking at reference statistics, or what elements librarians should consider when conducting an evaluation of their teaching. some of this information will be found in existing literature, and a scoping review of what has already been documented would be a good start. limitations this study is not intended to be generalized to all academic librarians. the purposeful sample allowed for depth and richness of information, and saturation in the data was reached, but not all academic librarians would necessarily fit within these findings. in addition, other academic library systems outside of canada may operate differently. academic librarians are generally regarded as academics or faculty in canada, and at many institutions they can obtain tenure. these factors may create a very different work environment and professional outlook from those working in other library sectors. doing similar research on other librarian groups would strengthen the key findings and applicability of this study. the data collection methods included diary keeping by the participants for a period of one month. the very act of having to keep the diary was something that was not a normal part of their practice, and thus may have impacted their behaviour. for example, they may have felt pressure to do more and be more methodical in their decision-making processes than normal. it is unlikely that false reporting occurred, however, since the follow-up interviews with participants allowed for indepth probing of the actual decision-making evidence based library and information practice 2012, 7.4 21 process, confirming what was previously reported. conclusion this paper has detailed research findings regarding types of information that academic librarians consider to be evidence, and the evidence sources that they use in practice. it answers the research questions, “what forms of evidence do academic librarians use when making professional decisions? why do they use these types of evidence?” two broad types of evidence were identified (hard and soft), which are generally used in conjunction with one another in order to ensure that all possible evidence sources applicable to the problem at hand are considered. neither type of evidence is sufficient on its own. librarians look at all evidence sources (hard and soft) with a critical eye, and try to determine a complete picture before reaching a conclusion. information about how librarians find evidence emerged from the data, showing that both proactive and passive approaches are used. this paper also answers the research question, “how do academic librarians incorporate research into their professional decision making?” it is clear that academic librarians do value research and do look for it to assist with their decision making. however, the published research is insufficient on its own. it may not be directly applicable, and the specifics of the question or problem which librarians are trying to solve take them to sources beyond the research literature. librarians value research literature, but do not use it in isolation. it is only one part of the overall evidence that a librarian needs to consider. both hard and soft types of evidence instill confidence but from different perspectives, and taken together have the most strength. these results provide a strong message that no evidence source is perfect. as a result, librarians bring different types of evidence together in order to be as informed as possible before making a decision. using a combination of evidence sources, depending upon the problem, is the way that academic librarians approach decision making. these results suggest that current practice does not fit with the most commonly used definition of eblip or the eblip model as noted in the literature. a change within eblip does not require a full rejection of the name, but rather a realization that more types of evidence can be included within the concept of evidence, and that doing so brings the eblip model closer to one that has truly considered the needs of librarians. acknowledgement this paper is the first from a doctoral study. future papers will look at how evidence sources are used in decision making, obstacles and enablers to evidence based decision making, and a fuller consideration of possible changes to the eblip model itself. references agor, w. h. (1989). intuition in organizations: leading and managing productively. newbury park, ca: sage publications. banks, m. (2008). friendly skepticism about evidence based library and information practice. evidence based library and information practice, 3(3), 86-90. booth, a. (2000, july). librarian heal thyself: evidence based librarianship, useful, practical, desirable? 8th international congress on medical librarianship, london, uk. booth, a. (2010). upon reflection: five mirrors of evidence based practice. health information and libraries journal, 27(3), 253-256. brophy, p. (2009). narrative-based practice. surrey, uk: ashgate. charmaz, k. (2006). constructing grounded theory: a practical guide through qualitative analysis. london: sage. clark, c. (2011). evidence-based practice and professional wisdom. in l. bondi, d. evidence based library and information practice 2012, 7.4 22 carr, c. clark, & c. clegg. (eds.). towards professional wisdom: practical deliberation in the people professions. (pp. 45-62). surrey, uk: ashgate. corbin, j., & strauss, a. (2008). basics of qualitative research: techniques and procedures for developing grounded theory (3rd ed.). thousand oaks, ca: sage. crumley, e., & koufogiannakis, d. (2002). developing evidence based librarianship: practical steps for implementation. health information and libraries journal, 19(4), 61-70. cybenko, g., & brewington, b. (1999). the foundations of information push and pull. in g. cybenko, d. p. o’leary, & j. rissanen (eds.). the mathematics of information coding, extraction and distribution. (pp. 9-30). new york: springer-verlag. dane, e., & pratt, m. g. (2007). exploring intuition and its role in managerial decision making. academy of management review, 32(1), 33-54. davies, h., nutley, s., & walter, i. (2008). why “knowledge transfer” is misconceived for applied social research. journal of health services research, 13(3), 188-190. duguid, p. (2005): “the art of knowing”: social and tacit dimensions of knowledge and the limits of the community of practice. the information society: an international journal, 21(2), 109-118. eldredge, j. (2002). evidence-based librarianship levels of evidence. hypothesis, 16(3), 10-13. eldredge, j. d. (2000a). evidence-based librarianship: an overview. bulletin of the medical library association, 88(4), 289-302. eldredge, j. d. (2000b). evidence-based librarianship: searching for the needed ebl evidence. medical reference services quarterly, 19(3), 1-18. evidence. (2010). in oxford dictionaries pro. retrieved 6 dec. 2012 from http://english.oxforddictionaries.com/ definition/evidence. foster, a., & ford, n. (2003). serendipity and information seeking: an empirical study. journal of documentation, 59(3), 321-340. fox, n. j. (2003). practice-based evidence: towards collaborative and transgressive research. sociology, 37(1), 81-102. gabbay, j., & le may, a. (2011). practice-based evidence for healthcare: clinical mindlines. new york: routledge. given, l. (2006). qualitative research in evidence-based practice: a valuable partnership. library hi tech, 24(3), 376386. guyatt, g. h. (1991). evidence-based medicine. acp journal club, 114(mar.-apr.), a16. howick, j., chalmers, i., glasziou, p., greenhalgh, t., heneghan, c., liberati, a., moschetti, i., phillips, b. & thornton, h. (2011). the 2011 oxford cebm levels of evidence (introductory document). retrieved 7 dec. 2012 from http://www.cebm.net/mod_product/de sign/files/cebm-levels-of-evidenceintroduction-2.1.pdf hunsucker, r. l. (2007). the theory and practice of evidence-based information work – one world? paper presented at eblip4: transforming the profession: 4th international conference, evidencebased library & information practice, university of north carolina-chapel hill, durham, nc, usa. retrieved 3 mar. 2012 from http://www.eblip4.unc.edu/papers/hu nsucker.pdf evidence based library and information practice 2012, 7.4 23 jonas, w. b. (ed.). (2005). mosby’s dictionary of complementary and alternative medicine. new york: elsevier koufogiannakis, d. (2010). the appropriateness of hierarchies. evidence based library and information practice, 5(3), 1-3. lave, j., & wenger, e. (1991). situated learning: legitimate peripheral participation. cambridge: cambridge university press. mckibbon, k. a., wilczynski, n., hayward, r. s., walker-dilks, c. j., & haynes, r. b. (1995). the medical literature as a resource for health care practice. journal of the american society for information science, 46(10), 737-742. polanyi, m. (1966). the tacit dimension. garden city, ny: doubleday & company, inc. rolfe, g., jasper, m., & freshwater, d. (2011). critical reflection in practice: generating knowledge for care (2nd ed.). london: palgrave macmillan. ryle, g. (1945). knowing how and knowing that: the presidential address. proceedings of the aristotelian society, 46, 1-16. schatzki, t. (1996). social practices: a wittgensteinian approach to human activity and the social. cambridge: cambridge university press. schön, d. a. (1983). the reflective practitioner: how professionals think in action. u.s.a: basic books. suny downstate medical center. (2004). the evidence pyramid. retrieved 20 jan. 2012 from http://library.downstate.edu/ebm2/21 00.htm usher, r., & bryant, i. (1989). adult education as theory, practice and research: the captive triangle. london: routledge. appendix a blog diary instructions for participants over the course of the month that you keep this diary, please write about any incidents where questions arise relating to your professional practice as a librarian. questions/problems could vary widely. please make note of each question, your thoughts about it and how you might approach solving the question. explain any action you took to answer the question, and what, if anything, you did about it. some questions may be answered immediately, while others may take days or weeks, or not be answered during the diary-keeping period at all. that is ok. just detail as much of the process you used in your decision making as possible. how did you come to make the decision you did? at any point in the process, please feel free to reflect on the decisions you made and whether they seem to be working. remember, there are no right or wrong responses. as a researcher, i am looking to understand the process that academic librarians go through in reaching decisions, and what types of evidence may be part of that decision. examples of professional questions/problems a librarian may be working on: o today i am deciding which print journals we might be able to safely weed from the collection. o i’ve been asked to determine the most appropriate hours of operation for the fall term. evidence based library and information practice 2012, 7.4 24 o we are going to be doing renovations to the building this year and i’m on a team looking into what changes would be best. o i am planning a one-hour information literacy session for first year biology students and am trying to determine the best method of delivering the information. o i was wondering how scopus journal coverage compares to that of biosis. do we need both? key elements to include in your diary blog entry: o the professional question/problem arising in practice. o things you did in working through the question/problem. what types of evidence did you use, if any? who or what did you turn to in this process to help you? o any roadblocks you encountered in your problem-solving process, and what you did as a result. o the end result/outcome if a conclusion was reached; or, steps you plan to take to reach a conclusion. o reflection on your decision making process. how do you feel about what you did; what would you change? were your sources of evidence sufficient? please write in your diary as professional practice questions occur. if no entries are received within the period of one week, you will be prompted with a reminder by the researcher. you may contact the researcher at any point, to either ask questions, or drop out of the study if you wish. there is no obligation on your part to participate, all participation is voluntary, and there are no repercussions for dropping out of the study. the blog you are using is private, so only you and i can access or read the content. denise koufogiannakis dkoufogi@gmail.com; dak@ualberta.ca 780-432-3427 (home) skype: dkoufogi appendix b interviewer’s guide guiding questions (to be adapted to each situation and allowed to flow from the context of what the participant feels is important to discuss): o thinking back on the diary keeping period, were their any specific incidents that stood out for you? o would you say that the things you recorded in your diary were fairly typical of a normal month for you? why/why not? o in your diary entry, one of the things you discussed was [x]. can you tell me more about your thoughts on this and what it means to you as an academic librarian? o you mentioned a question that arose in your practice (name the specific question/incident). can you tell me about this in a bit more detail? o what were some of the barriers or difficulties you encountered during the diary-keeping period? o what types of things do you consider to be ‘evidence’? can you tell me how important or not research is to you as a practitioner? how do you use research? what do you consider to be good research, and how do you use it? evidence based library and information practice commentary   reflections on using patrons’ stories as practice-based evidence   suzanne lewis library manager central coast local health district gosford, new south wales, australia email: suzanne.lewis@health.nsw.gov.au   received: 30 jan. 2016   accepted: 9 feb. 2016      2016 lewis. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     we all know the scenario: a patron approaches the librarian in the shelves, or emails the library manager, or telephones the reference desk with a story about how the library service met, exceeded, or failed to meet their expectations. apart from addressing specific problems (“the printing system keeps breaking down” or “i can’t log in to the online resources”), what do librarians do with the rest of the material that comes their way through these informal channels? is it evidence, information, or “anecdata”? in particular, what do librarians do with conflicting opinions? for example, some patrons love downloading e-books to their device and want everything available online, while others bemoan the complexities of e-book borrowing and prefer the reading experience afforded by hardcopy.   qualitative research, usability studies, and user experience explorations are not new in evidence   based library and information practice (eblip). however, the data generated has tended to rank low in the hierarchy of evidence, a hierarchy that has been challenged in recent years (koufogiannakis, 2010). this commentary will explore whether the narratives and stories received through informal communication channels between library staff and patrons are valid forms of evidence, and how (or whether) they can be used to inform decision making and demonstrate the value of library and information services.   the generally accepted definition of eblip is that it integrates the best available research evidence, practitioner experience, and library users’ values and preferences as the basis for decision making (eldredge, 2012). much continues to be written about the quality, quantity, and accessibility of research evidence in the library and information science (lis) field. in the last few years there has been a focus on library practitioners’ knowledge and experience as evidence (koufogiannakis, 2013). at present, there is increasing interest in the third element of eblip—the values and preferences of library users. patron journeys are receiving attention in forums such as the blog #ukanthrolib (http://ukanthrolib.wordpress.com/), which is “a  blog exploring ethnography, usability and user experience in libraries”.   in her keynote address to the 7th eblip conference, koufogiannakis (2013) highlighted two areas of research she considered a priority for the eblip community. one of these was “how do we ‘read’ the results of different types of evidence sources?” (p. 15). koufogiannakis described a range of types of evidence referred to in the academic literature. from this review it was clear that the evidence generated through informal communication with patrons—if it were to be considered evidence—is anecdotal, experiential and highly localized (p. 7). it also falls within the broad grouping of “soft” evidence sources, described by koufogiannakis as follows:   this type of evidence focuses on a story, and how details fit into a particular context. soft evidence provides a real-life connection, insights, new ideas, and inspiration. such types of evidence include input from colleagues, tacit knowledge, individual feedback from users, and anecdotal evidence. these types of evidence are more informal and generally not seen as deserving of the label evidence… (p. 8)   in fact, it could be argued that such evidence, particularly anecdotal evidence, is “anecdata”. anecdata has been defined variously as “unfounded perception” (harris-keith, 2014, p.150); “a compilation of correlated stories or other single pieces of information produced to appear like actual scientific data” (urban dictionary); and “data based on individual story telling that is subjective, malleable, and resists collection via formal mechanisms” (turner, owen & thomas, 2013). the term can be used positively and negatively, depending on context.   in fact, context is key. “different types of evidence need to be weighed within the context in which they are found, and only the practitioners dealing with that decision can appropriately assign value and importance within that context” (koufogiannakis, 2013, p. 15). virginia wilson, in her most recent “research in practice” column in this journal, claimed that because the questions for which librarians seek evidence arise from the local setting, context must remain front and centre in weighing the evidence gathered to address those questions (wilson, 2015).   also key to this issue is the nature of the anecdote itself. greenhalgh and hurwitz (1999) remind us that stories have “a finite and longitudinal time sequence” with a beginning, middle and end (p. 48). the stories that library patrons tell are often fragments of a larger story: the librarian often does not hear the end, and may only discover the beginning in retrospect. greenhalgh, an academic and practicing doctor, writes about the stories that patients tell their doctors. she observes that a patient’s problem is usually converted by the clinician from narrative into “the structured and standardized format that has come to be known as the ‘medical history’” (p. 50). but sometimes what is most important to the patient can be lost in translation. of course, the outcome for the patient is important, but so is the experience, and the same can be said with regard to library patrons.   gidman (2013) points to the work of a.w. frank, who “presents a typology for illness stories: restitution narratives refer to an illness which is treated and resolved; quest narratives report patients’ journeys through their illness (overcoming a range of obstacles)[;] and chaos narratives describe stories with multiple and complex issues which are not resolved” (p. 193). this is a really useful way of thinking about the stories told by library patrons and communicated via informal channels.   many are restitution narratives in which the patron encounters a problem which is successfully resolved with the help of the librarian. in this kind of narrative, the outcome is the most important part of the process for the patron and the librarian. these, of course, are the most satisfying narratives for librarians to hear, as they reinforce the belief that the collections and services provided by the library are useful and relevant.   many are quest narratives, such as the following, related to the author in her workplace (a hospital library) by a nurse who had been writing her master’s thesis and using endnote:   do you remember that time when i rang the library in tears because my word document  formatting on my thesis had gone all funny and made the document unreadable, even the backup file seemed corrupted. i was devastated and i spoke to you and you knew exactly what the problem was and sent me the link to a video which showed me how to fix it?... and i can’t tell you how relieved i was because i was just walking around the house absolutely shattered, fearing i had lost all that work. (j. burrows, personal communication, 17 november 2014)   librarians often hear these types of anecdotes in which they become involved in a larger narrative (in this case, the story of undertaking a master’s degree). the librarian is often able to assist the patron in overcoming obstacles so they can move on to the next part of the story, in which the library may not be involved at all. in this narrative, the experience is just as important as the outcome.   then there are the chaos narratives, such as the student who experiences multiple problems in their interaction with an education provider. in these kinds of situations, the library is only a small part of a much larger narrative and may or may not be able to assist with some issues. the library influences the experience but not the overall outcome for the patron, because the outcome is beyond the sphere of influence of the library.   when assessing the value of anecdotes as evidence, it is useful to bear in mind this typology of stories. the library may be front and centre in the narrative, able to influence both experience and outcome for the patron, or it may be more peripheral, unable to change the outcome but potentially able to improve the experience. the librarian may be involved in the narrative from beginning to end, or in one small part only. patrons’ stories are highly contextual, which is both a limitation and a strength, because they are so specific and meaningful to the individual.   the value of the information derived from stories or anecdotes may also lie in the effort required to obtain it. it should not be used in isolation as the basis for major changes in resources or services, but it can be used to inform further investigation, providing insight into patrons’ values, preferences and experiences.   references   anecdata (n.d.). in urban dictionary, retrieved 24 january 2016 from http://www.urbandictionary.com/define.php?term=anecdata   eldredge, j. d. (2012). the evolution of evidence based library and information practice, part i: defining eblip. evidence based library and information practice, 7(4), 139-145. http://dx.doi.org/10.18438/b8gc99     gidman, j. (2013). listening to stories: valuing knowledge from patient experience. nurse education in practice, 13(3), 192-196.   greenhalgh, t., & hurwitz, b. (1999). narrative based medicine: why study narrative? bmj, 318(7175), 48-50.   harris-keith, c. (2014). evaluating the staffing of an interlibrary loan unit: an exercise in data-driven decision making and debunking “anecdata”. journal of access services, 11(3), 150-158.   koufogiannakis, d. (2010). the appropriateness of hierarchies. evidence based library and information practice, 5(3), 1-3. http://dx.doi.org/10.18438/b88d1r   koufogiannakis, d. (2013). eblip7 keynote: what we talk about when we talk about evidence. evidence based library and information practice, 8(4), 6-17. http://dx.doi.org/10.18438/b8659r     turner, j., owen, c. & thomas, l. (2013) living the indie life: mapping creative teams in a 48 hour game jam and playing with data. in ie '13 proceedings of the 9th australasian conference on interactive entertainment: matters of life and death, acm digital library, melbourne, vic. abstract retrieved from http://eprints.qut.edu.au/64496/   #ukanthrolib. (2014). ukanthrolib: a blog exploring ethnography, usability and user experience in libraries. retrieved from http://ukanthrolib.wordpress.com/   wilson, v. (2015). evidence, local context, and the hierarchy. evidence based library and information practice, 10(4), 268-269. http://dx.doi.org/10.18438/b8k595       evidence summary   millennial generation students search the web erratically, with minimal evaluation of information quality   a review of: taylor, a. (2012). a study of the information search behaviour of the millennial generation. information research, 17(1), paper 508. retrieved from http://informationr.net/ir/17-1/paper508.html   reviewed by: dominique daniel information literacy and reference librarian oakland university rochester, michigan, united states of america email: daniel@oakland.edu   received: 26 nov. 2012 accepted: 10 feb. 2013      2013 daniel. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to identify how millennial generation students proceed through the information search process and select resources on the web; to determine whether students evaluate the quality of web resources and how they use general information websites.   design – longitudinal study.   setting – university in the united states.   subjects – 80 undergraduate students of the millennial generation enrolled in a business course.   methods – the students were required to complete a research report with a bibliography in five weeks. they also had to turn in interim assignments during that period (including an abstract, an outline, and rough draft). their search behaviour was monitored using a modified yahoo search engine that allowed subjects to search, and then to fill out surveys integrated directly below their search results. the students were asked to indicate the relevance of the resources they found on the open web, to identify the criteria they used to evaluate relevance, and to specify the stage they were at in the search process. they could choose from five stages defined by the author, based on wilson (1999): initiation, exploration, differentiation, extracting, and verifying. data were collected using anonymous user ids and included urls for sources selected along with subject answers until completion of all assignments. the students provided 758 distinct web page evaluations.   main results – students did not progress in orderly fashion through the search process, but rather proceeded erratically. a substantial number reported being in fewer than four of the five search stages. only a small percentage ever declared being in the final stage of verifying previously gathered information, and during preparation of the final report a majority still declared being in the extracting stage. in fact, participants selected documents (extracting stage) throughout the process. in addition, students were not much concerned with the quality, validity, or authority of their sources, reporting that the main criteria they used to evaluate a web resource were its understandability, the amount of information in the source, its accuracy, and its recency. during the last stage of the assignment the main criteria were understandability and the amount of information. finally, students used general information websites like wikipedia throughout the process, but especially while preparing the final report.   conclusion – the search behaviour of millennial students does not conform to existing search models. the models are appropriate but the execution of these models by students is problematic. students gathered documents, including general websites like wikipedia, through all stages of the assignment, including the preparation of the final report. they are likely to procrastinate and do some backfilling. furthermore they show little concern for the validity of sources: very few verified their sources and quality of the information gathered was not a priority for them. those findings point to a problem of perception rather than a lack of information search skills: millennial students know how to search and filter, but they do not believe that there is an objective standard to evaluate information and they have a non-critical view of information. more research about the causes of such perception should help us identify effective strategies to help students improve their searches.     commentary   this study builds on existing information search process models which were developed before the internet became a popular information source. there is relatively little research assessing how such models may have changed since (knight & spink, 2008). the study also adds to growing research about the information seeking habits of the millennial generation, which has shown that millennials have superficial search habits, a fragmented view of information, and a conception of information as product rather than process. a third research area this study explores is that of relevance criteria used by searchers to select sources that meet their information need. the study brings these aspects together into an investigation of students’ information search behaviour over time and concludes that in all three areas the behaviour and perceptions of millennials differ from those of previous generations. while not groundbreaking, it provides some insight into ways millennials themselves experience and conceptualize searching.   the study’s greatest contribution springs from its methodology: it used a self-reporting online tool to monitor students completing a real assignment in their own space and at their own pace, away from the researchers’ presence. yet subjects were not in a completely natural setting: they used a modified search engine that included data collection instruments. the author used yahoo, which may not be students’ usual search engine. although subjects were not required to use it, the vast majority stayed with the default because it was easier to enter the data. furthermore the data collection method is susceptible to all the limitations of self-reporting. reporting was a complex process that involved numerous criteria to choose from as well as specialized terminology, two issues that may have negatively affected student reporting. the author does not mention if the validity of the instrument was tested by external observation. it is regrettable that the instrument is not included in the report in its entirety, as it has no precedents (cristal checklist, n. d.).   other research has confirmed the study’s results about students’ tendency to procrastinate and engage in backfilling (head & eisenberg, 2009). particularly striking is the students’ definition of the search process primarily in terms of extraction of information, which points to a confused and limited conception of searching. however, the study’s conclusion that pre-internet search models are appropriate in the digital age is not warranted. although the author uses these models for his research instrument, he does not demonstrate how they are relevant for the study. more research is needed on the impact of unmediated web searching on search behaviors, including other models like berrypicking (bates, 1989; knight & spink, 2008).   for the author, the key factor accounting for the students’ search behaviour is millennials’ relativistic perception of information, but he gives no evidence for this and does not explore the reasons for the students’ apparent neglect of evaluation. research comparing perceived and actual search behaviours could yield more useful results. for example, project information literacy found that students did not think of resource evaluation as a distinct step in the search process but in effect did evaluate sources (head & eisenberg, 2010, p.18). it has also found that students’ search behaviours are not just determined by a subjective conception of information but by uncertainty and stress about the entire research process, which leads students to rely on predictable and convenient but limited routines (head & eisenberg, 2009, 2010). consequently, this study did not lead to innovative conclusions but like other similar studies could have practical implications for librarians who seek to help students construct more effective searches, and to help faculty design more successful assignments.     references   bates, m. j. (1989). the design of browsing and berrypicking techniques for the online search interface. online information review, 13(5) 407-424. doi: 10.1108/eb024320   cristal checklist on appraising a user study. in netting the evidence wiki. retrieved 11 feb. 2013 from http://nettingtheevidence.pbwiki.com/f/use.doc     head, a. j., & eisenberg, m. b. (2009). lessons learned: how college students seek information in the digital age. project information literacy progress report. retrieved 11 feb. 2013 from http://projectinfolit.org/pdfs/pil_fall2009_finalv_yr1_12_2009v2.pdf   head, a. j., & eisenberg, m. b. (2010). truth be told: how college students evaluate and use information in the digital age. project information literacy progress report. retrieved 11 feb. 2013 from http://projectinfolit.org/pdfs/pil_fall2010_survey_fullreport1.pdf   knight, s. a., & spink, a. h. (2008). toward a web search information behavior model. in a. spink & m. zimmer (eds.), web search: multidisciplinary perspectives. (pp. 209-234). berlin: springer.   wilson, t.d. (1999). models in information behaviour research. journal of documentation, 55(3) 249-270. doi: 10.1108/eum0000000007145 evidence summary   individuals with chronic conditions want more guidance from health professionals in finding quality online health sources   a review of: lee, k., hoti, k., hughes, j. d., & emmerton, l. (2014). dr google and the consumer: a qualitative study exploring the navigational needs and online health information-seeking behaviors of consumers with chronic health conditions. journal of medical internet research, 16(12), e262. http://dx.doi.org/10.2196/jmir.3706   reviewed by: cari merkley associate professor mount royal university library calgary, alberta, canada email: cmerkley@mtroyal.ca   received: 1 mar. 2016    accepted: 8 apr. 2016      2016 merkley. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to explore how and when individuals with chronic health conditions seek out health information online, and the challenges they encounter when doing so.   design – qualitative study employing thematic analysis.   setting – urban western australia.   subjects – 17 men and women between 19 and 85 years of age with at least 1 chronic health condition.   methods – participants were recruited in late 2013 at nine local pharmacies, through local radio, media channels, and a university's social media channels. participants were adult english speakers who had looked for information on their chronic health condition(s) using the internet. semi-structured face-to-face interviews were conducted with each participant, audio recorded, and transcribed. the transcripts were coded in qsr nvivo using two different processes – an initial data-driven inductive approach to coding, followed by a theory driven analysis of the data.   main results – three major themes emerged: trust, patient activation, and relevance. many of the participants expressed trust both in health professionals and in the efficacy of search engines like google. however, there was uncertainty about the quality of some of the health information sources found. searching for information online was seen by some participants as a way to feel more empowered about their condition(s) and treatment, but they reported frustration in finding information that was relevant to their specific condition(s) given the volume of information available. low health literacy emerged in participant interviews as an intrinsic barrier to effective online searches for health information, along with low patient motivation and lack of time. the many extrinsic barriers identified included difficulty determining the quality of information found, the accessibility of the information (e.g., journal paywalls), and poor relationships with health care providers.   conclusion – individuals look for online health information to help manage their chronic illnesses, but their ability to do so is influenced by their levels of health literacy and other external barriers to effective online navigation. consumers may prefer to receive recommendations from health professionals for high quality health websites rather than training in how to navigate and identify these resources themselves.   commentary   this study meets many of tracy’s (2010) eight "big tent" criteria of good qualitative research. according to tracy, “high quality qualitative methodological research is marked by (a) worthy topic, (b) rich rigor, (c) sincerity, (d) credibility, (e) resonance, (f) significant contribution, (g) ethics, and (h) meaningful coherence” (p. 839). firstly, the study addresses a topic of clear importance. according to the world health organization (2014), chronic diseases caused 68% of all deaths in 2012, an estimated 38 million people. few previous studies on health information seeking behaviour have focused on the particular needs of those grappling with chronic illnesses, and as such this study fills an important need (greyson, 2015; lee, hoti, hughers, & emmerton, 2014). the researchers demonstrate the qualities of "rich rigor," "meaningful coherence," and “sincerity” by providing a very detailed description of the data collection, interview questions, and process of data analysis, and by clearly identifying the reasons behind the choices made in the study's design (tracy, 2010). the credibility of the findings is further enhanced by the decision to triangulate the findings by subjecting the data to two different forms of analysis involving multiple researchers.   the article would benefit from more detail, such as more information about the backgrounds of participants. rather than providing the usual demographic table, only brief details about participants are provided. participants' highest levels of education and exactly how many fell into each age range are not made clear. age and education have been linked with levels of health literacy (kutner, greenberg, jin, & paulsen, 2006), and as such these details are relevant. it would have been helpful to have the researchers’ definition of chronic illness, particularly with regard to the recruitment process. providing the wording used to help individuals self-identify would also have been helpful, and would not have compromised the privacy of participants. particulars of the australian health care system in which participants were operating would place their comments about the accessibility of health care professionals into context. finally, like many other studies on health information seeking behaviour, no reference is made to existing theories around information seeking throughout the study (devon, 2015).   it is notable that while participants expressed interest in more guidance from health professionals in finding quality health sources, information professionals and librarians were not mentioned in the study as potential partners in this work. the authors are trained as pharmacists, and this professional lens and background may have led to the limited appearance of references from information science journals investigating similar concerns. in their follow up quantitative study with a similar population (lee, hoti, hughes, & emmerton, 2015), the researchers highlighted the possibility of health professionals taking on a larger role as a trusted curator of online health resources or search advisor. information professionals may also be able to play this role to some extent, but this research suggests we should also work to address some of the extrinsic barriers consumers encountered in how health information resources were designed and made available, rather than focusing too narrowly on training users to navigate around roadblocks. the findings of this 2014 study may be of interest to information professionals providing consumer health information or training health professionals in information literacy skills, highlighting areas for future collaboration.   references   greyson, d. (2015, june). everyday life health information seeking: mapping the research literature. presentation at the annual meeting of canadian health libraries association, vancouver, bc, canada. retrieved from http://conference.chla-absc.ca/sites/conference.chla-absc.ca/slides/sat_1330_grand_greyson_1aconsumerhealth.pdf   kutner, m., greenberg, e., jin, y., and paulsen, c. (2006). the health literacy of america’s adults: results from the 2003 national assessment of adult literacy (nces 2006–483). u.s. department of education. washington, dc: national center for education statistics. retrieved from http://nces.ed.gov/pubs2006/2006483.pdf   lee, k., hoti, k., hughes, j. d, & emmerton, l. m. (2015). consumer use of “dr google”: a survey on health information-seeking behaviors and navigational needs. journal of medical internet research, 17(12): e288. http://dx.doi.org/10.2196/jmir.4345   tracy, s. j. (2010). qualitative quality: eight ''big-tent'' criteria for excellent qualitative research. qualitative inquiry, 16(10), 837-851. http://dx.doi.org/10.1177/1077800410383121   world health organization. (2014). global status report on noncommunicable diseases 2014. geneva, switzerland: who press. retrieved from http://apps.who.int/iris/bitstream/10665/148114/1/9789241564854_eng.pdf?ua=1   article   a citation analysis of the classical philology literature: implications for collection development   gregory a. crawford director, penn state harrisburg library middletown, pennsylvania, united states of america email: gac2@psu.edu   received: 07 dec. 2012   accepted: 15 feb. 2013      2013 crawford. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – this study examined the literature of classical (greek and latin) philology, as represented by the journal transactions of the american philological association (tapa), to determine changes over time for the types of materials cited, the languages used, the age of items cited, and the specificity of the citations. the overall goal was to provide data which could then be used by librarians in collection development decisions.   methods – all citations included in the 1986 and 2006 volumes of the transactions of the american philological association were examined and the type of material, the language, the age, and the specificity were noted. the results of analyses of these citations were then compared to the results of a study of two earlier volumes of tapa to determine changes over time.   results – the analyses showed that the proportion of citations to monographs continued to grow over the period of the study and accounted for almost 70% of total citations in 2006. the use of foreign language materials changed dramatically over the time of the study, declining from slightly more than half the total citations to less than a quarter. the level of specificity of citations also changed with more citations to whole books and to book chapters, rather than to specific pages, becoming more prevalent over time. finally, the age of citations remained remarkably stable at approximately 25 years old.   conclusion – for librarians who manage collections focused on greek and latin literature and language, the results can give guidance for collection development and maintenance. of special concern is the continuing purchase of monographs to support research in classical philology, but the retention of materials is also important due to the age and languages of materials used by scholars in this discipline.     introduction                 citation analysis has been a mainstay in the literature of librarianship and information science. a search for the term “citation analysis” in library literature & information science index produced by h. w. wilson results in a list of over 1,600 articles for the years 1981 to 2012. when combined with the search term “collection development,” however, the results shrink to less than 60 articles. outside the field of librarianship and information science, citation analysis is used in a variety of fields, especially to determine leading journals, influential articles, and major authors. a search of psycinfo via apa psycnet yielded over 240 articles containing the phrase “citation analysis” for the period 1927 until 2012. even the mla (modern language association) international bibliography database includes several articles that contain the phrase “citation analysis.” in contrast, a search of the l’année philologique on the internet database covering 1924 to 2011 (the latest update) retrieves no articles specifically on citation analysis within the field of classical studies. l’année philologique is the primary database for the literature of the field of classical studies and currently indexes approximately 1,500 journals.   this research seeks to rectify this lack of research by examining citation patterns in classical studies, specifically classical philology, through an analysis of articles in the transactions of the american philological association, usually referred to as tapa. classical philology has a broad definition which covers most of the fields that are included in the domain of classics or classical studies including literature, languages, history, philosophy, art, religion, and material culture. of specific concern, however, is the study of literary and philosophical texts produced by the ancient greeks and romans.   literature review   the literature on citation analysis and its variants such as co-citation analysis has a long and storied history. according to broadus (1977), librarians have long used citation analysis for collection building and management. similarly, bowman (1991) argued that citation patterns could be used as one method for deciding the suitability of specific items for inclusion in a library’s collection. of special interest to bowman were the formats cited (for example, books and journals), languages of items cited, and the age of items cited. many researchers have studied specific fields to determine how citation analysis can be applied to collection development. for example, zhang (2007) examined the field of international relations, determining that monographs made up almost half the cited materials while journals contributed almost 40%. the vast majority of items cited were written in english, with foreign languages accounting for less than 4% of the total citations. liu (2007) applied citation analysis to the field of educational psychology. he specifically studied the clustering of specialties in educational psychology and stated that studies such as his “can inform librarians doing selection of journals … to meet their specific needs” (p. 117). in an earlier article, hitchcock (1990) examined the use of research materials in a single historical journal. she wrote:   citation analysis is a valid method of measuring the use of materials since it takes advantage of the author’s attempt to substantiate the findings of the research based on documented evidence. as a collection development tool, it benefits from the citation’s function of providing sources of further information on a subject. it is a reliable method as the data are readily available in print and not subject to response variables as are questionnaires. (p. 53)   hitchcock concluded her article, “libraries can best serve researchers of state and local history by becoming aware of the researchers’ use of primary sources, and providing services which will satisfy their information needs” (p. 54). this is also true of researchers in all academic fields.   budd and christensen (2003) examined the social sciences to see how expanding access to electronic information resources had changed citation patterns. they found that within the eight journals from the social sciences that they included in their analysis, few electronic resources were cited. in particular, they found that almost 47% of the citations were to journals and another 44% were to books. they wrote, “one inference that might be drawn from this indicator is that, for the time being at least, the academic world adheres to formal and traditional media for communication” (p. 645).   several authors have examined the field of classical studies, often in combination with other fields of the humanities. in a trio of articles, kellsey and knievel (2004), knievel and kellsey (2005), and kellsey and knievel (2012) studied citation patterns in various humanities fields, including classics. in their first article (kellsey & knievel, 2004), the primary goal was to determine the use of foreign languages by examining the citations in representative journals for a span of 50 years. in total, they counted 16,138 citations from 468 articles in 4 journals from history, classics, linguistics, and philosophy. for classics, they used the american journal of philology as their source. the results for this journal indicated that the use of foreign language materials had declined from 1962 to 2002. in 1962, over 45% of all citations were to foreign language materials while in 2002 slightly over 21% were to foreign language items. in their 2005 article, the authors analyzed 9,131 citations from the 2002 volumes of journals in eight humanities fields, including classics. again, citations from the american journal of philology were used as the source of data for the field of classics. this study broadened the scope of the analysis to include formats of materials studied as well as language. that particular volume of the american journal of philology yielded 996 individual citations with an average of 39.8 citations per article. over 76% of the citations were to monographs while slightly over 33% were to journals. almost 80% of the citations were to english language materials. finally, in their 2012 article, they examined citations from 28 monographs published by humanities faculty members with the goal of determining how these scholars accessed the materials they used. specifically, they queried whether the sources were owned by the faculty member’s academic library, how they were acquired (approval or firm order), their average age, and interdisciplinary usage as determined by the lc classification of the cited item.   for the field of classics, especially classical philology, two pieces of research stand out, tucker (1959) and dabrishus (2005). both of these master’s papers were written at the university of chapel hill. tucker’s goal was “to ascertain certain of the characteristics of the literature used by researchers in the field of classical philology” (p. 1) by studying the literature cited in the transactions of the american philological association. among the characteristics he examined were the form of publication, the age of the literature cited, the specificity of the citation, and the use of foreign language publications. tucker’s analysis included a total of 1,327 citations drawn from 33 articles in two volumes of tapa, volume 87 (1956) and volume 88 (1957). he only counted those citations to secondary sources, not the primary sources that were often the focus of the article itself. for example, the original text of euripides was not included in the citation analysis, but works about the text were included. specific results from tucker’s research are discussed below in the results section. dabrishus studied the citations included in three classics journals: the classical quarterly, classical antiquity, and mnemosyne. although she focused primarily on the use of periodicals, she did note that monographs were cited heavily, accounting for 76% of all citations, while periodicals received only 24% of the citations in her analysis. the three most frequently cited journals were the classical quarterly, bulletin de correspondence hellénique, and transactions of the american philological association. in total, the articles in the three journals included in the study cited 120 different journals of which over half were cited more than one time each.   aims   the overall goal of this research was to study changes over time in the way scholars have used the literature of classical (greek and latin) philology. based on previous research, the journal transactions of the american philological association (tapa) was used for the analysis and this study attempted to determine changes over time for the types of materials cited (e.g., monographs and journals), the languages of the cited materials, the age of the items cited, and the specificity of the citations. the results of the analyses provide data which may be used by librarians in making collection development decisions, especially the allocation of resources for monographs and journals in classical studies, the discarding of materials which are no longer relevant, and the placement of materials in storage.   methods   in order to understand how citation trends in classics have changed over time, the current research sought to replicate and update the research performed by tucker (1959). all citations to secondary sources from articles published in tapa for the years 1986 and 2006 were compiled into a spreadsheet. the use of these two years of tapa helped determine if there had been significant changes over time in the citation patterns for this specific journal, especially when compared to the original research which analyzed citation data drawn from the 1956 and 1957 volumes of tapa. in addition, using citations from the 2006 volume provided a way to determine the extent to which scholars in this field cite identifiable electronic resources (other than journals which, while electronically available, are usually cited as if they were used in a print version).   every citation included in each article appearing in the 1986 and 2006 volumes of tapa was examined and only those from secondary sources, that is, not the original texts being discussed in the article itself, were included in the analysis. citations to original greek and latin texts were, therefore, not included in the analysis. the data for each citation included the name(s) of the author(s), the title of the publication, the type or format of publication, the date of publication, the language of the publication, and the specificity of the citation. in addition, the age of the citation was determined by subtracting the date of the publication from the year in which the source article appeared in tapa. as determined by tucker, the type of publication included the following formats: book/monograph, journal/periodical, annual/yearbook, encyclopedia/dictionary, festschriften, dissertation/thesis, and other. the current research added electronic sources for the 1986 and 2006 articles. languages of citations included english, german, french, italian, latin, greek, and spanish. following the work of tucker, specificity focused on the length of the citation, i.e., 1 page, 2-10 pages, over 10 pages, an entire article (of a journal, annual, etc.), an entire book, a book chapter, and other. all citations to secondary materials were entered into the analysis, including ibid. and op. cit. citations.   results   in his research, tucker (1959) did not separate his results by year. thus, in the following tables and discussion, his results are given as he presented them, consolidating both years of his study into one set of data. in the two volume years of tapa that he included in his study, tucker examined 1,327 citations drawn from 33 articles, an average of 40.21 citations per article. as shown in table 1, the current research examined 34 articles and 3,323 citations. in 1986, there were 20 articles that included 1,421 citations, an average of 71.05 citations per article. by 2006, the number of articles had declined to 14, but the total number of citations had ballooned to 1,902, an average of 135.86 citations per article. thus, there is a statistically significant increase in the average number of citations per article between the 1956/57 and the 1986/2006 data (t=4.542, p<.001). in fact, there is also a statistically significant difference between the 1986 and the 2006 average number of citations per article (t=-2.598, p=.014). these results show that the number of items cited by authors of articles had grown considerably between 1956 and 2006. more recent authors cited more than 3 times as many sources as authors during the 1950s.   table 2 provides a breakdown in the types of publication cited in the examined articles. a chi-square test results in a statistically significant result (chi square=358.63, p<.001, df=12) indicating that the types of publications cited had changed significantly over the time span. in 1956/57, books and monographs accounted for 54.8% of the citation. this percentage grew to 67.2% in 1986 and 68.6% in 2006. citations to journals and periodicals remained fairly stationary (1956/57: 28.5%; 1986: 26.9%; and 2006: 29.7%). major changes are observed for the remaining types, except for other. citations to annuals and yearbooks fell from 9.6% of citations in 1956/57 to only 0.2% in 2006. similarly, citations to encyclopedias and dictionaries fell from 3.6% in 1956/57 to 0.3% in 2006. citations to festschriften and dissertations likewise fell dramatically over the timespan of the study. there is only one citation to an electronic resource in 2006, although by then the internet and world wide web had been publicly available for well over a decade. this may be misleading, however, since many journals in the field of classical studies, especially philology, had been available electronically for many years prior to 2006. the authors may have used electronic sources, but not cited them as such.     table 1 citations per tapa volume tapa volume & year number of articles number of citations average citations per article   87 (1956) & 88 (1957) 33 1327 40.21   116 (1986) 20 1421 71.05   136 (2006) 14 1902 135.86   total 67 4650 69.4   table 2 format of materials cited ** “other” includes newspapers, conference proceedings, and government documents a total of 935 different books or monographs were cited by the 1986 and 2006 articles. in 1986 there were 387 different books cited, while in 2006 there were 562. several books were cited in both years. tucker, unfortunately, did not list the total number of different books cited but only included the total number of citations to books.   of particular interest are the journals and periodicals which were cited within these articles. table 3 gives a breakdown of the ten titles which received the greatest number of citations for each year included in the study. the chart is arranged alphabetically with the number of citations given to that specific journal during each of the study years given in the columns. as can be seen, of the eighteen journals listed, only four were in the top ten for all three years: american journal of philology, classical philology, hermes, and transactions of the american philological association. four others were in the top ten for two years: arethusa, classical journal, classical quarterly, and journal of hellenic studies. of these, arethusa did not begin publication until 1968, well after tucker’s study. of the top 10 journals cited in the 1956 and 1957 volumes, only one (byzantinische zeitschrift) was not cited by any of the articles in the later volumes of tapa. tucker does not provide a listing of all the journals cited during the years of his study, but for 1986 and 2006 a total of 119 different journals received citations. in 1986, there were 93 different journals cited and in 2006, 101 different journals were cited. thus, it is evident that scholars in the field of classical philology cast a wide net when utilizing the research literature.   the language of the sources of citations also changed significantly over time (chi-square=601.40, p<.001, df=14). table 4 shows that english was, by far, the most frequently cited language for all years, accounting for 67.5% of all citations included in the study. in contrast, for the years 1956 and 1957 english accounted for less than half of the citations while german received 31.1% and french 11.3%. by 1986, german and french witnessed dramatic declines with german accounting for 23.9% and french for 6.3% while english grew to 67.2%. for the 2006 articles, english grew even more, accounting for 83.2% of the citations. german and french continued to decline (8.4% and 3.3% respectively) and italian increased slightly in comparison to the 1986 citations (3.0% in 2006 compared to 2.5% in 1986), yet did not approach the 7.7% in 1956 and 1957.   the level of specificity of citations also changed significantly over time (chi-square=168.13, p<.001, df=12). as shown in table 5, citations to a single page remained fairly steady over time, while citations to 2-10 pages declined as a percentage of the total citations. the major changes were in the number of citations to entire books and to book chapters, both of which grew greatly over the period.   finally, the study examined the age of the citations. table 6 gives the age breakdown of citations using the time spans originally established by tucker in his research. the age of the citation was determined by simply subtracting the publication year of a citation from the volume year of tapa. for example, if an item being cited by an article in volume 136 (2006) was published in 1997, the age of the citation was recorded as 9 years old. in contrast to the other changes noted above, the average age of citations remained very stable over time. for the 1956-1957 citations, the average age was 25.23 years. for the 1986 citations, the average age was 24.53 years and for 2006, the average age was 24.63 years.   discussion   table 3 the ten most cited journals by tapa volume journal vol. 87 (1956) & 88 (1957) vol. 116 (1986) vol. 136 (2006) american journal of archaeology   24   american journal of philology 35 16 23 arethusa   13 22 byzantinische zeitschrift 9     classical antiquity     17 classical journal 31 12   classical philology 12 11 69 classical quarterly 12 29   classical review 11     greece & rome     19 harvard studies in classical philology   24   hermes 17 36 18 journal of hellenic studies   13 35 philologus 12     phoenix     24 proceedings of the cambridge philological society     17 rheinisches museum fur philologie 11     transactions of the american philological association  63 19 38     table 4 language of citations   tapa volume & year language total english german french italian latin greek spanish other 87 (1956) & 88 (1957) 604 413 150 102 44 9 1 4   1327 116 (1986)   955   340   89   35   1   1   0   0   1421 136 (2006)   1583   160   63   58   18   15   5   0   1902   total   3142   913   302   195   63   25   6   4   4650     table 5 specificity of citations   this study of citations to the literature of classical philology support several observations. authors of articles included in tapa rely heavily not only on the literature of the field of classical philology, but also on the literature of related fields such as history, philosophy, and archaeology, as shown in the growth in the variety of items cited. according to the articles included in the study, the average number of citations more than tripled from 1956 and 1957 until 2006, indicating a greater reliance on previously published literature. it is difficult to speculate on all the reasons for this increase in the number of citations, especially considering the similarity in the length of the articles over the years, although the training of scholars in the humanities emphasizes the necessity of building upon the work of previous scholars. the availability of materials, especially journals through a variety of electronic databases and the ease of interlibrary loan, may have made resources more readily available to scholars, thus increasing the amount of material used in more recent research. the types of materials cited, while similar, did show statistically significant changes. specifically, the citations to books increased dramatically, from 54.8% of the total citations in 1956 and 1957 to 68.6% in 2006, although this is still less than the 76% reported by dabrishus (2005). such growth in the number of citations to monographs is a surprising finding when one considers the growth in the use of journals shown by most scholarly fields. the finding does underscore the monographic nature of the field of classical studies and the continuing importance of books to scholars within the humanities.   within tapa articles, citations to journals remained fairly steady at slightly less than 30%. the use of other materials, such as encyclopedias and festschriften, all declined, although their numbers represented a relatively small percentage in the types of materials cited during all years. one surprising result was the lack of specific citations to electronic resources, especially considering the tremendous growth of websites, e-only journals, online encyclopedias, etc. only one purely electronic resource was identified in this study, although many of the journals and monographs could have been accessed through electronic databases.     table 6 age of citations age of citations age in years vol. 87 (1956) & 88 (1957) vol. 116 (1986) vol. 136 (2006) total 0-5 283   292   164 739 6-10 181   203   352 736 11-15 96   161   345 602 16-20 144   233   256 633 21-25 85   126   162 373 26-30 95   63   139 297 31-35 77   43   106 226 36-40 46   30   110 186 41-45 39   14   73 126 46-50 55   21   50 126 51-75 138   124   73 335 76-100 39   88   45 172 100+ 47   20   27 94 no date 2   3   0 5 total 1327 1421 1902 4650     the analyses also show that a wide variety of books and journals were cited. thus, for developing a collection to support research in classical philology, books remain an important mainstay for scholarly work. such books include not only commentaries on specific classical authors, but also works on art, archaeology, literature, and philosophy. the array of journals consulted is also very broad, although there is a fairly small core of journals which received heavier use. thus, librarians have evidence that providing the core set of journals will provide a large proportion of the materials actually cited by classical scholars in their research. this result can help determine how to spend the scarce resources available for collection development.   the utilization of foreign language materials has greater implications for collection development and maintenance. in 1956 and 1957, english language materials accounted for only about half of the citations. by 2006, english materials represented 83.2% of all citations. during this time frame, the use of foreign language materials declined precipitously. for example, german language materials declined from 31.1% of all citations to 8.4% and french declined from 11.3% to 3.3%. these changes, however, may be deceiving, since many materials, especially books, may have been translated into english from the other languages in more recent years. still, these changes do show that scholars in the field of classical philology rely heavily on materials in english. these results mirror those found by kellsey and knievel (2004) and knieval and kellsey (2005), although the american journal of philology cited a higher proportion of foreign language materials than did tapa. as a result of such evidence, for many libraries collection development in the field of classical philology should focus primarily on english-language materials, although the evidence also reiterates the need for access to a wide variety of materials in other languages which may be provided through interlibrary loan or databases of foreign-language journals.   the specificity of citations has also changed over time. the main change is the dramatic increase in the number of citations to whole books and to chapters in books. this mirrors the results for the types of materials cited and shows an increased usage of monographs, indicating that the demand for scholarly monographs in classics continues to be high. tucker (1959) says, “the longer, more exhaustive treatment which a book can afford a topic could be a considerable factor in the most frequent choice of this form” (p. 16).   the most striking result of the present study is the consistency in the average age of citations within this field of approximately 25 years old. as tucker (1959) notes, “the researcher in this field perhaps does not feel so constrained to consult the most current literature” (p. 14). the field of classics in its broadest sense has a long history, stretching back centuries, and obsolescence of scholarly ideas is low. as can be seen from the age analysis, even materials from the nineteenth and early twentieth centuries are still cited by scholars. such citation patterns have major implication for retention policies. if scholars are regularly using such older materials, these books and journals need to be available. this could call into question the weeding of older books and journals or their placement in remote storage facilities. digitization of these older materials could also help solve the problems of storage.   conclusions   the study of citation patterns can provide the scholar and the librarian with varied insights into selected fields. this study sought to replicate and expand an earlier study and provides useful information on how scholars in the field of classical philology use sources in their research. the results, of course, are limited to only one scholarly field and cannot be generalized to other subjects. similar methodology, however, has been used frequently in the study of other fields. the results from this study and others can help librarians in their pursuit of providing materials needed by scholars for their research. of special concern is the retention of materials. in the case of classical philology, scholars make use of materials from a wide time span written in a variety of languages, although more recent research has relied increasingly on english language materials. in addition, they are heavy users of monographs, yet they still use a wide array of journal titles. thus, such materials need to be retained in research library collections. unlike other fields, especially in the sciences, which rely more heavily on current journals, classics continues to rely on both monographs and journals and ideas expressed in older materials can still have immense relevance to current research. as a result, librarians cannot make blanket decisions for retaining materials, such as format or age. they must consider the nature of the use of materials by subject discipline.     references   bowman, m. (1991). format citation patterns and their implications for collection development in research libraries. collection building, 11(1), 2-8.   broadus, r. n. (1977). the application of citation analyses to library collection building. advances in librarianship, 7, 299-335.   budd, j., & christensen, c. (2003). social sciences literature and electronic information. portal: libraries and the academy, 3(4), 643-651. doi: 10.1353/pla.2003.0077   dabrishus, m. (2005). the forgotten scholar: classical studies and periodical use (unpublished master’s thesis). university of north carolina at chapel hill, chapel hill, nc. hitchcock, e. r. (1990). materials used in the research of state history: a citation analysis of the 1986 tennessee historical quarterly. collection building, 10(1/2), 52-54.   kellsey, c., & knievel, j. e. (2004). global english in the humanities? a longitudinal citation study of foreign-language use by humanities scholars. college & research libraries, 65(3), 194-204.   kellsey, c., & knievel j. (2012). overlap between humanities faculty citation and library monograph collections, 2004-2009. college & research libraries, 73(6), 569-583.   knievel, j. e., & kellsey, c. (2005). citation analysis for collection development: a comparative study of eight humanities fields. the library quarterly, 75(2), 142-168. doi:10.1086/431331   liu, z. (2007). scholarly communication in educational psychology: a journal citation analysis. collection building, 26(4), 112-118.   tucker, b. r. (1959). characteristics of the literature cited by authors of the transactions of the american philological association, 1956 and 1957. (unpublished master’s thesis). university of north carolina at chapel hill, chapel hill, nc. doi:oclc/23993605   zhang, l. (2007). citation analysis for collection development: a study of international relations journal literature. library collections, acquisitions, & technical services, 31(3/4), 195-207. doi.org/10.1016/j.lcats.2007.11.001         evidence summary   transgender and gender-nonconforming populations experience unique challenges in health information environment developed for heteronormative audience   a review of: tenny, c. s., surkan, k. j., gerido, l. h., & betts-green, d. (2021). a crisis of erasure: transgender and gender-nonconforming populations navigating breast cancer health information. the international journal of information, diversity, & inclusion, 5(4), 132–149. https://doi.org/10.33137/ijidi.v5i4.37406    reviewed by: lisa shen business librarian & director of public services newton gresham library sam houston state university huntsville, texas, united states of america  email: lshen@shsu.edu   received: 9 dec. 2022                                                                accepted:  19 jan. 2023      2023 shen. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30289     abstract   objective – to understand the lived experiences of transgender and gender-nonconforming populations in seeking health information about breast cancer.   design – thematic literature review.   setting – four english-language databases featuring clinical, patient engagement, and library and information sciences (lis) research.   subjects – twenty-one published articles.   methods – the researchers chose three concepts (trans, lgbtq+, and breast cancer), identified related terms for each, and used these terms to conduct literature searches in four databases: pubmed, web of science, library literature & information science full text, and library, information, science & technology abstracts. search results were reviewed for relevance to the research objective. the researchers applied grounded theory to analyze the 21 selected articles through open, axial, and selective (thematic) coding. the qualitative research software nvivo was used to perform thematic analysis of each article, and a shared codebook was developed to ensure saturation of axial themes and consistency of coding amongst researchers.   main results – three overarching themes emerged from selective coding that exemplify experiences of transgender and gender-nonconforming persons seeking health information about breast cancer: access, erasure, and quality. compared to their cisgender peers, these historically marginalized populations and their caregivers experience more difficulty accessing the already limited breast cancer information, healthcare, and support services suited to their needs. in particular, transgender and gender-nonconforming patients are often burdened with choosing between receiving health information and care designed for heteronormative persons and risking self-disclosure and possible discrimination by culturally incompetent health professionals.   conclusion – the researchers noted the alarmingly limited resources available for gender-nonconforming patients seeking information and support for health matters other than mental health or sexually transmitted diseases. the researchers also called for increased efforts by lis curriculums and professionals to study and understand the needs of transgender and gender-nonconforming patrons, and to improve the quality and quantity of information resources specifically dedicated to these unique populations.   commentary   as humanity recovers from the worldwide covid-19 pandemic, this study provides timely advocacy for the creation of, and access to, quality health information for historically underserved populations. the article contributes to existing lis research by bringing attention to the lack of inclusive information sources for transgender and gender-nonconforming audiences about health topics not explicitly related to their sexual identities.   an examination of this study using the evidence based librarianship (ebl) critical appraisal checklist (glynn, 2006) yielded an overall validity below the accepted threshold of 75%. the majority of the validity issues were associated with the population and data collection sections of the ebl checklist. several ebl appraisal elements for those sections were not applicable to narrative literature reviews, and sectional validity ratings were therefore skewed by a limited number of lower-rated elements. however, the study design and results did achieve sectional validities above the threshold. the researchers provided sound rationale for choosing grounded theory to investigate their research question and included citations of all the articles they analyzed. norming practices employed during each coding stage were also intentional and appropriate for the chosen methodology.   nevertheless, database search protocols and article selection parameters for the study were frustratingly opaque, and such lack of replicable details raise significant concerns regarding validity of the findings. a simple keyword search in any of the four databases used in this study would return hundreds of articles indexed with the same search terms identified by the authors, yet only 21 were selected for the thematic analysis. it is unclear how the researchers constructed each literature search, how they determined whether a resulting article is “most relevant” (tenny et al., 2021, p. 134) to the research question, and how many articles were found then eliminated in the review process. timing of the searches and publication date range were also not addressed. moreover, each of the four chosen databases offers some unique search functions, such as the major concept limiters in pubmed, and it is unclear whether these functions were considered or used. consequently, those interested in replicating the study or applying similar methodology to a different issue or population would find it difficult, if not impossible, to reliably do so.   because of these validity concerns, readers are advised to consider the representativeness of findings with reservation. nonetheless, this article highlights a pertinent health information resource gap experienced by historically marginalized populations. not only do the study findings provide practical implications for lis professionals interested in improving equitable, inclusive access to health information for all patrons, but they may also inspire future researchers to investigate similar resource gaps in inclusive health information availability and access experienced by other underserved populations.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387–399. https://doi.org/10.1108/07378830610692154    tenny, c. s., surkan, k. j., gerido, l. h., & betts-green, d. (2021). a crisis of erasure: transgender and gender-nonconforming populations navigating breast cancer health information. the international journal of information, diversity, & inclusion, 5(4), 132–149.  https://doi.org/10.33137/ijidi.v5i4.37406    evidence summary   the urgency and importance of an active information seeking task influence the interruption of information encountering episodes   a review of: makri, s., & buckley, l. (2020). down the rabbit hole: investigating disruption of the information encountering process. journal of the association for information science and technology, 71(2), 127–142. https://doi.org/10.1002/asi.24233   reviewed by: barbara m. wildemuth professor emeritus, school of information & library science university of north carolina at chapel hill chapel hill, north carolina, united states of america email: wildemuth@unc.edu   received: 28 may 2020                                                             accepted:  17 sept. 2020      2020 wildemuth. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29785     abstract   objective – to understand when and why information encountering episodes are interrupted.   design – naturalistic observational and interview study.   setting – personal network of the study authors in london.   subjects – fifteen personal contacts of authors, aged 22-60, recruited via word-of-mouth and social media.   methods – each participant was asked to conduct a search on a self-chosen topic. the researchers took notes and recorded search interactions and think-aloud protocols. after the search, a follow-up interview asked whether the participant had unexpectedly encountered any interesting or useful information; if so, the researchers asked for more details about that episode. if not, they conducted a critical incident technique interview, focused on a memorable example of a past information-encountering episode. the researchers used inductive thematic analysis to analyze the data, augmented with constant comparison across the data and the themes to ensure analytical rigor.   main results – the most frequent point at which participants interrupted an information encountering (ie) episode was near its beginning, when the searcher noted an information stimulus but then immediately returned to the active information seeking task. ie episodes were also interrupted 1) after the searcher examined the encountered content but did not explore it further, and 2) after the searcher explored it but decided it was not useful.   the factors that influenced interruptions of ie episodes included the searcher’s reluctance to invest the time and effort needed to engage with the encountered information, due to the importance or urgency of the active information seeking task; the searcher’s reluctance to leave the active information seeking task, seeing ie as a distraction from that task; the searcher’s reluctance to multitask, i.e., to keep track of both the ie episode and the active information seeking task; the searcher’s reluctance to risk a dead end; the searcher’s reluctance to be seduced by the “shiny thing” of encountered information (p. 136) and to drift too far away from the active information seeking task; and the searcher’s reluctance to get “caught up” emotionally in the ie episode (p. 138), a “temptation that is satisfying only in the short-term” (p. 138).   conclusion – overall, the results help us understand when and why disruption of ie can occur. when an ie episode begins, the searcher is not able to estimate the time and effort required to pursue it or the fruitfulness of following it through. thus, factors associated with the primary information seeking task (e.g., its importance or urgency) and with the searcher (e.g., ability to multitask) tend to influence decisions about when to interrupt an ie episode.   commentary   information encountering is defined by the study authors as “a type of serendipitous information acquisition that involves passively finding unexpected information that was not purposively sought and is considered subjectively interesting, useful, or potentially useful” (p. 127, emphasis in original). an ie episode consists of several stages, including noticing an informational stimulus, stopping the active information seeking task, acquiring and examining the encountered content, exploring or following up on that content, capturing or using that content, and possibly resuming the active information seeking task (p. 128; based on erdelez, 2004; jiang et al., 2015). the authors of this study faced a challenge in clarifying their focus: we are more accustomed to thinking about information encountering as an interruption of an active information seeking episode, but this study focused on interruptions to an ie episode. readers will need to keep this focus in mind as they consider the findings from this study and the implications of those findings.   while the authors did an excellent job of defining the scope of ie and its interruptions, studies of searchers’ stopping behaviors during active information seeking (e.g., wu & kelly, 2014; maxwell et al., 2015; dedema & liu, 2019) are also pertinent. for example, maxwell et al. (2015) found that searchers most often stopped their review of results after seeing too many non-relevant results, i.e., when they’d reached the point of frustration. thus, the searcher’s affective response seems to play a role in interruptions to an active information seeking task as well as interruptions to an ie episode. both bodies of literature will be useful to practitioners as they train new searchers, design search systems, or provide assistance during searches.   the methods used to conduct the study are generally strong and appropriate for the study’s purposes (critical skills appraisal programme, 2018). because ie episodes cannot be assigned in a lab study, the overall design was necessarily naturalistic. because the occurrence of ie episodes is spontaneous, the sample of participants was necessarily a convenience sample and only five of the participants actually experienced an ie episode during the observation. the authors filled this gap in the data by conducting critical incident interviews with the other ten searchers. while this is a reasonable approach, other approaches were possible: using data only from those who experienced ie and expanding the sample until theoretical saturation was reached, or conducting critical incident interviews with all participants to compare and validate the quality of the data collected with each method. at a minimum, it would have been useful if the text of the paper had noted which data points came from observations versus interviews. rigorous data analysis, including detailed thematic analysis augmented with constant comparison of the themes, balanced this slight weakness in the data collection method. in addition, the description of the results provides enough examples from the data for the reader to be confident in the validity of the findings.   the findings of this study have implications primarily for search system design. one underlying question not asked is whether search system designers should encourage or discourage ie. since ie often yields references that are interesting or useful, one could argue that systems should support more and better ie. on the other hand, ie tends to distract the searcher from an ongoing active information seeking task; depending on the priority of that task, it may be more appropriate for systems to discourage ie and encourage a focus on the active information seeking task. a third possibility is that the system could detect circumstances in which it should encourage ie and those in which it should discourage ie, and respond appropriately. it would have been interesting to hear the authors discuss this question and the future research studies needed to clarify tradeoffs and the role of uncertainty before and during an ie episode.   the authors of the study do suggest some ways in which search engines can better support searchers’ decisions about whether to continue an ie episode or return to their active information seeking task. these include ways that systems could support the pursuit of ie (e.g., through a user-controlled “serendipity filter” or history-based highlighting) or the delay of ie to be taken up at a later time (e.g., by temporarily “parking” encountered information or sending notifications to review parked content during the searcher’s “dead time”). these recommendations seem fruitful for further development in general search engines, digital libraries, and online catalogs, but most of them will require that we first gain a better understanding of the role of ie in relation to a searcher’s active information seeking activities.    references   critical skills appraisal programme. (2018). casp qualitative checklist. retrieved from https://casp-uk.net/wp-content/uploads/2018/01/casp-qualitative-checklist-2018.pdf   dedema & liu, c. (2019). examination of online information search stopping behaviors and stopping rules by task type. proceedings of the annual meeting of the american society for information science & technology, 56. https://doi.org/10.1002/pra2.114   erdelez, s. (2004). investigation of information encountering in the controlled research environment. information processing & management, 40(6), 1013–1025. https://doi.org/10.1016/j.ipm.2004.02.002   jiang, t., liu, f., & chi, y. (2015). online information encountering: modeling the process and influencing factors. journal of documentation, 71(6), 1135–1157. https://doi.org/10.1108/jd-07-2014-0100   maxwell, d., azzopardi, l., järvelin, k., & keskustalo, h. (2015). searching and stopping: an analysis of stopping rules and strategies. proceedings of the 24th acm international conference on information and knowledge management (cikm '15), 315–322. https://doi.org/10.1145/2806416.2806476   wu, w., & kelly, d. (2014). online search stopping behaviors: an investigation of query abandonment and task stopping. proceedings of the annual meeting of the american society for information science & technology, 51(1), 1–10. https://doi.org/10.1002/meet.2014.14505101030     evidence based library and information practice 2012, 7.2 20 evidence based library and information practice article digital images in teaching and learning at york university: are the libraries meeting the needs of faculty members in fine arts? mary kandiuk visual arts, design and theatre librarian york university libraries toronto, ontario, canada email: mkandiuk@yorku.ca aaron lupton electronic resources librarian york university libraries toronto, ontario, canada email: aalupton@yorku.ca received: 21 dec. 2011 accepted: 27 apr. 2012 2012 kandiuk and lupton. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/bync-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – this study assessed the needs for digital image delivery to faculty members in fine arts at york university in order to ensure that future decisions regarding the provision of digital images offered through commercial vendors and licensed by the libraries meet the needs of teaching faculty. methods – the study was comprised of four parts. a web survey was distributed to 62 full-time faculty members in the faculty of fine arts in february of 2011. a total of 25 responses were received. follow-up interviews were conducted with nine faculty members. usage statistics were examined for licensed library image databases. a request was posted on the electronic mail lists of the art libraries society of north america (arlis-l) and the art libraries society of north america canada chapter (carlis-l) in april 2011 requesting feedback regarding the use of licensed image databases. there were 25 responses received. results – licensed image databases receive low use and pose pedagogical and technological challenges for the majority of the faculty members in fine arts that we surveyed. relevant content is the overriding priority, followed by expediency and mailto:mkandiuk@yorku.ca mailto:aalupton@yorku.ca evidence based library and information practice 2012, 7.2 21 convenience, which take precedence over copyright and cleared permissions, resulting in a heavy reliance on google images search. conclusions – the needs of faculty members in fine arts who use digital images in their teaching at york university are not being met. the greatest shortcomings of licensed image databases provided by the libraries are the content and technical challenges, which impede the ability of faculty to fully exploit them. issues that need to be resolved include the lack of contemporary and canadian content, training and support, and organizational responsibility for the provision of digital images and support for the use of digital images. introduction the increasing growth of digital images offered through commercial vendors and licensed by libraries has provided new opportunities for teaching and learning at universities. offering a significant number of high-resolution digital images with educational use permissions, licensed image databases are intended to facilitate the use of digital images in pedagogy and research. given the significant financial expenditures on image databases such as artstor by canadian academic libraries, it is critical to know whether the needs of library users are being met through these electronic resources. informal feedback from faculty members in fine arts at york university suggested that subscription image databases are not being used and pose pedagogical and technological challenges. this included messages from faculty members frustrated when trying to use licensed image databases (artstor in particular); poorly attended artstor training sessions on campus; the inclusion of web sources for images on course readings lists as opposed to licensed image databases; and requests from faculty members that york university participate in fadis (a shared common repository and content management system designed for the teaching, studying, and researching of art, architecture, and visual culture). in an effort to ensure that future decisions regarding the provision of digital images by the libraries meet the needs of teaching faculty, the authors conducted a fourpart study in 2011 to assess the needs for digital image delivery to faculty members in fine arts at york university. as recently as 10 years ago faculty members in fine arts at york university relied on a slide library, established in 1971 and housed in the department of visual arts, for images to support their teaching. the libraries meanwhile were responsible for monograph and periodical collections. a variety of factors contributed to the demise of the slide library – a deteriorating slide collection which included damage sustained during renovations, decreased staff support for its operation precipitated by budget cuts, as well as the advent of digital images via the web and licensed image databases provided by the libraries. the original plan to digitize the slide library collection – which at its pinnacle contained over 250,000 slides, including substantial canadian and contemporary content as well as unique material relating to prominent york art teachers – was never realized. york university libraries were an early canadian adopter of the artstor digital library, which was first licensed in 2005. this was followed in the same year by subscriptions to corbis images for education (no longer available for licensing) and camio, oclc’s catalog of art museum images online. at the time these image databases appeared to be a promising campus-wide solution that would meet the needs of teaching faculty both in and outside the faculty of fine arts for digital images with secured permissions for non-commercial, educational, and scholarly use as the slide library was quietly laid to rest. what gradually became evident was that despite initial enthusiasm, faculty members in fine arts were unable to exploit fully, if at all, these resources that the evidence based library and information practice 2012, 7.2 22 libraries had invested in so heavily financially, and the costs for which were increasingly difficult to rationalize. the challenge therefore was twofold – how to better support faculty members in their use of digital images in teaching and how to better exploit resources provided by the libraries that were not being used. literature review there were several major studies published from 2001 to 2006 examining the use of digital images in teaching and learning at american colleges and universities. these studies, on a much larger scale than ours, were conducted at a time when faculty members were still making the transition from the use of analog images to digital images. the visual image user study at penn state university, conducted over several years starting in 2001, was an extensive needs assessment study that explored the “use of pictures in higher education in order to inform the design of digital image delivery systems” (pisciotta, dooris, frost, & halm, 2005, p. 33). the project included the study of current and expected use of pictures by students and faculty, a survey of the image resources supporting those uses, and a review of current practices related to software and metadata. the summary of the critical factors influencing the willingness to use an image delivery system for teaching included: desired content; user-selected technology for classroom presentation; ability to create presentations with images from many sources; help with understanding permitted uses; methods of selecting, sorting, naming groups, and other personalization of portions of the data; and easy coordination with image-use systems. surveying 33 colleges and universities in the united states in 2006, green’s study, using digital images in teaching and learning: perspectives from liberal arts institutions (2006), focused on the pedagogical implications of the widespread use of the digital format, revealing issues of infrastructure and support that “need to be resolved before their deployment can be effective” (p. 3). it examined image sources, image use, technology and tools, support and training, and institutional infrastructure issues. it was the infrastructure issues that proved to be the biggest challenge of all. schonfeld, in the visual resources environment at liberal arts colleges (2006), examined the role images play in teaching and learning at seven liberal arts colleges in the united states in 2005 and 2006. the report focused on the issues of organizational structure and organizational culture and the role they played in supporting strategies for the provision of digital images. the role of the slide library or visual resources collection proved to be the key variable, and “those campuses at which the slide library takes a campus-wide perspective (rather than serving the art history department alone) seemed to see much easier and more successful transitions to digital images” (p. 1). in 2005, waibel and arcolio, as members of the rlg instructional technology group for oclc, set out to test “assumptions about how digital images are discovered, acquired and used – and about preferences for the future” (p. 1). their primary conclusion was that “image databases need to leverage the breadth and simplicity of online search engines such as google images search to achieve higher use” (p. 3). what is missing from the literature is current research relating to the use of digital images in teaching by fine arts teaching faculty. the main purpose of our study was to determine how digital images are located, stored, and used by fine arts faculty members in their teaching at a large university with a strong fine arts program; to examine the shortfalls of available image databases and barriers inhibiting their use; and to explore potential future models to support the use and availability of digital images and strategies to maximize the potential of existing digital resources. in addition, our study sheds light on the specific needs of fine arts teaching faculty in canada. the canadian research knowledge network (crkn, a national consortium comprised of 44 canadian artstor subscribers) is licensing image databases as part of its large-scale content acquisition and evidence based library and information practice 2012, 7.2 23 licensing initiatives designed to “build knowledge infrastructure and research capacity” at canadian universities (canadian research knowledge network, 2011). this study serves in part to evaluate the effectiveness of those initiatives. methods the information-gathering portion of our study was comprised of four parts. part 1 a web survey was distributed to 62 full-time faculty members in the departments of visual arts, design, fine arts cultural studies, and theatre in the faculty of fine arts in february of 2011. as one of the authors is the liaison librarian for these four departments, there was a particular interest in conducting a needs assessment. each of these departments provides a comprehensive, balanced program of creative work and academic studies, combining scholarly work with practical training. faculty members teach in a variety of settings, which include the lecture hall, classroom, laboratory, and studio. the survey was comprised of 26 questions (see appendix a). the faculty image use survey conducted as part of the penn state visual image user study proved very useful for the formulation of the questions for our survey. respondents were also provided with the opportunity to provide additional comments throughout the survey. part 2 at the end of the survey respondents were asked if they would be willing to be contacted for a follow-up interview. interviews were conducted in person where possible and otherwise by telephone during april 2011 (see appendix b). part 3 usage statistics were examined for licensed library image databases. statistics were compiled for artstor from 2005 to 2011 and for camio from 2007 to 2011. artstor and camio sessions and searches per fte for york university were also compared to the average of institutions within crkn and the provincial consortium ocul (ontario council of university libraries). part 4 as the final part of the information gathering, a request was posted on the electronic mail list of the art libraries society of north america (arlis-l) and the art libraries society of north america canada chapter (carlis-l) in april 2011 as follows: i have just finished conducting a survey of faculty members in fine arts at my institution regarding the use of digital images in teaching. preliminary results indicated that artstor is highly underutilized as a source for digital images and that faculty members are relying heavily on google images. i am interested in hearing whether the experience has been the same at your institutions. i am also particularly interested in hearing from those who have had success in promoting artstor at their institutions and where faculty members are using artstor on a regular basis in their teaching. feedback regarding the use of other licensed image databases in teaching is also welcome. results and discussion part 1 – the survey there was a 40% response rate, with 25 faculty members in total responding to the survey. the 25 responses received from faculty members were distributed across the following departments: visual arts (11 respondents – 44%), theatre (7 respondents – 28%), design (4 respondents – 16%), fine arts cultural studies (3 respondents – 12%). evidence based library and information practice 2012, 7.2 24 analog images how often are analog images used in teaching? there were 3 faculty members (12%) who reported always using analog images in their teaching; 4 (16%) reported using them frequently; 6 (24%) sometimes; and 12 (48%) not at all. the greatest reason for using analog images was that content suited their needs (11 respondents – 44%) and ease of use (7 respondents – 28%). those least likely to use analog images in their teaching were faculty members in visual arts and theatre. those most likely to use analog images were faculty members in design. why are analog images used? the reasons faculty members gave for using analog images included: preference for working with tangible objects; lack of access to a projection system; difficulty in manipulating digital images; preference for using their own person slide collections; and availability of images only in slide form. digital images how often are digital images used in teaching? the conversion from the use of 35 mm slides to digital images is described by sonia staum (2010), director at iupui herron art library, as “perhaps one of the most significant transitions for our collections in the past decade” (p. 77). faculty members in fine arts at york university appear to have made the transition (although not always successfully) from the use of analog to digital images, with 13 respondents (52%) reporting that they always use digital images in their teaching, 7 (28%) reporting frequent use of digital images in teaching, and 5 (20%) reporting that they sometimes use digital images in their teaching. no one indicated that they never use digital images. what sources are used for digital images? this question was divided into three parts: (1) licensed image databases, (2) creation of own images, and (3) external sources, including photo sharing sites, image collections and portals from other libraries, purchased cd collections, and web search engines. for licensed digital image databases, very low use of artstor was reported with only 1 respondent (4%) using it always and 4 respondents (16%) using it frequently. there were 7 respondents (28%) who reported using artstor sometimes, while another 10 (40%) reported no use of artstor whatsoever. not a single faculty member in design used artstor, which was puzzling given the inclusion of design collections in artstor (e.g., moma architecture and design collection, cooper union for the advancement of science and art graphic design collection). there was negligible use of camio reported by all respondents. as reported by waibel and arcolio in their study in 2005, we discovered what we already suspected, that “by and large, the library plays only a small role in supplying the faculty with digital image content” (p. 2). when asked to elaborate as to why they did not use licensed digital image databases in their teaching, faculty members’ responses included:  “images are scanned from my own book collection.”  “use my own personal images. i am a photographer. or i search google images.”  “use my own research on line and my own work.”  “i didn’t know about camio; i use a lot straight off the internet but not for lectures – for print info.”  “locate images from museum websites and anywhere else i can find them.”  “use ccca open source for contemporary canadian art.” as for sources used for the creation of their own digital images, the most frequent method used by all faculty members was using a digital camera (14 respondents – 56% always evidence based library and information practice 2012, 7.2 25 or frequently use) followed by scanning from books (10 respondents – 40% always or frequently use). faculty members in visual arts were the group most likely to create digital images by digitizing slides (7 respondents – 28% always or frequently). when specifically asked about which web/internet sources were used for digital images, the most often used source was google images search (17 respondents – 68% always or frequently use). faculty also reported frequent use of image collections from other libraries, museums, or archives (15 respondents – 60% always or frequently), followed by public photo sharing sites such as flickr (8 respondents – 32% always or frequently). additional sources most frequently cited include images scanned from a private library, printed materials such as books and magazines, and unique digital documents provided by other artists/educators. the majority of faculty members reported that they were able to combine images to meet their needs if more than one source was used. in fact, only one respondent (in visual arts) reported being unable to combine images for the reason of time constraints and file/software incompatibility. when asked about their favourite sites for digital images, faculty member responses are as follows:  web search engines/tools (e.g., google images, flickr, cooliris, youtube)  virtual museum websites (e.g., centre for contemporary canadian art, national arts centre: the secret life of costumes, web gallery of art)  museum/gallery websites (e.g., national gallery of canada: cybermuse, carnegie museum of art, guggenheim museum, moma, metropolitan museum of art, tate online, the barnes foundation)  library digital image collections (e.g., metropolitan toronto reference library: the canadian theatre record, digitized images from the bodleian libraries special collections, gallica digital library)  university digital image collections (e.g., university of amsterdam flickr collection)  stock photography websites (e.g., getty images, stock.xchng)  auction house websites (e.g., artnet)  personal websites (e.g., typefoundry: documents for the history of types and letterforms) what other licensed image databases should be made available? what was telling about this question is that 8 faculty members (32%) responded that they did not have enough knowledge to suggest any sources, revealing a general unfamiliarity with licensed digital image databases. the following resources grouped by department were suggested by respondents:  design: berg fashion library, aiga design archives  fine arts cultural studies: alinari, art resource  visual arts: vtape, fadis (3 respondents) when asked to indicate what criteria are most important to them, as is illustrated by figure 1, ease of finding the images they needed ranked highest followed by image resolution/quality. as was also revealed by waibel and arcolio in the oclc study, “almost every faculty member interviewed regarded google image search as a quick, reliable way of retrieving images for teaching. while the common deficiencies in terms of file size and color fidelity are apparent to them, ease of use and the search engine’s ability to deliver a suitable image for almost any request outweigh those shortcomings” (p. 2). furthermore, “in their dream of the future, faculty envision access to high-quality, rights-cleared, persistently available images with the same retrieval success rate as google image search” (p. 3). meanwhile, cleared copyright and permissions, a concern at the top of the library’s mind, received more of a mixed response. copyright, as was revealed in the evidence based library and information practice 2012, 7.2 26 interviews, is perceived as a barrier to expediency and convenience. what activities are important for teaching? as revealed in figure 2, creating image sequences for presentation was ranked highest followed by being able to integrate images from several sources, and then the ability to create your own digital images with a scanner/camera. what activities are desirable that are not currently possible? when we asked faculty members what they would like to do that current resources do not make possible, we received a variety of responses, but one common one was the ability to show two images side by side simultaneously. it was pointed out that it was possible to do this with the old slide projection, a point also raised by schonfeld, who commented that “not all digital image teaching tools have made it easy to bring together two images side by side, which has made it difficult for some instructors to mimic traditional art history teaching methods using digital solutions” (p. 7). a related response came from two members of fine arts cultural studies, who indicated that they would like to be able to project an image at the same time as a moving image with sound. other singular responses included: having access to the rare book room to scan images; more flexibility with copyright (specifically, the ability to use images in a course document/handout); access to more video content; access to a larger database of content; access to specifically more contemporary global art content; and more technical assistance with using images. many of these responses would come up in future questions. where or how are images stored? as is illustrated by figure 3, the most common place where images are currently stored is on faculty members’ personal computers, followed by a flash memory storage device. figure 1 how important is each of the criteria to you? evidence based library and information practice 2012, 7.2 27 what content management/courseware systems should be made available? this question revealed a lack of knowledge of other content management systems, with the most common answer being some variation of “i don’t know.” one respondent in design mentioned slideroom and plone, while one in visual arts mentioned fadis (the federated academic digital imaging system currently housed at the university of toronto). frustration was also expressed about the lack of space to mount a slide show and the need for a system highly compatible with moodle. what presentation software is used? powerpoint was the most popular response among theatre and visual arts faculty members, with the majority responding that they use this well-known microsoft product. it was not as popular in design, where most respondents said they use adobe acrobat. artstor presentation software was a distant third. where or how are digital images posted for review? the most common response was that faculty do not post images for student review. a number of faculty members did post images for review on a faculty/institutional server and local courseware systems. however one faculty member, who teaches an online course, indicated that the lack of space provided on the local course management system posed an obstacle to posting images for review. what are the challenges or obstacles faced when using digital images in teaching? as is illustrated by figure 4, a lack of content was identified as the number one obstacle, with “too few good sources” indicated as an issue by 11 respondents (44%). however, respondents also gave answers in the openended “other” section. these included: lack of figure 2 what activities are important for your teaching? evidence based library and information practice 2012, 7.2 28 technical knowledge to work with images; material being obscure, expensive, and difficult to obtain; the time it takes to obtain material; the poor resolution of most images; a lack of contemporary material; not enough digital space to hold images; lack of video; and a lack of finding aids for images. what are the deficiencies/challenges of licensed image databases? again, content proved to be a challenge, with 9 respondents (36%) indicating this was an issue with york’s licensed digital image databases artstor and camio. specifically, they indicated that these resources lack: canadian content (mentioned five times), contemporary content (mentioned twice), typography, indigenous content, and video. regarding the advent of image databases, sonja staum (2010) writes that “while these vast digital image repositories held promise for improved convenience due to their access-on-demand nature, the content in these resources often did not match the curricular needs of the respective target audience and as a result was not useful” (p. 80). many years later this still appears to hold true. the second most popular deficiency of licensed image databases was being unable to manipulate images satisfactorily (3 respondents – 12%). faculty members indicated that they found artstor “too complicated” and “laborious to use.” who provides technical support? “because images can be obtained easily online, it is falsely assumed that there needs to be little supportive infrastructure. nothing could be further from the truth,” states green (2006, p. 99). this was also our finding. most of our respondents (17 respondents – 68%) indicated that they turned to faculty it support for assistance. many (7 respondents – 28%) indicated that they had insufficient technical knowledge to use licensed image databases effectively. several (4 respondents – 16%) indicated that technical support is too overwhelmed to provide proper support for teaching faculty and that they relied on hired consultants, family, a paid technician, or a research assistant. very few (2 respondents – figure 3 where or how do you store your digital images? evidence based library and information practice 2012, 7.2 29 8%) said that they relied on support provided by licensed providers such as artstor. what type of artstor training has been received? again, mirroring our own experience, waibel and arcolio (2005) write: “while we heard about library attempts to make faculty more aware of licensed resources, these communications seem to largely bypass their audience” (p. 2). when asked what type of artstor training faculty members have received, the answer in every category (figure 5) from online handouts to onsite training sessions was consistently “have not used.” this was despite promotion by the libraries of artstor training and support services as well as a full-day artstor training session organized by the libraries and conducted by an artstor trainer on campus in fall 2006. has technical assistance been sought or received from artstor and was it useful? almost no one indicated that they had sought or received technical assistance from artstor. only two faculty members in theatre had sought assistance, with one member finding it very helpful and another indicating it was not, making it difficult to draw any conclusions. however, several faculty members had indicated in the past that the lack of a toll-free number for canadian artstor subscribers was an impediment to obtaining quick assistance (as well as the ability to participate in artstor webinar training sessions). figure 4 what are the challenges/obstacles you face when using digital images in teaching? evidence based library and information practice 2012, 7.2 30 part 2 – the interviews the interviews were used as an opportunity to elicit more information about the responses in the survey, and they revealed information that did not emerge in the survey. we were able to follow up directly with faculty regarding their individual responses. the nine faculty members (36% of respondents) who were interviewed were distributed across the following departments, which provided us with valuable insights into how digital images are used in different disciplines: visual arts (5), design (1), and theatre (3). we interviewed one faculty member in visual arts who uses only slides in his teaching (his own vast personal collection of 190,000 images created with a digital camera), and another in theatre who relies entirely on artstor for digital images in his teaching. neither of these individuals was typical or representative. most other faculty members use the web, pulling together images for their teaching from a variety of sources. the following summarizes what we learned and what issues emerged in the interviews. department of visual arts  artstor: concerns primarily relate to content (especially canadian) as well as technical challenges  web: overriding concerns relate to the quality of images and the patchwork of resources that need to be organized  fadis: it is perceived as a flexible alternative resource, offering more relevant as well as user-generated content, especially canadian (“critical mass of material”); york university’s concerns relating to copyright have impeded participation  visual literacy: students are perceived to be lacking in this skill  federated searching: there is a need for collective software that searches across databases quickly and simply  copyright: this is perceived to be a bigger issue in canada than in the united states (because of carfac, canadian artists’ representation/le front des artistes canadiens); york university is also perceived to be overly concerned about copyright as compared to other canadian institutions; the need to assemble figure 5 what type of artstor training have you received? evidence based library and information practice 2012, 7.2 31 images quickly takes priority over copyright considerations  technical support: there is a desperate need for more technical support; “budget cuts have eviscerated support systems” (which went from 3 full-time staff to 0.5); technical support is needed at short notice  libraries: there is a diversity of views about the role of the libraries, with some indicating that they would not expect the libraries to assist beyond a general level, while others felt that the libraries should take more responsibility and at the very least provide curating of sources (e.g., image portal web page) department of theatre  print materials: small classes allow for the use of books or hard prints relating to costume and set design and less reliance on digital images  web: there is a heavy reliance on the internet for images  artstor: this database is considered technically difficult and lacking in content with poor it/customer support (yet one faculty member indicated that what he wants is readily available in artstor and that it/customer support is responsive); concerns were expressed about the technical problems with updates and the lack of canadian content department of design  print materials: they are better suited to the needs of this one faculty member who scans digital images mostly from his own material  web: concerns were expressed regarding the poor resolution of images  artstor: this database is perceived to be lacking as it is not based on typography or graphic design the idea of one search across all of our library resources was mentioned several times in our interviews. a similar idea was also reported by waibel and arcolio (2005), who write that “the idea of searching across all licensed resources and the web at the same time found many proponents” (p. 3). the biggest issue that emerged in the interviews was the lack of a coordinated strategy for making the transition from analog to digital images, which was reported in numerous studies as the critical ingredient for success. as green (2006) states, “perhaps the biggest challenge of all is that of institutional response: of managing change and of thinking strategically about planning the necessary infrastructure for effective use of digital resources” (p. 15). green also discovered – which has been proven true at york – that “often issues were taken one at a time, without understanding how they were connected” (p. 15). part 3 – artstor and camio usage statistics the statistics revealed extremely low usage for camio but growing usage for artstor. one of the limitations of the data is that we were not able to identify the type of user (what table 1 artstor usage, 2005-2011 artstor 2005 2006 2007 2008 2009 2010 2011 sessions 700 1,608 1,932 1,464 2,268 1,944 843* sessions per fte 0.3 0.9 0.9 0.7 1.1 1 searches 9,584 20,307 24,987 23,640 38,559 33,648 5,756* searches per fte 5 10.7 12.5 11.9 19.4 18 *january-march 2011 data only evidence based library and information practice 2012, 7.2 32 department/program the user was in) and the status of the user (faculty, student, etc.). it was also impossible to tell if each search represented a unique user and whether a single user was conducting multiple searches. updated usage statistics were available at the time of writing and are included in table 1. artstor has, more or less, shown steady growth in usage since its acquisition in 2005. while numbers may appear high, it should be noted that york is below the crkn average in number of times accessed. between 2010-11 and 2011-12 the university of ottawa recorded over 452,000 searches, which is eight times york’s usage over the same period. interestingly, york is far above the crkn average number of searches. so while fewer people are using it at york, they are spending a lot of time using it. camio usage is far lower than that of artstor, and when the number of sessions and searches per fte is factored in, it still can barely be characterized as “regular usage.” camio is licensed through the ontario consortium ocul, with three subscribing institutions, including york. table 3 shows that york is above the ocul average in number of searches and sessions for the period december 2010 to november 2011. part 4 arlis/na and arlis/na canada electronic mail lists feedback the feedback received from art and architecture librarian and visual resources staff colleagues via the lists was very revealing. there were 25 responses received in total from 19 american institutions and 5 canadian institutions. there were 5 respondents (20%) who reported no success with artstor at their institutions (“i’m afraid that our experience is similar to yours” was a common response); 3 (12%) reported success with artstor; 4 (16%) reported limited success with artstor; 6 (24%) reported that they were preparing local collections for inclusion in artstor shared shelf; and 4 (16%) requested our survey and/or the results of our survey. the remaining 3 (12%) responses were not applicable. there appeared to be no discernible difference in the experience of canadian and american institutions, although several american institutions reported heavy use of local digital collections. of the 12% reporting success with artstor, the existence of a dedicated visual resources centre and/or visual resources librarian or curator, an aggressive promotion and instruction strategy, and the inclusion of in-house images through participation in artstor shared shelf seemed to suggest greater success with artstor. table 2 camio usage at ocul and york, december 2010-november 2011 camio 2007 2008 2009 2010 sessions 109* 169 254 326 sessions per fte 0.05 0.08 0.12 0.17 searches 516 553 806 1393 searches per fte 0.25 0.27 0.40 0.74 *only august-december 2007 data available table 3 camio usage at ocul and york, december 2010-november 2011 dec. 2010-nov. 2011 ocul total ocul average york total sessions 631 210 354 searches 2879 960 1490 evidence based library and information practice 2012, 7.2 33 artstor’s shared shelf allows “institutions to manage, actively use, and – should an institution so choose – share their institutional and faculty image collections” (artstor, 2012). one us college respondent indicated: “since we signed an agreement to add our own collections to artstor, we have been able to promote much more, since faculty and students can see our museum’s collections side by side with other collections in artstor.” meanwhile the comments received from those reporting no success using artstor reflected our own experience. concerns expressed related to the lack of contemporary and canadian content as well as the technical challenges associated with the use of artstor. respondents said: “artstor does not have the images faculty need/want and they must go elsewhere to locate needed images,” “contemporary canadian coverage is not great in artstor,” and faculty find “artstor to be unwieldy to use.” in addition artstor required “publicity and start-up training.” to summarize the results, licensed image databases receive low use and pose pedagogical and technological challenges for the majority of the faculty members in fine arts that we surveyed. relevant content is the overriding priority, followed by expediency and convenience, resulting in a heavy reliance on google images search. copyright considerations rank lower in priority and are perceived as a barrier to expediency and convenience. there is also a direct correlation between comfort level with technology and the use of digital images in teaching. licensed image databases are challenging to use and faculty members surveyed have insufficient training and technical support to fully exploit them. feedback received from librarians and visual resources staff at other institutions polled suggests that their experience mirrors our findings. conclusion our study illustrated clearly that the needs of faculty members in fine arts who use digital images in their teaching at york university are not being met. the greatest shortcomings with respect to licensed image databases provided by the libraries relate to content and technical challenges, including technical support, which impede the ability of faculty to fully exploit them. green (2006) states: finally, it might serve us well to recognize the complexity, difficulty and expense of deploying digital images and to regard the transition to using them as a longer, more ongoing process than we have expected up until now: a transition that will need careful managing. as smith college art historian dana leibsohn put it: “this notion of transition is interesting – but it has a really long tail and we have to think harder about it and what it means to be in transition for more like fifteen or twenty years, rather than the five to eight years we’ve been talking about. national initiatives will help; peer exchange will help – but i think we’re not thinking about transition as seriously as we should as an ongoing process.” (p. 100) the supportive infrastructure for the provision and use of images in teaching that existed in the faculty of fine arts was removed with the demise of the slide library, the advent of digital images readily available on the web, and the acquisition by the libraries of licensed image databases. the libraries meanwhile have not historically provided technical support for the use of images; nor do they have the staff resources to provide the kind of assistance required at short notice by faculty members teaching with digital images. with respect to the use of image databases, it was believed that the support for the use of those databases would and could be provided by the licensed digital image providers. this has resulted in faculty members in fine arts being left, in the words of one york art historian, as “one of the biggest art departments in the country with no solution.” there are a number of strategies that will be pursued by the libraries to address the issues and concerns that were identified in our study. the first involves working to resolve issues evidence based library and information practice 2012, 7.2 34 relating to the lack of canadian and contemporary content. the libraries are currently exploring participation in fadis and artstor shared shelf. they are also members of the ocul visual resources working group, which has been established with a mandate to “identify opportunities for collaboration across ontario’s universities that will improve access to visual resources and services” (patrick, 2011). this includes exploring additional opportunities for collaboration with other canadian universities to develop shared content and to lobby artstor for content that would support the needs of canadian users. it should be noted that at the time of writing there are several canadian universities that are considering cancelling their subscriptions to artstor (trent university has already cancelled) or have renewed for only one year in order to provide an opportunity for review (e.g., university of toronto). while we have renewed our artstor subscription for three years, we are reviewing other existing subscriptions with a view to cancelling image databases receiving extremely low use (such as camio) and working with faculty members to identify other potentially more relevant databases. on the basis of the feedback received from our survey the libraries are also exploring the creation of a library digital images web portal that would provide links to image sites. the second initiative is to address issues of training and support at the local level. this requires identifying the specific needs of faculty members with respect to training and support, working with appropriate partners at artstor and instructional technology staff in the faculty of fine arts to address these issues, and potentially expanding the role of the libraries with respect to artstor training and support. the third is to raise awareness of digital collections in artstor in an effort to increase its use, as well as increase awareness and understanding of copyright issues as they relate to the use of digital images, with the aim of promoting the use of artstor and other licensed image databases. the libraries are currently exploring the use of a search and discovery service which would have the potential to search digital images from licensed databases. the issue of copyright meanwhile is a challenging one. as was revealed in the interviews, faculty members, particularly those trained in the united states, perceive canada’s copyright laws to be overly restrictive (fair use vs. fair dealing) and york university’s enforcement of copyright very rigid. the last initiative involves working to resolve issues relating to organizational responsibility regarding the use of digital images in teaching (including the digitization, management, and integration of local/personal image collections and institutional image collections). this will entail working with the faculty of fine arts and other partners on campus to develop a coordinated and integrated approach to the provision of digital images and support for the use of digital images in teaching. acknowledgements an abbreviated version of this paper was presented at the seventh annual trtlibrary staff conference, held in toronto, ontario (may 2011), and as a poster presentation at the 40th annual art libraries society of north america conference, held in toronto, ontario (march 2012). references artstor. (2012). shared shelf. retrieved 14 may 2012 from http://www.artstor.org/shared-shelf/shtml/shared-shelf-home.shtml canadian research knowledge network. (2011). about. in canadian research knowledge network. retrieved 27 april 2012 from http://www.crkn.ca/about green, d. (2006). using digital images in teaching and learning: perspectives from liberal arts institutions. retrieved 3 jan. 2011 from http://www.academiccommons.org/fil es/image-report.pdf evidence based library and information practice 2012, 7.2 35 patrick, j. (2011, april 4). re: ocul visual resources working group. retrieved from ocul-l <ocull@listserv.uofguelph.ca> pisciotta, h., dooris, m. j., frost, j., & halm, m. (2005). penn state’s visual image user study. portal: libraries and the academy, 5(1), 33-58. schonfeld, r. c. (2006). the visual resources environment at liberal arts colleges. retrieved 3 jan. 2011 from http://dspace.nitle.org/bitstream/handl e/10090/6619/2006_4_3_schonfeld.pdf staum, s. (2010). swimming with the tides of technology in an art and design library: from amico to delicious to youtube. in a. gluibizzi & p. glassman (eds.), the handbook of art and design librarianship. (pp. 75-90). london: facet publishing. waibel, g., & arcolio, a. (2005) out of the database, into the classroom. retrieved 3 jan. 2011 from http://www.oclc.org/research/activities /past/rlg/culturalmaterials/outofthedb. htm evidence based library and information practice 2012, 7.2 36 appendix a digital images survey the increasing growth of digital images offered through commercial vendors has provided new opportunities for teaching and learning. given the significant financial expenditures on licensed digital image resources such as artstor by york university libraries it is important for us to know whether the needs of faculty and their students are being met through these electronic databases. in an effort to ensure that future decisions with respect to the provision of digital images by the libraries meet the needs of faculty and their students this survey is being conducted to assess the needs for digital image delivery to faculty in design, fine arts cultural studies, theatre and visual arts. definition of digital image: still picture in electronic file format in any form and of any subject including those derived from analog images such as scanned photographs and slides. it would be appreciated if you could take a few minutes to fill out this survey. if you have any questions please contact mary kandiuk or aaron lupton. thank you. 1. what department do you teach in? o design o fine arts cultural studies o theatre o visual arts 2. what position do you hold? o full-time faculty o other (please specify) 3. which type of setting best describes where you teach? please check all that are applicable. o lecture hall o classroom o laboratory o studio o other (please specify) 4. how often do you use analog images (images that are not in electronic form) in your teaching? o always o frequently o sometimes o never evidence based library and information practice 2012, 7.2 37 5. why do use analog images in your teaching? please check all that are applicable. o content suits my needs o ease of use o not comfortable using digital images o other or “na” if you do not use analog images 6. how often do you use digital images in your teaching? o always o frequently o sometimes o never 7. which of the following sources do you use for your digital images? please check all that are applicable and the frequency with which they are used. licensed digital image resources provided by the libraries: artstor o always o frequently o sometimes o never camio o always o frequently o sometimes o never other o always o frequently o sometimes o never if other, please specify or write "na" if never evidence based library and information practice 2012, 7.2 38 8. which of the following sources do you use for your digital images? please check all that are applicable and the frequency with which they are used. create own digital images using the following: digital camera o always o frequently o sometimes o never scan from books o always o frequently o sometimes o never slide digitization o always o frequently o sometimes o never other o always o frequently o sometimes o never if other, please specify or write "na" if never 9. which of the following sources do you use for your digital images? please check all that are applicable and the frequency with which they are used. locate own digital images using the following: evidence based library and information practice 2012, 7.2 39 public photo sharing sites (e.g. flickr) o always o frequently o sometimes o never image collections from other libraries, museums, or archives o always o frequently o sometimes o never image portals created by other libraries o always o frequently o sometimes o never image search engines (e.g. google image search) o always o frequently o sometimes o never purchase cd collections o always o frequently o sometimes o never other o always o frequently evidence based library and information practice 2012, 7.2 40 o sometimes o never if other, please specify or write "na" if never 10. if more than one source is used, are you able to combine digital images from these sources to meet your needs? o yes o no if no, why not? 11. what are your favourite sites for digital images? 12. are there any other licensed digital image resources you would like the libraries to make available? 13. how important are each of the following criteria to you? image resolution/quality o very o somewhat o not at all metadata (information about the image) o very o somewhat o not at all ease of finding the image you need o very o somewhat o not at all ability to indicate scale or size of the object o very evidence based library and information practice 2012, 7.2 41 o somewhat o not at all ability to retain the rights to an image o very o somewhat o not at all cleared copyright and permissions o very o somewhat o not at all having all content in one place o very o somewhat o not at all other o very o somewhat o not at all if other, please specify or write "na" if not at all 14. how important are each of the following activities for your teaching? presenting several images simultaneously o very o somewhat o not at all zooming in to show progressive detail in an image o very o somewhat evidence based library and information practice 2012, 7.2 42 o not at all altering images (cropping, changing contrast, etc.) o very o somewhat o not at all adding text or other media to accompany an image o very o somewhat o not at all creating image sequences for presentation o very o somewhat o not at all being able to interrupt or change sequences in the middle of a presentation o very o somewhat o not at all posting digital images for student review and study outside the classroom o very o somewhat o not at all being able to integrate images from several sources o very o somewhat o not at all creating your own digital images (scanning/camera) o very o somewhat evidence based library and information practice 2012, 7.2 43 o not at all tasking students to find/create digital images for their own creative work or assignments o very o somewhat o not at all other o very o somewhat o not at all if other, please specify or write "na" if not at all 15. what would you like to be able to do when teaching with digital images that you are currently unable to do? 16. where or how do you store your digital images? please check all that are applicable. o cd/dvd-rom o flash memory storage device o public photo sharing sites (e.g. flickr) o open source software for managing digital images (e.g. mdid) o open source digital repository (e.g. dspace) o proprietary digital repository (e.g. contentdm) o faculty or institutional server o personal computer o artstor o other (please specify) 17. are there any content management/courseware systems for digital images you would like to have available? 18. what is the presentation software for digital images that you use in your teaching? please check all that are applicable. evidence based library and information practice 2012, 7.2 44 o powerpoint o artstor o other (please specify) 19. where or how do you post digital images for student review? please check all that are applicable. o local courseware system o artstor o open source software for managing digital images (e.g. mdid) o open source digital repository (e.g. dspace) o faculty or institutional server o proprietary digital repository (e.g. contentdm) o public photo sharing site (e.g. flickr) o do not post images for student review o other (please specify) 20. what are the challenges or obstacles that you currently face using digital images in your teaching? please check all that are applicable. o too few good sources o suitable system for storing images is not available o suitable system for presenting images is not available o suitable system for posting images for student review is not available o loan, permissions, or copyright considerations o technology is lacking in the setting where i teach o lack of technical support o other (please specify) 21. if you experienced any of the challenges or obstacles listed below when using licensed databases such as artstor or camio, please indicate the name of the database in the corresponding text box. content is lacking please specify how the content is lacking (e.g. lacks canadian content) and which database: poor quality of images please specify which database: evidence based library and information practice 2012, 7.2 45 duplicate images please specify which database: images are insufficiently documented please specify which database: way of searching does not match the way images are organized or identified please specify which database: unable to manipulate images satisfactorily please specify which database: difficult to integrate images from other sources please specify which database: difficult to store images please specify which database: difficult to post/share images please specify which database: insufficient training please specify which database: technology is too complicated please specify which database: lack of technical support please specify which database: other please specify the challenge or obstacle and which database: 22. from whom do you receive technical support? please check all that are applicable. o licensed digital image resource provider (e.g. artstor) o faculty it support o other (please specify) 23. what type of artstor training have you used or participated in? please check all that are applicable and the degree to which it was useful. online handouts provided by artstor o useful o somewhat useful o not useful o have not used online training session provided by artstor o useful o somewhat useful o not useful o have not used evidence based library and information practice 2012, 7.2 46 video demonstrations provided by artstor o useful o somewhat useful o not useful o have not used onsite training session provided by artstor o useful o somewhat useful o not useful o have not used other o useful o somewhat useful o not useful o have not used if other, please specify or "na" if have not used 24. how have you sought/received technical assistance from artstor? please check all that are applicable and how frequently they were used. telephone o many times o several times o seldom o never e-mail o many times o several times o seldom o never evidence based library and information practice 2012, 7.2 47 other o many times o several times o seldom o never if other, please specify or write "na" if never 25. how would you rate the technical assistance you have sought/received from artstor? o very helpful o somewhat helpful o not helpful o have not sought technical assistance o other (please specify) 26. if you sought/received technical assistance from artstor and were not satisfied, why not? o not timely o didn’t resolve my problem o have not sought technical assistance o other (please describe) 27. would you agree to be contacted for a follow up interview? o yes o no if yes, please provide your name and email address evidence based library and information practice 2012, 7.2 48 appendix b interview schedule department position date of interview in person/telephone design associate professor, graphic design april 20, 2011 in person theatre associate professor, design april 4, 2011 telephone theatre associate professor, design april 7, 2011 telephone theatre professor, production april 13, 2011 in person visual arts associate professor, art history april 11, 2011 telephone visual arts assistant professor, art history april 13, 2011 in person visual arts associate professor, canadian art history april 13, 2011 in person visual arts associate professor, canadian art history april 20, 2011 in person visual arts professor, medieval art and architecture april 21, 2011 telephone research in practice   the open access conundrum   virginia wilson director, centre for evidence based library and information practice (c-eblip) university library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   received: 11 aug. 2015 accepted: 18 aug. 2015      2015 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   welcome to the brand new column, research in practice! as mentioned in the final ebl101 column in the last issue, research in practice is intended for those who “are using research in their practice, conducting research for their practice, and otherwise interested in the varied and vast topics pertaining to research in practice. the scope of the new column is broad, allowing for a variety of topics to be explored in a number of ways” (wilson, 2015, p. 175).   the most common misconception of evidence based library and information practice (eblip) is that it only involves using research in practice. of course, it’s not that simple, as there is more to eblip than research evidence, including user preference, what our library or info centre users want, need, or expect, and our professional expertise, the knowledge we bring to the table from our experience as practicing librarians. all three components need to be present before it’s really eblip. still, the research evidence piece is what gives eblip its zing, and it’s probably the easiest piece to leave off when pressed for time. i’ve always felt it needs a little special attention.   so, where to start in a brand new column that focuses on something as broad as research in practice? how about access? open access (oa), that is. recently on social media, a public librarian lamented that while compiling references for a writing project, she was faced over and over again with the paywall: the vendor page indicating that the article she wanted was going to cost $xx.xx to access. she wondered why any librarian would publish in a non-oa journal. retweets, replies, and commiserations followed, urging all librarians and library and information studies (lis) faculty to publish in oa journals. this got me thinking about why researchers still choose non-oa journals as publishing venues these days. i thought of the following reasons off the top of my head:   journal prestige impact factor not fully aware of the oa option and its implications for research use belief that oa journals do not employ rigorous peer review or perhaps any peer review belief that all oa journals are predatory or that the risk of this problem is too high belief that all oa journals charge a fee to publish failure to understand what the fee is actually buying   so, if i can think of a list that long in just a few minutes, it suggests that there is still some work to be done in terms of educating researchers about the realities of oa. of course, it begs the question, why don’t librarians and lis faculty know better? i would suggest that we do know better, but there are those who are making the choice not to publish oa for a variety of reasons:   prestige for tenure/promotion files high impact factor for tenure/promotion files belief that a larger audience can be reached with a non-oa journal concerns about making a mistake and publishing in a disreputable oa journal author can’t afford and/or justify a potential fee for publication or won’t approach a journal to see if it will waive or reduce the fee   the second list is not intended to be a list of excuses nor am i pointing fingers. there are many folks in librarianship publishing oa or making their research available in repositories. it’s important to acknowledge that uptake of oa publishing has been slower in the humanities and social sciences than it has been in the sciences (coonin & younce, 2009). but, a recent study shows that more librarians and lis faculty than ever are publishing in oa venues, although “librarians were confirmed to be the primary authors of oa articles on lis” (chang, 2015, p. 7). and, there are many reasons why authors are choosing to publish in oa journals:   support for the principles of oa advantage of speed of publication most suitable vehicle for making the research widely accessible objections to commercial publishing models copyright retention (nariani & fernandez, 2012, p. 183)   the reality of the paywall and the choice not to publish oa hampers evidence based practice efforts. what is this research being done for? from my perspective as an evidence based practicing librarian, it’s to apply, to use, to inform, and to teach. what happens if research is not accessible except at a great financial cost? it’s not applied or used, it does not inform, and it does not teach. or, if it does, it’s only to those who have privileged access to subscriptions. to have information stalled like that is not what librarians stand for.   and if we have no control over where researchers from other disciplines publish, surely we can look to our own discipline and make the right choices there. librarian practitioner-researchers and lis faculty members owe it to librarianship to make their research oa, either green or gold. as chang points out, a “key to the success of oa journals is that authors are willing to publish in oa journals” (2015, p. 2). and when it comes to eblip and research informing the practice of librarianship across all library sectors, only oa makes sense.   references   chang, y.-w. (2015). librarians’ contributions to open access journal publishing in library and information science from the perspective of authorship. the journal of academic librarianship. advanced online publication. http://dx.doi.org/10.1016/j.acalib.2015.06.006   coonin, b. & younce, l. (2009). publishing in open access journals in the social sciences and humanities: who’s doing it and why. acrl 14th national conference papers. seattle, wa, 85-94. retrieved from http://bit.ly/1htwo6s   nariani, r. & fernandez, l. (2012). open access publishing: what authors want. college and research libraries, 73(2), 182-195. retreived from http://crl.acrl.org/content/73/2/182.full.pdf+html   wilson, v. (2015). ebl101: riding into the sunset. evidence based library and information practice, 10(2), 174-175. retrieved from https://ejournals.library.ualberta.ca/index.php/eblip/article/view/24631/18435 research article   digital health and professional identity in australian health libraries: evidence from the 2018 australian health information workforce census   cecily gilbert research assistant centre for digital transformation in health the university of melbourne parkville, victoria, australia email: cecilyg@unimelb.edu.au   kathleen gray associate professor centre for digital transformation in health the university of melbourne parkville, victoria, australia email: kgray@unimelb.edu.au   kerryn butler-henderson associate professor of digital innovation in health and health pedagogy college of health and medicine university of tasmania launceston, tasmania, australia email: k.butlerhenderson@utas.edu.au   ann ritchie national manager alia health libraries australia canberra, act, australia email: annritchie@yahoo.com   received: 2 sept. 2019                                                                    accepted: 10 jan. 2020      2020 gilbert, gray, butler-henderson, and ritchie. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29640     abstract   objective – this research aimed to examine the characteristics of the current health library professional workforce in australia. the study also sought to explore the areas of health library competency domains and job functions that may reflect progress toward a specialized digital health information capability.   methods – health librarians’ responses to the may 2018 australian health information workforce census were analysed and compared with results obtained in earlier census counts. the health librarian characteristics were also compared with other health information occupations included in the census.   results – there were 238 usable health librarian responses. these indicate that the health librarian workforce continues to be a comparatively mature population, with substantial experience, increasing involvement in dataand technology-intensive functions, high levels of professional association membership, and participation in continuing education activities. notably there are emerging role titles and job functions which point to a greater digital health focus in the changing work realm.   conclusion – the health librarian workforce has adapted its skills, in line with the increased digital emphasis in health information work. however, as with other health information occupational groups, it is possible that health system planners and funders are not aware of librarians’ current functions and skills. this mature workforce may undergo significant attrition and consequent loss of expertise in the next decade. continued advocacy and strategic planning around these factors with workforce, healthcare quality, and educational organizations will be required.     introduction   information and communication technology advances are transforming the way that health care systems operate and the kind of care they provide (marques & ferreira, 2019). major technologies include telemedicine, smartphone apps, sensors and wearables for diagnostics and remote monitoring, reading and writing the genome, speech recognition and natural language processing, virtual and augmented reality, artificial intelligence based image analysis, predictive analytics, and rehabilitative robotics (topol, 2019). “the use and scale up of digital health solutions can revolutionize how people worldwide achieve higher standards of health, and access services to promote and protect their health and well-being,” according to the world health organization (who, 2019).   the digital transformation of health highlights the need to strengthen that part of the overall health workforce who are specialists in the information and communication methods and tools used in digital health. these practitioners are responsible for the development, maintenance, and governance of the systems used to manage health data, health information, and health knowledge. yet this workforce is poorly defined in general. for example, standards australia’s digital hospital handbook identifies only two relevant roles: chief clinical information officers and health information managers (standards australia, 2017). health information specializations such as health librarianship are often invisible to those responsible for digital health implementations and the consequent quality and safety of patient care (gray, gilbert, butler-henderson, day, & pritchard, 2019). there is not yet consensus on the capabilities required in the specialists who manage digital health information. a recent voluntary survey by the health informatics career pathways project illustrates this divergence: it identified a wide variety of skills, specializations, and job titles in the current non-clinical health informatics workforce in the united kingdom and ireland (cowey, 2019). professional education bodies such as digital health canada include multiple domains of knowledge and expertise in their informatics certification programs (digital health canada, 2019).   the ill-defined status of the health information labour force is also apparent in the incomplete workforce data available. in australia, for example, estimates of the size of the australian health information workforce, by health workforce australia (hwa) and other organizations between 2009 and 2013, ranged from around 6,000 to 11,800. these varying numbers were due to differences in the occupations that were included by the data-collecting bodies and to different data collection methods.   numbers for the professional workforce in australian health libraries are not included in these estimates. unlike many other work groups in health, librarians are not a regulated practitioner group, and there is no national board licence or registration needed to practice. health librarians are thus relatively unseen even in the health workforce. providers of librarianship education programs do not offer a medical or health information specialty subject, which means that course graduate numbers are not useful for counting new entrants to the health area. for these reasons, detailed descriptions of the australian health librarian workforce are reliant on efforts by professional bodies and interested researchers.   this paper focuses on data about the current health librarian workforce in australia, as a case to illustrate the workforce issues implicit in adoption of digital health. factors such as adequate supply, changes in work role, and preparation through specialist training will be considered.   literature review   a strong body of literature has appeared in the past decade on predicted changes in the health librarian profession. the scoping review by ma, stahl, and knotts (2018) described nine evolving and active roles, with embedded librarians as a strong theme. several authors have identified external and occupational drivers of change, and discussed the need for the profession to respond (brettle & urquhart, 2011; crum & cooper, 2013; henderson, 2014; holst et al., 2009; hopkins, 2017; king & lapidus, 2015; mcgowan, 2012; murphy, 2013). hallam et al. (2010) concisely stated the challenges and outlook for the health librarian workforce: “traditional library work is diminishing, professional boundaries are blurring, and emerging areas of work are being claimed by other professional groups” (p. 355).   health librarian workforce surveys   in the past decade surveys of librarians in health roles have revealed shifts in the responsibilities and skills of this group. sen, villa, and chapman (2014) collected data on health information professionals practicing across europe, as a means to understand their current roles, skills, professional development needs, and views on the impact of their work. the data were obtained through an online survey, focus groups, and individual interviews. the 513 respondents identified a wide range of roles: “evidence-based roles (e.g., literature searching and teaching/training) and management roles, including library-specific management roles, more generic management roles, communications roles and roles involving technology” (p. 12). the authors noted the breadth of roles across the sector, as well as within the context of individuals’ jobs. focus group members commented on the changes in their roles, due in part to technological changes.   dunikowski et al. (2013) surveyed united states health association libraries to gather details of their status, services, staffing, and technology. this was the most recent in a survey series conducted since 1980 by the health association libraries section of the medical library association and its predecessors. sixty organizations with a library responded. nominated changes in the roles of these libraries and staff included an increase in the volume and complexity of services. in addition, non-traditional services represented 10% or more of library staff workload. a number of these areas involved technology-enabled information work, such as managing archives, publications and citation support, records management, website involvement, and metadata support.   mclaughlin, spencer, zeblisky, liszczynskyj, and laera (2018) surveyed 383 solo hospital librarians. over two-thirds of the respondents worked in hospital systems and community hospitals; nearly half had 15 years’ experience or more working in a hospital setting, 75% worked full-time, and 84% held a master’s degree in library and information science. duties covered internal library operations (such as literature and reference services, managing information resources, and teaching) as well as external links with clients and groups, for example, committee work in patient safety, research and ethics committees, and clinical education. while noting the depth and breadth of this workforce, the authors also drew attention to challenges, such as recognition of the librarian’s specialist contribution to the organization’s purpose (p. 132).   benchmark surveys   recent benchmarking surveys of healthcare libraries also illustrated current services and staff characteristics. benchmarking aims to enable libraries to compare their performance by gathering statistics from similar sites. the hospital libraries benchmarking study by spencer, mamo, and billman (2019) obtained data from 180 north american respondents about services, client groups, funding, and activity. the libraries were predominantly small, with one or two staff, yet the majority offered a wide range of services, confirming the breadth of skills noted by mclaughlin et al. (2018). more than 80% held membership in their professional association. the authors recommended a large-scale longitudinal study of hospital libraries to obtain baseline data, so that benchmark surveys can be conducted regularly. this data would “be readily available for use with hospital administrators and for hospital library planning and advocacy” (p. 18).   earlier, ducas, demczuk, and macdonald (2015) benchmarked canadian health libraries against the 2006 canadian health libraries association/association des bibliothèques de la santé du canada standards for libraries and information services in canadian healthcare facilities. almost one-third of the 168 responses noted shortcomings in staffing compared to the level set in the standards. respondents indicated increased activity in the past five years in user assistance services and literature searching. the authors suggested updating the standards to reflect “the accelerated pace of transformation to health library practice” (p. 9).   taken together, the earlier surveys show that the majority of health librarians had significant experience in this specialty. increases in the volume and complexity of services in health libraries were observed, which were largely attributable to technological and environmental changes in the industry. of particular note is the rise of non-traditional services (whether libraryor user-initiated) which may challenge the status quo and prompt the need for re-skilling or ongoing education.   background on the australian health library workforce   australian library and information association’s (alia) health libraries australia (hla) section has initiated substantial research into the characteristics of health library services and the health librarian workforce in the past 10 years (blackwood & bunting, 2016; siemensma, ritchie, & lewis, 2017). this has been driven by the need to gain an informed picture of the make-up of the workforce—size, composition, education, work performed, and future training requirements. noting the forecast introduction of e-health and similar technologies into the healthcare environment, australian health library organizations also questioned the adequacy of existing education programs and ongoing training.   the findings of the workforce studies were detailed in two reports:   the 2009-11 health librarianship workforce and education research study (hallam et al. 2011), and the 2014-15 census of australian health libraries and health librarians (kammermann, 2016).   comparisons between the two studies are not straightforward due to differences in target respondents and the variations in questions used in each instrument. appendix a shows the aims and methods used in these two studies, and appendix b lists the key findings in the studies.   data from the 2014-15 census was extrapolated to develop the estimate that there were approximately 1,250 people in the australian health library workforce: 760 health librarians, 290 library technicians, and 200 non-library-qualified staff. both studies found the workforce was predominantly female, aged 40 years or older, and worked in the government health sector. at least one-third were eligible to, or intended to, retire within five years. the 2011 study found 70% of respondents had a bachelor’s degree or graduate diploma/certificate in library studies; fewer than 15% had a higher degree. however, this study also found a high interest in professional development, with 75% of respondents having undertaken 11 hours or more of continuing educational activities in the past year. this study also found that more than 80% of individual respondents used technology and systems to manage information, and expected that this competency requirement would increase in the future. in the 2016 research, more than 40% of the library services were providing some form of technology-related support services to their clients, including digital repositories, digitization services, and internet and intranet development or management. in addition, implementation of new software or growth in electronic resources and services were the most frequently mentioned service changes in the previous year.   in the wider health information occupations, hwa had recommended that data collection processes should be improved. the stakeholder groups mobilized to host a national health information workforce summit in 2016, with representation by professional bodies for health information managers, health informaticians, health librarians, clinical coders, and health service managers. the summit’s action plan recommended action to develop and conduct an australian health information workforce census (butler-henderson et al., 2017). this was implemented in may 2018.   aims   this study seeks to understand the current australian health library workforce, using data from the 2018 australian health information workforce census. characteristics such as age structure, education and experience, employment, professional affiliations, and future work intentions are examined to establish the areas of workforce changes when compared with earlier australian health library workforce research. the study also aims to examine the nature of competency domains and job functions that may suggest a shift towards a specialized digital health information capability.   methods   census details   the health information workforce census project aims to “quantify and qualify the australian health information workforce (hiw), specifically to delineate and count the workforce, consider the future configuration of workforce, identify health information workforce shortfalls, as well as current health information training and career pathways” (butler-henderson & gray, 2018a, para. 5).   the census is a collaborative research project being conducted with human research ethics approval by the university of tasmania and the university of melbourne, who are jointly responsible for the design and operation of the census. the university researchers are advised by a management group of stakeholder organization representatives: the australian digital health agency, alia health libraries australia, australasian college of health informatics, health informatics society of australia, health information management association of australia, and the victorian government department of health and human services (butler-henderson & gray, 2018a).   the census tool was developed between 2016 and 2018 by a multi-professional expert panel (butler-henderson et al., 2017). it comprised approximately 160 questions on data elements such as demographics, education, employment, competence domains, functions, certification, professional development, and intentions for future work in the health information workforce. elements were designed with the aim of staying relevant over at least 15 years. the intention is to conduct the australian census every two years, and international partners are being sought to run it in other countries. the census planning included provision for respondents to register for the longitudinal study, thus enabling linking of those individuals’ data from one census to the next.   the initial census was conducted online across australia in may 2018, with a paper census available on request. publicity was distributed in electronic media via a dedicated website (butler-henderson & gray, 2018a), social media channels, and electronic mail lists for the target professions. for example, the professional group hla’s endorsement for the census was evident in its promotional messages: health librarians were invited to complete the census through announcements in the hla newsletter, messages on the mail list aliahealth, and on hla’s three social media sites.   the census invited voluntary participation from individuals who self-identified as part of the health information workforce. they were defined as those who “work in a role where the primary function is related to developing, maintaining, or governing the systems for the management of health data, health information or health knowledge . . . for/with an organisation that operates in australia . . . and your role relates to the australian operations, and relates to the health sector” (butler-henderson & gray, 2018a, para. 6).   health librarian data in the census responses   our project to examine the census’ health librarian data was approved by the university of melbourne general practice human ethics advisory group (#1853443.1) in february 2019. the census privacy statement and the data management and access policy have been followed in this project. summary results for the full response set were published in late 2018 (butler-henderson & gray 2018b). there were 1,597 usable census responses in total. within that dataset, three criteria were used to identify respondents in the health librarian occupational group:   health librarian was chosen as the occupational group from the list provided; or the job title included the word: [librar*] or the respondent held a qualification that included the word [librar*] in the title.   these criteria will include health library technicians (the para-professional occupation) as well as health librarians.   eligible responses were extracted from the census database into a microsoft excel spreadsheet and imported into ibm® spss v25.0 for analysis. descriptive statistical analysis focused on employment and role characteristics, on markers of professional identity (educational background, continuing professional development, and professional memberships), and on intention to remain in the workforce. deductive thematic analysis focused on job functions; for this purpose, we used a recognized health librarians’ competency framework as our guide to categorize free text responses (alia, 2018).   there were 238 responses (14.4% of the total usable census responses) which met the above criteria and were included in our analysis. the figures are stated as headcount, not full-time equivalents. all survey questions were optional, thus responses to some questions do not total 238. specific response numbers and rates are included where relevant.   when compared with the 1,200 headcount estimated from the 219 institutional responses to the 2014-15 census, the 2018 figure represents approximately 20%-25% of the earlier response numbers. in the 2009-11 study there were 161 responses.  these figures suggest some consistency in response rates for the voluntary survey method.   results   the health librarian occupational group is a mature-age and largely female group. the average age was 54 years (range 28-72 years). in detail, 81.0% of this group is 45 years or older; there was a significant association between those who identified in the health librarian occupational group and being aged 45 years and older (χ2(1) = 67.613, p < 0.001). a majority of respondents (65.0%) had worked in health information roles for more than 10 years, confirming the experienced and mature nature of this workforce. table 1 shows the demographic features of the health librarian group and the full hiw group who responded to the census.   table 1 demographic characteristics   health librarians (n = 238) all hiw (n = 1,597) average age (years) median range 53.98 50-59 28-72 45.03 50-59 20-70s aged 45+ aged 60+ 81.0% 34.0% 52.0% 14.0% gender female male other/not answered   88.2% 11.3% 0.5%   78.1% 21.6% 0.3% citizenship australian citizen other resident   97.5% 2.5%   92.7% 7.3% born in australia 79.0% 74.5% identified as aboriginal or torres strait islander < 5.0% < 5.0% participation limited by disability or health condition 4.0% 3.4%   employment and organizational characteristics for the occupational group health librarians and for the full hiw census group are summarized in table 2.   table 2 employment and organizational characteristics employment characteristic (hl n / hiw n) responses health librarians all hiw number percentage number percentage time since qualification (223/1370) average 21 years   15 years   range 1-47 years   0-55 years   <5 years 15 6.7% 247 18.0% 5-9 years 28 12.6% 252 18.4% 10-19 years 68 30.5% 422 30.8% 20-29 years 57 25.6% 288 21.0% 30-39 years 39 17.5% 129 9.4% 40+ yrs 16 7.2% 32 2.3% major employment group (212/1142) manager 45 21.2% 413 36.2% professional 141 66.5% 558 48.8% clerical or admin 16 7.5% 167 14.7% technician or trade 10 4.7% 5 0.4% organization status (206/1106) public 170 82.5% 801 72.4% private 14 6.8% 187 16.9% public-private partnership 5 2.4% 33 3.0% not for profit 17 8.3% 85 7.7% organization type (206/1109) hospital 120 58.3% 701 63.4% educational facility 37 18.0% 23 2.1% state health department 19 9.2% 96 8.7% local health service 11 5.3% 117 10.6% other public institution 7 3.4% 169 15.3% other private < 5 < 3.0% < 5 < 0.5% employment status (206) permanent 194 94.2% not given 82.1% contract 7 3.4% not given 14.7% casual 5 2.4% not given 1.8% actively seeking hi work 19 8.8% not given 15.1% hours worked (206) average paid hours per week 28.6 hrs   32.6 hrs     respondents were invited to select the areas of competence they require to perform their health information work, using the five domains that underlie the certified health informatician australasia competencies framework (health informatics society of australia, 2013). multiple selections were possible. the results emphasize the dominance of technological and data science competencies in the census respondents’ view of the subject domains they need to work effectively. answers were as follows:   ·         information & communications technologies: 72.6% ·         data & information science: 61.4% ·         health & biomedical science: 53.6% ·         human & social science: 49.75% ·         management science: 46.8%   one question sought respondents’ view of their broad work category—seven occupational categories available were manager, professional, clerical, technician, sales, labourer or community worker. of those who replied, 66.5% chose professional, 21.2% chose manager, 7.5% chose clerical or administrative, and 4.7% chose the technical category.   the census asked for role title details. respondents gave an array of more than 65 position titles. five position titles were given in almost half the responses: librarian, library manager, library technician, senior librarian, and medical librarian. in the wide span of other role titles provided, twelve newer titles were listed that reflect the digital or electronic environment, including data officer, digital content coordinator, e-health facilitator, electronic resources librarian, electronic services or e-services librarian, health information coordinator, knowledge services advisor (or manager), library and literacy project officer, systems educator, and systems support librarian. other specialist role titles include: consumer health information coordinator, (medical or senior) research librarian, research information specialist.   respondents were invited to state the top five functions they performed in their health librarian role. analysis of the 849 responses showed continuing emphasis on direct user assistance, education, and information literacy. management of services, resources, and online systems were also well-represented, as shown in table 3.   table 3 job functions categorized according to hla health librarian competency areas alia hla competencies for health librarians 2018 census: named functions performed number (percentage) n = 849   c2 reference & research   assist clients seeking information search information resources perform systematic review tasks   137 (16.1%) 74 (8.7%) 14 (1.6%) total = 26.4% c3 resources   manage information resources arrange document supply acquire information resources   89 (10.4%) 57 (6.7%) 42 (4.9%) total = 22.0% c4 leadership & management manage the information service advocate, promote information service 113 (13.3%) 36 (4.2%) total = 17.5% c5 digital, e-health, technology & systems   maintain it systems perform data management tasks   82 (9.6%) 7 (0.8%) total = 10.4% c6 health literacy & teaching   provide education and training   135 (15.9%) c7 health research participate in research team   51 (6.0%)   unable to categorize 12 (1.4%)   the census asked about performance of unpaid or voluntary work. there were 205 responses; 14.0% of these respondents said they undertook unpaid or voluntary tasks. examples included board or committee roles (9.0%); writing, publishing, or reviewing (6.5%); event management (5.0%); and mentoring or advising (3.5%). respondents stated they worked an average five unpaid hours per week (range 0-21 hours).    educational background   as noted above, in australia there has been no specialty health librarian or health information professional qualification offered by educational bodies. the census question asked: “what is the highest formal educational qualification you have attained that you believe is relevant to your health information work?” respondents in the health librarian group stated a range of qualifications and course titles. more than a third (36.7%) hold a graduate certificate or graduate diploma (with underlying bachelor degree); 27.35% hold a bachelor degree; 15.4% have a certificate or diploma; 12.0% hold a master’s degree; and 3.8% have a doctorate. consistent with the age profile of the health librarian group, the average time since completion of the highest award was 21 years (range 1-47 years). in contrast, the overall hiw participant cohort had on average completed their health information qualification 15 years earlier.   continuing professional development   the 86% of health librarian respondents who indicated they had undertaken some form of professional development in the past year nominated 380 activities undertaken. work-based learning (35.0%), professional association activities (30.7%), and self-directed learning (27.8%) were popular, while 6.3% of respondents had completed a formal educational program. fourteen percent of respondents had not participated in any further learning in the previous year. in a subsequent question on future learning intentions, 83 participants (42.0% of responses) said they intended to undertake further learning or professional development about health information, 57 (28.8%) were unsure, and the remaining 58 (30.0%) chose “not applicable.”   professional membership and certification   two-thirds of the health librarian group held a membership in one or more professional associations; 91.0% were members of the hla section in the alia, while 7.6% were members of the victorian state-based body (health libraries inc.) and 3.8% were in the health information management association of australia. the trend is not as strong for the whole census cohort, where 44.5% stated that they do not hold any membership in a professional or industry association.   however, maintenance of a health information certification is far less common. certification in health information areas is available from a number of professional associations (e.g., alia) to recognize practitioners who voluntarily complete the association’s professional development or continuing education program cycle. only one quarter of the respondents held a certification; of these, 91.0% were certified practitioners with the alia health librarian specialty. the remainder held either a certified health informatician australasia award, or a health information management certification.   future work intentions   of the 198 health librarian respondents who answered the question “how many more years do you intend to remain in the paid health information workforce in australia?” 34.8% said they will leave within five years. in contrast, only 16.9% of the respondents from the full hiw group plan to leave within five years (see table 4).     table 4 intention to remain in the workforce   health librarians number (percentage) n = 198 all hiw number (percentage) n = 1,041 will remain more than 5 years 95 (48.0%) 719 (69.1%) will leave within 5 years 69 (34.8%) 176 (16.9%) unsure 34 (17.2%) 146 (14.0%)   the census also asked about post-work or post-retirement involvement in the health information area. forty-three respondents (21.7%) planned to continue in an unpaid or volunteer capacity; the envisaged median duration in this capacity was 6-10 years.   discussion   this description of the australian health library workforce is consistent in many ways with results of earlier studies in 2009-11 and 2014-15. the older age structure (average age 54 years, and 32.4% aged 60-69 years), length of experience in this work, and time since obtaining formal qualifications are largely unchanged. in comparison, the average age of the entire australian health workforce in 2017 was 43.6 years. just over 20% of the census respondents were born overseas. in contrast one-third of respondents at the full australian 2016 population census said they were born overseas (australian bureau of statistics, 2016). the health library workforce does not reflect the diversity of the broad australian population. future advocacy and educational and recruitment efforts will need to address this shortcoming, as has been recognised for the wider australian library workforce (alia, 2019).   responses on intent to continue in the paid health information workforce illustrate the challenge to planners and professional bodies representing the health librarian specialty. more than one-third (n = 69, 34.8%) of respondents plan to leave within five years. this is a much higher planned departure rate than the 16.9% for all census respondents. it is consistent with the figure of 36% of qualified library staff found in the 2014-15 census to be eligible to retire within five years (kammermann, 2016, p. 37). it is not clear if the foreshadowed “net loss” (p. 3) predicted by kammermann has eventuated. the health library sector will need to re-assess how to respond to and plan for the potential departure of up to one-third of the current workforce by 2023. continuing efforts to demonstrate the value that health librarians contribute to the parent organizations may help to combat job redundancies and library closures.   on average, health librarians attained their highest qualification 21 years earlier. active participation in continuing professional development and interest in upskilling reflect an awareness that the health care environment is changing, and health information workers need to reassess and refresh their knowledge, skills, and services to match. recent active research and advocacy by hla has resulted in the introduction of a professional development pathway and certified practitioner award that recognizes ongoing self-directed learning. the association has partnered with education providers to jointly develop educational courses, ranging from single-day workshops through to a masters-level semester-length subject, to enable new entrants and current health library professionals to gain a specialist qualification in this area.   another professional identity marker—association membership—is also reasonably strong, with two-thirds of the health librarian group holding a membership in a professional or industry association. although a national professional development scheme with a health specialist certification structure exists, the absence of a required licence to practice or a national registration scheme confirms the difficulty of establishing and retaining a clear identity in the changing workforce.   an examination of the role titles provided by respondents indicates both continuity and change. there is a high frequency of traditional titles such as librarian, library manager, and medical librarian. however, the uptake of digital or e-health labels in existing roles points to a broadening of the health information work field, consistent with the predicted changes in the wider health setting. role titles that include data, digital, e-health, electronic, information coordinator, knowledge, literacy, and systems suggest a recognition of change in the nature of information sources and the skills required to work with them.   the change in work focus is also evident in the areas of competence nominated by respondents as essential to perform their current health information role. information and communications technology and data science competencies were selected by 61.4%-72.6% of the participants, while other domains—health science, social science, and management—were chosen by approximately 50% of respondents. this response is more pronounced than the results for the entire census cohort; those were more evenly spread (43.6%-65.4%) across the five areas of competence.   in another perspective on competencies, the health library job functions that were most frequently mentioned in 2018 corresponded with the areas nominated as “most likely to increase” in the 2009-11 research, namely reference and research, resources, health literacy and teaching, and—to a lesser extent—digital, e-health, technology and systems. in addition, leadership and management was ranked highly in the 2018 responses, perhaps indicating that health librarians are taking on management roles currently where this was not widespread ten years earlier.   these changes in areas of competence are generally consistent with findings across the broader australian health information workforce, as reported by gray et al. (2019). there is not a readily recognisable specialization in the health workforce that is understood as being the “logical” profession to manage and govern digital health.   the outlook   since “health librarian” is not a recognized profession in the australian and new zealand standard classification of occupations (anzco), positions for health librarians or health information professional roles do not have formal educational or certification requirements in the australian workforce.   with this lack of standardization, there is potential for newly created or updated health information roles to overlook or dismiss the existing health librarian capabilities. examples can be found in current technology forecasting literature, agency roadmaps, and emerging literature. the article extract from adler-milstein, nong, and friedman (2019) illustrates this point:   the current state of knowledge management in healthcare delivery organizations relies on an outdated biomedical library model, and only a small number of organizations have developed enterprise-scale knowledge management approaches that “push” knowledge in computable form to frontline decisions. (p. 1)   the authors highlight the dynamic nature of health-related knowledge, and state that the “pull” model of the traditional library struggles to cope with the need to integrate knowledge into clinical practice. it is claimed that “a relatively small number of organizations” have adopted knowledge management infrastructure that enables evidence-based advice to be pushed to decision makers (p. 3). the article cites a 2006 example of work at a large united states health system to implement a scalable clinical decision support system (cdss). however, there are many more recent cases where health librarians have enabled provision of health information resources at the point of care as part of a cdss, for example, as described by fowler et al. (2014) and by ma et al. (2018). examples illustrating the application of librarians’ knowledge management expertise in the cdss are also available: frakes et al. (2017) described practice at vanderbilt university medical center creating evidence summaries and linking knowledge briefs to specific decision scenarios, while wright et al. (2009) outlined the role of librarians in managing metadata in health knowledge systems.   our analysis of the census data is hampered by the absence of a 2018 baseline figure for all health library positions. as expected in a census aimed at individual respondents, questions were not asked about the total number of positions in the organization’s library or information service, nor about job vacancy rates. the 2014-15 census sent to health library managers found a job vacancy rate of 9.6%. in the current census 19 respondents in the workforce indicated they were actively seeking work. each year brings anecdotal news of a small number of health libraries closing or merging within larger organizations, or reducing their staff quotas, with some health information services extending their boundaries by incorporating neighbouring districts. it would be useful to update the headcount and full-time equivalent numbers at regular intervals, as well as the count of health information services. a similar recommendation was made by spencer et al. (2019, p. 18) for united states hospital libraries.   more generalized data from the australian labour market employment projections to 2023 predicts the “librarian” occupational group will rise 6.4% from the current 15,400 figure to 16,400. this growth is modest when compared with the category “information and organisation professionals” which is predicted to increase by 16.1%, from 164,200 to 190,000 positions. it would be useful to know whether health information professionals are included in this latter estimate, and if so, how they are defined. this indicates that role and role title are significant in analyses of future outlook.   finally, the results of the census’ implementation in new zealand, with more than 450 responses, provides an opportunity in the future to undertake an inter-country comparison of results (day & grainger, 2019).   limitations   a key limitation in this study is that it relied on respondents’ self-selection to participate in the census. any voluntary survey or instrument raises a similar issue of potential bias in the sample who respond. in the case of the health information workforce census, extensive efforts were made to reach the desired groups using electronic communication channels, in the lead-up to the 1 may 2018 start date, and throughout the month that the online census was available.   the selection criteria used to extract the health library staff group from the full dataset of census responses were perhaps too inflexible. it is possible that eligible respondents did not select the occupational group “health librarian” and did not have a role or a qualification that included the word librar*.  however, the resulting set of 238 responses appears to be consistent with earlier australian health library staff surveys.   this form of selection criteria has also been used to extract other occupational groups from the full census dataset, such as health informatics (butler-henderson et al., 2019). its wider application suggests that the method is a best fit for this purpose. nonetheless, given these limitations, the findings should be regarded as indicative.   conclusion   the 2018 census for the health information workforce has provided up-to-date evidence on the current status of the health librarian workforce in australia. while it has confirmed the demographic and employment trends in the two earlier studies of health librarians undertaken since 2009, it has also revealed elements suggesting a stronger digital health information focus in both role titles and in the work being performed.   analysis of role functions and perceived competency requirements obtained in this census will assist with future role development and specification of the knowledge, skills, and attributes that new entrants will require. ideally this can be presented cohesively with similar requirements data for the other health information occupational groups captured in the census. these would be persuasive in advocacy with the australian digital health agency about its workforce and education program, currently aimed at upskilling clinical staff only.   there is merit in sharing these results with health information and health library professional bodies internationally, noting that the australian census tool is designed to be replicable in other countries. these strategies will assist in translating this research into workforce reform and support improved patient safety.   note about data access/availability of the census data   access to the de-identified census data will only be approved for non-commercial purposes (e.g., research). please review the data management and access policy at https://www.utas.edu.au/__data/assets/pdf_file/0003/1090776/data-management-and-access-policy-v1_0.pdf.   the data access application link is at https://redcap.utas.edu.au/surveys/?s=8y9rh44kkr.   references   adler-milstein, j., nong, p., & friedman, c. p. (2019). preparing healthcare delivery organizations for managing computable knowledge. learning health systems, 3(2), e10070. https://doi.org/10.1002/lrh2.10070   australian bureau of statistics (2016). 2016 census quickstats. retrieved from http://quickstats.censusdata.abs.gov.au/census_services/getproduct/census/2016/quickstat/036   australian government department of employment, skills, small and family business. (2018). employment projections—for the five years to may 2023. retrieved from http://lmip.gov.au/default.aspx?lmip/gaininsights/employmentprojections    australian library and information association (2018). alia hla competencies. retrieved from http://www.alia.org.au/sites/default/files/hla%20competencies_0.pdf   australian library and information association (2019). alia galleries, libraries, archives, museums and records workforce diversity trend report. canberra: alia.   blackwood, d., & bunting, j. (2016). the role of the health librarian: ways of working towards professional recognition in the australian health workforce. him-interchange, 6(3), 14-19.   brettle, a., & urquhart, c. (2011). changing roles and contexts for health library and information professionals. london: facet publishing.   butler-henderson, k., gray, k. (2018a). health information workforce census website. retrieved from http://www.utas.edu.au/business-and-economics/hiwcensus   butler-henderson, k., & gray, k. (2018b). australia’s health information workforce: census summary report 2018. retrieved from http://www.utas.edu.au/__data/assets/pdf_file/0003/1163487/australias-hiw-census-summary-report-2018.pdf   butler-henderson, k., gray, k., greenfield, d., low, s., gilbert, c., ritchie, a., . . . schaper, l. k. (2017). the development of a national census of the health information workforce: expert panel recommendations. studies in health technology and informatics, 239, 8-13. https://doi.org/10.3233/978-1-61499-783-2-8      butler-henderson, k., gray, k., pearce, c., ritchie, a., brophy, j., schaper, l. k., . . . ryan, a. (2019). exploring the health informatics occupational group in the 2018 australian health information workforce census. studies in health technology and informatics, 266, 44-50. https://doi.org/10.3233/shti190771   cowey, a. (2019 nov.). health informatics career pathways project, report a: summary report. [london]: health education england, with south, central & west commissioning support unit. retrieved from https://www.hee.nhs.uk/sites/default/files/documents/health%20informatics%20career%20pathways%20report%20a.pdf   crum, j. a., & cooper, i. d. (2013). emerging roles for biomedical librarians: a survey of current practice, challenges and changes. journal of the medical library association, 101(4), 278-286. https://doi.org/10.3163/1536-5050.101.4.009    day, k., and grainger, r. (2019). new zealand’s health information workforce: census summary report. retrieved from http://hiwcensusnz.blogs.auckland.ac.nz/2019/08/05/2019-summary-report/   digital health canada (2019). core health informatics education modules and learning objectives. retrieved from http://digitalhealthcanada.com/wp-content/uploads/2019/06/digital-health-canada-education-modules-learning-objectives-v05-2019.pdf   ducas, a., demczuk, l., & macdonald, k. (2015). results of a survey to benchmark canadian health facility libraries. journal of the canadian health libraries association/journal de l'association des bibliothèques de la santé du canada, 36(1), 3-10. https://doi.org/10.5596/c15-008   dunikowski, l. g., embrey, a. c., hawkes, w. g., riedlinger, j. e., taliaferro, m. g., & van hine, p. m. (2013). the health association libraries section survey: finding clues to changing roles. journal of the medical library association, 101(4), 318-322. https://doi.org/10.3163/1536-5050.101.4.015   fowler, s. a., yaeger, l. h., yu, f., doerhoff, d., schoening, p., & kelly, b. (2014). electronic health record: integrating evidence-based information at the point of clinical decision making. journal of the medical library association, 102(1), 52-55. https://doi.org/10.3163/1536-5050.102.1.010   frakes, e., fox, z., su, j., blasingame, m., epelbaum, m., desautels, s., . . . giuse, n. b. (2017). applying knowledge management best practices to capture, support, and archive systems-embedded clinical decision support evidence. paper presented at the mla 117th annual meeting, seattle, may 2017. retrieved from http://jmla.mlanet.org/ojs/jmla/article/view/421/599#s1-jmla-106-e1   gray, k., gilbert, c., butler-henderson, k., day, k., & pritchard, s. (2019). ghosts in the machine: identifying the digital health information workforce. studies in health technology and informatics, 257, 146-151. https://doi.org/10.3233/978-1-61499-951-5-146   hallam, g., ritchie, a., hamill, c., lewis, s., newton-smith, c., kammermann, m., & o'connor, p. (2010). australia’s health libraries: a research-directed future. library trends, 59(1-2), 350-372.   hallam, g., ritchie, a., hamill, c., lewis, s., o'connor, p., kammermann, m., . . . newton-smith, c. (2011). health librarianship workforce and education: research to plan the future. canberra: alia. retrieved from http://www.alia.org.au/sites/default/files/documents/our-communities/healthlibrarianshipworkforcereport.pdf   health informatics society of australia. (2013). health informatics competencies framework; certified health informatician australasia. retrieved from http://www.healthinformaticscertification.com/wp-content/uploads/2016/02/chia-competencies-framework_final.pdf   health workforce australia. (2013). health information workforce report, october 2013. retrieved from http://www.aims.org.au/documents/item/401   henderson, m. (2014). new roles and new horizons for health sciences librarians and libraries. in m. sandra wood (ed.) health sciences librarianship (pp. 403-418). lanham, md: rowman & littlefield.   holst, r., funk, c. j., adams, h. s., bandy, m., boss, c. m., hill, b., . . . lett, r. k. (2009). vital pathways for hospital librarians: present and future roles. journal of the medical libraries association, 97(4), 285-292. https://doi.org/10.3163/1536-5050.97.4.013   hopkins, e. (2017). knowledge management in healthcare libraries: the current picture. health information & libraries journal, 34(2), 103-105. https://doi.org/10.1111/hir.12183   kammermann, m. (2016). the census of australian health libraries and health librarians working outside the traditional library setting: the final report of the 2012 anne harrison award project. conducted between october 2014-february 2015. retrieved from http://www.alia.org.au/sites/default/files/census%20of%20aus%20hlth%20libs%202012-14_final%20report_2016.pdf     kiesau, k. (2003, june). health libraries australia: snapshot. hla news, 7-8. retrieved from http://pandora.nla.gov.au/pan/49496/20050831-0000/www.alia.org.au/groups/healthnat/hla/hla.news-june.2003.pdf   king, s. b., & lapidus, m. (2015). metropolis revisited: the evolving role of librarians in informatics education for the health professions. journal of the medical library association, 103(1), 14-18. https://doi.org/10.3163/1536-5050.103.1.003   ma, j., stahl, l., & knotts, e. (2018). emerging roles of health information professionals for library and information science curriculum development: a scoping review. journal of the medical library association, 106(4), 432-444. https://doi.org/10.5195/jmla.2018.354   mcgowan, j. j. (2012). evolution, revolution, or obsolescence: an examination of writings on the future of health sciences libraries. journal of the medical library association, 100(1), 5-9. https://doi.org/10.3163/1536-5050.100.1.003   mclaughlin, l., spencer, a., zeblisky, k., liszczynskyj, h., & laera, e. (2018). solo census: demographics, duties, needs and challenges. journal of hospital librarianship, 18(2), 127-135. https://doi.org/10.1080/15323269.2018.1437503   marques, i. c. p., & ferreira, j. j. m. (2019). digital transformation in the area of health: systematic review of 45 years of evolution. health and technology. https://doi.org/10.1007/s12553-019-00402-8   murphy, j. (2013). international trends in health science librarianship: part 7 taking stock. health information & libraries journal, 30(3), 245-252. https://doi.org/10.1111/hir.12034   sen, b., villa, r., & chapman, e. (2014). the roles, skills, training needs and contributions of health library and information professionals. journal of the european association for health information and libraries, 10(2), 11-14.   siemensma, g., ritchie, a., & lewis, s. (2017). shaping the professional landscape through research, advocacy and education an australian perspective. health information & libraries journal, 34(2), 171-176. https://doi.org/10.1111/hir.12180   spencer, a., mamo, e., & billman, b. l. (2019). benchmarking study of hospital libraries. hypothesis, 31(1). https://doi.org/10.18060/23805   standards australia (2017). digital hospitals handbook (hb 163:2017). sydney: standards australia.   topol, e. (2019). the topol review: preparing the healthcare workforce to deliver the digital future; an independent report on behalf of the secretary of state for health and social care. retrieved from http://topol.hee.nhs.uk/the-topol-review/   world health organization. (2019). digital health and innovation. retrieved from https://www.who.int/health-topics/digital-health   wright, a., bates, d. w., middleton, b., hongsermeier, t., kashyap, v., thomas, s. m., & sittig, d. f. (2009). creating and sharing clinical decision support content with web 2.0: issues and examples. journal of biomedical informatics, 42(2), 334-346. https://doi.org/10.1016/j.jbi.2008.09.003   appendix a   table 5 aims and methods used in the 2011 and 2016 workforce studies   health librarianship workforce and education research study (hallam et al., 2011) census of australian health libraries and health librarians (kammermann, 2016) date 2011 2016 aim determine future requirements for the australian health library workforce, and develop education framework for these needs. obtain data on characteristics of australian health library and information services (lis), and composition of their workforce. target individual australian health library and information practitioners. (managers surveyed separately.) managers of health lis, and known individual health librarians working outside traditional libraries. data collection method online survey available late february to early march 2010. census link sent to named managers and individuals; data collected october 2014 to february 2015. number of responses 161 219 responses, representing 328 health library services. response rate not stated 81%   appendix b   table 6 major findings in the 2011 and 2016 studies   health librarianship workforce and education research study (hallam et al., 2011) census of australian health libraries and health librarians (kammermann, 2016) demographic characteristics age 66% aged 40 +: 32% aged 41-50 yrs 34% aged 51-60 yrs 8% aged 61+ years not stated in detail. 36% of services had one or more staff aged 60 or more. gender female 86% male 14% 60% of services had 90% or more female staff. new graduates (qualified in past 5 years) 12% of respondents 18.2% of services had 1 or more new graduates on staff. retirement prospects 27% intend to leave sector within 5 years 36% of services had one or more staff eligible to retire within 5 years. employment characteristics area of health sector hospitals 53% govt dept 14% university 14% research body 2% other 17% hospitals 43% university 17% community org 14% professional college 3% other 23% sector status public 82% not for profit 11% private 8% other 4% no response 5% public 60% not for profit 20% private 14% geographic location 71% in capital city 25% in regional areas 75% in capital city 30% in regional areas education and professional characteristics highest formal educational award phd 2% master’s 11% grad certificate or diploma 40% honor’s 6% bachelor’s 32% topic was not included in census. had undertaken pd in past year 75% had undertaken 11 or more hours of pd in the past year. topic was not included in census. technology services and competencies currently use technology and systems 81% of individual respondents 67% of institutional respondents 42% of lis services provided technology-related support services to users. 30% managed digital repositories. 21% offered a digitization service. around one-quarter provided internet or intranet management and/or support. predicted future use of technology and systems 82% of individual respondents 69% of institutional respondents topic was not included in survey. service changes in past year topic was not included in survey. most frequently mentioned change (by 45 of 136 lis) was implementation of new software or growth in electronic resources and services.     microsoft word news4417 evidence based library and information practice 2008, 3:4 95 evidence based library and information practice news cilip training and development courses for january © 2008. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. cilip training & development are pleased to announce new training programs for 2009. note that cilip members receive up to 40% discount on all cilip training. copyright and licensing: commercial and legal 14 january 2009 gain a good understanding of copyright law and exceptions, the distinction between copying for commercial and noncommercial purpose and look at licensing as a way of expanding permissible copying. a popular half-day course that covers copyright and licensing specifically for those working in the commercial or legal sector. program highlights include: • the copyright exceptions • downloading from websites and commercial databases • copying of different types of material; business directories, annual reports • market research reports <www.cilip.org.uk/training/training/2009/co pylic/ copyrightandlicensingcommercialandlegalse ctor.htm>. advanced desk research 27 january 2009 an essential cilip training course for anyone who is required to undertake indepth desk-research. you will have the opportunity to research your own topic of interest during the day, developing your own project plan, preparing reports and presenting findings. an advanced course for researchers who want to refine their skills. program highlights include: • developing the research project plan • defining search terms and the complex use of boolean operators • indentifying specialist libraries on your field of interest • creating references and a bibliography <:www.cilip.org.uk/training/training/2009/re search/ advanceddeskresearchskills.htm> library management systems: procurement and implementation 22 january 2009 the role of the library management system (lms) in information provision is increasingly under question. many organizations are stuck with poor systems evidence based library and information practice 2008, 3:4 96 that no longer meet business requirements. program highlights include: • overview of the lms landscape • writing a specification • projecting managing the lms implementation process <www.cilip.org.uk/training/training/2009/lib info/librarymanagementsystemsprocuremen tandimplementation.htm>. setting up and delivering an information service 29 january 2009 a step-by-step lively and participative cilip training course that will help make best use of your resources as you set up or revive your existing information service. during the day you will learn how to maximise your effectiveness with time and budgetary constraints. program highlights include: • identifying your most influential users • issues to consider: copyright, licensing, data protection • are you ready to deal with enquiries? • persuading people to keep using your services • help at hand: networking and professional bodies <www.cilip.org.uk/training/training/2009/lib info/settingupanddeliveringaninformationse rvice.htm>. assertiveness for managers 15 january 2009 library and information managers are expected to work productively with a wide range of people in sometimes challenging situations. explore the use of assertiveness in management to feel more confident in work and assess your own behaviour in a variety of circumstances on this wellestablished cilip training course. program highlights include: • handling criticism and negative feedback • practical work on saying no, delegating difficult tasks • resolving conflict <www.cilip.org.uk/training/training/2009/m d/assertivenessformanagers.htm>. to browse the full selection of courses for 2009, visit the cilip web site < <www.cilip.org.uk/trainingcourses>. digitized indigenous knowledge collections can have beneficial impact on cultural identity and social ties evidence summary   digitized indigenous knowledge collections can have beneficial impact on cultural identity and social ties   a review of: liew, c. l., yeates, j., & lilley, s. c. (2021). digitized indigenous knowledge collections: impact on cultural knowledge transmission, social connections, and cultural identity. journal of the association for information science and technology, 72(12), 1575–1592. https://doi.org/10.1002/asi.24536   reviewed by: hilary bussell associate professor, research and education the ohio state university libraries columbus, ohio, united states of america email: bussell.21@osu.edu   received: 27 may 2022                                                             accepted:  20 july 2022      2022 bussell. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30179     abstract   objective – to explore the impact and significance of digitized and digital indigenous knowledge collections (d-ikc) on knowledge transmission, social connections, and cultural identity.   design – phenomenological explorative study.   setting – new zealand.   subjects – eight d-ikc users, including three academics, four undergraduate students, and one postgraduate student. six participants were women and two were men. all participants were of māori descent.   methods – eight semi-structured interviews ranging from 40 to 75 minutes were conducted in a face-to-face setting between june 2019 and august 2020. participants were recruited through the researchers’ personal and professional networks using a purposeful sampling technique. potential participants were provided with a copy of the interview guide during recruitment.   main results – the article reports on seven areas of results: use of collections, accessibility and discoverability, collection features and functionality, sharing of knowledge resources, reuse and repurposing of resources, perceived benefits of cultural and social connections, and development and provision of d-ikc. participants use d-ikc for academic work including coursework, teaching, and research as well as for personal interest and development, such as researching whakapapa (genealogy) and whenua (land) information, language revitalization projects, and creative works. all participants expressed preference for online access to the collections. participants discussed barriers to access not only for themselves but also for other members of their community, including difficulty using the platforms on mobile devices, lack of awareness about the collections, inadequate digital access, and lack of digital competence for searching and navigation. some participants noted inaccuracies in transcriptions that could lead to alteration of the meaning of words and deter engagement with d-ikc. all participants reported having shared knowledge resources they encountered in digitized collections. primary reasons for sharing information included helping classmates get access to educational materials and sharing resources with whānau (extended family) for genealogical research and land claims. common reasons for reusing or repurposing materials included language and dialect revitalization and creative work and performance. participants said they were more likely to share materials related to their tribal affiliation. participants also discussed information that would not be appropriate to share, such as information that is considered tapu (sacred), particularly if the material is outside of their tribal roots. notably, all participants said they had come across resources and information in d-ikc that should not be openly accessible at all. participants reported having gained linguistic and cultural knowledge as well as information about their cultural identity through their use of d-ikc. sharing this knowledge with their communities has helped strengthen social connections. some participants noted that their hapū (subtribe) planned to set up their own digital archives.   conclusion – overall, d-ikc can have a beneficial impact on individual and collective social identity and social ties. making these materials available online facilitates their wider access and use. however, memory institutions (mis) need to take steps to ensure that cultural values and knowledge are embedded into the development and stewardship of the collections. mis should employ more specialists from indigenous communities with deep understanding of customary practices and principles, encourage other staff to develop their understanding of the language and customs of the indigenous communities that their collections are rooted in, and develop partnerships with indigenous authorities to help guide them on issues relating to sacred knowledge and genealogical materials. the authors also recommend that mis develop outreach programs to raise awareness of the resources and to improve digital access and competencies.   commentary   this article contributes to the body of research on the role that cultural heritage institutions can play in the development of cultural identity, knowledge transmission, and social connections. specifically, the authors build on previous research into the development of knowledge sharing relationships through the use of digitized māori language collections (crookston et al., 2016) by exploring users’ perspectives, experiences, and expectations in engaging with these collections. based on this research, the researchers provide insights into how these collections can lead to knowledge sharing outside of an institutional context; they also articulate a number of considerations to guide mis in the provision of indigenous knowledge collections.   this study was evaluated using guidance from kuper et al. (2008). the article is particularly strong in the level of detail it provides in its methodology section and supporting data. the authors include a paragraph that addresses reflexivity by describing their positionalities as researchers. reflexivity is an important concept within qualitative research that explores the influence a researcher brings to the research process by acknowledging how their “gender, ethnic background, profession, and social status influence the choices made within the study” (liew, yeates, & lilley, 2021, p. 689). the authors describe how they brought a combination of “insider and outsider perspectives” to their collaboration in terms of their sociocultural identities, topical knowledge, and roles within the research study. important to note is that at least one of the authors is of māori descent while another is not; they note that the third author has expertise in kaupapa māori (māori principles and values), but they do not state whether he is of māori descent. the methods for data collection and analysis are described thoroughly. in addition to drawing on concepts from their literature review and several value-impact frameworks for assessing digital cultural heritage collections, the authors analyzed their data using kaupapa māori to make sure that specific cultural issues were not elided. the authors draw on these concepts throughout the article and provide a glossary of māori terms in an appendix. additional supporting data include participant profiles and summaries of key findings mapped to value-impact frameworks and traditional cultural values.   one area where additional details would have been welcome is in the discussion of sampling. the authors state that participants were recruited through personal and professional networks of the researchers, using a purposeful sampling technique. all of the participants were university students or academics, leading the reader to assume that university affiliation was one of the criteria; however, this is not stated outright, and what other inclusion criteria were used is unclear.   as the authors note, given the qualitative nature of this study, the findings are not meant to be generalizable to a larger population. nevertheless, they provide a number of recommendations and insights that will be relevant to mis that maintain indigenous cultural knowledge collections. these considerations are particularly important in light of the fact that the principle of open access to information resources can be in conflict with indigenous knowledge sharing paradigms (underhill, 2006) and given the destructive role that archives, libraries, and other mis have and continue to play through colonial collecting practices (christen, 2015).   references   christen, k. (2015). tribal archives, traditional knowledge, and local contexts: why the “s” matters. journal of western archives, 6(1), 3. https://doi.org/10.26077/78d5-47cf   crookston, m., oliver, g., tikao, a., diamond, p., liew, c.l., & douglas, s.-l. (2016). kōrero kitea: ngā hua o te whakamamatitanga/the impacts of digitised te reo archival collections. https://interparestrust.org/assets/public/dissemination/korerokiteareport_final.pdf   kuper, a., lingard, l., & levinson, w. (2008). critically appraising qualitative research. bmj, 337, 687–689. https://doi.org/10.1136/bmj.a1035   liew, c. l., yeates, j., & lilley, s. c. (2021). digitized indigenous knowledge collections: impact on cultural knowledge transmission, social connections, and cultural identity. journal of the association for information science and technology, 72(12), 1575–1592. https://doi.org/10.1002/asi.24536   underhill, k. j. (2006). protocols for native american archival materials. rbm: a journal of rare books, manuscripts, and cultural heritage, 7(2), 134–145. https://doi.org/10.5860/rbm.7.2.267   microsoft word es_brown.doc evidence based library and information practice 2006, 1:1  101 evidence based library and information practice       evidence summary    training may affect primary care staff access to the biomedical electronic evidence  base    a review of:  doney, liz, helen barlow, and joe west. “use of libraries and electronic information resources  by primary care staff: outcomes from a survey.” health information and libraries journal  22.3 (september 2005): 182‐188.     reviewed by:   marcy l. brown  clinical medical librarian, the western pennsylvania hospital, forbes regional campus  monroeville, pennsylvania, united states of america  e‐mail: wordsmith@alltel.net      received: 14 december 2005  accepted: 14 february 2006      © 2006 brown. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to assess use of existing local  libraries, the internet, and biomedical  databases by primary care staff prior to  implementation of the primary care  knowledge management projects.  additionally, to assess the need to train  primary care staff to use the internet and  biomedical databases.     design – cross‐sectional postal  questionnaire survey.    setting – nottingham and rotherham, two  cities in the trent region of the uk.    subjects – questionnaires were analyzed  from 243 general practitioners, practice  nurses, and practice managers in four  nottingham primary care trusts as well as  practices in the rotherham health authority  area.     methods – questionnaires and cover letters  were sent between may 2001 and february  2002. to encourage response, a postage‐paid  envelope was enclosed. a total of 709  questionnaires were sent in nottingham,  and 169 were returned for a response rate of  24%. in rotherham, 179 questionnaires were  sent and 61 returned, for a 34% response  rate. thirteen responses from a may 2001  pilot in rotherham were also included in the  data analysis. survey questions included a  variety of formats, including tick boxes and  open‐ended questions. data was entered  into an access database and analysis was  performed using stata software.     main results – reported use of libraries was  low overall, with only 30% of respondents  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:1  102 claiming to have used library facilities.  however, there was significant variation  among professional groups. practice nurses  (pns) had significantly higher usage of  libraries than general practitioners (gps)  and practice managers (p < 0.01). overall,  81% of the respondents used the internet for  work, with no significant variation by group.  forty‐four percent reported using  biomedical databases, with some significant  variation. gps and pns reported higher  usage of databases than practice managers  (p < 0.01). the most common reported  barrier to using both the internet and  biomedical databases was lack of training.  gps more frequently cited lack of time as a  barrier to using biomedical databases (p =  0.04). over half of all respondents reported  an interest in internet training, and over  60% reported an interest in database search  training. a significantly lower number of  practice managers wanted database training  (p = 0.02).    conclusion – based on the results of this  admittedly small study, additional training  is needed – and desired – by primary care  staff in both nottingham and rotherham.  developing and offering training in internet  searching and evaluation as well as use of  the biomedical databases is one important  way in which libraries can build  partnerships with primary care practitioners.  this will also enable added numbers of  primary care staff to access and use the  clinical evidence knowledge base.  additional studies are needed to identify  and overcome barriers to training.    commentary      this survey reinforces the conclusions of  similar studies (bellman et al.; cogdill et al.;  dawes and sampson) about the use of  knowledge‐based resources by physicians;  namely, that time and training are barriers  to internet and database usage. unlike the  other studies, lack of training was a larger  factor than lack of time. this may result  from the inclusion of groups other than  physicians in this study, or it may represent  a significant finding. regardless, this is an  outcome on which further research may  shed some light. training requires time. if a  respondent does not currently use libraries,  would she actually attend a library‐ sponsored training session?     the survey itself was not provided, and the  actual questions were not reproduced  within the article. therefore, the instrument  could not be evaluated. we do not know if  respondents were asked about public  libraries, health services libraries, or  whether a distinction was made within the  survey. we also do not know how library  “use” was defined, if at all. lack of question  clarity could skew responses about library  use. it was also difficult to evaluate  responses related to internet access without  viewing the survey questions. if  respondents were not asked about internet  use at home, but do use it at home for work‐ related purposes, would this have an effect  upon responses related to access?    the study response rate was low, which  mirrors physician response rates in other  postal surveys. the fact that these  practitioners responded may mean that they  are more interested in informatics issues  than the average gp or practice nurse. the  study authors do acknowledge this possible  response bias, which indicate that  conclusions about the need and desire for  training might not apply to the entire  primary care population of nottingham and  rotherham.     this study presents a fairly complete  snapshot of electronic resource use by gps,  practice nurses, and practice managers in  the trent region. it also reinforces the  findings of previous studies by outlining  barriers to using electronic resources.  however, the study neglects to explore one  evidence based library and information practice 2006, 1:1  103 of its three objectives: use of existing local  libraries. do local libraries currently offer  internet and database training? if so, are  primary care practitioners aware of these  training opportunities? this is a crucial area  for future research in this region, as well as  for any library that offers resources and  services to a clinical population.    works cited    bellman, philip, et al. “facilitating physician  access to medical reference  information.” the permanente journal  9.4 (fall 2005): 27‐32.                                                                     cogdill, keith w., et al. “information needs  and information seeking in community  medical education.” academic medicine  75.5 (may 2000): 484‐86.    dawes, martin, and uchechukwu sampson.  “knowledge management in clinical  practice: a systematic review of  information seeking behavior in  physicians.” international journal of  medical informatics 71.1 (august 2003):  9‐15.  microsoft word article_wright_10pt font evidence based library and information practice 2008, 3:3 3 evidence based library and information practice article three evidence based methods to compensate for a lack of subject background when ordering chemistry monographs robert a. wright manager tuttle neighborhood library, houston public library houston, texas, united states of america email: robert.wright@cityofhouston.net received : 31 january 2008 accepted : 06 august 2008 © 2008 wright. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – the aim of this article is to present evidence based methods for the selection of chemistry monographs, particularly for librarians lacking a background in chemistry. these methods will be described in detail, their practical application illustrated, and their efficacy tested by analyzing circulation data. methods – two hundred and ninety-five chemistry monographs were selected between 2005 and 2007 using rigorously-applied evidence based methods involving the library's integrated library system (ils), google, and scifinder scholar. the average circulation rate of this group of monographs was compared to the average circulation rate of 254 chemistry monographs selected between 2002 and 2004 when the methods were not used or were in an incomplete state of development. results – circulations/month were on average 9% greater in the cohort of monographs selected with the rigorously-applied evidence based methods. further statistical analysis, however, finds that this result can not be attributed to the different application of these methods. conclusion – the methods discussed in this article appear to provide an evidence base for the selection of chemistry monographs, but their application does not change circulation rates in a statistically significant way. further research is needed evidence based library and information practice 2008, 3:3 4 to determine if this lack of statistical significance is real or a product of the organic development and application of these methods over time, making definitive comparisons difficult. introduction it is often the case that the possession of a science degree of any kind leads librarians to subject responsibilities of a broadly scientific or technical nature. for example, a librarian with a bachelor’s degree in geology might be charged with subject responsibilities for physics or engineering in addition to the geosciences. in the author’s case, he collected materials for pharmacy and chemistry, areas of study related in different, incomplete ways to biology, the degree he earned as an undergraduate in 1988 and applied in research settings for 5 years after that. pharmacy, chemistry, and biology as fields of knowledge certainly all describe natural phenomena from a molecular standpoint. for instance, drug-receptor interactions, chemical reactions, and the processes of dna transcription and translation all occur at this level. but this foundation for understanding is only useful for making broad, limited connections among these subjects. the knowledge that molecules are the common actors is a far cry from understanding the intricacies of the clinical pharmacology of cancer drugs, chemical vapor deposition polymerization, or dna damage and repair. while librarians need not know as much as graduate students or research faculty about a subject, a rigorous academic background ending in a degree would seem an ideal preparation for serving their information needs. such a degree, whether at the bachelor, master, or ph.d. level, enables the librarian to share an essential intellectual landscape with his patrons. this landscape consists of key signposts or terminology, central ideas or organizing principles, distinct conceptual geographies or subcategories, and the palpable interconnections between all these features. the aim of this paper is to add to the existing set of techniques for collection building for those librarians who are responsible for acquiring chemistry monographs without the benefit of a chemistry degree. this paper presents a case study of the experience of one librarian collecting in one subject area at a particular university. these techniques, however, are also generally applicable to subject librarians with minimal content-specific educational background, regardless of the field for which selection choices must be made. literature review of general collection building methods the scope and depth of coverage of collection development and management in the library science literature is significant. a number of monographs published in the past ten years address general principles, guidelines, and universally-applicable best practices (gorman and miller; jenkins and morley; clayton and gorman; mack; johnson; evans and saponaro). a further set of recent monographs relates to collection building in a variety of specific subject areas, including the humanities (owens); the health sciences (richards and eakin); literature in english (day and wortman); music (maple and morrow; mccants); area studies (hazen and spohrer); and business (bergart and lewis). finally, the literature also includes reproductions of collection development policies (sylvia; mcabee, bevis, poe, and evidence based library and information practice 2008, 3:3 5 whitesel). these policies typically focus on deselection criteria and the appropriate types and formats of materials to collect. they are meant to offer general guidelines, and so do not address the kinds of questions facing subject selectors in their day-to-day work, such as, “should the acquisition of books on organometallic chemistry be given more priority than the acquisition of books on electrochemistry?” research which evaluates collections with materials-based and usage-based criteria offers broadly applicable methods to identify collection weaknesses and future acquisitions. materials-based research compares collections to an exemplary collection or to lists of titles obtained from respected, representative sources, such as textbook bibliographies (stelk and lancaster) or citations from teachers’ guides (ching and chennupati). usage-based research examines circulation data from existing collections to determine broad areas of patron use. these studies range from global examinations of the use of collections over time (montgomery, bulick, fetterman, and kent; burrell; kao, chang, and lin); to evaluations of recent usage of monographs on general subjects (fenske; kraemer); to the use of interlibrary loan statistics to show collection weaknesses (khalil). another broadly-applicable area of research in collection development and management relates to the development of expert systems or decision support systems. this research seeks to build systems that can standardize and automate best practices in order to streamline the selection process and improve its output (sowell; johnston and weckert; uzoka and ijatuyi). these systems are either weak at incorporating subject content into the decision-making process or fail to include actual usage data. literature review of collection building methods applied to chemistry sources in the literature that specifically address the challenges of chemistry monograph selection can be grouped for convenience into two categories. the first of these sources are lists of recommended titles which can be based on article length (sapp; culp; rudman) or book length (bottle and rowland; maizell; douville) and feature annotations of recent works in a variety of categories. titles, authors, prices, and isbns are provided to aid the selector. these lists evaluate and recommend specific titles which are topical, current, and essential. as such, they are quite helpful. their scope is limited, however, and by their nature they are decontextualized. because of their “top ten” nature, these lists feature books that would be appropriate for any academic library with a chemistry collection. a second category features sources that address context-specific strategies for selection. these include selection based on textbook citations (powell), citations from faculty-published scholarship (farina and snyder 146), direct consultation with faculty (barnett), and the use of approval plans (farina and snyder 141-42). selection based on textbook citations reflects the recommended readings that both faculty and enterprising students will encounter in the course of study and teaching. these readings, however, are likely to be much more useful for faculty instruction than for faculty research. on the other hand, selection based on citations from faculty scholarship could be quite valuable for faculty research. monographs, however, are not frequently cited in the chemistry research literature (barnett). direct consultation with faculty members could ensure that their instructional and research needs for monographs are met. evidence based library and information practice 2008, 3:3 6 only a few faculty members, however, respond to requests for selection suggestions, and they are often unaware of the holdings at the university of houston libraries (library). as well, suggestions from faculty are often already part of the library’s collection. approval plans are arrangements that libraries make with vendors for the provision of recently published books, as well as electronic or printed descriptions (called “forms”) of current or forthcoming titles. books that fit an agreed-upon set of criteria are automatically sent by the vendor to the library for review. the books are then either selected by subject librarians for inclusion in the library’s collection or sent back to the vendor. subject selectors likewise examine the electronic or printed forms and decide either to discard them or use them as a means for ordering the books they describe. the library uses blackwell’s book services as its approval plan vendor. approval plans offer the advantage of timely, automatic delivery of monographs and monograph descriptions based on a specific set of criteria. sets of criteria form profiles of subjects such as chemistry. the selection derived from an approval plan profile depends upon the degree to which the profile can be calibrated for specificity by the selector. for instance, the vendor may use the criterion “electrochemistry” when sending books or forms to a library. for a library serving patrons that are most interested in the electrochemistry of polymers, this criterion will often prove too general and result in books and forms being sent which are of marginal value to this library. aims as discussed above, a biology degree with minimal chemistry course work offers limited fluency in the lexicon of chemistry and only a basic understanding of its underlying principles. fluency is greater in areas of interdisciplinary study such as biochemistry, biomolecular science, environmental science, and medicinal chemistry, but the bulk of chemical specialties are beyond this level of fluency. a limited acquisitions budget makes this large gap in knowledge a serious difficulty. for without the ability to buy every currently published chemistry monograph, actual purchasing choices must be made from an array of sources, including: approval plans, choice reviews, and publishers’ catalogs. informing these choices are the questions, “will these books be relevant to the needs of faculty and students?” and “what evidence can i offer in support of relevance?” three methodologies are presented below that provide evidence to make these choices more relevant. these methods will be described in detail, their practical application illustrated, and their efficacy tested by analyzing circulation data. methods the integrated library system (ils) method the integrated library system (ils) method refers to the strategic use of ils-derived, title-level circulation statistics for making collection decisions. the ils referred to is millennium, version 6.0 from innovative interfaces, inc., this being the one used at the library. other ilss likely have the same functionality. the search/holds feature in the millennium circulation module searches the library’s holdings using a variety of fields. the title, author, and keyword fields may be searched as needed depending on the level of specificity required for an evidence based selection decision. a title search may reveal an earlier edition in the holdings. the circulation information for this edition, evidence based library and information practice 2008, 3:3 7 including the checked out, renewed, and last out fields in its millennium record provides sufficient evidence upon which to base the purchase of the newer edition. a new edition of a monograph is ordered if the old edition has been checked out an average of once/year or has been checked out in the past two years. these decisions can be justified in the first case because of the book’s long-term use and in the second case because of a recent spike in use. for first edition titles, one or more keyword searches are performed to determine the relevance of the book. poor circulation of recent subject-related titles indicates that the book in question is of marginal value to the collection. recent circulation of old titles indicates the need for more recent titles, so the prospective purchase would be made in this case. this method was used to select handbook of size exclusion chromatography and related techniques. a search of the ils revealed a heavily-used previous edition of this title, which had circulated ten times since 1995 and was currently checked out. a search of the ils is an ideal starting point for prospective titles that are also subsequent editions. it can often reveal the performance of earlier editions, which is sufficient evidence for a selection decision. a clear advantage of this method of making selection decisions is that, assuming the ils software is functioning properly, the selector has evidence that books similar to the one he is considering for purchase either have or have not been checked out by patrons from his library. circulation statistics from these related books are akin to results of experiments in which books were placed before patrons and a specific response was measured (i.e. did patrons check them out or not?). this evidence provides some certainty that the experiments the selector sets into motion with his new acquisitions will yield the same results, and that he was justified in not performing some experiments at all. this method is not necessarily effective at identifying titles that, if purchased, would fill a new or re-emerging pedagogical or research need. this may be because patrons choose not to check out books that are topically relevant but are judged by them to be out-dated. also, patrons may not check out books on a given topic because no such books exist in the collection. the ils method might help in some of these cases if the subject matter of existing books in the collection is closely related to that of the book in question, since then circulation data are available. but for monographs whose topics are historically new or only distantly related to topics covered by the existing collection, the ils approach will not be helpful. in this case, testing for the utility of prospective purchases requires different methods. the google-mediated method google-mediated searching of faculty and departmental web pages uses google’s advanced search feature. key terms taken from book titles, descriptions in publishers’ catalogs, descriptors from approval plan forms, etc. are entered as phrases or individual words. the search is then limited to the departmental domain name. in this case, “chem.uh.edu” is used to limit the search to the department of chemistry’s website at the university of houston. the results of this kind of search vary. they may include course syllabi, learning outcomes, powerpoint presentations, descriptions of research interests on faculty members’ departmental and personal homepages, e-prints, post-prints, bibliographies, and curricula vitae. the first edition of a book published in 2003, entitled computational materials science of polymers, was selected using this method. a evidence based library and information practice 2008, 3:3 8 search was performed using the terms “computational,” “materials science,” and “polymer.” the ten results were a variety of documents related to the uh department of chemistry, including: one syllabus, one set of course notes, one description of the undergraduate program, two faculty web pages describing research interests, and five full-text journal articles in pdf format maintained by faculty authors on their web pages. in the majority of these cases, the search terms lacked sufficient proximity to one another to make the results relevant. a typical example of unrelated terms found by the search above is the “course notes” result, which summarizes the occurrence of the terms in bold in its text as “…b. you do not have to read all of this material. …see science, 266, 1359 (1994…vii) a nice computational example is provided by the reaction of ketene (ch 2=c=o …” the result that did provide the needed evidence was one of the faculty web pages outlining research activity. on this web page, which has since been updated, is a numbered list of research interests. among these topics were two of particular importance, captured in the relevant summary of the site as “… (6) development of computer simulation methodology for material science and biotechnology. (7) polymer correlations in composite materials. …” (pettitt). the favorable impression given by this summary was confirmed by a visit to the site itself, which made clear that this faculty member’s research involves computational methods applied to polymeric materials. the book was ordered on the strength of this evidence. this method has the virtue of connecting directly to faculty research and teaching interests, which, in the case of faculty homepages, are often articulated in descriptive paragraphs and overviews. these narratives provide scope and context for faculty research. since they are written at a more general level than, for example, a journal article abstract, they address broader concepts, are dense with key terms and phrases, and make connections between concepts or fields of study. this means that these narratives are more easily comprehensible to the uninitiated, including librarians with little or no subject background. this also means that these text and content rich pages are good candidates for keyword searching by search engines such as google. while google-mediated searching of a chemistry department’s web pages can yield a considerable amount of unique information upon which to base monograph acquisition decisions, this information may be both incomplete and out-of-date. these problems are related to the decentralized nature of web authoring on the uh department of chemistry’s web site (chemistry). much of the content on this site is maintained by individual faculty members. as one might expect of a site run by busy researchers and administrators, faculty-specific content may not always be kept updated. to test this assumption, the faculty pages listed at <http://www.chem.uh.edu/faculty/> were examined, excluding those pages from adjunct and joint faculty, emeritus faculty, and visiting professors and lecturers. these pages were excluded because their owners were either part of another department with different web-authoring norms (joint faculty); were less professionally active (emeritus faculty); or were, due to affiliation, perhaps less likely to actively maintain their sites (adjunct faculty, visiting professors, and lecturers). while no “last updated” or “last modified” date is part of these pages, dated information listed in both the “honors, fellowships, etc.” and the “recent publications” sections of these sites evidence based library and information practice 2008, 3:3 9 was assumed to be a reliable measure of when the pages were last updated. the results of this analysis illustrate the varying levels of attention faculty authors pay to their sites. as of july 13, 2007, 50% (13) of the faculty pages had last been updated five years ago or more and only 15% (4) had been updated in 2007. see table 1: faculty web page updating in the uh department of chemistry. to confirm that a substantial number of these web pages were out-of-date, author searches were performed on july 18, 2007 in scifinder scholar, a comprehensive database of the chemical literature. these searches showed that a great deal of content in the form of published article titles was missing from faculty web pages. after comparing the publication lists from these pages with the scifinder scholar search results, it was found that only 19% (5) of them were up-to-date and reflected the actual published output of faculty members. the scifinder scholar method searching scifinder scholar (scholar) was the third method used for collection building. keywords derived from book descriptions were entered in the “research topic” search of the “explore” feature of scholar, which accepts natural language queries and provides users with different result sets based on different combinations of key terms. the results were limited before the fact by entering “department of chemistry, university of houston” in the “company name” filter. the results obtained by this method included journal articles, abstracts from conference proceedings, books, and patents published by current and former uh department of chemistry faculty members. a portion of these results were irrelevant because they are products of institutions having “houston” in their titles or addresses. it is unclear how to avoid retrieving these irrelevant results. result sets were typically less than 20 items, depending upon the specificity of the search terms. table 1 faculty web page updating in the uh department of chemistry year of the most recent web page update number of faculty web pages updated (26 total) percentage of faculty web pages updated number of faculty with most recent publications in a given year number of up-to-date faculty web pages 2007 4 15% 20 4 2006 6 23% 3 0 2004 2 8% 2 1 2003 1 4% 1 0 2002 1 4% 0 2001 5 19% 0 2000 7 27% 0 evidence based library and information practice 2008, 3:3 10 evidence for ordering the book, peroxides at the beginning of the third millennium: synthesis, properties, applications came from searching scholar using the term “peroxides.” this produced 29 results. sixteen of these results, more than half of the total, were not published by uh faculty. this is not an uncommon outcome with the scholar method, since it searches the institutional name and address by keyword. the unwanted results came from rice university (in houston, texas) and sam houston state university. more useful, but still not exactly addressing the target audience, were seven articles published by members of the uh department of chemical engineering. these intrainstitutional results are valuable as secondary justifications for monograph purchases, particularly if there are frequent collaborations between certain departments or if faculty members have joint departmental appointments. the remaining six results were authored by faculty in the uh department of chemistry and addressed in some part the application of peroxides as reagents in a variety of chemical reactions. this record of scholarly output was sufficient justification for the purchase of the monograph. the searches in scholar illustrate one of this resource’s primary strengths, its timeliness. since scholar is continually updated, searching with this database is the method best suited to probing the agreement between a book’s description and the most recent published chemical literature. in addition to timeliness, scholar is comprehensive. it covers the chemical literature back to the mid-1800s, with ~9,500 journals indexed (“scifinder for academics”). it also covers scientific conferences and meetings, medline for biomedical and biochemical searching, and patents from u.s. and foreign patent offices. the thoroughness of scholar’s coverage makes it highly likely that monograph descriptors used as search terms will retrieve the corresponding faculty-authored literature, assuming it exists. scholar’s comprehensiveness has value in other ways. through patent results, scholar has the virtue of revealing commercial applications of faculty research and thus providing connections between chemistry, technology, and engineering. by including abstracts from conferences and meetings, scholar may reveal areas of research that are just beginning to be explored and presented by faculty. this information can make selection decisions more proactive and make monograph collections more sensitive to trends in research. with this said, a disadvantage of this method may lie in relying too much on or overstating the value of scholar searches for determining emerging areas of research interest. no matter how timely, scholar is still a record of published work. while papers or posters presented at conferences may describe research in a more preliminary and provisional way than journal articles, they are nevertheless at a level of organization and development to be made part of the scientific record. for information about research interests that have yet to be expressed in publication, faculty web pages may be a better source, if they have been updated. direct communication with faculty members would also be invaluable in this regard. method of analyzing circulation statistics to support the claim that these three methodologies can build a collection of local value, the circulation statistics of chemistry monographs ordered from 2002 to 2007 were analyzed. this time period begins with the author’s first year of making monograph purchases at the library and evidence based library and information practice 2008, 3:3 11 ends two years after the author’s coordinated application of his three collection building methods. these methods were developed organically over time, so that prior to 2005, they were being used in a limited fashion for book selection. the year 2005 is nevertheless an important point of demarcation because it was at the beginning of this year that the methods were applied for the first time in a rigorous and comprehensive manner for selection decisions. this time marker offers a convenient point around which the impact of the methods on the circulation of selected monographs can be measured. consequently, data about monograph selections were arranged in two tables in appendix 1: chemistry monograph orders 2002-2004 (excluding approval orders and added copy orders) and chemistry monograph orders 20052007 (excluding approval orders and added copy orders) (see separate file). the first table shows monographs selected without rigorous application of the methods (2002-2004), and the second table shows those selected with rigorous application of them (2005-2007). the circulation rates of these two groups were then compared to determine if the consistent use of the methods resulted in a higher rate of circulation. the 2002-2004 and the 2005-2007 tables represent the circulation of all the chemistry monographs ordered for the library during these time frames, 254 and 295 monographs respectively. the following types of monographic orders were excluded: • approval orders these are monographs that are sent automatically from blackwell’s book services because they fit a relatively broad descriptive profile. they are often also judged by the author using the methods before acceptance or rejection. these monographs are not included, however, because their physical presence may engender an acceptance bias. • added copy orders these are excluded because the methods were not involved in their selection. they were selected based on their appearance on lists of highly-used monographs. • orders for monographs that are non-circulating internal-use statistics could be used for these books, but these data are only gathered periodically. so, this circulation data is excluded. • orders that have been cancelled, have not yet been received, have been withdrawn, are being processed, or are missing these orders have no circulation information. • orders to fill a patron suggestion or to alert a specific patron about a particular monograph in both of these cases, a patron is notified about the book and the book is held for them. unlike the other selected books, these have inducements to circulation, so their circulation data is excluded. the following data accompany the monographic information in the tables in appendix 1 (see separate file): • order date • received date • date on shelf (estimated to be approximately one month after receipt) • circulations (excluding renewals) as of may 1, 2008 • circulations/month as of the date on the shelf the order date, received date, and circulations were all taken from the records for monographs in millennium. the date on shelf was calculated by adding a month to evidence based library and information practice 2008, 3:3 12 the received date and rounding to the nearest month. the roughly month-long time frame for processing received monographs to shelf-readiness was arrived at through consultation with technical services staff at the library. the circulations/month figure was calculated by dividing the number of circulations as of may 1, 2008 by the number of months elapsed between the date on shelf and may 1, 2008. results when compared, the 2005-2007 cohort of monographs showed an increase in its average monthly rate of circulation versus the 2002-2004 cohort. the 2005-2007 average rate of circulation was 0.058 circulations/month, while the 2002-2004 average rate was 0.053 circulations/month. see below for table 2: statistical features of the 2002-2004 and the 2005-2007 monograph cohorts. this appears to represent an increase of approximately 9%, with the numbers of circulations/month on average 9% greater in the cohort using the evidence based methods in a consistent manner. however, when these two cohorts are considered as two large independent samples and standard statistical analysis is performed, this apparent increase in circulation is seen to be due to natural variations in rates of circulation among the two samples. in other words, there is no statistical evidence that the observed difference in rates of circulation among the 2002-2004 and the 2005-2007 cohorts is attributable to an increasingly rigorous application of the evidence based methods described here. in order for the difference in circulation between the two cohorts to be attributable to differences in monograph selection techniques, the test statistic 2 2 2 1 2 1 0 n s n s dyx z + −− = must have a value less than − 2.33 (stephens 213-14). in this formula, x and y are the means of two populations, in this case the average circulations/month of each cohort. 0 d represents the null hypothesis, in this case the state in which the average monthly circulation of the two cohorts is the same. this is the situation which results when the rigor with which the monograph selection methods are applied has no impact on circulation. the symbols s1 and s2 are the standard deviations of samples from the two cohorts. the symbols n1 and n2 correspond to the sizes of the samples drawn from the two cohorts for analysis. in this case, each sample size equals 30. the value of z derived from the data found in appendix 1 is − 0.294, considerably more than − 2.33, as can be seen below in table 2: statistical features of the 2002-2004 and the 2005-2007 monograph cohorts. discussion the methodologies for decision-making discussed in this article are three possible table 2 statistical features of the 2002-2004 and the 2005-2007 monograph cohorts 2002-2004 cohort 2005-2007 cohort average monthly circulation (mean) 0.053 0.058 standard deviation 0.058 0.080 sample size analyzed 30 30 test statistic (z) − 0.294 evidence based library and information practice 2008, 3:3 13 approaches to building a collection of chemistry monographs. other methodologies are conceivable, and a number have been used to a limited degree, such as: selection by approval plan, selection via patron suggestions, selection by publisher, and imitative selection based on another institution’s collection. while these strategies were once considered sufficient on their own terms, they are now validated by using the ils, google, or scifinder scholar methods. for instance, chemistry books that come on approval are not accepted until evidence has been identified in the ils, online, or through scholar for their future use. for while the fact that a book has been selected by an approval plan is itself a form of evidence, given that it must fit a certain profile of desired material, it may be judged as insufficient, since this profile is sometimes overly-broad. a judgment of insufficiency may also apply to selection by patron suggestion, since individual interest may not reflect institutional interest; to selection by publisher, since not everything by an essential publisher will be locally relevant; and to selection by imitation, since no two institutions have identical sets of research and teaching interests. regarding questions of sufficiency related to the three methods, it is fair to say that each method has somewhat different types of evidence and therefore different ways of measuring what amount of evidence is sufficient. for the ils approach, the evidence reflects actual monograph usage in a local context. as a result, this method is the most directly suited to answering the question, “if i purchase this book, will anyone check it out?” in some situations, making a decision is very straightforward, as in the case of a new edition of a monograph whose earlier edition has circulated well according to the ils. other situations are less clear, as when no earlier edition exists and keyword searching reveals related titles with no clear pattern of circulation. investigation of this title using one of the other two methods may be warranted in order to establish a solid case for or against a purchase. in the case of the google-mediated method, results vary widely in their utility as evidence. they must be investigated carefully to determine the extent of their conceptual match to a monographic subject as opposed to simply a keyword match. given that the use of all these methods reflects to some extent an absence of conceptual understanding of chemistry, it would seem that making this kind of distinction among results would be difficult. in practice, however, it is relatively straightforward and rests largely on both the proximity of terms in the results and their context within larger documents. spurious results can be easy to identify owing to the occurrence of search terms at widely dispersed points in texts. in useful results, by contrast, the search terms occur in close proximity to one another and tend to be in parts of texts that serve summarizing functions, such as titles, abstracts, or introductions. in cases in which the significance of search terms within the larger text is ambiguous, the clarifying use of one or both of the other methods is recommended. scifinder scholar results have more certain relevance because they are products of matches between search terms and a variety of summarizing elements in document records, including titles, abstracts, and index terms. the locations of terms in these results should be studied carefully, however, because they may co-occur in a way that makes them unrelated to one another or causes them to have different meanings. also, if search words are elements or compounds, these may match with index terms because of their roles in reactions that are tangential to or irrelevant to the subjects of the monographs being evidence based library and information practice 2008, 3:3 14 investigated. it should also be noted that indexing does not exist for every record in scholar, and so potentially relevant connections may be missed. in this case, the google method, which searches full text, might be employed. because of the limitations of these three methods, they are frequently used together until the ambiguity of a situation is resolved or until all avenues for evidence-gathering have been exhausted. the methods are typically used sequentially in this order: ils, google, and scholar. with regard to the analysis of circulation statistics, a monthly rate of circulation was used because monographs were compared that were ordered at widely different times. the circulations taken from millennium records show how often a book has been checked out by patrons. when comparing monographs that arrived on the shelf several years apart, these circulation numbers will tend to be higher for monographs that have been on the shelf longer. this comparison may give a false advantage based on longevity, since a younger monograph might circulate as much or more than an older monograph, given time. calculating the rates of circulation can overcome this longevity bias. for instance, a book on the shelf on may 2002 that has circulated eight times has a higher number of circulations than a book on the shelf on may 2006 that has only circulated four times. however, the 2006 book has a higher rate of circulation/month (4/24 = 0.167) than the 2002 book (8/72 = 0.111), as of may 1, 2008. conversely, the calculation of circulation rates may introduce a regularity bias by falsely assuming that books will circulate at a constant rate. given that the 2002-2004 and 2005-2007 cohorts of monographs are close in age and young in relation to the may 1, 2008 reference point, however, it seems likely that their average rates of circulation will remain steady for some time. this assumes that library patrons will be guided in their use of all these monographs in the near future primarily by subject-fit with their information needs and not by dates of publication. this also assumes that the information needs of patrons will not as a group change radically in a short time. during the six-year course of this case study, the department of chemistry has consistently maintained its major programs of research. a more convincing test of a long-term increase in circulation attributable to the use of the methods would involve revisiting the two cohorts at the effective end of the useful life of their monographs. circulation statistics of the cohort monographs could be compared when the majority of them have reached obsolescence, in perhaps another five to ten years. conclusion the methods outlined above were developed over time by a monograph selector seeking to compensate for a lack of academic background in chemistry. as such, these strategies are necessarily influenced by and flow from the selector’s particular experiences. so while these methodologies are rational and hopefully of value to others, they do not encompass the universe of approaches to this problem. other strategies could be employed, including chemistry coursework, extensive dialog with faculty members, reading chemistry textbooks, etc. the methods are admittedly reactive and somewhat inefficient, but they have formed a trusted and effective means of both addressing selection uncertainty and building a collection of local value. the three methods may thus be considered supplements to evidence based library and information practice 2008, 3:3 15 other approaches, an echoing of what librarians may already be doing in a modified way, or a process already superseded by better methods. in fact, the present study may perhaps best be viewed as an installment in the education of a chemistry librarian. the ils, google-mediated, and scholar methods may be considered evidence based in two senses. in the first sense, these are methods which require the inputs of locallyrelevant evidence, whether in the form of circulation statistics, faculty web pages, or citations of departmental research. in the second sense, however, there is no statistical evidence that the use of these methods to select monographs increases their circulation. following the 2002-2004 and 2005-2007 monographs through the lifetime of their usage, to a point when their members have for the most part stopped circulating, may change this picture and show a statistically significant effect of these methods. another test of the effectiveness of these methods to increase circulation of chemistry monographs would involve a more controlled adoption of them. the results of this research are complicated and to some extent undermined by the fact that the methods were adopted incompletely or in a piecemeal fashion for some portion of the analysis time period (2002-2004). this reflects the fact that these methods and the use of these methods as a coordinated system were being developed as they were being applied to real selection decisions. the application of the methods or some variant of them by librarians relatively new to chemistry collection development and collection development in general, would offer a much more controlled test of effectiveness. finally, the criteria for measuring sufficient evidence for monograph purchases may need to be reexamined and refined. studies could be conducted which investigate the relative importance of different kinds of evidence to the circulation of selected monographs. works cited barnett, philip. “combinatorial chemistry: a guide for librarians.” issues in science and technology librarianship 33 (2002) 15 jan. 2008 <http://www.istl.org/02winter/refereed.html>. bergart, robert, and vivian lewis. sudden selector’s guide to business resources. chicago: american library association, 2007. bottle, r. t., and j. f. b. rowland. information sources in chemistry. 4th ed. london: bowker-saur, 1993. burrell, quentin l. “using the gammapoisson model to predict library circulations.” journal of the american society for information science 41.3 (1990): 164-70. chemisty. dept. home page. university of houston. 6 sep. 2008 < http://www.chem.uh.edu/>. ching, joanna tan yeok, and k. r. chennupati. “collection evaluation through citation analysis techniques: a case study of the ministry of education, singapore.” library review 51.8 (2002): 398-405. clayton, peter, and g. e. gorman. managing information resources in libraries: collection management in theory and practice. london: library association publishing, 2001. evidence based library and information practice 2008, 3:3 16 culp, bartow. “chemistry: the central science.” library journal 125.13 (2000): 65-8. day, betty h. and william a. wortman. literature in english, a guide for librarians in the digital age. chicago: association of college and research libraries, 2000. douville, judith a. the literature of chemistry: recommended titles for undergraduate chemistry library collections. chicago: association of college and research libraries, 2005. evans, g. edward, and margaret zarnosky saponaro. developing library and information center collections. 5th ed. westport, ct: libraries unlimited, 2005. farina, alfred j., and monroe b. snyder. decision-making in the selection of science library materials for higher education: empirical findings and guidelines. chicago: american library association, 1967. fenske, ruth.e. “evaluation of monograph selection in a health sciences library.” bulletin of the medical library association 82.3 (1994): 26570. gorman, g.e., and ruth h. miller. collection management for the 21st century: a handbook for librarians. westport, ct: greenwood press, 1997. hazen, dan, and james henry spohrer, eds. building area studies collections. wiesbaden: harrassowitz, 2007. jenkins, clare, and mary morley. collection management in academic libraries. 2nd ed. brookfield, vt: gower, 1999. johnson, peggy. fundamentals of collection development and management. chicago: american library association, 2004. johnston, mark, and john weckert. “selection advisor: an expert system for collection development.” information technology and libraries 9.3 (1990): 219-25. kao, s. c., h. c. chang, and c. h. lin. “decision support for the academic library acquisition budget allocation via circulation database mining.” information processing and management 39.1 (2003): 13347. khalil, mounir a. “applications of an automated ill statistical analysis as a collection development tool.” journal of interlibrary loan, document delivery & information supply 4.1 (1993): 45-54. kraemer, alfred b. “evaluating usage of monographs: is it feasible and worthwhile?” collection management 26.1 (2001): 35-46. mack, daniel c. collection development policies: new directions for changing collections. binghamton, ny: haworth information press, 2003. maizell, robert e. how to find chemical information: a guide for practicing chemists, educators, and students. 3rd ed. new york: wiley, 1998. evidence based library and information practice 2008, 3:3 17 maple, amanda, and jean morrow. guide to writing collection development policies for music. lanham, md: scarecrow press, 2001. mcabee, sonja l., mary d. bevis, jodi w. poe, and george whitesel. houston cole library collection management and development policy. jacksonville, al: jacksonville state university, 2001. mccants, clyde t. opera for libraries: a guide to core works, audio and video recordings, books, and serials. jefferson, nc: mcfarland, 2003. montgomery, k. leon, stephen bulick, john fetterman, and allen kent. “costbenefit model of library acquisitions in terms of use: progress report.” journal of the american society for information science 27.1 (1976): 73-4. owens, irene. acquisitions and collection development in the humanities. new york: haworth press, 1997. pettitt, b. montgomery. faculty home page. 2006. dept. of chemistry, university of houston. 6 sep. 2008 < http://www.chem.uh.edu/faculty/p ettitt/>. powell, diana l. evaluation of the chemistry collection of a four-year college library by means of textbook citation analysis. kent, oh: kent state university, 1998. richards, daniel t., and dottie eakin. collection development and assessment in health sciences. lanham, md: scarecrow press, 1997. rudman, reuben. “recent books on popular chemistry: a ‘new’ 200-year-old book category.” choice: current reviews for academic libraries 41.7 (2004): 1227-45. sapp, gregg. “science at the ethical frontier; this year’s 39 best sci-tech books tackle scientific and moral questions.” library journal 119.4 (1994): 52-6. “scifinder for academics.” cas. 3 sep. 2008. american chemical society. 6 sep. 2008 <http://www.cas.org/products/sfaca d/index.html>. sowell, s.l. “expanding horizons in collection development with expert systems: development and testing of a demonstration prototype.” special libraries 80.1 (1989): 45-50. stelk, roger edward, and f. wilfrid lancaster. “the use of textbooks in evaluating the collection of an undergraduate library.” library acquisitions 14.2 (1990): 191-3. stephens, larry j. “inferences for two populations.” schaum's outline of theory and problems of beginning statistics. ed. larry j. stephens. new york: mcgraw-hill, 1998. sylvia, margaret. collection development policy, academic library, st. mary’s university. san antonio, tx: st. mary’s university, 1992. uzoka, f.m.e., and o. a. ijatuyi. “decision support system for library acquisitions: a framework.” the electronic library 23.4 (2005): 45362. microsoft word es 4071 evidence based library and information practice 2008, 3:4 57 evidence based library and information practice evidence summary birds of a feather flock together: the congruence of personality types within librarians’ subject specialties a review of: williamson, j.m., a.e. pemberton, and j.w. lounsbury. “personality traits of individuals in different specialties of librarianship.” journal of documentation 64.2 (2008): 273-86. reviewed by: john loy learning resources manager woodside library, callington road hospital brislington, bristol, united kingdom e-mail: john.loy@awp.nhs.uk received: 01 september 2008 accepted: 24 october 2008 © 2008 loy. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objectives – to investigate the personality traits of a range of librarians and information professionals using the personal style inventory (psi), and to investigate whether the personality traits of those in person-orientated library specialties differ from those in technique-orientated specialties. design – self-selecting survey. setting – solicitations to complete the survey were sent out via 10 e-mail discussion lists, and paper copies were made available at the annual american library association conference in 2002. subjects – 2,075 librarians and information science professionals. methods – participants completed the survey either in print format, as an e-mail attachment or a web form. the survey format was an adaptation of the psi scale using 13 of the accepted 16 scales, namely: • adaptability • assertiveness • autonomy • conscientiousness • customer service orientation evidence based library and information practice 2008, 3:4 58 • emotional resilience • extraversion • openness • optimism • teamwork • tough-mindedness • visionary-operational work style • work drive responses were analysed using a two-step cluster analysis technique, and participants were grouped into seven clusters. main results – the largest group of respondents was cataloguers at 23.7%, followed by other (health or law) 19.1%, academic reference librarians 13.2%, special librarians 12.3% with all other groups in single figures. respondents were divided up into the following seven clusters. • cluster 1, the “unadaptive” group - so labelled because several unadaptive traits such as low emotional resilience, low optimism, low teamwork, and low work drive are included. • cluster 2, “adaptive academic reference librarians” -high on customer service orientation, extraversion and teamwork, and low on tough-mindedness. • cluster 3, “adaptive cataloguers” - low on customer service orientation and possessing a more operational work style. • cluster 4, “adaptive special librarians” -high on autonomy, customer service orientation and extraversion. • cluster 5, “adaptive distance education librarians, public librarians, records managers, and school librarians” -possessing a visionary work style and scoring high on adaptability, assertiveness, customer service orientation, emotional resilience, high extraversion, openness, optimism, and teamwork; scoring low on tough-mindedness. • cluster 6, “adaptive other information professionals” -also possessing a visionary work style and with high scores on adaptability, assertiveness, autonomy, customer service orientation, emotional resilience, extraversion, openness, optimism, teamwork, and work drive. • cluster 7, “adaptive archivists and systems librarians” scoring high on assertiveness, openness, and toughmindedness. most clusters were comprised of a single occupational group, with only clusters 1 and 5 made up of individuals from more than one group. conclusion – the results indicate that different librarianship subspecialties can be differentiated by personality traits, and that individuals are likely to be drawn to either person-orientated or technique-orientated library specialties depending on their personality traits. commentary there is an episode of the classic bbc sitcom fawlty towers in which basil fawlty suggests his wife sybil should enter mastermind -a long-running tv quiz show -and for her specialist subject have “the bleedin’ obvious.” this comment did, perhaps uncharitably, pop into my head while reading williamson’s article. it is a thorough and workman-like piece of research, and an area around which not a lot of research has yet been done in the field of librarianship. but were the results not a foregone conclusion? perhaps they were, but it is important to remember that assumptions can be wrong. if we are to evidence based library and information practice 2008, 3:4 59 move towards an evidence-based profession, we must embrace research to help decision-making. what williamson gives us here are the beginnings of an evidence base to back up our professional instincts. the article includes much interesting background for those of us new to research into the effects of personality traits. williamson herself has written before on the subject and is clearly very comfortable in the realm. the literature search provides a fascinating round-up of studies and provides an excellent introduction. the work is very well referenced, though curiously an earlier article by williamson using a subset of the data presented here, and exploring related themes, is not referenced (williamson 2005). those working in the health arena particularly will find much to intrigue and amuse among the studies reported. previous research has shown that psychiatrists are more personorientated while machiavellianism, or cunning, is greatest in anaesthesiologists. the psi used here normally consists of 16 traits, and the authors provide a reassuring amount of detail to expand on how they arrived at the 13 personality traits measured. all items had been used in a wide range of circumstances and validated for a variety of jobs and organisations. these 13 traits are then expanded at some length, so that we have a reasonable idea of how to interpret, for example, “tough-mindedness.” conversely they do perhaps skate rather quickly over some of the limitations of their study, with background description of the statistical analysis being fairly thin. we are obliged to accept that most of the clusters contained only one occupational category - which seems very neat and conveniently tidy -with little explanation as to how this was actually achieved. another frustration is that details of gender, identified as a significant factor in previous studies of personality traits in librarianship, including work by williamson herself (goulding, williamson 2005), are not collected. geography is not taken into account and while we are told that responses were international, we get no indication of percentage of distribution. as paper surveys were distributed at the annual american library association conference in 2002, we can guess that this may indicate that responses are greatest from the usa. but as the authors do not provide a response percentage for each of the three survey formats, we can only speculate. nor do we learn anything about race or ethnicity, and we must not forget that respondents are exclusively englishlanguage speakers. i found it a particular disappointment of the study that we know next to nothing about the people who responded other than their particular specialty, how long they have been in it and time in their current post. for a study on personality type we do seem to learn rather more about the job than the people. perhaps this is unfair, as the authors do state that they are particularly interested in exploring the theory of congruence -how the same personality types are drawn towards similar job -rather than the personality traits themselves, and that is demonstrated by this research. so how can this be applied to practice? well, it’s certainly useful for those considering a career change or just starting out to take personality traits into account. similarly managers considering a restructuring of their services could benefit by actively considering staff personalities. might an unhappy cataloguer tucked away in the darkest recesses of the library be better suited to running information skills training evidence based library and information practice 2008, 3:4 60 sessions? or, might that be considered too obvious…? works cited goulding, a. et al. “professional characters: the personality of the future information workforce.” education for information 18 (2000): 7–31. williamson, j. “jungian/myers-briggs personality types of librarians in films.” reference librarian 78 (2002): 47-59. williamson j.m, a.e. pemberton, and j.w. lounsbury. “an investigation of career and job satisfaction in relation to personality traits of information professionals.” the library quarterly 75.2 (2005): 122-41. evidence based library and information practice 2012, 7.3 4 evidence based library and information practice article bridging the gap: understanding the differing research expectations of first-year students and professors meg raven coordinator of reference and instructional services mount saint vincent university library halifax, nova scotia, canada email: meg.raven@msvu.ca received: 3 may 2012 accepted: 3 aug. 2012 2012 raven. this is an open access article distributed under the terms of the creative commonsattributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – the project sought to understand the research expectations of first-year students upon beginning university study, and how they differed from the expectations of their professors, in order to provide more focused instruction and work more effectively with professors and student support services. methods – a survey of 317 first-year undergraduate students and 75 professors at mount saint vincent university in halifax, nova scotia, was conducted to determine what each expected of first-year student research. students were surveyed on the first day of the term in order to best understand their research expectations as they transitioned from high school to university. results – the gulf between student and professor research expectations was found to be considerable, especially in areas such as time required for reading and research and the resources necessary to do research. while students rated their preparedness for university as high, they also had high expectations related to their ability to use nonacademic sources. the majority of professors believed that students are not prepared to do university-level research, do not take enough responsibility for their own learning, should use more academic research sources, and should read twice as much as students believe they should. mailto:meg.raven@msvu.ca evidence based library and information practice 2012, 7.3 5 conclusions – by better understanding differing research expectations, students can be guided very early in their studies about appropriate academic research practices, and librarians and professors can provide students with improved research instruction. strategies for working with students, professors, and the university community are discussed. introduction librarians frequently hear from professors: “my students won’t look beyond google for sources”; “they copy indiscriminately without citing”; “they complain about reading anything longer than a screen.” the lament is different from students: “i don’t understand why i can’t use google or wikipedia”; “what’s the big deal about copying? everyone does it”; “i just don’t understand this long journal article – it’s written for an expert in the field, not me.” most academic librarians have lived these experiences. those who choose to work in the field of library instruction likely spend a great deal of time considering students’ and professors’ differing expectations of student research. while a sample of some of the research carried out is offered below, none of this research addresses in detail students’ research expectations upon beginning their university studies. professors and librarians acknowledge implicitly that most students arrive at university unprepared to conduct academic research but that as part of the learning experience their expectations will shift and align with those of their professors; however, this paper proposes that both professors and librarians will be better prepared to help first-year students advance their learning if we identify and better understand the research expectations with which students arrive at university. understanding exactly where students are beginning their studies will provide librarians with the information we need to create the most appropriate research instruction programs. the primary goal of this study was to identify how first-year students’ and professors’ expectations of student research differ, and thus explore the role librarians can play by working with both groups to bridge this gap. to this end, a study was undertaken at mount saint vincent university (msvu), in halifax, nova scotia, canada, that investigated first-year university students’ and professors’ expectations of the academic research process as conducted by firstyear students. msvu is a small, predominantly undergraduate university that specializes in liberal arts and selected professional studies. the student body numbers approximately 5,000, and the 80% female population reflects the university’s heritage as a former female school. embedded in the mission of the institution is a commitment to teaching and personalized education. all attempts are made to keep class size small, with 73% of classes enrolling fewer than 30 students (mount saint vincent university, 2012). the university’s strong commitment to collaborative teaching and learning provided an ideal arena to investigate differing research expectations and to propose concrete, yet collaborative facultylibrarian recommendations that could benefit students. literature review the volume of information literacy (il) literature is considerable and contains research that attempts to explain and offer interventions for the introductory scenarios that describe the very different research expectations of professors and students. much has been written evidence based library and information practice 2012, 7.3 6 on university students’ general research experiences, with the majority of contemporary work focusing on students’ use of online sources. van scoyoc and cason (2006), mcclure and clink (2009), griffiths and brophy (2005), and thompson (2003) all provide useful insights into students’ use of online resources for academic research and their inability to effectively evaluate the information they retrieve. these studies, coupled with work undertaken in the field of information-seeking behaviour (head, 2008), suggest that students are more concerned with how much time research will take than with the accuracy of the information found (weiler, 2005); that even though students have used abstracting and indexing databases, many will select only articles available in full text (imler & hall, 2009); and, finally, that many still prefer google (williamson, bernath, wright, & sullivan, 2007). other studies have explored the issue of student satisfaction with their research experience (belliston, howland, & roberts, 2007; martzoukou, 2008) and their satisfaction with library services (gardner & eng, 2005; harwood & bydder, 1998; voelker, 2006). findings suggest that students are generally happy with their research and library experiences (gardner & eng, 2005) but often prefer the convenience of their own homes when conducting research (vondracek, 2007). another important line of research has considered the role the university professor plays in students’ learning to carry out academic research. valentine (2001) looked at the disparity between students’ understanding and experience of a research assignment and the goal of the assignment as described by the professor. students typically evaluated an assignment based on the degree of effort required and the grades awarded, whereas the professor viewed a particular assignment based on its learning experience. mcguinness (2006) writes convincingly that there is “a tacit assumption among faculty that students would somehow absorb and develop the requisite knowledge and skills through the very process of preparing a written piece of coursework” (p. 577), and that becoming information literate simply requires participation in established academic research traditions such as research methods courses, computer skills classes, and library instruction. mcguinness goes on to describe faculty as believing that students will simply “pick up” information literacy skills, and if students are motivated to become information literate, they will learn. little seems to have changed since leckie (1996), in her classic article, criticized faculty who created assignments that required students to use skills which they had not yet developed. the studies identified above, however, do not adequately address the issue of research expectations. with the exception of scutter, palmer, luzeckyj, burke da silva, and brinkworth (2011), laskowski (2002), and long and tricker (2004), very little work has been done on the research expectations of students. (the bulk of student expectation research concentrates on students’ more general academic and career expectations and aspirations.) scutter et al. present important data on a range of first-year student expectations that includes how much time students expect to study for each course in which they are enrolled, but they do not address more detailed research expectations. laskowski tackles the issue of divergent research expectations between students and professors by focusing on students’ use of technology. her study shows that discrepancies exist between how and when students and professors believe technology should be used in academic research: “many students believe that their professors do not appreciate or understand the wide variety and scope of material available online and that they devalue online resources because of format rather than content” (p. 305). long and tricker surveyed only undergraduate students, not faculty, in the united kingdom to determine if their expectations of university-level research differed from their experiences. they found that evidence based library and information practice 2012, 7.3 7 students’ expectations do differ from their experiences, but not substantially. the study described below proposes that with a better understanding of both students’ and professors’ expectations of first-year student research, some light can be shed on what sometimes feels like a widening gulf between students’ research practices and professors’ research expectations. it is proposed that by adding research expectations as a variable in the information literacy equation, librarians and professors will be better equipped to assist firstyear students with their research. methods data collection involved the construction of two surveys: student expectations of the research process (appendix a) and faculty expectations of student research (appendix b). both surveys were administered with the approval of the mount saint vincent university research ethics board. the student survey was designed to gather data on students’ past research experiences and their expectations of universitylevel research. students were asked very specific questions about past research experiences and sources they had used and about more general activities that could influence research behaviours, such as use of technology and time spent reading. the faculty survey was constructed to complement and compare with data gathered from the student survey. the student survey was administered to firstyear classes only. this choice was made for two reasons: first, these classes were most likely to contain recent high school graduates, making it possible to learn more about student research expectations upon beginning university; and second, it was necessary to identify, for professors, a specific group of students to base their own responses on when completing the faculty survey. professors likely have very different research expectations of first-year and senior students. the first-year classes were chosen from across disciplines in an attempt to have broad student representation. eight introductory classes, with a total student count of 434, were surveyed on the first day of the 2008-09 academic year. this date was selected so that students would complete the survey before their professors had an opportunity to discuss with them their own research expectations. a librarian visited the classroom at a pre-arranged time and distributed hard-copy surveys that students could complete on the spot. a total of 317 student surveys (73% return rate) were completed. approximately 240 full-time and part-time professors at msvu were contacted by email and invited to complete a web-based survey. a total of 75 faculty surveys (31% return rate) were completed. results demographics and access to information communication technology (ict) the survey asked students to provide basic demographic information about themselves. eighty percent of respondents were female, 71% were in their first year of study, and 76% were age 20 or under. over 95% identified themselves as full-time students, and their declared majors represented a cross-section of disciplines: 38% social science and humanities; 22% sciences; 37% professional studies; 3% with undeclared majors. fifty-eight percent of students reported working while going to school, and of those, over 50% reported working more than 20 hours per week. in order to better understand students’ use of ict, and how it may impact their use of research resources, students were asked to indicate which technologies they could easily access. over 80% of students responded that they had ready access to a laptop, the internet, cell phone, texting, or an ipod (or similar device). when evidence based library and information practice 2012, 7.3 8 asked to indicate how much time they spent online in an average week during the past year participating in activities such as web browsing, social networking, email, or gaming, approximately 27% of students indicated they spent over 16 hours per week online; 39% spent 8-15 hours per week online; and 34% spent fewer than 7 hours per week online. high school experiences access to and use of technology are an important variable when considering how students may expect to conduct academic research. also important to consider are the experiences these students may have had with previous research in high school. students were asked to respond to questions about their use of the google search engine and research databases while in high school, and also to indicate how much instruction they had received on citation and plagiarism. specifically, students were asked if their teachers allowed them to use google (or other search engines) to do research for assignments. sixty-six percent indicated that they were allowed to use google “all the time” and 21% indicated “most of the time.” by contrast, only 12% of students indicated they used a research database “all the time” or “most of the time” to do research. far more common were the students (51%) who reported that they “rarely” or “never” used a research database. it is important to note that in the province of nova scotia, where 77% of students completed high school, school boards have subscriptions to the ebsco databases. students reported on levels of citation and plagiarism instruction while in high school. sixty-four percent of students indicated that high school teachers discussed the issues of citation and plagiarism with them “all the time” or “most of the time.” by contrast, when professors where asked how much instruction they believed students had received in high school, only 15% indicated they believed teachers spoke about these issues “all the time” or “most of the time.” the majority of professors indicated that they believed citation and plagiarism were discussed only “sometimes” (40%) or “rarely” or “never” (41%). table 1 first-year students’ high school research experiences all the time most of the time sometimes rarely/never students’ report that high school teachers allowed them to use google for research assignments in high school 66% 21% 8% 3% students’ use of databases for research assignments in high school 6% 6% 22% 51% students’ report of teachers discussing citation and plagiarism in high school 28% 36% 19% 14% professors’ belief that high school teachers discuss citation and plagiarism with students in high school 5% 10% 40% 41% note: not all totals equal 100% as some respondents did not answer all questions. evidence based library and information practice 2012, 7.3 9 research skills students and professors were asked to rank students’ preparedness to do university-level research and to indicate who they feel is most responsible for first-year students’ learning how to do research. figures 1 and 2 show the discord between students’ and professors’ views in these areas. in figure 1, 70% of students reported that they were “very prepared” or “somewhat prepared” to do university-level research. this level greatly exceeds how their professors view their preparedness, with 87% indicating that students are “not very prepared” to conduct such research. related to this is the question of who is responsible for students learning universitylevel research skills. it is interesting that while students rate their preparedness as high, figure 2 shows that only 50% take personal responsibility for learning the necessary research skills. by contrast, 80% of professors indicate that the students themselves are most responsible for learning these skills. figure 1 first-year students’ preparedness to do university-level research figure 2 who is most responsible for first-year students learning how to do research? evidence based library and information practice 2012, 7.3 10 students and professors were asked to rate students’ general internet searching skills and their academic research skills. figures 3 and 4 show that students and professors view students’ skills in these areas very differently. in figure 3, results found that almost 75% percent of students rated their general internet searching skills as “excellent” or “good,” whereas 84% of professors rated students’ skill as only “average” or “poor.” when students were asked to indicate how they rated their academic research skills, that is, the ability to find scholarly information, they were slightly less confident. as illustrated in figure 4, 49% still categorized themselves as “excellent” or “good.” here professors were quite clear in their rating of students’ research skills: a full 67% indicated skills were “poor” or “terrible.” students were also asked to indicate who they believe has the best internet searching skills, choosing from it professionals, librarians, professors, and students. they ranked it professionals as the best searchers 45% of the time, followed by librarians 37% of the time. students ranked themselves third (12%) and professors last (6%). figure 3 rating of first-year students’ general internet searching skills figure 4 rating of first-year students’ academic research skills evidence based library and information practice 2012, 7.3 11 reading and research much has been written about the decline in reading (see jameson, 2007; reedy, 2007; or salter and brook, 2007, for discussions of the decline in reading among college students). given the importance of reading in higher education, the current study sought to better understand how much time first-year students had spent reading in the past year, and how much time they expected to dedicate to reading to keep up with their school work and research during the upcoming year. professors were also asked to indicate how much time they expected first-year students to spend reading. figure 5 illustrates that there is a considerable gulf between how much time students expected to dedicate to reading and what professors expected of them in this regard. column one illustrates students’ reading experiences during the last year. on average they reported reading approximately 7.8 hours per week – just a little over one hour per day. column two illustrates students’ expected reading during the coming year. in this case, students were asked to indicate, regardless of how much they read in the past year, how much they expected to read in the coming year. students indicated that they expected to read more, predicting on average 9.8 hours of reading per week. column three illustrates professors’ expectations of student reading. even though students indicated that they would be reading more than in the past, their expectations did not approach professors’ reading expectation of, on average, 14.9 hours per week. students and professors were then asked to consider how long they anticipated it would take students to conduct the necessary research for a 10-page paper or assignment in an introductory course. figure 6 shows that again we see divergent research expectations between students and professors. figure 5 first-year students’ reading experiences and expectations (hours/week) note: “reading” was defined for students as any time spent reading in print or online format in order to accommodate various reading media but did not include time spent emailing, texting, gaming, social networking, or general web browsing. evidence based library and information practice 2012, 7.3 12 fifty-eight percent of students indicated it would take them less than 5 hours to research such a paper or assignment; by contrast 41% of professors indicated they expected students to spend at least twice that amount of time. appropriate research resources in an attempt to better understand how firstyear students and professors value the google search engine or other similar search engines as an academic research tool, students were asked to indicate how much research material they expected to locate by carrying out a google search, and professors were asked to indicate how much research material they expected/wanted students to find by searching google. the majority of professors (73%) indicated that google was an appropriate academic research tool for locating less than 20% of research material. in contrast, 70% of firstyear students expected to make use of google to locate between 50% and 100% of their research material. to understand what other resources students expected to use for academic research, and the resources professors expected/wanted students to use, both groups were asked to select from a list of over 40 electronic and print resources that they expected to use, or expected students to use, when carrying out academic research. table 2 summarizes the top five resources, ranked by figure 6 time required to research a 10-page paper/assignment table 2 resources students expect to use and sources professors expect/want students to use for academic research first-year students’ top 5 research resources professors’ top 5 research resources (for first-year student use) 1 books from home library 1 journals 2 google 2 library website 3 newspapers 3 books from home library 4 encyclopedias 4 library catalogue 5 library website 5 databases evidence based library and information practice 2012, 7.3 13 frequency of selection as an expected research resource. getting help if an assignment presented challenges, students were asked to consider where they would go for help, and professors were asked where they expected first-year students to seek help. figure 7 illustrates that students and professors both see the professor as the key assignment authority, followed closely by librarians. both groups also see fellow classmates as a good resource when help is needed. an interesting discrepancy found here is that students consider their friends almost as good a source for research help as librarians. seventy-three present of students will seek help from a librarian and 67% will go to friends. professors discount the value of input from friends (10%) and family (5%), whereas students expect to make considerable use of these groups. discussion high school research experiences the data presented here suggest that most firstyear students entering university directly from high school developed their research skills in an environment where google was the primary research tool. while the data do not tell us whether teachers advocated for the use of research databases, they do tell that students report rarely using them. high schools students look upon their teachers as research authorities. with google identified as the research tool of choice, more focused and consistent information literacy work needs to be done in teacher education programs (kovalik, jensen, scholman, & tipton, 2010), and school boards must reinvest in school library programs and teacherlibrarian positions (gunn & hepburn, 2003; heycock, 2003). one interesting, yet positive, finding of this study is that students clearly remembered figure 7 where first-year students expect to seek research assistance, and where professors want students to seek research assistance evidence based library and information practice 2012, 7.3 14 receiving a fair amount of instruction on citation and plagiarism during high school. like many professors, librarians are frequently confronted with students who seem unaware of conventional citation practices, and who do not have a good grasp of the concept of plagiarism. while it appears teachers are stressing the importance of these concepts, more focused research is required to uncover why students are not retaining what they suggest they have learned. perhaps students are not getting enough practice citing and writing, or perhaps there is not consistent instruction across high school classrooms. chao, wilhelm, and neureuther (2009) provide strong evidence that students’ ability to cite, paraphrase, and avoid plagiarism improves with practice. in universities with teacher education programs or links to high schools, there is still much work that can be done. current and future teachers will have the greatest impact on the research abilities of first-year students, and so it is imperative that librarians make them aware that students entering university continue to struggle with citation and plagiarism, and many are unfamiliar with the academic sources found in research databases. academic librarians who are able to partner with high school librarians will find the results of the oakleaf and owen study (2010) very helpful. it describes a successful collaboration involving syllabi review that helped prepare senior high school students for university-level research. librarians with subject responsibility for education may wish to consider approaching education curriculum groups to advocate for more integrated instruction in the areas of citation and plagiarism and in the use of databases and google. in addition, schools of education often provide in-service training for current teachers. msvu recently offered a wellreceived librarian presentation as part of an inservice session. topics covered included the identification of professional literature that outlines the challenges faced by many first-year university students and the sharing of first-year students’ initial research experiences. the first-year millennial student this study corresponds with the results of work done by englander, terregrossa, and wang (2010), and miller (2007), in which college students reported spending, on average, 14.3 hours and 17 hours per week online, respectively. the current research also confirms what a number of authors (abram, 2007; becker, 2009; sweeney, 2012; twenge, 2006) have written about generation y or millennial students’ high levels of self-confidence: students are arriving at university believing they are quite prepared to conduct university-level research, but only half are taking personal responsibility for learning how to do such research. by contrast, most professors rate firstyear students as not very prepared to do research and believe they must take personal responsibility for their own learning. these differing expectations need to be addressed with students early in their academic programs, and the idea of personal responsibility reinforced throughout their studies. while professors can identify their expectations for what students learn about research in the classroom, and the learning students are expected to pursue on their own by seeking out library research instruction and through independent activity (e.g., library tutorials), librarians have less direct access to students. this is an area where a more focused collaboration between professors and librarians could be nurtured. at msvu, when setting up instruction workshops with faculty, librarians have begun to ask explicitly what, if any, research skills faculty will be teaching in their classes and what students are expected to do on their own. this lets the librarians know where we fit in the equation and where attention should be focused. the information gleaned is useful regardless of the instruction format (50minute one-shot or multi-part seminar). while still at the informal information-gathering stage, evidence based library and information practice 2012, 7.3 15 there are plans to pursue a more detailed study that considers where various university constituents (i.e., faculty, student support services, and the international student centre) expect students to learn research skills. mounce (2010) provides a thorough review of the faculty-librarian collaboration literature as it relates to information literacy and the benefits afforded students. anthony (2010) also reviews this literature but with the added depth of providing tangible examples of programs in operation. what both reviews are lacking, however, are details on broadening the types of material covered by instruction librarians. these librarians are often drawn into the classroom to discuss the latest research tools when their time may be better spent initially on non-resource instruction addressing research expectations. instead of immediately launching into database selection and search strategies, dedicating time to a discussion of the basics of research, the time involved, the reading requirements, and the careful thought and preparation required may help students to understand that research is an involved process. taking time for discussion is important given how many students reported how little time they expected to spend on the research components of their assignments. preparing this kind of presentation with the professor ahead of time will allow students to hear from the librarian and from their professor, in tandem, that academic research takes time to learn and carry out. students must be encouraged to accept responsibility for this complex learning (ferlazzo, 2011). many librarians have seen assignments that require that a specified number of resources be consulted; we need to encourage professors to also provide details on how long the assignment should take students to research and write up. one surprising piece of evidence collected in this study has to do with how students rated their own internet searching skills. students consistently ranked themselves third, behind it staff and librarians. professors were ranked last in internet searching skills, which could lead to students being hesitant in going to their professors for some forms of research help. a study by gunn and hepburn (2003), and reinforced here, suggests that high school students are most comfortable seeking help from friends and classmates rather than from teachers. what librarians and professors should take away from this finding, especially in universities where library reference departments share physical space with an information or learning commons, is that students may see computing it staff as most knowledgeable in internet searching and they may opt to approach these staff members first or exclusively. alternatively, some students simply may not differentiate between the staff working in a learning commons (bickley, 2011) and may seek help from the first available person. at msvu we encourage a lot of communication between technical staff and librarians to ensure that research questions are directed to the appropriate person. short in-house training sessions or providing staff with the opportunity to job-shadow in other public service areas provides everyone with a better understanding of which questions should be handled where. reading and research the data gathered in this study supports the 2007 report to read or not to read, which details a general decline in reading and found that 39% of college freshmen did no reading for pleasure and 26% read no more than one hour per week. the report provides strong evidence linking reading to literacy scores and it cites “written communication” as the skill most lacking by employers hiring both high school and college graduates. the current study shows a large gap between student and professor expectations surrounding reading. a full 83% of professors believe students need to be reading at least 12 hours per week, whereas only 31% of students reported that they expected to read this much. gilbert and fister (2011) discuss the many academic benefits of pleasure reading and also explain that academic reading is quite difficult: students “often need help in learning how to do evidence based library and information practice 2012, 7.3 16 ‘close’ or in-depth analytical reading” (p. 475). building into information literacy workshops a statement or acknowledgement that the ability to read critically is challenging and takes time may help students be better prepared to tackle more advanced reading and not to shy away from lengthier journal articles. librarians at msvu are beginning to include in instruction workshops explicit statements informing students that the type of information they find in academic databases will usually require indepth analytical reading. explaining that it is common to have to read an article more than once and often with the help of a dictionary may normalize the experience for students. this is also an ideal time to remind them that there are academic support services available on campus if they feel they are struggling with this type of work. related to the findings on reading, and the lack of time students expect to take conducting research, is the matter of the resources they expect to use when conducting research. this is another category in which student and professor expectations varied considerably. while the list of research resources generated by the professors contains common academic research tools (journals, books, catalogues, databases), students appear to have selected sources with which they are familiar, or perhaps those they used in high school (books, google, newspapers, encyclopedias). one has to wonder, though, if rather than selecting the research tools they expected to use, students instead selected resources they thought we would want them to use when researching. follow-up research will be necessary to better understand these findings. it might be expected that students would use books and google, but also anticipated on the list might be wikipedia, electronic books, and general websites. the marked absence of newer (web 2.0) research technologies was common to both students’ and professors’ lists: both surveys asked respondents if they expected students to use blogs, podcasts, rss feeds, and videos for research purposes, but all were notably absent. neither group indicated that these were resources they expected to use for academic research. librarians preparing instructional sessions should not only seek guidance from professors as to what resources they want their students consulting, but we can also provide guidance on the diverse variety of tools available that can add depth to students’ research experiences. working with students faulty-librarian collaboration has always been central to library instruction (mounce, 2010) and this study supports the idea that it is increasingly important that librarians and professors work together to deliver a consistent message to students. especially during their first year, students need to hear a research refrain that is campus-wide and includes student academic support services (love & edwards, 2009). coupled with delivering a strong consistent research message is the practice of reminding students that while they are not expected to know how to do scholarly research when they arrive at university, they are expected to learn new ways of doing this academic work by embracing new research tools. one specific way librarians can focus their work is by acknowledging the positive. we must validate for students their past research experiences. students do not arrive at university as “blank research slates”: they have been googling their research questions for years. magolda (2012) discusses the concept of a learning partnership whereby professors are encouraged to “listen more carefully to students’ thinking and recognize that their experiences often prompt different, yet valuable interpretations” (p. 35). librarians could also explore this teaching method as another way to help students develop their research skills. if librarians and professors are overly critical of past research practices, we risk discouraging these novice academic researchers. we can encourage students to join the research dialogue by asking them to describe their own research experiences and evidence based library and information practice 2012, 7.3 17 expectations. giving positive feedback when we see that appropriate sources are being used, and giving suggestions for alternatives when an inappropriate source is selected, can help students refine expectations early in the research process. we can reinforce that google is the perfect tool for locating food guide standards, for example, but it is not an acceptable academic source for critiques of the standards. each discipline and course needs to have such a relevant example at its fingertips when a teachable moment arrives. exploring innovative ways to initiate these dialogues with students, and the outcomes, is another area for future research. working with professors and cross-campus support services working with professors is both rewarding and challenging. a number of authors (anthony, 2010; mcguinness, 2006; mounce, 2010) discuss the challenges librarians have engaging some faculty in information literacy initiatives. however, success stories are also available in the literature : corso, weiss, and mcgregor (2010) describe the embedding of il skills by a team of librarians, writing program coordinators, and professors; kenedy and monty (2011) discuss how student learning is enhanced as a result of a librarian and faculty member collaboration that ties together information literacy, research skills, and other essential post-secondary skills; kobzina (2010) describes partnering with faculty in the teaching of a specific course that addresses research skills for specialized subject areas. these examples illustrate that information literacy instruction can be broad-based and very rich. most instruction librarians are more than happy to partner with professors on curriculum or assignment review (brown & kingsley-wilson, 2010) to determine how information literacy can be addressed more explicitly. at msvu, librarians have begun to actively invite faculty to discuss syllabi and assignments with us regardless of whether or not we visit their classrooms. many professors seemed hesitant to seek out this kind of input when they were not willing to provide dedicated classroom time for library instruction. while we would prefer to also be invited to give an il workshop, we recognize that sometimes having access to syllabi can provide students with basic yet significant research information. one professor who had never seen value in having a librarian present during class time did agree to include library research information and a subject librarian’s contact information on the course syllabus. it was encouraging to see that reference traffic increased slightly in this area. this is a very small success story, but when we see how unprepared many first-year students are for university research, we decided that any contact with students – even only through an email – was better than no contact. while most librarians will actively seek out opportunities to engage professors at their home institution, librarians can also strive to get their messages out in alternate venues, for example, discipline-specific teaching journals and nonlibrarian conferences. engaging professors in their own domains may remind those who have partnered with librarians in the past to reconnect, and it may convince others of the teaching and research abilities of their librarian colleagues. at msvu, librarians take part in cross-campus research seminars where faculty and librarians are invited to present current research projects. teaching faculty members have been consistently interested in any work on student learning. while much has been written about librarian collaboration with faculty, far less work has been done on librarians partnering with other cross-campus support services (hollister, 2005). a few studies (love & edwards, 2009; swartz, carlisle, & uyeki, 2007) have more recently provided an excellent introduction to the mechanics of this type of collaboration and provide evidence that there is much to be gained when libraries partner with student support services. a disappointing result of the current study was the finding that only half of first-year evidence based library and information practice 2012, 7.3 18 students and 75% of professors reported that they expected students who need help to take advantage of student support services such as writing centres. in order to broaden library instruction services, more libraries may want to consider partnering with student support services such as writing and international student centres. walter and eodice (2005) caution that it is important for librarians to work with colleagues to find “common language through which learning objectives can be defined” (p. 220). librarians, who are used to partnering with faculty, may be unfamiliar with the learning objectives of student support services. it is incumbent upon us to not just take il needs to student support services, but to understand the values and goals of these units and whenever possible try to support their initiatives without duplicating them. as described earlier, a new initiative has msvu librarians explicitly linking the concept of in-depth reading to the retrieval of scholarly articles. providing a referral to a support service is always helpful, but introducing the concept of in-depth reading in a way that complements the instruction students get in a support unit just makes sense. students will perceive that there is a coordinated effort that may help them be more successful. msvu librarians are trying to become better informed about the office of students services, and as a result have been invited to sit on student retention and student experience committees. while little of this work links directly to our initiatives in il, librarians feel better informed about support services for students. we are optimistic that the time we put in now will benefit some of our own instruction initiatives in the future. one other area in which librarians should direct their attention relates to representation on committees that give them access to program and curriculum design, which will put them in a position to provide input on research and il skill development (anthony, 2010). such forums often allow administrators and student support staff to hear, sometimes for the first time, about some of the gaps in research expectations described in this study. the better everyone understands the unpreparedness of many firstyear students, the better we will be at bridging this gap and coordinating efforts to support students’ adjustment to university-level research. conclusion this study provides evidence that the research expectations of first-year students and professors vary considerably. students arrive at university believing that they have better online skills than their professors and that they are prepared to do university-level research; they are often overconfident about their research skills and therefore may not ask for help; they expect that it will take less time to do research than is in fact the case; and many are reading less than is likely necessary to grasp a subject in depth. while some professors will tell librarians that they know these facts, many may be struggling with what to do with the knowledge. librarians who work closely with both students and professors are afforded the unique view of both worlds and are ideally positioned to provide not only research instruction, but research insight to students, professors, and the wider university community. an unexpected outcome of this study is the acknowledgement that not only is the facultylibrarian relationship significant in students’ research development, but that there is also an important need for broad cross-campus collaboration. this article draws attention to the idea that students deserve to get consistent research messages across campus. the more we work with faculty and academic support services, the more we are able to provide integrated, coordinated instruction. when there is a strong campus-wide voice addressing research expectations, librarians can work with students with greater certainty. this study covers many topics at a general level and raises many further questions. there is a evidence based library and information practice 2012, 7.3 19 need for more focused work in a number of areas, specifically relating to students’ understanding of citation and plagiarism as they transition from high school to university, the sources students expect to consult for academic research purposes, and the broadening of library instruction portfolios to include instruction on critical thinking skills such as in-depth reading. references abram, s. (2007). millennials: deal with them! part i. school library media activities monthly, 24(1), 57-58. anthony, k. (2010). reconnecting the disconnects: library outreach to faculty as addressed in the literature. college & undergraduate libraries, 17(1), 79-92. doi:10.1080/10691310903584817 baxter magolda, m. b. (2012). building learning partnerships. change: the magazine of higher learning, 44(1), 32-38. doi: 10.1080/00091383.2012.636002 becker, c. h. (2009). student values and research: are millennials really changing the future of reference and research? journal of library administration, 49(4), 341364. belliston, c. j., howland, j. l., & roberts, b.c. (2007). undergraduate use of federated searching: a survey of preferences and perceptions of value-added functionality. college & research libraries, 68(6), 472-486. bickley, r. & corrall, s. (2011). student perceptions of staff in the information commons: a survey at the university of sheffield. reference services review, 39(2), 223-243. doi:10.1108/00907321111135466 brown, c., & kingsley-wilson, b. (2010). assessing organically: turning an assignment into an assessment. reference services review, 38(4), 536-556. doi:10.1108/00907321011090719 chao, c.-a., wilhelm, w. j., & neureuther, b. d. (2009). a study of electronic detection and pedagogical approaches for reducing plagiarism. delta pi epsilon journal, 51(1), 31-42. corso, g. s., weiss, s., & mcgregor, t. (2010). information literacy: a story of collaboration and cooperation between the writing program coordinator and colleagues 2003-2010. paper presented at the national conference of the council of writing program administrators (cwpa), philadelphia, pa. retrieved 21 aug. 2012 from http://www.eric.ed.gov/ericwebportal/de tail?accno=ed529197 englander, f., terregrossa, r. a., & wang, z. (2010). internet use among college students: tool or toy? educational review, 62(1), 85-96. doi:10.1080/00131910903519793 ferlazzo, l. (2011). how do you help students see the importance of personal responsibility? helping students motivate themselves: practical answers to classroom challenges (pp. 14-28). ny: eye on education. gardner, s., & eng, s. (2005). what students want: generation y and the changing function of the academic library. portal : libraries and the academy, 5(3), 405-420. gilbert, j., & fister, b. (2011). reading, risk, and reality: college students and reading for pleasure. college & research libraries, 72(5), 474-495. griffiths, j. r., & brophy, p. (2005). student searching behavior and the web: use of evidence based library and information practice 2012, 7.3 20 academic resources and google. library trends, 53(4), 539-554. gunn, h., & hepburn, g. (2003). seeking information for school purposes on the internet. canadian journal of learning and technology, 29(1). retrieved 21 aug. 2012 from http://hdl.handle.net/10515/sy5b853w9 harwood, n., & bydder, j. (1998). student expectations of, and satisfaction with, the university library. journal of academic librarianship, 24(2), 161-171. haycock, k. (2003). the crisis in canada’s school libraries: the case for reform and re-investment. executive summary. toronto, on: association of canadian publishers. retrieved 21 aug. 2012 from http://www.accessola.com/data/6/rec_docs/ execsummary_ha_e1e12.pdf head, a. j. (2008). information literacy from the trenches: how do humanities and social science majors conduct academic research? college & research libraries, 69(5), 427-445. hollister, c. (2005). bringing information literacy to career services. reference services review, 33(1), 104-111. doi:10.1108/00907320510581414 imler, b., & hall, r. a. (2009). full-text articles: faculty perceptions, student use, and citation abuse. reference services review, 37(1), 65-72. jameson, d. a. (2007). literacy in decline: untangling the evidence. business communication quarterly, 70(1), 16-33. kenedy, r., & monty, v. (2011). facultylibrarian collaboration and the development of critical skills through dynamic purposeful learning. libri: international journal of libraries & information services, 61(2), 116-124. doi:10.1515/libr.2011.010 kobzina, n. g. (2010). a faculty-librarian partnership: a unique opportunity for course integration. journal of library administration, 50(4), 293-314. doi:10.1080/01930821003666965 kovalik, c. l., jensen, m. l., schloman, b., & tipton, m. (2010). information literacy, collaboration, and teacher education. communications in information literacy, 4(2), 145-169. retrieved 21 aug. 2012 from http://www.comminfolit.org laskowski, m. s. (2002). the role of technology in research: perspectives from students and instructors. portal: libraries and the academy, 2(2), 305-319. leckie, g. j. (1996). desperately seeking citations: uncovering faculty assumptions about the undergraduate research process. journal of academic librarianship, 22(3), 201208. long, p., & tricker, t. ( 2004, sept.). do first year undergraduates find what they expect? paper presented at the british educational research association annual conference, university of manchester. retrieved 21 aug. 2012 from http://www.leeds.ac.uk/educol/documents/ 00003696.doc love, e., & edwards, m. b. (2009). forging inroads between libraries and academic, multicultural and student services. reference services review, 37(1), 20-29. doi:10.1108/00907320910934968 martzoukou, k. (2008). students' attitudes towards web search engines increasing appreciation of sophisticated search strategies. libri: international journal of evidence based library and information practice 2012, 7.3 21 libraries & information services, 58(3), 182201. mcclure, r., & clink, k. (2009). how do you know that? an investigation of student research practices in the digital age. portal: libraries & the academy, 9(1), 115-132. mcguinness, c. (2006). what faculty thinkexploring the barriers to information literacy development in undergraduate education. journal of academic librarianship, 32(6), 573-582. doi:10.1016/j.acalib.2006.06.002 miller, m. j. (2007). internet overuse on college campuses: a survey and analysis of current trends. in y. inoue (ed.), technology and diversity in higher education: new challenges (pp. 146-163). hershey, pa: information science publishing. mounce, m. (2010). working together: academic librarians and faculty collaborating to improve students' information literacy skills: a literature review 2000-2009. reference librarian, 51(4), 300-320. doi:10.1080/02763877.2010.501420 mount saint vincent university. office of institutional analysis. common university data. retrieved 25 june 2012 from http://www.msvu.ca/en/home/aboutus/ho me/institutionalanalysis/commonuniversity data/default.aspx oakleaf, m., & owen, p. l. (2010). closing the 12-13 gap together: school and college librarians supporting 21st century learners. teacher librarian, 37(4), 52-58. reedy, j. (2007). cultural literacy for college students. academic questions, 20(1), 32-37. salter, a., & brook, j. (2007). are we becoming an aliterate society? the demand for recreational reading among undergraduates at two universities. college & undergraduate libraries, 14(3), 27-43. doi:10.1300/j106v14n03&#x201702 scutter, s. d., palmer, e., luzeckyj, a., burke da silva, k., & brinkworth, r. (2011). what do commencing undergraduate students expect from first year university? the international journal of the first year in higher education, 2(1), 8-20. doi:10.5204/intjfyhe.v2i1.54 swartz, p. s., carlisle, b. a., & uyeki, e. c. (2007). libraries and student affairs: partners for student success. reference services review, 35(1), 109-122. doi:10.1108/00907320710729409 sweeney, r. t. (2012). millennial behaviors and higher education focus group results: how are millennials different from previous generations at the same age? retrieved 21 aug. 2012 from http://library1.njit.edu/stafffolders/sweeney/millennials/millennialsummary-handout.doc thompson, c. (2003). information illiterate or lazy: how college students use the web for research. portal: libraries and the academy, 3(2), 259-268. to read or not to read: a question of national consequence (2007). washington, dc: office of research & analysis, national endowment for the arts. retrieved 21 aug. 2012 from http://www.nea.gov/research/toread.pdf twenge, j. m. (2006). generation me: why today’s young americans are more confident, assertive, entitled – and more miserable than ever before. ny: free press. valentine, b. (2001). the legitimate effort in research papers: student commitment versus faculty expectations. journal of academic librarianship, 27(2), 107-115. evidence based library and information practice 2012, 7.3 22 van scoyoc, a. m., & cason, c. (2006). the electronic academic library: undergraduate research behavior in library without books. portal: libraries & the academy, 6(1), 47-58. voelker, t. j. e. (2006). the library and my learning community: first year students' impressions of library services. reference & user services quarterly, 46(2), 72-80. vondracek, r. (2007). comfort and convenience? why students choose alternatives to the library. portal: libraries and the academy, 7(3), 277-293. walter, s., & eodice, m. (2005). meeting the student learning imperative: supporting and sustaining collaboration between academic libraries and student services programs. research strategies, 20(4), 219225. doi:10.1016/j.resstr.2006.11.001 weiler, a. (2005). information-seeking behavior in generation y students: motivation, critical thinking, and learning theory. journal of academic librarianship, 31(1), 4653. williamson, k., bernath, v., wright, s., & sullivan, j. (2007). research students in the electronic age: impacts of changing information behavior on information literacy needs. communications in information literacy, 1(2), 47-63. evidence based library and information practice 2012, 7.3 23 appendix a student expectations of the research process 1. age: 2. sex:  female  male 3. in what year did you graduate from high school? 4. where are you from?  nova scotia  another canadian province:  somewhere else in the world: 5. in what year of study are you (include time spent at other universities)?  1st year  2nd year  3rd year  4th year  more than 4 years  other 6. major: (if undecided, give as much information as possible: arts, social science, science, professional studies.) 7. are you a full-time (3+ courses) or part-time (1-2 courses) student?  full-time  part-time 8. are you working at a job while going to school?  yes _______ hours per week.  no 9. are you volunteering anywhere while going to school?  yes _______ hours per week.  no 10. which of the following do you own or have easy access to? check all that apply.  laptop computer  desktop computer  internet: high-speed (fast connection)  internet: modem access (slow connection over phone line)  wireless internet  cell phone  cell phone with text messaging evidence based library and information practice 2012, 7.3 24  blackberry or similar pda  ipod/mp3 player (or similar device)  gaming consoles or devices high school research: q. 11-13 the next three (3) questions ask you to reflect on your experiences in high school. if you have been out of high school for too long or can’t remember, skip to question 14. 11. in high school did your teachers allow you to use google (or other search engines) to do research for your assignments?  yes, all the time  most of the time  sometimes  rarely  never  not sure 12. in high school did you ever use a research database (such as ebsco’s academic search) to do research?  yes, all the time  most of the time  sometimes  rarely  never  i’m not sure what a databases is 13. in high school, when teachers gave out an assignment, did they discuss the issues of citation and plagiarism with you?  yes, all the time  most of the time  sometimes  rarely  never  i’m not sure what citation and plagiarism are 14. during the last year, approximately how many hours per week did you spend reading books, magazines, journals and/or newspapers for school, work and/or pleasure? reading could be in print or online, but shouldn't include general web browsing, e-mail or gaming.  0-3 hours per week  4-7 hours per week  8-11 hours per week  12-15 hours per week  16-19 hours per week  20+ hours per week 15. during the last year, approximately how many hours per week did you spend online, e.g., general web browsing, facebook, e-mail, gaming, etc. evidence based library and information practice 2012, 7.3 25  0-3 hours per week  4-7 hours per week  8-11 hours per week  12-15 hours per week  16-19 hours per week  20+ hours per week 16. do you feel prepared to do university-level research?  yes, i feel very prepared  i am somewhat prepared  i’m not sure  i don’t think i’m very prepared  no, i know i’m not prepared 17. how would you rate your academic research skills? (your ability to find academic or scholarly information.)  excellent i almost always find what i’m looking for  good i usually find what i need  average sometimes it takes me awhile to find something useful  not very good i’m usually disappointed with my results  terrible i never find what i need 18. who do you think is responsible for you learning the skills necessary to succeed at carrying out university-level research? rank the following in order from 1 (most responsible) 6 (least responsible) professors librarians me student affairs (through their academic support programs) my friends or family other students 19. what percentage of your research material do you expect to find using google?  0-20%  21-40%  41-60%  61-80%  81-100% 20. you have just been assigned a 10-page paper/assignment for an introductory course. approximately how long would you spend on the research component of this assignment (before you start the real writing)?  1-3 hours  3-5 hours  5-7 hours  7-9 hours evidence based library and information practice 2012, 7.3 26  10+ hours 21. how much time do you expect to spend reading each week to keep up with all your courses? reading could be in print or online, but shouldn't include general web browsing, e-mail or gaming.  0-3 hours per week  4-7 hours per week  8-11 hours per week  12-15 hours per week  16-19 hours per week  20+ hours per week 22. how would you rate your overall internet searching skills?  excellent i almost always find what i’m looking for  good i usually find what i need  average sometimes it takes me awhile to find something useful  not very good i’m usually disappointed with my results  terrible i never find what i need 23. who do you think has the best internet searching skills? rank 1 (best) 4(worst). professors librarians students people working in it/computing 24. please indicate which of the following resources you expect to use for research purposes. check all that apply. electronic resources:  e-mail  im/chat (instant messaging)  google (or other search engines)  wikipedia  library web site  facebook, myspace (or similar social networking sites)  msvu’s online library catalogue (novanet)  other online catalogue (public library)  databases like ebsco’s academic search to find articles  online journals  online magazines  online newspapers  e-books (online books, reports)  scholarly, government, professional web sites  general, popular web sites  films, documentaries, dvds  games: computer or virtual evidence based library and information practice 2012, 7.3 27  youtube (or similar video sites)  itunes (or similar music sites)  flickr (or similar photo sites)  blogs  rss feeds  podcasts print resources:  journals  magazines  newspapers  books, reports from msvu library  books, reports from other universities (dalhousie, smu)  books, reports from the public library  encyclopedias  dictionaries  archival (historical) material other:  art  music  experts in the field  your own experiences  other:______________________________ 25. which of the following people do you expect to go to if you need help with your assignments? check all that apply.  professors  librarians  friends  classmates  student services (writing centre)  family  other: _______________________ appendix b faculty expectations of student research 1. primary department: 2. how long have you taught at the university level (msvu and other institutions): 3. are you a full-time or part-time faculty member?  full-time faculty  part-time faculty evidence based library and information practice 2012, 7.3 28  other 4. while students were in high school do you believe teachers allowed them to use google (or other search engines) to do research for their assignments?  yes, all the time  most of the time  sometimes  rarely  never  not sure 5. while students were in high school do you believe they ever used a research database (such as ebsco’s academic search) to do research?  yes, all the time  most of the time  sometimes  rarely  never  not sure 6. what percentage of first-year students do you think know what a research database is?  0-20%  21-40%  41-60%  61-80%  81-100%  not sure 7. when high school teachers give assignments to their students do you believe they discuss the issues of citation and plagiarism with them?  yes, all the time  most of the time  sometimes  rarely  never  not sure 8. what percentage of first-year students do you think know what citation and plagiarism are?  0-20%  21-40%  41-60%  61-80%  81-100%  not sure 9. during the last year, approximately how many hours per week do you think first-year students spent reading books, magazines, journals and/or newspapers for school, work and/or pleasure? reading could be in print or online, but shouldn't include general web browsing, e-mail or gaming. evidence based library and information practice 2012, 7.3 29  0-3 hours per week  4-7 hours per week  8-11 hours per week  12-15 hours per week  16-19 hours per week  20+ hours per week  not sure 10. do you believe the majority of first-year students are prepared to do university-level research?  yes, they are very prepared  they are somewhat prepared  i’m not sure  i don’t think they are very prepared  no, they are not prepared at all 11. how would you rate first-year students’ academic research skills? (their ability to find academic or scholarly information?)  excellent they almost always find what expect  good they usually find what i expect  average they find a combination of useful and un-useful results  not very good i’m usually disappointed with their results  terrible they never find what i expect  not sure 12. who do you think is responsible for first-year students learning the skills necessary to succeed at carrying out university-level research? rank the following in order from 1 (most responsible) 6 (least responsible) professors librarians the students themselves student affairs (through their academic support programs) friends or family other students 13. in first-year classes, what percentage of research material do you believe students expect to find using google?  0-20%  21-40%  41-60%  61-80%  81-100%  not sure 14. in first-year classes, what percentage of research material do you want/expect students to find using google?  0-20%  21-40% evidence based library and information practice 2012, 7.3 30  41-60%  61-80%  81-100%  not sure 15. you have just assigned a 10-page paper/assignment to an introductory class. approximately how long would you expect students to spend on the research component of this assignment (before the real writing starts)?  1-3 hours  3-5 hours  5-7 hours  7-9 hours  10+ hours  not sure 16. how much time do you expect first-year students to spend reading each week in order to keep up with all their course work (not just your course)? reading could be in print or online, but doesn’t include general web browsing, e-mail or gaming.  0-3 hours per week  4-7 hours per week  8-11 hours per week  12-15 hours per week  16-19 hours per week  20+ hours per week  not sure 17. how would you rate your first-year students’ overall internet searching skills?  excellent they almost always find what i expect  good they usually find what i expect  average they find a combination of useful and un-useful results  not very good i’m usually disappointed with their results  terrible they never find what i expect  not sure 18. please indicate which of the following resources you expect (want) first-year students to use for research purposes. check all that apply. electronic resources:  e-mail  im/chat (instant messaging)  google  wikipedia  library web site  facebook, myspace (or similar social networking sites)  msvu’s online library catalogue (novanet)  other online catalogue (public library) evidence based library and information practice 2012, 7.3 31  databases like ebsco’s academic search to find articles  online journals  online magazines  online newspapers  e-books (online books, reports)  scholarly, government, professional web sites  general, popular web sites  films, documentaries, dvds  virtual games  youtube (or similar video sites)  itunes (or similar music sites)  flickr (or similar photo sites)  blogs  rss feeds  podcasts print resources:  journals  magazines  newspapers  books,reports from msvu library  books, reports from other universities (dalhousie, smu)  books,reports from the public library  encyclopedias  dictionaries  archival (historical) material other:  art  music  experts in the field  your own experiences  other: ___________________________ 19. which of the following people do you expect first-year students to go to if they need help with their assignments? check all that apply.  professors  librarians  friends  classmates  student services (writing centre)  family  other: _______________________ article   laying the groundwork for a new library service: scholar-practitioner & graduate student attitudes toward altmetrics and the curation of online profiles   kathleen reed assessment and data librarian vancouver island university nanaimo, british columbia, canada email: kathleen.reed@viu.ca   dana mcfarland eresources librarian vancouver island university nanaimo, british columbia, canada email: dana.mcfarland@viu.ca   rosie croft university librarian royal roads university victoria, british columbia, canada email: rosie.croft@royalroads.ca     received: 15 feb. 2016   accepted: 21 mar. 2016       2016 reed, mcfarland, and croft. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – in order to inform a library service related to creating and maintaining online scholarly profiles, we sought to assess the knowledge base and needs of our academic communities. participants were queried about use, issues, and attitudes toward scholarly profile and altmetric tools, as well as the role librarians could play in assisting with the curation of online reputation.    methods – semi-structured interviews with 18 scholar-practitioners and 5 graduate students from two mid-sized universities.   results – while all participants had googled themselves, few were strategic about their online scholarly identity. participants affirmed the perception that altmetrics can be of value in helping to craft a story of the value of their research and its diverse outputs. when participants had prior knowledge of altmetrics tools, it tended to be very narrow and deep, and perhaps field-specific.  participants identified time as the major barrier to use of scholarly profile and altmetrics tools.   conclusions – librarians are well-placed to assist scholar-practitioners who wish to curate an online profile or use altmetrics tools. areas of assistance include: personalized support, establishment of goals, orientation to specific tools, orientation to altmetrics and scholarly promotion landscape, preparing users for potential difficulties, discussing copyright implications, open access education, and guidance with packaging content for different venues and audiences.   introduction   until recently, measurement of scholarly influence was the exclusive domain of specialized citation indexing tools, relying principally on citations to articles published in select journals to construct an understanding of individual scholar reputation. parallel developments in recent years have brought analytics and internet search optimization tools to any savvy internet user. these developments have given rise to altmetrics, the process of expanding the measurement of scholarly impact to include the social web, beyond traditional citations. the convergence of the tools and models of the past with the analytical tools of the online environment opens a space for innovation, and poses an interesting challenge for libraries to define a role.   for librarians to shape a service that may assist scholar-practitioners and graduate students to find their way with altmetrics and scholarly promotion, more needs to be known about how these groups perceive and engage with the tools available to them. our research explores this terrain, querying participants about what issues they face when trying to establish, grow, or measure a scholarly presence on the web, as well as how they negotiate these issues. from this we discern and suggest ways in which academic librarians can assist scholar-practitioners and students to create, discover, and manage elements of online reputation using traditional and emerging tools for measuring influence.   literature review   the idea of altmetrics dates from 2010, when jason priem, a doctoral candidate at the university of north carolina at chapel hill, first used the term on twitter. an influential manifesto followed, articulating the limitations of traditional filters of academic quality: article citations and journal impact factor (priem, taraborelli, groth, & neylon, 2010). subsequently, heather piwowar, researcher and altmetrics advocate, cited four potential advantages to altmetrics:   a more nuanced understanding of impact, showing us which scholarly products are read, discussed, saved and recommended, as well as cited. often more timely data, showing evidence of impact in days instead of years. a window on the impact of web-native scholarly products like datasets, software, blog posts, videos, and more. indications of impacts on diverse audiences, including scholars, but also practitioners, clinicians, educators, and the general public (piwowar, 2013)   despite the advantages articulated by piwowar, some researchers have balked at altmetrics, questioning if tweets and blog post mentions are a real indicator of impact (scott, 2012). others are concerned that altmetric data can be manipulated. one study tested how easy it is to game google scholar metrics, concluding it is “simple, easy, and tempting” (howard, 2013). however, attempting to game one’s scholarly influence is nothing new in academia; as the “publish or perish” model continues to weigh on researchers, there has been an increasing number of paper retractions in journals due to research fraud and increased journal vigilance (fang, steen, & casadevall, 2012; steen, 2011).   recognizing both the value and the uncertainty of altmetrics, many scholars and librarians including the authors of this paper choose to see traditional informetrics indicators and altmetrics as complementary (costas, zahedi, & wounters, 2015). rousseau and yee (2013) suggest that cronin & weaver’s 1995 term “influmetrics” is a more useful term than altmetrics, while “allmetrics” is used by plum analytics, a for-profit scholarly analytics company recently purchased by ebsco.   in our review of the literature, we noted that institutions focussed on applied research were not represented in altmetrics discussions. as librarians from universities with emphasis on applied programming, this lack of representation surprised us; this type of institution is home to many scholar-practitioners who undertake significant work that is often not published through traditional channels. for example, non-governmental organization reports, briefing notes, papers of all kinds, and instructional resources are frequently published by applied scholar-practitioners, but up until recently have been difficult to track for impact.    scholar-practitioners have been defined as a group who characteristically see their work “in relation to broader organizational, community, political, and cultural contexts [and] explicitly reflect on and assess the impact of their work” (mcclintock, 2004). in our applied and teaching-focused institutional settings, we consider that scholar-practitioners include both those who maintain professional practice in their teaching area, and those for whom teaching itself is the practice.   aims   as altmetrics now offers a way to gauge the level of influence that diverse types of published scholarly work may have, we believe there is an opportunity for libraries to define new services related to scholarly profile curation and management. in order to derive greatest benefit from altmetric tools, the intentional development of an online scholarly profile is necessary. “online scholarly profile” refers to a curated representation of the digital footprints left by scholar-practitioners on the web. this might include published articles, books, tweets, blogs, datasets, reports, comments, presentations, academia.edu profiles, or any other data that is published online by a scholar-practitioner. librarians have the opportunity to advise scholar-practitioners in how to most effectively keep track of and represent the influence of all these types of online evidence, and the implications of tool choice.   in order to shape a new evidence-based scholarly profile service at our libraries, we sought to assess the awareness and needs of our communities, asking the following questions:   what issues do scholars and graduate students who are also practitioners face when trying to establish, grow, or measure a scholarly presence on the web? how do scholars and graduate students who are also practitioners negotiate issues related to establishing or growing a scholarly presence on the web? in what ways can academic librarians assist scholar-practitioners to create and manage online reputation using traditional and emerging tools for measuring influence?   methods   in order to explore scholar-practitioner and graduate student attitudes and practices related to altmetrics and online scholarly profiles, semi-structured interviews occurred with 18 scholar-practitioners and 5 graduate students from two institutions: vancouver island university and royal roads university. interviews ranged from 45 minutes to 1.5 hours.   participants were chosen based on factors that included publication record, including both traditional and alternative channels of dissemination, and demonstrated interest in new modes of scholarly communication and networking. at vancouver island university, the annual institutional report on scholarly activity, which lists the scholarly output and service of all faculty members, was consulted. from this guide we identified individuals who were actively engaged in scholarship. we then researched these scholars through publicly accessible tools like google scholar, mendeley, and impact story to see if the work of these individuals was represented. subsequently, we contacted these potential participants, requested an interview, and offered to show individuals the impact of their work using reports and examples drawn from altmetrics tools, as available. graduate student participants were recruited via snowball sampling.     scholar-practitioners and students who chose to participate were queried about their impressions of traditional scholarly metrics, engagement with altmetrics and social media, and opinions regarding academic library potential roles in providing services related to altmetrics and scholarly reputation. participants were also shown the impact of their scholarly work using reports drawn from altmetrics tools, as available. traditional to emerging metrics tools including web of science, google scholar, mendeley, impact story, and plum analytics were demonstrated and discussed. interview data was transcribed using ms word, printed, and then coded by hand. the authors then met to discuss the themes that emerged from the transcripts.   results   participants were first queried about how they use the internet professionally, and their knowledge of impact and altmetrics. every participant had searched for his or her own name through google for professional purposes. very few of the participants were strategic about their online presence, not having built google scholar profiles, searched their names through altmetrics tools, or otherwise had a systematic approach to online identity. there was general awareness of impact related to journal impact factor, and a general perception that this calculation was important, although most participants could not articulate why.   the term ‘altmetrics,’ and associated tools, were new to most participants. participants affirmed the perception that altmetrics can be of value in helping to craft a story of the value of their research and its diverse outputs, as opposed to simplistic benchmarking. when participants had prior knowledge of altmetrics tools, it tended to be very narrow and deep, and perhaps field-specific. for example, one interviewee told us about a tool called the carbon capture report (http://wwwcarboncapturereport.org). this site tracks and ranks the tone and activities of individuals who post and publish about climate change and alternative energy in social media.   whether participants felt that altmetrics and scholarly profile tools were important to their own careers depended on several variables:   career stage: younger scholar-practitioners recognized that these tools are becoming increasingly important. several participants close to retirement also recognized value, not necessarily for themselves, but for their graduate students. we had several graduate students participate in our study at the urging of their supervisors. career aspirations (or lack thereof): several participants did not plan on changing jobs, so felt that altmetrics were unimportant to them. institutional structure and value placed on research: most vancouver island university participants commented that the lack of a tenure-based advancement system at their institution, and weak incentives to pursue an active program of scholarship, corresponded to a low incentive to track influence.  home discipline: scientists in our sample were more familiar with the tools that we presented and quickly understood their importance. we believe this to be related to a predisposition in the sciences toward quantitative methods. for example, upon hearing about orcid (http://orcid.org) numbers (persistent digital identifier numbers for individual researchers), one chemist in our study immediately perceived a parallel with chemistry abstract service (cas) numbers.  major grant applicants: those participants who actively applied for major research grants were enthusiastic about the possibility of demonstrating their impact beyond traditional means.   thus, to what extent interviewees were engaged with scholar-practitioner profile and altmetrics tools depended not only on whether people knew about them or not, but also on the perception of direct relevance. philosophical beliefs about social media and privacy boundaries also factored into decisions to use profile and altmetric tools. some participants were averse to posting about themselves online, while others felt it impossible to separate their private and public selves. other participants mentioned that how one is expected to behave within particular disciplines may play a role. in more traditional disciplines, promoting one’s own work on social media would be frowned upon, and the person labelled a braggart. in other fields, self-promotion is completely accepted.   by far the most common barrier to the use of these tools that participants noted was time. it takes substantial time to set up, track, and ensure that profiles are continuously updated. some participants mentioned that they delegated this type of work to graduate students. other participants mentioned not having the time to learn about and determine which particular tool(s) is best for their disciplines. the stage of development of tools also was influential in how and whether people decided to engage with tools. with high production values, including effective and informative visual displays customized to the scholar, plumx (https://plu.mx) was a favourite tool among participants. sites that looked less polished and had fewer features inspired less confidence among participants.   participants expressed some skepticism over what altmetrics tools could accomplish, how these sorts of measures might be ‘gamed,’ and how to make meaningful comparisons across or within disciplines. also, through a labour-management lens, there was some concern about faculty being reduced to numbers and then compared to one another.    many of the participants were surprised to see their professional identities represented online, having never or seldom uploaded professional content to the web themselves. in many of these cases, conference organizers, journal staff, or co-authors uploaded abstracts, conference presentations, and articles without active participation from our study participants. seeing how evidence of their work is already represented online, whether they like it or not, led participants to feel a need for guidance and assistance. when the interviewees were asked if such a service might be situated in the library, all affirmed a role for librarians. there was also general support for the idea that indicators of influence related to scholarly publication could be more diverse than citations, capturing different kinds of use, and that the influence of non-journal publications could also be indicated.   discussion   a number of ways in which librarians can support scholar-practitioners and graduate students with profile and altmetrics tools have been identified through this research:   personalized support   as outlined above, participants’ attitudes toward altmetrics and a potential library service in this area varied substantially based on stage of career, field of scholarly activity, institutional value placed on research and publishing in faculty role, formality of institutional promotion requirements, dependence on traditional impact measures to obtain and retain funding, time, awareness of tools, perceived readiness of tools, and disciplinary view of tools. consequently, users’ objectives will vary greatly, and any service needs to be highly personalized in nature. while an online library guide to altmetrics and curating scholarly profiles online would be a useful tool to introduce users to the concept of altmetrics and scholarly profiles, and may provide a starting point for the enterprising and self-motivated, it is only a first step.   establish goals   participants in this study listed several different reasons for wanting to build and measure a scholarly profile online: tracking the influence of their work outside of traditional journal literature, building an audience outside of the academy, and making connections to other researchers with interest in the same area but perhaps in other disciplines, and attracting graduate students and funders. librarians can highlight the opportunities and strengths afforded by various tools, and recommend particular services based on the goals of individuals.    tools   as professionals often connected with both social media and scholarly communication, academic librarians are in a prime spot to act as guides in orienting scholar-practitioners and students to the altmetrics environment and social media conventions for academic purposes.   one of the main places where librarians can assist scholar-practitioners and students who are starting out with establishing a scholarly presence online is to go over the tools available and make recommendations as to which to use. with so many tools and sites, participants asked: which tools offer the features that i am interested in? which ones are my colleagues using? what’s best for my discipline? one participant stated that developing a “road map” of exactly what tools to use, and what had to be done to maintain them effectively, would be helpful.   orientation to the altmetrics and scholarly promotion landscape   librarians can provide education and guidance on specific actions to take and tools to use, but we also have a role in encouraging users in developing nuanced perceptions of social media. research participants expressed a wide spectrum of attitudes toward social media and alternative channels of disseminating their work. these attitudes ranged from seeing social media as fraught with and characterized by flame wars and egocentrism to perceiving it as a critical, emerging venue for dissemination.     a key hurdle to utilizing social media is an aversion to self-promotion. a majority of participants in this study commented on being uncomfortable with promoting their own work in social media, and creating what one participant referred to as a scholarly “egosystem.” another participant worried about being “dismissed as a braggart.” although this study examined only a small sample of 23 individuals, trends emerged related to self-promotion conventions within disciplines. participants from interdisciplinary backgrounds, for example, had less concern about self-promotion than those in natural sciences. as all participants were of canadian nationality, we couldn’t help but wonder if our stereotypical national politeness and aversion to limelight played a role. a cross-cultural study of how academics view online promotion would be useful in determining what demographics most influence participation. regardless of the motivation for shying away from social media, librarians can assist service users by drawing a distinction between useful promotion of one’s own scholarly work, and egoism.     preparing users for the tough times   at the same time that they aspired to a broader audience, a handful of participants expressed fear that once released into the digital wilds beyond the academy, their work might be misinterpreted or misused. one participant explained that this already had happened to him: an article on climate change was cited as proof of “intelligent” design. others expressed feelings of frustration and uncertainty about how online data they create might be misused. issues with accounts being hacked, services spamming email accounts, and worries that research on controversial topics could be used against a scholar or their broader network (i.e., their graduate students) were also highlighted as serious concerns by participants. if librarians are going to assist our users in setting up profiles online and using various tools, we also share in a responsibility to prepare them for and assist when tough times arrive – including hacks, misrepresentation, and spamming. making users aware of the potential downside of engaging with profile tools should be a part of any service the library provides.    copyright   as scholarly work becomes more accessible and visible, scholar-practitioners and students have amplified responsibility to ensure that their published work is in compliance with copyright law. librarians can assist by educating about creative commons resources and models, enhancing copyright awareness on campus, and reviewing work that individuals wish to release to the public.   packaging content   many academics are accustomed to writing in a particular style for a scholarly audience. work produced for this specialized audience will not necessarily be accessible to the general public. as one of our participants commented,   i can’t just take out a [research finding], plop it on the internet, because you’re not going to reach a bigger audience. you’re probably going to reach people who already know about your work through citations and things like that. translational things take time, repackaging the content for a wider audience. ... i think as a scientist you have an obligation to share knowledge in as many different formats as you possibly can.    librarians can help scholar-practitioners and graduate students to identify their audience segments, and to present work in ways that are the most suitable for various forums.    open access (oa) education   conversations about scholarly profile flow naturally into a discussion of making work available in oa form, ensuring that the opportunity exists for scholarship to be viewed as widely as possible. librarians may assist scholar-practitioners to assert their author rights with publishers in order to republish content in oa form, as well as provide education on suitable oa repositories. there is some confusion among scholar-practitioners on the subject of appropriate open access repositories in which to deposit content; for-profit sharing sites like academia.edu or research gate are not in fact open, and do not comply with the tri-agency open access policy on publications (government of canada, 2015).   promoting the new service   in interviews, participants suggested a number of ways that librarians could reach researchers who would be interested in a library-led profile and impact service. several suggested talking to people on campus concerned with knowledge mobilization, as these are likely to be natural allies. knowledge mobilization is all about getting work out of the academy and making it useful in the community, so measuring the ways in which this happens is a relatively “easy sell” to these potential allies.   as mentioned above, many participants did not know that their work was already represented online. if scholar-practitioners are unaware of how their work is being disseminated and discussed in non-traditional venues, emailing publicly accessible impact reports can entice them to learn more and to take an active role in shaping their online profiles.   aside from direct contact with people whose work is already represented online, librarians can approach scholar-practitioners who are working on research projects – particularly those who are in the planning stages. in addition to providing data management planning guidance, librarians can assist scholar-practitioners in planning where, when, and how to mobilize and track use of the knowledge being produced and disseminated.   altmetrics are an excellent springboard to use to talk to graduate students and senior undergraduates, who are engaged in capstone projects, about beginning to curate a scholarly profile. while many of these students have not published in journals, they will often begin by giving posters or presentations at conferences. by uploading their posters to their institutional repository or their slides to a site like slideshare, these works are made accessible online and can be tracked for views and downloads. teaching graduate students to seed their profiles early and to deliberately apply online tools for understanding their scholarly influence serves them as they seek to establish themselves in their fields.   conclusion   the opportunity for libraries going forward is to augment their existing role with respect to scholarly profiles: libraries have long provided access to and interpreted citation indexes, cited references, and the traditional scholarly publication cycle. the emerging scene challenges libraries to support scholar-practitioners and students to engage with a broader, volatile, and evolving environment in which much may be gained or forfeited depending on how reputation is curated.   having demonstrated that an interest and need exists on our campuses for a library-led service related to scholarly profile and impact measurement, the next step in our research is to pilot a highly individualized consultation service for scholar-practitioners and students. as traditional library services change or disappear, we see this service as a natural extension of library expertise, and a meaningful way to provide outreach and support to our scholar-practitioners and graduate students.   references   costas, r., zahedi, z., & wounters, p. (2015). do “altmetrics” correlate with citations? extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. journal of the association for information science and technology, 66(10), 2003–2019.   cronin, b., & weaver, s. (1995). the praxis of acknowledgement: from bibliometrics to influmetrics. revista española de documentación científica, 18(2), 172–177. http://dx.doi.org/10.3989/redc.1995.v18.i2.654   fang, f. c., steen, r. g., & casadevall, a. (2012). misconduct accounts for the majority of retracted scientific publications. proceedings of the national academy of sciences of the united states of america, 109(42), 17028–33. http://dx.doi.org/10.1073/pnas.1212247109   government of canada. tri-agencies. (2015). tri-agency open access policy on publications. retrieved 14 march, 2016, from http://www.science.gc.ca/default.asp?lang=en&n=f6765465-1.   howard, j. (2013). rise of “altmetrics” revives questions about how to measure impact of research. chronicle of higher education. retrieved from https://chronicle.com/article/rise-of-altmetrics-revives/139557/   mcclintock, c. (2004). scholar practitioner model. in encyclopedia of distributed learning. thousand oaks, ca: sage. retrieved april 20, 2016 from http://knowledge.sagepub.com/view/distributedlearning/n134.xml   piwowar, h. (2013). altmetrics: what, why, and where?”. asis&t bulletin, 39(4), 8–9. retrieved from https://www.asis.org/bulletin/apr-13/aprmay13_piwowar.html   priem, j., taraborelli, d., groth, p., & neylon, c. (2010). altmetrics: a manifesto. retrieved february 15, 2016, from http://altmetrics.org/manifesto/   rousseau, r., & ye, f. y. (2013). a multi-metric approach for research evaluation. chinese science bulletin, 58(26), 3288-3290. http://dx.doi.org/10.1007/s11434-013-5939-3   scott, n. (2012). altmetrics are the central way of measuring communication in the digital age but what do they miss? in london school of economics and political science blog. retrieved february 15, 2016, from http://blogs.lse.ac.uk/impactofsocialsciences/2012/12/17/scott-altmetrics-central-digital-whats-missing/   steen, r. g. (2011). retractions in the scientific literature: is the incidence of research fraud increasing? journal of medical ethics, 37(4), 249–253. http://dx.doi.org/10.1136/jme.2010.040923     appendix – interview questions   tell me your impressions of scholarly metrics as they exist now? how do you see yourself in this system?    have you searched for yourself and your work online in the last year? how did you do so?   have you tried to build awareness of your work online? if so, how? if not, why not?   have you experimented in non-traditional venues? if not, why not?    do you see scholarly metrics as being important to building your academic career? if so, how?   do you have a professional presence in social media? what do you consider to be the advantages and disadvantages in terms of building reputation as a scholar-practitioner?   do you think your discipline or field is receptive to innovation with respect to forms of publication and associated metrics?   how do you integrate (or separate) your scholarly and personal lives online? describe this process (difficulties, technologies, etc.).   midway through the interview, demonstrate altmetrics in mendeley, google scholar, reader meter, impact story.org.   give us your impression of these tools.    would you be interested in meeting with other scholars/colleagues to discuss possible uses of new ways to measure influence across disciplines and emerging forms of scholarly communication?
   would you be interested in a workshop focused on exploring how to use library or social tools to assess your influence as a scholar-practitioner?   would you be interested in an appointment-based service where you could consult with a librarian on a practical strategy for managing and monitoring your online presence as a scholar-practitioner?   an analysis of the effect of saturday home football games on physical use of university libraries research article   an analysis of the effect of saturday home football games on physical use of university libraries   kerry sewell research librarian for the health sciences library associate professor william e. laupus health sciences library east carolina university greenville, north carolina, united states of america email: browderk@ecu.edu   received: 4 mar. 2021                                                                accepted: 25 oct. 2021        2021 sewell. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   data availability: sewell, k. (2021). effect of home football games on library gate counts [datasets, public data sources, and data analysis code for spss]. osf. https://osf.io/wpzx7/. gate counts_libraries_footballsaturdays [data visualization] tableau public. https://public.tableau.com/app/profile/kerry.sewell/viz/gatecounts_libraries_footballsaturdays/alsaveragefootballeffects    doi: 10.18438/eblip29942     abstract   objective – library science literature lacks studies on the effect of external events on the physical use of libraries, leaving a gap in understanding of would-be library patrons’ time use choices when faced with the option of using the library or attending time-bound, external events. within academic libraries in about 900 colleges and universities in the us, weekend time use may be affected by football games. this study sought to elucidate the effect of external events on physical use of libraries by examining the effect of saturday home football games on the physical use of the libraries in a large, academic institution.   methods – this study used a retrospective, observational study design. gate count data for all saturdays during the fall semesters of 2013-2018 were collected for the two primary libraries at east carolina university (main campus’ academic library services [als] and laupus, a health sciences campus library), along with data on the occurrence of home football games. the relationship between gate counts and the occurrence of home football games was assessed using an independent samples t-test.   results – saturday home football games decreased the gate count at both als and laupus. for als, the mean physical use of the library decreased by one third (34.4%) on saturdays with a home game. for laupus, physical use of the library decreased by almost a quarter (22%) on saturdays with a home game.   conclusion – saturday home football games alter the physical use of academic libraries, decreasing the number of patrons entering the doors. libraries may be able to adjust staffing based on reduced use of library facilities during these events.     introduction   students, faculty, and staff in higher education inhabit complex social worlds in which choices related to time and resource use are made to satisfy a variety of needs. for students, needs include scholarly work (research and learning), income-driven work, and social connection. excepting off-campus employment, campus buildings and services often provide the physical spaces in which these needs are fulfilled. libraries in higher education are one such space, providing areas for individual and group study, serving both social and individual scholarly needs.   demand for library spaces to support scholarly work spans the entire week and many large academic libraries meet the demand by providing extensive hours seven days per week. the widespread demand for expansive operational hours is not met with even levels of use throughout the week. shifts in the physical usage rates of academic libraries are broadly documented, with usage rates noted as declining on wednesdays and reaching a nadir on saturdays (dotson & garris, 2008; ferria et al., 2017; scarletto et al., 2013).   although general patterns of change in physical use of libraries throughout the typical week may be widely observed and documented, less is known about the effects of singular, large community events, many of which occur on weekends, on physical use of libraries. a broad, multi-database search (proquest search, lisa, lista, eric, econlit) for studies examining the effect of any major external event (e.g., sports, parades, natural disasters, local festivals) on the physical use of academic and public libraries revealed a dearth of literature in this area. the lack of fruitful searches suggests scholarly inattention to the ways that potential library users alter their behaviours in response to large-scale external events.   the reasons for the paucity of information on the effect of external events on physical use of libraries are unclear. it may be that the effects of such events are assumed as being known and thus not worthy of scholarly attention. however, the test of such assumptions is critical to our understanding of library users and their time use choices, as well as an opportunity to ensure that assumptions are matched by data. additionally, if tests of such assumptions are verified—or disproved—by data, decisions related to staffing and hours of operation around such external events may be made with more confidence. studies of the effects of external circumstances and large-scale events and circumstances (friday the 13th, hurricanes, sporting events) on use of emergency room services and outcomes serve as examples of how such tests can lead to better decisions about service delivery and staffing (drobatz et al., 2009; jena et al., 2017; jerrard, 2009; lo et al., 2012; mcgreevy et al., 2010; protty et al., 2016; schuld et al., 2011; shook & hiestand, 2011; smith & graffeo, 2005).   one such external weekend event within university life at nearly 900 american colleges and universities (next college student athlete (ncsa), n.d.) is a home varsity football game. although football games can occur on other weekdays, saturday home games are an especial draw due to lack of time conflicts with other pursuits. intramurally, home football games offer various game-related social and financial opportunities for students and faculty (chen et al., 2012; coates & depken, 2006; hardin, 2019; lyon-hill et al., 2015).   whether economically or socially driven, time dedicated to game-related pursuits is time lost on scholarly activities. libraries, serving as study spaces and providers of ready access to electronic and print resources, traditionally occupy the role of central physical space for scholarly pursuits. if football games alter scholarly behaviours and outcomes, it would be reasonable to expect a change in the number of individuals entering the library—determined by gate counts—during saturdays when a home football game occurs. the interplay of physical use of the academic library and the draw of varsity sports, however, has not been studied. the effect of varsity sports on physical use of academic libraries is examined in this study.   background   east carolina university (ecu) is a large, public, doctoral university serving roughly 29,000 students (23,081 undergraduate students, 4,739 graduate students, 537 dental and medical students, and 1,383 unclassified students) (east carolina university, n.d.), located in rural north carolina. ecu offers 174 different degrees, spanning health sciences and academic campuses.  the university has two distinct, though proximate campuses, with operationally independent libraries serving each campus; academic library services (als) and its branch music library serve the academic affairs campus and laupus library (laupus) serves the health sciences campus. both libraries serve their respective campuses seven days per week, though the hours offered each day differ.   ecu also includes an athletics department that organizes and funds more than 25 different sports. the largest of the sports sponsored by ecu is the football program, which accounted for 14.5% of ecu athletics revenue and 19.1% of the operating expenses in fiscal year 2019 (ecu athletics fiscal sustainability working group, 2020). the ecu football program brings in large crowds each fall, averaging between 30,000-45,000 attendees at home games in the 2019 season (national collegiate athletic association, n.d.-a).   literature review   variations in physical use of the library   libraries collect and report a variety of statistics on an annual basis. some of the statistics relate to use of online resources and services, while a small set of statistics relate to the use of libraries’ physical collections and spaces, namely in-person reference services, circulation of physical items, and gate counts. among the statistics documenting physical use of the library, gate counts, defined as “the total number of persons physically entering the library in a typical week” (national center for education statistics, 2017, p. 3), are the sole statistic that provide information on overall building use. gate counters are common across academic libraries, collecting hourly tallies of the number of people entering the library throughout the day with relatively high reliability (phillips, 2016). although some libraries employ card swipe systems for some or all of their opening hours in order to limit entry to campus affiliates and to collect select demographic data on the patrons entering their doors, the statistic reported to outside stakeholders is nonetheless a simple tally of the number of physical entries. the data are used to communicate the performance and value of the library as well as trends in student behaviours. typically, the data are sent to internal and external stakeholders including professional organizations, university governing councils, and institutional research departments, who send the data out for periodic national reports on higher education.   while a significant portion of libraries collect gate count data and report it in aggregate to stakeholders, beyond these annual reporting obligations, the data are either not widely reported on library websites, not used, or the exact nature of the data usage is unknown (driscoll & mott, 2008; martell, 2007; terrill, 2018). the data may otherwise remain entirely unused or be reserved for internal operational decisions only, though one study examining the data used and impetuses for changing operating hours and staff scheduling at medium-sized college libraries found that a low percentage of libraries used gate count data to inform their staffing decisions (brunsting, 2008). published library studies that have analyzed gate counts and card swipes did so as part of an assessment of the success of extending hours (lawrence & weber, 2012; scarletto et al., 2013), changing service models (albanese, 2003; jones, 2011), or in the context of declining use of print resources and reference services over longer periods of time (martell, 2007; opperman & jamison, 2008). beyond these studies, libraries publish relatively little on variations in physical use of library spaces, neither as a function of shorter, defined periods of time nor in response to external events.   as previously noted, the studies including gate counts and user surveys for normal (non-exam) weeks in their data collection methods find common patterns. library usage is highest at the beginning of the week and then progressively declines between wednesday and saturday (dotson & garris, 2008; ferria et al., 2017; scarletto et al., 2013). similar patterns are observed in the use of library electronic resources usage throughout the week (clotfelter, 2011). these observations complement student time use surveys that find that, as the week progresses and students begin the “social weekend,” comprising thursday night through sunday morning (finlay et al., 2012), time use shifts away from scholarly activities to activities fulfilling psychosocial and financial needs, namely employment activities or formally or informally organized social activities (finlay et al., 2012; greene & maggs, 2015; moulin & irwin, 2017; orcutt & harvey, 1991).   the effect of sports events on scholarly behaviours   among the formally organized activities available on saturdays during the fall semester, at universities with large sports programs, home football games provide both economic and social opportunities (chen et al., 2012). for students employed in service positions, games may lead to more work hours to meet increased consumer volume and spending on game day (coates & depken, 2006; lyon-hill et al., 2015). more significantly, football games provide various levels of social participation in events and rituals associated with game day (cohen et al., 2014). this is particularly true for weekend games, which are less likely to conflict with academic and work schedules, therefore drawing larger crowds and allowing for more time-intensive social activities.   social rituals and events surrounding the home game include tailgates, pregame rituals, and parties, along with remote group viewing via televised coverage. home game rituals and events consume significant amounts of time. the duration of football games alone in the 2019-2020 season averaged 3 hours and 18 minutes (national collegiate athletic association, n.d.-b). additionally, pregame events add to the already considerable amount of time dedicated to football games. data on the duration of many pregame events are lacking. however, studies of student drinking behaviours on football game days imply considerable time dedicated to pregame activities, with data and anecdotal evidence suggesting that students spend an average of five or more hours drinking on game days (glassman et al., 2007) and that drinking begins early in the morning (derringer & french, 2015). data about tailgating duration is also lacking, though one survey of tailgaters reported most respondents (51%) indicated that tailgate set-up occurs 3-4 hours prior to kickoff (tailgating institute, n.d.). the duration of the game itself, combined with the time dedicated to pregame drinking and tailgating suggests that, on saturdays with home games, most of the day is consumed by football-related activities.   participation in these events represents a significant time-trade for all participants, producing a diversionary disruption in normal activities for university and local communities. for faculty and students, the disruption in normal activities may mean a disruption in academic pursuits. time dedicated to watching the home game or to engaging in game-related social rituals is time lost to engagement in scholarly behaviours promoting learning and research, representing an opportunity cost among both students and faculty. literature from the field of economics provides evidence of the “cost” of university-sponsored sports events in terms of scholarly outcomes, with most studies indicating that university sports impact student and faculty scholarly outcomes.   most studies examining the effect of college football on scholarly outcomes have examined the effect of football games on various student outcomes. several economics studies document the effect of winning seasons on the numbers of college applications in the following year along with gpas and sat scores of applicants (pope & pope, 2009, 2014; toma & cross, 1998). since these studies focus on potential students and do not provide evidence of changed scholarly behaviours among students already attending a university, they are not considered here.   economics studies examining relationships between university sports and scholarly outcomes among fully enrolled students have focused on changes in gpa and graduation rates during football seasons with higher win percentages. the majority of studies indicate that both gpas and graduation rates are affected by a successful football season, though the direction of the effect differs across studies. regarding gpas, one study indicated that gpa declines during winning football seasons, with a larger decline in male students’ gpas than female students’ (lindo et al., 2012), although another study comparing gpas among college athletes and non-athletes found that gpa increased during winning seasons (mixon jr & trevino, 2005). literature on the relationship between football and graduation rates suffers from a similar lack of clear direction of affect. one study reported that winning seasons lead to declines in graduation rates (tucker, 1992) while another study found no evidence of negative impact of winning seasons on graduation rates (rishe, 2003).   faculty members’ scholarly behaviours also appear to be influenced by successful sports seasons. one study examining the effect of winning football seasons on the number of pages published among economics faculty in over 100 different economics departments (shughart et al., 1986) found that a winning season negatively impacted the scholarly output of economics faculty members. the authors note that “a tradeoff exists between success on the gridiron and success in the journals…when the local team is winning, there is more of an incentive for a professor to put off doing research on another academic article” (shughart et al., 1986, pp. 48–50).   notably, almost all the studies of the effect of university-sponsored sports events on scholarship among students and faculty have focused on how football games change scholarly outcomes, without examining how university sports events directly alter scholarly behaviours. only one study captured actual behavioural changes related to university sports events. charles clodfelter’s (2011) treatise on the effects of “big-time” university-sponsored sports on various aspects of academic life includes a study of the effect of university sports-related events on the use of a comprehensive, widely-used digital library resource called jstor. the study examined how an annual, nationally viewed university sports-related event (“selection sunday,” prior to the start of the ncaa basketball tournament) affected the use of jstor materials at many universities. clotfelter found that the use of digital materials in jstor decreased by an average of 6.7% during the week following selection sunday and that the majority of that decline occurred during the first two days of the ncaa tournament following selection sunday. as clotfelter (2011) states:   unless there existed other, unmeasured factors at work, the record of jstor usage [in the study] implies that the ncaa tournament had a measurable influence on the pattern of work in research libraries… and it reflects the power of the demand for the entertainment provided by this form of big-time college athletics. (p. 64)   clotfelter’s study is the only study examining the effect of sports events on the use of library resources, namely digital resources. no studies have examined the effect of football on library resource usage, or on the effect of sports events on physical use of the library.   aims   this study seeks to test the effect of external events on physical use of academic libraries by examining the relationship between home football games and gate counts on saturdays. in doing so, it indirectly examines time use choices among would-be saturday library users. the study also complements the economics literature on the effect of big-time sports on scholarship. the null hypothesis for this study was that home football games have no effect on physical use of the libraries on saturdays. the alternative hypothesis was that football games have an effect on the physical use of libraries on saturdays.   methods and materials   this study used a retrospective, observational study design. as the dates of saturday home games are not prespecified by the libraries and vary each semester, this is a natural experiment.   this study used library hourly gate count data for als and laupus for each saturday during the fall semesters of the years 2013-2018. the gate counters in both libraries are mounted above the libraries’ entry doors and run on sensource hardware and software (sensourceinc.com). other collected data points included data on the university’s football schedule, namely the occurrence of football games for each saturday of the fall semesters for the years of interest as well as the location of games (home vs. away) for saturdays when a game was scheduled. additional data elements were gathered as well, e.g., total enrollment for each of the years studied, the start time for the football games, the week of the semester for each saturday recorded, and annual academic calendar events coinciding with football games (e.g., fall break and homecoming). excepting gate count data, all data points come from publicly available sources.   the years studied included dates when major weather events (hurricanes matthew and florence) severely impacted the area, shutting down the university and its libraries. those dates were eliminated from the data set. additionally, data points for saturdays when holiday weekends occurred and one or both libraries were closed were eliminated from the dataset for the affected library. the full dataset was separated into two unique datasets for each library and a limited set of variables retained for the library-specific datasets. the original raw dataset, the full, cleaned, dataset reflecting the eliminated data points, the library-specific smaller datasets, and the codebook are all located in an open science framework (osf) project space for this study (https://osf.io/wpzx7/).   following data cleaning, the .csv files were loaded into spss 25.0.0.1 (64-bit version for windows 10). minimal recoding in spss was undertaken and data were examined for general patterns of use throughout a typical semester for each library. initial exploratory analysis of the difference in means for saturdays with a home game vs. saturdays with no home game was performed using box plots for simple visual comparison. subsequently, an independent samples t test was performed for each dataset. independent samples t-tests are used to determine whether a difference between the means for two groups differ significantly—that is to say that the difference in the means is not due to chance.   the spss syntax used for recoding, exploratory analysis, and the independent samples t-tests are all also available on the osf project space for this study. additionally, information about the sources for the publicly available data points are provided in the wiki for the data component of the project space.   data visualization was performed in tableau public. all visualizations are published on the tableau public site for this project.   results   for the years 2013-2018, of the 15 weeks making up the typical fall academic calendar, most dates were retained. of the 90 weeks for which data was initially gathered, 87 weeks were retained for analysis of the effect of home football games on physical use of als on saturdays; for laupus on saturdays, 85 weeks were retained for analysis of the effect of home football games on physical use. although no analysis of the additional influence of win percentage on the effect of physical use of the libraries on saturdays with home games was performed, the years studied included years with higher win percentages as well as lower win percentages (maximum win percentage .769 in 2013 and minimum win percentage of .250 in 2016-2018). in this way, the years studied are representative of years with high and poor football performance.   regarding overall trends in physical use of the libraries on saturdays at ecu, the data reveal that physical use of the libraries climbs throughout the first half of the semester (weeks one through six) before falling markedly on the saturday of the seventh week, a weekend marking the beginning of fall break. during the second half of the semester, physical use of the libraries is generally higher than during the first half, with peaks on the saturdays of weeks 9, 11, and 15. the saturday of the 14th week marks a second nadir in the gate count data, coinciding with thanksgiving break. regarding the peaks in gate counts during the second half of the semester, it is posited that the looming deadlines for large-scale assignments and exams drive these late-semester peaks. of the late-semester peaks in usage, the highest average gate count occurs on the saturday of the fifteenth week, just before the period of final examinations begins. notably, the rising gate counts and the amplitude of the peaks during the 9th and 11th weeks during the 2nd half of the semester are more pronounced for als (figure 1) than for laupus (figure 2).   football games played at home drive the gate count at both als and laupus. physical use of the libraries decreased on saturdays with home games. for als, the mean physical use of the library decreased by one third (34.4%) on saturdays with a home game (639.28 +/182.27 (sd) vs. 976.28 +/501.67, p<.001). for laupus, physical use of the library decreased by nearly a quarter (22%) on saturdays with a home game (154.57 +/62.66 (sd) vs. 120.63 +/33.13, p=.005). the effect of saturday home football games on gate counts is more consistently apparent at als (figure 3) than laupus (figure 4), with all weeks showing some evidence of effect. for laupus, gate counts during weeks 7, 8, and 13 appear to be less affected by home football games.   figure 1 average saturday gate counts for als, fall semester, 2013-2018.   figure 2 average saturday gate counts for laupus library, fall semester, 2013-2018.   figure 3 effect of home football games on average saturday gate count by week of fall semester, als, 2013-2018.   figure 4 effect of home football games on average saturday gate count by week of fall semester, laupus library, 2013-2018.   discussion   the findings of this study indicate that library patrons are not immune to the events surrounding saturday home football games. the data indicate that some aspect of saturday home football games alters time use even among those who might otherwise physically access the library. the exact cause of alterations in time use for regular saturday users of the library cannot be determined by this study and warrant further study. however, multiple factors (financial, social, environmental) may account for changes in weekend use of the library during home football games.   financial needs may alter student use of libraries by increasing employment-related opportunities and demands. would-be patrons may be unable to come to the library due to employer needs for more staff to serve the influx of local and visiting consumers in service industry positions such as food service and hospitality. this same factor may not influence employment-related time demands for students at universities in densely populated metropolitan areas where sports events have less effect on service industries; the relative geographic isolation of ecu may have a more pronounced influence on time use related to student employment (agha, 2013; agha & rascher, 2016; deschriver et al., 2021).   the larger effect size of home football games on physical use of the main academic campus library (als) than for the health sciences campus library (laupus) suggests that undergraduates are more likely to be influenced by the events surrounding saturday home football games. undergraduate students represent a more substantial percentage of the student population enrolled in degree programs for the main academic campus than for the health sciences campus (83% vs. 60% respectively for years 2016-2019) (east carolina university institutional assessment, planning, and research, 2021). additionally, undergraduate students, particularly freshmen, are more likely to live near the main academic campus and its proximal football stadium and thus have greater access to the social events surrounding the game. the increased effect of games on physical use of als may also reflect a stronger need for social connection and university community identity-building for students in the undergraduate cohort, needs which football games are well-situated to fill. as anderson and stone (1981) note, “sports teams are symbolic representations of a community and can provide individuals a sense of belonging to that community” (as cited in robinson et al., 2005, p. 44).   the same proximity to games may change library use in an entirely different manner, through environmental changes affecting decisions to use the libraries. namely, games may lead to alterations in municipal environments, such as increased noise levels in areas near the stadium (chase & healey, 1995) and fraternity or sorority houses, and through changed traffic patterns (humphreys & pyun, 2018; tempelmeier et al., 2020). these environmental and traffic changes arising from football games may alter library user behaviours. for would-be saturday library users, traffic congestion and campus noise levels related to games would be more pronounced for als than for laupus, given differences in proximity to the stadium (2.7 km driving distance vs. 5.4 km driving distance respectively).   if social, rather than environmental and employment factors, drive variations in use of the libraries on home football game saturdays, the findings of this study serve as a subtle (and unverified) caveat to published studies examining differences between users and non-users of academic and health sciences libraries (kramer & kramer, 1968; lemaistre et al., 2018; soria et al., 2013, 2015, 2017; sridhar, 1994; thorpe et al., 2016; toner, 2008; turtle, 2005). these studies do indicate that meaningful differences between users and non-users of academic and health sciences libraries exist, from demographic and socioeconomic factors (e.g., age, major, class level) to other factors such as lack of awareness of resources, lack of time to use libraries, and use of materials elsewhere (soria et al., 2015; sridhar, 1994; toner, 2008; turtle, 2005), with differences in library use shown to relate to gpa and dropout rates among undergraduates (kramer & kramer, 1968; lemaistre et al., 2018; soria et al., 2013, 2017; thorpe et al., 2016). while these library use factors and outcomes hold true, users of academic libraries on saturdays may not differ from non-users in one regard—they may be swayed away from library use by the draw of social events that football games uniquely offer. the effect of football event participation on student outcomes is indicated in published economics literature. as this study did not test the reasons for lower use of the libraries on saturdays, this cannot be verified by the current study and is worthy of further examination.   limitations   ecu’s libraries do not employ card swipe systems during saturday operational hours, hindering the ability to determine which user group (undergraduates, graduate students, community users, faculty) is less apt to use the library on saturdays with home football games. without more granular data, it is impossible to draw any conclusions beyond a general understanding of the direction and significance of the effect of football games on gate counts. given more specific data on saturday library users, a better analysis of the user group likely to be affected by varsity sports would be feasible.   this study did not examine the hours during which library usage was most likely to be affected by a home game. kickoff times varied considerably, making this analysis difficult. it is therefore unknown if the occurrence of a home game affects whole-day physical use of the library or if hours more closely grouped near the game are most affected. if the effect is time-bound with the game, libraries might consider reduced hours of operation related to kickoff time.   conclusion   the data from this study lead to the rejection of the null hypothesis. the data make clear that home football games influence would-be library patrons’ choice to physically access campus libraries. the lower number of patrons on saturdays with home football games might justify a reduced or altered level of staffing on those saturdays, particularly for libraries in proximity to a university’s football stadium. given that the observed reduction in gate counts was still less than 35% for the libraries examined in this study, library closure on saturdays with home football games would not be justified.   for libraries at universities facing budget cuts in tandem with the ever-present demand for expansive hours from student populations, such data would allow for more informed decision-making about staffing and hours in response to pre-planned, external events within the university as well as justification to student populations for reduced hours and staffing on those days.   future research on this topic area is recommended. replication of this study at other universities with large-scale sports programs is recommended to determine the generalizability of this study. furthermore, an examination of circulation and reference statistics for saturdays with home football games would determine if circulation and reference interactions decrease in tandem with gate count. similarly, assessing changes in use of cross-disciplinary, digital library resources (e.g., jstor or scopus) would supplement these findings, providing evidence of overall changed library-related behaviours on saturdays with home football games. additionally, further analysis of gate count data would be warranted to determine if users alter the days when they use the library as compensation for time spent in football-related activities on a saturday, (e.g., using the libraries more heavily on the friday or sunday surrounding a home game).   references   agha, n. (2013). the economic impact of stadiums and teams: the case of minor league baseball. journal of sports economics, 14(3), 227–252. https://doi.org/10.1177/1527002511422939   agha, n., & rascher, d. a. (2016). an explanation of economic impact: why positive impacts can exist for smaller sports. sport, business and management: an international journal, 6(2), 182–204. https://doi.org/10.1108/sbm-07-2013-0020   albanese, a. r. (2003). deserted no more: after years of declining usage the campus library rebounds. library journal, 128(7), 34–36.   brunsting, m. (2008). reference staffing: common practices of medium-sized academic libraries. journal of interlibrary loan, document delivery & electronic reserve, 18(2), 153–180. https://doi.org/10.1300/10723030802099251   chase, j., & healey, m. (1995). the spatial externality effects of football matches and rock concerts: the case of portman road stadium, ipswich, suffolk. applied geography, 15(1), 18–34. https://doi.org/10.1016/0143-6228(95)91060-b   chen, s. s.-c., teater, s., & whitaker, b. (2012). perceptions of students, faculty, and administrators about pregame tailgate parties at a kentucky regional university. international journal of developmental sport management, 1(2). http://ijdsm.net/wp-content/uploads/2019/03/steve-s-chen-perception-of-students....pdf   clotfelter, c. t. (2011). big-time sports in american universities. cambridge university press.  https://doi.org/10.1017/cbo9780511976902   coates, d., & depken, c. a. (2006). mega-events: is the texas-baylor game to waco what the super bowl is to houston. international association of sports economists working paper series, paper no. 06–06. https://ideas.repec.org/p/spe/wpaper/0606.html       derringer, n., & french, r. (2015, september 29). college drinking gameday diary: from 8 a.m. 1 a.m. across michigan. the center for michigan: bridge magazine. https://www.mlive.com/news/2015/09/unconscious_students_on_hospit.html   deschriver, t. d., webb, t., tainsky, s., & simion, a. (2021). sporting events and the derived demand for hotels: evidence from southeastern conference football games. journal of sport management, 35(3), 228–238. https://doi.org/10.1123/jsm.2020-0268   dotson, d. s., & garris, j. b. (2008, september). counting more than the gate: developing building use statistics to create better facilities for today’s academic library users. library philosophy and practice, 1–13. https://scholarcommons.sc.edu/cgi/viewcontent.cgi?article=1000&context=lib_facpub   driscoll, l., & mott, a. (2008). after midnight: late night library services: results of a survey of arl member libraries. association of research libraries.   drobatz, k. j., syring, r., reineke, e., & meadows, c. (2009). association of holidays, full moon, friday the 13th, day of week, time of day, day of week, and time of year on case distribution in an urban referral small animal emergency clinic. journal of veterinary emergency and critical care, 19(5), 479–483. https://doi.org/10.1111/j.1476-4431.2009.00452.x   east carolina university. (n.d.). ecu by the numbers. https://facts.ecu.edu/   east carolina university institutional assessment, planning, and research. (2021). enrollment by unit and major [data set]. ecu analytics portal. https://performance.ecu.edu/portal/?itemid=04309242-b548-43c6-84f4-a757fc7bf4c1   ecu athletics fiscal sustainability working group. (2020, may 14). athletics fiscal sustainability working group report. east carolina university. https://news.ecu.edu/wp-content/pv-uploads/sites/80/2017/07/athletics-fiscal-sustainability-working-group-report.pdf   ferria, a., gallagher, b. t., izenstark, a., larsen, p., lemeur, k., mccarthy, c. a., & mongeau, d. (2017). what are they doing anyway?: library as place and student use of a university library. evidence based library and information practice, 12(1), 18–33. https://doi.org/10.18438/b83d0t   finlay, a. k., ram, n., maggs, j. l., & caldwell, l. l. (2012). leisure activities, the social weekend, and alcohol use: evidence from a daily study of first-year college students. journal of studies on alcohol and drugs, 73(2), 250–259. https://doi.org/10.15288/jsad.2012.73.250   glassman, t., werch, c. e., jobli, e., & bian, h. (2007). alcohol-related fan behavior on college football game day. journal of american college health, 56(3), 255–260. https://doi.org/10.3200/jach.56.3.255-260   greene, k. m., & maggs, j. l. (2015). revisiting the time trade-off hypothesis: work, organized activities, and academics during college. journal of youth and adolescence, 44(8), 1623–1637. https://doi.org/10.1007/s10964-014-0215-7   hardin, t., & basden, s. (2019, october 28). businesses feel impact of ecu’s homecoming. wcti. https://wcti12.com/news/local/businesses-say-ecu-homecoming-helps-sales   humphreys, b. r., & pyun, h. (2018). professional sporting events and traffic: evidence from u.s. cities. journal of regional science, 58(5), 869–886. https://doi.org/10.1111/jors.12389   jena, a. b., clay mann, n., wedlund, l. n., & olenski, a. (2017). delays in emergency care and mortality during major u.s. marathons. the new england journal of medicine, 376(15), 1441–1450. https://doi.org/10.1056/nejmsa1614073   jerrard, d. a. (2009). male patient visits to the emergency department decline during the play of major sporting events. the western journal of emergency medicine, 10(2), 101–103.  https://www.ncbi.nlm.nih.gov/pmc/articles/pmc2691518/   jones, j. l. (2011). using library swipe-card data to inform decision making. georgia library quarterly, 48(2). https://digitalcommons.kennesaw.edu/glq/vol48/iss2/4   kramer, l. a., & kramer, m. b. (1968). the college library and the drop-out. college & research libraries, 29(4), 310–312. https://doi.org/10.5860/crl_29_04_310   lawrence, p., & weber, l. (2012). midnight-2.00 a.m.: what goes on at the library? new library world, 113(11/12), 528–548. https://doi.org/10.1108/03074801211282911   lemaistre, t., shi, q., & thanki, s. (2018). connecting library use to student success. portal: libraries and the academy, 18(1), 117–140. https://doi.org/10.1353/pla.2018.0006   lindo, j. m., swensen, i. d., & waddell, g. r. (2012). are big-time sports a threat to student achievement? american economic journal: applied economics, 4(4), 254–274. https://doi.org/10.1257/app.4.4.254   lo, b. m., visintainer, c. m., best, h. a., & beydoun, h. a. (2012). answering the myth: use of emergency services on friday the 13th. the american journal of emergency medicine, 30(6), 886–889. https://doi.org/10.1016/j.ajem.2011.06.008   lyon-hill, s., alwang, j. r., mawyer, a., burke, p., & budzevski, l. (2015). economic impact analysis of virginia tech football. http://hdl.handle.net/10919/94598   martell, c. (2007). the elusive user: changing use patterns in academic libraries 1995 to 2004. college & research libraries, 68(5), 435-445. https://doi.org/10.5860/crl.68.5.435   mcgreevy, a., millar, l., murphy, b., davison, g. w., brown, r., & o’donnell, m. e. (2010). the effect of sporting events on emergency department attendance rates in a district general hospital in northern ireland. international journal of clinical practice, 64(11), 1563–1569. https://doi.org/10.1111/j.1742-1241.2010.02390.x   mixon jr, f. g., & trevino, l. j. (2005). from kickoff to commencement: the positive role of intercollegiate athletics in higher education. economics of education review, 24(1), 97–102. https://doi.org/10.1016/j.econedurev.2003.09.005   moulin, m. s., & irwin, j. d. (2017). an assessment of sedentary time among undergraduate students at a canadian university. international journal of exercise science, 10(8), 1116–1129. https://digitalcommons.wku.edu/ijes/vol10/iss8/3   national center for education statistics. (2017). academic libraries survey (als). https://nces.ed.gov/statprog/handbook/pdf/als.pdf   national collegiate athletic association. (n.d.-a). ncaa statistics. https://stats.ncaa.org/rankings/change_sport_year_div   national collegiate athletic association. (n.d.-b). ncaa statistics: football 2018-2019 fbs miscellaneous reports: game length. https://stats.ncaa.org/rankings/change_sport_year_div   next college student athlete (ncsa). (n.d.). your comprehensive list of college football teams. https://www.ncsasports.org/football/colleges   opperman, b. v., & jamison, m. (2008). new roles for an academic library: current measurements. new library world, 109(11/12), 559-573. https://doi.org/10.1108/03074800810921368   orcutt, j. d., & harvey, l. k. (1991). the temporal patterning of tension reduction: stress and alcohol use on weekdays and weekends. journal of studies on alcohol, 52(5), 415–424. https://doi.org/10.15288/jsa.1991.52.415   phillips, j. (2016). determining gate count reliability in a library setting. evidence based library and information practice, 11(3), 68-74. https://doi.org/10.18438/b8r90p   pope, d. g., & pope, j. c. (2009). the impact of college sports success on the quantity and quality of student applications. southern economic journal, 75(3), 750–780. https://www.jstor.org/stable/27751414   pope, d. g., & pope, j. c. (2014). understanding college application decisions: why college sports success matters. journal of sports economics, 15(2), 107–131. https://doi.org/10.1177/1527002512445569   protty, m. b., jaafar, m., hannoodee, s., & freeman, p. (2016). acute coronary syndrome on friday the 13th: a case for re-organising services? the medical journal of australia, 205(11), 523–525. https://doi.org/10.5694/mja16.00870   rishe, p. j. (2003). a reexamination of how athletic success impacts graduation rates: comparing student‐athletes to all other undergraduates. american journal of economics and sociology, 62(2), 407–427. https://doi.org/10.1111/1536-7150.00219   robinson, m. j., trail, g. t., dick, r. j., & gillentine, a. j. (2005). fans vs. spectators: an analysis of those who attend intercollegiate football games. sport marketing quarterly, 14(1). https://fitpublishing.com/content/fans-vs-spectators-analysis-those-who-attend-intercollegiate-football-games   scarletto, e. a., burhanna, k. j., & richardson, e. (2013). wide awake at 4 am: a study of late night user behavior, perceptions and performance at an academic library. the journal of academic librarianship, 39(5), 371-377. https://doi.org/10.1016/j.acalib.2013.02.006   schuld, j., slotta, j. e., schuld, s., kollmar, o., schilling, m. k., & richter, s. (2011). popular belief meets surgical reality: impact of lunar phases, friday the 13th and zodiac signs on emergency operations and intraoperative blood loss. world journal of surgery, 35, 1945–1949. https://doi.org/10.1007/s00268-011-1166-8   shook, j., & hiestand, b. c. (2011). alcohol-related emergency department visits associated with collegiate football games. journal of american college health, 59(5), 388–392. https://doi.org/10.1080/07448481.2010.511364   shughart, w. f., tollison, r. d., & goff, b. l. (1986). pigskins and publications. atlantic economic journal, 14, 46–50. https://doi.org/10.1007/bf02316792   smith, c. m., & graffeo, c. s. (2005). regional impact of hurricane isabel on emergency departments in coastal southeastern virginia. academic emergency medicine: official journal of the society for academic emergency medicine, 12(12), 1201–1205. https://doi.org/10.1197/j.aem.2005.06.024   soria, k. m., fransen, j., & nackerud, s. (2013). library use and undergraduate student outcomes: new evidence for students’ retention and academic success. portal: libraries and the academy, 13(2), 147–164. https://doi.org/10.1353/pla.2013.0010   soria, k. m., fransen, j., & nackerud, s. (2017). beyond books: the extended academic benefits of library use for first-year college students. college & research libraries, 78(1), 8–22. https://doi.org/10.5860/crl.78.1.8   soria, k. m., nackerud, s., & peterson, k. (2015). socioeconomic indicators associated with first-year college students’ use of academic libraries. the journal of academic librarianship, 41(5), 636–643. https://doi.org/10.1016/j.acalib.2015.06.011   sridhar, m. s. (1994). non-use and non-users of libraries. library science with a slant to documentation and information studies, 31(3), 115–128.   tailgating institute. (n.d.). tailgating research—tailgater statistics and information. https://tailgating.com/tailgater-research-tailgater-statistics-and-information   tempelmeier, n., dietze, s., & demidova, e. (2020). crosstown traffic-supervised prediction of impact of planned special events on urban traffic. geoinformatica, 24, 339–370. https://doi.org/10.1007/s10707-019-00366-x   terrill, l. j. (2018). telling their story with data: what academic research libraries share on their websites. journal of web librarianship, 12(4), 232–245. https://doi.org/10.1080/19322909.2018.1514286   thorpe, a., lukes, r., bever, d. j., & he, y. (2016). the impact of the academic library on student success: connecting the dots. portal: libraries and the academy, 16(2), 373–392. https://doi.org/10.1353/pla.2016.0027   toma, j. d., & cross, m. e. (1998). intercollegiate athletics and student college choice: exploring the impact of championship seasons on undergraduate applications. research in higher education, 39, 633–661. https://doi.org/10.1023/a:1018757807834   toner, l. j. (2008). non-use of library services by students in a uk academic library. evidence based library and information practice, 3(2), 18–31. https://doi.org/10.18438/b8hs57   tucker, i. b. (1992). the impact of big-time athletics on graduation rates. atlantic economic journal, 20, 65–72. https://doi.org/10.1007/bf02300088   turtle, k. m. (2005). a survey of users and non‐users of a uk teaching hospital library and information service. health information & libraries journal, 22(4), 267–275. https://doi.org/10.1111/j.1471-1842.2005.00596.x   evidence based library and information practice evidence based library and information practice 2011, 6.2 1 evidence based library and information practice editorial what is evidence? denise koufogiannakis editor-in-chief collections & acquisitions coordinator, university of alberta libraries edmonton, alberta, canada email: denise.koufogiannakis@ualberta.ca 2011 koufogiannakis. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. lately, i have been pondering what we really mean when we say “evidence based practice”? in lis, we all know the definitions that have been proposed (booth 2000, eldredge 2000, crumley and koufogiannakis 2002), and which have not ever really been challenged. but have we ever said explicitly what qualifies as evidence in this model? the underlying assumption seems to be that evidence is research, hence, we are really talking about research-based practice, but we don’t actually use that term. higgs and jones (2000) note that evidence is “knowledge derived from a variety of sources that has been subjected to testing and has found to be credible.” the oxford english dictionary states that evidence is “something serving as a proof” (oed, 2011). neither of these definitions of evidence notes that evidence equals research; research is only one form of evidence. it certainly isn’t the only form of evidence – so what, then, constitutes evidence? rycroff-malone et al. (2004) state that that in order for evidence based practice to create a broader evidence base in nursing, “the external, scientific and the internal, intuitive” need to be brought together. the external, scientific is what evidence based practice has been focused on, in the form of scientific research, but rycroff-malone et al. note that other elements such as clinical experience, patient experience, and information from the local context also need to be considered. in library and information practice, what are the other forms of evidence we need to consider? i propose that while research evidence is of high importance to our profession and knowledge, lis practitioners need to first of all consider local evidence. local evidence is found in our working environment and specific to the context in which we carry out our work. it includes such things as our experiences with patrons in particular contexts, and what we observe to work in such situations, assessment of programs, feedback from our users, project evaluations, and accumulated experiences over the course of careers. these things are not easily shared and often do not find a place in publications because they are too local. but data that comes from a local context is in fact often the most important evidence source that mailto:denise.koufogiannakis@ualberta.ca� evidence based library and information practice 2011, 6.2 2 a lis professional can consult because it gives us information that is directly applicable to, and about our users. for example, usage stats on ejournals, feedback and comments about our services, usability testing on a website, titles on our interlibrary loan requests; these are just a few examples of local evidence that is invaluable to our decision making. this local data doesn’t often mean much to others, but it is of utmost importance to our local knowledge. the trick is to figure out what local information to collect, and how to use it. and remember to use it. this is where others’ experiences of how they use such local evidence can give us ideas and inspiration. as well, we hold a great deal of evidence in our professional knowledge that progressively is built up by library and information professionals over the course of their career. much of this is tacit, but worthwhile trying to draw out and make explicit. evidence is shown to us every single day as we practice our profession, we learn what works and what doesn’t in certain situations. we have practical, real-life experiences to draw upon that are wrapped in different contexts. as professionals we have foundations that form the basis of our knowledge, in a field where we have already learned from our education, training, and on-the-job experience. we build up skills and know-how that are not necessarily written down, but which provide us with a great deal of specialized knowledge. as we learn how to most effectively provide good service, or build quality collections for our users, or build relationships within our community, all these things provide us with evidence of how to be a better professional. that does not mean that we can just rely on these experiences, however, but rather that through reflection and critical thinking we will see where things may be improved and begin to investigate ways to make them better. without that initial evidence coming from our experience, we would not even know how or when to question or critique such things. hence, the initial evidence of “what do i know” becomes an area that is crucial to future research that will be of value to practice improvement. we need to start thinking about how we put all these different pieces of evidence together to make the best decisions possible. i don’t write this to dismiss what those in the eblip movement are doing (certainly, i am part of that movement, and hope to contribute to its progression). rather, i pose these things as considerations and challenges for us to make evidence-based/evidence-informed/practicebased evidence, better and more applicable to practitioners. so, what does this journal do to aid in pulling together these different pieces of evidence? well, first and foremost we publish in an open access manner so anyone who needs to can access the content we provide. the types of evidence we publish in this journal are varied, but we have taken several different approaches. firstly, we publish relevant research that has been vetted through peer review. this is a traditional role in publishing scholarly work, and we believe it is important. secondly, we publish evidence that comes from critically appraising previously published studies – this is a type of metaevidence wherein writers of evidence summaries must bring their professional experience and training to bear on the critiques they write. it lends readers another professional opinion (also vetted through peer review), to help educate and inform readers to make up their mind about the quality and value of the original research study. the same can be said of our “classics” which entail the same process but with works that have stood the test of time and still hold relevance for today. our ebl101 column is similar in that it synthesizes small aspects of evidence based practice or research that help educate the reader about particular elements of evidence based practice. finally, we publish commentaries and the using evidence in practice section, which provide practitioner insights and reflections about their work at a local level. these are useful in helping readers apply similar strategies to their own practice, or imbue them with ideas from which their own evaluation of their practice can occur. commentaries also allow new ideas to take shape, or critiques of particular aspects of evidence based library and information practice 2011, 6.2 3 evidence based practice, to reach readers, allowing for continued discussion and debate. pulling together the published evidence with local evidence and insights is not an easy task. i think it resides in each individual professional who wishes to do their best work while including research evidence to support that work. talking about being evidence based and making explicit some of the factors that do not make this process an easy one, is one of the first steps to enabling the process to become more meaningful and allow individual practitioners to have ownership of it in their own way. references booth, a. (2000). exceeding expectations: achieving professional excellence by getting research into practice. lianza conference. christchurch, new zealand. crumley, e.t., & koufogiannakis, d. (2002). developing evidence based librarianship: practical steps for implementation. health information and libraries journal, 19(2), 61-70. eldredge, j. d. (2000). evidence based librarianship: an overview. bulletin of the medical library association, 88(4), 289-302. evidence. (1989). in oxford english dictionary (2nd ed.). retrieved from http://www.oed.com /entry/65368. higgs, j. & jones, m.a. (2000). will evidencebased practice take the reasoning out of practice? in j. higgs & m.a. jones (eds.), clinical reasoning in the health professionals (2nd ed.) (pp. 307-315). oxford, u.k.: butterworth heineman. rycroff-malone, j., seers, k., titchen, a., harvey, g., kitson, a., & mccormack, b. (2004). what counts as evidence in evidence based practice? journal of advanced nursing, 47(1), 81-90. / evidence based library and information practice news   call for applicants for associate editor (evidence summaries) for evidence based library and information practice    2021. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29928     evidence based library and information practice (eblip) is seeking a new associate editor (evidence summaries). the successful candidate is expected to begin their role by august 2021 and serve a three-year term (with possibility of renewal). this role shares workload with a second associate editor.   evidence summaries are critical appraisal syntheses that provide analysis regarding the validity and reliability of the methodology used in original research articles. as such, they are a key component to aid readers in making informed decisions in their local practice. the team of experienced evidence summary authors follow a strict format to ensure consistency, and all evidence summaries undergo peer review to ensure quality. eblip publishes up to ten summaries in every issue.   the associate editor (evidence summaries) will be responsible for:   overseeing the complete editorial process for items submitted to the evidence summaries section, which includes: selecting articles for summary and assigning articles to evidence summary authors; ensuring authors adhere to the eblip evidence summary guidelines; assigning peer reviewers and monitoring the peer review process; communicating with evidence summary authors and peer reviewers; making acceptance decisions, ensuring required changes to manuscripts are made; and communicating with copyeditors and the production editor to ensure the final copy is as expected. communicating and consulting with the editor-in-chief on a regular basis. attending editorial team meetings (via zoom) on a quarterly basis and participating in the overall governance of the journal. ensuring that all necessary deadlines are met. maintaining a team of approximately five to eight evidence summary authors, each of whom serves a two-year term with a commitment to write four evidence summaries. periodically sending out calls for new evidence summary authors, reviewing applications, and assessing sample evidence summaries from candidates.   for more information about evidence summaries and other types of submissions accepted by eblip see: https://journals.library.ualberta.ca/eblip/index.php/eblip/guidelines   the ideal candidate for associate editor (evidence summaries) will be well-versed in evidence based practice, research methods, and critical appraisal. this position requires dedicated time on a regular basis, and it is therefore essential that interested persons ensure available time to devote to this position prior to applying. it is estimated that the workload is approximately 15 hours per month.   interested persons should send a cover letter, indicating strengths they would bring to the role, and a resume/cv as a single pdf file to ann medaille, editor-in-chief, at amedaille@unr.edu by april 30, 2021. the current associate editors (evidence summaries) are happy to answer specific queries about the role upon request. please email stephanie krueger (stephanie.krueger@techlib.cz) or christina wissinger (clw68@psu.edu).   *please note that evidence based library and information practice is a non-profit, open access journal and all positions are voluntary and unpaid.   about the journal   published quarterly, and hosted by the university of alberta, this peer-reviewed, open access journal has been publishing since 2006. it is targeted at all library and information professionals interested in an evidence based model of practice. by facilitating access to librarianship research via original research articles and evidence summaries of relevant research from the library literature, evidence based library and information practice will enable librarians to practice their profession in an evidence based manner.   please visit the evidence based library and information practice journal web site (https://journals.library.ualberta.ca/eblip/index.php/eblip) for more information.       evidence based library and information practice evidence based library and information practice 2011, 6.1 102 evidence based library and information practice news/announcements news from the library and information science research coalition (u.k.) 2011. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. eblip6 the library and information science (lis) research coalition will be very much in evidence at eblip6. professor hazel hall, director of the centre for social informatics at edinburgh napier university and executive secretary of the lis research coalition, is presenting the keynote paper on thursday, 30 june while, later the same day, the coalition is sponsoring a panel session entitled “meet the editors”. the lis research coalition will also be offering four sponsored places at the conference: two for practitioners, and two for phd students based in the u.k. further details of how to apply for a sponsored place will soon be available on the lis research coalition web site at http://lisresearch.org/ dream: developing research methods and excellence in november 2010, the lis research coalition was awarded a grant of £45,000 from the arts & humanities research council (ahrc) to support the dream project. principal investigator professor hazel hall and her co-investigator professor charles oppenheim are working with representatives of the lis research coalition member organizations to develop a series of training events and resources for lis researchers. a key goal of the project is to build capacity and capability in the development and implementation of innovative methods and techniques in undertaking lis research. the project started in january 2011 and runs until august 2012. the first dream project event will be a oneday conference at the british library in london on 19 july 2011. the programme will include an opening keynote presentation by dr blaise cronin (rudy professor of information science in the school of library and information science at indiana university, us), a closing keynote presentation, a one minute madness session and a choice of themed workshops, as well as a session on the dream project itself. four further dream project events will take place: • 25 october 2011 – workshop craighouse campus, edinburgh napier university, edinburgh (places for 30 participants) • 30 january 2012 – workshop, british library london (places for 30 participants) • april 2012 – workshop in edinburgh (exact date and location to be confirmed, places for 30 participants) http://lisresearch.org/� evidence based library and information practice 2011, 6.1 103 • 9 july 2012 – concluding event, british library london (places for 40 participants) these events will explore the scope of lis and related research, and the range of methods appropriate to research in the domain. experts in a variety of techniques will present method overviews, covering their strengths and weaknesses. a wide range of techniques will be evaluated beyond the standard qualitative and quantitative methods commonly deployed in lis research. it is hoped that a number of the participants will be able to attend all events (along with others who will elect to come to one or more). to register interest in these events, please e-mail hazel.hall@lisresearch.org. further information about the project will be added to the dream project web page at: http://lisresearch.org/dream-project/. for regular updates, please follow the dream project on twitter at @lis_dream. rilies: research in librarianship – impact evaluation study rilies (pronounced “realize”) is a study led by professor hazel hall and supported by the lis research coalition. running from february until the end of july 2011, the project explores the extent to which funded librarianship research projects influence library practice in the u.k. of particular interest are the factors that increase or hinder the impact of project outcomes on practice. currently little is known of the impact of particular research projects on the day-to-day work carried out by librarians. this makes it difficult to assess an important aspect of research undertaken. equally, although it is possible to cite reasons why a particular project may have a greater influence than another – such as the means by which it is conceived and implemented, how output is disseminated, what is disseminated, the profile of the research team or the target audience – knowledge of the relative value of these factors is low. already 200 lis practitioners and researchers have contributed to the project by completing the rilies poll. there will be further opportunities for participation in the later stages of the project. further information about the project will be added to the rilies project web page at: http://lisresearch.org/rilies-project/. for regular updates, please follow the rilies project on twitter at @lis_rilies mailto:hazel.hall@lisresearch.org� http://lisresearch.org/dream-project/� http://lisresearch.org/rilies-project/� / evidence based library and information practice article   evaluating approaches to quality assessment in library and information science lis systematic reviews: a methodology review   michelle maden   nihr nwc clahrc phd student liverpool reviews and implementation group university of liverpool liverpool, united kingdom email: michelle.maden@liverpool.ac.uk     eleanor kotas   information specialist liverpool reviews and implementation group university of liverpool liverpool, united kingdom email: eleanor.kotas@liverpool.ac.uk   received: 2 jan. 2016      accepted: 8 apr. 2016         2016 maden and kotas. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     objective – systematic reviews are becoming increasingly popular within the library and information science (lis) domain. this paper has three aims: to review approaches to quality assessment in published lis systematic reviews in order to assess whether and how lis reviewers report on quality assessment a priori in systematic reviews, to model the different quality assessment aids used by lis reviewers, and to explore if and how lis reviewers report on and incorporate the quality of included studies into the systematic review analysis and conclusions.   methods – the authors undertook a methodological study of published lis systematic reviews using a known cohort of published systematic reviews of lis-related research. studies were included if they were reported as a “systematic review” in the title, abstract, or methods section. meta-analyses that did not incorporate a systematic review and studies in which the systematic review was not a main objective were excluded. two reviewers independently assessed the studies. data were extracted on the type of synthesis, whether quality assessment was planned and undertaken, the number of reviewers involved in assessing quality, the types of tools or criteria used to assess the quality of the included studies, how quality assessment was assessed and reported in the systematic review, and whether the quality of the included studies was considered in the analysis and conclusions of the review. in order to determine the quality of the reporting and incorporation of quality assessment in lis systematic reviews, each study was assessed against criteria relating to quality assessment in the prisma reporting guidelines for systematic reviews and meta-analyses (moher, liberati, tetzlaff, altman, & the prisma group, 2009) and the amstar tool (shea et al., 2007).   results – forty studies met the inclusion criteria. the results demonstrate great variation on the breadth, depth, and transparency of the quality assessment process in lis systematic reviews. nearly one third of the lis systematic reviews included in this study did not report on quality assessment in the methods, and less than one quarter adequately incorporated quality assessment in the analysis, conclusions, and recommendations. only nine of the 26 systematic reviews that undertook some form of quality assessment incorporated considerations of how the quality of the included studies impacted on the validity of the review findings in the analysis, conclusion, and recommendations. the large number of different quality assessment tools identified reflects not only the disparate nature of the lis evidence base (brettle, 2009) but also a lack of consensus around criteria on which to assess the quality of lis research.   conclusion – greater clarity, definition, and understanding of the methodology and concept of “quality” in the systematic review process are required not only by lis reviewers but also by editors of journals in accepting such studies for publication. further research and guidance is needed on identifying the best tools and approaches to incorporate considerations of quality in lis systematic reviews. lis reviewers need to improve the robustness and transparency with which quality assessment is undertaken and reported in systematic reviews. above all, lis reviewers need to be explicit in coming to a conclusion on how the quality of the included studies may impact on their review findings.   introduction   systematic reviews aim “to systematically search for, appraise and synthesize research evidence” (grant & booth, 2009, p. 95). unlike traditional literature reviews, the systematic review aims to minimize bias by following a systematic and transparent approach to defining the question, searching the literature, extracting relevant data, assessing the quality of the literature, and synthesizing and drawing conclusions on the state of the evidence base (boland, cherry, & dickson, 2013; higgins et al., 2011; centre for reviews and dissemination [crd], 2009).   systematic reviews in lis, as in other professional domains, are becoming increasingly popular. paralleling the evidence based medicine paradigm, systematic reviews are considered high levels of evidence within the lis domain (eldredge, 2000) and incorporate into their methodology one of the core skills of lis professionals—literature searching. the number of systematic reviews in the lis field is growing (koufogiannakis, 2012), and journal editors are actively encouraging librarians to undertake them (sampson, 2014). however, lis research methods are diverse (brettle, 2009), and they are not easy to work with in the context of a systematic review (koufogiannakis, 2012). lis systematic reviewers, therefore, may be faced with the prospect of including and assessing the quality of a diverse evidence base.   this study aims to provide an overview of approaches to quality assessment used by lis systematic reviewers. the paper models and presents a collation of quality assessment tools. the results will be particularly useful for those undertaking lis systematic reviews and evidence summaries.   literature review   in defining quality as a “multidimensional concept which could relate to the design, conduct, and analysis of a trial, its clinical relevance, or quality of reporting,” jüni, altman, & egger (2001, p. 42) identify three approaches to quality assessment: internal validity, external validity, and reporting quality. internal validity refers to the robustness of the methods and the extent to which bias (i.e. systematic errors in studies that may lead to an overestimation or underestimation of the true result) is minimized and the results of the study can be considered reliable. external validity is the extent to which the study results can be considered generalizable to other settings or populations, and reporting quality assesses the completeness of the reporting.   quality assessment is an integral part of the systematic review process (boland, cherry, & dickson, 2013). details on how reviewers intend to assess the quality of the included studies should be outlined a priori in the review methods section (moher et al., 2009). in addition, the results of the quality assessment of the included studies should be presented and consideration given not only to the quality of the individual studies but also to how the overall quality of the included studies impacts on the validity of the review findings (shea et al., 2007). given the subjective nature involved in quality assessment, it is imperative that more than one reviewer undertakes this process.   a multitude of tools are available to help reviewers assess the quality of the included studies. tools for assessing the quality of research fall into three categories: scales, checklists, and domains (see box 1; west et al., 2002). despite a lack of consensus about which quality assessment tools are best, there is some agreement that scales are not the most appropriate tool. the shortcomings of scales are that different scales can produce different results (jüni et al., 2001; booth, 2007) and that users may assume each criteria on a scale is equally weighted. in the case of double-blinding in a randomised controlled trial, for example, such an assumption may not be appropriate (schulz, chalmers, hayes, & altman, 1995).   quality assessment of studies is essential to maintaining the integrity of the systematic review since the validity of the review conclusions rely upon the quality of the included studies (jüni et al., 2001; deeks et al., 2003; sampson, 2014). empirical research in the health and medical field found that poor methodological quality of trials can overestimate or underestimate the true result (schulz et al., 1995). combining such studies in a systematic review, therefore, can only serve to emphasize such biases even further.   there is limited empirical evidence of how quality assessment is incorporated in lis systematic reviews. a recent editorial undertook a brief examination of systematic reviews published in the journal of the medical library association (sampson, 2014). whilst the number of systematic reviews identified was small (four), only two of them assessed the quality of the literature of the included studies. with the growing popularity of systematic reviews within the lis field, it is timely to consider how and to what extent lis reviewers incorporate assessments of quality in lis systematic reviews.     box 1 categories of quality assessment toolsa checklist: “instruments that contain a number of quality items, none of which is scored numerically.”   scale: “instruments that contain several quality items that are scored numerically to provide a quantitative estimate of overall study quality.”   domain (or component): “individual aspect of study methodology (e.g. randomisation, blinding, follow-up) that has a potential relation to bias in estimation of effect.”   awest et al., 2002, p. 33     aims   to provide an overview of approaches to quality assessment in published lis systematic reviews. in particular, to:   assess whether and how lis reviewers report on quality assessment a priori in systematic reviews, model the different quality assessment aids used by lis reviewers, and explore if and how lis reviewers report on and incorporate the quality of included studies into the systematic review analysis and conclusions.   methods   the authors undertook a methodological study of published lis systematic reviews. a cohort of published systematic reviews of lis-related research was identified from the literature in november 2014, using an existing wiki that claims to document “all the known systematic reviews in library and information studies” (koufogiannakis, brettle, booth, kloda, & urquhart, 2015).   studies in the list were included if they were reported as a systematic review in the title, abstract, or methods section. meta-analyses that did not incorporate a systematic review and studies in which the systematic review was not a main objective were excluded. two reviewers independently assessed the studies against the inclusion criteria. in the event of disagreement, consensus was reached via discussion.   data were extracted on the type of synthesis, whether quality assessment was planned and undertaken, the number of reviewers involved in assessing quality, the types of tools or criteria used to assess the quality of the included studies, how quality assessment was presented in the systematic review, and whether the quality of the included studies was considered in the analysis and conclusions of the review.   to determine the quality of the reporting and incorporation of quality assessment in lis systematic reviews, two reviewers independently assessed each included study against criteria relating to quality assessment in the prisma reporting guidelines for systematic reviews and meta-analyses (moher et al., 2009) and the amstar tool (shea et al., 2007) (see box 2). the prisma guidance (moher et al., 2009) assesses the quality of the reporting in systematic reviews. within the 27-item checklist, four items relate to the reporting of quality assessment within systematic reviews. amstar (shea et al., 2007) is an 11-item checklist that aims to assess the methodological quality of systematic reviews. four of the items relate to quality assessment.     box 2 prisma and amstar quality assessment criteria prismaa   item #12: risk of bias in individual studies describe methods used for assessing risk of bias of individual studies (including specification of whether this was done at the study level or outcome level) and how this information is to be used in any data synthesis   item #15: risk of bias across studies specify any assessment of risk of bias that may affect the cumulative evidence (e.g., publication bias, selective reporting within studies)   item #19: risk of bias within studies present data on risk of bias of each study and, if available, any outcome level assessment   item #22: risk of bias across studies present results of any assessment of risk of bias across studies   amstarb   item #7: was the scientific quality of the included studies assessed and documented?   item #8: was the scientific quality of the included studies used appropriately in formulating conclusions?   item #10: was the likelihood of publication bias assessed?   amoher et al., 2009 bshea et al., 2007     studies were classed as adequate if they were explicit in meeting the prisma or amstar criteria, inadequate if they partially met the criteria, or unclear if the study did not report on the item. a narrative synthesis of the results is presented.   results   a total of 40 studies reported on the wiki met the inclusion criteria. of the 10 studies excluded, four were meta-analyses (aabø, 2009; ankem, 2006b; salang, 1996; saxton, 1997), five were not reported as systematic reviews (haug, 1997; julien, leide, & bouthillier, 2008; mairs, mcneil,   mcleod, prorok, & stolee, 2013; ward, stevens, brentnall, & briddon, 2008; williams, nicholas, & rowlands, 2010), and in one study (sampson, mcgowan, lefebvre, moher, & grimshaw, 2008), the systematic review was not the objective of the study.   table 1 outlines the characteristics of the 40 systematic reviews included in the analysis. more than half were published in the health lis field. the number of studies included in the systematic reviews ranged from 3 to 333. nine systematic reviews were undertaken by only one author, 15 reported two authors, and 16 reported more three or more authors. five systematic reviews reported including only one type of study design, 22 included more than one type of study design, and 13 did not report on study designs.   reporting of quality assessment methods in lis systematic reviews   table 2 outlines the approach to quality assessment undertaken in the included studies. fourteen of the 40 failed to state they would undertake quality assessment in the methods. of these 14, 10 were narrative reviews, three were qualitative, and one a meta-analysis with narrative review. of the 14 systematic reviews that did not report on quality assessment in the methods, six did report on the quality of the included studies in the results or discussion (ankem, 2006a; fanner & urquhart, 2008; genero, fernandez-saez, nelson, poels, & piattini, 2011; kelly & sugimoto, 2013; matteson, salamon, & brewster, 2011; wagner & byrd, 2004).   less than half of those that did undertake quality assessment (11 of 26) actually defined what they meant by quality. eight studies defined quality as an assessment of the methodological quality (koufogiannakis and wiebe, 2006; zhang, watson, & banfield, 2007; joshi & trout, 2014; golder & loke, 2009, 2010; perrier et al., 2014; gagnon et al., 2010; divall, camosso-stefinovic, & baker., 2013), and three studies (perrier et al, 2014; gagnon et al, 2010; divall et al., 2013) specified quality as the risk of bias. two studies defined quality assessment as assessing the quality of the study design, or level of evidence (manning fiegen, 2010; ndabarora, chipps, & uys, 2014). one study assessed the quality of the reporting (crumley, wiebe, cramer, klassen, & hartling, 2005).   fourteen studies reported the number of people involved in undertaking quality assessment. the number of authors undertaking quality assessment in a systematic review ranged from 1 to 8. three studies (joshi & trout, 2014; manning fiegen, 2010; perrier et al., 2014) reported an assessment of the inter-rater agreement of quality assessment between reviewers.  nine studies reported using quality assessment as part of their inclusion and exclusion criteria.   quality assessment tools   table 3 outlines the aids used by lis systematic reviewers. lis reviewers referenced four types of aids to quality assessment: 14 published tools designed specifically to assess quality, 3 research design books, 11 journal articles, and 1 web resource. twenty-nine unique aids were referenced.   twenty-one studies that reported on quality assessment in the methods section referenced the tools or aids used: 14 studies reported using one aid, four studies reported using two aids, and three studies used three or more aids (see table 3). seven studies modified the aids (see table 2), but in four of these studies (beverley, bath, & booth, 2004; brettle, 2007; brettle et al, 2011; gray, sutton, & treadway, 2012) what modifications were made is unclear. four studies reported using bespoke (i.e. custom-made) criteria to assess the quality of the included studies.   of the nine studies that referenced aids other than published tools, six reported the criteria on which their quality assessment would be undertaken (brettle et al., 2011; gagnon et al., 2010; golder & loke, 2009; crumley et al., 2005; sommestad, hallberg, lundholm, & bengtsson, 2014; weightman & williamson, 2005). three studies (beverley et al., 2004; booth, carroll, papaioannou, sutton, & wong, 2009 [for survey tool]); gray et al., 2012) did not report on the criteria used to assess the quality of their included studies.     table 1 characteristics of included studies characteristics number of reviews journal health information and libraries journal journal of the medical library association information research  journal of academic librarianship  journal of documentation reference services review bmc medical research methodology evidence based library and information practice health and social care in the community health informatics journal information development information management and computer security international journal of medical informatics journal of information science journal of the american medical informatics society journal of the american society for information science and technology journal of database management library and information science research libres: library and information science research electronic journal mousaion reference and user services quarterly   14 4 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 date published 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014   2 3 3 4 4 3 2 6 3 2 5 4 number of systematic review authors 1 2 3 4 5 8 10 13   9 15 8 2 3 1 1 1 type of systematic review (synthesis) narrative qualitative (e.g. meta-synthesis, meta-ethnography, framework) meta-analysis with narrative   30 7 3     of the 14 published tools specifically designed to assess the quality of research, nine were checklists, three were scales and one was domain-based. no single tool was preferred over others, with the glynn (2006), cristal checklist (booth, 2000; booth & brice, 2004), hcprdu evaluation tools (long et al., 2002a, 2002b, 2002c) and the cochrane effective practice and organisation of care (2010) each being cited by three studies. only one study (catalano, 2013) reported using a validated tool.   incorporation of quality assessment in systematic reviews   all studies incorporating quality assessment presented a narrative synthesis of their results. twelve studies (beverley et al., 2004; booth et al., 2009; brettle et al., 2011, brettle, 2007; catalano, 2013; divall et al., 2013; gagnon et al., 2010; golder & loke, 2009; ndabarora et al., 2014; perrier et al., 2014; sommestad et al., 2014; weightman & williamson, 2005) also tabulated their quality assessment. table 4 outlines the appraisal of the included studies according to the incorporation of quality assessment criteria in the prisma reporting guidelines and amstar quality assessment tool.   prisma assessment   of the 26 studies reporting a planned quality assessment in the methods, all but three did not report on how the data would be used in the synthesis, thereby failing to meet the first prisma quality criteria (item #12): “describe methods used for assessing risk of bias of individual studies (including specification of whether this was done at the study level or outcome level) and how this information is to be used in any data synthesis [emphasis added]” (moher et al., 2009, p. w-67). only one study (sommestad et al., 2014) reported an assessment of publication bias a priori in the methods (prisma item #15).   when reporting the results of the quality assessment, only five studies adequately “presented data on the risk of bias of each study and, if available, any outcome level assessment” (prisma item #19; moher et al., 2009, p. 266). twenty-one studies were classed as inadequate. of these, 18 studies (bergman & holden, 2010; beverley et al., 2004; booth et al., 2009; brennan, mattick, & ellis, 2011; brettle, 2003, 2007; brown, 2008; catalano, 2013; crumley et al., 2005; golder & loke, 2010; kelly & sugioto, 2013; koufogiannakis & wiebe, 2006; manning fiegen, 2010; ndabarora et al., 2014; perrier et al., 2014; sommestad et al., 2014; weightman & williamson, 2005; winning & beverley, 2003) reported summary data only. thus, the systematic reviews were either unclear about which of the included studies met each of the criteria assessed, or the systematic reviews were unclear about what criteria was used to assess the included studies. two studies reported only criteria relating to the validity and reliability of the outcome tool (ankem, 2005; ankem, 2006a) while one study failed to report on all studies in the quality assessment (gray et al., 2012). the remaining 14 studies were assessed as unclear as they did not report on quality assessment of the included studies in the results section of their review.     four studies “presented results of any assessment of risk of bias across studies” (prisma item #22; moher et al., 2009, p. w-67). two studies (brennan et al., 2011; perrier et al., 2014) assessed selective reporting; the other two (divall et al., 2013; sommestad et al., 2014) assessed publication bias. three of the studies (brennan et al., 2011; divall et al., 2013; perrier et al., 2014) presented a descriptive analysis while sommestad et al. (2014) presented an analytical analysis using a funnel plot.     table 2 quality assessment (qa) in lis systematic reviews study qa reported in methods authors defined qa no. of authors undertaking qa number of qa tools used model of qa (tools only) published, modified, or bespoke qa reported as an inclusion criteria ankem (2005) ✓ nr 1 nr nr nr ✓ ankem (2006a) nr nr nr nr nr nr nr bergman & holden (2010) ✓ nr nr 1 checklist published nr beverley et al. (2004) ✓ nr nr 3 checklist, scale 1 published 2 modified (unclear) nr booth et al. (2009) ✓ ✓ nr 3 checklist published nr brennan et al. (2011) ✓ nr nr 1 domain published nr brettle (2003) ✓ nr 1 1 checklist published nr brettle et al. (2011) ✓ nr 8a 2 checklist modified (unclear) nr brettle (2007) ✓ nr 1 2 checklist modified (unclear) nr brown (2008) ✓ nr nr nr nr nr nr burda & teuteberg (2013) nr nr nr nr nr nr ✓ catalano (2013) ✓ nr 1b 1 checklist published nr childs et al. (2005) nr nr nr nr nr nr nr cooper and crum (2013) nr nr nr nr nr nr nr crumley et al. (2005) ✓ ✓ 2 1 unclear bespoke (other journal article) nr divall et al. (2013) ✓ ✓ nr 1 domain published ✓ du preez (2007) nr nr nr nr nr nr nr duggan & banwell (2004) nr nr nr nr nr nr nr fanner & urquhart (2008) nr nr nr nr nr nr nr gagnon et al. (2010) ✓ ✓ 2 nr unclear bespoke (other journal articles) ✓ genero et al. (2011) nr nr nr nr nr nr nr golder & loke (2010) ✓ ✓ nr 1 unclear bespoke (own criteria) nr golder & loke (2009) ✓ ✓ nr 2 unclear modified (journal articles, web resource) nr grant (2007) nr nr nr nr nr nr nr gray et al. (2012) ✓ nr 3c 3/4d unclear modified (journal articles) nr joshi & trout (2014) ✓ ✓ 2 nr nr nr nr kelly & sugimoto (2013) nr nr nr nr nr nr nr koufogiannakis and wiebe (2006) ✓ ✓ 1 1 unclear published (other journal article) nr manning fiegen (2010) ✓ ✓ 6a 1 checklist published nr matteson et al. (2011) nr nr nr nr nr nr nr ndabarora et al. (2014) ✓ ✓ nr nr nr nr ✓ perrier et al. (2014) ✓ ✓ 2 2 scale published ✓ phelps & campbell (2013) nr nr nr nr nr nr nr rankin et al. (2008) ✓ nr nr 1 checklist published nr sommestad et al. (2014) ✓ nr nr 1 unclear modified (other journal article) ✓ urquhart & yeoman (2010) nr nr nr nr nr nr nr wagner & byrd (2004) nr nr nr nr nr nr ✓ weightman & williamson (2005) ✓ nr 2 1 unclear bespoke (books) ✓ winning & beverley (2003) ✓ nr nr 1 checklist published nr zhang et al. (2007) ✓ ✓ 2 1 scale modified nr note: nr = not reported. bespoke = custom-made. atwo reviewers appraised each paper bone study that was appraised by two reviewers cthree reviewers appraised three included studies collectively and then they appraised the rest individually. two reviewers checked all appraisals for accuracy. dauthors report using three tools but they reference four.     table 3 bibliography of quality assessment tools and resources used in lis systematic reviews quality assessment tools number of studies (and the studies) using the tool checklists   cristal booth, a. (2000). research. health information & libraries journal, 17(4), 232-235. booth, a., & brice, a. (2004). appraising the evidence. in booth & brice (eds.), evidence-based practice for information professionals a handbook. london, uk: facet publishing.   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-99.     hcprdu evaluation tools long, a. f., godfrey, m., randall, t., brettle, a., & grant, m. j. (2002a). hcprdu evaluation tool for qualitative studies. leeds: university of leeds, nuffield institute for health. long, a. f., godfrey, m., randall, t., brettle, a., & grant, m. j. (2002b). hcprdu evaluation tool for quantitative studies. leeds: university of leeds, nuffield institute for health. long, a. f., godfrey, m., randall, t., brettle, a., & grant, m. j. (2002c). hcprdu evaluation tool for mixed methods studies. leeds: university of leeds, nuffield institute for health.   koufogianniakis, d., booth, a., & brettle, a. (2006). reliant: readers guide to the literature on interventions addressing the need for education and training. library and information research, 30, 44-51.   kmet l. m., lee r. c., & cook l. s. (2004). standard quality assessment criteria for evaluating primary research papers from a variety of fields. edmonton: alberta heritage foundation for medical research (ahfmr). hta initiative #13.   atkins, c., & sampson, j. (2002). critical appraisal guidelines for single case study research. proceedings of the xth european conference on information systems (ecis), gdansk, poland, 6-8 june 2002.   morrison, j. m., sullivan, f., murray, e. & jolly, b. (1999). evidence-based education: development of an instrument to critically appraise reports of educational interventions. medical education, 33, 890-893.        3 (beverley et al, 2004; rankin et al., 2008; winning & beverley, 2003)   3 (bergman et al., 2010; catalano, 2013; manning fiegen, 2010)   3 (brettle, 2003a, 2007; brettle et al., 2011)               1 (brettle, 2007)     1 (booth et al., 2009)     1 (booth et al., 2009)     1 (koufogiannakis & wiebe, 2006) scales   downs, s. h. & black, n. (1998). the feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care -interventions, journal of epidemiology and community health, 52(6), 377-384.   nelson e.a. (1999). critical appraisal 8: questions for surveys. nursing times learning curve, 3(8), 5-7. wells g., shea b. j., o’connell, d., peterson, d., welch, v., losos, m., & tugwell, p. (n.d.). the newcastle-ottawa scale (nos) for assessing the quality of nonrandomised studies in meta-analyses. available at http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp       1 (zhang et al., 2007)   2 (beverley et al., 2004)   1 (perrier et al., 2014)   domain-based   cochrane effective practice and organisation of care (epoc) group. (2010). draft epoc methods paper: including interrupted time series (its) designs in a epoc review.         3 (brennan et al, 2011; divall et al., 2013; perrier et al., 2014) references to other publications study referencing the publication books   burton, d. (ed.) (2000). research training for social scientists. london: sage publications.   de vaus, d. a. (1991). surveys in social research, 3rd edition. london: allen & unwin.   gomm, r., needham, g., & bullman, a. (2000). evaluating research in health and social care. london: sage publications.   journal articles   boynton, p. m. (2004). hands on guide to questionnaire research: selecting, designing and developing your questionnaire. british medical journal, 328, 1312.   jamtvedt, g., young, j. m., kristoffersen, d. t., o’brien, m.,a., & oxman, a. d. (2006). audit and feedback: effects on professional practice and health care outcomes. cochrane database systematic reviews. http://dx.doi.org/10.1002/14651858.cd000259.pub2   lijmer j. c., mol, b. w., heisterkamp, s., bonsel, g. j., prins, m. h., van der meulen j. h., bossuyt, p. m. (1999). empirical evidence of design-related bias in studies of diagnostic tests. journal of the american medical association, 282, 1061-1066.   malhotra, m. k., & grover, v. (1998). an assessment of survey research in pom: from constructs to theory. journal of operations management, 16(4), 407-425.   mcdonald, s., crumley, e., eisinga, a., & villanueva, e. (2007). search strategies to identify reports of randomized trials in medline (protocol). cochrane database of systematic reviews. http://dx.doi.org/10.1002/14651858.mr000018.pub2   mi, m., & gilbert, c., m. (2007). needs assessment: prerequisite for service excellence. journal of hospital librarianship, 7, 31-52.   polgar, s., & thomas, s. a. (1995). critical evaluation of published research. in  introduction to research in the health sciences, 3rd edition (343–355). melbourne: churchill livingstone.   robinson, l., & bawden, d. (2007). evaluation of outreach services for primary care and mental health; assessing the impact. health information and libraries journal, 24, 57-66.   urquhart, c. j., & hepworth, j. b. (1996). comparing and using assessment of the value of information to clinical decision-making. bulletin of the medical library association, 84, 482-489.   weightman, a. & williamson, j. (2005). the value and impact of information provided through library services for patient care: a systematic review. health information and libraries journal, 22, 4-25.   yeoman, a. j., cooper, j. m., urquhart, c. j.,  & tyler, a. (2003). the management of health library outreach services: evaluation and reflection on lessons learned on the vivos project. journal of the medical library association, 91, 426-33.   web resource   intertasc information specialists’ sub-group search filter resource. (2008). issg search filter resource. retrieved from https://sites.google.com/a/york.ac.uk/issg-search-filters-resource/home       (weightman & williamson, 2005)   (weightman & williamson, 2005)   (beverley et al., 2004)         (booth et al., 2009)     (gagnon et al., 2010)       (crumley et al., 2004)       (sommestad et al., 2014)     (golder & loke, 2009)       (gray et al., 2012)     (gray et al., 2012)     (gray et al., 2012)     (gray et al., 2012)       (brettle et al., 2011)       (gray et al., 2012)       (golder & loke, 2009) abrettle 2003 was based on earlier versions of the hcprdu evaluation tools     table 4 prisma and amstar assessment of quality criteria study prisma amstar   prisma #12a prisma #15b prisma #19c prisma #22d amstar #7e amstar #8f amstar # 10g ankem (2005) inadequate unclear inadequate unclear inadequate inadequate unclear ankem (2006a) unclear unclear inadequate unclear inadequate unclear unclear bergman & holden (2010) inadequate unclear inadequate unclear inadequate inadequate unclear beverley et al. (2004) inadequate unclear inadequate unclear adequate inadequate unclear booth et al. (2009) adequate unclear inadequate unclear adequate unclear unclear brennan et al. (2011) inadequate unclear inadequate adequate inadequate adequate unclear brettle (2003) inadequate unclear inadequate unclear inadequate adequate unclear brettle et al. (2011) inadequate unclear adequate unclear adequate adequate unclear brettle (2007) inadequate unclear inadequate unclear inadequate adequate unclear brown (2008) unclear unclear inadequate unclear inadequate unclear unclear burda & teuteberg (2013) unclear unclear unclear unclear unclear unclear unclear catalano (2013) inadequate unclear inadequate unclear adequate inadequate unclear childs et al. (2005) unclear unclear unclear unclear unclear unclear unclear cooper and crum (2013) unclear unclear unclear unclear unclear unclear unclear crumley et al. (2005) inadequate unclear inadequate unclear inadequate inadequate unclear divall et al. (2013) inadequate unclear adequate adequate adequate inadequate unclear du preez (2007) unclear unclear unclear unclear unclear inadequate unclear duggan & banwell (2004) unclear unclear unclear unclear unclear unclear unclear fanner & urquhart (2008) unclear unclear unclear unclear inadequate inadequate unclear gagnon et al (2010) inadequate unclear adequate unclear adequate adequate unclear genero et al. (2011) unclear unclear unclear unclear inadequate inadequate unclear golder & loke (2010) inadequate unclear inadequate unclear inadequate inadequate unclear golder & loke (2009) inadequate unclear adequate unclear adequate unclear unclear grant (2007) unclear unclear unclear unclear unclear unclear unclear gray et al. (2012) inadequate unclear inadequate unclear inadequate unclear unclear joshi & trout (2014) inadequate unclear unclear unclear unclear inadequate unclear kelly & sugimoto (2013) unclear unclear inadequate unclear inadequate inadequate unclear koufogiannakis and wiebe (2006) adequate unclear inadequate unclear inadequate adequate unclear manning fiegen (2010) inadequate unclear inadequate unclear inadequate adequate unclear matteson et al. (2011) unclear unclear unclear unclear inadequate unclear unclear ndabarora et al. (2014) inadequate unclear inadequate unclear inadequate inadequate unclear perrier et al. (2014) inadequate unclear inadequate adequate unclear inadequate unclear phelps & campbell (2013) unclear unclear unclear unclear unclear unclear unclear rankin et al., (2008) inadequate unclear unclear unclear inadequate inadequate unclear sommestad et al. (2014) adequate adequate inadequate adequate adequate adequate adequate urquhart & yeoman (2010) unclear unclear unclear unclear unclear unclear unclear wagner & byrd (2004) unclear unclear unclear unclear inadequate inadequate unclear weightman & williamson (2005) inadequate unclear inadequate unclear adequate inadequate unclear winning & beverley (2003) inadequate unclear inadequate unclear inadequate inadequate unclear zhang et al. (2007) inadequate unclear adequate unclear adequate adequate unclear arisk of bias in individual studies; describe methods used for assessing risk of bias of individual studies (including specification of whether this was done at the study level or outcome level) and how this information is to be used in any data synthesis brisk of bias across studies; specify any assessment of risk of bias that may affect the cumulative evidence (e.g., publication bias, selective reporting within studies) crisk of bias within studies; present data on risk of bias of each study and, if available, any outcome level assessment drisk of bias across studies; present results of any assessment of risk of bias across studies ewas the scientific quality of the included studies assessed and documented? fwas the scientific quality of the included studies used appropriately in formulating conclusions? gwas the likelihood of publication bias assessed?     amstar assessment   only one quarter (10 of 40) of the systematic reviews included in this analysis adequately assessed and documented the scientific quality of the included studies (amstar item #7; shea et al., 2007). twenty studies were assessed as inadequate because, although quality assessment was documented, 14 studies (ankem, 2005; bergman & holden, 2010; brennan et al., 2011; brettle, 2003, 2007; brown, 2008; crumley et al., 2005; golder & loke, 2010; gray et al., 2012; koufogiannakis and wiebe 2006; manning fiegen, 2010; ndabarora et al., 2014; rankin 2008; winning & beverley, 2003) failed to report “some kind of result for each study” (amstar criteria #7; shea et al., 2007, p. 5) and six studies (ankem, 2006a; fanner & urquhart, 2008; genero, 2011;, kelly & sugimoto, 2013; matteson et al., 2011; wagner & byrd, 2004) did not report their quality assessment methods a priori. in one study (perrier et al., 2014), determining whether quality assessment was documented and assessed in accordance with the amstar item #7 was not possible because the link to the online supplementary table detailing the quality assessment was unavailable. the remaining eight studies failed to report on quality assessment at all; therefore, whether they met amstar item #7 was unclear.   in assessing the included studies against amstar item #8, which reads   was the scientific quality of the included studies used appropriately in formulating conclusions? the results of the methodological rigor and scientific quality should be considered in the analysis and the conclusions of the review, and explicitly stated in formulating recommendations. (shea et al., 2007, p. 5).   studies were classed as adequate if they incorporated how the quality of the included studies impacted on the validity of the systematic review findings in both the analysis and conclusion and also considered quality issues in their recommendations. the studies were classed as inadequate if they addressed the quality of the included studies in only one of these sections. they were classed as unclear if the studies did not report the quality of the included studies anywhere.   using the above criteria, only 9 of the 40 included systematic reviews adequately incorporated quality assessment in the analysis, conclusions, and recommendations. just one study (sommestad et al., 2014) met the final amstar quality criteria—assessing the likelihood of publication bias (item #10)—by providing a funnel plot.   five studies (du preez, 2007; fanner & urquhart, 2008; genero et al., 2011; kelly & sugimoto, 2013; wagner & byrd, 2004) incorporated some discussion of the quality of the included studies without explicitly reporting that quality assessment would be undertaken in the review methods.   discussion   the results section demonstrates great variation on the breadth, depth, and transparency of the quality assessment process in lis systematic reviews. nearly one third of the lis systematic reviews included in this study did not report on quality assessment in the methods. less than one quarter adequately incorporated quality assessment in the analysis, conclusions, and recommendations. quality assessment is an essential part of the systematic review process (moher et al., 2009; higgins et al., 2011; crd, 2009). without it, a systematic review loses one of the advantages it has over traditional literature reviews and is in danger of conforming to the old adage of “garbage in, garbage out” (yuan and hunt, 2009), where ignoring the impact of methodological quality may result in misleading conclusions (verhagen, de vet, de bie, boers, & van den brandt, 2001; mallen, peat, & croft, 2006). in particular, a lack of consistency in the understanding and application of the systematic review terminology appears to exist not only between lis authors but also across studies published in the same journal. for example, the majority (14) of lis systematic reviews were published in health information and libraries journal. of these, four reported only one author (brettle, 2003, 2007; brown, 2008, grant, 2007), and three did not assess the quality of the included studies (childs, blenkinsopp, hall, a. & walton, 2005, fanner & urquhart, 2008; grant, 2007).   the question is, does it matter if authors do not consider quality assessment in the analysis of a systematic review? although no empirical evidence within the lis domain suggests that the quality of the studies impacts on the validity of findings in lis-related systematic reviews, there is evidence that the quality of the included studies can yield differences in review results (voss & rehfuess, 2013). although guidance on the reporting of qualitative synthesis includes four items on the appraisal of the included studies (tong, flemming, mcinnes, oliver, & craig, 2012), the debate on whether to undertake quality assessment in qualitative systematic reviews is ongoing with insufficient evidence to support the inclusion or exclusion of quality assessment (noyes et al., 2015).   only nine of the 26 systematic reviews that undertook some form of quality assessment incorporated considerations of how the quality of the included studies impacted on the validity of the review findings in the analysis, conclusion, and recommendations. ignoring the extent to which the quality of the included studies may impact on the validity of the review findings, undertaking quality assessment in isolation makes the act of quality assessment within the systematic review a rather futile exercise (de craen, van vliet, & helmerhorst, 2005). the fact that lis systematic reviewers fail to incorporate how the quality of the included studies impacts on the overall review findings is not surprising given that similar studies in the field of health and medicine have shown only slightly better results (katikireddi, egan, & petticrew, 2015; de craen et al., 2005; moher et al., 1999; hayden, côté, & bombardier, 2006). the findings of this study agree with katikireddi et al. when they state that systematic review conclusions “are frequently uninformed by the critical appraisal process, even when conducted” (2015, p. 189).   conversely, a number of systematic reviews (du preez, 2007; fanner & urquhart, 2008; genero et al., 2011; kelly & sugimoto, 2013; wagner & byrd, 2004) raised the issue of the quality of the included studies in their discussion; however, their comments may not be valid since it was unclear how the quality of the studies was assessed. similarly, four studies (brennan et al, 2011; divall et al., 2013; perrier et al., 2014; sommestad et al., 2014) reported on publication or selection bias, but only one outlined their methods a priori (sommestad et al., 2014).   de craen et al. (2005) put forward a number of theories as to why systematic reviewers may not incorporate quality assessment into the analysis. firstly, reviewers may not know that quality assessment should be considered in the analysis, or secondly, they simply may not know how to incorporate the quality assessment into the analysis. conversely, it may be that the reviewers’ focus is more on the tools used to assess quality, many of which are designed to assess the quality of individual studies, rather than across a group of studies. this raises important questions over the nature of the guidance used by lis reviewers when undertaking a systematic review. a quick look at the guidance referred to in the systematic reviews in this study reveals that lis reviewers follow a range of guidance when undertaking a systematic review, from the more formal (e.g., higgins & green, 2011; crd, 2009) to single journal articles providing a rather short, introductory overview of the systematic review.   while there are numerous texts explaining how to conduct a systematic review, they are largely written from the perspective of the healthcare professional rather than the lis professional (e.g. booth, papaioannou, & sutton, 2012; crd, 2009; higgins & green, 2011). currently there is no comprehensive guidance with a focus on the different approaches to evidence synthesis written purely from a lis perspective with relevant guided examples of how to undertake and incorporate quality assessment in the analysis. the findings of this study appear to demonstrate a need for such a resource or series of guides. however, even when comprehensive guidance is available, such as in the healthcare domain, the findings of previous methodology studies examining the incorporation of quality assessment in systematic reviews (hayden et al., 2006; katikireddi et al., 2015) seem to suggest that reviewers still fail to address how the quality of included studies impacts on the validity of the review findings.   de craen et al. (2005) also suggest that reviewers may see the incorporation of quality assessment in the analysis as a “cumbersome procedure” which might “further complicate the interpretation of its results” (p. 312). it is certainly the case that the heterogeneous nature of the lis evidence base requires lis reviewers to consider the quality of studies across diverse research designs. this adds another level of complexity to the quality assessment process since different biases may arise according to the type of research design, which makes comparisons across studies more difficult. furthermore, quality assessment is something that is out of the comfort zone of many librarians (maden-jenkins, 2011).   critical to the understanding of how quality impacts on the review findings is the reviewers’ definition of quality. four definitions of quality were identified in lis systematic reviews: reporting quality, study design, methodological quality (internal and external validity), and risk of bias (internal validity). while an assessment of bias in research does rely on the quality of the reporting, assessing the quality of the reporting can become more of a descriptive exercise in recording whether or not methods were reported, rather than assessing whether the methods were adequately conducted in order to reduce bias. similarly, basing quality assessment on study design may lead reviewers to base quality on the level of evidence rather than the process used to conduct the study, which ignores the possibility that high levels of evidence, such as systematic reviews or randomized controlled trials, may have been poorly conducted and therefore susceptible to bias.   part of this problem may be that quality assessment tools that purport to assess methodological quality are, on further examination, actually assessing the reporting quality. the jadad tool (jadad et al., 1996) is a prime example of this where reviewers are asked to assess whether the study was described as a double-blinded randomized controlled trial. even the criteria used in amstar to critique the approach to quality assessment in systematic reviews goes no further than to address whether or not the methods were reported a priori. reviewers, therefore, should critique their own approach to quality assessment to ensure that the criteria or tool they select for quality assessment is appropriate and fit for purpose.   for those systematic reviews in this study that do report on quality assessment in the methods, there is need for greater transparency in the reporting process. this can be a fairly simple process of tabulating the quality assessment in tables or figures, such as in cochrane reviews. reporting on the quality assessment items for each study allows the reader to see exactly on what criteria (methodology, reporting, etc.) judgments of quality were made, while at the same time making it easier for reviewers to judge the overall quality of the evidence base.   identifying the type of tool and resources lis reviewers used to assess the quality of the evidence was not straightforward. the aids identified went beyond the use of tools developed specifically for quality assessment. the large number of different quality assessment tools identified reflects not only the disparate nature of the lis evidence base (brettle, 2009), but also a lack of consensus around criteria on which to assess the quality of lis research. given the diverse nature of the lis evidence base and the multiple study designs often incorporated into lis reviews (see table 1), quality assessment tools with a more generic focus on qualitative, quantitative, or mixed methods focus rather than a study design focus (e.g. randomized controlled trial) may help reviewers compare and contrast the quality of the included studies more easily. lis reviewers may wish to look at how reviews incorporating a wide variety of study designs approach quality assessment (e.g. the campbell collaboration).   due to the broad nature of some of the amstar and prisma criteria, it was sometimes difficult to interpret the criteria and make a clear judgment on some of the quality items assessed. for example, amstar item #8 asks “was the scientific quality of the included studies used appropriately in formulating conclusions?” (shea et al., 2007, p. 5). the accompanying notes suggest that “the results of the methodological rigor and scientific quality should be considered in the analysis and the conclusions of the review, and explicitly stated in formulating recommendations” (shea et al., 2007, p. 5). for example, some studies reported on the impact of quality assessment on the review findings in the analysis but not the conclusions, while others reported recommendations for improving the quality of future research but failed to assess the impact the quality of the included studies had on the review findings. the criteria also lacked transparency in assessing whether the tools and approaches to quality assessment were appropriate.   for those undertaking lis systematic reviews, consideration therefore should be given to the prisma and amstar criteria (box 2) for incorporating considerations of quality assessment in systematic reviews, specifically how the quality of the included studies may impact on the validity of the overall review findings. in addition, reviewers should ensure that whatever criteria or tool they use for quality assessment is fit for purpose. in other words, reviewers should critique their chosen set of criteria or tool to ensure it reflects the purpose of the quality assessment (e.g. methodological quality versus reporting quality). given that tools aiming to assess methodological quality often, on further examination, are found to actually assess reporting quality, further research on the appropriateness of tools and criteria selected to quality assessment in lis reviews is warranted. further research should also examine what criteria are necessary to adequately assess the quality of studies included in lis systematic reviews. above all, there is a need for tailored lis systematic review guidance with accompanying exemplar case studies of lis systematic reviews.   strengths and limitations   one reviewer of this study extracted data from all included studies. one of the reviewers (mm) also co-authored one of the included studies (brettle et al., 2011); therefore, a second reviewer (ek) checked the data extraction for accuracy. while we used an existing resource that listed published lis systematic reviews, it is possible that other published lis systematic reviews were not listed on the wiki. we included only studies that reported themselves as being a systematic review. other studies may have followed systematic review principles but were not explicit in labelling themselves as such. no attempt was made to contact the authors of the included studies for further clarification. this study did not seek to critique the reviewers choice of quality assessment tool but rather to identify the tools used and the approach for incorporating considerations of quality assessment in systematic reviews. finally, perhaps the major limitation in the way this study was conducted is that 18 of the included lis studies were published before the prisma guidelines (moher et al., 2009) were available, and 11 were published before amstar tool (shea et al., 2007) was available. however, even studies published after these dates show only a very small improvement in meeting the criteria (see table 4) and there is still a long way to go in improving quality assessment methods in lis systematic reviews.   conclusions   although quality assessment of included studies is an integral part of the systematic review methodology, the extent to which it is documented and undertaken in lis systematic reviews varies widely. the results of this study demonstrate a need for greater clarity, definition, and understanding of the methodology and concept of quality in the systematic review process, not only by lis reviewers but also by editors of journals who accept such studies for publication. due to the diverse nature of the lis evidence base, work still needs to be done to identify the best tools and approaches for incorporating considerations of quality in lis systematic reviews. what is clear from this analysis is that lis reviewers need to improve the robustness and transparency with which they undertake and report quality assessment in systematic reviews. above all, lis reviewers need to be explicit in coming to a conclusion on how the quality of the included studies may impact on their review findings. in considering this, lis reviewers can therefore increase the validity of their systematic review.   disclaimer: the views expressed are those of the author(s) and not necessarily those of the nhs, the nihr or the department of health.   references   aabø, s. (2009). libraries and return on investment (roi): a meta-analysis. new library world, 110(7/8), 311-324. http://dx.doi.org/10.1108/03074800910975142   ankem, k. (2005). types of information needs among cancer patients: a systematic review. libres: library and information science research electronic journal, 15(2). retrieved from http://libres-ejournal.info/   ankem, k. (2006a). use of information sources by cancer patients: results of a systematic review of the research literature. information research, 11(3). retrieved from https://dialnet.unirioja.es/servlet/revista?codigo=6956   ankem, k. (2006b). factors influencing information needs among cancer patients: a meta-analysis. library & information science research, 28(1), 7-23.  http://dx.doi.org/10.1016/j.lisr.2005.11.003   bergman, e. m. l., & holden, i. i. (2010). user satisfaction with electronic reference: a systematic review. reference services review, 38(3), 493-509. http://dx.doi.org/10.1108/00907321011084789   beverley, c. a., bath, p. a., & booth, a. (2004). health information needs of visually impaired people: a systematic review of the literature. health & social care in the community, 12(1), 1-24. http://dx.doi.org/10.1111/j.1365-2524.2004.00460.x   boland, a., cherry, m.g., & dickson, r. (2013). doing a systematic review. a student’s guide. london: sage publications ltd.   booth, a. (2000). research. health information & libraries journal, 17(4), 232-235. http://dx.doi.org/10.1111/j.1471-1842.2000.00295.x   booth, a. (2007). who will appraise the appraisers? —the paper, the instrument and the user. health information and libraries journal, 24(1), 72-76. http://dx.doi.org/10.1111/j.1471-1842.2007.00703.x   booth, a., & brice, a. (2004). appraising the evidence. in booth & brice (eds.), evidence-based practice for information professionals: a handbook (pp. 96-110). london, uk: facet publishing.   booth, a., carroll, c., papaioannou, d., sutton, a., & wong, r. (2009). applying findings from a systematic review of workplace-based e-learning: implications for health information professionals. health information and libraries journal, 26(1), 4-21. http://dx.doi.org/10.1111/j.1471-1842.2008.00834.x   booth, a., papaioannou, d., & sutton, a. (2012). systematic approaches to a successful literature review. london: sage publications ltd.   brennan, n., mattick, k., & ellis, t. (2011). the map of medicine: a review of evidence for its impact on healthcare. health information and libraries journal, 28(2), 93-100. http://dx.doi.org/10.1111/j.1471-1842.2011.00940.x   brettle, a. (2003). information skills training: a systematic review of the literature. health information and libraries journal, 20(suppl. 1), 3-9. http://dx.doi.org/10.1046/j.1365-2532.20.s1.3.x   brettle, a. (2007). evaluating information skills training in health libraries: a systematic review. health information and libraries journal, 24(suppl. 1), 18-37.   brettle, a. (2009) systematic reviews and evidence based library and information practice. evidence based library and information practice, 4(1), 43-50. retrieved from https://ejournals.library.ualberta.ca/index.php/eblip/index   brettle, a., maden-jenkins, m., anderson, l., mcnally, r., pratchett, t., tancock, j., thornton, d., & webb, a. (2011). evaluating clinical librarian services: a systematic review. health information and libraries journal, 28(1), 3-22. http://dx.doi.org/10.1111/j.1471-1842.2010.00925.x   brown, c. (2008). the information trail of the 'freshman 15'—a systematic review of a health myth within the research and popular literature. health information and libraries journal, 25(1), 1-12. http://dx.doi.org/10.1111/j.1471-1842.2007.00762.x   burda, d., & teuteberg, f. (2013). sustaining accessibility of information through digital preservation: a literature review. journal of information science, 39(4), 442-458. http://dx.doi.org/10.1177/0165551513480107   catalano, a. j. (2013). patterns of graduate students' information seeking behavior: a meta synthesis of the literature. journal of documentation, 69(2). http://dx.doi.org/10.1108/00220411311300066   centre for reviews and dissemination (crd). (2009). systematic reviews: crd's guidance for undertaking reviews in health care. york, uk: centre for reviews and dissemination, university of york.   childs, s., blenkinsopp, e., hall, a., & walton, g. (2005). effective e-learning for health professionals and students—barriers and their solutions. a systematic review of the literature—findings from the hexl project. health information and libraries journal, 22(suppl. 2), 20-32. http://dx.doi.org/10.1111/j.1470-3327.2005.00614.x   cooper, i. d., & crum, j. a. (2013). new activities and changing roles of health sciences librarians: a systematic review, 1990-2012. journal of the medical library association, 101(4), 268-277. retrieved from http://www.ncbi.nlm.nih.gov/pmc/journals/93/   crumley, e.t., wiebe, n., cramer, k., klassen, t.p., & hartling, l. (2005). which resources should be used to identify rct/ccts for systematic reviews: a systematic review. bmc medical research methodology, 5:24. http://dx.doi.org/10.1186/1471-2288-5-24   de craen, a. j., van vliet, h. a., & helmerhorst, f. m. (2005). an analysis of systematic reviews indicated low incorporation of results from clinical trial quality assessment. journal of clinical epidemiology, 58(3), 311-313. http://dx.doi.org/10.1016/j.jclinepi.2004.07.002   deeks, j. j., dinnes, j., d’amico, r., sowden, a. j., sakarovitch, c., song, f., petticrew, m., & altman, d. g.. (2003). evaluating non-randomised controlled intervention studies. health technology assessment, 7, 1-173. retrieved from http://researchonline.lshtm.ac.uk/id/eprint/8742   divall, p., camosso-stefinovic, j., & baker, r. (2013). the use of personal digital assistants in clinical decision making by health care professionals: a systematic review. health informatics journal, 19(1), 16-28. http://dx.doi.org/10.1177/1460458212446761   du preez, m. (2007). information needs and information-seeking behaviour of engineers: a systematic review. mousaion, 25(2), 72-94. retrieved from http://www.unisa.ac.za/default.asp?cmd=viewcontent&contentid=2008   duggan, f., & banwell, l. (2004). constructing a model of effective information dissemination in a crisis. information research, 9(3). retrieved from http://www.informationr.net/ir/   eldredge, j. d. (2000). evidence-based librarianship: an overview. bulletin of the medical library association, 88(4), 289-302. retrieved from http://www.ncbi.nlm.nih.gov/pmc/journals/72/   fanner, d., & urquhart, c. (2008). bibliotherapy for mental health service users part 1: a systematic review. health information and libraries journal, 25(4), 237-252. http://dx.doi.org/10.1111/j.1471-1842.2008.00821.x   gagnon, m.p., pluye, p., desmartis, m., car, j., pagliari, c., labrecque, m., fremont, p., gagnon, j., njoya, m., & legare, f. (2010). a systematic review of interventions promoting clinical information retrieval technology (cirt) adoption by healthcare professionals. international journal of medical informatics, 79(10), 669-680. http://dx.doi.org/10.1016/j.ijmedinf.2010.07.004   genero, m., fernandez-saez, a. m., nelson, h. j., poels, g., & piattini, m. (2011). research review: a systematic literature review on the quality of uml models. journal of database management, 22(3), 46-70. http://dx.doi.org/10.4018/jdm.2011070103   golder, s., & loke, y. k. (2009). search strategies to identify information on adverse effects: a systematic review. journal of the medical library association, 97(2), 84-92. http://dx.doi.org/10.3163/1536-5050.97.2.004   golder, s., & loke, y. k. (2010). sources of information on adverse effects: a systematic review. health information and libraries journal, 27(3), 176-190. http://dx.doi.org/10.1111/j.1471-1842.2010.00901.x   grant, m. j. (2007). the role of reflection in the library and information sector: a systematic review. health information and libraries journal, 24(3), 155-166. http://dx.doi.org/10.1111/j.1471-1842.2007.00731.x   grant, m., & booth, a. (2009). a typology of reviews: an analysis of 14 review types and associated methodologies. health information and libraries review, 26(2), 91-108. http://dx.doi.org/10.1111/j.1471-1842.2009.00848.x gray, h., sutton, g., & treadway, v. (2012). do quality improvement systems improve health library services? a systematic review. health information and libraries journal, 29(3), 180-196. http://dx.doi.org/10.1111/j.1471-1842.2012.00996.x   haug, j. d. (1997). physicians' preferences for information sources: a meta-analytic study, bulletin of the medical library association, 85(3), 223–32. retrieved from http://www.ncbi.nlm.nih.gov/pmc/journals/72/   hayden, j. a., côté, p. & bombardier, c. (2006). evaluation of the quality of prognostic studies in systematic reviews. annals of internal medicine, 144(6), 427-437. http://dx.doi.org/10.7326/0003-4819-144-6-200603210-00010   higgins j. p. t., & green, s. (eds.). (2011). cochrane handbook for systematic reviews of interventions, version 5.3.0. the cochrane collaboration, 2011. available from http://handbook.cochrane.org/   jadad a. r., moore r. a., carroll d., jenkinson, c., reynolds, d. j., gavaghan, d. j., & mcquay, h. j. (1996). assessing the quality of reports of randomized clinical trials: is blinding necessary? controlled clinical trials, 17(1), 1-12. http://dx.doi.org/10.1016/0197-2456(95)00134-4   joshi, a, & trout, k. (2014). the role of health information kiosks in diverse settings: a systematic review. health information and libraries journal, 31(4), 254-273. http://dx.doi.org/10.1111/hir.12081   julien, c.-a., leide, j. e., & bouthillier, f. (2008). controlled user evaluations of information visualization interfaces for text retrieval: literature review and meta-analysis. journal of the american society for information science and technology, 59(6), 1012-1024. http://dx.doi.org/10.1002/asi.20786   jüni, p. altman, d. g., & egger, m. (2001). assessing the quality of controlled clinical trials. british medical journal, 323(7303), 42-46. http://dx.doi.org/10.1136/bmj.323.7303.42   katikireddi, s. v., egan, m., & petticrew, m. (2015). how do systematic reviews incorporate risk of bias assessments into the synthesis of evidence? a methodological study. journal of epidemiology and community health 69(2), 189-195. http://dx.doi.org/10.1136/jech-2014-204711   kelly, d., & sugimoto, c. r. (2013). a systematic review of interactive information retrieval evaluation studies, 1967-2006. journal of the american society for information science and technology, 64(4), 745-770. http://dx.doi.org/10.1002/asi.22799   koufogiannakis, d. (2012). the state of systematic reviews in library and information studies. evidence based library and information practice, 7(2), 91-95. retrieved from https://ejournals.library.ualberta.ca/index.php/eblip/index   koufogiannakis, d., brettle, a., booth, a., kloda, l., & urquhart, c. (2015). lis systematic reviews. retrieved from http://lis-systematic-reviews.wikispaces.com/welcome   koufogiannakis, d., & wiebe, n. (2006). effective methods for teaching information literacy skills to undergraduate students: a systematic review and meta-analysis. evidence based library and information practice, 1(3), 3-43. retrieved from https://ejournals.library.ualberta.ca/index.php/eblip/index   long, a. f., godfrey, m., randall, t., brettle, a., & grant, m. j. (2002a). hcprdu evaluation tool for qualitative studies. leeds: university of leeds, nuffield institute for health. retrieved from http://usir.salford.ac.uk/12970/   long, a. f., godfrey, m., randall, t., brettle, a., & grant, m. j. (2002b). hcprdu evaluation tool for quantitative studies. leeds: university of leeds, nuffield institute for health. retrieved from  http://usir.salford.ac.uk/12969/   long, a. f., godfrey, m., randall, t., brettle, a., & grant, m. j. (2002c). hcprdu evaluation tool for mixed methods studies. leeds: university of leeds, nuffield institute for health. retrieved from http://usir.salford.ac.uk/13070/   maden-jenkins, m. (2011). healthcare librarians and the delivery of critical appraisal training. health information and libraries journal, 28(1), 33-40. http://dx.doi.org/10.1111/j.1471-1842.2010.00899.x   manning fiegen, a. (2010). systematic review of research methods: the case of business instruction. reference services review, 38(3), 385-397. http://dx.doi.org/10.1108/00907321011070883   mairs, k., mcneil, h., mcleod, j., prorok, j. c., & stolee, p. (2013). online strategies to facilitate health-related knowledge transfer: a systematic search and review. health information and libraries journal, 30(4), 261-277. http://dx.doi.org/10.1111/hir.12048   mallen, c., peat, g., & croft, p. (2006). quality assessment of observational studies is not commonplace in systematic reviews. journal of clinical epidemiology, 59, 765-769. http://dx.doi.org/10.1016/j.jclinepi.2005.12.010   matteson, m. l., salamon, j., & brewster, l. (2011). a systematic review of research on live chat service. reference & user services quarterly, 51(2), 172-190. retrieved from http://www.jstor.org/stable/refuseserq.51.2.172   moher, d., cook, d. j., jaded, a. r., tugwell, p., moher, m., jones, a., pham, b., & klassen, t. p. (1999). assessing the quality of reports of randomised controlled trials: implications for the conduct of meta-analyses. health technology assessment, 3(12), 1-98. http://dx.doi.org/10.3310/hta3120   moher, d., liberati, a., tetzlaff, j., altman, d. g., & the prisma group. (2009). preferred reporting items for systematic reviews and meta-analyses: the prisma statement. annals of internal medicine, 151(4), w65-94. http://dx.doi.org/10.7326/0003-4819-151-4-200908180-00135   ndabarora, e., chipps, j. a., & uys, l. (2014). systematic review of health data quality management and best practices at community and district levels in lmic. information development, 30(2), 103-120. http://dx.doi.org/10.1177/0266666913477430   noyes, j., hannes, k., booth, a., harris, j., harden, a., popay, j., pearson, a., cargo, m., & pantoja, t., on behalf of the cochrane qualitative and implementation methods group.  qualitative research and cochrane reviews. in j. p. t. higgins & s. green (eds.), cochrane handbook for systematic reviews of interventions version 5.3.0 (updated october 2015). the cochrane collaboration, 2015. available from http://qim.cochrane.org/supplemental-handbook-guidance   perrier, l., farrell, a., ayala, a. p., lightfoot, d., kenny, t., aaronson, e., allee, n., brigham, t., connor, e., constantinescu, t., mullenbach, j., epstein, h. a. b., & weiss, a. (2014). effects of librarian-provided services in healthcare settings: a systematic review. journal of the american medical informatics association, 21(6), 1118-1124. http://dx.doi.org/10.1136/amiajnl-2014-002825   phelps, s. f., & campbell, n. (2012). commitment and trust in librarian-faculty relationships: a systematic review of the literature. journal of academic librarianship, 38(1), 13-19. http://dx.doi.org/10.1016/j.acalib.2011.11.003   rankin, j. a., grefsheim, s. a., & canto, c. c. (2008). the emerging informationist speciality: a systematic review of the literature. journal of the medical library association, 96(3), 194-206. http://dx.doi.org/10.3163/1536-5050.96.3.005   salang, m. m. c. (1996). a meta-analysis of studies on user information needs and their relationship to information retrieval. journal of philippine librarianship, 18(2), 36-56.   sampson, m., mcgowan, j., lefebvre, c., moher, d., & grimshaw, j. (2008). press: peer review of electronic search strategies. ottawa: canadian agency for drugs and technologies in health. retrieved from https://www.cadth.ca/media/pdf/477_press-peer-review-electronic-search-strategies_tr_e.pdf   sampson, m. (2014). welcoming systematic reviews to the journal of the medical library association. journal of the medical library association, 102(3):143-145. http://dx.doi.org/10.3163/1536-5050.102.3.001   saxton, m. l. (1997). reference service evaluation and meta-analysis: findings and methodological issues. the library quarterly, 67(3), 267-289. retrieved from http://www.jstor.org/stable/40039723   schulz, k. f., chalmers, i., hayes, r. j., & altman, d. g. (1995). empirical evidence of bias. dimensions of methodological quality associated with estimates of treatment effects in controlled trials. journal of the american medical association, 273(5), 408-412. http://dx.doi.org/10.1001/jama.1995.03520290060030   shea, b. j., grimshaw, j. m., wells, g. a., boers, m., andersson, n., hamel, c., porter, a. c., tugwell, p., moher, d., & bouter, l. m. (2007). development of amstar: a measurement tool to assess the methodological quality of systematic reviews. bmc medical research methodology, 7:10. http://dx.doi.org/10.1186/1471-2288-7-10   sommestad, t., hallberg, j., lundholm, k., & bengtsson, j. (2014). variables influencing information security policy compliance: a systematic review of quantitative studies. information management & computer security, 22(1), 42-75. http://dx.doi.org/10.1108/imcs-08-2012-0045   tong, a., flemming, k., mcinnes, e., oliver, s., & craig, j. (2012). enhancing transparency in reporting the synthesis of qualitative research: entreq. bmc medical research methodology, 12:181. http://dx.doi.org/10.1186/1471-2288-12-181   urquhart, c., & yeoman, a. (2010). information behaviour of women: theoretical perspectives on gender. journal of documentation, 66(1), 113-139. http://dx.doi.org/10.1108/00220411011016399   verhagen, a. p., de vet, h. c., de bie, r. a., boers, m., & van den brandt, p. m. (2001). the art of quality assessment of rcts included in systematic reviews. journal of clinical epidemiology, 54(7), 651-654. http://dx.doi.org/10.1016/s0895-4356(00)00360-7   voss, p. h., & rehfuess. e. a. (2013). quality appraisal in systematic reviews of public health interventions: an empirical study on the impact of choice of tool on meta-analysis. journal of epidemiology and community health, 67(1), 98-104. http://dx.doi.org/10.1136/jech-2011-200940   wagner, k. c., & byrd, g. d. (2004). evaluating the effectiveness of clinical medical librarian programs: a systematic review of the literature. journal of the medical library association, 92(1), 14-33. retrieved from http://www.ncbi.nlm.nih.gov/pmc/journals/93/   ward, r., stevens, c., brentnall, p., & briddon, j. (2008). the attitudes of health care staff to information technology: a comprehensive review of the research literature. health information and libraries journal, 25(2), 81-97. http://dx.doi.org/10.1111/j.1471-1842.2008.00777.x   weightman, a. l., & williamson, j. (2005). the value and impact of information provided through library services for patient care: a systematic review. health information and libraries journal, 22(1), 4-25. http://dx.doi.org/10.1111/j.1471-1842.2005.00549.x   west, s., king, v., carey, t. s., lohr, k. n., mckoy, n., sutton, s. f., & lux, l. (2002). systems to rate the strength of scientific evidence, evidence report/technology assessment no. 47. (ahrq publication no. 02-e016). rockville, md: agency for healthcare research and quality. retrieved from http://archive.ahrq.gov/clinic/epcsums/strengthsum.pdf   williams, p., nicholas d., & rowlands, i. (2010). the attitudes and behaviours of illegal downloaders. aslib proceedings, 62(3), 283-301. http://dx.doi.org/10.1108/00012531011046916   winning, m. a., & beverley, c. a. (2003). clinical librarianship: a systematic review of the literature. health information and libraries journal, 20(suppl. 1), 10-21. http://dx.doi.org/10.1046/j.1365-2532.20.s1.2.x   yuan, y., & hunt, r. (2009). systematic reviews: the good, the bad, and the ugly. american journal of gastroenterology, 104(5), 1086-1092. retrieved from http://www.nature.com/ajg/index.html   zhang, l., watson, e. m. & banfield, l. (2007). the efficacy of computer-assisted instruction versus face-to-face instruction in academic libraries: a systematic review. the journal of academic librarianship, 33(4), 478-484. http://dx.doi.org/10.1016/j.acalib.2007.03.00   evidence based library and information practice commentary   celebrating collaboration: what i have learned as production editor   michelle dunaway production editor phd student, wayne state university detroit, michigan, united states of america email: michelle.dunaway@wayne.edu   received: 1 jan. 2016      accepted: 9 feb. 2016      2016 dunaway. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     when i joined evidence based library and information practice as editorial intern in 2011, i was a master of library and information studies student looking for opportunities to get involved in library and information studies (lis) service and scholarship in any way that i could. i began serving as production editor in 2013, and since then, eblip has been increasingly influential in my professional goals and interests. of the many transformative learning experiences that i have had during my time with eblip, among the most significant is learning that publishing involves a degree of collaboration that is truly remarkable.   as production editor, my contribution to the journal is arguably more technical than editorial in nature. after manuscripts have been peer reviewed, accepted for publication, and copyedited, i prepare them for publication; i lay out the text, figures, and tables in eblip format and style, and put them together as an issue. i take a great deal of pride in polishing and perfecting the final version of a manuscript; although things like line spacing, paragraph headers, and the placement of figures and tables do not involve any actual editing of substantive content, these things are essential to creating aesthetically pleasing, readable articles. although i thoroughly enjoy the artistic, visual aspect of my role, it is the ongoing teamwork during this final stage of the publication process that is truly gratifying. after i format the items, i send them back to their respective editors, who inspect the formatting and do a final proofread. then, i send the documents to the editorial intern, who visually inspects them yet again. inevitably, at each stage of this collaborative proofing process, something is identified that can be improved upon, such as a typographical error, an extra line of space between paragraphs, or the use of american english instead of canadian english. because such things are so easily overlooked, our collaborative process is essential and extremely effective.   in addition to the formatting and perfecting manuscripts prior to publication, my role as production editor involves a significant amount of behind-the-scenes work with our publishing system, open journal systems (ojs). the university of alberta hosts the instance of ojs that eblip lives on, and when i’ve experienced technical difficulties with ojs that i just could not resolve on my own, the university of alberta has been yet another team member who i have been able to rely on and learn from. as production editor, i am very fortunate to be able to reach out to directly to various people at the university of alberta for guidance and technical support.   along with gaining an understanding of the amount of collaboration that goes into the creation of a single issue of eblip, i have also developed deep admiration and respect for the eblip editorial team as well as eblip’s contributing authors, for their respective roles in advancing the open access movement. the hard work that authors do in order to produce and publish quality research should be rewarded by having the fruits of that labour reach as broad an audience as possible. through my experiences as production editor, i have learned that making this happen requires an enormously dedicated editorial team.  for this reason, and many other reasons, i am so proud to be part of eblip, and thrilled to be celebrating its 10th year of publication with my outstanding colleagues!   microsoft word ed_1808 evidence based library and information practice 2008, 3:2 1 evidence based library and information practice editorial plans, updates and musings lindsay glynn editor-in-chief acting head, public services, health sciences library memorial university of newfoundland st. john’s, newfoundland, canada e-mail: lglynn@mun.ca © 2008 glynn. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. eblip5 is just around the corner. time to plan your research paper or poster and prepare a submission. how often do you get the chance to present your work and mingle among international colleagues in sweden? this is a great opportunity to showcase your projects, see what others are doing and discover opportunities for collaborations. the call for papers will be announced this summer. further information can be found on the eblip5 blog at <http://blogs.kib.ki.se/eblip5/>. the editorial board of eblip is excited about the upcoming feature issue on evidence based practice in school librarianship. the two guest editors, whose names many of you will undoubtedly recognize due to their huge body of work, carol gordon and ross todd, have an impressive and fascinating list of articles lined up as well as several evidence summaries and commentaries. this feature issue will be published in december 2008. plans are also underway for a feature issue on evidence based practice in public librarianship for december 2009. julie mckenna, many of you will know from her work in both academic and public libraries, has agreed to be the guest editor. informal discussions at conferences and meetings seem to indicate that, although people are interested in evidence based practice, the skill sets necessary are often lacking. to address this, eblip will feature a new column titled ebp 101, debuting in the 3(3) issue. this regular column, co-authored by su cleyle (eblip co-founder and former editor-in-chief) and julie mckenna, will highlight and focus on one specific skill per issue. while su and julie have several ideas already, it would be highly useful for eblip readers to indicate skills that they would like to see addressed. feel free to send your requests and suggestions to me at lglynn@mun.ca. if you read my last editorial, you’ll recall my ebp anxiety regarding the construction of the new computer lab in my library. many people have been asking about the progress, and i’m happy to provide a report. since evidence based library and information practice 2008, 3:2 2 asbestos abatement was necessary to prepare the space, there is a temporary sealed wall surrounding the construction site, and therefore, there is nowhere to peep through to see what’s happening inside. at this point in the process, then, all i can do is assume that there is something going on behind the sealed walls. this blind trust is most definitely an unexpected part of the evidence based process and it is slightly surreal. what i have noticed during the process, however, is that one of the most important aspects of such a large endeavor is not planning or designing, but communication. as you can imagine, this project requires decision making and communication on multiple levels and among multiple teams including librarians, staff, technical consultants, architects, electricians, faculty, administration, etc. without constant and transparent communication, any attempt at an evidence based process may as well be thrown out the window. to illustrate, and i am embarrassed to write this, the tender for construction was awarded and work had begun when it suddenly came to light that nobody on the planning committee had actually seen the final blueprints. we had all assumed that the proverbial someone had seen them and that all was fine. it turned out that the approved blueprints were not exactly what we had planned and, as a result, some backpedaling was required. had we been communicating properly with all involved, this would not have happened. note to self: always communicate and never assume. i’ll keep you posted on the progress. there are few tools available with which to evaluate the impact or use of an open access journal such as eblip. some statistics can be utilized in terms of download counts and registered users/readers, however these do not provide any information regarding how the articles are being used, who is using them or the impact that our published research is having on practice. i would be most interested to entertain suggestions on how to measure this or even an offer to take on such a project (hmm...what an idea for an eblip5 paper or poster…), but in the meantime it would be very useful to hear individual narratives on how information from this journal was applied in practice. this would be welcome in the form of a letter to the editor, a commentary or a brief “what we did” narrative. i’m sure that you will enjoy this summer issue of evidence based library and information practice and will find the commentary, articles and evidence summaries both interesting and useful. the evidence summaries touch on everything from collaboration to institutional depositories to e-books; the articles describe librarian’s evaluations of bedside evidence based tools and students’ non-use of library services. the commentary will get you thinking about the historical and social culture of knowledge. there is, undoubtedly, something here to pique everyone’s interest. there must be a research paper somewhere that proves the emotional and mental benefits of reading trashy books and drinking mojitos in the summer sunshine. with that in mind, be sure to take a few days off and enjoy your summer! evidence based library and information practice 2011, 7.4 152 evidence based library and information practice news/announcements lirg research scan award – call for proposals 2012. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. the library and information research group (lirg) research scan award intends to encourage dissemination and take-up of research and innovation in library and information science. the value of the research scan award is £500. the lirg research scan is not intended to be an annual award, but rather a special means to address an urgent current issue in library and information science, by acting as a summary of relevant research. it is not intended to generate new research, but rather to collect, analyse and present existing research in a report for application by practitioners. scan award 2012/13 – call for proposals what do lis practitioners want from research? to inform the development of a research policy for the chartered institute for library and information professionals (cilip), lirg are funding a scan of the literature which will help address the question what do lis practitioners want from research? the scan should cover the published literature, across all domains of lis and focus on at least the following questions:  what kind of research is relevant to lis practitioners?  what do practitioners understand by "research" and how do they use it?  what are the barriers and facilitators to using research in practice? general conditions of the award applicants should have the experience to perform a comprehensive review of the quantitative and qualitative methodologies and best practice currently available. applications are open to lone researchers or research teams. researchers may be academics and/or practitioners. researchers must be residents of the uk. please see below for how to apply and deadlines. the research scan award is not intended to create new research but involves performing a literature review of existing research. synonymous terms for this activity are considered to be systematic review, scoping review and horizon scanning. if in doubt about what is required, please see below for contact email addresses for enquiries. the final intended output of the lirg research scan will be a revised version of the conference paper and will be published in lirg's journal, library and information research. the research scan award is a fixed sum of £500 which will be paid on acceptance of the final paper for publication in library http://www.lirg.org.uk/lir/ojs/index.php/lir http://www.lirg.org.uk/lir/ojs/index.php/lir http://www.lirg.org.uk/lir/ojs/index.php/lir evidence based library and information practice 2011, 7.4 153 and information research. the research scan award is a fixed sum and is not intended to cover institutional costs of the work (if any). outline of workplan the detailed workplan for the lirg research scan 2012/13 is: december 2012: call for proposals. 7 january 2013: submission deadline. by 21 january 2013: lirg to select and notify the successful researcher/team. january through june 2013: selected researcher/team to compile a draft research scan paper on the given topic, and submit it by email to the lirg awards co-ordinator. 15 june 2013: submission of the draft research scan paper. 1 july 2013: lirg pre-umbrella conference event. selected researcher/team to present, and get feedback on, the draft research scan paper from attendees at the event. note that lirg will pay travel expenses to the event for up to two researchers involved in the research scan. this is in addition to the research scan award itself. by the end of september 2013: library and information research, lirg's journal, to consider for acceptance the final version research scan report as a paper. after acceptance, the scan award will be paid to the selected researcher/team. the material from the scan report, paper and feedback from the lirg event will be used by lirg to inform policy development. research scan applications/enquiries there is no application form. applicants are invited to submit a brief proposal outlining how the scan would be conducted and what it will cover. applicants should also submit one a4 cv per would-be researcher, containing relevant professional experience and / or academic publishing record. the proposal, appropriateness, quality and depth of academic and professional achievements will form the criteria for the scan award. applications and/or enquiries should be sent by email to: alison brettle lirg awards co-ordinator room 190 mary seacole building university of salford frederick road salford m6 6pu tel: 0161 295 0447 email: lirg.awards@gmail.com http://www.lirg.org.uk/lir/ojs/index.php/lir mailto:lirg.awards@gmail.com evidence summary   pediatric residents and interns in an italian hospital perform improved bibliographic searches when assisted by a biomedical librarian   a review of: gardois, p., calabrese, r., colombi, n., lingua, c., longo, f., villanacci, m., miniero, r., & piga, a. (2011). effectiveness of bibliographic searches performed by paediatric residents and interns assisted by librarian. a randomised controlled trial. health information and libraries journal, 28(4), 273-284. doi: 10.1111/j.1471-1842.2011.00957.x   reviewed by: mathew stone assistant librarian bradford teaching hospitals nhs foundation trust bradford, united kingdom email: mathew.stone@bthft.nhs.uk   received: 27 nov. 2012 accepted: 22 jan. 2013      2013 stone. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to establish whether the assistance of an experienced biomedical librarian delivers an improvement in the searching of bibliographic databases as performed by medical residents and interns. design – randomized controlled trial. setting – the pediatrics department of a large italian teaching hospital. subjects – 18 pediatric residents and interns. methods – 23 residents and interns from the pediatrics department of a large italian teaching hospital were invited to participate in this study, of which 18 agreed. subjects were then randomized into two groups and asked to spend between 30 and 90 minutes searching bibliographic databases for evidence to answer a real-life clinical question which was randomly allocated to them. each member of the intervention group was provided with an experienced biomedical librarian to provide assistance throughout the search session. the control group received no assistance. the outcome of the search was then measured using an assessment tool adapted for the purpose of this study from the fresno test of competence in evidence based medicine. this adapted assessment tool rated the “global success” of the search and included criteria such as appropriate question formulation, number of pico terms translated into search terms, use of boolean logic, use of subject headings, use of filters, use of limits, and the percentage of citations retrieved that matched a gold standard set of citations found in a prior search by two librarians (who were not involved in assisting the subjects) together with an expert clinician. main results – the intervention group scored a median average of 73.6 points out of a possible 100, compared with the control group which scored 50.4. the difference of 23.2 points in favour of the librarian assisted group was a statistically significant result (p value = 0.013) with a 95% confidence interval of between 4.8 and 33.2. conclusion – this study presents credible evidence that assistance provided by an experienced biomedical librarian improves the quality of the bibliographic database searches performed by residents and interns using real-life clinical scenarios.   commentary searching for evidence is a core activity in the practice of evidence based medicine and hence there is an obvious opportunity for librarians, as expert searchers, to play a major supporting role. although previous studies have already examined a range of library based interventions aimed at improving search performance for health evidence, the lack of reliable research for one-to-one librarian support for healthcare staff is the justification for this study. indeed the authors voice a more general discontent with the quality of research in this area by highlighting the lack of experimental study designs employed and the subjective manner in which outcomes have often been measured.   the intervention tested in this study is one tailored more for one-off evidence based medicine projects such as devising local guidelines or commissioning new services, rather than as an adjunct to routine patient care where clinicians will presumably struggle to find the time to search bibliographic databases (with or without the support of a librarian) between patient appointments. the use of an assessment tool (albeit one that appears to have not been validated) to measure the quality of the search process is a main strength of this study. this combined with the random allocation of subjects to intervention/control groups (which reduces the impact of confounding variables) elevates the study to a very high level of evidence. the use of an objective assessment tool minimizes the effects of observer bias, a factor which has potentially distorted studies of similar interventions where subjects have been asked to assess their own search skills. the sample size of this study was small (only 18 participants), but it was adequately powered to produce a statistically significant result due to the large difference between the outcomes in the two groups. however, while we can be confident that librarian assistance causes an improved search performance, we can be less sure about the magnitude of this improvement. the wide 95% confidence interval signifies that we can only be confident that the extent of improvement lies somewhere between 4.8 and 33.2 points. this means librarians will have to ask themselves if they can justify providing this kind of one-to-one support if the true impact of this assistance is at the lower end of this confidence interval. a larger sample size would have produced a narrower confidence interval and therefore allowed us to assess the effect size of this intervention with greater precision. the generalizability of these results to the wider population of medical residents and interns is strengthened by the high participation rate in this study. 78% of invitees were recruited in to the study and so the possibility of a non-response bias (where only confident searchers agree to participate) resulting in an unreflective sample is minimized. however, the degree to which we can generalize from interns and residents to a more general clinical population is uncertain. for instance, one might expect interns and residents who are not long out of medical school to be comparatively better searchers of bibliographic databases than their more experienced clinical colleagues and consequently less likely to benefit from the intervention. but despite the possibility of a selection bias, the effect of which would most probably be to moderate the positive result of this intervention, this randomized controlled trial provides reliable evidence that bibliographic databases searches are improved when clinicians are supported by a librarian.    librarians can take a couple of key points from this study, one quite specific and the other more general. sub-category analysis reveals that the particular component of the intervention that was most beneficial to clinicians was the help translating pico terms into search terms, so librarians would be well advised to focus their efforts in this direction. more generally, librarians now have compelling evidence showing how they can improve the evidence based medicine process by helping the clinician to search the literature more effectively. librarians can therefore push confidently (if they are not doing so already) for greater involvement in evidence based medicine projects within their organizations.     microsoft word art_hulme.doc evidence based library and information practice 2006, 1:4  17 evidence based library and information practice     article    using cost effectiveness analysis; a beginners guide       claire hulme, phd  lecturer / research fellow  salford centre for nursing, midwifery and collaborative research   university of salford  salford, manchester, uk  e‐mail c.t.hulme@salford.ac.uk      received: 23 august 2006    accepted: 26 september 2006      © 2006 hulme. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    abstract    objective ‐ this report seeks to describe the key elements of cost effectiveness analysis  (cea) and to demonstrate how such analysis may be used in the library environment.    methods ‐ the paper uses a step‐by‐step approach to walk the non‐economist reader  through the basics of conducting a cost effectiveness study. it provides an outline of the key  elements of cea using examples from the library sector, and it presents a case study of a  cea in a hospital library. the case study compares two library services, mediated searching  and information skills training, to illustrate the application of cea and to highlight some of  its limitations.    results ‐ cea is a comparative analysis tool. its key elements include a study question  regarding a particular process or procedure that identifies both costs and effectiveness; a  justification of the study’s perspective; evidence of effectiveness; comprehensive  identification of all relevant costs, and appropriate measurement of costs and effectiveness.    conclusions ‐ cea enables comparison of services or interventions regarding particular  processes or procedures in terms of their costs, and it measures their effectiveness. the  results can be used to aid decision‐making about service provision.          http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:4  18 introduction    libraries, like other businesses and  organisations, have finite budgets. decisions  must be made to determine how that budget  is allocated.  cost effectiveness analysis  (cea) can help that decision‐making  process, yet few studies that address costs or  cost effectiveness of library services have  been published.  a recent review in the  health care library sector noted that much of  the literature is descriptive rather than  empirical, providing little hard evidence of  the effectiveness or cost effectiveness of  library services (brettle, “costs”).     this paper walks the non‐economist  through the essentials of conducting a cea.   it begins by outlining the key elements of a  cost effectiveness study. it then illustrates  how this type of analysis may be carried out  in the library sector by way of a case study  situated in a hospital library. the case study  compares two library services, mediated  searching and information skills training, to  illustrate the application of cea and to  highlight some of its limitations.      cost effectiveness analysis (cea)    cea is a comparative analysis of the costs  and effectiveness of alternative  interventions or services. in the library  environment this could be in areas as  diverse as competing methods of providing  training or the use of different databases for  information retrieval. the basic cea uses  standard techniques and methods  associated with evaluating the effectiveness  of any service, but it also adds the costs  associated with the provision of these  services into the mix.     hypothesis    the first step in carrying out a cea, as in  any study, is the formulation of a clear (and  ‘do‐able’) hypothesis or study question. the  objective of cea is to compare two or more  interventions or services in terms of the  effectiveness of each and its associated costs.  the hypothesis, therefore, should include  both cost and effectiveness elements. a  further consideration at this point is to  decide on the perspective for conducting the  study. is the library primarily interested in  determining the costs of a particular  program in order to provide competing  services (or even a ‘do nothing alternative’),  or is the library looking at wider costs? it is  wise to be clear about the parameters of the  study, so that results will address the  specific issues important in making program  and resource decisions.     equally important is the decision about  which services will be compared. does the  proposal include all appropriate methods of  providing the service in question? the cea  analysis should enable the library manager  to determine the most cost effective method  of achieving a specified goal; all competing  alternatives need to be included.  for  example, in a comparison of the  effectiveness of databases in retrieving  information, it is important to include all the  appropriate databases.  once the study  question has been formulated, the  perspective determined, and the alternative  services selected, it is time to develop the  study design.     effectiveness    at this stage it is important to decide  whether you need to collect data on the  extent to which the service achieves what it  sets out to achieve (the effectiveness) as part  of your study. think about whether there is  already robust evidence of the effectiveness  of the services being compared. for example,  if previous studies have shown that one  data retrieval database consistently  produces better results than another, is it  necessary to again measure how effective  each is again? you could just cite those  evidence based library and information practice 2006, 1:4  19 results and carry out only a cost comparison  of the competing alternatives. if you do  consider using existing evidence from  previous studies be sure that the results are  transferable to the services you are  comparing. for example, are the  demographics of the library customers the  same?       if you are collecting effectiveness data, what  type of evaluation will you conduct? will it  be, for example, a randomised controlled  trial (rct) or a case study? will you have a  control group, or will this be a ‘before and  after’ comparison study in which the sample  acts as its own control over time? in  quantitative studies where the rct is  typically held as the gold standard (gold),  the standard hierarchy of research design  applies.     having decided on the study design, the  next step is to outline the specific outcome(s)  of the service and what the service tries to  achieve.  is there one clear goal that is easily  quantifiable, or are there multiple outcomes?  how can you capture those outcomes in a  quantifiable form? there are a number of  factors that will help answer these questions.  the first is whether you use an endpoint or  an intermediate point; this is best explained  using an example. consider information  skills training. the endpoint of the training  is likely to be the application of relevant and  up to date information. an intermediate  point could be the ability of those trained to  retrieve relevant and up‐to‐date information  or an improvement in their information  retrieval skills. if an intermediate point is  used, the analysis should show that the  intermediate endpoint has relevance in its  own right, or that there is a link between the  intermediate outcome and the final outcome  that has been established by previous  research (drummond et al.). the  intermediate outcome should accurately  reflect the long‐term benefits of the  intervention.   after determining the outcome(s), the next  step is to decide how you will quantitatively  measure the extent to which the competing  services achieve their goal(s). some  outcomes may be readily quantifiable, but  often data is less easily expressed  numerically. in the health sector validated  measures are often readily available, such as  measures that quantify changes in health  related quality of life or measures of anxiety  and depression. however, these are less  prevalent for the type of cea likely to be  carried out in the library sector, and reviews  have shown a lack of validated measures to  quantify improvements in information  retrieval skills (brettle, “evaluating;”  koufogiannakis). this often means a heavy  reliance on the use of a proxy for the  outcomes being measured, and you should  consider whether this is appropriate. for a  study of information skills training, proxies  might include self‐reported changes in  students’ information skills (often using a  likert scale) and satisfaction questionnaires.  using more than one outcome measure may  confuse your research results.    a further factor to consider is when to take  those measures. consider again the example  of information skills training.  if you are  using students’ self‐reported improvement  in their skills, will you ask them to report  these immediately after the training session,  after a delay of three months to allow them  an opportunity to practice their skills, or  both?  has their perceived improvement in  their information skills training persisted  over time?     once you have determined the form your  study will take (e.g., an rct) and how the  outcome(s) of your study will be measured,  the next step is to identify your study  population, and to select your sample in line  with normal research guidelines. it is  important to remember that the sample  needs to be representative of your  population. you must also determine an  evidence based library and information practice 2006, 1:4  20 appropriate size for your sample. there are  a number of standard research methods  texts that can guide you (such as bowling).     costs    the next component of the cea that  requires your attention is determining the  costs associated with the provision of the  services included in your study. it is crucial  that all important and relevant costs be  identified (drummond et al.). there are  three stages in the cost analysis:  identification, measurement and valuation  (raftery).     the next step is to identify all the costs  associated with the provision of the services  in your analysis. costs arising from, or as a  consequence of, library services will fall to  the library as service provider, but may also  fall to the library’s larger organisation and  to the library customer. the cost to the  library is likely to include the librarians’  time and administrative costs. the costs to  the library customer may include out‐of‐ pocket expenses such as printing search  results or photocopying articles.     having previously set out parameters of the  study in terms of the perspective you are  taking, the costs to be included in the study  will depend in part on the viewpoint or  perspective of the evaluation. for example,  if the evaluation is from the viewpoint of the  library, the cost of out‐of‐pocket  expenditures to the customer need not be  included. even though it may not be  possible or necessary to measure and value  all of the costs and consequences of the  services under comparison, a full  identification of all important and relevant  costs should be provided.    costs to be considered include:     • staff time associated with the provision of  the service. this is likely to include, for  example, preparation time, but it may also  include the cost to the organisation of staff  to cover those attending and providing  training sessions.   • cost of consumables, including provision  of materials, hire of a training location or  advertising.  • overhead costs, such as administrative  support, computers, and database  packages.  • capital costs that represent investments at  a single point in time (drummond et al.),  such as the cost of a new computer  training classroom.  • out‐of‐pocket expenses, such as travel  costs or child care costs, to enable students  to attend a training course.    after identifying all important and relevant  costs, the inclusion and exclusion criteria for  the cea will lie in the bounds of the study  perspective. costs may be excluded from an  evaluation because they are common to both  services or are relatively small. these should  be identified, and their exclusion justified.  similarly, if inclusion of particular costs will  only confirm the results obtained without  them, then they may be omitted. once again,  reference should be made to them in the  study, including the reason for their  exclusion.    measurement and valuation    once you have identified all costs and  decided on your inclusion and exclusion  criteria, you must decide how you will  measure and value them. two strategies,  representing each end of the spectrum, can  be useful in measuring and valuation:  micro‐costing and gross‐costing (raftery), or  a combination of both may be used.   micro‐costing is a ‘bottom up’ method of  costing that involves taking a detailed  inventory of all the separate costs of items  evidence based library and information practice 2006, 1:4  21 involved. this may take the form of, for  example, reporting on and quantifying all  activities associated with a particular  activity. thus it may include the time spent  by each individual involved in the activity,  either directly or indirectly. this type of  analysis tends to be costly and runs the risk  of being specific to particular contexts  (raftery). although more laborious than the  gross‐costing method, micro‐costing  provides a more specific insight into the  relationships between characteristics of  activities and their costs, the economies of  scales of a production process, and the  relative importance of separate activities  (drummond and mcguire).     gross or top‐down costing allocates a total  budget to specific services (raftery). thus  the library may have a budget for training  that is apportioned to specific training  activities. this method provides less precise  estimates (drummond et al.), but it has the  advantages of consuming fewer resources  and providing better opportunity for  generalisation (drummond and mcguire).     data on quantities (e.g., training time or  computer records) can, in some situations,  be directly observed or recorded. when data  cannot be obtained directly, or if there is no  data or only imprecise data, then you may  be required to make an estimate or informed  guess. you should clearly outline the basis  for any estimates.    having identified and measured the  resources associated with a service, the next  step is to assign a price to those resources‐‐ to value them. for gross costing, the  valuation for the service provider is inherent  (from the budget). in micro‐costing,  financial records such as pay scales can be  used to value, for example, the time of those  involved or the cost to the library of a  database. micro‐costing can produce very  context‐specific data.     one area to consider when looking at how  to value resources is that of the opportunity  cost. the best way to describe opportunity  cost is through an example: a library staff  member is training users. using his or her  time this way means the library is forfeiting  the performance of alternative tasks that  might have been performed instead. in  addition, the library will lose any benefits  those alternative tasks might have brought.  that is, the use of resources in one way prevents  their use in others (palmer 1551). thus,  opportunity costs are the benefits lost because  the next best alternative was not selected (gold  403). valuing opportunity cost is difficult,  however, and unit costs are used more  frequently (palmer).     whatever the sources of the values you  place on items included in your study, you  should be clear and explicit about that  valuation. although it sounds obvious,  don’t forget to include the date (year) to  which the valuation refers, as prices change  over time.    the time period over which you track costs  is also important. for example, when  comparing two training services that have a  common initial outcome but one requires  students to have further training in the  future, the time period should reflect the  costs of this additional training. if training  costs are spread over a number of years, the  costs should be discounted. discounting is  the process of converting a future sum of  money to its present value.     in summary, the costs included in the study  will depend on the study’s perspective. the  exclusion of any costs from the analysis  must be justified. detail quantities and  prices separately, and give costs in both  their discounted and undiscounted forms  when applicable.   evidence based library and information practice 2006, 1:4  22 analysis and presentation of results    after collecting the data, it is time to analyse  the results. if you have used estimates for  components of the costs or outcomes, you  may wish to take into account how sensitive  the study results are to these estimates. a  sensitivity analysis can determine this.  briggs and gray outline a number of  methods by which the sensitivity analysis  may be carried out. the most common  methods are one‐way analysis and multi‐ way analysis. an alternative, although far  less robust way to take account of estimates,  is to present confidence intervals. (a  confidence interval specifies a range of  values that, with a known degree of  certainty, includes the unknown population  mean. it is calculated from the sample data.)     how should you present your findings?  your results are likely to tell you the total  costs of the service provided and may be  examined in detail to determine, for  example, average cost per training session.  similarly, depending on the study design,  you might compare outcomes of the two  services, or changes in outcomes pre‐ and  post‐training. if there are multiple outcome  measures, none of which you consider to be  the primary outcome, this would determine  how the results are presented. if the results  show one of the services to be less costly  and more effective in all outcomes, this can  be said to dominate and is therefore the  most cost effective. if each of the services  included has varying degrees of  effectiveness among multiple outcomes and  none is dominant, then the reader is left to  make his or her own judgement regarding  the most important outcome. ceas with no  single primary outcome and multiple  outcomes measures are known as cost  consequence analyses (cca).      if there is one primary outcome, you might  present your results as an incremental cost  effectiveness ratio. the incremental cost  effectiveness ratio is the difference in costs  between alternatives to the difference in  effectiveness between the same alternatives.  stated another way, the incremental cost  effectiveness ratio compares the additional  costs that one service or programme  imposes over another, with the additional  effects, benefits or utilities it delivers. by  using incremental cost effectiveness ratios it  is possible to determine, for example, how a  budget could be spent between two or more  services.    this outline of the key elements of a cost  effectiveness analysis is summarised in   box 1.    • the hypothesis or study question should include both costs and effectiveness.  • the perspective or viewpoint of the study needs to be justified; the nature of the outcomes  resulting from the intervention should all be clear.   • the study design needs to be developed.  • there should be evidence of the effectiveness of the services being compared; all outcomes  should be identified.  • the measurement of effectiveness should reflect the outcomes; measures should be  appropriate to the form of the study, relevant, and validated where possible.  • all costs should be identified and in line with the perspective of the study; inclusion and  exclusion criteria should be identified.  • the measurement of costs included in the study may use micro‐costing (considered most  accurate) or gross‐costing (provides greater opportunity for generalising) methods ‐‐ or a  combination of both. the method should be clearly outlined.  evidence based library and information practice 2006, 1:4  23 • costs and outcomes should be tracked over an appropriate period of time and provide an  accurate reflection of actual practices.  • where costs are spread over a number of years, they should be discounted.   • sensitivity analysis should be used to take account of uncertainty in any estimates, be they  costs or consequences.     box 1. key elements of a cost effectiveness analysis      this case study illustrates how cea  methods may be used within health care  libraries and some of the difficulties that  may be faced.    case study    the case study used to illustrate cea within  the library sector is part of a larger study  reported in full elsewhere (brettle, “costs;”  brettle, “effectiveness”). the study was  carried out in hospital based health care  libraries across the north west of england  in the spring of 2005. the libraries’ customer  base is health care workers. the purpose of  the larger study was to explore the  effectiveness and cost effectiveness of  different methods of providing information  for patient care. two strategies were  employed by the libraries to achieve this  aim. the first involved library staff  providing information and mediated  searches directly to clinicians on request; the  second strategy involved librarians training  clinicians to search for information  themselves. a baseline survey of the 102  libraries in the region conducted in 2004  showed libraries typically provided both  services (brettle, “baseline”).    the cea was incorporated into the larger,  questionnaire‐based study of the 102  libraries in the region, 26 agreed to  participate in the study. in each site  questionnaires exploring both mediated  searching practices and information skills  training were distributed to all library staff       (115), dedicated trainers (15), and to a  random sample of 30 clinicians (780). all   cea data were collected from these  questionnaires and showed the 26 library  sites were representative of the population  in terms of the numbers of staff, seats, and  computer terminals when compared with  the baseline survey data.        the cea study question compared two  library services: which is more cost effective –  information skills training or librarian mediated  searching? the uk north west health care  libraries unit (hclu), a strategic regional  unit interested in the effectiveness and cost  implications of providing both services,  commissioned the overall study. the study  was carried out from the perspective of the  library as service provider.    effectiveness    in respect of the effectiveness of the two  services, the endpoint could be interpreted  as provision of up‐to‐date and relevant  information for patient care to clinicians,  either directly (mediated searches) or  indirectly, by enabling them to access that  information themselves. however, it could  be argued that provision of information is  not an endpoint in itself, and that the most  important outcome is the use or influence of  that information in patient care. because the  larger study was questionnaire‐based, the  only feasible method by which to measure  the effect of information derived from  searches on patient care was to ask the  clinicians’ their perceptions. in addition,  evidence based library and information practice 2006, 1:4  24 clinicians were asked how satisfied they  were with the results of the searches. it was  anticipated that this would be a proxy for  search quality. thus the measures included:    • health professionals’ perception of the  effect of information from librarian  mediated searches on patient care.   • health professionals’ perception of the  effect of information from their own  searches on patient care.   • health professionals’ satisfaction with  librarian mediated searches.   • health professionals’ satisfaction with  their own searches.     all used a five‐point likert scale (from  highly positive to highly negative in the first  two measures and from very satisfied to  very dissatisfied in the latter two).    costs    the cost analysis used a micro‐costing or  ‘bottom up’ method. the questionnaire  asked respondents to provide details of  mediated searches and information skills  training. resource use was identified and  measured using the completed  questionnaires from librarians and health  professionals. questionnaires contained  demographic details including professions  and grades (from national grade scales) and  asked respondents for details of mediated  searches requested or performed and about  training they had undertaken or attended.     inclusion and exclusion criteria    it was assumed that resources such as  information technology facilities,  administration, and other overhead costs  were common to all settings, and they were  excluded from the analysis. by this we mean  these resources would need to be available if  only mediated searches were conducted (no  information skills training); or if only  information skills training was carried out  (no mediated searches); or if both mediated  searches and information skills training  were carried out. it was also assumed that  existing library facilities and resources were  used, and that no additional costs would be  incurred for a training room or computer  equipment, for example. the cost of training  materials such as handouts was believed to  be relatively small and was excluded from  the analysis.     the analysis assumed staff time was the  primary resource, and as such, the cost  element of the analysis was based solely on  this resource. staff time identified from the  questionnaires included these items:    • time spent by the librarian conducting  mediated searches.  • time spent by health professionals  sitting with the librarian during  performance of mediated searches.  • time spent by health professionals  performing literature searches.  • time spent by the librarian delivering  information skills training and in  training‐related activities.  • time spent by health professionals in  information skills training.    all staff time was valued using 2005 prices  and was based on salary cost only; the  analysis excluded ongoing costs (e.g.,  national insurance payments) and overhead  costs. library staff time values were derived  from their library grades as reported on the  questionnaires. national pay scales were  used to derive per hour estimates, assuming  a 46‐week work year and a 36‐ hour work  week. similarly, illustrative salaries were  estimated by profession for clinicians using  data from the unit cost of health and social  care 2004 (curtis).  each was inflated in  order to incorporate increases in national  salary scales that were already reflected in  the library staff salary scales.      evidence based library and information practice 2006, 1:4  25   mediated  searches  number (%)  clinicians’ own  searches ‐  post training  number (%)  information from search has a positive /  highly positive effect on patient care*    83/132 (62.88%)  66/103 (64.08%)  very satisfied / satisfied with search results*  116/132 (87.88%)  64/103 (62.14%)      table 1. effectiveness of searching    *for ease of reporting, the results have been presented categorically.      training method  mean time, including preparation and  related activities  average cost  one‐on‐one pre‐arranged  training    1.49 hours    £19.92  one‐on‐one ad hoc training  1.25 hours  £13.47    group training   (<20 trainees per session)    2.55 hours  £32.33    table 2. cost of information skills training to the library      results    each of the outcome measures and costs was  presented separately. in brief, the study      found that 54% (132/243) of the clinicians  had requested mediated searches, and 62%  (44/71) of the library staff carried out  mediated searches. with regard to training,  42% (103/243) of the clinicians had received  information skills training from their  hospital library, and 53% (38/71) of the  library staff had delivered training. table 1  shows little difference was seen in the  perceived effect on patient care resulting  from mediated search results and their own  search results. however, the proportion of  clinicians who were very satisfied or  satisfied with the results from mediated  searches (87.88%) was higher than that  expressed for clinicians’ own searches       (62.14%). librarians from different position  grades conducted the mediated searches.  the median grade of those carrying out  searches was used to calculate the per hour  cost of these searches. this figure was then  multiplied by the mean time of a search  (1.08 hours) to determine the mean cost of a  mediated search to the library‐‐£14.44.      the cost of training was calculated using the  same method. however, the analysis took  into account the different training formats to  reflect differences in the average delivery  time of each and differences in the grades of  those delivering the training (and therefore  differences in salary). the data showed that  training delivered to groups was on average  longer and was delivered by librarians of a  higher grade than other formats of training   evidence based library and information practice 2006, 1:4  26   profession  cost per hour  total cost per  search  staff nurse  £11.84  £15.15  ward manager  £15.61  £19.97  physiotherapist, occupational therapist,  radiographer (professions allied to medicine)  £16.20  £20.73  consultant (physician)  £44.46  £56.89    table 3. cost of clinicians’ searches     (table 2). while group training is the most  costly option, the cost falls when considered  on a ‘per person trained’ basis. for example,  if 10 people attend a training session, the  average cost would be £3.23 per person.    the mean cost of a search carried out by a  clinician (once trained) was calculated using  the same methods. the analysis used  professions representative of the sample (i.e.,  staff nurse, ward manager, therapist, and  consultant) and their associated salary scales.  the average time spent on a search (1.28  hours) was used to calculate the mean cost  for each profession (table 3).  it should be  noted that although these costs do not fall to  the library, they are helpful for illustrative  purposes.    discussion    although there was little difference between  the perceived effect on patient care from  either mediated searches or the clinicians’  own searches, there was clearly a higher  level of satisfaction with mediated search  results. in addition, mediated searches were  found to be less costly than any of the forms  of training. this seems to clearly indicate  that mediated searching is most cost  effective for the library, and that funding  should be shifted from training to mediated  searches. however, while budget allocation  decisions can be informed by the analysis,  this conclusion should be treated with  caution, as there are a number of other  factors to be taken into account. these are  explained below.    one of the first decisions to make when  carrying out a cea is to determine the  perspective from which to conduct the  study. this case study reported from the  libraries’ perspective, but what happens to  our conclusions if the wider perspective of  the health care provider (the hospital, in this  case) is considered? the raw data showed  that more than one in three mediated  searches was carried out with the clinician  present. adding their time to the search  would increase the mean cost of the search  to £38.23. similarly, adding the time spent  by clinicians attending training sessions  increases the mean costs of each type of  training. the data showed that while ad hoc  one‐on‐one training was the least costly  method of training, clinicians were likely to  have attended more than one of these  sessions (mean number of sessions 2.67),  thus increasing their cost to the library when  compared with other formats. an important  aspect of whether to include clinicians’ time  to train or carry out mediated searches is  whether their absence from their workplace  for this activity requires their tasks to be  covered by additional members of staff? if  not, is there any additional cost to the  hospital?    the results were further confounded by the  insight that the majority of clinicians who  had been trained in searching also requested   evidence based library and information practice 2006, 1:4  27 mediated searches. this poses the question  of whether training clinicians to do their  own searches leads to a decrease in the  number of mediated searches requested? or  does this increase the number of searches  (whether by library staff or clinicians)  carried out overall? if there is an increased  number of searches, does this lead to an  improvement in patient care?  a related  issue was the small proportion of clinicians  with limited or no training who performed  their own searches. this calls into question  the cost effectiveness of providing skills that  are not utilised.    despite there being little difference in the  mean times required for either library staff  or clinicians to perform searches, this was in  part explained by the maximum time each  group could or would allocate to a search.  this may also help to explain the difference  between groups in terms of their satisfaction  with search results. the larger study gave  insight into many of the confounding factors  discussed above, particularly the qualitative  responses which provided explanations for,  for example, the time allocated to searches.    a final area of importance to note is that this  cea was based on retrospective, self‐ reported data, and it relied on respondents’  recollections that should be considered as  estimates. further analysis is required to  evaluate the sensitivity of the study results  to those estimates.  conclusion      in common with other organisations and  sectors, libraries have competing demands  placed on their budgets. cea provides a  tool by which comparative evidence on the  cost and effectiveness of services can be  produced in order to inform budget  allocations and to determine which services  to provide. however, it should be noted that  decision‐making is complex. while the cost  and effectiveness of services are important,  there are likely to be factors outside the  analysis that also affect decisions. these  may include, for example, anticipated  advances in technology, planned re‐ configuration of services, or legislative  requirements. none should be viewed in  isolation; all should inform budget  allocation decisions.     this paper presents the basics of cea.  an  explanation and description of the key  elements of the analysis and examples  utilising familiar situations introduce the  reader to ideas for using cea in the  workplace. while the paper provides an  introduction to cea, there are a number of  texts that go into greater theoretical and  technical detail on conducting and critically  appraising ceas. box 2 presents several  useful cea resources.         drummond, michael f, and tom o jefferson. “guidelines for authors and peer reviewers of  economic submissions to the bmj.” bmj. 313.7052 (1996): 275‐83.    gold, marthe r., joanna e. siegel, louise b. russell, and milton c. weinstein. cost effectiveness  in health and medicine. oxford: oxford university press, 1996.    hulme, claire. “a practitioner’s guide to appraising cost effectiveness papers.” nurse  researcher. 8.4 (2001): 76‐86.      evidence based library and information practice 2006, 1:4  28 kobelt, gisela.  health economics: an introduction to economic evaluation. london: office of  health economics, 2002.     university of salford school of nursing, health care practice r&d unit. “evaluation tool for  critical assessment of an economic evaluation of a health intervention.” 25 aug. 2005.  30 nov.  2006 <http://www.fhsc.salford.ac.uk/hcprdu/economic.htm>.     university of york. centre for reviews and dissemination. review methods and resources.  “what is an economic evaluation?” 28 nov. 2006. 30 nov. 2006.  <http://www.york.ac.uk/inst/crd/crdreview.htm>.      box 2. sources of information on cea      works cited    bowling, ann. research methods in health.   investigating health and health  services. buckingham; briston, pa:  open university press, 1997.     brettle, alison, claire hulme, and paula  ormandy. effective methods of  providing information for patient care:  empiric project – baseline data  collection survey.  salford, university  of salford, 2004.  contact  a.brettle@salford.ac.uk for copies.    brettle, alison. “evaluating the  effectiveness of information skills  training: what the literature says.”   health libraries group conference,  eastbourne, uk. 12 july 2006.      brettle, alison, claire hulme, and paula  ormandy.  the costs and effectiveness  of information‐skills training and  mediated searching: quantitative  results from the empiric project.  health information and libraries  journal 23.4 (2006): 239‐47.    brettle alison, claire hulme, and paula  ormandy.  “effectiveness of  information skills training and       mediated searching: qualitative  results from the empiric project.”  health information and libraries  journal, 24 (2007): in press.    briggs, andrew h., and alistair m. gray.  “handling uncertainty when  performing economic evaluations of  health care interventions.” health  technology assessment. 3.2 (1999): 1‐ 134.    curtis, lesley, and ann netten, comps. unit  costs of health and social care 2004.   kent: personal social services research  unit, university of kent, 2004. 4 dec.  2006 <http://www.pssru.ac.uk>.    drummond, michael, and alistair mcguire.  economic evaluation in health care ‐  merging theory with practice. oxford:  oxford university press, 2001.    drummond, michael f, bernard j. o’brien,  greg l. stoddart, and george w.  torrance. methods for the economic  evaluation of health care  programmes. 2d ed. oxford: oxford  university press, 1997.    gold, marthe, joanna e. siegel, louise b.  russell, and milton c. weinstein. cost  http://www.fhsc.salford.ac.uk/hcprdu/economic.htm http://www.york.ac.uk/inst/crd/crdreview.htm http://www.pssru.ac.uk evidence based library and information practice 2006, 1:4  29 effectiveness in health and medicine.  oxford: oxford university press, 1996.    kobelt, gisela.  health economics: an  introduction to economic evaluation.  2d ed. london:  office of health  economics, 2002.     koufogiannakis, denise, and natashe weibe.   “effective methods for teaching  information literacy skills to  undergraduate students: a systematic  review and meta‐analysis.”  evidence  based library and information practice,  1.3 (2006): 3‐43. 4 dec. 2006  <http://ejournals.library.ualberta.ca/ind ex.php/eblip>.                                                              palmer, stephen, and james raftery.  “economic notes: opportunity cost.”  bmj. 318.7197 (1999) 1551‐2.    raftery, james. “costing in economic  evaluation.” bmj. 320.7249 (2000): 1597.                      http://ejournals.library.ualberta.ca/ind evidence summary   information horizons mapping is related to other measures of health literacy but not information literacy   a review of: zimmerman, m.s. (2020). mapping literacies: comparing information horizons mapping to measures of information and health literacy. journal of documentation, 76(2), 531–551. https://doi.org/10.1108/jd-05-2019-0090   reviewed by: eugenia opuda health & human services librarian assistant professor dimond library university of new hampshire durham, new hampshire, united states of america email: eugenia.opuda@unh.edu   received: 31 may 2020                                                             accepted:  30 sept. 2020      2020 opuda. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29787     abstract   objective – to evaluate information horizons mapping as a valid measure for assessing information literacy and health literacy compared to three validated information and health literacy measurements and level of educational attainment.   design – quantitative data analysis using multiple regression and the anker, reinhart, and feeley model as the conceptual framework.   setting – a small university-centered community in iowa city.   subjects – 149 members of the university community.   methods – the author conducted a power analysis to determine a minimum sample size required for maintaining study validity and selected the anker model of conceptual framing for health information-seeking behavior. this is a three-phased model that explores the information seeker’s predisposing characteristics, engagement in health information seeking, and outcomes associated with information seeking. recruited participants completed three assessments—the tool for real-time assessment of information literacy skills (trails), the health literacy skills instrument (hlsi), and the brief health literacy screen (bhls)—and drew information horizon maps illustrating what sources of information they tend to seek for health-related questions. the author calculated information horizon map results using a scoring system incorporating the number and quality of information sources identified in the maps and applied multiple linear regression analysis and spearman’s rank correlation coefficient to participants’ scores from all four assessments as well as their level of educational attainment to determine strengths of relationships between variables.   main results – in the information horizons map results, participants identified an average of 6.9 information sources with a range of 3–13 and received an average score of 18.8 in information source quality with a range of 4–45. the author applied multiple linear regression to predict the number of information source counts on the information horizons map based on hlsi, trails, and bhls assessment scores and level of educational attainment and found a significant relationship (p=0.044). a significant relationship also existed between quality of source scores on the map based on hlsi, trails, and bhls assessment scores and level of educational attainment (p=0.033). removing the educational attainment variable produced an even stronger significant result. spearman’s rank correlation coefficient supported the findings of the multiple regression analysis and revealed a strong relationship between source count and scores on the bhls (r=0.87) and hlsi (r=71) but a weak relationship between source counts and trails score and level of educational attainment. source quality had a weak relationship with bhls scores (r=0.24), a moderate relationship with the hlsi scores (r=0.50), and a weak relationship with trails scores and educational attainment.   conclusions – the data analysis suggests a significant relationship between information horizons mapping and health literacy but not information literacy or level of educational attainment. this data supports findings from the author’s previous research examining the relationship between information horizon maps and information literacy scores for refugee and immigrant women. it also suggests that information horizons mapping may facilitate storytelling that reflects the complexity of participants’ health literacy ability and may introduce the potential to assess low-literacy level populations. more research is needed to examine the quality and complexity produced in information horizons maps. this methodology may be applied to investigate better techniques for assessing the health literacy levels among populations that struggle with prose-based assessments.   commentary   this research builds upon the author’s previous work using information horizons mapping to assess health and information literacy among low-income refugee and immigrant women (zimmerman, 2018) and examines the efficacy of this tool, in comparison with other validated literacy tools, to measure literacies among a general population with the long-term goal of applying the tool to low-literacy-level populations. the information horizons methodology has been taught to library science students as a tool to measure information behavior among various populations (hartel, oh, & anh, 2018) and used by researchers to measure information literacy (eckerdal, 2013), but this article is one of the first to measure health literacy.   this study was appraised using the ebl critical appraisal checklist (glynn, 2006). the study methodology is appropriate to address the research questions, is strengthened by a conceptual framework of linear information-seeking behavior, and is detailed enough to facilitate study replication. analysis of the results was a strength of the study. the author  clearly articulates the process of creating the information horizons map scoring system based on currently existing examples in the literature; accounts for positive scoring of information horizon mapping by taking only positive scores from the hlsi, trails, and bhls assessments; and corroborates the results of multiple linear regression analysis by utilizing spearman’s rank correlation coefficient to determine the strength of correlations between variables. the author’s sample information horizon maps helped to visually articulate the methodology but could have included the source counts and quality scores for each of the example maps to convey information more effectively.   though the author conducted an analysis to determine a minimum sample size for study validity and exceeded the minimum population sample, the demographic, which mostly consisted of white, college-educated adults, lacks diversity and is not representative of a general population. the author is clear about this limitation and describes the study population as coming from an “unusually educated” university-centered community and with a significant lack of racial diversity (p. 548).   while the findings of this study are inconclusive in determining if they can apply to a broader population, the study serves as a stepping stone toward thinking critically about the development and utilization of heath literacy assessments, especially for low-literacy and underrepresented populations who experience increased health disparities (national academies of sciences, engineering, and medicine, 2017) and who may struggle with conventionally prose-based assessments. these study findings may help librarians personalize research consultations with medical students or be used for patient education training, identifying strengths and areas for growth in health literacy and information-seeking behavior. the information horizons map shows promising results to achieve this goal as well as the ability to detail the complexities of health information-seeking behavior in a strength-based rather than a deficit-focused approach.   references   eckerdal, j.r. (2013). empowering interviews: narrative interviews in the study of information literacy in everyday life settings. information research: an international electronic journal, 18(3). retrieved from http://informationr.net/ir/18-3/colis/paperc10.html#.xtppden7mt8   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387–399. https://doi.org/10.1108/07378830610692154   hartel, j., oh, c., & nguyen, a. t. (2018). teaching information behavior with the information horizon interview. journal of education for library and information science, 59(3), 67–79. https://doi.org/10.3138/jelis.59.3.2018-0017.07   national academies of sciences, engineering, and medicine; health and medicine division; board on population health and public health practice; committee on community-based solutions to promote health equity in the united states. (2017). the state of health disparities in the united states. in a. baciu, y. negussie, a. geller, & j. n. weinstein (eds.), communities in action: pathways to health equity. national academies press. retrieved from https://www.ncbi.nlm.nih.gov/books/nbk425844/   zimmerman, m. s. (2018). information horizons mapping to assess the health literacy of refugee and immigrant women in the usa. information research, 23(4), 1–21. retrieved from http://informationr.net/ir/23-4/paper802.html     microsoft word comm evidence based library and information practice 2008, 3:3 86 evidence based library and information practice commentary friendly skepticism about evidence based library and information practice marcus a. banks manager of education and information services uc san francisco library san francisco, california, united states of america e-mail: marcus.banks@library.ucsf.edu received: 01 sept 2008 accepted: 02 sept 2008 © 2008 banks. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. introduction on july 12, 2008 i blogged up a “screed” about evidence based librarianship, and realised almost immediately that i should have more accurately discussed “evidence based library and information practice,” or eblip (banks). by july 13 two leaders in the eblip community, pam ryan and denise koufogiannakis, had responded eloquently to this post. the next day former jmla editor t. scott plutchak pointed me to the video of a “friendly debate” he participated in with andrew booth about eblip last year (booth and plutchak). finally, on july 15 sarah mccord contextualised the conversation by pointing to a recent jama paper about varieties of valid evidence (berwick), to which ryan responded by noting examples of discussion in the eblip community about similar issues (given; ryan). i offer this excessively detailed chronology as anecdotal evidence for the power of blogs to rapidly generate thoughtful discussion. while the reaction to many blog posts can dwindle down to flame wars and insults, to me this experience demonstrates the potential of blogs to be a positive professional communication vehicle. i’m willing to stake a good deal on this slight evidence, even though it is undeniably flimsy by the standards of eblip. i have similar fuzzy feelings for wikis; for example, i find the “evidence based librarianship” wiki managed by dean giustini to be an excellent source of key information about this topic (giustini). my personal experience with this wiki is my only evidence for such a sweeping endorsement. booth, one of the chief proponents of eblip, recently lamented a general “evaluation bypass” as librarians rush to try out “web 2.0” tools like blogs and wikis (booth). evidence based library and information practice 2008, 3:3 87 surely my soft-minded embrace of blogs and wikis would not pass muster with a data-driven man like booth. and yet, i believe that a sense of infectious experimentation should also be part of our professional toolkit. i suspect that booth would not disagree with me if i phrase the matter this way, and would protest that he only wants to see “evaluation alongside innovation.” and in turn how could i object to something so equally obvious and compelling? so ultimately this disagreement turns not on ends but on emphases. i would be the first to cheer for a well-designed, randomised, replicable trial with the potential to improve a tangible aspect of information practice. but for me such studies do not have a monopoly on explanatory power. depending on the context, the so-called evidence pyramid can be a political statement as much as a statement of fact, and for many types of questions, the randomised controlled trial is neither possible nor desirable (sladen library; berwick). eblip defenders might object that this is a caricature that frames the matter too sharply. but i believe it is a fair response to the rhetorical thrust of the eblip movement. contributions of eblip before dwelling upon points of disagreement, it is important to recognise the contributions of eblip. the current definition of evidence based library and information practice is, “an approach to information practice that promotes the collection, interpretation, and integration of valid, important and applicable user-reported, practitionerobserved and research-derived evidence” (giustini). it is indeed critically important for librarians to seriously examine why we do what we do, because otherwise the crutch that “we’ve always done it that way” can become seductive. this definition promotes but does not command, and nicely balances the perspectives of users, librarians, and researchers. on a personal note, i sincerely appreciate the willingness of eblip’s proponents to engage with people of contrary viewpoints. in addition to her response on my blog, koufogiannakis immediately invited me to contribute this commentary. the debate between booth and plutchak is another example of an open exchange. however much we may disagree, we should all remember that this truly is a dispute between friends. friendly disagreements so, where do i disagree with my friends who promote eblip? i find three principal points of contention, each of which i will address in turn: o eblip is not sufficiently responsive to the reality that local contexts can trump even the most rigorous evidence. o eblip cannot answer many types of questions, particularly value-laden questions. o eblip can stifle innovation, in effect if not in design. not responsive to local contexts: in my blog screed, i hypothesise about an academic library that decides to abandon its in-house document scanning service for faculty members. the evidence is unequivocal that this service, however convenient it might be, is no longer cost-effective for the library. in some libraries, this evidence would be persuasive, and in all libraries it should be. but in many libraries, empirically irrelevant evidence based library and information practice 2008, 3:3 88 factors would prevail. library staff may resist the cancellation of this service, seeing it as a core of what they do; or influential faculty members may lobby for it to continue, causing the director to see it as a “loss leader” that generates goodwill. these are among a host of reasons why even persuasive evidence is often insufficient. while the concept of eblip makes allowance for practical realities (such as the need to maintain goodwill), it runs the risk of losing coherence if it grants too many exceptions. less doctrinally committed proponents of the use of evidence in libraries, such as myself, can have our cake and eat it too: we see the value of empirical evidence, but do not put disproportionate weight on this one form of knowing. much of my critique echoes similar criticisms lodged against evidence based medicine (ebm), the movement in which eblip has its roots. cohen et al. categorized many of the recurrent objections to ebm: an excessive reliance on empiricism; a narrow definition of what constitutes evidence; lack of evidence for the efficacy of ebm itself; limited utility for individual patient care (which is analogous to the limited utility of eblip for local library contexts); and a threat to the autonomy of the doctor-patient relationship (cohen et al.). proponents of ebm are likely to find these objections to be spurious (strauss and mcalister). i find them to be persuasive, as long as there is no backsliding towards the older model in which physicians had absolute authority and did not need to consult anyone. the same holds for concerns about eblip; they have merit, but i am not arguing that librarians should base our efforts on no evidence whatsoever. cannot answer all questions, especially valueladen questions: in his debate with plutchak, booth’s disdain for answering “philosophical” questions is striking. he prides himself on being a “pragmatist,” whose chief concern is with answering the nitty-gritty questions associated with running a library. i find this to be worthy but narrow. while excessive theorising about the role and purpose of libraries can become tiresome, we are now living through a long period of profound (and to my mind, enriching) professional destabilization brought on by the internet. it is a critical time for deep and unstructured thinking about our future. core values of what it means to be a librarian are at stake, and many of the questions we’re grappling with will never have “right” answers. sometimes the only way to learn is to stumble through the dark until you discover where you are going. stifling of innovation: koufogiannakis has responded to the claim that eblip can unwittingly stifle innovation, by positing that there is no inherent tension between the two (koufogiannakis). she argues that the two actually go hand in hand; an innovative product or service should be grounded in whatever evidence is available, and offered in a way that facilitates ongoing evaluation and refinement. while this is an admirable middle ground, booth’s concern about an evaluation bypass with regard to the use of web 2.0 tools in libraries is a more typical eblip formulation. he is right that evaluation of these tools’ effectiveness is necessary, but for the time being it is more important to adopt a flexible and experimental mindset. here’s a slightly absurd parenting analogy to illustrate my point: is it better to let your children play and then do their homework, as long as they get the homework done? or does the homework always come first, even if it takes so long that it’s dark outside by the time they finish? i say, let the kids play. if they never finish their homework — i.e., if there are still no serious analyses of the use evidence based library and information practice 2008, 3:3 89 of web 2.0 tools in libraries after several more years — then we have a problem. but right now, it’s light outside. conclusion and détente in the discussion that ensued after writing my blog screed, i noted that librarianship is merely the “object under discussion” for what is ultimately a disagreement about the nature of knowledge and how to approach the world. my (somewhat tongue-in-cheek) surmise is that most proponents of eblip are sensing (“s”) on the myers-briggs type indicator, while i rely (sometimes overmuch) on intuition (“n”) (wikipedia). in order to make good decisions, s’s want as much data — aka “evidence” — as they can find. n’s are more comfortable making inferences from limited information. i would rather make a mistake based on imperfect information, than wait for superior information only to see an important opportunity pass by. the case for waiting and gathering more information is just as valid. this personality distinction seems hardwired, so i predict that there will be friendly jostling about the validity of eblip in our professional community for years to come. what is not in dispute is that librarianship is a deeply service-oriented, collaborative profession. in the book people come first, michael gorman states it well: “libraries are in the service business. the most important product they have is service” (montanelli and stenstrom 2). one of the core satisfactions of librarianship is helping people find that critical nugget of information they would not have found on their own; this is true for all librarians, whether they are explicitly researchoriented or not. my colleagues in the medical library association recently released the welldeveloped “research imperative” (medical library association). superb research skills are definitely imperative, but for some of us and not for all. the rhetoric of eblip strongly suggests that we should all become research experts, but only a subset of librarians will ever have the inclination to develop a sophisticated understanding of research methods. in my view it would be more fruitful to groom this cadre of research leaders. then the rest of us could learn from these pathfinders as we carry out the essential business of evaluating our work. as long as we don’t stretch it past the breaking point, eblip can be a vital philosophical asset for our profession. works cited banks, marcus. “thoughts on evidence based librarianship: a screed.” 12 july 2008. 2 sept. 2008. <http://mbanks.typepad.com/my_w eblog/2008/07/thoughts-onevidence-based-librarianship-ascreed.html>. berwick, donald m. “the science of improvement.” journal of the american medical association 299.10 (2008): 1182-84. booth, andrew. “blogs, wikis and podcasts: the ‘evaluation bypass’ in action?” health information and libraries journal 24.4 (2007): 298-302. booth, andrew, and t. scott plutchak. “eblip: clear, simple, and wrong? a friendly debate.” 09 may 2007. 2 sept. 2008 <http://www.youtube.com/watch?v =_ilumv25oxg>. cohen, aaron m., p zoe stavri, and william r. hersh. “a categorization and analysis of the evidence based library and information practice 2008, 3:3 90 criticisms of evidence based medicine.” international journal of medical informatics 73.1 (2004): 3543. giustini, dean. “evidence-based librarianship.” 11 august 2008. 2 sept. 2008 <http://hlwiki.slais.ubc.ca/index.php ?title=evidencebased_librarianship>. given, lisa. “qualitative research in evidence-based practice: a valuable partnership.” library hitech 24.3 (2006): 376-86. koufogiannakis, denise. “on evidence based practice and innovation.” evidence based library and information practice 2.4 (2007): 108-11. 2 sept. 2008 <http://ejournals.library.ualberta.ca/ index.php/eblip/article/view/787/6 65>. medical library association. “the research imperative: the research policy statement of the medical library association.” 14 sept. 2007. 2 sept. 2008 <http://www.mlanet.org/research/p olicy/policy-01_toc.html> montanelli, dale s., and patricia f. stenstrom, eds. people come first: user centered academic library service. chicago: association of college and research libraries, 1999. ryan, pam. “ebl and library assessment: two solitudes?” evidence based library and information practice. 1.4 (2006): 7780. 2 sept. 2008 <http://ejournals.library.ualberta.ca/ index.php/eblip/article/view/136/1 77>. sladen library. “guide to sladen library evidence based medicine resources.” 2 sept. 2008 <http://sladen.hfhs.org/library/staff/ ebm-resource-pyramid.htm>. straus, sharon e., and finlay a. mcalister. “evidence-based medicine: a commentary on common criticisms.” canadian medical association journal 163.7 (2000): 837-41. 2 sept. 2008 <http://www.pubmedcentral.nih.go v/articlerender.fcgi?tool=pubmed&p ubmedid=11033714>. wikipedia. “myers-briggs type indicator.” 31 aug. 2008. 2 sept. <http://en.wikipedia.org/wiki/myers -briggs_type_indicator>. evidence summary   miscommunication and employee power dynamics may affect student navigation of library resources   a review of: almeida, n., & tidal, j. (2022). library wayfinding and esol students: communication challenges and empathy-based intervention. portal: libraries and the academy, 22(2), 453–474. https://doi.org/10.1353/pla.2022.0025     reviewed by: matthew bridgeman information and education librarian robert wood johnson library of the health sciences rutgers, the state university of new jersey new brunswick, new jersey, united states of america email: mcb226@libraries.rutgers.edu   received: 5 dec. 2022                                                                accepted:  8 jan. 2023      2023 bridgeman. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30287     abstract   objective – to map the experiences of students of english for speakers of other languages (esol) navigating an academic library.   design – a wayfinding study to evaluate how students navigate a library.   setting – an urban-based academic library at an institution of higher education.   subjects – students of english for speakers of other languages (esol).   methods – a mixed methods study including visual recordings, web screen capture, interviews, and surveys. subjects were recruited through email. twelve participants were selected and given an initial screening survey. they were given four tasks to complete: find a book in the stacks, find a book in the reserves, find a dvd in media, and find a database. they were equipped with a gopro camera and were given a think-aloud protocol (tap). they were then given a post-task debriefing interview. qualitative data were analyzed and coded. quantitative data like success of task and time to completion were also recorded. main results – success rate varied among tasks: finding a book in reserves had the highest rate at 75%, while finding a database had the lowest at 50%. time also varied from 12 minutes to find a book in the stacks to just under 6 minutes to find a database. seven of the 12 participants indicated they had prior library experience; however, they still encountered skill gaps. they lacked familiarity with the space, policies, website, and terminology. participants also struggled with library jargon and inconsistent use of jargon among staff and librarians.   conclusion – the researchers discovered there were discrepancies between language used in signs, directions provided by staff, and information provided on the website. signage was important because several participants made remarks on lack a familiarity with the library space. they would get lost and anxious. in addition, the video recordings and subsequent discussions among the staff and librarians showed issues arising from the power dynamics in the library organization. staff felt pressured to provide reference services when librarians were unavailable due to staffing shortages, which led to miscommunication. these conclusions lead to empathy-based training to address language discrepancies and experiences among staff. it also provided additional rationale for hiring.   commentary   this article is a wayfinding study that addressed the role communication plays in the library. wayfinding is defined as the ability of a person to orient themself in a space and navigate it. a key, though incidental, finding was the impact of the employee power dynamics on communication to students. this negatively affected how the students were aided in performing the study’s task. the researchers could have ignored this, as it was not in the original study design but only came to light when they were analyzing the data about the paraprofessionals and librarians.   since this is a mixed methods study, it was evaluated using “the cat: a generic critical appraisal tool” by perryman and rathbun-grubb (2014). the authors of the article are academic librarians with the necessary expertise in the field. the literature review is extensive and provides a suitable background and reasoning for the study. they were transparent in their study design and its limitations, such as its small sample size. twelve participants are not enough to support generalizable actions, but it can be the seed for further study. the subjects, being esol students, were appropriate for the objective. however, based on their findings, the employees at the library could also be considered inadvertent subjects. more detail on them would have been beneficial. demographic information could also aid in understanding the interprofessional power dynamics.   the article takes an empathetic approach by trying to fully encompass the first-person experiences of an individual for whom english is not their primary language. while the intended subjects were the patrons, the revelation of the relationships among the librarians and paraprofessionals was enlightening. the authors discovered they were not familiar with their staff colleagues’ professional roles and the language staff used to communicate with students. this resulted in inconsistent communication when helping patrons. the surveyed students were already unfamiliar with library jargon, so any discrepancies in terms added to the confusion. in addition, some employees used different terms to describe the same thing. an example they offer is “call number” and “code number” for finding books. with no context, this can be very confusing and resulted in many mistakes. this also spills over into signage. regarding terminology in signs, consistency is key (mandel & johnston, 2019). this begs the question: how do we get consistency in language to improve wayfinding? it affects not only esol students but also english proficient students.   the proposed solution was to review the power dynamics among the employees at the library. tensions among librarians and paraprofessionals stifled communication. understanding and resolving these tensions can solve several issues and improve communication and interprofessional understanding (hill, 2014). according to anasi (2020), “studies indicate that good work relationships form the foundation for increased productivity in any organization” (p.378). the authors’ approach of incorporating an empathy-based workshop led by a professional facilitator is one possible solution. through these workshops, the library personnel were able to explore causes of communication barriers such as staff shortages, gaps in training, and others. this led to more initiatives such as providing information literacy training for paraprofessional staff and student assistants and developing standardized protocols for referral when the library is understaffed. this goes beyond a typical solution involving signs and giving clearer instructions to employees. it could serve as a model for other libraries that are having similar issues as well as lead to approaching the issue from an interprofessional perspective and standardization of communication. this can be a compelling approach to better assisting not just esol students, but everyone involved with the library.   references   almeida, n., & tidal, j. (2022). library wayfinding and esol students: communication challenges and empathy-based intervention. portal: libraries and the academy, 22(2), 453–474. https://doi.org/10.1353/pla.2022.0025   anasi, s. n. (2020). perceived influence of work relationship, work load and physical work environment on job satisfaction of librarians in south-west, nigeria. global knowledge, memory and communication, 69(6/7), 377–398. https://doi.org/10.1108/gkmc-11-2019-0135   hill, c. (2014). the professional divide: examining workplace relationships between librarians and library technicians. the australian library journal, 63(1), 23–34. https://doi.org/10.1080/00049670.2014.890020   mandel, l. h., & johnston, m. p. (2019). evaluating library signage: a systematic method for conducting a library signage inventory. journal of librarianship and information science, 51(1), 150–161. https://doi.org/10.1177/0961000616681837   perryman, c., & rathbun-grubb, s. (2014). the cat: a generic critical appraisal tool. http://www.jotform.us/cp1757/thecat     microsoft word news4418 evidence based library and information practice 2008, 3:4 97 evidence based library and information practice news chla/absc call for contributed papers and posters © 2008. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. the canadian health libraries association / association des bibliothèques de la santé du canada invites you to submit contributed papers or posters for its 2009 annual meeting, to be held in winnipeg, manitoba, may 30-june 3. papers and posters may describe innovative programs/practices or new research findings and should relate to the overall conference theme “the sky’s the limit / horizons illimités”. call for contributed papers join us under winnipeg’s limitless prairie sky, as we explore the horizons of best practice, research, and the evolving role of health librarianship. though papers will be accepted from a broad range of topics we invite submissions that address, in particular, activities in the following areas: on the horizon • what is the future of e-health and eresources? • what innovative programs will support health libraries and health librarianship in the future? beyond the horizon • what opportunities exist for collaboration in the clinical practice settings? • how can librarians partner with other professionals to support evidence-based health and knowledge management and translation? convergent horizons • how will libraries adapt to the shifting landscape of information delivery? • how can libraries facilitate the convergence of media, technology and social networking tools? submission process to submit a paper for consideration, please send an e-mail to analyn baker <analyn_baker@umanitoba.ca> by 15 december 2008 with the following information: • title of the paper • short structured abstract (250 words or less) • author(s) name(s), address, email, and work phone number evidence based library and information practice 2008, 3:4 98 structured abstracts should follow jchla/jabsc instruction to authors at <http://pubs.nrccnrc.gc.ca/jchla/jchla26/c04-900.pdf>. all those submitting abstracts for contributed papers will be contacted by the program committee by 19 january 2009. please note that conference registration fees will not be waived for presenters of contributed papers. call for posters poster sessions comprise visual presentations of research results, innovative library programs or notable projects. they provide a less formal alternative to contributed papers and plenary sessions and an opportunity for conference delegates to interact directly with those responsible for the poster presentations. we invite poster submissions that reflect the overall conference theme of the “the sky’s the limit / horizons illimités” and in particular, those that relate to the subthemes outlined above. submission process to submit a poster for consideration, please send an email to analyn baker at <analyn_baker@umanitoba.ca> by 15 december 2008 with the following information: • outline of the topic • brief structured abstract (no more than 250 words) • name(s) and contact information of the presenter(s) including address, email, and work phone number structured abstracts should follow the jchla/jabsc instructions to authors at <http://pubs.nrc-cnrc.gc.ca/jchla/jchla25/c04900.pdf>. all those submitting abstracts for posters will be contacted by the poster committee by 19 january 2009. accepted poster presenters will be required to submit electronic versions (pdf, powerpoint, etc.) of their posters prior to the meeting to give attendees a chance to digest material ahead of time. please note that conference registration fees will not be waived for presenters of posters. poster details • poster presenters will be allotted a push-pin, display board (4' x 8') and table space • a limited number of electrical connections will be available for electronic poster presentations • if you are planning an electronic poster presentation, you must bring your own laptop computer as well as a standalone version of your presentation • you will be required to staff your print or electronic poster during a portion of the conference • posters will be open to viewing by conference delegates for two days of the conference poster design • include the title, the author(s), affiliation(s), and a description of the research, highlighting the major elements that are covered in the structured abstract • posters are visual – add pictures, graphs, charts etc. to make the poster interesting • keep text to a minimum and use a large font size so that the poster can be read from a distance • keep some white space don’t overwhelm the reader with too much text or graphics • use color creatively evidence based library and information practice 2008, 3:4 99 • consider laminating the poster with a low-glare or matte finish • consider bringing copies of your poster for hand-out • ensure that you poster is set up and taken down at the specified times poster links: <http://www.soe.uoguelph.ca/webfiles/agal vez/poster/> <http://educ.queensu.ca/~ar/poster.htm> <http://www.esf.edu/its/html/posterref.htm> <http://www.hsl.unc.edu/services/tutorials/ poster_design/home.htm> please contact the poster chair <analynbaker@umanitoba.ca> for additional information about themes and sample poster topics. microsoft word es_kloda.doc evidence based library and information practice 2007, 2:3  87 evidence based library and information practice       evidence summary    use google scholar, scopus and web of science for comprehensive citation tracking    a review of:  bakkalbasi, nisa, kathleen bauer, janis glover and lei wang. “three options for citation  tracking: google scholar, scopus and web of science.” biomedical digital libraries 3.7  (2006).    reviewed by:   lorie a. kloda  phd student, school of information studies, mcgill university  montreal, quebec, canada  e‐mail: lorie.kloda@mail.mcgill.ca        received: 01 june 2007    accepted: 12 july 2007      © 2007 kloda. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to determine whether three  competing citation tracking services result  in differing citation counts for a known set  of articles, and to assess the extent of any  differences.    design – citation analysis, observational  study.    setting – three citation tracking databases:  google scholar, scopus and web of science.    subjects – citations from eleven journals  each from the disciplines of oncology and  condensed matter physics for the years 1993  and 2003.    methods – the researchers selected eleven  journals each from the list of journals from  journal citation reports 2004 for the  categories “oncology” and “condensed  matter physics” using a systematic sampling  technique to ensure journals with varying  impact factors were included. all references  from these 22 journals were retrieved for the  years 1993 and 2003 by searching three  databases: web of science, inspec, and  pubmed. only research articles were  included for the purpose of the study. from  these, a stratified random sample was  created to proportionally represent the  content of each journal (oncology 1993: 234  references, 2003: 259 references; condensed  matter physics 1993: 358 references, 2003:  364 references). in november of 2005,  citations counts were obtained for all articles  from web of science, scopus and google  scholar. due to the small sample size and  skewed distribution of data, non‐parametric  mailto:kloda@mail.mcgill.ca http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:3  88 tests were conducted to determine whether  significant differences existed between sets.    main results – for 1993, mean citation  counts were highest in web of science for  both oncology (mean = 45.3, sd = 77.4) and  condensed matter physics (mean = 22.5, sd  = 32.5). for 2003, mean citation counts were  higher in scopus for oncology (mean = 8.9,  sd = 12.0), and in web of science for  condensed matter physics (mean = 3.0, sd =  4.0). there was not enough data for the set  of citations from scopus for condensed  matter physics for 1993 and it was therefore  excluded from analysis. a friedman test to  measure for differences between all  remaining groups suggested a significant  difference existed, and so pairwise post‐hoc  comparisons were performed. the wilcoxon  signed ranked tests demonstrated  significant differences “in citation counts  between all pairs (p < 0.001) except between  google scholar and scopus for cm physics  2003 (p = 0.119).”     the study also looked at the number of  unique references from each database, as  well as the proportion of overlap for the  2003 citations. in the area of oncology, there  was found to be 31% overlap between  databases, with google scholar including the  most unique references (13%), followed by  scopus (12%) and web of science (7%). for  condensed matter physics, the overlap was  lower at 21% and the largest number of  unique references was found in web of  science (21%), with google scholar next  largest (17%) and scopus the least (9%).  citing references from google scholar were  found to originate from not only journals,  but online archives, academic repositories,  government and non‐government white  papers and reports, commercial  organizations, as well as other sources.    conclusion – the study does not confirm  the authors’ hypothesis that differing  scholarly coverage would result in different  citation counts from the three databases.  while there were significant differences in  mean citation rates between all pairs of  databases except for google scholar and  scopus in condensed matter physics for 2003,  no one database performed better overall.  different databases performed better for  different subjects, as well as for different  years, especially scopus, which only includes  references starting in 1996. the results of  this study suggest that the best citation  database will depend on the years being  searched as well as the subject area. for a  complete picture of citation behaviour, the  authors suggest all three be used.    commentary    this study makes a contribution to current  research on citation databases similar to  jacso, yang and meho. (for a more complete  list of recent research on the subject, see  schroeder.) it adds a unique aspect by  investigating the citation counts for two  very specific subject areas – oncology and  condensed matter physics – that present  different publishing patterns. the citation  analysis method used is appropriate for  verifying the study’s hypothesis, however  there are some issues concerning the choice  of years and databases. web of science is  often considered the premier database for  citation searching; scopus and google scholar  are still in development. certainly, the  citation sets for 1993 reflect this, since scopus  has yet to add backfiles prior to 1996. since  this study was published, scopus has added  over 800 new journal titles to its database, a  number which would certainly impact the  results if this study were repeated. google  scholar remains the newest, and least  transparent of the databases, providing, for  example, no information as to which  publication dates are covered.    other methodological issues include the  small size of the sample, and the limitation  of the subject areas, making it difficult to  evidence based library and information practice 2007, 2:3  89 statistically generalise the findings. the  authors do not take into account  publications other than research articles. the  sample is drawn from thomson’s journal  citation reports, giving web of science the  advantage, as all these journals are indexed  within it. journal self‐citations can  sometimes be quite high, and a paper  published in a web of science‐indexed  journal, statistically speaking, will have a  greater chance of being cited than a paper  that is published in a journal which is not  indexed in the database. for the citation  search itself, the authors do not explain how  google scholar was searched, as this database  is not searchable in the same way as the  other, more traditional citation tracking  tools. the value of this research could have  been enhanced by including the researchers’  search strategy either in the text, or in an  appendix.    although more research needs to be done, it  is clear from the results of this study that no  one database will suffice for calculating an  article’s citation counts. librarians would be  well advised to use all three when possible  and to educate users to do the same. in  addition, it is recommended that all three be  used for locating research when an  alternative to traditional subject searching is  desired. research looking at third‐party  software to combine data from these  databases, or others that include citation  information, is needed (see yang and meho  for an example).      citation tracking is more popular than ever.  as more providers offer tools for  performing citation searches, librarians will  need to be familiar not only with the best  tools for the job, but with the surrounding  issues as well. the proliferation of these  databases indicates an increased importance  in citation tracking, not just for locating  related research, but for measuring  academic output and performance.  as  information professionals, we need to  educate researchers about the pitfalls of  using the impact factor and citation counts  in decision making for hiring, promotion,  tenure and funding. questions remain about  the validity of using these measures as  indicators of quality of research output, and  alternatives such as the eigenfactor  (bergstrom) and the h‐index (hirsch) should  be investigated.      works cited    bergstrom, c. “eigenfactor: measuring the  value and prestige of scholarly  journals.” college & research libraries  news 68.5 (2007): 314‐316.    hirsch, j.e. “an index to quantify an  individual’s scientific research  output.” proceedings of the national  academy of sciences 102.46  (2005):16569‐16572.    jacso, peter. “as we may search –  comparison of major features of the  web of science, scopus, and google  scholar citation‐based and citation‐ enhanced databases.” current science  89.9 (2005): 1537‐1547.    schroeder, robert. “pointing users toward  citation searching: using google  scholar and web of science.” portal:  libraries and the academy 7.2 (2007):  243‐248.    “scopus qualifies more than 800 new titles  to be added to its database.” press  release. amsterdam. 30 apr. 2007. 13  july 2007  <http://www.info.scopus.com/news/pr ess/pr_300407.asp>.               http://www.info.scopus.com/news/press/pr_300407.asp http://www.info.scopus.com/news/press/pr_300407.asp evidence based library and information practice 2007, 2:3  90 yang, kiduk and lokman i. meho. “citation  analysis: a comparison of google  scholar, scopus, and web of science.”  in grove, andrew, ed. proc. of the 69th  annual meeting of the american  society for information science and  technology (asist) 43, austin, 2006.    women of colour and black women leaders are underrepresented in architectural firms featured in key trade publications evidence summary   women of colour and black women leaders are underrepresented in architectural firms featured in key trade publications   a review of: mathews, e. (2021). representational belonging in collections: a comparative study of leading trade publications in architecture. library resources & technical services, 65(3).  https://journals.ala.org/index.php/lrts/article/view/7486   reviewed by: nandi prince assistant professor ursula c. schwerin library new york city college of technology new york, new york, united states of america email: nprince@citytech.cuny.edu   received: 27 may 2022                                                             accepted:  29 aug. 2022      2022 prince. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30180     abstract   objective – to measure how well women are reflected, specifically women of colour, in architectural trade publications.   design – quantitative diversity audit.    setting – architecture field.   subjects – architectural firms whose work appeared in four trade publications (architectural record, architectural review, l’architecture d’aujourd’hui, and detail) in 2019.   methods – a diversity audit was selected to analyze the representation of various subsets of women within the architecture core collections. the avery index was used to identify architectural firms featured in four trade publications. the quantitative study collected demographic data from 354 firms, featuring 726 women. within these firms, the author sought to identify women leaders and how many of those were women of colour. the author then used four guiding questions to analyze the journals: (1) individual journals’ coverage; (2) size of the firm; (3) type of firm, and (4) firms which issued a statement in support of the black lives matter movement and the likelihood of a woman of colour being in a leadership role.   main results – the key results for the studies guiding questions were: (1) the overall average of women leaders in the firms covered in the journals was 24% and for women of colour 6%. architectural record featured the highest proportion of firms with women in leadership roles (28%) and those with women of colour as leaders (9%); (2) women leadership was higher in smaller firms (large 24%; medium 20%; small 31%) as was women of colour in leadership (large 3%; medium 6%; small 9%); (3) insufficient data was found for meaningful analysis of the representation of women according to specialization within the architectural field; and (4) the firms that issued clear blm statements were highest in the us (15%) overall. architectural record, a us publication, featured the highest percentage of firms that made clear blm statements (27%).   conclusion – the study concluded that there was an underrepresentation of women, women of colour, and black women in architectural trade publications. the author’s position is that collection development practices should adequately reflect the library users they serve with acquisition actions that increase a more equitable representation. the author stated that the practical implications for this study fall under the rubric of remediation in the following areas: (1) balance inequities in architectural programs by increasing enrollment of women; (2) identify collections which lack inclusivity, balance them with curated electronic resources; and (3) collection policies should reflect readership and encourage a sense of professional belonging. in future studies, the author acknowledges that a qualitative study based on responses from architects would complement the current study.   commentary   the author’s study demonstrates the current trend in library literature to use diversity codes to analyze collections. a study by ciszek and young (2010) on diversifying collections in large academic libraries uses this methodological approach to critique the categorization of what academic libraries select. this selected sample of architectural trade journals provided a description of the current landscape for women as practicing architects as well as women in leadership positions. further, the author points out that the lack of gender and race parity in journals that are used for precedent research impacts the representational belonging of professionals who use these sources. precedent research is a corpus of information which documents architectural projects widely used in the field. the evidence based librarianship (ebl) critical appraisal checklist was used to evaluate this study (glynn, 2006). this tool determined that the quality of the evidence was high. the avery index is a recognized tool for architectural researchers and was used to narrow down the core publications within the field. some of the strengths of this study were how the firms were chosen and what demographic data was collected. to ensure the integrity of the data collected, architectural students were used to assemble the information from the firms because of their familiarity with specific industry terms. additionally, the author incorporated anti-racist methodology to limit the researchers’ bias.    the author acknowledges that visible identification of race and gender is not perfect. consequently, ten minutes was allocated to each woman leader to glean additional demographic information from their company websites, published interviews, and social media sites. the article does not fully discuss how the study participants whose race and gender was coded as probable were processed. the author did not address how racial identity differs by country; they do suggest that a qualitative study on self-identification is being considered. visual inference to racial and gender identifications and some tendencies toward u.s. centricity existed in the methodology. the study examined u.s. and non-u.s. firms and found that the statements issued in support of the black lives matter movement were higher in the united states. the author discussed data comparisons within the subset of women of colour and black women; however, they acknowledge that specific data for the four research questions were not always available for all.   the author makes helpful recommendations for librarians who make collection decisions in order to acknowledge the diversity of their readership, particularly from marginalized populations. the broad impact of this study underscores the importance of role models in architecture with special reference to the inclusiveness for subsets of women. the author posits that it is important for collections to reflect the diversity of their students to attract and support more women of colour to the discipline. they suggest that increasing the balance in the digital and print collections will remedy the nuances of maintaining a diversified collection, but they also point to the need for deeper systemic change within libraries.   references   ciszek, m. p., & young, c. l. (2010). diversity collection assessment in large academic libraries. collection building, 29(4), 154–161. https://doi.org/10.1108/01604951011088899   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387–399. https://doi.org/10.1108/07378830610692154    mathews, e. (2021). representational belonging in collections: a comparative study of leading trade publications in architecture. library resources & technical services, 65(3).  https://journals.ala.org/index.php/lrts/article/view/7486   evidence based library and information practice using evidence in practice   developing, implementing, and monitoring a system for the management of corporate policies: a partnership between risk and evidence   amanda minns head of evidence nhs lanarkshire bothwell, scotland, united kingdom email: amanda.minns@lanarkshire.scot.nhs.uk   carol mcghee corporate risk manager nhs lanarkshire bothwell, scotland, united kingdom email: carol.mcghee@lanarkshire.scot.nhs.uk   received: 2 sept. 2019                                                                    accepted: 18 feb. 2020      2020 minns and mcghee. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29641 setting   national health service lanarkshire (nhsl) is the third largest health board in scotland serving a population of approximately 655,000 across both rural and urban communities in lanarkshire and employing approximately 12,000 staff.   the aim of the project was to develop and implement a system to manage and monitor the corporate policies for nhsl, providing assurance that policies were contemporary and extant. a working group, jointly led by   the departments of corporate risk and evidence, was set up to manage and monitor corporate policies.  corporate policies are core documents within the organization that provide a statement of principles and/or actions to ensure consistent decision-making and resource allocation. for example, much of the human resources department’s actions are based on national guidelines which are then translated into local policies (e.g., maternity policy and retirement policy). other departments also translate actions that all staff must adhere to (e.g., copyright policy and password policy). ensuring we have a consistent way to communicate these core principles and required actions is the key function of a policy. at this early stage, the nature and depth of the requirements was not clear. however, it was known that policies were dissipated throughout the board without one governance system to allow them to be easily managed. given the essential nature of the content in many of the policies, having a system that provided the necessary assurance to the board was essential.   problem   from the earliest stage, it was identified that the landscape around policies within the board was complex and decentralized. the absence of control of policies over many years had already been recognized by the corporate risk manager.   it was unclear at the start the project of the size, scale and complexity of the work that was required to be undertaken. the first step was therefore to set up a project group bringing in other interested parties to ensure the work would have buy-in across the board. this group was chaired by the corporate risk manager and outlined the desired outcomes and scope the project work required. representation from corporate risk, evidence, information management technology, practice development, nursing midwifery and allied health professionals, medical education, staff side, pharmacy, and human resources were all essential to the development of this early work.   at the initial stage in the project and in discussion with the corporate risk manager, the evidence team was able to demonstrate that they had the necessary skills to help manage this project. initially the role for evidence was to help with categorization and key terms for developing a search function of policies online. however, as the project progressed, the involvement from evidence became more central and integral to the day-to-day running of the system.   evidence   evidence for this project spanned a number of areas, including the lived experience of the staff using policies every day, as it was recognized that this evidence would be as important as traditional printed evidence.    the first evidence search was to ensure nhsl had the most up-to-date information on the existing landscape. an amnesty on the existing policies started in 2009. this allowed a developing picture of the scale of the issue that enabled discussion, consideration, and views from the collective expertise of those in the working group to outline the vision of where nhsl needed to be in one year, two years, and beyond.   the evidence team was key to:   ·         searching the literature for definitions of policies, guidelines and protocols to help define the scope of the project;  ·         seeking out who else might be doing similar work, what approach had they taken, and how far had they progressed; ·         developing a policy for nhls: developing organisation policy.   a number of literature searches were completed at this early stage that helped form the nhsl policy for developing organisation policy, including definitions, and understand the approach others had taken, if any. ensuring evidence was used – both from the printed literature and from talking to those designated authors and leads who develop policies in the health board – greatly influenced how the work developed and matured. the definitions of the different types of documents is the same now as it was at the start of the project and this has ensured that we have consistently applied the definitions as the project has changed and developed. these definitions have also helped shape other work in the management of guidelines for nhs lanarkshire.   evidence searching was and continues to be the backbone of the development and updating of policies. for example, all policies must reference their evidence base, whether this is from other policies, national standards, legislation, scottish government circulars, or partnership information guidelines produced by nhs scotland, etc.   in continuing to improve the effective management of nhsl policies, seeking evidence on accessibility of policies was also important in ensuring nhsl adopted best practices and complied with nhs lanarkshire communications strategy 2013-2018.   implementation   the working group met quarterly from 2009 and was responsible for the development, implementation, monitoring, and continuous improvement of the organizational systems that will enable access to up-to-date and endorsed polices, in support of the board of nhs lanarkshire.   there were a number of complexities when it came to implementation, in particular: how to develop a system that has the right levels of controls, but allows a degree of flexibility to work well; and how to engage with the managers/staff/teams to bring them on board? from the literature review, a policy template was developed. all new policies had to use the template and over a period of three years (maximum review period) all policies due for an update were re-written in the new template policy. this was closely monitored by the evidence team, who returned policies to authors and leads if they were not in the approved format.   information management and technology was key in the development of the control systems. given the size and complexity of the board, it quickly became obvious that change was required: the repository for the polices moved from a paper policy system to an online one, which allowed for the monitoring of policy review dates, and enabled watermarking on policies with “out of date” when printed on each document for version control.     while there was no additional funding made available, the key to success was the use of the available subject expertise recognizing that the approach would be incremental and would be at a pace that both suited the project group and the change management for the policy authors and leads. listening to users was invaluable in the development and implementation, especially as the system of notification of review dates was implemented.  feedback from users influenced a change in notification times, triggered by the evidence team, to allow for the endorsing process through the governance groups to be extended to a more reasonable period. the evidence team implemented a revised trigger process, which has proven to be much more effective.    outcome   information management and technology built a bespoke page on the intranet as the central repository for policies and the evidence team built the behind the scenes online management system that tracked and monitored the policies. in conjunction with these two systems – a key performance indicator was agreed upon by the group – it is monitored on a monthly basis. it has enabled a reporting function that provides a level of assurance to the board on the management of corporate policies across the system (figure 1). the partnership between corporate risk and evidence and their relevant expertise in continuous monitoring and improvement has led to having a system that ensures high levels of compliance.   there has been a further change within the last 12 months to move the policies from the intranet site to the nhsl public site, which involved a quality review of policies ensuring they met agreed standards for posting on the public website. a further exercise was taken to check all polices for stating compliance with general data protection regulations. the head of evidence is also the nhs lanarkshire representative for working towards the one for scotland national policies project with the scottish government which has led to further changes within policies governance.   the overall systematic approach to the effective management of the corporate policies has also been used as an exemplar for other projects within the board, demonstrating a consistently good way to approach complex management of official documentation.   figure 1 corporate policy key performance indicators. all corporate policies are reviewed within the review date, and no later than 10 working days of the scheduled review date. the tolerance threshold has been set at 95%.   reflection   collaborative working and effective leadership from the risk and evidence heads of service has been the key to the success of this project. both teams recognized their expertise and were able to bring together others in a working group to ensure the project delivered impact and value. through the corporate risk management function, the corporate management team for nhsl were fully sighted on, and supported the improvements for the change and the expectations, helping unblock barriers as they arose. making sure there was strong evidence for change was vital. while there was some resistance at the start, there has been great benefit to having a controlled, systematic, consistent, and focused approach to a complex area of system control.   conclusion   continuous development was and continues to be a core principle of the working group.  while the membership of the working group has reduced and changed over nine years, the desire to continually improve the systems for staff continues.    from an evidence team point of view, be brave and put your head above the parapet. there are many skills within the evidence department that can strengthen, and bring added value to operational and governance arrangement across an organization. initially, there was a suggestion to outsource this work. had that happened, the current system would not have been developed with the control and flexibility required and with the same passion to continue to improve.   references   brown, c. (2013) nhs lanarkshire communications strategy:  2013-18. nhs lanarkshire, scotland.   nhs lanarkshire (2020) policies, procedures, strategies and plans. available at  https://www.nhslanarkshire.scot.nhs.uk/about-us/policy-procedure-plan/ [accessed 25 november 2019].   evidence based library and information practice 2011, 7.2 99 evidence based library and information practice news/announcements congratulations to our associate editors 2012. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. on april 26, 2012, the canadian library association (cla) released a statement where it said “the cla is pleased to announce that the winners of the 2012 robert h. blackburn distinguished paper award are lorie kloda, denise koufogiannakis, and katrine mallan, who are honoured for their 2011 paper entitled, "transferring evidence into practice: what evidence summaries of library and information studies research tell practitioners," published in information research (16:1, march 2011). this study examines the critical appraisals of library and information studies published in the form of evidence summaries within evidence based library and information practice from 2006-2008. critical appraisal is a crucial aspect of evidence based practice, and evidence summaries "provide a synopsis and critical appraisal of published research in order to facilitate the transfer of research into practice." evidence summaries save the time of the practitioners who read them and provide them with the benefit of others' expertise. the results of the study show that evidence summaries reveal more weaknesses than strengths in library and information research published within this journal. critical appraisal writers found frequent weaknesses in validity and reliability of library and information research studies, and this finding should have significant impact because it will, as the authors suggest, strengthen the body of research within the field by helping to identify weaknesses and pitfalls, whereas noted areas of strength will provide "examples of good practice that new researches can model." paradoxically, the summary writers still found that even with flawed methodology, the results of the research studies appraised were still applicable. research articles in the field yielded elements that were relevant to practice, even if the actual results came from weak methodology. this paradox caused the authors to ask why and how this could be the case, and subsequent research into this question could further improve the quality of research, as well as improve the quality of evidence summaries themselves. the robert h. blackburn distinguished paper award is named for dr. blackburn, the first president of cacul (1963-64). the award annually recognizes notable research published by canadian library association members. the robert h. blackburn distinguished paper award will be presented at the cla national conference and tradeshow taking place at the ottawa convention centre, may 30 june 2, 2012. http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ evidence based library and information practice commentary   the changing nature of evidence for eblip   alison brettle reader in evidence based practice school of nursing, midwifery, social work and social sciences university of salford salford, united kingdom email: a.brettle@salford.ac.uk   received: 4 mar. 2016    accepted: 4 mar. 2016      2016 brettle. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     it has been an honour and a pleasure to have been involved in the eblip journal in so many roles over the past 10 years. i was a peer reviewer for the first issue, then associate editor articles, editor in chief, and now an editorial advisor. time has flown very quickly. over recent months fewer editorial duties have enabled me to be involved in a wider range of eblip related projects. this has given me chance to reflect on the changing nature of evidence over the years.    my initial experiences of evidence based practice took place before the terms evidence based libraries, evidence based information practice, or evidence based library and information practice were coined. in 1993, fresh out of library school, i worked in a health research unit that specialized in outcome measurement, just as evidence based medicine was being discussed and taken up within the united kingdom. my manager at the time, a strong advocate for librarians who saw them as essential components in evidence based practice, encouraged me to do research and apply that evidence to my own information practice. we went on to develop and move forward a project that examined the feasibility of taking an evidence based approach to social care (long et al. 2002a). here, we found that it was possible to “be evidence based” but the nature of evidence wasn’t as clear cut as in medicine. questions were complex and difficult to focus, quantitative research methods weren’t always appropriate, the literature was scattered, and we needed to develop techniques and approaches for searching and appraising more diverse evidence (brettle & long, 2001; long et al. 2002 a, b, c, d). to me this always had many parallels with the evidence base in library and information science (lis) and the challenges of eblip.   common definitions of eblip (e.g., booth, 2000) did not stress research evidence over professional knowledge or user preferences; however research evidence (and in particular quantitative designs) has seemed to take priority within professional discourses (eldredge, 2002) about eblip. this has long been problematic within lis, as the evidence base doesn’t lend itself to such an emphasis. the questions that librarians ask may not always lend themselves to quantitative research designs and if they do there is unlikely to be the funding to conduct large scale rigorous studies. throughout all my roles within the evidence based library and information practice (eblip) journal, my ethos has been to publish “best evidence”. this may be about the “best” type of evidence for a particular question or it may be the “best” type of evidence that can be collected by a particular library at a particular time. for example, eblip has published systematic reviews (koufogiannakis & weibe, 2006), correlational studies (eng & stadler, 2015), quantitative analysis (newell, 2010), and qualitative studies (rankin, 2012). over time as a profession we can look at our evidence base and seek to improve it, but the evidence needs to fit the question and the context and, in the meantime, we need to use the best evidence we can find to help professional decision making.   one of my early eblip related projects was to conduct a systematic review on information skills training in health libraries (brettle, 2003). one of the key findings was a lack of rigorous studies on which to draw conclusions about effectiveness. reviews conducted around the same time about clinical librarians had similar findings (winning and beverley, 2003; cimpl & wagner, 2003) and not much changed over the next few years with more systematic reviews noting a lack of rigorous studies about health libraries and their services (weightman & williamson, 2005; brettle et al, 2011). more recently, however, i have begun to sense a change. i have just completed a systematic scoping review of the evidence for professionally trained and qualified library, information and knowledge professionals (brettle & maden, 2015). looking across all library sectors we found evidence of effectiveness, impact, and value for health librarians, school librarians, academic librarians, and public libraries. the evidence came from a wide range of study designs, with some sectors favouring particular approaches. for example the return on investment method was popular within public libraries; school and academic libraries favoured correlational or mixed method studies; and health libraries had the largest number of systematic reviews and randomized controlled trials. these types of evidence may be problematic for those who follow a traditional medical hierarchy of evidence, but these studies undoubtedly provide evidence. as well as the wide variety of study designs, what was particularly heartening was that quite a number of studies were published either as primary research or as evidence summaries in the eblip journal. there is still some way to go, however. for many library sectors we found no evidence, and there are still relatively few systematic reviews (koufogianakis & brettle, 2015). we also noticed much evidence in the grey literature that is difficult to find and appraise. these issues are a challenge for researchers – but even more so for practitioners who want to use evidence to help them make decisions.   i recently co-edited a book with eblip journal colleague, denise koufogiannakis that takes a wider view of the evidence that librarians use (koufogiannakis & brettle, in press). the eblip model put forward in the book emphasizes all types of evidence, whether research evidence, local evidence or the practitioners’ knowledge (koufogiannakis, 2013). the second part of the book examines evidence based practice and the evidence base in different library sectors. eblip has developed in different ways across sectors, and as found in our scoping review (brettle & maden, 2015), different sectors favour different types of evidence and study designs. for example in school libraries, action research has been a key feature, and in special and public libraries the concept of eblip is not so well known, but the need to use evidence to demonstrate value and impact to stakeholders is key.   in the future i’m looking forward to seeing the evidence base in lis develop further and seeing librarians learn from each other in finding new ways of approaching evidence based practice and using evidence in their practice. i’m sure that the eblip journal will play a key role in this for the next 10 years and beyond.  happy 10th birthday eblip journal!   references   booth, a. (2000). librarian heal thyself: evidence based librarianship, useful, practical, desirable? paper presented at the 8th international congress on medical librarianship, 2nd-5th july 2000, london, uk   brettle, a. (2003). information skills training: a systematic review of the literature. health information & libraries journal, 20(suppl 1), 3-9.   brettle, a., & maden, m. (2015). what evidence is there to support the employment of professionally trained library, information and knowledge workers? a systematic scoping review of the evidence.  salford, university of salford.   brettle, a., maden-jenkins, m., anderson, l., mcnally, r., pratchett, t., tancock, j., thornton, d., & webb, a. (2011). evaluating clinical librarian services: a systematic review. health information & libraries journal, 28(1), 3-22. http://dx.doi.org/10.1111/j.1471-1842.2010.00925.x   brettle, a. j., & long, a. f. (2001). comparison of bibliographic databases for information on the rehabilitation of people with severe mental illness. bulletin of the medical library association, 89(4), 353-362.   eldredge, j. (2002). evidence-based librarianship: levels of evidence. hypothesis, 16(3), 10-13.   dolman, h., & boyte-hawryluk, s. (2013). impact of the reading buddies program on reading level and attitude towards reading. evidence based library & information practice, 8(1), 35-46. http://dx.doi.org/10.18438/b8n89t   eng, s., & stadler, d. (2015). linking library to student retention: a statistical analysis. evidence based library & information practice, 10(3), 50-63. http://dx.doi.org/10.18438/b84p4d   koufogiannakis, d., & weibe, n. (2006). effective methods for teaching information literacy skills to undergraduate students: a systematic review and meta‐analysis. evidence based library & information practice, 1(3), 3-43. http://dx.doi.org/10.18438/b8ms3d   koufogiannakis, d. (2013). how academic librarians use evidence in their decision making: reconsidering the evidence based practice model (doctoral dissertation, aberystwyth university). retrieved from http://cadair.aber.ac.uk/dspace/handle/2160/12963   koufogiannakis, d., & brettle, a. (2015). systematic reviews in lis: identifying evidence and gaps for practice. 8th international evidence based library and information practice conference, brisbane, australia. retrieved from https://era.library.ualberta.ca/files/nk322d79d#.vkz-j1wrs71   koufogiannakis, d. & brettle, a. (in press). being evidence based in library and information practice.  london: facet.   long, a., godfrey, m., randall, t., brettle, a., & grant, m. j. (2002). feasibility of undertaking systematic reviews in social care, part iii. report.  nuffield institute of health, university of leeds, and health care practice r&d unit, university of salford, leeds and salford.   long, a. f., godfrey, m., randall, t., brettle, a., & grant, m. j. (2002). hcprdu evaluation tool for mixed methods studies. university of leeds, nuffield institute for health, leeds. retrieved from http://usir.salford.ac.uk/13070/1/evaluative_tool_for_mixed_method_studies.pdf   long, a. f., godfrey, m., randall, t., brettle, a., & grant, m. j. (2002). hcprdu evaluation tool for qualitative studies. university of leeds, nuffield institute for health, leeds. retrieved from http://usir.salford.ac.uk/12970/1/evaluation_tool_for_qualitative_studies.pdf   long, a.f., godfrey, m., randall, t., brettle, a., & grant, m.j. (2002). hcprdu evaluation tool for quantitative studies. university of leeds, nuffield institute for health, leeds. retrieved from http://usir.salford.ac.uk/12969/1/evaluation_tool_for_quantitative_research_studies.pdf   newell, t. s. (2010). learning in simulations: examining the effectiveness of information literacy instruction using middle school students' portfolio products. evidence based library & information practice, 5(3), 20-38. http://dx.doi.org/10.18438/b85k7t   rankin, c. (2012). the potential of generic social outcomes in promoting the positive impact of the public library: evidence from the national year of reading in yorkshire. evidence based library & information practice, 7(1), 7-21. http://dx.doi.org/10.18438/b8vg8c   winning, m.a., & beverley, c.a. (2003) clinical librarianship: a systematic review of the literature. health information & libraries journal. 20(suppl 1), 10-21.   weightman, a. l., & williamson, j. (2005). the value and impact of information provided through library services for patient care: a systematic review. health information & libraries journal, 22(1), 4-25. http://dx.doi.org/10.1111/j.1471-1842.2005.00549.x   microsoft word art_lorenzetti.doc evidence based library and information practice 2007, 2:1  3 evidence based library and information practice     feature article    identifying appropriate quantitative study designs for library research      diane lorenzetti  centre for health and policy studies  university of calgary  calgary, alberta, canada  e‐mail: dllorenz@ucalgary.ca      received: 02 december 2006    accepted: 12 february 2007      © 2007 lorenzetti. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.   abstract    this paper is concerned with the identification of quantitative study designs suitable for  library research. identifying a researchable question and selecting a research method best  suited to it are key to the successful design and execution of any research project.  each  research situation is unique, and each researcher must find the method that best suits both  their situation and the question at hand.  following a brief discussion of issues related to  question development, the author outlines a checklist that may assist the process of  selecting study designs for quantitative research projects. when faced with options in terms  of study design selection, pragmatic issues such as expertise, funding, time, and access to  participants may influence this decision‐making process.    introduction  identifying a researchable question and  selecting a research method best suited to it   are two important decisions that any  investigator must make when embarking on  a research project. it is these processes with  which this paper is concerned.     after a preliminary discussion of question  development, this paper attempts to shed  some light on research design selection by  introducing a means by which researchers  can match their questions to a selection of  appropriate study designs. the focus of this  paper will be on quantitative research.    what is quantitative research?  research designs may be qualitative,  quantitative, or mixed methods techniques  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:1  4 which employ both qualitative and  quantitative methods.    qualitative research focuses on exploring  the views of individuals. “observing events  from the perspectives of those involved”  (powell and connaway 3) is used as a  means to develop theories to explain events  or behavior (creswell, educational research;  powell and connaway). because a  qualitative research approach is exploratory,  it is often used when a topic is new or little  understood, or where “existing theories do  not apply” (creswell, research design 22).    quantitative research, on the other hand, is  primarily concerned with testing theories  and measuring the relationships between  variables or the impact these variables have  on outcomes of interest (couchman and  dawson; creswell, educational research).  whereas qualitative researchers use events  and individual experiences as a way of  developing theories, quantitative  researchers begin with a hypothesis or  theory and then try to prove or disprove it.  generally speaking, a quantitative approach  might be recommended if a research project  involves the identification of “factors that  influence an outcome, the utility of an  intervention or understanding the best  predictors of outcomes” (creswell, research  design 21‐22). within this broad category  are a variety of research designs, including  surveys, bibliometrics, and experimental  designs or techniques.    experimental techniques is a category of  quantitative research design wherein the  researcher attempts to determine the impact  of an intervention (e.g., a teaching technique,  electronic database, or collection  development policy) on an outcome or effect  of interest. an example of an experimental  design is a randomised controlled trial. in  experimental research, investigators are  interested in exploring issues of “cause and  effect” (creswell, educational research 283).  an investigation into whether or not  researchers who receive training in  literature searching skills are more  successful in obtaining funding from  granting agencies than those who do not is  an example of a research problem that could  be answered through an experimental  research design. here, the intervention is the  training and the outcome or effect is the  number of successful grant applications.    true experimental designs are those in  which research participants can be  randomly allocated to receive or not receive  an intervention of interest. quasi‐ experimental designs are experiments in  which researchers, for whatever reason, are  not able to randomly assign participants and,  instead, conduct research with naturally  occurring groups ‐ for example, students  registered in an introductory political  science course (creswell, educational  research).    developing a researchable question  before tackling the demanding task of  launching a research project, we should ask  ourselves the following question: what do  we want to know?    in a survey published in a 2001 issue of  hypothesis, medical librarians were asked to  “identify the most relevant research  questions currently facing the profession”  (evidence based librarianship  implementation committee 9). the variety  and complexity of these responses clearly  indicates that the profession continues to  grapple with many challenges. among other  things, we are concerned with issues of  resource selection, budgeting, library skills  training, the adoption of new technologies,  and the need to determine what impact our  profession has on the clients we serve  (evidence based librarianship  implementation committee). there appears  to be no lack of potentially researchable  issues of interest to our profession.  evidence based library and information practice 2007, 2:1  5 an awareness of a given topic and its  potential options or issues provides  researchers with the information they need  to decide on the scope and direction of their  research. those unfamiliar with aspects of a  given topic will find it useful to explore  their area of interest prior to developing  their research questions. a literature review  may identify whether or not research  already exists in librarianship or a related  field. conducting quality original research  can be time consuming. it may be that  adequate evidence already exists that can be  adapted to your needs.    the importance of developing specific  questions that accurately reflect research  needs cannot be overstated.    background versus foreground questions  questions may be loosely grouped into two  types: background and foreground. when  you need to gather together basic  underlying background knowledge on a  topic, you are dealing with a background  question (booth; richardson and wilson). in  addressing a background question, you  attempt to arrive at a greater understanding  of the complexity and diversity of the  literature as a whole, either as an endpoint  in itself or as a first step towards developing  a more focused foreground question. an  example of a background question might be:  “what online systems exist to facilitate the  delivery of distance education courses?”     now imagine that you have identified  through research that blackboard (an online  tool that allows students and instructors to  communicate over the internet) may be an  effective tool for the delivery of distance  education courses in your institution. you  would like to determine if it would be  effective by comparing blackboard to your  existing method for online course delivery.  a foreground question is very focused in its  research aim. it contains multiple  components, which are clear and can be  clearly stated (booth; richardson and  wilson). an example of a foreground  question might be: “is blackboard an  effective tool for online library literacy  course delivery?”     pico is a model that can help facilitate the  development of a researchable question.     pico stands for:  p population  i intervention(s) or exposure  c comparison   o outcome(s) of interest    population is the group under  observation/study (in the case of  collection use studies, the population  could well be the library’s collection of  books, journals, or other media)    intervention/exposure is the action or  intervention of interest    comparison is an alternative action or  intervention against which the  intervention of interest will be measured.    outcome is a measure of the desired  effect of the intervention/exposure    although pico was developed as a clinical  model, it can be adapted to other disciplines  of research, including librarianship.     so why use pico? identifying the pico or  building blocks of a question/problem can  identify gaps in question development and  may inspire investigators to rethink and/or  reevaluate an original premise, making for a  solid researchable question.     using the pico model may also point the  way towards specific research designs  amenable to particular types of questions of  which yours is but one. for instance, if it  appears that you will be comparing one  intervention to another, this identifies your  evidence based library and information practice 2007, 2:1  6 question as belonging to a family of  questions which can be effectively  researched by employing a certain type of  quantitative study design. on the other  hand, if it appears that the research question  properly belongs to that category previously  identified as a background question, this too  is valuable information that may inform the  choice of research design.    so how can this be used? imagine we are  evaluating the reference services we offer to  students. we would like to know if an in‐ person, drop‐in reference service would  absorb more or less librarian time than an  email reference service.      if we apply the pico model, the  components of this question might be  identified in this way:    population:  students  intervention/exposure: in‐person drop‐in  reference service  comparison: email reference service  outcome: frequency of use    the research question that results from this  exercise could be:     would an in‐person drop‐in reference service be  used more frequently by students than an email  library reference service?    although this example fits neatly into the  pico format, in many instances no  comparison exists in which case “no  intervention/exposure” (or placebo) is the  comparison.      rarely are questions born in their final  format – they almost always require some  tweaking. at this point you might decide to  re‐evaluate your question to focus on any  other outcomes of interest (e.g., amount of  time spent responding to each request) or  populations (e.g., university faculty in  addition to students or perhaps some subset  of the student population).    identifying the research method  once a researchable question has been  identified, the next step is to match it to an  appropriate research design. each design  has its own strengths and weaknesses.  while some may be ideal for tracking trends,  others are better at demonstrating cause and  effect or predicting outcomes. the nature of  a research question should guide the design  selection. at the same time, there may be  more than one research design that could be  appropriate for your project. as with  question development research design  selection can be, in some instances, an  iterative process.    in a 2002 article on levels of evidence in  evidence based librarianship, eldredge  identified three categories of research  questions: prediction, intervention and  exploration (ʺebl levels of evidenceʺ).  prediction questions anticipate outcomes;  intervention questions compare actions or  processes; and exploration questions may be  equated with background questions in that  they often describe or synthesize  observations or results (booth; eldredge  ʺebl levels of evidenceʺ). eldredge  organized specific research designs  according to their appropriateness for  answering one of these three question types  (ʺebl levels of evidenceʺ). this paper was  used as the basis for developing the  checklist that follows for the selection of  quantitative research designs.    using the checklist, librarians new to the  field of research may identify possible  research designs based on the focus or goal  of the question under consideration.  consider, in general terms, what you hope  to achieve through your research. determine  which of the activities and corresponding  study designs is conducive to the aims or  goals of your research.  evidence based library and information practice 2007, 2:1  7   you may find that more than one research  design fits your needs. when deciding on an  approach, consider not only the suitability  of the research design but the expertise,  resources, time and other variables that will  be integral to the successful completion of  this piece of research. in other words, what  resources will you need in order to research  your question in the manner you have  determined to be the most effective? if those  resources are not available, a different  design may need to be selected.    descriptions of research designs    bibliometrics  bibliometrics is a method for describing  patterns of publication within a body of  literature. it has been used in librarianship  to identify core literatures, classify  literatures, predict publishing trends,  describe patterns of book use, and chart the  dissemination of ideas (powell and  connaway 63).   checklist for selecting a quantitative study design from eldredge, “ebl levels of evidence”  match the desired aims of your research activity to one or more of the activities and  corresponding research designs outlined below. descriptions of the research designs follow the  checklist.  report on observations and analyze experiences with a process, service, program, technology or  individual:    case study (aka case report)  identify trends in attitudes, behaviors or characteristics where:    participants will be surveyed at one moment in time  cross sectional survey    participants will be surveyed at multiple points in time  longitudinal survey  identify patterns of publication within a given field or body of literature:    bibliometrics/citation analysis  predict whether the presence or absence of an exposure or event influences an outcome of  interest where:    participants will be observed from just prior to the exposure into the future  prospective cohort study    participants will be identified from historical records and followed into the present  retrospective cohort study    demonstrate if the presence or absence of an intervention, service, or process causes an outcome  of interest. here the investigator will attempt to test the strength of cause and effect relationships  where:      evidence based library and information practice 2007, 2:1  8 no comparison/control group is available and the intervention group will act as its own  comparison via before and after tests  single group controlled comparison study    participants cannot be randomly assigned to intervention or comparison/control groups and pre‐ existing groups must be used  controlled comparison study    participants can be randomly assigned to intervention or comparison/control groups  randomized controlled trial    participants may be followed over time and multiple observations (pre‐intervention, during the  intervention, and post‐intervention) may be made  time series  map key concepts and/or identify gaps in a research area:    scoping review  evaluate existing published evidence with respect to a specific research question:    systematic review  pool or combine the results of published studies to determine a measure of effectiveness:    meta‐analysis  calculate the probability of expected consequences of implementing different strategies:    decision analysis    citation analysis is a form of bibliometrics in  that it focuses on identifying patterns of  citation (powell and connaway 63). citation  analysis can be used to identify seminal  studies in fields of research, trace the  citation history of studies, or identify  influential writers within a field of research.     for more information on bibliometrics,  readers are advised to consult powell and  connaway’s basic research methods for  librarians.    case study  although case studies/case reports rank low  on most evidence hierarchies, they employ  few resources, are relatively easy to  undertake, and for these reasons, are often  the first layer of evidence published with  respect to emerging questions, issues, or     techniques. eldredge reported in 2004 that  case studies ranked with surveys and  qualitative methods as one of the three most  used research designs in librarianship (ʺebl:  an overviewʺ 294). case studies are a means  of reporting on in depth observations and  explorations of an individual, population,  process, technology, or service (the  cochrane collaboration secretariat;  creswell, research design 15; eldredge,  ʺinventoryʺ; fraenkel and wallen). no  baseline information is collected and no  comparison groups are used. for these  reasons, case studies are not capable of  proving cause and effect … although they   can begin the chain of evidence that leads to  a realization of why or how something  happens (eldredge, ʺinventoryʺ).    citation analysis  ‐ see bibliometrics  evidence based library and information practice 2007, 2:1  9 cohort study  a cohort is a population of individuals who  share a characteristic (ruane 99). in a cohort  study a naturally occurring group (the  cohort) is followed over time it is exposed or  not exposed to a particular event. the  outcomes of exposed or non exposed  members of the cohort are compared  (eldredge, ʺcohort studiesʺ; the cochrane  collaboration secretariat). a cohort study is  an observational study and researchers do  not manipulate the cohort in any way. they  simply observe the course of events in an  attempt to predict whether or not there may  be a relationship between the event and an  outcome or outcomes of interest (streiner  and norman). for this reason, cohort  studies are sometimes referred to as event‐ based designs (ruane). prospective cohort  studies follow participants “into the future”  (the cochrane collaboration secretariat 8);  whereas retrospective or historical cohort  studies identify participants “from historical  records and follow them from the time of  those records into the present” (the  cochrane collaboration secretariat 8).    for more information on cohort designs,  particularly as they relate to librarianship,  readers are advised to consult eldredge’s  2002 article on this topic (ʺcohort studiesʺ).    controlled comparison study  in controlled comparison studies, two  groups are compared (eldredge, ʺebl:an  overviewʺ 295). comparisons are made  between an intervention group receiving  intervention a and a comparison /control  group b receiving either intervention b or  no intervention (fraenkel and wallen 273).  the aim is to determine if these  interventions (e.g., two different  instructional techniques) have an effect on  an outcome of interest (e.g., knowledge  retention, course grades, etc…). participants  are not randomised to either group. instead,  controlled comparison studies attempt to  reduce bias (the chance that observed  outcomes will reflect, for example,  differences in the makeup of the two groups  rather than differences in the interventions)  by attempting to closely match  characteristics of participants in the  intervention and comparison/control groups  (eldredge, ʺebl: an overviewʺ 295). in the  case of the instructional techniques example,  we might want to ensure that members of  both groups are in the same year and  program of study, are comparable in age  and background, and have had similar past  exposure to library instruction. conducting  both pre‐tests and post‐tests also helps  account for variations in these naturally  occurring groups. controlled comparison  studies are referred to as a type of quasi‐ experimental design.     for more information on controlled  comparison studies, readers are advised to  consult creswell’s educational research  methods and powell and connaway’s basic  research methods for librarians both of  which contain detailed chapters on  experimental and quasi‐experimental  designs.    cross sectional survey ‐ see survey    decision analysis  decision analysis is a transparent decision  making method wherein all options and  possible consequences of each choice are  made explicit. it “involves identifying all  available alternatives and estimating the  probabilities of [and assigning values to]  potential outcomes associated with each  alternative (evidence based medicine  working group 409). existing research and  data are used to develop the model and  options, and the potential results of those  options are represented graphically as a  decision tree. researchers use the  probability and value of all outcomes to  develop a “quantitative estimate of the  relative merit of [each alternative]”  (evidence‐based medicine working group).  evidence based library and information practice 2007, 2:1  10 for more information, readers are advised  to consult keeney’s overview on this topic.    longitudinal survey ‐ see survey    meta‐analysis  meta‐analysis is the “use of statistical  techniques in a systematic review to  integrate the results of included studies”  (the cochrane collaboration secretariat 25).  by statistically pooling the results of  multiple studies, it is possible to determine,  with greater certainty, the effect or non‐ effect that an intervention has on an  outcome of interest (egger, smith, and  altman; powell and connaway).    although all meta‐analyses should begin  with a systematic review, not all systematic  reviews do or should include a meta‐ analysis. although it is possible to conduct a  meta‐analysis with as few as two studies, a  misleading conclusion can result if  researchers attempt to combine the results of  different study designs (e.g., rcts and  cohort studies) or include data that is not  comparable.      for more information on meta‐analysis  techniques, readers are advised to consult  systematic reviews in health care: meta‐ analysis in context.    prospective cohort study  ‐ see cohort study    randomised controlled trial  a randomised controlled trial (rct) is “an  experiment in which two or more  interventions…are compared by being  randomly allocated to participants” (the  cochrane collaboration secretariat 37). the  intervention group receives the intervention  of interest to the researchers and the  comparison/control group receives either no  intervention (ie placebo) or an alternate  intervention (creswell, educational  research 295). the ability to randomise  participants eliminates many of the  opportunities for bias that exist in other  research designs. that being said, it is not  always possible or appropriate to randomise  participants. for instance, imagine that your  library has a subscription to a particular  electronic database. it is expensive and you  are wondering if it is worth retaining. it may  be difficult (not to mention unethical) to  randomise students to either have or not  have access to a database or other service for  which there is already system‐wide access.  on the other hand, the benefits of email  versus in‐person reference service, one  process of handling document delivery  requests versus another, one‐time versus  multiple follow‐up library instruction  training, etc … may be successfully explored  through an rct design (eldredge, ʺthe rct  designʺ).     for more information on rct design and  application, readers are advised to consult  eldrege’s 2003 article on this subject  (eldredge, ʺthe rct designʺ), powell and  connaway’s basic research methods for  librarians, and creswell’s educational  research: planning, conducting, and  evaluating quantitative and qualitative  research.    retrospective cohort study ‐ see cohort study    scoping review  a scoping review is similar to a systematic  review in that it employs rigorous methods  to rapidly yet systematically locate and  review literature appropriate to the research  question under review. unlike systematic  reviews though, scoping reviews tend to  address broader, less focused questions,  include a variety of study designs, and tend  to not assess the quality of included studies  (arksey and oʹmalley). scoping studies are  often undertaken to map key concepts in a  research area and/or identify gaps in the  existing literature. they can be used to  quickly chart complex areas of research,  provide a visual picture of the range of a  evidence based library and information practice 2007, 2:1  11 field, and are also useful in identifying  topics where further research might be  warranted.     for more information on scoping reviews,  readers are advised to consult arksey and  o’malley’s article on this topic as it provides  both an overview and detailed method for  proceeding with this type of review.    single group controlled comparison study  a single group controlled comparison study  (also known as a single group pretest  posttest design) is a controlled comparison  study wherein only one group is under  observation by researchers. the group is  measured or observed both before and after  exposure to the intervention of interest.  unlike other controlled comparison studies,  the intervention group acts as its own  comparison or control (creswell, research  design; creswell, educational research;  fraenkel and wallen; powell and  connaway).    survey  surveys are “questionnaires administered to  a sample of a population in order to identify  trends in attitudes, opinions, behaviors or  characteristics” (creswell, educational  research 52). cross sectional surveys or  studies are referred to as snapshots because  they essentially capture information from a  group of respondents at one moment in time  and are useful for describing current  situations and conditions (ruane 93).  longitudinal surveys, on the other hand,  gather information at multiple points in  time and may be used to track changes in  values, opinions, attitudes, and behaviors  (ruane 95). longitudinal surveys may  repeatedly collect information from the  same individuals or generate new survey  samples each time the survey is  administered (creswell, research design;  fraenkel and wallen; ruane). a checklist for  designing a survey method can be found in  creswell’s research design: qualitative,  quantitative, and mixed methods  approaches (155).    for more information on survey designs,  readers are advised to consult fraenkel and  wallen’s how to evaluate research design  in education.    systematic review  a systematic review is “a review of the  evidence on a clearly formulated question  that uses systematic, explicit [and  reproducible] methods to identify, select  and critically appraise relevant primary  [original] research, and to extract and  analyze data from the studies that are  included in the review” (centre for reviews  and dissemination 4). unlike traditional  narrative reviews, systematic reviews are  comprehensive, unbiased, and systematic in  the ways in which literature is located,  selected, abstracted, and reported.      for an excellent overview of the process of  conducting systematic reviews, readers are  advised to consult the 2001 report produced  by the university of york’s centre for  reviews and dissemination entitled  undertaking systematic reviews of  research on effectiveness.    time series  controlled comparison studies and  randomised controlled trial designs assume  that the researcher will observe/measure  participants at a single point in time (twice if  a pre‐test is included in the research design).  if you are able to observe participants over a  period of time, you may wish to consider a  randomised or non‐randomised time series  approach. in a time series, multiple pre‐test  and post‐test measures or observations are  taken both before and after the intervention  (creswell, educational research; polit,  tatano‐beck, and hungler 177). the  strength of the time series design is in its  ability to track the impact of an intervention  over time.  evidence based library and information practice 2007, 2:1  12 discussion  in a statement released in 2001, the special  libraries association (sla) commented that  “both the need to make decisions quickly  and the lack of a clear connection between  much library and information science  research and the day‐to‐day problems faced  by librarians make seeking and applying  our own knowledge base a challenge.”  although it may be argued that librarians  continue to experience a certain lack of  professional support with respect to  engaging in research activities, we cannot  claim to be the only profession that is beset  with tight timelines when it comes to  making and implementing decisions.     for those of us not schooled in research  methods however, undertaking a research  project of any sort may appear to be a very  intimidating experience. certainly a brief  overview article of this sort will, at best,  only whet the appetite for more information.  although there is much that can and should  be read on this subject before embarking on  a research project, it is often in the  application that we learn the most.     partnering with an experienced professional  colleague may be the best way for a  budding researcher to become familiar with  the research process. if this is not an option,  we may wish to consider looking a bit  further afield. many granting agencies look  favorably on proposals submitted by  multidisciplinary teams. librarians have  skills that make them valuable partners in a  variety of research endeavors. those of us  who work in academic libraries are  surrounded by research faculty experienced  in leading multifaceted research  investigations, some of whom may be open  to mentoring a beginning researcher in  exchange for active participation on their  teams.    insofar as research designs are concerned,  the author wishes to reiterate that many  questions may be more effectively  addressed through qualitative research  techniques, and these approaches should be  considered when choosing an appropriate  design. for a discussion of qualitative  research designs in librarianship, readers are  advised to consult eldredge’s 2004  inventory of research designs (eldredge,  ʺinventoryʺ) and powell and connaway’s  basic research methods for librarians.  many researchers are also embracing the  need to incorporate both qualitative and  quantitative methods in their research  projects (creswell, research design).     conclusion  this paper has touched on the subject of  research design in only the most cursory  sense. no attempt has been made to  comprehensively represent the range of  research designs that exist and may be  applied or adapted to answering all  questions facing our profession. readers are  encouraged to “research” the process more  thoroughly, and speak with colleagues  before proceeding to engage in a research  endeavor.    although the focus of this paper has been  on select quantitative techniques, it is not  the intention of the author to recommend  the application of one study design or group  of designs over another in all research  situations. not every question can or should  be answered by a randomised controlled  trial. each research situation is unique, and  each researcher must find the method that  best suits both their situation and the  question at hand. when faced with options  in terms of study design selection, it may be  that pragmatic issues such as professional  expertise, time, and access to participants  will influence the decision‐making process.    sometimes the ideal study design from a  research question perspective is not ideal  from a practical standpoint; an important  characteristic of a researcher should be the  evidence based library and information practice 2007, 2:1  13 ability to recognize and creatively adapt to  the challenges of reality.     works cited     the cochrane collaboration secretariat.  ʺglossary of terms in the cochrane  collaboration. version 4.2.5.ʺ the  cochrane collaboration. may 2005. 18  feb. 2007  <http://www.cochrane.org/resources/h andbook/glossary.pdf>.   egger, matthias, george davey smith, and  douglas altman, eds. systematic  reviews in health care: meta‐analysis  in context. ed. m. egger, g. davey‐ smith, and d. g. altman. london: bmj  books, 2003.  evidence‐based medicine working group.  ʺhow to use a clinical decision  analysis.ʺ centre for health evidence.  2001. 18 feb. 2007  <http://www.cche.net/usersguides/deci sion.asp>.  arksey, hilary and lisa oʹmalley. ʺscoping  studies: towards a methodological  framework.ʺ international journal of  social research methodology 8.1 (feb.  2005): 19‐32.  booth, andrew. ʺformulating answerable  questions.ʺ evidence‐based practice  for information professionals: a  handbook. eds. andrew booth and  anne brice. london: facet publishing,  2004. 61‐70.  centre for reviews and dissemination.  ʺundertaking systematic reviews of  research on effectiveness: crdʹs  guidance for those carrying out or  commissioning reviews.ʺ  university  of york centre for reviews and  dissemination. mar. 2001. 18 feb.  2007<http://www.york.ac.uk/inst/crd/r eport4.htm>.  couchman, wendy and jane dawson.  nursing and health‐care research: a  practical guide. 2nd ed. london:  scutari press, 1995.  creswell, john w. research design:  qualitative, quantitative, and mixed  methods approaches. 2nd ed.  thousand oaks, ca: sage publications,  2003.  ‐‐‐. educational research: planning,  conducting, and evaluating  quantitative and qualitative research.  2nd ed. upper saddle river, nj:  merrill, 2005.  eldredge, jonathan d. ʺevidence‐based  librarianship: an overview.ʺ bulletin  of the medical library association 88.4  (oct. 2000): 289‐302.  ‐‐‐. ʺcohort studies in health sciences  librarianship.ʺ journal of the medical  library association 90.4 (oct. 2002):  380‐92.  ‐‐‐. ʺthe randomised controlled trial  design: unrecognized opportunities  for health sciences librarianship.ʺ  health information and libraries  journal 20.s1 (2003): 34‐44.  ‐‐‐. ʺinventory of research methods for  librarianship and informatics.ʺ journal  of the medical library association 92.1  (jan. 2004): 83‐90.  ‐‐‐. ʺevidence‐based librarianship levels of  evidence.ʺ hypothesis 16.3 (fall 2002):  10‐13.  evidence based librarianship  implementation committee. ʺthe most  relevant and answerable research  questions facing the practice of health  http://www.cochrane.org/resources/h http://www.cche.net/usersguides/deci http://www.york.ac.uk/inst/crd/r evidence based library and information practice 2007, 2:1  14 sciences librarianship.ʺ hypothesis  15.1 (2001): 9‐17.  guyatt, gordon and drummond rennie.  usersʹ guides to the medical literature:  essentials of evidence‐based clinical  practice. chicago: american medical  association press, 2001.  fraenkel, jack r. and norman e. wallen.  how to design and evaluate research  in education. 5th ed. boston: mcgraw  hill, 2003.  keeney, ralph l. ʺdecision analysis: an  overview.ʺ operations research 30.5  (sep./oct. 1982): 803‐38.  streiner, david l. and geoffrey r. norman.  pdq epidemiology. 2nd ed. hamilton,  on: b.c. decker, 1996.  polit, denise f., cheryl tatano beck, and b.  p. hungler. essentials of nursing  research: methods, appraisal, and  utilization. 5th ed. philadelphia:  lippincott williams & wilkins, 2001.  powell, ronald r. and lynn silipigni  connaway. basic research methods for  librarians. 4th ed. westport, ct:  libraries unlimited, 2004.  richardson, w. s. and m. c. wilson. ʺon  questions, background and  foreground.ʺ evidence based health  care newsletter 17 (nov. 1997): 8‐9.  ruane, janet m. essentials of research  methods: a guide to social science  research. malden, ma: blackwell  publishing limited, 2004.  special libraries association. ʺsla research   statement: putting our knowledge to  work.ʺ special libraries association.  jun. 2001. 18 feb. 2007  <http://www.sla.org/content/resources/ research/rsrchstatement.cfm>. http://www.sla.org/content/resources/ news/announcements   call for proposals: c-eblip fall symposium 2016      2016. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     c-eblip fall symposium: librarians as researchers wednesday, october 12, 2016 university library, university of saskatchewan    call for proposals   the third annual centre for evidence based library and information practice (c-eblip) fall symposium: librarians as researchers will take place on wednesday, october 12, 2016 at the university of saskatchewan. this one-day conference (with complimentary registration including an all-new networking breakfast, breaks, and lunch) is designed to facilitate sharing, collaboration, and networking with the focus on librarians in their researcher roles.  the symposium consists of a series of single-track sessions, along with a keynote address and time for networking. the sessions will focus not only on the research being done by librarians, but also on how librarians do research (tips, tricks, methodology, collaborative efforts, etc.) and why librarians do research (motivations or inspirations).   the librarian as researcher is becoming less of an anomaly and more just part of the job. conducting research and disseminating the results are important activities for practicing librarians. whether it’s a requirement for tenure and promotion, part of evidence based library and information practice, curiosity, contributing to one’s own practice, or   contributing to the field of librarianship, librarians approach research from the unique perspective of practice.   sessions are 20 minutes in length which includes time for questions. session topics should broadly relate to one of the focus areas of the symposium:    research being done by librarians – results of research projects, a project in its interim state, experiences with conducting research, etc. how and/or why librarians are conducting research – methods, techniques, tips, tricks, inspirations, motivations, philosophical underpinnings of conducting research.    session proposals should include the following:   title of the presentation description of the presentation (no more than 200 words) how the proposal links to a focus areas (research being done by librarians; how/why librarians are doing research) 2-3 outcomes or takeaways   applicants will be advised of the status of their proposal by the week of august 8, 2016.   submission deadline is monday, july 25, 2016.   submit proposals here: c-eblip proposal submission form   research article   arranging the pieces: a survey of library practices related to a tabletop game collection   teresa slobuski research services coordinator dr. martin luther king, jr. library san jose state university san jose, california, united states of america email: teresa.slobuski@sjsu.edu   diane robson associate librarian media library university of north texas denton, texas, united states of america email: diane.robson@unt.edu   pj bentley public services librarian west slope community library washington county cooperative library services portland, oregon, united states of america email: paulb@wccls.org   received: 18 may 2016  accepted: 13 nov. 2016      2017 slobuski, robson, and bentley. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – the purpose of this study is to explore collection development, cataloguing, processing, and circulation practices for tabletop game collections in libraries. this study used the term “tabletop games” to refer to the array of game styles that are played in real-world, social settings, such as board games, dice and card games, collectible card games, and role-playing games.   methods – an online survey regarding tabletop games in libraries was developed with input from academic, public, and school librarians. participants were recruited utilizing a snowball sampling technique involving electronic outlets and discussion lists used by librarians in school, public, and academic libraries.   results – one hundred nineteen libraries answered the survey. the results show that tabletop games have a presence in libraries, but practices vary in regard to collection development, cataloguing, processing, and circulation.   conclusion – results indicate that libraries are somewhat fragmented in their procedures for tabletop collections. libraries can benefit from better understanding how others acquire, process, and use these collections. although they are different to other library collections, tabletop games do not suffer from extensive loss and bibliographic records are becoming more available. best practices and guidance are still needed to fully integrate games into libraries and to help librarians feel comfortable piloting their own tabletop collections.     introduction   libraries have supported games and play for over a century. the early 20th century saw the emergence of toy libraries that were established to support families in need by lending toys, board games, and other realia that support play (moore, 1995). since the 1970’s, digital games have become the most visible and dominant medium of play in our culture and in libraries (nicholson, 2009). although video game sales remain prevalent, tabletop games have entered a new golden age, beginning with the emergence of designer games in the mid-1990s. since 2012, purchases of board games have risen annually by more than 25% as online retailers have made them available to the mass market (duffy, 2014). recent estimates placed total industry sales above $880 million in 2014 (icv2, 2015). this growth has been further accelerated by the advent of crowdfunding as a means to finance and pre-order new games (roeder, 2015).   as the hobby gains in popularity, librarians are reevaluating tabletop games as a viable collection for their patrons’ needs. for many librarians, it is not a matter of whether to include tabletop games in a library’s collection but a matter of how. even though the establishment of toyand game-lending collections predates the establishment of libraries’ video game collections, research about tabletop game collections lags behind the research on video game collections. librarians interested in collecting video games can find a plethora of information on incorporating video games into their programs and collections. however, a librarian interested in building a tabletop game collection will find relatively few resources to guide them.   despite well-established benefits of gaming and booming growth in the tabletop industry, only a small number of libraries circulate tabletop games. in a 2007 survey of 313 libraries, 44% circulated games with 27.9% of those libraries circulating board/card games (nicholson, 2009); that equates to 12% of libraries overall circulating board/card games. since this question has not been surveyed since 2009, it is difficult to gauge the current proportion of libraries that circulate games. this is not meant to imply that libraries are not incorporating tabletop games into their services in other ways. many libraries provide games in their children’s areas, host chess and go clubs, and run gaming programs (nicholson, 2009). however, the practice of developing, processing, cataloguing, and circulating a tabletop game collection is relatively rare.   tabletop games, much like other types of realia, can be daunting to libraries because of presumed cost, durability, and complexity. because few resources address those concerns, it is not surprising that few libraries have developed tabletop game collections despite the growth of the hobby. this study was conducted to gather information from the libraries that do have games collections in order to determine their procedures and practices. the authors address some of the perceived challenges and issues regarding tabletop game collections and offer ways to improve access and management of this type of special collection through the creation of standards and best practices. literature review   games in libraries   modern libraries include tabletop games in their services in different ways. to gain a better sense of the history of games in libraries, it is helpful to expand the scope to include other play media. in his 2013 article, “playing in the past: a history of games, toys, and puzzles in north american libraries”, nicholson makes clear how libraries have historically supported play through their programs, services, and collections.   hosting clubs and offering programs seem to be the earliest means by which libraries supported play. nicholson (2013) notes the earliest mention of games in libraries is a chess club at the mechanics’ institute library in 1850’s san francisco. the relationship between gaming communities and the library has evolved so that game clubs and gaming programs have become standard among many libraries’ offerings. nicholson’s survey of libraries (2009) found that 43% offered gaming programs, most of which included tabletop games. in 2007, the american library association began collaborating with game companies to provide free tabletop and digital games to libraries that participate in international games day. in 2010, around 1,800 libraries participated and in 2015, 2,157 libraries participated. the coordinators of the program surveyed participating libraries; among those who responded, 57% had offered gaming programs in the last year in addition to their international games day event (international games day @ your library, 2016). these numbers show that while there is not much formal documentation about libraries and tabletop gaming, many libraries are enthusiastically participating in the trend.   libraries have also supported play by building lending collections. toy libraries emerged during the great depression in north america and were the first to lend games in addition to toys and puzzles. moore’s a history of toy lending libraries in the united states since 1935 (1995) documents these types of collections. her research starts at the first toy lending library in a garage in 1930’s los angeles. the library ensured that families that could no longer afford toys, puzzles, or games could still access them. in 1970, the american library association’s children’s services division began reviewing toys for use in libraries (moore, 1995). today two associations, the usa toy library association (usatla) and the international toy library association (itla), exist to support libraries and librarians that manage toy collections.   despite this long history there is still reluctance to fully integrate games into the library. as bierbaum notes in her 1985 survey of realia in libraries, new media is often decried as the destroyer of libraries as we know them. (bierbaum, 1985). in order to cater to their users' interests, libraries incorporate new media, technology, and realia into their collections regardless of this outcry, but if their emerging collections are not as fully integrated as standard collections, they will be only partially accessible to the users they are intended to serve.   collection development   the need for collection management guidance is ongoing as both digital and analog games evolve. law (1976) stresses the need for librarians to become well-versed in game collection management. law’s concerns hold true today, including keeping up with game resources and literature to evaluate games for purchase, improving searching and finding in catalogues, the physical care of circulating games, and loss prevention (law, 1976). bastiansen and wharton (2015) note additional challenges for toy libraries, such as adequate staffing, collection visibility, and maintenance of materials.   current scholarly publications that directly address the topic of tabletop game collections are practically non-existent. a few publications, such as nicholson’s (2010) book, everyone plays at the library: creating great gaming experiences for all ages, provide advice for starter collections or outline characteristics of good games for libraries. a few articles focus on role-playing game collections. “dungeons and downloads: collecting tabletop fantasy role-playing games in the age of downloadable pdfs” and “dragons in the stacks: an introduction to role-playing games and their value to libraries” give overviews of major role-playing games. (sich, 2012; snow, 2008). however, no substantial writings were found that explore board game collection development.   determining what to buy is not the only issue that libraries face as they consider this type of collection. chadwell (2009) discusses the issues that managers face. many librarians and administrators see game formats as disruptive because new procedures and policies are often needed to handle games. however, these concerns are shortsighted because libraries are becoming more efficient in other areas, such as automatically delivered bibliographic records, shelf-ready item processing, and automated materials handling. this should allow libraries time to handle new formats as needed, but again this survey shows that librarians treat this type of format differently to other standard formats.   cataloguing special collections are considered hidden if not in the library catalogue. in the white paper “hidden collections, scholarly barriers: creating access to unprocessed special collections materials in north america’s research libraries” the contributors state why all collections should be catalogued if possible: uncatalogued collections are at greater risk of being lost or stolen, are inaccessible to the community, and access is staff dependent. (jones, 2003) in her 1985 study, bierbaum surveyed 218 public libraries about three-dimensional realia collections of which toys and games were the most popular category. of these libraries, 163 collected toys and games but many were not cataloguing these items.  this survey noted a lack of guidance in cataloguing non-print materials as a possible cause for the lack of catalogue records.   thirty years later there is still little in-depth information beyond the basic realia cataloguing rules set forth in the anglo-american cataloguing rules, second edition (aacr2) and resource description and access (rda). olson (2001) uses a tabletop game as an example and does state the need to include information about the number of players, recommended age, and purpose of the game.  in a slideshow presentation for the association for library collections and technical services (alcts), mcgrath (2012) includes helpful hints for tabletop game cataloguing. however, moore (2014) reflects different practices for game cataloguing. piascik (2002) briefly reviews the cataloguing and circulation of special materials but notes that sixty-nine percent of their materials lacked records in the online computer library center (oclc). the original cataloguing needed in such cases requires advanced knowledge if the catalogue records are to be complete and useful.   at this time, professional organizations have not provided best practices for cataloguing tabletop games or for assigning subject or genre headings to these materials. librarians continue to need more guidance in these areas if they are to provide satisfactory access to tabletop games.   game preservation and processing   more publications address the preservation of video games than of tabletop games. this imbalance exists because of the real danger of losing digital games to media format obsolescence. lowood et al. (2009) detail many of the issues surrounding video game preservation. there are no readily available publications for analog game preservation or processing. circulating collections require additional steps not necessary for personal use collections. piascik (2002) gives a few tips for processing games, including advice to use bags for pieces and to make creative use of conventional library materials. most of the information on tabletop game preservation is not library-specific and exists only on gaming blogs and in forum posts.    although articles about games and their value are readily available there is not enough current, in-depth research about tabletop games in libraries. tabletop game collections will remain niche experiments in libraries until literature that provides guidance for collection development, cataloguing, processing, lending, and preservation finds its way into professional and scholarly publications.   aims   the lack of resources and baseline data specific to tabletop collections in libraries contributes to misconceptions about practices and can hinder librarians who are considering establishing game collections. the researchers designed a survey to gather information about current practices for these materials with the belief that the results would provide practical information on how tabletop game collections are implemented and maintained both for librarians who are exploring the possibility and those who are looking to improve their existing collections. the specific goals of the study were to understand the norms and related issues regarding tabletop game collections. the study focused on the following research questions:   ·         are libraries cataloguing their tabletop game collections so they are discoverable? ·         are libraries circulating their tabletop game collections outside of their buildings? ·         what barriers are libraries facing in fully integrating tabletop games into the library?   methods   an online survey was developed to gather information from libraries with game collections. input was gathered from several academic, public, and school librarians to craft the questions. the survey was created using qualtrics research software and was tested by members of the american library association’s games and gaming round table. the responses were collected in june and july of 2015. participants were recruited utilizing a snowball sampling technique involving electronic outlets and discussion lists used by librarians. the request to participate included an explanation of the purposes of the research and a link to the survey.   results   demographics   the results included responses from 119 participants with 66% of the participants from public libraries, 28% from academic libraries, 3% from special libraries, and 3% from other (3 curriculum resource centers and 1 school library). the respondents were from urban (21%), suburban (26%), and rural (31%) areas, with 17% indicating mixed and a few libraries reporting other. total library budgets ranged from $50,000 to $5 million plus, with the majority from libraries with $1 million to $4.9 million budgets. of the 119 respondents 81% have a tabletop game collection and 19% do not.     table 1 overview of practices based on library type     which of the following best describes your library?     public academic special other total do you create catalogue records for tabletop games for your opac? yes 15 8 2 4 29 no 42 6 0 0 48 sometimes 10 6 1 0 17   total respondents 67 20 3 4 94 do you create item records with barcodes for your tabletop games? yes 17 13 2 4 36 no 33 3 0 0 36 sometimes 5 1 1 0 7   total respondents 55 17 3 4 79 do you circulate your tabletop games? yes 17 11 3 3 34 no 38 6 0 0 44   total 55 17 3 3 78 do you offer programming around your tabletop games? yes 41 11 1 2 55 no 13 6 2 2 23   total respondents 54 17 3 4 78     collection development   collection development and curation of tabletop game collections is unique to each library and its patrons’ needs. the survey included several questions about these practices. surveyed libraries’ collections range from very broad and informal ones that include mostly donations to well-curated collections that support institutional goals.   unlike other media, most games are unavailable through library vendors. nevertheless, games are being added to collections. there were 77 libraries that acquire games through both purchasing games (84%) and accepting donations (66%). of the 65 libraries that purchase games, most (74%) use online vendors such as amazon and barnes & noble, 57% purchase from physical game stores, 32% purchase from physical chain stores, 18% purchase from online game vendors such as cool stuff inc., funagain, and miniature market, 11% purchase from library vendors, and 5% purchase from “other,” including thrift stores, garage sales, and ebay.   the budget for purchasing tabletop games ranged from $0 (all donations) to over $500. out of 76 libraries, 46% have a budget of up to $249, 30% have $0, 12% have $250 $500, and 12% have over $500 to purchase games. libraries with lower budgets tended to favour general vendors, both online and physical. selection criteria range from purchasing popular, family friendly, or award-winning games to solely purchasing games that support coursework and classroom instruction.     table 2 vendor usage by game budget   what is your budget for purchasing tabletop games?   $0 $1 $249 $250 $499 $500+ total what type of vendor do you use to purchase your games? online general vendors (amazon, barnes & noble, etc.) 9 26 7 6 48 online game vendors (cool stuff inc., funagain, miniature market, etc.) 0 5 3 4 12 online library vendors (ingram, brodart, etc.) 0 3 1 3 7 physical chain stores (barnes & noble, books-a-million, target, etc.) 5 12 0 2 19 physical local stores (game shops, comic books stores, etc.) 8 17 5 5 35 other (please specify) 2 0 1 1 4   total respondentsa 11 34 9 9 63 a respondents could choose more than one vendor type.     although book donations to libraries are often castaways, donated games are not always from the back of someone’s closet. many game companies understand that more gaming is good for their business. there were 50 libraries that provided insight into game donations. of those 50, 45 accept donations from patrons, 25 receive games by participating in international games day, 19 accept donations from publishers, 17 from local businesses, and 8 from “other”, including staff and local thrift stores. donation criteria range from accepting only complete games in good condition to anything that is offered. some libraries accept any type of game regardless of age range or content, but others only accept games that are appropriate for the library’s collection needs.   cataloguing   despite the fact that cataloguing is a cornerstone for discovery in libraries, survey results reveal that cataloguing practices of tabletop game collections are inconsistent. there were 94 libraries that answered the question “do you create bibliographic catalogue records for tabletop games?” of those 94, 31% do, 51% do not, and 18% answered “sometimes.” there were 39 libraries that responded to a question regarding what types of tabletop games have catalogue records. board games are most frequently catalogued with 79% of those libraries reporting these kinds of records. libraries also catalogue card sets (54%), roleplaying guides (54%), and “other” (13%) games. “other” games include puzzles, totes with multiple games, and games tied to curriculum goals. libraries that “sometimes” create catalogue records were most likely to create them for roleplaying (rpg) guides with 73% of the libraries that sometimes catalogue tabletop collections having this practice. these results do not show an increase in game cataloging when compared to the findings of bierbaum’s survey of public libraries (bierbaum, 1985).     table 3 catalogue records for different tabletop game types   do you create catalogue records for tabletop games for your opac?   yes sometimes total what types of tabletop games get catalogue records? board games 23 8 31 card sets 16 5 21 rpg guides 10 11 21 other 3 2 5   total respondentsa 24 15 39 a respondents could choose multiple types of games.     standards in cataloguing also vary. of libraries surveyed, 22 libraries use oclc to catalogue their games. these libraries were asked to approximate the percentage of games they have catalogued which already had oclc records. there were 11 libraries that responded with 4 answering less than 25%; 5 answering 25% to 49%; and 2 answering 50% to 74%. no libraries reported that over 75% of games they catalogued already had records in oclc.   both subject headings and classification numbers are essential for access to collections. however, out of 36 libraries, only 22% find library of congress subject headings sufficient to aid in finding tabletop games in the opac. out of 39 libraries, 22 (56%) create local subject or genre headings using other resources, including board game geek (www.boardgamegeek.com), father geek (www.fathergeek.com), game descriptions, and reviews. librarians are also creating subject terms that include curricular area, grades, awards, and mechanisms. standard call numbers are not as widespread for these collections. of 37 answering libraries, 16% use library of congress (lc), 32% use dewey, 46% use local call numbers, and 5% use no call number “none”.     figure 1 subject heading sufficiency.     perhaps unsurprisingly, many libraries still have reservations regarding cataloguing and processing tabletop games. one respondent stated, “keeping all the pieces is not easy and we have become more wary of entering new items into the catalog.” besides material concerns, the unusual nature of these items causes some to be hesitant or doubtful of their cataloguing ability. one respondent’s comments could ring true for any size library when first starting to provide access to these materials: “i am in a one person library and i am in no way good at original cataloging which has held me back from cataloging a lot of our board games. best practices for original of board games would be great.” however, as seen in the above results, libraries are still attempting to provide access to their tabletop games. one library stated that they “add a color-coded sticker and letters to indicate the primary audience(s) for each game,” to aid browsing the collection. another notes that, “it's important to mark and indicate every item in game (i like to put in individual baggies), to ensure that materials are not missing when loaned & returned.” the complexity of most games could be daunting for a cataloguer unaccustomed to cataloguing realia since there are no best practices to follow.   as more libraries collect and catalogue games, the availability and quality of records in oclc should increase. the survey did not include perception questions for the 51% of libraries that indicated they are not currently cataloguing their tabletop games. however, we can infer by responses to other questions that the in-house usage and small size of many of these collections negate the perceived need for bibliographic records. the lack of sufficient subject and genre headings and classification is also a barrier for finding and using games. the combination of perceived complexity of cataloguing with a dearth of standards means that these collections are more hidden than others in the library.   processing   while cataloguing provides intellectual access to collections, processing is key to providing physical access. questions specific to processing tabletop games were included in the survey to address topics such as item records, barcoding, and physical processing of games.   as with cataloguing, processing procedures remain inconsistent for tabletop collections. of 79 respondents, there is an even split of 46% of libraries that create item records with barcodes for their tabletop collection and 46% that do not, while 9% only barcode sometimes[1]. even the process of barcoding is quite variable when compared to traditional collections with 38 respondents putting barcodes on the game box, 5 putting them on the game’s instructions, 5 putting barcodes on each of the bags or pieces/cards within the game, and 10 placing them on other parts of the game. also, barcoding is sometimes limited to game types, with one respondent noting that “rpg guides” are the only items that get barcodes, which may mean only those would receive item records.   from the answers to this survey, few games are processed to increase longevity and reduce wear-and-tear. however, of the 79 libraries that answered questions regarding processing, 22% reinforce the game’s box; 54% separate games pieces into bags; but only 6% put plastic or archival sleeves on cards to protect them from damage. for some libraries, how their collection is used negates the need for extensive processing. one respondent stated, “in our library the games have been considered just to be used within the building they are cheap and easily replaced. not much money or effort is put into ‘preserving’ them.”   however, some libraries with games for in-house use only do carry out extra processing. one library noted that they “put a security strip in the board game boxes so that the board game collection can only be used in the library.” another library “keep[s] the reinforced game boxes in a very visible area and [has] all of the guts behind the desk. we don't check them out or have them cataloged. we feel that this strikes a nice balance for our patrons to know that we have these games and that they are there to be played with, but also keep good track of the pieces, etc.” furthermore, some libraries are even more conscientious about their processing, especially those that provide out-of-library checkouts. one survey participant wrote, “i put library stickers and a library name stamp on everything.” another library provided a unique way to manage the many parts of some tabletop games without individual barcodes: “we weigh the various types of components of each game with a digital scale and attach this information to the game. that way, we can tell if all items have been returned when they're checked in.” while unusual, weighing could allow for clearer check-in procedures for circulation staff. as with cataloguing, the processing of tabletop games remains an area without clear library standards. this lack of standards leads to an unwarranted fear, not seen with print materials, concerning damage and loss for this format (see below).     table 4 barcoding and processinga   do you create item records with barcodes for your tabletop games?   yes no sometimes total what parts of the game get their own item records/barcodes: (choose all that apply) box 30 2 5 38 instructions 4 1 0 5 bags of each type of pieces/cards 3 0 2 5 others (please specify) 6 3 1 10   total respondents 35 5 7 47 do you leave your games in their original containers? yes 34 32 7 73 no 2 3 0 5   total respondents 36 35 7 78 do you separate game pieces into bags? yes 21 19 3 43 no 15 16 4 35   total respondents 36 35 7 78 do you sleeve your individual cards with plastic sleeves? yes 3 1 1 5 no 33 34 6 73  total respondents 36 35 7 78 a respondents could select multiple parts to have item records or barcodes. respondents also could answer regarding parts, containers, bagging pieces, or sleeving cards even if they had previously stated “no” or “sometimes” in regards to creating item records.     table 5 circulation practices for tabletop collections     count of responses what is the loan period for tabletop games in your collection? less than 1 day 6 1-3 days 4 4-7 days 5 7-14 days 10 14+ days 12   total respondentsa 34 can patrons place a hold on a tabletop game? yes 23 no 10   total respondents 33 can patrons renew a tabletop game? yes 26 no 8   total respondents 34 a respondents could select multiple options to indicate that some tabletop games have different loan periods than others.     circulation   cataloguing and processing a collection prepares it for potential circulation. librarians on social media and blogs have discussed how to circulate tabletop games without undue hardship on staff, and the researchers hope this survey offers some insight for libraries considering circulating their games.   much like cataloguing and processing, there are no best practices for circulating a tabletop collection. of 78 responding libraries, 44% report that they circulate tabletop games. however, if in-library, in-school, and out-of-library borrowing are considered together, it is clear that more libraries are circulating games. out of 77 answers, 65% of libraries report that they allow in-house library use only, 1% in-school only, and 34% lend outside of the library or off the premises. from the difference in the results from these two questions, it is clear that some libraries consider in-library/school use as circulation, while others do not. perhaps this is due to the lack of catalogue records for items that are indeed available for use in the building. of the 44% of libraries that indicated that they do circulate this collection, 82% let all of the library’s patrons check out games, 9% have age restrictions, and 3% have other restrictions such as checking out to faculty/staff members only. the loan period varies from less than 1 day to 14 plus days, with the majority (35%) being 14+ days. seventy percent of these circulating libraries allow holds to be placed on games, and 76% allow renewals.   as noted above in the cataloguing section, 51% of the respondents do not catalogue their collections, so actual visibility is important for finding their collections. of the 80 libraries that answered questions regarding tabletop collection storage, 25% store collections behind the counter but visible to patrons; 36% stored them behind the counter but not visible to patrons (closed stacks); and 39% store theirs in public areas (open stacks). of the 31 libraries with open stacks for their games, 23% are in the teen’s area, 19% are in the children’s area, 19% are with media items, and 65% are in “other,” which includes displays near front desks or entry points, community resource areas, lounge areas, curriculum collection areas, and in toy and game libraries.   loss prevention is one of the leading concerns that can cause a library to not circulate tabletop games. questions were included to help gauge procedures related to loss prevention. regular inventorying is one common method to prevent loss. as with cataloguing and processing, inventory procedures are varied with 48% of 80 respondents counting pieces at each return while 28% never count their pieces. more rarely, 18% count pieces yearly, 6% monthly, and 1% weekly. it is unclear if any of the 28% of libraries that do not inventory use alternative methods to ensure games are complete, such as the weighing system mentioned earlier.   although 73% of responding libraries conduct inventories, only 34% purchase new pieces when they are lost, and most (77%) do not charge patrons replacement fees. one library that charges a replacement fee noted, “none charged over last year at 5 branches. replacements have been minor.” another mentioned that they would charge but that the situation has yet to come up at their library. the types of pieces replaced include instruction booklets, game pieces, tokens, and cards. libraries that do replace pieces have many ways to manage the replacement process. one respondent wrote, “i sometimes buy duplicate copies of games at thrift stores and garage sales, so that i can use them for replacement parts as needed.” several noted that many games can be played even when some pieces are lost, so replacing the pieces is not always necessary. “we would make replacement judgments based on the specific game. we would try to work with users to get pieces back, but would charge if significant pieces were missing.” another noted that they were able to get the publisher to send them a replacement piece.   overwhelmingly, it seems the fear of lost pieces should not be a deterrent against circulating tabletop collections, considering comments such as: “the largest concern with circulating board game were missing or broken pieces [sic]. as of nearly a year of circulating 50+ games, we have had only one missing piece. it was gladly replaced by the publisher”; “we have not had any instances of lost pieces or damaged games, so we haven't developed too many policies yet to handle these issues”; and “lost pieces was the biggest fear, and it was for naught. although some pieces do go missing, it does not happen at a high rate. and many games are completely functional even if some components get lost.” these comments should assuage the fears that libraries that are new to collecting or circulating games may have.     table 6 tabletop programming by library type   which of the following best describes your library?   public academic other special total do you offer programming around your tabletop games? yes 41 11 2 1 55 no 13 6 2 2 23   total respondents 54 17 4 3 78 which programs do you provide which involve tabletop games? family board gaming events 28 1 0 0 29 game jams 2 1 1 0 4 board game design events 5 2 1 0 8 adult gaming events 17 9 1 0 27 teen gaming events 28 2 1 0 31 other tabletop gaming events 9 4 1 1 15   total respondentsa 41 11 2 1 55 a respondents could choose multiple programs they offer.     programming and events   programming and events continue to grow in all types of libraries. in nearly all libraries with tabletop collections, programming is a key element to the collection. although the majority of the survey did not consist of questions regarding programming, write-in responses such as, “we don't circulate games to patrons, just to staff for program use” occurred throughout the cataloguing, processing, and circulation sections.   when asked about offering programming around tabletop games, 78 libraries answered with the majority (71%) confirming they do offer programming with their collection. these programs include teen gaming events (56%), family board game events (53%); adult gaming events (49%), board game design events (15%), game jams (7%), and other events (27%) including game days, tournaments, lectures, and club meetings.   many of these libraries elicit help from outside agencies for gaming events. this help comes from staff and faculty (49%), teen/student clubs (29%), local board game meetups (24%), game retailers (13%), and professional agencies (4%). connections to volunteers, community game stores, and local gamers bring visibility to programs and help ease the pressure on staff. although not a focus for this research, it is clear that tabletop collections provide an outlet for libraries to connect to patrons as well as to other community stakeholders.   conclusion   this research provides an extensive first look at tabletop game collections in libraries. the results show that libraries are fragmented in their procedures for creating, employing, and maintaining these collections, which is not surprising since each library must develop their practices locally or at best through informal communications with other libraries. however, this research also reveals commonalities among many libraries. for those who catalogue tabletop games, they benefit from having bibliographic records available from oclc while at the same time they find that subject and genre headings remain inadequate. for those who circulate games, most find that the fear of lost pieces was misplaced and that circulation can be accomplished by using procedures that make sense for their location. overall, many comments show that while different from mainstream library formats, tabletop games can find a place in a library’s collection. understanding the surveyed libraries’ current practices should encourage other libraries to pilot their own tabletop game collections or increase access to their existing collections.   however, comments and the variety of responses to the survey questions reveal that libraries and researchers have much work to do in this area. the demand for tabletop games in the wider marketplace is increasing. libraries should be meeting the cultural, recreational, and educational needs of their users by meeting this demand, but they are falling behind. many tabletop games go out of print. libraries should be collecting tabletop games in order to preserve them for study and future use, but in this area they also fall behind. most libraries are not collecting or offering the format in any significant way while those that do must create local practices. for this reason, researchers and professional organizations should be developing resources and best practices that empower libraries to successfully meet the needs of their users. at the same time, as revealed by the survey results, libraries do not need to wait for codified standards in order to launch tabletop game collections that are discoverable, well-preserved, and available to borrow. current attempts can be imperfect while still providing significant access.   there should be no insurmountable barriers to incorporating this format into a library. most of the problems are based on misconceptions instead of reality. we should not let another 30 years pass before we start to fully integrate tabletop games into the library. standards would help ease unwarranted fears, but a shift in attitude about this type of collection also needs to take place. libraries have supported games and play for over a century, and now librarians and researchers have the opportunity to strengthen this tradition for another century by establishing standards and best practices for tabletop game collections.   references   bastiansen, c., & wharton, j. (2015). getting ready for play! toy collections in public libraries. children and libraries: the journal of the association for library service to children, 13(4). 13-16,29. http://dx.doi.org/10.5860/cal13n4.13   bierbaum, e. g. (1985). the third dimension. public library quarterly, 6(3). 33-50. http://dx.doi.org/10.1300/j118v06n03_05   chadwell, f. a. (2009). what's next for collection management and managers? user-centered collection management. collection management, 34(2), 69-78. http://dx.doi.org/10.1080/01462670902731081   duffy, o. (2014, november 25). board games' golden age: sociable, brilliant and driven by the internet. the guardian. retrieved from http://www.theguardian.com/technology/2014/nov/25/board-games-internet-playstation-xbox   icv2. (2015). hobby games market climbs to $880 million. icv2: the business of geek culture, 88(fall), retrieved from http://icv2.com/articles/markets/view/32102/hobby-games-market-climbs-880-million   international games day @ your library. (2016, january 12). retrieved from http://igd.ala.org/2016/01/12/igd-2015-final-roundup/   jones, b. m. (ed.). (2003). hidden collections, scholarly barriers: creating access to unprocessed special collections materials in north america’s research libraries [white paper]. retrieved from http://www.arl.org/storage/documents/publications/hidden-colls-white-paper-jun03.pdf   law, g. (1976, oct.). the librarianship of games. the 15th conference of north american simulation and gaming association. raleigh, nc, usa.   lowood, h. (ed.), monnens, d., vowell, z., ruggill, j. e., mcallister, k. s., & armstrong, a. (2009). before it's too late: a digital game preservation white paper. american journal of play, 2(2). retrieved from http://www.journalofplay.org/issues/2/2/special-feature/its-too-late-digital-game-preservation-white-paper   mcgrath, k. (2012). cataloging three-dimensional objects and kits with rda [powerpoint slides]. retrieved from http://downloads.alcts.ala.org/ce/03282012_rda_3d_kits_slides.pdf   moore, j. e. (1995). a history of toy lending libraries in the united states since 1935. [master’s research paper]. retrieved from eric database (ed 390 414).   moore. j.r. (2014). cataloging 3d objects & 2d graphics using rda and marc21 [powerpoint slides]. retrieved from http://olac2014.weebly.com/uploads/2/5/1/9/25198051/olacmoug2014-moore-3d2d.pdf   nicholson, s. (2009). go back to start: gathering baseline data about gaming in libraries. library review, 58(3), 203-214. http://dx.doi.org/10.1108/00242530910942054   nicholson, s. (2010). everyone plays at the library: creating great gaming experiences for all ages. medford, nj: information today.   nicholson, s. (2013). playing in the past: a history of games, toys, and puzzles in north american libraries. the library quarterly: information, community, policy, 83(4), 341–361. http://doi.org/10.1086/671913   olson, n. b. (2001). cataloging three-dimensional artefacts and realia. cataloging & classification quarterly, 31(3-4), 139-150. http://dx.doi.org/10.1300/j104v31n03_01   piascik, j. m. (2002). cataloging and processing of special format science curriculum materials. oclc systems and services, 18(3), 153-156. http://dx.doi.org/10.1108/10650750210439377   roeder, o. (2015). crowdfunding is driving a $196 million board game renaissance. in fivethirtyeight. retrieved from http://fivethirtyeight.com/features/crowdfunding-is-driving-a-196-million-board-game-renaissance/   sich, d. (2012). dungeons and downloads: collecting tabletop fantasy role-playing games in the age of downloadable pdfs. collection building, 31(2), 60-65. http://dx.doi.org/10.1108/01604951211229854   snow, c. (2008). dragons in the stacks: an introduction to role-playing games and their value to libraries. collection building, 27(2), 63-70. http://dx.doi.org/10.1108/01604950810870218     [1] percentage totals 101% due to rounding error. news   call for applicants for copyeditors for evidence based library and information practice  2021. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29929     evidence based library and information practice (eblip) is looking for two new copyeditors to join our copyediting team. ideally, candidates will be in place by may 2021, in time to begin work on the june issue.   the role of copyeditors is to:   accept or reject copyediting requests in a timely manner. meet copyediting deadlines set by editors. communicate with the lead copyeditor regarding circumstances that might require temporary absences from editing availability. utilize eblip copyediting guidelines alongside apa style guidelines. ensure that proper grammar and spelling conventions are addressed. verify citations and references to the extent possible with available bibliographic resources and advise editor of any not checked. identify inconsistencies or changes in format or style as necessary.   the ideal candidate will have copyediting or other detail-oriented editing experience, possess working knowledge of apa style, and have an interest in evidence based practice. the position requires dedicated time to ensure eblip publishing deadlines are met, and it is therefore essential that interested persons ensure available time to devote to this position prior to applying. it is estimated that the workload is approximately 5 to 10 hours per issue, and much of this work falls over a one-month period, on a quarterly basis, prior to the publication of each issue. interested persons should send a cover letter, indicating areas of strength they would bring to the role, and resume/cv as a single pdf file to heather healy, lead copyeditor, at heather-healy@uiowa.edu, by april 9, 2021.    *please note that evidence based library and information practice is a non-profit, open access journal and all positions are voluntary and unpaid. the positions are an excellent opportunity for continuing professional development and gaining experience in reviewing or critically appraising library-related research. **only those applicants who are selected or shortlisted will be contacted. about the journal   published quarterly, and hosted by the university of alberta, this peer-reviewed, open access journal is targeted at all library and information professionals interested in an evidence based model of practice. by facilitating access to librarianship research via original research articles and evidence summaries of relevant research from the library literature, evidence based library and information practice will enable librarians to practice their profession in an evidence based manner.   please visit the evidence based library and information practice web site (http://journals.library.ualberta.ca/index.php/eblip) for further information about the journal.   editorial   ebl 101 special issue   virginia wilson director, centre for evidence based library and information practice (c-eblip) university library, university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca      2016 wilson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   welcome to a special issue of evidence based library and information practice. this issue will serve to commemorate the journal as it begins its eleventh year of publication. it will also be a compilation of all of the ebl 101 columns from the past several years, columns which have been intended to provide an introduction to the concepts of evidence based library and information practice (eblip) as well as research methods.   picture it: may 23, 2008 – vancouver, bc – the canadian library association conference – at the evidence based librarianship interest group (eblig) breakfast meeting. five librarians gathered, sipping coffee (and likely tea), and enjoying the obligatory continental breakfast. after talk pertaining to the interest group, the conversation turned to the evidence based library and information practice journal and an observation was made around the notion that while willing, librarians might not feel they have the skills to practice in an evidence based way. to address this potential gap, the idea of a regular column focusing on manageable parts of the eblip process was put on the table. true to form, eblip ran with it, with julie mckenna and su cleyle writing the first ebl 101 column entitled “evidence based librarianship backgrounder” (eblip, 2008, 3(3)). and that was that! ebl 101 was off and running! lorie kloda took on the second column about asking the right question in eblip 3(4). shortly after that, i was asked to write the column on a regular basis. from eblip 3(3) to eblip 6(1), ebl 101 consisted of the steps of the eblip process. when that had come to its conclusion, then-editor in chief, denise koufogiannakis, suggested morphing ebl 101 into a column dedicated to various research methods using the same approachable style and short format. and off we went on research methods from eblip 6(2) to eblip 10(2).   in 2008 i had been at the university of saskatchewan for three years as an academic librarian and was looking ahead to the tenure process. so as well as having a propensity for eblip, i thought that this regular column business would look pretty good in my tenure case file. and so it did, but that wasn’t the reason i kept on with the column, writing 24 pieces over the years. i found that in writing with the aim of helping others to learn and grow, i was learning and growing. i became fascinated with dissecting the pieces of the elbip process, and later on, various research methods. i constantly thought about making these things accessible, helpful, straightforward, and doable.   the response to ebl101 has been very gratifying. the columns have been and continue to be tweeted, cited, and shared. i’m grateful that the editors at the eblip journal gave me the chance to contribute in this way. and i am thrilled by the opportunity to introduce this special issue of ebl 101 columns. this issue contains all past ebl 101 columns – the steps of eblip and a variety of research methods. i hope you find it useful as you continue on your path of eblip and research.   ebl 101   a new path: research methods   virginia wilson client services librarian murray library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   originally published in: evidence based library and information practice, 6(2), 85–87. https://ejournals.library.ualberta.ca/index.php/eblip/article/view/10233/8146     received: 26 apr. 2011  accepted: 28 apr. 2011      2016 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   for more than two years, this column has taken you through the steps of evidence based library and information practice (eblip). with the final step taken in the last issue (keeping in mind that eblip is an iterative process and that the notion of the final step is sometimes interpretive), it is time to choose a new path and take the first step on an exciting new journey. for the next while, i will explore the exhilarating world of research methods!   do i sound invigorated? i am! i’m no expert, by any means. but, i am a lifelong learner; a practitioner-researcher with a strong interest in research methods, so we are going to learn together. and of course this column is ebl 101, so the information will be introductory and by no means exhaustive. as of right now, i have no set plan on the exact methods i will tackle, nor the order in which i will wrestle them to the ground. so if you have any needs or suggestions, please let me know. for this first column on our new path, i’m going to talk about qualitative and quantitative research in general. yes, that is a big topic for a small column, so let’s see how it goes.   the rivalry between the toronto maple leafs and the montreal canadians (hockey for the non-canadians on board) has nothing on the rivalry between quantitative and qualitative research methods, or at least between the researchers devoted to them. qualitative scholars consigned quantitative research to the lower echelons of the scientific field because of its “subjective, interpretive approach” (denzin & lincoln, 2005, p. 2). qualitative researchers have defended the subjective, interpretive approach, stating that it does not reduce the subject to numbers, but rather uses a holistic approach to preserve the complexities of the subject (black, 1994, p. 425). granted, we are moving in the direction of being able to see the value in both types of methodology, and the rivalry is becoming a comfortable living arrangement (at least i think it is!). however, that was not always the case, and for a long time, only the seemingly solid numbers of quantitative data were seen to have value.   in the early days of eblip, a hierarchy of evidence was put forward which borrowed heavily from evidence-based medicine, placing randomized controlled trials near the top and case studies down at the bottom. research methods were given a place in the hierarchy based on their perceived robustness as methods, not taking into account the quality of the research that comes out of that methodology, which is highly subjective depending on the individual study (booth, 2010, p. 84). qualitative methods were given a nod, but the impression was one of the privileging of quantitative methods. given writes that “qualitative researchers and the results of their work remain marginalized in eblip” (2007, p. 16). she explains this marginalization by looking at the history of eblip: “in understanding the historical path that this movement has taken in our field (i.e. moving through the ranks of health librarianship to try to find a stronger voice across library contexts), it is little wonder that the controversial baggage about the quality of qualitative evidence has been transported along the way” (given, 2006, p. 382). every movement has growing pains and i believe it is safe to say that qualitative methods are coming into their own in eblip. lis as a social science is well-suited to both realms of methodology. i refer you to an editorial written by denise koufogiannakis in evidence based library and information practice (2010, 5.3). in it, she addresses the evidence hierarchy and points out that the choice of research methods should be driven by the type of research question asked. this viewpoint about choice emerges elsewhere (newman & benz, 1998, p. 14; bell, 2010, p. 6; silverman, 2010, p. 9).   the notion of research is often fraught, no matter from which discipline you look at it. and while i am going to focus the rest of the column on outlining some of the differences between quantitative and qualitative methodology, i urge you to explore the debate between the two. there seems to be as many opinions as there are researchers. as well, i should note that pitting one against the other via a laundry list of differences sets these two paradigms up as a dichotomy. many involved in the debate feel that a dichotomous relationship limits both methodologies (morgan & smircich, 1080, p. 491). we would be better served to realize that there are times when both types of methods can be used in the same research study (also known as mixed-methods research) for a depth and richness of perspectives and in order to triangulate findings. this is based on the premise that “multiple viewpoints allow for greater accuracy” (jick, 1979, p.602).   in its simplest definition, quantitative research works with numbers and qualitative research works with people or text. quantitative methods attempt to answer “how many?” and qualitative methods attempt to answer “why is this happening?”  quantitative research emerged from a positivistic paradigm, and the belief that there is one objective reality and it is our job to apprehend it. qualitative research follows the naturalistic paradigm, which posits that there are many realities, that the subject is involved in creating his/her own reality, and that it is our job to discover and explore those subjective realities. quantitative methods, using deductive processes and statistical analyses, attempt to confirm a hypothesis and test a theory, while qualitative methods, using inductive processes, explore a phenomenon and attempt to generate a theory. methods common to quantitative research include surveys, randomized controlled trials, and highly structured observation. methods common to qualitative research include in-depth interviews, focus groups, and participant observation.   of course this only scratches the surface. the number of books and articles on the subject of research methodology is astounding. just three examples include research design: qualitative, quantitative, and mixed method approaches by john w. creswell, the sage handbook of qualitative research, and introduction to quantitative research methods: an investigative approach by mark balnaves and peter caputi. the ways in which researchers utilize research methods are varied. in upcoming columns, i will make my way from one method to another and i hope you will join me along the way. it strikes me that the features found in the open journal system used by eblip would be very handy for having an ongoing discussion on research methods. look for the article tools on the right side of the page when you have opened up an article in the journal. you will have the options, among others, to email the author or post a comment. you can also use the “add a comment” feature found underneath the pdf version of the article. i welcome such interaction, as i believe that a community of like-minded individuals could generate conversations that would enrich us all.   references   bell, j. (2010). doing your research project (5th ed.). berkshire, uk: mcgraw hill.   black, n. (1994). why we need qualitative research. journal of epidemiology and community health, 48, 425-426.   booth, a. (2010). on hierarchies, malarkeys and anarchies of evidence. health information and libraries journal, 27, 84-88.   denzin, n. k. & lincoln, y.s. (2005). the sage handbook of qualitative research (3rd ed.). thousand oaks, california: sage.   given, l. (2007). evidence-based practice and qualitative research: a primer for library and information professionals. evidence based library and information practice, 2(1), 15-22.   given, l. (2006). qualitative research in evidence-based practice: a valuable partnership. library hi tech, 24(3), 376-386.   jick, t. d. (1979). mixing qualitative and quantitative methods: triangulation in action. administrative science quarterly, 24, 602-611.   koufogiannakis, d. (2010). the appropriateness of hierarchies. evidence based library and information studies, 5(3), 1-3.   morgan, g. & smircich, l. (1980). the case for qualitative research. academy of management review, 5(4), 491-500.   newman, i. & benz, c.r. (1998). qualitative-quantitative research methodology:exploring the interactive continuum. carbondale: southern illinois university press.   silverman, d. (2010). doing qualitative research (3rd ed.). london, sage.   microsoft word news2_1809 evidence based library and information practice 2008, 3:2 76 evidence based library and information practice news professional grant proposal writing workshop © 2008. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. the grant institute's grants 101: professional grant proposal writing workshop will be held at the university of phoenix ann arbor campus on 20-22 august 2008. interested development professionals, researchers, faculty, and graduate students should register as soon as possible, as demand means that seats will fill up quickly. all participants will receive certification in professional grant writing from the institute. for more information call (888) 824 4424 (213-817-5308 outside us) or visit the grant institute website <http://www.thegrantinstitute.com>. the grant institute's grants 101 course is an intensive and detailed introduction to the process, structure and skill of professional proposal writing. this course is characterized by its ability to act as a thorough overview, introduction and refresher at the same time. in this course, participants will learn the entire proposal writing process and complete the course with a solid understanding of not only the ideal proposal structure, but a holistic understanding of the essential factors, which determine whether or not a program gets funded. through the completion of interactive exercises and activities, participants will complement expert lectures by putting proven techniques into practice. this course is designed for both the beginner looking for a thorough introduction and the intermediate looking for a refresher course that will strengthen their grant acquisition skills. this class, simply put, is designed to get results by creating professional grant proposal writers. participants will become competent program planning and proposal writing professionals after successful completion of the grants 101 course. in three active and informative days, students will be exposed to the art of successful grant writing practices, and led on a journey that ends with a masterful grant proposal. grants 101 consists of three (3) courses that will be completed during the three-day workshop. fundamentals of program planning this course is centered on the belief that "it's all about the program." this intensive course will teach professional program development essentials and program evaluation. while most grant writing "workshops" treat program development and evaluation as separate from the writing of a proposal, this class will teach students evidence based library and information practice 2008, 3:2 77 the relationship between overall program planning and grant writing. professional grant writing designed for both the novice and experienced grant writer, this course will make each student an overall proposal writing specialist. in addition to teaching the basic components of a grant proposal, successful approaches, and the do's and don'ts of grant writing, this course is infused with expert principles that will lead to a mastery of the process. strategy resides at the forefront of this course's intent to illustrate grant writing as an integrated, multidimensional, and dynamic endeavor. each student will learn to stop writing the grant and to start writing the story. ultimately, this class will illustrate how each component of the grant proposal represents an opportunity to use proven techniques for generating support. grant research at its foundation, this course will address the basics of foundation, corporation, and government grant research. however, this course will teach a strategic funding research approach that encourages students to see research not as something they do before they write a proposal, but as an integrated part of the grant seeking process. students will be exposed to online and database research tools, as well as publications and directories that contain information about foundation, corporation, and government grant opportunities. focusing on funding sources and basic social science research, this course teaches students how to use research as part of a strategic grant acquisition effort. registration $597.00 tuition includes all materials and certificates. each student will receive: the grant institute certificate in professional grant writing the grant institute's guide to successful grant writing the grant institute grant writer's workbook with sample proposals, forms, and outlines registration methods on-line complete the online registration form at www.thegrantinstitute.com <http://www.thegrantinstitute.com/> under register now. we will send your confirmation by e-mail. by phone call (888) 824 4424 (213-8175308 outside us) to register by phone. our friendly program coordinators will be happy to assist you and answer your questions. by e-mail send an e-mail with your name, organization, and basic contact information to info@thegrantinstitute.com and we will reserve your slot and send your confirmation packet. research in practice   librarian research: making it better?   virginia wilson director, centre for evidence based library and information practice (c-eblip) university library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   received: 15 feb. 2016   accepted: 16 feb. 2016      2016 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   a few weeks ago i was having a conversation with a senior colleague in my field about librarians as researchers. this colleague mentioned that traditionally, librarian research hasn’t been “very good” and then asked me, “how do we make librarian research better?” as a proponent and supporter of librarians as researchers, and as one who knows we are getting more rigorous and adept with our research all the time, i kind of inwardly bristled, but attempted to answer the question. i couldn’t shake this question, though, and so i consulted with members of the centre for evidence based library and information practice (c-eblip), asking for their thoughts on how to make librarian research better. several of my colleagues* responded to my query of how to make librarian research better and some common themes emerged.   before i get to the themes, however, i want to explore the skepticism that one colleague felt concerning the notion of librarian research traditionally not being “very good.” since the start of my library career, i’ve heard disparaging talk about the “how we done it good” papers (the first time was in library school, sadly enough), case studies that are descriptive in nature and not necessarily generalizable. yes, the library literature has many such pieces. are they useful? it depends on what you’re looking for. are they generalizable? it depends on a number of factors. but to generalize about them all, much how the original observation lumped all “library research” into one thing, is something we want to avoid. my colleague pointed out that there was a time when quantitative research was the top of the pile and every other type of research was looked down upon. assumptions were challenged and now there are qualitative and mixed methods research projects that are as rigorous as you would want. i share my colleague’s view: any research that brings new knowledge to the forefront is important. have we been told that our research isn’t “very good” for reasons other than an objective and fair assessment of quality? my colleague stated that “when a tall grown man tells a young kid that he (the kid) is not tall enough then one needs to be a bit skeptical about the intention behind such a remark.” is the same thing happening here?   regardless, there’s always room for improvement in any discipline. as one colleague pointed out, we are not the exception. there is research published all along the continuum, from hogwash to exemplary, in all disciplines. (which brings up a whole new topic that will not be explored here. hello, peer review.)  so with the help of colleagues and of course my own thoughts on the matter, i have compiled six strategies to consider that may make librarian research “better”:   1.       confidence   i am not being overly hierarchical with this list, but i purposely chose confidence to be number one. this was my answer to the senior colleague’s question outlined in the first paragraph. i have talked to many librarians in different countries about research and, more often than not, the idea of confidence, or the lack of it, comes up. imposter syndrome, self-deprecation, low research self-esteem, lack of confidence in research skills – i would argue that these all play a role from time to time in even the most seasoned librarian practitioner-researcher. thoughts of “i can’t do this,” “i don’t know what i’m doing,” and “i don’t want anyone to read this stuff” are common. if you have been feeling like you are the only one who doesn’t know what is going on with all this research business, you are not alone. that doesn’t mean that you stop conducting research. however, a good dose of confidence can help us navigate the research process and there are ways to gain confidence for research. three ways follow.   2.       collaboration   collaborating with a colleague on a research project is one way to feel like you’re not going it alone. teaming up with a like-minded colleague allows you to have a second set of eyes (or more if the collaboration is larger), to split the work, to share the credit, and to learn and grow as a researcher. research quality can improve by collaborators being accountable to one another. if your research partner is at the same stage you are, you can learn together and learn from each other. if one collaborator has more experience, then mentorship can come into play. one of my colleagues feels that what is even stronger than collaboration alone is collaboration with a mentorship component. and mentorship is next on the list.   3.       mentorship   a research mentor, either one within or one outside of your discipline or area of interest, can help with confidence issues and can provide advice and suggestions in all areas of the research enterprise. research mentors can comment on drafts of work, help with grant writing, and suggest venues for dissemination. mentors can be found online (i.e., http://interlibnet.org/) or in person. libraries can have formal mentorship programs or the mentorship relationships can happen spontaneously. there are mentoring relationships that consist of peer mentors or of an experienced mentor and a novice mentee. age really isn’t a factor as there might be an early career librarian who is well versed in a specific methodology mentoring a later career librarian who is a novice in that area. at the university of saskatchewan library, c-eblip rolled out four research mentorship teams this year for new librarians, each consisting of four mentors and the mentee. we will be assessing the program to see if it meets is objectives – preparing faculty librarians for the tenure process and resulting in more robust programs of research.    4.       practice makes better   as with anything that takes a certain amount of skill, research takes practice. early endeavours may be fraught with “oh no, i should have” or “why didn’t i.” but just as piano playing, skiing, and tai chi improve with practice, so can research. the collaborations and mentorship spoke of above can help early career librarian researchers get where they are going, whether it’s tenure or improving practice. one of my colleagues pointed out that the thing to do is to just do it. take the plunge with useful guidance and advice from peers and mentors, and chances are the research will improve. and you will gain confidence!   5.       education – library schools; professional development   the required research methods course found in most canadian library schools is not enough. from my own experience in library school (and i’ve heard this anecdotally elsewhere), the program neither prepared me to conduct research as part of a librarian job, nor did it make it clear how important research can be to one’s practice – as a requirement for tenure/permanent status or as moving practice forward in a systematic way no matter what the library sector. as well, at least when i received my mlis, the thesis route was not encouraged. one colleague reported that early in the library school program, the class was actively discouraged from doing a thesis as part of the program. without much opportunity to be immersed in a research project, librarians are hired into institutions with research expectations without the necessary tools and knowledge. this dilemma has a chicken/egg feel about it. if libraries do not specify a requirement for research as part of the job description, then library schools are not going to make it a priority. and without a rigorous research foundation in library school, librarians are ill-prepared to undertake research on the job. librarians in all library sectors are conducting research. it is beneficial to practice and to practicing in an evidence based way.   so, fast forward to librarians out of library school and on the job. professional development  opportunities can include sessions on research methods, the research lifecycle, basic how-to knowledge, and other topics pertaining to research. professional development can help with instilling a research culture in an organization. once research becomes an accepted part of practice, one can hope that library schools will follow suit and offer broader research programming. as well, professional development can help us branch out from just doing what we know. another anecdotal complaint about librarian research is that is it most often done using surveys. and there is some merit to this observation. need to know some stuff? let’s do a survey. or, let’s do a survey. what question can we ask? don’t get me wrong: sometimes a survey is completely appropriate and i don’t want to give the impression that i find that methodology lacking. the process of choosing a research method is driven by the research question asked, and as one colleague pointed out, there are a lot of interesting and appropriate methods we can borrow from other disciplines. ) finding or creating professional development opportunities to learn more about research is going to help librarians doing research in the long run.   6.       develop specific research needs for specific areas of librarianship   many of the above strategies are specifically for the librarians who are also researching. this point calls on associations, groups, and organizations that focus on specific areas of librarianship to talk about what research is needed to move their specialties forward. one colleague talked about defining specific areas of research and around those, developing goals, expectations, and best practices in order to increase quality. there are already groups that are doing that, for example, the medical library association has the research imperative (http://www.mlanet.org/p/cm/ld/fid=545), and a list of topics for future research in cataloguing was released after 2010, the year of cataloguing, to name just two. finding a niche in a specialized research area and then working with others to define and target research topics can move the specialty forward as well as provide collaboration and mentorship opportunities with the goal of making librarian research better.    so, there you have it. just the tip of the iceberg, i’m sure. librarian research will keep on improving the more it is accepted in library culture and the more library schools prepare new graduates for conducting research. the library literature is filled with rigorous and valid research studies, studies that practicing librarians can use to make decisions in practice, to explore new ways to practice, and to inform their own research. as we keep contributing to the library and information studies body of knowledge, we will gain confidence and we will aid and inspire other librarians in practice and in research.   *many thanks to kristin bogdan, karim tharani, christine neilson, and donna frederick for thoughtfully engaging with my question and providing insights in aid of writing this column.    evidence based library and information practice 2011, 7.4 146 evidence based library and information practice news/announcements eplip 7 call for abstracts and conference update 2012. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. it’s full steam ahead for the 7th international evidence based library and information practice conference (eblip7) coming up from july 15-18, 2013 in saskatoon, saskatchewan, canada. the university library, university of saskatchewan is pleased to host the international eblip community with an interesting day of preconference workshops (july 15) and three days of conference sessions, posters, networking, engagement, and fabulous food! the deadline to submit abstracts for papers and posters has been extended until december 21, 2012. research papers, reviews of evidence, reports of innovative practice, hot topic discussion papers and poster presentations are invited. in the spirit of our conference title the possibilities are endless and in line with eblip7’s prairie roots, key themes include:  beyond the horizon visioning the future, investigation, discovery  ploughing new terrain cultivating and enhancing productivity through innovation, experimentation, transforming practice  harvesting the rewards assessing outcomes, impact, value  summer fallow lessons learnt, reflection, insight in other news, the organizers of eblip7 are very proud to announce the keynote speakers for this event. denise koufogiannakis will be the opening keynote speaker on tuesday, july 16 to open the conference. denise is the collections and acquisitions coordinator at the university of alberta libraries. as well, she is a researcher, editor, speaker, and mentor. denise has embraced evidence based library and information practice (eblip) since the very early days of its inception. publishing and speaking on eblip since 2001 and contributing to the lis evidence base through papers and book chapters, denise is a key driver in the international eblip movement. dan gardner will close the eblip7 conference, presenting the closing keynote address on the afternoon of thursday, july 18. dan is an ottawa-based journalist and author of the bestselling books future babble and risk. his writing has won or been nominated for most major prizes in canadian journalism, including the national newspaper award, the michener award, the canadian association of journalists award, the amnesty international canada media award for reporting on human rights, and a long list of other awards, particularly in the field of criminal justice and law. today, he is an opinion columnist who refuses to be pigeonholed as a liberal or a evidence based library and information practice 2011, 7.4 147 conservative and is positively allergic to all varieties of dogma. visit the eblip7 website for complete submission details and the online submission form, as well as the biographies for the keynote speakers: http://eblip7.library.usask.ca/#homepage to keep up to date on all eblip7 news and announcements, follow us on twitter (@eblip7) or join us on facebook: http://www.facebook.com/groups/eblip7/ http://eblip7.library.usask.ca/#homepage http://www.facebook.com/groups/eblip7/ research article   data literacy in the social sciences: findings from a local study on teaching with quantitative data in undergraduate courses   patricia b. condon assistant professor, research data services librarian university of new hampshire library durham, new hampshire, united states of america email: patricia.condon@unh.edu   eleta exline associate professor, scholarly communication librarian university of new hampshire library durham, new hampshire, united states of america email: eleta.exline@unh.edu   louise a. buckley associate professor, social sciences, public policy & government information librarian university of new hampshire library durham, new hampshire, united states of america email: louise.buckley@unh.edu   received: 29 apr. 2022                                                              accepted: 23 nov. 2022      2023 condon, exline, and buckley. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   data availability: due to the small, local sample used in this research study, participants were not asked to agree for their data to be shared publicly, so supporting interview transcripts are not available.   doi: 10.18438/eblip30138     abstract   objective – the university of new hampshire (unh) library conducted an exploratory study of the pedagogical practices of social science instructors at unh who teach using quantitative data in undergraduate courses. this study is connected to a suite of parallel studies at other higher education institutions that was designed and coordinated by ithaka s+r.   the four aims of this study were to explore the ways in which instructors teach and engage undergraduates in the social sciences using quantitative data; understand the support needs of these instructors; develop actionable recommendations for campus stakeholders; and identify opportunities for the development of resources, services, or activities in the library to support the use of quantitative data in the classroom.   methods – for the unh study, the research team recruited eleven participants through convenience sampling for one-on-one, semi-structured interviews. the study sample included lecturers, assistant professors, associate professors, and full professors across seven social science disciplines from the durham and manchester campuses.   results – courses using data provide a unique opportunity for students to gain experience by working with hands-on examples. the two overarching themes identified speak to both the motivations of instructors who teach with data and the challenges and opportunities they face: teaching with data for data literacy and scientific literacy and teaching with data for statistical, data, and tools skill building.   conclusion – data literacy is an important set of competencies in part because of the quality and quantity of data students encounter; they need to have the ability to critically evaluate data, methods, and claims. this study directed attention to an area that had not previously been examined at unh and is an important first step toward creating greater awareness and community of practice for social science instructors teaching with data. the unh library offers research data services and is exploring new ways of supporting data literacy. unh has opportunities to create additional supports for instructors and students that could improve student learning outcomes. such efforts may require cross-college or cross-department coordination as well as administrative support.     introduction   teaching undergraduate students to work with data is essential—not only for handling data in research-intensive fields but also for gaining valuable skills for entering the workforce and as informed members of society trying to interpret news stories and public policy. incorporating data use into the curriculum gives undergraduate students opportunities in the classroom to learn data literacy skills ranging from finding, collecting, and analyzing data to interpreting visualizations to effectively presenting an argument using data. because of data literacy’s corresponding relationship with information literacy, librarians are key stakeholders in conversations to foster data literacy education and instruction efforts across campus.   as the university of new hampshire (unh) library continues to develop its data services and information literacy programs, including support for data literacy, understanding the local practices and needs of instructors can help in making evidenced based decisions about services. additionally, informing campus stakeholders outside the library (such as the office of academic technology, the center for excellence and innovation in teaching & learning, and college deans and department chairs) of potential opportunities for interdisciplinary efforts to enhance instruction in this area can lead to increased awareness of data literacy across campus. knowing more about teaching practices used for incorporating data use into the curriculum can also support improved data literacy instruction and better prepare undergraduates with necessary data-related skills.   to learn more about teaching with data on campus and as part of a larger ithaka s+r research project, librarians at the unh library conducted a study of the pedagogical practices of social science instructors at unh who teach using quantitative data in undergraduate courses. ithaka s+r designed the project and invited participation from libraries. each participating library conducted a local research study using the methodology established by ithaka s+r. in this article, we summarize the findings from the study conducted at unh and discuss how the findings address the aims of the study. this was an exploratory qualitative study that used one-on-one semi-structured interviews. the goal of the study was to better understand instructors’ undergraduate teaching practices related to the use of quantitative data for meeting course learning objectives in the social sciences.   literature review   the literature on teaching with data in the social sciences is substantial and varied in topic. this review focuses on literature that discusses characterizing and fostering data and statistical literacy, instructional strategies for integrating quantitative skills into curricula, and barriers to learning data concepts. a sizable portion of the studies reviewed focus on research methods and statistics courses, often in sociology, although a cross section of social sciences disciplines is represented.   data and statistical literacy   data literacy and statistical literacy are separate but related concepts. they overlap and interact with the broader concepts of information literacy and scientific literacy and with adjacent concepts such as scientific thinking (hosein & rao, 2019), social scientific reasoning, and quantitative literacy (caulfield & persell, 2006). data literacy has typically been described as the ability to understand data and representations of data, to draw and support conclusions, and to evaluate claims, and it may extend to functional use of data such as data collection, analysis, and interpretation (carlson et al., 2011; wolff et al., 2016). in disambiguating statistical literacy and data literacy, wolff et al. (2016) define statistical literacy as assessing the validity of statistics being presented but with an awareness of the processes by which statistics and associated visualization were created. in this way, data literacy informs statistical literacy.   instructional strategies   instructors who teach with data employ an array of pedagogical strategies to foster students’ data literacy skills and knowledge of discipline-specific methodologies. many of these strategies fall into the broad categories of active learning, experiential learning, and student-centred learning that rely on engaged student participation in the learning process. examples include student-led creation of learning materials that can then feed back into the curriculum (clark & foster, 2017), working in collaborative groups that aim to enhance learning through discussion and accomplishing shared project goals (caulfield & persell, 2006; lovekamp et al., 2017), collecting data or analyzing available data (lovekamp et al., 2017), or conducting data projects in partnership with community organizations (nurse & staiger, 2019; wollschleger, 2019). neumann et al. (2013) suggest that the use of real-world data enhances the significance of the course experience for students. additional instructional strategies include using data analysis modules or labs (fellers & kuiper, 2020), surveying members of a course and analyzing the resulting data (brown, 2017), using games (lawrence, 2004), and data mining (hartnett, 2016).   barriers to learning   statistical anxiety and math anxiety are frequently discussed as barriers to learning in courses using data. summarizing the literature that defines these associated phenomena, cui et al. (2019) state that math anxiety has to do with the manipulation of mathematical symbols, while statistical anxiety has to do with understanding the language used to describe and interpret data within statistics. these anxieties are overlapping in definition and tend to coexist in individuals (cui et al., 2019). other studies found statistical anxiety to be related to student perceptions of past math performance, insufficient math background (condron et al., 2018; rode & ringel, 2019), other types of academic anxiety, and self-concept (beliefs about one’s intrinsic abilities) around quantitative skills (faber & drexler, 2019; macarthur, 2020). faber and drexler (2019) also found a connection between statistical anxiety and student beliefs about the usefulness (utility value) of learning statistics, which they suggest can be addressed by emphasizing the practical and vocational application of such skills.   reviews of the empirical evidence find that the negative relationship between statistical anxiety and academic performance may be overstated (macarthur, 2020; ralston et al., 2016) and require additional study to establish causation (filiz et al., 2020). instructors may attribute hesitancy to work with data as the result of insufficient math preparation or anxiety when the issue is a preference for non-mathematical methods (chamberlain et al., 2015). providing math and statistics support, such as drop-in consultations and peer tutoring, can help overcome these anxiety barriers (cantinotti et al., 2017; elbulok-charcape et al., 2019; intepe & shearman, 2020). a survey-based assessment of a project implementing the american sociological association’s and social science data analysis network’s integrating data analysis project modules (see hilal & redlin, 2004) into the curriculum at lehman college identified the removal of barriers, such as insufficient math skills, and a well-sequenced curriculum as key factors in teaching quantitative concepts (wilder, 2010).   aims   this review of the literature highlights that teaching with data in social science courses is a valued pedagogical approach for improving data literacy skills in undergraduates. the unh study investigates local pedagogical practices and instructor needs to inform the augmentation of services on campus. the aims of this study are to   1.      explore the ways in which instructors teach and engage undergraduates in the social sciences using quantitative data; 2.      understand the support needs of these instructors; 3.      develop actionable recommendations for campus stakeholders; and 4.      identify opportunities for the development of resources, services, or activities in the library to support the use of quantitative data in the classroom.   methods   the study undertaken at unh was connected to a suite of parallel studies conducted locally at other higher education institutions. ithaka s+r, a not-for-profit research and consulting organization that supports academic, cultural, and publishing communities, coordinated this multi-institutional effort. ithaka s+r conceptualized and designed the research project “teaching with data in the social sciences,” including developing the methodology and providing local teams guidance on conducting the project at their institutions (cooper, 2019). after obtaining irb approval, the unh research team carried out the research at our institution, including recruitment of participants, data collection and processing, and analysis and interpretation of our dataset to identify local themes (see condon et al., 2021 for a detailed methodology).   data collection   unh is a medium-sized flagship land, sea, and space grant research university that was established as an agricultural and mechanical arts school in 1866. unh enrolls around 15,000 students, with approximately 12,000 undergraduates and 3,000 graduate students across three campuses: durham, manchester, and the unh franklin pierce school of law in concord. the college of liberal arts (cola) on the durham campus houses most of the social science disciplines; however, there are social science disciplines represented in all the other colleges and on the manchester campus. cola has the largest enrollment of undergraduates with over 3,100 enrolled in spring of 2022.   the study sample at unh, recruited through convenience sampling, included eleven lecturers, assistant professors, associate professors, and full professors across seven social science disciplines on two campuses who engaged in teaching undergraduate students to work dynamically with quantitative data (e.g., collecting data through social science research methods, finding existing data to address a research question, using software tools to analyze data, and drawing conclusions from data). we conducted and recorded one-on-one interviews online via zoom, a video-conference software, during the fall of 2020. all research team members were trained on the informed consent process and interview guide to ensure consistency across interviews. recorded interviews were transcribed by a third party and de-identified by the research team.   data analysis   after the interviews were transcribed and de-identified, the research team conducted qualitative coding of the transcripts based on a coding process using grounded theory methodology that was recommended by ithaka s+r (strauss and corbin, 2014). using the qualitative analysis software nvivo, coding and analysis were done through an iterative process (see figure 1). all three team members conducted initial open coding on the same set of three interview transcripts to identify emergent codes in the data (phase 1). the team discussed and compared the initial codes, selected several core themes that emerged from the open coding, and determined a final set of focused codes (phase 2). once coding of all interviews was complete, the team identified overarching themes that emerged from the focused coding (phase 3) and used these to address the aims of the study (phase 4).    figure 1 qualitative coding process used by unh research team.   the following focused codes developed in phase 2 were used to analyze all transcripts:   ·        learning objectives. comments and reflections about learning objectives instructors have defined as desired outcomes for their students within courses.  ·        challenges understanding data. comments and reflections about challenges students experience in understanding data concepts and working with data.  ·        student prior learning. comments and reflections about expected preparation at the high school level or college level prior to the data course; student skills or viewpoints brought to the data course; perceived student anxiety about math or science; perceived student abilities with mathematical concepts or skills; student motivation; and perceived student challenges with basic software, technical familiarity, and access.  ·        locating and providing data for use. comments and reflections about finding datasets for use in teaching, qualities of data that instructors look for, common sources for usable data, and challenges of teaching with data.  ·        support outside the classroom (for both students and instructors). comments and reflections about where students go for support regarding data-related questions or needs outside the classroom. comments and reflections about professional development for instructors around teaching with data and learning new methodologies and tools.    results   from the focused coding, two overarching themes emerged that spoke to the motivations of instructors for teaching with data and the challenges and opportunities they face. table 1 presents the themes and subthemes that we derived from the focused coding.   table 1 themes and subthemes that emerged from the coding phase 3: themes phase 3: associated subthemes phase 2: focused codes from which themes/subthemes were derived teaching with data for data and scientific literacies students as consumers of data students interpreting data students learning the scientific method learning objectives challenges understanding data teaching with data for statistical, data, and tool skill building students working with data instructors providing data for use sources of support on campus learning objectives challenges understanding data student prior learning locating and providing data support outside the classroom   teaching with data for data and scientific literacies   the theme “teaching with data for data and scientific literacies” represents a desire among participants to introduce or strengthen a broad set of foundational skills that they believe students need to be successful. these skills include a wide range of competencies connected to critical thinking, from essential information and data literacies to understanding scientific methodologies and their underpinnings. this theme has three subthemes: students as consumers of data, students interpreting data, and students learning the scientific method.   students as consumers of data   although data, information, and scientific literacy skills development are not always explicit learning objectives in courses using data, they are addressed by most participants because of their importance as life skills and as foundations for more explicit learning objectives around scientific thinking and disciplinary research methods. when participants do emphasize learning data literacy among their implicit goals for a course, this is often expressed as helping students to “be savvier consumers” of data of all kinds (participant 05).   one driver for including data literacy as course objectives is the shared perception among participants that there is a flood of low-quality online representations of data and that much of what students will encounter in their everyday lives is, as one participant bluntly stated, “complete garbage” (participant 06). another participant mentions that “depending on where they're coming from in life, [students] may or may not have had any life experience to really give them a foundation to think about data from” (participant 09). as savvy consumers of data, students need to be equipped to assess the validity of claims. they need to be able to recognize misleading or false claims and to identify claims that might be intentionally deceptive, are based on faulty reasoning, or have poor methodology. conversely, students should be able to recognize sound methodology where the arguments and conclusions presented are supported by the data cited.   students interpreting data   there is a sense of urgency around helping students navigate the challenges of an information environment in which accuracy is sometimes secondary to messaging and “the difference between opinion and argument” can be difficult to discern (participant 07). students need to know enough about data to begin to ask interrogative questions, identify biases, and recognize misrepresentations:   are there other data sources that might tell a different story? or is there something about the way this data is being presented that biases the presentation toward a certain type of conclusion? . . . i want them to ask those kinds of questions of me. and then, also, of themselves, as they work with data. because we’re being bombarded all the time with information, and oftentimes with just conclusions and statements about this is the way things are (participant 01).   data literate students should be equipped to question the data and claims they encounter and to understand that data “doesn’t tell a story by itself” (participant 01), but is interpreted, analyzed, and presented by people. students also need a level of data proficiency to progress to more advanced courses, to be members of the workforce in which those skills are increasingly important, and to be “a good citizen” (participant 01) in a participatory democracy in which even accurately presented data can be used to tell conflicting stories.   students learning the scientific method   many courses using data have explicit learning objectives to introduce students to disciplinary research methods. students transition from data literacy as academic and life skills to contextualizing these skills in an understanding of the scientific method and social science disciplinary research methodologies, setting the stage for deeper learning about methods and knowledge building. to this end, participants use data as a tool to help acclimate students to scientific thinking:   it's a very important part of the process of helping students understand what scientists do, and what people who are real researchers do . . . it’s really emphasizing the use of data and empirical knowledge to make sense of what we see (participant 11).   the scientific method and its core principle of using empirical evidence to substantiate arguments is a different way of learning about the world than students may have encountered previously. it may even bump against other ways of learning that students have internalized, such as those with an arts and humanities focus where “the whole scientific method is sort of not their ballpark” (participant 05), and they may have developed a “pattern of learning” (participant 02) that works against their interpretation of numeric data, presumably a pattern based on textual rather than numeric analysis. additionally, popular notions on how to conduct research, such as participation in informal polls and surveys, may cloud students’ understanding of methods: “they’ve grown up in a world, unfortunately, where every fool with a modem and internet connection does what they call a survey, there is so much misperception about how to do survey research. i think that's really damaging” (participant 02).   teaching with data for statistical, data, and tool skill building   while the previous theme focused on contextualizing motivations for teaching with data in terms of a holistic understanding of data and scientific literacies, the theme “teaching with data for statistical, data, and tool skill building” focuses on working with datasets and the development of skills such as understanding statistical concepts, variables and measurements, and tools for working with data. participants expressed working with data as a continuum of concepts and skills that build upon one another. students tend to move from a basic understanding of variables —how variables are measured and how data is structured—to interpreting graphic visualizations of data and conducting and interpreting statistical analyses. finally, they move to providing a written interpretation of findings and their significance. this theme has three subthemes: students working with data, instructors providing data for use in the classroom, and sources of support on campus.   students working with data   the participants emphasized the challenges that students face when working with data. as noted by one participant, “i always have to try to remember how unfamiliar they are with using data” (participant 07). students struggle for a variety of reasons, including difficulty with or anxiety around math skills, lack of experience with or lack of retention of math or statistical concepts, obstacles learning to use analysis software, or problems specific to how data is structured and manipulated. students in a course are unlikely to have uniform knowledge or exposure to data concepts. this diversity of experience results in what one participant described as a “heterogeneous knowledge base” (participant 11) within a single course that makes teaching course content at the appropriate level more difficult. in some cases, software serves a pedagogical role in helping students practise analytical concepts; in other cases, participants consider working knowledge of software packages as transferable job skills.   a common theme from participants was about students’ math anxiety or difficulty with math skills. prior experiences around learning math in k-12 can lead some students to develop a mindset that they are not going to be good at math or science:   so, technical challenges [are] one aspect, obviously, of it. i feel like, depending on their comfort level with statistics, depending on their comfort level with math and with numbers, there’s a level of anxiety that goes with it. that, they see a lot of numbers and just freak out (participant 10).   it is each student’s “own math ability, understanding, and their comfort level with numbers that plays a role” (participant 10). participants viewed math anxiety as a barrier for students to overcome to gain a positive outcome that will be helpful after graduation. participants described students as being capable of working with mathematical concepts, statistics, and data but needing to overcome the mindset of not being able to.   instructors providing data for use   carryover of learning from one course to another and retention of knowledge by students was described by participants as inconsistent as it pertained to basic statistics or specific software. this lack of retention can impact the scaffolding of learning objectives; students’ tentative grasp of data concepts can lead to struggles with higher-order tasks such as data analysis, interpretation, and application of findings. in turn, providing data for students allows them to focus on specific skills or learning outcomes. if the learning objectives for the course do not include data collection or data processing, then including those activities distracts students from concentrating on data analysis and interpretation of findings. as one participant explained:   i . . . bring them data that’s already cleaned . . .. it makes it a lot easier . . .. then they can start to get into what the story is with the data, rather than thinking about, . . . what do you mean there are missing cases? . . . i think [the higher-order concerns] just throw them for a loop (participant 02).   while providing data required participants to spend time locating and preparing data prior to the start of the course, it saved time for the students. among our participants, it was most common to find students engaging in data collection or acquisition and data processing in research methods courses or courses concerning the scientific process. in some courses, locating data from instructor-vetted sources was required. but in courses where students collected or located their own data for use, those skills were tied to course learning objectives.   sources of support on campus   to address challenges faced by students who struggle with math, statistical software, and understanding data, participants noted limited support available outside the classroom. the most common support participants mentioned that was available for their students were graduate assistants, teaching assistants, and lab instructors. many participants expressed that they are “lucky enough to have a grad research assistant” (participant 03) as additional support for their courses, someone who could provide one-on-one attention to students who need extra help. most participants were unaware of other resources that their students might use outside of those that the instructor or course provided. although participants mentioned video tutorials as options, they warned against advanced tutorials (such as those from software vendors) that provide more detail than students require and likely only lead to further frustration and confusion.   discussion   our review of the literature suggests that social science instructors use a range of pedagogical strategies to teach data and statistical literacy concepts, but they encounter math and statistical anxiety as significant barriers to learning. our study supports these earlier findings and expands on why unh instructors see data concepts as essential to student academic and life success and what strategies for overcoming barriers to student learning they employ in the classroom. in this section, we discuss how the findings address the aims of the study including a discussion of evidenced based actions to support data instruction and learning.   our study findings directly address the first two of our study aims: (1) explore the ways in which instructors teach and engage undergraduates in the social sciences using quantitative data and (2) understand the support needs of these instructors. social science instructors teach with data in the classroom to support both general and course-specific learning outcomes that focus on building data and scientific literacies and skill building. the participants expressed challenges that students face working with data and how they, as instructors, mediate these challenges though course design and navigate the minimal support on campus. depending on a course’s learning objectives and content, participants concentrate on different combinations of analytical, conceptual, and technical skills.   table 2 recommended actions for local stakeholders based on findings recommended actions rationale     provide support for instructors in locating and sharing data for teaching identifying, locating, and cleaning datasets that are appropriate for students to use can be time consuming, and the findings suggest that instructors often provide datasets for students to analyze so that specific data literacy skills can be targeted. provide support for students who need extra help with math and statistics the findings suggest that instructors have some concern about the math skills and retention of their students and that this has the potential to interfere with understanding data concepts and statistical analysis concepts. while unh, like many campuses, has a centralized writing centre there is no general tutoring for math or specific support that caters to students working with math or statistics in social science contexts. provide enhanced software support for both students and instructors sometimes developing proficiency with a tool or software is a learning objective; other times the tool supports the learning objectives. decisions around which tool to use may be based on criteria such as type of data, user preference, pedagogical purpose, or disciplinary practice. provide learning opportunities for instructors on teaching with data, student skills retention, and new research methodologies and data analysis and visualization practices at unh, like many other universities, the center for excellence and innovation in teaching & learning already provides resources and professional development opportunities for best practices in teaching. librarians can build partnerships with teaching and learning centres to help expand opportunities that focus on teaching with data. enhance library support for teaching with data and foster partnerships with campus stakeholders to explore these collaborative actions support needed by social science instructors cannot be addressed by a single campus stakeholder. for many of the collaborative actions, the library’s role is as partner; however, there are areas for which the library can provide leadership or build on existing support for data-related activities.   courses using data provide a unique opportunity for students to gain experience by working with hands-on examples. there was a strong thread throughout many of the interviews associated with students learning to work with datasets and to engage with data and scientific literacies through application of concepts—learning by doing. this aligns with a comment made by one participant about the difference between understanding due to reading or listening and the deeper comprehension resulting from the actual experience of doing and practicing.   based on the findings, we propose five evidence based collaborative actions that local stakeholders can take to better support social science instructors teaching with data (see table 2). these proposed actions address the last two aims of our study: (3) develop actionable recommendations for campus stakeholders and (4) identify opportunities for the development of resources, services, or activities in the library to support the use of quantitative data in the classroom. in many cases, these are opportunities for stakeholders to collaborate or partner with one another and with the library. most build on or extend the support already provided by the library and other units on campus.   the study we conducted at unh is both a stand-alone project and part of a larger project that included 19 other institutions conducting local versions. while the unh study sheds light on teaching with quantitative data in the social sciences at unh, the findings and evidenced based actions are potentially transferable to other settings. additional insight into this topic at the local level can be found in the other local reports available in participating institutions’ institutional repositories. ithaka s+r prepared a consolidated report that analyzes interviews across all 20 institutions as a single dataset (ruediger et al., 2022). findings from that publication report on high-level themes identified across institutions.   limitations   there are some limitations to the methods used in this study. while the sample size for our local study was small and consisted of self-selecting participants, this is appropriate for an exploratory study. it is worth noting that “this study does not purport to be statistically representative nor are the recommendations meant to be prescriptive; rather, the report and its recommendations are intended to be suggestive of areas for further investigation” (ithaka s+r, n.d.). the data from this local project is included in and complemented by the capstone report from ithaka s+r that provides an aggregated analysis of interviews conducted at 20 institutions. this broader analysis provides additional perspective and context for this local study and mitigates the limitation of its small size. another limitation of this study is that the focus was on undergraduate social science courses using quantitative data. including graduate courses as well as the use of qualitative data would have provided a more holistic look at data literacy and teaching practices in social science courses. future work involves exploring these areas as well as courses beyond the social sciences.   conclusion   this exploratory study investigated the teaching practices of social science instructors at unh who engage with undergraduate students using quantitative data in the classroom. the participants we interviewed teach both general and discipline-specific data concepts as academic, work, and life skills. primary challenges discussed by the participants that students face in engaging with these topics are understanding math and statistical concepts, learning new software and computing skills, limited prior exposure to data, and lack of retention of content from earlier courses. participants addressed challenges in several ways in order to lower barriers to learning, including finding, vetting, and cleaning data for their students to use. participants could use additional support and new strategies to alleviate student challenges, and we presented recommended actions based on the findings of this study. data literacy is an important set of competencies in part because of the quality and quantity of data students encounter in their academic, work, and daily life; they need to have the ability to critically evaluate data, methods, and claims. this study directed attention to an area that had not previously been focused on at unh and is an important first step toward creating greater awareness of the challenges of teaching with data and creating opportunities for building a community of practice for social science instructors grappling with these issues. unh has opportunities to create additional supports for instructors and students that could improve student learning outcomes. in addition to library partnership, such efforts may require cross-college or cross-department coordination as well as administrative support.   funding statement   funding for this project was provided by the unh library.   acknowledgements   the researchers extend a sincere thank you to the study participants, each of whom was generous with their time and provided thoughtful remarks. the researchers also thank ithaka s+r and their staff, especially dylan ruediger and danielle cooper, for the high level of support provided to the local teams throughout the process, which, like so much else, was impacted by the pandemic.   author contributions   patricia condon: investigation (equal), formal analysis (equal), project administration (lead), writing – original draft (equal), writing – review & editing (equal) eleta exline: investigation (equal), formal analysis (equal), writing – original draft (equal), writing – review & editing (equal) louise buckley: investigation (equal), formal analysis (equal), writing – original draft (equal), writing – review & editing (equal)   references   brown, m. (2017). making students part of the dataset: a model for statistical enquiry in social issues. teaching statistics, 39(3), 79–83. https://doi.org/10.1111/test.12131   cantinotti, m., désormeaux-moreau, m., & balbinotti, m. (2017). mapping psychology students’ perspective on group peer-tutoring in statistics. mentoring & tutoring: partnership in learning, 25(4), 466–484. https://doi.org/10.1080/13611267.2017.1403609   carlson, j., fosmire, m., miller, c., & nelson, m. s. (2011). determining data information literacy needs: a study of students and research faculty. portal: libraries and the academy 11(2), 629–657. https://doi.org/10.1353/pla.2011.0022   caulfield, s. l., & persell, c. h. (2006). teaching social science reasoning and quantitative literacy: the role of collaborative groups. teaching sociology, 34(1), 39–53. https://doi.org/10.1177/0092055x0603400104     chamberlain, j. m., hillier, j., & signoretta, p. (2015). counting better? an examination of the impact of quantitative method teaching on statistical anxiety and confidence. active learning in higher education, 16(1), 51–66. https://doi.org/10.1177/1469787414558983   clark, t., & foster, l. (2017). ‘i’m not a natural mathematician’: inquiry-based learning, constructive alignment and introductory quantitative social science. teaching public administration, 35(3), 260–279. https://doi.org/10.1177/0144739417711219   condon, p., exline, e., & buckley, l. (2021). teaching with quantitative data in the social sciences at the university of new hampshire: an ithaka s+r local report. unh scholars repository. https://dx.doi.org/10.34051/p/2021.39     condron, d. j., becker, j. h., & bzhetaj, l. (2018). sources of students’ anxiety in a multidisciplinary social statistics course. teaching sociology, 46(4), 346–355. https://doi.org/10.1177/0092055x18780501   cooper, d. m. (2019, july 29). announcing two new s+r projects on supporting data work. ithaka s+r blog. https://sr.ithaka.org/blog/announcing-two-new-sr-projects-on-supporting-data-work/   corbin, j., & strauss, a. (2014). basics of qualitative research: techniques and procedures for developing grounded theory. sage publications.   cui, s., zhang, j., guan, d., zhao, x., & si, j. (2019). antecedents of statistics anxiety: an integrated account. personality and individual differences, 144, 79–87. https://doi.org/10.1016/j.paid.2019.02.036   elbulok-charcape, m., grandoit, e., berman, l., fogel, j., fink, l., & rabin, l. (2019). improving exam performance in an undergraduate statistics course for at-risk students through peer tutoring. psychology teaching review, 25(2), 3–17.   faber, g., & drexler, h. (2019). predicting education science students’ statistics anxiety: the role of prior experiences within a framework of domain-specific motivation constructs. higher learning research communications, 9(1). https://eric.ed.gov/?id=ej1221253   fellers, p. s., & kuiper, s. (2020). introducing undergraduates to concepts of survey data analysis. journal of statistics education, 28(1), 18–24. https://doi.org/10.1080/10691898.2020.1720552   filiz, m., early, e., thurston, a., & miller, s. (2020). measuring and improving university students’ statistics self-concept: a systematic review. international journal of educational research open, 1, article 100020. https://doi.org/10.1016/j.ijedro.2020.100020   hartnett, j. l. (2016). using data mining to teach applied statistics and correlation. teaching of psychology, 43(2), 142–146. https://doi.org/10.1177/0098628316636292   hilal, s. m., & redlin, m. m. (eds.). (2004). integrating data analysis: instructional materials for integrating data analysis into sociology courses. american sociological association.   hosein, a., & rao, n. (2019). the acculturation and engagement of undergraduate students in scientific thinking through research methods. in m. murtonen & k. balloo (eds.), redefining scientific thinking for higher education: higher-order thinking, evidence-based reasoning and research skills (pp. 157–175). springer international publishing. https://doi.org/10.1007/978-3-030-24215-2_7   intepe, g., & shearman, d. (2020). developing statistical understanding and overcoming anxiety via drop-in consultations. statistics education research journal, 19(1), 149–166. https://iase-web.org/ojs/serj/article/download/127/35   ithaka s+r. (2020). teaching with data in the social sciences ethics review instructions. [unpublished internal documentation].   lawrence, r. (2004). teaching data structures using competitive games. ieee transactions on education, 47(4), 459–466. https://doi.org/10.1109/te.2004.825053   lovekamp, w. e., soboroff, s. d., & gillespie, m. d. (2017). engaging students in survey research projects across research methods and statistics courses. teaching sociology, 45(1), 65–72. https://doi.org/10.1177/0092055x16673136      macarthur, k. r. (2020). avoiding over-diagnosis: exploring the role of gender in changes over time in statistics anxiety and attitudes. numeracy: advancing education in quantitative literacy, 13(1), 1–26. https://doi.org/10.5038/1936-4660.13.1.4   neumann, d. l., hood, m., & neumann, m. m. (2013). using real-life data when teaching statistics: student perceptions of this strategy in an introductory statistics course. statistics education research journal, 12(2), 59–70. https://doi.org/10.52041/serj.v12i2.304   nurse, a. m., & staiger, t. (2019). teaching data reproducibility through service learning. teaching sociology, 47(4), 350–357. https://doi.org/10.1177/0092055x19860577     ralston, k., macinnes, j., crow, g., & gayle, v. j. (2016). we need to talk about statistical anxiety. a review of the evidence around statistical anxiety in the context of quantitative methods pedagogy [working paper]. ncrm. https://eprints.ncrm.ac.uk/id/eprint/3987/   rode, j. b., & ringel, m. m. (2019). statistical software output in the classroom: a comparison of r and spss. teaching of psychology, 46(4), 319–327. https://doi.org/10.1177/0098628319872605   ruediger, d., cooper, d. m., bardeen, a., baum, l., ben-gad, s., bennett, s., berger, k., bonella, l., brazell, r., bruce, s., buckley, l., burrows, t., calvert, s., condon, p., curty, r., davis, h., exline, e., feerrar, j., finch, e., . . . yatcilla, j. (2022). fostering data literacy: teaching with quantitative data in the social sciences. ithaka s+r. https://doi.org/10.18665/sr.317506   wolff, a., gooch, d., montaner, j. j. c., rashid, u., & kortuem, g. (2016). creating an understanding of data literacy for a data-driven society. the journal of community informatics, 12(3), pp. 9–26. https://doi.org/10.15353/joci.v12i3.3275   wollschleger, j. (2019). making it count: using real-world projects for course assignments. teaching sociology, 47(4), 314–324. https://doi.org/10.1177/0092055x19864422 research article   swimming upstream in the academic library: exploring faculty needs for library streaming media collections   elsa loftis humanities and acquisitions librarian, assistant professor portland state university portland, oregon, united states of america email: eloftis@pdx.edu   carly lamphere science librarian reed college portland, oregon, united states of america email: lampherec@reed.edu   received: 2 feb. 2023                                                                 accepted: 23 aug. 2023      2023 loftis and lamphere. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30317     abstract   objective to compare portland state university’s (psu) local experience of using streaming media to national and international trends identified in a large qualitative study by ithaka s+r. this comparison will help librarians better understand if the psu library is meeting the needs of faculty with its streaming media collection through a series of faculty interviews.   methods and intervention two librarians from psu participated in a large, collaborative, two-part study conducted by ithaka s+r in 2022, with 23 other academic institutions in the united states, canada, and germany as part of this study, the authors conducted a series of interviews with faculty from psu’s social work and film studies departments to gather qualitative data about their use, expectations, and priorities relating to streaming media in their teaching. ithaka s+r provided guided interview questions, and librarians at psu conducted interviews with departmental faculty. local interview responses were compared to the interviews from the other 23 institutions.   results psu library had a higher rate of faculty satisfaction than in the larger survey. discussions raised concerns around accessibility of content, which was novel to psu, and did not meaningfully emerge in the broader study. local findings did line up with broader trends in the form of concerns about cost, discoverability, and lack of diverse content.    conclusions the data collected by ithaka s+r’s survey, which was the first part of their two-part study, is useful as it highlights the trends and attitudes of the greater academic library community. however, the second portion of the study’s guided interviews with campus faculty reinforced the importance of accessibility, the library’s provision of resources, and the relationships between subject liaisons and departmental instructors. it emphasized that portland state university’s library has built a good foundation with faculty related to this area but has not been able to provide for every streaming instructional need. reasons for this include limited acquisitions budgets, constraints of staff time, and market factors.     introduction   in 2021 and 2022, two librarians from portland state’s university’s millar library participated in a study facilitated by ithaka s+r “making streaming media sustainable for academic libraries” to identify emerging streaming media trends and needs on academic campuses. portland state university (psu) is a public urban university in oregon, with approximately 23,000 enrolled students, and is an r2 doctoral university with 201 degree programs. psu employs 1,690 research and instructional faculty (portland state university, 2022). ithaka s+r is a not-for-profit organization dedicated to helping academic and cultural communities navigate issues in higher education. their research projects are designed to “generate action-oriented research for institutional decision-making and act as a hub to promote and guide collaboration across the communities we serve” (ithaka s+r, 2022).   ithaka s+r’s project consisted of two phases focused on making streaming media more sustainable for academic libraries. the first phase was a u.s. and canada-wide survey sent to 1,493 individuals by invitation from ithaka s+r. this survey assessed and evaluated the competitive landscape of streaming media licensing for libraries. the results of that survey were shared with the investigators from the 24 participating libraries at the cohort-wide meeting in fall 2021, and then published widely (cooper et al., 2022).    phase two of the research project was a series of interviews conducted by librarians from the 24 college campuses. this article focuses on interview findings from portland state university’s faculty. two investigators from psu’s millar library conducted 10 interviews with faculty in the film studies and social work departments in the winter/spring of 2022.   ithaka s+r’s collaborative research project was an opportunity to survey faculty at psu about streaming media preferences in instruction to better understand how the library was meeting service and support needs. in turn, psu also contributed to developing strategies for libraries as they continue to navigate the complex landscape of streaming media in higher education. broadly, the survey reinforced the preliminary assumption that streaming media is increasingly important in academic library collections, both before and during the covid-19 pandemic. the survey feedback also revealed how faculty across different institutions incorporated streaming media into instruction. reviewing psu faculty’s interviews provided the library with information regarding how to tailor outreach and other services to better assist faculty and students accessing streaming media in their courses.   psu’s library was awarded the reimagine psu grant to participate in the ithaka s+r project. the grant emerged as an effort to transform campus services to better serve students during a time of pandemic transition. streaming services in instruction and access during the pandemic continues to be an equity issue, as many students do not have reliable high-speed internet or the devices required to access content in this way. learning more about the needs of our faculty and students will inform the library on how to better meet those challenges.   literature review   streaming media usage has risen on campuses nationwide, as documented in recent library literature and demonstrated on psu’s campus (wang & loftis, 2020). more recently, according to a survey by tanasse (2021), 96.7% of responding libraries offer streaming media. academic libraries grew accustomed to incorporating streaming films into their selection, acquisition, cataloging, and budgetary workflows for several years, albeit with variations in a climate of consistent change.   during the public health crisis of covid-19, many libraries restricted access to physical collections and universities rapidly shifted toward remote learning where possible (grove, 2021). this exacerbated the already growing demand for the streaming format that outpaced library acquisitions budgets (lear, 2022). regardless of whether libraries can afford continued subscription and license renewals indefinitely, it became clear that a “preference” for streaming media has evolved into “necessity,” based on trends in media consumption (lear, 2022), as well as instructors’ pedagogical aims and instructional realities. in addition to the growth of online courses, reasons for the new dominance of streaming include use of film to accommodate multiple learning styles, instructors adopting “flipped classroom models,” and physical media players becoming increasingly obsolete (adams & holland, 2017).   in addition to budgetary hardships and workflow complications for libraries, students experience a variety of difficulties with accessing and utilizing streaming media. covid-19 further underscored the problems of the digital divide in the united states, which is particularly significant for rural residents who struggle with readily available speedy internet service that can keep pace with these resources (lai & widmar, 2020). other variables for student access depend upon video quality, presence or absence of subtitles, closed captions, or audio description (peacock & vecchione, 2019).   in addition to addressing the digital divide exacerbated by the covid-19 pandemic, librarians and faculty members recognize the importance of developing and expanding their accessibility standards for streaming media collections. some libraries have begun to adopt policies of universal design for learning to increase digital equity and inclusion in collection access. this ranges from lending wi-fi hotspots, expanding library remote services, and most importantly, purchasing library materials that are accessible for all users (frank et al., 2021). while there are outlined recommendations for these practices from the national digital inclusion alliance (ndia), there are few recommendations for streaming media specifically beyond encouraging libraries to ensure streaming titles are closed captioned and offer transcripts. a last recommendation is relying on campus/institution accessibility services to help caption and make other media accessible when licensed content does not include closed captions or transcripts (frank et al., 2021).    it can be challenging to procure films that have accessibility features if they are not available on the market. the authors otherwise found a paucity of studies regarding specific streaming accessibility best practices, a gap which should be further explored in library literature.   in addition to exploring more specific accessibility needs for streaming media, libraries also noted the importance of surveying their faculty and students when making streaming media collection decisions. there have been several focus group research studies in recent years to assess and inform librarians making streaming media collection decisions such as developing collection decision workflows (vanullen et al., 2018) and usability preferences (hill & ingram-monteiro, 2021). once assessed, libraries strive to make collection decisions that inform their unique user populations and preferences for streaming media and share these focus group/survey models for future research. in a study by beisler et al. (2019) it is evident that students express a need for “streaming content that was credible and appropriate for academic purposes,” while faculty generally are concerned with content being discoverable, and reliable. while strides towards best practices have been made for accessibility, there are still gaps in universal best practices to collect this format. in the meantime, libraries must create boutique policies and approaches to try to satisfy the needs of their unique user populations.    in 2021, ithaka s+r launched a survey of academic libraries and released its report in june of 2022, which reinforced the importance of streaming media in library collections. a total of 96% of librarians surveyed said the “impact on instruction is the number one factor shaping library decision making in purchasing and renewing streaming licenses” above even cost considerations. another notable finding from the survey is that academic institutions were already heading this direction before the covid-19 pandemic accelerated the immediate need for streaming media (cooper et al., 2022). the study goes on to reveal that nearly half of the librarians surveyed (42%) strongly agree that demand for streaming media has increased since march of 2020, yet only 23% of librarians at doctoral institutions strongly agree that their library’s strategy around streaming media licensing has changed (2022). in summary, while costs and demand rise, the collective profession has not made a meaningful response in terms of reimagining how we select and acquire this format.   methodology   prior to conducting the interviews with faculty, both investigators participated in two training sessions with staff from ithaka s+r and the other project participants. the training consisted of sharing the interview materials such as the script (appendix a) with questions and tips on how to conduct a research interview, and built-in time to practice with other participants. the investigators submitted paperwork for irb approval for participation in the project and were granted human research protection program (hrpp) exemption prior to the faculty interviews.   the investigators chose to select faculty from film studies and social work departments due to their consistent need for streaming services in their instruction. a total of 10 faculty participants were recruited, five from each department. an initial call for volunteers was sent to the departments via email, and the librarians selected faculty that they knew were frequent requestors for media content. of the faculty approached, there was a 90% acceptance rate to participate in the interviews. of these, 50% had indefinite tenure at the institution, while the remaining participants were either at the rank of assistant professor, adjunct faculty, or at the rank of instructor. the anonymity of the conversations enabled the faculty to be candid in their remarks, and the researchers believe that this is why adjunct faculty did not appreciably respond much differently than full professors, for example. future studies should include other disciplines, but the librarians felt that social work and film studies were excellent choices due to their high use of streaming content, and their similarities and differences incorporating streaming media into their instruction.    each interview was conducted and recorded over zoom with auto transcription enabled to provide a starting point for packaging the final transcript sent to ithaka s+r in the spring of 2022 for analysis and inclusion in their final report. investigators reviewed the auto-generated transcript to make any necessary corrections, and de-identified the faculty to ensure anonymity in the broader study. researchers at ithaka s+r did the final coding and analysis as they compiled interviews from all 24 participating academic institutions (macdougall & ruediger, 2023).    psu’s interviews were reviewed by the librarians and put into a spreadsheet for general comparative analysis (appendix b). this provided faculty reactions at a glance, which enabled easier comparison between psu’s experience and the findings from the larger collected survey, which included 244 total interviews from faculty in a wide range of disciplines.   results   the interviews illustrate how psu’s faculty view and utilize streaming media and library services in their teaching practice. faculty were able to express their frustration and satisfaction with the current model of streaming services at millar library. some feedback varied between disciplines, and yet there were also commonalities across both social work and film departments.   social work faculty utilized streaming media in their instruction as supplemental material, a practical demonstration tool for concepts introduced in their courses, and as instruction materials for clinical practice courses. film faculty utilize streaming content as their base text for analysis and demonstrate properly executed technical production skills. faculty in both disciplines taught in different environments: online, hybrid, and in person.   the themes from the portland state university interviews are discussed below and are separated into the following sections: accessibility challenges, discovery, and cost containment. the valuable insight they offered also underscores the importance of faculty/librarian relationships, which was also spoken about in many interviews.    accessibility challenges   one of the most surprising insights offered during the faculty interviews was a heightened awareness of student accessibility. this concern was more pronounced among the psu faculty in comparison with the broader national study from other participating institutions, where these concerns were not meaningfully explored. accessibility is often conflated with discoverability and whether a film ‘can be accessed’, not specific concerns around usability. while generally satisfied with captioning services, faculty are interested in reimagining the current model of captioning services within the disability resources center (drc) department at psu. faculty want captioning as a basic feature of streaming media content and not only for students who have formal disability accommodations. many students who do not formally meet the requirements for submitting accessibility requests still benefit from captions due to a variety of different learning styles.  as one faculty member expressed: “i think it addresses a lot of learning styles when you have captioning and it shouldn’t be where we have to have a person that has to get an accommodation that you know, maybe 50% of a class would benefit from, but they don’t have accommodations. it's just that it works better to have the audio and the captioning at the same time.” faculty expressed how using streaming services helped to support different learning styles and increased focus on accessibility challenges would naturally follow suit. the library has built this concern into all of its purchasing, but gaps occur when faculty use streaming media that is not licensed by the library.   some faculty caption media themselves because they cannot rely on the auto captioning offered by some platforms. this includes media clips created by faculty members themselves to support instruction: one faculty member stated: “sometimes it’ll take me like 45 minutes to make two small clips…so you finally have these two clips. it’s like 10 minutes of content you share and then you find out ‘oh this doesn’t need to be captioned because it’s a foreign film it’s subtitled’ and i’m like ‘oh, but it's not like captioned-captioned.’” it is a time-intensive endeavor for faculty members already stretched thin with teaching and research.   due to the number of streaming platforms, it is common to encounter a variety of delivery options. a film might be viewed on a vendor platform, emailed in the form of an mp4 file to host on the library’s local platform, or mailed as a dvd that staff are to digitize. this can lead to complications in workflow for acquisitions and cataloging staff, but the important disparity is the lack of uniformity in accessibility options.    some vendors consistently offer closed captioning and transcripts with their films. some vendors supply the library with different versions of the film with audio descriptions for visually impaired viewers, but this is rare. non-english dialog is sometimes subtitled, but not always, and the variety of available languages of subtitle options is limited. the library is often limited by what is available since any given film is commonly only available for institutional purchase by one vendor. as noted by beisler et al. (2019), captions are a great pedagogical value to faculty for these reasons, as well as supplementing poor audio quality, and for students who might be viewing films in loud environments.    ultimately, when discussing whether a streaming film is ‘accessible’ the conversation is normally about whether closed captions are provided. this overlooks audio description services that assist learners with visual impairments, which is a rarity on the market, and therefore not often library provided. another aspect of accessibility is whether transcripts are provided, which again varies by vendor. also of concern is the relative accessibility of the hosting site where viewers may be searching for and accessing films, libraries should ensure that they are user friendly, able to be read by a screen reader, and so forth.    the fact that the library is sometimes pressed to refer students and researchers to outside streaming services (commercial, open web) means that it is not always able to assess or control the accessibility standards of the content students consume. the library only controls its own provisions, in compliance with the university’s standard for accessible digital procurement, (portland state university, 2023), ensuring acceptable accessibility levels. ultimately, the library’s ability to provide quality resources that are accessible and desirable to faculty and students hinges on whether our community is indeed utilizing library resources. when instructors and students are compelled to use sources not provided by the library, important considerations such as these can potentially be unaddressed.    discovery   psu faculty’s interview responses echoed that of faculty at the other 23 institutions represented in the ithaka s+r study. issues around discoverability were raised, as busy faculty often use the open web to find films before going to the library catalog or databases. the findings from ithaka suggest that youtube is the most common source of streaming films utilized by faculty. psu’s faculty disputed this to some extent, with 50% of surveyed faculty naming youtube or google specifically when asked where they search for content, whereas 80% cited the library’s catalog or subscription databases.    many faculty in the national study described library catalogs and databases as “confusing”, “unreliable”, and “impossible to navigate”. the ithaka report conveys “personal connections with librarians were fundamental to faculty satisfaction with library resources” (macdougall & ruediger, 2023). this was also the case with psu’s faculty, who consulted a combination of subject librarian, the catalog, and alternative discovery routes to locate resources and help facilitate access for their students. throughout all interviews, the subject librarian played a crucial role in helping navigate streaming media access and discovery. on psu’s campus, a study by wang and loftis (2020) was conducted to determine how streaming media was discovered, and the results reinforce the validity of that perception; faculty use the catalog, but also search other vendor platforms, finding those discovery systems more intuitive. in support of this, beisler et al. (2019), state in their study that “faculty expressed very clearly that having content available is not enough; streaming video must be easy to find and use and be reliable, or the content will not be used”.    cost containment    ithaka’s study revealed that keeping costs low for students is a major priority for instructors. local interviews also reinforced this, with 60% of interviewees responding that students should not have to pay additional rental fees or individual subscriptions for streaming films in courses. 40% of the faculty stated that they would reluctantly require students to pay for content, but none were pleased by the prospect. the interviewees who suggested that they would require students to pay out of pocket for film rentals indicated a ceiling of around $5 to $10 as the limit they would expect to burden students. keeping costs low for the library and individual students was understood as a necessity. this is reinforced in dotson and olivera’s 2020 case study about the affordability of course materials in general, and how faculty often try to mitigate students’ burdensome costs by increasingly using non-textual resources such as streaming media and open educational resources.    streaming media budgets have been a consistent concern on psu’s campus for several years as rising demand quickly outpaced the ability to contain costs. strategies ranged from using patron driven acquisitions (pda) models, exploring options via a consortia subscription, and requesting individual titles from filmmakers, for example. despite the various methods of streaming acquisition types available, a unified, cost-conscious strategy for the library is strongly desired and would benefit students and faculty.   local needs and strategies   information about streaming media appears in portland state university library general collection guidebook, which acts as the library’s outward-facing collection development policy (emery & loftis, 2020). the acquisitions librarian created a companion library guide dedicated to using films in courses, with information about the library’s decision-making process in licensing among other topics such as copyright, public performance rights, and other relevant information. it also outlines current criteria for purchasing single streaming licenses.   the interviews revealed that faculty have questions about how streaming media is paid for and understand that it is expensive to supply and sustain. funding is not always the key issue, however. in the case of some feature films, historical films, and foreign language content, institutional licensing is not always available to libraries, and what is available in collections that the library subscribes to is not always satisfactory. the library purchases educational streaming licenses for popular feature films when affordable and available. however, some content is simply not possible to license. these difficulties are captured in an article by lear (2022 p. 7) which discusses barriers in filling streaming requests: “because vendors did not have streaming rights in order to provide a license, or the distributor of the video did not provide institutional streaming licenses. and, of course, requests from streaming services such as netflix and hulu had a 0.0% success rate”. similar frustrations are felt in the psu library, where barriers such as these are encountered with requested titles, with a similarly dismal success rate for television series. in summary, there are varying levels of availability depending on the titles being sought.   conclusion   the results of the local faculty interviews were analogous to key findings in the broader survey, notably: how streaming is increasingly being used in instruction, keeping costs down for students as a high priority, and that this media is being accessed both from within the library and on commercial platforms in a patchwork.    that psu faculty were aware of and concerned about accessibility issues puts them ahead of the national trend in considering these issues as central. the library will play a role in advocacy in this area as we demand improvements from our vendors in this realm.   portland state university both reinforced and slightly diverged from the collected findings in that millar library has generated positive feedback in its ability to communicate and liaise with departmental faculty. the broader survey likewise found that faculty were happy with librarians and library services, however they indicated some general dissatisfaction with library subscriptions and ability to deliver content. the faculty answered a question that is crucial to the library, which is “to what extent are your current needs for incorporating video content in your courses being adequately met?” each psu interviewee answered that, to some degree, they felt well-supported. they each identified some frustrations, certainly, mostly due to availability of certain titles, and the ephemeral nature of many licenses, but it was largely positive and indicated goodwill towards the library staff and the library in general.    in the interviews, faculty were asked “what kinds of resources or other support would help you identify and assess opportunities for including video content in your classes?” the answers to this varied again by department. film studies faculty, nearly unanimously, answered that they would like to know more about the library budget, and what it could specifically support. social work faculty, on the other hand, were far more interested in learning about the library’s existing resources and how best to discover and utilize them. faculty from social work suggested workshops for faculty to learn where best to find resources, and they asked for more support from instructional designers with experience in online learning. social work faculty also discussed the locally created content that they or their students produced, either with lecture-capture or social media clips, and wondered how those types of streaming items could be housed and disseminated. ultimately, portland state university’s library built a good foundation with faculty related to its provision of streaming media, but more outreach can be done to offer support. some factors are simply out of the library’s control, such as limited acquisitions budgets, constraints of staff time, and market factors that render some content difficult to find and license.   streaming media is ubiquitous on portland state university’s campus just as it is on campuses across the globe. libraries must understand their faculty’s use, priorities, and barriers to making effective use of this technology. as a trusted provider of streaming media to faculty and students, a library ensures its relevance and stays true to its mission. if faculty are not adequately provided for in this area, they are obliged to seek content elsewhere, and this throws into question the quality, cost, copyright compliance, and accessibility of the materials they use in class.   author contributions   elsa loftis and carly lamphere were equally responsible for investigation, data curation, writing—original draft, writing—review & editing   references   adams, t. m., & holland, c. c. (2018). streaming media in an uncertain legal environment: a model policy and best practices for academic libraries. journal of copyright in education & librarianship, 1(2). https://doi.org/10.17161/jcel.v1i2.6550   beisler, a., bucy, r., & medaille, a. (2019). streaming video database features: what do faculty and students really want? journal of electronic resources librarianship, 31(1), 14–30. https://doi.org/10.1080/1941126x.2018.1562602   cooper, d. m., ruediger, d., & skinner, m. (2022, june 9). streaming media licensing and purchasing practices at academic libraries: survey results. https://doi.org/10.18665/sr.316793   dixon, j. a. (2017, september 7). the academic mainstream | streaming video. library journal. retrieved november 3, 2022, from https://www.libraryjournal.com/story/academic-mainstream-streaming-video   dotson, d. s., & olivera, a. (2020). affordability of course materials: reactive and proactive measures at the ohio state university libraries. journal of access services, 17(3), 144–163. https://doi.org/10.1080/15367967.2020.1755674   emery, j., & loftis, e. (2020). portland state university library general collections guidebook. https://pdx.pressbooks.pub/librarycollectionsguide/   farrelly, d (2016). digital video merrily, merrily, merrily, merrily wading into the stream. information today, inc. retrieved november 3, 2022, from https://www.infotoday.com/cilmag/nov16/farrelly--digital-video--merrily-merrily,-merrily-merrily-wading-into-the-stream.shtml   farrelly, d., & hutchison, j. (2014). atg special report: academic library streaming video: key findings from the national survey. against the grain, 26(5). https://doi.org/10.7771/2380-176x.6852   grove, t. m. (2021). academic library video services: charting a post-covid course. pennsylvania libraries, 9(2). 101–110. https://doi.org/10.5195/palrap.2021.262   hill, k., & ingram-monteiro, n. (2021). what patrons really want (in their streaming media): using focus groups to better understand emerging collections use. the serials librarian, 80(1–4), 30–33. https://doi.org/10.1080/0361526x.2021.1873701   ithaka s+r. (2023). about. https://sr.ithaka.org/about/   keenan, t. m. (2018). collaborating to improve access of video for all. reference services review, 46(3), 414–424. https://doi.org/10.1108/rsr-03-2018-0028   lai, j., & widmar, n. o. (2021). revisiting the digital divide in the covid ‐19 era. applied economic perspectives and policy, 43(1), 458–464. https://doi.org/10.1002/aepp.13104   lear, c. (2022). controlled digital lending of video resources: ensuring the provision of streaming access to videos for pedagogical purposes in academic libraries. journal of copyright in education and librarianship, 5(1), 1-19. https://doi.org/10.17161/jcel.v5i1.14807   loftis, e. (n.d.). streaming media overview. retrieved november 9, 2022, from https://pdx.pressbooks.pub/librarycollectionsguide/chapter/streaming-media-overview/   loftis, e., & keyes, j. (2019). navigating the sustainable stream. timberline acquisitions institute, mt. hood, or, united states.   macdougall, r., & ruediger, d. (2023). teaching with streaming video: understanding instructional practices, challenges, and support needs. ithaka s+r. https://doi.org/10.18665/sr.318216   peacock, r., & vecchione, a. (2020). accessibility best practices, procedures, and policies in northwest united states academic libraries. the journal of academic librarianship, 46(1), 102095. https://doi.org/10.1016/j.acalib.2019.102095   portland state university. (2022). facts: psu by the numbers. portland state university. retrieved november 8, 2022, from https://www.pdx.edu/portland-state-university-facts   portland state university. (2023). standard for accessible digital procurement. portland state university. retrieved november 8, 2022, from https://www.pdx.edu/technology/standard-accessible-digital-procurement   rapold, n. (2014, february 14). even good films may go to purgatory. the new york times. https://www.nytimes.com/2014/02/16/movies/old-films-fall-into-public-domain-under-copyright-law.html   rodgers, w. (2018). buy, borrow, or steal? film access for film studies students. college & research libraries, 79(4). https://doi.org/https://doi.org/10.5860/crl.79.4.568   tanasse, g. (2021). choice white paper: implementing and managing streaming media services in academic libraries. choice360. http://choice360.org/ librarianship/whitepaper vanullen, m.k., mock, e., & rogers, e. (2018). streaming video at the university at albany libraries. collection and curation, 37(1), 26–29. https://doi.org/10.1108/cc-01-2018-004   wang, j., & loftis, e. (2020). the library has infinite streaming content, but are users infinitely content? the library catalog vs. vendor platform discovery. journal of electronic resources librarianship, 32(2), 71–86. https://doi.org/10.1080/1941126x.2020.1739818   whitten, s. (2019). the death of the dvd: why sales dropped more than 86% in 13 years. cnbc. https://www.cnbc.com/2019/11/08/the-death-of-the-dvd-why-sales-dropped-more-than-86percent-in-13-years.html   appendix a semi-structured interview guide   streaming project   semi-structured interview guide   introduction   the ways that instructors can work with video content is evolving rapidly with the ascendancy of streaming platforms, including those the library licenses or are made freely available, over older formats like vhs and dvd. within this context, the library is conducting a study to understand the possibilities for fostering instructional use of video content at our university. i’d like to ask you questions about your current use, preferences, and future plans for incorporating video content in your teaching, and perspectives on the role that the library can play towards that.   before we begin, i’d also like to acknowledge that the landscape of available video content for educational use can be incredibly complicated, especially in terms of copyright terms and pricing models. those complexities are not the focus of our conversation, but of course they cannot be divorced from how we can use video content in our teaching. as we go please feel free to request we pause at any point if you’d like further explanation or clarification about video content in the context of the broader educational media landscape or any other aspect of our discussion.   current practices   i’d like to begin by exploring how you teach with video content, including vhs, dvd, and the content provided through streaming platforms.   do you currently use any video content in your classes? »       if yes, briefly walk me through what kinds of content you are using, and in what format/platform and length? ▪        for which classes do you use this content in? ▪        how does the content contribute to the pedagogical goals of the class? »       if no, why is that? [and if they have never used video content in their classes, skip to question 3]   how do you determine which video content you use in your classes? »       at what point in developing a course do you identify opportunities to include this content? do you typically have very specific titles in mind? »       where do you typically look for content? »       to what extent do delivery affordances determine whether you incorporate a specific video offering into your course? (e.g., delivery platform, accessibility options) »       do you consult with any other people to identify opportunities to incorporate video content into your class offerings?   to what extent are your current needs for incorporating video content into your courses being adequately met? »       has the pandemic changed your needs for incorporating video content into your courses in any way? »       are there any recent examples where you encountered barriers to incorporating specific content into your class? [e.g., unavailability of specific titles, copyright complexities] »       if yes, what were the barriers, and how did you work around them? ▪        did you work with any others to mitigate those barriers? ▪        is there anything else that could have been done to alleviate these challenges?   evolving expectations   next, i’d like to learn more about how your expectations are evolving around how video content can be incorporated in your classes.   has the availability of streaming content changed how you integrate video content into your teaching? »       what do you see as the greatest affordances of streaming content for your teaching? »       are they any downsides to incorporating streaming content into your teaching? »       is there anything that could be improved about streaming content offerings and/or functionalities to maximize the opportunities to incorporate it into your teaching?   has the availability of streaming content changed your expectations about how the costs of the video content should be covered? »       are there any instances where it is acceptable to require students to pay directly to access video content for educational purposes? »       how do your expectations with video relate to your expectations for how other forms of course content are paid for? e.g., textbooks, journal articles. »       what are the top factors that you think are important for determining the extent to which the university covers the costs of video content? which part(s) of the university should cover those costs?   what kinds of resources or other supports would help you identify and assess opportunities for including video content into your classes? »       would additional information about pricing structures, available titles, or format types affect your decision-making about what content to assign? »       ideally, how would you like to get this information and from whom?   wrapping up   i’d like to finish up with a few questions that put your perspectives into the broader context of your field and look towards future developments and needs.   how does your use of video content in your teaching compare to the practices of your peers? »       are there any kinds of video content or functionality that you would like to see more of? »       are there any developments in the areas that you teach that may affect how you or your peers would like to teach with video content in the next five years? is there anything else that is important for me to know about how you or your peers incorporate video content into teaching?   appendix b interview results   blue indicates film studies faculty and orange is social work faculty         evidence summary   quality of student paper sources improves after individual consultation with librarians   a review of: reinsfelder, t. l. (2012). citation analysis as a tool to measure the impact of individual research consultations. college & research libraries, 73(3), 263-277. retrieved from http://crl.acrl.org/content/73/3/263.abstract   reviewed by: laura newton miller collections assessment librarian carleton university ottawa, ontario, canada email: laura_newtonmiller@carleton.ca   received: 27 nov. 2012 accepted: 8 feb. 2013      2013 newton miller. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine whether the quality of sources used for a research paper will improve after a student receives one-on-one instruction with a librarian. to test citation analysis and a rating scale as means for measuring effectiveness of one-on-one consultations.   design – citation analysis.   setting – academic library of a large american university.   subjects – papers from 10 courses were evaluated. in total, 76 students were asked to meet with librarians. of these, 61 actually participated. another 36 students from the control group were not asked to meet with a librarian (although 1 partook in a consultation).   methods – librarians invited faculty to participate in a new service to help improve quality of student research papers. eligible courses included those with a required research paper component where papers could be evaluated at different times in the project. faculty instructed students in the class to meet with the librarian after a first draft of a paper was written. students from seven courses were asked to meet with a librarian. courses included english composition (2), geography (1), child development (1), occupational therapy (1), marketing (1) and women writers (1). three courses acted as control groups (all english composition). after meeting with students to make recommendations, librarians used a rating scale (measuring relevancy, authority, appropriate dates and scope) to review the quality of sources in both drafts and final papers.    main results – one-on-one consultations with a librarian resulted in sources being of a higher quality in the final paper. with the exception of authority, the differences between draft and final paper were statistically significant in all measures (overall quality, relevance, dates and scope). those in the control group showed no improvement in quality of sources between draft and final paper.   conclusion – quality of sources in final paper improves after one-on-one consultations with librarians. the use of a rating scale is helpful in objectively measuring quality of sources, although there is potential for subjective interpretation.     commentary   although citation analysis is commonly used to study library resources, this study takes a unique twist to the design by quantitatively examining the effects of individual research consultations. one-on-one instruction studies usually rely on more subjective tools such as satisfaction surveys and anecdotal evidence. using citation analysis and a new rating scale offers a fresh take on evaluating library impact. this paper is not only testing to see whether librarian consultation is effective, but also whether citation analysis is a useful tool to test that hypothesis. the answer is yes to both, with some caveats.   the ebl critical appraisal checklist (glynn, 2006) was used to determine various strengths and weaknesses of the study. the researcher’s rating scale was tested for reliability among raters and the author admits that the tool is far from perfect. however, he is to be commended for clearly explaining ways to improve the tool with better instructions and more descriptive categories and criteria, so that future researchers are fully aware of potential drawbacks. a copy of the rating scale with descriptions is included for other researchers’ future use.   this is a very readable article. however, there are concerns regarding the population and methodology. the researcher explains some of the issues regarding diversity of assignments and how faculty instructed students in acceptable sources, but it is challenging to know if there are just too many variables affecting the results. although there are benefits to one-on-one instruction (such as tailoring to individual student needs), there is a lack of standardization involved. for instance, it is not completely clear if the librarians were telling the students the actual resources to use, or were recommending places where they could find useful resources.    two librarians provided recommendations to students regarding resources and then subsequently scored student papers. there is potential for bias in the ways that they scored the sources, since they knew which students/papers they had assisted. the librarians might have looked more favourably on the papers/sources of those students that they had helped versus those students that they hadn’t.  having different people score the papers would have helped eliminate this bias.   we don’t know the librarians’ individual style or their subject expertise. the researcher is comparing resources of english composition, geography, child development, occupational therapy, marketing and women writers. these are very different subjects, and what one librarian recommends for resources could be quite different from the other based on their knowledge of the subject. limiting to only english composition papers for both groups would have made for stronger comparisons.   the use of tools such as rating scales and rubrics to measure student learning has received much attention in recent years. despite some drawbacks, this paper helps to support academic librarians looking for different measures of library impact that are substantive and improved over methods such as satisfaction surveys. however, time invested in individual consulting, evaluating and scoring citations for large classes is a potential challenge. future researchers should take the author’s recommendations to make the rating scale a more reliable tool, and to limit comparisons to similar or equivalent courses.     references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi: 10.1108/07378830610692154     supporting the intersections of life and work: retaining and motivating academic librarians throughout their careers    evidence based library and information practice       research article   supporting the intersections of life and work: retaining and motivating academic librarians throughout their careers   lori birrell associate dean for special collections university of arkansas fayetteville, arkansas, united states of america email: lori@birrell.us   marcy a. strong metadata projects librarian river campus libraries, university of rochester rochester, new york, united states of america email: mstrong@library.rochester.edu   received: 16 may 2021                                                              accepted: 17 dec. 2021      2022 birrell and strong. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29971     abstract   objective – this study uses the kaleidoscope career model (mainiero & sullivan 2006a) to determine key sources of motivation for library professionals during their careers and identifies strategies for how library administrators can better retain and inspire their staff.   methods – the authors adapted the kaleidoscope career model survey tool with permission from mainiero and sullivan. the authors used qualtrics to send out the adapted survey and in october 2019 emailed a call for participation with the survey link to six library electronic mailing lists. a total of 433 participants completed the survey. the authors reviewed the demographic data and charts qualtrics generated and used an open-coding method to analyze the qualitative responses to open-ended questions included in the survey. first, they read through those responses, identified common words, phrases, and ideas, which became initial codes. then the authors reviewed the codes and determined themes common in the data. each author coded and analyzed each question. those themes then informed the discussion and recommendations shared in this article.   results – nearly 60% of respondents identified as being in the authenticity phase, 15% in the challenge phase, and 18% in the balance phase. when asked if they felt supported, those in the authenticity phase reported the highest overall level of satisfaction, with those in the 47–52 years old cohort experiencing peak feelings of support. the study found that all early career practitioners seemed interested in continuing in a supervisory role. those older participants in the balance phase were less interested than those in the other two phases in continuing to supervise. those in the authenticity phase identified most strongly with being organizational leaders. by contrast, older participants in the balance phase did not identify strongly as leaders. those in the challenge phase showed strong interest in being leaders at an early age and that interest increased among older cohorts.   conclusion – this study is the first to analyze sources of motivation for academic librarians during the stages of their careers. when working with librarians who identify with the authenticity phase, administrators should work with their employees to develop career goals that are extrinsically based, such as what can be achieved through good work rather than striving for a dream position. librarians in the balance phase would benefit from early opportunities to develop leadership roles or serve in supervisory roles. these early opportunities better fit with their efforts to prioritize family later in life. librarians in the challenge phase are intrinsically motivated to achieve and strive. they may experience disappointment as newer career librarians continue to advance and as they begin to plateau later in life. leaders must consider the kinds of changes their organization can withstand as they strive to best support and foster the growth and development of all of their employees.     introduction   the concept of career is changing (lyons, et al., pp. 9–10), and as working professionals become more mobile and flexible in their definitions of a career, employers must learn to meet their needs in order to engage a motivated and experienced workforce. academic librarians and libraries are no exception. librarians wrestle with questions like how they bring their authentic selves to work, how they understand work and life balance throughout their careers, and how they perceive career advancement, and at what time. to meet the needs and expectations of their staff, library administrators must ask how to continue to retain a pool of driven and satisfied professionals, how to continue to support librarians as their needs change throughout their careers, and how to create workplace cultures that offer flexibility and space in support of a diverse workforce.   this study is an application of mainiero and sullivan’s kaleidoscope career model. the model uses three phases—authenticity, balance, and challenge—as a non-linear approach to understanding the mapping of career trajectories (figure 1). in the authenticity phase, professionals have a need to be genuine and to act in ways congruent with their values. in the balance phase, they desire a more balanced personal life. in the challenge phase, they seek exciting, stimulating work. professionals can be in one or several of the phases at any given time and can move through the phases as the circumstances and motivating forces in their lives change. for example, while a professional may start their career in the challenge phase, later in life they may find themselves more strongly identifying with balance or authenticity before ending their career back in challenge.   figure 1 kaleidoscope career model.   mainiero and sullivan (2006a) argue “because most organizations have not acted, individual workers have acted instead. women and men are working within and outside corporate boundaries to better blend their own needs to authenticity, balance, and challenge. these men and women are making adjustments to their careers to find a regression line that balances work and family. they are developing new definitions of success” (p. xii). though not in the corporate world, academic librarians, too, must grapple with the impact of economic background, age and ageism in the workplace, gender identity, care-giving roles, absence of diversity, ableism, and mental health when determining how best to shape their careers.   in this female-dominated profession, understanding how women perceive their career trajectories will help administrators and colleagues determine how to provide flexible organizational cultures to support their work. women’s professional lives are less often characterized by a linear trajectory. based on the findings, this study examines how academic librarians conceive of and perceive their career path trajectories as they relate to their overall sense of satisfaction with their careers to date and their feelings of support from their employers.   the initial findings of this study, drawn from 433 survey responses, suggest that for those participants who most strongly identify their current career phase as one of authenticity, they’re intrinsically motivated by values and sense of “fit” within their library or the profession. for those who strongly identify with challenge, they’re extrinsically motivated by traditional rewards, such as higher salary and more responsibilities. intertwined within each of these two phases is balance, as the characteristics of that phase heavily impact how academic librarians perceive their overall career paths. applying the kaleidoscope career model in the academic library context enables practitioners to equip their profession with the language and data to describe themselves as well as provide an opportunity for self-reflection. as a result, we can better understand how perceptions of career trajectory impact the industry and its ability to retain talent.   literature review   there exists a gap in the literature addressing career path changes. this literature review is divided into a discussion of career progression, a discussion of the increasing demand for a flexible workforce in response to changing expectations in higher education, and a discussion of job satisfaction. the literature that explores these areas focuses almost exclusively on different phases in one’s career, with the assumption of a linear or stagnant progression into management roles. while some of the work presented here explores these issues, little research in libraries has focused on cyclical or nonlinear progression. this study addresses this gap in the literature through the application of lisa mainiero and sherry sullivan’s kaleidoscope career model.   career progression       in their book the opt-out revolt: why people are leaving companies to create kaleidoscope careers, mainiero and sullivan (2006a) argue creating adaptive career paths have fallen to employees because most organizations have not proactively developed such structures of support. their study included survey instruments and interviews with men and women of different generations and industries. based on the data they collected, we developed a new framework for conceiving of career trajectories. they define the kaleidoscope career model as “a career created on your own terms, defined not by a corporation but by your own values, life choices, and parameters. like a kaleidoscope, your career is dynamic and in motion” (p. 11). the authors go on to argue that “as your life changes, you can alter your career to adjust to those changes rather than relinquishing control and letting a corporation dictate your life for you” (p. 111). their research revealed that “for men the prospect of a linear career within the same firm or industry is still highly valued” (p. 107). by contrast the authors argue “for women, a ‘career’—often defined as a series of interrupted jobs, transitions, and shifts—cannot be separated from a larger understanding of their lifestyle priorities” (p. 107). mainiero and sullivan conclude with an analysis of the impact of people’s changing perceptions of their career trajectories on industry: “for employers, understanding the importance of the kaleidoscope career is critical . . . until now, career paths and succession plans within corporations have [not] been based . . . on the . . . (challenge-balance-authenticity) kaleidoscope career pattern that characterizes most women” (p. 153). the study presented here employed the kaleidoscope career model survey tool and explores how the field of academic librarianship complements or complicates the findings in professions writ large.   two years after they completed their book, sullivan and mainiero (2008) published an article aiming to provide suggestions for reconsidering human resource development programs with women’s career trajectories in mind. they argue that by mid-career the women in their study were predominantly concerned about the issue of balance (p. 36). the authors underscore the ways in which women evaluate opportunities and make decisions, through the lens of relationalism (p. 37). drawing on their kaleidoscope career model career phases, they argue that to meet women’s needs and fit within their framework for decision making, organizations should consider how women perceive their current career phase. when working with those in the authenticity phase, organizations should focus on corporate social responsibility and company efforts to promote total wellness in mind, body, and spirit. organizational mission should align with women’s personal values and promote ethics and values (p. 38). by contrast when working with a woman in the balance phase, organizations should reward actual performance, regardless of “face time” in the office, and create actual “family friendly” programs that consider needs outside of work (pp. 39–40). finally, for those women in the challenge phase, the authors argue organizations should create equitable access to challenging, meaningful job assignments and training opportunities and should design career development programs with opportunities (pp. 40–41). these recommendations highlight the need to find solutions that fit with individual needs and goals, rather than treat one’s workforce as a monolith. based on the data presented in this study, librarians have similar unmet needs and desire differing levels and systems of support from their organizations throughout the lifecycle.   applying mainiero and sullivan’s (2006a, 2008) work to the field of health care with nurses in australia, o’neill and jepson (2017) conducted a two-phase study to better understand the interplay of women's kaleidoscope career intentions and life roles. they found “some women seek to transform their worker and leisure life roles as they desire authenticity in their life and will pursue paid and unpaid work as well as leisure activities to do so” (p. 971). volunteer work was one example of the kind of unpaid work these women might pursue. based on their data, the authors conclude “individuals with a high leisure life role commitment may seek authenticity in their late career and want to engage in leisure life roles that provide them with internal fulfilment and satisfaction” (p. 973). surprisingly, the authors found that women seeking balance struggle more when caring for aging parents than when caring for children. women also seemed to continue to pursue challenging work late into their career. these women may have fewer commitments to non-work life roles, such as caregiving or leisure pursuits than others in their study. the authors stressed the importance of considering the impact and flux of care responsibilities and other non-work life pursuits throughout the life cycle when recruiting and retaining women nurses. furthermore, they underscored the importance of providing both organizational support for evolving needs, and the role governmental programs play in women’s ability to successfully navigate care duties. published in 2017, this article is one of the first to apply the kaleidoscope career model to a female-dominated profession, which in that regard is similar to librarianship.        meeting the demand for a flexible workforce       narrowing down to the field of librarianship within higher education, maggie farrell (2013) highlights the changing nature of career progression in her article, “lifecycle of library leadership.” farrell contends “a librarian might move from a management position to a non-management position and then to a high-level leadership position. our organizations are far more fluid today, challenging us to rethink how an individual progresses within libraries” (p. 257). as farrell argues, academic libraries have continued to experiment with non-traditional organizational hierarchies and job duties. she states “one view of leadership development is that you progress from a position to a supervisor to a manager to a leader. another perspective is that positions change and individuals develop their skill sets but not necessarily in a linear fashion” (p. 264). farrell’s work does not include data indicating the experiences of those whose careers proceed in a nonlinear path.   though farrell acknowledges such a path exists, her discussion of management and leadership skill acquisition follows traditional assumptions about such senior roles. the hard and soft skills needed to be successful as first a manager and then, as farrell argues, as a leader do not come into play before one prepares to or enters those advanced positions. once one has those skills, should a senior leader choose to enter into a practitioner role again, “you can take these skills with you . . . whereas tradition outlined a linear, developing path for leadership development, our libraries today require aspects of these skills throughout our organization. leadership development at all levels of our libraries will enhance our work” (p. 264). it seems as though farrell is not necessarily advocating for leadership skill development at all levels; rather she recognizes the benefit of taking advantage of those skill sets once a senior manager returns to a role elsewhere in the organizational hierarchy.   michael ridley’s (2014) work, “returning to the ranks,” explores similar benefits to library organizations as farrell. he argues for those library deans and administrators who have term limits or choose to leave those roles and assume duties outside of library administration within their organization to consider that many “former chief librarians often have unique and valuable skill sets that can be exploited” (p. 4). in librarianship, we tend to think of career paths as a linear progression rather than cyclical. according to ridley, senior leaders often feel they experience a “professional de-skilling” as they move into administration (p. 3). their work becomes increasingly focused on external stakeholders, and their peer group shifts from librarians to senior administrators in other university units. facilitating this transition requires overcoming key challenges including how the former leader develops the most productive relationship with their new boss and the person’s transition to a new role, which may include a sabbatical or vacation time and—if applicable—being part of a union again. moreover, ridley highlights the dearth of librarians willing to enter into senior administrative roles and encourages decision-makers to develop “a more supportive policy and reward structure that facilitates returning to the ranks [which] might encourage librarians to explore management and administrative roles without feeling that they are somehow ‘leaving the profession’” (p. 9). ridley’s work makes a valuable contribution to the literature; his recommendations emerge from conversations with four senior leaders, including himself, who “returned to the ranks” of librarianship. these lessons learned offer a useful starting point for further analysis of librarian career paths.        sources of motivation and job satisfaction       determining sources of motivation and job satisfaction are two related areas that impact one’s career path and form the cornerstone of this study. in their 2009 article, mallaiah and yadapadithaya describe the findings from a survey they distributed to fifteen academic librarians working at universities throughout karnataka. focused on exploring intrinsic motivation, the authors concluded that library work, itself, and a sense of personal worth were two drivers. considering the broader implications of their work, mallaiah and yadapadithaya argue “motivation is culture specific, industry-specific, and organization-specific and context or situation-specific in nature” (p. 41). related to sources of motivation is job satisfaction.   authors adigwe and oriola (2015) found among nigerian librarians “with increased length of service, the importance of job satisfaction decreased for factors such as self-actualization and conditions of work, but the importance of pay increased” (p. 782). for those in american academic libraries seeking to increase their salaries, few pathways exist other than entering formal leadership and management positions. kathy pennell (2010) underscores an increasing interest in “shifting away from the use of narrowly defined job descriptions toward more flexible ones that are not skill based but are based on job roles. the flexibility allows the latitude necessary to provide opportunities for job rotation or stretch assignments to help develop high-potential employees” (p. 286). as a result, employees can better meet their professional and personal needs and goals throughout their careers, while employers gain a more satisfied and motivated workforce. this study contributes to the existing literature in three critical ways. first, no researchers to date have addressed the evolving needs of practitioners through a lifecycle model lens. the research presented here builds on mainiero and sullivan’s (2006a) work by applying the model to the academic librarian context. the kaleidoscope career model provides the library profession and organizations with an approach through which to critically reflect on their current practices, values, and support mechanisms. second, examining the career paths of those in senior as well as mid-level and entry-level positions in academic libraries fills a gap in the literature that to date has focused almost exclusively on senior-level positions. third, unlike previous analyses of librarians’ motivations, the conclusions presented here address both intrinsic and extrinsic motivation.   aims   this study addresses two interrelated questions: what motivates library professionals in doing their work, and what can library administrators do to retain and inspire their staff? library professionals should have a better understanding of what motivates them in their work, why they may or may not choose a traditional career advancement path, and how priorities in their life may shift over time and change their career perspectives. this study also underscores the important role library administrators play in understanding the individual motivations of their staff and supporting their employees with a more holistic approach throughout their careers. the application of the kaleidoscope career model, with its authenticity, balance, and challenge phases, is one framework through which library professionals and administrators can understand how such motivators could, and likely do, change throughout the lifecycle.   methods   we adapted the kaleidoscope career model survey tool (mainiero & sullivan, 2006b) with permission, which was obtained via email from mainiero and sullivan. this survey had been validated as part of the previous research projects mainiero and sullivan conducted. we piloted but did not validate the adapted survey tool. following the guidelines in fink’s (2013) how to conduct surveys, we pilot tested the survey by emailing a link to the adapted survey available through qualtrics to seven academic librarians and received feedback from five people. we made subsequent edits to the survey based on this feedback. changes to the tool included briefer terms on the statements of agreement scale and an adjustment in some language to be less corporate and more congruent with the academic library work environment. the survey tool included thirty statements with a five-point scale, allowing participants to express how much they agreed or disagreed with each statement. the answers to those thirty statements resulted in the score participants received, indicating which of the three kaleidoscope phases they most closely identified with. the remainder of the survey included questions about how they felt about their results (appendix), whether they felt supported by their library administration, if they supervise or want to supervise, and if they consider themselves a leader or want to be a leader. the survey also asked several demographic questions related to institutional affiliation, gender/gender identity, age, and time spent working in the profession. the questions were a mix of close-ended and open-ended questions. we developed the survey using qualtrics, which enabled us to easily capture participant responses and begin analysis after data collection.   after receiving approval from the institutional review boards at both the university of arkansas and the university of rochester, we sent a call for participation with the link to the survey to six library electronic mailing lists via email in october 2019. the mailing lists included rare books and manuscripts section of the association of college and research libraries (acrl), society of american archivists, university libraries section of the acrl division of the american libraries association (ala), college libraries section of the acrl division of the american libraries association (ala), and the former library leadership and management association division of ala. the email included an information letter describing the research project. participants had one month to complete the survey. we emailed two reminders as the survey window continued.   once the survey closed, we exported initial statistics from qualtrics to determine the breakdown of participants by career phase. we reviewed the demographic data and charts qualtrics generated and used an open coding method to analyze the qualitative responses to open ended questions included in the survey. first, we read through those responses and then identified common words, phrases, and ideas, which became initial codes. each of us coded and analyzed each question. we then shared our analyses with one another for reliability. once compared, we worked together to finalize codes for each question based on the context of the original participant responses. then we reviewed the codes and determined themes common in the data. those themes then informed the discussion and recommendations shared below.   results   the results are interpreted through three categories based on the questions asked in the survey: participants’ general demographic information, participants’ sense of administrative support, and participants’ interest in taking on or continuing in leadership or supervisory roles.       demographics   a total of 433 people completed the survey. the majority worked at 4-year doctoral-granting universities (table 1). nearly half of all respondents worked in public services with nearly one-fifth working in special collections/archives (table 2). about one-quarter of participants had twenty or more years of work experience in librarianship (figure 2). the largest group of participants (23%) were aged 34–40 (figure 3). the authors use the term early-career to refer to those participants aged 22-33; the term mid-career for those 34-52; and late-career for those 52 and older. the vast majority (74%) of participants identified as female (figure 4).   to contextualize the demographics in this study, the authors exchanged emails with acrl staff, who provided the 2018 acrl member survey data. of 3,029 respondents, 1% of respondents were aged 18–24 years old, 20% were 25–34, 25% were 35–44, 24% were 45–54, 22% were 55–64, and 9% were 65 and older. of the respondents, 77% were female (or 2,332.33 respondents), and 20% were male (or 605.8 respondents), with 1% indicated a different gender identity (or 30.29 respondents) and 2% preferred not to say (or 60.58 respondents). the age and gender demographics in this study were consistent with the 2018 acrl survey. the gender demographics were also in line with the arl annual salary survey 2018–2019 (morris, 2019), which found that in u.s. and non-u.s. libraries, men comprise 36.9% (or 3,541) of staff and women comprise 63.1% (or 6,050 of staff). table 1 institution type of participants institution participants 4-year doctoral-granting university 54% (235 participants) 4-year masters-granting university 16% (68) 4-year bachelors-granting university 10% (43) other (e.g., public, distance, special, nonprofit, seminary, library of congress, government agency, consortium, health sciences, research) 5% (24) 2-year community/vocational college 5% (21) left blank 10% (42)     table 2 functional area of work of participants functional area participants public services 41% (176 participants) special collections/archives 18% (74) administration 13% (57) technical services 10% (43) other 8% (35)     figure 2 years of experience.     figure 3 age ranges of participants.     figure 4 gender identity of participants.   focusing on the breakdown of participants in each career phase (figure 5), 60% of respondents identified as being in the authenticity phase, 15% in challenge, and 18% in the balance phase. some respondents identified as a combination of phases: 3% of all respondents identified as authenticity and balance, 3% as authenticity and challenge, and 1% as balance and challenge. we have not focused on these results, as the small percentages do not warrant generalizations and therefore do not factor into the study’s overall findings.     figure 5 breakdown of participants by career phase.     among those in the authenticity phase (figure 6) nearly one-fifth were entering mid-career and were between the ages of 34–40. this age group was the largest in the survey population. a mere 3% of participants were 65 years old or older. a similarly small percentage (4%) were aged 22–27 and were at the beginning of their careers.     figure 6 age ranges of authenticity participants.     nearly one-third of participants in the challenge phase (figure 7) were between the ages of 34–40 and 12% were aged 59–64 or nearing retirement age. there were fewer participants in the challenge phase in what is traditionally thought of as mid-career than in the authenticity phase.   figure 7 age ranges of challenge participants.     the largest group (30%) who identified as in the balance phase (figure 8) were aged 34–40 years old. similar to the age breakdowns of the other two phases, there were few participants early in their careers or nearing retirement.     figure 8 age ranges of balance participants. sense of administrative support   overall, most participants expressed positive reactions when asked if they felt supported by their administration. drawing on participants’ comments to this question, they define “supported” as having a supervisor who fosters accountability; displays behavior to indicate they trust workers, including offering flexible schedules, professional development, work–life balance, and autonomy to develop new projects and structure work more generally; advocates for workers; fosters creativity and collaboration; and engages with the work while not micro-managing. the phrases participants used when they described not feeling supported by their supervisors included not feeling respected, valued, or understood; feeling there was incompetent leadership in the organization; and not feeling connected to staff, patrons, or work culture.   figure 9 feeling of support from administration.   when asked if they felt supported (figure 9), those in the authenticity phase reported the highest overall level of satisfaction, followed by those in the challenge and then balance phases. among the participants who responded negatively, those in the balance phase represented the largest percentage of participants who did not feel supported by their supervisor at 38%. of the authenticity phase, 28% responded negatively, and of those in the challenge phase 31% responded negatively.   when considering the participants’ age together with their career phase, those in the authenticity phase experienced peak feelings of support between the ages of 47–52 with 81% of respondents answering positively (figure 10). that percentage steadily declines amongst older participants with only 29% of respondents ages 65–70 answering positively. those in the balance phase experienced peak feelings of support between ages 65–70 at 100% followed by ages 41–46 with 75% of respondents answering positively. the youngest (100% of those 22–27) and the oldest (100% of those 65–70) participants in the challenge phase reported feeling supported. unlike the participants in the other two phases, those in the challenge phase experienced feeling lower levels of support between 47–52 and 59–64 years old, with 57% and 40% respectively responding positively.     figure 10 feeling of support by age and career phase.   interest in leadership and supervisory roles   interest in serving in a management or leadership role varied across age groups as well as career phases. when asking participants about their interest, we defined management to mean a formal supervisory position and leadership to be a little more ambiguous, including non-formal roles such as project manager, mentor, or other influential role outside of direct supervision.   figure 11 overall interest in supervising or leading.   looking at the overall picture, there was high interest in becoming both a supervisor and an organizational leader from the youngest cohort (figure 11). however, interest diminished amongst the older age cohorts, with the lowest interest showing in the 47–52 age group, before rising slightly again.   this can further be broken down by career phase, which reveals more nuance about when interest peaks by age group (figure 12). for example, when asked their interest in becoming a supervisor, those between ages 28–33 were the peak age group in the authenticity phase. for the balance phase, the peak occurred with librarians between ages 22–27 who are at the very beginning of their careers. however, the peak was at a later age group, between ages 34–40, for the respondents in the challenge phase.   figure 12 interest in becoming a supervisor by age. (the gaps between columns represent those age ranges with no participant responses.)   when participants in the authenticity phase were asked about their interest in becoming a leader, those earlier in their careers responded more favorably than those age 34 and older (figure 13) those who were early in their careers and in the challenge phase expressed noticeable interest in becoming leaders.  interestingly, no participants aged 53–58 in the balance and challenges phases answered this question. interest in becoming an organizational leader showed a noticeable drop in age groups older than 34 across all phases.   figure 13 interest in becoming a leader by age. (the gaps between columns represent those age ranges with no participant responses.)   this sometimes reluctance in supervision and leadership work can seem a stark contrast to the overall interest shown by experienced librarians in either continuing in their role as a supervisor or in their perception of themselves as organizational leaders (figure 14). younger cohorts already working as supervisors or in leadership positions were less enthusiastic about continuing, with the lowest numbers in the 28–33 range.   figure 14 overall current perceptions of supervision and leadership. the data reveals the intersections of age with career phase, suggesting opportunities for administrators to nurture these professionals and encourage them to continue in supervisory or leadership roles.   common to each phase, practitioners seem interested in continuing in a supervisory role when earlier on in their careers (figure 15). however, older cohorts in the balance phase are less interested than those in the other two phases in continuing to supervise. for those in the authenticity phase, interest peaked at mid-career between ages 47–52. for those in the challenge phase, interest was highest among the 41–46 cohort.     figure 15 interest in continuing to supervise by age. (the gaps between columns represent those age ranges with no participant responses.)     those in the authenticity phase identified most strongly with being organizational leaders between the ages of 47–52 (figure 16). each cohort in the challenge phase strongly identified as leaders, with peak interest among those in mid-career and approaching retirement. cohorts in the balance phase do not identify as consistently with being leaders, with significant declines occurring among those ages 28–33, and again among the 59–64 years old. figure 16 consider themselves a leader by age. (the gaps between columns represent those age ranges with no participant responses.)   discussion   extrinsic and intrinsic motivations drive the decisions and goals participants have made in their careers. when asked to reflect on their perceptions about their career paths, and specifically to consider the level of support they experienced and their interest in assuming or continuing to serve in management or leadership roles, these sources of motivations surfaced. those in the balance phase felt the least supported when compared with those in the other two phases. when reflecting, one participant commented “at my previous institution, my boss wanted us to take time for ourselves, but he was also aggressive, critical, and unequal in his treatment.” this group is managing extrinsic sources of motivation including care and life responsibilities. that lack of support surfaced regardless of age, whereas in the other two phases participants of different ages experienced varying levels of support.   those in the authenticity phase responded more positively to assuming leadership, as opposed to management, roles. one such participant shared “i just took on new responsibilities and a new title (lateral move) that is giving me the opportunity to add value to my organization.” this finding speaks to the intrinsic motivation that participants expressed and their interest in the non-hierarchical nature of such duties that do not necessarily include supervisory responsibilities. higher salary, an advanced title, and the drive for greater responsibilities are the types of intrinsically focused motivations that push those in the challenge phase to pursue and remain in management and leadership roles. as one participant responded, “i came into librarianship as a second career after a divorce. my motivation has primarily been focused on advancement . . . to provide for my children. that being said, i also love a challenge, and leadership roles provide those more than other positions.” though this group certainly benefits from being supported, they possess a strong drive to pursue advancement on their own.   support   the most consistent feelings of support across all age groups, regardless of career phase, were expressed by those in the authenticity phase. one participant shared the following:   i am given respectful space to share both my opinions and my ideas. when i provide enough evidence of my position, i am generally permitted to move forward as i wish. when i have not, i am respectfully challenged to collect more information and strengthen my case. if an endeavor ultimately does not turn out to be successful, i still feel respected for trying.        as someone who has strong personal values, characteristic of the authenticity phase, this participant appreciates being given the opportunity to share their opinions and ideas. working for someone who then explains their decision helps the person to continue to feel respected in the workplace. in contrast to those in the challenge phase, those in the authenticity phase felt the least supported at the ages of 34–46 years old. as one participant in this cohort explained,   i am in the middle of changing careers and want to be more involved in heritage preservation, especially international, intangible and theoretical. so i am back in school myself. i feel that archivy [sic] is a calling, that it called me and for the last 20 years i have served, but i am burnt out and tired of the same old battles. also, this field does not pay well enough.   mid-career practitioners, who identify as in the authenticity phase have tried to fit themselves into the values mold of their organization and have not found a good fit. participants entering mid-career seem to experience a crossroads where they confront their own values and those of their organization. as a result, practitioners appear more inclined to make a career pivot that more closely aligns with their personal values.   the highest percentage of respondents who did not feel supported were those in the balance phase. mainiero and sullivan (2006a) point to support and flexibility as key drivers for women making career decisions. such support to juggle work and life responsibilities can come from their spouses or partners, employers, or family members. the responses from this study’s participants echo mainiero and sullivan’s (2006a) conclusion that the absence of such support in women’s quest for balance strongly impacts their career decisions (p. 193). such responses suggest that supervisors have not found or implemented adequate strategies to best meet the needs of those who balance family, relationships, caregiving, and personal health and emotional conditions throughout the lifecycle. one participant reflected on the supportive relationship they have with their supervisor:   my current supervisor is also a mother and is very supportive of taking time off to attend kid things, staying home with sick kid, etc. i feel that the administration at my current job are very understanding and supportive of work-life balance. i also have been supported in professional development and i know that my supervisor wants me to succeed in my career.        older participants reported an absence of support, suggesting that supervisors may not give as much attention to work–life balance issues throughout the lifecycle. as one participant explained, “i feel that i am on the b team and that the newer librarians have been given the support to shine.” when analyzing these results by age, it is worth noting that only one participant was in the 65–70 age group, so additional data would be needed to determine if those in the balance phase feel supported later in life.   the data indicates that for those aged 34–46 in the challenge phase, practitioners begin to take on advanced roles or move into management positions as they feel a strong sense of support from their supervisors. one participant responded “i have the resources i need and am encouraged to pursue my own interests and professional contributions.” intrinsically motivated, this participant’s comment highlights the individualistic nature, rather than values-driven or work–life balance focus, of those in the challenge phase. by contract, practitioners approaching retirement feel waning support. participants aged 47–52 experienced a lower level of support with 57% responding positively, while only 40% of those aged 59–64 felt supported. this finding suggests that once practitioners have less that challenges them professionally, they feel less supported to pursue their goals.   interest in leadership and supervisory roles   participants who identified themselves as being in the authenticity phase seemed much more interested in leading informally rather than advancing through formal management structures. one participant reflected: “i am currently in middle management and find the work challenging and fulfilling. i'm not sure i want to go further up the ladder because it might mean having to make decisions that are inconsistent with my values.” those in the authenticity phase are also strongly interested in their lives outside of work and a focus for them is work–life balance. participants’ interest in being a supervisor peaked at the 28–33 age range, which suggests the beginning of a values misalignment with their organization or profession. the issue of competing priorities and misaligned values led one participant to share that these struggles are “at the root of the burnout issue, especially for women who find it difficult to be managers at work and caretakers at home . . . many of us do not feel listened to or respected in dysfunctional academic libraries.” these early experiences and misalignment with their personal values lead practitioners to pull back from formal supervisory roles and seek out alternative career paths. practitioners seemed to shift and more strongly identify themselves as organizational leaders at the 34–40 age range. one participant stated:   i prefer to lead in less formal ways like chairing campus committees or being part of task forces or working groups. i find it more satisfying to work on a project and see it completed or implemented rather than having to deal with ongoing issues with no end in sight.   overall, these participants expressed more sources of extrinsic motivation, such as finding fulfillment through making a difference in students’ lives, rather than sources of intrinsic motivation, such as building a career through promotions. an informal role could position those in the authenticity phase to become strong leaders of project-based work with concrete objectives and timelines; thereby enabling these practitioners who are values-oriented to feel a sense of accomplishment, which can be harder to attain when in a formal supervisory role.   this source of motivation contrasts quite noticeably with participants who identified as being in the challenge phase. participants expressed interest in new positions, the opportunity to supervise and earn promotions and increased salaries. one participant stated their goal as:   yes, i would like to become more of an organizational leader, but not at my current place of employment. i would like to work at an organization where i felt there were more opportunities for the kind of work i enjoy doing, so that there would be clearer lines towards leadership opportunities.       this participant identified their career goal and sought to advance by leaving the organization and working in a library with an organizational culture oriented toward leadership opportunities. those in the challenge phase aged 28–33 responded positively with 60%, considering themselves to be organizational leaders earlier in their careers. of the three groups, this group showed the highest satisfaction at being a supervisor later in their career.   finally, those in the balance phase most strongly indicated that they do not feel prepared for management positions. rather than seek out leadership or supervisory responsibilities, those in the balance phase may find themselves asked to assume those roles before they have gotten the training or identified an advancement path as a career goal. one participant stated, “i became a leader somewhat unwillingly and in a time of need for our library. i often feel inadequate and unprepared in my work.” this participant’s experience underscores the impact of being extrinsically motivated. overall, those in the balance phase do not show a strong propensity for wanting to be supervisors, especially early in their careers. one participant commented: “i think the profession as a whole needs to reconcile how librarians can translate their skills across positions/organizations/etc. i have no idea how to leverage the experience i have to transition to a different type of library work.” such practitioners can feel stalled as they may be organizational leaders, but not supervisors, due to the limitations of the library’s hierarchy. taking a more a passive approach to their careers highlights the importance of training and organizational support for those prioritizing work–life balance.   looking across the career phases amongst those who are not already supervising, the strongest interest (58% of respondents) in assuming such a role came from the youngest cohort, who are the newest to the profession and most enthusiastic to take on the roles. but for those aged 34–40 that interest dropped to below 30% with subsequent age cohorts even less interested in assuming management roles. the surprisingly low interest in continuing to supervise amongst those in their late twenties and early thirties is also concerning. why don’t these young professionals want to keep supervising? one participant commented: “right now, i'm feeling very drained from having no support from my supervisor + having direct reports that clearly don't care for my supervisory role . . . external factors like low morale and lack of institutional support are affecting my views and values . . ..” their younger counterparts expressed strong interest in supervising, and yet this group of similarly aged individuals seemed uninterested in continuing to supervise. not getting enough training or support could be an indicator. the responsibilities of the position may compete too strongly with raising a family or participating in outside activities. the interest in continuing to supervise noticeably rises in the next age cohort (34–40), suggesting the impact of a degree of maturity, increased wisdom, and comfort due to job experience. additional research is needed to determine the root causes of this uninterest in the younger group.   leadership perception and interest from participants followed a similar pattern. one participant stated, “i am very interested in what library leadership looks like outside of the traditional management role or model. there are many, many ways to exercise leadership skills that do not involve becoming a direct supervisor or manager.” the 28–33 age group was least likely to consider themselves current leaders; this group also displayed the lowest interest in being a supervisor and seems to be struggling with issues related to both formal and informal leadership, in ways that the age cohorts that are a little younger and older, do not. one participant who identified as in the authenticity phase commented:   while i have had a few promotions earlier on in my career, i feel like i have more or less plateaued. i am not really seeking new opportunities or challenges because i don't feel that i can take much more on at this point in my life.        this sentiment exactly matches the results from the supervision question, suggesting, perhaps unsurprisingly, that older librarians are less interested in being leaders. experiencing burnout may be one cause. as one participant reflected, “i get tired and need more vacation and down time [than] in the past. at times i feel burned out with the long hours and social events required of my position.” it is notable that interest in continuing to supervise peaked among the mid-career group of 41–46, which suggests career burnout could be more likely to occur during this age range. each subsequent cohort also showed a 20% difference between interest in being a leader vs. interest in being a supervisor.   limitations   the gender breakdown of participants is the principal limitation of this study; 74% of the sample identified as female. therefore, the analysis and recommendations presented here may not be generalizable to those practitioners who identify as male or gender variant/non-conforming. the authors also acknowledge that they did not collect data on race, ethnicity, sexual orientation, disabilities, or other identity-related categorizations that could have provided further data on the social hierarchies inherent in libraries. further research is suggested on identifying library career motivation issues from an intersectional perspective.   recommendations   when considering how administrators and supervisors can best foster leaders and managers and support their work forces overall, this research yields several key contributions to the literature. administrators should seek out information about employees’ career phases as part of onboarding by implementing specific strategies. such strategies could help to identify sources of intrinsic and extrinsic motivation, career goals, personal values, and non-work-related interests and responsibilities. employers can then be better positioned to support their staff as they develop.   when working with librarians who identify with the authenticity phase, administrators should work with their employees to develop career goals that are extrinsically based, such as what can be achieved through good work rather than striving for a dream position. administrators should provide these librarians with the latitude to better align their job with work goals, such as giving someone who loves to teach the chance to teach more or take a leadership role in developing an instruction program. these librarians embrace opportunities to lead via projects, committees, and other non-hierarchical leadership work. administrators should proactively engage those librarians in the authenticity phase aged 34–46 in discussions of organizational values and priorities, which may help librarians to feel better aligned with their organizations. due to the high value those in the authenticity phase place on principles, administrators should include them, when possible, in institutional and departmental visioning and goal setting and allow them to align their work to the bigger picture. the majority of survey respondents (60%) identified with the authenticity phase; if this figure is consistent with the general library population, then library administrators would do well to offer numerous informal leadership opportunities and provide inclusive ways for librarians to influence the work culture.          librarians in the balance phase would benefit from early opportunities to develop leadership roles or serve in supervisory roles. these early opportunities better fit with their efforts to prioritize non-work-related responsibilities later in life. training must precede such opportunities to best support and encourage skill development. they should encourage their staff to seek out mentors as they consider potential new roles. administrators should also provide more hands-on support through conversation, feedback, and opportunities for stretch assignments.   for those who identify with the challenge phase, administrators should work with them to find early opportunities to fill a leadership role or supervise others. organizations should implement formal promotion guidelines, which will benefit all employees, and keep this group engaged. librarians in the challenge phase are intrinsically motivated to achieve and strive. they may experience disappointment as newer career librarians continue to advance while they begin to plateau later in life. regardless of age, these librarians continue to crave the latitude to redefine their position or take on new responsibilities to alleviate potential boredom.   whichever career phase a librarian identifies with, administrators should strive to nurture and support young supervising librarians in order to foster better managers and leaders and sustain their interest in the role. such strategies could include offering flexible scheduling to accommodate care duties, options to work part of their time remotely, or adjusting job duties as care duties demand. feeling as though the administration has their backs was the most common response from participants. as one participant shared, “my immediate supervisor . . . [is] very attentive and points out when i'm working towards burnout. the[y] remind me to try to balance everything.” librarians working in an organization that demonstrates it supports all of its employees will be more engaged and motivated. when considering strategies to maintain current levels of support or to address gaps, administrators should certainly get to know their employees to find out what kind of support would best work for them and what future roles and responsibilities best fit with their aspirations.   at their core, the recommendations described here are intended to develop and maintain a highly engaged workforce. clear communication, transparency, and creative problem solving will be key to implementing these recommendations. organizational culture heavily impacts personal behavior and a leader’s ability to bring about change (mainiero & sullivan, 2006a, p. 243). at a fundamental level, such leaders must consider the kinds of changes their organization can withstand as they strive to best support and foster the growth and development of all of their employees.   conclusion   the findings from this study underscore the importance of providing academic librarians flexibility and support as practitioners seek to craft their own career paths. such paths may include advancing into senior leadership positions and back out again, being fulfilled in a non-managerial position that gives practitioners time to spend on care responsibilities, or being in roles that align with their personal values and ethics. not mutually exclusive, this study illustrates how career paths intersect with life events, goals, and experiences. practitioners shift between those challenge, balance, and authenticity phases as their needs evolve over the course of their careers. each phase provides leaders with its own framework through which to communicate with their employees and best meet them where they are, in terms of their priorities and what they value or need at that particular time.   leaders can no longer afford to be complacent when it comes to talent development and retention. as this study highlights, practitioners are looking for more than just a paycheck in recognition of their time and contributions. rather, leaders should consider the intrinsic and extrinsic motivations that guide each of their staff members to provide opportunities that fit with the employees’ career phases and senses of themselves within that phase. these phases provide organizations with a new framework to imagine structuring work, roles, and support within libraries and to allow academic librarians a lens for viewing their careers that replaces the straight linear progression of the past. academic library leaders must recognize the changing needs of their workforce and strive to evolve their practices, policies, and cultures to best support their teams. author contributions   lori birrell: conceptualization, data curation, formal analysis, investigation, methodology, project administration, resources, visualization, writing – original draft, writing – review & editing marcy a. strong: conceptualization, formal analysis, investigation, methodology, project administration, visualization, writing – original draft, writing – review & editing   references   adigwe, i., & oriola, j. (2015). towards an understanding of job satisfaction as it correlates with organizational change among personnel in computer-based special libraries in southwest nigeria. the electronic library, 33(4), 773–794. https://doi.org/10.1108/el-01-2014-0018   barclay, s. r., stoltz, k. b., & chung, y. b. (september 2011). voluntary midlife career change: integrating the transtheoretical model and the life-span, life-space approach. career development quarterly, 39, 386–399. https://doi.org/10.1002/j.2161-0045.2011.tb00966.x   betz, n., & e. (2003). a proactive approach to mid-career development. the counseling psychologist, 31(2), 205–211. https://doi.org/10.1177/0011000002250478   farrell, m. (2013). lifecycle of library leadership. journal of library administration, 53, 255–264. https://doi.org/10.1080/01930826.2013.865390   fink, a. (2013). how to conduct surveys: a step-by-step guide (5th edition). sage.   franks, t. p.  (2017). should i stay or should i go? a survey of career path movement within academic, public and special librarianship. journal of library administration, 57(3), 282–310, https://doi.org/10.1080/01930826.2016.1259200   fyn, a., heady, c., foster-kaufman, a., & hosier, a. (2019). why we leave: exploring academic librarian turnover and retention strategies. association of college and research libraries conference proceedings http://www.ala.org/acrl/sites/ala.org.acrl/files/content/conferences/confsandpreconfs/2019/whyweleave.pdf   lacey, s., & stewart, m. p. (2017). jumping into the deep: imposter syndrome, defining success, and the new librarian. partnership: the canadian journal of library and information practice and research, 12(1), 1–15. https://doi.org/10.21083/partnership.v12i1.3979   lyons, s. t., schweitzer, l., & ng, e. s.w. (2015). how have careers changed? an investigation of changing career patterns across four generations. journal of managerial psychology, 30(1), 8–21. https://psycnet.apa.org/doi/10.1108/jmp-07-2014-0210   mainiero, l. a., & sullivan, s. (2006a). the opt-out revolt: why people are leaving companies to create kaleidoscope careers. davis-black publishing.   mainiero, l. a., & sullivan, s. (2006b). the kaleidoscope career self-assessment inventory: a guide to understanding the career parameters that drive your motivation for work. unpublished manuscript.   mallaiah, t. y., & yadapadithaya, p. s. (2009). intrinsic motivation of librarians in university libraries in karnataka. desidoc journal of library & information technology, 29(3), 36–42. http://dx.doi.org/10.14429/djlit.29.250    montgomery, d. l. (2002). happily ever after: plateauing as a means for long-term career satisfaction. library trends, 50(4), 702–716.   morris, s. (2019). arl annual salary survey 2018–2019. association of research libraries.   o’neill, m. s., & jepsen, d. (2017). women’s desire for the kaleidoscope of authenticity, balance, and challenge: a multi-method study of female health workers’ careers. gender, work, and organization, 26(7), 962–982. https://doi.org/10.1111/gwao.12317   pennell, k. (2010). the role of flexible job descriptions in success management. library management, 31(4/5), 279–290. https://doi.org/10.1108/01435121011046344   power, s. j., & rothausen, t. j. (2003). the work-oriented mid-career development model: an extension of super’s maintenance stage. the counseling psychologist, 31(2), 157–197. https://doi.org/10.1177%2f0011000002250479   ridley, m. (2014). returning to the ranks: towards an holistic career path in academic librarianship. partnership: the canadian journal of library and information practice and research, 9(2), 1–9. https://doi.org/10.21083/partnership.v9i2.3060   sullivan, s., & mainiero, l. a. (2008). using the kaleidoscope career model to understand the changing patterns of women’s careers: designing hrd programs that attract and retain women. advances in developing human resources, 10(1), 32–49. https://doi.org/10.1177%2f1523422307310110   sullivan, s. e., forret, m. l., carraher, s. m., & mainiero, l. a. (2009). using the kaleidoscope career model to examine generational differences in work attitudes. career development international, 14(3), 284-302. https://doi.org/10.1108/13620430910966442       appendix survey instrument   the kaleidoscope career model (mainiero, et al), uses three phases: challenge, authenticity, and balance, as a non-linear approach to understanding the mapping of career trajectories, agency and decision-making power to the individual, rather than the organization, based on the person’s own values and choices. applying this model in the academic librarian context, we seek to better understand where those pivot points exist for professionals, and more broadly how these perceptions impact their sense of satisfaction with their career trajectories, and sense of support they receive from their employers.   this survey contains questions about your experiences and/or feelings concerning how you conceive of and perceive of your career path.   we would like you to complete the whole survey, but you may skip any questions that you don’t feel comfortable answering or can discontinue your participation at any time. the survey data results will be kept for analysis purposes only and will not be released in any publication or report; they will be destroyed once the analysis is complete. only the investigators will have access to your individual responses. all the information received from you will be strictly confidential and will be stored on a password protected local (non-networked) hard drive. you will not be identified nor will any information that would make it possible for anyone to identify you be used in any presentation or written reports concerning this project. only summarized data will be presented in any oral or written reports.   your participation in this project is completely voluntary. you are free not to participate or to withdraw at any time, for whatever reason, without risk. no matter what decision you make, there will be no penalty or impact to your employment. the institutional review boards of the university of arkansas and the university of rochester approved this study. your participation in this survey indicates your consent to these terms.   for more information about this project you should contact: lori birrell by phone at 479-5758443, or by email at: lori@birrell.us or marcy strong by phone at 585-273-2325, or by email at strongstuff@gmail.com.   by clicking on the red arrow below, you are agreeing to participate in this survey.   [the kaleidoscope career model statements and answer scales have been redacted for publication.]   in what ways, if any, do the characteristics of the phase you scored the highest in describe your current thinking about your career path? (open ended)   in what ways, if any, do the characteristics of the phase you scored the highest in not describe your current thinking about your career path? (open ended)   do you feel supported by your library administration? please enter any details you'd like to share in the text box next to your response. yes _______________________________________ no ________________________________________ are you currently a supervisor (defined as managing faculty, staff, students, interns, or volunteers)? yes                        no   if yes, would you like to continue to be a supervisor in the future? yes                        no   if no, would you enjoy the opportunity to become a supervisor in the future? yes                        no   do you consider yourself to be a leader in your organization? (defined here as someone who: does project management tasks, large-scale decision making, coaching/mentoring of others). yes                        no   if no, would you enjoy the opportunity to become an organizational leader in the future? yes                        no   do you have any other thoughts about your career path, the self-inventory tool and kaleidoscope model, or this topic more generally that you’d like to share? (open ended)   the following are demographic questions: what kind of library do you currently work in? 4 year, doctoral degree granting university or college 4 year, masters degree granting university or college 4 year, bachelor degree granting university or college 2 year, community or vocational school other (please describe below)   what area of librarianship do you currently work in? (for this question, we’re asking about your primary job duty. department heads, please indicate the functional area you work in) administration it public services technical services special collections/archives other (please enter your area of librarianship in the text box.)   how many years have you worked in the library science profession? less than 2 years 2-5 years 6-10 years 11-15 years 16-20 years 25+ years   please select your age range. 22-27 28-33 34-40 41-46 47-52 53-58 59-64 65-70 71-75 75+   please identify your gender. male female gender variant/non-conforming other prefer not to answer       microsoft word comm_brettle.doc evidence based library and information practice 2007, 2:2  123 evidence based library and information practice     commentary    transforming the profession: reflections on evidence based library and information  practice, 4th international conference      alison brettle  research fellow/associate editor evidence based library and information practice  salford centre for nursing, midwifery and collaborative research  institute of health and social care  university of salford, salford, uk  e‐mail: a.brettle@salford.ac.uk      © 2007 brettle. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.        transforming the profession was the 4th  international evidence based library and  practice (eblip) conference and promised a  varied and inspiring mix of presentations and  keynote speakers together with opportunities  for meeting like‐minded colleagues from across  the world.  it was my 4th eblip conference and  for me, also offered a chance to hear people  speak whose work i have admired for some time,  catch up with colleagues and friends from  previous eblip and other conferences, and most  importantly, meet members of the eblip journal  team face to face for the first time.    the program offered something for all types of  libraries and librarians.  it  covered the use of  evidence based library and information practice  for a wide range of themes including  management, user needs, building the evidence  base, outcomes assessment, collection  development, decision making, instruction,  special libraries, finding evidence, education  and professional development and research in  progress.  criticism of previous eblip  conferences is that it has been dominated by  health library representation.   this should not  be surprising, given that the paradigm  originated in the health field, but this time the  number of presentations from other sectors  provided evidence of the spread of the  paradigm across the profession.  as a speaker in  two sessions and moderator of two more, i did  not always get to the slots i would have chosen.   this was not necessarily a bad thing as it  provided the chance for me to step outside my  comfort zone and listen to interesting speakers  in different areas.  some of these are highlighted  below.    the conference opened with an inspiring and  thought provoking keynote speech from  margaret haines on professionalism and  evidence based practice: reflections of a  university librarian.  margaret has had a varied  mailto:brettle@salford.ac.uk http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:2  124 career both in the united kingdom and canada  and suggested that for her, ebp is a means of  ensuring that what is being done is worthwhile  and is being done in the best possible way.  she  provided examples of attempts she had made to  implement evidence based practice at a national  and local level and described how, worryingly,  efforts that had appeared successful in one post  disappeared when she left.  this highlighted a  potential problem for evidence based library  and information practice in that it is of little use  leaving things to the “early adopters” or  champions if their efforts are going to be wasted  when they move on.  to be sustainable rather  than a passing fad, evidence based library and  information practice needs to be embedded and  recognised at all levels.    “choices in chaos”, a presentation by michelynn  mcknight, describing a project examining the  library services provided in community wide  disasters, is a good demonstration of an  expansion of the paradigm outside health.  it  also opened my eyes in that i had never  anticipated a need for library services in the  aftermath of hurricanes and other community  disasters, let alone considered there was a need  for an evidence base.  michelynn presented  cases where librarians provided support to  members of the public by establishing mobile  library services that allowed people left  homeless after disasters to use the internet to  sort out their lives.  her research will use a  grounded theory approach to study cases such  as these to provide an evidence based training  package for librarians who may need to provide  such services.    examining the outcomes of services was a  welcomed theme in a number of the  presentations i attended.  examining,  monitoring or assessing outcomes are key if  librarians are to truly measure the impact of  their services.  there were a number of  examples of outcome measurement in academic  library settings including ngata, toda and  kytomaki (students patterns of library use and  their learning outcomes), eric ackermann  (libqual and the evolution of library as a  place) and byrd and squires (the readiness of  the library director community to measure  outcomes rather than inputs and outputs).   rowena cullen, associate professor at victoria  university, new zealand, and one of the  keynote speakers, also visited this theme in her  talk about evidence and e‐government.  she  suggested that the use of online technology in  government has expanded rapidly but  measuring and evaluating whether this  technology has actually worked or made a  difference has lagged behind.  commenting that  it was difficult to measure effectiveness in e‐ government (but noting that this was not an  excuse for not trying) she suggested that the  traditional hierarchy of evidence approach was  unlikely to be relevant – or at least the  methodologies at the top of the evidence  hierarchy.   methods such as policy or program  evaluation or descriptive and formative  evaluation may be more appropriate.  one of the  issues that can be problematic is the issue of  attribution: if you apply evidence or changes to  e‐government how do you know which bit is  making the impact?  this is not a problem  unique to e‐government; however, it is an issue  relevant to outcome measurement in many areas.   she suggested that in policy evaluation it is  necessary to look at what is being measured and  examine whether the program is doing what it is  intended to do. confounding variables need to  be eliminated, all impacts need to be followed  through, measured and evaluated individually.   again, this is not unique to policy evaluation  but is a problem faced by any researcher trying  to carry out an evaluation.   for evidence based  librarians perhaps its good news as it means  there should be many examples in a range of  sectors upon which we can draw.    but what does it mean to be an evidence based  librarian?  this is a question under examination  by a team of librarians from the university of  queensland in australia, led by helen partridge  and presented by gillian hallam.  they are  carrying out a qualitative study to examine how  practitioners experience and conceive eblip.    evidence based library and information practice 2007, 2:2  125 to date, pilot interviews with a small number of  librarians have led to four categories  of  evidence based librarians: those who believe  ebl is a professional accident,  that ebl is  learning from and using research, that ebl  improves what the librarian does or what the  library offers, and those who believe that ebl is  integral to their job – ebl is what they do.   other presentations provided good examples of  librarians implementing an evidence based  approach to answer questions in their library or  service (perhaps they would fit in the third  category above!).  these included frank  cervone, who used an evidence based approach  to make changes to the design of the academic  library website; lisa toner, who examined why  some students made little or no use of the  academic library and the services offered to  them in order to provide an equitable service to  users; and matthew trekson, who used citation  analysis to determine whether the databases  provided by an agricultural library service met  the needs of the researchers using the library.    but is evidence based library and information  practice clear and simple or wrong?  this was  the topic debated by two eminent speakers:  andrew booth and t. scott plutchak. andrew  booth, chair of the international program  committee, not surprisingly argued that the  eblip approach was useful for evaluating many  questions faced by the profession.  for those  who have read andrew’s work or seen him  speak at other conferences, it would have been  worrying to see him argue in any other way.  t.  scott plutchak, from the university of alabama  at birmingham, on the other hand argued that  the evidence based model was inappropriate for  answering the “big questions” facing  librarianship.  once thrown open to the floor the  debate became quite heated to some who argued  that eblip was not new: it was something that,  as professionals, all librarians did as a matter of  course. others clearly disagreed that searching  for evidence and applying it to their practice  was not happening and should be embedded  within everyday practice.  the final vote was  split between the “fors” and the “againsts” with  a fair number of abstentions.  as one of the  abstainers i could see both sides of the argument:  evidence based library and information practice  is a useful tool and something that all librarians  should be involved in but sometimes it does not  appear possible to implement it.    for me, the conference lacked actual evidence  that librarians could take away and use in their  own practice.  this may be a reflection of the  presentations i attended and further  examination of all the presentations on the  conference website http://www.eblip4.unc.edu/  may prove me wrong.  overall there was a  positive shift away from discussions of  definitions of evidence based library and  information practice and its progress (which  have featured at previous eblip conferences) to  a demonstration that eblip is operational in a  wide range of sectors in many parts of the world.   however i would like to have seen more good  quality, generalisable research studies which  would help to build the evidence base for  practitioners.  i appreciate that finding time and  money to undertake research studies is a key  issue and barrier to research and one which i am  not sure how to overcome,  but perhaps as  rowena cullen pointed out, “being difficult is  not an excuse for not doing”.  i would also like  to have seen more systematic reviews.  as  author and team member of a range of  systematic reviews, i am undoubtedly biased in  this area.  some may argue that there is  insufficient evidence to review, however  reviews in lis exist and andrew booth spoke  about the potential role of qualitative systematic  reviews in eblip.  i also believe librarians have  a major role to play in systematic reviews.  they  are not only in a position to provide the  information expertise for systematic reviews, as  many librarians are doing in the health and  social science fields, but their skills put them in a  good position to undertake systematic reviews  in their own field.  systematic reviews involve  refining questions, undertaking systematic and  comprehensive literature searches, critical  appraisal of the evidence and synthesising the  results.  a large unspoken part of systematic  http://www.eblip4.unc.edu evidence based library and information practice 2007, 2:2  126 reviewing includes managing the results  obtained often using reference management  packages.  systematic reviews are frequently  carried out in teams.  librarians can refine  questions, many have in‐depth searching skills,  knowledge of resources, and knowledge of  reference management and already work in  teams.  some already have critical appraisal  skills, and there were opportunities at the post  conference workshops to develop these together  with skills in data synthesis.  if librarians  teamed up and pooled their skills and  knowledge it would be possible, and hopefully  not too difficult, to undertake more systematic  reviews.   it would be great to see them  presented at eblip5 in stockholm!        research article   an evaluation of methods to assess team research consultations   ashlynn kogut education & social sciences librarian texas a&m university libraries college station, texas, united states of america email: awkogut@library.tamu.edu   pauline melgoza science & engineering librarian texas a&m university libraries college station, texas, united states of america email: p-melgoza@library.tamu.edu   received: 19 dec. 2019                                                                             accepted: 27 may 2020      2020 kogut and melgoza. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29698     abstract objective – due to the individualized nature of consultations and institutional constraints, research consultations can be challenging to assess. at texas a&m university libraries, subject librarians use research consultations to teach information literacy to upper-division engineering student teams working on a technical paper project. this paper describes an action research project designed to evaluate which assessment method for consultations with student teams would provide the most actionable data about the instruction and the consultation logistics as well as optimize librarian time. methods – for three semesters, we simultaneously used up to four consultation assessment methods: one-minute papers, team process interviews, retrospective interviews, and questionnaires. we followed the action research cycle to plan the assessments, implement the assessments, reflect on the data collected and our experiences implementing the assessments, and revise the assessments for the next semester. each assessment method was distributed to students enrolled in an engineering course at a different point in the technical paper project. the one-minute paper was given immediately after the consultation. the team process interviews occurred after project deliverables. the questionnaire was distributed in-person on the last day of class. focus groups were planned for after the assignment was completed, but low participation meant that instead of focus groups we conducted retrospective interviews. we used three criteria to compare the assessments: information provided related to the effectiveness of the instruction, information provided about the logistics of the consultation, and suitability as an assessment method in our context. after comparing the results of the assessment methods and reflecting on our experiences implementing the assessments, we modified the consultation and the assessment methods for the next semester. results – each assessment method had strengths and weaknesses. the one-minute papers provided the best responses about the effectiveness of the instruction when questions were framed positively, but required the most staff buy-in to distribute. the team process interviews were time intensive, but provided an essential understanding of how students think about and prepare for each progress report. recruiting for and scheduling the focus groups required more time and effort than the data collected about the instruction and logistics warranted. the questionnaire provided student perspectives about their learning after the assignment had been completed, collected feedback about the logistics of the consultations, was easy to modify each semester, and required minimal librarian time.   conclusion – utilizing multiple assessment methods at the same time allowed us to determine what would work best in our context. the questionnaire, which allowed us to collect data on the instruction and consultation logistics, was the most suitable assessment method for us. the description of our assessment methods and our findings can assist other libraries with planning and implementing consultation assessment.     introduction   research consultations provide personalized instruction that is not available during one-shot instruction sessions, but they can create staffing challenges due to the amount of time librarians spend preparing for and providing the consultations (faix, macdonald, & taxakis, 2014). assessing research consultations can help librarians to design the service, to justify the time they spend providing consultations, and to determine the value of providing consultations (fournier & sikora, 2017). despite the benefits of assessing research consultations, formal consultation assessment seldom occurs in academic libraries due in part to the personalized nature and diverse learning goals of consultations (fournier & sikora, 2017).   at texas a&m university libraries, multiple librarians provide research consultations for engineering teams in a writing-intensive course. multiple librarians devote a significant amount of time each semester meeting with the teams, and the number of teams scheduling consultations has been increasing. we wanted to assess these consultations in order to gather evidence that allowed for continuous improvement of the consultations and that justified the staff time and library space needed to provide the consultations.   since consultations are under-represented in the assessment literature (fournier & sikora, 2015; miller, 2018; savage, 2015), guidance was limited about the most effective assessment methods. to address this gap in the literature, we developed an action research project that evaluated the suitability of four assessment methods for the research consultations provided to the engineering teams.   context   at texas a&m university, the engineering technology and industrial distribution department requires students to take a junior level, writing-intensive engineering course. one of the course writing components is a technical paper. the technical paper is a 16-page paper that focuses on a mechanical power transmission component (e.g., bearing, lubrication, gear box) in a product (e.g., airplane, car, wind turbine, diesel generator). this is the first time many students need to find technical information outside the manuals provided in their engineering labs. the junior and senior engineering students work in teams of four to complete the assignment over a 12-week period.   research consultations have been integrated into the course for the last 10 years. in 2016, the course instructor made the research consultation a requirement and recommended that students attend a second consultation. most students scheduled research consultations between weeks 6 and 10 of the semester. the learning outcomes for the one-hour research consultation were for students to become aware of the breadth of sources available for them to use in their paper and to become efficient in searching these sources. an online research guide and video tutorials were available to students as supplementary resources. a detailed description of the instruction topics covered during the consultation is available in a previously published conference paper (melgoza, 2017).   initially, all consultations were provided by the second author – a science and engineering librarian – and another science and engineering librarian. as course enrollment increased to a maximum of 180 students, they could no longer provide all of the consultations and the second author began to recruit additional librarians (see table 1). by fall 2017, six librarians (four science and engineering librarians and two non-engineering librarians) and a library assistant taught the teams. in addition to the one-hour consultation, librarians spent one to two hours preparing and had the possibility of a one-hour follow-up consultation. during the six weeks the consultations were held, one of the library’s consultation rooms was taken offline to accommodate the consultations.     table 1 student enrollment and consultation statistics  semester students enrolled in coursea total teamsa teams who met with a librarian fall 2017 173 49 27 spring 2018 171 47 47 fall 2018 169 44 43 aenrollment numbers and number of teams were provided by the course instructor.     due to the increasing amount of librarian time devoted to the consultations and the demand for library space, the second author started considering alternate ways of providing the consultations. before making changes, the second author wanted to assess the consultations. we had robust usage statistics about the numbers of students coming for consultations and use of the course guide (stephens, melgoza, hubbard, pearson, & wan, 2018), but this data provided no information about the effectiveness of the instruction or the logistics of the consultations.   to plan the assessment, the second author asked the first author for assistance because she had assessment experience and was a neutral party who did not provide consultations for this course. from the outset, analyzing student papers would not be an option because the course instructor preferred not to share the final student papers with the librarians. after an initial review of the consultation assessment literature, we determined that we did not have a clear path for determining the best assessment method. we wanted a method that would allow us to know more about the information students were remembering and applying from the consultations, as well as how students felt about the consultation experience. thus, we developed this action research project to evaluate different research consultation assessment methods.   literature review   consultation assessment   librarians have used various methods to assess research consultations including surveys (e.g., butler & byrd, 2016; drew & vaz, 2008), usage statistics (fournier & sikora, 2015), citation analysis (e.g., hanlan & riley, 2015; reinsfelder, 2012), pre and post testing (e.g., sikora, fournier, & rebner, 2019), focus groups (e.g., watts & mahfood, 2015), interviews (e.g., rogers & carrier, 2017), mystery shoppers (e.g., newton & feinberg, 2020), and examining students’ course grades (e.g., cox, gruber, & neuhaus, 2019; newton & feinberg, 2020). while most of these articles discuss the limitations of the particular method, direct comparison of different consultation assessment methods is limited. even when researchers used multiple consultation assessment methods, the discussions focused on the findings of the method, not the utility of each method (e.g., hanlan & riley, 2015; newton & feinberg, 2020; watts & mahfood, 2015).   only fournier and sikora’s (2015) scoping review provided an explicit discussion of the strengths and weaknesses of the three consultation methods they identified: usage statistics, surveys, and objective quantitative methods. usage statistics are useful for understanding the demand and planning the service (fournier & sikora, 2015). surveys can show user satisfaction and assist in making modifications to the service, but are limited by their subjective nature and positively skewed results (fournier & sikora, 2015). statistics and surveys are not the best methods to use to provide evidence of the outcomes of research consultations. rather, objective quantitative methods, like pre/post testing, provide a better way to assess the impact of consultations on student learning (fournier & sikora, 2015). since the use of objective quantitative methods would be challenging in our context, we looked for other ways to assess the outcomes and logistics of consultations.   qualitative methods offer an alternative way to assess the outcomes and logistics of consultations. both interviews and focus groups have been used to provide evidence of what students believed were the outcomes of research consultations (watts & mahfood, 2015; yee et al., 2018). interviews can be an initial step in creating a survey and can provide detailed information about outcomes students felt as a result of consultation (yee et al., 2018). open-ended survey questions can elicit responses about how students perceive the value of consultations (magi & mardeusz, 2013).   action research and assessment   action research is a method of inquiry that aims to improve practice (malenfant, hinchliffe, & gilchrist, 2016; reason & bradbury, 2006; suskie, 2018). action research projects focus on an issue derived from a specific context, are led by the librarian involved in the service, incorporate stakeholders in their design, make changes immediately based on the results, and utilize an evolving design (coghlan & brydon-miller, 2014; malenfant et al., 2016; woodland, 2018).   action research aligns well with library assessment projects. the unique contextual factors within an academic library often drive assessment projects. librarians and other stakeholders involved in the delivery of a service plan, evaluate, and make changes to the service based on the assessment data.   action research has been used in the library and information science discipline as the basis of assessment projects. multiple researchers have used action research to assess information literacy instruction (e.g., insua, lantz, & armstrong, 2018; lemire, sullivan, & kotinek, 2019; margolin, brown, & ward, 2018). in addition, researchers have used action research for other types of assessment including the enhancement of services (kong, fosmire, & branch, 2017) and planning library spaces (brown-sica, 2012; brown-sica, sobel, & rogers, 2010). using action research to determine a way to assess our consultations would allow us to build upon the hallmarks of the assessment cycle, while incorporating the aspects of action research that would keep our research design flexible as we encountered new information. our study adds to the literature on consultation assessment by directly comparing four assessment methods in terms of the data collected about the instruction and logistics as well as the ability to implement the method. aims the aim of this action research project was to determine which assessment method would be the best way for us to collect actionable feedback in order to continuously improve the team research consultations. the goals of the assessment were to assess the effectiveness of the instruction and the logistical aspects of the consultation service in order to maximize the use of available resources.    each assessment method was evaluated on three criteria: information provided related to the effectiveness of the instruction, information provided about the logistics of the consultation, and suitability as an assessment method. we defined effectiveness of the instruction by evaluating if what students reported learning from the consultation was related to the consultation learning outcomes. the logistics of the consultation was defined as student opinions about the timing of the consultation in relation to the assignment milestones, the length of the consultation, and the format of the consultation. the suitability of the assessment method was determined by considering the usefulness of the information collected and the amount of staff time needed to implement the assessment.   methods   we planned to implement one-minute papers, team process interviews, and focus groups as our assessment methods. one-minute papers are frequently used as an assessment technique in library instruction sessions (bowles-terry & kvenild, 2015). given their popularity in classroom assessment, we found limited discussion of the use of one-minute papers as an assessment technique for consultations. one-minute papers typically consist of two questions: one focused on what students learned and the other focused on what was confusing.   we chose the one-minute paper because it would allow us to assess students’ recall of information immediately after instruction. however, our institutional review board (irb) approval come too late in the fall 2017 semester to use the one-minute papers immediately after the consultation. instead, we used the irb-approved one-minute paper questions on the end of semester questionnaire in fall 2017. we did not initially plan to use a questionnaire to assess the consultations, but took advantage of an opportunity provided by the course instructor. once we starting using the one-minute paper after the consultation, we changed the questions on the questionnaire.   interviews and focus groups were chosen because we thought they would provide more in-depth responses from students. these methods had been used by other universities examining how consultations impact student learning (watts & mahfood, 2015; yee et al., 2018). two studies that used citation analysis concluded that qualitative data from the students about their research process would have been helpful to understand the results (hanlan & riley, 2015; sokoloff & simmons, 2015). based on these studies, we decided to use team process interviews to explore the process that teams used to find information at different points in the assignment. we planned to use focus groups in order to engage students in conversation about the consultations.   action research cycle   we used action research as a way to evaluate the assessment methods for team research consultations. action research includes a cycle of planning, acting and observing, reflecting, re-planning, acting and observing, and reflecting (kemmis, mctaggart, & nixon, 2014, p. 18). we planned how to collect data using one-minute papers, questionnaires, focus groups, and team process interviews. we acted and observed our implementation of the assessment methods. then, we reflected on the utility of the methods, compared the results of the assessments, and made changes to the assessments and the consultations. reflection occurred throughout the semester. we talked at least once a week about how the assessments and the consultations were going. small changes to the consultations and the assessment methods were made immediately based on the assessment data and personal observations. larger changes to the consultations were made after each semester.   stakeholders   we had three groups of stakeholders: the course instructor, librarians, and students. after we informed the course instructor of our assessment project, he offered his support and willingness to assist as needed. the instructor gave us a portion of the last class each semester to distribute the questionnaire. each semester we shared student responses to illuminate students’ confusion with the project and our changes in instruction.   librarians assisted with the data collection for the one-minute papers and the questionnaires. after collecting the one-minute papers, some librarians reviewed the responses to see what the students retained. some of the group reviewed the questions on the questionnaire. librarians received a summary of themes from the assessments as well as representative responses prior to the start of consultations for the next semester. the group discussed changes to make for the consultations based on the findings.   the students were not as involved as one would expect for an action research project. prior to distributing the questionnaire, we shared our past findings and asked students to share their honest assessment of our instruction and changes.   data collection and data analysis   our data collection spanned three semesters; it began in fall 2017 and ended in fall 2018. we used each assessment method at a different point in the assignment (see figure 1). the participants were junior and senior engineering students who were currently enrolled in the engineering course     figure 1 timeline of planned data collection and assignment milestones.     one-minute papers   the one-minute paper assessed the immediate impact of the instruction. one-minute papers were distributed to individual students after their team’s research consultation for two semesters, spring 2018 and fall 2018. the first author met individually with each librarian conducting research consultations to explain the data collection process and answer any questions. the librarian who conducted the consultation distributed the one-minute paper in hardcopy to students immediately after the consultation. the librarian stepped away from the table to give students privacy. students’ participation was voluntary, no incentives were used to increase participation, and student responses were anonymous. the librarian collected the one-minute papers and gave them to the first author for transcription and data analysis. we received 77 completed one-minute papers (see table 2).   in spring 2018, we piloted four versions of the one-minute paper in order to determine the questions that would provide the most useful information. librarians gave the different versions to the students randomly. table 3 presents the questions on the four versions and number of responses per version. after analyzing the spring 2018 data, we found that student responses to question 2 on version 2 were the most useful for highlighting additional topics to cover during the research consultation. the first question on each of the versions elicited similar responses from students. therefore, we only used version 2 of the one-minute paper in fall 2018.   the first author transcribed and analyzed the data from the one-minute papers. the coding followed the qualitative coding procedures outlined in creswell and guetterman (2019): noting words and phrases, assigning a descriptive code to the phrase, defining each code, merging similar codes, and developing themes by aggregating the codes. codes focused on resources and information that students described learning during the consultation. the codes were both descriptive terms and in vivo codes, which are code labels that use the same language from the student’s responses (creswell & guetterman, 2019). each question was coded independently. then, the codes for the first questions on each version and the codes for the second questions on each version were pooled to develop themes. the first author coded the data in atlas.ti each semester. after the initial coding each semester, we met to discuss the themes that emerged from the data.     table 2 participants by semester  semester one-minute papers team process interviews questionnaires retrospective interviews total number met with librarian did not meet with librarian fall 2017 n/a n/a 57 38 19 0 spring 2018 52 7 (3 teams) 68 57 11 3 fall 2018 25 3 (1 team) 95 93 2 n/a total 77 10 (4 teams) 220 188 32 3     table 3 one-minute paper questions version questions spring 2018 responses fall 2018 responses 1 1.      what do you think you will do differently after meeting with a librarian? 2.      what is still unclear about using library resources for your assignment? 17   2 1.      what did you learn? 2.      what would you like to learn more about? 15 25 3 1.      what was helpful? 2.      what was not helpful? 10   4 1.      what was the most important thing you learned during this consultation? 2.      what question remains unanswered? 10       team process interviews   team process interviews investigated how teams worked through the assignment. the second author recruited teams during the team’s initial research consultation with her. project reports were due about every two weeks, and the interviews were scheduled for the day after a project report was due, for a total of three interviews. the interviews were held in one of the library’s consultation rooms and snacks were provided. the same five questions were asked during each interview. questions focused on what type of information the team used to meet the requirements of the previous progress report and what kind of information the team needed to find for the next progress report (see appendix a). the second author took notes during the interview; interviews were not recorded. after each team process interview, we debriefed to discuss the data. the notes were analyzed for trends that could inform the instruction and the logistics of the consultation.   four teams (a, b, c, and d) participated in the team process interviews. all members of a team were encouraged to attend each interview. in spring 2018, the second author recruited three teams. team a had the same, single student attend all sessions. team b had four members attend the first session, three the second, and two the last session. team c had the same two students attend all sessions. in fall 2018, only one team (d) was recruited. three students attended the first session, and the same two students attended the last 2 sessions. the team process interview took 10 minutes and afterwards the second author asked the students if they wanted to stay for an additional consultation. all of the teams did stay for the consultation. they discussed their outline and asked for additional tips for locating the next set of information.   questionnaires   the questionnaire gathered feedback from the students who had met with a librarian and from the students who had not met with a librarian. we collected data using the questionnaire in fall 2017, spring 2018, and fall 2018. the paper questionnaire was distributed to the students attending the last class session of the semester. students received their graded project after the questionnaire was completed. a food incentive was provided, but students were not required to participate in order to have the incentive. the questionnaires were printed on different colors of paper to better keep track of those who met with a librarian and those who did not meet with a librarian. all questionnaire responses were anonymous. we collected 220 questionnaires (see table 2).   since we were using action research, the questions naturally changed as we instituted modifications based on the questionnaire responses (see appendix b). we dropped questions and added new ones. the fall 2017 questionnaire questions were based on the one-minute papers questions with added questions about citations. the spring 2018 questionnaire for students who had a consultation had questions that addressed the effectiveness of the instruction and the logistics. the questions included the information students learned from their consultation, what they could apply to future courses, and their feedback on having another team present during the consultation. for the students who did not meet with a librarian, we asked if any member of the team met with a librarian and if they shared any information, how they chose their topic, the search process and where the information was found, and if they were aware of the course guide.   for the fall 2018 questionnaire, we only made changes to the questions about the logistics of the consultation. the new questions were about their experience with a shorter consultation time, how often they met with a librarian, if they had needed to consult with a librarian in another engineering course, and feedback on the online tutorials. for the group that did not meet with a librarian, the new questions were about the tutorials and if they struggled to find information for their project.   after data transcription, the data analysis for the questionnaires followed a similar procedure to the one-minute papers. for open-ended questions, the first author used the qualitative coding procedures outlined in creswell and guetterman (2019). all responses for each question were pooled for analysis, but each question was analyzed separately. the first author used atlas.ti to apply the code labels to the student responses each semester. the list of code labels in atlas.ti provided a starting point for coding each semester, and additional codes were added when needed. for the questions that had a closed ended component, the first author used a set coding scheme (e.g., yes, no, maybe) to code the closed ended answer and analyzed the data using descriptive statistics.   focus groups/retrospective interviews   planning focus groups   after the team project was completed, we wanted to use focus groups to solicit feedback on the instruction and the timing of the consultation. our focus group recruitment was unsuccessful in fall 2017. we attempted to recruit students who had a research consultation via email, but we did not have an email address for every student who met with a librarian. we scheduled multiple time slots during the day and evening in the last full week of classes and offered an incentive of pizza, but no students indicated interest in participating. after the unsuccessful email recruitment, we tried to hold focus groups during the class meeting time on the last class day, but again no students were interested in participating.   for the spring 2018, we changed our recruitment method and added additional incentives. at the consultation sessions, we obtained the emails of all the present team members. the focus groups were scheduled for the days immediately following the questionnaire distribution. we recruited in-person when we went to distribute the questionnaires on the last class day. if a student indicated interest in participating, we handed them a slip of paper with instructions for signing up. we also offered a $10 gift card in addition to lunch. using the new recruitment technique, we had three students volunteer.   conducting and analyzing retrospective interviews   since we did not get enough volunteers to hold focus groups, we held two retrospective interviews. the first interview had two participants. the second interview only had one participant because this student had been a part of the team process interviews. we felt this student’s experience would be different from the other students and wanted to keep the participants with similar consultation experiences together. the first author conducted the interviews, and another library staff member, who did not provide consultations, observed. the same protocol was used for both interviews (see appendix c). the retrospective interviews were audio recorded. both the first author and the library staff member took notes during the interview. as soon as possible after each interview, the first author transcribed the notes and added additional details and observations. a summary-based approach was used for the data analysis (morgan, 2019). to do this, the first author compared the responses from each interview in order to summarize the information that could inform the instruction provided and the logistics of the research consultation.   results   the results are discussed by data collection method. for each method, we highlight how the collected data showed the effectiveness of the instruction, informed the logistical aspects of the consultation, and contributed to our analysis of the suitability of the method.   one-minute papers   relating to the effectiveness of the instruction, the one-minute paper responses to the first question on each version (see table 3) fell into three primary themes: resources and services, how to use the library or resources, and related to the assignment (see table 4).   the analysis of one-minute paper responses to the second question on each version showed certain questions would elicit more actionable responses. students did not answer questions that were negative in nature (e.g., “what is still unclear about using library resources for your assignment?”, “what was not helpful?”). typical responses to these questions were “none,” “n/a,” or positive responses, like “everything was explained thoroughly.” the positively framed “what would you like to learn more about?” question provided the most actionable responses. the responses were primarily about the assignment, but a few were about utilizing library resources. for example, “proper citations” and “maybe more specifics on key search words and which phrases might be the most effective in searching.”     table 4 example student reponses for one-minute paper themes theme examples of student reponses resources and services ·        databases ·        library resources ·        refworks ·        endnote how to use the library or resources ·        using combinations of words to search ·        navigating through databases related to the assignment ·        topic needs to be narrowed down ·        how to structure the paper ·        best way to organize/approach paper     the one-minute papers were an effective method for collecting data about the immediate effectiveness of the instruction, but not about the logistics of the consultation. therefore, this method would not fit both of our needs for effectiveness of the instruction and logistics of the consultation. this assessment method also had implementation challenges. while all librarians providing the consultations were willing to hand out the one-minute papers, not everyone did so consistently. in addition, we had to coordinate a centralized location to collect the responses and plan time to transcribe the data.   team process interviews   all four teams had different topics but still approached the project similarly. initially, the teams felt that the first consultation was sufficient for them to complete their project. though they agreed for their team to be interviewed, they really wanted continued access to the librarian in case they needed additional instruction.   in the first session of the team assessment, none of the teams demonstrated a complete understanding of the scope of the project. with each session, the teams gained confidence with understanding the scope of the project and used their newfound searching techniques to find information for the forthcoming project sections. other times, they struggled to compose searches for previously unexplored aspects of their topic.   as suspected, the teams were not following the course instructor’s timetable for writing the paper; they did not understand or embrace how the progress reports schedule was leading them to write their paper at a manageable pace. sometimes the teams did not submit what the course instructor required because they had competing assignments from other courses. teams appreciated having a regularly scheduled, structured appointment to discuss the project and get more focused information for each project submission. there was no consensus as to when a second library consultation should be offered.   this method was better suited to understanding how the teams work on the project, and therefore, should not be used for measuring the effectiveness of the instruction and logistics of the consultation. the team process interviews were an effective method for collecting data about how the teams’ research needs changed during the project, but did not need to be continued once we found similar results both semesters. the team process interviews would be good to use again if there were fundamental changes to the assignment. the high time commitment was also a disadvantage. while all teams could schedule up to two consultations, the personalized assistance given to the teams who agreed to be interviewed could give those teams an advantage.   questionnaires   at the end of the semester, students’ responses to the most important things they learned during the consultation focused on three themes: the assignment, awareness of library resources, and utilization of databases and resources. when answering about what they learned that could be used in future courses, student responses fell into two primary themes: awareness of library resources and utilization of databases and resources.   responses from students who did not personally meet with a librarian, but had team members who met with a librarian, supported the themes. these students responded that their teammates shared which databases to use and advice for choosing a topic.   in regard to the logistics of the consultations, student responses showed that they appreciated the personalized nature of the experience, including the focus on only their topic and the ability to ask specific questions about their topic. feedback to the idea of a librarian meeting with multiple teams at once was mostly negative. after shortening the consultations to 30 minutes in fall 2018, the majority of the responses indicated that the 30-minute length of the consultation was sufficient. however, a third of the responses expressed that students would like longer consultations or that 30 minutes is only sufficient in certain cases.     table 5 example student reponses for questionnaire themes theme examples of student responses assignment ·        how to layout our paper and what to focus on ·        ability to narrow down a research topic using library resources awareness of library resources ·        how to access the research databases ·        about the wide variety of sources that were available ·        i could go there for research paper help. i had no idea that was possible. utilization of databases and resources ·        keywords to search with to find sources directly related to my topic ·        research more effectively with credible resources     the questionnaire assessment method was suitable for assessing both the effectiveness of the instruction and the logistics of the consultations. this method allowed us to modify the questions each semester to collect data needed at that particular time. the challenges with the questionnaire method were writing questions in a way that elicited useful student responses, recruiting library staff to help with the in-class data collection, and deciphering and transcribing students’ handwritten responses. collecting data required a half-day time commitment of multiple librarians, but we were able to gather evidence from a meaningful portion of the students at one time. the continued use of this method depends on the continued support of the course instructor to allow us to collect data during class time.   retrospective interviews   in regard to the effectiveness of the instruction, we learned about what students believed were the outcomes of the consultation and the amount of general library information to be covered. students saw assistance in helping them decide which topic to choose as the primary outcome of the consultation. students felt that the consultations helped them have a better understanding of the types of information sources that were appropriate for the course paper. this included sources that are not necessarily scholarly, like patents, websites, and contacting industry people directly. the students disagreed about the amount of general library information that should be provided during the consultation.   the retrospective interview participants’ responses about the logistics of the consultations gave us additional insight about student expectations about the consultations and accessing resources. participants mentioned preparation both in regard to the librarian and the students. students expected the librarians to already know the good resources for the topic and share their personal experiences with the project. students also realized they personally needed to prepare beforehand to fully take advantage of the research consultation. the participants strongly preferred that a librarian only meet with one team at a time. students gave no clear answer about the timing of consultations.   focus groups were not a suitable method to collect assessment data for our research consultations. while we appreciated the in-depth responses provided by students about the effectiveness of the instruction and the consultation logistics, the challenge of recruiting students outweighed the insights we gained. due to the low participation, the amount of coordination required, and scheduling conflicts, we never conducted any focus groups and we only conducted retrospective interviews one semester.   discussion   comparing data for effectiveness of instruction   we found that the one-minute papers and the questionnaires were the best methods to assess the effectiveness of the instruction. students’ answers to both of these assessments aligned with the learning outcomes for the consultations: learn about the breadth of resources provided by the library and how to search the resources. mapping student responses to learning outcomes is one way to analyze one-minute papers to determine if the instruction is meeting its objectives (bowles-terry & kvenild, 2015). in the one-minute papers and the questionnaires, the same themes were found in student responses about what was learned, which demonstrated in our case the timing of the assessment did not influence the responses. one strength of the questionnaire was that the timing at the end of the semester allowed for a better understanding of whether students continued to use what they learned in the consultation (goek, 2019). however, when only using the questionnaire at the end of the semester, any changes to instruction had to wait until the next semester.   the retrospective and team process interviews provided the most detailed information about student beliefs about what they learned. from the retrospective interviews, we learned how students applied information from the consultation to complete the assignment, and the team process interviews helped us understand how students worked through the project and the challenges they encountered with finding information for the assignment. in order to make better use of the interviews, we could ask questions to clarify and provide context to the questionnaire responses, like other researchers have suggested (hanlan & riley, 2015; sokoloff & simmons, 2015).   comparing data for logistics of consultations   in terms of assessing the logistical aspects of the consultation, the questionnaire again provided us with the best assessment method. the questionnaire was easy to modify each semester to solicit student feedback on ways to modify the consultation service. the use of questionnaires to inform consultation logistics supports newton and feinberg’s (2020) finding that a survey was a good method to assess student satisfaction with the consultations in regard to scheduling and the location of the consultation.   the one-minute paper offered no information about the logistics, but the team process interviews and the retrospective interviews provided some student feedback about the logistics. while the retrospective interviews provided us with preferences about the consultation logistics, self-selection bias might have influenced the results. students were aware of us seeking information about meeting with multiple teams at once prior to the focus group. students participating in the team process interviews might have been influenced by the desire to have more personalized assistance. self-selection bias is a limitation that other researchers using interviews have also noted (e.g., rogers & carrier, 2017).   comparing utility of methods   in regard to the utility of each method in our context, we determined that the questionnaire was the best method for our environment due to the data collected and our ability to distribute the assessment (see table 6). questionnaires are one of the most frequently used methods to assess research consultations (fournier & sikora, 2015). we identified three reasons why questionnaires were the most suitable assessment method in our context, which provide additional insight into why questionnaires are often used. first, the questionnaire allowed us to collect data about the effectiveness of the instruction and the logistics of the consultation. the questions could easily be modified to meet our needs at a particular time. to continue to provide actionable data, the questions should not focus on user satisfaction, which has been shown to receive positive responses (fournier & sikora, 2015; newton & feinberg, 2020). positive feedback is flattering, but does not identify areas for service improvement. the questionnaire also allowed us to collect data from students who did not meet with a librarian.   second, the distribution of the questionnaire was more streamlined than other methods. since we distributed the questionnaires once each semester, the method avoided the challenges we encountered with the one-minute papers. as the questionnaire was distributed at the beginning of the class, we had a high response rate. our experience supports faix, macdonald, and taxakis (2014) who found they got a better response rate when distributing the survey in class.   third, the data analysis of the questionnaires was the easiest to integrate into our workflows. while the questionnaires took a few weeks to completely transcribe and analyze, the different topics of questions allowed us to prioritize the analysis of the questions, if required, to gather the information needed to make changes to the consultations. the one-minute papers also required a time commitment to transcribe and code the responses, and this analysis needed to occur immediately to make changes to the ongoing instruction. we only had time to do summaries of the interview data before needing to make changes for the next semester, which meant some of the data we collected was not utilized.     table 6 methods evaluation summary method effectiveness of instruction consultation logistics utility of method one-minute papers best learning outcomes data no data provided some utility team process interviews limited learning outcomes data limited actionable data difficult implementation questionnaires best learning outcomes data best actionable data best utility focus groups n/a n/a difficult implementation retrospective interviews limited learning outcomes data limited actionable data n/a     changes made to consultations   the fall 2017 questionnaire led to instructional changes in the spring 2018 semester. the librarians added showing students how to find the formatted citations in databases, and if that was not available, how to find the citation in google scholar. for help beyond that, librarians reminded students that the writing center was available. consultations for each team continued to be scheduled for one hour.   for the fall 2018 semester, the consultation was shortened from one hour to 30 minutes. during the first week of the semester, the second author met with each class section and provided an overview of the course guide, the new tutorials, and the assignment. thus, prior to the consultation, librarians told the teams via email to review the tutorials so that they could be better prepared for the session. librarians used the team process interview questions to help guide the consultation.   practical implications   our project illuminated multiple considerations for assessing consultations. first, framing questions on the one-minute papers positively elicited more responses than negatively framed questions. descriptions of how to use one-minute papers advise asking a question about points of confusion, the muddiest point, or what is unclear (bowles-terry & kvenild, 2015; schilling & applegate, 2012). while several versions of our one-minute papers had this type of question, we found that framing the question positively provided us with more actionable data. for example, instead of “what question remains unanswered?” we asked, “what would you like to learn more about?” this finding supports bowles-terry and kvenild’s (2015) caution about using a negatively framed assessment technique too often.   also, librarians should consider the possible role that each assessment method could play in student learning (oakleaf, 2009). reflection is part of the learning process (fosnot & perry, 1996). offering a one-minute paper at the end of the consultation provided students time to reflect on the session and could potentially deepen learning. the questions asked during the team process interviews also helped students frame their learning and what was needed next for their assignment. the questionnaire allowed students to reflect on what information sources they used throughout the semester. however, the retrospective interviews, while reflective, were more informative for the librarian than the students.   limitations   our study has several limitations. first, the involvement of stakeholders is key to action research. while we involved stakeholders in our project, stakeholder involvement in assessment design, data collection, and data analysis could be expanded. in particular, we could look for ways to include students. second, only one person coded all of the data. although we frequently discussed the findings during the coding process, having only one person code the data could have led to bias. finally, all of the data collected represent individual student perceptions. for our project, we did not feel the student perceptions were a large limitation, as we were able to see that students’ reports of learning mapped to the consultation learning outcomes. however, future assessments could use other methods like journaling, pre/post testing, and citation analysis of the project.   conclusion   we used action research to evaluate four assessment methods for consultations. the action research method allowed us to plan an assessment, implement the assessment, analyze the results, reflect on the effectiveness and utility of the assessment, and make changes to the assessment for the next semester. the cyclical nature of this project allowed us to make changes and continuously reflect on the usefulness of each method. after implementing one-minute papers, team process interviews, questionnaires, and retrospective interviews, we found that questionnaires were the best assessment method for our context. questionnaires provided the most actionable information about both the effectiveness of the instruction and the logistics of the consultation and were the easiest to administer. the continuous evaluation and modification of an assessment method allows for the development of an assessment that is the best for a particular context.   acknowledgements   thank you to t. derek halling, david hubbard, mike larson, bruce neville, chad pearson, ashley staff, jane stephens, and gary wan for assistance with data collection. thank you to michael r. golla for his support of our research project.   references   bowles-terry, m., & kvenild, c. (2015). classroom assessment techniques for librarians. chicago, il: association of college and research libraries.   brown-sica, m. (2012). library spaces for urban, diverse commuter students: a participatory action research project. college & research libraries, 73(3), 217-231. https://doi.org/10.5860/crl-221   brown‐sica, m., sobel, k., & rogers, e. (2010). participatory action research in learning commons design planning. new library world, 111(7/8), 302-319. https://doi.org/10.1108/03074801011059939   butler, k., & byrd, j. (2016). research consultation assessment: perceptions of students and librarians. the journal of academic librarianship, 42(1), 83–86. https://doi.org/10.1016/j.acalib.2015.10.011   coghlan, d. & brydon-miller, m. (2014). introduction. in the sage encyclopedia of action research. london : sage. https://doi.org/10.4135/9781446294406   cox, a., gruber, a. m., & neuhaus, c. (2019). complexities of demonstrating library value: an exploratory study of research consultations. portal: libraries and the academy, 19(4), 577-590. https://doi.org/10.1353/pla.2019.0036   creswell, j. & guetterman, t. (2019). educational research: planning, conducting, and evaluating quantitative and qualitative research (6th ed.). new york, ny: pearson.   drew, c., & vaz, r. (2008, june). global projects preparation: infusing information literacy into project based curricula [paper presentation]. 2008 annual conference & exposition, pittsburgh, pa, united states. https://peer.asee.org/3738   faix, a., macdonald, a., & taxakis, b. (2014). research consultation effectiveness for freshman and senior undergraduate students. reference services review, 42(1), 4-15.  https://doi.org/10.1108/rsr-05-2013-0024   fosnot, c. t., & perry, r. s. (1996). constructivism: a psychological theory of learning. in c. t. fosnot (ed.), constructivism: theory, perspectives, and practice (pp. 8-38). new york: teachers college press.   fournier, k., & sikora, l. (2015). individualized research consultations in academic libraries: a scoping review of practice and evaluation methods. evidence based library and information practice, 10(4), 247-267. https://doi.org/10.18438/b8zc7w   fournier, k., & sikora, l. (2017). how canadian librarians practice and assess individualized research consultations in academic libraries: a nationwide survey. performance measurement and metrics, 18(2), 148-157. https://doi.org/10.1108/pmm-05-2017-0022   goek, s. (2019). sign up now for project outcome for academic libraries. retrieved from https://www.acrl.ala.org/acrlinsider/archives/17406   hanlan, l. r., & riley, e. m. (2015, june). information use by undergraduate stem teams engaged in global project-based learning [paper presentation]. 2015 asee annual conference & exposition, seattle, wa, united states. https://doi.org/10.18260/p.24300   insua, g. m., lantz, c., & armstrong, a. (2018). in their own words: using first-year student research journals to guide information literacy instruction. portal: libraries and the academy, 18(1), 141-161. https://doi.org/10.1353/pla.2018.0007.   kemmis, s., mctaggart, r., & nixon, r. (2014). the action research planner: doing critical participatory action research. new york: springer.                https://doi.org/10.1007/978-981-4560-67-2   kong, n., fosmire, m., & branch, b. d. (2017). developing library gis services for humanities and social science: an action research approach. college & research libraries, 78(4), 413-427. https://doi.org/10.5860/crl.78.4.413   lemire, s., sullivan, t. d., & kotinek, j. (2019). embracing the spiral: an action research assessment of a library-honors first year collaboration. the journal of academic librarianship, 45(5), 102042. https://doi.org/10.1016/j.acalib.2019.05.010   magi, t. j., & mardeusz, p. e. (2013). why some students continue to value individual, face-to-face research consultations in a technology-rich world. college & research libraries, 74(6), 605-618. https://doi.org/10.5860/crl12-363   malenfant, k., hinchliffe, l., & gilchrist, d. (2016). assessment as action research: bridging academic scholarship and everyday practice. college & research libraries, 77(2), 140-143. https://doi.org/10.5860/crl.77.2.140   margolin, s., brown, m., & ward, s. (2018) comics, questions, action! engaging students and instruction librarians with the comics-questions curriculum. journal of information literacy, 12(2), 60-75.  https://doi.org/10.11645/12.2.2467   melgoza, p. (2017, june). mentoring industrial distribution students on their junior and senior papers [paper presentation]. 2017 asee annual conference & exposition, columbus, oh, united states. https://peer.asee.org/28663     miller, r. e. (2018). reference consultations and student success outcomes. reference & user services quarterly, 58(1), 16-21. https://doi.org/10.5860/rusq.58.1.6836   morgan, d. l. (2019). basic and advanced focus groups. thousand oaks, ca: sage.   newton, l. & feinberg, d. e. (2020). assisting, instructing, assessing: 21st century student centered librarianship. the reference librarian, 61(1), 25-41. https://doi.org/10.1080/02763877.2019.1653244   oakleaf, m. (2009). the information literacy instruction assessment cycle: a guide for increasing student learning and improving librarian instructional skills. journal of documentation, 65(4), 539-560. https://doi.org/10.1108/00220410910970249   reason, p. & bradbury, h. (eds.). (2006). handbook of action research: the concise paperback edition. thousand oaks, ca: sage.   reinsfelder, t. l. (2012). citation analysis as a tool to measure the impact of individual research consultations. college & research libraries, 73(3), 263-277. https://doi.org/10.5860/crl-261   rogers, e. & carrier, h. s. (2017). a qualitative investigation of patrons’ experiences with academic library research consultations. reference services review, 45(1), 18-37. https://doi.org/10.1108/rsr-04-2016-0029   savage, d. (2015). not counting what counts: the perplexing inattention to research consultations in library assessment activities. in d. m. mueller (ed.), creating sustainable community: the proceedings of the acrl 2015 conference, march 25–28, portland, oregon (pp. 577-584). chicago: association of college and research libraries. retrieved from http://www.ala.org/acrl/sites/ala.org.acrl/files/content/conferences/confsandpreconfs/2015/savage.pdf   schilling, k., & applegate, r. (2012). best methods for evaluating educational impact: a comparison of the efficacy of commonly used measures of library instruction. journal of the medical library association: jmla, 100(4), 258-269. https://doi.org/10.3163/1536-5050.100.4.007    sikora, l., fournier, k., & rebner, j. (2019). exploring the impact of individualized research consultations using pre and posttesting in an academic library: a mixed methods study. evidence based library & information practice, 14(1), 2–21. https://doi.org/10.18438/eblip29500   sokoloff, j., & simmons, r. (2015). evaluating citation analysis as a measurement of business librarian consultation impact. journal of business & finance librarianship, 20(3), 159-171. https://doi.org/10.1080/08963568.2015.1046783   stephens, j., melgoza, p., hubbard, d.e., pearson, c.j., & wan, g. (2018). embedded information literacy instruction for upper level engineering undergraduates in an intensive writing course. science & technology libraries, 37(4), 377-393. https://doi.org/10.1080/0194262x.2018.1484317   suskie, l. (2018). assessing student learning: a common sense guide (3rd ed.). san francisco: jossey-bass.   watts, j., & mahfood, s. (2015). collaborating with faculty to assess research consultations for graduate students. behavioral & social sciences librarian, 34(2), 70-87. https://doi.org/10.1080/01639269.2015.1042819   woodland, r. h. (2018). action research. in b. frey (ed.), the sage encyclopedia of educational research, measurement, and evaluation (pp. 37-38). thousand oaks, ca: sage. https://doi.org/10.4135/9781506326139.n18   yee, s., arnold, j., rankin, j., charbonneau, d., beavers, p., bielat, v., krolikowski, oldfield, m., phillips, s., & wurm, j. (2018). assessment in action wayne state university. retrieved from https://guides.lib.wayne.edu/aiawaynestate     appendix a team process interview protocol   1. what did you turn in for the project? 2. are you satisfied with your project report submittal? did you have sufficient information to submit for the update? 3. is there anything that you wish you had done differently? 4. what do you need to submit for your next project due date? 5. what kind of information do you need for the next due date?   appendix b questionnaire instruments   fall 2017   met with a librarian version questions 1 1. what was helpful? 2. what was not helpful? 2 1. what was the most important thing you learned during this consultation? 2. do you need help with citing resources in your project? which citation style did you use?   did not meet with a librarian version questions 1 1. were there any barriers to meeting with the librarian for the research consultation? 2. did you use the library idis class guide? which sections? 2 1. did you find or use any resources that the library should add to their idis 303 class guide or book collection? 2. do you need help with citing resources in your project? which citation style did you use?   spring 2018   met with librarian   1. what was the most important thing you learned during the consultation? 2. what did you learn about library resources that you could use in your future courses? 3. do you intend to schedule an appointment with a librarian in mmet 401 (currently idis 403)? why? 4. librarians currently meet with one team at a time. for the future mmet 301 (currently idis 303) team meetings, we are considering having multiple teams meet with a librarian at the same time and providing information using video tutorials. a. what aspects of the one-on-one team meeting were most beneficial to you? b. based on your research consultation experience, do you have any concerns about multiple teams meeting with a librarian at once? c. what information from the research consultation would you like see in video tutorial format?   did not meet with librarian   1. did anyone on your team meet with a librarian? 2. if someone from your team met with a librarian, what did they share with you about finding information for your team’s project? 3. how did your team choose a topic? 4. where did you find the information needed to write your paper? 5. how did you find the information needed to write your paper? 6. did you know the library created an idis 303 class guide to assist you with finding sources for your paper?   fall 2018   met with a librarian   1. what was the most important thing you learned during the consultation? 2. what did you learn about library resources and services that you could use in your future courses? 3. librarians currently meet with one team at a time, but have considered meeting with multiple teams at once. what aspects of the one-on-one team meeting were most beneficial to you? 4. what are your impressions about the 30-minute length of the consultation? 5. did you meet with a librarian multiple times? why or why not? 6. do you wish you had met with an engineering librarian before this class? if so, in which course or context? 7. do you intend to schedule an appointment with a librarian in mmet 401? why or why not? 8. did you view any of the library tutorial videos? 9. what information from the research consultation would you like to see in video tutorial format?   did not meet with a librarian   1. did anyone on your team meet with a librarian? 2. if someone from your team met with a librarian, what did they share with you about finding information for your team’s project? 3. do you intend to schedule an appointment with a librarian in mmet 401? why or why not? 4. did you know the library created an online mmet 301 class guide to assist you with finding sources for your paper? 5. how did your team choose a topic? 6. how did you find the information needed to write your paper? 7. did you have any difficulty finding the information for your paper? please describe. 8. did you view any of the library tutorial videos? 9. what information from the research consultation would you like to see in video tutorial format?     appendix c retrospective interview protocol   1. what did you enjoy about the class project? 2. what did you not enjoy about the class project? 3. how did the information that you needed early in the assignment compare to the information that you needed closer to the assignment due date? 4. think about how the research consultation fit within the flow of your research assignment. how would you describe the timing of your consultation: too early, just right, too late? why? 5. how did you approach finding resources for your paper after meeting with a librarian? 6. describe the resources that you used to find information for your project. 7. did you use any sources that were not mentioned by a librarian? 8. did you use the library’s get it for me service to obtain any resources? 9. how did you decide which sources to use and which not to use? 10. what could have made the research consultation experience better? 11. consider you are talking to a student who will be taking the idis 303 or idis 403 course next semester. what would you say about meeting with a librarian to that student? 12. is there anything else you would like to add about the research consultations?     microsoft word eblip101 evidence based library and information practice 2008, 3:4 79 evidence based library and information practice ebl 101 asking the right question lorie kloda phd student mcgill university school of information studies montreal, quebec, canada e-mail: lorie.kloda@mcgill.ca © 2008 kloda. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. for librarians, the idea of “asking the right question” is nothing new. as information professionals, we know that the real question is not the same as the first thing a patron asks at the outset of the reference encounter. similarly, those teaching information literacy recognize the importance of understanding one’s information needs as one of the first steps in the research process. the first step in the evidence based librarianship (ebl) process is to formulate an answerable question. eldredge draws a parallel between this step and the first step of problem based learning, in which learners are encouraged to express their uncertainties as precise information needs that can be answered using the literature. in the same way, even though you often begin with vague uncertainties regarding your information practice, ebl requires that you turn those uncertainties into more refined questions. for a question to be answerable, it must be precise or detailed enough to be conceivably answered by research. of course, it is easier to create a detailed question if you are familiar with the subject area, and formulating answerable question takes practice. the benefit to creating a precise, answerable question is that you will be more likely to make a decision based on the answer, should you find one. another benefit to formulating an answerable question is that it also enables efficient retrieval. as librarians, we all know the value of retrieving a set of literature that is not only high in recall, but high in precision as well. in other words, the concepts present in a detailed question will enable you to develop a search strategy that retrieves only very relevant results. formulating an answerable question, though, does not always mean that an answer will be available. lewis and cotter evidence based library and information practice 2008, 3:4 80 found a gap between the topics of questions asked by practitioners (mostly management and education) and those addressed by researchers (mostly information access and retrieval, and collections). in evidence based medicine and other health disciplines, the formulation of clinical questions is guided by the pico structure (for person or problem, intervention, comparison, and outcome). this structure, proposed by richardson and colleagues, was meant to be helpful in guiding physicians to formulate precise clinical questions. the pico structure, which allows for flexibility (some of the elements cab be omitted) continues to be employed by many health professionals. in library and information practice, the spice structure has been proposed by booth: • setting: the context (e.g., an academic library, law firm) • perspective: the stakeholder (group or individual) interested (e.g., graduate student, manager) • intervention: the service being offered (e.g., chat reference, library instruction workshop) • comparison: the service to which it is being compared (note that there may be no comparison) • evaluation: the measure used to determine success (e.g., usage statistics, visit to the reference desk after regular hours) as an example, an academic librarian work in a health sciences library may want to know if there are any disadvantages to staffing a chat reference service with paraprofessionals. in order to refine this question into a detailed, answerable question, the librarian can use the spice structure: • setting: academic health sciences library • perspective: students, faculty members • intervention: chat reference offered by professional librarian • comparison: chat reference offered by a paraprofessional • evaluation: user satisfaction using this example, the question can be restated as follows: in an academic health sciences library, does staffing a chat reference service with a librarian instead of a paraprofessional result in greater user satisfaction? keep in mind that asking questions is an iterative process, as librarians will recognize from the reference interview. it is a necessary and worthwhile endeavour to continually refine and reframe a question until it captures precisely the uncertainty you wish to resolve. this process takes some time and thought, and it is a good idea to make sure you, and if applicable, your colleagues, are in agreement on what exactly is the question before attempting to find an answer, otherwise you risk wasting time looking for, appraising, and applying evidence that is not even relevant to your original uncertainty! once you have formulated an answerable question, the next step is to identify the appropriate level of evidence for answering it. the next ebl 101 column will focus on matching question types to study designs. works cited booth, andrew. “formulating answerable questions.” evidence based practice: an information professional’s handbook. eds. andrew booth and anne brice. london: facet, 2004. 6170. evidence based library and information practice 2008, 3:4 81 eldredge, jonathan deforest. “evidencebased librarianship: formulating ebl questions.” bibliotheca medica canadiana 22.2 (winter 2000): 74-7. lewis, suzanne and lisa cotter. “have the most relevant and answerable research questions facing librarians changed between 2001 and 2006?” evidence based library and information practice 2.1 (mar. 14, 2007): 107-20. richardson, w. scott, marc c. wilson, jim nishikawa, and robert s. a. hayward. “the well-built clinical question: a key to evidence-based decisions.” acp journal club 123 (nov./dec. 1995): a12. news/announcements   eblip9 2017 conference announced      2016. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     save the date! the 9th international evidence based librarianship & practice (eblip9) conference will be held june 18-21, 2017 in philadelphia, pennsylvania usa. this gathering will emphasize evidence-based practice in libraries with pre-conference workshops, presentations, and networking opportunities. the conference host is drexel university which has a long tradition of practical training, civic engagement, and academic technology leadership. calls for participation for program sessions and pre-conference workshops, will go out in the coming months. keep abreast of the latest eblip9 news via the conference twitter account: @eblip9 or the conference website: http://libguides.library.drexel.edu/eblip9 if you have questions about this upcoming conference or potential ideas for pre-conference workshops, please contact john wiggins, co-chair of the local organizing committee at eblip9@drexel.edu   print book circulation longevity dropping at a small canadian university library evidence summary   print book circulation longevity dropping at a small canadian university library   a review of: belvadi, m. (2021). longevity of print book use at a small public university: a 30-year longitudinal study. insights, 34(1), 26. http://doi.org/10.1629/uksg.562   reviewed by: jordan patterson associate librarian a. p. mahoney library st. peter’s seminary london, ontario, canada email: jpatte46@uwo.ca   received: 6 june 2022                                                                    accepted:  20 july 2022      2022 patterson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30181     abstract   objective – to inform future collecting decisions by ascertaining the circulation longevity of print books within an academic library.   design – longitudinal data analysis of two circulation datasets.   setting – library catalogue of a small public university in canada.   subjects – 10,002 print books acquired between 1991 and 1996 with a first circulation year between 1991 and 2000 (part 1); 4,060 print books acquired and with a first circulation year between 2008 and 2011 (part 2a); 35,860 print books acquired since 1991 with a first circulation year between 2008 and 2011 (parts 2b).   methods – the researcher established two datasets by selecting books with viable circulation data from the institution’s holdings. using each book’s library of congress classification number, the researcher mapped each book to three other categorization schemes. the first scheme, becher-biglan typology, categorizes books as belonging to either applied or hard and pure or soft fields of study. the second scheme, called in the paper “major subjects,” uses a traditional broad subject categorization (e.g. arts, sciences, health, etc.), and the third scheme categorizes books by the academic programs at the researcher’s institution. the researcher then analyzed the circulation data through the lens of these three categorization schemes.   main results – part 1, which considered the collection’s older circulated books, found that books had an average circulation longevity of 10 years. about 14% of books circulated for only one year, and about 24% of books circulated for less than five years. among the newer books considered in part 2, 37% circulated for just one year and 64% had a circulation longevity of four years.   conclusion – books in applied and hard fields generally have greater longevity compared to pure and soft fields. books in professional and stem fields generally have greater longevity than books in the humanities and arts, contrary to conventional library wisdom. print book circulation longevity appears to be dropping. subscription and on-demand acquisitions options may prove to be a more efficacious use of resources than ‘just-in-case’ print collecting.   commentary   as libraries continue their march into the future, perhaps no issue in academic libraries today is so centrally pressing as the usage of print books. the pessimist might ask, if print books are not used, why should libraries spend all the time, energy, and money required to purchase, process, catalogue, shelve, and store them? or forego the opportunity of using that time, energy, and money space for another purpose? beyond merely functioning as a potential net drain on a library’s resources, a focus on print books may actively hinder the library in providing services patrons need and desire.   evaluated with glynn’s (2006) critical appraisal checklist, this study satisfies accepted standards of validity. the sampling for this study had to be precise to obtain viable data, but the researcher’s inclusion and exclusion criteria limited her from providing a fuller picture of print book circulation at her institution. while the study compares circulation statistics for items across different disciplines, one important point of comparison is passed over in the researcher’s decision to omit from her study any figures for items that did not circulate at all. the researcher made this decision to avoid distorting the statistics with a large number of items reporting “0 years” of circulation longevity; however, without the number of comparable items that never circulated (within the same timespans the paper considers), the circulation figures presented in the paper do not convey their full potential significance.   the difference would be, for instance, a circulated print item’s longevity compared to a broader print collection, as opposed to a circulated print item’s longevity compared to other circulated items. it is eye-opening to learn that, of recent books that circulate, 64% do so for four years at maximum; it would be instructive to learn also how many books never circulate at all. presented even briefly before proceeding with a finer analysis of circulated items, figures on uncirculated items would provide important context and bolster the researcher’s questioning of the value of ‘just-in-case’ print acquisitions. it’s a small quibble with an otherwise impressive and important study, and ultimately, non-circulating books were not the focus of this research. a comparison of circulated and non-circulated items could form the basis of a fruitful future study.   as the researcher demonstrates in her literature review, this is the only investigation into print resource longevity with a timeframe as long as thirty years. given the importance of the question, more studies would be welcome. libraries allocate more and more funds toward digital resources every year, at the cost of print, and this study makes a compelling case for continuing and even hastening that trend. attractive and efficient e-book packages allow libraries to optimize their resources toward the needs of the present, but any strategic choice requires sacrifices and trade-offs; librarians may well consider whether this option is optimized toward the needs of the future as well.   references   belvadi, m. (2021). longevity of print book use at a small public university: a 30-year longitudinal study. insights, 34(1), 26. http://doi.org/10.1629/uksg.562   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154     evidence based library and information practice 2009, 4:2 168 evidence based library and information practice evidence summary the actions of teacher-librarians minimize or reinforce barriers to adolescent information seeking a review of: meyers, eric m., lisa p. nathan, and matthew l. saxton. “barriers to information seeking in school libraries: conflicts in perceptions and practice.” information research 12:2 (2007): paper 295. reviewed by: julie mckenna deputy library director regina public library regina, saskatchewan, canada email: jmckenna@reginalibrary.ca received: 02 september 2008 accepted: 10 may 2009 © 2009 mckenna. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – to study high school teacherlibrarians and whether their actions and reactions are aligned with their perception of the role they play in creating an information seeking and learning environment. design – triangulation qualitative research undertaken over a 16 month period (fall 2005 – 2007). setting – six high school libraries in the puget sound region of the state of washington, united states. subjects – six teacher-librarians, each with a minimum of ten years experience and classroom teachers and students. this sample represented the range of school sizes, the rural, urban, and suburban mix, and the range of significant socioeconomic conditions (qualification for subsidized lunch and english as an additional language) in the region. methods – four interviews of one to two hours were held with each teacher-librarian during school hours. initial interviews were recorded by hand and a set question protocol was used (and included in the appendix). questions were asked about their professional background and training; their job duties, day to day activities and priorities; their perceptions as to how others mailto:jmckenna@reginalibrary.ca http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2009, 4:2 169 (e.g., peers and administrators) support the library; the goals of their library’s services; how students use the library; and their critical assessment of their role. subsequent interviews were undertaken within two days of a classroom visit to the library and also followed a set protocol of questions (appendix d). the second set of interviews was audio recorded and transcribed. two classroom teachers from each school were interviewed for 30 minutes and audio recorded using a set interview protocol (appendix c) within two days of class participation in library instruction. library observations ranging from two to three hours each occurred during a minimum of seven randomized times at each library. these observation sessions typically included class instructional sessions of thirty to ninety minutes. the observation protocols are described in an appendix to the study. consistent note-taking, varying of observation times and days of week, use of triangulated methods, comparison of emergent themes with other studies, audio-taping interviews, intercoder checks, analyzing data for observer effect, and a number of other approaches ensured validity. kuhlthau’s theory of intermediation and zone of intervention was used as a theoretical framework to categorize the teacher-librarians’ perceptions of their roles and their observed activities. harris and dewdney’s principles of information seeking behaviour were used as an analytic framework to study the difference between the teacher-librarians’ perceptions of their roles and their observed practices. these five roles are organizer of information; expert in locating material; identifier and instructor of general sources; advisor of search strategy; and mediator in the process of constructing meaning (kuhlthau). main results – the findings were framed in the six principles of information seeking (harris & dewdney) and were presented through use of narrative captured in both the observations and interviews. principle 1: information needs arise from the helpseeker’s situation. the high school students in the library to complete assignments about which the teacherlibrarians were not apprised; therefore the teacher-librarians were unable to assist the students in meeting information needs. principle 2: the decision to seek help or not seek help is affected by many factors. principle 3: people tend to seek information that is most accessible. issues of control were the greatest barrier to students’ successful information seeking behaviour. in the environments observed, the greatest balance of power was within the control of the teachers, including when and if the students would have access to the library, and whether the teacher-librarian would be informed of the assignment. within the library facility, the teacher-librarians demonstrated a high need for control and power over the students’ activities and behaviour, and the students themselves had almost no power. principle 4: people tend to first seek help or information from interpersonal sources, especially from people like themselves. principle 5: information seekers expect emotional support. the interpersonal style of each teacher-librarian had an affect on the nature of the students’ information seeking behaviour. the narratives demonstrated how the practices of staff, in particular, those actions that set expectations for student behaviour, had an affect on the actual information seeking activities undertaken by students. evidence based library and information practice 2009, 4:2 170 principle 6: people follow habitual patterns in seeking information. the narrative used to recount the unsuccessful instruction and research session demonstrates that unless students are convinced of the reasons why they should change their approach, they will not change habitual patterns in seeking information. students use familiar sources and their familiarity is with google and wikipedia. in order for them to understand why these sources alone are not adequate, the students would need to experience a situation that demonstrates this and would cause them to reconsider their habitual patterns. conclusion – students were not exposed to teacher-librarian behaviours and roles that would enable the development of information literacy skills. the absence of collaboration between teachers and teacher-librarians was detrimental to the support of students in their assigned tasks. students were not able to carry out information seeking practices with any autonomy and were given no meaningful reason or evidence as to why they should consider different practices. the failure to recognize that students have information habits that must be validated in order to assist them in changing or establishing new information seeking behaviours was problematic. the adolescents’ need for affective support was negated and had consequences that affected their information seeking experience. these teacher-librarians perceive that they fulfill roles in support of information literacy learning, but their behaviours and actions contradict this perception. teacher-librarians must be able to identify, analyze and change their behaviours and actions in order to better enable student achievement. commentary the findings of this study will be of interest to teacher-librarians, teachers, and school administrators. it could create awareness about how failure to collaborate and mitigate the barriers within the high school setting creates constraints for student development of information seeking skills. this study should provide insight about the mediator role teacher-librarians can fulfill and the context necessary for it to be successful. in particular, the context of information use in everyday life is required in order for adolescents to consider why and how their information seeking can improve. the study should also inform the preand inservice education of teacher-librarians to ensure that practices around rules and restrictions, and the validation of the importance and affect of interpersonal interactions enable positive roles as mediators and support student information seeking behaviours. this research does not identify why the gap between perception and actual behaviours so consistently occurred, or what measures could be taken to address this significant ongoing concern. in addition, this study was undertaken in a region that was moving to a learning environment with a commitment to authentic learning. the policies, guidelines or formalized context of library services was not made evident and this raises some question about the transferability of these research findings. the narrative anecdotes recounted in the findings do not provide any indication as to the frequency of such observed incidences. the sample provided is very small and without any context regarding the reliability of the result. the authors acknowledged that a significant constraint of this research has been the absence of student perspectives. this will be resolved in the next stage of this study, which will utilize a survey instrument with the students. evidence based library and information practice 2009, 4:2 171 works cited harris, roma m. and patricia dewdney. barriers to information: how formal help systems fail battered women. westport, ct: greenwood, 1994. kuhlthau, carol c. seeking meaning: a process approach to library and information services. westport, ct: greenwood, 2004. evidence based library and information practice 2011, 7.2 100 evidence based library and information practice news/announcements call for submissions: carl research grants 2012. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. the canadian association of research libraries (carl) offers research grants to librarians working in carl member institutions and to future library staff (students of library and information studies in canada). funds are awarded through the carl research in librarianship grant. carl has designed the research in librarianship grant to support projects involving structured evidence based research that proposes answers to real-world issues. two grants of up to $2,000 each per year will be awarded when merited. for more information see http://carlabrc.ca/en/research-libraries/grants/researchin-librarianship-grant.html. http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ evidence summary   midwest ecological study outlined the neighbourhood literacy environment and the inequitable access children have to books in public library branches   a review of: crosh, c., hutton, j., szumlas, g., xu, y., beck, a., & riley, c. (2022). inequities in public library branch access and children’s book circulation in a midwestern american city. the international journal of information, diversity, & inclusion (ijidi), 6(3), 68-81. https://doi.org/10.33137/ijidi.v6i4.38127    reviewed by: nandi prince assistant professor ursula c. schwerin library new york city college of technology new york, new york, united states of america email: nprince@citytech.cuny.edu   received: 9 dec. 2022                                                                accepted:  2 feb. 2023      2023 prince. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30290     abstract   objective – to explore the impact of the neighbourhood literacy environment (nle) by examining associations between public library locations, book circulation rates, and neighbourhood racial composition.   design – an ecological study using aggregated data sources.   setting – forty selected neighbourhood public libraries in the state of ohio, united states of america.    subjects – analysis of (1) existing circulation statistics from january 2014 to december 2018 for the neighbourhoods of cincinnati and hamilton public libraries; and (2) the american community survey (acs) data from 2018.   methods – among the key components studied for the population was the nle, which the authors defined as access to literacy materials in a neighbourhood. the data the authors examined for the targeted populations were race, age, poverty level, and library location. the two groups of variables computed were: (1) the connection between circulation rates of children’s books and child poverty; (2) the connection between circulation statistics and the proportion of people who self-identify as black in the neighbourhood. additionally, the researchers used the spearman’s rank order correlation coefficient (rs) to measure the relationships between the correlating variables within each neighbourhood library branch – number of books circulated per child; the census data of children who self-identified as black; and the children who were designated as 20% below the federal poverty level (fpl). the chi-square test was used to calculate associations between access to a library branch and child poverty in each neighborhood. in this study, the researchers only looked at the associations between variables at an aggregate level. the authors defined the terms they used in the study: (1) children were ages 0-18 years; (2) children’s books were literature intended for an audience from 0-18 years old; (3) the definition of poverty was taken from the u.s. census and classified as neighborhoods with 20% of children below the fpl.   main results – there were 40 library branches that served 81 neighbourhoods, of which there was only a 38% distribution in the high-poverty areas, compared with 58% for the low. approximately 24 million books were circulated during the 5-year period of 2014 -2018. the median circulation rate per child at the neighbourhood level was 22 books. the results showed steep variations in circulation rates per child across branch locations; the numbers range from 3 to 98 books per child across neighborhoods. the authors indicated that the increases and decreases in the circulation rates were tied to branch location and the area’s socioeconomic status. the primary finding of the data analyzed was a negative correlation between the population identified as black/african american and lower circulation rates in poorer neighbourhoods.   limitations identified by the authors were (1) the allocation of literacy resources per branch was unknown; (2) the in-library book user statistics in high-poverty neighbourhoods may not be accurately documented; (3) the precise allocations for literacy funds and the use of in-library resources for developing literacy skills need further study.   conclusion – the authors noted that race, economic status, and proximity to public libraries were pertinent factors in understanding inequitable access to books for children in the neighbourhoods studied. the nle was an important dynamic beyond the home; the availability of books and engagement with them were contributing factors to the development of literacy skills. the associations observed between the variables indicated that improving the nle matters and libraries must mindfully work to alleviate the disproportionately lower levels of access to books and their unfavorable outcome for children in low-income areas.   commentary   this study added to the existing literature on associations between poverty and access to literacy resources (e.g., neuman & celano, 2001; neuman & molan, 2016). based on the evidence of the circulated books-to-child ratio and the various data components studied (library location, poverty, and race), the nle was a significant determinant to the literacy development of children. branch libraries are part of their environment and therefore warrant the statistical analysis completed by the authors.   after evaluating the study with the cristal checklist for appraising a user study (booth, 2010), the generalizability to age group, location, and race applied by the researchers were assessed as appropriate for a population level study. this checklist also highlighted the strengths of the study: the authors’ clearly stated aims, the population size selected, the variables of the data collected, and the reproducibility of the methods.   the researchers outlined their hypothesis and the specific variables they used to understand the problem: (1) by making statistical comparisons of the aggregate circulation statistics of children’s books between branches to determine; (2) what connections exist because of the area’s economic status; and (3) what institutional partnerships and legislative acts can be made to the current structure to improve the studied population’s access to books. spearman’s correlation coefficient was a good choice because it ranked the dissimilar data of this population study. it displays the monotonic relationship, when one variable changes, as in high and low-poverty neighbourhoods. an examination of the high and low-poverty neighbourhoods reflected that the low-poverty residents had more libraries available to them. those statistics should be the reverse because low-poverty residents had more options available to them, including the income to purchase books and additional libraries from which to obtain free library books. additionally, the 3:98 ratio of circulated books in low to high-poverty neighbourhoods was a daunting statistic because the difference is 33-fold. the authors suggested that populations with lower circulation figures had access to under-resourced libraries.   there was inherent bias in the census data that was used for the study. the historical legacy of racial and ethnic identities in american society is complex and fraught with problems. implicit in the linear timeline, and the numbers from their computation of the variables, race and poverty contributed to the decisions of where the 40 libraries were placed in cincinnati going as far back as the late 1800’s. while this study focused on the number of black people who self-identified based on information obtained from acs, a subset of the u.s. census, future researchers may want to address how the census measures race and ethnicity of hispanics and of other ethno-racial categorizations of population groups. additionally, the methods section stated that the acs data were produced annually, but there was no explanation given as to why only the 2018 data (crosh et al., 2022, p. 71) was applied to the circulation statistics of 2014 through 2018. it is worth extending this study because population level data were only partially informative as they lacked the accuracy of data at an individual level (grimes & schulz, 2002). the aforementioned concerns did not overshadow the usefulness of this study, however, because it highlighted important findings that stakeholders and legislators should implement to eliminate the negative impacts of income inequality on the literacy development of black children. other library systems may also use this methodology to analyze their circulation data and better advocate for the economically disadvantaged children who are most reliant on services provided by public libraries to advance their literacy development. additionally, the results that showed decreased exposure for said population to resources that are cognitively enriching were at odds with the basic tenets of the library bill of rights. the study’s usefulness may also influence state and local government officials to redistribute resources to libraries that were underfunded in areas with a high concentration of black children. the data will be instrumental in brainstorming solutions for the continuous and complex problems facing neighbourhood branches.   references    booth, a. (2010). cristal checklist for appraising a user study. netting the evidence. retrieved from http://nettingtheevidence.pbwiki.com/f/use.doc     crosh, c., hutton, j., szumlas, g., xu, y., beck, a., & riley, c. (2022). inequities in public library branch access and children’s book circulation in a midwestern american city. the international journal of information, diversity, & inclusion (ijidi), 6(4), 68-81. https://doi.org/10.33137/ijidi.v6i4.38127    grimes, d. a., & schulz, k. f. (2002). descriptive studies: what they can and cannot do. the lancet (british edition), 359(9301), 145–149. https://doi.org/10.1016/s0140-6736(02)07373-7   neuman, s. b., & celano, d. (2001). access to print in low-income and middle-income communities: an ecological study of four neighborhoods. reading research quarterly, 36(1), 8-26. https://doi.org/10.1598/rrq.36.1.1   neuman, s. b., & moland, n. (2016). book deserts: the consequences of income segregation on children's access to print. urban education, 54(4), 126-147. https://doi.org/10.1177/0042085916654525   evidence summary   interesting patterns found when academic and public library use by foreign-born students is assessed using ‘super-diversity’ variables   a review of: albarillo, f. (2018). super-diversity and foreign-born students in academic libraries: a survey study. portal: libraries and the academy, 18(1), 59-91. https://doi.org/10.1353/pla.2018.0004   reviewed by: brittany richardson web services librarian, assistant professor university of tennessee at chattanooga library chattanooga, tennessee, united states email: brittany-richardson01@utc.edu   received: 2 sept. 2019                                                               accepted:  5 nov. 2019      2019 richardson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29637     abstract   objective – to evaluate the relationship between academic and public library usage and various characteristics of foreign-born students.   design – survey questionnaire.   setting – medium-sized public liberal arts college in the northeastern united states.   subjects – 123 foreign-born students enrolled at the institution in fall 2014.   methods – the researcher emailed a five-part survey to participants who indicated on a screening survey that they were foreign-born students currently enrolled at the college. of the participants emailed, 94 completed the survey. the survey used a super-diversity lens to assess academic and public library use by foreign-born students in relationship to multiple variables, including student status, race and ethnicity, immigration status, first-generation student status, gender, age, age of arrival in the united states (us), years living in the us, and zip code (used to approximate median income based on the us census bureau’s 2014 american community survey). respondents reported frequency of use on a likert-type scale of 1=never to 6=always. the author adapted items from the in library use survey instrument (university of washington libraries, 2011). usage types included: computer, wi-fi, staff assistance, electronic resources, physical resources, printing/scanning/photocopying, program attendance, and physical space. independent sample t-tests were used to evaluate mean differences in reported library usage based on demographic variables. the author used somers’ d statistical tests to explore the relationship between library use and age, age on arrival in the us, years lived in the us, and median income. the survey asked participants to describe both academic and public libraries in five words. to show term frequency, the author used word clouds as a visualization technique.   main results – the study reported on the results of the library use survey section. overall, foreign-born students used college libraries more frequently than public libraries. the author reported on findings that were statistically significant (p ≤ 0.5), focusing on those with mean differences ≥ 0.5. key findings included: undergraduate students used public libraries and wi-fi/e-resources onsite at college libraries more often than graduate students; first-generation students gathered at the library with friends more frequently; no significant difference was reported in library resource use by gender; and non-white students used the college library more frequently as a study space and for printing. the author was surprised no significant differences in usage were found between participants with permanent vs. temporary immigration status. somers’ d associations showed an inverse relationship between age and wi-fi use and age of arrival in the united states and likelihood of eating in the library. overall, both library types were positively described in open-ended responses as places with social and academic value.   conclusion – the author suggested the concept of super-diversity equips librarians with a more inclusive approach to studying library user perspectives and behaviors. the author used survey data and the association of college and research libraries (acrl) diversity standards (2012) to highlight library service considerations for foreign-born students. examples of suggested service improvements included supporting printing in unicode non-english fonts, cultivating a diverse library staff, and providing culturally appropriate library orientations and outreach. the author recommended that more research with foreign-born students was needed to assess culturally appropriate areas for eating and socializing, unique information needs, and expectations and awareness of library services. the author suggested first-generation students’ use of the library for socializing and non-white students’ higher use of libraries for studying as two areas for further qualitative study. the author also suggested creating services and partnerships between public and academic libraries could support foreign-born students, even recommending cross-training of library staff.   commentary   vertovec (2007) describes super-diversity as moving beyond ethnicity to evaluate the interplay of variables like gender, immigration status, religion, and others. library use by diverse populations has been studied focusing on single (whitmire, 2003) and multiple variables (herrera, 2016; nackerud, et al., 2013; stone & collins, 2013; sei-ching & kyung-sun, 2008). some studies used self-reported survey responses (sei-ching & kyung-sun, 2008; whitmire, 2003) while others assessed usage data (herrera, 2016; nackerud, et al., 2013; stone & collins, 2013). the author furthered these research efforts by exploring the relationship between library use and multidimensional characteristics of foreign-born students. the study simultaneously examined academic and public library use, making it unique among similar studies. most importantly, the author introduced the concept of super-diversity as a way to support a more inclusive exploration of diversity in libraries.   evaluating the study using glynn’s (2006) ebl critical appraisal checklist revealed some of its strengths. the methods were well articulated, including details on recruiting, survey administration, and analysis. results were clearly outlined, with a focus on findings that demonstrated statistical significance. the author also provided a detailed explanation of potential study limitations, with suggested improvements useful to practitioners pursuing similar research.   there were also areas for improvement. only a portion of the five-part survey instrument was provided. ambiguous question design may have caused underreporting of first-generation status. “i am the first in my family to get a college degree” was listed as one of nineteen possible responses for the question: “ideally, what's your intention for completing a degree? check all that apply” (p. 89). when asked to “list five words that best describe” each type of library, respondents used phrases (p. 71). reporting word frequency may have removed some context in the analysis of these responses. the author identified several study limitations: interpretation of likert-type responses as continuous variables; potential inaccuracies of self-reported survey responses; use of a convenience sample with a 10% margin of error (95% confidence interval); use of zip code to estimate income; and lack of formal survey validation. super-diversity was presented as a study framework, but there was no evidence of multivariate analysis to assess the interaction of variables, as suggested by vertovech (2007). the author noted a larger population would afford more data to analyze.   the results of the survey are of interest to libraries serving foreign-born students and provide a starting point for further examination of services for these users. by using the concept of super-diversity, the study offers a broader lens for exploring the relationship between diversity variables and library use. the development of increasingly robust studies using this lens may enhance inclusion efforts in libraries. areas for further research include the exploration of additional variables (e.g., sexuality, gender identity, disability, neurodiversity) and relationship to other diversity frameworks (e.g. intersectionality).   references   herrera, g. (2016). undergraduate library collection use and diversity: testing for racial and gender differences. portal: libraries and the academy, 16(4), 763-774. https://doi.org/10.1353/pla.2016.0051   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154    sei-ching j. s. & kyung-sun, k. (2008). use and non-use of public libraries in the information age: a logistic regression analysis of household characteristics and library services variables. library & information science research, 30(3), 207-215. https://doi.org/10.1016/j.lisr.2007.11.008   stone, g. & collins, e. (2013). library usage and demographic characteristics of undergraduate students in a uk university. performance measurement and metrics, 14(1), 25-35. https://doi.org/10.1108/14678041311316112    university of washington libraries. (2011). in library use survey. retrieved from https://www.lib.washington.edu/assessment/surveys/ilu   vertovec, s. (2007). super-diversity and its implications. ethnic and racial studies, 30(6), 1024-1054.  https://doi.org/10.1080/01419870701599465   whitmire, e. (2003). cultural diversity and undergraduates’ academic library use. the journal of academic librarianship 29(3), 2003, 148-161. https://doi.org/10.1016/s0099-1333(03)00019-3   microsoft word es_preddie.doc evidence based library and information practice 2007, 2:3  105 evidence based library and information practice       evidence summary    time, cost, information seeking skills and format of resources present barriers to  information seeking by primary care practitioners in a research environment    a review of:  andrews james e., kevin a. pearce, carol ireson, and margaret m. love.  “information‐seeking  behaviors of practitioners in a primary care practice‐based research network (pbrn).”  journal of the medical library association 93.2 (apr. 2005): 206‐12.    reviewed by:   martha ingrid preddie  medical librarian, medical library, federal medical center,   butner, north carolina, united states of america  e‐mail: preddie@email.unc.edu        received : 01 june 2007    accepted : 08 july 2007      © 2007 preddie. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to determine the information  seeking behaviors of primary care  practitioners in order to inform future  efforts towards the design of information  services that would support quality in  primary care.    design – a cross‐sectional survey.    setting – a primary care practice based  research network (pbrn) of caregivers who  serve a broad population while  simultaneously studying and disseminating  innovations aimed at improvements in  quality, efficiency and/or safety of primary  health care in the united states.      subjects – all primary care practitioners in  the pbrn including family practitioners,  general practitioners, nurse practitioners  and physician assistants.     methods – a questionnaire comprising  twenty‐six questions was distributed to 116  practitioners.  practitioners attached to  academic centres (who were also members  of the pbrn) were excluded in order “to  achieve a sample of practices more  representative of the primary care practising  population” (208).  descriptive data were  collected and analyzed.  spss v11.5 was  used for statistical analyses.    main results – there was a response rate of  51% (59 of 116).  fifty‐eight percent of the  respondents stated that they sought  information (excluding drug dosing or drug  mailto:preddie@email.unc.edu http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:3  106 interactions information) to support patient  care several times a week.  sixty‐eight   per cent sought this information while the  patient waited.  almost half of the  respondents had access to a small medical  library (48%) or a hospital library (46%),  while 21% used a university medical library.   approximately 14% had no immediate  access to a medical library.  almost 60% of  practitioners stated that they had an e‐mail  account.  thirty‐four percent agreed that the  use of e‐mail to communicate with patients  enhanced medical practice, while 24%  disagreed.  there was frequent prescribing  of internet‐based consumer health  information to patients by only 16% of the  practitioners, while internet support groups  were frequently recommended by 5%. the  main barriers to information seeking were  lack of time (76%), cost (33%), information  seeking skills (25%), and format of  information sources (22%).  the use of ebm  resources was fairly low, while there was a  high preference for ready reference and  interpersonal sources.  when compared  with print information resources, the use of  online resources was moderate.  a  significant correlation was found between  use of online sources and use of print  sources, namely, that practitioners who used  online sources more frequently, also sought  information from print sources more  frequently, with the inverse being true for  those who sought information less  frequently from either electronic or print  sources.    conclusion – primary care practitioners in  this rural pbrn used print and  interpersonal sources more than online  sources.  practitioners who are more likely  to use print sources are also more likely to  seek online information.  librarians working  in pbrn environments will need to identify  interventions that address barriers such as  time, cost, and information‐seeking skills.      commentary     this study investigates the information  seeking behaviour of primary care  practitioners in a practice based research  network (pbrn) in kentucky, usa. the  significance of this setting is that pbrns  provide not only for the provision of care to  a wide range of patients, but also for the  facilitation of research to advance primary  care practice in terms of quality, efficiency  and safety.  pbrns can also serve as a  vehicle for translating advances in health  information services and research into  practice.  the investigation of the  information seeking behaviours of  practitioners in the pbrn adds a unique  client group to the evidence base of  information needs and information seeking  behaviour research.     the methodology employed was a cross‐ sectional survey administered to the entire  eligible population.  the survey was  approved by an institutional review board  and tested.  completion of the survey  indicated informed consent.  it would have  been useful for the survey instrument to be  included in an appendix.  the unavailability  of the questionnaire (despite the provision  of a web link to it in the publication)  prevents analysis of its suitability for the  collection of data to meet the objective of the  study.  nevertheless, the survey is usually  the method of choice for research studies of  this nature.  the response rate of 51% (59  respondents) was adequate for analysis and  reporting, particularly since a comparison of  responders and non‐respondents revealed  similarities in terms of demographics and  practice types.  the results clearly outline  use of and barriers to the use of information  resources by pbrn practitioners, and the  conclusions accurately reflect the analysis.   the study would have benefited from an  analysis of the reasons behind the  differences that give rise to practitioners’  inclination to seek or not to seek information  evidence based library and information practice 2007, 2:3  107 on a frequent basis, a shortcoming that was  acknowledged by the researchers (210).       this research contributes to our  understanding of the information needs of  primary care practitioners in practice based  research networks.  the significance of the  article is that despite working in an  environment that is more oriented to  research, practitioners in this setting face  similar barriers (time, cost and information  seeking skills) to information use as their  counterparts in non‐research settings. as  implied by the researchers, the findings of  this study can be used to inform the  development of practice based research  infrastructure (208). possible solutions may  include tailored outreach programs that  seek to reduce the barriers to information  seeking and use.  these have been  demonstrated to be effective in other  settings, and may well fit the research‐ oriented environment of this study.  the use  of technology, as a means of developing  tools that provide access to multiple sources  via a single interface, warrants investigation.   outreach via internet based library  resources also has the potential to enhance  information access services.          evidence summary   metadata quality in institutional repositories may be improved by addressing staffing issues   a review of: moulaison sandy, h., & dykas, f. (2016). high-quality metadata and repository staffing: perceptions of united states–based opendoar participants. cataloging & classification quarterly, 54(2), 101-116. http://dx.doi.org/10.1080/01639374.2015.1116480   reviewed by: elizabeth stovold information specialist, cochrane airways group st george’s, university of london tooting, london, united kingdom email: estovold@sgul.ac.uk   received: 31 may 2016  accepted: 15 july 2016      2016 stovold. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to investigate the quality of institutional repository metadata, metadata practices, and identify barriers to quality. design – survey questionnaire. setting – the opendoar online registry of worldwide repositories. subjects – a random sample of 50 from 358 administrators of institutional repositories in the united states of america listed in the opendoar registry. methods – the authors surveyed a random sample of administrators of american institutional repositories included in the opendoar registry. the survey was distributed electronically. recipients were asked to forward the email if they felt someone else was better suited to respond. there were questions about the demographics of the repository, the metadata creation environment, metadata quality, standards and practices, and obstacles to quality. results were analyzed in excel, and qualitative responses were coded by two researchers together. main results – there was a 42% (n=21) response rate to the section on metadata quality, a 40% (n=20) response rate to the metadata creation section, and 40% (n=20) to the section on obstacles to quality. the majority of respondents rated their metadata quality as average (65%, n=13) or above average (30%, n=5). no one rated the quality as high or poor, while 10% (n=2) rated the quality as below average. the survey found that the majority of descriptive metadata was created by professional (84%, n=16) or paraprofessional (53%, n=10) library staff. professional staff were commonly involved in creating administrative metadata, reviewing the metadata, and selecting standards and documentation. department heads and advisory committees were also involved in standards and documentation selection. the majority of repositories used locally established standards (61%, n=11). when asked about obstacles to metadata quality, the majority identified time and staff hours (85%, n=17) as a barrier, as well as repository software (60%, n=12). when the responses to questions about obstacles to quality were tabulated with the responses to quality rating, time limitations and staff hours came out as the top or joint-top answer, regardless of the quality rating. finally, the authors present a sample of responses to the question on how metadata could be improved and these offer some solutions to staffing issues, the application of standards, and the repository system in use. conclusion – the authors conclude that staffing, standards, and systems are all concerns in providing quality metadata. however, they suggest that standards and software issues could be overcome if adequate numbers of qualified staff are in place.   commentary   in the first part of the article the authors reviewed the available literature to define what is meant by quality metadata and why it is important. they identified interoperability as being particularly significant, and discussed how the use of standards and best practices can facilitate interoperability, and therefore improve metadata quality. the literature review sets the scene very well for their investigation into repository metadata quality.   to evaluate the author’s survey, boynton and greenhalgh’s critical appraisal checklist for a questionnaire study (2004) was used as a guide. a survey questionnaire was an appropriate tool to address the stated aims of the research; however, the authors do not describe how they developed the survey, whether it was based on previously published or validated measures, or whether they piloted the survey first. the questionnaire is not included in the article or as an appendix so it is not possible to assess the suitability of the format or instructions, nor is it possible for readers to replicate the research in their own settings. the methods section is brief and does not give any detail about coding methods for the free-text questions.   the chosen sample repositories were all based in the united states of america, representing 14% of member american repositories and 1.7% of the total members. the authors do not explain their rationale for selecting this sample, nor discuss how the geographic limitation and small sample size may affect the applicability of their results. however, the analysis was appropriate, with results presented as both absolute numbers and percentages. the free text answers were interesting and added context to the quantitative data. clear themes emerged around consistency of applying standards, and available time to do such work.   despite some shortcomings in the reporting of their methods, the authors have provided a good overview of the issues relating to metadata quality. their literature review picked up on staffing issues and this is reflected in the results of their survey. the authors found that even when professional staff are involved in selecting standards, internationally recognized standards are not always used in repository settings. there is a role for librarians to advocate for the use of recognized metadata standards where appropriate, and a challenge to develop metadata creation and curation skills so there are adequately skilled staff available to do this work. cox, verbaan, and sen (2012) have conducted a useful audit of required competencies for librarians involved in research data management.   the authors sought to investigate the perceptions of metadata quality, while recognizing that the idea of ‘quality’ is subjective. future research could be aimed at trying to quantify the notion of quality, for example by conducting an audit of adherence to recognized metadata standards in repositories.   references   boynton, p. m., & greenhalgh, t. (2004). hands-on guide to questionnaire research: selecting, designing, and developing your questionnaire. bmj, 328, 1312-1315. http://dx.doi.org/10.1136/bmj.328.7451.1312   cox, a., verbaan, e., & sen, b. (2012). upskilling liaison librarians for research data management. ariadne, 70. retrieved from http://www.ariadne.ac.uk/issue70/cox-et-al     microsoft word es_hall.doc evidence based library and information practice 2007, 2:1  128 evidence based library and information practice       evidence summary    a national survey of the perceived monetary value of public library service for  norwegians: contingent valuation approach reveals a 1:4 cost‐benefit ratio      a review of:  aabø, svanhild. ʺare public libraries worth their price?” new library world 106.11/12 (2005):  487‐95.    reviewed by:   stephanie hall  chief librarian, powell river public library  powell river, british columbia, canada  e‐mail: shall@powellriverlibrary.ca        received: 3 december 2006  accepted: 3 january 2007      © 2007 hall. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to determine the perceived  monetary value of public library service to  norwegians (both library users and non‐ users).     design – four sets of 2 questions were  included in a bi‐monthly omnibus telephone  survey conducted by a private agency.    setting – norway.    subjects – a representative sample of  norwegian citizens (999 respondents). the  mode of selection used for the survey is  described by aabø as follows:    (1) municipalities were randomly  drawn from clusters based on  economic and industrial structure,  demographic structure and  geography; (2) starting addresses in  the municipalities were randomly  pulled from the national telephone  directory database; and (3) the  individual to be interviewed as  representative of the household was  the person above 15 years of age  with the most recent birthday. (491)    method – the contingent valuation method  was used, with questions assessing both  willingness to pay (wtp) and willingness to  accept compensation for loss of the service  (wta). four sets of 2 valuation questions  were asked via telephone. the second  question was an open‐ended question with  “principally the same wording” in each set  (492). the first question was worded  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:1  129 differently in each set, with two of the sets  assessing wtp and two assessing wta. the  questions were framed in the context of a  “pressed economic situation where the local  politicians discussed reallocation of the  funding of several public services, including  the public library” (492).     main results – three major findings were  reported:    1. the average value placed on library  service by the sub samples varied  between 400‐2000 norwegian kroner  (nok) per household. this can be  compared to the actual amount paid for  library service – 420 nok per  household. in general, the lower bound  (400 nok) reflects the wtp scenario,  while the upper bound (2000 nok)  reflects the wta scenario.  2. 94% of the norwegian population  perceives that they have property rights  to their local library.  3. because of the strong sense of rights  with regard to libraries, a stronger  weighting was given to the wta  scenario, with a final cost‐benefit ratio  reported as 1:4.    conclusions – the 1:4 cost‐benefit ratio  provides a rationale for continued  government funding of the public library in  norway as a cost‐effective and much‐valued  service at the national level. because  funding of public libraries also comes from  local government sources, the author  recommends further cost‐benefit research at  the municipal level (494).     commentary    an increasing number of studies are using  contingent valuation as a way to determine  the value of public libraries to communities  (for a convenient list of valuation studies see  abram). this methodology is often used to  evaluate non‐market products and services,  but is considered by many to be somewhat  unreliable, relying as it does on stated  preferences rather than revealed behaviour.  particularly at issue is the wta measure,  which is generally found to elicit higher  valuations than wtp.      in the current study, the reliance on the less  conservative wta measure is somewhat  mitigated by the tactic of framing the  questions in the context of fiscal restraint.  also, the exercise of weighing library service  against other government services (for  example elder care or schools) may have  helped to provide some perspective as  respondents attempted to arrive at a dollar  value in the hypothetical market.  nonetheless, given the lack of unanimity on  the use of contingent valuation as a  methodology, the study would likely have  benefited from some form of triangulation.    the paper is based on aabø’s 2005 phd  thesis, and does an admirable job of  condensing the material into 9 pages.  however, additional length would have  been welcome in exchange for a more  detailed discussion of the method and  findings, including question wording and a  breakdown of responses for the four  subsamples.    for public libraries, the research trend to  evaluate the impact of library service in a  monetary sense is a very useful tool for  advocacy, as well as for internal  reassessment. as aabø and her scholarly  colleagues continue their work in this area,  it is to be hoped that best practices for  credible valuations of library service will  emerge, simplifying the task of replicating  this valuable research.    works cited  aabø, svanhild. “the value of public  libraries: a methodological discussion  evidence based library and information practice 2007, 2:1  130 and empirical study applying the  contingent valuation method.” diss.  u of oslo, 2005.    abram, stephen. “the value of canadian  libraries: impact, normative data and  influencing funders.” sirsidynix. 2005.  7 jan. 2007  <http://www.sirsi.com/pdfs/company/ abram/stephenabram_valuecanadian libraries.pdf >.    http://www.sirsi.com/pdfs/company/ evidence based library and information practice evidence based library and information practice 2012, 7.4 4 evidence based library and information practice editorial responsibilities 2012. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. editor-in-chief: alison brettle associate editor (articles): wayne jones associate editor (evidence summaries): lorie kloda associate editor (reviews): denise koufogiannakis associate editor (commentaries, using evidence in practice, news): alison brettle production editor: katrine mallan editorial intern: michelle dunaway copyeditors: heather pretty (lead copyeditor), georgianne bordner, molly des jardin, linda ferguson, richard hayman, stacey penney, mary virginia taylor , alison yeoman indexing support: pam morgan http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/ ebl 101   research methods: triangulation   virginia wilson director, centre for evidence based library and information practice (c-eblip) university library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   originally published in: evidence based library and information practice, 9(1), 74–75. https://ejournals.library.ualberta.ca/index.php/eblip/article/view/21469/16225     received: 11 feb. 2014   accepted: 16 feb. 2014      2016 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   triangulation refers to using more than one particular approach when doing research in order to get richer, fuller data and/or to help confirm the results of the research. denzin via flick (2002) points out four different types of triangulation:   data triangulation: using different sources of data. this includes different times for data collection, difference places from which to collect the data, and different people who could be involved in the research study. flick (2002) indicates that “the starting point is to purposively and systematically involve personas and study groups, local and temporal settings in the study” (p. 226). investigator triangulation: using several people (or at least more than one) in the data gathering and data analysis processes. this would be a “systematic comparison of different researchers’ influences on the issue and the results of the research” (flick, 2002, p. 226). theory triangulation: approaching the data with multiple theories or perspectives in mind to “extend the possibilities for producing knowledge” (flick, 2002, p. 227). methodological triangulation: two subtypes are noted – within-method and between-method. using more than one method to gather data.   triangulation initially was undertaken as a way to increase the validity of research results. however, flick (2002) states that “triangulation is less a strategy for validating results and procedures than an alternative to validation which increases scope, depth and consistency in methodological proceedings” (p. 227). triangulation can be used in quantitative and qualitative research and it even seems as though triangulation is just another term for mixed-methods research. the journal of mixed methods research has a special issue devoted to analyzing and exploring the various ways triangulation is used in mixed-methods research, obviously differentiating the two terms. this topic was chosen for the special issue “based on the claims made by many scholars in the field that triangulation provides a justification for the use of mixed methods” (mertens & hesse-biber, 2012, p. 76).   triangulation has been taken to task on a couple of occasions. given the philosophical and epistemological nature of the various positions taken regarding triangulation, it is too complex to delve into in this relatively brief column. in brief, triangulation has been criticized for “subscribing to a naive realism that implies that there can be a single definitive account of the social world” as well as for assuming that “sets of data deriving from different research methods can be unambiguously compared and regarded as equivalent” (bryman, 2004, p. 3). despite possible controversies, triangulation in whatever form it takes (although the most common is probably methodological triangulation), has become a staple in social science research.   examples of studies using triangulation:   wahl, d., avery, b., & henry, l. (2013). studying distance students: methods, findings, actions. journal of library & information services in distance learning7 (1-2), 183-209. doi: 10.1080/1533290x.2012.705656   zuze, h. & weideman, m. (2013). keyword stuffing and the big three search engines. online information review37(2),268-286. doi: 10.1108/oir-11-2011-0193   bitso, c. & fourie, i. (2012). an investigation of information-seeking behaviour of geography teachers for an information service intervention: the case of lesotho. information research17(4). retrieved 23 feb. 2014 from http://www.informationr.net/ir/17-4/paper549.html#.uwo1i-zfdiu     resources:   o’cathain, a., murphy, e., & nicholl,j. (2010). three techniques for integrating data in mixed methods studies. bmj 341 doi: http://dx.doi.org/10.1136/bmj.c4587 (published 17 september 2010).   denzin, n. k. (19708). strategies of multiple triangulation. in n. k. denzin (ed.), the research act: a theoretical introduction to sociological methods. (pp.). chicago, il: aldine.   references   bryman, a. (2004). triangulation and measurement. retrieved 11 feb. 20134 from http://www.referenceworld.com/sage/socialscience/triangulation.pdf     flick, u. (2002). an introduction to qualitative research (2nd ed.). london: sage publications.   mertens, d. m. & hesse-biber, s. (2012). triangulation and mixed methods research: provocative positions. journal of mixed methods research 6(2), 75-79. doi:10.1177/1558689812437100     ebl 101   research methods: design, methods, case study…oh my!   virginia wilson liaison librarian murray library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   originally published in: evidence based library and information practice, 6(3), 90–91. https://ejournals.library.ualberta.ca/index.php/eblip/article/view/11231/8954     received: 13 aug. 2011 accepted: 16 aug. 2011      2016 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   as this column transitions into looking at various research methods, some preliminary topics must be explored in order to put research methods into context. last time i looked at qualitative and quantitative research in general, and discussed the background of both types of research and the tension between researchers invested in each of these methods. lately, the mixed methods approach has gained traction, and i will discuss that in forthcoming columns as well. but first, in order to situate research methods, this column will explore research design.   before undertaking research, there must be a plan, and that is where design comes in. but research design is not merely a work plan. research design is “the entire process of research from conceptualizing a problem to writing research questions, and on to data collection, analysis, interpretation, and report writing” (bodgan & taylor, 1975, p. 5, as cited in creswell, 2007). it is the “logical sequence that connects the empirical data to a study’s initial research questions, and ultimately, to its conclusions” (yin, 2003, p. 5, as cited in in creswell, 2007). the confusing part is that, while reviewing research literature as well as websites focusing on research, i have found that quite often the terms research design and research methods are used inconsistently. for example, the terms research design and research method are both used to refer to a case study, a third, and different, entity.   let’s think about “case study” for a minute. the chapter on qualitative case studies in the sage handbook of qualitative research states that a “case study is not a methodological choice but a choice of what to be studied,” and goes on to say that “by whatever methods, we choose to study the case” (stake, 2005, p. 443). the book case study research: design and methods “covers the distinctive characteristics of the case study as a research method” (yin, 2003, p. 2). so is the case study both a method and a design? is it a method sometimes and a design at others? and where is the definitive answer? there really isn’t one. just as i was about to throw my hands up in despair, i came across an article in the information systems field which puts forth a straightforward delineation between research design and research method that i am going to use for this column: “research strategy [or research design] is defined as ‘a way of going about one’s research, embodying a particular style and employing different methods.’ research method is defined as ‘a way to systemise [sic] observation, describing ways of collecting evidence and indicating the type of tools and techniques to be used during data collection’” (cavaye, 1996, p. 227).   based on this delineation, a case study is first and foremost a research design, and it makes sense that a variety of methods could be used to collect the evidence, depending upon which angle one is taking in approaching the research question. cavaye (1996) reminds us that case study research is multi-faceted and can be undertaken by using “a positivist or an interpretivist stance, can take a deductive or an inductive approach, can use qualitative and quantitative methods, [and] can investigate one or multiple cases” (p. 227). while cavaye does describe some characteristics of a case study method that are employed in case study research—it does not seek to control variables; it looks at a case in its natural context; focuses on (generally) one site; and it uses qualitative, in addition to quantitative, tools and techniques—even these characteristics seem more akin to design than to method (p. 229).   so for the purposes of this ongoing column, in which i will examine various research methods and provide some sources for further exploration, i will adopt the initial definitions from cavaye seen above. i may focus a later column on case study research in more detail. but as it is, i am ready to move on to a specific, hopefully more straightforward, method. next time around, i will look at content analysis.   references   cavaye, a.l.m. (1996). case study research: a multi-faceted research approach for is.                 information systems journal 6(3): 227-242.   creswell, j.w. (2007). qualitative inquiry & research design (2nd ed.). london: sage.   stake, r.e. (2005). qualitative case studies. in n.k denzin and y.s. lincoln (eds.), the sage handbook of qualitative research (3rd ed.). (pp.443-446), thousand oaks, ca: sage.   yin, r.k. (2003). case study research: design and methods. thousand oaks, ca: sage.   microsoft word art_song.doc evidence based library and information practice 2006, 1:1    69 evidence based library and information practice       article    evidence based marketing for academic librarians      yoo‐seong song  lir librarian & assistant professor of library administration  university of illinois at urbana‐champaign  urbana, illinois, united states of america  e‐mail: yoosong@uiuc.edu      received: 9 december 2005    accepted: 6 march 2006      © 2006 song this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective ‐ in developing marketing strategies for the business & economics library  (bel) at the university of illinois at urbana–champaign (uiuc), a survey was  designed to answer the following questions:       • should bel develop marketing strategies differently for east asian business   students?  • what services do graduate business students want to receive from bel?  • with whom should bel partner to increase visibility at the college of   business?     marketing research techniques were used to gather evidence upon which bel could  construct appropriate marketing strategies.   methods ‐ a questionnaire was used with graduate business students enrolled at uiuc.  the survey consisted of four categories of questions: 1) demographics, 2) assessment of  current library services, 3) desired library services, and 4) research behavior. the data  were analyzed using descriptive statistics and hypothesis testing to answer the three  research questions.    results ‐ east asian business students showed similar assessment of current services as  non‐east asian international business students. survey results also showed that  graduate business students had low awareness of current library services. the business  career services office was identified as a co‐branding partner for bel to increase its  visibility.    conclusion ‐ a marketing research approach was used to help bel make important  strategic decisions before launching marketing campaigns to increase visibility to  graduate business students at uiuc. as a result of the survey, a deeper understanding  of graduate business students’ expectations and assessment of library services was  gained. students’ perceptions became a foundation that helped shape marketing  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:1    70  strategies for bel to increase its visibility at the college of business. creating marketing  strategies without concrete data and analysis is a risky endeavor that librarians, not just  corporate marketers, should avoid.      introduction    corporations develop marketing strategies  for their products or services based on  careful analysis of research findings, and  marketing research has become a distinct  discipline due to its importance within the  field of marketing. according to  armstrong and kotler, marketing research  is the “systematic design, collection,  analysis, and reporting of data relevant to  a specific marketing situation facing an  organization” (113). aaker et al. explain  that marketing research “helps to improve  management decision making by  providing relevant, accurate, and timely  (rat) information” (1). put differently,  marketing research is a discipline based  on the need to identify and discover  evidence upon which corporations can  decide what products to develop or to  withdraw from the market, which  consumer groups they will target for their  advertising campaigns, and which  strategic partners to select. marketing  research does not stop at simply studying  consumer behavior or preferences, but  continues to help create concrete  marketing action plans.    the techniques of marketing research are  highly applicable to libraries that are  seeking to develop and communicate their  services to users. by applying techniques  of marketing research, librarians may  learn essential information seeking  behavior of diverse groups of users in  such areas of preference, satisfaction, and  expectations. furthermore, librarians can  answer such questions as what new  services should be offered, what existing  services should be evaluated for their  effectiveness, and with whom librarians  can collaborate to increase their visibility  and relevance to students and faculty  members.      this paper presents a case study that  discusses how the author applied  marketing research to study a specific  group of students and develop  appropriate marketing strategies based on  the results.      institutional background    the business & economics library (bel)  is one of 40 departmental libraries at the  university of illinois at urbana– champaign (uiuc). it mainly serves  faculty and students at the college of  business that typically enrolls about 3,500  undergraduate and graduate students  each academic year. interestingly, the  majority of graduate business students are  identified as international students;  consistently between 55 and 65 percent  since 1992, according to information  provided by uiuc. furthermore, business  information resources have increasingly  become available electronically, and most  business information resources at bel,  such as statistics databases, scholarly  journals, and financial analysis databases,  are now electronically accessible  regardless of time and location.  nevertheless, there has not previously  been an in‐depth user study to determine  how diverse business student groups at  uiuc react to both traditional and new  services offered by bel.      as bel was introducing new services, the  author sought to understand how  business students perceived bel and its  current services. in doing so, it was hoped  that marketing needs and opportunities  could be determined, and marketing  strategies developed, accordingly. as  mentioned earlier, the majority of the  graduate business students at uiuc are  international students bringing with them  drastically different previous library  experiences and expectations.   many corporations carefully design their  marketing campaigns to meet the needs of  specific populations, since consumer  behaviors vary widely depending upon  evidence based library and information practice 2006, 1:1    71 their demographics. ethnic marketing, for  example, closely assesses the needs of  various ethnic populations in the united  states and creates appropriate marketing  strategies with regards to packaging,  promotions, and retailing. with this in  mind, the author recognized the likely  need to devise different marketing plans  for diverse international business student  populations. instead of grouping all the  international business students into one  population as a homogeneous group, the  author sought to determine whether or not  international business students needed to  be segmented further based on their  nationalities, assuming that international  business students brought distinctively  different previous library experiences  from their home countries which may  have shaped their perceptions of the  library in general.    interestingly, over 70 percent of the  international business students at uiuc  are from four east asian countries – china,  japan, korea, and taiwan. if east asian  business students showed significantly  different reactions to library services from  other international business students, then  bel might need to consider devising  marketing plans especially for them,  which would require further research. to  determine this, the first step was to study  their perceptions of current library  services and determine if their perceptions  were widely different from those of other  international business students.    a conventional marketing research  methodology was followed.  a research  method was developed, data was collected  and analyzed, and the findings were  communicated.  for this study, the  targeted student group was limited to  graduate business students who were  enrolled in master’s degree programs such  as the m.b.a. and master of science in  finance, and doctoral programs in various  business fields.    research design    constructing an appropriate methodology  ultimately determines the validity of any  research effort. the author designed a  survey questionnaire that consisted of 25  questions asking about graduate business  students’ current library use and research  behavior. prior to conducting the survey,  the author obtained approval from the  institutional review board at uiuc to  comply with the university’s policy on the  ethical and legal conduct of human  subjects research. figure 1 illustrates the  research process that the author followed.              exploratory   research  descriptive   research  causal   research  primary   data  secondary   data  research   goals  data  collection  figure 1. research design process  evidence based library and information practice 2006, 1:1    72 research goals    due to the rapid adoption of the internet  as a research tool and changes in  pedagogical requirements for graduate  business programs, it was critical to assess  graduate business students’ perceptions  and expectations of library services. the  results of this research would then become  the foundation on which bel could  formulate its branding, advertising, and  service development strategies. three  questions formed the basis of the research:    • should bel develop marketing  strategies differently for east  asian business students?  • what services do graduate  business students want to receive  from bel?  • with whom should bel partner to  increase visibility at the college of  business?      at the time of this research, many  departmental libraries at uiuc were  introducing new services such as virtual  reference, term paper consultation  sessions, electronic databases tutorials,  and wireless internet access.  however  bel, as a departmental library, did not  have data that would indicate how  graduate business students in particular  reacted to such services, and how those  services truly helped increase bel’s  visibility and relevance. there was a  concern that bel might not be offering  services that graduate business students  actually needed the most. only in‐depth  research would reveal what services they  were hoping to receive, what services  should be continued, and those that  should be considered for termination.    research objectives    according to armstrong and kotler,  marketing researchers make a decision as  to which of three research objectives –  exploratory, descriptive, and causal – is  most likely to accomplish their research  goals (114). the three research types serve  distinctively different objectives.  exploratory research is helpful in  establishing hypotheses for further  research, and therefore, it is generally  used as a preliminary research tool for  descriptive and causal research.  descriptive research, on the other hand,  helps researchers determine consumer  behavior and market potential for  products. causal research is suitable when  researchers seek to understand cause‐and‐ effect relationships. as the goals clearly  demonstrate, the objective of this research  was to conduct descriptive research in  order to study customer behavior (in this  case, the behavior of graduate business  students).    data collection    as figure 1 illustrates, there are two types  of data: primary and secondary.  while  some secondary information found in a  literature review was helpful in  understanding business students’  information seeking behavior in general  terms, the availability of secondary  information specifically on graduate  business students was very limited.  furthermore, since the results would be  the basis of marketing efforts targeting  graduate business students at uiuc,  collecting primary data specific to them  was essential to make this research  relevant.    primary data collection   it was decided to conduct a written survey  via a questionnaire. the author had  previously used an internet‐based survey  for other research projects, and while it  was easy to use, the response rate did not  meet the author’s expectation. based on  this previous experience, a different  approach was taken with this survey. a  research assistant contacted graduate  business students individually, in the  hope that this approach would result in a  high response rate. students were  approached at various locations on  campus, and asked to participate if they  met the requirements (i.e. current full‐time  graduate business students at uiuc).  a  total of 259 responses were gathered. at  the time of the survey, 845 graduate  students were enrolled in the college of  evidence based library and information practice 2006, 1:1    73 business, which indicates a 30 percent  response rate. of the total respondents,  143 students (55 percent) were  international students. the response rate  may be considered low, but the actual  number of the respondents was high  enough to draw conclusions with a fairly  high level of confidence. bel had never  achieved a response rate higher than 30  percent for other research projects  previously.    survey questions  since the research was designed to study  graduate business students’ perceptions  and behaviors, the author used a likert‐ scale for most questions, a tool widely  used for behavioral research.  for example,  the respondents were given a statement  “how important is the availability of  personal space at the library?” and the  scale ranged from 1 (least important) to 5  (very important).  similarly, the  respondents were asked to select the  degree of agreement from 1 (strongly  disagree) to 5 (strongly agree) with a  statement such as “library workshops  introduced me to resources at the library  of which i had not known previously.”      the survey asked respondents to select  specific answers for some questions. it  included questions on their favorite  internet sites for research, the student  program offices (e.g. admissions office,  career services office) with which they had  most frequent contact, and library  offerings that they hoped to receive.    questions were grouped into separate  categories: 1) demographics, 2) assessment  of current library services, 3) desired  library services, and 4) research behavior.    each respondent was asked to indicate his  or her field of study, student status, and  the number of years spent in the united  states as a student if the respondent was  an international student. the respondents  were then asked to indicate the  importance of current library services such  as virtual reference, personal study area,  and the availability of electronic resources.  each respondent also had an opportunity  to list up to five services he or she wished  to receive from bel. lastly, the survey  asked several questions to study graduate  business students’ research behavior.   those questions included:    • which internet site do you visit  most frequently on a daily basis?  • when conducting research for  school projects or personal needs,  which internet site do you  primarily use?  • with whom do you have most  frequent contact? (e.g. curriculum  advisor, career counselor, student  organizations)    data analysis and reporting    once all the results were gathered, the  data was interpreted and analyzed. again,  having clear research goals helped guide  the data analysis stage.  as mentioned  earlier, there were three questions to be  answered from data generated by the  survey. the following sections illustrate  how the data for each question was  analyzed and interpreted.    question 1: do east asian business students  differ from other international business  students in terms of their assessment of library  services?    it was necessary to compare several  student groups when analyzing the data.  while all survey respondents were  graduate business students, they could be  further segmented based on various  factors. segmenting the graduate business  students is important in developing  marketing strategies because different  segments may require different types of  marketing efforts. for instance,  international business students may  possess drastically different information  seeking behaviors from those exhibited by  domestic business students. furthermore,  even among international business  students, information seeking behavior  may be different based on their  nationalities.     evidence based library and information practice 2006, 1:1    74 in the survey, respondents were asked to  identify their citizenship. a total of 25  countries were reported, and students  from the four east asian countries  constituted 66.2 percent of the total  respondents.  the rest of the respondents  were from southeast asia (19.4 percent),  europe (7.2 percent), south america (4.3  percent), and other (2.9 percent).      hypothesis testing was used to determine  if east asian business students had  different library expectations from other  international business students.  hypothesis testing is generally used with  empirical data to draw conclusions on the  difference among two or more samples.  for example, the survey asked the  question “how important is the  availability of personal study space at the  library?”, and this research sought to find  out if the level of importance of personal  study space was different by the place of  citizenship. the means for this question  for east asian and other international  business students were 4.46 and 4.59,  respectively. can we safely conclude that  east asian business students were placing  less emphasis on the availability of  personal study area at bel than did the  rest of the international business students,  or should the conclusion be that, in  actuality, there was no significant  difference between the two student  groups?      to approach this question, the data was  examined using a statistical test for the  difference among student groups. a major  difficulty in comparing the different  student groups was that the data revealed  that responses to the questions with the  likert‐scales were heavily skewed toward  the numbers four (somewhat important or  somewhat agree) and five (very important  or strongly agree).  while the t‐test is well‐ known and widely used for comparing  different populations, it could not be  applied since a critical requirement of the  t‐test is that the result must be a normal  distribution with a bell‐shaped curve, but  the data collected for this research did not  show a normal distribution. for this  reason, the chi‐square test was used to  compare different student groups, since it  is appropriate to test samples that are not  normally distributed but highly skewed.    five services were tested in order to better  understand how international business  students viewed traditional services and  new services (i.e. physical space and  electronic services). overall, the chi‐square  test showed that east asian business  students did not differ significantly from  the rest of the international business    importance of library services:  e. asian vs. non‐e. asian  1. space for personal study      critical value  chi‐square test stat  p‐value  5.991  1.718  0.424  2. public computers      critical value  chi‐square test stat  p‐value  7.815  6.173  0.103  3. electronic resources      critical value  chi‐square test stat  p‐value  5.991  3.440  0.179  4. virtual reference service      critical value  chi‐square test stat  p‐value  9.488  2.004  0.735  5. space for group meetings      critical value  chi‐square test stat  p‐value  7.815  0.156  0.984  table 1. importance of library services: east asian vs. non‐east asian international students  evidence based library and information practice 2006, 1:1    75 students in terms of their perceptions and  expectations of library services, and  therefore, no separate marketing strategies  for east asian business students were  suggested. table 1 provides the results of  the chi‐square test and shows that no  significant differences were found  between the two student groups. the chi‐ square statistics for all questions were  below the critical values, and p‐values also  demonstrated that there existed no  significant differences. when the chi‐ square statistics produce values larger  than the critical values, the results suggest  that the two samples being tested exhibit  different behaviors. as shown in table 1,  the chi‐square test statistics for all five  library services fell below the critical  values, and thus it can be concluded that  there existed no significant differences  between the two student groups with  regards to their perceptions of the five  library services listed in the survey. the p‐ values also suggest the degree of the  probability of wrongly concluding that  there are significant differences. table 1  reveals that the p‐values for all the  services exceeded .05 which is the level of  statistical significance. the conclusion can  be made that the two student groups did  not differ significantly in terms of their  perceptions of the five services.the survey  results showed that space for personal  study was the most important service for  both east asian and non‐east asian  students, followed by the availability of  public computers, electronic resources,  space for group meetings, and vrtual  reference services, in that order of  importance.    question 2: what services do graduate  business students want to receive from bel?    open‐ended questions were used to ask  respondents to list new services that they  hoped bel might offer. respondents  could list up to five services. in analyzing  the data, the author ranked the services  based on the number of times they were  listed by respondents.    surprisingly, the two library services that  graduate business students asked for the  most from bel were wireless access to the  internet (42 percent) and access to  electronic resources from remote locations  (37 percent). these two services were  already being offered at the time of the  research, but graduate business students  did not seem to be aware of them. signage  had been posted in the building stating  that bel was equipped with wireless  connectivity, but it was never marketed  aggressively. this result indicated that     bel needed to market its services more  proactively to graduate business students  so that they could take advantage of the  services currently being offered. it was  suggested that bel needed to emphasize  the offerings of wireless access to the  internet and access to electronic resources  from remote locations more aggressively  through marketing campaigns and  orientations for newly admitted students.  moreover, all m.b.a. students are now  required to have individual laptops, and  thus communicating such services is  essential.      question 3: who could be a strategic partner  for bel to increase its visibility?    to increase the visibility of bel to the  college of business, potential partners  were sought to create a co‐branding  strategy. armstrong and kotler define co‐ branding as “the practice of using the  established brand names of two different  companies on the same product,” which  creates “broader consumer appeal and  greater brand equity” (245). co‐branding  is effective when two organizations do not  compete against each other, and each has  expertise that is widely recognized. this  co‐branding strategy can easily be  replicated by academic libraries in  identifying appropriate partners. libraries  can expand their outreach through co‐ branding with other campus units in ways  that are similar to companies expanding  their markets by collaborating with other  companies already established in the  existing markets. at bel, potential  collaborators were sought for co‐branding  opportunities so that bel could increase  its visibility to graduate business students.  evidence based library and information practice 2006, 1:1    76 • most popular research topic (n=253)  1. career (73 percent)  2. current news (53 percent)    • most frequently used student services (n=243)  1. career counseling (82 percent)  2. faculty advice or curriculum advice (74 percent)    • most popular campus program (n=242)  1. job fairs (52 percent)  2. academic seminars (32 percent)  table 2. finding a co‐branding partner    the survey asked a number of questions  designed to find viable candidates to  become bel’s strategic partners. table 2  shows the results of these questions.  as is  clearly seen, the primary concern for  graduate business students is career  management, and the business career  services office (bcso) at the college of  business seemed to be the best strategic  partner available for bel. unlike other  graduate programs, graduate business  programs are categorized as professional  as opposed to academic, since the main  goal of business schools is to train  students to become business managers  who can solve complex problems in a fast‐ paced global business environment.  graduate business students also have  expectations that degrees such as the  m.b.a. will enhance their chances of  obtaining job offers with high salaries  from reputable companies. consequently,  the average salary for graduates, and job  placement rate, play a central role in  assessing the strength of various graduate  business schools when business  publications, such as business week, wall   street journal, and financial times, rank  graduate business programs.    based on these findings, bsco was  confidently identified as a co‐branding  partner for bel. bcso has great visibility  among graduate business students since         they frequently contact bcso for  assistance. the results of the survey  suggested that business students were  highly attentive to campus activities  dealing with career management issues,  such as effective interview skills, effective  resume writing, and building professional  networks. bcso would be an excellent  marketer of bel’s services, as business  students closely monitor announcements  released from bcso. in turn, bel could  provide substantial expertise as business  students could benefit significantly from  in‐depth company and industry research  available when they prepared their  resumes and cover letters. by having joint  seminars and workshops, bel would  leverage bcso’s high visibility, and bcso,  in return, would help business students  improve their job search processes. at  career‐related seminars hosted by bcso,  bel would demonstrate various business  databases, showing company and  industry information that students would  be interested in, and how to access  resources remotely so students can find  relevant information easily and quickly  without location or time constraints.    conclusion    linking research to practice is a critical  element in evidence based librarianship.  marketing research has been used by  marketing professionals to study user  needs when developing new products and  evidence based library and information practice 2006, 1:1    77 to assess consumer reactions to existing  products. the author used a marketing  research approach to help bel determine  important strategies as it was about to  launch new marketing campaigns to  increase its visibility to graduate business  students at uiuc. as a result of the survey,  a deeper understanding of graduate  business students’ expectations and  assessment of library services was gained.  the research findings suggest that bel  did not have to design marketing  strategies specifically targeting east asian   at uiuc. in addition, promotion of  services such as wireless internet  connectivity and remote access to  electronic databases at bel were   strengthened. the survey results also  helped bel identify bcso as its strategic  partner and a strong relationship with  bcso has begun to develop through  seminars and workshops on career  research. graduate students’ perceptions,  as indicated by the results of the survey,  became a foundation that helped shape  marketing strategies for bel’s efforts to  increase its visibility in the college of  business.                                                        works cited    aaker, david a., v. kumar and george s.  day.  marketing research 6th ed. new  york: john wiley & sons, 1998.    armstrong, gary and philip kotler.  marketing: an introduction. 7th ed.  upper saddle river, n.j.:  pearson/prentice hall, 2005.  evidence based library and information practice 2006, 1:1    78  (page number not for citation purposes)  appendix: survey instrument    part 1        1) gender  1. male  2. female    2) country of citizenship:                  3) program of study  • undergraduate   • master (major:          )  • ph.d. (major:            )  • other (please specify):                4) length of stay in the us (applicable only to international business students)    for how many months have you been studying in the united states?             year          months      part 2    for the questions 5 – 9, please indicate how important each of the following library services is to you at  uiuc.      very  important  moderately  important  neutral  of little  importance  not  important  5) personal study area (i.e. study space)  5  4  3  2  1  6) to use computers  5  4  3  2  1  7) to use library resources for research and  assignments  5  4  3  2  1  8) asking questions via internet chat  5  4  3  2  1  9) group meetings  5  4  3  2  1      part 3      10) when you look for information on the internet, what types of information do you seek most  frequently? (write more than one, if necessary)      11) among many student services at the college of business, with which student services do you  have most frequent contact?  1. faculty/curriculum counseling  evidence based library and information practice 2006, 1:1    79  (page number not for citation purposes)  2. career services and counseling  3. diversity services  4. student associations (please specify: _________________________)  5. other: (please specify: ____________________________________)    12) in which campus events are you most interested?  1. job fairs  2. academic seminars   3. presentations from invited guest speakers  4. field trips to companies  5. athletics/sports   6. multicultural events  7. other (please specify: _____________________________________)        part 4    13) what internet sites do you visit most frequently on a daily basis? (write more than one, if  necessary)        14)  what internet access do you currently have at your residence?  1. dial‐up (56k or less)  2. cable modem  3. dsl  4. high‐speed access provided by dormitory or apartment  5. other (please specify):                  15) do you access electronic library resources from home via the internet?  1. always  2. frequently  3. occasionally  4. rarely  5. never    16) when using electronic resources, do you access them from library computers or home  computers?  1. computers at the library  2. computer at home  3. other (please specify):                evidence based library and information practice 2006, 1:1    80  (page number not for citation purposes)  17)  have you asked librarians for help at the library (please circle one)? yes/no    17‐1. if you responded “yes” to question 27, what kind of questions did you ask?  check all that apply.  1. help locate and retrieve books or articles  2. help with research strategies  3. other (please specify):               17‐2. if you responded “no” to question 27, why did you not seek any assistance?    1. did not need to do research requiring using books, journals, or databases  2. already knew how to use the library without any help  3. did not expect that librarians would understand my area of study  4. other (please specify):               18)  what types of services do you wish to receive from the business & economics library?  (write more than one, if necessary)    19)  the library that i use the most frequently on campus is (e.g. aces, grainger, business &  economics, undergraduate):                  20) the library that i use second most frequently on campus is (e.g. aces, grainger, business &  economics, undergraduate):                  21) when conducting research for your projects, which internet sites do you primarily use?   1. search engines (e.g. google, yahoo!)  2. library home page  3. other: (please specify: __________________________________)    for the following questions (22‐25), please indicate the level of your agreement with the statement, using the following  scale.  this time, please answer based on your experience currently with the library at uiuc.      strongly  agree  somewhat  agree  neutral  somewhat  disagree  strongly  disagree  22) the internet is more useful than library  resources for my research or assignments at  uiuc.    5  4  3  2  1  23) professors or instructors at uiuc encourage  students to use library resources for research and  assignments.    5  4  3  2  1  24) library workshops introduced me to  resources at the library of which i had not known  previously.  5  4  3  2  1  25) my course assignments at uiuc usually  require some research utilizing library materials.    5  4  3  2  1    classics   eysenbach, tuische and diepgen’s evaluation of web searching for identifying unpublished studies for systematic reviews: an innovative study which is still relevant today   a review of: eysenbach, g., tuische, j. & diepgen, t.l. (2001). evaluation of the usefulness of internet searches to identify unpublished clinical trials for systematic reviews. medical informatics and the internet in medicine, 26(3), 203-218. http://dx.doi.org/10.1080/14639230110075459     reviewed by: simon briscoe information specialist national institute for health research (nihr) collaboration for leadership in applied health research and care (clahrc) south west peninsula university of exeter medical school exeter, united kingdom email: s.briscoe@exeter.ac.uk   received: 7 mar. 2016    accepted: 11 may 2016      2016 briscoe. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to consider whether web searching is a useful method for identifying unpublished studies for inclusion in systematic reviews.   design – retrospective web searches using the altavista search engine were conducted to identify unpublished studies – specifically, clinical trials – for systematic reviews which did not use a web search engine.   setting – the department of clinical social medicine, university of heidelberg, germany.   subjects – n/a   methods – pilot testing of 11 web search engines was carried out to determine which could handle complex search queries. pre-specified search requirements included the ability to handle boolean and proximity operators, and truncation searching. a total of seven cochrane systematic reviews were randomly selected from the cochrane library issue 2, 1998, and their bibliographic database search strategies were adapted for the web search engine, altavista. each adaptation combined search terms for the intervention, problem, and study type in the systematic review. hints to planned, ongoing, or unpublished studies retrieved by the search engine, which were not cited in the systematic reviews, were followed up by visiting websites and contacting authors for further details when required. the authors of the systematic reviews were then contacted and asked to comment on the potential relevance of the identified studies.   main results – hints to 14 unpublished and potentially relevant studies, corresponding to 4 of the 7 randomly selected cochrane systematic reviews, were identified. out of the 14 studies, 2 were considered irrelevant to the corresponding systematic review by the systematic review authors. the relevance of a further three studies could not be clearly ascertained. this left nine studies which were considered relevant to a systematic review. in addition to this main finding, the pilot study to identify suitable search engines found that altavista was the only search engine able to handle the complex searches required to search for unpublished studies.   conclusion –web searches using a search engine have the potential to identify studies for systematic reviews. web search engines have considerable limitations which impede the identification of studies.     commentary   background   eysenbach, tuische, and diepgen’s study is the first evidence-based evaluation of how searching the internet using a web search engine can contribute to the identification of studies for systematic reviews, in particular, unpublished clinical trials. the study deserves the status of classic due to its originality and continuing significance; in particular, for proposing and evaluating a systematic approach to web searching which to date is referenced in prominent guidelines for conducting systematic reviews (lefebvre, manheimer & glanville, 2011).   web searching is a common activity for information professionals in almost all library and information settings. systematic reviews, however, are perhaps more familiar to information professionals in health care research settings. systematic reviews answer research questions by identifying and appraising all the relevant studies (using pre-specified eligibility and quality criteria) and synthesizing the accumulated evidence (higgins & green, 2011). they are important in health care settings because there is too much research literature for practitioners to appraise individually. in addition, the methods and conclusions of systematic reviews are less biased than narrative reviews or expert opinion (higgins & green, 2011). it is important to identify unpublished studies, the focus of eysenbach et al., because they may contain findings which are more up-to-date than published studies. there is also evidence suggesting that studies with negative findings are less frequently published or take longer to reach publication (fanelli, 2010).   information professionals contribute to systematic reviews by identifying studies (harris, 2005). research has shown that their contributions improve the quality of systematic reviews (rethlefsen, farrell, osterhaus trzasko, & brigham, 2015). at the time eysenbach et al. was published in 2001, there had been several years of research on the identification of studies for health care systematic reviews using bibliographic databases. early examples of this research include studies by dickersin et al. (1994) and wilczynski et al. (1993) – see also the historical survey of methodological developments in this area by lefebrve et al. (2013). there were also established supplementary search methods for identifying studies, including checking reference lists, hand searching, and searching company trials registries, all of which were detailed in the systematic review guidance manual, the cochrane reviewers’ handbook (now titled the cochrane handbook for systematic reviews of interventions, hereafter, the cochrane handbook) (clarke & oxman, 1999). web searching did not have a prominent place amongst these search methods. this is a view eysenbach et al. verify with reference to the lack of a web searching section in the otherwise comprehensive cochrane handbook .   eysenbach et al. addressed the lack of research and guidance on web searching for systematic reviews, focusing on the use of web search engines to identify unpublished studies. the authors tested the hypothesis that retrospectively conducted web searches, which were adapted from the bibliographic database search strategies of completed systematic reviews, would retrieve previously unidentified and unpublished studies (specifically, clinical trials). they also set out to address practical issues such as the suitability of various search engines for the task.   main results   following the identification of 14 unpublished studies relating to 4 of the 7 included systematic reviews in the study, eysenbach et al. recommended that web searching using a search engine with appropriate search features should be conducted alongside other search methods. they also, however, noted that there was no evidence the searches they conducted affected the outcome of a systematic review. in particular they emphasized that none of the studies they identified contained results that remained unpublished due to negative results. (this would have contributed to the aforementioned evidence that studies with negative results are hard to publish and less likely to be included in systematic reviews (fanelli, 2010).) the authors concluded that web searching using a search engine should be conducted as it has the potential to affect the outcome of a systematic review.   this conclusion is important for being the first evidence-based recommendation on web searching for systematic reviews. the conclusion has been noted in subsequent editions of the cochrane handbook, which currently states that “[t]here is little empirical evidence as to the value of using general internet search engines such as google to identify potential studies”, citing eysenbach et al. as evidence (lefebvre et al., 2011). a forwards citation search on the citation index web of science reveals a total of eighteen citations of eysenbach et al.  the cochrane handbook citation is enough to ensure that health care information professionals with systematic review experience are likely to have seen, or learnt from mentors and on training courses, the main result and conclusion.   the web searching section in the cochrane handbook also advises that searchers might have more success identifying studies by targeting known key websites, such as pharmaceutical companies, than using web search engines. this is an important point considering the inaccessibility of a large portion of the web, known as the invisible or deep web, to the automated web-crawlers which index webpages for search engines (devine & egger-sider, 2013).this is highlighted by eysenbach et al. to improve the efficacy of using search engines the authors recommended that organizations involved in carrying out and funding trials should publish details “on a robot [i.e. web crawler] accessible web page…. using the standard format ‘randomized trial on (intervention) in (condition)’ … so that they can be indexed by search engines and found by systematic reviewers” (p. 216).   eysenbach et al. advocated for the establishment of prospective and ongoing trials registries. this would remove some of the difficulties of finding unpublished trials using web search engines, though the authors anticipated that the web would play an important part in “linking the evidence” between different registries (p. 215). recent developments in this area are detailed below in the discussion of specialized web resources.   pilot study results   in addition to the enduring impact of the main finding of eysenbach et al., the findings from the pilot study remain relevant. in order to effectively adapt bibliographic database search strategies for web search engines, the search engines require similar search features. to this end, the search features of 11 web search engines were assessed: altavista, excite, fast search, google, hotbot, infoseek, lycos, northern light, webcrawler, medical world search, and medhunt. only altavista offered all the required search features, i.e., boolean operators, phrase, proximity, and truncation searching, and capitalization recognition. subsequently, altavista was the only search engine used in the main study.   it remains the case today that bibliographic databases have more advanced search features than web search engines. there have been some improvements to the latter since eysenbach et al. was published. for example, google did not offer boolean searching when eysenbach et al. was published but it does at the time of writing, albeit with limitations. however, the main developments in web search engines have been moving away from complex searches where the user retains a degree of control, towards simple searches where the user increasingly relinquishes control to undisclosed algorithms which determine the relevancy and ranking of the webpages retrieved (granka, 2010; pariser, 2011). this is a challenge for information professionals with complex and detailed information needs, in that search strategy development is limited, frequent changes to algorithms compromise the reproducibility of searches, and bias is introduced in cases where the search history of the user informs the webpages which are retrieved (briscoe, 2015).    the problem of identifying relevant studies with a simple search interface has been exacerbated by the growth of the web. when eysenbach et al. carried out their research in december 1998 there were approximately 2,400,000 websites, whereas in march 2016 there were approximately 1,000,000,000 websites ("total number," 2016). subsequently, the search string (study or trial or random*) near asthma* near (education* or (self near management)), which retrieved 159 hits using altavista in december 1998 (p. 210), retrieved 389,000 hits using google on 4 march 2016. altavista was terminated in 2013 and is unavailable for testing ("yahoo to shut," 2013). the same search on 4 march 2016 in google scholar, which limits results to scholarly literature, retrieved a more focused 37,800 hits, although it is unclear whether the unpublished studies which eysenbach et al. searched for would be indexed in google scholar. the high numbers retrieved indicate that the approach eysenbach et al. used would need to be adapted in order for the results to be manageable. either the searches would need to be made more focused, or the screening of hits would need to be limited to a manageable number (godin, stapleton, kirkpatrick, hanning, & leatherdale, 2015).   the relatively simple search capabilities of web search engines and the growth of the web highlight the importance of assessing the tools and strategies used for web searching, following the example of eysenbach et al. in particular, in an age dominated by google, information professionals should be mindful to seek out and assess other search engines.   the development of specialized web resources   as a solution to the limitations of using web search engines for systematic reviews, eysenbach et al. advocated the creation of “specialized search engines” containing “expert knowledge on which [web]sites ongoing studies are published and [able to] access dynamic databases [i.e. the deep web] and meta-trial registers” (p. 214). no such search engine exists to date, although the launch of the web-based databases clinicaltrials.gov and the isrctn registry (both in 2000) have made it easier to identify unpublished studies, specifically, unpublished clinical trials.   google scholar is a specialized web search engine but it is unable to access the deep web as advocated by eyenbach et al. nonetheless, google scholar is an advance in web searching for the systematic review community, and in recent years there has been research on how it can contribute to systematic reviews. in the health care literature there has been research and debate about whether google scholar can replace bibliographic databases as the main source of studies for systematic reviews (boeker, vach, & motschall, 2013; gehanno, rollin, & darmoni, 2013; giustini & boulos, 2013), general comparisons (not primarily related to systematic review methods) of google scholar with the pubmed database (anders & evans, 2010; nourbakhsh, nugent, wang, cevik, & nugent, 2012; shultz, 2007), and in the environmental science literature, its ability to identify grey literature (haddaway, collins, coughlin, & kirk, 2015). there are varying views on how much google scholar can contribute to systematic reviews, but in most studies the inadequacy of the google scholar search interface for writing complex search strategies is a predominant theme, reflecting the pilot study findings of eysenbach et al.   conclusion   despite the limitations of web search engines and the underwhelming result of eysenbach et al., information professionals who contribute to systematic reviews are likely to continue to use them to identify literature. although there are web-based databases for health care literature, such as the clinicaltrials.gov and isrctn trials registries, web searches using search engines have the potential to retrieve literature not indexed in these resources, or which exist in web resources unknown to the searcher. more research is needed on the potential role of web searching for different types of literature and different types of systematic reviews. evaluations of search engines launched since eysenbach et al. was published are also required. eysenbach et al. will remain a benchmark for future research in these areas, and deserves to be recognized as a classic of the information science literature.   as an aid to future research, eysenbach et al. advocated that systematic review authors should “carefully document their internet search strategy in reports of systematic reviews (rather than just mentioning that ‘internet searches have been performed’) so that factors influencing the effectiveness and necessity of internet searches can be identified” (p. 215). this is a recommendation which research suggests requires more adherence (briscoe, 2015).   references   anders, m. e., & evans, d. p. (2010). comparison of pubmed and google scholar literature searches. respiratory care, 55(5), 578-583.   boeker, m., vach, w., & motschall, e. (2013). google scholar as replacement for systematic literature searches: good relative recall and precision are not enough. bmc medical research methodology, 13, 131. http://dx.doi.org/10.1186/1471-2288-13-131     briscoe, s. (2015). web searching for systematic reviews: a case study of reporting standards in the uk health technology assessment programme. bmc research notes, 8, 153. http://dx.doi.org/10.1186/s13104-015-1079-y   clarke, m., & oxman, a. d. (eds.) (1999). cochrane reviewers' handbook 4.0 (updated july 1999) in: the cochrane library, issue 4. oxford: update software.   devine, j., & egger-sider, f. (2013). going beyond google again: strategies for using and teaching the invisible web. london: facet publishing.   dickersin, k., scherer, r., & lefebvre, c. (1994). identifying relevant studies for systematic reviews. british medical journal, 309(6964), 1286-1291.   fanelli, d. (2010). do pressures to publish increase scientists' bias? an empirical support from us states data. plos one, 5(4), e10271. http://dx.doi.org/10.1371/journal.pone.0010271   gehanno, j. f., rollin, l., & darmoni, s. (2013). is the coverage of google scholar enough to be used alone for systematic reviews. bmc medical informatics and decision making,13, 7. http://dx.doi/10.1186/1472-6947-13-7   godin, k., stapleton, j., kirkpatrick, s. i., hanning, r. m., & leatherdale, s. t. (2015). applying systematic review search methods to the grey literature: a case study examining guidelines for school-based breakfast programs in canada. systematic reviews, 4, 138. http://dx.doi.org/10.1186/s13643-015-0125-0     granka, l. a. (2010). the politics of search: a decade retrospective. the information society, 26(5), 364-374. http://dx.doi.org/10.1080/01972243.2010.511560   haddaway, n. r., collins, a. m., coughlin, d., & kirk, s. (2015). the role of google scholar in evidence reviews and its applicability to grey literature searching. plos one, 10(9), e0138237. http://dx.doi.org/10.1371/journal.pone.0138237   harris, m. r. (2005). the librarian's roles in the systematic review process: a case study. journal of the medical library association, 93(1), 81-87.   higgins, j. p. t., & green, s. (eds.) (2011). cochrane handbook for systematic reviews of interventions version 5.1.0 (updated march 2011). the cochrane collaboration. available from www.cochrane-handbook.org   lefebvre, c., glanville, j., wieland, l. s., coles, b., & weightman, a. l. (2013). methodological developments in searching for studies for systematic reviews: past, present and future? systematic reviews, 2, 78. http://dx.doi.org/10.1186/2046-4053-2-78    lefebvre, c., manheimer, e., & glanville, j. (2011). searching for studies. in j. p. t. higgins & s. green (eds.), cochrane handbook for systematic reviews of interventions version 5.1.0 (updated march 2011). the cochrane collaboration. available from www.cochrane-handbook.org   nourbakhsh, e., nugent, r., wang, h., cevik, c., & nugent, k. (2012). medical literature searches: a comparison of pubmed and google scholar. health information and libraries journal, 29(3), 214-222. http://dx.doi.org/10.1111/j.1471-1842.2012.00992.x   pariser, e. (2011). the troubling future of internet search. the futurist, 45(5), 6.   rethlefsen, m. l., farrell, a. m., osterhaus trzasko, l. c., & brigham, t. j. (2015). librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. journal of clinical epidemiology, 68(6), 617-626. http://dx.doi.org/10.1016/j.jclinepi.2014.11.025   shultz, m. (2007). comparing test searches in pubmed and google scholar. journal of the medical library association, 95(4), 442-445. http://dx.doi.org/10.3163/1536-5050.95.4.442   total number of websites. (2016). retrieved on 4 march 2016 from http://www.internetlivestats.com/total-number-of-websites/#trend   wilczynski, n. l., walker, c. j., mckibbon, k. a., & haynes, r. b. (1993). assessment of methodologic search filters in medline. proceedings of the annual symposium on computer applications in medical care, 601-605.   yahoo to shut down pioneering altavista search site. (2013). retrieved on 4 march 2016 from http://www.bbc.co.uk/news/technology-23127361 evidence summary   user experience research techniques facilitate improvements for access and discovery tools managed by technical services librarians   a review of: hill, k. (2020). usability beyond the home page: bringing usability into the technical services workflow. the serials librarian, 78 (1–4), 173–180. https://doi.org/10.1080/0361526x.2020.1702857   reviewed by: abbey lewis stem engagement librarian university of colorado boulder boulder, colorado, united states of america email: abbey.b.lewis@colorado.edu   received: 17 aug. 2023                                                             accepted: 4 sept. 2023      2023 lewis. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30416     abstract   objective – to demonstrate how user experience research techniques can be incorporated into technical services work. as proof of this concept, the author describes a case wherein a team of librarians, including one in a technical services role, deployed a user experience study to determine if students were able to successfully use libguides and the a-z database list to find subject-specific resources. the study also aimed to gauge the potential for several a-z database list interface redesign options.   design – a case study of user experience techniques applied to technical services projects, including a classic usability test of existing tools and an a/b/c comparison of potential interface redesigns.   setting – the library at the university of north carolina greensboro (uncg), a public r2 university (doctoral university with high research activity).   subjects – eleven student participants recruited through convenience sampling.   methods – the research team recruited study participants who were in the library at the time of the study, deselecting students from uncg’s library school and those who were not currently affiliated with the university through an initial questionnaire. eleven student participants were ultimately selected and led through a series of tasks related to finding subject-specific databases using the a-z database list and libguides. after the tasks for the a-z database list were completed, students were asked for their impression of two additional database list interfaces. students were recorded throughout the tasks using the “talk aloud” method to provide researchers with insights on their thought processes and preferences. following the study, researchers listened to the recordings, coding them as successful or incomplete and noting their observations for use in generalized findings.   main results – eight of eleven participants used the library’s main search box to locate a general resource for their major on the library’s homepage. when shown the a-z database list, ten out of eleven participants used the list to find a database for their major, while one used the link to “research guides by subject” from that page. comparisons of three a-z database list interfaces showed that most students preferred the springshare content management system that allowed for filtering by subject area. when asked to find a research guide for their subject or major from the library’s homepage, nine out of eleven students clicked on the link labeled “research guides by subject.” starting from their subject guide, ten out of eleven could find a tab listing article databases. nine participants noted that the number of databases listed on the guides was daunting.   conclusion – results from the user experience study were used to support a redesign of the a-z database list using the springshare content management system. the author regarded the experience as a whole as demonstrating how technical services librarians can become involved in user experience work and incorporate findings from usability studies into their management and design of tools that promote access and discoverability.    commentary   as the author of the current study notes, “the idea of bringing usability into technical services is not unique to this paper” (hill, 2020, p. 174). in performing their work, technical services librarians shape users’ routes to resources. greater direct knowledge of users’ information-seeking behaviours and perceptions can inform improvements to access and discovery tools (cross & gullikson, 2020). this creates an opportunity to bridge “two seemingly disparate areas of library work… when staff expertise is recognized and valued” (madden, 2020, p. 145). this case study demonstrates a pathway for technical services librarians to engage in usability research in ways that positively influence the library’s ability to meet user needs through the tools and resources that technical services librarians already manage.   the cristal checklist for appraising a user study (n.d.) is used to assess the current study. as part of the consideration of a study’s validity, the checklist asks if researchers collecting data are also those responsible for delivering the service under examination. while this can create a conflict of interest, it can also allow those with the best technical knowledge of a service to rethink its delivery in light of user needs and experiences. this mirrors a point made by the author throughout the study: technical services librarians, as those responsible for the maintenance and development of access and discovery tools, are best positioned to resolve issues with those tools that are identified through user experience research (hill, 2020). also, regarding the cristal checklist’s assessment of a study’s validity, the author makes a strong justification for undertaking this research, noting a previous survey that uncovered difficulties in discovering subject-specific resources among distance education students.   however, this marks a disconnect between the population where the issue was first identified (distance education students) and the population under study. recruiting students passing through the library is ostensibly unlikely to yield participants from that population. still, the use of these tools extends beyond distance education students, making the findings from a more general and convenient population of students in the library still useful for determining common information-seeking behaviours and pitfalls. pre-screening questions were used to filter out those unaffiliated with the university or students in uncg’s library and information science graduate program, appropriately balancing considerations for convenience and relevant representation of the user population.   the author provides the success rates for students who were able to complete the tasks and adequately details the difficulties they encountered that led to failed tasks. students were sometimes able to complete tasks, albeit inefficiently, by using tools in manners other than those intended (i.e., using an a-z database list for subject browsing). these situations are described as well, illuminating the decisions behind some of the interface redesign choices that were influenced by the study. finally, the script used for guiding students through the tasks is provided as an appendix, showing clear, simple tasks that are good representations of the ways librarians expect subject-specific resource needs to be filled. the tasks can ultimately point to discrepancies in the ways that librarians and users view resource discovery. the author helpfully points out that while students might be able to find a link to subject-specific guides when asked directly, seeking out those guides does not always occur to students when they are presented with a less precise request, such as finding a general resource for their major.   while the user experience study itself yielded helpful findings for improving the ways that students discover subject-specific databases, it is important to note that the study’s greatest contribution is through recommendations for technical services librarians who wish to incorporate these techniques into their own practice. in addition to relating the findings back to the reconfiguration of technical services tools, the author provides a helpful and concise primer on usability methodology and offers best practices for collaborating with other teams within the library. this extends the applicability of the study far beyond the exact methods, tools, and user populations explored here and presents a broader view for technical services librarians to consider the scope of their work.   references   cristal checklist for appraising a user study. (n.d.). netting the evidence. http://nettingtheevidence.pbwiki.com/f/use.doc   cross, e., & gullikson, s. (2020). making a case for user experience research to drive technical services priorities. library resources & technical services, 64(2), 89–98. https://doi.org/10.5860/lrts.64n2.89   hill, k. (2020). usability beyond the home page: bringing usability into the technical services workflow. the serials librarian, 78(1–4), 173–180. https://doi.org/10.1080/0361526x.2020.1702857   madden, l. (2020). a new direction for library technical services: using metadata skills to improve user accessibility. serials review, 46(2), 143–146. https://doi.org/10.1080/00987913.2020.1782648   microsoft word news_ebl4.doc evidence based library and information practice 2006, 1:1  136 evidence based library and information practice     news    4th international conference on evidence‐based library and information practice      joanne gard marshall  local conference chair  e‐mail: marshall@ils.unc.edu      © 2006 marshall. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    announcing the 4th international conference  on evidence‐based library and information  practice (eblip4), may 6‐9, 2007 with  additional continuing education events on  may 10‐11, chapel hill, north carolina,  usa    this exciting international conference has  grown out of an increasing interest in using  the best available evidence to improve  library and information practice. the  conference will be of interest to all sectors of  the library and information community. it  will offer an opportunity for presenting high  quality research as well as discussing the  role and future of evidence‐based practice in  the profession.     the conference is sponsored by the school  of information and library science at the  university of north carolina at chapel hill   http://www.ils.unc.edu/. the site of the  conference will be the sheraton imperial  hotel in nearby research triangle park  http://www.sheratonrtp.com/. the hotel is a  10‐minute ride from the raleigh durham  international airport http://www.rdu.com/.     for further information contact: joanne  gard marshall, local conference chair.  e‐mail: marshall@ils.unc.edu  http://creativecommons.org/licenses/by/2.0 http://www.ils.unc.edu/ http://www.sheratonrtp.com/ http://www.rdu.com/ evidence summary   pubmed central: an essential resource for information professionals and researchers   a review of: cornell, a., bushman, b., & womack, k. (2011). analysis of journals that did not meet selection criteria for inclusion in the national library of medicine collection but have manuscripts in pubmed central. journal of the medical library association, 99, 168-170. doi: 10.3163/1536-5050.99.2.011   reviewed by: joanne l. jordan research information manager arthritis research uk primary care centre, keele university keele, staffordshire, united kingdom email: j.jordan@keele.ac.uk   received: 28 nov. 2012 accepted: 8 apr. 2013      2013 jordan. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – a review of the journals containing research listed in pubmed central (pmc), but not selected for inclusion in the national library of medicine (nlm) collection. the authors identified reasons why journals had not been included in the collection and if any met the nlm selection criteria and were appropriate for inclusion.   design – descriptive study.   setting – national library of medicine, united states.   subjects – 571 journals that were not included in the nlm collection but had research articles in pmc.   methods – in october 2009, a report was produced from the nlm library system listing journals tagged as having articles in pmc and not being in the nlm collection. information was gathered on the journals identified and these were checked against the collection development manual of the nlm and the nlm checklist used for selecting electronic journals. the reason for non-selection of the journal was recorded and the subject category, according to the library of congress classification, was noted.   recorded reasons why journals were not selected:   ·         less than 15% of articles were within scope of nlm collection ·         not enough articles published ·         coverage (lacking original research or not for a scholarly audience) ·         insufficient information to determine reason   for journals where the criteria seemed to be met, the decision on selection to the nlm collection was reviewed.   main results – the authors identified 571 journals that had articles in pmc but did not meet the criteria for inclusion in the nlm journal collection. the majority of these journals (73%) were outside the nlm scope and a further 10% had not published a sufficient number of articles to be considered. a further 3% were assessed as not intended for a scholarly audience or lacked original research and another 3% could not be reviewed due to lack of information available. there were 65 journals (11%) that were referred for further review as the selection criteria seemed to be met and 11 of these journals have subsequently been added to the nlm collection. this is in relation to 482 new print and electronic journals in total that were added to the nlm collection in 2009.   however, only 369 of the 571 journals (65%) had one or more articles included in pmc; of these, 238 had one article and 33 had more than four articles in the archive. the reason that some journals had no articles in pmc at the time of this review was due to the time it takes to process new articles and embargos set by the publishers that restrict immediate listing on open access databases such as pmc. a number of these journals may also be new and may not have had a sufficient number of articles or enough information available to be able to include them in the nlm collection. to add context, the authors state that pmc contained over 115,000 nih-funded articles by the end of november 2010.   the subject areas these non-selected journals were classified under included engineering (15%); medicine (14%); mathematics (10%); chemistry (10%); and computer science (9%). library science was assigned to 2% of the journals. the medicine journals were more likely than those in the other subject areas to be new journals without sufficient articles to be included in the nlm collection.   conclusion – when the journal title is out of the scope of the nlm collection, an individual article in that journal can still be included in pmc. this provides a solution to the problem of how to collect biomedical research that is not published in biomedical journals. this may be more important in the future as the field becomes more interdisciplinary. this also provides a useful resource for libraries and researchers searching for full-text biomedical articles.   the authors conclude that analyzing the articles from the journals not selected for inclusion in the nlm collection will provide helpful information about the types of biomedical research being published in non-biomedical journals. this will highlight particular areas the nlm should pay attention to in the future.     commentary   this paper highlights an important issue for health information professionals and researchers; a proportion of health-related research is not published in biomedical journals and therefore is not likely to appear in nlm bibliographic databases such as medline. the size of this problem will depend on the subject and type of research being conducted and will have less impact on searches for mainstream medical literature than those working closely with, e.g., the social sciences, information technology, engineering, and information science. those relying on medline to search for citations may miss articles unless specific subject databases are searched as well, and these can be less well known and more difficult to access and search.   for a systematic review, where it is vital that the search is comprehensive, even a few missing citations could jeopardize the rigour of the review and produce misleading results. anyone conducting a systematic review should be using a variety of different databases to retrieve citations, but may not have considered adding pmc to the list. searching pmc as well as medline may be one way to retrieve a few of the research papers that might otherwise have been missed.   the authors make references to the size of pmc at the time the paper was prepared. however, they have not given much detail about pmc and the non-nih funded articles that it contains. as of may 2012, pmc contains 2.4 million articles in over 3,000 journals according to their website (http://www.ncbi.nlm.nih.gov/pmc/). the archive contains not only journal articles resulting from nih-funded research, but also over 1,000 journals that have allowed the full-text articles from all their issues to be deposited in pmc. funders of research in other countries, such as the national institutes of health research (nihr) in the u.k., also require researchers to provide open access to articles in a repository. these articles are deposited in uk pubmed central, which then becomes part of pmc. the pmc archive is a key resource for information providers and researchers because it provides access to the full text of a diverse selection of health-related research that may otherwise be difficult to obtain.   while it is of concern that health-related research is published in journals not listed in the nlm collection, it is encouraging that this has not gone unrecognized and ways of capturing this information are being explored. the number of additional articles from journals that are not in the nlm collection is small in relation to the size of the archive at present. however, it is important to review this group of articles on an annual basis to ensure that the proportion of articles in journals not selected for the nlm collection is not growing. as the authors say, the world of health research has become more diverse and boundaries between disciplines are not distinct. organizations such as the nlm will need to keep up with future developments.   fostering evidence-grounded dialogue in a multi-institutional digital library using evidence in practice   fostering evidence-grounded dialogue in a multi-institutional digital library   leah duncan digital collections librarian louisiana state university libraries baton rouge, louisiana, united states of america email: lpowe17@lsu.edu   elizabeth joan kelly library web and applications development administrator louis: the louisiana library network baton rouge, louisiana, united states of america email: elizabeth.kelly@laregents.edu   sophia ziegler head of digital programs and services louisiana state university libraries baton rouge, louisiana, united states of america email: sziegler1@lsu.edu   received: 1 apr. 2022                                                                   accepted: 14 apr. 2022      2022 duncan, kelly, and ziegler. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30139     setting   this report presents the findings of a survey of louisiana-based librarians, archivists, and museum professionals’ processes for digitization selection and digital collection outreach and assessment. survey participants were administrators at cultural heritage institutions that contribute to the louisiana digital library (ldl), a state-wide resource for sharing digital heritage content.   the survey was part of a larger, grant-funded project, “...and 25 of our closest friends: the louisiana digital library as community-focused data (“the ldl,” n.d.) awarded by collections as data: part to whole (n.d.), with funds made possible by the andrew w. mellon foundation. the main goal of the project team was to bring together ldl professionals as a community of practice and explore the policy, ethics, and implementation of reconceptualizing the ldl as data. the ldl community had never before gathered in such a way because, structurally, the ldl is a decentralized network of contributing institutions. the survey team is composed of librarians who work with the ldl community to provide ongoing training, support, and development, while individual institutions have autonomy over their own collections.   problem   this decentralized model means there is not a unified vision for the role of the ldl in representing louisiana cultural heritage. different institutions pursue different goals related to the digital library’s research value, representational inclusivity, and scope. we also did not understand what steps, if any, individual institutions take to assess the use of their own content, and if that information ever influences what they choose to digitize and add to the library.    furthermore, participating institutions vary widely in their size, staffing, and endowments, and we did not know how many professionals at each institution work with the ldl, or how staffing levels influence digitization and collection assessment. without this basic information it is difficult to develop vision statements for the digital library, facilitate sustained planning sessions with our community of content administrators, or equitably distribute ldl resources.   evidence   using qualtrics, we distributed a survey to gain insight into the current practices of ldl institutions in relation to (1) digitization selection, and (2) assessment of collection content, use, and reuse (ziegler et al., 2020). the authors distributed the survey to 27 ldl content administrators who served as the primary decision-makers regarding their institution’s digital collections. in total, 22 content administrators responded, representing 15 academic libraries, three public libraries, two museums, one cultural heritage institution, and one respondent who chose not to disclose their institution type.   we found that digitization selection staffing and processes vary widely. fifty-five percent of respondents had one to two full-time employees contributing to digitization initiatives, and 27% have more than two. nine percent had less than one full-time employee contributing. the remainder of respondents did not have ongoing digitization initiatives. regarding workflow, 36% of responding institutions had certain individuals who chose what gets digitized, 25% based their digitization selection decisions on community and patron requests, 7% had a digitization selection committee, and 7% based selection on grant funding and donations. the remaining 25% did not have a solidified process for deciding what would be digitized from their collections. sixty-one percent of respondents expressed interest in modifying their selection strategy.   most respondents indicated they performed some form of collection assessment. fifty-eight percent assessed the content and/or scope of current digital collections, and 62% of those institutions used metadata assessment to evaluate content and scope. of the institutions that did not assess collection content or scope, 33% cited lack of training as a barrier, 27% cited lack of personnel, and another 27% cited lack of documented standards or best practices.   in addition to content assessment, 65% of participating institutions collected usage statistics for their collections. when asked why they collect usage statistics, 19% of respondents indicated a desire to demonstrate the impact of collections on scholarship, 17% to improve digital collections services, and 17% to improve digitization selection decisions.   only 29% of respondents collected reuse data. collected data most frequently included citations in academic scholarship, published or reposted digital objects in digital media such as websites and digital exhibits, and sharing of digital objects through social media. of the institutions that did not collect reuse data, 39% selected lack of documented standards or best practices as the main barrier, 26% selected lack of training, and 22% selected lack of personnel.   implementation   survey results suggest that the project team should provide training and shared best practices and principles to flexibly guide ldl participating institutions, which use a wide variety of processes for making digitization selection decisions and for assessing collection content, use, and reuse. any attempt to superficially impose a set of one-size-fits-all standards or workflows would unnecessarily alienate some of our institutions. our diverse community of institutions is better served by an ongoing dialogue around selection and assessment that leverages the expertise distributed across the ldl.    in an effort to begin building this dialogue, the project team facilitated the “ldl as data online speaker series”, during which we virtually hosted digital library practitioners and users from across the country. we recruited speakers who could address topics that respondents indicated would be of interest, such as selecting digital projects that fill representational gaps, and assessing users and reuse. these talks highlighted priorities such as inclusivity and community engagement, bringing them freshly to the forefront of community discussion.   we also hosted a content administrator focus group, during which we received more specific feedback about technical developments that would facilitate user-centered digitization selection and assessment.   finally, the project team established the “ldl as data fellowship”, which supported four researchers in creating small-scale digital projects using ldl collection data. the fellowship concluded with the “lunch with the ldl fellows brown bag series”, which allowed each fellow to present their project to ldl content administrators. this initiative not only produced specific ldl use cases, but also enabled discussion around ways in which our digital collections might be used and reused.   outcome   after the conclusion of the “ldl as data online speaker series” and focus group, we distributed a second survey to evaluate the impact of these initiatives on strategies ldl administrators use to select content for digitization and to assess collection content, scope, use, and reuse. fifty-seven percent of respondents indicated that the project enhanced their understanding of strategies for digitization selection, and 57% indicated that the project enhanced their understanding of assessing digital collections.   in addition to providing knowledge and best practices for selection and assessment, we hoped our efforts would foster stronger connections among ldl institutions. seventy-one percent of respondents indicated that the project helped to build community among ldl contributors and administrators. furthermore, when asked to explain how the project benefitted them, several respondents focused on the opportunity to exchange ideas with and learn from colleagues at other institutions. one respondent wrote,   meeting with other ldl administrators (along with the knowledge gained from the ldl as data speaker series) benefited me by providing an understanding of the various aspects of data and how we use that data....these meetings gave me a sense of what other ldl administrators were doing individually and as part of the ldl...not only did i gain so much value from the collaborative meetings and the speaker series as an ldl administrator, but this helped me in assessing our digitization and data needs.   another respondent wrote, “i can't stress enough the usefulness of the resources, communications, and collaborations. it has made me a better content administrator.”   there is still much room for improvement within the ldl as technology and as community. although 38% of respondents report that they intended or hoped to make changes to how their institution decides what to digitize, and 57% reported that they intended or hoped to make changes to how they assess collections and use/reuse, lack of personnel and funding remain a major barrier to such developments.    reflection   given the multi-institutional and decentralized structure of the ldl, we find it ineffective to strictly implement standards that would apply to every contributing institution, from small cultural heritage museums to large academic libraries. instead, we chose to take an evidence based approach to fostering community, productive dialogue, and flexible best practices.   working with over 30 unique and autonomous institutions is never straightforward, but the evidence we acquired through our survey allowed us to facilitate discussion around topics relevant to our diverse community of administrators. without this evidence, as project team members, we would have relied on our own experience to predict relevant topics, and likely would have excluded institutions that are dissimilar to our own. based on what the project team learned, we continue to grow the ldl community through events, workshops, and other learning opportunities that bring us together in our shared pursuit of providing a unified portal to the digital cultural heritage of our state.    author contributions   leah duncan: conceptualization, funding acquisition, investigation, methodology, writing – original draft elizabeth joan kelly: conceptualization, data curation, formal analysis, funding acquisition, investigation, methodology, writing – review & editing sophia ziegler: conceptualization, funding acquisition, investigation, methodology, project administration, writing – review & editing   references   collections as data: part to whole. (n.d.). retrieved april 30, 2022, from https://collectionsasdata.github.io/part2whole/   the ldl as data. louisiana digital library. (n.d.). retrieved may 5, 2022, from https://louisianadigitallibrary.org/ldlasdata     ziegler, s., powell duncan, l, & kelly, e. j. (2020). louisiana digital library collections as data intro survey: selected survey results to support reports and publications. lsu digital commons.  https://digitalcommons.lsu.edu/libraries_pubs/97   evidence based library and information practice editorial responsibilities      2019. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     editor-in-chief: lorie kloda   associate editor (evidence summaries): stephanie krueger, christina wissinger   associate editor (research articles): ann medaille, lisl zach   associate editor (classics & reviews): jane schmidt   associate editor (commentaries, using evidence in practice): lorie kloda   communications officer (news): tatiana bryant   production editor: rachel hinrichs   editorial intern: kimberly mackenzie   editorial advisors: lindsay alcock, alison brettle, michelle dunaway, jonathan eldredge, denise koufogiannakis   copyeditors: heather healy (lead copyeditor), georgianne bordner, julie evener, julie james, emily kingsland, alison moore, jane morgan-daniel, elizabeth stregger, elaina vitale   indexing support: kate shore news/announcements   call for applicants for eblip journal: associate editor and communications officer      2016. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   evidence based library and information practice (eblip), a peer reviewed open access journal published since 2006, is expanding and restructuring. the journal seeks to fill two positions to complete our editorial team. the first position is that of associate editor for the classics and reviews sections. the second position is that of communications officer. candidates will be expected to be able to begin their roles by october 2016 and serve a 3 year term (with possibility of renewal).   the associate editor (classics and reviews) will be responsible for:   overseeing the complete editorial process for items submitted to the classics and reviews sections (this includes assignment of peer reviewers, monitoring the peer review process, communicating with authors and peer reviewers, making acceptance decisions, ensuring required changes to manuscripts are made, communicating with copyeditors and the production editor to ensure the final copy is as expected). identifying classic studies to be summarized, and soliciting writers. communicating and consulting with the editor-in-chief on a regular basis. attending editorial team meetings, via skype, on a monthly basis and participating in the overall governance of the journal. ensuring that all necessary deadlines are met. communicating with potential authors and responding to queries.   contributions to the eblip classics section follow a structured format designed to highlight, summarize and critically appraise research studies that have stood the test of time and that have had (and continue to have) an impact on library and information practice. previous "classics" have included the work of william postell, constance mellon, carol kuhlthau, joanne marshall, and robert taylor. for an example of a classic, see http://ejournals.library.ualberta.ca/index.php/eblip/article/view/1760/3331   review articles provide a way for librarians to obtain an overview of the evidence on a particular topic, and stay current with the literature. reviews may take on several different forms, including an overview, systematic review, meta-analysis, literature review, state-of-the-art review, or umbrella review, among others.   for more information about the classics, reviews, and other types of submissions accepted by eblip see: https://ejournals.library.ualberta.ca/index.php/eblip/about/submissions#authorguidelines   the ideal candidate for associate editor (classics and reviews) will be well-versed in evidence based practice and research methods. this position requires dedicated time on a regular basis, and it is therefore essential that interested persons ensure available time to devote to this position prior to applying. it is estimated that the workload is approximately 8-10 hours per month.   the communications officer, a new position on the editorial team, will be responsible for:   developing and implementing a communications plan for the journal. developing, designing, writing, and editing material to promote the journal, as requested by the editor-in-chief. this can include email messages to mail lists such as calls for applicants, posting on social media, updating the announcements section of the eblip journal, and powerpoint presentation for conferences. communicating and consulting with the editor-in-chief on a regular basis. attending editorial team meetings, via skype, on a monthly basis. ensuring that all necessary deadlines are met.   the ideal candidate for communications officer will have demonstrable skills and knowledge in areas of communications, preferably in the scholarly/academic realm. excellent written communication skills in english are required. experience with developing and executing communication plans, and an interest in evidence-based librarianship and open access are considered assets. this position requires dedicated time on a regular basis, and it is therefore essential that interested persons ensure available time to devote to this position prior to applying. it is estimated that the workload is approximately 5-10 hours per month.   interested persons should send a statement of interest, indicating the post of interest, cover letter, as well as a resume/cv to lorie kloda, editor-in-chief, at lorie.kloda@concordia.ca by august 1, 2016.   **please note that evidence based library and information practice is a non-profit, open access journal and all positions are voluntary and unpaid.   about the journal:   published quarterly, and hosted by the university of alberta, this peer-reviewed, open access journal is targeted at all library and information professionals interested in an evidence based model of practice. by facilitating access to librarianship research via original research articles and evidence summaries of relevant research from the library literature, evidence based library and information practice will enable librarians to practice their profession in an evidence based manner.   please visit the evidence based library and information practice website (http://ejournals.library.ualberta.ca/index.php/eblip) for further information about the journal.   evidence summary   information literacy course yields mixed effects on undergraduate acceptance of the university library portal   a review of: chen, y. (2015). testing the impact of an information literacy course: undergraduates' perceptions and use of the university libraries' web portal. library & information science research, 37(3), 263-274. http://dx.doi.org/10.1016/j.lisr.2015.04.002   reviewed by: heather coates digital scholarship & data management librarian university library indiana university-purdue university indianapolis (iupui) indianapolis, indiana, united states of america email: hcoates@iupui.edu   received: 7 mar. 2016    accepted: 15 apr. 2016      2016 coates. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine the effects over time of a 3-credit semester-long undergraduate information literacy course on student perception and use of the library web portal.   design – mixed methods, including a longitudinal survey and in-person interviews.   setting – information literacy course at a comprehensive public research institution in the northeastern united states of america.                 subjects – undergraduates at all levels enrolled in a 3-credit general elective information literacy course titled “the internet and information access.”   methods – a longitudinal survey was conducted by administering a questionnaire to students at three different points in time: prior to instruction, near the end of the course (after receiving instruction on the library portal), and three months after the course ended, during the academic year 2011-2012. the survey was created by borrowing questions from several existing instruments. it was tested and refined through pre-pilot and pilot studies conducted in the 2010-2011 academic year, for which results are reported. participation was voluntary, though students were incentivized to participate through extra credit for completing the preand post-instruction questionnaire, and a monetary reward for completing the follow-up questionnaire. interviews were conducted with a subset of 14 participants at a fourth point in time.   main results – 239 of the 376 (63.6%) students enrolled in the course completed the preand post-instruction questionnaire. fewer than half of those participants (111 or 30% of students enrolled) completed the follow-up questionnaire. participants were primarily sophomores and juniors (32% each), with approximately one-quarter (26%) freshman, and only 10% seniors. student majors were concentrated in the social sciences (62%), with fewer students from science and technology (13%), business (13%), and the humanities (9%). the 14 participants interviewed were drawn from both highand low-use students.   overall, the course had a positive effect on students’ perception of usefulness (pu) and ease of use (peou), as well as usage of the library portal. this included significant positive changes in perceived ease of use and information quality in the short-term (from pre-instruction to post-instruction). the results were mixed for perceived usefulness and system quality. though there was mixed long-term impact on usage, the course does not appear to have had a long-term effect on pu and peou. the interview participants were asked questions to explore why and how they used the library portal, and revealed that both highand low-use students used the library portal for similar reasons: to find information for research papers or projects, to search the library catalogue for books, and in response to a mandate or encouragement from instructors.   conclusion – the study supports the theory that an information literacy course could change student perception and use of the library portal in the short-term. replicating this design in other settings could provide a systematic approach for assessing whether information literacy courses address learning outcomes over time. a longitudinal approach could be useful for comparing proficiency and information behaviors of those who take information literacy courses with those who do not.   commentary   this well-designed study has several strengths and offers a model for future research. the use of technology acceptance models to assess library resource use is an interesting approach, particularly when combined with instructional intervention. applying glynn’s (2006) critical appraisal checklist indicates that overall validity is good, particularly in relation to the study design, data collection, and results. however, readers should be cautious in generalizing the results given that the study used a non-random sample of a student population that may not be representative of their local student populations.   the primary strengths of the study are its careful design and execution. two well-tested models, technology acceptance model and the information systems success model, informed the development of the questionnaire, which was piloted twice. its face validity appears to be good. although timing may have been a factor in the attrition from the post-instruction to follow-up phases, this possibility was not discussed. the results are clearly reported and connected back to the hypotheses.   statistical analysis is an area for improvement in future studies. use of a one-tailed t-test only detects changes in the constructs (pu, peou, and portal usage) in one direction. this choice increases the ability to detect positive changes at the expense of detecting negative changes. additionally, it is unclear whether a key assumption for using the dependent t-test is met – the author does not report whether the differences between the paired scores are normally distributed.   the smaller sample size at follow-up raises two questions. were the long-term effects of instruction undetected because the sample size was too small? were the students who completed the follow-up phase different in some meaningful way from the students who did not? neither of these considerations is explored in the article. finally, readers would have benefited from deeper examination of the partially-supported hypotheses. in particular, what implications do they have for the validity of the questionnaire and use in future studies? how could those concerns be addressed or explored in future studies?   this study is particularly relevant to librarians engaged in course integrated information literacy instruction, instructional coordinators, and assessment librarians. it provides a model for examining the impact of information literacy instruction on student use of library resources. considerations for future studies include gathering additional information on student demographics and experience with particular library resources, as well as carefully considering the timing of the follow-up survey and interviews. finally, a preand post-skills assessment administered in conjunction with the technology acceptance questionnaire could be powerful for identifying potential relationships between information literacy skill level and acceptance of library resources. reference   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154   conference paper   our future, our skills: using evidence to drive practice in public libraries   gillian hallam adjunct professor, lis discipline science and engineering faculty queensland university of technology brisbane, queensland, australia email: g.hallam@qut.edu.au   robyn ellard senior program manager, public libraries state library victoria melbourne, victoria, australia email: rellard@slv.vic.gov.au   received: 11 aug. 2015  accepted: 19 oct. 2015      2015 hallam and ellard. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – the public library sector’s future prosperity is contingent upon a well-trained, experienced, and valued workforce. in a collaborative initiative, state library victoria (slv) and the public libraries victoria network (plvn) commissioned an in-depth research study to examine the skills requirements of staff across the state. the our future, our skills project sought to identify the range of skills used by public library staff today, to anticipate the range of skills that would be needed in five years’ time, and to present a skills gap analysis to inform future training and development strategies.   methods – the project encompassed qualitative and quantitative research activities: literature review and environmental scan, stakeholder interviews, focus groups and a workforce skills audit. the research populations were staff (individual survey) and managers (management survey) employed in 47 library services, including metropolitan, outer metropolitan and regional library services in victoria.   results – the high response rate (45%) reflected the relevance of the study, with 1,334 individual and 77 management respondents. the data captured their views related to the value of their skillsets, both now and in five years’ time, and the perceived levels of confidence using their skills. the sector now has a bank of baseline evidence which has contributed to a meaningful analysis of the anticipated skills gaps.   conclusions – this paper focuses on the critical importance of implementing evidence-based practice in public libraries. in an interactive workshop, managers determined the skills priorities at both the local and sectoral levels to inform staff development programs and recruitment activities. a collaborative slv/plvn project workgroup will implement the report’s recommendations with a state-wide workforce development plan rolled out during 2015-17. this plan will include a training matrix designed to bridge the skills gap, with a focus on evaluation strategies to monitor progress towards objectives. the paper provides insights into the different ways in which the project workgroup is using research evidence to drive practice. the world of public libraries is highly dynamic, with staff being challenged to provide customers with a broad and diverse array of services and programs in an environment characterised by tight public sector budgets, ever-evolving technologies, a changing customer base and an ageing workforce. in australia, the state library victoria (slv) has collaborated with public libraries victoria network (plvn), the peak body for victoria’s 47 public library services, to address the challenges. over the past decade a number of initiatives have been undertaken to envision the public library service of the future:  libraries building communities (state library victoria (slv), 2005); workforce sustainability and leadership (van wanrooy, 2006; considine, jakubauskas & oliver, 2008); connecting with the community (slv, 2008); tomorrow’s library (ministerial advisory committee (mac), 2012); and victorian public libraries 2030 (also referred to as vpl 2030) (slv, 2013a).   specifically, the vpl 2030 study (slv, 2013a) sought to establish a strategic vision for public library services in victoria. following extensive consultation and a series of workshops to explore community attitudes and needs, two scenarios for the future were developed: the community library and the creative library. the final report introduced a strategic framework which serves as a planning tool to ensure the sector’s ability to meet the community’s expectations for the two scenarios. it was emphasised that “a workforce of well-trained, experienced and valued public library staff will be at the heart of our success” (slv, 2013a, p.1).    this statement provided the impetus for a further state-wide study into the knowledge, skills and attributes which staff would need to deliver future-focused library services and programs in the community library and the creative library. the research activities in the victorian public libraries: our future, our skills project (slv, 2014) focused on the workforce planning issues which would underpin the successful achievement of the vpl 2030 goals.   objectives   the our future, our skills project sought to identify the range of skills currently used by public library staff in their work, to anticipate the skills which would be needed in five years’ time, and to present a skills gap analysis to inform future training and development strategies. the overarching objectives of the our future, our skills project were to develop a framework to articulate the core competencies required by the public library workforce for the 21st century, to conduct a skills audit of victorian public library staff in order to collect data about the current skills and to anticipate future requirements, and to deliver a report which analysed the audit findings and made recommendations on training needs and strategies to support the future delivery of public library programs and services in victoria.   based on the skills presented in the framework, three key questions were posed:   how important is each skill to your current role? how important do you think each skill will be to the same role in five years’ time? how confident do you feel in your ability to apply this skill in the work that you do?   managers of library services were asked to consider the skills within the context of the library service:   how important is each skill to the library service at the current time? how important do you think each skill will be to the library service in five years’ time?   the rich research data collected in the project will enable future recruitment and staff development practice to be guided by meaningful evidence. one of the biggest challenges faced by practitioners, however, is to understand how to translate this evidence into practice.   methodologies   the our future, our skills project involved a number of qualitative and quantitative research activities, including stakeholder interviews, a literature review and environmental scan, the development of a skills framework to guide the design of the survey instruments, a series of focus groups, and a skills audit of the public library workforce in victoria. in the earlier research project, considine, jakubauskas and oliver (2008) delineated three areas of workplace skills (cited in mounier, 2001):   cognitive skills – foundation or general skills obtained on the basis of general citizenship (for example, literacy, numeracy, general education competence) technical skills – the skills associated with the purchase of labour on the open market to perform particular tasks (for example, the ability to operate machinery/technology, recognised trade or professional skills) behavioural skills – personal skills associated with labour’s ability to deal with interpersonal relationships and to perform in the context of authority relations on the job (for example, communication, empathy, reliability, punctuality).   this overarching model was adopted for the literature review and environmental scan (slv, 2014, appendix 2) with the structure of the discussion built around these three skills areas. the changing world is driving the need for an increased focus on contemporary cognitive skills, or foundation skills (mounier, 2001) which are also described as 21st-century skills (ministerial council on education, employment, training and youth affairs, 2008; partnership for 21st century skills, 2008; institute of museum and library services, 2009). in order for citizens to successfully participate in and contribute to a dynamic society, a new range of literacies is required (unesco & ifla, 2012; unesco, 2013; institute of museum and library services, 2015a; institute of museum and library services, 2015b). the combination of information literacy, media literacy, digital literacy and technological literacy form a new metaliteracy (mackey & jacobson, 2011; o’connell, 2012; jacobson & mackey, 2013; mackey & jacobson, 2014). in order for public libraries to remain relevant and meaningful in the future, staff will need to demonstrate these 21st-century skills.   technical skills – or professional skills, as they are termed in this study – may be more familiar to library workers. traditionally, education and training in the library and information science (lis) field has led to proficiency in the relevant professional skills (hirsch, 2012). professional associations and other bodies have developed their own lis frameworks to define the typical areas of professional practice (sla, 2003; ala, 2009; lifelong learning uk, 2011; alia, 2012; lianza, 2013; cilip, 2013; gutsche & hough, 2014). the australian library and information association (alia) core knowledge, skills and attributes policy document (alia, 2012) guides the curricula of accredited library and information science (lis) education programs in australia. some critics argue, however, that lis education practice fails to meet the workforce requirements of the contemporary public library sector (partridge et al., 2011; pateman & willimen, 2013; bertot, sarin & percell, 2015).   information and communications technology (ict) skills, especially the competencies required for the application of web 2.0 technologies in libraries to facilitate participation, interaction and co-creation of content are becoming increasingly important (king, 2007; cullen, 2008; harvey, 2009; peltier-davis, 2009; partridge, menzies, lee & munro, 2010). mobile literacy is required by library staff in order to broaden access to library resources and services and to link emerging technologies with new opportunities to engage library users (murphy, 2011; saravani & haddow, 2011).   the final skills area, behavioural skills, has been widely discussed in the professional literature (partridge & hallam, 2004; chan, 2005; precision consulting, 2006; barrie, hughes & smith, 2009; oliver, 2011; alia, 2012). it is argued that the profession requires a richness and diversity of behavioural skills, with many employers stating that they wish to appoint staff who have the ‘soft skills’, i.e. the personal and interpersonal skills, that are pertinent both to the lis profession and to the wider employment environment (kennan, cole, willard, wilson & marion, 2006; ralph & sibthorpe, 2010; reeves & hahn, 2010; howard, 2010; partridge, menzies, lee & munro, 2010; partridge et al., 2011; haddow, 2012).  communication skills (wilson & birdi, 2008; working together project, 2008; abram, 2009; saunders & jordan; 2013), teamwork and collaboration skills (bagshaw, 2013), adaptability, and flexibility (chawner & oliver, 2013) are viewed as particularly important.   after the draft skills framework was reviewed by the project reference group, it was examined and discussed by library staff in a series of 15 focus groups held across victoria. the focus group activities involved a total of 133 participants, representing all levels of the workforce in small, medium, and large library services and corporations, as well as library educators from the higher education and the vocational education and training (vet) sectors. the framework was subsequently affirmed by the project reference group as the foundation for the our future, our skills survey activities. the final version of the framework includes 10 foundation skills, 30 professional skills and 19 behavioural skills (slv, 2014, appendix 3).   research subjects for the study were all staff and managers employed in public libraries in victoria. all 47 library services, including metropolitan, outer metropolitan, and regional library services, were invited to participate in the project.  the total number of potential respondents was 2,975.    two survey instruments were developed for the skills audit: the individual survey (slv, 2014, appendix 6), which was completed by individual staff in different library services, and the management survey (slv, 2014, appendix 7), which was open to selected senior staff with managerial responsibilities and an understanding of the strategic direction of their library service.    the questions in the individual survey focused on the individual staff member’s own skill sets and confidence levels, while the questions in the management survey examined the relevance of various skills to the library service as a whole. the individual surveys were made available to all staff employed in victorian public libraries, as prospective respondents, through an online platform. given the length of the questionnaires, the survey tool was designed to allow respondents to answer the questions progressively, rather than all in one session.   the individual survey comprised four sections:   demographics foundation skills professional skills behavioural skills.   an explanation of the scope of each skill area was provided, as well as descriptors which typically represent the area of practice. there were two open-ended questions at the end of each section to offer respondents the opportunity to provide an indication of where they might benefit from support and training, and to comment further on the skills area. at the conclusion of the survey, respondents were invited to indicate how they believed their role might change over the coming five years, and to outline any ‘hidden talents’ they had that might be of value to the library service.   the management survey was more condensed than the individual survey and asked only two questions for each of the skill areas: the importance of the skill set to the library service today, and the anticipated importance of that skill to the library service in five years’ time. descriptors were again provided for each skill. library managers had the opportunity, through open-ended questions, to give their views on why there might, or might not, be any change over the coming five-year period. they could also offer general comments about the three skills areas.   the draft survey instruments were made available for pilot testing. this study was underpinned by the principles of research quality to ensure that the overall study design and the research questions resulted in reliable and valid research findings. the reliability of the research design was considered in the development and testing of the survey instrument. the vocabulary used throughout the survey was kept consistent and descriptors were provided for each competency area to assist the respondent in relating the skills to their work role. for the pilot, the online questionnaires were reviewed by a small representative sample of library staff, drawn from different employment band levels and working with different library services in victoria. some of the pilot testers had participated in the focus groups, while others had not. some minor adjustments to the questionnaires were made in response to feedback from the pilot group.   the survey was open from 13 november 2013 to 20 december 2013. to verify the integrity of respondents, library staff members were required to register for the survey using their work email addresses and were then sent a system-generated password that enabled them to access the questionnaire. the research team worked closely with the project team to respond to any technical issues encountered. the stability and technical performance of the online platform were monitored closely throughout the survey period. the systems developer was able to monitor the registrations received to ensure there were no duplicate registrations. incomplete surveys were excluded from the analysis.   respondents were advised that their involvement in the survey was completely voluntary and that they could withdraw at any time without penalty. the research data collected remained anonymous and confidential; email addresses were replaced with sequential numbers to ensure respondents’ complete anonymity in the data analysis. the margin of error for the individual survey was calculated to be 3.7%; it was higher for the management survey, given a smaller sample size, at 8.9%.   results   the high response rate (45%) reflected the relevance of the our future, our skills study, with 1,334 valid responses to the individual survey and 77 valid responses to the management survey. responses to the individual survey were received from 45 library services, with response rates ranging from 7% to 100%. managers from 37 library services contributed to the research through the management survey. single responses were received from 19 libraries, while the remaining 18 library services provided between two and six management responses.    the vpl 2030 report invites those involved in the public library sector in victoria to begin to think strategically about ways in which “public library staff, programs and facilities can be better equipped to adapt and innovate to meet changing community needs towards 2030 and beyond” (slv, 2013b, p.2). the data collected in the individual and management surveys in the our future, our skills project represent a bank of baseline evidence which can contribute to a deeper understanding of the opportunities and challenges.   the picture of the current workforce presents confident and competent staff who deliver the library services that have long been valued by users. the respondents’ strengths tend to reflect the core knowledge and skills presented in alia’s policy document (alia, 2012) which is used to guide the professional and vocational education programs in australia. the vpl 2030 report stresses, however, that the status quo cannot continue; changing community attitudes and behaviours will have a significant impact on the role libraries play and on the programs and services they deliver. inevitably, the ability to successfully enable the current public library model to move to an alternative paradigm that is relevant to the evolving information and learning needs of different communities will depend on the competence and confidence of those working in the sector. public libraries will face the critical challenge of transitioning effectively and smoothly from a passive, product-based model to one which can deliver dynamic, service-based experiences (slv, 2013a, p.17). public library staff will need to become actively engaged with the evolving social trends of creativity, collaboration, mental engagement, learning and community connection.    one of the primary drivers for societal change will inevitably be the continuing influence of technology, as acknowledged in the vpl 2030 report: “technological advancements and improved access to technology continue to enable scientific breakthroughs and new social behaviours to emerge” (slv, 2013a, p.11). the creative library scenario is underpinned by developments in information and communications technologies (ict), while the push towards globalisation, which is directly linked to the adoption of new technologies, influences the community library scenario. as the vpl 2030 report presents only limited commentary about the skills requirements for these future public library scenarios, the research data has been examined to consider how the findings relate to three strategic perspectives: the technology environment, the creative library and the community library.   skills for the technology environment   the data collected through the our future, our skills surveys and the focus group discussions revealed that there was a very keen awareness amongst public library managers and staff about the challenges of the fast changing technology environment. without a doubt, digital literacy represents a fundamental foundation skill needed by library staff. staff in all roles and at all levels will increasingly need to demonstrate high levels of digital literacy as they apply their information and media skills in a dynamic online world. in the context of public libraries today, digital literacy skills were ranked as the fourth most important foundation skill by library managers (69% ‘extremely important’). literacy, cultural literacy and local awareness skills were identified as the three principal foundation skills for contemporary library staff, with literacy viewed as the paramount skill. it was overwhelmingly apparent, however, that in five years’ time, digital literacy skills would be just as important as traditional literacy skills, with 94% of managers rating this skillset as ‘extremely important’ and the remaining 6% stating it would be ‘important’. an enormous increase in significance was also anticipated by individual library staff, with those rating it as ‘extremely important’, jumping from 58% to 84% in the five year timeframe. it was recognised that all library staff would quickly need to become fluent in the area of digital literacy and that operational ict skills would become mainstream:   ict policy and planning development and management of ict systems in the library integration of social media and mobile applications into library operations provision of ict support to customers management of digital resources creation and maintenance of metadata schema.   behavioural skills also came under scrutiny: the dynamic and ever-evolving technology environment demands flexibility, with staff encouraged to respond positively and confidently to constant change and to willingly accept new work assignments and job responsibilities. creative thinking and problem solving were likely to become essential skills in a less predictable world: public library staff would need to be able to seek out and promote new ideas and to test novel approaches to resolving operational issues. a commitment to lifelong learning would be an imperative, with staff prepared to take responsibility for their ongoing learning and professional development through avenues of both informal and formal learning.   while technological developments were clearly going to make a significant impact on library operations in the coming years, one major area of concern for public libraries was the staff members’ present levels of confidence in utilising the various skillsets. the gap between managers’ expectations for the importance of the different skill areas and the number of staff who stated that they felt ‘very confident’ (likert scale 5) about utilising the skill in their current role is depicted in figure 1. relatively strong levels of confidence were recorded for the behavioural skills, but in the area of the professional skills relating to ict in libraries, the confidence levels were extremely low.       figure 1:  gap analysis: skills required in the technology environment.* *managers’ expectations (‘extremely important’) and individual confidence (‘very confident’).   figure 2:  gap analysis: skills required in the technology environment.* *managers’ expectations (‘extremely important’) and individual confidence (‘very confident’)     figure 2 presents the foundation skills and professional skills data as a spidergram; the managers’ expectations for future skills requirements for the library (dark blue) are contrasted with the individuals’ current levels of confidence. the graph shows both medium and high levels of confidence by presenting the aggregated responses for ‘confident’ (likert scale 4) and ‘very confident’ (likert scale 5) (mid blue), as well as the specific data for ‘very confident’ (likert scale 5) (light blue).   as the graph illustrates, while the current skill level appears to be low – especially in relation to the professional skills – there is a small kernel of staff who have the appropriate skill sets and a further group of staff who feel their skills are developing well. importantly, opportunities may exist within the workforce to draw on the skills of these people to contribute to informal staff development activities, e.g. through work shadowing and mentoring.   it is essential that public library staff commit to the process of skills development to be able to perform at a high level in this dynamic technology environment. as few areas of the public library of the future were likely to remain untouched by ict developments, there was a clear sense that the entire workforce would need to up-skill if staff were to operate productively in the world of electronic information and to help members of the community develop their own digital literacy skills. training and development activities to address the current low levels of digital literacy and ict skills should be regarded as a high priority for the victorian public library sector.   skills for the creative library   the depiction of a future creative library, as outlined in the vpl 2030 report, is heavily influenced by technology, particularly through the application of participative and collaborative tools to create and share digital resources in a range of media formats. the creative public library has been described as an active learning centre; community arts studio; and collaborative work space. as staff of the creative library become “facilitators of creative development, expression and collaboration” (slv, 2013a, p.21), they will need the skills and abilities to run a broad selection of creative and learning programs which contribute to building the inventive capacity of the community. in this environment, public library staff will be required to use a variety of skills to manage and coordinate both internal and external resources:   to facilitate content sharing to connect people to teach new skills ·         to nurture untapped talent to produce, record and edit creative content to host business collaboration to manage people to coordinate multiple diverse activities within the library and across different stakeholder groups.   the skills relevant to the creative library can be mapped to all three fields of the skills framework: digital literacy and cultural literacy as key foundation skills; cultural programming, creative making, and literacies and learning as professional skills; creative thinking, problem solving, customer engagement, building partnerships and alliances, and lifelong learning as important behavioural skills. at the same time, a wider range of professional skills should not be ignored, as staff in the creative library will need to draw on their understanding of the ict environment and their skills in information seeking, eresource management, information services, project management, marketing and promotion.   figure 3 correlates staff confidence levels with the managers’ expectations about the future importance of the specific areas of competency required for the creative library. high levels of confidence recorded for customer service and cultural literacy contrasted strongly with low levels of confidence recorded for the areas of literacies and learning, cultural programming, creative making and building relationships and alliances.      figure 3:  gap analysis: skills required for the creative library scenario.* *managers’ expectations (‘extremely important’) and individual confidence (‘very confident’)     in figure 4, the spidergram presents the comparison between the managers’ expectations for future skillsets and the individual respondents’ data: high levels of confidence (likert scale 5) and the combined levels of ‘confident’ (likert scale 4) and ‘very confident’ (likert scale 5).     figure 4:  gap analysis: skills required for the creative library scenario.* *managers’ expectations (‘extremely important’) and individual confidence (‘very confident’)     the graphs highlight the areas where skills development is essential if victorian public libraries are to achieve the aspiration of meeting the expectations of a creative community.    skills for the community library   the second scenario discussed in the vpl 2030 report was the community library. in this scenario the library plays the role of  “a learning village” (slv, 2013a, p.25), with the potential to play a central role as community learning centre; gathering place; brain gymnasium; repository, documenter and disseminator of local knowledge; and local business hub. the responsibilities of public library staff in this environment are broad ranging: to develop community capacity by connecting people who have either similar interests or complementary skills. the effective management and coordination of internal and external resources would again be integral to the success of the library.   the essential competencies for staff of the community library can be drawn from all three fields of the our future, our skills framework. significant foundation skills would include local awareness to comprehend the socio-demographic and cultural characteristics of the different populations who use library services, as well as those of non-users. while the staff who deliver programs and services in the community library would continue to draw on some of their more traditional skillsets (e.g. information seeking, resource management, people management, project management, and marketing and promotion), the most critical professional skills would relate to community development. the field of community development encompasses community needs analysis, for example, through socio-demographic analysis, community profiling and community mapping; community engagement, especially in relation to issues of social exclusion; and establishing productive relationships with other community groups and volunteers. this last skillset is closely aligned with the behavioural skills relating to building partnerships and alliances across the public and private sectors, which in turn would be augmented by skills in political and business acumen in order to contextualise the environment in which the library operates. other behavioural skills such as effective communication, customer engagement and empathy would continue to be important.      figure 5:  gap analysis: skills required for the community library scenario.* *managers’ expectations (‘extremely important’) and individual confidence (‘very confident’).     the community library scenario would therefore require a mix of skills that are, arguably, underdeveloped at the present time. although staff confidence levels were marginally stronger in this scenario, a degree of discord was still apparent when they were compared with the value placed on the skillsets by library managers, particularly in the context of community development skills. the gap between managers’ views of the future of these skill areas and the confidence felt by staff is presented graphically in figure 5.    this data is further amplified in figure 6 to show the combined individual responses for ‘confident’ (likert scale 4) and ‘very confident’ (likert scale 5).     figure 6:  gap analysis: skills required for the community library scenario.* *managers’ expectations (‘extremely important’) and individual confidence (‘very confident’).     strategies for the future   konrad (2010) stresses that the development of staff competence is intrinsically linked to organisational development, so library leaders face the challenge of ensuring that their staff have the right skills to work in an organisation that encourages and supports interdisciplinary teams and networks within and across the cultural sector. staff will also need to be able to respond and contribute to an organisation that has the capacity to embrace the “processes of change and development as a permanent condition for the sector” (konrad, 2010). the future scenarios of the creative library and the community library anticipate “a flexible and inclusive organisational culture that attracts and retains people with the right skills and attitude” (slv, 2013a, p.31) in order to design and deliver the programs and services that will place the public library service at the centre of an active and engaged community. the “right skills and attitude” encompass a range of the competency areas presented in the our future, our skills framework. some of these skillsets may be regarded as ‘traditional’ lis skills while others can be described as ‘new’ skills.    the research findings indicated that public library staff perform well in those areas where their skills have long been tried and tested: they are “well-trained, experienced and valued” (slv, 2013a, p.1). however, some of the competency areas that are directly relevant to the creative library and the community library can be described as underdeveloped. while some of the skills are beginning to become relevant to public library practice, staff levels of confidence are very low. this is particularly the case with the skillsets relating to cultural programming, creative making, literacies and learning, and community development. the final report for the our future, our skills project included a number of recommendations relating to the development of a set of priorities to help position the victorian public library sector for the delivery of future-focused programs and services, to use the skills framework as a multi-purpose workforce planning tool, and to develop a productive staff training and development framework.   translating evidence into practice   the aim of the our future, our skills research project was to establish an evidence-based assessment of the training needs of victorian public library staff. a major issue was, once the research activities had concluded and the research data had been analysed, how should the findings and recommendations be used in practice? while many public library managers may be becoming increasingly aware of the importance of evidence-based library and information practice, they face significant difficulties when it comes to translating evidence into practice.   the research data and recommendations were acknowledged to be key ingredients for future planning efforts. nevertheless, during the initial review and discussion of the final report by the workgroup which had been responsible for commissioning the project, it became clear that many practitioners in the library and information sector struggled with translating research evidence into everyday solutions. in order to assist librarians in interpreting the data and the final recommendations, a number of actions were implemented. in july 2014, victorian library service managers, ceos and senior staff were invited to attend an interactive workshop led by the researcher. the research process was described, the recommendations were examined and the priorities for skills development and training programs were discussed. workshop participants identified the skills priorities at both the local and sectoral levels to inform staff development programs and recruitment activities.   the workshop provided a good foundation for senior library managers to understand the importance of the recommendations, the rigour of the research and the value of applying evidence based practice in libraries. one of the recommendations in the final report was the development of a workforce action plan for victorian public libraries: this became the collaborative workgroup’s first undertaking. it was agreed that the action plan should align with the victorian public library sector’s strategic direction and agreed priorities, as outlined in the vpl 2030 document.   the project workgroup undertook an analysis of the two reports, vpl 2030 and our future, our skills, to establish the critical priorities for the training programs to be delivered during the 2014-2017 timeframe. the workgroup identified the levels and positions of staff that would benefit from specific training and development, as well as the preferred methods of delivery for the training programs. four key themes were identified:   building partnerships and alliances community development and engagement digital literacy 4.       collection development.   in developing the workforce action plan, the project workgroup scrutinised each of the four key themes to identify:   why the theme was critical to the work of public libraries what skills should be included under each theme what skills gaps currently existed what training and development response was required how to ensure that the response truly reflected the needs of the public library workforce (e.g. improving access to training opportunities for staff in regional libraries) which library staff would benefit most from the different types of training.   the collated information was translated into a training matrix to drive workforce development over the period 2015-2017. the value of the evidence collected through the our future, our skills project was acknowledged through a successful grant application to the r.e. ross trust, a charitable trust in victoria. one specific area of interest is to provide funding for initiatives which offer “improved access to and achievement of equity and excellence in public education, arts and culture”. the application submitted by slv recognised the innovative nature of the research work and highlighted the importance of implementing the recommendations. the funding will support the delivery of the training program in a range of formats across the state to ensure an equitable spread of professional development opportunities to ensure that public libraries and their staff are well equipped to service the needs of all victorian communities now and into the future.   in order to further build on the evidence base, evaluation strategies have been developed to allow the workgroup to monitor the progress made towards reducing the skills gaps. a two-stage evaluation involves assessing the impact of each training course with each participant at the end of the event to consider the effectiveness of the learning activities, and again four weeks later via a survey designed to explore the impact the training had on the attendees’ methods, attitude and approach to their work. after the first training event, it was found that levels of confidence had increased, with some respondents reporting significant changes:   i have put in a funding application for a lego mindstorm program after hearing how successful it was at geelong regional library service.   [i’ve] bought an ipad for my own professional learning and development.   [there’s been] more of a change in attitude underpinning my relationships with colleagues and community groups.”   all the events have had an impact on my thinking and will influence how i will structure my future career i am sure that it has already impacted on my practice.   conclusions   library managers face immense challenges to fully comprehend the nature and value of the evidence gathered through the our future, our skills project and to develop effective and workable strategies for a strong and successful public library sector. the library workforce will be integral to the sector’s future success and it is essential that managers develop a clear set of priorities to address skills needed in the future. some workplace tasks which are currently viewed as routine will inevitably be subject to ongoing change: technological developments will streamline mundane activities and some traditional library roles will become redundant. together, slv and plvn are well positioned to use the vpl 2030 strategic framework as the springboard to progressively introduce a range of training programs which will enable staff to develop new resources and services to meet the changing needs of the communities they serve.   in both the creative library and community library scenarios, emphasis is placed on members of the community striving to develop new knowledge and skills. accordingly, in this dynamic learning environment, it is essential that staff employed in public libraries are also motivated to see themselves as learners. at the conclusion of the study, managers of a number of individual library services requested the analysis of the data directly relating to their staff, to be compared with the aggregated state-wide data. key areas of strength could be identified, as well as those areas where the skill sets were particularly underdeveloped. the differences between metropolitan and regional libraries highlighted the opportunities to support knowledge exchange and skills development across the state though staff exchanges, job swaps and peer mentoring programs. as it will be important to monitor the impact of the workforce development plan, the survey instrument can be used again to collect updated evidence in order to measure the progress being made towards the upskilling of staff over time.   as libraries forge new directions, alternative career pathways will emerge, with roles that require people to draw on a different range of skills. the value of the research activities undertaken in the public library sector in victoria extends beyond this immediate context, not only to public libraries in other jurisdictions within australia and overseas, but also to other sectors of library and information practice. as a living document, the skills framework can be used as multi-purpose workforce planning tool to raise awareness among library staff about the importance and value of the range of skills which underpin high quality practice, to support performance planning and review processes in libraries, to review staffing structures in order to align skills requirements with library programs and services, to support the recruitment of high calibre library staff, and to advocate on library workforce issues with key stakeholders. the research should also stimulate debate between practitioners, educators and professional associations about the future direction of lis education with the goal of ensuring a strong future for the sector.    references   abram, s. (2009). the new librarian: three questions. information outlook, 13(6), 37-38.   american library association (ala) (2009). core competences of librarianship. retrieved july 1, 2015 from http://www.ala.org/educationcareers/careers/corecomp/corecompetences   australian library and information association (alia) (2014). the library and information sector: core knowledge, skills and attributes. retrieved on july 1, 2015 from http://www.alia.org.au/about-alia/policies-standards-and-guidelines/library-and-information-sector-core-knowledge-skills-and-attributes   bagshaw, m. (2013, march 1). is there no ‘i’ in ‘teamwork’? library journal: backtalk retrieved november 22, 2015 from http://lj.libraryjournal.com/2013/03/opinion/backtalk/is-there-no-i-in-teamwork-backtalk/   barrie, s., hughes, c., & smith, c. (2009). the national graduate attributes project: integration and assessment of graduate attributes in curriculum. sydney: australian learning and teaching council. retrieved july 1, 2015 from http://www.itl.usyd.edu.au/projects/nationalgap/resources/gappdfs/national%20graduate%20attributes%20project%20final%20report%202009.pdf   bertot, j.c., sarin, l.c., & percell, j. (2015). re-envisioning the mls: findings, issues, and considerations. college park, md: university of maryland college park, college of information studies. retrieved august 15, 2015 from http://mls.umd.edu/wp-content/uploads/2015/08/reenvisioningfinalreport.pdf    chan, d. (2005, jun.). core competencies for public librarians in a networked world.  data, information, and knowledge in a networked world. proceedings of the annual conference of the canadian association for information science. london, ontario. retrieved july 1, 2015 from http://www.cais-acsi.ca/ojs/index.php/cais/article/view/167/657   chartered institute of library and information professionals (cilip) (2013). professional knowledge and skills base. retrieved july 1, 2015 from http://cilip.org.uk/cilip/jobs-and-careers/professional-knowledge-and-skills-base    chawner, b., & oliver, g. (2013). a survey of new zealand academic reference librarians: current and future skills and competencies. australian academic& research libraries, 44(1), 29-39.     considine, g., jakubauskas, m., & oliver, d. (2008). workforce sustainability and leadership: survey, analysis and planning for victorian public libraries. sydney: university of sydney, workplace research centre. retrieved on july 1, 2015 from http://www.libraries.vic.gov.au/downloads/workforce_survey_analysis_and_planning_project/finalworkforcereport.pdf   cullen, j. (2008). catalyzing innovation and knowledge sharing: librarian 2.0. business information review, 25(4), 253-258.   gutsche, b., & hough, b. (2014). competency index for the library field. dublin, oh: webjunction. retrieved july 1, 2015 from http://www.webjunction.org/content/dam/webjunction/documents/webjunction/2015-03/competency%20index%20for%20the%20library%20field%20(2014).pdf   haddow, g. (2012). knowledge, skills and attributes for academic reference librarians. australian academic and research libraries, 43(3), 231-248.   harvey, m. (2009). what does it mean to be a science librarian 2.0? issues in science and technology librarianship, 58, 2. retrieved july 1, 2015 from http://www.istl.org/09-summer/article2.html   hirsch, s. (2012). preparing future professionals through broad competency planning. information outlook, 16(1), 9-11.   howard, k. (2010). programming not required: skills and knowledge for the digital library environment. australian academic & research libraries, 41(4), 260-275.   institute of museum and library services (imls) (2009). museums, libraries and 21st century skills. washington, dc: imls. retrieved 1 july, 2015, from http://www.imls.gov/assets/1/assetmanager/21stcenturyskills.pdf   institute of museum and library services (imls) (2015a). museums, libraries and 21st century skills. retrieved november 22, 2015 from https://www.imls.gov/issues/national-initiatives/museums-libraries-and-21st-century-skills   institute of museum and library services (imls) (2015b). museums, libraries and 21st century skills: definitions. washington, dc: imls. retrieved november 22, 2015, from https://www.imls.gov/impact-imls/national-initiatives/museums-libraries-and-21st-century-skills/museums-libraries-and-21st-century-skills-definitions   jacobson, t.e., & mackey, t.p. (2013, apr.). what’s in a name?: information literacy, metaliteracy or transliteracy. presentation at the meeting of acrl: imagine, innovate, inspire. association of college & research libraries conference 2013, indianapolis in. retrieved on 1 july, 2015 from http://www.slideshare.net/tmackey/acrl-2013   kennan, m.a., cole, f., willard, p., wilson, c., & marion, l. (2006). changing workforce demands: what job ads tell us. aslib proceedings, 58(3), 179-196.   king, d.l. (2007, july 11). basic competencies of a 2.0 librarian, take 2 [web log post]. retrieved november 22, 2015 from http://www.davidleeking.com/2007/07/05/basic-competencies-of-a-20-librarian/   konrad, i. (2010). future competence needs in public libraries. scandinavian public library quarterly, 43(4), 8-9. retrieved on july 1, 2015 from http://slq.nu/?article=denmark-future-competence-needs-in-public-libraries   library and information association of new zealand (lianza). taskforce on professional registration (2013). lianza professional registration board professional practice domains and bodies of knowledge. december 2013. retrieved november 22, 2015 from  http://www.lianza.org.nz/sites/default/files/updated%20lianza%20professional%20practice%20domains%20and%20bodies%20of%20knowledge%20-%20v3.pdf   lifelong learning uk (lluk) (2011). national occupational standards: libraries, archives, records and information management services. retrieved july 1, 2015 from http://www.slainte.org.uk/news/archive/1103/larims%20nos%20final%20approved%20feb11.pdf   mackey, t.p., & jacobson, t. e. (2011). reframing information literacy as a metaliteracy. college & research libraries, 72(1), 62-78.   mackey, t.p., & jacobson, t.e. (2014). metaliteracy: reinventing information literacy to empower learners. chicago, il: ala neal-schuman.   ministerial advisory council on public libraries (mac) (2012). review of victorian public libraries: stage 1 report. retrieved on november 22, 2015 from  http://www.dtpli.vic.gov.au/local-government/public-libraries/tomorrows-library-stage-1-and-2   ministerial council on education, employment, training and youth affairs (mceetya) (2008). melbourne declaration on educational goals for young australians. retrieved on july 1, 2015 from  http://www.curriculum.edu.au/verve/_resources/national_declaration_on_the_educational_goals_for_young_australians.pdf   mournier, a. (2001). the three logics of skill in french literature. australian centre for industrial relations research and training (acirrt) working paper for the board research project: the changing nature of work vocational education and training to enable individuals and communities to meet the challenges of the changing nature of work. sydney: university of sydney, acirrt.  retrieved on november 22, 2015 from  http://ses.library.usyd.edu.au/bitstream/2123/13397/1/wp66.pdf   murphy, j. (2011, april 1). social networking literacy for librarians. acrl paper update [web log post]. retrieved july 1, 2015 from http://joemurphylibraryfuture.com/social-networking-literacy-for-librarians-acrl-paper-update/   o’connell, j. (2012). learning without frontiers: school libraries and meta-literacy in action. access, 26(1), 4-7.   oliver, b. (2011). good practice report: assuring graduate outcomes. sydney: australian learning and teaching council. retrieved july 1, 2015 from http://www.olt.gov.au/resource-assuring-graduate-outcomes-curtin-2011   partnership for 21st century learning (p21) (2008). framework for 21st century learning. retrieved july 1, 2015 from http://www.p21.org/our-work/p21-framework    partridge, h., menzies, v., lee, j., & munro, c. (2010). the contemporary librarian: skills, knowledge and attributes required in a world of emerging technologies. library & information science research, 32(4), 265-271.   partridge, h. l., hanisch, j., hughes, h. e., henninger, m., carroll, m., combes, b., genoni, p., reynolds, s., tanner, k., burford, s., ellis, l., hider, p., & yates, c. (2011). re-conceptualising and re-positioning australian library and information science education for the 21st century: final report. retrieved november 22, 2015 from http://eprints.qut.edu.au/46915/   partridge, h., & hallam, g. (2004, sept.). the double helix: a personal account of the discovery of the structure of the information professional’s dna. in challenging ideas. alia 2004 biennial conference. gold coast, australia. retrieved july 1, 2015 from http://eprints.qut.edu.au/1215/   pateman, j., & willimen, k. (2013). developing community-led public libraries: evidence from the uk and canada. farnham, surrey, uk: ashgate publishing.   peltier-davis, c. (2009). web 2.0, library 2.0, library user 2.0, librarian 2.0: innovative services for sustainable libraries. computers in libraries, 29(10), 16-21.   precision consultancy (2006). employability skills: from framework to practice. an introductory guide for trainers and assessors. canberra: department of education, science and training. retrieved july 1, 2015 from http://www.velgtraining.com/library/files/employability%20skills%20from%20framework%20to%20practice%20%20an%20introductory%20guide%20for%20trainers%20and%20assessors.pdf   ralph, g., & sibthorpe, j. (2010). learning from job advertisements for new zealand special librarians. new zealand library & information management journal, 51(4), 216-235.   reeves, r. k., & hahn, t. b. (2010). job advertisements for recent graduates: advising, curriculum, and job-seeking implications. journal of education for library & information science, 51(2), 103-119.   saravani, s-j., & haddow, g. (2011). the mobile library and staff preparedness: exploring staff competencies using the unified theory of acceptance and use of technology model. australian academic & research libraries, 42(3), 179-190.   saunders, l., & jordan, m. (2013). significantly different?: reference service competencies in public and academic libraries. reference & user services quarterly, 52(3), 216-223.   sla (2003). competencies for information professionals in the 21st century. revised edition. retrieved july 1, 2015 from http://www.sla.org/about-sla/competencies/   state library of victoria (slv) (2005). libraries building communities: the vital contribution of victorian public libraries – a research report for the library board of victoria and the victorian public library network. melbourne: state library of victoria. retrieved on july 1, 2015 from http://www2.slv.vic.gov.au/about/information/publications/policies_reports/plu_lbc.html   state library of victoria (slv) (2008). connecting with the community. melbourne: state library of victoria.  retrieved on july 1, 2015 from http://www2.slv.vic.gov.au/pdfs/aboutus/publications/lbcreportcommunity.pdf   state library of victoria (slv) (2013a). victorian public libraries 2030: strategic framework. melbourne: state library of victoria. retrieved on july 1, 2015 from www.plvn.net.au/sites/default/files/20130527%20final%20vpl2030%20full%20report_web.pdf   state library of victoria (slv) (2013b). victorian public libraries 2030: strategic framework. summary report. melbourne: state library of victoria. retrieved on july 1, 2015 from http://plvn.net.au/sites/default/files/20130528%20final%20vpl2030%20summary%20report_web.pdf   state library of victoria (2014). victorian public libraries: our future, our skills: research report. melbourne: state library of victoria. retrieved on july 1, 2015 from http://www.plvn.net.au/sites/default/files/skills%20audit%20report%20final.pdf     united nations educational, scientific and cultural organization (unesco) & international federation of library associations (ifla) (2012). the moscow declaration on media and information literacy. retrieved july 1, 2015 from http://www.ifla.org/publications/moscow-declaration-on-media-and-information-literacy   united nations educational, scientific and cultural organization (unesco) (2013). media and information literacy for knowledge societies. moscow: unesco. retrieved july 1, 2015 from http://www.ifapcom.ru/files/news/images/2013/mil_eng_web.pdf   university of sydney, workplace research centre.  retrieved july 1, 2015 from http://www.libraries.vic.gov.au/downloads/public_libraries_unit/final_workforce_scoping_report_jul_06.pdf   van wanrooy, b. (2006). workforce sustainability and leadership: scoping research. sydney: university of sydney, workplace research centre. retrieved july 1, 2015 from http://www.libraries.vic.gov.au/downloads/public_libraries_unit/final_workforce_scoping_report_jul_06.pdf   wilson, k. & birdi, b. (2008). the right man for the job?: the role of empathy in community librarianship. sheffield: university of sheffield. retrieved july 1, 2015 from http://www.shef.ac.uk/polopoly_fs/1.128131!/file/ahrc-2006-8-final-report-04.08.pdf   working together project (2008). community-led libraries toolkit. retrieved july 1, 2015 from http://www.librariesincommunities.ca/resources/community-led_libraries_toolkit.pdf   english literature students at spanish university have positive perceptions towards but limited understanding of online resources evidence summary   english literature students at spanish university have positive perceptions towards but limited understanding of online resources   a review of: roig-marín, a., & prieto, s. (2021). english literature students' perspectives on digital resources in a spanish university. journal of academic librarianship, 47(6). https://doi.org/10.1016/j.acalib.2021.102461   reviewed by: lisa shen business librarian & director of public services newton gresham library sam houston state university huntsville, texas, united states of america   email: lshen@shsu.edu    received: 31 may 2022                                                             accepted:  20 july 2022      2022 shen. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30182     abstract   objective – to assess students’ perception, use, and format preferences of library resources.   design – online survey questionnaire.   setting – a public university in spain.   subjects – 134 second-year, third-year, and fourth-year undergraduate english language and literature students.   methods – an anonymous survey was built using google forms and shared with eligible participants during march and april 2021. survey participation was voluntary, although students were encouraged to respond and were provided with class time to do so. nonetheless, due to the ongoing covid-19 pandemic at the time of this study, courses were taught in a hybrid (both in-person and online) format and class attendance was not mandatory. the survey consisted of six multiple choice and four open-ended questions, and answers were required for all 10 questions.   main results – respondents were mostly satisfied with the available resources in supporting their studies in english literature and culture, with the majority preferring to access resources online (51%) or through both online and print formats (14%). convenience was the most commonly cited reason for favoring online access, while improved processing and learning were mentioned by those preferring print. a majority of respondents also indicated they have used online resources from either their home university library (72%) or other libraries (55%). conversely, 29% of the respondents were unable to identify any specific electronic resources.   conclusion – study results indicate that spanish undergraduate students majoring in english literature generally have a positive perception of library resources in supporting their studies and prefer online access over print. however, many of these students may also have an incorrect or limited understanding of how to differentiate between library resources, general websites, web search engines, or computer programs.    commentary   even though much research has been conducted about students’ perception and use of academic library resources, this article provides a timely contribution to the existing literature by focusing on the unique perspective of non-anglophone students enrolled in an english literature program and could be of interest to practitioners serving similar student populations. study findings both affirm results from similar recent research on different student groups’ preferences for print and online resources (mizrachi & salaz, 2020; zell, 2020) and provide new insights into the challenges faced by non-native english-speaking students in identifying and using english literature resources.   an examination of the study using the evidence based librarianship (ebl) critical appraisal checklist (glynn, 2006) yielded an overall validity of 71%, which is below the accepted threshold of 75%. nevertheless, the authors can be commended for disclosing the full survey instrument, providing a succinct but through analysis of the results, acknowledging potential limitations of the findings, and identifying areas for further investigations. despite several limitations in the study’s population selection and data collection practices, its design and results each scored over 80% in sectional validity. the authors also provided details about the research methodology at a level that would enable replication.   on the other hand, readers would benefit from more details about the student population. the researchers provide sound rationale for establishing parameters for their intended population but neglect to provide the total number of eligible participants. without the actual population size and consequently the survey response rate, it is difficult to assess whether the study results sufficiently representative the study population. the authors do not explain whether eligible participants absent during the in-class survey distributions were invited through other methods. the researchers also do not explain how they controlled for duplication of responses since links to the anonymous survey were shared in multiple classes during the data collection period.   moreover, inconsistencies in the survey questionnaire suggest the instrument was unlikely to have been validated. terms including “online,” “electronic,” and “digital” were used interchangeably to describe resources throughout the survey, which could have confused the non-native english-speaking respondents. in addition, only the last three survey questions explicitly referred to “library resources,” which may have led respondents to conclude that “resources” in the other seven questions included both library and non-library resources. these types of terminology inconsistencies may have contributed to respondents naming google or kindle as resource examples for supporting their studies. finally, answer options for q6 assumes that respondents have either found a resource to be user-friendly or that the user was unable to access the resource. this dichotomy of options likely reflects the researchers’ assumptions about the user-friendliness of library resources and fails to account for the experiences of students who did not find these library resources to be user-friendly but were able to access them regardless.   because of these validity concerns with the study population, survey instrument, and data collection method, readers are advised to consider the representativeness of findings from this article with some reservation. nonetheless, this article highlighted the unique perspectives of non-anglophone academic library users of english literature resources and provided a possible foundation for future scholars interested in further investigating similar topics.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387–399. https://doi.org/10.1108/07378830610692154   mizrachi, d., & salaz, a. (2022). beyond the surveys: qualitative analysis from the academic reading format international study (arfis). college & research libraries, 81(5), 808–821. https://doi.org/10.5860/crl.81.5.808   roig-marín, a., & prieto, s. (2021). english literature students' perspectives on digital resources in a spanish university. journal of academic librarianship, 47(6), article 102461. https://doi.org/10.1016/j.acalib.2021.102461   zell, h. m. (2020). digital vs print resources at african university institutions: a discussion document. african research and documentation, 138, 3–16. https://doi.org/10.1017/s0305862x0002327x     microsoft word es 4075 evidence based library and information practice 2009, 4:1 29 evidence based library and information practice evidence summary secondary school teachers don’t have time to engage in the most important aspects of information literacy due to curricular pressures a review of: williams, d. and c. wavell. “secondary school teachers’ conceptions of student information literacy.” journal of librarianship and information science 39.4 (2007): 199-212. reviewed by: david herron scholary developer karolinska institutet, university library stockholm, sweden e-mail: david.herron@ki.se lotta haglund head of information and public relations karolinska institutet, university library stockholm, sweden e-mail: lotta.haglund@ki.se received: 02 september 2008 accepted: 19 january 2009 © 2009. herron and haglund. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. objective – to examine school teachers’ understanding of student information literacy and to look at the implications of the findings for developing information literacy in students. design – qualitative research design (group discussions; verbal and written reflections). setting – secondary schools in the united kingdom. subjects – secondary school teachers with various subject specializations. methods – the study initially involved 31 secondary school teachers; 24 from seven schools in scotland and seven from two schools in england. participant teachers were self-selecting. the study took a phenomenographic approach. teachers’ perceptions of their students’ abilities and their experiences and evidence based library and information practice 2009, 4:1 30 perceptions of their role in developing these abilities were examined through data collection in three stages. during the first stage, group discussions established the teacher’s initial conceptions of information literacy. participants also filled out an information task grid which, together with focus questions, ensured that discussions were kept in the realm of practice. during the second stage of data collection, teachers observed and reflected on their students’ use of information in classroom activities. this was accompanied by informal site visits during which 26 of the participants were introduced to information literacy frameworks and definitions in order to contextualize reflections and discussions. field notes were used to record these informal meetings. six participants provided written feedback on reflections. in the third and final stage, a summary of themes from the first two stages was sent to 23 of the participants prior to a second group discussion in order to try to stimulate further reflection. discussions were audio recorded and transcribed. the transcribed data were analyzed for the dimensions of variation of teachers’ conceptions of student information literacy and key elements associated with each category of description were determined. main results – the data revealed six main conceptions of student information literacy among the teachers: “finding information,” “linguistic understanding,” “making meaning,” “skills,” “critical awareness of sources,” and “independent learning.” there was no particular hierarchy in their conceptualization, although independent learning seemed to be the ultimate goal. teachers’ conceptions in the beginning of the study focused on the ability to find information, whereas after a period of reflection and further discussions, a broader and more complex understanding of information literacy appeared. table 1 simplifies some of the data from the table in the article and shows conceptualizations as well as some of the contextual elements. table 1 major conceptions of student information literacy among teachers categories of description learning priority sense of control finding information unclear high skills low some critical awareness of sources subjectdependent some linguistic understanding high low making meaning high low independent learning high low teachers felt that they had at least some control over the development of student ability to find information, develop skills, and critically appraise sources, even though these were not the highest learning priority outcomes. however, in the three areas of ability which the teachers regarded as of high learning priority (linguistic understanding, making meaning and independent learning), they felt that they had little control over the development of these abilities in the students. conclusion – the teachers’ conceptions of student information literacy overlapped with the ideas in existing models and frameworks for information literacy. however, some areas of information literacy were not addressed by the teachers, i.e., ethical issues in the use of information. evidence based library and information practice 2009, 4:1 31 in the categories of description of high learning priority, the main reason for the low sense of control by the teachers seemed to be curricular pressures – “our content is prescribed and time allocated doing each of these things is prescribed so we have got limited room for maneuvering” (206). the teachers tended to regard information literacy as process and skills-orientated, with little connection to learning. the authors note that other research also suggests that school librarians also have difficulty relating information finding to learning in specific subject disciplines. there is a clear gap here as the authors point out that “there is a danger that students will not understand the significance of learning with information for academic achievement or for independent learning” (209). the authors suggest that the way forward for further development of information literacy in the classroom and library is through dialogue and that “librarian-teacher collaborations needs to be founded on shared understanding of the complex interrelationship between information and learning” (209). commentary the authors make an outstanding job of putting their findings in the wider context of research on conceptions of information literacy as well as existing standards and frameworks. the validity of the study is good, although it is unclear just how representative the 31 secondary school teachers really were. the sample was large for a qualitative study and the data was collected over three stages, making the reliability convincing. the elaboration of implications for the integration of information literacy into the school curriculum is well thought-out, constructive, and should be widely discussed in the field. a good, interesting qualitative study and a pleasure to read. evidence based library and information practice commentary   dealing with misquotations constructively   dr. pablo millares-martin general practitioner whitehall surgery wortley beck health centre leeds, west yorkshire, uk email: pablo.martin@nhs.net   received: 17 aug. 2018                                                                accepted: 31 oct. 2018      2019 millares-martin. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29493     introduction   in a recent university lecture i attended, a specific article by sackett et al. (1976) was used to support the claim that the management of hypertension was influenced by the physician’s year of graduation from medical school. as a clinician, i considered whether my year of graduation could really affect my decision-making, and it triggered an interest in finding more details on the matter. after all, sackett is one of the pillars of evidence-based medicine (ebm). in the process of my investigation, i asked the library services to provide me the article quoted in the lecture. when i received and read it, i was surprised to find that it included no reference to the quoted statement. while i would not expect that type of error from a university tutor, it was clear that anybody could be affected by trusting a source and not checking it as diligently as any piece of information should be before presenting to others. furthermore, the fact that the misquotation referred to a respected figure such as sackett made it more likely to be accepted without question because of all the papers he has written on ebm.   in consequence of this particular misquotation, two broad questions came to mind: if trust in reputable sources is broken, what should be the responsible answer and what can be done to reduce the incidence of misquotations?   information is presented constantly, whether in journals or at conferences, and on many occasions it is not first hand, but referenced to previous publications. in the majority of cases the link between the fact and the reference is clear, and additional understanding of the subject can be achieved by reading the primary source. unfortunately, misquotations are common: de lacey, record, and wade (1985) reported a level of 24% among six medical journals, while more recently jergas and baethge (2015) estimated no less than 6.7% in their meta-analysis.   a major misquotation is “said to occur if the reference contradicted, failed to substantiate, or was irrelevant to the author’s assertion in the article” (luo, li, domingo malina, andersen, & panchbhavi, 2013). no author should accept a secondary reference without checking the original document, and if the information required is not confirmed, a process is suggested in this paper to deal with the matter constructively and to reduce the incidence of future misquotations.   citation-mapping review (cmr) technique   the process of finding a valid reference for a piece of information that has been misquoted and uncovering the extent of a particular misquotation is here called citation-mapping review (cmr), and the two steps involved are as follows.   the starting point is identifying an article that has been misquoted, such as the article by sackett et al. (1976) already mentioned. since authors should be aware of high levels of misquotations, primary sources should always be fully checked, never relying on secondary sources alone. if the information claimed to be in that article is not there, then there is a wrong to be righted. the approach when misquotations are found should be not just to disregard the alleged source and to look for an alternative source of information, but rather to make a statement about the fact that a misquotation has been found. it is an important fact that should be shared to prevent further misquotations to the same article.   the second step is to use a modified forward citation chaining approach. forward citation chaining is a research technique described by ellis (1989) and very commonly used nowadays (george et al., 2006) to identify articles or other materials that cite a specific source. however, in the current application its purpose has been adapted so that the purpose is not to find additional information on the subject, but rather to determine if those other articles are presenting the same misquotation by referring to a source that has not been checked adequately for the data being presented. additionally, the process can be used to check if the information itself is valid and whether a different source can be used to sustain it. this process could even be used to expose plagiarism or self-plagiarism, understood as failing to indicate when the wording in the article has been transcribed directly from another paper (mudrak, 2017). to do so, the analysis of papers found has to be carried out in chronological order so that it could indicate how the misinformation was disseminated originally.   authors using this process could not only be writing a much stronger manuscript but also sharing a more complete picture of known facts. the following describes the cmr process using a practical example, as the technique is better shown through tracing a real misquotation.   cmr in action   in the specific article by sackett et al. (1976) cited during the lecture i attended, i found that rather than stating that the management of hypertension was influenced by the physician’s year of graduation from medical school, sackett was discussing the different attitudes in hypertension management between a family physician and an industrial physician and the fact that patient’s characteristics (e.g., blood pressure, age, and some of the indexes of organ damage) explained no more than 20% of the variability encountered in prescribing. therefore, the article was the source of a major misquotation as defined by luo et al. (2013) when used to consider how year of graduation affects hypertension treatment, or any treatment if a generalization was aimed for.   to identify how far this misquotation had made its way into the scholarly literature, and whether it was just a single casual mistake by the person giving the lecture or a spread of misinformation appearing in multiple sources, a bibliometric database that allowed forward chaining was required. the citation search was based google scholar, which has coverage similar to scopus and the web of science (harzing & alakangas, 2016), but can be used without an access fee.   as of august 2018, google scholar found that the sackett paper has been cited on 54 occasions. the results differed between the access at the start of the project (50 citations) to the final check. there were several inaccuracies noted on scholar, with some broken links (unaccesible documents), or citations with the wrong order of authors, or the wrong title but the same journal and pages, and also duplication of entries.   within these i clearly identified a total of 42 unique articles (29 in english, 9 in chinese, 2 in spanish, 1 in italian, and 1 in german) and 3 books. i excluded a small number of articles from analysis because of access limitations or language barriers; i only examined articles in english and spanish as no translator was needed for those languages. this left a total of 30 citations to consider for this case study.   when analyzing the citation itself i noted two key findings. first, that all 30 papers were referring to an incorrect year of publication using 1977, rather than the correct year 1976. this included those articles of which sackett was a co-author (cooke & sackett, 1996; evans et al., 1986; rosenberg & sackett, 1996; sackett & rosenberg, 1995a, 1995b, 1995c), although in one of them the comment “abstracted” to the citation was included (evans et al., 1986), which is a more accurate description of the paper itself for its shortness. second, most papers cited the page number improperly: in the printed version the correct page is 648a not just 648 (thought it was cited correctly by three papers: devlin et al., 1992; bellamy et al., 1994; and werk et al., 2000). in all cases it was referred to as a single page, not suggesting the possibility of a longer version of the article, which in any case would be expected to focus on the same analysis of differences between practices of family physicians and industrial physicians, and not on a different matter like the influence of year of graduation on treatment choices.   after identifying an appropriate forward chain of citations, it was essential to compare what sackett had written originally with how it had been interpreted and modified. for the cmr process, it is important to understand the ways information has been turned into different accounts. for this case study, i conducted the analysis by year of publication, exploring the possibility of articles referencing intermediary articles, and then using the intermediary articles’ references as their own. in this respect, the forward chaining concept described by ellis (1989) is given an added layer: textual criticism, linking articles through time.   first the original article: sackett et al. (1976) stated that “family physicians decided to treat 49%, and industrial physicians 76%, of referred hypertensives” (p 648a), but they made no mention of year of graduation of clinicians, just of the fact that “the combined effect of all [the] clinical features explained only 20% of the variance in the decision to start treatment” without an explanation of what factors were more relevant.   six of the initial eight citations that occurred up to 21 years later were made by the original group of authors (cooke & sackett, 1996; evans et al., 1986; rosenberg & sackett, 1996; sackett & rosenberg, 1995a, 1995b, 1995c). among the information provided in these early articles, authors indicated that “previous work by our group documented important deficiencies in the care of hypertensive patients, including decreased likelihood of treatment by older physicians of patients with high blood pressure (sackett et al., 1977 [sic])” (evans et al., 1986, p. 501). evans et al. (1986) also proved no benefit from a mailed education program and indicated “previous work by our group documented […] a very strong negative correlation (—.55, p<.001) between the time since a physician' graduation and his or her knowledge of hypertension” (p. 501). the source for that statement was evans et al. (1984), which was confirmed.   devlin, bellamy, and bayliff (1992) clearly pointed out that “drug prescribing habits may also be, in part, a function of year of graduation from medical school” (p. 23), similar to bellamy, brooks, campbell, drane, and dupen (1994) describing “it has been demonstrated that year of graduation is one of the determinants of prescribing practice” (p.215 ), both wrongly quoting sacket et al. (1976). later in 1995 three separate articles stated the exact same words “in one study of actual clinical behaviour, the decision to start antihypertensive drugs was more closely linked to the number of years since medical school graduation in the doctor than to the severity of target organ damage in the patient (sackett et al., 1977)” (sackett & rosenberg, 1995c, p. 621; sackett & rosenberg, 1995a, p. 250; sackett & rosenberg, 1995c, p. 331).   cmr example positive findings   this assessment exercise was effective in providing an accurate resource for the statement misquoted at the “start point”, the focus on finding if it was true that the management of hypertension was influenced by the physician’s year of graduation from medical school. the initial citation from the 1976 sackett article did not provide the evidence to sustain that fact, and there was a need for clarity, for confirming the information from a different source, or for considering the possibility yet unconfirmed. the cmr process was successful in this particular example. it was possible to endorse the finding, based on a different article by members of sackett’s research team: there is a confirmed correlation between decline in medical knowledge and years since graduation from medical school (evans et al., 1984).   further misquotation from the example   the next step in the process could be to unmask further misquotations for the benefit of other readers looking at this subject, so that as an author it could be possible to determine and expose the degree of inadequate information in other papers, providing strength to the author’s own manuscript. further misquotations are shown below:   roche and richard (1991) stated “it is interesting to note in other health areas, e.g. treatment of hypertension, that doctors have been found to retain the style of practice learned during their early medical training, even when later evidence dictates that they should change” (pp. 1058-59) quoting sackett et al. (1976). roche, stubbs, sanson-fisher, and saunders (1997) referred to “medical practitioners become relatively fixed in their clinical style soon after qualifying” (p. 78), but then cited not only sackett et al. (1976) but two others, including byrne & long (1976) and maguire, fairbairn, and fletcher (1986a, 1986b); these later papers were found to be focused into the quality of training and its effects on diagnosis.   in a paper from grimes and learman (1996), the 1976 sackett citation was used but it did not refer to any text in the article, an indication not of misquotation but of possible editorial error. in the following years, authors continued referring to sackett’s 1976 article:   ·         brouwers and browman (1999) indicated “a strong negative relation between years since graduation and a provider’s knowledge of important advances in the detection, evaluation, and management of clinical problems, such as hypertension” (p. 1236). ·         daya (2001) expressed “the decision to initiate treatment is more closely linked to the number of years since the physician’s graduation than to the severity of the target-organ damage in the patient” (p. 382). ·         macdermid (2004) likewise indicated “clinical behaviors could be predicted more accurately from the number of years a clinician had been out of school than from the severity of the target organ damage in patients with hypertension” (p. 105). ·         samarkos (2006) similarly mentioned “the decision to start antihypertensive medication was more closely associated with the time from medical school graduation rather than the degree of target organ damage” (p.30 ).   other papers follow a different route, adding also evans et al. (1986) as a reference and using very similar words or paraphrasing:   ·         irvine and hunt (2001) wrote  “…the disparity between our increasing diagnostic skills and clinical judgement and our declining mastery of contemporary knowledge (evans et al., 1986) and clinical performance (sackett et al., 1977)” (p.2 ). ·         spanish author gutiérrez sougarret (2001) indicated that the disparity between our diagnostic skills and clinical judgment, which increases with experience and our up-to-date knowledge (citing evans et al., 1986) and clinical performance (citing sackett et al., 1977) that are in decline. ·         de leon (2005) indicated “the disparity between our diagnostic skills and clinical judgment, which increases with experience, and our up-to-date knowledge (evans et al., wrongly dated 1996 rather than 1986) and clinical performance (sackett et al., 1977) which decline” (p.73). ·         “the disparity between our diagnostic skills and clinical judgement, which increase with experience, and our up-to-date knowledge [(evans et al., 1986)] and clinical performance [(sackett et al., 1977)], which decline (zaidi et al., 2007, p. 556; koh et al., 2010, p 3) ·         virgilio et al. (2007) wrote “as their clinical judgment and diagnostic skills improve, their knowledge of current diseases and treatments often declines (evans et al., 1986; sackett et al., 1977)” (p. 295).   ramsey et al. (1991) found a negative correlation between knowledge and years since medical graduation, and the reference is used with the 1976 sackett misquotation and a different wording by:   ·         berg (1998) indicating that “year of graduation from residency is an important predictor of the drug formulary used by the average physician” (p. 217). ·         robinson et al. (2000) pointing out that “changes in medical knowledge over time are related to insufficient acquisition of up-to-date knowledge after graduation from medical school” (pp. 347-348). ·         stamp and kruzins (2001) stating that “knowledge and practice of up-to-date care was found to be negatively associated with the years since graduation” (p. 19 ).   werk et al. (2000) were more thorough, summarising “these findings are consistent with observations that awareness and adoption of expert recommendations deteriorates with time elapsed from medical school (sackett et al., 1977; evans et al., 1986; ramsey et al., 1991)” (p. 589), with one of the three references wrong.   this misdirection was also found in sutherland (2003, 2008a, 2008b), stating with the same words that “a study of hypertension showed that the main factor determining whether doctors decided to prescribe antihypertension drugs was not the severity of organ damage, as would be expected, but the number of years since the doctor graduated from medical school (sackett et al., 1977, 1998)” (sutherland 2008b, p. 112 ).   more recent references to sackett et. al can be found as well:   ·         ahmadi et al. (2013) wrote that “early exposure to evidence-based medicine has been shown to have significant effects on future clinical decision-making (sackett et al., 1977)” (p. e101). ·         spanish author echeverry-raad (2015) wrote that health training in colombia, as in other latitudes, is not providing this human talent with tools that allow them to self-actualize and learn long term, as they gain experience, a phenomenon that has been alerted for more than 40 years, citing sackett et al., 1977, as well and shin et al. 1993, as evidence.   in summary, i found many statements regarding how year of graduation of the clinician was linked to a decline in knowledge, but discovered the information was generally misquoted, although i did find two valid references. the most relevant collateral finding was that no article referred to data actually present in the original 1976 sackett et al. paper, on the perceived management differences between a family physician and an industrial physician, and more on the other quite relevant work that sackett has published over the years on evidence based medicine, for which he needs to be remembered as a pioneer.   discussion   new researchers are advised on using citations properly. no author should accept a secondary reference without checking the original document, and if the information required is not confirmed, a process like the one described here should be followed to provide clarity on the matter. when conducting a literature review, if a misquotation is found or suspected, it is possible to use cmr to find an alternative, more authoritative source. a positive finding cannot be guaranteed, and neither are we able to provide a probability of success. however,  since misquotations are common, it is important to have the confidence not only to attempt to match a statement to a valid reference but also to expose the problem, and to stop the spread of misinformation.   the cmr process is like a detective job, in that it is hard work but it can provide valuable results. going through articles chronologically can give some indication of how the misinformation progressed. more importantly, it can strengthen an article if it is used to clarify misquotations, presenting both the inappropriate and the adequate citations, and in a way alerting future readers of previous errors, potentially putting a stop to the spread of the misquotation.   misquotations should be recognised, so “the truth, the whole truth, and nothing but the truth" is published. we consider that focusing on this technique can help to further knowledge, to provide certainty regarding the primary sources of information, and to expand on them. ideally an conscientious author should write or present the information found in the example in a very different format, such as the following:   there is a correlation between decline in medical knowledge and years since graduation (evans et al., 1984; ramsey et al., 1991), information that has been misquoted to sackett et al. (1976) by as many as 30 papers between 1986 and 2015 (the latest three being de leon, 2005; ahmadi et al., 2013; and echeverry-raad, 2015).   strengths and limitations of the cmr process   this case study is based on an article with a small number of forward citations, which were easily manageable. it serves as an example to explain better how the cmr process takes place, and how the findings are linked to a particular misquotation. from this example, a clear picture emerges of how the original misquotation persisted through decades. using a single example in this case allows for a focus on the technique and a suggested presentation on findings, where as a considered analyses  of multiples misquotations could make this paper too long and distract from the process.   the cmr process takes a considerable amount of time, and doing it for several references while writing a single article could be considered to be too much work, but there may be no need to apply the analysis to all citations, but only to ones that when checked initially appear to be misquotations. one goal of this approach is that the number of misquotations should go down over time. although it was possible to find more adequate sources for the statement of interest, it has to be appreciated there will be examples of assertions misquoted in literature that cannot be as easily linked to alternative articles. in those circumstances it would require a considerable amount of time for a different benefit, stopping the spread of misinformation.   at present there are electronic tools to make citation easier and cross-reference citations, such as citethisforme (http://www.citethisforme.com), reciteworks (http://reciteworks.com), or citationmachine (http://www.citationmachine.net), but misquotations can still be found. identifying and correcting misquotations remains a slower process.   conclusions   the power of reviewing all cited sources thoroughly, through the cmr or a similar process, cannot be overestimated. this process is one of the tools a researcher can use to stop misquotations from propagating further. the right ethical conduct for a researcher should be to expose found misquotations so we enrich our knowledge. the cmr process is a different approach to looking at information misquoted in scientific journals. it is not only about finding the right link between a relevant piece of information and its source, but also assessing, for a particular paper, how often it is referred to on the contained information and how often it is misquoted. if a new culture of research included not only presenting valid sources but also commenting on misquotation findings, a better understanding of the issues presented would result.   two steps are suggested, a simple ‘start point’ which includes analysing the article found to be misquoted, to be certain nothing has been missed, and a follow-up ‘modified forward chaining’ to assess a number of articles that followed that misquotation to assess the extent of the problem. how thorough is the assessment is to be decided by the researchers, but sharing that piece of work in their papers would benefit us all.   references   ahmadi, n., dubois, l., mckenzie, m., brown, c. j., maclean, a. r., & mcleod, r. s. (2013). role of evidence-based reviews in surgery in teaching critical appraisal skills and in journal clubs. canadian journal of surgery, 56(4), e98-102. https://doi.org/10.1503/cjs.007512   bellamy, n., brooks, p. m., campbell, j., drane, d., & dupen, f. (1994). an australian survey of current prescribing practices of methotrexate in rheumatoid arthritis. australian and new zealand journal of medicine, 24(2), 214-216. https://doi.org/10.1111/j.1445-5994.1994.tb00559.x   berg, a. o. (1998). dimensions of evidence. journal of the american board of family medicine, 11(3), 216-223. https://doi.org/10.3122/15572625-11-3-216   brouwers, m. c., & browman, g. p. (1999). development of clinical practice guidelines: surgical perspective. world journal of surgery, 23(12), 1236-1241. https://doi.org/10.1007/s002689900655   byrne, p.s., & long, b.e.l. (eds.). (1976). doctors talking to patients. london, uk: h. m. stationery office.   cooke, i. e., & sackett, d. l. (1996). 1 evidence-based obstetrics and gynaecology. baillière's clinical obstetrics and gynaecology, 10(4), 535-549. https://doi.org/10.1016/s0950-3552(96)80003-4   cozzolino, a., feola, t., simonelli, i., puliani, g., pozza, c., giannetta, e., ... isidori, a. m. (2018). somatostatin analogs and glucose metabolism in acromegaly: a meta-analysis of prospective interventional studies. the journal of clinical endocrinology & metabolism, 103(6), 2089-2099. https://doi.org/10.1210/jc.2017-02566   daya, s. (2001). evidence-based medicine. in d. k. gardner, a. weissman, c. m. howles, & z. shoham (eds.). textbook of assisted reproductive techniques: laboratory and clinical perspectives (p. 381-392). london, uk: martin dunitz ltd.   de lacey, g., record, c., & wade, j. (1985). how accurate are quotations and references in medical journals? british medical journal, 291, 884-886. https://doi.org/10.1136/bmj.291.6499.884   de leon, a. s. (2005). evidence-based medicine: what it is, what it is not. philippine journal of ophthalmology, 30(2) 73-74. http://www.paojournal.com/   devlin, j. w., bellamy, n., & bayliff, c. d. (1992). observations and effects of educational consults on allopurinol prescribing. the canadian journal of hospital pharmacy, 45(1), 21-27. https://www.cjhp-online.ca/   echeverry-raad, j. (2015). aspectos generales y académicos de los aspirantes y los admitidos a la especialidad en pediatría de la universidad nacional de colombia [general and academic aspects of the candidates and those admitted to the pediatrics specialty in the national university of colombia]. revista de la facultad de medicina, 63(1), 33-45. https://doi.org/10.15446/revfacmed.v63n1.45953   ellis, d. (1989). a behavioural approach to information retrieval system design. journal of documentation, 45(3) 171-212. https://doi.org/10.1108/eb026843   evans, c. e., haynes, r. b., birkett, n. j., gilbert, j. r., taylor, d. w., sackett, d.l., … hewson, s. a. (1986). does a mailed continuing education program improve physician performance? results of a randomized trial in antihypertensive care. jama, 255(4), 501-504. https://doi.org/10.1001/jama.1986.03370040075027   evans, c. e., haynes, r. b., gilbert, j. r., taylor, d. w., sackett, d. l., & johnston, m. (1984). educational package on hypertension for primary care physicians. canadian medical association journal, 130(6), 719-722. http://www.cmaj.ca/content/130/6/719.short   george, c., bright, a., hurlbert, t., linke, e.c., st clair, g., & stein, j. (2006). scholarly use of information: graduate students' information seeking behaviour. information research, 11(4). http://www.informationr.net/ir/11-4/paper272.html   grimes, d. a., & learman, l. a. (1996). theory into practice: within a department. baillière's clinical obstetrics and gynaecology, 10(4), 697-714. https://doi.org/10.1016/s0950-3552(96)80013-7   gutiérrez sougarret, b (2001). anestesia basada en evidencia: medicina basada en evidencia [evidence-based anesthesia: evidence-based medicine]. revista mexicana de anestesiología, 24(4) 199-200. retrieved october 30, 2018, from http://www.medigraphic.com/pdfs/rma/cma-2001/cma014d.pdf   harzing, a. w., & alakangas, s. (2016). google scholar, scopus and the web of science: a longitudinal and cross-disciplinary comparison. scientometrics, 106(2) 787-804. https://doi.org/10.1007/s11192-015-1798-9   irvine, e. j., & hunt, r. h. (eds). (2001). evidence-based gastroenterology. b. c. decker inc.: hamilton, on.   jergas, h., & baethge, c. (2015). quotation accuracy in medical journal articles—a systematic review and meta-analysis. peerj, 3, e1364. https://doi.org/10.7717/peerj.1364   koh k., gill j. s., & pillai s. k. (2010). evidence-based clinical practice a critical analysis of its justification and limitations. malaysian journal of psychiatry, 19(1) 1-6. http://www.mjpsychiatry.org/index.php/mjp/article/view/83   kundoor, v., & ahmed, k. k. m. (2010). uncovering negative results: introducing an open access journal “journal of pharmaceutical negative results.” journal of young pharmacists, 2(4), 339-341. https://doi.org/10.4103/0975-1483.71618   logan a. g., & haynes, r. b. (1986). determinants of physicians' competence in the management of hypertension. journal of hypertension, 4(5), s367-369.   luo, m., li, c. c., domingo molina, i. v., andersen, c. r., & panchbhavi, v. k. (2013). accuracy of citation and quotation in foot and ankle surgery journals. foot & ankle international, 34(7), 949-955. https://doi.org/10.1177/1071100713475354   macdermid, j. c. (2004). an introduction to evidence-based practice for hand therapists. journal of hand therapy, 17(2), 105-117. https://doi.org/10.1197/j.jht.2004.02.001   maguire, p., fairbairn, s., & fletcher, c. (1986a). consultation skills of young doctors: i—benefits of feedback training in interviewing as students persist. british medical journal, 292, 1573-1576. https://doi.org/10.1136/bmj.292.6535.1573   maguire, p., fairbairn, s., & fletcher, c. (1986b). consultation skills of young doctors: ii—most young doctors are bad at giving information. british medical journal, 292, 1576-1578. https://doi.org/10.1136/bmj.292.6535.1576   mudrak, b. (2017). self-plagiarism: how to define it and why you should avoid it. retrieved october 30, 2018, from http://www.aje.com/en/arc/self-plagiarism-how-to-define-it-and-why-to-avoid-it/   ramsey p. g., carline d., lnui t. s., larson e. b., logerfo j. p., norcini j. j., & wenrich, m. d. (1991). changes over time in the knowledge base of practising internists. jama, 266(8) 1103-1107. https://doi.org/10.1001/jama.1991.03470080073032   robinson, e., bachrach, l. k., & katzman, d.k. (2000). use of hormone replacement therapy to reduce the risk of osteopenia in adolescent girls with anorexia nervosa. journal of adolescent health, 26(5), 343-348. https://doi.org/10.1016/s1054-139x(99)00121-4   roche, a. m., & richard, g.p. (1991). doctors' willingness to intervene in patients' drug and alcohol problems. social science & medicine, 33(9), 1053-1061. https://doi.org/10.1016/0277-9536(91)90010-a   roche, a. m., stubbs, j. m., sanson-fisher, r. w., & saunders, j. b. (1997). a controlled trial of educational strategies to teach medical students brief intervention skills for alcohol problems. preventive medicine, 26(1), 78-85. https://doi.org/10.1006/pmed.1996.9990   rosenberg, w. m., & sackett, d. l. (1996). on the need for evidence-based medicine. therapie, 51(3), 212-217.   sackett, d. l., haynes, r. b., taylor, d. w., gibson, e. s., roberts, r. s., & johnson, a. l. (1976). clinical determinants of decision to treat primary hypertension. clinical research, 24(5), a648.   sackett, d. l., & rosenberg, w. m. c. (1995a). on the need for evidence-based medicine. health economics, 4, 249-254. https://doi.org/10.1002/hec.4730040401   sackett, d. l., & rosenberg, w. m. c. (1995b). on the need for evidence-based medicine. journal of public health. 17(3), 330-334. https://doi.org/10.1093/oxfordjournals.pubmed.a043127   sackett, d.l., & rosenberg, w. m. c. (1995c). the need for evidence-based medicine. journal of the royal society of medicine, 88(11), 620-624. https://journals.sagepub.com/home/jrs   samarkos, m. g. (2006). the philosophy of evidence-based medicine. hospital chronicles, 1(1), 27-35. retrieved from http://hospitalchronicles.gr/index.php/hchr/article/view/4   shin, j. h., haynes, r. b., & johnston, m. e. (1993). effect of problem-based, self-directed undergraduate education on life-long learning. cmaj: canadian medical association journal, 148(6), 969-976. http://www.cmaj.ca/content/148/6/969.short   stamp, g. e., & kruzins, g. s. (2001). a survey of midwives who participated in a randomised trial of perineal massage in labour. australian journal of midwifery, 14(1) 15-21. https://doi.org/10.1016/s1445-4386(01)80030-6   sutherland, w. (2003). evidence-based conservation. conservation in practice, 4(3), 39-42. https://doi.org/10.1111/j.1526-4629.2003.tb00068.x   sutherland, w. (2008a, july 29). evidence-based conservation. conservation. retreived from http://conservationmagazine.org/2008/07/evidence-based-conservation/   sutherland, w. j. (ed.) (2008b). the conservation handbook: research, management and policy. oxford, uk: blackwell science. https://doi.org/10.1002/9780470999356   tripathy, d. (2015). negative is positive: a plea to publish all studies regardless of outcome. american journal of hematology/oncology, 11(4). https://www.gotoper.com/publications/ajho/2015/2015apr/negative-is-positive-a-plea-to-publish-all-studies-regardless-of-outcome   virgilio, r. f., chiapa, a. l., & palmarozzi, e. a. (2007). evidence-based medicine, part 1. an introduction to creating an answerable question and searching the evidence. journal of the american osteopathic association, 107(8), 295-297. https://jaoa.org/article.aspx?articleid=2093462   werk, l. n., steinbach, s., adams, w. g., & bauchner, h. (2000). beliefs about diagnosing asthma in young children. pediatrics, 105(3), 585-590. https://doi.org/10.1542/peds.105.3.585   zaidi, z., hashim, j., iqbal, m., & quadri, k. m. (2007). paving the way for evidence-based medicine in pakistan. journal of pakistan medical association, 57(11), 556-560. https://jpma.org.pk/article-details/1244   evidence based library and information practice 2012, 7.2 96 evidence based library and information practice ebl 101 research methods: interviews virginia wilson liaison librarian, murray library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca received: 8 may 2012 accepted: 20 may 2012 2012 wilson. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. last time we looked at focus groups as a qualitative research method. this time the focus shifts to interviews. sitting down face to face with a research participant and asking probing and insightful questions can be daunting—especially when feeling the pressure to be probing and insightful! however, if your research question dictates that interviews are the best method of data gathering, you need to dive in and get started. how do you know if interviewing is a good fit for your research? beck and manuel (2008) suggest that if you want to understand or explore finely shaded human issues, if your question seems best answered in prose rather than with numbers, and if you want to explore a trend or an experience looking for themes, then the interview is a good choice for data gathering (p. 82). there are a few types of interviewing styles to choose from: structured interviews, semistructured interviews, and unstructured interviews. as may be evident, structured interviews involve asking the same set of questions to each research participant. there is no room to move beyond the set of questions. this style of interview is often used in conducting surveys. in qualitative research, the structured interview is quite limiting. semi-structured interviews allow for more flexibility. they involve having a set of guiding questions that will keep the interview on track. however, the researcher can follow topics of interest during the interview without having to adhere to a structured set of questions. for unstructured interviews, the researcher will have an idea of the avenues he/she wants to explore, but the interview is more like a conversation—flexible and unrestricted. because the conversation can, and is expected, to go anywhere, comparing data between interviews becomes more difficult. interviews can be conducted in person, over the telephone, or electronically using a program such as skype. the advantage to being face to face is the ability to see facial expressions and body language. these can be mailto:virginia.wilson@usask.ca evidence based library and information practice 2012, 7.2 97 jotted down as notes as the interview progresses and may be useful in the data analysis portion of the research. conversely, interview participants may be shy or uncomfortable with a face to face conversation or unfamiliar with the technology. using the telephone or skype is cost effective, as no travel is involved, and may provide a certain level of comfort to participants. the factors must be weighed carefully in order to choose the best method of conducting the interviews. beck and manuel (2008) break down interviewing as a research method into a series of steps: 1. identify participants. once participants are known to you, make sure you build enough time into the project to schedule the interviews. meeting everyone’s needs can be time consuming. 2. decide on the type of interview (in person, telephone, etc.). 3. decide on the facilities in which to conduct the interviews. this may be out of your control if you decide to meet the participants on their own turf. if you do travel to meet participants, be mindful of the potential risk involved. ensure someone knows where you are and consider developing a fieldwork policy if your organization does not have one. 4. check and test your equipment. the last thing you need is a voice recorder with dead batteries. 5. design an interview schedule that is well-paced and that includes topics to explore. 6. obtain formal signed consent. 7. pace the interview, giving adequate time for each question and response. 8. follow proper protocol which is usually determined prior to beginning the research at the ethics approval stage (adapted from beck & manuel (2008), pp. 95-96). one important thing to consider before undertaking interviews as a data gathering method is the different types of people you may encounter. these types can include dominant talkers, long-winded participants, the “expert” type, argumentative types, and the shy person. the interviewer needs to be prepared with techniques and tactics to move the interview along, quell a potential argument, or draw out a reticent participant. there are many resources to help you get started in using interviews as a research method. here are just a few: dilley, p. (2000). conducting successful interviews: tips for intrepid research. theory into practice, 39(3), 131-137. retrieved 21 may 2012 from http://www.jstor.org/stable/i264807 kvale, s., & brinkmann, s. (2009). interviews: learning the craft of qualitative research interviewing (2nd ed.). los angeles, ca: sage publications. seidman, i. (2006). interviewing as qualitative research: a guide for researchers in education and the social sciences (3rd ed.). new york, ny: teachers college press. and here are some examples of research that uses interviewing as a data gathering method: duncan, v., & holtslander, l. (2012). utilizing grounded theory to explore the information seeking behavior of senior nursing students. journal of the medical library association,100(1), 20-27. doi:10.3163/1536-5050.100.1.005 valentine, b. (2001). the legitimate effort in research papers: student commitment versus faculty expectations. journal of academic librarianship, 27(2), 107-115. doi: 10.1016/s0099-1333(00)00182-8 major, j.a. (1993). mature librarians and the university faculty: factors contributing to librarians’ acceptance as colleagues. college and research libraries, 54(6),463469. retrieved 21 may 2012 from eric database (ej473071). http://www.jstor.org/stable/i264807 evidence based library and information practice 2012, 7.2 98 there is not enough room in this column to delve into data analysis, and basically that depends on which theoretical approach you are taking, e.g., grounded theory. if you think that interviewing might be the way to go for your research, look to the literature and discuss your ideas with experienced colleagues. as well, if you are just beginning a research project using the interview method, you are not alone. i am, too! next time around, the topic will be bibliometrics. references beck, s. e., & manuel, k. (2008). practical research methods for librarians and information professionals. new york, ny: neal-schuman. evidence based library and information practice using evidence in practice   but what about us? developing an inclusive approach to library insight   selena killick associate director, library services the open university milton keynes, buckinghamshire, united kingdom email: selena.killick@open.ac.uk   received: 3 jan. 2020                                                                  accepted: 6 aug. 2020      2020 killick. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29704   setting   the open university (ou) is the u.k.’s largest academic institution dedicated to distance learning, with over 170,000 students. established in 1969, we were the first online university waiting for the internet to be invented. the library was established when the university formed to provide print collections for campus-based academic staff. as electronic publishing grew, we commenced offering services to students. today 100% of our journal collections and around 75% of our books are electronic. the online library attracts half a million unique users per annum. each year, 91% of our students study a module with embedded digital and information literacy skills or library materials, and we know that those who use library resources and attend library tutorials get better results (killick et al., 2018; nurse et al., 2018).   the library has a strong culture of assessment, which ensures that service improvements are underpinned by evidence. this is achieved through our student library research panel, a representative community of 500 students who work with us as co-creators (dick & killick, 2016). although there is a natural self-selection bias, both users and nonusers of the library agree to participate. this inclusive approach ensures that our strategies meet the needs of the whole student community.   we gather insight through a range of techniques, including interviews, surveys, focus groups, usability tests, love letters and breakup letters, touchstone tours, card sorting (for information architecture projects), and directed storytelling—all at a distance (stiles, 2017). the students have partnered with the library on a variety of projects, including the procurement of a library management system and discovery tools (dick & killick, 2016).   problem   as the library’s assessment culture has grown, one concern about our approach has continued to surface: the student library research panel membership only includes students. they are the largest community we serve and, due to the distance learning model, difficult to gain feedback from. while we do conduct some user experience research with our predominantly campus-based research students (jenkins, 2017), we do not routinely work with other key stakeholders. insight from academic staff, responsible for curriculum creation and predominantly based in milton keynes, is typically anecdotal feedback. insight from our 4,000 associate lecturers, who are located across the u.k. and are responsible for teaching and supporting our students on a part-time basis, is even scarcer. the library was planning service improvements based on student insight alone and not seeing a fuller picture.   evidence   to gain a wider understanding of the needs of our whole community, we embarked upon the library needs project. to ensure we captured the views of the whole community, the project sought insight from staff for the first time, as well as students.   initially, the project team analyzed the library insight we already had. this included a review of the insight from the panel members and campus-based research students, along with feedback captured through institutional surveys (for example the national student survey qualitative data).   after securing appropriate institutional ethical approvals, the team embarked upon primary research with members of the ou community, specifically:   ·        academic staff based on campus and in other parts of the u.k. ·        associate lecturers ·        research students on campus and in other parts of the u.k. ·        students (undergraduate and graduate—known as “postgraduate” in the u.k.)   using a snowball technique to recruit participants, people who had previously worked with the library were asked to assist in recruiting people with whom we may not yet have spoken. a total of 33 people volunteered to participate in the research from all parts of the u.k., academic faculties, and community groups (academics, associate lecturers, research students, and students).   using a directed storytelling conversational approach, we gathered insights into the participant’s needs, perceptions, and expectations of the library. prior to meeting, the team developed a series of light-touch questions for the different communities, with follow-up prompts if the conversation required it. for example, an academic staff member’s questions included the following:   ·        can you describe what you think of when i say library services? ·        can you tell me about an experience/your last experience of using a service provided by the ou library (researcher note: if no experience of using the ou library, any other library)? ·        can you describe the last time you needed to find something out for your research or you needed to add something to a module you were preparing?  ·        [when discussing future needs]: o   how has your academic practice been changing over the last few years? o   how do you think it will change over the next few years? o   can you think of anything you would want from library services to help you meet these changing needs?   the focus was on allowing the participant to have a conversation with the researcher and to lead the discussion; the questions were used as prompts rather than a script. we held face-to-face conversations for users based at the milton keynes campus and met over the telephone with those who work and study remotely.   all interviews were recorded and transcribed before conducting conversation analysis and thematic analysis. the team used the findings in an immersive workshop with the extended leadership team (elt), where the insight was combined with their professional expertise and the university’s strategic aims, to develop the departmental operational plan and for the forthcoming year.   while the primary research gained from the students corresponded with the research previously undertaken with the student panel, wider insights from the other community groups were surprising. positively, we are seen as a prized resource that is central to the work of the university. participants spoke about the value of the library to their work, research, and study. in line with our continual-improvement culture, however, a number of opportunities to improve were identified by the elt.   one of the key areas for improvement is the physical library. since commencing services for students, the strategic focus has been on the online environment. coupled with increasing financial pressures, the physical support services have been reduced. this has not impacted the distance learning students, but it has impacted our campus-based academic staff and research students.   implementation   in response, the library has partnered with the estates department to develop a new strategy for the physical library. using a mixed-methods approach, we have gathered further detailed insight to inform this, including observational studies of the building, hourly headcounts for each floor, and exit interviews.   as physical services have been declining for several years, so, too, have visits to the building. nonusers of the current building were included in the research scope to overcome this. we installed a self-service, anonymous “postcard to the library” station in our catering outlets. the postcards prompted feedback by opening with statements such as “i like to use the library building because … ” or “i don’t like to use the library building because … ”, eliciting rich qualitative feedback from both users and nonusers of the library.   mini guerrilla interviews at various locations across the campus were also conducted. to get the participants thinking more widely than a traditional library environment, the team asked broader questions around existing campus spaces and any gaps in current provision overall. an online survey replicating the mini guerrilla interview questions was also employed. this was sent to staff members who regularly visit the milton keynes campus but are based in our offices across the u.k., a key stakeholder group who had the potential to be missed from our on-campus sampling (stiles & killick, 2019).   the insight was fascinating, powerful, and, at times, heartbreaking to read. respondents have a strong emotional connection with the physical library and what it represents as a symbol for the university. some respondents highlighted the strengths of the physical library, such as the calm, quiet, and light working space, which we are keen to maintain. others, however, described it as soulless, empty, and unused. this provided a powerful call to action; we knew we really must do better.   following the same approach as previously used, the insight was summarized and presented to the elt.   outcome   the findings, combined with our professional expertise and institutional strategic aims, have resulted in the new physical library strategy that is now being implemented. through partnering with the estates department, we were able to use the evidence to inform a multi-year plan for building redevelopments. the first phase has been to provide space to support knowledge exchange through collaborative working and events. in order to preserve the valued quiet working environment, workspaces were zoned, and quiet study spaces were moved from the open ground floor atrium to the floor above, which provides a naturally quieter environment. the ground floor atrium has been renovated into a flexible workspace, allowing large exhibitions and showcase events to be hosted in the library, combined with collaborative working at other times. feedback has been exceptionally positive from the ou community, and more phased developments are planned in the future.   the key learning from this process has been the importance of capturing the views of our whole community, including the users and nonusers, from all user groups. we are now extending the library student research panel to become the library research panel. we want to expand the membership to ensure we continue to gather insight from a representative sample of our whole community. this will establish our collaborative working culture with the community we support, ensuring we continue to develop the library to meet their ever-changing needs and expectations.   reflection   using evidence is an important part of our organizational culture, enabling us to develop our services in line with user needs. the key strength of the library needs project was the directed storytelling methodology, allowing us to identify several strategic improvement projects (one of which has been the building renovation), which we would never have discovered using a closed research technique. recruitment of participants through the snowball technique was effective given the time constraints of this research; however, this led to a self-selection bias of library users over nonusers.   our key recommendation is to design your insight collection methods to include nonusers from the outset. purposely gathering insight from people outside of the library building enabled us to understand the reasons for non-usage and to develop a strategy to overcome this. the future expansion of the library research panel is designed to allow more insight to be gained from our nonusers to ensure future strategies are more inclusive of our whole community.   references   dick, s., & killick, s. (2016). delighting our customers: building services collaboratively with learners at a distance. in s. baughman, s. hiller, k. monroe, & a. pappalardo (eds.), proceedings of the 2016 library assessment conference: building effective, sustainable, practical assessment, october 31–november 2, 2016, arlington, va (pp. 543–548). association of research libraries. http://oro.open.ac.uk/id/eprint/57807   jenkins, d. (2017). how research students at the open university conduct research: insights from cognitive mapping. sconul focus, (69), 18–22. http://oro.open.ac.uk/id/eprint/59313   killick, s., nurse, r., & clough, h. (2018). the continuing adventures of library learning analytics: exploring the relationship between library skills training and student success. in s. baughman, s. hiller, k. monroe, & a. pappalardo (eds.), proceedings of the 2018 library assessment conference: building effective, sustainable, practical assessment, december 5–7, 2018, houston, tx (pp. 188–199). association of research libraries. https://doi.org/10.29242/lac.2018.15   nurse, r., baker, k., & gambles, a. (2018). library resources, student success and the distance-learning university. information and learning sciences, 119(1/2), 77–86. https://doi.org/10.1108/ils-03-2017-0022   stiles, k. (2017). ux research with distance learners. in a. priestner (ed.), user experience in libraries yearbook 2017: stories, techniques, insights (pp. 127–131). ux in libraries. http://oro.open.ac.uk/id/eprint/52664   stiles, k., & killick, s. (2019, june 18–19). completely shelfless: reinventing a physical library for an online community [conference session]. fifth annual international user experience in libraries conference (uxlibsv), egham, surrey, united kingdom. http://oro.open.ac.uk/62190/   evidence summary   parents of young children select picture books based on information not found in bibliographic records   a review of: švab, k. & žumer, m. (2015). the value of a library catalog for selecting children's picture books. cataloging & classification quarterly, 53(7), 717-737. doi: 10.1080/01639374.2015.1044059   reviewed by: ruby warren user experience librarian university of manitoba libraries winnipeg, manitoba, canada email: ruby.warren@umanitoba.ca   received: 20 may 2016  accepted: 22 july 2016      2016 warren. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine how parents select picture books for their children, and which bibliographic data are important when selecting a specific version of a title with multiple interpretations.   design – qualitative, with interviews and task-based controlled observational studies.   setting – a public library in slovenia.   subjects – 36 parents of children between one and 6 years of age.   methods – the researchers recruited parents via convenience sampling in non-library, family-oriented locations (parks, playgrounds, beaches, and others). participants were all interviewed regarding their methods of picture book selection and their use of library catalogues. participants were then given six print bibliographic records for copies of cinderella, available in libraries, and asked to select a book for their child based solely on these records. they were then presented with their selection and interviewed regarding their satisfaction with the book selected and their decision-making process. finally, the researchers presented participants with all six physical copies of the book that had been represented by bibliographic records, and asked participants to select one of the books for their child. the researchers then interviewed participants regarding what information about the physical books should be included in records to assist in their decision-making.   main results – interviews indicated that the majority of participants did not use the library catalogue to select books for their children, and did not expect librarian or bookseller assistance. many participants expressed browsing behaviours as the primary method of obtaining new picture books, and the strongest criteria for picture book selection among participants were subjective judgements regarding illustrations and content. however, when asked to use just bibliographic records to select a version of cinderella, most participants selected a title using the author field and year of publication. 67% of participants were then dissatisfied with their selection due to factors such as illustration type, font size, and length or complexity of text. when choosing from all six physical copies, most participants disregarded condition issues and selected the oldest edition, favouring its colourful illustrations and textual length.   conclusions – the authors concluded that illustrations and book content were more important than other factors, including physical condition of the book, and that existing library catalogues were inadequate for picture book selection. they suggested that library catalogues should include further information about picture books, such as cover images, sample pages, book condition, and information about the type of text (whether it is the original, abridged, or an adaptation). they supported this by explaining that participants used the bibliographic fields already available (author, year of publication) to try and guess at what they actually considered important (the aforementioned suggested fields). in addition, they believed that their study indicated that users require a transparent and systematic way to review and compare versions of a given text. finally, the authors recommended further study using enriched bibliographic records and additional data collection methods, such as focus groups and questionnaires. the authors have several further studies in this area planned.   commentary   available literature on picture book selection is quite extensive, but bibliographic record enrichment to aid picture book selection has not been adequately researched thus far. while the available literature indicates how parents may choose picture books for their children in general (saracho & spodek, 2010), and that “the role of accompanying parents [is] pivotal” (larkin-lieffers, 2012), it does not address the ways that catalogue records are failing to support parental selection behaviours. the authors suppose that libraries need to redesign their bibliographic systems to meet user needs, and this fuels their research questions: what user needs are bibliographic data failing to meet, and how could bibliographic records be better? the answer to these questions could have significant practical implications for bibliographic record structure and electronic service design in public libraries.   the qualitative methods used by this study are entirely appropriate for the stated research questions, which require answers of insight into parental thought processes and behaviour. the authors explicitly and satisfactorily justified the need for a combination of observational and interview methods in order to compare self-reported participant intentions with actual behaviour.   this study scored 7 out of a possible 10 points using the critical appraisal skills programme qualitative research checklist (2013). areas of weakness in the study’s methodology included a declaration of structured interviews but an absence of an interview instrument appended to the article; select examples of the “type” of questions asked are given, but there was no explicit interview structure explained. in addition, the form of the data (whether participant responses were digitally recorded, transcribed, or simply noted by the researchers) was unclear, and there was no explanation of the researchers’ methods of analyzing the available data. this, and the reservations that always accompany a convenience sample, could limit direct applicability of these findings to broader populations; however, the questions and design ideas raised by this research merit further investigation.   ultimately, this study is easily replicable and could be useful for informing practice, with some caveats. further research and testing is needed to determine exactly how to design electronic catalogue services and bibliographic records to better suit book selection needs. while this particular study is most relevant to children’s librarianship and public bibliographic design, further research regarding book selection and bibliographic records could have a substantial impact on cataloguing and electronic service design in any library dealing with visually unique or alternative format materials. testing the importance of various criteria in the selection of these materials could inform innovations in bibliographic record enhancement, or in the provision of additional digital browsing or selection tools.   references   critical appraisal skills programme. (2013). qualitative research checklist. retrieved from http://www.casp-uk.net/wp-content/uploads/2011/11/casp-qualitative-research-checklist-31.05.13.pdf   larkin-lieffers, p. (2001). informational picture books in the library: do young children find them? public library quarterly, 20(3), 3-28. doi:10.1300/j118v20n03_02   saracho, o. & spodek, b. (2010). families’ selection of children’s literature books. early childhood education journal, 37(5), 401-409. doi: 10.1007/s10643-009-0365-5     microsoft word comm 4343 evidence based library and information practice, 2008: 3.4 76 evidence based library and information practice commentary on skepticism about eblip: friendly or otherwise! andrew booth director of information and reader in evidence based information practice school of health and related research (scharr) university of sheffield sheffield, great britain, united kingdom e-mail: a.booth@sheffield.ac.uk received: 14 november 2008 accepted: 27 november 2008 © 2008 booth. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. i congratulate the eblip editorial team on publishing the provocative piece, “friendly skepticism about evidence based library and information practice” (banks), in the previous issue of this journal. it is surely a healthy sign for the movement when one of its foremost organs opens its pages to such “friendly skepticism.” indeed, only one year ago in this very journal, anne brice and i bemoaned the absence of such reflective soul-searching: “. . . there is little evidence of genuine innovation and debate” (booth and brice 94). does such skepticism about evidence based library and information practice (eblip) pose any threat to the paradigm? let us consider some of the criticisms raised by banks and others. first of all, advocates of eblip cannot afford to be “data-driven.” the paucity of evidence across all types of questions and all the major library and information domains requires a bestavailable evidence approach. indeed, we cannot even afford to be problem-led. as busy practitioners we seek the solution to practical library issues, and we do not have the luxury of spending an inordinate amount of time focusing on the perfect question. our optimal response is to be neither problem-led nor data-driven, but to be “solution-focused” (booth and brice 103). in truth, many of us have been drawn to eblip because it provides a realistic and practicable antidote to decision-making based on hunches or gut instincts. in proposing objections to eblip it is important that arguments are advanced, not simply rehearsed. in their seminal article about common criticisms of evidence based medicine straus and mcalister make a constructive start by separating the limitations of medicine per se from the limitations of evidence based medicine. arguably it is equally appropriate to attempt to separate the limitations of the library profession in general from the specific limitations of evidence based library and information practice. for example, the fact that librarianship is not a evidence based library and information practice, 2008: 3.4 77 research-minded profession is a characteristic of the problem, not a limitation of the solution. furthermore, inordinate attention to the preferences of users, as described in banks’ commentary, may indeed be symptomatic of our inability as a profession to initiate a wider dialogue incorporating a perspective from the research literature and securing recognition of the value of the viewpoint afforded by a professional librarian. certainly it is our professional duty to resist any suggestions that user preferences should be allowed to completely and singularly determine whether or not a service is provided. in the second stage of their rebuttal, straus and mcalister turn their attention to the more common criticisms of evidence based practice in librarianship: • it is too time-consuming. • it privileges research over other types of evidence. • it makes the ideal the enemy of the good. these criticisms are based on a misunderstanding of evidence based practice or, at the very least, an incomplete or outdated understanding of this decision-making model. as i commented in a recent article, we should acknowledge the contribution of the widest variety of research investigations to our day-today practice. fundamentally as a profession we are pragmatic principally because we have to be so. (booth, “using evidence” 49) to take but one example, the alleged “privileging” of quantitative over qualitative research is currently being widely redressed and will do much to put the all-important perspectives derived from service users on a more rigorous footing. above all, we cannot allow criticisms to imply that evaluating a service is a greater waste of time than continuing to deliver an entirely useless non-evidence based service. evaluation does not require a lengthy process. my favourite library research study involves dropping books through different types of book drops and evaluating them in terms of damage! this is far from rocket science. ask not whether we can afford to undertake eblip; ask instead, “can we afford to not pursue it?” this was the topic of the discussion between scott plutchak and myself at the 4th evidence based library and information practice conference (chapel hill-durham, north carolina, 2007). the question as to whether eblip should be pragmatic, philosophical, or both continues to loom over the movement and resurfaces in banks’ article. perhaps we are guilty of having manufactured a climate where, in the absence of genuine, well-founded debate, adherents of eblip gravitate naturally to controversy – substituting heat for the absence of light? if such is the case, the real extent of dissension will likely be less pronounced than such artificially constructed rhetorical debates might suggest. indeed there are many assumptions and values expressed by such skeptics with which to concur. yes, the emphasis must be focused on real decisions that make a difference. yes, a good understanding of the needs and preferences of our users must mediate the evidence. but no, we must not let evaluation stifle creativity and innovation. by espousing such criticism and reflection such skeptics are perhaps closer to being evidence based practitioners than they would either wish or have us believe! the paradigm may well be flawed, but such limitations cannot be considered to be fundamental; they relate instead to the immaturity of its development. eblip does not command a blind allegiance, but it demands ongoing reflection and self-scrutiny. as with the cochrane collaboration, the very people who are most active in eblip are those most aware of what it can and cannot do and where it is and is not appropriate. indeed those of us who have been associated with the broader evidence based library and information practice, 2008: 3.4 78 context of evidence based practice for many years have already undergone extensive therapy and counseling to help us overcome these alleged deficiencies. we have moved on and overcome them. eblip 2.0, if we dare slip into such populist jargon, is quite a different “animal” from that first unleashed at the onset of the movement. we realize that, with less than two dozen randomized controlled trials in the known library universe, we are unlikely to overturn completely entrenched and unhelpful behaviours with rcts or systematic reviews (or perhaps even guidelines). we will have to employ the widest range of study designs and types and sources of best available evidence. this coincides with developments in many fields to define “evidence” in broader terms than those initiated by the narrow medical model. at the same time, this falls far short of the unsatisfactory verdict that “everything is evidence.” as information specialists we have always been custodians of quality and should continue to be so. this being the case, we should welcome the choice by any skeptic, friendly or otherwise, to raise objections and counterproposals through the pages of this journal. as julius caesar might have attested, “keep your friends at arm's length ... and your enemies even closer!” works cited banks, marcus. “friendly skepticism about evidence based library and information practice.” evidence based library and information practice 3.3 (2008): 86-90. 3 dec. 2008 <http://ejournals.library.ualberta.ca/ index.php/eblip/article/viewpdfinterst itial/473/3354>. booth, andrew. “an evidence based approach to collection management.” evidence based practice for information professionals: a handbook. eds. andrew booth and anne brice. london: facet, 2004: 185-95. ---. “using evidence in practice: retrospective.” health information and libraries journal. 25.special issue 1 (2008): 49-51. booth, andrew, and anne brice. “prediction is difficult, especially the future”: a progress report." evidence based library and information practice 2.1 (2007): 89-106. 3 dec. 2008 <http://ejournals.library.ualberta.ca/inde x.php/eblip/article/view/99/242>. straus, sharon e, and finlay a. mcalister. “evidence based medicine: a commentary on common criticisms.” canadian medical association journal, 163.7 (2000): 837-41. research article   perceptions of the skills of graduates in the library and information science and technology degree of p. porto, portugal: a statistical data analysis of the alignment between students, teachers, and employers   susana martins senior lecturer, library and information science and technology porto accounting and business school (iscap) of polytechnic of porto porto, são mamede infesta, portugal email: susanamartins@iscap.ipp.pt   isabel cristina lopes senior lecturer, library and information science and technology porto accounting and business school (iscap) of polytechnic of porto porto, são mamede infesta, portugal email: cristinalopes@iscap.ipp.pt   milena carvalho senior lecturer, library and information science and technology porto accounting and business school (iscap) of polytechnic of porto porto, são mamede infesta, portugal email: milenacarvalho@iscap.ipp.pt   received: 5 mar. 2021                                                                accepted: 4 june 2021         2021 martins, lopes, and carvalho. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   doi: 10.18438/eblip29944     abstract   objective – the objective of this research was to identify the four professional competences of graduates in library and information sciences and technologies (list) considered most pertinent, from the points of view of students, teachers, and employers. we also sought to compare the perceptions of the different groups. the study was based on the premise that alignment of these perceptions may enhance the employability of list graduates.   methods – a questionnaire, used and validated by arias-coello et al. (2014), was further translated by martins and carvalho (2018). the questionnaire consisted of a set of questions regarding four dimensions: information management; communication and interpersonal relationships; domain and application of information technologies; organization management. we sent the survey by email to the target audience; it was available to complete in april and may 2018. data analysis included calculating mean and standard deviation, as well as shapiro wilk normality tests, statistical tests for multiple comparisons, anova, kruskal wallis test, friedman test, wilcoxon test, and pearson's correlation.   results – in relation to certain dimensions, one could think that age would be a determining factor, but this has not been proved. in fact, results showed that age is not a factor that influenced the importance attributed to different competences in the several dimensions. the respondents' academic degrees and areas of knowledge were linked to significant differences transversally. the kruskal wallis test indicated that students, teachers, and employers perceived the importance of information and communication technologies (ict) skills equally. as for the other competences, overall there were significant differences between students and employers, and there were significant differences between students and teachers regarding the perceived importance of organization management skills. there were also significant differences between teachers, students, and employers regarding the perceived importance of communication skills. we also found that responses within the teacher group had less dispersion of answers, therefore there was greater internal agreement. the opposite occurred in the employer group.   conclusion – the differences detected in the perception of the different groups were minor. however, it is necessary to create initiatives for the alignment of the perceptions of students and employers, because if all groups have the same perception, they will develop and value the same skills, responding to the needs of the labor market, thus promoting the employability of list graduates. the inclusion of a curricular internship, even one of short duration, in the first year of the degree could also be a way of endeavoring to bring together the expectations of both groups. these suggestions are part of a proposal to change and update the study plan and enhance the performance of course management.     introduction   at the beginning of the 21st century, the portuguese higher education system underwent major changes for compliance to the bologna declaration. as higher education lecturers, the authors experienced this change in the first person and, in addition to the change in form, the assumptions inherent to higher education have also changed. on the one hand, at the level of teaching, new methodologies, which should benefit from information and communication technologies (ict), were implemented, as well as information literacy and active teaching strategies, such as the use of laboratories and tutoring. on the other hand, at the level of learning, students became responsible for their own learning, taking up a proactive posture and becoming aware that their learning does not end with the completion of the degree, but will take place throughout their life: lifelong learning.   these changes, among other objectives, were intended to facilitate the employability and mobility of the youngest students (european higher education area, n.d.).   amaral (2005) believed that the change of paradigm that underlies the effective implementation of the principles contained in the bologna declaration presupposes an adaptation of the degrees to the new professional and social reality.   information literacy is of vital importance in this context. the evolution of information science has led to a user-centered paradigm, as well as to the enhancement of the importance of information and of the social role of success in shaping a more democratic and inclusive society (comissão europeia, 2009). it is now certain that the work of the information professional must be based on the needs and interests of the users and, at the same time, must support their activity in the mediation of information, that is, in the construction and consolidation of mechanisms and instruments that allow users to appropriate the information they need, to stimulate and facilitate access to and use of information.   in the library and information sciences and technologies (list) degree, we believe that the training of professionals and the increase of their ict skills is necessary in order to achieve the full valorization of information as a human and social phenomenon. that is why each year the list meetings (encontros de ctdi) are organized. the list degree and the master in business information are both degrees of iscap-p, porto (portugal) which have been structured as stipulated in the euro-referencial i-d (2005). this is the document that, to this date, characterized the "skills and aptitudes of european information-documentation professionals" (volume i) and their "qualification levels of european information-documentation professionals” (volume ii). as said in the document, it was intended for a number of categories of users (information-documentation professionals in progress, employers-recruiters, people wishing to be oriented to these activities, trainers) and implied various uses (writing a curriculum vitae, career advancement, self-assessment, development of a training program). their content presupposed a certain standardization of the profession which, despite the changes, continues to have to search and find information, describe, organize it, and make it accessible to those who need it.   with this work we intend to assess the perceptions of three different groups (students, teachers, and employers) linked to list regarding information management, communication skills, information technologies and management. the study is based on the assumption that the alignment of these perceptions, although not always perfect, may, in our understanding, enhance the employability of list graduates. conclusions will be drawn regarding the data collected, as well as their subsequent analysis.   this study originated from a spanish study entitled “professional skills of post-graduates in the master in documentation, libraries and archives management from the perspective of employers, students and teachers” published by arias-coello et al. (2014), in which the professional skills of the graduates of a master's degree in documentation, libraries and archives were identified precisely considering three perspectives: student, teacher, and employer. in 2018, martins and carvalho presented a paper which, at the time, was based on a descriptive statistical analysis of the data collected. the data collection tool was translated and adapted by the authors to their national reality.   literature review   according to ribeiro and silva (2004), the euro-referential has a vision about the information and documentation professional and about his profession that “is defined by its fundamental mission of researching, treating, producing and disseminating information incorporating added value in order to meet the needs of information, expressed or not, by a target audience, and propose information resources, usually consisting of ‘documents’ (texts, images, sounds)” (p. 4). naturally, the constant mutability of the context and technologies requires a persistent adaptation of the professional and the profession itself. also, according to the authors, the components of the information professional's performance are knowledge (know and know how to do) and skills (referred to as know-how to be).   pinto and ochôa (2006) stated that “the strategies to rethink the profession imply career enhancement, both by professionals and by employers" (p. 39). lou and pang (2003, as cited in pinto & ochôa, 2006) categorized these strategic initiatives in three areas: promotion of employability, improvement image with employers, and strengthening of external contacts. effectively aligning perceptions of the information and documentation profession is essential for the success of the employability indicator.   for ribeiro (2008), the traditional training (custodial, historicist, and patrimonialist) of information professionals centered their performance in the cultural world although, due to technological evolution, another technical approach anchored in the need to describe, organize, and make information available has emerged. it would have been, however, in the middle of the 20th century that a paradigm shift occurred, as a consequence of the explosion of scientific and technical information, together with the development of information technology and with the treatment and recovery of information. this new context converged to a new reality, with new demands, not only at the level of the profession, but also at the disciplinary level and in the training of professionals.   indeed, throughout the years, programs and projects for the sensitization, training, and adaptation of professionals in this area have emerged, developed both by professional associations and higher education entities. in particular, we wish to highlight changes in the academic training of information professionals in portugal, where the main pioneers in 2001 were the faculty of arts and humanities of the university of porto (with a degree in information science, jointly taught with the faculty of engineering) and the school of industrial studies and management of the polytechnic institute of porto (with a list degree and the master in business information, structured as stipulated in the euro-referential).   it should be noted that the euro-referential is the document that characterized the "skills and aptitudes of european information-documentation professionals" and the "qualification levels of european information-documentation professionals. according to the euro-referential i-d (2005), "this euro-referential was carried out by professionals from a european perspective. the skills and aptitudes demonstrated in the different occupations of the information-documentation profession (archivist, librarian, documentalist, alert service, etc.) were identified and compared. this tool is intended for a number of categories of users (information-documentation professionalsin progress, employers-recruiters, people wishing to be oriented to these activities, trainers) and various uses (writing a curriculum vitae, career advancement, self-assessment and development of a training program)” (p. 11). these guidelines presupposed a certain standardization of the profession which, despite the changes, continues to have to research and find the information, treat it, and make it accessible to those who need it.   we are not aware of studies that have similar objectives to this in national territory. some studies have been carried out, such as pradhan (2015) and kumar (2010), which show that employability of list graduates is around 100% and that communication skills as well as information and technology (it) skills are of great importance regarding employability. the importance of a hands-on practice is also present in these studies, alongside the need to develop adaptation skills to a constantly changing environment. good communication skills, for example, conduct a negotiation that brings together multiple actors with antagonistic interests, taking into account the strategic interests of the company, problem-solving attitude, good knowledge of it, presentation skills, and ability to provide services with motivation and commitment. these are some of the assets which the authors consider likely to increase the probabilities of employment.   this research was based on a 2014 study developed by arias-coello et al. entitled “professional skills of post-graduates in the master in documentation, libraries and archives management from the perspective of employers, students and teachers,” and on a more recent study by martins and carvalho (2018), who developed a first analysis of the data collected.   we believe it is important to know the extent to which there is convergence in what concerns the immediate actors of higher education and the labor market, namely finalist students, teachers, and employers, in particular with regard to the perceptions about the relevant skills for professional practice. we also believe that this convergence will facilitate greater success in employability and that actions, at the level of the degree, can be conducted to refine these perceptions and obtain the desired convergence.   aims   in this research, we assessed the perceptions of three different groups linked to a list degree (lists’ last year students, list teachers, and list graduates’ potential employers) regarding four dimensions. the study is based on the assumption that the alignment of these perceptions may enhance the employability of list graduates.   we based the study on the premise that identical expectations and perceptions of professional needs among the different actors in the training process in higher education and in the consequent entry into professional life are drivers of greater employability. the research questions that guided this work were: are the perceptions of students, teachers and employers regarding different sets of skills all aligned? and if not, what can we do to promote that alignment and consequently foster employability?   methods   regarding the methodology, we applied the same questionnaire used in and validated by the study of arias-coello et al. (2014), and formatted it in google forms. the questionnaire consisted of 4 dimensions, subdivided into 29 factors, which resulted in 29 questions. the four dimensions were:   1.      information management; 2.      communication and interpersonal relationships; 3.      domain and application of information technologies; 4.      organization management.   the response scale comprised values between 1 and 10 to rate the perceived importance of each skill, with 10 being the highest value of importance (most important) and 1 the lowest value (least important). the questionnaire was made available electronically via email to the three groups: a) the finalist students (n=28); b) the teachers of the degree (n=12) and c) the employers (here represented by the institutions of traineeship) (n=84), which were first contacted by telephone. the groups a) and b) were previously contacted in person in order to raise awareness for the completion of the questionnaire that was subsequently sent by email. the group c) was contacted by telephone. during the period of the data collection, from april to may of 2018, weekly reminders were made by phone and by email. the response rate of potential employers was below expected, only 26% (22 entities), the response rate of the students and teachers was close to 100%, respectively 82% (23 last year students) and 92% (11 teachers).   the data collected was later exported. ibm spss was the software used for data analysis, alongside descriptive statistics—namely mean and standard deviation—as well as shapiro wilk normality tests, statistical tests for comparisons such as anova, kruskal wallis, wilcoxon and friedman tests, and pearson's correlation. the results obtained are described in the next section.   figure 1 boxplot of the importance of information management skills for students, teachers, and employers.   results   the analysis of individual competences within each dimension, separated by the respondents' role (student / teacher / employer) in relation to the degree, and taking into account the importance of information management skills for the three groups of subjects, revealed that, on average for students, the most important competence was “knowledge to navigate communication networks through search engines and other tools” (for details on the descriptive statistics, see appendices a, b, c, and d). the second most important was “ability to analyze and synthesize information”, whilst the third most important was “knowledge about information sources, retrieval and its storage.”   figure 1 shows a box around the quartiles, the median in the center of the box, and the minimum and maximum values assigned by students, teachers and employers to each aspect evaluated in this study. the stars and circles represent outliers.   regarding the teachers, this group appeared, as a whole, as the group with the least dispersion in the answers and presented the two most important answer options: “knowledge about information sources, retrieval and its storage” and “ability to analyze and synthesize information”, followed by a parameter which also included two answers, namely “ability to manage information management and control systems” and “knowledge in database management.” the third most important were “knowledge about content management” and “knowledge of standardized methods of description, presentation and transmission of information.”   in regard to employers, this group presented, in general, a greater dispersion in the respondents' answers. “knowledge in database management” proved to be the first most important answer, followed by “ability to analyze and synthesize information”, whereas “knowledge to navigate communication networks through search engines and other tools” was the third most important.   developing a similar analysis, yet considering the importance of communication skills for students, teachers and employers, figure 2 evinces that, for students, the first most important competence is the “ability to work as a team,” the second most important is the “ability to train, coordinate and direct teams,” and the third most important is the “ability to communicate orally and in writing in their native language.”   regarding the teachers, similarly to what happened with the previous item, this group was also the one with the lowest dispersion of responses. the first most important option was “ability to communicate orally and in writing in their native language,” followed by “ability to speak and write in english,” while in the third most important this group presented two tied options: “ability to work as a team” and “ability to socialize with classmates and superiors.”   in regard to employers, this group was the one with the greatest dispersion in responses. the most important answer was shown to be the “ability to socialize with classmates and superiors,” followed by the “ability to work as a team,” leaving the “ability to relate to users” for third.   figure 2 boxplot of the importance of communication skills for students, teachers, and employers.   figure 3 boxplot of the importance of skills in information technology for students, teachers, and employers.   if we take into account the importance of skills in information technology for students, teachers and employers (figure 3 and appendix c), we can say that, regarding the average, for students the first most important option was the “domain of web applications,” while the second most important was the “knowledge of electronic resources and applications to manage a changing technical process.” as for the third most important, two options arose: “sufficient technical knowledge to solve unforeseen problems” and “knowledge about management and design of intranet and web pages.”   throughout this analysis, for teachers (this group had less dispersion in their answers) the most important option was the “ability to implement a management system,” the second most important was the “domain of web applications,” and the third most important was “sufficient technical knowledge to solve unforeseen problems” (see figure 3). with regard to employers, overall, the dispersion of responses by this group was the highest. the first most important option was the “domain of web applications,” the second most important was “knowledge of electronic resources and applications to manage a changing technical process,” and the third most important was “sufficient technical knowledge to solve unforeseen problems.”   finally, regarding the importance of the organizational management skills (figure 4 and appendix d), in tune with the previous analysis, for students, the first most important was the “ability to manage projects,” the second most important was “problem solving ability,” and the third most important was the “ability to analyze and organize statistical data.” as for teachers, and in agreement with what happened in the other dimensions, this was the group that presented least dispersion in the answers. the first most important proved to be “problem solving ability,” followed by “analytical ability to combine and organize complex information,” while the third most important remained the “ability to analyze and organize statistical data.” with regard to employers, also in line with what happened in the other dimensions, they were the ones who presented the greatest dispersion in the answers given. for them, the first most important was “problem solving ability,” the second most important was the “analytical ability to combine and organize complex information,” and the third most important was the “ability to manage projects.”   figure 4 boxplot of the importance of organizational management skills for students, teachers, and employers.   for each individual who answered the questionnaire, we calculated the average of the scores attributed in the questions for each of the four dimensions, as well as the global average of the scores attributed in all questions. we analyzed these five quantitative variables using inferential statistical tests. shapiro wilk normality tests were performed for the four dimensions and for the global average (see appendix e), considering the three groups surveyed. with a significance level of 5%, the shapiro wilk normality tests revealed that only the variable of the average importance of information management skills, spread by the three roles in relation to the degree, could be considered normally distributed, while the other variables could not be considered normally distributed.   thus, to determine whether students, teachers and employers assigned equal average importance to the skills listed, the anova test was used for the first variable and the kruskal-wallis non-parametric test for the others.   in figure 5, the average of the importance of skills in the four dimensions are compared regarding the three roles under analysis, and also the global average of all skills. students and teachers seemed to assign more importance to communication and interpersonal skills and employers to organizational management skills.   figure 5 boxplot of the average importance of skills in the four dimensions and the global average.   throughout the anova test and the kruskal wallis test (tables 1 and 2), it was found that there were significant differences for students, teachers, and employers regarding the importance of skills in dimensions 1, 2, 4 and, globally, at the level of 5% of significance. the kruskal wallis test indicated that students, teachers, and employers perceived the importance of ict skills equally.   multiple comparisons with the tukey hsd test (table 3) resulted in significant differences, at the level of 5% of significance, between the average importance of information management skills for students and employers, but not between students and teachers, nor between teachers and employers.   the statistical tests carried out for multiple comparisons also indicated that there were no differences between teachers and employers in the importance attributed to communication and interpersonal skills, but there were significant differences, at the level of 5%, between teachers and students and between students and employers.   tukey hsd tests also indicated that there were no differences between teachers and employers or between students and employers in the importance attributed to the organization management skills, yet that there were significant differences, at the level of 5%, between teachers and students.   statistical tests for multiple comparisons indicated that there were no differences between teachers and employers nor between teachers and students in the global average importance attributed to all skills under analysis, and that there were only significant differences, at the level of 5%, between students and employers (table 3).   table 1 anova test to determine whether students, teachers, and employers assigned equal average importance to information management skills average importance of information management skills sum of squares df mean squares f p-value between groups 21.197 2 10.599 4.067 0.022 within groups 148.530 57 2.606     total 169.728 59         table 2 kruskal wallis tests to determine whether students, teachers, and employers assigned equal average importance to skills in several dimensions   average importance of communication and interpersonal skills average importance of information technology skills average importance of organizational management skills global average importance of skills chi-square 11.469 4.324 6.096 7.523 df 2 2 2 2 p-value 0.003 0.115 0.047 0.023   table 3 tukey hsd tests for multiple comparisons of the importance of skills regarding the different roles skills roles test statistic std. error p-value average importance of information management skills employer – teacher -0.74786 0.58061 0.408 employer – student -1.31308 0.46208 0.017 teacher – student -0.56522 0.59176 0.608 average importance of communication and interpersonal skills teacher employer -0.202 6.203 0.974 teacher – student 15.638 6.285 0.013 employer – student 15.437 4.954 0.005 average importance of organizational management skills teacher employer -4.963 6.269 0.428 teacher – student 14.310 6.389 0.025 employer – student 9.347 4.989 0.061 global average importance of skills employer – teacher 1.101 6.280 0.861 employer – student 13.018 4.998 0.009 teacher – student 11.917 6.401 0.063   regarding the analysis of the average competences in each dimension, aggregated for all roles, the reduced p-values obtained in the shapiro wilk normality tests indicated that the aggregated variables did not have a normal distribution (table 4), therefore non-parametric tests were used to analyze the aggregated variables.   regarding the comparison of the aggregated average skills in the four dimensions, the friedman test indicated that the four dimensions were not considered by the respondents to have equal importance (c2=19.183, with 3 degrees of freedom produced a p-value<0.0005).    wilcoxon's tests for multiple comparisons of the four dimensions (table 5) indicated that, at the 5% level of significance, there were no significant differences between the average importance of skills in organizational management and in information management, nor between average importance of skills in organizational management and in information technology. there was statistical evidence to consider that there were differences when comparing the averages of the importance of skills in the remaining dimensions.   when separating by area of knowledge of the respondent, the only variable that the shapiro wilk test considered normally distributed was the variable on the average importance of the skills in information management. for this variable, the anova test was used, and, for the others, the kruskall wallis test.   in the analysis of significant differences in the average importance of skills separated by area of knowledge, the anova test produced a p-value of 0.024, which indicated that there were significant differences in the importance of skills in information management for the various areas of knowledge of the people who responded to the survey. the kruskal wallis test indicated that there were significant differences at the level of 5% for the various areas of knowledge regarding the importance of skills in information technologies (p-value=0.034), organizational management skills (p-value=0.005) and globally (p-value=0.021), but not in communication and interpersonal skills (p-value=0.055).   if we separate the groups by academic degree, the only variables that the shapiro wilk test considered normally distributed were the variables on the average importance of the skills in information management and organizational management. the anova test was used for these variables, and the kruskall wallis test for the others. these test results indicated that there were only significant differences in the importance of communication and interpersonal skills (c2=12.702 and p-value=0.005) and globally (c2=9.109 and p-value=0.028) for the various academic degrees.   regarding age, pearson's correlation test showed that there were no significant linear relationships between the respondent's age and the importance attributed to competences in any of the dimensions (all correlations were less than 0.134 in absolute value).   table 4 shapiro wilk test for the importance of skills in the four dimensions (normality tests)   shapiro-wilk test statistic df p-value average importance of information management skills 0.948 59 0.014 average importance of communication and interpersonal skills 0.913 59 0.000 average importance of information technology skills 0.937 59 0.004 average importance of organizational management skills 0.921 59 0.001 global average importance of skills 0.919 59 0.001   table 5 wilcoxon tests for multiple comparisons     test statisticsa z p-value average importance of communication and interpersonal skills average importance of information management skills -3.032b 0.002 average importance of information technology skills average importance of information management skills -2.326c 0.020 average importance of organizational management skills average importance of information management skills -1.010c 0.313 average importance of information technology skills average importance of communication and interpersonal skills -4.275c 0.000 average importance of organizational management skills average importance of communication and interpersonal skills -3.775c 0.000 average importance of organizational management skills average importance of information technology skills -1.102b 0.270 a wilcoxon signed rank test b based on negative ranks. c based on positive ranks.   discussion   as stated in the introduction, the present study was based on the premise that identical expectations and perceptions of professional needs among the different actors in the training process in higher education and in the consequent entry into professional life are drivers of greater employability.   however, the desired alignment in relation to the perceptions of the different actors in the teaching and learning process did not exist in the way it was assumed to exist by the authors because the skills acquired by graduates will be able to, in theory, meet the needs of the labor market. indeed, a notorious gap exists, above all, between the perceptions assumed by students and their future employers.   a research question that guided this work was: are the perceptions of students, teachers and employers, regarding different sets of skills, all aligned? as seen, they were not fully aligned. however, there was a group that was closer to the other groups: teachers. in fact, considering the several tests and data analysis, there were significant differences at the level of 5% of significance between the average importance of information management skills for students and employers, but not between students and teachers, nor between teachers and employers. they also showed that there were no differences between teachers and employers in the importance attributed to communication skills, but there were significant differences, at the level of 5%, between teachers and students and between students and employers. the statistical tests for multiple comparisons indicated that there were no differences between teachers and employers or between students and employers in the importance attributed to the organizational management skills, yet that there were significant differences, at the level of 5%, between teachers and students; and there were no differences between teachers and employers nor between teachers and students in the global average importance attributed to all skills, and that there were only significant differences, at the level of 5%, between students and employers. the reason could be the age difference and the incipient knowledge of professional practice by students.   in view of the data collected and their interpretation, it is necessary to put into practice some actions that promote maximum alignment, namely clarification actions targeted at students and degree candidates. in-office training sessions on the skills of information professionals and on ways they can work in an organization, addressing different dimensions and competences, would be a further action that could have a positive impact. the creation of a curricular unit in the first year aimed at bringing students closer to their employers also seems to be a viable way of promoting this connection.   the skills of an information professional are broad and diverse and can be used in business in order to promote its efficiency. this principle will be the guiding principle of the whole process.   conclusion   skills currently required of the information professional are diverse and from different spheres. the euro-referential information-documentation, which framed and normalized the profession of information professionals in europe, bears witness to this, including diversified competences, some transversal, and assuming different levels. the acquisition of these skills by students in the information science area will therefore be essential for entering the job market.   this work was based on the premise that the perception of the necessary competences for the exercise of the information profession among the three groups involved in the teaching and learning process of the degree in list are properly aligned.  however, the basic premise has not been proven. there was no total alignment and, overall, the largest differences existed between students and employers, which can be an obstacle to the employability of new graduates. based on this result, different actions, such as information actions for both groups, in-office training for the employers, and the inclusion of a curricular internship, even one of short duration, in the first year of the degree, are proposed in order to harmonize the perceptions of the groups and therefore clarify and promote the relevant competences for entering and working in the labor market.   the main limitation of this work was the low response rate obtained from potential employers. another limitation was the fact that the implementation of some of these suggestions, in particular, of the internship in the first curricular year of the degree, require a lengthy process to obtain authorization from the national entity that accredits higher education degrees in portugal.   as future work, it is suggested to repeat this study after implementing the actions identified as necessary, so as to identify the success of the proposed approach.   author contributions   susana martins: conceptualization, formal analysis, writing – review & editing, investigation, methodology isabel cristina lopes: formal analysis, writing – review & editing, investigation, methodology, validation milena carvalho: conceptualization, formal analysis, investigation, methodology   references   amaral, a. (2005). bolonha, o ensino superior e a competitividade económica. in o processo de bolonha e a formação dos educadores e professores portugueses. porto: profedições. pp. 35–45.   arias-coello, a., simón-blas, c., & simón-martín, j. (2014). competencias profesionales de los postgraduados en el máster de gestión de la documentación, bibliotecas y archivos desde la perspectiva de empleadores, estudiantes y professores. [professional skills of post-graduates in the master in documentation, libraries and archives management from the perspective of employers, students and teachers.] revista española de documentación científica, 37(3). https://doi.org/10.3989/redc.2014.3.1121   carvalho, m. (2014). estudo da mediação e do uso da informação nos arquivos distritais. coimbra: universidade de coimbra.   comissão europeia. (2009). recomendação da comissão europeia sobre literacia mediática no  ambiente digital para uma indústria audiovisual e de conteúdos mais competitiva e uma sociedade do conhecimento inclusiva. jornal oficial da união europeia, l 227/9. retrieved from http://eur-lex.europa.eu/lexuriserv/lexuriserv.do?uri=oj:l:2009:227:0009:0012:pt:pdf   conselho europeu das associações de informação e documentação. (2005). euro-referencial d. lisboa: incite-associação portuguesa para a gestão da informação. retrieved from http://www.eseig.ipp.pt/documentos/doc_noticias/euroreferencial_p.pdf   european higher education area and bologna process. (n.d.). employability in the bologna process. retrieved october 10, 2020, from http://www.ehea.info/page-employability-in-the-bologna-process   kumar, b. (2010). employability of library and information science graduates: competencies expected versus taught—a case study. desidoc journal of library & information technology, 30(5), 74-82. https://publications.drdo.gov.in/ojs/index.php/djlit/article/view/621   martins, s. (2015). tecnologias de informação, literacia e bibliotecas do ensino superior da área metropolitana do porto. porto: universidade portucalense infante d. henrique.   martins, s. & carvalho, m. (2018). professional skills of the graduates in library and information sciences and technology from the point of view of the potential employers, students and professors. edulearn18 proceedings, 2814-2820. https://doi.org/10.21125/edulearn.2018.0751   morgado, j. c. (2009). processo de bolonha e ensino superior num mundo globalizado. educação & sociedade, 30(106), 37–62. https://doi.org/10.1590/s0101-73302009000100003   ochôa, p. & pinto, l.g. (2006). observar a profissão: fundamentos, metodologias e práticas. a imagem das competências dos profissionais de informação-documentação: relatório. lisboa: observatório da profissão de informação-documentação. 29-59.   pradhan, s. (2015). study of employability and needed skills for lis graduates. desidoc journal of library & information technology, 35(2), 106-112. https://publications.drdo.gov.in/ojs/index.php/djlit/article/view/8312     ribeiro, f. (2008). a formação dos profissionais de informação na universidade do porto: um modelo teórico-prático inovador assente numa perspectiva integrada. retrieved from https://repositorio-aberto.up.pt/bitstream/10216/14050/2/formacaoup000073241.pdf   ribeiro, f. & silva, a. m. (2004). formação, perfil e competências do profissional da informação. in actas do congresso nacional de bibliotecários, arquivistas e documentalistas 8.     appendix a   mean and standard deviation regarding the importance of information management skills for students, teachers, and employers     role student teacher employer     mean standard deviation   mean standard deviation   mean standard deviation   knowledge about content management   8.30 1.460   7.73 .905   6.73 2.031   knowledge about information sources, retrieval and its storage   8.35 1.668   8.00 1.095   6.85 2.130   ability to analyze and synthesize information   8.43 1.502   8.00 1.000   7.12 1.840   knowledge of data base management   8.09 1.756   7.82 1.168   7.15 1.891   ability to manage information management and control systems   8.04 1.796   7.82 1.250   7.00 2.000   knowledge about information providers and users   8.00 1.567   6.91 1.136   6.58 2.101   capacity for planning information systems   8.09 1.411   7.45 1.440   6.81 2.191   knowledge of standardized methods of description, presentation and transmission of information   8.09 2.151   7.73 1.191   7.00 1.980   knowledge to navigate communication networks through search engines and other tools   8.70 1.743   7.55 1.635   7.04 1.990   valid n   23     11     26         appendix b   mean and standard deviation regarding the importance of communication and interpersonal skills for students, teachers, and employers     role student teacher employer   mean standard deviation   mean standard deviation   mean standard deviation ability to socialize with classmates and superiors   8.83 1.193   7.73 1.191   7.79 1.668 ability to work as a team   9.04 1.224   7.73 .905   7.83 1.606 ability to relate to users   8.87 1.180   7.64 1.027   7.87 1.392 ability to communicate orally and in writing in their native language   8.91 1.379   8.09 1.136   7.58 1.640 knowledge transfer skills   8.70 1.222   7.45 .934   7.46 1.693 ability to train, coordinate and direct teams   8.96 1.364   7.18 .982   7.52 1.470 ability to speak and write in english   8.39 1.828   7.82 1.328   7.48 1.537 valid n   23     11     21       appendix c   mean and standard deviation regarding the importance of skills in information technology for students, teachers, and employers     role student teacher employer   mean standard deviation   mean standard deviation   mean standard deviation domain of web applications   8.39 1.438   7.64 1.027   7.04 1.907 knowledge of electronic resources and applications to manage a changing technical process   7.83 1.642   7.18 1.079   6.96 2.107 sufficient technical knowledge to solve unforeseen problems   7.91 2.234   7.45 1.508   6.88 2.142 knowledge of systems’ architecture   7.52 2.064   6.82 1.722   6.42 1.880 knowledge about management and design of intranet and web pages   7.91 1.832   7.27 1.272   6.54 1.816 ability to implement a management system   7.70 2.439   7.73 1.272   6.46 2.064 valid n (listwise)   23     11     26       appendix d   mean and standard deviation regarding the importance of organizational management skills for students, teachers and employers     role student teacher employer   mean standard deviation   mean standard deviation   mean standard deviation ability to manage human and financial resources   7.65 2.124   6.00 1.612   6.77 2.519 ability to manage and implement policies, deontological, social and legal codes   7.57 2.212   5.73 1.421   6.73 2.539 ability to create and organize services for the user   8.04 1.492   7.18 1.471   6.85 2.461 analytical ability to combine and organize complex information   8.22 1.650   7.73 1.272   7.04 2.522 ability to manage projects   8.61 1.530   7.00 1.000   6.96 2.441 problem solving ability   8.57 1.409   7.82 .874   7.19 2.530 ability to analyze and organize statistical data   8.26 2.050   7.64 1.433   6.81 2.367 valid n   23       11   26       appendix e   shapiro wilk test (normality test)     role shapiro-wilk   test statistic df p-value average importance of information management skills student 0.931 23 0.115 teacher 0.945 11 0.575 employer 0.953 25 0.291 average importance of communication and interpersonal skills student 0.857 23 0.004 teacher 0.934 11 0.455 employer 0.915 25 0.039 average importance of information technologies skills student 0.947 23 0.249 teacher 0.966 11 0.838 employer 0.861 25 0.003 average importance of organizational management skills student 0.936 23 0.149 teacher 0.900 11 0.183 employer 0.910 25 0.031 global average importance of skills student 0.941 23 0.186 teacher 0.954 11 0.691 employer 0.920 26 0.046   microsoft word art_charbonneau.doc evidence based library and information practice 2007, 2:4  46 evidence based library and information practice     article    demystifying survey research: practical suggestions for effective question design      deborah h. charbonneau  coordinator for information access and delivery and principal investigator, urban health  partners  vera p. shiffman medical library  wayne state university  detroit, michigan, usa  email: dcharbon@med.wayne.edu       received: 20 july 2007    accepted: 12 november 2007      © 2007 charbonneau. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    abstract    objective ‐ recent research has yielded several studies helpful for understanding the use of  the survey technique in various library environments. despite this, there has been limited  discussion to guide library practitioners preparing survey questions. the aim of this article  is to provide practical suggestions for effective questions when designing written surveys.    methods ‐ advice and important considerations to help guide the process of developing  survey questions are drawn from a review of the literature and personal experience.    results ‐ basic techniques can be incorporated to improve survey questions, such as  choosing appropriate question forms and incorporating the use of scales. attention should  be paid to the flow and ordering of the survey questions. careful wording choices can also  help construct clear, simple questions.    conclusion ‐ a well‐designed survey questionnaire can be a valuable source of data. by  following some basic guidelines when constructing written survey questions, library and  information professionals can have useful data collection instruments at their disposal.           mailto:dcharbon@med.wayne.edu http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:4  47 introduction  survey research is often undertaken for a  number of reasons. simply put, the survey is  a “type of research in which a sample of  individuals is asked to respond to  questions” (case 190). library surveys may  attempt to obtain input on awareness of  library services, ease of access, quality and  relevance of services, effectiveness of  outreach efforts, and reasons why services  or resources are not being used (bertot and  davis; plosker). according to novotny,  examples of survey research include “a  questionnaire distributed after a library  instruction session; a user satisfaction form  given to every person entering the library; a  telephone survey of a random sample of city  residents; and a small group interview with  some students about the library’s policies”  (20). the survey approach can also reveal  service issues and opportunities, identify  unmet needs, and obtain input for strategic  planning.     in survey research, questionnaires are often  used as “the primary data‐collection  instruments” (busha and harter 61). as  such, survey questionnaires can be designed  to assess the needs of both current and  potential customers (plosker 65). thus,  survey questions can explore issues of  satisfaction with current services, perceived  needs for other kinds of information, other  outside sources of information that are used,  how libraries can provide better service, and  the perceived role of the library.     the survey process begins by determining  what topics or areas of interest would  benefit from surveying. for example,  gauging user reaction to a new service or  evaluating the effectiveness of an outreach  program. while some surveys (e.g.  nationally conducted surveys) sometimes  cover a broad range of topics, one strategy is  to limit an inquiry to a narrowly defined  issue which will allow for an appreciation of  a single topic’s complexity. thus,  identifying the survey’s specific purpose  and considering how the data acquired from  the survey will be used are important  considerations. after understanding fully  the survey’s purpose, what information is  needed, and how that information will be  used, then the types of survey questions to  be asked will be determined (fink 8).     the survey approach encompasses a variety  of methods of data collection and the  manner in which answers to research  questions are obtained must also be taken  into account. for instance, surveys can be  conducted over the telephone (plosker 66).  surveys may also be administered by an  interviewer in face‐to‐face encounters, such  as interviewing individuals in person or  interviewing people together in small  groups. with self‐administered questionnaires,  questions are administered in writing and  respondents are asked to complete the  questionnaire themselves. a common form  of self‐administered questionnaire is the  mail survey (babbie 266). there is increasing  and widespread use of electronic versions of  questionnaires, such as email or web‐based  surveys.  fink highlights some of the  similarities and differences between online  surveys and other self‐administered  questionnaires, such as traditional paper‐ and‐pencil surveys. useful summaries of  web‐based surveys also help identify many  considerations associated with web‐based  surveys, such as time and cost, response  rates, and follow‐up procedures (gunn;  franklin and plum; and hayslett and  wildemuth).  thinking about data collection  methods when designing the questionnaire  is important because issues, such as having  adequate resources and available expertise  for analyzing and interpreting them, are  important considerations to be addressed  before the questionnaire is distributed.    as noted, a survey “is a method of collecting  information directly from individuals”  (novotny 19). a common element of  evidence based library and information practice 2007, 2:4  48 surveys is observations or measurements  (line 13; powell and connaway 84). walden  describes a range of sources in survey  research methodology to serve both the  novice and the expert.  powell and  lorenzetti assist with the identification of  study designs suitable for library research.  gothberg, novotny, and powell and  connaway offer helpful discussions on the  various types of survey studies in library  and information science research, including  issues of sampling, data collection, and data  analysis.     numerous studies in librarianship have  “relied upon the survey approach” (busha  and harter 88). a review of the recent  library and information science literature  has yielded several, mostly descriptive,  studies about how the survey technique has  been deployed in various library  environments (for example, see: chen and  chen; franklin and plum; perkins and yuan;  shenton and johnson; tennant et al.; walter).  major themes in the literature concentrate  on the information‐seeking behavior of  users or evaluating a specific library service  or program. while much of the existing  research is useful for understanding the use  of surveys in general, there has been limited  discussion in the library and information  science literature on the process of  constructing effective survey questions  suitable for library research (janes; jerabek  and mccain; novotny). as a result, further  guidance to assist library practitioners  preparing written survey questions is  warranted and will be addressed in this  article.    when well‐constructed and implemented  effectively, survey questionnaires can be  useful data collection tools. there are  several key steps in constructing a well‐ designed survey questionnaire. a survey  questionnaire is “an instrument specifically  designed to elicit information that will be  useful for analysis” (babbie 253). as such,  peterson states “questionnaire construction  is one of the most delicate and critical  research activities” (13). the aim of this  article is to provide practical suggestions  and advice useful to library practitioners  trying to craft their own survey questions.  table 1 highlights some of the commonly  seen problems with survey questions that  this article seeks to address.   recommendations are drawn from several  sources, including the works of babbie,  converse and presser, fowler, janes, plosker,  and personal experience (for example, see:  charbonneau et al). it is not intended to be  an exhaustive list, but rather a set of guiding  recommendations for constructing survey  questions gleaned from a selective review of  the literature. this includes choosing  appropriate question forms, techniques for  measuring attitudes, strategies for helping  respondents recall past activities and the  importance of pretesting.     • negative wording or leading statements  • double‐barreled (single question has multiple parts)  • use of jargon, vague, or confusing language  • not specific for recall of past activities (need to narrow the reference period)  • agree‐disagree statement does not fully capture range of intensity (try using scales)    table 1. common problems with survey questions        evidence based library and information practice 2007, 2:4  49 suggestions and considerations    use simple language and avoid jargon     in general, survey questions should observe  several guidelines. researchers have  emphasized the need for “simplicity,  intelligibility, and clarity” when crafting  survey questions (converse and presser 10).  as janes notes, “people will answer the  question you ask them, not necessarily the  question you wanted to ask them” (322).  thus, the wording of survey questions is  critical. in fact, the wording of questions can  influence the way in which people respond.  using simple and clear language should not  be underestimated in survey research.  mcmain and jerabek suggest avoiding  jargon, such as scientific terms, library, or  vendor‐specific terminology, and instead  using everyday language whenever possible.  when appropriate, definitions or  explanations of terms should be provided.  this is in agreement with several other  studies that found that the use of library  jargon and terminology can be problematic  for users (for example, see: chaudhry and  choo; cobus et al.; hutcherson; mcgillis  and toms).  therefore, it is important to be  cognizant of the terms that users prefer, or  those that users are more likely to be  familiar with, and to construct questions  accordingly. respondents should be able to  “read an item quickly, understand its intent,  and select or provide an answer without  difficulty” (babbie 258).    asking specific, rather than general  questions, is typically a good rule to follow.  questions should be “precise so that the  respondent knows exactly what the  researcher is asking” (babbie 255). moreover,  if respondents are being asked to select the  one best answer from among the options  that are provided, then this should be  clearly stated in the instructions. using  negative language or leading statements  should be avoided because the appearance  of such language in a questionnaire can lead  to unnecessary confusion or  misinterpretation. for example, words such  as control, restrict, or oppose convey negative  meanings and should be avoided (converse  and presser 14). the following question is a  classic example of a leading statement: “do  you own a library card?” (fowler 36).  according to fowler, “when a question is  phrased like this, there is a tendency for  respondents to think that the researcher  expects a “yes” answer” (36). therefore, one  possible alternative may be: “many people  get books from libraries. others buy their  books, subscribe to magazines, or get their  reading materials some other way. do you  have a library card now, or not?” (fowler  36). this wording provides some legitimacy  and some reasons why the “no” response is  acceptable. other examples of negative  wording and leading statements include  questions beginning with “are you aware  that” and “do you agree.”    open‐ended vs. close‐ended questions    choosing to incorporate open‐ended  questions, close‐ended questions, or a  combination of both question forms is also  an important consideration.  in general,  close‐ended questions are easy to tabulate  and analyze because respondents must  chose from among the offered alternatives.  in the case of closed‐ended questions, the  “respondent is asked to select an answer  from among a list provided by the  researcher” (babbie 254). limiting the  number of responses available “ensures that  everyone is using the same terminology”  (novotny 35). as such, closed‐ended  questions may also be referred to as forced‐ questions (bertot and davis 59). for example,  close‐ended questions can be constructed as  “yes, no, uncertain” or “multiple choice.”  when designing a close‐ended question, it is  important to consider all of the possible  responses and to address these in your  question so that respondents know what  evidence based library and information practice 2007, 2:4  50 information to provide (novotny 39).  questions about occupation, education level,  gender, age, and other demographic  characteristics are commonly constructed as  close‐ended questions. other examples of  close‐ended questions include:    • did you find the workshop helpful? (yes  or no)  • would you recommend this [workshop  or library service] to a co‐worker? (yes  or no)  • would you like the library to offer a  one‐hour workshop on [topic or  resource]? (yes or no)    open‐ended questions are another option  available to consider when creating  questions. open‐ended questions are “essay  types that allow the user to express more in‐ depth input as well as allow for opinions  and views” (plosker 67). basically, open‐ ended questions are when “the respondent  is asked to provide his or her own answer to  the question” (babbie 254). as such, open‐ ended questions can elicit responses from  the respondents in their own words.   examples of open‐ended questions include:    • what other services would you like to  see us offer?  • what suggestions would you make to  improve the [library service]?   • what do you like most about the  [library service] currently provided?   • what did you find helpful about the  workshop?  • what services were you looking for  today that were not found on the  [library’s] web site?  • what was your reason for using the  [library or library service] today?    thus, open‐ended questions can be utilized  to gain a richer understanding from the  users’ perspectives. furthermore, the use of  probes, such as asking for an example or if  there is anything else that respondents  would like to add to their answer, is another  useful approach to elicit added input  (converse and presser 43).  probe questions  are most frequently used to follow‐up  answers to prior open‐ended questions  (peterson 31).    in addition, open‐ended questions are  particularly useful when not all of the  possible responses can be identified when  constructing response options. converse  and presser state “when not enough is  known to write appropriate response  categories, open questions are to be  preferred” (34). however, it is also  recommended that “no more than 10  percent of total survey questions be open‐ ended” (plosker 67). this is due in part  because of the effort required on behalf of  the researcher to process and analyze such  narrative responses to open‐ended questions.  as fowler notes “answers in narrative form  produce data that researchers sometimes  find difficult to work with” (178). in  particular, answers must be coded; someone  must read the complexity of the answers  and put them into meaningful categories for  analysis which is a different process from  when respondents answer in a more  structured way. this recommendation for  the use of open‐ended questions is also due  in part because open‐ended questions  require more time on behalf of the  respondents to answer and may add  significantly to the survey time. it is  important to keep in mind that “survey  research involves an imposition on those  surveyed” (hernon 83). therefore, a  concerted effort to ensure that respondents  are not overwhelmed or experience survey  fatigue is imperative.     multiple questions on a topic    some items warrant more than one question  for investigation. by asking similar  questions a number of times, any “distorting  effects that may occur as a result of the  evidence based library and information practice 2007, 2:4  51 respondent misinterpreting a single  question” can be reduced (novotny 35).  therefore, another recommendation is to  ask about an item in several different ways.  as a general strategy, one can “look for  questions that cast light at different angles”  (converse and presser 47). this can be  accomplished using composite measures that  typically involve the measurement of an  attitude or behavior in which several items  are devised to help measure a single concept  (babbie 156). for example, crafting  questions regarding usage of digital  reference services and other electronic  resources can be combined to create a  composite of online user behavior (bejune  and kinkus 188). line also notes “much  greater precision and objectivity can be  obtained by employing several questions on  the same matter” (63). as such, using  multiple measures can help to reveal the  complexity of attitude about an issue.  multiple questions that seek to explore  satisfaction may include: “how satisfied are  you with todayʹs visit to our building,”  “how satisfied are you with todayʹs visit to  our web site,” and “how helpful was the  library staff today in answering your  question.” thus, such composite measures  provide a richer context of inquiry.     incorporating scales     surveys are “excellent vehicles for  measuring attitudes” (babbie 252). in fact,  one of the most popular ways that attitude  is measured are agree‐disagree statements;  however, this form has come under scrutiny  for not offering a range of possible options  for respondents (converse and presser 38).  consequently, one recommendation is to  incorporate the use of scales, which is a type  of composite measure “composed of several  items that have a logical structure among  them” (babbie 157). as such, scales offer the  advantage of moving beyond simple agree‐ disagree statements and are useful for  assessing “a dimension of attitudinal  position, with intensity, and how strongly a  position is felt” (converse and presser 38).     the likert scale is a commonly used format.  likert items are those using such response  categories as “strongly agree, agree,  disagree, and strongly disagree” (babbie  174).  somewhat similar to the likert format,  the semantic differential format asks  respondents to choose between two  dichotomous positions. however, the  semantic differential rating scale is  represented as a “seven‐point scale labeled  at either end by opposing positions”  (converse and presser 37). respondents are  asked to rate something in terms of two,  opposite adjectives (slight through extreme;  i.e. good‐bad) and the various points on the  scale bridge the distance between the two  opposites (babbie 175). for example, a  semantic differential rating scale can be  used to measure satisfaction. (see table 2).  a semantic information measure is based on  the “assumption that the more content  elements implied by a statement, the more  information it provides” (tague‐sutcliffe 70).  over the years, several likert‐type scales  and several variations of the semantic  very important  1  2  3  4  5  6  7  not important  very helpful  1  2  3  4  5  6  7  not helpful  excellent  1  2  3  4  5  6  7  seriously inadequate  friendly  1  2  3  4  5  6  7  unfriendly  polite  1  2  3  4  5  6  7  rude    table 2. examples of semantic differential scales  evidence based library and information practice 2007, 2:4  52   differential rating scale have been offered  (peterson 75). nonetheless, both the likert  and semantic differential scales attempt to  improve the levels of measurement through  the use of “standardized response categories  in survey questionnaires to determine the  relative intensity of different items” (babbie  174).  narrowing the reference period     other strategies are appropriate for when  respondents are being asked to recall past  events and activities. one useful strategy is  called narrowing the reference period  (converse & presser 21). this technique  provides a common frame of reference and  enhances the validity of the reporting of the  past. in fact, it may be helpful to reduce the  reference period to the immediate past. for  example, respondents can be asked to  indicate how often they used a specific  library service or resource within the last  week, over the past weekend, or yesterday.  likewise, the critical incident technique is a  method that has been used in library use  studies (for example, see: andrews; bush et  al.; siegel et al.; urquhart et al.).  in such  cases, individuals were asked to report on  actual instances “which contributed  significantly to the activity or behavior  under investigation” (line 51). for instance,  asking respondents about incidents in which  searches using a specific database were  either effective or ineffective is an example  of using the critical incident technique.  furthermore, using landmarks is another  useful technique to anchor the timing of  events (converse and presser 22). for  example, survey questions inquiring about  information behaviors or utilization of  services since the start of the new calendar  year or the start of a semester may prove  fruitful.         other considerations     double‐barreled questions are a common  problem and can be easily avoided. a  double‐barreled question is when a “researcher  asks respondents for a single answer to a  question that actually has multiple parts”  (babbie 255). for instance, consider the  following survey question: “please indicate  which types of information you need to support  your teaching and research.” a good rule to  follow is whenever the word and appears in  a question, check whether the question is  asking multiple items (babbie 255). a better  way to phrase the question is to re‐write the  question as two separate questions: “please  indicate which types of information you need to  support your teaching” and “please indicate  which types of information you need to support  your research.” a list of response categories  for respondents to select from should follow  each question.  additional examples of  multiple statements commonly seen in  surveys include:    • the library staff is readily available,  courteous, professional, and inviting.  • study space in the library is readily  available, quiet, and conducive to study.  • the library communicates information  about hours, services, and resources  adequately.  • the library delivers articles and books  requested through interlibrary loan in  a reasonable timeframe.  • the library’s web site is informative and  up‐to‐date.  • the library has sufficient evening and  weekend hours of service.    each of the above statements is asking about  multiple items. as such, the statements  should be closely examined and re‐written.  for instance, the statement: “the library has  sufficient evening and weekend hours of service”  can be improved when revised as two  separate statements: “the library has sufficient  evidence based library and information practice 2007, 2:4  53 evening hours of service” and “the library has  sufficient weekend hours of service.”    once the survey questions are crafted,  layout and design issues relating to the  construction of the questionnaire itself  should be addressed. the questionnaire  format should “maximize white space in the  form,” be uncluttered, and pleasing to the  eye (janes 324). it is also customary to  include a “cover letter stating the purpose of  the research and providing an explanation  of the importance of providing a response”  (losee and worley 145).    importance of pretesting    pretesting (or piloting) a survey  questionnaire is always recommended  (converse and presser 51). a pretest is a  “kind of dress rehearsal” done for  clarification and refinement (spaeth 71).  bertot and davis state that “the library  needs to engage in a survey design process  that includes pre‐tests of both the survey  questions and forms (be they electronic or  print)” (59).  respondents are “sensitive to  the context in which a question is asked”  (converse and presser 39). thus, pretesting  survey questions will help elucidate  whether the instructions were clear and  whether or not the questions provided  answers to the questions as posed “in the  sense of producing meaningful information”  (peterson 46).    pretesting also assists in identifying  question problems and practical aspects of  the questionnaire itself; such as making sure  the questions are clear and the questionnaire  is reasonably easy to complete (fowler 132).  pretests can be instrumental in helping to  catch problems of poor ordering or  confusing wording; especially if the survey  is tested in face‐to‐face encounters. when  surveys are pretested in situations where  body language and nonverbal cues can be  observed, important insight can be gleaned  to refine the survey questions and overall  questionnaire format (janes).      peterson states “all aspects of a  questionnaire’s structure, from question  sequencing to appearance [and] individual  question wording and format, should be  assessed” to determine any potential  problems (115).  in addition, novotny  suggests that unexpected responses may  indicate a problem “with the question’s  wording, or the set of options provided”  (42). additional problems to look for when  pretesting include “respondents skipping  questions, selecting more than one answer  to the same question, [and] making notes in  the margins” which are all indications that  the survey instrument is unclear in some  way (novotny 42).  furthermore, pretesting  the questionnaire and looking at the  consistency of the responses to the questions  helps to demonstrate the reliability of the  questionnaire and may highlight potential  problems with data collection and analysis.  lee provides a useful overview of reliability  and validity issues that researchers should  be aware of when conducting survey  research in libraries.    conclusion  in summary, a well‐designed survey  questionnaire can be a valuable source of  data and as such surveys are popular in  library research. chauvel and despres argue  that the survey technique “brings an issue  into focus by defining and detailing its  various characteristics” and allows library  practitioners to respond in a relatively quick  manner (208). drawing from the existing  literature, a number of useful techniques to  improve survey questions emerge. for  instance, the use of different types of  questions or specific rating scales (i.e. likert  scale or semantic differential scale) can be  used to measure the degree and intensity of  the respondents’ attitudes. careful wording  choices can also help construct clear and  easy‐to‐understand questions. furthermore,  evidence based library and information practice 2007, 2:4  54 an effort should be made to ensure that only  one question for each item is being asked  and that negative words are not being used  to bias responses. in conclusion,  constructing carefully written questions and  a well‐designed survey questionnaire can  help illuminate the needs and desires of  both current and potential library clientele,  shape or reshape services, or guide strategic  planning decisions.       works cited    andrews, j. ʺthe use of critical incident  research technique in an academic  library.ʺ library and information  research news 14.50 (1991): 22‐7.  babbie, e. the basics of social  research. 3rd ed. belmont, ca:  wadsworth publishing, 2005.     bejune, m., and j. kinkus “developing   composite of user behavior to inform  decisions about new and existing  library services.” reference services  review 34.2 (2006): 185‐92.    bertot, j.c., and d.m. davis. planning and  evaluating library networked services  and resources. westport, ct: libraries  unlimited, 2004.    bush, a, d.w. king, and c. tenopir.  ʺmedical facultyʹs use of print and  electronic journals: changes over  time and in comparison with  scientists.ʺ journal of the medical  library association 92.2 (2004): 233‐41.     busha, c.h., and s.p. harter. research  methods in librarianship: techniques  and interpretation. new york, ny:  academic press, 1980.    case, d.o. looking for information: a  survey of research on information  seeking, needs, and behavior. san  diego, ca: academic press, 2002.    charbonneau, d.h., e.b. marks, a.m.  healy, and c.f. croatt‐moore.  “collaboration addresses information  and education needs of an urban  public health workforce.” journal of  the medical library association 95.3  (2007): 252‐254.    chaudhry, a. s., and m. choo.  ʺunderstanding of library jargon in  the information seeking process.ʺ  journal of information science 27.5  (2001): 343‐9.     chauvel, d., and c. despres. ʺa review of  survey research in knowledge  management: 1997‐2001.ʺ journal of  knowledge management 6.3 (2002):  207‐23.     chen, c.w., and m.w. chen. ʺa study of  service quality on the usersʹ  satisfaction and reused intention: an  example of an academic library.ʺ  journal of educational media and  library sciences 44.1 (2006): 61‐82.     cobus, l., v.f. dent, and a. ondrusek.  ʺhow twenty‐eight users helped  redesign an academic library web  site: a usability study.ʺ reference &  user services quarterly 44.3 (2005):  232‐46.     converse, j.m., and s. presser. survey  questions: handcrafting the  standardized  questionnaire.   thousand oaks, ca: sage, 1986. fink,  a. how to ask survey questions.  thousand oaks: sage publications,  2003.    fowler, f.j. improving survey questions:  design and evaluation. thousand  oaks: sage publications, 1995.   evidence based library and information practice 2007, 2:4  55 franklin, b., and t. plum. “successful web  survey methodologies for measuring  the impact of networked electronic  services (mines for libraries).” ifla  journal 32.1 (2006): 28‐40.    gothberg, h.m. ʺthe library survey: a  research methodology rediscovered.ʺ  college and research libraries 51.6  (1990): 553‐9.     gunn, h. “web‐based surveys: changing  the survey process.” firstmonday 7.12  (2002)<http://www.firstmonday.org/iss ues/issue7_12/gunn/>    hayslett, m.m., and wildemuth, b.m.  “pixels or pencils? the relative  effectiveness of web‐based versus  paper surveys.” library & information  science research 26 (2004): 73‐93.      hernon, p. ʺsurvey research: time for some  changes.ʺ journal of academic  librarianship 26.2 (2000): 83‐4.     hutcherson, n.b. ʺlibrary jargon: student  recognition of terms and concepts  commonly used by librarians in the  classroom.ʺ college and research  libraries 65.4 (2004): 349‐54.     janes, j. ʺsurvey research design.ʺ library  hi tech 19.4 (2001): 419‐21.     jerabek, j.a., and l.m. mcmain. ʺassessing  minds want to know: developing  questions for assessment of library  services supporting off‐campus  learning programs.ʺ journal of library  administration 41.1/2 (2004): 303‐14.     lee, d. ʺsurvey research: reliability and  validity.ʺ library administration and  management 18.4 (2004): 211‐2.     line, m.b. library surveys: an introduction  to the use, planning procedure, and  presentation of surveys. 2nd ed.  london: clive bingley ltd., 1982.     lorenzetti, d. “identifying appropriate  quantitative study designs for library  research.” evidence based library and  information practice 2.1 (2007): 3‐14.     losee, r.m., and k.a. worley. research and  evaluation for information  professionals. san diego, ca:  academic press, 1993.    mcgillis, l., and e. g. toms. ʺusability of  the academic library web site:  implications for design.ʺ college and  research libraries 62.4 (2001): 355‐67.     novotny, e. “surveys and questionnaires.”  evaluating reference services a  practical guide. ed. jo bell whitlatch.  chicago: american library association,  2000.    perkins, g.h., and h. yuan. “genesis of a  web‐based satisfaction survey in an  academic library: the western  kentucky university libraries’  experience.” library administration &  management 14.3 (2000): 159‐66.     peterson, r.a. constructing effective  questionnaires. thousand oaks, ca:  sage publications, 2000.    plosker, g. r. ʺconducting user surveys:  an ongoing information imperative.ʺ  online 26.5 (2002): 64‐8.     powell, r.r. “recent trends in research: a  methodological essay.” library and   information science research 21.1  (1999): 91–119.    powell, r.r., and l.s. connaway. basic  research methods for librarians.  fourth edition. westport, ct: libraries  unlimited, 2004.  http://www.firstmonday.org/issues/issue7_12/gunn http://www.firstmonday.org/issues/issue7_12/gunn evidence based library and information practice 2007, 2:4  56 schlichter, d.j., and j.m. pemberton. ʺthe  emperorʹs new clothes? problems of  the user survey as a planning tool in  academic libraries.ʺ college and  research libraries 53.3 (1992): 257‐65.     shenton, a.k., and a. johnson. ʺthe  employment of online questionnaires  within user studies research in school  libraries.ʺ library and information  research 30.95 (2006): 25‐31.      siegel, e r., et al. “evaluating the impact of  medline using the critical incident  technique.” proceedings of the annual  symposium on computer application  in medical care (1991): 83‐87.     spaeth, j. l. ʺperils and pitfalls of survey  research.ʺ proceedings of the allerton  park institute, robert allerton park,  monticello. illinois, 27‐29 oct 91.  illinois university at urbana‐ champaign, graduate school of  library and information science, 1992.     tague‐sutcliffe, j. measuring information:  an information services perspective.  san diego, ca: academic press, 1995.    tennant, m.r., et al.. ʺevaluation of a  liaison librarian program: client and  liaison perspectives.ʺ journal of the  medical library association 94.4  (2006): 402‐409.     urquhart, c., et al. ʺcritical incident  technique and explicitation  interviewing in studies of information  behavior.ʺ library and information  science research 25.1 (2003): 63‐88.     walden, g. r. ʺselected reference sources in  polling and survey research  methodology, 1971‐1996.ʺ reference  services review 24.4 (1996): 49‐66.     walter, s. ʺmoving beyond collections:  academic library outreach to  multicultural student centers.ʺ  reference services review 33.4 (2005):  438‐58.       evidence summary   there is a lack of standardization in the collection development and circulation policies of prison library services   a review of: conrad, s. (2012). collection development and circulation policies in prison libraries: an exploratory survey of librarians in us correctional institutions. the library quarterly, 82(4), 407-427. doi: 10.1086/667435   reviewed by: michelle dalton librarian university hospital limerick limerick, ireland email: michelledalton@gmail.com   received: 14 dec. 2012  accepted: 8 apr. 2013      2013 dalton. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to explore how collection development policies currently support the role and purpose of prison libraries, and to explore if the accessibility of circulation records impacts on patron privacy.   design – online survey questionnaire and a case study analysis of the existing policy statements of selected correctional institutions.   setting – the prison library sector in the united states.   subjects – 17 librarians and library staff across ten states in the united states.   methods – an eight-question online questionnaire was used to explore the existing collection development and circulation policies in prison libraries, and the level of adherence to the guidelines of the association of specialized and cooperative library agencies (ascla) and the american correctional association (aca). in addition, participants were encouraged to forward any circulation or collection development policy statements for more detailed analysis. each policy was then reviewed to assess the degree of alignment or otherwise with the american library association’s (ala) prisoners’ right to read guidelines (2010).   main results – the results of the survey found that 24% of libraries had no formal collection development policy, and at least 53% of libraries had no circulation policy statement. in these instances, the libraries were typically subject to the local policies and procedures of the correctional institution. the purpose of the library and its collection was primarily viewed as: providing recreational reading material; maintaining contact with the outside world and enabling re-entry into the community; and supporting vocational skills and lifelong learning. in selecting materials, the results indicated that a broadly similar approach to that of public libraries was adopted by most institutions, with the exception of any material that may pose a safety or security threat to the institution. in one institution the use of library services or resources for legal purposes or to provide legal assistance was also clearly prohibited in the collection development policy, although approximately half of the libraries did state that providing legal material was one of their roles. the lengthy and arduous approval process for ordering books and other materials (up to ten months in one instance) was reported by several participants due to the layers of bureaucracy and controls inherent in the prison setting. with regard to circulation records and confidentiality issues, 35% of libraries deleted such records instantly upon return of the items, compared with 30% that archived them. a further 29% only retained information from the current and most recent patrons for the purposes of assessing and charging for damaged items.   conclusion – the author found the prison library sector to be a relatively challenging environment. in this context, following the existing guidelines and best practice as recommended by the ala and others, and establishing clear and ethical policy statements can help libraries to support the needs and rights of patrons more effectively.     commentary   the author addresses a sector that has received comparatively little attention in the library and information practice literature to date. the study is framed in the context of recent legal and media controversy which has drawn (unproven) parallels between a prisoner’s access to reading material and the subsequent crimes that may be committed. the latter is obviously a broad and important social issue, not just of relevance within the library and information research community.   the detailed discussion in the literature review highlights some important issues in the sector, in particular the unique ethical dilemmas and challenges faced by prison librarians. indeed it is difficult for policy statements to fully capture the delicate and nuanced balance that must be struck between protecting the institution and upholding prisoners’ rights to access material. moreover, librarians also have to contend with the perceptions of the institutional administration (and even the public), who may see providing full access to library services for prisoners, including recreational reading material, as being in conflict with the intrinsic concept of punishment. these predicaments may help to explain some of the inconsistencies in collection development and circulation policies across institutions that the survey results illuminate.   however, the small sample size in the study does undermine the validity of the results and the ability to make subsequent inferences to the broader population. while this is a problem that is openly acknowledged by the author, there are other methodological concerns that are not identified; for instance, no information is provided as to whether the survey was piloted in advance of its distribution. the fact that participants were required to enter their name and institution may have discouraged some individuals from completing the survey, and introducing anonymity may have helped to generate a larger sample. furthermore, as 6 of the 17 institutions are based in colorado this may introduce some degree of bias into the results. although the author does find evidence of intrastate inconsistencies, indicative that policies tend to be determined at the institutional rather than the state level, this fact does not altogether eliminate this concern (boyton & greenhalgh, 2004).   while reference is made to relevant ala policies, and the recommendations of the ascla and aca, the questionnaire does not appear to survey library staff directly on their awareness or use of such tools (or at least this information is not presented in the article). this may have been a useful addition in order to explicitly assess the level of consciousness and penetration of the existing guidelines. the author also states that these guidelines are at least two decades old, however does not comment as to whether there is a need for an updated approach to reflect present realities, and perhaps this was a question worthy of inclusion. in this context, benchmarking current policies against what may be out-dated standards arguably weakens the potential importance and value of the results.   in terms of how the data are presented by the researcher, the figures included are somewhat confusing as the axes are generally unlabelled. it is probable that the y-axis in figures 1 and 2 refers to the number rather than the percentage of institutions, however this is not specified, and indeed in figure 3 percentages are used which reduces clarity even further.   the study adopts a narrow focus, but will certainly be of relevance to those working in the prison library sector by prompting them to examine their current collection development and confidentiality policies. it also raises issues of interest to a broader audience (particularly given the generally acknowledged similarities with the public library sector), namely the accessibility of circulation records and the potential censorship of library materials and policies by internal or external stakeholders. how libraries can resist such influences, and uphold best practice guidelines, is an important consideration for us all.     references   american library association (2010). prisoners’ right to read. in intellectual freedom manual. retrieved 28 april 2013 from http://www.ifmanual.org/prisoners   boynton, p. m., & greenhalgh, t. (2004). hands-on guide to questionnaire research: selecting, designing, and developing your questionnaire. bmj, 328, 1312. doi: http://dx.doi.org/10.1136/bmj.328.7451.1312   ebl 101   research methods: sampling   virginia wilson librarian, murray library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   originally published in: evidence based library and information practice, 9(2), 45–47. https://ejournals.library.ualberta.ca/index.php/eblip/article/view/22186/16578     received: 16 june 2014  accepted: 19 june 2014      2016 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   research often involves choosing a certain number of people or items in order to answer a question. a frequent question is how does one choose a sample? another question is how many individuals or items is enough? what sample size will give the best results for the question at hand? well, the answer to the latter is: “it depends”. it depends on the question, on the method one uses to administer the research project, on what kind of results one is hoping for.   sampling is a consideration in both qualitative and quantitative research. survey methodology, interviews, focus groups, bibliometrics, content analysis, usability testing, etc., all rely on an appropriate number of people or items being selected and examined. for the purposes of this brief column, i’ll look at sampling as it pertains to survey methodology, as much of this information can be applied to other types of research methodologies. a valid sample must be considered in order to obtain generalizability in quantitative research and trustworthiness in qualitative research.   there are various types of sampling methods, including nonprobability sampling and probability sampling. below is a very brief examination of the methods under each, adapted from basic research methods for librarians, 5th ed. (connaway & powel, 2010). sampling is a complex exercise, depending on the type. as usual, the brevity of this column necessitates only the briefest overview of the topics.   nonprobability sampling: the researcher cannot be sure of a “specific element of the population [i.e. the particular grouping that is being looked at] being included in the sample” (p. 117). the weakness of a nonprobability sample is that it “does not permit generalizing from the sample to the population because the researcher has no reassurance that the sample is representative of the population” (p. 117). still, these types of samples are easier and often cheaper to obtain than the alternative (which we will get to later), and they can be adequate depending upon the research question.   there are various types of nonprobability samples:   •     accidental (or convenience) sampling: selecting the cases at hand until the desired number of people/items is reached. •     quota sample: the same as accidental sampling, except that “it takes steps to ensure that the significant, diverse elements of the population are included” (p. 118). •     snowball sample: a cumulative sample is generated by starting with a few people and asking them to recommend more people. •     purposive sample: based on the researcher’s “knowledge of the population and the objectives of the research” (p. 119). •     self-selected sample: people self-identify with the desired population criteria and select themselves to participate in the study.   probability sampling: this type of sampling comes closer to the objective of sampling, that is, “to select elements that accurately represent the total population from which the elements were drawn” (p. 119). the critical piece in probability sampling is that “every element in the population has a known probability of being included in the sample” (pp. 119-120).   there are various types of probability samples:   •     simple random sampling: this is the basic sampling method in survey research and it “gives each element in the population an equal chance of being included in the sample” (p. 120). the simple random sample is generated most often by using a table of random numbers. there are variations of the random sample, differentiated by the way the samples are generated. ø  systematic sample: involves “taking the every nth element from a list until the total list has been sampled” (p. 123). ø  stratified random sample:  the population elements are divided into categories, then independent random samples are selected from each category. ø  cluster sample: the population (not the population’s elements as in stratified random sampling) are divided up into clusters and samples are drawn from the clusters. this is particularly helpful when a population cannot be easily listed for sampling purposes.   this has been a whirlwind trip through types of sampling, as the other main point i would like to address is the “how many” question. how many people/items are enough to be representative of any given population? the rule of thumb for sample sizes is the larger the better. however, time, funding, and a host of other issues also have a role to play in determining how big to go. connaway and powell state that there are four criteria that you can think about to help determine the necessary sample size:   1.     the degree of precision required (the less accuracy needed, the smaller the sample you can get away with) 2.     the variability of the population (the greater the variability, the larger the sample size) 3.     the method of sampling (i.e. “stratified sampling requires fewer cases to achieve a specified degree of accuracy” (p. 129). 4.     how the results are to be analyzed (small samples have limitations in terms of the types of statistical analyses that can be used)   there are formulas that can be used to determine the ideal number. luckily, for the mathematically challenged (like me) there are tables and calculators that researchers can use that already have the formulas applied:   ·         table: http://www.research-advisors.com/tools/samplesize.htm   ·         a sample size calculator: http://www.raosoft.com/samplesize.html  ·         calculator: http://www.surveysystem.com/sscalc.htm  ·         simple random sample calculator: http://www.nss.gov.au/nss/home.nsf/pages/sample+size+calculator      other resources related to sampling:   beck, s.e. & manuel, k. (2008). practical research methods for librarians and information professionals. new york: neal-schuman. (includes sampling across a variety of research methods).   bouma, g. d., ling, r., & wilkinson, l. (2009). the research process, canadian edition. don mills, on: oxford university press. (includes a chapter on selecting a sample and a table of random numbers).   cochran, w.g. (1977). sampling techniques. 3rd ed. new york: wiley.   kish, l. (1995). survey sampling. new york: wiley.   references   connaway, l.s. & powell, r.r. (2010). basic research methods for librarians, 5th ed. santa barbara, ca: libraries unlimited.     news   call for review papers: evidence based library and information practice    2023. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30421     evidence based library and information practice (eblip) welcomes review articles on topics of relevance to practitioners in library and information studies. reviews may take several different forms, including systematic reviews, scoping reviews, narrative reviews, meta-analyses, or state-of-the-art reviews, among others.    in its eighteenth year of publication, eblip is a peer reviewed, open access journal published quarterly on the 15th of march, june, september, and december. the purpose of the journal is to provide a forum for librarians and other information professionals to discover research that may contribute to decision making in professional practice.   further information regarding author guidelines for reviews and the online submission process can be found on the eblip website at https://journals.library.ualberta.ca/eblip/index.php/eblip/guidelines. questions about possible submissions can be directed to dr. lisl zach, associate editor (reviews), at lisl@marsez.com.   ebl 101   research methods: content analysis   virginia wilson liaison librarian murray library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   originally published in: evidence based library and information practice, 6(4), 177–179. https://ejournals.library.ualberta.ca/index.php/eblip/article/view/12180/13124     received: 27 oct. 2011   accepted: 27 oct. 2011      2016 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   content analysis, a method which can be used qualitatively or quantitatively for systematically analyzing written, verbal, or visual documentation, goes back to the 1950s and the study of mass communication (white & marsh, 2006, p. 22). key themes emerge from the documents after they are classified and coded. the content can come from a wide variety of sources: books, manuscripts, drawings, photographs, recorded conversations, videotaped events, messages on electronic mailings lists and online forums, blog posts, etc. content is analyzed by breaking it up into conceptual chunks that are then coded or named. qualitative analysis develops the categories as the analysis takes place. the results are used to make inferences about the messages in the text. quantitative analysis starts with a hypothesis and a predetermined coding scheme that is designed to test the hypothesis. the results are described using statistics. kimberly neuendorf and klaus krippendorff are two of many contemporary scholars in the area of content analysis. neuendorf sees the method as primarily quantitative, while krippendorf believes that counting is not a prerequisite of content analysis. verbal categories and the listing of quotes are considered as valid as numbers and counting.   there are two types of content analysis: conceptual analysis and relational analysis. conceptual analysis is largely what was described above: the content is coded for certain words, concepts, or themes, and the analyst makes inferences based on the patterns that emerge. relational analysis builds on conceptual analysis by delving into the relationships between the concepts and themes that surface from the analyzed text. relational analysis is popular because of its flexibility, but this flexibility can also be a drawback when reliability and trustworthiness are necessary in the research. when the analysis is too flexible, the research becomes impossible to replicate. therefore, a codebook, a coding form, rules, and often more than one coder are necessary to give stability to the content analysis process.   content analysis is a method commonly used in the social sciences and is therefore a viable choice for lis research. in fact, there are many published lis research studies that have used the content analysis methodology. i have selected a few as an illustration:   dahl, c. (2001). electronic pathfinders in academic libraries: an analysis of their content and form. college& research                 libraries, 62(3), 227-237.   du, y., stein, b., & martin, r.s. (2007) content analysis of an lis job database: a regional prototype for a collaborative model. libri, 57, 17-26. retrieved from http://www.librijournal.org/pdf/2007-1pp17-26.pdf   julien, h., mckechnie, l. & hart, s. (2004). a content analysis of affective issues in library and information science systems work [summary of a research note delivered at the isic 2004 conference, dublin, 1-3 september, 2004], information research, 10(1). retrieved from http://informationr.net/ir/10-1/abs6   koufogiannakis, d., slater, l., & crumley, e. (2004). a content analysis of librarian research. journal of information science, 30(3), 227-239. doi: 10.1177/0165551504044668   morais, y., & sampson, s. (2010). a content analysis of chat transcripts in the georgetown law library. legal reference services,29(3), 165-178.   yontar, a., & yalvac, m. (2000). problems of library and information science research in turkey: a content analysis of journal articles 1952-1994. ifla journal26(1), 39-51.   the basic steps a researcher takes in approaching a content analysis is as follows:   develop a research question (if using qualitative analysis) or a hypothesis (if using quantitative analysis). define the population. this step can take place at several places during the course of the study: at the research question development state or later in the process depending on what is returned from choosing a particular population. then there are different methods of choosing a sample: random sampling methods (simple random sampling, systematic random sampling, cluster sampling, stratified sampling) and non-random sampling (purposive sampling, convenience sampling). select a research design. there are several tasks to be accomplished in this step: choose units of analysis to study; create a coding scheme; and, in the case of quantitative analysis, develop a numbering system. gather data: quality control is a major concern when coding. agreement tests must be conducted between coders to insure acceptable levels of inter-coder reliability. the researcher must try to avoid subjectivity and the appearance of subjectivity. interpreting the evidence: findings must directly address the research question or hypothesis. tables, bar graphs, numbers, etc., all must be explained and interpreted in light of the question asked.  (adapted from beck & manuel, 2004, chapter 3)   in practice, content analysis can be “time-consuming and labour-intensive” (beck & manuel, 2004, p. 37). using this research method can help to reveal trends and themes, but it cannot attribute cause. however, it is one of the top research methods used in lis research, and can be just the thing when an analysis of multiple texts is required. coming up in the next issue, a look at using focus groups.   works consulted   course content page from the university of texas school of information. retrieved from http://www.ischool.utexas.edu/~palmquis/courses/content.html   hsieh, h-f., & shannon, s.e. (2005). three approaches to qualitative content analysis. qualitative health research, 15, 1277-1288, doi: 10.1177/1049732305276687   recommended reading   krippendorff, k. (2004). content analysis: an introduction to its methodology (2nd ed.). thousand oaks, ca: sage.   neuendorf, k. a. (2002). the content analysis guidebook. thousand oaks, ca: sage.   white, m.d., & marsh, e.e. (2006). content analysis: a flexible methodology. library trends, 55(1), 22-45, doi: 10.1353/lib.2006.0053   references   beck, s.e., & manuel, k. (2004). practical research methods for librarians and information professionals. new york: neal-schuman.   white, m.d., & marsh, e.e. (2006). content analysis: a flexible methodology. library trends 55(1), 22-45 doi: 10.1353/lib.2006.0053   news/announcements   eblip article wins 2016 jesse h. shera award      2016. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     the american library association library research round table selected, for the 2016 jesse h. shera award for distinguished published research, the publication “far from a trivial pursuit: assessing the effectiveness of games in information literacy instruction” by eamon tewell and katelyn angell. this paper was published in evidence based library and information practice (eblip), volume 10, issue 1 in 2015.   see the official announcement from ala and more information about the award at http://www.ala.org/news/press-releases/2016/03/tewell-and-angell-win-2016-jesse-h-shera-award-distinguished-published microsoft word news_practice.doc evidence based library and information practice 2007, 2:1  162 evidence based library and information practice     news    evidence in practice award for uk librarians       © 2007 eblip. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    the second evidence in practice award in  conjunction with the third uk clinical  librarian conference is now open for entries.  we hope to bring together case studies that  can be shared and used to persuade decision  makers of the value of library and  information services.    the second evidence in practice award is  sponsored by the national library for  health and bmj group clinical evidence. it  is open to any library & information  professional in the uk who, working with  health professionals, has made a difference  in patient care. you may be working in any  sector (e.g. primary or secondary care,  consumer health information) and at any  type of health organization (e.g. a  professional body or a voluntary  organization).    partnerships of information and health  professionals are invited to enter the  competition by providing a structured  abstract, describing (in approximately 750  words) how health information provided by  their library or information service has  impacted patient care or service delivery.    entries can be submitted up to march 31,  2007 after which anonymous case studies  will be judged by an independent panel  combining clinical and information  expertise. the judges’ decision will be final.    the winners will each receive a personal  digital assistant, financial support towards  attendance at a professional conference or  course of their choice, plus a free delegate  place at the third clinical librarian  conference.    the award will be presented at the third  uk clinical librarian conference, 11th &  12th june 2007, st williamʹs college, york  minster, where the award winners will have  an opportunity to share their example of  successful practice. for more details, follow  the link from http://www.uhl‐ library.nhs.uk/clinlibconf2007.htm or  contact     linda ward  library services manager  university hospitals of leicester nhs  libraries  education centre library, lgh, gwendolen  rd. le5 4pw. tel: 0116 2588124 fax:  0116 2588078  medical library, gh, groby rd. le3 9pq.  tel: 0116 2502303 fax: 0116 2563334  http://creativecommons.org/licenses/by/2.0 http://www.uhl%e2%80%90library%00 evidence based library and information practice 2007, 2:1  163   email: linda.ward@uhl‐tr.nhs.uk  or lmw12@le.ac.uk    please take a short while to reflect on how  you have made a difference and share it  with your professional colleagues.    on behalf of the eip award / cl conference  team. news/announcements   evidence based library and information practice releases special issue on ebl 101      2016. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     the journal evidence based library and information practice (eblip), now in the 11th year of publication, released its first special issue: an issue dedicated exclusively to the ebl 101 column, which ran in the journal from 2008 to 2015. the special issue is a compilation of all 24 columns published during this period, in their original form, in one place. the special issue also includes an editorial by virginia wilson, author of the vast majority of the columns.   the special issue was released on march 4, 2016 as an anniversary gift to the readers of eblip, past, present, and future. it is available on the journal website at https://ejournals.library.ualberta.ca/index.php/eblip/issue/view/1571. as with all journal content, it is available open access. evidence summary   public library users are challenged by digital information preservation   a review of: copeland, a. j. (2011). analysis of public library users’ digital preservation practices. journal of the american society for information science and technology, 62(7), 1288-1300. doi: 10.1002/asi.21553   reviewed by: robin e. miller assistant professor and research & instruction librarian mcintyre library university of wisconsin – eau claire eau claire, wisconsin, united states of america email: millerob@uwec.edu   received: 30 nov. 2012 accepted: 22 jan. 2013      2013 miller. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to discover the factors that influence digital information preservation practices and attitudes of adult public library users.   design – mixed methodology combining matrix questionnaires, interviews, and visual mapping.   setting – urban public library on the east coast of the united states.   subjects – 26 adult members of a public library’s friends group.   methods – the researcher conducted semi-structured interviews with 26 participants. all participants drew maps to indicate the types of information they value and why, and their preferences for information storage and maintenance. qualitative data were supplemented by a matrix questionnaire on which 22 participants identified the types of digital information they maintain, and modes of storage.   main results – some public library users may store and organize information inconsistently, utilizing a variety of digital devices. technical, social, and emotional context influences choices about organization, sharing of information, and shortand long-term preservation. users reported placing a higher value on born digital information, and information that they had shared with others.   conclusion – public librarians may have a role in facilitating growth of patron knowledge about creation, storage, preservation, and sharing of personal digital information.     commentary   while corporate and academic libraries have investigated the long-term storage and preservation of their patron and institutional data, the author makes a compelling case that we know less about the everyday digital information preservation practices of individual public library users.   designed to validate a qualitative inquiry, the author’s methodology offers a time-intensive model for inquiry about information use and preservation. the author carefully explains strategies employed to validate the results. interviews were transcribed, coded, and analyzed with nvivo 7.0. to ensure coding consistency, two independent judges evaluated the coding of a sample of the interview transcripts. the researcher used an instrument developed by savolainen and kari (2004) to evaluate the maps drawn by each participant.   the author outlines three questions about public library users and the characteristics of the digital information they maintain, their motives, and the factors that influence their choice to preserve digital information. all participants reported storing digital photographs, email, music, and word processed documents using a variety of tools, with varying degrees of technical problems and challenges. while the author suggests that public libraries could become more involved in patrons’ private computing, studying participants who own a computer curtails the library practitioner’s ability to apply findings about participant behaviour to public library users in general.   the study’s most notable finding is that users most valued digital information they had shared with others, and information sharing was influenced by the emotional connection represented by the file. emotional attachments to information strongly influenced participants’ rationale for preserving files for the long-term, and the social and emotional context of information creation influenced choices about organization, naming conventions, and sharing.   the validity of this research is limited by the characteristics and size of the participant pool. the author offers no justification for recruiting volunteers from the friends of the library, a decision which yielded an unrepresentative sample of public library users in general. in addition, this research does not explore how often or how participants used the library, or whether subjects desired library instruction about digital information preservation. in light of these factors, the author cannot generalize about public library users’ “everyday life” modes of digital information creation or storage, common motives for preserving information, or how public librarians might intervene to enhance users’ skills and expertise.   librarians might hope that the outcome of this research would inform the development of group, individual, or point-of-need instruction. the author suggests that public library patrons may benefit from instruction or consultation about methods of prioritizing, storing, and organizing digital files. however, this research would be more useful to practitioners if it offered evidence that the diverse population of public library users needs and desires library instruction and services about digital information management.     references   savolainen, r., & kari, j. (2004). placing the internet in information source horizons: a study of information seeking by internet users in the context of self-development. library & information science research, 26(4), 415-433. doi: 10.1016/j.lisr.2004.04.004   microsoft word es_mckenna.doc evidence based library and information practice 2007, 2:3  108 evidence based library and information practice       evidence summary    danish post‐secondary students use public libraries for study purposes    a review of:  pors, niels ole. “the public library and students’ information needs.” new library world  107.1226/12272 (2006): 275‐85.    reviewed by:   julie mckenna  librarian, university of regina  regina, saskatchewan, canada  e‐mail: juliemileskatie@yahoo.ca        received: 01 june 2007     accepted: 29 july 2007      © 2007 mckenna. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to determine whether and how  danish university and higher education  students use public libraries for study  purposes.    design – online survey.    setting – post‐secondary programs in  denmark.    subjects – 1,575 students in university‐level  programs or other higher education  programs (vocational three‐to‐four‐year  programs) in denmark.    methods – a sample of students was drawn  from the national database of students by  selecting every student born on the 15th of  every month (approximately 4,900 students).   a letter describing the study and with an  invitation to fill out an online questionnaire  was sent to all students in the sample.  there  were 1,694 valid responses (approximately  35% response rate).  students following  short vocational programs were deemed to  be under‐represented and these subjects  were omitted from the analysis of this report,  which reflects the response of 1,575 students.    the online questionnaire gathered  demographic details (gender, age,  educational institution, study topic, study  year, geographical location, access to the  internet, etc.) and used 110 questions or  statements to gather information about  student information‐seeking behaviour  related to study purposes.  these included  use of the physical library and satisfaction  with services, use of search engines,  awareness and use of library web‐based  services, study behaviour, and participation  in information literacy activities.  mailto:juliemileskatie@yahoo.ca http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:3  109   main results – for the purposes of this  study, “academic library is used as a generic  term covering university libraries, research  libraries, educational libraries and all other  kind of libraries outside the field of public  libraries” (p. 278).      the survey results confirmed many of the  previous international reports of student  information‐seeking behaviour: 85% of  students use the academic library for study  purposes; fewer than 10% of all students are  able to cope without any library use;  students in technology and engineering, the  sciences and arts, architecture and music  have a higher rate of non‐use of their  academic libraries; a large percentage of  students access the electronic resources from  home; the physical library is still considered  important to students; google is used  extensively and is nearly the exclusive  choice for search engine.    the public library is used for study  purposes by about 58 percent of all students  with the highest use (76%) by students in  higher education institutions (hei); students  of education, social topics and psychology  are very frequent public library users.   female students in hei were the most  frequent users of the public library  independent of study subject or year, or any  other demographic variable.  seven per cent  of students rely exclusively on the public  library for study purposes and first‐year  hei students in the subject areas of  education, social topics and psychology are  over‐represented in this group (which  additionally has less internet access from  home than the other students).    students perceive nearly all aspects of  service in the academic library as superior;  hei students rate ambience, electronic  resources and speed of inter‐library loan  provision in the public library as higher  than the academic.    university students give a low rating to the  collections of public library, although the  students use the public library principally to  supplement the collections available in their  academic libraries.  another high use of  public libraries by hei students is for inter‐ library loans placed through the national  resource sharing system.  public library  reference services are used often by only one  per cent of students and only two per cent  use the public library on a regular basis for  “study related group activities.”    conclusion – students use physical libraries  to a great extent to support their studies and  students have embraced digital access to  collections, especially access from home.   google is the most heavily used search  engine and is used by nearly all students;  use of google complements and  supplements library use.      nearly 60% of all students use public  libraries for study purposes and to  supplement the collections of their academic  library, but they find that the public library  collections are insufficient to meet their  needs.  the inter‐library loan policies of  public libraries are more lenient and  accommodating to student needs and may  drive the high use of public libraries.   students form a large constituency of the  public library user population and they  generally rate most aspects of service as  substandard to those of academic libraries.   there is a call for review of the public  library’s role in meeting the information  needs of students, and in particular, those of  hei programs who are most dependent on  the public library.    commentary    this study is an ambitious attempt to define  the use of public libraries by post‐secondary  students in denmark and will be of interest  to both public and academic librarians.  the  evidence based library and information practice 2007, 2:3  110 information policy environment in denmark  is described in this article, and this provides  a framework about the nature of library  services for danish citizens and the  country’s tremendous national resource  sharing environment.    the study considers post‐secondary  education student use of libraries in  denmark and raises questions about how to  meet the needs of students within the public  library system.  these questions are  important considerations for all who strive  to serve students in a seamless environment  of digital and traditional library services  that requires cooperation across library  sectors.    the research design and methodology of the  study are well described.  the seemingly  simple construct of “study purposes” was  not defined in the study and it would be  beneficial, especially given the international  context, to learn the scope of definition. the  report refers to results as “actual  behaviour,” yet the study utilizes an online  survey through which students self‐report  their behaviour.  without access to the  questions and statements utilized in the  instrument, it is difficult to assess the  potential biases in the survey response.    the percentage of students indicating  regular public library use is very high; it  would be very interesting to pursue this  study in other international regions to  determine whether the same rates of public  library use exist among students elsewhere.    this research provides an insight into  student use of public libraries in denmark  and is an important exploratory study, the  results of which could contribute to further  understanding about student use and  expectations of public libraries.        microsoft word art_price.doc evidence based library and information practice 2007, 2:3  43 evidence based library and information practice     article    using a prompt sheet to improve the reference interview in a health telephone  helpline service      toni price  national institute for health and clinical excellence  manchester, uk  email: toni.price@nice.org.uk    christine urquhart  department of information studies  university of wales aberystwyth  llanbadarn campus  aberystwyth, uk   email: cju@aber.ac.uk    janet cooper  truro college  college road  truro, uk  email: janetcooper@trurocollege.ac.uk      received:  15 may 2007     accepted: 19 july 2007      © 2007 price, urquhart, and cooper. this is an open access article distributed under the terms of the  creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits  unrestricted use, distribution, and reproduction in any medium, provided the original work is properly  cited.    abstract    objective ‐ the study examined whether a prompt sheet improved the reference interview  process for health information advisers working at nhs direct, a 24‐hour telephone  helpline that provides confidential health care advice for the public in england.    methods ‐ a randomised control trial was conducted at eight nhs direct sites across  england in 2003‐04. newly recruited health information advisers (n=30), full and part‐time,  were randomly allocated to a control group (n=15) or intervention group (n=15), and 26  mailto:price@nice.org.uk mailto:cju@aber.ac.uk mailto:janetcooper@trurocollege.ac.uk http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:3  44 completed the study. existing health information advisers were involved in the planning  and design of the intervention. the prompt sheet included prompts for demographic  information, reason for call, condition/treatment plan, existing knowledge of caller, special  needs of the caller, handling a call empathetically, and conclusion. testing of reference  interview expertise was done at the end of basic training, and two months later, using the  same questions. the ten test questions were based on common questions received by nhs  direct. a relevance framework for possible responses was drawn up for each question for  scoring test responses, with more relevant responses scoring higher than less relevant  responses.    results ‐ the average score of prompt (experimental) and non‐prompt (control) participants  increased on the second test, for each of the 10 questions. the prompt group improved  significantly more overall than the control group. there was variation within the groups.  sixteen health information advisers showed a net increase in their score over all ten  questions (10 experimental group, six control group). the post‐test score for an individual  on a particular question did sometimes decrease from the pre‐test score, but all 26 improved  on at least one question. previous call handler experience did not appear to influence the  extent of improvement, but length and type of experience in the post may have an influence.    conclusion ‐ the trial demonstrated the benefit of a simple and inexpensive prompt sheet  for some, though not all, newly recruited health information advisers to improve their  reference interview technique.     introduction    studies of reference interviewing may  consider a variety of perspectives –  theoretical principles of the process,  expression of inquirer needs, setting, subject  scope variations and training requirements.  from the theoretical perspective, several  studies have re‐examined the questioning  process within the reference interview  (tuai), the role of the librarian – as expert or  therapist and non‐expert (stover), or simply  as equal with the inquirer (mabry).  strategies used in the process of questioning  could follow the principles of the cognitive  interview (moody and carter). others have  challenged the ‘funnelling’ process and note  the problem of premature closure (ross and  dewdney). the theoretical principles of the  reference interview may be drawn from  speech act theory (searle) as suggested by  dewdney and mitchell, to understand the  function of what is being said. alternatively,  interview questions may be based on  reflective listening, the empathic responding  techniques as practised in counselling and  psychotherapy (nelson‐jones). basic  ‘attending skills’ were adapted by jennerich  for use in librarianship and the 12 skills for  successful reference interviewing include  nonverbal skills (eye contact, gestures,  posture, facial expression and tone of voice)  and verbal skills (remembering, avoiding  premature closure, reflection, paraphrasing,  using encouragement, closing, giving  opinions or suggestions, asking open  questions) (jennerich and jennerich). for  telephone interviews, the visual cues are  absent and there is more emphasis on the  voice, the tone, and a protocol for referral to  other experts or for calling back (ross,  nilsen and dewdney 126).     taylor referred to the levels of expressed  information need of the inquirer as visceral,  conscious, formalized and compromised.  for services that provide health information  to the public, the theories of information  behaviour that seem particularly pertinent  are the sense‐making theory of dervin and  models of the information search process  that recognise that the inquirer may stop  evidence based library and information practice 2007, 2:3  45 temporarily at a particular stage (e.g. in the  formulation stage of the six‐stage process of  information seeking described by kuhlthau).  wilson and walsh stress the emotional  aspects of information seeking and this is  important for health information seeking,  when inquirers may not wish to obtain all  the relevant information at once, preferring  to wait until they feel they can cope with the  facts (baker).    recent discussion in the literature on setting  considers how virtual reference services  operate in chat or e‐mail environments (dee;  diamond and pease; lee; ross, nilsen and  dewdney 185). in academic libraries,  reference services may now be reorganised  to provide a variety of services to meet, cost‐ effectively, the different needs of users  (simmonds). the lack of desired  functionality and management statistics  may make use of commercial electronic  reference software problematic for some  universities acting in a consortium (bains).  an alternative approach is to redesign the  home page of a library service, so that it can  perform some of the functions of the  reference interview (bowman) particularly  for students who may use the internet as  their first point of access for information.     the subject scope of the interview may  influence the organisation and support  required. in the health sector, a well  prepared search strategy is required. for  evidence‐based practice and clinical queries,  the pico set of structured questions has  been developed (straus et al.). pico stands  for patient and/or problem, intervention,  comparison intervention and clinical  outcomes. for clinical queries this type of  structuring should work more effectively for  searching in databases such as medline,  but this may depend on the type of  information the user is seeking. a  comparison of a pico structured reference  interview form compared to a minimally  structured form, in a before and after study  in six multidisciplinary health libraries in  england (n=185 pico requests, n=195  minimally structured requests) found that  the ebm structured forms were associated  with fewer items retrieved, and more  precise search results (booth, o’rourke and  ford). the librarians generally preferred the  minimally structured form, and in fact  demanded that the pico structured form  include a free text box. for health policy and  management information, a looser structure  may be necessary, and a new mnemonic has  been proposed as a search aid, eclipse,  which stands for expectation, client group,  location, impact, professionals, service  (wildridge and bell).     as far as training is concerned, most  manuals on reference interviews include  ‘how to do it’ recommendations on training  staff in reference interview skills, and some  of this advice is based on evidence. ross,  nilsen and dewdney (226) cite evidence that  verification and follow‐up increase the  success of the reference transaction, and that  the microskills training advocated by  jennerich helps librarians to ask more open  and sense‐making questions. the habit  hardest to break may be that of asking too  many closed questions.     barriers to a successful reference interview  may be related to expression of the question,  and inquirers may be particularly reticent  when discussing personal health topics.  there are many reasons why such inquiries  may not be successful for the inquirer (table  1, synthesising literature from (mainly)  baker; ross, nilsen and dewdney; taylor).   the reference interview requires the  librarian to demonstrate the following  attitudes:  • friendliness and approachability  • courtesy, being polite  • willingness to listen  • patience  • sensitivity to inquirer’s needs  evidence based library and information practice 2007, 2:3  46   problem  notes  implication  ask undemanding question,  not related to need  may relate to lack of  confidence in the librarian, or  viability of the question  real need not expressed  question expressed poorly  may not have sufficient  vocabulary to express the  problem precisely, or,  conversely, use too much  jargon  real need not expressed  precisely and completely  lack of ease in asking  question about personal  sensitive issues  health topics particularly  problematic  requires skills on the  librarian’s side to establish  appropriate empathy  unwillingness to reveal  reasons for needing  information  health topics particularly  problematic  may not be able to match need  with appropriate information   question may be too broad  inquirer may feel more certain  that the answer will be  obtained   danger of overload  question may be too vague  inquirer may be at too early a  stage in the information  seeking process  may require time for  negotiation of question  inquirer feels a ‘failure’ for  having to ask   inquirer may be aggressive or  apologetic  requires skills on the  librarian’s side to establish  appropriate empathy    table 1. inquirer problems in the reference interview      • confidence  • persistence  • sixth sense – in knowing that  something is not quite right (grogan;  ross, nilsen and dewdney).  the evidence for the negotiation process in  reference interviews indicates the following  types of question, for particular purposes:  • open questions (to encourage  inquirer to expand)  • neutral questions (type of open  question ‘tell me more about…’ to  help check the focus  • closed questions (when a specific  answer is required, e.g. whether  illustrations are required or not)         (grogan; jennerich and jennerich; ross,  nilsen and dewdney).    reflective listening, or active listening,  employs the following techniques to ensure  empathic understanding:  • paraphrasing, to check that  interpretation by the listener is  correct  • summarising, at the end, to check  that the entire question has been  answered, and that the  interpretation is correct.  • use of silence, to enable the inquirer  to reflect themselves, without  interruption (jennerich and  jennerich).    evidence based library and information practice 2007, 2:3  47   problem  notes  implication  question accepted as face  value  question, although  convincing, may need further  checking  inquirer’s real needs not met  librarian appears  unapproachable (queues)  may require some service  reorganisation  inquirer does not use the  service again  intrusive questions  questions need to appear  relevant  inquirer may cease to co‐ operate with the negotiation of  the question  lack of privacy in the setting  sensitivity to needs important  inquirer may not provide  sufficient information to  resolve question  premature closure  (false  focus, kennedy, cole and  carter)   time pressures may lead to  inadequate reflective listening.  partial answer at best    table 2. librarian problems in the reference interview      for telephone interviewing for health  information it is important that a shared  understanding is achieved between inquirer  and the service provider (table 2, based  mainly on jennerich and jennerich; ross,  nilsen and dewdney; taylor).     in summary, the guidelines for reference  interviewing are based on some, often old  evidence. some of the guidelines and theory  have been revisited when designing, and  evaluating telephone and virtual reference  services, but there is a gap in the evidence  on the type of ongoing support required for  reference inquiry staff. the purpose of this  study was to examine how new recruits to a  large telephone helpline service could be  supported after initial training on reference  interview skills, to ensure a high quality of  service to the public.     setting for study  nhs direct is a national 24‐hour nhs  (national health service) telephone helpline  providing confidential advice and  information to the public in england. calls  are charged at the local call rate, and there is  an interpreting service for those whose first  language is not english. there is a  complementary online service, nhs direct  online. an evaluation of nhs direct by the  national audit office found that nhs  direct was operating safely and that advice  to callers errs on the side of caution. nhs  direct was encouraging inquirers to make  more appropriate use of nhs services  (national audit office).    when a call comes into the service, the call  handler (assistant health information  adviser) uses a screening protocol to  immediately detect callers who need an  emergency response. the remaining callers  are divided into those who need nurse  assessment (due to symptoms mentioned)  and those who have an information need.  around 80% of calls involve symptoms and  are therefore routed to a nurse. the  remaining 20% may be referred to a health  information adviser, if the assistant health  information adviser is unable to deal with  the inquiry. typically, the assistant health  information adviser will provide details  about location of doctors or dentists. the  evidence based library and information practice 2007, 2:3  48 full reference interview process starts with  the health information adviser, and in some  cases the inquiry may be referred to an  information team. if, during the course of  the interview with the caller, the health  information adviser judges that nurse  assessment is required, the call will be  transferred to a nurse.     this study focuses on health information  advisers who come from a range of  backgrounds, with no requirement for  previous experience of working in the nhs.  there are no minimum qualifications other  than a ‘good general education’ required for  this role. all receive training before starting  to answer calls, but the extent of this  training varies from region to region. some  regions provide mentoring support, or  refresher training, and the format of the  advisers’ work varies    study aims and objectives  calls to the nhs direct service continue to  increase, resulting in pressures on the  advisers to respond quickly but effectively.  there are four ways in which the health  information adviser’s techniques could be  enhanced: 1) extending the call assessment  system to include information triage, with  on‐screen prompts to assist the adviser; 2)  prompt sheet used manually; 3) specific and  specialist group training; and 4) individual  training. in practice, option 1) would be  costly to design and deliver, and options 3)  and 4) are part of ongoing staff development.  the views of the health information advisers  were obtained using a questionnaire (n=268,  response rate 88/268, 32.8%), and the most  popular option was refresher group training  (option 3) followed by the prompt sheet  (option 2). although the prompt sheet was  not the most popular option, it was the  simplest option, and one which could be  trialled easily. the objective was therefore to  trial the prompt sheet in a randomised  control trial of new recruits, to assess  whether it was effective in enhancing the  reference interview technique of the  advisers. the research work contributed to a  dissertation for an msc(econ) in health  information management for the researcher  (tp); the other authors are the course leader  (cu), and the dissertation supervisor (jc).    sample  at the time of the study (2003‐2004) the  structure of nhs direct was changing, and  that meant that no single site was recruiting  sufficient new health information advisers  to do a trial at one site, thus minimising  potential variables in site support.  recruitment to the study was done over a  period of months, in three waves, with 30  new health information advisers recruited  voluntarily from eight sites. random  allocation was by a sealed opaque envelope  type of system, allocation proceeding site by  site. two participants withdrew due to ill  health, one left for other employment, and  one was excluded on grounds of previous  employment as a health information adviser.  new advisers were judged to have the most  potential for improvement in reference  interview techniques and the trial was  therefore limited to new advisers. it was not  possible to exclude those who were working  part‐time (n=7 part‐time staff completed the  study) or might have had previous  experience within nhs direct in a call‐ handling role as an assistant health  information adviser (n=5 experimental, n=10  control, had been call handlers). where  possible, the recruitment, pre‐ and post‐ testing was done in person by the researcher  (tp), but some post‐testing had to be  delegated, under instruction, to the health  information managers on site (table 3).   participants withdrew from site 4 (one), site  7 (one) and site 8 (two withdrawals), leaving  26 who completed.    ethics  the aims and objectives were explained to  the eligible health information advisers,   evidence based library and information practice 2007, 2:3  49   site number  allocation to  prompt sheet  (n=15)  allocated to control  group  (n=15)  visits by researcher  1  4  2  both pre‐ and post‐test  2  3  1  both pre‐ and post‐test  3  1  2  several visits, pre‐test only  4  0  4  none  5  0  1  none  6  2  1  pre‐test only  7  2  2  both pre‐test and post‐test  8  3  2  pre‐test (one participant tested later)  table 3. site and participant allocations  noting that the researcher had to identify the  responses to allow comparison of pre‐ and  post‐responses for the analysis. however,  no respondent was identifiable outside the  study. as this was categorised as an audit  for nhs purposes, full nhs research ethics  approval was not required. the study  followed the ethical guidelines of the  department of information studies,  university of wales aberystwyth.     methods  a prompt sheet was devised by the  researcher (tp) by first considering the  stages of a call, and the type of questions,  and subject areas that should be covered.  the prompt sheet was piloted with a group  of six health information advisers, of  varying experience. the revised sheet was  circulated to five experienced health  information advisers, two of whom were  involved with the pilot, and minor  amendments made.    the prompt sheet reminded the advisers to  check:  • what is the caller’s inquiry (specific  illness, diagnosed/suspected or a  worry, tests/surgery – done or  planned), drugs (dosage, period  taken), complementary therapies?  • who is the call about (self, partner,  family, who else is involved – gp,  social worker etc)  • how (recent or longstanding is the  illness, how much is already known,  how did they find out?)  • why (have they called today –  specific need?)  • outcome (to read, send, further  research required?) remember to  quote sources  • organisations (phone number,  address, website)  • special needs (urgent, and by when,  particular requirements)  • conclusion (restate what nhs  direct will do now (and/or when)    the same questions were used at pre‐test  (after initial training but before taking calls  from the public) and post‐test. questions  were designed to be sufficiently diverse to  capture the required range of expertise, and  ten questions were devised, with the help of  the six health information advisers in the  pilot. the assessment matrix comprised pre‐ determined categories and content of  responses (highly relevant, relevant,  potentially relevant, minimal relevance) and  responses outside that list had to be  considered on their merits. the question  evidence based library and information practice 2007, 2:3  50 topic chosen to pilot the assessment matrix  was a common inquiry topic, asthma, and  the matrix (and scoring system) examined to  ensure that the questions (and scoring  framework) could differentiate performance.  emphasis was placed on the type of  information the adviser would need to  obtain. the responses from the six advisers  in the pilot could be allocated among the  relevance categories, and the majority of the  advisers’ responses fell into the highly  relevant category, as expected.     the questions chosen for the pre‐ and post‐ test were based on experience of the type of  topics received at nhs direct and the type  of detail initially offered by an inquirer. the  type of calls (about drugs, meningitis,  treatment options, and cancer) would be  covered in training, and novice health  information advisers should provide  adequate responses.  the ten questions  chosen were:  1. my friend has a breast lump and is  worried  2. i have been told my child’s skin  problem is eczema and i need more  information  3. i saw the gp about some  menopausal symptoms but am  worried about taking the hrt he  prescribed  4. a work colleague has been  diagnosed with meningitis and i  want to know more about the illness  5. my teenage son has acne, and i’d  like to know more about treatment  6. i’m taking prozac and want to know  more about it  7. i’d like some information on knee  replacements  8. i have cancer  9. i’m on warfarin  10. the gp is going to arrange tests    the five experienced health information  advisers were asked for comments on the  ten questions, but no suggestions were  received. the scoring system gave higher  scores to highly relevant responses, but the  total possible score varied with each  question as the number of questions that  could be asked varied, as did their relevance.  for every question, existing knowledge,  specific concern, and information need  should be assessed, and status of diagnosis  is usually very important. callers may be  asking for information for themselves,  although posing the question as for a friend,  and it is often potentially relevant to  establish that. table 4 provides an example  score framework for question one.  in the pre‐test, the experimental and control  group participants were given as much time  as they wanted to answer the questions, up  to a 25 minute maximum for all ten  questions in a written test. after testing, the  advisers were divided into their groups, and  the control group returned to their duties,  while the experimental group was given  brief instructions on the prompt sheet, in  which they were requested to consult the  prompt sheet regularly, and notified that  they would be asked to complete an  evaluation sheet on the use of the prompt  sheet at the end of the study. the  experimental group subjects were asked not  to discuss or share the prompt sheet with  other staff. in practice, it was difficult to  assess whether compliance with use of the  prompt sheet was thorough, or whether the  experimental groups shared the prompt  sheet with the control group. the same  questions were asked at post‐test,  approximately two months later, under the     evidence based library and information practice 2007, 2:3  51 highly relevant  (score 3)  relevant   (score 2)  potentially relevant  (score 1)  minimal relevance  (score ‐1)  diagnosed?  type  self or other  medicines  gp/consultant  involvement  gender  age  other medicines  symptoms  treatment/care plan  co‐morbidity  menstrual cycle  duration of  episode/timescales  medical history  prognosis knowledge    tests  family history  support    existing knowledge        specific concern        information need        possible score =24  possible score = 10  possible score =5  total possible score =  39  table 4. assessment matrix for question one      same timing and test conditions as the pre‐ test.    results  results are presented for the 26 participants  who completed the trial. for the pilot  asthma question, the average score was 12.3,  and the question scores in the trial were  similar.     changes in overall performance   the average scores obtained across pre‐test  and post‐test, and across all questions  ranged from 7.9 to 14.4. on average, scores  improved from pre‐test to post‐test, and  without exception the average scores  improved for each question (table 5). the  mean score per question at pre‐test was 9.6  (standard deviation 1.66), and the mean  score post‐test was 11.1 (standard deviation  1.61). the post‐test and pre‐test scores were  significantly different (t‐test, t=12.5, dof =9,  p<0.01). making a more conservative  assumption that the t‐test conditions of  normal distribution are not satisfied, and  that non‐parametric testing should be  applied, the pre‐test and post‐test scores  remain significantly different (wilcoxon     signed ranks test, t=0, n=10, p<0.01, one‐ tailed test).    differences between the experimental group  (prompt) and control group were examined  in various ways and are described below.  the experimental group improved more  than the control group on eight of the ten  questions. the control group improved  more than the experimental group on only  two questions (question 4 and question 10)  and the difference for question 10 was small  (table 6). question 5 showed the most  improvement of all the questions and the  majority of the improvement was from the  experimental group. the experimental  group’s overall improvement on the  questions was significantly better than that  of the control group (wilcoxon signed ranks  test, t=4, p<0.05, two‐tailed test). to assess  whether the improvement was due to an  increase in the number of potentially  relevant responses, at the expense of the  highly relevant responses, the differences in  total ‘highly relevant’ scores of the  experimental group and the control group  were compared. over ten questions, the  experimental group’s score for highly     evidence based library and information practice 2007, 2:3  52     pre‐test  post‐test  difference  q1  breast lump  10.4  11.7  1.3  q2  childhood eczema  10.7  12.3  1.6  q3  hrt/menopause  9.1  10.1  1.0  q4  meningitis  8.2  9  0.8  q5  teenage acne  7.6  9.7  2.1  q6  prozac  10.3  11.9  1.6  q7  knee replacement  7.9  9.5  1.6  q8  cancer  13.2  14.4  1.2  q9  warfarin  9.1  10.8  1.7  q10  tests arranged by gp  9.3  11.2  1.9  table 5. average scores for each question, pre‐test and post‐test            table 6. difference in increase in scores at post‐test    relevant responses increased from 1083 to  1272 (an increase of 189 points, equating to  63 additional aspects noted) and the control  group increased from 1110 to 1215 (an  increase of 105 points, or 35 additional  aspects). for the relevant responses, the  score for the experimental group changed  from 102 to 154 (an increase of 52 points, or  26 additional aspects noted), and the control  group increased their score from 116 to 128  (an increase of 12 points, 6 additional  aspects).         changes in individual performance  sixteen health information advisers showed  a net increase in their score over all ten  questions, and ten of these were from the  experimental group, six were control. the  post‐test score for an individual on a  particular question did sometimes decrease  from the pre‐test score, but all 26 improved  on at least one question. fifteen (9  experimental, 6 control) improved at post‐ test on five or more questions.             experimental (prompt)    control  q1  breast lump  22  11  q2  childhood eczema  30  13  q3  hrt/menopause  19  8  q4  meningitis  5  14  q5  teenage acne  49  6  q6  prozac  34  7  q7  knee replacement  36  4  q8  cancer  19  11  q9  warfarin  28  18  q10  tests arranged by gp  23  26  evidence based library and information practice 2007, 2:3  53   hia #  prompt  (experimental)/control  ft/pt  previous  call‐ handler?  net  score  increase  site  4  prompt  ft  n  75  1  14  control  ft  y  67  4  18  control  ft  y  52  6  28  prompt  ft  y  50  8  10  control  pt  y  42  1  8  prompt  pt  y  41  3  6  prompt  ft  n  38  2  3  prompt  pt  n  31  1  1  prompt  ft  n  25  1  11  control  ft  n  18  2  25  control  ft  y  16  3  5  prompt  ft  n  10  2  15  control  ft  y  10  4  7  prompt  ft  n  8  2  29  prompt  ft  y  5  8  17  prompt  ft  y  2  6    table 7. characteristics of the 16 health information adviser (hia) ‘improvers’      the group was too small to permit proper  statistical analysis of some of the factors that  might also affect improvement. for example,  some of the participants had previous  experience as a call handler, and some  would have more time to practise as they  worked full‐time rather than part‐time. of  the sixteen ‘improvers’ who had shown a  net increase in score over ten questions,  seven had not had previous call handling  experience, and nine had (table 7). a mann‐ whitney u test indicated that there was no  significant difference (n1=7, u1=29.5, n2=9,  u2=33.5) between those with previous call  handling experience and those without. a  similar mann‐whitney u test for the effect  of full time or part‐time was just significant  (n1=13, u1=6, n2=3, u2=13, p <0.05) as u=6 just  meets the level for significance. while  plausible, this would need further testing  with a larger sample. the data indicated  that some sites were not represented among     however, only one participant was  recruited from site 5. for site 7, it is possible  that there were some site‐specific conditions,  as all three of the participants there were in  the top scorers at pre‐test, but none were in  the top 16 improvers.  the characteristics of the 22 of the 26  participants were examined to assess  whether there was a possible ceiling effect  operating among some of the participants  (table 8). this would occur if participants  were already skilled to a high level after  initial training, and might be indicated by  high pre‐test and post‐scores, but no  improvement. only four health information  advisers (hia 5, 17, 21 and 22) fall into this  group. another four (hia 6, 10, 11 and 14)  scored well at pre‐test, post‐test and were in  the top 13 improvers. five (hia 7, 16, 19 20,  and 30) feature only in top pre‐test group,  but do not appear to have improved much.     evidence based library and information practice 2007, 2:3  54   hia#  top 13  pre‐test  top 13  post‐test  top 13  improvers  1      y  3    y  y  4    y  y  5  y  y    6  y  y  y  7  y      8    y  y  10  y  y  y  11  y  y  y  14  y  y  y  15      y  16  y      17  y  y    18    y  y  19  y      20  y      21  y  y    22  y  y    25      y  28    y  y  29      y    table 8. characteristics of individual changes among health information advisers (hias)      five were in both the top group at post‐test  and among the top improvers. four of these  had received the prompt instructions.     discussion  the findings of this study indicate the  effectiveness of the prompt sheet in  improving the reference interview skills of  new recruits to the telephone helpline  service. several of the training methods  advocated for reference librarians, such as  peer coaching (ross, nilsen and dewdney)  or class training are costly in terms of staff  time. in the nhs direct setting, initial  training followed by use of the prompt sheet  appeared successful in improving the  quality of the reference transactions. the  results also suggest that there are individual  differences as some staff did not improve,  whereas others improved significantly. this  individual variation, together with the small  sample size, limits the generalisability of the  findings. some of this improvement may be  a result of factors beyond the control of the  study such as discussions with managers,  following an audit of calls. experience may  help, but the prior experience of being a call  handler did not seem to affect the scale of  improvement, although more experience in  the health information adviser post at nhs  direct may be an influence. however, the  study did not measure the actual number of  hours worked by the participants, nor was it  possible to assess the type of inquiries that  were handled by the participants over the  period between pre‐test and post‐test. the  impact of the prompt sheet intervention  may be limited to a short period after initial  evidence based library and information practice 2007, 2:3  55 training. another factor that needs to be  taken into consideration is the degree of  flexibility in duties, as health information  advisers may sometimes help out with call  handling duties. the main limitation, as  already mentioned, was the small number of  participants.    the final version of the prompt sheet was  structured but not as detailed as earlier  versions and this may have contributed to  the success of the intervention. other  studies (booth, o’rourke and ford;  wildridge and bell) also indicate that  librarians may prefer a looser structure,  rather than following a rigid protocol for the  reference inquiry.     performance at a test may not be a good  judge of performance in routine inquiry  work, but to test otherwise would have been  very difficult. sites for nhs direct are  geographically dispersed, and observation  would have been time consuming. in  addition, there is no guarantee that  observed performance is identical to  performance when not observed.  comparison of performance against  different questions, from different inquirers  would be unreliable. on the other hand,  supplying written answers is not the same  as conducting a reference interview, and  some health information advisers may be  better, or worse at the interaction, and  sequence of the interview, moving from  response to next question, than their written  responses might indicate. the pilot test,  with the experienced health information  advisers, indicated that the score would  very rarely exceed one third of the total  possible score.     the ten questions used were typical of nhs  direct inquiries. there seems no particular  reason why question 5 (on acne) should  have shown the most improvement in the  quality of response, but there was sufficient  range in responses to suggest that the  questions were effective in differentiating  performance. the questions and scoring  worked as might be expected for the range  of skills among the participants. even if  some participants were already ‘highly  skilled’,  the questions and scoring helped to  identify overall improvement (or not), and  the different weighting given to highly  relevant responses helped to differentiate  between good and very good performances.     research indicates that those seeking health  information vary in their attitudes towards  the timing and the quantity of information  required. some want to know everything as  soon as possible and others prefer not to  know everything immediately (baker). the  responses to the questions were scored on  the assumption that most inquirers wanted  a full response, although some information  is often sent on later. an audit of calls  would be necessary to decide whether some  callers genuinely do not want a full response  at the time, and how health information  advisers can assess when the inquirer has  obtained sufficient information for their  particular needs. by sending more  information separately, the health  information adviser provides a compromise  that may work well.    conclusion  the findings of the study are particularly  relevant to call centres or health information  services that handle a large number of  inquiries. the findings indicate that a simple  prompt sheet helps most health information  advisers to structure their reference  interviews with clients, to ensure that all  relevant aspects of the inquiry are covered.  the prompt sheet is not a substitute for  initial training, but probably helps to  consolidate training. there were some  indications that there are some differences  in practice among the nhs direct sites and  these may affect the degree of improvement  that may be obtained for some individuals.  although the experimental group’s  evidence based library and information practice 2007, 2:3  56 improvement was significantly better than  that of the control group, the findings in no  way indicate that individual performance is  consistently better. there were differences  within the groups that suggest that some  people may have a better aptitude for  inquiry work. individual responses to some  inquiry topics vary and some health  information advisers may handle certain  inquiries better than others.      further research might assess how the  prompt sheet was used, and how frequently  it was used. there was no objective means  of assessing compliance with use of the  prompt sheet by the experimental group,  and it was not possible to be sure that  control group had not accessed the prompt  sheet. research with a larger cohort, at one  site, might help to ensure that experimental  and control groups are kept separate, as far  as possible, to reduce ‘contamination’. audit  of live calls would provide another  perspective of the performance, and the  nature of advanced expertise, as the written  test only provided an indicator of true  performance in the conversational turns of a  reference interview.     declaration of interests, and authorship  during the study the first author (tp) was a  health information manager at one of the  nhs direct sites. the paper was prepared  by cu, in collaboration with tp, and with  comments from jc.      acknowledgements  the authors thank the participants for  taking part in the trial, and nhs direct for  their approval of the study.  we are very  grateful to the reviewers and the editorial  team for their constructive comments on a  previous version of the paper.     a version of this paper was presented at the  4th international evidence based  librarianship conference, held in chapel  hill, north carolina, usa (may 2007).  see  http://www.eblip4.unc.edu/index.html      works cited    bains, simon. “testing collaborative  electronic reference services: the  experience of uk university research  libraries in the use of oclc’s  questionpoint.” virtual reference  desk conference. chicago, nov. 2002.  14 aug. 2007.  <http://data.webjunction.org/wj/docum ents/12629.pdf>.    baker, lynda. “preference for physicians as  information providers by women with  multiple sclerosis: a potential cause  for communication problems.” journal  of documentation 53.3 (1997): 251‐262.    booth, andrew, alan j. o’rourke and nigel  j. ford. “structuring the pre‐search  reference interview: a useful  technique for handling clinical  questions.” bulletin of the medical  library association 88.3 (2000): 239‐246.    bowman, vibiana. “the virtual librarian  and the electronic reference  interview.” internet reference services  quarterly 7.3 (2002): 3‐14.    dee, cheryl. “chat reference service in  medical libraries: part 1‐ an  introduction.” medical reference  services quarterly, 22.2 (2003): 1‐13.     dee, cheryl. “chat reference service in  medical libraries: part 2 ‐ trends in  medical school libraries.” medical  reference services quarterly 22.2  (2003): 15‐28.     dervin, brenda. “information needs and  uses.” annual review of information  science and technology 21 (1986): 3‐33.  http://www.eblip4.unc.edu/index.html http://data.webjunction.org/wj/documents/12629.pdf http://data.webjunction.org/wj/documents/12629.pdf http://data.webjunction.org/wj/documents/12629.pdf http://data.webjunction.org/wj/documents/12629.pdf evidence based library and information practice 2007, 2:3  57 dewdney, patricia and gillian michell.  “asking ‘why’ questions in the  reference interview: a theoretical  justification.” library quarterly 67.1  (1997): 50‐71.    diamond, wendy and barbara pease.  “digital reference: a case study of  question types in an academic  library.” reference services review  29.3 (2001): 210‐218.    grogan, denis. practical reference work.  2nd edition. london: clive bingley,  1992.    jennerich, elaine z. and edward j. jennerich.  the reference interview as a creative  art. englewood, co: libraries  unlimited. inc., 1997.    kennedy, lynn, charles cole and susan  carter. “the false focus in online  searching: the particular case of  undergraduates seeking information  for course assignments in the  humanities and social sciences.”  reference and user services quarterly  38.3 (1999): 267‐73    kuhlthau, carol c. “inside the search  process: information seeking from the  user’s perspective.” journal of the  american society for information  science 42.5 (1991): 361‐371.    lee, ian j. “do virtual reference librarians  dream of digital reference  questions?” australian academic and  research libraries 35.2 (2004): 95‐110.     mabry, celia h. “the reference interview as  partnership: an examination of  librarian, library user and social  interaction.” reference librarian (83/84)  (2003): 41‐56.    moody, janette and elizabeth carter.  “application of the cognitive  interview by the reference librarian.”  reference and user services quarterly  38.4 (1999): 389‐393.    national audit office. nhs direct in  england: report by the comptroller  and auditor general, hc 505 2001‐ 2002. london: national audit office,  2002. 14 aug. 2007  <http://www.nao.org.uk/pn/01‐ 02/0102505.htm>.    nelson‐jones, richard. the theory and  practice of counselling psychology.  london: cassell, 1982.    ross, catherine s. and patricia dewdney.  “negative closure. strategies and  counter‐strategies – the reference  transaction.” reference and user  services quarterly 38.2 (1999): 151‐163.    ross, catherine s., kirsti nilsen and patricia  dewdney. conducting the reference  interview: a how‐to‐do‐it manual for  librarians. london: facet publishing,  2002.    searle, john r. speech acts. cambridge:  cambridge university press, 1969.    simmonds, patience l. “the 21st century  reference department: working to  provide quality service to users.”  reference librarian (81) (2003): 89‐103.    stover, mark. “the reference librarian as  non‐expert: a post‐modern approach  to expertise.” reference librarian  (87/88) (2004): 273‐300.    straus, sharon e., w. scott richardson,  william s. rosenberg, paul glasziou  and r. brian haynes. evidence‐based  medicine: how to practice and teach  http://www.nao.org.uk/pn/01%e2%80%9002/0102505.htm%00%00 http://www.nao.org.uk/pn/01%e2%80%9002/0102505.htm%00%00 evidence based library and information practice 2007, 2:3  58 ebm. 3rd ed. edinburgh: churchill  livingstone, 2005.    taui, cameron k. “using the concept of  teamwork to increase why‐based  questioning in the academic reference  interview.” portal: libraries and the  academy 3.4 (2003): 663‐670.    taylor, robert s. (1968). “question‐ negotiation and information seeking  in libraries.” college and research  libraries 29.3 (1968):178‐194.     wildridge, valerie and lucy bell. “how  clip became eclipse: a mnemonic  to assist in searching for health  policy/management information.”  health information and libraries  journal 19.2 (2002): 113‐115.    wilson, tom d. and christina walsh.  information behaviour: an inter‐ disciplinary perspective: a report to  the british library r&d department  on a review of the literature. london:  british library, 1996. 14 aug. 2007 <  http://informationr.net/tdw/publ/infbe hav/>    http://informationr.net/tdw/publ/infbehav http://informationr.net/tdw/publ/infbehav http://informationr.net/tdw/publ/infbehav http://informationr.net/tdw/publ/infbehav research article   academic e-book usability from the student’s perspective   esta tovstiadi sr. assistant librarian, eresources librarian college libraries state university of new york college at potsdam potsdam, new york, united states of america email: tovstiem@potsdam.edu   natalia tingle assistant professor, business collections & reference librarian university libraries university of colorado boulder boulder, colorado, united states of america email: natalia.tingle@colorado.edu      gabrielle wiersma associate professor, head of collection development university libraries university of colorado boulder boulder, colorado, united states of america email: gabrielle.wiersma@colorado.edu    received: 14 june 2017                                                                  accepted: 17 july 2018      2018 tovstiadi, tingle, and wiersma. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29457     abstract   objective – this article describes how librarians systematically compared different e-book platforms to identify which features and design impact usability and user satisfaction.   methods – this study employed task-based usability testing, including the “think-aloud protocol.” students at the university of colorado boulder completed a series of typical tasks to compare the usability and measure user satisfaction with academic e-books. for each title, five students completed the tasks on three e-book platforms: the publisher platform and two aggregators. thirty-five students evaluated seven titles on nine academic e-book platforms.   results – this study identified each platform’s strengths and weaknesses based on students’ experiences and preferences. the usability tests indicated that students preferred ebook central over ebsco and strongly preferred the aggregators over publisher platforms.   conclusions – librarians can use student expectations and preferences to guide e-book purchasing decisions. preferences may vary by institution, but variations in e-book layout and functionality impact students’ ability to successfully complete tasks and influences their affinity for or satisfaction with any given platform. usability testing is a useful tool for gauging user expectations and identifying preferences for features, functionality, and layout.     introduction   librarians select materials based on a variety of criteria such as content, format, availability, and cost (anson & connell 2009; roncevic 2013). it is assumed and expected that the content of a book does not vary because it is published in cloth or paperback. electronic books challenge this expectation. while some e-books replicate the content and presentation of their print equivalents, others transform the initial work into something that hardly resembles the print version (kichuk, 2015; wiersma & tovstiadi, 2017). technical limitations and design choices on different e-book platforms create variations in the presentation, layout, and even content. these variations often go unnoticed because they are not obvious without direct comparison and evaluation.   in 2015, librarians at the university of colorado boulder tested thirty-four elements that are important to usability and the end-user experience and identified inconsistencies between e-book platforms, such as layout, navigation, metadata, and search results (wiersma & tovstiadi, 2017). this study builds on previous research by exploring some of those elements from the user’s perspective. students examined the same title on three different platforms and completed a series of tasks to compare the usability and measure user satisfaction with each platform. through this study we gained a greater understanding of student expectations and preferences that can be used to guide e-book purchasing decisions.   literature review   e-book usability issues identified by librarians   there is no shortage of articles in the literature admonishing the poor usability of academic e-book platforms. bivens-tatum (2014) described the various platforms as a “vast array of substandard choices,” noting that restrictions on use often cause patrons to give up on using e-books (para. 3). digital rights management (drm) and restrictions on downloading, printing, and saving e-books for offline use or for future reference are frequently cited explanations for the low acceptance of the medium (slater, 2010; thomas & chilton, 2016).   library journal’s e-book usage in u.s. academic libraries (2012) provided the librarian’s standpoint about e-book usability. the survey results revealed issues such as a “complex downloading process” and “difficult to read on screen/online” as some of the top barriers to e-book usage (p. 8). these issues persisted in the 2016 survey, alongside problems such as “platform not user friendly” and “can’t read offline or download” (library journal, p. 47).   mune and agee (2015) developed a template for evaluating different platforms by function, including navigation, offline availability, and full-text searching. while their study focused on accessibility as related to users with print disabilities, they found that “single publisher platforms (such as gale, palgrave, and springer) appear to offer more features and have more flexibility overall compared to aggregators (such as proquest and acls humanities) that include books from a variety of publishers in their collections” (p. 222).   cataldo and leonard (2015) compared 14 e-book platforms, and studied seven common features including format; user accounts; personal bookshelves; mobile accessibility; and the ability to annotate, download, and print. in addition to variation among platforms, they also found variation within aggregator platforms due to publisher restrictions. while user preferences may vary, they concluded that “it is crucial to understand the needs of your patrons, and more specifically on how the features, functionality and accessibility of the e-books meet those needs” (conclusion section, para. 2).   one the largest studies, the jisc national e-books observatory project, analyzed e-book use in more than 120 universities in the united kingdom and concluded that there was a strong need for e-book platforms that are designed with usability principles (2009). the call for consistent design is echoed throughout the literature (hobbs & klare, 2016; muir & hawes, 2013) as is the call for improved usability (slater, 2010).   e-book usability issues identified by users   a number of usability studies have been conducted on academic e-book platforms. carter, et al. (2013) used a survey to identify engineering students’ attitudes towards and experiences with e-books, finding that students expressed a number of concerns, including issues with navigation, format, printing, and downloading.   using the think-aloud method, berg, hoffman, and dawson (2010) compared a set of e-books with the print counterparts. the researchers instructed 20 undergraduate participants to complete information retrieval tasks using both print and e-books. students used different navigation and search strategies depending on format, and their expectations for e-book functionality were unmet.   a mixed-methods study by zhang, niu, and promann (2017) included a task-based usability test of 12 participants, including undergraduates, graduate students and faculty members. the user tests and follow-up survey call for improved consistency among e-book platforms since platforms that do not follow general web conventions appear to require more effort from the user.   o’neill (2009) compared the usability of ebrary, ebl, and myilibrary using task-based methodology with 10 undergraduates and graduates. the study identified a number of common usability issues with e-books, including functionality such as printing and navigation. muir and hawes (2013) observed 14 undergraduate physics students interact with two e-books on the netlibrary and myilibrary platforms. their findings support previous studies by highlighting issues with navigation and searching. in addition, the researchers developed a set of desired e-book features based on user needs.   this study builds on the work of previous studies by examining a greater number and additional types of platforms and suggesting that librarians can use test results as evidence to inform selection and purchase decisions. further, this study goes beyond determining whether e-book platform features exist, it evaluates how usable they are from the student perspective.   aims   the aims of this study are to:   ●        identify specific functionality and features that students prefer on e-book platforms ●        understand how differences in e-book platforms impact the user experience ●        describe how librarians can factor user experience into the selection of e-books   methodology   usability testing is a method of evaluating a product or service by testing it with a representative group of users. “the goal is to identify any usability problems, collect qualitative and quantitative data and determine the participant's satisfaction with the product” (u.s. department of health & human services, 2018). in this study, the authors observed students as a representative group of academic e-book users. after receiving irb approval for human subjects research, they posted information about the study in an online campus newsletter and the library’s social media channels and offered students a $10 amazon gift card for completing the study. they recruited one doctoral, one masters, and three undergraduates to test each title. although five students is a small sample of the entire student population, according to the nielsen norman group, “test[ing] 5 users lets you find almost as many usability problems as you'd find using many more test participants” (nielsen, 2012). this convenience sample of students was further limited to select students who had majors related to the subject of the sample title in order to replicate an authentic experience that an individual student might have with an e-book.   this study used “task-based usability,” a technique where users complete typical tasks on a website while an observer records if and how they were able to accomplish the task. during the test, students completed tasks on three e-book platforms. the tasks mimicked behaviour that students might naturally exhibit while using an e-book, such as printing, downloading, searching within the book, and navigating to a specific page. for the tasks, see the appendix. we observed their actions and noted whether, as well as how, they completed the tasks. using the “think-aloud protocol,” students were asked to verbalize their thoughts and expectations. this enabled the researchers to compare the actual results with students’ expectations in order to measure user satisfaction with the product. the authors took notes about each test and recorded the audio and on screen navigation.   the authors used a convenience sample of e-books that were available on ebook central, ebsco, and a publisher platform. the library acquired access to the sample titles on additional platforms as needed.   for each title, five students completed the tasks on three platforms: the publisher platform and two aggregators. the order in which the platforms were tested was randomized, in order to temper the potential bias of consistently testing one platform first or last, and students completed all of the tasks on one platform before moving onto the second and third.   after the tasks were completed on each platform, students ranked the platforms in order of preference, using physical printouts of the e-book landing pages as references. this gave students an opportunity to compare the platforms and provide additional feedback.   results   in each usability test, the student completed a series of tasks on each of three platforms. results from each of the tasks are described below. altogether 35 students (21 undergraduates, 7 masters, 7 doctoral) tested sample titles on seven publisher (brill, cambridge, duke, oxford scholarship online, springer, science direct, and wiley) and two aggregator platforms (ebsco and ebook central). at the end of the test, students were asked to rank the three platforms in order of preference. sixty percent of students rated ebook central as their preferred platform, followed by ebsco (26%) and individual publisher platforms (14%).   task 1: evaluating the landing page   the usability tests started on a landing page, which is typically the first page that a user sees when they click on a link to an e-book from a search engine, the library catalog, or a discovery layer. we asked students what they expected to see on this type of page and what information was most useful. most students expected to see the basic bibliographic elements needed to cite a book (e.g., title, author, and publication information) as well as a brief summary or abstract. some students expected an isbn or doi, which they indicated was helpful for citing a book or figuring out which edition they were using. while most platforms provide all of this information, the placement on the page and order in which it was presented varied.   task 2: evaluating the bibliographic information   when students were asked how they interpreted the bibliographic information (title/subtitle, authors, dates, keywords/subject headings) on the landing page it seems likely that students would accept the bibliographic information presented at face value and not question its accuracy. as they navigated between platforms, students understood each of these components individually but were puzzled when the information for the same book varied on different platforms. some students pointed out discrepancies in the metadata for publication dates, subtitles, and author information when it varied by platform. these discrepancies would likely have gone unnoticed but were obvious when students were asked to identify and interpret this information on each platform.     figure 1 availability on proquest ebook central.     figure 2 concurrent user level on ebsco.     some of the students noticed the subject terms but were not impressed unless they were hyperlinked and more descriptive than just repeating words in the title of the book. most students were unfamiliar with library jargon like “lc subject headings,” but they generally understood that keywords and categories were meant to describe the book. some users indicated that they would skim through a summary or book description to help them decide if it was appropriate for their research, but most people assumed that if they ended up on this page, it was because they already knew that they needed this book.   students wanted information about the availability of the book, but almost all of them misinterpreted the information that described the permitted number of simultaneous users or the number of copies. for example, ebook central and ebsco included information about availability based on the type of license that the library purchased (see figures 1 and 2).   while a librarian might interpret that “access to 3 copies of this book” refers to a license that allows three users to access the e-book simultaneously, many students misinterpreted the number of available online copies as the number of print copies they could find in the library. this was particularly misleading because our library typically does not purchase books in multiple formats, and so it is very unlikely that we would have both an e-book and print copy, much less multiple print copies.   task 3: finding and using citation tools   next, we asked students to interact with some of the information on the landing page. we asked them how to cite the book using any available tools or information, using the prompt: “you need to cite this book for your paper. how would you use this page to do that?” a native citation generating tool is one of the features that is referenced as being a benefit or expected feature of e-books (cassidy, 2012). many students noticed the native citation generator immediately. some of the students were introduced to the native citation generator while interacting with the first of three platforms, then learned to look for a similar tool on subsequent platforms. although a majority of the students were able to find the citation tools easily on the aggregator platforms, consistent naming and use of icons between platforms would improve usability.   we encountered two kinds of native citation generating tools, which may account for some of the inconsistency in labeling the tool. in one type (cite), the citation is displayed on demand in one of several citation styles for the user to copy and paste as needed. the other variety (export) is a downloadable file for use with a bibliographic management tool such as endnote, zotero, bibtex, or refworks. some of the platforms included both kinds of tools. the platforms that offered one generally offered the download version, although sometimes they provided a plain text download option, which could then be opened in a tool like notepad and then copied and pasted. students reacted much more positively to the versions that showed the citation without having to open another program or download any files, even when the style they commonly used was not on the list.     table 1 finding citation tools platform found citation tools easily found after some time/ with some difficulty did not find brill (n=5) 20% (1) 20% (1) 60% (3) cambridge (n=5) 60% (3) 20% (1) 20% (1) duke (n=5) 40% (2) 60% (3) ebook central (n=33)* 58% (19) 9% (3) 33% (11) ebsco (n=32)* 56% (18) 3% (1) 41% (13) oxford scholarship online (n=4)* 25% (1) 25% (1) 50% (2) science direct (n=5) 20% (1) 40% (2) 40% (2) springer (n=5) 100% (5) wiley (n=5) 40% (2) 60% (3) *data was not available for some students.     in completing this task, students mentioned a variety of other tools for creating or managing citations. popular programs included easybib, citation machine, knight cite, mendeley, and zotero. in several cases, the students explained that they would google either the title or the isbn number along with “cite.” whether they would use the native tool or another means, many students in this study also mentioned the importance of including all the elements needed to cite manually the e-book on the landing page.   task 4: navigating to a specific chapter or page number   next, the students were asked to navigate to a specific chapter and page number. this task was designed to observe how students preferred to navigate through an e-book (e.g., using linked table of contents, searching, or scrolling) and if they were able to find the correct page. the majority of students (69%) found the correct page easily. most students used the linked table of contents from the landing page or in the navigation pane of an e-book reader, but they were frustrated when it was not linked in the pdf versions. many students also used the page number box to “jump” to a specific page when that feature was available.     table 2 success navigating to a specific page platform found appropriate page easily found with some difficulty did not find brill (n=5) 100% (5) cambridge (n=5) 40% (2) 60% (3) duke (n=5) 80% (4) 20% (1) ebook central (n=32)* 72% (23) 22% (7) 6% (2) ebsco (n=31)* 74% (23) 23% (7) 3% (1) oxford scholarship online (n=4)* 75% (3) 25% (1) science direct (n=5) 100% (5) springer (n=5) 60% (3) 40% (2) wiley (n=4)* 75% (3) 25% (1) *data was not available for some students.     some students scrolled within the reader to find a specific page. sometimes scrolling was preferable to jumping from page to page, and other times it was necessary because the e-book did not include page numbers. some downloaded the entire book and navigated within the downloaded file. for some platforms, it was necessary for students to go back to the landing page and then open the pdf of the correct chapter. a few students noted that page numbers that were displayed in the pdf reader did not always match the page number printed on the page. if they struggled to find the correct page, students often used creative solutions to find it. although frustrating, students seemed to tolerate these inconsistencies if they had experienced them before.   students also struggled to find the appropriate page when the platform did not display page numbers, such as the epub version of e-books in ebook central and ebsco. if students are unable to locate a specific page in an e-book this might prevent them from successfully completing an assigned reading, locating a cited reference, or creating a citation. platform design played a major role in students’ ability to complete this task, and they preferred platforms that included linked tables of contents, the ability to “jump” from page to another, and clearly displayed page numbers.    task 5: finding and using annotation tools   next, we invited students to interact with the text by employing available annotation features. we asked students, “as you are reading, you want to take notes for your class. would you do that here in the e-book? if so, how?” at the time of testing, annotation tools such as highlighting and note taking were only available on the aggregator platforms.    we rated the ease with which our testers found the annotation tools on a scale of 1 to 3, with 3 being “found easily” and 1 being “did not find.” students had an easier time finding annotation tools in ebook central than ebsco, with 63% (23) finding the tools easily in ebook central (n=32) compared to 11% (4) in ebsco (n=32). this may be due to the placement of the “my notes” link in ebsco or to the multiple ways of accessing the tools in ebook central (see figures 3 and 4).   the annotation tools within the ebook central platform were easy for our users to find and use, especially the highlighting tool. ebook central situated its annotation tool icons (highlight, add note, and bookmark) in the toolbar at the top of the reading pane (see figure 4). additionally, when a user selected a section of text within the reading pane, a popup emerged with options for copying or highlighting the selection as well as an option to add a note.   the annotation tools in the aggregators led to popups about creating an account. due to the way the task scenarios were ordered, this may have been the user’s first encounter with the need to create an account on the platform in order to fully use the available tools. students were quick to close the popup window, often without a pause to read closely. one student said about creating accounts in ebook central, “unless i was really desperate for what they had in here, i would probably see if i could find it on any other source that didn’t require me to sign up.” this sentiment was echoed in various ways across many of the tests.   students had pre-existing habits and strong preferences for note-taking that influenced their response to this question. many students said that they would prefer to download the pdf and highlight or annotate within the pdf file or on a printed copy of the file. some of the alternative note-taking options students noted included a physical notebook, evernote, google keep, mendeley, a word document, or text files. they expressed some interest in the tools, particularly the highlight tool found in ebook central; however, very few of the students we tested (17%) affirmed that they would likely use the annotation tools, expressing concern about the long-term availability of the notes they take or having to create an account to take or keep notes. although commonly indicated as an important feature, some studies report that students may not take advantage of these tools (jisc, 2009; muir & hawes, 2013).     figure 3 “my notes” link in ebsco.     figure 4 annotation tools in ebook central.     task 6: searching and evaluating results   next, we asked students to find information on a specific topic within the e-book. we didn’t want to lead them to use the search tool, rather we wanted to understand how they naturally looked for content in e-books. previous studies found that students use multiple navigation strategies to locate information in e-books, such as searching, navigating the table of contents or index, or scrolling through pages (muir & hawes, 2013). in our study, most students employed multiple search strategies throughout the test but generally used more on the first platform and fewer on the last platform. on the first platform they tested, 63% of students responded to this question by searching for the topic within the book, either using ctrl+f (12 students) or the e-reader searching tools (10 students). thirteen students started with either the index or the table of contents (toc). of those who started with the index or toc on the first platform, on subsequent platforms all 13 students started by searching instead of using the index. all students were asked to use the search feature even if that was not their first choice for finding information within the e-book.   a surprising number of students tried keyboard shortcuts instead of the search box on the platform. over half (54%) of students used ctrl+f to search for a term in the book at least once during the testing, but they experienced varying levels of success. generally, students employed different search strategies on subsequent platforms based on the results from the first platform. if ctrl+f did not work on the first platform, students might continue to try it on subsequent platforms, but they also tried other search mechanisms.   by the third platform, the majority of students (63%) used only one strategy: searching through the e-book’s reader tool or using ctrl+f. they may have done this because they realized that the librarians would ask them to search as part of the test, or they may have done this because they realized that searching was more efficient and effective than other strategies.   the fact that students modified their information-seeking strategy from the first platform to the last platform suggests that they learned as they interacted with e-books. on the first platform, students exhibited a wider range of information seeking behaviour and often took more time to look for tools or complete tasks than on subsequent platforms. for example, many students did not consider searching for a term in the first e-book, but by the last platform, nearly every student opted to search within the e-book rather than try a different strategy, such as using the table of contents or the index.     table 3 how students would find specific information in a book strategy number of students/sample size ctrl+f (keyboard shortcut) 19/35 looking in the index 11/35 looking in other parts of the book (chapter titles, preface, etc.) 3/35 looking in the table of contents 11/35 searching using the platform or reader’s search tool* 35/35 *librarians prompted students to search if the students did not search on their own.      how many search results did you expect?   along with the task of searching within the e-book using the platform’s search feature, we asked students to evaluate the following: how many search results they expected, how the results were ranked, and what they expected to see if they clicked on a result. many students guessed or had an idea of how many results to expect based on the number of results from previously tested platforms. some students (16) indicated some expectation that the number of results should be similar across all platforms, indicating on the second or third platform an expectation that the number of results be similar to the number they encountered on the previous platform(s).   it was difficult for some students to estimate how many search results to expect if the subject matter of the book was outside of their field. the authors attempted to have students test a sample title related to their academic discipline, but it was difficult to offer an exact match for each major. of the 11 students who mentioned being unsure of how many results they expected or said that they had no expectation, five attributed this to their lack of subject knowledge.   how do you think these search results are ranked? why is this one (point to top one) first?   a majority (74%) of students expressed uncertainty or confusion about how the results were displayed or ranked on at least one of the platforms. this is understandable considering the wide range of search results they encountered during the tests. e-book platforms tend to display search results at the chapter, page, or keyword level. students seemed to understand that keyword-level search results listed each time a keyword appeared in the text. keyword results were overwhelming when the search term appeared more than a few times in the text and the students had to scroll through dozens of results.   chapter-level results were confusing because the search term was not always highlighted or included in the search results, so it was difficult to understand why each result was a good match. most students appreciated when the search results displayed a snippet of text that surrounded their search term and ideally also highlighted or bolded their search term. this helped them quickly identify the keyword and provided helpful context to determine the best match(es). students were also frustrated if after navigating to the appropriate chapter, their search term was not indicated within the text.   students were also confused by some of the default sorting options. most students (63%) expected or believed that search results would be displayed in the order in which they appear in the book. some students (26%) were able to figure out relevance ranking, but many were confused when results were not displayed in “chronological” or “page number order”. students were confused when a platform displayed a list of pages or chapters out of order in order to represent relevance.   ebook central had the most intuitive display because search results were grouped within each chapter and relevance was indicated by a bar graph that clearly represented term frequency. at the time of testing, ebsco's platform did not provide an overall number of keyword results nor did it give another option for sorting results, which made it very difficult for users to interpret which results were most relevant.   where do you expect to go when you click on this search result?   most students expected that their search term(s) would be highlighted in the results (74%) and that clicking on a result would link them to the part of the page, or at least the page in general, where that term appeared (60%).   task 7: printing, saving, and downloading   a surprising number of students (40%) tried to use keyboard or mouse shortcuts, such as ctrl+p or right-clicking, to print, save, or download. the majority of these students tried these strategies on the first platform and abandoned after they didn’t work. unfortunately, these types of commands do not work on the majority of platforms. based on our findings, e-book platforms should consider making their sites responsive to these commands. at the very least, the platform could respond to a keyboard shortcut by moving the user’s cursor to the appropriate icon or link on the website. this would not only improve the user experience for many users who just prefer to use shortcuts, but it may also improve usability for students using screen readers or other assistive technology.   eventually, almost all students (97%) utilized the e-book reader or pdf printing icons. when asked what they expected to see when downloading the book, most students expected a pdf. many students remarked positively when the pdf contained a citation, and they particularly appreciated the ability to select the citation style before downloading the pdf.   it was helpful when actions such as printing, downloading, and saving functioned similarly across the platform or, ideally, mimicked functionality on other websites. if a student figured out one process, then it was easier for them to master other processes. although the outcomes are similar, students were confused when platforms used unclear terminology such as an option to print a “section” rather than a chapter, and they were frustrated by warnings about “exceeding your print allowance” when attempting to download a chapter.  our observations indicated that it is important to present information about printing or download limits at the point of need instead of just listing that information on the landing page. users need this information within the context of performing an action or when it limits their ability to take action, but otherwise this information is superfluous and confusing.   both aggregators required students to login or create an account in order to download the entire book, whereas the publisher platforms did not. when prompted to log in by the aggregator sites, most students summarily ignored the pop-up notification and tried to find other ways to print or download the book. a few students tried to subvert the drm on the aggregators by taking screenshots or saying that they would go to “other sites” or ways of accessing this e-book, even though they understood that it was unethical and illegal to do so.     table 4 students expressing difficulty finding print options by platform platform percent expressing difficulty brill (n=4)* 25% (1) cambridge (n=5) 20% (1) duke (n=5) 0% (0) ebook central (n=31)* 6.4% (2) ebsco (n=33)* 12% (4) oxford scholarship online (n=5) 40% (2) science direct (n=5) 60% (3) springer (n=5) 20% (1) wiley (n=4)* 25% (1) *data was not available for some students.     table 5 students expressing difficulty finding download/offline reading options by platform platform percent expressing difficulty brill (n=5) 20% (1) cambridge (n=3)* 33% (1) duke (n=5) 80% (4) ebook central (n=31)* 45% (14) ebsco (n=32)* 31% (10) oxford scholarship online (n=5) 20% (1) science direct (n=5) 0% (0) springer (n=5) 0% (0) wiley (n=4)* 25% (1) *data was not available for some students.     table 6 of the three tested, which platform is your first preference? platform number of students brill (n=5) 0 cambridge (n=5) 1 duke (n=5) 0 ebook central (n=35) 22 ebsco (n=35) 8 oxford scholarship online (n=5) 1 science direct (n=5) 1 springer (n=5) 1 wiley (n=5) 1     task 8: which platform would you prefer to use?   the final task asked students to rank the platforms in order of preference, using physical printouts of the e-book landing pages as a reference. of the students studied, 60% rated ebook central as their preferred platform, followed by ebsco (26%) and individual publisher platforms (14%). some of the characteristics that distinguished ebook central were the intuitive layout of the search results including the bar graph that indicated whereand how many search results were included in each chapter and the clearly visible icons and menus that made it easy to accomplish tasks such as printing and downloading.   in general, students preferred the platforms that offered full-text searching, identified the number of search results, highlighted search terms within the results, and presented search results in an intuitive order (either relevancy or the order in which they appear in the book). they also preferred platforms that allowed them to easily highlight in multiple colors. students had mixed opinions about the reading pane layout of most aggregators but seemed to prefer the toolbars in both aggregators because the icons clearly identified the tools that students needed most (e.g., printing and downloading). they also expressed varying opinions about platforms that prompted them to login or create an account in order to print, download, or save content.   discussion   in addition to learning about students’ preferences for features and functionality, we also learned that they are quick to blame themselves when things do not work as expected. regardless of whether it was a system error or user error, many students assumed that lack of functionality was somehow related to their limited knowledge about the subject of the book or unfamiliarity with the platform. on the other hand, when a platform was more intuitive to use, students were happy to demonstrate how to use the site and seemed more assertive in their comments. this was perhaps the strongest evidence that platform design impacts user experience and that librarians need to understand how platforms vary in order to purchase content on platforms that optimize user experience.   study limitations and recommendations for further research   although the usability testing revealed local user preferences, the results are not generalizable to all students. the convenience sample of students who participated in the study may not represent our larger student population in terms of previous e-book experience or fields of study. the majority of students in the convenience sample were from stem (science, technology, engineering, or math) majors, and we recognize that these students might use e-books differently than other disciplines. we also know that many of the participants had some experience with e-books prior to the study. we collected information regarding prior e-book use in a pre-screening survey but were unable to use this information because the pre-screening survey did not include a consent form. both areas would be interesting areas for further research.   there are many ways to identify usability problems and measure user satisfaction. this study used the think-aloud protocol and task-based usability. while these techniques are designed to produce qualitative and quantitative data, there are limitations and room for error in their application. for example, task-based usability is predicated on an observer leading the user through a script of predefined tasks, but it is difficult not to deviate from the script in order to follow the subject’s flow of information seeking behavior. the study was also limited by the tasks that we asked students to complete. we tried to create tasks that mimic what we thought was typical student behavior, and that may have skewed our results. for example, we did not ask students to download the entire e-book, although if we had, we hypothesize that fewer students would prefer the aggregator platforms. likewise, the testing environment was based on a false need for information. students may have behaved or answered differently if the need was real and attached to an outcome that mattered to them, such as a grade on an assignment.   the think-aloud protocol also has certain limitations. some students respond more naturally to verbal communication; others might have given us different answers if they were asked to respond in writing. some students may also have been influenced by the perceived “power dynamics” of a faculty librarian and student relationship. this was mitigated by the consent form and the script that assured students that this wasn’t “a test of your knowledge, and there are no right or wrong answers.” however, their responses might have been skewed towards what they thought the librarian wanted to hear.   finally, the students’ ranking of e-book platforms was limited to the platforms that they examined. because more students examined the aggregators than the publisher platforms, the results regarding the aggregators are arguably more valid. if students had been asked to test a different publisher platform, it may have changed their opinion relative to the aggregators that they tested. however, since most books are only available on a publisher and one or more aggregator platforms and not different publisher platforms, the comparison between publisher and aggregator platforms remains the most relevant finding rather than comparing one publisher platform to another.   conclusion   there are relatively small differences between major e-book aggregators in terms of cost, content, and coverage. as such, user feedback about their preferred platform was critical to selecting a default option for the library’s approval plan and demand driven e-book programs. this study identified strengths and weaknesses of academic e-book platforms based on students’ experiences and preferences. these characteristics can be used alongside other factors such as pricing and accessibility when selecting a title that is available on more than one platform.   the results of the usability tests in this study indicated a preference for ebook central over ebsco and a strong preference for the aggregators over publisher platforms. we expected that students would prefer the publisher platforms because those platforms rarely impose limits on printing and downloading. students in this study, however, struggled to navigate the publisher platforms and the overall perception was that they are not as easy to use as aggregators with clearly defined menus and icons. this suggests that students value usability and are even willing to accept some printing and downloading restrictions in exchange for an intuitive, user-friendly platform. although students will find a way to access the materials they need, all e-book providers should follow usability design principles that serve the needs of students.   this study explored students’ information seeking behaviour on e-book platforms and identified features and functionality that students prefer on these platforms. it confirmed the results of many previous studies that found that usability issues influence user perceptions and success rates using e-books. until we are able to build completely intuitive resources, having a better understanding of user expectations will help us select books on the platforms that best meets user expectations.   references   anson, c., & connell, r. (2009). e-book collections. washington, dc: association of research libraries.   berg, s. a., hoffmann, k., & dawson, d. (2010). not on the same page: undergraduates’ information retrieval in electronic and print books. the journal of academic librarianship, 36(6), 518–525. https://doi.org/10.1016/j.acalib.2010.08.008   bivens-tatum, w. (2014). the mess of ebooks. peer to peer reviewed. retrieved from http://lj.libraryjournal.com/2014/10/opinion/peer-to-peer-review/the-mess-of-ebooks-peer-to-peer-review/   carter, d.s., grochowski, p. f., lalwani, l. n., nicholls, n. h., & samuel, s.m. (2013). students, vendor platforms, and e-textbooks: using e-books as e-textbooks. asee annual conference. retrieved from http://www.asee.org/public/conferences/20/papers/7427/view   cassidy, e. d., martinez, m., & shen, l. (2012). not in love, or not in the know? graduate student and faculty use (and non-use) of e-books. the journal of academic librarianship, 38(6), 326–332. https://doi.org/10.1016/j.acalib.2012.08.005   cataldo, t. t., & leonard, m. (2015, spring). e-stem: comparing aggregator and publisher e-book platforms. issues in science and technology librarianship (80). https://doi.org/10.5062/f4fj2dsp   hobbs, k., & klare, d. (2016) are we there yet?: a longitudinal look at e-books through students’ eyes. journal of electronic resources librarianship, 28(1), 9–24. https://doi.org/10.1080/1941126x.2016.1130451   jisc. (2009). jisc national e-books observatory project: key findings and recommendations. jisc national e-books observatory project. retrieved from http://observatory.jiscebooks.org/reports/jisc-national-e-books-observatory-project-key-findings-and-recommendations/   kichuk, d. (2015). loose, falling characters and sentences: the persistence of the ocr problem in digital repository e-books. portal: libraries and the academy, 15(1), 59–91. https://doi.org/10.1353/pla.2015.0005    library journal. (2012). ebook usage in u.s. academic libraries. the digital shift. retrieved from http://www.thedigitalshift.com/research/ebook-usage-reports/academic/   library journal. (2016). ebook usage in u.s. academic libraries. retrieved from https://s3.amazonaws.com/webvault/research/lj_2016_ebookusage_academiclibraries.pdf muir, l., & hawes, g. (2013). the case for e-book literacy: undergraduate students’ experience with e-books for course work. the journal of academic librarianship, 39(3), 260–274. https://doi.org/10.1016/j.acalib.2013.01.002   mune, c., & agee, a. (2015, mar.). ebook showdown: evaluating academic ebook platforms from a user perspective. paper session presented at the meeting of the association of college and research libraries, 2015 conference, portland, or. retrieved from https://works.bepress.com/ann_agee/19/   nielsen, j. (2012). how many test users in a usability study? nielsen norman group: evidence-based user experience research, training, and consulting. retrieved from https://www.nngroup.com/articles/how-many-test-users/   o’neill, l.c. (2009). a usability study of e-book platforms. retrieved from https://cdr.lib.unc.edu/record/uuid:9a109741-0d0e-4a11-b02b-21893c32f8b9   roncevic, m. (2013). criteria for purchasing e-book platforms. library technology reports, 49(3), 10–13. retrieved from https://journals.ala.org/index.php/ltr/article/view/4305   slater, r. (2010). why aren’t e-books gaining more ground in academic libraries? e-book use and perceptions: a review of published literature and research. journal of web librarianship, 4(4), 305–331. https://doi.org/10.1080/19322909.2010.525419   thomas, j., & chilton, g. (2016). library e-book platforms are broken: let’s fix them. in s. m. ward, r. s. freeman, & j. m. nixon (eds.), academic e-books: publishers, librarians, and users (pp. 249–262). west lafayette, in: purdue university press.   u.s. department of health & human services. (2018). what & why of usability. retrieved from https://www.usability.gov/how-to-and-tools/methods/usability-testing.html   wiersma, g., & tovstiadi, e. (2017). inconsistencies between academic e-book platforms: a comparison of metadata and search results. portal: libraries and the academy, 17(3), 617–648. https://doi.org./10.1353/pla.2017.0037   zhang, t., niu, x., & promann, m. (2017). assessing the user experience of e-books in academic libraries. college & research libraries, 78(5). https://doi.org/10.5860/crl.78.5.578   appendix usability tasks   1)       task 1: evaluating the e-book landing page (landing page, title bibliographic info, native reader) a)       have you ever seen this webpage before? b)      what information do you expect to see here? i)        what information is most helpful?   2)       task 2: evaluating the bibliographic information a)       did you notice this date? what does this date mean to you? (any date on landing page; publication or otherwise) b)      why do you think these names are here (point to authors, editors, etc. names)? c)       did you notice these? (point to subject terms). what do these mean to you?   3)       task 3: finding and using citation tools a)       you need to cite this book for your paper. how would you use this page to do that?   4)       task 4: navigating to a specific chapter or page number a)       how would you start reading the e-book from this page? b)      what do you expect to see when you click on this (read e-book, open e-book, etc.;)? c)       your professor told you to start reading at chapter ##. it starts on page ##. starting from this page, how would you do that?   5)       task 5: finding and using annotation tools a)       (after they navigate) as you’re reading, you want to take notes for your class. would you do that here in the e-book? if so, how?   6)       task 6: searching and evaluating results a)       you need to find information on _______ in this book. how would you do that? [if searching is not their first response, prompt them to search within the book] b)      [note how many results] c)       how many search results did you expect? d)      how do you think these search results are ranked? why is this one (point to top one) first? e)       which search results are more useful? f)        where do you expect to go when you click on this search result?   7)       task 7: printing, saving, and downloading a)       you want to print this page to read later. how would you do that? b)      you want to save this chapter to read later. how would you do that? a.       what would you expect to see if you downloaded the book?   8)       task 8: which platform would you prefer to use? a)       now that you have seen 3 different versions of this book, which would you prefer to use? [the student will be given three print-outs; one showing a screenshot of each landing page of the e-books that they used during the testing.] b)      rank the versions in order of preference.     the use of search request forms can identify gaps in a consumer health library collection evidence summary   the use of search request forms can identify gaps in a consumer health library collection   a review of: giannopoulos, e., snow, m., manley, m., mcewan, k., stechkevich, a., giuliani, m. e., & papadakos, j. (2021). identifying gaps in consumer health library collections: a retrospective review. journal of the medical library association: jmla, 109(4), 656–666. https://doi.org/10.5195/jmla.2021.895   reviewed by: matthew bridgeman information and education librarian robert wood johnson library of the health sciences rutgers, the state university of new jersey new brunswick, new jersey, united states of america email: mcb226@libraries.rutgers.edu   received: 6 june 2022                                                               accepted:  20 july 2022      2022 bridgeman. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30187     abstract   objective – the objective of this study was to determine if search request forms, which are used when a patron’s request for information cannot be fulfilled at the time of contact with the library team, can be used to identify gaps in consumer health library collections while offering some explanation for the gaps.   design – retrospective case study of search request forms.   setting – a consumer health library at an academic cancer center in canada.   subjects – library patrons: patients, patient family, other members of the center, and unspecified.   methods – the researchers reviewed 260 search request forms submitted between 2013 and 2020. of those, 249 records met inclusion criteria and were analyzed and coded. coding included patron type, cancer diagnosis, information delivery, and content themes. this information was then used to identify gaps in the library collection and the reasons for the gaps. main results – patients were the primary patrons, asking 62.9% of the questions, followed by family members at 22.5%. the most common cancer type researched was breast at 23.3%, then hematology at 16.5%. gynecology, gastrointestinal, genitourinary, and sarcoma were next between 10% and 8.4%. the remaining cancer types ranged between 6.0 % and 2.0%, with brain being the lowest. of the questions asked, 60% revealed a gap in the collection. the gaps included rare cancer diagnosis, treatment options, and prognosis. there were data collected on why the information was unavailable. while 53% of the gaps were a result of limited health consumer information, 25% were a result of paywall restrictions or content restricted to members.   conclusion – search request forms can be an effective tool in evaluating gaps in collections. in this study, the researchers were able to identify that breast cancer patients made up the most significant proportion of patrons, and the biggest gaps in the collection were related to their treatment decisions. one opportunity to bridge this gap is through collaboration with clinical teams in developing patient friendly resources on this topic. in addition, inter-institutional collaboration between libraries may also help. continued review of forms can help inform collection decisions to better meet the needs of patrons.   commentary   consumer health libraries play a significant role in helping cancer patients navigate the over-abundance of information available. they provide reliable information to help with treatment, prognosis, and diagnosis. the collections at these libraries need to be accurate and readily available. however, collection development is a complex and challenging part of librarianship, especially when considering factors like use, budget, and reliability. there are a variety of procedures and policies used to assist librarians, but they are focused on the quality of the content, for example, relevance, credibility, currency, and so on (papadakos et al., 2014). in this study, giannopoulos et al. (2021) offered an interesting mode for assessing gaps in a consumer health library at a cancer institute by analyzing search requests.     the glynn (2006) critical appraisal tool was used to evaluate this study. giannopoulos et al. (2021) clearly described data collection; however, they did not mention where the data was stored, nor gave an example of the mentioned search request form. the researchers did mention that the data will not be made available due to personally identifiable information. they clearly identified inclusion/exclusion criteria, namely requests unrelated to cancer were excluded. the data they provided is detailed and well-presented.   the conclusions giannopulos et al. (2021) reached would be helpful for their specific institution. they identified gaps in their collections that fail to meet the needs of their patrons. in addition, they also explored why these gaps existed and offered some interesting solutions, such as collaboration among institutions to develop their own consumer health information. that could potentially fill gaps in collections outside the collaborators and may serve as a model for future projects. the paywall restrictions unfortunately are more difficult to overcome, but they only made up 25% of the reasons for gaps.   however, giannopulos et al.’s (2021) study cannot be generalized to other libraries yet. the population was not large enough to make general conclusions. a follow up study involving multiple institutions utilizing the same search request form could provide more significant insight. overall, this was an interesting study that offered a novel approach to collection development. utilizing search request forms for collection gap analysis should be explored further.   references   giannopoulos, e., snow, m., manley, m., mcewan, k., stechkevich, a., giuliani, m. e., & papadakos, j. (2021). identifying gaps in consumer health library collections: a retrospective review. journal of the medical library association: jmla, 109(4), 656–666. https://doi.org/10.5195/jmla.2021.895   glynn. l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387–399. https://doi.org/10.1108/07378830610692154   papadakos, j., trang, a., wiljer, d., cipolat mis, c., cyr, a., friedman, a. j., mazzocut, m., snow, m., raivich, v., & catton, p. (2014). what criteria do consumer health librarians use to develop library collections? a phenomenological study. journal of the medical library association : jmla, 102(2), 78–84. https://doi.org/10.3163/1536-5050.102.2.003   evidence summary   information literacy skills are positively correlated with writing grade and overall course performance   a review of: shao, x., & purpur, g. (2016). effects of information literacy skills on student writing and course performance. the journal of academic librarianship, 42(6), 670-678. http://dx.doi.org/10.1016/j.acalib.2016.08.006   reviewed by: rachel e. scott integrated library systems librarian university libraries university of memphis memphis, tennessee, united states of america email: rescott3@memphis.edu   received: 3 mar. 2017    accepted: 7 apr. 2017      2017 scott. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to measure the correlation of tested information literacy skills with individual writing scores and overall course grade.   design – online, multiple-choice survey.   setting – public research university in north carolina, united states of america.   subjects – freshmen students enrolled in either first-year seminar (uco1200) or basic english writing course (eng1000).   methods – a 25-question, forced-choice test was piloted with 30 students and measured for internal consistency using cronbach’s alphas. the survey instrument was slightly revised before being administered online via selectsurvey, to 398 students in 19 different sections of either uco1200 or eng1000, during class sessions. the test measured students’ information literacy skills in four areas: research strategies, resource types, scholarly vs. popular, and evaluating websites. the preliminary questions asked for each student’s name, major (by category), number of library instruction sessions attended, and the names of library services utilized. the students’ information literacy scores were compared to their writing scores and overall course grades, both of which were obtained from course instructors. the information literacy scores were also analyzed for correlation to the number of library instruction sessions attended or the types of library services utilized.   main results – information literacy skills positively correlated with writing scores (n=344, r=-.153, p=0.004) and final course grades (n=345, r=0.112, p=0.037). pearson’s correlation coefficients results demonstrated relationships between writing scores and the information literacy test section “scholarly versus popular sources” (n=344, r=0.145, p=0.007), and final grade and information literacy test sections “types of sources” (n=345, r=0.124, p=0.021) and “website evaluation” (n=345, r=0.117, p=0.029). the impact of using other library services or of attending multiple information literacy sessions was not statistically significant.   conclusion – students’ mastery of tested information literacy skills directly correlates to their writing and final course grades. the study confirms the need for faculty and library collaboration to create well-integrated library instruction and services, and advocates for librarians to become integral to campus initiatives for student learning and success.   commentary   there is a growing body of literature linking academic libraries to various measures of student success. megan oakleaf has published extensively on assessing the academic library’s contributions; a 2016 article by her focuses on librarian involvement in institutional-level initiatives. rockman’s 2002 paper shows that institutions have long collaborated across departments and campuses to integrate information literacy into the general education curriculum to support institutional goals. by investigating the correlation of information literacy skills instruction with writing scores and overall class grades, the authors of the study at hand provide a unique and compelling contribution to these areas of the literature.   the “reader’s guide to the literature on interventions addressing the need for education and training” facilitates the evaluation of a study’s design, educational context, results, and relevance (koufogiannakis, booth, & brettle, 2006). the objective of the study was clearly articulated. the first three research questions, analyzing the correlation of information literacy skills to both writing skills and course grade and identifying the key information literacy skills for both, were clearly stated and investigated. the fourth research question, assessing the effect of library uses on student performance, is too broad and cannot be systematically addressed. the relevant survey question: “library services you have used in order to complete your writing assignments (choose all that apply)” asks students not to apply skills, as in the rest of the test questions, but to understand or remember library jargon (e.g., “rap session,” “online tutorials,” “library service desks”) (p. 675).    the teaching method, mode of delivery, instruction topics and amount of instructional contact time were not detailed. uco1200 and eng1000 sections were required to meet specific assignment criteria to qualify for study participation. it was unclear if and how these heightened requirements created a different educational intervention. name, academic class level, and major were the only demographic information collected through the instrument, and it is unknown if the subjects were representative of the university’s undergraduate population.   the study’s results are clearly explained, but some details are missing. the authors did not mention an irb (institutional review board) or a consent process. they also did not account for the differences between the number of students completing the information literacy skills test (p=398) and the number with available writing scores (p=344) or final grades (p=345); presumably these students dropped the course. the relevant data is presented and analyzed using spss statistical software.    this article’s positive contribution to the literature is the validation of its premise, namely that information literacy skills can be learned through instruction and use of library resources and services. the study makes a compelling argument for the continued integration of tailored library instruction in the general education curriculum; targeting first-year students can have a timely impact on academic success.   references   koufogiannakis, d., booth, a., & brettle, a. (2006). reliant: reader’s guide to the literature on interventions addressing the need for education and training. library & information research, 30(94), 44-51. retrieved from http://www.lirgjournal.org.uk/lir/ojs/index.php/lir/article/view/271/318   oakleaf, m. (2016). getting ready & getting started: academic librarian involvement in institutional learning analytics initiatives. the journal of academic librarianship, 42(4), 472–475. http://dx.doi.org/10.1016/j.acalib.2016.05.013   rockman, i. f. (2002). strengthening connections between information literacy, general education, and assessment efforts. library trends, 51(2), 185-198. retrieved from http://hdl.handle.net/2142/8465   evidence based library and information practice using evidence in practice   conducting state-wide research to inform practice in new south wales public libraries   ellen forsyth consultant, public library services state library of new south wales sydney, new south wales, australia email: ellen.forsyth@sl.nsw.gov.au   cameron morley manager, public library services state library of new south wales sydney, new south wales, australia email: cameron.morley@sl.nsw.gov.au   kate o’grady consultant, public library services state library of new south wales sydney, new south wales, australia email: kate.ogrady@sl.nsw.gov.au   received: 13 aug. 2015  accepted: 18 sept. 2015      2015 forsyth, morley, and o’grady. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   setting the state library of new south wales (nsw) is one of eight state and territory libraries within australia. each of these libraries plays an important role within its jurisdiction, enabling people to access rich collections and connecting them to information and to each other. the libraries are legislated to collect published materials and preserve cultural heritage, ensuring that both are available in the future. additionally, the state libraries all play key roles with the network of public libraries within their states.   the state library of nsw provides specialist advice and support to local councils providing public library services to the people of nsw across 367 locations. a dedicated team within the state library administers state government funding to councils, provides leadership on matters affecting public libraries, and identifies and researches industry-wide issues. much of the relationship between nsw public libraries and the state library of nsw is governed by legislation, specifically the library act 1939 and the library regulation 2010. this paper focuses on the specific role of researching industry-wide issues.   problem the state library of nsw established the public library network research committee in 2002. the committee was established to oversee the research program, to ensure that the public library network had input into identifying research projects, to recommend projects, and to ensure research implementation. the key factor addressed by the committee and used in identifying potential projects is the lack of research, in specific areas, about public libraries within nsw.   the following are examples of recent research projects:   ·         nsw public libraries local studies audit: this project looks at local studies collections and services with an aim to develop best practice guidelines for collection development, managing collections, providing access and service delivery, and collaborating to develop a state-wide strategy for local history collections. ·         mobile library and outreach service models: the goal of this project is to identify the current range and scope of nsw public library mobile and outreach service models and to explore the strengths, weaknesses, benefits, and opportunities of different outreach models used within nsw public libraries with the aim of developing good practice guidelines. ·         regional library models project: this project aims to ascertain the types of arrangements that are currently in place between councils for regional or cooperative library services across nsw and to explore other possibilities for management models. ·         guidelines for e-collections in nsw public libraries: the focus for this project is to identify e-collection usage patterns by format (e-books, e-audio, e-magazines), identify factors that affect usage of e-collections, and identify current e-reader or tablet usage patterns across nsw public libraries. the project outcomes include good practice guidelines for e-collection development, display, access, and collection evaluation.   evidence each project in the program has its own set of requirements. the methods for managing and delivering on the project were designed to be specific to each project. this may be a partnership with another organization, use of an external contractor, in-house research, or a combination of these methods. the research committee identified evidence for each project through stakeholder knowledge of issues of importance to the nsw public library network where information was lacking, out of date, or not specific to the nsw public library network.   prior to the start of the mobile library and outreach services project, state library staff had identified a lack of comprehensive relevant literature on mobile and outreach services for australian and specifically nsw public libraries. there were few relevant documents available and none that comprehensively covered both mobile and outreach services. the project group undertook a literature review to assess the information available and gather relevant information to inform the project outcomes. it became clear that a nsw specific publication, looking more broadly at outreach services as well as mobile library services, was required. the group collected evidence from public library stakeholders through an online survey to all library managers with visits and interviews to provide detailed information for 10 case study libraries on their range of outreach services.   the regional library models project looks at council agreements and arrangements in the delivery of library services. the need for this project arose after a 2011 change to section 12 of the library act enabled councils to propose alternate models for regional library management, and the change failed to lead to any applications from councils. the premise was that councils may be unsure of the type of arrangements they could enter into. the project group needed evidence to prove or disprove this premise and research to present possibilities for future management models.   the state library engaged the centre for local government (clg) at the university of technology sydney to undertake research to explore and recommend regional management models for nsw public libraries. a literature review was undertaken to look at current arrangements in nsw public libraries and explore models throughout australia and internationally.   clg conducted in-depth interviews (11 in total) with a representative selection of staff from the state library and relevant staff associated with regional and cooperative library models from across nsw. clg distributed an online survey to all library managers across nsw. the survey received 58 responses from 45 libraries, including 24 from stand-alone libraries, 31 from libraries involved in some form of sharing arrangement, 14 based on a regional library model, 13 based on a cooperative library model, and four based on other types of models.   evidence collected in the local studies audit showed that the combined collection of local studies material across the state is at least 2.9 million items and over 21,329 metres. for the libraries that hold these materials, almost 50% have no records on trove (a national database of digitized library material), with less than 16% of local studies collections being fully searchable on trove. over 40% of the councils surveyed have no digitization program, and there is very limited born digital content being collected. the audit identified the following issues:   some local studies collection material is digitized but not always online with clear and accurate rights statements. little born digital content is added to local studies collections, meaning limited recent material is collected. few collections have had significance assessments. inadequate collecting occurs for languages other than english and about multicultural and indigenous communities. in a small number of areas, there is no local studies collecting taking place. some collections have many boxes or shelves of material stored with no detailed descriptions of content and in some instances little knowledge of content, which may mean that some of the uncatalogued material does not deserve the storage space it is using.   the e-collections project focused on 13 nsw public libraries with high use of e-resources by clients. library staff were interviewed about the performance of their e-collections. these libraries all had turnover rates above the state-wide median of 3.46 loans per annum for these collections based on the 2012–13 data.   for the libraries with higher turnover rates:   collections are actively developed and tailored, e-collections are visible on the library website homepage, the titles are accessible in the catalogue (except in two instances), staff are trained in how to use and download e-books and e-audio, staff actively promote e-books in the library, and in many there are signs to highlight the availability of e-books, and training is provided to clients in how to use and download e-books and e-audio.   key factors emerged as a result of the project. compulsory training for staff and optional training for the public appears to have a positive effect on use. this ensures that all staff realize that assisting the public with e-resources is part of their responsibility. providing library clients with introductory sessions on e-readers and tablets that includes assistance with the use of their own devices appears to encourage use of library e-collections.   in all of these situations, the evidence was compelling because it came directly from those providing the services the research committee was exploring. the beauty of the research program is that the state library has ease of access to the public library stakeholders.   implementation   the local studies audit has resulted in the state library facilitating a course for nsw public library staff in digital repository management and in an increased focus on advocating the importance of standards and policy at the local level. the mobile library and outreach service project has delivered a report with case studies and guidelines for mobile and outreach services. as of september 2015, the regional library models project is entering its final phase, which will see the development of recommended regional and cooperative models. the e-collections project has identified key practices that libraries can implement to increase the use of these collections.   outcome   with many of these projects, the outcomes are far reaching and may not be able to be implemented immediately due to external factors. however, the reports and recommendations are made available for nsw councils via the state library of nsw website so that the information is easily accessible when required.   reflection   the approach of using evidence to inform practice delivers long-term benefits to nsw public libraries and provides access to research that no individual library could otherwise afford. there is no shortage of research needs, and because the projects provide practical outcomes, the state library has access to a large pool of willing participants from public libraries. there is a tension between this enthusiasm and receiving sufficient responses from the subjects. the projects require a diverse state-wide view to develop evidence based models of relevance to all public libraries. another challenge for the state library is encouraging uniform take-up of project recommendations by councils.   these projects deliver a state-wide overview that enables big picture planning. the research projects and groups provide a valuable community of interest, delivering positive and usable outcomes, and will be a key factor in a successful future for nsw public libraries.   references   national library of australia. trove. retrieved from http://trove.nla.gov.au/   new south wales government. library act 1939. retrieved from http://www.legislation.nsw.gov.au/viewtop/inforce/act+40+1939+first+0+n/   new south wales government. library regulation 2010. retrieved from http://www.legislation.nsw.gov.au/maintop/view/inforce/subordleg+445+2010+cd+0+n   state library of new south wales. (2015). retrieved from http://www.sl.nsw.gov.au/   evidence summary   reference management practices of postgraduate students and academic researchers are highly individualized   a review of: melles, a., & unsworth, k. (2015). examining the reference management practices of humanities and social science postgraduate students and academics. australian academic & research libraries, 46(4), 250-276. http://dx.doi.org/10.1080/00048623.2015.1104790   reviewed by: kimberly miller learning technologies librarian albert s. cook library towson university towson, maryland, united states of america email: kimberlymiller@towson.edu   received: 20 may 2016  accepted: 15 july 2016      2016 miller. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to understand patterns in reference management practices of postgraduate students and faculty members at one institution.   design – mixed methods online survey and semi-structured interviews.   setting – public research university in australia.   subjects – the survey included responses from 81 postgraduate students. semi-structured interviews were conducted with 8 postgraduate students and 13 faculty members.   methods – the researchers distributed an 18-item survey via email to approximately 800 people who previously registered for endnote training sessions. survey participants were also recruited via a website advertisement. the researchers recruited postgraduate student interview participants from the list of survey respondents. librarians invited faculty members to participate in the semi-structured interviews. interview audio recordings were transcribed and coded for data analysis.   main results – the survey found that 71.4% (n=55) of respondents used reference management software (rms) and 29% (n=22) did not. over half of the students who did not use an rms described other ad hoc or “manual” (p. 255) methods for organizing and tracking references. the majority of participants reported using endnote (67.53%, n=52), while few respondents reported using other rms tools like zotero (1.3%, n=1) or mendeley (1.3%, n = 1). software awareness (49.32%, n=36), recommendations from faculty members (30.14%, n=22), and university support (47.95%, n=35) were the primary motivations for choosing a specific rms. other important factors included ease of use (32.88%, n=24) and integration with microsoft word (46.58%, n=34). students preferred rms features that support the process of accessing and using references in a paper, and reported that technical problems were the most common frustrations. the researchers found that student interview respondents were more likely to use rms (75%, n=6) than were faculty member respondents (31%, n=4). interview respondents varied in which rms features they used, like importing references, pdf management, or “cite while you write” plug-ins (p. 258). few interviewees used the rms’s full functionality, either due to variations in workflow preferences or lack of awareness. similar to survey respondents, interviewees who did not use an rms reported their own personal practices for managing references. the time and learning curve necessary to become proficient with a particular rms, as well as how the rms fit into a particular task or workflow, influenced respondents’ decisions about software selection and use. faculty members were split with their advice to students about using an rms, with some respondents advocating that an rms can save time and trouble later in their writing processes, while others took a more cautious or hands-off approach.    conclusion – the authors conclude that measuring rms use or non-use does not reflect the real world complexity behind student and faculty member reference management practices. they suggest that librarians may want to rethink focusing on rms as the sole reference management solution. librarians should also recognize that institutional availability and support may influence users’ rms choices. a user-centred approach would account for rms and non-rms users alike, and librarians should “develop a more flexible perspective of reference management as part of an approach to researchers that aims to understand their practices rather than normatively prescribe solutions” (melles & unsworth, 2015, p. 265). instruction workshops should help students and faculty members select features or systems that match their existing research processes, rather than exclusively demonstrate the mechanics of a particular rms.   commentary   from information literacy instruction and research management (childress, 2011) to a larger suite of “researcher services” (shanks & arlitsch, 2016), academic librarians see reference management as a contemporary service to students, faculty, and staff. the current article encourages librarians to understand whether and how their existing approach to reference management responds to their user populations’ needs. pivoting the focus from specific software usage to the larger scope of a researcher’s work situates reference management as an essential component of the scholarly research lifecycle.   reviewing the article with glynn’s (2006) critical appraisal checklist suggests strengths in study design. mixed methods design allows for a multidimensional understanding of reference management practices. including the study instruments means librarians can complete a similar study of their local population. the article’s user-centred orientation also aligns with user-focused trends in the library literature. while the study’s findings about the personal, idiosyncratic nature of reference management are valuable, it is unclear how participants, pulled largely from a pool of individuals already interested in an rms workshop, reflect the wider university population among study variables. the methodology does not illustrate how many participants were recruited from the endnote workshop email list versus recruited from the website posting, or how many possible respondents had access to the website during the recruitment timeframe. survey participants are described as “postgraduate arts students” without clarifying which disciplines they come from. while the article mentions disciplinary affiliations for the interview participants, the sample size for each discipline is not included. in reporting the survey results, the per-question n-value is also unclear and can, without explanation, differ from the overall number of respondents. for example, the n-value drops from n=81 in table 1 to n=55 in table 2. it is unclear if discrepancies are due to survey attrition or survey design.   the study suggests that reference management practices are highly individualistic and personal, implying that a one-size-fits-all approach to workshops and system recommendations does not serve all users. as the authors note, “investigations should focus on what students and academics are trying to achieve, rather than how they do it” (p. 263). with this in mind, librarians should develop services that help match tools to a person’s needs, rather than adapt a person’s workflow to a particular tool. librarians must also consider their timing in such interventions, given that once users select a reference management system they are unlikely to change their practice. finally, libraries with tight budgets, who are weighing the cost of providing rms subscriptions, may be interested in the finding that university support for reference management software influences users’ decisions when selecting from among rms options.   references   childress, d. (2011). citation tools in academic libraries: best practices for reference and instruction. reference & user services quarterly, 51(2), 53-62. retrieved from http://www.jstor.org/stable/refuseserq.51.2.143   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154%20   shanks, j., & arlitsch, k. (2016). making sense of researcher services. journal of library administration, 56(3), 295-316. http://dx.doi.org/10.1080/01930826.2016.1146534   evidence summary   multiple sessions for information literacy instruction are associated with improvement in students’ research abilities and confidence   a review of: henry, j., glauner, d., & lefoe, g. (2015). a double shot of information literacy instruction at a community college. community & junior college libraries, 21(1-2), 27-36. http://dx.doi.org/10.1080/02763915.2015.1120623   reviewed by: kelley wadson librarian bow valley college calgary, alberta, canada email: kwadson@bowvalleycollege.ca   received: 3 mar. 2017    accepted: 21 apr. 2017      2017 wadson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to evaluate the impact of providing multiple information literacy (il) sessions, instead of a single “one-shot” session, to students in face-to-face and online english courses.                       design – non-experimental, using pre-test and post-test surveys for one group, and only a post-test survey for the other group.   setting – a small community college in north carolina, united states of america.   subjects – 352 students enrolled in 2 successive 3-credit english courses, excluding those under the age of 18, for a total of 244 participants.   methods – the researchers selected two english courses, eng 111 and eng 112, of which most students were required to take at least one to earn a degree or certification. after consulting with faculty, the researchers designed two workshops for each course that integrated active and group learning techniques. the eng 111 workshops covered pre-searching (e.g., mind mapping and selecting search terms) and database searching in the first session, and website analysis and research (e.g., urls, google’s advanced search, and the evaluative craap test) in the second session. the eng 112 workshops covered subject database searching in the first session and evaluative analysis of magazine and scholarly journal articles in the second session. instructors provided web-based tutorials to online course sections as a substitute for the face-to-face sessions. course assignments were the same for both online and face-to-face classes.   the researchers used anonymous online surveys. eng 111 students completed pre-test and post-test surveys for their two workshops during the fall 2014 semester. the surveys consisted of seven fill-in-the-blank and multiple-choice questions measuring pre-searching, research, and website analysis skills, and three likert-type 1-5 rating scale questions measuring comfort levels. engl 112 students completed their post-test survey in the spring 2015 semester, which consisted of the same three 1-5 rating scale questions measuring comfort levels, to further test the effectiveness of multiple sessions.   main results – the eng 111 pre-test survey had 244 (66.67% female and 33.33% male) respondents and the post-test had 150 (72.37% female and 28.69% male) respondents. when comparing results, scores increased for pre-searching, specifically understanding of methods for brainstorming search terms (9%), and for all measures of website analysis and research, namely understanding of library databases (7.63%), choosing correct evaluative criteria (4.49%), recognizing reliable top-level domains (tlds) .edu (1.15%) and .gov (11.21%), and google’s advanced search (10.43%). post-test scores decreased on the measures of understanding of a thesis statement (7%) and narrowing a topic if there’s too much information (6%). for comfort levels, neutral responses did not vary much, but there was a shift in responses from “not comfortable” to “somewhat comfortable” and “very comfortable.” across three measures, namely getting started with a research paper, library research skills, and writing an academic research paper, participants’ “not comfortable” responses decreased and their “comfortable” responses increased. the eng 112 post-test survey had 29 (60.71% female and 39.29% male) respondents and measured the same comfort levels. in addition, responses showed further improvement for all three questions. within-subject analysis of both surveys showed slight gender variations. on several pre-test and post-test measures, females scored lower than males in understanding of databases, google’s advanced search, and website analysis.   conclusion – the researchers conclude that expanding il instruction from a single “one-shot” to four sessions had a positive impact on student learning, particularly the ability to evaluate websites and to use google’s advanced search. student participants expressed increased comfort levels and confidence in their research skills. to address decreases on the post-test survey described above, the researchers planned to focus more on research topic narrowing and using thesis statements alongside the research process in future il sessions. in terms of instructional strategy, the researchers found timing the workshops closely with the course assignments was helpful and concluded that the use of hands-on, interactive elements was successful in engaging and assessing students’ understanding in the workshops.   commentary   this article adds to the substantial body of literature in the research areas of il instruction in academic libraries (detmering, johnson, sproles, mcclellan, & linares, 2014). in particular, the researchers cite considerable evidence supporting their application of active learning and multiple instructional sessions as a substitute for the “one-shot” instruction. although not explicitly mentioned, it also integrates elements of faculty-librarian collaboration and embedded librarianship, both of which have been shown to improve the effectiveness of il instruction (mounce, 2010; hamilton, 2012).   this review used the reliant tool for evaluating research on educational and training interventions in library and information science (lis) (koufogiannakis, booth, & brettle, 2006). according to this checklist, there are flaws in the study design and results that affect this article’s internal validity and relevance to lis practitioners.   the educational context and research instruments are mostly well-explained and appended to the report, but there is considerable ambiguity in the study’s purpose and design. the researchers state the article examines the effectiveness of expanding il instruction from one to four sessions. however, this is not formulated clearly as a research question or statement of purpose and, perhaps consequently, the article lacks discussion of how and why the research design was selected. additional limitations include a lack of advanced statistical analysis, such as cross-tabulation and the chi-square test of statistical significance, no differentiation in data collection methods or analysis between face-to-face and online students, and lack of pilot testing for the surveys. there is also limited description of the population; community colleges are generally recognized to be quite diverse in terms of age, ethnicity, and academic level of students. except for gender and age, the researchers do not specify what demographic questions were included in the surveys nor do they describe the institutions’ demographic make-up, which could aid lis practitioners in assessing the study’s relevance and applicability.   the one-shot is a mainstay of information literacy instruction and as the authors note, a well-recognized limitation is its attempt to cover too much content. librarians have been testing various strategies to address this, such as more effective assessment, e-learning, and even full-credit courses (mery, newby, & peng, 2012). a key strength of this article is its description of a strategy that is manageable in scope; developing e-learning and full-credit courses is not an option for many libraries, particularly smaller institutions like community colleges.   based on its design, this article shows an associative rather than a causal relationship. lis practitioners may find it helpful as a pre-experimental or case study providing descriptive insights into faculty-librarian collaboration, active learning techniques, and the potential for multiple sessions to lower library anxiety and bolster students’ confidence in their research skills.   references   detmering, r., johnson, a. m., sproles, c., mcclellan, s., & linares, r. h. (2014). library instruction and information literacy 2013. reference services review, 42(4), 603-715. http://dx.doi.org/10.1108/rsr-07-2014-0028   hamilton, b. j. (2012). embedded librarianship: tools and practices. chicago, il: ala techsoure.   koufogiannakis, d., booth, a., & brettle, a. (2006). reliant: reader’s guide to the literature on interventions addressing the need for education and training. library & information research, 30(94), 44-51. http://www.lirgjournal.org.uk/lir/ojs/index.php/lir/index   mery, y., newby, j., & peng, k. (2012). why one-shot information literacy sessions are not the future of instruction: a case for online credit courses. college & research libraries, 73(4), 366-377. retrieved from http://crl.acrl.org/index.php/crl/issue/archive   mounce, m. (2010). working together: academic librarians and faculty collaborating to improve students' information literacy skills: a literature review 2000–2009. the reference librarian, 51(4), 300-320. http://dx.doi.org/10.1080/02763877.2010.501420     research article   the role of institutional repositories in the dissemination and impact of community-based research   cara bradley research & scholarship librarian university of regina library regina, saskatchewan, canada email: cara.bradley@uregina.ca   received: 17 may 2021                                                              accepted: 7 aug. 2021        2021 bradley. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   data availability: bradley, c. (2021). canadian community-based research unit outputs, 2010-2020 (v1) [data]. scholars portal dataverse. https://doi.org/10.5683/sp2/gyvkn6   doi: 10.18438/eblip29972     abstract   objective – the goals of this study were to 1) characterize the quantity and nature of research outputs created by or in cooperation with community-based research units (cbrus) at canadian universities; 2) assess dissemination practices and patterns with respect to these outputs; 3) understand the current and potential roles of institutional repositories (irs) in disseminating community-based research (cbr).   methods – the researcher consulted and consolidated online directories of canadian universities to establish a list of 47 english language institutions. working from this list of universities, the researcher investigated each in an attempt to identify any cbrus within the institutions. ultimately, these efforts resulted in a list of 25 cbrus. all but 1 of these were from universities that also have irs, so 24 cbrus were included for further analysis. the researcher visited the website for each cbru in february 2021 and, using the data on the site, created a list of each project that the cbru has been involved in or facilitated over the past 10 years (2010-2020). an excel spreadsheet was used to record variables relating to the nature and accessibility of outputs associated with each project.   results – these 24 cbrus listed 525 distinct projects completed during the past 10 years (2010-2020). the number of projects listed on the cbru sites varied widely from 2 to 124, with a median of 13. outputs were most frequently reports (n=375, which included research reports, whitepapers, fact sheets, and others), with journal articles (n=74) and videos (n= 42) being less common, and other formats even less frequent. the dissemination avenues for these cbru projects are roughly divided into thirds, with approximately one third of the projects’ results housed on the cbru websites, another third in irs, and a final third in “other” locations (third party websites, standalone project websites, or not available). some output types, like videos and journal articles, were far less likely to be housed in irs. there was a significantly higher deposit rate in faculty or department-based cbrus, as opposed to standalone cbrus.   conclusion – the results of this study indicate that academic libraries and their irs play an important role in the dissemination of cbr outputs to the broader public. the findings also confirm that there is more work to be done; academic librarians, cbru staff, and researchers can work together to expand access to, and potentially increase the impact of, cbr. ideally, this would result in all cbru project outputs being widely available, as well as providing more consistent access points to these bodies of work.     introduction   most canadian universities, like similar institutions worldwide, have a tripartite mandate that includes teaching, research, and service. some researchers and their institutions have devoted considerable time and resources to conducting cbr, an undertaking that combines research and service in an effort to investigate pressing community issues. cbr (with variations known by names like community-based participatory research, community-engaged research, collaborative research, and others) is widely viewed as one way that universities can build relationships with and have an impact on their wider communities, and be “of and not just in their community” (watson, 2003, as cited in macpherson et al., 2017, p. 298).   cbr in its truest form is a partnership between academics and community members to investigate research topics of common concern. ideally, cbr sees the involvement of community partners throughout the entire research process, from identifying the question or problem, through designing the research study and collecting data, and on to sharing and disseminating the research findings widely (not only among other academics, but crucially among the community participants in the research and the wider population). by addressing real, local concerns, cbr has the potential to improve the lives of residents. even a cursory look at the titles of cbr outputs reveals that many focus on important social justice issues and that a number of individuals and groups stand to benefit from broad sharing of these research findings. access to these research results or “informational justice” (mathiesen, 2015) is an important consideration in ensuring that cbr achieves its greatest possible impact.   widespread dissemination has frequently been highlighted as an area where cbr falls short, but there have been few efforts to objectively assess what happens to cbr outputs upon completion, and no studies on how academic libraries (who routinely assist researchers with dissemination to academic audiences) contribute to cbr dissemination efforts. thus, the goals of this study were to 1) characterize the quantity and nature of research outputs created by or in cooperation with cbrus at canadian universities; 2) assess dissemination practices and patterns with respect to these outputs; 3) understand the current and potential roles of irs in disseminating cbr.   literature review   several studies highlight both the importance and difficulty of communicating the results of cbr beyond academia. most of these studies question key stakeholders in cbr to collect their assessments of the challenges and state of cbr communication. bodison et al. (2015) provide an example of this type of work. they conducted a discussion forum with multiple stakeholders in cbr and found that “research findings are rarely meaningfully communicated back to those who participated, if communication about the findings occurs at all” (bodison et al., 2015, p. 817). such studies are useful and provide direction for improvements, but they do not provide objective assessments of current cbr dissemination practices. one exception is chen et al. (2010), who conducted a systematic review of cbr publications to assess efforts in disseminating findings beyond scholarly journal articles, in order to find out what is really happening regarding wider dissemination. they found that despite the fact that widespread dissemination of findings is a key tenet of cbr, “substantial challenges to dissemination remain” (chen et al., 2010, p. 377). to date, this is one of very few studies assessing what actually happens to cbr results once projects are concluded. even less has been written about what role academic libraries might play in the dissemination of cbr. the most relevant literature investigates library contributions to increasing research impact within the academy. this is supplemented by more recent work that has started to consider the role of academic libraries in dissemination outside of the academy and contributions to public engagement and the common good.   recent years have seen academic libraries expand from primarily supporting teaching and learning in their universities, to an increased emphasis on support for faculty and graduate student research. as recently as 2011, maccoll and jubb noted that “it is hard to avoid the conclusion that libraries in recent years have been struggling to make a positive impact on the scholarly work of researchers, but having relatively little effect” (p. 5). this is gradually changing, driven in no small part by increasing requirements around national research impact assessment initiatives like the research excellence framework requirements in the uk and the impact assessment requirement as a component to the excellence in research for australia (era) national framework. librarians are increasingly called upon to assist their organizations in demonstrating the impact of their research through using conventional and alternative (“alt”) metrics. corrall et al. (2013) surveyed academic librarians in four countries to better understand the scope and nature of their support for research activities in their institutions. the results confirmed that national research assessment exercises had breathed new life into bibliometric services in many libraries and that “the focus of bibliometric activity . . . has shifted from collection development to research evaluation and impact assessment for individual researchers, academic groups, organizational units and whole institutions” (corrall et al., 2013, p. 666). they concluded, though, that there remain “significant opportunities for further engagement” in this type of work (corrall et al., 2013, p. 666).   in 2014, kennan et al. revisited their results to further analyze the skills required for librarians to succeed in supporting both research impact assessment and research data management. they found that many librarians reported needing additional training and skills development to undertake this work with confidence. nicholson and howard (2018), in their study of the gap between core competencies required for research support work (as evidenced in position postings) and the skills of library and information professionals, similarly found that “it would be beneficial to build upon the skillsets of current and new lis professionals” regarding research engagement and impact topics (p. 144).   given et al. (2015) looked more specifically at the need to disseminate scholarly research and expand its impact to those outside of the academy; they interviewed 10 australian academics in an effort to better understand their conceptions of research impact in both academic and non-academic settings, to gain participants’ insight into “existing or needed university-based supports to foster societal engagement” (p. 4). they found that academics generally felt ill-equipped to disseminate their work beyond traditional channels (scholarly journals and conferences) and these academics “did not identify any existing library supports that could be applied to their work in the societal impact space” (given et al., 2015, p. 6). the researchers encouraged further efforts, commenting that “academic librarians and information science researchers can be proactive . . . to ensure that researchers and institutions are well-informed and well-prepared to engage with their communities in appropriate and productive ways” (given et al., 2015, p. 8).   one of the most common ways for researchers to extend the impact of their work beyond the academy is through the creation of research outputs that differ from traditional journal articles and scholarly books. more accessible outputs like whitepapers and policy documents are increasingly likely to reach and impact policy makers, just as videos, recordings, fact sheets, websites, and blog posts may be more easily accessed and readily understood by the general public. many of these outputs fall under the broad category of grey literature and some researchers have started to investigate the role of irs in collecting, providing access, and preserving these outputs. searle notes that “librarians involved in scholarly communication must move quickly beyond a limited set of formal publication types towards a wide range of more complex and arguably more at-risk research outputs” and that “grey literature struggles to find a place in library strategies despite the evidence of its high value to communities outside academia” (agate et al., 2017, p. 2). the following year, marsolek et al. (2018) conducted a study of the discoverability of grey literature in irs and commercial databases; they found that 95% of the 115 irs included in their study contained grey literature, but concluded that only 63% of irs seemed to be actively working to collect it (p. 15). theses and dissertations were the most commonly collected grey literature found in irs, while others like technical reports, working papers, blogs, standards, and protocols were much less likely to be included. marsolek et al. (2018) concluded that:   the marriage between irs and grey literature could elevate the value of irs to the research community. irs could make a substantial difference in ensuring grey literature’s preservation, increasing its reach, and, in many cases, providing a form of legiti­macy to these items published outside traditional realms. (p. 17)   moore et al. (2020) explored how use of irs to provide access to grey literature can also help universities increase public engagement and achieve community service goals; they saw an important role for irs in the “recognition, dissemination, and preservation of the outputs of community-based research”, outputs which are often grey literature (p. 117). moore et al. (2020) state how a repository containing grey literature produced during the course of cbr helps the university to “present a more holistic picture of its community partnerships and institutionalize public engagement into something much more integral and essential to campus (and local) culture” (p. 117). they describe how the repository at the university of minnesota became a “conduit between campus units and community partners” (moore et al., 2020, p. 117). in the process, the ir began to “play a strategic role in public engagement . . . by acting as a common good to showcase, contextualize, disseminate, preserve, and institutionalize this content” and came to “support the research, teaching, and outreach mission of an engaged campus, provide a service as a public good, and contribute to an informed citizenry in society” (moore et al., 2020, p. 126). this echoes makula’s (2019) description of the university of san diego’s repository as moving from its position as “primarily a platform, a system, or a service” to becoming “a bridge between the university of san diego and the outside world, an instrument helping to build and nurture institutional-community relationships, foster collaboration, and cultivate good will” (para. 12).   heller and gaede’s (2016) work expands the notion of the ir as a common good by emphasizing that “libraries must move beyond pragmatic justification for institutional repositories . . . [and] understand their work in the context of social justice, lest they become complicit in unjust scholarly communication systems” (p. 3). they articulate a “social justice impact metric” based on search engine access to social justice-related repository content, as well as access to all repository content by developing countries, to express the social justice impact of irs (heller & gaede, 2016, p. 3). they offer this metric as a way for other librarians to assess their own open access activities in terms of their level of success in contributing to the public good, by reaching members of the public who would not otherwise have access to this important content. perhaps even more important than the metric they offer, though, is the insight that:   open access to the scholarly and creative output of our institutions contributes a vital academic good insofar as prestige and reputation are concerned, but the social good is something extraordinary and should excite us more. in reclaiming our role as facilitators of democratic discourse, we demonstrate the change we believe in and live out our bibliography. (heller & gaede, 2016, p. 15)   mathiesen (2015) offers the theory of “information justice” as a framework for better understanding the contributions of library work to social justice. she describes “informational justice” as a facet of social justice concerned with people as “seekers, sources, and subjects of information” (p. 199). she notes that:   what makes informational justice of central concern, and thus why libraries and other information services are particularly important, is the fact that informational injustice produces and reinforces other forms of social injustice, while information justice undermines systems of social injustice. indeed, informational justice serves as a good proxy for social justice writ large, because opportunities to receive and share information are central means for enhancing all aspects of people’s lives. (mathiesen, 2015, pp. 204-5)   mathiesen’s (2015) elaboration of “idistributive justice”, which is terminology for “equitable distribution of access to information” is particularly relevant when thinking about the library’s role in making cbr more widely available to those who may benefit from but lack access (p. 207). for librarians engaged in the many facets of research impact work for institutions, it is important to ask whether they are doing all they can to extend research impact to the broader community beyond academia and contribute to informational justice.   methods   university involvement with cbr is difficult to quantify and track. some is coordinated by units, either at the department, faculty, or institutional level, that facilitate partnerships between community organizations and researchers. research associated with cbrus was chosen as the subject of study for this paper because it provides a manageable starting point for exploring the nature and accessibility of cbr outputs.   even this approach is not without its challenges. the language for referring to this type of research varies and seems to be in transition, including names such as “research shop”, “community-based research”, “community-engaged research”, and “community-based participatory research”, among others. as well, there is no comprehensive list of department, faculty, or university cbrus in canadian universities. as such, the researcher consulted and consolidated online directories of canadian universities to establish a list of 47 english language institutions. universities or colleges that are smaller affiliates of larger institutions were excluded based on difficulties distinguishing their contributions from those of their larger partner or parent organizations.   working from this list of universities, the researcher investigated each in an attempt to identify any cbrus within the institution. this involved viewing lists of research centres and institutes on each university’s webpage, searching these institutional webpages for variations of “community-based research”, and conducting google searches combining this concept with the name of each institution. multi-institution cbrus (for example, nova scotia’s clari) were excluded due to the anticipated difficulty of tracking outputs in the repositories of specific institutions at later stages in the research process. ultimately, these efforts resulted in a list of 25 cbrus. all but one of these were from universities that also have irs, so 24 cbrus were included for further analysis.   the researcher visited the websites for each cbru in february 2021 and used the data on the websites to create a list of projects that the cbrus had been involved in or facilitated over during the past 10 years (2010-2020). the researcher used an excel spreadsheet to record variables relating to the nature and accessibility of outputs associated with each project. these variables included:   -        type of outputs (document, video, website, and others) -        availability of output in its entirety (i.e., full-text, entire video, journal articles, and others) on: o   cbru websites o   irs (the names of projects and lead researchers were also searched in the irs, even in instances where there was no link to the ir from the cbru webpage) o   third party websites o   dedicated project websites   the researcher then analyzed the findings to learn more about the dissemination of cbr and, in particular, the role of the ir in disseminating the results of this research.   results   as mentioned above, this methodology produced a list of 47 english-language canadian universities, within which 24 cbrus were identified in institutions that also have irs. as shown in figure 1, these cbrus were housed in 19 institutions, with some having 2 distinct cbrus. sixteen of the cbrus were at the institutional level (that is, not located within a specific faculty or department); 7 were housed within faculty or departments, and 1 was a faculty member’s laboratory.   between them, these 24 cbrus listed 525 distinct projects completed during the past 10 years (2010-2020). projects that were clearly still underway or in progress were excluded from the analysis, given they could not yet be expected to have produced outputs for analysis. the number of projects listed on the cbru sites varied widely from 2 to 124, with a median of 13. figure 2 shows a breakdown of outputs by type. the number of outputs exceeds the number of projects because some projects produced more than one output type.   as figure 2 clearly shows, reports (which includes research reports, whitepapers, fact sheets, and others) was the largest category of outputs (n=375). “unique” (n=13) includes output types that only appeared once across all the data (e.g., electronic book, blog, storytelling event, among others), while “unclear” (n=36) includes projects whose description suggests that there was an output generated, but its nature is not specified nor is the work provided.   figure 1 type of community-based research unit.   figure 2 cbru outputs by type.   after characterizing the types of outputs emerging from cbrus, the study sought to assess if and how research outputs were made accessible to interested readers. some outputs were available in more than one place (e.g., ir and cbru websites), so the total output locations in figure 3 exceed the 525 projects included in the analysis. the “cbru website” includes outputs (n=197) available in their entirety (full report, entire project video, and others) on the units’ webpages. “institutional repository” similarly indicates that an entire output has been deposited in the ir (n=193). “third-party website” describes instances where the cbru websites link to a third-party website where the research output can be found (n=104). “project website” indicates that the cbru site links to a stand-alone website, created to share the results of that particular project (n=19). “available for purchase” refers to instances where the cbru websites either link to (n=22) or provide citations without links (n=9) to a journal article that requires an institutional subscription or personal purchase to access the research output. sixty-five projects are categorized as “not available” because the cbru websites suggest that there have been outputs from the research, but there is no access information provided or the only method provided is a dead link.   figure 3 dissemination of research outputs.   figure 4 journal articles by publication type and ir availability.     third-party websites figure prominently in the dissemination of research outputs from the cbrus, with 104 of the projects (19.8%) using this as a means of sharing results. these thirdparty sites can be divided into three broad categories: video sites like youtube and vimeo (37 videos), journal websites (74 articles), and websites of partner or funding organizations that contain the research outputs (n=24). another 19 projects (3.6%) have separate project websites to share results. importantly, in terms of access, there were 10 dead links from projects listed on cbru websites to third-party or project websites.   outputs were not freely available for 72 of the projects (13.7%). this included 65 projects that indicated reports or other outputs existed and either did not provide access or a link, or else provided a dead link, as well as 7 for which outputs could only be viewed by purchasing access to paywalled journal articles that were not available in the corresponding irs. overall, there were 31 paywalled articles identified as sites for research output, but most supplemented other output methods and did not therefore impede access to the outcomes of the project, except for the 7 highlighted above. this compares to 43 open access articles listed as outputs of these research projects. figure 4 shows the breakdown of journal articles by publication type, as well as the portion of each type that are also deposited in the irs (4/31 or 12.9% of paywalled articles and 12/43 or 27.9% of open access articles).   overall, a total of 193 (36.8%) of the projects resulted in research outputs than can be found in the institutions’ repositories. figure 5 shows that there are some notable differences in the rates of outputs deposited in irs when the data were further broken down. the 7 faculty or department-based cbrus had an ir deposit rate of 69.4% (154 of 222 projects), while the institutional-level cbrus only had an ir deposit rate of 13% (39 of 299 projects). none of the projects emerging from the faculty member research laboratories were captured by their irs. thus, although only 7 of the 24 cbrus (29.2%) were faculty or department based, they accounted for 154 of the 193 (79.8%) projects for which research outputs were deposited in irs.   figure 5 ir deposit for projects, by cbru type.   interestingly, only 177 of the 193 projects found in irs contained a link from the cbru websites to the relevant repository contents. thus, the output of 16 projects (8.2%) are in fact held in irs but would not be found by readers or researchers viewing the cbru websites.   while 36.8% of research outputs from these cbrus can be found in the corresponding irs, figure 6 shows that the frequency with which these outputs are deposited varies widely depending on the nature of the output. 48% of reports (n=180) have been deposited, while the same can be said of 21.6% of journal articles (n=16). deposit rates are much lower for items that are not typical word or pdf files; only 9.5% of videos (n=2) and 5% of posters (n=2) have been deposited.   discussion   the dissemination avenues for these cbru projects are roughly divided into thirds, with approximately one third of the project results housed on cbru websites, another third in irs, and a final third in “other” (third party websites, standalone project websites, or not available). this demonstrates a level of inconsistency among dissemination practices that would make it difficult for individuals interested in this type of research to know how to proceed in locating it. although posting research outputs on cbru, third-party, or standalone websites may aid findability in the short term, sole use of these sites generates problems over the long term. the problems of “content drift,” where the contents of webpages change over time and “url decay” (i.e., urls no longer active) have been well-documented (jones et al., 2016; oguz & koehler, 2016). irs, by contrast, provide “safe storage, persistent urls, backup, and possibly migration if it is needed in the future” and reduce cbru website and file hosting workloads (marsolek et al., 2018, p. 5). many cbru-involved outputs remain relevant over the longer term, and continued access is important for faculty members seeking to include these materials in promotion and tenure applications.   figure 6 ir deposits by output type.   there was also a marked difference in the deposit rate for different output formats. “reports” which included word and pdf text files, were deposited at a much greater rate than alternative formats like videos and posters, among others. the reason for this is unclear but warrants further investigation, since research has shown that some of these alternative formats have the greatest potential to impact the general public. possible explanations include ir collection policies that align with traditional (print) collection policies, the failure of librarians to actively collect materials in these formats, or lack of awareness among the campus community (cbru staff and researchers) that other formats are also welcome in irs. it was somewhat surprising that journal articles emerging from these cbru projects were not more consistently included in the irs (only 21.6% had been deposited), given that the collection of journal articles has long been a priority for many irs and many libraries have developed policies, workflows, and advocacy tools to support journal article collection.   cost was less of an access barrier to cbru-involved work than expected; while 31 paywalled journal articles emerged from the work of these cbrus, there were only 7 cases where this prevented all access to the research findings. the other 24 paywalled articles were supplemented with freely available reports or summaries available elsewhere (ir, cbru site, third-party site, standalone site). the lack of availability of any findings associated with a research project was, conversely, more of a problem that anticipated, with 65 (12.4%) projects providing no information about outputs or providing only a dead link. there were also instances where the output was available, but findability was an issue. in several instances, cbru outputs could only be found in irs, but there was no indication on the cbru site that this was the case. this has implications for accessibility, as only those who thought to conduct a separate search of the ir would have access to the full research output. also interesting was the discrepancy between the ir deposit practices of institutional vs. faculty or departmental cbrus. faculty or department cbrus deposited at a far greater rate than institution-level cbrus (69.4% vs. 13%). this large difference warrants further investigation, as it may provide insights into how deposit rates by institutional cbrus can be increased. many canadian academic libraries still operate with some variation of a subject liaison librarian model, usually supplemented by functional positions (scholarly communications librarian, systems librarian, among others). it would be valuable to better understand whether the relationship between the subject liaison librarians and faculty or departmental cbrus is important to achieving this relatively high rate of deposit, and how this success could be transferred to institutional cbrus, whose staff may not have (or be aware of) a connection with a subject specialist.   there were a few instances of institutions that had adopted unique practices of dissemination that do not fit neatly into the results above, but are relevant to note as examples of possible approaches to expanding the reach of cbr. the university of british columbia’s dtes portal (https://dtesresearchaccess.ubc.ca) is an impressive effort to expand access and awareness to research results relevant to the issues facing vancouver’s downtown eastside. the mandate of the dtes portal differs somewhat from that of the cbrus included in this study, in that they aim to collect material of interest to the community regardless of creator or origin (not necessarily involving academia) and to profile this material in a standalone database. their curatorial statement (https://infohub-2019.sites.olt.ubc.ca/files/2020/07/curatorial-statement-2020-final.pdf), however, also indicates that they collaborate with the ubc ir in their collection of relevant ubc research outputs. many institutions lack the resources to create a standalone topic repository of this nature, but the dtes portal does provide a model that might be embedded within existing irs. at another institution, the cbru website simply links to the relevant section of the ir that lists all of the cbru projects (including the full outputs). this means that all cbru items are included in the ir, saving the cbru the work of creating and maintaining a list of projects and associated outputs. these are examples of different ways for academic libraries to approach utilizing their irs to collaborate with cbrus in the dissemination of cbr.   there are some limitations to the methods used in this study. cbrus represent only a portion of the cbr undertaken at canadian universities. it would be useful conduct a study of researchers doing cbr without the involvement of cbrus in order to understand if their dissemination practices differ from those observed in this study. another limitation is the reliance on the cbru websites to identify projects as well as outcomes. it is possible that some cbru-involved projects were not listed on the websites and therefore these outputs were excluded from the analysis. a future study might reduce this risk by asking cbrus to provide a list of all the projects in which they were involved over a given time frame. additionally, it is possible that in some instances cbrus or researchers have chosen to communicate results to community members in other ways that would not be captured in this type of study (e.g., a seminar presenting results to community members or a report sent directly to a partnering community organization). this would be a suitable way to communicate with research participants and community stakeholders, but it prevents other individuals and organizations from benefiting from the results of the research. surveys, interviews, or focus groups with cbru staff and affiliated researchers might be the best way to supplement the results of this study and deepen understanding of cbru research dissemination practices and the role that academic libraries and their irs might play in this process.   conclusion   the results of this study indicate that academic libraries and their irs play an important role in the dissemination of cbr outputs to the broader public. the findings also confirm that there is more work to be done; academic librarians, cbru staff, and researchers can work together to expand access to and potentially increase the impact of cbr. ideally, this would result in all cbru project outputs being widely available, as well as providing more consistent access points to these bodies of work. irs are not, by any means, the entire solution to the complex issues of cbr dissemination, but their more consistent use would be one piece of the puzzle. additional services and supports for cbr could build upon the relationships established in implementing such a service, providing a way for academic librarians to contribute to the common good and amplify the social justice efforts of their universities. this work is one way to “reclaim . . . our role as facilitators of democratic discourse” (heller & gaede, 2016, p. 15) and contribute to the realization of what mathiesen (2015) termed “informational justice” (p. 199).   references   agate, n., clement, g., kingsley, d., searle, s., vanderjagt, l., waller, j., schlosser, m., & newton, m. (2017). from the ground up: a group editorial on the most pressing issues in scholarly communication. journal of librarianship and scholarly communication, 5(1), ep2196. https://doi.org/10.7710/2162-3309.2196   bodison, s. c., sankaré, i., anaya, h., booker‐vaughns, j., miller, a., williams, p., & norris, k. (2015). engaging the community in the dissemination, implementation, and improvement of health‐related research. clinical and translational science, 8(6), 814-819. https://doi.org/10.1111/cts.12342   chen, p. g., diaz, n., lucas, g., & rosenthal, m. s. (2010). dissemination of results in community-based participatory research. american journal of preventive medicine, 39(4), 372-378. https://doi.org/10.1016/j.amepre.2010.05.021   corrall, s., kennan, m. a., & afzal, w. (2013). bibliometrics and research data management services: emerging trends in library support for research. library trends, 61(3), 636-674. https://doi.org/10.1353/lib.2013.0005   given, l. m., kelly, w., & willson, r. (2015). bracing for impact: the role of information science in supporting societal research impact. proceedings of the association for information science and technology, 52(1), 1-10. https://doi.org/10.1002/pra2.2015.145052010048   heller, m., & gaede, f. (2016). measuring altruistic impact: a model for understanding the social justice of open access. journal of librarianship and scholarly communication, 4(0), ep2132. https://doi.org/10.7710/2162-3309.2132   jones, s. m., van de sompel, h., shankar, h., klein, m., tobin, r., & grover, c. (2016). scholarly context adrift: three out of four uri references lead to changed content. plos one, 11(12), e0167475. https://doi.org/10.1371/journal.pone.0167475   kennan, m. a., corrall, s., & afzal, w. (2014). “making space” in practice and education: research support services in academic libraries. library management, 35(8/9), 666-683. https://doi.org/10.1108/lm-03-2014-0037   maccoll, j., & jubb, m. (2011). supporting research: environments, administration and libraries. https://www.oclc.org/content/dam/research/publications/library/2011/2011-10.pdf   macpherson, h., davies, c., hart, a., eryigit-madzwamuse, s., rathbone, a., gagnon, e., buttery, l., & dennis, s. (2017). collaborative community research dissemination and networking: experiences and challenges. gateways: international journal of community research and engagement, 10, 298-312. https://doi.org/10.5130/ijcre.v10i1.5436   makula, a. (2019). “institutional” repositories, redefined: reflecting institutional commitments to community engagement. against the grain, 31(5). https://www.charleston-hub.com/2019/12/v315-institutional-repositories-redefined-reflecting-institutional-commitments-to-community-engagement/   marsolek, w. r., cooper, k., farrell, s. l., & kelly, j. a. (2018). the types, frequencies, and findability of disciplinary grey literature within prominent subject databases and academic institutional repositories. journal of librarianship and scholarly communication, 6(1), ep2200. https://doi.org/10.7710/2162-3309.2200   mathiesen, k. (2015). informational justice: a conceptual framework for social justice in library and information services. library trends 64(2), 198-225. https://doi.org/10.1353/lib.2015.0044   moore, e. a., collins, v. m., & johnston, l. r. (2020). institutional repositories for public engagement: creating a common good model for an engaged campus. journal of library outreach and engagement 1(1), 116-129. https://doi.org/10.21900/j.jloe.v1i1.472   nicholson, j., & howard, k. (2018). a study of core competencies for supporting roles in engagement and impact assessment in australia. journal of the australian library and information association, 67(2), 131-146. https://doi.org/10.1080/24750158.2018.1473907   oguz, f., & koehler, w. (2016). url decay at year 20: a research note. journal of the association for information science and technology 67(2), 477-479. https://doi.org/10.1002/asi.23561   editorial   evidence summary theme: data creation, access, and services   fiona inglis associate editor (evidence summaries) liaison librarian, science wilfrid laurier university waterloo, ontario, canada email: finglis@wlu.ca      2023 inglis. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30423     this is our first themed selection of evidence summaries since completing our exploration of the six domains of librarianship identified by koufogiannakis et al. (2004). in this and future issues, we will be choosing themes based on trends that we are seeing in library research and emerging topics of interest. we already have some great topics planned for the next few issues, but if you have a suggestion for a topic, you are very welcome to share it with us.   for this issue, we have chosen the very broad topic of data and the ways in which libraries are engaged in the use, access, creation, and storage of many different kinds of data. research on this topic is occurring in all types of libraries and heritage institutions and covers a wide range of domains including information access and retrieval, collections, outreach, and education.   the six evidence summaries in this issue cover data services and data availability within the academic research environment, data collection and access to government data in public libraries, and open data and born-digital data in museums and archives.  we are also happy to report that five of the six articles being appraised are available open access.   we hope that these summaries will provide inspiration for services and research in your own context.   references   koufogiannakis, d., slater, l., & crumley, e. (2004). a content analysis of librarianship research. journal of information science, 30(3), 227–239. https://doi.org/10.1177/0165551504044668 evidence based library and information practice commentary   evidence based health sciences librarians   jonathan deforest eldredge associate professor biomedical informatics research, training, and scholarship health sciences library and informatics center university of new mexico albuquerque, new mexico, united states of america email: jeldredge@salud.unm.edu   received: 28 feb. 2016   accepted: 4 mar. 2016      2016 eldredge. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     evidence based library and information practice (eblip) has become the most visible and enduring institution of our international eblip community of practice (wenger, 1998; eldredge et al., 2015). congratulations to the hundreds of colleagues dedicated to creating this inter-sectoral and international peer-reviewed forum that has been so open to exploring many diverse viewpoints while embracing the critical importance of evidence! librarians from every sector know that eblip decision making consists of taking into account the users’ preferences, one’s professional expertise, and the best available evidence. regardless of one’s specific library sector, our practices are heavily influenced by our common librarian (and i would suggest our eblip) ancestor john cotton dana. he insisted on turning our profession away from the physical trappings of libraries. dana instead focused on our shared cause with our user communities (dana, 1916a; dana, 1916b). academic, public, special, and school librarians alike, for the past century, have continued to assess their users’ information needs and to find ways to meet those needs. librarians want to remove all barriers between their users and the desired information.   health sciences librarians (hsls) similarly seek to fully integrate themselves within their communities of users. for most hsls, this means that they collaborate with other health professionals in pursuit of the clinical, teaching, or research missions of their academic centre or hospital. it can be a fast-paced, high-stakes environment where other health professionals depend upon hsls to be accurate and comprehensive. in recent years hsls, have unshackled themselves from physical libraries due to a high proportion of collections resources now in digital format (plutchak, 2012). their collaborations with other health professionals can occur far from their physical buildings. this trend cannot be considered new (pratt, 1991), although it has been accelerating for the past decade (cooper & crum, 2013). in line with this trend i was rarely in my physical library during the early years of my career when i was a chief of collection resources. instead, i was frequently out meeting with members of my user community so i could better understand their information needs.   hsls have integrated many of the norms, values, and standards held by the health professionals.  who are collaborative members of their user communities (eldredge, 2014). some of these specific values include accountability, credibility, replicability, and transparency. professionals can no longer hide behind a veil of professional autonomy. hsls, like all health professionals in this environment of accountability, can be challenged on their decisions and must be able to respond in a transparent manner. hsls must produce the kinds of evidence, when explaining their decisions, that will convince their health professional colleagues. the health professions generally embrace evidence based practice (ebp) and this approach permeates the organizational cultures of most health care organizations. ebp specifically pertains to the clinical, educational, and research aspects of the health professions. while some minor differences exist in what various health professions integrate as evidence into their practices, the core characteristics of ebp allow the different health professions to speak the same evidentiary language to their physician, nurse, pharmacist, public health, physical or occupational therapist colleagues. eblip similarly enables hsls to speak that same language and enlist similar forms of evidence. for example, if asked to defend a budget for collection resources, by using eblip approaches hsls can marshal the types of compelling evidence such as a cohort study or a randomized controlled trial that will convince decision makers. in the clinical realm, hsls who understand the underlying principles of ebp and possess these skills, can speak the same evidence-based language as clinicians. hsls can teach the first two steps in the ebp process (question formulation and searching) and assist with teaching the third step of critical appraisal by utilizing filters that isolate higher forms of evidence. this dynamic underscores the need for hsls to downplay their differences and emphasize their similarities in practicing their specific form of ebp. this approach will help hsls to enhance communication, develop new roles, and possibly even gain additional respect from the health professionals with whom they collaborate.   hsls were central to the creation of ebp in the health professions. the historical evidence indicates that these other health professions needed hsls to create ebp. for example, hsls developed certain sophisticated tools such as pubmed for identifying and interpreting authoritative evidence for making decisions (eldredge, 2008). hsls continue to contribute their essential skills to ebp within other health professions, since all forms of ebp in the health professions rely largely upon authoritative research-based information as the basis for most evidence.   this co-creation dynamic makes it inaccurate to depict hsls as imitating other health professions’ respective forms of ebp. to be accurate, hsls and other health professions’ variants of ebp co-evolved. hsls hold the distinct position among all types of librarians that they not only engage in their own variation of ebp, they also provide the evidence sources and the services to make ebp possible for the health professions.   when eblip began publishing ten years ago hsls were defining, in large part, the eblip process and the levels of evidence (eldredge, 2002). hsls had co-created ebp so it was only natural that librarians from other sectors would adapt what already existed. within several years other types of libraries were joining in the eblip movement and challenging the hsl approach to eblip with its heavy health professions flavor. in recent years there have been attempts to develop a unifying model of eblip that would apply to all sectors of librarianship. will that happen? perhaps, but i remain skeptical because the forms of evidence that are acceptable in the health professions can be vastly different from the forms of evidence that might be the currency of management studies, for example. in the meantime, we have so much to learn from one another in our community of practice, in all its diversity and enthusiasm, here at evidence based library and information practice for many years to come. happy anniversary!   references   cooper, i. d., & crum, j. a. (2103). new activities and changing roles of health sciences librarians: a systematic review, 1990-2012. journal of the medical library association, 101(4), 268-77.  http://dx.doi.org/10.3163/1536-5050.101.4.008   dana, j. c. (1916a). hear the other side [1898]. in: libraries: addresses and essays. new york: h.w. wilson, 8-14. [reprinted in: librarian at large: selected writings of john cotton dana. edited by c.a. hanson. washington, dc: special libraries association, 1991]   dana, j. c. (1916b). making a library known [1905]. in: libraries: addresses and essays. new york: h.w. wilson, 1916, 115-126. [reprinted in: librarian at large: selected writings of john cotton dana. edited by c.a. hanson. washington, dc: special libraries association, 1991]   eldredge, j. d. (2002 fall). evidence-based librarianship: levels of evidence. hypothesis, 16(3), 10-13. retrieved from http://www.mlanet.org/p/cm/ld/fid=737&tid=511&sid=647         eldredge, j. d. (2008). evidence-based practice. in m.s. wood (ed.), introduction to health sciences librarianship (241-269). new york: haworth press.   eldredge, j. d. (2014). the evolution of evidence based library and information practice part iii: revitalizing the profession through eblip. evidence based library and information practice, 9(1), 62-73. http://dx.doi.org/10.18438/b84c7f    eldredge, j. d., holmes, h. n., & ascher, m. t. moving the eblip community’s research forward. evidence based library and information practice, 10(2), 170-173. http://dx.doi/org/10.18438/b8j60s   plutchak, t. s. (2012). breaking the barriers of time and space: the dawning of the great age of librarians. journal of the medical library association, 100(1), 10-19. http://dx.doi.org/10.3163/1536-5050.100.1.004   pratt, g. f. (1991). liaison services for a remotely located biotechnology research center. bulletin of the medical library association, 79(4), 394-401. retrieved from http://www.ncbi.nlm.nih.gov/pmc/articles/pmc225587/   editorial evidence based library and information practice 2009, 4:2 8 evidence based library and information practice article creation of a research community in a k-12 school system using action research and evidence based practice susan d. ballard director, library media and technology londonderry school district londonderry, new hampshire, united states of america e-mail: sballard@londonderry.org gail march library media specialist londonderry high school londonderry, new hampshire, united states of america e-mail: gmarch@londonderry.org jean k. sand library media specialist matthew thornton school londonderry, new hampshire, united states of america e-mail: jsand@londonderry.org received: 15 january 2009 accepted: 27 april 2009 © 2009 ballard, march and sand. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective the purpose of this study was to apply skills developed from an action research training model (gordon) in the design of two action research projects to ensure that students are ethical users of ideas and information. it was deemed necessary to assess prior knowledge and attitudes of students and teachers in order to identify issues to be addressed. methods both projects employed the use of survey instruments, which presented students with scenarios involving aspects of information use, and asked mailto:sballard@londonderry.org mailto:gmarch@londonderry.org mailto:jsand@londonderry.org evidence based library and information practice 2009, 4:2 9 whether or not the actions in the scenarios were examples of ethical use. the high school survey was administered to 381 students in tenth grade english classes. the elementary survey was administered to 87 students in fifth grade. a more comprehensive survey was administered at the high school level. for each student behavior addressed by the survey, there were two questions: one eliciting the teacher’s perception of how often students engaged in that behavior, and the second how often the teacher had to confront a student about the same behavior. participation was voluntary, and 36 teachers took the survey. results surveys administered at the high school level showed that most students have a good understanding of the ethical use of information regarding clear instances of plagiarism. students’ understanding was less clear in two major areas: creating a bibliography that accurately reflects the sources used to create the work, and the level of collaboration or assistance that is appropriate in completing a research assignment. the teacher surveys showed some discrepancy between perception of the frequency of certain types of unethical student behavior and how often teachers challenged students on that behavior. the surveys showed that teachers found plagiarism to be the most prevalent behavior, while obtaining copies of exams and buying papers were the least frequently occurring behaviors. at the elementary level, results indicated that understanding how to properly cite sources was a major area of concern. students were also confused about whom to ask for help during the research process. instructional intervention was developed and applied. the survey was re-administered with the addition of items that were based on the interventions. responses showed a marked improvement in understanding by at least 20%. some responses improved by as much as 60%. conclusion the study validated the action research training model as the first dimension and conceptual framework that informs and guides instructional practices of school library media specialists and teachers in a k-12 school district. after using the model to examine student-teacher knowledge and understanding of ethical use of information (second dimension), there was recognition of the need to clarify the school’s position on the ethical use of information for teachers and students and provide intentional instruction and interventions for students beginning at an earlier grade level. after being made aware of the results, teachers were anxious to work with library media specialists to address issues and to look for opportunities within research units to collaborate. introduction: background of the action research project why would an award-winning school district engage in a rigorous and challenging action research project? the school district was already aware of evidence based practice as defined by todd: ‚evidencebased practice is where day-by-day professional work is directed toward demonstrating the tangible impact and outcomes of sound decisions making and evidence based library and information practice 2009, 4:2 10 implementation of organizational goals and objectives‛ (7). the school district had embraced this concept of evidence based practice in their decision making and in their teaching. they had established a district data team representative of all schools, conducted a comprehensive data inventory as well as a gap analysis to determine what additional data were needed in order to improve teaching and learning. however, they were not convinced that they were seeing benefits through improved transactions between school librarians and teachers, school librarians and students, and teachers and students. as todd notes, evidence based practice offers six key benefits: 1. it provides local evidence at the school level that library initiatives make a visible contribution to learning, and that administrators, teachers and parents can see the real impacts<; 2. it convinces administrators and community funders that the money invested in the school library is worth it<; 3. it demonstrates the teacherlibrarian’s commitment to learning outcomes<; 4. it helps teacher-librarians plan more effective instructional interventions and information services<; 5. it contributes to job satisfaction<; 6. it moves beyond anecdotal, guess work, hunches, advocacy, and touting of research findings< (7). the moment of realization came in october 2001 at the 10th annual aasl (american association of school librarians) conference and exhibition held in indianapolis, indiana. dr. carol kuhlthau and dr. ross todd, both from rutgers university, presented a transformational session entitled, ‚research process and evidence based practice‛. two school library media professionals from londonderry, new hampshire were in attendance and afterward connected with dr. carol gordon, then at boston university, now at rutgers. the two library media professionals began a conversation with dr. gordon about their recognition of what they described as a ‚missing ingredient‛ in the londonderry program. although recipients of the aasl school library media program of the year award in 2000, they had an intuitive sense that the program needed to make the next step. the vision generated by dr. kuhlthau and dr. todd of what ‚should be‛ was reinforced by the conversation with dr. gordon, who proposed the idea of action research because it was well-suited to improving teaching and learning. ‚action research, as a tool of evidence based practice, structures reflective practice‛ (gordon). action research is problem-focused, context-specific, and future-oriented, and aims at improvement and involvement (hart and bond). although well-conceived in its purpose and well-described in its intent, there is, however, a lack of consensus about its methodology. boomer defined action research as a ‚deliberate, group or personally owned and conducted, solutionoriented investigation‛ (8). anderson, herr, and nihlen defined it as ‚insider research done by practitioners using their own site as the focus of their study. <it is oriented to some action or cycle of actions that practitioners wish to take to address a particular situation‛ (2). the components of action research are reflection, inquiry and action (patterson and shannon). as a result, a plan of action, or action research, was developed to move the program to a dimension where it would be evidence based library and information practice 2009, 4:2 11 both collaborative and authentic. the hope was that this effort would ensure enhanced student achievement and success as well as improving the professional practice of the school library media specialists through their reflective practice in the design, delivery, and assessment of instruction. the library media professionals’ moment of realization was the connection they made between evidence based practice, as it was defined by todd, and action research, which offered a structure for the gathering and analysis of evidence. the next step was for the director of library media and technology services to marshal the resources (i.e. support from the superintendent) to sanction a proposal to work with a university-level research mentor, find a source of funding and gain the approval of the londonderry school board. another critical component was to introduce the concept of action research to the other school library media specialists and convince them that because action research is problem-focused and provides solution-oriented investigation, it would serve the program and address ongoing concerns regarding improved student learning. there was definite resistance on the part of the already over-scheduled library media specialists to commit to something new, and that challenged them to grow professionally. the university mentor came to the school to conduct an orientation on action research and methodologies for the team of eight school library media specialists, the director and the technology trainer/integrationist. she provided a basic understanding of action research as a tool for systematic, intentional inquiry into one particular aspect of their professional practice in order for them to better understand and improve their work. after that, as one library media specialist observed, the district traveled the ‚bumpy road of action research.‛ working with their mentor, school library media specialists learned how to pose researchable questions and write proposals (appendix 1). using various sources such as diaries, surveys, questionnaires, interviews, and observations, data were collected and shared with participating teachers, administrators, parents and the school board. when the research projects concluded, participants completed an action research summary (appendix 2) which, in addition to addressing methods and results, provided an opportunity for reflective consideration of how the study had changed their practice and what they had learned. it was soon obvious that through sustained and guided efforts by their research mentor, support from one another and a keen interest from senior administrators, the library media specialists had engaged and found the work meaningful. the added value of the collaborative nature of the work and the empowerment of the school library media specialists to make decisions and take responsibility for their professional growth was meaningful. as the library media specialists grew more comfortable using action research and shared what they had learned with the administration, it was determined that it was critical to provide ongoing professional development opportunities as well as support their work as mentors to other teachers in the use of evidence based action research methodologies. this strategy would help to address the school district’s strategic goal to achieve and sustain a high level of professional competency of all staff members. it would assist district educators to better inform and improve their practice and most importantly, meet the research learning needs of students. this need was one that resonated with both the school library media specialists and their classroom colleagues. despite their award-winning program and data sets that showed a high evidence based library and information practice 2009, 4:2 12 level of student achievement, they felt that students continued to struggle with the research process and resulting projects appeared more ‚repackaged‛ than ‚reflective‛ of understanding the topic or issue. in particular, educators felt that in working on research assignments and projects, students did not clearly understand the concept of ethical use of information. thus, in addition to other areas of investigation, action research was employed by high school and elementary school library media specialists, in collaboration with classroom teachers, to focus on the critical objective of ensuring that students are ethical users of ideas and information. as a result, the connection between action research and professional development became clear. action research incorporates many of the qualities of an ‘ideal’ staff development program. it is individualized and can be used by a teacher at any developmental level. it assumes teachers are knowledgeable and gives them power to make decisions. it can be carried out collaboratively. it is an on-going process and for that reason can be more effective than a typical one day in-service presentation. one of the more significant qualities of action research is that it puts the teacher in the position of accepting more responsibility for her (his) own professional growth (wood 1617). conceptual framework for two action research studies on the ethical use of information what emerged from this project was a twodimensional model (gordon) for doing action research in the context of school library instruction (figure 1). the subject of two of the action research studies was the ethical use of information by children and teens. the first dimension of the model provided the conceptual framework through the establishment of the understanding and use of action research as a tool of evidence based and reflective practice (gordon). in the second dimension, school library media specialists (with assistance from participating teachers) designed, administered and analyzed results of two action research studies related to student and teacher understanding of the ethical use of ideas and information in order to address immediate concerns and increase their understanding of why students struggle with this issue and what interventions are required. this model is based on a formal research study performed by the university researcher that was concurrent with the action research projects specific to each school library (gordon). dimension one: action research as a tool of evidence based practice the conceptual framework of the study rests on the use of action research as a tool of evidence based practice. action research can engage educators in examining the effectiveness of their methods when they have identified an area of concern and use the research process to gather evidence for their theses (gordon). as previously noted, through their work with the university mentor, the school library media specialists had learned how to pose researchable questions. they recognized that not only did the research questions need to be measurable, but also meaningful to their work with students. it was also critical that the school library media specialists and teachers recognized the importance of selecting and developing the appropriate data collection instrument. whether to utilize diaries or journals; pre and post-surveys, questionnaires or tests; rubrics, interviews, or observations, was evidence based library and information practice 2009, 4:2 13 fig. 1. conceptual framework for two studies on the ethical use of information by children and teens evidence based library and information practice 2009, 4:2 14 carefully considered because the instrument had bearing on the reporting of results (evidence) and what was learned. the school library media specialists had been trained by the university mentor to ground their projects in educational theory. as the action researchers considered their research questions, they not only had to think about data collection methods and timelines, but also determine with which educational theorist the approach best aligned. this was a transformational element that allowed for meaningful discussions and exchanges among all participants about how students learn and what interventions are most appropriate (and when) in order to ensure their success. research questions and theoretical framework the two studies presented here were conducted by one high school and two elementary library media specialists and addressed the question, ‚do our students understand what is ethical or unethical in terms of the use of ideas and information?‛ both studies are based on understanding students’ prior knowledge as it relates to learning and the constructivist theories of dewey, piaget, and vygotsky in order to apply appropriate interventions. research has shown that a learner's prior knowledge often confounds an educator's best efforts to deliver ideas accurately. a large body of findings shows that learning proceeds primarily from prior knowledge, and only secondarily from the presented materials. prior knowledge can be at odds with the presented material, and consequently, learners will distort presented material. neglect of prior knowledge can result in the audience learning something opposed to the educator's intentions, no matter how well those intentions are executed in an exhibit, book, or lecture (roschelle 37). additionally, in a review of the literature on academic dishonesty and plagiarism among students, ercegovac and richardson posited that the ‚seminal writings by john dewey (1909), jean piaget (1997) and lawrence kohlberg (1976) could provide a solid theoretical framework in moral reasoning and a good starting point to build on. their work should be considered by education and library communities in any efforts to design well-grounded academic honesty policies and programs for learners across the educational spectrum‛ (301-2). this coupled with the authors’ additional suggestion that ‚attention also is needed in the areas of mapping research results to pedagogical units and specific disciplinary lesson plans, diagnostic and assessment tools that librarians, media specialists, and instructors could customize for their curricular needs‛ (ercegovac and richardson 313) aligns well with the use of action research to investigate learning problems in order to determine and apply instructional interventions. the high school library media specialist determined it was necessary to assess the attitudes of students and teachers to identify the issues that needed to be addressed. while anecdotal evidence surfaced in among the teachers, little concrete information existed to establish students’ understanding of what was and was not ethical use of ideas and information. additionally, teachers’ perceptions of how effectively students could utilize information needed to be understood in order to establish learning expectations, develop effective instruction, and to plan for professional development activities. evidence based library and information practice 2009, 4:2 15 at the elementary level, the two library media specialists and four fifth grade teachers also set out to assess students’ understanding of the ethical use of information. their research addressed the following questions: did fifth grade students know the difference between ethical and unethical use of information? and would the teaching of information literacy skills, such as note taking and bibliography, in conjunction with the fifth grade curriculum change student knowledge and practices with regard to ethical use of information? limitations of the studies it should be noted here that the findings of these two studies are limited in their generalizability to a wider population. qualitative research is not concerned with effect size or generalizability to a population because it is local and context-specific. instead, it is said to be transferable to similar populations. criteria for transferability were met through triangulation, where multiple methods of data collection were used. the findings of these studies are transferable to similar tenth grade high school classes, in the case of the first study, and to similar grade four level elementary classes, in the case of the second study, within the schools where the action research occurred. the replication of both studies in successive years renders the findings transferable to fourth and tenth grade students in the district over time. dimension two: study one ethical use of information at the high school level methods for the assessment of student understanding of ethical use of information at the high school level, members of the library media staff developed a survey for students (appendix 3) and adapted a related survey for faculty. they were based (with permission) to varying degrees on surveys developed by the academic integrity project at central connecticut state university. the teacher survey was more comprehensive than the student survey, with paired questions designed to determine teachers’ perceptions of how often students engaged in unethical behavior related to academic integrity, and also the number of times the teachers had challenged students for those behaviors. the behaviors surveyed included copying or allowing others to copy during an exam, using unauthorized materials or devices during exams, submitting work that was not the student’s own, fabricating research or laboratory data, plagiarism, buying or selling research papers, and reporting cheating by other students. participation by faculty was voluntary, and 36 high school teachers took the survey. it was determined that the student survey, while covering many of the topics in the teacher survey, needed to be presented in a more student-friendly manner. therefore, the student survey presented twelve scenarios involving aspects of ethical use of information, and asked the students whether or not the actions taken by students in the scenarios were examples of ethical use of information. two additional questions elicited responses about what students knew about the school’s position on ethical use of information, and how they had received information (if any) about the school’s position. the student survey was administered to 381 tenth grade students during their english classes. staff also developed an answer key for use by english teachers who wished to take the opportunity to stimulate discussion with students. evidence based library and information practice 2009, 4:2 16 findings the results of student surveys showed that most students have a good understanding of the ethical use of information regarding direct instances of plagiarism. for the scenario of a student who fabricates data and the corresponding citation, 87% of responding high school students identified the behavior as unethical. the scenario of a student who copies complete sentences from a source without attribution was identified as unethical by 90% of high school students. the scenario involving cutting and pasting a table from a web site and submitting it as original work was reported unethical by 87% of high school students. a similar percentage of students recognized turning in a paper written by someone else as unethical (84%). students’ understanding of the ethical use of information was less clear in two major areas: creating a bibliography that accurately reflects the sources used to create the work, and the level of collaboration or assistance that is appropriate in completing a research assignment. at the high school level, 48% of the students felt it was ethical to include random citations to meet the minimum number required by the teacher, while an additional 18% reported they ‚didn’t know‛. only 56% of high school students thought it was unethical to ‚guess‛ which book was the source of a quote used in a research paper. student responses showed some confusion over what level of assistance is acceptable for an assignment. high school students overwhelmingly reported brainstorming ideas as a group to be ethical (92%), as well as asking a librarian for assistance in internet searching (93%). however, thirty percent of high school students thought it was ethical to have a relative conduct the search for information for a project, while 10% reported they ‚didn’t know‛. while students overwhelmingly agreed it was ethical to have a parent make suggestions for improvements for a paper (95%), a surprising 56% of high school students thought is was acceptable to have a friend who was a better writer revise and make improvements to a paper. most students reported that they knew ‚some‛ or a ‚fair amount‛ about the school’s policies on the ethical use of information (64%). however, 28% knew ‚little‛ or ‚nothing,‛ and only 8% felt they knew ‚a lot‛. most students said they received the information from their teacher (82%), but only 49% said they received the information in the class syllabus. the teacher survey illustrated a variation in teachers’ perceptions of student behavior and the student behaviors they had challenged. for the questions involving students copying or allowing others to copy during an exam, 53% of teachers reported the typical student had done it more than several times. based on survey response structure, this would mean 3 to 5 times (in high school career). however, 24% of the teachers reported having never challenged a student for that behavior. while 56% of the teachers reported that the typical student turned in another student’s work (or prepared work for another student to turn in) occasionally, often or very often, 32% of the teachers reported never having challenged a student on that behavior. the teachers’ responses indicated that the most prevalent unethical student behaviors are failing to cite resources (quotes, phrases, figures or data), and in copying information from websites and presenting it as their own work. eighty-three percent of the teachers reported that the typical student exhibits both of these behaviors. fifty-nine percent of the teachers reported that they challenge students on these two behaviors often to very often. evidence based library and information practice 2009, 4:2 17 teachers agreed that the typical student did not practice two behaviors: obtaining or distributing an exam in advance (86% never or seldom) and buying papers (88% never or seldom). similarly, teachers reported they had never challenged a student on this behavior (72% for obtaining a copy of an exam; 83% for student buying an exam). additionally, teachers reported that the typical student seldom or never reported cheating or plagiarism to faculty (64%). student responses to the survey indicate they have a good understanding that it is unethical to copy the work of others, or to fabricate information or data. however, there is less clarity on the ethics of fabricating their sources. this indicates a lack of understanding of the purpose and value of a bibliography. also, students are unclear in their understanding of how much assistance from others is appropriate. there is not a clear understanding of the difference between getting assistance with searching for information and having someone else search for you. similarly, there is less clarity about whether having someone else revise and improve a paper is different from receiving suggestions for improvement. teachers need to make their expectations clear for what level of assistance is appropriate. while the student surveys indicated students are aware that failing to cite sources, submitting work that is not their own, and fabricating data are unethical, teachers report that it is the unethical behavior most likely to occur. indications are that students understand the message that they need to use information in an ethical manner, but have difficulty putting that understanding into practice. teachers need to look beyond one-time admonishments to avoid plagiarism, and to develop instruction and design projects that will build the skills that allow students to confidently create work that reflects their knowledge. implications for practice teachers did not always see the library media specialists as equal partners. while initially concerned that teachers might be inclined to consider follow-up and intervention solely the responsibility of the library media specialists, after the results of the study were shared and discussed, overall awareness of student confusion was raised and recognized as everyone’s responsibility. teachers were anxious to work with the library media specialists to address the issues and to look for opportunities within research units to collaborate. there was also recognition that the school needed to clarify its position regarding the ethical use of information. a statement was developed and included in the student handbook, but it was further recommended to consider the development of a districtwide statement endorsed at the highest level (the school board and senior administration). the proposed statement would not emphasize punitive measures, but rather address the creation of a culture of academic honesty as a value and the norm. it was suggested that this statement be communicated to parents, because of their role in developing and influencing student attitudes. dimension two: study two ethical use of information at the elementary school level methods ‚are the following examples fair or unfair ideas?‛ this was a question asked of fifth grade students on a survey that posed various scenarios regarding ethical use of information (appendix 4). the survey was evidence based library and information practice 2009, 4:2 18 adapted from the ethical use of information survey administered at the high school level. the wording and scenarios were changed to better suit an elementary student population. the survey was administered to 87 students in two fifth grade classrooms at each of two elementary schools (total of four classrooms) prior to a unit of study with a required research component. using the preliminary results of the survey, two library media specialists and four fifth grade teachers determined what areas of ethical use needed focus and attention. they developed lessons to address these areas of weakness as a part of the thematic unit of study in order to assess whether teaching information literacy skills in conjunction with the unit of study would change student knowledge and practices related to the ethical use of information. after the students had completed their unit of study and submitted their research projects the survey was re-administered. four additional questions were added to allow unstructured student responses and to better assess the success of the process. 1. what is plagiarism? (explain in your own words) 2. why would someone be tempted to plagiarize? 3. draw the copyright symbol. 4. explain some things that you can do to make sure that you do not plagiarize and violated copyright laws. the students’ projects were also assessed for ethical use of information. table 1: areas of concern from preand post-survey responses (f=fair; nf=not fair; ns=not sure) question pre-survey responses post-survey responses question #2: okay to ask a librarian for help. 31%=nf or ns 10%=nf or ns question #5: copies two complete sentences from web page without using quotation marks. 34%= f or ns 3%=f or ns question #6: lists a source she didn’t use to meet teacher’s three-source requirement. 44%=f or ns 14%= f or ns question #9: copy/paste a chart into your report without including the citation in your bibliography. 30%=f or ns 3%= f or ns question #10: download copyrighted material. 32%=f or ns 16%=f or ns question #11: make a video copy for two friends. 38%=f or ns 2%=f or ns question #12: allow friend to install your new computer software. 80%=f or ns 12%=f or ns question #13: burn a cd for a friend 81%=f or ns 20%=f or ns question #14: allow friend to photocopy your music book. 47%=f or ns 26%=f or ns evidence based library and information practice 2009, 4:2 19 the survey results were reviewed with the students and explanations were provided as to the fairness or unfairness of each scenario. students had many legal questions and were truly interested in understanding copyright. a parent of a fifth grader who was an intellectual property lawyer volunteered to give a presentation to the students and answer any questions that pertained to copyright infringement. findings the initial survey results indicated areas of misunderstanding amount the students that were of concern to the participating library media specialists and teachers. of particular concern were questions 2, 5, 6, 9, 10, 11, 12, 13, and 14. the library media specialists were particularly surprised by question two which asks if it is fair or unfair to ask the librarian for help. thirty-one percent of the students thought that this was unfair and the library media specialists felt strongly that they needed to rectify this misperception. table 1 contains the combined results of the post-survey for both elementary schools. implications for practice after analyzing initial survey results and determining the areas of concern, the library media specialists decided to focus on developing lessons on note taking, using bibliography cards, developing a bibliography, and what it means to plagiarize. lessons addressing these information literacy skills as well as a lesson on copyright (appendix 5) were presented prior to the students beginning their thematic unit research project. the lesson on copyright (what does copyright mean, which materials are copyrighted, and what constitutes copyright infringement) was taught to address the four additional question items added to the survey; the survey was then re-administered at the end of the unit. the elementary library media specialists decided to teach the information literacy lessons that were implemented for this project to each fifth grade class and continually emphasize that it is ethical to use the library media specialist as a resource. another action item was to revise the grade level expectations included in the document londonderry school district information literacy benchmarks to incorporate the idea of ethical use of information starting in first grade and to continually reinforce these benchmarks through the fifth grade. other intervention recommendations included: 1. develop a district level policy on plagiarism. 2. include a statement in each student/parent handbook on plagiarism, possibly along with the homework statement. 3. include a statement on plagiarism on each school web site. 4. encourage teachers to design assignments that don’t lend themselves to plagiarism. 5. purchase resources for student research at the students’ level or below so that students will be able to understand the material and therefore be better able to take notes in their own words. 6. encourage reading teachers to emphasize how to read non-fiction. 7. begin the process of taking notes and citing sources starting in second grade. conclusions about the results of the two studies and about action research ethical use surveys at the high school level show that most students have a good understanding of the ethical use of information regarding clear instances of plagiarism. the scenario of a student who copies complete sentences from a source without attribution was identified as unethical by 90% of high school students. evidence based library and information practice 2009, 4:2 20 scenarios involving cutting and pasting a table from a web site and turning in a paper written by someone else were also correctly identified by the majority of the students. students’ understanding of the ethical use of information was less clear in two major areas: creating a bibliography that accurately reflects the sources used to create the work, and the level of collaboration or assistance that is appropriate in completing a research assignment. the teacher surveys showed some discrepancy between their perception of the frequency of some types of students’ unethical behavior and how often the teachers challenged students on that behavior. the surveys showed teachers felt plagiarism was the most prevalent student behavior, while obtaining copies of exams and buying papers were the least frequently occurring student behavior. at the elementary level, post-survey responses related to the identified areas of concern showed a marked improvement by at least 20% after instructional intervention. some responses improved by as much as 60%. answers to the additional survey questions demonstrated that most students understood what plagiarism and copyright are and ways that they could make sure that they do not violate copyright law. the use of action research as a means to provide a rich and meaningful professional development opportunity for school library media specialists and teachers has transformed instructional practice in the londonderry school district. despite these traditional challenges of collaboration, there was a strengthening of bonds between the library media specialists and participating teachers. the change in dynamics was precipitated by the unique expertise of the library media specialists in designing and implementing action research. this earned the respect of teachers and the enthusiasm of their students. (gordon) in fact, a collaborative research culture has been created in which all those responsible for student learning engage in meaningful reflection on instructional and program practices at the local level through an examination of evidence provided through the process. since the inception of the use of action research in the school year 2001-02, the school district has conducted 27 action research projects and there are two currently underway. action research has become part of the way the district does business and when new initiatives are considered, there is a required action research component in order that the initiative be assessed and evaluated in terms of impact and sustainability. areas of investigation explored over the last seven years include: how can we get high schoolers off the internet and into subscription databases? how do middle schoolers and high schoolers take notes and how can we teach this skill better? can third graders reach higher order thinking skills through project work? are our end-users satisfied with the current level of library service? additional representative action research projects are provided in appendix 6. the school library media specialists have emerged as leaders of evidence based evidence based library and information practice 2009, 4:2 21 practice in the school district and have data and evidence on how students learn and achieve in an information environment. data are regularly shared with decisionmakers and stakeholders to inform practice and provide improved instruction and service to students. shared results of the various action research projects include a formal report and presentation to the school board. senior administrators and board members have been impressed with the findings and have encouraged the continuation of efforts to the point where funding, initially provided by federal dollars, has been incorporated into the general operating budget. school library media specialists are viewed as experts in implementation and leaders in the improvement of instructional practices. proposed new initiatives are often vetted through a required action research component and most recently, a new element was incorporated into the school district’s strategic plan which calls for the implementation of an action research course for students at londonderry high school. this course will enable students to conduct a semester-long research project, providing the opportunity for the student to be actively engaged in authentic inquiry on a topic of their own choosing, at a deeper level than might be possible in other projects. regarding the example of action research used to assess student understanding of the ethical use of information, high school students are getting the message that it is unethical to copy the work of others, or to fabricate information or data. however, the teacher surveys indicate that it is still difficult for students to put that understanding into practice. teachers will need to move beyond one-time admonishments for plagiarism, and endeavor to build the skills and understanding that allow students to confidently develop and support their own ideas and work. as a result of our findings at the elementary schools, the library media specialists and teachers have reflected on how improve their own practice. it was decided to continue to teach lessons on information literacy to all fifth grade students, to begin to discuss copyright and plagiarism beginning in the early grades, and to revise district information literacy benchmarks. works cited anderson, gary l., kathryn g. herr and ann nihlen. studying your own school: an educator’s guide to qualitative practitioner research. thousand oaks, ca: corwin press, inc., 1994. boomer, garth. ‚addressing the problem of elsewhereness: a case for action research in our schools.‛ reclaiming the classroom: teacher research as an agency of change. eds. dixie goswami and peter stillman. portsmouth, nh: boynton/cook publishers, 1987. 413. dewey, john. democracy and education: an introduction to the philosophy of education. 1916. new york: free press, 1966. dewey, john. moral principles in education. (reprinted from 1909 edition by permission of the philosophical library.) new york: greenwood press, 1969. ercegovac, zorana and john v. richardson, jr. ‚academic dishonesty, plagiarism included, in the digital age: a literature review.‛ college evidence based library and information practice 2009, 4:2 22 and research libraries. 65 (2004): 301-18. ‚faculty survey of student academic misconduct.‛ academic integrity project. eds. antonia moran, j. d. conway, and james conway. 18 apr. 2004. central connecticut state university. 22 aug. 2008 <http://www.ccsu.edu/ academicintegrity/ faculty%20survey.asp>. gordon, carol. ‚a study of a threedimensional action research model for school library programs.‛ school library media research online 5 (2006): 5 january 2009 <http://www.ala.org/ala/aasl/aaslpu bsandjournals/slmrb/slmrcontents/v olume9/actionresearch.cfm> hart, elizabeth and meg bond. action research for health and social care: a guide to practice. buckingham, uk: open university press, 1995. kohlberg, lawrence. ‚moral stages and moralization: the cognitivedevelopmental approach.‛ moral development and behavior: theory, research and social issues. ed. dalam t. lickona. new york: holt, rinehart & winston, 1976. 31-53. kuhlthau, carol c. and ross todd. ‚research process and evidence based practice.‛ 10th annual aasl conference and exhibition, indianapolis, in, usa, october 2003. patterson, leslie and patrick shannon. ‚reflection, inquiry, and action.‛ teachers are researchers: reflection and action. eds. leslie. patterson and carol santa. newark, de: international reading association, 1993. 7-11. piaget, jean. the child’s conception of movement and speed. new york: basic books, 1970. piaget, jean. the moral judgment of the child. new york: simon & schuster, 1997. roschelle, jeremy. ‚learning in interactive environments: prior knowledge and new experience.‛ public institutions for personal learning: establishing a research agenda. eds. john h. falk and lynn d. dierking. washington, dc: american association of museums, 1995. 37-51. ‚survey of student academic misconduct.‛ academic integrity project. eds. antonia moran, j. d. conway, and james conway. 18 apr. 2004. central connecticut state university. 22 aug. 2008 <http://www.ccsu.edu/ academicintegrity/ student%20survey.asp>. todd, ross. ‚learning in the information age school: opportunities, outcomes, and options.‛ international association of school librarianship conference, durbin, south africa, july 2003. vygotsky, lev. thought and language. cambridge, ma: mit press, 1986. wood, patricia. ‚action research: a field perspective.‛ american educational research association conference, new orleans, la, usa, april, 1988. http://www.ala.org/ala/aasl/aaslpubsandjournals/slmrb/slmrcontents/volume9/actionresearch.cfm http://www.ala.org/ala/aasl/aaslpubsandjournals/slmrb/slmrcontents/volume9/actionresearch.cfm http://www.ala.org/ala/aasl/aaslpubsandjournals/slmrb/slmrcontents/volume9/actionresearch.cfm http://www.ala.org/ala/aasl/aaslpubsandjournals/slmrb/slmrcontents/volume9/actionresearch.cfm evidence based library and information practice 2009, 4:2 23 appendix 1 londonderry school district action research proposal submitted by: date: research question or hypothesis instructional unit or initiative a. topic: b. timelines: start: interim timelines (if applicable): finish: b. description of project or unit. please include details that would help someone replicate your unit. curriculum or other objectives? student or other product? assessments (formative and summative)? lessons taught: support materials? data collection: please include at least two methods (e.g. interviews, questionnaires, grades, observation journal entries, photographs, student products/projects, formative assessments, focus groups, case studies, and content analysis). evidence based library and information practice 2009, 4:2 24 appendix 2 – londonderry school district action research summary researcher: school: date of report: project date: research question/hypothesis: summary: (100 word abstract) why is this question important to your practice? describe the teaching unit: describe data collection methods: . major findings: what i learned that surprised me: how this will change my practice: evidence based library and information practice 2009, 4:2 25 appendix 3 londonderry school district student survey – high school part 1 in your opinion, are the following examples of fair (ethical) and appropriate use of ideas and information, or not? answer key 1. nicole, kelly, joe and andy meet after school to discuss their assignment to write a paper on global warming. they brainstorm some ideas together, and then each individual begins to focus in on developing their own paper. yes, this is ethical and it is a good idea to review the assignment with others to get a sense of different ways to approach your individual paper. 2. cassidy visits the school library and asks a member of the library staff for assistance in searching the internet for information for the report on global warming. yes, this is ethical and cassidy is wise to consult with an information specialist to get some assistance in beginning her research. 3. jason has an uncle who works in a research laboratory. he emails his uncle to search the internet for him to get the information for his global warming report. this is a gray area. if the teacher has made it clear that students are expected to do their own research, then jason is out of line because he is asking his uncle to do the searching. if the teacher was not clear on instructions, at the very least, jason is shortchanging himself because he is not learning anything about the resources. 4. mike decides that his paper will be better if he is able to cite lab data results on global warming, but he runs out of time to do the research, so he “makes up” the information, and also indicates in his bibliography a source for the data. this is very unethical. providing made up information is never right. perhaps if mike really feels that there has not been enough time to complete the assignment, he might request an extension. 5. jen has located a very helpful paper on global warming posted at a high school web site in an australia. she downloads the entire paper and uses some ideas and information to develop her own paper, and credits the source in her bibliography. yes, this is ethical. as long as jen credits the ideas and information to the source she is ok, although there may be better sources of information than a high school web site! 6. brian finds a great article in an electronic data base that is perfect for his global warming paper. he copies two complete sentences from the resource and includes them verbatim (word for word) in his report without using quotation marks, footnotes or other means of attribution. brian is exhibiting unethical behavior. this is a clear case of plagiarism by using someone else’s work and not giving credit. 7. cindy wrote down a great quote about the effects of global warming in her notes, but can't remember which book it came from. so, she makes her best guess, and lists that book in her bibliography. cindy needs to go back to the resources she utilized and find the correct source for her quote so she can give accurate citation in the bibliography, otherwise, this is unethical. 8. jim found four great sources for his global warming research project, but his teacher required jim is wrong. a bibliography is not a list of resources that you found, but a list of evidence based library and information practice 2009, 4:2 26 seven sources. so, before typing his bibliography, he found three more websites and included them on the list. those that you actually used in completing a project or a paper. 9. judy asks her dad to review a draft of her global warming paper in order to spot errors, or make suggestions for changes or improvements, before she completes a final draft. this is fine and it is often helpful to ask an independent reader to go over your work so that they can point out glaring errors or sections that may need clarification or refinement. 10. mark connects with his best friend nate to look over his global warming paper and asks nate, a gifted writer, to go ahead and revise any errors and change anything that he feels will improve the paper. mark is also wise to ask a “gifted” friend to read his paper but he is wrong to give him a free hand to change anything as the paper is now at best a joint publication of mark and nate, and at worse, it is now nate’s paper. 11. john finds results from a study on global warming at a university web site. it includes a very interesting table which he cuts and pastes into his report, and submits as his original work. this is completely wrong and another example of an act of plagiarism – using someone else’s work and pretending it is your own. 12. ingrid’s cousin in florida had a similar assignment on global warming last year. her cousin copies the paper and faxes it to ingrid, who makes some minor changes, and turns the paper in as her original. by making a few changes, ingrid has done nothing to change the fact that this is another instance of plagiarism – even if her cousin was agreeable, ingrid is submitting someone else’s work as her own and that is wrong. part 2 13. how much do you know about your school’s position on (fair) ethical use? (check the appropriate response): a) nothing b) very little c) some d) a fair amount e) a lot 14. – 17. have you received information on your school’s position from: yes no 14) a b a course syllabus? 15) a b a teacher (i.e., discussed in class)? 16) a b other students? 17) a b other? (indicate where and include in comments section on answer form) 18) your age: a) 1214 b) 15 17 19) year of graduation: a) 2010 b) 2007 20) your gender: a) male b) female thank you for your participation part 1 © londonderry school district, 2005 evidence based library and information practice 2009, 4:2 27 part 2 used with permission, from the academic integrity project, central connecticut state university, antonia moran, j.d. and james conway, ph.d., project directors evidence based library and information practice 2009, 4:2 28 appendix 4 londonderry school district student survey elementary schools are the following examples fair or unfair ideas? fair unfair not sure 1. joe, mary and bobby meet after school to discuss their explorer reports and help each other with ideas. 2. john goes to the library and asks the librarian to help him search the internet for information for his report on hurricanes. 3. mike needs 2 written pages for his report, but he only has enough information for 1 page. he decides to “make-up” information in order to complete his 2 pages. 4. jen found a very helpful book for her explorer report. she writes down some important information she found in the book in her own words, and she fills out a bibliography card to show where her information came from. 5. lindsey copies 2 complete sentences from a web site and includes them in her report without using quotation marks. 6. jane needs 3 sources of information for her new hampshire report. she has only used 2. she lists a source she did not use in her bibliography so she will have the 3 she needs. 7. judy asks her dad to look over her report and make suggestions for improvement. 8. josh asks his mom to look over his report and she makes the changes for him. 9. will cuts and pastes a chart for his hurricane report from a web site and doesn’t include the web site in his bibliography. 10. ben downloads copyrighted music from the internet. evidence based library and information practice 2009, 4:2 29 11. alex makes copies of a video he bought for 2 of his friends. 12. mary has new software for her computer. she lets her friend, madison, install the software on her computer, too. 13. sean burns a cd for his friend with his favorite songs from his cd collection. 14. jillian lets her friend photocopy a guitar music book so her friend will not have to buy it. © londonderry school district, 2005 evidence based library and information practice 2009, 4:2 30 appendix 5 – londonderry school district lesson plan unit: ethical use of information school: matthew thornton elementary grade level: 5 library media specialist: jean k. sand classroom teachers: stella skoropowski, maureen mckay overview: students have taken a survey on the ethical use of information. after reviewing the survey questions with the students, a parent who is an intellectual property lawyer spoke to the children and answered their questions. the students are working on a report, which will be evaluated for ethical use, and following this, a post-survey will be administered. before the post-survey and questionnaire, this lesson will be taught to reinforce the students’ knowledge and understanding of plagiarism and copyright. objectives: the students will be able to: recognize the copyright symbol understand what copyright means understand what plagiarism is content/activities: lms asks what ―stealing‖ means explains what ―property‖ is—things someone owns discuss why stealing is wrong—(hurts others, against the law, wrong to take what is not yours) lms reads a book about stealing follow up discussion about stealing and the consequences discussion of stealing ―ideas‖—things people create. make a list of things (stories, poems, music, software) that are ―intellectual property‖. wrong to steal ideas? yes, just like physical things show copyright symbol define copyright define plagiarism materials: overhead and transparency sheet vis a vis marker story book evidence based library and information practice 2009, 4:2 31 appendix 6 representative action research projects students’ ethical use of information author: hs library media specialist abstract: are current practices (school policy, instructional practices, library services) effective in ensuring that students are ethical users of information? are we addressing the factors that lead to student plagiarism? for this project, a two-session workshop was held for teachers interested in the topic where staff development credit was offered. a survey was administered to determine participants’ perceptions of plagiarism and cheating at the high school level but there were a very small number of teachers who participated and so these results are not representative of the entire teacher population. the recommendations of the participants show these teachers are aware that there are steps that could be taken. however, other issues such as the importance of and pressure for good grades, and the indifferent attitude of some students, will require a much larger effort to address. study descriptors: survey, library media, technology, assessment/evaluation, staff development, climate/culture data collection methods: survey, focus group, discussion does the time given to quality research improve student knowledge and output? author: high school social studies abstract: this project looked at how to make the research process more concrete by providing a step-by-step program that will guide students independently down the path of researching a topic. the process is outlined in separate pages for each of the steps and before going to the next step, the work has to be approved by the teacher. this process was well-received by the sophomore u.s. history students, as many of them did not know the individual steps that went into research. study descriptors: social studies, library media, assessment/evaluation, inquiry/research data collection methods: questionnaire, student work, teacher journal technology in the ece (earth and its changing environment classroom) authors: high school library media specialists abstract: does the use of technology in the science classroom help students understand the concepts better? the freshmen science classes at londonderry high school are considered ―technology rich‖ because there are classroom computers to be used by students. but how do the students actually use the computers? do the students feel as thought they learn better or differently with the computers? 295 students were surveyed in total, garnering useful information about students’ interactions with technology. study descriptors: technology integration, survey, library media, science, technology, assessment/evaluation, instructional strategies data collection methods: survey, observation assessing an online learning forum author: high school reading evidence based library and information practice 2009, 4:2 32 abstract: literacy is a primary goal of londonderry high school for the school year. in response to this initiative, a website called learning theories was developed. learningtheories.org is a website designed to for the discussion of young adult literature for both pleasure and as an assessment tool. teachers and students are given equal authority on the book club style forum and need only to log on once registered. this study looked at whether using an online forum was a practical means of assessment. study descriptors: reading, technology integration, technology, staff development, assessment/evaluation, at-risk students, learning styles, instructional strategies, communication, literacy data collection methods: case study, questionnaire, interview will 9th grade students demonstrate improvement in evaluating, selecting and assembling information, as well as crediting sources, if provided with a choice of note-taking templates to assist in the research process? authors: high school library media specialist with social studies teacher abstract: eighty-one ninth grade students in three world history classes participated in a research project titled ―taking a closer look at renaissance art.‖ previous years’ projects revealed that students had difficulty putting ideas into their own words and crediting their sources of information. this problem of plagiarism, whether intentional or inadvertent, was the impetus for determining what interventions could improve students’ skills in the research process. results from the study showed that more students completed the project compared to overall freshmen results from the previous year. also, more students in this study met or exceeded the composite benchmarks compared overall to freshmen from the previous year. younger high school students need structured guidance in the research process. something as simple as a note-taking template can help students stay organized, and emphasis on the process can help students avoid taking shortcuts that defeat the purpose of the project. study descriptors: assessment/evaluation, inquiry/research, library media, social studies, instructional strategies data collection methods: survey if the early research steps are emphasized, i.e. pre-reading and creating questions, will the students see the benefit in their completed project? will i see a difference in their ability to support their topic as opposed to report writing? author: high school social studies abstract: a good summary is the famous dickens quote, ―it was the best of times, it was the worse of times.‖ this project has been done for a number of years, but each year the project has been changed slightly to reflect the different classes. this project was given to three heterogeneously grouped freshmen world history classes. emphasis was placed on understanding the difference between ―reporting‖ and ―researching.‖ moving from report to research can be difficult, but with planning and resources, it is possible. study descriptors: assessment/evaluation, inquiry/research, library media, social studies, instructional strategies data collection methods: questionnaire, survey. library services survey evidence based library and information practice 2009, 4:2 33 author: high school library media specialist abstract: a first-year library media specialist developed and distributed a library services survey, which looked at how well the library media center was meeting the students’ needs. the survey was distributed to 196 students in total, roughly 50% freshmen and 50% juniors. there were also a few short interviews done with students from each of the classes that asked more specified questions. the students were very candid with their remarks, and made some great suggestions for improvements to the library media center. more importantly, this survey and interview process allowed a person new to the school the opportunity to learn more about what the students would like and what they need from their library media center to be successful. study descriptors: assessment/evaluation, library media, survey, instructional strategies, communication, climate/culture data collection methods: survey, interview research skills in the high school science classroom author: high school science abstract: it is so important for students to improve their research skills, to be able to separate fact from fiction, and to be able to defend their viewpoints using reputable supporting evidence. this is especially important in the science field when investigating through experimentation. this research project looked at how to engage students in the research process by both giving them a choice in their topic and by providing assistance and guidance throughout the research process. the overall topic to be studied was nanotechnology, but each student had a choice of what part to study and which direction they took their own research. the goal was for the students to become an ―expert‖ on their topic. study descriptors: technology, assessment/evaluation, collaboration/teaming, inquiry/research, library media, science, instructional strategies data collection methods: survey does the iim research method help students synthesize information from multiple resources and reduce plagiarism? author: high school science abstract: sixty-five tenth grade college preparatory biology students participated in a human disease research project. the tenth grade biology curriculum has been aligned with the newly adopted state frameworks and human disease is an addition to the curriculum. the project was designed to allow students to apply what they had learned throughout the school year and learn about how their bodies can fight disease. students used the iim research model to conduct their research, as well as to analyze and organize their information. students had the option to work alone or in groups on a topic of their choice and each step of the research process was outlined for them with specific due dates. students were given class time to use the resources in the school library. the final products varied from brochures to power point presentations and special consideration was paid to looking to see if the students had truly synthesized information rather than just reporting on the information. study descriptors: assessment/evaluation, collaboration/teaming, inquiry/research, library media, science, instructional strategies evidence based library and information practice 2009, 4:2 34 data collection methods: survey how can i improve my students’ ability to read nonfiction and informational text in order to increase their ability to understand science concepts? author: middle school science abstract: various types of information text were integrated into daily science lessons. in addition to textbook material, magazine and journal articles were utilized through a current events component. the instructor employed a methodology to provide students with strategies for reading comprehension: pre-reading activities (examples and demos) investigation of the organization of the text prior to reading; vocabulary review before reading; text clues to identify critical information; use of pictures, captions and graphics to enhance understanding. study descriptors: science, reading, instructional strategies, informational text data collection methods: pre/post test, student survey, questionnaire and conferences how is student writing in a practical level eleventh grade english class changed by the use of alphasmart technology? author: high school english abstract: in these days of high stakes writing, it is imperative students become more comfortable with writing in a classroom setting. researchers from the technology and assessment study collaborative of the lynch school of education at boston college analyzed test performance and computer use habits of 986 fourth grade students from 55 classrooms in nine massachusetts school districts. they found the more students used computers to prepare powerpoint presentations, surf the web or play games, the worse they performed on the exam. the more students used computers to write school papers, the better they performed on the mcas english/language arts exam (2004) which suggests that we need to provide our students access to technology specifically for word processing rather than simply unguided access to computers. this project serves to answer the question of how is student writing in a practical level eleventh grade english class changed by the use of alphasmart technology. specifically, this study sought to answer the following questions: do students who have historically struggled with writing find the use of alphasmarts makes the writing process less tedious? does the use of alphasmarts make students more willing to revise their work? a subordinate purpose for this study was to discover how students’ perceptions of themselves as writers change through the use of alphasmart technology. study descriptors: library media, technology, technology integration, english, writing, assessment/evaluation, instructional strategies data collection: interview, observation analysis skills during the research process authors: middle school library media specialists with social studies teachers abstract: as our students have developed their research skills using search engines, web portals, and databases, what analysis guides their selection? do the recommendations of teachers, library media specialists, or peers help them make a selection? or, are they evidence based library and information practice 2009, 4:2 35 using a less critical approach? how does selecting appropriate search tools aide or frustrate them in developing the final product? study descriptors: assessment/evaluation, collaboration/teaming, inquiry/research, instructional strategies data collection methods: questionnaire, student reflection, survey do math skills books, in conjunction with problems of the day, increase students’ grades in computation, specifically in the four basic operations on decimals and fractions on standardized tests? author: middle school math abstract: targeted students weekly assignments from their math skills book as well as a problem-of-the-day, designed to cover the area in which they have historically scored lowest (i.e. four basic operations using decimals and fractions. study descriptors: technology, assessment/evaluation, math, instructional strategies data collection methods: pre and post-test, standardized test results, student conferences, survey how can we improve library services? authors: elementary library media specialists abstract: a survey/interview was developed to gather data about student attitudes, perceptions, and needs of the elementary school library media center and its staff. previously the library media specialist’s had surveyed the professional staff regarding library media services. at this time we felt the necessity to survey our important users – the students. after the data was entered and analyzed using excel, the library media specialist’s noted their findings and planned to implement the appropriate changes to their practice and services so they can better serve our school community. the same survey was given at each of the three elementary schools, with separate reports for each school. study descriptors: survey, library media, assessment/evaluation, instructional strategies, communication, climate/culture data collection methods: survey, interview is this ok, is this not ok? do 5 th grade students know the difference between ethical and unethical use of information? authors: elementary library media specialists with 5 th grade teachers abstract: can the teaching of information literacy skills (note-taking, bibliographies…) in conjunction with fifth grade thematic units of instruction change student knowledge and practices with regard to the use of information? instructional units were designed to teach students note-taking skills, how to cite sources, and how to do a bibliography. by teaching these skills, the teachers and librarians hoped to have a positive affect on student behaviors with regard to the ethical use of information. due to this intervention, there was a marked difference between the pre and post questionnaire results, which demonstrated the change in attitude and student understanding of the material presented. study descriptors: assessment/evaluation, instructional strategies, communication data collection methods: pre/post questionnaire, observation journals, student evaluation of the process. evidence based library and information practice 2009, 4:2 36 does the use of specific math software with grade 4 students increase their math fact fluency? authors: technology integrationist and 4th grade teacher abstract: 4 th grade students in a control class used specific math software ton a daily basis. schedules were posted to ensure that all students had at least 15 minutes/day to use the program. utilization data was recorded automatically through the program and a variety of reports formats available related to student progress. level: elementary study descriptors: math, math software, assessment/evaluation data collection methods: program data, teacher journal, student questionnaire after participating in a teacher induction process for a year, how do new teachers to londonderry view their practice? in what danielson domain(s) and component(s), can these ideas be categorized? author: teacher induction team member (5 th grade teacher) abstract: in order to assess the effectiveness of the londonderry teacher induction program in regard to participant understanding of the domain of professional responsibility, 34 teachers new to londonderry (experience ranges from no experience to 14 years) participated in a study designed to determine if the program’s activities were meaningful and constructive. level: elementary study descriptors: induction programs, professional practice, teaching domains and components data collection methods: questionnaire how well do students apply skills taught in class for the reading of non-fiction to the gathering of information for an independent research project? author: elementary library media specialist level: elementary study descriptors: assessment/evaluation, collaboration/teaming, inquiry/research, reading, instructional strategies data collection methods: interview, questionnaire abstract: this project looked at how well students apply skills taught for the reading of non-fiction to the gathering of information for an independent reading project. three teachers worked in collaboration for the project: a reading specialist, a library media specialist, and a grade 4 classroom teacher. for the assignment, students embarked on a shuttle flight to the international space station where each team of 6 students will have a specific job, such as a pilot, engineer, or scientist, as well as a special assignment like moon walkers, sun trippers, galaxy searchers, and satellite repair crew. all students will research general information on space flight and then gather information for their specific job assignment and outside activity. ebl 101   research methods: the most significant change technique   virginia wilson director, centre for evidence based library and information practice (c-eblip) university library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   originally published in: evidence based library and information practice, 9(3), 121–123. https://ejournals.library.ualberta.ca/index.php/eblip/article/view/22918/17169     received: 11 aug. 2014 accepted: 29 aug. 2014      2016 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   in this ebl101 column i am exploring a technique that has largely been used to evaluate international development programs. the most significant change technique (msc) was developed by rick davies and jessica dart in the early 2000s to evaluate complex interventions. it takes place during the lifespan of the intervention program, so it is a process of continuous evaluation. while i have not encountered this method used in library and information studies, it strikes me that a technique such as this would be useful in variety of situations: evaluating instruction, appraising public programs, assessing various initiatives in any area of the library (client services, technical services, etc.), and others that i am probably just not seeing right now.   the methodology is participatory, so its use would be a good chance to have direct contact and conversation with various stakeholders. for example, library patrons, library staff, higher administration, the public—whoever is involved with whatever is being looked at and changed. while this method could perhaps be used on smaller projects in the library, i see its usefulness as being centred more upon large, organization-wide developments and changes, as a way to continuously monitor the situation and make adjustments as the project progresses. examples of larger projects include the implementation of a different organizational structure, the design, development, and building of a library facility, advancement initiatives, or a large-scale research project that is national in scope.   dart and davies (2003) refer to msc as a “dialogical, story-based evaluation tool” that eschews “conventional monitoring against quantitative indicators” in favour of the “collection and participatory interpretations of ‘stories’ about change” (p. 138). these stories are “examples of significant program outcomes are collected and presented to designated groups of stakeholders who deliberate on the value of these outcomes in a systematic and transparent manner” (dart, 2005, p. 261). though stories are the focus of the analysis, dart and davies (2003) indicate that “the central aspect of the technique is not the stories themselves, but the deliberation and dialogue that surrounds the process of selecting significant changes” (p. 138).   on his news website, davies (2008) states that msc is most useful in the following situations:   where it is not possible to predict in any detail or with any certainty what the outcome will be where outcomes will vary widely across beneficiaries where there may not yet be agreements between stakeholders on what outcomes are the most important where interventions are expected to be highly participatory, including any forms of monitoring and evaluation of the results   msc consists of seven key steps (although the guideline document listed below under resources by the same authors outlines 10 steps):   the selection of domains of change to be monitored unlike performance indicators, which are specific and focused, the domains of change are broad and loose, allowing for program participants to define them for themselves. the domains are identified by stakeholders. the reporting period “stories of significant change are collected from those most directly involved” over a time period decided upon at the start of the project. the time period can be extended if more stories are needed. the participants the participants are those who are involved with the program in question, such as beneficiaries, clients, and field staff. phrasing the question stories are gathered by using one simple question: “during the last month, in your opinion, what was the most significant change that took place in the program?” the structure of participants the stories are then analyzed and “filtered up through the levels of authority typically found within an organization or program”, with each level selecting the most significant change stories to be sent on up the ladder. feedback continuous communication amongst stakeholders participating in and reviewing the stories is a key component, so that the feedback can be incorporated into each subsequent round of story collection. verification verification can take place by visiting the sites of the events described in the stories for follow up. (adapted from dart & davies, 2003, pp. 138-139)   for example, the above steps might be mapped on to a library or information management process in the following way when a complex intervention is chosen for continuous evaluation, e.g., the move to a programmatic approach to academic library instruction. domains to be monitored are selected by participants in this change process, and by those who the change affects, such as librarians, students, library staff, and others from the larger institution (professors from various colleges who utilize library instruction, administrators from colleges who are participating in the construction of a programmatic approach to instruction, etc.). stories are gathered from the participants regularly over the academic year to provide continuous monitoring of the change intervention. stories are the result of asking a simple question, such as “what was the most significant change that happened this month as a result of the programmatic approach to instruction recently implemented?” those monitoring the change analyse and examine the stories at every level, with the different analyses going up the organizational structure to be further analysed. all participants provide and share continuous feedback, providing more information about the change occurring. follow up should happen, with various participants talking to other participants about the change taking place, verifying the analyses of the stories.   the msc technique could be an interesting framework with which to assess and evaluate our professional practice. dart and davies (2003) state that the key strength of msc “lies in its ability to facilitate a dynamic dialogue between designated stakeholders” (p. 152). as librarians and information professionals, we should be looking for new and innovative ways to communicate with our users, clients, patrons, and with each other as we strive to provide the best services possible.   other resources and examples of msc in action   davies, r., & dart, j. (2005). the ‘most significant change’ (msc) technique: a guide to its use. retrieved from http://www.alnap.org/resource/8102     lunch, c. (2007). the most significant change: using participatory video for monitoring and evaluation. participatory learning and action, 56, 28-32. http://www.iied.org/participatory-learning-action   wilder, l., & walpole, m. (2008). measuring social impacts in conservation: experience of using the most significant change method. oryx, 42(4), 529-538.     http://dx.doi.org/10.1017/s0030605307000671   willetts, j., & crawford, p. (2007). the most significant lessons about the most significant change technique. development in practice, 17(3), 367-379. http://dx.doi.org/10.1080/09614520701336907   references   dart, j. (2005). most significant change technique. in s. mathison, (ed.), encyclopedia of evaluation (pp. 261-263). thousand oaks, california: sage.   dart, j., & davies, r. (2003). a dialogical, story-based evaluation tool: the most significant change technique. american journal of evaluation, 24(2), 137-155. http://dx.doi.org/10.1177/109821400302400202   davies, r. (2008, apr. 13). most significant change (msc). monitoring and evaluation news. retrieved from http://mande.co.uk/special-issues/most-significant-change-msc/   ebl 101   research methods: focus groups   virginia wilson liaison librarian murray library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   originally published in: evidence based library and information practice, 7(1), 129–131. https://ejournals.library.ualberta.ca/index.php/eblip/article/view/16359/13267     received: 31 jan. 2012   accepted: 04 feb. 2012      2016 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   conducting focus groups is a qualitative research method that allows researchers to collect a large amount of data from a substantial group of people in a relatively short amount of time. focus groups explore how people perceive, feel about, or view a certain service, product, topic, etc. this method gained popularity in marketing and market research in the last several decades. there is an unverified anecdote that tells of the ford focus vehicle being named by an extremely bored focus group! however, focus groups need not be boring, and the use of these groups for research has become more popular in the social sciences and the health sciences (bell, 2010, p. 165). the data produced is unique as well. given the “synergistic potentials” of focus groups, they “often produce data that are seldom produced through individual interviewing and observation and that result in especially powerful interpretive insights” (kamberelis & dimitriadas, 2005, p. 903). in other words, get people talking and the insights will often go further because of the group dynamic.   focus groups are more commonly conducted face to face, with participants, researchers, and focus group facilitator all together in one place. however, it is becoming more common for focus groups to be conducted virtually, via real time chat or video conferencing. although beck and manuel (2008) assert that “library and information science practitioner-researchers have not yet used electronic interviews or virtual focus groups to any significant degree,” these methods are less costly in terms of administration and data transcription (p. 94-95).  it is conceivable that the benefits will outweigh the challenges of conducting virtual research into the future.   now, what are the basics of conducting focus groups? the basic structure consists of a group of six to ten participants taking part in an organized but flexible conversation that will last one to two hours. a moderator facilitates the discussion, notes are taken, and the entire conversation is usually recorded for later transcription. the role of the moderator is an important one, and if an expert moderator is not available, training should be explored. the “moderator sets the tone of the session . . . [and] needs to strike a balance between ease and formality that encourages the free flow of information but that also requests that participants take the session seriously” (beck & manuel, 2008, p.96).   of course there are issues that can arise when a group meets for a discussion. some people tend to dominate group discussion while others rarely speak. it is the moderator’s job to make sure that all viewpoints are heard. beck and manuel (2008) outline some types of focus group participants that can be challenging in a group discussion: dominant talkers, long-winded participants, the expert, the argumentative type, the shy person. it is important to think ahead of ways to deal with the various personalities one may encounter in a focus group.   careful consideration should go into populating a focus group. once the topic is decided, you need to think of who would be the best people to inquire about it. other things to think about are how many participants in total will the study include? how many groups will be conducted? should each group contain a varied mix of people or be more homogeneous? beck and manuel (2008) point out that “the research problem itself primarily drives the answers to these questions through its purposes and goals” (p. 87).   if you are considering conducting focus group research, there are many resources out there to get you started, including the following books:   glitz, b. (1998). focus groups for libraries and librarians. new york, ny: forbes.   greenbaum, t. l. (2000). moderating focus groups: a practical guide for group facilitation. thousand oaks, ca: sage.   liamputtong, p. (2011). focus group methodology: principles and practices. los angeles: sage.   walden, g. r. (2008). focus groups: a selective annotated bibliography. lanham, md: scarecrow press.   additionally, it’s a good idea to review research articles that have used focus groups as a research method:   carlock, d. m., & perry, a. m. (2008). exploring faculty experiences with e-books: a focus group. library hi tech, 26(2), 244-254. doi:10.1108/07378830810880342    courtois, m. p., & turtle, e. c. (2008). using faculty focus groups to launch a scholarly communication program. oclc systems & services: international digital library perspectives, 24(3), 160-166. doi:10.1108/10650750810898192   fagerheim, b. a., & weingart, s. j. (2005). using focus groups to assess student needs. library review, 54(9), 524-530. doi:10.1108/00242530510629542   macmillan, d., mckee, s., & sadler, s. (2007). getting everyone on the same page: a staff focus group study for library web site redesign. reference services review, 35(3), 425-433. doi:10.1108/00907320710774292   waters, m. (1996). a children's focus group discussion in a public library: part one. public library quarterly, 15(2), 5-6.   focus groups can generate quick and plentiful data for a research project. their benefits include a relatively low cost, highly detailed data, synergistic results from group participation, and flexibility. some drawbacks of focus groups include problems with dominant personalities, potential scheduling hassles, the possibility of group think (comes from the desire for harmony within a group which is trying to minimize potential conflict), and outcomes that are highly dependent on the moderator (adapted from beck & manuel, 2008, p. 79). however, when you want to discover how a group of people perceives a particular service or topic, a focus group can be the way to go. next time, we will take a look at a method that is related to focus groups – the interview.   references   beck, s. e., & manuel, k. (2008). practical research methods for librarians and information professionals. new york, ny: neal-schuman.   bell, j. (2010). doing your research project: a guide for first-time researchers in education, health and social sciences. new york, ny: open university.   kamberelis, g., & dimitriadas, g. (2005). focus groups: strategic articulations of pedagogy, politics, and inquiry. in n. k. denzin & y. s. lincoln (eds.), the sage handbook of qualitative research, (3rd ed.) (pp. 887-907). thousand oaks, ca: sage.    evidence summary   both online and in-person library orientations have positive effects on graduate students’ information literacy skills   a review of: gall, d. (2014). facing off: comparing an in-person library orientation lecture with an asynchronous online library orientation. journal of library & information services in distance learning, 8(3-4), 275-287. doi:10.1080/1533290x.2014.945873   reviewed by: lisa shen head of reference and instructional services newton gresham library sam houston state university huntsville, texas, united states of america email: lshen@shsu.edu   received: 8 mar. 2016    accepted: 20 apr. 2016      2016 shen. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to assess and compare the effectiveness of online and face-to-face library orientations.   design – pretest/posttest.   setting – a public university in the united states of america.   subjects – graduate students in a master of social work program.   methods – at the beginning of the fall 2013 semester, students in 3 different sections of a social work research methods course were asked to complete a 17-question assessment to evaluate their information literacy skills and knowledge. then, 1 section (campus a) received a 50-minute in-class library orientation from a librarian, while the other 2 sections (campus b and off campus) received orientation through asynchronous online video tutorials.   post library orientation, research labs were then held for all three class sections, during which students received some hands-on time working with a librarian. after the hands-on labs, students were invited to complete the posttest assessment, which consisted of the same 17 questions.   main results – the researcher collected 59 pre-test and 27 posttest responses, although many respondents did not answer all the questions. moreover, none of the posttest responses from the off campus students was deemed usable by the researcher.   after attending the library orientation and lab sessions, students were more likely to choose the library or a librarian as their starting point for research (19% pretest, 40% posttest). students’ ability to identify book or chapter title in a citation (48% pretest, 92% posttest), and determine whether common knowledge required citations (87% correct in pre-test, 100% posttest) also appeared to improve after the library sessions. in addition, students’ skills in assessing the scholarliness and credibility of an article by its abstract also improved. while there were some anecdotal variations between responses between campus a and campus b groups, no statistically significant differences were noted.   conclusion – the study results suggest that regardless of format, library orientations and hands on lab session had positive effects on graduate students’ information literacy skills and knowledge.   commentary   this article strives to make a timely comparison between the effectiveness of online and in-person library instructions, and provides a survey of the related literature. the idea for the study is sound and would provide a good starting point for future research. however, due to flaws in the research design and execution, there are significant concerns with the study’s findings. an examination of this research using the ebl critical appraisal checklist (glynn, 2006) yielded an overall validity of 32%, below the accepted threshold of 75%.   nonetheless, the author should be commended for identifying and acknowledging several study limitations, including small sample size, high attrition rate, and errors in the assessment instrument. these observed limitations, such as the lack of proofreading before questionnaire distribution, will help others to minimize similar issues in future research.   on the other hand, readers would have benefited from some additional details from the author. for instance, the rationale and sources used in developing the assessment instrument were not addressed, the content of the in-person orientation and online tutorials were not discussed, and a copy of the actual instrument was unavailable. consequently, it would be difficult for interested readers to adopt and replicate similar studies.   there were also indications of personal biases in the assessment design and evaluation. in one instance, the author showed preferences for “ask a librarian” and “head to the library” as good choices for the question about starting points for one’s research, but neglected to provide online library resources as an answer option.  likewise, after acknowledging that “there [was] not a correct answer” for a different question about database selection, the author then evaluated the responses based on the choice he had believed to be “the best.” similarly troubling, none of the posttest responses from the off campus group was deemed usable by the author without further clarification.   moreover, the study’s actual population size, and thus the overall response rate, was unavailable. because participation was anonymous and voluntary, not only was the 54% (32 of 59) pretest/posttest attrition a concern, but one also cannot confidently conclude that the posttest responses were made by respondents who had completed the pretest. response rate for the individual questions was also not consistently provided, even though the author noted that most respondents skipped questions.   therefore, because of all these issues with the research design and analysis, readers are advised to consider the findings from this study with caution. however, with some improvements such as the addition of a coding method to track individual respondents’ pretest/posttest performance, this article can serve as a good starting point for those interested in similar research.   reference   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. article   using photo-elicitation with native american students to explore perceptions of the physical library   karen neurohr community outreach and assessment librarian oklahoma state university library stillwater, oklahoma, united states of america email: karen.neurohr@okstate.edu   lucy bailey social foundations and educational studies, director of gender and women's studies oklahoma state university stillwater, oklahoma, united states of america email: lucy.bailey@okstate.edu     received: 16 feb. 2016   accepted: 4 april 2016        2016 neurohr and bailey. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – this research project explored native american students’ perceptions of the edmon low library at oklahoma state university (osu). the study sought to understand how native american students perceived the role of the academic library in their lives, and which elements of the library students depicted and described as holding meaning for them.   methods – photo-elicitation, a form of visual research and a participatory research method, was the primary method chosen to explore students’ perceptions of the library. to qualify for this study, students self-identified as native american and as frequent library users. they also had completed three or more semesters of study at osu. five students followed a photo prompt for taking at least fifteen pictures of the library, then participated in two separate interviews with the primary researcher. participants also completed a demographic/questionnaire form, answered semi-structured questions, and ranked the photos they took.   results – this study produced several emergent findings. first, students expressed uncertainty about the library’s books. second, functional library tools such as express printers and library signage played a valuable role for facilitating student work. third, the method of photo-elicitation was enjoyable for students and served as library discovery. fourth, native american resources and exhibits in the library had varied salience for students.   conclusion – limited research focuses on native american students in academic libraries, particularly on how students use and experience the library. exploring how individual students who identify as native american perceive the university library enhanced our understanding of how libraries in predominantly white institutions (pwis) can best serve and support students. this study provided insight into the method of photo-elicitation interviews. this research also provided practical benefits for student participants through increased library knowledge.   introduction   the edmon low library is located at the heart of oklahoma state university, (osu), a land-grant, research i institution. a modified georgian brick building graced by a fountain and sweeping library lawn, the library is a campus icon. this six-floor building, which opened in 1953, has a public seating count of 1800. the main entrance opens to a marble lobby and grand staircase. the first floor includes service and checkout desks, current periodicals, the largest computer cluster on campus, numerous printers, group study rooms, and a coffee shop. the other five floors house varied material and structural resources: books; group tables and individual study carrels; gallery space; soft and hard furniture; a computer training room; and the math learning success center. in fall 2015 the osu undergraduate headcount was 21,046 students, with 1,138 self-identifying as american indian/alaskan native and 29 students self-identifying as native hawaiian/pacific islander (oklahoma state university, 2015). the diversity of the student body is steadily increasing at this predominantly white institution (pwi). the university continues to rank high nationally on degrees conferred on native american students (top native american, 2014). however, as administrators at other institutions have noted, retaining the majority of native american students both within pwis and in tribal colleges remains an unrealized institutional goal (belgarde & loré, 2007; brayboy, fann, castagno, & solyom, 2012). at osu, the first-year retention rate and six-year graduation rate of full-time freshmen native students, as well as other racial/ethnic groups, still lags behind those of white students (oklahoma state university, 2013). oklahoma has the second highest state population of american indian/alaskan native people alone-or-in-combination in the united states (norris, vines & hoeffel, 2012). nine percent of the population in oklahoma identifies as american indian/alaskan native (united states census bureau, 2014), and oklahoma has 38 federally-recognized tribes (national conference of state legislatures, 2016). nationally, native americans struggle with high rates of poverty, unemployment, high school attrition (krogstad, 2014), and are under-represented in postsecondary degree completion. a congressional mandated study of underrepresented racial/ethnic groups in higher education found that, at 14.4%, american indian/alaskan natives have the lowest six-year graduation rates among the groups noted (ross et al., 2012). nationwide, federal and institutional entities have developed programs such as trio, gear up, and college horizons to serve, advance, and retain native american and other underrepresented students (brayboy et al., 2012), but little research explores the library’s potential to serve underrepresented students in this broader mission.   the association of college and research libraries (acrl) produced the value of academic libraries initiative to address higher education’s emphasis on “assessment, accountability, and value” (oakleaf, 2010, p. 7). oakleaf (2010) conducted a comprehensive research review and report of the state of contemporary libraries and provided suggestions for a research agenda since that time, research into the library’s role in student retention, learning, experiences, and perceptions has increased.    to explore the role that libraries play in the lives of some native american students, this qualitative study shares student perspectives and images through the method of participant-produced photo-elicitation. this promising methodology can provide insights into students’ perceptions of libraries because students are the primary agents in choosing and conceptualizing the visual and verbal data. their reflections can inform future research and practice. the research questions were:   how do native american students perceive the role of the academic library in their lives? which elements of the library do students depict and describe as holding meaning for them?   literature review   decades ago heyser (1977) recognized the need for more research into how libraries can better serve native american people. this area of research is slowly increasing. scholarship offers recommendations for librarians serving indigenous people to learn about their histories and cultural rights, and to develop culturally-relevant library services (burke, 2007; hills, 1997; eric clearinghouse on rural education and small schools, 1971; rockefeller-macarthur, rockefeller, & macarthur, 1998; roy & frydman, 2010; roy, hogan, & lilley, 2012; spencer, 1985; webster, 2005).   some studies have focused on college and university libraries specific to native peoples. zuber-chall (2007) provides a historical and current perspective of haskell indian nations university and advocates for increasing research and practice to serve indigenous libraries. studies of tribal college libraries have examined college administrators’ perceptions (metoyer-duran, 1992) and library employee roles (dilevko & gottlieb, 2002), and have described developing collections of culturally-relevant books (koelling, 1995), and supporting and preparing students (patterson & taylor, 1996).   recent library studies recommend avenues of support for underrepresented college students, including library outreach to summer academic programs (love, 2009), multicultural student service centers (aguilar & keating, 2009), american indian studies programs (alexander, 2013), and indigenous support programs in australian academic libraries (hare & abbott, 2015).   studies focused on native americans in higher education rarely, if ever, mention the academic library (brayboy et al., 2012; carney, 1999; conroy, 2013; huffman, 2008, 2010; oosahwe, 2008; reyhner & eder, 2007; tierney, 1992). one exception is garrod and larimore’s (1997) study which included students’ fond descriptions of their early library experiences. one student in that study described three areas she considered “homes” during college: her dorm room and two different campus libraries (p. 195). ultimately, native american students’ perceptions of the library and its role in their lives and their academic success remain relatively unexplored areas of research.   methodology   this multicase study of five native american students attending osu uses participant-produced photo-elicitation as the primary mode of inquiry to explore the phenomenon of interest: students’ perceptions of the academic library. case studies use varied data sources to explore “how” or “why” questions (yin, 2009). for stake (1995), a case researcher’s goal is to focus on the “particularity and complexity of a single case, coming to understand its activity within important circumstances” (p. xi). in multicase study, examining each individual case in detail contributes to understanding of the whole (stake, 2005).   furthermore, multicases are linked in some ways (stake, 2005). the cases for this study are linked in at least two ways. first, all participants self-identified as native american who use the physical library at least three times a month. the osu library’s previous libqual® survey results helped inform this frequency; furthermore, the frequency indicated that students perceived the library as a resource in some way and had developed familiarity with at least some aspects of the library. the second link across the cases is students’ persistence toward graduation through completing at least three semesters of college, an important characteristic of the student retention rates at osu. new students who leave the university tend to leave during or shortly after their freshman year. having completed at least three semesters of study indicates that students have passed the first major marker in attrition patterns and are progressing toward degree completion.   photo-elicitation, a form of visual or image-based research and a participatory research method, is a creative method for exploring students’ perceptions of the library. sometimes called photo-interviewing, photo-elicitation was first tested and described by collier in the 1950s (harper, 2012, p.157). in his comparative study, collier (1957) found that interviews based on photographs were more definite and effective than interviews conducted without photographs. whether used as a primary or supplementary tool of data-gathering, a primary aim of photo-elicitation is to use images as prompts to elicit an interviewee’s comments, meanings, and perspectives. the inquiry method ideally honors a collaborative, conversational exchange that enables participants to share meanings and contribute to the direction of the interview.   the premise of such an approach is that images hold and evoke varied meanings and values that can offer unique insights and stimulate productive conversations about an individual’s world views. the images a researcher might incorporate into their research design may be as diverse as the goals of the study, including advertisements, photographs, paintings, video-clips, and web based images. similarly, depending on the study purpose, either the researcher or the participant can provide or produce the images that serve as vehicles for discussion or the researcher/participant can produce the images—and the meaning—in collaboration. in this study, we incorporated only participant-generated images as we were invested in fostering students’ agency in deciding which photographs they would take, how they would take them, and how they would teach others the meaning they held (lapenta, 2011).   very few library research studies have utilized photo-elicitation. shao-chen (2006) conducted a case study of how past library experiences influenced current library perceptions of first-year taiwanese graduate students. her methods included both participant and researcher-created photographs, along with interview questions based on libqual+ dimensions. she framed her coding analysis on libqual categories. the study included the finding that past library experiences shaped participants’ current perceptions. briden (2007) utilized participant-produced photographs to study students’ lives and research behavior at the university of rochester. six years later briden and george (2013) utilized participant-produced photographs to study academic work places of students. additional photo-elicitation library studies include duke and asher (2011) and gabridge, gaskell, and stout (2008).   methods   as of 2015, five students have participated in this institutional review board approved study: amanda, kellie, megan, sage, and charlie [pseudonyms]. the researchers masked all potentially-identifying information (e.g., hometown, tribal affiliation, etc.) to protect identity. all participants identified as members of tribal nations in oklahoma. they ranged in age from 19 to 23, and from sophomore to senior status at the time of interviewing. our study participants were the agents and instruments in the process of data collection through deciding which photos to take, where to take them and the meaning they created and shared about those images.   the coordinator of native american affairs in the office of multicultural affairs assisted in recruiting students through word-of-mouth, posting fliers in her office, and forwarding invitations through email to members of the native american student association (nasa). the primary researcher offered an incentive of ten dollars per hour of participant’s time, a sufficiently-modest amount to prevent coercion. exchanging money or a gift card for participants’ time reflects the ethical and methodological principle of reciprocity: providing something of value to participants in exchange for students providing their valuable perspectives for research (patton, 2002).   the primary investigator (pi) met with students two different times in her office and recorded the interviews with an olympus digital recorder. megan and sage’s interviews and the pi’s member checking, a method to check transcription accuracy and clarify and expand data (creswell, 2009), occurred during their sophomore year. charlie completed his interview and member check during his junior year. amanda completed our interview during her junior year and the member check when she was nearing graduation and applying to graduate schools. finally, kellie participated during her senior year and the member check occurred after she graduated and was making plans for graduate school. while member checks might occur at any time after an interview, transcription, or analytic stage, the varying dates in which students participated reflected their preferences and busy schedules. the member checks of those nearing graduation or post-graduation suggested a greater deal of retrospection and reflection.   meetings were scheduled for the students’ convenience when the library is typically less busy, to minimize intrusion of library users. after the pi met the students, the first phase consisted of reviewing and signing the consent form, completing a biographical intake sheet and short survey, and responding to three initial semi-structured interview questions:   how did you first learn about the library? describe any interactions you have had with library employees. in what ways does the library make a difference for you?   the short survey and initial questions were designed to prompt students’ reflections on their library use and experiences. this information also provided key contextual information for the photographs. when students were uncertain about library resources or services, or asked questions, the pi (a librarian) jotted a note, “parking” the idea until the interview ended. at that point, she answered each question, demonstrated online tools, and/or accompanied students to the library stacks to demonstrate how books are shelved.   in the second phase of the first meeting, the pi showed participants how to use the camera and reviewed the photo prompt:   take a least fifteen different photos of the edmon low library. photos should represent some meaning for you. meaning may be of things you use in the building or online, favorite things, things you dislike or like, things that are confusing or easy, or new things you discover. photos can be of exterior or interior things. photos cannot show the faces of people.   the description focused on student-driven meaning and provided prompts as suggestions. the primary researcher loaned students a digital camera for the activity and reminded students that they would discuss each photo when they returned. when participants returned, the pi explored their experience of taking photos, uploaded the images, and discussed them one at a time. later, the pi made back-up copies and removed the images from the camera to prevent others from seeing the photos. she transcribed the interviews verbatim to prepare for the second meeting focused on member-checking and photo-ranking.   the second meeting with participants took place between one month and twelve months after the initial interview, focusing on three activities. first, the pi asked a series of questions to clarify responses based on initial analysis of the first interview transcript. second, the pi asked participants to discard one photograph and explain why, and, finally, she asked participants to rank the top photos and explain the reasons. the pi audio-recorded these “member checks” and transcribed them verbatim as part of the analytic process (poindexter, 2002).   the data reflected several similarities among participants. while the study criteria specified at least three visits to the library per month, all students indicated they came to the library one to three times per week. similarly, they all used a variety of spaces and seating in the library, and they all actively participated in the native american student association. students (cases) differed across age, gender, major field of study, classification, and place of residence. also, use of the library’s online resources varied in frequency: sage and kellie said they used the online resources one to three times per week; amanda said she used the online resources more than three times per week, and megan and charlie described their use as more than three times per month.    both researchers analyzed the data in multiple ways and over time. as is characteristic of qualitative analysis in which the meaning-making process proceeds throughout and after data collection, the pi noted analytic insights during interviews, during immersion in the transcription process (poindexter, 2002), and through systematic analysis with the data corpus. in particular, the researchers relied on the concept of “pursuing members’ meanings” (emerson, fretz & shaw, 2011) to analyze data, seeking understanding through interviews and descriptions of photographs of what participants considered meaningful, relevant, and important about the library (p. 129). for example, we used the term “resources” to capture diverse elements of the library such as printers, books, and electrical outlets to reflect students’ perspective of what they found relevant and meaningful to their library use. similarly, using in vivo codes, the photographs were categorized by topic and meaning using the participants’ own words (glaser & strauss, 1967; saldaña, 2013; strauss & corbin, 1990), evaluated for convergence and divergence (guba, 1978), and substantive significance (patton, 2002, p. 467). all photographs were evaluated for comparison across the multi-cases.   findings   visual sociologist harper (2012) suggests that participant photos usually reflect two types: “images of mundane objects that are special because of their role in the lives being documented,” or “visualized metaphors, sentiments and emotions” (p. 205). the interviewing process revealed the various meanings participants ascribed to the objects and images in their photographs. participants took the following number of photographs: amanda, 17; kellie, 17; sage, 15; megan, 17; and charlie, 15 for a total of 84. surprisingly, the only exterior photo of osu’s historical library depicted its tower only, with the meaning ascribed to the chimes, not the iconic building.   students took pictures of objects they used, liked, or that helped them, and that seemed at first glance to be ‘mundane’ as harper (2012) notes, objects such as printers, computers, signs, and elevators. all five participants took photographs of individual study carrels, and four photographed the vending machine. the highest number of sentiment photos depicted objects pertaining to native cultures: two photos of native books, two photos of native americans in traditional tribal attire, and two photos of the library’s browsing room because of the native programming which has occurred there.   discussion   finding one: uncertainty about book use   several students indicated they appreciated books and reading but also expressed limited experience with this library resource and confusion about how to utilize it. in one case a student described an unpleasant experience with a library employee when they tried to check out a textbook on reserve. another felt uncertain about how the books were organized. of the 84 photographs students took, only four represented books. only sage (sophomore at the time of interviewing) and charlie (junior) had ever checked out a book from this library. amanda (junior), who had never checked out a book at osu, said, “all the rows and rows of books and i’ve always [wondered], ‘oh there’s so many books’ but i never really use any of them, so i just kind of wonder if people use them!”   megan, whose first photograph represented books, expressed her investment in books and their clear symbolic association with the library. yet her comments and questions related to her photographs of books also highlighted her limited understanding of how to access this library resource. she said,   my instant thought was to take a picture about books because i really do like to read, but i know nothing about how to check out a book, where to find the books i would be interested in, what books are reference books, what books, you know, where i could find books for my major or anything.   figure 1 photo taken by megan.   her second photograph of books highlighted both the limited experience described above and her feelings, when she discovered books of interest, that she had missed a key opportunity to explore an area of interest:    i had no idea. i guess i mean i would have known, if i would have thought about it, but i was just walking by a desk, [where] i was gonna to take pictures… and there were …tribal books [voice rising with surprise]….ah, i mean it’s a library so clearly, you know, when you think about it, you would think they would have like native american books and stuff, but it had just never come to my mind, and i was like “wow! maybe i would check some of those out,” if i would have been aware.     figure 2 photo taken by megan.     despite visiting campus between one and three times per week, and her affection for books, her awareness of the ways the library might serve her reading interests increased through this multicase study, echoing sage’s comment, “i like to read a lot of books on native american issues.” this finding underscores the importance of considering how libraries might better highlight and link students’ cultural interests with library resources. in addition, this data seems to reflect the trend scholars and librarians have noted of declining book use in academic libraries (association of research libraries, 2012; davis, 2011; de rosa, cantrell, carlson, gallagher & hawk, 2011; gardner & eng, 2005). yet the data also suggests that, for at least several of these students, one reason for this trend might be that students do not have the experience, information, and support to use such resources more fully.   finding two: valuable role of functional library tools   the varied pictures of objects that outsiders might see as ‘mundane’ were important to students because they facilitated their academic work. all students referred to the key equipment they use – particularly computers and printers – and the majority of the pictures represented such objects as scanners, printers, and computers. for example, an interesting set   of photographs represented common signs, such as a finding aid sign for electrical outlets. kellie described her quest for these signs because they enable her to charge her laptop:   i find these signs to be golden…i always am aware of these signs and i turned the corner and i saw this and i knew this was a picture that needed to happen.     figure 3 photo taken by kellie.   similarly, in pointing to his picture of “the quick print station” charlie said:   it’s on the first floor, and this is really helpful because if i have an assignment due, or a paper or something like that, i’ll just head to campus maybe a few minutes early, and i’ll go to one of these computers, get on my email or the online classroom, and i’ll just print off what i need for class, or assignments that are due and stuff like that.     figure 4 photo taken by charlie.   these pictures represent both the value of library resources to students because they aid in streamlining their academic work as well as, more broadly, the gradual shifting use of the library as information technology has expanded. what was previously a repository of texts, is now a multi-purpose space re-envisioned for the needs of contemporary students.   finding three: photo-elicitation as library discovery   participants remarked on the novelty of the research methodology and seemed to enjoy the photo activity. three participants said they envisioned specific photos once they heard the researcher’s prompt, while two participants described their photo taking, at first, as “random.” overall, participants seemed to approach the photograph-taking process in terms of representing how they use library resources, what they value in the library, and how participating in photography prompted them to discover new aspects of the library. they were surprised at some of the objects they encountered through the camera lens that they hadn’t noticed previously. megan remarked:   i found a lot of things. i would just walk by things and say [to self], ‘oh, i’d never really noticed that.’ when i was really looking for things, i started to realize a lot more that there is to the library.   this comment reflects a broader pattern in the photo methods in which revisiting a familiar environment with heightened awareness and a new purpose can uncover, as harper (2012) remarked, “previously unknown or unconsidered dimensions of social life” (p. 121). for instance, amanda was not aware that the library provided ipads or self-checkout stations. sage discovered the library’s exterior live webcam, and charlie discovered the current periodicals section.    the initial survey/checklist also revealed students’ varied levels of awareness concerning library services. only two participants were aware of the library’s textbook reserves, which sage wished he had known about earlier. a first generation student with financial constraints, he described a critical experience one semester in which a biology instructor deviated from the syllabus requirement to use only the newest edition of the textbook. sage purchased that edition, then he learned an older edition would suffice. he said, “that was pretty brutal for me... buying a hundred and fifty dollar textbook, that wasn’t even worth a hundred fifty.”   finding four: varied salience of native american resources and exhibits   students took seven pictures that focused explicitly on native american events or cultural exhibits. although these pictures comprise a small number of the photos overall, students narrated their meaning at length. all participants referred to aspects of the library which related to their individual nation or native american identity. sage took two pictures of native americans in a special “images of oklahoma” exhibit, and mentioned that one photo related to his culture and the other represented preserving traditional culture ways. three participants discussed the nasa cultural events they attended in the library’s browsing room.   other photographs underscored the support library resources provided for student events and for facilitating the submission of tribal paperwork, such as particular rooms and scanners. two of megan’s photographs prompted her discussion of absences – what wasn’t in the library – first, a fax machine that would facilitate her ability to efficiently submit her tribal scholarship documentation and a flyer promoting nasa events:   i thought that the little bulletin board was really cool because i noticed a lot of things that i would be interested in that i would otherwise not have any idea about, although i wish more of our native american stuff was on there [chuckle]…i noticed we didn’t have anything for [native american] awareness week on there.   figure 5 photo taken by megan.   participant data offers an important caution relevant to research with any cultural groups, which is that motivations and allegiances are multidimensional. a participant may always identify with their individual nation or native american culture, but individual interests, personality, and fields of study also shape perceptions of the library and of higher education in pwis. megan emphasized the distinction between what she likes to do as an individual, vs. what she thinks other native american students like to do. for example, she clarified:   i mean i like it here but i know a lot of people that i am friends with that are native american and they don’t come. like no one comes to the library. a lot of my friends like in nasa (native american student association) and in stuff, i’m like, “hey, do you want to go to the library?” they’re like, ‘no.’ …but i definitely feel like i don’t see very many of my native american friends here.   kellie’s perspective underscores the vital role student perceptions of the library might play in their willingness to walk through library doors and avail themselves of the resources. she said:   when i think of a library i think of something very institutional and educational…sometimes that can be a bit intimidating especially for someone of a minority. you think, “oh well, i don’t really belong here.” i hear a lot of my students [say things like], “i don’t deserve this” or “that is just too hard” or “that’s only where the smart people go.   similarly, megan suggested the library had a role in making visible culturally-relevant books and events: “i think native americans would feel…that would warm their heart a little bit just to know that [the library] respects that culture.” kellie suggested that the library should “break that stereotype” and convey “this is an environment for all people, and it’s not just only the really smart people [who] go here.”   for kellie personally, however, one photo and her words, shown here as a poetic re-presentation, convey her own intentionality upon entering the library:   when i’m stepping into this magnificent door i enter a different frame of mind, an academic frame of mind. study, focus, get this done.   figure 6 photo taken by kellie.   the other three participants offered divergent views of the library as a common space for native american students. amanda elaborated that people who say they don’t like studying in the library, just really don’t like studying at all.   conclusion   this study is significant for several reasons. first, it expands the limited research available on native american students’ perceptions of the library. second, it advances knowledge about the meaning of library tools/resources for students persisting toward degree completion, including those elements that may seem “mundane” at first glance, but that students cumulatively use in multidimensional ways to support their institutional and academic work. third, researchers have not used the method of photo-elicitation interviews widely for either academic libraries or university settings. it provides a way of “seeing” that demonstrates the multidimensional meaning of familiar objects and uncovers questions students have about library resources and how to use them.   through both method and findings, the research has practical benefits for librarians who seek to serve native american students and/or who manage the physical facilities of libraries. providing exhibits, programs, and materials related to native culture is important, but cultivating awareness of those offerings is equally important. the study also provides practical benefits for student participants. first, several participants described financial constraints and indicated they were motivated to participate because of the research incentive. second, students’ knowledge of library resources and services increased through participating, and this may have positive academic implications for students.   this study’s findings might be distinctive to the participants and the site of this study. interviews always provide self-report and partial perspectives that observation data about what participants do can enhance. although this study rests on a small sample that limits its broad applicability, the research focus, multiple data sources, and methodology prompt unique insights upon which future research can build. currently, the institutional review board protocol for this study at our institution remains open as we intend to continue data analysis, add another data source, and recruit additional participants to explore how perceptions of the library change over time. the methodology also offers promise for studying individuals in other contexts; the pi has used similar methods to collect data with first-generation students, and data analysis in that study is underway.   a unique opportunity to share study findings arose in july 2014 when the cherokee college preparatory institute was held at osu. the week-long college readiness program connected fifty-four native american high school juniors and seniors with twenty-one representatives from twelve universities. the pi was invited to coordinate an “introduction to the academic library.” students toured the library, practiced using online resources, and attended an informal presentation of the research data. findings also informed the session content which focused on library resources with native cultural connections. these high school students learned what some persisting osu native american college students think about the library. the students were attentive and asked insightful questions afterward.   research with native americans requires awareness of and sensitivity to the history of research on native american issues and people marked by colonialism, exploitation, misunderstanding and misrepresentation that has often devalued native peoples and ways of knowing (huffman, 2010; klug, 2012; mihesuah, 1998; mihesuah & wilson, 2004; shotton, lowe & watterman, 2013). given the diversity of sovereign nations and the individuals who identify as native american, the history of exploitative research (smith, 2012), and the complexity of students’ experiences in pwis, researchers both within and outside of the culture must take into account the complexity of identity (brayboy, 2000), and the ways indigenous people consider their identities and needs to be salient in their library experience.   in 2012 the association of college and research libraries (acrl) racial and ethnic diversity committee developed cultural competency standards for academic libraries the eleven acrl standards cover cross-cultural leadership, skills and knowledge, library collection development, provision of library programs and services, linguistic and workforce diversity, and research (association of college and research libraries, 2012). these resources may help broaden the cultural competency of librarians who are members of dominant cultural groups. future research studies could focus on exploring other ways the library might better serve native american students, as well as other underrepresented groups.   acknowledgements   the authors sincerely thank the eblip reviewers for their comments. this research was originally delivered at the 2014 library assessment conference, and an early version of that paper was produced in the conference proceedings.   references   aguilar, p., & keating, k. (2009). satellite outreach services program to under-represented students: being in their space, not on myspace. the reference librarian, 50(1), 14–28. http://dx.doi.org/ 10.1080/02763870802546365   alexander, d. l. (2013). american indian studies, multiculturalism, and the academic library. college and research libraries, 74(1), 60–68. http://dx.doi.org/10.5860/crl-311   association of college and research libraries. (2012) diversity standards: cultural competency for academic libraries. (2012). retrieved from http://www.ala.org/acrl/standards/diversity     association of research libraries (2012). service trends in arl libraries, 1991–2012. retrieved from http://www.arl.org/storage/documents/service-trends.pdf     belgarde, m. j. & loré, r. k. (2007). the retention/intervention study of native american undergraduates at the university of new mexico. in a. seidman (ed.), minority student retention: the best of the journal of college student retention: research, theory, & practice (pp. 167-194). amityville, ny: baywood publishing.   brayboy, b. m. (2000). the indian and the researcher: tales from the field. international journal of qualitative studies in education, 13(4), 415–426. retrieved from http://dx.doi.org/10.1080/095183900413368   brayboy, b. m. j., fann, a. j., castagno, a. e., & solyom, j. a. (2012). postsecondary education for american indian and alaskan natives: higher education for nation building and self-determination, ashe higher education report, 37(5), 53–72. http://dx.doi.org/10.1002/aehe.3705   briden, j. (2007). photo surveys: eliciting more than you knew to ask for. in n. f. foster & s. gibbons (eds.). studying students: the undergraduate research project at the university of rochester (pp. 40–47). chicago, il: association of college and research libraries. briden, j., & george, s. (2013). picture my work. in n. f. foster, (ed.). studying students: a second look (pp. 25–44). chicago, il: association of college and research libraries.   burke, s, k. (2007). the use of public libraries by native americans. the library quarterly, 77(4), 429–461. http://dx.doi.org/10.1086/520998   carney, c. m. (1999). native american higher education in the united states. brunswick, nj: transaction publishers.   collier, j. (1957). photography in anthropology: a report on two experiments. american anthropologist, new series. 59(5), 843-859.   conroy, j. c. (2013). native american student perceptions regarding barriers, supporting factors, and retention strategies at two midwestern post-secondary institutions (doctoral dissertation). retrieved from proquest. (3589835).   creswell, j. w. (2009). research design: qualitative, quantitative, and mixed methods approaches. los angeles, ca: sage. davis, d. m. (2011). trends in academic libraries, 1998 to 2008. retrieved from http://www.ala.org/research/sites/ala.org.research/files/content/librarystats/academic/als%209808%20comparison.pdf     de rosa, c., cantrell, j., carlson, m., gallagher, p., hawk, j., & sturtz, c. (2011). perceptions of libraries, 2010: context and community, a report to the oclc membership. retrieved from http://oclc.org/content/dam/oclc/reports/2010perceptions/collegestudents.pdf     dilevko, j., & gottlieb, l. (2002). making a difference in their own way: the role of library directors and non-directorial staff at tribal college libraries. the journal of academic librarianship, 28(5), 306–318. http://dx.doi.org/10.1016/s0099-1333(02)00322-1   duke, l. m., & asher, a. d. (2011). college libraries and student culture: what we now know. chicago, il: ala editions.   eric clearinghouse on rural education and small schools. (1971). manual for providing library services to indians and mexican americans (ed047872). las cruces, nm: new mexico state university.   gabridge, t., gaskell, m., & stout, a. (2008). information seeking through students’ eyes: the mit photo diary study. college & research libraries, 69(6), 510–523. http://dx.doi.org/10.5860/crl.69.6.510   gardner, s. & eng, s. (2005). what students want: generation y and the changing function of the academic library. portal: libraries and the academy 5(3), 405–420.   garrod, a., & larimore, c. (eds.). (1997). first person, first peoples: native american college graduates tell their life stories. ithaca, ny: cornell university press.   glaser, b. g., & strauss, a. l. (1967). the discovery of grounded theory: strategies for qualitative research. hawthorne, ny: aldine publishing.   guba, e. g. (1978). toward a methodology of naturalistic inquiry in educational evaluation. los angeles, ca: center for the study of evaluation, university of california.   hare, j., & abbott, w. (2015). library support for indigenous university students: moving from the periphery to the mainstream. evidence based library and information practice, 10(4), 80–94. http://dx.doi.org/10.18438/b86w3q   harper, d. (2012). visual sociology. new york: routledge.   heyser, r.g. (1977). native americans and libraries: a citation-entry analysis (doctoral dissertation). retrieved from proquest. (7808954).   hills, g.h. (1997). native libraries: cross-cultural conditions in the circumpolar countries. lanham, md: the scarecrow press.   huffman, t. (2008). american indian higher educational experiences: cultural visions and personal journeys. new york, ny: peter lang.   huffman, t. (2010). theoretical perspectives on american indian education: taking a new look at academic success and the achievement gap. lanham, md: altamira.   klug, b. j. (ed.). (2012). standing together: american indian education as culturally responsive pedagogy. lanham, md: rowman & littlefield education.   koelling, h. (1995). a model collection process for specialized environment: recommending reference titles for tribal community colleges. community and junior college libraries, 8(1), 63–68. http://dx.doi.org/10.1300/j107v08n01_08   krogstad, j.m. (2014). one-in-four native americans and alaskan natives are living in poverty. in pewresearchcenter. retrieved from http://www.pewresearch.org/fact-tank/2014/06/13/1-in-4-native-americans-and-alaska-natives-are-living-in-poverty/   lapenta, f. (2011). some theoretical and methodological views on photo-elicitation. in e. margolis & l. pauwels (eds.), the sage handbook of visual research methods (pp. 201-213). thousand oaks, ca: sage.   love, e. (2009). a simple step: integrating library reference and instruction into previously established academic programs for minority students. the reference librarian, 50(1), 4–13. http://dx.doi.org/10.1080/02763870802546357   metoyer-duran, c. (1992). tribal community college libraries: perceptions of the college presidents. the journal of academic librarianship, 17(6), 364-369.   mihesuah, d. a. (ed.). (1998). natives and academics: researching and writing about american indians. lincoln, ne: university of nebraska press.   mihesuah, d. a. & wilson, a. c. (eds.). (2004). indigenizing the academy: transforming scholarship and empowering communities. lincoln, ne: university of nebraska press.   national conference of state legislatures. (2016). federal and state recognized tribes. retrieved from http://www.ncsl.org/research/state-tribal-institute/list-of-federal-and-state-recognized-tribes.aspx#ok   norris, t., vines, p. l., & hoeffel, e. m. (2012). american indian and alaskan native population: 2010. in united states census bureau. retrieved from http://www.census.gov/prod/cen2010/briefs/c2010br–10.pdf   oakleaf, m. (2010). value of academic libraries: a comprehensive research review and report. chicago, il: association of college and research libraries. retrieved from http://www.ala.org/acrl/sites/ala.org.acrl/files/content/issues/value/val_report.pdf   oklahoma state university. (2013). retention of students: retention and graduation rates of new freshmen by ethnicity. retrieved from https://irim.okstate.edu/sites/default/files/ studentprofile/2013/pdf/2013retention.pdf   oklahoma state university. (2015). five-year academic ledger: stillwater campus. retrieved from  https://irim.okstate.edu/sites/default/files/academicledgers/ oklahomastateuniversity.pdf   oosahwe, e. s. (2008). strategizing success: narratives of native american students in higher education (doctoral dissertation). retrieved from proquest. (3304450). patterson, l., & taylor, r. h. (1996). tribally controlled community college libraries: a paradigm for survival. college and research libraries, 57(4), 316–329. http://dx.doi.org/10.5860/crl_57_04_316   patton, m. q. (2002). qualitative research & evaluation methods. thousand oaks, ca: sage publications.   poindexter, c. c. (2002). meaning from methods: re-presenting narratives of an hiv-affected caregiver. qualitative social work, 1(1), 59–78. http://dx.doi.org/10.1177/147332500200100105   reyhner, j. & eder, j. (2007). higher education of american indians. in h. s. wechsler, l. f. goodchild, & l. eisenmann (eds.), ashe reader: history of higher education (pp. 694–702). boston: pearson custom publishing.   rockefeller-macarthur, e., rockefeller, r. & macarthur, p. (1998). american indian library services in perspective: from petroglyphs to hypertext. jefferson, nc: mcfarland & company.   ross, t., kena, g., rathbun, a., kewalramani, a., zhang, j., kristapovich, p., & manning, e. (2012). higher education: gaps in access and persistence study, 2012. retrieved from http://nces.ed.gov/pubs2012/2012046.pdf   roy, l. & and frydman, a. (2014, march 3). library services to indigenous populations: case studies. in international federation of library associations. retrieved from  http://www.ifla.org/files/assets/indigenous-matters/publications/indigenous-librarianship-2013.pdf   roy, l., hogan, k., & lilley, s. (2012). balancing access to knowledge and respect for cultural knowledge: librarian advocacy with indigenous peoples’ self-determination in access to knowledge. in j. lau, a. m. tammaro, & t. j. d. bothma (eds.), libraries driving access to knowledge (pp. 163–189). berlin/boston: de gruyter saur.   saldaña, j. (2013). the coding manual for qualitative researchers (2nd ed.). thousand oaks, ca: sage publications. shao-chen, l. (2006). perceptions of u. s. academic library services of first-year graduate students from taiwan: a photo-elicitation study (doctoral dissertation). retrieved from proquest umi dissertations publishing. (umi no. 3234607)   shotton, h. j, lowe, s. c., & waterman, s. j. (eds.). (2013). beyond the asterisk: understanding native students in higher education. sterling, va: stylus.   smith, l. t. (2012). decolonizing methodologies: research and indigenous peoples. (2nd ed.). london: zed books.   spencer, a. f. (1985). a perception study of the accomplishment of the goals for indian library and information service in native american educational institutions (doctoral dissertation). retrieved from proquest. (8522753).   stake, r. e. (1995). the art of case study research. thousand oaks, ca: sage publications.   stake, r. e. (2006). multiple case study analysis. new york: guilford press.   strauss, a. l., & corbin, j. m. (1990). basics of qualitative research: grounded theory procedures and techniques. newbury park, ca: sage publications. tierney, w. g. (1992). official encouragement, institutional discouragement: minorities in academe – the native american experience. norwood, nj: ablex publishing.   the top native american degree producers. (2014, november 20). diverse: issues in higher education, 31(21), 14–17.   united states census bureau. (2014). state and county quickfacts: oklahoma retrieved from http://quickfacts.census.gov/qfd/states/40000.html   webster, k. (ed.). (2005). library services to indigenous populations: viewpoints and resources. chicago: american library association.   yin, r.k. (2009). case study research: design and methods. (4th ed.). thousand oaks, ca: sage publications.   zuber-chall, s. (2007). tommaney library at haskell indian nations university: an ethnographic case study (doctoral dissertation). retrieved from proquest. (3276489).     appendix demographic/questionnaire   participant: __________________________________________________________________ gender: male _____ or female _____ age: _____ marital status: single _____ or married _____ children: no _____ or yes _____ if yes, how many? _____ list your hometown and state: ___________________________________________________ are you the first in your immediate family to attend college? no _____     yes_____   education   list your year in school: _____________________________ list your major field(s) of study: _______________________________________________ do you live on campus? no ____ yes ____ did you transfer to osu from another college? no _____   yes _____   which of the following best describes how frequently you come to the library? (check one) 1-3 times a week    _____               more than 3 times a week _____ other, list: __________ 1-3 times a month   _____              more than 3 times a month _____              1-3 times a semester _____            more than 3 times a semester _____            which of the following best describes how frequently you use the library’s online resources? (check one) 1-3 times a week _____                   more than 3 times a week _____ other, list: __________ 1-3 times a month _____                more than 3 times a month _____              1-3 times a semester _____            more than 3 times a semester _____                 which of the following library spaces/seating have you used? (check all that apply) 1st floor computers/printers _____                            café libro  _____ 2nd floor browsing room _____                                soft seating  _____ 2nd floor reading room _____                                   group study rooms  _____ 2nd floor seating by the exhibits _____                    group study tables in the open _____ 2nd floor computer instruction room _____         individual study desks (study carrels) _____ 3rd floor _____                                                                  writing center outpost in the library _____ 4th floor _____                                                                 math learning success center  _____ 5th floor _____                                                                 other (please list): ______________________ basement  _____     which of the following library services/resources have you used? (check all that apply) library desktop computers _____                              large computer monitors _____ library scanners _____                                                 checked out an ipad _____   checked out a laptop _____                                         textbooks on reserve _____                                                interlibrary loan _____                                                  asked librarian for help _____ library reserves _____                                                 wireless network connection _____       checked out a book _____                                            library search box on homepage ______                 library printers _____                                                    chat box on library homepage _____ library databases _______                                            digital library signage _____                                              library website _____                                                    used book in library, didn’t check it out _____ electronic white board/projector in group study room _____      other, please list:  __________________   research article   digging in the mines: mining course syllabi in search of the library   keven m. jeffery digital technologies librarian san diego state university library san diego, california, united states of america email: kjeffery@mail.sdsu.edu   kathryn m. houk health & life sciences librarian san diego state university library san diego, california, united states of america email: khouk@mail.sdsu.edu   jordan m. nielsen entrepreneurship, marketing & business data librarian san diego state university library san diego, california, united states of america email: jnielsen@mail.sdsu.edu   jenny m. wong-welch stem librarian san diego state university library san diego california, united states of america email: jwongwelch@mail.sdsu.edu   received: 2 sept. 2016    accepted: 2 jan. 2017           2017 jeffery, houk, nielsen, and wong-welch. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective the purpose of this study was to analyze a syllabus collection at a large, public university to identify how the university’s library was represented within the syllabi. specifically, this study was conducted to see which library spaces, resources, and people were included in course syllabi and to identify possible opportunities for library engagement.   methods a text analysis software called qda miner was used to search using keywords and analyze 1,226 syllabi across eight colleges at both the undergraduate and graduate levels from the fall 2014 semester.   results of the 1,226 syllabi analyzed, 665 did not mention the library’s services, spaces, or resources nor did they mention projects requiring research. of the remaining 561, the text analysis revealed that the highest relevant keyword matches were related to citation management (286), resource intensive projects (262), and library spaces (251). relationships between categories were mapped using sorensen’s coefficient of similarity. library space and library resources (coefficient =.500) and library space and library services (coefficient-=.457) were most likely to appear in the same syllabi, with citation management and resource intensive projects (coefficient=.445) the next most likely to co-occur.   conclusion the text analysis proved to be effective at identifying how and where the library was mentioned in course syllabi. this study revealed instructional and research engagement opportunities for the library’s liaisons, and it revealed the ways in which the library’s space was presented to students. additionally, the faculty’s research expectations for students in their disciplines were better understood.     introduction   librarians have long seen syllabi as a valuable way to gauge how effectively library services have been integrated into the curriculum. in 2015, the san diego state university library leveraged a campus syllabus collection to do a broad analysis of how effectively the library was integrating itself into the curriculum. the san diego state university (sdsu) syllabus collection was initiated after a 2011 request from the student government for syllabi to be made available in digital format before the deadline for course registration. students were interested in having access to the course requirements, especially factors like assignments, fieldwork, or required travel that may not be available in the course catalog description. at the time of the request, syllabi were mandated by the university senate to be made available only in print from department offices. the documents were therefore not easily available to students who might be registering for classes remotely.   even though the primary goal for creating an open and accessible syllabus database was to provide easier access to course information for students, other potential uses for the syllabus collection have emerged. in addition to being an open syllabus repository, it also represents a storehouse of data about courses, faculty, and students at sdsu. in 2015, four librarians in the university’s library mined the syllabus collection to discover how the library was being referenced and used at the university.   creating the syllabus collection   a working group led by the dean of the division of undergraduate studies identified the library as a partner on the project due to its having the experience and resources to manage existing collections of university documents, such as digital theses and course calendars. the library offered to support the project using a dspace instance, the same software used by the library for other campus publications. as of summer 2015, 90% of academic departments were participating at some level in the syllabus collection, and the collection had surpassed 8,500 documents. from june 2014 through may 2015, over 1 million syllabi had been downloaded from the database suggesting the collection has fulfilled the original goal of providing access to students and those interested in the university course offerings.   issues like intellectual property were relatively easily overcome with the option to use a course information template instead of the syllabus, but challenges remain. even though the campus supports the database, there is no real incentive for participating, so gaining participation from the last 10% of departments may be a challenge. while uploading documents is not a hard task for administrative staff, taking approximately one minute per document, it is sometimes still a challenge obtaining the syllabi from the teaching faculty. there is a suggested metadata standard, but there is no enforcement of the standard for the collection. as section codes are not often included, it is not easy to connect the syllabus in dspace to the course calendar to accurately determine the level of participation.   literature review   prior to beginning the analysis, a literature search was undertaken using databases specific to library science, such as library literature & information science index and library, information science & technology abstracts, along with more general subject databases, such as ebsco academic search premier and proquest research library, as well as the proquest summon discovery tool. the authors performed independent searches for articles dealing broadly with syllabi analysis and decided as a group which articles were appropriate to the project. most of the studies examined, out of necessity, looked at small samples of documents that could be obtained directly from faculty or class sites available on the web. these analyses have been conducted in a variety of ways, including by random sampling, by targeting specific student populations/courses, and by focusing on specific degree or major programs.   syllabi analyses involving random samples   rambler (1982) identified a random sample of 162 courses from the pennsylvania state university winter 1979 course schedule and collected syllabi and course documents directly from faculty. she then rated these according to a three-point scale for library usage, finding that 63% of the courses required no library use (p. 156) and that library use increased with class level. rambler found that only 8% of the courses analyzed made heavy use of the library (pp. 158-159). smith, doversberger, jones, parker, & pietraszewski (2012) looked at a similarly sized sample, first identifying the 5,173 course sections offered in spring 2009 by the university of notre dame. they then eliminated graduate courses, laboratory sections, and directed research classes. they also eliminated syllabi from sections known to have a library component. of the remaining 1,496 sections, they selected a random sample of 300 classes and obtained 144, or 52%, of the documents for the sample. the syllabi were then rated for library use according to a four-point scale. they found 43% of the syllabi examined required no library use, and only 38% required use of the library beyond course reserves, with library use increasing with class level (pp. 266-267).   williams, cody, & parnell (2014) started with a list of 3,125 class sections offered by the university of north carolina at wilmington in the fall 2002 and spring 2003 semesters and identified 828 available via the “free web.” of these 828, they identified 253 upper-level courses in 34 disciplines for analysis. they found 41% of classes used the library for research papers or projects, 18% used the library for reserve materials, 16% required library use for special projects and book reviews, 12% offered extra-credit library assignments, and 11% offered optional use of materials not on reserve (p. 271).   syllabi analyses involving special student populations/courses   vanscoy & oakleaf (2008) obtained the course lists for a random sample of 350 first-semester freshmen students from the north carolina state university registrar. they obtained a complete set of syllabi for 139 students from the internet or directly from instructors. they found 97% of the 350 students were required to find research resources with the number jumping to 100% for the 139 students where a complete syllabi sample was available (pp. 569-570). o’hanlon (2007) examined winter quarter, 2006 syllabi for writing courses and senior capstone courses at ohio state university, analyzing 71 syllabi provided by instructors or found on the internet (p. 174). these 71 syllabi represented 44, or 30%, of course sections for the writing course and 27, or 55%, of the senior capstone courses (p. 181). fifty-nine percent of writing course syllabi indicated a writing assignment requiring external research (p. 182), and 70% of the senior capstone courses mentioned the same. o’hanlon in looking for research related lectures in the syllabi found that while some courses offered supplemental support, “no indication of class lectures by instructors or librarians on research methods was found in these syllabi” (p. 183).   syllabi analyses involving majors or programs   boss & drabinski (2013) examined a comprehensive set of 79 undergraduate and graduate course syllabi obtained directly from the school of business at long island university. they then searched the syllabi for the word “library” and rated the syllabi according to a set of questions developed from the association of american colleges & universities information literacy standards (pp. 267-268). the authors found that while 51 of the syllabi included a research assignment, only 22 directed students to the library or a librarian (p. 270). dewald (2003) examined syllabi for courses required for the completion of a b.s. in business administration at penn state university. the author looked at examples from the 2000–2001 and 2001–2002 academic years and rated library usage according to a four-point scale (p. 35). dewald found that 48.9% had no library use, 31.6% required library use for short assignments, and 18.3% required significant research assignments (p. 39).   aims   by examining a large group of syllabi during a specific timeframe, the librarians conducting this study sought to identify how the library was referenced in courses at the university. it was expected that most mentions of the library in course syllabi would be related to spaces within the library’s physical location rather than personnel or services. it was hoped that the following key questions could be answered during this research study:   is the library mentioned in the course syllabus? if the library is mentioned, what is the context? which colleges at the university mention the library more frequently? are there opportunities for the library or librarians to provide research support or otherwise engage with course instructors and students?   methods   as of may 28, 2015, there were 8,433 total syllabi in the collection dating back to the 2011 pilot. for the purpose of this project, the syllabi from fall 2014 were chosen for examination due to multiple factors. first, the set of syllabi were cross-disciplinary and would provide data across all colleges and most subject areas on campus. second, the 1,258 syllabi in the fall 2014 set were relatively higher in total number when compared to other semesters. third, the analysis was started in the spring 2015 semester, and fall 2014 was the most recent set of syllabi available to analyze.   as the dspace software housing the collection was not managed in-house, it was not possible to simply download the collection metadata and files. we were, however, able to obtain a spreadsheet of the metadata for all documents uploaded to the syllabus collection prior to february 2015. a script was then written in the server-side scripting language (php) that visited the handle uniform resource identifier for each dspace record in the spreadsheet and downloaded every document in the collection containing the string “2014.” during the download process, the collection name was added to the start of the real document name, meaning each document in a “2014 fall” collection could be easily identified and added to the pool of documents to be analyzed.   after obtaining the fall 2014 syllabi set of documents, appropriate text-mining software had to be identified. the software had to support batch ingestion of large amounts of pdf and word documents, have the ability to search across the entire contents of each document, and provide the ability to tag the discovered content with keyword codes. ultimately, qda miner was chosen for this project due to its ability to support qualitative data analysis through coding, annotation, and retrieval of the large syllabus collection. it is important to note two key aspects of using this software: 1) the software is only compatible with the windows operating system, and 2) when importing word documents, the text formatting was thrown off and unicode characters were added to some of the text content. to counteract this, all documents were converted to pdfs.   after importing the pdfs, metadata was applied to each document. this metadata included the associated college, subject, and course level represented in each syllabus. next, the librarians brainstormed a list of keywords during multiple meetings to use when searching across the syllabi. these keywords were related to either the library and its services or spaces or the courses’ research assignments. keywords related to the library and its services or spaces were used to identify if or how the teaching faculty referenced the library as well as what services or spaces were promoted. keywords related to the courses’ research assignments were used in order to identify opportunities for subject librarians to promote the library’s research services. similar keywords were grouped together to form codes. the codes include library spaces, library services, it services, librarian-led instruction, independent instruction, resources, people, campus space in the library, citation management, and research. table 1 shows the keywords and their corresponding code category.     table 1 codes categories and keywords code categories keywords library spaces library classroom, student computing center, media center, reference, special collections, sdsu library, love library, library library services reference help, circulation/course reserves, exam space, interlibrary loan it services computers, software, technical assistance, email, blackboard librarian-led instruction library session independent instruction self-guided tour, plagiarism resources databases, media collection, pin, research guide, ebook, book, article/journal, syllabus collection, microform people name, librarian, general campus space in the library writing center, financial lab, tutoring/math center citation management apa, mla, chicago style, bibliography research research paper, literature review, capstone, senior project, thesis, literature search, data management     figure 1 the digital syllabus collection hosts a total of 8,433 syllabi, with 1258 syllabi from the fall 2014 semester—approximately 15% of the total collection.     results & analysis   analysis of sample set   twelve hundred and fifty-eight syllabi from fall 2014 courses were ingested into qda miner for analysis out of a total of the nearly 8,500 syllabi in the entire collection. thirty-two were unable to be labeled and coded due to missing text and poor conversion by the software. the final corpus size of 1,226 syllabi represents approximately 17% of the total planned classes for sdsu during fall 2014, as outlined by the 2014–2015 course catalog.   seventy-one of 96 campus subjects were represented in the corpus, along with seven colleges and the division of undergraduate studies. the colleges are represented by their short codes as follows: college of arts and letters (cal), college of business administration (cba), college of health and human services (chhs), college of education (coe), college of engineering  (eng), college of professional studies and fine arts (psfa), college of science (sci), and the division of undergraduate studies (oth). figure 2 depicts the relative prevalence of syllabi from each college in the sample. cal provided the most syllabi, with 520, while eng provided the fewest with only 26.     figure 2 relative number of syllabi from each college in the corpus and the total number of syllabi from each college: cal = 520, cba = 156, chhs = 58, coe = 124, eng = 26, psfa = 234, sci = 73, oth = 35.     table 2 scheduled classes a, number of syllabi available, and percentage of scheduled classes represented for each college in fall 2014 anumbers include all sections of courses.     relative to the number of planned classes for the fall 2014 semester, cba provided the highest percentage of syllabi (44%) while sci provided the lowest percentage (4%). table 2 compares the number of scheduled classes, the number of syllabi, and the percentage represented in the corpus from each of the eight colleges. the corpus contains syllabi from 77 unique subjects. rhetoric and writing (rws), history (hist), and english (engl) were the top subject contributors of syllabi, with 84, 77, and 62 respectively. fifty-five percent of subjects had fewer than 10 syllabi in the sample, with 14% of subjects having only one syllabus each.   codes & keywords results   of the 1,226 syllabi in the corpus, more than half did not mention any library spaces, services, or resources, nor did they mention any papers or projects requiring research. the following results are based on the remaining 561 syllabi. the least frequently used keyword codes included the following: senior project and math/tutoring center had no mentions, and the keywords blackboard, syllabus collection, librarian title, tour, microform, data management, wells fargo, interlibrary loan, and literature search had fewer than five mentions each. the most popular keyword codes overall were research paper (173), apa (125), and mla (123), as depicted in figure 3. after the keywords were condensed into 10 codes, the three most frequent codes in the syllabi were citation management (286), research (262), and library spaces (251). figure 4 depicts the three most frequently used keyword codes mapped to each of the top three codes.     figure 3 number of occurrences in the corpus of the three most popular keywords.     figure 4 frequency of individual keyword codes from the top three code categories of the corpus.     figure 5 frequency of code occurrence in the corpus and likelihood of co-occurrence with other codes in the same syllabus.     relationships between codes were mapped using sorensen’s coefficient of similarity. library space and library resources (coefficient =.500) and library space and library services (coefficient-=.457) are most likely to appear in the same syllabus, with citation management and resource intensive projects (coefficient=.445) next likely to co-occur. these two clusters are somewhat related to each other, as they all have loose ties to library space, but the codes of librarians, librarian-led instruction, and self-guided instruction have almost no co-occurrence frequency with research, citation management, or library services. figure 5 shows a 2d representation of code frequency and strength of co-occurrence with other codes. line thickness indicates the strength of sorensen’s coefficient.   syllabi from history were the only subgroup to have mentioned keywords representing all 10 codes. general studies, english, management information systems, child & family development, and sociology all used keywords mapping to 90% of the codes. at the college level, cal and cba mapped to 100% of the codes, while eng was the only college to map to less than 90% of the codes. table 3 shows the number of code mentions from the syllabi of each college.     table 3 number of category codesa represented in syllabi of each college acitation management and library spaces are the two most used codes across all disciplines, followed by library resources.     figure 6 percentage of syllabi, out of the 1,226 syllabi sample, mentioning instruction codes versus a resource-intensive project.     of our corpus of syllabi, only 38 mentioned librarian-led instruction and 18 of these syllabi were from rhetoric and writing, which is a core curriculum course. in contrast, there were almost twice as many (67) syllabi mentioning independent instruction, typically from requirements to complete the library’s plagiarism tutorial or interactive tour. eight percent of syllabi mention any type of library instruction, while 21% mention some sort of research. figure 6 highlights the 18% gap between mentions of research and librarian-led instruction sections, and the 16% gap between independent instruction and research.   study limitations   while the syllabus collection study helped to uncover broad patterns and opportunities for library interventions, there were a number of limitations. first, the sample chosen for this study was syllabi uploaded during the fall 2014 semester. a more accurate picture of the library’s presence in the syllabi would likely be revealed if the librarians analyzed the entire collection of syllabi from the last 5 years, rather than focusing on one semester. second, there is not complete course coverage within each subject area of the syllabus collection. even though the vast majority of subjects are represented within the collection, only certain courses within each subject area actually appear within the collection. in order to have a better understanding of the subject areas and possible library interventions, the library would need to reach out to departments to ensure that there is a syllabus on file for each course taught within a subject area. third, a full content analysis was not performed on the syllabi. the syllabi were searched for specific words and phrases, and the results were contextualized by viewing the sentences surrounding the search hits. more context for how the library is mentioned in the syllabi could be discovered if a full content analysis was performed.   discussion   a collection of syllabi can provide access to vast amounts of data about a university’s community. mining this data can provide libraries with much-needed information about their communities and inspire new methods of outreach and engagement. the information gleaned from syllabi can have an impact on a library’s collections, service points, instructional activities, spaces, and technologies. in the case of sdsu, the initial syllabus collection investigation has revealed multiple opportunities for the library to intervene. of the over 1,200 syllabi examined, only 38 included information about a librarian. additionally, over 250 syllabi included requirements for research or intensive resource use. there is clearly a mismatch between the number of courses requiring research and those that mention librarians. librarians at sdsu can capitalize on these findings to offer research and information literacy instruction support.   from a subject or department standpoint, there is much to be gained. this study revealed that many history syllabi refer to the library, yet subject support from the library consists of several librarians serving niche areas within the department. this finding led to recommendations that subject coverage be provided in a more organized manner, which resulted in establishing a coordinator who works with all librarians providing support for history. moving forward, individual subject librarians have planned syllabi-analysis projects based on this study in order to uncover specific needs within the schools, departments, and colleges they support. this will allow for a more targeted approach to engaging library users with relevant resources and services. it will also give subject librarians the data they need to develop and improve their services.   conclusion   in this study, syllabi were analyzed from the entire university, across most levels and departments. the results revealed major differences across academic disciplines with regards to if or how the library is mentioned in syllabi. despite its limitations, this study does demonstrate how academic librarians can perform a text-mining analysis of syllabi to shed light on the information needs of their campus communities. it also revealed gaps where the library could intervene and provide support, especially in the area of research support. key areas of outreach for liaison librarians were identified, particularly in history and writing courses. additionally, student research expectations were further illuminated across disciplines. it is no surprise that research is different from one discipline to the next, but this study sheds some light on the research expectations faculty have for the students in different disciplines.   while there are many examples of librarians evaluating syllabi collected from the web or directly from instructors, programs, and colleges; this study was unique in utilizing syllabi from a central campus repository and leveraging text-mining software. a central repository of syllabi decreases the time and effort required for collection and access, while qda miner significantly reduces the burden of hand coding text documents. we conclude that our research has produced a replicable method for text mining digital syllabi, whether they are in a central repository or individually collected, and for identifying areas for improved services to faculty and students that other libraries could use to their advantage.   references   boss, k. & drabinski, e. (2014). evidence-based instruction integration: a syllabus analysis project. reference services review, 42(2), 263-276. http://dx.doi.org/10.1108/rsr-07-2013-0038   dewald, n. (2003). anticipating library use by business students: the uses of a syllabus study. research strategies, 19, 33-45. http://dx.doi.org/10.1016/j.resstr.2003.09.003   o’hanlon, n. (2007). information literacy in the university curriculum: challenges for outcomes assessment. libraries and the academy, 7(2), 169-189. http://dx.doi.org/10.1353/pla.2007.0021   rambler, l.k. (1982). syllabus study: key to a responsive academic library. the journal of academic librarianship, 8(3), 155-159. retrieved from https://eric.ed.gov/?id=ej264858   smith, c., doversberger, l., jones, s., parker, j., pietraszewski, b. (2012). using course syllabi to uncover opportunities for curriculum-integrated instruction. reference & user services quarterly, 51(3), 263-271. http://dx.doi.org/10.5860/rusq.51n3.263   vanscoy, a., & oakleaf, m. (2008). evidence vs. anecdote: using syllabi to plan curriculum-integrated information literacy instruction. college & research libraries, 69(6), 566-575. http://dx.doi.org/10.5860/crl.69.6.566   williams, l. m., cody, s. a., & parnell, j. (2004). prospecting for new collaborations: mining syllabi for library service opportunities. the journal of academic librarianship, 30(4), 270-275. http://dx.doi.org/10.1016/j.acalib.2004.04.009   evidence summary   link established between lbgt-friendly campus climate index scores and web-based resources of academic libraries   a review of: ciszek, m. p. (2011). out on the web: the relationship between campus climate and glbt-related web-based resources in academic libraries. the journal of academic librarianship, 37(5), 430-436. doi: 10.1016/j.acalib.2011.06.007   reviewed by: kathleen reed assessment & data librarian vancouver island university nanaimo, british columbia, canada email: kathleen.reed@viu.ca   received: 30 nov. 2012 accepted: 22 jan. 2013      2013 reed. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to explore whether academic institutions that score highly on the lgbt-friendly campus climate index also have well-developed web-based library resources to support glbt (gay, lesbian, bisexual, transgender) students.    design – website analysis, percentage, and binary logistic regression analysis.   setting – library websites of colleges and universities in four american geographic regions: northeast, midwest, south, and west.   subjects – there were 259 colleges and universities that participated in the 2010 lgbt-friendly campus climate index.   methods – the author visited the library websites of all institutions and surveyed available glbt-related resources.  the criteria for online resources included: 1) a research guide for glbt studies or geared toward glbt students, 2) an individual identified within the research guide as a contact for glbt-related resources, and 3) a subscription to ebsco’s glbt life database.   whether or not the academic libraries had the above resources was then analyzed with each institution’s score on the climate survey scale.  the author controlled for geographical location, religious affiliation, and campus setting of the college or university.     main results – there is a positive direct relationship between whether a library makes glbt resources available on the web and campus climate. however, only 25% of libraries surveyed published a research guide, 18% named a contact individual, and 31% subscribed to glbt life.   conclusion – while parent institutions commit to glbt students by taking the lbgt-friendly campus climate index survey, academic libraries lag behind providing online resources for this community.    the author recommends academic libraries:   create a top-level glbt research guide. provide contact information for a staff person assigned to provide glbt-related research assistance. assign a resource selector for glbt-related resources. subscribe to glbt-related databases. partner with glbt organizations on campus to improve collections. promote glbt-related collections to the campus community. perform an assessment of the information and resource needs of glbt campus community members. ensure the glbt community is included in programming and services.     commentary    glbt-related research in the library and information arena has been ongoing for several decades, but tends to focus on public libraries and collections. ciszek’s article seeks to expand the collections focus into glbt resources made available via academic library websites.   there are several issues with this survey.  while the author mentions that lgbt-friendly campus climate index scoring occurs on the basis of 50+ questions that each institution uses internally to assess the environment for lgbt students, he fails to adequately address how this self-assessment affects his study. without any holistic review of campus attitudes and perceptions of glbt campus community members, it is difficult to determine an institution’s actual climate. additionally, ciszek intertwines personal and academic glbt information seeking and does not address how they are connected, if at all. finally, as ciszek points out in his article, a significant issue is that the glbt life database – one of the three resources he uses to determine if a library is adequately contributing to creating a positive glbt campus climate – is often bundled with other ebsco products (p. 434). this database may also be financially out-of-reach for smaller institutions, regardless of how glbt-positive their climates may be.     academic libraries have an important role in making campuses glbt-friendly and this article has an excellent bibliography and recommendations for how libraries can work toward this goal. however, this reviewer was disappointed by methodological weaknesses and theoretical gaps. in the future, research on links between the existence of glbt online resources and campus climate should be based on more thorough climate studies, and account for the relationship between personal and academic information seeking in academic libraries.  ideally, future research would factor in not just existence of library resources as a sign of contributing to glbt-friendly campus climates, but use, quality, and perception among library users.   research article   chat transcripts in the context of the covid-19 pandemic: analysis of chats from the askaway consortia   barbara sobol public services librarian university of british columbia, okanagan kelowna, british columbia, canada email: barbara.sobol@ubc.ca aline goncalves information literacy/reference librarian yukon university library whitehorse, yukon, canada email: agoncalves@yukonu.ca mathew vis-dunbar data and digital scholarship librarian university of british columbia, okanagan kelowna, british columbia, canada email: mathew.vis-dunbar@ubc.ca  sajni lacey learning and curriculum support librarian university of british columbia, okanagan kelowna, british columbia, canada email: sajni.lacey@ubc.ca    shannon moist head of reference services douglas college new westminster, british columbia, canada email: moists@douglascollege.ca  leanna jantzi head, fraser library simon fraser university surrey, british columbia, canada email: leanna_jantzi@sfu.ca aditi gupta engineering and science librarian  university of victoria libraries victoria, british columbia, canada email: aditig@uvic.ca    jessica mussell distance learning and research librarian university of victoria libraries victoria, british columbia, canada email: jmussell@uvic.ca    patricia l. foster public services/askaway coordinator university of british columbia, point grey campus vancouver, british columbia, canada email: patricia.foster@ubc.ca    kathleen james instruction librarian mount royal university calgary, alberta, canada email: kjames@mtoroyal.ca    received: 12 dec. 2022                                                                   accepted: 13 mar. 2023      2023 sobol, goncalves, vis-dunbar, lacey, moist, jantzi, gupta, mussell, foster, and james. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   data availability: sobol, b., goncalves, a., vis-dunbar, m., lacey, s., moist, s., jantzi, l., gupta a., mussell, j., foster, p., & james, k. (2023). data: chat transcripts in the context of the covid-19 pandemic: analysis of chats from the askaway consortia. open science framework, v1. https://doi.org/10.17605/osf.io/feswq   doi: 10.18438/eblip30291     abstract   objective – during the covid-19 pandemic, the majority of post-secondary institutions in british columbia remained closed for a prolonged period, and volume on the provincial consortia chat service, askaway, increased significantly. this study was designed to evaluate the content of askaway transcripts for the 2019-2020 and 2020-2021 academic years to determine if the content of questions varied during the pandemic.   methods – the following programs were used to evaluate the dataset of more than 70,000 transcripts: r, python (pandas), voyant tools and linguistic inquiry and word count (liwc).   results – our findings indicate that the content of questions remained largely unchanged despite the covid-19 pandemic and the related increase in volume of questions on the askaway chat service.   conclusion – these findings suggest that the academic libraries covered by this study were well-poised to provide continued support of patrons through the askaway chat service, despite an unprecedented closure of physical libraries, a significant increase in chat volume, and a time of global uncertainty.     context   this study was designed to evaluate the content of askaway transcripts for the 2019-2020 and 2020-2021 academic years in response to this question: did the types of questions, or the substance of those questions, change during the covid-19 pandemic? our hypothesis was that the questions would differ when compared with the pre-pandemic period. we were curious about what could be learned about academic library patron needs during this time, based on changes in language usage and the types of questions asked. what could we ascertain about reference needs in this time period, and how could that help us to prepare for future service disruptions? could anything be discovered about unique information needs during a pandemic?   chat reference in post-secondary libraries in british columbia, canada is provided through askaway, which is described as “a collaborative service that is supported and staffed by post-secondary libraries from bc and the yukon” and comprises 29 member libraries (bc eln, 2022a). the member libraries represent a diverse set of institutions, from private two-year colleges to large publicly funded research universities that span an enormous geographic area and represent both rural and urban settings (bc stats, 2018). askaway was established in 2006 and has since been an important part of library services at all participating institutions. when the pandemic was declared in march 2020 (world health organization, 2022), physical libraries were closed and chat reference was perceived by most libraries as the primary means of service provision. this shift is described by hervieux (2021) as moving chat reference from the margins of a service model to a “vital community service in a time of great uncertainty” (p. 267), a sentiment echoed by radford et al. (2021) who describe chat during the pandemic as a “premier essential user service” (p. 106).   the long-term closure of academic libraries in british columbia was unprecedented, with many libraries remaining closed for up to 18 months (bc eln, 2021). during the height of the pandemic, demand for askaway chat reference services increased by 62% over pre-pandemic years. post-secondary students also reported severe disruptions in their studies, finances and career plans (statistics canada, 2020). foreshadowing our findings, lapidus (2022) and watson (2022) nonetheless found that “for some institutions at least, the existing online reference infrastructure was capable of absorbing the demand during the pandemic” (watson, 2022, p.11).   in order to address the pandemic-specific questions that we had of the dataset, the standard practice of comparing academic years was deemed to be inadequate. as such, we created a timeline for analysis tied to key dates in the pandemic. september 2019 through march 2020 is the pre-pandemic timeline; the pandemic was declared on march 11, 2020, and bc post-secondary institutions closed on different dates throughout the month. april 2020 through december 2020 is the main pandemic period when institutions were fully online and library buildings were primarily closed. january 2021 through august 2021 represents a lessening of restrictions in bc and the reopening of libraries on different dates in advance of the 2021-2022 academic year; we refer to this period as late pandemic. the natural ebb and flow of the academic year is not reflected in the pandemic timelines and each time period includes regular term peaks and intersession breaks.   literature review   academic library chat reference analysis   virtual reference is defined as a “reference service initiated electronically for which patrons employ technology to communicate with public services staff without being physically present” (reference and user services association, 2017). academic library virtual reference services began in the 1990s and were in widespread use by the 2000s, allowing services to reach users at their point of need (francoeur, 2001; sloan, 1998). research has found that chat services have resulted in a decrease in library in-person visits as more people access web resources on home computers (francoeur, 2001; harlow, 2021). with physical space closures and public safety measures being implemented by libraries during the covid-19 pandemic, reliance on chat reference increased which resulted in a renewed urgency to examine this topic (de groote & scoulas, 2021; hervieux, 2021; kathuria, 2021). while there are many studies evaluating different aspects of chat reference, this literature review is focused on methodological approaches for unearthing meaning and evaluating language in academic library chat transcripts.   in an effort to discover and understand the complex behaviors, experiences, and interactions between virtual reference chat users and librarians, chat transcript analysis has moved beyond usage statistics and standardized question tagging to a more contextualized examination of transcripts using transcript-harvested data-based topic modeling, sentiment analysis, and visualizations (wang 2022, chen & wang, 2019; ozeran & martin, 2019). this developing analysis trend has technical limitations in its implementation and the lack of a standardization for evaluation (chen, 2019; grabarek & sobel, 2012; harlow, 2021; kathuria, 2021; ozeran & martin, 2019). grabarek and sobel (2012) highlight the challenges of anonymous data in evaluating social and emotional meaning. further exploration with larger datasets and chat transcripts over longer and various date ranges for comparison may elucidate more areas of interest, and visualization tools will be helpful for analysis (chen & wang, 2019; ozeran & martin, 2019). sharma, barrett and stapelfeldt (2022) utilize a python library and tableau for visualization, demonstrating the utility of mixed method analysis. walker and coleman (2021) explore machine learning as a method for examining the complexity of chats with a large dataset.   the use of coding methods is heavily utilized in chat transcript analysis to examine meaning and satisfaction, yet large datasets often make this impractical without relying upon sampling. schiller (2016) used the cultural-historical activity theory framework to conduct their analysis, generating a codebook and a cluster analysis to determine relationships. logan, barrett, and pagotto (2019) used spss to code chat user satisfaction based on transcripts and exit surveys. harlow (2021) coded nursing chat transcripts using atlas.ti to evaluate reference efficacy. logan and barrett (2018) coded a sample of chats to evaluate the relationship between provider communication style and patron willingness to return; chi-square tests were used to assess this relationship. kathuria (2021) utilized a two-part method of grounded theory tagging followed by sentiment analysis using r to evaluate positive and negative sentiments. grounded theory has been used in many studies as part of a mixed methods analysis to examine meaning (harlow, 2021; mungin, 2017; smith et al., 2016).   chat reference in libraries during covid-19   a number of studies which examine chat services and transcripts in the context of the covid-19 pandemic have already been published, and many found an increase in chat volume when academic libraries, along with their institutions, closed their doors and shifted to remote instruction and services (de groote & scoulas, 2021; hervieux, 2021; kathuria, 2021; lapidus, 2022; radford et al., 2021). when comparing chat transcripts between fall 2019 and fall 2020, hervieux (2021) found that while “percentages of each type of interaction were fairly similar…with known items, circulation and reference queries making up the majority of the questions asked,” there was a “substantial difference” in questions about branch library information due to covid-protocols and procedures applied to study spaces (p. 275-6).  kathuria (2021) found that “questions about accessing and returning the physical collection grew the most during covid” (p. 112). other questions that increased included those regarding fines and fees, library hours and technical troubleshooting (kathuria, 2021). an increase in questions regarding course and assignment support and assignments has also been noted (hervieux, 2021; kathuria, 2021). alternatively, watson (2022) compared the university of mississippi libraries’ pandemic chat data to a pre-pandemic period and found no increase in chats and no significant difference in word frequency in chat transcripts. graewingholt et al., (2022) argue that review of chat transcripts, regardless of the pandemic context, can further support revisions, adjustments, and improvements to library services.   multiple methods of analysis for examining chat reference during the covid-19 pandemic have been utilized. hervieux (2021) used qualitative coding and quantitative metadata analysis to examine both the complexity and duration of chats. hervieux (2021) concluded that more questions were being asked, more downtime during a chat was occurring, and that “librarians and patrons use more relational cues during their interactions” (p. 277). kathuria (2021) used a grounded theory of analysis relying on coding and sentiment analysis in r and found an increase in negative sentiment when comparing pre-pandemic and pandemic chats. radford, costello & montague (2021) relied on surveys and interviews to inform their examination of patron chat behavior and service perceptions. degroote & scoulas (2021) also used patron surveys and paired this with statistical analysis to examine library use patterns during covid-19. lapidus (2022) conducted statistical analysis of metadata to understand reference services overall, including chat. watson (2022) analyzed metadata and word frequency utilizing nvivo and voyant in a multi-method approach not dissimilar from that reported in this study. finally, graewingholt et. al. (2022) started with machine classification followed by manual coding to understand trends in questions and inform training. consensus on how to best evaluate large chat datasets has not yet emerged within the literature.   methods and tools   data acquisition   askaway chat data for this study covering september 1, 2019 to august 31, 2021 were obtained from software vendor libraryh3lp. names, email addresses, student numbers, and other information that could lead to patron or service provider identification were removed by libraryh3lp in accordance with the bc eln privacy policy. in addition, four categories of chats were removed prior to data acquisition: chats from a university that withdrew from the service early in the timeline being studied, chats where the privacy script was employed by service providers, practice chats, and chats fewer than five seconds in length.   based on these criteria, libraryh3lp provided two datasets: askaway transcripts, containing chats between patron and provider with each chat as an individual text file for a total of 70,728 chats; and askaway metadata, containing chat metadata, including start date, start time, queue, duration, and tags. tags are standardized categories applied to chats by service providers (askaway, 2022). this second data set consisted of 73,483 rows, one row for each chat, suggesting 2,754 additional records than were included in the transcript data set; this discrepancy is discussed below.   data description   askaway transcripts are composed of four elements: a header with metadata, system text (like a welcome message), provider text, and patron text. these transcripts, like most chat data, are not well-structured. the informal nature of chat communication and the lack of standardization or error correction across chats (as an example, 'thank you', vs 'thank-you', vs 'thankyou', vs 'thnkyou'), present challenges in derived analyses. these challenges are further exacerbated by a variety of other features of chat data: the use of shorthand, such as emoticons and acronyms; a need to be expressive in a text environment, resulting in things like excessive punctuation; fast typing resulting in misspellings and excess white space; and content pasted from other sources introducing a variety of printed and non-printed characters. these features of the data set make even basic descriptive statistics, such as word counts, challenging. conceptually, we can see this when comparing the terms 'meta data' and 'metadata', counted as two words and one word, respectively.   askaway metadata is highly structured data and consists of two categories: system-generated and provider-generated. system-generated metadata include variables such as time stamps, institution, and duration. provider-generated metadata consists of tags, of which there are thirty available (askaway, 2022). service providers select those most appropriate to the chat to represent the interaction; multiple tags can be selected and there is no free-text option. in march 2020, askaway advised service providers to apply the tag “other” to covid-19 chats and in june 2020, askaway introduced a new covid-19 tag to indicate if a question was specifically related to an aspect of the pandemic.   libraryh3lp was unable to provide an explanation for the discrepancy between number of transcripts provided and number of chats suggested by the metadata dataset. to investigate this further, metadata was extracted from the transcripts and compared against the metadata dataset. while a definitive conclusion could not be derived, noted anomalies such as the duration time stamps occasionally being off by a second, suggest minor errors in the collection of data attributed to this inconsistency. representing just under 4% of the transcript data, this discrepancy was considered manageable for the purposes of this study.   we added a field to the metadata to classify participating institutions by the size of their student body using the value of full-time equivalent (fte). fte numbers were derived from bc eln (2022b), and divide post-secondary institutions into three categories, as shown in table 1.     table 1 askaway post-secondary institutions by fte institution size fte count number of institutions small 4,999 or less 18 medium 5,000-9,999 9 large 10,000 or more 4     data cleaning & preparation for analysis   askaway transcript data was iteratively cleaned and organized using r. the text chats were initially merged into a single data set, and all system-generated text and metadata stripped from the main corpus; metadata remained associated with the text and allowed for subsequent subsetting of the data. several subsets of the data were produced, including patron-only transcripts, provider-only transcripts, and time series transcripts. as part of this process, general cleaning included removal of excess white space, conversion to lowercase, and stripping of punctuation. single text files of each subset were then produced for analysis.   the askaway metadata dataset arrived clean with no post-processing needed, other than including additional data about the fte category. to prepare the data for use in liwc, a small modification to the original datasets was made: replacing the strings ‘https://’ and ‘http://’ with the string ‘url,’ as the element  ‘:/’ was being interpreted as an emoticon and skewing the scores for the tone dimension analysis.   selection of text analysis tools   with a dataset of 70,728 items, the research team used multiple approaches in an attempt to find patterns and meaning. uncertain of which tools and techniques would be the most appropriate for examining meaning in large bodies of chat text, we first ran a random sample of the liwc, askaway transcript (n = 3,800) through a series of text analysis applications. this allowed us to assess the general format and contents of the dataset and to identify the most appropriate tools to address our research questions. the test analysis was run in the following programs: r, atlas.ti, nvivo, python (pandas), voyant, liwc, and openrefine. based on functionality and researcher expertise, r and python (pandas) were selected for metadata and quantitative chat analysis. liwc and voyant were selected as tools to explore meaning in the chat transcripts.   voyant tools   voyant tools is a suite of web-based textual analysis tools used in digital humanities (sinclair & rockwell, 2022). voyant allows for text files to be explored and visually represented in an easy to manipulate interface; it is particularly useful for examining patterns within texts. of particular relevance to this analysis were the cirrus, document terms, terms berry, and trends tools; these were used to determine patterns and meaning within each dataset and between them.   linguistic inquiry and word count (liwc)   liwc is a software program focused on identifying people’s social and psychological states from the language they use. liwc achieves this by calculating word count distributions in psychologically meaningful categories (tausczik & pennebaker, 2010). the current version, liwc-22, uses over 100 built-in dictionaries consisting of words, word stems, emoticons, and other verbal structures to capture several psychological categories (liwc, n.d.). since our objective was to assess if there was a difference between the types of inquiries received by askaway throughout the pandemic, and not to evaluate linguistic characteristics per se, we focused on the summary dimensions, described in table 2.     table 2 liwc summary dimensions summary dimensions description analytic captures the use of formal, logical, and hierarchical thinking patterns. low scores correspond to intuitive, personal, and less rigid language. high scores suggest more formal or academic language correlated with higher grades and reasoning skills. clout refers to language related to social status, confidence, or leadership. authenticity describes self-monitoring language, associated with levels of spontaneity. higher scores in this category mean use of less “filtered” language, while prepared texts tend to have lower scores. tone puts positive and negative emotional tones into a single summary variable. the higher the number, the more positive the language, while scores below 50 suggest a more negative tone.      analysis   metadata analysis   using both r and python (pandas), the metadata for the full dataset was analyzed for insights into question types, distributions and trends. in general, the data indicate changes in volume more so than any other element. for example, the pandemic was officially declared on march 11, 2020 and march 23, 2020 was the busiest single day on askaway throughout the study period. figure 1 below depicts the distribution of questions asked on askaway between september 2019 and august 2021, totaled by month, and colour-coded to represent the pandemic timeline.     figure 1 total chats by month and pandemic timeline.     the askaway service supports a variety of post-secondary institutions that differ in size, program type, geography, and urban/rural locality. figure 2 depicts the distribution of questions asked over time by the size of the post-secondary institution by enrollment. while the pattern in the data follows the same trajectory as is depicted in figure 1, when the data is broken out by institution size, we see that medium-sized institutions consistently account for a high proportion of questions asked. we also see that, after the pandemic was declared, the number of questions asked from the larger institutions accounts for a substantial portion of the increase in volume. despite these increases in volume from larger institutions, which include the research-intensive universities, there is no evidence in our data that the question types themselves were altered.     figure 2 timeline of total chats by institution size and pandemic timeline.     askaway tags are well-defined for collective usage (askaway, 2022) and analysis of them shows very clearly that, from the service provider’s perspective, the type of questions asked on askaway did not change substantially during the pandemic. figure 3 demonstrates the remarkable consistency in the types of questions asked throughout our study period through a display of the top 10 tags in our dataset, isolated by pandemic period. note that there are 12 tags in figure 3. the top 10 tags for each period were extracted and then collated; not all tags below would have been in the top 10 for the dataset as a whole. the top 7 tags in figure 3 were in the top 10 for all 3 periods, as was the tag 'technical'. the remaining 4 tags were in the top 10 for only 1 or 2 periods (no question in 'pre', covid-19 in 'early', interlibrary loan in 'pre' and 'main', and referred in 'early' and 'main').   a few other aspects of the tags warrant highlighting. circulation, referrals to “home library,” and directional questions all increased from april 2020 onward. interlibrary loan (ill) questions are absent in the main pandemic dataset as ill services were unavailable globally for much of that time. the presence of general referrals directing patrons to services outside of the library only in the mainand late-pandemic data suggests the important role that libraries played as a general campus service. the covid-19 tag emerges in the top 10 tags during the main pandemic period but does not persist beyond december 2020. despite these differences, we see remarkable consistency especially in those that account for the majority of the volume of activity.     figure 3 top ten chat tags displayed by pandemic timeline.     transcript analysis   table 3 details the transcripts in the pandemic timeline which form the basis of our subsequent analysis.     table 3 pandemic timeline datasets timeline transcript datasets chats per dataset pre-pandemic: september 2019 march 2020 all chats (patron & provider) patron only provider only 18,968 main-pandemic: april 2020 through december 2020 all chats (patron & provider) patron only provider only 28,440 late-pandemic: january 2021 august 2021 all chats (patron & provider) patron only provider only 23,320     table 4 voyant tools analysis of patron chat   pre-pandemic main-pandemic late-pandemic average words per sentence 39.5 37.8 36.1 vocabulary density 0.030 0.024 0.027 readability index 9.603 9.406 9.493 total words 1, 683, 539 2, 959,334 2, 341, 374 unique words 50, 039 71, 373 62, 735     table 4 displays the summary report for the patron chat for each time period in voyant tools. these are values that voyant tools applies as a default to all analyses: average words per sentence, vocabulary density, readability, total words and unique words (sinclair & rockwell, 2022). the readability index is a calculation based on the breakiterator class, which is a natural-language coding technique to determine word boundaries and syntax in text (oracle, 2022). while these values are not inherently insightful, when examined in comparison across the time periods they demonstrate remarkable similarity on each metric, reinforcing our overall findings.   word frequency was used to explore patron voice in the transcripts across each pandemic time period. there are a total of 214,418 unique words in all of the patron transcripts. using r, we extracted the 1,000 most frequently used words, consisting of 4 or more characters, from each time period. there are 1,135 unique terms in total that meet these criteria, of which 871 are in all 3 time periods, and 264 are in only 1 or 2 of the time periods. figures 4-6 present word clouds of the top 115 words from patron-only transcripts with the size of the word mapped to its frequency.     figure 4 pre-pandemic patron chat.     figure 5 main-pandemic patron chat.     figure 6 late-pandemic patron chat.     liwc was employed to identify general trends in the sentiments expressed in our dataset, an approach similar to kathuria’s (2021) sentiment analysis. the liwc dimensions of analytic, clout, authentic, and tone were used to evaluate potential changes in both patron and provider transcripts. linguistic scores between patron and provider were compared to evaluate similarities or differences in language used. table 5 presents the scores and change percentages for the liwc dimensions for both patron and provider transcripts.     table 5 liwc dimensions analysis by pandemic timeline   word count analytic % change clout % change authentic % change tone % change provider chats pre 3,633,999 52.66 n/a 81.13 n/a 28.69 n/a 80.78 n/a main 6,149,448 52.62 -0.07 79.2 -2.37 27.03 -5.78 76.43 -5.38 late 4,997,628 54.51 3.60 80.24 1.31 25.99 -3.85 77.37 1.22 patron chats pre 1,682,967 39.28 n/a 15.2 n/a 55.89 n/a 89.32 n/a main 2,959,063 37.92 -3.46 15.33 0.85 56.72 1.48 89.52 0.22 late 2,340,394 38.7 2.05 15.72 2.54 56.71 -0.01 91.3 1.99   analytic scores were nearly identical for providers pre-pandemic and in the beginning of the pandemic, indicating similar levels of formality in language. an increase of 3.59% was observed in the late phase of the pandemic. overall, analytic scores remained relatively similar for both providers and patrons, with small changes in between periods (less than 4% change for both user groups). when comparing scores between patron and provider chats, the analytic score was considerably higher for providers, indicating the prevalent use of formal language by askaway librarians.   clout was consistently high for provider responses, with a 2.38% decrease in the main pandemic stage, and a return to near pre-pandemic scores in the late pandemic stage. for patron chats, even though a slight increase in clout occurred over time, the levels remained low throughout all phases. these numbers indicate that the language used by providers translates to higher status, confidence, or leadership when compared to patron chats and that rates did not change substantially during the pandemic.   levels of authenticity were consistently low for provider chats in relation to patron chats, and these levels decreased as the pandemic progressed. authenticity was high for patron chats, with levels consistently above 50% and a slight increase during the pandemic. for provider chats, a more pronounced decrease in authenticity occurred.   both providers and patrons had positive emotional tone (scores consistently higher than 50), but patrons had higher positive language than providers both before and during the pandemic. an inverse trend was observed: while positive language in provider chats declined during the pandemic (5.78% decline from pre-pandemic levels and 3,84% decline between pandemic phases), scores for patron chats had a small increase (less than 3% between pre-pandemic and pandemic levels).   discussion   this study aimed to evaluate the type and substance of chat reference questions in an effort to understand a vital aspect of academic library services during the covid-19 pandemic. researcher expectations were that the type of questions would differ, in large part because the experience of providing only virtual services seemed so different. however, similar to hervieux (2021), our expectations are not supported by the data. by examining askaway transcripts and metadata, we found homogenous results which demonstrated more consistency than difference in the types of questions asked.   the use of voyant tools to explore patterns and interpret meaning in the transcripts focused on the patron transcripts. when examining word frequency over the pandemic timeline, the numbers indicate remarkable consistency in the words used by patrons over time. despite all of the social and economic impacts of the pandemic, the shift to online-only classes, and the closures of our physical libraries, this snapshot captured in the word cloud figures depict the overwhelming use of askaway for library-specific questions that focus on research involving citation and locating and accessing sources of information. analysis using the terms berry feature duplicates the cirrus results, and the trends tool did not prove insightful with the transcripts.   liwc shows a slight increase in analytic scores in the late pandemic which might be correlated with increased use of pre-scripted language as a provider strategy to deal with the increased volume on askaway. this would also explain declining rates in the authenticity dimension as pre-scripted language is considered less authentic in liwc. though it is difficult to confirm exactly why this is the case without qualitatively examining more closely the interactions between providers and patrons, it is possible that certain patterns in provider messages and scripts, such as higher use of articles, can indicate higher analytic thinking and formality (jordan et al., 2019). analytic scores of patron responses had more fluctuation between phases than those of providers, however, similar scores in the pre and late periods suggest that the main pandemic period may have in fact been a bit of an anomaly.   as observed by kacewicz et al. (2014), the use of first-person plural pronouns and an "outer-focus" language correspond with higher scores in the liwc clout dimension, and this might serve as a potential explanation for the large discrepancy between provider and patron chats. for example, several askaway scripts use the pronoun "we" as part of their composition, particularly the script used at the end of a chat, so the use of plural personal pronouns may be contributing to higher provider scores in this category, as opposed to other types of words that correlate to higher confidence or social status. regardless, levels of clout did not change considerably during the pandemic, with changes to percentages remaining lower than 3% for both patron and provider chats.   the decrease in the liwc authenticity dimension for providers in the main and late pandemic may again be associated with increased use of pre-scripted messages, which tend to be formulated using neutral language and with higher use of third-person pronouns. since high authenticity is correlated with use of firstand third-person singular pronouns (i, he, she), as described by kalichman & smyth (2021), the increase of pre-scripted language that does not match those characteristics can help explain the low scores for providers and the differences when compared to patron chats. we can infer from these numbers that patrons use more spontaneous language when compared to providers, and that levels of spontaneity for patrons have not changed substantially during the pandemic.   the difference in emotional tone scores between patron and provider chats may be explained by certain patterns in provider responses and in how some askaway pre-formatted scripts are written. for example, the word 'lost' is assigned to a negative emotional category in the liwc dictionary. it also happens to be part of a script used to check if patrons are still online after a period of inactivity (check in lost script). coincidentally, “lost” was the negative word that appeared most frequently in provider chats before and during the pandemic. similarly, “worries” also had high frequency in provider chats, but this word appears to be part of the expression “'no worries,” an alternative to “you’re welcome.” this suggests that emotional tone should be viewed with caution in this dataset, as individual words may not accurately represent the actual tone of a chat. the changes observed may be associated with the higher frequency of certain words due to increased number of chats, rather than a substantive change in emotional tone.   the main finding of consistent patron questions from april 2020 august 2021 has important implications for academic library service provision and future planning. first, consistency in the question types points to similar patron expectations for chat interactions, regardless of class format and library building operations. second, consistency in the question types, despite the large increase in volume, points to a need for flexible staffing responses in times of disruption or closure with sufficient training to respond to research, citation, and a broad scope of library service questions. third, our findings have implications for staff training and expectation management in times of disruption, whether planned or unexpected. while we know that patrons' lives were upended during the pandemic, what they expected of their academic libraries, at least as evidenced through chat interactions, did not change. future studies that compare patron expectations with patron behavior, in times of both normalcy and disruption, would further bolster this argument.   limitations   finally, as there are serious limitations to evaluating chat using quantitative methods alone, due to the fragmented nature of chat interactions, and because the volume of consortia chat does not easily lend itself to qualitative analysis, an improvement in the nuance of tags applied by providers would assist future assessment of the value of chat. as chat is poised to continue as an important element within the academic library service ecosystem, additional nuance in facilitating quantitative assessment of all reference services would be a welcome improvement.   conclusion   this article reports on the analysis of over 70,000 chat transcripts from a diverse set of post-secondary institutions across british columbia and the yukon and finds that, despite a significant increase in volume during the pandemic, question types were remarkably consistent with those asked prior to the pandemic. the professional literature has long advised that academic libraries devote more attention to virtual services (francoeur, 2001) but closing the physical operations of libraries during the pandemic significantly altered the urgency of this call (radford et al., 2021). de groote and scoulas (2021) utilized a multi-method approach to understand the impact of covid-19 on academic library use and found ongoing value for patrons in virtual service offerings. the insights offered in this paper lend confidence in articulating patron needs for chat reference as more than a supplemental service, but rather a cornerstone of service provision, during both stable and uncertain times. echoing the findings of mawhinney and hervieux (2022), this paper also provides support to the argument that the questions asked by chat patrons are complex, with the largest segment of our dataset tagged as research in focus.   author contributions   barbara sobol: conceptualization, formal analysis, methodology, project administration, visualization, writing original draft aline goncalves: formal analysis, methodology, visualization, writing original draft mathew vis-dunbar: methodology, visualization, writing – review & editing sajni lacey: literature review, writing review & editing shannon moist: writing review & editing leanna jantzi: writing – original draft aditi gupta: analysis, writing – review & editing jessica mussell: methodology, writing review & editing patricia l. foster: literature review, writing original draft kathleen james: literature review, writing review & editing. we would also like to acknowledge cristen polley at bc eln for facilitating access to the data.   references   askaway. (2022). tags. https://askaway.org/staff/tags   bc stats. (2018). college regions: detailed wall map. https://www2.gov.bc.ca/assets/gov/data/geographic/land-use/administrative-boundaries/college-regions/map_wall_college_regions_2018nov18.pdf   british columbia electronic library network. (2021). askaway actions and achievements 2021 [accessible version]. https://bceln.ca/sites/default/files/reports/aa_actions_achievements_2021_accessible.pdf   british columbia electronic library network. (2022a). participating institutions. https://bceln.ca/services/learning-support/askaway/participating-institutions    british columbia electronic library network. (2022b). partner libraries: fte information. https://www.bceln.ca/partner-libraries/fte   chen, x., & wang, h. (2019). automated chat transcript analysis using topic modeling for library reference services. proceedings of the association for information science and technology, 56(1), 368-371. https://doi.org/10.1002/pra2.31   de groote, s. and scoulas, j.m. (2021). impact of covid-19 on the use of the academic library. reference services review, 49(3/4), 281-301. https://doi.org/10.1108/rsr-07-2021-0043   francoeur, s. (2001). an analytical survey of chat reference services. reference services review, 29(3), 189-204. https://doi.org/10.1108/00907320110399547   grabarek roper, k., & sobel, k. (2012). anonymity versus perceived patron identity in virtual reference transcripts. public services quarterly, 8(4), 297-315. https://doi.org/10.1080/15228959.2012.730396   graewingholt, m., coslett, c., cornforth, j., palmquist, d., greene, c.r., & karkhoff, e. (2022). chatting into the void: scaling and assessing chat reference services for effectiveness. in m. chakraborty, s. harlow, & h. moorefield-lang (eds.), sustainable online library services and resources: learning from the pandemic (pp. 19-34). libraries unlimited.   harlow, s. (2021). beyond reference data: a qualitative analysis of nursing library chats to improve research health science services. evidence based library and information practice, 16(1), 46-59. https://doi.org/10.18438/eblip29828   hervieux, s. (2021). is the library open? how the pandemic has changed the provision of virtual reference services. reference services review, 49(3/4), 267–280. https://doi.org/10.1108/rsr-04-2021-0014   jordan, k. n., sterling, j., pennebaker, j. w., & boyd, r. l. (2019). examining long-term trends in politics and culture through language of political leaders and cultural institutions. proceedings of the national academy of sciences, 116(9), 3476–3481. https://doi.org/10.1073/pnas.1811987116   kacewicz, e., pennebaker, j. w., davis, m., jeon, m., & graesser, a. c. (2014). pronoun use reflects standings in social hierarchies. journal of language and social psychology, 33(2), 125–143. https://doi.org/10.1177/0261927x13502654     kalichman, s. c., & smyth, j. m. (2021). “and you don’t like, don’t like the way i talk”: authenticity in the language of bruce springsteen. psychology of aesthetics, creativity, and the arts. https://doi.org/10.1037/aca0000402   kathuria, s. (2021). library support in times of crisis: an analysis of chat transcripts during covid. internet reference services quarterly, 25(3), 107–119. https://doi.org/10.1080/10875301.2021.1960669   lapidus, m. (2022). reinventing virtual reference services during a period of crisis: decisions that help us move forward. medical reference services quarterly, 41(1), 41-53. https://doi.org/10.1080/02763869.2022.2021033   liwc. (n.d.). understanding liwc summary measures. https://www.liwc.app/help/liwc#summary-measures   logan, j. & barrett, k. (2018). how important is communication style in chat reference? internet reference services quarterly, 23(1-2), 41–57. https://doi.org/10.1080/10875301.2019.1628157   logan, j., barrett, k., & pagotto, s. (2019). dissatisfaction in chat reference users: a transcript analysis study. college & research libraries, 80(7), 925–944.   mawhinney, t., & hervieux, s. (2022). dissonance between perceptions and use of virtual reference methods. college & research libraries, 83(3), 503.   https://doi.org/10.5860/crl.83.3.503   mungin, m. (2017). stats don’t tell the whole story: using qualitative data analysis of chat reference transcripts to assess and improve services. journal of library & information services in distance learning, 11(1/2), 25–36. https://doi.org/10.1080/1533290x.2016.1223965   oracle. (2022). java documentation: about the breakiterator class. https://docs.oracle.com/javase/tutorial/i18n/text/about.html   ozeran, m., & martin, p. (2019). “good night, good day, good luck”: applying topic modeling to chat reference transcripts. information technology and libraries, 38(2), 59-67. https://doi.org/10.6017/ital.v38i2.10921   radford, m., costello, l., & montague, k. (2021). surging virtual reference services: covid-19 a game changer. college & research libraries news, 82(3), 106–113. https://doi.org/10.5860/crln.82.3.106   reference and user services association. (2017). guidelines for implementing and maintaining virtual reference services. https://www.ala.org/rusa/sites/ala.org.rusa/files/content/guidelinesvirtualreference_2017.pdf   sharma, a., barrett, k., & stapelfeldt, k. (2022). natural language processing for virtual reference analysis. evidence based library and information practice, 17(1), 78-93. https://doi.org/10.18438/eblip30014   schiller, s. z. (2016). chat for chat: mediated learning in online chat virtual reference service. computers in human behavior, 65, 651-665. https://doi.org/10.1016/j.chb.2016.06.053   sinclair, s. & rockwell, g. (2022). voyant tools (version 2.6.1). [text analysis app]. https://voyant-tools.org/   sloan, b. (1998). electronic reference services: some suggested guidelines. reference and user services quarterly, 38(1), 77-81.   smith, m., conte, j., & guss, s. (2016). understanding academic patrons’ data needs through virtual reference transcripts: preliminary findings from new york university libraries. iassist quarterly, 40(1), 20–26. https://doi.org/10.29173/iq624   statistics canada. (2020, may 12). impacts of the covid-19 pandemic on postsecondary students. the daily. https://www150.statcan.gc.ca/n1/daily-quotidien/200512/dq200512a-eng.htm   tausczik, y. r., & pennebaker, j. w. (2010). the psychological meaning of words: liwc and computerized text analysis methods. journal of language and social psychology, 29(1), 24–54. https://doi.org/10.1177/0261927x09351676   walker, j. & coleman, j. (2021). using machine learning to predict chat difficulty. college & research libraries 82(4), 683-707. https://doi.org/10.5860/crl.82.5.683   wang, y. (2022). using machine learning and natural language processing to analyze library chat reference transcripts. information technology and libraries, 41(3), 1-10. https://doi.org/10.6017/ital.v41i3.14967   watson, a. p. (2023). pandemic chat: a comparison of pandemic-era and pre-pandemic online chat questions at the university of mississippi libraries. internet reference services quarterly, 27(1), 25-36. https://doi.org/10.1080/10875301.2022.2117757   world health organization. (2023). coronavirus disease: covid-19 pandemic. https://www.who.int/europe/emergencies/situations/covid-19   research article   what do australian library and information professionals experience as evidence?   ann gillespie post doctoral research fellow queensland university of technology brisbane, queensland, australia email: anngillespie@outlook.com   faye miller research assistant university of southern queensland toowoomba, queensland, australia email: faye.miller@canberra.edu.au   helen partridge professor and pro-vice chancellor (scholarly and information and learning services) university of southern queensland toowoomba, queensland, australia email: helen.partridge@usq.edu.au   christine bruce professor queensland university of technology brisbane, queensland, australia email: c.bruce@qut.edu.au   alisa howlett coordinator, evidence based practice university of southern queensland toowoomba, queensland, australia email: alisa.howlett@usq.edu.au   received: 2 aug. 2016    accepted: 2 jan. 2017           2017 gillespie, miller, partridge, bruce, and howlett. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – this article presents the findings of a project which established an empirical basis for evidence based library and information practice (eblip). more specifically, the paper explores what library and information professionals experienced as evidence in the context of their professional practice.   methods – the project consisted of two sub-studies. the public library sub-study was conducted using ethnography. over a 5-month period, a member of the research team travelled to a regional public library on 15 occasions, staying between 3 and 4 days on each visit. the researcher observed, interacted, and became involved in the day-to-day activities of this library. these activities were recorded in a journal and added to the researcher’s insights and thoughts. additionally, 13 face-to-face interviews with staff in positions ranging from the operational to the executive were conducted. the academic sub-study was conducted using constructivist grounded theory. semi-structured interviews were conducted either in person or via skype, with 13 librarians from australian universities. interviewees were in a diverse array of roles, from liaison librarian to manager and library director.   results – the project found that the australian academic librarians and the public librarians who participated in the project experienced six elements as evidence: observation, feedback, professional colleagues, research literature, statistics, and intuition. each of these will be described and highlighted with examples from each of the two studies.   conclusions – the findings of this study revealed many similarities in the way that library professionals from both studies experienced evidence. evidence was not hierarchical, with evidence from many sources being valued equally. in contextualizing evidence and applying to the local environment, library professionals were able to draw upon more than one source of evidence and apply their professional knowledge and experiences. in this way evidence was more nuanced.     introduction   this article presents findings from a three-year project, which explored the ways in which australian lis professionals experience evidence based practice (ebp). two interconnected sub-studies provided an empirical basis for ebp in the context of the australian library and information profession: 1) academic librarians’ experience of ebp, using constructivist grounded theory methodology and 2) public librarians’ experience of ebp, using ethnographic methods. the two contrasting qualitative research approaches enabled the facilitation of deeper insights into how lis professionals can experience ebp and also what they experienced as evidence. the concept of “evidence” in the eblip context is seldom interrogated. research evidence does not always provide the necessary guidance to make decisions in professional practice, yet it takes “front and centre” position in eblip discourse. this article specifically focuses on comparing the findings on what was experienced as evidence across the two sub-studies to describe what constitutes these forms of evidence in the context of librarians’ professional practice. to contextualize what lis professionals experience as evidence, a review of the literature outlines the current state of research into the various sources of evidence used for evidence based library and information practice (eblip), followed by an overview of the two sub-studies’ methodological approaches and findings.   literature review   over the past 15 years, since an initial re-modeling of the decision-making framework from its medical origins, what constitutes as “evidence” in evidence based library and information practice (eblip) has been debated in the literature. from the first eblip framework proposed by eldredge (2000), “published research” has taken centre stage and often times continues to be assumed as the only type of evidence in eblip discourse (koufogiannakis, 2013, p. 8).   an early definition of eblip from booth (2002) builds on eldredge’s (2000) framework and identifies sources of evidence other than research to inform improvements to practice or “professional judgments” (2002, p 53).   evidence-based librarianship is an approach to information science that promotes the collection, interpretation and integration of valid, important and applicable user-reported, librarian observed, and research derived evidence. the best available evidence, moderated by user needs and preferences, is applied to improve the quality of professional judgments (booth, 2002, p. 53).   yet despite these additional sources of evidence included in booth’s (2002) definition, koufogiannakis (2011, p. 42) highlights the ongoing omission of “user-reported” and “librarian-observed” sources in the eblip literature. koufogiannakis (2011, p. 53) uses a practice based perspective to identify local information and professional knowledge as other evidence to consider in practice, forming a more “realistic view” of evidence. similarly, todd (2009, p. 89) categorizes research evidence as “evidence for practice,” one of three dimensions of evidence in a “holistic,” conceptual approach to looking at evidence used in professional practice. two other dimensions of evidence, evidence in practice and evidence of practice, are identified in todd’s (2009) model. user-reported evidence and results of evaluation programs are examples of “evidence of practice” (todd, 2009, p. 89).   this aligns with “local evidence” sources as described by koufogiannakis (2011, p. 50) to include usage data, feedback, and librarian observation. according to koufogiannakis (2011), local evidence is directly applicable as it is concerned with addressing the needs of the users of the library or information service. koufogiannakis (2011, p. 42, 44) argues for these additional types and sources of evidence to be considered equally with research evidence and says that they are not any less worthy, but simply different.   an understanding of “evidence” in eblip is evolving, both from acknowledgements of different types and sources of evidence in the literature, as well as findings from empirical studies seeking to identify evidence in organizational contexts (gillespie, 2014; koufogiannakis, 2012; partridge, edwards & thorpe (2010). research evidence is found to not be the only type of evidence to inform practice (koufogiannakis, 2012, p.18). koufogiannakis (2012, p. 10) grouped sources of evidence used by academic librarians into two types – hard and soft evidence. “hard” evidence has “concrete” information attached to it and types include published literature, statistics from the particular product or service, and local evaluation (koufogiannakis, 2012, p. 11). “soft” evidence focuses on “the story of how things fit together in context” and includes input from colleagues, feedback, and tacit knowledge (koufogiannakis, 2012, p. 11). koufogiannakis (2012, p. 10) found that practitioners were unsure of what constituted evidence; there is some hesitation as to the relevance and quality of research evidence. but regardless of the source, they were willing to consider whatever may inform decision-making (koufogiannakis, 2012, p. 10). this study confirms that research evidence alone is not enough to inform professional practice.   with a range of evidence sources identified by library and information professionals, koufogiannakis (2012, p. 9) found evidence that use in practice is dependent on the situation and type of problem being faced. a pilot study by partridge, edwards, and thorpe (2010) is the first australian study to explore variations in experiences of eblip by a cross-sector group of library and information professionals. participants described their experiences of the role of evidence in their daily practice. for example, an experience of evidence based practice “as service improvement,” where the professional’s focus is on best practice, looking at, and benchmarking against other library and information services, is undertaken (partridge, edwards, thorpe (2010, p. 286). the range of evidence identified in participants’ experiences with evidence based practice was associated with its use and “submission” as part of decision-making processes and culture within their organizations (partridge et al., 2010, p. 291). evidence used by australian library and information professionals in partridge et al. (2010) included research literature, as well as surveys, organizational strategy, and feedback, which is consistent with the sources of evidence advocated by koufogiannakis (2011; thorpe, partridge and edwards, 2008). findings of this study suggest that identifying types or sources of evidence and assigning its value is influenced by the situation and how the evidence is used in making decisions. koufogiannakis (2013, p. 9) argues that evidence identified and used in practice cannot be prescriptive, and must consider local context and circumstances; that the role of eblip is about using evidence and figuring out what is best for the situation or problem.   this then poses the question of what is “best available” evidence, the determination of which booth (2002) and koufogiannakis (2011) say only the library and information professional can do through appraisal and assigning value to evidence pertinent to making decisions in a given situation or context. within a professional practice setting, day-to-day realities can influence how evidence is encountered, gathered, and used. for example, influential stakeholders of an organization, such as a ceo, were found to determine the types of evidence gathered for a decision or task in a study which explored evidence based practice of special librarians (howlett and howard, 2015). further to this, koufogiannakis (2013) found that who owns the decision – the individual librarian or a group within an academic library – has an impact on how evidence is used, either for confirming a decision or to influence or convince. while similarities exist across library and information practice, empirical findings suggest variations in what is “best available.” understanding these variations will better position the existing eblip model to achieve its aims in making effective “value added” decisions in the provision of library and information services.   the research project   the three-year project commenced in 2013 and included two interconnected sub-studies. sub-study one explored academic librarians’ experience of evidence based practice (miller, partridge, bruce, yates, & howlett, submitted). constructivist grounded theory (charmaz, 2006) was the research approach employed. data collection consisted of semi-structured interviews with thirteen academic librarians recruited from australian universities. participants were recruited via a purposive sampling approach. participants were identified through publicly available information about staffing and organizational structures that is provided on university library websites. participants were approached via email inviting them to take part in the study. participants were identified to ensure variation in key aspects such as roles (e.g., liaison, reference and information librarians, library executives, team managers, and directors) and university. interview questions were designed to allow participants to describe their experiences of evidence based practice. in keeping with the grounded theory approach, there was one primary interview question: can you tell me about your experience of using evidence in your professional practice? in addition, a range of follow-up questions was also used to probe or elicit further information from participants about responses they provided.   data collection and analysis was undertaken simultaneously, with “each informing and focusing the other” (charmaz, 2006). this is a key element of grounded theory. typically, data is collected initially from a small pool of participants. this data is analyzed and the results inform the direction of further data collection, including sampling strategies. the researcher returns to the field continually until theoretical saturation is achieved. the findings from this sub-study provide a holistic view of academic librarians’ experience of evidence based practice. six categories of experience were constructed, which described librarians’ experiences of evidence based practice as: empowering, intuiting, affirming, connecting, noticing, and impacting. each category was identified through analysis of responses from more than one participant. it is beyond the scope of the current paper to give a detailed discussion of each category. further details regarding the sub-study’s findings are presented in miller et al. (submitted).   sub-study two explored how evidence based practice was experienced in one australian public library. ethnography was the research approach employed (fetterman, 1998). summertown library is the site for this investigation. summertown is a pseudonym used to protect the identities of the participants. the same pseudonyms have consistently been used in other articles reporting on this project (gillespie, partridge, bruce & howlett, 2016). the summertown library is a service provided by the summertown city council, which serves a large provincial town of over 180,000 residents. three branches are strategically located, with another branch planned in the near future to cater for the growing spread of the population. summertown is a coastal town that has a port and is the service centre for outlying mining and industrial industries. in more recent times it has become a site for migrant and refugee families.   one member of the research team travelled to summertown 15 times over a 6 month period. each visit was of 3 to 4 days. initial visits took a “big net” approach where the researcher was immersed in as many activities in the library as possible (e.g., shelving, assisting customers on the floor, culling, storytelling, assisting in the mobile van, and offering assistance wherever possible). at day's end the researcher recorded the events in a journal. included were unobtrusive observations, comments, interactions with staff, and attempts to interpret what the researcher was seeing. in addition, thirteen participants from within the library staff, representing diversity in the operational units and management levels, were interviewed. additionally, the researcher collected a range of print materials, including promotional leaflets of library activities, strategic plan and related timelines, planning pro forma, and feedback forms.   ethnographic data analysis is iterative “as it builds on ideas throughout the study” (fetterman, 1998, p. 92). analysis is a refinement of the data with the researcher trying to fit selections of the data into the bigger picture; in this case, experiences of evidence based practice. the researcher's reflections and interpretations, observations, interactions, and field notes provided the data for the current study. in keeping with the ethnographic approach, the findings are presented as a thematic narrative. evidence based practice is experienced in summertown public library through four interconnected and interdependent cultural orientations: valuing, being, learning, and leading. it is beyond the scope of the current paper to give a detailed discussion of each cultural orientation. further details regarding the sub-study’s findings are presented in gillespie et al. (2016).   in both sub-studies, data collection was designed to allow the participants to reveal their own experiences and understanding of evidence based practice and evidence. the research team did not impose a pre-determined definition or understanding of these concepts. this approach was in keeping with the two research methods employed and with the overarching aim of the project, which was to build an empirical basis for evidence based practice grounded in the lived experiences and realities of library and information science practitioners.   what is evidence?   in both studies, observations, feedback, professional colleagues, research literature, statistics, and intuition were recognized as evidence. each of these will be described and highlighted with examples from each of the studies.   observations as evidence   observations as evidence could be deliberate and controlled or unexpected and serendipitous. observation was recognized by tracy, a public librarian, “as a very powerful tool.” it can raise awareness of clients’ behaviours, demographics, and usage patterns, confirm professional judgment, and expose information concerning continual improvement of services and resources that may not be available from statistical data sources.   in the public library sub-study, especially among operational staff, observation was generally unexpected and not controlled. the observations were generally not recorded as they were seen, but they were often reported or passed on in conversation informally and in more formal meeting and planning situations. in the example which follows, observation served to raise awareness guiding taya, who was leading children's storytime, to seek supporting evidence. taya relates her observations of participation and attendance at storytime sessions:   we saw that our audiences for the mentoring and storytime sessions were increasing. and we were having a lot more multicultural people come to storytime.   taya observed that many of the families attending storytime were from diverse cultural backgrounds. at the end of sessions the families are asked to complete feedback forms. these provide some useful information, but the forms do not ask demographic questions. the growing cultural mix of families attending storytime could only be gathered from observations.   similarly, in the academic library study, observations of client or staff behaviours and usage patterns were experienced as forms of evidence, as one academic librarian described:   . . . that’s why i really love getting out and working on the desk for a couple of hours every day or going into classes and . . . teaching because you still pick up on ways to improve or identify . . . similar problems that the students are having maybe on a website or with searching or . . . just understanding their behaviours in regards to finding information as well. (participant 1).   for academic librarians in this study, observational evidence can be gained informally, such as in the above example, or from formal web data analytics observed during a daily task, for example: . . . i’ve got a library guide, which is all about how to reference in apa style . . . and i was looking at the statistics for that site ‘cause i really want to know how many students are actually accessing it. and it is one of the most popular library guides that we do have . . . what i found interesting was that the most used page within that guide itself was how to reference a website, not how to reference a journal article or a book from the library . . . . so that gives . . . evidence to me that . . . although we really try to focus our . . . sessions on using library resources . . . the students are still using websites . . . and wanting to reference them (participant 1).   in contrast, examples of observations by executive and management levels were more deliberate, although these too were unrecorded. for example, tonya, as executive manager of the public library, spends time on the library floor every week with the purpose of seeing first-hand what is happening in the library and getting a “feel” of work flows and responses from the staff as they interact with customer requests. when she is on the floor, customers do not realize that she is the executive manager, and tonya does not respond to them in that way either. she responds as any of the operational level library staff would do. in this way, she is deliberately observing staff interactions, work flows, customer concerns, and activities. this observational evidence keeps tonya in touch with the day-to-day library activities.   feedback as evidence   similar to observational evidence, feedback could be formally or deliberately sought, or could come from incidental encounters. the collection and analysis of data from customer satisfaction surveys was systematically carried out by the summertown council on a regular basis. additionally, customers were asked to fill in feedback forms after being involved in activities. mostly responses were positive, with “more of this” being quite common. the collated data of customer feedback forms, in conjunction with the customer satisfaction surveys, provided an overall picture of customer satisfaction. the results presented a positive image for the library and its operations; however, the customer feedback forms are a requirement of the council and are generic in nature. there were concerns among library managers that this type of evidence does not indicate level of impact, or provide information which might assist in future planning.   examples of evidence gained from incidental feedback were in the form of emails and in face-to-face encounters. betty explained, “quite often we will have a thank you . . . 90% of the time you'll get positive feedback.” maggie valued incidental customer feedback in this comment, “ . . . someone comes up to you or a few people come up to you after and say, wow, that was really good.”   for academic librarians, evidence is the corroboration of supportive feedback received and shared by colleagues, clients, and institutions, as illustrated in the following quote from an academic librarian:   i think i’m performing . . . effectively when my colleagues give me positive feedback . . . . i think managers can give you . . . lots of positive reinforcement about where you’re going . . . . i think it’s that 360 thing . . . you get it from all directions (participant 8).   similarly, for public librarians, this valuing was witnessed through the ways skills and achievements of staff were acknowledged and shared. open acknowledgments shared face-to-face and among staff was affirmative evidence. examples of feedback included shared responses and incidents, usually a firsthand encounter and emails, relating to customer reactions, events staff had attended, and feedback of a more general nature. affirmative evidence as part of conversation was an ongoing and everyday occurrence among all levels of staff.   other examples of feedback as evidence were in relation to workplace performance. this feedback could be face-to-face in meetings between staff and their supervisors. maggie valued this type of feedback.   . . . my supervisor is very good . . . if i'm doing a good job she'll tell me i'm doing a good job . . . that just prompts me to do better, you know.   it was during the interview with xavier, that he reflected that anecdotes gathered in face-to-face encounters with customers could provide valuable feedback. awareness among staff about the value of this feedback and the need to document the anecdote would move the evidence from being an unexpected encounter to a strategic approach in capturing this type of evidence.   for academic librarians, feedback is collected through listening and questioning, which can be used to enhance or change services and/or practice, as the following quote from a liaison librarian explains:   i might be . . . walking along a corridor, and an academic will actually . . . come out of their office . . . "thanks . . . i like . . . the library . . . what service they're offering, or what you did in that class the other day,” . . . whereas the formal feedback might be they'll send me an email after class to say . . . "we hope you can continue doing . . . joint classes,” . . . it's good to seek it out and get that formal, and sometimes you don't need to, they'll just tell you informally, which is great as well. i think i like that one better . . . . and if i haven't explained it well, i can tell. they'll ask me the same kind of questions again, or if i'm on the right track they might ask me . . . a more advanced question that . . . continues the conversation . . . you're sort of using that feedback . . . you're using that as evidence (participant 3).   professional colleagues as evidence   interactions with professional (industry or university librarian) colleagues at conferences are experienced by academic librarians as evidence. these interactions include sharing experiences and informal networking with librarian colleagues from other universities and institutions to experiment and gather new ideas to implement within their own library.   . . . attending conferences . . . events and webinars, and those types of things, where librarians from outside of my workplace are sharing their experiences, or their achievements, or projects they’ve worked on, getting a chance to see what everyone else is doing and then picking up on, “that’s what i’m doing,” or, “that’s something that i want to do” (participant 1).   additionally, they are sharing and collecting resources with other universities to demonstrate improved processes. they are also collaborating with outside subject experts to improve selection quality and learning resources. academic librarians also benefit from professional colleagues as evidence for benchmarking across similar libraries with good practice models to inform planning their own library. sharing and collecting ideas from other libraries was also evident in the public library sub-study. flora, the manager of collection development, was able to investigate and later implement innovative ways of displaying and arranging the non-fiction collection. she consulted with professional colleagues and visited other public libraries. in gaining first-hand evidence from outside sources, flora was able to report to management to plan and implement changes.   research literature as evidence   academic librarians are maintaining awareness of professional literature to evaluate specific library activities and make decisions in terms of industry standards and best practice, where applicable. they also maintain awareness of scholarly literature to increase credibility of evidence presented and service contribution to university contexts.   . . . individual librarians have . . . done literature searches. in terms of just reviewing particular services, we have done literature reviews just to see what evidence is out there for best practice (participant 5).   using scholarly literature did not feature strongly in the interviews in the public library sub-study. however, from incidental conversations with staff, the literature influenced and informed before decisions were made. cailey, from the children’s services section of the library, related an incident where floor staff were wishing to remove the book spinners used to display books in the junior and teenage areas. the book spinners were considered to be difficult to keep tidy and to relocate books when needed. cailey was able to bring to a management meeting literature which supported the use of book spinners, especially for junior and teenage customers. due to bringing these insights, the book spinners stayed. additionally, they were moved to more prominent positons in the children’s and teens’ areas of the library.    statistics as evidence   the summertown libraries and their council collected and collated many statistics. for instance, the library collected circulation and membership statistics generated from the library management system; visitor numbers were an indication of traffic in the different libraries and customer participation data demonstrated how many attended the different activities that were offered. the council quantified and plotted customer satisfaction surveys. this data was useful for accountability, to plot trends over time, to indicate workload such as periods and areas of high use, and likewise, underutilization.   flora, the collection development manager, considered that statistics were vital in her role. statistical data was used for budgets and user requests, and identified areas of high demand. the collection of data from user statistics revealed a need to extend the inter-library loan scheme. data generated from the library management system assisted staff in culling the collection, as well as identifying gaps and future purchases. the data was interrogated and selected to gain specific information for the long term management of the collection.   tonya, the executive manager, termed much of the data that was collected as lag data; that is, this type of data was evidence of past events. it was considered useful, but of limited value in the big picture of the library. tonya felt that she needed a bigger picture of the community landscape, and the statistical data being collected did not provide these insights. the library was able to provide many services of the type that are generally associated with the role of libraries, but she considered that there was much that could be specifically designed to meet the needs of the local population. tonya began by actively seeking statistical data as evidence from within the council. this type of data included demographic information such as age and ethnicity, population density, population growth, locations of growing, and changing population. additionally, from sources further afield, she sought data and indications of trends such as community needs in a changing economic environment.   academic librarians in this study are adept at “keeping an eye on” usage patterns from statistical data, but they are uncertain about how to use this evidence once identified. one of the main experiences reported by librarians is the perception that, while more challenging to capture “mental notes” for future use, qualitative data gathered from informal conversational feedback were more insightful and useful for decision making than quantitative data gathered from client surveys or databases where only numbers of interactions have been recorded (participant 3).   intuition as evidence   for academic librarians, intuition, encompassing wisdom and understanding of library staff/clients’ behaviours, is being used as evidence to solve problems and redesign library services. as the following senior level academic librarian expressed in relation to using her intuitive understanding of staff under her supervision:   i’ve learnt to trust my gut, and … i’ve learnt not to be scared to invite someone into the office and say “are you alright?” (participant 11).   another librarian who teaches information literacy classes described the intuitive evidence of knowing she is teaching effectively as:   . . . sometimes it’s more of a perception or an intuition you know when you’re teaching a class and you can see the students . . . the light go on in their eyes . . . . many times you can just visually see it . . . you know they’ve understood and they’ve comprehended . . . (participant 9).   nadia, the team leader of the summertown library’s customer service section, considered that gaining understandings of many aspects of the organization and the people who worked within it helped to build a picture about what is happening; that intuition is something that is built over time using a variety of sources to come to conclusions: “ . . . a lot of what you do is still gut instinct.”    nadia draws on her professional knowledge as an experienced team leader and her professional experiences from working in the organization. these guide her intuition, which in turn guides her actions. this can be explained as there being two parts to the practitioner’s expression of the term “intuition”; that is, professional experience and professional knowledge, and there is a nuanced difference between these two concepts. when evidence from whatever source is presented, the practitioner looks at this and makes a judgment based on professional and past experiences. this action relies on professional experience. when the practitioner questions and seeks further evidence in different or better ways, in order to gain more information, or to confirm or deny the evidence, professional knowledge comes into play. nadia explained it in this way:   in that in terms of evidence, don’t just rely on, on what you’re being told or how you’re being trained because at the end of the day, we are a government organization and we are trained a certain way. you’ve got to think outside the box.   professional knowledge is a measured and thoughtful response to the evidence; the practitioner is defining the purpose of the evidence, considering ways to explore it further, and drawing upon professional knowledge.   evidence in context   this study has revealed that professionals in both the public library and academic library shared similar views about what they considered to be evidence. the similarities bring attention to and emphasize the many different types of evidence that inform and confirm everyday practice. however, there were some differences. these were related to the way in which the studies gathered, analyzed, and presented the data, rather than the perceptions of evidence from each group. the grounded theory academic library study provided many contexts, with each interview coming from a different library setting, but the revelations from these interviews were limited to how much the participants were willing to share; there were no first hand observations from the researcher. in contrast, the ethnographic public library study revealed much contextual data from one library setting. the first hand observations and insights of the researcher provided many examples of library professionals’ experiences of evidence based practice, providing data of a more nuanced nature and rich contextual information which face-to-face interviews on their own may not provide.   conclusion   this study has revealed that what is experienced as evidence by academic and public library professionals is similar in many ways. in both studies, evidence based practice was a lived experience. observations, feedback, professional colleagues, research literature, statistics, and intuition were recognized as evidence. however, many of these types of evidence were used in conjunction with each other as a means to support or confirm. all evidence types were treated equally by the participants and there was not a hierarchical structure of evidence types. library professionals drew upon their professional knowledge and experiences to draw conclusions from the various types of evidence. this nuanced approach of contextualizing evidence, drawing from many sources and applying it to the local environment, demonstrates an experiential engagement with evidence based practice.   this is in contrast to early writing on evidence based practice in librarianship as expressed by brice and hill (2004), where evidence from the research literature was held in the highest regard, with less recognition of the practitioner observed and user reported evidence types. this study serves to explain and elaborate in practice based terms the early eblip definition provided by booth (2002). previously, evidence based practice as an experience had been explored by gillespie (2014), koufogiannakis (2013), and partridge et al. (2010). this study adds to this growing empirial base; it highlights that there is no one way to be an evidence based practitioner and that many sources of evidence are utilized by library professionals to guide and inform practice and in decision making.   this study has provided many insights about the nature of evidence among library professionals in academic and public library contexts. the examples in this paper provide lived experiences of library professionals gathering and using many sources of evidence in their everyday work environments.   acknowledgements   this project was funded by the australian research council. the authors would like to thank the many australian library and information professionals who kindly participated in this project.   references   booth, a. (2002). from ebm to ebl: two steps forward or one step back? medical reference services quarterly, 21(3), 51-64. http://dx.doi.org/10.1300/j115v21n03_04   brice, a., & hill, a. (2004). a brief history of evidence-based practice. in a. booth & a. brice (eds.), evidence-based practice for information professionals: a handbook (pp. 13-23). london: facet publishing.   charmaz, k. (2006). constructing grounded theory: a practical guide through qualitative analysis. london: sage publications.   eldredge, j. d. (2000). evidence-based librarianship: an overview. bulletin of the medical library association, 88(4), 289-302. retrieved from http://www.ncbi.nlm.nih.gov/pmc/articles/pmc35250/   fetterman, d. m. (1998). ethnography: step by step. thousand oaks, ca: sage.   gillespie, a. m. (2014). untangling the evidence: introducing an empirical model for evidence-based library and information practice. information research, 19(3). retrieved from http://eprints.qut.edu.au/76198   gillespie, a. m., partridge, h., bruce, c., & howlett. a. (2016). the experience of evidence-based practice in an australian public library: an ethnography. information research, 21(4). retrieved from http://www.informationr.net/ir/21-4/paper730.html   howlett, a., & howard, z. (2015) exploring the use of evidence in practice by australian special librarians. information research, 20(1). retreived from http://www.informationr.net/ir/20-1/paper657.html#.wjjrarsykua   koufogiannakis, d. (2011). considering the place of practice-based evidence within evidence based library and information practice (eblip). library & information research, 35(111), 41-58. retrieved from http://www.lirgjournal.org.uk/lir/ojs/index.php/lir/article/view/486/527   koufogiannakis, d. (2012). academic librarians’ conception and use of evidence sources in practice. evidence based library and information practice, 7(4), 5-24. http://dx.doi.org/10.18438/b8jc8j   koufogiannakis, d. (2013). eblip7 keynote: what we talk about when we talk about evidence. evidence based library & information practice, 8(4), 6-17. . http://dx.doi.org/10.18438/b8659r   miller, f., partridge, h., bruce, c., yates, c., & howlett, a. (submitted). how academic librarians’ experience evidence based practice: a grounded theory model. library and information science research.   partridge, h., edwards, s. l., & thorpe, c. (2010). evidence-based practice: information professionals' experience of information literacy in the workplace. in a. lloyd & s. talja (eds.). practising information literacy: bringing theories of learning, practice and information literacy together. (pp. 273-297). amsterdam: chandos. http://dx.doi.org/10.1016/b978-1-876938-79-6.50013-3   thorpe, c., partridge., & edwards, s. l. (2008). are library and information professionals ready for evidence based practice? paper presented at australian library and information association (alia) biennial conference 2-5 september 2008, alice springs, northern territory, australia.   todd, r. j. (2006). school libraries and evidence-based practice: an integrated approach to evidence. school libraries worldwide, 12(2), 31-37.   todd, r. j. (2009). school librarianship and evidence based practice: progress, perspectives, and challenges. evidence based library & information practice, 4(2), 78-96. http://dx.doi.org/10.18438/b8bs62   evidence based library and information practice using evidence in practice   increasing objectivity in eresource selection using a priority matrix     megan l. anderson research & curriculum librarian library and media services          fanshawe college london, ontario, canada email: manderson@fanshawec.ca   linda l. crosby research & curriculum librarian library and media services fanshawe college london, ontario, canada email: lcrosby@fanshawec.ca   received: 28 aug. 2018                                                                  accepted: 12 nov. 2018      2016 anderson and crosby. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29499   setting   library and media services (lms) at fanshawe college is located in london, ontario, canada. lms is an academic library providing a variety of services and resources to approximately 14,000 full-time equivalent (fte) students and 2,800 faculty and staff. a significant number of students attend regional campuses, with no on-site library services, thereby increasing the need for a strong eresource collection. there are three faculty librarians at this college, two of whom investigated the use of a priority matrix for eresource selection and renewal.   problem   librarians at fanshawe college faced a major dilemma. a significant eresource budget cut, combined with a depressed canadian dollar, made it impossible to retain all the databases in the collection. the ensuing decision-making process left the librarians repeatedly fighting their collection management instincts. the process was challenging, in part because each librarian had her own emotional investment in particular databases. the librarians believed there must be a way to objectively assess which databases should be retained or added to the collection. this objectivity is vitally important because, as walters (2016) explains “regardless of the library’s . . . selection model, collection development librarians must be able to explain their decisions to librarians, faculty, and administrators with primary interests in areas other than collection development” (p. 10).   the librarians were also curious to see if their instincts aligned with an objective, rational review of the data. a priority matrix format has proven successful at this library when applied to other projects. the librarians decided to see if this format could also be successful when applied to eresource collection management. further, the key to solving this problem was to find or create a tool that allowed eresource decisions to be made easily and systematically.   evidence   the evidence component of this project was twofold: a literature review informed the decision-making during development of the matrix, and local and vendor data was used in the matrix to rank existing eresource subscription products. the literature provided an excellent starting point for determining what factors were important to consider in this evaluation. the investigating librarians were quite familiar with the consideration and application of indicators such as usage statistics given that this type of evaluation is “focused on demand, as indicated by usage” (kohn, 2013, p. 89). local data included information such as the number of students registered in a program. vendor data provided content and coverage details. concrete criteria, as opposed to the more abstract concepts upon which the librarians might have relied instinctively, was also discovered. for example, walters (2016) focuses on the idea of brand recognition when stating that “relevant papers . . . will be found only if the patron first recognizes that the online resource . . . has a reasonable chance of including relevant works” (p. 13).   implementation   the process began with an environmental scan including a survey of electronic mail lists and completion of a literature review. ideally, the investigating librarians hoped a “plug and play” solution was already in existence. after the search yielded no promising results, they resolved that a priority matrix would be created. microsoft (ms) excel seemed like a natural solution as it is capable of mathematical formulas, is possible to customize, and is cost effective.   the next step was to compile a list of the appropriate criteria. table 1 lists the selection of weighted criteria. “frequency of course offering,” an unweighted criterion, is reserved for use when a resource is at risk of cancellation. at that point, the librarians need to review how often the course is offered because it affects usage statistics, particularly with very specific and specialized eresources such as quicklaw.     table 1 weighted criteria criteria weight content x10 required resource x10 cost sharing x10 cost x8 # of applicable programs x8 cost per expected user x8 currency of content x8 licensing & authentication x6 ease of use x6 overlap of content x6 depth of coverage x6 opportunity cost x4 vendor support x2 perpetual access x1 brand recognition x1 % of budget assigned to applicable school(s) x1     table 2 priority matrix weights and rationales criteria rationale priority the priority number calculated for a particular resource is calculated after the resource has been put through the matrix. content (x10) content of a particular resource is one of, if not the, most important factors in determining a resources value. our beliefs on this particular criterion were reinforced by mangrum and pozzebon’s 2012 studya, and walters’ 2016 articleb.  as such, this criterion was assigned the top possible value score of 10. required resource (x10) resources required for programs to maintain accreditation are, naturally, more important than others and therefore this criterion was assigned a value score of 10.  cost sharing (x10) given the current economic climate, the amount of money a program or school is able to contribute to a resource heavily influences our ability to make a purchase, resulting in this criterion begin assigned a value score of 10. cost (x8) cost is one of the most important considerations when reviewing potential purchases, however it is not one of the top considerations and so was assigned a value score of 8. # of applicable programs (x8) the number of programs that may find a particular resource useful speaks directly to value for money.  something may have a low initial cost, but may not be useful – thereby having low value for money. this is equally as important as the initial cost, so was also assigned a value score of 8. cost per expected user (x8) as important as the overall cost, the cost per expected user of a particular resource is equally important and speaks to value for money. some resources are specialized, and it is not reasonable to compare their usage statistics to those of resources intended for a more general audience. this criterion should create a more equitable playing field. this criterion has been assigned a value of 8. actual cost per use (x8) the number of uses any particular resource has requires further context.  for example, a resource may have 1,000 uses that are only $0.02/use or they may have 100 uses that are $3.50/use.  this further contextualization allows accurate assessment of value for money and return on investment. this criterion has been assigned a value of 8, in line with the weight of other cost criterion. currency of content (x8) currency of content is almost as important as overall content. while a database may have lots of title holdings, it is important to consider how current the content is – for example, heavily embargoed resources are not particularly useful and reduce the value of the resource.  a value of 8 has been assigned to this criterion. licensing & authentication (x6) licensing, including permitted use, and authentication method are important as they influence the usability of a particular resource.  this criterion has been assigned a value of 6. ease of use (x6) patrons are more likely to make use of a database that is intuitive and user friendly. to that end, this is a relatively important criterion, but since learning how to use databases is part of a college education the value is lesser than it would be in other types of libraries. as such, this criterion has been assigned a value of 6. overlap of content (x6) it is important to consider how much the content of a resource overlaps with content in the existing collection, both print and electronic, to ensure we are not paying for the same resource twice unless it is justified.  to reflect this, a value of 6 has been assigned. depth of coverage (x6) backfiles, and their relative importance, varies by database and discipline, which is why this criterion has been assigned a mid-range value of 6. opportunity cost (x4) what would the cost to the library be if we had to buy all of the relevant content individually, rather than as part of the database package? this is important to consider, but not as important as many other factors and therefore has been assigned a value of 4. vendor support (x2) it is important to note how many technology-based incidents are associated with a particular database.  however, how many of said incidents will be tolerated is largely dependent on other criteria, with a much higher value, and for this reason this criterion has been assigned a value of 2. perpetual access (x1) lack of perpetual access is certainly not a deal breaker, however it is an additional value that should be considered.  it was assigned a value of 1 to reflect this. brand recognition (x1) as per walters, “relevant papers…will be found only if the patron first recognizes that the online resource…has a reasonable chance of including relevant works.”c this criterion has been assigned a value of 1. % of budget assigned to applicable school(s) (x1) the percentage of the overall budget assigned to the applicable school(s) must be considered to ensure that all schools are being equitably represented in library holdings. frequency of course offering this criterion is not weighted, and is not routinely used in assessing resources.  use should be limited to resources that are on the bubble as the frequency of course offerings may influence the use, or lack thereof, of particular resources. amangrum & pozzebon, 2012. bwalters, 2016. cibid.     after compiling the list, the investigating librarians took the next step to assign a weight to each criterion to ensure that the relative importance of each was considered. for example, if a database package is near-perfect in terms of content, should ip authentication, or lack thereof, dissuade collections librarians from making a purchase or renewing a subscription? by weighting each criterion, situations where a less important criterion overrules a more important criterion, thereby skewing decision-making, can be avoided. the weights and associated rationales are found in table 2. settling on the criteria weighting was the last step before building the matrix in ms excel.   one of the investigating librarians created an ms excel spreadsheet that contains six worksheets: evaluation; results; criteria description; criteria weighting rationale, charts; and database data. the priority matrix then went live on november 1, 2016.   evaluation: the collections librarians determine scores for the criteria for each eresource and enter the data into this worksheet. the collections librarians determine scores collectively if a resource is multi-disciplinary. if a resource is discipline-specific, the librarian responsible for collections within the discipline will establish the score.   results: scores for each resource are automatically populated from the evaluation worksheet and auto-calculated according to weight. each resource is assigned a score of one to four. the score then determines the decision that is made. an explanation of the decisions is found in table 3.     table 3 purchase or renewal decisions rating decision 1 high priority purchase / renewal; robustly meets all requirements 2 generally meets all requirements; purchase / renew if funds available 3 meets minimal requirements; purchase / renew with caution 4 does not meeting basic requirements; do not purchase / renew     criteria description: this worksheet defines each criterion and describes what to look for when assigning a score.   criteria weighting rationale: this worksheet contains a list of each criteria, the weight assigned to each, and associated rationale behind each weight assignment.   charts: this worksheet uses the data generated in the evaluation worksheet and displays it as images rather than numbers for optimal visual data representation.   database data: the eresource specialist proactively inputs raw database data, such as cost, usage, and cost sharing, needed by the librarians to make their retention and selection decisions.    the final step was to present the product to the senior manager and the non-investigating librarian colleague. an example of a completed priority matrix and ranking, such as that found in table 4, were included in this presentation.   outcome   the priority matrix has been in use since november 1, 2016 as ad hoc renewals have come in. utilization of the matrix identified required minor tweaks, three of which are of note. while “cost per expected user” was included in the initial criteria, “actual cost per use” had inadvertently been omitted from this list. “actual cost per use” is, of course, of tremendous importance so it was added to the list of criteria and assigned a weight of eight. the investigating librarians quite quickly realized that two priority matrices are necessary: one for renewal and retention of databases, and one for new subscriptions. this is a critical differentiation since a criterion such as “actual cost per use” is not available and should not be applied to a potential new resource. additionally, the investigating librarians reworded some criteria descriptions to make their scope encompassing or applicable when evaluating non-traditional databases like simplyanalytics or statista.     table 4 sample completed priority matrix and ranking database data worksheet sample resource a cost sharing 0 cost $27,363 expected users 2,637 cost per expected user $10.38 actual use 16,879 actual cost per use $1.62 depth of coverage 1977 vendor support no issues perpetual access n % of budget assigned to school 22% evaluation worksheet sample resource a content 4 required resource 3 cost sharing 0 cost 2 # of applicable programs 4 cost per expected user 2 actual cost per use 4 currency of content 3 licensing & authentication 4 ease of use 3 overlap of content 4 depth of coverage 4 opportunity cost 4 vendor support 4 perpetual access 0 brand recognition 0 % of budget assigned to school 4 results worksheet sample resource a priority 2 renew / cancel r content 40 required resource 30 cost sharing 0 cost 16 # of applicable programs 32 cost per expected user 16 actual cost per use 32 currency of content 24 licensing & authentication 24 ease of use 18 overlap of content 24 depth of coverage 24 opportunity cost 16 vendor support 8 perpetual access 0 brand recognition 0 % of budget assigned to school 4 total 308     since implementation, the librarians have an annual eresource collection meeting during which all existing subscriptions, as well as desired additions, are evaluated using the priority matrix. the librarians pass these decisions on to the eresource specialist who acquires, renews, or cancels resources accordingly. the investigating librarians monitored the application of the matrix for the next year to enhance and refine it whenever necessary or possible. as well, the possibility of applying this same approach to other resource types such as streaming media collections will be explored in future. using the matrix for decisions is a welcome change to the process. it allows for more efficient decision-making, and increases the ability to articulate any contentious collections decisions in a manner that is clear to both non-practitioners and practitioners.   reflection   the addition of evidence into eresource collections decisions was challenging in some ways, yet relatively simple in others. the librarians already used a significant amount of evidence, but not in a uniform or consistent manner. additionally, many of the evidence-based decisions made prior to the implementation of the matrix were at an instinctual level, causing the challenge to lie in slowing down the process and identifying what pieces of evidence were being used intuitively. vendor-supplied data provided some challenges to the process, as the type of data tracked and supplied to the library is not consistent between vendors. counter-compliant statistics were used whenever possible to “compare data received from different publishers and vendors” (counter, 2018, para. 3).   conclusion   a failing canadian dollar and a declining eresources budget compelled the librarians at fanshawe college to address the way eresources selection and retention decisions were made. additionally, the librarians needed to be able to appropriately articulate to non-library-science practitioners why new resources could not be added and existing resources were being eliminated. by reviewing the literature and applying local and vendor data in a consistent manner, the librarians could make objective decisions, rather than relying on their instincts. after applying the matrix, it became clear that the librarians’ instincts were actually fairly consistent with what the hard data demonstrated, as there were no significantly unexpected outcomes in terms of retention decisions. its application did, however, require one librarian to realize she was continuing to advocate for a database that, despite being a good fit for the needs of a particular program, was just not being used. furthermore, the matrix allowed the librarians to demonstrate to non-library-science practitioners that the budget is at its bare minimum, and further cuts would decimate the collection.     references   counter. (2018). about counter. retrieved from https://www.projectcounter.org/about   kohn, k. c. (2013). usage-based collection evaluation with a curricular focus. college & research libraries, 74(1), 85-97. https://doi.org/10.5860/crl-295   mangrum, s., & pozzebon, m. e. (2012). use of collection development policies in electronic resource management. collection building, 31(3), 108-114. https://doi.org/10.1108/01604951211243506   walters, w. h. (2016). evaluating online resources for college and university libraries: assessing value and cost based on academic needs. serials review, 42(1), 10-17. https://doi.org/10.1080/00987913.2015.1131519     news/announcements   call for papers – rails 2016      2016. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   the research applications, information and library studies (rails) is australasia’s premier research conference for information and library studies and related disciplines. this annual conference brings together educators, researchers and practitioners within the information professions to encourage a culture of informed and innovative research practice. the 2016 conference will be held in wellington, new zealand on 6-8 december 2016.    we invite educators, practitioners, and research students to submit proposals on the conference theme hono tangata: rangahaua kia mārama—bridging the gap: from research to practice in information studies. full papers (20 minutes with 10 minutes for q&a) are sought, as well as proposals for panel discussions and posters showcasing works in progress or completed research projects. the closing date for all submissions is 1 august 2016.   all proposals as are to be submitted through the easychair system at https://easychair.org/conferences/?conf=rails2016   further information about the conference is available on the conference website (http://railsconference.com/ ) or by contacting the conference chair, dr. brenda chawner at brenda.chawner@vuw.ac.nz   evidence summary   low levels of teacher information literacy awareness and collaboration between librarians and teachers in information literacy instruction   a review of: mckeever, c., bates, j., & reilly, j. (2017). school library staff perspectives on teacher information literacy and collaboration. journal of information literacy, 11(2), 51-68. https://doi.org/10.11645/11.2.2187   reviewed by: ruby warren user experience librarian university of manitoba libraries winnipeg, manitoba, canada email: ruby.warren@umanitoba.ca   received: 6 june 2018    accepted: 9 aug. 2018      2018 warren. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29459     abstract   objective – researchers sought to determine school library staff perspectives on the information literacy knowledge held by secondary school teachers, and teacher relationships with the library.   design – interviews analyzed with thematic and axial coding.   setting – secondary schools in northern ireland.   subjects – 21 schools across northern ireland were selected as a sample, including urban, rural, integrated, grammar, and secondary schools. 16 schools ultimately participated.   methods – semi-structured interviews were conducted with one library staff member at each selected secondary school. interview audio and notes were transcribed and coded thematically both manually by the researchers and using nvivo. categories were identified by open coding, then relationships identified via axial coding.   main results – the majority (10 of 16) of library staff members interviewed expressed that they had not been asked about information literacy by teachers, and only one library staff member described a truly collaborative instructional relationship with teaching staff. the majority of staff expressed either that teachers were familiar with concepts related to information literacy but did not know the name for them, or, that they thought information literacy was entirely unfamiliar to teachers at their school. staff frequently cited competing priorities (for example, standardized testing) and limited class time as potential causes for teachers not focusing on information literacy concepts.   conclusion – both cultural and policy changes need to be made in schools to prioritize information literacy as a core competency for both students and teachers. the researchers call for greater intra-school collaboration as a means to achieve this cultural change.   commentary   this is the first study to evaluate the knowledge of information literacy among teachers in northern ireland. it is interesting because rather than asking teachers to assess themselves, the researchers instead asked school library staff to assess teacher familiarity with information literacy. this perspective is particularly valuable because library staff are more likely to be familiar with the concept of information literacy, as well as more familiar with information literary as a term. as such, school library staff are thus able to more accurately report on the ways teachers pursue engagement with the library to support information literacy instruction. context provided by the authors indicates that professional librarian positions in schools are extremely uncommon in northern ireland – no national information strategy framework exists, and therefore this study provides welcome insight by focusing on information literacy education where there is often no librarian available to support it. the findings of this study align with recent interviews of teaching staff in alberta, canada (another location with few school librarians), where teaching staff self-reported that they are unfamiliar with the term information literacy, and that a variety of competing priorities or time constraints affect how they approach imparting information literacy concepts (smith, 2013). a lack of library collaboration, lack of familiarity with information literacy, and lack of time seem to be concerns with secondary school information literacy education that are consistent across recent related literature (lee, reed, & laverty, 2012; stockham & collins, 2012; togia, korobili, malliari, & nitos, 2015).   this study scores 88% validity when evaluated against the criteria for qualitative research in glynn’s (2006) ebl critical appraisal checklist. the initial sample, chosen for inclusivity of different types of secondary schools in northern ireland, achieved acceptable breadth to represent different types of secondary educational institutions and experiences, and school exclusion criteria (e.g., not having a staffed library) were clearly defined. although multiple institutions declined to participate, the final sample of 16 schools appears to have achieved saturation — the point at which no new ideas are being introduced by participants — and is therefore adequate for insight.   the authors’ coding process is explained in detail, and the semi-structured interview outline is provided in the appendix for easy study replication. greater clarity could have been offered on whether the authors collaboratively coded transcripts, and an inter-coder reliability calculation would have further boosted the study’s face validity. identified themes and sub-themes are well presented in the article and appear to clearly follow from interview excerpts, and future research opportunities are highlighted.   this research may be useful for advocates and policy makers looking to encourage librarian presence in secondary schools, and for those attempting to improve information literacy curriculum or collaborative education practices in their schools. study results clearly indicate a need for collaborative information literacy support for teaching staff, many of whom lack familiarity with the subject.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154   lee, e., reed, b., & laverty, c. (2012). preservice teachers’ knowledge of information literacy and their perceptions of the school library program. behavioral & social sciences librarian, 31(1), 3-22. https://doi.org/10.1080/01639269.2012.657513   smith, j. (2013). secondary teachers and information literacy (il): teacher understanding and perceptions of il in the classroom. library & information science research, 35(3), 216-222. https://doi.org/10.1016/j.lisr.2013.03.003   stockham, m., & collins, h. (2012). information literacy skills for preservice teachers: do they transfer to k-12 classrooms? education libraries, 35(1-2), 59-72. https://doi.org/10.26443/el.v35i1-2.316   togia, a., korobili, s., malliari, a., & nitsos, i. (2015). teachers’ views of information literacy practices in secondary education: a qualitative study in the greek educational setting. journal of librarianship and information science, 47(3), 226-241. https://doi.org/10.1177/0961000614532485   <html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/tr/rec-html40"> <head> <meta http-equiv=content-type content="text/html; charset=unicode"> <meta name=progid content=word.document> <meta name=generator content="microsoft word 15"> <meta name=originator content="microsoft word 15"> <link rel=file-list href="es_goldstein_29516_files/filelist.xml"> <link rel=edit-time-data href="es_goldstein_29516_files/editdata.mso"> <!--[if !mso]> <style> v\:* {behavior:url(#default#vml);} o\:* {behavior:url(#default#vml);} w\:* {behavior:url(#default#vml);} .shape {behavior:url(#default#vml);} </style> <![endif]--><!--[if gte mso 9]><xml> <o:documentproperties> <o:author>mallan, katrine</o:author> <o:template>normal</o:template> <o:lastauthor>hinrichs, rachel</o:lastauthor> <o:revision>4</o:revision> <o:totaltime>2</o:totaltime> <o:created>2019-03-14t14:18:00z</o:created> <o:lastsaved>2019-03-14t14:59:00z</o:lastsaved> <o:pages>1</o:pages> <o:words>1216</o:words> <o:characters>6936</o:characters> <o:company>microsoft</o:company> <o:lines>57</o:lines> <o:paragraphs>16</o:paragraphs> <o:characterswithspaces>8136</o:characterswithspaces> <o:version>16.00</o:version> </o:documentproperties> <o:officedocumentsettings> <o:allowpng/> </o:officedocumentsettings> </xml><![endif]--> <link rel=datastoreitem href="es_goldstein_29516_files/item0009.xml" target="es_goldstein_29516_files/props010.xml"> <link rel=themedata href="es_goldstein_29516_files/themedata.thmx"> <link rel=colorschememapping href="es_goldstein_29516_files/colorschememapping.xml"> <!--[if gte mso 9]><xml> <w:worddocument> <w:trackmoves>false</w:trackmoves> <w:trackformatting/> <w:hyphenationzone>21</w:hyphenationzone> <w:punctuationkerning/> <w:validateagainstschemas/> <w:saveifxmlinvalid>false</w:saveifxmlinvalid> <w:ignoremixedcontent>false</w:ignoremixedcontent> <w:alwaysshowplaceholdertext>false</w:alwaysshowplaceholdertext> <w:donotpromoteqf/> <w:lidthemeother>de</w:lidthemeother> <w:lidthemeasian>x-none</w:lidthemeasian> <w:lidthemecomplexscript>x-none</w:lidthemecomplexscript> <w:compatibility> <w:breakwrappedtables/> <w:snaptogridincell/> <w:wraptextwithpunct/> <w:useasianbreakrules/> <w:dontgrowautofit/> <w:splitpgbreakandparamark/> <w:enableopentypekerning/> <w:dontflipmirrorindents/> <w:overridetablestylehps/> </w:compatibility> <w:browserlevel>microsoftinternetexplorer4</w:browserlevel> <m:mathpr> <m:mathfont m:val="cambria math"/> <m:brkbin m:val="before"/> <m:brkbinsub m:val="&#45;-"/> <m:smallfrac m:val="off"/> <m:dispdef/> <m:lmargin m:val="0"/> <m:rmargin m:val="0"/> <m:defjc m:val="centergroup"/> <m:wrapindent m:val="1440"/> <m:intlim m:val="subsup"/> <m:narylim m:val="undovr"/> </m:mathpr></w:worddocument> </xml><![endif]--><!--[if gte mso 9]><xml> <w:latentstyles deflockedstate="false" defunhidewhenused="false" defsemihidden="false" defqformat="false" defpriority="99" latentstylecount="371"> <w:lsdexception locked="false" priority="0" qformat="true" name="normal"/> <w:lsdexception locked="false" priority="9" qformat="true" name="heading 1"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 2"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 3"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 4"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 5"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 6"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 7"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 8"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 9"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 6"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 7"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 8"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 9"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 1"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 2"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 3"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 4"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 5"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 6"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 7"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 8"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 9"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="normal indent"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="footnote text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="annotation text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="header"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="footer"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index heading"/> <w:lsdexception locked="false" priority="35" semihidden="true" unhidewhenused="true" qformat="true" name="caption"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table of figures"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="envelope address"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="envelope return"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="footnote reference"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="annotation reference"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="line number"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="page number"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="endnote reference"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="endnote text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table of authorities"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="macro"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="toa heading"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number 5"/> <w:lsdexception locked="false" priority="10" qformat="true" name="title"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="closing"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="signature"/> <w:lsdexception locked="false" priority="1" semihidden="true" unhidewhenused="true" name="default paragraph font"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text indent"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="message header"/> <w:lsdexception locked="false" priority="11" qformat="true" name="subtitle"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="salutation"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="date"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text first indent"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text first indent 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="note heading"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text indent 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text indent 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="block text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="hyperlink"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="followedhyperlink"/> <w:lsdexception locked="false" priority="22" qformat="true" name="strong"/> <w:lsdexception locked="false" priority="20" qformat="true" name="emphasis"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="document map"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="plain text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="e-mail signature"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html top of form"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html bottom of form"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="normal (web)"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html acronym"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html address"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html cite"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html code"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html definition"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html keyboard"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html preformatted"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html sample"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html typewriter"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html variable"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="normal table"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="annotation subject"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="no list"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="outline list 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="outline list 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="outline list 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table simple 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table simple 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table simple 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table classic 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table classic 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table classic 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table classic 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table colorful 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table colorful 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table colorful 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 6"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 7"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 8"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 6"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 7"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 8"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table 3d effects 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table 3d effects 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table 3d effects 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table contemporary"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table elegant"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table professional"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table subtle 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table subtle 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table web 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table web 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table web 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="balloon text"/> <w:lsdexception locked="false" priority="59" name="table grid"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table theme"/> <w:lsdexception locked="false" semihidden="true" name="placeholder text"/> <w:lsdexception locked="false" priority="1" qformat="true" name="no spacing"/> <w:lsdexception locked="false" priority="60" name="light shading"/> <w:lsdexception locked="false" priority="61" name="light list"/> <w:lsdexception locked="false" priority="62" name="light grid"/> <w:lsdexception locked="false" priority="63" name="medium shading 1"/> <w:lsdexception locked="false" priority="64" name="medium shading 2"/> <w:lsdexception locked="false" priority="65" name="medium list 1"/> <w:lsdexception locked="false" priority="66" name="medium list 2"/> <w:lsdexception locked="false" priority="67" name="medium grid 1"/> <w:lsdexception locked="false" priority="68" name="medium grid 2"/> <w:lsdexception locked="false" priority="69" name="medium grid 3"/> <w:lsdexception locked="false" priority="70" name="dark list"/> <w:lsdexception locked="false" priority="71" name="colorful shading"/> <w:lsdexception locked="false" priority="72" name="colorful list"/> <w:lsdexception locked="false" priority="73" name="colorful grid"/> <w:lsdexception locked="false" priority="60" name="light shading accent 1"/> <w:lsdexception locked="false" priority="61" name="light list accent 1"/> <w:lsdexception locked="false" priority="62" name="light grid accent 1"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 1"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 1"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 1"/> <w:lsdexception locked="false" semihidden="true" name="revision"/> <w:lsdexception locked="false" priority="34" qformat="true" name="list paragraph"/> <w:lsdexception locked="false" priority="29" qformat="true" name="quote"/> <w:lsdexception locked="false" priority="30" qformat="true" name="intense quote"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 1"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 1"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 1"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 1"/> <w:lsdexception locked="false" priority="70" name="dark list accent 1"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 1"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 1"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 1"/> <w:lsdexception locked="false" priority="60" name="light shading accent 2"/> <w:lsdexception locked="false" priority="61" name="light list accent 2"/> <w:lsdexception locked="false" priority="62" name="light grid accent 2"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 2"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 2"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 2"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 2"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 2"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 2"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 2"/> <w:lsdexception locked="false" priority="70" name="dark list accent 2"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 2"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 2"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 2"/> <w:lsdexception locked="false" priority="60" name="light shading accent 3"/> <w:lsdexception locked="false" priority="61" name="light list accent 3"/> <w:lsdexception locked="false" priority="62" name="light grid accent 3"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 3"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 3"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 3"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 3"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 3"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 3"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 3"/> <w:lsdexception locked="false" priority="70" name="dark list accent 3"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 3"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 3"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 3"/> <w:lsdexception locked="false" priority="60" name="light shading accent 4"/> <w:lsdexception locked="false" priority="61" name="light list accent 4"/> <w:lsdexception locked="false" priority="62" name="light grid accent 4"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 4"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 4"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 4"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 4"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 4"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 4"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 4"/> <w:lsdexception locked="false" priority="70" name="dark list accent 4"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 4"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 4"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 4"/> <w:lsdexception locked="false" priority="60" name="light shading accent 5"/> <w:lsdexception locked="false" priority="61" name="light list accent 5"/> <w:lsdexception locked="false" priority="62" name="light grid accent 5"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 5"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 5"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 5"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 5"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 5"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 5"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 5"/> <w:lsdexception locked="false" priority="70" name="dark list accent 5"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 5"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 5"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 5"/> <w:lsdexception locked="false" priority="60" name="light shading accent 6"/> <w:lsdexception locked="false" priority="61" name="light list accent 6"/> <w:lsdexception locked="false" priority="62" name="light grid accent 6"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 6"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 6"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 6"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 6"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 6"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 6"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 6"/> <w:lsdexception locked="false" priority="70" name="dark list accent 6"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 6"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 6"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 6"/> <w:lsdexception locked="false" priority="19" qformat="true" name="subtle emphasis"/> <w:lsdexception locked="false" priority="21" qformat="true" name="intense emphasis"/> <w:lsdexception locked="false" priority="31" qformat="true" name="subtle reference"/> <w:lsdexception locked="false" priority="32" qformat="true" name="intense reference"/> <w:lsdexception locked="false" priority="33" qformat="true" name="book title"/> <w:lsdexception locked="false" priority="37" semihidden="true" unhidewhenused="true" name="bibliography"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" qformat="true" name="toc heading"/> <w:lsdexception locked="false" priority="41" name="plain table 1"/> <w:lsdexception locked="false" priority="42" name="plain table 2"/> <w:lsdexception locked="false" priority="43" name="plain table 3"/> <w:lsdexception locked="false" priority="44" name="plain table 4"/> <w:lsdexception locked="false" priority="45" name="plain table 5"/> <w:lsdexception locked="false" priority="40" name="grid table light"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light"/> <w:lsdexception locked="false" priority="47" name="grid table 2"/> <w:lsdexception locked="false" priority="48" name="grid table 3"/> <w:lsdexception locked="false" priority="49" name="grid table 4"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 1"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 1"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 1"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 1"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 1"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 1"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 1"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 2"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 2"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 2"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 2"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 2"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 2"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 2"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 3"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 3"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 3"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 3"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 3"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 3"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 3"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 4"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 4"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 4"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 4"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 4"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 4"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 4"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 5"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 5"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 5"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 5"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 5"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 5"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 5"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 6"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 6"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 6"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 6"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 6"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 6"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 6"/> <w:lsdexception locked="false" priority="46" name="list table 1 light"/> <w:lsdexception locked="false" priority="47" name="list table 2"/> <w:lsdexception locked="false" priority="48" name="list table 3"/> <w:lsdexception locked="false" priority="49" name="list table 4"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 1"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 1"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 1"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 1"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 1"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 1"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 1"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 2"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 2"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 2"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 2"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 2"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 2"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 2"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 3"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 3"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 3"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 3"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 3"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 3"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 3"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 4"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 4"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 4"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 4"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 4"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 4"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 4"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 5"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 5"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 5"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 5"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 5"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 5"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 5"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 6"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 6"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 6"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 6"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 6"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 6"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 6"/> </w:latentstyles> </xml><![endif]--> <style> <!- /* font definitions */ @font-face {font-family:"ms mincho"; panose-1:2 2 6 9 4 2 5 8 3 4; mso-font-alt:"ｍｓ 明朝"; mso-font-charset:128; mso-generic-font-family:roman; mso-font-pitch:fixed; mso-font-signature:1 134676480 16 0 131072 0;} @font-face {font-family:latha; panose-1:2 0 4 0 0 0 0 0 0 0; mso-font-charset:1; mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:262144 0 0 0 0 0;} @font-face {font-family:"cambria math"; panose-1:2 4 5 3 5 4 6 3 2 4; mso-font-charset:0; mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:-536870145 1107305727 0 0 415 0;} @font-face {font-family:calibri; panose-1:2 15 5 2 2 2 4 3 2 4; mso-font-charset:0; mso-generic-font-family:swiss; mso-font-pitch:variable; mso-font-signature:-536859905 -1073732485 9 0 511 0;} @font-face {font-family:tahoma; panose-1:2 11 6 4 3 5 4 4 2 4; mso-font-charset:0; mso-generic-font-family:swiss; mso-font-pitch:variable; mso-font-signature:-520081665 -1073717157 41 0 66047 0;} @font-face {font-family:"palatino linotype"; panose-1:2 4 5 2 5 5 5 3 3 4; mso-font-charset:0; mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:-536870265 1073741843 0 0 415 0;} @font-face {font-family:"\@ms mincho"; panose-1:2 2 6 9 4 2 5 8 3 4; mso-font-charset:128; mso-generic-font-family:roman; mso-font-format:other; mso-font-pitch:fixed; mso-font-signature:1 134676480 16 0 131072 0;} /* style definitions */ p.msonormal, li.msonormal, div.msonormal {mso-style-unhide:no; mso-style-qformat:yes; mso-style-parent:""; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca;} p.msocommenttext, li.msocommenttext, div.msocommenttext {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"comment text char"; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca;} p.msoheader, li.msoheader, div.msoheader {mso-style-priority:99; mso-style-link:"header char"; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; tab-stops:center 3.15in right 6.3in; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca;} p.msofooter, li.msofooter, div.msofooter {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"footer char"; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; tab-stops:center 3.15in right 6.3in; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca;} span.msocommentreference {mso-style-noshow:yes; mso-style-priority:99; mso-ansi-font-size:8.0pt; mso-bidi-font-size:8.0pt;} p.msobodytext, li.msobodytext, div.msobodytext {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"body text char"; margin:0in; margin-bottom:.0001pt; mso-pagination:none; mso-layout-grid-align:none; text-autospace:none; font-size:12.0pt; font-family:"arial",sans-serif; mso-fareast-font-family:calibri; mso-bidi-font-family:latha;} a:link, span.msohyperlink {mso-style-noshow:yes; mso-style-priority:99; color:blue; text-decoration:underline; text-underline:single;} a:visited, span.msohyperlinkfollowed {mso-style-noshow:yes; mso-style-priority:99; color:purple; mso-themecolor:followedhyperlink; text-decoration:underline; text-underline:single;} p.msocommentsubject, li.msocommentsubject, div.msocommentsubject {mso-style-noshow:yes; mso-style-priority:99; mso-style-parent:"comment text"; mso-style-link:"comment subject char"; mso-style-next:"comment text"; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca; font-weight:bold;} p.msoacetate, li.msoacetate, div.msoacetate {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"balloon text char"; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:8.0pt; font-family:"tahoma",sans-serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca;} p.msobibliography, li.msobibliography, div.msobibliography {mso-style-noshow:yes; mso-style-priority:37; mso-style-next:normal; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca;} p.msonormal0, li.msonormal0, div.msonormal0 {mso-style-name:msonormal; mso-style-unhide:no; mso-margin-top-alt:auto; margin-right:0in; mso-margin-bottom-alt:auto; margin-left:0in; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast;} span.commenttextchar {mso-style-name:"comment text char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"comment text"; mso-ansi-font-size:10.0pt; mso-bidi-font-size:10.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; mso-ansi-language:en-ca;} span.headerchar {mso-style-name:"header char"; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:header; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; mso-ansi-language:en-ca;} span.footerchar {mso-style-name:"footer char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:footer; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; mso-ansi-language:en-ca;} span.bodytextchar {mso-style-name:"body text char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"body text"; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"arial",sans-serif; mso-ascii-font-family:arial; mso-fareast-font-family:calibri; mso-hansi-font-family:arial; mso-bidi-font-family:latha; mso-ansi-language:en-us;} span.commentsubjectchar {mso-style-name:"comment subject char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-parent:"comment text char"; mso-style-link:"comment subject"; mso-ansi-font-size:10.0pt; mso-bidi-font-size:10.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; mso-ansi-language:en-ca; font-weight:bold;} span.balloontextchar {mso-style-name:"balloon text char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"balloon text"; mso-ansi-font-size:8.0pt; mso-bidi-font-size:8.0pt; font-family:"tahoma",sans-serif; mso-ascii-font-family:tahoma; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:tahoma; mso-bidi-font-family:tahoma; mso-ansi-language:en-ca;} p.default, li.default, div.default {mso-style-name:default; mso-style-unhide:no; mso-style-parent:""; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; mso-layout-grid-align:none; text-autospace:none; font-size:12.0pt; font-family:"palatino linotype",serif; mso-fareast-font-family:"ms mincho"; mso-bidi-font-family:"palatino linotype"; color:black; mso-ansi-language:en-ca; mso-fareast-language:ja;} span.normalchar1 {mso-style-name:normal__char1; mso-style-priority:99; mso-style-unhide:no; mso-style-parent:""; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman";} span.shorttext {mso-style-name:short_text; mso-style-priority:99; mso-style-unhide:no; mso-style-parent:""; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman";} span.heading2main {mso-style-name:"heading 2 main"; mso-style-unhide:no; mso-style-parent:""; mso-ansi-font-size:12.0pt; font-family:"arial",sans-serif; mso-ascii-font-family:arial; mso-fareast-font-family:arial; mso-hansi-font-family:arial; mso-bidi-font-family:arial; color:black; font-weight:bold; mso-bidi-font-weight:normal; font-style:normal; text-underline:black;} span.mention1 {mso-style-name:mention1; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; color:#2b579a; background:#e6e6e6;} span.unresolvedmention {mso-style-name:"unresolved mention"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; color:#605e5c; background:#e1dfdd;} .msochpdefault {mso-style-type:export-only; mso-default-props:yes; font-size:10.0pt; mso-ansi-font-size:10.0pt; mso-bidi-font-size:10.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:calibri; mso-fareast-theme-font:minor-latin; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; mso-ansi-language:de;} /* page definitions */ @page {mso-footnote-separator:url("es_goldstein_29516_files/header.htm") fs; mso-footnote-continuation-separator:url("es_goldstein_29516_files/header.htm") fcs; mso-endnote-separator:url("es_goldstein_29516_files/header.htm") es; mso-endnote-continuation-separator:url("es_goldstein_29516_files/header.htm") ecs;} @page wordsection1 {size:595.3pt 841.9pt; margin:70.85pt 70.85pt 56.7pt 70.85pt; mso-header-margin:35.4pt; mso-footer-margin:35.4pt; mso-columns:2 even 35.4pt; mso-header:url("es_goldstein_29516_files/header.htm") h1; mso-paper-source:0;} div.wordsection1 {page:wordsection1;} --> </style> <!--[if gte mso 10]> <style> /* style definitions */ table.msonormaltable {mso-style-name:"table normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; mso-ansi-language:de;} </style> <![endif]--><!--[if gte mso 9]><xml> <o:shapedefaults v:ext="edit" spidmax="1026"/> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext="edit"> <o:idmap v:ext="edit" data="1"/> </o:shapelayout></xml><![endif]--> </head> <body lang=en-us link=blue vlink=purple style='tab-interval:35.4pt'> <div class=wordsection1> <p class=msonormal><i style='mso-bidi-font-style:normal'><span lang=en-ca style='font-family:"palatino linotype",serif'>evidence summary<o:p></o:p></span></i></p> <p class=msonormal><i style='mso-bidi-font-style:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></i></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:11.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>data librarians’ skills and competencies are heterogeneous and cluster into those for generalists and specialists<o:p></o:p></span></b></p> <p class=msonormal><span lang=en-ca style='font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>a review of:<o:p></o:p></span></b></p> <p class=msonormal style='margin-left:35.3pt;text-indent:-35.3pt'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>federer, l. (2018). defining data librarianship: a survey of competencies, skills, and training. <i style='mso-bidi-font-style:normal'>journal of the medical library association</i> <i style='mso-bidi-font-style:normal'>106</i>(3), 294–303. <span class=msohyperlink><a href="https://doi.org/10.5195/jmla.2018.306">https://doi.org/10.5195/jmla.2018.306</a></span><br style='mso-special-character:line-break'> <![if !supportlinebreaknewline]><br style='mso-special-character:line-break'> <![endif]></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>reviewed by:<o:p></o:p></span></b></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>scott goldstein<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>web librarian<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>appalachian state university libraries<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>boone, north carolina, united states of america<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>email: <span class=msohyperlink><a href="mailto:goldsteinsl@appstate.edu">goldsteinsl@appstate.edu</a></span><o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>received: </span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>29 oct. 2018</span><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><span style='mso-tab-count:5'>                                                                 </span>accepted: </span></b><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>27 dec. 2018<b style='mso-bidi-font-weight:normal'><o:p></o:p></b></span></p> <p class=msonormal><span lang=en-ca style='font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:9.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:arial;mso-ansi-language:en-us;mso-no-proof:yes'><!--[if gte vml 1]><v:shapetype id="_x0000_t75" coordsize="21600,21600" o:spt="75" o:preferrelative="t" path="m@4@5l@4@11@9@11@9@5xe" filled="f" stroked="f"> <v:stroke joinstyle="miter"/> <v:formulas> <v:f eqn="if linedrawn pixellinewidth 0"/> <v:f eqn="sum @0 1 0"/> <v:f eqn="sum 0 0 @1"/> <v:f eqn="prod @2 1 2"/> <v:f eqn="prod @3 21600 pixelwidth"/> <v:f eqn="prod @3 21600 pixelheight"/> <v:f eqn="sum @0 0 1"/> <v:f eqn="prod @6 1 2"/> <v:f eqn="prod @7 21600 pixelwidth"/> <v:f eqn="sum @8 21600 0"/> <v:f eqn="prod @7 21600 pixelheight"/> <v:f eqn="sum @10 21600 0"/> </v:formulas> <v:path o:extrusionok="f" gradientshapeok="t" o:connecttype="rect"/> <o:lock v:ext="edit" aspectratio="t"/> </v:shapetype><v:shape id="picture_x0020_4" o:spid="_x0000_i1025" type="#_x0000_t75" alt="cc-ca_logo_xl" style='width:7.5pt;height:7.5pt;visibility:visible; mso-wrap-style:square'> <v:imagedata src="es_goldstein_29516_files/image003.png" o:title="cc-ca_logo_xl" cropright="49941f"/> </v:shape><![endif]--><![if !vml]><img border=0 width=10 height=10 src="es_goldstein_29516_files/image001.png" alt="cc-ca_logo_xl" v:shapes="picture_x0020_4"><![endif]></span><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:arial'><span style='mso-spacerun:yes'> </span>2019 </span><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif'>goldstein. this is an open access article distributed under the terms of the creative commons</span><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"cambria math"'>‐</span><span lang=en-ca style='font-size: 9.0pt;font-family:"palatino linotype",serif'>attribution</span><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"cambria math"'>‐</span><span lang=en-ca style='font-size: 9.0pt;font-family:"palatino linotype",serif'>noncommercial</span><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"cambria math"'>‐</span><span lang=en-ca style='font-size: 9.0pt;font-family:"palatino linotype",serif'>share alike license 4.0 international (<span class=msohyperlink><a href="http://creativecommons.org/licenses/by-nc-sa/4.0/">http://creativecommons.org/licenses/by-nc-sa/4.0/</a></span>), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.<o:p></o:p></span></p> <div style='mso-element:para-border-div;border:none;border-bottom:solid black 1.0pt; mso-border-bottom-alt:solid black .25pt;padding:0in 0in 1.0pt 0in'> <p class=msobodytext style='border:none;mso-border-bottom-alt:solid black .25pt; padding:0in;mso-padding-alt:0in 0in 1.0pt 0in'><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-ca'><o:p>&nbsp;</o:p></span></p> <p class=msobodytext style='border:none;mso-border-bottom-alt:solid black .25pt; padding:0in;mso-padding-alt:0in 0in 1.0pt 0in'><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-ca'><o:p>&nbsp;</o:p></span></p> <p class=msobodytext style='border:none;mso-border-bottom-alt:solid black .25pt; padding:0in;mso-padding-alt:0in 0in 1.0pt 0in'><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif;color:blue; mso-ansi-language:en-ca'>doi</span><span lang=en-ca style='font-size:9.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-ca'>: 10.18438/eblip29516<o:p></o:p></span></p> <p class=msobodytext style='border:none;mso-border-bottom-alt:solid black .25pt; padding:0in;mso-padding-alt:0in 0in 1.0pt 0in'><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-ca'><o:p>&nbsp;</o:p></span></p> </div> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>abstract<o:p></o:p></span></b></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>objective</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> – to better define the skills, knowledge, and competencies necessary to data librarianship.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>design</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> – electronic survey.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>setting</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> – unknown number of research institutions in english-speaking countries with a focus on north america.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>subjects</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> – unknown number of information professionals who follow data-related interest group electronic mail lists or discussions on twitter.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>methods</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> – author distributed an electronic survey via electronic mail lists and twitter to information professionals, particularly those in biomedicine and the sciences, who self-determined that they spend a significant portion of their work providing data services. the survey asked respondents to rate the importance of various skills and expertise that had been selected from a review of the literature. in addition to other quantitative analysis, author performed cluster analysis on the final dataset to detect subgroups of similar respondents.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>main results</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> – 82 valid responses were received. most respondents supported more than one academic discipline and spent at least half of their time on data-related work. competencies in the “personal attributes” category (such as interpersonal, written, and presentation skills) were rated as most important, while those in the “library skills” category were rated as least important. a cluster analysis detected two groups that could best be described as subject specialists and data generalists. subject specialists focus on a smaller number of disciplines and view a smaller number of tasks as important to their work compared to data generalists. in addition, data generalists are more likely to report spending most of their time on data-related work.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>conclusion</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> – data librarianship is a heterogeneous profession with many skillsets at play depending on the work environment, but the existence of two overarching subgroups – subject specialists and data generalists – deserves further study and may have implications for a number of stakeholders. hiring institutions may consider the breadth of their user population’s needs before recruitment. educational institutions as well as other on-the-job training opportunities may do well to focus more on “soft skills” as this is deemed more important by data librarians.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>commentary</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> – in the past decade, there has been growth in the number of libraries offering data services, defined as services to researchers in relation to managing data. examples of data services include data management guidance, data curation, and data visualization. whereas tenopir et al. (2012) found that only a small minority of u.s. and canadian libraries offer any sort of services, a recent content analysis of library websites by yoon and shultz (2017) has revealed over 180 schools with services in place, though to varying degrees. other scholarship has focused on the competencies required of librarians in data services roles, such as xia and wang’s (2014) analysis of social science data librarian job postings. the author’s survey contributes to the literature by asking self-designated data librarians in biomedical and scientific fields how these competencies and skills are actually utilized in practice.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>this summary relies on boynton and greenhalgh’s (2004) critical appraisal tool. two aspects of the paper are worth highlighting here. first, the author performs cluster analysis on the categorical survey data to group respondents into the categories of generalist and specialist. this is an innovative and welcome analytical technique in lis practitioner research. in addition, the data used in the analysis are openly available, well-documented, and reproducible. the questionnaire generally fares well against the critical appraisal tool. the items were developed in conjunction with a review of the literature and pilot tested. skills were rated on importance using a five-point likert scale ranging from “not at all important” to “absolutely essential” with an additional option for “don’t know or n/a.” the biggest issue was the sampling method, which relied on a convenience sample from electronic mail lists and social media. this limits the generalizability of the results, although the author mentions this as a limitation and generally refrains from making wide-sweeping claims in the paper. (differences between subject specialists and data generalists were tested with unpaired-samples t-tests and p-values were reported, but these should have been adjusted for multiple tests or else explicitly presented as exploratory.)<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>the study has unconventional implications for library science students and librarians looking to get into data librarianship. for students, it suggests they may be served more by developing soft skills and seeking discipline-specific skills rather than focusing on library-specific data management or curation courses. for librarians, on-the-job training and professional development opportunities, especially in a specific discipline if they wish to specialize, might prove more beneficial. strong comfort with self-education may be a highly valuable skill to develop and certainly one to promote during the hiring process. the study also highlights potential barriers to data librarians that deserve further study. for instance, an open-ended survey comment indicated that “our researchers have shown a strong bias towards working with ‘one of their own.’” it is unclear where this reluctance comes from, but it may be from a lack of awareness of the skills and competencies data librarians possess, suggesting that proactively demonstrating what they have done and can do to assist researchers would do much to reverse misconceptions.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>references<o:p></o:p></span></b></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:35.3pt;text-indent:-35.3pt'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>boynton, p. m., &amp; greenhalgh, t. (2004). selecting, designing, and developing your questionnaire. <i style='mso-bidi-font-style:normal'>bmj</i>, <i style='mso-bidi-font-style:normal'>328</i>(7451), 1312–1315. <span class=msohyperlink><a href="https://doi.org/10.1136/bmj.328.7451.1312">https://doi.org/10.1136/bmj.328.7451.1312</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:35.3pt;text-indent:-35.3pt'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:35.3pt;text-indent:-35.3pt'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>tenopir, c., birch, b., &amp; allard, s. (2012). academic libraries and research data services: current practices and plans for the future [white paper]. retrieved october 26, 2018, from association of college &amp; research libraries: <span class=msohyperlink><a href="http://www.ala.org/acrl/sites/ala.org.acrl/files/content/publications/whitepapers/tenopir_birch_allard.pdf">http://www.ala.org/acrl/sites/ala.org.acrl/files/content/publications/whitepapers/tenopir_birch_allard.pdf</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:35.3pt;text-indent:-35.3pt'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:35.3pt;text-indent:-35.3pt'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>xia, j., &amp; wang, m. (2014). competencies and responsibilities of social science data librarians: an analysis of job descriptions. <i style='mso-bidi-font-style:normal'>college &amp; research libraries</i>, 75(3), 362–388. <span class=msohyperlink><a href="https://doi.org/10.5860/crl13-435">https://doi.org/10.5860/crl13-435</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:35.3pt;text-indent:-35.3pt'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:35.3pt;text-indent:-35.3pt'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>yoon, a., &amp; schultz, t. (2017). research data management services in academic libraries in the us: a content analysis of libraries’ websites.<i style='mso-bidi-font-style:normal'> college &amp; research libraries</i>, 78(7), 920–933. <span class=msohyperlink><a href="https://doi.org/10.5860/crl.78.7.920">https://doi.org/10.5860/crl.78.7.920</a></span><o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> </div> </body> </html> commentary   in and out of the rabbit hole: unpacking the research proposal   marjorie mitchell research librarian university of british columbia okanagan library kelowna, british columbia, canada email: marjorie.mitchell@ubc.ca   received: 13 jan. 2017    accepted: 26 mar. 2017       2017 mitchell. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   introduction   the research proposal is almost a fairy tale document. and i don’t mean it is fantasy or beyond belief (although some probably are), but rather that, at the time of composing, a research proposal is not quite research, not quite fiction, and a whole lot of optimism about a happy ending. the happy ending is not guaranteed, either.   there are research proposals (ideas on the back of a napkin) and research proposals (a far more formal proposal for either a phd or large funding grant such as a sshrc grant). i think, for most of us here at the c-eblip fall symposium, specifically practitioner researchers, the research proposal will fall somewhere between these two extremes. i suspect that most research proposals written by practitioner researchers are shorter in length, possibly from 1 to 3 pages. for this presentation i am going to share with you the process i’ve been going through for my latest research project (working title – walking the walk: librarians sharing their research data).   composing research proposals remains one of the invisible or “inside the black box” parts of research. in an attempt to unpack and refine the process, i will briefly walk through the steps and missteps, including some highlights of conversations with people i consulted for advice, background readings, and pre-research that i have done.    the method i am using within this presentation is autoethnography. for those of you not familiar with this method a concise definition is qualitative research where the researcher/author utilizes a method of written self-reflection to connect and position their story and experiences to a broader cultural, social and political context (maréchal, 2010). briefly, i will use my experience writing a research proposal for a specific research project to illustrate some of the benefits and drawbacks of investing time, itself a valuable resource, in advance of undertaking a research project.   i learned about autoethnography as a research method from an anthropology – fine arts student who was writing her master’s thesis. she was examining her place as a metis artist – defining what it meant to be metis, how the work of metis artists was often not identified as such, and what that “erasure” meant – a very political as well as a very personal work. working with her led me to believe there were many more places where this method could be applied with fruitful results.   as i was looking at my way of creating a research proposal, i was also situating it within the body of practitioner-researcher scholarship, even more specifically within academic librarian practitioners. my work is not as political, nor as personal, but i do think it contributes to the small c cultural definition of librarian researchers.   my research process begins with an idea. the first concrete step i take with my idea is to conduct a little “pre-research” – a literature review that takes absolutely no more than 30 minutes. that amount of time seems to be enough for me to determine whether the idea has already been investigated and my question answered, or to determine whether i feel there is a gap in the literature. maybe the last work that was done on the question was so old as to no longer be relevant. maybe it was done in a setting that was not directly comparable to the one i was considering. maybe the idea is worth taking to the next step.   if an idea is “testable” outside the literature, i do a short test. sometimes this involves sending a couple of emails seeking information. maybe i ask a few people i know whether they know of information about my idea. i have found it pays to beware of “good ideas” – often many people have the same good idea at the same time and sometimes my ideas are not as fresh as i think they are. i have learned the hard way it is far better to spend a short amount of time checking out an idea – the quick and dirty lit review, followed by a very small sample to test an idea is far, far better than a few weeks (or more) developing a fully-fledged research proposal only to discover it has some fatal flaw – like it’s been done before, or someone else is currently doing it.   there is no single right way to write a research proposal. there is no one right way to create a research proposal, but there appears to be widespread support that research proposals are a useful tool (fain, 2013).  yesterday, i learned a new-to-me tool for creating research proposals – identifying institutional stakeholders, services librarians and the library could be using or offering to support the stakeholders, then figuring out what method would be appropriate to study that (henderson, 2016).   many things will influence what your research proposal will look like. don’t get hung up on the form of it, unless the form is important to the proposal’s purpose, such as a funding application where you will be judged not only on the intellectual content of your proposal, but also on your ability to write well and follow instructions. in those specific instances, follow the instructions closely. make it easier for the adjudicators to say yes to your proposal.   a really good question is why write a research proposal? why not just jump right in and do a full literature review, or start designing that questionnaire to circulate, or pull down the datasets from your ils? a research proposal gets the idea out of your head and into tangible form. as i mentioned, more formal research proposals – such as those you might submit for funding – have greater structure. however, all useful research proposals have a few things in common. without simply being a checklist, it can also provide you with a list of “to-do” items to nudge you forward at the times you might feel stuck. it may contain questions you will need to wrestle with as your work through an ethics proposal. it may help you define what your answer will look like so you will know when your research is complete. some other benefits to having a research proposal include having a document you could use to   enlist research partners get validation for your idea         find gaps in your plan begin to build a network of support for your research (colleagues send you literature you didn’t know existed.   this stage is one i call scary, because this is the point at which i really start to talk about my idea with other people. it’s one thing to write something down in the privacy of my personal notebook. it’s entirely something different to share that with others, even if the others are friendly, kind people who want to see me do well and want to help me.   there are some drawbacks to having a research proposal (really…):   use all your time creating the “perfect research proposal” be criticized for your idea                             fall into the comparison trap (their research proposal was better than mine).   there will always one more question that needs to be answered to “complete” the research proposal – one more cost that needs to be noted, one more timeline that needs to be fleshed out and so on until your idea becomes old and stale. a research proposal is a tool, not an end product. i believe having a research proposal is more useful than not having one only up to the point where it (the research proposal) has ceased to move your research forward.   if you don’t have a formal research partner at least have a trusted colleague who will help you do reality checks. believe it or not – trusted colleagues are the best support system for doing research. sometimes it is even better if your trusted colleague don’t “know” what you are researching so they can ask you the naïve, direct questions that you had glossed over or hadn’t thought about – yes, reality checks are hugely important. even if they aren’t a research partner, if you have a colleague who will call you on it when you go down the rabbit hole or on the wild goose chase or pick your favorite metaphor for getting sidetracked, then you are indeed a lucky researcher.   in closing, written research proposals can support the research process. i encourage you to try incorporating them into your practice. feel free to discard them if they don’t provide you with any value.   reference list   fain, j. a. (2013). reading, understanding, and applying nursing research (4th ed.). philadelphia: f.a. davis co.   henderson, m. (2016, october). practical research for librarians: making our research relevant. workshop, saskatoon, sk.   maréchal, g. (2010). autoethnography. in a. j. mills, g. durepos & e. wiebe (eds.), encyclopedia of case study research (vol. 2, pp. 43-45). thousand oaks, ca: sage publications.     microsoft word es_wilson.doc evidence based library and information practice 2007, 2:3  114 evidence based library and information practice       evidence summary    user studies differ across some disciplines and may not be very effective    a review of:  dervin, brenda and carrielynn d. reinhard. “researchers and practitioners talk about  users and each other. making user and audience studies matter‐‐paper 1.”  informationresearch 12.1 (oct. 2006). 6 aug. 2007 <http://informationr.net/ir/12‐ 1/paper286.html>.    reviewed by:   virginia wilson  shirp coordinator, health sciences library, university of saskatchewan  saskatoon, saskatchewan, canada  e‐mail: virginia.wilson@usask.ca        received: 02 june 2007     accepted: 22 july 2007      © 2007 wilson. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – as part of a multi‐staged project,  this study seeks to identify the unanswered  questions about users as found in three  fields: library and information science (lis),  human computer interaction, and  communication and media studies, as well  as the convergences and divergences across  these fields.    design – a multi‐phased, qualitative study  involving individual face‐to‐face and  telephone interviews, as well as self  interviewing and focus groups.    setting – the fields of lis, human computer  interaction, and communication and media  studies as examined in interview situations.    subjects – 83 international experts across   the three fields, as well as 31 local experts  from central ohio, usa.    methods – the majority of the 83  international experts in the fields of lis,  human computer interaction, and  communication and media studies were  interviewed by telephone (some in person).  thirty‐one local experts (7 public and 24  academic library directors) were  individually interviewed and also took part  in focus groups. the sense‐making  methodology was used as an interview  approach with its emphasis on bridging  gaps. neutral interview questions were used  to tease out the gaps in certain situations— in the case of this project, the gaps involve  communication and the unanswered  questions about users. brenda dervin  http://informationr.net/ir/12%e2%80%901/paper286.html%00%00 http://informationr.net/ir/12%e2%80%901/paper286.html%00%00 mailto:wilson@usask.ca http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:3  115 developed this approach, which has been  transformed and adapted by dervin and a  host of other lis researchers over the past 25  years. it is a metatheoretical approach that  has “evolved into a generalized  communication‐based methodology seen as  useful for the study of human sense‐making  (and sense‐unmaking) in any context”  (dervin 729).  the sense‐making metatheory  is implemented three ways in the method:  “in the framing of research questions; in the  designing of interviewing; and in the  analyzing and concluding processes of  research” (dervin 737). in the research  under review for this summary the answers  to the gap‐identifying questions allow  different disciplines to begin to  communicate and understand each other.   using sense‐making in focus groups  involves self interviewing (diaries, journals)  and group discussions.    interviews were transcribed using the  “smooth verbatim approach” in which non  fluencies such as repetition, hesitancies, and  partial words are eliminated. care was  taken to ensure anonymity, as this is  necessary in the first step of the sense‐ making approach. the transcripts were  analyzed for themes to capture a broad  picture of what the participants struggle  with across disciplinary and research‐ practice divides. analysis was carried out  by using comparative coding developed in  early grounded theory combined with the  sense‐making methodology’s emphasis on  gaps and bridging gaps. the “quotable  quote” was the unit of analysis, and  thematically representative quotes were  selected from the transcriptions.    main results – in an attempt to analyze  communication across and within  disciplines, the researchers did a thematic  analysis on the interviews conducted with  their international and local experts. the  thematic analysis found 12 major themes,  which included a total of 75 sub themes. the  12 major themes include the following:  participants wanted to make a difference  with their work; participants agreed that  current user research is not doing the job;  there are fundamental disagreements about  users and user studies; there are  fundamental disagreements about the  purposes of using user studies; there are  external forces that make carrying out and  applying user studies difficult; there was a  lengthy list of differing suggestions for  improving user studies; interdisciplinary  communication across the three fields that  do user studies is not effective; it was agreed  that interdisciplinary contact is difficult;  communication across the research/practice  divide is not going well; some participants  saw academic researchers as the problem,  while some participants viewed  practitioners as the problem; and most  participants agreed that contact across fields  and the research/practice divide would be  beneficial.    the researchers contend that this analysis is  one of many that could be done on the  information retrieved from the interviews.  their goal was not to find the definitive  answers, but to describe the difficulties that  participants are having across disciplines  and across the research/practice divide in  terms of communication and relating to user  studies. the researchers wanted to tease out  implications for communication and to  illustrate the multiplicity that they found.    conclusion – it is difficult for this study to  draw conclusions except in the most general  sense, as it is part of a larger, multi‐staged  research project. however, this study did  find that although participants across fields  wanted a synthesis, they also expressed  their inability to understand syntheses from  fields other than their own. there were  some who wanted more theories, while  some claimed there were too many theories  already. there was much criticism about  communication across disciplines, but few  evidence based library and information practice 2007, 2:3  116 solutions offered. the researchers can offer  up no “magic wands” as solutions for these  results, but they do suggest that the modes  of communication traditionally used in user  studies research are not working.    commentary    this paper is the first report of what will be  a 5‐stage research project. taken alone, the  paper offers up some interesting qualitative  revelations having to do with cross‐ disciplinary notions of the terms user,  audience, and user studies. the researchers  interviewed a total of 114 experts across the  three fields of lis, human computer  interaction, and communication and media  studies. the international experts included  participants from all three fields, while the  local expert pool was comprised of only  librarians. it is not made clear in the paper  why the local pool did not include members  of the other two disciplines, except perhaps  that the researchers were constrained by the  parameters of their research funding.     there were some limitations in the process  of compiling the experts, which the  researchers outline in the paper: of the 83  international experts, only 10 were not  living in the us; the researchers faced  budget constraints which hampered their  efforts to expand the interviews outside of  the us; and the sample favoured academics  – only 17% of the participants were, at the  time of the study, full‐time practitioners.  the researchers contend that the limitations  had little impact on the first stage of the  study because of its focus on  communication issues.    the qualitative procedures used in this  study are clearly described to the extent that  they can be in a paper of this length. the  researchers give enough information to  inform the reader, as well as for the reader  to follow up on the different methods if  desired. there is one gap in the description  of the qualitative approach taken, where the  researchers write that “standard quality  control procedures usually used in  qualitative research were applied” when  discussing the interview transcriptions (11  of 50). there is no elaboration on what these  quality control procedures entail.    the thematic analysis revealed disparate  notions of the value of user studies across  disciplines, mainly due to the fact that  different terminology is used – disciplines  often do not speak the same language. this  revelation is helpful for lis practitioners.  when searching for evidence across  disciplines in the area of user studies, it is  useful to know that different disciplines can  use the terminology differently. that does  not mean that cross‐disciplinary evidence  cannot be used; however an awareness of  the differences is necessary in order to fully  use the information. there was also the  trend of the experts not finding that user  studies are doing the job. this can be seen as  a call for different approaches to user  studies, and these approaches can be  informed by the multi‐disciplinary aspects  revealed by the research in this study.      work cited    dervin, brenda. “on studying information  seeking methodologically: the  implication of connecting metatheory  to method.” information processing  and management. 35.6 (nov. 1999):  727‐50.      evidence summary   low-level evidence suggests that perceived ability to evaluate and trust online health information is associated with low health literacy   a review of: diviani, n., van den putte, b., giani, s., & van weert, j. c. (2015). low health literacy and evaluation of online health information: a systematic review of the literature. journal of medical internet research, 17(5), e112. http://dx.doi.org/10.2196/jmir.4018   reviewed by: lindsay alcock head, health sciences library memorial university of newfoundland st. john’s, newfoundland, canada email: lalcock@mun.ca   received: 28 feb. 2016  accepted: 15 apr. 2016      2016 alcock. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to review, based on research evidence, the correlation between low health literacy and four outcomes of interest: (1) the ability to evaluate online health information based on (2) perceived reliability and accuracy, (3) trust in the internet as an information source, and (4) the application of established evaluation criteria.   design – systematic review and narrative synthesis.   setting –medline, psycinfo, web of science, cinahl, and communication and mass-media complete as well as articles discovered through the snowball method.   subjects – 38 studies identified through a systematic literature search.   methods – an exhaustive list of potential articles was gathered through searching five online databases and google scholar, and hand searching of references. inclusion and exclusion criteria were applied in a two-phase screening process in which two researchers participated to address reliability. data, including study characteristics and metadata, predictors, assessment methods, and outcomes, were extracted from relevant studies, and then synthesized narratively.    main results – following duplication removal 13,632 records were retrieved, 254 of which were identified for full-text assessment. thirty-eight studies met the eligibility criteria. all studies were non-experimental and therefore graded as a low level of evidence; 35 were cross-sectional designs, 1 a focus group, and 2 were observational studies. studies varied widely in population definition and sample size and were published between 2001 and 2013, primarily in north america.   overall, a positive association was identified between health literacy and outcomes related to the ability to evaluate or trust internet health information, while findings were inconsistent related to perceived quality of information and the application of evaluative criteria. four studies examined the impact of health literacy levels on one or more of the outcomes of interest.   the most prevalent outcome measure studied was trust in online health information, and the least prevalent was the use of evaluative criteria. the ability to trust online health information was assessed primarily through self-reporting, half of which utilized the ehealth literacy scale, the majority of which indicated a positive correlation between education level or low health literacy and the perceived or actual ability to evaluate online health information.   no studies on perceived information quality were found to utilize health literacy as an indicator. a positive association between educational level and trust in health information on the internet was reported in ten studies, while two articles noted a similar correlation based on proxies for health literacy, including english language proficiency and comprehension comfort. in terms of the use of evaluation criteria, only one study focused on health literacy, indicating that those with low health literacy evaluate online health information based on search result placement, celebrity endorsement, image quality, and site authorship, and that they trust university researchers more than government or religious authorities to provide health information.   no association was shown between readability or physician-provided online information and evaluation criteria while one study demonstrated that study participants with higher education tended to check author credentials more often when evaluating a website.   conclusion – effective and informed evaluation of online health information is impacted by low health literacy.   commentary   given the lack of a standard definition of low health literacy, the numerous markers for literacy levels and the multiple indicators related to internet evaluation, the authors have effectively synthesized the available research in this systematic review. there is no indication that the search strategy was peer reviewed using press, and it is questionable whether or not term/concept selection such as telemedicine and racial/ethnic group was appropriate. the authors note, however, that the search strategies were deliberately broadened to include all potential proxies and that irrelevant articles were eliminated during the screening phases. the article screening process is not standard for a systematic review and therefore reliability is somewhat limited. one reviewer performed title/abstract and full-text screening. a second reviewer assessed 10% of the abstracts and all full text articles meeting eligibility criteria. disagreements were resolved in consensus meetings rather than by a third reviewer.   recognizing that information evaluation is multi-faceted, the authors identified four components, or research questions, to investigate. the authors demonstrated informed consideration of low health literacy indicators by including proxies (e.g., general literacy, reading, or educational level) in the literature searches and inclusion/exclusion criteria thus ensuring comprehensiveness. while this approach strengthened the synthesis through the inclusion of closely related research it also served to decrease the homogeneity of the eligible studies. the inclusion of subgroup analysis from larger studies also reduces the strength of the review and external validity. that all included studies were non-interventional lowers the level of evidence and resulted in a narrative review, however that is logical given the subject area. the authors acknowledge these limitations of the study as well as the fact that sufficient data was not available to compare across different levels of health literacy and that self-reporting has inherent biases.   due to the heterogeneity of the studies, a synthesis of quantitative results was not possible. the results were therefore presented narratively and enhanced with five detailed tables identifying the study type, population description, sample size, health literacy predictor/proxy, measurement, and results based on each outcome of interest.   the benefits of this systematic review lie primarily in its identification of gaps in the literature and limitations of current research on this topic to provide solid evidence correlating literacy levels with online health information evaluation. suggestions for further research, as identified by the authors, aim to tie together the body of knowledge of information seeking behaviour, appropriate evaluation criteria, and the related impact of literacy levels thus allowing health information providers to best meet the needs of all health consumers.   microsoft word ebl101 evidence based library and information practice 2009, 4:1 51 evidence based library and information practice ebl 101 matching question types to study designs virginia wilson shirp coordinator, health sciences library university of saskatchewan saskatoon, saskatchewan, canada e-mail: virginia.wilson@usask.ca received: 31 jan 2009 accepted: 31 jan 2009 © 2009 wilson. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. in the last ebl 101 column, lorie kloda discussed asking the right questions, and outlined the formulation of an answerable question. in evidence based librarianship, the question is the foundation upon which everything else rests. the question needs to be focused enough to find precise evidence while taking into account the key concepts involved in the situation. however, before rushing off to begin searching the literature after constructing the best question ever, take a moment to think about the type of question you’ve formulated. the question type can point you in the direction of the study design best suited to answering your particular question. often, one’s prior level of knowledge of the topic will determine the type of question asked. first, decide if the question is a background question or a foreground question. a background question is one that is more general in nature and one that asks about fundamentals and facts. these types of questions might arise among novice practitioners, or among librarians who are encountering a new issue in the field for the first time. an example of a background question is, “what are the possible solutions to plagiarism in a post-secondary situation?” these types of questions may be answered by consulting a handbook, by conducting a literature review, or by eliciting thoughts and opinions from colleagues. by contrast, the foreground question presumes prior knowledge of the subject, and the practitioner usually has a couple of alternatives in mind from which to choose. when a foreground question is generated, one is most likely at the point of decision making. an example of a foreground question is, “among teen public library patrons, do after school study programs result in higher marks at school?” evidence based library and information practice 2009, 4:1 52 these types of questions are often answered by turning to the literature and finding relevant research studies (booth 62). but what type of research study should one look for? this is where the next grouping of question types comes into play. jonathan eldredge has written about question types and levels of evidence. he determined that there are three types of questions generated by library professionals: • prediction questions • intervention questions • exploration questions prediction questions typically predict an outcome under particular circumstances. an example of such a question would be, “are students who have attended information literacy sessions more likely to continue their studies?” these types of questions are often answered by using a cohort study; that is, a study that involves a defined population that is closely monitored over time to determine the outcome of being exposed to a particular phenomenon. andrew booth reported that prediction questions, and thus cohort studies, have investigated topics such as information resource use, outreach, education, and marketing (63). intervention questions are aimed at finding particular outcomes by comparing different actions (or interventions). these questions often compare an innovation to a traditional way of doing things. an intervention question might look like this: “do medical students learn searching skills more effectively from librarians or teaching faculty?” (eldredge 11). the classic research design for an intervention question is a randomized controlled trial (rtc). an rtc involves taking two similar groups and exposing them to the different actions; that is, one group is taught searching skills by librarians and one group is taught by teaching faculty. because the groups are similar in their make-up, any changes can most likely be attributed to the intervention. intervention questions could involve teaching, delivering a reference service, or maintaining a collection (booth 64). exploration questions closely resemble background questions. these questions typically ask or imply a “why” query. one example is, “why do non-library users not use their library?” qualitative research methods are best suited for the exploration question as these types of questions are more likely to be openended, and to have a need to explore ranges of behaviors and reasons for those behaviors. study designs include focus groups, ethnographic studies, observation, interviewing, and historical analysis (booth 65). now that you have decided what type of study design is best suited to your question, it is time to examine the research evidence. next time: looking to the literature—domains to help determine where to look. works cited booth, andrew. “formulating answerable questions.” evidence based practice: an information professional’s handbook. eds. andrew booth and anne brice. london: facet, 2004. 61-70. eldredge, jon. “evidence-based librarianship levels of evidence,” hypothesis 16.3 (2002): 10-14. evidence based library and information practice editorial responsibilities      2016. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     editor-in-chief: lorie kloda   associate editor (evidence summaries): heather pretty   associate editor (articles): rebekah (becky) willson, lisl zach   associate editor (classics): jonathan eldredge   associate editor (reviews): denise koufogiannakis   associate editor (features, commentaries, using evidence in practice, research in practice, news): lorie kloda   production editor: michelle dunaway   editorial intern: melissa griffiths   editorial advisors: lindsay alcock, alison brettle, katrine mallan, pam ryan   copyeditors: richard hayman (lead copyeditor), georgianne bordner, bryan chan, julie evener, heather healy, stacey penney, maria tan, mary virginia taylor, nikki tummon, alison yeoman   indexing support: meral alakus   microsoft word es_1791 evidence based library and information practice 2008, 3:3 64 evidence based library and information practice evidence summary the utilization of wireless handheld computers with medline is an effective mechanism for answering clinical questions at the point of care a review of: hauser, susan e., dina demner-fushman, joshua l. jacobs, susanne m. humphrey, glenn ford, and george r. thoma. “using wireless handheld computers to seek information at the point of care: an evaluation by clinicians.” journal of the american medical informatics association 14.6 (nov./dec. 2007): 807-15. reviewed by: martha ingrid preddie doctoral student, school of information and library science university of north carolina at chapel hill chapel hill, north carolina, united states of america email: mipreddie@gmail.com received: 02 june 2008 accepted: 31 july 2008 © 2008 preddie. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – to assess the effectiveness of wireless handheld computers (hhcs) for information retrieval in clinical environments and the role of medline in answering clinical questions at the point of care. design – a prospective single-cohort study. setting – teaching rounds in the intensive care units and general medicine wards in two hospitals associated with a university’s school of medicine in the united states. subjects – five internal medicine residents with training in evidence-based practice. methods – while accompanying medical teams on teaching rounds for approximately four consecutive weeks, each resident used md on tap (an application for handheld computers) on a treotm 650 pda/cell phone to find answers in real time, to questions that were raised by members of the medical teams. using a special version of md on tap, each resident initialized a userid. serving as evaluators, the residents described and categorized clinical scenarios and recognized questions. they also evidence based library and information practice 2008, 3:3 65 formulated search terms, searched medline and identified citations determined to be useful for answering the questions. an intermediate server collected details of all medline search query transactions, including system response time, the user (based on userids), citations selected for viewing, the saving of citations to hhc memory, as well as use of the linkout and notes features. in addition evaluators submitted daily summaries. these summaries included information on the scenarios, clinical questions, evidencebased methodology (ebm) category, the team member who was the source of the question, the pubmed identifiers (pmids) of relevant citations, and comments. at the end of the data collection period, each evaluator submitted a summary report consisting of a qualitative and quantitative evaluation of his experience using medline via the handheld device to find relevant evidence based information at the point of care. the report also focused on the usefulness of md on tap features, along with suggestions for additional features. data analysis encompassed matching the text of daily summaries to transaction records in order to identify sessions (containing a scenario, clinical question, one or more search queries, citation fetches and selected pmids). a senior medical librarian/expert indexer reviewed all the citations selected by evaluators and graded each citation as a (useful for answering the question), b (provided a partial answer) or c (not useful for answering the question). only those graded a were regarded as “relevant.” for the purpose of analysis a session was deemed to be successful “if at least one of the citations selected by the evaluator as relevant was also classified as relevant” (810) by the expert indexer. similarly, an individual query was successful “if at least one of the citations among the results of the query was relevant, that citation was viewed by the evaluator during rounds, and it addressed the clinical question as recorded in the daily summary” (810). various types of relationships were analyzed including the characteristics of clinical questions vis-a-vis successful sessions, search strategies in relation to successful queries, and the association between md on tap features and successful queries. sas/sudaan version 9.1 was used for statistical analysis. main results – evaluators answered 68% (246 of 363) clinical questions during rounding sessions. they identified 478 “relevant” citations, an average of 1.9 per successful session and 1.3 for each successful question. session lengths averaged 3 minutes and 41 seconds. characteristics of the evaluator (training, interest, experience and expertise) were a significant predictor of a session’s success. the significant determinants of query success were “the number of search terms that could be mapped to medical subject headings (mesh)” (812), the number of citations that were found for a query, and the use of md on tap’s auto-spellcheck feature. narrative comments from the evaluators indicated that using medline on a hhc at the point of care contributed positively to the practice of evidence -based medicine. conclusion – wireless handheld computers are useful for retrieving information in clinical environments. the application of several mesh terms in a query facilitates the retrieval of medline citations that provide answers to clinical questions. the md on tap program is a valuable interface to medline at the point of care. commentary this study investigates the usefulness of wireless handheld computers and the value of medline as a resource for finding evidence based library and information practice 2008, 3:3 66 answers to point of care clinical questions. the methodology employed was a cohort study of five participants. despite the small number of participants, the use of a prospective cohort study was suitable as the data collection method of choice. additionally, the application of the research methodology and the data analysis processes were clearly detailed, thus allowing for replication of the study. nevertheless, a major shortcoming of this research is that the study population was not totally representative of all eligible users of handheld computers. inclusion and exclusion criteria were not stated explicitly. however, the selected participants were affiliated with a clinical elective in medical informatics and were all residents who had been trained in evidence-based practice. it is likely that this training predisposed them to being more adept at constructing and executing ebm-type searches of databases. this can be surmised from the fact that their md on tap medline training averaged only 25 minutes. this prior training in evidencebased practice would have facilitated favourable results pertinent to the objectives of the study. this occurrence introduced some amount of bias into the study, and the possibility exists that the study outcomes could have been different if clinicians who were not residents, or physicians with longer years of practice who had not been trained in evidence-based practice, had also been included as evaluators. the researchers acknowledged that the small number of participants limited the strength of the study, since they were unable to statistically confirm the value of features such as the date limit and the clinical query hedge. they also admitted that the process of checking the primary literature to answer all questions (while on the rounds), produced an artificial situation that would have impacted the overall success rate. with regard to opportunities for further research, the researchers suggested an examination of the relationship between clinicians’ background and training and search success. also indicated is a comparative research study assessing the searching of identical questions using the same information databases on a handheld computer versus a desktop. given the inadequacies of the study emanating from the homogeneity of the study population and the small number of participants, both of which affect the external validity of this cohort study, a larger study with a more heterogeneous population is also warranted. this would render a higher level of generalizability. despite its shortcomings this study is significant because it “is the first reported analysis of online information searching by clinicians using wireless hccs in which search actions are automatically recorded and associated with both the questions being addressed and the perceived value of the search results” (814). the quantitative measures of search strategy and search success add tangible evidence to the literature on the use and value of wireless handheld computers, for information retrieval in clinical settings. the findings revealed that the evaluators’ “individual characteristics of training, experience, expertise or interest” (811) was the sole “significant predictor of a successful session” (811). these results suggest that medical librarians need to continue to engage in user training, to build competencies among clinicians in the formulation of effective search strategies for the retrieval of evidence-based information. evidence based library and information practice 2008, 3:3 67 with regard to the medline database, such training should highlight the use of multiple mesh terms in the construction of search queries. editorial   welcome new members of the editorial team   lorie kloda editor-in-chief associate university librarian, planning and community relations concordia university montreal, quebec, canada email: lorie.kloda@concordia.ca      2016 kloda. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   at the time the last issue of evidence based library and information practice (eblip) was published, we posted openings for two positions: an associate editor (reviews and classics) and the newly created communications officer. i’m pleased to say that both positions have now been filled. diane cooper, clinical informationist at the national institutes of health library, in maryland, us, has been appointed as the associate editor (reviews and classics). her term began this september. along with the rest of the editorial team as well as the editorial advisors, i look forward to working with diane and to see the continuation of two important sections of the journals.   classics have been an occasional but regular feature of the journal – every few issues includes one or more classics. this type of peer reviewed article is similar to an evidence summary as it summarizes and comments on a research publication, but is also distinct, as it focuses on an older, “classic” publication and has a demonstrated and ongoing impact on librarianship. this issue includes three classics – more than we have ever published in a single issue. i would like to thank jonathan eldredge for his role as associate editor (classics) over the past three years. jonathan has shepherded several classics through to publication, sourcing the original publications to be reviewed, and locating authors to write the articles. review articles, which are also peer reviewed, are newer for eblip. the review section began in 2012 in volume 7 of the journal, under the associate editorship of denise koufogiannakis, and several review articles have been published since then. i thank denise for her work in setting up the review articles section and overseeing it during the past few years.   our new communications officer is tatiana bryant, special collections librarian at the university of oregon, us. tatiana will have an important role not only in helping promote eblip but in supporting authors who publish in the journal in promoting their own work. she will make use of tradition communication channels as well as social media as part of her role. this is a new position on our editorial team and i have no doubt that tatiana will hit the ground running when she begins her appointment on october 1st.   we are fortunate, at eblip, to have so many eager volunteers who contribute their time, energy, and knowledge in the dissemination of evidence in support of the work of information professionals internationally.   evidence summary   differences between the perception and use of virtual reference services for complex questions   a review of: mawhinney, t., & hervieux, s. (2022). dissonance between perceptions and use of virtual reference methods. college & research libraries, 83(3), 503–525. https://doi.org/10.5860/crl.83.3.503     reviewed by: kathy grams associate professor of pharmacy practice massachusetts college of pharmacy and health sciences boston, massachusetts, united states of america email: kathy.grams@mcphs.edu   received: 22 aug. 2023                                                             accepted: 11 nov. 2023      2023 grams. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30426     abstract   objective – to investigate the differences that exist between the users’ perception of virtual reference tools (chat, email, and texting) and how these virtual reference tools are used.   design – multimodal research that includes a descriptive summary of user perspectives of virtual reference tools and a descriptive and correlation analysis of question categories (complexity, reference interview, question category, and instruction) compared to the type of virtual reference.   setting – a large university library in montréal, québec, canada.   subjects – a summary of in-person interview results from 14 virtual reference users and a sample of chat (250), email (250), and texting (250) transcripts.   methods – the authors describe their research as part of a larger project. in phase one, which was published in a previous report,1 the first author interviewed 14 users and collected their preferences among virtual reference tools and factors that impacted their use. participants were interviewed in fall 2019. they were eligible if they used one or more virtual methods. in phase two, the users’ perceptions among virtual reference tools were compared to the analysis of question complexity in a sample of chat, email, and texting transcripts. transcripts were collected from january 1, 2018, to december 31, 2019. text conversations were grouped as a single transcript. a total of 250 texts were collected and were matched in number with a random sample of chat and email transcripts; 750 transcripts were analyzed. the transcripts were coded by question type, question complexity, and the presence of reference interviews and instruction. the read scale was used to categorize questions by complexity and read 3 and above were deemed to be complex. a codebook was used for consistency and intercoder reliability. a random 10% of transcripts were coded by both authors with an agreement of 84%. after discussion, agreement reached 100%. the remaining 90% of the transcripts were coded by the first author. the chi-square test of independence (x2) was used to determine if there was a difference in the frequency of the delivery method in the categories analyzed. cramer’s v was used to determine the strength of associations.   main results – the authors state the main findings signify “dissonance between users’ perceptions of virtual reference methods and how they actually use them.” results from the user interviews suggest that participants felt that chat and texts should be used for basic questions and that email be used for more complex ones. they appreciated the quick answer from text for things such as library hours, and the back-and-forth nature of the chat for step-by-step instruction but did not believe these were suited for complex questions. participants expressed that an email to the library liaison rather than the library general email is the best for research questions. of note, library liaison emails were not collected as part of the virtual reference tools for this research project. the results from the transcript evaluation revealed that chat interactions were used for complex questions as reflected by the read scale rating. questions were categorized from read 1 (requiring the least amount of effort) to read 5 (requiring considerable effort and time) with the following results: read 1 0% chat, 0% email, 13% text; read 2 4% chat, 8% email, 43% text; read 3 72% chat, 75% email, 38% text; read 4 20% chat, 15% email, 6% text; and read 5 4% chat, 2% email, 0% text. the authors demonstrated a moderate strength of association between the delivery method and the read scale (v=0.41), reference interview (v=0.43), question category (v=0.34), and instruction (v=0.21). there were significant differences between the delivery method and complexity, p< 0.001. the email and chat transcripts were more complex than text and the chat transcripts were marginally more complex than email. chat transcripts were also more frequent in reference and instruction categories, p<0.001. the types of questions were divided into 10 categories: reference/ research, library systems, problem with access, interlibrary loan, known item, access policies, collection acquisitions, library physical facilities, hours, and other. the most popular question types for chat transcripts were reference/research questions (24%), library systems (17%), problem with access to e-resources (14%), interlibrary loans (14%), and known items (13%). the most popular question types for email were reference/research (18%), library systems 16%), problem with access (15%), and access policies (16%). the most popular for text transcripts were reference/ research (15%), library systems (18%), library physical facilities (18%), and hours (16%).   conclusion – mawhinney and hervieux establish that disagreement exists between the users’ perception of and the use of virtual reference services. after researching the types of questions and level of complexity associated with each virtual reference tool, the authors were able to provide a list of practical implications of their research to improve documentation and workflow and make suggestions for staffing needs. they recommend multiple reference methods, training on the reference interview and virtual methods chosen, advertising virtual resources, and making chat available on the website in places of research. they found that their institution had a high number of questions categorized as access policies and they suggested that easier ways to report problems be considered.   commentary   this research was appraised using the critical appraisal review form developed by letts (2007).   mawhinney and hervieux conducted a comprehensive literature review on the perception of virtual reference services. they describe a conflict in the literature regarding the use of chat as a virtual tool. chat has been reported as both an unsuitable tool for reference and research projects as well as an acceptable tool for all questions. chat exchanges have been reported to be more complex than email and also reported to be less complex. the justification for their research was clear, and the participant users and virtual tool samples were clearly described.    the authors discuss appropriate limitations to their research. they mentioned that one limitation to be considered was that transcripts of library liaison emails were not collected as part of their research project and conclude that they are likely rated as more complex on the read scale and suggest further investigation would be needed to confirm.   the authors include that this research was done at one institution and may not apply to others. they mention that the perception of question complexity, how users perceived their own questions, may vary among users. another limitation regarding transcript collection was described. in august 2019, mcgill library moved from using questionpoint to libchat. these two virtual reference services are different in delivery and in the way they account for text transactions. the authors discuss that they accounted for the differences by including an equal number of chats, emails, and texts from questionpoint and libchat.   the last limitation mentioned was that this study collected perceptions of virtual tools prior to the covid-19 pandemic. the authors mention that the use of all virtual tools increased during the pandemic, staffing virtual reference was reevaluated, and that there was a need to make virtual references more visible. what authors did not mention is that perceptions of virtual reference post covid-19 may have changed as well. users may have adapted.   the study aimed to investigate differences between users’ perception of and the use of chat, email, and texting as virtual reference tools and raises the possibility of other limitations.   mawhinney and hervieux (2022) describe mcgill university at the time of publication as a publicly funded institution with an enrollment of 40,000 students. participants were recruited through online and on-campus solicitation and described as both men and women; as undergraduate (5), masters (4) and doctoral (2) students, faculty (2), and alumni (1). the sample was described by the first author as being based on “theoretical saturation” where interviews were discontinued when the author did not gain “additional insights” from the interviews.1 questions were appropriate to elicit from the user how the question type influenced the choice of virtual reference. however, fourteen participants is a very small sample size and may not reflect the perceptions of users from an institution of this magnitude which is a potential bias.   potential bias is reflected in the analysis of text chat and email transcripts. not all questions were coded independently by two authors. the first author, who interviewed all participant users, coded 90% of the transcripts. although there was an 84% match in the first 10% of questions coded, with 100% agreement after discussion, it is possible there could be a slight change in the percent complexity per virtual reference tool in the overall results. this, however, is unlikely to change the overall message. a count and a measure of complexity also does not imply if the user obtained a complete answer, or if there was resolution of the problem after using the virtual tool. the authors state that transcripts were assessed for level of complexity, question category, and the presence of a reference interview and instruction. there is no description if the use of the virtual tool was successful. a user could potentially use text or chat for a complex question and then move to general library email or library liaison email because they did not receive a complete answer, or their issue was not resolved. this may be outside the scope of this research. mawhinney and hervieux provide suggestions that are useful to library practice that can help address this.   the authors suggest improved policies and workflows. they recommend librarians staff virtual references such as chat and general email, and that library assistants and/or students staff text reference. the authors suggest that due to the number of complex questions, more training is needed on the reference interview and methods of virtual reference, and that the user be made aware when a question needs to be transferred to a subject specialist. they support virtual reference tools be placed where users conduct library research.   librarians can have an impact on virtual reference services, including how they are used and where they are located, and how they are staffed to respond to complex questions.   references   letts, l., wilkins, s., law, m., stewart, d., bosch, j., & westmorland, m. (2007) critical review form – qualitative studies (version 2.0). retrieved from http://www.peelregion.ca/health/library/eidmtools/qualreview_version2_0.pdf   mawhinney, t. (2020). user preferences related to virtual reference services in an academic library. the journal of academic librarianship, 46(1), 102094. https://doi.org/10.1016/j.acalib.2019.102094   mawhinney, t., & hervieux, s. (2022). dissonance between perceptions and use of virtual reference methods. college & research libraries, 83(3), 503–525. https://doi.org/10.5860/crl.83.3.503       ebl 101   research methods: scoping studies   virginia wilson director, centre for evidence based library and information practice (c-eblip) university library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   originally published in: evidence based library and information practice, 9(4), 97–99. https://ejournals.library.ualberta.ca/index.php/eblip/article/view/23308/17776       received: 02 nov. 2014 accepted: 16 nov. 2014      2016 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   this time around, i’m going to take a look at scoping studies or scoping reviews. a scoping study consists of a fairly comprehensive search of the literature around a particular topic. when i came across this methodology, my first question was, “how are these different from systematic reviews?” which i’ve written about previously (wilson, 2013). as i looked deeper, i’ve discovered that a scoping study seems to sit somewhere between a literature review and a systematic review and is “one method among many that might be used to review literature” (arksey & o’malley, 2005, p. 20). this column will provide an overview of the scoping study methodology, some further reading on the subject, and some citations of examples of scoping studies in library and information studies.   but what exactly is a scoping study? a number of definitions have been put forward and several have been collected in a paper by levac, colquhoun, and o’brien (2010). for this column, i’m going to use the definition quoted by arskey and o’malley (2005): a scoping study aims “to map rapidly the key concepts underpinning a research area and the main sources and types of evidence available” (mays, roberts & popay, 2001). levac, colquhoun, and o’brien (2010) also include a definition from the canadian institutes of health research which states that scoping studies “are exploratory projects that systematically map the literature available on a topic, identifying the key concepts, theories, sources of evidence and gaps in the research” and are often “preliminary to full syntheses” (p. 2 of 9). the name of this methodology has been a bit confusing, as it has been referred to as “scoping study”, “scoping review”, “scoping literature review”, and “scoping exercise” in various studies. perhaps its relative newness as a defined methodology means that standard terminology has yet to be adopted.   in order to illustrate how a scoping study differs from a systematic review, i have put together a table (table 1).   the literature outlining and advancing the methodology of scoping studies is fairly recent. arksey and o’malley (2005) presented a framework that they adopted for the undertaking of a scoping study (p.22), to which i have added clarifying points:   stage 1: identifying the research question – this guides the development of search strategies stage 2: identifying relevant studies – the depth of this often depends of time and budget constraints stage 3: study selection – some search results will not be applicable to the research question stage 4: charting the data – material is sorted according to key issues and themes stage 5: collating, summarizing and reporting the results   an optional, although strongly recommended, stage 6 was also proposed, consisting of a consultation exercise “to inform and validate findings from the main scoping review” (p. 23). levac et al (2010) also recommend incorporating this stage, as it “adds methodological rigor and should be considered a required component” (p. 7 of 9).  stage 6 involves consultations with stakeholders who may be able to “provide additional references about potential studies to include in the review as well as valuable insights” about other issues pertinent to the review (arksey & o’malley, 2005, p. 29). in 2013, daudt, van mossel, and scott published an article further enhancing the methodology from the perspective of a large, inter-professional team’s experience using the original arksey and o’malley steps. they too agree that stage 6 should be a fully incorporated rather than optional step.   why might one conduct a scoping study? arskey & o’malley (2005) offer up four reasons:   1.       to examine the extent, range and nature of research activity... [and when] mapping fields of study 2.       to determine the value of undertaking a full systematic review 3.       to summarize and disseminate research findings... 4.       to identify research gaps in the existing literature... (pp. 21-22)     table 1 adapted from arksey & o’malley (2005) and grant & booth (2009) systematic reviews scoping studies focus on a well-defined research question address broader topics specific study designs can be identified prior to searching based on the question different study designs may be applicable and included attempt to provide answers from a narrow range of “quality assessed studies” (arksey & o’malley, p. 20) less likely to assess quality of studies included the goal is thorough, comprehensive searching how complete the searching is depends on time and scope constraints     a scoping study is a viable and useful methodology for a rapid scan of literature on a specific topic. i strongly recommend delving into the papers that i have consulted for this column; these are listed in the reference list. and, if you do conduct a scoping study, you might consider contributing your own experience to the literature as well.  to get a broad overview of the many review strategies you might explore, check out the article by grant and booth (2009).   the following are some examples of the scoping study methodology used in library and information studies.   baxter, g. j. & connolly, t.m. (2014). implementing web 2.0 tools in organisations: feasibility of a systematic approach. the learning organization, 21(1), 6-25. http://dx.doi.org/10.1108/tlo11-2012-0069 gardois, p., colombi, n., grillo, g., & villanacci, m.c. (2012). implementation of web 2.0 services in academic, medical and research libraries: a scoping review. health information and libraries journal, 29, 90-109. http://dx.doi.org/10.1111/j.1471-1842.2012.00984.x norwood, j. & skinner, b. (2012). implementing rfid in a hospital library: a scoping study. health information and libraries journal, 29(2), 162-165. http://dx.doi.org/10.1111/j.1471-1842.2012.00987.x younger, p. (2010). internet-based information-seeking behaviour amongst doctors and nurses: a short review of the literature. health information and libraries journal, 27(1), 2-10. http://dx.doi.org/10.1111/j.1471-1842.2010.00883.x   references   arksey, h. and o’malley, l. (2005). scoping studies: towards a methodological framework. international journal of social research methodologies, 8(1), 19-32. http://dx.doi.org/10.1080/1364557032000119616   daudt, h. m. l., van mossel, c., & scott, s. j. (2013). enhancing the scoping study methodology: a large, inter-professional team’s experience with arksey and o’malley’s framework. bmc medical research methodology, 13(48). http://dx.doi.org/10.1186/1471-2288-13-48   grant, m. j. & booth, a. (2009). a typology of reviews: an analysis of 14 review types and associated methodologies. health information and libraries journal, 26, 91-108. http://dx.doi.org/10.1111/j.1471-1842.2009.00848.x   levac, d., colquhoun, h., & o’brien, k. k. (2010). scoping studies: advancing the methodology. implementation science, 5(69). http://dx.doi.org/10.1186/1748-5908-5-69   mays. n., roberts, e., popay, j. (2001). synthesizing research evidence. in: fulup, n., allend, p., clarke, a., black, n. (eds). studying the organisation and delivery of health services: research methods (pp.188-220).
london: routledge.   wilson, v. (2013). research methods: systematic reviews. evidence based library and information practice, 8(3), 83-84. retrieved from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/20437/15740 ebl 101   research methods: interviews   virginia wilson liaison librarian murray library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   originally published in: evidence based library and information practice, 7(2), 96–98. https://ejournals.library.ualberta.ca/index.php/eblip/article/view/17196/14053     received: 08 may 2012  accepted: 20 may 2012      2016 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   last time we looked at focus groups as a qualitative research method. this time the focus shifts to interviews. sitting down face to face with a research participant and asking probing and insightful questions can be daunting—especially when feeling the pressure to be probing and insightful! however, if your research question dictates that interviews are the best method of data gathering, you need to dive in and get started. how do you know if interviewing is a good fit for your research? beck and manuel (2008) suggest that if you want to understand or explore finely shaded human issues, if your question seems best answered in prose rather than with numbers, and if you want to explore a trend or an experience looking for themes, then the interview is a good choice for data gathering (p. 82).   there are a few types of interviewing styles to choose from: structured interviews, semi-structured interviews, and unstructured interviews. as may be evident, structured interviews involve asking the same set of questions to each research participant. there is no room to move beyond the set of questions. this style of interview is often used in conducting surveys. in qualitative research, the structured interview is quite limiting. semi-structured interviews allow for more flexibility. they involve having a set of guiding questions that will keep the interview on track. however, the researcher can follow topics of interest during the interview without having to adhere to a structured set of questions. for unstructured interviews, the researcher will have an idea of the avenues he/she wants to explore, but the interview is more like a conversation—flexible and unrestricted. because the conversation can, and is expected, to go anywhere, comparing data between interviews becomes more difficult.   interviews can be conducted in person, over the telephone, or electronically using a program such as skype. the advantage to being face to face is the ability to see facial expressions and body language. these can be jotted down as notes as the interview progresses and may be useful in the data analysis portion of the research. conversely, interview participants may be shy or uncomfortable with a face to face conversation or unfamiliar with the technology. using the telephone or skype is cost effective, as no travel is involved, and may provide a certain level of comfort to participants. the factors must be weighed carefully in order to choose the best method of conducting the interviews.   beck and manuel (2008) break down interviewing as a research method into a series of steps:   identify participants. once participants are known to you, make sure you build enough time into the project to schedule the interviews. meeting everyone’s needs can be time consuming. decide on the type of interview (in person, telephone, etc.). decide on the facilities in which to conduct the interviews. this may be out of your control if you decide to meet the participants on their own turf. if you do travel to meet participants, be mindful of the potential risk involved. ensure someone knows where you are and consider developing a fieldwork policy if your organization does not have one. check and test your equipment. the last thing you need is a voice recorder with dead batteries. design an interview schedule that is well-paced and that includes topics to explore. obtain formal signed consent. pace the interview, giving adequate time for each question and response. follow proper protocol which is usually determined prior to beginning the research at the ethics approval stage (adapted from beck & manuel (2008), pp. 95-96).   one important thing to consider before undertaking interviews as a data gathering method is the different types of people you may encounter. these types can include dominant talkers, long-winded participants, the “expert” type, argumentative types, and the shy person. the interviewer needs to be prepared with techniques and tactics to move the interview along, quell a potential argument, or draw out a reticent participant.   there are many resources to help you get started in using interviews as a research method. here are just a few:   dilley, p. (2000). conducting successful interviews: tips for intrepid research. theory into practice, 39(3), 131-137. retrieved 21 may 2012 from http://www.jstor.org/stable/i264807   kvale, s., & brinkmann, s. (2009). interviews: learning the craft of qualitative research interviewing (2nd ed.). los angeles, ca: sage publications.   seidman, i. (2006). interviewing as qualitative research: a guide for researchers in education and the social sciences (3rd ed.). new york, ny: teachers college press.   and here are some examples of research that uses interviewing as a data gathering method:   duncan, v., & holtslander, l. (2012). utilizing grounded theory to explore the information seeking behavior of senior nursing students. journal of the medical library association,100(1), 20-27. doi:10.3163/1536-5050.100.1.005   valentine, b. (2001). the legitimate effort in research papers: student commitment versus faculty expectations. journal of academic librarianship, 27(2), 107-115. doi: 10.1016/s0099-1333(00)00182-8   major, j.a. (1993). mature librarians and the university faculty: factors contributing to librarians’ acceptance as colleagues. college and research libraries, 54(6),463-469. retrieved 21 may 2012 from eric database (ej473071).   there is not enough room in this column to delve into data analysis, and basically that depends on which theoretical approach you are taking, e.g., grounded theory. if you think that interviewing might be the way to go for your research, look to the literature and discuss your ideas with experienced colleagues. as well, if you are just beginning a research project using the interview method, you are not alone. i am, too! next time around, the topic will be bibliometrics.   references   beck, s. e., & manuel, k. (2008). practical research methods for librarians and information professionals. new york, ny: neal-schuman.   evidence summary   stem and non-stem library users have increased their use of e-books   a review of: carroll, a. j., corlett-rivera, k., hackman, t., & zou, j. (2016). e-book perceptions and use in stem and non-stem disciplines: a comparative follow-up study. portal: libraries and the academy, 16(1), 131-162. https://doi.org/10.1353/pla.2016.0002   reviewed by: stephanie krueger head, office of specialized academic services czech national library of technology prague, czech republic email: stephanie.krueger@techlib.cz   received: 1 mar. 2017    accepted: 17 apr. 2017      2017 krueger. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to compile a set of usability and collection development suggestions and to examine a possible statistical correlation between visiting the physical library, online resource use, and e-book use.   design – online questionnaire survey.   setting – major public research university in maryland, united states of america.   subjects – 47,209 faculty, students, and staff.   methods – this survey is a follow-up to a similar 2012 study at the same institution. survey respondents completed 14 multiple-choice and up to 8 open-ended questions about academic e-book discovery, perception, and usage patterns for both stem and non-stem respondents using the qualtrics online research platform. seven of eight open-ended questions were conditional (i.e., dependent on answers to multiple-choice questions), thus the number of questions answered by respondents could vary. the survey was available from october 1 to november 22, 2014, and promoted across a variety of communication channels (email, library website, social media, print flyers and handouts). incentives for completing the survey included one ipad mini and eight u.s. $25 amazon gift cards.   main results – 1,911 (820 stem and 1,091 non-stem) self-selected students, faculty, and staff from a total campus population of 47,209 faculty, students, and staff (4.2% response rate) participated in the survey, excluding 277 additional responses representing library personnel (70) and individuals not affiliated with the institution (207).   64% of respondents indicated more e-book use than three years before, with only 21.9% of respondents noting they never use e-books for academic purposes compared to 31% in 2012. 32.5% of respondents noted daily or weekly use of e-books for scholarly pursuits, with undergraduates reporting the most frequent use: 38.6% daily/weekly use versus 37.2% for graduate students, 16.2% for faculty, and 14.2% for staff. 38% of respondents reporting daily/weekly use were from stem disciplines; 31.3% were from non-stem fields.   computers, not e-readers, were the primary devices used for accessing e-books: 72.5% of respondents reported using laptops or desktops to this end versus tablets, 37.9%; mobile phones, 36.7%; kindles, 25.6%; nooks, 5.9%; and other e-readers, 3.3%. top “mixed device access” responses were tablet/mobile phone/computer (98 responses); mobile phone/computer (93 responses); and tablet/computer (81 responses).   the top three discovery tools respondents reported using for finding e-books were commercial sites (35.9%), free websites (26.8%), and the library website (26.2%). a weak-positive spearman’s rho rank correlation of 0.25 provides some evidence that respondents who visit the library often are likely to use online resources and e-books. 35% of respondents reported they use e-books online “most of the time,” and 67% of respondents indicated they print out e-book content for use. responses to the question “what, if anything, would make you more likely to use e-books for academic purposes?” included easier access via the library website (48% of respondents), better functionality for highlighting/annotating (44%), reduced cost (43.2%), easier downloading (38.5%), more e-books in area of research interest (37.3%), more textbooks (37.2%), and ownership of a dedicated e-reader (35.6%).   in 2012, 52% of respondents reported never having downloaded an e-book for offline use. this percentage dropped notably in this study, with only 11.5% of respondents indicating they had never downloaded for later use.   conclusion – while this study indicates both stem and non-stem respondents at this institution are increasingly using e-books, preferences for electronic versus print format varied according to content type and type of user (e.g., stem or non-stem, undergraduate or graduate, student/faculty/staff). key recommendations for usability and collection development include: improving discovery and awareness mechanisms, purchasing some content (e.g., references works, style guides) in e-format while ensuring multiple simultaneous use, taking advantage of print plus electronic options to serve users with different format preferences, and encouraging vendors to allow digital rights management free downloading and printing.   commentary   this study adds to the corpus of institutional surveys about academic e-book use. frame (2014) provided a review of such studies, including implications for collection development, while rayner and coyle (2016) highlighted more recent discussions in this area. cross-institutional, global surveys in this field of investigation, such as mckiel (2011), are still rare.   the local nature of this study and the low overall response rate (4.2%) limit generalization of findings, but it is still possible to compare themes identified here to those identified in other local surveys. because of this, the study shows “face validity” (bryman, 2012, p. 171). for example, raynor and coyle (2016) found – as in this study – significant online use as well as appreciation of easy downloading. mckiel (2011) identified discovery difficulties, also an important barrier to use in this study, with 47% of respondents in his study stating “i do not know how to find e-books” (p. 149). additional thematic comparisons across the e-book survey corpus should be addressed in future studies.   the 2012 predecessor survey could, in some ways, be considered to be a “pilot” questionnaire (boynton & greenhaigh, 2004, p. 1372) because investigators modified the 2014 instrument’s wording and scope to include stem respondents, focus on academic (not recreational) use, and solicit more feedback about format preferences. despite these minor modifications, the 2014 survey re-tested 2012 concepts, indicating reliability – at least for this particular institutional setting – by illustrating stability over time (bryman, 2012). in future surveys, wording of questions about discovery and specific e-book vendors might be further simplified in order to avoid local jargon and provide more generalizable results (bryman, 2012). mckiel (2011) provided examples of clear, simple wording for discovery-related terms.   while this study follows the boynton and greenhaigh (2004) questionnaire research precepts, key underlying details about the spearman’s rho correlation, including rank data, are not provided. a data table and/or scatterplot for rank data would enrich future studies (boynton & greenhaigh, 2004).   while investigators presented data tables highlighting demographic aspects of the study, tables showcasing unique aspects of the study, notably preferences for different content types, are not present. future studies might include format preference tables to facilitate easier comparison of responses across formats.    one of the most original contributions of this article is table 3, implications for collection development decision-making. here, investigators link interpretations of survey results to specific collection development actions. seeing how investigators interpreted survey findings to inform their actions might inspire other institutions to conduct their own local surveys using a same or similar instrument in order to confirm or dispute interpretations made by others. more critical analysis across institutions might, in turn, lead to better investigative cooperation across institutional and geographical boundaries and increase confidence in findings, as noted in suggestions for future research (p. 151).     references   boynton, p. m., & greenhaigh, t. (2004). hands-on guide to questionnaire research: selecting, designing, and developing your questionnaire. bmj, 328, 1372-1375. https://doi.org/10.1136/bmj.328.7451.1312   bryman, a. (2012). social research methods (4th ed.). new york, ny: oxford university press.   frame, r. m. (2014). a comparison of use between e-books acquired in bulk packages, individual titles, and user requested titles at a health sciences library (master’s thesis). retrieved from https://cdr.lib.unc.edu/   mckiel, a. (2011). 2011 global student e-book survey. proceedings of the charleston library conference, 143-154. http://doi.org/10.5703/1288284314890   rayner, s., & cole, d. (2016). books right here right now at the university of manchester library. insights, 29(2), 172–180. http://doi.org/10.1629/uksg.309     research article   experiences of visible minority librarians and students in canada from the vimloc mentorship program   yanli li business and economics librarian wilfrid laurier university library waterloo, ontario, canada email: yli@wlu.ca   valentina ly research librarian university of ottawa library ottawa, ontario, canada email: vly@uottawa.ca   xuemei li data services librarian york university library toronto, ontario, canada email: lixuemei@yorku.ca   received: 13 feb. 2023                                                               accepted: 21 june 2023      2023 li, ly, and li. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30325     abstract   objective – the purpose of this research is to examine the experiences of mentors and mentees in the formal mentorship program offered by the visible minority librarians of canada network (vimloc) from 2018-2022. findings from this research will help mentors and mentees understand how to establish an effective mentoring relationship. professional library associations and libraries can also gain valuable insights to support the visible minority library professionals within their own mentorship programs.   methods – between 2018 and 2022, 113 mentors and 145 mentees participated in four sessions of the vimloc mentorship program. the vimloc mentorship committee designed and delivered a survey for mentors and a survey for mentees at the end of each session. over four sessions, 81 mentors and 82 mentees completed the surveys, representing a 72% and 57% completion rate, respectively. fisher's exact tests were performed to examine if there were significant differences between mentors and mentees in their perceptions regarding ease of communication, relationship, helpfulness of mentorship, likeliness of keeping in contact, and importance of having a visible minority partner.   results – the mentees perceived mentoring support to be more helpful than the mentors perceived it themselves. the mentees were more likely to keep in contact with their mentors beyond the mentorship program while the mentors did not show as much interest. the mentees who had a positive experience from the formal mentorship program were found to be more likely to mentor others in the future, whereas the same effect did not hold true for the mentors. on the other hand, some findings were the same for both mentors and mentees. both stated that effective communication would facilitate a good mentoring relationship, which in turn, would lead to positive outcomes and greater likelihood of keeping in contact beyond the mentoring program. there was also consensus of opinion about the most important areas of mentoring support and some essential skills for building a successful mentoring relationship.    conclusion – this research contributes to the literature by using an empirical research method and comparative analyses of the experiences between mentors and mentees over four sessions of the vimloc mentorship program. the study focuses on the perceptions of participants regarding their communication, relationship, helpfulness of mentorship, associations between their past and present mentoring experiences, areas of support, importance of having a visible minority partner, and essential skills for building a successful mentoring relationship. mentors and mentees differed significantly in how they perceived the helpfulness of mentorship support and how likely they would like to maintain the ties beyond the program. for both sides, effective and easy communication was found to be critical for building a good mentoring relationship and achieving a satisfactory experience.     introduction   the purpose of this research is to examine the experiences of mentors and mentees who have participated in the formal mentorship program offered by the visible minority librarians of canada network (vimloc). this study focuses on the perceptions of participants regarding their communication, relationship, helpfulness of mentorship, associations between their past and present mentoring experiences, areas of support, importance of having a visible minority (vm) partner, and essential skills for building a successful mentoring relationship. there are limited quantitative studies that address formal mentoring relationships in the field of librarianship, and the existing literature primarily focuses on american settings (jordan, 2019). this empirical study compares the experiences of mentors and mentees, which is rarely seen in other mentorship research. findings from this paper will help mentors and mentees understand how to establish an effective mentoring relationship. professional library associations and libraries can also gain valuable insights to support the visible minority library professionals within their own mentorship programs, especially in regions where the populations are predominantly caucasian.   vimloc formed in 2012 with a mission to connect, engage, and support visible minority librarians in canada. the employment equity act defines visible minorities (vms) as “persons, other than aboriginal peoples, who are non-caucasian in race or non-white in colour” (government of canada, 2021). maha kumaran and heather cai (2015) conducted the first vimloc survey in 2013 which reported that vm librarians lacked mentorship and networking opportunities with other minorities in the country. accordingly, the vimloc mentorship program was inaugurated in 2013 and on an ongoing basis from 2013-2015 nine pairs were matched, becoming the first formal mentorship program for vm librarians to be mentored by vm librarians in canada (kumaran, 2013). many of the formal mentorship programs available for librarians require association fees, are limited to a small geographic area, or are tied with other limited opportunities like residency programs, which are mostly seen at american institutions (garrison, 2020; harper, 2020). one reason for establishing vimloc was to improve informal professional connections (majekodunmi, 2013); the mentorship program is a formal way of creating these networking opportunities. vimloc is free of charge and participation in the mentorship program is open to librarians and master of library and information science (mlis) students across canada who identify as vms.   after a hiatus, the vimloc mentorship program was reinitiated in 2018. the vimloc mentorship committee (referred to as “the committee”) recruited vm librarians at every career stage to be mentors and paired them with mentees. each session ran for two months (october-november in 2018, and may-june in 2020, 2021, 2022). the 2018 session occurred at the end of the year from november to december. after seeing the high demand for visible minorities to mentor each other, the committee found there was merit to continuing the mentorship program. they observed that 58% of the mentees were students and 40% of mentors worked in an academic library. the committee decided that offering a mentorship program from may to june would best suit academic work schedules and be ideal for students graduating from their programs. repeating the mentorship program in may-june 2019 would be too soon after the 2018 session closed, so the committee decided to postpone the next session until may-june 2020; the subsequent sessions in 2021 and 2022 followed the same timeline. a survey was sent separately to mentors and mentees at the end of each session. the survey responses were used to assess the mentorship program and provided rich information about the mentorship experiences of the participants. the data collected therein formed the basis of this research.   literature review   barriers for vms in librarianship   vms in the library profession can face additional invisible obstacles that their caucasian counterparts may not. many of these obstacles have been explained in the literature. for example, gohr (2017) identified that new library professionals looking for their first position in a competitive job market might have to take on unpaid work experiences to build up their cv. however, for vms, taking on unpaid work is considered a privilege and financial barrier that their caucasian counterparts might not face as often.    even after getting a library job, there are multiple descriptions of workplace barriers for vm librarians in the literature. a consistently cited barrier is that vm librarians need to assimilate themselves into a caucasian-dominated work culture (brown et al., 2018; gohr, 2017; lee & morfitt, 2020). the result, according to brown et al. (2018), is that vm librarians feel a pressure to police how they present themselves through their words, behaviour, and appearance, which can cause personal anguish. for instance, research found that 38.6% of vm academic librarians “did not feel free to speak their mind and express their views openly,” which adds to that stifling of their true selves (kandiuk, 2014, p. 510).   while fitting in is not the only option, vm librarians who do not conform may face more microaggressions in the workplace (brown et al., 2018). others described how this work situation could amplify feelings of imposter syndrome (farrell et al., 2017; lee & morfitt, 2020). brown et al. (2018), hathcock (2015), johnson (2007), and thornton (2001) all discussed the feeling of loneliness in the workplace, either from the physical isolation at work events or from the emotional isolation of pretending to fit in. hathcock (2015) further elaborated on the loneliness of pretending to fit into a structure of white librarianship and needing someone, like a role model or mentor, to show them how to do so. in thornton’s (2001) study of black female librarians in the us, she determined that many respondents felt some level of isolation and that was one source for low levels of job satisfaction. furthermore, 71% of black female librarians had experienced some level of racial discrimination at work, which compounded with feelings of isolation and negatively impacted morale. comparatively, in a canadian academic setting, kandiuk (2014) reported that only 42.8% of the vm academic librarians found that their colleagues were welcoming or somewhat welcoming to difference and diversity, 23.3% were not or somewhat treated with respect and as an equal member, and 16.1% felt they were not or only somewhat valued by work colleagues for their knowledge and work contributions. these seemingly common experiences of isolation for vms in libraries can lead to low morale and negative work environments that make them want to leave the job and/or the profession altogether (kendrick & damasco, 2019; olivas, 2014; thornton, 2001).    mentorship identified as a solution   when referencing all the barriers and negative experiences of being a culturally diverse library professional, a common solution proposed by vm librarians throughout the literature was mentorship. for example, in johnson’s (2016) interviews of academic librarians of colour, respondents indicated that a lack of mentorship was a barrier to their progress in their career and that any form of mentoring or networking was better than nothing. echoing that sentiment, in an open-ended question about how their workplace could support vm librarians in canadian academic institutions, the most common survey response was having mentorship and that “equity-related mentoring” opportunities to support vm librarians needed to be created (kandiuk, 2014). moreover, olivas (2014) found that many vm study participants needed to seek mentorship opportunities beyond their institution because what they received from their library was not adequate.   many studies focus on mentoring students before or after lis (library and information science) programs to increase the number of vm students and graduates, which in turn, will increase representation and diverse candidate pool. montiel-overall and littletree (2010) spoke of a specific program to recruit and ensure graduation of latino and native american in an lis program; one of the methods used to ensure high retention was mentorship. mccook and lippincott (1997) analyzed american statistics in the 1980s to 1990s to find that lis schools that graduated a higher number of diverse students employed mentoring as a recruitment strategy. the vimloc mentorship program also welcomed current mlis students who identified as vm to apply as mentees, providing them with an opportunity to build a professional network and access other resources that vimloc can provide.   mentorship benefits for vms   since mentorship has often been proposed as a solution to combat the barriers in the profession, the benefits of mentorship for vm librarians in these circumstances need to be elaborated. some of the more obvious benefits of vm mentorship listed in the literature are related to getting advice from experts with lived experience, refining their cv, and career planning and progression (alston, 2017; bonnette, 2004; boyd et al., 2017). however, considering the circumstances, there is great emphasis on emotional support that mentors have provided. for example, it could be “a shoulder to cry on, a relatable voice, and honesty” (alston, 2017, p. 159). it could also be social supports through “socialization into the profession” (boyd et al., 2017, p. 492), or “demystifying and interpreting library culture and politics and assisting the mentee with tips on how to work within the given institutional structure,” along with connecting them with people within their network (moore et al., 2008, p. 76). overall, this blend of practical advice, along with psychosocial support from mentors has been found to improve job satisfaction for vm librarians (alston, 2017).   benefits of vm mentors   oftentimes, vm mentees are paired with caucasian mentors due to the imbalance of representation within the profession (ford, 2018). many have noted that this can be problematic as vm mentees may feel the pressure to assimilate more into the dominant culture of the profession and that caucasian mentors cannot provide vm mentees with supports that a vm mentor could (brown et al., 2018; hathcock, 2015). while the effect of mentorship for vms is profound, it has been suggested in the literature that having vm mentors providing mentorship to vm mentees can amplify the benefit. for example, espinal et al. (2018) suggested how a mentoring program specifically for vm librarians would be beneficial to alleviate some of the pressure put on them in a white-dominated profession. for vm lis students that had vm mentors, they found that there was a “shared understanding of their experience” (hussey, 2006, p. 76) and that vm mentors can help them navigate through difficult situations based on their own lived experiences (hussey, 2006). from their national survey and qualitative semi-structured interviews of chinese american librarians, ruan and liu (2017) found that one of the major themes from respondents was mentorship. they revealed respondents’ praises for the chinese american librarians association (cala) mentorship program where chinese american librarians mentored each other. they noted that a benefit of vm mentors was providing assistance to overcome communication and other cultural barriers. similarly, moore et al. (2008) touched upon dealing with situations or issues that would be unique to vms in the profession and how the vm mentor could provide advice about how to deal with it effectively. furthermore, it has been noted that people from nonminority groups might not detect microaggressions the way a racialized person might, but vm mentors are able to validate situations of microaggressions and provide the vm mentee with reassurance of their experiences (alabi, 2015). anecdotally, cho (2014) examined his experiences with other asian librarian mentors. he found that having vm mentors benefitted him as it gave him the opportunity to share their experiences together and reflected upon them. not only did it benefit the vm mentees, a study of vm mentors at academic libraries found that mentoring new librarians was professionally rewarding and increased the vm librarian’s likelihood of pursuing managerial roles (bugg, 2016).   while there are many benefits to be gained from vm mentors, a negative aspect of vms mentoring vms is that there are so few mentors available, especially those with managerial experiences (hoffman, 2014; moore et al., 2008). thus, extra burden is placed on vm mentors, especially when mentorship does not count as credited work. for example, cooke and sánchez (2019) highlighted that mentorship was unpaid volunteer work and did not contribute towards tenure application for academic librarians. nevertheless, some vm librarians still took on the added task of mentorship to ensure the growth of diversity in the profession (harper, 2020; vanscoy & bright, 2017).   communication in a mentoring relationship   the literature about mentoring programs for vms in libraries focuses more on practical guidelines that highlight the importance of communication, which is pivotal to building and maintaining the relationship. for example, the guidance provided states that regular contact and showing concern allows the vm mentee to openly share and discuss personal concerns with candour (abdullahi, 1992; hernandez, 1994). likewise, in harrington and marshall (2014), the respondents from canadian academic libraries highly rated the importance of the following mentorship activities: sharing experiences, sharing confidential information, and actively listening to concerns. all these elements go into building a mentoring relationship through communication. anecdotally, olivas and ma (2009) described their positive mentoring relationship as vm librarians due to their “[c]lear communication with each other, on a continual basis” (p. 6) which they did through emails and phone calls to discuss their professional experiences.   communication issues in mentorship can occur due to cross-cultural and intergenerational differences, among many. amongst vm librarians there are many factors that can lead to a breakdown in communication, which becomes a challenge for the mentoring pair. for example, among the 17 challenges to mentorship listed in adekoya and fasae’s (2021) study of academic librarians in nigeria, “ineffective meetings, communication and feedback between the mentor and mentee” ranked fourth highest. in another study, a respondent identified issues with communication for the breakdown with their mentor, which created a negative mentorship experience (zhang et al., 2007).   methods   survey design and delivery   to gather information about the participants’ mentoring experiences and to help improve the vimloc mentorship program, the vimloc mentorship committee designed and delivered a survey for mentors and mentees separately at the end of each session. after ethics approvals from wilfrid laurier university and york university, online survey questionnaires were created using qualtrics xm. nine questions from the surveys used in the first 2013 vimloc mentorship session were slightly updated. an additional 15 new questions were included in the survey for mentees and mentors respectively, with the researchers referring to other studies to inform the new questions (goodsett & walsh, 2015; harrington & marshall, 2014). the mentorship committee also consulted other vimloc committee members for feedback. the survey links were sent to mentors and mentees via email. the surveys remained open for one month with a reminder email sent two weeks before the deadline. an informed consent letter was provided at the start of the survey, which indicated that the participants could choose not to participate, withdraw at any point, or skip any questions in the survey. all responses were collected anonymously.   during the mentorship program, when there was a shortage of mentors or when a mentor had the expertise that could benefit more than one mentee, one mentor could be approached to be matched with two mentees. as such, the questionnaires for mentors were slightly different depending on how many mentees were mentored by them. the survey for mentors who assisted one mentee contained six multiple-choice questions about communication, eight multiple-choice questions about interaction, three multiple-choice questions and one open-ended question about mentorship experience, and one multiple-choice question and three open-ended questions about mentorship program assessment (appendix a). in the survey for mentors who assisted two mentees, the questions about communication and interactions (q4-17 in appendix a) with the first mentee were repeated for the second mentee. the survey for mentees included six multiple-choice questions about communication, eight multiple-choice questions about interaction, two multiple-choice questions and one open-ended question about mentorship experience, and two multiple-choice questions and three open-ended questions about mentorship program assessment (appendix b).   data analysis   the survey questions were designed to allow comparison of experiences between mentors and mentees in the program. four questions (q13-16) from the mentor survey (appendix a) and the mentee survey (appendix b) were similar, with minor changes in the wording for the target audience. for instance, in the question about ease of communication, the mentors were asked “how easy was communication with your mentee?” this question was reworded for mentees to read “how easy was communication with your mentor?” similarly, this research compared the perceptions of mentors and mentees regarding how they felt about their relationship with their mentoring partner (mentoring relationship), how helpful the mentor was in assisting their mentee (helpfulness of mentorship), and how likely they would connect beyond the program (likeliness of keeping in contact). these indicators were all measured using a 5-point likert scale. for instance, the response options for the question about ease of communication included “very easy,” “easy,” “moderately easy,” “difficult,” and “very difficult.”   moreover, in-depth analyses were conducted regarding the associations between four items: ease of communication, mentoring relationship, helpfulness of mentorship, and likeliness of keeping in contact. this study also examined the associations between the respondents’ present and previous mentorship experience, and associations between their present experience and intentions to mentor in the future. these analyses were based on fisher's exact test, a statistical test used to determine whether there is a significant association between two categorical variables if 20% or more of the cells in the contingency tables have expected frequencies less than five. the freeman-halton extension of the fisher's exact test for more than 2 x 2 (two-row by two-column) contingency tables was employed (ibraheem & devine, 2013; kim, 2017). stata 13 was used for all data analyses. for the open-ended questions about the skills that are important for a successful relationship and the most satisfying aspect of the vimloc mentoring program, answers from respondents were coded inductively by one researcher, with several answers having multiple elements listed, thus given multiple codes. subsequently, some codes were aggregated due to similarity. another assessment of the responses and second review of the code assignments was completed. frequencies for each code were calculated for the mentor and mentee groups separately.     vimloc mentorship program participants and survey respondents   from 2018 to 2022, 113 mentors and 145 mentees participated in the vimloc mentorship program. amongst mentors, the largest three ethnicities were chinese (38%, n=43), south asian (17%, n=19), and black (10%, n=11). nearly 60% (n=64) were working at academic libraries, followed by public libraries (22%, n=25), and special libraries (17%, n=19). their experience as a librarian ranged from 2-29 years. of the 113 mentors, 24% (n=27) were in management positions. regarding mentees, chinese (21%, n=31), south asian (20%, n=29), and black (15%, n=22) were also the most prevalent minorities. over half the mentees (55%, n=80) were library school students. this mirrored the finding of harrington and marshall (2014), where library school students were found to expect mentorship opportunities more than librarians. those who had librarian experience for less than five years made up 26% (n=38). among the mentees, 9% (n=13) had a master's degree in librarianship from outside canada. it was noted that a mentor or mentee might engage in more than one mentoring session, and a mentee might also serve as a mentor either in different sessions or even within the same session. the counts and percentages presented above included all repeat participants. as shown in table 1, there were 73 participants in 2018, with the greatest number of mentees (n=48) across the four sessions, which reflected the pressing needs of visible minority mentees after a few gap years since the first vimloc mentorship program was launched in 2013. the number of participants experienced a dramatic drop in 2020 due to covid-19 and bounced back in 2021 and 2022.   table 1 vimloc mentorship program participants and survey respondents   year  program participants   survey respondents   survey completion rates mentors  mentees mentors  mentees mentors  mentees 2018 25 48 18 (16) 25 72% 52% 2020 24 26 16 (2) 20 67% 77% 2021 33 39 25 (8) 22 76% 56% 2022 31 32 22 (1) 15 71% 47% total 113 145 81 (27) 82 72% 57% note: the number in brackets indicates the number of mentors who assisted two mentees.   in total, 81 mentors and 82 mentees completed the surveys, representing a 72% and 57% completion rate, respectively. although more mentees participated in the mentorship program, they had a lower survey completion rate compared to mentors in all the years, except in 2020 (see table 1). we analyzed 190 responses, including 108 responses from mentors and 82 responses from mentees. all questions were optional, therefore the number of responses for individual questions varied.     results   mentoring support activities   the mentees were asked, “what aspects did your mentor help you with?” this question was reworded for the mentors to read, “what aspects did you help your mentee with?” each group was provided with a list of 20 mentoring support activities to choose whatever aspects they provided as mentors or received as mentees. both groups identified the same five areas of mentorship support as their top experiences: sharing professional experience, providing encouragement, providing career counselling, sharing experience about the day-to-day work life, and addressing professional concerns (figure 1).   figure 1 most reported five mentoring support activities.   the perspectives of mentors and mentees were separately analyzed regarding four items: ease of communication, mentoring relationship, helpfulness of mentorship, and likeliness of keeping in contact beyond the program. fisher's exact tests were further conducted to examine if mentors and mentees significantly differed in their perceptions of each aspect.   ease of communication   the mentoring pairs communicated in various ways: email, video chat, online chat/instant messaging, in-person, and telephone. email and video chat were reported as the most effective methods of communication. this could be due to the covid-19 pandemic that made in-person interactions difficult or impossible. the survey respondents were asked how easy their communication was with the mentoring partner. of the 107 responses from mentors, 82% (n=88) indicated “very easy” or “easy” compared to 88% (n=72) of the 82 responses from the mentees. meanwhile, 6% (n=6) of the mentor responses indicated “difficult” or “very difficult,” compared to 2% (n=2) of the mentee responses (figure 2). the fisher's exact test was performed to determine if the perspectives of mentors and mentees significantly differed. the test resulted in a p-value of 0.543, which is greater than the commonly used significance level of 0.05. based on this analysis, the perspectives of the two groups regarding the ease of communication were not significantly different.   figure 2 perceptions of ease of communication.     figure 3 perceptions of the mentoring relationship.   mentoring relationship   there were 106 responses from the mentors and 82 responses from the mentees that described their mentoring relationships. of these responses, 87% (n=71) of the mentees indicated that their mentoring relationship was “very good” or “good,” compared to 84% (n=88) of the mentors, while 13% (n=11) of the mentees described their relationship as “fair,” “poor,” or “very poor,” compared to 17% (n=18) of the mentors (figure 3). overall, it seemed that the mentees were more likely to report a positive relationship than the mentors. however, fisher's exact test result indicated non-significant differences between the two groups (p = .203).   helpfulness of mentorship   in the survey for mentees, they were asked how helpful the mentors had been in assisting them. in the survey for mentors, the same question was reworded as how helpful they had been in assisting their mentees. of the 107 responses from mentors, 81% (n=86) indicated “very helpful” or “helpful” compared to 92% (n=75) of the 81 responses from mentees. mentors (20%, n=21) were more likely to have a neutral attitude, feel their support was “unhelpful” or “very unhelpful,” compared to the mentees (7%, n=6) (figure 4). overall, the mentees showed a great appreciation for the mentoring support they received; by contrast, the mentors seemed to be modest when evaluating their own value in assisting their mentees. the fisher's exact test indicated a significant difference in perceptions between the two groups (p = .000).   figure 4 perceptions of the helpfulness of mentorship.   likeliness of keeping in contact   as the mentorship program ran for only two months, the vimloc mentorship committee suggested that mentors and mentees could continue to connect with each other even after the program ended if they would like. there were 107 responses from mentors and 80 responses from mentees submitted regarding how likely they would be to keep in contact with their mentoring partner. of these responses, 58% (n=46) of the mentees were “very likely” to keep the relationship, much higher than that of the mentors (30%, n=32). compared to the mentees, more mentors indicated “somewhat likely,” “neither likely nor unlikely,” “somewhat unlikely,” and “very unlikely” (figure 5). overall, it seemed that the mentees were more interested in maintaining ties with their mentors, while the mentors were not as enthusiastic to do the same. their perceptions were found to be significantly different based on the fisher's exact test (p = .002).   figure 5 perceptions of the likeliness of keeping in contact.   in addition, fisher's exact tests were conducted separately for mentors and mentees to examine the associations between the four items: ease of communication, mentoring relationship, helpfulness of mentorship, and likeliness of keeping in contact. statistically highly significant relationships were identified for both mentors and mentees between any two of the above four elements (all p = .000). these results suggested that those who felt it was easier to communicate with their mentoring partner were more likely to report a good relationship and to feel that the mentors were helpful in assisting mentees. the pairs who established a better relationship were more likely to feel that the mentoring support was helpful. easier communication, building a better relationship, and the feeling that the mentoring support was more helpful were all associated with a greater likelihood of keeping in contact beyond the mentorship program.   skills for building a successful mentoring relationship   when mentees were asked, “what skills do you think would be important to build a successful relationship with the mentor?,” there were 64 open-ended responses, while 62 mentors responded to their equivalent question that asked, “what skills do you think would be important to build a successful relationship with the mentee?” as presented in table 2, respondents described 24 important skills or elements, some identifying multiple elements within their response. for example, a mentor’s open-ended response was, “patience and listening skills to determine needs from mentee. sometimes they don’t know what they need to know until they feel comfortable enough to open up.” it was coded as “patience,” “listening,” and “openness.” the most common answer amongst both groups was communication (mentees n=28, mentors n=22). another common response that ranked highly amongst both groups was some form of open communication, sometimes characterized as being able to open up to the other (mentees n=13, mentors n=10). in addition, setting goals or expectations was important for the mentoring relationship, ranking within the top five most common responses for both groups, with mentees indicating 10 times and mentors indicating eight times. the rest of the responses had more fluctuation between mentors and mentees with regard to rating. honesty rated high amongst mentees (n=11), but only five mentors included it in their response, however, both ranked within the top 10 common responses. both flexibility or adaptability with scheduling meetings (mentees n=5, mentors n=6) and asking questions (mentees n=6, mentors n=5) ranked within the top 10 responses amongst both groups. in some cases, what mentors perceived as important, such as being a good listener (n=16), being empathetic (n=10), being knowledgeable (n=6), being respectful (n=5), and being encouraging (n=5) did not rank high with mentees. likewise, mentees perceived being organized (n=9), being friendly (n=6), being willing to learn (n=5), and the frequency of meetings (n=5) to be important when mentors did not rank them as highly. both groups barely mentioned the category of interpersonal skills, but they specified elements of it separately such as communication.   previous mentorship experiences and intentions to mentor   associations of past mentorship experience with vimloc mentorship experience   of the 81 mentors, 58% (n=47) had been mentored by an lis professional, formally or informally before joining the vimloc mentorship program. over one third (35%, n=28) engaged in a formal mentorship program for the first time. to examine if their previous mentorship experience would relate with their vimloc mentorship experience, fisher's exact tests were run to check the associations between having prior mentee experience and the following three variables, one at a time: ease of communication, mentoring relationship, and helpfulness of mentorship. as shown in table 3, no significant relationship was identified. having formal mentor experience before did not make a significant difference in their present experience either.   of the 81 mentees, 77% (n=62) indicated it was their first time participating in a formal mentorship program. having prior formal mentorship experience was not significantly related with their perceptions of ease of communication, mentoring relationship, and helpfulness of mentorship in the vimloc mentorship program.   associations of vimloc mentorship experience with intentions to mentor in the future   for mentors, feeling helpful in assisting their mentees in the vimloc mentorship program was not significantly associated with how likely they would be to mentor again in the future (table 3). for mentees, however, helpfulness from the vimloc mentorship experience would significantly increase the likelihood of becoming a mentor in the future (p = .002).   table 2 skills considered important for building a successful mentoring relationship response what skills do you think would be important to build a successful relationship with the mentee? mentors (n=62) what skills do you think would be important to build a successful relationship with the mentor? mentees (n=64) communication 22 28 listening 16 4 openness 10 13 empathetic 10 2 setting expectations/goals 8 10 knowledgeable 6 3 flexibility of scheduling 6 5 honesty 5 11 respectful 5 4 encouraging 5 2 asking questions 5 6 open-minded 4 3 curious 4 1 experienced 4 1 interpersonal skills 3 3 patience 3 0 friendly 3 6 timely responses 3 2 frequency of interactions 3 5 being organized/prepared 3 9 trust 2 3 positive attitude 2 4 willingness to learn 2 5 acting professionally 1 1   table 3 associations between past, present mentoring experience, and future intentions to mentor indicators mentors mentees having prior mentee experience & ease of communication p = .681 n/a having mentee experience & relationship p = .680 n/a having mentee experience & helpfulness of mentorship p = .451 n/a having formal mentorship experience & ease of communication p = .463 p = .399 having formal mentorship experience & relationship p = .485 p = .577 having formal mentorship experience & helpfulness of mentorship p = .427 p = .325 helpfulness of mentorship & going to mentor again in the future p = .063     n/a helpfulness of mentorship & going to mentor in the future n/a p = .002* note: p values from fisher's exact tests. * significant at 0.01 level   importance of having a visible minority partner   when asked how important it was to them that the mentoring partner was a visible minority, 75% (n=81) of the 108 responses from mentors versus 83% (n=68) of the 82 responses from mentees indicated “very important” or “important.” an equal share indicated “moderately important,” both at 9%. more mentors 14% (n=16) than mentees 8% (n=7) indicated “slightly important” or “not at all important” (figure 6). however, these differences in perspectives were not found to be significant in the fisher's exact test (p = .633).   figure 6 perceptions of the importance of having a visible minority partner.   most satisfying aspects of the vimloc mentorship program   both mentors (n=51) and mentees (n=63) shared comments about what was the most satisfying aspect of the vimloc mentorship program, many providing multiple aspects, which provided great insights into the unique contributions that vm mentors could make in supporting vm mentees. when coding the responses from the mentors, they predominantly described the satisfaction of helping someone (n=25). when the specific type of help was indicated, assistance with job hunting (n=4) and advice about career development (n=4) were the most common, closely followed by discussions about adjusting to the work culture (n=3) or helping the mentees towards their goals (n=3). many mentors remarked how they were able to connect with someone new to expand their network (n=18). another rewarding aspect that was noted was the ability to share their experiences with others (n=10). however, it was not just a one-way flow of experiences from mentors to mentees, as seven mentors noted that they also learned something from the experience. for example, one mentor mentioned that it was satisfying “[g]etting to understand the perspectives of other new visible minority librarians. hearing about their accomplishments, as well as their challenges in the current library landscape is eye opening.” less frequently mentioned, but still notable was that the mentors enjoyed providing the mentees with encouragement (n=6) and the ability to give back (n=3), meaning they had also received similar professional support and wanted to return the favor to someone else in the profession.   when mentees shared what they thought was the most satisfying aspect, the ability to network and make new connections with other library professionals was the most popular response (n=32). the second most common response was related to the help they received from the mentor (n=29). with most mentees being early on in their library career, they needed more specific help with job hunting (n=8), career guidance (n=4), and awareness of the work culture (n=4). mentees appreciated having their mentors share their experiences (n=19), with four saying that it made them feel less alone and three that described how their mentors’ experiences were something they could role model. touching upon many of these aspects, one mentee disclosed that they “loved being able to connect and hear from someone who was a visible minority in my desired profession. it can be easy to feel a bit alone in this caucasian-dominated field and i enjoyed hearing my mentor share their experiences and how they navigate the workplace.” five mentees expressed their satisfaction with the encouragement they received from their mentors and finally, three noted satisfaction with the mentorship program itself and the comfort “[k]nowing that something like that exists.”   within answers from both mentors (n=12) and mentees (n=18) there were notable mentions about being connected with someone who was also a visible minority and shared experiences as visible minorities within the library profession. for example, one mentor described that “[i]t’s a nice way to network with someone with similar experiences and have conversations about being a visible minority person in a workplace. i don’t see any other avenue where issues of being a visible minority could be discussed in a professional setting.” furthermore, a mentee elaborated that they “have a caucasian mentor through another program, who i cannot talk to about anything related to race/identity as a visible minority/immigrant. the most valuable part of the vimloc mentorship program was the opportunity to connect with a fellow visible minority and ask questions related to that.”   discussion   mentors and mentees indicated that email and video chat were the most effective methods of communication. this finding was in agreement with binder et al. (2022) in which mentors and mentees who interacted through web conferencing tended to report higher satisfaction of their mentoring experiences, but against the result of jordan (2019) indicating that skype or video chat was not as popular. the two groups also reported the same top five areas of support activities. likewise, harrington and marshall (2014) found these five aspects were important components of a mentoring relationship, which were categorized under career guidance, psychosocial support, and role model. this could be associated with the mentees’ career stages and their corresponding needs. more than half of the mentees in the vimloc mentorship program were library school students. their questions were more frequently about librarians’ professional work and general career preparation. in contrast, fewer questions were asked regarding promotion and tenure, as well as research and scholarship activities that librarians put on their professional development agenda years after entering the profession. moreover, the mentees did have concerns related to their vm identities. the mentors who had similar racial backgrounds and experiences navigating the caucasian-dominated library landscape could provide comfort, encouragement, and inspiration to the mentees.   the research findings revealed that mentors and mentees significantly differed in how they perceived the value of mentoring assistance. a higher proportion of mentees found mentorship was helpful while a higher percentage of mentors felt their assistance was unhelpful to mentees. the differences could be due to the fact that 55% of the mentees in the mentorship program were library school students and 26% were early career librarians. without entering the library profession, the student mentees typically needed guidance in career planning, job search, and interview process. the mentors were all librarians who have gone through the journey from library schools to the job market. they had the experience and capacity to assist the mentees, thus making the mentees feel that it was helpful. for the mentees who were early-career librarians, despite the anonymity of the surveys, they might have been cautious about giving negative feedback about someone in the profession, as they had not secured permanent positions yet.   the mentors and mentees also had significantly different perspectives regarding whether they would like to keep in contact after the mentorship session ended. mentees were more likely to maintain the ties than mentors. they might have different reasons for continuing the relationship beyond the program. despite the informal format, mentees could still benefit from the connection if the mentors were willing to continue providing advice and support. as vms are underrepresented in the library world, the mentors would be invaluable resources for the mentees to draw on in the future. for the mentors, some were willing to connect again because they could learn and grow themselves while supporting their mentees, or they found it personally rewarding to continue helping mentees to succeed. meanwhile, other commitments might hinder many mentors from continuing the relationship. one respondent commented:   i had a mentee with whom i had reviewed their resume and cover letter. i did not mind doing this a few times during the mentorship. they asked me to review multiple applications a few months after the mentorship ended and after our final wrap up meeting. by this time, i had to let them know that i could only review their application once more due to my other commitments.   the finding from this research suggested that positive experiences from the vimloc mentorship program would have a great impact on mentees’ intention to mentor in the future. this could be explained by the “spillover effect” derived from the benefits that the mentees received from the present mentoring relationship (ragins & scandura, 1999). the act of being mentored creates more mentors, which also seems to hold true according to the literature on vm mentorship. when early-career minority librarians were surveyed, they described their willingness to become a mentor in the future based on their positive experiences as a mentee (olivas & ma, 2009). in practice, vm librarians in higher education from johnson’s (2016) dissertation benefited from their mentee experience so that they actively sought out mentor opportunities “[i]n the spirit of service to others” (p. 100). this is similar to cho’s (2014) personal experience; he described the help he received from vm mentors at his institution when he was first starting his career and how he now participates in mentorship programs for vm librarians as a rewarding way to give back. comparably, boyd et al. (2017) found that residency programs could help develop library leadership skills in vm librarians and in turn, they would likely go on to recruit vms to the profession and mentor early career vm librarians.   this research has some implications for library associations and libraries seeking to support vm library professionals within their own mentorship program. first, as the findings indicated, the mentees who had positive experiences from the present mentoring relationship were more likely to mentor others. therefore, mentorship program managers need to take a proactive role in improving the mentee experience so that more mentees can become mentors to support others. this may strengthen the pipeline for greater recruitment and retention among vm librarians.   second, the research results showed that previous mentorship experience for both mentors and mentees did not make significant differences in how they communicated and built relationships in the present program. hence, regardless of their mentorship experience level, it is essential to give participants some guidance at the start of the mentorship session to ensure that they all are on the same page. based on our open-ended question about the skills that would be important to build a successful relationship with a mentee/mentor, future mentoring programs for vm librarians should focus on supporting communication skills, especially through encouraging people to open up. this is especially important with vm groups within the profession since it can ease some of the negative experiences from the workplace such as isolation and microaggressions. furthermore, they should ensure that mentors and mentees can clearly articulate their goals or expectations and formulate good questions to ask each other to gain a beneficial experience for the mentoring pairs.   third, various approaches could be taken to facilitate the mentoring relationship. the vimloc mentorship committee provided guidelines to each participant. however, due to the dynamic nature of each mentoring pair and their relationship, those guidelines could not be used to resolve some issues that occurred. the committee’s timely intervention was necessary when issues came up. examples of issues included: communication being lost or delayed due to scheduling difficulties from being in different time zones or being unable to fit the meetings into respective schedules; mentees not showing much interest in interacting, being uncommunicative, or not having clear expectations of the program; mentors feeling it was hard to build a relationship within two months, or being unable to answer questions outside their field of work. although the committee could not intervene in all problems; they took various approaches to help smooth things out when possible. for instance, they sent out two check-in emails (one for all the mentees and the other one for all the mentors) after the mentoring session was launched to make sure the pairs had connected with each other and started a conversation. they helped mentees to find an alternative mentor if the mentee did not find that the mentor was a good match. when there was not a good fit from the pool of mentors, an informal mentor was sought out and recommended to the mentee to connect with. when tensions occurred between a mentoring pair and they reached out to the committee, the committee members held one-on-one meetings with individuals in an attempt to gather more information, to understand their respective expectations and concerns, and to facilitate the mentoring pair’s mutual understanding and communication. it is important that “if a pairing is not compatible or causes harm, then allow the pair to disengage with dignity” (burke & tumbleson, 2019, p. 12), and reassign the mentee to a mentor with a better fit whenever possible (goodsett & walsh, 2015).     limitations   first, demographic information of the mentorship program participants, such as ethnicity, geographic location, work experience, and the type of library they work at, was gathered in the program application forms, however, such information was not collected again in the surveys. as the surveys were anonymous, there was no way of linking the survey respondents with their applications. hence, survey respondents’ mentorship experiences could not be examined based on their demographic characteristics.   second, as the completion rate of the mentees was 15% lower than that of the mentors, it was possible that some mentees who had negative experiences chose not to fill out the surveys, and as a result, their experiences might not be reflected in this research.   third, this research was based on the data gathered from the four mentoring sessions over 2018-2022. except in 2018, the mentoring sessions occurred during the covid-19 pandemic. this context could have affected the mentorship experiences of survey respondents. for instance, communication became more difficult and might have affected relationship building negatively due to the pandemic. further research efforts can seek to examine more sessions beyond the pandemic.   conclusion   there are numerous studies on mentorship within the library profession as a whole. research on mentorship for vm is increasing as mentoring support has been perceived to be beneficial to vm mentees individually and to the diversification of the library profession. this study contributed to the literature through an empirical research method and comparative analyses of the experiences between mentors and mentees in the vimloc mentorship program. statistically significant differences were identified between the two groups. mentoring support was perceived to be more helpful by the mentees than by the mentors. the mentees were more likely to keep in contact with their mentors beyond the mentorship program, while the mentors did not show as much interest. a positive experience in the present mentoring relationship would increase the intention of mentees to mentor others in the future, whereas the same effect did not hold true for the mentors. on the other hand, some findings were shared by both mentors and mentees, including the belief that effective communication would facilitate a good mentoring relationship, which in turn, would lead to positive outcomes and greater likelihood of keeping in contact beyond the mentoring program. mentor and mentee responses indicated they both agreed on the most important areas of mentoring support and some essential skills for building a successful mentoring relationship. in addition to contributing to the librarianship literature, the practical implications of this research are profound as there are very few mentorship programs characterized by vms mentoring vms. the experiences shared in this research will be helpful to library associations and libraries who are interested in operating a mentorship program for vm library professionals.   author contributions   yanli li: conceptualization, data curation, investigation, methodology, quantitative analysis, visualization, writing – original draft, writing – review & editing valentina ly: conceptualization, investigation, methodology, qualitative analysis, writing – original draft, writing – review & editing xuemei li: conceptualization, investigation, methodology, writing – original draft, writing – review & editing   references   abdullahi, i. (1992). recruitment and mentoring of minority students. journal of education for library and information science, 33(4), 307-310. https://doi.org/10.2307/40323194   adekoya, c. o. & fasae, j. k. (2021). mentorship in librarianship: meeting the needs, addressing the challenges. the bottom line, 34(1), 86-100. https://doi.org/10.1108/bl-09-2020-0063   alabi, j. (2015). racial microaggressions in academic libraries: results of a survey of minority and non-minority librarians. the journal of academic librarianship, 41(1), 47-53. https://doi.org/10.1016/j.acalib.2014.10.008   alston, j. k. (2017). causes of satisfaction and dissatisfaction for diversity resident librarians a mixed methods study using herzberg’s motivation-hygiene theory [doctoral dissertation, university of south carolina]. https://scholarcommons.sc.edu/etd/4080/   binder, a., o’neill, b., & willey, m. (2022). meeting a need: piloting a mentoring program for history librarians. library leadership & management, 36(2), 1-18. https://doi.org/10.5860/llm.v36i2.7524   bonnette, a. e. (2004). mentoring minority librarians up the career ladder. library administration & management, 18(3), 134-139. https://doi.org/10.5860/llm.v36i2.7524   boyd, a., blue, y., & im, s. (2017). evaluation of academic library residency programs in the united states for librarians of colour. college & research libraries, 78(4), 472-511. https://doi.org/10.5860/crl.78.4.472   brown, j., ferretti, j. a., leung, s., & méndez-brady, m. (2018). we here: speaking our truth. library trends, 67(1), 163-181. https://doi.org/10.1353/lib.2018.0031   bugg, k. (2016). the perceptions of people of colour in academic libraries concerning the relationship between retention and advancement as middle managers. journal of library administration, 56(4), 428-443. https://doi.org/10.1080/01930826.2015.1105076   burke, j. j., & tumbleson, b. e. (2019). mentoring in academic libraries. library leadership & management, 33(4), 1-19. https://doi.org/10.5860/llm.v33i4.7348   cho, a. (2014). diversity pathways in librarianship: some challenges faced and lessons learned as a canadian-born chinese male librarian. in d. lee & m. kumaran (eds.), aboriginal and visible minority librarians: oral histories from canada (pp. 89-102). scarecrow.   cooke, n. a., & sánchez, j. o. (2019). getting it on the record: faculty of colour in library and information science. journal of education for library and information science, 60(3), 169-181. https://doi.org/10.3138/jelis.60.3.01   espinal, i., sutherland, t., & roh, c. (2018). a holistic approach for inclusive librarianship: decentering whiteness in our profession. library trends, 67(1), 147-162. https://doi.org/10.1353/lib.2018.0030   farrell, b., alabi, j., whaley, p., & jenda, c. (2017). addressing psychosocial factors with library mentoring. portal: libraries and the academy, 17(1), 51-69. https://doi.org/10.1353/pla.2017.0004   ford, a. (2018, november 1). underrepresented, underemployed: in the library-job search, some face special barriers. american libraries magazine. https://americanlibrariesmagazine.org/2018/11/01/underrepresented-underemployed/   garrison, j. (2020). academic library residency programs and diversity. portal: libraries and the academy, 20(3), 405-409. https://doi.org/10.1353/pla.2020.0020   gohr, m. (2017). ethnic and racial diversity in libraries: how white allies can support arguments for decolonization. journal of radical librarianship, 3, 42-58. https://journal.radicallibrarianship.org/index.php/journal/article/view/5   goodsett, m., & walsh, a. (2015). building a strong foundation: mentoring programs for novice tenure-track librarians in academic libraries. college & research libraries, 76(7), 914-933. https://doi.org/10.5860/crl.76.7.914   government of canada. (2021). employment equity act. https://laws-lois.justice.gc.ca/eng/acts/e-5.401/page-1.html#h-215195   harper, l. m. (2020). recruitment and retention strategies of lis students and professionals from underrepresented groups in the united states. library management, 41(2/3), 67-77. https://doi.org/10.1108/lm-07-2019-0044   harrington, m. r., & marshall, e. (2014). analyses of mentoring expectations, activities, and support in canadian academic libraries. college & research libraries, 75(6), 763-790. https://doi.org/10.5860/crl.75.6.763   hathcock, a. (2015, october 7). white librarianship in blackface: diversity initiatives in lis. in the library with the lead pipe. https://www.inthelibrarywiththeleadpipe.org/2015/lis-diversity/   hernandez, m. n. (1994). mentoring, networking and supervision: parallelogram, vortex, or merging point? in d. a. curry, s. g. blandey & s. m. martin (eds.) racial and ethnic diversity in academic libraries: multicultural issues (pp. 15-22). the haworth press.   hoffman, s. (2014). mentoring diverse leaders in academic libraries. in b. l. eden & j. c. fagan (eds.), leadership in academic libraries today: connecting theory to practice (pp. 77-91). rowman & littlefield.   hussey, l. k. (2006). why librarianship? an exploration of the motivations of ethnic minorities to choose library and information science as a career. in d. e. williams, j. m. nyce & j. g. williams (eds.) advances in library administration and organization (pp. 153-217). emerald group publishing limited. https://doi.org/10.1108/s0732-0671(2009)0000028007   ibraheem, a. i., & devine, c. (2013). a survey of the experiences of african librarians in american academic libraries. college & research libraries, 74(3), 288-306. https://doi.org/10.5860/crl-292   johnson, p. (2007). retaining and advancing librarians of colour. college & research libraries, 68(5), 405-417. https://doi.org/10.5860/crl.68.5.405   johnson, k. (2016). minority librarians in higher education: a critical race theory analysis [doctoral dissertation, marshall university]. https://mds.marshall.edu/etd/1019   jordan, a. (2019). an examination of formal mentoring relationships in librarianship. the journal of academic librarianship, 45(6), 1-9. https://doi.org/10.1016/j.acalib.2019.102068   kandiuk, m. (2014). promoting racial and ethnic diversity among canadian academic librarians. college & research libraries, 75(4), 492-556. https://doi.org/10.5860/crl.75.4.492   kendrick, k. d., & damasco, i. t. (2019). low morale in ethnic and racial minority academic librarians: an experiential study. library trends, 68(2), 174-212. https://doi.org/10.1353/lib.2019.0036   kim h. y. (2017). statistical notes for clinical researchers: chi-squared test and fisher's exact test. restorative dentistry & endodontics, 42(2), 152-155. https://doi.org/10.5395/rde.2017.42.2.152   kumaran, m. (2013). visible minority librarians of canada at ontario library association super conference, 2013. partnership: the canadian journal of library and information practice and research, 8(2), 1-9. https://doi.org/10.21083/partnership.v8i2.2888   kumaran, m., & cai, h. (2015). identifying the visible minority librarians in canada: a national survey. evidence based library and information practice, 10(2), 108-126. https://doi.org/10.18438/b8zc88   lee, e., & morfitt, p. (2020). imposter syndrome, women in technical services, and minority librarians: the shared experience of two librarians of colour. technical services quarterly, 37(2), 136-147. https://doi.org/10.1080/07317131.2020.1728125   majekodunmi, n. (2013). diversity in libraries: the case for the visible minority librarians of canada (vimloc) network. feliciter, 59(1), 31-33. https://yorkspace.library.yorku.ca/xmlui/handle/10315/35701   mccook, k., & lippincott, k. (1997). planning for a diverse workforce in library and information science professions. https://files.eric.ed.gov/fulltext/ed402948.pdf   montiel-overall, p., & littletree, s. (2010). knowledge river: a case study of a library and information science program focusing on latino and native american perspectives. library trends, 59 (1), 67-87. https://core.ac.uk/download/pdf/4827147.pdf   moore, a. a., miller, m. j., pitchford, v. j., & hwey jeng, l. (2008). mentoring in the millennium: new views, climate and actions. new library world, 109(1/2), 75-86. https://doi.org/10.1108/03074800810846029   olivas, a. p. (2014). understanding underrepresented minority academic librarians’ motivation to lead in higher education [doctoral dissertation, university of california, san diego]. https://escholarship.org/uc/item/5gq790c0   olivas, a., & ma, r. (2009). increasing retention rates in minority librarians through mentoring. the electronic journal of academic and special librarianship, 10(3), 1-9. https://digitalcommons.unl.edu/cgi/viewcontent.cgi?article=1119&context=ejasljournal   ragins, b. r., & scandura, t. a. (1999). burden or blessing? expected costs and benefits of being a mentor. journal of organizational behaviour, 20(4), 493-509. http://www.jstor.org/stable/3100386   ruan, l., & liu, w. (2017). the role of chinese american librarians in library and information science diversity. international journal of librarianship, 2(2), 18-36. https://doi.org/10.23974/ijol.2017.vol2.2.39   thornton, j. k. (2001). african american female librarians. journal of library administration, 33(1–2), 141-164. https://doi.org/10.1300/j111v33n01_10   vanscoy, a. & bright, k. (2017). including the voices of librarians of colour in reference and information services research. reference & user services quarterly, 57(2), 104-114. https://journals.ala.org/index.php/rusq/article/view/6527   zhang, s. l., deyoe, n., & matveyeva, s. j. (2007). from scratch: developing an effective mentoring program. chinese librarianship: an international electronic journal, 24, 1-16. http://www.white-clouds.com/iclc/cliej/cl24zdm.pdf   appendix a survey questionnaire 2022 for mentors (with one mentee)   q1 thank you for participating in this survey! we would like to hear your feedback on the 2022 vimloc mentorship program so that we can continue to improve it. please look over the informed consent letter before proceeding. o i have read and understand the above information. i agree to participate in this study. o i have read and understand the above information. i do not want to participate in this study. q2 how did you hear about the vimloc mentorship program? (please select all that apply) o vimloc group o school o conference o colleague o social media (e.g. facebook, twitter, linkedin, etc.) o friend o contacted by the vimloc mentorship committee o other  q3 what are your reasons for participating as a mentor in the vimloc mentorship program? (please select all that apply) o  professional development o  promotion o  meeting requirements for tenure o  passion for helping others o  networking o  sharing experience o  other communication:   how many times have you used the methods below to communicate with your mentee:  q4 e-mail (you sent to the mentee):  0, 1, 2 …9, 10 and above  q5 online chat/instant messenger: 0, 1, 2 …9, 10 and above  q6 skype or video chat: 0, 1, 2 …9, 10 and above  q7 in person: 0, 1, 2 …9, 10 and above  q8 telephone: 0, 1, 2 …9, 10 and above  q9 what was the most effective method of communication? o  email o  online chat/instant messenger o  skype or video chat o  in person o  telephone o  other (please specify) ____________ interactions: q10 in your early contacts with your mentee, did they discuss their mentorship program expectations with you? o yes o no  q11 how important was it to you that the mentee was a visible minority? o very important o important o moderately important o slightly important o not at all important  q12 did you discuss any experiences about being a visible minority in the profession with your mentee? o yes o no  q13 how easy was communication with your mentee? o very easy o easy o moderately easy o difficult o very difficult  q14 how would you describe your relationship with your mentee? o very good o good o fair o poor o very poor  q15 how likely are you to keep in contact with your mentee after the program ends? o very likely o somewhat likely o neither likely nor unlikely o somewhat unlikely o very unlikely  q16 how helpful do you think you were in assisting the mentee? o very helpful o helpful o neither helpful nor unhelpful o unhelpful o very unhelpful  q17 what aspects did you help your mentee with? (check all that apply) o  provide encouragement o  provide career counseling o  help with job seeking skills (cover letter, resume, interview, etc.) o  assist with networking o  help with setting mentee's professional goals o  share own professional experience with the mentee o  share experience about the day to day work life o  help with orientation to library culture and workplace expectations o  advise on how to adapt in an organization as a visible minority o  address the mentee's professional concerns o  provide knowledge of a discipline or subject area o  assist with research and scholarship (grant writing, research methods, etc.) o  assist with promotion and tenure (preparation of materials, procedure, criteria, etc.) o  share experience or improve skills in instruction o  share experience or improve skills in collection management o  share experience or improve skills in reference services o  share experience or improve skills in leadership o  share experience or improve skills in community involvement or outreach o  share experience or improve skills in technology-related library work o  other mentorship experience: q18 in the past, were you ever mentored by an lis professional, formally or informally? o yes o no q19 is this your first experience as a mentor through a formal mentorship program? o yes o no  q20 based on this experience, how likely would you be to mentor again in the future? o very likely o somewhat likely o neither likely nor unlikely o somewhat unlikely o very unlikely  q21 what skills do you think would be important to build a successful relationship with the mentee? ___________________________________ vimloc mentorship program assessment: q22 what has been the most satisfying aspect about the vimloc mentorship program? ________________________________________________________________ q23 what has been the least satisfying aspect about the vimloc mentorship program? ________________________________________________________________ q24 how did you feel about the level of interaction commitment required (two interactions a month) for the vimloc mentorship program? o too much o about right o too little  q25 what would you suggest to improve the vimloc mentorship program?________   appendix b survey questionnaire 2022 for mentees q1 thank you for participating in this survey! we would like to hear your feedback on the 2022 vimloc mentorship program so that we can continue to improve it. please look over the informed consent letter before proceeding. o i have read and understand the above information. i agree to participate in this study. o i have read and understand the above information. i do not want to participate in this study. q2 how did you hear about the vimloc mentorship program? (please select all that apply) o vimloc group o school o conference o colleague o social media (e.g. facebook, twitter, linkedin, etc.) o friend o contacted by the vimloc mentorship committee o other q3 what are your reasons for participating as a mentee in the vimloc mentorship program? (please select all that apply) o knowing more about the profession o seeking guidance in career direction o moving up in your career o seeking advice on how to transfer skills obtained from your home country o learning about the skills and qualifications needed for a librarian-related job o networking o learning about how to adapt in an organization as a visible minority o other communication:  how many times have you used the methods below to communicate with your mentor: q4  e-mail (you sent to the mentor): 0, 1, 2 …9, 10 and above q5 online chat/instant messenger: 0, 1, 2 …9, 10 and above     q6 skype or video chat: 0, 1, 2 …9, 10 and above q7 in person: 0, 1, 2 …9, 10 and above q8 telephone: 0, 1, 2 …9, 10 and above q9 what was the most effective method of communication? o email o online chat/instant messenger o skype or video chat o in person o telephone o other (please specify) __________ interactions: q10 in your early contacts with your mentor, did you discuss your mentorship program expectations? o yes o  no q11 how important was it to you that the mentor was a visible minority? o very important o important o moderately important o slightly important o not at all important q12 did you and your mentor discuss any issues or experiences about being a visible minority in the profession? o yes o  no q13 how easy was communication with your mentor? o very easy o easy o moderately easy o difficult o very difficult q14 how would you describe your relationship with your mentor? o very good o good o fair o poor o very poor   q15 how likely are you to keep in contact with your mentor after the program ends? o very likely o somewhat likely o neither likely nor unlikely o somewhat unlikely o very unlikely q16 how helpful was your mentor in providing assistance? o very helpful o somewhat helpful o neither helpful nor unhelpful o somewhat unhelpful o very unhelpful q17 what aspects did your mentor help you with? (check all that apply) o provide encouragement o provide career counseling o help with job seeking skills (cover letter, resume, interview, etc.) o assist with networking o help with setting your professional goals o share own professional experience with you o share experience about the day to day work life o help with orientation to library culture and workplace expectations o advise on how to adapt in an organization as a visible minority o address your professional concerns o provide knowledge of a discipline or subject area o assist with research and scholarship (grant writing, research methods, etc.) o assist with promotion and tenure (preparation of materials, procedure, criteria, etc.) o share experience or improve skills in instruction o share experience or improve skills in collection management o share experience or improve skills in reference services o share experience or improve skills in leadership o share experience or improve skills in community involvement or outreach o share experience or improve skills in technology-related library work o other mentorship experience: q18 is this your first experience as a mentee through a formal mentorship program? o yes o no q19 based on this experience, how likely would you be to mentor in the future? o very likely o somewhat likely o neither likely nor unlikely o somewhat unlikely o very unlikely q20 what skills do you think would be important to build a successful relationship with the mentor?___________ vimloc mentorship program assessment: q21 what has been the most satisfying aspect about the vimloc mentorship program?____ q22 what has been the least satisfying aspect about the vimloc mentorship program?___ q23 how did you feel about the level of interaction commitment required (two interactions a month) for the vimloc mentorship program? o too much o about right o too little q24 overall, do you feel the vimloc mentorship program has met your expectations? o strongly agree o somewhat agree o neither agree nor disagree o somewhat disagree o strongly disagree q25 what would you suggest to improve the vimloc mentorship program?___________     editorial   happy 10th anniversary to evidence based library and information practice!   lorie kloda editor-in-chief associate university librarian, planning and community relations concordia university montreal, quebec, canada email: lorie.kloda@concordia.ca      2016 kloda. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     exactly 10 years ago, on march 15, 2006, the first issue of evidence based library and information practice (eblip) was published online as an open access journal. in some ways, 10 years may not seem like a long time, but in the world of online publishing, i think a decade of consistent quarterly issues is a significant achievement and one worth celebrating.   to celebrate, we have a few surprises to accompany volume 11, issue 1 of the journal. first, we have several commentaries from past members of the journal’s editorial board. the first commentary is by denise koufogiannakis, pam ryan, lindsay alcock, and su cleyle, the founding editors of eblip. other commentaries have been written to mark the occasion by past and present associate editors (evidence summaries) denise koufogiannakis, myself, and heather pretty, past editor-in-chief alison brettle, associate editor (classics) jonathan eldredge, associate editor (articles) lisl zach, and production editor michelle dunaway.   accompanying the publication of issue 1 of the journal for 2016 is a special issue on ebl 101. a few days prior to this issue’s publication on march 15, 2016, we released the special issue, a compilation of all 24 previously published ebl 101 columns in one place, accompanied by an editorial by the column’s frequent author, virginia wilson. the eblip editorial board considers this our anniversary gift to our readers, and we hope that having all these columns in one place will make learning about and practicing evidence based librarianship that much easier.   also new in this issue is the addition of a digital object identifier (doi) for each article published in eblip. the editorial board is extremely grateful to the university of alberta learning services for assuming the costs of assigning dois to not only all future issues of the journal, but to retrospective issues as well. this means that every paper published in eblip now has a stable and unique identifying link. we also want to thank sonya betz, digital initiatives projects librarian at university of alberta libraries for working diligently with her team to get the doi feature implemented in time for this anniversary issue.   finally, as the editor-in-chief and someone who has been with the journal in one role or another since its inception, i would like to extend my gratitude to everyone who has made this journal a success: members of the editorial board, the editorial advisors, copyeditors, indexers, peer reviewers, members of the writing assistance team, and of course all of our readers and champions over the years. this journal is the result of an enormous amount of effort by scholars and practitioners in the field of library and information studies from many sectors and regions. and it is also our reward.   happy 10th anniversary, eblip!   doing more with a dm: a survey on library social media engagement research article   doing more with a dm: a survey on library social media engagement   jason wardell health & life sciences librarian university libraries university of dayton dayton, ohio, united states of america email: jwardell1@udayton.edu   katy kelly coordinator of marketing and engagement university libraries university of dayton dayton, ohio, united states of america email: kkelly2@udayton.edu   received: 4 apr. 2022                                                                accepted: 20 july 2022      2022 wardell and kelly. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30141     abstract   objectives – this study sought to determine the role social media plays in shaping library services and spaces, and how queries are received, responded to, and tracked differently by different types of libraries.   methods – in april and may of 2021, researchers conducted a nine-question survey (appendix a) targeted to social media managers across various types of libraries in the united states, soliciting a mix of quantitative and qualitative results on prevalence of social media interactions, perceived changes to services and spaces as a result of those interactions, and how social media messaging fits within the library’s question reporting or tracking workflow. the researchers then extracted a set of thematic codes from the qualitative data to perform further statistical analysis.   results – the survey received 805 responses in total, with response rates varying from question to question. of these, 362reported receiving a question or suggestion via social media at least once per month, with 247 reporting a frequency of less than once per month. respondents expressed a wide range of changes to their library services or spaces as a result, including themes of clarification, marketing, reach, restriction, collections, access, service, policy, and collaboration. responses were garnered from all types of libraries, with public and academic libraries representing the majority.   conclusion – while there remains a disparity in how different types of libraries utilize social media for soliciting questions and suggestions on library services and spaces, those libraries that participate in the social media conversation are using it as a resource to learn more from their patrons and communities and ultimately are better situated to serve their population.     introduction   social media use by libraries as institutions is a well-established research topic. existing published research tends to focus on content strategies at a practical level, such as case studies and how-tos. in contrast, this study intends to fill a gap within the literature about current practices of social media management and the direct engagement happening between libraries and the communities they serve. the literature review will focus on social media and libraries in terms of the current landscape, user engagement, and managers’ perspectives.   literature review   current landscape   as a free communication tool, social media increases the capacity for companies, institutions and groups to promote themselves, view what people are saying about them, and converse with customers. according to edison research (2021), 82% of the total u.s. population over the age of 12 use social media, an increase from 79% in 2019 and 80% in 2020 (p. 20). the pew research center (2021, april 7) reports usage as around 72% and that facebook and youtube are the most used platforms, also stating that those companies’ “user base is broadly representative of the population as a whole.” institutions can use social media surveys like these to inform their strategy, depending on their intended audience and the content they produce.   accordingly, libraries in the u.s. that manage social media accounts use facebook more than any other social platform while “twitter is the next most popular platform, used by 67% of libraries, followed by instagram, used by 56% of libraries” (oclc webjunction, 2018, february 13). these platforms provide opportunities to share content widely and communicate one-on-one with people. most libraries use their social media to share upcoming events and event photos, while some choose to engage directly with their communities by offering reader’s advisory or research help (oclc webjunction, 2018, february 13). the american library association (ala, 2018) approved a set of social media guidelines for libraries, including creating a social media policy, staffing and managing the platforms, and making strategic decisions about intended audience and one-way or two-way communication. the guidelines conclude with the potential positive outcomes of libraries using social media, such as presenting the opportunity for “libraries to engage with users and to make significant contributions to shared knowledge. this robust civic engagement leads to an informed citizenry and a healthy society, while also demonstrating the great value of our institutions” (ala, 2018). the ala’s guidelines offer both encouragement and caution, striking a balance of outlining opportunities as well as consequences libraries could face.   user engagement   the literature also weighs the opportunities, positives, and negatives of using social media to engage with users as individual institutions. researchers conduct content analyses of social media accounts to arrive at conclusions about trends and strategies. a study by kushniryk and orlov (2021) analyzes the twitter posts and interactions of 12 large public libraries in north america and concludes with suggestions of how libraries have opportunities to better leverage twitter by engaging in dialogic communication. they suggest building better relationships by “replying to inquiries, providing feedback, commenting, and retweeting messages” (p. 6). practitioners and researchers emphasize the importance of continuously developing a communication strategy, or keeping social media social, with practices such as such as surveying your intended audiences’ social media habits and preferences (howard, huber, carter and moore, 2018) and moving beyond simply broadcasting messages and towards building connections and having conversations to “develop relationships, improve real-world services and resources, affect policy, and meet target goals (trucks, 2019, p. 12).   some have compared traditional services to what’s now possible with social media. in the introduction to their study of academic librarians’ perspectives, ahenkorah-marfo and akussah (2017) write “[reference librarians] employed face-to-face conversations with users. of late, however, the service environment increasingly demands digital reference service, more especially, synchronous service” (p. 1). on the other hand, researchers express caution concerning the negative outcomes of using social media to engage with patrons. katopol (2017) warns that it enables greedy behavior because a library’s social media presence needs constant attention and requires staff to be available 24/7 through cell phone and email (p. 3). kliewer (2018) calls attention to privacy concerns and the problematic practices of social media companies. social media engagement tactics can run counter to longstanding library ethics and principles.   managers’ perspectives   some research, including this study, seeks out social media managers in libraries to ask about their practices and perspectives. one of the earliest surveys was presented by rogers (2009, may 22) and reveals library use of blogs, social networking, and instant messaging to market and promote library services. the excitement and potential for reaching more people and meeting them online, where they were increasingly spending time, is a major theme of survey respondents (p. 6). another theme is respondents’ perceiving the lack of staff time as a major barrier to participating in these tools (pp.6-7). in 2014, taylor & francis group published a white paper about libraries’ practices and future opportunities with social media. their survey results show that promoting events, services, and resources were all top priorities, followed by more engagement-centered objectives such as connecting with new students, engaging with the academic and local community, and as a customer service tool (p. 8). social media tracking and assessment varied, some citing the fact that they don’t have a significant number of users to warrant writing a report, like other libraries choose to do (p. 21).   other manager-specific perspectives are discussed in a study that surveyed art librarians; 71% agreed or strongly agreed that social media can increase visitors and collection use in their library (sulkow et al., 2019, pp. 308-309) and in their case study, a manager cites that “content creation, regular engagement, image editing, and other time-consuming activities are forms of labor that are often hidden from coworkers and administration” (p. 315). unlike the study presented below, these do not focus on direct messaging or engagement with social media users. the themes in this literature review, such as staff time, reporting, and providing services, are relevant to the survey results.   aims   this study’s aim was to determine the role social media messaging plays in shaping library services and spaces across all types of libraries. the authors sought to explore how and how often libraries of different types directly engage with their patrons on social media and how queries are managed.   methods   to better understand the current use of social media by libraries to solicit and respond to questions and feedback, the researchers wrote an online survey intended for managers of social media within all types of libraries in the united states. for this research, “managers of social media” was defined as any individual who is responsible for monitoring or posting on social media on a library’s behalf. all library social media managers were encouraged to respond; there was no limit per library.   following irb approval, the 9-question online survey (see appendix a) launched in april 2021. each question was optional. between april 28 and may 14, the researchers and their colleagues shared the invitation to participate, focusing the project’s initial communication on ohio library workers through ohio-specific electronic mailing lists. between may 18 and june 4, the invitation was shared to a national audience by the researchers and their colleagues using the american library association’s discussion boards, library marketing-specific facebook groups, and various professional association electronic mailing lists. the survey was intentionally distributed to professional organizations for multiple types of libraries to ensure data collection and representation from public, academic, school, government, and special libraries. appendix b includes a timeline of the invitation sharing to each communication channel and their approximate reach.   between april 28 and june 4, 2021, 805 people responded to the survey. the researchers independently analyzed and coded responses to the survey’s two open-ended questions using qualtrics xm software for qualitative and quantitative analysis, identifying 12 themes within q7 responses and 6 in q8 after comparing findings and reconciling discrepancies.   results   survey results revealed experiences and practices of social media managers in libraries. the first two questions filtered out respondents who were ineligible to participate. all 100% affirmed they were willing to take the survey. of these, 23 respondents said they were not social media managers in response to question 3 and the survey ended for them. the remaining 763 participants, self-identified social media managers, continued the survey. since the individual questions were optional, the total number of responses varied from question to question.   the subsequent questions inquired about the person’s own engagement experiences while using their library’s social media account(s). of those who answered question 4 (n=615), nearly 92% reported receiving questions or suggestions; 8% have not. for the respondents of question 5 (n=614), nearly 97% reported that they respond to questions or suggestions. there were 29 people who answered q5 affirmatively, but either skipped or said no to q4.   the next section of the survey asked participants to report the frequency of engagement and practices related to questions or suggestions received on social media. it includes the survey’s two open-ended questions that the researchers coded and analyzed.   most commonly, respondents reported receiving questions or suggestions less than once a month (n=249). the next highest number of respondents (n=187) said they received questions or suggestions weekly. the remaining options presented in the survey were monthly (n=134), daily (n=33), and more than once a day (n=9). see figure 1.     figure 1 q6: “approximately how often do you receive questions on your library’s social media?”     question 7 asked survey respondents “what has changed in your library services or spaces as a direct result of questions or suggestions received via social media?” the 12 themes that emerged from the researchers’ qualitative coding process on changes to library services or spaces are listed in the next section. each individual response was given one or more of these codes, describing how the library’s services or spaces changed or didn’t change.   description of codes, with representative samples   samples have been abbreviated for clarity. the anonymized data set of responses is available upon request.   clarification (n=143): a response to a question via social media. this includes quick, ready reference questions in addition to access-related questions. this is considered a change to services, as it represents a new platform by which to communicate with patrons.   ●       public library in new york: “people want to verify services and hours of operations as well as ask questions about programs” ●       government library in kentucky: “[the] majority of questions we receive are how to research, and we direct them to the appropriate page on our website.” ●       academic library in pennsylvania: “[questions] are generally just basic ones like "is the library open today?"”   marketing (n=49): an adjustment to messaging off social media. this includes changes to physical signage, website content, or advertising. this is considered a change to spaces, either physical or digital.   ●       academic library in ohio: “mostly it has been people dming questions or letting us know about noise complaints…we did increase signage as a result.” ●       public library in pennsylvania: “it helped me identify where we can improve communication with the public such as where and what we include on flyers and brochures” ●       public library in massachusetts: “we became aware of people with autism preferring the term "autism acceptance," and changed our signs and sm accordingly.”   reach (n=103): an increase in usage of social media on the part of the library social media manager. this includes consciously increasing the content, maintenance, or monitoring of a library’s social media. this is considered a change to services.   ●       government library in mississippi: “[social] media is an easy way for people to contact us. it helps us to be available to all of our patrons, not just those who can make it into the building or call us on the phone.” ●       public library in california: “social media gives us a bit of a thermometer on what people know, want to know, and don't know about our services.”   restriction (n=5): a decrease in usage of social media. this includes automatic responses saying the library does not check social media messages, messaging redirecting users to traditional service points, or anything to dissuade users from contacting the library over social media. this is considered a change to services.   ●       public library in massachusetts: “we have more explicit autoresponders on social media telling people we don't monitor in realtime [sic], and alerting them to [phone, email,] chat options instead.” ●       public library in massachusetts: “[we] had an autoresponder asking folks to email us…especially so the more detailed requests that involve different people could have everything altogether instead of spread out through a chat. also because messages sent to the page were easy to accidentally miss and often sent at odd times when no one was online.”   collections (n=33): an addition to the library collection instigated by communications using social media. this includes physical and digital purchases and subscriptions as well as coordination for donations. this is considered a change to services.   ●       public library in pennsylvania: “we have purchased a couple books based off of some suggestions on social media.” ●       public library in pennsylvania: “sometimes patrons' messages help us with collection development, as they message us through facebook to ask us if a certain title is available or if we can purchase a certain title.”   access (n=33): a change to the ways in which a patron might interact with the physical library space. this includes changes to hours, reconfiguration of seating, improved wifi. this is considered a change to services or spaces.   ●       public library in new york: “it's hard to measure exactly, but we speeded up our timeline for reopening our doors to the public because of "suggestions" -more like annoyed comments! -from patrons.”   service (n=76): a change to or addition of patron-facing programming. this includes in-person events, online versions of previously offered services, or the creation of new platforms for patron interaction. this is considered a change to services.   ●       academic library in california: “we notice trends, when there is a preponderance of questions, it means that [we] have to address the topic, either on our social media or with addressing changes itself on our library space. for example, our campus has many parents, and since we received many questions about children in the library, we created a children's space.” ●       public library in pennsylvania: “we are able to handle more online reference questions and online programs due to social media”   policy (n=16): an internal change to how the library—in whole or in part—responds to certain situations. this includes policies on social media, mindfulness and continuity in messaging, and procedures when it comes to recording social media messages as reference transactions. this is considered a change to services.   ●       public library in pennsylvania: “we have formed a team of public service staff responsible for monitoring and answering questions on social media. previously this task was the responsibility of marketing.” ●       public library in new york: “we had many questions that we realized staff had different answers to. it made us rethink, rewrite, or write new procedures that made clarifications for both staff and patrons.”   collaboration (n=8): a new connection between departments within the library or between the library and external partners, either instigated by or founded on social media communication.   ·        academic library in ohio: “lots of partnerships with other departments on campus. the questions are in the forms of tagging us in an event or initiative to share with our audience. but we have had comments questioning our intent when posting about race. we took immediate internal action by meeting and consulting with the university social media contact.”   in addition, three “no-change” codes were identified:   nothing (n=124): the respondent states that there has been no change to library spaces or services as a result of social media interaction.   ·        public library in maine: “not much. because i am technically not a member of our patrons services team, i let people know what is being said but not much changes.” ·        public library in florida: “very little. i report the suggestions and questions but it's rare department heads or administrators actually act on the feedback, unfortunately.”   n/a (n=19): the respondent claims that the question is not applicable to their library, and no further clarification is given. the authors considered this to be different from a “nothing” response, in that it suggests there has been no opportunity for change, whereas “nothing” suggests the opportunity existed, but no change was made.   unclear (n=9): the wording of the response is ambiguous or uninterpretable.    ·        academic library in florida: “unsure, information doesn't usually reach back to me.”   a total of 618 respondents elaborated on their experiences in response to this question. overall, the top three themes across all libraries were clarification, nothing, and reach.   question 8 inquired “how are these questions and suggestions reported and/or tracked?” the six themes that emerged were:   ·        not (n=289): they are not reported or tracked. ·        included (n=100): they are included in the library’s overall reporting or tracking mechanism. ·        referral (n=69): they are referred to the appropriate person or department. ·        separate (n=37): they are tracked separately as social media engagement. ·        yes, other (n=16): they are reported and tracked but in other ways. ·        unknown (n=2): the respondent does not know.   there were 513 responses to this question. the majority of participants (56%) stated that questions or suggestions are not reported and/or tracked. the other top themes showed that social media engagements are included in overall library counts (19%) or that questions and suggestions are referred to someone else by the social media manager (13%).   the final survey questions investigated the demographics of respondents. question 9 asked “what type of library do you work for?” 612 participants answered this question; every type of library was represented by at least one respondent. the majority represented public libraries (61.60%) while workers in academic libraries (26.63%) represented the second largest group of participants.   question 10 asked respondents to report the state where their library is located. social media managers in pennsylvania represented 26.6% of the respondents while libraries in ohio and texas had high representation as well.     figure 2 q4: “have you received questions or suggestions…” by library type.     the researchers also compared results across library types to explore any differences or similarities between them. out of all types of libraries, public libraries were the most likely to receive messages over social media, and k-12 school libraries were the least likely. figure 1 shows whether libraries receive questions or suggestions over social media, by library type. apart from government libraries, for whom all of those who responded to the survey reported responding to social media questions or suggestions, public libraries were also the most likely to respond to these inquiries, as seen in figure 3.     figure 3 q5: “do you respond to questions or suggestions…” by library type.     in addition to receiving a greater percentage of messages over social media, public libraries also receive much more frequent communication, the most common response stating they receive messages weekly, whereas all other types of libraries predominantly reported receiving messages less than once a month. figure 4 shows frequency by library type.     figure 4 q6: social media contact frequency, by library type.     types of libraries also differed in how they saw social media messaging as affecting change and how they reported or tracked questions or suggestions received over social media. public libraries, for instance, ranked clarification (n=98) and reach (n=69) as their two most frequent changes, while academic libraries’ most frequent change was nothing (n=43). see table 1 for the theme frequency according to library type.     table 1 q7: “what has changed in your library services or spaces…” by library type code public school (k-12) academic government special other total clarification 98 2 31 5 2 5 143 nothing 67 1 43 7 4 2 124 reach 69 2 18 5 4 5 103 service 57 1 13 3 0 2 76 marketing 30 0 15 2 0 2 49 collections 26 1 5 0 0 1 33 access 19 1 12 0 0 1 33 n/a 10 0 8 1 0 0 19 policy 9 0 6 1 0 0 16 unclear 5 0 3 0 1 0 9 collaboration 4 1 2 1 0 0 8 restriction 2 0 2 0 0 1 5     though more respondents from public libraries identified some manner of change resulting from social media messaging, they were less likely to report or track these messages in any formal way. whereas around 45% of academic, government, and special libraries each report not tracking social media messages, 66% of public libraries neither track nor refer incoming questions or suggestions received on social media. figure 5 shows percentages of how each type of library tracks or reports questions or suggestions received on social media.     figure 5 q8: message tracking or reporting, by library type.     table 2 q8: message tracking or reporting, by library type q8 public   school   acad.   gov.   special   other   not 63.29% 200 85.71% 6 44.14% 64 45.45% 10 45.45% 5 33.33% 4 referral 14.24% 45 0.00% 0 13.79% 20 13.64% 3 9.09% 1 0.00% 0 included 11.39% 36 0.00% 0 33.79% 49 27.27% 6 18.18% 2 58.33% 7 separate 6.96% 22 0.00% 0 6.90% 10 13.64% 3 9.09% 1 8.33% 1 yes, other 3.80% 12 14.29% 1 1.38% 2 0.00% 0 9.09% 1 0.00% 0 unknown 0.32% 1 0.00% 0 0.00% 0 0.00% 0 9.09% 1 0.00% 0 total   316   7   145   22   11   12     discussion   survey results show that a library's social media presence provides a valuable patron interaction point beyond being a platform for simply sharing content and soliciting feedback. for many libraries, social media has joined other methods of interaction—such as phone, email, face-to-face, etc.—as an integral service point. those libraries that provide even a modicum of interaction on a social media platform are better at reaching their patrons where they are, and the most engaged among them report connecting with their users in ways that might not happen otherwise. while nearly 92% of respondents of q4 acknowledge receiving some manner of question or suggestion from a patron on social media, nearly 97% of them report using their social media to address questions or suggestions. more respondents reported responding than receiving, which may suggest the use of social media to address questions or suggestions received elsewhere, or it may be a result of multiple individuals on a social media team having responsibility for receiving, reporting, and responding to messages. this uncertainty is discussed further in this study’s limitations and opportunities for future research.   engaging with questions and suggestions   across all libraries, 59% receive at least one question or suggestion via social media per month, and 37% receive at least one per week. for public libraries, communication over social media is more frequent, with nearly 51% receiving at least one question or suggestion per week. other library types primarily report receiving fewer than one per month, though with the exception of k-12 school libraries, at least 25% report receiving one or more per month. those libraries that engage in proactive methods to garner social media communications have a more positive experience in receiving and utilizing social media feedback. among libraries the authors identified as increasing social media reach in response to messages received—a group including all types of libraries—the frequency of received messages is greater, with 72% receiving at least one message per month and 50% receiving at least one per week.   this study does not assess patron communication tendencies, but rather the libraries’ response to social media interaction. in response to q7, 20% of survey respondents claimed that nothing changed, but then went on to describe how social media has become a platform to receive and respond to ready reference questions. the act of responding to these questions constitutes a change in service: a new platform by which to communicate with a patron base that may not have reached out via other methods. this change is subtler than something collections or service-related, where a patron explicitly asks for something not previously offered, and as a result the library changes its offerings. instead, the act of reaching out over social media is the implicit ask—“will you respond?”—and as reported in the survey, the answer is not always “yes.” in a minority of cases, older, more traditional methods of communication such as email or phone are preferred, and library social media services have been restricted to direct patrons to reach out through those official channels. a wisconsin public library anticipated this tendency, writing, “we also offer direct links to our website/events/registration instead of just saying "go to our website.”” some respondents also identified the tendency to attract bad-faith interactions with “trolls” or individuals using the anonymity of the internet to justify mean-spirited or hurtful criticism. opening the door to interaction on social media also invites these individuals to participate. across the board, survey respondents who identified this sort of behavior also report ignoring or blocking the offenders. in some cases, social media served as an impetus to increase messaging mindfulness. some survey respondents noted requests to change certain phrasing to more acceptable terminology, both in functional library tools—"feedback has influenced naming practices in our catalog” from a texas academic library—and on public-facing media platforms—“we became aware of...the term “autism acceptance” and changed our signs and sm accordingly” from a massachusetts public library. others said their social media interactions led to including depictions of a wider variety of individuals when posting images. the quick feedback afforded to and expected by users of social media often make it a good venue for suggestions, provided the library is open to receiving feedback in this way. several libraries identified an improvement in their communication style after receiving social media feedback. “it helped me identify where we can improve communication…what we include on flyers and brochures [and] information we make available through social media.” others identify changing “where and when we post announcements,” with a goal of making services “more customer friendly.” tracking and reporting practices   when it comes to incorporating social media comments and suggestions into a tracking or reporting workflow, there are some notable differences in how libraries handle this task. while public libraries report receiving the most frequent communication via social media, they also report having the least codified tracking structure for any questions or suggestions received in that way. one washington academic library notes, “so far they are few and far between, so not tracking currently.” this is a recurring theme throughout the responses: many libraries share that the volume of social media questions and suggestions is too low to warrant tracking. however, this suggests that should the number of social media interactions increase, it might be worth recording. as one texas academic library responds: “i am in the process of developing a social media engagement reporting schedule to help track social media engagement in general. since our engagement is typically very low, there is not much to report or track, which is why this hasn't been done in the past.” smaller libraries, too, identify less of a need to track or report social media interactions. one minnesota public library responds: “if i received questions/suggestions, i would not track them. i might report them to the city administrator. i am the only librarian in the library.” similarly, from a michigan public library, “we are small, so i report to our interim manager.” in these cases, where incorporating social media comments into reported reference statistics is perhaps unnecessary, they still see value in referring the social media feedback up to an administrator. there is a trend of directly informing others about social media feedback but not incorporating it into existing structures. several libraries refer social media suggestions directly to administration, with 17 responses sharing any feedback directly with their director, dean, or senior management. this may suggest that social media feedback carries more importance than feedback received through traditional channels since it is afforded a direct route to library administration. one of the study’s goals was to find out how social media interactions are managed, including opportunities to track, report, and refer. findings show that most social media managers do not track questions received through social media channels, such as through existing workflows like reference question reporting or referring to other departments. without this information, how could library social media managers best support their continued use of these tools? perhaps new parameters such as measuring the quantity and quality of engagements and changes to services or collections could be new, evidence-based parameters for success. this data would help showcase the impact and intentionality of a library’s social media presence, more so than other available analytics such as number of followers, likes, or views.   timeliness   responses to this survey were clearly impacted by its timing, distributed as it was in the second year of the covid-19 pandemic. during this worldwide event with major ramifications for the safety and practicality of in-person library services, many libraries either added or expanded their online presence, and social media communications became an extension of this. one pennsylvania public library remarked that social media "helped us innovate pandemic services and work toward keeping popular ones in a modified fashion." while this is one of 34 responses to q7 or q8 directly mentioning either covid or the pandemic, any change to clarification, reach, or virtual services would be a benefit to a patron base that was incapable of visiting the library in person. thus, we can assume that many more of the suggestions and comments received by librarians were a direct result of covid-related communication needs. as libraries have reopened (and while some never closed), covid-19 also led to changes to physical spaces. one minnesota k-12 school library received social media feedback from students who "commented that they liked the reduced seating in the learning commons" for covid-related physical distancing requirements, which had the unintended effect of "[allowing] them a more quiet area to work." other libraries variously reported boosting wifi signals, adjusting hours of operation, incorporating pop-up outdoor events, and other demonstrations of flexibility in reaction to pandemic demands. what remains to be seen is how these services and communication channels will change in a post-pandemic world. follow-up research is warranted to further explore the effects of covid-19 on libraries' social media use.   limitations and future research   first, while the authors were elated at the number and range of survey responses and the trends derived from commonalities in the qualitative portions, the low number of responses from school (k-12), government, special, and other libraries besides public and academic makes it difficult to draw generalized conclusions from those library types. any future studies interested in one or more of those populations should target them directly via their professional organizations, mailing lists, and interest groups rather than large, generic organizations. also in relation to survey responses, study limitations arose since the researchers invited all who identified as library social media managers to participate; there was no limit per library. survey respondents’ self-identification as a social media manager presented the possibility of a wide range of job duties and expertise represented across all participants. in addition, multiple responses from the same library could have affected the sample size, and add an uncertainty about whether individuals responded according to their own experience or assumed they were answering on behalf of their library. lastly, the broad and exploratory nature of the qualitative portion of this survey led to some confusion among the respondents. both q7 and q8 were written in an ambiguous way, so while some responses covered services, spaces, questions, and suggestions, many more addressed just one or two of those aspects. additionally, there was room for interpretation when it came to our definitions of "what has changed" in library services or spaces and how social media interactions might be "tracked." utilizing the coding data generated in this study, future exploration into this topic might limit the amount of open-ended questions and instead provide a list of options for both how things have changed and how social media interaction is reported, with limited space to write in explanations or examples.   the survey results and analysis reveal opportunities for deeper research regarding library social media management. future research questions might include:   ●       the authors noted a theme of social media messages having a faster path directly to library administration. do questions or suggestions from people using social media get higher priority over those received through traditional reference channels? a survey of library administrators as to their impression of social media effectiveness may reveal additional insight. ●       there was little agreement among libraries as to how to best track and report on social media interaction. further, there was great disparity between those libraries claiming to get a lot out of social media interaction and those claiming to get none. are there trainings or competencies on library social media best practices that should be developed for library social media managers?   conclusion   this study shows a variety of practices related to communicating and social media, including sharing, listening, tracking, and making changes. although social media is usually part of an overall communication strategy, it can become just another mechanism to just share news and updates. many libraries, however, are using it as a resource to learn more from their patrons and communities to better serve their population. this increases a library’s approachability and reach.   author contributions   jason wardell: conceptualization (equal), data curation (equal), formal analysis (equal), investigation (equal), visualization (lead), writing – original draft (equal), writing – review & editing (equal) katy kelly: conceptualization (equal), data curation (equal), formal analysis (equal), investigation (equal), visualization (supporting), writing – original draft (equal), writing – review & editing (equal)   references   ahenkorah-marfo, m., & akussah, h. (2017). information on the go: perspective of academic librarians on use of social media in reference services. international information & library review, 49(2), 87-96. https://doi.org/10.1080/10572317.2016.1278190 american library association. (2018, june). social media guidelines for public and academic libraries. https://www.ala.org/advocacy/intfreedom/socialmediaguidelines edison research. (2021). the infinite dial 2021. http://www.edisonresearch.com/wp-content/uploads/2021/03/the-infinite-dial-2021.pdf  howard, h. a., huber, s., carter, l. v., & moore, e. a. (2018). academic libraries on social media: finding the students and the information they want. information technology and libraries, 37(1), 8-18. https://doi.org/10.6017/ital.v37i1.10160 katopol, p. (2017). the library as a greedy institution. library leadership & management, 31(2), 1–7. https://doi.org/10.5860/llm.v31i2.7251 kliewer, c. (2018). library social media needs to be evaluated ethically. public services quarterly, 14(2), 170–182. https://doi.org/10.1080/15228959.2018.1447418 kushniryk, a., & orlov, s. (2021). ‘follow us on twitter’: how public libraries use dialogic communication to engage their publics. library & information science research, 43(2), 1–7. https://doi.org/10.1016/j.lisr.2021.101087 oclc webjunction. (2018, february 13). social media and libraries survey summary. https://www.webjunction.org/news/webjunction/social-media-libraries-survey.html  pew research center. (2021, april 7). social media fact sheet. https://www.pewresearch.org/internet/fact-sheet/social-media/  rogers, c. (2009, may 22). social media, libraries, and web 2.0: how american libraries are using new tools for public relations and to attract new users. presented at the german library association annual conference: deutscher bibliothekartag 2009 in erfurt. https://dc.statelibrary.sc.gov/bitstream/handle/10827/6738/scsl_social_media_libraries_2009-5.pdf?sequence=1  sulkow, c., ferretti, j. a., blueher, w., & simon, a. (2019). #artlibraries: taking the pulse of social media in art library environments. art documentation: bulletin of the art libraries society of north america, 38(2), 305–323. https://doi.org/10.1086/706630 taylor & francis group (2014). use of social media by the library: current practices and future opportunities [white paper]. taylor & francis group. trucks, e. (2019). making social media more social: a literature review of academic libraries’ engagement and connections through social media platforms. in j. joe & e. knight (eds.), social media for communication and instruction in academic libraries. (pp. 1-16). igi global. https://doi.org/10.4018/978-1-5225-8097-3.ch001   appendix a survey    invitation to participate in research surveys and interviews research project title: library social media management and engagement you have been asked to participate in a research project conducted by (researchers) from (institution). we are looking for managers of social media at all types of libraries in the united states to answer a 5-minute survey with mostly yes or no questions. for the purposes of this survey, we are defining “managers of social media” as any individual who is responsible for monitoring or posting on social media on a library’s behalf. all library social media managers are encouraged to respond; there is no limit per library. the purpose of this project is to better understand the current use of social media by libraries to solicit and respond to questions and feedback. ●       you should read the information below, and ask questions about anything you do not understand, before deciding whether or not to participate. ●       your participation in this research is voluntary. you have the right not to answer any question and to stop participating at any time for any reason. answering the questions will take about 5 minutes. ●       you will not be compensated for your participation. ●       all of the information you tell us will be confidential. ●       if this is a recorded interview, only the researcher and faculty advisor will have access to the recording and it will be kept in a secure place.   ●       if this is a written or online survey, only the researcher and faculty advisor will have access to your responses. if you are participating in an online survey: we will not collect identifying information, but we cannot guarantee the security of the computer you use or the security of data transfer between that computer and our data collection point. we urge you to consider this carefully when responding to these questions. ●       i understand that i am only eligible to participate if i am over the age of 18. please contact the following investigator with any questions or concerns: (researcher and contact information) if you feel you have been treated unfairly, or you have questions regarding your rights as a research participant, please email (email address) or call (phone). 2. do you agree to participate in this survey? ●       yes ●       no 3. are you a manager of a library's social media account(s)? (note: for the purposes of this survey, a "manager" is anyone responsible for monitoring or posting on social media on the library's behalf.) ●       yes ●       no 4. have you received questions or suggestions on your library's social media? ●       yes ●       no 5. do you respond to questions or suggestions on your library social media? ●       yes ●       no 6. approximately how often do you receive questions or suggestions on your library's social media? ●       more than once a day ●       daily ●       weekly ●       monthly ●       less than once a month 7. what has changed in your library services or spaces as a direct result of questions or suggestions received via social media? 8. how are these questions and suggestions reported and/or tracked? 9. what type of library do you work for? ●       public ●       school (k-12) ●       academic ●       government ●       special ●       other 10. in which state is your library located?  ●       included all 50 states, washington d.c., and puerto rico   appendix b timeline of the survey invitation sharing to each communication channel and their approximate reach   invitation date, 2021 channel members at invitation date, approx., if known april 28 academic library association of ohio (alao) listserv 645 april 28 ohio library council (olc) marketing and pr division   april 28 ohio library support staff institute (olssi)   april 28  society of ohio archivists 376 april 29 academic library association of ohio (alao) programming, outreach, and marketing interest group (promig) listserv 39 may 3 ohio educational library media association (oelma)   may 3, may 10 ohiolink may weekly updates   may 6 ohionet may newsletter   may 18 facebook group: association of college and research libraries (acrl) library marketing and outreach interest group  5,400 may 18 facebook group: libraries & social media 13,400 may 18 american library association (ala) connect: ala all members 45,000 may 18 ala connect: acrl library marketing and outreach interest group 1,400 may 19 medical library association   may 19 society of american archivists 6,369     news/announcements   call for applicants: concordia university library’s researcher-in-residence program      2016. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   concordia university is one of canada’s most dynamic universities in one of north america’s most livable cities. it reflects a unique blend of commitments to diversity, accessibility, social responsibility, and innovation. with over 46,000 students (6,300 international) and nearly 1,700 full-time and part-time professors, concordia is recognized for offering a rich academic experience combined with strong community engagement.   this is a time of change for concordia university library. a new library space was recently inaugurated in the grey nuns building. a major renovation and expansion project is transforming the webster library, at the sir george williams campus (downtown), over the period of 2015 to 2017. an innovative technology program is being developed and will contribute to enhance the students experience in the libraries.   concordia library recently developed a strategic plan (https://library.concordia.ca/about/plan/) for 2016-2021. some of the highlights from the plan’s goals and objectives include:   the transformation of service offerings to provide an outstanding user experience the launch of the concordia university press to disseminate engaging research in the form of open access scholarly books the creation of a platform for the preservation and dissemination of oral history material the promotion of research and innovation in library and information studies   in light of this, the newly launched concordia library researcher-in-residence program has been created to promote research in the library and the use of research by practitioners. the program offers the opportunity for scholars or doctoral students to focus on an area of inquiry in a supportive and enriching environment, and to interact with concordia library staff and its resources.   the presence of the researcher-in-residence in concordia library has as one of its goals to foster a culture of research. it is hoped that throughout the residency, both the researcher and staff will become engaged in the use of research in library practice and the concept of evidence-based librarianship.   the residency period is from january 1 to december 1, 2017. proposals which include alternative start dates and those of different duration will also be considered.   for the inaugural researcher-in-residence program, concordia library invites proposals relating to any of the library’s strategic plan goals as well as the following themes:   the role of the library in a next-generation university the role of the library in the digital humanities collection development in an increasingly connected world use of space for learning and research digital libraries communication and outreach to users efficacy of library instruction professional development and capacity building of library staff   researchers from library and information studies, as well as other disciplines (e.g., anthropology, computer science, education, informatics, media studies, sociology) are welcome to apply.   scope   as part of the program, concordia library will provide:   ·         dedicated workspace in concordia library ·         access to computing facilities ·         access to library resources, print and electronic collections, and technological infrastructure ·         opportunities for the researcher-in-residence to interact with concordia library staff ·         other types of in-kind support may be made available   in return, the researcher-in-residence is expected to participate in the intellectual life of concordia library. this participation may take various forms, including:   ·         presenting research in progress to staff ·         providing learning opportunities in the form of seminars or workshops for a campus or library audience on topics related to their own expertise ·         participating in selected activities in concordia library, depending upon the relevance to their research, their expertise, and interest. such activities include: ·         committee meetings ·         special events ·         brown bag talks, seminars, round tables, or other information sessions ·         concordia library research forum ·         meeting individually with librarians and library staff (e.g., office hour drop-ins, or by appointment)   funding   ·         for a phd student or postdoctoral fellow, a stipend of $20,000, plus up to $2,500 for travel/research expenses ·         for a faculty member on sabbatical leave, research funds of up to $20,000 (budget required)   eligibility   phd students, postdoctoral fellows, and full-time members of academic staff are eligible to apply. the proposed research should make a contribution to the knowledge base of librarianship, libraries, or information studies. professors and librarians preparing for a sabbatical are encouraged to apply.   application and instructions   applications should include information about the applicant, including the expertise brought to the program:   ·         letter of intent, including ·         a description of study / project proposal ·         proposed in-library activities (outline of activities to be undertaken during the period of residency) ·         proposed research outputs ·         curriculum vitae ·         short biography (100 words) ·         for phd student applicants, two letters of recommendation are also required. one of these should be from the applicant’s doctoral supervisor. letters of recommendation should be submitted electronically directly by the referee. ·         for members of academic staff, a budget should also be included   selection criteria   submissions will be evaluated by a selection committee composed of the university librarian and four librarians using these criteria:   ·         quality and originality of the proposed research project ·         demonstrated ability to complete the proposed project based on previous accomplishments ·         significance of the proposed research for librarianship, libraries, or information studies   deadline   the application process opens april 26, 2016   the deadline for receiving the complete application, letters of recommendation, is august 8, 2016 by 5:00 pm edt. applications may be submitted in either english or french.   applications should be submitted as a single pdf file to lorie kloda, associate university librarian, planning & community relations, at lorie.kloda@concordia.ca notification of the successful candidate will be given in september 2016.   the full details of the program may also be found at: https://library.concordia.ca/about/researcher-in-residence/   evidence summary   secret shopping is an effective tool for identifying local patterns in library user experience   a review of: boyce, c. m. (2015). secret shopping as user experience assessment tool. public services quarterly, 11(4), 237-253. doi:10.1080/15228959.2015.1084903   reviewed by: kelley wadson librarian bow valley college calgary, alberta, canada email: kwadson@bowvalleycollege.ca   received: 27 june 2016  accepted: 19 oct. 2016      2016 wadson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to assess library user experience (ux) at two entry-level service desks to determine the need for, and inform the aspects in which to improve, services and staff training.   design – observational study using secret shopping.   setting – a small, private university in illinois, united states of america.   subjects – library employees, comprised primarily of student assistants; and 11 secret shoppers, comprised of 5 faculty members, 4 staff members, and 2 first-year students from the university.   methods – recruitment methods for shoppers consisted of the campus electronic mail list, flyers, directed requests, and a $10 gift certificate to the campus bookstore following their participation. both groups (library employees and secret shoppers) were briefed on the purpose of the study and completed informed consent forms. shoppers attended face-to-face training sessions in which they selected two questions to ask from a list of options, one for each service desk in the library. shoppers were not told at which desk to ask their questions. the list of options included informational and research assistance questions; shoppers also had the option of asking a question based on an actual information need. two service desks were involved: one for circulation and one for research/it support. since it staff and librarians were excluded from the study, shoppers were directed to tactfully end the transaction if a referral to an expert was made.   within two weeks of the training session, shoppers made two separate visits to the library at a time convenient to them to ask the question and observe the transaction at each of the two service desks. for each round of secret shopping, shoppers completed an electronic evaluation form afterward on the qualtrics platform. the evaluation form consisted of yes/no, multiple-choice, and open-ended comment questions with two questions employing skip logic for a total of 29 possible questions. the questions covered the following variables both quantitatively and qualitatively: how well the question was answered, the customer service skills (responsiveness, approachability, and respectfulness) of the library employee, and if applicable, the quality of the referral to other library staff or services.   main results – the shoppers evaluated a total of 21 transactions: 11 for the circulation desk and 10 for the research/it support desk (1 shopper did not evaluate this desk). eighteen of the transactions were in-person and three were by phone. eight of the questions asked were based on the participants’ actual information need.   on the variable of satisfaction with the answer received, the research/it support desk scored higher than the circulation desk. the circulation desk received 7 very satisfactory ratings, 3 satisfied ratings, and 1 neutral rating; whereas the research/it support desk received 10 very satisfactory ratings and 1 satisfied rating. the lower scores of the circulation desk may be related to the variables of responsiveness and approachability, as library employees on the circulation desk were scored lower in these areas and observed as not paying attention in two interactions and this was not observed at all at the research/it support desk. however, the study did not collect sufficient data to test this relationship. all shoppers gave positive ratings on whether they were treated respectfully and if the library employee waited for the shopper to state the question fully and with the exception of one transaction, clarify the question if necessary.   responses to the open-ended comment questions were reviewed by investigators, who found that in five cases the transaction would have been improved by consulting library faculty. on the variables related to customer service, responses were generally positive but in several transactions the library employees failed to appear attentive and ready to help the shopper.   conclusion – the author found secret shopping was an effective tool for evaluating library ux to identify both positive and negative patterns and better inform responses to areas in need of improvement. the author identified two areas for improvement to the library. first, library employees at the circulation desk require additional training that would encourage them to refer transactions to library faculty where necessary. second, although evaluation of customer service skills was generally positive, library employees will also receive additional training that will emphasize listening and role-playing scenarios. these areas for improvement will also support the library’s plans to combine research and circulation functions into a single service desk on the entry level and move the it support desk to the third floor.   commentary   secret shopping is a form of participant observation for evaluating customer service that has been widely used in the retail sector (zorica, ivanjko & spiranec, 2014). the author notes that libraries have been employing secret shopping since 1970, but mainly in public libraries. the article cites four studies of academic libraries as of 2013 and seven studies of public libraries. this study thus contributes to the growing discussion and application of secret shopping in academic libraries, which includes more recent publications by crowe and bradshaw (2015) and zorica, ivanjko, and spiranec (2014).   using glynn’s ebl critical appraisal checklist (2006), the study’s limitations are related most significantly to population and, to a lesser extent, data collection and results. the study aimed to recruit five participants from each of the university’s main stakeholder groups: faculty, staff, and students. the actual total figure was 11 participants, only 2 of them students. the author noted that comments from faculty and staff were richer than those from the students. the study thus exhibits selection bias, whereby the sample was not fully representative of the entire population. for data collection, the author did not state at what point after the transaction participants completed the evaluation form. two weeks was allotted for both rounds of secret shopping, but it is unclear whether the participant completed the form immediately or waited several hours or possibly longer. there was thus a risk of recall bias, in that the data relied on the shopper’s memory of the event rather than the actual event itself.   unfortunately, the results have low external validity as the sample size was too small for generalizable conclusions beyond a local context. however, this highlights two strengths of the study. firstly, the study exhibits strong ecological validity as it was conducted in a natural setting with actual patrons who were encouraged to present a real information need by selecting their own question. secondly, the author used trends identified through previous research, namely campus surveys, to inform and enhance the precision of the evaluation form.   the implications for practitioners are related to the study’s methodology rather than its results. as ux and service delivery become increasingly important to academic libraries (bell, 2014), this study adds to the research applying secret shopping as an alternative to what some have claimed is an over-dependence on surveys in lis research (halpern, eaker, jackson, & bouquin, 2015). in sum, secret shopping is an effective tool for exposing or verifying local patterns in library ux. for more reliable results, it should be combined with other methods or designed to avoid this study’s limitations by seeking a larger sample size or requiring participants to complete multiple rounds at specified time intervals, for example.   references   bell, s. j. (2014) staying true to the core: designing the future academic library experience. portal: libraries and the academy, 14(3), 369–382. doi: 10.1353/pla.2014.0021   crowe, k., & bradshaw, a. k. (2016). taking a page from retail: secret shopping for academic libraries. evidence based library & information practice, 11(1), 40-56. doi:10.18438/b85s6h glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi: 10.1108/07378830610692154   halpern, r., eaker, c., jackson, j., & bouquin, d. (2015). #ditchthesurvey: expanding methodological diversity in lis research. in the library with the lead pipe, 1-13. retrieved from http://www.inthelibrarywiththeleadpipe.org    zorica, m. b., ivanjko, t., & spiranec, s. (2014). mystery shopping in libraries are we ready? qualitative & quantitative methods in libraries, 2, 433-442. retrieved from http://www.qqml.net/     microsoft word art4081 evidence based library and information practice 2008, 3:4 18 evidence based library and information practice article measuring the extent of the synonym problem in full-text searching jeffrey beall metadata librarian university of colorado denver denver, colorado, united states of america e-mail: jeffrey.beall@ucdenver.edu karen kafadar rudy professor of statistics college of arts and sciences, indiana university bloomington, indiana, united states of america e-mail: kkafadar@indiana.edu received: 03 september 2008 accepted: 23 october 2008 © 2008 beall and kafadar. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – this article measures the extent of the synonym problem in full-text searching. the synonym problem occurs when a search misses documents because the search was based on a synonym and not on a more familiar term. methods – we considered a sample of 90 single word synonym pairs and searched for each word in the pair, both singly and jointly, in the yahoo! database. we determined the number of web sites that were missed when only one but not the other term was included in the search field. results – depending upon how common the usage is of the synonym, the percentage of missed web sites can vary from almost 0% to almost 100%. when the search uses a very uncommon synonym ("diaconate"), a very high percentage of web pages can be missed (95%), versus the search using the more common term (only 9% are missed when searching web pages for the term "deacons"). if both evidence based library and information practice 2008, 3:4 19 terms in a word pair were nearly equal in usage ("cooks" and "chefs"), then a search on one term but not the other missed almost half the relevant web pages. conclusion – our results indicate great value for search engines to incorporate automatic synonym searching not only for user-specified terms but also for high usage synonyms. moreover, the results demonstrate the value of information retrieval systems that use controlled vocabularies and cross references to generate search results. introduction and context of the study full-text searching generates results by matching a word or words in a search query with words in a database. the synonym problem in full-text searching occurs when a searcher looks for information on a topic and enters a search using a single term to represent the topic but does not also enter any synonyms for that topic. for example, a search for information on dentures with only the word "dentures" as a search term could miss documents that refer to this concept by its synonym "false teeth", because the two terms have no words in common. for most full-text searching, “value-added” features such as controlled vocabularies and cross references are not present. these features serve to retrieve and co-locate documents on a given topic in search results regardless of the terms used in the full text of searched documents. this article seeks to measure the extent of the synonym problem in full-text searching. more precisely, this study looks at single word pairs of synonyms, and for each term measures the proportion of documents that are missed when one term is searched, and the proportion of documents that contain only the synonym. this study is limited to traditional full-text search engines, that is, search engines that match words in a search query with words in full-text documents and return results. full-text search engines and synonyms with the advent of the internet, full-text searching has proliferated, and with it the desire to retrieve as much information about a topic as possible. a problem that arises with such searches is the potential for the search to return only a subset of the web sites with relevant information because the search concept can be referenced by more than one term. the concept can be described by simple nouns ("false teeth" and "dentures"), or by broader terms, such as "botany" and "plant science", or "aurora borealis" and "northern lights". a search in most search engines on the term "botany" (or "aurora borealis") may well miss web pages that refer to the discipline only as "plant science" (or "northern lights"). a few authors have commented on this effect. for example, in the oxford guide to library research mann writes: when all is said and done, keyword searching necessarily entails the problem of the unpredictability of the many variant ways the same subject can be expressed, within a single language (“capital punishment,” “death penalty”) and across multiple languages (“peine de mort,” “pena capitale”). and no software algorithm will solve this problem when it is confined to dealing with only the actual words that it can retrieve from within the evidence based library and information practice 2008, 3:4 20 given documents (or citations or abstracts) themselves. (102) beall refers to this problem as the “synonym problem” and states, “in full-text searching, synonyms hinder effective information retrieval when a searcher enters a term in the search box and the system only returns results that match the term and does not return results that refer to the concept only by one of its synonyms” (“weaknesses” 439). for example, some use the term “botany” and others use “plant science” to describe the same concept. a search in most search engines on the term “botany” would probably miss web pages that refer only to the discipline as plant science (beall “death” 6). fugmann uses the term “paraphrase lexicalization” to describe the disconnect between a user's search terms and the terms used in relevant documents. he exemplifies the synonym problem by giving an example of a searcher looking for information on insecticides and missing documents that refer to them as pesticides. he states, “…an inquirer expects all documents to be retrieved in which the concept of the search request is dealt with and in fact independent of how it happens to have been expressed by an author” (223). dagan et al. describe the synonym problem from an information science perspective. their study “investigates conceptually and empirically the novel sense matching task, which requires [one] to recognize whether the senses of two synonymous words match in context” (449). they describe this phenomenon as “lexical substitution”. their study does not measure the synonym problem but attempts to lay the groundwork for an algorithmic solution to it. while only a few authors have noted the synonym problem, even fewer have attempted to measure it. the challenges to measuring the extent of the synonym problem include defining an appropriate measure, and designing a study to quantify it. to our knowledge, no previous study has been conducted to measure the synonym problem. this article attempts to fill that void. on its web page, google describes a “synonym search”, but it provides very little information about this type of search. on one of its help pages google states, “if you want to search not only for your search term but also for its synonyms, place the tilde sign ("~") immediately in front of your search term” (“web search help center”). we suspect that rather few google users are aware of this feature, and even fewer take advantage of it. google offers no further explanation of this feature. slightly more information is provided in the patent application granted to google in 2002 and issued in 2005 for a process that essentially functions as an algorithmic synonym search, rather than a deterministic synonym search (by matching synonyms from a preconstructed list). according to the patent's abstract: methods and apparatus determine equivalent descriptions for an information need. in one implementation, if adjacent entries in a query log contain common terms, the uncommon terms are identified as a candidate pair. the candidate pairs are assigned a score based on their frequency of occurrence, and pairs having a score exceeding a defined threshold are determined to be synonyms. (dean et al. 2005) we assume that the phrase “equivalent descriptions” here means "synonyms", but it is unclear whether google has implemented the process described in this patent into its evidence based library and information practice 2008, 3:4 21 current search algorithms. for proprietary reasons, search engine companies release very little information about the algorithms they employ to generate results. bade says “… the exact nature of the formulae used remains largely unknown to the public since these are valuable intellectual property for their owners” (831). at least one library online catalog product offers a synonym search feature. the innovative interfaces, inc. online catalog allows libraries to program in synonyms. once a synonym pair has been programmed into the system, a keyword search on either of the two words in the pair returns results as if both search terms had been entered. this feature is not used so much for synonyms as it is for variant spellings, such as british and american variants like “labor” and “labour”. methods our original plan was to generate a random sample of synonym groups and then to search them in both google and google book search. as our source for synonyms, we planned to use printed thesauri from the reference section in the auraria library on the campus of the university of colorado denver. after collecting the data, we planned to do a statistical analysis to answer our research question. difficulties with synonyms we soon realized that exact synonyms are rare, and words listed as synonyms in thesauri are close in meaning but frequently are not true synonyms. one example of a false pair of synonyms is the pair “waterfall” and “cascade”. while close in meaning, there is a significant semantic difference between these two terms. we sought to study synonym groups that were as semantically identical as possible. we suspected that the use of "non-exact synonyms" such as "waterfall" and "cascade" would result in even more missed web pages, and hence an even more severe problem than what we ultimately observed. difficulties with google before we began to collect data we performed numerous test searches, which immediately revealed two significant problems for conducting this research with google. the first problem was that the google search software does not allow nested boolean searching. that is, if a term contains more than one word, google will not allow a searcher to apply the boolean operator “not” to the phrase. this was a significant problem for us, because our study objective required us to search for one term but not the other. as an example, for the synonym pair “leprosy” and “hansen’s disease”, ideally we would perform the following search in google: leprosy -“hansen’s disease” the minus sign within google activates the boolean operator “not” in the search, and the quotation marks indicate a term to be searched as a phrase. unfortunately, the google search engine lacks the functionality to correctly perform this type of search. our test searches showed that when we tried to use nested boolean terms, the phrases we attempted to exclude often appeared in the pages retrieved by the search. this would prevent us from accurately measuring the number of resources missed due to the synonym problem. to address the difficulties with synonyms from thesauri, we abandoned printed thesauri as a source for a random selection of synonyms and turned instead to controlled vocabularies. controlled vocabularies also are frequently called thesauri; they list the preferred term for a concept followed by a list of the variant evidence based library and information practice 2008, 3:4 22 terms or "cross references". the library of congress subject headings is an example of a controlled vocabulary, and as one of the most comprehensive we selected this controlled vocabulary as the source for our random selection of synonyms. at this point we encountered our second major problem with google: an apparent inconsistency in the search results in the google database. one of the valuable features of the google database that benefits information retrieval research is that results of each search include the total number of web sites retrieved. however, as we were conducting our test searches, we found this count to be highly variable. in some cases, for example, the same search performed at two different times retrieved significantly different numbers of "web pages found”. we illustrate this problem with a more detailed description of our study design. for our study we required data on the number of web pages found from the following searches for each word pair, expressed in figure 1 as a venn diagram. the venn diagram in figure 1 expresses the following boolean logic: a not b (represented by only the area in the left circle that is shaded green) b not a (represented by only the area in the right circle that is shaded blue ) a and b (represented by the blue-green shaded area in the center) a or b (represented by the entire shaded area of the diagram) our study depends critically on the numbers of web pages found for each of these four searches. one would expect that the sum of the numbers of web pages found from the first three searches should equal the number of web pages found by the fourth search. however, in our test searches we observed discrepancies between these two results, sometimes as large as several million. indeed, the four individual numbers from the four searches often varied substantially. we postulated several explanations for the wide discrepancy. first, it could arise if google actually applies its patented "synonym search" feature described in its help pages. second, if google's cited "number of web pages found" is not a deterministic count, but rather is a statistical estimate based on the current version of the search algorithm being used, then one would expect variability in the estimate at different times. a third possible explanation arises from the fact that "every search in google is part of an experiment" (pregibon and lambert), so searches of the same query at different times may result in different algorithms being applied. finally, discrepancies could reflect actual changes in the number of web pages available due to new and deleted web pages over time. however, we suspect the numbers of new and deleted web pages on widely diverse topics would not vary much, fig. 1. a venn diagram that illustrates the data gathered for each word pair. evidence based library and information practice 2008, 3:4 23 casting doubt on this fourth possibility as a plausible explanation for the extreme variability we observed in our searches, some of which involved rather obscure terms. slight discrepancies would have been tolerable, but for our study such huge discrepancies rendered google searches too variable for our purposes. for this reason, we turned to alternative search engines. revised methods because of the inability of yahoo! (as well as google) to perform nested boolean searches, we decided to limit this study to only single word synonym pairs. (we would like to repeat this study on more complicated synonym-phrase pairs when a nested boolean search feature is implemented in one of the search engines.) we generated a random list of synonym word pairs from the library catalog at the auraria library (university of colorado denver). using the search functionality in the "staff" mode of the library's catalog, we created a list of all topical subject authority records that contained at least one cross reference. because the auraria library serves three institutions of higher education, including a comprehensive university, the scope of the headings in the library is unusually broad. our generated list contained 39,511 records. we then used a program available through the r project for statistical computing <http://www.r-project.org> to generate 100 random numbers distributed uniformly across the range [1, 39511], which identified the indices of the 39,511 records selected for this study. as indicated above, we limited our study to single word pairs of synonyms, meaning that both the heading and the cross reference had to be single words. we imposed two further conditions on the word pairs for this study to avoid the potential for the 100 pairs to include geographicor location-specific terms. the two conditions thus relate to the structure and composition of the library of congress subject headings (lcsh) thesaurus. first, we skipped records whose cross references were also cross references from another record. second, we insisted on semantically exact word pairs. the lcsh does group semantically related concepts on a single record. for example, the lcsh heading for "mountains" has a "see reference" for the word "hills". while similar, these two concepts are semantically different, even though lcsh groups them together on a single subject authority record for convenience. to impose the two further conditions, we eliminated a pair and went to the next record in the list of 39,511 if the main heading / cross reference pair (a) involved more than one word for either the main heading or the cross reference; (b) contained a cross reference that was itself a cross reference; or (c) contained terms that were not semantically exact. in ten instances, the random numbers were so close together that no valid single word synonym pair appeared between the previous and the next randomly-selected record. thus our final sample consisted of ninety pairs of single word synonyms. all searches were conducted by the first author (beall). results we applied our study plan to search 100 (later revised to 90) synonym, single word pairs in the yahoo! database. the word pairs and the data are presented in the appendix. the searches were conducted in march and april, 2007. when we gathered the data, we realized that the searches, like full-text searching, would not be perfect. for example, one of our synonym pairs was biologicals / biologics. it is likely that one of the terms is the name of a company, or is a word in a foreign language and in many contexts is not a synonym of the other, a situation that would affect our data. but evidence based library and information practice 2008, 3:4 24 there were far too many search results to examine to determine their context, and the type of searching we are studying, full-text searching, is also burdened by the same problem. we acknowledge this potential contamination by proper company names, but believe it to be quite small. the pertinent data from this study are the percentages of total references ("a and b") found by searching for "a only" (i.e., number of pages found in search for word a only, divided by the total number of pages found in a search for either ("a or b")) and likewise searching for “b only”. usually, one of a or b is the more common word, so the percentage for one will be higher, often substantially higher, than the percentage for the other word. figure 2 displays via boxplots the data for the more common of the words in the pair (max(%a,%b)), the data for the less common of the words in the pair (min(%a,%b)), and the difference in the two percentages (diff(max–min)). for convenience in this article, we will designate “a” as the more common word and “b” as the less common word (i.e., a search on “a” returned more web pages than a search on “b”). figure 3 displays information similar to the third box in figure 2, but on an item-by-item basis. for example, the highest percentage among these word pairs occurred for word pair #53: “mitochondria” but not “chondriosomes” found 99.992% of the 2,000,189 web pages, while “chondriosomes” but not “mitochondria” found only 0.006% of the web pages. the designated line in figure 3 connects these two proportions: 0.99992 (left side) and 0.00006 (right side). from this display it is clear that if one succeeds in identifying the more common word the search will yield most of the references, but if one asks for the less common word the search will miss almost all the web pages. for about 10-20 of the word pairs the words will each find about half of the available pages. for example, in word pair #72 each of the two searches, on "preparedness" only and on "readiness" only, returns about half of the total number of web pages, but also will completely miss the other half. if one happens to select the more common of the words in the pair, one is often likely to capture most of the references (on average, about 88% of the references), but in 10 of the 90 pairs a search for even the more common of the words in the pair returned less than 55% of the available web pages. see table 1 for the list of these 10 word pairs. in a search, the proportion of missed web pages depends on whether one searched the more common or the less common word in the synonym pair. in these 10 word pairs fig. 2. boxplot showing results missed from the perspective of the moreand less common word. evidence based library and information practice 2008, 3:4 25 even a search on the more common term returned less than 55% of the web pages found if both words were used in the search (i.e., 45% or more web pages were missed when using only one term in the word pair). how costly can the search be, in terms of missed web pages, if one were to search on the less common of the two words in the pair? figure 2 shows that, when the more common ("a") of the words is used in the search, often one captures 78% or more of the total web pages (the lower quartile of the percentages of web pages found using "a only" is 78%, as demonstrated by the lower edge on the left-most box in the boxplot). conversely, if one were so unlucky as to have selected the less common word, one is likely to capture no more than 20% of the web pages (the upper quartile of the percentages of web pages found using "b only" is 20%, as shown by the upper edge of the middle box in the boxplot). even when the more common word is searched, the left box in figure 2 shows that 25% of the searches returned only 50-78% of the available web pages. these results suggest that the "cost" of web searches for information about a topic can be rather high if one unfortunately enters the less common term, which may be frequent depending upon one's native language. (for example, australians often use the term "jumper" for the american term "sweater", and the british use the term "biscuits" for the american term "cookies".) the third box in figure 2 shows that the difference in "percentage of web pages found" can be very large -often as high as 50-95% (lower quartile and upper quartile) -depending on which word was selected for the search. figure 3 shows both percentages for each word pair (a, the more common, on the left; b, the less common, on the right), connected by a dashed line. often one of the words in the word pair is much more common than the other word. but for about one-fourth of the words in our study, both percentages are near 50%: a search for one term or the other fails to capture half of the web pages, regardless of whether one selected the "more" or "less" common word. even when the more common word is searched, the left box in figure 2 shows that 25% of the searches returned only 50-78% of the available web pages. these results suggest that the "cost" of web searches for information about a topic can be rather high if one unfortunately enters the less common term, which may be frequent depending upon one's native language. (for example, table 1 a b a (not b) b (not a) a and b a or b prop. max prop min 2 afrocentrism afrocentricity 57900 63800 1040 128000 49.8 45.2 20 cooks chefs 18500000 23600000 2200000 44200000 53.4 41.9 24 discrimination bias 44400000 35900000 2910000 83200000 53.4 43.1 26 egoism egocentricity 1150000 10600 980 3000000 38.3 3.5 27 electromagnetism electromagnetics 953000 750000 46500 1760000 54.1 42.6 50 marmots marmota 325000 299000 8900 645000 50.4 46.4 69 picornaviruses picornaviridae 35400 39200 2360 84200 46.6 42.0 72 preparedness readiness 17200000 16500000 986000 39000000 44.1 42.3 81 salafiyah salafiyya 17500 12100 66 38700 45.2 31.3 93 tinsmithing tinwork 16200 13900 79 41700 38.8 33.3 99 waka tanka 1530000 1580000 10700 3110000 50.8 49.2 evidence based library and information practice 2008, 3:4 26 australians often use the term "jumper" for the american term "sweater", and the british use the term "biscuits" for the american term "cookies".) the third box in figure 2 shows that the difference in "percentage of web pages found" can be very large -often as high as 50-95% (lower quartile and upper quartile) -depending on which word was selected for the search. figure 3 shows both percentages for each word pair (a, the more common, on the left; b, the less common, on the right), connected by a dashed line. often one of the words in the word pair is much more common than the other word. but for about one-fourth of the words in our study, both percentages are near 50%: a search for one term or the other fails to capture half of the web pages, regardless of whether one selected the "more" or "less" common word. discussion while small in scope, this study demonstrates the severity of the synonym problem in web searching. because of cultural or sociological differences in terms, the use of one term instead of its more common counterpart could result in highly incomplete web searches, raising only a fraction of the available web pages on this fig. 3. display of proportions of web pages found when searching on "more common word" (left side) versus "less common word" (right side). segments connect proportions. the greatest discrepancy occurs with word pair #53, "mitochondria" (0.99992) versus "chondriosomes" (0.00006). the least discrepancy occurs with word pairs #72 ("preparedness", 0.496; "readiness", 0.476) and #99 ("tanka", 0.507; "waka", 0.490). evidence based library and information practice 2008, 3:4 27 topic. for example, our study included the word pair “appraisers/assessors”; the former term is more common in some societies (73%), while the latter is more familiar in other contexts (but which captures only 26% of the web pages found by using both terms). for other word pairs, both words are used roughly equally often, but not in the same document, and hence a search on either word, but not the other, misses almost half the web pages found by searching on both (e.g., preparedness 49.6%: readiness 47.6%). some search engines (such as google inc.) appear to offer synonymsearching capability, and based on our study, such a feature would result in more complete searches. this study involves some important limitations which we need to acknowledge. first, the study is limited in its sample size. we selected only 100 pairs and our study design yielded data on only 90 word pairs, which is not a huge study but is definitely large enough to demonstrate the variability that can arise with the synonym problem. in addition, the uncertainties on the estimates of the reported percentages of missed web pages with each word in the search pair includes the uncertainties in the algorithms used by the search engine. one must keep in mind the extent that the search engine algorithms themselves are based on some sort of sampling strategy that returns estimates on "approximate number of web fig. 4. the fewer the hits in a search, the more precise the estimate of number of web pages found. evidence based library and information practice 2008, 3:4 28 pages found", which then affect our reported percentages. clearly a larger study that involves replication is warranted to yield better estimates of the variability in the percentages reported here. in a few instances, the data returned were illogical, in that the number of web pages found from a search of (a or b) exceeded the sum of the numbers of web pages found from the three searches combined (a not b) + (b not a) + (a and b). two factors probably contributed to such events. first, the distributed system architecture can change quickly and repeatedly, resulting in different values at different times. second, the search engine reports only an estimate, not a precise value, of the number of web pages found. these estimates are less precise for larger numbers of web pages found, as illustrated in figure 4. ideally, the difference between n1 = #web pages found from "(a or b)" and n2 = #web pages found from "(a not b) + (b not a) + (a and b)" should be zero. figure 4 shows the logarithm (base 10) of the absolute value of this difference, log(|n1-n2|), versus log(n2). as the number of web pages found increases, the discrepancy between n1 and n2 grows, with large n2 (on the order of 100 million web pages) resulting in discrepancies of over 1 million. for the most part, though, this figure shows that the discrepancy is usually less than 1%, but can sometimes be as large as 10%. as search engine algorithms improve, we expect fewer large discrepancies of this type. this study attempts to address the extent of the synonym problem by comparing the numbers of web pages found by only one of the two words in a synonym word-pair but not the other word. however, a user's main interest may be in capturing not the total number of web sites for a given concept but rather the number of most relevant web sites. the results from this type of study would indeed be interesting, but we see two immediate problems in attempting to conduct such a study. first, one would have to define what is meant by "most relevant". the easiest definition would be "top 25 web sites", but some of those "top 25" could be duplicates, irrelevant, non-authoritative, or paid by advertisers. moreover, human subjectivity would be involved in assessing "relevance". at some future time, search engines may offer functionalities that would reduce the human effort in this timeintensive, possibly subjective, laborious process, and we would consider such a study at that point. another issue may be whether our study measured "semantic exactness" rather than "extent of the synonym problem". our criteria for word-pair synonyms in this study included one criterion that was aimed at achieving a high degree of homogeneity in semantic exactness, but this criterion did involve some human judgment. the confounding of these two concepts, "semantic exactness" and "extent of synonymy", may be difficult to resolve with present technology. conclusion the extent of the synonym problem in fulltext searching depends on whether one searches the more common of the synonyms. overall, the measure of what’s missed is as high as 30% in a large (90%) fraction of common word-pairs. information discovery systems need to take the synonym problem into account and develop solutions for it, both probabilistic and deterministic. this study should be repeated with a wider and more systematic variety of synonym pairs from defined subject areas; searches that include phrases instead of single words in the pairs; replication, to determine the evidence based library and information practice 2008, 3:4 29 variability in the reported percentages; and more search engines. the methodology here could result in the establishment of a benchmark data set against which various search engines can evaluate their search algorithms in terms of their ability to minimize the synonym problem. additionally, the data demonstrate the value of vocabulary control and cross references in providing more precise search results. acknowledgements: the authors wish to thank the associate editor and two anonymous referees for their thoughtful comments on an earlier version of this manuscript. this work was prepared in part with support from army research office, grant #w911nf0510490, awarded to the university of colorado denver (kafadar). works cited bade, david. “relevance ranking is not relevance ranking or, when the user is not the user, the search results are not the search results.” online information review 31.6 (2007): 831-44. beall, jeffrey. "the weaknesses of full-text searching." journal of academic librarianship 34.5 (2008): 438-44. ---. "the death of full-text searching." pnla quarterly 70.2 (2006): 5-6. dagan, ido, oren glickman, alfio gliozzo, efrat marmorshtein, and carlo strapparava. “direct word sense matching for lexical substitution.” coling-acl 2006: proceedings of the 21st international conference on computational linguistics and 44th annual meeting of the acl, sydney, australia, 17-21 jul. 2006: 449-56. 11 nov. 2008 <http://eprints.pascalnetwork.org/archive/00002759/01/p0 6-1057.pdf>. dean, jeffrey a., georges harik, benedict gomes, and noam shazeer. methods and apparatus for determining equivalent descriptions for an information need. google inc., assignee. patent 6,941,293. 6 sep. 2005. fugmann, robert. “the complementarity of natural and controlled languages in indexing,” subject indexing: principles and practices in the 90's : proceedings of the ifla satellite meeting held in lisbon, portugal, 17-18 august 1993, and sponsored by the ifla section on classification and indexing and the instituto da biblioteca nacional e do livro, lisbon, portugal. eds. robert p. holley, et al. munich: saur, 1995. 215-30. mann, thomas. the oxford guide to library research. 3rd ed. oxford: oxford up, 2005. pregibon, daryl, and diane lambert. "understanding online advertisers." joint statistical meeting, denver, co, usa, 5 august 2008. “web search help center.” google. 2008. 11 nov. 2008 <http://www.google.com/help/refin esearch.html>. evidence based library and information practice 2008, 3:4 30 appendix table 2 the data collected in this study. “a” is designated as the more common word in the synonym pair, “b” as the less common word. number terms max min a (not b) b (not a) a and b a or b 1. a. adivasis b. adibasis 0.9979 0.0016 109,000 177 50 110,000 2. a. afrocentricity b. afrocentrism 0.5198 0.4717 63,800 57,900 1,040 128,000 3. a. aluminum b. aluminium 0.6850 0.2853 63,400,000 26,400,000 2,750,000 92,500,000 4. a. anomie b. anomy 0.8556 0.1429 443,000 74,000 776 521,000 5. a. appraisers b. assessors 0.7258 0.2597 6,400,000 2,290,000 128,000 8,850,000 6. a. arctiidae b. lithosiidae 0.9971 0.0016 85,200 138 106 85,800 7. a. arthropods b. arthropoda 0.6061 0.2905 1,400,000 671,000 239,000 2,310,000 8. a. berberis b. barberries 0.9407 0.0565 368,000 22,100 1,090 393,000 9. a. b. 10. a. biologics b. biologicals 0.7582 0.2275 2,290,000 687,000 43,200 3,030,000 11. a. bleaching b. blanching 0.9281 0.0715 3,390,000 261,000 1,510 3,660,000 12. a. buddhists b. lamaists 0.9998 0.0001 3,070,000 386 276 3,090,000 13. a. b. 14. a. bullying b. bullyism 0.9999 mm 10,100,000 984 160 10,100,000 15. a. cachexia b. cachexy 0.9899 0.0069 245,000 1,720 784 253,000 16. a. cannibalism b.anthropophagy 0.9946 0.0044 2,440,000 10,800 2,330 2,470,000 17. a. catalans b. catalonians 0.9941 0.0056 2,100,000 11,900 643 2,130,000 18. a. b. 19. a. chimneys b. smokestacks 0.8340 0.1627 2,820,000 550,000 11,200 3,410,000 20. a. chefs b. cooks 0.5327 0.4176 23,600,000 18,500,000 2,200,000 44,200,000 21. a. deacons b. diaconate 0.9085 0.0578 2,720,000 173,000 101,000 3,000,000 22. a. deburring b. burring 0.8388 0.1557 513,000 95,200 3,410 618,000 23. a. b. evidence based library and information practice 2008, 3:4 31 24. a. discrimination b. bias 0.5336 0.4314 44,400,000 35,900,000 2,910,000 83,200,000 25. a. dreams b. dreaming 0.8367 0.1246 133,000,000 19,800,000 6,160,000 159,000,000 26. a. egoism b. egocentricity 0.9149 0.0843 1,100,000 93,000 978 190,000 27. a.electromagnetism b. electromagnetics 0.5447 0.4287 953,000 750,000 46,500 1,760,000 28. a. embezzlement b. defalcation 0.9738 0.0230 2,460,000 58,200 7,900 2,540,000 29. a. errors b. mistakes 0.7010 0.2570 162,000,000 59,400,000 9,690,000 231,000,000 30. a. eurocentrism b. eurocentricity 0.9806 0.0178 104,000 1,890 171 113,000 31. a. eviction b. dispossession 0.8953 0.1005 4,590,000 515,000 21,800 5,120,000 32. a. extraversion b. extroversion 0.6307 0.3372 432,000 231,000 22,000 684,000 33. a. faience b. fayence 0.7405 0.2580 861,000 300,000 1,790 1,170,000 34. a. fasteners b. fastenings 0.9559 0.0373 12,100,000 472,000 86,800 12,700,000 35. a. fireworks b. pyrotechnics 0.9453 0.0402 32,200,000 1,370,000 495,000 34,200,000 36. a. forearm b. antebrachium 0.9996 0.0003 4,510,000 1,310 578 4,500,000 37. a. formaldehyde b. formalin 0.7469 0.2196 2,500,000 735,000 112,000 3,350,000 38. a. gelatin b. gelatine 0.7867 0.2003 4,360,000 1,110,000 72,400 5,550,000 39. a. greenhouses b. hothouses 0.9841 0.0152 5,090,000 78,600 3,390 5,170,000 40. a. gums b. gingiva 0.9664 0.0263 4,780,000 130,000 36,200 4,940,000 41. a. heme b. hematin 0.9862 0.0119 975,000 11,800 1,800 995,000 42. a. hydrogeology b. geohydrology 0.9327 0.0525 917,000 51,600 14,600 978,000 43. a. intellectuals b. intelligentsia 0.8409 0.1303 7,810,000 1,210,000 268,000 8,710,000 44. a. ischemia b. ischaemia 0.8551 0.0967 2,060,000 233,000 116,000 2,380,000 45. a. kayasthas b. kayasths 0.8249 0.1613 1,790 350 30 2,420 46. a. kimchi b. kimchee 0.7670 0.2070 930,000 251,000 31,500 1,220,000 47. a. lakes b. lochs 0.9858 0.0130 73,400,000 731,000 90,200 73,100,000 48. a. larrea b. covillea 0.9985 0.0003 568,000 133 82 558,000 49. a. libertinage b. libertinism 0.8570 0.1421 468,000 77,600 460 563,000 50. a. marmots b. marmota 0.5135 0.4724 325,000 299,000 8,900 645,000 51. a. 0.9999 0.0001 80,700 5 1 80,700 evidence based library and information practice 2008, 3:4 32 mechanoreceptors b. mechanicoreceptors 52. a. micropipettes b. micropipets 0.9662 0.0321 72,500 2,410 124 81,300 53. a. mitochondria b. chondriosomes 0.9999 0.0001 2,000,000 144 45 2,000,000 54. a. monazite b. cryptolite 0.9979 0.0018 112,000 207 24 113,000 55. a. mutuality b. mutualism 0.7695 0.2295 798,000 238,000 1,070 1,050,000 56. a. natriuresis b. natruresis 0.9982 0.0012 59,600 71 35 60,000 57. a. norsemen b. northmen 0.7713 0.2206 465,000 133,000 4,860 613,000 58. a. ochre b. ocher 0.8799 0.1151 1,460,000 191,000 8,260 1,660,000 59. a. ointments b. salves 0.6694 0.2958 1,430,000 632,000 74,300 2,140,000 60. a. ontogeny b. ontogenesis 0.8210 0.1428 638,000 111,000 28,100 778,000 61. a. organotherapy b. opotherapy 0.8937 0.066 2,750 203 124 3,420 62. a. b. 63. a. b. 64. a. paramecium b. paramaecium 0.9282 0.0712 314,000 24,100 207 339,000 65. a. parsis b. parsees 0.7687 0.2228 177,000 51,300 1,960 237,000 66. a. pediatrics b. paediatrics 0.9002 0.0806 18,200,000 1,630,000 388,000 20,200,000 67. a. perimenopause b. premenopause 0.8164 0.1475 631,000 114,000 27,900 772,000 68. a. photogravure b. heliogravure 0.8237 0.1579 395,000 75,700 8,840 487,000 69. a. picornaviridae b. picornaviruses 0.5094 0.4600 39,200 35,400 2,360 84,200 70. a. pollination b. pollinization 0.9988 0.001 2,080,000 2,070 382 2,100,000 71. a. porpoises b. phocoenidae 0.9720 0.0252 709,000 18,400 2,000 773,000 72. a. preparedness b. readiness 0.4959 0.4757 17,200,000 16,500,000 986,000 39,000,000 73. a. b. 74. a. b. 75. a. procellariiformes b. tubinares 0.9599 0.0293 55,700 1,700 627 60,300 76. a. promethium b. illinium 0.9933 0.0053 137,000 735 185 144,000 77. a. radiologists b. roentgenologists 0.9998 0.0001 1,740,000 221 133 1,770,000 evidence based library and information practice 2008, 3:4 33 78. a. religiosity b. religiousness 0.8444 0.138 1,040,000 170,000 21,600 1,240,000 79. a. rodents b. rodentia 0.9456 0.0407 6,070,000 261,000 88,000 6,420,000 80. a. sago b. sagu 0.8664 0.1319 1,340,000 204,000 2,690 1,550,000 81. a. salafi ̄yah b. salafiyya 0.5899 0.4079 17,500 12,100 66 38,700 82. a. metalloids b. semimetals 0.6761 0.3187 77,000 36,300 595 116,000 83. a. b. 84. a. shepherds b. sheepherders 0.9855 0.0143 6,500,000 94,000 1,610 6,600,000 85. a. b. 86. a. shrews b. soricidae 0.8917 0.0895 485,000 48,700 10,200 551,000 87. a. skunks b. mephitidae 0.9983 0.0008 1,270,000 980 1,210 1,280,000 88. a. slavists b. slavicists 0.9725 0.0253 27,100 704 61 30,400 89. a. somite b. metamere 0.9646 0.0317 76,400 2,510 294 85,300 90. a. spires b. steeples 0.8532 0.1362 2,950,000 471,000 36,400 3,460,000 91. a. stigmata b. stigmatization 0.7671 0.2323 1,420,000 430,000 1,080 1,860,000 92. a. summer b. summertime 0.9779 0.0135 396,000,000 5,480,000 3,470,000 403,000,000 93. a. tinsmithing b. tinwork 0.5368 0.4606 16,200 13,900 79 41,700 94. a. trilobites b. trilobita 0.9052 0.0841 339,000 31,500 3,990 384,000 95. a. urea b. carbamide 0.9579 0.0356 3,790,000 141,000 25,500 3,970,000 96. a. vietnamese b. annamese 0.9997 0.0002 45,000,000 10,100 8,800 45,200,000 97. a. violin b. fiddle 0.6499 0.2931 20,400,000 9,200,000 1,790,000 31,500,000 98. a. virilization b. virilism 0.8697 0.1218 67,500 9,450 665 82,800 99. a. tanka b. waka 0.5063 0.4903 1,580,000 1,530,000 10,700 3,110,000 100. a. wrasses b. labridae 0.6217 0.3072 118,000 58,300 13,500 196,000 evidence based library and information practice editorial responsibilities      2016. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     editor-in-chief: lorie kloda   associate editor (evidence summaries): heather pretty   associate editor (articles): rebekah (becky) willson, lisl zach   associate editor (classics): jonathan eldredge   associate editor (reviews): denise koufogiannakis   associate editor (features, commentaries, using evidence in practice, research in practice, news): lorie kloda   production editor: michelle dunaway   editorial intern: melissa griffiths   editorial advisors: lindsay alcock, alison brettle, katrine mallan, pam ryan   copyeditors: richard hayman (lead copyeditor), georgianne bordner, bryan chan, julie evener, heather healy, stacey penney, maria tan, mary virginia taylor, nikki tummon, alison yeoman   indexing support: meral alakus   evidence summary   valued academic library services are not necessarily the ones that are used most frequently, students’ service and social media communication priorities should also be considered   a review of: stvilia, b., & gibradze, l. (2017). examining undergraduate students' priorities for academic library services and social media communication. the journal of academic librarianship, 43(3), 257-262. https://doi.org/10.1016/j.acalib.2017.02.013   reviewed by: michelle dubroy sciences librarian griffith university southport, queensland, australia e-mail: m.dubroy@griffith.edu.au   received: 26 june 2018  accepted: 12 aug. 2018      2018 dubroy. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29463     abstract   objective – to examine how undergraduate students rate the importance of different categories of library services and library social media postings.   design – online survey.   setting – large research university in the united states.   subjects – 159 undergraduate students enrolled in 3 information technology classes.   methods – participants were asked to rate the importance of different library service categories on a 7-point likert scale. the library service categories were (1) access to information and computer resources, (2) study support services, (3) support for club meetings, and (4) q&a services. participants were also asked to rate the importance of nine different categories of library social media postings, also on a 7-point likert scale. the categories of social media postings were (1) event, (2) resources, (3) community building, (4) operations updates, (5) study support, (6) q&a, (7) survey, (8) staff, and (9) club. students were also asked to identify which library services they currently use.   main results – validly submitted surveys totaled 104 (response rate 65%). respondents rated access to information and computer resources (m=5.9) and study support services (m=5.9) as being of the highest importance, with no statistically significant difference being found between these ratings. respondents rated q&a services (mean not reported) and support for club meetings (m=4.8) as being of significantly lower importance than the baseline (access information and computer resources). in terms of service usage, using the library to study (87%) and to access information and computer resources (59%), were the top two most reportedly used services.   respondents rated social media postings relating to operations updates (m=5.6), study support (m=5.5) and events (m=5.4) as being of highest importance, with no significant difference between the ratings of these three categories. respondents rated all other categories of social media postings (survey, m=4.7; staff, m=4.4; means for remaining categories not reported) as being of significantly less importance than the baseline (operations updates). for just over half the social media posting categories (5/9, 56%) importance rankings found in this study agree with engagement rankings the authors found in a previous study (stvilia & gibradze, 2014).   conclusion – the results of this study suggested frequency of use alone cannot be used to determine the value students place on a library’s services, as students may perceive equal value in services they use at different frequencies. the authors, therefore, argued there is a strong need to inexpensively predict users’ perceptions of service value without relying on usage metrics alone. because a level of agreement was found between social media engagement (determined in the authors’ 2014 study) and importance rankings (found in this study), the authors proposed further research be done to determine whether and how an analysis of library social media engagement can be used as an inexpensive way to predict the perceived importance and value of a library’s services. while the authors recognized it may not be appropriate to generalize the results of this study to a wider student population, they suggested the findings may be applicable to similar groups of students (i.e., undergraduate information technology students).   commentary   as higher education institutions acquire a more corporatized culture, academic libraries increasingly need to demonstrate their value and contribution to strategic objectives (oakleaf, 2010). value can be defined as a customer’s “overall assessment of the utility of a product based on perceptions of what is received and what is given” (zeithaml, 1988, p. 14). the perceived importance of a service plays a role in this assessment, with consumers being willing to give more and expecting to receive higher quality when procuring services which are important to them (ostrom & iacobucci, 1995). the authors of this study, therefore, sought to contribute evidence of academic library users’ service priorities as a way to ultimately infer the value users perceive in these services.   the study was reviewed using two critical appraisal tools (boynton & greenhalgh, 2004; glynn, 2006) and a number of strengths and weaknesses were found. a copy of the survey instrument was not included in the publication, but it was reportedly pretested by a small group of students. informed consent was obtained.   the authors used an ordered logistic regression analysis to determine the significance of any differences between average importance ratings, and this ensured their conclusions were not based on insignificant findings. the authors did not analyze how demographic factors may have affected respondents’ priorities, nor did they discuss the potential presence of self-selection bias. further, service categories were not mapped to social media post categories, so the connection between these is unclear.   while the authors discovered some high-level insights, the survey questions were not posed in a way that elicited precise responses. the service and social media posting categories were broad, uneven and included both academic and non-academically focused services. it is not known what subset of services within a category led respondents to rate that category as important or unimportant. further, importance was a subjective measure that was not defined by the authors or placed in relation to a specific outcome. a student might have rated a service as being important, but it is unclear why that service is important to them, to what outcome they believe it contributes (e.g., academic achievement, sense of belonging), and what it would mean to the student if the service was no longer offered.   this study explored one approach for understanding the value clients place on a library’s services. valued services are not necessarily the ones that are used most frequently, so libraries need to ask their clients about their priorities, and then promote and deliver services that matter. more specific practice implications are limited, however, and greater benefit could be obtained through defining more nuanced service categories and by exploring students’ service priorities in relation to specific outcomes or objectives.   references   boynton, p. m., & greenhalgh, t. (2004). selecting, designing, and developing your questionnaire. bmj, 328(7451), 1312-1315. https://doi.org/10.1136/bmj.328.7451.1312   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154   oakleaf, m. (2010). the value of academic libraries: a comprehensive research review and report. retrieved from acrl value of academic libraries website: http://www.ala.org/acrl/sites/ala.org.acrl/files/content/issues/value/val_report.pdf   ostrom, a., & iacobucci, d. (1995). consumer trade-offs and the evaluation of services. journal of marketing, 59(1), 17-28. https://doi.org/10.2307/1252011   stvilia, b., & gibradze, l. (2014). what do academic libraries tweet about, and what makes a library tweet useful? library & information science research, 36(3), 136-141. https://doi.org/10.1016/j.lisr.2014.07.001   zeithaml, v. (1988). consumer perceptions of price, quality, and value: a means-end model and synthesis of evidence. journal of marketing, 52(3), 2-22. https://doi.org/10.2307/1251446   evidence summary   making life easier for the visually impaired web searcher: it is now clearer how this should and can be done, but implementation lags   a review of: sahib, n. g., tombros, a., & stockman, t. (2012). a comparative analysis of the information-seeking behavior of visually impaired and sighted searchers. journal of the american society for information science and technology, 63(2), 377–391. doi: 10.1002/asi.21696   reviewed by: r. laval hunsucker information and collection specialist emeritus university libraries, universiteit van amsterdam amsterdam; silversteyn 80, breukelen, the netherlands email: amoinsde@yahoo.com   received: 1 dec. 2012     accepted: 10 feb. 2013      2013 hunsucker. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine how the behaviour of visually impaired persons significantly differs from that of sighted persons in the carrying out of complex search tasks on the internet.   design – a comparative observational user study, plus semi-structured interviews.   setting – not specified.   subjects – 15 sighted and 15 visually impaired persons, all of them experienced and frequent internet search engine users, of both sexes and varying in age from early twenties to mid-fifties.   methods – the subjects carried out self-selected complex search tasks on their own equipment and in their own familiar environments. the investigators observed this activity to some extent directly, but for the most part via video camera, through use of a screen-sharing facility, or with screen-capture software. they distinguished four stages of search task activity: query formulation, search results exploration, query reformulation, and search results management. the visually impaired participants, of whom 13 were totally blind and two had only marginal vision, were all working with text-to-speech screen readers and depended exclusively for all their observed activity on those applications’ auditory output. for data analysis, the investigators devised a grounded-theory-based coding scheme. they employed a search log format for deriving further quantitative data which they later controlled for statistical significance (two-tailed unpaired t-test; p < 0.05). the interviews allowed them to document, in particular, how the visually impaired subjects themselves subsequently accounted for, interpreted, and vindicated various observed aspects of their searching behaviour.   main results – the investigators found significant differences between the sighted participants’ search behaviour and that of the visually impaired searchers. the latter displayed a clearly less “orienteering” (o'day & jeffries, 1993) disposition and style, more often starting out with already relatively long and comprehensive combinations of relatively precise search terms; “their queries were more expressive” (p. 386). they submitted fewer follow-up queries, and were considerably less inclined to attempt query reformulation. they were aiming to achieve a satisfactory search outcome in a single step. nevertheless, they rarely employed advanced operators, and made far less use (in only 4 instances) of their search engine’s query-support features than did the sighted searchers (37 instances). fewer of them (13%) ventured beyond the first page of the results returned for their query by the search engine than was the case among the sighted searchers (43%). they viewed fewer (a mean of 4.27, as opposed to 13.40) retrieved pages, and they visited fewer external links (6 visits by 4 visually impaired searchers, compared with 34 visits by 11 sighted searchers). the visually impaired participants more frequently engaged in note taking than did the sighted participants.   the visually impaired searchers were in some cases, the investigators discovered, unaware of search engine facilities or searching tactics which might have improved their search outcomes. yet even when they were aware of these, they very often chose not to employ them because doing so via their screen readers would have cost them more time and effort than they were willing to expend. in general, they were more diffident and less resourceful than the sighted searchers, and had more trust in the innate capacity and reliability of their search engine to return in an efficient manner the best available results.   conclusion – despite certain inherent limitations of the present study (the relatively small sample sizes and the non-randomness of the purposive sighted-searcher sample, the possible presence of extraneous variables, the impossibility of entirely ruling out familiarity bias), its findings strongly support the conclusion that working with today’s search engine user interfaces through the intermediation of currently available assistive technologies necessarily imposes severe limits on the degree to which visually impaired persons can efficiently search the web for information relevant to their needs. the findings furthermore suggest that there are various measures that it would be possible to take toward alleviating the situation, in the form of further improvements to retrieval systems, to search interfaces, and to text-to-speech screen readers. such improvements would include:   more accessible system hints to support a better, and less cognitively intensive, query formulation; web page layouts which are more suitable to screen-reader intermediation; a results presentation which more readily facilitates browsing and exploratory behaviour, preferably including auditory previews and overviews; presentation formats which allow for quicker and more accurate relevance judgments; mechanisms for (a better) monitoring of search progress.   in any event, further information behaviour studies ought now to be conducted, with the specific aim of more closely informing the development of user interfaces which will offer the kind of support that visually impaired internet searchers are most in need of. success in this undertaking will ultimately contribute to the further empowerment of visually disabled persons and thereby facilitate efforts to combat social exclusion.     commentary   the last 15 years have witnessed the appearance of a very considerable, and now in fact quickly growing, number of publications dealing with the problems and requirements of visually impaired persons in the context of searching for, selecting, and making use of information and resources on the web. the study reviewed here confirms numerous earlier research findings and furthers, in particular, our understanding of how these users interact with search systems, while more fully exploring to what extent, and for what reasons, this interaction is significantly different from that which one observes among searchers who are not visually impaired. it is a useful contribution, as well, because it again explicitly focuses our attention on what could be done to render internet searching less time-consuming and cognitively burdensome, and accordingly more rewarding, for visually impaired individuals.   aside from the inherent limitations of the research design, as conceded above, the study’s presentation also displays certain shortcomings. quite remarkably and inexplicably, the researchers tell us nothing at all about the setting in which they conducted their study. furthermore, they specify neither what the sampling frame was for the random sample of visually impaired searchers, nor exactly what population(s) the samples were meant to represent. also, the demographic information which they provide is surprisingly limited; we find here no indications, for example, of educational level, of socio-economic status, or of ethnic or cultural identity. it is, therefore, even apart from any lingering uncertainty regarding the internal validity or the reliability of this research, easily imaginable that at least some readers will, justifiably, feel unsure as to just how pertinent the study’s findings actually are within their own specific environments. we should also note that, while the authors do review and cite some of the important earlier research on their topic, they reference none of the relevant non-english-language literature. even then, remarkable omissions remain. we encounter here, for example, no mention of the very interesting and still highly pertinent study by theofanos and redish (2003). moreover, our present authors’ lengthy “discussion” section leaves the reader largely in doubt as to which of the conclusions there being drawn are in fact based specifically on their study’s own new findings, and which on the whole accumulated body of research up to and including this study. however that may be, the conclusions themselves, along with the recommendations which accompany them, strike this reviewer not only as justified, but indeed as having clear and compelling implications, possibly even as amounting to a mandate of sorts, for assistive technology designers as well as for search engine interface developers, if not indeed for website developers in general. mates (2012) has, after all, recently written: “a disconcerting fact is that many websites and applications are becoming less accessible rather than more” (p. 12). and this is in spite of the fact that many of the proposed approaches to usability improvement are already well understood, and would be relatively easy to follow through on. the findings of the present study are of course in themselves neither generalizable nor necessarily transferable, but they are, taken together with those of related research, distinctly indicative of what steps are possible and appropriate, and therefore these findings do have practical evidentiary value.   nowhere in this article do we find any mention of a possible role for the library or information science and services (lis) professional, or indeed any suggestion what role human or institutional intermediation of any kind can fulfill in making things easier for visually impaired searchers. the researchers look solely to software enhancements for whatever solutions to the existing problems may be available. nevertheless, lis practitioners – public services librarians and library web services developers in particular, as well as anyone involved in accessibility evaluation – would be well advised take notice of, and to take into account, findings such as those emerging from this study, while decidedly also keeping an eye out for any promising fresh developments, such as certainly for example some published too recently to be mentioned in the article here under review: for example yang, hwang, and schenkman’s (2012) experimental “specialized search engine for the blind”; important new research on the growing accessibility problems associated with dynamically changing webpages (brown, jay, chen, & harper, 2012); innovative software approaches like behaviour-driven transcoding (lunn, harper, & bechhofer, 2011) or a prototype webpage restructuring system (guercio, stirbens, williams, & haiber, 2011); and, by no means least of all, kerkmann and lewandowski’s (2012) proposed accessibility evaluation framework. lis professionals and their organizations can pride themselves on a venerable tradition of striving to ensure broad and efficient access to information, literature, and recorded knowledge for all, regardless of disability. staying abreast of the results produced by this kind of research on accessibility enhancement will ensure that the practitioner remains aware of a valuable pool of potential evidence on which he or she can draw in making decisions which will serve collectively to sustain, perhaps indeed to strengthen, that tradition.     references   brown, a., jay, c., chen, a. q., & harper, s. (2012). the uptake of web 2.0 technologies, and its impact on visually disabled users. universal access in the information society, 11(2), 185-199. doi: 10.1007/s10209-011-0251-y.   guercio, a., stirbens, k. a., williams, j., & haiber, c. (2011). addressing challenges in web accessibility for the blind and visually impaired. international journal of distance education technologies, 9(4), 1-13. doi: 10.4018/ijdet.2011100101.   kerkmann, f., & lewandowski, d. (2012). accessibility of web search engines: towards a deeper understanding of barriers for people with disabilities. library review, 61(8/9), 608-621. doi: 10.1108/00242531211292105.   lunn, d., harper, s., & bechhofer, s. (2011). identifying behavioral strategies of visually impaired users to improve access to web content. acm transactions on accessible computing, 3(4), article no. 13. doi: 10.1145/1952388.1952390.   mates, b. t. (2012). information power to all patrons. in c. booth (ed.), making libraries accessible: adaptive design and assistive technology (= library technology reports 48:7) (pp. 7-13). chicago: american library association. doi: 10.5860/ltr.48n7.   o'day, v. l., & jeffries, r. (1993). orienteering in an information landscape: how information seekers get from here to there. in s. ashlund (ed.), bridges between worlds: conference on human factors in computing systems: interact '93 and chi '93: amsterdam, the netherlands, 24-29 april 1993 (pp. 438-445). new york: acm press. doi: 10.1145/169059.169365.   theofanos, m. f., & redish, j. (2003). guidelines for accessible – and usable – web sites: observing users who work with screenreaders. interactions, 10(6), 38-51. doi: 10.1145/947226.947227. expanded “authors’ version” retrieved 4 feb. 2013 from http://www.redish.net/content/papers/interactions.html    yang, y.-f., hwang, s.-l., & schenkman, b. (2012). an improved web search engine for visually impaired users. universal access in the information society, 11(2), 113-124. doi: 10.1007/s10209-011-0250-z.   microsoft word art1838 evidence based library and information practice 2008, 3:4 4 evidence based library and information practice article the information seeking behavior of undergraduate education majors: does library instruction play a role? jason martin assistant librarian, university of central florida libraries curriculum materials center orlando, florida, united states of america e-mail: mjmartin@mail.ucf.edu received: 23 july 2008 accepted: 09 october 2008 © 2008 martin. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – this study investigated the information seeking behavior of undergraduate majors to gain a better understanding of where they find their research information (academic vs. non-academic sources) and to determine if library instruction had any impact on the types of sources used. methods – the study used a convenience sample of 200 students currently enrolled as undergraduates at the university of central florida’s college of education. a chi square test of association was conducted to determine if the proportion of undergraduate education majors who use academic sources as compared to non-academic sources varied depending on whether the students had attended at least one library instruction session. results – the majority of students surveyed find their research information on the freely available web, even though they admit that academic sources are more credible. at an alpha level of .05, types of sources used for research were not statistically significantly related to whether the student attended library instruction sessions (pearson χ2 (1, n = 200) = 1.612, p = .447, cramer’s v = .090). conclusion – these results are supported by other studies that indicate that today’s college students are using freely available internet sites much more than evidence based library and information practice 2008, 3:4 5 library resources. little to no association appears to exist between “one-shot” library instruction sessions and the sources used by students in their research. serious consideration needs to be given to multiple library instruction sessions and to for-credit library courses over one-shot classes. introduction a february 2007 editorial in the washington post stated that judges had cited wikipedia four times as often as the encyclopedia britannica in their judicial opinions over the previous year. the editorial goes on to praise wikis, youtube, and other “opensource projects” as an “unstoppable movement toward shared production of knowledge” (sunstein). while sites such as wikipedia are valuable for a myriad of reasons, including the community creation of knowledge, librarians, teachers, and other information professionals must wonder at the reasons why judges, extremely learned men and woman, would choose wikipedia over an esteemed source such as the encyclopedia britannica for their opinions and what, if any, evaluation techniques they used when selecting this resource. similar concerns arise regarding the information seeking behavior of students in higher education. college students’ strong preference for quickly and easily accessible web sites is an issue for librarians, college professors, and others in higher education. opting for information quickly available on the internet hinders the development of students’ research skills and provides them with only a small fraction of the information available on any given topic. students relying only on internet resources will not only be deficient in their knowledge of a subject, but also in how to find more information on that subject. information seeking can be defined as “the interactions between people, the various forms of data, information, knowledge, and wisdom that fall under the rubric of information, and the diverse contexts in which they interact” (todd 27). liao, finn, and lu divide information seeking into three broad categories: initiating, searching, and locating (9). others have argued that information seeking should not be seen in such rigid and linear frames. instead, they suggest that the process of finding information should be viewed as subjective and influenced by previous experiences, knowledge, and opinions (weiler 51). however one approaches the concept of information seeking, it is clear that this is an important skill for students to possess. those individuals who are deficient in information seeking skills have difficulty in knowing when information is needed, the value of libraries in finding information, and how to evaluate the sources they do find (gross 155). without these skills, students will perform poorly in the classroom, making the professor’s job more difficult and ultimately reflecting poorly upon the university. the problems, however, extend beyond the classroom. these same skills are needed when graduates seek home or small business loans, research options for their retirement plan, or seek to make informed decisions in local or national elections. research and evaluation skills learned in the classroom are needed throughout life. the information seeking behavior of “nextgen” or “millennial” students is a matter of great concern for those in higher education. the difference in credibility between a web and a print source document is negligible to these college students (abram and luther 34). indeed, long and shrikhande report, “students often simply type terms in google and scan the results until information on their topic is found. no assessment of quality, reliability, or accuracy evidence based library and information practice 2008, 3:4 6 generally occurs” (358). while some have argued that the growth of the internet should be seen simply as the development of a new research methodology, rather than as a decline of research skills, it appears that more and more students are forsaking the library altogether (o’brien and symons 411). several studies lend legitimacy to these assertions. a study conducted in the united kingdom in 2005, found that 45% of the students in that study began their academic research with google, while another 23% used a different commercial search engine such as yahoo!, lycos, or altavista. over two-thirds of the students in this study began their research on the internet rather than in the academic library (griffiths and brophy 545). one reason for this may be that students simply find the internet easier to use than the library. the study further found that students had difficulty using library resources and were willing to sacrifice quality for ease of use (griffiths and brophy 548). today’s college students are definitely at ease with the internet. a report from the pew internet and american life project reported that 86% of college students have gone online, and that 20% of today’s students began using a computer between the ages of 5 and 8 (jones 2). the study also found that college students are positive about the internet, using it for both academic and social/recreational needs. the study found that 78% of the students used the internet for fun, and 73% of them admitted using the internet more than they used the library (jones 2-3). in fact, 80% of the students stated they used the library less than three hours a week. many remarked that finding information on the internet was easier than using the library (jones 12-3). ease of use is an important component in the information seeking behavior of millennial generation college students. academia is filled with jargon that only the most experienced understand. added to this difficulty is the archaic and technical language used in library catalogs and database subject headings (bodi 111). these barriers make it difficult for time-pressed students to find what they need for their classes. while faculty are often ecstatic at not finding much or any information on their research topics, students become frustrated and opt for the internet because it gives them the quantity they crave (bodi 111). the preference for the internet over the library is not limited to inexperienced researchers. one study found no real difference in library usage among freshman, sophomore, junior, and senior college students (van scoyoc and cason 51). although an oclc study did find that 7 out of 10 students use the library’s site for at least some of their research, 43% of those students who do not use the library’s site for research do so because they think they can find better information elsewhere (oclc 6). millennial college students make heavy use of the web in their class projects and research. a study that examined the bibliographies of student papers found that the number of citations for web sites rose from an average of 11.3 per bibliography (or 9% of the total number of references per bibliography) in 1996 to 14.4 per bibliography (or 13% of the total number of references per bibliography) in 2001 (davis 46). web citations in student bibliographies peaked in 2000, with an average of 22% per bibliography. the decline in percentage is directly attributable to new restrictions placed by professors regarding the type and frequency of web citations students were permitted to use (davis 47). davis found that faculty were not opposed to students using web sites in their research, but that they now routinely apply restrictions on what and how many web sources students may use in their papers (45). most faculty agree that the internet is an excellent source of information, but they are concerned that evidence based library and information practice 2008, 3:4 7 students are not able to properly evaluate the sources they have found (herring 255). these concerns over web sites and resource evaluation appear well founded. an oclc study found that while two-thirds of students polled felt they could determine what sites were best to use, 58% of students believe that sites with advertisements are just as reliable as sites without advertisements (oclc 4). a number of authors have attempted to determine the effectiveness of library instruction. in an oft-cited study, lois pausch and mary popp found that few critical assessments of library instruction exist in the literature. most of what has been published are informal surveys of students that measure the students’ satisfaction with a particular class (pausch and popp). brettle reviewed the research on information skills training in the health sector during the time period 1995-2002 and found that many of the studies were poorly designed, executed, and reported (6). further, brettle found that many of the studies reviewed relied on subjective measures to test the efficacy of instruction rather than on validated instruments using objective measures. in 2006 koufogiannakis and wiebe undertook a review of the literature on teaching information literacy skills. their findings report a lack of overall quality in the studies reviewed. many of the published studies suffered from faulty reporting and failed to use a validated instrument. twenty percent of the studies performed no statistical analysis (koufogiannakis and wiebe 19). a 2004 study conducted by beile and boote attempted to critically assess the effectiveness of library instruction using preand post-tests. while they found a statistically significant difference in test scores, the population was small (49 students) and consisted solely of graduate education majors (6). most other studies, however, show that library instruction has a minimal impact on students’ information seeking behavior. in her systematic review of the literature, brettle wrote, “the results revealed very limited evidence to show that training does improve skills” (7). after reviewing the literature and performing a meta-analysis, koufogiannakis and wiebe reported only that library instruction was better than no instruction (19). andrew robinson and karen schlegl undertook a bibliometric analysis of student research papers. their study found that library instruction had little impact on the types of sources cited. the students’ choice of resources was most influenced by the instructors’ directions to the students; when the instructors enforced penalties related to student grades, the students cited more scholarly sources (robinson and schlegl 280). in 2002 emmons and martin studied the effects of library instruction on an english writing class; they found a statistically significant increase in scholarly journal citations following library instruction (554). their overall findings indicated that library instruction “made a small difference in the types of materials students chose and how they found them” (557-8). aim the aim of this study was to examine the information seeking skills of undergraduate education majors at the university of central florida (ucf). specifically, this study attempted to discern the types of sources (academic vs. non-academic) undergraduate education majors used to find information for their research. the study also sought to determine whether an association existed between library instruction sessions and the types of sources used. the research was funded with a $1,000 grant sponsored by the ucf quality enhancement plan (<http://www.if.ucf.edu/>). evidence based library and information practice 2008, 3:4 8 sample the university of central florida enrolled almost 49,000 students at the start of the fall 2007 semester. of those students, 3,605 were undergraduate education majors. the study used a convenience sample of 200 currently enrolled undergraduate education majors. participants volunteered after seeing advertisements for the survey or after a curriculum materials center (cmc) employee asked if they would like to take a short, online survey. an incentive of $5 was offered to all participating students. those who agreed to participate were shown how to access the survey in the cmc. once the survey was confirmed as complete by the principal investigator, participants each received $5. instrumentation the survey consisted of 14 questions (appendix a). the survey was administered online using survey monkey questionnaire software (<http://www.surveymonkey.com/>). the survey asked questions about four areas of information seeking behavior: • the research habits of students (questions 1,10, and 11); • the ease of using the library’s resources, and how important convenience is to the student in selecting resources. (questions 2, 3, and 9); • where students find most of their research information (questions 4,5, and 8); • evaluating sources (questions 6 and 7). table 1 student information seeking behavior if you were researching a topic for a class project like a paper or presentation, where would you find most of your information? internet library resources ask friends ask experts other 72%, n=144 27.5%, n=55 .5%, n=1 0 0 if you were researching a topic for a personal reason, where would you find most of your information? internet library resources ask friends ask experts other 88%, n=176 2.5%, n=5 6%, n=12 3.5%, n=7 0 table 2 research resources which of the following sources do you use most in your research? internet book academic journal newspaper or popular magazine all sources equally 65%, n=130 8%, n=16 16%, n=32 0 11%, n=22 which of the following do you think is the most credible source? internet book academic journal newspaper or popular magazine all sources are equally credible 2%, n=4 20%, n=40 59%, n=118 1%, n=2 18%, n=36 evidence based library and information practice 2008, 3:4 9 additional demographic questions asked participants about their class standing, the number of hours per day spent on the internet, and the number of library instruction sessions attended. results all 200 surveys were deemed usable, and no one from the original sample opted out of the research. table 1 shows that the internet was the predominant choice of almost threefourths of the respondents for class-related research. nearly 9 out of 10 used the internet for personal research. even though these students realized that library resources were more credible than internet sources, they still chose to use internet sources instead of academic library sources for both personal and class work. the question remains as to why these students would make that choice. ease of access may be an answer, as may the students’ high comfort level with the internet. table 3 addresses these ideas. while almost 90% of respondents felt that the library’s resources were not hard to use, 78% were still more comfortable using the freely available internet instead of the library’s resources. dishearteningly, 52% of the respondents based their decisions more on convenient access than on the authority of the resource. effects of library instruction another important question is about the effect of library instruction on the students’ choice of resources. a chi-square test of association was conducted to determine whether the proportion of undergraduate education majors who used academic sources in comparison to those who used non-academic sources varied depending on whether the students had taken at least one library instruction session. the null hypothesis (h0: x2 = 0) states that the proportions are equal, while the alternative hypothesis (h1: x2 ≠ ) is that the proportions are not equal. table 3 resource selection – library or internet i think the library’s resources are hard to use. very much disagree somewhat disagree somewhat agree very much agree 43.5%, n=87 45%, n=90 11%, n=22 .5%, n=1 i am more comfortable using the internet than the library’s resources. very much disagree somewhat disagree somewhat agree very much agree 2.5%, n=5 20%, n=40 54%, n=107 24%, n=48 i would use a source because it is convenient to use even though it is not the best source on my topic. very much disagree somewhat disagree somewhat agree very much agree 17%, n=34 31%, n=62 44%, n=88 8%, n=16 evidence based library and information practice 2008, 3:4 10 the independent variable, library instruction, was assessed with question 14: “not counting cmc tours, how many library instruction sessions have you attended?” choices ranged from zero sessions to five or more. all the responses of zero (n=61) were grouped into the category “no library instruction.” all the responses from one session to five or more (n=139) were grouped into the category “library instruction.” the dependent variable, “types of sources used,” was assessed with the question “which of the following sources do you use most in your research?” all the responses for “internet sites” were grouped into the category “internet.” all the responses for “book and academic journal” were used for the category “academic sources.” the responses for “i use all these sources equally” were grouped together in the category “all equally.” no one selected the response “newspapers or popular magazines.” table 4 illustrates the chisquared test of association table. all variables were independent of each other, and all cells had at least five expected frequencies, so all assumptions for the chisquare test of association were met. at an alpha level of .05, the type of information resource used for research was not statistically significantly related to whether the student attended library instruction sessions (pearson χ2 (1, n = 200) = 1.612, p = .447, cramer’s v = .090). students who had attended a library instruction session were proportionally just as likely to use academic and non-academic sources as those students who had not attended a library instruction session. the measure known as ‘effect size’ evaluates the strength of the association being tested (morgan, reichert, and harrison 15). it may be seen as the practical significance of a test result. in this study the cramer’s v value of .090 indicates a small effect size. tables 5 and 6 illustrate the findings. since the effect size is small, it can be thought of as having less practical significance to the field of table 4 chi-square test of association amount of library instruction types of resources used academic all equally internet total library instruction count 33 13 94 140 expected count 33.6 15.4 91 140 no library instruction count 15 9 36 60 expected count 14.4 6.6 39 60 total count 48 22 130 200 expected count 48 22 130 200 evidence based library and information practice 2008, 3:4 11 information literacy and library instruction. effect size considered with sample size determines power, which is the probability that a test will reject a false null hypothesis. a test with a small effect size might not generate enough power to detect statistical significance (morgan, reichert, and harrison 16). discussion although this study is limited in that without a true random sample and larger sample size the results cannot be generalized to the entire population, the results are nonetheless disappointing, if not surprising, for those interested in library instruction. the fact that students surveyed here performed most of their research, whether for class or personal reasons, on the freely available web is supported by the findings of griffiths and brophy, davis, and the pew internet report on the use of the library and the internet by college students (jones). moreover, 79% of students surveyed stated that academic sources (e.g., books and journals) are more credible than the internet, yet they still rely heavily on internet sources for their research. griffiths and brophy concluded that students have difficulty using library resources, so they turn to the internet with which they are much more comfortable. this study found that while students did not find the library difficult to use, they were more comfortable using the freely available internet. some might argue that with so many library resources being online, the distinction between the internet and library resources is blurred, and that students may have difficulty differentiating between the two. this may have been the case in this study, since definitions of “internet resources” and “library resources” were not provided in the survey. however, personal experience at the library’s reference desk and in library table 5 chi-square test results value df asymp. sig. (2-sided) pearson chi-square 1.612a 2 .447 likelihood ratio 1.548 2 .461 n of valid cases 200 a. 0 cells (.0%) have expected count less than 5. the minimum expected count is 6.60. table 6 symmetric measures value approx. sig. nominal by nominal phi .090 .447 cramer's v .090 .447 n of valid cases 200 evidence based library and information practice 2008, 3:4 12 instruction sessions suggests that students do make the distinction between resources on the library’s site and those available freely through a search engine such as google. additionally, while google scholar further erodes the separation of academic sources and the freely available web, personal experience again suggests that undergraduate students are not using google scholar. again, this may be due to students being unaware not only of the differences between academic and nonacademic sources, but also the appropriateness of using those sources. this study found no association between library instruction and the types of sources used by students. this is supported by the findings of emmons and martin and robinson and schlegl. furthermore, the studies by davis and robinson and schlegl found that instructor guidelines played a more significant role in student citations than did library instruction. this raises a crucial question as to how much students are learning about research from simply following the rules written in their class syllabi. if students are not citing internet sources simply because they are told to use more academic sources, it is possible that they will revert to using the internet when they are not specifically instructed to do so, and they would not have gained a deeper understanding of the critical importance of using academic sources. this is important, since almost 90% of the students in this study said they use the internet as a primary tool for personal research. however, beile and boote found that the greatest increases in post-test scores occurred among students who had previous library instruction (6). a 2006 bibliometric study conducted by wang found a statistically significant difference in the citations of students who had taken a forcredit library course as compared to citations listed by those students who had not taken the course. those who had taken the course cited more scholarly sources (wang 85). further, wang reviewed the guidelines set forth by the professors and found that none of them specified an academic penalty for having too many nonacademic citations (wang 87). this suggests that for-credit library classes or multiple library instruction sessions may prove more effective in changing students’ information seeking behavior than the traditional “oneshot” library instruction class. these studies could have an important impact on how academic libraries approach library instruction. libraries have long used the “one-shot” library instruction session where a professor brings his/her class to the library for a session on how to use the library. while this approach does have some value as an introduction to the library for new students, perhaps it is time for libraries to seriously consider alternative practices. academic libraries might be better served to invest their limited resources in for-credit library classes, mandatory multiple library instruction sessions, or in integrating librarians into the class curriculum. these changes in practice will not be easy. not only would these approaches require more time and effort, but the devaluing or possible eradication of one-shot library instruction classes strikes at a core belief of academic librarians. while the vast majority of library instruction at the university of central florida libraries consists of one-shot classes aimed mainly at freshman composition students, the library has made efforts to enhance the instruction program. in conjunction with course development and web services (cdws), the library has created online information literacy modules that can be used by the teaching faculty in their online or face-toface classes (<http://infolit.ucf.edu/faculty/>). these modules focus on different areas of research, evidence based library and information practice 2008, 3:4 13 and more are forthcoming. they include content, practice, and assessment, so an instructor can see how well students understand the information. the ucf libraries also offer “embedded librarians” as an integral part of online classes. they answer questions, create tutorials, and work with the instructors on creating proper research assignments. during the seven academic years in which this service has been offered, ucf librarians have been embedded in 187 classes reaching almost 5,600 students. although no formal assessment of library skills has been made of students in classes with embedded librarians, further investigation is planned. conclusion this study found no association between library instruction and the use of traditional academic library resources in student research. academic libraries are currently investing staff and time to order to teach information literacy, and yet the truly important question of how to effectively change students’ perception of research methodology remains unanswered. is information literacy and its generic offshoot library instruction truly effective? perhaps the solution lies outside the library in the types of assignments students are given and how they are graded. do libraries need to rethink and redesign how they organize and allow access to information? or have we truly entered a new age of research where quantity, easy access, and keyword searching are more important than controlled vocabulary and peer-review? this study makes no claims to answering these questions, no one study alone can, but it is important that as the library profession moves forward it develop a research agenda and theoretical foundation which will eventually answer these questions. works cited abram, stephen, and judy luther. "born with the chip." library journal 129.8 (1 may 2004): 34-7. beile, penny m., and david n. boote. "does the medium matter?: a comparison of a web-based tutorial with face-toface library instruction on education students' self-efficacy levels and learning outcomes." research strategies 20.1/2 (mar. 2004): 57-68. bodi, sonia. "how do we bridge the gap between what we teach and what they do? some thoughts on the place of questions in the process of research." journal of academic librarianship 28.3 (may 2002): 109-14. brettle. alison. “information skills training: a systematic review of the literature.” health information and libraries journal 20.supp.1 (jun. 2003): 3-9. davis, philip m. "effect of the web on undergraduate citation behavior: guiding student scholarship in a networked age." portal: libraries & the academy 3.1 (jan. 2003): 41-51. emmons, mark, and wanda martin. "engaging conversation: evaluating the contribution of library instruction to the quality of student research." college & research libraries 63.6 (nov. 2002): 545-60. <http://www.ala.org/ala/mgrps/divs/ac rl/publications/crljournal/2002/novemb er02/emmons.pdf> 1 nov. 2008. griffiths, jillian r., and peter brophy. "student searching behavior and the web: use of academic resources and google." library trends 53.4 (spring 2005): 539-54. evidence based library and information practice 2008, 3:4 14 gross, melissa. "the impact of low-level skills on information-seeking behavior." reference & user services quarterly 45.2 (winter 2005): 155-62. herring, susan davis. "faculty acceptance of the world wide web for student research." college & research libraries 62.3 (2001): 251-8. 2 nov. 2008 <http://www.ala.org/ala/mgrps/divs/ac rl/publications/crljournal/2001/may01/h erring.pdf> jones, steve. the internet goes to college: how students are living in the future with today's technology. pew internet and american life project. 15 sept. 2002. 2 nov. 2008 <www.pewinternet.org/pdfs/pip_colle ge_report.pdf>. koufogiannakis, denise and natasha wiebe. “effective methods for teaching information literacy skills to undergraduate students: a systematic review and meta-analysis.” evidence based library and information practice 1.3 (2006): 3-43. 2 nov. 2008 <http://ejournals.library.ualberta.ca/ind ex.php/eblip/article/view/76/153>. liao, yan, mary finn, and jun lu. "information-seeking behavior of international graduate students vs. american graduate students: a user study at virginia tech 2005." college & research libraries 68.1 (jan. 2007): 525. 2 nov. 2008 <http://www.ala.org/ala/mgrps/divs/ac rl/publications/crljournal/2007/crljanua ry07/liao07.pdf>. long, casey m., and milind m. shrikhande. "improving information-seeking behavior among business majors." research strategies 20.4 (2005): 357-69. morgan, susan e., thomas reichert, and tyler r. harrison. from numbers to words: reporting statistical results for the social sciences. boston, ma: allyn & bacon, 2002. o'brien, heather l., and sonya symons. "the information behaviors and preferences of undergraduate students." research strategies 20.4 (2005): 409-23. oclc. “how academic librarians can influence students' web-based information choices.” oclc white paper on the information habits of college students. june 2002. 2 nov. 2008 <http://www5.oclc.org/downloads/com munity/informationhabits.pdf>. pausch, lois m. and mary pagliero popp. “assessment of information literacy: lessons from the higher education assessment movement.” 1997. 23 june 2008. in: acrl 1997 national conference papers. chicago: association of college and research libraries. 2 nov. 2008 <http://www.ala.org/ala/mgrps/divs/a crl/publications/whitepapers/nashville /pauschpopp.cfm>. robinson, andrew m. and karen schlegl. "student bibliographies improve when professors provide enforceable guidelines for citations." portal: libraries & the academy 4.2 (april 2004): 275-90. sunstein, cass r. "a brave new wikiworld." the washington post [washington, dc] 24 feb. 2007: a,19. todd, ross j. "adolescents of the information age: patterns of information seeking and use, and evidence based library and information practice 2008, 3:4 15 implications for information professionals." school libraries worldwide 9.2 (july 2003): 27-46. van scoyoc, anna m., and caroline cason. "the electronic academic library: undergraduate research behavior in a library without books." portal: libraries & the academy 6.1 (jan. 2006): 47-58. wang, rui. 2006. “the lasting impact of a library credit course.” portal: libraries & the academy 6.1 (jan. 2006): 79-92. weiler, angela. "information-seeking behavior in generation y students: motivation, critical thinking, and learning theory." journal of academic librarianship 31.1 (jan. 2005): 46-53. evidence based library and information practice 2008, 3:4 16 appendix a 1.) i enjoy researching. 1 very much disagree 2 somewhat disagree 3 neither agree nor disagree 4 somewhat agree 5 very much agree 2.) i think the library’s resources are hard to use. 1 very much disagree 2 somewhat disagree 3 neither agree nor disagree 4 somewhat agree 5 very much agree 3.) i am more comfortable using the internet than the library’s resources. 1 very much disagree 2 somewhat disagree 3 neither agree nor disagree 4 somewhat agree 5 very much agree 4.) if you were researching a topic for a class project such as a paper or presentation, where would you find most of your information? (check only one.) on the internet using library resources asking friends asking experts other 5.) if you were researching a topic for a personal reason, where would you find most of your information? (check only one.) on the internet using library resources asking friends asking experts other 6.) which criteria do you use to evaluate web sites? (check all that apply.) accuracy authority objectivity currency i do not evaluate web sites 7.) which of the following do you think is the most credible source? internet site book academic journal newspaper or popular magazine all are equally credible 8.) which of the following sources do you use most in your research? (check only one.) internet sites books academic journals newspapers or popular magazines i use all these sources equally evidence based library and information practice 2008, 3:4 17 9.) i would use a source because it is convenient to use even though it is not the best source on my topic. 1 very much disagree 2 somewhat disagree 3 neither agree nor disagree 4 somewhat agree 5 very much agree 10.) i perform a good deal of research for my classes. 1 very much disagree 2 somewhat disagree 3 neither agree nor disagree 4 somewhat agree 5 very much agree 11.) i am required to write research papers for my classes. 1 very much disagree 2 somewhat disagree 3 neither agree nor disagree 4 somewhat agree 5 very much agree 12.) what is your ucf student status? freshman sophomore junior senior 13.) how many hours a day do you spend on the internet on average? 0 1-3 4-6 7 or more 14.) not counting cmc tours, how many library instruction sessions have you attended while a student at ucf? 0 1 2 3 4 5 or more evidence based library and information practice editorial responsibilities      2016. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.       editor-in-chief: lorie kloda   associate editor (evidence summaries): heather pretty   associate editor (articles): rebekah (becky) willson, lisl zach   associate editor (classics): jonathan eldredge   associate editor (reviews): lorie kloda   associate editor (commentaries, using evidence in practice, research in practice, news): lorie kloda   production editor: michelle dunaway   editorial intern: melissa griffiths   editorial advisors: lindsay alcock, alison brettle, denise koufogiannakis, katrine mallan, pam ryan   copyeditors: richard hayman (lead copyeditor), georgianne bordner, bryan chan, julie evener, heather healy, julie james, stacey penney, maria tan, nikki tummon, alison yeoman   indexing support: meral alakus   microsoft word art_fisher.doc evidence based library and information practice 2007, 2:4  36 evidence based library and information practice     article    evidence based management as a tool for special libraries      bill fisher  school of library & information science  san jose state university  san jose, california, usa  e‐mail: bill.fisher@sjsu.edu    dav robertson  national institute of environmental health sciences  p.o. box 12233  research triangle park, north carolina, usa  e‐mail: robert11@niehs.nih.gov      received: 29 july 2007    accepted: 12 november 2007      © 2007 fisher and robertson. this is an open access article distributed under the terms of the creative  commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    abstract    objective ‐ to examine the evidence based management literature, as an example of  evidence based practice, and determine how applicable evidence based management might  be in the special library environment.    methods ‐ recent general management literature and the subject‐focused literature of  evidence based management were reviewed; likewise recent library/information science  management literature and the subject‐focused literature of evidence based librarianship  were reviewed to identify relevant examples of the introduction and use of evidence based  practice in organizations.  searches were conducted in major business/management  databases, major library/information science databases, and relevant web sites, blogs and  wikis.  citation searches on key articles and follow‐up searches on cited references were  also conducted.  analysis of the retrieved literature was conducted to find similarities  and/or differences between the management literature and the library/information science  literature, especially as it related to special libraries.    mailto:fisher@sjsu.edu mailto:robert11@niehs.nih.gov http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:4  37 results ‐ the barriers to introducing evidence based management into most organizations  were found to apply to many special libraries and are similar to issues involved with  evidence based practice in librarianship in general.  despite these barriers, a set of resources  to assist special librarians in accessing research‐based information to help them use  principles of evidence based management is identified.    conclusion ‐ while most special librarians are faced with a number of barriers to using  evidence based management, resources do exist to help overcome these obstacles.       introduction  using evidence based management in a  special library environment can seem like a  daunting task; however, this does not need  to be the case.  the following situation  provides one scenario to support the use of  evidence in library decision‐making.      the staff of a special library supporting an  environmental sciences organization knew  they needed to make some changes to better  serve their clientele.  like many libraries,  they were trying to find the appropriate  balance between traditional and digital  services, programs and collections.  while  the library’s staff felt they had a good idea  about what to keep, what to eliminate and  what to alter, they also realized that  bringing in a consultant to independently  determine what should be done would go a  long way toward having those decisions  accepted by the organization’s management.   the consultant was brought in and found  that the library was already gathering some  of the data that would be used to support  some of the decisions that would be made.   this not only reduced the time needed by  the consultant, but also saved the library  money.  while some of the staff’s earlier  assumptions were confirmed by the  consultant’s report, other assumptions staff  made were not supported by the evidence.   based on the evidence, staff were persuaded  to reconfigure their service model,  introducing new programs and services and  dropping those that the evidence  demonstrated were not as important to the  library’s customers as staff had believed.  the evidence further led to a redesign of the  library’s web site and even a  reconfiguration of the library’s physical  space.  this library can now demonstrate  greater use of both physical and digital  resources, and this may not have been the  case if the staff worked solely from their  original assumptions.  this example  demonstrates that gathering evidence to be  used in managerial decision‐making is not  as difficult as it may seem.      scope and methodology  although evidence based practice is still a  relatively recent concept, a number of fields,  most notably medicine and the health  sciences, have begun to utilize evidence to  inform many of their activities.  libraries,  led by those affiliated with medical and  health science organizations, have been  active in this arena.  the general  management of an organization (regardless  of the type of organization) has also been  identified as an arena where evidence based  practice could be useful, and this has led to  the development of the sub‐field of evidence  based management.  this paper will review  the literature defining evidence based  management and discuss how it can be  introduced and applied in an organizational  setting.  this information will then be  evaluated for its applicability to special  libraries, especially those that exist in the  for‐profit and government/non‐profit  environments.  while most medical/health  sciences libraries would be considered  evidence based library and information practice 2007, 2:4  38 special libraries, for the purpose of this  paper, they will not be included to any  extent.    literature searches were conducted in both  the management and library literature.  the  library literature search included the  following databases: library literature &  information science full text; library,  information science & technology abstracts;  and eric.   the management literature  search was conducted in the following  databases: abi/inform; business full text;  and business source premier.  both  literature searches were undertaken for the  years 1990 to 2007. the web of science,  including both social science citation index  and science citation index, was also  searched and then the cited reference  search and find related records functions  were used selectively.  various  permutations of the terms evidence based  management, special libraries, and evidence  based practice were employed for the initial  screening. the aim was initially to  emphasize recall, find as many items as  possible and then apply further search  criteria to enhance precision.  titles and  descriptors/keywords of the items retrieved  were scanned for anything of interest,  (primarily terms which could be used as  synonyms for the primary search terms).    abstracts were then reviewed for the more  promising items; and finally the full text of  the most relevant articles was reviewed.   from this latter group of material, the  references cited by those authors were  checked for additional leads to useful  literature.  occasionally, this would lead to  the review of a journal’s table of contents for  specific theme‐based issues.  all of the  issues of evidence based library and  information practice were also examined.      beyond the journal literature, a number of  other sources were reviewed.  the end‐of‐ chapter references from the evidence based  practice handbook (booth and brice) were  examined; as were the papers delivered at  the first three evidence based library &  information practice conferences  (<http://www.eblip.net> for access to the  first and second conferences and  <http://conferences.alia.org.au/ebl2005> for  access to the third conference).  relevant  web sites and blogs were also evaluated,  and the most useful are presented later in  this paper.    change in attitude  before examining the case of special libraries,  it is necessary to understand how evidence  based management is described in the  management field itself.  as the opening  example demonstrates, just gathering the  data/evidence is not enough.  many  organizations have the information they  need to be more effective; the question  seems to be what these organizations do  with this information once they realize they  have it or make an effort to gather it.  in fact  the evidence itself may not be a problem at  all.  as stanford professors jeffrey pfeffer  and robert sutton explain, evidence based  management requires a change in attitude, a  change in how a manager thinks about  decision‐making, so for many organizations  creating acceptance for a culture for  evidence based management is the major  challenge (“evidence‐based” 64).  evidence  based management is a process, and with  any process the best way to begin is with  small, incremental steps that will eventually  culminate in change on an organization‐ wide basis.      pfeffer and sutton, writing in the harvard  business review, gave managers a few steps  to follow in bringing about this change in  attitude.  first, demand evidence – the best  way to get an organization to become  evidence based is for the organization’s  leaders to ask for the evidence that supports  decisions, recommendations and the like.   the danger in this, of course, is the manager  needs to address the evidence that is  http://www.eblip.net http://conferences.alia.org.au/ebl2005 evidence based library and information practice 2007, 2:4  39 presented, which may well conflict with the  manager’s preconceived ideas of how things  should be.  this leads to the second step of  examining the logic and critically evaluating  any evidence that is presented.  pfeffer and  sutton cite a medical author who relates a  joke among physicians: “if you want to have  an operation, ask a surgeon if you need one”  (“evidence‐based” 65).  as mentioned in the  opening example, the special library’s staff  believed having recommendations come  from an outside consultant would give them  more credibility with the organization’s  management.   pfeffer and sutton’s third  point is to treat the organization as an  unfinished prototype.  any organization  that feels it has nothing to learn first has  everything to learn, and second won’t  commit any resources to trying to improve.   decision‐making in an environment like this  is all about maintaining the status quo and  justifying why change should not happen.   try something new in a limited way, gather  the evidence on how things go, and than  adapt, revise and retry as needed based on  the evidence.  finally, an attitude of wisdom  needs to spread throughout the organization  (“evidence‐based” 70‐73).  most people  rarely feel they have all the information they  need to make a truly knowledgeable  decision, so while they need to act based on  the most informed knowledge we have at  the time, they also need to keep questioning  what is known and the basis for a decision  to see if new evidence comes to light.   additionally, for this to be effective, people  need to be in a frame of mind that is  accepting of any new evidence.  did  something work because it was well  planned and executed, or did it work  because of sheer luck?  people need to  question their successes as much as their  failures to determine why things went well,  so they can build upon those successes.    pfeffer and sutton further identify three  questionable management practices that can  be remedied by using evidence based  management: casual benchmarking, doing  what (seems to have) worked in the past,  and following deeply held yet unexamined  ideologies.  they propose six standards for  managers to use under evidence based  management for “generating, evaluating,  selling, and applying business knowledge”:    • treat old ideas like old ideas.  • be suspicious of breakthrough ideas and  studies—they almost never happen.  • celebrate communities of smart people  and collective brilliance, not lone  geniuses or gurus.  • emphasize the virtues and drawbacks  (and uncertainties) of your research and  proposed practices.  • use success and failure stories to  illustrate practices supported by other  evidence, not necessarily as valid  evidence.  • take a neutral approach to ideologies  and theories.  base management  practices on the best evidence, not what  is in vogue.  (hard facts 41)    be that as it may, evidence based  management does not appear to have  proliferated through the business world or  the business literature in the same way that  total quality management, reengineering,  the balanced scorecard or other  management ideas of the past 20‐25 years.   while evidence based management may  have a solid toehold on the rock‐face of  credibility, it is still just a toehold.  just as  evidence based practice in librarianship has  its advocates, pfeffer and sutton are clearly  leading the charge for evidence based  management.  pfeffer and sutton are the  driving forces behind the evidence based  management web site and blog  (<http://www.evidence  basedmanagement.com> and   <http://www.evidence based  management.com/blog/index.php>).  links  off the web site lead to over a dozen  publications by pfeffer and/or sutton on  http://www.evidencebasedmanagement.com http://www.evidencebasedmanagement.com http://www.evidencebasedmanagement.com/blog/index.php http://www.evidencebasedmanagement.com/blog/index.php evidence based library and information practice 2007, 2:4  40 evidence based management, as well as  syllabi for business school classes that take  an evidence based approach.  denise  rousseau, in her capacity as president of the  academy of management, has also  championed the benefits of evidence based  management.  “the promises of evidence‐ based management are manifold.  it affords  higher‐quality managerial decisions that are  better implemented, and it yields outcomes  more in line with organizational goals.  those who use evidence (e and e) and learn  to use it well have comparative advantage  over their less competent counterparts. . . . a  focus on evidence use may also ultimately  help to blur the boundaries between  researchers, educators, and managers,  creating a lively community with many  feedback loops where information is  systematically gathered, evaluated,  disseminated, implemented, re‐evaluated,  and shared” (267‐68).    the gap between the results of many  research studies and the information which  practitioners say will be useful to them has  been noted in many fields, including  management and librarianship.  rousseau  notes that for evidence based management  to truly be worthwhile, that gap needs to  shrink.  she posits that it will take a  concerted effort to make this happen.   faculty need to incorporate more evidence  into their instruction, students need to use  evidence in their assignments, researchers  need to find/create evidence that will be  useful in the field, and practitioners need to  identify and utilize appropriate evidence in  their decision‐making and daily practice  (266‐267).  however, there is no quick fix.   any organization that seeks to incorporate  evidence based management will find it a  time‐consuming and frustrating process.   additionally, what works for one  organization may not work for another even  in the same industry.  the best, most  rigorously obtained evidence is worthless if  applied to an organization without the  informed judgment of those in the  organization itself.    special libraries and evidence based  management  from an organizational perspective, it  would be beneficial for special librarians to  apply the principles of evidence based  management to their work environments.   first, since most special libraries are  particularly vulnerable to reductions,  downsizing, or even closure, demonstrating  that decision‐making is supported by  evidence could help to substantiate the  businesslike nature of operations.   most of  the environments where special libraries are  found don’t have to support those libraries,  so presenting an evidence based focus could  help minimize questions of the library’s  value to the organization.  second, adopting  an evidence based approach may not be an  option if the parent organization utilizes  evidence based management or even if the  librarian’s manager is a proponent of  evidence based management.  it is not  surprising that medical and health‐related  libraries have taken the lead with evidence  based applications, since the medical/health  care field is far ahead of other disciplines in  applying evidence based practices.  third,  the only thing most special librarians have  in excess is work to do, so being able to  identify and eliminate functions that don’t  really help one accomplish anything is  extremely useful.  as a profession there are  numerous things any library does because it  is a library, but do all those really need to be  done in every library in every organization?    there are very few examples of evidence  based management in special libraries  outside medical libraries.  this supports the  viewpoint that one of the reasons why the  profession doesn’t embrace evidence based  practice more fully is the lack of an evidence  base (brice, booth, and benson 16).  two  papers about using evidence based  management in special libraries were  evidence based library and information practice 2007, 2:4  41 discovered: one dealt with the use of  information in decision‐making in the  banking industry in the uk (reid, thomson,  and wallace‐smith 86‐109), and the second  was a meta‐analysis of studies on the use of  information in decision‐making in six  different environments in the uk, canada  and the united states (grieves 78‐85).  as  brice, booth and benson indicate, this is  partly a result of our over‐reliance on  anecdotal evidence and on our own  experience (17), which leads to the  numerous how‐i‐done‐it‐good type articles  that fill the pages of our professional  journals.  the lack of an evidence base is  also noted by lerdal as she assessed the  value of evidence based librarianship for  law librarians (33‐60).     obstacles for special libraries  for special librarians, the other factors  identified by brice, booth, and benson also  ring true.  one of these is the lack of time  many special librarians face due to staff size;  there are too many things that need to be  done for the staff available to do them.  the  irony here is the special librarian may not  have the time to find evidence to help  manage the library because he/she is busy  finding evidence for others in the  organization to assist them in managing  their operation.  another factor, prevalent  among special librarians, is an emphasis on  practical rather than theoretical knowledge.   this may be a result of staff size and  workload, leading special librarians to look  for ideas they can implement quickly and  easily, often without taking the time to  evaluate just how effective the new idea  might be.  however the special libraries  association (sla) provides a number of  opportunities for disseminating research  project results to its members, including its  annual conference.  finally, there is the idea  that librarians don’t have the skills to  critically appraise the evidence they do find.   again, even those special librarians with the  ability to analyze and evaluate the research  they can access probably don’t have the time  or get any workplace reinforcement to do so.   and for those who may not have these skills  (or have not used them recently), they can  be learned/relearned relatively easily.  these  obstacles are not just applicable to special  librarians, koufogiannakis and crumley, in  looking at the status of research in the field  overall, identified access to the research  literature, time and support from the  workplace, experience with research, and  funding as obstacles to be overcome to make  evidence based librarianship more viable in  any library environment (333‐335).    it can be seen from the literature that there  are identifiable reasons why information  professionals and managers in general do  not use or engage in evidence based  research.  these include the inability to  generalize from specific case‐studies or to  apply general studies to local settings, and  an emphasis on practice rather than theory.   a study by kathlyn turner in new zealand  found that special librarians in government  agencies were less likely to use research  than their academic counterparts and that  the smaller the library staff size, the less  likely the librarians were to use research.  turner also found that attending  conferences correlated with increased  research use and that the second most‐cited  reason information professionals consult  research, after personal professional  development, is “to assist with managerial  activities…such as problem‐solving,  decision‐making, planning and/or  evaluation” (4).indicating a potential  opportunity for evidence based  management.    however, information professionals who  work in the corporate or government sector  are susceptible to the same limitations as  other corporate managers.  rousseau in her  address to the academy of management  emphasized the problems resulting from not  using research and from managers not  evidence based library and information practice 2007, 2:4  42 having a systematic understanding of the  principles governing organizations and  human behaviour. she asserted, “the reality  is that managers tend to work in settings  that make valid learning difficult.  this  difficulty is compounded by the widespread  uptake of organizational fads and  fashion….in such settings managers cannot  even learn why their decisions were wrong,  let alone what alternatives would have been  right” (261).  but, rousseau concludes,  “evidence‐based management leads to valid  learning and continuous improvement”  (261).    strategies for implementing evidence  based management  despite these obstacles it is believed that  most special librarians would benefit from  adopting some form of evidence based  management.  clearly demonstrating the  ability to make focused decisions supported  by the best available evidence will assist  special librarians in being recognized as  competent information/knowledge  managers within any organization.   examining strategies drawn from nursing  management might be a potential way  forward. in a paper on evidence based  management for nurse executives, lauren  williams (249) puts forth three groupings of  strategies: strategies to establish an evidence  based management culture (making it clear  that the leaders of an organization support  and actively encourage evidence based  management); to create the capacity to  change to evidence based management (e.g.  management research forums and  leadership development programs to  integrate evidence based management into  practice); and to sustain that change (e.g.  evidence based management in performance  plans, establishing collaborative  partnerships between practitioners and the  academic community and to create an  evidence based management collaborative  database).    these strategies are supported by a  prescription for applying evidence based  practice in special libraries put forward by  the international sla’s research committee  in 2001 (marshall). their recommendations  are similar but more specific than those put  forward by williams,  concentrating on  what individual managers can do, rather  than what management with a capital ‘m’  should do to institute evidence based  management.  beyond a personal  professional commitment to using the best  evidence, they recommend discussions with  colleagues, participating in data collection  and sharing evidence and analysis (44).     resources for special libraries  given the barriers to using evidence based  practice, what resources would enhance the  use of evidence based management in  special libraries?  additionally, what factors  would increase these resources’ acceptance  and use by the special library community?   though not exhaustive, a list of factors  might include the following: a) the source of  evidence needs to be readily accessible,  which in most instances means  electronically available or at the very least a  print‐based serial publication of some kind;  b) both library and non‐library  environments should be included to help  expand horizons and adopt a broader view  of what might be applicable to our home  institutions; c) some type of initial  evaluation of the information needs to have  been made so the special librarian can assess  the resource with confidence. this applies to  articles from peer‐reviewed publications,  web sites or publications sponsored by a  reputable source; and d) a standard format  for reviews of evidence based literature  needs to be used so the special librarian can  quickly determine the relevance and  validity of the review and its potential use.  from the range of material reviewed, some  resources that may assist information  professionals in applying evidence based  evidence based library and information practice 2007, 2:4  43 management are provided in an annotated  list below:    1. pfeffer and sutton’s evidence‐based  management web site  (<http://www.evidence‐ basedmanagement.com/>). includes an  extensive bibliography as well as  frequently updated columns, concrete  examples of evidence based  management in use, and examples of  evidence based management being  taught (primarily in business schools).    2. evidence based library and information  practice e‐journal  (<http://ejournals.library.ualberta.ca/ind ex.php/eblip/index>).  research articles  and analytical summaries of the  evidence‐based literature in  librarianship.  it reflects the general  finding that the majority of evidence  based information in special  librarianship focuses on operational  issues — which can be used by  managers — but not on management  per se.      3. libraries using evidence  (<http://www.newcastle.edu.au/service/l ibrary/gosford/ebl/>), and especially its  eblip toolkit  (<http://www.newcastle.edu.au/service/l ibrary/gosford/ebl/toolkit/>).  this web  site is full of well‐organized information,  but it is the toolkit that is a  straightforward tool for carrying out  and evaluating evidence based research.    4. lindsay glynn’s “a critical appraisal  tool for library and information  research.” in library hi tech.  glynn  reviewed appraisal models in several  fields and devised a tool consisting of a  list of questions to ask about a research  study to judge its validity, applicability  and appropriateness.  this kind of  appraisal tool would work well for  quantitative studies of library functions  but would be difficult to apply to  general management studies that deal  with qualitative and prescriptive topics.    5. graham walton’s “theory, research  and practice in library management:  new column for library management”.   this new column reviews a different  area of library management each time,  beginning with the topic of “flexibility”  (pp. 165‐171).  it provides guidance  within an evidence based framework,  but to date has not  provided practical  analysis relevant to special libraries.    6. outsell, inc.  (<http://www.outsellinc.com/>).  outsell  undertakes market research for the  information industry and provides  consulting services to special libraries  and their organizations.  a large  database of survey results from  corporations and government agencies  has been accumulated which can be  used by subscribers to apply to local  special libraries.  outsell analysts  periodically produce reports  summarizing the data on selected topics  and recommend steps managers can use  to apply that evidence.  they also  specialize in performing needs  assessments and strategic assessments.      7. special libraries association (sla)  (<http://www.sla.org/>).  from the  strategies identified by the sla  research committee for implementing  eblip  (<http://www.sla.org/content/resources/ research/rsrchstatement.cfm>),  to the  sla information portal on lis research  (<http://www.sla.org/content/resources/i nfoportals/research.cfm>), sla provides  a variety of resources to use for  evidence based practice in special  libraries.  other resources are dispersed  among the newsletters, web pages and  http://www.evidence%e2%80%90basedmanagement.com%00%00%00%00 http://www.evidence%e2%80%90basedmanagement.com%00%00%00%00 http://www.evidence%e2%80%90basedmanagement.com%00%00%00%00 http://ejournals.library.ualberta.ca/index.php/eblip/index http://ejournals.library.ualberta.ca/index.php/eblip/index http://www.newcastle.edu.au/service/library/gosford/ebl http://www.newcastle.edu.au/service/library/gosford/ebl http://www.newcastle.edu.au/service/library/gosford/ebl/toolkit http://www.newcastle.edu.au/service/library/gosford/ebl/toolkit http://www.outsellinc.com http://www.sla.org http://www.sla.org/content/resources/research/rsrchstatement.cfm http://www.sla.org/content/resources/research/rsrchstatement.cfm http://www.sla.org/content/resources/infoportals/research.cfm http://www.sla.org/content/resources/infoportals/research.cfm evidence based library and information practice 2007, 2:4  44 blogs of the sla subject‐oriented  divisions as well as papers presented at  the annual conference.    8. various compilations of evidence based  practice in other fields, such as the what  works clearinghouse in education  (<http://www.whatworks.ed.gov/>), the  campbell collaboration in justice and  social sciences  (<http://www.campbellcollaboration.org />), and the cochrane collaboration in  medicine  (<http://www.cochrane.org/index.htm>),  especially for learning how they do  critical appraisals and systematic  reviews.  another useful critical  appraisal tool is the critical appraisal  skills programme  ( <http://www.phru.nhs.uk/pages/phd/ casp.htm>).       conclusion   since this paper began with an example of  evidence based management in a special  library environment, it seems appropriate to  conclude with another such example.   several years ago, when the topic of  metadata was emerging, and google had  not  become widely accepted, a special  librarian saw a great need to improve the  public’s ability to retrieve relevant public  health information from his organization’s  web site.  the common wisdom at that time  was that one should hire indexers —  preferably librarians — to provide metadata  for the web pages.  the special librarian  sought the assistance of a colleague from a  local library school, and they devised a  project to address the question, “can web  page authors themselves create high quality  metadata?”  they set up an experiment and  found that in fact the authors (content  creators) could create useful metadata of  most types, but not all types.  subsequently,  they found that what worked best was for  librarians to review the subject metadata  created by the authors for consistency and  for full subject coverage.  these findings  were then used for management decisions  involving resources.  this example  reinforces rousseau’s comments about  collaboration between practitioners and  academic researchers and shows one way  practitioners and academic researchers can  work together to develop evidence for  evidence based management and challenge  the conventional wisdom.  while there is an  extensive literature on both the pros and  cons of practitioner‐oriented research from  the academic sector, this is not the place to  pursue this further.  suffice it to say for now,  this is a highly relevant topic to the  widespread adoption of evidence based  management and will need further  investigation in the future.    this and the earlier example illustrate the  usefulness of evidence based management  in a special library. anyone who manages in  a library setting can and should make use of  evidence based management, especially the  aspect of it that says to question the  conventional wisdom and to base your  decisions on the best available evidence.   this is especially true for those in the  corporate or government sector where  libraries are being downsized and  information professionals have to develop  new roles for themselves.  it is only by  incorporating the evidence based  management principles into every aspect of  their managerial roles that these librarians  will be able to convince higher management  of their value and continue to contribute to  their respective organizations.  the authors  hope the information above will give  colleagues the impetus to take the initial  steps in creating a workplace environment  supported by evidence based practice.            http://www.whatworks.ed.gov http://www.campbellcollaboration.org http://www.campbellcollaboration.org http://www.cochrane.org/index.htm http://www.phru.nhs.uk/pages/phd/casp.htm http://www.phru.nhs.uk/pages/phd/casp.htm evidence based library and information practice 2007, 2:4  45 works cited    booth, andrew and anne brice, eds.  evidence‐based practice for  information professionals: a  handbook. london: facet publishing,  2004.    brice, anne, andrew booth, and nicola  benson. “evidence based librarianship:   a case study in the social sciences.”   world library and information  congress: 71st ifla general  conference and council, oslo, norway.  aug. 2005. 22 nov. 2007  <http://www.ifla.org/iv/ifla71/program me.htm>.    glynn, lindsay. “a critical appraisal tool  for library and information research.  “library hi tech 24.3 (2006): 387‐399.    grieves, maureen. “the impact of  information use on decision‐making:  studies in five sectors – introduction,  summary and conclusions.”  library  management19.2 (1998): 78‐85.    koufogiannakis, denise and ellen crumley.  “research in librarianship: issues to  consider.”  library hi tech 24.3 (2006):  324‐340.    lerdal, susan nissen. “evidence‐based  librarianship: opportunity for law  librarians?” law library journal 98.1  (2006): 33‐60.    marshall, joanne. g. “influencing our  professional practice by putting our   knowledge to work.”  information  outlook 7.1 (january 2003): 40‐44.    pfeffer, jeffrey and robert i. sutton.  “evidence‐based management.”  harvard business review 84.1  (january 2006): 62‐74.    _ _ _.  hard facts, dangerous half‐truths  and total nonsense. boston: harvard  business school press, 2006.    reid, christine, julie thomson, and jayne  wallace‐smith. “impact of information  on corporate decision making: the  uk banking sector.” library  management 19.2 (1998): 86‐109.    rousseau, denise m. “is there such a thing  as ‘evidence‐based management’?”   academy of management review 31.2  (april 2006): 256‐269.    turner, kathlyn j. “do information  professionals use research published  in lis journals?” 68th ifla council  and general conference, glasgow,  scotland. aug. 2002. 22 nov. 2007  <http://www.ifla.org/iv/ifla68/papers/0 09‐118e.pdf>    walton, graham. “theory, research and  practice in library management: new  column for library management.”  library management 28.3 (2007): 163‐ 164.    williams, lauren l. “what goes around  comes around: evidence‐based   management.”  nursing  administration quarterly 30.3 (july‐ september 2006): 243‐251 http://www.ifla.org/iv/ifla71/programme.htm http://www.ifla.org/iv/ifla71/programme.htm http://www.ifla.org/iv/ifla68/papers/009%e2%80%90118e.pdf http://www.ifla.org/iv/ifla68/papers/009%e2%80%90118e.pdf microsoft word news4162 evidence based library and information practice 2008, 3:4 89 evidence based library and information practice news/announcements how spicy can it get? librarian meets librarian at the 3rd international evidence based library and information practice conference, brisbane, 2005 suzanne lewis gosford hospital library nsw, australia e-mail: slewis@nsccahs.nsw.gov.au ray a’court macquarie university nsw, australia e-mail: racourt@mq.edu.au © 2008 lewis and a’court. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. when health librarian lisa cotter from nsw, australia, set out to attend the 3rd international evidence based library and information practice conference held in brisbane in 2005, her aim was to meet andrew booth, eblip guru. little did she know that she would meet two andrews and go on to marry one of them three years later! her experience prompted library colleagues of both parties to ask the following research question: how does meeting andrew spencer, librarian from macquarie university, nsw, australia, compare to meeting andrew booth in terms of quality of life outcomes? colleagues of lisa and andrew attempted to answer this question. their first step was to break the research question down using the spice methodology: setting third international evidence based library and information practice conference, brisbane, australia, 2005 perspective librarian and eblip groupie lisa cotter intervention meeting andrew booth comparison meeting andrew spencer evaluation outcomes measured using quality of life indicators evidence based library and information practice 2008, 3:4 90 action research methods, mainly observation, were used to obtain evidence. the subject, lisa cotter, was observed interacting with the comparison, andrew spencer: a cup of coffee during a break in conference proceedings; the odd glance over a poster display; excitedly relating amusing reference queries. there was a slight catch in andrew’s voice when he talked about lisa’s coding. it was not long before both parties were dropping the odd suggestive boolean operator. both lisa and andrew complimented each other on how evidence based their practice really was. andrew booth made a valiant attempt to divert lisa’s attention by referring to her as 'an evidence based practice supermodel', but andrew spencer trumped this by offering lisa the enticements of a web 2.0 environment. soon lisa and andrew were well past the stage of using a controlled vocabulary. outcomes were measured using quality of life indicators. the outcomes of the intervention – a couple of papers published in this journal and a conference paper or two – rated far lower than the outcomes of the comparison which included a wedding on 13 september 2008, held at the state library of new south wales, and a baby expected in january 2009. this is evidence of the highest quality according to all critical appraisal methodology. therefore, lisa and andrew’s colleagues, friends and families concluded that while the desired intervention (meeting andrew booth) was achieved, the comparison (meeting andrew spencer) resulted in high quality outcomes which were applied to everyday practice – love, happiness, family and joy. update thomas paul cotter spencer arrived seven weeks early on friday, 14 november 2008, weighing 4.65 pounds (2.1 kilograms). mother, father, and baby are all doing well. thomas is still being cared for in hospital but is making great progress and is expected to be home well before christmas. microsoft word news_eblig.doc evidence based library and information practice 2007, 2:2  129 evidence based library and information practice     news    eblig report: interest group meets in atlantic canada and new co‐convenors are  chosen      virginia wilson  coordinator, saskatchewan health information resources partnership (shirp)  university of saskatchewan  e‐mail: virginia.wilson@usask.ca    lyn currie  head, education library  university of saskatchewan  e‐mail: lyn.currie@usask.ca        © 2007 wilson and currie. this is an open access article distributed under the terms of the creative  commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    may was a month of travel for many  evidence‐based librarianship interest  group (eblig) members. from may 6 – 11,  the 4th international evidence based library  and information practice conference was  held in durham‐chapel hill, north carolina.  from may 22 – 26, eblig members  attended the canadian library association  (cla) conference in st. john’s,  newfoundland. as the second anniversary  of the formation of eblig has come around,  the inaugural co‐conveners, lyn currie and  virginia wilson, are wrapping up their two‐ year term.  congratulations to sue fahey of  memorial university of newfoundland and  renée degannes, canadian dental  association, ottawa, who have taken over  the reins and accepted a two‐year term as  eblig’s new co‐conveners.    eblig has been active this past year, mainly  in the area of conference workshops and  sessions. at the organizers’ invitation, we  successfully submitted and presented a half‐ day post‐conference workshop for the 4th  international evidence based librarianship  conference. how to assess the evidence: a  critical appraisal tool for library and  information research, facilitated by lindsay  glynn, of memorial university of  newfoundland, was an extremely well‐ received session with nearly 30 participants  in attendance.    mailto:wilson@usask.ca mailto:currie@usask.ca http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:2  130 virginia wilson and stephanie hall created  a practical toolkit on evidence‐based library  and information practice specifically for  public librarians:  http://ebltoolkit.pbwiki.com/. this toolkit  was unveiled at the 2007 cla conference in  st. john’s by stephanie hall.     a wiki was launched in 2007 entitled  eblibrarianship: the wiki of the evidence  based librarianship interest group (eblig)  to facilitate interest group collaboration,  sharing, and information access, as well as a  means of communication:  http://eblibrarianship.pbwiki.com/     eblig members are active in the eblip  community and beyond:     • eblig members edited and  contributed to an ebl‐themed issue  of feliciter.    • eblig members are participating as  editors, on the editorial advisory  board, and on the evidence  summary team of the open access  journal, evidence based library and  information practice.    • as a result of a posting on the  eblibrarianship wiki for the cla  conference session, evidence‐based  librarianship: a toolkit for public  libraries, virginia wilson and  stephanie hall have been invited to  present a 1‐hour audio conference  for the education institute in the fall  of 2007.     international membership to the interest  group is available without having to become  a full member of cla. for only $30 cad,  people residing outside of canada can join  eblig and take advantage of networking  and continuing education opportunities.  more information on international  membership is available at  http://www.cla.ca/about/igroups/evidence_b ased.htm.     the past two years have been challenging  and exciting. an active and engaged  membership goes a long way in making this  interest group vital and productive.    http://ebltoolkit.pbwiki.com http://eblibrarianship.pbwiki.com http://www.cla.ca/about/igroups/evidence_b research article   organizational factors as predictors of knowledge management practices in federal university libraries in nigeria   cyprian i. ugwu post-doctoral research fellow university of south africa pretoria, 003 south africa email: cyprian.ugwu@unn.edu.ng   received: 21 nov. 2016  accepted: 12 apr. 2018        2018 ugwu. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip28601     abstract   objective – university libraries in nigeria are facing challenges arising from poor funding, increasing user demands, and a competitive information environment. knowledge management has been accepted by information professionals as a viable management tool, but issues surrounding its application require empirical investigation. the aim of this study is to determine the organizational factors that are correlates and predictors of knowledge management practices in federal university libraries in nigeria.   methods – the study was based on a correlational research design. twenty heads of university libraries in nigeria responded to a structured questionnaire developed by the researcher. the questionnaire was validated by experts and its internal reliability was 0.78 obtained through cronbach’s alpha procedures. the data collected were analyzed using mean, standard deviation, one-way anova, pearson’s product moment correlation coefficient, and regression analysis.   results – the study found that management support and collaboration were the most significant predictors of knowledge management practices in federal university libraries in nigeria.  even though human resources policy and rewards systems had positive correlations with knowledge management practices, their correlation coefficients were not significant.   conclusion – the success of knowledge management in university libraries in nigeria depends on some contextual factors such as the support given by the management staff and the extent of collaboration among staff.       introduction   the information environment in which academic libraries operate today is changing rapidly. it is also clear that this changing information environment is largely due to the internet and digital revolution, thus creating new roles for librarians. baruchson-arbib and bronstein (2002) have identified three roles for information professionals arising from the new information environment. first, library managers have to secure access to information not available in their libraries because no library is self-sufficient and capable of providing all the information needed by users. second, information professionals are expected to provide user-centred services aimed at providing the right information to the right user. third, lis professionals should be more proactive and assertive in the new information environment. knowledge management is a necessity as a result of these shifting roles (maponya, 2004). inkinen, kiano, and vanhala (2015) defined knowledge management practices as a set of management activities that enable an organization to deliver value from its knowledge resources.   university libraries today should provide services to fulfil the roles created by the new information environment. unfortunately, university libraries in nigeria lack adequate information resources, have poorly organized collections, lack internet connectivity, have poorly developed electronic resources, and suffer from decreasing budgetary allocations (igbo & imo, 2011). mabawonku (2004) stated that the information resources of the nigerian university libraries were “overstretched” and “inadequate” (p. 67). he added that most of the books were outdated and journal subscriptions irregular. the solution may lie with knowledge management which has the potential to help libraries to deliver quality services and to be more innovative (islam, agarwal, & ikeda, 2017; obeidat, al-suradi, masa’deh, & tarhini, 2016; plessis, 2007; shang, lin, & wu, 2009). however, since the emergence of knowledge management over two decades ago, much attention in knowledge management studies has been on technological solutions (brun, 2005; detienne, dyer, hoopes, & harris, 2004; yang, 2007). though knowledge management depends on people management and human-related factors, there is a lack of empirical evidence on enablers for knowledge management success in university libraries (ajiferuke, 2003). the aim of this paper is to contribute to the knowledge management literature through a quantitative analysis of the organizational factors that predict knowledge management practices in university libraries based on the views of university librarians. because librarians have different views of knowledge management, this topic was approached from a specific viewpoint that resulted in a relatively small sample size. that is, instead of considering the views of all academic librarians, the study considered only the views of university librarians. these university librarians are library directors and are expected to play the role of knowledge management officers in their libraries. specifically, this paper seeks to determine: (1) the relationship between organizational factors and knowledge management practices in federal university libraries in nigeria, and (2) the organizational factors that predict knowledge management practices. based on these specific objectives, the following research questions were formulated:   1.       which knowledge management practices are used in federal university libraries in nigeria? 2.       which organizational factors for knowledge management practices are used in university libraries? 3.       how do organizational factors correlate with and predict knowledge management practices?   review of the literature   knowledge management   knowledge management has no accepted definition. as a result, scholars from different disciplines are debating the meaning of knowledge management from different schools of thought and different dimensions (nonaka &takeuchi, 1995).   the scholars affiliated with these schools of thought have different perceptions of knowledge management. sveiby (1996) summarized these schools of thought into two. according to sveiby, the first school of thought believed that knowledge management was about the management of information. researchers in this group viewed knowledge as objects that could be identified and handled or processed in any information system. they also equated knowledge with information access with a focus on building and managing knowledge stocks (alavi & leidner, 2001). by seeing knowledge management as the management of information, these researchers believed that knowledge management was all about technology. the second school of thought believed that knowledge management was about management of people (sveiby, 1996). the researchers in this group also believed that knowledge management was concerned with knowledge flows or knowledge processes in organizations. they maintained that these knowledge processes are found within the organizational environment. this may be clearly understood from the dimensions of knowledge management.   the dimensions of knowledge management, according to brun (2005), include people, process, and technology. okunoye (2003) and handzic (2001) both identified processes and enablers as dimensions of knowledge management. according to okunoye, “when we talk about knowledge management, we are primarily talking about supporting the knowledge processes with enablers” (p. 34). the enablers in this definition are regarded as organizational factors. the implication of okunoye’s definition is that, first, the management of knowledge begins with the identification of the internal processes of the organization. second, the enablers or organizational factors that support the processes should be identified.   knowledge management process   the knowledge management process, according to davenport (1993), includes acquisition, creation, packaging, and application or re-use of knowledge. galagan (1997) expanded this and proposed a knowledge management process that consisted of gathering new knowledge, accessing knowledge, representing knowledge, embedding knowledge, transferring knowledge, using knowledge, facilitating knowledge, and measurement. rufai and seliaman (2004) listed the elements of the knowledge management process as creating knowledge, capturing knowledge, representing knowledge, updating knowledge, disseminating knowledge, and validating knowledge. according to rufai and seliaman, knowledge is created as people determine new ways of doing things or develop know-how. sometimes if knowledge did not reside within the organization, external knowledge could be brought in. the knowledge that was created needed to be stored in its raw form in a database. most organizations used many different types of knowledge repositories to capture new knowledge (wang, 2002).      table 1 emergent content categories of knowledge management practices categories of knowledge management practices description (from theory-based literature) knowledge identification this refers to the knowledge activities aimed at identifying users’ needs and requirements for the purpose of providing them with a variety of quality services. activities underlying knowledge identification include establishing contacts with users, studying the university curricula, participating in teaching and research in the university, and conducting user surveys (balague, duren, & saarti, 2015; ilo, 2004; maponya, 2004). knowledge acquisition this refers to activities directed at seeking and obtaining knowledge from external sources and also from the internal environment. these activities include networking with other libraries; attending training programmes, conferences, seminars and workshops; using library collections; and collating internal profile of staff (balague et al., 2015; ilo, 2004; maponya, 2004). knowledge creation this refers to the analysis of knowledge gathered from internal and external sources for the purpose of creating new knowledge. this analysis takes the form of cataloguing of online information resources, creation of databases of experts, indexing of knowledge generated in the university, and building knowledge repositories (balague et al., 2015; ilo, 2004; maponya, 2004). knowledge dissemination this refers to activities aimed at making knowledge resources and services accessible to users. this can be achieved by using library alert systems, library mailing lists, new technologies, groupware, internet, library presentations, virtual tours of the library, intranet, and library guides (balague et al., 2015; ilo, 2004; maponya, 2004).       wang further argued that new knowledge must be placed in context to be actionable and be made available in a useful format to anyone in the organization who needed it. maponya (2004) drew on the previous literature to identify the key dimensions of knowledge management processes as acquiring, capturing, creation, and sharing of knowledge.   applying knowledge management successfully in a university library requires the selection of knowledge management processes that cover completely the range of library activities or operations and services. according to ugwu, idoko, and enem (2013), the key to knowledge management is capturing the knowledge of library processes or how libraries get their work done. the knowledge management processes selected for this study were adapted from the international labour organization (ilo) (2004) model. the knowledge management processes as identified by the inspection unit of the ilo consisted of the identification of required knowledge, capturing knowledge, organizing knowledge, and sharing knowledge. the meaning and descriptions of these knowledge processes are shown in table 1; in addition, see appendix 1 for the item measures of the categories of knowledge management process.   organizational factors   some authors have tried to identify the factors that could influence knowledge management.  for instance, the following factors have been found to influence knowledge management: information systems, organizational structure, reward systems, processes, people, and leadership (gupta & govindarajan, 2000; holoweztki, 2002; martin, 2000). thomas (2006) suggested that organizations needed a small number of the following factors: top management leadership, human resources policy, compensation schemes, collaboration, and measurement. other studies have identified these factors as culture, technology, leadership, human resources practices, and innovation (donate & guadamillas, 2011; yusuf & wanjau, 2014). holsapple and joshi (2000) carried out a literature review that yielded eight factors that potentially influenced knowledge management in organizations. the authors later expanded these factors to produce 16 factors which were grouped into three as the major kinds of forces influencing knowledge management in organizations, namely managerial influence, resource influence, and environmental influence. it was from these studies that the organizational factors for this study were selected. the factors include management support, human resources development, reward systems, and collaboration. the meaning and descriptions of these factors are shown in table 2; in addition, see appendix 1 for the item measures of the categories of organizational factors.     table 2 emergent content categories of organizational factors categories of organizational factors description (from theory-based literature) management support this refers to the extent to which knowledge management efforts are promoted by the library leadership, where the library leadership refers to the individual or individuals responsible for allocating resources for knowledge management and for specifying knowledge management initiatives for the library, explaining the importance of knowledge management to staff, building trust among staff, developing a written knowledge management policy or formulating knowledge management goals as well as leading by example (thomas, 2006; von krogh, nonaka, & rechsteiner, 2011). human resources development this refers to the activities that are intended to encourage staff to participate in knowledge management initiatives of the library. these activities include training of staff to acquire knowledge management competencies, rotating staff on the job, ensuring that staff are placed in the right positions in the library, and developing appropriate procedures for staff retention (syed-ikhsan & rowland, 2004; thomas, 2006). reward systems reward systems consist of activities that motivate staff to embrace knowledge management, or mechanisms developed in the library to recognize and appreciate the knowledge behaviour of staff. these activities or mechanisms include recognition of staff achievement, appreciation of knowledge management efforts of staff, and monetary rewards such as incentives and other benefits (hasanali, 2006; thomas, 2006). collaboration this refers to the extent to which individuals communicate, cooperate, and help one another through sharing of knowledge and expertise. this can be encouraged through social networks such as teams, work groups, and communities of practice and through mentoring of staff and effective communication flows in the library (syed-ikhsan & rowland, 2004; thomas, 2006).       organizational factors and knowledge management process   several researchers have explored the relationship between organizational factors and knowledge management processes (brewer & brewer, 2010; donate & guadamillas, 2011; rosmaini &woods, 2007; schulte &wang, 2004; singh, 2008; thomas, 2006). positive interactions were found between human resources activities and knowledge management activities (brewer & brewer, 2010; cabrera & cabrera, 2005). brewer and brewer identified the human resources activities as teamwork, promotion of positive attitudes, socialization programmes, team performance appraisal, and reward system. thomas’s study revealed positive and significant relationships between organizational factors and knowledge management activities. thomas found these factors to consist of management support, human resources policy, collaboration, and reward system. donate and guadamillas (2011) provided empirical evidence on the relationship between knowledge management and organizational elements such as culture, leadership, human resources practices, and innovation. other factors that have been found to correlate positively with knowledge management activities include leadership (singh, 2008), communication flows (rosmaini &woods, 2007), and collaboration and training (schulte &wang, 2004). some authors have also found that reward systems are very effective in motivating knowledge workers to partake in knowledge management activities (al-adaileh & al-atawi, 2011; chua, 2009; hansen, nohria, & tiemy, 1999). ajiferuke (2003) and bouthillier and shearer (2002) have advocated that similar studies should be carried out in a university library environment. based on these studies, the following hypotheses were formulated:   h1: there will be a significant joint relationship between organizational factors and knowledge management practices in university libraries in nigeria.   h2: there will be a significant relative relationship between organizational factors and knowledge management practices in university libraries in nigeria.                     methods   research type   a quantitative research approach was adopted in this study. leedy and ormrod (2005) stated that quantitative methodology is used to answer questions about relationships among measured variables with the purpose of explaining, predicting, and controlling phenomena and further added that it may be distinguished as being a traditional, experimental, or positivist approach. thus, quantitative methodology is suitable for this study as it establishes the relationships between variables or helps to test hypotheses or to determine the predictive values of variables. further, the most suitable design for this study based on this approach is a correlational research design. correlational research is used to determine the relationships between two or more variables, ordinarily through the use of correlation coefficients (joyner, rouse, & glatthorn, 2013). kumar (2014) stated that the main purpose of a correlational study is to discover or establish the existence of a relationship or association or interdependence between two or more aspects of a situation. this design was chosen because it would help to determine the relationship between the organizational factors as independent variables and knowledge management practices in university libraries as the dependent variable.   measures   scales for the measurement of knowledge management processes and organizational factors were drawn from prior literature. to measure these elements, the researcher used a five-point likert scale that ranged from 1 = strongly disagree to 5 = strongly agree. the organizational factors identified by thomas (2006) were utilized in this paper: management support, human resources policy, reward systems, and collaboration. measures assessing management support were developed from prior studies (thomas, 2006; von krogh et al., 2011) that encompassed two functional dimensions, namely, stating organizational goals and building trust among staff. the second factor, human resources policy, was measured through job placement and staff training. the measurement items of these organizational elements were drawn from studies by syed-ikhsan and rowland (2004) and thomas (2006). further, item measures relating to reward systems were developed from previous studies on extrinsic and intrinsic reward systems (hasanali, 2000; thomas, 2006). measures for collaboration were also developed from previous studies (syed-ikhsan & rowland, 2004; thomas, 2006) that dealt with two of its aspects, namely learning activities and communication flows. the measures of the dependent variable, knowledge management practices or knowledge management process, were based on the international labour organization’s (2004) study on knowledge management process. four categories of knowledge management process were adapted from this study, namely knowledge identification, knowledge acquisition, knowledge creation, and knowledge dissemination. further, item measures for each of the categories of knowledge management process were developed from previous research literature (balague et al., 2015; maponya, 2004).   research participants and data collection   a survey questionnaire entitled “organizational factors as predictors of km questionnaire” (see appendix 1) was used to collect data for hypotheses testing. before implementing the survey, the instrument was reviewed by four experts—three of whom were senior lecturers in the field of library and information science and the fourth was a professor of educational measurement and evaluation all from the university of nigeria, nsukka—in order to identify problems with wording, content, and question ambiguity. after some changes were made based on their suggestions, the modified questionnaire was piloted on 10 management staff of the university of nigeria, nsukka library system so as to determine the internal consistency of the research instrument using cronbach’s alpha procedures. cronbach’s alpha is one numerical coefficient used to measure the reliability of summated scales such as likert scales. it estimates the internal consistency of scales (gliem & gliem, 2003). since a 5-point likert scale was used to measure the responses of the participants, the researcher found cronbach’s alpha the most appropriate procedure for establishing internal consistency of the research instrument for this study. the scales used in this study were found to be reliable as their cronbach’s alpha values were 0.78 and 0.86 for knowledge management process and organizational factors, respectively.   the copies of the questionnaire were sent via email to all the university librarians at the federal university libraries in nigeria. these libraries were chosen because they were supposed to have established library and information services with fully developed library operations. there are presently 33 federal university libraries corresponding to the number of federal universities in nigeria. the university librarians were chosen because of their experience in library operations and services. introductory letters and the survey were emailed to 29 university librarians whose email addresses were obtained from the attendance list of 33 university librarians from the federal university libraries who attended the 2015 meeting of the committee of university librarians of nigerian universities. attendance at this meeting consisted of university librarians from federal, state, and private universities in nigeria. no ethical clearance was required for this study. however, the researcher assured the participants of their safety and anonymity. in order to protect the identity of the subjects, no names, email addresses, or library names were gathered. furthermore, each participant was asked to voluntarily participate in this study, spend 10 to 15 minutes responding to the questionnaire, and return the survey within 10 days. in total, 20 (67%) of the subjects completed and returned the survey.   data analysis   to summarize the data collected on knowledge management practices and organizational factors, means and standard deviations were used and the mean scores were ranked. the pearson’s product moment correlation coefficient was used to determine the relationships between organizational factors and knowledge management practices. the two hypotheses (h1 and h2) were tested at 0.05 probability levels using multiple regressions and anova. all the statistical tests used in this study were computed with the aid of statistical packages in social sciences (spss).   results   what knowledge management practices are in use in university libraries in nigeria?   the knowledge management practices in federal university libraries in nigeria are as described and presented in table 3. to describe the responses on the knowledge management practices, the mean and standard deviation were estimated for each item. table 3 shows that the highest responses in knowledge management activities were in the area of knowledge acquisition, which involves using library resources (mean = 3.49, sd = 0.61), online resources (mean = 3.22, sd = 0.72), and consultation with colleagues (mean = 3.03, sd = 0.77). the next highest knowledge management activity was participation of librarians in the teaching and research activities in the university (mean = 3.16, sd = 0.92), which falls in the knowledge identification domain. in the area of knowledge dissemination, the highest knowledge management activity was library presentations and demonstrations (mean = 2.94; sd = 0.98).   what organizational factors for knowledge management are in use in university libraries in nigeria?   the purpose of table 4 is to describe the organizational factors for knowledge management practices in federal university libraries in nigeria. the analysis of data depicted in table 4 shows that there are factors in the university library environment in nigeria that are likely to promote knowledge management activities. the most important among these factors include mentoring of staff (mean = 3.57, sd = 0.77), interest in the job (mean = 3.48, sd = 0.72), recording of staff achievements (mean = 3.36, sd= 0.63), enhancing job satisfaction (mean = 3.28, sd = 0.66), and rotating staff on the job (mean = 3.27, sd = 0.77). from these major factors, mentoring of staff is concerned with the nature of collaboration in the library. interest in the job as well as recording staff achievements and enhancing staff job satisfaction refers to the nature of reward systems, whereas rotation of staff on the job and emphasizing the importance of knowledge management relate to human resources policy and management support, respectively.     table 3 means, standard deviations, and ranks of responses on knowledge management processes km process              mean sd rank knowledge identification participating in the teaching and research activities in the university   3.16   0.92   1 contact with users 2.83 0.75 2 survey results and academic programmes   2.82   0.73   3 overall mean for knowledge identification 2.95 0.80 knowledge acquisition library collections as source of knowledge   3.49   0.61   1 internet as source of knowledge 3.22 0.72 2 consultations with colleagues, interviews and experience   3.03   0.77   3 overall mean for knowledge acquisition 3.25 0.70 knowledge creation creating databases   2.66   1.05   1 cataloguing of online resources 2.62 0.82 2 indexing of knowledge generated in the university   2.52   0.79   3 overall mean for knowledge creation 2.60 0.87 knowledge dissemination library presentations and demonstrations   2.94   0.98   1 use of  library notices, phones, email, library alert system and fax 2.92 0.97 2 use of university newsletters 2.55 1.15 3 overall mean for knowledge dissemination 2.80 1.03       table 4 means, standard deviations, and ranks of responses on organizational factors organizational factors mean sd rank management support emphasizing the importance of knowledge management   3.36   0.72   1 welcoming contributions from staff 3.21 0.79 2 identifying knowledge gaps and updating library policies regularly 3.20 0.72 3 aligning knowledge management policy with the library’s vision 3.13 0.67 4 maintaining an open door policy 3.08 0.87 5 making knowledge management policy available to staff   2.91   0.78   6   having a written knowledge management policy   2.86   0.74   7 overall mean for management support 3.11 0.76 human resources policy recording of staff achievement   3.36   0.63   1 rotating staff on the job 3.27 0.77 2 allowing staff to undertake formal training 3.20 0.66 3 sponsoring staff to conferences and workshops 3.11 0.68 4 identifying staff with valuable tacit knowledge 2.99 0.67 5 encouraging self-improvement of staff 2.95 0.85 6 short courses on knowledge management for staff   2.80   0.73   7 overall mean for human resources policy 3.05 0.73 reward system increasing interest in the job   3.48   0.72   1 enhancing job satisfaction 3.28 0.66 2 providing positive feedback on the job 3.12 0.63 3 conducting staff annual performance appraisals 2.76 0.76 4 providing other incentives and benefits to staff   2.66   0.70   5 overall mean for reward system 3.11 0.68 collaboration mentoring of staff   3.57   0.77   1 support for both top-down and bottom-up communication 2.89 0.94 2 formulation of communities of practice or research groups 2.80 0.87 3 encouraging staff to help one another 2.56 0.75 4 support for top-down communication only 2.47 0.79 5 support for bottom-up communication only 2.42 0.72 6 overall mean for collaboration 2.79 0.81       table 5 means, standard deviations, and correlations between organizational factors and km practices   variables m sd 1 2 3 4 5 6 7 8 1 knowledge identification 2.95 0.80 1.00               2 knowledge acquisition 3.25 0.70 .431 1.00             3 knowledge organization 2.60 0.87 .362 .432 1.00           4 knowledge dissemination 2.80 1.03 .503 .376 .467 1.00         5 management support 3.11 0.76 .627 .347 .455 .540 1.00       6 human resources dev. 3.05 0.73 .583 .243 .331 .386 .666 1.00     7 reward systems 3.11 0.68 .581 .297 .231 .401 .652 .636 1.00   8 collaboration 2.79 0.81 .497 .413 .355 .565 .384 .424 .533 1.00     table 6 summary of one-way anova of the relationship between organizational factors and km practices model sum of squares df mean square f sig regression residual total 9,626.081 18,392.408 28,018.489 3 16 19   3,208.694 1,149.526 2.79 0.021     table 7 regression analysis of the relative contributions of organizational factors to knowledge management practices   unstandardized  coefficient standardized coefficient t sig. model                b std error beta     constant management support human resources dev. reward system collaboration 14.671 .254 .177 .128 .245 2.693 .090 .109 .104 .074    .337 .178 .138 .218 5.522 3.122 1.634 1.237 2.440 0.000 .002 .106 .219 .017 adj. r = .5648; adj. r2 = .319; standard error of estimate = 9.708       what are the correlates of knowledge management practices?   in this study, the organizational factors are the independent variables whereas the dependent variables are the knowledge management practices. table 5 shows a correlation matrix demonstrating the relationship between organizational factors and knowledge management practices.   1.       the mean scores on the knowledge management variables ranged from 2.60 to 3.25, indicating that in the federal university libraries in nigeria there is some level of knowledge management practices with supportive organizational factors, but that these are not strongly present. 2.       the high standard deviation scores on knowledge organization and knowledge dissemination showed that the opinions of the respondents were polarized, or rather that the respondents were having issues with knowledge organization and dissemination. this might have influenced the mean scores, or such a situation might be responsible for the reporting of some level of knowledge management activities in the university libraries. 3.       the mean scores on the organizational elements ranged from 2.79 to 3.11, indicating that these elements were not strongly utilized to provide support for knowledge management activities in the university libraries. 4.       the respondents’ opinions on the organizational factors did not show wide variability. the coefficient of variation was as low as 16%. 5.       the correlation matrix depicted in table 5 revealed a positive correlation between organizational factors and knowledge management practices. 6.       knowledge identification had the strongest positive correlation with organizational factors. the correlation coefficient between these variables ranged from 0.50 to 0.63. 7.       management support had the strongest positive correlation with knowledge management practices. the correlation coefficient between management support and knowledge management variables ranged from 0.35 to 0.63. 8.       knowledge acquisition had the weakest positive correlation with organizational factors. the correlation coefficient between these variables ranged from 0.24 to 0.41. 9.       reward system had the weakest positive correlation with knowledge management variables. the correlation coefficient between reward systems and knowledge management variables ranged from 0.23 to 0.58.   hypotheses testing (hi and h2)   joint relationship between organizational factors and knowledge management practices (h1)   h1 was formulated to test at 0.05 probability level the joint relationship between organizational factors and knowledge management practices. the analysis was done using one-way anova. table 6 shows a joint relationship between organizational factors and knowledge management practices. the result of the one-way anova revealed that the f-test was significant at 0.05 probability levels. this implies that at least one of the organizational factors was a significant predictor. this provides support for h1. therefore, the combined effect of the organizational factors on knowledge management practices was significant.   relative relationship between organizational factors and knowledge management practices (h2)   h2 was formulated to test at 0.05 probability level the relative relationship between organizational factors and knowledge management practices. this hypothesis was formulated to provide answers to the fourth research question: what are the predictors of knowledge management practices? regression analysis was employed in the analysis. table 7 shows the results of the regression analysis on the individual contributions, or the predictive values of the organizational factors in relation to knowledge management practices in university libraries in nigeria. the knowledge management process was regressed on the organizational factors. the factors, as shown in table 7, with the most significant contributions are management support (beta = 0.337, t = 3.122, p < 0.05) and collaboration (beta = 0.218, t = 2.440, p < 0.05). other factors like human resources policy and reward systems showed no significant contributions to knowledge management practices. these factors accounted for 32% of the variance in knowledge management practices in federal university libraries in nigeria. this evidence shows that h2 is not fully supported because reward systems and human resources policy were not significant predictors.   discussion   the results of the study showed the important measures of knowledge management practices and those of organizational factors as well as the correlates and predictors of knowledge management practices in federal university libraries in nigeria.   knowledge management practices   the findings of this study revealed that knowledge identification, knowledge acquisition, knowledge creation, and knowledge dissemination are important measures of knowledge management activities in federal university libraries in nigeria. the responses of the university librarians on each of these measures indicate that university libraries are gradually engaging in knowledge management activities. the greatest areas of knowledge management activities were knowledge acquisition and knowledge dissemination. the specific activities carried out in the area of knowledge acquisition consist mainly of using library resources and online resources for knowledge services. the libraries were equally engaged in library presentation and demonstrations as a knowledge dissemination activity. however, emphasis on knowledge management process appears to be more on manual operations than automated systems. these results are not surprising because technological infrastructures have not been fully developed in university libraries in nigeria. second, many librarians in these libraries have yet to come to terms with the use of web 2.0 and web 3.0 in providing library and information services. the findings support the km process in academic libraries identified by maponya (2004) and balague et al. (2015). according to maponya, the knowledge management process in academic libraries involves the capturing, sharing, or dissemination and utilization of knowledge. maponya further identified specific knowledge management activities in academic libraries as participation in the teaching and research activities of the university (knowledge identification), collating internal profiles of academic librarians (knowledge creation), establishing knowledge link or contacts (knowledge acquisition), and using both internal and external media to disseminate knowledge.   organizational factors   the findings also revealed that management support, human resources policy, reward system, and collaboration constituted the organizational dimensions of the knowledge management implementation process in university libraries in nigeria. the mean scores obtained on the organizational elements show that university librarians have neutral responses about their roles in knowledge management. however, the highest responses of the librarians on the organizational factors used for knowledge management practices are in the areas of providing management support and having a structured reward system that defines specific intrinsic and extrinsic reward activities to encourage staff participate in knowledge management. these findings support mosoti and masheka’s (2010) study that maintained that knowledge management should be implemented as part of organizational leadership. the findings are also in conformity with the studies by king (2000), thomas (2006), gold, malhotra, and segars (2001), holowetzki (2002), holsapple and joshi (2000), and yassin, salim, and salari (2013) that laid the foundation for the empirical investigations of the organizational related factors for knowledge management success and identified such factors as leadership, organizational culture, human resources activities, communication flows, and non-hierarchical organizational structure.   correlates and predictors of knowledge management practices   the positive relationship between organizational factors and knowledge management practices is in conformity with the results of similar studies in other public organizations (thomas, 2006). this finding is not unexpected because lis professionals have accepted knowledge management as either a re-branding of librarianship and information management or a new dimension of both disciplines (husain & nazim, 2013).the factors with the most significant contributions are management support and collaboration. this finding is not consistent with those of thomas (2006) that showed a significant correlation between organizational factors and knowledge management systems. the possible explanations for these contradictory results may be that the respondents could not distinguish between management support and such factors like human resources policy and reward systems since both might be included in the management tools needed to facilitate knowledge management. second, the respondents might have felt that knowledge management issues revolved around management support and collaboration. these findings show that management support and collaboration are two critical success factors for knowledge management practices in federal university libraries in nigeria. this means that university librarians are expected to provide management support and foster collaboration among staff for the success of knowledge management practices.   practical implications of the findings   this study has three implications for university libraries in nigeria. first, the study reveals that knowledge identification has the most positive correlation with organizational factors. this implies that university libraries should pay more attention to identifying the needs of users in their knowledge management initiatives. more knowledge activities in this area should be intensified to ensure that user needs are not only identified but also met. to identify user needs, librarians should be encouraged to participate in teaching and research in the university and to maintain regular contact with users. second, the study reveals that management support is the most significant predictor of knowledge management practices. this means that the success of knowledge management depends largely on the extent of support provided by the library leadership. providing knowledge management leadership has been supported in the literature as a necessary condition for knowledge management success (singh, 2008). the present study reveals that this support takes several forms of commitments from the library leadership such as welcoming contributions from staff, updating library policies and procedures, and maintaining an open door policy. these leadership activities or commitments will help to build trust among the library staff. further, library leadership must try to explain the importance of knowledge management to staff and ensure that knowledge management policy is aligned with the library’s vision. explaining the importance of knowledge management to staff is crucial because its understanding will help in the formulation of knowledge management policies or goals for the purpose of integrating them into the libraries’ vision. finally, the study also reveals a significant correlation between collaboration and knowledge management practices. this implies that learning and communication are required for the success of knowledge management in the university libraries. this learning should take the form of group learning where staff are allowed to engage in mentoring and encouraged to help one another, or where staff are helped to form research groups as well as communities of practice. an effective communication system is equally important; a system that encourages top-down and bottom-up communication is supported by this study.   the study also has implications for researchers interested in knowledge management and managing university libraries in the era of change. the information environment is changing rapidly, and it is threatening the survival of academic libraries the world over. research has been ongoing in this regard from total quality management through learning organizations to knowledge management now. more research is therefore needed in the area of knowledge management in libraries to update current practice and provide enhanced services to library users.   limitations of the study   the following may be considered as the limitations of this paper. first, the number of university librarians studied was small, and this may have affected the findings. as a result, the findings are not generalizable. future research could benefit from using larger samples by involving all the university librarians in nigerian universities. second, common organizational elements for knowledge management process have been analyzed, but specific conditions may be necessary for clearer understanding of the relationships between these elements and the knowledge management process. as has been obtained from this study, human resources policy or practices and reward systems did not interact significantly with the knowledge management process. future studies could look in greater depth at the relationships between specific human resources practices and the knowledge management process, or specific compensation schemes and the knowledge management process.   third, a quantitative technique was used as the main data collection method. self-reported data collection techniques such as questionnaires are associated with quantitative techniques and may create a response bias. studies have shown that self-reported measures also create methods effects (podsakoff, mackenzie, lee, & podsakoff, 2003). methods effects have the potential to bias correlations and regression weights and, as such, correlated errors often affect the significance of statistical tests (sharma, yetton, & crawford, 2009). to this effect, it is possible that the support for all the hypotheses, which were tested at 0.05 significance levels, could be inflated. though it is unlikely that respondents would be able to anticipate the patterns of relationships among the variables studied, the researcher tried to minimize methods bias by ensuring the anonymity of the respondents and by withholding any forms of incentives during data collection. however, it is suggested that future research in this area should apply harman’s (1960) single factor and marker variable to statistically test common methods bias. it is also suggested that future research could consider using multiple methods of data collection, which should include more qualitative techniques.   fourth, the dimensions of the knowledge management process chosen in this study might have affected the findings as several dimensions of the knowledge management process can be found in the knowledge management literature. this study may have overlooked other important dimensions that could have correlated significantly with the identified organizational elements or factors. future studies could eliminate this kind of method effect by enhancing the validity of the measurement scale through convergent and discriminant validity tests. the essence of these tests is to enable the researcher to obtain good measures of what he or she wishes to measure. finally, further information was not provided as explanations to some of the item measures of the knowledge management categories and those of organizational factors. though the item measures were validated and their internal reliabilities determined, wrong interpretations of these measures by the participants could have some influence on the findings. future studies could address this issue by providing additional information to help the participants respond with less difficulty to the questionnaire items.   conclusion and recommendations   based on the findings of the study, the following recommendations are made:   1.       a knowledge management strategy should be formulated for knowledge management practices in federal university libraries in nigeria. 2.       the strategies for knowledge management practices should consist of knowledge identification, acquisition, organization, and dissemination. 3.       organizational commitment for knowledge management practices in university libraries should include management support and collaboration. 4.       research should be intensified on knowledge management in libraries to uncover more contextual factors needed for knowledge management success.   furthermore, though both management support and collaboration have significant correlations with knowledge management practices, the percentage contribution of these factors to knowledge management (32%) was not strong enough. this means that more activities or efforts are still needed in the two organizational dimensions for efficient knowledge management practices in university libraries in nigeria.     references   ajiferuke, i. (2003). role of information professionals in knowledge management programs: empirical evidence from canada. informing science, 6, 247–257. retrieved from https://www.informingscience.org/journals/informingscij/overview   al-adaileh, r. m., & al-atawi, m. s. (2011). organizational culture impact on knowledge exchange: saudi telecom context. journal of knowledge management, 15(2), 212–230. https://doi.org/10.1108/13673271111119664   alavi, m., & leidner, d. e. (2001). review: knowledge management and knowledge management systems: conceptual foundations and research issues. mis quarterly, 25(1), 107–136. retrieved from http://aisel.aisnet.org/misq/   balague, n., duren, p., & saarti, j. (2015). benchmarking the knowledge management practices in selected european higher education libraries. qualitative and quantitative methods in libraries, 4, 331–341. retrieved from http://www.qqml-journal.net/index.php/qqml/index   baruchson-arbib, s., & bronstein, j. (2002). a review to the future of the library and information science profession. journal of the american society for information science and technology, 53(5), 397–408. https://doi.org/10.1002/asi.10051   bouthillier, f., & shearer, k. (2002). understanding knowledge management: the need for an empirical perspective. information research, 8(1), 251–267. retrieved from http://www.informationr.net/ir/   brewer, p. d., & brewer, k. k. (2010). knowledge management, human resource management, and higher education: a theoretical model. journal of education for business, 85, 330–335. https://doi.org/10.1080/08832321003604938   brun, c. (2005). abc of knowledge management. retrieved from http://www.fao.org/fileadmin/user_upload/knowledge/docs/abc_of_km.pdf   cabrera, e. f., & cabrera, a. (2005). fostering knowledge sharing through people management practices. international journal of human resources management, 16(5), 720-735. https://doi.org/10.1080/09585190500083020   chua, a. y. k. (2009). the dark side of successful knowledge management activities. journal of knowledge management, 13(4), 32–40. https://doi.org/10.1108/13673270910971806   davenport, t. h. (1993). process innovation: reengineering work through information technology. boston: harvard business school.   detienne, k. b., dyer, g., hoopes, c., & harris, s. (2004). toward a model of effective knowledge and directions for future research, culture, leadership and ckos. journal of leadership & organizational studies, 10, 26–43.   donate, m. j., & guadamillas, f. (2011). organizational factors to support knowledge management and innovation. journal of knowledge management, 15(6), 890–914.   galagan, p. (1997). smart companies. training and development, 51(12), 20–25.   gliem, j. a., & gliem, r. r. (2003). calculating, interpreting and reporting cronbach’s alpha reliability coefficient for likert-type scales. paper presented at the midwest research to practice conference in adult, continuing and community education, ohio state university, columbus, oh.   gupta, a. k., & govindarajan, v. (2000). knowledge flows within multinational corporations. strategic management journal, 21, 473–496.   gold, a. h., malhotra, a., & segars, a. (2001). knowledge management: an organizational capabilities perspective. journal of management information system, 18(1), 185–214.   harman, h. h. (1960). modern factor analysis. chicago, il: university of chicago press.   handzic, m. (2001). knowledge management: a research framework. in proceedings of the 2nd european conference on knowledge management (eckm) (pp. 35–42). bled, slovenia.   hansen, m.t., nohria, n., & tiemy, t. (1999). what is your strategy for managing knowledge? harvard business review, 77(2), 106–116.   hasanali, f. (2002). critical success factors of knowledge management. retrieved from http://www.apgc.org/free/articles   holowetzki, a. (2002). the relationship between knowledge management and organizational culture: an examination of cultural factors that support the flow and management of knowledge within an organization. retrieved from http://aim.uoregon.edu/research/pdfs/ holowetzki2002.pdf   holsapple, c, w., & joshi, k. d. (2000). an investigation of factors that influence the management of knowledge in organizations. journal of strategic information systems, 9(2–3), 253–261.   husain, s., & nazim, m. (2013). concepts of knowledge management among library & information science professionals. international journal of information dissemination and technology, 3(4), 264–269.   igbo, h. u., & imo, n. t. (2011). challenges of accessibility of information resources by the postgraduate users of a nigerian university. the information technologist, 7(2), 1–5.   inkinen, h. i., kianto, a., & vanhala, m. (2015). knowledge management practices and innovation performance in finland. baltic knowledge management, 10(4), 432–455.   international labour organization (2004). knowledge management at the international labour organization. joint inspection unit. geneva: international labour organization.   islam, m. a., agarwal, n. k. & ikeda, m. (2017). effect of knowledge management on service innovation in academic libraries. ifla journal, 43(3), 266–281.   joyner, r. l., rouse, w. a., & glatthorn, a. a. (2013). writing a winning thesis or dissertation: a step-by-step guide (3rd ed.). thousand oaks, ca: corwin.   king, w. r. (2000). playing an integral role in knowledge management. information system management, 17(4), 59–61.   kumar, r. (2014). research methodology: a step-by-step guide for beginners (4th ed.). london: sage.   leedy, p. d., & ormrod, j. e. (2005). practical research: planning and design (8th ed.). upper saddle river, nj: pearson education.   mabawonku, i. (2004). library use in distance learning: a survey of undergraduates in three nigerian universities. african journal of library, archives and information science, 14 (2), 61–72.   maponya, p. n., (2004). knowledge management practices in academic libraries: a case study of the university of natal, pietermaritzburg libraries. retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.137.8283&rep=rep1&type=pdf   martin, b. (2000). knowledge management within the context of management: an evolving relationship. singapore management review, 22(2), 17–36.   mosoti, z., & masheka, b. (2010). knowledge management: the case for kenya. the journal of language, technology & entrepreneurship in africa, 2(1), 35–56.   nonaka, i., & takeuchi, h. (1995). the knowledge-creating company. oxford: oxford university press.   obeidat, b.y., al-suradi, m. m., masa’deh, r., & tarhini, a. (2016). the impact of knowledge management on innovation: an empirical study of jordanian consulting firms. management research review, 39(10), 1214–1238.   okunoye, a. o. (2003). knowledge management and global diversity: a framework to support organization in developing countries. finland: university of turku.   plessis, m. (2007). the role of knowledge management in innovation. journal of knowledge management, 1(4), 20–29.   podsakoff, p.m., mackenzie, s. b., lee, j. y., & podsakoff, n.p. (2003). common method biases in behavioural research: a critical review of the literature and recommended remedies. journal of applied psychology, 88(5), 879–898.   rosmaini t., & woods, p. c. (2007). relationship between corporate knowledge management and firm’s innovation capability. international journal of technology and management, 8(1), 62–79.   rufai, r., & seliaman, m. e. (2004). towards a knowledge management model for universities. retrieved from http://ickm.upm.edu. my/parallel%20session%202/raimi%20&%20seliamantowards%20a%20km%20model%20for%20universitiesdoc   schulte, w. d., & wang, p. o. j. (2004). perceptions of organizational factors of successful knowledge management: an exploratory study of knowledge workers in taiwan. retrieved from http://www.aibse.org/ proceedings/proceeding%2004/%20of20organization%20factors%20of%20successful.doc   shang, s., lin, s., & wu, y. (2009). service innovation through dynamic knowledge management. industrial management & data system, 109(3), 322–337.   sharma, r., yetton, p., & crawford, j. (2009). estimating the effect of common method variance: the method – method pair technique with an illustration from tam research. mis quarterly, 33(3), 473–712.   singh, s. k. (2008). role of leadership in knowledge management. journal of knowledge management, 12(4), 3–15.   sveiby, k. (1996).what is knowledge management? retrieved from http:// sveiby.com.au/knowledge management.html   syed-ikhsan, s. o. s., & rowland, t. (2004). knowledge management in a public organization: a study on the relationship between organizational elements and the performance of knowledge transfer. journal knowledge management, 8(2), 95–111.   thomas, t. d. (2006). an empirical investigation of factors providing knowledge management system success (doctoral dissertation). retrieved from   https://sedonaweb.com/attach/schools/mwsfaculty/attach/diss-bt1234mws.pdf   ugwu, c. i., idoko, n. a., and enem, f. n. (2013). university libraries, knowledge management and higher education in digital environment: the synergistic relationship. paper presented at the 12th annual conference, annual general meeting of the nigerian library association, enugu state chapter, enugu.   von krogh, g., nonaka, i & rechsteiner, l. (2011). leadership in organizational knowledge creation: a review and framework, journal of knowledge management studies, 12(2), 16–26.   yang, j. t. (2007). knowledge sharing: investigating appropriate leadership roles and collaborative culture. tourism management, 28, 530–543.   yassin, f., salim, j. & salari, n. (2013). the influence of organizational factors on knowledge sharing using ict among teachers. procedia technology, 11, 272–280.   yusuf, m. m., & wanjau, k. (2014). factors affecting implementation of knowledge management practices in state corporation in the national treasury in kenya, international journal of management technology, 2(2), 9–18.   wang, s. (2002). knowledge maps for managing web-based business. industrial management and data systems, 102(7), 357–364.     appendix questionnaire   instruction: choose as appropriate in the boxes provided in sections a and b as shown below:   1.       strongly agree = 5 2.       agree = 4 3.       neutral = 3 4.       disagree = 2 5.       strongly disagree =1 section a: knowledge management (km) process s/n indicate your level of agreement on km activities in your library 1 2 3 4 5   identifies knowledge through:           1 contact with users           2 participating in the teaching and research activities in the university           3 survey results and academic programmes             acquires knowledge through:           4 consultation with colleagues, interviews and personal experience           5 library collection           6 internet resources             creates knowledge by:           7 creating different databases to add value           8 indexing knowledge generated in the university           9 cataloguing of online resources to enhance access             disseminates knowledge through:           10 library presentation and demonstrations           11 use of library notice, phones, email, library alert system and fax           12 university newsletter             other (specify)             section b: organizational factors for knowledge management (km) s/n indicate your level of agreement on the following organizational activities  are ongoing in your library for km 1 2 3 4 5   management support           1 my library has a written km policy           2 aligns km policy with the library’s vision           3 makes km policy available to staff           4 emphasizes the importance of km           5 maintains open door policy           6 welcomes contributions from staff           7 identifies knowledge gaps and updating library policies regularly             human resources policy or practices           8 sponsors staff to conferences/workshops           9 encourages self-improvement of staff           10 approves formal training of staff           11 approves short courses in knowledge management for staff           12 rotates staff on the job           13 identifies staff with valuable tacit knowledge             reward systems           14 records staff achievements           15 increases staff interest in the job           16 enhances job satisfaction           17 provides positive feedback           18 holds annual performance appraisals of staff           19 provides incentives to staff from time to time             collaboration           20 my library has communities of practice or research groups           21 mentoring of staff is encouraged           22 encourages staff to help one another           23 supports top-down communication only           24 supports bottom –up communication only           25 supports both top-down and bottomup communication             other (specify)               review article   does the read scale work for chat? a review of the literature   adrienne warner assistant professor, learning services librarian university of new mexico albuquerque, new mexico, united states of america email: adriennew@unm.edu   david a. hurley discovery and web librarian university of new mexico albuquerque, new mexico, united states of america email: dah@unm.edu   received: 29 mar. 2021                                                                  accepted: 13 may 2021       2021 warner and hurley. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   doi: 10.18438/eblip29947     abstract   objective – this review aims to determine the suitability of the read scale for chat service assessment. we investigated how librarians rate chats and their interpretations of the results, and compared these findings to the original purpose of the scale.   methods – we performed a systematic search of databases in order to retrieve sources, applied inclusion and exclusion criteria, and read the remaining articles. we synthesized common themes that emerged into a discussion of the use of the read scale to assess chat service. additionally, we compiled read scale designations across institutions to allow side-by-side comparisons of ratings of chat interactions.   results – this review revealed that librarians used a variety of approaches in applying and understanding read scale ratings. determination of staffing levels was often the primary goal. further, librarians consistently rated chat interactions in the lower two-thirds of the scale, which has implications for service perception and recommendations.   conclusion – the findings of this review indicated that librarians frequently use read scale data to make staffing recommendations, both in terms of numbers of staff providing chat service and level of experience to adequately meet service demand. evidence suggested, however, that characteristics of the scale itself may lead to a distorted understanding of chat service, skewing designations to the lower end of the scale, and undervaluing the service.     introduction   researchers have been strategizing for decades about how best to capture what happens in reference interactions. the reference effort assessment data (read) scale (gerlich & berard, 2007) was one of several tools developed in response to librarians’ “deep dissatisfaction” (novotny, 2002, p. 10) with the reference statistics being collected in the early 2000s. the then-common practice was to record simple counts, often as hash marks on paper, for each mode (e.g., desk or telephone) in one of two categories: “directional” or “reference”. the read scale, by contrast, is a six-point scale indicating the amount of effort a librarian expends on each reference interaction. answers are rated 1 if they require no specialized knowledge or consultation of resources. a rating of 6 indicates “staff may be providing in-depth research and services for specific needs of the clients” (gerlich, n.d.). in introducing the read scale, gerlich and berard (2007) emphasized that their goal was to change the focus from “how many” and “what kind” to the knowledge, skills, and abilities needed to provide the service. they suggested this data could be used for “a retooling of staffing strategies,” to “increase positive self-awareness of the professional librarian” (2007, p. 9) and “for training and continuing education, renewed personal and professional interest, and reports to administration" (gerlich & whatley, 2009, p. 30). while other classification systems exist (see maloney & kemp, 2015, for examples), the read scale has become a standard in many libraries—a state that is reflected in its integration into both open source and commercial reference management products (e.g., sarah, 2012).   gerlich and berard (2010) undertook a large-scale, multi-institution viability study in 2007. however, the data reflected the predominantly face-to-face nature of reference at that time. with 15 institutions reporting 8,439 transactions, 91% were face-to-face. chat transactions accounted for only 1% of the interactions in the study. the landscape has shifted dramatically since then. chat is now a major source of reference transactions in academic libraries (asher, 2014; belanger et al., 2016; nicol & crook, 2013; ward & phetteplace, 2012), accounting for more than 20 percent of all reference transactions at the university of new mexico in 2019, with in-person reference transactions down to just 55 percent. in 2020, the majority of reference transactions were conducted via chat, likely due to covid-19 related building closures.   the software that mediates chat reference interactions typically generates rich metadata, from timestamps to transcripts, allowing, and likely encouraging, innumerable assessment strategies. indeed, chat reference evaluation has been the focus of hundreds of research articles since its first implementation in academic libraries in north america in the mid 1990s (matteson et al., 2011), with metrics including number of chat interactions in a given timeframe, number of missed chats, frequency and length of interaction, turns taken between librarian and user, word counts, and type of referring url (luo, 2008). these automatically generated metrics are often combined with more qualitative measures such as types of questions asked; the presence of reference interviews and instructional elements; and quality, completeness, and tone of librarians’ answers (luo, 2008).   some chat-specific, or at least virtual reference specific, assessment tools have been developed, both for services overall (hirko, 2006; white, 2001), and for transcript analysis in particular (mungin, 2017). however, transcript analysis is both time consuming (mungin, 2017) and limited (belanger et al., 2016; rabinowitz, 2021). much of the literature on chat reference has been case studies with limited generalizability, with mclaughlin (2011) suggesting the need for standard approaches and reporting formats across libraries.   there has been recognition generally that standards and assessments are not necessarily transferable from face-to-face to online reference (ronan et al., 2003), and some researchers have attempted to determine applicability across modes (schwartz & trott, 2014). however, though the read scale is widely used with chat reference, and is recommended as a metric "applicable across reference services" in the reference and user services association's guidelines for implementing and maintaining virtual reference services (2017), no equivalent to the viability study has been undertaken for chat. library services have changed considerably in the nearly two decades since the read scale was developed, and it cannot be assumed that a tool developed for in-person contexts will then be appropriate for chat reference in 2021. in that sense, there is no evidence that the read scale is appropriate for chat. as a first step in investigating the viability of the read scale for chat, we review the literature on chat reference that uses the read scale.   aims   there are several ways in which a literature review can help us determine the suitability of the read scale for chat. first, we gain insight into how and why librarians are using the read scale; that is, we want to see what librarians using the read scale for chat reference are trying to understand and what decisions they are trying to make with read scale data. we are interested if these are the same uses that gerlich and berard (2007) anticipated, and if the nature of read scale data is appropriate for these uses. second, we can examine the ratings that are reported for chat reference across institutions, much as gerlich and berard did for all reference transactions. here we are interested in the ratings themselves, including what patterns are evident, but also in any modifications being done to the read scale, as well as how the data are interpreted and reported in the literature. we use “librarians” to refer to all library workers who provide or analyze chat service. we use “chat agents” to describe the workers who field chat in situ, distinguished from those who later analyze trends.   thus, we have two broad questions to guide our review:   ●        how do librarians use the read scale to assess chat? ●        how do librarians rate chats on the read scale?   taken together, these two strands present a picture of the read scale in practice, and whether or not it is an appropriate tool for the assessment of chat reference.   method   we took a systematic approach to collecting pertinent professional literature using a mixed methods review synthesis (heyveart et al., 2016). first, we explicitly defined inclusion and exclusion criteria for the literature we would review. then we developed a search strategy to locate literature that would meet our inclusion criteria. next, we read the literature to identify themes and patterns, and iteratively reread and assigned categories until we reached consensus on both the categories that were present in the literature, and the specific categories present in each document.   inclusion and exclusion criteria   we defined “professional literature” to include articles, white papers, book chapters, and conference proceedings and presentations. for clarity, we use the term “article” to refer to all document types. to be included, articles must have directly discussed the read scale as applied to chat reference, other than reporting data from a different source, such as in the literature review section of an article. exclusion criteria were review articles that did not present otherwise unpublished data and articles in which read scale data and discussion of chat cannot be distinguished from other modes of reference. we aimed for a global scope and so explicitly did not exclude material based on publication language or library type.                 search strategy   we searched library & information sciences abstracts (lisa), library, information science & technology abstracts (lista), proquest dissertations & theses global full text global, web of science core collection, google scholar, ebsco discovery service (eds), and the e-lis repository. though platforms vary in the exact construction of searches, our search had two concepts: the read scale and chat reference. while we could be confident that the presence of the phrase “reference effort assessment data” referred to the scale developed for library reference services, not all relevant documents spelled out the acronym. further complicating matters, the phrase “read scale” was used in ways that are not related to the read scale. therefore, our search required either the use of the full name of the scale, or the acronym and a variation on the word library and the word reference. the second concept was chat reference, which we defined to include any synchronous online reference service, such as chat or instant messaging. because e-lis, which does not index full-text, had only one result for “read scale” and none for “reference effort assessment data”, we did not include that concept in the query. instead, our query looked only for the concept of chat reference, and we manually checked each “reference work” subject result for the read scale concept.   table 1 database search strategies database database note search edsa, lisa, lista, proquest dissertations and theses full text global, web of science core collection (chat* or trigger* or "instant messag*") and (("read scale" and librar*) or “reference effort assessment data”) google scholar google allows neither truncation nor parentheses. "reference effort assessment data" or "read scale" and library or libraries or librarians and reference and chat or "instant messaging" or "instant message" e-lis e-lis’s search interface consists of a series of fields and search operators. the keywords field allows text entry while subjects is a drop down menu. field: keywords any of: chat messaging messenger field: subjects any of: ....ij. reference work. a every eds configuration is unique. while the total number of databases at unm is in the hundreds, the databases within our eds that produced hits on our query were: academic search complete, applied science & technology source, british library document supply centre inside serials & conference proceedings, business source complete, complementary index, directory of open access journals, education research complete, eric, escholarship, gale academic onefile, gale onefile: computer science, library, information science & technology abstracts, sciencedirect, social sciences citation index, and supplemental index.   themes and patterns   we read each article and noted patterns and themes related to our guiding questions, including descriptions of how read was used, the read score data itself, and any resultant service outcomes. we were interested in similarities across institutions as well as differences in implementation or interpretation.   results   our search strategy yielded 141 unique items. after applying the inclusion and exclusion criteria to the search results, we had a total of 18 articles that we included in the review. all were from academic institutions, of which two were outside the united states. the data, patterns, and themes found in those articles are presented below.                 rating comparability   gerlich and berard (2010), as part of testing its viability across many institutions, normed   read scale designations with coordinators at each institution who then normed the scale with their local reference agents. larson et al. (2014) highlighted consistent application of the scale as an issue when a chat service employed both local and consortial chat agents. however, the rest of the articles did not have any cross-institution norming, though five (in addition to gerlich and berard), performed some form of inter-rater norming for their own data (belanger et al., 2016; kemp et al., 2015; keyes & dworak, 2017; maloney & kemp, 2015; stieve & wallace, 2018). in the remainder, no mention was made of any norming of read scale ratings, other than warner et al. (2020) who explicitly stated that interrater reliability was not tested as part of the study.   rating process   also potentially limiting the comparability across institutions is who rated the chats. in eight articles, the ratings were applied post-hoc by the researchers, in three cases, the reference agents rated their own interactions, while in six articles, it is not stated who rated the chats. kohler (2017) rated chats algorithmically, and compared the ratings to those done by the chat agents, concluding that the algorithm was as good or better at rating the effort required by the chat agents than the agents themselves.   rating questions or answers   asher (2014), cabaniss (2015), maloney and kemp (2015), ward and phetteplace (2012), and ward and jacoby (2018) used the read scale to rate incoming questions, while others looked at the outbound responses of the librarians: the university of turin used the ratings to “uniformly categorize the type of responses our patrons received” (bungaro et al., 2017, p. 4). kohler (2017) reported rockhurst university used the read scale to understand “the effort, skills, knowledge, teaching, techniques, and tools” used by librarians (p. 138). keyes and dworak (2017), mavodza (2019), stieve and wallace (2018), valentine and moss (2017), and warner et al. (2019) did not address whether they were assessing the complexity of either questions or answers.   local adaptations   five institutions represented in the literature adjusted the read scale. kohler (2017) added a 0 rating to algorithmically rated transactions that the algorithm was not able to rate. while these were largely staff demonstrations, they were included in the analysis. belanger et al. (2016) dropped level 6 of the scale before assessment started, though no rationale was given. keyes and dworak (2017) shifted the scale to 0-5, without explanation. no articles mentioned adjustments to extend the scale beyond level 6.   kayongo and van jacob (2011) added 26 sub-categories to the read scale, e.g. within level 3, there are categories such as “l3 complex known item search [do we own?],” “l3 online resource problem,” and “l3 simple citation verification”. stieve and wallace (2018) added the word “circulation” as a bullet point in the definition of level 2 and expanded several of the examples that accompany the definitions.   local interpretations   specific read scale ratings were not always interpreted to mean the same level of expertise. belanger et al. (2016) defined points 3 and above as “requiring a complex response” (p. 12). bungaro et al. (2017) only reported the split between the highest 3 categories (i.e., 4, 5, and 6), for which a subject specialist was justified and the lowest 3, for which a 'generic' librarian was sufficient. kemp et al. (2015), comparing complexity of questions across modes, considered questions “complex” at ratings 3 and above and “basic” at 2 and below. they made a further distinction that a score of 4 or above required librarian or subject librarian expertise. using a 0-5 scale, keyes and dworak (2017) noted that 0-2 were considered to be “clearly” able to be addressed by graduate students. in presenting the data, they also grouped 4 and 5, but without explanation. maloney and kemp (2015) grouped 4 and above as requiring advanced expertise.   while mavodza (2019) reported counts for each point on the scale, she presented the groupings of the counts at 2 and lower, writing “[w]hen analyzing the same chats from the read level, most of them were at the one and two difficulty levels at 939 and 230, respectively” (p. 128).  even though level 3 had only 40 occurrences fewer than level 2, she grouped the level 3 count (n=190) with level 4 (n=98). she grouped the level 5 (n=26) and 6 (n=15) together. these groupings suggest both affinities of content within the groups as well as an intellectual divide between the groups.   data points used in conjunction with read scale   many times, the read scale ratings were used in conjunction with other data points, time measures being most common. belanger et al. (2016), gerlich and berard (2010), and maloney and kemp (2015) each analyzed the read scale ratings by point in the academic semester or quarter in order to understand the relationship between the academic calendar and the complexity of reference transactions. cabaniss (2015), kayongo and van jacob (2011), and ward and jacoby (2018) compared ratings across times of day in order to understand busyness patterns by hour. cabaniss (2015) also looked at days of the week in order to find the busiest and least busy days.                 the second most-frequent variable used with the read scale was the topic of the interaction. bungaro et al. (2017) were interested in the frequency of psychology-related chats and whether they were, on average, more complex than others. cabaniss (2015) and mavodza (2019), following the same research protocol, assigned four categories to chats: general information, technical, known item lookup, and reference. kohler’s (2017) algorithm weighed certain words more than others in order to algorithmically assign a read scale rating. ward and jacoby (2018), studying referrals given in chat, compared the read scale to their own categories of referral needed, referral provided, appropriate referral, and referral gap. they found that referrals happened more often as the complexity of the question increased, and also found that the referral gap, or rate at which a referral was warranted but not given, also went up with the complexity of the interaction as designated by the read scale.   other metrics used in conjunction with the read scale to determine service patterns included staffing types (keyes & dworak, 2017), delivery mode (asher, 2014; gerlich & berard, 2010; maloney & kemp, 2015; ward & jacoby, 2018), length of interaction, and referral type (ward & jacoby 2018).   goals of read scale assessment   in some cases, such as in larson et al. (2014), the read scale chat data were reported as part of an overall assessment of reference services, for which there was no particular insight or assessment-related decision that the authors were using the read scale to understand. in one case, the read scale ratings were used as a cut off for including transactions in a different assessment effort (valentine & moss, 2017). gerlich and berard (2010) and belanger et al. (2016) were interested in assessing the read scale itself, the latter concluding that the read scale did not provide a sufficiently nuanced understanding of the complexities of chat service provision.   several uses for the read scale were found in multiple articles, and these are summarized below.   staffing needs   eleven articles used the read scale to determine appropriate staff allocation, with the most common approach being to match read scale ratings with the level of expertise required of the reference staff. often the six-point scale was reduced, in the data analysis, to two broad categories: ratings that indicate a need for professional expertise, and those that do not. this sort of grouping is implied by the logo for the read scale, which depicts lines separating 5 and 6 from the lower numbers on the scale. this division is not explicit in the scale’s definition, however, and as discussed above, institutions located this split at varying points on the scale.   researchers found evidence to support a variety of staffing recommendations, both in terms of number of staff assigned to providing the service and the level of experience of staff providing the chat service. kemp et al. (2015) and maloney and kemp (2015), studying the implementation of a proactive chat system at the university of texas-san antonio library, found an increase in complex questions through the proactive system. because they defined read scale 4 and above as needing librarian-level attention, an increase in librarians was needed to field the more frequent, complex questions, and they subsequently used a multi-pronged approach to get more librarians into the chat staffing mix (kemp et al., 2015). similarly, bungaro et al. (2017) found evidence to maintain and keep subject librarians answering a subject-specific chat channel, whereas they assumed generalist librarians could adequately answer read scale 1-3. further evidence was found to keep professional librarians answering chat, arguing that outsourcing late night chat service to consortial librarians resulted in decreased quality of service (kayongo & van jacob, 2011).   while some researchers found evidence for staffing increases, ward and jacoby (2012) found that a rearrangement of staff to match hour-by-hour patterns in chat may optimize which staff are most likely to receive the most complex questions. however, the researchers ultimately recommended a reduction in the experience level of chat staff, shifting to graduate students exclusively. in another study, keyes and dworak (2017) found evidence to recommend a reduction in professional level/experience. keyes and dworak noted that undergraduate students could adequately field chat at the read scale 1-3 level.   while many supported point-of-need staffing with experienced or professional librarians, some suggest the tiered model was adequate to address complex questions that could not be answered at the time they were asked. in their study examining the rates of referrals and whether they occurred as needed, ward and phetteplace (2012) found that while the rates of referral went up with the read scale designation, so did the rate of interactions that should have included a referral but did not. the referral ecosystem was further complicated with the read scale when staff members sometimes referred questions they rated as complex, but gave little effort to answer on the spot (belanger et al., 2016).   comparison of reference configurations   researchers in five articles used the read ratings to compare different types of reference service. four of the five compared passive and proactive chat configurations (demars et al., 2018; kemp, et al., 2015; maloney & kemp, 2015; warner et al., 2020). each of these researchers found that proactive chat systems resulted in more complex transactions than passive chat configurations. stieve and wallace (2018) compared sources of chat transactions, finding that read scale ratings were higher from within the university’s learning management system as compared to chats originating from the library website.   gerlich and berard (2010) compared all modes of reference at the participating institutions: walk-up directional, walk-up reference, phone directional, phone reference, email, and chat. the difference between “directional” and “reference” phone and walk-up modes was not explained. chat transactions made up only about 1% of the transactions in the study. a greater emphasis was on distinguishing between “walk up” in-person desk interactions and those in-person interactions that happened in hallways and offices: “the off-desk comparisons show… that the percentage of questions answered off-desk for most of the institutions require a much higher level of effort, knowledge, and skills from reference personnel than at the public service point” (gerlich & berard, 2010, p. 125)   other   several researchers noted the higher degree of difficulty in fielding chat rather than face-to-face interactions, difficulties that were not represented in the read scale. chat agents may have to field multiple chats at the same time (cabaniss, 2015; demars, 2018; keyes & dworak, 2017), and the necessity of typing succinct directions without the ability to rely on non-verbal cues was challenging (gerlich & berard, 2010). librarians at one institution reported that multiple factors coalesced into chat interactions being deemed as more stressful (ward & phetteplace, 2012).   the ratings   while the read scale is a 6-point scale, the 1-4 range of the scale was heavily used, while the 5-6 range was not. belanger et al. (2016), gerlich and berard, (2010), kohler (2017), stieve and wallace (2018), and ward and jacoby (2018) found that zero chats were rated a 5 or 6. kayongo and van jacob (2011), cabaniss (2015), and mavodza (2019) found that the majority of chat interactions fell within the 1-3 range. the one outlier was bungaro et al. (2017), who found that almost half (46%) of their chat interactions happened at the 4 or above designation, though it is unclear how many were rated at each level.   not every source provided a breakdown by each number, with some providing percentages of ranges of read scale designations. however, of those that did, the data showed the vast majority of transactions occurring in the 1-4 range. within the read scale 1-4 range, most chats are rated a read scale 2 or 3 (see figure 1).   table 2 summary of articles, including read scale breakdowns author, publication date, (institution examineda) chat sample size breakdown by read scale designation as reported by sourceb timeframe of chats assessed with read scale sampling method asher, 2014 (university of indiana, bloomington) 149 unstated 2006-2013 convenience belanger et al., 2016 (university of washington) 3721 chat transcripts 1=10% 2&3= 80% 4=10% 5=0 6=n/a   fall quarter 2014 (september-december) convenience bungaro et al., 2017 (university of turin; italy) 121 1-3= 53.8% (n=65) 4-6= 46% (n=56) 2014-2016 convenience cabaniss, 2015 (university of washington) 608 1=28% (n=169) 2=45% (n=275) 3=23%(n=142) 4=4%(n=27) 5=0% (n=2) 6=unstated 3 sample weeks in winter term 2014 (january-march) convenience demars et al., 2018 (california state university, long beach) unstated 1=7-14% 2=32-32% 3=38-40% 4=7-15% 5=1-3% 6=.5-1% gleaned from table 2016-2018. compared three chat configurations. cluster gerlich and berard, 2010 (multiple) test 1: 98 test 2: 317 test 1: 1=13% (n=13) 2=19% (n=19) 3= 45% (n=44) 4=22% (n=22) 5=0% (n=0) 6=0% (n=0)   test 2: 1=6% (n=19) 2=24% (n=76) 3=47% (n=150) 4=21% (n=66) 5=2% (n=6) 6=0% (n=0) test 1: 3 weeks in february, 2007 test 2: spring semester, 2007 test 1: convenience, test 2: adaptive kayongo and van jacob, 2011 (university of notre dame) 2517 1=21.1% (n=531) 2=28.3% (n=712) 3=40.4% (n=1017) 4=5.9% (n=1480 5=4.2% (n=105) 6=.1% (n=4) november 2007-may 2010 convenience kemp et al., 2015 (university of texas, san antonio)   test 1: unspecified test 2: 287 test 3: 228 test 1: 3= 44% 4 and above= 21%   test 2: (triggered) 1,2 =19% 3 and above= 81%   test 3: (non-triggered) 1,2=37% 3 and above= 63% test 1: six sample weeks from fall 2013 to spring 2014. tests 2 and 3: november 2013 test 1: convenience tests 2 and 3: cluster keyes and dworak, 2017 (boise state university) 454 0-1= 19% (n=78) 2= 31% (n=131) 3= 39% (n=164) 4-5= 11% (n=48) may 2014-sept 2016 convenience kohler, 2017 (rockhurst university) 1109 0=7% (n=80) 1=15% (n=166) 2=33% (n=366) 3=41% (n=456) 4=3% (n=35) 5=1% (n=6) 6=0% (n=0) fy2015fy2016 convenience larson et al., 2014 (university of maryland) 39 1=5% (n=2) 2=49% (n=19) 3=21% (n=8) 4=23% (n=9) 5=3% (n=1) 6=0% (n=0) 2013 convenience maloney and kemp, 2015 (university of texas, san antonio) 2492 1=3% (n=85) 2=31% (n=764) 3=39% (n=968) 4=26% (n=654) 5=1% (n=19) 6=0% (n=2) 6 sample weeks: fall 2013 and spring 2014 cluster mavodza, 2019 (zayed university; united arab emirates) 1498 1=63% (n=939) 2=15% (n=230) 3=13% (n=190) 4=7% (n=98) 5=2% (n=26) 6=1% (n=15) feb 2013-feb 2018 convenience stieve and wallace, 2018 (university of arizona) 382 1=0% (n=2) 2=34% (n=128) 3=48% (n=184) 4=9% (n=33) 5=3% (n=13) 6=0% (n=0) fall and spring semesters for 2014 and 2015. cluster valentine and moss, 2017 (university of kansas) 30 3 and above=100% (n=30) fall semester 2016 cluster ward and jacoby, 2018 (university of illinois, urbana-champaign)   1120   1=30 (3%) 2=378 (34%) 3=636 (57%) 4=76 (7%) 5 and 6=unstated april 2015 convenience ward and phetteplace, 2012 (university of illinois. urbana-champaign) unspecified unspecified   sept and oct 2010 convenience warner et al., 2020 (university of new mexico) 4617 unspecified july 2016 to july 2018 cluster a  country of institution is united states of america unless otherwise indicated. b when the source gives numbers only, review authors have converted these to percentages and rounded to the whole number, but we have not used percentage-only data to derive hard numbers. because of this, percentages are recorded in the 100-101 range.   figure 1 read scale ratings for chat interactions from articles that provided breakdowns by each read scale number. gerlich and berard (2010) reported two tests in “testing the viability of the read scale.”   discussion   perhaps unsurprisingly, a primary goal in using the read scale at many institutions was to understand staffing needs. many analyzed read scale designations by time of day, day of week, and time in the academic period in order to find patterns of busiest times and most complex questions, following assessment trends luo identified in 2008. understanding demand for service is an admirable goal, but unfortunately even when service patterns are present, staffing models may not provide the flexibility to checkerboard staff shifts to match them. and, while optimizing staffing may be possible to a degree, in practice, complex questions can arrive at any point in the day, week, or year and service coordinators must determine what is adequate, rather than optimal, service. additionally, read scores were used to make inferences about the level of expertise needed, most often a recommendation for decreased expertise, and none of the sources articulated how they would determine when they had sub-adequate staffing levels on chat. this use of the read scale, while seemingly common, is troubling.   the most striking feature of the read scale data was the paucity of chat transactions rated 5 or 6. indeed, this was reflected in gerlich and berard’s (2010) viability study data. none of the 15 testing institutions reported any chat transactions falling into the 5 or 6 ratings. however, it would be premature to conclude that patrons are not asking complex questions, based solely on ratings of the responses to those questions. as gerlich and berard developed the read scale, they theorized that more complex questions were answered away from the reference desk, such as in hallways or offices, and they built this assumption into the scale. for example, the definitions provided for the 5 and 6 ratings assume or presuppose the interaction occurring outside of initial reference contexts. level 5 suggests that “consultation appointments might be scheduled” (gerlich, n.d.) while level 6 specifies that “(r)equests for information cannot be answered on the spot” (gerlich, n.d.). in practice, this makes the higher end of the scale unavailable to chat transactions. as of this writing, it is not feasible for a librarian to initiate a chat with a specific patron or schedule a consultation via chat, so any question that requires follow up is necessarily answered in a different mode. the scale gives weight to the mode in which the librarian answers reference questions, rather than the mode in which the patron asks it.   depending on how the read scale ratings are used by an institution, the data could present a distorted view of the chat service. for example, if a question comes in via chat that is beyond the abilities of that librarian, they might create a ticket that the relevant subject specialist will claim. while the eventual answer would likely be a 5 or 6, the chat transaction would rate as a 1, as it took no effort. if the staff member used knowledge or training about who specifically to refer the question to, or instructed the patron in how to schedule a consultation, it might appropriately rate at level 2 or 3. in any case, it would be at the low end of the scale, indistinguishable from simple item searches or questions about hours. using read scale data to determine the level of expertise required of chat staff is problematic when questions that staff could answer comfortably and those they lacked the expertise to answer at all are represented the same way in the data.   it is also notable that almost every study in this review tabulated read scale designations based simply on frequency, either as raw counts or percentages. this gives equal weight to all questions; imagine a one hour reference shift in which a librarian has two questions at level 4, spending 15 minutes on each working with the patrons to choose a database, develop and refine search strategies, and so on. that librarian also spends ten seconds on each answer to questions about the hours. if there are five such questions, the hour was predominantly level 1, even though less than a minute was spent on level 1 answers, and half the hour was spent on level 4 responses. this approach essentially recreates the hash mark system that the read scale was developed to replace, with the added misinterpretation that most questions can be answered without any specialized knowledge or skill.    this unequal weight distribution is true for all modes of reference, but is especially significant for chat, where librarians report fielding multiple questions at a time. answering several questions about hours or renewals while a patron requiring more complex assistance is running searches is efficient. an approach to assessment that devalues this practice is flawed.   measuring the skills needed to answer a question makes sense when trying to assess how to staff the service. it can be less helpful when trying to assess the value of the service to our patrons. helping a patron navigate a confusing record or misconfigured link in a database may be a simple question from the librarian’s perspective, perhaps rating a 2 on the read scale. from the patron’s perspective, getting access to the resource may be critically important. a low read score might be desirable in this case: the librarian was able to solve the patron’s problem quickly and easily. yet, the interpretation of the read scale may rank the worthiness of some questions or answers over others, a common trap inherent in scales (lemire et al., 2016).   given the language limiting chat to the lower end of the read scale, we might see a rash of adaptations of the definitions and examples, creating the ability of chat to enter into the final third of the read scale. however, of the few institutions that adapted the read scale verbiage to their local assessment environments, mode-agnostic wording was not added at the upper levels, and mode-specific language was not deleted. because the majority of institutions did not alter the language of the scale at all, it may be that critical analysis of the viability of the tool explicitly for chat was absent or that librarians agreed with the existing wording of the scale. if librarians adhere to the existing wording of the scale, they implicitly agree that regardless of time invested or librarian resources used, chat content will never equate to the same content delivered face-to-face in a traditional consultation. if librarians assess chat in direct comparison to face-to-face, or indeed other modes of delivery, the assessment reflects bias by creating an implicit hierarchy of service.   while the read scale language may constrict chat transactions to the lower two-thirds of the scale, experience with complex content may have a similar effect. less experienced chat agents do not have as deep a skillset with which to call upon in chat interactions, leaving their choice of read scale rating constricted. so, if an institution staffs chat with less experienced undergraduate students, the read scale ratings should reflect that restricted ability, resulting in lower-rated chats. in contrast, highly experienced librarians have the ability to answer inquiries farther up the read scale. further, the judgement of complexity is compounded by more experienced librarians’ increased familiarity and expertise: librarians who walk students through the intricacies of searching databases may be more familiar with the content and thus rank those interactions as less complex.   limitations   the findings of this literature review are limited in several ways. because we were attempting to understand the variety of situations in which the read scale has been deployed to measure chat, we did not limit the scope of material to academic journal articles. the resultant formats provide variable depth of information. for example, only half of the articles included provided the breakdown of read scale ratings by each of its six levels, while the others reported only clusters of ratings. some provided in-depth explanations of read scale ratings results and extensive analysis, and others did not. for some of the articles, chat reference was a brief focus of a comprehensive reference service evaluation. more broadly, many institutions used the read scale to assess reference interactions, as evidenced by our initial searches, however it is unclear exactly how many institutions used the read scale to assess chat. evidence from community colleges and additional international sources could inform future research and help to address this knowledge gap. finally, we recognize that not all chat service coordinators publish their assessment efforts, so direct research with this population may provide insight into read scale reach, variations in application, and changes in the quantity and experience levels of chat agents after using it for assessment.   conclusion   pomerantz et al. (2008) argued that when assessing chat reference, “’good enough’ data is better than no data when one is aware of the limitations of the data” (p. 27). the read scale has been adopted across the globe to assess online synchronous interactions, yet this tool was developed before chat reference became commonplace (matteson et al., 2011). in this literature review, we set out to understand how librarians use the tool to make decisions and understand resultant data when examining chat reference service. this was a first step to understanding whether the read scale has withstood the test of time as chat has evolved to play a much bigger role in reference service.   based on the data that have been published and reviewed herein, the read scale systematically undervalues chat reference transactions for many of the assessment goals for which it is used. ultimately, the way the read scale is applied and interpreted with regards to chat reference is at odds with the intent of the tool. rather than capturing the knowledge, skills, and abilities needed to provide adequate reference service, it overemphasizes the simplest, least time consuming questions, with the effect of making chat reference appear not worth the cost of staffing with experienced professionals.   we are not providing recommendations here about the correct level of staffing for chat reference. our concern is that the assessment that libraries do to make those decisions are appropriate and accurate. we therefore propose a number of recommendations for practice.   recommendations for practice   based on our understanding of how the read scale has been used by librarians to conflate complexity, experience, and value, we invite practitioners to reflect on the following recommendations:   1.       use caution when comparing face-to-face, chat, email, text, and phone interactions to each other using the existing definitions and examples provided in the read scale documentation. 2.       examine and update the definitions and examples of each level of the read scale through the lens of each reference delivery method your library provides. 3.       if you are trying to understand the expertise needed to staff the service, consider rating every question that staff could not answer as a 6. these are questions that cannot be answered “on the spot” with your current staffing model. 4.       if you are trying to understand the service overall, consider reporting the time spent on each level of answer, rather than the number or percent of questions at each level. alternately, you could develop a system of weights for each read scale number in order to represent expertise needed for the length of the interaction.   author contribution statement   adrienne warner: conceptualization, data curation, formal analysis, investigation, methodology, supervision, visualization, writing – original draft, writing – review and editing david a. hurley: conceptualization, data curation, formal analysis, investigation, writing – original draft, writing – review and editing   references   asher, a. (2014). who’s asking what? modelling a large reference interaction dataset. proceedings of the 2014 library assessment conference, 52-62. https://www.libraryassessment.org/past-conferences/2014-library-assessment-conference   belanger, j., collins, k., deutschler, a., greer, r., huling, n., maxwell, c., papadopoulou, e., ray, l. & roemer, r. c. (2016). university of washington libraries chat reference transcript assessment. https://www.lib.washington.edu/assessment/projects/chat-reference-assessment-project-2014-2016-final-report   bungaro, f., muzzupapa, m. v., & tomatis, m. s. (2017). extending the live chat reference service at the university of turin a case study. iatul annual conference proceedings, 1–12. https://docs.lib.purdue.edu/iatul/2017/challenges/1/     cabaniss, j. (2015). an assessment of the university of washington’s chat reference services. public library quarterly, 34(1), 85–96. https://doi.org/10.1080/01616846.2015.1000785   demars, m. m., martinez, g., aubele, j., & gardner, g. (2018, december 7). in your face: our experience with proactive chat reference [conference presentation]. carldig-s fall 2018 program, los angeles, ca, united states. http://eprints.rclis.org/33775/      gerlich, b. k. (n.d.). read scale bulleted format. the read scale©: reference effort assessment data. http://readscale.org/read-scale.html   gerlich, b. k., & berard, g. l. (2007). introducing the read scale: qualitative statistics for academic reference services. georgia library quarterly, 43(4), 7-13. https://digitalcommons.kennesaw.edu/glq/vol43/iss4/4   gerlich, b. k., & berard, g. l. (2010). testing the viability of the read scale (reference effort assessment data)©: qualitative statistics for academic reference services. college & research libraries, 71(2), 116-137. https://doi.org/10.5860/0710116   gerlich, b. k., & whatley, e. (2009). using the read scale for staffing strategies: the georgia college and state university experience. library leadership & management, 23(1), 26-30. https://journals.tdl.org/llm/index.php/llm/article/download/1755/1035   heyvaert, m., hannes, k., & onghena, p. (2016). using mixed methods research synthesis for literature reviews: the mixed methods research synthesis approach (vol. 4). sage publications.   hirko, b. (2006). wally: librarians' index to the internet for the state of washington. internet reference services quarterly, 11(1), 67-85. https://doi.org/10.1300/j136v11n01_06   kayongo, j., & van jacob, e. (2011). burning the midnight oil: librarians, students, and late-night chat reference at the university of notre dame. internet reference services quarterly, 16(3), 99–109. https://doi.org/10.1080/10875301.2011.597632   kemp, j. h., ellis, c. l., & maloney, k. (2015). standing by to help: transforming online reference with a proactive chat system. the journal of academic librarianship, 41(6), 764–770. https://doi.org/10.1016/j.acalib.2015.08.018   keyes, k., & dworak, e. (2017). staffing chat reference with undergraduate student assistants at an academic library: a standards-based assessment. the journal of academic librarianship, 43(6), 469–478. https://doi.org/10.1016/j.acalib.2017.09.001   kohler, e. (2017, november 3). what do your library chats say?: how to analyze webchat transcripts for sentiment and topic extraction. brick & click libraries conference proceedings, 138–148. https://files.eric.ed.gov/fulltext/ed578189.pdf larson, e., markowitz, j., soergel, e., tchangalova, n., & thomson, h. (2014). virtual information services task force report. university of maryland university libraries. https://doi.org/10.13016/m2gt5fh7n   lemire, s., rutledge, l., & brunvand, a. (2016). taking a fresh look: reviewing and classifying reference statistics for data-driven decision making. reference & user services quarterly, 55(3), 230-238.   luo, l. (2008). chat reference evaluation: a framework of perspectives and measures. reference services review, 36(1), 71–85. https://doi.org/10.1108/00907320810852041   maloney, k., & kemp, j. h. (2015). changes in reference question complexity following the implementation of a proactive chat system: implications for practice. college & research libraries, 76(7), 959–974. https://doi.org/10.5860/crl.76.7.959   matteson, m. l., salamon, j., & brewster, l. (2011). a systematic review of research on live chat service. reference & user services quarterly, 51(2), 172-190. https://doi.org/10.5860/rusq.51n2.172   mclaughlin, j. e. (2011). reference transaction assessment: survey of a multiple perspectives approach, 2001 to 2010. reference services review, 39(4), 536-550. https://doi.org/10.1108/00907321111186631   mavodza, j. (2019). interpreting library chat reference service transactions. reference librarian, 60(2), 122–133. https://doi.org/10.1080/02763877.2019.1572571   mungin, m. (2017). stats don't tell the whole story: using qualitative data analysis of chat reference transcripts to assess and improve services. journal of library & information services in distance learning, 11(1-2), 25-36. https://doi.org/10.1080/1533290x.2016.1223965   nicol, e. c., & crook, l. (2013). now it's necessary: virtual reference services at washington state university, pullman. the journal of academic librarianship, 39(2), 161-168. https://doi.org/10.1016/j.acalib.2012.09.017   novotny, e. (2002, september). reference service statistics & assessment. association of research libraries. https://www.arl.org/resources/spec-kit-268-reference-service-statistics-a-assessment-september-2002/   pomerantz, j., mon, l. m., & mcclure, c. r. (2008). evaluating remote reference service: a practical guide to problems and solutions. portal: libraries and the academy, 8(1), 15-30. https://doi.org/10.1353/pla.2008.0001   rabinowitz, c. (2021). you keep using that word: slaying the dragon of reference desk statistics. college & research libraries news, 82(5), 223-224. https://doi.org/10.5860/crln.82.5.223   reference and user services association. (2017). guidelines for implementing and maintaining virtual reference services. http://www.ala.org/rusa/sites/ala.org.rusa/files/content/resources/guidelines/guidelinesvirtualreference_2017.pdf   ronan, j., reakes, p., & cornwell, g. (2003). evaluating online real-time reference in an academic library: obstacles and recommendations. the reference librarian, 38(79-80), 225-240. https://doi.org/10.1300/j120v38n79_15   sarah. (2012, march 21). libanswersnew features are live! springshare. https://blog.springshare.com/2012/03/21/libanswers-new-features-are-live/   schwartz, h. r., & trott, b. (2014). the application of rusa standards to the virtual reference interview. reference and user services quarterly, 54(1), 8-11. https://journals.ala.org/index.php/rusq/article/view/3993/4505   stieve, t., & wallace, n. (2018). chatting while you work. reference services review, 46(4), 587–599. https://doi.org/10.1108/rsr-09-2017-0033   valentine, g., & moss, b. d. (2017, march 28). assessing reference service quality: a chat transcript analysis [conference presentation]. association of college and research libraries conference, baltimore, md.  http://hdl.handle.net/1808/25179   ward, d., & jacoby, j. (2018). a rubric and methodology for benchmarking referral goals. reference services review, 46(1), 110-127. https://doi.org/10.1108/rsr-04-2017-0011   ward, d., & phetteplace, e. (2012). staffing by design: a methodology for staffing reference. public services quarterly, 8(3), 193–207. https://doi.org/10.1080/15228959.2011.621856   warner, a., hurley, d. a., wheeler, j., & quinn, t. (2020). proactive chat in research databases: inviting new and different questions. the journal of academic librarianship, 46(2), 102-134. https://doi.org/10.1016/j.acalib.2020.102134   white, m. d. (2001). digital reference services: framework for analysis and evaluation. library & information science research, 23(3), 211-231. https://doi.org/10.1016/s0740-8188(01)00080-9   evidence summary   hong kong students consider virtual reference a vital service and it can aid in many stages of learning   a review of: tsang, a. l. y., & chiu, d. k. w. (2022). effectiveness of virtual reference services in academic libraries: a qualitative study based on the 5e learning model. the journal of academic librarianship, 48(4), article 102533. https://doi.org/10.1016/j.acalib.2022.102533   reviewed by: samantha j. kaplan research & education librarian, liaison to the school of medicine duke university medical center library & archives durham, north carolina, united states of america email: samantha.kaplan@duke.edu   received: 31 aug. 2023                                                             accepted: 7 nov. 2023      2023 kaplan. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30430     abstract   objective – understand how virtual reference services (vrs) impact students’ learning using the 5e model (engage, explore, explain, elaborate, evaluate) as a theoretical framework.   design – exploratory qualitative study.   setting – major university in hong kong.   subjects – there were 10 participants between the ages of 18 and 35, including undergraduate and postgraduate students and one alumnus of the university.   methods – online synchronous semi-structured interviews of 30 minutes via zoom. interview data were transcribed and analyzed thematically according to the 5e learning model.   main results – whatsapp was the preferred form of vrs, over zoom, email, or phone. vrs can facilitate better awareness of library resources and supports resource exploration. whatsapp vrs is particularly valuable for students who may find other modes intimidating, overly formal, or inaccessible due to time constraints. vrs has grown in importance since the covid-19 pandemic.   conclusion – vrs provided via instant messaging is a valued service for students, but libraries, library websites, and librarians can all work to improve awareness of the option and possible uses. future work is needed to understand how demographics may influence patrons’ attitudes and experiences of vrs.    commentary   this article was appraised using the critical appraisal skills programme’s casp qualitative checklist (2018). in terms of the validity of results, there is a clear statement of research aims: to explore library users’ perceptions and preferences of vrs in an academic library in hong kong. however, the methodology is not entirely appropriate, as the authors state it was an exploratory qualitative study using the 5e model as a theoretical framework but also report frequency counts and observed demographic differences in participants’ perceptions of vrs, hallmarks of quantitative approaches (tsang & chiu, 2022, p. 7). while they acknowledge that further studies are needed to understand demographic differences, the purposive sample of 10 participants is not enough to know if their observations are valid at all.   the authors provide the interview questions, which address the research issue. unfortunately, the authors do not provide nearly enough information on their recruitment strategy, other than stating that they “adopted purposive sampling to select participants who fit a particular purpose or criteria” (tsang & chiu, 2022, p. 3). while they do summarize their participants’ age and the program in which they are enrolled, information about how they were recruited and selected would be valuable. the casp checklist asks whether the relationship between researchers and participants has been considered, and this is another area where the authors have omitted clarity. by not disclosing how they recruited participants and any possible relationship between them, they somewhat undermined the reliability of the findings.   while the findings are reported in great detail, with a clear chain of evidence between themes, quotations, and codes, more information about the actual data analysis process would increase confidence. the authors do not indicate if software was used for coding, if there was a codebook, how said codebook might have been developed, if they collaborated on coding, and how conflicts may have been resolved. one assumes the 5e learning model, which guided their framework and research question development, played a significant role. the authors explicate the phases of the 5e model with participant quotes in their analysis. the 5e model is an interesting choice for this study because it relies on student perceptions of the effectiveness of vrs without any additional assessment. the interview guide is structured in a way that could potentially predispose students to confirmatory answers, as it consists of structured and semi-structured questions that presume vrs effects on learning, potential to improve outcomes, and other positives rather than coming from a more neutral place.   the actual findings are valuable, if challenging to trust based on the methodological reporting gaps outlined above. the authors report that whatsapp is the preferred form of vrs, compared to zoom, email, or telephone. in their coding, this is partially because patrons can keep and retrieve saved records, though participants did note privacy concerns. this is interesting, given that a 2015 study by yang and dalal, which randomly sampled 362 four-year colleges’ websites, reported that 47.5% of college libraries provided instant messaging as a vrs and that the “top five chat web-based/software products used by libraries in the sample include libraryh3lp, questionpoint, libchat, zoho, and knowitnow24/7” (p. 71). these are products that are embedded in library websites, as opposed to whatsapp, which is a mobile app available to patrons without necessarily navigating to a library website. this distinction may make whatsapp or similar messaging apps far more accessible to patrons, though privacy concerns and setting realistic expectations about response times may be challenging for libraries. thus, given that some of the more conceptual findings are based on whatsapp as a preferred vrs, readers should be cautious about transferring this to their own settings.   the authors are right to comment on the covid-19 pandemic making vrs more important than ever. their findings in this area add to a growing body of literature body about the emergence of vrs as essential to patrons at this time.   references   critical appraisal skills programme. (2018). casp qualitative checklist. https://casp-uk.net/images/checklist/documents/casp-qualitative-studies-checklist/casp-qualitative-checklist-2018.pdf   tsang, a. l. y., & chiu, d. k. w. (2022). effectiveness of virtual reference services in academic libraries: a qualitative study based on the 5e learning model. the journal of academic librarianship, 48(4), article 102533. https://doi.org/10.1016/j.acalib.2022.102533   yang, s. q., & dalal, h. a. (2015). delivering virtual reference services on the web: an investigation into the current practice by academic libraries. the journal of academic librarianship, 41(1), 68–86. https://doi.org/10.1016/j.acalib.2014.10.003 research article   space use in the commons: evaluating a flexible library environment   andrew d. asher assessment librarian indiana university bloomington libraries bloomington, indiana, united states of america email: asherand@indiana.edu   received: 15 jun. 2016   accepted: 21 feb. 2017        2017 asher. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – this article evaluates the usage and user experience of the herman b wells library’s learning commons, a newly renovated technology and learning centre that provides services and spaces tailored to undergraduates’ academic needs at indiana university bloomington (iub).     methods – a mixed-method research protocol combining time-lapse photography, unobtrusive observation, and random-sample surveys was employed to construct and visualize a representative usage and activity profile for the learning commons space.   results – usage of the learning commons by particular student groups varied considerably from expectations based on student enrollments. in particular, business, first and second year students, and international students used the learning commons to a higher degree than expected, while humanities students used it to a much lower degree. while users were satisfied with the services provided and the overall atmosphere of the space, they also experienced the negative effects of insufficient space and facilities due to the space often operating at or near its capacity. demand for collaboration rooms and computer workstations was particularly high, while additional evidence suggests that the learning commons furniture mix may not adequately match users’ needs.        conclusions – this study presents a unique approach to space use evaluation that enables researchers to collect and visualize representative observational data. this study demonstrates a model for quickly and reliably assessing space use for open-plan and learning-centred academic environments and for evaluating how well these learning spaces fulfill their institutional mission.       introduction   as part of its efforts to transform library spaces and environments to meet students’ learning, collaboration, technology, and research needs more effectively, the herman b wells library at indiana university bloomington (iub) opened a newly renovated and redesigned learning commons in fall 2014. occupying the entire first floor (approximately 25,000 square feet) of the west wing of iub’s main research library, the learning commons was designed as a technology-focused learning centre that provides services and spaces tailored to undergraduates’ academic requirements with the goal of supporting a learning-centred paradigm of library use (see bennett, 2009).        to enable the diverse range of learning activities encompassed by this usage paradigm, the learning commons was designed to maximize flexible study and work spaces and was intended to represent a deliberate break from the previous service model. prior to the renovation, the learning commons’ space was configured as an “information commons” with 260 desktop computers in mostly hardwired and immobile computer-lab style rows, and with library and technology support services anchored to large desks (see forrest & halbert, 2009, pp. 93-96 for a summary and diagram of this space). in contrast, the redesigned space features a variety of multi-purpose spaces and contains two classrooms (one configured with media tables and one in a traditional teaching lab layout), a writing support and tutoring centre, 18 collaboration rooms with large-screen monitors, work tables, and whiteboards (12 configured with media collaboration tables containing built-in laptop and device display adaptors), 68 individual computer workstations, and multi-purpose seating for about 400 people comprised of a mix of tables, booths, soft benches, chairs, and lounge areas (see figure 1). all of these spaces are available for student use 24/7, except for the classrooms and writing centre, which may be reserved for workshops and programming. an array of walk-up services is provided in a “genius bar” style service hub containing desks for library circulation, course reserves, and equipment check out, directional and basic reference assistance, research consultation, technology and computer support, and peer mentors for help in navigating student services, degree planning, and career development. the configuration of these service hub desks is designed to be flexible, and the composition of the services offered varies based on the time of the semester and demand.    the emphasis on flexibility in the learning commons’ design assumes that users will engage in a variety of information production and consumption tasks using many types of devices (see delcore, teniente-matson, & mullooly, 2014). in this way, the learning commons can be understood as occupying the centre of a continuum between low-intensity informal spaces and high-intensity formal study spaces (see delcore et al., 2014; priestner, marshall, & modern human, 2016), and its mix of spaces and furniture are intended to support people working throughout this spectrum.   conducted approximately 9 months after its opening, this study sought to evaluate not only these assumptions about the learning commons’ design, but also its effectiveness as a learning space, by observing students’ adoption and usage of its facilities and services in their everyday academic activities.       figure 1 the floor plan of the learning commons with workstations, mixed use seating, and the service hub highlighted. (stock photographs provided by iub libraries communications. used by permission.)     literature review   beginning around 2000, the creation of “learning commons” was part of a larger trend in universities to shift teaching and learning pedagogies from an emphasis on a “culture of teaching,” to a “culture of learning” that recognizes the importance of the social dimensions of learning activities (turner, welch, & reynolds, 2013, p. 228; bennett, 2003, p. 10). in libraries, learning commons spaces tend to be seen as an evolution and extension of the “information commons” model, which reframes spaces originally intended to primarily support students’ information-seeking activities as locations for students to participate in information processes and produce knowledge in “a vibrant, collaborative, [and] technology-infused space” (accardi, cordova, & leeder, 2010, p. 312; turner et al., 2013, p. 230; somerville & harlan, 2008, pp. 1-36; bonnand & donahue, 2010).   a commitment to an understanding of students as intentional learners is a key aspect of the learning commons concept, and bennett asserts that these spaces should be “one of the chief places on campus where students take responsibility for and control over their own learning, and [should] employ library staff to enact the learning mission of the university through being educators” (2009, p. 194). a learning commons therefore supports the social dimensions of learning by providing spaces that enable a “variety of teaching and learning relationships” so that students can meet and work with fellow students, faculty, librarians, and other university staff and support units (head, 2016, p. 8). to fulfill this mission, learning commons require flexible spaces that are both formal and informal and that “accommodate both solitary and collaborative learning behaviors” (bennett, 2007, p. 18; see also head, 2016, pp. 2, 13-14; turner et al., 2013, p. 231).     although they represent a significant capital investment for universities and libraries, head (2016, p. 25) observes that relatively few academic library learning space renovation projects conduct systematic post-occupancy assessments, instead tending to rely on goals developed during the design process. nevertheless, bennett points out the importance of both initial post-occupancy performance evaluation for assessing how well a learning space meets the needs of its users in practice, and for continuing this evaluation “persistently” to assure the space’s ongoing effectiveness (2007, pp. 15, 23).   aims   the opening of the wells library’s learning commons presented an opportunity to help address the gap in ongoing assessment of learning spaces by enabling iub librarians to conduct a post-occupancy evaluation of the learning commons’ new work environments and to develop methods for periodic long-term assessments of the space. this study was designed to evaluate the learning commons by exploring a series of research questions about the ways individuals and groups were using its spaces and services on an everyday basis, including: “what types of students are using (or not using) the learning commons, and for what purposes?”; “what tasks and activities are taking place, and what are students trying to accomplish?”; “are students aware of the technology resources and services available and are these resources meeting students’ needs?”; and finally, “are the underlying assumptions about learning commons design and use requirements supported by students’ everyday practices?” answering these questions allowed librarians and administrators to appraise the efficacy of the learning commons’ design and assess how well it was fulfilling its intended mission as a learning space.   methods   data collection design & instruments   faced with the challenge of systematically studying a large 24-hour space, this study developed a mixed-method research protocol that combined time-lapse photography, direct unobtrusive observation, and random-sample walk-up surveys to gather a representative and multi-modal activity profile of the learning commons. this approach not only enabled the research team to quickly assess the usage of the learning commons, but also created tools that can be reused to rapidly and meaningfully evaluate changes in services, policies, or space configurations in the future.   the overall occupancy and use of open study spaces was evaluated using 10 time-lapse cameras placed along the interior perimeter of the learning commons so that its full area could be photographed automatically at regular intervals. usage data for group study rooms were collected using in-person unobtrusive observation and head counts.    walk-up surveys were conducted with both individuals and groups working in the learning commons. these surveys collected demographic, user experience, and satisfaction information using both open and close-ended questions (see appendices a & b), and were developed with the input of the learning commons operation team, which included representatives from all service units that cooperate and provide services in the space. the surveys were field tested with a small group of students to verify the clarity of questions, the time required for individual and group participants to complete the survey (about 5-10 minutes), and the time required by a research team member to complete a round of surveying according to the study’s sampling design (about 45 minutes-1 hour).    all instruments and procedures for this study were reviewed and approved by the iub institutional review board.[1]        figure 2 daily gate counts for the learning commons during the study period. peaks are typically mondays or tuesdays, while valleys are typically saturdays (low usage on march 14-21 was due to spring break).      data collection procedures   when utilizing observation and survey-based research methods in spaces like the learning commons, ensuring a representative sample of the space’s usage over time can be particularly difficult. the occupancy, types of users, and the activities taking place in a library space can vary dramatically over the course of a day, week, or semester (figure 2), making studies of these spaces potentially vulnerable to underlying structural bias within their sampling design. a formalized sampling technique is therefore useful to construct a study that accurately reflects a space’s use characteristics.    to this end, the learning commons study randomly selected 175 data collection times from all possible 5-minute increments between march 1 and may 8, 2015, covering the second half of the spring semester. these data collection times were used for both the automated time-lapse photographs and the actively collected observation and survey data.   observation and survey data collection was completed by a research team consisting of one librarian and seven graduate research assistants. these research team members were trained in the study’s sampling methods and data collection procedures by the study’s principal investigator, who also coordinated and oversaw the data collection process.      at each randomly selected data collection time the research team member first collected observation data and head counts for the learning commons’ group study rooms. once these observations were complete, the researcher then collected walk-up surveys from a group occupying one randomly selected room, as well as three or four randomly selected individuals from throughout the learning commons’ space. a tablet computer running qualtrics web-based survey software was used to generate the random selection and to conduct the surveys, as well as to guide the researcher through data collection procedure from beginning to end to ensure data were collected in a standardized way by all research team members.      figure 3 an example zone with numbered seats used for the random selection of individuals for walk-up surveys.     for the group surveys, a group study room was selected randomly until a group agreed to participate or at least four groups had been asked. for the individual surveys, the learning commons was divided into zones of roughly equal size. a zone was randomly selected first, and then a seat number was randomly selected from within that zone until an individual agreed to participate (figure 3) or at least four individuals had been asked. a new learning commons zone was then selected and the process was repeated until three or four surveys had been collected. in cases where there were so few people in the learning commons that randomly selecting an occupied seat was unlikely (e.g., during early morning hours), the researcher was allowed to override the selection and approach a person in any occupied seat to ask them to complete a survey.        data analysis   a combination of quantitative and qualitative approaches was employed to analyze the collected data. the time-lapse photographs were reviewed at the sampled data collection times to ascertain how many people were using the learning commons and to construct heat maps of how different areas of the space were utilized (see also khoo, rozaklis, hall, kusunoki, & rehrig, 2014 for a similar approach to heat mapping). observation data of the group study rooms were used to calculate occupancy rates, as well as to evaluate which technologies were utilized in the rooms. the survey results from individuals and groups were analyzed to obtain descriptive statistics about user demographic information, time spent in the learning commons’ space, and awareness and satisfaction with available services. finally, answers to the surveys’ qualitative questions were coded thematically and categorized for analysis using nvivo qualitative data analysis software to identify and understand patterns in users’ experience of and affective attitudes towards the learning commons.     results   in total, all 175 sampled data collection times were completed for the time-lapse photographs, while 95 data collection sessions were completed for the group study room observation and walk-up surveys, resulting in the collection of 304 individual surveys and 96 group surveys. data collection for the observations and surveys was hindered by the practical difficulties of conducting surveys on a 24-hour schedule (particularly with regard to the ability and willingness of graduate research assistants to conduct lengthy observation and survey procedures in the overnight hours). this number of observations produced a margin of error of 5.62% for individual surveys and 9.98% for group surveys, at a 95% confidence interval, which, although higher than what would be desirable for a statistical study, is adequate for the primarily descriptive goal of outlining the use of the learning commons during this time period. while i believe these observations are sufficiently robust to support the validity of the findings and conclusions presented in this article, it is nevertheless possible that not completing all the sampled times may introduce some degree of error into the observations, especially given that overnight times were more likely to be missed than times during the day.   user demographics   the demographic data collected during the learning commons survey revealed patterns in the types of students using the space that differed substantially from expectations based on iub’s enrollment figures.   kelley school of business students accounted for 37% of the undergraduate students surveyed in the learning commons, while these students comprise 21% of iub’s enrollment (figure 4). college of arts and science (cas) students comprised the next highest group at 29% of undergraduate users—slightly lower than the 34% expected by its enrollment, and inside the survey’s margin of error. however, cas  students in humanities disciplines accounted for only 1% of the students using the learning commons, compared to about 15% of undergraduate enrollment.      figure 4 undergraduate use of the learning commons by iub school of enrollment     figure 5 use of the learning commons by year of study and international student status.     undergraduates early in their educational career used the learning commons at the highest level, with first and second year students accounting for 49% of its use (compared to about 18% of enrollment) (figure 5). usage appears to decline with the third and fourth years of study, while graduate students accounted for about 16% of users—lower than the 22% expected from their enrollment, but not surprising given that the learning commons is targeted primarily for undergraduate use.   at 28% of the surveyed users, international students comprised a much larger proportion of the learning commons’ users than would be expected based on their university-wide enrollment of 13%. this high proportion of international students also resulted in a higher-than-expected number of self-identified asian students using the space (27% of users versus 6% of enrollment), while observed usage by other self-identified ethnic and gender groups generally corresponded to expected enrollment patterns.   space utilization   observations obtained from the time-lapse cameras demonstrated strong patterns in the study space utilization of the learning commons. while the entire space was in use fairly extensively, there was a clear hierarchy in users’ preferences. as shown in appendix c, computer workstations were the most in-demand areas and were occupied 45-63% of the time. tables were the next most used category of furniture, typically occupied from 16-36% of the time, while soft seating and lounge areas were the least used, usually at 16% of the time or less. with 175 observations, the sampling design of this aspect of the study enables the calculation of confidence intervals for each of these observed frequencies. for example, for observed values above 36%, the confidence interval is approximately +/-7% at a 95% confidence level (see bernard and killworth (1993) for a detailed explanation of this calculation at varying observed frequencies).     on average, 10.5 of the 18 collaboration rooms in the learning commons were occupied during the observation times. every room exhibited an average occupancy rate of above 50%, while the four most popular rooms exceeded 70% occupancy (appendix d). the confidence interval for all of these observed frequencies is approximately 7% at a 95% confidence level.  rooms configured in the media-table layout were more popular than those with circular tables and chairs, and at the learning commons’ busiest times of 4-8 p.m. and 8 p.m.-12 a.m., the group study rooms were almost completely occupied (at 15/18 and 16/18 on average respectively).     however, based on the number of seats occupied, the group study rooms were often not used to capacity. the average group size was 2.27 people per room, while the average capacity is 5.5 (with room capacities ranging from 4 to 7). people using the group study rooms also did not appear to be using the technology provided in the rooms to as high a degree as was anticipated—the average number of large-screen monitors in use in the group study rooms was only 5 of 18, compared to 22 laptops that students had brought with them.   in general, the learning commons’ users reported planning to stay for relatively long blocks of time. a total of 78% of individuals and 71% of groups said they planned to stay in the learning commons for least 2 hours, and only about 5% said they would stay less than 1 hour (table 1). when asked in the qualitative section of the surveys what they wanted to accomplish while at the learning commons, a majority (55.3%) of individuals described a specific academic task such as completing projects or papers. another 39.1% said “studying,” while 17% said “preparing for an exam.” groups followed a similar pattern, with 62% mentioning a specific task, 60% studying, and 18% preparing for an exam.    table 1 intended duration of work in the learning commons     user experience and satisfaction   user satisfaction with the learning commons was generally very high, with 87% of users indicating that they were either “satisfied” or “very satisfied” overall.   when asked why they had decided to come to the learning commons, individuals emphasized the atmosphere and availability of computer workstations, while groups emphasized the collaborative space and its associated technology (e.g., whiteboards and large computer screens), as well as the availability of private and quiet spaces.    when asked what was best about the learning commons, both individuals and groups again mentioned available technology, the overall environment (especially private and quiet spaces, even though the learning commons is not designated as a quiet space), the furniture, and the availability of computers. in general, individuals tended to highlight features that support working alone, while groups noted features that support collaboration. conversely, many of the same items were also discussed when users were asked what was the worst thing about the learning commons. one third of learning common users stated that there were not sufficient study spaces, and both groups and individuals complained about insufficient or unavailable technology, furniture, computers, and collaboration rooms. noise levels and inadequate soundproofing were also regularly mentioned as problems.      figure 6 learning commons service awareness.     figure 7 learning commons service use.     of the services available in the learning commons, students were more likely to be aware of technology support services, writing tutorial services, and library circulation than peer mentoring and research consultation support (figure 6), but they were much more likely to have utilized technology services than library services (figure 7). only 8% of respondents reported using research consultations compared to 51% who had used the university information technology services (uits) support centre. however, 60% of users reported that they had asked for help from the learning commons staff at least once. the learning commons’ users continued to favor obtaining assistance in-person, with 72% saying that if they needed help they would most prefer to get it at a walk-up desk (online chat was the second most preferred method at just 9%).   when students had used the learning commons’ services, satisfaction was uniformly very high at around 70% for all services. satisfaction with technology-related services (it training, uits support, and technology center consulting) was even higher at around 90% (figure 8). while the most common answer was “nothing” when asked what additional services they would like in the learning commons, a  handful of users reported a desire for more tutoring in a variety of subject areas (particularly math, foreign languages, and writing).     figure 8 learning commons service satisfaction.     discussion    the distinct patterns observed in the demographics of the learning commons’ users likely result from a combination of space design, location, and pedagogical factors. the learning commons appears to serve some types of students very well, such as the business students that comprise the largest user group. the intensive use of the space by these students is likely in part due to the wells library’s close proximity to the business school, which is located less than one block away (and whose own library is often occupied at full capacity), as well as the collaborative work requirements of many business courses. the importance of the learning commons as a group meeting location is further suggested by the prevalence of students majoring in social science and stm disciplines, curricula that also typically include a number of courses that emphasize collaborative and group-based projects. conversely, the relative absence of humanities students may suggest that the open and group-oriented environment of the learning commons does not serve the needs of these students. it is not clear from the survey data whether this is because these students are engaged in more solitary work that is not facilitated by the space or because they need resources that are unavailable, and this finding warrants additional study.   compared to what would be anticipated by enrollment, students arriving at the learning commons’ service desks are more likely to be early in their undergraduate careers, more likely to be studying in the business school, and more likely to be international students. librarians, staff, and graduate assistants working at these service desks should be especially trained and prepared to address the needs of these groups. follow-up studies or surveys might seek to specifically identify if there are additional needs of both high-use and low-use groups of students that could be met either via current learning commons services or by collaboration with other campus units: for example, esl support or other international student services, curriculum-targeted workshops, tutoring or research consultation, or services and programming designed to reach out to non-using groups of students, such as humanities majors.   the overall success and popularity of the learning commons produces many of the problems identified in this study. while users enjoyed the overall environment and atmosphere of the space, they often complained that it was too crowded and had insufficient collaboration rooms, available furniture, and workstations. in a survey of library space choice, cha & kim (2015, p. 277) identified the amount of space, noise level, crowdedness, and comfort of furnishings to be the four most important factors students consider in choosing to use a space, so it is perhaps not surprising that this cluster of characteristics appears simultaneously in both positive and negative evaluations of the learning commons. the observed problems in all of these areas can also ultimately be linked to the learning commons routinely operating at or near its capacity.    the usage patterns of the learning commons furniture and group study rooms suggest that many of the spaces’ resources are in extremely high demand. combined with the relatively high observed use of workstations, the desire for additional computers suggests that the nearly 75% reduction of workstations (from 260 to 68) after the learning commons’ renovation might have been too extreme, and that the capacity of computing resources located in the learning commons is not adequate for users’ needs. while reducing the number of workstations was a deliberate decision to help make the learning commons’ space more flexible, and many of the removed workstations were redistributed to other spaces in the building (the net loss was only about 80 computers), users clearly experience the diminished number of computers as a deficiency of the space. nevertheless, given the extensive overall use of the learning commons, workstations, tabletop work surfaces, and group work spaces might be in such high demand that almost any amount provided would be perceived as insufficient.   users’ preference for tables likely reflects students’ need for hard work surfaces for laptops, books, and other materials, a finding similar to holder and lange, who also found that students indicated a preference for “traditional furniture such as tables and desk chairs” (2014, p. 15). in terms of space planning, it is probably worth considering allocating a higher proportion of seats to workstations and table seating instead of soft seating and lounge areas. with a current mix of 53% tables, 25% soft seating, and 17% workstations, the learning commons’ most in-demand seating is also the least available, while a quarter of available seats are under-utilized or used principally during the busiest times when no other places are available.     shifting some soft seating and lounge areas to workstations or tabletop surfaces might help alleviate demand on these resources and increase the capacity of the space, although the learning commons’ managers should also carefully observe how the delicate balance between space and furniture types might affect use. as khoo et al. (2014, pp. 617-618) observe, the perceived occupancy of a space is often as important as its actual occupancy, and depending on the type of furniture and its layout, a space can feel full from the standpoint of the user even if many seats remain open—in some cases even if half of places remain unused (gibbons & foster 2007, p. 28). similarly, priestner et al. argue that library work spaces need to provide users with enough available “study territory” so that each seat feels inviting, and they demonstrate that in some cases occupancy can counterintuitively be increased by decreasing the number of seats in a space to provide more territory to each seat (2016, pp. 22-24 ). khoo et al. conclude that “practical occupancy limits for open plan study spaces could be significantly lower than the theoretical maximum seating” (2014, p. 618).    within an open environment like the learning commons, that is already perceived and experienced as busy and crowded during many of its open hours, simply adding additional seats and furniture might exacerbate the problem even if the absolute capacity is increased. to determine an optimal layout and furniture mix, the learning commons’ managers and administrators might consider an iterative prototyping approach to adjusting the space (priestner et al., 2016, pp. 5-7), in which a series of changes are made to the space’s configuration and the effects on usage and user behaviour are carefully observed at each step. in this way the flexibility that was designed into the learning commons could be effectively leveraged to balance the demand for both solitary and collaborative spaces, to continue to improve the experience of the space for its users, and to more fully respond to students’ learning needs.     despite these capacity issues, the relatively long planned study sessions reported by both individuals and groups suggests that the learning commons adequately supports the goal of creating a space that “acknowledge[s] the social dimension of . . . learning behaviors and that enable[s] students to manage socializing in ways that are positive for learning . . .” by “encourage[ing] more time on task and more productive studying” (bennet, 2007, p. 17). this sustained time in the learning commons is important to its effectiveness as a learning space, and confirms the presence of an audience for library and university support services.     while the high levels of satisfaction with the services available in the learning commons are encouraging, the relatively weak usage of the services available suggests that the learning commons is not yet delivering on its goal of delivering point-of-need learning. particularly disappointing was the low use and mostly moderate awareness of learner-focused services such as research consultations, peer mentors, it training, and writing tutorial services. this low use of library services relative to technology support services further indicates that there may be a disconnect between the types of help and assistance students perceive to be available and the broader range of services that are offered, and that more programming may be necessary to develop students’ identification of the learning commons as a multifaceted learning space.    conclusions   the renovated learning commons is clearly a popular and well-used collaboration and study space used for a variety of academic tasks and activities. however, it is less certain the degree to which it is fulfilling the “learner-centered” paradigm of design (bennett, 2009) that asserts the need for providing flexible spaces that support not only the multifaceted, frequently changing, and self-managed learning activities of students, but also the diverse types of teaching and learning relationships encompassed by the social dimensions of learning (turner et al., 2013, p. 231; bennett, 2013, p. 38).   the high rates of occupancy observed in the learning commons’ group study rooms and open study spaces suggest that it is offering attractive locations for many types of student work, while the high overall satisfaction with the redesigned space supports the efficacy of shifting toward a more flexible approach to the provision of space and library and technology services.   this popularity may result in the learning commons’ falling short in providing adequate spaces for all types of students and student activities. while it contains areas for both solitary and group work, the learning commons’ design and furniture configuration emphasizes collaborative activities. as is illustrated by the disciplinary distribution of students using the learning commons, the space appears to attract students in curricula that tend to have high numbers of group-oriented assignments. the success of the learning commons as a collaborative space may be pushing out students in need of a more solitary work environment.    by supporting collaborative relationships among students, the learning commons effectively facilitates one aspect of the social dimension of learning. nevertheless, the low reported identification and usage of available services besides it and technology support indicates that additional outreach is needed to build relationships between students, librarians, and other service providers such as the writing centre and peer tutors, so that students begin to identify the learning commons as a multifaceted learning space.    the results of this study’s initial post-occupancy evaluation of students’ everyday use of the learning commons thus illustrates a space that has been well received by students and meets many of their educational needs, but only partially fulfills its goals as a learning-centred space. while the learning commons successfully enables some of the social dimensions of learning by providing a variety of collaborative spaces and supporting technologies for students to engage with one another and information resources, it has not yet fully integrated relationships with other library and campus services. as with any space committed to a learning-centred paradigm, developing these relationships within the learning commons is a continuous project, needing ongoing outreach, service development, and evaluation efforts to ensure its success.    acknowledgements   this study was funded by the iub libraries. the author would like to thank joseph eldridge and brian winterman for their assistance in data collection and analysis.   references   accardi, m. t., cordova, m., & leeder, k. (2010). reviewing the library learning commons: history, models, and perspectives. college & undergraduate libraries, 17(2–3), 310–329. http://dx.doi.org/10.1080/10691316.2010.481595   bennett, s. (2003). libraries designed for learning. washington, dc: council on library and information resources. retrieved from https://www.clir.org/pubs/reports/pub122/pub122web.pdf   bennett, s. (2007). first questions for designing higher education learning spaces. the journal of academic librarianship, 33(1), 14–26. http://dx.doi.org/10.1016/j.acalib.2006.08.015   bennett, s. (2009). libraries and learning: a history of paradigm change. portal: libraries and the academy, 9(2), 181–197. http://dx.doi.org/10.1353/pla.0.0049   bernard, h. r., & killworth, p. d. (1993). sampling in time allocation research. ethnology, 32(2), 207–215. http://dx.doi.org/10.2307/3773773   bonnand, s., & donahue, t. (2010). what’s in a name? the evolving library commons concept. college & undergraduate libraries, 17(2–3), 225–233. https://doi.org/10.1080/10691316.2010.487443   cha, s. h., & kim, t. w. (2015). what matters for students’ use of physical library space? the journal of academic librarianship, 41(3), 274–279. http://dx.doi.org/10.1016/j.acalib.2015.03.014   delcore, h., teniente-matson, c., & mullooly, j. (2014). the continuum of student it use in campus spaces: a qualitative study. educause review online. retrieved from http://er.educause.edu/articles/2014/8/the-continuum-of-student-it-use-in-campus-spaces-a-qualitative-study   forrest, c., & halbert, m. (eds.). (2009). a field guide to the information commons. lanham, md: scarecrow press.   gibbons, s., & foster, n.f. (2007). library design and ethnography. in n. f. foster & s. gibbons (eds.), studying students: the undergraduate research project at the university of rochester. chicago, il: association of college and research libraries. retrieved from http://www.ala.org/acrl/sites/ala.org.acrl/files/content/publications/booksanddigitalresources/digital/foster-gibbons_cmpd.pdf   head, a. j. (2016). planning and designing academic library learning spaces: expert perspectives of architects, librarians, and library consultants. retrieved from https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2885471   holder, s., & lange, j. (2014). looking and listening: a mixed-methods study of space use and user satisfaction. evidence based library and information practice, 9(3), 4–27. http://dx.doi.org/10.18438/b8303t   khoo, m., rozaklis, l., hall, c. e., kusunoki, d., & rehrig, m. (2014). heat map visualizations of seating patterns in an academic library. iconference 2014 proceedings. http://dx.doi.org/10.9776/14274   priestner, a, marshall, d., & modern human (2016). the protolib project: researching and reimagining library environments at the university of cambridge. retrieved from https://futurelib.files.wordpress.com/2016/07/the-protolib-project-final-report.pdf   somerville, m. m., & harlan, s. (2008). from information commons to learning commons and learning spaces: an evolutionary context. in b. schader (ed.), learning commons: evolution and collaborative essentials (pp. 1–36). oxford: chandos publishing.   turner, a., welch, b., & reynolds, s. (2013). learning spaces in academic libraries: a review of the evolving trends. australian academic & research libraries, 44(4), 226–234. http://dx.doi.org/10.1080/00048623.2013.857383     appendix a learning commons space use assessment survey for individuals   1.       why did you decide to come to the learning commons today?   2.       what would you like to do or accomplish while you are here?   3.       how long will you stay in the learning commons for this visit? ·         less than 30 minutes ·         30 minutes to 1 hour ·         1-2 hours ·         2-3 hours ·         more than 3 hours   4.       what is the best thing about the learning commons space?   5.       what is the worst thing about the learning commons space?   6.       how many times in the last seven days have you used the learning commons space?   7.       what would make you want to use the learning commons more often?   8.       are there sufficient study spaces in the learning commons? ·         yes ·         no ·         i don't know/ i'm not sure   9.       if you need help with something you are working on, how would you most prefer to get assistance? ·         in person at a walk-up help desk ·         online chat ·         email ·         text message ·         telephone ·         in person by appointment ·         other ____________________   10.    how easy is it for you to get help in the learning commons? ·         very easy ·         easy ·         neutral ·         difficult ·         very difficult ·         i don't know   11.    have you ever asked for help from the staff in the learning commons? ·         yes ·         no ·         i don't know/ i'm not sure   12.    [if yes selected for #11] thinking about only the most recent time you asked the staff of the learning commons for help, what did you need help with?   13.    [if yes selected for #11] how effective were the learning commons' staff in answering your question? ·         very ineffective ·         ineffective ·         neither effective nor ineffective ·         effective ·         very effective   14.        prior to this survey, i was aware that this service is available in the learning commons i have used this service   yes no yes no technology center consulting m    m    m    m    uits support center m    m    m    m    writing tutorial services m    m    m    m    research consultations m    m    m    m    library circulation m    m    m    m    university division peer mentors m    m    m    m    it training m    m    m    m        [for the services used] how satisfied were you with the following service:  ·         very satisfied ·         satisfied ·         neutral ·         dissatisfied ·         very dissatisfied   [for the services not used] how likely are you to use the following service:  ·         very likely ·         likely ·         undecided ·         unlikely ·         very unlikely   15.    what additional services would you like to see offered in the learning commons?    16.    what is your overall satisfaction with the learning commons? ·         very dissatisfied ·         dissatisfied ·         neutral ·         satisfied ·         very satisfied   demographic questions   d1. what is your age?   d2. what gender do you identify with?  ·         male ·         female ·         i don't identify with either of these. i identify as: ____________________   d3. what is your year of study? ·         first year ·         sophomore ·         junior ·         senior ·         graduate ·         faculty member ·         other   d4. what is your major or department?   d5. what race or ethnicity do you most identify with? ·         black or african american ·         hispanic or latino ·         white or caucasian ·         asian ·         american indian or alaska native ·         native hawaiian or other pacific islander ·         i don't identify as any of these. i identify as: ____________________   d6. are you an international student? ·         yes ·         no   if yes, what is your country of citizenship:     appendix b learning commons space use assessment survey for groups   1.       how many people are in your group?   2.       why did your group decide to come to the learning commons today?   3.       what would your group like to do or accomplish while you are here?   4.       how long will your group stay in the learning commons for this visit? ·         less than 30 minutes ·         30 minutes to 1 hour ·         1-2 hours ·         2-3 hours ·         more than 3 hours   5.       is your group: ·         working together on a single assignment or project for a course ·         working or studying together but on different assignments ·         working on an extracurricular project ·         socializing or working on something not related to your studies ·         working on something else-what? ____________________   6.       if your group is working together on a course assignment or project, what course is it for?    7.       what is the best thing about the learning commons space?   8.       what is the worst thing about the learning commons space?   9.       what would make your group want to use the learning commons more often?   10.    are there sufficient group study spaces in the learning commons? ·         yes ·         no ·         i don't know/ i'm not sure   11.                what is your group's overall satisfaction with the learning commons? ·            very dissatisfied ·            dissatisfied ·            neutral ·            satisfied ·            very satisfied     appendix c  heat map of the learning commons open study areas     appendix d heat map of the utilization of the learning commons’ collaboration rooms     [1] classified as exempt. protocol #36077373. commentary   finding out is better: becoming a librarian-researcher   ann glusker librarian for sociology, demography, & research methods social research library university of california, berkeley berkeley, california, united states of america email: glusker@berkeley.edu   received: 21 dec. 2022                                                                 accepted: 21 mar. 2023      2023 glusker. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30293     “supposing is good, but finding out is better.” – mark twain   i became a librarian because i love research. specifically, i love the process of finding things out. it almost doesn’t matter what—if i have a sense that the answer is there to be found, i want to dive in and find it, as many of us do. as a reference librarian at heart, it matters to me that the result of a search will be of benefit to some or many people, but still, beyond that, i find the process intrinsically satisfying.   i entered librarianship with a background in population studies and public health research. as i gained experience as a librarian, i began to engage professionally in various ways, including doing research. this paper outlines my process in becoming, and embracing my identity as, a librarian-researcher. it also offers possibilities for how all of us who work in libraries can take steps to incorporate this important focus into our work.   my research background   i date the beginning of my research career to the moment that i began a life-changing course at the university of pennsylvania called “introduction to demography.” it would probably not be life-changing for anyone else, but it was wholly unexpected for me to be so intrigued by a subject. demography was, and still is, the academic subject about which i am most passionate, and which i find the most fascinating. i still revere the professor who taught it and was galvanized by the experience into leaving my career counseling job at the university of pennsylvania in philadelphia, to start my phd in sociology/demography at the university of washington in seattle. it took me seven years to get that phd, a normal timespan, during which i was a research assistant on three different projects, worked with some 15 data sets using four different statistical tools/packages, and took eight courses consisting solely of research methodology (most of which i don’t remember now).   i discovered the reference librarianship aspect of my research life during the ten years that i was a public health epidemiologist for a local city-county health department after getting my phd. i worked with an even wider range of datasets, analyzed and reported on public health data in a range of venues, and answered data requests from government agencies, researchers, students, the media and the public. finding health-related information for people was so rewarding that i decided to make reference-related work a central career focus and went back for my master’s degree in library and information science (mlis). yet, even after having written a dissertation, a book, several papers, and multiple public health reports, and with all my considerable training, i would not have called myself a researcher at the time i started my mlis degree.   librarianship was a revelation. i became serious about pursuing it after two wonderful librarians gave me the “librarian test” and i passed (“after you see a movie, do you run right home to research what it was about?” “of course, doesn’t everyone?”). while i was thrilled to have finally found the right career, i felt i was moving further away from the research world as i conceived of it. my main research-related concern during my mlis program was getting the research methods course waived. i wasn’t seeing librarians as researchers then because my viewpoint was so narrow and narrow-minded, but i see now that the very thing that attracted me to librarianship was the possibility of finding answers using sophisticated strategies and tools, carried out using rigorous processes. librarians were doing this every day in varied settings, similarly to epidemiologists, and academics, whether in the context of answering a reference question from a patron, or of setting out to understand a professional phenomenon in their own worlds.   embracing my identity as a librarian-researcher   in my first job as a librarian, in a medical setting, after about a year of orienting to my new professional culture, i started to think about my professional path. i was amazed at how different librarianship was than public health and demography had been, in terms of what could be presented at conferences and what was being published. it seemed comparatively open and accessible. giving a professional presentation no longer meant facing a terrifying gauntlet of methodological criticisms and competitiveness. in librarianship, intellectual curiosity, creative investigative methods, and precision in communicating content were there, but often with the crucial aspect of having a goal of improving services and processes in order to serve constituencies. the content of the research products from librarians felt varied, and immediately applicable to my questions and concerns. for example, i liked that the many available examples of case studies had an investigative quality that allowed them to be embraced as an initial step toward enhanced practice.   soon i started reading and following the library journal from the university of alberta, evidence based library and information practice. it dovetailed perfectly with teaching evidence based practice to clinicians as a medical librarian, and i have carried its viewpoint and lessons with me in my subsequent career. this journal also gave me one of the best learning experiences in critical appraisal of research that i have had in any of my careers, that of writing evidence summaries (short evaluative pieces about library research papers).   until this point, i had been defining research through a narrow lens of quantitative and science-based inquiry (with some judgmental tendencies remaining from my earlier studies), but now i was able to broaden my perspective. over time i came to believe that research is defined as the process of investigating a question, involving systematically and rigorously gathering and analyzing information to answer it. although there are disagreements about appropriate methods for arriving at the answer, the methods are merely a tool, while answering the research question is the central concern.   through this shift in perspective, i began to consider myself a researcher, defined in a new way for me. i had many professionally-related questions and worked to answer them, as i saw others doing. i researched and communicated my findings to library communities. people were interested in my findings. i was still able to participate in these avenues in my new position as a public librarian, researching and creating a conference poster on the question of best practices for working with health consumers’ numeracy issues.   by the time i had my first academic librarian job, an outreach position at the university of washington, i had finally embraced my identity as a librarian-researcher. i was answering questions i had about how things worked, using rigorous processes. i was creating new content to advance practice, and i was engaging professionally with my field in more rewarding ways than i ever had. i was no longer conducting much quantitative analysis, but that no longer felt central to me as a marker of worth of the research or the researcher.   however, for my own story, there is another chapter, which has led to my writing this paper. in 2019, i went to the university of california, berkeley. it was my first academic library liaison position, after almost 15 years as a librarian with a phd. i was and am the liaison to the sociology and demography departments, and i consider this the perfect job for me. i could feel the rehydration of my desiccated connections to sociology and demography as disciplines. i re-read the classic textbook, shryock and siegel’s the methods and materials of demography, with nostalgia and joy. i read my faculty members’ research profiles with deep interest, and, surprisingly to me, understanding. i felt solidly placed in academia writ large, and while it was not required, it seemed more natural that in this setting i might engage in what i thought of as research with a capital r. i felt i should once again do research conducted in the way i had learned as a demography graduate student 20+ years earlier, the kind a demography faculty member would recognize and respect. i took a coursera course in r (the statistical computing language) to get ready.   as it turned out, i didn’t need statistical training; i needed the open-mindedness and time to learn new research methodologies. the two new research projects i engaged with were both qualitative rather than quantitative studies. one was an ithaka s+r sponsored study examining big data research practices on campuses nationwide. the other was a team project on factors affecting the morale of library staff (as opposed to that of librarians), for which i obtained principal investigator status, applied for funding and human subjects research approval, arranged for a qualitative data analysis software tool license, negotiated bureaucracies to hire, pay, and supervise a transcriptionist, shepherded the project, and more. we learned qualitative methods from our experienced team member, and we interviewed, transcribed, coded, analyzed, and presented. we found the information to be so rich that we created four separate topics, one for each of us to pursue.   it was when i was analyzing and writing up the results from my topic area—the role of management in staff morale—that i started to connect with how much my earlier graduate study could inform even my qualitative research work. i had disconnected the two until this point, but now i realized that the depth and experience in research that i brought, even though it was two decades old, was helpful. i did an extensive literature review (more than 90 articles) and created detailed charts of the demographics of our respondents and their institutions and comparing them to published data. furthermore, after reading about the depth and complexity of grounded theory methodology in qualitative research, i worked to incorporate a theoretical model into my approach. a number of the papers i reviewed drew from grounded theory methodology but didn’t actually use the methodology in its intended form to create a theory arising from the data gathered. i saw this as a loss, since it would have been really interesting to see the more complete realization and visualization of their theoretical approaches. the resulting paper, “‘viewed as equals’: the impacts of library organizational cultures and management on library staff morale” (glusker et al., 2022), was—and will remain—the one in my librarian career in which i used my background as a social science researcher the most. in particular, until this point i hadn't connected theory and theoretical models to my research as a librarian, and i hadn't sought out the appropriate research methods for my questions. getting back into the processes of original research, including human subjects review and grant applications, was a trigger for me to revisit those and other practices i had learned earlier, while of course i should have been doing them all along.   as i reflected on this experience, i felt that my social science research background provided me with strong preparation for the process i eventually enacted, but it was not entirely necessary to the success of the project. there is a large and strong body of practice-focused literature by librarian-researchers, and i see amazing research, more in-depth and creative than this, and more technically adept, in the library science literature every day. at the same time, when reviewing the literature for the paper, i saw instances in which i felt that more quantitative research training might make librarian researchers feel the same deep satisfaction i do in extracting hidden patterns and trends from quantitative data, and also that qualitative research training would improve the quality and rigor of many library science studies.   incorporating a research focus   if we define research as the process of investigating a question through methodically gathering and analyzing information to answer it, then in the end, how does one become a librarian-researcher? my outline of my own experience raises more questions than it answers. perhaps we are all researchers already if we say that there is a continuum along which librarians lie, in considering themselves and striving to be researchers, rather than some dividing line between researcher and non-researcher. we could posit that we are all researchers because finding things out is our core value. even more existentially, we can consider what being a librarian-researcher means to each of us, professionally and personally. i believe that if there is the good fortune to have workplace support and bandwidth, any librarian in almost any setting can be a researcher. the avenues for promoting, and interest in consuming, a range of research products are there.   my hope is that we explicitly incorporate research into our professional identities, and that we don’t let others define or narrow that meaning for us. this can be challenging when our self-identities as librarians and researchers may be at odds with what is needed and rewarded in our workplaces. for example, in academia, librarians may find their research identities being shaped by faculty standards and the need to get tenure; in medical settings librarians may have to put research last since it is not considered a central function in terms of library services, and is not counted in performance reviews; and in public libraries, librarians may be told that research is “not what public librarians do.” in addition, even if there is support for research, it can be challenging to carve out time for an activity that may not be seen as “core”.  the best-case scenario would be, because library administrators value informed decision-making, that developing and answering research questions becomes (or perhaps already is) a standard procedure for improving services to users. either way, i hope we can find ways to be scholar-researchers, practitioner-researchers, and reference-researchers in whatever ways that fit for us and are possible in the settings in which we find ourselves. sociologist andrew abbott (1998) notes: “librarians… are used to relearning their jobs every decade or so, and that is in fact the paradigmatic experience in most professions” (p. 442). if our self-identity as librarians can be dynamic, our self-identity as researchers can be as well. i have worked as an academic, medical, and public librarian, and i know how impossible this sounds—but i also know it is possible.   part of this is acknowledging and promoting the research-related activities we are already involved in as part of our librarian identities. as with data literacy, where librarians may feel they are not “data people” but actually are already doing detailed and important data-related work, many of us are already research-involved in some way. we can recognize and appreciate what we already do—looking at circulation patterns, reading research-related articles, attending conference presentations in which research is presented, doing patron evaluation surveys at the end of our programs. if these spark more in-depth original research, that’s wonderful, and if they spark nothing, and we are not drawn to formal research at all, that’s also fine and we are in good company.   for those of us who are drawn to research, there is always more to learn about it. most research-related papers end with some ideas for future research, which can be rich troves of questions to be explored in a variety of creative ways. even if we feel that we have a good background in research, there is always the latest new thing to discover; for example, reading a paper on librarian-researchers and their networks (kennedy et al., 2020) has inspired me to take a course in social network analysis. i’d also encourage us all not to fear quantitative analysis. it may seem daunting to see a spreadsheet with incomprehensible variable names and thousands of rows, but perhaps there is a thrill in extracting patterns and new information for the first time, and it doesn’t have to take years to learn how to do it. easy and accessible online tools such as databasic (databasic.io), rawgraphs (rawgraphs.io), and datawrapper (datawrapper.de) mean that anyone can take a spreadsheet of interest, upload it, and immediately see percentages and visualizations that both answer and raise more questions. i also encourage us to respect qualitative analysis done well; it is a process which requires intensive effort and rigor and is far from the easy way out some perceive it to be.   especially these days, it is becoming easier and less expensive to pursue research and methods training, and i am often asked how much and what kind of training is needed to perform research. i would never suggest that anyone suffers through research training in which they aren’t interested. if someone feels drawn to researching their professional questions, there are many ways to produce the answers. i feel sad when i see librarians in a grim forced march toward finishing a research project they feel is needed for their professional status; i think to myself, “if only the research process could be intrinsically rewarding to them!” i don’t think there is any set answer to this question—my thought is that it depends on setting and individual proclivities. however, some starting places might be to look at the literature and explore what types of research and research methods appeal, to find a research team on which to participate and learn, to take some basic methods workshops with hands-on components to see what feels doable, and to begin to research and present on processes and topics which are manageable in scope and close to hand. we should all recognize that it takes time and practice to feel more confident in taking on large projects and methodologies, and most often large projects involve a team approach, meaning that the research methods expertise may sit with other team members, but all contribute.   last but not least, i’d like to see more librarian research follow the tenets of the evidence based practice movement. i’ve already mentioned the journal evidence based library & information practice, and i’d like to also recommend a handbook for being evidence based as a librarian, being evidence based in library and information practice. it “brings together recent theory, research and case studies from practice environments across the broad field of librarianship to illustrate how librarians can incorporate the principles of evidence-based library and information practice (eblip) into their work” (koufogiannakis & brettle, 2016, p. 3). being researchers can enhance our own practice, along with helping others. if we conduct ourselves and our work lives with evidence at the core, we take control of our identities as librarians and librarian-researchers, whatever that means to each of us.   this is all because supposing is good, but finding out is better.   acknowledgements   i’d like to thank etienne van de walle and samuel preston, who believed in me and encouraged me to become a demographer; gail kouame and lisa oberg, who gave me the librarian test; susan edwards and celia emmelhainz, who have been mentors and supporters of my research life at uc berkeley; and erin foster and carol perryman, who emboldened me to write this paper.   references   abbott, a. (1998). professionalism and the future of librarianship. library trends, 46(3), 430–443.   glusker, a., emmelhainz, c., estrada, n., & dyess, b. (2022). “viewed as equals”: the impacts of library organizational cultures and management on library staff morale. journal of library administration, 62(2), 153–189. https://doi.org/10.1080/01930826.2022.2026119   kennedy, m. r., brancolini, k. r., & kennedy, d. p. (2020). an exploratory study of accomplished librarian-researchers. evidence based library and information practice, 15(1), 179-217. https://doi.org/10.18438/eblip29655   koufogiannakis, d. & brettle, a. (2016). being evidence based in library and information practice. facet publishing.   evidence based library and information practice 2012, 7.3 121 evidence based library and information practice ebl 101 research methods: bibliometrics virginia wilson librarian, murray library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca received: 1 aug. 2012 accepted: 8 aug. 2012 2012 wilson. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. bibliometrics! metrics about biblios: a set of measurements about a book, or books, or text. the oxford english dictionary defines bibliometrics as “the branch of library science concerned with the application of mathematical and statistical analysis to bibliography; the statistical analysis of books, articles, or other publications” (2012). as a research method, beck and manuel state that “essentially, if you can count something without too much debate over what it is—and it has something to do with any type of recorded information source—it could become the subject of a bibliometric study” (2008, p. 165). bibliometrics can also be referred to as informetrics, webometrics, scientometrics, and cybermetrics. the different terms basically reflect the types of information to which the analysis is applied. bibliometrics involves the measurement of data not intrinsic to the text, i.e., the text does not need to be read in order for analysis to be conducted. instead, extrinsic measures such as affiliation of author, word frequency, usage statistics, and, most commonly citations are measured. there is a school of thought that combines bibliometrics with content analysis (extrinsic and intrinsic) but for the purposes of this column, i intend to focus on bibliometrics alone. while bibliometrics has been around for decades, and “is one of the oldest research methods in library and information science,” it’s only since the advent of high powered computer technology that bibliometrics has exploded (beck & manuel, 2008, p. 166). “bibliographic databases, citation indexes, and statistical programs” have added ease, depth, and breadth to bibliometric analysis (powell & connaway, 2004, p. 63). another major development for bibliometrics has been the internet and the publishing of “an increasingly broad range of research-related documents, from articles to email discussion lists, allowing the creation of a range of new metrics relating to their access and use” (thelwell, 2008, p. 605). mailto:virginia.wilson@usask.ca evidence based library and information practice 2012, 7.3 122 a quick scan of the articles returned on a search for “bibliometrics” and “librarian research” in the library and information science (lisa) database shows a variety of ways that this research method has been used: sengupta, i. n. (1990). bibliometrics and the identification of core periodicals. herald of library science, 29 (3-4), 226245. mancall, j. c., & drott, m. c. (1979). materials used by high school students in preparing independent study projects: a bibliometrics approach. library research, 1(3), 223-236. houston, w. (1983). the application of bibliometrics to veterinary science primary literature. quarterly bulletin of the international association of agricultural librarians and documentalists, 28(1), 6-13. hersberger, j., & demas, c. (2001). the current state of public library research in select peer reviewed journals: 1996-2000. north carolina libraries, 59(1), 10-14. retrieved 20 aug. 2012 from http://www.ncl.ecu.edu/index.php/nc l/article/viewfile/288/306 powell and connaway pulled together applications of bibliographic research outlined by a number of authors into the following list: 1. improving the bibliographic control of a literature 2. identifying a core literature, especially journals 3. classifying a literature 4. tracing the spread of ideas and growth of a literature 5. designing more economic information systems and networks 6. improving the efficiency of information handling services 7. predicting publishing trends 8. describing patterns of book use by patrons 9. developing and evaluating library collections (2004, p. 63) bibliometrics can also be embraced by librarians as a core service relating to the scholarly communication process. by honing skills in bibliometric analysis, librarians can develop expertise which “can be used to help [their] faculty prove their scholarly contribution and achieve success in their tenure process” (kear & colbert-lewis, 2011, p. 470). as always, this column can only give a minimum overview of any research method, and immersion into the method is necessary when undertaking the research. here are some resources that can get you started on your bibliometric journey: andrés, a. (2009). measuring academic research: how to undertake a bibliometric study. oxford: chandos publishing. de bellis, n. (2009). bibliometrics and citation analysis: from the science citation index to cybermetrics. lanham, md: scarecrow press. thelwall, m. a. (2009). introduction to webometrics: quantitative web research for the social sciences. san rafael, ca: morgan & claypool publishers. wolfram, d. (2003). applied informetrics for information retrieval research. westport, ct: libraries unlimited. it seems like a logical progression to go from the tried and true method of bibliometrics to a newer and quicker type of analysis called altmetrics, so that’s what i’ll look at next time. and if you have any comments, questions, or suggestions for this column, please comment on this article by signing in to the open journal system as a reader and joining in the conversation. http://search.proquest.com/lisa/indexingvolumeissuelinkhandler/23477/library+research/01979y10y01$23oct+1979$3b++vol.+1+$283$29/1/3?accountid=147 evidence based library and information practice 2012, 7.3 123 references beck, s. e., & manuel, k. (2008). practical research methods for librarians and information professionals. new york, ny: neal-schuman. kear, r., & colbert-lewis, d. (2011) citation searching and bibliometric measures: resources for ranking and tracking. college and research library news, 72(8), 470-474. retrieved 21 aug. 2012 from http://crln.acrl.org/content/72/8/470.ful l.pdf+html powell, r. r., & connaway, l. s. (2004). basic research methods for librarians (4th ed.).westport, ct: libraries unlimited. thelwell, m. (2008). bibliometrics to webometrics. journal of information science, 34(4), 605-621. doi:10.1177/0165551507087238 bibliometrics. (2012). in oxford english dictionary. retrieved 21 aug. 2012 from www.oed.com/view/entry/241665. http://crln.acrl.org/content/72/8/470.full.pdf+html http://crln.acrl.org/content/72/8/470.full.pdf+html http://www.oed.com/view/entry/241665 microsoft word news_eblig.doc evidence based library and information practice 2006, 1:1  137    evidence based library and information practice     news    eblig is up and running – jump on board!      virginia wilson, shirp coordinator, health sciences library  university of saskatchewan  saskatoon, sk, canada  e‐mail: virginia.wilson@usask.ca    lyn currie, head, education library  university of saskatchewan  saskatoon, sk, canada  e‐mail: lyn.currie@usask.ca      © 2006 wilson and currie. this is an open access article distributed under the terms of the creative  commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    after a flurry of e‐mail activity early in 2005,  which included expressing interest in a new  canadian library association (cla) interest  group and signing a petition to that effect,  the evidence‐based librarianship interest  group (eblig) held its inaugural meeting at  the cla conference in calgary in june of  2005. the breakfast discussion covered the  mission and mandate of the new group,  establishing a listserv, the selection of a  convener, a proposal for a new ebl journal  and a host of different issues surrounding  evidence based practice in librarianship.     at that first meeting hosted by su cleyle,  the following terms of reference for eblig  were discussed and approved:    1. to represent the interests of  librarians involved in evidence‐ based librarianship and library‐ related research.     2. to organize continuing education  opportunities for librarians in this  area.     3. to provide a means of  communication between librarians  involved in evidence‐based  librarianship.    4. to assist librarians with the  dissemination of library research  utilizing an evidence‐based model.     5. to govern an evidence‐based  librarianship journal called evidence  based library and information practice  by establishing the journalʹs guiding  principles and ensuring that the  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:1  138    journalʹs editorial board is in place  on an annual basis.    after a call for nominations on the eblig  listserv, lyn currie and virginia wilson,  both of the university of saskatchewan,  were acclaimed co‐conveners of the interest  group for a two‐year term.    activities of eblig to date have included:    • establishment of the eblig listserv  hosted by memorial university of  newfoundland    • creation of an editorial board for  the new open access journal  evidence based library and  information practice (eblip)  comprising su cleyle and lindsay  glynn at memorial university of  newfoundland, and denise  koufogiannakis and pam ryan at  the university of alberta    • commissioning of a logo for eblig  and the new journal hosted by the  university of alberta     • sponsorship of the 3rd international  evidence based librarianship  conference in brisbane, australia  october 2005    • distribution of postcards,  announcing and promoting eblip   at the brisbane conference, creating  a buzz about the canadian  contribution to ebl, the new journal  and the interest group  • webcast presentation by su cleyle  on ebl in canada for the cla  continuing education series in  october, 2005    • successful submission to cla 2006  to run a panel session in the  leadership track on good leaders  use the evidence: implementing  evidence‐based practice in libraries  with denise koufogiannakis and  pam ryan of the university of  alberta, and margaret haines of  carleton university as the featured  speakers    • invitation to submit a workshop  proposal for the 4th international  evidence based librarianship  conference in charlotte, north  carolina  in 2007    the eblig listserv functions as an online  meeting place for librarians interested in  ebl. open to both interest group members  and non‐members, the listserv sees postings  from canadian, british, american, and  australian librarians. it also serves the  invaluable purpose of allowing eblig  members to get together to discuss issues  and ideas when physical distance does not  allow for frequent meetings.   to join this  discussion list and to receive all postings,  send a ʺsubscribe ebligʺ message to  listserv@morgan.ucs.mun.ca.    eblig is proud to be a sponsor and  supporter of evidence based library and  information practice. we congratulate the  energetic editorial team on this evidence‐ filled first issue.  ebl 101   riding into the sunset   virginia wilson director, centre for evidence based library and information practice (c-eblip) university library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   originally published in: evidence based library and information practice, 10(2), 174–175. https://ejournals.library.ualberta.ca/index.php/eblip/article/view/24631/18435     received: 02 may 2015  accepted: 13 may 2015      2016 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   time flies when you’re having fun and when you’re writing the ebl 101 column! and with that pithy sentiment, i’m pleased to let you know that this is the final ebl 101 column in evidence based library and information practice (eblip). i’m pleased because ebl 101 has had an exciting run spanning eight years! it has become a go-to resource for those beginning to think about evidence based library and information practice (eblip) and for overviews of research methods. if the citations of the research methods columns are any indication, scholars from within library and information studies (lis) and without have found value in the overviews. i’m also pleased for another reason that i’ll get to in a moment.   i started writing the ebl 101 column in 2009 (vol 4, no 1), continuing the steps of eblip.   when those steps were finished, i moved on to writing concise overviews of research methods, including examples of the methods from the lis literature and resources for further information as there’s only so much one can do in 2 or 3 pages of a column. here’s an overview of all the columns as a whole. aside from the first two ebl 101 columns, the rest where written by me: ebl 101: steps of eblip   ·         vol. 3, no. 3 (2008) evidence based librarianship backgrounder – su cleyle, julie mckenna ·         vol. 3, no. 4 (2008) asking the right question – lorie kloda ·         vol. 4, no. 1 (2009) matching question types to study designs ·         vol. 4, no. 2 (2009) looking to the literature: domains to help determine where to look ·         vol. 4, no. 3 (2009) looking to the literature: open access and free sources of lis evidence ·         vol. 4, no. 4 (2009) conducting your own research: something to consider ·         vol. 5, no. 1 (2010) an introduction to critical appraisal ·         vol. 5, no. 2 (2010) applicability: what is it? how do you find it? ·         vol. 5, no. 3 (2010) evaluating the results of evidence application, part one ·         vol. 5, no. 4 (2010) evaluating the results of evidence application, part two: at the practice level ·         vol. 6, no. 1 (2011) disseminating your research ebl 101: research methods   ·         vol. 6, no. 2 (2011) a new path: research methods ·         vol. 6, no. 3 (2011) design, methods, case study...oh my! ·         vol. 6, no. 4 (2011) content analysis ·         vol. 7, no. 1 (2012) focus groups ·         vol. 7, no. 2 (2012) interviews ·         vol. 7, no. 3 (2012) bibliometrics ·         vol. 8, no. 1 (2013) altmetrics ·         vol. 8, no. 2 (2013) mixed methods research ·         vol. 8, no. 3 (2013) systematic reviews ·         vol. 8, no. 4 (2013) action research ·         vol. 9, no. 1 (2014) triangulation ·         vol. 9, no. 2 (2014) sampling ·         vol. 9, no. 3 (2014) the most significant change technique ·         vol. 9, no. 4 (2014) scoping studies   quite the list! there are plans in the works for a "virtual issue" that would bring all the ebl 101 columns together in one place for readers and we’ll keep you posted about that. also (and this is the part i’m really pleased about), on the horizon and taking ebl 101’s place will be a new column that i’m very excited to begin research in practice. this column will be designed for practicing librarians who are using research in their practice, conducting research for their practice, and otherwise interested in the varied and vast topics pertaining to research in practice. the scope of the new column is broad, allowing for a variety of topics to be explored in a number of ways. there could be interviews, guest columnists, co-authored columns...it boggles the mind!   i would like to thank my past editors of the ebl 101 column, denise koufogiannakis and alison brettle, for their patience, help, and guidance, and for the opportunity to explore my thoughts in a column format. and thank you to lorie kloda, eblip’s new editor-in-chief, for being open to the idea of reconfiguring the column. eblip’s editorial board has had some exciting ideas for the column and i look forward to continuing in this capacity with the journal. look for research in practice in the next issue of eblip.   article   eblip and active learning: a case study   helen buckley woods information specialist information resources group health economics and decision science scharr university of sheffield sheffield, united kingdom email: h.b.woods@sheffield.ac.uk   received: 20 dec. 2012   accepted: 13 may 2013      2013 woods. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to determine how librarians use evidence when planning a teaching or training session, what types of evidence they use and what the barriers are to using this evidence. the case study also sought to determine if active learning techniques help overcome the barriers to using evidence in this context.   methods – five librarians participated in a continuing education course (cec) which used active learning methods (e.g. peer teaching) and worked with a number of texts which explored different aspects of teaching and learning. participants reflected on the course content and methods and gave group feedback to the facilitator which was recorded. at the end of the course participants answered a short questionnaire about their use of educational theory and other evidence in their planning work.   results – findings of this case study confirm the existence of several barriers to evidence based user instruction previously identified from the literature. amongst the barriers reported were the lack of suitable material pertaining to specific learner groups, material in the wrong format, difficulty in accessing educational research material and a lack of time. participants gave positive feedback about the usefulness of the active learning methods used in the cec and the use of peer teaching demonstrated that learning had taken place. participants worked with significant amounts of theoretical material in a short space of time and discussion and ideas were stimulated.   conclusions – barriers to engaging with evidence when preparing to teach may be addressed by provision of protected time to explore evidence in an active manner. implementation would require organisational support, including recognition that working with research evidence is beneficial to practice.   introduction   library and information professionals, working in a variety of settings, invariably have to undertake some kind of teaching, whether it is called user instruction, training, lecturing or other variants of the term. differing approaches are taken to planning teaching sessions and differing levels of experience and teaching qualifications exist in the library and information science (lis) community (julien & genuis, 2011). although there is considerable literature on librarians as teachers, there is little research which investigates librarians’ use of evidence in preparing teaching or “evidence based teaching.” questions arise such as what kinds of evidence lis professionals use or whether there are particular barriers to using evidence that relate to the teaching role.   inspired by constructivist theories of learning and active learning techniques (for example, peer teaching) the author was keen to learn more about the relationship between health librarians and their use of evidence when preparing to teach. the jigsaw method of teaching that utilises group work and other non-passive approaches to teaching was used. this method is based on the “jigsaw classroom” (aronson, 2000). the author hypothesized that unless interaction with evidence is active then it will have little effect on the teaching process (coomarasamy & khan, 2004). she opportunistically used an international workshop at which she was a facilitator, to explore such a hypothesis.   literature review   this project examines three interlinked aspects of library and information practice. to situate the case study within an appropriate context a selective literature review will focus on these key areas: evidence based library and information practice, information literacy instruction, and active learning methods. it is important to review key evidence based library and information practice (eblip) literature in order to define terminology, examine what is meant by “evidence” and identify what the barriers to eblip are, particularly with reference to a librarian’s teaching role. this case study is about librarians as teachers and as such it is important to define what is meant by “information literacy” and how practitioners engage with evidence in order to prepare to teach. some explanation of the area of active learning is given as a precursor to the methods used in this study.   evidence based library and information practice   evidence based library and information practice is a way of working which evolved from evidence based medicine. health librarians have been the initial and enduring champions of eblip playing a central role in evidence based medicine in particular in the development of systematic reviews. subsequently they soon became aware that they could take an evidence based approach to their own work. the lis community was not alone in this “nagging awareness” (booth & brice, 2004, p.7) with other professionals, for example in social care and education, also engaging with evidence based practice.   in its broadest sense, eblip involves practitioners engaging with research to inform their practice. anne mckibbon provided a focussed definition of eblip, which was the developed by booth (booth & brice, 2004) into an understanding that   “evidence-based librarianship (ebl) is an approach to information science that promotes the collection, interpretation and integration of valid, important and applicable user-reported, librarian observed, and research-derived evidence. the best available evidence, moderated by user needs and preferences, is applied to improve the quality of professional judgements” (p.7). this definition mirrors sackett’s (2000) definition of evidence based medicine (ebm) “[the] integration of best research evidence with clinical expertise and patient values” (p.1), and draws together the three elements of evidenced based practice (ebp): evidence, knowledge and experience of the practitioner, and what is appropriate for the particular client/user/patient in question. a key characteristic of booth’s definition is the way evidence is defined, in that it includes evidence from service users and practitioner observations. booth and brice (2004) also talk about a common misunderstanding about eblip and the library literature, with practitioners believing that they have to apply a traditional hierarchy of evidence in eblip where this is often not appropriate for the types of questions asked in library and information science (p. 9). the phrase “the best available evidence” refers to this aspect of eblip. the use of what kelly et al. (2010) call “epistemologically and methodologically diverse evidence” (p. 1059) can be seen in professional areas much closer to the original ebm movement, such as systematic reviews to answer complex public health questions.   koufogiannakis (2011) considers whether, as suggested in booth’s definition of eblip, the use of evidence is pluralistic and embraces evidence other than research derived evidence. she argues that in practice it does not, with the emphasis placed on research evidence and within that a focus on particular types of research evidence as demonstrated in the traditional hierarchy. she states “anything other than positivistic, scientific evidence has been demoted...the notion of a hierarchy and what is represented therein, is so far removed from the reality of practitioners’ experiences and what is valuable to them” (p.43). this raises two key points: firstly, the focus of eblip has centred on the “research evidence” aspect of the three elements, and that the local context and the practitioner’s knowledge and experience have been wrongly undervalued and under promoted. the second point is that within the research element aspect of eblip, qualitative research is not being given its rightful value as high-quality formalised research and this is a cultural problem which needs to be addressed. the message is clear that all elements of the eblip model are important and all types of evidence need to be engaged to arrive at the best decisions in practice.   as part of eblip, librarians will have questions to answer about user instruction, such as “what is the best way to structure my small group sessions on critical appraisal with post graduate students”, “what is the best way to conduct an induction lecture to 200+ new users?”, and “how do i engage my year 7 boys with the fiction collection in our weekly supervised reading session?” to answer such questions librarians face the challenge of engaging with a variety of types of research evidence from multiple disciplines aimed at different audiences with different purposes.   these questions may also be answered by focussing on what koufogiannakis (2011) calls local evidence (e.g., evaluation forms from teaching sessions) and professional knowledge (e.g., in a practical session it becomes clear to the facilitator that some students need extra support with their it skills in order to participate fully). the literature beyond that of eblip will now be considered to contextualise this case study and reflect how information professionals fulfil the need for evidence to improve their practice in this area.   information literacy   “information literacy lies at the core of lifelong learning. it empowers people in all walks of life to seek, evaluate, use and create information effectively to achieve their personal, social, occupational and educational goals. it is a basic human right in a digital world and promotes social inclusion of all nations” (high-level colloquium on information literacy and lifelong learning, 2005, p.3.).   this text, taken from the “alexandria proclamation” written by international information literacy (il) stakeholders, provides an indication of how far reaching the concept of information literacy has become in many contexts across the world. through this proclamation and the work of the earlier expert meeting in prague in 2003, il was linked to concepts such as the knowledge economy and the information society. a humanitarian element may also be clearly perceived within the il movement with the concept linked to, and practice developed under, the banner of “lifelong learning” and “social inclusion” (webber & johnston, 2003).   the main focus of activity in il practice is within the education sphere, particularly within higher education (he). in he, library and information professionals and faculty staff are engaged with teaching students information skills. library staff also train researchers to make the best use of the academic resources they need. due to the high level of activity in he, including numerous papers on best practice in teaching information skills and “how to” books, this focus on information skills is sometimes confused with the wider concept of il. lloyd (2010) examines the concept of il in various areas, defining it as more than a “textual practice, as we commonly conceive it to be in the library or educational setting” (p. xvi). she describes it as a social practice which involves people reaching shared understandings and engaging with information in particular cultural environments.   in truth, library and information science practitioners are focussed on providing for the needs of their user communities, whether this is to equip undergraduates to complete their degree, to help a member of the public access travel information on the internet, or to support a researcher seeking evidence for a systematic review. despite the growing eblip movement, which advocates the use of research evidence by practitioners, there is a divide between the lis academic community’s research into il (which provides the theoretical framework) and the needs of practitioners in the lis community. this divide was highlighted by eve and shenk (2006), in their qualitative study about the impact and use of research by lis practitioners. they found that typically practitioners are reluctant to engage with research material and have little time to undertake their own research or publish examples of best practice. conversely, researchers are perceived to be guilty of a lack of engagement with the practitioner community and of limited dissemination of their work. the research in librarianship – impact evaluation study (rilies) report found “a disconnect between lis research and the practitioner community” (cruickshank, hall & taylor-smith, 2012, p.4). suggested recommendations to ameliorate this included: involving practitioners in research projects at an early stage, making research evidence more accessible and practitioners, employers and the uk professional body offering support and encouragement for engagement with research.   librarians as teachers   a plethora of papers examine all aspects of the teaching role that information professionals inhabit. two recent papers examined this aspect of librarian’s work and consider how librarians prepare for teaching. in a national survey of canadian librarians involved in information literacy training, julien and genuis (2011) explored librarians’ experiences of the teaching role. as part of this survey participants were asked how they prepared for teaching activities, including mention of a specific category of reading professional or research literature; 75.1% of respondents used professional reading as a method of preparing for their instructional role. these researchers found a beneficial effect for those respondents who undertook professional reading in that they were more likely to feel prepared for teaching work and were more comfortable with the teaching aspect of their role (p.107). participants were given the opportunity to provide additional open comments; and particular interpersonal challenges were noted by the researchers. for example, participants described their lack of knowledge of educational theory, teaching methods, and curriculum design. only 39.7% of those surveyed had received any formal training in teaching (p.106).   bewick and corrall (2010), focussed on u.k. subject librarian’s acquisition and use of pedagogical knowledge. their findings concurred with those of julien and genuis, (2011) with participants’ primarily acquiring knowledge informally, through on the job training and trial and error. less than a third of respondents had undertaken an extended course or training programme, but more than half had participated in a short course to improve teaching knowledge and skills. a minority of respondents mentioned personal reading as a method of gathering pedagogical knowledge, but the exact number is unknown as this was reported within a wider category of “other methods”. the authors report that most participants “felt confident about teaching and thought their knowledge sufficient” (2010, p.97).   librarians then are faced with a vast array of different types of evidence to help them prepare for information literacy teaching and there is evidence that there is a general lack of engagement with research by practitioners both in terms of research consumption or production. however, professional reading has been cited by practitioners as a preferred way of preparing for teaching and it has been found to have a beneficial effect in how they felt about teaching, (julien & genuis, 2011). this paper will now move on to look in more depth at the barriers to engaging with evidence and the use of active learning as a method to engage with research.   barriers to eblip   eve and shenk (2006) research anticipated barriers to eblip later cited in the literature review by booth (2011). booth’s review identified seventeen themes, including lack of time, poor access to the evidence base, lack of organizational support, and inappropriate orientation of research. a study by turner (2002, as cited in booth, 2011) is examined in more detail with “time constraints” being cited as the number one barrier to practitioners consulting the research literature. booth unpacks this barrier and suggests two concomitant parts: what the time is needed for (e.g., to acquire, appraise and apply research), and the reasons why the time is lacking (e.g. workload and management responsibilities). while acknowledging “the competing horizons of short-term deliverables and longer-term professional development” (p. 11), also implicit within this tension is the prioritisation of eblip and whether motivation is lacking to engage with research evidence. indeed booth states, “there remains an ongoing need for more success stories so that practitioners can realise the value of the evidence-based approach, particularly when ranged against more established alternatives for library decision making and planning” (p.15). the review examined barriers and facilitators to using evidence in all aspects of library and information practice, so booth’s findings can be applied to the use of evidence in user-education, but due to the generic application there were few examples specifically related to information literacy support.   active learning as eblip method   active learning refers to a learner-centred approach to teaching that is based on constructivist theories of learning, developed most prominently by the work of bruner who posited that learners are active participants in any learning situation; they work with new information to process and construct new meanings. bruner also highlighted the importance of social interaction in the learning process (culatta, 2013). the constructivist theory of learning and its practical applications are succinctly encapsulated by chickering and gamson.   “learning is not a spectator sport. students do not learn much just by sitting in classes listening to teachers, memorizing pre-packaged assignments, and spitting out answers. they must talk about what they are learning, write about it, relate it to past experiences, apply it to their daily lives. they must make what they learn part of themselves” (chickering & gamson, 1987, p.4).   when using a constructivist approach, learners also need to engage in higher order thinking tasks such as analysis, synthesis, and evaluation (bonwell & eison, 1991). bloom’s taxonomy provides a useful theoretical underpinning for the jigsaw method used in this study. bloom classifies three different aspects of learning; the cognitive domain, which is about the use of knowledge, the affective domain, which is about the use of values (for example listening and respecting other’s views in the classroom, (clarke, 2004) and the psycho-motor domain which is about skill development. they are sequential in their use, so for example it is not possible to synthesise facts until you know what those facts are. it is also possible to use the different domains in order to build up the levels over a period of time for example throughout a module or course (atherton, 2011). so far so good, but is there any evidence that librarians use active learning methods themselves in their own preparation for their teaching role?   young and vielle (2011) report that librarians use multiple methods to keep up to date with current research, policy and practice, for example by accessing social media, using email lists, as well as personal reading and attendance at training days, conferences and other continuing professional development (cpd) activities. another approach to keep research part of practice is to introduce a discussion group around particular journal articles or professional materials. such discussions can happen within an existing meeting or, in a more formalised way, can take the form of a journal club thereby mirroring medical professional groups that follow the “five steps of evidence based practice” (centre for evidence based medicine, 2009). it is within this method of professional development that we see an opportunity for practitioners to engage with active learning.   several authors report case studies and reviews of journal clubs and discussion groups for lis practitioners (haglund & herron, 2008; young & vilelle, 2011; kraemer, 2006; pearce-smith, 2006). the wider medical literature reports a variety of different methods employed under the banner of “journal club”. a recent review (harris et al., 2011) found that common elements in journal clubs in medical settings included “using principles of adult learning” which was defined as “promoting active learner participation” (2011, p.38). another common element was “using multifaceted approaches to teaching and learning” (2011, p.38) which are basically active learning methods e.g. small group discussion. they conclude that one element of a successful journal club is to design the club using principles of adult learning, however they report that:   “adult learning was an ingredient in some studies across all areas, but the influence of adult learning is difficult to assess because it may have been underreported when the educational intervention was described. this may also be the case with multifaceted approaches to teaching and learning…” (2011, p.9).   this selected review of the literature has explored different areas of research in order to contextualise and inform the aims of the case study presented. to summarise there is little literature on how librarians use or don’t use evidence (including practice based evidence, secondary sources and primary research) to prepare for teaching; although there is well-documented evidence to suggest that there is an overall research practice gap. professional reading and informal “on the job” learning are reported as preferred methods of preparing for teaching (julien & genuis, 2011; bewick & corrall, 2010). key barriers to eblip are a lack of time and opportunity, but specific barriers which pertain particularly to “evidence based il instruction” are unknown. there is some evidence to support the effectiveness of active learning as an eblip method.   aims   this study investigates the following research questions:   how do librarians use evidence when planning a teaching or training session? what types of evidence do librarians use when planning a teaching or training session? what are the barriers to using evidence when preparing for a teaching session? do active learning techniques help overcome barriers to eblip and therefore improve teaching practice?   methods   to investigate the relationship between active learning and eblip the author conducted a continuing education course (cec) at the european association for health information and libraries (eahil) 2011 workshop, “active learning and research partners in health,” held at koç university, istanbul, turkey.  as this small case study was practitioner led action research it was not necessary to obtain ethics approval, however participants granted permission to use their feedback and comments in a publication or presentation about the cec. an example of the consent form can be seen in appendix a.   the case study took the form of a cec with five participants. individuals were required to read journal articles or book chapters before the session and worked with these texts to discuss, teach, and reflect on ideas and concepts elicited from them on the day of the workshop. the author facilitated the cec, presented on the concept of active learning, and led the group-work that included feedback and reflection on the cec method and the content of the texts. the session closed with a focused discussion providing feedback regarding the method. this conversation was recorded, transcribed, and analysed. please see appendix b for the questions used to lead this discussion. immediately after the close of the cec participants answered a short questionnaire about their use of educational theory and evidence in their planning work, shown below as appendix c.   the case study focused on active learning and used an adaptation of an instructional method, originally employed with school children called, the ‘jigsaw classroom’ (aronson, 2000). the jigsaw method describes a method of working with small groups and is designed to be participatory for learners and to encourage deep learning (fry, ketteridge, & marshall, 2003). it is recognised to develop the learner’s cognitive and affective domains through its collaborative nature (wong & driscoll 2008; robledo-rella, neri, & noguez, 2010). teachers can select from a variety of collaborative methods when working with small groups, such as discussion, answering/asking questions, and peer teaching. the jigsaw method is very rich in that it incorporates a number of methods and develops a number of skills within one exercise. it maximises potential learning (bloom’s cognitive domain) but also encourages and develops other skills in the learners, such as presentation skills, listening skills, organisation skills, and time management skill (the affective domain), (chapman, 2009). this multifaceted learning method enables the teacher to provide an opportunity for all the learners in their group to be engaged at some point in the exercise. this is achieved by incorporating diverse aspects and demands on the students that require different skills and engage different learning styles.   a jigsaw exercise enables participants to assimilate significant amounts of new information in a short space of time by engaging with such material using a variety of active methods. the basic structure of the jigsaw exercise can be adapted to different numbers of participants or to different settings. in the cec, the session began with participants sitting together where they chose, but each person had been previously allocated to a different “expert group” reflecting the material they had. as there were only five participants in total, the participants were assigned to a group of three and a group of two. each small group worked on two topics.   after initial introductions and setting the context and purpose of the cec, including a reflective exercise, the jigsaw then began with delegates having time to review the materials they had read individually before moving to their “expert groups”. each group discussed the material they had read and then planned how they would teach this material to other delegates. participants then paired with someone from the other “expert group” and taught their peers about what they had learned by delivering a short presentation.   the author selected reading materials for the session that fulfilled several criteria. the aim was to use diverse materials in terms of type of material and its intended readership, and it was important to mix content to include both original research and summaries or syntheses of theory, in order to use a broad range of evidence. four distinct groups in the jigsaw exercise were assigned materials indicated on these topics: planning learning (webb & powis, 2004); working with small groups (jaques, 2003; kaufman, 2003); working with large groups (cantillon, 2003; verlander & scutt, 2009); and motivating learners (newstead & hoskings, 2003). of these, three short papers were taken from the british medical journal series “abc of learning and teaching in medicine” (cantillon, 2003; jaques, 2003; kaufman, 2003) and gave a good introduction to the topics of teaching small groups, large groups and applying educational theory in practice. the bmj papers were examples of pithy writing that provide a topic summary and are underpinned by theory. such articles can be useful materials to introduce into this type of training session, as they provide a good introduction to a broader theoretical landscape that can be further explored at a later date. it is not always necessary to use papers which can be straightforwardly applied to practice or papers which are aimed specifically at an lis audience. concepts and ideas from other disciplines can spark ideas for use in lis practice. articles from other disciplines can also provide a further intellectual challenge and enjoyment for the reader. however, articles should not be chosen for novelty value, and there must be a core of meaning which is relevant to the lis practitioner.   the author also selected two book chapters. one chapter was aimed at higher education practitioners and discussed learner motivation (newstead & hoskings, 2003). this chapter provided a broad overview of the key theories and concepts related to student motivation, with many links to further reading. as it was aimed at an he audience, it had more breadth than a text aimed at il practitioners. the other chapter, by webb and powis (2004), was infused with educational research. this book is aimed at an lis practitioner audience and was also a good example of how to introduce some pedagogical theory into a discussion at an appropriate level, and therefore complementary to the newstead and hoskings chapter. the final reading was verlander and scutt’s (2009) article wherein the link between evidence and implementation in practice was clearer to envisage for the reader as the research was practitioner based. together, these diverse readings served to raise participant awareness of the kinds of materials that were available, or reminded participants of the types of suitable sources to inform their teaching roles.   after the jigsaw exercise, the group reformed as one, and participants were then invited to take a few moments to reflect individually on the technique. participants then shared comments with one another regarding the effectiveness of the method and the likelihood they could utilise the method or an adaptation of the method in their own teaching. they were also asked to describe how it felt to be a participant and whether they had identified any particular messages or points for discussion from the literature used in the session. the resulting group feedback was recorded with the participants’ permission. after this the session was brought to a close and participants completed a short questionnaire about the use of evidence and educational theory in the preparation of teaching sessions. the group feedback was transcribed and the author used this transcription and the recording to organize the responses into simple categories that reflected the four research questions stated above.   demographically, the participants represented from five countries: cyprus, norway, slovenia, poland, and australia. participants served as mid-career health information professionals and service managers at both academic institutions and health care settings.   results   how do librarians use evidence when planning a teaching or training session? what types of evidence do librarians use when planning a teaching or training session?   some participants reported that they did use articles or papers when preparing to undertake a teaching session, but they felt unable to pursue extensive research. one respondent noted: “i use these articles sometimes as i have many different tasks to do and didn’t have time to go more in depth in this matter.” participants also reported using evaluation questionnaires and materials to support a specific project, such as creating a web-based tutorial. current awareness services were also cited as a method of keeping up to date on all professional areas. a number of participants commented that the session had raised their awareness of materials and ideas that could inform their information literacy work which they would investigate after the session.   participants described superficial attempts to incorporate educational theories in planning learning. some respondents reported using techniques that were derived from a theoretical base, but did not report a familiarity or comfortable use of theory. one respondent reported increased awareness of using educational theory in practice but suggested that preparing teaching materials was given priority due to time constraints:   “i should put more attention for using theories of learning in teaching. i have concentrated more on learning material because lack of time and engagement in different activities.”   an experienced professional reported that they had knowledge of a constructivist approach to learning but that in practice it was difficult to keep sessions true to this methodology, noting   “i try to have a constructivist approach but it is easy to slip back into old-fashioned lecturing. the need to be in control … [a] lack of confidence in using active learning techniques.”   what are the barriers to using evidence when preparing for a teaching session?   participants presented several barriers to using evidence to prepare for teaching via their questionnaire responses. several issues related to the material itself, and a lack of suitable material pertinent to the particular characteristics of the students to be taught was felt to be a barrier. the level of material available was also felt to be a barrier, as review articles would be more useful to read than primary studies. difficulty in locating articles was also cited as a barrier, which may seem unusual coming from information professionals, but working in a health information environment meant that some participants were unable to access non-medical databases. one participant noted:   “a barrier to finding evidence in articles is locating them – having access to a suitable database, we only have medical ones e.g. medline, cinahl and psychinfo, so articles would be already medical teaching orientated.”   participants also reported the challenge of understanding educational research articles when coming from a librarianship background, and of not having undertaken a teaching qualification. more than one respondent reported lack of time as a barrier, particularly in the face of competing priorities. one respondent suggested the need for high quality review articles as a solution to this barrier. a lack of confidence to utilise teaching methods reported in the research literature was another barrier to eblip.   do active learning techniques help overcome barriers to eblip and therefore improve teaching practice?   feedback about the methods used was very positive, although participants noted that it was challenging and quite demanding, with one delegate reporting that it “certainly took me out of my comfort zone” but that “it was good, it engages you straight away”. the role of the facilitator was noted as key to the success of the method in terms of time management and creating a comfortable atmosphere for the group. “i think that is one of the roles of the facilitator to make the learning space comfortable and people willing to come forward rather than staying back.” the same delegate reflected that the whole group had been disciplined with the time management and also committed to the spirit of the exercise, being enthusiastic and respecting each other’s views:   i must say i felt very secure in this group. i mean i know x, i know you from before, but even so i don’t know either of you and i think everybody is very pleasant and kind to each other. you know there are no problems with the group. that makes it feel more secure and be prepared to open up more.   the peer teaching element of the exercise was felt to be a useful part of the process, with one delegate commenting: “i think it was effective… if i read it and then i have to talk about this, to pass my knowledge is more effective than just reading.” this is the essence of why active learning is a useful technique, to “teach” the article, even in terms of a brief summary, the participant needs to understand the material better than they otherwise would by just reading it.   the group feedback elicited several comments regarding the significant amount of material covered in the session: “in a way we have learnt or we have read six articles when in fact we have only actually had to read three each” and “we have sort of got the sort of summary of three more.” one participant commented that it was reassuring that they were “all in the same boat” another noted “and also the realisation that you are then going to listen to your colleagues who are equally not familiar with the topic is good.”  the session also piqued the interest of one delegate to investigate the papers they hadn’t read in more detail: “and i also feel that i am now curious. i would like to read the other three articles.”   participants reflected on how they could adapt the method and whether it would work in their own work teaching situations. the session gave participants insights into how it felt to be a learner again and what kinds of activities were useful to learners. one participant identified on the peer teaching aspect and how they could use this with their groups:   “well i thought i could adapt it and therefore using different strategies for the group. i teach small groups. it is different strategies to get them involved instead of me talking and then them doing little exercises. i could talk and then they perhaps could do a little presentation back and then maybe that’s a way of learning.”   another delegate thought the method could be usefully applied to staff training at their institution:   “because we are on four sites and we don’t get together very much and we actually use a skype type system … i think you could even adapt it to that…we have different channels, so you could say you go to that channel and talk about that and then people come together again in the main channel.”   the delegate commented that this approach would work well and be a welcome alternative to sitting “passively with earphones on and you just fall asleep.”   the group discussed how well the reading material had been chosen to reflect key areas of interest and make the best use of the time available. one participant remarked:   “i think that this material was really very carefully chosen and that i can get some main themes, large groups, small groups, various situations. so maybe i know these things but this session we get the themes very concentrated.”   another delegate commented that engaging with the material had enabled them to “think more… deep[ly], about certain topics, so then i think it is easier to understand our learners.”   the group also felt that the material had given them insights into both their own practices and that of their learners. for example, one person commented on the usefulness of doing an “audit” as described in the webb and powis (2004) chapter. the group also discussed the issue of learner motivation that had been the focus of one of the expert groups. they felt that this gave them a useful insight into what was going on beneath the surface with their own learners.   discussion   findings of this case study confirm the existence of several barriers to evidence based user instruction identified previously (booth, 2011; eve & shenk, 2006). respondents also cited specific examples of these barriers, including a lack of suitable material pertaining to their specific learner group, the lack of review level material, and a difficulty in accessing educational research material, while overall a lack of time was the main barrier reported. the lack of pedagogical training for librarians was also reported and expressed as a difficulty in interpreting and understanding particular forms of evidence. some of these barriers to eblip (regarding materials) could be overcome in a workshop, journal club, or by the use of a facilitator responsible for choosing and circulating appropriate materials before discussion sessions.   the group feedback on the usefulness of the jigsaw learning method was positive. the group felt they had learned a significant amount, both about theory and teaching practice, from the material itself and also from the meta-learning aspect of reflecting on and experiencing the method in practice. they also reported an emotional element to the experience, such as feeling what it was like to be a learner again, feeling actively engaged with the session, and a sense of camaraderie with the other participants. such elements made the experience richer and more likely to resonate and remain with participants, as they had to give something of themselves to participate in the session. the positive comments from participants about the material and how they would use the ideas in practice concur with julien and genuis’ (2011) findings that reading had a positive effect on a librarian’s role as teacher.   two key barriers cited in the literature were lack of time and also lack of motivation to undertake eblip activities. using active learning methods increased understanding and made more effective use of the time available. such methods also led easily into a discussion on the relationship with practice both in terms of methods and materials that had been covered. this gave participants the opportunity to consider the local context and professional experience elements of the eblip model. participating in this type of session necessitates an investment and therefore rewards participants with a feeling of achievement. it also effectively creates or strengthens the connection between group participants through shared experience. often some humour is generated at the end of the session quite naturally, when participants reflect on the experience. these components all help to create a positive experience and therefore one which participants would be more likely to repeat. this is especially beneficial when practitioners are labouring under a heavy workload and need additional motivation to make time for eblip activities. the chosen method needs to maximise benefit to participants and their organisation.   limitations   this case study conforms to the mode of action research (stenhouse, 1981), an exploratory investigation by a practitioner. the group was small, with only five participants, and as such the findings from this study cannot be generalised. however the theories and ideas generated can be used when working with larger groups and can be further developed. the investigation was confined to the original workshop setting, as there was no follow up with participants, and therefore no further data was collected on the links between the workshop and participants’ subsequent practice. however, the initial reflections related to potential impact on practice were collected during the session.   conclusion   information professionals have a wide variety of evidence to choose from to assist in preparation for their role as information literacy practitioners. respondents reported the use of journal articles and evaluation questionnaires specifically in their preparation to teach. other materials were used for general current awareness or to undertake a specific task, such as creating a web based tutorial. there are numerous barriers to eblip documented in the literature, and respondents reported a number of barriers that concurred with those found in the literature. specific examples include the challenge of finding materials that related to a particular group of learners and the difficulty in understanding educational research papers without the benefit of a teaching qualification. this study found some evidence to support the effectiveness of active learning methods as an eblip tool. professional reading was also found to be beneficial to how librarians feel about their teaching role.   this case study provides a snapshot of common issues information professionals face in their practice, and identifies barriers to eblip which are specific to the il teaching role. in a small case study such as this, it is not possible to prove or disprove a hypothesis about the effectiveness of active learning and eblip, but the aim of exploring these themes was met. the feedback from participants endorses the evidence from the literature that active learning is a useful eblip tool. barriers to engaging with evidence when preparing to teach may be addressed by provision of protected time to explore evidence in an active manner. organisational support would be required to implement such an intervention, with a recognition that working in an evidence based way is worth pursuing, and that it will make a difference to practice.   several conclusions can be drawn from the case study. participants worked with significant amounts of theoretical material in a short space of time, and teaching one another about this material, they were able to demonstrate that some learning had taken place. through discussing ideas, they created a very positive atmosphere in the room, not a bad thing to engender at work. this type of method creates an opportunity to engage socially and individually with different types of evidence.   further research   the author is presently taking this initial experience forward within a workplace-based “active journal club,” with plans for further evaluation on the impact on practice by following up participants use of evidence after the group sessions. a larger study to compare an active journal club or discussion group such as described in this case study with a more passive format would strengthen the results of this research. this type of study would help answer the question of how effective active learning methods are in helping to overcome barriers to eblip and improve il practice.   acknowledgments   i would like to thank the workshop members for their excellent participation and hard work; we also had fun. i would like to thank the scharr public health collaborating centre who contributed to funding my place at the 2011 eahil conference. thank you to sue wilson for transcribing the recording of the session feedback. thank you to colleagues for giving valuable comments and suggestions on earlier drafts of this paper.     references   aronson, e. (2000). jigsaw classroom. retrieved 29 mar. 2013 from http://www.jigsaw.org/   atherton, j. s. (2011). learning and teaching: bloom's taxonomy. retrieved 14 dec. 2012 from http://www.learningandteaching.info/learning/bloomtax.htm/   bewick, l., & corrall, s. (2010). developing librarians as teachers: a study of their pedagogical knowledge. journal of librarianship and information science, 42(2), 97-110. doi:10.1177/0961000610361419   bonwell, c., & eison, j. (1991). active learning: creating excitement in the classroom. eric digest. washington: ashe-eric higher education reports. retrieved 20 dec. 2012 from http://www.eric.ed.gov/ericwebportal/detail?accno=ed340272   booth, a. (2011). barriers and facilitators to evidence-based library and information practice: an international perspective. perspectives in international librarianship, 1(1), 1-15. doi:10.5339/pil.2011.1   booth, a., & brice, a. (2004). evidence-based practice for information professionals: a handbook. london: facet publishing.   cantillon, p. (2003). abc of learning and teaching in medicine: teaching large groups. british medical journal, 326(7386), 437-440. doi:10.1136/bmj.326.7386.437   centre for evidence based medicine. (2009). what is ebm? retrieved 20 dec. 2012 from http://www.cebm.net/index.aspx?o=1914     chapman, a. (2009). bloom’s taxonomy: learning domains. retrieved 20 dec. 2012 from http://www.businessballs.com/bloomstaxonomyoflearningdomains.htm   chickering, a.w., & gamson, z.f. (1987). seven principles for good practice in undergraduate education. american association of higher education bulletin, 39(7), 3-7. retrieved 28 mar. 2013 from http://www.eric.ed.gov/ericwebportal/detail?accno=ed282491   clark, d. r. (2004). bloom’s taxonomy of learning domains: affective domain. retrieved 29 dec. 2012 from http://www.nwlink.com/~donclark/hrd/bloom.html     coomarasamy, a., & khan, k.s. (2004). what is the evidence that postgraduate teaching in evidence based medicine changes anything? a systematic review. british medical journal, 329(7473):1017. doi:10.1136/bmj.329.7473.1017   cruickshank, p., hall, h., & taylor-smith, e. (2012). enhancing the impact of lis research projects: rilies project report. retrieved 20 dec. 2012 from http://www.researchinfonet.org/wp-content/uploads/2012/01/rilies_report_final.pdf     culatta, r. (2013). constructivist theory (jerome bruner). retrieved 19 aug. 2013 from http://www.instructionaldesign.org/theories/constructivist.html   eve, j., & schenk, n. (2006). research and practice: findings from the interactions project. library and information research, 30(96), 36-46. retrieved 20 dec. 2012 from http://eprints.brighton.ac.uk/3006/   fry, h., ketteridge, s., & marshall, s. (eds.). (2003). a handbook for teaching and learning in higher education: enhancing academic practice (2nd ed.). london: kogan page.   haglund, l., & herron, d. (2008). implementing eblip to stimulate professional development. journal of the european association for health information and libraries, 4(3), 3. retrieved 20 dec. 2012 from http://www.eahil.net/journal/   harris j., kearley, k., heneghan c., meats e., roberts, n., & perera, r. (2011). do journal clubs promote evidence based practice? a systematic review. beme guide no. 16. medical teacher, 33(1), 9-23. retrieved 20 dec. 2012 from http://www.medicalteacher.org/   high-level colloquium on information literacy and lifelong learning. (2005). report of a meeting sponsored by the united nations education, scientific, and cultural organisation (unesco), national forum on information literacy (nfil) and the international federation of library associations and institutions (ifla). alexandria, international federation of library associations and institutions. retrieved 20 dec. 2012 from http://archive.ifla.org/iii/wsis/info-lit-for-all.htm   jaques, d. (2003). abc of learning and teaching in medicine: teaching small groups. british medical journal,326(7387),492-4. doi:10.1136/bmj.326.7387.492   julien, h., & genuis, s.k. (2011). librarians’ experiences of the teaching role: a national survey of librarians. library & information science research, 2(2), 103-111. doi:10.1016/j.lisr.2010.09.005   kaufman, d. (2003). abc of learning and teaching in medicine: applying educational theory in practice. british medical journal, 326(7387): 213-6. doi:10.1136/bmj.326.7382.213   kelly, m., morgan, a., ellis, s., younger, t., huntley, j., swann, c. (2010). evidence based public health: a review of the experience of the national institute of health and clinical excellence (nice) of developing public health guidance in england. journal of social science & medicine, 71(6), 1056-1062. doi:10.1016/j.socscimed.2010.06.032   koufogiannakis, d. (2011). considering the place of practice-based evidence within evidence based library and information practice (eblip), library & information research, 35(111), 41-58. retrieved 20 dec. 2012 from http://www.lirgjournal.org.uk/lir/ojs/index.php/lir/index   kraemer, e.w. (2006). keeping up with the journals: a library journal club at oakland university. journal of academic librarianship, 33(1), 136-137. retrieved 20 dec. 2012 from http://www.journals.elsevier.com/the-journal-of-academic-librarianship/   lloyd, a. (2010). information literacy landscapes: information literacy in education, workplace and everyday contexts. oxford: chandos.   newstead, s.e., & hoskings, s. (2003). encouraging student motivation. in h. fry, s. ketteridge, & s. marshall (eds.), a handbook for teaching and learning in higher education: enhancing academic practice (3rd ed.). (pp. 26-41). new york: routledge.   pearce‐smith, n. (2006). a journal club is an effective tool for assisting librarians in the practice of evidence‐based librarianship: a case study. health information & libraries journal, 23(1), 32-40. retrieved 20 dec. 2012 from http://onlinelibrary.wiley.com/journal/10.1111/%28issn%291471-1842   robledo-rella, v., neri, l., & noguez, j. (2010). collaborative learning for physics courses at tecnologico de monterrey, mexico city campus. international journal of engineering education, 26(1), 136-140. retrieved 20 dec. 2012 from http://www.ijee.ie/   sackett, d.l. et al. (2000). evidence-based medicine: how to practice and teach ebm (2nd ed.). edinburgh: churchill livingstone.   stenhouse, l. (1981). what counts as research? british journal of educational studies, 29(2), 103-114. doi:10.1080/00071005.1981.9973589   verlander, p., & scutt, c. (2009). teaching information skills to large groups with limited time and resources. journal of information literacy, 3(1), 31-42. doi:10.11645/3.1.220   webb, j., & powis, c. (2004). auditing: finding out what your learners need. in j. webb & c. powis (eds.). teaching information skills: theory and practice. (pp. 31-77). london, england: facet publishing.   webber, s., & johnston, b. (2003). information literacy in the united kingdom: a critical review. in c. basili (ed.), information literacy in europe. (pp.258-283). rome: italian network research council.   wong, c. k,. & driscoll, m. (2008). a modified jigsaw method: an active learning strategy to develop the cognitive and affective domains through curricular review. journal of physical therapy education, 22(1), 15-23. retrieved 20 dec. 2012 from http://www.ptnow.org/journals/detail.aspx?cid=4859a056-5320-447e-9420-c207d8dad87c   young, p., & vilelle, l. (2011). the prevalence and practices of academic library journal clubs. journal of academic librarianship, 37(2), 130-136. doi:10.1016/j.acalib.2011.02.004     appendix a permission to use contribution in journal article   i plan to write a paper about evidence based practice and teaching methods. i would like to include your comments and contributions from our session today. i will be undertaking an analysis and synthesis of the discussions from today and will use these to support the writing of the paper. i will anonymize any comments used and acknowledge all the workshop participants at the end of the article.   if you are happy with the above, please sign and date below to show your agreement with the statement.   i am happy to have my contributions (written or verbal) used anonymously in a journal article. i understand i will be acknowledged in any such article.                signed:.............................................................................................     date:................................................................................................        appendix b questions for group reflection following the cec session   how did it feel to be a learner?   was it effective?   how useful would it be for you as a facilitator?   how would you adapt the session?   consider the four themes – what was notable for you about the material we looked at?   appendix c post-session questionnaire   please answer the four questions below:   do you use articles on teaching methods in your practice? for example papers which emphasise the practical application of theory in a teaching scenario?   what are the barriers to using evidence in your practice? what would make this easier?   are you aware of using theories of learning in your teaching? for example do you have a particular theoretical approach in mind as you prepare?   please give any other comments or feedback on this continuing education course below:      evidence based library and information practice       evidence summary   pubmed’s native interface remains the best tool for systematic searching of its biomedical citations   a review of: wildgaard, l. e., & lund, h. (2016). advancing pubmed? a comparison of third-party pubmed/medline tools.  library hi tech, 34 (4), 669-684. http://dx.doi.org/doi: 10.1108/lht-06-2016-0066   reviewed by: ann glusker research & data coordinator national network of libraries of medicine, pacific northwest region university of washington health sciences library seattle, washington, united states of america email: glusker@uw.edu   received: 1 mar. 2017                                                                   accepted: 17 apr. 2017      2017 glusker. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to compare the functionality of third-party pubmed tools for searching biomedical citations in pubmed, in the specific context of systematic searching.   design – comparative analysis of software functionality.   setting – online, freely accessible search software.   subjects – sixteen third-party tools for searching and managing the full range of pubmed citations (tools which focused on specific disciplines were not included).   methods – tools for analysis were identified in two ways; those discussed in two published articles were used, and a supplementary pubmed search was performed. the initial list of 76 possibilities was assessed for study inclusion on 4 criteria: covering the entire range of pubmed content; being freely available; not limiting to a particular bio-medical discipline; and incorporating online pubmed/medline content. after assessment, 16 tools were chosen for further analysis (the authors provide a list and description of the tools in their table i). each was examined in relation to 11 crucial operational aspects. result sets were tested against a control (a literature search result set on a particular clinical question which was determined by physicians to yield relevant results, details of which are provided by the authors in an online appendix).    main results – the 11 identified aspects related to tool functionality were examined for each tool selected, with results grouped into three sets of factors: 1) supporting the search (field codes, filters, limits and boolean operators); 2) managing the search (output, related articles, links to articles, number of results, exporting); and 3) documenting the search (saving the search and search history). in some cases, the tests had to be adjusted to accommodate the tool's specifications. in table ii the authors present a grid with the results of the testing, on each of the 11 aspects, for each tool.   the authors found that with many tools it was not straightforward, if even possible, to filter and limit in order to get more specific result sets. few tools were effective at suggesting related articles within the tool itself, instead linking the user out to pubmed, and only two tools provided the same number of citation results as the comparison pubmed search. in addition, the display of results often made it difficult to assess result sets; and only two tools provided the option to save searches and see search history. furthermore, due to unexpected tool limitations, it was not possible to assess the relevance of citation result sets delivered by the third-party tools, as compared with the control pubmed search.   conclusion – close analysis of the tools studied indicated that they were not created in order to support systematic searches. they lack support for filtering/limiting, saving or exporting searches, which are central functionalities to the work of performing such searches. while some of the tools studied may still be in the early phases of development, and while several of them, in enhancing pubmed searches in particular ways, may suggest additional profitable strategies for performing a systematic search, not one of them can replace the functionalities of the native pubmed interface. it remains the best tool for searching and managing the full range of pubmed citations, for the purposes of performing systematic searches. commentary   this study was an addition to existing literature – specifically articles the authors consulted by lu (2011) and keepanasseril (2014), which merely listed and described tools – in that the authors analyzed the functionality of tools using a detailed set of criteria and a validated search as a control. while it turned out that the third-party tools examined are not suited to use for systematic searching, they may be useful for other search enhancements. the authors state that these tools “are beneficial as they give immediate, dynamic visual assessment of relationships between authors, topics and term hierarchies, etc. in the bibliographic data, giving a strong starting point in evaluating and selecting literature to include in a systematic search” (p. 679). for example, in 2010 kristine ogden outlined several aspects of the tools hubmed and quertle which helped her in clinical searches: a citation finder which pulls pubmed records for citations in a bibliography; the ability to run a pubmed search automatically on other sites such as googlescholar; the use of natural language to find relevant citations; and separate tabs for keyword search results and citations. she also appreciated the clean user interfaces of these tools.   furthermore, this paper makes an important contribution toward supporting medical librarians and others who work with systematic reviewers, in showing the crucial importance of the systematic searching that underlies such reviews. it also gives librarians a framework for helping systematic reviewers assess third-party tools to help with those searches.   for this evidence summary, methodologies were systematically assessed using glynn’s critical appraisal checklist (2006). the checklist was designed to evaluate population-based studies, and so some of its criteria did not apply to this study, but it does focus on freedom from bias and representativeness of the subjects studied. in this instance, one question is whether there are more effective third-party tools for systematic searching which may not be freely available, but rather exist behind pay walls – could including them have changed the results? also, there may be excellent tools that were designed for specific clinical areas which were excluded from this study. in table iii the authors list excluded third-party tools, which could prove a resource for future analysis.     furthermore, in addition to listing closed projects and dead links, table iv lists potentially relevant third-party tools and sites under construction. the authors are not denigrating the third-party tools they tested (in fact, they mentioned wanting to re-test them, and entries in table iv may be a starting place). future methodological advances may contribute to the creation of systematic pubmed search tools. as described in gonzalez et al. “computational methods contribute…by bringing knowledge from literature, either extracted or curated, together with high-throughput data sets, to identify both known and new relationships” (2016, p. 39). while the context for such computational methods relates to text and data mining, there is every reason to expect that they may eventually contribute to systematic analysis of pubmed citations such as wildgaard and lund seek.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154   gonzalez, g. h., tahsin, t., goodale, b. c., greene, a. c., & greene, c. s. (2016). recent advances and emerging applications in text and data mining for biomedical discovery. briefings in bioinformatics, 17(1), 33-42. https://dx.doi.org/10.1093/bib/bbv087   keepanasseril, a. (2014). pubmed alternatives to search medline: an environmental scan. indian journal of dental research, 25(4), 527o. http://dx.doi.org/doi:10.4103/0970-9290.142562   lu, z. (2011). pubmed and beyond: a survey of web tools for searching biomedical literature. database: the journal of biological databases and curation, baq036. http://dx.doi.org/doi: 10.1093/database/baq036   ogden, k. (2010, march 16). why use a third-party tool to interact with pubmed? retrieved from https://nnlm.gov/pnr/dragonfly/2010/03/16/hubmed_quertle       research article   movement-based programs in u.s. and canadian public libraries: evidence of impacts from an exploratory survey   noah lenstra assistant professor library and information studies school of education university of north carolina at greensboro greensboro, north carolina, united states of america email: lenstra@uncg.edu   received: 18 apr. 2017   accepted: 25 oct. 2017      2017 lenstra. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – past research suggests that approximately 20-30% of public libraries in the united states offer movement-based programs, that is programs that encourage, enable, or foster physical activity and physical fitness. little is currently known about the impacts of these programs, in the u.s. or elsewhere. this study addresses the questions: what impacts do movement-based programs in public libraries have and what variations exist between urban and rural libraries.   methods – the researcher aimed to explore these questions through an exploratory survey of u.s. and canadian public libraries that have offered movement-based programs. the survey was completed by self-selecting staff from 1,157 public libraries in the u.s. and canada during spring 2017. analysis focuses on those portions of the survey that address the impacts of movement-based programs.   results – results show that throughout north america, public libraries provide movement-based programs for all age groups. the most consistently reported impact of these programs is new library users. furthermore, on average respondents report that participation in these programs slightly exceeding their expectations. these facts may account for the finding that 95% of respondents report that they intend to continue offering movement-based programs at their libraries.   conclusion – more research using a randomized survey design is needed to better assess this emerging programming area in a more comprehensive manner. nonetheless, this study provides needed evidence on the impacts of movement-based programs in many north american public libraries. hopefully this evidence will contribute to more conversations and research on the roles of public libraries in public health and wellness.     introduction   this article analyzes an emerging type of public library program: movement-based programs. these are programs that encourage, enable, and foster physical activity and physical fitness (lenstra, 2017). the literature review below shows that although there is both research-based evidence that approximately 20-30% of public libraries in the united states offer movement-based programs and anecdotal evidence that these programs are offered by public libraries elsewhere in the world, the impacts and outcomes of these programs have received little attention. this paper addresses this gap by presenting the results from a survey of north american public libraries that have offered movement-based programs.   since little was known about the impacts of movement-based programs in public libraries, an exploratory survey design was used to address the following research questions: what impacts do movement-based programs in public libraries have and what variations exist between urban and rural libraries. results show that these programs tend to bring new users into libraries, contribute to community building as well as to health and wellness. most respondents (95%) state that they intend to continue offering movement-based programs at their public libraries. the article concludes by discussing how these results can productively inform our understanding of the evolving roles of public libraries in relation to public health and wellness.   literature review   the literature on movement-based programs in public libraries consists of three types: 1) the inclusion of questions about movement-based programs in surveys that focus on other facets of public librarianship, 2) case studies in which researchers were participants in the experimental cases analyzed, and 3) short, journalistic program reports shared in channels without peer-review or expectations of adherence to research frameworks. this literature shows that approximately 20-30% of u.s. public libraries have offered some form of movement-based programming. furthermore, the case studies and journalistic reports suggest that these programs are also offered elsewhere around the globe. although this literature suggests that movement-based programs tend to resonate with the populations served, no research has yet analyzed in detail what impacts movement-based programs have. as a result, the profession has yet to develop the means to communicate about physical activity in public libraries to policy makers, to broader stakeholders, or even to itself.   survey-based research   surveys conducted during the last decade find that movement-based programs have been offered in many public libraries throughout the united states. a randomized survey of gaming programs in public libraries (nicholson, 2009, p. 206) found that “physical games” that require moving the body were the fourth most common type of gaming program offered in public libraries. a follow-up study using convenience sampling that included school and academic libraries found that “the most popular game activity reported in 2006 gaming programs in libraries was the dance dance revolution series, with 44% of library programs [reported] using this game” (nicholson, 2009, p. 209).   more recently, two surveys conducted in 2014 attest to the presence of yoga and other fitness classes among the regular offerings of u.s. public libraries. among other questions, the 2014 digital inclusion survey, conducted by the information policy and access center at the university of maryland, asked a random sample of public libraries a series of questions related to health programs and services they provided. one question asked respondents to state whether or not their libraries had during the past year offered “fitness classes (e.g., zumba, yoga, tai chi, other).” the survey found that approximately 22.7% of u.s. public libraries had offered some sort of fitness class (bertot, real, lee, mcdermott, & jaeger, 2015, p. 62), with these types of programs most common in suburban libraries (33.9%) and least common in rural libraries (12.6%).   another survey conducted in 2014 came to similar conclusions. the library journal programming survey asked a convenience sample of library journal subscribers working in public libraries to answer questions about yoga programs offered by their libraries. the survey found that 33% of respondents had offered yoga programs during the last twelve months (library journal, 2014). of those public libraries that had offered yoga, 77% said they offered it for adults, 27% for teenagers, and 40% for children. of these three surveys, only library journal’s produced evidence on the impacts of movement-based programs: 23% of libraries with yoga programs said they had been very popular, 43% said popular, 28% said somewhat popular, and only 6% said not at all popular.   case study research   the earliest research-based case study of movement-based programs in public libraries was conducted by two public librarians in the early 1990s. public librarians in connecticut collaborated with a local aerobics instructor to develop a series for teenaged girls that included fitness classes. interviews with the teenaged participants revealed that the fitness components of the program led to increased self-esteem and increased interest in regular physical activity (quatrella & blosveren, 1994). it is unclear if the program continued after the trial study. in any case, approximately 15 years later a group of librarians from the louisiana state university health sciences center launched a series of programs for youth in local public libraries that included exercise instruction (woodson, timm, & jones, 2011). by tracking the participants in these programs, the authors determined that the programs were successful in that the children who participated had fun while learning about health and wellness.   more recently, three research-based case studies on movement-based programs in public libraries were published in 2015 and 2016. health science librarians from washington university in st. louis, missouri partnered with the local public library system to administer a community survey on health information needs. the survey found that “exercise” was the topic the public most wanted to see more of at the library (engeszer et al., 2016, p. 64). in response, the partners developed a series of programs that included yoga, beginning exercise, and zumba that was subsequently offered throughout the st. louis public library system.   a similar study took place in the small town of farmville, north carolina, where the public library partnered with a nearby library and information science professor to develop programs and services that promote healthy lifestyles (flaherty & miller, 2016). the library loaned pedometers to patrons and the researcher interviewed those who participated. participants reported liking the program and asked for more movement-based programs at the library. in response, the library organized a 5k race and a mile fun walk/run in spring 2015, which has since become an annual library-sponsored program. based on the success of these initiatives, the public librarian became the wellness coordinator for the town.   in lethbridge, alberta, public librarians collaborated with local and provincial partners to develop a "library of things" initiative that involved checking out supplies that could be used in physical activities (cofell, longair, & weekes, 2015; weekes & longair, 2016). the librarians assessed the program by monitoring circulation trends and collecting feedback from participants. they found that the circulating materials contributed to increasing physical literacy and physical activity among participants.   collectively these studies show that diverse types of movement-based programs tend to be popular with public library patrons. nonetheless, these case studies are based in particular places. without analysis of libraries outside of those locations it is difficult to make generalizations about the impacts of these types of programs beyond the particular cases presented.   short reports of programs authored by public librarians   in addition to the peer-reviewed research literature discussed above, short reports concerning programs in public libraries have been published outside peer-reviewed channels. these reports illustrate other types of movement-based programs offered in libraries. in addition to the types of programs discussed above, this literature reports on movement-based programs for early literacy (e.g. music and movement) (dietzel-glair, 2013; kaplan, 2014; prato, 2014), library-based community gardens (peterson, 2017), dancing (green, 2013; st. louis public library, 2014), storywalks® (maddigan & bloos, 2013), outdoor activities like walking and bicycling (hill, 2017; richmond, 2012), and fitness challenges (hanson, 2012).[1] furthermore, these reports illustrate that movement-based programs are being offered in public libraries in canada (maddigan & bloos, 2013), the united kingdom (vincent, 2014), romania (eifl, 2016), namibia (hamwaalwa, teasdale, mcguire, & shuumbili, 2016), china (zhu, 2017), and singapore (national library board of singapore, 2017).   a lack of evidence on the impacts of innovations in public library programs   one would perhaps expect that the growth of movement-based programs in public libraries would naturally lead to a growth of data collection on the spread and impacts of these programs. however, the continued lack of evidence based research on innovations in public library programs and services complicates matters. in a guest editorial to a special issue of eblip focused on public libraries, ryan (2012) writes that   despite this welcome inclusion in eblip, public librarian participation is notably low. this mirrors the grim reality of low public librarian research and publication rates, as well as the small overall percentage of lis research articles about public library practice. (p. 5)   in a recent follow-up to this special issue, cole and ryan (2016) note that “the current state of evidence based practice and research on, and to inform, public library practice lags significantly behind that of other library sectors” (p. 120). as a result of this state of affairs, there continues to be a great need for research both on how public libraries are innovating, as well as on the impacts of these innovations.   within the u.s. public library profession, one means of enabling librarians to integrate evidence into their evolving practices has been the development of the project outcome toolkit. the u.s. public library association’s project outcome seeks to create standardized evaluation tools that public librarians can use to assess the impacts of their services and programs (anthony, 2016; oehlke, 2016). nonetheless, despite this laudable goal there are significant gaps in the coverage of project outcome. in particular, the toolkit provides no means of assessing how libraries contribute to health and wellness. project outcome focuses on assessing what it calls “seven essential library service areas,” including: “civic/community engagement, early childhood literacy, education/lifelong learning, summer reading, digital learning, economic development, and job skills” (public library association, 2017, n.p.). despite a plethora of studies showing that public libraries impact population health and wellness (e.g. gillaspy, 2005; morgan, dupuis, whiteman, d’alonzo, & cannuscio, 2017; rubenstein, 2016), project outcome does not include any tools to assess these outcomes. as a result, more work is needed to understand how public libraries impact health as well as to prepare public librarians to incorporate evidence into this service area. according to public health scholars and policy-makers, regular physical activity is one of the best things for good health (kohl et al., 2012). the researcher aimed to investigate the impacts of movement-based programs in public libraries to better understand the impacts of physical activity in public libraries.   aims and methods   study design   since little was known about the general impacts of movement-based programs in public libraries an exploratory survey design was used to address the research questions:   what impacts do movement-based programs in public libraries have? what variations exist between urban and rural libraries? the focus on disentangling differences between urban and rural libraries relates to a continued divide between these two types of public libraries in the u.s., with entire professional associations focused around the concerns of these two groups (i.e. the association for rural & small libraries and the urban libraries council).   in any case, in creating the data collection instrument (appendix a), the author looked to past surveys of public libraries (e.g. bertot et al., 2015), as well as to past literature on movement-based programs. in addition, the survey was piloted with three public librarians, one each from illinois, north carolina, and new brunswick. these librarians helped inform the language used in the final survey.   data collection   public libraries throughout north american were invited to self-select for participation in the survey. the researcher hopes that in the future this self-selecting sample can be supplemented by a randomized sample of public libraries. data collection was carried out via an online questionnaire using qualtrics. the url to the questionnaire was sent to public librarians in the u.s. and canada through state and provincial library electronic mailing lists, as well as through announcements from state and provincial libraries to public libraries in their regions. in addition, the survey was disseminated through national electronic mailing lists used by public librarians (e.g. publib) and on the project’s website. between february 14 and march 23, 2017 a self-selecting sample of 1,828 public librarians began the “let’s move in libraries survey”.   data analysis   respondents were invited to complete as much or as little of the survey as they wished. after removing partial responses (n=570) and responses from libraries that had never offered any movement-based programs (n=101), a sample of 1,157 libraries remained for analysis.   the data were integrated with data from the institute of museum and library services fy 2014 public libraries survey (imls, 2016) to sort the respondents into “urban,” “suburban,” “town,” and “rural” libraries, as well as to sort the respondents by region. according to imls (2016) the major distinction between urban/suburban and town/rural libraries is that the former are libraries located within urban metropolitan areas and the latter are libraries located outside those metro areas. all canadian respondents (n=62), as well as 49 u.s. respondents could not be integrated with the imls dataset. these 101 respondents were sorted by hand, using the methods of the imls, into these 4 geospatial divisions.   to transform the data in ways that would allow for quantitative comparisons between urban and rural libraries, the verbal options from which respondents selected were translated into numbers. see table 1 below for an example of how this process was carried out. the number in the “average across all programs” column on the right side of the table illustrates how comparisons were made among libraries. for instance, in the example below library 1 reported the most satisfaction with program participation. the fact that program participation “fell below expectations” in one of the movement-based programs offered at library 3 led to its composite measure being lower. similar techniques enabled comparisons among libraries in terms of the extent to which movement-based programs had brought new users into libraries, and the extent to which the media had reported on movement-based programs in libraries.   findings   description of sample   figure 1 shows respondents’ physical locations. although respondents are located in many parts of north america, this self-selecting sample does not constitute a statistically representative sampling of all public libraries that offer movement-based programs. nonetheless, as table 2 shows, the respondents do represent many types of communities, with a nearly even split between libraries located within urban metro areas (54%) and libraries located outside metro areas (46%).   overall, respondents reported that their libraries had offered a wide variety of movement-based programs for a wide array of age groups. yoga programs were the most commonly reported type of program, offered in 65% of the responding libraries (figure 2), followed by movement-based early literacy programs (55%), gardening (41%), dancing (36%), and storywalks® (29%). most of the more frequently offered types of movement-based programs were reported more frequently in urban and suburban libraries than in town and rural libraries. however, other programs, including storywalks®, “other,” outdoor activities, fitness challenges, and library of things initiatives were slightly more likely to be reported in town and rural than in urban and suburban libraries.   respondents reported offering movement-based programs for all age groups. among respondents, 73% had offered programs for pre-k audiences, 52% for school-aged youth, 39% for tweens and teenagers, 65% for adults, and 42% for senior citizens. in addition, 38% reported movement-based programs for all ages and 37% reported programs for families (see figure 3). urban and suburban libraries were more likely to have offered movement-based programs for all of the groups asked about except for “all ages” programs, which were slightly more common in town and rural libraries.     table 1 example of data analysisa. library early literacy yoga tai chi zumba dancing outdoor activities storywalks gardening fitness challenges library of things other average across all programs library 1 exceeded expectations exceeded expectations exceeded expectations exceeded expectations exceeded expectations exceeded expectations n/a library 2 exceeded expectations exceeded expectations exceeded expectations met expectations met expectations n/a library 3 exceeded expectations exceeded expectations exceeded expectations fell below expectations n/a l1 3 3 3 3   3   3       3 l2 3 3     3 2   2       2.6 l3 3 3   3           1   2.5 a based on three libraries’ responses to the question “how would you characterize participation levels in programs?” (appendix a)     table 2 survey respondents by type, compared to u.s. population of public libraries (bertot et al., 2015).   location of respondents u.s. library locations urban 18% (n=204) 17% (n=2779) suburban 36% (n=419) 26% (n=4369) town 28% (n=327) 20% (n=3298) rural 18% (n=207) 37% (n=6249) all 100% (n=1157) 100% (n=16695)       figure 1 visualization of where respondents are physically located in north america, n=1157.       figure 2 percentage of respondents that have offered movement-based programs.     figure 3 the audiences of movement-based programs in public libraries.       part 2: the impacts of movement-based programs in public libraries   to understand the impacts these programs have had, this section first analyzes the different ways libraries have assessed their movement-based programs. it then analyzes the satisfaction of library staff with participation levels, before looking at to what extent programs have brought new users to libraries and to what extent programs have received attention from local media. this section concludes by analyzing the outcomes to which these programs have contributed.   1. assessment techniques   the principal technique libraries use to assess the impacts of movement-based programs has been to count the number of participants. approximately 90% of respondents said that they use this method. the remaining 10% reported doing no assessment. surveys and interviews were supplementary assessment techniques sometimes used by approximately 30% and 20% of respondents, respectively.   2. participation levels   based on these assessment techniques, libraries generally reported satisfaction with how many people had participated in their movement-based programs. based on the analytical techniques discussed above (see methods), on average libraries reported participation levels that slightly exceeded their expectations. respondents were asked for each type of movement-based program they had offered whether participation fell below (coded to “1”), met (“2”), or exceeded (“3”) expectations. the average satisfaction level across all respondents was 2.2, with statistically significant differences between urban/suburban (m=2.240, sd=0.474) and town/rural (m=2.145, sd=0.481) libraries, conditions: t(1110)=3.3414 =, p=0.0009. in other words, although on average all libraries reported participation levels that slightly exceeded expectations, urban and suburban libraries were more likely than rural and town libraries to report participation levels meeting and exceeding expectations.   figure 4. outcomes to which movement-based programs in public libraries contributed.     3. users   the most consistently reported impact of movement-based programs was that these programs brought new users into libraries. for each type of movement-based program offered, respondents were asked whether the program had (coded to “2”) or had not (“1”) brought new users to their libraries. a significant number of respondents (n=183, or 16% of the sample) did not know the answer to this question. nonetheless, among those libraries that did know, the vast majority reported new users coming to libraries because of their participation in movement-based programs. the overall average was 1.86. there was a significant difference between urban/suburban (m=1.904, sd=0.228) and town/rural (m=1.817, sd=0.317) libraries, conditions: t(972)=4.942 p=0.0001. in other words, the tendency for movement-based programs to bring new users to libraries was more accentuated in urban libraries.   4. media   even more respondents (n=242, or 21% of the sample) did not know whether or not the media had reported on their libraries’ movement-based programs. nonetheless, among those who did know the answer to this question, the composite average was 1.55 (“2”=yes, “1”=no). furthermore, there with a statistically significant difference between urban/suburban (m=1.505, sd=0.442) and town/rural (m=1.591, sd=0.446) libraries, conditions: t(912)=2.958, p=0.0032. in other words, movement-based programs tended to receive slightly more media coverage in more rural libraries.   5. outcomes   finally, respondents were asked, based on any feedback and evidence they may have collected, if their movement-based programs had contributed to health or wellness, community building, outreach, literacy, or other outcomes. overall, only slight variation existed between urban/suburban and town/rural respondents (see figure 4). interestingly, the most commonly reported outcome was not health or wellness (76%), but rather community building (80%). in addition, over 50% of respondents said that at least one of their movement-based programs had contributed to outreach (52%) or to literacy (58%), suggesting that movement-based programs contribute to multiple outcomes in the public libraries that offer them.   the final measure of the impact of movement-based programs in public libraries comes from the answer to the question: will libraries continue to provide these types of programs in the future? nearly 95% of respondents (n=1094) said their libraries plan to continue offering movement-based programs.   discussion   similar to the library journal survey (2014) that asked about yoga programs in u.s., this study found that movement-based programs have been offered for multiple age groups. there does not appear to be any one primary age group for these types of programs. nevertheless, the high percentage of respondents that reported programs for pre-k youth suggests that movement may be most integrated into library programs for this age group, an assertion bolstered by the many program development tools that discuss how to incorporate movement into programs for pre-k audiences in public libraries (e.g. dietzel-glair, 2013; kaplan, 2014; prato, 2014). the extent to which movement has been integrated into library programs for other age groups is less clear. however, in at least some libraries it does appear that movement-based programs for diverse age groups has become a normal part of library programming.   in any case, the results from this survey also suggest that urban and suburban libraries may be offering slightly different types of programs than their rural and town counterparts. in particular, the survey found that programs that do not require the use of an indoor meeting space, or that take place outside the library (such as storywalks®, outdoor activities, library of things initiatives, and fitness challenges) were offered more often in town and rural libraries than in urban and suburban libraries. on the other hand, the differences reported were slight. more research will be needed to determine if the types of movement-based programs offered in public libraries differ by the types of communities served.   the evidence on the impacts of movement-based programs adds to our understanding of how public libraries impact health and wellness. past research has investigated how public libraries impact health through consumer health information services (e.g. rubenstein, 2016), but has not focused directly on the question of how public libraries impact health by fostering active lifestyles. being physically active throughout all stages of life is one of the most important things people can do to be healthy (kohl et al., 2012). better understanding the impacts of this emerging programming area could potentially contribute to the development of tools to assess how public libraries impact health and wellness, which could potentially be included in the u.s.-based project outcome toolkit (public library association, 2017), as well as in other assessment tools being developed elsewhere (cole & ryan, 2016). although more research is needed, the findings from this exploratory study suggest that movement-based programs contribute both to health and wellness as well as to community building. furthermore, the fact that so many libraries reported new users being brought to libraries because of these types of programs suggests that these programs also contribute to community engagement in libraries.   limitations   the principal limitation of this work derives from its exploratory nature. rather than survey a randomized sample of all public libraries in the u.s. and canada, the researcher instead recruited a self-selecting sample of public libraries, relying primarily on state and provincial mediators to disseminate this survey to public librarians in their regions. future work should more rigorously test and refine these exploratory results by using a randomized study design to enhance our knowledge and understanding of how widespread these types of programs have become and what impacts these types of programs have.   despite this limitation, this study shows that many public libraries throughout north america do offer a wide variety of movement-based programs and most plan to continue offering these programs. based on these facts, more research is needed to understand why this programming area has emerged, how it works, and what impacts it is having. in addition to more quantitative data, we also need qualitative studies that look in depth at the evolution and impacts of movement-based programs as they have emerged and evolved in particular public libraries.   conclusion   past surveys of public libraries show that movement-based programs have been offered in 20-30% of u.s. public libraries (bertot et al., 2015). furthermore, case studies and journalistic reports show that movement-based programs also occur elsewhere. nonetheless, despite this evidence little was known about the impacts these programs have had beyond the particular cases discusses in past case studies and reports. this study added to this literature by reporting data from a self-selecting sample of 1,157 u.s. and canadian public libraries that have offered movement-based programs. the most consistently reported impact of movement-based programs in libraries is that they bring new users into public libraries. complicating assessment of the impacts of these programs is the fact that a majority of respondents did no assessment of their programs beyond counting the numbers of participants. the need for more research on this topic is great; this article has sought to provide needed evidence on this emerging programming area in order to support future conversations and studies.   references   anthony, c. (2016). project outcome: looking back, looking forward. public libraries online, http://publiclibrariesonline.org/2016/01/project-outcome-looking-back-looking-forward/.   bertot, j. c., real, b., lee, j., mcdermott, a. j., & jaeger, p. t. (2015). 2014 digital inclusion survey: survey findings and results. university of maryland, college park: information policy & access center (ipac). http://digitalinclusion.umd.edu/sites/default/files/uploads/2014digitalinclusionsurveyfinalrelease.pdf.   cofell, j., longair, b., & weekes, l. (2015). physical literacy in the library or, how we ended up loaning out rubber chickens. pnla quarterly, 80(1), 34-36. https://arc.lib.montana.edu/ojs/index.php/pnla/article/view/295/218   cole, b., & ryan, p. (2016). public libraries. in d. koufogiannakis & a. brettle (eds.), being evidence based in library and information practice (pp. 105-120). london: facet.   dietzel-glair, j. (2013). books in motion: connecting preschoolers with books through art, games, movement, music, playacting, and props. chicago: american library association.   eifl. (2016). keeping fit in romania: pietrari local public library helps the community get back into shape. http://www.eifl.net/eifl-in-action/creative-use-ict-innovation-award.   engeszer, r. j., olmstadt, w., daley, j., norfolk, m., krekeler, k., rogers, m., colditz, g., anwuri, v.v., morris, s., voorhees, m., & mcdonald, b. (2016). evolution of an academic–public library partnership. journal of the medical library association: jmla, 104(1), 62-66. doi: 10.3163/1536-5050.104.1.010   flaherty, m. g., & miller, d. (2016). rural public libraries as community change agents: opportunities for health promotion. journal of education for library and information science, 57(2), 143-150.   gillaspy, m. l. (2005). factors affecting the provision of consumer health information in public libraries: the last five years. library trends, 53(3), 480-495. http://hdl.handle.net/2142/1738   green, j. (2013). thoughts on the power of arts with older adults. creative aging toolkit for public libraries, http://creativeagingtoolkit.org/power-of-the-arts-with-older-adults/.   hamwaalwa, n., teasdale, r. m., mcguire, r., & shuumbili, t. (2016). promoting innovation in namibian libraries through leadership training. proceedings of ifla 2016, http://library.ifla.org/1509/1/189-hamwaalwa-en.pdf.   hanson, t. (2012). featured: estherville’s couch-to-5k. the association for small and rural libraries blog, http://arsl.info/esthervilles-couch-to-5k/.   institute of museum and library services. (2016). data file documentation. public libraries survey. fiscal year 2014. washington, d.c.: institute of museum and library services. https://www.imls.gov/sites/default/files/fy2014_pls_data_file_documentation.pdf.   hill, j. (2017). book-a-bike: increasing access to physical activity with a library card. in m. robison & l. shedd (eds.), audio recorders to zucchini seeds: building a library of things (pp. 43-51). santa barbara: libraries unlimited.   kaplan, a. (2014). get up and move! why movement is part of early literacy skills development. university of wisconsin madison school of library and information studies. http://vanhise.lss.wisc.edu/slis/2014webinars.htm.   kohl, h. w., craig, c. l., lambert, e. v., inoue, s., alkandari, j. r., leetongin, g., kahlmeier, s.,& lancet physical activity series working group. (2012). the pandemic of physical inactivity: global action for public health. the lancet, 380(9838), 294-305. http://dx.doi.org/10.1016/s0140-6736(12)60898-8   lenstra, n. (2017). let’s move! fitness programming in public libraries. public library quarterly, 36(4), 1-20. http://dx.doi.org/10.1080/01616846.2017.1316150.   library journal. (2014). public library maker & non-book related programming report. unpublished manuscript, american library association.   maddigan, b. c. & bloos, s. c. (2013). community library programs that work: building youth and family literacy. santa barbara: libraries unlimited.   morgan, a. u., dupuis, r., whiteman, e. d., d’alonzo, b., & cannuscio, c. c. (2017). “our doors are open to everybody”: public libraries as common ground for public health. journal of urban health, 94(1), 1-3. https://link.springer.com/article/10.1007/s11524-016-0118-x   national library board of singapore. (2017). exercise programs. https://www.nlb.gov.sg/golibrary2/c/30307529/result/term/exercise.   nicholson, s. (2009). go back to start: gathering baseline data about gaming in libraries. library review, 58(3), 203-214. https://doi.org/10.1108/00242530910942054   oehlke, v. (2016). pla president’s report, ala 2016 annual conference. chicago: public library association. http://www.ala.org/pla/sites/ala.org.pla/files/content/about/board/vailey_oehlke_presidents_report.pdf.   peterson, j. (2017). growing library garden programs. webjunction. http://www.webjunction.org/news/webjunction/growing-library-garden-programs.html.   prato, s. (2014). music and movement at the library. association for library service to children blog, http://www.alsc.ala.org/blog/2014/11/music-and-movement-at-the-library/.   public library association. (2017). project outcome: measuring the true impact of public libraries. https://www.projectoutcome.org/.   ryan, p. (2012). eblip and public libraries. evidence based library and information practice, 7(1), 5-6. https://journals.library.ualberta.ca/eblip/index.php/eblip/article/view/16557/13672   quatrella, l., & blosveren, b. (1994). sweat and self-esteem: a public library supports young women. wilson library bulletin, 68(7), 34–36   richmond, a. (2012). running and reading at rye public library. granite state libraries, 48(3), 12. http://fliphtml5.com/gjox/riia/basic.   rubenstein, e. l. (2016). breaking health barriers: how can public libraries contribute? public library quarterly, 35(4), 331-337. https://doi.org/10.1080/01616846.2016.1245006   st. louis county library. (2014). bean stalk ballet, 5.1.14. flickr. https://www.flickr.com/photos/slclevents/sets/72157644064466020/.   vincent, j. (2014). an overlooked resource? public libraries’ work with older people–an introduction. working with older people, 18(4): 214–222.   weekes, l. & longair, b. (2016). physical literacy in the library—lethbridge public library, lethbridge, alberta, canada. international information and library review, 48(2), 152-154.   woodson, d. e., timm, d. f., & jones, d. (2011). teaching kids about healthy lifestyles through stories and games: partnering with public libraries to reach local children. journal of hospital librarianship, 11(1), 59-69. https://doi.org/10.1080/15323269.2011.538619   zhu, j. (2017). perfect combination of reading and sports. proceedings of ifla 2017, http://library.ifla.org/id/eprint/1869.   appendix a let’s move in libraries survey   q1. these questions ask for some background information on your library. what is the zip code, or postal code, of your library's physical location?   q2. if you would like to provide it, what is the name of your library?   q3. survey part 1. this survey first asks about programs or services your library has offered in the past or currently offers in the present. at the end of the survey you will be given the opportunity to discuss programs or services your library is planning, but has not yet offered to the public. has your library ever offered any programs or services that include (select all that apply)? [note: responses to q3 were carried forward for the remainder of the survey]                 movement-based programs for early literacy (e.g. music and movement)               yoga               tai chi               zumba               dancing               walking, hiking, bicycling, or running               storywalks               gardening               fitness challenges (e.g. pedometer challenge, biggest loser programs, couch to 5k)               fitness equipment that can be checked out, including passes for gyms or aquatic centers               other programs or services               no programs or services involving movement   q3.b. [if “other programs or services” selected than this question appears.] what other movement-based programs or services has your library offered?    q4. survey part 2. you are now invited to participate in the second part of this survey. this part of the survey consists of 16 questions that ask about the administration of the programs and services your library offers, or has offered in the past. it should take about 10 minutes to complete. would you like to participate in the second part of this survey?  [if respondents select “no” they skip to q26.]   q5. these questions ask about the timing of programs and services your library offers, or has offered. [carried forward programs] first offered by your library:               after jan. 1, 2016               before jan. 1, 2016               don’t know   q6. since your library started offering these programs and services, how regularly, on average, has your library offered them to the public? [carried forward programs] offered:               only once               more frequently than once a month               once a month               less frequently than once a month               not applicable               don’t know   q7. on which days and times has your library offered the following [carried forward programs] (select all that apply)               weekday mornings               weekday afternoons               weekday evenings               weekend mornings               weekend afternoons               weekend evenings               not applicable               don’t know   q8. these questions ask about who these programs/services are for, and also who participates in them. for which audiences are these [carried forward programs] targeted? (select all that apply)               youth, birth-5               school-aged youth               tweens and teens               adults               senior citizens               families               all ages               don’t know   q9. how would you characterize participation levels in these programs? [carried forward programs] participation:               exceeded expectations               met expectations               fell below expectations               don’t know   q10. this question asks about the reasons your library offers these programs. for each of the programs your library offers, please indicate which of the following are reasons for the program. if multiple reasons, please select multiple responses.               lifelong learning               literacy               health and/or wellness               community engagement               other               don’t know   q11. please discuss other reasons, if any, your library offers these programs.   q12. these questions ask about how programs and services in your library relate to other spaces and programs in your service area. please answer to the best of your ability. where are your library's programs and services physically located?               within a community room or auditorium located within the library               within another space in the library               outside the library               not applicable               don’t know   q13. if you have other information about the location of these programs and services, please record it here.   q14. who leads or directs these programs and services? (select all that apply). [carried forward programs] led by:               librarians or library paraprofessionals               paid contractors               partner institutions or groups               individual volunteers               other               don’t know   q15. if your library developed these programs and services with partners (e.g. parks departments, public health departments, ymcas, etc.), please specify who these partners are here.   q16. these questions ask about the management and administration of these programs and services. are these programs/services under the supervision of a particular division of your library? if so, which ones. (select all that apply). [carried forward programs] supervised by:               the library as a whole               adult services               teen services               youth services               programming, outreach, or lifelong learning staff               other               don’t know   q17. if needed, please discuss here how these programs and services fit within your organizational hierarchy.   q18. for the following programs and services, are any of the following ever required? (select all that apply). [carried forward programs] sometimes or always require participants:               register in advance               sign a waiver of liability               pay a fee               do something else               no requirements for participation               don’t know   q19. how are these programs and services funded? (select all that apply). [carried forward programs] funded by:               regular library budget               programming budget               friends of the library               donations               grants               other               don’t know   q20. how have programs been marketed? (select all that apply). [carried forward programs] marketed through:               print flyers               newspaper advertisements or articles               website               online calendar               social media               word of mouth               other               don’t know   q21. how have the programs and services been assessed (select all that apply)? ). [carried forward programs] assessed through:               head counts of participants               surveys of participants               interviews with participants               no assessment               other               don’t know   q22. what other administrative issues or challenges has your library had to address in organizing these programs and services?   q23. these questions ask about the impacts of these programs and services. has the media reported on the fact that your library is offering [carried forward programs]?               yes               no               don’t know   q24. this question asks about how these programs and services engage your community. have these [carried forward programs] brought new users into your library?               yes               no               don’t know   q25. based on feedback and evidence you have collected, have these [carried forward programs] contributed to any of the following (select all that apply)?               health and/or wellness               literacy               community building               outreach               other               don’t know q25.b. if "other impacts" selected, please discuss them here.   q26. in the future, does your library plan to provide any programs or services that include (select all that apply)?               movement-based programs for early literacy (e.g. music and movement)               yoga               tai chi               zumba               dancing               walking, hiking, bicycling, or running               storywalks               gardening               fitness challenges (e.g. pedometer challenge, biggest loser programs, couch to 5k)               fitness equipment that can be checked out, including passes for gyms or aquatic centers               other programs or services               no programs or services involving movement   q26.b. [if “other programs or services” selected than this question appears.] what other movement-based programs or services does your library plan to offer in the future?    q27. thank you for taking the time to fill out this survey. if you have additional comments about these programs or services, or about this survey, please record them here.    q28. if you would like to be entered into the raffle for one of the ten (10) $50 gift certificates from amazon.com, please record your email address here.                   [1] for more information on the diverse types of movement-based programs offered in libraries, consult the website http://www.letsmovelibraries.org/. evidence based library and information practice commentary   using eblip to prepare future information professionals   lisl zach managing consultant informatics insights llc philadelphia, pennsylvania, united states of america email: lisl@marsez.com   received: 28 feb. 2016   accepted: 4 mar. 2016      2016 zach. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     as a library and information studies (lis) educator for many years, my goal has always been to send students out into the workforce well prepared to take on leadership roles as information professionals. to this end, i aim to develop their skills in the areas of critical thinking and problem solving. to support this aim, i rely on the principles of evidence-based practice to bring real-life examples into class discussions, whether based on my own experience and research or on appropriate sources in the research literature. by basing learning on lessons drawn from actual practice, students can better understand the range of options that may be available to them in terms of solutions to a particular problem. this approach is particularly useful in the areas in which i teach—the design and evaluation of information services in organizations.   as we know, evidence based practice provides a structured approach to addressing critical decisions by basing the proposed responses on the best available research in the field. it has been my privilege and pleasure for the past two years to be part of the editorial team that helps to bring much of this relevant research to the field through articles published in the journal of evidence based library and information practice (eblip). as an associate editor for articles, i have the opportunity of seeing manuscripts describing important research conducted around the world in settings as disparate as high tech research university libraries and low tech rural communities. each of the manuscripts that is accepted for publication in eblip provides some special insight into library and information practices throughout the field, and each article can be used by others in the field as a basis for their own research and practice.   in my introductory course on basic research methods, where i used eblip articles extensively as required readings, i tried to cultivate in my students an understanding of the critical research skills required by information professionals to perform successfully in today’s (and tomorrow’s) competitive job market. using evidence based practice as a framework for identifying and articulating the issues to be studied or the problems to be solved helps students to think systematically about the environment in which they are functioning and the needs of the users whom they are serving. articles published in eblip provide examples of different research designs that can be applied in a wide range of settings.   the coming generations of students will see changes in the work place in regard to the acquisition, organization, access, preservation, and use of information that we can only begin to envisage; our role as educators is to provide these students with the tools that will allow them to respond nimbly to the changes in our environment that are bound to occur. many of these tools will, and should, rely on lessons learned from others in the field, and eblip plays a key role in disseminating these important lessons.   commentary   collaborating to increase the evidence base in library and information practice   margaret henderson associate professor director, research data management vcu libraries virginia commonwealth university richmond, virginia, united states of america email: mehlibrarian@gmail.com   received: 15 jan. 2017    accepted: 26 mar. 2017      2017 henderson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   as i was thinking about ebl while preparing to come and give this talk, i started thinking about evidence based practice in general. i know you didn’t come here for a history lesson, but i think we can learn something from a quick look at how ebp has evolved.   the most common early examples used when discussing ebm are from the nineteenth century: john snow and florence nightingale. john snow (wikipedia, n.d.) is famous for figuring out that cholera was spread through water, eventually resulting in the famous pump map of the london cholera outbreak in 1854. in 1856, florence nightingale started pushing for changes in medical care, notably sanitation, to save the lives of soldiers and patients in hospitals, based on data collected during her work as a nurse during the crimean war (mcdonald, 2001). i have also seen references to earlier medical research, especially epidemiology research.   when reviewing these early examples, i noticed that they really just applied the scientific method, which can be used by anyone (nerdy baby, 2017):   1.       make an observation 2.       form a hypothesis 3.       perform the experiment 4.       analyze the data 5.       report your findings 6.       invite others to reproduce the results   what we now recognize as evidence based medicine, including the familiar evidence pyramid (wikimedia commons, 2016), 5 a’s (ask, acquire, appraise, apply, assess, e.g. university of north carolina health sciences library, 2016 ) and venn diagrams showing ebm as the intersection of evidence, patient, and doctor expertise (weill cornell medical college medical library, n.d.), involves more than just the scientific method. ebm involves systematically collecting all the research and literature on a particular clinical query together, and analyzing it to provide a complete picture and make a better decision on the care to provide a patient. claridge and fabian (2005) provide a good history of evidence based medicine and the term itself doesn’t show use in medline until 1993, then shows a linear increase in medline citations starting in 1995.    despite the fairly recent origin of ebm – or at least to me it seems recent since i remember learning about ebm in 1998 at a meeting of medli, the medical & scientific libraries of long island (n.d.) during my time as director at the cold spring harbor laboratory library and archives – the idea of systematically analyzing multiple research papers on a topic to decide on the best action for a problem has been adopted by many disciplines. the urban institute (turner, 2014) is using evidence based policy making, specifically for government spending and tax policies. the laura and john arnold foundation (n.d.) is looking at policies for social problems such as unemployment or homelessness. evidence is being used for management styles and decision making (barends, rousseau, & briner, 2014). it is being used to design better schools and hospitals (whitemyer, 2010). evidence is being considered in education as well; in some cases to decide on the best methods for teaching, but also, using student data to design specific strategies for success, although there is some concern about privacy issues in these areas (killian, 2013).   and of course, evidence is used in librarianship. we can see the beginnings of evidence based practice in early papers by jonathan eldredge (2000), andrew booth (2002), and ellen crumley and denise koufogiannakis (2002). having spent most of my career as a biomedical sciences librarian, i’m happy that all of these are in journals covering medical librarianship but, as jonathan eldredge pointed out in 2000, we were helping to teach ebm (as evidenced by my class back in 1998), so it isn’t a stretch that medical librarians should start thinking about evidence for their own work around that time.   in 2006, eblip was the first journal to focus on evidence based librarianship, with the object of helping librarians make more informed decisions based on the best available evidence. the evidence summaries are especially helpful for busy librarians, saving time by providing a synopsis of the important points in a research article, and covering a wide range of topics and journals.   in 2012, denise koufogiannakis wrote about the state of lis systematic reviews and announced the wiki she had created to gather known systematic reviews in library and information studies. looking over the reviews in the lis systematic review wiki (http://lis-systematic-reviews.wikispaces.com/welcome) gives us an idea of the state of evidence based practice in library and information studies. we can see that many of the systematic reviews are in specific niche areas, not surprising considering the wide range of subjects covered by lis. as marcia bates (2015) pointed out, lis covers all disciplines, and includes many information sub-disciplines, so it is always going to be a challenge to find commonalities that can be compared in a systematic review. but overall, there aren’t that many systematic reviews, given that librarians have been talking about evidence based practice for about 20 years.   the problem is the lack of data and research. as the leader of one of the mla research agenda systematic review projects (eldridge, ascher, holmes, & harris, 2012), i know first-hand that finding appropriate evidence to answer a question can be difficult.   our question is: “do health sciences libraries and librarians have any measurable (statistically significant) positive impacts on consumer health, the outcomes of medical care, the productivity of biomedical researchers and the knowledge obtained by graduates of biomedical and health sciences training programs, and at what total cost?” we presented a poster at mla 2015 (henderson et al., 2015) and we still aren’t finished (team members finding the time to work on a project outside their usual job duties can be a problem with some collaborative projects). even though we started with a set of over 4000 articles, we ended up with just over 100 that fit all criteria, and even fewer had actual data. and when there was data, the papers covered very dissimilar measures, so there was no way to combine results to bring together a more robust result, as noted by wagner and byrd (2004) in a systematic review of clinical medical librarian effectiveness.    the bottom line: we need to do more lis research that results in data!   so, when i was planning a survey on the research data needs of virginia commonwealth university (vcu) faculty earlier this year, i pulled questions from other papers to allow me to do some comparisons. this is a step towards being able to compare results and pull together a large body of evidence. the following are a few of the questions i asked, and the responses i received, compared with one of the papers i used as inspiration.   the results on data formats faculty are collecting compared to those at northwestern, in a 2015 report of a survey by cunera buys and pamela shaw are shown in figures 1 and 2.   i can compare most of the reasons for not sharing data with the responses to a 2015 study by federer, lu, joubert, welsh, and brandys at the nih library (2015). (tables 1 and 2)   responses on how much data is being stored, and where it is being stored, can be compared with results from katherine akers and jennifer doty at emory in 2013 (table 3, figures 3 and 4):     figure 1 faculty data formats from vcu survey.     figure 2 type/format of data from northwestern study (buys and shaw, 2015).     table 1 reasons for not sharing data, from vcu faculty survey privacy or protection of subjects 148 data require secure/restricted access 94 data might be misinterpreted/misused 71 to protect my intellectual property rights 71 might not get credit (e.g., citation, acknowledgment, authorship) 59 requires too much time/effort 50 not licensed to share data 47 data of little value to others 26 i don't know where to share it 25 commercialization/patent concerns 25 no repository exists for my type of data 21 other 5     table 2 reasons for not sharing data from nih study (federer et al., 2015)     table 3 current data storage amounts by faculty at vcu approximately how much digital research data are you currently storing? (choose one)   frequency percent valid megabyte range 40 16.9 gigabyte range 72 30.5 terabyte range 43 18.2 don’t know 76 32.2 total 231 97.9 total 236 100.0     figure 3 current data storage locations of vcu faculty.     figure 4 digital research data storage amounts and locations from emory survey (akers & doty, 2013).     but it isn’t enough to just look for other studies and make comparisons; we need to be more intentional about what we are doing.   we need to collaborate from the start to plan research that is applicable to multiple libraries, such as a recent study looking at the evolving needs of researchers in information and data management (cain, cheek, kupsco, hartel, & getselman, 2016). the authors hope to develop new information service models that can be used by others, based on surveys conducted at two research-intensive universities. marshall, et al. (2013), conducted a large, multi-site study to learn about the impact of medical libraries on patient care. this large collaboration included physicians, residents, and nurses at 118 hospitals, making the results more generally applicable than a single site study.   we need to make sure that, where possible, we change our promotion and tenure requirements to include credit for open and shared resources. and we make sure that we include time for research in our work plans.   we need to relate our research to assessment measures so it can be a bigger part of the job we do. librarians often shy away from research because there isn’t enough time (fox, 2007), so building research into the assessment we should already be doing is a good way to get started.   we need to relate our research to the research interests of faculty outside of the library, especially with the current focus on interdisciplinary research. our research will be more useful if it can inform decisions outside of the library, and have impact throughout our institutions. there are issues about student success or researcher support that are common everywhere and we should do the work together. yesterday, i quickly tweeted a question about who worked with faculty outside of the library, and on what, just to give you a few examples:    cynthia hudson vitale works on data citation practices, ehr research on transparency and qualitative health data sharing. plus systematic reviews. roy brown works on nurses magnet status research and teaching ebn in curriculum. plus systematic reviews. patricia anderson works on bibliometrics, comics, wearable tech, systematic reviews, video game design, and educational design. abigail goben works on scholarly impact, bibliometrics, copyright education, electronic health record federated search tools, and health information literacy.   and to make sure we can publish and share any insights we have from surveys or questionnaires we use, we should always get irb/ethics board approval before starting our research.   we need to make our articles, research instruments, and data open so research can be replicated using the same survey instruments or methods, and so that data can be combined to create a more robust evidence base. the acrl value of academic libraries bibliography (http://acrl.ala.org/valueography /) allows sharing of research in a blog post, but there is no good way to find a listing of all the studies submitted and no way to share data. i recommend setting up an osf (https://osf.io) project or depositing materials in the new socarxiv preprints (https://osf.io/preprints/socarxiv). librarians are pushing for open access; some researchers are pushing for open science. wouldn’t it strengthen our position to have our research out there where it can be scrutinized?   the reasons we give to researchers in other areas for sharing are the same for us. sharing research methods and results:   helps to avoid duplication, thereby reducing costs and wasted effort. promotes scientific integrity and debate. enables scrutiny of research findings and allows for validation of results. leads to new collaborations between data users and data creators. improves research and leads to better science. enables the exploration of topics not envisioned by the initial investigators. permits the creation of new datasets by combining data from multiple sources. increases citations. a study by piwowar, day, and fridsma (2007) showed a 69% increase in citations. and we can use the same repositories to make our data available.   dataverse http://dataverse.org/ figshare http://figshare.com/ open science framework https://osf.io/   zenodo https://zenodo.org/   your institutional repository   i have been a librarian for 30 years now, and i’ve seen many changes, but i’m still enthusiastic about the profession and see a wonderful future for us. i hope you will see the value in lis research and feel enthusiastic for our future as well.   references   akers, k. g., & doty, j. (2013). disciplinary differences in faculty research data management practices and perspectives. international journal of digital curation, 8(2), 5-26. http://dx.doi.org/10.2218/ijdc.v8i2.263   barends, e., rousseau, d.m., & briner, r.b. (2014) evidence-based management: the basic principles. amsterdam: center for evidence-based management. retrieved from https://www.cebma.org/wp-content/uploads/evidence-based-practice-the-basic-principles-vs-dec-2015.pdf   bates, m.j. (2015). the information professions: knowledge, memory, heritage. ir information research, 20(1). retrieved from http://informationr.net/ir/20-1/paper655.html   booth, a. (2002). from ebm to ebl: two steps forward or one step back? medical reference services quarterly, 21(3), 51-64. http://dx.doi.org/10.1300/j115v21n03_04   buys, c. m., & shaw, p. l. (2015). data management practices across an institution: survey and report. journal of librarianship and scholarly communication, 3(2). http://dx.doi.org/10.7710/2162-3309.1225   cain, t. j., cheek, f. m., kupsco, j., hartel, l. j., & getselman, a. (2016). health sciences libraries forecasting information service trends for researchers: models applicable to all academic libraries. college & research libraries, 77(5), 595-613. http://dx.doi.org/10.5860/crl.77.5.595   claridge, j.a. & fabian t.c. (2005). history and development of evidence-based medicine. world journal of surgery, 29(5), 547-553. http://dx.doi.org/10.1007/s00268-005-7910-1   crumley, e. and koufogiannakis, d. (2002), developing evidence-based librarianship: practical steps for implementation. health information & libraries journal, 19(2), 61–70. http://dx.doi.org/10.1046/j.1471-1842.2002.00372.x   eldredge, j. d. (2000). evidence-based librarianship: an overview. bulletin of the medical library association, 88(4), 289-302.   eldredge, j. d., ascher, m. t., holmes, h. n., & harris, m. r. (2012). the new medical library association research agenda: final results from a three-phase delphi study. journal of the medical library association: jmla, 100(3), 214-218. http://dx.doi.org/10.3163/1536-5050.100.3.012   federer, l. m., lu, y.-l., joubert, d. j., welsh, j., & brandys, b. (2015). biomedical data sharing and reuse: attitudes and practices of clinical and scientific research staff. plos one, 10(6), e0129506. http://dx.doi.org/10.1371/journal.pone.0129506   fox, d. (2007). finding time for scholarship: a survey of canadian research university librarians. portal: libraries and the academy, 7(4):451–462. http://dx.doi.org/10.1353/pla.2007.0041   henderson, m. e., crum, j. a., fatkin, k. j., gagnon, m.-m., nguyen, t., taylor, m., & vrabel, m. (2015). do health sciences libraries and librarians have an impact on the cost of health care and research? a systematic review. annual meeting, medical library association, austin, tx, usa. retrieved from http://scholarscompass.vcu.edu/libraries_present/43/    killian, s. (2013). top 10 evidence based teaching strategies. in the australian society for evidence based teaching. retrieved from http://www.evidencebasedteaching.org.au/evidence-based-teaching-strategies/   koufogiannakis, d. (2012). the state of systematic reviews in library and information studies. evidence based library and information practice, 7(2), 91-95. http://dx.doi.org/10.18438/b8q021   laura and john arnold foundation (n.d.) evidence-based decision making. retrieved from http://www.arnoldfoundation.org/initiative/evidence-based-policy-innovation/evidence-based-decision-making/   marshall, j. g., sollenberger, j., easterby-gannett, s., morgan, l. k., klem, m. l., cavanaugh, s. k., oliver, k.b., thompson, c.a., romanosky, n., & hunter, s. (2013). the value of library and information services in patient care: results of a multisite study. journal of the medical library association: jmla, 101(1), 38-46. http://dx.doi.org/10.3163/1536-5050.101.1.007   mcdonald, l. (2001). florence nightingale and the early origins of evidence-based nursing. evidence based nursing, 4(3), 68-69. http://dx.doi.org/10.1136/ebn.4.3.68   medical and scientific libraries of long island (n.d.). medli’s history: programs. retrieved from https://medli.org/about/history/programs/   nerdy baby (2017). every baby knows the scientific method. retrieved from http://www.nerdybaby.com/every-baby-knows-the-scientific-method-mini-poster-11x17/   piwowar, h.a., day, r.s., & fridsma, d.b. (2007). sharing detailed research data is associated with increased citation rate. plos one, 2(3): e308. http://dx.doi.org/doi:10.1371/journal.pone.0000308   turner, m.a. (2014). job market and labor force. in urban institute. retrieved from http://www.urban.org/urban-wire/bipartisan-call-better-evidence-inform-policy   university of north carolina health sciences library (2016). surgical residents: ebm review and practice. retrieved from http://guides.lib.unc.edu/c.php?g=204277&p=1347834#s-lg-box-4079097   wagner, k. c., & byrd, g. d. (2004). evaluating the effectiveness of clinical medical librarian programs: a systematic review of the literature. journal of the medical library association, 92(1), 14–33.   weill cornell medical college medical library (n.d.). evidence-based medicine. retrieved from http://med.cornell.libguides.com/ebm   whitemyer, d. (2010). the future of evidence-based design. in international interior design association. retrieved from http://www.iida.org/content.cfm/the-future-of-evidence-based-design   wikimedia commons (2016). research design and evidence.svg. retrieved from https://commons.wikimedia.org/wiki/file:research_design_and_evidence.svg   wikipedia (n.d.). "john snow". retrieved from https://en.wikipedia.org/w/index.php?title=john_snow&oldid=759493613       ebl 101   research methods: bibliometrics   virginia wilson librarian, murray library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   originally published in: evidence based library and information practice, 7(3), 121–123. https://ejournals.library.ualberta.ca/index.php/eblip/article/view/17975/14278     received: 01 aug. 2012 accepted: 08 aug. 2012      2016 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   bibliometrics! metrics about biblios: a set of measurements about a book, or books, or text. the oxford english dictionary defines bibliometrics as “the branch of library science concerned with the application of mathematical and statistical analysis to bibliography; the statistical analysis of books, articles, or other publications” (2012). as a research method, beck and manuel state that “essentially, if you can count something without too much debate over what it is—and it has something to do with any type of recorded information source—it could become the subject of a bibliometric study” (2008, p. 165). bibliometrics can also be referred to as informetrics, webometrics, scientometrics, and cybermetrics. the different terms basically reflect the types of information to which the analysis is applied.   bibliometrics involves the measurement of data not intrinsic to the text, i.e., the text does not need to be read in order for analysis to be conducted. instead, extrinsic measures such as affiliation of author, word frequency, usage statistics, and, most commonly citations are measured. there is a school of thought that combines bibliometrics with content analysis (extrinsic and intrinsic) but for the purposes of this column, i intend to focus on bibliometrics alone.   while bibliometrics has been around for decades, and “is one of the oldest research methods in library and information science,” it’s only since the advent of high powered computer technology that bibliometrics has exploded (beck & manuel, 2008, p. 166). “bibliographic databases, citation indexes, and statistical programs” have added ease, depth, and breadth to bibliometric analysis (powell & connaway, 2004, p. 63). another major development for bibliometrics has been the internet and the publishing of “an increasingly broad range of research-related documents, from articles to email discussion lists, allowing the creation of a range of new metrics relating to their access and use” (thelwell, 2008, p. 605).   a quick scan of the articles returned on a search for “bibliometrics” and “librarian research” in the library and information science (lisa) database shows a variety of ways that this research method has been used:   sengupta, i. n. (1990). bibliometrics and the identification of core periodicals. herald of library science, 29 (3-4), 226-245.   mancall, j. c., & drott, m. c. (1979). materials used by high school students in preparing independent study projects: a bibliometrics approach. library research, 1(3), 223-236.   houston, w. (1983). the application of bibliometrics to veterinary science primary literature. quarterly bulletin of the international association of agricultural librarians and documentalists, 28(1), 6-13.   hersberger, j., & demas, c. (2001). the current state of public library research in select peer reviewed journals: 1996-2000. north carolina libraries, 59(1), 10-14. retrieved 20 aug. 2012 from http://www.ncl.ecu.edu/index.php/ncl/article/viewfile/288/306   powell and connaway pulled together applications of bibliographic research outlined by a number of authors into the following list:   improving the bibliographic control of a literature identifying a core literature, especially journals classifying a literature tracing the spread of ideas and growth of a literature designing more economic information systems and networks improving the efficiency of information handling services predicting publishing trends describing patterns of book use by patrons developing and evaluating library collections (2004, p. 63)   bibliometrics can also be embraced by librarians as a core service relating to the scholarly communication process. by honing skills in bibliometric analysis, librarians can develop expertise which “can be used to help [their] faculty prove their scholarly contribution and achieve success in their tenure process” (kear & colbert-lewis, 2011, p. 470).   as always, this column can only give a minimum overview of any research method, and immersion into the method is necessary when undertaking the research. here are some resources that can get you started on your bibliometric journey:   andrés, a. (2009). measuring academic research: how to undertake a bibliometric study. oxford: chandos publishing.   de bellis, n. (2009). bibliometrics and citation analysis: from the science citation index to cybermetrics. lanham, md: scarecrow press.   thelwall, m. a. (2009). introduction to webometrics: quantitative web research for the social sciences. san rafael, ca: morgan & claypool publishers.   wolfram, d. (2003). applied informetrics for information retrieval research. westport, ct: libraries unlimited.   it seems like a logical progression to go from the tried and true method of bibliometrics to a newer and quicker type of analysis called altmetrics, so that’s what i’ll look at next time. and if you have any comments, questions, or suggestions for this column, please comment on this article by signing in to the open journal system as a reader and joining in the conversation.   references   beck, s. e., & manuel, k. (2008). practical research methods for librarians and information professionals. new york, ny: neal-schuman.   kear, r., & colbert-lewis, d. (2011) citation searching and bibliometric measures: resources for ranking and tracking. college and research library news, 72(8), 470-474. retrieved 21 aug. 2012 from http://crln.acrl.org/content/72/8/470.full.pdf+html    powell, r. r., & connaway, l. s. (2004). basic research methods for librarians (4th ed.).westport, ct: libraries unlimited.   thelwell, m. (2008). bibliometrics to webometrics. journal of information science, 34(4), 605-621. doi:10.1177/0165551507087238   bibliometrics. (2012). in oxford english dictionary. retrieved 21 aug. 2012 from www.oed.com/view/entry/241665.   research article   a mismatched group of items that i would not find particularly interesting: challenges and opportunities with digital exhibits and collections labels   melissa harden product owner, center for research computing university of notre dame notre dame, indiana, united states of america email: mharden@nd.edu    anna michelle martinez-montavon online education and technology consultant indiana university south bend south bend, indiana, united states of america email: ama9@iu.edu     mikala narlock director, data curation network university of minnesota minneapolis, minnesota, united states of america email: mnarlock@umn.edu   received: 21 june 2022                                                                  accepted: 18 oct. 2022      2022 harden, martinez-montavon, and narlock. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30194   abstract   objective – the authors sought to identify link language that is user-friendly and sufficiently disambiguates between a digital collection and digital exhibit platform for users from a r1 institution, or a university with high research activity and doctoral programs as classified in the carnegie classification of institutions of higher education.   methods – the authors distributed two online surveys using a modified open card sort and reverse-category test via university electronic mailing lists to undergraduate and graduate students to learn what language they would use to identify groups of items and to test their understanding of link labels that point to digitized cultural heritage items.   results – our study uncovered that the link terms utilized by cultural heritage institutions are not uniformly understood by our users. terms that are frequently used interchangeably (i.e., digital collections, digital project, and digital exhibit) can be too generic to be meaningful for different user groups.   conclusion – because the link terms utilized by cultural heritage institutions were not uniformly understood by our users, the most user-friendly way to link to these resources is to use the term we—librarians, curators, and archivists—think is most accurate as the link text based on our professional knowledge and provide a brief description of what each site contains in order to provide necessary context.     introduction   the main library website serves as a prominent access point for the entire enterprise of library resources, including print and online materials, services, and spaces. as polger (2011) described “[t]he library website is a living document” (para. 1), changing as new information, resources, services, or features are made available. because a library website makes a wide range of information available to users, it is imperative that the resources linked from the site are labeled, described, and contextualized in a meaningful way. in order to allow users to make informed decisions about which links are relevant to their needs, link labels must make sense.   at the time this study was conducted, there was a link in the main navigation of our library’s website labeled “digital collections,'' which pointed to a platform that included digital exhibits from our rare books department entitled “digital exhibits and collections.” at the same time, we were about to launch a new site called “marble,” a platform for providing access to digitized cultural heritage materials from our campus art museum, university archives, and rare books departments. we needed to determine an appropriate link label for this new site that succinctly and appropriately described it without introducing confusion. the digital exhibit platform primarily featured digital exhibits—interactive sites that feature curatorial text and collection highlights—while the new site primarily featured digital collections– akin to a catalog of items that are made available with basic descriptive data. this marked the first time our library had two separate platforms for these two distinct use cases. as a result, the name of the original platform, digital exhibits and collections, would no longer be accurate because the two functions were split into two different sites, with collections moved to marble. therefore, we needed to identify link labels for both of these sites that accurately, meaningfully, and concisely described and differentiated the two resources in a way that made sense to users. we also wanted to ensure that any link language leveraged would be useful and understandable to both novice and advanced users, as our campus population includes undergraduate and graduate students, postdoctoral scholars, and faculty members from a wide variety of disciplines.   the need to add a link to the new digital collections site in the library website’s main navigation provided an opportunity to learn what link labels users thought best described the types of specialized resources found on the digital exhibits platform and digitized collections site. in our local context, because one site primarily featured digital collections and the other platform primarily featured digital exhibits, we needed to find out what meaning, if any, the term “digital exhibits” had for our users to determine if this would be a useful link label. we sought to test language that would resonate with users and strike a balance between the overly general and overly specific.    we first conducted an environmental scan of association of research library (arl) websites to see if there was consistent link label language used by our peer libraries for digital exhibits and digital collections links. finding none, we looked to the literature to see if other studies had been conducted for this purpose. because no study was identified that tested these specific terms (i.e., “digital collections” and “digital exhibits”) against each other, we determined that we needed to design our own study for this purpose. therefore, we designed a user study to help us answer the question, “what meaningful, user-centered, and concise link language accurately describes and differentiates between two sites?” specifically, we were interested in testing how well the terms “digital collections” and “digital exhibits” resonated with users, including both undergraduate and graduate students. the intention of this dual audience was to see if there was a difference between novice and advanced users. given that the library website is the main portal for the entire student body, we wanted to be sure that any link language used would be understandable for all.   literature review   for the purposes of this research, we focused on the published articles and presentations on user experience testing that focus on link labels (or link language) and card sorting activities. this selected literature review provided a solid grounding in the theories and best practices of both to ensure we were not duplicating research efforts and allowed us to build on previous work so we could be thoughtful and intentional in how we designed our surveys. because our study falls at the intersection of link labels and card sorting exercises, we have divided our literature review into these two categories below.   link labels   link labels are the words or phrases that display a hyperlink. it is important that link titles contain meaningful natural language but are also specific enough to give a clear impression of where those links point. there are several factors users consider when determining whether or not to click on a link, one of which is the link label (budiu, 2020). through various studies, it is clear that using terminology that is too broad to be helpful has been identified as an issue in library card sorts (duncan & holliday, 2008; hennig, 2001; lewis & hepburn, 2010). alternatively, using very specific language may cause link text to be too long (dickstein & mills, 2000), leading to visual clutter. while there may be some variation in findings based on individual library context, one common theme is that links with branded names are unlikely to be interpreted correctly by users (gillis, 2017; hepburn & lewis, 2008; kupersmith, 2012). based on this information, we did not consider using the branded names—the institutionally-specific name used to market and promote a resource or service—of our digital exhibits platform and collections site as potential link labels on the main library website and sought to test language that would resonate with users and strike a balance between the overly general and overly specific.   a user-centered approach means using website language that resonates with users; however, as francoeur (2021b) pointed out, it can be challenging “to balance the demands to be concise, clear, and understandable” (para. 2). this is especially true when libraryor archives-specific language or jargon is used (burns et al., 2019). the terms used to describe materials by information professionals have distinct meanings: reducing them to common terms is not only a disservice to our profession and unique skill set, but also to our users, as overly generic terms can be equally confusing. moreover, leveraging the appropriate term can serve as an educational opportunity, teaching even casual browsers of content the nuances between terms, leading to more efficient searching and use of materials in the future. burns et al. (2019), described the importance of the challenge accurately: “more than just an issue of semantics, the branding and labeling we employ in digital library interfaces plays a critical role in helping users find, utilize, and understand archival and special collections in the online environment” (p. 5).   the study burns et al. (2019) conducted presented the closest match to the information we wanted to discern in our own case. in this study, the researchers reviewed terminology used by arl member libraries to identify which terms were most commonly used to label digitized cultural heritage collections. they identified a variety of terms, noting that the inconsistent use of terms may be confusing to users. burns et al. designed a survey-based study to identify the terminology that users were “most likely to associate with different materials commonly found in digital libraries” (p. 5) and the terms that “are potentially confusing and likely to be misunderstood by users” (p. 6). their task-based questions asked users where they would click to find various items that typically appear in digital libraries and were meant to help identify terminology that would disambiguate the term “digital library.” the label options provided to respondents included digital history collections, digital library, digital archives, and digital collections (burns et al., 2019, p. 7). their results suggested that there is little to no consensus about the interpretation of these various labels, highlighting the challenge that exists in meeting various web usability goals with respect to link labels and terminology.   while the study conducted by burns et al. (2019) addressed a very similar question to ours and provided a useful framework for the design of our study, theirs did not test or address the use of “digital exhibit” as a link or navigation label. similarly, there was no mention of best terms for digital collections and digital exhibits in the document “library terms that users understand” (kupersmith, 2012). in our local context, because one site primarily features digital collections and the other platform primarily features digital exhibits, we needed to find out what meaning, if any, the term “digital exhibits” has for our users to determine if this would be a useful link label. a deeper dive into the relevant literature indicated that the term “digital exhibits” has not been adequately studied from a user-experience perspective. when studying digital libraries, the researchers in usability studies have tended to focus on terms like “digital library” or “digital archive” without much analysis as to why those were the terms selected (e.g., burns et al., 2019; kelly, 2014). instead, terms like “digital collections” and “digital exhibits” can be, and often are, used interchangeably in library literature without much analysis. in some instances, the words are used synonymously, e.g., the book digital collections and exhibits is exclusively about exhibits, yet the author published under both terms (denzer, 2015).   for the purposes of this research, we defined digital collections as a catalog of items that are made available with basic descriptive data, and users search and sort as they wish; digital exhibits, on the other hand, are highly mediated online experiences that feature specially selected items, extensive curatorial text, and often a predetermined path to explore the content.   card sorting   card sorting has been used by several libraries to test site structure and nomenclature, and is in fact a frequent testing option for libraries (brucker, 2010). a key benefit of card sorting is that users can propose their own organizational and mental models for information and are not influenced by pre-existing structure (faiks & hyland, 2000). there are numerous variants of the card sorting exercise, all of which are used for different purposes (spencer, 2009). for example, some studies leveraged an open card sorting activity, which allowed testing participants to sort cards into categories they create (e.g., dickstein & mills, 2000; lewis & hepburn, 2010; robbins et al., 2007; sundt & eastman, 2019; whang, 2008). others have used closed card sorting tests, in which users were provided categories and asked to put content into the pre-defined groupings (e.g., diller & campbell, 1999; faiks & hyland, 2000; guay et al., 2019; hennig, 2001; paladino et al., 2017; rowley & scardellato, 2005). others have used a hybrid approach (e.g., paladino et al., 2017), in which participants could sort into predefined groups or create their own. lastly, while card sorting is commonly used for in-person user experience testing, it has also been leveraged remotely (e.g., ford, 2013).   reverse category tests have been used by some academic library teams to validate or expand upon results from prior card sorting activities in preparation for larger website redesign projects (hennig, 2001; sundt & eastman 2019; whang, 2008). in these cases, users were asked where they would click to find specific items, resources, types of information, and others, and were provided different categories as answer options. these categories corresponded with main navigation categories.   aims   in order to determine a concise, specific, and non-duplicative term to label the newly launching digital library platform, marble, and relabel the existing digital exhibits platform, digital exhibits and collections, the authors set out to answer the question: which link terms do our users not only understand, but also find meaningful? how can we, as librarians and information professionals, sufficiently differentiate between terms like “digital collections” and “digital exhibits”— terms that are often used interchangeably but have specific meanings?   method   to address this challenge, our team developed a multi-phased, institutional review board (irb) approved study that was conducted in spring 2021, when the authors all worked at hesburgh libraries, university of notre dame. first, with the help of student workers, we conducted an initial review of different link titles used on the websites of arl member libraries. this work was critical to confirm that there was no consensus on or consistency in application of various terms, as well as to identify potential terms to test in parts two and three of our study. secondly, we developed an initial survey that was a modified open card sort—we provided users with items already sorted into groups and asked them to supply labels for the groups. thirdly, using the terms provided from the first survey, as well as from the arl members’ websites, we developed a second survey, a reverse category test, in which users were asked to identify which items or features they would expect to find based on various link terms. data was analyzed to identify patterns and themes, and ultimately to inform decisions about link language on the main library website. this section provides more details on our methods and decisions.   arl link language environmental scan   we chose to focus the initial scan on arl libraries because they are part of our peer network; many of these libraries dedicated similar time and resources to their digital collections and exhibits. we recruited student workers to browse the websites of all 148 arl libraries and record any link related to digital exhibits or online collections on the home page or main navigation; anything that was online and could be even tangentially related was captured, including terms such as “digital library” and “digital resources.” for sites that leveraged explanatory language to clarify their links (e.g., “this site is for x, y, z”) the title of the link, as well as the additional contextual information, was captured for a holistic view of how these libraries presented their digital collections. while we did capture a few sites with branded names, we did not include those in our analysis, as those would be too specific to the institution and not helpful for our purposes.   user surveys   modified open card sort   as a follow-up to our environmental scan of arl libraries, we conducted two surveys of notre dame students. the first survey was designed to operate like a modified open card sort—we provided users with items already sorted into groups and asked them to supply labels for the groups (see appendix a for full survey). it was critical that the questions on the survey did not include the language we were testing so as to not predispose our users to the language we expected (nielsen, 2009). the first group of items contained screenshots of items and descriptions from the digital collections platform, and the second group of items contained screenshots from portions of digital exhibits. all screenshots were cropped to remove any site branding or logos. for both groups of items, we asked users two open-ended questions: “how would you describe this group of items?” and “what name would you give a group of items like this?” both of these questions were designed to better understand how users interpret these items and elicit potential terminology, without suggesting common library terms, like “collections” or “exhibits.” while the second question directly asked respondents to provide a group label, the first one was purposefully meant to elicit longer responses, with the aim of understanding users’ thinking and collecting additional link label terms to test in the second survey. we did not use a formal coding method to analyze the survey results but rather noted the frequency of the terminology supplied by our users for the link labels as well as in the free-text descriptions of content. we focused on identifying unique terms that were used most often, as well as less-used terms that might appear on library websites.   reverse category test   the second user survey was designed to operate like a reverse category test, much like the one described in the burns et al. (2019) study. the key difference between their study and ours is our inclusion of digital exhibits in the test. the first set of six questions asked students, “if i clicked on a link called [link label], i would expect to see…” where the link label was changed to test different words or phrases. the link labels tested were: “digital collections,” “digital exhibits,” “digital artifacts,” “digital showcase,” “digital archive,” “digital projects,” and “other” where students could provide their own text. we chose these labels based on our own local context (“digital collections” and “digital exhibits” have meanings and scopes that are well understood in our own library system), the student responses from the first survey (“digital artifacts” and “digital showcase”), and our analysis of arl library websites (“digital archive” and “digital projects”).   for each link label, respondents were asked to select all answer options that applied. answer options included: artwork (e.g., photographs, paintings, sculptures), ephemera (e.g., posters, broadsides), e-books, archival or specialized collections (e.g., rare books, manuscripts, diaries, letters), scholarly journals and articles, item details (e.g., metadata), explanatory information about the items, historical and social context of the items, interactive features (e.g., dynamic timelines and maps), related items, datasets and databases, and other (with a write-in option). some of these answer options were included based on student responses from the first survey, and some of them were supplied by the authors.   the second set of questions on the second survey asked: “below are examples of content linked from a library website. what link would you follow to get to each item?” the examples contained screenshots of digitized items from our digital collections and digital exhibit platforms, plus descriptive text and metadata (see appendix a for full survey). in order to minimize the visual differences between the way that items in these two platforms are presented, we used screenshots of just the item image and cut and pasted the exhibit text or collection metadata into the survey platform so that the formatting from the different sites would not influence student answers. the answer options were the same as the labels for the first part of the survey (“digital collections,” “digital exhibits,” “digital artifacts,” “digital showcase,” “digital archive,” and “digital projects” along with an “other” write-in option). students were only able to select one of these choices.    these surveys were distributed online through two key mechanisms: the university-wide weekly update, which reaches all undergraduate and graduate students, and electronic mailing lists of different disciplines, including political science and art history, which reach more targeted groups of undergraduates and graduate students. the goal with using the university-wide email was to reach and solicit input from users that might rarely leverage online cultural heritage materials, such as students from disciplines like business, engineering, or psychology. we chose the political science and art history specific electronic mailing lists because we assumed these students would be familiar with online cultural heritage materials and might represent our more advanced users. by approaching the survey distribution from this perspective, we hoped to represent both novice and advanced users.   for both surveys, students who completed the survey were entered into a chance to win a $50 gift card; for the first survey, one gift card was offered, and for the second, three $50 cards were offered. these incentives were funded as part of a grant from the andrew w. mellon foundation. contact information was collected separately from the survey instrument via a google form to keep responses anonymous.   we reviewed results from the second survey to determine if there were consistent patterns among responses that might suggest link labels and terminology that were commonly understood across user groups.   results   arl link language environmental scan   unsurprisingly, we found most institutions used terms like digital collections, digital archives, and digital library; however, those phrases were not necessarily applied consistently across institutions. for example, link labels of “digital collections” on two different library sites did not necessarily point to the same types of resources. some of these digital collections brought together curated, digitized resources from the library’s archives and special collections, while others also included scholarship from university researchers or journals published by the library or university. we found that links related to digital collections were often located alongside or underneath headings such as “collections” or “specialized collections.” for libraries that provided access to and publicized online exhibits, these were more frequently discoverable alongside digital scholarship projects or events as a companion to physical exhibits.   with no clear consensus on link labels found through the literature or our environmental scan, we decided to conduct two user surveys to learn more about how our students understand the terminology relevant to our question. while there are numerous ways to evaluate digital libraries, we needed to find solutions that would more closely match our circumstances. as has been demonstrated over the past few decades, because “digital libraries are designed for specific users and to provide support for specific activities… [they] should be evaluated in the context of their target users and specific applications and contexts” (chowdhury et al., 2006, p. 671. for that reason, we chose to focus on the needs of our students and test them directly through online surveys.   user surveys   modified open card sort   in the first survey, 52 participants started the survey. twenty individuals did not complete the survey, and incomplete answers, including partials, were removed from the dataset for analysis. one respondent who started the survey noted that these items were “a mismatched group of items that i would not find particularly interesting,” and did not complete the survey. this left a total of 32 responses. respondents were evenly split between graduate and undergraduate students (n=16). notre dame has 8,874 undergraduate students and 3,935 graduate students (university of notre dame, 2022), so our sample represents a heavier skew toward graduate students than the general student body. respondents skewed heavily towards college of arts and letters students (n=32), though this makes sense because the college of arts and letters is the largest college on campus, with 3,000 undergraduate and 1,100 graduate students. the next largest college is the mendoza college of business, with 1,700 undergraduate and 625 graduate students (mendoza college of business, 2022), for about 18% of the student body. in contrast, the school of architecture, the smallest school, only awarded 1.6% of all the degrees conferred at notre dame between july 1, 2020, and june 30, 2021 (office of strategic planning & institutional research, 2022). some students listed multiple affiliations, e.g., college of arts and letters and college of science, which is why the total adds up to more than the 32 participants.   due to the open-ended nature of the survey questions, students’ answers varied wildly. when asked what term they would give to a group of items librarians might refer to as digital collections, many used terms like historical (n=10, 31%), art (n=7, 22%) and artifacts (n=6, 19%). one user also wondered why these items would be grouped together in one space.   when asked to assign a term to what librarians might call a digital exhibit, students often focused on the content of the exhibit instead of the media. we had selected two extant exhibits from our platform: one about women’s right to vote, and one about printed representations of animals. as a result, many students used terms like “socio political” or “drawings” to describe the two. others were again wondering why these items would be grouped together. at least three respondents suggested a variation of “collections” and two suggested “exhibit” or “exhibition.” based on students’ responses to this survey, we identified two terms of interest to test: digital artifacts, because of the frequency in which it showed up in answers to the first set of questions, and digital showcase, because it seemed to capture the respondents’ focus on “art” in both sets of questions.   table 1 first survey respondents by school or college school/college count school of architecture 0 college of arts and letters 21 mendoza college of business 5 college of engineering 2 keough school of global affairs 2 the graduate school 4 the law school 0 college of science 5 total 39   reverse category test   in the second survey, 45 participants started the survey and 3 did not complete it. we once again removed all incomplete answers from the dataset for analysis. of the 42 participants who completed the survey, 31 were undergraduate students, 10 were graduate students, and 1 was a postdoctoral research assistant. respondents again skewed heavily towards the college of arts and letters (n=27). some students listed multiple affiliations, e.g., college of arts and letters and college of science, which is why the total adds up to more than the 42.   table 2 second survey: respondents by school or college school/college count school of architecture 0 college of arts and letters 27 mendoza college of business 5 college of engineering 6 keough school of global affairs 2 the graduate school 8 the law school 0 college of science 10 total 58   the questions in the second survey were more close-ended, which allowed us to observe some interesting patterns. when we asked users what they would expect to see in each of the six potential link labels (digital collections, digital exhibits, digital artifacts, digital showcase, digital archive, and digital projects), we found that many expected to find archival or specialized collections in digital collections (n=36), digital artifacts (n=34), and digital archives (n=33), while slightly fewer expected to find those materials in digital exhibits (n=27). more than half the students also expected to find e-books (n=26) and scholarly journals and articles (n=24) in digital collections, and a significant number of students expected to find scholarly journals and articles (n=22) and datasets and databases (n=15) in digital archive.   when it came to features that librarians, archivists, and curators often associate with our work, student expectations did not always line up. while a majority of students did expect to see explanatory information about the items (n=22) and historical and social context of the items (n=26) in digital exhibits, fewer expected to see item details (e.g., metadata) in any of the link labels (n= between 8 and 10).   there was a wide range of responses to the questions that asked students to apply labels to specific examples of digitized items. in other words, there was no identifiable pattern, suggesting that there is not widespread shared meaning of these terms.   table 3 aggregated responses to the question “what would you expect to see if you clicked on the following link?”   table 4 aggregated responses from the second survey to the question, “what link would you click to get to each of the following?” item name digital collections digital exhibits digital artifacts digital showcase digital archive digital projects where it lives on our website shoes n=8 (19%) n=24 (57%) n=2 (5%) n=3 (7%) n=4 (10%) n=1 (2%) digital collections peru’s first newspaper n=4 (10%) n=8 (19%) n=17 (40%) n=1 (2%) n=12 (29%) 0 digital exhibits collegiate jazz festival n=8 (19%) n=5 (12%) n=4 (10%) n=5 (12%) n=17 (40%) n=3 (7%) digital collections meeting d’aviation nice. n=14 (33%) n=7 (17%) n=4 (10%) n=3 (7%) n=13 (31%) n=1 (2%) digital collections the author; the journal n=9 (21%) n=12 (29%) n=11 (26%0 0 n=7 (17%) n=3 (7%) digital exhibits the ferrell bible n=6 (14%) n=6 (14%) n=16 (38%) n=3 (7%) n=11 (26%) 0 digital exhibits new edition of a general collection of the ancient irish music n=16 (38%) n=4 (10%) n=7 (17%) n=4 (10%) n=8 (19%) n=3 (7%) digital collections   discussion   the results of our study suggested that, among our users, there was no consistent understanding of the terms most commonly used by libraries to describe the types of digitized cultural heritage items found on our digital exhibits and collections platforms. without a common understanding or interpretation of these terms, using those terms alone as link labels would not be enough for most users to clearly understand the types of information they would find by following those links. similarly, the other terms suggested by our students were not well-understood across the board in the follow-up survey. there were several terms that showed up more frequently in the student responses, such as “archives” and “artifacts.” however, these terms have specific meaning to librarians, curators, archivists, and other researchers. as mentioned earlier, using them to describe our digitized and contextualized content would not be entirely accurate and may in fact cause confusion for more advanced researchers who have specific associations with these terms. as such, we would not use these terms as link text to broadly label entire digital collections or exhibits sites because of the additional confusion this would cause for internal workflows and the work of another key constituency: advanced researchers.   additionally, more graduate students selected the term “digital projects” than undergraduates in the second survey. while this term seemed to resonate well with this small group of users, it is an all-encompassing term that may not provide enough information to users about what they might find at a link with that label. other researchers using card sorting have found that some terms can be so broad and vague as to be unhelpful, such as “resources” (hennig, 2001) and “services” (hennig, 2001; duncan & holliday, 2008). we suspect that “digital projects” might be one such term.   while beyond the scope of this study, an interesting data point emerged related to scholarly articles, e-books, and datasets: nearly half of respondents indicated they would expect to see e-books and scholarly journal articles in digital collections (e-books, n=26, and scholarly journal articles, n=24) and digital archives (e-books, n=19, and scholarly journal articles, n=22). a significant number of participants would expect to see datasets and databases in digital archives (n=15) (see table 3). future research could explore this to better understand why users expected to find datasets and databases in digital archives, and whether it, too, is a term that is poorly understood.   based on these results, we determined the most user-friendly way to link to these resources was to use the term we—librarians, curators, and archivists—think is most accurate as the link text based on our professional knowledge and provide a brief description of what each site contains in order to provide necessary context. for example, the link entitled “digital exhibits,” could also include the brief description: “in-depth explorations of a theme using items from our collections, curated by our librarians and staff.” this solution allows librarians and archivists to refer to various digital collections or exhibits sites in a clear, distinct, and consistent manner, and the brief description provides necessary additional context and serves as a teaching tool to help our users understand what we mean by these link label terms. the brief description, when appropriate, can also include words that students might be looking for, such as “archives” and “artifacts,” without making inaccurate claims about our digital exhibits or collections. lastly, this approach also allows our library to leverage specific and meaningful terms to help educate users on library resources. less than a quarter of respondents expected to see descriptive information (metadata) or related items in any of the options (see table 3). in leveraging precise language to concisely describe the links and supplementing that with additional descriptive text, we can educate users not only on the meanings of these words, but also on the different types of resources and support available to them. in other words, while our profession has a tendency to use terms like digital collections and digital exhibits interchangeably, it is critical that we use terms precisely—not necessarily because students intuitively know the difference, but because this is an opportunity to educate users on different ways to access content online.   these descriptions could be added in a variety of ways. the link could be accompanied by a brief phrase or sentence to provide context. this option would require web content or menu structure that allows for links to have additional text next to them. another option is to provide the descriptive text in a tooltip that appears when a user hovers over the link. the method for providing descriptive text could be tested further in a usability study.   limitations   this study has a few limitations. first, this was a relatively small sample size: 32 respondents completed the first survey, and 42 respondents completed the second survey. while there was a mixture of undergraduates and graduate students who responded to each survey, the total number of respondents to each survey overall was not large. additionally, respondents skewed heavily toward affiliation with the college of arts and letters. approaches to research and experience with and awareness of digital collections and digital exhibits may be different among students with different primary college affiliations. due to limited responses from some of the colleges, there were not enough data to be able to determine whether there were any significant disciplinary differences for preferred link labels. finally, these surveys were sent only to students at the university of notre dame. while these survey results represent the thoughts of students at one campus, the results may provide a launching point for other institutions’ usability studies.   conclusion   in this research study, we set out to learn more about users’ understandings of terms related to digital libraries, specifically to disambiguate a digital collections site from a digital exhibits site. following a literature review, the authors conducted an environmental scan of arl libraries’ websites to get a clearer picture of how peer institutions were approaching this distinction. without a consensus, the authors conducted two surveys of undergraduate and graduate students at an r1 institution. the results suggested that there was no clear understanding of various terms among users. we suggested the best path forward in labeling the links of these sites was to provide additional contextual information to help educate users and make links clearer.   these examples demonstrated the importance of partnering user input and feedback with professional expertise. while our first instinct may be to leverage language that is most familiar to some users, this approach not only minimizes our professional contributions and expertise, but also can be confusing to other users. this study affirmed the importance of using meaningful language: while broad terms like “digital project” might be catchy, they are ultimately too broad to be helpful. there was no consensus among the undergraduate and graduate students surveyed as to what these terms might actually lead to.   this study, building on the work of previous scholars (burns, 2019) included examples of digitized items from our library system; however, the terms tested were not necessarily specific to our context. therefore, the results of this study may provide useful guidance or considerations for other libraries and archives attempting to identify appropriate link language on their own organizations’ websites or as a jumping off point for developing their own user studies. as libraries, archives, museums, and other cultural heritage organizations continue to distribute content online, build and implement new services, and even consolidate and sunset previous services, using specific terms to clearly label and disambiguate links will be of continued importance.   author contributions   melissa harden: conceptualization, methodology, writing – original draft, writing – review & editing anna michelle martinez-montavon: formal analysis, investigation, writing – original draft, writing – review & editing mikala narlock: funding acquisition, writing – original draft, writing – review & editing   references   brucker, j. (2010). playing with a bad deck: the caveats of card sorting as a web site redesign tool. journal of hospital librarianship, 10(1), 41-53. https://doi.org/10.1080/15323260903458741   burns, d., sundt, a., pumphrey, d., & thoms, b. (2019). what we talk about when we talk about digital libraries: ux approaches to labeling online special collections. weave: journal of library user experience, 2(1). https://doi.org/10.3998/weave.12535642.0002.102   budiu, r. (2020, february 2). information scent: how users decide where to go next. nielsen norman group. https://www.nngroup.com/articles/information-scent/   chowdhury, s., landoni, m., & gibb, f. (2006). usability and impact of digital libraries: a review. online information review, 30(6), 656-680. https://doi.org/10.1108/14684520610716153   denzer, j. (2015). digital collections and exhibits. rowman & littlefield.   dickstein, r., & mills, v. (2000). usability testing at the university of arizona library: how to let the users in on the design. information technology and libraries, 19(3), 144–151.   diller, k. r., & campbell, n. (1999). effective library web sites: how to ask your users what will work for them. proceedings of the integrated online library systems meeting, 14, 41–54.   duncan, j., & holliday, w. (2008). the role of information architecture in designing a third-generation library web site. college & research libraries, 69(4), 301-318. https://doi.org/10.5860/crl.69.4.301   faiks, a., & hyland, n. (2000). gaining user insight: a case study illustrating the card sort technique. college & research libraries, 61, 349–357. https://doi.org/10.5860/crl.61.4.349   ford, e. (2013). is digital better than analog? considerations for online card sort studies. college & research libraries news, 74(5), 258–261.   francoeur, s. (2021a, march 5). more thoughts on language usage and design. beating the bounds: library stuff i’m thinking about. https://www.stephenfrancoeur.com/beatingthebounds/2021/03/05/more-thoughts-on-language-usage-and-design/   francoeur, s. (2021b, march 4). watching the language parade of our users. beating the bounds: library stuff i’m thinking about. https://www.stephenfrancoeur.com/beatingthebounds/2021/03/04/watching-the-language-parade-of-our-users/   gillis, r. (2017). “watch your language!”: word choice in library website usability. partnership: the canadian journal of library and information practice and research, 12(1), article 1. https://doi.org/10.21083/partnership.v12i1.3918   guay, s., rudin, l., & reynolds, s. (2019). testing, testing: a usability case study at university of toronto scarborough library. library management, 40(1/2), 88–97. https://doi.org/10.1108/lm-10-2017-0107   hennig, n. (2001). card sorting usability tests of the mit libraries’ web site: categories from the user’s point of view. in n. campbell (ed.), usability assessment of library related web sites: methods and case studies (pp. 88–99). lita, american library association.   hepburn, p., & lewis, k. m. (2008). what’s in a name? using card sorting to evaluate branding in an academic library’s web site. college & research libraries, 69(3), 242-251. https://doi.org/10.5860/crl.69.3.242   kelly, e. j. (2014). assessment of digitized library and archives materials: a literature review. journal of web librarianship, 8(4), 384–403. https://doi.org/10.1080/19322909.2014.954740   kupersmith, j. (2012). library terms that users understand. uc berkeley library. http://escholarship.org/uc/item/3qq499w7   lewis, k. m., & hepburn, p. (2010). open card sorting and factor analysis: a usability case study. the electronic library, 28(3), 401–416. http://dx.doi.org/10.1108/02640471011051981   mendoza college of business. (2022). about mendoza. retrieved september 11, 2022, from https://mendoza.nd.edu/about/   nielsen, j. (2009, august 23). card sorting: pushing users beyond terminology matches. nielsen norman group. https://www.nngroup.com/articles/card-sorting-terminology-matches/   office of strategic planning & institutional research. (2022). common data set 2021-2022 [data set]. university of notre dame. https://www3.nd.edu/~instres/cds/2021-2022/cds_2021-2022.pdf   paladino, e. b., klentzin, j. c., & mills, c. p. (2017). card sorting in an online environment: key to involving online-only student population in usability testing of an academic library web site? journal of library & information services in distance learning, 11(1–2), 37–49. https://doi.org/10.1080/1533290x.2016.1223967   polger, m. (2011). student preferences in library website vocabulary. library philosophy and practice, 1-16. https://digitalcommons.unl.edu/libphilprac/618   robbins, l. p., esposito, l., kretz, c., & aloi, m. (2007). what a user wants: redesigning a library's web site based on a card-sort analysis. journal of web librarianship, 1(4), 3-27. https://doi.org/10.1080/19322900802111346   rowley, p., & scardellato, k. (2005). card sorting: a practical guide. access, 2, 26–28   spencer, d. (2009). card sorting: designing usable categories. rosenfeld media.   sundt, a., & eastman, t. (2019). informing website navigation design with team-based card sorting. journal of web librarianship, 13(1), 37–60. https://doi.org/10.1080/19322909.2018.1544873   university of notre dame. (2022). about: notre dame at a glance. retrieved september 11, 2022, from https://www.nd.edu/about/   whang, m. (2008). card-sorting usability tests of the wmu libraries’ web site. journal of web librarianship, 2(2–3), 205–218. https://doi.org/10.1080/19322900802205940   appendix a survey instruments distributed through electronic mailing lists survey 1                 survey 2     if     evidence summary   external and internal citation analyses can provide insight into serial/monograph ratios when refining collection development strategies in selected stem disciplines   a review of: kelly, m. (2015). citation patterns of engineering, statistics, and computer science researchers: an internal and external citation analysis across multiple engineering subfields. college and research libraries, 76(7), 859-882. http://doi.org/10.5860/crl.76.7.859   reviewed by: stephanie krueger head, office of specialized academic services czech national library of technology prague, czech republic email: stephanie.krueger@techlib.cz   received: 30 june 2016  accepted: 19 oct. 2016      2016 krueger. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine internal and external citation analysis methods and their potential applicability to the refinement of collection development strategies at both the institutional and cross-institutional levels for selected science, technology, engineering, and mathematics (stem) subfields.   design – multidimensional citation analysis; specifically, analysis of citations from 1) key scholarly journals in selected stem subfields (external analysis) compared to those from 2) local doctoral dissertations in similar subfields (internal analysis).   setting – medium-sized, stem-dominant public research university in the united states of america.   subjects – two citation datasets: 1) 14,149 external citations from16 journals (i.e., 2 journals per subfield; citations from 2012 volumes) representing bioengineering, civil engineering, computer science (cs), electrical engineering, environmental engineering, operations research, statistics (stat), and systems engineering; and 2) 8,494 internal citations from 99 doctoral dissertations (18-22 per subfield) published between 2008-2012 from cs, electrical and computer engineering (ece), and applied information technology (ait) and published between 2005-2012 for systems engineering and operations research (seor) and stat.   methods – citations, including titles and publication dates, were harvested from source materials and stored in excel and then manually categorized according to format (book, book chapter, journal, conference proceeding, website, and several others). to analyze citations, percentages of occurrence by subfield were calculated for variables including format, age (years since date cited), journal distribution, and the frequency at which a journal was cited. top journals for selected subfields were identified based on the percentages of authors citing them in each dataset and, for interdisciplinary journals, according to how often citations for them appeared in subfield groups.   main results – for each subfield group, distinct patterns emerged for both internal and external analysis in terms of format, currency, and preferred journals. regarding format of material cited, journals were dominant for external citations and ranged between 40% of citations (cs) to 94% (bioengineering) of formats cited. formats were more distributed for internal citations, with ece, seor, and stat exhibiting journal dominance (61%, 30%, and 59% of citations, respectively) and conference proceedings dominant in cs (43%) and ait (30%). regarding currency, almost all cited items (>98% for external citations and 96% for internal citations) were published within the last 50 years, with electrical engineering showing the highest percentage of materials cited within the past five years for external citations (47%). for internal citations, applied information technology illustrated the most use of materials in the five-year timeframe (46%). top journals for each subfield in which only external data were analyzed include journal of biomechanics (bioengineering 54%), engineering structures (civil engineering 47%), water research (environmental engineering 60%). for cs and ait, the top journal was communications of the acm (external cs citations 29%; internal cs 32%; internal ait 36%). for electrical engineering, the top journals were electronics letters (21% external citations) and proceedings of the ieee (50% internal citations). seor was broken into three categories (systems engineering, seor, and operations research), with systems engineering being the top journal according to external citations for the subfield of the same name (48%) and air traffic control quality as the leading seor journal (25% internal citations only). management science (77% external citations only) was the top journal for operations research. top stat journals were annals of statistics (96% internal citations) and journal of the american statistical association (60%). science was the top interdisciplinary journal for external citations (10%) and ieee: transactions on pattern analysis and machine intelligence for internal citations (13%).   conclusion – an approach to citation analysis integrating both internal and external components is useful for institutions aiming to develop balanced stem collections as well as for collection assessment and budgeting purposes and enables adjustment of serial/monograph ratios to create custom local serial/monograph ratio “blends.” in this institution’s case, internal data suggested a 59:41 serial/monograph ratios versus an external data ratio of 75:25, which indicated that a blended ratio of 67:33 might be appropriate for this institution based on an average of both ratios. in the future, cross-institutional collaboration for external analyses would make it easier for institutions to focus on internal analyses in order to develop appropriate local serial/monograph ratio blends.   commentary   citation analysis, considered a branch of bibliometrics (hoffmann & doucette, 2012), has been used in a variety of settings and across disparate populations in an attempt to describe how users interact with resources, making key assumptions in terms of validity that citations represent accurate snapshots of resource use in time and are of high quality (beile, boote, & killingsworth, 2004). as kelly notes in her literature review, many prior citation analysis studies have attempted to apply research findings to inform collection development, but they have used citation sets (i.e., datasets) that are 1) too narrow for use across institutions or disciplines, or 2) too general to be applicable to individual institutional settings. kelly, by including both external (global) and internal (local) datasets, attempts to overcome such limitations and to point the way toward future studies that might be comparable, reproducible, and therefore more broadly valid – all goals which prior studies have failed to achieve (hoffmann & doucette, 2012).    while failing to provide a methodological “holy grail” for reasons regarding sampling outlined below, kelly’s study does follow guidelines developed by hoffmann and doucette (2012) for citation analysis studies: the author clearly describes the rationale for her study as well as the two samples (i.e., datasets) under investigation. she describes the specific steps undertaken to conduct her analysis, enabling reproducibility, and offers straightforward presentation of research results via analysis of variables for well-defined subfields. the presentation of variables includes comparisons between external and internal datasets, the former of which might be re-used and therefore applicable in future studies as a kind of control against which internal citations from other institutions, source types, or disciplines could be compared. reproducibility could have been enhanced with a deeper description of how, for external citations, the varying impact indicators for thomson reuters web of knowledge, isi journal citation index, and scimago journals and country rank were reconciled with one another in the creation of the journal source lists.   one crucial way in which kelly’s approach could be improved in relation to the hoffmann and doucette methodological criteria would be by providing explanations for why the datasets selected could be considered representative samples. in this study, the target thresholds of 1,500 external citations per subfield and 1,200 internal citations per dissertation subfield appear to have been arbitrarily selected; while they might have been chosen as saturation points (hoffmann & doucette, 2012), this is not explicitly stated. and though kelly notes dissertation citations were selected at random, there is no description of the randomization process.   since kelly identifies the importance of conference papers for some disciplines (cs and electrical engineering, ece for both external and internal citations, and ait for internal citations), future studies focusing on these disciplines might potentially be enriched with a conference paper dataset (or datasets), in which citations from conference proceedings – categorized into serial or monograph format – would be additionally analyzed and included in blended serial/monograph ratios.   in terms of broader significance, the external component of this study provides libraries unable to conduct their own studies with ammunition for justifying the purchase or retention of key english language subscriptions in selected stem subfields. for libraries interested in conducting their own similar studies, this article provides them with a roadmap, although the process described is labor intensive and might be streamlined with automated citation harvesting and management of citations in database form instead of spreadsheets.   references   beile, p. m., boote, d. n., & killingsworth, e. k. (2004). a microscope or a mirror? a question of study validity regarding the use of dissertation citation analysis for evaluating research collections. the journal of academic librarianship 30(5), 347-353. http://doi.org/10.1016/j.acalib.2004.06.001   hoffmann, k. and doucette, l. (2012). a review of citation analysis methodologies for collection management. college & research libraries 73(4), 321-335. http://doi.org/10.5860/crl-254     microsoft word comm5082 evidence based library and information practice 2009, 4:1 43 evidence based library and information practice commentary systematic reviews and evidence based library and information practice alison brettle research fellow (information) salford centre for nursing midwifery and collaborative research, university of salford salford, great britain, united kingdom e-mail: a.brettle@salford.ac.uk received: 30 jan 2009 accepted: 31 jan 2009 © 2009 brettle. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. introduction one of the key tools of evidence based practice for practitioners and policy makers are systematic reviews. “rigorous summaries of all the research evidence relating to a specific question…often addressing the effectiveness of an intervention” that employ “a rigorous methodology for searching, research, retrieval, appraisal, data extraction, synthesis, and interpretation” (ciliska, cullum, and marks 135). they present a comprehensive summary of research based knowledge that can aid both practitioners and policy makers in decision making. following a systematic approach to both the search and synthesis of the included research, reduces the risk of bias and random errors inherent in a standard literature review process (droogan and song 16). when undertaking a systematic review, details of all elements of the search, appraisal, and synthesis process are documented and reported to ensure transparency. this enables readers to assess the quality of the review process and to determine the extent to which it has been rigorously conducted and is free from bias. systematic reviews and library and information professionals the systematic review industry has offered information professionals an opportunity to play a significant supportive role within evidence based practice, demonstrate their traditional skills of finding and managing information, and highlight their skills as expert searchers (beverley, booth, and bath 66). furthermore, librarians who are involved in the production of systematic reviews, particularly in healthcare, have demonstrated their ability as evidence based library and information practitioners by evaluating aspects of their expert searching practice in order to provide an evidence based for searching within systematic reviews (boynton et al.; mckibbon, wilczynski and haynes; sampson and mcgowan; sampson, evidence based library and information practice 2009, 4:1 44 mcgowan et al.; sampson, zhang et al.; sampson, barrowman et al.; zhang, sampson, and mcgowan). but what about systematic reviews as a tool to help library and information practitioners make decisions about their practice? systematic reviews can help us make sense of our research base, implement and justify services, and ensure that individual practitioners do not have to collect and analyse primary studies for every decision made (mckibbon 205-207). however, systematic reviews in library and information science are relatively uncommon. a recent paper located seven reviews published in library and information science journals between 19962006 (ankem). although this is not likely to be the exact number of existing systematic reviews in the domain, it provides an interesting comparison to the 5000+ systematic reviews that have been undertaken in health and social care (centre for reviews and dissemination, crd databases http://www.crd.york.ac.uk/crdweb/). papers discussing librarians’ involvement in systematic reviews tend to focus on the searching elements (falzon and booth; mcgowan and sampson; scherrer and dorsch; harris), although some discuss a wider role for librarians (beverley, booth, and bath; swinkels, briddon, and hall). in my own experience, becoming involved as a team member and providing literature searches for systematic reviews gave me the confidence to undertake systematic reviews of my own practice (brettle "information skills training: a systematic review of the literature; brettle "evaluating information skills training in health libraries: a systematic review"). this in turn provided me with the knowledge and skills to become more fully integrated within review teams conducting systematic reviews at the boundaries of health and social care (brettle, hill, and jenkins; dugdill et al.; hill and brettle). this involved critical appraisal, review management, data synthesis, report writing, and providing mentorship throughout the process to subject specialists who have limited experience of conducting systematic reviews. mckibbon outlines the importance of systematic reviews for librarians and provides a step by step guide to their production (mckibbon "systematic reviews"). furthermore, in the same article, mckibbon notes a range of reasons for undertaking systematic reviews including summarising the volume of information available, integrating information in areas where there is little information available, resolving discrepancies between studies with conflicting results, planning for new research, and providing teaching materials. however, systematic reviews have most commonly been used to address questions of effectiveness, although they can be relevant to any area where it is important to identify research trends or determine the overall balance of evidence in relation to a particular question (macdonald). they can also incorporate a range of research designs (both quantitative and qualitative), although these may present methodological and practical challenges in finding and synthesising literature for their inclusion (mays, pope, and popay). given that librarians already play a significant role in finding information for systematic reviews, and, according to the required competencies of our professional bodies (hunter and wake; medical library association), require the skills to carry out the appraisal and synthesis parts of systematic reviews, why are more systematic reviews not undertaken by librarians to answer questions related to their own practice? furthermore, why do the majority of librarians focus on the traditional searching elements of the review process rather than the critical appraisal? barriers to conducting systematic reviews time undoubtedly systematic reviews take time; mckibbon (mckibbon "systematic reviews") estimates 600 hours for a narrowly focussed evidence based library and information practice 2009, 4:1 45 review. this time factor is a serious issue for evidence based library and information practice, and is not a feasible way forward for everyday practitioners who wish to answer a pressing question in their own environment. a debate about the time involved and the need for systematic reviews to advance eblip would perhaps be useful in raising the issue up the agenda of organisations who may provide funds for library related systematic reviews. putting the issue of time to one side, there are methodological issues that also need to be faced if more systematic reviews are to be undertaken in eblip. these are outlined below and arose during the production of two systematic reviews in evidence based library and information practice (brettle "information skills training: a systematic review of the literature; brettle "evaluating information skills training in health libraries: a systematic review"). what evidence should be included? systematic reviews which follow a medical model of effectiveness, frequently include controlled studies to provide high quality evidence on “what works”. however, this type of evidence is frequently lacking in the library literature as was found in brettle ("information skills training: a systematic review of the literature"). an alternative approach is to include a range of evidence which “gains insight into the complexity of interventions and perspectives of users” (long et al. 22). this could include quantitative and qualitative research studies using a range of designs; and although this approach may not provide definitive answers to questions of effectiveness (according to a medical model of what constitutes good evidence), it does provide a summary of the evidence which exists, a baseline from which to move forward and highlights where and what research is needed. all factors which are useful to practising librarians. it has been argued that evidence based library and information practice should be based on a “core-centred” or all encompassing model of research evidence, rather than a hierarchical one (crumley and koufogiannakis 68). using this model, it is appropriate for systematic reviews in the library domain to take a wide view of relevant evidence and include a variety of designs appropriate to the topic or review question at hand. assessing study quality a related issue to deciding which studies to include is determining the quality of studies selected for inclusion in the review. methodologies such as randomised controlled trials are viewed as “good quality evidence” because of the ability of their design to reduce bias in the study, assuring that the effects of the intervention are due the intervention alone. however, this does not automatically mean that the study has been well conducted. furthermore, some questions or situations are not suited to such study designs and would more appropriately be addressed by using, for example, a qualitative approach. a range of critical appraisal tools exist that provide a means of extracting detailed information and a means of judging the quality and relevance of a study. these may be library specific (booth and brice; glynn; koufogiannakis, booth and brettle) or adapted from other disciplines such as healthcare, but do not offer a definitive answer regarding the quality of a study. how does a reviewer decide which studies to include in the review – all those which fit the inclusion criteria, or just those above a certain quality threshold? if so, how is this threshold determined? one approach is to limit the review to controlled study designs as in a review by garg and turtle, but this runs the risk of including a very limited number of studies and reach limited conclusions, which may not relate to practice. weightman and williamson included a range of designs and developed what they named a “crude measure of quality,” based on established research methodology and only included those which were above this quality threshold. this approach should ensure that the conclusions were based on the best quality evidence available, providing their definition of a “good quality study” was reliable. but what evidence based library and information practice 2009, 4:1 46 does make a good library study? it would be useful for further work to establish some kind of consensus. by examining the quality of library studies, it may be possible to highlight issues of weak study design that can be addressed by future research, and thus long term improve the overall quality of library related research. searching issues handbooks and guidance on systematic review production stress the need for comprehensive searching to locate relevant studies (centre for reviews and dissemination; higgins and green). information specialists have conducted a wide range of studies in relation to searching health care resources to improve the comprehensiveness and accuracy of searching (mckibbon, wilczynski, and haynes; sampson, mcgowan et al.). within social care systematic reviews, a number of studies have confirmed the need to search widely and use a variety of resources to ensure comprehensive coverage (brettle and long; golder, mason, and spilsbury; mcnally and alborz). the library literature parallels that of social care with studies crossing a range of domains such as library literature, education and management. thus it is likely that a range of databases need to be searched in order to obtain comprehensive coverage, but work needs to be undertaken to quantify this and provide guidance for searchers of the library related literature. this would provide evidence and tools to underpin evidence based library and information practice, and redress the balance of librarians conducting research into effective searching of resources relevant to the practice of other professions, but not that of their own! a systematic review to address these issues these issues related to systematic review methodology are not limited to library and information practice. similar issues have been found by the author in reviews that cross the boundaries of health and social care and much could be learned for library related systematic reviews by examining the methodological literature in these fields. what may be more problematic for librarians is the time involved and the skills needed to conduct systematic reviews. all these issues are currently being faced by a group of uk librarians who are undertaking a systematic review related to clinical librarianship (the north west clinical librarian systematic review group). the group has come together (and successfully obtained a small amount of funding) to conduct a review which seeks to answer a question facing their practice – that of how to evaluate their service. uk nhs library policy guidelines have recommended that libraries evaluate the impact of their library services and that clinical librarian (cl) services be implemented in all acute hospitals (hill). evidence to date has been inconclusive about the effectiveness of cl services (wagner and byrd; winning and beverley), but rigorous evaluations of cl services will be needed to demonstrate their value over time. given the various models of service provision, there is unlikely to be a “one model fits all” approach to evaluating cl services, therefore the review will examine these issues and provide guidance for future evaluations. the group comprises seven (mainly) clinical librarians from the north west region of the uk and myself as a facilitator and researcher who specialises in the conduct of systematic reviews. at the first meeting of the group, the majority of members suggested that their involvement in the project was a practical opportunity for them to gain research and critical appraisal skills as well as insight into the complete systematic review process that they often support. furthermore, operating as a group is a “safe environment” and provides confidence and support for those members who are less familiar with research methods. in order to provide insight into the process of library related systematic reviews the group are maintaining reflective diaries of the highs and lows of involvement in the project and a record of the time spent on the review. it is hoped that publications arising from these can be used to evidence based library and information practice 2009, 4:1 47 advance systematic reviews in librarianship by providing guidance for other librarians who wish to conduct systematic reviews. conclusion systematic reviews are an important means of summarising evidence and are useful for librarians who are seeking to find evidence for their practise. furthermore, they provide an opportunity for librarians to demonstrate and develop their skills in expert searching and critical appraisal. however, there are a limited number of reviews in existence in library and information practice – possibly due to the practical and methodological challenges involved in conducting them. as a profession which is seeking to demonstrate their value and improve their evidence base, it is important that we seek to overcome these challenges, progress research in this area, and provide a sound evidence base for library and information practice. note the clinical librarian systematic review mentioned in this commentary is seeking to identify effective methods of evaluating clinical librarian services from the published and grey literature. if you have (or are aware of) any grey literature reports which involve the evaluation of a cl service, please get in touch with the author of this article. works cited ankem, kalyani. "evaluation of method in systematic reviews and meta-analyses published in lis." library & information research 32.101 (2008): 91104. beverley, catherine a., andrew booth, and peter. a. bath. "the role of the information specialist in the systematic review process: a health information case study." health information & libraries journal 20.2 (2003): 65-74. booth, andrew, and anne brice. "research." health information & libraries journal 18 (2001): 175-7. boynton, janette, julie glanville, david mcdaid, and carol lefebvre. "identifying systematic reviews in medline: developing an objective approach to search strategy design." journal of information science 24.3 (1998): 137-54. brettle, alison, andy hill, and peter jenkins. "counselling in primary care: a systematic review of the evidence." counselling & psychotherapy research 8.4 (2008): 207-14. brettle, alison. "evaluating information skills training in health libraries: a systematic review." health information & libraries journal 24 (2007): 18-37. ---. "information skills training: a systematic review of the literature." health information & libraries journal 20.suppl 1 (2003): 3-9. brettle, alison j., and andrew f. long. "comparison of bibliographic databases for information on the rehabilitation of people with severe mental illness." bulletin of the medical library association 89.4 (2001): 353-62. centre for reviews and dissemination. "undertaking systematic reviews of research on effectiveness: crd's guidance for carrying out or commissioning reviews." 2001. university of york. <http://www.york.ac.uk/inst/crd/report4 .htm>. evidence based library and information practice 2009, 4:1 48 ciliska, donna, nicky cullum, and susan marks. "evaluation of systematic reviews of treatment of prevention interventions." evidence based nursing : an introduction. eds. nicky cullum, et al. oxford: blackwell publishing, 2008. 135-44. crumley, ellen, and denise koufogiannakis. "developing evidence-based librarianship: practical steps for implementation." health information & libraries journal 19.2 (2002): 61-70. droogan, jacqueline, and fujan song. "the process and importance of systematic reviews." nurse researcher 4.1 (1996): 15-26. dugdill, lindsey, alison brettle, claire hulme, serena mccluskey, and andrew long. "workplace physical activity interventions: a systematic review." international journal of workplace health management 1.1 (2008): 20-40. falzon, louise, and andrew booth. "realise-ing their potential?: implementing local library projects to support evidencebased-health care." health information & libraries journal 18 (2001): 65-74. garg, anupama, and kathleen m. turtle. "effectiveness of training health professionals in literature search skills using electronic health databases: a critical appraisal." health information & libraries journal 20.1 (2003): 33-41. glynn, lindsay. "a critical appraisal tool for library and information research." library hi tech 24.3 (2006): 387-99. golder, su, anne mason, and karen spilsbury. "systematic searches for the effectiveness of respite care." journal of the medical library association 96.2 (2008): 147-52. harris, martha r. "the librarian’s roles in the systematic review process: a case study." journal of the medical library association 93.1 (2005): 81-7. higgins, julian p.t., and sally green, eds. cochrane handbook for systematic reviews of interventions. version 5.0.1 [updated september 2008]. the cochrane collaboration, 2008. <http://www.cochrane-handbook.org>. hill, andrew, and alison brettle. "the effectiveness of counselling with older people: results of a systematic review." counselling & psychotherapy research 5.4 (2005): 265-72. hill, peter. "report of a national review of nhs library services in england: from knowledge to health in the 21st century." 2008. national library for health. 5th may 2008 <http://www.library.nhs.uk/aboutnlh/re view>. hunter, tracey, and michelle wake. "guide to working in health information: revised edition." chartered institute of library and information professionals: health libraries group, 2005. 16 december 2008 <http://www.cilip.org.uk/specialinterest groups/bysubject/health/publications/w orkguide/>. koufogiannakis, denise, andrew booth, and alison brettle. "reliant: reader's guide to the literature on interventions addressing the need for education and training." library & information research 30.94 (2006): 44-51. long, andrew f., mary godfrey, tracy randall, alison brettle, and maria grant. feasibility of undertaking systematic reviews in social care. leeds: university of leeds, nuffield institute evidence based library and information practice 2009, 4:1 49 for health. university of salford, health care practice r&d unit, 2002. macdonald, geraldine. using systematic reviews to improve social care. london: social care institute for excellence (scie), 2003. mays, nicholas, catherine pope, and jennie popay. "systematically reviewing qualitative and quantitative evidence to inform management and policymaking in the health field." journal of health services research & policy 10.suppl 1 (2005): 6-20. mcgowan, jessie, and margaret sampson. "systematic reviews need systematic searchers." journal of the medical library association 93.1 (2005): 74-80. mckibbon, k. ann. "systematic reviews and librarians." library trends 55.1 (2006): 202-15. mckibbon, k. ann, and nancy l. wilczynski. "retrieving randomized controlled trials from medline: a comparison of 38 published search filters." health information & libraries journal (2008). accessed 12 december 2008. mcnally, rosalind, and alison alborz. "developing methods for systematic reviewing in health services delivery and organization: an example from a review of access to health care for people with learning disabilities. part 1. identifying the literature." health information & libraries journal 21.3 (2004): 182-92. medical library association. "medical library association: educational policy: professional competencies for health sciences librarians". (2007): web page from professional organisation. medical library association. 28 april 2008. <http://www.mlanet.org/education/polic y/executive_summary.html>. sampson, margaret., et al. "can electronic search engines optimize screening of search results in systematic reviews: an empirical study." bmc medical research methodology 6:7 (2006). < http://www.biomedcentral.com/14712288/6/7> sampson, margaret, and jessie mcgowan. "errors in search strategies were identified by type and frequency." journal of clinical epidemiology 59.10 (2006): 1057-63. sampson, margaret, et al. "managing database overlap in systematic reviews using batch citation matcher: case studies using scopus." journal of the medical library association 94.4 (2006): 461-3, e219. ---. "an alternative to the hand searching gold standard: validating methodological search filters using relative recall." bmc medical research methodology 6:33 (2006). <http://www.biomedcentral.com/14712288/6/33> scherrer, carol s., and josephine l. dorsch. "the evolving role of the librarian in evidence based medicine." bulletin of the medical library association 87.3 (1999): 322-8. swinkels, annette, jason briddon, and jane hall. "two physiotherapists, one librarian and a systematic literature review: collaboration in action." health information & libraries journal 23 (2006): 248-56. wagner, kay c., and gary d. byrd. "evaluating the effectiveness of clinical medical librarian programs: a systematic evidence based library and information practice 2009, 4:1 50 review of the literature." journal of the medical library association 92.1 (2004): 14-33. weightman, alison l., and jane williamson. "the value and impact of information provided through library services for patient care: a systematic review." health information and libraries journal 22.1 (2005): 4-25. winning, m. alison, and catherine. a. beverley. "clinical librarianship: a systematic review of the literature." health information & libraries journal 20 suppl 1 (2003): 10-21. zhang, li, margaret sampson, and jessie mcgowan. "reporting the role of the expert searcher in cochrane reviews." evidence based library & information practice 1.4 (2006): 3-16. evidence based library and information practice 2011, 7.4 148 evidence based library and information practice news/announcements call for submissions: information: interactions and impact (i³) 2012. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. deadline: friday 18th january 2013 i³ focuses on the quality and effectiveness of the interaction between people and information and how this interaction can bring about change. the conference will look beyond the issues of use and accessibility of technology to questions about the way people interact with the information and knowledge content of today's systems and services, and how this might ultimately affect the impact of that information on individuals, organisations and communities. we invite the submission of high quality papers that report original research or critically discuss underlying methodological issues associated with the main themes of the conference. papers may reflect ongoing or completed research studies and should not have been previously published or be currently under consideration for publication elsewhere. we would particularly welcome papers which address two or more of the following conference themes:  the quality and effectiveness of user/information interactions (e.g. information literacy);  patterns of information behaviour in different contexts;  impact of information or information services on people, organisations, communities and society (e.g. social, learning, cultural and economic outcomes of engagement with information); and  more effective decision making. submissions are invited for: • full papers (40 minutes duration: 35 minutes, 5 minutes for questions); • short papers (15 minutes duration: questions at end of session);and • round table discussions (60-80 minutes duration). authors who are accepted to give full papers at i³ are also invited to submit a full paper for consideration by the editorial board of information research. for those papers accepted by information research, support for the process of html conversion will be provided by the i³ conference team. contributors should submit abstracts of 1000 words for full papers, and 300-500 words for short papers and round table discussions. these should be submitted electronically by downloading the online submissions form, located on the call for papers section of the website. this should then be emailed (as an http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca http://informationr.net/ir/index.html http://www.i3conference.org.uk/ evidence based library and information practice 2011, 7.4 149 attachment) to i3submissions@rgu.ac.uk. the conference language will be english. submissions will be anonymously reviewed by two members of an international programme committee specialising in one or more of the conference research streams. notification of acceptance will be emailed to authors and will also include the comments of the reviewers. full details of the requirements can be found in the call for papers section of the website. the deadline for receipt of abstracts is 18th january 2013. for any further information please see http://www.i3conference.org.uk, or contact the conference team at i3information@rgu.ac.uk. professor dorothy williams, i³ conference director mailto:i3submissions@rgu.ac.uk http://www.i3conference.org.uk/ mailto:i3information@rgu.ac.uk microsoft word es_kloda.doc evidence based library and information practice 2006, 1:2      33 evidence based library and information practice       evidence summary    reference librarians use electronic sources six times more than print sources to  answer patrons’ questions    a review of:  bradford, jane t., barbara costello, and robert lenholt. “reference service in the digital age: an  analysis of sources used to answer reference questions.” the journal of academic  librarianship 31.3 (may 2005): 263‐72.    reviewed by:  lorie a. kloda  instructional technology librarian, life sciences library  mcgill university montreal, quebec, canada  e‐mail: lorie.kloda@mcgill.ca    received:  1 march 2006   accepted:  3 april 2006      © 2006 kloda. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to test the hypothesis that  electronic sources are used by librarians  more often than print sources to answer  questions at the reference desk.    design – use study.    setting – small, private university in the  united states.    subjects – five full‐time and two part‐time  librarians working at the reference desk for  four months (two months in the fall of 2002,  and two months in the spring of 2003).    methods – the study recorded every  question asked by library patrons during the  two study periods, with the exception of  non‐library related directional questions and  hardware problems. for each question  recorded, librarians, while working solo at  the reference desk, paraphrased the  question and recorded the source(s) used to  answer each question. although questions  were recorded regardless of source – in  person, via email, or by telephone – the  medium in which each question was asked  and answered was not recorded. for the  first half of the study period (fall 2002  semester), librarians kept manual records  that were subsequently transcribed into a  spreadsheet by a student assistant.  in the  second half of the study period (spring 2003  semester), the librarians entered data  directly into a spreadsheet. the study’s data  monitor (who was also a study participant)  was responsible for ensuring the integrity of  the data and for assigning a category to each  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:2      34 source. the source category ‘librarian’ was  problematic in that it was unclear whether  or not the actual source of the answer was  the librarian or a source located by the  librarian.  after the first half of the study,  the procedure was changed to require that a  reference librarian was to label a source  used to answer a question as ‘librarian’ only  if the answer came from a librarian’s own  knowledge, and if it did not require  consultation of an outside source. categories  were generated on the fly, as the data  monitor reviewed the recorded questions  and sources. by the end of the study, 23  categories had emerged. while all sources  for answers were categorized, questions  themselves were not. during the second  part of the study, the gender of the patron  asking the question at the reference desk  was also recorded.    main results –the results for the fall 2002  and spring 2003 semesters were similar. for  the entire study period, librarians used a  total of 3,487 sources to answer 2,491  questions.  sources fell into 23 different  categories.  the top 5 categories used to  answer reference questions were databases  (23.92%), librarians (23.6%), library  catalogue (15.03%), internal web page  (12.27%), and reference books (9.38%).  the  top five categories accounted for 84.2% of all  sources used. for 75% of the questions,  librarians referred to a single source for an  answer. almost 60% of the sources used to  answer questions were electronic. of the  internal web pages used to answer  questions, the library’s online journal title  listings accounted for 76%. reference books  were used to answer questions in less than  10% of cases. less than 2% of the library’s  print reference collection (173 of 9,587 titles)  was consulted to answer reference questions  during the study period. the approximate  60:40 ratio of questions asked by female to  male patrons corresponded to the  university’s student body ratio.    conclusion – the results of this study  confirm the researchers’ hypothesis that  librarians use electronic sources with greater  frequency than they use print sources to  answer patrons’ reference questions. the  surprising finding in this study is the  proportion (approximately one quarter) of  reference questions answered by the  librarians themselves, without the need to  consult an outside source, either print or  electronic. the study suggests that a large  proportion of the reference collection goes  unused in answering patrons’ questions and  that librarians often answer a question using  only a single a source of information. a  reshelving study analyzing use of the  reference collection is underway to  supplement the results of the current study.    commentary    this study asks an important question for  academic libraries: do reference librarians  use electronic sources more often than print  sources to answer patrons’ questions? the  answer is a resounding “yes.”  unfortunately, the authors did not record  the method by which questions were asked  (in person, via email, or by telephone). such  data could have provided information about  whether or not sources differed depending  on the request method. overall, the results  of the study are useful for understanding  usage of the reference collection, the library  website, the catalogue, and the librarians  themselves in answering patrons’ questions.  the authors also note that many reference  questions asked do not require the expertise  of a professional librarian, although library  policy dictates that the desk be staffed with  one at all times.     the study periods were chosen to ensure a  high number of reference questions. since  all questions asked by patrons during the  study period were included in data  collection, there is no doubt that the data are  indeed representative of questions asked at  the library reference desk at that particular  institution.  while the programs of study  available at the institution might have  influenced the types of questions asked (and  evidence based library and information practice 2006, 1:2      35 subsequently, the types of sources used to  answer them), the results are strong enough  to suggest that this was not due to chance  and that these results are likely not unique  to the location or time of the study.  possible  confounding factors include librarian age,  prior training, and time pressure – any of  which may have impacted selection and use  of an electronic source versus a print source  or even the decision to use one particular  source rather than another to answer a  question. the modification of data collection  between the fall and spring recording  periods provides a cautionary example of  the need for beta testing of research  processes. this change in methodology may  have influenced the results.    the assignment of reference questions to  categories is somewhat problematic, as these  categories were not determined before the  study began, but were chosen by a single  librarian following data collection.     the authors do not elaborate as to how the  23 categories emerged from the data. also  problematic is the fact that data was                                                 collected by the study subjects themselves,  who were also the researchers and were  well‐aware of the study’s hypothesis. this  could possibly have introduced bias into the  data.    the implications of this study for academic  libraries are manifold. the authors state that  the manner in which reference questions are  answered can have an impact on training,  staffing levels at the reference desk,  collection development, and web page  design.  the results strongly encourage  librarians to consider the investment made  in purchasing and maintaining print  reference collections, in contrast with the  emphasis placed on library website content  and design. further research could explore  the accuracy of answers to reference  questions, particularly those for which only  one (electronic) source was consulted.  evidence summary   public librarians reflect belief in intellectual freedom through collection development activities   a review of: oltmann, s. m. (2016). public librarians' views on collection development and censorship. collection management, 41(1), 23-44. http://dx.doi.org/10.1080/01462679.2015.1117998   reviewed by: melissa goertzen collection development analysis & support librarian columbia university libraries new york, new york, united states of america email: mjg2227@columbia.edu   received: 23 may 2016  accepted: 26 july 2016      2016 goertzen. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to examine public librarians’ perspectives on censorship and intellectual freedom in relation to collection development activities.   design – survey combining questions from previous studies by moody (2004) and harkovitch, hirst and loomis (2003) with additional questions regarding intellectual freedom and demographics.   setting – public libraries in the state of ohio.   subjects – 251 directors and librarians responsible for collection development.   methods – the researcher created a survey in qualtrics, a software that supports online data collection and analysis. it contained thirty-two structured and open-ended questions and took approximately 15-25 minutes to complete. to recruit participants, an explanatory letter and survey link were sent to every public library director in the state of ohio. directors were also asked to share the survey with librarians under their leadership who were responsible for collection development. to analyze the data set, cross-tabulations were run to identify statistically significant correlations between demographic and community variables.   main results – the response rate was 43% (108 out of 251). participants agreed with the american library association’s (ala) definition of intellectual freedom, and to build collections that neither promote nor suppress specific ideas or beliefs. only 3.7% of respondents reported decisions not to purchase materials due to fear of negative feedback from the community. nearly 40% of participants reported conflict between personal and professional values at some time. all said that this dilemma had no bearing on professional collection development decisions. contrary to anecdotal evidence that suggests librarians in rural or conservative communities are less likely to purchase controversial materials, the researcher found that community and political variables were not statistically significant; across the board, participants were most concerned with building balanced, well-developed collections. gender, however, was statistically significant in terms of pressures felt to restrict access to materials; male librarians reported a higher number of instances where they felt internal or external pressures of this nature. however, as the number of male respondents was relatively low (15 out of 108 participants), the researcher did not draw concrete conclusions as to why this discrepancy exists.   conclusion – study findings demonstrate a strong professional allegiance to intellectual freedom as defined by the ala. in a practical sense, the participant group applied the principles of intellectual freedom to collection development activities regardless of demographic, community, or political variables.   commentary   intellectual freedom is a core value of the library profession. it is defined as “the right of library users to read, seek information, and speak freely as guaranteed by the first amendment” (ala, 2016, para. 1). the study at hand documents public librarians’ perspectives on intellectual freedom and censorship, and how these concepts are applied in branch libraries. while this is not a new topic of discussion, the researcher highlights a number of findings that contradict what was previously known anecdotally regarding the impact of community or political variables on collection development decisions. her conclusions add to a growing body of research that documents how concepts of intellectual freedom and censorship inform professional activities and the development of well-rounded collections (downey, 2013; moody, 2005; whelan, 2009).   strengths of the study include the suitability of the methodology to the central research question, well-defined criteria for the selection of participants, and the thorough discussion of study findings. the presentation of both quantitative and qualitative data produced thought-provoking results. it is interesting how many participants agreed with statements about intellectual freedom, felt pressure to restrict access, and so on, and the author supports this finding with specific examples from participants regarding how these issues impact collection development activities. the discussions brought a human element to the work and challenged the reviewer to consider what decisions she would have made in similar situations.   the uneven distribution of male and female librarians in the participant group limits the implications of the study. oltmann states that gender is a statistically significant variable in terms of pressure felt by professionals to restrict access, but does not draw concrete conclusions as to why this is the case due to a low number of male participants. a second limitation is the fact that the survey focused exclusively on physical collections. it would have been interesting to discover if perceptions of intellectual freedom and censorship differ when public librarians work with non-physical collections, as online resources are their own beast. oltmann acknowledges both of these limitations and suggests them as areas for future study. because of the potential for future work in this area, it was unfortunate that the survey was not included as an appendix. while the tables included in the article report on findings from specific questions, the reviewer was also interested to view the survey in its entirety.   despite these limitations, the study brings value to the library profession. oltmann developed a tool that effectively captures attitudes and perceptions that exist within the public library system in the state of ohio. because some findings contradict what was previously thought or observed within the library profession, it would be of value, as oltmann suggested, to distribute the survey across other states, and perhaps even other countries, to compare and contrast results.   references   american library association. (2016). intellectual freedom. retrieved from http://www.ala.org/advocacy/intfreedom   downey, j. (2013). self-censorship in selection of lgbt-themed materials. reference &user services quarterly, 53(2), 104-107. retrieved from https://journals.ala.org/rusq/index   harkovitch, m., hirst, a., & loomis, j. (2003). intellectual freedom in belief and practice. public libraries, 42(6), 367-374. retrieved from http://www.ala.org/pla/publications/publiclibraries   moody, k. (2004). censorship by queensland public librarians: philosophy and practice. australasian public libraries and information services, 17(4), 168-185.   moody, k. (2005). covert censorship in libraries: a discussion paper. the australian library journal, 54(2), 138-147.  http://dx.doi.org/10.1080/00049670.2005.10721741   whelan, d. l. (2009). dirty little secret: self-censorship is rampant and lethal. school library journal, 55(2), 26-31. retrieved from http://www.slj.com/     evidence summary   longitudinal analysis of undergraduate e-book use finds that knowledge of local communities drives format selection and collection development activities   a review of: hobbs, k., & klare, d. (2016). are we there yet?: a longitudinal look at e-books through students’ eyes. journal of electronic resources librarianship, 28(1), 9-24. http://dx.doi.org/10.1080/1941126x.2016.1130451   reviewed by: melissa goertzen collection development analysis & support librarian columbia university libraries new york, new york, united states of america email: mjg2227@columbia.edu   received: 12 oct. 2016   accepted: 17 jan. 2017      2017 goertzen. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine undergraduate students’ opinions of, use of, and facility with e-books.   design – a qualitative study that incorporated annual interview and usability sessions over a period of four years. the protocol was informed by interview techniques used in prior studies at wesleyan university. to supplement the body of qualitative data, the 2014 measuring information service outcomes (miso) survey was distributed; the researchers built five campus-specific e-book questions into the survey.   setting – a small university in the northeastern united states of america.   subjects – 28 undergraduate students (7 per year) who attended summer session between the years of 2011-2014 recruited for interview and usability sessions; 700 full-time undergraduate students recruited for the 2014 miso survey.   methods – the method was designed by a library consortium in the northeastern united states of america. the study itself was conducted by two librarians based at the single university. to recruit students for interview and usability sessions, librarians sent invitations via email to a random list of students enrolled in the university’s summer sessions. recruitment for the 2014 miso survey was also conducted via email; the survey was sent to a stratified, random sample of undergraduate students in february 2014.   interview sessions were structured around five open-ended questions that examined students’ familiarity with e-books and whether the format supports academic work. these sessions were followed by the students’ evaluation of specific book titles available on myilibrary and ebrary, platforms accessible to all libraries in the ctw consortium. participants were asked to locate e-books on given topics, answer two research questions using preselected e-books, explain their research process using the above mentioned platforms, and comment on the overall usability experience. instead of taking notes during interview and usability sessions, the researchers recorded interviews and captured screen activity. following sessions, they watched recordings, took notes independently, and compared notes to ensure salient points were captured.   due to concerns that a small pool of interview and usability candidates might not capture the overall attitude of students towards e-books, the researchers distributed the 2014 miso survey between the third and fourth interview years. five additional campus-specific e-book questions were included. the final response rate was 33%.   main results – the results of the interviews, usability studies, and miso survey suggest that although students use print and electronic formats for complementary functions, 86% would still select print if they had to choose between the formats. findings indicate that e-books promote discovery and convenient access to information, but print supports established and successful study habits, such as adding sticky notes to pages or creating annotations in margins. with that being said, most students do not attempt to locate one specific format over another. rather, their two central concerns are that content is relevant to search terms and the full-text is readily available.   study findings also suggest that students approach content through the lens of a particular assignment. regardless of format, they want to get in, locate specific information, and move on to the next source. also, students want all sources – regardless of format – readily at hand and arranged in personal organization systems. pdf files were the preferred electronic format because they best support this research behaviour; content can be arranged in filing systems on personal devices or printed when necessary. because of these research habits, digital rights management (drm) restrictions created extreme frustration and were said to impede work. in some cases, students created workarounds for the purpose of accessing information in a usable form. this included visiting file sharing sites like pirate bay in order to locate drm free content.   findings demonstrated a significant increase in student e-book use over the course of four years. however, this trend did not correspond to increased levels of sophistication in e-book use or facility with build-in functions on e-book platforms. the researchers discovered that students create workarounds instead of seeking out menu options that save time in the long run. this behaviour was consistent across the study group regardless of individual levels of experience working with e-books. students commented that additional features slow down work rather than creating efficiency. for instance, when keyboard shortcuts used to copy and paste text did not function, students preferred to type out a passage rather than spend time searching for copy functions available on the e-book platform.   conclusion – academic e-books continue to evolve in a fluid and dynamic environment. while the researchers saw improvements over the course of four years (e.g., fewer drm restrictions) access barriers remain, such as required authentication to access platform content. they also identified areas where training sessions lead by librarians could demonstrate how e-books support student research and learning activities.   the researchers also found that user experiences are local in nature and specific to campus cultures and expectations. they concluded that knowledge of local user communities should drive book format selection. whenever possible, libraries should provide access to multiple formats to support a variety of learning needs and research behaviours.   commentary   dozens of studies published over the last decade have focused on e-book use, particularly at the undergraduate level. while the study at hand supports what was previously known – students scan texts for key concepts (hernon, hopper, leach, saunders & zhang, 2007), lack a general awareness of platform functionalities (cassidy, martinez & shen, 2012), and express frustrations with drm restrictions (hyman, moser & segala, 2014) – it does include a unique feature that brings value to the information profession: a methodology for conducting a longitudinal analysis that explores the research behaviours of local user communities.   because the study conclusions emphasized the importance of understanding local research behaviours, the inclusion of the full protocol in the appendix adds significant value to the paper. this, combined with the researchers’ descriptions of recruitment techniques and how the methodology was executed over a period of four years, provides a detailed roadmap that librarians can adopt or customize in order to document the e-book landscape at their home institution.   although the methodology was certainly of interest, the organization and presentation of study results diminished the overall strength of the paper. although three methods were used to collect data, the researchers lump all findings together and attempt to discuss them at once. at times it was difficult to determine if a discussion was informed by results of the interview sessions, usability tests, survey, or a combination. it would have been interesting to read the results of each method separately, followed by a discussion of observed trends across the entire data set.   to add to the above mentioned point, it seemed that a discussion of miso survey results was largely omitted from the paper. although the researchers state that survey results did not reflect findings from interview sessions and usability tests, they did not explain how or why. this discrepancy is one of the most interesting aspects of the study findings and it would have been fascinating to learn more about the researchers’ thoughts and observations. it also seemed like a missed opportunity to present research questions for future study.   despite these limitations, the paper demonstrates how to document local attitudes and research needs surrounding e-book collections. the researchers gathered interesting sentiments regarding e-book experiences; one thought that stood out was that “the e-books we were testing seem to be designed not to be misused rather than to be used” (hobbs & klare, 2016, p. 13). insights such as this can inform how funds are allocated, services are developed, and materials are selected at individual institutions. in the future, it would be interesting to run the study across a group of libraries within a consortium to examine how user attitudes and behaviours compare across collecting groups.   references cassidy, e. d., martinez, m., & shen, l. (2012). not in love, or not in the know? graduate student and faculty use (and non-use) of e-books. the journal of academic librarianship, 38(6), 326-332. http://dx.doi.org/10.1016/j.acalib.2012.08.005   hayman, j. a., moser, m. t., & segala, l. n. (2014). electronic reading and digital library technologies: understanding learner expectation and usage intent for mobile learning. educational technology research and development, 62(1), 35-52. http://dx.doi.org/10.1007/s11423-013-9330-5   hernon, p., hooper, r., leach, m., saunders, l.,& zhang, j. (2007). e-book use by students: undergraduates in economics, literature, and nursing. the journal of academic librarianship, 33(1), 3-13. http://dx.doi.org/10.1016/j.acalib.2006.08.005   hobbs, k., & klare, d. (2016). are we there yet?: a longitudinal look at e-books through students’ eyes. journal of electronic resources librarianship, 28(1), 9-24. http://dx.doi.org/10.1080/1941126x.2016.1130451     evidence summary   ethnographic methods are becoming more popular in lis research   a review of: khoo, m., rozaklis, l., & hall, c. (2012). a survey of the use of ethnographic methods in the study of libraries and library users. library & information science research, 34(2), 82-91. doi: 10.1016/j.lisr.2011.07.010   reviewed by: diana k. wakimoto online literacy librarian/archivist california state university, east bay university libraries hayward, california, united states of america email: diana.wakimoto@csueastbay.edu   received: 26 oct. 2012   accepted: 22 jan. 2013      2013 wakimoto. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to determine the number of ethnographic studies of libraries and library users, where these studies are published, how researchers define ethnography, and which methods are used by the researchers.   design – literature survey.   setting – the researchers are located at drexel university, philadelphia, pennsylvania, united states of america.   subjects – 81 ethnographic studies of libraries and library users.   methods – the researchers conducted a literature survey, starting with a pilot study of selected library and information science (lis) journals, to find ethnographic studies and to determine key terms in research using ethnographic methods. the researchers used these terms in the main study to identify more lis research using ethnographic methods. the same journals used in the pilot study were then searched online as part of the main study, along with three lis databases (lisa, lista, llis). the researchers also searched the open web in order to capture grey literature in the lis field. all literature found, including those sources found through secondary citations, was screened for inclusion in coding. studies with non-lis settings were excluded as were studies that utilized non-ethnographic methods. the screened studies were coded to determine categories of methods used.   main results – the researchers found 81 articles, reports, and conference presentations that used ethnographic methods, which they compiled into a bibliography. this is an order of magnitude larger than that found by previous literature surveys. of these studies, 51.9% were published after 2005. the majority (64.2%) of the studies were published in journals. many studies did not provide clear or detailed definitions of ethnography and the definitions that were provided varied widely. the researchers identified themes which had been used to support ethnographic methods as a research methodology. these included using ethnographic methods to gain richer insight into the subjects’ experiences, to collect authentic data on the subjects’ experiences, and to allow flexibility in the methods chosen. they also included the use of multiple data collection methods to enable data triangulation. the five main method categories found in the literature were: observation, interviews, fieldwork, focus groups, and cultural probes.   conclusion – based on the relatively large number of ethnographic studies identified when compared to previous literature surveys and on the upward trend of publication of ethnographic research over the past five years, the authors noted that their overview study (and resultant compilation of literature from disparate sources) was important and time-saving for researchers who use or are beginning to use ethnography as a research methodology.     commentary   this study provides an overview of research in the lis field using ethnographic methods, which is published in many disparate sources. as such, it is situated to extend lis research literature and align it with other social sciences, such as anthropology and sociology, which often use ethnographic methods. the researchers position their study as novel, given that no other researcher or research team has yet completed a comprehensive survey and bibliography of research using ethnographic methods in the lis field. as the researchers were able to identify 81 studies that used ethnographic methods, there is obvious interest in the lis field in researching multiple aspects of libraries and library users using these methods. as a starting point for background research on previous studies using ethnographic methods, it is a useful article for librarian practitioners.   the study is well-written and the researchers clearly define their research questions and link their results back to these questions. they also acknowledge the limitations of the study and how the study will need to be updated to keep current with the expanding literature using ethnographic methods. the study is valid using the critical appraisal tool by glynn (2006) for calculating study validity, if it is assumed that the researchers analyzed studies for use of ethnographic methods and not studies that were themselves ethnographies. there is the possible issue of conflating ethnographic studies with studies using ethnographic methods throughout the study. ethnographic methods, such as interviews and observation, are used by many types of qualitative methodologies (berg, 2007) that would not be considered ethnographies. the researchers appeared to use the phrases “ethnographic studies” and “ethnographic methods” interchangeably throughout the study, although these are two different concepts.   a question raised by the lack of precision in using the phrases “ethnographic studies” and “ethnographic methods” is the categorization of studies as ethnographies when the researchers never identified their studies as ethnographies or their methods as ethnographic (briden & marshall, 2010; cmor, chan, & kong, 2010). more information about the definitions of ethnography found in the literature would have been useful, especially those definitions that were “induced indirectly” from the literature studies (p. 84). these issues could have resulted in the researchers’ overestimation in the number of ethnographic studies in the lis literature, while at the same time accurately recording the number of studies that used ethnographic methods. these issues aside, this study’s extensive bibliography will be useful to those who want an overview of some of the most popular qualitative methods used in lis research and to those contemplating using ethnographic methods. the compiled bibliography of research studies is a valuable reference and starting point for those interested in learning more about the application of qualitative methods in lis research settings, especially in the area of observation and interviews. through reading the cited literature, librarians will gain a better understanding of various qualitative methodologies used in lis research and thereby be better able to select an appropriate methodology for their next research project.     references   berg, b. l. (2007). qualitative research methods for the social sciences (6th ed.). boston, ma: pearson.   briden, j., & marshall, a. (2010). snapshots of laptop use in an academic library. library hi tech, 28(3), 447-453. doi: 10.1108/07378831011076684   cmor, d., chan, a., & kong, t. (2010). course-integrated learning outcomes for library database searching: three assessment points on the path of evidence. evidence based library and information practice, 5(1), 64-81.   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi: 10.1108/07378830610692154   evidence summary   a historical view of library instructional podcasts demonstrating they were beneficial to students and staff at a new zealand college of learning   a review of: jowitt, a. (2008). perceptions and usage of library instructional podcasts by staff and students at new zealand’s universal college of learning (ucol). reference services review, 36(3), 312–336. https://doi.org/10.1108/00907320810895396   reviewed by: mackenzie kathmann master of information studies, candidate 2022 university of ottawa ottawa, ontario, canada email: mackenzie.k@hotmail.com    received: 30 mar. 2021                                                                 accepted:  12 july 2021      2021 kathmann. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29948     abstract   objective – to examine usage of a specific set of library instructional podcasts and the potential of the format for effective library instruction.   design – concurrent mixed methods survey.   setting – multiple campuses at a polytechnic college in new zealand.   subjects – a total of 86 self-selected, non-random students and staff.   methods – web-based survey, piloted before a broader launch, with open and closed questions in one survey instrument (surveypro) regarding six sample podcasts accessible via the college’s library website. the researcher used closed questions to gather quantitative data with likert and verbal frequency scales and used concurrent triangulation to ensure balance with qualitative open-ended question responses for proper later interpretation.   main results – of the 86 participants in the study, 71.1% responded that the five library podcasts were “very good.” the study determined that the most useful podcast was called “my account” and helped students and staff activate and use their library accounts. overall, students enjoyed the five library podcasts slightly more than staff. the orientation walking tour was the least popular podcast. the researchers hypothesized that this was because the podcast did not fit the users’ preferred medium, which was computer based. even listeners who owned a portable media device preferred using a media player on their computer to access the podcasts. the participants preferred to listen to the podcasts during the day. the participants found that the 24/7 availability and the ability to listen to the material repeatedly were particularly helpful features.   conclusion – based on the research results, students and staff found library instructional podcasting advantageous because of its ease of access and constant availability. some participants mentioned ways to improve the quality of the podcasts, but they found them to be an effective new medium overall. additional research is needed to evaluate podcasts as an instructional medium.   commentary   the study builds upon the body of research around podcasting, podcast users, podcasts as educational tools, and new technologies in libraries (nelson, 2021) and recognizes a gap in the knowledge available about new zealand podcast users and those in other countries. the study uses rogers’ diffusion of innovations theory as its framework.   the research was thorough when appraised with a critical review checklist for mixed methods studies (hong et al., 2018). the researcher noted a potential shortcoming in that people who are familiar with podcasts could be more inclined to respond to the survey, and therefore the results could be biased towards that group. the researcher encountered issues regarding the sample, and the participants are not necessarily indicative of their target group of students from diverse backgrounds, including online and distance learners. the author of the study noted that technical issues with the survey impeded distance learners’ ability to complete it. online students were a key demographic that would benefit from podcasting technology, and so their input was crucial to the study. although the researcher attempted manual solutions where possible, this is a drawback of the study. further, while 200 participants began the survey, only 86 entered more than their id numbers. this indicates that the survey was somehow inaccessible or turned the participants off from responding after they had begun. the researcher did not explore this avenue further to identify why the response rate was low.   considering rogers’ diffusion of innovations theory framework, the findings of this study are useful when considering how much podcasting has grown since this study was conducted in 2008. now that podcasts are considered mainstream technology, it would be interesting to investigate whether the researcher’s theory, that podcasting would follow rogers’ five-stage decision-making process, was correct. this article provides an important historical perspective on the ever-evolving world of new media.   the author’s findings encourage libraries to adopt new technologies for student learning. the study shows that students and staff appreciated podcasting as an educational tool. these findings are relevant to the modern student, as podcasting is still used in education today. with that said, other mediums are becoming more popular. for example, short videos and video-recorded lectures are being used more frequently. in fact, in my experience, video-assisted learning is more common than podcasting for educational purposes and visual aids are highly beneficial to learning. as such, it would be useful to investigate the benefits of podcasting versus the benefits of video learning. those potential findings, along with the findings of this study, could be used to make budgetary decisions and to support librarians requesting funding for new technologies. almost 15 years later, it would be useful to investigate whether the findings of this research were applied in new zealand schools and what the status of this technology is today.   references   hong, q. n., pluye, p., fàbregues, s., bartlett, g., boardman, f., cargo, m., dagenais, p., gagnon, m-p., griffiths, f., nicolau, b., o’cathain, a., rousseau, m-c., & vedel, i. (2018). mixed methods appraisal tool (mmat). http://mixedmethodsappraisaltoolpublic.pbworks.com/w/page/127425845/download%20the%20mmat      nelson, r. (2021). podcasting services in academic libraries: a case study. college & undergraduates libraries, 1–16. https://doi.org/10.1080/10691316.2021.1897722   evidence summary   significant value found in mentoring programs for novice tenure-track academic librarians   a review of: goodsett, m., & walsh, a. (2015). building a strong foundation: mentoring programs for novice tenure-track librarians in academic libraries. college & research libraries, 76(7), 914-933. http://dx.doi.org/10.5860/crl.76.7.914   reviewed by: saori wendy herman, mlis, ahip head of education and access services hofstra northwell school of medicine hempstead, new york, united states of america email: saori.w.yoshioka@gmail.com   received: 15 june 2016  accepted: 14 oct. 2016        2016 herman. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to examine the effectiveness of mentoring programs for novice tenure-track academic librarians, and to identify critical elements that define a successful mentoring program in various academic library settings.   design – survey questionnaire with a voluntary phone interview.   setting – academic libraries in the united states of america.   subjects – 283 librarians participated in a survey questionnaire. researchers conducted additional interviews with 6 out of the 12 librarians who had volunteered on the survey questionnaire and who met the inclusion criteria.   methods – researchers recruited participants through two professional e-mail lists: the information literacy instruction discussion list (ili-l) listserv and the american library association’s new members round table (nmrt) listserv. interested participants completed a secured online survey that was hosted using surveymonkey. the researchers then coded and analyzed the collected survey data using the same software. at the end of the online survey, participants were given the opportunity to volunteer for an additional interview. potential interviewees were selected if mentoring programs were available for tenure-track librarians at their institutions. once selected, researchers contacted potential interviewees and conducted interviews. the interviews were transcribed, the data anonymized, and original recordings deleted. researchers coded the anonymized interview data to identify common themes.   main results – researchers identified six themes from the survey data and interview transcripts: traits of an effective mentor; configuration of mentoring programs; elements of effective mentoring programs; mentoring partnerships within or beyond the library; role and training of mentors and mentees; and the mentor/mentee relationship. overall, the survey and interview data suggest that mentoring programs provide valuable assistance with professional tenure-related activities, and facilitate less-tangible effects such as an increased understanding of an institution’s culture and an improvement of communication and time-management skills. the data also provides insight into effective program elements and areas for improvement.   conclusion – this study suggests that there is significant value in implementing a mentoring program. the results from this study can be used by academic libraries that are considering implementation of a mentoring program or improving an existing one.   commentary   as the authors note, the implementation of mentoring programs in libraries or the mentoring effects on new tenure-track librarians are topics that have been covered extensively in the literature (bosch, ramachandran, luévano, & wakiji, 2010; freedman, 2009; ghouse & church-duran, 2008; osif, 2008). however, the authors recognize that most of the studies only provide “isolated perspectives from specific libraries” (p. 917) rather than providing a broad overview. as a result, the authors sought to fill this gap in the literature by conducting a study that takes a “broader look at the perspectives of mentors, mentees, and program facilitators across a wide variety of academic libraries that employ tenure-track librarians” (p. 917).   this reviewer critically appraised the article using glynn’s ebl critical appraisal checklist (2006). the main issue affecting the validity of this research is the lack of diversity in the sample and the small number of responses from the target group. the recruitment pool was limited to librarians on the ili-l and nmrt listservs, although no reason is given for limiting distribution of the survey in this way. this resulted in the study excluding an unknown number of librarians who may have been eligible to participate. furthermore, due to the narrow focus of the two listservs, the number of participants identifying themselves as “novice tenure-track librarians” was very low (only 36 respondents from a total of 283 respondents). the authors acknowledge both of these limitations in the study.   another issue to note is that of the 283 survey participants, only 156 indicated that they were in an academic library setting, while the remaining participants did not respond. it is unclear whether the data from the unresponsive participants were removed from analysis since the survey was intended to target academic librarians. the authors do not address how the survey instrument and interview questions were created. as a result, it was unclear whether or not the instruments were validated.   aside from the validation issue, the questions on the survey and interview were clear enough to elicit precise answers. furthermore, with the exception of the questions focusing on librarianship, the instruments can be modified for studying other academic departments. this speaks to the generalizability of the study instruments. the authors helpfully include both the survey and interview questions as appendices.   with regard to observer bias during the data collection phase of the research, it should be noted that the authors themselves conducted the interviews. they do not mention any possible measures taken to minimize inter-observer bias. both survey and interview data were coded by the authors, but again, they do not mention whether the coding was done independently of one another, nor whether there was a third researcher available to resolve any disagreements in the coding.   this study provides a broad look at mentorship in academic libraries, with a special focus on novice tenure-track librarians. with some modifications to the sampling process and additional information on the survey instrument creation and interview process, this study would be worth exploring for future research.   references   bosch, e. k., ramachandran, h., luévano, s., & wakiji, e. (2010). the resource team model: an innovative mentoring program for academic librarians. new review of academic librarianship, 16(1), 57-74. http://dx.doi.org/10.1080/13614530903584305   freedman, s. (2009). effective mentoring. ifla journal, 35(2), 171-182. http://dx.doi.org/10.1177/0340035209105672   ghouse, n., & church-duran, j. (2008). and mentoring for all: the ku libraries’ experience. portal: libraries and the academy, 8(4), 373-386. http://dx.doi.org/10.1353/pla.0.0022   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154   osif, b. a. (2008). successful mentoring programs: examples from within and without the academy. journal of business & finance librarianship, 13(3), 335-347. http://dx.doi.org/10.1080/08963560802183096     research article   repercussion of using internet sources: dilemma for research communities   nilakshi sharma librarian government model college deithor, karbi anglong, assam, india email: snilakshi4@gmail.com     s. n. singh professor department of library and information science mizoram university aizawl, india email: drsinghsn@yahoo.com   received: 5 apr. 2022                                                                     accepted: 13 mar 2023      2023 sharma and singh. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30144     abstract   objectives – consultation of internet sources for educational and research purposes is the new normal. as a result of information communication technology, information creation and access are more convenient. the current study was carried out to investigate the proportion of use of internet sources for research purposes by research scholars of three central universities of north east india, namely tezpur university, mizoram university, and rajiv gandhi university.   methods – the researchers collected data from 123 respondents through an online questionnaire that was distributed through different social media platforms. the study was conducted among research scholars (phd and m.phil) of mizoram university, rajiv gandhi university, and tezpur university.   results – the research results show that research communities are moving toward digital platforms for searching and consulting their required resources. most of the respondents consult internet sources for writing their research reports, but they do not format the references properly. some research scholars do not follow any referencing style for citing web documents, and respondents do not have much awareness about the differences between urls and dois. research communities also face problems due to the inaccessibility of online documents cited by former researchers.   conclusion – since most of the respondents are not familiar with the use of web archives, the current study suggests that higher education institutions should arrange awareness programs on the use of web archives. research communities should follow the proper referencing formats to acknowledge others’ works. publishers should mandate a citation style for authors and verify the accuracy of the references before publishing articles or other works.     introduction   the publishing industry has expeditiously adopted e-publishing over print publishing in the last few decades. the internet has introduced new avenues for disseminating information quickly through e-publishing at minimal cost compared to print publishing. most electronic publishing sources are available over the internet and swiftly accessible, which is indeed a privilege for research communities. digital repositories and their relationship with the open access (oa) movement constitute a new trend in scholarly communication worldwide that will set new standards for information sharing and management (wani, gul, & rah, 2009). in their study, halloumeh & jirjees (2016) found that both faculty members and graduate students highly preferred electronic journals over print journals. e-resources occupy a significant portion of the global literature (baskar, 2017).   researchers cannot move on the right track with their research without scrutinizing publications of previous research in that field. to build basic understanding of a topic, researchers must consult previous works. consulting resources is easier since information communication technology made it possible to access almost all sources of information from anywhere at any time without much physical or financial effort. however, while there are advantages of internet sources, there are also some drawbacks. all documents available on the web have an address called a universal resource locator (url), but these web addresses may vanish over time for reasons such as changing the document location, server issues, change in domain name, etc. this phenomenon is known as link rot, or broken links. along with several other studies, a study by bansal (2021) showed that a huge number of urls cited in reputed journals were missing, posing a problem for researchers.   unavailability of previously cited resources may create confusion, and finding such a gap may be puzzling to researchers. vanishing resources also impact author citations. to discover the impact that broken links have on research, the present study was carried out among research scholars from different fields at three central universities in north east india.   literature review   with the emergence of information communication technology, publishers have shifted from traditional modes of publishing to web publishing. web publishing and the use of web information sources go hand in hand. studies have claimed that the use of web resources by research communities are increasing. kumar (2019) stated that advantages like the ability to hyperlink, use of a graphical interface, and remote access are the major reasons for the popularity of the web as a 24/7 information hub. the study revealed that almost 84% of the research scholar respondents used the web daily for writing articles, acquiring information, and other academic purposes.   on the other side, many scientists have shown that the use of web resources is not reliable, since web addresses vanish and documents may become unavailable for readers after a few years. in their study, bansal and parmar (2020) found that web resources cited in the current science journal have disappeared from their original locations with a common error message: 404 file not found. however, according to their research, the average half-life for these missing urls was 1.76; that means it would take almost 2 years for half of the url citations to vanish. lack of constancy in web references implies that persistent availability of online information resources is not sure or guaranteed. kumar and sushmitha (2019) found that, out of a total 1105 urls cited in annals of library and information studies from 2006 to 2015, only 625 urls remained active, and 480 urls were missing. they also tested the accessibility of urls and their recovery through the time travel tool. the internet archive had the highest number (56.46%) of missing urls, followed by bibliotheca alexandria web archive (15%).   kumar (2018) studied the prevalence and persistency of urls cited in articles between 2006 and 2015 of 12 lis journals from reputed publishers. results revealed that the lis researchers were highly dependent on web resources for scholarly information, and as a result, url citations in lis scholarly literature have increased in the last few years. the number of url citations increased from 1,870 in the year 2006 to 5,673 in the year 2015. a total of 93.80% of respondents used url citations in their scholarly articles; meanwhile 64.02% of respondents mentioned that they choose to use web resources due to easy accessibility.    in another study, kumar and kumar (2017) used the w3c link checker tool to find out the availability of urls and found that 1310 urls were active, whereas 823 urls were missing. the researchers recovered a total of 484 urls with the internet archive, but the remaining 339 urls were not found. arao, santos, & guedes (2015) studied the half-life and obsolescence of the citations included in dissertations and theses from the post-graduate program of science of the belles-lettres school of the federal university of rio de janeiro. the half-life study of citations was limited to only books and book chapters and showed that some fields have a relatively long half-life period. the results indicated that the half-life of science literature from 2007–2008 is 14 years, and the half-life of science literature from 2011–2012 is 15 years.   but as every problem has a solution, web archiving may be a solution to this serious issue. gul, mahajan, & ali (2014) mentioned that lifelong availability of web resources is not guaranteed, but solutions like web archiving can prevent the decay of web citations. ben-david & huurdeman (2014) claimed that the field of web archiving is at a turning point. previously, in web archiving, the single url has been the dominant unit for preservation and access. access tools such as the internet archive’s wayback machine reflect this notion as they allow consultation, or browsing, of one url at a time. in recent years, however, the single url approach to accessing web archives is being gradually replaced by search interfaces. ben-david & huurdeman address the theoretical and methodological implications of this transition to search; they recommend that web archiving should be appreciated and implemented to preserve web resources, which may help future researchers or lay persons find information.   aims   this study was carried out to fulfil the following objectives:   to find out the use of internet sources among researchers. to know the purpose of using internet sources. to understand the web citing behaviour of researchers. to understand the problems faced by researchers due to broken references.   scope and methodology   the study was conducted among phd and mphil research scholars of mizoram university, rajiv gandhi university, and tezpur university. basic information about these three institutions is given in table 1.     table 1 basic information of institutions name of the institution establishment year state responding departments total respondents mizoram university (mu) 2001 mizoram 5 62 rajiv gandhi university (rgu) 1984 arunachal pradesh 3 22 tezpur university (tu) 1994 assam 3 39 total 11 123     the researchers systematically structured an online questionnaire (available in appendix) in english. the questions were formulated based on the analysis of studies related to the availability and persistency of web references. it was shared through different social media platforms like whatsapp, facebook, and researchgate. the distribution of the questionnaire started 20 december 2021, and 123 responses were received by 7 february 2022. data were statistically analysed and presented in different tabular and graphical forms. institutional review of our research methods and survey were not required.   limitations   the questionnaire included questions requesting details such as age. these questions were required so that the data could be analysed according to scholar demographics. however, researchers have been careful to maintain the privacy of respondents.   results   demographic characteristics of respondents   from five departments at mu, three departments at rgu, and three departments at tu, totals of 62, 22, and 39 respondents respectively were received. as shown in table 2, more female research scholars than male research scholars responded to the questionnaire, and most of the respondents were pursuing a phd. the age group 26–30 represented 53 respondents (the largest number of respondents from any of the age categories), while only three respondents were in the age group >40.     table 2 demographic structure of respondents demography frequency   cumulative frequency relative frequency cumulative relative frequency sex female 67 67 0.54 0.54 male 53 120 0.43 0.98 prefer not to say 3 123 0.02 1.00 age <26 21 21 0.17 0.17 26–30 53 74 0.43 0.60 31–35 35 109 0.28 0.89 36–40 11 120 0.09 0.98 >40 3 123 0.02 1.00 degree pursuing mphil 46 46 0.37 0.37 phd 77 123 0.63 1.00     purpose of using the internet   all respondents (100%) from all the universities used the internet for research activities (see figure 1). all respondents of mizoram university used the internet for educational purposes. majority of respondents also used the internet for social connectivity with others and entertainment purpose. additionally, respondents, i.e., 37% of mu, 45% of rgu and 67% of tu use internet for purposes like online marketing, internet banking, shopping etc.     figure 1 purpose of using the internet.     use of internet sources in different research activities   research scholars mostly used various internet sources for the literature review of their research areas (see figure 2). although literature review is a process that helps in developing research knowledge like understanding methodologies and tools, how to verify results, and so on. most of the researchers considered literature review as a separate process. they indicated collecting sources related to their research area to be added in the literature review section of their theses. no respondents from rgu were found to use internet sources to understand the methodologies. meanwhile 17% of respondents from tu prioritized internet sources for using data collection tools, and 30%, 27%, and 10% of respondents from mu, rgu, and tu respectively prioritized use of internet sources for collecting data.     figure 2 use of internet sources in different research activities.     types of internet sources used for research purposes   as shown in figure 3, all respondents from all three universities used e-journals and e-books as information sources. majority of respondents also used electronic theses and dissertations (etd). all respondents from rgu and tu used websites as information sources; on the other hand, only 79% of mu respondents used websites to collect information. blogs were an important source of information for 65%, 29%, and 78% of respondents from mu, tu, and rgu respectively. all the sources used by respondents have web addresses that they included in the references of their research reports.     figure 3 types of internet sources used.     document searching techniques   fifty-seven percent, 49%, and 61% of respondents from mu, rgu, and tu respectively searched for a document using the url of the source (see figure 4). only 25%, 27%, and 3% of respondents from mu, rgu, and tu respectively used the digital object identifier (doi) of the document, which may be a reason why researchers could not access many of the documents. dois are more stable compared to urls, but the results show that researchers tend to search web documents through their urls. some scholars search documents using keywords or by searching the title of the document in search engines.     figure 4 document searching techniques of respondents.     citing behaviour of research scholars   today’s research publications are tomorrow’s research guides. all the references included in today’s research reports may be searched by other researchers for many decades to come. therefore, it is essential to study the citing behaviour of today’s researchers, so that the behaviours can be appreciated, modified, or rectified if needed. the use of reference management software (rms) was not popular among respondents: very few respondents preferred to use rms to create references (see figure 5). most of the respondents added all the available details of the web source, which may not have included the retrieval date. however, 42%, 23%, and 19% of respondents from mu, rgu, and tu respectively created references as per the guidelines given by a particular reference style. some respondents added only the web address of the source.     figure 5 citing behaviour of research scholars.     problems faced due to inaccessible urls   all research scholars responded positively to the question “have you ever witnessed the http error during your document search?” for various reasons, web addresses had vanished, and the user had faced problems. a problem faced by research communities is inaccessibility of resources, which may result in confusion among scholars. a few scholars mentioned that their research was derailed when they could not access the important sources. one scholar wrote that “since my research were based on secondary sources, i was reliable on the internet sources to some extent. but while i found that some of the resources are relocated and some have gone missing, it was like disaster to me.” another researcher wrote that “while i was looking for available literatures to my research area, a lot of literatures were available at that time and i was satisfied with it. but while i started downloading the files, i found some of the most important papers were missing.”   understanding about web archives   to assess scholars’ level of understanding about web archives, the questionnaire included the question “do you know web archives can help you to access your missing resources?” in the results, one respondent shared their experience of using the internet archive (suggested by their research supervisor) for searching missing documents. however, notably, 47% respondents answered the question with “no,” showing a lack of awareness among research communities about using web archives.   discussion   along with the increasing rate of using internet sources for research activities, confusion about inaccessibility of important sources is also increasing. research reports are the intellectual output of an institution; they contain rich data and are a major source of information for present and future generations, but those future generations of researchers will be unable to access this information if web addresses are missing. it is the duty of today’s research communities to preserve our intellectual output for future generations. this study interprets that using the internet is the new normal for society. the internet is acting as a bridge to link today’s society with knowledge, directly contributing to the process of building a knowledge society.   this study also found that although literature review should be an ongoing process carried out through the research process to aid in developing research knowledge, understanding methodologies and tools, verifying results, etc., many researchers consider the literature review to be an independent part of research. they tend to collect sources that are related to their research topic for the purpose of simply adding a literature review to their research report.   along with other sources, research scholars use blogs as information sources. compared to respondents from the other universities, respondents of tezpur university showed a lower interest in blogs as information sources. this may be due simply to the researchers’ personal interests. however, some research scholars may consider blogs as unauthentic, non-reliable information sources since they do not present filtered or processed information.   focusing on availability of web information sources, dois are more stable than urls, and researchers will have an easier time searching as well as provide a benefit to later researchers by using dois instead of urls. publishers should also focus on providing dois for online articles. due to the inaccessibility of important resources, research scholars face different issues at different phases of their research, which may impact the number of citations received by an author or a document.   based on the current study’s results, we suggest that researchers should create references properly by using rms. researchers and publishers should prioritize permanent web addresses like dois or purls over urls. publishers should verify the accuracy of web references before publishing papers. awareness of web archives cannot fully resolve the problem of inaccessibility but may reduce it to some extent. therefore, institutes of higher education should organize awareness programs on the use of web archives, rms, and similar tools. content creators and users should use web archives to preserve important information sources. authors should have a web-archiving system to preserve their own intellectual output for future references. publishers should consider assigning dois to previously published articles. librarianship is all about managing information with different tools and techniques. librarians should lead this movement to make the knowledge society aware of this issue and find solutions to it.   conclusion   inaccessibility and loss of references may directly impact the research productivity of researchers and institutions. knowledge creators should emphasize using proper information sources and adding proper references by using rms. web references should be properly formatted according to citation guidelines, and the permanence of web addresses should be maintained. web archives should be used, and for that, awareness must be created among researchers. the current study has helped to explain the dilemma of research communities and to suggest solutions.   author contributions   nilakshi sharma: methodology, data curation, writing – original draft, writing – review & editing s. n. singh: conceptualization, visualization, investigation   references   arao, l. h., da costa santos, m. j. v., & da silveira guedes, v. l. (2015). the half-life and obsolescence of the literature science area: a contribution to the understanding the chronology of citations in academic activity. qualitative and quantitative methods in libraries, 4, 603–610. http://www.qqml.net/index.php/qqml/article/view/271/261   bansal, s. (2021). decay of url references cited in desidoc journal of library & information technology. library philosophy and practice, article 5811. https://digitalcommons.unl.edu/libphilprac/5811   bansal, s., & parmar, s. (2020). decay of urls citation: a case study of current science. library philosophy and practice, article 3582. https://digitalcommons.unl.edu/libphilprac/3582/   baskar, a. (2017). e-resources and its uses in library. journal of critical reviews, 7(08), 3588–3592. http://www.jcreview.com/admin/uploads/files/61c717c2ea6a74.46602997.pdf     ben-david, a., & huurdeman, h. (2014). web archive search as research: methodological and theoretical implications. alexandria, 25, 93–111. https://doi.org/10.7227/alx.0022   gul, s., mahajan, i., & ali, a. (2014). the growth and decay of urls citation: a case of an online library & information science journal. malaysian journal of library & information science, 19(30), 27–39. https://ajba.um.edu.my/index.php/mjlis/article/view/1781   halloumeh, k. a., & jirjees, j. m. (2016). electronic versus print journals in academic libraries in abu dhabi: preference & problems. advances in journalism and communication, 4, 113–126. https://doi.org/10.4236/ajc.2016.44011     kumar, a., (2019). the use of web and its resources among research scholars of social science at guru ghasidas university (a central university), bilaspur, chhattisgarh, india: a study. library philosophy and practice, article 2457.  https://digitalcommons.unl.edu/libphilprac/2457   kumar, d. v. (2018). url prevalence and persistence: measuring the efficiency of internet archive and google in the recovery of vanished urls [doctoral thesis, tumkur university]. shodhganga. http://hdl.handle.net/10603/227210   kumar, v., & sampath kumar, b t. (2017). finding the unfound: recovery of missing urls through internet archive. annals of library and information studies, 64(3), 165–171. http://op.niscair.res.in/index.php/alis/article/view/16709   kumar, d. v., & sushmitha, m. (2019). recovery of missing urls cited in annals of library and information studies: a study of time travel. annals of library and information studies, 66(1), 24–32. http://op.niscair.res.in/index.php/alis/article/view/22420   wani, z. a., gul, s., & rah, j. a. (2009). open access repositories: a global perspective with an emphasis on asia. chinese librarianship: an international electronic journal, 27. http://www.white-clouds.com/iclc/cliej/cl27.htm     appendix questionnaire   greetings of the day,   this survey is for research purposes only. we are currently working on an area named 'repercussion of using internet sources: dilemma for research communities'. we request you to cooperate with us and provide the correct details. we will surely maintain the privacy of the respondents.   thank you.   * required   basic details   name * (open-ended response)   name of the institution * (open-ended response)   name of the department * (open-ended response)   date of birth * example: january 7, 2019   gender * mark only one oval. ·         male ·         female ·         prefer not to say ·         other:   degree pursuing * mark only one oval. ·         mphil ·         phd   year of joining the current program *   details related to topic   do you use internet? * mark only one oval. ·         yes ·         no ·         other:   please mention purpose of using internet? * check all that apply. ·         educational purpose ·         research purpose ·         for social connectivity ·         for entertainment ·         other:     for which of the research activities do you use internet sources? * check all that apply. ·         to develop basic idea of research ·         for literature review ·         to explore methodologies ·         to collect data ·         to use online data collection tool ·         to verify results ·         other:   what types of internet sources do you use? * check all that apply. ·         e-book ·         e-journals ·         electronic theses and dissertations ·         e-magazine ·         e-newspaper ·         e-reports ·         websites ·         blogs ·         other:   while searching an online resource, what technique do you use? * check all that apply. ·         i search through the url ·         i search through doi ·         i directly search the title on search engines ·         i search through keywords ·         other:   do you cite all online sources used for the research purpose? * mark only one oval. ·         yes ·         no ·         i cite, but not all sources ·         other:   what method do you use to cite an online document? * check all that apply. ·         i use reference management software ·         i add all available details of the source ·         i add all available details except url/doi ·         i follow the instruction of the journal ·         i add only the url or doi ·         other:   have you ever witnessed the http error during your document search? * (open-ended response)   have you ever heard about 'web archives'? * mark only one oval. ·         yes ·         no ·         maybe   do you know web archives can help you to access your missing resources? * (open-ended response)   any other additional information? (open-ended response)   what do you recommend to overcome the problem of missing web resources? (open-ended response)   evidence based library and information practice editorial responsibilities      2017. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     editor-in-chief: lorie kloda   associate editor (evidence summaries): heather pretty   associate editor (research articles): rebekah (becky) willson, lisl zach   associate editor (classics & reviews): i. diane cooper   associate editor (commentaries, using evidence in practice, research in practice): lorie kloda   communications officer (news): tatiana bryant   production editor: michelle dunaway   editorial intern: rachel hinrichs   editorial advisors: lindsay alcock, alison brettle, jonathan eldredge, denise koufogiannakis, katrine mallan, pam ryan   copyeditors: richard hayman (lead copyeditor), georgianne bordner, julie evener, heather healy, julie james, stacey penney, nikki tummon, alison yeoman   indexing support: meral alakus   microsoft word art_mcknight.doc evidence based library and information practice 2007, 2:3  59 evidence based library and information practice     article    choices in chaos: designing research into librarians’ information services improvised  during a variety of community‐wide disasters in order to produce evidence‐based  training materials for librarians      michelynn mcknight, phd, ahip  assistant professor,   louisiana state university school of library and information science  baton rouge, louisiana, usa  email: mmck@lsu.edu    lisl zach, phd  assistant professor,   drexel university college of information science and technology  philadelphia, pennsylvania, usa  email: lisl.zach@ischool.drexel.edu      received:  4 april 2007    accepted: 27 july 2007      © 2007 mcknight and zach. this is an open access article distributed under the terms of the creative  commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    abstract    objective ‐ disasters can and do happen anywhere, and library clients’ information needs  change suddenly during community‐wide disasters. library research and literature about  disasters emphasize issues relevant to the protection of staff and collections and the return  to normal services. most of these reports lack any discussion about the special services  librarians can provide during disasters. this project investigated the ways librarians  develop new services during disasters and examined the most appropriate methods of  gathering and analyzing data regarding such services. results of this preliminary research  aided preparation of a proposed larger scale project.    methods ‐ this paper reports on the design of a research project to study librarians’  information services during a variety of community‐wide disasters. the choice of a research  design was informed by the results of earlier preliminary studies and other research method  explorations.    mailto:mmck@lsu.edu mailto:zach@ischool.drexel.edu http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:3  60 results ‐ a study was proposed using multiple case study and grounded theory methods to  gather data on such activities.    conclusion ‐ the resulting proposed project was funded by the united states institute for  museum and library services national leadership program “investigating library and  information services during community‐based disasters: preparing information  professionals to plan for the worst,” this larger project will research and develop evidence  based training materials for library students and practitioners.  in this paper the preliminary  studies are outlined together with the proposed methodology and timeline for the larger  project.      “all disasters are chaos – but you always have choices”     – kim moore (director of emergency services, norman regional hospital,  norman, ok) discussing emergency responses after oklahoma city’s may 3, 1999  outbreak of f4 and f5 tornadoes and the bombing of the murrah building april  19, 1995. (personal communication, june 1999.)        introduction    community‐wide disasters are neither rare  nor limited to certain geographical areas of  the world. according to the latest world  disaster report there were 744 such disasters  in 2005 and an average of 614 such disasters  per year in the years 1996‐2005 (walter 207).  places that may not be susceptible to  hurricanes, earthquakes, or tsunamis may  suffer from floods, wildfires, volcanic  eruptions, tornadoes, or other windstorms.  accidental technological disasters  (unrelated to wars or acts of terrorism) can  result from power blackouts, explosions,  toxic releases, industrial and transportation  accidents, and other causes. populations  served by all kinds of libraries, information  services, and resource centers are directly  affected by these disasters.    “disaster planning” in library and  information science parlance currently  means planning for system and collection  protection during a localized emergency,  but not necessarily planning for provision of  information services during or after a  community‐wide disaster. an “emergency”  can be defined as follows:     … a situation or an occurrence of a  serious nature, developing  suddenly and unexpectedly, and  demanding immediate action …  generally of short duration, for  example, for a week or less. it may  involve electrical failure or minor  flooding caused by broken pipes.  (36 usc sec.1236.14 (2005)    by contrast, a “disaster” can be defined as  “an unexpected occurrence inflicting  widespread destruction and distress and  having long‐term adverse effects” (36 usc  sec. 1236.14 (2005). back‐up plans for  normal services based on agreements with  another library in the same community may  work in a brief localized emergency, but not  in a community‐wide disaster lasting more  than a few days. appendix a lists examples  of u.s. libraries affected by such disasters in  recent years.   evidence based library and information practice 2007, 2:3  61 textbook studies on implementing new  library services tell us about the value of  thoughtful data collection for information  needs analysis, planning, testing, and  evaluation of services (bopp and smith 177‐ 305). however, what really happens when a  community’s information needs suddenly  change, and there is no time to plan? how  do librarians continue to serve a community  when the users’ needs have changed  dramatically, and the usual access to  information sources is severely reduced or  non‐existent? after the hurricanes of 2005  battered most of the gulf of mexico coast of  the united states, information providers –  even those with traditional “disaster plans”  in place, were taken completely by surprise.  how do librarians deal with a barrage of  questions about food, health services, social  security checks and other necessities of life?  how do they support displaced children  without access to books to complete  homework assignments? how do they assist  researchers trying to meet proposal  deadlines with no internet access?    there is a need for evidence‐based  guidelines and education modules to  prepare students and practitioners to  provide such services during community‐ wide disasters. to develop such materials  requires research into instances when  librarians have quickly created new services  to meet the emergency needs of their client  communities. this was the objective of the  preliminary studies described in the first  part of this paper: what did librarians do  when there was no time for planning?    disasters vary, and so do the services  needed by communities. are there any  common patterns of librarians’ planning  and implementation of such services in a  wide variety of communities and disasters?  the preliminary studies led to the larger  research question: “how can we discover  patterns describing how librarians develop  new information services needed when  disaster strikes the community?” other  questions were also implied: “what data  will answer the question?” and “how can  such data be gathered?”  suitable methods  of data collection and analysis were  examined in order to develop a research  proposal for funding from the united states  institute for museum and library services  (imls). the new proposal was approved for  a research project that will focus on  librarians’ responses to disasters in the u.s.  the eventual results of this research should,  however, have implications for librarians’  responses to disasters throughout the world.   the methodology and timeline of the larger  project are described in the second half of  this paper.      literature review  there is a long history of library disaster  planning literature dealing with resource  protection during a localized emergency.  the available research on library disaster  planning is almost entirely limited to  protection and restoration of library staff,  systems, collections, and physical plants  during local emergencies. in schools of  library and information science, “disaster  planning” is usually covered in conjunction  with general management, collection  development, or preservation – not in the  context of information services. it is not  covered anywhere in the current edition of  bopp and smith’s reference and information  services: an introduction, nor in william a.  katz’ basic information services, both widely  used texts for information services courses.    two current examples of disaster planning  texts are halsted, jasper, and little’s disaster  planning: a how‐to‐do‐it manual for librarians  with planning templates on cd‐rom, and  kahn’s disaster response and planning for  libraries. both emphasize assignment of staff  responsibilities to insure the security and  recovery of a library and its staff after a  major emergency.    evidence based library and information practice 2007, 2:3  62 searches of the major bibliographic  databases covering library and information  science literature provide numerous  accounts telling how individual libraries  dealt with collection protection in a  particular localized emergency, such as a  power or plumbing failure. the provision of  library services during the emergency and  recovery period is rarely mentioned,  however, except in the context of restoration  of normal service. it is seldom discussed as a  service response to community information  needs created by the disaster (baggett;  benefiel and mosley; knisley; rogers).  alire’s disaster response and planning for  libraries is the only current text with a  substantial section on the restoration of  normal service after a community‐wide  disaster. it includes several accounts of  service restoration in the colorado state  university library after a flood.    some reports (e.g., schlotzhauer, watkins)  describe the addition of links on a library’s  web site, but this is more about added  resources than about new services  responsive to the changing needs of the  community. this could be viewed as  surprising for a profession that emphasizes  service first and the resources for providing  the service second, particularly in an era in  which distance services and virtual  reference services are growing  exponentially.    a few accounts describe librarians’ services  improvised during a single disaster or  during a small number of similar disasters.  they are characterized by describing the  “heroic efforts” of specific individual  librarians, and they tend to be anecdotal and  idiosyncratic. their recommendations are  not based on generalizable evidence  (fletcher; simmons and dover; schwanke;  tarin; taylor; whelan).    there is limited coverage, especially from  europe, of individual instances of public  library support services for government  officials and public health workers, but none  of the recommendations are based on  broader research into meeting community‐ wide information needs in more than one  disaster (heyworth; polloni and harkins;  riley and meadows). another related  example is harris, wathen, and chan’s  account of ontario libraries’ reference  services for consumers during the 2003  outbreak of severe acute respiratory  syndrome (sars) in toronto.    there is a lack of evidence in the  professional literature describing how  libraries can deal with community‐wide  disasters.  this paper describes a series of  preliminary studies that gathered data on  how librarians in the u.s. have provided  services in such disasters.  these provided a  foundation to solicit funding for a larger  scale research project that will develop  evidence based training materials for library  students and practitioners. this paper also  outlines the proposed methodology and  timeline for the larger project.    preliminary studies  in differing contexts, the authors were  participants in the implementation of  extraordinary information services in  louisiana during and immediately after  hurricanes katrina and rita in 2005. the  studies developed over time, using several  methods to gather quantitative and  qualitative data to discover what services  were improvised during that disaster.     within days of hurricane katrina’s landfall,  informal assessments of the responses of  librarians to the disaster were collected. this  included anecdotal information through  personal observation, participant  observation, formal interviews,  conversations, and monitoring electronic  mail lists. it appeared that librarians around  the state were responding to the disaster in  many different ways, and that few, if any,  evidence based library and information practice 2007, 2:3  63 were following any predetermined plan.  (dawson and mccook; dempsey; fletcher;  kniffel; mccoy; mcknight “medical  reference ”;  mcknight “information  services”; mcknight “health sciences”;  mcknight “mobile shelter”; meraz; morris;  spencer; tadman; zach “first‐hand  thoughts”; zach “no time to plan”) this  prompted formulation of an overarching  research question to guide this assessment:  “did louisiana libraries have any  systematic approach to providing new  services to meet users’ extraordinary needs  after hurricane katrina, august 29, 2005?”   two sub‐questions emerged in those  locations where libraries were providing  services,        • “what services were provided?”  • “how did libraries determine what  was needed?”    less than a month after hurricane katrina,  these research questions were extended to  include louisiana librarians’ service  responses to hurricane rita (september 23,  2005).    methods  in october 2005, a 22‐question survey  (appendix b) was announced on all of the   louisiana library mail lists including those  of the louisiana library association (lla),  louisiana academic library information  network consortium (lalinc), local  chapters of the american library  association (ala), the medical library  association (mla), the society of american  archivists (saa) and the special library  association (sla).  the survey was also sent  to school media center coordinators and  public library administrators. the survey  was developed and administered with  surveymonkey® online survey software.  after gathering descriptive information  about the current state of the library, the  survey asked respondents to identify new  information services that their organizations  were providing in response to the  hurricanes. selected respondents who  indicated they had created these new  information services participated in short  open‐ended interviews (by telephone and in  person) about their projects. because this  study developed in response to events,  rather than being planned as a formal pilot  study, the exact number of interviews was  not tallied.  the findings of the survey were  later augmented with participant  observations of specific new information  services. these studies were conducted by  faculty and students from louisiana state  university school of library and  information science.    results  because of the distribution method (a link to  the survey site in a generic e‐mail message),  it is impossible to calculate the total number  of librarians who received the link. usable  responses were received from 269 librarians  who reported they worked in the types of  institutions in table 1.  the majority of responses came from  librarians whose institutions had sustained  little or no direct damage from the  hurricanes. of those responding to the  question, 84.1 % reported little or no  damage to their collections and 78.2 %  reported little or no damage to their  buildings. others, however, were less  fortunate, and several responses were from  librarians who had evacuated from their  communities and did not know the  condition of their buildings or collections.  almost half (48.9 %) of those responding  reported that they had continued operations  without interruption; at the time of the  survey, 8.5 % of those responding had  moved operations to another location, and  12.1 % were closed until further notice.           evidence based library and information practice 2007, 2:3  64 academic libraries  94  35%  archives or historical repositories  29  11%  public libraries  52  19%  school media centers  37  14%  special libraries or other  54  20%  no response to the “type” question  3  1%  total number respondents  269  100%    table 1: survey respondents reported by library type      of those who did have general disaster  plans in place before the hurricanes, only 25  (less than 10 %) said that they used their  plan. reasons given for not implementing  the plans included:  the damage to the  buildings and surrounding areas was  greater than anticipated by the plans, lack of  authority, lack of funds, or lack of personnel.    seventy‐five (25%) librarians reported  having added new services in direct  response to the hurricanes. the services  included extended service hours, new  outreach locations, new websites, new  evacuee‐specific services, and special  services for displaced adults and children.  some extended loan periods or suspended  fines. others provided wireless internet  access, delivered reading material to  evacuee shelters, provided program  matching services to guide displaced  students in their choice of new schools.  libraries provided office space to displaced  librarians, issued temporary borrowing  cards to displaced users, provided free  photocopying, and read stories to children  in evacuee shelters. some began new  computer lab instruction programs,  provided refreshments, facilitated  communication with their local emergency  operations center, arranged transportation  for evacuees, collected donations, helped  evacuees search online for missing family  members and pets, and helped evacuees  replace drivers’ licenses, birth certificates,  and other important documents. most that  instituted new services indicated that they  would continue these services as long as  they were needed.    librarians who implemented new services  reported that their emergency needs  assessment was based on personal  observations, requests from users, and  discussions with internal staff, governing  boards, and information professionals  outside of the institution. thirty‐nine (14%)  respondents said that they did have some  sort of plan in place that addressed the  potential need for new information services  in response to a crisis.    the librarians who were interviewed as part  of the follow‐up survey focused on the level  of general chaos in the immediate aftermath  of the hurricanes and on the need to take  quick and decisive actions based on  whatever situation they might have  encountered at a given moment. their  response showed that their post‐katrina  responsibilities went far beyond collection  protection, technical services, and financial  management. they emphasized the point  that during times of crisis it is essential for  librarians to be flexible enough to respond  to the needs of their users and that a useful  tool for any librarian to have is a very open  mind about what types of services may fall  within the aegis of the library (zach “first‐ hand thoughts”). participant observation,  observation, and interviews produced data  evidence based library and information practice 2007, 2:3  65 which has been analyzed and reported  elsewhere (mcknight “medical reference”).    designing a larger study  data collected in the preliminary studies did  not answer the “what” and “how” questions  needed for a study of responses during  multiple disasters. based on that fact and on  subsequent discussions with researchers  experienced in qualitative methodologies, a  larger study was designed to gather  evidence of common practice patterns in  diverse disasters. this study would  incorporate a research method based on in‐ depth interviews, multiple case studies, and  narrative data analysis to build a grounded  theory of librarians’ information services  during a variety of kinds of community  disasters.     creswell explains the value of grounded  theory research as being a focus on the  object of the research: “[for] the  development or generation of a theory  closely related to the context of the  phenomenon being studied” (56). baker,  wuest and stern describe this situational  focus more explicitly:    “… [the] researchers’ purpose in  grounded theory is to explain a  given social situation by identifying  the core and subsidiary processes  operating in it. the core process is  the guiding principle underlying  what is occurring in the situation  and dominates the analysis because  it links most of the other processes  involved in an explanatory  network.” (1357)  approximately twenty participants will be  chosen to provide a sample of information  professionals who have faced different types  of community‐based disasters in a variety of  institutional settings. selection criteria for  the interviews will include: type of disaster  faced, size or type of institution, geographic  location, and scheduling constraints. data  will be collected using a semi‐structured  interview protocol, and the interviews will  be audiotaped and transcribed for  referential adequacy. participants will be  asked to sign an informed consent form, and  data collected in the interview phase of the  study will be kept confidential in most cases.   specific identifiable experiences may be  reported in the case studies developed  during the second phase of the project  (described later in this paper), with the  permission of the interviewees. interviewees  will be given the opportunity to review all  case study materials before they are released  (“member checks”). in addition to the data  collected during the interviews, the  researchers will examine any other available  data sources (e.g., existing disaster plans,  flyers or brochures for new services,  archived web sites). these sources will be  used to provide additional richness to the  personal narratives.    the interviews will be carried out using a  “multiple‐case‐studies” design (zach  “using”). the multiple‐case‐studies design  will allow the researchers to use subsequent  interviews to confirm or disprove the  patterns identified in earlier sessions. in  practice, this means that the interview  protocol may be revised during the course  of the study in response to new information.  the interviews will be sequenced, so that  the first group of interviewees will have  many characteristics (e.g., size or type of  institution, type of disaster) in common  (“literal replication”). these interviews will  provide (as far as possible) a baseline of  types of responses and obstacles  encountered by practitioners. the remaining  interviews will be selected to explore and  confirm or disprove the patterns identified  in the initial interviews (“theoretical  replication”) and to expand the list of  responses and obstacles encountered. this  approach will allow the researchers to  identify common elements of disasters  evidence based library and information practice 2007, 2:3  66 response that can be codified and presented  as best practices.    data will be coded using qsr nvivo 7 text  analysis software. nvivo is a qualitative  research tool that allows researchers to  search text and organize concepts efficiently  for further analysis. it is also helps  researchers collect data from different  sources into a single, compatible format for  coding.     based on the results of the data collection  from librarian interviews, the researchers  will identify five to seven common elements  that appear to exemplify a successful  approach to planning and delivering new  information services in response to  community‐based disasters. common  elements to be addressed will include, but  will not be limited to, the following:     •   type of information need identified  (e.g., health, basic needs, specialized  data)  • type of user (e.g., existing, new,  transient, long‐term, adult, youth)  • type of resources sought (e.g.,  public or proprietary)  • type of new or modified service  provided  • type of obstacles overcome     the researchers will develop case study  materials to exemplify the best practices  identified during the data collection and  analysis phase. the material developed will  be a combination of generic case studies  (demonstrating one or more common  elements found across several interviews)  and individual explanatory case studies (in  which a single situation is described in  depth to provide pedagogical tools for  discussing a variety of responses to the  issues). each case study will include a  general description of the particular  community‐based disaster, the responses  taken by the information professional(s)  involved, and a guide for discussing the  issues involved in the situation.    this methodology was put forward as a  proposal and was successful in receiving  funding from the united states institute for  museum and library services national  leadership program. the study will be  under the oversight of an advisory  committee (table 2) whose members will  communicate primarily via e‐mail and a  project blog.  how the study will progress  a large pool of libraries is being developed  using personal contacts, a literature review,  and through a call for participation in  dozens of librarians’ electronic mail lists. a  surveymonkey.com site  http://www.surveymonkey.com/s.asp?u=718 283553604 has been established for  interested parties. to date it has been  relatively easy to find libraries in  communities that have experienced  disasters. people are usually eager to tell  their stories, because disasters have  profound emotional effects. friends and  colleagues frequently suggest names of  librarians and libraries affected by disasters.  the challenge has been to identify the few  who actually created new information  services. a project web site will provide  open access to data throughout the course of  the study. the initial data gathering phase  of the study should conclude by the end of  2007, the analysis will be conducted in 2008,  and the results subsequently published in  2009.    the results of this study will be made  available to practitioners in the field as a  compendium of best practices in the form of     http://www.surveymonkey.com/s.asp?u=718 evidence based library and information practice 2007, 2:3  67   theresa jay dickson, associate director, pioneer library system, norman, oklahoma  carol l.ginsburg, senior consultant, bst america, new york, new york   deborah halsted, associate director, houston academy of medicine/texas medical center  library, houston, texas  delia neuman, associate professor, college of information studies, university of maryland ‐  college park, maryland  sue sherif, school library/youth services coordinator alaska state library, anchorage, alaska  table 2. advisory committee  case‐study examples of various types of  disaster response scenarios. curriculum  modules based on the results will also be  implemented in the researchers’ courses  taught at louisiana state university and  drexel university. these modules will be  made available to other library and  information science programs. in this way  the experiences of practitioners in the field  may be transmitted to future information  professionals. all too often, information  professionals have had to improvise quickly  to provide services in times of disasters. the  results of this research project will save  them precious time and allow them to make  vital information resources available more  effectively during times of crisis.    conclusion  librarians need evidence‐based case studies  and educational material to help them learn  how to identify needed information services  during any kind of community‐wide  disaster and to respond to these needs  creatively. since this preparation is not  currently included in lis education,  standards and guidelines or research  literature, there is a need for reliable studies  of these phenomena in a variety of libraries  and a variety of disasters. following  preliminary work and an examination of  various quantitative, qualitative and mixed  data gathering methods, a study was  designed and received funding for the next  two years.       acknowledgement  a version of this paper was presented at the 4th  international evidence based librarianship  conference, held in chapel hill, north carolina,  usa (may 2007).  see  http://www.eblip4.unc.edu/papers/mcknight.pdf      works cited    alire, camila. library disaster planning and  recovery handbook. new york: neal‐ schuman, 2000.    baggett, james l. “handle with care:  disaster planning and recovery.”  alabama libraries 53.2 (2003): 5.    baker, cynthia, judith wuerst, and phyllis  noerager stern.  “method slurring: the  grounded theory/phenomenology  example.” journal of advanced  nursing 17.11 (nov. 1992): 1355‐60.    benefiel, candace r., and pixey ann mosley.  “coping with the unexpected: a  http://www.eblip4.unc.edu/papers/mcknight.pdf evidence based library and information practice 2007, 2:3  68 rapid response group in an academic  library.” technical services quarterly  17.1 (1999): 25‐35.    block, marylaine, and ann kim. “all  (librarian) hands on deck: librarians  lead the way on the long journey to  recovery and rebuilding.” lj movers  and shakers. spec. issue of library  journal 131 (15 mar. 2006): 22‐31.    bopp, richard e., and linda c. smith.  reference and information services:  an introduction. 3rd ed. englewood,  co: libraries unlimited, 2001.    creswell, john. w. qualitative inquiry and  research design. 2nd ed. thousand  oaks, ca: sage, 1998.    dawson, alma, and kathleen de la peña  mccook. “rebuilding community in  louisiana after the hurricanes of  2005.” reference & user services  quarterly 44.4 (summer 2006): 292‐6.    dempsey, beth. “responding to disaster:  lessons from the shelter library.” lj  buyers guide. spec. issue of library  journal 130 (dec. 2005): 6‐8.    fletcher, adelaide myers. “no point of  reference: a hurricane of medical  information needs.” journal of  hospital librarianship 62.2 (2006): 1‐14.    halsted, deborah d., richard p. jasper, and  felicia m. little. disaster planning: a  how‐to‐do‐it manual for librarians  with planning templates on cd‐rom.  new york: neal‐schuman, 2005.    harris, roma c., nadine wathen, and  donna chan. “public library  responses to a consumer health  inquiry in a public health crisis: the  sars experience in ontario.”  reference & user services quarterly  45.2 (2005): 147‐54.    heyworth, felicity. “global consultation  meeting on disaster information  centres.” feliciter 45 (1999): 149‐50.    kahn, miriam b. disaster response and  planning for libraries. 2nd ed. chicago:  american library association, 2003.    kniffel, leonard. “you have to see it for  yourself.” american libraries 37.5  (2006): 9‐13.    kniseley, susan. “disaster planning for your  library.” nebraska library association  quarterly 37.1 (spring 2007): 8‐14.    mccoy, mary j. “the forces of nature or  dancing with the three ladies.” texas  library journal 81.4 (2005): 142.    mcknight, michelynn. “medical reference  in louisiana after katrina.” alabama  health libraries association, auburn,  ga, usa, 15 nov. 2005.    mcknight, michelynn. “information  services in the chaos of disaster:  southern louisiana medical reference  in the wake of hurricane katrina.”  medical library association, phoenix,  az, usa, 23 may 2006.    mcknight, michelynn. “health sciences  librarians’ reference services during a  disaster: more than collection  protection.” medical reference  services quarterly 25.3 (2006): 1‐12.    mcknight, michelynn. “mobile shelter  library meets information needs of  evacuees.” lsu community  connections 6.2 (2006): 2.    mcknight, michelynn, and lisl zach.  “choices in chaos: designing research  evidence based library and information practice 2007, 2:3  69 to investigate librarians’ information  services improvised during a variety  of community‐wide disasters and to  produce evidence‐based training  materials for librarians,” eblip4:  transforming the profession: 4th  international conference, evidence‐ based library & information practice,  university of north carolina ‐ chapel  hill, durham, nc, 7 may 2007.    meraz, gloria. “in times of trouble.” texas  library journal 81.3 (2005): 90.    moore, kim. personal interview. june 1999.    morris, george. “shelter library: michigan  woman spearheads effort to get  temporary library set up for  hurricane evacuees in baton rouge.”  advocate [baton rouge, la] 30 nov.  2005 : magazine 14+  <www.2theadvocate.com/stories/11200 5/ent_shelter001.html>.    polloni, diana, and barney harkins.  “picking up the pieces: an  organizational profile of the library  disaster centre.” library management  17.1 (1996): 37‐40.    riley, julie, and jack meadows. “the role of  information in disaster planning: a  case study.” library management 16.4  (1995): 18‐24.    rogers, michael. “libraries okay after  blackout.” library journal 128.15 (15  sept. 2003): 17‐8.    schlotzhauer, nonny. “disaster relief:  starting points for learning.” college  & research libraries 67.2 (feb. 2006):  91‐6.    schwanke, harriet “hurricane andrew:  community information services  disaster relief – cis to the rescue.”  southeastern librarian 44 (1994): 69‐71.    simmons, silvia, and marilyn dover. civil  disasters: the role of public libraries  following a crisis in the community.  london: library association, 1993.    spencer, forrest glenn. “info pros respond  to disaster: rallying to help  employers and each other.”  information outlook 9.10 (2005): 13‐8.    tadman, andrew. “reflections on ochsner  medical library: a year after the  storm.” louisiana libraries 69.2 (2006):  3‐5.    tarin, patricia a. “crisis sparks need for  information: fire in the canyon.”  library journal 113.15 (15 sept. 1988):  47‐9.    taylor, dee dee. “hurricane survival  tactics: a public library offers  emergency service of epic  proportions.” american libraries 35.10  (nov. 2004): 42‐3.    walter, jonathan, ed. world disasters  report: focus on neglected crises.  geneva: international federation of  red cross and red crescent societies,  2006.    watkins, christine. “chapter report ‐  disaster planning makes (dollars and)  sense.” american libraries 27.8 (sept.  1996): 8‐9.    whelan, debra lau. “california fires shut  libraries: librarians continue to offer  their services during state‐of‐ emergency.” school library journal  49.12 (2003): 22‐3.    http://www.2theadvocate.com/stories/112005/ent_shelter001.html http://www.2theadvocate.com/stories/112005/ent_shelter001.html evidence based library and information practice 2007, 2:3  70 zach, lisl. “first‐hand thoughts on the  storm.” information outlook 9.10  (2005): 14‐5.    zach, lisl. “when there is no time to plan:  responding to the information needs  of hurricane katrina’s victims.”  recognizing the success of reference:  7th annual virtual reference desk  conference, san francisco, ca, 14‐15  november 2005.    zach, lisl. “using a multiple‐case‐studies  design to investigate the information‐ seeking behavior of arts  administrators.” library trends 55.1  (2006): 4‐21.  71 appendix a  examples of u.s. libraries providing services during community‐wide  natural or accidental disasters 1997‐2007  (data collected by the authors)    date  disaster  library  library type  1997  flood  colorado state libraries  academic  1998  earthquake  olympia timberland library, wa  public  1998  tornadoes  hanson‐mccook county regional  library, sd  public  1998  flood  jonsson library of government  documents, stanford, ca  special library  1999  flood  edgecombe county courthouse  archives, nc  archives and historical  collection  1999  tornadoes  south oklahoma city community  college, oklahoma city, ok  academic  1999  tornadoes  moore public library, moore, ok  public  1999  tornadoes  norman regional hospital library,  norman, ok  special library  2000  tornadoes  fort worth central public library, tx  public  2000  earthquake  engineering library and fisheries  oceanography library, university of  washington, seattle, wa  special library  2000  blackout  helen de roy medical library,  providence hospital and medical  centers, mi  special library  2001  flood  houston academy of medicine – texas  medical center, houston,tx  archives and historical  collection  2001  tornadoes  national agricultural library, md  special library  2003  wild fire  san diego state university, san diego,  ca  academic  2003  wild fire  san bernadino public library, ca  public  2003  wild fire  escondido public library, ca  public  2003  wild fire  san diego county library, ca  public  2003  blackout  new york public library  public  2004  flood  university of hawaii at manoa  academic  2004  hurricanes  union county public library, nc  public  2005  hurricane  louisiana state university health  sciences center library  academic  2005  hurricane  archdiocese of new orleans, new  orleans, la  archives and historical  collection  2005  hurricane  pass christian public library, ms  public  2005  hurricane  vermilion parish library, la  public  2005  hurricane  delisle public schools, delisls,ms  school  2005  hurricane  mckinley high school, la  school  2005  flood  bonne ecole elementary, slidell, la  school  72 2005  flood  chalmette high school, st. bernard  parish,chalmette, la  school  2005  hurricane  ochsner clinic and foundation  library, la  special library  2006  flood  smithridge elementary school, reno,  nv  school  2006  toxic chemical  (chlorine) spill  byrd elementary and levelle‐ mccampbell middle school, aiken  county, ga  school    73 appendix b  information professionals responses to hurricanes katrina and rita  [distributed via surveymonkey.com 10/18/2005]    1. where is/was your information organization?  (if you are responsible for several libraries that were  affected differently by hurricanes katrina and/or rita please forward this survey to the individuals  who can answer questions at the local level.)  o city/town  o parish/county  o state  o zip code    2. please describe your information organization:  o academic library  o archives/historical repository  o public library  o school media center  o special library/information center  o other (please specify)    3. what was the pre‐hurricane size of your staff?  o number of full‐time staff _________  o number of part‐time staff _________    4. what was the pre‐hurricane size of your collection?  (please select whichever measures are most  appropriate for your organization.)   o volumes  o linear feet  o manuscript pieces  o microforms  o electronic resources  o other (please describe)    5. did your collection sustain physical damage as a result of hurricanes katrina and/or rita?  o no damage to collection  o minor damage to collection  o moderate damage to collection  o significant damage to collection  o don’t know condition of collection/unable to access  o other (please describe)    6. did your building sustain physical damage as a result of hurricanes katrina and/or rita?  o no damage to building  o minor damage to building  o moderate damage to building  o significant damage to building  o don’t know condition of building/unable to access  o other (please describe)  74   7. did your information organization lose equipment and/or furniture as a result of hurricanes katrina  and/or rita?  (please check all that apply.)  o computers  o copiers  o microform readers  o desks/chairs/tables  o shelving/storage units  o other (please describe)    8. did your staff size change as a result of the hurricanes?  o yes  o no    9. if yes how?  o decreased by: ____  o increased by: ____    10. in the aftermath of hurricanes katrina and/or rita did your library:  o continue operations without interruptions  o continue operations on a reduced basis  o suspend operations temporarily  o move to a new location  o close until further notice  o other (please describe)    11. if you are not currently providing services when do you expect to resume operations?  o approximate date: ________    12. if you are currently providing services did you add any new services in direct response to the  hurricanes?   o yes  o no    13. if yes please identify each new service added:  (please check all that apply.)  o extended hours  o new outreach/satellite locations  o new website with links to disaster relief information  o additional reference/referral services for displaced/evacuated users  o special services for displaced/evacuated children  o other (please describe)    14. how did you determine that there was a need for new services?  (please check all that apply.)  o personal observation  o requests from users  o discussions with internal/branch staff  o discussions with library board/advisory committee  o discussions with other information professionals (outside of organization)  75 o other (please describe)    15. if you added any new services in direct response to the hurricanes are you still offering them?  o yes  o no  o some but not all    16. please describe any obstacles that you encountered in adding and/or maintaining new services.    17. please list any services that you have eliminated as a result of the hurricanes.    18. did you have a disaster and/or recovery plan in place at the time of the hurricanes that specifically  addressed the potential need for new information services in response to a crisis?    o yes  o no    19. if you had a disaster and/or recovery plan have you been able to implement it?  o yes  o no    20. if not why not?  (please check all that apply.)  o damage was far greater than anticipated  o did not predict nature of the crisis  o lack of authority to implement plan   o lack of funds  o lack of personnel  o other (please describe)    21. please share any experiences and/or suggestions that you think would be helpful to other  information professionals facing a crisis situation.    22. if you would be willing to be interviewed about your libraryʹs responses to hurricanes katrina  and/or rita please fill out the following information:  o your name  o organizationʹs name  o phone  o email    microsoft word ed_glynn_10pt font evidence based library and information practice 2008, 3:3 1 evidence based library and information practice editorial evidence based practice outside the box lindsay glynn editor-in-chief acting head, public services, health sciences library memorial university of newfoundland st. john’s, newfoundland, canada e-mail: lglynn@mun.ca © 2008 glynn. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. i love food. i love cooking, baking, testing, and eating. i read about food preparation, food facts, and food service. over the years i’ve developed my fair share of knowledge about cooking and i’m a decent cook, but i’m no chef. i guess i’m what you’d call a “foodie”. however, i have the good fortune to have a friend who is a chef and owns one of the best, and certainly the most innovative, restaurants in town. during this summer i hosted a cooking class in my home for my family with my chef friend as instructor. the tex-mex barbecue theme was a big hit (you can contact me for recipes, if you like), but much more fascinating was the explanation of the science behind the cooking. it turns out that there is a term for this: molecular gastronomy. another term, and hence the genesis of my “eureka!” moment of the summer, is evidence based cooking. good cooking is not just following a recipe (not all of which are evidence based) but at its best is the culmination of heaps of tested information regarding why and how chemical and environmental factors work together to result in a gastronomical delight. for example, will brining or marinating a pork chop make it moister? and, if brining, what temperature should the water be, how long should it soak, and how much salt is needed? why does pounding meat increase its tenderness? what will keep guacamole from browning better – the pit or lime juice? what does baking soda do in a chocolate cake? eggs or no eggs in fresh pasta? like most librarians, i tend not to take information at face value. i want to know where information comes from and whether or not it is valid, based on specific factors. i’ve come to notice that evidence based, or evidence informed, practice is everywhere and has a tremendous impact on our lives. why do you rotate the tires on your car? evidence shows that the front tires wear evidence based library and information practice 2008, 3:3 2 more quickly (think about all those 3-point turns, the braking, etc) and therefore, switching the front tires with the back tires will increase the life span of the tires. ergonomics is based heavily on human factors research and usability testing; theme park design is largely affected by studies on the psychology of waiting; the study of local weather patterns and temperatures directly affects landscape architecture; advertising is strategically designed and presented based upon sociological studies such as demographics and viewing/reading behaviour. could one argue that natural evolution is a result of nature’s long-term evidence-based practice? i know that i’m not alone in finding this fascinating, especially since the basis of these studies is information. while my own work may focus more on medical and health information, i feel that as an information specialist i can take pride in the fact that the soul of my field, information, is essential to the world around me. think about this when you feel yourself getting cynical about research and the validity of your profession (don’t tell me you’ve never been there). i will be the first to admit that evidence is not always the be-all and end-all. there is a lot to be said for spontaneity (i.e. “hmmmm…i wonder if a sprig of fresh thyme would taste good in the mushroom sauce?”) and happenstance (i.e. “well, whaddaya know... penicillin!”). i certainly don’t think that every decision needs to be researched. but thank goodness for many research based decisions, without which the world would be a very different place and, heaven forbid, the perfect crème brulée may not exist. evidence based library and information practice using evidence in practice   sources of evidence to inform scholarly communication librarianship   allyson mower associate librarian and head, scholarly communication & copyright j. willard marriott library, university of utah salt lake city, utah, united states of america email: allyson.mower@utah.edu     received: 13 july 2018   accepted: 23 july 2018      2018 mower. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29468 setting   like many scholarly communication librarians, i have worked towards the goal of free and open access to scholarly articles and books by establishing an advocacy program at the university of utah, a public research university in salt lake city with approximately 2,000 faculty researchers. advocacy efforts from 2008 to 2014 included organizing open access week events, developing workshops about scholarly journal publishing, drafting a public access policy, promoting deposit in the institutional repository, and starting an open access article publishing fund.   problem   the advocacy program had mixed results. attendance at open access week events and publishing workshops, on average, was very low (the exceptions being the keynote lectures by john willinsky in 2009 and by john wilbanks in 2010). the public access policy failed because university administrators did not see the value of additional work time being dedicated to the deposit of articles since pubmed central was coming on board at the time. contributions to the institutional repository represented a small fraction of the institution’s overall research output and open access publishing occurred very little amongst the most prolific authors at the institution, based on a study conducted by my colleagues and myself (amos et al., 2012). the open access article publishing fund was popular amongst graduate students and assistant professors during the 2013-2014 pilot phase, but it did not receive ongoing funding from university administrators in 2014 or 2015.   this range of experiences led me to reconsider at what level the goal of public and open access resonated with researchers at the institution and whether or not the expectations placed on them by their peers and the institution’s leaders incentivized free and open distribution of research articles and books. rather than continue with the advocacy program, i decided to take a step back and look for additional sources of evidence to determine the level of commitment to public and open access and to further inform a scholarly communication program.   evidence   i started with the institution’s history. i had read the founding charter and early employment policies for a previous research project and expanded that research by looking at the works written by employees and their distribution. this required that i develop a new research methodology called university bibliography, which i detail below. after compiling the historical data, i then analyzed current employment statements for retention, promotion, and tenure (rpt) to see what contemporary researchers expect of themselves, their peers and, by extension, what the institution supports in terms of scholarly communication.   these sources of evidence uncovered valuable information and provided a foundation from which to develop new scholarly communication services, also detailed below, that focus less on advocacy and more on meeting discipline-specific needs of researchers as detailed in faculty-led employment expectations.   for example, the university bibliography revealed that, historically, faculty at the institution utilized commercial distribution as the primary means of communication. one of the first books from an author at the university, titled the school and primary songster, was published by coalter & snelgrove in 1889, a commercial publisher in salt lake city, utah. the examined variant of the book did not list a price (no records from coalter & snelgrove survive), but re-published copies of the book are currently for sale on amazon for $22.95. the book was held by libraries (n=18) so a form of public access was realized, but on a small scale. similarly, the first journal article from an author at the university was published in the commercial publication science: a weekly record of scientific progress in 1892. at the time, the journal had 3,000 individual subscribers (subscription price unknown) and the individual article is still for sale as part of the science platform offered by the american association for the advancement of science (mower, 2017). it is not known if libraries were amongst those 3,000 subscribers, but over 3,000 libraries currently subscribe to science, according to worldcat.   the bibliographic data also showed that historic faculty did not limit the audience of their communication to fellow scholars; they wrote for broad audiences and published when and where possible in order to make information from their areas of interest known to those inside and outside the classroom. evan stephens, who wrote the school and primary songster, did so to “awaken [people’s] intellectual faculties” and to provide “practical reading of vocal music in public schools or classes” (stephens, 1889).   similar to findings from the historical data, current data from rpt statements revealed an incongruence between public access ideals and employment expectations in the area of research. unlike historical faculty, however, currently employed researchers at the institution expect each other to only communicate findings to fellow scholars and not to those outside their disciplines. like historical faculty, though, current researchers do not expect each other to offer research output free of cost to the general public. on the point of free access to the public, both historical and modern faculty at the institution agreed.    these findings led me to more deeply consider what the best approach to a scholarly communication program on campus would be. it seems worth noting here that the unique feature of faculty employment is the ability to set the criteria and standards by which to judge each other’s work performance. it is not administrators alone who determine work expectations and evaluate performance. faculty share (alongside department chairs, deans, provosts, and presidents) in the drafting of criteria, standards, and guidelines utilized in the peer review process. the criteria for research, especially, can serve as an indicator of what a department or college faculty value when it comes to scholarly communication. it serves as the main source of evidence for scholarly communication librarianship, in my opinion. to take it one step further, helping faculty understand that they can update these criteria if or when their values change becomes a crucial component of scholarly communication librarianship. knowing the institution’s scholarly communication history greatly informs these conversations and, in my experience, brings clarity to a scholarly communication librarian’s role within an academic institution.   university bibliography   to better understand the university’s scholarly communication history, i researched what authors wrote, tracked how those works were distributed, and pinpointed when research, writing, and publishing become an employment expectation. in this process, i created a descriptive bibliography on an institutional level. descriptive bibliography, in general, seeks to uncover the states and variants of printed books by any given writer (yee, 2007). a university bibliography is somewhat different in that it intends to show the distribution and ownership of books and articles by more than a single author. the organizing principle becomes the institution rather than the writer, but details about individual authors are included. the bibliography is descriptive because it covers basic information such as title of the work, author of the work, author’s discipline, publication date, and, as much as possible, the historical distribution of the work, the rights holder, and the number of copies currently held by libraries or digital archives.   compiling the university bibliography required several sources, starting with historical employment records and policies. these can be difficult to obtain, but i was able to find them by doing research at university archives. the purpose of consulting historical policies about employment was to obtain details on work requirements. it also contributed to a more accurate list of historical employees to search in bibliographic databases. if an employee was hired and listed as a faculty member, i included the name in the data set because the institution expected faculty to teach, at the outset, and later expected them to research, write, and publish.   with the data set of historical employment, i searched various sources to determine if an individual employee authored any books or journal articles. the bibliographic sources included library catalogs, institutional finding aids, and journal article indices. starting with the library’s catalog and finding aids, i searched the employee’s name. if the catalog retrieved any works, i added those details to my data set. i also broadened my search to include cooperative and national catalogs such as worldcat, hathitrust, and internet archive to discover distribution and physical ownership of the work outside the university. to determine if the historical employee wrote either magazine or journal articles, i searched the name using web of science, scopus, and google scholar. the details of any written works found in these sources was added to the data set.   i used the data to also search for any historical copyright ownership records available from the u.s. copyright office through the internet archive or the book renewal records available from stanford university libraries. if the employee authored a journal article and not enough was known about the journal to search historical copyright records, i consulted university microfilms international’s (umi) american periodicals 1741-1900: an index to the microfilm collections to find any publisher or editor names associated with the journal.   umi’s index provided valuable data on the history of journals in america, but discovering the distribution of a single journal article was difficult. it required web-based searching on the journal itself in the hope that someone else had studied or tracked the historical distribution of the journal. such studies were only found on a couple of occasions, which i would then include in the online bibliographic entry.   determining readership and impact of a scholarly article was equally difficult. i relied on google scholar to obtain basic citation statistics to help convey information about impact. most historical articles had not been cited and i have yet to uncover any personal narratives detailing the readership of specific scholarly works. researching article distribution presents many challenges, some of which cannot be fully resolved, but even the most cursory distribution data paints a more complete picture of the history of scholarly communication at the university.   employment statements for retention, promotion, and tenure   in addition to wanting a historical understanding of scholarly communication at the university, i also needed in-depth familiarity with existing scholarly communication requirements. to that end, i analyzed current employment statements in order to uncover what faculty in various departments at the institution expected of each other when it came to research and publishing. i saw it as a way to inform what type of scholarly communication services to provide.   i compiled a convenience sample through a web-based search. the sample consisted of six department level statements from five colleges. i read through each statement and looked more closely at what each department said regarding research. i found that all of them expected original, independent research to get communicated with peers in the discipline. none of them listed required journals and none of them expected the research to get shared free of cost to the general public. none of the employment statements required grant funding to progress in ranks, either. the focus remained on original, independent, and sustained research to get critiqued by those in the discipline. the most telling feature of the current statements was the fact that half of them included publishing quotas as a promotion and tenure requirement. this was not an original feature of the historical employment policies at the institution. determining when it got introduced is research-in-progress and will be valuable information in my work as a scholarly communication librarian.    implementation   this type of analysis led to a greater realization of the importance of developing scholarly communication services that match what faculty authors expect of themselves and each other. i synthesized and re-interpreted the content of the rpt statements into a tailored reference interview, which i then incorporated into a new service called researcher and author services:   scholarly communication reference interview 1.      what do you consider your research area to be? 2.      are there key historical moments that have influenced what your peers study today? 3.      are there big changes going on in your discipline right now? 4.      what research outputs are common in your field? 5.      what research outputs do your peers value? 6.      what are the research expectations for your department or college? 7.      what questions would you like to investigate? 8.      what works could the library purchase to support your research? 9.      which library service would further your research? the list of services that get discussed as part of question #9 represent a collaborative effort with several librarians at the university. representing a range of librarianship, services include collection development, data visualization, copyright management, digital scholarship services, and data management. outcome   i have held several one-on-one consultations with faculty seeking to better define their research area, determine what has already been done in their fields, strategize on how to advance the field, and purchase the exact primary material the researcher needs to further her work. the consultations also focus on perceived hurdles such as copyright permission and management, how to find the right journal, and what to expect after submitting a manuscript.   reflection      conducting historical research and consulting employment statements provided valuable evidence to inform a scholarly communication program and incorporate additional elements that went beyond advocating for free and open access to scholarly articles and books. the research provided greater perspective and appreciation for the ways in which authors at a university operate as employees. the most striking realization was that faculty influence the criteria and standards used to evaluate their performance and develop their own expectations regarding distribution and audience. the research led to more tailored scholarly communication services. if advocacy returns as a component of the scholarly communication program, perhaps it could focus on bringing awareness to the existence of publishing quotas in a department’s rpt statement and utilized to initiate dialogue with the department faculty about what they deem essential for contributing to their discipline’s scholarly communication.      references   amos, k., mower, a., james, m. a., weber, a., yaffe, j., & youngkin, m. (2012). exploring publishing patterns at a large research university: implications for library practice. evidence based library and information practice, 7(3), 32-50. https://doi.org/10.18438/b86311   mower, a. (2017). a1. in the university of utah, its authors, and their works, 1883-1970. retrieved from: https://authors.lib.utah.edu/category/section-a/a1/   stephens, e. (1889). the school and primary songster. salt lake city, ut: coalter & snelgrove.   yee, m. m. (2007). cataloging compared to descriptive bibliography, abstracting and indexing services and metadata. cataloging & classification quarterly, 44(3/4), 307-327. retrieved from: https://escholarship.org/uc/item/18g6j3nb       research article   exploring the impact of individualized research consultations using pre and posttesting in an academic library: a mixed methods study   lindsey sikora research librarian health sciences library university of ottawa ottawa, ontario, canada email: lindsey.sikora@uottawa.ca   karine fournier research librarian health sciences library university of ottawa ottawa, ontario, canada email: karine.fournier@uottawa.ca      jamie rebner masters of human kinetics university of ottawa ottawa, ontario, canada email: jrebn066@uottawa.ca   received: 30 aug. 2018                                                                  accepted: 29 jan. 2019      2019 sikora, fournier, and rebner. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29500     abstract   objective – academic librarians consistently offer individualized help to students and researchers. few studies have empirically examined the impact of individualized research consultations (ircs). for many librarians, ircs are an integral part of their teaching repertoire. however, without any evidence of an irc’s effectiveness or value, one might ask if it’s worth investing so much time and effort. our study explored the impact of ircs on students' search techniques and self-perceived confidence levels. we attempted to answer the following questions: 1) do ircs improve students’ information searching techniques, including the proper use of keywords and/or subject headings, the accurate use of boolean operators, and the appropriate selection of specialized resources/databases? 2) do ircs influence students’ confidence level in performing effective search strategies?   methods – our study used a mixed-methods approach. our participants were students from the faculties of health sciences and medicine at the university of ottawa, completing an undergraduate or graduate degree, and undertaking a research or thesis project. participants were invited to complete two questionnaires, one before and one after meeting with a librarian. the questionnaires consisted of open-ended and multiple choice questions, which assessed students' search techniques, their self-perceived search techniques proficiency and their confidence level. a rubric was used to score students' open-ended questions, and self-reflective questions were coded and analyzed for content using the software qsr nvivo.   results – twenty-nine completed pre and posttests were gathered from february to september 2016. after coding the answers using the rubric, two paired-samples t-tests were conducted. the first t-test shows that students’ ability to use appropriate keywords was approaching statistical significance. the second t-test showed a statistically significant increase in students’ ability to use appropriate search strings from the pretest to the posttest. we performed a last paired-samples t-test to measure students’ confidence level before and after the appointment, and a statistically significant increase in confidence level was found. conclusion – out of three paired t-tests performed, two showed a statistically significant difference from the pretest to the posttest, with one t-test approaching statistical significance. the analysis of our qualitative results also supports the statement that ircs have a positive real impact on students’ search techniques and their confidence levels. future research may explore specific techniques to improve search strategies across various disciplines, tips to improve confidence levels, and exploring the viewpoint of librarians.       introduction   in the current digital age, a university student’s challenge is not finding information, but rather locating the appropriate, validated, and trustworthy information required. librarians support students in this challenge in various ways, including in-class instructions, specialized workshops, and reference desk assistance. more specifically, individualized research consultations (ircs) between librarians and students have been increasing, with librarians spending less time at the reference desk. this shift in service appears to be a trend in many academic libraries. for the purpose of this study, ircs were defined as scheduled appointments that aim to help students with their research projects, including, but not limited to, the literature review process.   in a scoping review, fournier and sikora (2015) found that though ircs have been taking place for decades, the impact of these meetings on a student’s information literacy (il) skills is challenging to measure. the authors reviewed 20 articles for assessment methods, with the following techniques identified: 1) usage statistics; 2) surveys; 3) objective quantitative methods. while many libraries use statistics and surveys for assessment purposes, only three articles examined using objective quantitative methods as a measure of the impact of ircs on il skills (fournier & sikora, 2015). it is extremely difficult to evaluate an irc service objectively (schobert, 1982). however, this does not mean it should not be attempted. the three studies trying to measure this impact used different approaches. donegan, domas, and deosdade (1989) wanted to demonstrate impact between group instruction and term paper counselling, while erickson and warner (1998) examined whether getting individual tutorials vs. no tutorials would change the assessment. the authors were unable to demonstrate a statistically significant difference in the impact of ircs on students’ il skills. reinsfelder (2012) found a statistically significant difference in his study, which investigated il skills using citation analysis to compare students’ draft and final papers in a course. reinsfelder concluded that “some quantitative evidence demonstrating the positive impact of individual research consultation” (p. 263) had been proven.   as there is a paucity in the literature surrounding objective quantitative methods evaluating the impact of ircs on students’ il skills after meeting with a librarian, we sought to present a new method using pre and posttests to examine students’ database searching skills by using a rubric to analyze their search strategies.     literature review   it is well known that interactions occurring at the traditional library reference desk are declining (association of research libraries, 2015). however, the demand for librarians to offer more personalized, in-depth services to students and faculty has remained stable, or even risen (covert-vail & collard, 2012). these services often involve a librarian’s comprehensive knowledge of resources and strategies tailored to locate the appropriate information. ircs can serve as one way to connect students to librarians with such expertise.   user surveys and feedback forms have provided librarians with comments from students, illustrating the usefulness of ircs (butler & byrd, 2016). researchers have discussed the benefits ircs can provide for students, such as the “overwhelming usefulness” students often reported following a one-on-one meeting with a librarian (butler & byrd, 2016), the opportunity to aid in developing students’ problem-solving skills (fields, 2006), the overall positive patron experience with academic library research consultations (rogers & carrier, 2017), or the increase in goodwill between libraries and faculty members that extends beyond the library environment (handler, lackey, & vaughn, 2009). while these interactions have positive connotations to encourage ongoing relationships between librarians and students, they are subjective in nature, and do not provide an objective method to analyze a student’s success in developing future research skills.   over the last several decades, few researchers have attempted to assess ircs quantitatively, as it is challenging to quantitatively prove their effectiveness. reasons for these challenges vary, including not having the appropriate instrument to evaluate ircs, the topics of the ircs can be difficult to compare, and librarians have various ways in which they conduct their ircs. nevertheless, researchers have tried to surmount this challenge by utilizing different quantitative approaches. bergen and macadam (1985) analyzed the number and type of students (male vs. female, freshman to seniors, in various departments) who used a voluntary one-on-one instruction service. in 1989, donegan et al. used objective quantitative methods such as post-instruction testing by creating a multiple choice test that was given to students immediately following an instruction session. reinsfelder (2012) and sokoloff and simmons (2015) examined ircs using citation analysis within the management and business fields. reinsfelder evaluated the quality of citations used in undergraduate papers, before and after meeting with students individually, whereas sokoloff and simmons created an il rubric, adapted from the association of american colleges and universities rubric, to analyze the performance standards of their group of students. however, no researchers have specifically assessed the impact of ircs in the health sciences and medicine fields.   acrl framework for information literacy for higher education   the design for our pre/posttest questionnaire, as well as our rubric for assessing the students’ search strategies, was informed by the new acrl framework for information literacy for higher education sixth concept: searching as strategic exploration (association of college and research libraries, 2016). this framework states that searching for information is often nonlinear and iterative, requiring the evaluation of a range of information sources and the mental flexibility to pursue alternate avenues as new understanding develops. it goes on to state that as the searching process is complex and often daunting for students, meeting with a librarian permits them to become a more advanced searcher by allowing them to “search more broadly and deeply to determine the most appropriate information with the project scope” (p.9). we hypothesize that by matching students’ information needs and search strategies to the appropriate search tools, such as specialized bibliographic databases, we are able to help them design and refine their search strategies as necessary, based on their search results.   pre and posttesting methodology as previously stated, studies dedicated to the quantitative assessment of ircs are scarce, and even fewer using a pre and posttest methodology have been found in the literature. in light of this gap, we reviewed the literature evaluating group instructions using a pre and posttest method, focusing on their methodology and test design, in order to prepare our questionnaires.   many studies use multiple choice questions as their pretest and posttest design to assess il skills. multiple choice questions have been used to assess one-shot sessions (bryan & karshmer, 2015), il credited courses (goebel, neff, & mandeville, 2007) and library instructions classes (chiarella, khadem, brown, & wrobel, 2014; ivanitskaya, duford, craig, & casey, 2008). they have also been used to compare online vs. face-to-face library instructions, whether with one-shot face-to-face instructions (mery, newby, & peng, 2012), or with face-to-face workshops (shaffer, 2011). understandably, multiple choice questions provide quantifiable data to assess students’ il skills, making multiple choice the evaluation method of choice in many studies. however, other types of assessment techniques have appeared in the literature. open-ended questions have been used for pre and posttesting to capture students’ understanding of il concepts (cook & walsh, 2012; gross & latham, 2013; wakimoto, 2010).   further, pre and posttesting methodologies have been found to be successful outside of the library literature. shivaraju, manu, vinaya, and savkar (2017) evaluated knowledge of didactic lecturing among medical students through a pre and posttest questionnaire based evaluation technique. they analyzed how much students were aware of pharmacology concepts before the lecture, and evaluated the students’ learning of key concepts following the lecture. their results found that students’ understanding improved following the lecture, as they were able to improve their focus towards the lecture, which improved their overall performance in pharmacology. these findings are also corroborated in other medical schools using this methodology in medical education (cramer & mahoney, 2001; muthukumar, d’cruz, & anandarajan, 2013).    self-efficacy theory   bandura’s (1977, 1997) self-efficacy theory was used as an inspiration to help design our questionnaires. more specifically, research taking place in a library setting using self-efficacy theory was sought out. the term self-efficacy “refers to a person’s belief in his or her own capability to perform specific activities or tasks” (ren, 2000, p.323). ren (2000) tested students before and after library instruction on the following qualities: their self-perceived search performance, their attitude about acquiring search skills, and their emotions while completing an assignment. the author concluded that in order “for self-efficacy to increase, students must have adequate searching practice, experience learning accomplishments and not be overwhelmed with negative emotions such as confusion and frustration” (ren, 2000, p. 327). serap kurbanoglu (2003) explored the relationship between university students’ il and their self-efficacy beliefs. the author concluded that more research needs to be conducted to better understand how self-efficacy beliefs affect individuals’ information problem solving behaviours and lifelong learning activities.    aims   for our project, we issued a pre and posttest questionnaire, evaluating students’ searching techniques in medical databases such as medline (via ovid), before and after meeting with a librarian. we also wanted to gain insight into their self-perceived ability to search the databases by measuring their self-efficacy. we then assessed their search strategies with a rubric we designed (table 1). research questions   a)       do ircs improve students’ searching techniques, including the proper use of keywords and/or subject headings, the accurate use of boolean operators, and the appropriate selection of specialized resources/databases? b)      do ircs influence students’ confidence levels in performing effective search strategies?   objectives   our study’s primary goal was to evaluate the impact that ircs have on students’ search techniques and their confidence levels, with the following objectives:   a)       assessing students’ search techniques before and after they meet individually with a librarian. b)      discovering what factors influenced students’ self-perceived search techniques proficiency and their self-perceived confidence level of such search techniques. c)       determining if an irc influences students’ confidence levels in performing effective search strategies. d)      exploring students’ expectations and their satisfaction levels with ircs.   methods   population   the university of ottawa has over 40,000 students in attendance (university of ottawa, n.d.). there are 4,500 students within the departments of the faculty of health sciences, which include nursing, rehabilitation, nutrition, human kinetics, and interdisciplinary health sciences. the faculty of medicine includes the school of medicine, postgraduate students, epidemiology and public health, population health and bench science programs, totaling 2,250 students. participants included a convenience sample of university of ottawa students who were completing an undergraduate or graduate degree in the faculties of health sciences or medicine, and also undertaking a research or thesis project.   data collection   in order to assess the impact of ircs on students’ searching techniques, a mixed-methods approach was used. pre and posttesting were used, and ethics approval was received from the university of ottawa, office of research ethics and integrity, file number was h12-14-03.   the first round of data collection took place in 2015, but without a monetary incentive, very few participants completed the posttest (n = 9). additional academic disciplines were also involved in this round of data collection including management, social sciences, arts, and humanities. we found that the topics and resources covered in ircs can fluctuate greatly between disciplines. for that reason, we decided that the second round of data collection would be concentrated on a more homogenous group: health sciences and medicine. this method would allow a better comparison group between students. the first round of data collection acted as a pilot, allowing a review of the questionnaires, with several questions being adjusted to increase clarity. data from the first round of data collection is not included in the results listed below.   the second round of data collection took place from february to september 2016. even with a monetary incentive, it was challenging to recruit participants (n = 29). the pre and posttest questionnaires can be found in appendix b.   in addition to the authors, two other librarians employed by the university of ottawa were included in the second round of data collection. when a student contacted a librarian for assistance, a recruitment email was sent to the student, which contained a brief description of the study, and the links to the consent form and the pretest questionnaire. at the end of the first questionnaire, participants were asked if they wished to complete the second questionnaire (posttest). if their answer was affirmative, the online survey system (fluidsurveys) would send them an invitation one week after the first questionnaire was filled up. using this method of recruitment allowed complete anonymity for the participants, that is, none of the librarians, including the present study’s authors, providing ircs knew if the students they were helping had participated in the study or not. this anonymity helped to reduce bias, in the sense that librarians wouldn’t change their approach or their attitudes toward students depending on whether they participated in the study or not. librarians were asked to use a search strategy worksheet (see appendix a) with every student they met during an irc for the duration of the study, whether they were participating in the study or not. this worksheet is frequently used during regular ircs at this library, outside the scope of this study, therefore, no training of the librarians was required.    the questionnaires consisted of open-ended and self-reflective questions (see appendix b for the pre and posttest full questionnaires). the open-ended questions assessed students’ search techniques, specifically their choice of keywords, synonyms, subject headings, and the creation of a search string with the appropriate use of boolean operators. the self-reflective questions assessed students’ self-perceived proficiency with search techniques, their confidence level in their search techniques, and their expectations of (before) or their satisfaction with (after) the irc. to preserve anonymity, once data collection was complete, students’ personal information was removed and replaced with an anonymous identifying number (e.g., “student 1”) in both questionnaires.   rubrics were created as a multi-purpose scoring tool to assess student performance. while rubric development can stop after the performance criteria have been identified and performance levels established (wolf & stevens, 2007), more comprehensive rubrics include another step in which each of the cells in the matrix contains a description of the performance at that level. we created a rubric to code open-ended questions on search techniques (table 1), capturing details to assess the appropriate use of keywords and the search strategy. the rubric scoring was completed by one of the study’s authors (jr).      table 1 rubric used to assess the pre and posttest results for the appropriate use of keywords and the search strategy requirement insufficient (0) acceptable (1) superior (2) uses appropriate keywords no keywords provided, or if keywords provided, very little connection to the research question or topic and are too broad. no use of synonyms. keywords provided are connected to the research question or topic, but not all subjects are covered. keywords are somewhat focused and not too broad. synonyms used, if applicable. very little, or no use of subject headings (optional). keywords provided are connected to the research question or topic and all subjects are covered. keywords are well focused. appropriate use of synonyms, if applicable. appropriate use of subject headings (optional). builds appropriate search string no search string provided. search string provided with some errors or missing elements (e.g.: not all keywords are present; mistakes in the use of boolean operators) search string provided with no errors and all elements are present (all keywords are present, no errors with the use of boolean operators)     results   our sample size was small, with only 29 completed pre and posttests. pre and posttest self-reflective, open-ended answers were coded and analyzed with the use of the software qsr nvivo. multiple choice and likert scale questions were analyzed using spss. results are presented following the study’s outlined objectives.   the first objective was to assess students’ search techniques before and after they met individually with a librarian. to do so, we asked participants to provide their list of keywords and search strings before meeting with a librarian, if they already had done some searching by themselves (e.g., (marine or ocean) and (biology or science)). to assess if their keyword and search string selection were accurate and appropriate, we asked participants to state their research topic or question. we were then able to use our rubric (table 1) to code their answers. two paired-sample t-tests were conducted to evaluate the impact that a consultation with a librarian had on students’ ability to appropriately use keywords and build search strategies. the first t-test showed that students’ ability to use appropriate keywords from the pretest (m = 1.00, sd = .66) to the posttest (m = 1.34, sd = .72, t (28) = -1.98, p > .05, two-tailed) was approaching statistical significance. the mean increase in score was .345 with a 95% confidence interval ranging from -.70 to .01. the eta squared statistic (.12) indicated a large effect size. the second t-test showed a statistically significant increase in the students’ ability to use appropriate search strategies from the pretest (m = .21, sd = .41) to the posttest (m = .76, sd = .79), t (28) = -3.59, p = .001 (two-tailed). the mean increase in score was .55 with a 95% confidence interval ranging from -.87 to -.24. the eta squared statistic (.32) indicated a large effect size.                 the second objective was to discover factors that influenced students’ self-perceived search techniques proficiency and confidence level. we asked participants which factors influenced their confidence level before and after the irc. before the appointment, both positive and negative factors were stated in almost equal measure, with positive factors rated slightly higher. negative factors were grouped by the following themes:   1)       lack of available research 2)       research topic difficulty 3)       lack of prior knowledge 4)       difficulty using databases   positive factors were categorized under the themes:   1)       prior knowledge 2)       help from other people (colleagues, supervisors)   after the appointment, the factors that influenced students’ confidence level were almost all positive, and were grouped under the following themes:   1)       new or prior knowledge 2)       support from others 3)       strength of research question or search string   there were no statistically significant differences found between any of the themes presented.    our third objective was to determine if the irc influenced students’ confidence levels in performing effective search strategies. to answer that objective, we measured student confidence level before and after the appointment. we asked participants how confident they were in finding relevant sources of information, using a scale from 1 (“not confident at all”) to 10, (“very confident”). in the pretest, the mean was 5.85 (table 2), and in the posttest, the mean was 7.24 (table 3).   we performed a paired-samples t-test to evaluate the impact that meeting with a librarian had on students’ confidence with regard to finding relevant sources of information. there was a statistically significant increase in confidence level from the pretest (m = 5.93, sd = 1.46) to the posttest (m = 7.24, sd = 1.46), t (28) = -4.34, p < .001 (two-tailed). the mean increase in confidence was 1.31 with a 95% confidence interval ranging from -1.93 to -.69. the eta squared statistic (.40) indicated a large effect size.    we also asked participants in the posttest if the appointment with a librarian influenced their confidence level, and 96.6% of respondents said “yes” (table 4). when asked to describe how the appointment with a librarian changed their confidence level, participants provided positive comments, which we compiled under three main themes:   1)       finding useful resources 2)       learning how to properly search databases 3)       learning how to execute a search strategy   furthermore, students were asked how the appointment with a librarian might have influenced various elements of their research project. a response rate of 63% of participants mentioned that the appointment with a librarian influenced their keyword selection (table 5). table 6 illustrates that 45% of students mentioned that the irc influenced their search strategy, while 10% mentioned that it did not. however, it should be noted that 45% of respondents didn’t have a search strategy before meeting with a librarian, which is why they answered “does not apply” to that question (table 6).     table 2 pretest confidence level on a scale from 1 to 10   minimum maximum mean on a scale from 1 to 10, where 1 represents “not confident at all” and 10 represents “very confident”; how confident are you with finding relevant sources of information? 3 9 5.85     table 3 posttest confidence level on a scale from 1 to 10   minimum maximum mean on a scale from 1 to 10, where 1 represents “not confident at all” and 10 represents “very confident”; how confident are you with finding relevant sources of information? 4 10 7.24     table 4 appointment with librarian influenced students’ confidence in their search techniques   frequency percentage (%) yes 28 96.6 no 1 3.4 total 29 100     table 5 students’ keyword selection had changed after meeting with a librarian   frequency percentage (%) yes 17 63 no 10 37 total 27 100     table 6 students’ search strategy had changed after the appointment with a librarian   frequency percentage yes 13 44.8 no 3 10.4 does not apply 13 44.8 total 29 100     the last objective explored students’ expectations and their satisfaction levels with the irc. students were asked if their expectations were met after meeting with a librarian using a likert scale from 1 to 10, where 1 represented “expectations not met at all” and 10 represented “exceeded expectations”, and 86.1% of respondents answered 7 or higher. we also asked participants to describe how their expectations were or were not met. participants’ answers were grouped into three themes:   1) my expectations were met since i learned the appropriate resources and information-seeking knowledge. 2) my expectations were met because i learned how to search properly. 3) my expectations were not met because the appointment time was used to teach me how to use the resources rather than to find all available information.   discussion   our study demonstrated that students who met with a librarian for an irc improved their search strategies. although there wasn’t a significant statistical difference indicated on the pre/posttest questionnaire with regard to the students’ ability to use appropriate keywords, there was a statistically significant increase in the students’ ability to use appropriate search strategies overall. these strategies may include the choice of keywords, synonyms, subject headings, and the creation of a search string with the appropriate use of boolean operators. this indicates that while individual keywords still pose a challenge for students, their overall strategies for searching have holistically improved.   additionally, there was a statistically significant increase in the students’ confidence with regard to retrieving relevant sources of information, after having met with a librarian. the analysis of our qualitative results also supported the positive impact that ircs have on students’ search techniques, as participants indicated that their expectations were met as they learned how to search properly, and how to use the appropriate resources.   although participants’ confidence levels significantly increased after meeting with a librarian, we noted that 12 out of 29 respondents indicated a confidence level of 7 or higher prior to the appointment with a librarian, and mainly stated “prior knowledge” as a factor influencing their confidence level. prior knowledge may include previous searching experience for another research paper or with a particular database, or familiarity when searching for their specific research topic. it could then be inferred that many participants had a high self-perceived confidence in their own search techniques prior to meeting with a librarian. as maddux and volkmann (2010) stated: “people who maintain strong self-efficacy beliefs during self-regulatory efforts are […] more likely to persevere” (p. 317). in other words, to help oneself to self-regulate (the process by which people control their thoughts, feelings and behaviours), one has to believe in one’s own capabilities to perform the task at hand in order to do it. the students had likely completed previous searches and felt confident leading up to their meeting with the librarian.   another possible reason for this high confidence level could be the information-seeking behaviours exhibited by the digital generation. keshavarz, esmaeili givi, and vafaeian (2016) studied il self-efficacy in graduate students and found that a high degree of their self-efficacy stemmed from their confidence levels, as well as their motivation and proficiency. their results are consistent with what we discovered. however, once they meet with a librarian, they learned how to use new resources they hadn’t previously considered, with new search techniques that they did not possess previously (keshavarz et al, 2016). with the plethora of scientific literature easily retrievable from the internet, many students might think they are self-sufficient, or do not require professional help, but once they learn what specialized databases and strong search strategies can provide, they appreciate the new knowledge they have acquired after meeting with a librarian.   our study is unique, as it is one of the first to quantitatively examine student improvement with search strategies in the health sciences. while our methods were not without validation (i.e., the use of a rubric), it does allow future research to build on it in order to create methods that can become validated and reliable. it may also demonstrate a quantitative return on investment (roi) for libraries, showing the impact that librarians play in the role of student learning, however, this would require further research. librarians often must defend their impact in a research environment quantitatively, and this may be one manner in which it could be measured.    limitations   our study is not without limitations. firstly, there were only 29 completed questionnaires for the pre and posttesting period. a higher response rate would increase the significance of the results. also, the sample of students was a convenience sample, and therefore, not representative of the student population.   secondly, assessing individualized consultations is challenging, as the field of study involved or the type of sources needed are dependent on the research question. as such, individualized consultations are not identical. therefore, attempting to compare them is challenging due to the different variables each consultation brings to the table. we tried to limit the variability as much as possible by limiting the fields of study (only health sciences and medicine), and requesting participants be involved with a research project. thirdly, our rubric was not validated. it is true that rubrics can positively contribute to student learning and program improvement by ensuring that the learning target is more clear, guiding the instructional design and delivery, and making the assessment process more accurate and fair  (wolf & stevens, 2007). however, without piloting and assessing the rubric properly, in order to adapt it as needed, the validity of the process can be questioned. although we did pilot our rubric during our first round of data collection with all disciplines, performing a second pilot with our more targeted audience of only health sciences and medicine students would have been beneficial.    future studies on this topic should include qualitative data from interviews conducted with librarians to examine their perceptions of an effective irc. as well, specific focus groups with students may also alert librarians to challenges and barriers that were not originally anticipated. additional research involving ircs is certainly needed, and future studies could examine the similarities and differences between disciplines in order to adequately meet the unique needs of students in those fields. conclusion   with the study’s limitations in mind, we can affirm that, overall, ircs have a positive impact on students’ search techniques and their confidence levels. library services are rapidly changing, and assistance to students takes many forms. in-person, one-on-one tailored help is tremendously appreciated by students and should be kept as an additional service offered to students. anecdotally, the health sciences library at the university of ottawa has seen an increase in ircs provided at a distance via skype. this could be an additional method to continue offering this dedicated individualized assistance to students going forward.   conflict of interest   two grants aided in the execution of our research project. we received $600 from the university of ottawa library research grant, which was used to provide a $10 incentive for completing the pretest, and another $10 incentive for the posttest. the full amount received was used for incentives. we believe that the incentive helped gather enough completed posttests, as many students often completed the first questionnaire but didn’t usually bother to complete the second one. we received a second grant of the amount of $1,500 from the carl research in librarianship grant. funds from that grant were used to pay for a research assistant (jr), who was instrumental by helping with our data analysis.   references   association of research libraries. (2015). service trends in arl libraries, 1991-2015. retrieved from https://www.arl.org/focus-areas/statistics-assessment/statistical-trends#.xikvuihki70   association of college and research libraries. (2016). framework for information literacy for higher education. chicago, il: author. retrieved from http://www.ala.org/acrl/standards/ilframework   bandura, a. (1977). self-efficacy: toward a unifying theory of behavioral change. psychological review, 84(2), 191-215.   bandura, a. (1997). self-efficacy: the exercise of control. new york, ny: w.h. freeman.   bergen, k., & macadam, b. (1985). one-on-one: term paper assistance programs. rq, 24(3), 333-340. retrieved from  https://www.jstor.org/stable/pdf/25827393.pdf   bryan, j. e., & karshmer, e. (2013). assessment in the one-shot session: using pre-and post-tests to measure innovative instructional strategies among first-year students. college & research libraries, 74(6), 574–586. https://doi.org/10.5860/crl12-369   butler, k., & byrd, j. (2016). research consultation assessment: perceptions of students and librarians. the journal of academic librarianship, 42(1), 83-86. https://doi.org/10.1016/j.acalib.2015.10.011   chiarella, d., khadem, t. m., brown, j. e., & wrobel, m. j. (2014). information literacy skills retention over the first professional year of pharmacy school. medical reference services quarterly, 33(3), 302–312. https://doi.org/10.1080/02763869.2014.925693   cook, p., & walsh, m. b. (2012). collaboration and problem-based learning. communications in information literacy, 6(1), 59–72. retrieved from https://pdxscholar.library.pdx.edu/comminfolit/vol6/iss1/6/   covert-vail, l., & collard, s. (2012). new roles for new times: research library services for graduate students. retrieved from the association of research libraries website: http://www.arl.org/storage/documents/publications/nrnt-grad-roles-20dec12.pdf   cramer, j. s., & mahoney. m. c. (2001). introducing evidence based medicine to the journal club, using a structured pre and post test: a cohort study. bmc medical education, 1(6). https://doi.org/10.1186/1472-6920-1-6   donegan, p. m., domas, r. e., & deosdade, j. r. (1989). the comparable effects of term paper counseling and group instruction sessions. college & research libraries, 50(2), 195–202. https://doi.org/10.5860/crl_50_02_195   erickson, s. s., & warner, e. r. (1998). the impact of an individual tutorial session on medline use among obstetrics and gynaecology residents in an academic training programme: a randomized trial. medical education, 32(3), 269-273. https://doi.org/10.1046/j.1365-2923.1998.00229.x   fields, a. m. (2006). ill‐structured problems and the reference consultation: the librarian's role in developing student expertise. reference services review, 34(3), 405–420. https://doi.org/10.1108/00907320610701554   fournier, k., & sikora, l. (2015). individualized research consultations in academic libraries: a scoping review of practice and evaluation methods. evidence based library & information practice, 10(4), 247-267. https://doi.org/10.18438/b8zc7w   goebel, n., neff, p., & mandeville, a. (2007). assessment within the augustana model of undergraduate discipline-specific information literacy credit courses. public services quarterly, 3(1–2), 165–189. https://doi.org/10.1300/j295v03n01_09   gross, m., & latham, d. (2013). addressing below proficient information literacy skills: evaluating the efficacy of an evidence-based educational intervention. library & information science research, 35(3), 181–190. https://doi.org/10.1016/j.lisr.2013.03.001   handler, l., lackey, m., & vaughan, k. (2009). "hidden treasures": librarian offices hours in three health sciences schools. medical reference services quarterly, 29(4), 336-350. https://doi.org/10.1080/02763860903249076   ivanitskaya, l., duford, s., craig, m., & casey, a. m. (2008). how does a pre-assessment of off-campus students’ information literacy affect the effectiveness of library instruction? journal of library administration, 48(3-4), 509–525. https://doi.org/10.1080/01930820802289649   keshavarz, h., esmaeili givi, m., & vafaeian, a. (2016). students’ sense of self-efficacy in searching information from the web: a pls approach. webology, 13(2), 16-31. retrieved from http://eprints.rclis.org/32203/   maddux, j. e., & volkmann, j. (2010). self-efficacy. in r. h. hoyle (ed.), handbook of personality and self-regulation (pp. 315–331). oxford: wiley-blackwell.   mery, y., newby, j., & peng, k. (2012). why one-shot information literacy sessions are not the future of instruction: a case for online credit courses. college & research libraries, 73(4), 366-377. https://doi.org/10.5860/crl-271   muthukumar, s., d’cruz, s. m., & anandarajan, b. (2013). introduction of pre-test and post-test enhances attentiveness to physiology lectures: students’ perceptions in an indian medical college. international journal of biomedical advanced research, 4(5), 341-344. retrieved from https://ssjournals.com/index.php/ijbar/article/view/319   reinsfelder, t. l. (2012). citation analysis as a tool to measure the impact of individual research consultations. college & research libraries, 73(3), 263–277. https://doi.org/10.5860/crl-261   ren, w.-h. (2000). library instruction and college student self-efficacy in electronic information searching. the journal of academic librarianship, 26(5), 323–328. https://doi.org/10.1016/s0099-1333(00)00138-5   rogers, e., & carrier, h. s. (2017). a qualitative investigation of patrons’ experiences with academic library research consultations. reference services review, 45(1), 18–37. https://doi.org/10.1108/rsr-04-2016-0029   serap kurbanoglu, s. (2003). self‐efficacy: a concept closely linked to information literacy and lifelong learning. journal of documentation, 59(6), 635–646. https://doi.org/10.1108/00220410310506295   shaffer, b. a. (2011). graduate student library research skills: is online instruction effective? journal of library & information services in distance learning, 5(1–2), 35–55. https://doi.org/10.1080/1533290x.2011.570546   shivaraju, p. t., manu, g., vinaya, m., & savkar, m. k. (2017). evaluating the effectiveness of preand post-test model of learning in a medical school. national journal of physiology, pharmacy and pharmacology, 7(9), 947-951. https://doi.org/10.5455/njppp.2017.7.0412802052017   schobert, t. (1982). term-paper counseling: individualized bibliographic instruction. rq, 22(2), 146-151. retrieved from https://www.jstor.org/stable/pdf/25826898.pdf   sokoloff, j., & simmons, r. (2015). evaluating citation analysis as a measurement of business librarian consultation impact. journal of business and finance librarianship, 20(3), 159–171. https://doi.org/10.1080/08963568.2015.1046783   university of ottawa. (n.d.). enrolment. in institutional research and planning. retrieved 20 feb. 2018 from https://www.uottawa.ca/institutional-research-planning/resources/facts-figures/fact-book/enrolment   wakimoto, d. k. (2010). information literacy instruction assessment and improvement through evidence based practice: a mixed method study. evidence based library & information practice, 5(1), 82–92. https://doi.org/10.18438/b80616   wolf, k., & stevens, e. (2007). the role of rubrics in advancing and assessing student learning. journal of effective teaching, 7(1), 3–14. retrieved from https://files.eric.ed.gov/fulltext/ej1055646.pdf     appendix a search strategy worksheet   search statement / topic 1. search question or topic:       list as many as you need 2. major concepts: 1. 2. 3. 4. 3.                                                                 search terms:                         concept 1      and      concept 2          and        concept 3         and       concept 4           or         or         or         or         or         or           resources / databases to use:     source: rielding, ann marlow. 2002. learning to learn. new york: neil shuman publisher.     appendix b pre and posttest questionnaires   assessing individualized research consultations – pretest questionnaire  1.      briefly state your research topic or question.    2.      have you identified keywords for your research project?    yes   no 3.      what are your keywords?   4.      have you created a search strategy (search string)? for example: (marine or ocean) and (biology or science).    yes   no 5.      what is the search strategy (search string) you would use for your research project? for example: (marine or ocean) and (biology or science).   6.      have you already located relevant sources of information (articles, books, reports, websites, etc.) for your research project?    none   some sources   many sources 7.      what resources or tools have you used to find these relevant sources of information (e.g. web site, library, database, etc.)?    8.      on a scale 1 to 10, where 1 represents “not confident at all” and 10 represents “very confident”; how confident are you with finding relevant sources of information?   9.      what factors influence your confidence level (e.g. prior knowledge, degree of difficulty, etc.)?   10.  what are you hoping to achieve/get out of your appointment with a librarian?   11.  any additional comments?   12.  what is your gender?   male   female   there isn’t an option that applies to me 13.  to which age group do you belong?   19 years old or less   20 to 25 years old   26 to 30 years old   31 years old and over 14.  are you doing an:   undergraduate degree   graduate degree 15.  in which year are you?   1st year   2nd year   3rd year   4th year   other: ______________________ 16.  if you are doing a graduate degree, is it:   post graduate certificate   master degree   doctoral degree   other: 17.  in what faculty are you registered in?   faculty of health sciences   faculty of medicine 18.  in the last three months, have you had a formal library presentation in class, or have you attended a library workshop?   yes   no 19.  describe in a few words what was covered in the library presentation, or in the library workshop you attended:   20.  in the last three months, have you met individually with a librarian?    yes   no 21.  briefly state what you discussed.       assessing individualized research consultations posttest questionnaire 1.      after meeting with a librarian, has your research topic or question been modified?   not at all   slightly modified   modified completely 2.      briefly state your research topic or question whether it has been modified or not.   3.      if you had provided keywords for your research project before your appointment with a librarian, have they changed now?   yes   no 4.      please provide your keywords whether they have changed or not.   5.      if you had provided a search strategy (search string) in the previous questionnaire, have it changed now?    yes   no   does not apply 6.      please provide your search strategy (search string), whether it has changed or not. for example: (marine or ocean) and (biology or science).   7.      what resources/tools are you using, or will you use to find relevant sources of information for your research project (e.g. web site, library, database, etc.)?   8.      on a scale 1 to 10, where 1 represents “not confident at all” and 10 represents “very confident”; how confident are you with finding relevant sources of information?   9.      what factors influence your confidence level (e.g. prior knowledge, easy or degree of difficulty, etc.)?   10.  has the appointment with a librarian influenced in any way your confidence in your search techniques?   yes   no 11.  please describe how the appointment with a librarian has changed or has not changed your confidence level.   12.  on a scale 1 to 10, where 1 represent “expectations not met at all” and 10 represent “exceeded expectations”; how the appointment with a librarian met your expectations?   13.  please describe how they were or weren’t met.    14.  any additional comments?       this work is licensed under a creative commons attribution-non commercial-sharealike 4.0 international license.   evidence based library and information practice 2012, 7.4 99 evidence based library and information practice evidence summary novice academic librarians provide insight into choosing their careers, graduate school education, and first years on the job a review of: sare, l., bales, s., & neville, b. (2012). new academic librarians and their perceptions of the profession. portal: libraries and the academy, 12(2), 179-203. doi: 10.1353/pla.2012.0017 reviewed by: carol d. howe reference librarian/associate professor gabriele library, immaculata university immaculata, pennsylvania, united states of america email: chowe@immaculata.edu received: 2 aug. 2012 accepted: 26 oct. 2012 2012 howe. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to study the ways in which novice academic librarians’ perceptions of librarianship develop from the time they decide to attend library school through their first 6 to 24 months of library work. design – grounded theory method utilizing two qualitative research techniques: one-onone, face-to-face interviews and document analysis. setting – the libraries of three texas universities, three texas four-year colleges, and one texas community college. subjects – 12 professional academic librarians who graduated from eight different graduate schools. participants were 6 to 24 months into their professional careers and had little or no pre-professional experience. methods – the researchers sought participants through mailings, emails, electronic mailing list postings, and referrals from other participants. they conducted a small pilot study with two novice librarians to refine their research methodology. the researchers interviewed additional participants and analyzed the interview transcripts until categories of interest were identified and saturated. saturation occurred at 12 participants, not including the pilot participants. each interview was 30-45 minutes. the researchers recorded the interviews and systematically coded the transcripts using activist imagery. four of the mailto:chowe@immaculata.edu http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/ evidence based library and information practice 2012, 7.4 100 participants gave the researchers their “statement of purpose” essay that they used when applying for graduate school. these documents were also discussed with participants and analyzed. main results – from the data they collected, the researchers identified six categories of interest regarding librarians’ perceptions of librarianship: deciding upon a career, experiencing graduate school, continuing education, defining the work, evaluating the work, and (re)imagining the future. in considering librarianship as a career, the participants had not been entirely sure what it entailed, but they utilized what they did know about libraries and librarianship to generally deem the profession solid, safe, and/or noble. they had further explored librarianship to determine its compatibility with their personal characteristics. such personal reflection had led participants to graduate school where they gained a real understanding of librarianship. the participants had not generally found graduate school to be academically challenging. they had also valued practical over theoretical instruction. once in the workplace, the participants noted the value of continuing education to strengthen the skills they had learned in graduate school. participants benefitted the most from informal mentoring and on-the-job training, i.e. “learning by doing” (p. 192). as novice librarians, the participants had learned to feel their way around their job expectations and note the differences between their responsibilities and those of paraprofessionals in the library. as the novice librarians further defined their work, they had also learned that academic librarianship is the sum of many parts, including collaboration with peers. in evaluating their work, the participants noted that they had come to distinguish “real” academic library work, that which uses their expertise and helps society, from “other” work such as clerical work (pp. 195-196). the sixth and final category was “(re)imagining the future.” most of the participants predicted having advanced as academic librarians in the next five years but were otherwise unsure about what their futures would hold. conclusion – the researchers made a number of valuable observations in their work with novice librarians. as the step of deciding upon a career seemed to be a murky quest, they thought it would be helpful to analyze public opinion of librarianship and use that information to offset misperceptions about what librarians do. this might help those considering librarianship to make informed and conscious decisions. the study data also provided insight into graduate school. the fact that the participants did not consider graduate school to be rigorous concerned the researchers. they feared that librarians entering the field might not deem it a serious profession. because the participants favored practical over theoretical classes, the researchers thought it important for graduate schools to teach theoretical concepts in a way that is more satisfying to students. they felt that other applied fields, such as nursing, might provide examples of how to do so. the researchers also noted that graduate schools could do more to prepare students for life on the job. as new librarians reported favouring “real” work over “other” work, the researchers felt that students should hear it first in graduate school that all the work librarians do is an important and necessary part of academic librarianship. as most participants were uncertain about what their futures as academic librarians might look like, the researchers thought that graduate school professors should address that issue as well. data from this study also gave insight into how employers might best serve new librarians. the researchers suggest looking to new teacher induction programs to get ideas for orienting new librarians to the profession. orientation might include a combination of formal and informal techniques such as peer mentors, peer observation, new librarian training, and new librarian handbooks in the first year of employment. finally, the researchers proposed ideas for future research. they believe it might be helpful to study experienced academic librarians or new public librarians for comparison to this study. evidence based library and information practice 2012, 7.4 101 commentary other studies have examined students’ perceptions of librarianship, new librarians’ perceptions of specific aspects of librarianship, and experienced librarians’ perceptions of the profession. this study is the first to examine new librarians’ perceptions of academic librarianship as a continuum: the evolution of a budding librarian into a professional. the grounded theory method proved an effective way to bring the most meaningful data to the surface. the six distinct categories that emerged give a broad sense of the path that a new librarian follows. the study has some significant weaknesses, however, that cast doubt onto the validity of the results. most obvious is the small number of participants combined with the fact that all participants were from texas libraries. in addition, the researchers did not include much information about their interview strategy and whether specific questions were consistently asked. this might have explained to some extent the small number of participants, or in other words, why the researchers felt that saturation occurred at 12 participants. also regarding the interviews, the researchers indicated that they cross-coded 20% (translating to about two and a half) of the interview transcripts. the small number of cross-coded transcripts leads one to question the coding consistency. finally, the terminology describing the pilot portion of the study was confusing in that the two pilot participants were referred to as participants one and two, and the 12 actual study participants were referred to as participants 3 through 14. because of its small size, one cannot use the data from this study to reliably generalize to the larger population of novice academic librarians. it does, however, provide an interesting initial exploration of their perceptions of the profession. many readers will relate to the participants’ imprecise method of choosing librarianship as a career, the rewards and frustrations of graduate school, and the process of slowly but surely creating one’s identity as a professional librarian. the strengths of this study are twofold. first, it provides ample avenues for future research. this might include similar studies with more, randomly selected participants from different geographic locations. as stated by the researchers, it would also be interesting to compare this data to that from experienced librarians or public librarians. second, the researchers fleshed out several important themes: the influence that public perception of librarianship has on those considering entry into the field, the role of graduate school in preparing future librarians, and ideas for effective post-graduate training. this data, combined with data from potential future research of this sort, might be noteworthy to several populations. those considering librarianship would find an insider’s perspective from potential peers invaluable. graduate schools looking to increase their impact as well as employers looking to effectively support their new librarians would also be wise to examine the results of such studies. research in practice   research in practice: mythbusting eblip   virginia wilson director, centre for evidence based library and information practice (c-eblip) university library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   received: 2 may 2017    accepted: 15 may 2017      2017 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   over the years, i’ve heard lots of reasons that library folks might shy away from evidence based library and information practice (eblip). for some, it’s even a hot-button issue, usually focusing on the use of the term “evidence”. i often wonder if the thought is that eblip precludes doing anything else to facilitate decision-making. i’ve always seen it as an important part of my professional toolkit and don’t look at it along partisan lines. so when asked to present a conference session about eblip to the 2015 saskatchewan library association conference, i included a section about eblip myths – some of the reasons that i’ve heard for avoiding eblip in professional practice.  everyone’s experience is different but perhaps some of these will resonate.   myth 1: eblip is so restrictive. it only advocates using research evidence.   eblip does encourage the use of valid and reliable research results. these can be from the published literature or can be results found through doing your own research on a particular topic related to your practice. so, the myth here is that it only advocates using research evidence. i can see where this comes from. the idea of “evidence” points to formal research results, and often to quantitative results. however, recent research, such as that by koufogiannakis (2012), suggests that librarians use a wide variety of what they term “evidence”.  some of this comes from the published research and some of it comes from elsewhere. it’s all about expanding the definition of evidence in the eblip context to include our own professional knowledge and expertise and what our users or patrons prefer or need, as well as the formal research results. i’ve written more about this particular myth in a past research in practice column (wilson, 2015).   myth 2: eblip doesn’t take into account all my years of experience.   basically, this myth was dealt with in myth 1, but further to that, eblip does indeed consider knowledge and expertise gained over the course of a career. it would be silly if it didn’t! the idea is not to discount a librarian’s knowledge but to enhance it by ensuring that library users or stakeholders and the published research are consulted. we often hear the phrase “let’s not reinvent the wheel”. eblip, and its three-pronged approach, is all about not reinventing the wheel. it’s about being thorough and inclusive in order to make the best possible professional decision.    myth 3: evidence based practice is only used in academic libraries and in medicine.   while it is true that evidence based medicine was the model adopted by health sciences librarians in the early 2000s as they got the ball rolling for eblip, it is not true that it just stayed that way. not only can eblip be used in every library sector (public, school, special, and academic), evidence based practice has expanded across many disciplines including, to name just a few:   •       evidence based crime prevention •       evidence based policy development •       evidence based software engineering •       evidence based scheduling •       evidence based social work •       evidence based nursing •       evidence based management •       evidence based dentistry •       evidence based policing •       evidence based business •       evidence based conservation   the point of this, from a library perspective, is that while different libraries or library sectors may have different approaches to decision-making based on organizational structures, budgetary considerations, and the like, eblip can be used as one way to ensure that all sides of the problem are considered. the evidence based library and information practice journal published a special issue on public libraries in 2012. in the editorial introducing the special feature, ryan (2012) stated that “eblip is one area where librarians from every sector can work together, sharing a common interest in evidence based professional practice. eblip at its best puts aside sector silos and offers a broad perspective for our work in all library types” (p. 5).   myth 4: eblip requires too much specialized knowledge.   i do acknowledge that the term “too much” is relative. the specialized knowledge this myth refers to is research knowledge, and the ability to critically appraise the research evidence you find. there are checklists and rubrics available to help go through a research article with a critical eye. as well, a lot of critical appraisal is common sense. as librarians and library workers, we are trained to assess information sources, to determine if the information is reliable and credible. it’s much the same with the published research, which has the added benefit of being peer reviewed. while peer review does not guarantee quality, it does suggest that a second or third pair of eyes have looked at the research in a critical fashion. also, i would suggest that engaging with research in librarianship is quite like reading the newspaper or watching television news, in that we hopefully are critical of it: not critical in a negative sense, but critical in the sense of not accepting everything at face value and being aware of where the information comes from and who might benefit from seeing it presented a particular way.   myth 5: i don’t have the time so it isn’t for me.   it is true that one of the major barriers to practicing in an evidence based way is a lack of time. however, there are a couple of things to keep in mind. like anything, when a process or a workflow is first starting, it does take time for it to be smooth and to eventually save time. eblip is like that. there will be a learning curve, but as the world of libraries continues to grow and change, we all have the potential to face many learning curves in our daily work. once eblip as a process is internalized, it will feel like it is something you just do, that it is just the way you work. and because you are using this process to make decisions, you will hopefully save time overall by making the best possible decision using the best available evidence. also, even incorporating bits and pieces of eblip into your practice can reap benefits. so i would suggest that you do some personal reflection or talk to colleagues to determine if you really don’t have the time, or is something else going on.   as more and more librarians are conducting and disseminating research, the base of good, practice-based research is growing larger. it’s there to help in decision-making and problem-solving in practice. our own professional experience and knowledge is the basis from which we approach our work. our experience is valid, has value, and comes from many places: formal education, past disciplinary studies, and our own critical perspectives of the world around us. and our users/patrons/clients have needs and desires as they pertain to their library and information experiences. be sure to ask them. if you can accept these three things, you are an evidence-based practitioner.   references   koufogiannakis, d. (2012). academic librarians’ conception and use of evidence sources in practice. evidence based library and information practice, 7(4), 5-24. http://dx.doi.org/10.18438/b8jc8j   ryan, p. (2012). eblip and public libraries. evidence based library and information practice, 7(1), 5-6. http://dx.doi.org/10.18438/b89k72   wilson, v. (2015). evidence, local context, and hierarchy. evidence based library and information practice, 10(3), 268-269. http://dx.doi.org/10.18438/b8k595   evidence summary   flexible work agreements: here to stay but uneven in equity and promoting success   a review of: hosoi, m., reiter, l., & zabel, d. (2021). reshaping perspectives on flexible work: the impact of covid-19 on academic library management. portal: libraries and the academy 21(4), 695-713. https://doi.org/10.1353/pla.2021.0038     reviewed by: samantha j. kaplan research & education librarian, liaison to the school of medicine duke university medical center library & archives durham, north carolina, united states of america email: samantha.kaplan@duke.edu   received: 5 july 2022                                                                     accepted: 18 oct. 2022      2022 kaplan. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30200     abstract   objective – the article seeks to assess the current state and the future of flexible work agreements (fwas) in research libraries.   design – the authors held semi-structured interviews with 31 individuals in library leadership roles. setting – large american or canadian research libraries during the covid-19 pandemic. subjects – 31 individuals in senior leadership roles (ex: associate dean, director) at the top 50 research libraries in north america (based on the association of research libraries investment index). methods – interviews were conducted and recorded over zoom with participant, investigator, and note taker. investigators developed a quantitative coding instrument based on a selection of the interviews, then coded all interviews independently. coded data were evaluated for broader themes in a collaborative fashion. main results – all participants had employees working partially or fully remotely at the time of the interviews. half of participants observed gains in productivity during the pandemic, although even more commented on technology challenges. other positives included remote project success and more inclusive meetings; other negatives included caregiving and job duties that did not allow for remote work. conclusion – while fwas were widely available pre-pandemic, they were not normative. the majority of participants think flexible work will only increase in libraries and will influence recruitment and retention of employees, as well as utilization of library space.   commentary   the covid-19 pandemic thrust many industries and workers into remote and flexible work overnight, including large academic and research libraries. several years into this pandemic, remote and flexible work options remain while generating larger questions about the future of library work and library spaces.   utilizing lett's et al. (2007) critical review form for qualitative studies clarifies certain strengths and weaknesses of the article which reports the findings from interviewing senior administration and leadership at the largest research libraries in north america. while the purpose of the research is clearly stated with a review of relevant literature, the study design is not clearly named by the authors and no theoretical perspective is identified. without a named theoretical influence, it is impossible to discern if the methods are congruent with the philosophical underpinnings. however, the article does excel at describing the sampling process, but it would benefit from details about the consent process, as many of the participants hold positions of influence in powerful libraries. there is also minimal information about the participants; faculty status, pronouns, and generation are the only demographic characteristics provided. knowing how long these individuals had occupied their roles or their area's covid-19 restrictions would have made their responses more meaningful. the authors employed a quantitative coding instrument, but it is difficult to judge the audit trail without any report of agreement rate among raters or information about sources of disagreement. the description of data analysis methods is much improved by providing the coding instrument.   even though the authors used an interview instrument that was a mix of structured and semi-structured questions, their use of the term “validity” to describe their findings, and their description of their coding process suggests a quantitative perspective. a survey may have been a more appropriate methodology choice to achieve their desired goal.   the findings have significant implications for academic libraries of all sizes in north america, though by the authors' own acknowledgment, data were gathered early in the pandemic (august and september of 2020). since that time, many library workers have returned to work part or full-time, especially given the availability of effective vaccines and high-quality masks. the economic atmosphere, as well as the labor market, have also undergone significant shifts that have influenced flexible work's role in the workplace, retention, and recruitment. it is of vital importance to remember that this research was conducted prior to vaccine mandates and that depending on one's place of residence, there were varying political, economic, and campus pressures to return to the workplace.   the authors' focus on senior library leadership allows for a more aggregate picture of flexible work in libraries – particularly the inherent complexity if not impossibility of implementing fwas with equity as some roles require on-site presence. the participants also painted a greyer picture of flexible work, particularly in technology and caregiving. while libraries have spent decades discussing the digital divide's implications for patrons, this research and the issue of flexible work makes an important contribution that it can be a workplace issue, with not all employees prepared to work at home. while participants did report productivity gains by their staffs, especially with remote projects, the authors could have interrogated this further. shifting many library services and almost all library workers to remote work necessitated creating new workflows and policies. at the same time, library workers experienced increased demand from patrons. people may have been more productive because there was significantly more work to do, with flexible work as a confounder. this article captures flexible work in academic libraries at a particular moment in time, but their future and how they will change library workflows, workforces, and spaces is still in process. these findings should be examined by library administrators implementing flexible work agreements as a guide to some of the potential issues their staffs may experience.   references   hosoi, m., reiter, l., & zabel, d. (2021). reshaping perspectives on flexible work: the impact of covid-19 on academic library management. portal: libraries and the academy, 21(4), 695–713. https://doi.org/10.1353/pla.2021.0038   letts, l., wilkins, s., law, m., stewart, d., bosch, j., & westmorland, m. (2007) critical review form – qualitative studies (version 2.0). retrieved from http://www.peelregion.ca/health/library/eidmtools/qualreview_version2_0.pdf       ebl 101   research methods: altmetrics   virginia wilson librarian, murray library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   originally published in: evidence based library and information practice, 8(1), 126–128. https://ejournals.library.ualberta.ca/index.php/eblip/article/view/18900/14823     received: 03 feb. 2013   accepted: 08 feb. 2013      2016 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   it’s no secret that scholarly communication is changing. the internet, the open access movement, the proliferation of institutional repositories, and the use of social networking tools, as well as the questioning of peer review and impact factors, to name just a few things, have altered the scholarly publication landscape. massive amounts of research content, both full content and citations are available on the internet: the traditional research paper, blogs, academic repositories, online citation managers, and even tweets and facebook posts. if the accepted ways of publishing are changing, then it makes sense that different ways of measurement should be explored in order to get the complete picture of the impact of the work. this is where altmetrics comes in and it’s exciting stuff!   the altmetrics manifesto states that scholars are increasingly moving their work to the internet. “these new forms [citation managers, blogs, other social sharing sites] reflect and transmit scholarly impact: that dog-eared (but uncited) article that used to live on a shelf now lives in mendeley, citeulike, or zotero–where we can see and count it. that hallway conversation about a recent finding has moved to blogs and social networks – now, we can listen in. the local genomics dataset has moved to an online repository – now, we can track it. this diverse group of activities forms a composite trace of impact far richer than any available before. we call the elements of this trace altmetrics.” (http://altmetrics.org/manifesto/)   based on these alternative metrics, altmetrics is “the creation and study of new metrics based on the social web for analyzing, and informing scholarship” (altmetrics.org). what paths do our reactions to a particular article take in the social web? plos refers to this new landscape as the “scholarly ecosystem” (http://article-level-metrics.plos.org/alt-metrics/). altmetrics isn’t just about traditional citation-based indicators. nor is it just about hits; these can be inflated by robot-crawlers and other zealous clickers. so work has been done on finessing use stats and hit count work in order to get a more meaningful measure of online usage in all its variety. as an emerging field of scholarship and development, altmetrics is moving ahead in leaps and bounds. it is becoming common to see journal publishers and repositories implementing tools that will let authors see the impact of their scholarly publications.   and of course there is a role for librarians and information professionals to play in this new field of measurement. while new, the altmetrics field is exploding with new ideas and products. a librarian’s expertise in scholarly communication and in finding the resources and data needed for researching scholars can be invaluable to the institution. there is much to learn in this emerging field. i’ve included a list of resources that can take you further into the realm of altmetrics, and as always, i appreciate comments on this column. you can log in as a reader to the eblip journal and interact from there.   the altmetrics manifesto   priem, j., taraborelli, d., groth, p. & neylon, c. (2010), altmetrics: a manifesto, (v.1.0), 26 october 2010. http://altmetrics.org/manifesto (the comments at the end of the manifesto are an interesting conversation as the concept of altmetrics develops.)   articles   taraborelli, d. 2008. soft peer review: social software and distributed scientific evaluation. retrieved from http://eprints.ucl.ac.uk/8279/   neylon c. & wu s (2009) article-level metrics and the evolution of scientific impact. plos biol 7(11): e1000242. doi:10.1371/journal.pbio.1000242 retrieved from http://www.plosbiology.org/article/info:doi/10.1371/journal.pbio.1000242 priem, j. & hemminger, b.m. (2010). scientometrics 2.0: toward new metrics of scholarly impact on the social web. first monday, 15(7) http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/viewarticle/2874/2570 roemer, r.c. & borchardt, r. (2012). from bibliometrics to altmetrics: a changing scholarly landscape. college & research libraries news vol. 73 no. 10 596-600 retrieved from http://crln.acrl.org/content/73/10/596.full   altmetrics scholarship   plos altmeterics collections http://www.ploscollections.org/article/browseissue.action?issue=info:doi/10.1371/issue.pcol.v02.i19 the altmetrics group on mendeley http://www.mendeley.com/groups/586171/altmetrics/ here you can find more scholarly articles on altmetrics added by mendeley users.   products[1]:   altmetric it http://www.altmetric.com/ they offer open data for individuals, including a free bookmarklet to be used on recent scholarly articles to see how much attention they have received online. there is also an api, free for non-commercial use, used to mash up altmetrics data with other data. the following link is for an interesting blog post from altmetric.org on the free services and apps they’ve developed for libraries and institutional repositories: http://altmetric.com/blog/altmetrics-in-academic-libraries-and-institutional-repositories/ impactstory (formerly total-impact) http://impactstory.org/ an altmetric aggregator. in terms of pricing, this information is from the website: we charge to collect metrics; the data is free and open once it’s been collected. you let us know to start collecting metrics on something when you register it with us. the first 1000 items you register are free. registering more than 1000 items will have an annual fee to provide sustainability for our nonprofit service (waivers available in some cases); the fee will depend on how many items you'd like to register. plum analytics http://www.plumanalytics.com/index.html this company collects impact metrics in five major categories: usage, captures, mentions, social media, and citations. one of the two founders of this service is a librarian. they gather metrics about what they refer to as “artifacts” and these include: articles, book chapters, books, clinical trials, datasets, figures, grants, patents, presentations, source code, and videos. you can see the metrics they include and where they find them here: http://www.plumanalytics.com/metrics.html altmetric for scopus http://altmetric.uservoice.com/knowledgebase/articles/83246-altmetric-for-scopus denise koufogiannakis explains on her blog that “the altmetric service will capture information from social media sites such as twitter and facebook, mainstream media, and reference managers such as mendeley,to illustrate how scholarly articles are being used beyond academia” (2012). plos article-level-metrics (alms) http://article-level-metrics.plos.org/ metrics are tracked for every article published by plos and the full alm data set, which is updated monthly as a .csv file, is always freely available online for all plos-published articles. sciencecard http://sciencecard.org/ sciencecard is a web-service that collects article-level metrics using the content from twitter, mendeley, pubmed central, citeulike, wikipedia and crossref. other services continue to be added.   references   altmetrics.org. (2010). altmetrics. retrieved from http://altmetrics.org/about/   koufogiannakis, d. (2012). altmetrics for scopus. collections information. retrieved from http://collectionsinfo.blogspot.ca/2012/09/altmetric-for-scopus.html?spref=tw   [1] the author is not affiliated with and does not endorse any organization represented here. they are included for information purposes only and are to be implemented at the user’s own discretion.   microsoft word es_walker.doc evidence based library and information practice 2007, 2:3  97 evidence based library and information practice       evidence summary    low volume, funding, staffing and technical problems are key reasons for  discontinuation of chat reference services    a review of:  radford, marie l., and m. kathleen kern. ʺa multiple‐case study investigation of the  discontinuation of nine chat reference services.ʺ library & information science  research 28.4 (sept. 2006): 521‐47.    reviewed by:   stephanie r. walker  associate librarian for information services, brooklyn college – city university of new york   brooklyn, new york, united states of america  e‐mail: swalker@brooklyn.cuny.edu         received: 11 june 2007     accepted: 24 july 2007      © 2007 walker. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to investigate the reasons  behind the cessation of 9 virtual (chat)  reference services.      design – multiple case‐study investigation    setting – academic, public, and special  libraries in the united states.    subjects – five academic libraries, 1 public  library, and 3 library consortia.    methods – the initial method used to  identify discontinued chat reference services  was via a posting to dig_ref, the digital  reference services listserv. from this initial  posting, librarians on the list also suggested  other cases, for a total of 7. two cases later  came to the authors’ attention and were  added. the 9 cases included academic  libraries, a public library, and consortia who  used tools provided by four major software  vendors, giving a good cross‐section of the  types of libraries participating in vr and the  major software packages available at the  time. a contact person for each discontinued  service was invited to participate. the  participants were geographically scattered,  so data were collected via e‐mail  questionnaires, telephone interviews, and  examination of available documents  describing the services, audiences,  demographics, operations, and any  decision‐making processes either for  starting or discontinuing the service. the  authors then compared and contrasted  results obtained from the 9 services. the  authors analysed reasons for  mailto:swalker@brooklyn.cuny.edu http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:3  98 discontinuation of services, the decision  making processes, volume of questions,  service hours, marketing, evaluation, and  whether the service might be restarted at  some time in the future.      main results – although the vr services  discussed were being delivered by diverse  organizations, some commonalities emerged.  all 9 services used existing staff and added  chat reference to their duties. in all but one  case, the hours of service offered when vr  began were within the hours that the  libraries were already open to the public. in  the exception (case e), there were hours of  service after the libraries were closed, from  7‐10 p.m. these shifts were covered by  librarians working from home.     decisions to either begin or suspend  services were generally made by small  informal groups or committees. some cases  began with pilot studies and received  positive responses. other than these pilot  studies, little pre‐planning appeared to have  been done. for example, there were no  extensive advance surveys or studies to  determine potential demand or markets for  the service, or projected costs and benefits.  only one service had a strong, multi‐stage  evaluation plan. in future launches of  virtual reference services, both extensive  pre‐planning and detailed measures for  evaluation could be helpful in avoiding  some problems.      major reasons for discontinuation fell into 6  categories: funding problems, staffing  problems, technical problems, institutional  culture conflicts, low usage overall or low  usage by target populations. a table (table 1)  was included that actually listed 7 categories  of reasons for discontinuation, but the last  one listed, “software change,” is discussed  as part of “technical problems” in the text,  though it is separated in the table (527). this  is not immediately clear in the article, and  thus the table is slightly confusing. four  cases reported funding problems as the  major reason for discontinuation. low  volume or low volume for target population  were primary reasons for discontinuation by  4 cases, and secondary reasons for  discontinuation by 4 cases. if one combines  primary and secondary categories in this  table, low volume is the most frequently  cited reason for discontinuation. low  volume was determined to be “driven by a  complex combination of variables including  marketing strategies, insufficient hours of  operation, and [failure to provide] an ample  amount of time for a service to gain  momentum” (527‐528).      funding was cited as the primary reason for  cancellation in 4 cases. technical problems  were listed as primary or secondary factors  for suspension of service in 2 cases, and  software change was the secondary factor  behind the suspension of service in 2 cases;  one service reported that up to one third of  chat sessions were ‘lost” (527).  in many  cases, these sites were early adopters of vr,  or even beta test sites, so technical issues are  perhaps not surprising.      staffing was the primary or secondary  reason for discontinuation in a total of 3  cases.  staffing problems included  insufficient questions to keep staff interested;  however, concerns about extending service  hours and thereby putting additional  pressure on existing staff were also raised.  clashes in institutional cultures were  reported as secondary reasons for  discontinuation of the service in 2 consortia.  it should be noted that the sample size is  small; as the authors point out, the literature  on unsuccessful or discontinued virtual  reference services is quite limited.      pop‐up surveys were used by some cases to  evaluate service success and user  satisfaction. user satisfaction was reported  as being high, but return rates for the  surveys were very low, as can be expected  evidence based library and information practice 2007, 2:3  99 since pop‐ups are often blocked or ignored.  the authors recommend alternative means  of evaluation, such as focus groups and  more detailed surveys. the authors also  discuss alternative technologies and user  expectations for services such as im or sms  text messaging, and suggest evaluation of  user expectations and desires in these areas.      one question which arose was what  constituted success in chat reference – was it  usage/volume, or some other measure, such  as whether chat reference queries were  answered correctly more frequently than  reference questions asked in other ways,  such as by telephone or in person. other key  concerns identified include the importance  of strong marketing, service usability, and  the importance in a consortial environment  of having someone responsible on a full‐ time basis for directing and spearheading  the project. many of the suspended services  reported that they could potentially become  re‐interested in vr service, as issues are  resolved and the technologies available  mature.     finally, the authors provide detailed  recommendations for single and consortial  vr services, and offer directions for future  research. some suggestions for future  research include ways to bridge the  apparent gulf between expectations and  technological preferences of users versus  those of librarians and others staffing these  services, as they often appear to have  marked differences. other areas of interest  could include causes for low question  volume, and possible relationships among  various factors contributing to such poor  usage along with an investigation of what  makes successful services work well.  research on im or sms reference services as  compared to those using other chat  technologies and cost‐effectiveness studies  or evaluations based on goals and measured  outcomes are also needed.    conclusion – vr services have had erratic  success rates and bumpy beginnings, but  many libraries report continued interest the  libraries in this study all indicated that they  would consider, or are considering, re‐ starting such services. issues of staffing,  technological usability and reliability,  institutional culture, funding, marketing,  and volume are key. as services and  technologies mature, interest in offering vr  services in one form or another continues to  grow.       commentary     many of the libraries which chose to  discontinue their vr service were early  adopters of the service and the technology,  and thus seem to have faced difficulties  which have been at least partially addressed  by more successful currently operational  services. much has changed since the late  1990s and early 2000s in vr service  environments and offerings. to list just a  few examples: some major technological  issues have been addressed; more users  have access to high‐speed internet  connections and are familiar with  technologies such as im and sms; some  consortia have grown and strengthened; and  some services have well‐established  marketing practices (or other practices)  which can be shared. any of these factors  might have influenced overall success for  these 9 services. for example, technological  advances would have lessened frustration,  expansion of consortia has allowed some  libraries to collaborate on staffing and hours  of service, and improved marketing could  lead to improved usage. not all  environments will be amenable to vr  services, of course, but it is wise to examine  and address issues such as staffing,  institutional culture, technological feasibility,  funding, potential audiences for the service,  and marketing plans before even beginning  a new service launch. pre‐planning for all  aspects of the service is important, as is a  evidence based library and information practice 2007, 2:3  100 clear articulation of measures for evaluation  and criteria for success.     this article is extremely useful as  background for any library or consortium  exploring the possibility of launching vr  services. it could usefully be combined with  the reading of other studies of chat reference  services which are “successful,” and with  reading of studies of virtual reference  services which report relatively low volume,  but which have continued to operate for  some years. an analysis of what is different  in “successful” or particularly busy chat  services, or of reasons why some low  volume services persist despite usage which  accounts for only a relatively small  percentage of total reference queries could  be done. it would be interesting to  determine whether other criteria besides  usage statistics are considered particularly  important in those cases.     virtual reference services have been around  for some time now. many have existed since  at least the late 1990s, yet much remains  unknown. in the larger scheme of things, 10  years or so is still a relatively short period of  time in which to assess the success of an  entire service modality, particularly in a  rapidly altering and technology‐dependent  environment. virtual reference services will  continue to change and adapt, and more  study of factors which can contribute to  success in launching and operating such  services is vital.       research article   the relationship between student demographics and student engagement with online library instruction modules   mary c. thill associate professor ronald williams library northeastern illinois university chicago, illinois, united states of america email: m-thill@neiu.edu   james w. rosenzweig assistant professor john f. kennedy library eastern washington university cheney, washington, united states of america email: jrosenzweig@ewu.edu   lisa c. wallis associate professor ronald williams library northeastern illinois university chicago, illinois, united states of america email: l-wallis@neiu.edu   received: 28 mar. 2016   accepted: 11 july 2016         2016 thill, rosenzweig, and wallis. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to investigate whether there are any demographic trends affecting student engagement with online library instruction which might have implications for practice, the authors designed a case study to examine the relationship between student demographic characteristics and engagement with online library instruction modules in english 102 courses at a single university.   methods – the authors recruited 181 students from english 102 (eng 102), a research-based composition course, to participate in the study. eng 102 instructors asked all participants to complete an online library instruction module embedded in the university’s course management system, either before in-person library instruction or in lieu of face-to-face library instruction. no external incentive was provided for online module completion. the research team measured levels of student engagement by recording the amount of time students spent on each page of the online module. in collaboration with the office of institutional research, the authors then pulled demographic data on each participant using the university’s student information system. pearson chi-square tests were performed to determine whether there were any notable associations between levels of student engagement and student age, grade point average, gender, and race/ethnicity.   results – observable trends tied age and higher grade point average to higher levels of engagement with online instruction. there was additionally a slight trend linking female participants to higher levels of engagement than their male peers. in the category of race/ethnicity, the two largest subgroups, hispanic and caucasian students, exhibited similar levels of engagement.   conclusions – the authors conclude that there may be demographic implications for practice in designing online library instruction programs, especially when considering student age and academic performance indicators. they also conclude that, owing to this case study’s limited sample size, further study is warranted to investigate these conclusions, and to further examine the possible impact of gender and race/ethnicity on engagement with online library instruction modules.   introduction   today’s librarians are educators, advocating for the integration of information literacy into the college curriculum through embedded librarianship, collaborations on curriculum design, and co-teaching opportunities. however, in reality many librarians must continue to deliver traditional one-shot sessions and make the most of limited opportunities to engage directly with students in the classroom. this is driven largely by their partners in the teaching faculty who, contending with the demands of their content-packed syllabi, often feel that they must limit the amount of classroom time devoted to library instruction. in this environment, faculty and librarians alike are considering hybrid and fully online models as attractive methods to enrich or even replace face-to-face information literacy instruction.   the research team worked at northeastern illinois university, a four-year master’s level public hispanic-serving institution (hsi) in the midwestern united states. hsis, by definition, have a student body comprised of at least 25% hispanic students at the undergraduate or graduate level, or both (hispanic association of colleges & universities, n.d.). the team became interested in this research after their library received a significant internal grant to develop a set of professional-quality animated videos for information literacy. this raised the question of how best to incorporate videos into the existing instructional program. the authors of this paper were accepted into the assessment in action program of the association of college and research libraries, which provided peer-to-peer support and professional development to design a study that examined the potential effects of the new videos on student engagement, success, and completion.   the natural partners for this investigation were instructors of english 102: composition ii (eng 102), a research-based writing course delivered to undergraduates. the department of english requires all sections of eng 102 to visit the library at least once to receive a standardized lesson in searching library databases. although many composition instructors prefer traditional face-to-face library instruction, a set of instructors expressed interest in migrating to a mixed or even fully online lesson. for that reason, the research team developed three pedagogical approaches to study the effect of the videos: a hybrid “flipped classroom” model, a fully online module, and a traditional face-to-face session with no video enhancements.   the study examined the different attitudinal and cognitive effects of the three modes of instruction, as well as investigated what factors, if any, correlated with success in each format. the specific element of the broader study that will be addressed in this article is the role that demographic factors played in persistent engagement with online library instruction modules. some students completed all assigned online modules, but most engaged only in part, or not at all. the research team sought to identify those students who were most engaged with online library instruction, and whether any demographic factor differentiated them from their less engaged peers.   literature review   little scholarly research has been published on the populations served by online library instructional modules, despite the fact that academic libraries increasingly rely on these modules to address the trend in higher education towards offering more classes and degree programs in online formats. what literature exists provides an incomplete and even contradictory picture of student engagement with online library materials. while koohang (2004) found that male students were more likely to self-report high levels of confidence and satisfaction engaging with the online library, a very early study by alexander and smith (2001) showed that more women than men expressed a preference for online library instruction. jowitt’s study in new zealand (2008) indicated that the population of students self-selecting to use library-produced podcasts to learn library skills was largely female.   the landscape of the academic library has changed fairly dramatically since the early 2000s, however, as libraries have developed new models for resource and service delivery that increasingly extend into digital spaces. one recent study that addressed student engagement with online materials and instruction in the academic library was soria, fransen, and nackerud’s research with first-year students at the university of minnesota-twin cities, in which they found that student engagement with the library correlated with higher grade point average (gpa) (2013). their measure of student engagement, however, was very broad, extending to checking out books, using library databases, and other less intensive means of engaging with the library, in addition to receiving library instruction. therefore, the applicability of their findings to this study’s more narrow focus on student engagement with online library instruction is unclear. bowen’s recent study of libguides and websites as delivery vehicles for information literacy instruction did collect some self-reported demographic data from participants. however, as the collection of demographic data was incidental to the study’s purposes and from a limited sample of participants, bowen did not offer analysis of demographic performance or draw conclusions about engagement levels by subgroup (2014, p. 156). with few clear, recent studies addressing the demographics of library patrons using online library modules and how those demographic groups engage with those modules, it is difficult to make any evidence-based assertions about these subjects.   for that reason, the authors conducted a review of literature on the more extensive research that has been done regarding online credit-bearing college courses. the authors were especially interested in any research that indicates whether different demographic groups are likely to be more or less engaged, persistent, and ultimately successful in the online environment. the published research on these questions has focused particularly on the demographic categories of age and gender.   older students have generally been identified as one of the target demographic groups for online classes and degree programs. early research suggested that older, non-traditional students were more likely to enroll in online courses (dutton, dutton & perry, 2002; halsne & gatta, 2002). jaggars and xu (2010), however, found that this was true only of first-year students in their sample, and that over the course of their entire college career, students over 25 were no more likely to enroll in online courses than students under 25. the early research by dutton et al. suggested that this tendency of older students to enroll online was driven largely by considerations of convenience, especially balancing their studies with work and family obligations (2002, p. 11). callaway and alflayyeh (2011) investigated these claims, and found that, while age did not seem to correlate with convenience factors, older students tended to enroll in online coursework based on perceptions of quality factors. a recent study by kunhimohamed (2012), however, indicated that older age in students correlated slightly with both number of messages posted in their online coursework and the total time they spent accessing the course materials. this finding suggests that higher voluntary engagement might be expected of older students in similar online modules designed by the library, but delivered via the same content management system (cms) used for online coursework.   regarding gender as a demographic category, the literature is consistent in identifying women as more likely to enroll in online coursework and more engaged with the content of that coursework once enrolled. research has consistently found that online students were more likely to be female than male (halsne & gatta, 2002; jaggars & xu, 2010). early research by marley (2007) indicated that women were significantly less likely than men to fail in their online coursework, and marley suggested that this academic success might stem from a greater dedication of time and effort on the part of female students. subsequent research has borne this out: kunhimohamed (2012) found that female students were slightly more likely than males to post messages and engage with course content. yoo and huang (2013) found that female students reported a greater sense of intrinsic motivation in their online coursework than men did. shen, cho, tsai, and marra (2013) found that gender was a significant predictor of self-efficacy in both handling tools in a cms and online course completion, with women having the advantage in both cases. yukselturk (2010) examined gender in a male-dominated turkish engineering program and found that the female students were more likely to be “active” in their course engagement than male students. on the whole, the literature sends a very consistent message that women are more likely to have the skills to engage in online coursework, the motivation to do so, and the commitment to engage in ways that improve their chances of success. as a result, the research team’s expectation was high that female students would be more engaged in voluntary online library coursework, especially when delivered via the institution’s cms, than their male counterparts.   given the existing body of library literature, the authors elected to design a study that would address online library instruction in a contemporary context, employing modules loaded into the university’s existing cms, rather than podcasts as used by jowitt (2008) or other, older forms of online content delivery. the study was narrowly focused on engagement with online instruction itself, to see if the broader claims about library engagement in general, made by soria et al. (2013), apply to the specific context of library pedagogy. in response to fairly extensive research on engagement in online learning by different demographic groups in higher education, the authors chose to continue that line of research into online library instruction. they hoped to determine whether these trends persist when the student is taking a brief library course, as opposed to longer, credit-bearing courses.   aim   the authors had one principle aim for this research: to investigate whether there were any demographic trends affecting student engagement with online library instructional modules which might have implications for practice. the ultimate goal was to present a case study that would illuminate for the first time the demographic landscape of online library instruction.   method   context   at northeastern illinois university, eng 102: composition ii is a research-based writing course with high enrollment. while not a requirement for the completion of the baccalaureate degree, it is a required course in a variety of majors, ranging from the natural sciences and social sciences to the humanities. in addition to standard library instruction requirements, standard elements of all sections of eng 102 include learning outcomes, quantity and types of research-based writing assignments, and handbook selection. english department faculty teach all courses. most sections of eng 102 follow different instructor-selected course topics, and the final research assignment is most often outside of students’ chosen disciplines. in spring semester 2014, there were 18 sections offered.   recruitment   the english department requires certain standard elements, including learning outcomes and final research paper assignments, across all sections of eng 102. nevertheless, the department’s culture places a high value on the freedom of individual instructors to shape the curriculum, and there is consequently variation from one section to the next in how library instruction is performed. therefore, participant recruitment was conducted in two stages. first, the research team approached faculty assigned to teach eng 102 sections in spring semester 2014 requesting permission to randomly assign their sections to one of the three pedagogical approaches: fully online, hybrid, or traditional face-to-face. faculty support for the project was high, and therefore 14 of the 18 sections participated in the study. every participating section was a traditionally scheduled eng 102 class that met regularly in an assigned classroom on campus. the second phase of recruitment involved approaching students from participating sections to sign consent forms. the research team offered a modest incentive, valued under five american dollars, in exchange for students’ participation. student enrollment for flipped and online classes was 69.6%.   modes of instruction   the project involved three pedagogical approaches: traditional face-to-face, hybrid, and fully online. the research team designed lessons for all approaches that shared the same learning goals related to searching for and accessing journal articles in the library’s largest multidisciplinary database. this article focuses solely on the online modules delivered to the hybrid and fully online sections participating in the study.   students in both the fully online and hybrid classes received library instruction through the same online module embedded into the university’s course management system, d2l. the online module consisted of seven online pages, altogether containing four online tutorials, each under two minutes in length, short supplementary text, and three worksheets to be completed offline. the worksheets were not collected or graded. english composition instructors offered students no grade-based or additional incentives to complete the online coursework. therefore, completion was entirely voluntary. instructors in the hybrid classes asked students to complete the online module prior to their class’s assigned work day at the library. instructors in the fully online classes asked students to complete the online module prior to beginning research for their term paper.   data collection   by delivering online instruction through d2l, which requires users to log in with their unique student identification (netid), the research team could view the exact amount of time each student spent on each page of the online course module. for reporting purposes, students who spent no time on any page of the online module were labeled as “avoiders.” the research team divided students who spent any time at all in the online module into two groups: “completers” and “partial completers.” to be labeled a completer, a student needed to have spent as much or more time on each page of the module as the length of time needed to watch the video content for each page. students who fell short of that standard on one or more pages, but who spent any amount of time in the online module at all, were labeled partial completers.   the research team collaborated with their university’s office of institutional research to access demographic information on students. at the conclusion of the study, data were pulled from banner, the university’s enterprise resource planning system (erp), which is used to track, organize, and store student information. the research team requested student age, gender, race/ethnicity, cumulative gpa, and student level (i.e., freshman, sophomore, junior, or senior).   results   student participants were predominantly female (61%), which reflects the ongoing trend in american higher education of increasing enrollment by women. the most recent data from the national center for education statistics indicate that, in 2012, 56.8% of students in degree-granting secondary institutions in the united states were women (national center for education statistics, 2014).   students were predominantly 18-22 years of age. in the united states, students have historically begun college immediately after high school and completed their undergraduate work in four years, though enrollment trends are slowly changing (national center for education statistics, 2015). for the purposes of this study, we labeled those students aged 18-22 as “traditional.” alternatively, students who were older at the time of this study were grouped as “non-traditional.”   as with the university population as a whole, the largest segment of participants identified as hispanic of any race, followed by caucasian/white, asian, and african american/black. since 2007, the u.s. department of education has required institutions of higher education to collect ethnicity and race data as two questions: ethnicity (hispanic or non-hispanic) and race (at a minimum white, black or african-american, asian, american indian or alaska native, native hawaiian or other pacific islander). institutions may add other race categories if they feel they are relevant to the state or institution (u.s. department of education, 2008). the study institution, for instance, has added “nonresident alien,” for reasons unclear to the research team, as a nonresident alien could be of any race.   most students were freshmen in their second semester, which is when eng 102 is typically taken. see table 1 for descriptive statistics.     table 1 overall composition of participant sample (n = 181)   n % gender                 male                 female   70 111   38.7 61.3 agea                 traditional (18-22 years)                 non-traditional (23+ years)   139 42   76.8 23.2 race/ethnicityb                 african american/black                 asian                 caucasian/white                 hawaiian pacific                 hispanic of any race                 native american                 nonresident alien   8 26 51 3 85 1 3   4.4 14.4 28.2 1.7 48.0 0.6 1.7 student level                 freshman                 sophomore                 junior                 senior   107 58 10 6   59.1 32.0 5.5 3.3 categorized cumulative gpac                 less than 1.00                 1.00 – 1.99                 2.00 – 2.99                 3.00 – 3.99                 4.00   7 11 76 7 12   3.9 6.1 42.0 41.4 6.6 amean age = 20.97 bcategories determined by the campus office of institutional research & assessment. excludes 4 cases (2.2%) who did not report race/ethnicity. cmean gpa = 2.83     for the independent variables of age and cumulative gpa, analysis revealed observable trends, but no statistically significant associations. age, a scale variable, was coded into categories prior to cross tabulation analysis using a pearson chi-square test (p = .129). student participants were categorized as either traditionally aged (18-22 years) or non-traditionally aged (23 or older). as shown in table 2, non-traditionally aged students were more likely to be completers (35.7%) than were traditionally aged students (20.9%). at the other end of the spectrum, traditionally aged students were more likely to be avoiders (45.3%) than were non-traditionally aged students (33.3%).     table 2 age levels for avoiders, partial completers, and completers n (% of age category)   traditional (18-22 years) non-traditional (23+ years) avoider 63 (45.3) 14 (33.3) partial completer 47 (33.8) 13 (31.0) completer 29 (20.9) 15 (35.7)     table 3 race/ethnicity of avoiders, partial completers, and completers n (% of age category)   afr. amer./ black asian cauc./white hispanic other avoiders 3 (37.5) 5 (19.2) 24 (47.0) 40 (47.1) 3 (42.9) partial completer 4 (50.0) 11 (42.3) 16 (31.0) 26 (30.6) 3 (42.9) completer 1 (12.5) 10 (38.5) 11 (22.0) 19 (22.4) 1 (14.2)     for cumulative gpa prior to spring semester 2014, level of engagement increased with mean student gpa: avoiders’ mean cumulative gpa entering the semester was 2.72, while partial completers’ mean cumulative gpa was 2.81, and completers’ mean cumulative gpa was 3.07.   analysis by gender suggested a minor trend, but, again, no statistically significant association (p = 0.377). female students were slightly more likely than male students to engage with the online library module: 27% of women were completers, compared with 20% of men, whereas 38.7% of women were avoiders, compared with 48.6% of men.   as shown in table 3, race/ethnicity analysis did not reveal any meaningful trends. the two largest ethnic groups, caucasians and hispanics of any race, were almost identical in their level of engagement at every level (22% of caucasians were completers, as were 22.4% of hispanics; 47% of caucasians were avoiders, as were 47.1% of hispanics). the third largest ethnic group, asians, did engage at a noticeably higher level (38.5% of asians were completers, and only 19.2% were avoiders), but the small size of the group severely limits any conclusions that can be drawn from this data.   discussion/future directions   the observable trend for age in the data corresponds with the trend observed by kunhimohamed (2012), indicating that age has a small effect on engagement. specifically, older students were more willing to spend time interacting with the online instructional module. an implication of the data is that online library modules similar to those used in the study may not be likely to reach a traditionally-aged population of first year students, as nearly half of them never accessed the module, and the vast majority of those who did access the module failed to complete it. as instructors did not attach an external incentive, such as a completion grade, to the online library module, it is possible that the use of an incentive could alter this engagement pattern. a possible conclusion from this study is that higher production values in the library videos alone may not engage younger students. both incentives and active learning components need to be piloted in programs serving a large percentage of traditionally aged students. a library that plans to implement online instruction modules for on-campus students as a part of its larger instructional program might be most successful by using those modules in programs that serve primarily non-traditionally aged populations, as that may yield higher levels of engagement and more effective outcomes for students.   the observable trend for gpa in the data supports the results found by soria et al. (2013), indicating that gpa has a small effect on engagement with the library. as students’ cumulative gpa rose, so too did the students’ willingness to interact with the online instructional module. the authors feel that this result was perhaps the least surprising outcome in the study. higher performing students would normally be expected to complete course assignments, including the assigned online library module, more conscientiously, even absent a specific external incentive. this trend suggests that lower performing students may be the ones most likely to be left behind in online instruction. institutions looking to provide online library instruction should be mindful of the needs of this group in the design and implementation of online instruction modules.   the data show a very slight tendency for female students to engage more persistently than male students. this effect is consistent with the findings of multiple studies mentioned in the literature review (marley, 2007; kunhimohamed, 2012; shen et al., 2013) that found this distinction between the genders in engagement with online coursework in general. that consistency implies that the small difference between the genders in this data may be more significant than the sample size would otherwise warrant, but based only on the data from this study, the authors do not conclude that there are significant implications for practice. given the trend in the literature, institutions that have identified concerns about low academic engagement by male students might wish to proceed with some caution in relying too heavily on online library instruction modules.   a comparison of the data for the two largest ethnic groups in the study (caucasians and hispanics) demonstrates no trend of any kind, as the two groups were virtually identical at every level of engagement with the module. one of the smaller ethnic groups in the study, asian-american students, did show evidence of substantially higher levels of engagement, but given the smaller size of that group, the authors cannot draw conclusions that would impact practice. the university where this study took place is a hispanic-serving institution with substantial populations of both caucasian and hispanic students. as a result, the comparison of chief interest for potential application here was between those two groups.   limitations   the greatest limitation of the study was the size of the sample (181 students), which may have hindered the research team from identifying statistically significant results in areas where there were observable trends. the research team’s recruitment efforts were inhibited by several leaks of private information that were highly publicized in the media just prior to spring semester 2014. students were consequently cautious about signing the informed consent forms that would allow the researchers to access their banner data.   furthermore, this study focused on a distinctive population. students of eng 102 are generally first year students participating in a required course and writing on topics that may fall outside of their chosen majors. because the institution did not offer composition as a distance learning course at the time the study was conducted, all students were enrolled in fully on-campus courses. thus, this project can make claims only about on-campus students in an english composition class. distance students, students taking upper-level courses, or students taking courses within their major may behave differently from the population studied and demonstrate different levels of engagement with online library instruction.   while the authors consciously attempted to maximize the internal validity of this research study, it was not possible to control every potentially confounding variable. multiple faculty from the english department taught the different sections of eng 102. therefore, students in the various sections may have received divergent instructions regarding expectations for the library module and its relevance to their research assignment. the study also does not report data about students’ library and technology experiences outside of eng 102, such as prior library instruction, or access to the internet from off-campus. one major demographic category not recorded in the banner data provided by the institution was socioeconomic level, which the authors would have included in their analysis had it been available. although the research team designed the online module according to principles of best practice, there are numerous alternative approaches to the delivery of online instructional content that could have impacted the student engagement patterns observed in this study. due to these potentially confounding factors, the research team cannot make claims regarding causation.   lastly, this study’s measure of student engagement (literally, time spent logged in on each of the pages of the tutorial) has the advantage of being concrete and measurable. however, it does not reflect the qualitative aspects of student engagement that would be important in many contexts. moreover, because there was no analogous measure of engagement for the face-to-face classes, the authors were unable to make comparisons of engagement between students participating in online instruction and traditional instruction.   recommendations   the authors ground their analysis of this study in their belief that an effective library instruction program employs diverse pedagogical approaches in order to educate the largest possible number of students. their institution, therefore, intended to use student engagement data as a means of indicating where online modules might be implemented most effectively in the library’s instruction plan. the academic librarians who formed the research team had hoped that students of eng 102 would respond enthusiastically to online instruction, given the convenience of the medium and its potential to expand the reach of the library instruction program.   the generally low level of engagement with online instruction across all demographic groups was a disappointing outcome of the study, but one that was highly instructive. despite the potential convenience of online learning, both for students and librarians, the population under study did not engage with the online module at a high enough level to warrant its widespread adoption for eng 102 classes at the authors’ institution. the institution continues, however, to explore online instruction delivery methods and develop alternative modules for online instruction, under the direction of a newly-hired e-learning librarian.   the authors would recommend that future studies of online library instruction explore how modifications to both the design and delivery of the online module might affect engagement. one particularly promising modification is the use of incentives (especially the most common incentive in higher education: a grade for completion). incentives might raise engagement levels across all demographic groups, and furthermore might alter some of the trends observed in this study. another potential modification is the use of online modules with a different student population, such as upper-level undergraduate or graduate students. the authors initially elected to deliver the module through the institution’s cms for ease of student access, but alternative platforms should be considered and studied, especially those that would allow for more interactive module designs.   given the widespread use in academic libraries of traditional face-to-face instruction, the authors would recommend that future studies compare engagement in that setting to engagement in the online setting. it is possible that some groups reluctant to engage in one setting will be better served in another, especially given the major differences inherent in the two pedagogical approaches.   conclusion   this paper presents a case study that investigates the relationship between student demographic characteristics and engagement with online library instruction modules in english 102 courses at a single university. the demographic factors studied were age, cumulative gpa, gender, and race/ethnicity. while no statistically significant associations were found in any demographic category, observable trends tied age and higher gpa to higher levels of engagement with online instruction. these trends are substantial enough to influence current practice at the authors’ institution. there was additionally a slight trend linking female participants to higher levels of engagement than their male peers. in the category of race/ethnicity, hispanic and caucasian students exhibited similar levels of engagement. given the limited sample size for the study overall, the authors hope that future investigators will continue to examine the questions raised by this study, in order to more conclusively determine whether demographic characteristics of the student population should guide the design and implementation of online library instruction.   references   alexander, l. b., & smith, r. c. (2001). research findings of a library skills instruction web course. portal: libraries and the academy, 1(3), 309-328. retrieved from https://muse.jhu.edu/journals/portal_libraries_and_the_academy/ bowen, a. (2014). libguides and web-based library guides in comparison: is there a pedagogical advantage? journal of web librarianship, 8(2), 147-171. doi:10.1080/19322909.2014.903709 callaway, s. k., & alflayyeh, s. m. (2011). understanding critical distance learning issues: toward a comprehensive model predicting student satisfaction. information resources management journal, 24(4), 61-76. doi:10.4018/irmj.2011100104 dutton, j., dutton, m., & perry, j. (2001). do online students perform as well as lecture students? journal of engineering education, 90(1), 131-136. doi: 10.1002/j.2168-9830.2001.tb00580.x halsne, a. m., & gatta, l. a. (2002). online versus traditionally-delivered instruction: a descriptive study of learner characteristics in a community college setting. online journal of distance learning administration, 5(1). retrieved from  http://www.westga.edu/~distance/ojdla/spring51/halsne51.html hispanic association of colleges & universities. (n.d.). hispanic-serving institution definitions. retrieved from http://www.hacu.net/hacu/hsi_definition.asp jaggars, s. s., & xu, d. (2010). online learning in the virginia community college system. retrieved from: http://ccrc.tc.columbia.edu/publications/online-learning-virginia.html jowitt, a. (2008). perceptions and usage of library instructional podcasts by staff and students at new zealand's universal college of learning (ucol). reference services review, 36(3), 312-336. doi.org/10.1108/00907320810895396 koohang, a. (2004). students’ perceptions toward the use of the digital library in weekly web-based distance learning assignments portion of a hybrid programme. british journal of educational technology, 35(5), 617-626. doi:10.1111/j.0007-1013.2004.00418.x kunhimohamed, b. b. v. (2012). student participation and grade performance in an undergraduate online learning environment. retrieved from https://dspace.library.colostate.edu/ marley, j. l. (2007). gender differences and distance education: major research findings and implications for lis education. journal of education for library and information science, 48(1), 13-20. retrieved from http://www.alise.org/jelis-2 national center for education statistics (2014). total fall enrollment in degree-granting postsecondary institutions, by attendance status, sex of student, and control of institution: selected years, 1947 through 2023. retrieved from http://nces.ed.gov/programs/digest/d13/tables/dt13_303.10.asp national center for education statistics. (2015). chapter 3: postsecondary education. in digest of education statistics, 2013. retrieved from http://nces.ed.gov/programs/digest/d13/ch_3.asp shen, d., cho, m., tsai, c., & marra, r. (2013). unpacking online learning experiences: online learning self-efficacy and learning satisfaction. internet & higher education, 19, 10-17. doi:10.1016/j.iheduc.2013.04.001 soria, k. m., fransen, j., & nackerud, s. (2013). library use and undergraduate student outcomes: new evidence for students' retention and academic success. portal: libraries and the academy, 13(2), 147-164. doi:10.1353/pla.2013.0010 u.s. department of education. (2008). policy questions on the department of education’s 2007 guidance on collecting, maintaining, and reporting data by race or ethnicity. retrieved from: http://www2.ed.gov/policy/rschstat/guid/raceethnicity/questions.html   yoo, s. j., & huang, w. d. (2013). engaging online adult learners in higher education: motivational factors impacted by gender, age, and prior experiences. journal of continuing higher education, 61(3), 151-164. doi:10.1080/07377363.2013.836823 yukselturk, e. (2010). an investigation of factors affecting student participation level in an online discussion forum. turkish online journal of educational technology, 9(2), 24-32. retrieved from http://www.tojet.net/   evidence summary   personal publications lists serve as a reliable calibration parameter to compare coverage in academic citation databases with scientific social media   a review of: hilbert, f., barth, j., gremm, j., gros, d., haiter, j., henkel, m., reinhardt, w., & stock, w.g. (2015). coverage of academic citation databases compared with coverage of scientific social media: personal publication lists as calibration parameters. online information review 39(2): 255-264. http://dx.doi.org/10.1108/oir-07-2014-0159   reviewed by: emma hughes freelance information professional norwich, england email: emma.e.hughes@outlook.com   received: 1 aug. 2016   accepted: 19 oct. 2016      2017 hughes. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – the purpose of this study was to explore coverage rates of information science publications in academic citation databases and scientific social media using a new method of personal publication lists as a calibration parameter. the research questions were: how many publications are covered in different databases, which has the best coverage, and what institutions are represented and how does the language of the publication play a role?   design – bibliometric analysis.   setting – academic citation databases (web of science, scopus, google scholar) and scientific social media (mendeley, citeulike, bibsonomy).   subjects – 1,017 library and information science publications produced by 76 information scientists at 5 german-speaking universities in germany and austria.   methods – only documents which were published between 1 january 2003 and 31 december 2012 were included. in that time the 76 information scientists had produced 1,017 documents. the information scientists confirmed that their publication lists were complete and these served as the calibration parameter for the study. the citations from the publication lists were searched in three academic databases: google scholar, web of science (wos), and scopus; as well as three social media citation sites: mendeley, citeulike, and bibsonomy and the results were compared. the publications were searched for by author name and words from the title.   main results – none of the databases investigated had 100% coverage. in the academic databases, google scholar had the highest amount of coverage with an average of 63%, scopus an average of 31%, and lowest was wos with an average of 15%. on social media sites, bibsonomy had the highest coverage with an average of 24%, mendeley had an average coverage of 19%, and the lowest coverage was citeulike with an average of 8%.   conclusion – the use of personal publication lists are reliable calibration parameters to compare coverage of information scientists in academic citation databases with scientific social media. academic citation databases had a higher coverage of publications, in particular, google scholar, compared to scientific social media sites. the authors recommend that information scientists personally publish work on social media citation databases to increase exposure. formulating a publication strategy may be useful to identify journals with the most exposure in academic citation databases. individuals should be encouraged to keep personal publication lists and these can be used as calibration parameters as a measure of coverage in the future.   commentary   measuring coverage and impact of information scientists work is ever changing in the advent of scientific social media (bar-ilan et al, 2012). this study used a new calibration method of personal publications lists to compare coverage of publications from information scientists in both academic citation databases and scientific social media. the study was appraised using the ebl critical appraisal checklist (glynn, 2006). the strength of this study lies in the use of new calibration parameter of personal publication lists. the study scored high for data collection and for study design meaning that the study could be replicated. contacting the information scientists of the institutions to confirm that the authors had full publication lists ensured that they could achieve fairly accurate analysis of results although use of identifiers, such as orcid, may have provided stronger accuracy.   the objectives and research questions of this study were clearly focussed. the methodology builds on an approach used in a previous study (kirkwood, 2012) and clearly described the design, data collection, and analysis. however, it is not wholly explained why the particular three academic databases and three scientific social media databases used were selected over others, but the methodology would be easy to replicate and the limitations were also discussed.   this study focussed on information scientists working in german-speaking institutions and highlights the potential limitations of publishing in non-english language journals. the research found that the coverage of papers from dusseldorf were low in web of science due to the fact that many information scientists publish in german information science journals, which are not indexed by the database. though the authors acknowledge that there is no recent evidence on language skills of information scientists, it is possible that this is the deciding factor when information scientists are at the research publishing stage. this suggests that library and information professionals should continue to promote their work, particularly where work is not indexed in traditional databases and in instances when non-english language publications have less coverage.   the study limited the search to scholarly publications and not informal published documents. exploration of differences in coverage between document type and coverage of papers in library specific databases such as lisa and lista, would be interesting areas for further investigation.   glynn’s critical appraisal checklist advises that if overall validity of a study scores >75% then the study is valid. overall this study scored 77%.   the methodology used in this study will be of interest to information and library practitioners who want to show their research impact both on academic citation databases and social media. the research also highlighted the need to keep personal publication lists and the value of self-indexing on appropriate scientific social media for library and information professionals.   references   bar-ilan, j., haustein, s., peters, i., priem, j., shema, h., and terliesner, j. (2012). “beyond citations: scholars’ visibility on the social web”, in proceedings of the 17th international conference on science and technology indicators. montréal, canada. pp. 98–109.   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692145   kirkwood, p.e. (2012). “faculty publication checklists: a quantitative method to compare traditional databases to web search engines”, paper presented at the 119th american society for engineering education annual conference & exhibition, san antonio, tx, 10-13 june. www.asee.org/public/conferences/8/papers/3879/download     evidence based library and information practice 2009, 4:2 161 evidence based library and information practice evidence summary topic-specific infobuttons reduce search time but their clinical impact is unclear a review of: del fiol, guilherme, peter j. haug, james j. cimino, scott p. narus, chuck norlin, and joyce a. mitchell. ‚effectiveness of topic-specific infobuttons: a randomized controlled trial.‛ journal of the american medical information association 15.6 (2008): 752-9. reviewed by: shandra protzko information specialist, tucker medical library, national jewish health denver, colorado, united states of america e-mail: protzkos@njhealth.org received: 12 february 2009 accepted: 25 march 2009 © 2009 protzko. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. objective – to assess whether infobutton links that direct users to specific content topics (‚topic links‛) are more effective in answering clinical questions than links that direct users to general overview content (‚nonspecific links‛). design – randomized control trial. setting – intermountain healthcare, an integrated system of 21 hospitals and over 120 outpatient clinics located in utah and southeastern idaho. subjects – ninety clinicians and 3,729 infobutton sessions. methods – to ensure comparable group composition, subjects were paired and randomly allocated to the study groups. clinicians in the intervention group had access to topic links, while those in the control group had access to nonspecific links. all subjects at intermountain healthcare use a web-based electronic medical record system (emr) called help2 clinical desktop with integrated infobutton links. an infobutton manager application defines the content topics and resources; in this case, micromedex® (thomson healthcare, englewood, co) provided access to the topic links. the medication order entry module, the most popular of the outpatient modules, was selected to test the two configurations of infobuttons. a focus group of seven help2 users aided the researchers in determining the most salient topics to be displayed as a part of the intervention group's user-interface. the study measured infobutton session duration, or time spent seeking information, the number of mailto:protzkos@njhealth.org evidence based library and information practice 2009, 4:2 162 infobutton sessions conducted, and the outcome and impact of the information seeking. a postsession questionnaire displayed randomly in 30% of sessions measured outcome and impact. the study was conducted between may and november, 2007. this project was funded in part by the national library of medicine. main results – subjects in the intervention group spent 17.4% less time seeking information than those in the control group (35.5 seconds vs. 43 seconds, p = 0.008). the intervention group used infobuttons 20.5% more often (22 sessions vs. 17.5 sessions, p = 0.21) than those in the control group, a difference that was not statistically significant. twenty-five subjects answered the post-session survey at least once for a total of 115 (9.9%) responses out of 1,161 possible sessions. the information seeking success rate was equally high in both groups (87.2% intervention vs. 89.4% control, p = .099). subjects reported high positive clinical impact (i.e., decision enhancement or learning) in 62% of successful sessions. subjects conveyed a moderate or high level of frustration in 80% of responses associated with unsuccessful sessions. conclusion – topic links provide a slight advantage in the clinical decision-making process by reducing the amount of time spent searching. but while the session length difference between the control and intervention groups is statistically significant, it is less clear whether the difference is clinically meaningful. as previous studies have indicated, infobuttons are able to answer clinical medication questions with a high success rate. it is unclear whether topic links have a clinically significant impact, or rather, whether they are more effective than nonspecific links. the authors believe that the study results ‚should generalize to highfrequency, medication-related infobutton users in other institutions‛ (758). commentary infobuttons are decision-support tools designed to use the context of an emr interaction to retrieve clinically relevant content and to provide links to information resources. topicspecific infobutton links are intended to be more efficient than nonspecific links by more closely matching a clinician’s implied information need at the point of care. given that topic links are more difficult to implement, it is critical to know whether they offer a clinical benefit. the authors compare and contrast the study to previous work; they cautiously assert that the literature reinforces the study results in a few cases. they thoroughly account for the limitations of the study, and the conclusions accurately reflect the analysis. the inclusion of survey details and a careful description of the methodology allow other researchers in similar settings to replicate the project. the authors describe in detail the study population, setting, and group composition. because the study focused on a subpopulation of experienced, frequent infobutton users, as the authors point out, the user success rate may have been exaggerated and may not be applied to low-frequency users. since the questionnaire measured the success rate and outcome based on self-assessment, it was prone to bias, and may not objectively gauge clinical outcomes. the authors suggest alternate strategies to overcome this limitation. moreover, as the authors make clear, approximately one third of the subjects were not enrolled until halfway through the study and thus may not have had enough time to perceive much difference between the infobutton configurations. the authors received support with the statistical analysis, however one erratum creates confusion about the number of subjects who actually used infobuttons during the study period. the authors first state that 90 out of 104 subjects used infobuttons, and of those who did not, most subjects never accessed the medication order entry module. later the authors say that 102 of 104 accessed infobuttons. presumably the first statement is correct. it is a small error in an otherwise well-written report. evidence based library and information practice 2009, 4:2 163 while topic links provide a slight advantage in the clinical decision-making process by reducing the amount of time spent searching, the absolute difference in session length between the control and intervention groups in the study may not be considered significant enough to warrant a change in practice. the analysis of unsuccessful sessions (12% of survey responses) – some of which were due to code mapping problems, incomplete indexing or lack of content – underscores the need for continuous monitoring as a knowledge management practice. as the authors suggest, future studies are needed to evaluate the effectiveness of infobuttons in other emr contexts, such as laboratory results modules, for example. since the specific characteristics of an information resource such as micromedex® influence effectiveness, it would be useful to explore ‚methods that are able to more accurately predict the information needs that arise in a given context as well as the resources that are most likely to fulfill these needs‛ (758). like all technology trends, infobuttons provide both an opportunity and a challenge. they have the potential to improve efficiency, enhance clinical resources, and positively affect patient care. since 1997, several of the authors of the current study have made major contributions to the literature on infobuttons (although it is worth mentioning that, to date, there is no medical subject heading for ‘infobuttons’ in medline). this study contributes to our knowledge of decision-support tools and helps decision makers weigh the benefits and shortcomings of the current state of infobuttons. evidence summary   social networking tools for informal scholarly communication prove popular for academics at two universities   a review of: al-aufi, a., & fulton, c. (2015). impact of social networking tools on scholarly communication: a cross-institutional study. the electronic library, 33(2), 224-241. http://dx.doi.org/10.1108/el-05-2013-0093   reviewed by: aoife lawton systems librarian health service executive dr. steevens’ library, dr. steevens’ hospital dublin 8, ireland email: aoife.lawton@hse.ie   received: 1 mar. 2016    accepted: 15 apr. 2016      2016 lawton. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to investigate the adoption, use, perceived impact of, and barriers to using social networking tools for scholarly communication at two universities.   design – cross-institutional quantitative study using an online survey.   setting – academics working in the disciplines of the humanities and social sciences at two universities: one in europe and one in the middle east.   methods – an online survey was devised based on a previous survey (al-aufi, 2007) and informed by relevant research. the survey was piloted by 10 academics at the 2 participating universities. post pilot it was revised and then circulated to all academics from similar faculties at two universities. three follow up emails were sent to both sets of academics. the data was analyzed using statistical package for the social sciences (spss) software.  descriptive and inferential statistics were analyzed using anova tests.   main results – the survey achieved a 34% response rate (n=130). the majority of participants were from the university based in the middle east and were male (70.8%). most of the responses were from academics under 40 years of age. the use of notebooks was prevalent at both universities. “notebooks” is used as a term to describe laptops, netbooks, or ultra-book computers.  the majority reported use of social networking tools for informal scholarly communication (70.1%), valuing this type of use. 29.9% of respondents reported they do not use social networking tools for this purpose. barriers were identified as lack of incentive, digital literacy, training, and concerns over internet security. among the non-users, barriers included low interest in their use and a perceived lack of relevancy of such tools for scholarly communication. the types of tools used the most were those with social connection functions, such as facebook and twitter. the tools used the least were social bookmarking tools. a one-way analysis of variance (anova) test indicated that there was no significant difference at the 0.05 level between the use of social networking tools at both universities, with the exception of using tools to communicate with researchers locally and with publishers at one of the universities.   both universities use tools for communication with peers and academics internationally. the responses were mainly positive towards the perceived usefulness of social networking tools for informal scholarly communication.   conclusion – the authors conclude that despite the small sample of the community of academics investigated, there is a general trend towards increasing use and popularity of social networking tools amongst academics in the humanities and social sciences disciplines. as technology advances, the use of such tools is likely to increase and advance among academics. the authors point to pathways for future research including expanding the methods to include interviews, focus groups, and case studies. another angle for research of interest is interdisciplinary differences in the use of prevalent tools such as facebook and twitter.   commentary   this research investigates an emerging field of social networking tools in the context of informal scholarly communication. the literature review reveals that some research has investigated this topic. the authors’ research offers seven categories of social networking tools: social connections, multimedia sharing, professional, academic, blogging, social bookmarking, and cross-platform mobile apps.  however, as technology and social networking tools are ever-evolving, the literature is quickly outdated as newer tools emerge and are adopted. the current research study offers an insight at a distinct point in time of their use amongst a particular academic sector, humanities and social sciences, which is valuable. it provides an insight into the prevalence of social networking tools among academics working in vastly different countries operating under different higher education systems, which is also of interest.   the specificity of the population and presentation of results score high on glynn’s critical appraisal checklist (2006). the inclusion criteria are outlined. although the sample size is small, the interpretation of the results is relevant to the population studied. ethical considerations were taken into account; however, this is not described in any detail. it is unclear how many questions were contained in the survey. results are presented in eight tables, and questions make use of likert scales. however, the data analysis using anova is not appropriate given that all of the results are nominal, ranks, or likert-type responses.  the use of a chi-square would have been more statistically sound. a good analysis of the results is given with discussion in relation to the data collected.   there is a comparison made between the two academic institutions under study, but there is no description given of the it infrastructure in place at either university. it would have been interesting to include the wider socio-economic context for the adoption of social networking tools in both countries.   the survey questions are not included in the article or appendix, and it is not clear if all of the questions that were asked are described. it would have been useful to have included the survey so that it could be replicated by further studies. the authors outline clearly and precisely the limitations of the study and offer several ideas and suggestions for future directions which would build upon the current research.   librarians working in academic settings will find this study to be of interest in that it explores the use of social networking tools in two universities. librarians working in each of the universities in question may benefit from reading this research in full, as it offers a unique insight into adopted social networking tools of academics at their institutions. in addition, librarians may benefit from adopting findings from this research into their own engagement with researchers, for example in the areas of outreach, instruction, research, collection development, and collaboration.   references   al-aufi, a. (2007). networked research and scholarly communication in a developing arabic country: an investigation of sultan qaboos university, oman (unpublished doctoral dissertation). curtin university of technology, perth. boone, h. n., jr., & boone, d. a. (2012). analyzing likert data. journal of extension, 50(2), 1-5. http://www.joe.org/joe/2012april/tt2.php   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154   evidence summary   medical students in the united states reveal their ideal expectations to help planners of a new library   a review of: aronoff, n. (2016). surveying medical students to gauge library use and plan for a new medical library. medical reference services quarterly, 35(2), 187-203. http://dx.doi.org/10.1080/02763869.2016.1152144   reviewed by: aislinn conway phd fellow national university of ireland galway galway, ireland email: a.conway18@nuigalway.ie   received: 13 june 2017  accepted: 26 july 2017      2017 conway. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract objective – to help plan for a new library by exploring student use of existing library services and identifying their priorities for the new space.   design – online survey, sent via email.   setting – medical school at a university in new york.   subjects – 585 medical students.   methods – the researchers emailed a 45-item online survey to all medical students enrolled at the school. responses were anonymised and all questions were non-mandatory.   main results – 27% of students (157 out of 585) took part in the survey by answering at least one question. the questions were categorised into the following six topic areas:   1. use of space and expectations for the new library space: more than half of the participants (67%) indicated that they rarely or never came to the library during the academic year in question. of the students who reported frequenting the library on a daily, weekly, or monthly basis, the majority indicated that they preferred independent study to group study. the following resources were ranked as very important for an ideal library space: sufficient electrical outlets, strong wireless connectivity, printing facilities, individual and quiet study spaces, comfortable seating, online resources, computers, windows/natural light, and group study spaces. open-ended responses indicated that students desire close proximity to food and beverage services, large study tables to accommodate reading materials and technology, improved opening hours, and satisfactory bathroom facilities.   2. where medical students study: of the participants, one third of students reported studying at home, 21% chose to describe the physical characteristics of their place of study rather than name a place, 18% of students studied in multiple places, and 16% studied in the library. the remainder studied in another library, cafés, or other locations on campus. online resource use was much higher than borrowing figures with the majority of students indicating that they had never borrowed a print book (77%), a reserve book (90%), or a dvd (96%). in addition, 92% indicated never consulting a print reference book. online resources were used at least once a semester by 90% of students.   3. resource use and expectations: most students used lecture notes, presentations, websites, personal copies of books, clinical decision support tools, online tutorials or video content, electronic journal articles, recorded video or audio lectures, medical apps, electronic books, clinical practice guidelines, or pocket manuals or pocket guides. print books from the library were the least exploited resources with only 13% of students reporting their use. 83% of students ranked online resources as the most important feature of an ideal library.   4. equipment use and expectations for equipment and technology: in terms of equipment required for an ideal library space, 88% of students indicated printers, 78% computers, and 69% scanners. therefore, easy access to electrical outlets and strong wireless connections were hugely important.   5. services: book or article requests were only sought monthly or once per semester by 18% and 7% of students respectively. more than half of students (54%) felt that assistance from a librarian was a very important or important feature of an ideal library space. however, 68% never consulted a librarian in the past and of those who did they did not do so frequently. in-person or email contact with a librarian was preferred over other methods of communication. 52% of respondents were not interested in training provided by the library. of those who were, online and virtual training was preferred by 51% when compared to face to face instruction.   6. additional feedback: the vast majority of students (90%) indicated that they would be interested in using the library outside of the existing opening hours of 9:00a.m. to 5:00p.m., monday to friday. regarding the overall library service, 53% of students were satisfied or very satisfied, 26% were neither satisfied nor unsatisfied, and 21% were unsatisfied or very unsatisfied. lighting, electrical outlets, and having a place to get food and drink were also prioritized by students in this section of the survey.   conclusion – the author concluded that since convenience was considered an important factor by students when choosing their place of study, the increased proximity of the new library should attract more students. in accordance with student preferences, both individual and group study spaces are planned for the new library. sufficient electrical outlets and a glass façade increasing the amount of natural light will feature in the building. core textbooks and reference books will be made available in a small area onsite despite the fact that this did not feature in the original plan. computers and printers will also feature in the new library for students who require equipment to facilitate their study activities. a computer lab to accommodate 30 students will enable face to face instruction on library resources. a professional librarian will not be based at the new library. in-person services will be available at another library with sufficient staffing.   commentary     this study adds to a small body of literature addressing the needs of medical students in relation to library services. a study by norton (2013) asked multiple user groups about their preferences for a new library. the results were similar in both papers highlighting the importance of online technologies, associated infrastructure, and the importance of creating comfortable study areas.   exploring survey responses to help understand determinants of library usage by medical students at the current site provides valuable insight into what the participants view to be characteristics of an ideal library.   glynn’s eblip critical appraisal checklist (2006) will be used as a critical appraisal tool for this evidence summary, specifically sections b and d which cover data collection and results.   the survey was hosted on an online platform which collects all data. although the author provided citations to a number of studies on which the survey instrument is based, they did not specify what has been included or excluded from these studies. the author does not report measuring cronbach’s alpha (tavakol & dennick, 2011) to assess reliability or whether any testing for validity was carried out. there is no mention of the survey instrument being piloted before distribution to students. the instrument is not published with the article, making it difficult to assess whether all the findings were reported or how the wording of the questions may have impacted on the results. the survey was distributed directly before the graduation of fourth year students and during first and second year students’ exams, so this may not have been the optimal time to recruit student participants. however as an incentive, all students who took part were given a chance to win one of five gift cards.   the author discusses confounding variables such as location of the current library and timing of the survey that may have impacted on the survey results. the conclusions reflect the analysis but also highlight the fact that this survey of medical students is only one user type and the task force will also plan ahead taking into account the needs of these users.   the results are presented as numbers and percentages of participants and statistical devices to determine significance such as p values and confidence intervals were not utilised. there is some repetition in the article regarding the results and students’ priorities. some of the data could have been reported more concisely to improve readability of the results.   external validity was not important to the researcher in this study. the goal was to identify the views of the students at their university so they could be incorporated into plans for the library at that site. however, other researchers interested in medical students’ views of libraries and library services could adapt this study to help them investigate contextual issues specific to their own organizations.   the author provides implications for further research such as the need to investigate how the library might facilitate student learning, what other resources could be offered to students, and how the lack of a large book collection will impact students. the researcher intends to distribute the same survey to students after the new library opens in a follow up study, giving students time to start using it and to establish new behaviours. this will help to gain insight into what has worked, what could be improved, and under which circumstances.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154      norton, h. f., butson, l. c., tennant, m. r., & botero, c.e. (2013). space planning: a renovation saga involving library users. medical reference services quarterly, 32(2), 133–150. http://dx.doi.org/10.1080/02763869.2013.776879   tavakol, m. & dennick, r. (2011). making sense of cronbach’s alpha. international journal of medical education, 2, 53-55. http://dx.doi.org/10.5116/ijme.4dfb.8dfd     microsoft word art_ed_1.doc evidence based library and information practice 2006, 1:1    1 evidence based library and information practice       editorial      lindsay glynn  editor‐in‐chief    public services librarian and instruction coordinator, health sciences library   memorial university of newfoundland   st. johnʹs, newfoundland, canada  e‐mail: lglynn@mun.ca      © 2006 glynn. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      evidence based librarianship: we hear about  it, we read about it, some of us practice it,  some of us aren’t quite sure what it’s all  about.  information about ebl is relatively  prolific of late and can be found in  numerous journals, websites and books.  it  is indexed in databases and article indices  across the disciplinary board – medline,  cinahl, eric, psycinfo, dois, lista, etc.   evidence based library and information  practice is the first journal whose primary  focus is on ebl: what it is, how it affects the  decisions we make, how we look at and  perform research, and how it allows us to  make more informed decisions based on the  best available evidence.  eblip, not unlike  the practice of ebl, is international in scope  and unlimited in its application and utility.      getting eblip off the ground has been a  fascinating and rewarding experience, and i  have several key people to thank for making  the process seamless and satisfying.  eblip  is the brainchild of su cleyle, of memorial  university of newfoundland and denise  koufogiannakis, of the university of alberta.   su and denise got the ball rolling early in  2005 while researching ebl on sabbatical  and quickly pulled together an enthusiastic  team of editors.  as original editor‐in‐chief,  su’s vision, drive and unbeatable  organization resulted in eblip becoming a  reality more quickly and completely than  we could have imagined.  as associate  editor (evidence summaries), denise has  begun to compile and disseminate a  significant and highly useful collection of  critically appraised synopses of research  articles.  the evidence summaries section  allows eblip to reach well beyond  publishing research and adds a component  that our field has, until now, been lacking.   in addition to her tireless work in this area, i  thank denise for keeping the momentum  going throughout the entire start‐up process.      eblip is published by university of alberta  learning services using ojs (open journal  systems).  we are very fortunate to have  pam ryan, from the university of alberta,  as our production editor.  in addition to the  mammoth task of creating an open access,  peer reviewed electronic journal, we also  were faced with the tasks of familiarizing  ourselves with new software and creating a  visual appearance.  pam’s knowledge,  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:1    2 ability and creativity, not to mention her  endless patience with the rest of the editorial  team, have resulted in the professional  presence of eblip.  her work is not  transparent to the readers, but paramount to  the editorial team and the journal.      finally, i would like to draw attention to  two teams that are essential to the success of  eblip: the evidence summaries team and  the editorial advisors (peer reviewers).  the  diversity, specialization and professionalism  of the teams’ membership allow us to  explore all avenues of ebl and its practice  and utility in all environments.  members  from our teams range from journal editors  to university professors to school and  systems librarians.  their contributions, not  only to eblip but also to the body of ebl  knowledge, are substantial.                                                        as readers, i hope that you will find the  information in eblip useful, interesting,  provocative, and applicable to your work.  i  hope that you will be able to use the  information we present to you to aid in your  decision making regarding policies,  collections, procedures, marketing, etc., and  i particularly look forward to hearing how  you may have used this information in your  practice (whether it worked or not!).   evidence summary   experiences, benefits, and challenges of virtual teamwork for public libraries in the us midwest during the covid-19 pandemic   a review of: singh, k., & bossaller, j. s. (2022). it’s just not the same: virtual teamwork in public libraries. journal of library administration, 62(4), 512–534. https://doi.org/10.1080/01930826.2022.2057130   reviewed by: eugenia opuda health & human services librarian associate professor dimond library university of new hampshire durham, new hampshire, united states of america email: eugenia.opuda@unh.edu   received: 1 sept. 2023                                                               accepted: 10 oct. 2023      2023 opuda. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30432     abstract   objective – to learn about public libraries’ transition to virtual teams before and during the covid-19 pandemic, and to uncover the benefits, as well as challenges, of using technology and tools for virtual teamwork.   design – qualitative, online, semi-structured interviews with thematic analysis of data.   setting – public libraries in the midwestern united states.   subjects – eight leaders of public libraries or library systems and a state library.   methods – the authors conducted a snowball sampling technique to recruit participants in the midwestern united states. through 30 to 60 minute zoom sessions, the authors conducted interviews with the study participants. each session was led by one interviewer and a note-taker and all interviews, with one exception, were recorded. interviewers relied on an interview protocol, shared in the appendices of the paper, that addressed demographic questions as well as questions related to the study objectives. the authors comment that they reached data saturation after conducting interviews with eight subjects and decided to stop recruitment. using the notes from the interview and zoom session transcripts, the authors individually analyzed the data and then collectively discussed the themes as well as similarities and differences of participant responses.   main results – study respondents were midto late-career professionals in medium to large organizations. all respondents were white and a majority were women (n = 6) with only two men. a majority (n = 7) had a master’s degree in library and information science, and one had a phd in a related field. thematic analysis of the eight interviews uncovered several broader themes, including changes to staffing structures during the pandemic, the adoption of new technologies, the impact and experiences of using new technologies, the implementation of remote work, and the changes in services during the pandemic. prior to the pandemic, library staff were involved in a variety of teams at various levels, including within the library, among consortia, and even throughout community organizations. these teams continued to meet during the pandemic using various technologies. the covid-19 pandemic created the need to create new teams quickly to address safety protocols and the continuation of library services. many libraries closed their physical locations during the pandemic with many in-person services offered digitally or re-envisioned to comply with current safety protocols. technology adoption varied depending on the size of the library and the ease of use and familiarity with certain tools. many teams adopted video conferencing tools to continue to meet as a group, platforms to share documents, and messaging apps to enhance group communication. some libraries applied for grant funding to expand their technology access or digital services. though there was widespread adoption of technology, libraries prioritized communication among their teams, which resulted in less emphasis on technology security and privacy.   benefits of the adoption of new technologies were enhanced participation and accessibility at team meetings and virtual programming, expansion in skill development and training for staff, and the ability to continue working while adhering to covid-19 safety protocols. teams were able to meet virtually across multiple locations, and the online format of meetings allowed for more equitable participation among attendees. travel costs and travel time were eliminated, and the online meeting chat feature ensured that typically quieter participants could engage with the conversation. additionally, libraries were able to host larger library programs online by eliminating travel cost and time. the covid-19 pandemic created the need for rapid team formation in order to address safety protocols, ensure continuity of library services, and support the use of digital services. challenges included a steep learning curve for some staff using new technologies, a lack of access to adequate devices or broadband for library workers living in rural areas, the development of virtual meeting fatigue leading to disengagement, and a lack of a sense of community. though many libraries embraced remote work, front line staff were still expected to return to work physically. while remote work helped in some ways, it also presented a challenge for staff who did not have quiet spaces to focus, appropriate work setups at home, or consistent access to good internet.   conclusion – the covid-19 pandemic forced many libraries to adapt quickly to a new environment of digital and hybrid teamwork while still attempting to maintain community-oriented services. library leaders explored how flexible working environments could enhance employee engagement by using technology and addressed many challenges in adopting new technologies and making them accessible for staff. as well, library leaders addressed staff morale by extending grace to staff members, providing professional development in new areas of librarianship, and creating scheduling shifts for remote and in-person work. many organizations returned to physical meetings when it became safe to do so but also continued to offer hybrid work options and use some technology adopted during the pandemic. the pandemic required flexible and creative problem solving. the experience enabled library leaders to identify the benefits and challenges of adopting new technology, maintaining service continuity, emphasizing the need to think about information security, and adapting and creating teams as needed.   commentary   the authors address a gap in the literature on how public libraries adapted during the covid-19 pandemic, specifically through forming virtual teams, adopting digital technologies, and revisioning library services. similarly, chamakiotis and colleagues (2021) conducted a state-of-the-art literature review to examine how virtual teams changed before and during the pandemic and highlight the impact that the pandemic had on team reconfiguration and characteristics. chamakiotis et al. (2021) specifically outline how leaders of virtual teams can adapt for team success by establishing work-life guardrails for the team, adapting former virtual team practices, keeping team well-being at top of mind, creating new methods of team engagement, and fostering creativity and innovation. research by efimov et al. (2022) uncovers a positive correlation between virtual leadership styles and employee mental health outcomes, job satisfaction, and perceptions of isolation. schlaegel et al. (2023) investigate how geographical covid-19 severity and individual self-regulation impacts individual performance further impacting engagement and success of virtual teams. a more specific case example explores how public and academic libraries transitioned partnerships and teamwork through the pandemic to continue community outreach initiatives and build health literacy at a critical moment (swanberg et al., 2022).   i used an adapted version of russell and gregory’s (2003) qualitative research appraisal questions to evaluate this article (suarez, 2010). the authors present three research questions that investigate how teamwork evolved before and during the covid-19 pandemic, and they explore the benefits and challenges of tools on team development and sustainability. the use of qualitative, semi-structured interviews was appropriate to address the research questions, though the sampling method could be clearer. the authors note that they sent invitations to librarians in the midwestern region, but they did not provide more details about how they initially identified librarians, or why they explicitly selected only library leaders to be interviewed as opposed to other library managers who have decision-making power over team formation and technology adoption. it was also unclear whether the authors used other tools for data collection and analysis aside from hand-written notes and zoom transcripts. however, the authors’ description of data analysis seems appropriate.   in their findings, the authors provide an in-depth description of the impacts, benefits, and challenges of using new technologies on individual work as well as teamwork. they explore a wide variety of experiences and highlight the practical, financial, geographical, and learning challenges of virtual teamwork but also underscore the benefits such as increased accessibility and engagement with team meetings. the authors describe the types of teams that existed prior to the pandemic and note that they did not change during the pandemic or return to their initial team structures after returning to in-person or hybrid work. the authors also indicate how teams used technology to enhance and facilitate meetings, informal communications, and document sharing. the findings of this article illuminate pathways for public libraries to nimbly respond to emergencies and uncover the overall benefit of virtual teams for meeting accessibility and equity in participation, ensuring library service continuity. results also underscore the need to improve staff preparation for remote working environments, especially for those with limited workspace setups and insufficient internet access. however, there was no discussion of whether team makeup, team leadership, expectations, or dynamics changed during the shift to virtual work. in their implications, the authors briefly introduce the concept of agile project management, which is outside of the scope of their original research objectives. moreover, there was little exploration of how library leadership guided and facilitated the creation of teams and the impact leadership had on team success. exploring these questions could help illustrate a fuller picture of the challenges and successes of teamwork in virtual settings and if leadership approaches need to adapt based on the setting.   this study highlights the capacity of library teams to adapt quickly to emergency situations and shows how teams responded to technological challenges as they arose. the authors additionally note some benefits of technology practices that have carried over into hybrid and in-person work environments. study responses are limited to high-level leadership positions in a specific geographic region, which does not fully capture the scope of the experience of public libraries and their staff. the specific timeframe during which the authors conducted these interviews is unclear; this is key information given the rapidly changing context of the covid-19 pandemic. successful and unsuccessful virtual team dynamics and expectations could have been explored further to identify best practices for creating and sustaining virtual and hybrid teams. future research should focus on the perspectives of public library staff who can share their direct experiences working on virtual teams to paint a more robust picture of the experiences of virtual teamwork.   references   chamakiotis, p., panteli, n., & davison, r. m. (2021). reimagining e-leadership for reconfigured virtual teams due to covid-19. international journal of information management, 60, 102381. https://doi.org/10.1016/j.ijinfomgt.2021.102381   efimov, i., rohwer, e., harth, v., & mache, s. (2022). virtual leadership in relation to employees’ mental health, job satisfaction and perceptions of isolation: a scoping review. frontiers in psychology, 13, 960955. https://doi.org/10.3389/fpsyg.2022.960955    schlaegel, c., gunkel, m., & taras, v. (2023). covid‐19 and individual performance in global virtual teams: the role of self‐regulation and individual cultural value orientations. journal of organizational behavior, 44(1), 102–131. https://doi.org/10.1002/job.2671    singh, k., & bossaller, j. s. (2022). it’s just not the same: virtual teamwork in public libraries. journal of library administration, 62(4), 512–534. https://doi.org/10.1080/01930826.2022.2057130   suarez, d. (2010). evaluating qualitative research studies for evidence based library and information practice. evidence based library and information practice, 5(2), 75–85. https://doi.org/10.18438/b8v90m   swanberg, s. m., bulgarelli, n., jayakumar, m., look, e., shubitowski, t. b., wedemeyer, r., yuen, e. w., & lucia, v. c. (2022). a health education outreach partnership between an academic medical library and public library: lessons learned before and during a pandemic. journal of the medical library association, 110(2), 212–221. https://doi.org/10.5195/jmla.2022.1413     evidence based library and information practice editorial responsibilities      2018. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     editor-in-chief: lorie kloda   associate editor (evidence summaries): carol perryman   associate editor (research articles): ann medaille, lisl zach   associate editor (classics & reviews): christina wissinger   associate editor (commentaries, using evidence in practice): lorie kloda   communications officer (news): tatiana bryant   production editor: michelle dunaway, rachel hinrichs   editorial intern: rachel hinrichs, kimberly mackenzie   editorial advisors: lindsay alcock, alison brettle, jonathan eldredge, denise koufogiannakis   copyeditors: richard hayman (lead copyeditor), georgianne bordner, julie evener, heather healy, julie james, alison moore, jane morgan-daniel, elizabeth stregger, elaina vitale   indexing support: kate shore evidence summary   seed libraries can be a promising but challenging way to support community engagement and social innovation in public libraries   a review of: peekhaus, w. (2018). seed libraries: sowing the seeds for community and public library resilience. library quarterly, 88(3), 271-285. https://doi.org/10.1086/697706   reviewed by: abigail alty master of information studies, candidate 2023 university of ottawa ottawa, ontario, canada email: aalty093@uottawa.ca   received: 28 mar. 2021                                                                 accepted:  19 july 2021      2021 alty. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29954     abstract   objective – to describe and investigate the establishment, operation, function, purpose, and benefit of seed libraries within public libraries and local communities.   design – exploratory study.   setting – public seed libraries in arizona, california, minnesota, new york, pennsylvania, and wisconsin.   subjects – 10 librarians actively involved in creating or maintaining seed libraries.   methods – 60-75 minute interviews, primarily over the phone, with subjects selected by means of purposive sampling.   main results – according to the participants interviewed, starting and operating a seed library requires front-end effort from the “host” library, active participation by a dedicated librarian and community members, as well as ongoing funding, usually on an annual basis (estimated by one participant to be $2,500/year, mostly for the purchase of seeds). participant descriptions of their seed library operations differed, but most had a dedicated seed librarian. participants noted that primary activities included deciding what seeds to put in the collection, arrangement of the seed collection, development of checkout and return procedures, and ongoing education. several participants noted that such operational work was seasonal and not steady. none of the libraries included in this study had enough seeds donated to sustain their collections, but rather they relied on purchasing seeds in bulk or asking for donations from seed companies. cataloging procedures varied in terms of complexity, and participants from one library system reported the use of a seed library cataloging template as being helpful. all participants noted they gave patrons containers to return seeds. while educating patrons in formal sessions is often difficult for reasons such as resource limitations, the interview informants agreed that seed libraries fit into the missions of public libraries by furthering information sharing, access to resources, and knowledge development. conclusion – seed libraries are an active service that assist public libraries in responding to social challenges and in engaging with their local communities as a type of knowledge commons. seed libraries align with public libraries’ shift in priorities from increasing physical collections to enriching lives by providing knowledge and tools to support food autonomy, self-sufficiency, civic engagement, and community education. these libraries are a novel service that engage and attract patrons and support libraries’ positions as community hubs.   commentary   this study adds to the limited research that exists about seed libraries (dean & mezick, 2020; ingalls, 2019). the author’s literature review highlighted that the current information on seed libraries is predominantly non-academic and limited to “brief, one-off accounts of various seed library projects” (p. 272). additional research is needed to understand the larger impacts of seed libraries on communities and the long-term sustainability of seed libraries.   this exploratory and descriptive qualitative study demonstrated various strengths when evaluated with the critical review form for qualitative studies (letts et al., 2007). results were presented clearly, and the study provided a detailed and balanced overview of how participants’ seed libraries were implemented and managed. the study design was appropriate, and a telephone interview format was successful in achieving the desired results of better understanding how seed libraries were established and operated. the author offered valuable insights for seed library practitioners.       the author indicated that a purposive sampling strategy was used. the process to select the ten key informants was not outlined, other than to indicate that seed library staff were contacted via email or telephone and asked to participate in the study. it was also unclear in the study which, if any, ethics procedures were followed. the author discussed the promise of seed libraries but did not support the argument with data from the study results. future studies might include an interview guide, quotes from interviewees, or explanations of how interview transcripts were reviewed. additionally, the author could have provided supplemental details about the study setting and sample to assist in determining transferability to other settings.   as preliminary research on seed libraries, the article is significant to library and information practice. the findings of this research would be useful to an organization interested in starting a seed library, as the study explores different systems and their benefits. the research could also be valuable for librarians currently operating seed libraries who may want to improve on existing processes or practices and can learn from the experiences of the study’s participants. alternatively, librarians may find this study beneficial when seeking information about the implementation of non-traditional collections.   references   dean, h. m., & mezick, j. (2020). an examination of seed libraries across two academic institutions. urban library journal, 26(1), article 3. https://academicworks.cuny.edu/ulj/vol26/iss1/3   ingalls, d. (2019). from seed to harvest: growing the macdonald campus seed library. new review of academic librarianship, 27(1), 97-112. https://doi.org/10.1080/13614533.2019.1699838   letts, l., wilkins, s., law, m., stewart, d., bosch, j., & westmorland, m. (2007). critical review form – qualitative studies (version 2.0). mcmaster university. https://healthsci.mcmaster.ca/docs/librariesprovider130/default-document-library/guidelines-for-critical-review-form-qualitative-studies-english.pdf       <html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/tr/rec-html40"> <head> <meta http-equiv=content-type content="text/html; charset=unicode"> <meta name=progid content=word.document> <meta name=generator content="microsoft word 15"> <meta name=originator content="microsoft word 15"> <link rel=file-list href="es_williamson_29525_files/filelist.xml"> <link rel=edit-time-data href="es_williamson_29525_files/editdata.mso"> <!--[if !mso]> <style> v\:* {behavior:url(#default#vml);} o\:* {behavior:url(#default#vml);} w\:* {behavior:url(#default#vml);} .shape {behavior:url(#default#vml);} </style> <![endif]--><!--[if gte mso 9]><xml> <o:documentproperties> <o:author>mallan, katrine</o:author> <o:template>normal</o:template> <o:lastauthor>hinrichs, rachel</o:lastauthor> <o:revision>4</o:revision> <o:totaltime>3</o:totaltime> <o:created>2019-03-14t14:34:00z</o:created> <o:lastsaved>2019-03-14t15:03:00z</o:lastsaved> <o:pages>1</o:pages> <o:words>1660</o:words> <o:characters>9462</o:characters> <o:company>microsoft</o:company> <o:lines>78</o:lines> <o:paragraphs>22</o:paragraphs> <o:characterswithspaces>11100</o:characterswithspaces> <o:version>16.00</o:version> </o:documentproperties> <o:officedocumentsettings> <o:allowpng/> </o:officedocumentsettings> </xml><![endif]--> <link rel=datastoreitem href="es_williamson_29525_files/item0009.xml" target="es_williamson_29525_files/props010.xml"> <link rel=themedata href="es_williamson_29525_files/themedata.thmx"> <link rel=colorschememapping href="es_williamson_29525_files/colorschememapping.xml"> <!--[if gte mso 9]><xml> <w:worddocument> <w:trackmoves>false</w:trackmoves> <w:trackformatting/> <w:hyphenationzone>21</w:hyphenationzone> <w:punctuationkerning/> <w:validateagainstschemas/> <w:saveifxmlinvalid>false</w:saveifxmlinvalid> <w:ignoremixedcontent>false</w:ignoremixedcontent> <w:alwaysshowplaceholdertext>false</w:alwaysshowplaceholdertext> <w:donotpromoteqf/> <w:lidthemeother>de</w:lidthemeother> <w:lidthemeasian>x-none</w:lidthemeasian> <w:lidthemecomplexscript>x-none</w:lidthemecomplexscript> <w:compatibility> <w:breakwrappedtables/> <w:snaptogridincell/> <w:wraptextwithpunct/> <w:useasianbreakrules/> <w:dontgrowautofit/> <w:splitpgbreakandparamark/> <w:enableopentypekerning/> <w:dontflipmirrorindents/> <w:overridetablestylehps/> </w:compatibility> <w:browserlevel>microsoftinternetexplorer4</w:browserlevel> <m:mathpr> <m:mathfont m:val="cambria math"/> <m:brkbin m:val="before"/> <m:brkbinsub m:val="&#45;-"/> <m:smallfrac m:val="off"/> <m:dispdef/> <m:lmargin m:val="0"/> <m:rmargin m:val="0"/> <m:defjc m:val="centergroup"/> <m:wrapindent m:val="1440"/> <m:intlim m:val="subsup"/> <m:narylim m:val="undovr"/> </m:mathpr></w:worddocument> </xml><![endif]--><!--[if gte mso 9]><xml> <w:latentstyles deflockedstate="false" defunhidewhenused="false" defsemihidden="false" defqformat="false" defpriority="99" latentstylecount="371"> <w:lsdexception locked="false" priority="0" qformat="true" name="normal"/> <w:lsdexception locked="false" priority="9" qformat="true" name="heading 1"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 2"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 3"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 4"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 5"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 6"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 7"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 8"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 9"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 6"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 7"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 8"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 9"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 1"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 2"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 3"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 4"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 5"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 6"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 7"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 8"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 9"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="normal indent"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="footnote text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="annotation text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="header"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="footer"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index heading"/> <w:lsdexception locked="false" priority="35" semihidden="true" unhidewhenused="true" qformat="true" name="caption"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table of figures"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="envelope address"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="envelope return"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="footnote reference"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="annotation reference"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="line number"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="page number"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="endnote reference"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="endnote text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table of authorities"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="macro"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="toa heading"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number 5"/> <w:lsdexception locked="false" priority="10" qformat="true" name="title"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="closing"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="signature"/> <w:lsdexception locked="false" priority="1" semihidden="true" unhidewhenused="true" name="default paragraph font"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text indent"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="message header"/> <w:lsdexception locked="false" priority="11" qformat="true" name="subtitle"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="salutation"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="date"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text first indent"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text first indent 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="note heading"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text indent 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text indent 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="block text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="hyperlink"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="followedhyperlink"/> <w:lsdexception locked="false" priority="22" qformat="true" name="strong"/> <w:lsdexception locked="false" priority="20" qformat="true" name="emphasis"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="document map"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="plain text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="e-mail signature"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html top of form"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html bottom of form"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="normal (web)"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html acronym"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html address"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html cite"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html code"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html definition"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html keyboard"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html preformatted"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html sample"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html typewriter"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html variable"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="normal table"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="annotation subject"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="no list"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="outline list 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="outline list 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="outline list 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table simple 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table simple 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table simple 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table classic 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table classic 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table classic 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table classic 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table colorful 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table colorful 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table colorful 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 6"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 7"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 8"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 6"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 7"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 8"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table 3d effects 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table 3d effects 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table 3d effects 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table contemporary"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table elegant"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table professional"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table subtle 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table subtle 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table web 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table web 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table web 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="balloon text"/> <w:lsdexception locked="false" priority="59" name="table grid"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table theme"/> <w:lsdexception locked="false" semihidden="true" name="placeholder text"/> <w:lsdexception locked="false" priority="1" qformat="true" name="no spacing"/> <w:lsdexception locked="false" priority="60" name="light shading"/> <w:lsdexception locked="false" priority="61" name="light list"/> <w:lsdexception locked="false" priority="62" name="light grid"/> <w:lsdexception locked="false" priority="63" name="medium shading 1"/> <w:lsdexception locked="false" priority="64" name="medium shading 2"/> <w:lsdexception locked="false" priority="65" name="medium list 1"/> <w:lsdexception locked="false" priority="66" name="medium list 2"/> <w:lsdexception locked="false" priority="67" name="medium grid 1"/> <w:lsdexception locked="false" priority="68" name="medium grid 2"/> <w:lsdexception locked="false" priority="69" name="medium grid 3"/> <w:lsdexception locked="false" priority="70" name="dark list"/> <w:lsdexception locked="false" priority="71" name="colorful shading"/> <w:lsdexception locked="false" priority="72" name="colorful list"/> <w:lsdexception locked="false" priority="73" name="colorful grid"/> <w:lsdexception locked="false" priority="60" name="light shading accent 1"/> <w:lsdexception locked="false" priority="61" name="light list accent 1"/> <w:lsdexception locked="false" priority="62" name="light grid accent 1"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 1"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 1"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 1"/> <w:lsdexception locked="false" semihidden="true" name="revision"/> <w:lsdexception locked="false" priority="34" qformat="true" name="list paragraph"/> <w:lsdexception locked="false" priority="29" qformat="true" name="quote"/> <w:lsdexception locked="false" priority="30" qformat="true" name="intense quote"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 1"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 1"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 1"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 1"/> <w:lsdexception locked="false" priority="70" name="dark list accent 1"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 1"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 1"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 1"/> <w:lsdexception locked="false" priority="60" name="light shading accent 2"/> <w:lsdexception locked="false" priority="61" name="light list accent 2"/> <w:lsdexception locked="false" priority="62" name="light grid accent 2"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 2"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 2"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 2"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 2"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 2"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 2"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 2"/> <w:lsdexception locked="false" priority="70" name="dark list accent 2"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 2"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 2"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 2"/> <w:lsdexception locked="false" priority="60" name="light shading accent 3"/> <w:lsdexception locked="false" priority="61" name="light list accent 3"/> <w:lsdexception locked="false" priority="62" name="light grid accent 3"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 3"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 3"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 3"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 3"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 3"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 3"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 3"/> <w:lsdexception locked="false" priority="70" name="dark list accent 3"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 3"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 3"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 3"/> <w:lsdexception locked="false" priority="60" name="light shading accent 4"/> <w:lsdexception locked="false" priority="61" name="light list accent 4"/> <w:lsdexception locked="false" priority="62" name="light grid accent 4"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 4"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 4"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 4"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 4"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 4"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 4"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 4"/> <w:lsdexception locked="false" priority="70" name="dark list accent 4"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 4"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 4"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 4"/> <w:lsdexception locked="false" priority="60" name="light shading accent 5"/> <w:lsdexception locked="false" priority="61" name="light list accent 5"/> <w:lsdexception locked="false" priority="62" name="light grid accent 5"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 5"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 5"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 5"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 5"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 5"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 5"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 5"/> <w:lsdexception locked="false" priority="70" name="dark list accent 5"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 5"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 5"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 5"/> <w:lsdexception locked="false" priority="60" name="light shading accent 6"/> <w:lsdexception locked="false" priority="61" name="light list accent 6"/> <w:lsdexception locked="false" priority="62" name="light grid accent 6"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 6"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 6"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 6"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 6"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 6"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 6"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 6"/> <w:lsdexception locked="false" priority="70" name="dark list accent 6"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 6"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 6"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 6"/> <w:lsdexception locked="false" priority="19" qformat="true" name="subtle emphasis"/> <w:lsdexception locked="false" priority="21" qformat="true" name="intense emphasis"/> <w:lsdexception locked="false" priority="31" qformat="true" name="subtle reference"/> <w:lsdexception locked="false" priority="32" qformat="true" name="intense reference"/> <w:lsdexception locked="false" priority="33" qformat="true" name="book title"/> <w:lsdexception locked="false" priority="37" semihidden="true" unhidewhenused="true" name="bibliography"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" qformat="true" name="toc heading"/> <w:lsdexception locked="false" priority="41" name="plain table 1"/> <w:lsdexception locked="false" priority="42" name="plain table 2"/> <w:lsdexception locked="false" priority="43" name="plain table 3"/> <w:lsdexception locked="false" priority="44" name="plain table 4"/> <w:lsdexception locked="false" priority="45" name="plain table 5"/> <w:lsdexception locked="false" priority="40" name="grid table light"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light"/> <w:lsdexception locked="false" priority="47" name="grid table 2"/> <w:lsdexception locked="false" priority="48" name="grid table 3"/> <w:lsdexception locked="false" priority="49" name="grid table 4"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 1"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 1"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 1"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 1"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 1"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 1"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 1"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 2"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 2"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 2"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 2"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 2"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 2"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 2"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 3"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 3"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 3"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 3"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 3"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 3"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 3"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 4"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 4"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 4"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 4"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 4"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 4"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 4"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 5"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 5"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 5"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 5"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 5"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 5"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 5"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 6"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 6"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 6"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 6"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 6"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 6"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 6"/> <w:lsdexception locked="false" priority="46" name="list table 1 light"/> <w:lsdexception locked="false" priority="47" name="list table 2"/> <w:lsdexception locked="false" priority="48" name="list table 3"/> <w:lsdexception locked="false" priority="49" name="list table 4"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 1"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 1"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 1"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 1"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 1"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 1"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 1"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 2"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 2"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 2"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 2"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 2"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 2"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 2"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 3"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 3"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 3"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 3"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 3"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 3"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 3"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 4"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 4"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 4"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 4"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 4"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 4"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 4"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 5"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 5"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 5"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 5"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 5"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 5"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 5"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 6"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 6"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 6"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 6"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 6"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 6"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 6"/> </w:latentstyles> </xml><![endif]--> <style> <!- /* font definitions */ @font-face {font-family:"ms mincho"; panose-1:2 2 6 9 4 2 5 8 3 4; mso-font-alt:"ｍｓ 明朝"; mso-font-charset:128; mso-generic-font-family:roman; mso-font-pitch:fixed; mso-font-signature:1 134676480 16 0 131072 0;} @font-face {font-family:latha; panose-1:2 0 4 0 0 0 0 0 0 0; mso-font-charset:1; mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:262144 0 0 0 0 0;} @font-face {font-family:"cambria math"; panose-1:2 4 5 3 5 4 6 3 2 4; mso-font-charset:0; mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:-536870145 1107305727 0 0 415 0;} @font-face {font-family:calibri; panose-1:2 15 5 2 2 2 4 3 2 4; mso-font-charset:0; mso-generic-font-family:swiss; mso-font-pitch:variable; mso-font-signature:-536859905 -1073732485 9 0 511 0;} @font-face {font-family:tahoma; panose-1:2 11 6 4 3 5 4 4 2 4; mso-font-charset:0; mso-generic-font-family:swiss; mso-font-pitch:variable; mso-font-signature:-520081665 -1073717157 41 0 66047 0;} @font-face {font-family:"palatino linotype"; panose-1:2 4 5 2 5 5 5 3 3 4; mso-font-charset:0; mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:-536870265 1073741843 0 0 415 0;} @font-face {font-family:"\@ms mincho"; panose-1:2 2 6 9 4 2 5 8 3 4; mso-font-charset:128; mso-generic-font-family:roman; mso-font-format:other; mso-font-pitch:fixed; mso-font-signature:1 134676480 16 0 131072 0;} /* style definitions */ p.msonormal, li.msonormal, div.msonormal {mso-style-unhide:no; mso-style-qformat:yes; mso-style-parent:""; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca;} p.msocommenttext, li.msocommenttext, div.msocommenttext {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"comment text char"; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca;} p.msoheader, li.msoheader, div.msoheader {mso-style-priority:99; mso-style-link:"header char"; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; tab-stops:center 3.15in right 6.3in; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca;} p.msofooter, li.msofooter, div.msofooter {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"footer char"; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; tab-stops:center 3.15in right 6.3in; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca;} span.msocommentreference {mso-style-noshow:yes; mso-style-priority:99; mso-ansi-font-size:8.0pt; mso-bidi-font-size:8.0pt;} p.msobodytext, li.msobodytext, div.msobodytext {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"body text char"; margin:0in; margin-bottom:.0001pt; mso-pagination:none; mso-layout-grid-align:none; text-autospace:none; font-size:12.0pt; font-family:"arial",sans-serif; mso-fareast-font-family:calibri; mso-bidi-font-family:latha;} a:link, span.msohyperlink {mso-style-noshow:yes; mso-style-priority:99; color:blue; text-decoration:underline; text-underline:single;} a:visited, span.msohyperlinkfollowed {mso-style-noshow:yes; mso-style-priority:99; color:purple; mso-themecolor:followedhyperlink; text-decoration:underline; text-underline:single;} p.msocommentsubject, li.msocommentsubject, div.msocommentsubject {mso-style-noshow:yes; mso-style-priority:99; mso-style-parent:"comment text"; mso-style-link:"comment subject char"; mso-style-next:"comment text"; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca; font-weight:bold;} p.msoacetate, li.msoacetate, div.msoacetate {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"balloon text char"; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:8.0pt; font-family:"tahoma",sans-serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca;} p.msobibliography, li.msobibliography, div.msobibliography {mso-style-noshow:yes; mso-style-priority:37; mso-style-next:normal; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca;} p.msonormal0, li.msonormal0, div.msonormal0 {mso-style-name:msonormal; mso-style-unhide:no; mso-margin-top-alt:auto; margin-right:0in; mso-margin-bottom-alt:auto; margin-left:0in; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast;} span.commenttextchar {mso-style-name:"comment text char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"comment text"; mso-ansi-font-size:10.0pt; mso-bidi-font-size:10.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; mso-ansi-language:en-ca;} span.headerchar {mso-style-name:"header char"; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:header; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; mso-ansi-language:en-ca;} span.footerchar {mso-style-name:"footer char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:footer; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; mso-ansi-language:en-ca;} span.bodytextchar {mso-style-name:"body text char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"body text"; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"arial",sans-serif; mso-ascii-font-family:arial; mso-fareast-font-family:calibri; mso-hansi-font-family:arial; mso-bidi-font-family:latha; mso-ansi-language:en-us;} span.commentsubjectchar {mso-style-name:"comment subject char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-parent:"comment text char"; mso-style-link:"comment subject"; mso-ansi-font-size:10.0pt; mso-bidi-font-size:10.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; mso-ansi-language:en-ca; font-weight:bold;} span.balloontextchar {mso-style-name:"balloon text char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"balloon text"; mso-ansi-font-size:8.0pt; mso-bidi-font-size:8.0pt; font-family:"tahoma",sans-serif; mso-ascii-font-family:tahoma; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:tahoma; mso-bidi-font-family:tahoma; mso-ansi-language:en-ca;} p.default, li.default, div.default {mso-style-name:default; mso-style-unhide:no; mso-style-parent:""; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; mso-layout-grid-align:none; text-autospace:none; font-size:12.0pt; font-family:"palatino linotype",serif; mso-fareast-font-family:"ms mincho"; mso-bidi-font-family:"palatino linotype"; color:black; mso-ansi-language:en-ca; mso-fareast-language:ja;} span.normalchar1 {mso-style-name:normal__char1; mso-style-priority:99; mso-style-unhide:no; mso-style-parent:""; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman";} span.shorttext {mso-style-name:short_text; mso-style-priority:99; mso-style-unhide:no; mso-style-parent:""; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman";} span.heading2main {mso-style-name:"heading 2 main"; mso-style-unhide:no; mso-style-parent:""; mso-ansi-font-size:12.0pt; font-family:"arial",sans-serif; mso-ascii-font-family:arial; mso-fareast-font-family:arial; mso-hansi-font-family:arial; mso-bidi-font-family:arial; color:black; font-weight:bold; mso-bidi-font-weight:normal; font-style:normal; text-underline:black;} span.mention1 {mso-style-name:mention1; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; color:#2b579a; background:#e6e6e6;} span.unresolvedmention {mso-style-name:"unresolved mention"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; color:#605e5c; background:#e1dfdd;} .msochpdefault {mso-style-type:export-only; mso-default-props:yes; font-size:10.0pt; mso-ansi-font-size:10.0pt; mso-bidi-font-size:10.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:calibri; mso-fareast-theme-font:minor-latin; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; mso-ansi-language:de;} /* page definitions */ @page {mso-footnote-separator:url("es_williamson_29525_files/header.htm") fs; mso-footnote-continuation-separator:url("es_williamson_29525_files/header.htm") fcs; mso-endnote-separator:url("es_williamson_29525_files/header.htm") es; mso-endnote-continuation-separator:url("es_williamson_29525_files/header.htm") ecs;} @page wordsection1 {size:595.3pt 841.9pt; margin:70.85pt 70.85pt 56.7pt 70.85pt; mso-header-margin:35.4pt; mso-footer-margin:35.4pt; mso-columns:2 even 35.4pt; mso-header:url("es_williamson_29525_files/header.htm") h1; mso-paper-source:0;} div.wordsection1 {page:wordsection1;} --> </style> <!--[if gte mso 10]> <style> /* style definitions */ table.msonormaltable {mso-style-name:"table normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; mso-ansi-language:de;} </style> <![endif]--><!--[if gte mso 9]><xml> <o:shapedefaults v:ext="edit" spidmax="1026"/> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext="edit"> <o:idmap v:ext="edit" data="1"/> </o:shapelayout></xml><![endif]--> </head> <body lang=en-us link=blue vlink=purple style='tab-interval:35.4pt'> <div class=wordsection1> <p class=msonormal><i style='mso-bidi-font-style:normal'><span lang=en-ca style='font-family:"palatino linotype",serif'>evidence summary<o:p></o:p></span></i></p> <p class=msonormal><i style='mso-bidi-font-style:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></i></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:11.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>librarians’ reported systematic review completion time ranges between 2 and 219 total hours with most variance due to information processing and instruction<o:p></o:p></span></b></p> <p class=msonormal><span lang=en-ca style='font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>a review of:<o:p></o:p></span></b></p> <p class=msonormal style='margin-left:35.3pt;text-indent:-35.3pt'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>bullers, k., howard, a. m., hanson, a., kearns, w. d., orriola, j. j., polo, r. l., &amp; sakmar, k. a. (2018). it takes longer than you think: librarian time spent on systematic review tasks. <i style='mso-bidi-font-style: normal'>journal of the medical library association</i>,<i style='mso-bidi-font-style: normal'> 106</i>(2), 198-207. <span class=msohyperlink><a href="https://doi.org/10.5195/jmla.2018.323">https://doi.org/10.5195/jmla.2018.323</a></span> <o:p></o:p></span></p> <p class=msonormal style='margin-left:35.3pt;text-indent:-35.3pt'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>reviewed by:<o:p></o:p></span></b></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>peace ossom williamson<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>director for research data services <o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>university of texas at arlington libraries<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>arlington, texas, united states of america<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>email: <span class=msohyperlink><a href="mailto:peace@uta.edu">peace@uta.edu</a></span><b style='mso-bidi-font-weight: normal'><o:p></o:p></b></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>received: </span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>16 nov. 2018</span><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><span style='mso-tab-count:5'>                                                                </span>accepted: </span></b><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>7 jan. 2019<b style='mso-bidi-font-weight:normal'><o:p></o:p></b></span></p> <p class=msonormal><span lang=en-ca style='font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:9.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:arial;mso-ansi-language:en-us;mso-no-proof:yes'><!--[if gte vml 1]><v:shapetype id="_x0000_t75" coordsize="21600,21600" o:spt="75" o:preferrelative="t" path="m@4@5l@4@11@9@11@9@5xe" filled="f" stroked="f"> <v:stroke joinstyle="miter"/> <v:formulas> <v:f eqn="if linedrawn pixellinewidth 0"/> <v:f eqn="sum @0 1 0"/> <v:f eqn="sum 0 0 @1"/> <v:f eqn="prod @2 1 2"/> <v:f eqn="prod @3 21600 pixelwidth"/> <v:f eqn="prod @3 21600 pixelheight"/> <v:f eqn="sum @0 0 1"/> <v:f eqn="prod @6 1 2"/> <v:f eqn="prod @7 21600 pixelwidth"/> <v:f eqn="sum @8 21600 0"/> <v:f eqn="prod @7 21600 pixelheight"/> <v:f eqn="sum @10 21600 0"/> </v:formulas> <v:path o:extrusionok="f" gradientshapeok="t" o:connecttype="rect"/> <o:lock v:ext="edit" aspectratio="t"/> </v:shapetype><v:shape id="picture_x0020_4" o:spid="_x0000_i1025" type="#_x0000_t75" alt="cc-ca_logo_xl" style='width:7.5pt;height:7.5pt;visibility:visible; mso-wrap-style:square'> <v:imagedata src="es_williamson_29525_files/image003.png" o:title="cc-ca_logo_xl" cropright="49941f"/> </v:shape><![endif]--><![if !vml]><img border=0 width=10 height=10 src="es_williamson_29525_files/image001.png" alt="cc-ca_logo_xl" v:shapes="picture_x0020_4"><![endif]></span><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:arial'><span style='mso-spacerun:yes'> </span>2019 </span><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif'>williamson. this is an open access article distributed under the terms of the creative commons</span><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"cambria math"'>‐</span><span lang=en-ca style='font-size: 9.0pt;font-family:"palatino linotype",serif'>attribution</span><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"cambria math"'>‐</span><span lang=en-ca style='font-size: 9.0pt;font-family:"palatino linotype",serif'>noncommercial</span><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"cambria math"'>‐</span><span lang=en-ca style='font-size: 9.0pt;font-family:"palatino linotype",serif'>share alike license 4.0 international (<span class=msohyperlink><a href="http://creativecommons.org/licenses/by-nc-sa/4.0/">http://creativecommons.org/licenses/by-nc-sa/4.0/</a></span>), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.<o:p></o:p></span></p> <div style='mso-element:para-border-div;border:none;border-bottom:solid black 1.0pt; mso-border-bottom-alt:solid black .25pt;padding:0in 0in 1.0pt 0in'> <p class=msobodytext style='border:none;mso-border-bottom-alt:solid black .25pt; padding:0in;mso-padding-alt:0in 0in 1.0pt 0in'><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-ca'><o:p>&nbsp;</o:p></span></p> <p class=msobodytext style='border:none;mso-border-bottom-alt:solid black .25pt; padding:0in;mso-padding-alt:0in 0in 1.0pt 0in'><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-ca'><o:p>&nbsp;</o:p></span></p> <p class=msobodytext style='border:none;mso-border-bottom-alt:solid black .25pt; padding:0in;mso-padding-alt:0in 0in 1.0pt 0in'><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif;color:blue; mso-ansi-language:en-ca'>doi</span><span lang=en-ca style='font-size:9.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-ca'>: 10.18438/eblip29525<o:p></o:p></span></p> <p class=msobodytext style='border:none;mso-border-bottom-alt:solid black .25pt; padding:0in;mso-padding-alt:0in 0in 1.0pt 0in'><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-ca'><o:p>&nbsp;</o:p></span></p> </div> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>abstract<o:p></o:p></span></b></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>objective</span></b><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'> – to investigate how long it takes for medical librarians to complete steps toward completion of a systematic review and to determine if the time differs based on factors including years of experience as a medical librarian and experience completing systematic reviews. <o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>design</span></b><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'> – survey research as a questionnaire disseminated via email distribution lists. <o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>setting </span></b><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>– at institutions that are members of the association of academic health sciences libraries (aahsl) and librarians at association of american medical colleges (aamc) or american osteopathic association (aoa) member institutions.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>subjects</span></b><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'> – librarians of member institutions who have worked on systematic reviews.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>methods</span></b><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'> – on december 11, 2015, aahsl library directors and librarian members of aamc and aoa were sent the survey and the recommendation to forward the survey to librarians on staff who have worked on systematic reviews. reminders were sent on december 17, 2015, and the survey closed for participation on january 7, 2016. participants who had worked on a systematic review within the past five years were asked to indicate experience by the number of systematic reviews completed, years of experience as a medical librarian, and how much time was spent, in hours, on the following: initial consultations/meetings; developing and testing the initial search strategy; translating the strategy for other databases; documenting the process; delivering the search results; writing their part of the manuscript; other tasks they could identify; and any instruction (i.e., training they provided to team members necessary for completion of the systematic review). participants also further broke down the amount of their time searching, by percentage of time, in various resources, including literature indexes/databases, included studies’ references, trial registers, grey literature, and hand searching. participants were also given space to add additional comments. the researchers reported summary statistics for phase one and, for phase two, excluded outliers and performed exploratory factor analysis, beginning with principal components analysis (pca), followed by a varimax rotation, to determine if there was a relationship between the time on tasks and experience.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>main results </span></b><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>– of the 185 completed responses, 105 were analyzed for phase one because 80 responses were excluded due to missing data or no recent experience with a systematic review. the average respondent had between 1 and 6 years of experience: 1-3 years in librarianship (49.5%) and 4-6 years (23.8%). the time reported for completion of all tasks ranged from 2 to 219 hours with a mean of 30.7 hours. most of the variance (61.6%) was caused by “information processing” and “interpersonal instruction/training” components. search strategy development and testing had the highest average time at 8.4 hours. within that category, databases accounted for 78.7% of time searching, followed by other searching methods. for remaining systematic review tasks, their averages were as follows: translating research (5.4 hours), delivering results (4.3 hours), conducting preliminary consultations (3.9 hours), instruction (3.8 hours), documentation (3.0 hours), additional tasks that were written-in by respondents (2.2 hours), and writing the manuscript (1.8 hours). the most common written-in tasks were development of inclusion/exclusion criteria, critical appraisal, and deduplication. other write-ins included retrieving full-text articles, developing protocols, and selecting a journal for publishing the systematic review.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>for the second phase of analysis, 12 responses were excluded as extreme outliers, and the remaining 93 responses were analyzed to detect a relationship between experience and time on task. prior systematic review experience correlated with shorter times performing instruction, consultation, and translation of searches. however, librarian years of experience affected the percentage of time on task, where greater years of experience led to more time spent consulting and instructing than the percentage for librarians with fewer years of experience. librarians with greater than 7 years of experience skewed trends toward shorter time on task, and, with their data excluded, years of experience showed weak positive correlation with instruction and consultation. <o:p></o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>conclusion</span></b><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'> – because the average librarian participating on systematic review teams has had few prior experiences and because the times can vary widely based on assigned roles, duties, years of experience, and complexity of research question, it is not advised to establish expectations for librarians’ time on task. this may be why library administrators have disparate expectations of librarians’ involvement in systematic reviews and find it difficult to allocate and anticipate staff time on systematic review projects. while it may not be possible to set specific overarching guidelines for librarians’ expected time on systematic review tasks, librarian supervisors and library directors planning for their staff to offer systematic review services should work to develop extensive understanding of the steps for conducting and assessing systematic reviews in order to better estimate time commitments.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>commentary<o:p></o:p></span></b></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>librarians typically serve in searching and data management roles on systematic review teams, and many provide expertise in question design and content evaluation (dudden &amp; protzko, 2011; spencer &amp; eldredge, 2018). two recent studies investigated total time for systematic reviews and meta-analysis (gann &amp; pratt, 2013; saleh, ratajeski &amp; bertolet, 2014); however, this is the first study to investigate the librarian’s time on each task contributing to systematic reviews. the study involved a survey questionnaire emailed to various lists, and the evidence could be stronger if the research were supported by an observational study of librarians, where they tracked their time and effort while conducting one or several systematic reviews. the variation in reporting may also be reduced by stratifying times by systematic review quality. there may be a relationship between time on task and quality of the final product if, for example, fewer databases were searched, search strategies did not include subject headings and were not customized for each database, additional relevant terms were missed, and grey literature was not explored.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>the critical appraisal checklist developed by boynton and greenhalgh (2004) includes claims of validity and reliability in the reporting of survey research, and the study includes these attributes when explaining the methods of exclusion, quantitative analysis, and interpretation. the authors were transparent in their research by providing the instrument that was used. however, the researchers did not describe what were the statistical criteria for exclusion of outliers and thresholds for factor loadings and whether these were preestablished prior to data analysis, and, in figure 2 and table 3, there are numbered components upon which the data relies; however, there is no explanation of what factors (other than factors 1 and 2) with which those numbers align.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>the article did not report how the survey was piloted, although the survey was relatively brief with questions that were straightforward. the options for selection and text entry were appropriate. this allowed for clean data and clear structure for analysis. open-ended commentary elicited further information from participants. one notable strength in the survey was that it allowed for participants to input other tasks that weren’t accounted for in the original list. this allowed the researchers to catch common systematic review steps, like developing a protocol and conducting appraisals, that they would have otherwise omitted. <o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>ultimately, this study introduces a new way of thinking and compartmentalizing of tasks, and it is clear that the vast majority of time spent by the librarian on systematic reviews is time spent searching. training on how to search more efficiently may be one of the most impactful ways of reducing time spent on systematic reviews, allowing librarians to better involve themselves in systematic reviews while still performing other job duties.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>references<o:p></o:p></span></b></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:35.3pt;text-indent:-35.3pt'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>boynton, p. m., &amp; greenhalgh, t. (2004). hands-on guide to questionnaire research: selecting, designing, and developing your questionnaire.&nbsp;<i>bmj: british medical journal</i>, <i>328</i>(7451), 1312-1315. <span class=msohyperlink><a href="https://doi.org/10.1136/bmj.328.7451.1312">https://doi.org/10.1136/bmj.328.7451.1312</a></span> <o:p></o:p></span></p> <p class=msonormal style='margin-left:35.3pt;text-indent:-35.3pt'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:35.3pt;text-indent:-35.3pt'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>dudden, r. f., &amp; protzko, s. l. (2011). the systematic review team: contributions of the health sciences librarian. <i style='mso-bidi-font-style:normal'>medical reference services quarterly, 30</i>(3), 301-315. <span class=msohyperlink><a href="https://doi.org/10.1080/02763869.2011.590425">https://doi.org/10.1080/02763869.2011.590425</a></span> <o:p></o:p></span></p> <p class=msonormal style='margin-left:35.3pt;text-indent:-35.3pt'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:35.3pt;text-indent:-35.3pt'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>gann, l. b., &amp; pratt, g. f. (2013). using library search service metrics to demonstrate library value and manage workload. <i style='mso-bidi-font-style: normal'>journal of the medical library association, 101</i>(3), 227-229. <span class=msohyperlink><a href="https://doi.org/10.3163/1536-5050.101.3.015">https://doi.org/10.3163/1536-5050.101.3.015</a></span> <o:p></o:p></span></p> <p class=msonormal style='margin-left:35.3pt;text-indent:-35.3pt'><span lang=en style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:35.3pt;text-indent:-35.3pt'><span lang=en style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en'>saleh, a. a., ratajeski, m. a., &amp; bertolet, m. (2014). grey literature searching for health sciences systematic reviews: a prospective study of time spent and resources utilized. <i style='mso-bidi-font-style: normal'>evidence based library and information practice, 9</i>(3), 28-50. <span class=msohyperlink><a href="https://doi.org/10.18438/b8dw3k">https://doi.org/10.18438/b8dw3k</a></span> </span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p></o:p></span></p> <p class=msonormal style='margin-left:35.3pt;text-indent:-35.3pt'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:35.3pt;text-indent:-35.3pt'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>spencer, a. j., &amp; eldredge, j. d. (2018). roles for librarians in systematic reviews: a scoping review.&nbsp;<i>journal of the medical library association</i>,&nbsp;<i>106</i>(1), 46-56. <span class=msohyperlink><a href="https://doi.org/10.5195/jmla.2018.82">https://doi.org/10.5195/jmla.2018.82</a></span></span><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:35.3pt;text-indent:-35.3pt'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:35.3pt;text-indent:-35.3pt'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> </div> </body> </html> microsoft word news2_10pt font evidence based library and information practice 2008, 3:3 99 evidence based library and information practice news eblip5 call for papers © 2008. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. the international programme committee for the 5th international evidence based library and information practice (eblip5) conference invites you to submit oral presentations and posters for the conference, bridging the gap, to be held in stockholm, sweden from 29 june – 3 july 2009. we seek to build on the success of past conferences by including more disciplines, more sectors, more countries and more professional roles than ever before! the international programme committee for eblip5 invites research papers, reports of innovative practice and hot topic discussion papers. key themes include, but are by no means limited to, the following: bridging the gap…the who? • ...between disciplines communications and information studies; health and social care • ...between sectors public and private sectors; health library sector/wider library information service sectors including academic, public, and special libraries; archivists and local libraries and collections • ...between cultures developed and developing world; librarianship cultures; english speaking/other speaking countries bridging the gap…the what? • ...the “implementation gap” principles and practice; research/practice; research/policy; experiment/implementation; tradition/innovation; innovation and disinvestment; reflection and action, knowledge/action; personal learning to professional practice • ...the “expectation gap” library services and user needs; the egeneration and the physical library • ...the “experimentation gap” – data and results; r&d and eblip; information seeking research/information literacy teaching; implementation and evaluation evidence based library and information practice 2008, 3:3 100 • ...the “skills gap” – practitioner knowledge and skills; librarians and research skills; leadership and implementation bridging the gap…the how? • ...using marketing/advocacy • ...using data mining and management tools • ...through collaborative working (with teachers, with academics) • ...through communication (internal and external) • ...through management support and leadership • ...through international co-operation abstract submission for oral presentations and posters is by e-mail to eblip5@kib.ki.se. important dates: • abstracts to be submitted by 31 october 2008. • notification of acceptance will be received by 31 january 2009. • confirmation of participation by mid-february 2009. • registration opens in march 2009. • deadline for submission of full papers june 2009. see the conference video at <http://www.youtube.com/watch?v=b5nf2eezpw>. evidence summary   implementation of proactive chat increases number and complexity of reference questions   a review of: maloney, k., & kemp, j. h. (2015). changes in reference question complexity following the implementation of a proactive chat system: implications for practice. college & research libraries, 76(7), 959-974. http://dx.doi.org/10.5860/crl.76.7.959   reviewed by: sue f. phelps health sciences and outreach services librarian washington state university vancouver library vancouver, washington, united states of america email: asphelps@vancouver.wsu.edu   received: 2 mar. 2017    accepted: 17 apr. 2017      2017 phelps. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine whether the complexity of reference questions has changed over time; whether chat reference questions are more complex than those at the reference desk; and whether proactive chat increases the number and complexity of questions.   design – literature review and library data analysis.   setting – library of a doctoral degree granting university in the united states of america.   methods – the study was carried out in two parts. the first was a meta-analysis of published data with empirical findings about the complexity of questions received at library service points in relationship to staffing levels. the authors used seven studies published between 1977 and 2012 from their literature review to create a matrix to compare reference questions based on the staffing level required to answer the questions (e.g., by a nonprofessional, a generalist, or a librarian). they present these articles in chronological order to illustrate how questions have changed over time. they sorted questions by the service point at which they were asked, either through chat service or at a reference desk.   in the second part of the study authors used the read scale to categorize the complexity of questions asked at the reference desk and via proactive chat reference. they collected data for chat reference for six one-week periods over the course of eight months to provide a representative sample. they recorded reference desk questions for three of those same weeks. both evaluators scored the data for a single week to norm their results, while the remaining data was coded independently.   main results – the complexity of questions in the seven articles studied indicated change over time, shown in tables for desk and chat reference. one outlier, a study published in 1977 before reference tools and resources moved online, reported that 62% of questions asked could be answered by nonprofessionals, 38% by a trained generalist, and only 6% required a librarian. the six other studies were published after 2001 when most resources had moved online. of the questions from these six, authors found a range of 74-90% could be answered by a non-professional, 12-16% by a generalist, and 0-11% required a librarian. once chat reference was added there was more variation reported between studies, with generalist questions at 30-47% of those reported and 10-23% requiring a librarian.   though the underlying differences in the study designs do not allow for formal analysis, the seven studies indicate that more complex questions are asked via chat service than at the reference desk. each staffing level was grouped and averaged for comparison. the 1977 study shows nonprofessional questions at 62%, generalist questions at 32%, and librarian questions at 6%. reference desk questions in the post-2001 articles indicated 81% nonprofessional, 13% generalist, and 5% librarian questions. post-2001 chat questions were at 49% nonprofessional, 36% generalist, and 15% at librarian level.   in the second part of the study, the data coded using the read scale and collected from the proactive chat system showed an increased number and complexity of questions. the authors identified 4% of questions were rated at a level 1 (e.g., directional, library hours), 30% at level 2 (e.g., known item searching), 39% at level 3 (e.g., reference questions), and 27% at level 4 requiring advanced expertise (e.g., using specialized databases or data sets). authors combined questions at levels 5 and 6 due to low numbers, and did not describe these when reporting their study. in comparison, 15% of reference desk questions were at a level 3 on the read scale, and 1% were at level 4.   conclusion – proactive chat reference service increased the number and the complexity of questions over those received via the reference desk. the frequency of complex questions was too high for nonprofessional staff to refer questions to librarians, causing reevaluation of the tiered service model. further, this study demonstrates that users still have questions about research, but for users to access services for these questions “reference service must be proactive, convenient, and expert to meet user expectations and research needs” (p. 972).   commentary   the authors have made excellent use of library literature to create a matrix for the evaluation of the complexity of questions at different service points based on the expertise needed to answer questions. though there is already much published about online reference services, the use of proactive chat reference is just appearing in the literature, so a more thorough explanation of the service would have been useful. for example, zhang and mayer’s (2014) description of proactive chat provides appropriate context.   when evaluated using glynn’s (2006) critical appraisal tool, this study is valid, with scores >75% in each section: data collection (83%), study design (80%) and results (83%). the overall score for validity was 82%.   the study was conducted in two parts: the meta-analysis of the literature and an analysis of data collected by the author’s library. the authors do not provide much detail on how they conducted the meta-analysis of the literature to address the first two research questions, though they do report on their rationale for the seven studies selected for analysis. they included the matrix created from those seven articles with outcomes that are clearly described in tables, graphs, and narrative form.   their data collection is clearly described for the third research question, making it easier to duplicate. to determine the complexity of reference questions they conducted their analysis using the read scale, a validated instrument. they gathered data from chat transcripts and from the reference desk over representative times. the two researchers coded the reference questions after going through a norming process. however, the data from chat transcripts may present more objective data than the data collected at the reference desk, where different librarians could interpret the level of questions differently.   a significant finding to academic librarians is that patrons still have complex research questions that they are willing to ask through a proactive chat service. this study gives librarians “the opportunity to once again provide individual reference service at the point of need” (p. 972). it also raises the practical issue of increased staffing to manage increased chat activity. since the questions that arrive via proactive chat tend to be more complex, it is possible that more librarians, instead of non-professional staff, will be required, adding to already tight library budgets.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154   zhang, j. and mayer, n. (2014). proactive chat reference. college & research library news, 75(4), 202-205.   evidence based library and information practice commentary   bogus evidence   r. laval hunsucker information and collection specialist emeritus university libraries, universiteit van amsterdam amsterdam; silversteyn 80, breukelen, the netherlands email: amoinsde@yahoo.com   received: 31 jan. 2013   accepted: 21 feb. 2013      2013 hunsucker. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     the intrinsic uncertainty of research integrity   the year 2012 was a good year for research fraud, or at least a good year for illustrating what the eventual outcomes of research fraud can be. in february, anesthesiologist yoshitaka fujii was dismissed from toho university for having fabricated data for at least 172 research articles. in november, the university of kentucky and the u.s. federal government brought to a close the case against eric smart, a diabetes and cardiovascular disease specialist, who had fabricated or falsified data in 21 articles and other research documents. also in november, the final report appeared in the case of diederik stapel, a social psychologist at tilburg university, who had published 55 fraudulent articles and infected the dissertations of numerous of his doctoral students who had based their work in part on his fabricated and falsified data sets. less than two months before the appearance of the stapel report, another investigatory committee had submitted its follow-up report on the affair around the internationally respected cardiologist don poldermans, whom the erasmus medical center had fired in late 2011 for research misconduct.   there was a time when researchers as a matter of course upheld the “pretense that research misconduct is too rare to matter” (macilwain, 2012a, p. 1417). yet many have now come to suspect, even openly to proclaim, that those cases of research misconduct which are in fact eventually exposed probably amount to only the tip of the iceberg, and that the actual detrimental effects of that misconduct on science and scholarship are now already quite substantial. such suspicions and pronouncements are not based solely on subjective impression or anecdotal evidence. there now exists a small but growing body of research into the extent and diversity of the problem. meta-studies carried out in recent years have indicated that some measure of intentional research misconduct (in the widely accepted sense of fabrication, falsification, or plagiarism in the proposing, performing, or reporting of research) is at work within perhaps 10% to 20% of all research, while additional questionable research practices (qrps) bring to over 50%, at minimum, the share of the research effort likely to be producing misleading, erroneous, or altogether worthless results (fanelli, 2009). frequently, it is the highly prestigious peer-reviewed journals that publish these results. fang, steen, and casadevall (2012) found that well over half of the articles indexed in pubmed as retracted had been retracted on grounds of actual or suspected research misconduct. the study by john, loewenstein, and prelec (2012) yielded, for psychology, a misconduct and/or qrp rate higher than 90%. their findings suggest that certain qrps “may constitute the de facto scientific norm” (p. 524). many researchers apparently consider such practices a necessity in order for them to survive in their work (martinson as cited in bonetta, 2006, p. 875). and an extra complication for the scholarly enterprise is that the discovery of misconduct in itself is not enough, since, as hernon and altman (1999) write: “we know that only a few studies are discredited; an overwhelming majority remain in the literature untainted, even though their falsity has been ascertained; and that many continue to be cited for years after the misconduct has been exposed” (p. 402).   a growing concern   those who have studied questions of research integrity are fond of pointing out that misconduct by no means remains confined to those fields in which its exposure has happened to receive the greatest publicity, or in which it has aroused the greatest public interest or concern. it can, and presumably does, occur in every field – certainly in any field where empirical research plays an important role. furthermore, to a certain extent it is by nature self-perpetuating. according to john, loewenstein, and prelec (2012), the unrealistically elegant results achieved through research misconduct and other qrps “can lead to a ‘race to the bottom,’ with questionable research begetting even more questionable research” (p. 531). but is it in fact the case that fraudulent practices are on the increase? fang, steen, and casadevall (2012), at any rate, speak of an “ongoing retraction epidemic”, and state that the “percentage of scientific articles retracted because of fraud has increased ∼10-fold since 1975” (p. 17028). such a finding is indeed very much in line with the widespread opinion among researchers and other stakeholders, in various fields, that research misconduct is not only on the rise but also becoming easier to commit successfully (according to stapel himself, too easy), while “the probability of being found out is minimal” (stroebe, postmes, & spears, 2012, p. 682). the director of the u.s. office of research integrity openly admits “there are also more and more ways for people who want to cheat to do so” (wright as cited in macilwain, 2012a, p. 1419). something, then, has got to be done. it has gradually become clear that science and scholarship are not self-correcting, at least not sufficiently. macilwain (2012b) speaks of “a generation of denial” which has come to an end, now that the worldwide research community is finally taking research misconduct seriously and has put the development and implementation of countermeasures firmly on the agenda (heijden et al., 2012; interacademy council, & iap, 2012; panel on responsible conduct of research, 2011; second world conference on research integrity, 2010).   the case of library and information research   nonetheless, there are disciplines, certainly in the social sciences and humanities, which have never concerned themselves much, if at all, with the question of research misconduct within their ranks, and continue largely to ignore it as even a potential problem. one of these is – ironically, one might well think – library and information studies (lis). in a search of this discipline’s literature, i could locate only two publications (burke et al., 1996; curry, 2005) that touch more than perfunctorily on the question. the former characterizes research misconduct as not demonstrably an issue in lis and unlikely to become one; the latter suggests that, though it most likely should be an issue, lis professionals will probably remain collectively unwilling to treat it as one. the modern era of publicity and public concern regarding serious violations of research integrity began in the early 1980s, at the time of the darsee affair. it took another fifteen years  before the lis literature produced its first publication broaching the subject of possible research misconduct in its own field. that publication, a speculative but noncommittal editorial, did at least assert an ambition to “elicit further conversation”, as well as possibly “a review article which would inform us all in more depth on this important topic” (burke et al., 1996, p. 200). the further conversation seems never to have materialized, and that review article has still to be written. hernon and calvert (1997) assumed in passing that there was “probably not” a “serious problem” (p. 88) in our field, but wondered whether it wouldn’t at least be a good idea to conduct an up-to-date review of relevant standards, policies, and procedures – a review which to my knowledge also never took place. even with curry’s subsequent contribution, our knowledge remains no deeper than it was in 1996, and since 2005 there has again been nothing but silence on the topic.   given the entire absence of any research into, or even of any informed speculation on, the extent and nature of possible research misconduct in lis, we can only speculate concerning the actual situation. clearly, fraud and other forms of research misbehaviour are a proven and acknowledged factor in the worlds of medical, psychological, biological, and physics research, to name but a few obvious examples. that they should then somehow be absent from the world of lis research seems improbable in the extreme. but do we at least have reasons to believe that they are probably less prevalent in lis than in, for example, the fields just named above? yes, we do. do we have reasons to believe that they may be more prevalent in lis than in those and other fields? yes, we have those as well. there are good arguments which one could advance in support of either view, based on all we have learned from the many published descriptions, investigations, and analyses of known cases of fraudulent researchers in numerous disciplines. or is it perhaps better, at least until further notice, simply to operate on the working assumption that the lis research world is a more or less normal research world, and thus provisionally to infer that at least one in every ten lis research studies may well to some degree be fraudulent, while at least half of them will have incorporated one or more questionable research practices? pending the kind of deeper understanding which burke et al. (1996) had hoped would be forthcoming, but has not been, such a working assumption and such an inference would indeed not appear to be an irresponsible choice.   but do not we in fact owe it to the profession to go further than that? it is now seventeen years since the entire editorial board of library & information science research identified the issue of fraud in lis research as an “important topic” about which more should be known, yet the profession’s reaction has remained one perhaps best described by library/information school professor william fisher when he wrote (1999) “we are fortunate these practices do not seem to be a major problem for the lis literature, so we will not dwell on them” (p. 66). but ought we really to just go on cheerfully about our business while contenting ourselves with the conclusion that research misconduct does not seem to be a major problem in our neck of the woods? fisher cites no evidence and adduces no arguments that might serve to justify such a relatively unconcerned attitude. if it is justified, then we should at least be able to point out how we know that it is. if it is not justified, then the sooner we know that the better. the sooner we are in a position to estimate the extent and to begin to describe the nature of the problem in our field, the better off we, and the field, will be. the same goes for the detection and the investigation of specific cases – not so much out of a desire to stigmatize or to penalize wayward colleagues, as out of a sense of obligation to cleanse and correct the research record where appropriate. and let us not overlook a consideration of equal or in fact even greater importance. as bosch points out (2012), “the details of such cases also highlight what future action is needed to prevent similar misconduct” (p. 1680).   no evidence without integrity   if research misconduct is in principle an “important topic” for the lis research community at large, one would think it ought to be a matter of particular concern to anyone with even a casual interest in evidence based practice (ebp), to say nothing of committed ebp advocates or practitioners. yet up to now, quite remarkably it seems to me, there has been no indication, indeed hardly the slightest hint, that such is the case. to what extent, and how, do fraud and other qrps actually impact upon the evidentiary value of the research literature – in lis or for that matter anywhere else? here again, we cannot but resort to speculation. to my knowledge, there exist no more than two publications (lelgemann & sauerland, 2010; neugebauer, becker, sauerland, & laubenthal, 2009) which have addressed the relationship between research misbehaviour and evidence based practice. both deal explicitly with the establishment of specialized clinical guidelines, and have only limited relevance for the lis domain.   given the situation as so far sketched above, lis professionals set on founding their practice upon the best available evidence from research may be tempted to respond by arguing that the factor of potentially fraudulent research may indeed render the task confronting us a bit more complicated and challenging than we had previously imagined but that, even so, we as ebpers already have an instrument capable of effectively dealing with that task. if only we persist in our commitment to a rigorous and systematic habit of critical appraisal of all potentially pertinent evidence, there should be little reason for us to fear any contaminating influence of research misconduct on the decisions that we take. comforting as this reassurance may at first sound, its validity is unfortunately open to serious doubt. as already noted, the record of success in detecting probable scientific misconduct, to say nothing of conclusively proving such misconduct, has been decidedly poor. as trikalinosa, evangeloua, and ioannidis (2008) point out, “there are no strong alert signs to hint that a paper is fraudulent. ... overall, a fraudulent article looks much the same as a nonfraudulent one. ... even blatant papers of falsification may require careful scrutiny to be revealed” (p. 469). and no wonder. those who have ultimately been exposed as, or who have eventually confessed to being, committers of fraud have tended to be highly competent or even unusually talented researchers, not infrequently the holders of important positions within prestigious institutions. such wrongdoers can be very adept at masking their own violations of research integrity, and have at their disposal the facilities and influence which support them in doing so. this has long been known. marathe (1989) painted an insightful portrait of the typical dishonest researcher as a highly intelligent person with a good reputation, who is “much aware of his [or her] competence,” “used to success,” and “does not intend to be caught” (p. 259). under these circumstances, we can hardly expect that standard critical appraisal routines will normally be able to lay bare the unethical practices behind the publications of intentionally fraudulent researchers. we should likewise not assume that our traditional information literacy and critical thinking skills are well fitted to this task, or that the intermediation of what eldredge (2012) has termed the new evidence “translator” will offer much practical relief in this context (p. 141). of the various techniques customarily suggested for identifying instances of suspected research misconduct, some (e.g., peer review, editorial control, co-author alertness) have repeatedly shown themselves incapable of actually doing so (relman, 1983; stroebe, postmes, & spears, 2012). others (e.g., research auditing, replication, whistle blowing) have proven to some degree effective especially in the exact and life sciences but are, for differing reasons, much less suitable within an area such as lis.   indeed, we would probably be well advised to pin few hopes on our prospects of ever becoming very successful at the detection of fraud. if the record of success has been decidedly disappointing in the “harder” sciences, the odds against booking significant successes in lis would seem to be extraordinarily large. a more promising approach is likely to be one oriented less toward the detection, and much more toward the prevention, of fraud and qrps. but how can one best go about anticipating and forestalling violations of research integrity from the outset? observers have not been at a loss for ideas and recommendations, such as: a fundamental overhaul to the system of incentives, rewards, and academic/professional recognition such that the quality, conclusiveness, and transparency of research and its reporting become more decisive than quantity and speed of publication; agreements that journals will henceforth devote more space to the publication of negative or “null” results and to the reporting of replicative research, and that universities and other relevant organizations will institutionalize stronger incentives and recognition for researchers producing such publications; far less emphasis on the attainment of “mediagenic” research results; measures aimed at mitigating the increasingly fierce competition for (diminishing) research funding; the mandatory archiving and long-term unhindered accessibility of all raw research data, protocols, and analysis codes; the requirement that each named author formally accept full co-responsibility for the entirety of a published research report; reduction or elimination of “honorary” authorships; increasing the likelihood that a fraudulent researcher will be caught and penalized, for example by encouraging whistleblowers through guarantees of anonymity or career protection. often heard is the suggestion that the best means of systematically reducing the occurrence of research misbehaviour in the long run is to ensure that the training of future researchers includes a comprehensive and mandatory research integrity component. anderson et al. (2007), however, have shown that the positive effect of integrity training and mentoring has possibly been greatly overestimated, and that in some respects the effect may even be negative. anderson herself (2007) proposes that institutionalized group mentoring might yield positive results, but it is her further notion of a culture of “collective openness” which most appeals to this commentator. she sees that as “a mechanism for sustaining attention to the responsible conduct of research on an everyday basis” in an environment in which “everyone is not only encouraged but expected to question each others’ decisions and work, so that mistakes and oversights, as well as misbehavior, will be noticed and corrected”, where there is “a collective sense of responsibility for the integrity of the work” and where “not challenging questionable behavior or decisions is unacceptable” (p. 392). sobel (2012), too, strongly emphasizes structural openness as “paramount” in any effort to curb misconduct before it occurs. openness after the fact is also crucial: if, and as soon as, misconduct is discovered, the last thing that should happen is that it be swept under the carpet, for example, as a face-saving tactic for the institution or journal involved (bosch, 2012).   coping with research misconduct   there is certainly something to be said for each of the suggested measures listed above. taken together, they could possibly even make a quite significant difference for the good. still, it would be an illusion to believe that they can free lis or any other field altogether of fraud and other qrps. that problem will always be with us. any lis professional who values our research literature as a source for evidence of potential utility in the improvement of professional practice should keep in mind that there are not only one but, rather, two inherent potential threats to such utility. though they differ in two important respects, their effect is more or less the same. as burke et al. (1996) have put it, “given the nature of lis research, fraud and falsification would be difficult to spot. poorly conducted and poorly reported research, however, is a well-identified problem, and largely has much the same result as fraud – misleading research findings” (p. 206). one important difference, then is that while we have, as ebp-aficionados, developed fairly effective ways of recognizing and dealing with the hazards of sloppy research, we have apparently not even begun to think about ways of dealing with the hazards of fraudulent research, hazards which in any event are much trickier to localize. that’s the bad news. the good news, and the other important difference, is that genuinely fraudulent research in lis is almost certainly far less prevalent than sloppy research in lis. that circumstance can give us hope that by consciously adopting preventive measures we can eventually reduce its frequency to an even less problematical level. it should then be possible to move even further toward the objective of neutralizing whatever bogus evidence continues to exist. in this connection, too, it is fortunate that “[t]he number of systematic reviews published in lis each year seems to slowly be growing” (koufogiannakis, 2012, p. 93), for it is by basing our decisions where possible on the syntheses of research evidence provided by well-executed systematic reviews, rather than on the findings of individual studies or fortuitous collections of articles, that we can most effectively evade any potential contaminating influence attributable to undetected fraudulent or otherwise questionable research practices infecting the literature of lis and other disciplines.      references   anderson, m. s. (2007). collective openness and other recommendations for the promotion of research integrity. science and engineering ethics, 13(4), 387-394. doi:10.1007/s11948-007-9047-0   anderson, m. s., horn, a. s., risbey, k. r., ronning, e. a., de vries, r., & martinson, b. c. (2007). what do mentoring and training in the responsible conduct of research have to do with scientists' misbehavior? findings from a national survey of nih-funded scientists. academic medicine, 82(9), 853-860. doi:10.1097/acm.0b013e31812f764c   bonetta, l. (2006). the aftermath of scientific fraud. cell, 124(5), 873-875. doi:10.1016/j.cell.2006.02.032   bosch, x. (2012). research integrity on the horizon. the lancet, 379(9827), 1679-1680. doi:10.1016/s0140-6736(12)60317-1   burke, m., chang, m., davis, c.,  hernon, p., nicholls, p., schwartz, c., shaw, d., smith, a., & wiberley, s. (1996). editorial: fraud and misconduct in library and information science research. library & information science research, 18(3), 199-206. doi:10.1016/s0740-8188(96)90040-7   curry, a. (2005). unreliable research: are librarians liable? ifla journal, 31(1), 28-34. doi:10.1177/0340035205052640   eldredge, j. d. (2012). the evolution of evidence based library and information practice, part i: defining eblip. evidence based library and information practice, 7(4), 139-145. retrieved from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/18572/14514   fanelli, d. (2009). how many scientists fabricate and falsify research? a systematic review and meta-analysis of survey data. plos one, 4(5), e5738. doi:10.1371/journal.pone.0005738   fang, f. c., steen, r. g., & casadevall, a. (2012). misconduct accounts for the majority of retracted scientific publications. proceedings of the national academy of sciences of the united states of america, 109(42), 17028-17033. doi: 10.1073/pnas.1212247109   fisher, w. (1999). when write is wrong: is all our professional literature on the same page? library collections, acquisitions, and technical services, 23(1), 61-72. doi:10.1016/s1464-9055(98)00126-2   heijden, p. f. v. d., fokkema, j., lamberts, s. w. j., mols, g. p. m. f., hartogh, g. a. d., stouthard, m. e. a., & post, a. a. (2012). de nederlandse gedragscode wetenschapsbeoefening: principes van goed wetenschappelijk onderwijs en onderzoek, revised version. den haag: vereniging van universiteiten vsnu. retrieved from http://www.vsnu.nl/files/documenten/domeinen/onderzoek/code_wetenschapsbeoefening_2004_%282012%29.pdf  (english translation: retrieved from http://www.vsnu.nl/files/documenten/feiten_en_cijfers/the_netherlands_code_of_conduct_for_scientific_practice_2012.pdf).   hernon, p., & altman, e. (1999). misconduct: infecting the literature, but do we really care? the journal of academic librarianship, 25(5), 402-404. doi:10.1016/s0099-1333(99)80061-5.   hernon, p., & calvert, p. j. (1997). research misconduct as viewed from multiple perspectives. in e. altman & p. hernon (eds.), research misconduct: issues, implications, and strategies (pp. 71-89). greenwich, ct [etc.]: ablex.   interacademy council, & iap (2012). responsible conduct in the global research enterprise: a policy report. amsterdam: iac secretariat; trieste: iap secretariat. retrieved from http://www.interacademies.net/file.aspx?id=19789   john, l. k., loewenstein, g, & prelec, d. (2012). measuring the prevalence of questionable research practices with incentives for truth telling. psychological science, 23(5), 524-532. doi:10.1177/0956797611430953   koufogiannakis, d. (2012). the state of systematic reviews in library and information studies. evidence based library and information practice, 7(2), 91-95. retrieved from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/17089/14045   lelgemann, m., & sauerland, s. (2010). gefälschte studien und nicht publizierte daten: auswirkung auf die erarbeitung von leitlinien und evidenzbasierten empfehlungen. zeitschrift für evidenz, fortbildung und qualität im gesundheitswesen, 104(4), 284-291. doi:10.1016/j.zefq.2010.03.035   macilwain, c. (2012a). scientific misconduct: more cops, more robbers? cell, 149(7), 1417-1419. doi:10.1016/j.cell.2012.06.001   macilwain, c. (2012b). the time is right to confront misconduct. nature, 488(7409), 7. doi:10.1038/488007a   marathe, s. (1989). scientific fraud. nature, 340(6231), 259. doi:10.1038/340259c0   neugebauer, e. a. m., becker, m., sauerland, s., & laubenthal, h. (2009). wissenschaftsbetrug/gefälschte studien: auswirkungen auf die s3-leitlinie? deutsches ärzteblatt, 106(15), a703. retrieved from http://www.aerzteblatt.de/archiv/64093/wissenschaftsbetrug-gefaelschte-studien-auswirkungen-auf-die-s3-leitlinie   panel on responsible conduct of research (2011). the tri-agency framework: responsible conduct of research. ottawa: secretariat on responsible conduct of research. retrieved from http://www.rcr.ethics.gc.ca/eng/policy-politique/framework-cadre/   relman, a. s. (1983). lessons from the darsee affair. the new england journal of medicine, 308(23), 1415-1417. doi:10.1056/nejm198306093082311   second world conference on research integrity (2010). singapore statement on research integrity. retrieved from http://www.singaporestatement.org/downloads/singpore%20statement_a4size.pdf   sobel, b. e. (2012). on thwarting the seeds of scientific fraud.  coronary artery disease, 23(8), 560-562. doi:10.1097/mca.0b013e32835a05e9   stroebe, w, postmes, t., & spears, r. (2012). scientific misconduct and the myth of self-correction in science. perspectives on psychological science, 7(6), 670-688. doi:10.1177/1745691612460687   trikalinosa, n. a., evangeloua, e., & ioannidis, j. p. a. (2008). falsified papers in high-impact journals were slow to retract and indistinguishable from nonfraudulent papers. journal of clinical epidemiology, 61(5), 464-470. doi:10.1016/j.jclinepi.2007.11.019   evidence based library and information practice using evidence in practice   using data to break apart journal packages   alison ambi science research liaison librarian memorial university of newfoundland st. john’s, newfoundland & labrador, canada email: aambi@mun.ca   erin alcock science research liaison librarian memorial university of newfoundland st. john’s, newfoundland & labrador, canada email: ekalcock@mun.ca   pamela morgan information resources librarian memorial university of newfoundland st. john’s, newfoundland & labrador, canada email: pmorgan@mun.ca   amanda tiller-hackett humanities collection development librarian memorial university of newfoundland st. john’s, newfoundland & labrador, canada email: atiller@mun.ca   received: 4 nov. 2016    accepted: 14 nov. 2016      2016 ambi, alcock, morgan, and tiller-hackett. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     setting   memorial university of newfoundland is the only university in newfoundland and labrador and has approximately 20,000 students. the university aims to provide both a comprehensive undergraduate program and research-intensive graduate programs. because of the university’s geographic isolation it is particularly important for the library to provide a comprehensive collection. unlike in larger centres, researchers can not avail themselves of the resources held by nearby institutions. we therefore tend to adopt a cautious and conservative approach to cancellations or weeding.   problem   over the past decade there has been a steady increase in journal subscription prices while the materials budgets of academic libraries has remained fixed or even decreased. when this trend coincided with a precipitous devaluation of the canadian dollar beginning in 2013, many canadian academic libraries were thrust into a serials budget crisis.   librarians at memorial had implemented many smaller cost-saving measures during 2013 and 2014, however, it became evident that trimming the low-hanging fruit would not suffice. librarians needed to set their sights on significantly larger cancellation targets to make ends meet. a committee was formed to evaluate our cancellation options.   based on an analysis of the target savings, the subscriptions available for renewal at the end of 2015, and strategies to avoid disadvantaging any particular discipline, the committee recommended cancelling four large multi-disciplinary publisher journal packages – cambridge, oxford, springer, and wiley – and buying back only the most crucial titles. this was a dramatic proposal by any standard, but especially for memorial. it was, however, unavoidable.   evidence   to determine which journals were critical for supporting our researchers and students, we gathered cost, use, and alternate access data for each of the 4,231 titles in the 4 packages, and we solicited feedback from the university community. we chose not to consider impact factors in our analysis because of the flaws and limitations of this metric.   pinning down a definitive title list and compiling use data was challenging for a number of reasons. title lists are in constant flux due to publisher mergers and acquisitions, new journals launched by publishers annually, journals that change names, merge, split into two, or are discontinued, journals that become open access, and the tendency of society journals to move from one publisher to another. the sub-packages within some of these “big deal” packages presented another challenge, since there were some journals which were not priced individually and could not be purchased individually.   community feedback data   in june 2015 we conducted a survey of the university community. we informed respondents about the plan to cancel the four publisher packages and the rationale for doing so. we asked the respondents to provide lists of journals they deemed “essential” for their research, “important” for their research, and those that they needed for their students and/or teaching activities.   approximately 16% of the faculty/researchers on campus responded to the survey. of the 1,224 titles mentioned in the survey, approximately 1/3 were in one of the 4 packages under review.  the combined list price of the 1,224 journals was significantly more than the cost of the 4 packages, so we knew that we would not be able to retain them all.   of the cambridge, oxford, wiley, and springer titles that were not mentioned in the survey, 88 showed significant use and had over 200 annual article downloads. there were 3 titles with jr1 averages in the vicinity of 1000 which had not been mentioned in the survey. at the opposite end of the scale, 17 titles showing jr1 averages of 0 were mentioned by survey respondents.   given the low response rate and the fact that some very high-use journals had not been mentioned, it was clear that we could not rely heavily on the survey data to make our decisions.   journal metrics   cost data: when we started our analysis the publishers had not yet released their 2016 price lists. we therefore approximated 2016 prices by estimating a 5% increase over the 2015 list prices.   use data: counter data was available from all four publishers to some extent. for each title we calculated:   average of jr1[1] over the past 3 years average of jr5[2] over the past 3 years (where possible)   alternate access data: we conducted an overlap analysis between each publisher package and our aggregator subscriptions to identify titles with alternate availability and the embargo lengths for each title. just over 50% of the journals in the 4 packages were available with a 12 month embargo or better.   combining cost, use, and alternate access data:   we calculated the cost per download for journals to which we had embargoed access we wanted to establish a threshold for current year use at which a subscription might be more cost effective than relying on document delivery. using a cost estimate of $30 to provide a single article through our document delivery service, and estimating that the volume of document delivery requests would not exceed 25% of jr5, we identified journals for which the cost of current year document delivery requests was at risk of exceeding the cost of purchase. our choice of a 4:1 ratio was an over-estimate. subsequently published literature has suggested that the ratio is likely to be considerably lower. nabe and fowler (2015) estimate the ratio at 10:1 and according to scott (2016) it might be as low as 17:1.     table 1 summary of variables calculated for each journal title variable name method of calculation jr1 average [(jr1 2012) +(jr1 2013)+(jr1 2014)]/3 jr5 average [(jr1 2012) +(jr1 2013)+(jr1 2014)]/3 cost (2015 list price in usd) x (5% price increase) x (currency conversion factor) x (tax factor) cost/download (cost) / (jr1 average) best embargo the shortest embargo available in our aggregator products estimated cost of document delivery our institution uses $30 as an estimate of the cost to process a single document delivery request anticipated volume of currentyear document delivery requests (jr5) x (0.25) anticipated document delivery burden [(jr5 average) x (0.25) x ($30)] – cost       (if this was greater than zero for journals available with a 12-month embargo there was a risk that the cost of anticipated document delivery requests would exceed the cost of subscription)     implementation   by setting thresholds for use, cost/download, and volume of community feedback and considering the availability of embargoed access, we were able to use microsoft excel to filter out a list of the most crucial titles. this short initial list had a combined list price well within what we could afford to spend. information about the proposed fate of all journals in the packages, and about alternate access for the titles slated for cancellation was distributed to the university community for a second round of feedback.   we received many requests from faculty to reverse cancellation decisions. we revisited the cost, use, and alternate access data for the requested titles in the context of the qualitative information provided by researchers. to the extent possible, we added those titles for which a case could reasonably be made.   outcome   of the 4,231 titles in the 4packages, we retained subscriptions to 220 (5%), saving a total of approximately $800,000 when compared to what we would have paid had we renewed the 4 packages.    of the remaining titles, 1,942 (49%) were available with a 1 year embargo in our aggregator products, and 75 (2%) were available with no embargo. removing the handful of titles that had either become open access, moved to a different publisher, discontinued publications, etc., the final number of journals that we truly cancelled with no alternate access options was 1,758 (44%). see figure 1.     figure 1 final buy-back results.     of the 1,758 cancelled journals with no alternate access, many had very low use. for 202 titles (11%) there had been no article downloads in the past three years (jr1 average = 0), and for 913 titles (51%) there were 10 or fewer article downloads per year averaged over the past 3 years (jr1 average <=10).   there were journals we did not want to cancel and we are watching the document delivery statistics for these titles closely. it has not yet been a full year since the cancellations, but an initial analysis of our document delivery data has not revealed any significant demand for articles from any of the cancelled journals.   from a purely cost-saving perspective our process was successful. the data we gathered and how we used it to make our decisions reduced our subscription expenditure in 2016 and allowed us to avoid increased document delivery costs. we succeeded in dramatically decreasing our journal subscriptions while minimizing the impact on the university community.   reflection   although we succeeded in saving enough money, we acknowledge that from other perspectives, the outcome of this process was not ideal.   document delivery can provide researchers with access to any of the content we have cancelled, but we have lost easy and rapid access to content that might have been useful to our community. the literature indicates that there is little impact on document delivery when a big deal is cancelled (calvert, gee, malliett, & flemming, 2013; mcgrath, 2012). some students and researchers may be deterred from accessing this content using document delivery. others will turn to illegal or underground sources like sci-hub and peer networks which will skew the data, making future subscription decisions more difficult.   volume of use is only one way to measure the relative value of a journal within an institution, and it has several flaws and limitations. our numbers game inherently favoured the research interests of larger departments and research groups on the campus, journals with interdisciplinary appeal, and journals in disciplines where the primary research literature is more heavily used by undergraduate students. the value of journals representing emerging research directions, smaller departments and research groups, and disciplines in which primary literature is typically accessed only by graduate students and faculty, cannot easily be judged on the basis of use.   faculty tend to use very different criteria from librarians when assigning value to journals, and in some disciplines they care deeply about impact factors. user-perceived criteria are often personal (“i have published in this journal”) or prestige-based (“it would be embarrassing for the institution if we cancelled this famous journal”) and consequently more emotionally determined and less susceptible to arguments based on use and cost. as a result, many of those engaged in research and teaching at our institution feel that resources of high value to them have been cancelled. it might have been possible to bring the outcome of our analyses closer to faculty perceptions by systematically including data about local publishing activities.   subscribing individually to the journals we chose to retain has also placed a significant burden on our acquisitions department and complicated the process of making the journals available through our discovery layer. the impact of this process on less tangible institutional resources has been significant, especially if one includes the time devoted by the cancellations committee.   on a more positive note, there is generally a more comprehensive understanding among the faculty and administration at our institution of trends in academic journal publishing and the consequences for academic libraries. the cancellations project provided many opportunities for librarians to discuss these issues with our colleagues in academia.   conclusion   the librarians engaged in the cancellations process learned an enormous amount about the complexities of academic journal publishing. in addition, the scale of the project, which involved simultaneously grappling with four packages, forced us to learn a lot about data manipulation and analysis. this has enabled us to approach our next task the analysis of the taylor & francis package in 2016 with more efficiency and nuance.   when we embarked on this project in 2015, we knew that many other canadian libraries would be forced to cancel “big deals”, but few had publicized their cancellation activities. because we were pioneers of a sort, it was difficult in 2015 to convince our community that the situation at memorial was not particularly unique. now that many other canadian institutions have publicized their cancellation efforts, our faculty are more conversant with the issues in academic journal publishing, there is a heightened awareness of the budgetary crisis in our province, and our review process in 2016 is progressing with less passionate opposition. we hope that documenting our experiences will be of some benefit to other librarians faced with this task.   we will continue to monitor document delivery requests for cancelled titles, as well as the use data for titles that were retained to determine if annual adjustments to our subscription list is warranted. for the majority of the cancelled titles we have perpetual access to issues published during the years to which we subscribed to the packages. it is therefore not surprising that we have not yet observed any increases in document delivery requests.   now that we have broken free from several publisher packages we have the flexibility to compare the relative values of individual journals without the need to consider the publisher.   there is no doubt that the “big deal” package is the most cost-effective and efficient way to subscribe to the journals from many academic publishers. however, no matter how cost effective it is, if the cost is higher than libraries can afford, we will have no choice but to continue breaking the packages apart.   references   calvert, k., gee, w., malliett, j., & flemming, r. (2013). is ill enough? examining ill demand after journal cancellations at three north carolina universities. proceedings of the charleston library conference, charleston, south carolina. http://dx.doi.org/10.5703/1288284315297   mcgrath, m. (2012). fighting back against the big deals: a success story from the uk. interlending & document supply, 40(4), 178-186. http://dx.doi.org/10.1108/02641611211283831   nabe, j., & fowler, d. c. (2015). leaving the “big deal” … five years later. the serials librarian, 69(1), 20-28. http://dx.doi.org/10.1080/0361526x.2015.1048037   scott, m. (2016). predicting use: counter usage data found to be predictive of ill use and ill use to be predictive of counter use. the serials librarian, 71(1), 20-24. http://dx.doi.org/10.1080/0361526x.2016.1165783     [1] jr1 for a journal = the total number of articles downloaded during one year [2] jr5 for a journal = the number of articles downloaded in a given year that were published in that same year microsoft word art 4192 evidence based library and information practice 2009, 4:1 4 evidence based library and information practice article inquiring informationists: a qualitative exploration of our role rex r. robison informationist national institutes of health library, bethesda, md email: rexr@nih.gov mary e. ryan informationist national institutes of health library, bethesda, md email: ryanm@mail.nih.gov i. diane cooper informationist national institutes of health library, bethesda, md email: cooperd@mail.nih.gov received: 29 september 2008 accepted: 11 february 2009 © 2009 robinson et al. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – the goal of this study is to explore the impact of an informationist program at the national institutes of health (nih) library and to provide a basis for further program assessment. in 2001 the nih library began its informationist program, where librarians with training in both biomedicine and information science work alongside researchers. the goal of the program is to facilitate researchers’ access to and usage of information resources. methods – the researchers used qualitative interviews with key informants to characterize the current informationist services of user groups. subjects were selected to capture a variety of activities that would show patterns of how the program assists the researchers of various nih groups. following the interviews, evidence based library and information practice 2009, 4:1 5 the authors extracted recurring and significant themes from the subjects’ comments. results – interview subjects provided their views on the informationists’ skills, impact, and team participation. research results documented that informationists helped find resources, provided instruction, and worked as part of the research team. the nih groups currently using this service value their informationists’ knowledge of library resources and their ability to access information needs quickly. the informationists’ skills in finding information save the researchers time, increase the efficiency of the research team, and complement the contributions of other team members. training by informationists was found useful. informationist services led to increased self-reported library use, albeit in some cases this use was entirely via the informationist. conclusions – informationists saved researchers time by obtaining requested information, finding esoteric or unfamiliar resources, and providing related training. these activities appeared to be facilitated by the acceptance of the informationist as part of the research team. this exploratory study provides background that should be useful in future, more extensive evaluations. introduction the national institutes of health (nih), a part of the u.s. department of health and human services, is the primary federal agency for conducting and supporting medical research in the united states. nih is made up of 27 different institutes and centers with over 18,600 employees. this group includes researchers in nih laboratories and clinics, clinical staff in the research hospital, extramural program staff who oversee the grants for research outside nih, and the leadership and administrative staff. the nih library seeks to serve the information needs of this diverse group. since the nih staff is spread across several buildings and cities, the nih library continually works to improve outreach and service, such as providing over 95% of its clinical and research journals online. in 2001, the nih library established an informationist program as another means to reach the nih staff. “informationist” was a term first proposed in 2000 (davidoff and florance 996). in this editorial the authors suggested that clinicians could delegate their information needs to informationists just as they order lab work, x-rays, and other procedures from other health care professionals. they imagined an informationist would be a librarian who provides not only information services but who also brings those skills to the clinical care floor or the research laboratory outside of the library departments. the nih informationists are librarians trained in both biomedical subjects and information science, and who work alongside researchers. since 2001, the program has expanded over time to include 14 informationists working with over 40 groups in 16 institutes and centers (whitmore, grefsheim and rankin 135). each informationist is assigned to work with one to six groups of nih researchers. these groups vary in size, function (clinical, research, administrative), and subject area. all the informationists have library science degrees. about half of them have degrees and/or work experience in science or evidence based library and information practice 2009, 4:1 6 medicine, and all receive training relevant to the work of their groups (robison 341-3). this exploratory study sought to assess the impact of the informationist program, but little was known about the informationists’ collective activities. there was broad awareness of the projects on which individual informationists had worked, including searching literature databases, answering reference questions on hospital rounds and in grand rounds, providing instruction, finding collaborators and experts, as well as creating web pages, wikis, and databases. survey data (whitmore, grefsheim and rankin 139) indicated that users were pleased with informationist services. however, a general schema for the types of projects undertaken and information on what users particularly appreciated about the services were lacking. this study was initiated to gather data about the roles informationists play within their teams. the goal was to gain insight into and generate research questions about informationists’ impact on facilitating access to information and the effect that has on nih research. understanding the potential impact of the informationist program requires understanding the work of the nih staff. nih is similar in many ways to other large research hospitals, but it is clearly different in other ways, such as the absence of medical students or an urgent care facility. moreover, the groups served by the library are diverse and include physicians, nurses, scientists, and administrators, engaging in clinical trials, bench research, patient care, budgets, grants, and writing papers. studies on the information-seeking behavior of physicians has been shown to depend on the purpose of the question (teaching vs. patient care vs. research) (gorman 730), the specialty of the physician (bennett et al. 3), the type and location of the facility (forsythe et al. 186; gruppen 167-9), and the personality of the physician (gruppen 167-9; mckibbon, fridsma and crowley 141-3). for example, clinical settings are more concerned with patient care, and so tend to generate questions that are more specific and urgent (forsythe et al. 193; mcknight 148-50; bennett et al. 3). therefore, it was decided to perform a qualitative research study, rather than another broad survey of nih staff (see grefsheim and rankin). the nih library already collects quantitative data on the utilization of resources and users’ assessments of services, but this data does not provide the complete picture. by identifying recurrent and dominant themes from interviews, the goal was to gain a deeper understanding of the relationships informationists have with their groups and to assess the impact that the informationists have on those groups and on their work. these themes would elaborate on the relatively simple quantitative data and indicate areas for further research. methods interviews the key informant interview was selected as the data gathering technique for this study. the value of the interview process lay in the different perspectives and perceptions that the six key informants provided. also, since these interviews primarily took place in the informants’ offices, the environment was considered another source of “observed data.” the research employed a guided, semistructured interview technique. there was a predetermined set of questions, but these questions could be reordered during the interview; the wording of the questions could be flexible; and the interviewer was free to respond and provide clarification. moreover, the interviewer could generate follow-up questions to probe beyond initial responses to the defined interview evidence based library and information practice 2009, 4:1 7 questions. these types of “probes” are designed to prompt subjects to elaborate on topics in their own ways and are particularly valuable in generating a broad spectrum of qualitative responses (berg 93). each interview was conducted by two informationists from a team of three (the authors), one who asked most of the questions and the other who operated the recorder, took notes, and acted as observer. the interviewer pairs were varied for different interviews. the interviewers were always two of the authors, except for the interview with subject #2, where another librarian was the observer. the informationist who actually worked with the informant was never present during the interviews, but in some cases that informationist did introduce the informant to the interview team. interviews took place in the offices of the subjects, except in the case of subject #6, who lacked an appropriate workspace for a private interview. this interview was held in a private office in the nih library. after asking permission, which was granted in all cases, interviews were recorded with an olympus ws-100 digital voice recorder. each interview began with a brief statement explaining that the purpose of the interview was to better understand the informationist program, not to judge the quality of an informationist’s work. the subject was then asked a range of questions (appendix) on the work that the informationist had undertaken. subjects six nih staff members who were considered frequent users of the informationist service were selected as key informants (table 1). subjects were asked, typically by their assigned informationist, if they would assist the library in assessing the program by participating in an interview of 30 minutes or less. ethical approval was not required for this study, since nih does not require it for program evaluations conducted among its staff by central service organizations such as the nih library. id # degree gender title 1 phd, rn female clinical nurse scientist 2 phd male lab chief 3 md male unit head 4 md male consult service director 5 md male branch chief 6 md female staff clinician table 1. subject characteristics informants were purposefully selected, that is, they were nih researchers and clinicians who had been working with the informationist program for at least a year. for this study, the number of interviews was less important than making sure that key informants represented a range of different groups and viewpoints served by the program. this approach is consistent with the qualitative research literature. gilchrist and williams list three reasons for using key informants: “(a) to gather information efficiently, (b) to gain access to information otherwise unavailable to the researcher, and (c) to gain a particular understanding or interpretation of cultural information.” they also add that the “informant needs to be thoroughly enculturated and currently active within his or her own culture.” (gilchrist and williams 74-5) analysis the data analysis utilized a grounded theory approach. grounded theory is a systematic qualitative research methodology that provides for the generation of theory from collected data. it provides the researcher with a model for data collection, analysis, and emergent theory development. evidence based library and information practice 2009, 4:1 8 grounded theory utilizes a unique approach, in that it moves from the specific to the more general. unlike other research methods, grounded theory is inductive and develops from the data, building toward a theory. grounded theory builds, rather than tests, theory (dick). this methodology has been widely used in nursing and other health care research (annells 57; burgoyne et al. 3; gavois, paulsson and fridlund 103; persson and ryden 356; weber 654). in grounded theory, the research question is explored through data gathered from key informants and observation. concepts are identified from the data, connections are observed, and theories emerge. a grounded theory study works through overlapping phases of data collection, note-taking, coding, sorting, and emergent theme development (dick). in the process of collecting and interpreting data, a point of diminishing returns or a saturation point is eventually reached, and subsequent interviews may add nothing more to the identified themes. after each interview, the recording was saved on the library staff's shared drive, accessible by each investigator but not to staff outside the library. each investigator independently reviewed the six taped interviews and extracted themes and statements relevant to the informationists’ impact on the subjects’ works. these elements formed the basic unit for the analysis process. the investigators then met to coordinate and categorize the identified elements. the investigators were largely in agreement, but they resolved differences by individually elaborating on their selected elements and building consensus to create or merge categories. the major themes discussed below emerged from these categories. results table 2 lists six themes, grouped into three categories, the skills of informationists, impact of informationists and perceptions regarding informationists and representative quotes from transcripts. these themes are discussed more fully below. skills of informationists 1. literature retrieval subjects clearly valued the informationists’ ability to retrieve and evaluate the literature. this activity dominated informationists’ activities. (“the most help that i’ve received has usually been on references.”) in just this small sample, there was a wide variety of tasks in which informationists participated, including simple document delivery and reference checking, developing and executing the methods for review papers and meta-analyses, and turning a question from clinical rounds into a pubmed® query and summarizing the results. subjects referred to the informationists’ “expertise,” often stating explicitly that this expertise filled a gap for the team, despite the fact that team members held advanced degrees in other areas (see informationist as team member, below). in particular, they appreciated the informationists’ ability to find literature from databases other than pubmed, e.g., using bioinformatics databases or sources that contain publications outside pubmed’s scope. (“it adds to the creativity tremendously by broadening the perspective on where we can get information….”) 2. informationist as portal going beyond literature searching, informationists found databases, websites, evidence based library and information practice 2009, 4:1 9 scientific collaborators in academia, outreach partners in public libraries, and potential contacts in manufacturing. they also learned specialized scientific software, gave lectures over conference calls, and researched historical usage of medical jargon. when asked about future projects, subjects could foresee informationist involvement in running a lecture series and building a web site. subjects noted informationists’ searching skills (“she’s a resource, a go-to person.”), their (perceived) privileged access to resources (“we do interdisciplinary type of stuff… she has access to databases that will help us search out those parts.”), and their willingness to get really immersed in the task (“he was better equipped than most of the researchers were.”). at the very least, subjects valued these non-bibliographic services, because they didn’t “have the talent or manpower” in their own groups. impact of informationists 3. saving time five of the six subjects specifically mentioned that the informationist saves them time. this efficiency derives from several factors: the informationist’s expertise (“i don’t have time, i don’t want to spend the time learning … when i’ve got someone to help me”); their accessibility (“for you to leave the unit, go down to the library, talk to a reference librarian … it’s just horrible.”); or merely being someone else who can help (“you know, there’s only so many searches you can do in the course of a few days”). informationists also help researchers keep up-to-date in their fields by helping them develop weekly updates for their database searches and by notifying them of recent additions to the library’s collections and resources. 4. providing training which impacts on practice a consensus emerged that training, both formal and informal, provided by the informationist is viewed as important and has a positive impact on research performance and practices. one informationist, for example, provides instruction in database searching, bibliographic management software, and evidence-based research to nih clinical center nursing staff. as a result of evidencebased practice training, the staff are questioning some routines and procedures, saying, “show me the evidence.” another informationist trains researchers in the use of software that helps identify specific genes and gene pathways, as well as linking to relevant literature. with his guidance, researchers created a network of gene pathways crucial to their study using this software. the typical classes taught by informationists and other librarians are endnote®, pubmed, web of science®, scopus™, and evidencebased practice. the full list of the library’s training topics is available at <http://nihlibrary.nih.gov/resourcetraining/ default.htm?selectedvalue=tutorial>. (one subject reported that training in software programs such as endnote “makes paper writing much easier.”) the informationists also provide updates on library resources and services. (“it would be more work staying up-to-date without the informationist service.”) as with other services, the classes depend on the duties of the staff being served. extramural staff who administer grants to researchers around the country might use scopus and web of science to find bibliometric data for applicants and reviewers. intramural staff who conduct their own research and follow patients in clinical trials may require literature regarding clinical care, social evidence based library and information practice 2009, 4:1 10 science, or bioinformatics, and they are guided through the appropriate resources. 5. effect on library usage in general, the informationist service seems to be associated with higher library usage. themes representative quotations skills of informationists literature skills “[j]ust using tools and giving them to the biologist isn’t really working. critical reading of literature is something that not every graduate student, although they should, is experienced in.” “it’s been very helpful, very valuable to have someone with her expertise.” informationist as portal “in my mind she’s a resource, a go-to person. so if i need something and she can’t do it, she’ll let me know how i can do it.” “i view the informationist service as a pathway to a variety of resources and to different perspectives.” impact of informationists time “that’s her expertise … how to get information and get the right information and get it in a way that’s not onerous, you know, where it takes you forever and a day.” “i don’t have time. i don’t want to spend the time trying to learn how to set up these [alerts] myself…. it’s a waste of my time.” training – impact on practice his “willingness to take on programs and learn them has been immensely valuable. … he taught other people in our group about [informatics software]. we referred [clinical] researchers to him for training.” “it would be more work staying up-to-date without the informationist service.” library usage “i do less and less, i just ask [the informationist].” “[n]urses are using library resources more as a result of informationist contact.” “i am using the library more through [the informationist].” perceptions of informationists informationist as team member “the other thing that completely won us over is that … for you to leave the unit, go down to the library, talk to a reference librarian, maybe get some help or some direction, and then fly back to the unit, it’s just horrible. people simply won’t do it. so _____, being a nurse herself and getting the lay of the land in the clinical areas, she goes to them and sits down with them at their computer (the clinical computer stations) right there in the clinic or the unit that they’re familiar with. she asks them questions and gets information there, logged in on their computer, so it’s always right there.” “it would be much less efficient [without the informationist], dealing with somebody that doesn’t know how our team functions and what are the major issues involved.” table 2. themes and quotations from interviews evidence based library and information practice 2009, 4:1 11 several subjects reported that they or their group were using library resources more since working with the informationist, though in at least two cases, this usage was essentially via the informationist services. in fact, there was one researcher who reported relying almost exclusively on the informationist to supply information needs. since the informationist has been with his group, he stated, “i hardly ever go to the library anymore.” and, in terms of online resources, “i do less and less, i just ask [the informationist].” perceptions of informationist 6. informationist as team member all those interviewed were enthusiastic with having an informationist as part of their team. having an information specialist based in their work environment helped clinicians not only obtain access to needed information immediately, but also learn of relevant new library resources available for their use. subjects reported a number of ways that informationists integrated into the group, including having a desk in the group’s area; attending and/or presenting at seminars, rounds, and journal clubs; and coming to the group’s work area to answer questions or provide training. discussion why do nih staff use informationists? in the modern era of google, atms, and 24hour service stations, what compels a top scientist to trust someone else with part of their research? through key informant interviews, six major roles played by informationists were identified: bibliographic expert, portal to unfamiliar resources, time-saver, teacher, library interface, and teammate. a strongly recurring theme in the interviews was the feeling that our subjects could not keep as close to the literature without an informationist’s help. accordingly, numerous studies (reviewed in dawes and sampson 14; coumou and meijman 57) have found that time tends to be the biggest obstacle to health professionals finding information. when interviewed, one sample of nurses (mcknight 149) reported that time was not only a logistical problem but an ethical one when dealing with patients. furthermore, a qualitative study by brettle, hulme, and ormandy (27) found that even when librarians had helped health care workers retrieve information, time was still the most limiting factor in following through and applying that information to practice. in addition, researchers have cited many other obstacles to finding information (coumou and meijman 57-59; covell, uman and manning 598; mcknight 149; dawes and sampson 14; bennett et al. 3), which could be broadly grouped as either intrinsic (problems forming a search strategy or evaluating results, low expectations of getting results) or extrinsic (poorly designed indexes, unintuitive databases, cost). subjects generally reported that they were using library resources more since acquiring an informationist. this may be the result of raising awareness of available online products and services through training by the informationists. however, it may not have been clear to subjects whether the original question referred to usage of the physical library, the library’s website, or the resources available through the library. another ambiguity was whether accessing resources vicariously through the informationist counted as library usage. it is thus unclear whether “effect on library usage” would be a useful measurement in future research, though it was established that users are using informationists as a third interface to the library’s resources (besides the information desk and the website). evidence based library and information practice 2009, 4:1 12 most of the other themes/roles identified describe functions or abilities of any librarian. the exception is the idea of the informationist as a teammate who joins the physicians/researchers in their place of work. as mentioned above, this was a key feature of the program in the minds of the interviewees, as well as davidoff and florance. in addition, becoming a team member helped the informationist to understand the user group’s research area. by gaining additional knowledge of their work, the informationist could develop training programs, alerting services, and other information services that are specifically geared for their groups’ specialties. the experience of the informationist program at vanderbilt (giuse et al. 250) has also been that although the effectiveness of the program still needs validating, it was better to have an informationist working alongside clinicians than each working independently. interestingly, another defining element of the profession, biomedical training, was not mentioned except by subject #1, who appreciated that the nurse-informationist understood the demands on other nurses’ time. it should be noted, however, that we did not explicitly solicit comments on this point in our interviews, though a separate study by our library (in preparation) shows that subject-area knowledge is indeed valued, particularly by long-term users of the service. interviews allowed the collection of richer and less structured input from subjects, but this method has limitations. for example, our subjects had to step away from work to be interviewed. this allowed them to focus their attention on the interview, but it also effectively removed them from environmental demands and stimuli that might have illuminated how the informationists were used and corroborated subjects’ self-reports. future studies might include observations of the informationistsubject interaction, or at least of subjects engaging in information behavior for their work. another aspect of interviewing is the limitation to the subjects’ perceptions of reality. it could be argued that perception is actually what matters, since a well-stocked but unappreciated library is likely to lack support at budget meetings. to supplement the subjective perceptions from this study, quantitative data on literature search requests are being collected to provide a more comprehensive picture. these interviews helped identify six features important to users. this information confirms that informationists should be ready to serve in a wide variety of roles for their users, roles which require technical skills as well as the ability to maintain longterm collaborative relationships. the six roles could also be the focus of marketing initiatives to additional nih staff, since these are aspects that staff themselves recognized and appreciated. moreover, the data suggest further questions. it would be interesting and helpful to verify, or even quantify, the time saved by informationists. by comparing the time savings and comparative salaries of patrons and informationists, it may be possible to place a monetary value on that time. the interviews revealed that many of the roles could be played by other librarians, and others could be provided by a well-designed and carefully maintained website. the added benefits of personal contact, especially of being embedded with a group of users, should be explored further. for example, past experience (whitmore, grefsheim and rankin 138) has indicated that informationist users are initially somewhat skeptical and request fairly routine tasks, but that they eventually request collaboration on more sophisticated and complex projects. a longitudinal study could explore how informationists and information providers in general, build evidence based library and information practice 2009, 4:1 13 rapport and generate support from their clientele. conclusion informationists bring their librarian skills to research teams, aiding and complementing the skills of other team members. researchers value the time it saves them to have a teammate who finds information of all kinds, as well as one who can teach them to do their own information retrieval. these aspects should be explored in future research and promoted in future marketing. evidence based library and information practice 2009, 4:1 14 works cited annells, merilyn. "triangulation of qualitative approaches: hermeneutical phenomenology and grounded theory." journal of advanced nursing 56.1 (oct. 2006): 55-61. bennett, nancy l., linda l. casebeer, robert kristofco, and blanche c. collins. "family physicians' information seeking behaviors: a survey comparison with other specialties." bmc medical informatics and decision making 5.1 (22 mar. 2005): 9. berg, bruce l. qualitative research methods for the social sciences. 6th ed. boston: pearson/allyn & bacon, 2007. brettle, alison, claire hulme, and paula ormandy. "effectiveness of information skills training and mediated searching: qualitative results from the empiric project." health information and libraries journal 24.1 (mar. 2007): 24-33. burgoyne, louise n., catherine woods, rosarie coleman, and ivan j. perry. "neighbourhood perceptions of physical activity: a qualitative study." bmc public health 8 (mar. 2008): 101. coumou, herma c. h., and frans j. meijman. "how do primary care physicians seek answers to clinical questions? a literature review." journal of the medical library association 94.1 (jan. 2006): 55-60. covell, david g., gwen c. uman, and phil r. manning. "information needs in office practice: are they being met?" annals of internal medicine 103.4 (oct. 1985): 596-9. davidoff, frank, and valerie florance. "the informationist: a new health profession?" annals of internal medicine 132.12 (20 jun. 2000): 9968. dawes, martin, and uchechukwu sampson. "knowledge management in clinical practice: a systematic review of information seeking behavior in physicians." international journal of medical informatics 71.1 (aug. 2003): 9-15. dick, bob. "grounded theory: a thumbnail sketch." 11 june 2005. 29 january 2009. <http://www.scu.edu.au/schools/gc m/ar/arp/grounded.html>. forsythe, diana e., bruce g buchanan, jerome a. osheroff, and randolph a. miller. "expanding the concept of medical information: an observational study of physicians' information needs." computers and biomedical research 25.2 (apr. 1992): 181-200. gavois, helena, gun paulsson, and bengt fridlund. "mental health professional support in families with a member suffering from severe mental illness: a grounded theory model." scandinavian journal of caring sciences 20.1 (mar. 2006): 102-9. gilchrist, valarie j., and robert l. williams. "key informant interviews." doing qualitative research. eds. benjamin f. crabtree and william l. miller. thousand oaks, ca: sage publications, inc., 1999. 71-88. evidence based library and information practice 2009, 4:1 15 giuse, nunzia b., et al. "evolution of a mature clinical informationist model." journal of the american medical informatics association 12.3 (may-june 2005): 249-55. gorman, paul n. "information needs of physicians." journal of the american society for information science 46.10 (1995): 729-36. grefsheim, suzanne f., and jocelyn a. rankin. "information needs and information seeking in a biomedical research setting: a study of scientists and science administrators." journal of the medical library association 95.4 (oct. 2007): 426-34. gruppen, larry d. "physician information seeking: improving relevance through research." bulletin of the medical library association 78.2 (apr. 1990): 165-72. mckibbon, k. ann, douglas b. fridsma, and rebecca s. crowley. "how primary care physicians' attitudes toward risk and uncertainty affect their use of electronic information resources." journal of the medical library association 95.2 (apr. 2007): 138-46. mcknight, michelynn. "the information seeking of on-duty critical care nurses: evidence from participant observation and in-context interviews." journal of the medical library association 94.2 (apr. 2006): 145-51. persson, lars-olof, and anna rydén. "themes of effective coping in physical disability: an interview study of 26 persons who have learnt to live with their disability." scandinavian journal of caring sciences 20.3 (sept. 2006): 355-63. robison, rex r. "informationist education." medical reference services quarterly 27.3 (fall 2008): 339-47. urquhart, christine. "health service users of information." managing knowledge in health services. eds. andrew booth and graham walton. london: library association publishing, 2000. 15-30. weber, scott. "a qualitative analysis of how advanced practice nurses use clinical decision support systems." journal of the american academy of nurse practitioners 19.12 (dec. 2007): 652-67. whitmore, susan c., suzanne f. grefsheim, and jocelyn a. rankin. "informationist programme in support of biomedical research: a programme description and preliminary findings of an evaluation." health information and libraries journal 25.2 (june 2008): 135-41. evidence based library and information practice 2009, 4:1 16 appendix semi-structured interview schedule what has this person done for you lately? what is the value of having an informationist? how would you have obtained the information you needed without an informationist? give an example of how an informationist has helped you. in what environment has an informationist helped you (meetings; lab; rounds)? do you use the library more/less now that you have an informationist? what additional roles do you see for the informationist? evidence based library and information practice using evidence in practice   retroactive collection development: gap identification through citation analysis   kealin m. mccabe research and learning services librarian university of northern british columbia prince george, british columbia, canada email: mccabek@unbc.ca   received: 16 june 2016   accepted: 19 july 2016      2016 mccabe. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. setting   the university of northern british columbia (unbc) is a small research intensive university located in prince george, british columbia (bc), in bc’s central interior. the university opened its doors in 1994, serving as a primarily undergraduate university. since its inception, it has grown to include 18 different graduate degrees. the university is supported by the geoffrey r. weller library, which serves faculty and students doing research on both local and regional campuses.   problem   the 2014/2015 academic year marked the 25th anniversary of the university. as a young institution, our library collection, upon analysis, appears quite immature. in the past, comprehensive in-house studies have been conducted relating to our journal collections, specifically in the areas of economics, political science, and international studies. in the process of building our monograph collection, particular attention had always been paid to the purchase of current items as they relate to course offerings and the research interests of faculty and graduate students. therein lies the problem. past evaluations have shown the effectiveness of our journal collection, but we have failed to assess the strength of our monograph collection in these areas and its ability to meet the research needs of our students and faculty.   as is common with most libraries, collection budgets continue to diminish, while the price of materials continues to skyrocket. librarians must be judicious in their selection process. the abundant funding of the 1990’s is no longer occurring, meaning that librarians’ approach to collection development must be strategic and purposeful (knievel, wicht, & connaway, 2006). based on this evidence, the following exercise was undertaken to assess the strength of our current collection and the potential absence of core texts published prior to the library’s opening in 1994.   evidence   retroactively filling a library’s collection is important. a 2013 western illinois university (wiu) study sought to identify core journal and monograph titles through the examination of phd. level syllabi on a national (united states) level (johnson, stierman, stierman, & clark, 2013). their analysis showed that 24% of the monograph titles cited in top-tier journals had a publication date of 6-10 years, thus “…showing that currency is not always king” (johnson et al., 2013, p. 204).   in order to analyze the gaps in our collection at unbc, we used the following process. thompson reuters journal citation reports (jcr now incite) was used to identify high impact journal titles within the discipline of economics. this list was further refined through the examination of each publications’ scope to determine relevancy, which narrowed the list to 20 journal titles within the area of development economics. the bibliographies of the 50 most cited articles were examined within each high impact journal to identify monograph titles. the year of publication for the high impact articles ranged from 1956-2014 with a median date of 2001.   in total, 1000 bibliographies were examined resulting in the identification of 11,800 monograph titles. exclusion criteria were applied to remove reference texts (e.g. dictionaries, handbooks, encyclopedias), government documents, bulletins, and conference proceedings from the results. endnote desktop citation management software was used to manage the bibliographic records of identified monograph titles. the titles were either exported directly into endnote or entered manually. in instances where articles or their bibliographies were not readily available through our existing collection, the physical items were requested through interlibrary loans (ills).   classification of publication recency was based on fasae’s (2012) categorizations, as illustrated below in table 1. of the identified titles, 43% (n=5,044) were cited two or more times. the average year of publication for these titles was 1983. the findings indicate that within the discipline of development economics, 78% of the cited monograph titles are classified as “not recent”, indicating that older texts play a vital role in informing experts during the research process.     table 1 recency classification a recency classification year recency no. of citations percentage < 5 years very recent 91 2% 6 10 years recent 214 4% 11-15 years not very recent 805 16% 16 years + not recent 3934 78% total 5044 100% a  classification based on fasae (2012).   a list of monograph titles cited 4 or more times (n= 329) in the literature was compiled and compared to the library’s current holdings. the titles on this list were designated as being “core” titles necessary for a strong collection. of these titles, 49% (n=161) were not in our collection.   this process was then applied to faculty publications and graduate student theses.  analysis of these publications confirmed the above findings. faculty publications and graduate student theses relied on monographs with a publication date of 16 years or more, 36% and 44% respectively, thus confirming the importance of older texts in the research process.   implementation   given the evidence, current funds already allotted to the purchase of monographs was used to fill the gaps. monograph funds are allocated to disciplines taught at the university. due to the interdisciplinary nature of the field of development economics, funds from the other disciplines (political science and international studies) were used. this was done without compromising or undercutting one discipline’s fund. all 161 titles were purchased using existing funds. it is important to note that 69% (n=111) of the titles purchased were originally published prior to the university opening its doors.   outcome   the outcome of this study will only become evident with time. to measure the impact of retroactive collection development, the following measures are being proposed. circulation of the retroactively purchased titles will be monitored to measure their impact and use in addition to tracking citations in student theses and faculty publications. further attention will also be paid to requests placed via our ill service. in addition to this, we will continue to monitor ill requests for titles of monographs being requested within the discipline of development economics.   reflection   overall, this process has allowed us to develop a core collection as determined by the experts within development economics instead of relying on traditional book reviews and the occasional patron request. the analysis allowed us to see how influential older texts are in the creation of new scholarship. the overall methodological approach for this project provided great insights, but a few issues were encountered during this comprehensive study. the collection of data was labour intensive and time consuming. bad data was problematic where the records contained incorrect information. this meant that much of the data exported had to be manually manipulated in order to be used. humans are not perfect and neither are their bibliographies. many articles referenced items incorrectly, including errors in author names and dates of publication.   this study relied heavily on the web of science database. while the records for journal article citations were generally accurate, the citations for cited monographs were rife with errors and oftentimes incomplete.   the study began by utilizing refworks as the citation management software, which was user friendly and allowed for the import of duplicate records, which was key to the completion of this study. one of the major obstacles encountered during this time was the library’s decision to discontinue its subscription to refworks in favour of endnote web. this meant that the records already collected had to be exported to endnote and that extra care had to be taken when continuing to export records, as endnote does not allow for duplicates in the same library. this meant that a new process for importing and organizing bibliographic records had to be undertaken in order to ensure that duplicates were recorded accurately.   this approach would prove useful in identifying key reference texts that should be purchased or subscribed to on an ongoing basis. reference texts are oftentimes very expensive and measurement of their use can be difficult, to say the least. the process of exploring the extent of their use through expert citations could prove useful in ensuring that libraries are purchasing titles that are of high caliber and value. while reference texts were excluded from this study, their inclusion could have proven useful in identifying heavily used items that are worth collecting.   this process works well for the exploration of a small and focused area of a collection, specifically, sub disciplines like development economics. this process would not be appropriate to use for a broad sweeping analysis of a collection area in a larger discipline. this type of analysis would be useful in instances where in-depth knowledge of the collection is necessary such as budget requests, the development and support of new advance degree programs, external reviews, or in instances where a college transitions to a degree granting institution.   overall, the evidence collected emphasizes the importance of retroactively filling your book collection for new academic libraries. it demonstrates that emphasis should not be placed on electronic access to journals and the purchasing of newer titles alone. the results of this evidence informed assessment indicate that equal weight should be given in the purchase of older and well-established titles in the creation and maintenance of a well-rounded collection.   references   fasae, j. k. (2012). citation analysis of dissertations and theses submitted to the department of agricultural economics and extension, federal university of technology akure, nigeria. library philosophy and practice. retrieved from http://digitalcommons.unl.edu/libphilprac/      johnson, d. t., stierman, j. k., stierman, j. p., & clark, b. f. (2013). using syllabi to determine the core journal and monographic literature in doctoral finance programs. journal of business & finance librarianship, 18(3), 191-207. http://dx.doi.org/10.1080/08963568.2013.794666   knievel, j. e., wicht, h., & connaway, l. s. (2006). use of circulation statistics and interlibrary loan data in collection management. college & research libraries, 67(1), 35-49. http://dx.doi.org/10.5860/crl.67.1.35        microsoft word class4127 evidence based library and information practice 2008, 3:4 82 evidence based library and information practice classic predicting future information resource utilization under conditions of scarcity: the first cohort study in health sciences librarianship a review of: postell, william dosité. “further comments on the mathematical analysis of evaluating scientific journals.” bulletin of the medical library association 34.2 (1946): 107-9. figure 1: william dosité postell, 1908-1982. pioneer of the cohort design in librarianship permission granted for eblip to publish by william postell, jr. at tulane university, new orleans, la . reviewed by: jonathan d. eldredge associate professor, school of medicine and health sciences library and informatics center university of new mexico albuquerque, new mexico, united states of america email: jeldredge@salud.unm.edu evidence based library and information practice 2008, 3:4 83 received: 19 september 2008 accepted: 14 november 2008 © 2008 eldredge. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – to predict future use of journal titles for making subscription decisions. design – retrospective cohort study. setting – louisiana state university school of medicine library in new orleans. subjects – all library users, estimated to consist of primarily faculty members or their designees such as research assistants. methods – estelle brodman’s previous citation analysis and reputational analysis (1944) that produced a list of eleven topranked physiology journal titles served as the catalyst for postell’s retrospective cohort study. postell compiled data on all checkouts for these specific eleven journal titles in his library for the years 1939 through approximately 1945. main results – postell performed a spearman rank-difference test on the rankings produced from his own circulation use data in order to compare it against journal title rankings produced from three other sources: (1) citation analysis from the references found in the annual review of physiology based upon a system pioneered in 1927 by gross and gross; (2) three leading national physiology journals; and, (3) a reputational analysis list of top-ranked journals provided by the faculty members at the columbia university college of physicians and surgeons department of physiology. postell found a relatively high correlation (.755, with 1.000 equaling a perfect correlation) between his retrospective cohort usage data and the reputational analysis list of top-ranked journals generated by the columbia faculty members. the two citation analyses performed by brodman did not correlate as highly with postell’s results. conclusion – brodman previously had questioned the use of citation analysis for journal subscription purchase decisions. postell’s retrospective cohort study produced further evidence against basing subscription purchases on citation analysis. postell noted that the citation analysis method “cannot always be relied upon as a valid criterion” for selecting journals in a discipline. commentary libraries currently devote major portions of their annual budgets toward purchasing journal subscriptions. similarly large budgetary outlays motivated gross and gross to pioneer their use of a rudimentary form of citation analysis based upon all of the references found in the single year 1926 in the journal of the american chemical society to identify top-ranked journals in chemistry (385-9). estelle brodman documented 22 derivative applications of the citation analysis method published since 1927, noting that some researchers had enlarged their coverage to include multiple leading journals as sources for to obtaining the pool of references for their citation analyses (479). brodman conducted a reputational analysis list of top-ranked journals provided by the faculty members at the columbia university evidence based library and information practice 2008, 3:4 84 college of physicians and surgeons department of physiology as an alternative method to the citation analysis. brodman concluded, on the basis of her spearman rank-difference correlations, that the citation analysis method generally led to “unscientific” and “untrustworthy results” when making journal subscription decisions. brodman concluded further that “in spite of these extremely grave drawbacks, the (citation analysis) method will probably continue to be employed by librarians until the library profession is presented with a better one.” william d. postell conducted the first cohort study in health sciences librarianship by measuring what journal titles his library users actually checked out. he correlated his ranked frequency usage data with brodman’s reputation analysis and citation analyses, finding the latter lacking in reliability. in the process, he provided our profession with one of its most durable designs for applied research. by contemporary eblip critical appraisal standards, postell’s 1946 retrospective cohort study does not meet most standards of methodological rigor. yet, postell shifted librarians’ emphasis away from citation analysis with this article by introducing what would become one of our profession’s most enduring applied research methods: the cohort study design. postell also indirectly transitioned librarianship away from the citation analysis method, with its limited validity for journal selection (garfield), while offering a less authorityoriented alternative to brodman’s reputational analysis. how does postell’s 1946 article stand up to two contemporary critical appraisal checklists? booth and brice’s critical appraisal checklist prompts reviewers to evaluate any reported research results in accordance with the three major areas: (1) closeness to the truth, (2) credible and replicable results, and (3) applicability to professional practice (108-9 and 263). postell’s study fulfills most of the booth and brice checklist’s first criteria area by offering a close representation of the truth by providing a clear question, couching it in the context of other studies, comparing his results to relevant studies, and representing all library users. yet, postell’s study does not indicate who collected the data so it appears to not meet this specific criterion for the first area for critical appraisal. postell’s classic study clearly does not meet booth and brice’s second broad criteria area of credibility and replicability. postell does not even address these issues, including the area of documenting any study limitations. a biographical account of postell’s study (eldredge, “scc milestone”) suggests that postell conducted his study during the unusual historical circumstances of an intensively rapid medical school training program at lsu during world war two, thereby raising the possibility that his study was subject to “history” as a threat to validity. this kind of threat to validity occurs when researchers study a population during atypical circumstances that might cause behavior to deviate from the norm and therefore be non-representative of most situations (miller and salkind; shadish, cook and campbell; neuman). characterizing u.s. medical school libraries’ practices during world war ii, walker described “the accelerated (academic) program, reduced staff, the increasing number of ‘our’ doctors who are away and of other doctors who are here, the changed emphasis in teaching and research…” (326), suggesting monumental deviations from normal activities. she also notes, of special relevance to this classic review, the extra emphasis paid during this era to the subject areas of “physiological and chemical research.”(329). for reasons of this history threat to validity, it would be difficult for evidence based library and information practice 2008, 3:4 85 anyone to generalize the usage behavior that postell measured to users’ behavior during a different historical period or set of circumstances. the third area of booth and brice’s checklist, applicability to one’s own practice, suggests again that the unique historical circumstances of postell’s study, coupled to its occurrence at such distant date, cannot validate this first cohort study’s results for modern application to practice. postell’s study consequently does not meet most of booth and brice’s modern standards of critical appraisal. glynn’s 2006 critical appraisal checklist prompts reviewers to examine the four broad areas of (1) population, (2) data collection, (3) study design, and (4) results in a research study. postell’s study appears to fulfill all of the relevant or known criteria in the first population area, except for the possible aforementioned problems, again, with a history threat to validity due to the unusual war time circumstances affecting the population of journal users. postell’s study remains silent on most issues raised by the second data collection category of glynn’s checklist whereas postell’s choice of a cohort study for the third category in the glynn checklist seems appropriate for answering the specific research question. this study does well on the fourth results section of the glynn checklist, except that postell apparently never delved into the issues of confounding variables or external validity. overall, then, postell’s 1946 study does not fare well when appraised by glynn’s 2006 checklist. given its poor performance when evaluated by these two contemporary critical appraisal checklists, how should we interpret then the significance of postell’s 1946 classic retrospective cohort study? principally, we should recognize that postell ingeniously introduced the previously unused cohort research methodology in health sciences librarianship. the cohort design subsequently became a powerful tool for both collection development and library education practices. historically, the cohort study design can be traced back to james lind’s first study in 1747 of british sailors contracting scurvy, louis’ 1835 bloodletting outcomes study in paris, and semmelweiss’ 1848 childbed fever study in vienna. yet, by even the 1930s, the cohort study design still was not a widely used research method in any field (eldredge, “cohort studies”). postell published on diverse topics such as medical history (“principles”), library instruction for medical students (“formal training”; “further notes”), and library public relations (“stimulating interest”). postell’s historical work on 19th century medicine in louisiana (“principles”) offers the most likely inspiration for his adapting the cohort study design to health sciences librarianship. postell noted in this 1942 article that 19th century physicians in louisiana were influenced by both american and french medical approaches. physicians’ practice of bloodletting declined in louisiana, however, when pierre charles alexandre louis’ research in paris declared it ineffective. although postell never refers to it by its formal modern name, louis had used the cohort study design to dispute the practice of bloodletting. postell later most likely adapted the population, exposure, outcome(s) sequential structure of louis’ cohort study design to answer his own prediction form of research question. postell’s contribution to applied research seems all the more remarkable when one surveys the health sciences literature in the bulletin of the medical library association during the years 1911-1946. ballard’s call for uniformity and comparability of library statistics focused upon holdings data, while barely mentioning usage data, except apparently as gross data only in service evidence based library and information practice 2008, 3:4 86 demonstrating that library collections were used at all. osler’s “the science of librarianship” solely revolved around the bibliography, classification, and the general management aspects of libraries. while osler did praise american libraries for their accessibility, a nod to the efforts of john cotton dana (kingdon), he did not mention usage data in any connection to the “science” of our field. jenkin’s classic on cost benefit analysis of highly cited journals against price data never contemplated use data in her analysis. we can be certain that postell’s cohort study led to numerous replications and adaptations within health sciences librarianship during the next two decades, principally through information resource usage studies (kilgour, “annual report”; morse, beatty & hodge; keys, kilgour, “use: part i”; kilgour, “use: part ii”; kurth; fleming and kilgour; staudt, halbrook, and brodman). today, the cohort study design occupies the pinnacle of single study for evidence in the eblip levels of evidence for prediction questions (eldredge, “evidencebased librarianship”). therefore, we need to credit william postell for his remarkable introduction to health sciences librarianship of one of its perennial methods of modern applied research now used frequently in service to major information resource purchasing decisions. works cited ballard, james f. “uniformity in library statistics.” bulletin of the medical library association 5.2 (1915): 21-6. 24 nov. 2008. <http://www.pubmedcentral.nih.go v/tocrender.fcgi?iid=7237>. booth, andrew, and anne brice. “appraising the evidence.” evidence-based practice for information professionals. eds. andrew booth and anne brice. london: facet publishing, 2004: 104118. [also in north america reprinted in introduction to health sciences librarianship. ed. sandra wood. binghamton, ny: haworth press, 2007: 241-70.] brodman, estelle. “choosing physiology journals.” bulletin of the medical library association 32.4 (1944): 47983. 24 nov. 2008. <http://www.pubmedcentral.nih.go v/tocrender.fcgi?iid=6148>. eldredge, jonathan. “cohort studies in health sciences librarianship.” journal of the medical library association 90.4 (2002): 380-92. 24 nov. 2008. <http://www.pubmedcentral.nih.go v/articlerender.fcgi?artid=128954>. eldredge, jon. “evidence-based librarianship: levels of evidence.” hypothesis 16.3 (fall 2002): 10-13. 24 nov. 2008. <http://research.mlanet.org>. eldredge, jonathan d. “scc milestone in ebl history. south central connection. 2003 february; 13 (2): 10, 14. 24 nov. 2008. <http://www.sccmla.org/newsletter/ connection_archive.htm>. fleming, thomas p. and frederick g. kilgour. “moderately and heavily used biomedical journals.” bulletin of the medical library association 52.1 (1964): 234-41. 24 nov. 2008. <http://www.pubmedcentral.nih.go v/tocrender.fcgi?iid=6294>. garfield, eugene. “how isi selects journals for coverage: quantitative and qualitative considerations.” current contents 22 (28 may 1990): 185-93. 24 nov. 2008. evidence based library and information practice 2008, 3:4 87 <http://garfield.library.upenn.edu/v olume13.html>. glynn, lindsay. “a critical appraisal tool for library and information research.” library hi tech 24.3 (2006): 387-99. gross, p.l.k., and e.m. gross. “college libraries and chemical education.” science 66.1713 (october 28, 1927): 385-9. jenkins, r.l. “cost analysis of medical periodicals.” bulletin of the medical library association 22.3 (1934): 11523. 24 nov. 2008. <http://www.pubmedcentral.nih.go v/tocrender.fcgi?iid=7195> keys, thomas e. applied medical library practice. springfield, il: charles c. thomas publisher, 1958. kilgour, frederick g. annual report of the yale university medical library 1952/1953. new haven, ct: yale university, 1953. kilgour, frederick g. “use of medical and biological journals in the yale medical library. part i. frequently used titles.” bulletin of the medical library association 50.3 (1962): 42943. 24 nov. 2008. <http://www.pubmedcentral.nih.go v/tocrender.fcgi?iid=6288>. kilgour, frederick g. “use of medical and biological journals in the yale medical library. part ii. moderately used titles.” bulletin of the medical library association 50.3 (1962): 4449. 24 nov. 2008. <http://www.pubmedcentral.nih.go v/tocrender.fcgi?iid=6288>. kingdon, frank. john cotton dana. newark, nj: the public library and museum, 1940. kurth william h. survey of the interlibrary loan operation of the national library of medicine. bethesda, md: national library of medicine, 1962. miller, delbert c, and neil j. salkind. handbook of research design and social measurement. 6th ed.. thousand oaks, ca: sage publications, 2002, 49-51. morse, e.h, beatty, w.k., & hodge h.m. “annual report on the library.” transactions and studies of the college of physicians of philadelphia 21.4 (1954): 138-50. neuman, william lawrence. social research methods: qualitative and quantitative approaches. boston: allyn and bacon, 2006. osler, sir william. “the science of librarianship.” bulletin of the medical library association 7.4 (1918): 70-4. 24 nov. 2008. <http://www.pubmedcentral.nih.go v/tocrender.fcgi?iid=7252>. postell, william dosité. “the formal training of medical students in the use of the library.” journal of the association of american medical colleges 15.4 (1940): 241-4. postell, william dosité. “stimulating interest in a medical library.” bulletin of the medical library association 29.3 (1941): 141-6. 24 nov. 2008. <http://www.pubmedcentral.nih.go v/tocrender.fcgi?iid=7166>. postell, william dosité. “the principles of medical practice in louisiana during the first half of the nineteenth century.” bulletin of the medical library association 30.3 (1942): 191-7. 24 nov. 2008. evidence based library and information practice 2008, 3:4 88 <http://www.pubmedcentral.nih.go v/tocrender.fcgi?iid=6138>. postell, william dosité. “further notes on the instruction of medical school students in medical bibliography.” bulletin of the medical library association 32.2 (1944): 217-20. 24 nov. 2008. <http://www.pubmedcentral.nih.go v/tocrender.fcgi?iid=6146>. shadish, william r., thomas d.cook, and donald t. campbell. experimental and quasi-experimental designs for generalized causal inference. boston, ma: houghton mifflin company, 2002. staudt, cecilia, barbara halbrook, and estelle brodman. “a clinical librarians’ program – an attempt at evaluation.” bulletin of the medical library association 64.2 (1976): 236-8. 24 nov. 2008. <http://www.pubmedcentral.nih.go v/tocrender.fcgi?iid=6340>. walker, louise. “the medical school library in wartime.” bulletin of the medical library association 31.4 (1943): 326-31. 24 nov. 2008. <http://www.pubmedcentral.nih.go v/tocrender.fcgi?iid=6144>. microsoft word es_1798 evidence based library and information practice 2008, 3:3 83 evidence based library and information practice evidence summary personalized information service for clinicians: users like it a review of: jerome, rebecca n., nunzia bettinsoli giuse, s. trent rosenbloom, and patrick g. arbogast. “exploring clinician adoption of a novel evidence request feature in an electronic medical record system.” journal of the medical library association 96.1 (jan. 2008): 3441, with online appendices. reviewed by: gale g. hannigan professor, texas a&m medical sciences library college station, texas, united states of america e-mail: g-hannigan@tamu.edu received: 05 june 2008 accepted: 16 june 2008 © 2008 hannigan. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – to examine physician use of an evidence-based medicine (ebm) literature request service available to clinicians through the institution’s electronic medical record system (emr). specifically, the authors posed the following questions: 1) did newly implemented marketing and communication strategies increase physicians’ use of the service? 2) how did clinicians rate the relevance of the information provided? 3) how was the information provided used and shared? design – ten-month, prospective, observational study employing a questionnaire, statistics, a focus group, and a “before and after marketing intervention” analysis. setting – adult primary care outpatient clinic in an academic medical centre. subjects – forty-eight attending and 89 resident physicians. methods – in 2003, a new service was introduced that allowed physicians in the adult primary care center clinic to request evidence summaries from the library regarding complex clinical questions. contact with the library was through the secure messaging feature of the institution’s evidence based library and information practice 2008, 3:3 84 electronic medical record (emr). from march through july 2005, the librarian employed “standard” publicity methods (email, flyers, posters, demonstrations) to promote the service. a focus group in july 2005 provided feedback about the service as well as recommendations about communicating its availability and utility. new communication methods were implemented, including a monthly electronic “current awareness” newsletter, more frequent visits by the librarian during resident clinic hours, and collaborations between the librarian and residents preparing for morning report presentations. at the end of the study period, a 25-item web-based questionnaire was sent to the 137 physicians with access to the service. main results – during the 10-month study period, 23 unique users submitted a total of 45 questions to the ebm literature request service. more questions were from attending physicians than residents: 36 (80%) vs. 9 (20%). at least one of the 23 users asked 12 (26%) of the questions. utilization did not significantly change after the mid-study intervention. at the end of the study, 48 physicians (35%) completed the survey (32 attending physicians and 16 residents). while 94% of the respondents indicated awareness of the service, only 40% indicated using it. the 19 who used the service, on average, agreed that the information provided was relevant and “sometimes leads to a change in my clinical practice” (p.37). those who indicated that they shared the information (n=15) mostly did so with other attending physicians and residents, but also mentioned sharing with fellows, patients, and nurses. information was typically shared verbally but also by distributing a printout, forwarding by email, and forwarding within the emr message system. the information was used primarily for general self-education, instruction of trainees, and confirmation of a current plan. conclusion – the newly implemented marketing and communication strategies did not significantly increase the use of the ebm literature request service. those who used the service found it relevant and often shared the information with others. based on a small number of respondents and survey information, the librarian-provided ebm literature request service was “wellreceived” (39). commentary jerome et al. present a detailed account of their study and results. there is extensive data analysis and many comparisons between attending physicians’ and residents’ responses with statistical significance levels reported, but the number of respondents is too small to be conclusive. the fact that one physician accounted for more than one-quarter of the questions suggests that at least one person found the service very useful. as an early adopter, this physician, as a role model, may be the best marketing strategy for the service. in their introduction and discussion, the authors emphasize the importance of clinicians’ access to current information. by embedding the ebm literature request service in the emr, the authors have taken an important step in integrating convenient access to information services into the clinical context using technology. by the authors’ account, the service is designed to assist in answering complex clinical questions. the example of a request and response given in the appendix has to do with the natural history of syringomyelia. what the librarian provided was a summary based on several articles from multiple databases as well as links to patient education materials on the web. lack of awareness of the service does not seem to be an issue. during this study, evidence based library and information practice 2008, 3:3 85 physicians were given gift cards for their participation and considerable effort was invested in promoting the service, yet response and use were low. low use of the service may be due to a population of physicians used to answering their own clinical questions (something many of us are training them to do), or an environment where physicians don’t have many “complex” questions (the outpatient clinic), or the increased availability and use of point-of-care clinical information resources that provide sufficient, if not exhaustive, information (another tool made possible by technology). complex clinical questions can take a lot of time to answer; the authors indicate from three to more than ten hours. satisfaction with the service was high, but it often is when there is no direct cost to the user. but, when asked to pay for the service and the time actually invested, would the program or library director decide it was “worth it”? many clinical librarian programs that reported high user satisfaction and even demonstrated positive results (1) are no longer in existence either due to general library budget cutbacks or the unwillingness of the department served to assume the real cost of the service. the authors discuss scalability, but cost-benefit is also important. an interesting future study might be to investigate the cost-benefit to all participants (clinicians, library, patients) in the provision of specialized information services. work cited weightman, a.l. and j. williamson. “the value and impact of information provided through library services for patient care: a systematic review” health information and libraries journal 22 (2005):4-25. evidence summary   academic librarians at institutions with lis programs assert that project management training is valuable   a review of: serrano, s. c. & avilés, r. a. (2016). academic librarians and project management: an international study. portal: libraries and the academy, 16(3), 465-475. http://dx.doi.org/10.1353/pla.2016.0038   reviewed by: elaine sullo coordinator, information and instructional services himmelfarb health sciences library the george washington university washington, district of columbia, united states of america email: elainej@gwu.edu   received: 9 may 2017    accepted: 11 aug. 2017      2017 sullo. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to investigate academic librarians’ project management education and training, project management skills and experiences, and perceptions of project management courses within the library and information science (lis) curriculum.   design – online questionnaire.   setting – 70 universities worldwide with lis programs and at least one project management course.   subjects – 4,979 academic librarians were invited to complete the online questionnaire.   methods – from the identified institutions, the authors invited academic librarians to participate in a 17-question survey via e-mail. the survey was available in both english and spanish and was validated via a pilot trial. a total of 649 individuals participated, for a response rate of 13%. the survey included questions related to geographic region and institution affiliation, university education and librarian training associated with project management, project participation and use of project management software or methods, and project management courses in lis curriculums, and a final open-ended comment section.   main results – of the 649 librarians who participated in the survey, 372 were from north and south america (58%). the next highest number of responses came from europe (38%), followed by low response rates from africa, asia, and oceania. respondents reported working in a variety of library departments and identified themselves as being one of a director or manager, assistant librarian, or library page. of the 436 respondents who reported having a university degree, 215 attended an lis master’s level program, and 12 studied at the doctoral level. the majority of respondents indicated they have had training in project management, participating in formal coursework, conferences, webinars, or other self-directed learning methods. of the 459 academic library staff responding to the question, 40% considered project management courses of “high importance in the university curriculum” and 26 % responded that project management courses were “extremely important in their field of expertise and working environment” (p. 472). the consensus among participants was that project management courses should be included in both undergraduate and graduate level lis curricula.   conclusion – the high participation of librarians in project management, compared to the limited formal education received, suggests that courses in project management, including software and methodology, are needed in lis university curricula. additionally, less than 40% of academic librarian survey respondents were trained in lis; other professions are working as librarians and therefore may have insufficient knowledge and skills to manage the projects they direct. the research results confirm the relationship between strategic planning and project management skills. the authors conclude that universities should revise their lis curricula to include and require additional project management courses.   commentary    as noted by the authors, many articles have been published that describe project management experiences of librarians and information professionals, librarians’ perceptions of project management tools, and the use of project management software and techniques. this article is a follow up to a previous article by the authors (aviles, serrano, & simon, 2014), and adds to the current knowledge on the topic.   the study was evaluated using the cristal checklist for appraising a user study (cristal, n.d.). the researchers attempted to study a focused issue in terms of the population studied (academic librarians) and the outcomes measured (the authors conducted both quantitative and qualitative analysis of survey data). the authors surveyed “academic librarians,” but included librarians with the roles of director or manager, assistant librarian, and library page. for the purposes of this study, it seems that library pages are considered to be academic librarians, which is not the case in many institutions.   the survey instrument, while described in the text, was not included in the article, which makes for a study that cannot be easily replicated. furthermore, the article does not indicate what platform was used to administer the survey (e.g., survey monkey, zoomerang, etc.); it only states that the survey was “an online self-administered questionnaire.”   the librarians surveyed in this study were employed at academic institutions with lis programs with at least one course on project management, but the authors did not provide a rationale for including only librarians at institutions with lis programs. for a broader perspective, the authors could replicate this study by including librarians at all academic institutions.   another limitation of this study was the low number of responses from librarians in specific geographic regions, namely africa, asia, and oceania. the authors describe their process for identifying the 70 participating universities but it is not clear who was invited to participate. the article does not divulge how many invitations were sent to each continent or geographic area. additionally, because the survey was only available in english and spanish, some librarians may have disregarded the invitation to participate due to language barriers. although the authors stated that they conducted an international study, the low number of responses and lack of data from many geographic regions brings this point into question.   the survey results appear to demonstrate that the majority of respondents are involved in project work, although many did not receive project management training during their lis programs.  from this, the authors concluded that project management courses should become part of lis curricula and possibly a requirement, but there is no clear evidence in the study data to support the authors’ conclusions.   it is difficult to consider applying the researchers’ conclusions to other academic libraries based on the shortcomings of this research. although study respondents may have indicated that project management courses are valuable, the deficiencies in the study methodology, especially the lack of participating international universities, does not provide clear evidence for the authors’ deductions, and does not allow for the study’s findings to transfer to other academic institutions.   references   avilés, r. a., serrano, s. c., & simón, l. f. r. (2014). international presence of project management in the university curricula in library and information science. qualitative and quantitative methods in libraries, 2, 367-375. http://www.qqml.net/   cristal checklist for appraising a user study. (n.d.) in nettingtheevidence.pbwiki.com. retrieved from http://nettingtheevidence.pbwiki.com/f/use.doc     research article   video game equipment loss and durability in a circulating academic collection   diane robson games and education librarian university of north texas libraries denton, texas, united states of america email: diane.robson@unt.edu   sarah bryant reference and instruction librarian western wyoming community college rock springs, wyoming, united states of america email: sbryant@westernwyoming.edu   catherine sassen principal catalogue librarian university of north texas libraries denton, texas, united states of america email: catherine.sassen@unt.edu   received: 4 jan. 2023                                                                  accepted: 17 may 2023      2023 robson, bryant, and sassen. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30294     abstract   objective – this article reviewed twelve years of circulation data related to loss and damage of video game equipment, specifically consoles, game controllers, and gaming peripherals such as steering wheels, virtual reality headsets, and joysticks in an academic library collection.   methods – the authors analyzed data gathered from game equipment bibliographic and item records. only data related to the console system, game controllers, and peripherals such as steering wheels, virtual reality headsets, and joysticks were evaluated for rate of circulation, loss, and damage. cables and bags were not evaluated because the replacement cost for these items is negligible when considering long-term budgeting and maintenance of a game collection.   results – the majority of video game equipment can be circulated without unsustainable loss or damage. the library has been able to continue circulating video game equipment without undue replacement costs or loss of access for its patrons.   conclusion – although equipment will occasionally break or be lost, libraries should not let this unduly affect consideration when starting a video game collection.     introduction   although some academic libraries are starting to add video game collections to support research and recreation, there still seems to be a reluctance to collect items outside the norm. these items do require different skill sets to manage but libraries already deal with materials and software that require some specialization. although loss and damage for these items should be a concern, the costs related to collecting game equipment are no more exorbitant than other library technology.   video games are a big part of our culture with 66% of americans playing video games weekly; 83% of players begin new relationships and develop friendships through play (entertainment software association, 2022). in addition to recreation, these collections serve scholars in their study and creation of games. libraries should not let concerns about loss, damage, and other difficulties related to these materials hinder adding a collection that can supplement recreation, engagement, innovation, and scholarship in the library.   the university of north texas (unt) media library has been circulating a growing video game collection for over ten years. this article provides details about the game equipment in this collection. game consoles and controllers are some of the biggest costs for a collection. this research will examine the durability, lifespan, loss, management, and maintenance of these items. this evidence will be useful to libraries considering the viability and costs of establishing, expanding, or managing a collection in their own library.   given the limited life span of video game equipment and the expense of replacement components, managers of video game collections stand to benefit from detailed research on equipment durability in a circulating collection. however, none of the studies found in the literature review presented such research. the present article is designed to fill this research gap.   literature review   administering an academic library video game collection involves a variety of challenges ranging from changes in game design, game technology, and purchasing arrangements to evolving opportunities in support of educational and recreational needs on campus. as robson et al. (2020) noted, “managing challenges related to content, access, equipment, space, and outreach, with the goal of effectively supporting students, staff, and faculty can be exasperating but is also exciting and rewarding” (p. 3).    although most of these challenges have been covered extensively in the literature, libraries should devote more attention to equipment, considering its crucial role in the successful operation of a video game collection. the scope of this literature review is limited to the maintenance and durability of video game equipment in libraries. this review draws on research reports, feature articles, and publications providing guidance on library video game collection management. for more information about academic library video game collections in the contexts of collection development, library instruction, outreach, cataloguing, assessment, gaming spaces, and virtual and augmented reality, see robson et al. (2020).   equipment durability and longevity   video game components inevitably wear out with continued use. a library may decide to purchase extra components as backups to save time in locating them when replacements are needed (williams & chimato, 2008). other options may include repairing equipment and using 3d printing to create replacement parts (panuncial, 2019).   equipment compatibility is a concern in video game collections because games that can be played on older consoles may not necessarily be played on newer consoles (cross et al., 2015). a library may collect legacy consoles to mitigate this situation. it is possible to find older equipment on ebay or in pawn shops (robbins, 2016); however, the library should prohibit the circulation of these consoles, considering the historic importance of equipment that is no longer in production (robbins, 2016). the library also may restrict access to games on cartridges because of preservation concerns. a strategy to provide access to older games while preserving legacy consoles would be to use video game emulators (cross et al., 2015).   care and handling   the condition of a video game is influenced by the environment in which it is housed and the way it is handled (byers, 2003; leblanc, 2021). environmental conditions include temperature, humidity, moisture, solvents, light exposure, dust, debris, and smoke. handling effects include scratches, smudges, marking, labels, and wear. byers (2003) and mcdonough et al. (2010) provide guidance on mitigating these factors.   circulation policies and practices affect the integrity of video game collections (buller, 2017; goodridge & rohweder, 2021). libraries should develop policies covering lending and use issues as well as replacement fees to document expectations for users. after each circulation, staff should ensure that all equipment components have been returned. they should check discs for damage and clean them. other post-circulation tasks may include verifying that consoles function properly and removing any data left by players. circulating collections must have appropriate physical processing to protect the items and facilitate the check-in process (robson et al., 2017).   aims   in this article, we discuss the durability and loss of video game consoles, controllers, and peripherals in an academic video game collection over twelve years, as well as management and maintenance decisions needed to sustain equipment and increase its longevity. can an academic library sustain a video game collection, or will loss and damage be unsustainable?   durability is a consideration for all purchases but is particularly important for non-consumable items. video game equipment will be set up and taken down, held for hours in sometimes sweaty hands, and dropped. there is no standard metric for game equipment durability in a library setting. use determines longevity and use varies greatly between the user and the game played. the ability to repair wear and tear is also a consideration. there is no easy answer and often budgets will determine each library’s ability to sustain this type of collection.   loss, defined as items missing or not returned, is another question entirely. academic libraries conducting inventory projects have reported a variety of loss rates. for example, an inventory conducted at the university of mississippi libraries found losses ranging from 9% to 16%; this library set its acceptable loss rate at 4% (greenwood, 2013). a library inventory at seton hall university in the 1990s estimated a 14% missing rate for their collection with no desired loss rate stated (loesch, 2011).   the acceptable rate of loss is specific to each collection, its size, and user needs. each library will need to determine if their loss rate hinders research, instruction, and play, and if their budget can sustain a game collection through such losses. this research does not intend to determine a universal rate of loss for libraries with game collections but will examine current loss rates at the unt media library.   methods   we limited the scope of data to consoles, controllers, and game peripherals such as wii pads and steering wheels. this research study focused on this equipment because the replacement costs of a console, controller, or peripheral are much higher than plugs, cables, or headphones. the durability of these materials is important when considering the long-term costs of maintaining a game collection.   we collected data from bibliographic and circulation records for 497 consoles, controllers, and peripherals. these data included circulation statistics generated automatically and notes about loss and damage from december 1, 2009, through the data capture on february 1, 2022. information in library item records included the item create date, last check-in (return date), total checkout, renewals, and status. the lifespan was calculated from the item record create date and last check-in date for lost/paid/damaged items. the available lifespan was calculated with the item record create date and the data capture date (february 1, 2022). the total circulation was the sum of the total checkout and renewals values. the status included available, lost and paid, lost, billed, on search, missing, and discarded. the statuses for lost and paid, lost, billed, on search, and missing were all bundled into billed/lost/paid because these items were not returned to the library. discarded items were either those returned with either general wear and tear or consoles damaged by patrons. notes included content added to the bibliographic records for non-consumable items, documenting purchases and any loss or damage that happened over an item's lifetime in the library, as well as the circulation count at the time of discard. legacy collection items were not included in the durability values for this review. the legacy collection consists of older equipment that is considered obsolete or difficult to replace.  overview   the unt media library is one of four libraries that serve the educational and research needs of about 48,600 faculty, staff, and students across two campuses in denton, texas. the media library houses non-print, audiovisual, tabletop, and video game collections. the video game collection began in 2009 with a small grant and has grown to include services and collections to support student recreation, research, and coursework. this collection is used for student and staff programming, coursework, and university game-related initiatives such as esports and a game studies and design degree.   the media library is a collaborative space that encourages engagement around play. in 2018–2019[1] the media library served 104,890 patrons at its circulation desk and in its spaces. there were 12,427 pc reservations and 10,456 game station reservations. game-related equipment was checked out 7,427 times. before 2009, this space was primarily a quiet space for viewing audiovisual reserves. as audiovisual collections moved online, viewing carrels were no longer needed so their space needs decreased. this allowed the game space to increase to include in-house reservable space with 10 console stations, 22 pcs, virtual reality devices, and tables for gaming and play. most of the collection circulates out to faculty, staff, and students with a few exceptions for older, costly, or larger devices.   in late 2009, the video game and console collection consisted of modern consoles. it included a nintendo wii, a playstation 3, an xbox 360, controllers for each console, and a few peripherals such as guitars, a keyboard, and a steering wheel. today the collection includes modern consoles such as the xbox series x and playstation 5, as well as older legacy consoles and equipment such as the wii, playstation 3, and xbox 360.   obsolescence is a concern with library collections. new consoles are released about every three to five years which brings obsolescence into consideration much earlier than some other library materials.   the media library embraced this cycle of renewal with its film collection and decided to do the same with the game collection. not only have library staff developed procedures to keep our collection of seventh-generation consoles (wii, xbox 360, ps3) in the collection; we have reached even further into the past to develop a legacy collection of older equipment to support research and instruction. the game collection now houses equipment that spans the second console generation (1976–1992; gallagher & park, 2002) to modern devices in the ninth console generation (2020–present).   the equipment in this collection is curated to meet the needs of faculty, staff, and students. several different budgets are used to purchase and maintain the collection. all video game content, i.e., discs and cartridges, is purchased with the general materials budget. a yearly game equipment budget is used to purchase new equipment and legacy items. the library accepts gifts that have helped a collection of older content and consoles grow without additional costs. smaller items, such as batteries, bags, and cleaners, needed to maintain the longevity of the collection are purchased through a supply budget.   the library’s definition of a non-consumable durable item vs. a consumable item has shifted as staff learned how to manage this type of collection, with the budget reflecting the need to purchase some items yearly because of wear and tear. consumables such as batteries, bags, cables, and headphones degrade much quicker than non-consumable items such as console systems, controllers, and peripherals. each year new purchases, replacements, and maintenance are considered when determining purchases.   a video game collection, like any new collection, does cost money, and library staff had fears about unsustainable loss. early procedures played to these fears by requiring patrons to sign extra documentation reiterating their responsibilities at each checkout. these additional procedures added time to check out and did not minimize loss, so these procedures were relaxed to a simple checkout/check-in in the library system in 2014. details in the circulation record are sufficient to note responsibility for materials.   damage is another consideration, but a video game collection is much like any other collection. there will be damage and parts will break. this means understanding that some parts do break and the library will need to expand the idea of what a consumable is to include additional easily replaced items like cables and headphones. the library’s long-term goal is to mitigate unsustainable loss and damage with proper procedures and maintenance.   collection   the unt media library includes a collection of circulating content, consoles, and peripherals that circulate outside the library to faculty, staff, and students. the reserve collection circulates in-house and supports play in library game spaces. the legacy collection, which includes older consoles from generations two through six, is used in-house for research and instruction only. the game collection continues to grow across generations with consoles from the eighth generation, specifically the nintendo switch and playstation 4, still in high demand.   this collection circulates like most other physical items in a library. students can check out three items for 3 days. faculty and staff can check out 10 items for 7 days. circulating equipment may be placed on hold. faculty, staff, and official student groups are allowed to book materials. a booking holds the item for a specific date and ensures items are available for class or group events.   consoles the overall data displayed in figure 1 show that the circulating console collection has not suffered unsustainable loss or breakage. over the life of this collection, 6% of consoles were discarded because of damage. none of this damage was intentional; it was merely wear and tear. no specific console system suffered more damage than another.   analysis shows 10% of consoles were lost, paid, or billed. although this loss rate is fairly high, it has not hindered the library’s ability to provide equipment to access content. seen another way, the library only lost one copy out of the four available for the nintendo 3ds.   unlike a book on an open shelf, these items are held more securely when not checked out, so loss is often tied to a specific patron and the library has a chance of recovering the cost of the item. most losses are due to students withdrawing from school while an item is in their possession. sudden closures from covid-19 increased the recent loss rate, as students who had items did not return to university or re-enroll.   figure 1 cumulative total for consoles from 2010–2022 (available, billed/lost/paid, damaged/discarded). there is a slight degree of uncertainty in data related to the difference in notes added to the bibliographic records related to loss and damage.   table 1 median values and statistics for consoles (2010–2022) consoles   median lifespan/ years median circulation oldest console/ years highest circulation loss damage total consoles nintendo               3ds 3 78 6 184 0 1 4 ds 10 237 10 237 0 0 2 switch 3 59 4 576 0 0 9 wii 4 46 11 230 1 0 9 wii u 4 24 7 223 1 2 10 playstation               playstation vita 6 101 6 101 0 0 1 playstation portable 6 43 9 122 0 0 3 playstation 3 6 233 11 364 1 1 7 playstation 4 5 45 6 293 0 0 10 xbox               xbox 360 3 61 10 305 4 1 12 xbox one 6 162 6 278 1 0 11   a closer look at the data for circulating consoles, as illustrated in table 1, shows how durable some systems have been over this time period, with some consoles circulating hundreds of times outside the library. game console system durability exceeded early expectations. the median age of the circulating console collection is five years. this median age reflects the addition of newer generations of consoles, but also new-to-the-library older consoles. the median circulation total across the circulating console collection is 61.   console system cases have been sturdy with little actual physical breakage. most of the items discarded as damaged had software or system failures.   game controllers   game controllers are also more durable than first predicted, with a breakage rate of 14% and a loss rate of 5%, as illustrated in figure 2. these data are skewed because four off-brand gamecube controllers, which were purchased when budgets were tight, broke almost immediately. the library has learned that breakage is often very early in the controller’s life and paying the price up front for a name-brand controller is the best option for longevity.   figure 2 cumulative total for controllers from 2010–2022 (available, billed/lost/paid, damaged/discarded). there is a slight degree of uncertainty in data related to the difference in notes added to the bibliographic records related to loss and damage.   each console circulates with two game controllers in the console bag. additional controllers are available in their own bags. a closer look at circulating controllers and video game peripherals, as illustrated in table 2, shows how durable some of these items have been with several controllers circulating over 1,000 times. damage is higher for modern controllers, specifically the joy-con, ps4, and xbox one.   table 2 median values and statistics for controllers (2010–2022) game controllers   median lifespan/ years median circulation oldest highest circulation loss damage total controllers nintendo               gamecube 3 225 9 1,030 0 15 54 wii 4 129 12 1,148 3 2 62 wii nunchuk 6 70 11 267 3 4 33 wii u pro 4 140 4 205 0 0 10 wii u gamepad 5 394 7 963 1 1 8 joy-con 1 13 4 248 4 3 73 switch pro 1 7 3 211 1 0 17 playstation               playstation 3 6 77 11 1,130 2 5 27 playstation 4 4 177 6 580 2 11 55 xbox               xbox 360 6 183 11 1,232 4 7 28 xbox one 4 110 6 329 2 11 52   peripherals   the library also circulates peripherals. most of the peripherals listed in table 3 circulate inside and outside of the library. virtual reality headsets are in-house checkouts for use within the library. the guitar hero and rockband sets and their accessories were the one peripheral type pulled from circulation outside of the library because of damage. these sets are still used during special events but are no longer available for regular use.   table 3 median values and statistics for video game peripherals (2010–2022) video game peripherals   median lifespan/ years median circulation oldest highest circulation loss damage peripherals             wii wheel 11 65 11 77 0 0 wii perfect shot 5 1 6 2 0 0 wii balance board 6 32 9 59 0 0 wii dance pad 9 36 11 55 0 0 nintendo zapper 3 5 3 7 0 0 switch ring fit 1 4 1 4 0 0 ps3 move camera 10 29 10 57 0 0 ps4 vr headset 4 151 4 249 0 0 htc vive pro 4 290 4 290 0 0 ps3 rockband  7 59 7 65 0 1 ps3 guitar hero 7 10 7 32 0 0 xbox 360 guitar hero 8 13 8 45 0 1 xbox 360 rockband 8 13 8 13 0 1 wii guitar hero 1 14 1 14 0 1   legacy equipment   one way that the library has ensured equipment is available for older content is by developing a legacy collection. over the past 4 years this collection has been curated to support research and instruction as shown in table 4. in the future when production of a console system ceases, two copies will be pulled from circulation and added to the legacy collection, which is housed in media cabinets. consoles are also purchased to supplement this collection to support play of historical formats.   table 4 list of legacy consoles atari 2600 sega master system nintendo entertainment system sega genesis super nintendo entertainment system sega saturn sony playstation nintendo 64 sega dreamcast sony playstation 2 nintendo gamecube microsoft xbox sony playstation 3 microsoft xbox 360 nintendo wii nintendo wii u sony playstation 4 microsoft xbox one   consoles in the legacy collection do not circulate outside of the library. therefore, when circulating items enter this collection, the metrics for durability change. once an item makes it into this collection, circulation count is no longer a metric for determining usage. usage is determined by tracking class attendance for those using the collections to supplement coursework and individual research reservations.   durability is no longer a leading factor for items in the legacy collection. the goal shifts to maintenance to increase longevity. as this collection is only 4 years old, processes and procedures related to its management and maintenance are still evolving. at this time, one console, a sega master system, suffered an electronic failure, and another, a nintendo entertainment system, no longer produces sound. these failures are a bigger concern for this collection, but time will tell if it is sustainable as an educational resource.   management and maintenance   the statistics above show that a game collection can be circulated without unsustainable loss or breakage. one of the biggest hurdles related to beginning this type of collection is overcoming undue fears about equipment loss in relation to other library collections. developing a management and maintenance plan to mitigate loss and breakage can help positively persuade those making decisions related to these types of collection. over the past ten years the media library has modified its management, cleaning, and maintenance plans to increase the longevity of this collection. collection management   circulating video game equipment and its peripherals are housed in closed stacks or behind the circulation desk on reserve. all of the equipment in the media library is catalogued for an accurate inventory, but not all of it is visible to patrons. consoles are processed with an item record for each included item (e.g., hdmi cables, power cables, controllers). equipment is circulated in a barcoded bag. the bag item record is what displays to the patron in the library system and is used to place holds, bookings, and manage the equipment as a whole. each console circulates with two controllers and batteries. individual controllers and other game-related peripherals are also available for checkout in their own processed bags.   at checkout, all item barcodes are scanned to the patron’s account. at check-in, items are checked in and the service desk staff ensures that equipment is cleaned and organized neatly in the bag. batteries circulate in their own barcoded box and are removed from controllers and bags at check-in. the media library uses rechargeable batteries.   in-house non-circulating equipment and controllers are also catalogued. in-house materials include consoles, controllers, virtual reality devices, and gaming peripherals such as joysticks, steering wheels, and drum kits. in-house items are on reserve behind the front desk. they are checked in and out like outside circulating items but are not bagged.   these in-house items are used in our gaming space. this space includes 10 game stations and 22 gaming pcs. all of these stations are reservable using springshare’s libcal for up to 4 hours a day. entire spaces can be reserved by faculty, staff, and student groups for classes and engagement.   each reservable game station includes at least two cable-locked consoles with access to the internet and online game platforms such as playstation network, xbox game pass, and nintendo. internet access is locked to each console’s mac address. each station includes a high-definition television with a switch selector for input control. students can play their own digital content or library-owned physical content. play must be saved to the cloud both to mitigate loss, as anyone can delete content on these stations, and to ease console content space issues. patrons are allowed to bring in their own consoles, controllers, and peripherals but cannot access the internet through ethernet on all outside devices besides the switch console.   cleaning and maintenance   as the continuous use of gaming equipment grew, so did the need for cleaning and maintenance. consoles, controllers, peripherals, and headphones are examined at check-in and cleaned. items are wiped down and any microphone covers are changed out. a deeper clean is done each month. student staff retrieve each console, wipe it down and check it for damage, and check the circulation bag for damage or debris.   circulating remotes are deep cleaned. deep cleaning involves using a soft pick to clean out any dirt in the crevices and a cotton swab to clean around toggles and buttons with an alcohol solution. any peeling barcodes or labels are replaced at this time. cables are wiped down and managed with rubber bands. during the first months of the covid-19 pandemic, all equipment was quarantined for a 3-day period between checkouts until the library bought a sanitizing light station.   broken items or devices with software issues are checked out to a media library problem items patron record. student staff record the problem onto a problem item paper slip, attach the slip to the item, and place it into a tray. each item is double-checked by student staff for the noted issue and resolved as needed. problems might include consoles that need formatting or updates, broken cables, controllers that do not respond or have drift, a part that is missing, or an item record issue.   the addition of high-use gamecube controllers led to library repair procedures for controllers. repairs began simply with new joystick toggle caps but now include replacing joysticks in the joy-con to mitigate drift and replacing broken power jacks in handheld consoles. many of the components in modern consoles are easily replaceable. the library purchases kits and screwdrivers to clean and repair most of our controllers. select working parts from broken controllers are kept to make other repairs. these parts include the rubber button pads, triggers, toggles, and springs.   proper maintenance is important. not only does it help keep equipment clean and working, but it also allows staff to take a closer look at equipment to find problems before they become a bigger issue. cleaning equipment and performing simple repairs increases the longevity of circulating controllers.   legacy collection maintenance   new equipment in generation 7 and beyond, such as the wii, playstation 3, and xbox 360, are maintained like the non-legacy circulating collections. older equipment can require more maintenance. equipment should be cleaned, but consideration should be taken before applying any chemical processes to brighten or renew the look. for example, a console that has changed color because of smoke might look better if restored or whitened but this may make the plastic brittle. a careful review of renewal processes should be done to ensure that it will not decrease lifespan.   older consoles, and the cartridge games that are played in them, have pins that need care and cleaning. metal pins corrode over time but are easily cleaned with an alcohol solution and very fine grit sandpaper. the library uses the 1upcard cleaners on content cartridges and console pins before use. if a warm breath is used to get a cartridge to play, cleaning with a cotton swab and alcohol should be done before storing the item. although older consoles are very sturdy, maintenance should be done with care.   there are repair kits with more modern boards available for older consoles. care should be taken when repairing these older items. an old console case with all new parts is more like an emulator than the original device. emulators mimic a console and can play old content but are generally not using technology specific to the original, so play would differ. each library’s needs can dictate what types of repairs are acceptable for its equipment. if an older console is not something a library wishes to support, there are emulators for purchase that will play most older game cartridges as well as some preloaded with retro games.   conclusion   this study examined the sustainability of the consoles, controllers, and peripherals in a video game collection in an academic library. prior to incorporating video games and their associated equipment into the collection, there were concerns that such a collection could pose problems for collection management; these issues included increased staff procedures needed for circulation, the expense of replacing lost equipment, dealing with damage, and obsolescence. library staff continue to develop procedures to efficiently manage, clean, and maintain these collections to decrease wear and tear in the hopes that it increases longevity and reduces replacement costs. over the survey period of 12 years, there was indeed equipment loss and damage; however, the library was still able to meet the needs of students, faculty, and researchers and the cost of replacing or repairing the items was negligible. in conclusion, the researchers believe that including circulating consoles in a video game collection is a valued addition to a library that can supplement programming, boost innovation, and support burgeoning scholarship without being unsustainable.   author contributions   diane robson: conceptualization, data curation, formal analysis, visualization, writing – original draft (lead) sarah bryant: data curation, formal analysis, visualization, writing – review & editing catherine sassen: writing – original draft (supporting), writing – review & editing   references   buller, r. (2017). lending video game consoles in an academic library. portal: libraries and the academy, 17(2), 337–346. https://doi.org/10.1353/pla.2017.0020   byers, f. r. (2003). care and handling of cds and dvds: a guide for librarians and archivists. council on library and information resources. https://doi.org/10.6028/nist.sp.500-252   cross, e., mould, d., & smith, r. (2015). the protean challenge of game collections at academic libraries. new review of academic librarianship, 21(2), 129-145.  https://doi.org/10.1080/13614533.2015.1043467   entertainment software association. (2022, june). 2022 essential facts about the video game industry. https://www.theesa.com/wp-content/uploads/2022/06/2022-essential-facts-about-the-video-game-industry.pdf   gallagher, s., & park, s. h. (2002). innovation and competition in standard-based industries: a historical analysis of the us home video game market. ieee transactions on engineering management, 49(1), 67–82. https://doi.org/10.1109/17.985749   goodridge, m., & rohweder, m. j. (2021). librarian's guide to games and gamers: from collection development to advisory services. libraries unlimited.   greenwood, j. t. (2013). taking it to the stacks: an inventory project at the university of mississippi libraries. journal of access services, 10(2), 77–89. https://doi.org/10.1080/15367967.2013.762266   leblanc, k. (2021). the quagmire of video game preservation. information today, 38(5), 16–17.   loesch, m. f. (2011). inventory redux: a twenty-first century adaptation. technical services quarterly, 28(3), 301–311. https://doi.org/10.1080/07317131.2011.571636   mcdonough, j., olendorf, r., kirschenbaum, m., kraus, k., reside, d., donahue, r., phelps, a., egert, c., lowood, h., & rojo, s. (2010). preserving virtual worlds: final report. national digital information infrastructure and preservation program. http://hdl.handle.net/2142/17097   panuncial, d. (2019). librarians, start new game: how academic librarians support videogame scholars. american libraries, 50(11/12), 42–45. https://americanlibrariesmagazine.org/2019/11/01/librarians-start-new-game-videogame-collections/   robbins, m. b. (2016). invest in the classics. library journal, 141(13), 58. https://www.libraryjournal.com/story/invest-in-the-classics-games-gamers-gaming-august-2016   robson, d., parks, s., & miller, e. d. (2017). building game collections in academic libraries: a case study at the university of north texas. in m. robison & l. shedd (eds.), audio recorders to zucchini seeds: building a library of things (pp. 171–186). libraries unlimited.   robson, d., sassen, c., & rodriguez, a. (2020). advances in academic video game collections. the journal of academic librarianship, 46(6), 102233. https://doi.org/10.1016/j.acalib.2020.102233 williams, j. m., & chimato, m. c. (2008). gaming in d.h. hill library, nc state university. in a. harris & s. e. rice (eds.), gaming in academic libraries: collections, marketing, and information literacy (pp. 66–75). association of college and research libraries.     [1] pre-covid-19 data were used to provide a more accurate picture of library usage. article   educating assessors: preparing librarians with micro and macro skills   rachel applegate chair, department of library and information science school of informatics and computing indiana university purdue university indianapolis indianapolis, indiana, united states of america email: rapplega@iupui.edu   received: 3 feb. 2016     accepted: 24 mar. 2016       2016 applegate. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to examine the fit between libraries’ needs for evaluation skills, and library education and professional development opportunities. many library position descriptions and many areas of library science education focus on professional skills and activities, such as delivering information literacy, designing programs, and managing resources. only some positions, some parts of positions, and some areas of education specifically address assessment/evaluation skills. the growth of the library assessment conference, the establishment of the arl-assess listserv, and other evidence indicates that assessment skills are increasingly important.   method – four bodies of evidence were examined for the prevalence of assessment needs and assessment education: the american library association core competencies; job ads from large public and academic libraries; professional development courses and sessions offered by american library association (ala) divisions and state library associations; and course requirements contained in ala-accredited masters of library science (mls) programs.   results – while one-third of job postings made some mention of evaluation responsibilities, less than 10% of conference or continuing education offerings addressed assessment skills. in addition, management as a topic is a widespread requirement in mls programs (78%), while research (58%) and assessment (15%) far less common.   conclusions – overall, there seems to be more need for assessment/evaluation skills than there are structured offerings to educate people in developing those skills. in addition, roles are changing: some of the most professional-level activities of graduate-degreed librarians involve planning, education, and assessment. mls students need to understand that these macro skills are essential to leadership, and current librarians need opportunities to add to their skill sets.   introduction   over the last twenty years, libraries in general and academic libraries in particular have experienced a significant pro-assessment (evaluation) cultural wave. this is something that is becoming the norm in academic accreditation in general, and in the library field specifically. the question is whether current practitioners and current students have the opportunities to acquire the relevant assessment skills, which are different from what can be called the “practice” set (such as information assistance and instruction, information organization) and general professional values (such as knowledge of legal and ethical contexts and advocacy).   in this study, the word “evaluation” is used throughout. in higher education, the word “assessment” is generally reserved for a specific subset of evaluation: the assessment of student learning outcomes. when assessment of other areas (such as student affairs) occurs, it is generally termed “evaluation.” evaluation is also the more commonly used term in k-12 education and social services contexts. evaluation is distinct from research. according to the definitions for the use of human subjects in research, research aims to produce “generalized information.” in america, the code of federal regulations states that, “research means a systematic investigation, including research development, testing and evaluation, designed to develop or contribute to generalizable knowledge” (u.s. department of health and human services, 2009).   evaluation, on the other hand, is used for internal, organizational purposes, such as demonstration of value to stakeholders, improvement of existing functions, and design of new services, which have been collectively described as “the gathering of information for managerial decision-making” (applegate, 2013, p. 1). for instance, an analysis of whether mathematics resources can support a new doctoral program in mathematics at university a is evaluation. an exploration of how mathematics researchers access scholarly communication would be research. the distinction between evaluation and research lies primarily in the ends to which the data is put, rather than in the specific techniques used to conduct the evaluation or research.   higher education has placed increasing value on evaluation in accreditation, both institution-wide and for professional specializations. educational associations seek to demonstrate the value of their work. the council for higher education accreditation’s statement on the value of accreditation (2010) spells out the goal of “promoting accountability and identifying successful improvement efforts” (p.2). this followed changes in federal regulation based on the higher education opportunity act of 2008 and enacting regulations from 2010 and 2011 (higher learning commission, 2014). state and federal governments are keenly interested in accountability, given the significant funds given directly to institutions or indirectly through student aid and loans, as shown in the accrediting agency recognition criteria, u.s. department of education (2014).   the american library association’s (2009) core competences for librarianship speak to the responsibilities of graduate-level librarians and spell out the importance of both research for understanding of practice, and evaluation for effective management of libraries (ala, 2009). there are eight core areas, of which two (25%) mention evaluation; of 42 specific sub-points, four (10%) mention evaluation.   8c. the concepts behind, and methods for, assessment and evaluation of library services and their outcomes. 6a. the fundamentals of quantitative and qualitative research methods. 4c. the methods of assessing and evaluating the specifications, efficacy, and cost efficiency of technology-based products and services. 5d. information literacy/information competence techniques and methods, numerical literacy, and statistical literacy.   evaluation received explicit prominence in the 2008 standards for accreditation of mls programs, and even more emphasis in the 2015 standards (ala office for accreditation, 2008; 2015). from the preambles, both the 2008 and the 2015 documents state:   systematic planning is an ongoing, active, broad-based approach to… (b) assessment of attainment of goals, objectives, and learning outcomes; (c) realignment and redesign of core activities in response to the results of assessment…   the curriculum standard says:   (2008) ii.7 the curriculum is continually reviewed and receptive to innovation; its evaluation is used for ongoing appraisal, to make improvements, and to plan for the future. evaluation of the curriculum includes assessment of students' achievements and their subsequent accomplishments. evaluation involves those served by the program: students, faculty, employers, alumni, and other constituents.   (2015): ii.5 procedures for the continual evaluation of the curriculum are established with input not only from faculty but also representatives from those served. the curriculum is continually evaluated with input not only from faculty but also representatives from those served including students, employers, alumni, and other constituents. curricular evaluation is used for ongoing appraisal, to make improvements, and to plan for the future. evaluation of the curriculum includes assessment of students' achievements.   and the students standard (both 2008 and 2015) says:   iv.6 the school applies the results of evaluation of student achievement to program development. procedures are established for systematic evaluation of the degree to which a program's academic and administrative policies and activities regarding students are accomplishing its objectives. within applicable institutional policies, faculty, students, staff, and others are involved in the evaluation process.   the institute for museum and library services (2008) emphasizes outcomes-based evaluation for its grants directly, and through the library services and technology act funding to states. its webography on evaluation contains materials published in 1994 to 2004.   how do current and future librarians educate themselves to meet the need to evaluate (assess) library and information organizations? there is a micro-level of assessment that consists of understanding specific tools, such as survey design and data analysis, both generic (e.g., instructional testing) and library-specific (e.g., bibliometrics). there is also a macro-level that consists of understanding the role of assessment in managing libraries and in communicating with libraries’ users and parent institutions and communities.   in sum, assessment of libraries is something that a variety of stakeholders consider important. it is important internally for effective management, and externally, funders, donors, and governments expect it.   this descriptive study examined the prevalence of microand macro-evaluation skills on two sides: the job side, and the education side, for pre-service and in-service librarians. by combining data to provide an overall view of this landscape, this study lays the groundwork for further examination of the most effective and efficient venues for achieving this essential competency for libraries and information agencies.   methods   this study explores two descriptive, prevalence-related research questions.  rq-1: what is the prevalence of evaluation skills or responsibilities in library-based positions? rq-2: what is the prevalence of opportunities for education for librarians in evaluation skills?   for each research question, a population, a random sample, or a purposive sampling of items made up relevant data sets, and for each data set, qualitative coding was applied to arrive at a quantitative measurement of prevalence. a summary of these data sets can be found later in table 1.   rq-1 positions: operationalization   there are two data sets for this research question. one is idealized or prescriptive, while the other is descriptive or actual. the first data set (data set a) is the set of core competences and sub-points laid out in the ala core competences. the second data set (data set b) consists of a body of job position advertisements retrieved from a random sample (n = 20 each) of member libraries of the association of research libraries (arl) and the urban libraries council (ulc), as of spring 2014. this random selection of institutions, and using the institution’s own job posting sites, has been shown to provide the best representation of job ads, as opposed to using job-ad sites such as ala joblist or the chronicle of higher education (applegate, 2010). all full-time jobs were included, regardless of whether they were librarian-specific or required an mls.   there were 20 urban libraries council institutions selected by random number generation. of these, five had no current job openings. the new york public library listed 55 openings, while 15 other institutions listed 23 positions. twenty association of research libraries members were selected by random number generation. of these, five had no job openings listed while the remaining 15 libraries had 50 jobs listings among them.   it is worth noting that the boston public library (bpl) is a member of the urban libraries council and also the association of research libraries, and was selected in the arl random sample. new york public library (nypl) is also a member of the arl but was selected in the ulc sampling. the analysis examined the ads with boston public library positions in the arl group (as sampled) and another analysis divided the libraries into three groups: public, public-research (bpl and nypl), and research.   there were a total of 128 jobs identified. the researcher then coded each job at one of three levels of evaluation skills or responsibilities using coding level descriptions developed prior to coding. that is the coding represented an a priori categorization rather than a grounded content analysis.   0–no mention: position titles in this category include: library information assistant; major gifts officer; senior applications developer; public services librarian ii; librarian.   1–minimal mention: positions that were coded in this category included words such as cost tracking, generic "benchmarking," evaluating effectiveness (no more detail); evaluation tasks comprised less than half of the listed responsibilities. for example, an advertisement for library services manager (assistant director) indicated that “the successful candidate… cost effectiveness, monitor expenditures, continually benchmark approaches. another advertisement for project manager included in its qualifications “strong quantitative and qualitative data analysis skills, as well as experience conducting research”. an advertisement for staff secretary listed “compiling and reporting statistics” as one of the position’s responsibilities, and a librarian i position advertisement included “prepares statistical and/or narrative reports, memoranda and correspondence”.   2–significant mention: positions advertisements in this category includes the words "data" and either “analysis” or “gathering”, with related duties reflecting less than half of listed responsibilities. for example, an advertisement for a branch manager included responsibilities such as “tracks and analyze operational data and statistics; creates financial, statistical and narrative reports on branch library operations. makes presentations to library staff, the board of trustees and other groups.” an advertisement for librarian iii-children’s services: “collects, maintains and evaluates data relating to children's services, branch performance and program effectiveness.” a posting for library manager listed responsibilities like “develops and implements strategies to enhance the onsite user experience, including using statistics and metrics to tailor services to meet local community needs and drive circulation and attendance.” an advertisement for social media marketing associate included in its responsibilities “generate reports and translate data into actionable insights that will inform editorial decisions and content tactics…. run ab tests to optimize campaigns”.   3–primary role: assessment, evaluation, data-gathering or analysis mentioned as more than half of listed responsibilities. for example, an advertisement for business analyst states: “the it business analyst (ba) is a liaison between the information technology group and nypl business groups. the ba performs professional duties related to the review, assessment, and development of business processes. he/she will focus on the effective use of resources, both people and technology.” an advertisement for an assessment and statistics coordinator position included similarly pervasive use of words and phrases associated with assessment and analysis.   rq-2 education: operationalization   this part of the study draws on three data sets concerned with education for professionals.                 data set c: professional development courses or sessions offered by the american library association   this data set consists of professional development courses or sessions offered by divisions of the american library association as of spring 2014. this set included all online courses, all webinars, and listed ala annual meeting sessions. the “archives” were not accessed. these sessions were coded as either including or focusing on evaluation, or not.   examples of sessions coded as “evaluation-no” included:   personal digital archiving disaster response common core floating collection: how it can work   examples of sessions coded “evaluation-yes” included:   evaluating print book and e-book patron-driven acquisitions holdings comparisons: why are they so complicated? effective subscription management and alternatives a tale of two libraries: data evaluation through the eyes of an academic librarian and a public librarian                 data set d: state library association conference presentations   the data set consists of sessions presented at state library association conferences. these were taken from a purposive sampling of seven states for 2014 and one state for both 2013 and 2014, for a total of eight conferences.   alabama (april 2014) florida (may 2014) louisiana (march 2014) montana (april 2014) new hampshire (april 2014) new york (fall 2013) north carolina (10/2013 and 10/2014) washington (may 2014)   a total of 476 sessions were included. these sessions were coded as evaluation-no, or into one of two evaluation-yes groups, either results or techniques. the line between results and techniques was somewhat fuzzy and some analysis combines them.   evaluation-no: these were primarily how-to and update programs. they included professional techniques (“basics of preservation,”), content (“mysteries set in florida,”), management (“revolutionize your library with strong partnerships!”), and the community (“conversations with the montana state library commission.”)   evaluation-yes-results: for these programs, it appears that data was gathered, but the primary focus of the session was on what the data told the researchers and evaluators what to do next. example sessions:   ·         parents, alumni and libraries: what customers really believe about the library ·         turning the tables: assessing student worker satisfaction in peerstaffing models ·         rethinking reference: if it's broke, fix it! ·         patrons on performance: the library web as users see it ·         redefining outreach: creating a perception of person accessibility ·         outsourcing? an evaluation of vendor assistance in tech services ·         hispanic americans and public libraries: assessing health information needs and working together in an e-health environment   evaluation-yes-techniques: these sessions were specifically about how to conduct evaluation/research and data collection techniques, or, sometimes, explanations of why it should be done. in these cases the focus was on gathering data, not on why the data is gathered. example sessions include:   ·         how to listen to your patrons: maximizing value and outcomes through community insight ·         excel with excel ·         google analytic with how-to-directions ·         listening to your patrons: tools and approaches for gathering insight from your community ·         you've got data, now use it: innovative methods for better understanding public library use   data set e: courses that are or were required in ala-accredited masters of library science programs   this final data set consists of courses that are or were required in ala-accredited mls programs. these were examined at two time periods, 2005 and 2014, as reported to the association of library and information science education (alise). the first time period was selected as occurring before the spike in emphasis on evaluation in the late 2000s discussed in the literature review; the second was the most recent data available at the time of the study. three types of courses were captured, those about research, evaluation and management. management was included because of the tight integration of evaluation into the administration/ management section of the ala competencies. there were 48 degrees reported in 2004 and 50 in 2014. if a university offered multiple accredited degrees, the requirements for the degree that closest to a general “master of library science” were examined.   results   in 2014, both skills and needs represent about 10% of opportunities and requirements.   rq-1: what is the prevalence of evaluation skills or responsibilities in library-based positions?   this research found that approximately 10-30% of positions expect evaluation skills or include evaluation responsibilities, with no difference by type of library (public or academic/research). in data set a, the ala professional competencies mentioned some aspect of evaluation in 2 of 8 competencies (25%), and 4 of 42 sub-points (10%). in terms of job postings evidenced in data set b, out of 123 total jobs posted, 32% had at least some mention of an evaluation role. for 15% of postings, the mention was minor or in passing, 15% had a more explicit mention, but at less than half of listed responsibilities, and for 2% (2 positions) it was the major role (more than half of duties) for that position. conversely, the majority 68% of listed positions had no mention at all of evaluation or data responsibilities. this included professional librarian positions, such as “librarian” or “public services librarian ii.” other mentions were relatively meager.   there was a huge range of levels of responsibility in the descriptions, and they did not seem related to whether evaluation was present. two very different positions coded at the same “minimal” level for evaluation activity were “staff secretary—compiling and reporting statistics” and “library services manager…. cost effectiveness, monitor expenditures, continually benchmark approaches.” the two positions for which evaluation was the primary role included one primarily “librarian” (university of houston: assessment and statistics coordinator) and one of a professional support person (new york public library: business analyst).   when analyzed by type of institution, positions at public-research libraries (boston public library and new york public library, members of both the urban libraries council and the association of research libraries), and research libraries (arl libraries excluding ulc dual-members) were the only institutions to list primarily-evaluation positions. however, these institutions were also slightly more likely to have descriptions that had no mention of evaluation: research institutions listed 75% with no mention; public-research listed 68%, and public (ulc excluding arl dual-members) institutions listed only 63%.   table 1 data sources by research question   research question data set n (total), sample type, and date rq-1: what is the prevalence of evaluation skills or responsibilities in library-based positions?   a: ala core competences 8 core competency sets 42 specific sub-competencies population 2009   b: job postings at arl and ulc institutions 128 job postings random sample spring 2014 rq-2: what is the prevalence of opportunities for education for librarians in evaluation skills?   c: professional development courses offered by ala divisions 341 sessions population spring 2014   d: professional development sessions at state library association conferences 496 sessions purposive sample of 8 conferences 2013 and 2014   e: required courses at ala-accredited mls programs 48 programs, 67 courses; 50 programs, 74 courses; population 2005 and 2014     table 2 level of evaluation responsibility in job advertisements   type of library evaluation in described duties public research public research total none 50 34 64% 76% 68% minimal 11 7 14% 16% 15% less than half 16 3 21% 7% 15% more than half 1 1 1% 2% 2% total 78 45     rq-2: what is the prevalence of opportunities for education for librarians in evaluation skills?   for professional development, less than 10% of offerings involved evaluation skills. for pre-professional education, “research” and “management” are common requirements but evaluation is less present. data set c reveals that, as of spring 2014, there were 341 programs offered by 11 ala divisions: all online-recorded, live webinars, and conference sessions listed as “continuing education,” of which 24 (or 7%) were about evaluation generally or about a specific evaluation technique. out of 11 divisions, five had relatively few professional development courses/sessions listed (42 total sessions) of which none were about evaluation. yalsa had a large number of offerings at 38, of which only one was about evaluation. for the other divisions, the range of evaluation as a percentage of courses ranged from 6% to 18%. notably, the management-related division library leadership and management association (llama) had the highest percentage at 18%.   data set d includes seven states’ professional conference programs, found using a maximum variety purposive sampling varying by state size, region of the country, and presence or absence of graduate library programs. one state (north carolina) had two years examined (2013 and 2014). out of 496 total sessions discovered, only 29 (approximately 6%) had some relation to evaluation, either in terms of reporting results, or of teaching evaluation techniques.   graduate education for librarians typically consists of a wide variety of optional courses and some required courses. the balance between required and optional depends on the goals of individual programs, but the programs are unified here by the common factor of accreditation by the american library association. alise statistics cover most accredited libraries schools, though there are some gaps in the data for some programs in some years (association of library and information science educators, 2010, 2014). programs are asked to describe course requirements for their accredited degrees. both management and research course requirements remained stable when compared at two different points in a ten-year period, with 71 (72% of) programs requiring training in management and 58 (60%) requiring research methods. evaluation had a noticeable increase, with a low of 10% of programs in 2005 to 16% of programs in 2014.     table 3 continuing education offerings by ala division course/webinar involves evaluation no yes total percentage yes american association of school librarians (aasl) 6 6 0% association for library services to children (alsc) 8 8 0% association of specialized and cooperative library agencies (ascla) 7 7 0% library information technology association (lita) 13 13 0% united for libraries 8 8 0% young adult library services association (yalsa) 37 1 38 3% association for library collections and technical services (alcts) 114 7 121 6% association of college and research libraries (acrl) 30 2 32 6% reference and user services association (rusa) 16 2 18 11% public library association (pla) 55 7 62 11% library leadership and management association (llama) 23 5 28 18% total 317 24 341  7%   table 4 state library association conference sessions session involves evaluation no yes-results yes-technique total percentage yes new york 69 1 70 1% louisiana 62 2 64 3% new hampshire 31 1 32 3% alabama 49 2 51 4% montana 48 2 50 4% north carolina 133 4 7 144 8% washington 28 1 2 31 10% florida 47 5 2 54 13% total 467 10 19 496 6%   table 5 required courses for master of library science degrees courses 2005 number 2005 percentage 2014 number 2014 percentage management 34 71% 36 72% research 28 58% 30 60% evaluation 5 10% 8 16% programs 48   50       across programs a management course was the most prevalent course requirement. management courses had titles such as “library/management/administration of/in libraries/information organizations,” and frequently were by-type (academic, school, etc.). three others in 2014 were “achieving organizational excellence,” “management and systems analysis,” and “organizational management & strategy / management without borders.”   almost all research courses had simple titles of “introduction to research/methods” or “research methods.” three others were “contextual inquiry and project management,” “designing principled inquiry,” and “educational research & measurement.”   courses that were counted as focusing on evaluation were included “assessing information needs,” “evaluation of resources and services,” “evaluation of information systems,” “evaluation methods,” and “library planning, marketing and assessment.”   there was some overlap between categories. the course “management and systems analysis,” was counted as a management course and as an evaluation course. “research & evaluation for lis” and “research & evaluation methods” were counted in both the research and evaluation categories. also, in some programs, students could take either research or evaluation courses.   given that many, and probably most, program requirements involve options and substitutions, with differences by specializations, and also some variation in reporting, this is a very fuzzy data set. nevertheless, evaluation itself appears in required coursework for at least some programs, and has had some slight gains over the past 10 years.   table 6 overall results by research question research question results rq-1: need: what is the prevalence of evaluation skills or responsibilities in library-based positions? a-ala core competences 10-25% b-open jobs at arl and ulc institutions 32% rq-2: opportunity: what is the prevalence of opportunities for education for librarians in evaluation skills? c-professional development courses offered by ala divisions 7% d-professional development sessions at state library association conferences 6% e-required courses at ala-accredited mls programs 15% (evaluation) 58% (research) 71% (management)   discussion   within these data sets, and accounting for their limitations, there appears to be a mismatch between the need for evaluation (assessment) skills and the formal opportunities for librarians (library staff) to obtain those skills. while few library positions, even at very large systems and institutions, are solely dedicated to evaluation activities, data collection and analysis is part of about one-third of positions advertised at these libraries. however, less than 10% of continuing education opportunities, whether by state associations or by american library association divisions, focus on evaluation skills (or results).   association events, conferences, and courses are an important way for current information professionals to keep up to date, especially when life-long learning is not just a motto but an essential part of an information professional’s life (long & applegate, 2008). there appears to be an opening for increased attention to this area of education. this is also an area for a cumulative virtuous circle. experts in evaluation can present results and instruction in techniques to a widening pool of practitioners who in turn spread a culture, capability, and commitment to the use of data in decision-making. over the years the arl library assessment conference has grown in prominence and size, supplemented by the launch of the arl-assess listserv in 2014, and the development of a public library assessment workshop.   besides professional continuing education, there is pre-professional preparation. that is, programs of library and information science have the responsibility to prepare graduates to perform, understand, and develop further in the principles and practices of their profession. library education at the graduate level has had a high level of interest in or requirements for research-specific skills, undoubtedly influenced by the place of the mls degree as a graduate or professional degree at universities. there is a perennial discussion about the relevance of the mls to professional practice, and this paper avoids entering that broad debate here.   there is, however, a specific issue that is relevant to understanding the place of evaluation education in professional preparation: the distinction between research and evaluation. conceptually, are these the same, and pragmatically, does coursework in research methods prepare a student to conduct managerially-oriented assessment?   on the conceptual question, the assessment in higher education listserv (assess@lsv.uky.edu) has a user population made up primarily of people working at colleges and universities, in academic programs and also in centralized assessment offices. one perennial question and debate in this forum is whether evaluation or assessment is “research” as defined by the federal government or the institution’s institutional research board (irb) or other office for the protection of human subjects in research. federal definitions define “research” as generalized knowledge, and on campuses that in turn can be operationalized as something to be published, presented, or disseminated to an external audience. in contrast, non-research evaluation is often treated as internally oriented: “if the investigator does not intend to use the information for publication or presentation outside of the investigator’s department or organization, the research will not contribute to generalizable knowledge and irb review is not required” (indiana university, 2014).   this leaves a gap in understanding the dissemination of methodology and of case-instances that may contribute to a generalizable understanding. for example, suppose you conduct a study with your math majors of their use of your e-book collection on mathematics. this is for one’s own use in collection management. yet, an audience may want to know how to conduct such studies. or another scholar may want to know the status of e-books about mathematics and other science areas: using the specific to illuminate the general. methodologically, there can be important and useful overlaps in research or evaluation data techniques and data collection designs. faculty in library programs that require or offer research methods courses can use the practical importance of evaluation to educate their students about the overall value of such courses: many library students believe they will not conduct formal “research” so tend to think of this as entirely theoretical.   this prevalence study describes in part the role and place of evaluation in library practice, showing the degree of importance accorded to assessments skill in institutions and in library professional development. it forms part of a larger, ongoing conversation about the preparation and function of mls-educated librarians in information organizations. the extent to which the mls is managerial, evolving in addition to, and perhaps away from purely technical professional skills, is reflected in the description of evaluation as an essential component of leadership (component 8c).   conclusion   it is hard to design a quantitative equation encompassing offerings and needs, where a equals b, or even where a results in b, for the concerns under consideration in the study. even the percent or prevalence of evaluation mentions in courses or in job ads are far from exact. the trend is clear, though, that there seems to be more extensive need for evaluation skills than there are structured offerings educating people in those skills. when lis educators organize their programs of study to prepare graduates to meet the needs of practice, they need to thoughtfully consider what the core requirements are. evaluation is specifically mentioned and indeed emphasized in the ala competences document, and is reflected in new job position descriptions.   for existing librarians, roles will change. just as a wave of rda and frbr workshops, webinars, and books were published to assist technical services librarians in making the transition to newer forms of organizing information, opportunities are needed to continually enhance the ability of library leaders to manage and to meet external demands for accountability and improvement.   references   american library association. (2009). ala’s core competences of librarianship. retrieved from http://www.ala.org/educationcareers/careers/corecomp/corecompetences   american library association, office for accreditation. (2008). standards for accreditation of master's programs in library and information science. chicago: american library association, office for accreditation. retrieved from http://www.ala.org/accreditedprograms/standards   american library association, committee on accreditation. 2015. standards for accreditation of master’s programs in library and information studies. chicago: american library association, office for accreditation. retrieved from http://www.ala.org/accreditedprograms/standards   association of library and information science educators. (2010). library and information science education statistical report 2009. chicago: association of library and information science educators.   association of library and information science educators. (2014). alise statistical report, 2014 [data set]. retrieved from http://www.alise.org/statistical-reports-2   applegate, r. (2010). job ads, jobs, and researchers: searching for valid sources. library & information science research, 32(2),163-170. http://dx.doi.org/10.1016/j.lisr.2009.12.005   applegate, r. (2013). practical evaluation techniques for librarians. santa clara, ca: libraries unlimited.   council for higher education accreditation. (2010). the value of accreditation. retrieved from http://www.chea.org/pdf/value%20of%20us%20accreditation%2006.29.2010_buttons.pdf   higher learning commission, (2014). federal compliance program packet. retrieved from https://www.hlcommission.org/policies/federal-compliance-program.html   indiana university, office of research compliance. (2014). human subjects: levels of review. retrieved from http://researchcompliance.iu.edu/hso/hs_level_review.html   institute for museum and library services. (2008). webography [outcomes based evaluations]. retrieved from https://www.imls.gov/grants/outcome-based-evaluation/webography   long, c. e., & applegate, r. (2008). bridging the gap in digital library continuing education: how librarians who were not “born digital” are keeping up. library administration and management, 22(4), 172-18. https://journals.tdl.org/llm/index.php/llm/index   u.s. department of education. (2014). accreditation in the united states. subpart b: the criteria for recognition. basic eligibility requirements. retrieved from http://www2.ed.gov/admins/finaid/accred/accreditation_pg13.html   u.s. department of health and human services. (2009). code of federal regulations, title 45 public welfare, part 46: protection of human subjects. retrieved from http://www.hhs.gov/ohrp/humansubjects/guidance/45cfr46.html editorial   eblip receives the seal of approval   lorie kloda editor-in-chief associate university librarian, planning and community relations concordia university montreal, quebec, canada email: lorie.kloda@concordia.ca      2017 kloda. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   i have some very good news to report. after a long delay due to overwhelming volume, evidence based library and information practice (eblip) has been successful in its reapplication to the directory of open access journals (doaj). since its inception, all of the journal’s article-level metadata has been indexed in the doaj, and will continue to be. in addition to a successful reapplication, eblip was also qualified for the doaj seal. the seal is awarded to those open access journals which adhere to an additional set of seven criteria that demonstrate best practices. eblip meets all of these criteria by having an archiving practice in place, providing digital object identifiers for all its articles, and allowing authors to retain copyright over their work, among other things. i am very pleased to be able to make this announcement, as the editorial board of the journal is committed to providing quality content in an open access format, and to doing so in the most evidence-based manner possible. we are continuously striving to maintain the highest standards of publishing to provide access to journal content.   there are several more announcements to accompany the above. eblip recently appointed a new associate editor (classics and reviews), dr. christina l. wissinger, who is a librarian at pennsylvania state university in the u.s. we have also appointed a new editorial intern, rachel hinrichs, a recent graduate and librarian at indiana university purdue university in indiana. also joining the team is kate shore, a librarian at memorial university of newfoundland, canada as the indexing coordinator. kate’s role will be to ensure the journal’s contents are indexed in the doaj and to explore other indexing opportunities for the journal, as well as best practices.   this issue of eblip includes features from the recent symposium of the centre for evidence based library and information practice (c-eblip) that took place in late 2016. the organizer of the symposium, virginia wilson, is also a long-time contributor to the journal. virginia wrote many of the articles in the very popular ebl 101 column and more recently was the author of the research in practice column. this issue of the journal includes virginia’s final submission to the column. i am sure that though she will no longer be formally linked to eblip through the column that we can expect virginia to continue to make contributions to the journal and to the practice of evidence based librarianship. the editorial board and i are sad to see her step down from this role, and wish virginia the best of luck in her continuing role as director of the c-eblip.   and finally, it is that time again when the journal renews its list of peer reviewers. i have begun contacting peer reviewers to determine if they are willing to renew for another term. eblip is always looking for potential reviewers with experience in various sectors and with knowledge of different research methodologies. if you are interested in peer reviewing for the journal please let me know.   news   call for applications:  evidence based library and information practice evidence summaries writers    2017. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     eblip seeks to add five writers to the evidence summaries team. evidence summaries provide critical appraisal syntheses for specific research articles. these research synopses provide readers with information regarding the original research article's validity and reliability, thus providing information on the presence or absence of evidence with which to make informed decisions. evidence summaries team members are required to write two evidence summaries per year, with a two year commitment to the journal. evidence summaries cover all areas of library and information studies and we encourage applications from information professionals in areas such as school, public, and special libraries, as well as academic settings. interested persons should send a statement of interest, indicating areas of strength they would bring to the role, as well as a brief résumé to heather pretty (associate editor, evidence summaries) hjpretty@mun.ca by october 15, 2017. applicants who are shortlisted will be asked to submit a sample evidence summary. **please note that evidence based library and information practice is a non-profit, open access journal and all positions are voluntary and unpaid. the positions are an excellent opportunity for continuing professional development and gaining experience in reviewing or critically appraising library-related research. **only those applicants who are selected or shortlisted will be contacted by the editors. about the journal: published quarterly by the university of alberta, this peer reviewed, open access journal is targeted at all library and information professionals interested in an evidence based model of practice. by facilitating access to library and information studies research via original research articles and evidence summaries of relevant research from the library literature, evidence based library and information practice enables information professionals to practice their profession in an evidence-based manner. please visit the evidence based library and information practice website (http://ejournals.library.ualberta.ca/index.php/eblip) for further information about the journal. ebl 101   research methods: mixed methods research   virginia wilson librarian university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   originally published in: evidence based library and information practice, 8(2), 275–277. https://ejournals.library.ualberta.ca/index.php/eblip/article/view/19571/15257     received: 13 may 2013  accepted: 06 june 2013      2016 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   research paradigms. qualitative research. quantitative research. the firm adherence to one or the other can cause a lot of upheaval, as can be seen in the long-standing paradigm wars between the two. researchers from both sides of this ontological divide have been convinced of the “rightness” of their methodologies and beliefs around their specific approach to research. the side taken often depends upon from which disciplines the researchers are coming, as some disciplines work strictly quantitatively while some work from the qualitative perspective. increasingly, however, we are seeing mixed methods used in research studies. in a 2004 article, johnson and onwuegbuzi argued from the position that “there is now a trilogy of major research paradigms: qualitative research, quantitative research, and mixed methods research” (p. 24).   as always, i can only scratch the surface of any topic in this column, and mixed methods research (also referred to as mixed research) is no exception, especially as its history is steeped in philosophical understandings of research and knowledge acquisition in general. the mixed methods paradigm attempts to get in the middle of the two other approaches (quantitative and qualitative), seeking to respect both by using both in a research study. according to johnson, onwuegbuzie, and turner (2007), “mixed methods research is, generally speaking, an approach to knowledge (theory and practice) that attempts to consider multiple viewpoints, perspectives, positions, and standpoints (always including the standpoints of qualitative and quantitative research)” (p. 113). these researchers encourage a broad interpretation of “methods” in mixed methods research to allow for “inclusion of issues and strategies surrounding methods of data collection (e.g., questionnaires, interviews, observations), methods of research (e.g., experiments, ethnography), and related philosophical issues (e.g., ontology, epistemology, axiology)” (johnson, onwuegbuzie, and turner, 2007, p. 118).   looking at literature dealing with mixed methods research is a bit like navigating a maze: there’s a lot of it and it all looks much the same at the start. the johnson, onwuegbuzie, and turner (2007) article is particularly helpful, as it not only gives a history of this methodology, but also lists key players in the mixed methods mix, and looks specifically at the definitions that have emerged in the past 2 decades. the definitions are especially useful for a column such as this. the following are from the johnson, onwuegbuzie, and turner (2007) article:   mixed methods research is a research design (or methodology) in which the researcher collects, analyzes, and mixes (integrates or connects) both quantitative and qualitative data in a single study or a multiphase program of inquiry. (creswell, p. 119)   mixed methods research is a systematic integration of quantitative and qualitative methods in a single study for purposes of obtaining a fuller picture and deeper understanding of a phenomenon. mixed methods can be integrated in such a way that qualitative and quantitative methods retain their original structures and procedures (pure form mixed methods). alternatively, these two methods can be adapted, altered, or synthesized to fit the research and cost situations of the study (modified form mixed methods). (chen, p. 119)   mixed methods means the combination of different qualitative and quantitative methods of data collection and data analysis in one empirical research project. this combination can serve for two different purposes: it can help to discover and to handle threats for validity arising from the use of qualitative or quantitative research by applying methods from the alternative methodological tradition and can thus ensure good scientific practice by enhancing the validity of methods and research findings. or it can be used to gain a fuller picture and deeper understanding of the investigated phenomenon by relating complementary findings to each other which result from the use of methods from the different methodological traditions of qualitative and quantitative research. (kelle, p. 120)   mixed methods research refers to the use of data collection methods that collect both quantitative and qualitative data. mixed methods research acknowledges that all methods have inherent biases and weaknesses; that using a mixed method approach increases the likelihood that the sum of the data collected will be richer, more meaningful, and ultimately more useful in answering the research questions. (preskill, p. 121).   there are many more definitions of mixed methods research out there. while there are some variations in the content, most contain similar core ideas of using both qualitative and quantitative methods in order to more fully explore a research question. mixed methods research requires balance and an open mind to the benefits of various approaches to research.   in terms of mixed methods in library and information science (lis) research, raya fidel asks, “are we there yet?” fidel’s article is a must read, as she explores mixed methods research from an lis perspective, coming to the conclusion after an analysis of 465 articles published in four major lis research journals that only 5% of these articles used mixed methods, and that the methodology was not mentioned by name in the articles (p. 269). for her study, fidel uses the definition of mixed methods research written by tashakkori and creswell (2007) because of its inclusive nature: mmr is ‘research in which the investigator collects and analyzes data, integrates the findings, and draws inferences using both qualitative and quantitative approaches or methods in a single study or program of inquiry’ (p. 4)” (fidel, 2008, p. 269). however, given that the definitions of mixed methods research vary, that fidel found mixed methods research even though the terminology was not explicitly used in the research article, and that her research project examined only 4 lis research journals, it is not surprising to find that mixed methods research is being undertaken in lis. for example, see the following:   brettle, a., hulme, c., & ormandy, p. (2006) the costs and effectiveness of information-skills training and mediated searching: quantitative results from the empiric project. health information & libraries journal, 23(4), 239–247. doi: 10.1111/j.14711842.2006.00670.x   kwon, n. (2008). a mixed-methods investigation of the relationship between critical thinking and library anxiety among undergraduate students in their information search process. college & research libraries, 69(2), 117-131.   digging into mixed methods research would be beneficial to lis researchers. coming at a research question from multiple methodological perspectives in the same study will add a depth and breadth to the findings, and open up options for data collection and analysis.   resources:   on the web   the twitter hashtag #mixedmethods will yield tweets having to do with mixed methods research. developing mixed methods research with dr. john w. creswell, video, http://www.youtube.com/watch?v=psvsd9fax38 (an excellent encapsulation in about 6 minutes) what is mixed methods research? john creswell, video. http://www.youtube.com/watch?v=1oanitlpyx8 (15 min) creswell, j. (2012) introduction to mixed methods research, keynote address for the caqd conference, university of marburg, germany, march 8, 2012. http://prezi.com/qsksml6l-_vi/introduction-to-mixed-methods-research best practices for mixed methods research in the health sciences, national institutes of health, http://obssr.od.nih.gov/mixed_methods_research   journal of mixed methods research, sage publications, http://mmr.sagepub.com   books   creswell, j. w. & plano clark, v.l. (2011). designing and conducting mixed methods research. los angeles: sage.   hesse-biber. s.n. (2010). mixed methods research: merging theory with practice. new york: guilford press.   tashakkori, a., & teddlie, c. (eds.). (2003). handbook of mixed methods in social and behavioral research. thousand oaks, ca: sage.   teddlie, c. & tashakkori, a. (2009) foundations of mixed methods research: integrating quantitative and qualitative approaches in the social and behavioral sciences. los angeles: sage.   references   fidel, r. (2008). are we there yet?: mixed methods research in library and information science.  library and information science research, 30, 265-272.   johnson, r.b. & onwuegbuzie, a.j. (2004). mixed methods research: a research paradigm whose time has come. educational researcher, 33(7), 14-26. stable url: http://www.jstor.org/stable/3700093   johnson, r.b., onwuegbuzie, a.j. & turner, l.a. toward a definition of mixed methods research. journal of mixed methods research 1, 112-133.  doi:10.1177/1558689806298224   microsoft word news1_10pt evidence based library and information practice 2008, 3:3 98 evidence based library and information practice news eblip original research award © 2008. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. the editorial board of evidence based library and information practice is pleased to announce the development of an annual award for the best original research article published in its journal. the editorial board will select the winning article from original manuscripts published in each volume. a cash award will be presented to the winning author(s) following the publication of the final issue per volume and will be announced in the first issue of the following year. more information will be forthcoming on the journal’s website <http://ejournals.library.ualberta.ca/index.p hp/eblip>. evidence summary   demographic variables are associated with differing perceptions of a broad range of public library benefits   a review of: sin, s.-c. j., & vakkari, p. (2015). perceived outcomes of public libraries in the u.s. library & information science research, 37(3), 209-219. http://dx.doi.org/10.1016/j.lisr.2015.04.009   reviewed by: sara sharun campus librarian okanagan college library penticton, british columbia, canada email: ssharun@okanagan.bc.ca   received: 1 mar. 2016    accepted: 8 apr. 2016      2016 sharun. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine the frequency and nature of perceived beneficial outcomes of public libraries on individuals, and to identify demographic differences in these perceived outcomes.   design – self-administered, online questionnaire asking respondents to rate the frequency of benefits they received from public libraries in 22 areas of life including education, work, and business; everyday activities; and leisure activities.   setting – united states of america.   subjects – 1010 respondents from 49 states: 50% female, 76% white, 55% urban or suburban.   methods – correspondence analysis was used to visualize relationships between demographic variables and perceived outcomes. exploratory factor analysis was used to identify structures among the outcomes and summarize data into three core dimensions: everyday activities and interests; reading and self-education; and work and formal education. multiway anovas were used to test the significance of demographic differences on perceived outcomes.   main results – the most highly ranked areas of perceived benefits were reading fiction and non-fiction, self-education during leisure time, interest in history or society, and health. outdoor activities, exercise, and sport ranked the lowest. respondents in younger age groups reported benefits in “education and work,” as did ethnic minorities and people with lower household incomes. “everyday life” benefits were reported by male, suburban, white, middle-income respondents. “reading and self-education” benefits were reported by high-income, older age groups, white, and female respondents. two demographic groups did not correspond to any benefit categories: those who did not graduate high school and those over age 65.   conclusion – there are significant differences among demographic groups in how the benefits of public libraries are perceived, and these demographic differences have implications for program planning, marketing, and outreach in public libraries. specifically, libraries should work to increase and improve service to less-advantaged groups, including low-income earners and ethnic minorities, and make available more services and resources relevant to older people.   commentary   this study attempts to fill a gap in the lis assessment literature by defining and examining outcomes, rather than outputs, of public libraries. the authors make an interesting case for the need to measure broader, more inclusive outcomes and their impact on individuals and communities, instead of the more discrete and quantifiable outputs of specific library programs or services. the study, which was based on methods used in a previous study conducted in finland (vakkari & serola, 2012), is well designed. the results are clearly stated (although the terminology used may pose a challenge for readers without a statistical background) and conclusions are accurately reflected in the data.                 while the study design and reporting of results are strong, this study’s validity is questionable when the study population and data collection method are considered under critical appraisal criteria (glynn, 2006). the sample does not seem to be representative of the larger population (i.e., respondents were 76% white). an internet survey company was used to gather data, and it is not clear how the sample was recruited or selected. these issues may call into question the significance of the sample and the validity of claims made regarding demographic differences. the demographic differences in perceptions may be true for the sample, but the authors do not effectively argue that the sample is representative of the population. the strength of the evidence presented would have been stronger had the authors provided more rationale for their data collection methods. this would not only be helpful for other researchers looking to conduct similar research, but would also have bearing on the comparisons made to the finnish data, which was collected with different methods from a different population sample.   the authors give due attention to the limitations of surveys and self-reported data, but a stronger case for the reliability of their data could be demonstrated by presenting full details of the questionnaire, which would allow for insight into how respondents might have been influenced by the language of the survey, and how the manner in which categories and examples were presented to respondents might account for differences in responses from people with different demographic backgrounds. more information on category descriptions would also help validate the factor analysis and the establishment of three factors underlying the discussion of demographic differences in perceived outcomes.   so much of lis research is done at the case-study level, producing results that are usually not generalizable. this study presents an opportunity, in the form of a potentially robust measurement tool, to move beyond case studies and gather comparative data. information on the reliability and validity of the survey tool, as well as access to the questionnaire, would be welcome by lis researchers as a tool for benchmarking and comparing data from diverse populations, and would help facilitate the type of research the authors recommend. this way, comparisons made between nations or other large populations might be more reliable, and librarians would have data that they could then triangulate with local data, thereby improving the quality of evidence on the benefits of public libraries to communities.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154   vakkari, p., & serola, s. (2012). perceived outcomes of public libraries. library & information science research 34(1), 37–44. http://dx.doi.org/10.1016/j.lisr.2011.07.005   evidence summary   journal articles are the most widely used information resource for research and teaching in all academic disciplines   a review of: borrego, á., & anglada, l. (2016). faculty information behaviour in the electronic environment: attitudes towards searching, publishing and libraries. new library world, 117(3/4): 173-185. doi:10.1108/nlw-11-2015-0089   reviewed by: dominique daniel humanities librarian for history and modern languages oakland university rochester, michigan, united states of america email: daniel@oakland.edu   received: 25 may 2016  accepted: 15 july 2016      2016 daniel. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine faculty’s information behaviour and their perception of academic libraries in the current transition between print and electronic scholarly communication.   design – online survey.   setting – a consortium of 12 large universities in spain.   subjects – more than 17,380 faculty members.   methods – the researchers used a questionnaire based on a subset of the questionnaire used for the ithaka s+r faculty survey, with 20 closed and 2 open-ended questions. the survey was implemented via google forms and sent through mailing lists. the number of recipients was not known, but university statistics for 11 of the 12 universities list 17,380 faculty (statistics were not available for one university, located in a different administrative area). the questions aimed to identify the types of documents used by scholars for teaching and research, the search tools used, the strategies used to keep up-to-date in their disciplines, preferences for print or electronic books, the sources used to access documents, their preferred channels to disseminate their own research, and their views regarding library services.   main results – the response rate was 12.7%. based on the results, scholarly journals were the most used information resource for research across all academic disciplines, with 94% of respondents rating them as important. for teaching, faculty preferred to use textbooks for undergraduates, and journal articles for master’s students. to search the literature, faculty chose bibliographic databases and internet search engines over the library catalog and physical collections, although the catalog was the first choice for known-item searches. respondents favored print to read entire books or chapters but preferred the electronic format for skimming. of the respondents, 78% rated the library as an important channel to access resources, while 61% also considered free online materials important. if the material was not available at their library, 71% frequently chose to search for a free online version and 42% used the inter-library loan service. for their own research, faculty have published in scholarly journals more often than other channels and have selected the journal based on its impact factor (77.5% ranked it as important) and on its area of coverage (73.4%). when asked to rank library services, faculty placed paying for resources highest, with 86.2% identifying it as important. next were facilitating teaching and helping students develop information literacy skills. finally, a majority of faculty considered themselves highly dependent on the library.   conclusion – journal articles are the most widely used information resource for research and teaching purposes, regardless of discipline. this includes arts and humanities, which are known for heavy monograph usage. articles are also scholars’ preferred channel for publishing. with regards to books, faculty have mixed feelings about print and electronic formats. spanish faculty display information behaviours similar to their british and american counterparts, as documented in the ithaka s+r 2012 surveys. blogs and social networks are not widely used in spite of growing attention to such channels for research output and altmetrics. open access is also relatively unimportant for faculty when they choose where to publish. a majority of respondents still consider library services as important, for collections as well as teaching and learning support, which may present opportunities for librarians.   commentary   librarians know how important good relationships with faculty are, not only as patrons but also as teachers of the students the library supports. librarians therefore welcome studies such as this one that provide overviews of faculty information behaviour. while the surveys conducted by ithaka s+r since 2000 are limited to the united states and united kingdom, researchers in this study applied ithaka s+r questions to spanish professors. this makes it possible to compare information behaviour in a different academic culture. the authors found no major differences between the two groups. they felt that this may reflect the “globalized scientific endeavor,” as academics worldwide rely on the same documents and tools, and use the same strategies to keep up to date and access information.   the study relied on the 2012 ithaka s+r survey, the most recent one available at the time. since then the 2015 survey results have been published (housewright, schonfeld, and wulfson, 2012; wolff, rod, and schonfeld, 2016). the researchers pre-tested their questionnaire with research support librarians, who are presumably knowledgeable about academics. the usefulness of pre-tests is shown by the ithaka s+r survey, which revealed that faculty had difficulty with definitions of terms like institutional repository. (wolff et al., 2016) the study mentions the disciplinary makeup of the respondent population, which included health sciences faculty who are omitted from the ithaka s+r surveys. the study does not break down results by discipline, however, except for one question – the types of documents used for research. it would be interesting to see if this study’s results match those of the ithaka s+r surveys that show marked differences by discipline. for example, humanists are less interested in e-books than faculty in other disciplines; and the use of blogs and social media to disseminate research is somewhat more common for humanists and social scientists (wolff et al., 2016). age matters as well (wolff, et al., 2016).   a stimulating part of the survey dealt with the kind of library support services the faculty would like. the most valued service was acquisition of resources, but close behind was teaching support. this seems to indicate that faculty are increasingly aware of their students’ weak research skills, an interesting change from the 2012 ithaka s+r survey when faculty valued the purchasing and archival role of their library significantly more (housewright et al., 2012). this trend was later confirmed in the 2015 ithaka s+r survey, which shows a large increase in the perceived role of the library in helping undergraduates since 2012 (wolff et al., 2016). this, along with the fact that faculty still see themselves as highly dependent on the library for research, provides hope for librarians who wish to ward off a decline of users’ support.   references   housewright, r., schonfeld, r. c., & wulfson, k. (2012). ithaka s+r us faculty survey 2012. new york, ny: ithaka s+r. retrieved may 16, 2016 from http://www.sr.ithaka.org/publications/us-faculty-survey-2012/   wolff, c., rod, a. b., & schonfeld, r. c. (2016). ithaka s+r us faculty survey 2015. new york, ny: ithaka s+r. retrieved may 16, 2016 from http://sr.ithaka.org/?p=277685   microsoft word es_hall.doc evidence based library and information practice 2006, 1:2      36 evidence based library and information practice       evidence summary    ‘quick reads’ may promote literacy without stigma: findings from eight uk public  libraries   a review of:  mcloughlin, carla, and anne morris. ʺuk public libraries: roles in adult literacy provision.ʺ  journal of librarianship and information science 36.1 (march 2004): 37‐46.    reviewed by:  stephanie hall  chief librarian, powell river public library  powell river, british columbia, canada  e‐mail: shall@powellriverlibrary.ca    received: 2 march 2006           accepted: 15 april 2006      © 2006 hall. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to examine the role of public  libraries in the provision of adult literacy  services, with a detailed look at both the  successes and concerns of the libraries under  study; to provide recommendations for best  practice in establishing or reviewing adult  literacy services.    design – a series of case studies using  written reports and semi‐structured  interviews.    setting – eight public libraries in the uk  involved in literacy service provision or  reader development services.    subjects – eight senior staff members in  charge of library literacy programming.    method – a written report of literacy  service initiatives was solicited from each  participating library. a single interview was  conducted with a staff member in charge of  literacy service at each of the eight  participating libraries. fact‐checking  telephone interviews were conducted at  three locations where adult literacy  programs were in early stages. more in‐ depth, face‐to‐face interviews were  conducted at the five libraries with better  established programs. each type of  interview consisted of a set of scripted  questions supplemented by individualized  questions based on the written reports.    main results – there are four key areas of  results to be summarized from this study:    http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:2      37 • adult literacy collections – the authors  observed three main approaches to  branding literacy collections:     − emphasis on reading for  pleasure (with collections  entitled ‘quick reads’ or ‘first  choice’);     − emphasis on reading for skills  development;       − discreet labelling enabling stock  recognition without advertising  that the reader is borrowing  literacy materials.     the authors conclude that the ‘quick  reads’ approach was the most  successful in highlighting the collection  without stigmatizing it and in  promoting the pleasure of reading. the  importance of maintaining relevant,  attractive books was highlighted, with  collections targeting both entry level  readers and emergent readers.     • approaches for supporting adult literacy –  the libraries used reader development  extensively as a strategy to support  adult literacy efforts.  staff tied literacy  offerings to other programs or services  of interest (for example, promoting  adult literacy services alongside audio‐ visual collections and internet access).  adult learners were also targeted for  library tours, reading groups, and  assistance with book selection for the  literacy collection. some of the libraries  hired new staff from outside the library  profession, choosing candidates with  prior experience in basic skills  development or community work.    • methods of attracting adults with poor  literacy ‐‐ partnership was identified as  a key strategy for the libraries studied.  partnerships were formed with  numerous agencies, including the  probationary service, a community  centre (where the library’s ‘reader in  residence’ was installed), a college, and  a peugeot factory. networking with  other literacy service providers and  coalitions was also an important  strategy, particularly as a way to  increase the library’s profile as a literacy  service provider. perhaps the simplest  strategy for attracting adults with poor  literacy was to identify areas of the  library districts where literacy skills  were lowest and then to target literacy  service to those regions.    • sustainability and mainstreaming ‐‐ early  planning for sustainability was crucial.  incorporating funding for literacy  staffing and collections into the core  budget and annual library plan was also  an important step. while some libraries  hired new staff, and one library staffed  the literacy project with volunteers,  using existing staff for adult literacy  work proved to be more efficient and  sustainable. instilling a sense of  ownership in the project for both staff  and users of the literacy services by  involving them in the development and  promotion of literacy service and  collections was another strategy  employed to ensure longevity of the  service.     conclusions – the most successful form of  library literacy service provision was found  to be the reader development approach  (promoting reading for enjoyment and  building reading activities around existing  interests). the libraries studied showed an  understanding of the wide range of reading  levels and interests among adult learners.  potential barriers for libraries in the  provision of adult literacy service “include  restrictive funding criteria, limited staff  capacity, and a bidding culture that remains  unsympathetic to public library  circumstances” (44).  the authors make five  recommendations for best practices in adult  literacy service provision:    evidence based library and information practice 2006, 1:2      38 • eclectic adult literacy collections:  collections should be fresh and  appealing  and should incorporate  engaging non‐fiction.    • standardized criteria for adult literacy stock:  standardized criteria should be  developed by a basic skills agency,  preferably at a national level.    • equality for adult readers: approach adult  readers as people who read for  enjoyment or who are ‘getting back to  reading’, rather than as those needing to  ‘improve’ their reading.    • maximum access: ensure a diverse and  well‐stocked collection of books that is  easy for adult learners to locate.    • community profiling: optimize service  delivery by profiling your community’s  literacy levels.    commentary    the study successfully consolidates past  research and the wealth of detail generated  by the eight case studies.  as this was a  student project, limited resources may  explain several weaknesses in the study’s  design. however, these weaknesses do limit  the reliability of the findings. only one  informant (in each case a literacy program  director) was interviewed at each location,  removing the opportunity for cross‐ validation. since directors were asked to  evaluate the success of their own programs,  a significant potential for bias must be  assumed. because interviews were varied in  length and format, the study cannot be  replicated, and the opportunity for formal  cross‐case comparison is minimized.     the authors specify that this research is  intended to redress a perceived lack in the  “identification of best practice and  recommendations for library authorities  intending to establish or review their adult  literacy provision” (39), rather than to  replicate existing research. nonetheless, it  would have been interesting to see in more  detail how quantitative data (where  available from the subject libraries) support  or detract from the conclusions reached in  the study.  for example, although the  authors determine that a ‘quick reads’  approach was the most successful, it is  unclear how this was measured. were there  higher circulation counts on those literacy  materials?     although some of the recommendations  may seem unsurprising (for example, the  admonition to develop appealing  collections), the attempt to crystallize the  experiences of the agencies under review  into concrete recommendations is eminently  worthwhile. furthermore, the authors’  suggestion that standardized criteria for  adult literacy materials be developed at the  national level could save valuable time for  local institutions. another useful item  would be a standardized toolkit for literacy  program evaluation, and some work has  already begun in this area (for example, in  the laser foundation’s libraries impact  project, and in the  national literacy trust’s  mapping the territory and vital link  initiatives).    this research will be of interest to public  librarians establishing or reviewing adult  literacy services.     works cited    “libraries impact project.” a report  commissioned by the laser foundation.  pricewaterhousecoopers llp. july 2005.  10 may 2006  <http://www.bl.uk/about/cooperation/p df/laserfinal6.pdf>.     “mapping the territory, the links between  museums, archives and libraries, and  adult basic skills.” national literacy  trust. 2006. 10 may 2006.  <http://www.literacytrust.org.uk/mappi ng/>.   http://www.bl.uk/about/cooperation/p http://www.literacytrust.org.uk/mappi evidence based library and information practice 2006, 1:2      39 “the vital link, bringing libraries and  adult literacy together through  reading for pleasure.” national literacy  trust. 2006.10 may 2006  <http://www.literacytrust.org.uk/vitallin k/index.html>              http://www.literacytrust.org.uk/vitallin news   call for papers: eblip10, 15-19 june, university of strathclyde, scotland, uk      2018. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29504     conference theme: "using evidence in times of uncertainty"   the use of evidence is becoming more and more important as a routine part of library planning and decision making. whilst in an era of library de-funding, calls for accountability, fake news, and complex information environments, evidence is becoming an essential component of advocacy.   this inclusive conference theme will encourage practitioners and researchers to think about what evidence is, how we collect it, and how we use it in these uncertain times.   the conference aims to offer a stimulating and varied programme of workshops, presentations, posters and discussions designed to appeal to library and information professionals convinced about using evidence-based approaches as well as those who may have previously thought that evidence-based practice (or attending an eblip conference) wasn’t for them. pre-conference workshops covering a range of topics relating to research, evaluation and evidence based practice will be held on 15-16 june 2019. the main conference will take place 17-19 june 2019. the conference will provide a practical and accessible forum for librarians and information practitioners from all sectors to discover, use and disseminate evidence that may contribute to decision-making and advocacy in today’s professional practice. this includes (but is not inclusive to) those who:   are interested in the evaluation of library services or library assessment need to demonstrate the impact or value of their library service wish to build an evidence-based approach to their practice collect data about their services but don’t know what to do with it wish to include the user experience in library decision making are interested in methods for evidence-based library and information practice   eblip conferences tend to be relatively small and particularly friendly places offering a stimulating environment for librarians across sectors to meet and share ideas. we welcome librarians attending an eblip conference for the first time and have provided a lively social programme to complement the academic content.   proposal submissions:   submissions are invited in the form of 1) papers (for an oral presentation 20 minutes) or 2) posters. for papers and posters, abstracts of no more than 350 words should be submitted using a structured abstract form (title, aim, methods, results, discussion/conclusion).    submissions should address an area related to the conference theme – “using evidence in times of uncertainty.” all submissions will be subject to double blind peer review by members of the international programme committee. all submissions must be via the easychair conference management system at https://easychair.org/conferences/?conf=eblip10.   following the conference, authors are encouraged to submit their papers to practitioner focussed open access journals. following each eblip conference, the eblip journal invites authors to contribute to a special feature.  see https://journals.library.ualberta.ca/eblip/index.php/eblip/issue/view/1942 for the feature from eblip9.   key dates:   7 september 2018 – call for papers opens 30 november 2018 – call for papers closes 15 january 2019 – authors of submissions notified of decision 15 february 2019 – deadline for authors of submissions to confirm participation spring 2019 – registration opens   for further information, please consult the conference website – https://eblip10.org   submission:   to submit an abstract access: https://easychair.org/conferences/?conf=eblip10   if you already have an easychair account, login.  alternatively, you can create an account.   click on submissions click on add submission (top right hand corner of screen) enter the author details enter your abstract – please remember – this should be in a structured form (title, aim, methods, results, discussion/conclusion) and not more than 350 words  indicate whether your abstract should be considered as a paper or poster   for programme queries, please contact the international programme chair, alison brettle – a.brettle@salford.ac.uk   please follow us on twitter @confeblip and post using #eblip10.     evidence summary   academic libraries report minimal standardization and oversight of libguide content   a review of: logan, j., & spence, m. (2021). content strategy in libguides: an exploratory study. the journal of academic librarianship, 47(1), article 102282. https://doi.org/10.1016/j.acalib.2020.102282   reviewed by: sarah bartlett schroeder research & instruction librarian university of washington bothell/cascadia college campus library bothell, washington, united states of america email: sarahkb6@uw.edu   received: 30 mar. 2021                                                                 accepted:  24 may 2021      2021 bartlett schroeder. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29955     abstract   objective – to determine what strategies academic libraries use to govern creation and maintenance of their libguides.   design – online survey questionnaire.   setting – a selection of academic libraries that use springshare’s libguide system, mainly in the united states and canada.   subjects – academic libraries with administrator level access to libguides at 120 large and small, private and public schools.   methods – researchers made their online questionnaire available on a springshare lounge and recruited participants through electronic mailing lists. respondents were self-selected participants. the survey consisted of 35 questions, including several about their institution’s size and type, the number of libguides available through their library, and how their guides are created and reviewed. there was space available for comments. the survey stated that the researchers’ goal is to complete an “environmental scan of content strategies” in libguides at academic institutions.    main results – of the 120 responding institutions, 88% are located in either the united states or canada and 53% reported that they do have content guidelines for libguide authors. content guidelines might include parameters for topics, target audiences, or purpose. parameters for structural elements, including page design, content reuse policies, naming conventions, and navigation, were most commonly represented at those institutions that reported having guidelines. seventy-seven percent of respondents reported that their libguides do not go through a formal review process prior to publication.   regarding libguide maintenance, 58% reported that libguides are reviewed as needed, while 27% indicated a more systematic approach. in most cases, the libguide reviewer is the author, though sometimes a libguide administrator may take on a review role. the most common considerations for libguide review are currency, accuracy, usage, and consistency. of the responding institutions, 74% reported that they do not conduct any user testing of their guides.   two of the biggest barriers to introducing and maintaining libguide guidelines identified in the survey were lack of time and a sense of librarian ownership over content and workflow. the strong culture of academic freedom may make some librarians resistant to following institutional guidelines. survey respondents noted that, where content guidelines are present, they tend to address “low hanging fruit” issues, such as page design and naming conventions, rather than more complex issues around tone and messaging.   conclusion – content creators tend to have many competing priorities, so a workflow and guideline system might help librarians spend less time on their guides. despite a large amount of research on libguide best practices regarding content strategy, few institutions seem to be taking systematic steps to implement them. further research examining the experiences of libguide authors and administrators and on the effectiveness of content strategy practices is necessary.   commentary   as noted in the introduction to this study, much of the previous research on libguide content management has focused on best practices. some researchers have explored content management at the institutional level. notably, mcdonald and burkhardt (2021) summarized findings from a survey of content managers at academic libraries and found that “web content strategy as a community of practice in academic libraries is operating at, or just above, a basic level” (p.15). this is similar to logan and spence’s findings. a previous article by mcdonald & burkhardt (2019) explored the relevance of cohesive content strategy given the proliferation of libguide content at large institutions.   this review used a tool from the center for evidence-based management (n.d.) to evaluate the survey’s design quality. this survey included clear research questions regarding how academic libraries govern their libguides and what strategies they use and successfully gathered information on the prevalence of libguide guidelines at academic institutions.   survey participants were self-selected based on their participation in online forums and electronic mailing lists and their willingness to take the survey, a shortcoming acknowledged by the researchers several times. however, data concerning the location, size, and degree-type of responding institutions was collected and reported, and the survey instrument was included as an article appendix. responses to a number of questions included in the survey, such as those addressing libguide workflow, could have potentially added depth to the article discussion, but were not reported in the results.   this study provides few recommendations for organizations seeking to improve libguide cohesion, as though it is not yet known how content guidelines influence the quality of guides. nonetheless, organizations may be able to review and compare their own internal libguide practices in relation to the data presented, considering their content strategies from a different perspective.   references   center for evidence-based management. (n.d). critical appraisal of a survey. https://www.cebma.org/wp-content/uploads/critical-appraisal-questions-for-a-survey.pdf   mcdonald, c., & burkhardt, h. (2019). library-authored web content and the need for content strategy. information technology & libraries, 38(3), 8–21. https://doi.org/10.6017/ital.v38i3.11015   mcdonald, c., & burkhardt, h. (2021). web content strategy in practice within academic libraries. information technology & libraries, 40(1), 1–46. https://doi.org/10.6017/ital.v40i1.12453   news   join fellow researchers & practitioners at the eblip9 conference      2017. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     plan to join other researchers and practitioners at the 9th international evidence-based library & information practice conference, june 18-21, 2017.  hosted by drexel university, the eblip9 conference is dedicated to exchange of ideas and networking around evidence-based practices.   this year marks the first time since 2007 that the eblip conference will be held in the united states, and what better place for a conference than philadelphia, a city known for its rich history, numerous museums, top-tier universities and iconic landmarks, as well as the famous philly cheesesteak and other culinary delights.    we are excited to bring three inspiring keynote speakers with their unique perspectives on evidence-based practices. from canada, pam ryan, director for service development & innovation at the toronto public library, will address the everyday practices of librarians. dean yin deng, college of computing & informatics at drexel university, usa, will explore data in building the 21st century economy; and professor alison brettle, school of nursing, midwifery & social work at the university of salford, uk, will examine collecting and using evidence for advocacy. in addition, attendees will have a rich selection of more than 40 presentations, poster sessions and numerous networking opportunities, as well as optional pre-conference workshops to customize their program.   early bird registration closes on april 30, 2017. register online today!   full conference registration gives you: the complete three-day eblip conference program opening night welcome reception at the historic free library of philadelphia networking conference dinner at the elegant crystal tea room a variety of other networking opportunities with your peers and colleagues daily lunch and session break refreshments   the association of research libraries (arl) is sponsoring four scholarships to cover conference registration for historically underrepresented racial and ethnic groups to attend the 2017 eblip conference. the deadline for online applications is friday, march 31 at 11:59pm est. four additional scholarships will be awarded to first-time attendees who entered the association of college & research libraries’ [acrl] sponsored competition.    we also thank our other 2017 conference sponsors for their generous support: elsevier, medical library association (mla), the university of saskatchewan’s centre for evidence based library & information practice, and the university of tennessee knoxville’s school of information sciences. additional sponsorship opportunities are still available. the complete conference program is forthcoming, so please check our website www.eblip9.org for regular updates or follow us on facebook and twitter.   we hope to see you here in philadelphia!   danuta nitecki & john wiggins co-chairs, eblip9 conference drexel university   editorial   we want you to hear from us, and we want to hear from you!   lorie kloda editor-in-chief associate university librarian, planning and community relations concordia university montreal, quebec, canada email: lorie.kloda@concordia.ca      2016 kloda. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     the current issue of evidence based library and information practice (eblip) is a full one. in addition to research articles and evidence summaries, we have yet another feature on the most recent library assessment conference which took place in seattle, washington, in 2014. this is the second time we have had a feature from this conference, and we look forward to continuing the tradition with the upcoming 2016 library assessment conference.   also in this issue are several news items linked to the journal and to evidence based librarianship. conference and meeting announcements, including the 9th international conference on evidence based library and information practice (eblip9) and the centre for evidence based library and information practice (c-eblip) symposium, are included. look for features in the journal on these meetings in the future. an item announcing a call for a library researcher-in-residence is a timely accompaniment to the research in practice column, co-authored by virginia wilson and selinda berg on the benefits of such a program.   recently, an eblip publication received the jesse h. shera award for distinguished published research. this is not the first time an article in this journal has won an award – eblip continues to publish high-caliber research that makes an impact on the profession.   and finally, our journal continues to grow. in the news/announcements section as well, see our call for two positions: an associate editor (reviews and classics) and the newly created communications officer. the latter position is anticipated to expand the journal’s reach, to coordinate promotion of the journal, including all of its achievements, and to identify further opportunities for collaboration. i’m very excited about this position, as i think it is an excellent opportunity for someone with an interest in open access scholarly publication and with the appropriate skills and also and opportunity for the journal to develop a stronger relationship with its current readers and to reach new audiences. if you have ideas or suggestions about how you would like to hear about eblip news, by all means, please get in touch with me or another member of the editorial board.   research article   identifying social care research literature: case studies from guideline development   claire stansfield senior research associate evidence for policy and practice information and coordinating (eppi-) centre social science research unit, ucl institute of education, university college london london, united kingdom email: c.stansfield@ucl.ac.uk kristin liabo senior research fellow university of exeter medical school exeter, united kingdom email: k.liabo@exeter.ac.uk   received: 24 nov. 2016  accepted: 22 june 2017        2017 stansfield and liabo. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – systematic searching is central to guideline development, yet guidelines in social care present a challenge to systematic searching because they exist within a highly complex policy and service environment. the objective of this study was to highlight challenges and inform practice on identifying social care research literature, drawing on experiences from guideline development in social care.   methods – the researchers reflected on the approaches to searching for research evidence to inform three guidelines. they evaluated the utility of major topic-focused bibliographic database sources through a) determining the yield of citations from the search strategies for two guidelines and b) identifying which databases contain the citations for three guidelines. the researchers also considered the proportion of different study types and their presence in certain databases.   results – there were variations in the ability of the search terms to capture the studies from individual databases, even with low-precision searches. these were mitigated by searching a combination of databases and other resources that were specific to individual topics. a combination of eight databases was important for finding literature for the included topics. multiple database searching also mitigates the currency of content, topic and study design focus, and consistency of indexing within individual databases.   conclusion – systematic searching for research evidence in social care requires considerable thought and development so that the search is fit for the particular purpose of supporting guidelines. this study highlights key challenges and reveals trends when utilising some commonly used databases.     introduction   as people are living longer with more complex conditions, there is a need for a more integrated health and social care system. in 2012, the national institute for health and care excellence (nice) in england broadened its remit on health to develop national quality standards and guidelines for social care. the nice collaborating centre for social care was set up in 2013 and has addressed cross-cutting themes spanning health and social care through the provision of guidelines. social care "generally refers to all forms of personal care and other practical assistance for children, young people and adults who need extra support" (nice, 2014, glossary). the nice guidelines contain recommendations for individual health and social care practitioners, local authorities, health and social care commissioners, providers of services, and other organizations (nice, 2014). the procedures for developing social care guidelines were informed by the processes used for clinical guidelines. guideline committees develop the guidelines and are supported by evidence review teams, who undertake systematic literature searches and review the best available evidence (nice, 2014).   both health and social care service fields contain complex systems, and there are similarities in reviewing research evidence in these areas. however, as part of developing social care guidelines, important differences in the respective research traditions meant that reviewers and information scientists in the new collaborating centre had to consider whether a unique approach was needed. the purpose of this study is to focus on lessons learned from the systematic searching undertaken to support the evidence reviews that inform the development of guidelines. drawing on analyses of three searches for social care guidelines, we describe some challenges and reflect on the utility of these searches. the three guidelines focused on social care support for people across three distinct topics: 1) home care—delivering personal care and practical support to older people living in their own homes (nice, 2015); 2) transition between inpatient mental health settings and community or care home settings (nice, 2016b); and 3) transitions from children’s to adults' services for young people using health or social care services (nice, 2016a). the latter two guidelines also covered support for people using health services.   literature review: the peculiarities of searching for social care research   challenges of seeking diverse literature for guidelines have been recognised in public health (levay, raynor, & tuvey 2015). like public health, social care sits in between other services by its very nature and purpose. this is reflected in the research literature, which uses a diverse terminology and is published within a range of disciplines, such as social sciences, health, criminal justice, and housing (clapton, 2010). the literature is varied in format, with reports and unpublished literature making up a significant proportion (clapton, 2010). to account for this, golder, mason, & spilsbury (2008) suggest searching a number of different sources that cover different disciplines and using broad search strategies that encompass many variants of terminology. a number of case studies in this field recommend that social care systematic reviews utilise databases drawn from the broad fields of health, social sciences, and social care or those that contain multidisciplinary sources (brettle & long, 2001; clapton, 2010; golder et al., 2008, mcelhinney, taylor, sinclair, & holman, 2016; mcfadden, taylor, campbell, & mcquilkin, 2012; mcginn, taylor, mccolgan, & mcquilkan, 2016; steventon, taylor, & knox, 2016; taylor, wiley, dempster, & donnelly, 2007; taylor, dempster, & donnelly, 2003). clapton (2010) found that a minimum of seven or eight databases needed to be searched to capture the relevant references for three reviews on looked-after children (children under care), and the exact selection of databases is highly dependent on topic. mcginn et al. (2016) show that it is difficult to predict the best databases across several social care topics. national context is also important. the reviews studied by clapton (2010) informed a uk context, and she suggests searching uk-focused databases to add local context and reduce north american bias from commonly used databases.   developing search strategies to capture the diverse terminology and research literature within social care literature is therefore a challenge. steventon et al. (2016) considered approaches for a search about risk communication and risk concepts in dementia care. they found that care as a concept was too diffuse as it encompassed location of care, types of carer, range of professionals involved in care, specific care services, quality of care, service policy, and practice issues. golder et al. (2008) observe that alternatives for the term carer include phrases such as husbands supporting their wives or children caring for their elderly relatives, and there may be specific terms for paid and unpaid staff, (e.g., care worker or volunteer). they also note national differences, whereby the term carer is common in the united kingdom, but caregiver or caretaker are terms used in the united states.   given that social care research is considered difficult to identify, it is of interest to assess how well systematic searches locate what is present within a database. a thoughtful search strategy "considers the aim of searching, ensuring that the appropriate methods are used; what the most relevant sources of studies are likely to be; the benefits and drawbacks of searching each source; the resources available; ... appropriate search terms; and the benefits and costs of different combinations of sources within the available resources" (brunton, stansfield, caird, & thomas, 2017, p. 105). the case studies referred to above are based upon analysis of search results to assess which studies were identified from which database. while these findings help to indicate a database's usefulness to individual topics, they depend on the search strategies used.   more informative is the analysis by golder et al. (2008) for a review on the effectiveness of respite care for carers of frail older people. they found that for the majority of databases their search strategies failed to retrieve some relevant references that were stored in these databases, despite using a very broad search. reasons were that the bibliographic details lacked one of the concepts in their search, one of the concepts was expressed using ambiguous phrases, or records did not contain abstracts. they found that the studies from their review could be identified using their search strategy on six databases (ageline, embase, health management information consortium (hmic), medline, psycinfo, social sciences citation index (ssci)) plus reference checking and contacting authors. they checked which databases contained these studies and found that the same six sources collectively contained all the included references. the search strategy used in their systematic review identified unique references (i.e., items found from only one of the resources searched) in four databases: ageline, embase, psycinfo, and ssci. the authors found that reference checking and contacting authors are also valuable sources of unique relevant references and provide materials not available through the use of databases.   a similar investigation by bayliss & dretzke (2006) found that in seven out of nine databases investigated, their searches failed to locate relevant studies that were present for a technology assessment report on a parenting intervention. reasons included: the bibliographic details lacked either the study design or subject elements used in their search, the index terms searched, and the absence of abstracts. the difficulty of missing items is not limited to social care. in analyzing the citations of nine systematic reviews of diagnostic test accuracy, preston, carroll, gardois, paisley, & kalthenthaler (2015) found 11% of citations were indexed in either medline or embase but were not retrieved by the searches used for each review. we are aware that search strategies can never be perfect and will never retrieve every relevant reference (brettle et al., 1998, brunton et al., 2017, p. 98), and other constraints include the time and resources available to search (brunton et al., 2017, p. 97). while the balance between sensitivity and precision in systematic searches needs to be grappled with across various research fields, we suggest social care searching merits further investigation, based on the literature reviewed here and our own experiences of working on social care guidelines as well as systematic reviews in health and education.   aims and objectives   this study aims to draw on the experiences of identifying social care research for three guideline topics in order to highlight challenges and inform practice. we write from the perspectives of an information scientist charged with designing the searches based on the scope of the guideline and a reviewer charged with screening citations and appraising included studies. we hope that by analyzing the utility of our own approaches we can share knowledge on how social care searching can be developed further. specifically, our objectives are to:   a)       reflect on the challenges of searching for three guideline topics; and b)       evaluate the utility of major topic-focused bibliographic database sources for identifying research.   methods   overview of the approach to literature searching for the three guidelines   each guideline was preceded by a referral to the nice collaborating centre for social care, based on a population need identified through policy and practice. this referral was developed into a scope following consultation with stakeholders. the scope outlines the importance of the topic and the remit of the guideline in terms of populations, settings, and interventions. as part of developing the guidelines, each topic contains approximately seven sets of research questions relating to the effectiveness of interventions, people’s experiences of them, and barriers and facilitators to service delivery or interventions. table 1 provides examples of each type of research question for each topic.   each literature search utilized over 20 bibliographic databases comprising international and uk-focused health, social science, social care, and economic databases. the searches were supplemented by mainly uk-focused website searches, specialist registers, and catalogues, some citation searching, and contributions from the guideline development team. the search resources differed across the three topics, and the analysis presented here focuses on general databases in health, economics, social sciences, and social care. the full search strategies and reviews are reported elsewhere (nice, 2016a, 2016b, 2015). the original database searches were updated after one year to identify new research on the effectiveness of interventions. the searches were followed by screening and appraising studies for relevance to the review questions. studies deemed relevant to the review questions were then included to inform recommendations within the guidelines.     table 1 examples of types of research questions for each guideline types of research questions guidelines home care: delivering personal care and practical support to older people living in their own homes transition between inpatient mental health settings and community or care home settings transition from children’s to adults' services for young people using health or social care services effectiveness of the interventions what approaches to home care planning and delivery are effective in improving outcomes for people who use services? what is the effectiveness or impact of interventions, components of care packages, and approaches designed to improve discharge from inpatient mental health settings? what is the effectiveness of interventions designed to improve transition from children’s to adults’ services? people’s experiences what are users’ and family carers’ experiences of home care? what are the views and experiences of people using services in relation to their admission to inpatient mental health settings from community or care home settings? what are young people’s experiences of transitions? what works well? barriers and facilitators to specific interventions what are the significant features of an effective model of home care? how do different approaches to assessment, care planning, and support (including joint working) affect the process of admission to inpatient mental health settings from community or care home settings? what factors help and hinder purposeful and planned transitions from children’s or adolescents’ to adults’ services, as identified by young people, their families and carers, practitioners, and research?       table 2 databases searched and the database platform database platform british education index (bei), cinahl plus, econlit, eric, medline ebsco british nursing index (bni) hdas amed, embase, medline, psycinfo, health management information consortium (hmic), social policy and practice (spp) ovid assia, eric, international bibliography of social sciences (ibss), library and information science abstracts (lisa), pais, pilots, sociological abstracts, social services abstracts, worldwide political science abstracts (wpsa) proquest social sciences citation index (ssci) web of science     methods of analysis   the search protocols and key internal documents related to developing the search strategies for each guideline were revisited. common challenges were identified, and key ways to address these were noted.   a three-step process was used to investigate the utility of database sources. first, for two of the guidelines (child to adult services and mental health setting transitions) the citations used to provide research evidence were traced back to their original sources. these two guidelines contained 81 and 71 citations, respectively. the citations were checked against the original search results prior to duplicate checking and prior to the update searches. for the homecare guideline, citations located outside databases were also investigated. second, the presence of the 225 citations from all three guidelines was checked in 20 major topic-focused bibliographic databases at least one year after the original searches. this was undertaken by searching the fragments of titles for each citation. databases hosted within the same platform were searched together and are listed in table 2. these include many of those that had been searched for each topic plus some additional databases. these databases were selected for being important topic-focused databases and convenience of analysis. third, the sources of the original searches and the studies present within the databases were compared for two guidelines, and we observed some reasons for disparities across selected citations.   analysis was undertaken using the systematic review management tool, eppi-reviewer 4 (thomas, brunton & graziosi, 2010). eppi-reviewer 4 is a web-based electronic software for managing systematic reviews that is based on hundreds of reviews supported by or conducted at the eppi-centre (http://eppi.ioe.ac.uk/cms). it facilitated the assigning and analysis of codes for each citation relating to review questions, study designs, and databases.   results   designing the search strategies for the three guidelines    as part of the guideline development process, the information scientist developed searches that aimed to be sensitive in retrieving most of the studies available but balanced with retrieving a manageable number of citations to screen. as the guidelines were led by a stringent timeframe, comprehensiveness of searches needed to be balanced with the time available to complete the reviews, which included the reviewers’ need for time to screen and review studies. the number of records screened from the database searches ranged from around 14,500 to 21,400 per topic, after removal of duplicates. the number of citations of studies that were used to inform each guideline ranged from 71 to 81 per topic, which equates to an individual search precision of under 0.6%.     table 3 common challenges and selected examples across the guidelines challenges guidelines homecare for older people transition between mental health inpatient and community settings transition from children’s to adults’ services population encompasses individuals to organizations or settings individual homecare staff and social care agencies individual patients returning home and service transitions such as secondary care to primary care young people transitioning and children’s and adult health and social care services some relevant controlled vocabulary have broader focus than the topic and some mesh term examples home nursing discharge continuity of patient care patient transfer continuity of patient care patient care planning irrelevant studies retrieved in test searches clinical studies on home nursing in medical databases studies on prevalence or characteristics of the population transition has multiple meanings (e.g., physical and emotional development; life change transitions, such as parenting, education and employment)       table 3 summarizes, with selected examples, some of the challenges in developing a search strategy. a particular challenge for the guideline topics described here was that their titles and referrals did not follow a traditional pico structure (population, intervention, comparator, outcome), and neither did many of their questions. most notably, no topic operated with specific outcomes for the interventions and all included open-ended questions. to address this, the information scientist aimed to work closely with the reviewers to clarify ambiguous aspects of the scope and the review questions. for each guideline, the concepts common to each review question, such as populations and setting or context, were identified. in these instances, it was possible to construct one literature search to address the review questions for each guideline topic. a diverse range of search terms were needed for each concept and developed from several test searches.   across all topics, the population concept encompassed various groups of individuals and organizations. for example, the population concept in the homecare topic included older people, homecare staff, carers, social services, or integrated services. for the mental health setting transitions topic, the population was informed by the setting; it included people who were either entering or leaving inpatient mental health settings. relevant literature might describe the population in terms of people with a mental health disorder and indicate that they are in hospital, or it might describe the mental health unit.   a second concept was used in each topic. for the two topics on transitions, this involved a setting element (such as discharge from hospital to home or moving to adult services), but it also involved a process of transition and included interventions, such as transition planning or treatment education. for homecare, this concept related to the setting and intervention (e.g., care in the home). articulating this second concept was challenging for all topics owing to the diversity of terminology present in relevant literature.   there was also a problem of context. in the topic on child to adult services, the focus was on care transitions in both in health and social care services. however, transition is also a term used to describe facets outside this focus. for example, it can mean transition in terms of adolescent physical and emotional development or life changes, such as parenting, educational achievement, and employment. as some literature about education and developmental transitions is interlinked with research on care transitions, the former topics could not be automatically excluded from the search. to help counter this, a broad range of qualifying terms was used in the free-text searches so that transition had to appear with terms that were indicative of care or transition planning (e.g., care, pathways, readiness, failures, or schemes). where possible, proximity searching was used, which involved deciding on an arbitrary distance of words between transition and other relevant search terms. for this topic, we decided not to search education databases, but we searched and browsed uk government websites related to the education system for relevant research relating to health and social care service transitions.   the focus of transition between two settings or inpatient mental health settings and community was particularly challenging to articulate. as well as discharge, admission, or transition, there could be a variety of ways to describe the process, such as a person leaving hospital, moving home, returning to the community, or receiving aftercare services. there could also be a change of service provider, such as moving between primary and secondary care. a range of free-text and controlled terms was used to capture this literature for individual and service level transitions in a focused way.   some relevant controlled vocabulary terms tended to have broader focus than the topic. for example, the medical subject heading (mesh) term continuity of patient care was used in both transitions topics. controlled vocabulary focused on transitions to services is usually absent. the mesh term transition to adult care was introduced in 2012. to identify earlier literature, the mesh terms continuity of patient care, patient handoff, patient transfer, and patient care planning were each used in combination with mesh terms for children and adolescent services. to increase search precision, the mesh term patient care planning had to occur with the term adult in the title or abstract.   all searches retrieved large numbers of irrelevant studies, and we took steps to reduce this. for mental health setting transitions an inclusive study design filter was used in databases that yielded very large search results to capture trials, cost-effectiveness and qualitative studies, and research on people’s views or opinions on services. the purpose for using the filter was to reduce the yield of studies on prevalence or describing specific characteristics of the population. we decided to search the education databases british education index and eric using a focused search that aimed to capture studies on students returning to school following time in hospital without capturing literature about educational transitions. one particular difficulty that could not be resolved in the homecare topic was being unable to distinguish between clinical studies and social care in large health databases. the mesh term home nursing was relevant to the homecare topic but also captured clinical aspects of homecare beyond the guideline focus. for the child to adult services topic, we initially considered using recently published systematic reviews to avoid duplication of effort, but this was not possible during the searching and screening stages as the existing reviews we found did not fully cover a group of interest or some aspects of interest for that group. for example, while there was good coverage on effectiveness and views in some health settings, this material did not provide evidence on related areas, such as barriers and factors to specific interventions. there also seemed to be a gap in reviews on social care transitions for young people with physical disabilities. in compiling the evidence for the guideline, once screening had taken place, some findings of systematic reviews were considered collectively where appropriate.   it was important that appropriate database and website sources were searched to reflect the range of sectors, settings, and outcomes within the scope of each individual guideline. we supplemented database searches by browsing searching on websites for different population groups.     table 4 citations found from the systematic review searches for two guidelines database child to adult services n=81, % (number unique) mental health setting transitions n=71, % (number unique) assia 14 14 bni 20 (1) 17 (3) central 1 34 cinahl 30 n/a cinahl plus n/a 38 (1) dare 2 0 econlit 0 0 embase 41 (1) 44 eric n/a 1 (1) hmic 5 (2) 14 ibss 1 3 medline 48 (5) 59 (3) nhs eed 0 3 psycinfo 35 (5) 56 (2) ssci 40 (1) 44 (2) spp 31 (7) 11 (2) social services abstracts 2 3 social work abstracts 0 0 sociological abstracts 1 0 zetoc n/a 4 (1) nspcc inform 4 n/a other sources (unique) 14 (websites, trials registry, early scope work) 6 (citation searching, trials registry, websites)     one way to manage the time needed to conduct the evidence reviews was to only include studies published after a particular date. these varied across the guideline topics and for individual questions. individual evidence reviews were sometimes limited to the uk context. decisions on such restrictions were in agreement with the guideline committee for that topic, were used where deemed appropriate for a specific reason (such as changes in practice), and were used after searching and screening had taken place. the actual searches were carried out at the farthest time point and not limited by geography. language exclusions were applied within the databases as the final step in the search for two topics so that the number of citations prior to this exclusion was transparent.   references identified by the search strategies for two guidelines   table 4 shows the yield of relevant citations identified from each resource using the systematic searches for the child to adult services and the mental health setting transition topics. nearly all the databases searched yielded some studies, and comparing both topics, performance was similar for most of the databases. notable differences are central (the cochrane trials database), which was much higher yielding for the mental health setting transitions topic, and social policy and practice (spp), which was higher yielding for the child to adult services topic. embase, medline, and ssci each yielded over 40% of the citations for the child to adult services topic. for the mental health setting transitions topic, psycinfo and medline, perhaps not surprisingly, yielded over 55% of studies. social work abstracts and econlit did not yield any studies. some studies were only found in one database, and these databases differed between the two topics.   considering the contribution of databases collectively, the following eight databases yielded 89% (135/152) of studies: british nursing index (bni), cinahl, embase, hmic, medline, psycinfo, spp, and ssci. the remaining studies were found from sources outside the databases listed, plus one study for each topic was found from focused searches of eric and zetoc. sources outside bibliographic databases were important for identifying a small number of studies not located elsewhere, yielding 14% of citations for child to adult services and 6% of studies for mental health setting transitions. this contrasts with the homecare topic where 23% were found from sources outside bibliographic databases.   references present within the databases for three guidelines   table 5 shows the number of studies present in each database for the three guideline topics. the majority of studies were present within these databases. individual databases containing the most studies varied by topic. for child to adult services, ssci, cinahl, and embase each contained 52% of studies, or 70% collectively (n=57/81). for mental health setting transitions, psycinfo contained 90% of studies (n=64/71), and for home care, spp contained 77% of studies (n=56/73). a large amount of overlap exists across the databases. meanwhile, 10% (n=23/225) of studies were located only in one of the databases searched. the majority (8%) of these were from spp, plus two citations from hmic, and one from embase. some citations were not present in any of the databases searched for the child to adult services (7%, or six citations) and homecare topics (4%, or three citations). twelve citations were found only in one place. of these, 10 were from spp, and two were from hmic.   for the child to adult services topic, the minimum combination of databases to get all of the citations was embase, spp, and psycinfo. for mental health setting transitions the combinations were less clear, owing to large overlap between the databases. for the homecare topic, cinahl plus, hmic, and spp collectively yielded all the studies present within the databases. for all of the topics, seven databases provided 96% (n=215/225) of citations (cinahl plus, embase, hmic, medline, psycinfo, spp, and ssci). eric yielded one unique study, and the remaining nine studies were not present in any database.     table 5 citations present in each database for the three guideline topics database child to adult services, n=81 % (number unique) mental health setting transitions, n=71 % (number unique) homecare, n=73 %  (number unique) amed 4 11 7 assia 19 24 27 bei 2 1 0 bni 23 30 14 cinahl plus 52 65 53 econlit 0 0 1 embase 52 (1) 79 30 eric 9 4 (1) 0 hmic 11 23 62 (4) ibss 1 4 15 lisa 0 0 1 medline 51 77 32 pais 5 0 1 pilots 0 1 0 psycinfo 41 90 22 sociological abstracts 1 0 8 ssci 52 77 34 social services abstracts 7 8 18 spp 43 (9) 21 77 (8) wpsa 0 0 0 not present in any of these databases 7 0 4       one of the main differences between health research and social care is that in health the concept of intervention is well established, and the randomised controlled trial (rct) is the gold standard for investigating the role an intervention can play in service delivery. in clinical guidelines focussing on drug interventions, the rct is central and searches on these topics may include methods filters. these filters exclude studies that do not meet established standards for health research, thus increasing the precision of searches. in social care, less work is done on interventions as such.     instead, many studies investigate approaches and ways of working, and the role of the rct is less prominent. therefore, considering which study designs were found in which database is relevant (figure 1). four databases yielded over three times more studies on people's views and experiences than on effectiveness (assia, bni, spp, and hmic). cinahl and spp provided the most studies about people’s views. studies evaluating cost effectiveness formed a very small portion of studies, and these were present in most of the databases. for figure 1, some of the studies had overlapping categories, and the other category relates to studies concerning factors that help or hinder an intervention and that do not fit within other study designs.         figure 1 citations present within each database for different study designs, % of total, n=225 (values under 2% are not annotated).     table 6 sources present within selected databases by study type study design n=225 cost n=15 systematic reviews n=26 views and experiences n=123 effectiveness n=59 other n=13 total number of citations retrieved by combination cinahl plus, hmic, and either embase or medline 100% (15) ssci, psycinfo, and spp 96% (25) cinahl plus, hmic, spp, and ssci 92% (113) cinahl plus, embase, ssci, and spp 100% (59) either embase or medline, and spp 100% (13) not present in a database 1 8     table 6 provides a matrix of database sources and study designs to help visualize where certain types of studies are located. we recognize, however, that due to database overlap other possible combinations could yield the same studies. the data for table 6 was compiled using the highest yielding databases for a study design, and the databases where the unique citations were found.   comparing the performance of the databases searches   anyone who has conducted complex searches across multiple databases is aware of the curiosities encountered at times due to spelling mistakes in the original title or abstract or due to indexing interpretation. unpicking every thread of idiosyncrasy encountered across these searches would be extremely time consuming, but a selective picture of why some citations were not located by the search in some databases was gained for the child to adult services and the mental health setting transition topics. the reasons some studies were not located often applied across several databases.   some studies were not present in the database due to the time lag of adding publications to a database. from our estimates, this is less concerning for citations from bni, hmic, and medline, which each had two items missing due to this issue. the database with the largest time lag issue was embase (18 citations), followed by cinahl plus and ssci (10 citations each).    controlled vocabulary and keywords within individual databases were important for finding some citations, and this was sometimes the reason a citation present in multiple databases was identified by our searches only in one place. for example, one study present in eight databases was found in only ssci due to the citation containing the word transition in the keyword plus field. the benefit of controlled vocabulary is not consistent within databases. for example, our searches in psycinfo found two citations for the mental health setting transitions topic, owing to the subject headings psychiatric hospital admission and psychiatric hospitalization. the citations were both present in medline and ssci but not located by the search strategy. conversely, a study indexed in medline with the mesh term psychiatric hospitals was not found using the subject headings within psycinfo.   some citations, found in databases elsewhere, were missed owing to controlled vocabulary and indexing being broader than our search strategy. for example, three studies were not found in hmic because our population terms in the controlled vocabulary were more specific than that applied by the indexers. we opted not to use the controlled term transitional programs in cinahl owing to its broader scope than our focus. in ssci, three studies were not found due to filtering out studies focused on education without containing health and social care subjects; however, this approach also reduced the number of references to screen by around 400. as mentioned earlier, the number of final hits is important to the reviewers who are charged with screening within a tight timeframe. furthermore, screening studies that are situated outside of social care but imply social care in the abstract is slow and at times difficult.   the use of free-text searching with proximity for certain words to reduce the number of irrelevant citations resulted in studies being missed. at least one study was identified in the british nursing index (bni) that was missed in other databases for the mental health setting transitions topic. unlike the other database searches, no proximity was used as it was easier to translate the search without it and the relative yield of search results was low. another reason for missing studies was the absence of one search concept. for example, one study was not found in spp because it had no transition terms.    there were also other limitations from the databases. abstracts were absent from citations in some databases but were present in others. two citations were not found in hmic owing to a date limit being applied to the search, and these two citations did not contain a date in the date field. (the date was included in another field.) searching the notes field in addition to the abstract field would have been helpful for this database. furthermore, two items in two databases were found at the time of searching but were not present when checked at a later date.   discussion   while some findings are largely technical and specialist, they have a direct relevance to policy and practice. with the aim to develop research-based social care and increase our understanding of cost-effective services in this field comes the need to search efficiently and effectively for relevant research.   key challenges and implications   the development of guidelines is largely underpinned by methods developed within the health field, and following these within a social care framework can be challenging. our analysis shows that it is possible to conduct systematic and useful searches for social care guideline development within this context. because most of the questions were driven by their relevance to practice, it was not clear how well some areas were researched. having a potential paucity of literature, coupled with challenges in identifying it, drove the searches to be sensitive within resource constraints. this might not be possible to achieve or appropriate for all social care guidelines.   designing search strategies for the three guidelines in this analysis was challenging because they asked a range of questions across broad topics about both individuals and services and encompassed multiple outcomes. however, our searches were able to capture most of the evidence from the collection of databases searched. collective searches identifying literature for several questions within each guideline topic meant that fairly sensitive searches could be undertaken. the contribution of studies sourced outside databases was fairly low for the topics on child to adult services and mental health setting transitions. this could be due partly to the sensitivity of the databases searches and partly that less relevant literature existed outside the databases. this is surprising, given the challenges encountered in translating the concepts of transitions into search terms for the database searches. in contrast, for the homecare topic, 23% of literature was identified from supplementary searches outside databases, though just 4% of literature was not actually present in the databases.   attempts to increase precision of database searches meant that some citations were missed within individual databases, but our analysis shows that searching across multiple databases mitigated this problem. this was aided by the large amount of overlapping and relevant content found across the databases.   utility of major topic-focused bibliographic databases   choosing which databases to search is based on a number of factors, including likelihood of high yields, unique yields, convenience to search, sensitivity of the search available, functionality of the databases, and combination of databases searched. from this study, the trend points to using medline for currency and precision of searches: embase, hmic, and spp for unique content; psycinfo for mental health topics; bni for broader searching; and cinahl for studies on people's views. ssci yielded more relevant studies than other social sciences databases and located some studies not found in other databases. the keywords plus field in ssci (which is generated from the reference list of each citation) proved useful to search as an alternative to an indexed controlled vocabulary. eric was important for education topics. we observed there are potential issues with applying date limits, and as such, in some databases, it is prudent to check this by comparing the search results with an exclusion search of citations outside of the date limits required.   inability to locate some citations varied according to database, topic, and individual citations in our analysis of the child to adult services and the mental health setting transition topics. searching the following eight databases for the two topics would locate all the studies: bni, cinahl, embase, hmic, medline, psycinfo, spp, and ssci. this same combination could potentially locate all the studies from homecare; however, complete data is not available to check this with the original searches. over 20 databases were searched for each guideline, and our findings suggest a potentially smaller number of databases would yield the same results. this information is particularly useful in undertaking further searches in areas related to these topics and for update searches of the guidelines. given that the most useful databases vary depending on topic, these findings are tentative when applying to other topics.   it is important to be aware of specialist resources relevant to a topic, and this is not fully considered here. for example, for the child to adult services guideline, the nspcc inform child protection database was also searched, and searching trials registries and systematic review databases is often appropriate. however, the aim of this study was to consider general topic-specific databases rather than specialized sources.    context with other research   to locate social care studies, flexibility of approaches is needed in developing search strategies. this study shows this, and so have others (clapton, 2010; forbes & griffiths, 2002; mcnally & alborz, 2004). our findings on important databases to search are consistent with five of the six important databases from golder et al. (2008), which related to respite care for carers, though we did not investigate ageline. our study also confirms there are still difficulties with lack of abstracts.   although it is accepted that multiple database searching is important, our study informs practice on which databases might be more useful to focus on when carrying out literature searches in social care topics, with the caveat that every new topic faces the challenge of articulating a search, finding suitable search terms, and knowing where to find studies. our results also highlight the variability in indexing studies across databases. it further suggests variability of indexing within databases on social care topics, within the broad remit of the guidelines included here. analyzing the citations present within databases by their study design was undertaken to explore the usefulness of these databases to inform particular types of questions. the findings particularly highlight the predominance of citations of studies about people’s views in certain databases; however, the ability to identify these particular citations from the actual searches is not tested.   limitations of this research   there are a number of limitations to the findings presented here. two databases, hmic and spp, are particularly rich in u.k.-relevant content, so their applicability and coverage to social care topics from other countries are unclear. given that the majority of studies on people’s views and experiences were intentionally selected from the u.k. this influences transferability of findings to other countries. it is also worth considering that some studies in the guidelines contributed more than others, and analyzing the influence of these goes beyond the scope of this analysis.   the citations of research evidence used in the analysis were from the search strategies developed for the guidelines and were not compared with other search strategies. however, as part of the guideline development process, the guideline committee and public stakeholders have the opportunity to provide research evidence that may have been missed.   understanding where citations are most likely to be found informs decisions on utilising specific resources. this is particularly useful for topics that are difficult to search for or where a paucity of literature is anticipated. however, only assessing where items present does not provide a complete picture. for example, although eric contained 9% of studies from the child to adult transition topic, the sensitivity and practicality of searching eric with our search strategy is not known. using eric was important to locate one study for the mental health settings transitions topic, but a deliberately precise search was used. the utility of the interdisciplinary database scopus was not fully explored here; it was found, post-hoc, to contain a majority of citations for all three guidelines, including 95% of citations from mental health settings transitions topic, though the sensitivity of a search needed to capture these is unknown. a final limitation is that this study compares three guidelines undertaken at different points in time. differing date limits were used across review questions, with a focus on recent literature where this was considered appropriate. cinahl plus was used in the analysis of where citations were present in which databases, but only cinahl (which has less content) was searched for the child to adult services guideline.   conclusions   developing guidelines and systematic reviews in social care involves identifying social care research that is relevant, but not limited, to integrated health and social care services. broad questions to inform integrated or multi-disciplinary service development are challenging to articulate into concepts that can be translated into terms for searching and require considerable thought and development. for social care practitioners who want to use evidence in their practice and for policy makers in the same field, guidelines informed by evidence reviews and systematic reviews are good ways of grasping a coherent body of literature. therefore, it is important that the challenges of identifying such literature through systematic searching are addressed. this study highlights challenges and reveals trends in identifying social care research from database sources. there is variation in the ability of the search terms to capture the studies from individual databases, even with low-precision searches. however, this is mitigated by searching a combination of databases and searching other resources and websites that are specific to individual topics. we identified a combination of eight databases that were important for finding literature for these topics. multiple database searching also mitigates issues related to the currency of content, topic and study design focus, and consistency of indexing within individual databases.   author contributions and acknowledgements   cs conceived of the study and undertook the main analysis. cs and kl reflected on the findings and drafted the manuscript. an earlier version of this study was presented by cs at the european association of health information and libraries (eahil), seville 8-10 june 2016. thank you to catherine swann, sarah lester, and ginny brunton for their feedback on the early work.   funding   this study was undertaken as part of the work of the nice collaborating centre for social care, which received funding from the national institute for health and care excellence. the views expressed in this publication are those of the authors and not necessarily those of the institute.   kristin liabo's time on this paper was supported by the nihr collaboration for leadership in applied health research and care south west peninsula. the views expressed are those of the authors and not necessarily those of the nhs, the nihr, or the department of health.   references   bayliss, s., & dretzke, j. (2006). health technology assessment in social care: a case study of randomized controlled trial retrieval. international journal of technology assessment in health care, 22(1), 39-46. https://dx.doi.org/10.1017/s0266462306050823   brettle, a., & long, a. (2001). comparison of bibliographic databases for information on rehabilitation of people with severe mental illness. bulletin of the medical library association, 89(4), 353-362. retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/pmc57964/   brettle, a., long, a., grant, m., & greenhalgh, j. (1998). searching for information on outcomes: do you need to be comprehensive? quality in health care, 7(3), 163-167. http://dx.doi.org/10.1136/qshc.7.3.163   brunton, g., stansfield, c., caird, j., thomas, j. (2017) finding relevant studies. in: d. gough, s. oliver, & j. thomas (eds.), an introduction to systematic reviews. 2nd edition. london: sage.   clapton, j. (2010). bibliographic databases for social care searching (knowledge and research: report 34). london: social care institute for excellence.   forbes, a., & griffiths, p. (2002). methodological strategies for the identification and synthesis of ‘evidence’ to support decision-making in relation to complex healthcare systems and practices. nursing inquiry, 9, 141–155. https://dx.doi.org/10.1046/j.1440-1800.2002.00146.x   golder, s., mason, a., & spilsbury, k. (2008). systematic searches for the effectiveness of respite care. journal of the medical library association, 96(2), 147-155. https://dx.doi.org/10.3163/1536-5050.96.2.147   levay, p., raynor, m., & tuvey, d. (2015). the contributions of medline, other bibliographic databases and various search techniques to nice public health guidance. evidence based library and information practice, 10(1), 50-68. http://dx.doi.org/10.18438/b82p55     mcelhinney, h., taylor, b., sinclair, m., & holman, mr. (2016). sensitivity and specificity of electronic databases: the example of searching for evidence on child protection issues related to pregnant women. evidence based midwifery, 14(1), 29-34.   mcfadden, p., taylor, b., campbell, a., & mcquilkin, j. (2012). systematically identifying relevant research: case study on child protection social workers' resilience. research on social work practice, 22(6), 626-636. https://dx.doi.org/10.1177/104973151245320   mcginn, t., taylor, b., mccolgan, m., & mcquilkan, j. (2016). social work literature searching: current issues with databases and online search engines. research on social work practice, 23(6), 266-277. https://dx.doi.org/10.1177/1049731514549423   mcnally, r., & alborz, a. (2004). developing methods for systematic reviewing in health services delivery and organization: an example from a review of access to health care for people with learning disabilities. part 1. identifying the literature. health information and libraries journal, 21, 182–192. https://dx.doi.org/10.1111/j.1471-1842.2004.00512.x   national institute for health and care excellence (nice). (2014, december 2016 update) developing nice guidelines: the manual (process and methods [pmg20]). retrieved from nice website: https://www.nice.org.uk/process/pmg20   national institute for health and care excellence (nice). (2015, september). home care: delivering personal care and practical support to older people living in their own homes (nice guideline [ng21]). retrieved from nice website: https://www.nice.org.uk/guidance/ng21   national institute for health and care excellence (nice). (2016, february). transition from children’s to adults' services for young people using health or  social care services (nice guideline [ng43]). retrieved from nice website: https://www.nice.org.uk/guidance/ng43   national institute for health and care excellence (nice). (2016, august). transition between inpatient mental health settings and community or care home settings (nice guideline [ng53]). retrieved from nice website: https://www.nice.org.uk/guidance/indevelopment/gid-scwave0711   preston, l., carroll, c., gardois, p., paisley, s., & kaltenthaler, e. (2015). improving search efficiency for systematic reviews of diagnostic test accuracy: an exploratory study to assess the viability of limiting to medline, embase and reference checking. systematic reviews,;4:82. https://dx.doi.org/10.1186/s13643-015-0074-7   stevenson, m., taylor, b. j., & knox, j. (2016). risk in dementia care: searching for the evidence. health, risk & society, 18(1-2), 4-20. http://dx.doi.org/10.1080/13698575.2015.1119256   taylor, b., dempster, m., & donnelly, m. (2003). hidden gems: systematically searching electronic databases for research publications for social work and social care. british journal of social work, 33(4), 423-439. https://dx.doi.org/10.1093/bjsw/33.4.423   taylor, b., wiley, e., dempster, m., & donnelly, m. (2007). systematically retrieving research: a case study evaluating seven databases. research on social work practice, 17(6), 697-706. https://dx.doi.org/10.1177/1049731507304402   thomas j, brunton j, graziosi s (2010) eppi-reviewer 4: software for research synthesis. eppi-centre software. london: social science research unit, ucl institute of education   microsoft word class_booth.doc evidence based library and information practice 2007, 2:4  92 evidence based library and information practice       classic    nursing professionals use and value information but favour work‐based sources and  colleagues in preference to libraries      a review of:  urquhart, c., and r. davies. ”evince: the value of information in developing nursing  knowledge and competence.” health libraries review 14.2 (1997): 61‐72.       reviewed by:   andrew booth  reader in evidence based information practice, university of sheffield   sheffield, united kingdom   e‐mail: a.booth@sheffield.ac.uk         received: 26 june 2007      accepted: 21 october 2007      © 2007 booth. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to examine the impact of  information on the clinical knowledge and  practice of nurses, midwives and health  visitors.    design – two surveys: a one‐page critical  incident questionnaire survey sent weekly  over four weeks, and a questionnaire  attached to information requests and  searches, followed up by interviews.    setting – uk health information providers  serving nurses (national information  providers, national health service trust  libraries, higher education funded library  and information services, and a health  promotion library).  subjects – a random sample of 210 nurses,  midwives and health visitors were targeted  in the critical incident survey, and 776 of  those requesting information or searches at  participating library and information centres  received questionnaires for the second  survey.    methods – opinion leaders were consulted  to inform a pilot study. a critical incident  type questionnaire survey was then  administered to a random sample of 210  nurses, midwives and health visitors. the  same one‐page questionnaire was sent  weekly (for four weeks) to 10% of a  randomly selected sample of staff at each  site. staff were asked to identify one  occasion during that week when they  needed information, the purpose of the  mailto:booth@sheffield.ac.uk http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:4  93 information needed, the sources chosen to  answer the query and how successful the  quest was. the impact of the information  provided by the library and information  services on present and future professional  practice was examined through a  complementary survey. responses were  coded using three categories of competence:  assessment, monitoring of care and  evaluation of care. follow‐up interviews  then explored the nature of the incident  described or the quality of information  provided.      main results – the response rate for the  critical incident survey was 52% (434 out of  840 completed questionnaires returned)  with 78% (163/210) of participants replying  at least once. the total response rate for the  second survey was 40% (311/776). ninety  percent of respondents stated that the  information they obtained from the library  or information service added to their  knowledge, and 86% had been able to use  some information immediately. sixty‐one  percent reported that information had  refreshed their memory, and 75% agreed  that information substantiated what they  had known or suspected. seventy‐six  percent of respondents agreed that they  needed to obtain more information on the  topic, while 23% had expected to find  something else. eighty‐eight percent of  respondents indicated that they would share  the information with colleagues. ninety‐six  percent of respondents agreed that the  information obtained would contribute to  future practice. seventy percent felt that the  information would or did help in evaluation  of practice outcomes, 68% for improved  quality of life for patient and/or family and  61% in interpersonal relations with  clients/patients. other majority responses  were for audit or standards of care (57%),  monitoring of care (56%) and legal or  ethical issues (51%). base or ward sources  were used in 72% of patient care related  incidents and colleagues in 56% of these  incidents. of 148 incidents involving use of  a library, the primary purposes were  personal updating (62%), coursework (54%),  patient care – specific drug or therapy (44%),  and teaching staff, students or colleagues  (39%).    conclusion – the findings demonstrate the  value of information to nursing  professionals while acknowledging that the  library is not necessarily a principal source  of such information. ward‐based resources  and information from colleagues continue to  play a dominant part in information use.  rather than reducing uncertainty, the value  of information may lie in encouraging  uncertainty and reflective practice.  information professionals must seek a  greater understanding of situations and  methods of presentation by which they  might encourage reflective practice.    commentary    this study extends a previous methodology  to examine the information needs and uses  of nurses (urquhart and hepworth). rather  than simply replicating the previous value  study, this research was adapted to the  specific needs of nurses, using nursing  competencies to classify purposes for  information use rather than using stages of  the diagnostic and therapeutic process.    isi web of knowledge citations from the  health information and nursing literatures  attest that this is a landmark study.  furthermore it is located within a significant  body of literature on the impact of health  information, a literature that, in turn, traces  its pedigree back to the 1992 rochester  study (marshall). it is also illustrative of  urquhart’s associated portfolio exploring  methodological issues. interestingly, these  methodological and topical issues have led  to recent citations in at least two information  studies from the developing world (ghana  and uganda).  evidence based library and information practice 2007, 2:4  94 this research has several methodological  weaknesses. first, the critical incident  technique (urquhart et. al.), where  participants are asked to identify an incident  of note, is subject to both recall bias  (imperfect recollection) and reporting bias  (selection of unrepresentative incidents). it  is also extremely vulnerable to response bias  (participants with something positive to  report are more likely to respond). the  research team does, however, ask about all  information incidents, not simply those  related to use of a library, thus aiming at a  more realistic picture of information  behaviours. the authors are apologetic  about their response rates although, as any  researcher from the health domain will  recognize, 40% is closer to celebration than  commiseration! finally, by asking not  simply whether the participants used the  information supplied but also whether they  were likely to use it in the future, the  researchers open up the possibility of  inflating the importance of specific  information. indeed, the researchers  occasionally place a favorable interpretation  on data that could be subject to alternative  explanations. for example, respondents who  have to come back for further information  are not seen as being unsatisfied, but rather  the satisfied customer may be the customer  who returns for more information, not the  customer who makes only one trip.    this ten‐year old study exhibits the  occasional imperfection, but where is its  rightful place within the evidence base for  information behaviour? certainly its scale,  as funded by the british library, makes it  one of the most significant studies of the  decade, if not of the twenty‐five years of its  host journal. its concern with impact reflects  the prevailing zeitgeist of effectiveness and  efficiency that characterised the mid‐1990s.  for the reflective evidence based  practitioner, however, perhaps its greatest  contribution comes in laying down a marker  against which subsequent studies might  strive to compete – and to exceed!  retrospective designs have known  limitations with regard to selective reporting  and their susceptibility to bias. while critical  incident studies continue to command a  place in the investigation of user  information behaviour (radford), in this era  of evidence based practice we can aspire to  the emergence of prospective longitudinal  studies that comprehensively track the fate  of a representative sample of information  requests. such requests should be evaluated  both in terms of their immediate impact on  clinician knowledge, attitudes and  behaviours and, more significantly, in their  impact on the care of specific individual  patients.      works cited    marshall, j.g. ʺthe impact of the hospital  library on clinical decision making:  the rochester study.ʺ bulletin of the  medical library association 80. 2  (1992): 169‐77.    radford, m. l. “the critical incident  technique and the qualitative  evaluation of the connecting libraries  and schools project.” library trends  55.1 (2006): 46–64.    urquhart, c., et al. “critical incident  technique and explicitation  interviewing in studies of information  behavior.” library & information  science research 25.1 (2003): 63‐88.    urquhart, c., and j. hepworth. “the value  of information supplied to clinicians  by health libraries: devising an  outcomes‐based assessment of the  contribution of libraries to clinical  decision‐making.” health libraries  review 12.3 (1995): 201‐13.    ebl 101   research methods: altmetrics   virginia wilson librarian, murray library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   received: 3 feb. 2013     accepted: 8 feb. 2013      2013 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     it’s no secret that scholarly communication is changing. the internet, the open access movement, the proliferation of institutional repositories, and the use of social networking tools, as well as the questioning of peer review and impact factors, to name just a few things, have altered the scholarly publication landscape. massive amounts of research content, both full content and citations are available on the internet: the traditional research paper, blogs, academic repositories, online citation managers, and even tweets and facebook posts. if the accepted ways of publishing are changing, then it makes sense that different ways of measurement should be explored in order to get the complete picture of the impact of the work. this is where altmetrics comes in and it’s exciting stuff!   the altmetrics manifesto states that scholars are increasingly moving their work to the internet. “these new forms [citation managers, blogs, other social sharing sites] reflect and transmit scholarly impact: that dog-eared (but uncited) article that used to live on a shelf now lives in mendeley, citeulike, or zotero–where we can see and count it. that hallway conversation about a recent finding has moved to blogs and social networks – now, we can listen in. the local genomics dataset has moved to an online repository – now, we can track it. this diverse group of activities forms a composite trace of impact far richer than any available before. we call the elements of this trace altmetrics.” (http://altmetrics.org/manifesto/)   based on these alternative metrics, altmetrics is “the creation and study of new metrics based on the social web for analyzing, and informing scholarship” (altmetrics.org). what paths do our reactions to a particular article take in the social web? plos refers to this new landscape as the “scholarly ecosystem” (http://article-level-metrics.plos.org/alt-metrics/). altmetrics isn’t just about traditional citation-based indicators. nor is it just about hits; these can be inflated by robot-crawlers and other zealous clickers. so work has been done on finessing use stats and hit count work in order to get a more meaningful measure of online usage in all its variety. as an emerging field of scholarship and development, altmetrics is moving ahead in leaps and bounds. it is becoming common to see journal publishers and repositories implementing tools that will let authors see the impact of their scholarly publications.   and of course there is a role for librarians and information professionals to play in this new field of measurement. while new, the altmetrics field is exploding with new ideas and products. a librarian’s expertise in scholarly communication and in finding the resources and data needed for researching scholars can be invaluable to the institution. there is much to learn in this emerging field. i’ve included a list of resources that can take you further into the realm of altmetrics, and as always, i appreciate comments on this column. you can log in as a reader to the eblip journal and interact from there.   the altmetrics manifesto   priem, j., taraborelli, d., groth, p. & neylon, c. (2010), altmetrics: a manifesto, (v.1.0), 26 october 2010. http://altmetrics.org/manifesto (the comments at the end of the manifesto are an interesting conversation as the concept of altmetrics develops.)   articles   taraborelli, d. 2008. soft peer review: social software and distributed scientific evaluation. retrieved from http://eprints.ucl.ac.uk/8279/   neylon c. & wu s (2009) article-level metrics and the evolution of scientific impact. plos biol 7(11): e1000242. doi:10.1371/journal.pbio.1000242 retrieved from http://www.plosbiology.org/article/info:doi/10.1371/journal.pbio.1000242 priem, j. & hemminger, b.m. (2010). scientometrics 2.0: toward new metrics of scholarly impact on the social web. first monday, 15(7) http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/viewarticle/2874/2570 roemer, r.c. & borchardt, r. (2012). from bibliometrics to altmetrics: a changing scholarly landscape. college & research libraries news vol. 73 no. 10 596-600 retrieved from http://crln.acrl.org/content/73/10/596.full   altmetrics scholarship   plos altmeterics collections http://www.ploscollections.org/article/browseissue.action?issue=info:doi/10.1371/issue.pcol.v02.i19 the altmetrics group on mendeley http://www.mendeley.com/groups/586171/altmetrics/ here you can find more scholarly articles on altmetrics added by mendeley users.   products[i]:   altmetric it http://www.altmetric.com/ they offer open data for individuals, including a free bookmarklet to be used on recent scholarly articles to see how much attention they have received online. there is also an api, free for non-commercial use, used to mash up altmetrics data with other data. the following link is for an interesting blog post from altmetric.org on the free services and apps they’ve developed for libraries and institutional repositories: http://altmetric.com/blog/altmetrics-in-academic-libraries-and-institutional-repositories/ impactstory (formerly total-impact) http://impactstory.org/ an altmetric aggregator. in terms of pricing, this information is from the website: we charge to collect metrics; the data is free and open once it’s been collected. you let us know to start collecting metrics on something when you register it with us. the first 1000 items you register are free. registering more than 1000 items will have an annual fee to provide sustainability for our nonprofit service (waivers available in some cases); the fee will depend on how many items you'd like to register. plum analytics http://www.plumanalytics.com/index.html this company collects impact metrics in five major categories: usage, captures, mentions, social media, and citations. one of the two founders of this service is a librarian. they gather metrics about what they refer to as “artifacts” and these include: articles, book chapters, books, clinical trials, datasets, figures, grants, patents, presentations, source code, and videos. you can see the metrics they include and where they find them here: http://www.plumanalytics.com/metrics.html altmetric for scopus http://altmetric.uservoice.com/knowledgebase/articles/83246-altmetric-for-scopus denise koufogiannakis explains on her blog that “the altmetric service will capture information from social media sites such as twitter and facebook, mainstream media, and reference managers such as mendeley,to illustrate how scholarly articles are being used beyond academia” (2012). plos article-level-metrics (alms) http://article-level-metrics.plos.org/ metrics are tracked for every article published by plos and the full alm data set, which is updated monthly as a .csv file, is always freely available online for all plos-published articles. sciencecard http://sciencecard.org/ sciencecard is a web-service that collects article-level metrics using the content from twitter, mendeley, pubmed central, citeulike, wikipedia and crossref. other services continue to be added.     references   altmetrics.org. (2010). altmetrics. retrieved from http://altmetrics.org/about/   koufogiannakis, d. (2012). altmetrics for scopus. collections information. retrieved from http://collectionsinfo.blogspot.ca/2012/09/altmetric-for-scopus.html?spref=tw   [i] the author is not affiliated with and does not endorse any organization represented here. they are included for information purposes only and are to be implemented at the user’s own discretion. evidence summary   flipped library instruction does not lead to learning gains for first-year english students   a review of: rivera, e. (2017). flipping the classroom in freshman english library instruction: a comparison study of a flipped class versus a traditional lecture method. new review of academic librarianship, 23(1), 18-27. http://dx.doi.org/10.1080/13614533.2016.1244770   reviewed by: kimberly miller learning technologies librarian albert s. cook library towson university towson, maryland, united states of america email: kimberlymiller@towson.edu   received: 12 may 2017  accepted: 11 july 2017      2017 miller. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to determine whether a flipped classroom approach to freshman english information literacy instruction improves student learning outcomes.   design – quasi-experimental.   setting – private suburban university with 7,000 graduate and undergraduate students.   subjects – first-year english students.   methods – students in six sections of first-year “english 2” received library instruction; three sections received flipped library instruction and three sections received traditional library instruction. students in the flipped classroom sections were assigned two videos to watch before class, as an introduction to searching the library’s catalog and key academic databases. these students were also expected to complete pre-class exercises that allowed them to practice what they learned through the videos. the face-to-face classes involved a review of the flipped materials alongside additional activities.   works cited pages from the students’ final papers were collected from all six sections, 31 from the flipped sections and 34 from the non-flipped sections. a rubric was used to rate the works cited pages. the rubric was based on the association of college and research libraries’ information literacy competency standards for higher education (acrl, 2000), standard two, outcome 3a, and included three criteria: “authority,” “timeliness,” and “variety.” each criterion was rated at one of three levels: “exemplary,” “competent,” or “developing.”   main results – works cited pages from the students who received non-flipped instruction were more likely to score “exemplary” for at least one of the three criteria when compared to works cited pages from the flipped instruction students (68.6% vs. 52.7%). differences were found in the scores for “timeliness” (88.2% non-flipped scored “exemplary” compared to 58% flipped), and “variety” (55.9% non-flipped scored “exemplary” vs. 35.5% flipped). this pattern was not found for the “authority” category, in which 61.8% of non-flipped works cited pages scored “exemplary” vs. 64.5% of flipped works cited pages.   conclusion – the results suggest that the flipped library instruction approach did not improve student learning outcomes. the study’s findings are limited by the small sample size, the unknown impact of the variability of research assignments between sections, and the lack of control over whether students in the flipped sections completed the pre-class assignments. the author also notes that future research should examine how well the content of flipped library instruction mirrors that of non-flipped instruction sessions. the study concludes that the flipped classroom model needs further research to understand whether it is a strong fit for one-shot library instruction.   commentary   information literacy instruction remains an essential element of academic libraries’ missions; however, frustration with the “one-shot” model for course-embedded instruction persists (julien, gross, & latham, 2017, p. 12). as such, librarians continue to explore instructional approaches that promise to improve student learning outcomes, enabling them to make the most of limited contact time with learners. with its typical blend of pre-class preparation and in-class application, the flipped classroom is one model that offers librarians expanded opportunities to connect with learners and extend information literacy instruction.   evaluating the current study with koufogiannakis, booth, and brettle’s (2006) reliant instrument finds that the study provides a clear objective, the details of the flipped classroom intervention, and the rubric used to score the students’ works cited pages. the study’s data support the author’s conclusion that the flipped intervention alone did not improve student learning outcomes, as measured by the works cited rubric. many academic libraries are transitioning towards using the framework for information literacy for higher education (acrl, 2015), so librarians who wish to replicate this study may find value in translating the article’s standards-based (acrl, 2000) rubric criteria to the newer information literacy framework.   however, the article does not provide critical student population information necessary to fully understand the study’s design or evidence. no overall sample size (i.e., how many students received instruction) is provided for either the flipped or non-flipped instruction sessions, and there is no demographic information to demonstrate whether students in the flipped and non-flipped groups were similar along study variables. it is also unclear whether the six sections were randomly assigned to the flipped or traditional sessions. while six total sections received instruction over two semesters (p. 23), there is no information about which instruction methods (flipped vs. non-flipped) were used each semester. this may introduce potential confounds, such as maturation, if students’ experience levels or other characteristics differed from one semester to the next. without such information, it is difficult to evaluate whether study results are due solely to the educational intervention.    considering the educational context, there is no description of the lesson design for the traditional classes or how it compared to the content of the flipped lesson.  controls for measuring whether students in the flipped classes completed their pre-class work would be required to understand what element of the flipped approach should be improved. for example, incentives for student participation vs. the actual structure and content of the flipped lesson.    considering how the flipped tutorial content, with its focus on search and retrieval skills, aligns with rubric criteria (focused primarily on evaluation concepts) may provide an additional facet to interpreting study results.   this study provides evidence that, like any education intervention, the flipped classroom model by itself may not improve student learning outcomes. instead, librarians must carefully consider how well different instructional models could work in relation to local scenarios, classes, and content. the current study’s discussion of considerations for flipped lesson design, lesson content, and outcomes measurement provides librarians with a starting place to continue expanding research into flipped classroom effectiveness.   references   association of college & research libraries. (2000). information literacy competency standards for higher education. http://www.ala.org/acrl/standards/informationliteracycompetency   association of college & research libraries. (2015). framework for information literacy for higher education. http://www.ala.org/acrl/standards/ilframework   julien, h., gross, m., & latham, d. (2017). survey of information literacy instructional practices in u.s. academic libraries. college & research libraries. advance online publication. http://crl.acrl.org/   koufogiannakis, d., booth, a., & brettle, a. (2006). reliant: reader’s guide to the literature on interventions addressing the need for education and training. library & information research, 30(94), 44-51. http://www.lirgjournal.org.uk   evidence based library and information practice editorial responsibilities      2017. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     editor-in-chief: lorie kloda   associate editor (evidence summaries): carol perryman   associate editor (research articles): rebekah (becky) willson, lisl zach   associate editor (classics & reviews): christina wissinger   associate editor (commentaries, using evidence in practice): lorie kloda   associate editor (feature): lorie kloda   communications officer (news): tatiana bryant   production editor: michelle dunaway   editorial intern: rachel hinrichs   editorial advisors: lindsay alcock, alison brettle, jonathan eldredge, denise koufogiannakis   copyeditors: richard hayman (lead copyeditor), georgianne bordner, julie evener, heather healy, julie james, jane morgan-daniel, stacey penney, elizabeth stregger,  nikki tummon, elaina vitale   indexing support: kate shore   microsoft word es_perryman.doc evidence based library and information practice 2007, 2:3  101 evidence based library and information practice       evidence summary    tracking theory building and use trends in selected lis journals: more research is  needed    a review of:  kim, sung‐jin, and dong y. jeong. “an analysis of the development and use of theory in  library and information science research articles.” library & information science  research 28.4 (sept. 2006): 548‐62.    reviewed by:   carol perryman  trln doctoral fellow, school of information & library science, university of north carolina at  chapel hill  chapel hill, north carolina, united states of america  e‐mail: cp1757@gmail.com        received: 29 may 2007    accepted: 08 july 2007      © 2007 perryman. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective ‐ the authors measure theory  incidents occurring in four lis journals  between 1984‐2003 in order to examine their  number and quality and to analyze them by  topic. a third objective, only identified later  in the text of the study, was to compare  theory development and use between  korean and international journals. research  questions asked include whether lis has its  own theoretical base as a discipline, and  what characteristics the theoretical  framework has.    design – bibliometric study.    setting – journal issues selected from four     lis journals for the time span from 1984 ‐  2003.    subjects – two international journals,  journal of the american society for information  science and technology (jasist) and library  and information science research (lisr) were  selected based on their high ranking in the  social sciences citation index (ssci) impact  factors. two korean journals, journal of the  korean society for information management  (jksim) and journal of the korean society for  library and information science (jkslis) were  selected.     methods ‐ after having determined a  definition of theory, and identifying  different levels of theory, the authors set up   mailto:cp1757@gmail.com http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:3  102 rules for the identification of theory  incidents, which are defined as “events in  which the author contributed to the  development or the use of theory in his/her  own paper” (550).    content analysis of 1661 research articles  was performed to measure theory incidents  according to working definitions. interrater  reliability was ensured by conducting  independent coding for “subfield  classification, identification of theory  incidents, and quality measurement” (555),  using a sample of 199 articles (random  selection not specified), achieving 94‐97%  interrater reliability. incidents, once  identified, were evaluated for quality using  dubin’s “efficiency of law” criteria,  involving measures of relatedness,  directionality, co‐variation, rate of change,  and “profundity,” defined as the depth to  which theory is incorporated into the  research study.    main results ‐ 21.79% (n=362) of the articles  contained theory incidents that were  analyzed and evaluated. trend  measurement indicated an overall increase,  although a slight decrease was shown in the  year range 1993‐2003.  international journals  accounted for 61.33% of theory incidents,  compared to 38.67% for the korean journals.   t‐testing showed that differences in means  between korean and international journals  were not statistically significant.    topical theory areas were ranked by  frequency. the top five areas were shown to  be nearly identical between korean and  international journals. anova was  performed with significant results in the  difference between efficiency ratings.     conclusion – the authors find that the  overall proportion of theory incidents  including both theory development and use  increased through the 20‐year time span  examined, and that lis has established its  own theoretical framework based upon the  frequency of incidents.     commentary    several significant flaws affect the  applicability of an otherwise interesting and  valuable study. the selection of journals for  inclusion in this study is questionable on  four different points. first, the authors chose  jasist and lisr based upon their high  impact factor in ssci, for lis journals. the  use of these impact factors to select the two  international journals presents problems  that are not discussed by the authors of this  study, although it is possible that journal  space limitations precluded a discussion of  that nature. ssci is known to index only 30  lis journals, an issue that the authors make  no attempt to address.     second, the ranking of “top” journals by  ssci is not accomplished by any measure of  research quality, but by citation frequency.  according to gorman and calvert, “the fact  that paper x is cited y times is not an  indication of its quality – it is available, it is  in a journal held by many libraries, the  author (or publisher or editor) is particularly  good at self promotion” (101). even if we  accepted the rankings, there is still no  indication that the selected journals are  representative of all international lis  journals. in fact, it is not proved whether the  four journals are representative of their  respective categories (international and  korean), yet kim and jeong claim  generalizability from their results.    third, jksim and jkslis were selected  based upon their status as “top‐ranked in  korean lis society,” a statement which is  not further clarified (553).     fourth, the decision to sample issues of  jasist rather than to include the entire  volume of publication for the year range  chosen because of the comparatively large  evidence based library and information practice 2007, 2:3  103 output of that journal renders later statistical  analysis problematic, and furthermore, may  not be representative of the journal’s  publications. there is no apparent effort to  consider whether that content, for example,  may have included special issues or  supplements. the decision is further  questionable in light of the fact that the  number of issues does not necessarily  correlate with volume of research output,  but the authors appear not to have  considered this when making their selection  decision.     the authors are careful in the literature  review section to explain prior studies of  theory use in disparate literatures, yet the  same care is not carried over to the  methodology section. the determination to  identify theory incidents based upon  whether the concept “has ever been named  as a theory in any article in the sample” (553)  might be interpreted to mean that concepts  broadly accepted as theories, but merely not  mentioned in the issues examined for this  study, were rejected as theory incidents –  possibly resulting in underreporting of data  throughout the study.    the authors excel in clarifying concepts  related to theory building and use, and in  their literature review section in describing  the context for this study. unfortunately,  there are no directly comparable studies  because earlier efforts measure different  elements or do not measure both  construction and use of theory, as these  authors do. to this reviewer, it appears that  the authors overreach what can reasonably  be achieved within the constraints of a brief  research paper.     an additional problem that should be noted  is the poor quality of editing for this paper:  key statements are muddied by unclear  phrasing. for example, the statement “there  would be room for the intervention of other  factors such as the productivity increase in  article publications, though,” is  grammatically awkward and sheds little  light on a discussion about factors that may  have influenced the apparent increase of  theory incidents in lis publication (560).    due to insufficient data, some of the results  are not reproducible. providing data on  theory incidents by individual journal title  might have enlarged upon the usefulness of  the study. in addition, including articles  from the entire jasist run would have  given a more accurate accounting of theory  incidents. as it is, the figures are not  meaningful, and since the statistics are not  broken out by journal title the reader cannot  determine if any one of the journals diverges  in any significant way from the others. in  the discussion section the authors claim that  a total of 41.4% of the 1661 articles engage in  either theory building or use, yet no table or  explanation of how this figure is derived is  provided. adding their own percentages for  theory incidents (25.95% for theory use, and  21.79% for theory building) we arrive  instead at 47.74%.    additional data would also assist with  comprehension of the anova results.  the  authors provide an f statistic and overall p‐ value demonstrating significant results in  the difference between efficiency ratings,  but do not give individual p‐values to verify  the results. it is far from clear whether the  differences being tested are those between  international and korean means, or between  total means for each topical category.  an  additional column in table 4 (557) would  have greatly aided the reader.    this article is most useful in its well‐ constructed general methodology and  categorization of theory topics, as well as in  its careful explanation of theory research,  and would assist researchers in constructing  a more applicable study. the greatest  weaknesses are in the selection of journals  and in the explanation of the results,  evidence based library and information practice 2007, 2:3  104 damaging comprehensibility and  applicability for readers. the authors would  have strengthened their argument by  acknowledging the flaws associated with  the journal selection process. because of  these flaws, applicability of their findings is  severely limited.       work cited    gorman, g.e. and philip j. calvert. “journal  quality in the asian region: results of  a pilot study for the ifla round table  of library and information science  journals.” serials librarian 41.1 (2001):  99‐112.  microsoft word es 4149 evidence based library and information practice 2008, 3:4 49 evidence based library and information practice evidence summary information literacy strategy development: study prescribes strategic management framework for academic institutions a review of: corrall, sheila. "information literacy strategy development in higher education: an exploratory study." international journal of information management 28 (2008): 26-37. reviewed by: shandra protzko information specialist, tucker medical library, national jewish health denver, colorado, united states e-mail: protzkos@njc.org received: 25 september 2008 accepted: 25 october 2008 © 2008 protzko. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – to examine the development of information literacy (il) strategies in higher education by assessing content and presentation of il strategy documentation, and to explore the application of corporate strategy concepts and techniques to il strategy. design – comparative, multi-case study. qualitative analysis. setting – u.k. universities. subjects – twelve information literacy strategy documents from ten institutions. methods – google was searched for il strategy documents (restricted to the ac.uk domain), the lisinfoliteracy discussion list was queried, and the web sites of all u.k. universities were searched for a total sample of 12 documents at 10 institutions. results of the data capture were discussed in the context of the literature on strategic management. main results – corporate strategy tools and techniques are extensive in the literature, trending toward an emphasis on holistic thinking and marketing concepts. many themes identified in the documents were consistent with the literature. while the format and style varied, all documents emphasized the integration of il into subject evidence based library and information practice 2008, 3:4 50 curricula. all stressed the need to build collaborative partnerships between library/information staff and academic staff. significantly, many strategies aimed to reach the broader institution, although poor articulation undermined this ambitious goal. in three, il intervention was intended for the whole university community. however, the target audience often was not well defined. seven of the il strategies identified additional partnerships to effect change at the policy level. another key theme was the adoption of recognized il standards; seven proposed the sconul (1999) model. all strategies recognized the importance of learning outcomes; six stated them explicitly. prominent was the integration of e-learning resources, namely online tutorials. many strategies recognized the need for marketing and advocacy activities. half considered professional or staff development issues, as supported in the literature. all strategies explained in detail the context of their il proposals, citing external challenges (growth of digital information, employer demand), external evidence (official reports, benchmarking statements, studies), and internal evidence (stakeholder concerns, institutional strategies) to support the need for il. the documents specified a range of teaching modes from informal reference desk encounters to strategic positioning in relation to the broader community. half defined or described il. seven documents were labelled strategies, but many did not comply with content elements defined in the literature. other features of the literature poorly represented in the documents included: the need for well articulated objectives, mission and vision statements, attendance to broader strategic issues, stakeholder analysis, and the prioritizing of il activities with portfolio development. only two had action plans. seven documents were in the public domain. conclusion – information literacy is recognized as an essential competence for participation in higher education, the workplace and society, and information professionals have long promoted il. in response to the changing information environment they have sought to formalize policies and strategies to embed il institutionally, working collaboratively with key stakeholders in the process. in this study of strategic documentation from 10 u.k. universities, il objectives were consistent although the format and style of documentation varied. the author concludes that one or more strategic management models or tools available could improve il strategy development, consistency, and coherency. most importantly, an overarching strategic management framework should be used to resolve ambiguity and inconsistency, improve articulation, and maximize the effectiveness of strategy documents, thus avoiding weaknesses identified in the study. as noted by the author, the library literature has progressed to include strategic management concepts evidenced, in part, by the number of libraries using kaplan and norton’s scorecard system. but improvements can be made: conforming to strategic planning norms could strengthen il strategy. elements of models from the public or private sectors might be tailored to meet the specific needs of il strategies. further research could identify suitable strategy models for il development. the process of implementing il strategy should also be considered in future research. the author notes it would be interesting to explore the relationship between il strategies and other organizational strategies and to compare il strategies in other sectors. evidence based library and information practice 2008, 3:4 51 commentary the aims of the study were clear: advance an understanding of il strategy development by exploring the characteristics of information literacy strategies in u.k. universities from a strategic management perspective, and discuss possible contributions of corporate strategy concepts to il practice. without knowing how many universities there are in the u.k. it is unclear whether or not the sample size was large enough to be sufficiently representative. it is also not obvious how the themes were derived from the literature. the method of data collection was clearly described and the results conveyed succinctly with the use of tables. the study is reliable in that it could be reproduced. however, it was confusing that 12 documents were examined while the results were enumerated as 10 institutions. moreover it seems prudent to consider the question of bias when the reader takes into account the author’s leadership roles in academic institutions. what, if any, was the relationship between the researcher and the participants? did the author influence il documentation included in the study? potential conflict of interest is not addressed. the importance of il to the individual and to society cannot be overstated. the author rightly points out that, “[f]urther research is needed to review existing strategy models, tools and techniques and assess their suitability for il strategy development” (corrall 35). a discussion as to why a strategic management perspective is an appropriate and useful model from which il practitioners could work would have strengthened the suggestions proposed in the study. as the author suggests, it would be valuable to look at the relationships between il strategy and other organizational strategies. the results and the author’s suggestions will be of interest to those formulating and implementing il strategies. the study provides insight into current practice and highlights key themes in the literature that are either covered or absent in the sample of il strategy documents. it serves to highlight a need for either better strategic management models or greater awareness and adoption of existing tools and techniques. il practitioners would do well to use this study as a starting point for discussion. works cited corrall, sheila. "information literacy strategy development in higher education: an exploratory study." international journal of information management 28 (2008): 26-37. kaplan, r. s., & d. p. norton “the balanced scorecard: measures that drive performance.” harvard business review 70.1 (1992): 71-9. sconul. information skills in higher education: briefing paper. london: society of college, national, and university libraries (sconul). october 1999. 3 nov. 2008 <http://www.sconul.ac.uk/groups/inf ormation_literacy/papers/seven_pilla rs2.pdf>. research article   the library assessment capability maturity model: a means of optimizing how libraries measure effectiveness   simon hart policy, planning and evaluation librarian university of otago library dunedin, new zealand email: simon.hart@otago.ac.nz   howard amos university librarian university of otago library dunedin, new zealand email: howard.amos@otago.ac.nz   received: 19 july 2018                                                                   accepted: 11 oct. 2018      2018 hart and amos. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29471     abstract   objective – this paper presents a library assessment capability maturity model (lacmm) that can assist library managers to improve assessment. the process of developing the lacmm is detailed to provide an evidence trail to foster confidence in its utility and value.   methods – the lacmm was developed during a series of library benchmarking activities across an international network of universities. the utility and value of the lacmm was tested by the benchmarking libraries and other practitioners; feedback from this testing was applied to improve it. guidance was taken from a procedures model for developing maturity models that draws on design science research methodology where an iterative and reflective approach is taken.   results – the activities decision making junctures and the lacmm as an artifact make up the results of this research. the lacmm has five levels. each level represents a measure of the effectiveness of any assessment process or program, from ad-hoc processes to mature and continuously improving processes. at each level there are criteria and characteristics that need to be fulfilled in order to reach a particular maturity level. corresponding to each level of maturity, four stages of the assessment cycle were identified as further elements of the lacmm template. these included (1) objectives, (2) methods and data collection, (3) analysis and interpretation, and (4) use of results. several attempts were needed to determine the criteria for each maturity level corresponding to the stages of the assessment cycle. three versions of the lacmm were developed to introduce managers to using it. each version corresponded to a different kind of assessment activity: data, discussion, and comparison. a generic version was developed for those who have become more familiar with using it. through a process of review, capability maturity levels can be identified for each stage in the assessment cycle; so too can plans to improve processes toward continuous improvement.   conclusion – the lacmm will add to the plethora of resources already available. however, it is hoped that the simplicity of the tool as a means of assessing assessment and identifying an improvement path will be its strength. it can act as a quick aide-mémoire or form the basis of a comprehensive self-review or an inter-institutional benchmarking project. it is expected that the tool will be adapted and improved upon as library managers apply it.     introduction   the improvement of processes has become increasingly important in libraries, especially within the higher education context. this has been in response to wider economic pressures that have seen limited budgets and the rise of accountability (lilburn, 2017). libraries have prioritized the need to demonstrate a return on investment, show that users’ needs are being met, remain relevant, offer (added) value, and align with wider strategic imperatives (matthews, 2015; oakleaf, 2010; sputore & fitzgibbons, 2017; tenopir, mays & kaufman, 2010; urquhart & tbaishat, 2016). a drive for efficiency and effectiveness has culminated in calls to foster cultures of quality, assessment, and evidence based decision-making (atikinson, 2017; crumley & koufogiannakis, 2002; lakos & phipps, 2004). business as usual is no longer enough. doing more with less while continuing to improve is the new norm. applying assessment processes and improving upon them has become imperative for library mangers (hiller, kyrillidou, & oakleaf, 2014). the challenge is how can assessment be conducted and improved efficiently and effectively. this paper documents the development of a tool—the library assessment capability maturity model (lacmm)—that can meet this need.   literature review   the issue of library assessment is well documented (heath, 2011; hufford, 2013; town & stein, 2015). signposts, “how to” manuals, and examples of practice are readily available (oakleaf, 2010; wright & white, 2007). a range of comprehensive books have been published (appleton, 2017; brophy, 2006; heron, dugan, & nitecki, 2011; matthews, 2015).   the tools to measure effectiveness are continually evolving—from the questionnaire employed by the advisory board on college libraries across carnegie libraries in the 1930s (randel, 1932) to orr’s framework for quantitative measure for assessing the goodness of library services (orr, 1973) to more contemporary tools like libqual+® surveys (association of research libraries, 2012) and web based assessment tools offered by counting opinions (n.d.). significant investment has been made to strengthen librarians’ assessment practices, for example through the acrl program assessment in action: academic libraries and student success (hinchliffe & malenfant, 2013). work has been undertaken to identify factors important to effective library assessment (hiller, kyrillidou, & self, 2008) as well as to identify factors influencing an assessment culture (farkas, hinchliffe, & houk, 2015). in discussing the history of library assessment, heath (2011) noted that “recent years have seen a collaborative culture of assessment reach its full maturity” (p. 14).   despite the rich literature that exists on assessment practices, the concept of maturity in assessment has only received limited attention in libraries. cosby (1979) popularized the concept of maturity of business processes by considering them in stages building on each other, offering an effective and efficient means for the analysis and measurement of the extent to which a process is defined, managed, assessed, and controlled. the application of capability maturity within a framework emerged out of the software engineering industry where paulk, curtis, chrissis, and weber (1993) conceived a capability maturity model (cmm). subsequently, cmms have been applied in a range of other industries and organizations to assess the level of capability and maturity of critical processes, such as project management capability (crawford, 2006), people capability (curtis, 2009), and contract management process capability (rendon, 2009).   a cmm has five levels of capability maturity, as illustrated in figure 1 (adapted from paulk, curtis, chrissis, & weber, 1993). each level represents a measure of the effectiveness of any specific process or program, from ad-hoc immature processes to disciplined, mature, and continuously improving processes. the cmm provides criteria and characteristics that need to be fulfilled in order to reach a particular maturity level. actual activities are compared with the details at each level to see what level these best align to. consideration of the details in the levels above where activities align provide guidance on where improvement can be made (becker, knackstedt, & pöppelbuß, 2009).   the first reported instance of the cmm being utilized in developing a maturity model in a library setting was by wilson and town (2006). here the cmm was used as a reference model to develop a framework for measuring the culture of quality within an organization. as part of her doctoral research, wilson (2013) went on to develop a comprehensive and useful quality maturity model (qmm) and quality culture assessment instrument for libraries (www.qualitymaturitymodel.org.uk). subsequently the cmm has been used to develop maturity models in library settings to map knowledge management maturity (mu 2012; yang 2009, 2016) and digital library maturity (sheokhshoaei, 2018). only wilson (2013) and sheokhshoaei (2018) provided a detailed account of how their model was developed.   there are other instances of developing maturity models in a library setting. gkinni (2014) developed a preservation policy maturity model; however, this used a maturity assessment model promoted by de bruin and rosemann (2005). howlett (2018) has announced a project to develop an evidence based maturity model for australian academic libraries. it will describe characteristics of evidence based practice and identify what library mangers can implement to progress maturity at a whole organization level. at this stage, it is not known whether this will follow the structure of the cmm.     figure 1 capability maturity model.     there are limited instances of the application of cmms within the library literature. an early version of the qmm was applied by tang (2012) in benchmarking quality assurance practices of university libraries in the australian technology network. egberongbe and willett (2017) refer to an assessment of quality maturity level in nigerian university libraries that applied the prince 2 maturity model from the field of project management. similarly, within a university library in sri lanka, wijetunge (2012) reported using a version of a knowledge management maturity model; however, like willett (2017), this also did not apply a cmm in its development.   aims   this paper shares the lacmm, a tool that can assist library mangers with improving assessment. the lacmm offers managers an effective tool where, through a process of self-review, assessment processes can be simplified and considered in a stage-by-stage manner along an anticipated, desired, and logical path to identify how well developed and robust processes actually are. it offers efficiency as it acts as a diagnostic tool that helps to identify a course of action to optimize performance. the process of developing this tool is presented with an evidence trail to foster confidence in its utility and value.   methods   the lacmm was developed during a series of library benchmarking activities across a group of seven universities from across the world, the matariki network (https://matarikinetwork.org/). the authors of this paper coordinated the development of the lacmm and managed the benchmarking activities. one author is a library director (h.a.) and the other (s.h.) has assessment responsibilities as a significant component of his role. the network libraries shared in the development of the lacmm as they addressed the following question: if we enable and support the academic endeavour, how do we measure our effectiveness? guidance was taken from becker, knackstedt, and pöppelbuß (2009), who offered a procedures model for developing maturity models that draws on design science research methodology (hevner, 2004). this provided a clear flow of activities and decision-making junctures, emphasising an iterative and reflective approach.   the benchmarking activities included structured case studies from each of the university libraries that were assessed and best practice examples and resources that were shared. decisions were made through consultation via shared discussion documents. these conversations occurred during three day-long annual meetings between 2013 and 2017 when the seven library directors met as part of a series of matariki humanities colloquia that had emerged as part of the network activities. prior to each meeting staff from the libraries responded to a series of questions with reference to their library’s case study. the responses were shared via an online collaborative workspace. using the workspace allowed each library to come to the activity as resources allowed. each case study could be reviewed prior to the meeting where more questions could be answered and each library could report on what they learned from considering each other’s best practice examples. this process ensured a rich and productive interaction during the meetings (hart & amos, 2014).   benchmarking topics focused on activities and practices for library programs that supported teaching, research, and the student experience. aligned to wider strategic priorities, the topics included transition of first year students to university life, library space that support students’ experiences, planning for change to support research, how the library helps researchers measure impact, and the cost and contribution to the scholarly supply chain. as the library directors considered possible areas of improvement, the need to improve assessment processes was acknowledged. early on in the benchmarking process, the library directors agreed to investigate, as a separate but aligned activity, the use of a cmm for library assessment as a shared response to address “how we measure our effectiveness” (hart & amos, 2014, p. 59).   to encourage wide application of the tool, the authors promote the use of terms “assessment” and “evaluation” as interchangeable within the library context. while some argue for a distinction between assessment and evaluation (hernon & dugan, 2009) it needs to be recognized that this call is made within the context of higher education, where historically care has been taken to differentiate between assessing learners and evaluating things or objects (hodnett, 2001). in contrast, hufford (2013) concedes that among librarians the use of each term is ambiguous, and their uses have changed over time.   results   problem definition   the idea of developing a guide or roadmap that a cmm could offer appealed to the library directors within the network. they acknowledged that there were plenty of good case studies, resources, and lists of what had to be in place to advance a culture of assessment. for example, see bibliographies by hufford (2013) and poll (2016). while these are useful to learn about what others are doing, they did not offer systematic guidance on how to improve assessment processes within current and planned activities and programmes. it was confirmed that testing the model across a group of international libraries would strengthen its application to a wider audience (maier, moultrie & clarkson, 2012; wendler, 2012).   applying the cmm to library assessment was further validated when one of the partner libraries shared their experience using the revised australasian council on online, distance and e-learning (acode) benchmarking tool, which focuses on technology-enhanced learning (mcnaught, lam, & kwok, 2012; sankey, 2014a). the acode tool includes eight benchmarks with each containing a series of criteria-based performance indicators using a 1 to 5 scale of capability. it comprises a two-phased application, where it is applied in a self-assessment process and then used to develop a team response within or between institutions (sankey, 2014b). this example was useful as it allowed the library directors to conceptualize what a lacmm may look like and how it may be utilized. it was recognized that through the benchmarking activities the library directors could review their assessment processes against criteria, compare with what others had done, and draw upon this to improve practices.   comparison with existing models   having defined the problem and agreed upon an approach, the next stage of the procedures model required comparison with existing models. here wilson’s (2013) comprehensive qmm was considered. the qmm included 40 elements grouped into 8 facets. those elements that focussed on assessment processes included progress monitoring, performance measurement, gathering feedback, collation of feedback, responding to feedback, and acting on feedback. despite this focus, the qmm was rejected for this activity because of its complexity and size. the aim was to provide an efficient tool that would not overwhelm those using it. it was also rejected because overall the facets did not provide direct alignment to library assessment. instead, it focused on the broader concept of quality of which assessment is a smaller part. it was noted that, when it came to assessment, the qmm tended to focus more on feedback and not on assessment as a process. as noted earlier, with no other suitable model dealing with the issue of library assessment available, the need to develop a distinctive lacmm was confirmed.   iterative model development   to provide guidance in determining the characteristics of a lacmm, the literature on library assessment was reviewed. bakkalbasi, sundre, and fulcher’s (2012) work on assessing assessment was considered. in presenting a toolkit to evaluate the quality and rigor of library assessment plans, their work draws on the elements of the assessment cycle. the elements include (1) establishing assessment objectives, (2) selecting and designing methodologies and collecting data, (3) analyzing and interpreting data, and (4) using the results. it was decided that focusing on these elements would reduce the complexity of the design and simplify the development of the lacmm. a template of the lacmm was determined, as illustrated in figure 2.   the lacmm template was shared with library managers and assessment practitioners at international forums. presentations were made at the 11th northumbria international conference on performance measurement in libraries and information services 2015, the oclc 7th asia pacific regional council conference 2015, and the council of australian university librarians forum: demonstrating and measuring value and impact 2016. during the discussions at these presentations, attendees confirmed the utility, value, and simplicity of the model (amos & hart, 2015; hart, 2016; hart & amos, 2015).   as part of the shared development of the lacmm, each library in the matariki network was invited to populate the model as an additional part of a benchmarking activity. they were asked to consider the assessment applied in the case study they were reporting on in the benchmarking activity, to rank the level of capability for each stage of assessment in the project, and then to provide notes of the criteria for each of these. when only three of the seven libraries completed this task with varying degrees of success, the project lead decided to change tack to get more buy in. the decision was made, in line with the iterative nature of the procedures model, that a group of library staff at the university of otago would draft criteria for the network libraries to consider in the next benchmarking activity.     figure 2 library assessment capability maturity model template.     the otago staff selected for this task all had experience in either business management and or assessment roles. they included the university librarian, the resources assessment librarian, the library programmes manager, and the policy planning and evaluation librarian. drawing on their practice and knowledge, these staff met several times to discuss, develop, and revise criteria. following this, a draft version was then tested with the staff at otago who were responsible for undertaking the next benchmarking activity.   in reviewing the version completed by otago staff as part of the benchmarking activity, the project lead noted that a number of different kinds of assessment activities had been documented. furthermore, the different types of activities were reported on in the different assessment stages of the lacmm. for example, survey data were covered in objectives, methods, and results, while group interviews were reported on in analysis. reflecting on this, the project lead decided to use the otago criteria group to produce three versions of the model for different types of assessment activities. the wording of the criteria in each corresponded to the particular assessment activity:   1.       data, to cover assessment activities that included usage data and surveys 2.       discussion, to cover assessment activities that included group interviews and focus groups 3.       comparison, to cover assessment activities that included benchmarking, case studies, standards, or best practice examples   to add more clarity, descriptions were provided for each of the levels of capability maturity and the stages of the assessment cycle (see figures 3, 4, and 5). these three versions were then distributed to the matariki libraries as part of the next benchmarking activity.   testing the model   distributing three versions of the lacmm, including specific criteria for each, proved a successful strategy with six of the seven libraries completing them. the library that did not submit indicated that the project they reported on did not lend itself to assessment activities. overall, four libraries reported on one type of assessment activity that was applied in the project, and two libraries reported on two types of activities. each library ranked their capacity maturity across each of the four stages of the assessment cycle, providing evidence about how they met the criteria.   applying the model provided each library the opportunity to review their performance and see where they could improve. following this, each of the libraries’ responses were shared among one another and then discussed at a face-to-face meeting. this meeting provided the opportunity to clarify any issues and seek more tacit information from each other on assessment processes and resources—in particular, from those who scored a higher level of capability maturity.   at the meeting, feedback on the criteria and templates for different assessment processes in the lacmm were received and then confirmed. feedback primarily focused on the wording used. fine tuning terminology across a group of international libraries helped to provide wider appeal and utility. the library directors agreed that having a template for different kinds of assessment activities assisted their staff to complete the model in the first instance. however, as their staff become familiar with using the lacmm, the directors agreed that using one generic version for any type of assessment activity would be sufficient. the directors confirmed the usefulness of the tool and decided that they had sufficiently addressed the question of how they measure their effectiveness. having built a structure and precedence for collaborating and sharing resources through the benchmarking activities, the directors agreed to refocus on other projects that support scholarly communications and digitizing collections. nevertheless, most committed to applying the lacmm in projects at a local level. two directors commented that it was hard to get their staff interested in participating in benchmarking. however, it was acknowledged that within the activities each partner had the flexibility to come to the benchmarking as resources allowed. as town (2000) asserts, “benchmarking is as much a state of mind as a tool; it requires curiosity, readiness to copy and a collaborative mentality” (p. 164).   in line with the procedure model, further testing of the generic lacmm was carried out when it was shared with the council of australian university librarians value and impact group. the group acts as a community of practice with practitioners from new zealand and australian university libraries with a quality or communication role. overall the practitioners confirmed its utility and value. they suggested including more examples in the assessment activities and that brief “how to” instructions be included. the generic version that resulted from this testing is shown in figure 6. when advancing to using the generic lacmm, it is useful to understand that the term “data” used in each of the criteria statements refers to “what is collected from each of the different assessment activities.”     figure 3 library assessment capability maturity model for data.     figure 4 library assessment capability maturity model for discussion.     figure 5 library assessment capability maturity model for comparison.     figure 6 library assessment capability maturity model generic version.     discussion   put simply, the lacmm is designed to assist library managers in assessing their assessment activities and in identifying how these can be improved until they are optimized through continuous improvement. in the first application of the lacmm, there is benefit in using a recent piece of work or an example that is considered leading practice. managers can choose a piece of work that included assessment activities or that was an assessment activity. for example, the assessment activity could be something that was carried out to inform an initiative or to review the effectiveness of an initiative.   once a piece of work has been selected, the next step is to identify the kinds of assessment activities that were applied in terms of data, discussion, or comparison (see figures 3, 4, and 5). then, for each kind of assessment activity, managers should make notes on what was carried out at each stage of the assessment cycle, including objectives, methods and data collection, analysis and interpretation, and use of results. these notes should then be compared with the criteria listed at each level of capability maturity from the initial level upwards to the optimized level for each of the stages of the assessment cycle. all of the criteria at a particular level must be met for that level to be attained. this comparison should be carried out for each kind of assessment activity applied in the piece of work.   when managers are familiar with using the lacmm for the different kinds of assessment activities, they can then move to using the generic model. here it is useful to understand that the term “data” refers to “what is collected for each of the different assessment activities.”   when comparing a piece of work, managers may identify that the first three elements of the assessment cycle meet the criteria for the defined level because the assessment processes in the piece of work are documented, standardized, and integrated. however, when it comes to the use of results, what was carried out may only meet the criteria for the repeatable level. for example, the piece of work may have inconsistent reporting with no audit trail of how results are applied. for guidance on improving this element, a manager can review the criteria in the capability level and apply those criteria in the next project. in addition, managers, especially those who attain projects with higher levels of capability, could share their experiences of using the lacmm and the processes and resources they applied.    having applied the lacmm to a representative range of assessment activities, a manager can characterize their whole assessment program. this may be a useful exercise to help set targets for improving capability across the library or for benchmarking. however, as was seen through testing the lacmm, comparing examples of leading practice where tangible examples could be shared was also beneficial.   the lacmm has advantages over other tools and processes available. in only considering the four stages of the assessment cycle, the lacmm is not as complex as wilson’s (2013) qmm, which includes 40 elements grouped into 8 facets. by focusing on assessment processes in a stage-by-stage manner, self-review is simplified. the lacmm offers efficiency as both a self-review tool and as a means of identifying improvements. although this tool will add to the plethora of resources already available (see farkas, hinchliffe, and houk, 2015 and hiller, kyrillidou, and self, 2008), the simplicity of the tool as a means of assessing assessment and identifying an improvement path is its strength. it can act as a quick aide-mémoire and form the basis of a comprehensive self-review or an inter-institutional benchmarking project (sankey, 2014b).   the benchmarking exercises provided a unique opportunity to develop the lacmm where it could be applied and tested against actual case studies of best practice across an international group of university libraries. the development utilized staff experience at different levels of the organization, including both practitioners and leaders. the results at decision-making junctures were verified at international forums of library managers and assessment practitioners. drawing on design-science research methodology (hevner, 2004) was also beneficial. the iterative approach allowed methods to be trialled and revised as required. the schedule of annual meetings with each benchmarking exercise stretched over a year provided ample time for reflection in the shared development of the lacmm as a useful artifact. being flexible with timeframes allowed each partner to come to the exercise as resources allowed (hart & amos, 2014). the successful use of the design science research methodology demonstrates the potential of this approach to other library and information practitioners.   several limitations to the lacmm and its development must be acknowledged. first, the lacmm is sequential in nature and represents a hierarchical progression. some may argue that real life is not like that. some may legitimately be content to be at a certain level and not prioritize resourcing to improve practice. second, the authors acknowledge that bias may have influenced the development of the lacmm because it became the only means for participating libraries to respond to the question of how they measure their effectiveness. however, when deciding this path, no other options were put forward by other network partners. third, limitations exist because the lacmm was developed solely within the context of university libraries. input from other areas within the wider library and information management sector would provide additional insight into the relevance and usefulness of the lacmm.   the lacmm does not replace the comprehensive and useful qmm as a means of assessing the quality of library quality (wilson, 2015). it does, however, provide an effective and efficient means of assessing library assessment.   conclusion   the lacmm is an effective tool that, through self-review assessment processes, can be simplified and considered in a stage-by-stage manner along an anticipated, desired, and logical path to identify how mature assessment processes actually are. managers can compare their effort with each level of capability maturity from the initial level through to the optimized level across each of the four stages of the assessment cycle (objectives, methods and data collection, analysis and interpretation, and use of results. the lacmm offers efficiency as it acts as a diagnostic tool that helps identify a course of action to improve performance. criteria at each level of capability maturity at the particular stage of the assessment must be met to move up a level. the level above a particular stage provides guidance on how assessment process can be improved.   it is anticipated that providing the evidence trail of the development of the lacmm will further foster confidence in its utility and value. it is expected that the tool will be adapted and improved upon as library managers apply it. as this resource is being shared with a creative commons attribution–noncommercial–sharealike license, it will support other practitioners in sharing their work with and improving the lacmm as a means of optimizing how libraries measure their effectiveness.   acknowledgements   the authors would like to acknowledge the contributions of their colleagues at the university of otago library, those across the matariki network of universities, and others that participated in various forums in the shared development the lacmm.   naku te rourou nau te rourou ka ora ai te iwi.   references amos, h., & hart s. (2015). from collaboration to convergence: a breakthrough story from an international network of libraries. paper presented at oclc 7th asia pacific regional council conference. melbourne, australia.   appleton, l. (2017). libraries and key performance indicators: a framework for practitioners. kidlington, england: chandos.   association of research libraries. (2012). libqual+, charting library service quality. washington, dc: association of research libraries. retrieved december 12, 2013 from https://www.libqual.org/documents/admin/charting library service quality.ppt   atkinson, j. (2017). academic libraries and quality: an analysis and evaluation framework. new review of academic librarianship, 23(4), 421-441. https://doi.org/10.1080/13614533.2017.1316749   bakkalbasi, n., sundre, d. l., & fulcher, k. f. (2012). assessing assessment: a framework to evaluate assessment practices and progress for library collections and services. in s. hiller, m. kyrillidou, a. pappalardo, j. self, and a. yeager. (eds.), proceedings of the 2012 library assessment conference (pp. 533-537). charlottesville, va: association of research libraries.   becker, j., knackstedt, r., & pöppelbuß, d. w. i. j. (2009). developing maturity models for it management. business & information systems engineering, 1(3), 213-222. retrieved from https://doi.org/10.1007/s12599-009-0044-5   brophy, p. (2006). measuring library performance. london, england: facet.   counting opinions. (n.d.). about us [online]. retrieved december 12, 2013 from www.countingopinions.com   crawford, j. k. (2006). the project management maturity model. information systems management, 23(4), 50-58. https://doi.org/10.1201/1078.10580530/46352.23.4.20060901/95113.7   crosby, p. b. (1979). quality is free. new york: mcgraw-hill.   crumley, e., & koufogiannakis, d. (2002). developing evidence‐based librarianship: practical steps for implementation. health information & libraries journal, 19(2), 61-70. https://doi.org/10.1046/j.1471-1842.2002.00372.x   curtis, b., hefley, b., & miller, s. (2009). people capability maturity model (p-cmm) version 2.0 (no. cmu/sei-2009-tr-003). pittsburgh pa: carnegie-mellon university, software engineering institute. retrieved from http://www.dtic.mil/dtic/tr/fulltext/u2/a512354.pdf   de bruin t. & rosemann, m. (2005). understanding the main phases of developing a maturity assessment model. paper presented at the 16th australasian conference on information systems maturity assessment model. sydney, australia. retrieved from http://eprints.qut.edu.au/25152/   egberongbe, h., sen, b., & willett, p. (2017). the assessment of quality maturity levels in nigerian university libraries. library review, 66(6/7), 399-414. https://doi.org/10.1108/lr-06-2017-0056   farkas, m.g., hinchliffe, l.j., & houk, a.h. (2015). bridges and barriers: factors influencing a culture of assessment in academic libraries. college & research libraries, 76(2), 150-169. https://doi.org/10.5860/crl.76.2.150   gkinni, z. (2014). a preservation policy maturity model: a practical tool for greek libraries and archives. journal of the institute of conservation, 37(1), 55-64. https://doi.org/10.1080/19455224.2013.873729   hart, s., & amos, h. (2014). the development of performance measures through an activity based benchmarking project across an international network of academic libraries. performance measurement and metrics, 15 (1/2), 58-66. https://doi.org/10.1108/pmm-03-2014-0010   hart, s., & amos, h. (2015). building on what works: towards a library assessment capability maturity model. paper presented at 11th northumbria international conference on performance measurement in libraries and information services, edinburgh, scotland.   hart, s., (2016). evaluating library assessment capabilities: a measure of effectiveness. paper presented at caul forum: demonstrating and measuring value and impact. sydney, australia. retrieved from https://www.youtube.com/watch?v=vniexdbsxls   heath, f. (2011). library assessment: the way we have grown. library quarterly, 8(11), 7-25. https://doi.org/10.1086/657448   hernon, p., & dugan, r. e. (2009). assessment and evaluation: what do the terms really mean? college & research libraries news, 70(3), 146-149. https://doi.org/10.5860/crln.70.3.8143   hernon, p., dugan, r. e., & nitecki, d. a. (2011). engaging in evaluation and assessment research. santa barbara, ca: libraries unlimited.   hevner, a.r., march, s. t., park, j., & ram, s. (2004). design science in information systems research. mis quarterly, 28(1), 75-105. https://doi.org/10.2307/25148625    hiller, s., kyrillidou, m., & oakleaf, m. (2014). the library assessment conference-past, present, and near future! journal of academic librarianship, 40(3-4), 410-412. https://doi.org/10.1016/j.acalib.2014.05.013   hiller, s., kyrillidou, m., & self, j. (2008). when the evidence is not enough: organizational factors that influence effective and successful library assessment. performance measurement and metrics, 9(3), 223-230. https://doi.org/10.1108/14678040810928444   hinchliffe, l., & malenfant, k. (2013, december 9). assessment in action program open online forum [webcast]. chicago, il: association of college and research libraries. retrieved from https://www.youtube.com/watch?v=0g0uqxj6cum&feature=youtu.be   hodnett, f. (2001). evaluation vs assessment. las cruces, nm: new mexico state university.   howlett, a. (2018). time to move eblip forward with an organizational lens. evidence based library and information practice, 13(3), 74-80. https://doi.org/10.18438/eblip29491   hufford, j. r. (2013). a review of the literature on assessment in academic and research libraries, 2005 to august 2011. portal: libraries and the academy, 13(1), 5-35. https://doi.org/10.1353/pla.2013.0005   lakos, a., & phipps, s. (2004). creating a culture of assessment: a catalyst for organizational change. portal: libraries and the academy, 4(3) 345-361. https://doi.org/10.1353/pla.2004.0052   lilburn, j. (2017). ideology and audit culture: standardized service quality surveys in academic libraries. portal: libraries and the academy, 17(1), 91-110. https://doi.org/10.1353/pla.2017.0006   maier, a. m., moultrie, j., & clarkson, p. j. (2012). assessing organizational capabilities: reviewing and guiding the development of maturity grids. ieee transactions on engineering management, 59(1), 138-159. https://doi.org/10.1109/tem.2010.2077289   matthews, j. r. (2015). library assessment in higher education (2nd ed.). westport, ct: libraries unlimited.   mcnaught, c., lam, p., & kwok, m. (2012). using elearning benchmarking as a strategy to foster institutional elearning strategic planning. working paper 11. hong kong: centre for learning enhancement and research, the chinese university of hong kong. retrieved from http://www.cuhk.edu.hk/clear/research/wp11_mcnlk_2012.pdf   mu, y. (2012). appraisal of library knowledge management capability evaluation and empirical analysis. information studies: theory & application, 35(3), 83-86. retrieved from http://en.cnki.com.cn/article_en/cjfdtotal-qbll201203020.htm   oakleaf, m. (2010). the value of academic libraries: a comprehensive research review and report. chicago, il: association of college and research libraries.   orr, r. h. (1973). measuring the goodness of library services: a general framework for considering quantitative measures. journal of documentation, 29(3), 315-332. https://doi.org/10.1108/eb026561   paulk, m. c., curtis, b., chrissis, m. b., & weber, c. v. (1993). capability maturity model, version 1.1. institute for software research paper 7. retrieved from http://dx.doi.org/10.1109/52.219617   poll, r. (2016). bibliography on the impact and outcome of libraries. retrieved from http://www.ifla.org/files/assets/e-metrics/bibliography_impact_and_outcome_2016.pdf   randall, w. m. (1932). the college library. chicago, il: american library association.   rendon r. g. (2009). contract management process maturity: empirical analysis of organizational assessments. monterey, ca: naval postgraduate school. retrieved from https://apps.dtic.mil/docs/citations/ada633893   sankey, m. (2014a). benchmarks for technology enhanced learning: a report on the refresh project 2014 and inter-institutional benchmarking summit. retrieved august 11, 2015 from www.acode.edu.au/mod/resource/view.php?id=269   sankey, m. (2014b). benchmarking for technology enhanced learning: taking the next step in the journey. in b. hegarty, j. mcdonald, and s. k. loke (eds.), rhetoric and reality: critical perspectives on educational technology: proceedings ascilite dunedin 2014 (pp. 668-672). dunedin, new zealand. ascilite.   sheikhshoaei, f., naghshineh, n., alidousti, s., & nakhoda, m. (2018). design of a digital library maturity model (dlmm). the electronic library. https://doi.org/10.1108/el-05-2017-0114   sputore, a., & fitzgibbons, m. (2017). assessing ‘goodness’: a review of quality frameworks for australian academic libraries. journal of the australian library and information association, 66(3), 207-230. https://doi.org/10.1080/24750158.2017.1344794   tang, k. (2012). closing the gap: the maturing of quality assurance in australian university libraries. australian academic & research libraries, 43(2), 102-119.   tenopir, c., mays, r. n., & kaufman, p. (2010). lib value: measuring value and return on investment of academic libraries. school of information sciences. retrieved from http://trace.tennessee.edu/utk_infosciepubs/50   town, j. s. (2000). benchmarking the learning infrastructure: library and information services case studies. in m. jackson, and h. lund (eds.), benchmarking for higher education (pp. 151-164). buckingham, england: ou press.   town, j. s., & stein, j. (2015). ten northumbria conferences: the contribution to library management. library management, 36(3). https://doi.org/10.1108/lm-11-2014-0135    urquhart, c., & tbaishat, d. (2016). reflections on the value and impact of library and information services: part 3: towards an assessment culture. performance measurement and metrics, 17(1), 29-44. https://doi.org/10.1108/pmm-01-2016-0004   wendler, r. (2012). the maturity of maturity model research: a systematic mapping study. information and software technology, 54(12), 1317-1339. https://doi.org/10.1016/j.infsof.2012.07.007   wijetunge, p. (2012). assessing knowledge management maturity level of a university library: a case study from sri lanka. qualitative and quantitative methods in libraries, 1(3), 349-356. retrieved from http://www.qqml-journal.net/index.php/qqml/article/view/70   wilson, f. (2013). the quality maturity model: assessing organisational quality culture in academic libraries (doctoral dissertation). london, england: brunel university, school of information systems, computing and mathematics. retrieved from https://bura.brunel.ac.uk/handle/2438/8747   wilson, f. (2015). the quality maturity model: your roadmap to a culture of quality. library management, 36(3), 258-267. https://doi.org/10.1108/lm-09-2014-0102   wilson, f., & town, j. s. (2006). benchmarking and library quality maturity. performance measurement and metrics, 7(2), 75-82. https://doi.org/10.1108/14678040610679461   wright, s., & white, l. s. (2007). spec kit 303 library assessment. washington, dc: association of research libraries.   yang, z., & bai, h. (2009). building a maturity model for college library knowledge management system. international conference on test and measurement, ictm 2009 (pp.1-4). hong kong, ieee. https://doi.org/10.1109/ictm.2009.5412987   yang, z., zhu, r., & zhang, l. (2016). research on the capability maturity model of digital library knowledge management. proceedings of the 2nd international technology and mechatronics engineering conference. chongqing, china: itoec. https://doi.org/10.2991/itoec-16.2016.63   editorial   open data for evidence based practice   lorie kloda editor-in-chief associate university librarian, planning and community relations concordia university montreal, quebec, canada email: lorie.kloda@concordia.ca    2018 kloda. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29505     as editor-in-chief i’m very pleased to announce the journal’s new policy on data sharing. the policy is available on the eblip website submission page (at the very end): https://journals.library.ualberta.ca/eblip/index.php/eblip/about/submissions. the editorial team spent almost a year discussing the idea of such a policy – whether it was worth having one, whether it would advocate for mandatory or voluntary data sharing, and how detailed it should be. we drafted and revised the policy to be sure that our intentions were clear and to enable authors wishing to share data to be able to do so in a straightforward manner. the end result, as you will see, is rather detailed and references data standards and recommendations from universities and organizations.   since its inception, eblip has been an open access journal, advocating for access to research in order to support evidence based decision making. open data is an extension of the same principle as sharing research data can benefit scholarship in many ways. though researchers in library and information studies may still be new to the idea of disseminating data alongside their published articles, data sharing is becoming increasingly popular and eblip wanted to be prepared for data submissions and to encourage future authors to prepare to store and disseminate their research data ethically. if you have comments regarding the journal’s data sharing policy, please contact me or a member of the editorial team to share these.   in this issue, i would like to welcome our new production editor, rachel hinrichs. rachel is experienced with the journal as she served as the editorial intern for the past year and a half. she takes over from michelle dunaway, who is stepping down after six years (that’s 24 issues!) in the role. before being appointed as production editor in 2012, michelle was also the editorial intern starting in 2011. on behalf of the editorial team i would like to express my sincere appreciation for michelle’s contributions to eblip over the previous eight years. we will not, however, have to say farewell to michelle entirely, as she will stay on with the journal as an editorial advisor and we can continue to take advantage of her expertise. with rachel now taking on the role of production editor, kimberly mackenzie has been appointed as editorial intern.   research article   facilitating global art conversations: availability of art scholarship in latin america   alexander c. watkins assistant professor university libraries university of colorado boulder boulder, colorado, united states of america email: alexander.watkins@colorado.edu   received: 2 aug. 2016     accepted: 13 nov. 2016      2016 watkins. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – as art history becomes an increasingly global discipline, the question of geographically equitable access to the scholarly knowledge produced at universities in europe and north america remains unexamined. this study aims to begin to answer that question by investigating the availability of art scholarship in latin america.   methods – sixty university libraries in latin america were checked for various kinds of access to two major art history journals.   results – the study found that access rates were low, and that the types of access available were suboptimal.   conclusion – the results suggest that the current level of access is insufficient to support global scholarly conversations in art history and that current modes of dissemination of scholarship are not reaching key audiences.     introduction   the study of latin american art is a major endeavour at universities in europe and north america, but how much of the knowledge produced at these universities is available to latin american scholars? this study aims to answer the question of whether latin american scholars have access to the art scholarship of europe and the united states. it specifically focuses on scholars at latin american research universities working towards open scholarship. recently, local art histories have flourished as the discipline expands from its origins in western europe to become a global enterprise, but even with this global expansion of study we have not necessarily seen a concomitant expansion of scholarly communication. the question of access is a key part of facilitating a global scholarly conversation, as access to the theories and analyses that are current in the scholarly literature of the global centre are required for international communities to engage with these ideas and communicate their own perspectives back to the centre.   a note on terminology   there are a multitude of terms to describe the unequal global distribution of wealth caused by the legacy of colonialism. in this paper, i use the centre-periphery terminology and model as adapted to the world of scholarly publishing by suresh canagarajah (2002). the global centre consists primarily of the united states and western europe; however, there are many centres within countries that are part of the global periphery, which also take advantage of colonial mechanisms to concentrate wealth. there are also peripheries within centre nations, excluded from the prosperity that is concentrated in certain areas of that nation. the term centre scholarship in this paper includes scholarship produced by academics at institutions with significant social and financial capital concentrated but not exclusively found in north america and western europe. this unequal distribution of resources and reputation has a particular impact on access to the scholarly literature of the centre in the periphery, as this literature is made available most commonly on a toll access basis.   literature review   the traditional publishing model in which scholars give their articles to publishers, and those publishers sell the articles back to scholars and their universities is one that has hampered access to knowledge around the world. the rapid rate of increase in journal prices means that scholars must be associated with well-funded universities to access the full breadth of the scholarly literature, and this tends to disproportionately affect the access of periphery scholars. as early as 1995, this issue was discussed in scientific american, which recounts journal cancellations in libraries across the developing world. although latin america was found to have the most access to major journals of the three regions surveyed, libraries in africa, india, and latin america were all found to be lacking access to necessary serials collections (gibbs, 1995a). sri lankan scholar suresh canagarajah vividly recounts his experiences trying to get access to scholarly literature, when at his university it was unthinkable to get the latest scholarly journal or book. he speaks directly to the fact that his work and that of his peers was hampered without this access (canagarajah, 2002). consequently, many periphery scholars must employ slow, expensive, or convoluted work-arounds to deal with a lack of access, such as emailing article authors or traveling specifically to visit centre libraries (bonaccorso et al., 2014). the purchasing power of developing world libraries is further taxed by the extreme prices charged by academic publishers (arunachalam, 2003; davison, harris, licker, & shoib, 2005). specifically in latin america, limited financing means that university libraries often have incomplete collections with little ability to plan for the long term (holdom, 2005; terra figari, 2007). however, at the date of writing there has been little to no research on holdings of specific journals by periphery libraries, especially in humanities disciplines. this has made it hard to quantify the extent of this lack of access to journals. the problematic repercussions of this lack of access is often framed as one of distributive justice, where academic paywalls have recreated patterns of social exclusion and the dominance of the centre over the periphery (alperín, fischman, & willinsky, 2008; gómez & bongiovani, 2012).   insufficient access to scholarly publications creates a barrier to periphery scholars publishing in centre academic journals. keeping up with the frontier of knowledge development in the scholarly literature of the centre is impossible without access to current journals and databases (teferra, 2004). this lack of access creates a tendency to emphasize foundational works and to omit the latest developments of the centre (terra figari, 2007). this puts periphery scholars at a distinct disadvantage when publishing in journals of the centre, as the peer-review process requires writers to reference the most current centre scholarship (gibbs, 1995b; willinsky, 2006). without access to the current literature of the centre, periphery scholars are left out of the scholarly conversation and excluded from full participation in the process of knowledge creation (canagarajah, 2002; holdom, 2005). this creates a situation in which developing countries have the art, but in a striking parallel to colonial exploitation of raw materials, it has to be analyzed in the centre to be turned into scholarly knowledge accepted by the centre (canagarajah, 2002).   latin america is in many ways at the forefront of creating open scholarly knowledge. for example, open access (oa) publishing has been readily adopted in latin america. indeed, the oa model is much more prevalent in latin american than in most other regions; a full 51% of online journals in latin america are open access (alperín et al., 2011, 2008). growing internet connectivity and the historical lack of visibility of latin american print journals has meant that oa e-publishing gives scholars in latin america new opportunities to disseminate their research (holdom, 2005). several factors have enabled the wide adoption of oa in latin america including the lack of an entrenched scholarly publishing industry and first-hand experience by latin american scholars with the consequences of limited access (alperín et al., 2008). not only are e-journals flourishing, but open access repositories, databases where copies of articles are archived and made freely available, have allowed scholars to make their work openly available even when they publish in toll access journals (alperín et al., 2008; johnston, 2010).   aims   the study’s goal was to determine the availability of centre art journals at latin american universities that are practicing open scholarship. the literature review revealed that latin american scholars are making their work openly accessible to global scholars, but do scholars at these universities have access to the core journals of centre art scholarship?   methods   the first step was identifying latin american universities that are practicing open scholarship. this study used the opendoar database to select institutions. opendoar lists universities with institutional repositories by country. these institutions have created and support databases where their affiliates can deposit their scholarly products and have them made openly available, demonstrating participation in the open access movement. additionally, the resources and staff necessary to operate a repository suggest a certain minimum level of funding. universities with a singular focus like engineering or medicine were eliminated as out of scope. due to language limitations only institutions in spanish speaking countries were selected. after excluding institutions that did not meet the criteria, there were a total of 78 institutions; however, for 18 of these, reliable subscription information could not be located, so the final sample was 60 university libraries.   the study investigated access to two art history journals: the burlington magazine and the art bulletin. these two journals were selected because they are core journals for art history. the burlington magazine is the longest continually published art periodical in english. published in the united kingdom, it set the standard for scholarly art history publications, cementing its reputation with a string of well-respected editors (fawcett & phillpot, 1976). beginning publication not long after, the art bulletin rose in prominence to become arguably the most influential art journal (fawcett & phillpot, 1976). it is published in the united states of america by the college art association. neither journal focuses exclusively on latin america; instead the articles, editorials, letters, and reviews in these journals are key sites of the scholarly back-and-forth that generates new scholarly knowledge in the centre. because the goal was to determine latin american scholars’ ability to participate in the broader scholarly conversation going on in centre art history, these journals were selected specifically because of their importance to the discipline as a whole, rather than because of a focus on latin american art. selecting journals that solely study latin american art would have risked pigeonholing latin american scholars and suggesting that they are only able to work on local topics, while centre scholars enjoy the whole purview of global art to study.   the sixty university library websites and catalogs were investigated for access to each journal. each of the various ways that universities had access to the journal was recorded. as an additional check to catalog and website searching, an e-mail in english and spanish was sent to these sixty libraries, in order to confirm that availability had not been missed. the responses that were received were then checked against the information gathered from the websites. we found that the emails verified the information found on websites and catalogs.   this study has several limitations. firstly, it includes only two art history journals and relatively expensive ones at that. however, these journals represent major loci of scholarly conversations in art history. scholars attempting to write art history that is publishable in centre journals would find themselves confronting a nearly unbridgeable lacuna in their research without access to articles from these journals. while this is a limitation, the selected journals are used as indicators of problematic access to centre art history scholarship as a whole. additionally, the study only examines latin american universities with institutional repositories. therefore, it does not necessarily reflect the situation at all latin american universities, only those with repositories. however, if institutions with repositories are outliers, the funding commitment and know-how that a repository represents suggests that their libraries may be more well-funded than the average. as the study only looks at these major universities, it consequently left out smaller, perhaps more art-focused institutions such as museum libraries. while access at specialty libraries is certainly an interesting question, a major concern of this study was the ease and convenience of reading these journals at research universities, where the majority of scholars are concentrated. while scholars may be willing to make extraordinary efforts to track down a single key article, these hurdles waste scholars’ time, and preclude them from keeping up with general trends and emerging ideas, which are made possible by easy access through one’s institution.   results   the results show that most latin american scholars at institutions with repositories lack access to these two major journals of art history. a large percentage of institutions had no access to either publication. where there was access it was generally suboptimal, often only available after an embargo or at the mercy of a commercial vendor which could drop coverage at any time.   the study found several ways that libraries provide access to these two journals. they may subscribe directly to print or electronic editions. older issues of both journals are available through a subscription to jstor, though not current issues, as there is an embargo of five years for the burlington magazine and the art bulletin in jstor. libraries that subscribe to some ebsco or proquest full-text packages such as academic search premiere can also get access to the art bulletin. however, this access is unstable, as these content aggregators can drop the full-text access at any time. additionally, the extent of back issue access varies among packages, and none provide full access to the entire back file. there were five kinds of access for each journal possible at each institution.   subscription: a direct subscription to the print or electronic version of the journal. considered full access. jstor: access through jstor to back issues, but lacking the five most recent years due to the embargo. aggregator: access via proquest and ebsco to current issues and varying amounts of back file, but this access is unstable and unreliable. jstor and aggregator: access to current issues via an aggregator as well as reliable access to back issues through jstor. considered full access. no access.     figure 1 the burlington magazine is unavailable at most latin american universities.     figure 2 the art bulletin is more accessible, but much of the access is unreliable.     as shown in figure 1, only a single institution had access to recent issues of the burlington magazine. in a full 72% of institutions, the burlington magazine was not available at all. another 27% had access through jstor to articles, but only with a five year embargo.   as shown in figure 2, art bulletin was unavailable at 42% of the institutions, while 58% had access of some kind to recent issues. the journal’s relative availability is due to its inclusion in the typical package of full-text journals subscribed to through content aggregators such as ebsco and proquest. indeed, only two institutions (3%) had direct subscriptions to the art bulletin. this means that 55% of institutions had access to recent issues of the art bulletin entirely through aggregators. problematically, however, access through content aggregators is not stable. ebsco or proquest could cut art bulletin from their packages or the art bulletin could decide to withdraw, and these institutions would be left with no access to the journal, not even to back issues. additionally, for those institutions with only aggregator subscriptions, there is a lack of access to a substantial amount of the back file. some institutions also had subscriptions to jstor (27%) that gives them stable access to back issues. when combined with their aggregator subscriptions, these institutions, along with those with direct subscriptions, were considered to have full access to the art bulletin. overall, 30% of institutions had full access to the art bulletin either through subscription or a combination of jstor and aggregator access.   figure 3 shows a cross-sample of both journals, in which 42% of institutions had no access to either journal. a combined 30% had some access to one journal (always the art bulletin), either full access (3%) or unstable access (27%), while having no access to the burlington magazine. only a combined 28% of institutions had some access to both journals. in this small group that had access to both journals, only one institution had full access to both journals without an embargo, while the rest had full access to the art bulletin but only had access to the burlington magazine after a five-year embargo.     figure 3 the combined picture of the two journals shows that many institutions lack access to either journal, while very few institutions have access to both.     discussion   the results show a concerning lack of access to these centre-published art history journals at latin american universities with institutional repositories. this study found that even when access does exist it is often delayed by five years. the results suggest that latin american scholars at these institutions will have difficulty reading art history articles published in centre journals in a timely way. as the literature has shown, limited access to centre journals hinders periphery scholars’ ability to publish in these same journals because of the difficulty of staying up-to-date on the most recent centre theory (gibbs, 1995b; willinsky, 2006). therefore, lack of access limits periphery scholars’ ability to fully participate in centre discourse. they will be challenged to communicate their theories, ideas, and interpretations to centre scholars, and they will have difficulty debating the work of centre scholars, even when that scholarship is on the art of latin america. lack of access thus helps to perpetuate a colonial system of art history knowledge creation in which new knowledge is created and given authority by those in the centre, and where the art and ideas of latin america only enters the scholarly discourse after being analyzed by centre scholars.   due to barriers to participation in centre scholarship, the ideas and theories of these periphery scholars are likely to be published in local journals. these publications have a high chance of remaining unseen by centre scholars. previous studies have shown that much of the art history published in latin america, though often made available through open access, is difficult to find through conventional research methods (alperín et al., 2011; evans, thompson, & watkins, 2011; holdom, 2005). as a result of inadequate information access, there is breakdown in global scholarly communication, where art history ideas are not being transmitted between centre and periphery. the theory and analyses created in both the periphery and the centre remain in separate spheres, rather than becoming engaged in meaningful dialogue and productively building on one another. thus the scholarly conversation in art history is impoverished, losing key voices while privileging those scholars with greater information resources.   solving the access problem for these latin american universities will require a change in traditional systems of knowledge distribution in the discipline. simply having latin american institutions increase their journal subscriptions is not a viable solution. when many libraries are cancelling subscriptions, and the rising cost of existing subscriptions exceeds inflation, this is simply untenable (hoskins & stilwell, 2011; spencer & millson-martula, 2006). centre scholars should question whether only publishing in a scholarly journal, even (and perhaps particularly) top tier toll-access journals, adequately disseminates their work to the global scholarly community. open access publishing is a well-established alternative model in which access is free for the reader. open access has already been adopted by many latin american scholars: all the institutions in this study already have institutional repositories, and open access journals are far more popular in latin america than in the united states. if centre scholars were to increase their adoption of open access practices, their scholarship would become far more accessible and easily available to latin american scholars. importantly this would start to alleviate access problems in latin america and facilitate global conversations.   conclusion   access to scholarship is often overlooked in calls for a more global art history. but far from being a secondary concern, it is a key requirement for scholarly conversations that truly integrate global perspectives and move away from an inherently limited centre-out model of scholarship. more openly available scholarship is necessary if global voices are to participate in centre art history, and if centre and periphery discourses are to be joined into a single, richer discussion. it seems that traditional models for access to and dissemination of scholarship are not up to this task. the evidence of substandard access to centre art history scholarship suggests there is further work to be done investigating access to information in the periphery, as well as the effect of this limited access on the work of scholars in a range of other disciplines. it is this author’s sincere hope that this work will catalyze and build towards sustainable solutions, as well as motivate individual scholars to help create a more global discourse by moving toward open access scholarship.   acknowledgements   the author wishes to thank his research assistant, amanda saracho, whose language skills and time spent combing library websites were key to the writing of this article.   references   alperín, j. p., fischman, g. e., & willinsky, j. (2011). scholarly communication strategies in latin america’s research-intensive universities. educación superior y sociedad, 16(2). retrieved from http://pkp.sfu.ca/files/iesalc_final.pdf   alperín, j. p., fischman, g., & willinsky, j. (2008). open access and scholarly publishing in latin america: ten flavours and a few reflections. liinc em revista, 4(2). dx.doi.org/10.18617/liinc.v4i2.269   arunachalam, s. (2003). information for research in developing countries — information technology, a friend or foe? the international information & library review, 35(2–4), 133–147.   bonaccorso, e., bozhankova, r., cadena, c., čapská, v., czerniewicz, l., emmett, a., … tykarski, p. (2014). bottlenecks in the open-access system: voices from around the globe. journal of librarianship and scholarly communication, 2(2). dx.doi.org/10.7710/2162-3309.1126   canagarajah, a. s. (2002). a geopolitics of academic writing. pittsburgh, pa: university of pittsburgh press.   davison, r., harris, r., licker, p., & shoib, g. (2005). open access e-journals: creating sustainable value in developing countries. pacis 2005 proceedings, 21. retrieved from http://aisel.aisnet.org/pacis2005/21/   evans, s., thompson, h., & watkins, a. (2011). discovering open access art history: a comparative study of the indexing of open access art journals. the serials librarian, 61(2), 168–188.   fawcett, t., & phillpot, c. (eds.). (1976). the art press: two centuries of art magazines. london: the art book company.   gibbs, w. w. (1995a). information have-nots. scientific american, 272(5), 12b–14. https://doi.org/10.1038/scientificamerican0595-12   gibbs, w. w. (1995b). lost science in the third world. scientific american, 273(2), 92–99. https://doi.org/10.1038/scientificamerican0895-92   gómez, n., & bongiovani, p. c. (2012). open access and a2k: collaborative experiences in latin america. in j. lau, a. m. tammaro, & t. j. d. bothma (eds.), libraries driving access to knowledge (pp. 343–371). berlin: de gruyter saur.   holdom, s. (2005). e-journal proliferation in emerging economies: the case of latin america. literary & linguistic computing, 20(3), 351–365. https://doi.org/10.1093/llc/fqi033   hoskins, r., & stilwell, c. (2011). library funding and journal cancellations in south african university libraries. south african journal of libraries and information science, 77(1), 51–63. http://dx.doi.org/10.7553/77-1-66   johnston, m. (2010). changing the paradigm: the role of self-archiving and institutional repositories in facilitating global open access to knowledge. access to knowledge: a course journal, 2(1). retrieved from http://ojs.stanford.edu/ojs/index.php/a2k/article/view/429   spencer, j. s., & millson-martula, c. (2006). serials cancellations in college and small university libraries. the serials librarian, 49(4), 135–155. https://doi.org/10.1300/j123v49n04_10   teferra, d. (2004). striving at the periphery, craving for the centre: the realm of african scholarly communication in the digital age. journal of scholarly publishing, 35(3), 159–171.   terra figari, l. i. (2007). diseminación del conocimiento académico en américa latina. in anuario de antropologia social y cultural en  uruguay (pp. 193–204). montevideo, uruguay: editorial nordan. retrieved from http://www.unesco.org.uy/shs/fileadmin/templates/shs/archivos/anuario2007/articulo_15.pdf   willinsky, j. (2006). the access principle: the case for open access to research and scholarship. cambridge, ma: mit press.   research article   through the students’ lens: photographic methods for research in library spaces   shailoo bedi director, academic commons & strategic assessment university of victoria libraries university of victoria victoria, british columbia, canada email: shailoo@uvic.ca   jenaya webb public services and research librarian ontario institute for studies in education (oise) library university of toronto libraries university of toronto toronto, ontario, canada email: jenaya.webb@utoronto.ca    received: 15 jan. 2017    accepted: 29 apr. 2017      2017 bedi and webb. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – as librarians and researchers, we are deeply curious about how our library users navigate and experience our library spaces. although we have some data about users’ experiences and wayfinding strategies at our libraries, including anecdotal evidence, statistics, surveys, and focus group discussions, we lacked more in-depth information that reflected students’ real-time experiences as they move through our library spaces. our objective is to address that gap by using photographic methods for studying library spaces.   methods – we present two studies conducted in two academic libraries that used participant-driven photo-elicitation (pdpe) methods. described simply, photo-elicitation methods involve the use of photographs as discussion prompts in interviews. in both studies presented here, we asked participants to take photographs that reflected their experiences using and navigating our library spaces. we then met with participants for an interview using their photos as prompts to discuss their experiences.   results – our analysis of students’ photos and interviews provided rich descriptions of student experiences in library spaces. this analysis resulted in new insights into the ways that students navigate the library as well as the ways that signage, furniture, technology, and artwork in the library can shape student experiences in library spaces. the results have proven productive in generating answers to our research questions and supporting practical improvements to our libraries. additionally, when comparing the results from our two studies we identified the importance of detailed spatial references for understanding student experiences in library spaces, which has implications beyond our institutions.   conclusion – we found that photographic methods were very productive in helping us to understand library users’ experiences and supporting decision-making related to library spaces. in addition, engaging with students and hearing their interpretations and stories about the photographs they created enhanced our research understandings of student experiences and needs in new and unique ways.   introduction   students’ images can elicit stories that are not easily captured through other research methods. they can generate rich descriptions of library spaces and reveal new and important insights into the ways our users experience, navigate, and perceive the library. however, when planning changes and improvements to library spaces or services, librarians often rely on methods such as surveys or focus groups to seek input from users. as halpern, eaker, jackson, & bouquin (2015) note, “over-reliance on the survey method is limiting the types of questions we are asking, and thus, the answers we can obtain” (p. 1). although surveys and focus groups are valuable for many types of research, they are limited in providing "in the moment,” experiential data about how students use our library spaces. in surveys and focus groups, students may be asked to recall their perceptions or provide hindsight thinking about their experiences with library spaces, services, or resources. however, when equipped with cameras students can photographically document their experiences in a library space as they move through it. the exercise of collecting images and discussing them during follow-up interviews allows for deeper consideration of the perceptions and experiences of being in a particular library space.   over the past couple of decades, many libraries have focused their efforts on becoming user-centered, dynamic learning environments developed to support student success. this change in focus has rendered much discussion in the library and information science (lis) literature about the library as the “third place” (ferria et al., 2017; harris, 2007; montgomery & miller, 2011). authors often point out that despite the proliferation of online resources the library’s physical space is still critical to our users (brown & lippincott, 2003; harris, 2007; montgomery & miller, 2011). thus, continued exploration into library user experience of library space, design, and wayfinding is worthy of attention.   we propose that visual research methods, specifically photographic methods, have a much larger role to play in describing, interpreting, and understanding library users’ experiences. although visual research methods remain underrepresented in the lis literature, there are a few compelling examples that demonstrate great promise for lis research, especially for research focused on physical library spaces and the student experience. moreover, visual research methods offer valuable evidence for decision-making and user-focused improvements to library space and design.   in this article we present two studies that use visual research methods. the first uses participant-driven photo-elicitation (pdpe) to understand users’ wayfinding strategies at the university of toronto’s ontario institute for studies in education (oise) library, with the added goal of making user-focused improvements to directories and signage. the second explores students’ experience in the library spaces at the university of victoria using photo-narrative as a way of guiding decisions for upcoming renovations and to understand the student experience in library space. in addition to focusing on questions in our immediate institutional contexts, we also make the broader argument that students’ experiences with the library are interwoven with the spaces and objects they encounter, and that visual methods, and photographic methods in particular, can reveal new and important insights into the ways library users experience, navigate, and perceive library spaces.   literature review   visual methods are well established across the social sciences and encompass a wide range of approaches, techniques, and types of images. among many other purposes, they can be used as a way for researchers to document social processes (hartel & thomson, 2011), as part of ethnographic approaches to elicit information from participants (foster & gibbons, 2007), or as a way to engage and empower communities (julien, given, & opryshko, 2013). pollak (2017) provides a summary of visual methods used in the social sciences and argues visual approaches are well suited to lis researchers “exploring information worlds filled with vagueness, contradiction, fluidity, and movement” (p.105). the literature on visual methods is vast, but weber (2008) offers a comprehensive summary list of reasons to use images in research   images can be used to capture the ineffable, the hard-to-put-into-words. images can make us pay attention to things in new ways. images are likely to be memorable. images can be used to communicate more holistically, incorporating multiple layers, and evoking stories or questions. images can enhance empathic understanding and generalizability. through metaphor and symbol, artistic images can carry theory elegantly and eloquently. images encourage embodied knowledge. images can be more accessible than most forms of academic discourse. images can facilitate reflexivity in research design. images provoke action for social justice. (p. 44)   visual research methods have slowly begun to gain ground as part of a move toward more holistic approaches to studying libraries and library users. the groundbreaking ethnographic project, studying students: the undergraduate research project at the university of rochester, led by foster and gibbons (2007), employed a wide range of methods including visual methods in which students produced photographs, maps, and drawings as part of the research process. briden (2007) discusses the rochester project’s use of photo-elicitation interviews as a way to have students share “details about their lives in a way that conventional interviews alone could not achieve” (p.47). researchers put cameras in the hands of participants and provided them with a list of 20 photo prompts such as “all the stuff you take to class”, “your favorite place to study”, and “your favorite part of the day” (p. 41). the resulting images, in conjunction with interviews, brought together a vivid description of students’ lives at the university of rochester, and helped shed light on how the library factored into the total student experience.   more recent examples of visual methods applied in library contexts also show significant promise for providing new insights into our users’ experiences (haberl & wortman, 2012; julien, given, & opryshko, 2013; lin & chiu, 2012; neurohr & bailey, 2016; newcomer, lindahl, & harriman, 2016; treadwell, binder, & tagge, 2012). furthermore, these studies point to the constructive, user-centered input that visual methods can provide for making improvements to library spaces and services. for example, newcomer et al. (2016) used photo-elicitation as part of a broader ethnographic project to solicit student input on the design of a new arts campus at their institution. in their conclusion they highlight the value of ethnographic approaches for gathering unexpected data from user populations, and note that the results of their study have already been used to inform planning for the new arts campus.   there are many considerations in using photographs in research, one of the primary ones being that photographs are not neutral; they are contextual, intentional products. by using photographs in conjunction with interviews in methods such as photo-elicitation and photo-narrative, researchers can investigate these complexities and understand the photographer’s intentions. narrating through an image means storytelling about things and experiences related to what has been photographed; it does not mean telling or describing only what can be seen in a picture (collier, 2001; pink, 2001). the photo by itself is not an independent data point or an objective representation of data. rather the photo is an interpretation of the creators’ subjective experience (liebenberg, 2009). the process of photographic research methods, such as photo-narrative or photo-elicitation, goes beyond describing each photograph taken by the participant, and includes an interview that incorporates questions about what it is we are seeing, and what it is that we are not seeing and why. questions about what was happening before and after a given photo are also critical to understanding the contextual details (liebenberg, 2009; pink, 2001). ultimately, photographs can be thought of as a starting point in photo-elicitation. as weber (2008) writes, photographs and other artworks “provide a versatile and moveable scaffolding for the telling of life history, life events, life material” (p.48).   aims   as librarians and researchers, we wanted to know more about how our library users experience our library spaces. although we had data from our own libraries about space use and wayfinding gained through anecdotal evidence and assessment instruments like statistics, surveys, and focus group discussions, we recognized that we were missing more in-depth research information that reflected specific and “in the moment” student experiences in our spaces. as a result, the aim of both studies presented here is to gather data that provides detailed and in-depth knowledge about user experiences in our library spaces.   the overarching research question that frames the two studies is: how do students use and navigate our library spaces? while both projects have goals for local service improvements, this work will also contribute to the lis literature by expanding our understanding of how students’ experiences within the library are tightly interwoven with the spaces and objects they encounter during their visits. by examining and comparing the use of photographic methods in two independent studies and argue that photographic methods have broad applicability for researchers interested in library space and design.   methods   the studies we present in this article use two types of photo-elicitation methods to examine student experiences in library spaces. described simply, photo-elicitation involves the use of photographic data to provide discussion stimuli in interviews. the photos used in the interview can be photos taken or collected by the researcher, but more commonly, are photographs taken by the research participants themselves that are then later discussed in the photo-elicitation interview. this method is often referred to as participant-driven photo-elicitation (pdpe). the studies presented here both use forms of pdpe where research participants took the photographs used in the research.   in her summary of the literature, rose (2012) identifies four main strengths of photo-elicitation interviews:   photo-elicitation interviews evoke different information than other social science techniques. in other words, “things are talked about in these sorts of interviews that don’t get discussed in talk-only interviews” (p. 305). photographs can be helpful in shedding light on the more mundane or day-to-day activities of participants’ lives. having research participants take photographs and then discuss them in interviews, gives participants a “distance from what they are usually immersed in and allows them to articulate thought and feelings that usually remain implicit” (p. 306). pdpe gives participants more power in the research process. putting research participants in charge of making photographs, rather than simply answering a researcher’s questions, “gives them [participants] a clear and central role in the research process” (p. 306). photo-elicitation also facilitates collaboration between the researcher and participants that other methods do not.   by using photo-elicitation, we sought to gain new insights into students’ day-to-day experiences in library spaces. moreover, we hoped to engage in a more user-driven type of research. the particular adaptations of photo-elicitation applied in each situation, and the research instruments used, are described below.   oise library, university of toronto   context   the oise library’s wayfinding study was designed to gain insight into the challenges and successes that users face when navigating the oise library space. the oise library is 1 of 44 libraries in the university of toronto libraries system, with collections and services that support graduate students and faculty in the field of education. the library houses a main stacks collection and several special collections, including a historical education collection, a juvenile fiction collection, and a curriculum resources collection for teacher candidates.   research questions   reference desk interactions, directional statistics, informal observations, and anecdotes from staff at the library’s service desk all indicate that users experience difficulties navigating the library space and locating resources. however, our current knowledge falls short of understanding what actually happens when students leave the desk. moreover, we have very little meaningful knowledge of users’ personal experiences navigating the library space. in seeking to fill these gaps, my research sought to answer the following questions:   how do users navigate the library? how do users locate places and items in the library? where are they successful? where, specifically, do they encounter barriers? how do they perceive libraries?   recruitment   i used a broad recruitment strategy directed to students in the first year of their programs at oise. i targeted new oise students because i sought participants with a range of library experiences but also wanted to include students who were new to the oise library space. i sent invitations to all of oise’s incoming students via the library’s personal librarian program emails (~1,000 students). over 20 students responded, and i was able to recruit 17 of those to complete the study. the 17 participants represented a combination of frequent oise library users, students who had never been to the oise library but had experiences in academic libraries, students who described themselves as having rarely used any library (academic or public), and several international students who considered the experience quite different from library experiences in their home countries.   method: participant-driven photo-elicitation   at its basic level, photo-elicitation is a method that employs photographs in interviews. i asked the research participants to complete a short, independent photo survey followed by a one-on-one interview to discuss the photos they made. for the photo survey activity, i asked participants to walk through the oise library and complete tasks that they might carry out on a trip to the library, including locating books (see appendix a oise library participant photo survey tasks). i asked them to photographically document their efforts and decisions along the way. i reinforced that the intent of the tasks was not to test their ability to locate library materials, but simply to get them moving through the library space. whether or not they located the items was not important.   once participants had completed the photo survey tasks, we met for an interview to discuss their photos and unpack their experiences. the timing of interviews immediately following the photo survey meant that participants could easily recall the intention of most of the photos, and their feelings and experiences were still fresh in their minds. the interviews themselves were very loosely structured, which allowed discussion to emerge from the photos. after several introductory questions, most of the interview questions were quite broad, and were driven by the participants’ photos: “why did you take this one?” or “what’s happening here?” (see appendix b oise library interviewer’s guide). this allowed the discussion to move beyond the description of the photograph (“this is the stairwell”) and start in-depth discussions about participants’ experiences with specific objects and spaces.   my primary aim in using this style of structured pdpe for the oise library study was to focus on navigation within the library. although i did provide specific tasks for participants to complete, i wanted to ensure, as much as possible, that the data meaningfully reflected their experiences in the space. in other words, i wanted to be able to focus on the decisions they made, the photos they decided to take, and their explanations of what was important in this exercise. this can be contrasted with methods where researchers are present (e.g., haberl & wortman, 2012) or where video cameras are used to document participants’ every move (e.g., kinsley, schoonover, & spitler, 2016). by handing them the camera and allowing them to work through the tasks and space on their own, i was asking participants to independently decide what was important, what they wanted to photograph, to show and talk about.   university of victoria   context   at the university of victoria libraries, we recently received some funding to explore potential physical changes to all three campus libraries, with the help of external consultants and architects. the interest in implementing a research study using photo-narrative was to generate data on how students use the library space as they are using it and what they think about the space and design and how that impacts their experience. as mentioned, photo-narrative is a type of photo-elicitation and differs only in the final presentation of results where the photos and interview are used together to create a narrative of telling of the experience. also, a photo-narrative approach lends itself to include an exhibition component. although there are three libraries on campus, the photo-narrative study focused on the mearns centre for learning/mcpherson library, the main library. the reason for this limited scope was to make this research project more manageable.   research questions   for this particular study, the main research questions included:   how are students using the library space? how do they shape or re-shape the spaces? what type of learning is going on in that library space? what is missing from the space and design that might impact their learning or general experience of the space?   these questions served to guide the project.   recruitment   aiming to generate a broad set of student experiences through the photo-narrative study, all current undergraduate and graduate students were eligible to participate in the study. after gaining human research ethics approval, i employed print and virtual promotional posters using the slogan “let your photos tell the story” (appendix c university of victoria promotional poster). in addition, i sent emails to all department secretaries on campus asking them to put out a recruitment email on their student listservs. my goal was to recruit between 10 to 15 students who use and experience the library space on a regularly basis. student research participants were not required to be professional photographers. also, no incentive was offered other than an enlarged image of a student photograph that would be mounted as a thank you for their participation. recruitment began in september 2016, and 10 students took part in the study between september and december 2016.   method   as students expressed interest to participate, i asked them to meet with me briefly to review the research project, to sign a participant consent form, and to review the ethics and privacy issues if taking photos that might include other students. i also clarified with the students that they would keep the intellectual property for their images and that they could choose to keep their name attached to their photos. i provided the student research participants with lanyard tags that identified them as student research participants and encouraged them to spend some time collecting photos that represented their experience and use of library space and design. although most students opted to use their smart phone cameras, some expressed interest in using a higher resolution camera that they either owned or else borrowed from the university of victoria libraries’ music and media unit. once student participants had completed collecting their photos, i asked them to meet for a semi-structured interview where we would review their top 10-12 images that represented their experiences (see appendix d university of victoria interview questions).   another component of this photo-narrative research project includes an opportunity for participants to co-curate an exhibition in 2017 with me, featuring select photos from each participant with an opportunity for viewers of the exhibit to leave comments. the comments collected in the guestbook will be part of the overall data collection for this photo-narrative study. exhibiting as a method of inquiry is occasionally used in combination with photo-narrative, not just as a method of research  dissemination, but also to serve the purpose of data collection. in this way, exhibition as a method of inquiry has the potential to strengthen research participants’ connections to other viewers and their environment (gubrium & holstein, 2012). gathering viewer input is focused on the shared experiences reflected in the exhibited photos, and not on the quality of the image. research participants were asked to select the images they would like included in an exhibit and could elect to remain anonymous in the display. the exhibition component of the research was explained to all participants at the point of recruitment, and consent to participate in the research clearly highlighted all components of the research process.   results   oise library   participants’ photographs and the subsequent interviews for the oise library wayfinding project yielded sophisticated descriptions of their experiences navigating the library space. between january 2015 and january 2016, a total of 17 participants made 533 photographs, ranging from 4 photos from one participant to 75 from another. the follow-up interviews yielded 536 minutes of interview recordings, with interviews ranging from 20-44 minutes.   although the data analysis is not yet complete, initial results point to some key areas where signage can be improved to help make the journey through the library easier for users. in the first stage of analysis i examined the recorded interviews and the accompanying photographs, listening for mentions of things related to the photo tasks i provided. i did not code the photographs separately from the interview transcripts. rather, they are stored together in nvivo and analyzed as part of the same dataset. to address my research questions, i focused on gaining insights into the successes and challenges participants faced in navigating the library, noting any suggestions or recommendations they made. while the interviews reflected a wide range of experiences and suggestions, three broad themes emerged from this first phase that have proven valuable for recommendations for improvements. they include: (1) the overall layout of the library; (2) the consistency of directional prompts, including naming conventions and the visual consistency across collections, signage, call numbers labeling, and catalogue records; and (3) the terminology used for directional cues in the space. for the purposes of this article, i will briefly discuss how photo-elicitation helped shed light on problems with the consistency of directional prompts as well as the signage terminology at the oise library.   many participants described the process of locating items as connecting “clues” (or directional prompts) they encountered over the course of their journeys. they described observing clues in places like call numbers, signage, the names of collections, or by the titles of the books, and then making a guess about their next steps. many participants remarked that these clues did not always lead clearly to the next step in ways they expected. one participant provided a brilliant example of where inconsistent naming across signs, call numbers, and the catalogue record caused a temporary barrier in her search for the second book on the list.   the first photograph in figure 1, taken near the entrance to the children’s literature collection, prompted an in-depth discussion about the inconsistent directional prompts participant 6 encountered. for example, she pointed out that the book’s call number included the letters “juv fic,” but the catalogue record indicated that the item was located in the “children’s literature” collection, not the juvenile fiction collection. in the section itself, there are signs that read “juvenile fiction” and “the margo sandor collection,” as well as labels that read “children’s literature collection (clc)” but at the time there was no sign that clearly indicated she had arrived in the children’s literature collection area. ultimately, she said, “i just wasn’t sure what to trust” (participant 6).       figure 1 participant 6’s photo of the children’s literature collection in the oise library, and the same photo as annotated by the researcher after the interview.     in several other interviews, participants pointed to terminology on key library signage and made comments that challenge what we often take for granted when describing library spaces and collections. for example, figure 2 sparked a frank discussion with one student about the term “stacks”:   so, i saw that stacks was on the second floor, so i went up and then i got lost and i wasn’t sure where i was anymore. and to be honest [pointing to above photo], i don’t know what stacks means…. and then i felt silly, i didn’t want to ask cause i thought that was a stupid question [participant 7].   as library insiders, we know that there are gaps in the trail of clues our users attempt to follow. the photo-elicitation data, tied to specific places and particular items in the oise library, provided detailed insights into a library outsider’s journey. participants' photographs, coupled with their thoughtful discussions about library environments, provide an alternate view that can re-open our eyes to things like signs and even common terms such as “stacks” that have become second nature to those of us who work in libraries. as weber (2008) notes, participants’ photographs can make us pay attention in new ways. viewing my own library space in new ways allowed me to pinpoint specific problems, such as inconsistencies in signage, or problematic library terms, and to make suggestions for improvements.   in addition to providing evidence to support improvements to the oise library’s signage, my initial analysis has revealed new and unexpected insights into aspects of users’ library experiences that went beyond my research questions. as a result, i plan to review the data to explore additional themes that emerged around student-library relationships. the initial analysis has already revealed some of the complexities about how students inhabit library spaces, including how they work together (or don’t) to develop etiquettes to share space and resources, the connections and ownership they feel with the particular locations and items in the library, how they work around library policies and processes to accomplish what they need, as well as the things make them anxious, and the things make them happy.     figure 2 participant 7’s photo showing a sign for the stacks at the oise library.     figure three shows a photo made by participant 1 to capture their favourite study spot. the participant explained that the combination of natural light and electrical outlets made this place “prime real estate.” the photo also led to a long discussion about the use of library spaces for events, the importance of quiet study spaces, and the sense of ownership students feel for their favourite library places. i hope the second phase of analysis will reveal more examples like this and open the door to potential new research questions regarding the student culture in library spaces.   university of victoria libraries   the research project is not yet complete at the university of victoria. although the data collection from the student research participants has been completed, along with the accompanying interviews, and the interview data has been analyzed, the exhibition of photos is still forthcoming, and scheduled for late spring 2017. since the guestbook comments are considered part of the data collection for this study, the results are therefore incomplete. however, i am able to share some emerging trends and themes from the photo collection and interviews with participants.   of the 10 participants, 6 students are graduate students, 3 in master’s programs and 3 in doctoral programs. the remaining 4 are undergraduate students. also worthy of note is that 6 of the 10 participants identified as international students. all but 3 agreed in having their name identified with the images they had taken, while the others will be identified with pseudonyms for the exhibition of photos and in any publications that include samples of the photos from the study. in total, i recorded 314 minutes of interview time from 10 participants. the collection of photos exceeded the 10 to 12 images i requested from each participant. the photos were not coded separately from the interview. rather, the themes emerged as part of the discussion with the research participant that included their photos. this is an important part of a visual research method, in that themes are not generated from the perspective of the researcher but are co-constructed with the participant and the researcher.   from reviewing the photos and interview data with each participant, the preliminary themes include furniture, technology, lighting, artwork, and group learning space. within these themes there was much discussion about how each aspect was working within each category, and also how each could be improved to make the student experience even better. although there is much to share and highlight from the results, i will limit the discussion to only two of the themes: furniture and lighting.     figure 3 participant 1’s photo showing their favourite study spot in the oise library.   the photos that students took of the furniture, and as demonstrated through their interview discussion, highlighted a strong appreciation of the variety of furniture available to them, including large comfy chairs or sofas, individual study carrels, large desks in the learning commons workstations, big open table spaces, and the ever popular person-shaped bouloum lounge chairs. although this variety was much appreciated, several students took images of how worn-out some of the furniture fabric has become, making them less appealing. several commented on their reluctance to sit in such spaces, but they often had no choice because the library very busy and full. one participant commented, “…most days you are lucky to even get a seat, so you just take what you can get. really many students are sitting on the floor between stacks…” this comment also pointed to another aspect of the furniture theme, which is that we simply do not have enough furniture to meet student needs. this was conveyed through photos that also highlighted students spread out on the floor with their laptops, books, coats, and backpacks.   the theme around the lighting also had many equally positive reflections, including areas in the library where lighting could be improved. several students took images of the large windows facing west that are almost floor to ceiling and look out on the grassy quads and water fountain (see figure 4).   students’ comments about such images were overflowing with praise about the abundance of natural light. one international student commented,   …coming from china and my experience with my undergraduate library, we had very few windows. the lighting was almost always fluorescent tubes. i feel my day is lucky when i have the opportunity to sit at one of the large windows to study and…enjoy the view of such a beautiful campus.yet there were also many photos of areas in the library that are dark and ominous (see figure 5).     figure 4 participant’s photo of natural light at uvic mcpherson library.     figure 5 participant’s photo of a dark corner of uvic mcpherson library.     one student mentioned that as a graduate student they   have an assigned carrel in one of the darkest and most closed off spaces in the library. i really appreciate the carrel but find i move to another carrel next to a window if it is not being used. i don’t like feeling like [i] am in a cave, especially when i have hours ahead of me working on a laptop and reading and composing.   meanwhile, another graduate student added,   if i can get a carrel by the window, i find i stay longer to do my work…on the days, i can only get a study space in the dark areas of the library, i don’t find i am as productive and i have a tendency to leave before i am done what i planned.   the study suggests that the quality of the lighting in the library impacts how long a student will stay in the library and how much work they might complete.   discussion   strengths   one of the commonly reported strengths of methods such as photo-elicitation or photo-narrative is that the power of data collection is shifted, in part, into the hands of the research participant (liebenberg, 2009; pink, 2001; rose, 2012; schwartz, 1989). in the two studies presented here, participants’ photographs acted as the main prompts during the interview, and this, in turn, allowed the research participants drive more of the conversation. in this context, the research participant is placed in an active role in co-constructing knowledge with the researcher. harper (2002) describes the collaborative work inspired by photo-elicitation well, noting “when two or more people discuss the meaning of photographs they try to figure out something together” (harper, 2002, p. 23). furthermore, the opportunity to incorporate an exhibition aspect with a visual research method (as in the university of victoria study), allows for more voices to be included in the data-gathering phase of the research (gubrium & holstein, 2012).   another key strength of photographic methods is their applicability to spatial research. the methods used in the two studies presented here allowed researchers to follow lines of questioning with participants that tied participants’ experiences directly to particular locations, objects, signs, furniture, etc. we are interested in how students’ experiences are interwoven with particular spaces and objects, and the photographs served as visual queues that prompted space-specific discussions that would be lost in other research methods. for example, the photograph in figure 1 prompted a description of various signs visible from a particular location in the library. then, building on the description of the space, the participant’s comments broadened into a conversation about the difficulties that inconsistent cataloguing, signage, and labeling systems cause for library users. this type of detailed spatial reference could not be elicited through methods such as focus groups or surveys.   some of the key benefits of visual images for library research are also very practical. for instance, both researchers found that working with images facilitated the interview process and established a level of comfort between the researcher and the participant. this is in line with collier’s (2001) comment that photographs can be seen as an “ice-breaker,” a medium that creates a comfortable space for discussion.   learning and recommendations   as with any research project, we identified things we would do differently next time as researchers. one key challenge we both experienced using photographic methods was with the large amount of data that was collected. at the university of victoria, despite the criteria to the research-participants to bring only their top 10-12 photos to the interview with the researcher, students wanted to share many more photos than that. at times, this became quite overwhelming in guiding the students to be more analytical about their images and experience, since many students are accustomed to taking copious amounts of photos with today’s technology and image-rich culture, perpetrated by the internet and social media. for those considering a similar approach, it might make sense request a budget to buy disposable cameras with a finite number of exposures that would ensure the same number and quality of images among research participants.   at the oise library, participants also created a lot of photos, 533 in total. because of the “journey” style process, participants were documenting as they completed the photo tasks, so allowing them to make an unlimited number of images worked well. these images also flowed easily at the interview stage and really shaped the telling of participants’ stories. however, because of the number of participants involved, data analysis was labour intensive. for researchers planning to use a data collection method such as pdpe, the number of participants is also an important consideration. as key patterns and themes begin to emerge in the interview process, consider whether more data is needed to address the questions at hand. for the oise library study, 10 participants would likely have provided the insights needed to address the research questions.   a significant and unexpected outcome of this project relates to the contributions of international students. when international students at the university of victoria were asked why they were interested in the study, many spoke about the comfort they had in the image-based nature of this research project. for many of the international students, english was an additional language, but with the focus of the study on photographs they felt there was a common language between them and the researcher, especially during the interview phase. similarly, one international student participant at the oise library also pointed to the potential of photographs for research with international students. although the participant expressed concern about her english not being very good, she also explained that she was excited for the chance to engage in research in a way that allowed her to articulate her ideas through reference to her photographs.   international students comprise a diverse group of users that have traditionally been on the periphery in terms of engagement with library research projects at many institutions. the interviews with international students in our two studies suggest that they feel a positive connection to photographic research methods because the use of images created an inclusive method to facilitate participation by a diverse community of users. this outcome, while unexpected, is consistent with julien, given, & opryshko’s 2013 article that draws on feminist theory and puts forward photographic methods as a way to highlight the voices of marginalized communities. this outcome also inspires the need for more careful thought around the theoretical frames that are associated with visual research methods, including freire’s (1970) foundational work in critical education, which aimed to empower disadvantaged or marginalized communities, as well as the work of visual researchers such as wang & burris (1994), who drew on freire’s work and feminist theory to develop the photovoice method as a research tool for bringing voices to marginalized groups. the theoretical underpinnings as well as the potential benefits of photographic methods for international students and other marginalized student populations are areas for further exploration.   conclusions   preliminary findings from the two studies presented support trends in the lis literature that point to the value of photographic methods in library research. we feel that photographic methods have a strong role to play in understanding how spaces and objects shape user experiences. additionally, we found that photographic methods are well suited for providing unexpected insights and engaging participants in meaningful discussions about libraries. although as researchers we set the criteria and parameters of the projects and developed the photographic tasks and interview questions, the fact that our participants were moving through the spaces by themselves and deciding what to photograph led to many moments of realization for us during the interviews. whether it was the discovery of a long forgotten (and misleading) directional sign, a personal admission that not understanding library terminology was embarrassing, or an in-depth discussion about a student’s favourite place in the library, these unexpected lines of discussion provided fresh perspectives on the spaces we take for granted.   given that many libraries are focusing their efforts on becoming user-centered learning environments, it is critically important for librarians to continue to ask research questions that help solve the needs of our users in our physical spaces and to promote better physical access. expanding our research methods allows us to reach our users in different ways, and to promote better engagement with them, and ultimately gain additional, and perhaps even more meaningful, data. the photo-elicitation data presented here has already proven productive in generating answers to our research questions and supporting practical improvements to the library. additionally, participants’ willingness to describe the intentions of their photographs and engage in in-depth conversations about libraries led to many unexpected insights for us. in fact, it is in these moments (our “aha” moments) when we learn something completely new about how users experience our libraries, that we enjoy this research the most.   acknowledgments   this paper was originally presented at the 2016 centre for evidence based library & information practice (c-eblip) symposium at the university of saskatchewan. the authors would like to thank virginia wilson for her amazing work at c-eblip and for connecting our two research projects.   references   briden, j. (2007). photo surveys: eliciting more than you knew to ask for. in s. gibbons, & n. f. foster, (eds.). studying students: the undergraduate research project at the university of rochester (pp. 40-47). chicago, il: association of college and research libraries.   brown, m. b., & lippincott, j. k. (2003). learning spaces: more than meets the eye. educausereview, 1, 14-16.   collier, j. (2001). approaches to analysis in visual anthropology. in t. van leeuwen and c. jewitt (eds.) handbook of visual analysis. (pp. 35-60). london, uk: sage.   ferria, a., gallagher, b. t., izenstark, a., larsen, p., lemeur, k., mccarthy, c. a., mongeau, d. (2017). what are they doing anyway?: library as place and student use of a university library. evidence based library and information practice, 12(1), 18-33. http://dx.doi.org/10.18438/b83d0t   foster, n. f., & gibbons, s. (2007). studying students: the undergraduate research project at the university of rochester. chicago, il: association of college and research libraries.   freire, p. (1970). pedagogy of the oppressed. new york, ny: continuum publishing.   gubrium, j.f., & holstein, j.a. (2012). narrative practice and the transformation of interview subjectivity. in j. f. gubrium, j.a. holstein, a. b. marvasti, & k.d. mckinney, 27-43. , (eds.). the sage handbook of interview research: the complexity of the craft. 2nd ed. (pp. 27-43). thousand oaks, ca: sage.   haberl, v., & wortman, b. (2012). getting the picture: interviews and photo elicitation at edmonton public library. libres: library and information science research electronic journal, 22(2), 1-20. http://www.libres-ejournal.info/   halpern, r., eaker, c., jackson, j. & bouquin, d. (2015). #ditchthesurvey: expanding methodological diversity in lis research. in the library with the lead pipe. retrieved from http://www.inthelibrarywiththeleadpipe.org/2015/ditchthesurvey-expanding-methodological-diversity-in-lis-research/   harper, d. (2002). talking about pictures: a case for photo elicitation. visual studies, 17(1), 13-26. http://dx.doi.org/10.1080/14725860220137345   harris, c. (2007). libraries with lattes: the new third place. australasian public libraries and information services, 20(4), 145-152.   hartel, j., & thomson, l. (2011). visual approaches and photography for the study of immediate information space. journal of the association for information science and technology, 62(11), 2214-2224. http://dx.doi.org/10.1002/asi.21618   julien, h., given, l. m., & opryshko, a. (2013). photovoice: a promising method for studies of individuals’ information practices. library & information science research, 35(4), 257–263. http://dx.doi.org/10.1016/j.lisr.2013.04.004   kinsley, k. m., schoonover, d., & spitler, j. (2016). gopro as an ethnographic tool: a wayfinding study in an academic library. journal of access services, 13(1), 7-23. http://dx.doi.org/10.1080/15367967.2016.1154465   liebenberg, l. (2009). the visual image as discussion point: increasing validity in boundary crossing research. qualitative research, 9(4), 441-467. http://dx.doi.org/10.1177/1468794109337877   lin, y-c., & chiu, m-h. (2012). a study of college students’ preference of servicescape in academic libraries. journal of educational media & library sciences, 49(4), 630-636. http://joemls.dils.tku.edu.tw/index.php?lang=en   montgomery, s. e., & miller, j. (2011). the third place: the library as collaborative and community space in a time of fiscal restraint. college & undergraduate libraries, 18(2-3), 228-238.   neurohr, k. a., & bailey, l. e. (2016). using photo-elicitation with native american students to explore perceptions of the physical library. evidence based library and information practice, 11(2), 56-73. http://dx.doi.org/10.18438/b8d629   newcomer, n. l., lindahl, d., & harriman, s. a. (2016). picture the music: performing arts library planning with photo elicitation. music reference services quarterly, 19(1), 18–62. http://dx.doi.org/10.1080/10588167.2015.1130575   pink, s. (2001). doing visual ethnography: images, media and representation in research. london: sage.   pollak, a. (2017). visual research in lis: complementary and alternative methods. library and information science research, 39(2), 98-106. http://dx.doi.org/10.1016/j.lisr.2017.04.002   rose, g. (2012). visual methodologies: an introduction to researching with visual materials (3rd ed.). london: sage publications.   schwartz, d. (1989). visual ethnography: using photography in qualitative research. qualitative sociology, 12(2), 119-154. http://dx.doi.org/10.1007/bf00988995   treadwell, j., binder, a., & tagge, n. (2012). seeing ourselves as others see us: library spaces through student eyes. in l. m. duke, & a. d. asher, (eds.). college libraries and student culture: what we now know (pp. 127-142). chicago, il: american library association.   wang, c., & burris, m.a. (1994). empowerment through photo novella: portraits of participation. health education quarterly, 21(2), 171–186. http://dx.doi.org/10.1177/109019819402100204   weber, s. (2008). visual images in research. in j. g. knowles & a. l. cole (eds.). handbook of the arts in qualitative research: perspectives, methodologies, examples, and issues, (pp. 42-54).  london: sage publications.     appendix a oise library participant photo survey tasks   library a photo survey participant instructions   instructions: please use the following tasks to guide your visit to the library and take photos along the way. please return with the ipad to the oise library service desk when you have completed the tasks. have fun!     task 1: locate the following item:   antler, j., & biklen, s. k. (1990). changing education: women as radicals and conservators. albany: state university of new york press. (call number 305.420973 c4562)   please take photos along the way. consider things like:   signage locations and layout tools you use, actions you take things that helped as well as problems/barriers     task 2: make your way to the following item:   cleary, b., & tiegreen, a. (1981). ramona quimby, age 8. new york: dell. (call number juv fic c623rq)   please take photos alone the way. consider things like:   signage locations and layout tools you use, actions you take things that helped as well as problems/barriers     task 3:  anywhere in the library, take photographs of the following: one or more places in the library where you felt lost something that helped or assisted you something you really like something you really dislike  anything else that you take note of (good or bad!)   appendix b oise library interviewer’s guide   thank the student again for participating in the study.  review the purpose of the study and the participant’s own participation. review the participant’s consent options, withdrawal options, compensation options, ways the data will be stored and used, and the reason for the project.   we are interested in what students really do in the library, how they locate information, and what types of useful guides and barriers might exist in the oise library. we’ll be talking about the photographs you took last week and i’ll be recording this session.   a few questions to get started:   prior to this project, how often had you used the oise library? how would you describe your overall experience navigating the oise library?   following these initial questions, the interview will be guided by going through the participants’ photographs and associated tasks. questions will be open-ended and will seek to elicit descriptions related to understanding the actions of participants and how they navigated the library space. for example:   this photograph looks like it is associated with task #1 from the photo survey list. tell me about what’s happening here… why did you take this one? where did you go next? what did you do next?   once all the photographs have been examined, the pi will ask the following questions:   1.        what was your least favorite activity in the photo tasks? why? 2.        what are the key things you would change to improve the oise library experience? 3.        next time you have to locate something in the oise library, what would you do? would you try anything different? 4.        do you have any other suggestions, thoughts, or questions?   following completion of the interview, the pi will thank the participant again, sign off on completion of participation and provide the incentive funds. the pi will ask whether/how the student would like to be contacted with follow-up about the research project and whether they would be interested in continuing to participate on providing input to the oise library on service improvements.   appendix c university of victoria promotional poster     appendix d university of victoria interview questions   interview questions research project: student experience of library space told through student photo-narratives   this interview questions used with research participants in a semi-structured interview to discuss their top 10 to 12 photos. tell me about the photographs you have selected. why have you selected these images? if you were to describe your images, what would you say about them? how do these images resonate with your experiences of library space? what struck you as particularly interesting or concerning about the space? did you learn something about your use of space you were not aware of before this project? if so, what? from this experience, what are the top 3 things that library could do to improve your experience of library space?   for the photography exhibit, what impression about student use of library space would you like viewers to walk away with? and why?     research article   teaching systematic searching methods to public health graduate students: repeated library instruction sessions correlate with better assignment scores   john pell assistant librarian social work & public health library hunter college new york city, new york, united states of america email: jpell@hunter.cuny.edu   received: 20 oct. 2016   accepted: 11 mar. 2017       2017 pell. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – the objective of this study was to devise an assessment plan to determine if repeat attendance at two library instruction sessions is statistically associated with overall assignment scores or specific assignment qualities.   methods – the author used spss to calculate correlations between attendance and assignment scores and cross tabulations between attendance and assignment item analysis scores.   results – repeat attendance at two library instruction sessions was statistically associated with higher overall assignment scores and higher scores on specific assignment sections. the effect is statistically significant.   conclusion – students who attended two library instruction sessions applied skills and concepts practiced in those sessions on a subsequent research assignment. not all skills and concepts practiced in the session were applied. acquisition of more technical skills such as boolean searching may require a greater number of follow-up sessions.   introduction   systematic reviews are a high quality form of evidence in fields concerned with evidence based practice. systematic reviews are at the peak of “evidence pyramid” models that rank evidence quality. numerous reports on research agendas emphasize the importance of consulting and carrying out more systematic reviews (hawke, burns, & landorf, 2009; howes, doyle, jackson, & waters, 2004; kite, indig, mihrshahi, milat, & bauman, 2015; stewart, 1996; szajewska, 2013; whelan, 2014). the attention paid to systematic reviews has proven something of a boon to librarians since completion of a systematic review requires expertise in literature searching. prominent “gold standard” manuals of systemic review methodologies such as the cochrane handbook and the institute of medicine standards have thrown a spotlight on the search expertise of librarians by recommending teaming up with librarians to carry out a systematic search (higgins & green, 2011; research & medicine, 2011). this emphasis on librarian involvement in systematic review teams has been validated by research into the quality of systematic reviews. this research has shown that having a librarian co-author on a systematic review correlates with higher quality systematic review methodologies (rethlefsen, farrell, osterhaus trzasko, & brigham, 2015).   considering this background, there are several good reasons for librarians who support students and researcher in evidence based fields to promote librarian-led training in systematic searching methods to graduate students. it promotes the literature searching expertise of librarians to students and faculty, it can prepare students for a position as a research assistant, and students can apply what they learn from the training to subsequent research assignments.    this paper is specifically concerned with demonstrating, that under the right circumstances, students can learn and apply systematic searching skills to successfully complete research assignments.   aims   this paper describes an assessment method to test the following questions:   1. are library research assignment scores correlated with other assignment scores? 2. is attendance at library instruction sessions associated with better assignment scores? 3. what assignment characteristics are associated with attendance at library instruction sessions?   this paper will present the results of an assessment plan developed to answer these questions using data compiled by instructors of a cohort of public health graduate students.   literature review   there is a robust body of published assessments of librarian-led training in literature searching for medical students. there are comparatively fewer examples of assessments of learning outcomes from literature search training for graduate students in other fields. there are even fewer examples of assessments of training in systematic search methods for students outside of professional development programs for librarians (conte et al., 2015). there is at least one example of an effort to teach systematic searching to undergraduate nursing students that shows improved evidence summary outcomes as a result (whalen & zentz, 2015). the literature on database training for medical students suggests that librarian-led trainings can be effective at improving the evidence based literature searching skills of medical students. the literature reveals that much of this evidence, pointing to a positive effect, comes from studies with weak designs (garg & turtle, 2003; just, 2012; maggio & kung, 2014). a rigorously designed double blind clinical trial did not find any effect on medical student search skills following a single training session (ilic, tepper, & misso, 2012). however, maggio and kung propose that this null effect could be consistent with a paradigm in which longitudinal designs and follow-up training sessions are required for effective retention of skills and knowledge (2014).    effects associated with librarian-led trainings in literature searching for medical students can include increased confidence and use of demonstrated resources (miller, 2014; rafferty, 2013). however, some studies offer conflicting findings. for instance, training does not always increase confidence, it can also raise awareness of the complexity of expert literature searching and increase requests for librarian assisted searches (addison, glover, & thornton, 2010).    much of the assessment literature on librarian involvement with graduate students in fields outside of medicine focuses on needs assessments and student preferences for topics and mode of instruction. the results of these assessments emphasize the need for development of subject-specific content (baruzzi & calcagno, 2015; critz et al., 2012; fong, wang, white, & tipton, 2016; o’malley & delwiche, 2012; roszkowski & reynolds, 2013; tomaszewski, 2012). the study i describe in this paper is unique in terms of content area, method of assessment, and student population.   methods   students in a graduate-level public health course (n = 68) had the opportunity to attend two optional two-hour instruction sessions that supported a required library research assignment. forty-three students attended both sessions. twenty-five students either attended one session or did not attend any sessions. this study compares the performance of the forty-three students who attended both sessions to the performance of the twenty-five students with incomplete attendance.   the instruction sessions were led by faculty librarians with american library association-accredited master of library science or master of library & information science degrees. they provided the students with active learning exercises in stating research questions in a population/intervention/comparison/outcome (pico) format, looking up medical subject heading (mesh) terms related to the concepts in the research question, developing a boolean-based search strategy that includes keywords and subject heading combinations, identifying literature reviews in pubmed that relate to the research question and hand searching bibliographies for relevant studies, selecting databases other than pubmed/medline according to the need of the research question, documenting manual search criteria, and using citation management software to format in-text citations and lists of works cited in ama style. these exercises targeted the same set of skills that the library research assignment required.   students submitted the completed assignments to the librarian instructors, who then evaluated the assignments with a rubric. librarian instructors received training in the use of the rubric through norming sessions intended to ensure that the librarians applied the rubric consistently.   the data set for the assessment consists of collected attendance notes from the library instruction sessions, completed grading rubrics from the library research assignment, and scores on another assignment submitted in the same class.    the author used spss to calculate a pearson correlation between the library assignment scores and scores on another research assignment given in the same course. the purpose of the other assignment was to deliver a public health brief, which is a summary of the current research relating to an assigned topic. the author calculated chi-square cross tabulations in spss between attendance at both sessions and assignment score, defining “higher score” as a score above the median score for the cohort.   an institutional review board reviewed the protocol for data analysis for this project and determined that it qualified for an exemption from review of human subjects as a research study of existing data, documents, or records.   results   the author found that a pearson correlation showed library research assignment grades were substantially (r = .534) correlated to grades on the other individual research assignment given in the same course. this relationship is statistically significant (p<.01).   the author found that a chi-square test showed students who attended both library instruction sessions were likely to score above the median assignment score. this relationship is statistically significant (p<.01).   chi-square tests showed that attendance at both library instruction sessions had a statistically significant association with retrieval of a literature review related to the subject of the student’s research (p = 0.005), pico structured research questions (p = 0.006), and clear statements of inclusion and exclusion criteria with a logical relationship to the research question (p = 0.01). these three assignment characteristics showed the strongest statistical association with attendance at both library instruction sessions.   attendance at both library instruction sessions was also statistically associated with an accurate summary of primary sources and correct use of ama citation style (p = 0.03) and, to a lesser extent, use of controlled vocabulary and boolean logic (p = 0.07). however, these associations, though suggestive, were not statistically significant.   discussion   the training sessions described in this study taught students how to systematically search the public health literature. they were not training sessions on how to do systematic reviews. while the sessions introduced the concept of a systematic review and required students to practice some of the skills involved in producing systemic reviews, the training sessions did not cover a comprehensive array of the skills and knowledge required to carry out a full systematic review. grey literature, clinical trial registries, and publication bias are just a few of the systematic review search skills and concepts left out of the training sessions in this study. the objective of the sessions was to develop skills and knowledge that could serve as a scaffold for further development of more sophisticated search skills. the objective of these sessions was not to produce students capable of conducting systematic reviews. given that comprehensive systematic review courses can take 24 – 36 hours to complete (johns hopkins university, 2017) it would seem inadvisable to attempt to introduce a full array of systematic review concepts and skills in the short sessions assessed in this study. the results of this study suggest some possible limits to what can be accomplished in four hours, especially when training students with little to no prior knowledge about systematic reviews.   one of the most statistically significant associations with attendance at the instruction sessions concerns the retrieval of a literature review. a high percentage of students who did not attend both library instructions failed to submit literature reviews related to their research questions. when interpreting this result, it is important to remember that students may be able to identify literature reviews on a topic but may not have the skills required to efficiently search for and access relevant literature reviews for an assignment. one strategy is to teach these students how to use the “publication type” field in pubmed. this could significantly improve retrieval skills, as it did for the cohort that attended both sessions.   in contrast to the students’ improved abilities in the areas of literature searching, search criteria, and pico question formulation, students’ performance on the boolean searching tasks in the library research assignment did not show a statistically significant association with attendance at the library instruction sessions, despite strong emphasis on boolean searching. it may be that boolean coding skills were too far removed from the experience of this cohort to be significantly absorbed after two instruction sessions.   this study has some limitations. the data was gathered from a convenience sample of graduate students enrolled in a public health class and students self-selected into the library instruction sessions. it is important to bear these limitations in mind when considering the question of whether the library instruction sessions were a causal factor in the achievement of a higher score on the library assignment or the other individual research assignment in the course.   this study was not designed to test the impact of variation in instructor skills and experience on student outcomes. the training sessions employed a student-centered, active-learning pedagogy intended to mitigate for individual differences among instructors and their skills and experiences. students who attended two sessions often had different instructors for each session. despite these measures, the fact remains that this study did not collect data on variations in instructor skills and experiences; therefore, it cannot conclusively resolve questions about the impact of individual instructors on student outcomes.   individual variation may have also affected the rubric scores. although instructors received training intended to standardize their use of the rubric, this training cannot guarantee the elimination of instructor disagreement about rubric scores. since the instructors each scored non-overlapping segments of the sample, it is not possible to quantify the inter-rater reliability for the instructors who contributed scores to this study.   the correlation between repeated attendance at library instruction sessions and higher assignment scores may show the transferability of skills and concepts acquired in the library session. on the other hand, this correlation may merely show that the best students showed up for both library training sessions. taken alone, the results of the pearson correlation and the cross tabulations relating voluntary, repeated library instruction attendance to assignment scores could be an artifact of a comparison between students with sufficient time and motivation to attend additional instruction sessions and students without such resources. however, the cross tabulations relating library instruction attendance to performance on different sections of the library research assignment provide evidence to suggest that students who attended two instructional sessions retained and applied specific skills from those sessions (retrieval of literature reviews, documentation of manual search criteria, and pico formatting of research questions) on the subsequent assignment. when these results are considered together, a stronger case emerges for the causal impact of repeated library instruction sessions on assignment scores. the item analysis of the assignments strongly suggests that students who attended two training sessions retained and applied specific content from those sessions on a subsequent assignment.   further studies with graduate public health students are needed to describe best practices for curriculum plans that ensure appropriately repeated training and exercise in library research methods with this student population.   conclusions   the limited success of the two-session plan may be taken as evidence of the importance of repeated instruction sessions addressing systematic search skills for graduate students. more technical skills such as boolean searching or the use of citation management software in conjunction with ama formatting may require more follow-up sessions to increase student performance.   references   addison, j., glover, s. w., & thornton, c. (2010). the impact of information skills training on independent literature searching activity and requests for mediated literature searches. health information and libraries journal, 27(3), 191–197. http://doi.org/10.1111/j.1471-1842.2009.00871.x   baruzzi, a., & calcagno, t. (2015). academic librarians and graduate students: an exploratory study. portal: libraries and the academy, 15(3), 393–407. http://doi.org/10.1353/pla.2015.0034   conte, m., maceachern, m., mani, n., townsend, w., smith, j., masters, c., & kelley, c. (2015). flipping the classroom to teach systematic reviews: the development of a continuing education course for librarians. journal of the medical library association, 103(2), 69–73. http://doi.org/10.3163/1536-5050.103.2.002   critz, l., axford, m., baer, w. m., doty, c., lowe, h., & renfro, c. (2012). development of the graduate library user education series. reference services review, 40(4), 530–542. http://doi.org/10.1108/00907321211277341   fong, b. l., wang, m., white, k., & tipton, r. (2016). assessing and serving the workshop needs of graduate students. the journal of academic librarianship, 42(5), 569–580. http://doi.org/10.1016/j.acalib.2016.06.003   garg, a., & turtle, k. m. (2003). effectiveness of training health professionals in literature search skills using electronic health databases--a critical appraisal. health information and libraries journal, 20(1), 33–41. http://doi.org/10.1046/j.1471-1842.2003.00416.x   hawke, f., burns, j., & landorf, k. b. (2009). evidence-based podiatric medicine: importance of systematic reviews in clinical practice. journal of the american podiatric medical association, 99(3), 260–266. http://dx.doi.org/10.7547/0980260   higgins, j., & green, s. (eds.). (2011). cochrane handbook for systematic reviews of interventions (version 5.1.0). the cochrane collaboration. retrieved from http://www.cochrane-handbook.org   howes, f., doyle, j., jackson, n., & waters, e. (2004). evidence-based public health: the importance of finding “difficult to locate” public health and health promotion intervention studies for systematic reviews. journal of public health (oxford, england), 26(1), 101–104. http://doi.org/10.1093/pubmed/fdh119   ilic, d., tepper, k., & misso, m. (2012). teaching evidence-based medicine literature searching skills to medical students during the clinical years: a randomized controlled trial. journal of the medical library association: jmla, 100(3), 190–196. http://doi.org/10.3163/1536-5050.100.3.009   johns hopkins university. (2017). introduction to systematic review and meta-analysis. retrieved from https://www.coursera.org/learn/systematic-review   just, m. l. (2012). is literature search training for medical students and residents effective? a literature review. journal of the medical library association: jmla, 100(4), 270–276. http://doi.org/10.3163/1536-5050.100.4.008   kite, j., indig, d., mihrshahi, s., milat, a., & bauman, a. (2015). assessing the usefulness of systematic reviews for policymakers in public health: a case study of overweight and obesity prevention interventions. preventive medicine, 81, 99–107. http://doi.org/10.1016/j.ypmed.2015.08.012   maggio, l. a., & kung, j. y. (2014). how are medical students trained to locate biomedical information to practice evidence-based medicine? a review of the 2007-2012 literature. journal of the medical library association: jmla, 102(3), 184–191. http://doi.org/10.3163/1536-5050.102.3.008    miller, l. (2014). first year medical students use library resources emphasized during instruction sessions. evidence based library & information practice, 9(1), 48–50. http://dx.doi.org/10.18438/b8f316   o’malley, d. & delwiche, francis a. (2012). aligning library instruction with the needs of basic sciences graduate students: a case study. journal of the medical library association, 100(4), 284–290. http://dx.doi.org/10.3163/1536-5050.100.4.010   rafferty, r. (2013). the impact of library instruction: do first-year medical students use library resources specifically highlighted during instructional sessions? journal of the medical library association, 101(3), 213–217. http://dx.doi.org/10.3163%2f1536-5050.101.3.011   research, c. on s. for s. r. of c. e., & medicine, i. o. (2011). finding what works in health care: standards for systematic reviews. national academies press.   rethlefsen, m. l., farrell, a. m., osterhaus trzasko, l. c., & brigham, t. j. (2015). librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. journal of clinical epidemiology. http://doi.org/10.1016/j.jclinepi.2014.11.025   roszkowski, b., & reynolds, g. (2013). assessing, analyzing, and adapting: improving a graduate student instruction program through needs assessment. behavioral & social sciences librarian, 32(4), 224–239. http://doi.org/10.1080/01639269.2013.837798   stewart, l. a. (1996). the importance of systematic reviews and meta-analyses in the practice of evidence-based medicine. annals of the academy of medicine, singapore, 25(4), 483–484.   szajewska, h. (2013). importance of systematic reviews and meta-analyses in pediatric nutrition. world review of nutrition and dietetics, 108, 1–10. http://doi.org/10.1159/000351479   tomaszewski, r. (2012). information needs and library services for doctoral students and postdoctoral scholars at georgia state university. science & technology libraries, 31(4), 442–462. http://doi.org/10.1080/0194262x.2012.730465   whalen, k. j., & zentz, s. e. (2015). teaching systematic searching in a baccalaureate nursing research course. worldviews on evidence-based nursing / sigma theta tau international, honor society of nursing, 12(4), 246–248. http://doi.org/10.1111/wvn.12090   whelan, k. (2014). editorial: the importance of systematic reviews and meta-analyses of probiotics and prebiotics. the american journal of gastroenterology, 109(10), 1563–1565. http://doi.org/10.1038/ajg.2014.258   evidence summary   medical school librarians need more training to support their involvement in evidence based medicine curricula   a review of: maggio, l. a., durieux, n., & tannery, n. h. (2015). librarians in evidence based medicine curricula: a qualitative study of librarian roles, training, and desires for future development. medical reference services quarterly, 34(4) 428-440. http://dx.doi.org/10.1080/02763869.2015.1082375   reviewed by: aislinn conway phd fellow national university of ireland galway galway, ireland email: a.conway18@nuigalway.ie   received: 1 mar. 2016    accepted: 20 apr. 2016      2016 conway. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to describe the self-perceived role of librarians in developing evidence based medicine (ebm) curricula and identify their current and desired level of training to support these activities.   design – multi-institutional qualitative study.   setting – nine medical schools in canada and the united states of america.   subjects – nine librarians identified by medical school faculty as central to the provision of ebm training for medical students at their institution.   methods – the researchers designed a semi-structured interview schedule based on a review of the literature and their own experiences as librarians teaching ebm. the topics covered were; librarians’ perceptions of their roles in relation to the curriculum, the training required to enable them to undertake these roles, and their professional development needs. the interviews were conducted by telephone and then audio-recorded and transcribed verbatim. the authors present five main themes; curricular design, curricular deployment, curricular assessment, educational training, and professional development. profiles were developed for each participant based on the latter two themes and from this information common characteristics were identified.   main results – the participants described the importance of collaboration with faculty and student bodies when designing a curriculum. information literacy instruction and specifically literature searching and forming a research question were taught by all of the participants to facilitate curricular deployment. some of the librarians were involved or partly involved in curricular assessment activities such as formulating exam questions or providing feedback on assignments. educational training of participants varied from informal observation to formal workshops offered by higher education institutions. all librarians indicated a willingness to partake in professional development focused on teaching and ebm.   the subjects’ perceptions of their roles are supported by dorsch and perry’s themes of the librarian’s role in curricular design, deployment, and assessment. the educational training received by participants included formal training and experiential and self-directed learning activities.   finally, the librarians identified their professional development needs going forward. the majority of participants indicated that they would like to attend workshops run by universities or the medical library association. others wanted to invite and host guest speakers at their own institutions. librarians identified financial restraints and geographic location as barriers to attending professional development events.   conclusion – librarians can be actively involved in the delivery of ebm instruction in medical schools. however, they require additional educational opportunities to enable them to develop in this role. online training could be a viable option for self-directed learning to overcome financial and geographic constraints.   commentary   blanco, capello, and dorsch (2014) and maggio et al. (2016) have also reported on librarians’ roles in ebm design, implementation, and assessment. the strength of this study is that it identifies librarians’ perceptions of their own training needs. this study is presented as a starting point for exploring barriers to education and training requirements for librarians involved in ebm curricula. it was conducted within a larger qualitative study which aimed to capture barriers, as perceived by faculty, to ebm learning for medical students (maggio et al., 2016). the authors pre-selected dorsch and perry’s framework (dorsch & perry, 2012) to analyse the role of health sciences librarians in providing ebm instruction.   the consolidated criteria for reporting qualitative research (coreq) is a tool designed for studies featuring qualitative interviews. checklist items 29-32, which specifically refer to aspects of the reporting process, are: quotations presented, consistency of data and findings, clarity of major themes, and clarity of minor themes. (tong, sainsbury, & craig, 2007)   the study does not adequately satisfy the first two coreq checklist items above.  first, the authors present quotations to communicate participant perspectives within the thematic analysis and state that they attempted to include participants across the institutions through quotations presented. however, the quotations are not assigned to identifiers (e.g., participant number), so it is not demonstrated that a range of participants have contributed to the data. second, the data in this study is consistent with findings in the wider literature particularly in relation to the librarian as teacher and their curricular responsibilities. nevertheless, the authors find that the participant data highlights the lack of librarian assessment of learning, despite the variety of resources available to ebm teachers.   the presentation of major and minor themes demonstrates the strengths of the reporting aspects of the research article. the five major themes are clearly presented and each is explored in a dedicated section in the results. diverse cases were reported, such as the singular study participant using a validated tool to assess student learning. some minor themes also emerged from the findings focusing on specific gaps in teaching knowledge and skills, for example, the necessity of learning how to teach on the job due to the lack of pedagogical curriculum in academic library training courses.   a practical implication of this study as noted by the authors, is the need for increased opportunities for continuing education for librarians. in the parent study, faculty identified barriers to ebm learning for students (maggio et al., 2016). in order for librarians to position themselves on a par with other faculty members and contribute to ebm curricula in a valuable way, they should be educated to a level that allows them to effectively perform in this role.   further research is required to assess the extent to which the findings could be extended to guide the planning and development of training for librarians not based at medical schools. the authors recognise that the study findings may not be generalizable to a wider population of librarians since the methods employed to select participants involved a process whereby only a specific population met the inclusion criteria.   in a larger survey by blanco et al. (2014), the authors found that librarians were not sharing innovative approaches. this study would have benefited from investigating whether participants are effective at communicating ebm teaching strategies.   references   blanco, m. a., capello, c. f., dorsch, j. l., perry, g. j., & zanetti, m. l. (2014). a survey study of evidence-based medicine training in us and canadian medical schools. journal of the medical library association,102(3), 160–168.                 http://dx.doi.org/10.3163/1536-5050.102.3.005   dorsch, j. l. & perry, g. j. (2012). evidence-based medicine at the intersection of research interests between academic health sciences librarians and medical educators: a review of the literature. journal of the medical library association, 100(4), 251-257.                 http://dx.doi.org/10.3136/1536-5050.100.4.006          maggio, l. a., ten cate, o., chen, h. c., irby, d. m., & o'brien, b. c. (2016). challenges to learning evidence-based medicine and educational approaches to meet these challenges: a qualitative study of selected ebm curricula in u.s. and canadian medical schools. academic medicine, 91(1), 101-106. http://dx.doi.org/10.1097/acm.0000000000000814               tong, a., sainsbury, p., & craig, j. (2007). consolidated criteria for reporting qualitative research (coreq): a 32-item checklist for interviews and focus groups. international journal of quality in health care, 19(6), 349-357 http://dx.doi.org/10.1093/intqhc/mzm042       microsoft word comm_marshall.doc evidence based library and information practice 2006, 1:1  110 evidence based library and information practice     commentary    evidence‐based library and information practice: the time is now      joanne gard marshall  alumni distinguished professor, school of information and library science  university of north carolina at chapel hill  chapel hill, north carolina, united states of america  e‐mail: marshall@ils.unc.edu      © 2006 marshall. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      over the last decade we have seen the  concept of evidence based practice (ebp)  emerge not only in the health professions  but also in fields such as social work, public  policy and even business. as a group that is  dedicated to helping others find the  information they need to do what they do  better, it is high time that library and  information professionals put ebp to work  for themselves.  but major changes in how  we approach what we do take time and  there are usually several inputs required  before a new movement takes hold. i would  argue that we are getting to the point where  these inputs are accumulating and that the  time is right to begin engaging in major  efforts to develop, evaluate and apply our  own version of ebp.    since medicine was one of the first  professions to develop and use ebp, it is not  surprising that health science librarians have  been in the forefront of adopting evidence  based librarianship. the cost containment  issues in health care generally have been  strong drivers for examining current  physician practices in relation to outcomes  and building the scientific knowledge‐base  upon which to make clinical decisions. ebp  is being mandated through public policy  documents and various funding agencies,  making the concept even more prominent.  evidence based medicine (ebm) uses a  hierarchy of research design to judge the  quality of evidence. as a result, a strong  culture of critical appraisal of the literature,  culminating in systematic reviews of  existing research studies, has developed as  part of ebm. other professions have begun  to develop their own versions of ebp,  providing several approaches that librarians  can look to as models.    we have examples of ebp from other fields,  but what has been happening within  librarianship itself that could be noted as  steps toward adoption? a major source of  leadership in library and information  science (lis) resides in our professional  associations. increasingly, these associations  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:1  111 are adopting statements that support doing  and using lis research, such as the medical  library association and the special  libraries association. associations may also  include research among the competencies  expected of practitioners. many lis  graduate programs include a research  methods course in their curriculum and the  first course specifically on evidence based  librarianship was offered at the university  of north carolina at chapel hill in 2005‐06.  research and evidence based practice are  also starting to appear in library strategic  planning documents.     one of the most interesting aspects of the  rise of ebp in the library field has been its  international nature. the first international  conference was held in sheffield, uk in 2001,  the second in edmonton, canada in 2003,  and the third in brisbane, australia in 2005.  starting small, the conference has essentially  doubled in attendance with each successive  meeting. the fourth international  conference on evidence based library and  information practice (eblip4) will be held  in north carolina, usa from may 6‐9, 2007,  with an additional two days on may 10‐11  available for continuing education courses.  the australia conference was sponsored by  alia, the australian library and  information association, and brought  together not only health sciences librarians  but practitioners from public, academic,  school and special libraries. this multi‐ sectoral interest is expected to continue.    another sign that we are on our way  towards ebp in lis is the increasing number  of publications on the topic. andrew booth  and anne brice’s handbook, evidence‐based  practice for information professionals (2004)  contains 20 chapters covering topics from  why we need ebp, to appraising and  applying the best available knowledge in  specific areas of lis practice such as  reference, education, collection management,  online searching and administration.  increasing numbers of journal articles and  conference presentations are appearing on  ebp as a concept. we are also seeing an  increase in the publication of research‐based  articles on a myriad of library topics. for  several years, the medical library  association has required the submission of  structured abstracts for its annual meeting  and journal that specify the research  methods used. these efforts have  demonstrated the ebp is an attainable goal  for library and information professionals.  so it appears that the time is right to make a  major push towards evidence based practice  in lis. at the lianza conference in 2000,  andrew booth defined evidence‐based  librarianship (ebl) as “an approach to  information science that promotes the  collection, interpretation, and integration of  valid, important and applicable user‐ reported, librarian‐observed, and research‐ derived evidence. the best‐available  evidence moderated by user needs and  preferences, is applied to improve the  quality of professional judgements.” while  there are major challenges ahead, including  developing the version of ebp that will meet  our needs and building the knowledge base  to support our practice, we are well  positioned to move ahead. the time is now!  works cited    booth, andrew, and anne brice, eds.  evidence‐based practice for information  professionals: a handbook. london:  facet publishing, 2004.     booth, andrew. “exceeding expectations:  achieving professional excellence by  getting research into practice.”  lianza conference 2000. christchurch,  new zealand, 2000. 3 march 2006.  <http://www.shef.ac.uk/~scharr/eblib/ex ceed.pdf http://www.shef.ac.uk/~scharr/eblib/ex conference paper   a training needs analysis case study at legal aid queensland   claudia davies library manager legal aid queensland brisbane, queensland, australia email: cdavies@legalaid.qld.gov.au   richard vankoningsveld librarian legal aid queensland brisbane, queensland, australia email: richard.vankoningsveld@legalaid.qld.gov.au   received: 18 aug. 2015  accepted: 27 nov. 2015       2015 davies and vankoningsveld. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – legal aid queensland library services plan, design and deliver a training program comprising seven individual components including induction training, individual instruction, group training, and regional office training. the program is guided by a formal, detailed training plan developed by experienced library trainers. training needs assessment however has been informal and infrequent. library services commenced a formal training needs analysis (tna) in 2014. it was designed to gather evidence for decision making around future training activities.   methods – the tna was comprised of five quantitative and qualitative components: analysis of catalogue and knowledge management database usage statistics; analysis of paid subscription usage statistics; analysis of library reference and research request data; work shadowing library clients; and interviews with selected team managers.   results – each component revealed different aspects of our clients’ training needs. the results of the tna revealed gaps in current training and education services, and opportunities for maximizing the effectiveness of our training program. the tna also provided information about our products and services beyond implications for training.   conclusion – as a result of the tna the laq library has made changes to it training activities, which we believe are now more closely aligned with client needs and organizational objectives than they were in the past. introduction   legal aid queensland (laq) provides legal help to financially disadvantaged queenslanders in criminal, family and civil law matters. laq library services provides a full range of services to approximately 250 lawyers as well as social workers, executive management staff, policy staff and support staff across 14 locations. the library has the equivalent of 3.3 full time staff. our two librarians and two library technicians all provide training services.   training has been a part of the library’s role for the last 30 years. a formal library training plan was first produced in 2008; a major revision was undertaken in 2014. the plan now consists of:   1.       the strategic plan which outlines training policy and the theoretical underpinnings and context of the library’s training program; 2.       the operational plan which outlines seven training components; 3.       the training needs analysis methodology; and 4.       the annual training schedule.   the seven current training components include:   ·         induction training – new legal and support staff receive an introduction to library services from a library staff member as part of their induction process. the session with a library staff member is tailored to the specific role of the inductee within the organization and is delivered face to face or via screen casting depending on the staff member’s location.  ·         individual training – provided to all library clients at point of need. it is the preferred method of library training. it includes face to face and remote training via screen casting, and typically involves working through a specific research problem with the client. it also includes several self-directed learning initiatives including factsheets and help guides, and video on demand continuing professional development (cpd). ·         group training – formal training events are offered by the library including regular small group workshops, and occasional lecture style cpd presentations. ·         regional office training – library staff visit regional offices once each year to provide face to face training to regional library clients, including formal group training and individual training as required. ·         library awareness – raising awareness within the organization of the services and resources offered by the library. we achieve this through information sessions, blog items, web news updates, and emails about new resources to individual clients and teams. daily emails alerting staff to new judgments and weekly legislation updates help lawyers keep up to date with developments in the law. ·         external training – raising awareness of the laq library and the broader organization within the legal library sector through staff presenting at conferences, writing articles and papers and active participation in professional associations. ·         library staff professional development – increasing library staff capacity to provide quality services to our client through actively engaging with our own professional development activities, including self-directed learning, attending in-house training sessions, and external conferences and other professional development opportunities.   the training needs analysis process was designed to provide an evidence basis for developing future training activities and assessing the effectiveness of the current training program.   as this was the first structured tna undertaken, it further asked whether evidence from library database usage logs, publisher-provided usage statistics, research request data analysis, and qualitative evidence from client stakeholder interactions could reliably be used to improve the relevance and effectiveness of our training program.   for the purposes of this first tna we chose to analyse data from the 2014 calendar year only. the decision to limit our analysis to this period was taken because:   ·         it was the most recent complete set of data we had for all the components of the analysis; ·         we determined that it provided a dataset large enough to give meaningful results; and ·         it was a manageable dataset given the time constraints on completing the analysis.   literature review   mcgehee and thayer (1961) first observed that training should be underpinned by systematic research into training needs. they introduced a model framework of organizational, operational and individual analyses that needed to be assessed as a whole to inform effective training. examples of librarians using training needs analyses can be found in the literature though most provide little insight to their methodology. additionally, most take a less holistic approach, often relying solely on written self-assessment instruments. for example, johnson refers to a database training needs [self] analysis form (2005), and surveys were used in a study reported by bresnahan and johnson (2013) and turner, rosen and wilkie (2003). interviews with management can complement survey data (oldroyd, 1995).  allred (1995) questioned the validity of self-assessment as a basis for planning training in his report on a seemingly unsuccessful attempt at a training needs assessment for librarians which employed surveys and a training audit.   beaumont (2002) analysed search-tracking logs at the state library of victoria to describe the searching behaviour of visitors to the library and determine the extent to which they were able to learn from unsuccessful search results and rephrase queries.   rossett (1987) postulated that tnas should use a range of techniques and tools including: extant data analysis, needs assessment and subject matter analysis, interviews, observations, group discussions and surveys. employing a variety of methodologies can shed light on different aspects of the tna including optimal performance, actual performance, the causes of sub-optimal performance, feelings about tasks and potential solutions.   drawing on the work of beaumont’s search log analyses and rossett’s variety of methodologies, the library set out to use the empirical data available to produce a multifaceted view of training needs within laq though a variety of techniques and tools – while avoiding self-assessment instruments.   methods: the five components of laq’s tna   catalogue and knowledge management   database usage statistics   the first component of the tna was an analysis of query logs generated by our library catalogue and internal knowledge management (km) databases. the logs provided empirical data about the searching behaviours of laq staff when using these resources.   the query logs are automatically generated by our lms system and are enabled for our catalogue and all client facing internal km databases. the logs capture the following information:   ·         date of search; ·         start and end time of search; ·         ip address of searcher; ·         database searched; ·         search string; and ·         number of results returned   logs were analysed monthly. the data was imported into an ms excel spreadsheet which used a number of formulae to determine where and when the databases were being used, and what information clients were searching for. the monthly data was subsequently copied into an annual ms excel spreadsheet for the final analysis.   the ip address data revealed whether a user was located in a brisbane or regional office. for users in our brisbane office, we were able to identify the floor in the building that the search originated from. regional users connect to these databases via a vpn connection and consequently ip address data for these users did not reveal any usable information about their location.   a search resulting in zero results being returned was classed as a ‘failed’ search for the purpose of the tna.   subscription database usage statistics   the laq library purchases subscription access to a number of database products provided by three principle external suppliers, as well as a range of other products from various suppliers. usage statistics provided by two principle suppliers, lexisnexis and thomson reuters, were selected for the tna as they provided the largest datasets.   laq uses ip-fixed access to these databases and consequently statistics could only be obtained for the organization as a whole. the statistics provided by each supplier varied in scope, range and quality, but typically included information about the number of searches, document views and downloads for each individual subscription over a period of time. unfortunately, deeper search-level information was not available.   given the variation in data provided by each supplier, statistics were analysed manually using ms excel with separate findings for each supplier included in the results.   library reference and research request statistics   a core role of the laq library is the provision of a reference and research service to its clients. the scope of this service ranges from simple reference queries through to complex legal research. an analysis of the number, complexity and type of requests received by the library provided insight into the legal research needs of our clients. through this analysis some inferences were drawn about the legal research skills required of laq staff, and potential skills gaps that might currently exist.   a record of each reference/research query the library receives is captured in the library’s reference database. library staff enter these records following completion of the request categorising them by complexity (ready reference, simple or complex) and type (e.g., case law research; document delivery). the analysis looked for commonality and variation in request frequency, type and complexity across organizational units and geographic locations (regional offices). individual clients were de-identified in the data and were not included in the analysis.   ms excel was again used to analyse the raw data extracted from our research request database.   work shadowing library clients   library staff shadowed library clients in two different organizational units as they performed their normal duties, providing qualitative data about: their information use and needs; the resources they used; the potential gaps between their needs and skills; problems they encountered when researching; and opportunities for developing training or services for staff in those organizational units.   the two units were chosen on the basis that the library staff knew little about their day-to-day work. team 1 provides telephone advice to laq clients across many areas of law; team 2 represents laq clients in summary matters in the magistrates court.   support for the shadowing was obtained from the relevant team managers and the participating library clients, and permissions were gained from laq clients prior to each session.   four library staff participated in the shadowing, with a total of five shadowing sessions taking place (two with team 1, and three with team 2). each shadowing session lasted between 2-4 hours.   library staff took notes during the sessions and discussed findings with the library client at the conclusion of the session. notes were later collated into an ms word document, allowing for further analysis.   interviewing team managers   face to face interviews with six team managers were carried out by library staff to obtain qualitative data on the training needs of the staff in each manager’s team. each interview ran for approximately 15 minutes and was attended by the team manager and two library staff.   managers were emailed two questions to consider prior to the interview so that they could reflect on their team’s requirements.   within a context of services provided by the library:   1.       what 3 things do you really want your team members to be able to do? 2.       which of these, if any, do they currently struggle with most?   the manager’s responses to the questions were discussed in the face to face interview. where required, follow-up questions were used during the interviews to clarify or expand on answers given, and to discover whether the manager could provide recommendations on the preferred format and timing of future library training.   library staff made notes during each interview which were then collated into an ms word document, allowing for further analysis of responses.   results catalogue and knowledge management database usage statistics   general   the full dataset for the tna comprised 40,389 searches across our catalogue and 11 km databases. time and date analysis showed fairly constant usage throughout the working day but some seasonal variation during the year. ninety-eight per cent of searches were conducted between 7am and 7pm. of the remaining 2%, searches were performed in all hours except from 2-3am. monthly usage varied from 2,299 searches in december to 4,376 searches in july.   usage did not correlate with school terms – july and september, which contain school holidays, were busy months; november – which does not contain school holidays – gave the second fewest number of searches (see figure 1).     figure 1 total searches logged (2014).     seventy per cent of searches originated from the brisbane offices, where 70% of laq lawyers are located.   for the brisbane searches logged, 85% originated from the 3rd and 4th floors which house the criminal division and in-house counsel (barristers), the smaller executive and policy units, and the library – see figure 2. the remaining 15% of searches were dispersed across three floors in the main brisbane office and two annexes (north quay and markerston street). these locations house laq’s family and civil law divisions, and other administrative units.   interpretation of training issues. there was considerable variation in search numbers between months. if trends appear over time, consideration should be given to scheduling training for the quieter months of the year. monthly usage figures challenged our perception that school holidays are our quietest periods. however, further analysis of a larger dataset would have to be analysed to confirm usage trends over time.   the 30% of legal staff located in regional offices executed 30% of searches of the library’s databases. this indicates that there is a continuing need for library staff to provide training and point-of-need support to regional offices through appropriate and convenient channels. training priorities need to be aligned with the needs of teams and divisions which use library resources the most, i.e. the criminal practice and in-house counsel. however, low usage by other teams must be interpreted in the context of the relevance of our knowledge management databases to their areas of practice. for example, we would expect the use of our km databases to be much higher for the criminal appellate specialists, who make detailed submissions in higher courts, than for duty lawyers appearing in the magistrates court who have minimal time to prepare for their appearances.     figure 2 location of users brisbane offices.     interpretation of additional issues. the relatively even spread of queries across business hours indicates that reference staff must always be available to assist and advise clients during business hours.   while the bulk of searches occur between 8am and 5pm, consideration should be given to whether we can provide after-hours support without compromising the library staff’s work/life balance.   catalogue searches   the dataset of catalogue searches comprised 21,490 queries.   these searches fall into five distinct categories, including:   ·         free text using a traditional catalogue search form on the laq intranet; ·         predefined searches via web term lists (130 links on the library’s intranet pages to collections of key resources by topic, and curated by library staff); ·         predefined searches for continuing ·         professional development (cpd) resources; ·         predefined searches for our family law notes (fln) current awareness service; and ·         predefined searches for specific catalogue records via an laq news feed on the homepage of the organisation’s intranet (see figure 3).     figure 3 categories of catalogue search.     of the web term list searches, 28% were to legislation topics and 25% to case law topics. sixteen of the 130 web term lists were not searched at all in 2014.   free text searches via catalogue search form. our catalogue search form has four main fields – global search, title, author and subject, and three other limiting fields – date, type (e.g. book, journal) and format (e.g. electronic, paper). when users performed a free text search using the intranet search form 96% used a single search field only.  overall, free text searches produced a 15% failure rate (i.e. searches returning no results). the author field yielded a 2% failure rate while title and subject proved less successful with 27% and 30% failure rates respectively. when two fields were searched the failure rate reached 31%.   the ability to do a global search was introduced in october 2014 and was used in 247 searches with a failure rate of 19% to the end of 2014.   in february 2015, the database search forms were rewritten to comply with current web standards and overcome some ux issues. following this change the overall failure rate dropped from 15% to 3.5%.   interpretation of training issues. the dramatic fall in failure rates for searches following recoding of search pages, coupled with the relatively low usage of the catalogue search form suggests that training in effective searching of the library catalogue should be a low priority.     figure 4 km database searches by area of law.     interpretation of other issues. the popularity of predefined searches suggests that the users are relying on web term lists to access library resources in preference to searching the catalogue manually. these topics need to be reviewed systematically to ensure they remain relevant and current. additionally, the time the library invests in current awareness emails and web news items is worthwhile since we can clearly demonstrate that staff access library resources via these routes.   our analysis of web term list use showed that a number of key topics including legislation and case law were heavily used. there is scope to review the less ‘popular’ topics. this may involve reducing the number of topics, amalgamating and relabelling topics, and adding new topics.   km databases   ninety-two per cent of the 16,984 searches of laq’s km databases were to criminal law databases. searches of the library’s criminal judgments and comparable sentences databases accounted for 79% of searches. a further 3% were to general databases e.g. legislation, and only 5% of traffic was to civil and family law databases combined – see figure 4.   of the 9,176 searches of the criminal judgments database, 42% were executed by users clicking on links to predefined searches such as current awareness emails sent to staff in the criminal law division.   manual searches of the criminal judgments database via a search form on the library’s intranet page resulted in a total failure rate of 18%.  searches using specific fields had varying success rates. for example, the court field had a surprisingly high 66% failure rate; court number 21%; and decision date 25%.   the comparable sentences database has a complex search form that allows users to perform searches matching very specific criteria. however, 90% of searches used only a small number of the available fields including charge, age, criminal record and plea. twenty-one per cent of searches were limited to appeal sentences (see table 1).     table 1 adult comparable sentences database search fields used in manual searches field % of total searches including this field charges 90% age of offender 41% under 25 y/n 39% criminal record 34% plea 34% appeal sentences 21% court 20% particulars and comments 9% noncustodial sentences 6% full text 6% $ value of property 5% armed 5% alcohol/drugs 4% sentence date/range 3% offences in company 2% known to accused 2% gender 2% cooperation with authorities 2% employed 1% psych problems 1% dependants 1% previous convictions 1% judge <1% aboriginal/tsi <1%     the overall failure rate for the comparable sentences database was 33%.   interpretation of training issues. there are clearly implications for training in the high failure rates when using the comparable sentences database, and to a lesser extent the criminal judgments database. changes to database search forms implemented in 2015 saw a reduction in the number of failed searches, however the 2014 tna results suggest that strategies to improve this situation should include more regular training in using the database, and further investigating search form ux/functionality changes to help eliminate specific searching errors.   while it is understandable that fields such as charge, age, criminal record, plea and court would be the most-searched fields in the comparable sentences databases, low usage of other fields suggests that further training is needed in performing more complex searches.   interpretation of other issues. in addition to extra training, the high failure rates for manual searches of the comparable sentences and criminal judgments database may be reduced further by exploring search form ux/functionality changes to help eliminate specific searching errors.   the high percentage of traffic to criminal law databases supports the library’s policy of directing the largest proportion of our time to maintaining and developing databases in this area.   the high occurrence of predefined searches in our criminal judgments database indicates current awareness services to our criminal lawyers are widely used. provision of these types of service should be continued. consideration should be given to what improvements in the scope and relevance of such services might be made.   subscription database usage statistics   thomson reuters   the highest usage of thomson reuters was for accessing case law (55% of searches) and commentary (28%) – see figure 5. journal usage at 11% of the total was relatively low in comparison.   usage figures (see figure 5) show that for most products, users were likely to look at only one or two results following each search. exceptions to this were unreported judgements searches where the average was over two cases, and civil law commentary publications, where only around half of the searches performed resulted in viewing of a document in the search results.   the rates at which users downloaded documents they viewed were generally low – 4% for case law and journals, 14% for commentary and 27 % for the legal encyclopaedia laws of australia.   no data was provided for searching verses browsing of publications on thomson reuters.     figure 5 thomson reuters searches by content type.   table 2 thomson reuters views as percentage of searches subscription views as % of searches reference 128% laws of australia 123% lawyers practice manual 138% commentary 145% criminal 149% procedure 157% civil 57% journals 138% adrj 141% qld lawyer 135% family law review 152% criminal law journal 129% alj 192% caselaw 114% firstpoint 96% unreported judgments 239% law reports 107%     lexisnexis   for all types of publication except commentary services, searching accounted for greater than 90% of interactions. for commentary services searching was still dominant but was reduced to 62% – see figure 6.     figure 6 searching vs browsing of publications on lexisnexis.   figure 7 percentage of research requests by complexity (2014).     the figures for searching verses browsing by subject were very similar, with searches accounting for 90% of behaviour for all subjects except criminal law. the library subscribes to 29 titles on lexisnexis yet carter’s criminal law of queensland accounted for 26% of all interactions.   interpretation of training issues. usage figures for subscription services show that the most critical information needs of the library’s clients are finding case law, and to a lesser degree legal commentary. therefore, these should be the focus for training activities.   while journal use was significantly lower than for other resources, this was not unexpected given that the legal practice is focused on service provision rather than academic research. journal usage should however be benchmarked against similar organisations.   figures for viewing documents on thomson reuters show that, on average, users are finding one or two results per search worth further attention. this figure is reasonable for locating commentary on a particular section of legislation, or locating specific cases. the figure is low however for situations where users are looking for cases on a point of law, for example via firstpoint or unreported judgments. this again suggests a need for providing more training in case law research.   download rates on thomson reuters are quite low. without past results for comparison, more investigation is needed to understand this figure. there may be issues with effective search techniques that could be addressed with training.   the lexisnexis data showed that users browse commentary services much more frequently than they do other kinds of online publications. one possible explanation is that users are less comfortable searching for commentary and continue to use online commentary services like print resources. this hypothesis needs to be tested through follow up research. interviewees (in component 5 – interviews with team managers) however also reported that they wanted more training in using commentary services, supporting this theory.   library reference and research request statistics   seventy-four per cent of research requests received in 2014 were classified as simple such as requests for specific legal cases or legislation, and simple catalogue or comparable sentences searches. twenty-three per cent were classified as complex – see figure 7.   a quarter of the requests came from regional offices, a quarter from the civil and family law divisions combined, a quarter from the criminal law division, and 15% from our in-house counsel. the 10 teams who used us most came from across all the legal divisions and the executive management team.   figure 8 shows the types of requests received in 2014. nearly half related to case law research; 10% were legislation related; and 7% were more general legal research requiring a mix of primary and secondary legal sources.     figure 8 types of research requests received (2014).     interpretation   training issues. the high percentage of simple reference queries indicates that there is still a need for training in basic library and research skills. further, the majority of requests involved case law and comparable sentences research, again indicating that we should be concentrating our training efforts in these areas.    other issues. analysis of reference queries by teams shows that teams which use us the most come from all divisions and locations confirming that we need to provide training and convenient communication channels across the organization. additional promotion of library services could increase the use of the library’s research services amongst low use teams including regional offices.   work shadowing library clients   library staff performed five work shadowing sessions with library clients in two different organizational units. all four library staff members expressed a desire to participate in work shadowing both teams selected for the 2014 tna. sessions were organized following the shadowing where library staff could discuss their observations and impressions.   the responses from clients in both teams were similar. lawyers in these teams have limited time for legal research. observation of interactions between the lawyers and their clients confirmed that they rely heavily on experience and prior knowledge of the law to provide efficient quality advice and assistance to their clients. clients in the telephone advice team identified a need for current awareness services which covered state and commonwealth legislation and case law relevant to their practices.   interpretation   training issues. the responses from the shadowing sessions made it clear that there is little need for library training for clients in these particular teams, other than the need for relevant current awareness services.   the level of engagement and enthusiasm the library staff showed for this exercise however indicated that work shadowing is a worthwhile professional development opportunity for library staff. it increases the staff’s knowledge of the business of the organisation and therefore should be included in annual performance agreements.   other issues. the shadowing sessions showed that lawyers in these teams need up-to-date skills and knowledge to provide advice and representation. the range of legislation and case law alerting services from commercial and government sources however is overwhelming and few are sufficiently customisable.   the library identified legislation updates as a service we could provide to our clients, and consequently created a database of recently updated legislation of relevance to laq’s areas of practice. updates containing a link to a predefined query of the legislation database are emailed to library clients each week. access to the database is also available from our intranet. the email alerts summarize the changes to relevant legislation, and provide links to primary sources such as the bill and act as passed, and secondary sources such as history of the legislation.   interviewing team managers   six team managers (approximately 10% of legal team managers) from across the organization were interviewed, and the findings from these interviews were subsequently analysed.   all the interviewees expressed a requirement for case law training, and half mentioned comparable sentences training specifically. some interviewees also suggested training in researching legislation; basic general research skills; and effective searching of commentary services was also needed. most managers reported however that their teams were already competent at researching legislation.   one manager reported that they would rather come to the library for help with complex research than getting training in advanced research skills.   other issues raised included knowing what resources were available from the library and the need to keep legislation and case law knowledge current.   managers were also asked about their teams’ preferred mode of delivery. none of the interviewees expressed an interest in self-paced web-based training videos, such as webinars.   all interviewees indicated that cpd points were an incentive to attend training, and 15 minute sessions within team meetings were also requested. additionally, managers provided information about the times of day that would result in higher participation rates.   interpretation   training issues. the library should provide regular training on case law and comparable sentences research in a mixture of formats from 15 minutes to 1 hour to cater to all teams’ schedules. additionally, basic skills training in legislation and commentary services should be made available regularly.   library staff need to liaise with team managers when scheduling training and provide sufficient notice of future training sessions.   consideration should be given to tailoring course content to specific teams, rather than the current tendency for generic training.   discussion   through the tna process the library was able to elicit data about the actual searching and browsing behaviour of staff; their skills levels and research needs; and to identify gaps between actual and ideal performance by using a range of data sources. these included a mix of empirical data (database usage statistics), subject matter analysis (research requests) and observational evidence (job shadowing and interviews). the combination of these techniques provided a more complete and accurate analysis than could have been achieved through any single methodology.   some components provided unique insights into training and other issues. for example, catalogue and km database usage analysis revealed the scale of a usability issue with the library’s database search forms for which we were subsequently able to develop a technical solution. client interviews provided very specific data about optimal skill requirements and scheduling of training programs that would otherwise have been unknown to library staff planning future training.   certain themes recurred throughout the components. effective case law research was shown to be the most important skill for lawyers. in-house counsel and criminal law teams were shown to be our biggest users and their research tools the most frequently accessed.   the evidence provided by the tna supported some assumptions the library had, for instance that the in-house counsel’s primary need was for high quality case law research and that the criminal law practice needs ongoing training in case law, comparable sentences and commentary services. it showed us we were underestimating some user behaviours such as how much users searched online subscriptions (compared to browsing). it also gave some unexpected results such as how difficult users find searching particular internal database fields.   the tna relied on data that was readily available to us. however, it took considerably more time to complete the analysis than was estimated at the project’s outset. subsequent tnas however will be simpler and quicker because the development work has now been done including:   determining appropriate tna components and documenting methodologies; designing spreadsheets to automate query log and research query analyses; and retrospectively classifying research requests to allow tna analysis.   the analysis of 2014 data was distilled into a training needs analysis report, the first of an annual series. the annual report will provide a sound evidence base for developing a training program in 2015 and beyond.   there have been a great number of other benefits beyond the library’s training program, including:   senior management awareness of library services; evidence for demonstrating the library’s value to the organization; improved reporting of library usage statistics; improved relationships with library clients; highlighting of ux issues with internal databases; and professional development for library staff.   future developments   the tna has produced results that we can translate into action. our next step is to review our training schedule and develop and deliver training activities based on the results of the tna. we will also continue with a program of internal database redevelopment to improve the manual searching experience for library clients.   the 2014 tna has provided sufficient value to schedule it for the first quarter of every year.  the library however is not sufficiently resourced to perform every component of the tna each year. including fewer components per year will allow us to analyse the selected components more deeply. for instance, analysis of ‘failed searches’ or individual subscription titles usage could be included some years.   query logs are harvested every month. they produce meaningful data through automatic analysis, and therefore will be included in the tna each year. the statistical spreadsheets will be updated to accommodate changed search parameters and measure their impact.   work shadowing provided an opportunity for developing a new service but revealed little about how lawyers use information resources on the job. future shadowing should include teams with more complex legal research needs.   future tnas may include online user surveys. future annual training reports will also include trend analysis across years. of particular interest are:   ·         trends in database usage including predictability of monthly peaks and troughs; ·         changes to search failure rates and sophistication of search behaviours; ·         changes to subscription usage patterns; and ·         changes to the rate or types of reference queries especially in subjects targeted by training.   further, we should experience an increase in participation in training sessions if we can better target and schedule training sessions.   conclusion   the process developed for a multi-component tna successfully met the objective of describing training needs within laq as an evidence basis for developing training activities. the five components provided a mix of empirical, observational and anecdotal evidence and produced a multi-faceted picture of training needs at laq. the library will use the results to develop a program of training aligning with our client groups’ needs and skills gaps. the process will be repeated annually to describe trends and provide insight into the effectiveness of training efforts.   references   allred, j. (1995). co-operative training, open learning and public library staffs: interim report november 1992 and final report february 1994. library management, 16(4), 46.   beaumont, a. (2002). the 3 bears not too big, not too small, just right or how search access logs can be used to improve success rates for searchers. in proceedings from e-volving information futures,11th biennial conference & exhibition 1. (pp. 229-254). port melbourne vic: victorian association for library automation.   bresnahan, m. m., & johnson, a. m. (2013). assessing scholarly communication and research data training needs. reference services review, 41(3), 413-433. http://dx.doi.org/10.1108/rsr-01-2013-0003   johnson, a. (2005). training the trainees: developing effective programmes and partnerships in legal practice. legal information management, 5(1), 34–41. http://dx.doi.org/10.1017/s1472669604002324   mcgehee, w., & thayer, p. w. (1961). training in business and industry. new york: john wiley & sons.   oldroyd, r. (1995). staff development and appraisal in an "old" university library. librarian career development, 3(2), 13-16. http://dx.doi.org/10.1108/09680819510083435   rossett, a. (1987). training needs assessment. englewood cliffs, nj: educational technology publications.   turner, a., rosen, n., & wilkie, f. (2003). joining forces: developing a network to raise awareness of digital library resources in health care. vine, 33(4), 161-167. http://dx.doi.org/10.1108/03055720310510873   microsoft word es 4409 evidence based library and information practice 2009, 4:1 39 evidence based library and information practice evidence summary undergraduate students perceive reference encounters to be teaching and learning activities a review of: gremmels, g. s., and k. s. lehmann. “assessment of student learning from reference service." college & research libraries 68.6 (2007): 488-501. reviewed by: megan von isenburg duke medical center library dumc 3702, 10 bryan-searle drive, durham, nc 27710 e-mail: megan.vonisenburg@duke.edu received: 26 november 2008 accepted: 19 january 2009 © 2009. von isenburg. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. objective – the study explores the instructional nature of reference encounters from the perspective of students and librarians. specifically, the study asks: 1) whether students perceive reference interactions to be instructional, 2) whether what they learn is the same as what the librarians intended to teach, and 3) whether they connect reference-based instruction with any formal information literacy classes in which they may have participated. design – survey questionnaire with two parts: one for students and the second for reference librarians, administered twice (revisions to the study and to the questionnaire were made in between). setting – wartburg college in iowa, united states. wartburg is a 1,800-student private, residential, coeducational college with a strong course-integrated information literacy program. subjects – an unknown number of undergraduate students who were perceived by reference librarians to have asked instructional questions at the reference desk (264 surveys were collected from students, but some students may have completed more than one survey as the first implementation of the study allowed repeated participation for students with more than one instructional reference encounter) and four librarians. 40 methods – the study was conducted two times during the years 2003 and 2004 (referred to here as study a and study b) and findings are reported for each study separately. the data collection instrument in both implementations was a paper survey that was divided into two sections and perforated to collect information from two perspectives: the student and the librarian. the surveys were numbered to facilitate matching between the two sections after the survey was completed and returned by the student/librarian pair. potential student participants in the study were identified at the reference desk: each time a librarian deemed a reference question to be instructional, he or she invited the student to complete a short survey. if the student agreed, the librarian tore off and kept the librarian portion of the survey and gave the second section to the student to complete. students and librarians deposited their sections into a box and sections were re-matched and moved to a secure location every few days. on their section of the survey, students were asked whether the librarian who assisted them taught them anything while answering their question, and if so, to describe what the librarian taught them in their own words. additionally they were asked if what the librarian taught them built upon skills learned in a library session held for the present assignment or from a previous class, if applicable. on the librarians’ section of the survey, there were two questions: what the librarian intended to teach the student (to be chosen from a checklist) and whether the librarian thought the student understood that intention during the reference encounter. comments were allowed. some revisions were made to the survey instrument between study a and study b, including two of note: a check list matching that on the librarians’ survey was added to the student survey question about what the librarian taught the student, and the student participant selection criteria were changed to limit only one response per student. at the end of both studies, data analysis was done, including entering data into a spreadsheet and transcribing both student and librarian descriptions of learning into a narrative document (for study a) or qualrus, a software program for qualitative research (for study b). the study authors independently compared the librarians’ and students’ descriptions of what was taught and labelled the match as either “related”, “inconclusive” or “not related” (for study a) or “strong match,” “acceptable match,” or “no match” (for study b). disagreements were discussed and authors came to an agreement for each. main results – response rates for study a (85%) and study b (78%) were high. most students indicated that they believed the librarian taught them something (94% for study a and 98% for study b). findings on whether the students learned what the librarian intended to teach are mixed. for study a, 60% of the student responses were deemed matches, 20% were not related and an additional 20% were inconclusive. for study b, the authors report their findings in a different manner since the student survey included a checklist that matched the librarian survey in addition to a narrative description of what the librarian taught them. the findings therefore include whether the librarian and student surveys matched in the open-ended descriptions, in the category checklist, in both, or in neither. in this second study, 21% matched in the description only, 36% matched in the category only, 21% matched in both the description and the category, and 22% showed no match at all. this puts the overall match rate for study b at 78%. 41 surveys were analyzed to determine which categories were most likely to be matched. for study a, 62% of the matches were in the “tool” category, 4% in “terms,” 16% in ”strategy,” 10% in ”database,” and 8% in ”other.” for study b, 42% of the matches were in the ”tool” category, 22% in “terms,” 16% in “strategy,” 16% in “database” and 4% in “other.” additional findings relate to the connections between the reference encounters and previous information literacy sessions. approximately one third of students had participated in an information literacy session (33% for study a and 34% for study b) for their class. of these students, most connected what the reference librarian taught them and what they had learned in their in-class instruction (89% for study a and 95% for study b). the rate for how many students connected reference-based instruction with a prior information literacy class was lower, but still quite high (77% for study a and 74% for study b). conclusion – the vast majority of student participants perceived that the reference encounters were instructional and most also connected what they learned from the librarian in the reference encounter to librarian-led information literacy sessions in their current or previous classes. this suggests that post-session reference assistance could help reinforce information literacy principles in a one-on-one situation, perhaps closer to the time of need. for this follow-up instruction to be effective, reference librarians may want to refer directly to what was taught in the information literacy sessions, which could help place new skills in the context of those with which the students are already familiar. the authors also suggest that short reference encounters at the desk may not be appropriate for this kind of instruction and propose that scheduled consultations of up to 30 minutes may better meet students’ needs. rates for matching what the student thought the librarian taught them and what the librarian intended to teach them were lower, if still relatively high at 60% (study a) and 78% (study b). the authors attribute this lower rate in study a to conservative coding, the inability of researchers to follow up on what the students meant by their answers, and the difficulty students might have in describing what they refer to as “tacit knowledge.” undoubtedly, the openended nature of the students’ responses led to difficulties in matching how they might describe a skill or tool to how a librarian would describe it. adding a checklist of categories for students to select from for study b certainly made it easier to overcome the problem of students not using (or perhaps even knowing) the same terminology to describe skills as librarians and helped to achieve a higher match rate. “tools” was the category that was most likely to be matched in both study a and study b. the authors did not speculate on the reasons for this finding. commentary the authors position this study in the context of a new era of reference evaluation that is focused on student learning as opposed to earlier studies that investigated the accuracy of reference librarians’ answers, patron satisfaction, and librarianpatron communication. this is an important and rich area for new research as libraries continue to document decreasing readyreference transactions and rising instructional activities, and as librarians “are increasingly an integral part of the teaching and learning infrastructure at their institutions” (association of research libraries 5). 42 the two implementations of this study offer an initial exploration of the instructional nature of reference transactions that could help set the stage for further research in the area. the most noteworthy outcome is that the undergraduate students involved in the study perceived reference encounters to be teaching and learning activities. this finding suggests that librarians should structure reference transactions to optimize student learning, including discussing concepts introduced in prior in-class sessions and scheduling consultations when possible to allow for sufficient time for pre-planning and for guided practice with the students. the authors note that this is the first study to investigate whether students learn what librarians intend to teach. while a survey of both librarians and students is a seemingly logical way to approach this question, the reliance on self-reporting makes it difficult to collect data that demonstrates student learning. the addition of categories to the student survey in study b facilitated analysis and eliminated some potential barriers involving students’ unfamiliarity with information literacy concepts and terminology. however, it remains unclear whether students would be able to differentiate between the inherently related categories accurately or whether their ability to select the same category as the librarian indicates that they learned that skill or concept. in addition to revising the survey instrument, the authors made methodological improvements to the study design between study a and study b. allowing only one survey per student refined the selection process. however, because the librarians who assisted them chose student participants, an element of bias may have still been operating in both implementations. further research is needed to fully assess student learning in reference encounters. future studies might address whether the librarian taught the skills and concepts that the students needed to complete the assignment, whether the students were able to apply, retain and master those skills or concepts, and finally, whether those students who received help from a librarian did better on their assignments than those students who may not have received help. works cited association of research libraries. statistics 2005-06: a compilation of statistics from the one hundred and twentythree members of the association of research libraries. ed. martha kyrillidou and mark young. <http://www.arl.org/bm~doc/arlstats0 6.pdf> . washington: association of research libraries, 2008. accessed 21 nov 2008. evidence based library and information practice evidence based library and information practice 2009, 4:3  evidence based library and information practice       evidence summary    typology of librarian status created for u.s. land grant universities is applicable to  u.s. research libraries    a review of:  bolin, mary k. “librarian status at u.s. research universities: extending the typology.” journal of  academic librarianship 34.5 (august 2008): 416‐24.    reviewed by:  kate kelly   royal college of surgeons in ireland, the mercer library  dublin, ireland  email: katekelly@rcsi.ie    received: 21 may 2009          accepted: 12 august 2009      © 2009 kelly. this is an open access article distributed under the terms of the creative commons attribution  license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and  reproduction in any medium, provided the original work is properly cited.    abstract    objective – to describe and categorize the  status of librarians at 119 american research  libraries using a typology of librarian status  first developed for 50 u.s. land grant  universities.    design – survey.    setting – u.s. research universities.    subjects – 119 american research universities.  included are those universities whose library  is a member of the association of research  libraries (arl), in addition to land grant  universities who are not also arl members,  and any flagship state universities who are  neither arl nor land grant universities. all  subjects are classified as either “research –  very high” or “research – high” in the  carnegie classification of institutions of  higher education. the 119 institutions  represent a total census of the selected  population.    methods – the websites of the 119 institutions  were surveyed and data on institutional  characteristics such as governance, size and  geography collected. additionally, data  describing librarian status characteristics such  as administrator title, rank systems and tenure  status was gathered from sources such as  promotion and tenure documentation, faculty  handbooks, and policy manuals available on  websites. data was compiled on a spreadsheet  and imported into spss which was used to  create frequencies and cross tabulations. data  was categorised and cross‐tabulated using a  typology of status originally applied to 50 land  grant universities in a previous study. the  typology comprises four possible status types  for librarians:        60 mailto:katekelly@rcsi.ie http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2009, 4:3  type 1 – faculty: professorial ranks.  type 2 – faculty: other ranks with tenure.  type 3 – faculty: other ranks without  tenure.  type 4 – non‐faculty: professional or    academic staff.    main results – in the 119 institutions  surveyed, librarians held faculty status at 74  (62%) institutions, of which 63 (51%) provided  tenure track positions.  at the remaining 45  (38%) institutions, librarians were considered  non‐faculty.     of the 50 “land grant” institutions in the  population, 40 (80%) had librarians with  faculty status and 35 (70%) provided tenure  track.  ten universities (20%) considered  librarians non‐faculty.     of the 97 arl libraries in the population, 55  (57%) had librarians with faculty status and 44  (45%) provided tenure track.  non‐faculty  librarians were found at 42 (43%) of these  institutions.    of the 90 public institutions in the population,  68 (76%) had librarians who were faculty, 57  (64%) provided tenure track, and 22 (24%) had  non‐faculty librarians.     among the 29 private institutions the status  ratios were reversed with only 4 (13%)  institutions having librarians ranked as tenure  track faculty (type 1 or type 2) and 23 (80%)  having non‐faculty librarians.    in the total population (119) type 3 “faculty:  other ranks without tenure” was the least  common category, 48% (57) of libraries were  headed by a dean, 67% (80) of institutions had  librarian representation on faculty senate and  as the size of an institution increased the  likelihood of librarians having faculty rank  decreased. “dean” was the most popular  administrator title in the population; it  correlated with professorial rank and was  strongly associated with tenure.  having  tenure was, in turn, strongly associated with  faculty senate representation. in the northeast  census region type 4, non‐faculty staff  predominated while type 1 was rare; in the  midwest there was an almost even split  between type 1 and type 4; in the south there  was a fairly even spread across all four types,  and in the west a fairly even spread across  types 1, 2 and 4.    finally, the data showed that as the size of an  institution increased, the likelihood of  librarians having faculty rank decreased.    conclusion – the typology created for land  grant universities can be extended and applied  to a wider population.  it is valid and reliable  both for organizing information about  librarian status and for comparing institutions  and population segments.      commentary    reading bolin’s original paper on the  application of the typology created for 50 us  land grant institutions is essential to put this  paper on extending the typology into context.    reading bolin’s original paper on the  application of the typology created for 50 u.s.  land grant institutions is essential to put this  paper on extending the typology into context.   this study will be of interest primarily to  academic librarians in the u.s.  one of the  many interesting results is that u.s. librarians  at private institutions included in the  population have markedly different status  from those at public institutions, as measured  by the characteristics identified in the study.    the paper is generally well written and the  presentation of data is clear. the literature  review clearly places the study within the  framework of the long‐running and ongoing  debate on the role and status of academic  librarians in the us.  the explanations of  prototype theory and linguistics adequately  explain the decisions behind the typology  created. limitations to the study are  articulated and further research areas are  identified.    61 evidence based library and information practice 2009, 4:3  62 while bolin is clear about the inclusion of  subjects within the population and includes  details of the 119 universities in the appendix,  the carnegie classification lists 199  institutions designated as “research – very  high” or “research – high” and the number of  “land grant” universities varies depending on  which cohort of “land grants” is selected, 1862,  1890 or 1994. this means it is unclear which  institutions have been excluded, whether there  was any reason for this, (for example, stanford  university and tufts university were not  found in the list of institutions studied), and  whether it would make a difference.  furthermore, the methodology for reviewing  data on web sites could have been described in  more detail and the exact source of all  supplementary data provided.  the typology is not applicable in countries  where the terms “faculty” or “tenure” have  little or no association with academic  librarians.  however, those interested in the  role and status of academic librarians  elsewhere could consider using the bolin’s  methodology to generate a locally relevant  typology.      works cited    mary k. bolin, “a typology of librarian  status at land grant universities.”  journal of academic librarianship  34.3 (may 2008):220–30.  evidence summary   comparison of e-book acquisitions strategies across disciplines finds differences in cost and usage   a review of: carrico, s.b., cataldo, t.t., botero, c., & shelton, t. (2015). what cost and usage data reveals about e-book acquisitions: ramifications for collection development. alcts, 59(3). retrieved from https://journals.ala.org/lrts/article/view/5752/7199   reviewed by: laura costello head of research & emerging technologies stony brook university libraries stony brook, new york, united states of america email: laura.costello@stonybrook.edu   received: 30 nov. 2016 accepted: 8 feb. 2017      2017 costello. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to compare e-book cost-usage data across different acquisitions styles and disciplines.   design – case study.   setting – a public research university serving an annual enrollment of over 49,000 students and employing more than 3,000 faculty members in the southern united states.   subjects – cost and usage data from 15,006 e-books acquired by the library through packages, firm orders, and demand-driven acquisitions.   methods – data was collected from publishers and vendors across the three acquisitions strategies. usage, cost, and call number information was collected for the materials purchased via firm order or demand driven acquisitions and these were sorted into disciplines based on the call number assigned. discipline, cost, and use were determined for each package collection as a whole because information on individual titles was not provided by the publishers. the authors then compared usage and cost across disciplines and acquisitions strategies.   main results – overall, e-books purchased in packages had a 50% use rate and an average cost per use of $3.39, e-books purchased through firm orders had a 52% use rate and an average cost per use of $22.21, and e-books purchased through demand driven acquisitions had an average cost per use of $8.88 and 13.9 average uses per title. package purchasing was cost effective for science, technology, engineering, and mathematics (stem) materials and medicine (med) materials. demand driven acquisition was a particularly good strategy for humanities and social sciences (hss) titles.   conclusion – there are differences between the acquisitions strategies and disciplines in cost and use. firm orders had a higher cost per use than the other acquisitions strategies.   commentary   this study examined cost per use across three acquisitions styles and three disciplinary groups. the results agree with studies from other institutions that have found demand driven acquisitions to be a cost-effective strategy for e-books, particularly when cost per use is considered (downey, zhang, urbano, & klinger, 2014; herrera, 2012). the findings also agree with studies that have found a good cost per use, but low percentage of use in package acquisitions (lannon & mckinnon, 2013; sprague & hunter, 2008). further research examining data from multiple institutions could help to determine whether low percentage of use in subscription collections is a discovery issue for individual libraries or a selection and packaging issue for publishers.   the authors also found it difficult to justify firm ordering as a cost-effective strategy for e-book content. though there have been strong critiques of using methods like cost per use and percentage of use to evaluate the value of firm ordered books (fry, 2015) there is little evidence to support this strategy in the e-book environment, where turnaround time for purchases is generally much shorter. the authors identify the difficulty in drawing a line between firm orders and demand driven orders. firm orders often represent demand from faculty members and students through traditional channels. the changes to the subject university’s acquisitions methods seem practical and in line with the findings they reported.   the study presented a relatively small sample size, especially for materials that were purchased via firm order and demand driven acquisitions. materials acquired in packages represented 13,027 out of 15,006 monographs in the study. generalizability is a problem common to most, if not all of the single site acquisitions studies of this kind. more acquisitions research from consortiums and conglomerate data, like michael levine-clark’s work with combined ebl and ebrary data (levine-clark, 2015) will contribute more significantly to our understanding of the way disciplines and acquisitions strategies impact use. the findings from the subject university are strongly in line with other studies of this kind so the conclusions made are appropriate for local application, even if they are not broadly generalizable.   the authors also relied on combined subject designations and use for each of their package collections. because the authors used three broad disciplinary categories and did not consider large interdisciplinary packages, this likely did not significantly influence the results. future studies approaching this question might use a common subject classification for all acquisitions strategies, especially when considering databases with general content. future studies could also move beyond descriptive statistics for ranking and examine the statistical significance of differences in cost and cost per use.   this study represents an important contribution to the landscape of findings on this topic, particularly for collections and acquisitions librarians. the results of this article persuaded the authors to change acquisitions processes to favor more cost-effective methods. further research, including more meta-analysis of acquisitions strategies, is needed before these results can be generalized to all libraries, but librarians assessing their own collections will find this a good template for comparison.   references   downey, k., zhang, y., urbano, c., & klinger, t. (2014). a comparative study of print book and dda ebook acquisition and use. technical services quarterly, 31(2), 139-160. http://dx.doi.org/10.1080/07317131.2014.875379   fry, a. (2015). conventional wisdom or faulty logic? the recent literature on monograph use and e-book acquisition. library philosophy and practice, 2015(1), 1-27. retrieved from http://digitalcommons.unl.edu/libphilprac/1307   herrera, g. (2012). deliver the ebooks your patrons and selectors both want! pda program at the university of mississippi. serials librarian, 63(2), 178-186. http://dx.doi.org/10.1080/0361526x.2012.700780   lannon, a., & mckinnon, d. (2013). business e-books: what can be learned from vendor supplied statistics? journal of business & finance librarianship, 18(2), 89-99. http://dx.doi.org/10.1080/08963568.2013.767121   levine-clark, m. (2015). e-book usage on a global scale: patterns, trends and opportunities. insights: the uksg journal, 28(2), 39-48. http://dx.doi.org/10.1629/uksg.240   sprague, n., & hunter, b. (2008). assessing e-books: taking a closer look at e-book statistics. library collections, acquisitions and technical services, 32, 150-157. http://dx.doi.org/10.1016/j.lcats.2008.12.005     evidence summary   prison library services in croatia need improvement to meet international standards of universal rights to access   a review of: šimunić, z., tanacković, s.f., & badurina, b. (2016). library services for incarcerated persons: a survey of recent trends and challenges in prison libraries in croatia. journal of librarianship and information science, 48(1), 72-89. http://dx.doi.org/10.1177/0961000614538481   reviewed by: carol perryman assistant professor texas woman’s university denton, texas, united states of america email: cp1757@gmail.com   received: 26 may 2016  accepted: 15 july 2016      2016 perryman. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to compare the status of prison libraries in croatia to prior studies and ultimately, to guidelines for prison library services (lehmann & locke, 2005). two research questions were asked: 1) how are croatian prison libraries organized and managed? and 2) what kind of library collections and services are offered to incarcerated persons in croatia?   design – quantitative survey.   setting – 23 croatian prison libraries.   subjects – persons in charge of prison libraries.   methods – a paper survey was mailed to all 23 croatian prisons in 2013. the survey consisted of 31 questions grouped into 3 categories: general library information, management of the library, and use. analysis provided descriptive statistics.   main results – twenty-one responses (91%) were received. for the 10 institutions providing data on library holdings size, the numbers ranged from 450 to 6122, but per capita figures were not possible to calculate as no responses provided prison population size. most (65%) maintained an entry book for new acquisitions, while one library kept a card catalogue. half performed collection assessment on an annual basis.   while all but 1 of the prisons had libraries, most (16 of 20) reported that funding was not provided on a regular basis; 13 had space allocated specifically for library purposes, but none were staffed by trained librarians, instead using prison staff or prisoners. only two libraries practised regularly-scheduled collection development, with half acquiring materials solely through donations resulting in limited topical coverage. all collections included monographs, but only around 25% carried newspapers, magazines, music, or videos. while use of the libraries was high, most responses reflected severely limited educational, rehabilitative or cultural programming and access to the internet, and lack of space for collections and reading purposes.   conclusion – libraries in croatia fail to meet international standards for staffing, collections, and services. recommendations for immediate improvement are made, including legislative advocacy and funding, improved public library involvement, and the creation of national standards aligned with international standards.   commentary   the authors provide an overview of prison library research worldwide and of the state of prisons in croatia. overall, however, incomplete descriptions of methodology and the lack of access to survey questions affect the quality of this study. the literature review is extensive and well documented, beginning with an overview of international prison libraries and followed by a section focused on croatia’s prison libraries. these reviews occupy nearly half the length of the article, offering the reader new to this topic a valuable overview of international circumstances in prison libraries, including international agreements that have not been adhered to. with regard to description of methodology, the authors devote a scant two paragraphs to survey questionnaire topics, distribution, and response. no mention is made of question types (e.g., open versus closed-ended, multiple choice, etc.) or order, validation of the survey, or methods of analysis. although readers may assume that the survey was based upon previous studies for comparison purposes, this is not stated.   the study population is representative of the overall population, as the authors attempted to survey all croatian prison libraries, with a high response rate (91%). written permission was obtained from prison officials, and the survey did not collect personal information. whether the authors are involved in service to the settings examined is unclear.   some explanations of results are equally unclear. in particular, the authors discuss prison collection sizes, stating that “in a 2003 study, it was calculated that the size of library collections varied significantly across institutions,” but then state that “although seven respondents stated that their libraries had newspapers, only one respondent provided additional information and indicated that his/her library subscribed to two newspapers” (p.81). it is unknown whether this latter statement refers to the referenced study, or to their own survey results.   the methodologies, analysis, and discussion are somewhat less impressive than the well-researched literature review. overall, there is an overuse of tables (12 in total), which is unnecessary to the readers’ understanding. authors make no mention of limitations of the study, and include only general information about the questions. had the survey questions been included, evaluation and re-use would have been greatly aided.   the research adds to the literature on global prison libraries, which is sparse and limited in detail. a more in-depth literature review, published as a separate work, would enrich this corpus, aiding efforts for greater compliance with prescriptive standards. for practitioners, administrators, social justice researchers, and those employed in global standards organizations, the information provided is a valuable addition and update. however, additional detail would help those who hope to build on the study or to emulate its methods, so that future comparisons are possible. the work is of limited use in direct application, as survey questions are broad in scope, but information about extant practices, collections, and services (e.g., catalogue use and non-book materials made available) aid readers’ awareness.   critical evaluation was performed using the lis-specific glynn checklist (2006), but some aspects of the review process were made more difficult because the referenced croatian surveys which are built upon in this study (de villa, 2007; horvat & nebesny, 2004)  are available only in that language.   references   de villa, s. (2007). zatvorske knjižnice u republici hrvatskoj. master’s thesis, university of zagreb, croatia.   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154   horvat a. & nebesny t. (2004). zatvorske knjižnice u hrvatskoj. in: a. belan-simić and a. horvat (eds.) drugi i treći okrugli stol slobodan pristup informacijama: zbornik radova. zagreb: hrvatsko knjižničarsko društvo, (pp. 129–136).   lehmann v. & locke j. (2005). guidelines for library services to prisoners. the hague: international federation of library associations and institutions. retrieved from http://archive.ifla.org/vii/s9/nd1/iflapr-92.pdf.       evidence summary   twitter users with access to academic library services request health sciences literature through social media   a review of: swab, m., & romme, k. (2016). scholarly sharing via twitter: #icanhazpdf requests for health sciences literature. journal of the canadian health libraries association, 37(1), 6-11. http://dx.doi.org/10.5596/c16-009   reviewed by: elizabeth stovold information specialist, cochrane airways group st george’s, university of london tooting, london, united kingdom email: estovold@sgul.ac.uk   received: 15 may 2017                                                                  accepted: 11 july 2017      2017 stovold. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to analyze article sharing requests for health sciences literature on twitter, received through the #icanhazpdf protocol. design – social media content analysis. setting – twitter. subjects – 302 tweets requesting health sciences articles with the #icanhazpdf tag. methods – the authors used a subscription service called rowfeeder to collect public tweets posted with the hashtag #icanhazpdf between february and april 2015. rowfeeder recorded the twitter user name, location, date and time, url, and content of the tweet. the authors excluded all retweets and then each reviewed one of two sets. they recorded the geographic region and affiliation of the requestor, whether the tweet was a request or comment, type of material requested, how the item was identified, and if the subject of the request was health or non-health. health requests were further classified using the scopus subject category of the journal. a journal could be classified with more than one category. any uncertainties during the coding process were resolved by both authors reviewing the tweet and reaching a consensus. main results – after excluding all the retweets and comments, 1079 tweets were coded as heath or non-health related. a final set of 302 health related requests were further analyzed. almost all the requests were for journal articles (99%, n=300). the highest-ranking subject was medicine (64.9%, n=196), and the lowest was dentistry (0.3%, n=1). the most common article identifier was a link to the publisher’s website (50%, n=152), followed by a link to the pubmed record (22%, n=67). articles were also identified by citation information (11%, n=32), doi (5%, n=14), a direct request to an individual (3%, n=9), another method (2%, n=6), or multiple identifiers (7%, n=22). the majority of requests originated from the uk and ireland (29.1%, n=88), the united states (26.5%, n=80), and the rest of europe (19.2%, n=58. many requests came from people with affiliations to an academic institution (45%, n=136). these included librarians (3.3%, n=10), students (13.6%, n=41), and academics (28.1%, n=85). when tweets of unknown affiliation were excluded (n=117), over 70% of the requests were from people with academic links. other requesters included journalists, clinicians, non-profit organisations, patients, and industry employees. the authors examined comments in the tweets to gain some understanding of the reasons for seeking articles through #icanhazpdf, although this was not the primary focus of their study. a preliminary examination of the comments suggested that users value the ease, convenience, and the ability to connect with other researchers that social media offers. conclusion – the authors concluded that the number of requests for health sciences literature through this channel is modest, but health librarians should be aware of #icanhazpdf as another method through which their users might seek to obtain articles. the authors recommend further research into the reasons why users sometimes choose social media over the library to obtain articles.   commentary   when a research article is unavailable through a journal subscription or open access arrangement, library users would have traditionally made an interlibrary loan request. the internet and social media offer researchers an alternative method for obtaining journal articles, and the authors of this study have examined one such method.   the study was assessed using glynn’s (2006) critical appraisal tool, and scored above the defined threshold of 75% for overall validity. the data collection methods are clearly described, and the study design is appropriate for the objectives of the study. the authors have followed a similar method used in a previous study (gardner & gardner, 2015), thus building on previous research. the authors present their results clearly with absolute numbers and percentages, and their conclusions reflect their results and discussion. they identify future research directions, and encourage readers to use the information available through #icanhazpdf for their own research.   there were some limitations highlighted by the tool which are also identified by the authors in their discussion section. the study population is a convenience sample of twitter users who have posted public requests; therefore, the results are not generalizable, although the author’s findings were broadly consistent with the study by gardner and gardner (2015). guidelines for the #icanhazpdf protocol stipulate that tweets should be deleted when the request is fulfilled; therefore the authors could not be sure that they captured all eligible requests. the authors did not comment on the time period for their data collection, but it is possible that three months’ worth of data may not be representative of all article requests over the course of an academic year.   the number of health sciences requests via #icanhazpdf over the three month period is low, and on the basis of this study is probably not a cause for concern for health librarians. what this study highlights is that people with academic affiliations, who should have access to library services, are seeking scholarly research through social media, which raises the question of why they have chosen to bypass the library. understanding the motivations of users seeking research in this way is key to successfully targeting library services to user needs. a full analysis of this topic was outside the scope of the current study and should be pursued in future work. even so, health science librarians should be aware of scholarly sharing networks, including twitter, so they can educate and engage their users in the principles of responsible sharing of research articles without compromising copyright laws.                 references   gardner, c. c., & gardner, g. j. bypassing interlibrary loan via twitter: an exploration of #icanhazpdf requests. in d. m. mueller (ed.). creating sustainable community: acrl 2015, march 25-28, portland, oregon: conference proceedings. (pp. 95-101). chicago, il: association of college and research libraries. retrieved from: http://www.ala.org/acrl/sites/ala.org.acrl/files/content/conferences/confsandpreconfs/2015/gardner.pdf    glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399.  http://dx.doi.org/10.1108/07378830610692154 microsoft word es_haigh.doc evidence based library and information practice 2006, 1:2    27 evidence based library and information practice       evidence summary    obtaining copyright permission to digitize published works remains a significant  barrier    a review of:  george, carole a.  “testing the barriers to digital libraries: a study seeking copyright  permission to digitize published works.”  new library world 106.1214/1215 (2005): 332‐ 42.    reviewed by:  susan haigh   senior policy officer, library and archives canada  ottawa, ontario, canada  e‐mail: susan.haigh@lac‐bac.gc.ca    received: 20 march 2006            accepted: 3 may 2006      © 2006 haigh. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to assess the effectiveness and  efficiency of the copyright permission‐ seeking process and to suggest  improvements in order to improve  outcomes.     design – workflow study.    setting – carnegie mellon university  libraries, pittsburgh, pa, usa.    sample – a random sample of titles  published 1999‐2001 was selected from the  library’s circulating collection. after  eliminating duplicates, technical reports,  theses, dissertations, and missing items, the  sample comprised 337 titles. of these titles,  70% were books, and 56% were from  commercial publishers.  from this a  working sample of 273 titles was derived,  comprising those titles protected by  copyright and with the rights owner clearly  indicated. about 73% of this working  sample appeared to be out‐of‐print; their  median publication year was 1981.       method – in this two year study (1999‐2001),  a random sample of books was selected, and  pertinent bibliographic and copyright  holder information researched and recorded.  permission letters were sent and, six weeks  later, follow‐up letters were sent to non‐ respondents.  the letter allowed  respondents four options:    1. grant full permission to digitize the  work and provide unrestricted web  access;    evidence based library and information practice 2006, 1:2  28 2. grant permission to digitize the  work and provide read‐only web  access, limited  to carnegie mellon  university users;    3. declare that they do not hold the  rights, and hopefully provide  information to identify and locate  the actual rights holder;    4. deny permission for digitization.     results were then recorded and analyzed.    main results – of the 273 letters mailed, a  clear ‘yes’ or ‘no’ reply was obtained for just  over half (52%) of the documents. sixteen  percent of the rights holders could not be  found (the letter was returned, or a referral  proved impossible to locate and contact).  another 25% of the copyright holders  simply did not reply, and 7% were  otherwise problematic. of the 143 ‘yes’ or  ‘no’ responses, 54% denied permission,  while 46% granted permission. (note: these  percentage figures appear to be erroneously  reversed in table 1 of george’s article.)   therefore, of the overall working sample of  273 titles, permission to digitize was  obtained for only 24% of the titles.     a substantial portion of the permissions (41  of 66, or 62%) carried some restriction. this  represents 15% of the total working sample.  only a few restriction requests were deemed  too great to make use of the permission.     commercial publishers who made up 58%  of the working sample granted permission  at the lowest rate (13%).     response time averaged three months from  the time the initial letter was sent until a  ‘yes’ or ‘no’ response was received.  negative responses averaged a bit longer  than positive responses (101 days to 124  days). however, some of this time was  attributable to delays in issuing follow‐up or  redirected request letters (a step required in  60% of cases), owing to the limited staff  resources at carnegie mellon.     the copyright ownership had changed in  23% of the sample, requiring more than one  and up to three different addresses to be  contacted before a response was received or  the effort was terminated.      conclusions – the study concluded that the  permission rate would remain low unless  additional efforts were made in the  permission‐seeking process (e.g., personal  contacts in addition to letters and emails), or  unless more selective approaches were  employed (e.g., targeting non‐commercial  publishers).  it also concluded that the  process to seek copyright permissions was  neither quick nor easy, suggesting the need  for dedicated staff time and a readily  accessible database of publisher contact  information.  as a result, subsequent  projects have improved their permission‐ seeking process, focusing on more non‐ commercial publishers or older publication  dates, and asking publishers for blanket  consent for all of their out‐of‐print titles.       commentary    the useful data developed by this study is  based on a broad working sample of  published material and helps to identify the  least amount of staff effort needed to obtain  permissions. some context as to u.s.  copyright law might have been useful.  there is an assumption that all readers are  aware of u.s. copyright law, including  length of copyright.    while the study methodology was  appropriate, there were a couple of gaps in  data, pertaining particularly to staff time.  first, the time it took to examine the original  random sample of 337 titles and research  copyright holders (in order to conclude that  some would not be able to be located) did  evidence based library and information practice 2006, 1:2  29 not seem to be factored into the overall  consideration of staff time required. then,  once the working sample was determined,  the first step in the process was to determine  the identity of the copyright holder and  obtain contact information. how much time  did this take on average? it is described as  ‘considerable’, but not quantified.  the  author does mention that it would be  helpful in future projects to monitor the staff  time expended on each step of the process.  but without such data, the study’s aim to  determine efficiency cannot be fully  explained.     the low rate of permission to digitize (24%),  the relatively high rate of restrictions (62%),  and the high rate of non‐response (up to  30% for some publisher types) are all  notably discouraging findings. as google  has discovered with its google print library  project and the u.s. authors guild lawsuit  which it occasioned, copyright remains a  major obstacle to building comprehensive  retrospective digital collections of published  material.     and yet, as the author notes, there is a  growing demand for digital information.  carnegie mellon has determined to continue  its efforts, but with a honed approach. over  time, this will build critical mass, but  inevitably (and unfortunately) the resulting  collections will fall short on more recent and  more commercial titles.  there is no  discussion of whether the demands and  priorities of students and faculty will be met  by the revised scope of the library’s  digitization effort.     the author does not comment on the fact  that many publishers denied permission,  even, presumably, for long out of print titles,  or that many publishers simply chose to  ignore permission requests. the conclusion  does mention that one of their subsequent  projects, the million books collection, which  presumably employs their refined  approaches and strategies, achieves a 54%  permission rate—an improvement, but  hardly ideal.     those seeking to digitize copyright library  material may want to think twice about  doing so after reading this study. if they  persist, its findings provide some guidance  toward setting realistic expectations and  timeframes, as well as pragmatic workflow  processes, for optimal success.  evidence summary   an action research approach helps develop gis programs in humanities and social sciences   a review of: kong, n., fosmire, m., & branch, b. d. (2017). developing library gis services for humanities and social science: an action research approach. college & research libraries, 78(4), 413-427.  http://dx.doi.org/10.5860/crl.78.4.413   reviewed by: laura costello head of research & emerging technologies stony brook university libraries stony brook, new york, united states of america email: laura.costello@stonybrook.edu   received: 1 dec. 2017     accepted: 21 feb. 2018      2018 costello. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29381       abstract   objective – to develop and improve on geographic information systems (gis) services for humanities and social sciences graduate students and faculty members using an action research model.     design – case study.   setting – a public research university serving an annual enrollment of over 41,500 students in the midwestern united states of america.   subjects – faculty members and students in the humanities and social sciences that expressed interest in gis services.      methods – an action research approach was used which included data collection, analysis, service design, and observation. interviews with eight individuals and groups were conducted including four faculty members, three graduate students, and one research group of faculty and graduate students. data from interviews and other data including emails and notes from previous gis meetings were analyzed and the authors used inductive and axial coding to interpret data and organize it into thematic areas. this analysis was used to develop an action plan for the library, then the results of the activity were assessed informally.   main results – the interviews revealed three thematic areas for library gis service: research, learning, and outreach. the authors generated library service connections for each of these areas including providing data plan design support and server space to enable future collaboration. the action plan developed by the authors resulted in increased engagement including active participation in an annual gis day, attendance at workshops, course-integrated gis sessions, around 40 consultations on gis subjects over a two-year period, and increased hits on the library’s gis page. surveys from preand post-tests in the workshops increased participants’ spatial awareness skills.   conclusion – using an action research approach, the authors were able to identify needs and develop a successful model of gis service for the humanities and social sciences.   commentary   this article examines a method for moving gis services forward in an academic library. this function in libraries is not new; large scale efforts to integrate gis literacy and service have been associated with libraries since the early 1990s (argentati, 1997). the practice of providing these services has changed over time and this study is well-situated to scale existing gis programs to new disciplines and help librarians develop the audiences and resources they have cultivated through gis work.   the study employs an action research model which functions as a cycle to continuously produce insights, convert those insights to actions, and analyze the results of those actions. the original interview pool was small at eight participants who were not randomly chosen, but the researchers felt this was sufficient because of the specialized scale of gis work in the humanities and social sciences on campus and strong repetition in the themes of the responses. this repetition could indicate that the interview was representative of all patrons engaged in gis work in the humanities and social sciences and therefore meets the requirements of the critical appraisal (glynn, 2006). the authors used data curation profiles to develop content for the participant interviews which they felt helped structure the interviews away from personal research interests and towards more generalizable data on participants’ needs and expectations. the use of action research in this study was a unique choice and was effective for the goals of the authors. a future cycle of this method could be used with more formal outcomes assessment and a bigger sample size to deepen practice in this area.   these interviews allowed the authors to structure an action plan for library service. the impact of a holistic approach like this, which includes increased outreach, development of learning objects and activities, and the restructuring of a service program, is extremely difficult to measure. the authors provide both qualitative and quantitative indicators in the results, but a clearer picture of the impact of these efforts might emerge in subsequent publications.   this article strongly positions the library as a purveyor of this interdisciplinary service, particularly for institutions that do not have geography programs, and provides a good structure for scaling gis services to the humanities and social sciences. the activities outlined in the article require relatively high levels of investment and expertise in gis, so they may not be relevant to institutions that are just embarking on this service, but libraries with existing investments and mature gis programs will find a robust outline for expanding these programs toward humanities and social sciences researchers.   references   argentati, c. d. (1997). expanding horizons for gis services in academic libraries. the journal of academic librarianship, 23(6), 463-468. http://dx.doi.org/10.1016/s0099-1333(97)90170-1   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech                24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154         research article   understanding factors that encourage research productivity for academic librarians   kristin hoffmann associate librarian university of western ontario london, ontario, canada email: khoffma8@uwo.ca   selinda berg librarian leddy library, university of windsor windsor, ontario, canada email: sberg@uwindsor.ca   denise koufogiannakis associate university librarian university of alberta libraries edmonton, alberta, canada email: dak@ualberta.ca   received: 21 july 2017                                                                   accepted: 23 oct. 2017        2017 hoffman, berg, and koufogiannakis. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   data availability: hoffmann, k., berg, s., & koufogiannakis, d. (2019). factors encouraging research productivity for academic librarians survey instrument and data. ual dataverse, v1. https://doi.org/10.7939/dvn/jpbc81   doi: 10.18438/b8g66f     abstract   objective – this project identifies the factors that contribute to the success of librarians as active researchers. research success is generally aligned with productivity and output, and the authors are therefore interested in understanding the factors that encourage research productivity. this fills a gap in the literature on librarians as researchers, which has tended to focus on barriers rather than enablers.    methods – for this quantitative study, we distributed an online survey to 1,653 potential participants across canada and received 453 usable responses for a 27% response rate. the survey asked participants to report their research outputs and to answer questions that addressed three categories of factors: individual attributes, peers and community, and institutional structures and supports. we then statistically analyzed participant responses in order to identify relationships between the research output variables (weighted output score and number of peer-reviewed articles) and the three categories, the factors within those categories, and the constituent components.   results – participants’ research output consisted largely of presentations, non-peer-reviewed articles, peer-reviewed articles, and posters. all three categories of factors were significantly related to research output, both for a calculated weighted output score and for number of peer-reviewed articles. all of the factors identified within those categories were also significant when tested against weighted output score, but intrinsic motivations was not a significant factor when tested against number of peer-reviewed articles. several components of factors were also not significant for number of peer-reviewed articles. age was the only significant component of demographics. three components of education and experience were significant: whether participants had received research training after completing their mlis, whether they were working on an advanced degree, and the institution where they had obtained their mlis.   conclusions – research productivity is significantly impacted by all three categories: individual attributes, peers and community, and institutional structures and supports. fostering an environment that focuses on all of these areas will be most likely to promote research output for librarians. at the same time, this study’s findings point to particular aspects that warrant further investigation, such as the nature and effect of institutional support and librarians’ motivations for doing research.     introduction   how do we know what enables librarians to be successful researchers? what particular factors contribute to librarians’ dissemination of research? why are some librarians more productive researchers than others?   these are important questions because scholarship, including the dissemination and publication of research, is a professional responsibility for many canadian and american academic librarians. recent initiatives and conversations from the united kingdom and australia suggest that librarians there are also considering ways in which they can support and embrace practitioner-led research. in north america, librarians are often evaluated on their scholarly output as a component of tenure and promotion requirements (sassen & wahl, 2014).   research productivity can be an important element of librarians’ career development and career progression; however, librarians’ enthusiasm and capacity to achieve and maintain a scholarly record is inconsistent. while some librarians have excelled in this aspect of their responsibilities, others have struggled (walters, 2016; o’brien & cronin, 2016). there have been numerous approaches to supporting librarians in their efforts to be productive researchers; however, the impact of these supports has not been well studied.   literature review   the literature is replete with narratives and descriptions of the resources and structures available to support the research success of academic librarians. common supports include writing support groups (campbell, ellis & adebonojo, 2012; exner & harris houk, 2010; fallon, 2012; tysick & babb, 2006), journal clubs (fitzgibbons, kloda, & miller-nesbitt, 2017), support groups or forums for research conversations (carson, colosimo, lake, & mcmillan, 2014; hall & mcbain, 2014; miller & benefiel, 1998; sapon-white, king & christie, 2004), mentorship programs (cirasella & smale, 2011; stephens, sare, kimball, foster, & kitchens, 2011), research skills development initiatives (edwards, jennerich, & ward, 2009; jacobs & berg, 2013; mcbain, hall, & culshaw, 2013; schrader, shiri, & williamson, 2012), research leaves or release time, and funding (smigielski, laning, & daniels, 2014). alongside these supports, canadian academic librarians are actively developing communities within and outside of their institutions to foster a positive research culture across canada (carson et al., 2014; jacobs & berg, 2013; meadows, berg, hoffmann, torabi, & gardiner, 2013; mierke & williamson, 2017; wilson, 2017). two key initiatives towards this goal are the librarians' research institute sponsored by the canadian association of research libraries (carl, 2017) and the centre for evidence based library and information practice (c-eblip, 2017).   there are also numerous articles which describe the level, context, and environment in which librarians conduct their research and scholarship (harrington & gerolami, 2014; pickton, 2016; shaw & szwajcer, 2016). much of the research to date has focused on institutional context. within these articles, authors often address the challenges that librarians face when conducting research and the barriers that may prevent them from being productive researchers (black & leysen, 1994; brown, 2001; fox, 2007; kennedy & brancolini, 2012; lessick et al., 2016; o’brien & cronin, 2016; powell, baker & mika, 2002; shaw & szwajcer, 2016; spring, doherty, boyes, & wilshaw, 2014). commonly noted challenges and barriers include time constraints, lack of support, and lack of research training or experience. to a large extent, the literature highlights factors that impede rather than enable librarians to conduct research. however, some researchers have also asked librarians about their motivations for and perceived benefits of doing research, and those findings present a more positive view: librarians publish for both personal and professional development (o’brien & cronin, 2016), they value personally fulfilling research opportunities (hollister, 2016), and they feel that research helps demonstrate the value of library services and contributes to their evaluation and improvement (lessick et al., 2016).   recently, there has been increasing interest in understanding the research productivity of librarians. in a survey of the research productivity of post-tenure librarians, hollister (2016) asked respondents to share their perceptions of research production preand post-tenure. interestingly, hollister did not quantify the research productivity of individual respondents, but simply asked if respondents had produced particular types of research output. a majority of respondents reported having produced or intending to produce research post-tenure. walters (2016) investigated the influence of four institutional variables (university-wide research activity, eligibility for sabbaticals, university control, and enrollment) on the scholarly productivity of librarians at research universities in the united states. he found that librarians’ research productivity was influenced by university-wide research activity and faculty status. baro and ebhomeya (2012) investigated the research productivity of librarians in nigeria. they found that there was no significant difference in research output between librarians and lecturing faculty. despite obstacles of long hours, heavy workloads, and limited publication options, baro and ebhomeya encourage librarians to recognize and embrace publication as a responsibility for promotion, in similar ways as nigerian faculty have. as demonstrated by this research, interest may be shifting from a focus on barriers that prevent librarians’ scholarly output toward a focus on understanding the level of research done by librarians and the contexts that foster their research productivity.   still, while there is increasing interest in the research productivity of librarians, the factors which increase productivity for librarians have not yet been fully explored. within other academic disciplines, research examining factors that contribute to research productivity are more common (brew, boud, namgung, lucas, & crawford, 2016). research productivity studies have used a variety of methods (e.g., self-reports, bibliometrics) across a wide array of contexts (e.g., different institutions or disciplines). the research environment of librarians is often suggested to be distinct from that of other academic disciplines, requiring unique supports and structures, due to the different nature of academic librarian work. our previous study that identified literature on research productivity both within and outside of academic librarianship suggests many common factors (hoffmann, berg & koufogiannakis, 2014). however, it is not known whether the statistically significant factors for librarians are the same as those of other academics, because there has been little empirical research about factors that influence the research productivity of librarians.   aims   this study aims to fill a gap in the literature by identifying antecedents to the research success of librarians. research success is generally aligned with productivity and output, and we are therefore interested in understanding the factors that encourage research productivity by way of research outputs.    our goal is to develop a better understanding of the factors that influence librarians’ research productivity in canadian academic libraries. table 1 shows the categories and factors examined in this study in order to address the following research questions:   1.       what factors have a positive effect on research productivity? 2.       which of the three categories of factors – individual attributes, peers and community, and institutional structures and supports – are most influential for librarians’ research productivity?   table 1 factors examined in this study (hoffmann et al., 2014) individual attributes peers and community institutional structures and supports demographics education and experience intrinsic motivations personal commitment to research personality traits collaboration community mentoring peer support extrinsic motivations institutional supports   methods   this quantitative study used an online survey for data collection. the online survey was based on previously published research that identified potential factors that may contribute to librarians’ research productivity. the knowledge resulting from this first phase of the research was conducted via content analysis, and the results are described in a previous publication (hoffmann et al., 2014).  we considered the survey instruments used in those studies found via the content analysis to inform the development of survey questions. building from the results of the first phase of the research, survey questions captured participants’ research outputs and explored factors related to three categories: individual attributes (including demographics), peers and community, and institutional structures and supports.   in developing the survey, we made several key decisions to address the research questions. as noted above, we wanted to draw on previous research, identified in the first phase of our project, to give the survey a solid foundation in the existing literature on research productivity. we wanted to determine relationships between factors and research productivity outputs, rather than simply describing participants’ research environments, so we designed questions with binary yes or no answers, which could easily be used to calculate statistical measures. we also decided to focus on what individual participants did, rather than what was available to them; for example, we asked “did you take a sabbatical or other research leave?” rather than “do you have the option to take a sabbatical or other research leave?” we included a question for open-ended comments so that participants could elaborate on answers or add other factors that they felt we had not addressed, since we anticipated that the yes or no answers might leave participants feeling that the complexities of their situations were not captured.   we iteratively pre-tested the survey instrument with twelve librarians who understood the topic, but were not part of the specific population we intended to survey (canadian university librarians), made adjustments based on feedback, and then re-tested the questions. the survey instrument is provided in appendix a.   potential participants were all librarians who worked at the 75 canadian research knowledge network (crkn) member institutions, which are listed in appendix b. we chose these institutions because they are the largest grouping of canadian university academic librarians, and could therefore provide the largest sample of librarians who are likely to have research as part of their job responsibilities. because our population included both englishand french-speaking universities, our survey and recruitment materials were professionally translated into french.   we mined the public websites of each crkn member institution to obtain email addresses of potential participants. each individual received an email invitation to participate in the study. we also recruited through listservs, facebook, and twitter. we emailed the study invitation to 1,683 potential participants in april 2016. we received “mail undeliverable” messages from 30 email addresses, so 1653 potential participants received the invitation.   we asked participants to detail the number of research outputs they had in the past five years. most of the existing literature on research productivity has focused on peer-reviewed journal articles as the measure of research output. based on our understanding of research conducted by librarians, we felt that it was also appropriate to include conference presentations, posters, non-peer-reviewed articles, and books. the types of research output that we included all had some aspect of vetting and featured a dissemination process that the researcher needed to follow. as such, we did not include blogs or other self-posted forms of dissemination. in a comment field, we invited participants to elaborate on non-traditional forms of dissemination, such as blogs. finally, we asked participants to only note their research output that was related to lis. while some canadian academic librarians work at institutions where they are explicitly allowed to research in any discipline, including creative works, others are at institutions that explicitly state that their research must be relevant to librarianship, and we wanted our survey to focus on the kind of research that we all have in common. importantly, participants self-reported their research output, we did not ask for publication details, and so it was the participants themselves who determined whether their output was related to lis. we did not ask participants to indicate their level of involvement or whether they were sole or a co-author on works reported.   table 2 weights for each type of research output output type weight poster 0.5 presentation 1 conference proceeding 1 non-peer-reviewed article 3 book chapter 5 edited book 6 peer-reviewed article 9 authored book 10     in our analysis, we used a weighted output score to have one overall measure of productivity for each participant, and to account for the fact that not all outputs are equal. the weights for each type of output, noted in table 2, were reached via a paired comparison analysis. in this process, we compared each type of output against every other type of output and then we assessed the relative potential impact and contribution of each pair in relation to dissemination. once each pair had been weighted, we added the relative weights to arrive at an overall weight for each type of output. to calculate the weighted output score for each survey participant, we multiplied the overall weights by the number of research outputs of each type to arrive at a weighted output score. for example, if a participant gave their research output as two posters and two presentations, their overall weighted output score was three.   results   we received 556 responses to the survey. after removing incomplete responses, we had 453 responses for a 27% response rate, representing 93% of the crkn member institutions. table 3 and figures 1 and 2 summarize demographic characteristics of our survey respondents. our participants comprised a representative sample of canadian academic librarians, as compared to the 2015 census carried out by the canadian association of professional academic librarians (capal, 2016).   table 3 overview of survey respondents     % gender female 78.5   male 21.5   other < 1 language english 89   french 11 workplace category undergraduate 19   comprehensive 33   medical / doctoral 49   research output   participants disseminated a range of research output over the past five years, from none at all to multiple types and numbers of output. presentations made up nearly half of the total reported output. most of the participants’ output consisted of presentations, non-peer-reviewed and peer reviewed journal articles, and posters – these four types accounted for 89.5% of the total output. table 4 summarizes participants’ reported research output.   as described above in the methods section, we calculated a weighted output score for each participant. the distribution of participants’ weighted output scores is shown in figure 3. the mean score was 21. there were 53 participants who reported no output of any kind, many participants had very low weighted output scores, and a small number of participants had very high weighted output scores. we used a box plot in spss to identify extreme values. all weighted output scores above 67 were identified as outliers and removed from the analysis. as well, we decided to focus our analysis on those participants who had demonstrated some regular engagement in research and therefore we set a lower limit for a weighted output score of three, and removed all participants with weighted output scores below that.   figure 1 age ranges of participants.   figure 2 number of years since participants completed their mlis (or equivalent).   table 4 participants’ reported research output over the past five years (2011-2016) output type min.a max.a mean median st. dev. total number reported % of output reported presentation 0 27 4.1 3 4.7 1846 47.7 non-peer-reviewed article 0 36 1.3 0 3.4 609 15.7 peer-reviewed article 0 14 1.1 0 2.0 492 12.7 poster 0 10 1.0 0 1.6 462 11.9 conference proceeding 0 10 .6 0 1.4 283 7.3 book chapter 0 3 .3 0 .5 116 3 authored book 0 3 .1 0 .3 34 .1 edited book 0 3 .1 0 .3 27 .1 totals           3869 100   figure 3 histogram of participants’ weighted output scores.   since the distribution of weighted output scores does not approximate a normal distribution, we used non-parametric statistical tests to examine the relationship between weighted output score and the identified factors. we used the mann-whitney u test with variables that have two nominal groups, the kruskal-wallis anova for variables with more than two groups, and spearman’s rho for correlations of ordinal variables. for the mann-whitney and kruskal-wallis tests, the null hypothesis is that there is no difference in the distributions; when the null hypothesis is rejected, the difference in the distributions is found to be significant at the .05 level.   effect of factors on research productivity   the questions in the survey addressed eleven factors (table 1) that made up the three overarching categories: individual attributes, peers and community, and institutional structures and supports. each question mapped to one of the factors, as shown in appendix a. some questions or factors straddle more than one category; however, for simplicity, each question was mapped to one factor within one category (hoffmann et al., 2014). to confirm mappings for the yes or no questions, we ran correlations of the responses. in our previous research we had identified a single factor of motivations for research, which we further refined into two factors, intrinsic motivations and extrinsic motivations, as we analyzed the correlations.   to analyze the effects of our identified factors on research productivity, we tested at three levels: the three overarching categories in aggregate, selected factors within those categories,[1] and the individual questions that formed the components of the factors. for each of those three levels, we tested against two measures of research productivity: weighted output score and number of peer-reviewed journal articles.   all three categories were significant, both for weighted output score and number of peer-reviewed articles. in other words, many elements contribute to librarians’ research productivity. there was no single category – not individual attributes, nor peers and community, nor institutional structures and supports – that emerged as being clearly more important than the others, but rather all three were significantly correlated with research output. however, there were noteworthy findings within the eleven factors which made up the three categories, especially when we tested against different measures of research productivity.   within the factor of demographics, only age was significant, and it was only significant when tested against the weighted output score; there was no difference in the distribution of number of peer-reviewed articles over various age ranges. the significance in age came from lower weighted output scores for participants in the age ranges 55-59 and 60-64, as shown in table 5. the other components of demographics evaluated were gender, marital status, and whether a respondent cared for dependents. none of those were significant, neither for weighted output score nor number of peer-reviewed articles.   table 5 median research productivity for age range   weighted output score peer-reviewed articles age range n min. max. median min. max. median 25 29 7 7 53 13.5 0 4 0 30 34 31 3 52 15.0 0 2 0 35 39 58 3 66.5 16.5 0 4 1 40 44 57 3 60 17.0 0 4 1 45 49 50 3 54.5 20.5 0 4 1 50 54 37 3 59 17.0 0 4 1 55 59 22 3 61 10.5 0 6 0 60 64 15 3 67 9.0 0 2 0 65 70 5 3 45 20.0 0 5 0   table 6 median research productivity for institutions where participants obtained their mlis   weighted output score peer-reviewed articles institutiona n min. max. median min. max. median university of alberta 26 3 60 25.0 0 4 1 university of british columbia 33 4 55 16.0 0 4 1 dalhousie university 29 3 61 14.0 0 5 0 mcgill university 34 3.5 59 21.5 0 4 1 université de montréal 28 3 46.5 11.75 0 6 0 university of toronto 56 3 67 17.25 0 5 0 u.s. programsb 14 3 48 11.75 0 2 0 western university 96 3 66.5 17.0 0 2 1   within the factor of education and experience, there were three significant components for both weighted output score and for number of peer-reviewed articles. participants who had received research training after completing their mlis and those who were currently working on an additional advanced degree were more likely to have higher research output. also, there was a statistically significant difference between institutions from which participants received their mlis. table 6 presents median scores by institution for both measures of research productivity. specifically, post-hoc mann-whitney tests indicate that graduates of université de montréal had significantly lower output (both weighted output score and number of peer-reviewed articles) than graduates of mcgill, alberta, british columbia, or western. graduates from u.s. programs also reported a significantly lower number of peer-reviewed articles than those from mcgill, alberta, british columbia, or western, and they had significantly lower weighted output scores than participants from mcgill or alberta. none of the other aspects of education and experience were significant, for either measure of research productivity.   the other nine factors were comprised of the yes or no questions, and we tested both the factors and the individual components. when we tested the nine remaining factors against the weighted output score, all of them were significant. however, when we tested the factors against the number of peer-reviewed articles, intrinsic motivation was no longer a significant factor, and there were many fewer components that were significant on their own. tables 7, 8, and 9 show the significant components for the categories of individual attributes, peers and community, and institutional structures and supports, respectively.   within the individual attributes category (table 7) there is a lot of variation in which components are significant when tested against weighted output scores or number of peer-reviewed articles. when peer-reviewed articles was used as the measure of research productivity, none of the intrinsic motivation components are significant and only 8 out of 29 components in the category are significant.   the peers and community category (table 8) shows less variation in which components are significant when tested against weighted output scores or number of peer-reviewed articles. most components in the peer support factor are not significant, but most components of the other factors are significant. within the collaboration factor, the component “i have done research on my own” is one where answering “no” meant higher collaboration; however, participants who answered “yes” were more likely to have higher weighted output scores.   the institutional structures and supports category (table 9) also shows little variation. interestingly, the component “i do research only because it is a requirement of my job” is the only one that is not significant against weighted output score and is significant against number of peer-reviewed articles, and those who answered “yes” were more likely to have produced lower numbers of articles.   table 7 components of the individual attributes category and their significance as determined by the mann-whitney u test, significant at the .05 level factors and components weighted output score number of peer-reviewed articles intrinsic motivations i do research to contribute to more informed decision making in librarianship. – – i do research to contribute to better library services. – – i do research for my personal interest. – – i do research for professional growth. – – i do research to contribute to greater library visibility on campus. – – i do research to advance my career. significant – i do research to build stronger relationships with faculty members. significant – i do research to build a professional reputation for myself. significant – i do research to contribute to a stronger profession. significant – personal commitment to research i always have a research project that i’m working on. significant significant i schedule dedicated time for research. significant significant i am currently working on a research project. significant significant i have participated in activities that support lis research (e.g. peer review, editor of a journal, providing writing assistance to a colleague, etc.). significant significant i do research that is meaningful to my practice. significant – i consider research to be a priority. significant – i believe it is important for librarians to contribute to the profession via research. significant – i read research literature on a regular basis. significant – i work on research outside of regular work hours. significant – i have used personal funds to support my research and dissemination (e.g.: personal professional development funds or self funded). significant – personality traits i can achieve my research goals. significant significant i am confident that i have the ability to do research. significant significant i finish the research projects that i start. significant significant i can easily identify questions that could be answered through research. significant significant i enjoy speaking with colleagues about my research. – – i enjoy presenting at conferences. – – i do research to satisfy my curiosity. – – publishing gives me a personal sense of satisfaction. significant – i enjoy doing research. significant – i enjoy writing for publication. significant –   table 8 components of the peers and community category and their significance as determined by the mann-whitney u test, significant at the .05 level  factors and components weighted output score number of peer-reviewed articles collaboration i have done research with other people (co-researchers) at my institution. significant significant i have done research with other people (co-researchers) from other institutions. significant significant i have done research on my own. significant – community i feel like i belong to a research community. significant significant i have consulted with an expert to get help on a specific aspect of my research. significant significant i have a network of peers at my institution with whom i talk about research. significant significant i know people who have similar research interests to mine. significant significant i attend conferences in order to connect with others who have similar research interests. – – i have a network of peers from other institutions with whom i talk about research. significant – professional associations are a source of research community for me. significant – mentoring i have been mentored in relation to research activities. significant significant i have mentored others in relation to their research activities. significant significant peer support i have participated in a peer support group related to research. significant significant i ask my colleagues for feedback on my research. – – i have participated in a journal club. – – i have participated in a writing group. – –   table 9 components of the individual attributes category and their significance as determined by the mann-whitney u test, significant at the .05 level  factors and components weighted output score number of peer-reviewed articles extrinsic motivations i have received merit increments or promotion due to my research activities. significant significant i am (formally or informally) expected to participate in research and scholarship. – – i do research only because it is a requirement of my job. – significant institutional supports i have received funding for my research. significant significant i have hired a research assistant to help with research tasks. significant significant i have taken a sabbatical or other kind of leave to work on a research project. significant significant i have space where i am able to work effectively on my research. significant significant i have time to do research within my job. significant significant i am encouraged and supported by my library to do research. – –     finally, we examined participants’ open-ended comments. most comments corresponded to one of the factors that we had identified, especially to elements of institutional structures and supports: time and perceived institutional support. within comments about time, some participants specifically said that they did not have time for research because they had an administrative role, which is an area we did not explore. an unexpected theme emerged around precarious employment; participants who were in contract or part-time positions described an inability to plan for research (for example, because of the time needed to submit research ethics applications) and less supported by their institutions to do research.   discussion   returning to our research questions, we found that the three categories – individual attributes, peers and community, and institutional structures and supports – all had a positive effect on librarians’ research productivity. this is an important finding, since it reinforces that many elements contribute to librarians’ research productivity. figure 4 is a visual representation of how the three categories work together. an environment that embraces all three areas, by encouraging individual attributes, fostering peer and community interaction, and providing institutional supports, will be likely to promote research productivity among librarians.   while our findings unquestioningly show that all three categories have a significant effect on research productivity, they also show that within those categories, there are components that are particularly interesting or that warrant further examination.   for a female-dominated profession such as librarianship, it is both interesting and encouraging that gender did not have a significant effect on research productivity, especially since gender has been shown to be significant for academics generally (aiston & jung, 2015).   within the factor of education and experience, it is not surprising that graduates of the université de montréal have significantly lower research output; until 2007 when the university of ottawa’s program started, montréal’s was the only canadian mlis program offered in french, and many of the participants who commented in french said that research was not part of their job responsibilities. since librarians have long bemoaned the inadequacy of research training received in mlis programs (black & leysen, 1994), it is perhaps also not surprising that having received such training was not significant. however, it is encouraging that participants who received research training after completing their mlis and those who were working on advanced degrees reported significantly higher productivity. further examining these components may help to understand how such experiences can best help librarians in their research endeavours.   a close examination of the institutional structures and supports category shows that three components are not significant for weighted output score:   ·         i am encouraged and supported by my library to do research. ·         i am (formally or informally) expected to participate in research and scholarship. ·         i do research only because it is a requirement of my job.   however, the last of these components is significant when tested against number of peer-reviewed articles; those who answered “no” are more likely to have higher numbers of articles. in other words, participants who do research for reasons other than a job requirement publish more peer-reviewed articles. this suggests that intrinsic rewards might be stronger motivators than institutional expectations, although the components that we examined for intrinsic motivations were not significant when tested against number of peer-reviewed articles. as such, more investigation is needed into what motivates librarians to do research, especially for publication of peer-reviewed articles.   figure 4 three categories contributing to librarians’ research productivity.   in considering these results, it is evident that some librarians who feel that they have the expectation and support to do research are productive researchers, while others are not. similarly, some librarians are productive researchers despite feeling unsupported and not being expected to do research. it is reasonable to imagine that institutional expectations are “powerful motivators” (hollister, 2016, 369) and yet these findings do not support that.   this is a provocative finding when considered together with the open-ended comments that participants provided, where they overwhelmingly expressed a desire to experience a supportive institutional environment for research, and frustration with library environments that were not supportive or that conveyed mixed messages. for example, participants said that research was “an unfunded mandate,” that “management … values our research activity as long as our ‘regular’ work doesn’t suffer,” and that “we are not encouraged to devote much time to research, yet we are expected to in order to obtain continuing status, prestige, annual report grades [sic], etc.”   institutional factors therefore warrant more study. what do librarians mean by “feeling supported” to do research? what does it mean that so many participants wanted to feel supported and expressed frustration with a perceived lack of support, and yet these factors did not have a significant effect on research output? what role do institutional expectations play as motivators for producing research?  in our previous study (hoffmann et al., 2014), we had identified a factor of positive organizational climate, which we did not explore in this study. the current research points to the need to investigate organizational climate in order to gain a fuller understanding of librarians’ research productivity.   we also see some striking differences in factors and components that are significant when tested against weighted output score, but are not significant when tested against number of peer-reviewed articles. this is especially evident in the individual attributes category; intrinsic motivation is only a significant factor when considered against weighted output score, and there is considerable variation in which components are significant.   these findings suggest that, as a collective, librarians must first consider what type of research output they value, in order to have the clearest possible understanding of the factors that will foster their productivity. if librarians want to encourage dissemination of peer-reviewed articles, they may want to focus on a narrower range of factors in order to foster that research output. while peer-reviewed articles are the standard measure of productivity in many disciplines, and by extension, they are considered the goal output for researchers in those disciplines, it is not clear that they are the primary desired research output for librarians. this is seen in the number of peer-reviewed articles reported by our participants, only 12.7% of the total reported research output. this is also supported by shaw and szwajcer’s findings (2016) that only 32% of their sample of conference presentations were also published as peer-reviewed articles. in contrast, tsafe, chiya, and aminu’s (2016) analysis of nigerian librarians found that 69% of total output was journal articles, although they did not distinguish between peer-reviewed and non-peer-reviewed articles, perhaps indicating that dissemination preferences vary by geography.   this apparent tension around the type of research outputs that are highly valued may be related to uncertainty about the value attributed to research that is closely tied to the practice of librarianship. again looking at the individual attributes category, several of the components that are not significant when tested against peer-reviewed articles reflect a focus on professional engagement:   ·         i do research to build stronger relationships with faculty members. ·         i do research to build a professional reputation for myself. ·         i do research to contribute to a stronger profession. ·         i believe it is important for librarians to contribute to the profession via research. ·         i do research that is meaningful to my practice.   participants who focus on peer-reviewed articles may therefore be less motivated to tie their research to their practice. or it may be that other venues are perceived to be better for disseminating research that is related to practice. in open-ended comments, participants again expressed uncertainty around this element. for example, “there can be pressure from within your library to do certain types of research (very practice-oriented to your specific library), which might not align with your personal research interests,” and “... i mentioned that i was not highly interested in research but enjoyed presenting at conferences and feel that research and conference presenting are different, my supervisor sees these two as research.”   as mentioned above, canadian academic librarians have been working to develop a research culture for themselves. as this continues, it will be helpful to include conversations about how we, as a profession, want to value and promote various types of research output, what we mean by research that is tied to practice, and how we value that research. further study of these questions may also result in more informed conversations.   overall, this study confirms that the categories and factors we identified in our previous research are relevant and important. the issue of precarious employment in academia has gained attention in recent years, although challenges with contract or part-time work are not new (feldman & turnley, 2004), and this may be an additional element to explore in future studies.   the tool that we have developed may be useful for examining research productivity in the future, perhaps as research culture becomes more finely tuned for canadian academic librarians. it may also be useful for surveying other populations beyond canada in order to see if there is any variation in which factors are significant and to see what more we can learn about librarians’ research output and productivity.   there are several limitations to our chosen study design. our study participants were self-selected, so the results reflect a self-selection bias. as well, we were not able to control participants’ responses to ensure that they were replying as we intended. for example, questions about research output asked participants to provide counts for the last five years, but it is possible that someone reported publications over their career. the questions with bivariate variables (yes or no answers) were helpful for our analysis, but also limited the level of detail in the responses and restricted the scope of possible statistical tests that we could run. because we focused our analysis on participants who had some regular engagement with research, as determined by weighted output scores between 3 and 67, we do not know what factors are significant for the participants excluded from analysis. finally, the quantitative approach of this study means that we are not able to capture the full complexity of individual factors; though we can identify which factors are statistically significant, but we cannot explain why this is the case. we are also not able to take into account the context of the individuals who participated in the survey; for example, whether a particular situation in someone’s institution or personal life has affected their research productivity.   conclusions   the findings from this quantitative study contribute to a greater understanding of librarians’ research productivity and the factors that contribute to research success. while we might have hoped for the findings to reveal a ‘magic bullet’ for research output, they instead reveal even more complexity. research productivity is significantly impacted by individual qualities, by interaction and support from peers and community, and by strong institutional supports. these findings suggest that librarians and library administrators focus on all three of these areas in order to promote research productivity.   at the same time, these findings raise additional questions and highlight aspects where more investigation is needed. our participants’ expressed desire for supportive institutional climates is in tension with the finding that feeling supported by one’s institution and feeling expected to do research are not significantly related to research output. further examination of librarians’ motivations for doing research, and of the interplay between intrinsic and extrinsic motivations, may help to illuminate the role of a supportive institutional climate.   also related to motivations, our findings suggest that it will be important to explore questions about the value of research that is closely related to practice and the value of various types of research output. this may help to build a more cohesive research culture and may also help reveal factors that are key for different types of research, since we saw that the significance of some factors, and of some components within factors, varied depending on the measure of research productivity that we examined.   we plan to further extend this research with qualitative exploration into one or more of the areas above. we hope that this study’s findings will prompt others to also explore librarians’ research from the perspective of enablers rather than barriers.   acknowledgements   the authors wish to thank kristi thompson, data librarian at the university of windsor, for advice and assistance on data analysis. we also thank our 12 colleagues who pre-tested our survey instrument and provided valuable feedback.   references   aiston, s. j., & jung, j. (2015). women academics and research productivity: an international comparison. gender and education, 27(3), 205-220. https://doi.org/10.1080/09540253.2015.1024617   baro, e. e., & ebhomeya, l. (2012). a comparative study of the publication output of librarians and academics in universities in the south-south zone of nigeria. journal of scholarly publishing, 43(2), 200-219. https://doi.org/10.1353/scp.2012.0009    black, w. k., & leysen, j. m. (1994). scholarship and the academic librarian. college & research libraries, 55(3), 229-241. https://doi.org/10.5860/crl_55_03_229   brew, a., boud, d., namgung, s. u., lucas, l., & crawford, k. (2016). research productivity and academics’ conceptions of research. higher education, 71(5), 681-697. https://doi.org/10.1007/s10734-015-9930-6   brown, j. m. (2001). time and the academic librarian. portal: libraries and the academy, 1(1), 59-70. https://doi.org/10.1353/pla.2001.0009   campbell, k., ellis, m., & adebonojo, l. (2012). developing a writing group for librarians: the benefits of successful collaboration. library management, 33(1/2), 14-21. https://doi.org/10.1108/01435121211203284   canadian association of professional academic librarians. (2016). 2016 census of canadian academic librarians. user guide and results summary. retrieved from http://capalibrarians.org/wp/wp-content/uploads/2016/12/census_summary_and_user_guide_december_16_2016.pdf   canadian association of research libraries. (2017). librarians’ research institute. retrieved from http://www.carl-abrc.ca/strengthening-capacity/workshops-and-training/librarians-research-institute/   centre for evidence based library and information practice (2017). vision and mission. retrieved from http://library.usask.ca/ceblip/about/vision--mission.php   carson, p., colosimo, a. l., lake, m., & mcmillan, b. (2014). a “partnership” for the professional development of librarian researchers. partnership: the canadian journal of library and information practice and research, 9(2). https://doi.org/10.21083/partnership.v9i2.3037   cirasella, j., & smale, m. a. (2011). peers don’t let peers perish: encouraging research and scholarship among junior library faculty. collaborative librarianship, 3(2), 98-109. retrieved from http://digitalcommons.du.edu/collaborativelibrarianship/vol3/iss2/5   edwards, p. m., jennerich, e. z., & ward, j. l. (2009). supporting a culture of library research at the university of washington at seattle. in e. connor (ed.), an introduction to staff development in academic libraries (pp. 77-89). new york, ny: routledge.   exner, n., & houk, a. h. (2010). starting the write way: comparing two library scholarly development programs. library leadership & management, 24(4), 178-182. retrieved from https://journals.tdl.org/llm/index.php/llm/index   fallon, h. (2012). using a blended group learning approach to increase librarians’ motivation and skills to publish. new review of academic librarianship, 18(1), 7-25. https://doi.org/10.1080/13614533.2012.654673   feldman, d. c. & turnley, w. h. (2004). contingent employment in academic careers: relative deprivation among adjunct faculty. journal of vocational behavior, 64(2), 284-307. https://doi.org/10.1016/j.jvb.2002.11.003   fitzgibbons, m., kloda, l., & miller-nesbitt, a. (2017). exploring the value of academic librarians’ participation in journal clubs. college & research libraries, 78(6), 774-788. https://doi.org/10.5860/crl.78.6.774   fox, d. (2007). finding time for scholarship: a survey of canadian research university librarians. portal: libraries and the academy, 7(4), 451-462. https://doi.org/10.1353/pla.2007.0041   hall, l. w., & mcbain, i. (2014). practitioner research in an academic library: evaluating the impact of a support group. the australian library journal, 63(2), 129-143. http://dx.doi.org/10.1080/00049670.2014.898238   harrington, m. r., & gerolami, n. (2014). highs and lows: an examination of academic librarians’ collective agreements. in j. dekker & m. kandiuk (eds.), in solidarity: academic librarian labour activism and union participation in canada (pp. 151-169). sacramento, ca: library juice press.   hoffmann, k., berg, s. a., & koufogiannakis, d. (2014). examining success: identifying factors that contribute to research productivity across librarianship and other disciplines. library and information research, 38(119), 13-28. retrieved from http://www.lirgjournal.org.uk/lir/ojs/index.php/lir/index   hollister, c. v. (2016). an exploratory study on post-tenure research productivity among academic librarians. journal of academic librarianship, 42(4), 368-381. https://doi.org/10.1016/j.acalib.2016.04.021   jacobs, h. l. m., & berg, s. a. (2013). by librarians, for librarians: building a strengths-based institute to develop librarians’ research culture in canadian academic libraries. journal of academic librarianship, 39(3), 227-231. https://doi.org/10.1016/j.acalib.2013.02.003   kennedy, m. r., & brancolini, k. r. (2012). academic librarian research: a survey of attitudes, involvement, and perceived capabilities. college & research libraries, 73(5), 431-448. https://doi.org/10.5860/crl-276   lessick, s., perryman, c., billman, b. l., alpi, k. m., de groote, s. l., & babin, t. d. (2016). research engagement of health sciences librarians: a survey of research-related activities and attitudes. journal of the medical library association: jmla, 104(2), 166-173. https://doi.org/10.3163/1536-5050.104.2.015   mcbain, i., culshaw, h., & hall, l. w. (2013). establishing a culture of research practice in an academic library: an australian case study. library management, 34(6/7), 448-461. https://doi.org/10.1108/lm-08-2012-0053   meadows, k. n., berg, s. a., hoffmann, k., torabi, n., & gardiner, m. m. (2013). a needs-driven and responsive approach to supporting the research endeavours of academic librarians. partnership: the canadian journal of library and information practice and research, 8(2). https://doi.org/10.21083/partnership.v8i2.2776   mierke, j., & williamson, v. (2017). a framework for achieving organizational culture change. library leadership & management, 31(2). retrieved from https://journals.tdl.org/llm/index.php/llm/index   miller, j. p., & benefiel, c. r. (1998). academic librarians and the pursuit of tenure: the support group as a strategy for success. college & research libraries, 59(3), 260-265. https://doi.org/10.5860/crl.59.3.260   o’brien, t., & cronin, k. (2016). research output of academic librarians from irish higher education institutions 2000-2015: findings from a review, analysis, and survey. new review of academic librarianship, 22(2-3), 203-224. https://doi.org/10.1080/13614533.2016.1181666   pickton, m. (2016). facilitating a research culture in an academic library: top down and bottom up approaches. new library world, 117(1/2), 105-127. https://doi.org/10.1108/nlw-10-2015-0075   powell, r. r., baker, l. m., & mika, j. j. (2002). library and information science practitioners and research. library & information science research, 24(1), 49-72. https://doi.org/10.1016/s0740-8188(01)00104-9   sapon-white, r., king, v., & christie, a. (2004). supporting a culture of scholarship for academic librarians. portal: libraries and the academy, 4(3), 407-422. https://doi.org/10.1353/pla.2004.0044   sassen, c., & wahl, d. (2014). fostering research and publication in academic libraries. college & research libraries, 75(4), 458-491. https://doi.org/10.5860/crl.75.4.458   schrader, a. m., shiri, a., & williamson, v. (2012). assessment of the research learning needs of university of saskatchewan librarians: a case study. college & research libraries, 73(2), 147-163. https://doi.org/10.5860/crl-235   shaw, c. e., & szwajcer, a. l. (2016). publication rate of presentation abstracts presented at the canadian health libraries association (chla/absc) annual meetings from 2004-2009. performance measurement and metrics, 17(3), 252-262. https://doi.org/10.1108/pmm-07-2016-0034   smigielski, e. m., laning, m. a., & daniels, c. m. (2014). funding, time, and mentoring: a study of research and publication support practices of arl member libraries. journal of library administration, 54(4), 261-276. https://doi.org/10.1080/01930826.2014.924309   spring, h., doherty, p., boyes, c., & wilshaw, k. (2014). research engagement in health librarianship: outcomes of a focus group. library & information science research, 36(3-4), 142-153. https://doi.org/10.1016/j.lisr.2014.06.004   stephens, j., sare, l., kimball, r., foster, m., & kitchens, j. (2011). tenure support mechanisms provided by the faculty research committee at texas a&m university libraries: a model for academic libraries. library management, 32(8/9), 531-539. https://doi.org/10.1108/01435121111187897   tsafe, a. g., chiya, u., & aminu, b. a. (2016). scholarly publications of librarians in universities in nigeria: 2000-2012 – a bibliometric analysis. library philosophy and practice. retrieved from http://digitalcommons.unl.edu/libphilprac/1394   tysick, c., & babb, n. (2006). perspectives on... writing support for junior faculty librarians: a case study. journal of academic librarianship, 32(1), 94-100. https://doi.org/10.1016/j.acalib.2005.10.005   walters, w. h. (2016). the faculty subculture, the librarian subculture, and librarians’ scholarly productivity. portal: libraries and the academy, 16(4), 817-843. http://doi.org/10.1353/pla.2016.0054   wilson, v. (2017). the c-eblip fall symposium: librarians as researchers. evidence based library and information practice, 12(2), 4-5. https://doi.org/10.18438/b8rs9x   appendix a survey instrument (english version)   in the survey text below, each question is annotated with an abbreviation to indicate the factor to which it is mapped. these annotations are provided for this paper and were not included in the survey instrument. d            demographics ee          education and experience em         extrinsic motivations im          intrinsic motivations is            institutional supports pcr       personal commitment to research pt          personality traits col       collaboration com     community m            mentoring ps           peer support   factors influencing research productivity   the survey is expected to take less than 20 minutes and includes questions related to: a.) education and professional experience; b.) factors influencing research productivity; c.) demographic information; d.) research outputs;   some of the questions are simple yes or no questions and require you to choose the best answer that reflects your situation or your feelings.   the study seeks participation from canadian academic librarians (at ckrn institutions) who are and who are not active researchers. for this study, we are using the definition of research provided by the tri-council policy statement: research is “defined as an undertaking intended to extend knowledge through a disciplined inquiry and/or systematic investigation.”   the researchers are interested in canadian academic librarians’ contributions to library and information studies (lis) research. while it is recognized that librarians may undertake research outside of lis, the researchers are gathering information in this study only on lis-related research.   by taking this online survey i am indicating that i have read the information letter and voluntarily agree to participate in the research study.   please remember to print a copy of the information letter for your records.   where do you currently work? ee drop-down menu of the 75 ckrn institutions.   do you have tenure (or equivalent) or are you in a tenure-track (or equivalent) position? ee yes no   what year did you complete your mlis degree (or equivalent)? ee drop-down menu of years   where did you obtain your mlis degree (or equivalent?) ee dalhousie university mcgill university university of alberta university of british columbia université de montréal university of ottawa university of toronto western university other, please specify...   during your mlis program (or equivalent), did you complete any of the following: ee check all that apply. research methods course independent research study thesis   since completing your mlis (or equivalent), have you taken any formal research training? ee check all that apply. university-level research course full or half-day research workshop carl librarians' research institute thinking qualitative workshop series online non-credit research course other, please specify...   do you have an advanced degree in addition to your mlis (or equivalent)? ee check all that apply. yes, thesis-based masters yes, non-thesis-based masters yes, phd no additional degree other, please specify...   are you currently working towards an additional degree? ee check all that apply. yes, thesis-based masters yes, non-thesis-based masters yes, phd no additional degree other, please specify…   please indicate whether or not each statement applies to you. (presented in random order) pcr i consider research to be a priority. yes | no pcr i am currently working on a research project. yes | no pcr i always have a research project that i’m working on. yes | no pcr i do research that is meaningful to my practice. yes | no pcr i believe it is important for librarians to contribute to the profession via research. yes | no pcr i work on research outside of regular work hours. yes | no pcr i schedule dedicated time for research. yes | no pcr i have participated in activities that support lis research (e.g. peer review, editor of a journal, providing writing assistance to a colleague, etc.) yes | no pcr i have used personal funds to support my research and dissemination (e.g.: personal professional development funds or self funded). yes | no pcr i read research literature on a regular basis. yes | no is i am encouraged and supported by my library to do research. yes | no is i have time to do research within my job. yes | no is i have space where i am able to work effectively on my research. yes | no is i have taken a sabbatical or other kind of leave to work on a research project. yes | no is i have hired a research assistant to help with research tasks. yes | no is i have received funding for my research. yes | no com i have a network of peers at my institution with whom i talk about research. yes | no com i have a network of peers from other institutions with whom i talk about research. yes | no com i know people who have similar research interests to mine. yes | no com professional associations are a source of research community for me. yes | no com i attend conferences in order to connect with others who have similar research interests. yes | no com i feel like i belong to a research community. yes | no com i have consulted with an expert to get help on a specific aspect of my research. yes | no col i have done research with other people (co-researchers) at my institution. yes | no col i have done research with other people (co-researchers) from other institutions. yes | no col i have done research on my own. yes | no ps i have participated in a peer support group related to research. yes | no   please indicate whether or not each statement applies to you. (presented in random order) ps i have participated in a writing group. yes | no ps i have participated in a journal club. yes | no ps i ask my colleagues for feedback on my research. yes | no em i have received merit increments or promotion due to my research activities. yes | no em i am (formally or informally) expected to participate in research and scholarship. yes | no em i do research only because it is a requirement of my job. yes | no pt i enjoy doing research. yes | no pt i enjoy writing for publication. yes | no pt i am confident that i have the ability to do research. yes | no pt i can achieve my research goals. yes | no pt i enjoy presenting at conferences. yes | no pt i enjoy speaking with colleagues about my research. yes | no pt publishing gives me a personal sense of satisfaction. yes | no pt i can easily identify questions that could be answered through research. yes | no pt i do research to satisfy my curiosity. yes | no pt i finish the research projects that i start. yes | no im i do research to advance my career. yes | no im i do research for my personal interest. yes | no im i do research to contribute to better library services. yes | no im i do research for professional growth. yes | no im i do research to build a professional reputation for myself. yes | no im i do research to contribute to more informed decision making in librarianship. yes | no im i do research to contribute to greater library visibility on campus. yes | no im i do research to build stronger relationships with faculty members. yes | no im i do research to contribute to a stronger profession. yes | no m i have been mentored in relation to research activities. yes | no m i have mentored others in relation to their research activities. yes | no   what is your gender? d female male other   what month and year were you born? d   what is your marital status? d single married living with partner divorced separated widowed other, please specify...   do you have children or adults who depend on you for care? d child(ren) under 18 years of age child(ren) over 18 years of age other adult dependent upon me for care no children or dependent adult   can you think of other factors that were not fully captured in the previous questions that have affected your research productivity? if so, please share them here. open text box   thinking back over the last five years, please indicate how many times you have disseminated your lis-related research in each of the following venues: the researchers are interested in the research outputs of canadian academic librarians related to library and information studies (lis). while it is recognized that librarians may undertake research outside of lis, do research that is not disseminated, or disseminate research in non-traditional formats, in this question the researchers are gathering information about specific ways of disseminating lis-related research.   presented a poster at a conference (both peer reviewed and not) drop-down 0-50 gave an oral presentation at a conference (both peer reviewed and not) drop-down 0-50 published in conference proceedings drop-down 0-50 published a non-peer reviewed journal article drop-down 0-50 published a peer reviewed journal article drop-down 0-50 published a chapter in a book (contributed chapter) drop-down 0-50 authored a book (solo or co-author) drop-down 0-50 edited a book (collection of contributed chapters) drop-down 0-50   the scholarly landscape is changing and researchers are disseminating their research outputs in new ways. please list any ways that you have disseminated your research that were not included in the previous question. open text box   appendix b canadian research knowledge network (crkn) member institutions   acadia university algoma university athabasca university bishop’s university brandon university brock university cape breton university carleton university concordia university concordia university college of alberta dalhousie university école polytechnique de montréal hec montréal kwantlen polytechnic university lakehead university laurentian university macewan university mcgill university mcmaster university memorial university of newfoundland mount allison university mount royal university mount saint vincent university nipissing university nscad university ocad university queen’s university royal military college of canada royal roads university ryerson university saint mary’s university simon fraser university st. francis xavier university the king’s university college of alberta thompson rivers university trent university trinity western university université de moncton université de montréal université de sherbrooke université du québec:   école nationale d’administration publique   école de technologie supérieure   institut national de la recherche scientifique   université du québec à chicoutimi   université du québec à montréal   université du québec à rimouski   université du québec à trois-rivières   université du québec en abitibi-témiscamingue   université du québec en outaouais   télé-université du québec université laval université sainte-anne university of the fraser valley university of alberta university of british columbia university of calgary university of guelph university of lethbridge university of manitoba university of new brunswick university of northern british columbia university of ontario institute of technology university of ottawa university of prince edward island university of regina university of saskatchewan university of toronto university of victoria university of waterloo university of windsor university of winnipeg vancouver island university western university wilfrid laurier university york university           [1] we could not test demographics or education and experience as factors, only the individual questions comprising those factors, because the forms of the questions did not lend themselves to being combined in aggregate. research article   academic librarians’ knowledge of bibliometrics and altmetrics   tara malone assistant professor and librarian department of health sciences library and information management robert m. bird library university of oklahoma health sciences center oklahoma city, oklahoma, united states of america email: tara-malone@ouhsc.edu   susan burke associate professor university of oklahoma school of library and information studies norman, oklahoma, united states of america email: sburke@ou.edu   received: 18 apr. 2016   accepted: 9 july 2016           2016 malone and burke. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to measure the knowledge and opinions that academic librarians have of established and emerging research metrics.   methods – an online survey was distributed to all academic librarians in oklahoma during summer 2015.   results – librarians were less familiar with altmetrics than with bibliometrics, but they viewed altmetrics as effective and were interested in receiving training to learn more about them. librarians who had been in the profession for over five years knew more about both bibliometrics and altmetrics than newer librarians.   conclusions – technological advances and changes in the ways that research products are shared have led to the possibility of and need for new ways of measuring research impact. altmetrics have emerged to fill this need, but academic librarians need more familiarity and training to be able to fulfill a role as providers of these metrics.   introduction   with the advent of social media, digital publishing, and born-digital research, scholarly research impact is changing. traditional methods of evaluating research impact, such as journal impact factor (jif) and citation counts, have long served as benchmarks of research productivity. more recently, alternative metrics for assessing impact, altmetrics, have emerged. in addition to citation counts, altmetrics track the impact of individual research articles and other forms of scholarly output via media attention, article views and downloads, database inclusion, and more (cooper, 2015).   the roles of academic librarians are evolving to include the provision of bibliometrics services to researchers, including both established and emerging measures. as there are more tools available for tracking altmetrics, yet no established standards for conducting altmetrics analysis, it is challenging for academic librarians to help investigators and students understand and use these new measures to complement traditional metrics. while the literature strongly advocates for academic librarians to provide scholarly communication metrics, the fluency of librarians in these methods has yet to be established. this study aims to provide data on this topic.   literature review   research impact, old and new   citation analysis and the jif were conceived by eugene garfield and are widely considered the forerunners of modern bibliometrics (carpenter, 2014; herther, 2013). originally intended to assist librarians in assessing journal subscriptions, the jif is now commonly used during tenure and promotion review to assess the quality of a researcher’s work by making assumptions about the value of publications in part by whether they appeared in high impact journals. this common misapplication of the jif has mired the metric in controversy. criticisms of the jif include “gaming” the numbers through self-citation; compulsory citations imposed at the behest of journals; questions of mathematical validity; failures to replicate the metric’s calculations; and lack of comparability between disciplines (bollen, van de sompel, smith, & luce, 2005; brown, 2014; carpenter, 2014; neylon & wu, 2009). much of the jif is derived from citations to a small percentage of its articles, and a lack of context surrounds most citation counts (belter, 2015).   measuring research quality based on citation counts can be problematic due to the years it takes for a citation’s impact to be revealed, and the lack of journals in some disciplines (adams & bullard, 2014; brown, 2014; neylon & wu, 2009). new measures based on citation counts have emerged with varying degrees of success. among them are the h-index, defined as the number of an author’s papers with citations equal to or higher than h (hirsch, 2005); the g-index, which places more emphasis on highly cited papers; and google’s i10-index, based on the number of a researcher’s publications that have garnered at least 10 citations (gutierrez, beall, & forero, 2015).   while the jif and citation counts continue to be instrumental in shaping individual researchers’ careers, these measures have not kept pace with the explosion of the digital dissemination of scholarship (carpenter, 2014). as technology increasingly reshapes the research environment, more scholarly works are being shared via such social media sites as researchgate, academia.edu, facebook, twitter, or even pinterest, as well as through blogs and online reference managers (adie & roe, 2013; bar-ilan et al., 2013).   altmetrics track mentions of journal articles, data sets, presentations, and other research products on social media; bookmarks and downloads in online reference managers; mentions in popular media; data or slide sharing sites; and other web-based forums where scholarship is being shared (acrl research planning and review committee, 2014; adams & bullard, 2014; bonn, 2014). scholarly publishers are moving into the altmetrics field. for example, wiley, elsevier, and nature have formed partnerships with altmetrics.com, an early aggregator of web impact indicators (bornmann, 2014; brigham, 2014; information today newsbytes, 2014). elsevier recently purchased the online reference manager mendeley, while plum analytics and their research database, plumx, were purchased by ebsco in 2014. as a whole, interest in and adoption of altmetrics tools seem to be growing steadily (roemer & borchardt, 2013).   excitement about the potential of altmetrics to revolutionize research impact measurement is extensive in the lis field. one of these new measures’ attractive features is their speed. traditional citations may take years to yield measurable impact, while altmetrics can theoretically reveal impact in weeks, days, or even minutes (brown, 2014; dinsmore, allen, & dolby, 2014; lapinski, piwowar, & priem, 2013; piwowar & priem, 2013). also important is the possibility of measuring a broader scope of materials and “products” in addition to traditional manuscripts (bornmann, 2014, 2015; herther, 2013; howard, 2015; lapinski et al., 2013; piwowar & priem, 2013).   there are caveats to using altmetrics including: author disambiguation and numbers gaming (bornmann, 2014; brigham, 2014; brown, 2014; galligan & dyas-correia, 2013); the sheer volume of data needed to track altmetrics through the internet is daunting (adie & roe, 2013); and traditional measures like citation analysis and jif are embedded in the promotion and tenure process (bazeley, waller, & resnis, 2014).   the chief critique of altmetrics is the lack of empirical examination, standardization, or regulation (bornmann, 2014; brigham, 2014; carpenter, 2014; herther, 2013; lapinski et al., 2013). at this stage of development, altmetrics are more suited to complementing traditional metrics than to supplanting them. the national information standards organization (niso) has identified 25 key areas in which altmetrics need standardization, including the identification of research types to be tracked, more empirical investigation into the use of altmetrics as research impact measures, and strategies to address potential manipulation (carpenter, 2014; gunn, 2014; herther, 2013; national information standards organization, 2014) .   the emerging role of the academic librarian   research libraries are being called upon to provide improved research support services and access to bibliometric tools. this is a natural extension of the lis field in which bibliometrics practice and research has been situated. new metrics provide the opportunity for academic librarians to practise their skills such as database navigation and analysis, familiarity with tools such as web of science and scopus, and experience with the university promotion and tenure processes, in a new context (åstrom & hansson 2012; bladek 2014; brigham, 2014; brown, 2014; gumpenberger, wieland, & gorraiz, 2012; herther 2013; kennan, corrall, & afzal, 2014; maccoll 2010a; maccoll 2010b; roemer and borchardt 2013; roemer & borchardt, 2015a).   despite the notion that bibliometrics and altmetrics services are an excellent fit for the academic library, it may be that academic librarians are not, as a whole, educationally prepared to provide them. formal training in bibliometrics is a rarity for librarians, which has led to a call for the incorporation of bibliometrics education into the generalized lis curriculum. in many other countries where interest and emphasis may be strong, formal education or training also is lacking (bladek 2014; kennan, corrall, and afzal 2014; zhao 2011).   current research on altmetrics and academic librarians   roemer and borchardt (undated) state that “academic librarians have been and continue to be involved with altmetrics at every level,” but quantitative measures of that involvement are scarce in the lis literature. in a november 2015 conference paper, konkiel, sutton, and levine-clark reported on some aspects of a large-scale survey of u.s. academic librarians’ knowledge and use of altmetrics. while the bulk of those data are not yet published and available for review, preliminary results from the study indicate a low familiarity with or use of altmetrics when compared to more traditional metrics such as the jif or citation counts. respondents to the team’s survey also indicated low numbers of reference encounters surrounding metrics of any kind. librarians’ interest in altmetrics and bibliometrics is not just a u.s. phenomenon; it is international. kennan, et al. (2014) surveyed librarians in australia, the u.k., ireland, and new zealand in 2012 and reported on their opinions predominately on bibliometrics, with some mention of altmetrics. there is also an unpublished research report on the use of altmetrics by spanish librarians and scholars (gonzález-fernández-villavicencio, dominguez-aroca, calderón-rehecho, & garcia-hernández, 2015). the scarcity of published studies illustrates the need for more concrete knowledge on this topic. what do academic librarians truly know about altmetrics?   aims   before best practices can be established in the area of providing research impact services to faculty or students, it is important to ascertain the level of awareness that academic librarians have of new and traditional metrics and the services that provide them. research questions for the study were:   are academic librarians more familiar with established measures of research impact (bibliometrics) than with the emerging field of altmetrics? what are the attitudes of academic librarians toward altmetrics versus traditional measures (bibliometrics)? are academic librarians being called upon by faculty to provide information about new research impact measures, and if so, what has characterized these interactions? what are academic libraries doing, or what could they be doing regarding altmetrics?   the development of the research questions and identification of the independent variables for the study were based on two concepts. first, simple logic suggested that bachelor’s degree and higher granting institutions would be more engaged with publication and research impact, and librarians there would be more likely to be familiar with research impact measures because they would be called upon to engage with them. it also seemed logical that reference librarians would be more likely to explore research impact in their interactions with faculty and students. the idea that librarians newer to the profession would have more knowledge of the newer metrics reflects thomas kuhn’s (1962) suggestion that new ideas are generally promulgated by new generations and not by the older professionals. in other words, technological advances have made it possible to track the impact of research outputs in new ways, as well as boosting the ease of traditional bibliometrics. but has the availability of new metrics generated a paradigm shift or even a revolution in tracking research impact?   methods   this study targeted academic librarians in the state of oklahoma. publicly available information on academic library websites for all 2-year and 4+ year colleges and universities in the state (n=38) was used to gather names and email addresses of librarians, and all but three institutions had this information posted. in this study, “librarian” was defined as a person whose job title was “librarian,” and was not limited to people with specific educational backgrounds (such as an mlis). in total, 228 librarians, 38 at 2-year institutions and 190 at 4+ year institutions, were identified and in july 2015 were emailed a survey solicitation with a link to the survey instrument. two follow up emails were sent in august to generate more responses. as an incentive for participation, the opportunity to win a $20 amazon gift card was offered. the survey consisted of seven open-ended questions and thirteen closed-ended questions including seven likert-type, two “check all that apply,” and four others. these questions were designed to answer the research questions provided above. see appendix for the survey instrument.   limitations of the study   only librarians in oklahoma were surveyed and these results may not be reflective of librarians outside the state. this was a population survey, not a sample survey, so inferential statistics (such as significance testing) were not appropriate. three-quarters of those surveyed did not respond and it is unknown whether they differed on some characteristics from those who did respond. since survey responses were anonymous, a comparison of responders to non-responders is not possible. with these limitations it is not appropriate to generalize the study findings outside of the group of librarians who responded. additionally, the survey presented the term “altmetrics” without defining it. respondents may have had different definitions in mind when answering the survey questions.   results   survey response rate   a total of 58 usable responses were received for an overall response rate of 25.4%. by type of institution, ten librarians at 2-year institutions responded, for a response rate of 26.3%. responses were received from 48 librarians at 4+ year institutions resulting in a response rate of 25.3% for that group.   description of the variables   there were two types of information used as dependent variables. one type was respondents’ knowledge or opinions of various research impact measures as indicated by their answers to survey questions. the second type of dependent variables was scores on two items. one of these scores, bibliometrics familiarity, was computed by adding up the number of bibliometrics items from question 3 with which respondents reported familiarity. their score on this item could range from zero (not familiar with any of the listed items) up to five (familiar with all of them). the actual scores from these respondents ranged for zero to four, with a mean score of 1.71. the other score item, altmetrics familiarity, was computed by adding up the number of altmetrics items from question 2 with which respondents were familiar. again, the theoretical range was from zero to five, and for this item the actual range was also zero to five. the mean on this score was .74. additional dependent variables included a number of open-ended questions that asked about faculty, student, and librarian interest in learning about these measures and services, outreach efforts, and current initiatives.   the study’s independent variables included type of institution: 2-year (17.2%, 10) versus 4+-year (82.8%, 48); number of years as a librarian, from five or fewer years (34.5%, 20) to six or more years (65.5%, 38); and primary job responsibilities divided into reference and user services (60.3%, 35) versus non-reference positions (39.7%, 23). the job responsibility categorization was created from respondents’ answers to a closed-ended question and an “other, please specify” section which allowed people to elaborate on their job duties.   data analysis   the first research question was: “are academic librarians more familiar with established measures of research impact (bibliometrics) than with the emerging field of altmetrics?” when asked how familiar they were “with the concept of altmetrics to assess research impact” fewer than 10 percent (8.6%, 5) of respondents rated themselves as very familiar, and one-quarter indicated that they were “not familiar at all” (25.9%, 15). the majority were in the middle at “slightly familiar” (34.5%, 20) or “not very familiar” (31.0%, 18).   respondents were then given a list of bibliometric methods and asked to indicate the ones with which they were familiar (this was a “check all that apply” question. see question 3 in the appendix). most respondents knew some of the listed methods, with only 20.7% (f=12) indicating that they did not know any of the ones listed. the most common familiarity was with citations counts (74.1%, 43) and jif (65.5%, 38). around one-quarter (24.1%, 14) were also familiar with the h-index, but almost no respondents knew about the i10-index (5.2%, 3) or the g-index (1.7%, 1).   respondents were less likely to have knowledge of altmetrics tools. when given a list of tools (question 2), two-thirds (63.8%, 37) were not familiar with any of the ones listed. around one-quarter had heard of altmetric.com (25.9%, 15) and mendeley (24.1%, 14). less well-known were impactstory (13.8%, 8) and plumx (8.6%, 5). with the “other, please specify” option, one respondent wrote in plos. see table 1.     table 1 respondents’ familiarity with bibliometrics methods and altmetrics tools   % and f n=58 familiarity with bibliometrics methods      citation counts 74.1% (f=43)    journal impact factor 65.5% (f=38)    h-index 24.1% (f=14)    i10 index   5.2% (f=3)    g-index   1.7% (f=1)    none 20.7% (f=12)     familiarity with altmetrics tools      altmetric.com 25.9% (f=15)    mendeley 24.1% (f=14)    impactstory 13.8% (f=8)    plumx   8.6% (f=5)    other (plos)   1.7% (f=1)    none 63.8% (f=37)     table 2 comparison of means across types of independent variables   6+ years as librarian mean, n 5 or fewer years as librarian mean, n bibliometrics familiarity 2.03 (n=38) 1.10 (n=20) altmetrics familiarity .89 (n=38) .45 (n=20)         4+ year school 2-year school* bibliometrics familiarity 1.81 (n=48) 1.20 (n=10) altmetrics familiarity .83 (n=48) .30 (n=10)         reference & user services non-reference librarians bibliometrics familiarity 1.69 (n=35) 1.74 (n=23) altmetrics familiarity .77 (n=35) .70 (n=23) *n for this category is small. view these numbers with caution.     in order to compare knowledge across different independent variables, the data were condensed into two scales, “bibliometrics familiarity” and “altmetrics familiarity,” as described previously. a mathematical average (mean) score was calculated and indicated that, on average, respondents were familiar with nearly two (1.71) bibliometric methods, and almost one (.74) altmetrics tool. note that the large number of respondents with no knowledge of these items pulls the mean score down: 20.7% (12) claimed no knowledge of bibliometrics and 63.8% (37) had heard of none of these altmetrics. when those respondents were set aside and means calculated for those who were familiar with one or more items, mean scores showed knowledge of slightly more than two (2.15) bibliometrics and slightly more than two (2.15) altmetrics. in the following analysis the mean scores include the “none” answers.   mean scores for bibliometrics familiarity and altmetrics familiarity were compared across independent variables. people with six or more years in the profession were familiar with nearly twice as many bibliometrics (2.03 to 1.10) and altmetrics (.89 to .45) as those with five or fewer years of experience. those at 4+ year colleges and universities were more familiar with bibliometrics (1.81 to 1.20) and altmetrics (.83 to .30) than those at 2-year schools, but since the n for the 2-year schools is quite small, these results should be viewed with caution. there was little difference of knowledge between people who worked in reference and user services compared to those who held other types of positions (1.69 to 1.74, and .77 to .70). see table 2.   research question 2 asked, “what are the attitudes of academic librarians toward altmetrics versus traditional measures (bibliometrics)?” the most widely used traditional metrics are jif and citation counts. the h-index, g-index, and i10-index use calculations based on citation count and could be considered less commonly used traditional metrics. the research question is difficult to answer because few respondents had an opinion about the less commonly used metrics. in fact, when asked about their opinions of particular metrics most chose “i don’t know” as their answer for hirsch’s h-index (75.9%, 44), google’s i10-index (93.1%, 54), and egghe’s g-index (94.8%, 55). while most respondents (67.2%, 39) also said they didn’t know about altmetrics as an effective measure of individual research productivity, enough answered this question for a comparison to jif and citation counts. about two-thirds of respondents (65.5%, 38) held the opinion that citation counts are effective for assessing an individual investigator’s research impact. this was about twice as many as those who thought journal impact factor (39.7%, 23) or altmetrics (31.0%, 18) were effective. see table 3.     table 3 opinions about effectiveness of various measures of research impact (n=58)   effective   ineffective don’t know citation counts 65.5% (38) 13.8% (8) 20.7% (12) jif 39.7% (23) 19.0% (11) 41.4% (24) altmetrics 31.0% (18) 1.7% (1) 67.2% (39) h-index 20.7% (12) 3.4% (2) 75.9% (44) i-10 index 6.9% (4) 0.0% (0) 93.1% (54) g-index 1.7% (1) 3.4% (2) 94.8% (55)     respondents were asked for their open-ended comments about areas for improvement for research impact measures and altmetrics. from the eleven responses, respondents felt that there are problems with traditional citation-based measures that might be able to be addressed with altmetrics. the problems listed included the idea that highly cited authors might publish “garbage that is later withdrawn from publication” but the article received a high citation count. uncited literature has “an important role in the body of research as a whole” but that isn’t recognized through citation counts. traditional resources like web of science are difficult to use and altmetrics are easier and more current ways of measuring impact. problems that respondents recognized with the use of altmetrics were that there is a lack of standards for what these metrics mean. a saved article doesn’t mean it was “used in a meaningful way.” these metrics should not be over-valued “because they can be manipulated,” and they “should be used only with extreme caution.” several respondents also recognized a need for more awareness among faculty and librarians, and that workshops would be ideal.   research question 3 asked, “are academic librarians being called upon by faculty to provide information about new research impact measures, and if so, what has characterized these interactions?” very few librarians reported that faculty or students requested information about altmetrics. in fact, 89.7% (52) had zero such requests. a small number (8.6%, 5) received one to five information requests, and one respondent (1.7%) reported six to ten requests. this survey question was accompanied by an open-ended option. examples given of requests included: citation analysis of professor’s publications, mentions of a faculty member in popular media, usage counts (full text, citations, data, etc.), and help in determining the impact factor of an obscure journal.   the final research question was, “what are academic libraries doing, or what could they be doing regarding altmetrics?” this was addressed with a number of open ended questions about a variety of types of outreach, initiatives, and training.   respondents were asked what types of outreach were currently being offered at their institutions on altmetrics and traditional research impact measures. four librarians reported covering altmetrics in campus workshops, sessions, and discussions at faculty orientation. two had written libguides on altmetrics. one library was in the process of marketing their altmetrics information and another expected to start using altmetrics in a new digital commons. three didn’t know of any efforts on their campuses. concerning traditional metrics outreach, five librarians mentioned offering citation counts, and three jif. there were three comments about individual consultations and five on workshops that their libraries offer. one stated libguides and two said they didn’t know. when asked what types of outreach they believed could help faculty and students learn about research impact measures and altmetrics, eleven respondents mentioned various sorts of trainings and opportunities including classes, workshops, webinars and online videos, and conferences or seminars. two thought the best way would be to start demonstrating what is available and what works. two others said that faculty and students on their campuses were not interested in learning new things. respondents were mostly not aware of initiatives underway at their institutions to capture altmetrics data.   the librarians in the study were very interested in learning more about altmetrics. in fact, 84.2% (48) said that they would attend if a free workshop was offered at their institution. when asked what tools could help them learn more about research impact measures and altmetrics, ten mentioned some type of training, although several were careful to point out that it should come from knowledgeable sources such as acrl, digital commons, or vendors. specific training types mentioned included sessions, workshops, webinars and online videos, guides, descriptions, and outlines. one respondent stated they could learn on their own using google, and two made general comments that they were interested in learning more.   discussion   this study has a few measures that can be compared to findings from other studies, and it has a number of unique findings to report. first and most importantly, the data revealed that there appears to be a dearth of knowledge among academic librarians in oklahoma about altmetrics tools, and most of the librarians who responded to the survey were not familiar with newer forms of bibliometrics (h-index, etc.). among respondents, librarians who had been in the profession over five years were more familiar with both altmetrics and bibliometrics. citation counts and journal impact factor have been used for many years, and it is unsurprising that librarians are likely to be familiar with their advantages and limitations. however, other measures such as the more recent bibliometric calculations (i10 index, g-index, and h-index) and altmetrics are newer and generally less well-known.   konkiel et al. (2015) found that at universities with the highest carnegie classification level, 30% of most academic librarians and 50% of “scholarly communication support” librarians reported having “very expert familiarity” with altmetrics, while in the current study, only 8.6% claimed that they were “very familiar.” it is not surprising that librarians at research-intensive universities would have more familiarity than librarians from a mixture of college and university levels. the current study seems to bear out findings in the literature that for many library professionals, more education regarding bibliometrics and altmetrics is needed. while konkiel et al. found that a very specific job category of librarians had more familiarity with altmetrics, the current study compared reference and user services librarians with non-reference librarians and found no meaningful difference in knowledge between job types.   despite calls in the literature for librarians to be involved with providing altmetrics for the scholars on their campuses, most respondents reported that they had received no such requests from faculty. a small number of the librarians in the study had taught about altmetrics in various workshops and discussions, and some had written libguides or used other marketing. a handful of librarians offered standard bibliometrics services individually or in workshops. while the current study confirms the literature’s reflection of librarians’ interest in bibliometrics and altmetrics, respondents’ experiences in this study reveal little evidence of research support requests from other members of their institutions. this is in keeping with preliminary findings from konkiel et al. (2015).   most respondents wanted to learn more about altmetrics and expressed interest in various types of training. two international studies have also reported that academic librarians in a variety of countries are interested in altmetrics and bibliometrics training by their institutions, through vendors, or for lis schools to add this to their curricula (gonzález-fernández-villavicencio, et al., 2015; kennan, et al., 2014).  while roemer and borchardt (2015b) advocate that early career librarians should be providers of altmetrics in their jobs, this study found that early career librarians were less likely to be familiar with research metrics than their more experienced counterparts. this finding did not support kuhn’s (1962) observation that new discoveries are championed by newer professionals rather than those already established in their careers. additionally, while altmetrics may have the potential for generating a paradigm shift in the way research impact is measured in the lis profession, this small study suggests that it remains to be seen whether these newer metrics will revolutionize lis practice in this area in any wide-spread manner.   conclusions   technological advances are pushing the dissemination of research into new venues and traditional bibliometrics are not capturing the impact of these new practices. altmetrics offer new ways to measure research impact. the literature advocates for librarians to take up these cutting-edge technologies. however, there is little hard data in the literature showing if and how librarians are using altmetrics. future studies that produce concrete evidence would be valuable to the profession. one potential direction for future research might include in-depth interviews with librarians who are using altmetrics in their jobs to discover how they are using them and for what purposes. information-seeking behaviour studies of both scholars and librarians who are searching for information on research impact could be valuable. it might also be useful to explore the opinions of tenured faculty who make decisions on tenure review boards as to whether or not they positively view alternative metrics. there is much work to be done before the potential for and use of altmetrics in academia is well-understood.   references   acrl research planning and review committee. (2014). top trends in academic libraries: a review of the trends and issues affecting academic libraries in higher education. college & research libraries news, june 2014, 294–302. retrieved from http://crln.acrl.org/     adams, t. m., & bullard, k. a. (2014). a case study of librarian outreach to scientists: collaborative research and scholarly communication in conservation biology. college & undergraduate libraries, 21(3-4), 377–395. http://dx.doi.org/10.1080/10691316.2014.925415   adie, e., & roe, w. (2013). altmetric: enriching scholarly content with article-level discussion and metrics. learned publishing, 26(1), 11–17. http://dx.doi.org/10.1087/20130103   åstrom, f., & hansson, j. (2012). how implementation of bibliometric practice affects the role of academic libraries. journal of librarianship and information science, 45(4), 316–322. http://dx.doi.org/10.1177/0961000612456867   bar-ilan, j., sugimoto, c., gunn, w., haustein, s., konkiel, s., larivière, v., & lin, j. (2013, nov.). altmetrics: present and future – panel. proceedings of the association for information science and technology. 50(1). http://dx.doi.org/10.1002/meet.14505001013   bazeley, j. w., waller, j., & resnis, e. (2014). engaging faculty in scholarly communication change : a learning community approach. journal of librarianship and scholarly communication, 2(3), ep1129. http://dx.doi.org/10.7710/2162-3309.1129   belter, c. w. (2015). bibliometric indicators: opportunities and limits. journal of the medical library association, 103(4), 219–221. http://dx.doi.org/10.3163/1536-5050.103.4.014   bladek, m. (2014). bibliometrics services and the academic library: meeting the emerging needs of the campus community. college & undergraduate libraries, 21(3-4), 330–344. http://dx.doi.org/10.1080/10691316.2014.929066   bollen, j., van de sompel, h., smith, j. a., & luce, r. (2005). toward alternative metrics of journal impact: a comparison of download and citation data. information processing & management, 41(6), 1419–1440. http://dx.doi.org/10.1016/j.ipm.2005.03.024   bonn, m. (2014). tooling up: scholarly communication education and training. college & research library news, march 2014, 132–135. retrieved from http://crln.acrl.org/   bornmann, l. (2014). do altmetrics point to the broader impact of research? an overview of benefits and disadvantages of altmetrics. journal of informetrics, 8(4), 895–903. http://dx.doi.org/10.1016/j.joi.2014.09.005   bornmann, l. (2015). usefulness of altmetrics for measuring the broader impact of research: a case study using data from plos and f1000prime. aslib journal of information management, 67(3), 305–319. http://dx.doi.org/10.1108/ajim-09-2014-0115   brigham, t. j. (2014). an introduction to altmetrics. medical reference services quarterly, 33(4), 438–447. http://dx.doi.org/10.1080/02763869.2014.957093   brown, j. d. (2014). citation searching for tenure and promotion: an overview of issues and tools. reference services review, 42(1), 70–89. http://dx.doi.org/10.1108/rsr-05-2013-0023   carpenter, t. a. (2014). comparing digital apples to digital apples: background on niso’s effort to build an infrastructure for new forms of scholarly assessment. information services and use, 34(1-2), 103–106. http://dx.doi.org/10.3233/isu-140739   cooper, i. d. (2015). bibliometrics basics. journal of the medical library association, 103(4), 217–218. http://dx.doi.org/10.3163/1536-5050.103.4.013   dinsmore, a., allen, l., & dolby, k. (2014). alternative perspectives on impact: the potential of alms and altmetrics to inform funders about research impact. plos biology, 12(11), e1002003. http://dx.doi.org/10.1371/journal.pbio.1002003 galligan, f., & dyas-correia, s. (2013). altmetrics: rethinking the way we measure. serials review, 39(1), 56–61. http://dx.doi.org/10.1016/j.serrev.2013.01.003   gonzález-fernández-villavicencio, n., dominguez-aroca, m., calderón-rehecho, a., & garcia-hernández, p. (2015). what role do librarians play in altmetrics? retrieved from http://eprints.rclis.org/25481/1/altmetrics.pdf      gumpenberger, c., wieland, m., & gorraiz, j. (2012). bibliometric practices and activities at the university of vienna. library management, 33(3), 174–183. http://dx.doi.org/10.1108/01435121211217199   gunn, w. (2014). mendeley: enabling and understanding scientific collaboration. information services and use, 34(1-2), 99–102. http://dx.doi.org/10.3233/isu-140738   gutierrez, f. r. s., beall, j., & forero, d. a. (2015). spurious alternative impact factors: the scale of the problem from an academic perspective. bioessays, 37(5), 474–476. http://dx.doi.org/10.1002/bies.201500011   herther, n. k. (2013). niso project brings scientific evaluation into the 21st century with altmetrics. information today, retrieved from http://newsbreaks.infotoday.com/newsbreaks/niso-project-brings-scientific-evaluation-into-the-21st-century-with-altmetrics-90409.asp   hirsch, j. e. (2005). an index to quantify an individual's scientific research output. proceedings of the national academy of sciences of the united states of america, 102(46), 16569-16572.   howard, j. (2013). new metrics providers help keep libraries in the research-tracking game. chronicle of higher education 59(38) a6-a7.   information today newsbytes. (2014). information today, 31(7), 3.   kennan, m. a., corrall, s., & afzal, w. (2014). “making space” in practice and education: research support services in academic libraries. library management, 35(8/9), 666–683. http://dx.doi.org/10.1108/lm-03-2014-0037   konkiel, s., sutton, s., & levine-clark, m. (2015, november). use of altmetrics in u.s.-based academic libraries. presentation at the altmetrics conference (2:am). amsterdam, the netherlands. retrieved from http://altmetricsconference.com/?p=68   kuhn, thomas s. (1962). the structure of scientific revolutions. chicago, il: university of chicago press.   lapinski, s., piwowar, h., & priem, j. (2013). riding the crest of the altmetrics wave: how librarians can help prepare faculty for the next generation of research impact metrics. college & research libraries news, 74(6), 292–300. retrieved from http://crln.acrl.org/content/74/6/292.full   maccoll, j. (2010a). library roles in university research assessment. liber quarterly, 20(2), 152–168. http://dx.doi.org/10.18352/lq.7984   maccoll, j. (2010b). research assessment and the role of the library. report produced by oclc research. retrieved from http://www.oclc.org/research/publications/library/2010/2010-01.pdf   national information standards organization. (2014). niso altmetrics standards project white paper. retrieved july 20, 2016, from http://www.niso.org/apps/group_public/document.php?document_id=13295   neylon, c., & wu, s. (2009). article-level metrics and the evolution of scientific impact. plos biology, 7(11), e1000242. http://dx.doi.org/10.1371/journal.pbio.1000242   piwowar, h., & priem, j. (2013). the power of altmetrics on a cv. bulletin of the american society for information science and technology, 39(4), 10–13. http://dx.doi.org/10.1002/bult.2013.1720390405   roemer, r. c., & borchardt, r. (2013). institutional altmetrics and academic libraries. information standards quarterly, 25(2), 14–19. http://dx.doi.org/10.3789/isqv25no2.2013.03   roemer, r. c., & borchardt, r. (2015a). altmetrics and the role of librarians. library technology reports, 51(5), 31–37. http://dx.doi.org/10.5860/ltr.51n5   roemer, r. c., & borchardt, r. (2015b). new grads, meet new metrics: why early career librarians should care about altmetrics & research impact. in the library with the lead pipe (blog). retrieved april 18, 2016 from http://www.inthelibrarywiththeleadpipe.org/2015/new-grads-meet-new-metrics-why-early-career-librarians-should-care-about-altmetrics-research-impact/   roemer, r. c., & borchardt, r. (undated). keeping up with … altmetrics. retrieved from http://www.ala.org/acrl/publications/keeping_up_with/altmetrics   zhao, d. (2011). bibliometrics and lis education: how do they fit together? proceedings of the american society for information science and technology, 48, 1–4. http://dx.doi.org/10.1002/meet.2011.14504801190     appendix: survey questions   q1: how familiar are you with the concept of altmetrics to assess research impact? 1 = very familiar, 2 = slightly familiar, 3 = not very familiar, 4 = not familiar at all   q2: with which of the following altmetrics-related services are you familiar? please check all that apply.                 q2a: altmetric                 q2b: impactstory                 q2c: mendeley                 q2d: plumx                 q2e: i have never heard of any of these services                 q2f: other, please specify   q3: with which of the following methods of assessing research impact are you familiar? please check all that apply.                 q3a: journal impact factor                 q3b: citation counts                 q3c: h-index                 q3d: g-index                 q3e: i-10 index                 q3f: none                 q3g: other, please specify   q4: in your opinion, how effective is the journal impact factor in assessing an individual investigator’s research impact? 1 = very ineffective 2 = ineffective 3 = somewhat ineffective 4 = i don’t know 5 = somewhat effective 6 = effective 7 = very effective   q5: in your opinion, how effective are citation counts in assessing an individual investigator’s research impact? 1 = very ineffective 2 = ineffective 3 = somewhat ineffective 4 = i don’t know 5 = somewhat effective 6 = effective 7 = very effective   q6: in your opinion, how effective is the h-index in assessing an individual investigator’s research impact? 1 = very ineffective 2 = ineffective 3 = somewhat ineffective 4 = i don’t know 5 = somewhat effective 6 = effective 7 = very effective   q7: in your opinion, how effective is the g-index in assessing an individual investigator’s research impact? 1 = very ineffective 2 = ineffective 3 = somewhat ineffective 4 = i don’t know 5 = somewhat effective 6 = effective 7 = very effective   q8: in your opinion, how effective is the i-10 index in assessing an individual investigator’s research impact? 1 = very ineffective 2 = ineffective 3 = somewhat ineffective 4 = i don’t know 5 = somewhat effective 6 = effective 7 = very effective   q9: during the past year, how many information requests regarding altmetrics have you received from a faculty member or student at your institution? 1= 0, 2 = 1-5, 3 = 6-10, 4 = more than 10   q10: if you have received an information request from a faculty member or student at your institution regarding altmetrics in the past year, please describe some of these interactions and the specific information requested. if not applicable, please select n/a.   q11: in your opinion, how effective are altmetrics in assessing an individual investigator’s research impact? 1 = very ineffective 2 = ineffective 3 = somewhat ineffective 4 = i don’t know 5 = somewhat effective 6 = effective 7 = very effective   q12: what, if any, types of outreach does your library offer regarding altmetrics?   q13: what, if any, types of outreach does your library offer regarding traditional research impact measures? examples could include the journal impact factor, citation counts, the h-index, the g-index, or the i-10 index.   q14: if a free workshop was offered at your institution regarding research impact assessment and altmetrics, would you attend? 1= yes, 2 = no   q15: please describe any initiatives underway at your institution to collect altmetrics data.   q16: in your opinion, in what areas do research impact measures and altmetrics need improvement?   q17: what tools could help you learn more about research impact measures and altmetrics?   q18: what types of outreach do you believe could help faculty and students learn more about research impact measures and altmetrics?   q19: how many years have you served as a librarian? 1 = <1 year, 2 = 1-5 years, 3 = 6-10 years, 4 = more than 10 years   q20: in what department do you primarily operate as a librarian? 1= reference, 2 = ill, 3 = serials, 4 = systems, 5 = special collections, 6 = other, please specify         review article   library instruction for graduate nursing students: a scoping review   adelia grabowsky health sciences librarian ralph brown draughon library auburn university auburn, alabama, united states of america email: abg0011@auburn.edu   katherine spybey former adjunct professor nursing department calhoun community college decatur, alabama, united states of america email: katiespybey@gmail.com    received: 7 apr. 2022                                                                     accepted: 17 oct. 2022      2022 grabowsky and spybey. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30145      abstract   objective – the number of graduate nursing programs in the u.s. has increased significantly in recent years. this scoping review seeks to examine the range of literature discussing librarian instruction for graduate nursing students to identity the types of studies being published, the characteristics of instructional sessions, knowledge gaps which may exist, and the evidence available for a subsequent systematic review evaluating instructional effectiveness.   methods – guidelines established by the prisma statement for scoping reviews (prisma-scr) were used to conduct this review. concepts for library instruction and graduate nursing students were searched in six databases as well as google scholar. the two authors used titles/abstracts and when necessary, full-text to independently screen identified studies. conflicting screening decisions were resolved by discussion.   results – data was extracted from 20 sources. thirteen of the sources were descriptions of classes or programs, one was a program evaluation, two were mixed methods studies that looked at library use and program support respectively but did not assess instruction, two were surveys of students’ feelings and attitudes about instruction, and two were quasi-experimental studies which included pre-post instruction quizzes. the most popular format for library instruction was online (synchronous or asynchronous) instruction. most sources did not include information about the timing or duration of instruction. in addition, most sources did not reference instructional theory although a few mentioned aspects of instructional theory such as active learning. only one source mentioned using a specific model to develop instructional content. while several sources mentioned assessment of student learning, only four studies included the results of assessment.   conclusions – sources reporting on instruction for graduate nursing students consisted primarily of descriptions of programs or instructional sessions. many of the descriptive studies lacked essential information such as specifics of format, timing, and duration which would aid replication at other institutions. only four sources were research studies that evaluated instructional effectiveness.     introduction   the number of graduate nursing programs in the u.s. as well as enrollment in those programs has been increasing steadily (jonas philanthropies, 2015). although librarians and nursing faculty might imagine that students enter graduate school with information literacy (il) skills already fully developed, researchers have found that many students, including those in graduate nursing programs, struggle with finding, evaluating, and using information effectively (robertson & felicilda-reynaldo, 2015). therefore, graduate nursing students may benefit from librarian-led instruction intended to improve information literacy skills.    while librarians might consider using the same information and instructional techniques employed in undergraduate nursing classes, graduate students tend to differ from undergraduates in meaningful ways. graduate nursing students are likely to be older, may have been out of school for many years, and may have additional family or work responsibilities (salani et al., 2016). in addition, graduate nursing students are expected to develop more advanced information literacy skills than undergraduates to facilitate translating evidence into practice, identifying gaps in practice, and disseminating their scholarship (american association of colleges of nursing [aacn], 2021). finally, as adult learners, graduate nursing students may have a greater need for library instruction that allows them to be self-directed, to have their prior experience taken into account, and to understand why they are learning and how the new knowledge will be helpful in real-world situations (knowles et al., 1998; ross-gordon et al., 2017).   aims   this scoping review seeks to identity and summarize the published literature related to library instruction provided to graduate nursing students. the following research questions guided the study:   what types of studies are being published? what characteristics of instructional sessions are included in published literature?   methods   guidelines established by the prisma statement for scoping reviews (prisma-scr) were used to conduct this review (tricco et al., 2018). no protocol was prepared for the review. one author (ag), a health sciences librarian with prior experience creating searches for systematic and scoping reviews, developed and executed all searches. six databases were searched on july 30, 2019 with concepts for library instruction and graduate nursing students along with related synonyms and subject headings (see appendix a for complete searches). cinahl; medline; eric; library literature & information science index (h.w. wilson); and library, information science & technology abstracts were searched concurrently though the ebsco interface while library & information science abstracts (lisa) was searched through the proquest interface. the searches were rerun on december 7, 2021 to update content before publication submission. hand searching consisted of examining the reference lists of reviews included in the search results and screening the first 100 results of a search run in google scholar. all results were exported to an endnote library (version x9). after deduping, sources were exported to excel spreadsheets for screening.   inclusion/exclusion criteria   types of participants   the population of interest was graduate nursing students. studies that included only undergraduate students or professional nurses were excluded; however, studies that involved more than one level of student (e.g., undergraduates and graduate students) or more than one type of student (e.g., nursing and pharmacy students) were included as long as specific information about graduate nursing students could be extracted.   concept   sources had to include some type of librarian-led instruction. that instruction could be provided wholly by the librarian(s) or in partnership with other institutional faculty or staff. there were no restrictions on format of instruction; sessions could be provided in-person or virtually, and either synchronously or asynchronously.   context   due to the change from print-focused to electronic resources beginning in the late 1990s and subsequent changes to library instruction, sources had to have been published in or after 1994.   types of sources of evidence   no restrictions were placed on type of source. book reviews, article reviews, editorials, and evidence syntheses were excluded. all other source types including articles, book chapters, dissertations, and theses were included. due to language restrictions of the reviewers and lack of funding for translation services, all sources had to be written in english.   screening   the number of sources screened at each stage is shown in figure 1.  numbers in parentheses are the total of the initial search and the bridge search. separate figures for each search are provided in square brackets. at each level (title/abstract and full-text) the two authors independently screened sources, then met to compare decisions. conflicting screening decisions were resolved by discussion. after the full-text screening, 20 sources were retained for synthesis.   data extraction   a data extraction form was created using excel. variables on the form included population; location; extent of instruction (class or program); standards/guidelines/theories used to develop instruction; format, timing, and duration of instruction; content taught; additional support offered; methodology; assessment; and additional notes (see appendix b). one author (ag) extracted data from each source and the second author (ks) checked the extracted data for accuracy and completeness.   results   overview of sources   the 20 sources included in this review were primarily journal articles (n=19; bernstein et al., 2020; dorner et al., 2001; francis & fisher, 1995; guillot & stahr, 2004; guillot et al., 2010; hinegardner & lansing, 1994; hodson-carlton & dorner, 1999; honey et al., 2006; layton & hahn, 1995; leasure et al., 2009; lemley, 2016; milstead & nelson, 1998; schilperoort, 2020; thompson, 2009; welch et al., 2016; whitehair, 2010; whiting & orr, 2013; wills et al., 2001; wimmer et al., 2014). the one exception was a book chapter (deberg, 2014). publication dates ranged from 1994 to 2020 with zero to two publications each year. most instruction took place in the united states (n=18; bernstein et al., 2020; deberg, 2014; dorner et al., 2001; francis & fisher, 1995; guillot & stahr, 2004; guillot et al., 2010; hinegardner & lansing, 1994; hodson-carlton & dorner, 1999; layton & hahn, 1995; leasure et al., 2009; lemley, 2016; milstead & nelson, 1998; schilperoort, 2020; welch et al., 2016; whitehair, 2010; whiting & orr, 2013; wills et al., 2001; wimmer et al., 2014), although there was one source from canada (thompson, 2009) and one from new zealand (honey et al., 2006).     from: moher d., liberati a., tetzlaff j., altman d. g., the prisma group (2009). preferred reporting items for systematic reviews and metanalyses: the prisma statement. plos med 6(7): e1000097. doi:10.1371/journal.pmed1000097. for more information, visit www.prisma-statement.org.   figure 1 prisma flow diagram.   four sources included instruction for more than one level of student. one of the four included master’s, dnp, and phd students (whitehair, 2010), two included master’s and phd students (francis & fisher, 1995; layton & hahn), and one included master’s and dnp students (lemley, 2016). the remaining sources included only one level of students. master’s was the most common (n=9; dorner et al., 2001; guillot & stahr, 2004; guillot et al., 2010; hinegardner & lansing, 1994; hodson-carlton & dorner, 1999; honey et al., 2006; schilperoort, 2020; thompson, 2009; wills et al., 2001) followed by phd (n=3; milstead & nelson, 1998; welch et al., 2016; wimmer et al., 2014) and dnp (n=3; bernstein et al., 2020; deberg, 2014; whiting & orr, 2013). the remaining source referred only to graduate nursing students without indicating what level(s) were included (leasure et al., 2009).   characteristics of sources (see appendix b)   format of instruction   the 20 identified sources included descriptions of format for 21 classes and programs. the most popular format for library instruction was virtual (n=7); however, only one source used online synchronous instruction (wimmer et al., 2014). other virtual options included interactive tutorials (n=4; dorner et al., 2001; hodson-carlton & dorner, 1999; schilperoort, 2020; welch et al., 2016), videos (n=1; deberg, 2014), and a static webpage (n=1; milstead & nelson, 1998). five additional sources used hybrid methods with both virtual and face-to-face (f2f) components. two of the five used f2f followed by online tutorials (honey et al., 2006; leasure et al., 2009), one used f2f followed by a recording (deberg, 2014), one used f2f followed by optional individual virtual sessions (guillot & stahr, 2004), and one used both f2f and synchronous instruction followed by optional individual sessions (whitehair, 2010). four sources included only f2f instruction; however, it is important to note that three of those four were from 1994 and 1995, the earliest years included in this review (francis & fisher, 1995; hinegardner & lansing, 1994; layton & hahn, 1995). the fourth f2f source occurred later but involved instruction on spss using library computers (thompson, 2009). three of the remaining five sources reported on librarians who were embedded in a course or courses throughout the semester (guillot et al., 2010; lemley, 2016; wills et al., 2001). the final two did not indicate the format of instruction (bernstein et al., 2020; whiting & orr, 2013).   timing of instruction   three sources involved embedded librarians (guillot et al., 2010; lemley, 2016; wills et al., 2001) and one a static webpage (milstead & nelson, 1998) so instruction could be considered to be available throughout the class. there were 17 classes described in the remaining 16 studies. there was no indication of when instruction took place during the semester for eight of those classes (deberg, 2014; francis & fisher, 1995; guillot & stahr, 2004; hinegardner & lansing, 1994; layton & hahn, 1995; leasure et al., 2009; thompson, 2009; whiting & orr, 2013). the remaining nine reported instruction which took place early in the semester, i.e., before class started or within the first month (bernstein et al., 2020; deberg, 2014; dorner et al., 2001; hodson-carlton & dorner, 1999; honey et al., 2006; schilperoort, 2020; welch et al., 2016; whitehair, 2010; wimmer et al., 2014). in addition, some authors reported that instruction was tied to course assignments or course content (bernstein et al., 2020; deberg, 2014; dorner et al., 2001; guillot & stahr, 2004; hinegardner & lansing, 1994; hodson-carlton & dorner, 1999; thompson, 2009), that library assignments were required/graded (francis & fisher, 1995; schilperoort, 2020), and that assistance (guillot et al., 2010) or tutorials (dorner et al., 2001) were provided at point of need. finally, seven authors reported inclusion in the course learning management system which provided access to syllabi, assignments, discussion boards, and class email lists (dorner et al., 2001; guillot et al., 2010; lemley, 2016; whitehair, 2010; whiting & orr, 2013; wills et al., 2001; wimmer et al., 2014). duration of instruction    very few sources reported how long instruction lasted. most that did mention duration were discussing either f2f sessions or the f2f session of hybrid instruction. durations reported included two mentions each of one-hour sessions (guillot & stahr, 2004; whitehair, 2010), two-hour sessions (francis & fisher, 1995; layton & hahn, 1995), and three-hour sessions (hinegardner & lansing, 1994; thompson, 2009). only schilperoort (2020) mentioned the length of instructional tutorials, reporting an average time of 15 to 30 minutes to complete the self-paced tutorial.   content of instruction   fourteen of the 20 sources included introducing students to databases, in many cases mentioning specific health science databases such as cinahl and medline (bernstein et al., 2020; deberg, 2014; dorner et al., 2001; francis & fisher, 1995; guillot & stahr, 2004; hinegardner & lansing, 1994; honey et al., 2006; layton & hahn, 1995; leasure et al., 2009; lemley, 2016; schilperoort, 2020; welch et al., 2016; whitehair, 2010; wills et al., 2001). nine of those 14 sources also included specific instructional content related to searching skills such as choosing keywords, finding subject headings, and using boolean operators or filters (bernstein et al., 2020; dorner et al., 2001; francis & fisher, 1995; hinegardner & lansing, 1994; layton & hahn, 1995; leasure et al., 2009; lemley, 2016; schilperoort, 2020; whitehair, 2010). although all instruction might be assumed to discuss library services, 11 sources explicitly mention introducing library services in general or specific services such as how to access full-text, use interlibrary loan or contact a librarian for help (guillot & stahr, 2004; guillot et al., 2010; honey et al., 2006; layton & hahn, 1995; leasure et al., 2009; lemley, 2016; milstead & nelson, 1998; thompson, 2009; whitehair, 2010; whiting & orr, 2013; wimmer et al., 2014). five instructors included content about citing sources (dorner et al., 2001; guillot et al., 2010; lemley, 2016; welch et al., 2016; whiting & orr, 2013), and four included instruction on evaluating research sources (bernstein et al., 2020; dorner et al., 2001; hodson-carlton & dorner, 1999; leasure et al., 2009).   additional content mentioned more than once included: bibliographic management software (n=3; hinegardner & lansing, 1994; leasure et al., 2009; welch et al., 2016), developing research questions (n=3; deberg, 2014; guillot et al., 2010; welch et al., 2016), evaluating evidence/levels of evidence (n=3; deberg, 2014; lemley, 2016; schilperoort, 2020), and resources to find research instruments (n=2; dorner et al., 2001; francis & fisher, 1995). finally, there was content mentioned by only one author including current awareness services (whitehair, 2010), data concepts and using spss (thompson, 2009), off-campus access (francis & fisher, 1995), and in a pre-2000 source, how to use email and the internet (layton & hahn, 1995).   additional support     in many cases students were offered additional support beyond the actual instructional session(s). the most common type of support offered was online discussion boards/rooms within learning management systems (n=5; dorner et al., 2001; hodson-carlton & dorner, 1999; lemley, 2016; whiting & orr, 2013; wills et al., 2001). other support included encouraging students to contact a librarian or a library help desk with questions (n=3; lemley, 2016; thompson, 2009; whitehair, 2010), offering individual sessions (n=3; bernstein et al., 2020; deberg, 2014; wills et al., 2001), holding chat sessions for group help (n=2; dorner et al., 2001; hodson-carlton & dorner, 1999), sending follow-up emails (n=2; guillot & stahr, 2004; guillot et al., 2010), providing information about additional training opportunities (n=2; honey et al., 2006; leasure et al., 2009), offering a research guide (n=1; wimmer et al., 2014), and providing a brochure (n=1; honey et al., 2006).   assessment of instruction   most of the sources (n=16) did not assess the effectiveness of library instruction. instead, authors provided descriptions of how instruction was implemented in a specific class or classes (n=6; deberg, 2014; guillot & stahr, 2004; guillot et al., 2010; hinegardner & lansing, 1994; wills et al., 2001; wimmer et al., 2014), how instruction was implemented in a new program of study (n=3; francis & fisher, 1995; honey et al., 2006; lemley, 2016), or how instruction was implemented in both a program and one or more specific classes (n=7; dorner et al., 2001; layton & hahn, 1995; leasure et al., 2009; milstead & nelson, 1998; welch et al., 2016; whitehair, 2010; whiting & orr, 2013). three of those 16 sources were research studies, but the research was intended to assess library use (honey et al., 2006), students’ satisfaction with library services and resources (whiting & orr, 2013), or the practicalities of providing instruction (guillot & stahr, 2004) rather than instructional effectiveness.   several authors did mention assessing the effectiveness of instruction with varied means including pre/posttests and evaluations; however, no results of assessment were provided (deberg, 2014; dorner et al., 2001; francis & fisher, 1995; layton & hahn, 1995; welch et al., 2016). four authors provided anecdotal evidence of instructional success derived from informal feedback from faculty or students (deberg, 2014; dorner et al., 2001), course evaluations (guillot et al., 2010), or colleagues at the reference desk (francis & fisher, 1995).   only four sources were research studies assessing the effectiveness of library instruction. two were quasi-experimental studies utilizing pre and posttests of knowledge with additional open-ended questions about student confidence (hodson-carlton & dorner, 1999; schilperoort, 2020). the other two studies surveyed students about their feelings and attitudes concerning instruction (bernstein et al., 2020; thompson, 2009). results of the research studies assessing instructional effectiveness are shown in table 1. there were mixed results from surveys of student confidence, with three studies reporting increased confidence (bernstein et al., 2020; hodson-carlton & dorner, 1999; schilperoort, 2020) and one study reporting students almost equally divided among more confident and less confident (thompson, 2009). both studies with pre and postquizzes reported that the percentage of correct answers increased on the postquiz (hodson-carlton & dorner, 1999; schilperoort, 2020).   learning theories/standards/guidelines   fourteen of the 20 sources did not mention using any standards, guidelines, or theories to inform development of instructional content (dorner et al., 2001; francis & fisher, 1995; guillot & stahr, 2004; hinegardner & lansing, 1994; hodson-carlton & dorner, 1999; layton & hahn, 1995; leasure et al., 2009; lemley, 2016; milstead & nelson, 1998; thompson, 2009; welch et al., 2016; whiting & orr, 2013; wills et al., 2001; wimmer et al., 2014). in the remaining six sources, two authors mentioned library standards with honey et al. (2006) referencing the australian and new zealand information literacy framework (bundy, 2004) and guillot et al. (2010) referencing both the association of colleges and research libraries (acrl, 2000) information literacy competency standards for higher education and the acrl (2008) standards for distance learning library services. three authors referenced nursing standards with bernstein et al. (2020) and deberg (2014) citing the essentials of doctoral education for advanced nursing practice (american association of colleges of nursing, 2006) and whitehair (2010) citing the practice doctorate nurse practitioner entry-level competencies (national panel for np practice doctorate competencies, 2006).   only two authors mentioned using a specific learning model or theory to develop instructional content. whitehair (2010) used both the student-centered model of kraft and androwich and kuhlthau’s model of the information search process. schilperoort (2020) mentioned using both constructivist learning theory and andragogy (adult learning theory) to develop an interactive tutorial. six additional authors (dorner et al., 2001; francis & fisher, 1995; hinegardner & lansing, 1994; hodson-carlton & dorner, 1999; layton & hahn, 1995; leasure et al., 2009; welch et al., 2016) did mention elements such as active learning, hands-on learning, point-of-need instruction, or accommodating different skill levels which would be consistent with adult learning theory or constructivist approaches (knowles et al., 1998; ross-gordon et al., 2017).   table 1 results of research studies assessing instructional effectiveness author(s), date, population, location methodology specifics results of assessment surveys bernstein et al., 2020, dnp students, united states survey of student feelings and attitudes no indication of how many students completed the survey. results were given as broad statements rather than as numbers or percentages. most students felt they understood the components of nursing literature. most students felt confident in using databases to find relevant literature. students valued the integration of the library and the writing center into the class and felt both should be included in future classes. thompson, 2009, master’s students, canada survey of students’ feelings and attitudes no indication of how many students completed the survey. results were given as broad statements rather than as numbers or percentages most students agreed content was relevant. students were divided about whether the class increased their comfort with undertaking future quantitative projects. students were divided about whether they felt more comfortable reading and interpreting quantitative research. most students felt the assignment was too difficult. quasi-experimental studies hodson-carlton & dorner, 1999, master’s students, indiana quasi-experimental (pre & postquiz plus open-ended questions) 30 students took the prequiz and 24 took the postquiz. (6 students did not complete the course so did not take the post-quiz). 88% (21/24) answered the 6 post-module questions correctly compared to 63% (19/30) pre-module. post-module 79% agreed or strongly agreed they were able to assess the quality of web healthcare information from an advanced nursing conceptual approach. schilperoort, 2020, master’s students, california quasi-experimental (pre-post quiz plus survey of confidence with some open-ended questions) 59 students completed the pre and postquiz. 57 were required to do so as part of a class, the other 2 chose to complete the module voluntarily. 13 students provided additional comments. the percentage of correct answers increased on the post-test for each of 5 questions. the biggest change (+46%) occurred in a question asking students to rank by level of evidence. all students felt much more (49%) or somewhat more (51%) confident in their ability to identify high-level research. all students felt much more (59%) or somewhat more (41%) confident in their ability to use library resources to find various types of evidence. additional comments were positive.   challenges and benefits   some challenges seemed to be almost universal while others were related to specific types of instruction. the need for collaboration between nursing faculty and librarians was mentioned by almost all authors (bernstein et al., 2020; deberg, 2014; dorner et al., 2001; francis & fisher, 1995; guillot & stahr, 2004; guillot et al., 2010; hodson-carlton & dorner, 1999; honey et al., 2006; layton & hahn, 1995; leasure et al., 2009; lemley, 2016; schilperoort, 2020; welch et al., 2016; whitehair, 2010; wimmer et al., 2014). in contrast, the time-consuming aspects of instruction were mentioned primarily when discussing embedded librarianship (guillot et al., 2010; lemley, 2016) or when offering individual consultations (bernstein et al., 2020; deberg, 2014; guillot & stahr, 2004). dorner et al. (2001) also mentioned time as a challenge when discussing the need to update videos frequently because of database interface changes, a problem echoed in schilperoort’s (2020) recommendation to review and update tutorials at the beginning of each semester or use. one benefit mentioned for tutorials is that even when created for a specific class, they can also be offered as standalone sources of instruction (hodson-carlton & dorner, 1999; schilperoort, 2020). other challenges reported for embedded librarianship include unrealistic expectations of students (guillot et al., 2010) and role confusion, i.e., students asking questions of the librarian which should be directed to nursing faculty (guillot et al., 2010; lemley, 2016). benefits of embedded librarianship included extended rapport with students (guillot et al., 2010), the ability to be proactive (lemley, 2016), and the ability to broadcast messages to an entire class (guillot et al., 2010; lemley, 2016).   other instructional challenges mentioned include difficulties in providing equal access to off-campus students (dorner et al., 2001; francis & fisher, 1995; milstead & nelson, 1998), technological costs associated with virtual instruction (guillot & stahr, 2004), and nursing faculty turnover (dorner et al., 2001; lemley, 2016).   discussion   this scoping review sought to identify and summarize literature on librarian-led instruction for graduate nursing students. like previous research (salani et al., 2016), many of the reviewed sources suggest that the needs of graduate nursing students differ from those of undergraduates in multiple ways. graduate nursing students tend to be older (guillot & stahr, 2004; honey et al., 2006; whiting & orr, 2013) and to be working while attending school (dorner et al., 2001; francis & fisher, 1995; guillot & stahr, 2004; honey et al., 2006; thompson, 2009; whitehair, 2010; whiting & orr, 2013). in addition, many graduate students have been out of school for several years (guillot & stahr, 2004; guillot et al., 2010; lemley, 2016; whitehair, 2010; whiting & orr, 2013) and may have increased family responsibilities (guillot & stahr, 2004; whitehair, 2010).   sources reporting on library instruction for graduate nursing students consisted primarily of case reports, i.e., descriptions of instructional sessions, tutorials, or programs rather than research studies evaluating instructional effectiveness. descriptions, particularly of new programs or classes, can be helpful for librarians looking for different ways to approach instruction, however, these descriptions often lacked details which would aid in replicating library sessions or tutorials at other institutions. although all sources provided some information about instructional content and most sources indicated the format of instruction, in many cases, other information such as timing and duration which would assist in replicating the session was missing.   although several authors mentioned assessing instructional effectiveness, few reported assessment results which could also aid in replication decisions. in addition, the studies that did assess results varied in significant ways. two looked only at student’s feelings and attitudes (berstein et al., 2020; thompson, 2009) which provides an incomplete measure of effectiveness. the remaining two studies assessed both changes in knowledge and attitude (hodson-carlton & dorner, 1999; schilperoort, 2020) which offers a more complete assessment of learning. although published 21 years apart, both of the studies reported on the creation of a web-based, point-of-need tutorial. the older tutorial was intended to teach students to evaluate the quality of websites, while the newer taught students to find evidence based information and evaluate levels of evidence. both studies reported an increase in student knowledge after instruction.   finally, although authors may have developed instruction and assessment based on learning theories, standards, or guidelines, with a few exceptions, there was little indication of which standards and/or theories were used and how those standards/theories influenced instructional development.   implications   findings illustrate the need for librarians to provide more detail in published class descriptions so that sessions can be replicated by others. also helpful would be more explicit information about instructional theories, standards, or guidelines used to develop class content. more importantly, librarians should consider adopting or creating assessment strategies to determine the effectiveness of instruction for graduate nursing students, and then publish the results of those assessments for the benefit of others. only a robust assortment of published assessment studies will enable a clearer understanding of the effectiveness of library instruction for graduate nursing students.   limitations   searching always involves compromise between comprehensiveness (finding all relevant sources) and precision (finding a minimum of irrelevant sources). this study sought to err on the side of comprehensiveness in two ways: (a) by searching both subject headings and keywords in the title, abstract, and subject heading fields and (b) by using compound searching (x and y) rather than quoted phrase searching (“x y”).  however, there are still limitations to the search. for example, there may be other words or phrases used in the literature to refer to graduate nursing students or library instruction that were not included in this search strategy. in addition, search results were limited to results in english, which would have limited the inclusion of studies completed outside the united states.   conclusion   this scoping review examining published literature of librarian-led instruction for graduate nursing students found that most of the sources were descriptions of classes or programs which did not report any results from measures of instructional effectiveness. an additional three sources evaluated programs or library use but did not assess instruction. all sources reported some characteristics of instructional sessions, but few provided enough information to allow others to accurately replicate instruction at other institutions. only four sources provided measures of instructional effectiveness. two included surveys of students’ feelings and attitudes about instruction, and two were quasi-experimental studies which included pre-post knowledge quizzes. the lack of evidence related to the effectiveness of librarian-led instruction for the population of graduate nursing students reveals a gap in library research and suggests there is insufficient evidence to warrant a systematic review evaluating this topic.   author contributions   adelia grabowsky: conceptualization, data curation, formal analysis, investigation, methodology, project administration, writing – original draft, writing – review & editing katherine spybey: conceptualization, data curation, formal analysis, investigation, writing – review & editing   references *references included in scoping review   american association of colleges of nursing. (2006). the essentials of doctoral education for advanced practice nursing.  https://www.pncb.org/sites/default/files/2017-02/essentials_of_dnp_education.pdf    american association of colleges of nursing. (2021). the essentials: core competencies for professional nursing education. https://www.aacnnursing.org/portals/42/academicnursing/pdf/essentials-2021.pdf   association of college and research libraries. (2000). information literacy competency standards for higher education. https://alair.ala.org/handle/11213/7668   association of college and research libraries. (2008). standards for distance learning library services. https://www.ala.org/acrl/standards/guidelinesdistancelearning    *bernstein, m., roney, l., kazer, m. & boquet, e. h. (2020). librarians collaborate successfully with nursing faculty and a writing centre to support nursing students doing professional doctorates. health information and libraries journal, 37, 240-244. https://doi.org/10.1111/hir.12327   bundy, a. (2004). australian and new zealand information literacy framework, https://www.utas.edu.au/__data/assets/pdf_file/0003/79068/anz-info-lit-policy.pdf   *deberg, j. (2014). reflections on involvement in a graduate nursing curriculum. in a. e. blevins (ed.), curriculum-based library instruction (pp. 165-170). rowman & littlefield.   *dorner, j. l., taylor, s. e. & hodson-carlton, k. (2001). faculty-librarian collaboration for nursing information literacy: a tiered approach. reference services review, 29(2), 132-40. https://doi.org/10.1108/00907320110394173   *francis, b. w. & fisher, c. c. (1995). multilevel library instruction for emerging nursing roles. bulletin of the medical library association, 83(4), 492-8.   *guillot, l. & stahr, b. (2004). a tale of two campuses: providing virtual reference to distance nursing students. journal of library administration, 41(1-2), 139-52. https://doi.org/10.1300/j111v41n01_11   *guillot, l., stahr, b. & meeker, b. j. (2010). nursing faculty collaborate with embedded librarians to serve online graduate students in a consortium setting. journal of library & information services in distance learning, 4(1/2), 53-62. https://doi.org/10.1080/15332901003666951   *hinegardner, p. g. & lansing, p. s. (1994). nursing informatics programs at the university of maryland at baltimore. bulletin of the medical library association, 82(4), 441-3.   *hodson-carlton, k. & dorner, j. l. (1999). an electronic approach to evaluating healthcare web resources. nurse educator, 24(5), 21-6. https://doi.org/10.1097/00006223-199909000-00013   *honey, m., north, n., & gunn, c. (2006). improving library services for graduate nurse students in new zealand. health information and libraries journal, 23(2), 102-9. https://doi.org/10.1111/j.1471-1842.2006.00639.x   jonas philanthropies. (2015). new aacn data confirms enrollment surge in schools of nursing. https://tinyurl.com/y3u262gd   knowles, m. s., holton, e. f., & swanson, r. a. (1998). the adult learner (5th ed.). butterworth-heinemann.   *layton, b. & hahn, k. (1995). the librarian as a partner in nursing education. bulletin of the medical library association, 83(4), 499-502.   *leasure, a. r., delise, d., clifton, s. c., & pascucci, m. a. (2009). health information literacy: hardwiring behavior through multilevels of instruction and application. dimensions of critical care nursing, 28(6), 276-82. https://doi.org/10.1097/dcc.0b013e3181b4003c   *lemley, l. (2016). virtual embedded librarianship program: a personal view. journal of the medical library association, 104(3), 232-4. http://dx.doi.org/10.3163/1536-5050.104.3.010   *milstead, j. a., & nelson, r. (1998). preparation for an online asynchronous university doctoral course: lessons learned. computers in nursing, 16(5), 247-58.   national panel for np practice doctorate competencies. (2006). practice doctorate nurse practitioner entry-level competencies. https://www.pncb.org/sites/default/files/2017-02/nonpf_dnp_competencies.pdf   robertson, d. s. & felicilda-reynaldo, r. f. (2015). evaluation of graduate nursing students' information literacy self-efficacy and applied skills. the journal of nursing education, 54(3, suppl), s26-s30. https://doi.org/10.3928/01484834-20150218-03   ross-gordon, j. m., rose, a. d., & kasworm, c. e. (2017). the adult learner. in foundations of adult and continuing education (pp. 215-253). jossey-bass.   salani, d., albuja, l. d., & azaiza, k. (2016). the keys to success in doctoral studies: a preimmersion course. journal of professional nursing, 32(5), 358-363. https://doi.org/10.1016/j.profnurs.2016.01.005   *schilperoort, h. m. (2020). self-paced tutorials to support evidence-based practice and information literacy in online health sciences education. journal of library & information services in distance learning, 14(3-4), 278-290. https://doi.org/10.1080/1533290x.2021.1873890   *thompson, k. (2009). torturing nurses with data: building a successful quantitative research module. iassist quarterly, 33(3), 6-9. https://doi.org/10.29173/iq112   tricco, a. c., lillie, e., zarin, w., o'brien, k. k., colquhoun, h., levac, d., moher, d., peters, m. d., horsley, t., weeks, l., hempel, s., akl, e. a., chang, c., mcgowan, j., stewart, l., hartling, l., aldcroft, a., wilson, m. g., garritty, c., . . . straus, s. e. (2018). prisma extension for scoping reviews (prisma-scr): checklist and explanation. annals of internal medicine, 169(7), 467-73. https://doi.org/10.7326/m18-0850   *welch, s., cook, j. & west, d. (2016). collaborative design of a doctoral nursing program online orientation. nursing education perspectives, 37(6), 343-4. https://doi.org/10.1097/01.nep.0000000000000053   *whitehair, k. j. (2010). reaching part-time distance students in diverse environments. journal of library & information services in distance learning, 4(3), 96-105. https://doi.org/10.1080/1533290x.2010.503166   *whiting, p. & orr, p. (2013). evaluating library support for a new graduate program: finding harmony with a mixed methods approach. the serials librarian, 64(1-4), 88-98. https://doi.org/10.1080/0361526x.2013.760329   *wills, c. e., stommel, m. & simmons, m. (2001). implementing a completely web-based nursing research course: instructional design, process, and evaluation considerations. journal of nursing education, 40(8), 359-362. https://doi.org/10.3928/0148-4834-20011101-07   *wimmer, e., morrow, a., & weber, a. (2014). collaboration in etextbook publishing: a case study. collabrative librarianship, 6(2), 82-86. . appendix a search strategies   initial searches were completed on july 30, 2019. bridge searches were run on december 7, 2021.   cinahl; medline; eric; library literature & information science index (h.w. wilson); and library, information science & technology abstracts   (graduate nursing students or students, nursing, graduate or students, nursing doctoral or students, nursing, masters or education, nursing, graduate or msn or dnp or ((masters or phd or doctoral or graduate student*) and nurs*)) and (library orientation or library user education or library instruction or ((librar* or information literacy) and (instruction or workshop or orientation or session or class)))   search notes:   subject headings and keywords associated with the two concepts of graduate nursing students and library instruction were included in the search (see table a1 for list of included subject headings).  medline, cinahl, eric, and psycinfo were searched concurrently through the ebsco interface. while it is possible to use field codes to restrict search terms to specific fields, a more comprehensive search is possible with the “select a field” option. when using “select a field” all search terms are searched in the author, subject, keyword, title, and abstract fields which reduces the chance of missing relevant results. more information about using the “select a field” option can be found here: https://help.ebsco.com/interfaces/ebsco_guides/general_product_faqs/fields_searched_using_select_a_field_drop_down_list).   all searches were limited to english. the initial search was limited to 1994 through july 2019. the bridge search was limited to july 2019 through december 2021.   table a1 subject headings for each database   concept – graduate nursing students concept – library instruction database subject headings subject headings cinahl students, nursing, graduate students, nursing, masters students, nursing, doctoral library user education medline education, nursing, graduate libraries eric graduate students nursing students doctoral students library instruction library literature & information science index students library orientation library, information science, & technology abstracts students library orientation   library & information science abstracts (lisa) (searched through proquest interface):   (graduate nursing students or students, nursing, graduate or students, nursing doctoral or students, nursing, masters or education, nursing, graduate or msn or dnp or ((masters or phd or doctoral or graduate student*) and nurs*)) and (library orientation or library user education or library instruction or ((librar* or information literacy) and (instruction or workshop or orientation or session or class)))   search notes:   all searches were limited to english. the initial search was limited to 1994 through july 2019. the bridge search was limited to july 2019 through december 2021.   google scholar (first 100 results examined) (graduate nursing students | msn | dnp | ((masters | phd | doctoral | graduate student) and nurse)) ((library or information literacy) and (instruction | workshop | orientation | session| class))   search notes:   the initial search was limited to 1994 through 2019. the bridge search was limited to 2019 through 2021.   appendix b sources included in scoping review   table b1 characteristics of sources *s/g/t are standards, guidelines, or theories used to develop instruction. author(s), date, population, location class or program   s/g/t* format (f2f = face-to-face) timing, duration content taught additional support methodology, assessment, other notes bernstein et al., 2020, dnp students, united states class dnp intro level class   essentials of doctoral education for advanced practice nursing no indication of format. first week of class. no indication of class duration.     reading and evaluating nursing research; database searching focused on advanced features such as filters. follow-up research appointments with librarian. survey. survey of feelings and attitudes. instruction tied to course assignments. deberg, 2014, dnp students, iowa two classes 1. primary care and older adult ii                          2. finding evidence for practice.   essentials of doctoral education for advanced practice nurses.   1. hybrid -f2f lecture, recorded for distance students. 2. virtualonline videos of database demos and lectures.   1. & 2. no mention of timing or duration.     class 1 -structuring clinical questions, evaluating evidence strength, utilizing clinical and literature databases. class 2 databases demoed, no specifics. 1. & 2. individual meetings via phone, email, or web. case report.  1. & 2. assessment mentioned but no results provided. 1. anecdotal evidence of success from nursing faculty and conversations with students. not clear if f2f lecture in class 1 was delivered by librarian or nursing faculty. instruction was tied to course assignments. dorner et al., 2001, master’s students, indiana both -   program was tiered approach in bsn and msn. specific msn class – nurs 605.   no s/g/t mentioned. virtual – online tutorials, each with a pre and postquiz, developed for specific courses and inserted at point of need. module for nurs 605 was assigned during first two weeks of semester and contained multiple tutorials. no mention of duration or number of tutorials.     nurs 605 citations, boolean, keyword, and subject searching, evaluation of sources, web searching, cinahl, apa. additional content included in other classes medline, psyclit, index medicus, science citation index, dissertation abstracts, mental measurements yearbook, tests in print. online discussion boards, online chat sessions for small groups. case report. each tutorial of the module had a pre and postquiz, however no results were provided. informal feedback solicited from students was consistently positive. instruction was tied to course assignments. librarians given 'instructor' access to course blackboard site. francis & fisher, 1995, master’s and phd students, florida program   no s/g/t mentioned. f2f no indication of timing.   two sessions, each two hours long.     cinahl/medline (search strategies including limits, controlled vocab ), catalogue, science citation index, index medicus, dissertation abstracts, hospital literature index, mental measurements yearbook, tests in print, test critiques.   additional content for off-campus users: using databases from off-campus. case report. mentions assessment but no results provided. anecdotal evidence librarians reported that nursing students asked fewer basic questions. instruction was tied to course work. students were required to participate, assignments were graded, or credit was received for participation. guillot & stahr, 2004, master’s students, louisiana class nurs 600 theoretical foundations of advanced nursing   no s/g/t mentioned. hybrid traditional bibliographic instruction followed by optional individual virtual sessions. no indication of timing. session was 1 hour with 20 minutes spent scheduling individual sessions. duration of individual sessions varied. health science databases, library services, virtual reference. individual virtual sessions were tailored to each student with students expected to have chosen relevant search terms before the meeting. follow-up email with a transcript of the virtual session. program evaluation. focus was assessment of practicalities of providing the program, no assessment of instructional effectiveness mentioned. instruction tied to assignment.   guillot et al., 2010, master’s students, louisiana class nurs 500/600 theoretical foundations of advanced nursing   information literacy competency standards for higher education; standards for distance learning library services. embedded available throughout semester.   assistance provided at point of need. content driven by student questions on discussion board. questions for the specific semester included assessing library resources remotely, using interlibrary loan, apa, and help with research questions. broadcast email about how to access assigned articles. case report. anecdotal evidence that students were enthusiastic about the service (derived from course evaluations). librarian embedded into course management system. hinegardner & lansing, 1994, master’s students, maryland class computer applications in nursing and health care   no s/g/t mentioned. f2f no indication of timing. 3-hour session.     computerized literature searching, databases, search strategy development, file management software. none mentioned. case report. no assessment mentioned. focus of article is development of nursing informatics program. instruction tied to class assignment. hodson-carlton & dorner, 1999 master’s students, indiana class nur 605   no s/g/t mentioned. virtual interactive web instructional module. module took place in the 3rd or 4th week of the semester. no indication of duration of module. evaluation of web resources using seven evaluation criteria (scope, audience, authority, currency, accuracy, purpose, and organization). one synchronous chat session; asynchronous online bulletin board which included both a nursing faculty member and a librarian. quasi-experimental. pre-post quiz with six true/false questions about web information. open-ended questions about perceptions also included in postquiz. instruction tied to class assignments. honey et al., 2006, master’s students, new zealand program   australian & new zealand information literacy framework hybrid f2f plus online tutorials and web-based resource pages. f2f orientation beginning of semester. no indication of duration. course related sessions provided within classes. f2f orientation nursing specific resources, library tutorials, workshops, librarian contact info. voluntary f2f sessions catalogue, nursing-specific databases including cinahl, e-journals. informational brochure about library resources for nursing students.   small f2f voluntary sessions. mixed methods (student surveys plus library staff interviews). assessment of library use but no mention of assessment of instructional effectiveness. focus of the study is a survey of use of technology by nursing students and changes made as a result. layton & hahn, 1995, master’s and phd students, maryland both program and two classes; msn class computer technologies in nursing. phd class technology applications in nursing research.   no s/g/t mentioned. both classes f2f msn class 2) two-hour sessions. no indication of timing.   phd class two sessions. no indication of timing or duration. msn class internet, email, databases, search strategies, controlled vocab, medline, cinahl, psycinfo, library services. phd class internet, email, electronic mailing lists, databases, search strategies, controlled vocab, medline, cinahl, psycinfo, library services. none mentioned. case report. assessment mentioned but no results provided. all instructional sessions include lecture, demo, and hands-on training with students performing exercises on the computer. leasure et al., 2009, graduate students (level not specified), oklahoma both program and two graduate nursing classes.   no s/g/t mentioned. hybrid both f2f and online tutorials. early graduate nursing class no indication of timing or duration of f2f instruction.   graduate research course no indication of timing or duration. early course  databases, searching (keywords, controlled vocab, limits, boolean operators), website evaluation. online tutorial – webpage evaluation.   research course – advanced and command line searching, full-text, bibliographic management. additional free training sessions were available to individuals wishing to improve their skills. case report. no assessment mentioned. instructional sessions consisted of lecture plus live demo searches followed by discussion among students, librarian, and nursing faculty member. lemley, 2016, master’s and dnp students, alabama program   no s/g/t mentioned (did reference best practices for embedded librarians). embedded available throughout semester.   assistance provided at point of need. driven by questions. individual questions answered include apa, definitions of research types, where to search, cinahl. encouraged to contact the librarian by phone, email, or discussion board. online videos and tutorials for specific databases and ill. case report. no assessment mentioned. librarian listed as instructor in course management system. milstead & nelson, 1998, phd students, pennsylvania both program and nursing phd course -politics and health policy development.   no s/g/t mentioned. virtual (webpage) webpage available throughout the course. frequently used library functions/resources. vendor rep provided instruction for class on westlaw database. case report. mentioned assessment of library use and access, but no results provided. no assessment of instructional effectiveness mentioned. primary focus is development of program, limited discussion of library involvement, no librarian author on article. schilperoort, 2020, master’s students, california two clinical classes (no specifics on class name).   andragogy, constructivist learning theory virtual asynchronous interactive video tutorial. embedded in lms. self-paced, estimated 15 to 30 minutes to complete tutorial.   identifying level of evidence and locating library resources to find evidence. none mentioned. quasi-experimental. pre-post tests, survey of confidence with some open-ended questions. unique focus on clinical courses. tutorial was required; assignment was graded credit or no-credit. thompson, 2009, master’s students, canada class – research methods course   no s/g/t mentioned. f2f no indication of timing of class. compared two iterations with differing durations. first iteration was 1) 3-hour class, second iteration was 3) 3-hour classes.   1st iteration (3 hr. class) – lecture on basic concepts of data & quantitative research, demo of basic analysis in spss, hands-on practice with provided dataset. 2nd iteration (3 – 3 hr. classes) – 1st 3-hour class lecture on data concepts, 2nd 3-hour class, hands-on practice with spss, 3rd 3-hour class, answering questions and one-on-one assistance. assistance at the academic data center on a walk-in basis. survey (students’ feelings/ attitudes). 1st iteration – anecdotal evidence (instructor reported high grades on assignment). 2nd iteration – student survey. instruction tied to class assignment.   welch et al., 2016, phd students, georgia program -orientation   no s/g/t mentioned. virtual (online interactive modules). access before classes began, but not clear if modules had to be completed before classes began. no indication of duration of modules. 4 modules topics included scholarly writing, apa, library databases, lit reviews, research questions/ hypotheses, popular vs. scholarly, theoretical frameworks, endnote, planning a research study, research ethics. none mentioned case report. reports meeting as a group to discuss orientation assessments and evaluations but no results provided. describes shift to online modules for student support. whitehair, 2010, master’s, dnp, and phd students, kansas both program and two classes dnp capstone course phd on-site sessions.   practice doctorate nurse practitioner entry-level competencies; kuhlthau’s model of the information search process; kraft and androwich’s student-centered model. hybrid f2f, synchronous online instruction, videos. orientation preclass. dnp course beginning of semester, recorded; no indication of duration. q&a session several weeks later. phd students – 1st  & 3rd week included 1-hr library sessions; 2nd week individual meetings. orientations critical resources, off-site access. dnp capstone course lit searching, video tutorials, resources. phd sessions -1. library services, website, databases, 2. voluntary meetings. 3. complex searching, refining searches, current awareness services. one-on-one interaction with library liaisons was encouraged and available in person, via phone, online conferencing, and instant messaging. case report. no assessment mentioned. son faculty encouraged to add library contact info to the syllabus and to set up "ask a librarian" discussion boards in all courses. whiting & orr, 2013, dnp students, indiana both program, orientation   no s/g/t mentioned.     no indication of format of orientation. no indication of timing or duration of orientation. content that changed as a result of the research – improved explanation of ill and document delivery, more time spent on citing and citation resources, greater emphasis on nine nursing journals added to the collection in support of the new dnp program. librarians maintained a "library support" section within the general blackboard site. mixed methods. analysis of research paper reference lists and survey of library resources/services satisfaction but no assessment of instructional effectiveness mentioned. focus is support of dnp program over three years rather than instruction. wills et al., 2001, master’s students, michigan class nursing 811 concepts of research and evaluation for advanced practice nurses   no s/g/t mentioned. embedded available throughout semester.   cinahl, medline, proquest direct, and other health-science databases. individual consultations via email or f2f. discussion room in webtalk for questions and where the librarian posted content. case report. there was an end-of-class evaluation, but no assessment of library support was reported. focus is the development of an online nursing class in the master’s program, including info about library support. wimmer et al., 2014, phd students, utah class research with diverse populations   no s/g/t mentioned. virtual -synchronous second week of class was an orientation to library resources with question-and-answer session. no indication of duration. no information beyond that it was an orientation to library resources. librarian assisted with full-text, remote access, and ill. research guide for evidence-based nursing shared via course management system. case report. no assessment mentioned. focus is describing librarians' involvement in the creation of an e-textbook by students in the class.   evidence summary   the 360-degree temporal benefits model reimagines value-based assessment of user-centred design services   a review of: kautonen, h., & nieminen, m. (2018). conceptualising benefits of user-centred design for digital library services. liber quarterly, 28(1), 1-34. https://doi.org/10.18352/lq.10231     reviewed by: melissa goertzen consultant and information manager halifax, nova scotia, canada email: goertzen.melissa@gmail.com   received: 3 june 2018                                                                    accepted: 12 oct. 2018      2018 goertzen. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29473     abstract   objectives – the study has two central objectives: to examine the conceptual elements of evaluating and managing user-centred design (ucd) performance in library settings; and to propose a new framework, the 360-degree temporal benefits model (360°tb model), that assesses value-based evaluation of ucd performance in libraries.   design – data collection and analysis were conducted through literature reviews, case studies, semi-structured interviews, questionnaires, and reviews of digital library service documents.   setting – two digital library service environments in finland that use ucd approaches: one located at the national digital library and the other at a medium-sized special library.   subjects – there were 17 participants representing internal and external stakeholder groups such as digital service designers, end-users, and consumer organizations.   method – through a literature review, the authors studied several topics related to ucd services including digital services, design management, public value frameworks, and services. they examined literature from two theoretical perspectives: 1) performance management, which explains why and how performance evaluation is necessary for public services, and 2) temporality, the concept of time in relation to service provision. this lens allowed the authors to identify existing knowledge gaps in professional literature and define key concepts. the literature review informed the framework for the 360°tb model.   two digital library settings tested the model and served as case studies in the paper. data collection activities in this phase included reviews of existing project documentation and semi-structured interviews with stakeholders, at which time participants were also asked to complete an online questionnaire. the authors recorded and transcribed the interviews and combined these results with comments derived from questionnaires. finally, participants received the data collected from their interview sessions and were asked to review and validate their answers.   main results – the most significant result is the development of the 360°tb model. the framework combines three components to evaluate ucd design: the identification of stakeholders; the benefits of ucd services; and the temporal phases (e.g., process-time, use time, and future service provisions) of ucd design efforts and outcomes. the authors summarize the relationship between the components of the framework as follows: “a stakeholder anticipates benefits of the design in different phases” (p. 8).   regarding the case studies, the authors captured a range of diverse opinions through semi-structured interviews and questionnaires. participants in case 1 selected a range of benefits and there was little consistency in responses. however, two-thirds of participants in case 2 selected quality of services as the most desirable benefit of ucd, while the remaining one-third selected options such as process time and societal problem solving.   the participants stated that the 360°tb model provided authority in matters of design goals. it was challenging to capture temporality in design performance because it is not easy to specify goals or state the anticipated benefits of design activities in library settings. this is because the impact of design is indirect and cannot be easily quantified or isolated from the larger context of the library environment. the model provides a method to justify managerial choices regarding ucd and frame service changes around phases of development (e.g., process-time, use-time, and future service provisions).   conclusion – the 360°tb model pushes assessment activities beyond organization-centric evaluations and into intra-organizational and polycentric perspectives. it reaches beyond the boundaries of the institution to capture diverse viewpoints and service needs of external stakeholders. finally, the 360°tb model bridges the theoretical gap between public value frameworks and real-world information environments through the use of three key concepts: stakeholders, benefits, and phases.    commentary   for decades, librarians have experimented with and used performance indicators that provide evidence for the quality of library services. many of these assessments justify the benefits of ucd activities by accounting for the impact and outcome of services (best, 2010; rosenberg, 2004; wiebe, 2010). however, the profession has not developed a framework that evaluates the act of designing services. in order to fill this gap, the authors developed a study around one central question: what elements are essential when evaluating and managing ucd performance in libraries?   the paper covers the process of conceptualizing, developing, and testing the 360°tb model, which is unique because it takes a 360° view of stakeholders’ opinions, perspectives, and needs. essentially, the model captures the context surrounding ucd activities in order to support successful design plans in the present and allows services to evolve with the future needs of internal and external stakeholders. this characteristic is the framework’s greatest strength; by examining all stakeholder groups at a high level, ucd designers can identify conflicting interests or goals while services are still on the drawing board. once the values of the user community are identified, librarians can adjust service models or locate opportunities for stakeholder buy-in before services are released to the public. through this form of evaluation, libraries engage in proactive and strategic service management.   based on the results of the case studies, the authors conclude that the 360°tb model is not a mature tool for practitioners. going forward, they wish to develop the framework and transform it into a practical tool that supports ucd service design. there are two areas that the authors did not account for that would strengthen the model in future development phases. the first is to account for the limitations of ucd design alongside the anticipated benefits. since the framework predicts service evolution from the point of creation through future iterations, accounting for shortcomings places librarians in a proactive position: counteract the shortcomings or work with stakeholders to locate solutions. the second is to define the metrics and data sources that are appropriate for use with the 360°tb model. without this definition, it is difficult to standardize assessment practices and compare results of ucd services within the same institution or across the library profession.     since the 360°tb model is still in development, it would have been beneficial to librarians to learn about recruitment methods, interview questions, or the structure of the questionnaire in an appendix. as the paper currently stands, it would be difficult for others in the profession to replicate the study or explore how it can be applied to ucd service development at their institution.   overall, the authors developed an interesting assessment framework that has great potential and fills an observed gap in professional knowledge: the evaluation of the act of service design. in the future, it would be of value to the profession if the authors continue to publish papers about the development of the 360°tb model and discuss their methodology in greater detail.   references   best, k. (2010). the fundamentals of design management. lausanne, switzerland: ava publishing.   rosenberg, d. (2004). the myths of usability roi. interactions, 11(5), 22-29. https://doi.org/10.1145/1015530.1015541   wiebe, e. (2010). temporal sensemaking: managers’ use of time to frame organizational change. in t. hernes & s. maitlis (eds.), process, sensemaking, and organizing (pp. 213-241). oxford, uk: oxford university press. https://doi.org/10.1093/acprof:oso/9780199594566.003.0011   microsoft word art_clark.doc evidence based library and information practice 2007, 2:3  3    evidence based library and information practice         article    information skills survey: its application to a medical course       catherine clark   associate science librarian   the university of western australia library   perth, western australia   e‐mail: caclark@library.uwa.edu.au     dr. ralph catts   senior research fellow, applied educational research scheme   institute of education, university of stirling   airthrey castle, stirling, scotland   e‐mail: ralph.catts@stir.ac.uk       received: 26 april 2006   accepted: 16 may 2007      © 2007 clark and catts. this is an open access article distributed under the terms of the creative  commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits  unrestricted use, distribution, and reproduction in any medium, provided the original work is  properly cited.      abstract    objective ‐ to investigate the validity and reliability of the information skills survey for  assessment of information literacy in higher education (caul iss) (catts, “administration  manual”) for identifying the information literacy skills of first and fourth‐year medical  students.     the caul iss is a standardised, 20 item self‐report inventory of information literacy skills  of higher education students. it exists in two forms, namely a generic form and a law  discipline specific form. this paper is concerned with the suitability of the generic form of  the survey for use with medical students.     methods ‐ the generic form of the caul iss was administered to 86 first‐year and 120  fourth‐year medical students and the reliabilities were computed. in addition, students  were asked to respond to two open‐ended questions about their information literacy.  subsequently, having noted that the fourth‐year students rated themselves significantly  mailto:caclark@library.uwa.edu.au mailto:catts@stir.ac.uk http://creativecommons.org/licenses/by/2.0 http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:3  4 more severely on seven of the 20 items, four of this cohort were interviewed to identify the  extent to which the caul iss addressed the range of their techniques for information use  and to identify any specific content validity issues in the application of the caul iss scale  to this population.     results ‐ the reliability of the caul iss was confirmed for both years, but the evidence  from an analysis of the individual items and from the open‐ended questions and the  interviews indicated that the fourth‐year students adopt a wider range of information  gathering techniques, appropriate to their clinical experience than measured by the survey.  the caul iss demonstrated content validity for first‐year students but its scope was not  sufficient for content validity for fourth‐year students. further investigations are required to  determine the full scope of competencies required for content validity with the fourth‐year  cohort.    conclusion ‐ the evidence suggests that the generic form of the caul iss is suitable for  use to estimate the information skills of first year medical students. for fourth‐year students,  suggestions are made for the use of the caul iss in the context of their additional situated  information literacy.     introduction    lifelong learning can be defined as: “a  continuously supportive process which  stimulates and empowers individuals to  acquire all the knowledge, values, skills and  understanding they will require throughout  their lifetimes and to apply them with  confidence, creativity and enjoyment in all  roles, circumstances and environments”  (will 5).    to become lifelong learners, individuals  must have access to needed information,  and must also be able to judge the quality of  the information to which they are exposed  (candy 8). therefore, information literacy is  an essential component of the set of generic  capabilities needed for lifelong learning.    the definition of information literacy has  been the subject of debate. the majority of  definitions have the learner, rather than the  teacher, as a common focus. one frequently  used definition is that of the american  library association presidential committee  on information literacy: “ultimately,  information literate people are those who  have learned how to learn. they know how  to learn because they know how knowledge  is organised, how to find information and  how to use information in such a way that  others can learn from them. they are people  prepared for lifelong learning, because they  can always find the information needed for  any task or decision at hand”(ala final  report 1).     a definition specific to higher education is  given by the society of college, national  and university libraries: “information  literacy encompasses library user education,  information skills training and education,  and those areas of personal or transferable  ‘key’ skills relating to the use and  manipulation of information in the context  of learning, teaching and research issues in  higher education“(sconul 3).    information literacy is therefore a wider  concept than that described by librarians as  the information seeking process (u  auckland 1):     • defining the topic   • selecting and using resources   evidence based library and information practice 2007, 2:3  5 • locating information   • evaluating resources   • documenting the research     being information literate in one subject area  does not mean that this expertise can  necessarily be carried over to another  subject area (candy 7; george 282). for  example, the information skills required in  humanities are quite different from those  required in a discipline such as engineering  (palmer and tucker 19). candy, in his  chapter ʺmining in cyberiaʺ(144), notes that  information literacy in each discipline may  incorporate a range of different sources and  strategies.  information literacy also requires  the ethical and legal use of information,  which again is a situated practice.     information literacy is therefore discipline‐ specific and hence requires application in  the teaching and learning programmes  within each discipline.     the identification of the importance of  information literacy in lifelong learning has  led to a focus on information literacy  instruction in the teaching and learning  programmes of university libraries (marcum  2; orr and cribb 43). university librarians  recognise that if information literacy is to be  acquired, training must be managed using a  planned instruction programme (lupton  25).     in universities, this has translated into a  push to include information literacy as part  of course curricula, rather than stand‐alone  sessions run by the library. many librarians  now work closely with faculty to ensure that  information literacy is included in curricula  along with other generic attributes, such as  communication skills (bundy 1). the  emphasis on curricula development has  required librarians to collaborate with  academic colleagues and put more time into  liaison and course development. this,  combined with the change from content to  skills‐based information training, has  become a growing source of demand for  resources (gedeon 2; marcum 2; orr and  cribb 49).     in australia, the emphasis on information  literacy in higher education was evidenced  by the publication of a major information  literacy work. information literacy standards  was produced in 2001 by caul in  conjunction with the australian and new  zealand institute for information literacy  (anziil). this publication was updated in  2004 and published as the australian and  new zealand information literacy framework  (bundy). both of these publications were  designed to guide higher education  institutions in developing effective  information literacy programmes and  excerpts from each publication are  presented in appendix 1.     evaluation of information literacy programmes   the assessment of individual students’  learning in information literacy programmes  can include quantitative, qualitative and  practice demonstrations (palmer and tucker  25). assessment embedded within the  course has been able to measure individual  students’ information literacy learning in a  discipline or class context. this has been  done in a number of ways including the  production of bibliographies and search  strategies (lupton 26; iannuzzi 305).     the evaluation of programme effectiveness  requires data collection from groups of  students. hence the interest at this level is  on the aggregate scores and the stability of  the group parameters, not the individual  parameters because it is the programme  effectiveness that is the focus. while the  research literature has many examples of  information literacy programmes, the  evaluation of groups of students has proved  to be more problematic and research  reporting evaluations that use validated  evidence based library and information practice 2007, 2:3  6 research tools is very limited (palmer and  tucker 25, bernath and jenkin 4, brettle 6,  iannuzzi 305, koufogiannakis and wiebe  19). a search of the literature for the period  1990 to 2006 (embase, eric, lisa, lista,  medline) did not identify any reported  evaluations of information literacy  programmes using validated survey  instruments to evaluate changes in  information skills throughout the medical  course.     the use of a standardized tool to evaluate  the information literacy skills of large  groups of students would allow  comparisons to be made between the  effectiveness of information literacy  programmes in different populations within  the same institution or in benchmarking  across institutions. the effect of changes to  teaching and learning strategies for  information literacy could also be more  systematically measured.     information literacy programmes in medicine   in some disciplines, such as clinical  medicine, information literacy forms part of  a broader movement that is aimed at  ensuring that the skills of clinicians reflect  current best practice. in health care, this has  been termed evidence‐based clinical  practice (ebcp). ebcp follows a five‐step  process that includes finding the most  judicious and explicit clinical evidence, and  applying it to an individual patient.  librarians have begun to promote teaching  in ‘finding the evidence’ and are working to  ensure that it is part of the university  curriculum in a wide range of health science  areas (jacobs 327; mckibbon 25). to be able  to find the best evidence, practitioners must  be information literate within the medical  context. sackett promotes the role of  librarians in assisting practitioners to find  evidence‐based information (sackett 210).     the ebcp movement has led to a change in  focus for medical librarians. training has  been developed to ensure that practitioners  become information literate, rather than  simply being orientated in their use of a  particular library. studies have been  undertaken on the ability of continuing  medical education to change the habits of  health practitioners (dornan 670; fox and  bennett 466). however, it has been difficult  for any studies to pinpoint the effect of  information literacy programmes in  changing the habits of clinicians.  information literacy is rarely taught in  isolation and it is hard to separate the effects  of training by librarians from that learnt by  other means (e.g. consultation with  colleagues).     the trend towards ebcp in the clinical  situation has been coupled with a change in  the way that medical students learn at  university. the shift from lecture‐based  teaching to that of problem‐based learning  (from teacher‐centred to student‐centred),  has required students to be more  sophisticated in the techniques they use to  locate information. there has also been a  change in the direction of education away  from the principle that everything you need  to know can be learned at university,  towards an approach that regards university  education as a way to learn the critical  thinking and research skills required for a  lifetime of learning.     caul information skills survey   the lack of a suitable validated instrument  to evaluate information literacy  programmes has been recognized in the  united states and australia. the us  institute for information literacy prepared a  project plan in january 2004 that has as one  of its goals to“develop criteria for assessing  information literacy programs”(institute  information literacy 5). in australia,  concerns about how to measure the  evidence based library and information practice 2007, 2:3  7 information literacy skills of groups of  students were identified by caul and  addressed with the publication of the  information skills survey for assessment of  information literacy in higher education. the  caul iss was designed and developed by  the caul information literacy assessment  project team. the caul iss in its generic  form consists of twenty questions in a self‐ assessed closed questionnaire, with students  ranking their information skills on a scale of  0 (never) to 3 (always). the use of self‐report  as a valid means of gaining evidence from  students has been supported by several  writers (lally and myhill 27; marsh 722) and  is widely used in evaluating student  experiences. for instance, in australia the  course experience questionnaire (wilson,  et al 34) relies on self‐report.     the caul iss was designed using the  information literacy standards (caul ils)  which were revalidated for use with the  second edition, published as the australian  and new zealand information literacy  framework (anz ilf) (bundy) (appendix  1). the generic form of the survey was  benchmarked with students studying  education. the reliability (the ability to  reproduce the scores) has been  demonstrated to meet the criteria for  standardised test for the scale overall  (r=0.87). satisfactory levels of reliability  (between 0.54 and 0.78) were reported for  each of the sub‐scales that measured  standards two to six as listed in the anz  ilf. these results compare favourably with  other standardised instruments used in  educational research (see for instance,  marsh et al 717). the caul iss project  team noted that the survey developed with  students in education was designed for use  in a range of social science contexts. the  project team recommended that the caul  iss should be tested with students in other  disciplines, including science and  engineering. this study investigated this  issue in relation to the discipline of  medicine.     validity     validity is fundamental to the utility of a   measurement tool because it confirms that  the instrument measures what it purports to  measure, and must be examined in the  context in which the survey is used. four  aspects of validity have been identified as  necessary to establish the quality of a survey  (lally and myhill 17). these forms of  validity are termed content, construct,  concurrent and predictive validity. for the  caul iss the content validity was  demonstrated by expert judges matching  survey questions with the relevant  standards. evidence of construct validity  was demonstrated in the statistical analysis  of the caul iss, with details reported in the  caul iss technical manual. the concurrent  validity of the caul iss was established by  comparing student self‐report on the survey  with assessments by librarians of the  individual’s information skills, using  observation and interview. evidence of  predictive validity requires research in  specific contexts to demonstrate whether the  caul iss predicts the information literacy  performance of students. the generic form  of the caul iss has demonstrated content,  construct and concurrent validity within the  context in which the survey was developed,  but catts (“administration manual” 2) calls  for consideration of validity in other higher  education contexts including medicine  which would avoid the costly task of  creating a new form of the survey.   the central research questions for this study  were:     • is the caul iss a reliable  instrument for providing data on  information literacy outcomes for  groups of first year and advanced  medical students?   evidence based library and information practice 2007, 2:3  8 • is the caul iss valid for  identifying information literacy  skills for the above cohorts of  students?     methodology   the university of western australia   this study was conducted at the university  of western australia (uwa). uwa has an  enrolment of approximately 16,000 students  in nine faculties. the bachelor of medicine  and bachelor of surgery is a six‐year  undergraduate and four‐year postgraduate  programme offering integration of science  and clinical teaching.     the information literacy programme for  medical students at uwa is based around  the ʹspirallingʹ curriculum with incremental  development and a revisiting of important  concepts over six years. a curriculum‐ integrated approach has been taken with the  development of information skills  integrated into the teaching, learning and  assessment of curriculum objectives and  content. the information literacy  programme is based around outcomes  required of a medical graduate and uses the  anz ilf (bundy) and its associated  learning outcomes. an extract from the  uwa information literacy framework for  medical students (first and fourth‐year) is in  appendix 3. in 2006, library staff conducted  100 information literacy sessions for faculty  and students.   study design   the caul iss was administered to first‐ year medical students in october 2004 and  fourth‐year medical students in january  2005 as described below.     to confirm the content validity of the  survey, at the time the caul iss was  administered, students were asked to  answer two open‐ended questions on  whether or not they thought the caul iss  covered all areas of information literacy that  would be required by medical students in  their academic year.     sampling and recruitment procedures   respondents   first year students were assumed to have  general information literacy knowledge  similar to other groups of first year students  with whom the instrument has been  validated. it was therefore hypothesised that  the generic version of the caul iss would  be content valid and reliable for this group.  first‐year undergraduate medical students  at uwa complete a general course of study  and have not yet been exposed to  information sources exclusive to medicine.     students in their fourth year have been  exposed to a significant amount of discipline  specific knowledge (see examples in  appendix 3). ideally, the study would have  followed up with the first‐year cohort some  three years later, but immediate information  was sought for the purpose of course  review. although this is a limitation of the  study, the content validity and reliability of  the caul iss in different years of the course  can still be determined by gathering data  from two different cohorts.     all first and fourth‐year students who were  attending prescribed lectures, were  informed of the purpose of the research,  invited to complete the anonymous survey  instrument and given the option to  withdraw. all students in attendance at the  lectures agreed to complete the survey.  eighty‐six first year students, representing  46% of the cohort, and one‐hundred and  twenty fourth‐year students, representing  86% of the fourth year cohort provided data.  students under the age of 18 were not  permitted to participate for ethical reasons  (parental permission would have been  evidence based library and information practice 2007, 2:3  9 required). students under the age of 18  comprised approximately 10% of the first  year student cohort at the time the survey  was completed. however, there were no  students under the age of 18 at the lectures  where the caul iss was completed.   students who were not at the prescribed  lectures were not invited to complete the  survey.     materials   the contents of the information skills  survey are confidential to ensure that  responses are not rehearsed or students  encouraged to misrepresent their  knowledge and practice. such actions would  harm the validity of the survey to the extent  that students were prompted to modify  their behaviours. however, recently caul  agreed to the publication of the survey in  academic journals, with the proviso that  readers are asked to note that it is to be used  in conjunction with the caul iss  administration manual which can be  accessed from the caul web site  (www.caul.edu.au). consequently the  survey is included as appendix 2.    to confirm the content validity of the caul  iss in the context of medicine, students  were asked at the time of completing the  survey to give a written response to two  additional questions:     are there any skills related to  research and information gathering  for your first/fourth‐year medical  studies that are not covered by the  survey?  if so, what are they?   are there any questions in the  survey that are not applicable to  your first/fourth‐year medical  studies?  is so, what are they?     gathering student opinions complemented  the expert judgements used in the survey  design, and allowed a direct consideration  of content validity in the situation where its  use was to be trialled.     interview procedure   having noted that the fourth‐year students  rated themselves significantly more severely  than did first year students on some seven  items, four students from this cohort were  interviewed in a post‐hoc analysis to seek to  identify the extent to which the caul iss  addressed the range of their techniques for  information use. the selection of these four  students was on the recommendation of  academic staff from those students who  were available. two male and two female  students were selected (the medical course  at uwa comprises approximately 50  percent male and 50 percent female  students). interviews were conducted in a  semi‐structured way with the interviewer  asking a series of questions and students  responding to build an understanding  through discussion as promoted by holstein  (56). an outline of the topics discussed in  the interviews is included in appendix 5.     reliability   the reliability of a measurement instrument  estimates how consistently each item  measures the construct of interest, which in  this case is information literacy. a perfectly  reliable instrument in which the responses  to all items by any participant are entirely  consistent for that participant would  produce a reliability of one across the group  of participants. if each individual responded  completely at random to each item in the  survey the reliability for the group would be  zero. hence the higher the reliability that is  reported, the better is the measurement  instrument.   the caul iss was administered to the  groups of students on only one occasion.  coefficient alpha (sometimes called  cronbach alpha) was calculated to  http://www.caul.edu.au evidence based library and information practice 2007, 2:3  10 determine the reliability of the caul iss.  coefficient alpha is recommended as the  best estimate for cases where the survey is  administered once and is the most common  measure of the reliability of measurement  instruments in social science (bryman and  cramer 77). a high alpha value would  demonstrate that the survey questions  produce positively inter‐correlated  responses that measure the same concept  (bryman and cramer 77). for standardised  tests it is recommended that alpha should be  above 0.85 for decisions that affect  individuals and above 0.65 for inferences  about groups (frisbie 29).     results  reliability     using cronbach alpha, the reliability was  calculated for first year students at 0.85 and  0.84 for fourth‐year students demonstrating  the consistency of the caul iss for both  years. these results confirm the reliability of  the caul iss because they are consistent  with the reliability of 0.87 reported in the  caul iss technical manual .     performance on survey   the caul iss item numbers and their  alignment to the core standards from the  information literacy framework, together with  a comparison of mean scores, and standard  deviations and reliabilities are shown in  tables 1 and 2. the level of significance of  differences is shown in table 3. items that  are significant at the 5% level are in shaded  bold italics. note that there are no questions  in the caul iss relating to standard 1 (the  information literate person recognizes the  need for information and determines the  nature and extent of the information  needed).     the mean score for first‐year students on the  20 item scale (possible range 0 to 60) was  37.19 with a standard deviation of 8.14. the  mean score for fourth‐year students on the  20 item scale was 34.88 with a standard  deviation of 7.72.     the difference in favour of first‐year  students is evident in all five subscales. the  caul iss questions grouped by anz ilf  core standards mean scores for years 1 and  4 are reported in table 1. the mean score  was lower for the fourth‐year students in  each of the subscales. the significance of  this result is discussed below. it is noted that  the reliabilities for the sub‐scales were  consistent with the results reported in the  caul iss technical manual (catts, 2005).     as the items in the caul iss each provide  data on an ordinal scale with a maximum  range of four, the statistical significance of  any difference in performance between first  and fourth‐year students for each of the  twenty items was determined using a non‐ parametric test (mann‐whitney). this is a  conservative test that does not depend upon  the assumption of a normal distribution.  seven of the twenty items were statistically  significant at the 5% level. table 3 shows the  results of these calculations. there were  seven items that were significantly different  and these are shaded in the table. there is a  significant difference in results for at least  one item from each of the subscales.  however, the results of the tests were in an  unexpected direction as the first‐year  students had higher ratings than the fourth‐ year students on the items that were  significant. the specific items are listed in  appendix 4 and the reasons and  implications for this finding are considered  in the discussion that follows.     validity of caul iss for medical students     to investigate the validity of the caul iss  for medical undergraduates, students were  asked to give written responses to two  additional open‐ended questions. the   evidence based library and information practice 2007, 2:3  11 australian and new zealand information literacy framework  core standard number and associated caul information skills  survey numbers  mean   1st‐year   mean  4th  year   anz ilf standard 2 finds needed information effectively and  efficiently   (caul iss numbers 3,12,16,19)   8.00   7.52   anz ilf standard 3 critically evaluates information and the  information seeking process   (caul iss numbers 5,8,11,14)   6.84   6.12   anz ilf standard 4 manages information collected or generated   (caul iss numbers 1,2,6,9)   5.52   5.12   anz ilf standard 5 applies prior and new information to  construct new concepts or create new understandings   (caul iss numbers 4,15,17,18)   7.60   7.20   anz ilf standard 6 uses information with understanding and  acknowledges cultural, ethical, economic, legal, and social issues  surrounding the use of information   (caul iss numbers 7,10,13,20)   9.24   8.92     table 1. comparison of subscale means for first‐year and fourth‐year students    responses of the first year students indicated  that the caul iss survey items represented  a satisfactory sample of their information  literacy strategies. the analysis of the first‐ year students who responded to the open‐ ended questions is reported below. of the  sixty respondents, 30/60 (50%) reported that  they could not identify any skills related to  research and information gathering for their  medical studies that were not covered by the  survey. a further 8/60 (13%) gave no  response and 2/60 (3%) stated that they  “don’t know”. the remaining 18/60 (30%) of  first‐year students listed comments that  relate to elements from the anz ilf  information literacy standards. these  included finding information effectively and  efficiently (information literacy standard 2),  evaluating information (information  literacy standard 3) and a comment relating  to legal issues in using information  (information literacy standard 6). these  responses suggest that the caul iss survey  items represented a satisfactory sample of  their information seeking strategies. in   answering the question as to whether there  were any questions in the survey that were  not related to their first year studies, 34/60  (57%) students reported that they had no  suggested changes, 10/60 (17%) gave no  response and 3/60 (5%) stated that they  “don’t know”. a further 7/60 (12%) students  gave comments relating to applying new  information to construct new concepts  (information literacy standard 5) and 13/60  (22%) of students wrote comments relating  to legal issues in using information  (information literacy standard 6). these   evidence based library and information practice 2007, 2:3  12   australian and new zealand information literacy framework  core standard number and associated caul iss numbers  mean   sd   alpha   anz ilf standard 2 finds needed information effectively and  efficiently   (caul iss numbers 3,12,16,19)   7.71   1.86   0.50   anz ilf standard 3 critically evaluates information and the  information seeking process   (caul iss numbers 5,8,11,14)   6.43   2.32   0.73   anz ilf standard 4 manages information collected or generated   (caul iss numbers 1,2,6,9)   5.30   2.16   0.55   anz ilf standard 5 applies prior and new information to construct  new concepts or create new understandings   (caul iss numbers 4,15,17,18)   7.36   2.17   0.65   anz ilf standard 6 uses information with understanding and  acknowledges cultural, ethical, economic, legal, and social issues  surrounding the use of information   (caul iss numbers 7,10,13,20)   9.06   2.16   0.55     table 2. estimates of subscale means and sd for combined first‐year and fourth‐year responses  (total of 196 responses)     results suggest that the content of the caul  iss is valid for this group of 1st year  students.     a similar analysis was conducted of the  responses to the written questions by the  fourth‐year students. while the majority of  responses related to elements of the anz  ilf standards, a small number  ofrespondents indicated that the situated  nature of medical information literacy is an  issue. of the forty‐two fourth‐year students  who responded to the open‐ended questions,  25/42 (60%) of the fourth‐year students  reported that they had no suggested  additions to the survey in relation to the  skills they needed to research and gather  information for their fourth‐year medical  studies. a further 7/42 (17%) recorded no  response and one student (2%) responded  with the comment “don’t know”. unlike the  first‐year students, the fourth‐year students  did not report any additional comments  related to standards 2 and 3 but one did  make a comment relating to the legal use of  information (information literacy standard  6). however, of particular note were a series  of comments relating to the use of  information in the clinical setting indicating  that the situated nature of medical  information literacy was an issue. for  example, 3/42 (7%) of fourth‐year students  commented that the survey did not cover  the importance of comparing research to  actual clinical experience with comments  such as “talking to skilled clinicians”,  “gathering information from human  sources”. a further 4/42 (10%) of students  evidence based library and information practice 2007, 2:3  13   caul iss  item  number   anz ilf core standard  number   mean   1st‐year   mean   4th‐year   mann‐ whitney u   p values   1   4   1.57   1.13   3708   <.001   2   4   1.12   0.93   4448   .060   3   2   2.03   .804   4874.5   .465   4   5   1.68   1.34   3947.5   .004   5   3   1.70   1.29   3703   <.001   6   4   1.68   1.92   4355.5   .075   7   6   2.51   2.48   4820   .357   8   3   1.84   1.51   3949   .002   9   4   1.14   1.13   5019   .723   10   6   2.45   2.48   5151   .980   11   3   1.85   1.78   4854   .434   12   2   1.59   1.37   4303.5   .035   13   6   2.03   2.00   4908   .673   14   2   1.50   1.58   5021   .724   15   5   1.84   1.87   5058   .873   16   3   2.47   2.42   5014   .697   17   5   1.92   1.95   5156   .992   18   5   2.20   1.98   4320   .032   19   2   1.88   1.76   4755   .289   20   6   2.29   2.02   4342.5   .039     table 3. level of significance of caul iss questions as determined by non‐parametric tests  evidence based library and information practice 2007, 2:3  14 reported comments relating to the use of  consolidated information sources such as  clinical evidence and one student noted that  the survey did not cover the need to be  aware of library subscriptions to otherwise  password protected sources.     the analysis of the second follow‐up  question for the fourth‐year students  highlighted that the majority of the students  (27/42, 64%) could not identify any items on  the survey that were not applicable to their  fourth‐year studies and a further 5/42 (12%)  of students gave no response. however, 4/42  (10%) of the students reported that the  questions related to organising information  (information literacy standard 4) were not  applicable to their fourth‐year studies. in  addition 2/42 (5%) reported comments  relating to applying new information to  construct new concepts (information  literacy standard 5) and 7/42 (17%) of  students wrote comments relating to legal  issues in using information (information  literacy standard 6).     interview analysis   the unpredicted differences between the  first‐year and fourth‐year cohorts in the  survey responses and the analysis of the  follow up questions suggested that there  were differences in the way the fourth‐year  students used information that required  further investigation. this was gathered in  follow up interviews with four fourth‐year  students (see appendix 5).     during the interviews, students described  some of the medical resources they use  regularly and how they can search these  more effectively. the students discussed the  process they use to find the answers to  clinical questions that arise in the hospitals.  one student commented “my first point of  call normally is to go to the textbooks. first i  tend to refresh the whole problem and the  topic in my mind and, say pancreatitis, i  might go and find out in a basic pathology  book [from years 1 and 2] and something a  little bit more about the management of it  and then for more specific questions i  generally tend to start at medline. the  evidence‐based stuff [e.g. clinical evidence]  since the lecture at the start of the year”.   when asked about the differences that they  perceived between finding information  required for their studies in the pre‐clinical  versus the clinical years, the fourth‐year  students covered a number of areas.  selected quotations are: “well, probably the  first port of call would be to speak to the  immediate senior. just ask them for some  resources that they might suggest”, “i was  only really exposed to clinical evidence this  year and some of the other databases”, “i  think on the wards we refer to senior staff”,  “you’ll get some senior staff who are very  good at suggesting [sources of  information].” this response illustrates the  shift from a naïve learner dependent upon  texts, manuals and other documents, toward  a professional practitioner engaged in  information exchange within the profession.     discussion     the reliability of the generic form of the  caul iss for medical students is  demonstrated through the high cronbach  alpha scores for both the first and fourth‐ year students. the high overall reliability is  a necessary pre‐requisite for concurrent  validity – if the survey was not reliable the  scores could not be reproduced, and hence  one would expect low correlations with  other evidence of information literacy.  however, the detailed analysis revealed that  the responses from fourth‐year students to  some of the items produced lower self  ratings and this raised a question about the  predictive validity of the generic caul iss  for comparing first and fourth‐year  students.       evidence based library and information practice 2007, 2:3  15 issues with specific items   on each of the items, the first year students  out performed the fourth‐year students. for  those seven items where the difference is  significant, possible explanations for the  differences are described below.     anz ilf core standard 2: finds needed  information effectively and efficiently (caul  iss item 12: i have a system for searching for  information on a subject).     item 12 relates to using a system for finding  information. first‐year students may have  rated themselves significantly higher here  because they are using the same system for  finding information that they have been  using for a number of years. in their  university studies, the first‐year students  would have been researching a topic in  much the same way they did at high school  (approximately 85% of first year medical  students at uwa are school leavers).  finding information for first year  assignments revolves around using  textbooks, pre‐determined journal articles  and web sites. the system that has been  used in, for example, high school would still  be adequate for these first year students.  they have yet to become aware of the wide  gamut of formal and informal information  resources that are required by medical  practitioners.     in contrast fourth‐year students have  become aware of additional resources used  in the clinical setting and for more thorough  research, but their junior status may limit  their access to professional networks. the  fourth‐year students identified databases  such as the cochrane library and clinical  evidence as resources used in addition to  medline. although fourth‐year students are  aware of the range of material available to  them, they may still be trying to develop a  system in which they can identify the  relevant resources for a variety of clinical  and non‐clinical information needs. fourth‐  year students may find it more difficult to  develop a system as they have more parts to  fit together.     anz ilf core standard 3: critically evaluates  information and the information seeking process  (caul iss item 5: i critically evaluate each  information sources i use, and caul iss item  8: i evaluate information i read for criteria  including accuracy and relevance).     items 5 and 8 relate to evaluating  information. item 5 relates to evaluating all  information sources that are used while item  8 specifies evaluating for criteria such as  accuracy and relevance. the first‐year  medical curriculum has a strong emphasis  on evaluating information. by fourth‐year,  students should be obtaining information  only from reputable sources such as  refereed journals. item 5 relates specifically  to evaluating information sources. a  possible explanation for this difference  between the two groups is that the fourth‐ year students are evaluating their  information sources at an early stage of their  searching. that is, the evaluation is done at  the stage of choosing the information  resource itself rather than the specific  information that is found within. further  investigation is needed to confirm whether  the response by fourth‐year students can be  explained by their confidence in the  information resources they are using. if they  believe these resources contain reliable  information, this would explain why they  do not consider it so important to evaluate  each time they select from a specific  information resource.     anz ilf core standard 4: manages  information collected or generated (caul iss  item 1: i have a system that helps me organise  the information i need).     evidence based library and information practice 2007, 2:3  16 item 1 relates to having a system to organise  information. clinical coursework in fourth‐ year requires organising different types of  information to that used in the pre‐clinical  years. fourth‐year students are being  exposed to patient information, and to  hospital guidelines and procedures. most of  these would be new to the fourth‐year  students and could explain why they do not  rate themselves as highly as first‐year  students. first year students are organising  a limited amount of information from  familiar resources (books, journals, web  sites) whereas the fourth‐year students are  organising those as well as the clinically  specific information.     anz ilf core standard 5: applies prior and  new information to construct new concepts or  create new understandings (caul iss item 4:  when i get a new idea, i work out how to explain  it effectively, and caul iss item 18: i compare  information as i’m reading with what i already  know).     standard 5 relates to applying found  information to construct new concepts. item  4 regards effectively explaining new ideas  and item 18 relates to comparing found  information with what the student already  knows. it is unclear why the fourth‐year  students would rate themselves lower on  these items than the first year students. one  possible explanation is that the fourth‐year  students are no longer engaged in research  essays that require them to look at  information from a wide variety of books  and journals and then summarize these and  add their own ideas. rather, in the clinical  setting, the students are required to apply  the best available evidence to the individual  patient. therefore their focus may be more  on making the best judgement based on  known protocols of practice rather than  formulating new concepts. resources used  in the clinical setting (such as clinical  evidence) are pre‐packaged sources of  information and the students are not  required to explain new ideas in the same  way that they do in a standard  undergraduate essay. teaching and learning  revolves around case studies and problem‐ based learning.     anz ilf core standard 6: uses information  with understanding and acknowledges cultural,  ethical, economic, legal, and social issues  surrounding the use of information (caul iss  item 20: i comply with stated restrictions on the  use of intellectual property).     item 20 relates to intellectual property  which is not a term that is often used in  information literacy teaching and learning  activities in the uwa medical course. first  year students are possibly not as aware of  the restrictions of intellectual property and  so rated themselves more highly than the  fourth‐year students who “know what they  don’t know”.     a short survey need not cover every aspect  of information literacy in order to be a valid  survey. it appears from the responses of first  year students that this applies and the  survey can be considered valid in their  context. however, for fourth‐year students,  it appears that regard to the information  sources that best inform evidence based  practice need to be considered.   the significant difference between the two  groups of students in seven of the survey  items and the lower self‐rating of the fourth‐ year students may indicate that the content  is not sufficiently broad to reflect the  situated information literacy of the higher  level medical students.     this is confirmed by the responses to the  additional questions and the interviews  with the higher level students. the  information skills that students need for the  higher (clinical) years of the medical course  are different to that for the lower (pre‐ clinical) years. analysis of the areas of  difference showed that the fourth‐year  evidence based library and information practice 2007, 2:3  17 students’ approach to finding, evaluating  and using information has begun to change  in line with their exposure to clinical  practice. first year students focus their  information skills around the use of  textbooks, journal articles and web sites (in  this way first year medical students are  similar to the other first year cohorts for  whom the caul iss was designed).   fourth‐year medical students are becoming  aware that there is a much broader range of  information resources to draw from,  including colleagues. information resources  for fourth‐year students must include those  they develop as they are exposed to the  clinical setting, and the consequential  lowered self‐rating on the generic caul  iss, which illustrates their increasing  awareness of the complexity of information  retrieval. the information from the  additional questions and interviews  indicates that the caul iss content is not  valid for higher level medical students. this  is in line with the concept of situated  information skills anticipated by the  developers of the caul iss.     in their responses to the questions posed at  the conclusion of the administration of the  caul iss, 30% of the first year students  identified information relating to standard 2  as not being included in the caul iss. the  four caul iss items that relate to standard  2 are general items on searching for  information (i use a combination of search  tools including library catalogues and web  search engines, i have a system for searching  for information on a subject, if my searching  returns too much irrelevant information, i  change my keywords, i decide how best to  find the information i require for a  particular task) and do not specifically  mention, for example, using databases to  find journal articles. database searching is a  requirement for first year medical students.  their responses may indicate that either  they do not associate the items with data  base searching, or that as novice researchers  they do not associate generic statements  about search methods with the procedures  they employ.     fourth‐year students identified two main  areas of difference in the additional  questions and in the interviews relating to  the clinical setting. the first is the need to  consult with colleagues and the second is  the specific skills that are required to find  information to answer clinical questions.  these are areas that could be included in the  development of a form of the caul iss  suitable for use with advanced medical  students.     conclusion   the generic caul iss is reliable when used  with both first and fourth‐year medical  students. the validity of the generic caul  iss for first year medical students is  consistent with the findings in the technical  manual and it can be used for this group.  for fourth‐year students validity has not  been confirmed and further research in the  area should focus on designing and  validating a discipline specific version of the  survey. this could include additional  questions on recognizing the need for  information, using colleagues as a source of  information and the use of consolidated  information resources integrated with  relevant questions from the current version  of the caul iss. these findings are  consistent with the situated nature of  information literacy in higher education.       works cited  ala. final report from the presidential  committee on information literacy.  washington, dc, 1989.      bernath, v., and j. jenkin. ʺevaluation in  curriculum development for  information literacy: an australian  evidence based library and information practice 2007, 2:3  18 example using a canadian  questionnaire.ʺ 4th international  lifelong learning conference.  yeppoon, queensland: central  queensland university, 2006.      brettle, a. ʺinformation skills training: a  systematic review of the literature.ʺ  health information & libraries journal  20 suppl. 1 (2003): 3‐9.     bruce, c. ʺinformation literacy as a catalyst  for educational change: a background  paper.ʺ white paper prepared for  unesco, the us national commission  on libraries and information science,  and the national forum on  information literacy for use at the  information literacy meeting of  experts. prague, the czech republic,  2002.     bryman, a. and cramer, d. quantitative  data analysis with spss 12 and 13.  east sussex: routledge, 2005.    bundy, a. australian and new zealand  information literacy framework:  principles, standards and practice. 2nd  ed. adelaide: anziil, 2004.     candy, p. lifelong learning and  information literacy (white paper).  prague, the czech republic: unesco,  2002.     candy, p. ʺmining in cyberia.ʺ  information  literacy around the world: advances  in programs and research. eds. c.  bruce and p. candy. wagga wagga:  centre for information studies, charles  sturt university, 2000.      catts, r. information skills survey:  technical manual. canberra: council  of australian university librarians,  2005.      catts, r. information skills survey for  assessment of information literacy in  higher education: administration  manual. canberra: caul, 2003.     catts, r. ʺlifelong learning and higher  education: reflections and prospects.ʺ  3rd international lifelong learning  conference. yeppoon, queensland,  2004.      caul. information literacy standards.  canberra: caul, 2001.     dornan, t., et al. ʺevaluation of self‐ directed clinical education: validation  of an instrument.ʺ medical education  38.6 (2004): 670‐8.     fox, r. d, and n. l. bennett. ʺcontinuing  medical education: learning and  change: implications for continuing  medical education ʺ bmj 316 (1998):  466‐68.    frisbie, d. ʺreliability of scores from  teacher‐made tests.ʺ educational  measurement: issues and practice 7.1  (1984): 25‐35.    gedeon, j., c. radcliff, and l. oʹconnor.  ʺproject sails: facing the challenges on  information literacy assessment.ʺ  aahe assessment conference.  denver, colorado, 2004.     george, r., et al. ʺinformation literacy ‐ an  institution‐wide strategy.ʺ aarl 32.4  (2001): 278‐93.   holstein, j. the active interview. thousand  oaks: sage publications, 1995.     iannuzzi, p. ʺwe are teaching but are they  learning: accountability, productivity  and assessment.ʺ journal of academic  librarianship 25.4 (1999): 304‐05.     evidence based library and information practice 2007, 2:3  19 institute for information literacy. best  practices and assessment of  information literacy programs: a  project plan prepared for the  association of college and research  libraries a division of the american  library association. chicago, il:  institute for information literacy, 2004.     jacobs, s. k., p. rosenfeld, and j. haber.  ʺinformation literacy as the  foundation for evidence‐based  practice in graduate nursing  education: a curriculum‐integrated  approach.ʺ journal of professional  nursing 19.5 (2003): 320‐8.     koufogiannakis, d., and n. wiebe.  ʺeffective methods for teaching  information literacy skills to  undergraduate students: a systematic  review and meta‐analysis.ʺ evidence  based library and information  practice, 2006. vol. 1.      lally, m, and m myhill. teaching quality:  the development of valid instruments  of assessment. canberra: australian  government publishing service, 1994.     lupton, m. ʺcurriculum alignment and  assessment of information literacy  learning.ʺ  australian and new  zealand information literacy  framework: principles, standards and  practice. ed. a bundy. 2nd ed.  adelaide: australian and new zealand  institute for information literacy, 2004.   marcum, james w. ʺrethinking information  literacy.ʺ library quarterly 72.1  (2002): 1‐26.     marsh, h. ʺstudentsʹ evaluations of  university teaching: dimensionality,  reliability, validity, potentials biases,  and utility.ʺ journal of educational  psychology 76 (1984): 707‐727.    mckibbon, a. pdq: evidence based  principles and practice. hamilton,  ontario: bc decker inc, 1999.    orr, d., and j. cribb. ʺinformation literacy‐‐ is it worth the investment?ʺ australian  academic & research libraries 34.1  (2003): 42‐51.     palmer, s., and b. tucker. ʺplanning,  delivery and evaluation of  information literacy training for  engineering and technology  students.ʺ aarl march (2004): 16‐34.      sackett, d. evidence‐based medicine: how  to practice and teach ebm. 2nd  edition. edinburgh: churchill  livingstone, 2000.     sconul. learning outcomes and  information literacy. newport, wales:  society of college, national and  university libraries, 2004.     siegel, s., and j. castellan. nonparametric  statistics for the behavioral sciences.  2nd ed. new york: mcgraw‐hill, 1988.     snavely, l., and n. cooper. ʺthe  information literacy debate.ʺ journal  of academic librarianship  1997.january (1997): 9‐14.     university of auckland, library.  ʺinformation skills: the research  process ‐ 5 steps to success  ʺ.  auckland, 2005.  university of  auckland. 15 april 2007.  <http://www.library.auckland.ac.nz/ins truct/research.htm>     will. an action agenda for lifelong  learning for the 21st century. brussels:  world initiative for lifelong learning,  1994.     http://www.library.auckland.ac.nz/instruct/research.htm http://www.library.auckland.ac.nz/instruct/research.htm evidence based library and information practice 2007, 2:3  20 wilson, k.l., a. lizzio, and p. ramsden.  ʺthe development, validation and  application of the course experience  questionnaire.ʺ studies in higher  education 22.1 (1997): 33‐53.               evidence based library and information practice 2007, 2:3  21 appendix 1     information literacy standards (caul, 2001)     1. the information literate person recognizes the need for information and determines the  nature and extent of the information needed   2. the information literate person accesses needed information effectively and efficiently   3. the information literate person evaluates information and its sources critically and  incorporates selected information into their knowledge base and value system   4. the information literate person classifies, stores, manipulates and redrafts information  collected or generated   5. the information literate person expands, reframes or creates new knowledge by integrating  prior knowledge and new understandings individually or as a member of a group   6. the information literate person understands cultural, economic, legal, and social issues  surrounding the use of information and accesses and uses information ethically, legally and  respectfully   7. the information literate person recognizes that lifelong learning and participative citizenship  requires information literacy       australian and new zealand information literacy framework (bundy 2004 p.11)   core standards   these standards identify that the information literate person:     1. recognizes the need for information and determines the nature and extent of the information  needed   2. finds needed information effectively and efficiently   3. critically evaluates information and the information seeking process   4. manages information collected or generated   5. applies prior and new information to construct new concepts or create new understandings   6. uses information with understanding and acknowledges cultural, ethical, economic, legal,  and social issues surrounding the use of information   evidence based library and information practice 2007, 2:3  22   appendix 2   caul information skills survey questions     1   i have a system that helps me organise the information i need.   2   i keep accurate details of everything i read.   3   i use a combination of search tools including library catalogues and web search engines.   4   when i get a new idea, i work out how to explain it effectively.   5   i critically evaluate each information source i use.   6   when i make notes about the information i am reading, i include the author and title.   7   i reference websites that i have used in my assignment.   8   i evaluate information i read for criteria including accuracy and relevance.   9   i develop a system to keep track of the information i find and its sources.   10   i apply my institution’s policies regarding plagiarism.   11   in selecting information, i evaluate the quality of the information.   12   i have a system for searching for information on a subject.   13   i need to keep relearning because life is constantly changing.   14   i revise my research plan and strategy if i need to gather more information or data.   15   i present the information in a medium that suits the audience.   16   if my searching returns too much irrelevant information, i change my keywords.   17   when i consider information i have found, i state the key ideas in my own words.   18   i compare information as i’m reading with what i already know.   19   i decide how best to find the information i require for a particular task.   20   i comply with stated restrictions on the use of intellectual property.     evidence based library and information practice 2007, 2:3  23 appendix 3   university of western australia     medicine information literacy 2007       il outcomes based on caul il framework   learning reinforcement   faculty learning  outcomes by year   graduate  outcomes   1st‐year   session 1   lab   fahb   the student is able to :     • recognise the need to seek librarian help in the  information seeking process   • distinguish between the various types of material on  reading lists. (eg. books, journals, chapter in a book,  internet resource)   • find and access items on reading lists     lecturers encourage use of library  resources for assignments     pbl resources list includes a range of  resources   yr 1   demonstrate  ability to access a  range of  information  resources   use effective  communication  skills & styles     apply the  principles of  lifelong learning  continuing  education     session 2   webct   fcp       the student is able to :     • clarify the meaning of an [assignment] topic using  reference sources   • list the keywords in an [assignment] topic   • identify types of resources (e.g. books, journal  articles) likely to be useful   • construct an appropriate search strategy for  specified resources   • determine the reliability [trustworthiness] of  resources (eg. web sites, journal articles)   • demonstrate an understanding of the purpose and  search strategy included in fahb  assignment. strategy is evaluated  and lecture given to students by  librarian on performance of group  as a whole as part of assignment  feedback     vancouver style required for all  submitted material (pbl,  assignments etc)     lecturers encourage range of      evidence based library and information practice 2007, 2:3  24 coverage of different information access tools   • begin to recognise the need to locate a variety of  resources, representing a range of viewpoints   • cite sources and quotes accurately (avoid  plagiarism)   • use vancouver correctly to cite sources       resources in assignments and rural  week report     pbl have shorter reference lists and  students are encouraged to add more  1st‐year   il outcomes   learning reinforcement     faculty learning  outcomes by year   graduate  outcomes   session 3   lab   fcp   the student is able to :     • plan an appropriate search strategy   • evaluate results and revise search strategy  accordingly   • identify sources of health information including  population data, major health issues in rural areas,  health services and facilities in rural areas   • discuss the authority of information found   • cite resources correctly using vancouver     community health information  required for rural report   yr 1   demonstrate  ability to access a  range of  information  resources   use effective  communication  skills & styles     apply the  principles of  lifelong learning  continuing  education   session 4   lab   fcp   the student is able to :     • identify the special characteristics of medical  literature (structured thesaurus, health statistics)   • identifies when different resources are appropriate  (journals, books, databases, web sites)   students email articles from session  to themselves to show/discuss in  next tutorial       evidence based library and information practice 2007, 2:3  25 • recognises that the search strategy has to be  designed for the particular resource (database, web)   • gain an overview of a topic using review articles       il outcomes     learning reinforcement   faculty learning outcomes by year   4th‐year   session1   1‐on‐1  reference  interview   ebm   the student is able to :     • find papers to answer their individual question in  the following tutorial session   individual reference interviews  inform discussion in following  tutorial   yr 4   use appropriate information  management methods   session 2   webct     the student is able to:   • construct a well built clinical question   • understand the parts of a clinical question   • apply the clinical model to a patient   • identify and search the sources of synthesized  evidence‐based information;   • access evidence‐based resources to inform  decisions   • comment on the reliability and authority of  information found   • base a clinical decision on the evaluated  information found   webct backed up with a teaching  slide series that they go through  either before the question, or at  different points along the way.     question may be assessed and  compulsory     evidence based library and information practice 2007, 2:3  26 appendix 4     the seven items on which fourth‐year students rated themselves more severely than did first  year students are as follows:     1. i have a system that helps me organise the information i need.   2. when i get a new idea, i work out how to explain it effectively.   3. i critically evaluate each information source i use.   4. i evaluate information i read for criteria including accuracy and relevance.   5. i have a system for searching for information on a subject.   6. i compare information as i’m reading with what i already know.   7. i comply with stated restrictions on the use of intellectual property.     appendix 5   topics discussed in the interviews     1. formal and informal sources that are resources for finding information while working in  the hospitals as opposed to those used in the pre‐clinical years.   2. specific types of information sources e.g. textbooks, evidence‐based summaries.   3. importance of timeliness in information retrieval in the clinical setting.   4. limiting searching to particular study types e.g. randomised controlled trials.   5. critically evaluating sources of information.   6. clinicians involvement in directing students to information.       all the quantitative analysis reported was undertaken using spss version 12.   there were 84 first year students and 112 fourth‐year students.   caul australian and new zealand information literacy framework        news   call for book chapter proposals: visualizing the library: a primer on visual research methods in library and information sciences      2018. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29506     we are seeking proposals for a chapter in our upcoming book, visualizing the library: a primer on visual research methods in library and information sciences.   publisher: facet publishing editors: shailoo bedi, university of victoria, and jenaya webb, university of toronto   why contribute?   while various qualitative methods and ethnographic approaches have seen an increase in the library and information sciences (lis) literature, visual methods are only beginning to gain speed in the discipline. this book would be the first on the topic of visual research methods in the field of lis as such we invite you to be part of this groundbreaking work.   the purpose of the book is to provide a strong theoretical and methodological context for the use of visual research methods for information studies and to showcase examples of visual research methods “in action”. as our users increasingly document, share, and narrate their lives through images, we seek to harness the power of images to understand their experiences in information spaces.   the following article provides further context for the project: bedi, s., & webb, j. (2017). participant-driven photo-elicitation in library settings: a methodological discussion. library and information research, 41(125), 88-103. retrieved from http://www.lirgjournal.org.uk/lir/ojs/index.php/lir/article/view/752   book format: part 1 will be written by shailoo bedi and jenaya webb to provide an overview of visual research methods, a discussion of theoretical approaches to images, a survey of the types of visual research methods in use in lis and other disciplines, and a chapter on working with visual data. part 2 will showcase contributions from researchers and practitioners using visual research methods in a variety of contexts (e.g., galleries, libraries, archives, and museums). going beyond a step-by-step “how to” guide, this book will provide readers with practical approaches to applying visual research methods as a methodological approach while providing a grounding in research theories and the overarching theoretical foundations underpinning visual research methods. thus, readers will come away equipped to apply visual methods in their research and practice along with the ability to frame their research in theory.   we hope the methodological and theoretical discussions in the book will inspire researchers, practitioners, and students alike and help provide a rigorous and scholarly context for their work.   interested?   if you are interested in this opportunity and would like to contribute a proposal, please contact shailoo bedi (shailoo@uvic.ca) and jenaya webb (jenaya.webb@utoronto.ca).   timelines:   chapter proposals (500-800 words) are due october 31, 2018. authors will be notified in november whether their proposal has been selected for expansion to a full chapter. full chapters will be about 5000 words in length, and will be due may 31, 2019.       microsoft word news4414 evidence based library and information practice 2008, 3:4 92 evidence based library and information practice news canadian network for innovation in education international conference © 2008. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. mark your calendars for the 2nd annual canadian network for innovation in education (cnie) international conference 2009, to be held in ottawa, on, from 10 may 2009 – 13 may 2009. this exciting event will be hosted by the canadian healthcare association (cha). theme learning and technology: a capital idea! conference streams • building effective learning environments: innovations in instructional design and course development • the prescription for learning: integrating educational technology into the health system • breaking learning barriers: new technology and media applications to create accessible learning • k to 12 innovation: engaging the digital learner • transforming education: constructing a student centered world with open learning and distance education • leading the way: effective policies, partnerships and administrative practices with an expected attendance of over 400 national and international delegates working in the fields of educational technology, health education, k-12 education, multi-media design, and distance learning, the 2009 cnie international conference offers a unique opportunity for learning, networking and idea exchange. for more information, please visit the 2009 cnie international conference website <http://www.learningconference.ca/>. evidence based library and information practice editorial responsibilities      2016. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     editor-in-chief: lorie kloda   associate editor (evidence summaries): heather pretty   associate editor (articles): rebekah (becky) willson, lisl zach   associate editor (classics): jonathan eldredge   associate editor (reviews): lorie kloda   associate editor (features, commentaries, using evidence in practice, research in practice, news): lorie kloda   production editor: michelle dunaway   editorial intern: melissa griffiths   editorial advisors: lindsay alcock, alison brettle, denise koufogiannakis, katrine mallan, pam ryan   copyeditors: richard hayman (lead copyeditor), georgianne bordner, bryan chan, julie evener, heather healy, julie james, stacey penney, maria tan, mary virginia taylor, nikki tummon, alison yeoman   indexing support: meral alakus   evidence based library and information practice commentary   formalized curiosity: reflecting on the librarian practitioner-researcher   virginia wilson librarian, murray library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   received: 3 feb. 2013     accepted: 21 feb. 2013      2013 wilson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     introduction   research is formalized curiosity. it is poking and prying with a purpose.                                                                               zora neale hurston (1942)   there’s a well-documented gap between research and practice. a google search for scholarly articles using the term “research practice gap” yields 2,530 hits as of this writing, while a search using the discovery layer at the university library, university of saskatchewan, for the same search terms yields 1,038 hits. there are a large number of articles which explore bridging the research/practice gap. so what will fill that gap in librarianship? partnerships between lis scholars and librarians have been suggested, and this can certainly help to mitigate the research/practice gap. each group has things that the other group needs. practitioners often have funding barriers, a real or perceived lack of research skills, and uneven access to the research literature. scholars have less access to certain data that can only be obtained from practice situations, and a partnership with library practitioners can provide greater access to real life locations, users, and situations. as well, a partnership can help ensure that what the scholars are researching is relevant to the practitioners. however, scholar/practitioner partnerships sometimes are not practical, even in our age of social networking. in canada, for example, there is a dearth of library schools to cover our vast physical space. physical proximity can play a role in whether or not a partnership is successful. timeliness also is a factor. practitioners sometimes need to “hit the ground running" and get their research done in order to inform practice. the logistics of a partnership can be time-consuming. as well, i am estimating that there are far more library and information professionals than there are university library scholars, so it’s really up to us to fill that gap ourselves in many cases.   that is where the notion of the practitioner-researcher comes in. this is not a new concept. healthcare, education, and social work to name just three, have a history with the practitioner-researcher. the literature in these areas is filled with examples, dilemmas, problems, solutions, and illustrations of the practitioner-researcher model. and given that evidence based library and information practice (eblip) encourages practitioner research (see crumley and koufogiannakis (2002) for their practical definition of eblip), one of the next steps for eblip is to turn attention to the librarian practitioner-researcher as an encouraged and formalized role. not every information professional will conduct research, just as not every nurse, social worker, or teacher conducts research. but many will, and the rest of us will use this research in its various forms. organizational supports are needed to legitimize this role and to reinforce its necessity in library practice.   practitioner-researcher: definitions   so what is a practitioner-researcher? the simplest definition would be that it is a practitioner who conducts research. in fact, peter jarvis, who wrote the seminal book on the subject, the practitioner-researcher: developing theory from practice defines it as just that: “practitioners who do research” (jarvis, 1999 p. 3). cochran-smith and lytle define teacher research as “systematic, intentional inquiry by teachers” (1990, p. 2). shaw, who writes about the practitioner-researcher in a social work context, claims that “it is not adequate to define practitioner research simply as research carried out by practitioners without grounding it on the basis of purpose.” (2005, p. 1232).  he prefers mcleod’s definition, which states that practitioner research is “research carried out by practitioners for the purpose of advancing their own practice” (shaw, 2005, p. 1232). although mcleod is referring to practitioner research, i would say the same thing about the practitioner-researcher to a certain extent. practitioner-researchers largely conduct research to inform their own practice and to make decisions around practice issues. shaw defines practitioner involvement in research as the “evaluation, research, development, or more general inquiry that is small scale, local, grounded, and carried out by professionals who directly deliver those self-same services” (2005, 1232). in the health context, yanos and ziedonis’s definition of a clinician-researcher is “an individual who both conducts research and provides direct services” (2006, p. 249), just like librarians who are practitioner-researchers. bentz and shapiro talk about the scholarly practitioner in their book mindful inquiry in social research and define it as “someone who mediates between her professional practice and the universe of scholarly, scientific, and academic knowledge and discourse. she sees her practice as part of a larger enterprise of knowledge generation and critical reflection” (1998, p. 66).   this is my view of what a practitioner-researcher is: rather than being on the outside looking in, the practitioner-researcher is someone on the inside looking around, observing and attempting to understand what’s going on for the benefit of how things are working on the inside. it is the practitioner reflecting on practice, being curious about practice in a formalized way, and wanting to know more about practice in order to make that practice better.   jarvis (2000), via watson-boone, describes three types of practitioner-researchers:   those who undertake formal or continuing education that includes “studying one or more aspects of their practice” (p. 85). an example would be a librarian who decides to do the phd and chooses topics from her practice to delve into. those who carry out projects to inform policy decisions. an example would be librarians who conduct research to consider new approaches to library services. those who do research to satisfy their own curiosity. for example, academic librarians who continue their research after getting tenure because they are interested in the topic.   watson-boone writes a powerful statement that to me gets to the nub of the practitioner-researcher: “practitioner-researchers believe that continuous learning about their practice is fundamental to understanding and adapting themselves and their work to changing work requirements and that without such learning one cannot maintain a specialty or be an expert” (2000, p. 86). she also states that “within academic librarianship, it may be that the major difference between being a practitioner and being a practitioner-researcher is not one’s publication rate, but rather how deliberately each librarian incorporates [the steps of research] into routine work habits” (2000, p. 85). this is an important point. because of standards for tenure and promotion, librarians are required to do research and disseminate it for career advancement. with practitioner-researchers, it goes further than that. research is done to inform practice, to improve decision-making, to make sense, and to satisfy curiosity. mitchell, lunt, and shaw propose that “practitioner-researchers occupy a hybrid culture that is neither practitioner nor researcher” (2010, p. 20). i can accept the notion of a hybrid culture, but instead of saying neither practitioner nor researcher, i would suggest that we are both practitioner and researcher. the dual role can allow us to practice with much fuller knowledge of our work.   why is the practitioner-researcher necessary in librarianship?   in his book the practitioner-researcher: developing theory from practice, peter jarvis states that “practitioner-researchers are able to report aspects of practice at a depth that traditional forms of research might well not capture, precisely because they are practitioners” (24). because so much of our decision-making deals with issues of a practice nature, having a recognized body of research from the practitioner perspective would be an addition and an enhancement to the scholarly lis literature available. a professional dialogue in the research literature between lis scholars and practitioners would add robustness to the research conducted by both parties, which would serve to augment the outputs from both as well.   in healthcare, yanos and ziedonis have concluded that “patient-oriented clinician-researchers can serve as effective ‘bridgers’ between the research and practice communities and can facilitate both the development of clinically relevant research and the dissemination of evidence-based treatments into routine clinical services” (2006, p. 253). translated into lis, the librarian practitioner-researcher could perform that same bridging role between the two camps of librarians: the scholars and the practitioners. the two authors also observe that “it is often stated that the field [of medicine] would stagnate without the involvement of researchers who have direct clinical experience with the health conditions and the service systems being studied” (yanos & ziedonis 2006, p. 259). in support of that thought, mcgowan and dow claim that “no discipline can advance without a research agenda, and academic librarians are in a unique position to do research” (1995, p. 349).   the perceived legitimacy of the role   are practitioner-researchers “real” researchers? a prominent complaint about some publications in librarianship is that there are very many cases of the “how we done it good” papers: authors engage in superficial description without looking at the larger context or doing much analysis. this does not only occur in librarianship. brooker and macpherson observe in a paper focused on the educational field that they have seen “a proliferation of personal experiences and recollections of past occurrences which are being promoted under the banner of research (1999, p. 218). they go on to state that in order to be taken seriously, “practitioner researchers must have a sense of responsibility to think clearly in terms of purposes for the research, modes of research investigation, ways of documenting research strategies and outcomes, and ways of interpreting these outcomes and drawing implications for further action and investigation” brooker & macpherson (1999, p. 210). in other words, practitioner-researchers must conduct and report on “real” research, that is, a systematic investigation of a question or an issue using definable methodology and leading to a conclusion. however, it must be noted that practitioner-researchers seek “to understand, rather than control, the conditions in which practice occurs” (jarvis, 1999, p. 99). jarvis states that “the practitioner-researchers’ own practice is unique, so the findings from practice situations cannot be applied to other situations” (1999, p. 84). i would argue that, while technically jarvis may be correct, finding evidence from a practice that is similar to your own would yield benefits nonetheless.   the disciplines   the practitioner-researcher model is found in many disciplines, including nursing, social work, and education. i will speak a bit about each of these disciplines, although the review of the literature in all cases is representative rather than exhaustive.   nursing   evidence based nursing practice has placed building research capacity front and centre in the uk (deave, 2005). jarvis, in his article about practitioner-researchers in nursing, has stated that “practice has become a site for learning” (2000, p.33). due to the fast pace and transitory nature of practice, “every practice situation has become a potential research situation” (jarvis, 2000, p.32). literature about research in nursing observes that while nurses are encouraged to use the research evidence to inform their practice, the problem is that the evidence is lacking when it comes to practice situations (closs, 2000). various programs have been put in place to encourage practising nurses to do meaningful research which is based in practice, but the usual barriers of time, research skills, and management buy-in are at play here as well. the call here is for further research training, facilitation between practice and research, and more grant funds for this type of research in practice in order to build research capacity. as well, nurses need to believe that their own distinct contributions have value (wilson-barnett, 2001). in terms of role conflict, the tension between the roles of practising nurse and researcher, a study undertaken by deave around job advertisements for the research nurse position, suggests that the research nurse often works away from practice and only has contact with patients in the researcher role. this distance from caring practice runs contrary to the underpinnings of nursing, and “the researcher may be left feeling unsatisfied at being unable to help the individual” (deave, 2005, p. 653). in australia, the need has been recognized for clinical researchers. one program has three interesting aims: to support clinical research “with potential to lead to improved health outcomes”; to “foster training of clinical researchers, particularly those with a capacity for independent research”; and to “ensure effective translation of research outcomes” (brown & sorrell, 2009, 628).   social work   ian shaw asks an interesting question about the research being done by practitioner-researchers: “is practitioner research simply a street market version of mainstream research, or is it a distinct genre of research?” (2005, p.1231). this hearkens back to the perceived legitimacy of the role. is it real or is it a knock off? mccrystal has written about a study he did on a practitioner research training program for social workers in the uk, and has stated that “practitioner research does not entail any particular method or strategy of research, and is not in itself a special category of research” (2000, p. 361). he goes on to say that to be credible, social work research much be undertaken with the same rigorous standards in terms of methodology and interpretation that should be found in social science research. in his study, mccrystal found that 99% of the social workers surveyed “believed that research could be an asset to professional practice” but that only 7 respondents suggested that they themselves become actively involved in undertaking research (2000, p. 364 and 366).    education   in education, practitioner research is often termed action research, and the practitioner-researcher has been around the teaching profession for quite a few years, emerging in the uk in the 1960s (elliott, 1990, p. 1). however, there has been resistance to legitimizing practitioner research in education by the academic community (anderson & herr, 1999). school-based inquiry by teachers has been marginalized as a form of teacher development but not recognized as a form of knowledge production (zeichner, 1995, p. 153). throughout the literature, there is agreement about the satisfaction of engaging in inquiry about their own practice that is garnered by teachers. being teacher-researchers helps teachers better understand their own practice. they become resources for others, they begin to read in a critically responsive way, and they collaborate with students to answer the questions that are important to both groups (cochran-smith & lytle, 1990, p. 8).   practical issues   finding time for research   i work in academic librarianship in canada, where the standards for tenure and promotion include a research component. i was attracted to academic librarianship, as i know some of my colleagues were, because of the research piece. the opportunity to conduct research in an academic setting as well as to practise as a librarian is attractive. as members of the university faculty, librarians are required to develop a program of research in order to achieve tenure and to make our way up through the ranks. our guidelines for assignment of duties acknowledges this requirement by quantifying the time we should spend on research endeavours: 20% of our work assignment for pre-tenured librarians, and 15% of our assignment once tenure has been achieved. it can be a challenge to combine a research program with one’s daily job responsibilities. the fact that it is a requirement motivates academic librarians to follow the practitioner-researcher route. librarians in other sectors or countries will not necessarily have this motivation and their time challenges will be more daunting.   support: financially and from management   when librarians do research on an ad hoc basis, often the standardized supports are not there. financial concerns and lack of support from management can hamper one’s best intentions. it can take a while for the buy-in to occur in an organization. one way to achieve buy-in is to show the value of the research being done.   role conflict   one of the issues around being a practitioner who does research is role conflict. ethical conflicts, especially in the health field, are a large cause of confusion and role conflict. there can be tension between the roles, or as yanos and ziedonis state, “...confusion or conflict that often occurs when an individual functions in multiple roles simultaneously—termed ‘interrole conflict’ by social psychologists” (2006, p. 251).   balancing quality with utility   the research we do has to be useful. additionally, it should be of a standard that allows others to use it too. with constraints like timelines and support issues, there may be the feeling that while the research undertaken can inform our own individual and subjective practices, we might feel hesitant to disseminate it. so, there must be a focus on balancing quality with utility in order to make the best use of the research. there are methodological solutions, but they will not work in all circumstances. there could be replications of studies, synthesis of studies, and perhaps multicentre collaborations to get more generalizable results.   next steps   determine the needs of practitioner-researchers so they can be better supported. focus on effective dissemination of research findings so they are accessible and usable by practitioners. urge lis educators to incorporate more practitioner research into the curriculum to expose students to the wide variety of research and research possibilities that are available to them as practitioners.   conclusion   standing on the line between scholar and practitioner, the librarian practitioner-researcher is in a distinctive position to examine closely and to test issues of a practice nature from a unique perspective. lawrence stenhouse once said that “it is teachers who, in the end, will change the world of the school by understanding it.” (quoted in johnson, 1993). the field of librarianship must have practitioner-researchers who can participate in changing the world of the library by understanding it. the output of these researchers must be positioned in such a way in the body of lis research so that maximum benefit can be derived from this type of practical research. mitchell, lunt, and shaw state for social work that “for the impact of practitioner research studies to be maximized there should be a broad-based dissemination strategy” (2010, p.22) and that practitioner research should be “promoted as a means to stimulate researchmindedness and capacity” (2010, p. 21). the same can be said for librarianship. practical research undertaken from within the space of an intellectual discipline will provide well-rounded and robust evidence to the field. peter jarvis stated that “research is now not removed from the daily round of practice: it is being demystified and democratized. it is being undertaken, to a great extent but not exclusively, by practitioners, a trend that should grow and develop in this age of learning” (2000, p. 35). the presence of librarian practitioner-researchers is crucial if evidence based library and information practice is to move forward in a practical as well as theoretical way.     references   anderson, g. l. & herr, k. (1999). the new paradigm wars: is there room for rigorous practitioner knowledge in schools and universities? educational researcher, 28(5), 12-40. doi:10.3102/0013189x028005012   bentz, v. m. & shapiro, j. j. (1998). mindful inquiry in social research. thousand oaks, ca: sage publications.   brooker, r. & macpherson, i. (1999). communicating the processes and outcomes of practitioner research: an opportunity for self-indulgence or a serious professional responsibility? educational action research, 7(2), 207-221. doi:10.1080/09650799900200088   brown, g. v. & sorrell, t.c. (2009). building quality in health—the need for clinical researchers. the medical journal of australia, 190(11), 627-629.   closs, j. (2000). research for nursing: whose job is it anyway? nurse education today 20(6) 423-425. doi:10.1054/nedt.2000.0500   cochran-smith, m. & lytle, s.l. (1990). research on teaching and teacher research: the issues that divide. educational researcher, 19(2), 2-11. doi:10.3102/0013189x019002002   crumley, e. & koufogiannakis, d. (2002). developing evidence-based librarianship: practical steps for implementation. health and information libraries journal, 19(2), 61-70. doi:10.1046/j.1471-1842.2002.00372.x   deave, t. (2005). research nurse or nurse researcher: how much value is placed on research undertaken by nurses? journal of research in nursing, 10(6), 649-657. doi:10.1177/174498710501000608   elliott, j. (1990). teachers as researchers: implications for supervision and for teacher education. teaching & teacher education, 6(1), 1-26. doi:10.1016/0742-051x(90)90004-o   jarvis, p. (1999). the practitioner-researcher: developing theory from practice. san francisco, ca: jossey-bass.   jarvis, p. (2000). the practitioner-researcher in nursing. nurse education today, 20(1), 30-35. doi:10.1054/nedt.2000.0428   johnson, b. (1993). teacher-as-researcher. eric clearinghouse on teacher education. accessed feb 25, 2013. http://www.ericdigests.org/1993/researcher.htm   mccrystal, p. (2000). developing the social work researcher through a practitioner research training program. social work education, 19(4), 359-373. doi: 10.1080/02615470050078366   mcgowan, j.j. & dow,e.h. (1995). faculty status and academic librarianship: transformation to a clinical model. the journal of academic librarianship, 21(5), p. 345-350. doi: 10.1016/0099-1333(95)90059-4   mitchell, f., lunt, n., & shaw, i. (2010). practitioner research in social work: a knowledge review. evidence & policy, 6(1), 7-31.   research_is_formalized_curiosity_it_is_poking_and. (n.d.). columbia world of quotations. retrieved march 04, 2013, from dictionary.com website: http://quotes.dictionary.com/research_is_formalized_curiosity_it_is_poking_and   shaw, i. (2005). practitioner research: evidence or critique? british journal of social work, 35(8), 1231-1248. doi:10.1093/bjsw/bch223   watson-boone, r. (2000). academic librarians as practitioner-researchers. the journal of academic librarianship, 26(2), 85-93. doi:10.1016/s0099-1333(99)00144-5   wilson-barnett, j. (2001). research capacity in nursing. international journal of nursing studies, 38, 241-242.   yanos, p. t. & ziedonis, d.m. (2006). the patient-oriented clinician-researcher: advantages and challenges of being a double agent. psychiatric services 57(2): 249–253. doi: 10.1176/appi.ps.57.2.249   zeichner, k. m. (1995). beyond the divide of teacher research and academic research. teachers and teaching: theory and practice, 1(2), 153-172. doi: 10.1080/1354060950010202   microsoft word comm_bothmer.doc evidence based library and information practice 2007, 2:3  117 evidence based library and information practice     commentary    digital archiving of primary research data      a. james bothmer, mals, ahip  assistant professor and director   health sciences library/learning resources center  creighton university  2500 california plaza  omaha, ne  68178  e‐mail: jbothmer@creighton.edu     robert p. heaney, md  john a. creighton university professor and professor of medicine  creighton university school of medicine  2500 california plaza  omaha, ne  68178  e‐mail: rheaney@creighton.edu    ramon m. fusaro, md, phd  adjunct professor of preventive medicine and public health  school of medicine and adjunct professor of pharmaceutic sciences  school of pharmacy and health professions  creighton university  2500 california plaza  omaha, ne  68178  and  professor of internal medicine (dermatology)  college of medicine  university of nebraska medical center  984360 nebraska medical center  omaha, ne  68198‐4360  e‐mail: rfusaro@unmc.edu    received: 09 july 2007    accepted:  10 july 2007      © 2007 bothmer, heaney, and fusaro. this is an open access article distributed under the terms of the  creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits  mailto:jbothmer@creighton.edu mailto:rheaney@creighton.edu mailto:rfusaro@unmc.edu http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:3  118 unrestricted use, distribution, and reproduction in any medium, provided the original work is properly  cited.      only a relatively small portion of the data  generated from research projects performed  at universities ever sees the light of day.   final successful experiments are usually the  only data that are published.  for economic  reasons, journals publish minimal details of  a study’s methodology needed for others to  replicate the findings. research is a cultural  and scientific treasure of intellectual effort;  somehow, therefore, universities should  preserve all raw data from research  conducted within their domains, successful,  unsuccessful, published, and unpublished.   academic health sciences libraries need to  take the lead in this effort by creating and  linking institutional repositories (ir) to  form a true national database of all  intellectual experimentation.  these efforts  can be implemented without changing the  status quo of traditional publishing and they  could be introduced by reallocating existing  resources without extensive technical  development.  these publications would  still need internal universities’ faculty peer  review to assure the integrity and quality of  intellectual accomplishments.    digital technology has affected how scholars  disseminate and preserve their research.  it  is now economically possible to archive  information with digital technology making  it possible for libraries to catalogue it for  greater access by the scholarly community  (lynch).  the current technology also makes  it possible to include appropriate  restrictions for prepublication concealment,  patentability and patient confidentiality.   institutional repositories were devised as a  solution for preserving and making  accessible the scholarly output of an  institution’s researchers (lynch; harnad;  crow).  institutional repositories are  defined here “as a formally organized,  digital collection of the intellectual output of  an academic campus community and comes  directly from the faculty, staff, and students  of the university” (singarella 2).  there are  essentially two schools of thought regarding  an ir.  one articulated by lynch argues that  an ir serves to disseminate ‘grey literature’  including such documents as pamphlets,  bulletins, visual conference presentations,  and other materials that are typically not  included in traditional publications (lynch).   the alternate view expressed as early as  1995 indicated that an ir could effectively  take the place of traditional publishing, or at  least be a competitive entity (harnad).      the purpose of this opinion piece is to argue  that a national network of irs could be used  to archive and preserve published and  unpublished raw data in institutional  resources.  access to raw data could prevent  unnecessary replications of investigations,  provide data for university internal review  of faculties’ research, and permit access by  other scientists to details of all research data  that could be highly useful in light of newer  insights and concepts.  the authors believe  it is a regular occurrence for researchers to  wish they had access to unpublished raw  data in a given publication, either to analyze  it in a different way or to utilize  measurements made in a given study but  not published because they did not seem to  bear on the primary study’s outcomes.  the  availability of raw research data of reported  clinical studies would increase the  plausibility for either acceptance or rejection  of previous proposed findings in evidence  based medicine studies.      in order to make a case for harvesting  research data (published or unpublished),  we offer two examples that illustrate this  need: (a) the need for access to the actual  measured values if one is to reanalyze the  evidence based library and information practice 2007, 2:3  119 experiments of other investigators using  different models and/or testing different  hypotheses; and (b) the need for  publications to contain full methodological  details, the disclosure of which would help  explain or contradict published findings.  without such access by other investigators,  erroneous conclusions may be enshrined in  the literature effectively forever. in the first  example, below, the error in an earlier  publication would likely ultimately have  been found out, but only at the cost of  unnecessary duplication of work already  done, but misinterpreted.  in the second  example, a promising treatment might have  been lost forever because of the inadvertent  use, in this instance, of deteriorated  chemical reagents, unrecognized as such by  the investigators.     a pertinent archival retrieval experience of  the first example comes from a recent in  print publication describing the relationship  of serum vitamin d concentration and that  of its principal metabolite 25‐hydroxy‐ vitamin d [25(oh)d] (hollis et al.,  “circulating”). the authors had used a type  of curve fitting that led them to conclude  that the concentration of 25(oh)d would  reach a maximum at a certain serum vitamin  d level and rise no further. one of us [rph]  had generated a similar set of measurements  and wished to subject both sets to a different  kind of curve fitting.  the raw data were  obtained by personal contact with the  original investigator, and when combined,  the two datasets constituted essentially the  totality of the world experience and hence  had unique value for unraveling important  questions in a rapidly expanding field of  investigation. as it turned out, the data  actually showed that serum 25(oh)d rose  without limit as serum vitamin d rose, a  finding at variance with the original  authors’ analysis.  even more significant, the  combined set allowed further and crucial  insights into vitamin d metabolism that had  not been apparent within the individual  datasets (hollis et al., “25‐hydroxylation”).     the second example is a complex  investigation lasting over two decades.  the  investigation concerns the use of sequential  bedtime skin applications of two separately  stored chemicals (dihydroxyacetone {dha}  followed by lawsone) that produce a  melanoidins‐sunscreen in the top keratin  layer of the skin that lasted for 1‐2 weeks.  (fusaro).  this skin‐bound sunscreen gives  sunlight protection for both normal persons  and photosensitive persons who are either  allergic to or intolerant of ultraviolet (uv‐a  or uv‐b) wave lengths of sunlight without  loss from perspiration.  a clinical trial (rice)  of the above medication procedure reported  complete sunlight protection of thirty  photosensitive patients without any failures  of protection.  one of us (rmf) contacted  the author of this study in order to obtain  access to unpublished data.  these data  showed that the majority of the patients  tested the limits of their sunlight tolerance  and were also protected against uv‐b  sunburn for 6‐8 hours a day (spf 18‐24) over  a 7‐month period (fusaro and rice,  “maillard reaction”).  in the study by rice  (rice), the author tested the use of  previously stored mixtures of dha/lawsone.   the same photosensitive patients received  no significant sunlight protection as the  compounds had degenerated and use of the  mixture in patients was abandoned.  a  previously classified u.s. army study  (fitzpatrick and pathak) reported using two  different methods for skin applications of  stored mixtures of the same reagents and  different sunburn testing procedures in two  groups of the four volunteers that failed to  provide significant sunburn protection  compared to a paba sunscreen (fitzpatrick  and pathak).  however, the authors later  reported (pathak, fitzpatrick and frenk) the  four volunteers as one group in spite of the  differences in topical applications of the  dha/lawsone mixture and two different  evidence based library and information practice 2007, 2:3  120 sunburn testing procedure. two u.s.  senators were successful in obtaining the  raw data from this classified study  (fitzpatrick and pathak).  the variable  inadequate protective results of this u.s.  army clinical trial of four individuals gave  concern that the dha/lawsone mixture had  chemically deteriorated while in storage.   this deterioration was noted over a decade  later by one of the u.s. army’s original  investigators for the first time in a single  paragraph about the use of dha/lawsone  mixture as part of a complete review of all  available sunscreens (pathak).  the review’s  author noted that the deficiency of the  stored mixture of the two compounds in  their earlier study had changed color prior  to its skin application and was washed off of  the test subjects when they later bathed in  water (fitzpatrick and pathak).  this  confirmed the chemical deterioration of the  mixture and the failure to produce a skin‐ bound, melanoidins‐sunscreen with high  spf sunburn protection in the u.s. army  study.       from personal contacts with the vitamin d  investigators involved in the first example  and a relentless in‐depth search over two  decades of published and unpublished data  of the long‐lasting sunscreen in the second,  the two archival investigations uncovered  the raw data needed to produce new  conclusions.     huge barriers exist to the creation of a  nationally linked network of institutional  repositories.  for example, the centers for  disease control and prevention (cdc) have  created a large number of datasets from  research.  until recently these huge and  valuable datasets have not been easily  identifiable.  metadata was developed for  these cdc datasets creating a system that  allows researchers to search over 95% of the  databases within cdc.  unfortunately, the  metadata and system is available only to  cdc researchers thus marginalizing its  usefulness (matters et al).  another barrier  to populating an ir with unpublished  research data includes resistance and  reluctance of some researchers to contribute  their raw data.  davis has written that each  discipline has a normative culture, largely  defined by their reward system and  traditions.  if the goal of an ir is to capture  and preserve the scholarship of oneʹs faculty,  this institutional cultural diversity will need  to be addressed.      one strategy to overcome a reluctance to  deposit research in an ir is to implement  embargoed access restrictions (i.e.,  prepublication concealment, etc.), which  will help ensure acceptance and compliance  by research investigators, as it will protect  their creativity.  overcoming these barriers  will take time as well as open dialogue and  collaboration among researchers, librarians  and it staff.    after reviewing issues of journals such as  the journal of clinical investigation from the  1930s, 1940s, and 1950s, it appears to the  authors that it was historically more of a  common practice to publish raw data as part  of an article. there has been a shift in the  past 50 years or so from publicly accessible  research data to no access unless by  personal contact; moreover, most research  data are unfortunately discarded either by  authors or universities, and then even  personal contact is unavailing.     the controversial circumstances of the  protective effectiveness of the melanoidins‐ sunscreen concept resulted in a grand  rounds lecture on a university’s website  that discussed the issue in‐depth. this  grand rounds presentation is an open  access publication, which can be reviewed  by anyone through the internet (fusaro and  rice, presentation). this type of university  website publishing fills a niche for unlimited  in‐depth, detailed communications of all  university research or controversial concepts  evidence based library and information practice 2007, 2:3  121 and differs from the few new journals that  publish only negative results (begley).     given the explosive growth of research data  generated in the past 50 years, it would  seem important that universities invest  money, time and talent in digital archiving  and indexing of all raw research data  produced within their scholarly  communities.  although the authors  recognize the establishment of an  institutional repository could require the  institution to incur ongoing financial  burdens for staffing, equipment, and  preservation, much of this effort could result  in a savings by centralizing functions  (gibbons).  what is called for is for  universities across the country to focus on  creating institutional repositories linked  through a common metadata and search  engine (e.g. google scholar).  this could be  the solution for providing local controlled  access to all details of published and  unpublished experiments generated in their  academic institutions.  as singarella points  out, the ir can “build on a growing  grassroots faculty practice of self‐posting  research online” (singarella 20).   libraries  are the logical administrative entity to carry  out this function.  indeed, it is expertise in  material submission, metadata application,  access control, discovery, distribution, and  preservation that library staff can offer.   only the library can claim expertise in all of  these core functions (gibbons).      our examples of new conclusions obtained  from reviewing the archival data of  published reports illustrate the value of  digital archiving of all raw research  performed at universities; moreover, this  new information needs to be disseminated  without any delay. if new conclusions are in  conflict with accepted dogma and rejected  for publication in standard journals, the use  of a network of university institutional  repositories offers a global alternative  medium to stimulate further research, and  to promote acceptance of new findings by  others (fusaro and rice, presentation).   universities exist not only to educate  students but also to create and disseminate  knowledge for the betterment of humanity.  if that knowledge resource is not accessible,  if no one can find it or if no one knows  where it is or that it even exists, of what use  is it?       works cited    begley, s. ʺnew journals bet ʹnegative  resultsʹ save time and money.ʺ wall  street journal 16 sept 2006 : b1.    crow, r. ʺthe case for institutional  repositories: a sparc position  paper.ʺ arl bimonthly report aug.  223 (2002). 23 mar. 2007.  <http://www.arl.org/resources/pubs/br /2002br.shtml.>.    davis, p. ʺinstitutional repositories:   evaluating reasons for non‐use of  cornell universityʹs installation of  dspace.ʺ d‐lib magazine mar/apr  13.3/4 (2007). 23 mar. 2007.  <http://www.dlib.org/dlib/march07/da vis/03davis.html>.    fitzpatrick, t.b., and m.a. pathak.   “chemical binding of protective  agents to human stratum corneum.”   contract #da49—092‐aro‐85.   arlington, va:  department of united  states army, army research office,  1968.    fusaro, r. m. patent 3,920,808. 18 nov. 1975.   fusaro, r. m., and e. g. rice. ʺthe  maillard reaction for sunlight  protection.ʺ annals of the new york  academy of sciences 1043 (2005): 174‐ 83.     http://www.arl.org/resources/pubs/br/2002br.shtml http://www.arl.org/resources/pubs/br/2002br.shtml http://www.dlib.org/dlib/march07/davis/03davis.html http://www.dlib.org/dlib/march07/davis/03davis.html evidence based library and information practice 2007, 2:3  122 ‐‐‐.  presentation.  “keratin bound  sunscreens.”  2004.  27 sept. 2006.  <http://newadonis.creighton.edu/hsl/ grandrounds/grandrounds‐2004‐ october.html>.     gibbons, susan. ʺestablishing an   institutional repository.ʺ library  technology report 40.4 (2004): 1‐67.     harnad, s.  “overture:  the subversive  proposal.”  scholarly journals at the  crossroads:  a subversive proposal for  electronic publishing.  an internet  discussion about scientific and  scholarly journals and their future.   eds. a.s. okerson and j.j. o’donnell.  washington, dc:  association of  research libraries, 1994.  27 mar. 2007  <http://www.arl.org/sc/subversive/i‐ overture‐the‐subversive‐ proposal.shtml>.    hollis, b. w., et al. ʺ25‐hydroxylation of  vitamin d3:  relationship to  circulating vitamin d3.ʺ journal of  clinical endocrinology and  metabolism  in press.    hollis, b. w., et al. ʺcirculating vitamin d3  and 25‐hydroxyvitamin d in humans:  an important tool to define adequate  nutritional vitamin d status.ʺ journal  of steroid biochemistry and molecular  biology 103.3‐5 (2007): 631‐4.     lynch, c. a. ʺinstitutional repositories:  essential infrastructure for scholarship  in the digital age.ʺ arl bimonthly  report feb 226 (2003). 23 mar. 2007.  <http://www.arl.org/resources/pubs/br /br226/br226ir.shtml>.    matters, m. d., et al. ʺdeveloping metadata  to organize public health dataets.ʺ  amia annual symposium  proceedings 1047 (2005).  cd‐rom.    pathak, m. a., t. b. fitzpatrick, and e. frenk.  ʺevaluation of topical agents that  prevent sunburn‐‐superiority of para‐ aminobenzoic acid and its ester in  ethyl alcohol.ʺ new england journal  of medicine 280.26 (1969): 1459‐63.     pathak, m. a. ʺsunscreens: topical and  systemic approaches for protection of  human skin against harmful effects  of solar radiation.ʺ journal of the  american academy of dermatology  7.3 (1982): 285‐312.     rice, e. g. ʺdihydroxyacetone  naphthoquinone protection against  photosensitivity.ʺ dermatologica 153.1  (1976): 38‐43.     singarella, t.  online report.  “an  overview of institutional  repositories.”  association of  academic health sciences libraries  charting the future committee.  mar.  4 (2005).  23 mar. 2007.      http://newadonis.creighton.edu/hsl/grandrounds/grandrounds%e2%80%902004%e2%80%90october.html%00%00 http://newadonis.creighton.edu/hsl/grandrounds/grandrounds%e2%80%902004%e2%80%90october.html%00%00 http://newadonis.creighton.edu/hsl/grandrounds/grandrounds%e2%80%902004%e2%80%90october.html%00%00 http://www.arl.org/sc/subversive/i%e2%80%90overture%e2%80%90the%e2%80%90subversive%e2%80%90proposal.shtml%00m http://www.arl.org/sc/subversive/i%e2%80%90overture%e2%80%90the%e2%80%90subversive%e2%80%90proposal.shtml%00m http://www.arl.org/sc/subversive/i%e2%80%90overture%e2%80%90the%e2%80%90subversive%e2%80%90proposal.shtml%00m http://www.arl.org/sc/subversive/i%e2%80%90overture%e2%80%90the%e2%80%90subversive%e2%80%90proposal.shtml%00m http://www.arl.org/sc/subversive/i%e2%80%90overture%e2%80%90the%e2%80%90subversive%e2%80%90proposal.shtml%00m http://www.arl.org/resources/pubs/br/br226/br226ir.shtml http://www.arl.org/resources/pubs/br/br226/br226ir.shtml evidence summary   undergraduate students can provide satisfactory chat reference service in an academic library   a review of: keyes, k., & dworak, e. (2017). staffing chat reference with undergraduate student assistants at an academic library: a standards-based assessment. journal of academic librarianship, 43(6), 469–478. https://doi.org/10.1016/j.acalib.2017.09.001   reviewed by: heather macdonald health and biosciences librarian macodrum library carleton university ottawa, ontario, canada email: heather.macdonald@carleton.ca   received: 25 feb. 2018   accepted: 24 apr. 2018      2018 macdonald. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29414       abstract   objective – to determine whether undergraduate students can provide quality chat reference service.   design – content analysis of undergraduate student, professional librarian, and paraprofessional staff responses in chat reference transcripts.   setting – academic library.   subjects – 451 chat reference transcripts.   methods – chat reference transcripts from may 2014–september 2016 were collected. five categories of answerer were coded: librarian in the reference department (libr), librarian from another department (libnr), staff without a master of library science (staff), staff with a master of library science (+staff), and student employee (student). a random sample of 15% of each category of answerer was selected for analysis. the answerer categories were collapsed to librarians, staff, and students for the results section.    four criteria were used to code chat reference transcripts: difficulty of query, answerer behaviour, problems with transcript answer, and comments from coders. coding for difficulty was based on the read scale (reference effort assessment data). answerer behaviour was based on the rusa guidelines (reference and user services association). behaviours assessed included: clarity, courtesy, grammar, greeting, instruction, referral, searching, sign off, sources, and whether patrons were asked if their question was answered. all coding was done independently between the two researchers, with very good interrater reliability. data for variables with disagreement were removed from the analysis. the chi-square test was used to analyze the association between variables. analysis also included patrons’ ratings and comments about their chat experience. content and tone were assessed for each patron comment.   main results – answerer behaviours showed a significant difference between groups for 3 of the 10 behaviours assessed: courtesy (p=0.031), grammar (p=0.001), and sources (0.041). the difference between groups for courtesy was: staff (88%), librarians (76%), and students (73%). grammar was correct in most transcripts, but there was a significant difference between the answerer groups: librarians (98%), staff (90%), and students (73%). there was a significant difference between groups that offered sources: librarians (63.8%), staff (62.5%), and students (43.8%).    there was no significant difference between the answerer groups for the other seven behaviours. overall, 31% of transcripts showed that answerers asked if a patron’s query was answered or if they needed further help. the analysis showed that 79% of transcripts were coded as clear or free of jargon. greetings were found in 65% of transcripts. instruction was indicated in 59% of transcripts. referrals were offered in 27% of all transcripts. of the transcripts where searching was deemed necessary, 82% showed evidence of searching. a sign off was present in 56% of all transcripts. transcripts with noted problems were deemed so because of lack of effort, being incomplete or incorrect, having no reference interview, or the answerer should have asked for help. there was no significant difference between answerer groups with respect to problem questions.   of the 24% of patrons who rated their chat experience, 90% rated it as good or great, and no significant difference was found between answerer groups. question difficulty was coded 50% at level 0-2 (easier), 39% at level 3 (medium difficulty), and 11% at level 4-5 (more difficult).   conclusion – undergraduate students are capable of providing chat reference that is similar in quality to that of librarians and staff. however, increased training is needed for students in the areas of referrals, providing sources, and signing off. students do better than librarians and staff with greetings and are more courteous than librarians. there is room for improvement for staff and librarians offering chat services. tiered chat reference service using undergraduates is a viable option.   commentary – bravender, lyon and molaro’s 2011 paper asked “should chat reference be staffed by librarians?” they found that non-librarian staff provided a cost effective alternative to librarians. keyes and dworak take this one-step further and ask whether undergraduate students can provide quality chat reference service. the authors provide extensive background information on reference services in academic libraries, tiered reference and chat service models with undergraduate students, and assessing chat reference quality.    this commentary uses the cat critical appraisal tool (perryman & rathbun-grubb, 2014) to guide the appraisal. the authors clearly state their objective and explain the impetus for this study. after their library changed chat reference platforms, a decision was made in july 2014 to include access services staff as answerers. in the fall of 2015, undergraduate students were added to the roster. the data collected was from may 2014 through september 2016. it may have made more sense to present the data starting from the date all answerer groups were involved in the chat service, especially as this paper focused on the student group.   the results were presented in several tables with clear explanations. for the most part the presentation of the results was logical and easy to follow. however, there were a few inconsistencies. the abstract stated that 451 transcripts were analyzed, but the results talked about 454 (minus 68) transcripts. the layout of table 6 was unclear, as the first line in the table appeared to be a header when in fact it was actually data similar to the rest of the table. for the behavioural variable “searching,” the data in table 6 was not consistent with the results section. the p-value in table 6 was 0.040, but in the results section the p-value was 0.099. this is problematic, as the significance threshold used was p<0.05. the numbers for the behavioural variable “sources” were different in table 6 and the results section as well.   although there were some issues with tables matching the results, the methodology was solid and well executed. this methodology can be used by libraries to analyze their own chat reference services to identify strengths and weaknesses and improve training. the authors’ conclusions were grounded in the evidence presented. this study provides evidence that undergraduate students are capable of providing chat reference services. administrators and reference department managers may want to consider the benefits of students working in roles more traditionally filled by librarians. meanwhile, reference librarians will want to consider how their roles continue to change and what this means for the profession.   references           perryman, c. & rathbun-grubb, s. (2014). the cat: a generic critical appraisal tool. in jotform – form builder. retrieved from http://www.jotform.us/cp1757/thecat   bravender, p., lyon, c., & molaro, a. (2011). should chat reference be staffed by librarians? an assessment of chat reference at an academic library using libstats. internet reference services quarterly, 16(3), 111-127. https://doi.org/10.1080/10875301.2011.595255     article   one-shot or embedded? assessing different delivery timing for information resources relevant to assignments   amy van epps associate professor of library science engineering librarian, coordinator of instruction purdue university libraries west lafayette, indiana, united states email: vanepa@purdue.edu   megan sapp nelson associate professor of library science engineering librarian purdue university libraries west lafayette, indiana, united states email: mrsapp@purdue.edu   received: 9 aug. 2012     accepted: 7 jan. 2013      2013 van epps and nelson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – this study aims to determine if the timing of library in-class presentations makes a difference in the type and quality of resources students use for each of four assignments in an introductory speech class. this comparison of content delivery timing contrasts a single, 50-minute lecture early in the semester with four approximately 12-minute lectures offered just before each assignment.   methods – first-year engineering students taking fundamentals of speech communication provide the study group. each speech assignment requires students to turn in an outline and list of references. the list of references for each student was given to the librarians, after the assignments were appropriately anonymized, for analysis of resource type, quality of resource, and completeness of citation. researchers coded a random sample of bibliographies from the assignments using a framework to identify resource type (book, periodical, web, facts & figures, unknown) and quality, based on intended audience and purpose (scholarly, entertainment, persuasion/bias), and compared them to each other to determine if a difference is evident. the authors coordinated what material would be presented to the students to minimize variation between the sections.   results – the study found a statistically significant difference between groups of students, demonstrating that the frequent, short library instruction sessions produce an increased use of high-quality content. similarly, the sections with multiple library interactions show more use of periodicals than websites, while completeness of references is not significantly different across teaching methods.   conclusions – more frequent and timely interaction between students and library instruction increases the quality of sources used and the completeness of the citations written. while researchers found statistically significant differences, the use of a citation coding framework developed for specific engineering research and design tasks means the analysis done in this study is not as accurate as it might be with a framework designed for analyzing the resources required for researching and writing speech assignments.     introduction   this paper evaluates student references included in assignments when a single presentation (“one-shot”) and embedded instruction techniques are used, and contributes to the ongoing conversation among instruction librarians regarding which method is most effective. as awareness of the skills needed by students that are encompassed in information literacy grows, requests for librarians to participate in classes also grows, and finding ways to most effectively teach the content so it does not need to be repeated in later years is critical. purdue university is working toward a more embedded approach for information literacy whenever possible. nearly all incoming freshmen at purdue are required to take the fundamentals of speech communication course. demonstrating and implementing more effective teaching techniques for this course will impact a large majority of freshmen students across disciplines. having some empirical evidence to support the benefits of this model facilitates the conversation with faculty, (particularly engineering faculty) who appreciate data-driven decision making.    literature review   one-shot library sessions are generally considered to be less impactful than other instruction presentation styles (badke, 2009; hollister & coe, 2003). orr, appleton, and wallin (2001) make a clear argument for moving away from the “one-shot” instruction model:   it has became [sic] clear that the “one-off,” demonstration-style information skills classes delivered out of curriculum context do not necessarily coincide with the students’ need for information, are sometimes not valued by the students, and do not necessarily prepare them for the challenges of research, problem solving and continuous learning. where possible, librarians prefer to use an across-the-curriculum model that incorporates the process of seeking, evaluating, and using information into the curriculum and consequently, into all students’ experiences. (p. 457)   one-shot instruction sessions have been tested for impact upon student work with varying outcomes (byerly, downey, & ramin, 2006; fain, 2011; martin, 2008). generally, the increased integration of content into the curriculum leads to more positive student outcomes (jacobs & jacobs, 2009; stec, 2006).   the integration of information literacy into the curriculum presents the most opportunity for successful knowledge transfer of information literacy, as well as the highest barrier to entry for librarians (bean & thomas, 2010; brendle-moczuk, 2006; hall, 2008; hollister & coe, 2003; jacobs & jacobs, 2009; weaver & pier, 2010).  integration into the curriculum has benefits both for acquired skills for the students as well as for exposure and comfort with the librarian/instructor (bean & thomas, 2010; gandhi, 2005; weaver & pier, 2010). project information literacy research has determined that a major need for undergraduate researchers is to have context for the learning objectives. providing instruction in the context of an assignment fills a crucial need for undergraduates (head & eisenberg, 2009a). communication courses, by virtue of the secondary research required to prepare basic speeches, are particularly good venues for curriculum-embedded information literacy (hall, 2008; weaver & pier, 2010). creating speeches on a variety of topics should allow students to explore a variety of resources. however, as head and eisenberg have found, “most respondents, whether enrolled in a twoor four-year institution, almost always turned to a small set of information resources, no matter which research context they were trying to satisfy” (2009b, p. 32).   the variety of assignments encourages expanding the freshman students’ information toolkit, thereby increasing available tools for future assignments. freshman engineers generally are unskilled in the practice of information literacy skills, as shown by the predominance of websites in freshman bibliographies (yu, sullivan, & woodall, 2006). yu et al. (2006) emphasized “finding, interpreting, and citing books, journal articles, and web sites” (p. 21) as the primary skills that are necessary for freshman engineers. hsieh & knight (2008) concluded that the traditional lecture is ineffective for teaching freshman and sophomore engineers. the information literacy skills needed by first-year engineering students are generally part of an introduction to design. bursic and atman (1997) investigated the differences in information-gathering skills between seniors working on a design project and those just beginning to learn design. the designs from the first-year students are less complete and lack the contextual awareness and understanding of usefulness and applicability of designs that develop as a result of information gathering.   this study investigates the performance of first-year engineering students during an introduction to a communications course when exposed to two different modes of presentation, a just-in-time model and a one-shot model.  the literature indicates that the just-in-time model of instruction is likely to be more effective at building information literacy skills among the students (hall, 2008; martin, 2008; weaver & pier, 2010). using a citation analysis model developed specifically to examine bibliographies and outline deliverables of engineering undergraduate students (wertz, ross, fosmire, cardella, & purzer, 2011), this article seeks to demonstrate that the mode of instruction results in an increased information literacy of a students in a class and expands on a work-in-progress conference paper (van epps & sapp nelson, 2012).   aims   research question   is there a noticeable difference in the quality, type of resource, and completeness of the references in student assignments when “just-in-time” instruction is used as opposed to a “one-shot” session? the researchers’ hypotheses are that the sections which received the just-in-time instruction will have more references and better citations, in quality, type of resource, and completeness, than the section which received the one-shot session at the start of the semester. all three of the unique questions embedded in the research question as stated will be tested and reported.   methods   setting/courses   researchers studied a group of first-year engineering students enrolled in three sections of com 114, fundamentals of speech communication, a course that focuses on oral communication skills for students in all disciplines. several sections of the class are associated with learning communities (student access transition & success, 2011a, 2011b), and as a result have only engineering students enrolled. in preparation for assignments in com 114, two different course instructors contacted engineering librarians to have them present library resources to assist students with the information gathering portion of the four speech assignments to be completed during the semester. two sections received information in four 12-minute, integrated information literacy instruction sessions (otherwise known as “just-in-time”), prior to the assignment that the instruction was intended to support. one section was given a traditional “one-shot” instruction session of 50 minutes during the second week of the semester, before any of the assignments had been given. all of the students received an equivalent duration of library instruction, just divided differently. instruction librarians used the same materials and supporting libguide for all sessions offered. the libguide (http://guides.lib.purdue.edu/com114engr) uses four tabs, one for each assignment. during the one-shot session, all four tabs were addressed during the 50 minutes, while during the mini-lectures, the librarian presented a single tab in each session. the libguide and accompanying instruction provides guidance for the students in selecting from a variety of sources appropriate within the context of the assignment. the library instruction focused on the best resources for the types of speeches the students would be giving, in support of the course objective of being able to “use supporting material properly and effectively” when making a presentation (http://www.cla.purdue.edu/communication/documents/com114_syllabus2011.pdf). all com 114 classes are taught in traditional lecture-style classrooms with a computer and projector available in the front of the room. in all cases, librarians used a demonstration/lecture-style of material presentation.   description of assignments   table 1 presents an overview of each of the four assignments, including the focus of the speech, expected deliverables, and an indication of whether the assignment is for individual or group submission.       table 1 expected deliverables for com 114 engineering living learning community students assignment 1 informative speech – engineering innovation outline & bibliography individual submission assignment 2 informative speech – process speech outline & bibliography individual submission assignment 3 persuasive speech – charitable donation outline & bibliography individual  submission assignment 4 group presentation – description of an engineering innovation outline & bibliography group submission (3-4 individuals)     see appendix a for the complete assignment descriptions.                 sample   the population consists of all students enrolled in three engineering learning community sections of com 114 included in this study (n=75). the data consists of the student deliverables (outlines and bibliographies) for all individual and group assignments in these sections. the full data set for four assignments in the three sections provided a total of 234 outlines and bibliographies. equal sample sizes were used to represent the just-in-time and one-shot sections. this was done to avoid skewed data which may have resulted from having two sections of the class receiving just-in-time (jit)/embedded teaching (n=51) and only one receiving one-shot instruction (n=24). the sample analyzed consisted of five papers for each individual assignment per teaching team and three of the group papers from each team. researchers randomly selected papers from the set of possible papers for each teaching team, and used two methods to randomly select assignments to review, based on how the data was delivered to the librarians. the assignments from the mini-lectures classes were numbered sequentially and a random number generation website was used to identify which assignments would be analyzed. for the one-shot section assignments, copies were printed and researchers randomly selected the correct number of assignments from the pile.   data analysis procedure   after removing any identifying information, instructors sent the student assignments to the librarians. the librarians then coded the references in each bibliography for type of information resource used, quality of the resource based on its scholarly content and lack of bias, and the completeness of the reference included. the coding framework is a modification of that used by wertz et al. (2011) and can be found in appendix b. librarians then compared the quality of resources used, the completeness of citations, and the types of resources used for the particular assignment across the sections for each instructional team. a simple z-test for comparison of difference between proportions was then used for each rating given to the references.   while it was impossible to control for the instructor/librarian teaching style variations and differences inherent from having different students in each class, librarians coordinated the content presented and used the same libguide to ensure all students shared a common resource to return to for guidance as the semester progressed. in this way researchers controlled as many variables as possible to control easily. though they did not use a set script for delivery of their respective presentations, the two librarians involved have similar teaching styles.   one difference between the sections is due to multiple librarian visits that provide an opportunity for a quick follow-up conducted as a guided conversation of not more than three minutes. this provided the students a chance to reflect upon which tools they used in the previous assignment, how successful they felt they were with the tools, and why those tools were appropriate for the previous assignment. however, this discussion did not impact upon the upcoming assignment, as each assignment required the use of different resources. the discussion did establish that some features of databases (i.e., boolean logic and operators, limiters, and faceted searching) reappear across tools.   inter-rater reliability   researchers used a simple percent-agreement figure to calculate the consensus estimate of inter-rater reliability (irr). this calculation involved taking the number of items coded identically by different raters and dividing by the total number of items rated (stemler, 2004). both raters analyzed an initial sample of 8 items from the original 234 items, representing one of each assignment for each instructional method, using the framework developed by wertz et al. (2011). each citation is rated for type of material, quality of resource based on both audience and treatment (bias), and completeness of the citation, creating four ratings for each citation. after rating the initial eight items, the two librarians met, checked how their use of the framework aligned, and discussed differences to develop a common understanding of the coding framework. the consensus estimate of inter-rater reliability was calculated as 85.1%; a value above 70% for irr indicates strong agreement between raters in application of the framework (stemler, 2004). the largest source of variation between raters came in determining complete, incomplete, and improper citations, which accounts for 44% of the differences in codes applied. these differences were discussed so that raters could reach consensus prior to coding the full data. finding a sufficiently high agreement rate between raters meant the authors could trust that the individual analysis of the citations would be sufficiently similar and that each could rate half of the references lists to distribute the load. raters then divided the student outlines based on which presentation method was used, such that each rater had half of the students they taught and half from the other class. more clarity on improper and incomplete reference and what constitutes “easily traceable” could bring the irr up to 91.6%. defining a reference as findable meant that basic users could locate the item, rather than requiring the skills of a librarian, who would use the other bits of information present and require more time to track it down.   coding framework modifications   during discussion between the two raters to verify agreement on use of codes, several modifications were proposed to the coding framework. some required modifications resulted from applying the framework to non-engineering-specific assignments and clarifying the application for the current research. a full description of the modifications made from the original used in wertz et al. (2011) can be found in the work-in-progress conference paper (van epps & sapp nelson, 2012).   results   references analysis   the sample of 36 bibliographies included 233 references for analysis to determine student use of resources and the ability to document those sources. the bibliographies included an average of 6.5 references per outline (233/36=6.5), which may seem high for first-year students in a speech class. the high average can partially be explained by the team assignment that contained an average of 16.8 references per outline (101/6=16.8) for all teams, thus skewing the average. without the team assignment, the average number of references per outline is 4.4 (132/30=4.4). while this is still slightly higher than expected, based on an average of 3.57 references in first-year student papers found by knight-davis and sung (2008), it is a reasonable number given the first assignment required two sources and the remaining three assignments all asked for a minimum of three citations.   when analyzing the number of references, the teaching team discovered that the one-shot session students averaged 3.87 (58/15=3.8667) references per individual assignment, and that the mini-lectures session students averaged 4.93 (74/15=4.9333) references per outline.   resource quality   using the quadrants presented by wertz et al. (2011), as illustrated in figure 1, the 233 references were rated for quality. of the full set of 233, 6 were removed from the quality assessment because they were coded as general web (gweb) resources or unknown (unkn), and with a broken link it was impossible to determine audience or intent of the resource.     figure 1 quality of resources     as shown in figure 2, the remaining 227 references were analyzed: 34.8% were high quality (scholarly and informative), 59.5% were medium quality (popular and informative, or scholarly and biased), and only 5.7% were low quality (popular and biased or entertainment).     figure 2 percent for each quality     cross-section analysis   for the cross-section analysis, researchers divided the assignments into two sets by type of library instruction the students received, one-shot or four mini-lectures. the one-shot session included 109 references and the mini-lectures session included 124 references. both groups had three citations that were removed due to broken links or unknown materials type.   the one-shot section presented the following break-down of references by quality: 2.7% unable to be classified due to broken links, 22.9% high quality, 65.2% medium quality, and 9.2% low quality. the mini-lectures section presented a different pattern, with 43.6% high quality, 51.6% medium quality, and 2.4% low quality. figure 3 shows the differences between sections based on the quality of resources used. high (z=3.31, p<.001), medium (z=-2.06, p<.05), and low (z=2.24, p<.05) quality ratings all show statistically significant differences between the sections.     figure 3 quality of resources cited     analysis of the references based on the type of resources used (figure 4) shows a statistically significant difference between sections for use of periodicals (z=6.52, p<.001) and web resources (z=-6.50, p<.001). the mini-lectures section exhibits more use of periodical sources, while the one-shot section used more web resources.     figure 4 type of resources     figure 5 shows the variation of types of resources used for each assignment in both groups. each assignment shows a pattern very similar to the overall type of resources analysis. the students who received the mini-lectures show more variation in the types of resources used, while the students who received the one-shot lecture do not appear to have changed their information use patterns, consistently using mostly web resources.     figure 5 types of resources used by assignment     figure 6 shows the differences between sections for the completeness of the references. the only statistically significant difference can be seen in the incomplete category (z=2.03, p<.05) and may reflect differences between raters more than differences in student abilities. librarians did not teach proper apa format, and identification of a reference as complete required only the presence of all elements of the reference, not full punctuation and formatting. for the majority of the assignments in both teams (55.7% jit; 60.6% one-shot), the students included all necessary elements for a complete citation.     figure 6 completeness of references analysis     discussion   results indicate that the presentation of information just prior to the completion of an assignment led to an increased number of high-quality resources being cited in student bibliographies. this supports the researchers’ hypothesis. those students who were exposed to the just-in-time sessions performed in a way that indicates that four 12-minute sessions throughout the term improves knowledge transfer of information literacy skills. while the same content was presented, the librarian offering the mini-lectures noted the ability for quick follow-up from the preceding assignment and a progression in the learning about library resources. while this practice generated a small difference in delivery, it was a natural outgrowth of repeated visits to the class and a desire from the students to understand why the sources for the preceding assignment were not adequate for the current assignment.   the fact that the primary learning goals of the course were not technical (i.e., a speech communications course) influenced the use of popular and informative resources (medium quality at 59.9%). the researchers were unsurprised by this result, particularly given the topic of assignment 3, the persuasive speech about a charitable organization. researchers coded 93.4% of the resources as informative, while only a small percentage of the resources were coded as biased, even for the charity assignment, a likely situation for integrating biased information. course instructors provided the grading and feedback returned to the students. therefore, the authors have no indication of the content, quality, or consistency of feedback that students were given on practice of information literacy skills as evidenced in the outlines and bibliographies.   the analysis of the number of complete references per assignment revealed consistent patterns across sections of 50%-65% complete on all four assignments. again, librarians did not teach reference formatting, and completeness simply signals that all the necessary components were present. the majority of complete references pattern holds even for the third assignment, where the necessary resources were mostly websites. the authors see this as an encouraging sign that students understand that more than just a url is required to identify websites in citations.   conclusions   the statistical analysis of student bibliographies indicates that the introduction of information literacy instruction for several brief lectures in conjunction with gateways or assignments in the curriculum improves outcomes. it cannot be determined if the changes in instruction model are the sole reason for observed variations, or if the section instructors impacted the outcomes through differences in teaching or grading.   this project presents intriguing possibilities for future research. a continuation of the study reported here within a different course, focusing on technical information, could explore if information literacy skills practiced in speech class are transferred into technical courses. repeating a similar experiment, but using two or more sections of the speech class taught by the same instructor, could indicate the extent that instructor input impacts the outcomes of this experiment.  building upon the observation that the group speech had much higher-quality resources and more complete citations, a study may also investigate if the use of group work helps to improve the information literacy skills of the group as a whole.    acknowledgements this paper is an extension of a conference paper that presented preliminary findings and was published in the proceedings of the 2012 asee annual conference and exposition by the same authors. the authors want to thank jennifer hall and elizabeth wilhoit for requesting the library instruction sessions and assisting the research by providing anonymized copies of student work. the research was approved by the purdue university institutional review board (irb), protocol number 1109011287.     references   badke, w. (2009). ramping up the one-shot. online, 33(2), 47-49.   bean, t. m., & thomas, s. n. (2010). being like both: library instruction methods that outshine the one-shot. public services quarterly, 6(2-3), 237-249. doi:10.1080/15228959.2010.497746   brendle-moczuk, d. (2006). encouraging students’ lifelong learning through graded information literacy assignments. reference services review, 34(4), 498-508. doi:10.1108/00907320610716404   bursic, k. m., & atman, c. j. (1997). information gathering: a critical step for quality in the design process. quality management journal, 4(4), 60-75.   byerly, g., downey, a., & ramin, l. (2006). footholds and foundations: setting freshmen on the path to lifelong learning. reference services review, 34(4), 589-598. doi:10.1108/00907320610716477   fain, m. (2011). assessing information literacy skills development in first year students: a multi-year study. the journal of academic librarianship, 37(2), 109-19. doi:10.1016/j.acalib.2011.02.002   gandhi, s. (2005). faculty-librarian collaboration to assess the effectiveness of a five-session library instruction model. community & junior college libraries, 12(4), 15-48. retrieved 3 mar. 2013 from http://www.tandfonline.com/doi/abs/10.1300/j107v12n04_05   hall, r. a. (2008). the “embedded” librarian in a freshman speech class: information literacy instruction in action. college & research libraries news, 69(1), 28-30. retrieved 3 mar. 2013 from http://crln.acrl.org/content/69/1/28.full.pdf+html   head, a., & eisenberg, m. (2009a). finding context: what today’s college students say about conducting research in the digital age. seattle, wa. retrieved 3 mar. 2013 from http://projectinfolit.org/pdfs/pil_progressreport_2_2009.pdf   head, a., & eisenberg, m. (2009b). lessons learned: how college students seek information in the digital age. seattle, wa. retrieved 3 mar. 2013 from http://projectinfolit.org/pdfs/pil_fall2009_finalv_yr1_12_2009v2.pdf   hollister, c. v., & coe, j. (2003). current trends vs. traditional models: librarians’ views on the methods of library instruction. college and undergraduate libraries, 10(2), 49-63. doi:10.1300/j106v10n02_05#   hsieh, c., & knight, l. (2008). problem-based learning for engineering students: an evidence-based comparative study. the journal of academic librarianship, 34(1), 25-30. doi:10.1016/j.acalib.2007.11.007   jacobs, h. l. m., & jacobs, d. (2009). transforming the one-shot library session into pedagogical collaboration: information literacy and the english composition class. reference & user services quarterly, 49(1), 72-82. retrieved 3 mar. 2013  from http://rusa.metapress.com/content/q7508652x22vl4t1/fulltext.pdf   knight-davis, s., & sung, j. s. (2008). analysis of citations in undergraduate papers 1. college & research libraries, 69(5), 447-458. retrieved 3 mar. 2013 from http://crl.acrl.org/content/69/5/447.full.pdf+html   martin, j. (2008). the information seeking behavior of undergraduate education majors: does library instruction play a role? evidence based library and information practice, 3(4), 4-17. retrieved 3 mar. 2013 from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/1838/3696   orr, d., appleton, m., & wallin, m. (2001). information literacy and flexible delivery: creating a conceptual framework and model. the journal of academic librarianship, 27(6), 457-463. doi: 10.1016/s0099-1333(01)00263-4   stec, e. (2006). using best practices: librarians, graduate students and instruction. reference services review, 34(1), 97-116. doi:10.1108/00907320610648798   stemler, s. e. (2004). a comparison of consensus, consistency, and measurement approaches to estimating interrater reliability. practical assessment, research and evaluation, 9(4). retrieved 7 jan. 2013 from http://pareonline.net/getvn.asp?v=9&n=4   student access transition & success. (2011a). college of engineering learning communities. retrieved 5 mar.  2012, from http://www.purdue.edu/sats/learning_communities/profiles/engineering/index.html   student access transition & success. (2011b). learning communities. retrieved 5 mar.  2012, from http://www.purdue.edu/sats/learning_communities/index.html   van epps, a. s., & sapp nelson, m. (2012). one or many? assessing different delivery timing for information resources relevant to assignments during the semester: a work-in-progress. proceedings of the asee annual conference and exposition. san antonio, tx: american society for engineering education.   weaver, k. d., & pier, p. m. (2010). embedded information literacy in the basic oral communication course: from conception through assessment. public services quarterly, 6(2-3), 259-270. doi:10.1080/15228959.2010.497455   wertz, r. e. h., ross, m. c., fosmire, m., cardella, m. e., & purzer, s. (2011). do students gather information to inform design decisions? assessment with an authentic design task in first-year engineering. annual conference and exposition of the american society for engineering education (pp. ac 2011-2776). asee.   yu, f., sullivan, j., & woodall, l. (2006). what can students’ bibliographies tell us? evidence based information skills teaching for engineering students. evidence based library and information practice, 1(2), 12-22. retrieved 3 mar. 2013 from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/8/123       appendix a description of speech assignments   network learning community com 114 speech assignments   speech #1: informative length: 3-4 minutes         description: in this speech, you will present to the class about one of the top engineering innovations of the 20th century. you will be given a list of topics from your instructor. you will explain to the class what the innovation was and what impact this innovation has had on the way that people live, work, or how we understand the world. this assignment will require a small amount of research, and each presentation must include two sources. this assignment emphasizes organization and delivery. it is important that you present the material in an appropriate organizational pattern for an oral presentation. you must have an introduction, body, and conclusion. this will help your audience understand and retain the information you provide. you also will be asked to pay specific attention to your delivery.   speech #2: informative length: 4-5 minutes description: in this speech we will be focusing on how to report information to different audiences with differing levels of knowledge. for this assignment the class will be divided into groups of three. each small group will be assigned a machine, process, or technological innovation works. each individual in the group will also be assigned a target audience; fellow engineers, potential consumers, or high school juniors. although the groups of three will have the same topic and will present on the same day, you do not need to collaborate on your presentations. your task will be to explain how this machine, process, or technology works in a way that is appropriate for your target audience. this presentation must be based on at least 3 sources and use an appropriate organizational pattern and include a clear intro, body, and conclusion.   speech #3: persuasive length: 5-6 minutes description: for this presentation you are going to persuade your classmates to support a charity or nonprofit organization by donation their time, money, or tangible goods. you are going to persuade your audience to volunteer or to donate money or other tangible goods. you will use a problem-solution format. first explain what the problem is and then explain why your audience should support the organization you chose to help that problem. for example, you might want to persuade your audience to donate blood. you would first talk about the problem which is the need for blood and possible blood shortages and then explain how being a blood donor can help solve that problem. you can also talk about the personal benefits one might get from supporting the cause you chose. these can be national or local organizations.   speech #4: group presentation length: 30-35 minutes notes: 1 typed sheet or 1 4x6 notecard per person               description: in this speech, you must take various concepts/products (a car, a computer, a home, a classroom, a restaurant, etc.) and completely rethink the object or space to make it more user-friendly and/or efficient. you must develop visuals of your new product so the audience can visualize it. your audience for this speech is a venture capital firm, so be sure to “pitch” your product as well as you can.     appendix b coding framework for speech outlines   research article   university community engagement and the strategic planning process   laura newton miller assessment librarian carleton university ottawa, ontario, canada email: laura.newtonmiller@carleton.ca   received: 27 sep. 2017    accepted: 6 feb. 2017      2018 newton miller. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29351     abstract   objectives – to understand how university libraries are engaging with the university community (students, faculty, campus partners, and administration) when working through the strategic planning process.   methods – literature review and exploratory open-ended survey to members of caul (council of australian university librarians), carl (canadian association of research libraries), conzul (council of new zealand university librarians), and rluk (research libraries uk) who are most directly involved in the strategic planning process at their library.   results – out of a potential 113 participants from 4 countries, 31 people (27%) replied to the survey. libraries most often mentioned the use of regularly-scheduled surveys to inform their strategic planning, which helps to truncate the process for some respondents, as opposed to conducting user feedback specifically for the strategic planning process. other quantitative methods include customer intelligence and library-produced data. qualitative methods include the use of focus groups, interviews, and user experience/design techniques to help inform the strategic plan. the focus of questions to users tended to fall towards user-focused (with or without library lens), library-focused, trends and vision, and feedback on plan.   conclusions – combining both quantitative and qualitative methods can help give a fuller picture for librarians working on a strategic plan. having the university community join the conversation on how the library moves forward is an important but difficult endeavour.  regardless, the university library needs to be adaptive to the rapidly changing environment around it. having a sense of how other libraries engage with the university community benefits others who are tasked with strategic planning.       introduction   contributing to student success and demonstrating value are growing trends in academic libraries (connaway, harvey, kitzie, & mikitish, 2017; acrl research planning and review committee, 2016). in trying to discover how one engages with the university community, it becomes “imperative to co-create, rather than dictate, value to users” (peacemaker & stover heinz, 2015, p. 270). engaging users to help inform library strategic plans emphasizes our concerted effort to become more user-centred organizations (white, 2012). strategic planning is a necessary undertaking in most university libraries, but information about how academic libraries are involving and engaging with the university community in that planning is limited. through a literature review and an open-ended, exploratory survey with librarians from australia, canada, new zealand, and united kingdom, the focus of this paper is to understand how university libraries are engaging with the university community (students, faculty, campus partners, and administration) when working through the strategic planning process.   literature review   direct communication and engagement with the university community helps “an organization adapt quickly to the unpredictable and rapidly changing environments most organizations face” (cervone, 2014, p. 163). strategic planning is a formal process that involves an organization envisioning the future and developing the procedures and processes needed to achieve it (goodstein, nolan, & pfeiffer, 1993). most higher education institutions ask two fundamental questions when planning: what should we be doing (scanning external conditions) and how well do we do what we do (evaluating internal operations) (mcintyre, 2012).   whether a business or non-profit body (including a library, specifically), understanding the needs of the users or customers of an organization is part of strategic planning. “listening to the voice of the customer will ensure that the library understands the perceptions of its customers and the value the library provides rather than drawing conclusions and inferences using the myopic vision of a library’s assumptions and beliefs” (matthews, 2005, p. 101). planned strategy and identified customer needs must be aligned and clearly communicated in order to promote a sense of unique user defined value (germano & stretch-stephenson, 2012).   university community engagement means the involvement of faculty, students, campus partners, or administrators in the library strategic planning process. the literature on library strategic planning and university community engagement is surprisingly limited. many papers refer to its importance, but the main theme of the paper relates to something else (e.g., a specific aspect or a case study on a library’s planning process). some refer to the importance or strategies of the communication of the plan to its audience(s), but not necessarily on the “how” of involving users in the actual planning process (jacob, 1990; matthews & matthews, 2013; mcnicol, 2005).   germano & stretch-stephenson come close with a paper that focuses on marketing’s role in strategic planning. they state that strategic plans have a better chance of being successfully implemented if the focus is “reoriented towards understanding patrons in a way that considers their needs as well as the role the library plays in their overall information seeking behaviours” (2012, p. 75). this is not an easy task. connecting patron attitudes and needs with planning service goals is a significant difficulty for university libraries (germano & stretch-stephenson, 2012).   quantitative methods   decker and höppner (2006) discuss the use of customer intelligence in academic libraries to inform planning. examples of customer data include anonymized lending data and data found through card swipes when entering the library or using various services (which may not be anonymized). examining library patron data has been a topic of increasing discussion in assessment conferences and papers, much of it surrounding the need to demonstrate library value to university administration (matthews, 2012; beile, choudhury, & wang, 2017; renaud, britton, wang, & ogihara, 2015). however, there are concerns regarding the privacy of this data (varnum, 2015; chen et al., 2015).   strategically analyzing a variety of data routinely gathered in the library helps to provide clear direction for future decision making. examples include interlibrary loan and circulation statistics, gate counts, usage data (electronic resource usage data, website visit, or log files), and service data (equipment lending or help desk visits) (huff-eibl, miller-wells, & begay, 2014).   looking external to the library, national surveys help to understand different user groups, with institution-specific information available through many institutional research offices at individual universities. these surveys measure students’ engagement, experience, and satisfaction levels of different aspects of university life. libraries and their institutions can glean information not only on students’ research and learning needs, but also their levels of engagement and satisfaction with library services, resources, and space. these include (but are not limited to) nsse (national survey of student engagement, us & canada), cgpss (canadian graduate and professional student survey), nss (national student survey, higher education funding council for england), and ausse (australasian survey of student engagement).   libqual+ is a web-based survey offered by the association of research libraries that measures library users’ views and levels of service quality (arl, n.d.). it can be used for both quantitative and qualitative analysis (survey data versus comments). there are some papers specifically related to libqual+ and its use in strategic planning. stewart saunders (2008) found that although libqual+ ultimately informed purdue university library’s planning, the survey data initially limited their focus and confused the issues of strategic versus operational planning. while mostly used as a measurement system to address various objectives in the strategic plan, the university of florida used libqual+ results as a “discussion-starter” with their primary user groups (shorb and driscoll, 2004, p. 176). piorun (2011) interviewed five leaders of strategic planning in academic health libraries. all five libraries used libqual+ as the primary way of reaching stakeholders to inform strategic planning. in this and other papers on libqual+, all libraries also analyzed the comments to help get a deeper understanding of needs.   qualitative methods   bowling green state university libraries (haricombe & boettcher, 2004) and american university library (becher & flug, 2005) informed their strategic plans by carefully mapping focus group questions to the quantitative data from their libqual+ surveys. focus groups can help validate or challenge quantitative survey findings. as part of a larger discussion on the library planning process, nutefall (2015) briefly mentioned the use of focus groups of students and faculty to gather feedback on three priority areas (engagement, collections, and learning environments) that the planning committee deemed had the most impact on the university community. higa-moore, bunnett, mayo, and olney (2002) integrated focus groups into the university library’s long-term planning process. although there were benefits of raising the library’s profile and reinforcing the perception of being customer-oriented, a key disadvantage is that it is very labour-intensive and expensive to carry out on a large scale. moreover, precautions need to be taken to ensure that diverse perspectives are sought, as the views of highly engaged users may create disproportionate influence (peacemaker & heinze, 2015).   user experience (also referred to as ux) “is a suite of techniques based around first understanding and then improving the experiences people have when using our library services. it utilises ethnography and design to achieve this” (potter, n.d.). ethnographic research (including the use of observation, field notes, interviews, and other techniques) can provide rich information that helps to understand the experience of the user. priestner and borg (2016, p. 3) state that “…we are not our users, and just because they carry out tasks in a way that is alien to us does not mean that their way is wrong or broken. instead, we need to see their approach as an opportunity to learn and discover.” although an example of community college as opposed to library-specific strategic planning, mcintyre (2012) described the use of qualitative methods like ethnography as providing important context for identifying and prioritizing strategic planning goals. with more universities focusing on the “student experience,” examining how users are experiencing academic libraries is something that cannot be ignored (priestner & borg, 2016).   other methods   a small number of papers discussed other methods to involve users in the strategic planning process. the university of arizona libraries used a combination of metrics (i.e., circulation data, gate counts, and number of questions asked at service points), feedback comments (library report card), unmet customer needs, observational data (headcounts plus type of activity), libqual+ results, and an annually administered survey to understand the voice of the customer (huff-eibl, miller-wells, & begay, 2014).  eastern washington university libraries had an unprecedented opportunity of receiving funding for a strategic planning grant to hold a two-week institute for faculty to develop a vision for the future of the library. as incentive, faculty stipends were equivalent to teaching a summer course. although there were many presentations from a variety of library staff on various aspects to consider for the strategic plan, one of the highlights of the institute was attending a panel of four undergraduate students (with different majors) who spoke about their research habits (miller, 2009). university of california san diego libraries used an “open-ended” approach to gathering feedback from students and faculty. the planning working group wanted to encourage users to “think broadly about the role of information and existing and possible new library services and spaces” (p. 3) and thus mostly avoided questions about traditional library services. although not going into the details of how they spoke to users (it wasn’t clear if they conducted interviews or focus groups or something less formal), they did get “on-the-fly” input from a link on their website and from white boards in public areas which users could use to respond to specific questions (williams, nickelson dearie & schottlaender, 2013).     table 1 response rate and country of origin country # respondents potential total participants response rate australia 11 39 28% canada 13 29 45% new zealand 4 11 36% united kingdom 3 34 9%       aims   through a combination of a literature review and an exploratory, open-ended survey, this paper will seek to understand how university libraries are engaging with their community (students, faculty, campus partners, and administration) when creating a strategic plan. what kind of techniques are libraries using to engage with the university community? what kinds of questions are they asking their users to help them form the strategic plan? what has been helpful (and not) in this process? getting a sense of how other libraries are engaging with the university community will benefit others who are tasked with strategic planning. the emphasis of this research is on the process to get to the strategic plan and not on the strategic plan itself.   method   in the fall of 2016, the researcher sent an online, open-ended, exploratory survey via email to the executive directors of caul (council of australian university librarians), carl (canadian association of research libraries), conzul (council of new zealand university librarians), and rluk (research libraries uk). the executive directors were asked to send the survey to their members, requesting the person most directly involved in the strategic planning process to answer the survey (e.g., the university librarian, an associate university director, an assessment librarian, or other). respondents were asked questions related to the process used for creating their strategic plans, including what was helpful (or not), kinds of questions asked of the community, and words of advice for those starting a strategic planning process. (see the appendix for the complete survey).   results   the main focus of this article is to examine different approaches regarding the engagement of the university community (students, faculty, campus partners, and administration) when undertaking a strategic planning process. out of a potential 113 participants from 4 countries, 31 people (27%) replied to the survey. university locations are divided below, but are discussed as a whole throughout the paper because of the emergence of themes regardless of location. a total of 28 out of the 31 respondents stated that their library had a strategic plan. two responded “no,” and one skipped this question.   out of the 28 respondents with a strategic plan, 23 mentioned some sort of user engagement in their replies. libraries used a variety of techniques to gain user feedback for strategic planning purposes. some only mentioned one, while others pointed out that they used a “variety of tools” and a “combination of activities.”  the average (mean) number of methods used by university libraries is two, with the most often occurring number (mode) being one and two activities. three universities use four activities. figure 1 illustrates the kinds of methods used to engage with the university community.     figure 1 methods used to engage with the university community.       quantitative methods   quantitative methods highlight the use of data generated from a number of sources. this can include internal library systems (e.g., electronic usage statistics, circulation or other service statistics), university systems (e.g., student admissions data), and survey responses (either library or university-based).   e-resource data & data analytics (university & library)   quantitative methods found in the open-ended survey responses focus on customer intelligence, library-produced data, and survey results. only one library (making three comments throughout the survey) discussed taking advantage of e-resource use data and data analytics from university and library-based resources to help inform strategic planning. they commented that “library use data for e-resources, and services such as online chat, etc. have … better informed our planning” and they use “data analytics from university systems including learning management and library systems.”   university offices of institutional research can also provide data on student and faculty populations. one respondent explained:   all is helpful [in strategic planning] but especially data that is now available from the university’s business intelligence system including student, admissions, student load and projections, finance, staff, research and quality. university staff have access to aggregated data which means that our teaching, liaison and research librarians, and repository staff can access and use this for their own planning throughout the year.   survey data   libraries most often mentioned the use of regularly-scheduled surveys to inform their strategic planning (n=13 libraries). these included libqual+, satisfaction surveys, ithaka, insync, university surveys, and other surveys (non-specified). many libraries expressed the reality that the timing of the strategic planning process was quite restrictive (newton miller, in press). stephan (2010) found that having much of the data already gathered was very helpful when working on a strategic plan on a very short timeframe. regularly-scheduled assessment activities seem to help truncate the process for some respondents and help to reduce “over-surveying,” as opposed to conducting user feedback specifically for the strategic planning process. one respondent said:   every 3 years we conduct an extensive client satisfaction survey that asks clients what is important to them and how we are performing.  the responses are benchmarked against other academic libraries …. there are so many surveys used now within higher education that there is a backlash from clients about over-surveying so we are very careful to communicate by closing the feedback loop and not over survey and waste clients' time.   another commented: “we run a bi-annual client library survey which identifies opportunities for improvement as well as a performance assessment which is benchmarked across our sector.”   qualitative methods   qualitative methods emphasize the use of focus groups, interviews, and user experience/design techniques to help inform the strategic plan by better understanding user needs.   focus groups, meetings, & interviews   undergraduates, graduates, and faculty were involved in focus groups in seven libraries. three libraries mentioned “meetings” with users. six libraries spoke of using client interviews in the strategic planning processes. examples include using “interviews with library users, staff and key stakeholders, including deputy vice-chancellors and other leaders within the university”. one respondent stated “11 interviews with university stakeholders”, and another utilized “extensive stakeholder one-on-one interviews with focus not on library but on stakeholder needs”.   user experience   although user experience (ux) and design techniques are relatively new in librarianship, there were five universities that made comments relating to its use in the strategic planning process. one “watched how they move about in the library; ask them to complete tasks and see how they do it”. another stated that they “focus on user experience and try to frame questions from the users’ perspective, using their language, so there is not a mismatch or confusion of what you are asking them because of library jargon”. one spends time “at select points of the academic year just observing client behaviour in the libraries and interacting with library services and resources”. a respondent noted that “more recently, design thinking methodology is being deployed to explore targeted strategies such as online delivery, digital learning objects, website design and communications”.   other methods   it is worth noting other methods of gathering feedback from users that do not fit into a neat category. one library had an “open house” with 69 student participants. another library has staff attend staff-student panels (organized by the student union) to identify “burning issues” and ask what would make their library service better for them. this library also runs marketing campaigns on topics such as e-books, which can also help inform the plan. one library conducted a student to student “street survey” of “non-library users” at four campus locations (although this was found to be unsuccessful: “it turns out, that group is hard to find!”). finally, one library tried to involve student leaders in the development of their plan, with mixed success, depending on the commitment and interests of the leaders.   what was helpful   identifying and understanding patron needs through consultation was certainly helpful in the strategic planning process. one respondent stated that they “wanted to use surveys to ensure our activities responded to identified needs; we used focus groups to find out what students value from our services and which they don't notice”. one noted that “honest feedback on where they saw that the library could add value [was helpful]; a broad view rather than a solely internal view”. the same respondent found “collecting statements which showed that we are not communicating well and not meeting needs” was also useful for planning. another respondent found “particularly focusing on stakeholder needs [was beneficial] as the plan anticipated numerous university priorities that emerged subsequently”.   respondents mentioned the conversations with senior executives as particularly useful, stating “the direct conversations with new senior executive staff was … helpful in providing a sense of strategic direction and an awareness of short-term priorities”. another remarked that “individual meetings with key decision makers (deans, provost, etc.) worked very well”.   what would they do differently   most comments around user engagement focused on wanting to do more of it. usually because of tight timelines, some were not able to involve students, faculty, and others in the university community as much as they would have hoped. one noted “we definitely need to engage with our stakeholder groups more through the process, including staff, students and the community”. this was echoed in other statements, including “we did not have any students, professional staff from other service departments, or any academic staff present, and i would consider inviting representatives in the future” and “try to engage with more stakeholders: it was difficult to convince them to spend the time”. timing affected some planning processes, leading one librarian to remark “our timelines were dictated by the university; [we] would have liked to have had more time (or a different time of year) to better engage students and faculty”. finally, one respondent encouraged libraries to “engage as broadly as possible and don’t be discouraged that both positive and negative voices will come to the fore”.   kinds of questions asked   besides the method used for gaining insight into user needs, it was interesting to learn what libraries ask their users. questions tended to fall towards user-focused (with or without library lens) (n=16 comments), library-focused (n=9 comments), trends and vision (n=6 comments), and feedback on plan (n=2 comments). user-focused questions varied: some focused on the user within the library context, and others within a more general research/needs context.   user focus with a library lens:   user focused questions centre on user needs, but within the context of the library. examples include questions like “how do you use the library? what else could the library be doing to support your success?” (undergraduate students), “how do you use the library? how can the library assist you?” (graduate students), and “how well are we supporting your teaching and research activities?” (faculty). other examples include “what do you expect from the library?” and “how are you using the library?”   user focus without the library lens:   user focused questions zero in directly on users and their needs, without a focus on the library. one respondent explained that questions were about “their needsnot their needs of the library. there is a differenceand an advantage as it discourages them from answering via their perceptions of what the library can provide to them”. another commenter stated that they used “the ithaka s&r faculty survey as the primary faculty input into our planit asks faculty about their behaviors and research and teaching habits, more than asking them about what they want in the future of the library”.   library-focused   library-focused questions are different than user-focused with a library lens, focusing more on existing services than on the user specifically. these include questions like “what services are declining, increasing and what new services might we need? how will we provide these services and what roadblocks do we have to overcome?” another library asks similar questions, including “what can we improve? what are we not doing that we could be doing? how can we better serve interdisciplinary needs and research? finally, one respondent mentioned the following questions which are library-focused:   we target the questions to the level of staff/student being asked. however, some general questions could bewhat are the top 5 things we do well? what are the top 5 things we could improve? what services should we offer that we currently don’t? what would your ideal library service provide? can you tell me about a company or department you think provides great service? why do you think that? other comments?   trend and vision   trend and vision questions are exactly that: questions that focus on current trends and users’ vision of the future. examples include “what are the drivers for change? what are the key global, national or local trends which will impact library services over the next 5 years?”, “what do you see as future trends?” “impacts in the last ten years. impact of technology”, “what’s our business, now and in the foreseeable future?”, “open ended questions like ‘what should the library look like in 2020?’” feedback of draft plan   two comments focused on getting feedback on the draft strategic plan. one comment reflects that this is the only kind of feedback received from users, asking “mostly whether they broadly agree that the plan captures the priorities, that the context is stated accurately and whether anything is missing.” another respondent remarked that along with other engagement with the university community, there is also “an open commenting period when we launched the draft plan.”   one respondent mentioned that the amount of feedback from the university community in response to the circulated draft was actually not helpful in the process.   discussion   university libraries use a variety of methods to engage with their community in order to feed into the strategic planning process. simplistically, quantitative methods answer the “what” and “how many” questions, while qualitative methods answer the “why” and the “how.” because of time constraints, many libraries depend on quantitative methods such as regularly-generated survey data instead of leading user feedback endeavours specifically for the planning process. surprisingly, only one library mentioned using library-produced data to help inform the strategic plan. however, this may be a result of the open-endedness of the survey. in other words, some other libraries may use library-produced data, but because the survey is based on memory, the respondents failed to mention this as part of data-gathering. library-produced data may also be included in the “variety of activities” that were mentioned by respondents.   using survey data to help inform the strategic plan can have the benefit of its already being available to library staff. libraries can get the opinions of many users in a relatively quick timeframe. responses can also be benchmarked either with other libraries or with past surveys of the same library. however, there can be disadvantages to the survey, including low response rates and time needed to create the survey. ethics review or survey approval may be needed at an institutional level. (survey approval is needed at some institutions in order to avoid “over surveying” the university community.) finally, it can be difficult to understand the context behind the data that is produced from surveys alone.   some libraries mention the use of interviews, focus groups, and user experience techniques to help further understand quantitative findings. these approaches can provide needed context to quantitative data, allowing the opportunity for deeper analysis of issues. they can provide information on stakeholder priorities and values, and can also give libraries a sense of what users are not noticing. user experience techniques allow library staff to see what users do and not necessarily rely just on what users say. these techniques can provide very rich and deep understanding of the user. as with all qualitative techniques, time is needed to create questions, recruit participants, and deeply analyze results. some may perceive a risk of relying on only a limited number of perspectives.   the limited number of responses made dividing user groups into students, faculty, and university administration difficult, but it appears that interviews made with high ranking university officials were helpful for those working through the strategic planning process. whether the interviews were part of regular meetings with this group or were formed specifically for the process is unclear. meetings between high ranking library administration and their peers within the university may help some in getting feedback for the strategic plan in a time-sensitive fashion.   limitations   this was an exploratory study, and an open-ended survey was purposely used to gauge initial understanding of user engagement and the strategic planning process. it is limiting, however, because of the level of detail given in some responses. responses are based on memory of personal reflections on strategic planning. it is not certain when strategic planning took place. interviews would have given the opportunity for elaboration. a survey with fields and definitions to choose from would have also made for more clear descriptions (e.g., definition of “meeting”), but would have limited the open-endedness of responses. a combination of strategies would be helpful for future study on this topic.   conclusion   taking the different voices of so many in the university into account to inform the strategic plan is difficult. combining both quantitative and qualitative methods (including the growing popularity in user experience and design techniques) can help give a fuller picture for librarians working on a strategic plan. prioritizing those voices will be dependent on the culture of each university. what is interesting is the kinds of questions libraries are asking their users to help in their planning. are we asking the right kinds of questions? if we are to be user-centred institutions, should our questions be user focused or library focused? should user focused questions have a library theme attached to them? or should we be asking user focused questions without guiding the users as to how the library should help them? examining questions asked of users in library strategic planning papers will be the focus of future research.   this paper aims to help those tasked with strategic planning to understand how other libraries engage with the university community. the university library needs to be adaptive to the rapidly changing environment around it. having the university community join the conversation in how the library moves forward in this environment is an important but difficult endeavour. setting priorities and mitigating expectations is no easy task, but one of vital importance if the library is to create a strategic plan that is meaningful for all its users in the university community.   references   acrl research planning and review committee. (2016). 2016 top trends in academic libraries: a review of the trends and issues affecting academic libraries in higher education. college & research libraries news, 77(6), 274-281. https://doi.org/10.5860/crln.77.6.9505     association of research libraries. (n.d.). libqual+: charting library service quality. retrieved from https://www.libqual.org/home   becher, m. l. & flug, j. l. (2005). using student focus groups to inform library planning and marketing. college & undergraduate libraries, 12(1-2), 1-18. http://dx.doi.org/10.1300/j106v12n01_01   beile, p., choudhury, k., & wang, m. c. (2017). hidden treasure on the road to xanadu: what connecting library service usage data to unique student ids can reveal. journal of library administration, 57(2), 151-173. http://dx.doi.org/10.1080/01930826.2016.1235899   cervone, h. f. (2014). improving strategic planning by adapting agile methods to the planning process. journal of library administration, 54(2), 155-168. http://dx.doi.org/10.1080/01930826.2014.903371     chen, h., doty, p., mollman, c., niu, x., yu, j., & zhang, t. (2015). library assessment and data analytics in the big data era: practice and policies. proceedings of the association for information science and technology, 52(1), 1-4. http://doi.org/10.1002/pra2.2015.14505201002   connaway, l. s., harvey, w., kitzie, v., & mikitish, s. (2017). academic library impact: improving practice and essential areas to research. chicago: association of college and research libraries. retrieved from http://www.ala.org/acrl/sites/ala.org.acrl/files/content/publications/whitepapers/academiclib.pdf   decker, r. & höppner, m. (2006). strategic planning and customer intelligence in academic libraries. library hi tech, 24(4), 504-514. http://dx.doi.org/10.1108/07378830610715374   germano, m. a. & stretch-stephenson, s. m. (2012). strategic value planning for libraries. the bottom line, 25(2), 71-88. http://dx.doi.org/10.1108/08880451211256405   goodstein, l. d., nolan, t. m., & pfeiffer, j. w. (1993). applied strategic planning: a comprehensive guide. new york: mcgraw-hill.   haricombe, l. j. & boettcher, b. j. (2004). using libqual+ data in strategic planning. journal of library administration, 40(3-4), 181-195. http://doi.org/10.1300/j111v40n03_14   higa-moore, m. l., bunnet, b., mayo, h. g., & olney, c. a. (2002). use of focus groups in a library’s strategic planning process. journal of the medical library association, 90(1), 86-92. https://www.ncbi.nlm.nih.gov/pmc/articles/pmc64762/   huff-eibl, r., miller-wells, j., & begay, w. (2014). understanding the voice of the customer: practical, data-driven planning and decision making for access services. journal of access services, 11(3), 119-134. http://doi.org/10.1080/15367967.2014.916150   jacob, m. e. l. (1990). strategic planning: a how-to-do-it manual for librarians. new york: neal-schuman publishers.   matthews, j. r. (2005). strategic planning and management for library managers. westport, ct: libraries unlimited.   matthews, j. r. (2012). assessing library contributions to university outcomes: the need for individual student level data. library management, 33(6/7), 389-402. https://doi.org/10.1108/01435121211266203   matthews, s. a. & matthews, k. d. (2013). crash course in strategic planning. santa barbara, ca: libraries unlimited.   mcintyre, c. (2012). qualitative client-based research: tools for planning. planning for higher education, 40(4), 13-22. retrieved from https://www.scup.org/page/phe/read/article?data_id=31449&view=article   mcnicol, s. (2005). the challenges of strategic planning in academic libraries. new library world, 106(11/12), 496-509. http://dx.doi.org/10.1108/03074800510634982   miller, j. l. (2009). “reinventing the library” at eastern washington university: an intensive strategic planning experience with faculty. technical services quarterly, 26(2), 96-106.  https://doi.org/10.1080/07317130802260792   newton miller, l. (in press). what is helpful (and not) in the strategic planning process? an exploratory survey and literature review. library leadership & management.   nutefall, j. (2015). how an outside facilitator helped us create a better strategic plan. library leadership & management, 29(3), 1-8. https://journals.tdl.org/llm/index.php/llm/article/view/7112   peacemaker, b., & stover heinze, j. (2015). moving users, moving results: exploring customer engagement for deeper relationships. college & undergraduate libraries, 22(3-4), 261-272. http://dx.doi.org/10.1080/10691316.2015.1081084     piorun, m. (2011). evaluation of strategic plans in academic medical libraries. library & information science research, 33(1), 54-62. https://doi.org/10.1016/j.lisr.2010.05.002     potter, n. (n.d.). ux in libraries resource list: a structured introduction to ux and ethnography. retrieved from https://www.ned-potter.com/ux-in-libraries-resource-list/   priestner, a. & borg, m. (eds.). (2016). user experience in libraries: applying ethnography and human-centred design. new york: routledge.   renaud, j., britton, s., wang, d., & ogihara, m. (2015). mining library and university data to understand library use patterns. the electronic library, 33(3), 355-372. http:dx.doi.org/10.1108/el-07-2013-0136   shorb, s. r. & driscoll, l. (2004). libqual+ meets strategic planning at the university of florida. journal of library administration, 40(3-4), 173-180. http://dx.doi.org/10.1300/j111v40n03_13   stephan, e. (2010). strategic planning on the fast track. library leadership & management, 24(4), 189-198. https://journals.tdl.org/llm/index.php/llm/article/view/1856/1129   stewart saunders, e. (2008). drilling the libqual+ data for strategic planning. performance measurement and metrics, 9(3), 160-170. http://doi.org/10.1108/14678040810928390      varnum, k. (2015). editorial board thoughts: library analytics and patron privacy. information technology and libraries, 34(4), 2-4. http://doi.org/10.6017/ital.v34i4.9151   white, g. w. (2012). developing a marketing plan for the library by and for students. in l. snavely (ed.), student engagement and the academic library (pp. 95-103). santa barbara, ca: libraries unlimited.   williams, j., nickelson dearie, t., & schottlaender, b. e. c. (2013). bottom-up strategic planning: the uc san diego libraries experience. library leadership & management, 27(3), 1-13. https://journals.tdl.org/llm/index.php/llm/article/view/7031     appendix survey questions   where is your university located? canada united kingdom australia new zealand   what is your role in the library? library director/university librarian associate/assistant university librarian assessment librarian other   do you have a strategic plan for your library? yes/no (if no, thank you for your help. please go the end of the survey).   strategic planning   can you tell me about the process you used for creating your strategic plan (e.g., interviews/focus groups with library staff/users, meetings/consultations with campus partners, etc.)   what was helpful/what worked in that process?   what was not helpful/did not work in that process?   is there anything you would do differently next time?   what kinds of questions do you ask to library staff? are they different for different staff groups?   what kinds of questions do you ask to library user groups (e.g., undergraduate and graduate students, faculty, campus partners, etc.)   yearly updates   do you conduct any yearly updates/check-ins of your plan? yes/no (if no, go to final comments section)   if yes, what process do you use? (e.g., interviews/focus groups with library staff/users, meetings/consultations with campus partners, etc.)   what was helpful/what worked in the yearly update/check-in process?   what was not helpful/did not work in the yearly update/check-in process?   what kinds of questions do you ask to library staff? are they different for different staff groups?   what kinds of questions do you ask to library user groups (e.g., undergraduate and graduate students, faculty, campus partners, etc.)   final comments   do you have any words of advice for those starting a strategic planning process?   do you have any further comments?   microsoft word es_lewis.doc evidence based library and information practice 2007, 2:3  111 evidence based library and information practice       evidence summary    questions remain about whether ebm competencies learned in medical school are  retained in residency      a review of:  dorsch, josephine l., meenakshy k. aiyer, krishna gumidyala and lynne e. myer. “retention of  ebm competencies.” medical reference services quarterly 25.3 (fall 2006): 45‐57.      reviewed by:   suzanne lewis  library manager, central coast sector, northern sydney central coast health  gosford, new south wales, australia  e‐mail: slewis@nsccahs.health.nsw.gov.au        received: 29 may 2007    accepted: 12 july 2007      © 2007 lewis. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to assess medical graduates’  use of evidence based medicine (ebm) in  residency, self‐perception of ebm skills,  attitudes toward ebm, and the impact of a  formal ebm curriculum in their third  undergraduate year.    design – a longitudinal follow‐up study by  self‐administered questionnaire.    setting – internal medicine residency  programs in us hospitals.    subjects – a convenience sample of 2001  and 2002 graduates of the university of  illinois college of medicine at peoria  (uicom‐p) (n=78), and their respective  residency program directors (n=72).    methods – a student graduate  questionnaire (sgq) was sent to all  members of uicom‐p’s 2001 and 2002  graduating classes who had completed the  ebm course during their m‐3 internal  medicine clerkship. a program director  questionnaire (pdq), similar to the sgq,  was sent to the graduates’ residency  program directors. the research instrument  was tested with a pilot group prior to use,  but not validated. the questionnaires  consisted of 4 main sections. the first  section examined formal and informal ebm  programs in the graduates’ residency  curriculum. the second section consisted of  a self‐assessment of ebm skills by the  mailto:slewis@nsccahs.health.nsw.gov.au http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:3  112 residents and an assessment of those skills  by their program directors. the third section  asked graduates to compare their ebm skills  to those of their fellow residents who had  not been students at uicom‐p. similarly, in  the third section of the pdq, program  directors were asked to compare the ebm  skills of uicom‐p graduates and non‐ uicom‐p graduates participating in the  residency program. the last section  concerned professional and demographic  characteristics. copies of the surveys were  mailed out to non‐responders after 6 weeks.  results were collated but statistical analysis  was not applied.    main results – the response rate was 32%  for residents and 35% for program directors.  the number of incomplete surveys was not  reported. forty‐four percent of all  respondents reported having a formal ebm  curriculum for residents, and 76% reported  an informal curriculum. for both formal and  informal programs, the most common  teaching formats were journal clubs,  followed by lectures, teaching rounds,  morning reports, bedside consultations,  ambulatory clinics and seminars. in section  two of the questionnaires, both residents  and program directors rated the residents’  ebm skills similarly. however, the residents  rated their skills in searching the literature  and application of findings to clinical  practice higher than the program directors.  program directors also rated the residents’  skills in understanding statistics and tests  higher than the residents themselves.  twenty‐four percent of both residents and  program directors rated the uicom‐p  graduates as “very competent” or  “extremely competent” in ebm skills (50).  only 35% of program directors and 27% of  residents rated the uicom‐p graduates’  ebm skills as “usually better” or “always  better” than their peers who were not  uicom‐p graduates (50).      conclusion ‐ the authors of this study   conclude that, for uicom‐p graduates, “it   might be implied from these results that the  ebm skills gained during medical school  were retained through their medical school  graduation and into their residency  training” (51). however, this study has  methodological weaknesses which make it  difficult to draw any definite conclusions  from the results.    commentary    the authors themselves identify 3 serious   limitations of the study. the first limitation  is that, due to ethical considerations, the  anonymity of the respondents had to be  preserved. returned surveys could not be  matched to the individual students who had  undertaken the ebm course in their m‐3  internal medicine clerkship. as a result, a  direct relationship between student  performance in the undergraduate ebm  course and retaining ebm competencies in  postgraduate years could not be established.  therefore the authors were unable to  answer the central question of their study:  “whether the incorporation of a formal ebm  curriculum in a third‐year im clerkship has  a long‐term impact on evidence‐based  practice in residency” (46). in effect, all they  were measuring was the residents’  perceptions of their current ebm  competencies.     the second limitation of the study is that the  method used to measure ebm competency  was a subjective questionnaire eliciting the  subjects’ perceptions of their skills, rather  than an objective measure such as a test  administered by a third party. the authors  identify a third limitation of the study ‐ that  it did not determine whether the non‐ uicom‐p graduates had had any formal  ebm instruction in their undergraduate  medical program. this makes it difficult to  draw any meaningful conclusions from  section three of the questionnaire, which  evidence based library and information practice 2007, 2:3  113 attempted to identify differences between  the ebm skills of uicom‐p graduates and  graduates of other medical schools.    a limitation of the study that the authors do  not address is that the source of the  residents’ ebm skills is not identified. did  they learn these skills in their m‐3 internal  medicine clerkship, in their formal or  informal residency programs, or from some  other source? the low response rate to the  surveys compounds the methodological  flaws discussed above. one reason for the  low response rate may have been the messy  layout of sections ii and iii of the  questionnaire.     finally, the reporting of this study could  have been improved. more contextual detail,  particularly regarding the us medical  education and residency system, would  have been helpful for an international  audience. in addition, only the student  graduate questionnaire was supplied in the  appendix. the program director  questionnaire should have also been  included.    if the institutional review board had  allowed it, the study would have been  strengthened by being reframed as a cohort  study tracking a group of medical students  who received ebm training in their third  undergraduate year at uicom‐p, with ebm  skills objectively measured immediately  before and after the undergraduate training  and then in their residency. a control group  of medical students who did not receive  ebm training as part of their undergraduate  curriculum, but underwent identical testing  of ebm competency, would have further  increased the validity of the study. however,  in practice it would be extremely difficult to  identify such a control group. it would be  unethical to exclude half of the  undergraduate students at uicom‐p from  the ebm training, and obtaining details of a  control group of matched students from  another medical school that did not include  ebm skills in its curriculum would be  difficult.    the authors are correct in stating that  “methods for follow‐up studies that can  draw a more direct cause and effect  relationship between the undergraduate  ebm curriculum and later practice are  needed” (52). their literature review  highlights the lack of methodologically  sound studies in this area. a cochrane  systematic review attempted to assess the  effects of teaching critical appraisal skills to  health professionals on the process of care,  patient outcomes and knowledge of health  professionals, and concluded that “the  evidence supporting all outcomes is  weakened by the generally poorly designed,  executed and reported studies that we  found” (parkes 1).    this study is of particular interest to health  librarians involved in teaching evidence  based medicine skills, and of more general  interest to all librarians who have ever been  involved in information skills training and  wondered if the material they teach is  retained over time. however, there are  significant flaws in the design of this study  which limit its usefulness.      work cited    parkes, j. c. hyde, j. deeks, and r. milne.  “teaching critical appraisal skills in  health care settings.” cochrane  database of systematic reviews 3  (2001). 29 may 2007.  <http://www.mrw.interscience.wiley.c om/cochrane/clsysrev/articles/cd00127 0/frame.html>.  http://www.mrw.interscience.wiley.com/cochrane/clsysrev/articles/cd001270/frame.html http://www.mrw.interscience.wiley.com/cochrane/clsysrev/articles/cd001270/frame.html http://www.mrw.interscience.wiley.com/cochrane/clsysrev/articles/cd001270/frame.html evidence summary   undergraduate students’ research and information skills continue to change in their second year   a review of: hulseberg, a., & twait, m. (2016). sophomores speaking: an exploratory study of student research practices. college & undergraduate libraries, 23(2), 130-150. http://dx.doi.org/10.1080/10691316.2014.981907   reviewed by: kimberly miller learning technologies librarian albert s. cook library towson university towson, maryland, united states of america email: kimberlymiller@towson.edu   received: 30 nov. 2016 accepted: 17 jan. 2017      2017 miller. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to understand sophomore undergraduate students’ research practices.   design – mixed methods online survey and participant interviews.   setting – a small liberal arts college in the midwestern united states of america.   subjects – the sample consisted of 660 second-year students; 139 students responded to the survey (21% response rate). in-depth interviews were conducted with 13 of the 139 survey respondents.   methods – a 13-item survey was emailed to sophomore students during october 2012. to analyze the results, the authors and a library student intern developed a coding scheme to apply to open-ended survey questions.   survey respondents could also volunteer for in-depth interviews. a total of 50 survey respondents volunteered, and 14 were invited for in-depth interviews between december 2012 and january 2013. the interview protocol included open-ended questions about students’ research experiences. students were also asked to identify and discuss one recent research project. interviews were audio and video recorded; data from one interview was lost due to technology failure, resulting in data analysis of 13 interviews. interview transcripts were coded by an anthropology doctoral student, the study authors, and a library student assistant.   main results – the survey found that students completed fewer research projects and used fewer library resources as sophomores than they did as first-year students. for example, only 4.9% (n=7) of students reported completing zero research assignments in their first year, compared with 34.5% (n=48) in their second year. when asked if there were library resources or skills they wanted to know about sooner in their academic career, students’ top reply was “nothing” (34.5%, n=48), followed by “navigating the physical space” (15.8%, n=22), “librarians/staff & reference desk” (11.5%, n=16), and “effective searching & evaluating sources” (10.8%, n=15). male and female students’ responses differed, with male students less likely overall to express interest in library resources. while 42.4% (n=59) of students replied that they would consult with a librarian for help with their research projects, this option ranked third after professors (83.5%, n=116) and peers (70.5%, n=98). again, responses varied by gender, with female students (49.5%, n=49) more likely than male students (26.3%, n=10) to contact a librarian about a research project.   most interview participants replied that searching online, including library resources, was their research starting point. students most often selected research topics, based on their interest, from a professor-approved list. students identified “relevant content, familiarity . . . , and credibility” (p. 138) as important source evaluation characteristics. the majority of students also used library information sources in their research, including databases, research guides, and the catalogue. students most often mentioned struggling with “finding sources/identifying keywords” (n=6) and “finding known items” (n=6). unlike survey respondents, interview participants unanimously reported consulting with a librarian. most students (n=11) received library instruction as first-year students, and some suggested that this instruction helped them feel comfortable asking for help. finally, most students felt that their research habits improved from their first year to their second year, specifically with regards to “their research technique, improved confidence . . . and an expanded source horizon” (p. 143).   conclusion – the authors recommend continuing strong information literacy support to first-year students, as well as working with faculty members and other campus partners to promote reference services to sophomores. when compared to previous research, the current study reports a higher percentage of students seeking librarian assistance; however, because some students also reported confusion about when and how to ask for help, further analysis could explore how reference librarians capitalize on peer and faculty “referral networks” (p. 145). finding that students face significant challenges early in the research process was consistent with previous research, and future study might reveal more about this specific phenomenon in sophomores. interviews should also be extended to include students who are non-library users. finally, the authors suggest that the findings provide no evidence of a “sophomore information literacy slump” (p. 146).   commentary   demonstrating how academic library work aligns with wider institutional priorities requires that librarians explore how their services connect to students’ academic experiences. professional documents, like the framework for information literacy for higher education (association of college and research libraries, 2015), also encourage librarians to support students’ long-term information literacy development. in order to make strategic decisions about library instruction programs and reference services in this environment, librarians need insight into students’ research experiences and expectations. this study contributes to our understanding of how students develop research skills over time.   as a case study, the authors acknowledge that findings may have limited external validity, and discuss how their results compare to previous research. components of the study’s design, including its use of mixed methods and potential for replication, emerge as strengths when evaluated with glynn’s (2006) critical appraisal checklist. including survey and interview instruments within the article allows librarians to repeat the investigation on their own campuses. a rich picture of sophomore students’ information behaviour emerges because of the two data collection methods. quantitative results demonstrate patterns in students’ behaviours (e.g., how often they complete research papers or the types of resources consulted), while the qualitative themes add depth to understanding thoughts, feelings, and intentions behind these behaviours.   however, because the survey was deployed during october, the data may not represent the entire sophomore experience. likewise, interviewing students during december and january does not account for the spring semester. this means study findings cannot reflect research experiences that occurred during the second half of students’ sophomore year. since all interviewees reported consulting with a librarian, it is also unclear whether the interview data accurately reflects students who do not use reference services. if students who previously benefitted from a library consultation are predisposed to volunteer for an interview about conducting research, it is worth investigating how they compare to the rest of the population on other study variables. the authors acknowledge these limitations and provide recommendations for improving upon them in future research.   while the findings suggest that students experience growth from their first to second year on-campus, there is room to expand the library’s reach. outreach and instruction librarians should continue investigating how referrals from faculty members and peers influence whether students seek out library services. librarians should explore and strengthen partnerships with other student support offices, as well as faculty members in academic departments. for example, the study authors created an outreach opportunity when they shared their research results directly with faculty members. librarians may look for similar opportunities to discuss student research experiences within their local academic communities. interview responses also suggest that first-year library instruction connects students to library services into their sophomore year. as we learn more about the association between academic library use and student learning (association of college and research libraries, 2016), librarians are trying to keep students connected to our services that correlate with academic success. this study lends support to the role strategic instruction initiatives and cultivating referrals play in this endeavor.   references   association of college and research libraries. (2016). value of academic libraries statement. retrieved from http://www.acrl.ala.org/value/wp-content/uploads/2016/07/value-of-academic-libraries-statement-final.pdf     association of college and research libraries. (2015). framework for information literacy for higher education. retrieved from http://www.ala.org/acrl/standards/ilframework      glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154       commentary   cultivating your academic online presence   shannon lucky assistant it librarian university library university of saskatchewan saskatoon, saskatchewan, canada email: shannon.lucky@usask.ca joseph e. rubin assistant professor  department of veterinary microbiology university of saskatchewan saskatoon, saskatchewan, canada email: joe.rubin@usask.ca   received: 13 jan. 2017    accepted: 13 may 2017       2017 lucky and rubin. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   introduction you have an online presence whether you acknowledge it or not. it might align positively with who you are in your professional life but, unless you have been attending to it, chances are it will disappoint you. googling your name and institution will instantly reveal what you look like to colleagues and the public online. try searching your primary research area do you or your work show up? while cultivating your online scholarly identity will take some time and attention, ignoring it could be damaging if you appear nonexistent or dramatically out of date   in your field. there are a few easy things you can do to take control of your online identity with the goal of making sure you show up where it matters, and that the information is curated and relevant to your professional identity.   the pressure on academics to produce research, teach, and engage in professional service and administrative tasks is considerable and we do not advocate dedicating hours of your week to blogging or monitoring your twitter account. we are both pre-tenure academics working in different scholarly fields and environments, but we share a common interest in how we can use online platforms to further our careers. joe is a microbiologist and assistant professor in the department of veterinary microbiology at the university of saskatchewan (u of s) where he teaches, supervises multiple graduate students, and heads a lab that focuses on antimicrobial resistance research. shannon is an assistant librarian at the u of s in library systems and information technology where she splits her time between professional practice and research into digital archiving for non-expert communities. our academic lives represent two very different modes of scholarship; joe leads a research group closely connected to his teaching portfolio, while shannon’s research is largely independent of her professional practice and is more solitary. regardless, there are benefits for all kinds of scholars in actively managing some aspects of your online identity.   you may see our advice as just one more task that sounds like a good idea, but one you really don’t have time for. we argue that if you are clear about your goals and how you will spend your time and energy, there are significant benefits for your research career. neither of us has extensive online footprints, but we are both tenure-track faculty who aspire (and have struggled) to make our online presence something that benefits our careers yet doesn’t eat up all our productive time. while we cannot speak to the best way to become a scholarly internet celebrity, in this article we will share what we have learned through the process of choosing online platforms, populating them with content, and maintaining them once the novelty wears off.   why should i worry about my online presence?   the internet and internet users will not automatically differentiate between your personal and professional activities. a google search might bring up your profile from your institution’s website next to one of your publications, next to your vacation photos on facebook. it doesn’t mean that you can’t have both personal and professional lives online. making conscious choices about how you craft your online presence can keep these areas distinct and help focus your professional audience on the content you want them to see. for some researchers, this can involve crafting distinct online identities for your personal and professional lives. for others, their research might be so tightly enmeshed with their everyday lives that having a split personality online doesn’t make sense. either way, the important thing to remember is to look at your online identity through the eyes of a future colleague, collaborator, student, or manager. the level of online exposure you choose is personal and it is worth thinking carefully about setting guidelines for yourself before expanding your digital footprint.   define your goals the most important first step is to articulate what your goals are for building your online presence. it can be as simple as wanting something accurate, professional, and concise to show up when people search for you online, or it could be as ambitious as wanting to attract collaborators for a large project. building a website or being on snapchat should not be a goal in itself think critically about what you want to get out of the time and effort you will put into this project.   when we began thinking about our own online presences we asked ourselves three essential questions: who is this for? what do i want to say? why am i doing this?   joe had been planning to create a lab website for several years and already had some goals for the project. here, he articulates how his needs as a researcher guided his planning process.                  as a researcher, i see my online presence as essential for student recruitment and defining my professional interests and research expertise. the creation of a clear and concise description of my research program and area of specialization was important for me to create to have a resource for external persons (e.g. media, prospective students, granting agencies, and others) who might be interested in contacting me.   as a junior faculty member, i remember how frustrating it was to try to learn about prospective graduate or post-doctoral supervisors from poorly maintained institutional profiles or fragmented and non-curated sources such as pubmed, google scholar, or linkedin. for prospective trainees (primarily undergraduate or graduate students), it is very important to be able to identify faculty members whose research programs are current, productive, and align with the trainee’s goals.   joe’s specific goals for curating his online presence are:   to maintain an up to date list of publications including linked articles to briefly describe current research areas to describe the research group, including current lab members and their interests to publish high-quality scientific images under a creative commons license to facilitate the development of teaching materials by other educators to demonstrate a dynamic, engaged, and positive laboratory culture by maintaining an up to date blog of lab activities   joe’s goals are clear, directly related to success criteria for his scholarly career, and in line with his professional values and interest in supporting open educational resources and celebrating the success of his lab members. in the next section, we will talk about how you choose where to begin building.   choose a platform there is no shortage of online platforms to invest your time in and share your information on. it can be overwhelming to consider every site you could end up maintaining your institutional profile, linkedin, orcid, twitter, researchgate, academia.edu, mendeley, facebook, blogs, websites, and more. many researchers, particularly early-career academics, feel pressure to engage in any and all platforms that will improve the visibility of their work. while this can be beneficial, maintaining an online presence is time-consuming and can distract from other duties. taking time to choose your platform(s) with purpose will help you realize the greatest benefit from your investment.   if building your research network or getting your work seen by others in your field is a primary goal, the best place to begin is to look at where leaders and colleagues in your field are and join in. the same advice applies to researchers who want to connect with the public or the media. take some time to research where your audience already looks for information online and go there too. it’s critical to balance your level of interest with the investment required for the platform options. if you cringe to think about spending time on twitter every day, then it’s not the choice for you, even if you fear what that looks like to your contemporaries. don’t rush to set up many different profiles to fill up that first page of google results. many sparsely populated profiles doesn’t look any better than one or two platforms that were clearly intentionally set up. map out a strategy to connect the purpose of each platform you use to one or more of your goals.   meeting joe’s five goals: the case for a personal website   if we look at joe’s five goals, most of them involve sharing information in a place where it can be easily discovered and accessed by anyone, and that allows a high level of personalization, frequent updates, and the sharing of images. for him, conversations with colleagues and making new connections typically occur via email or in person, so social networking sites like linkedin or twitter don’t address a real need. instead, we decided to focus on building a website (www.therubinlab.com) to highlight the work of joe’s research group. we chose the website creation tool wix that allows you to customize existing templates, register a custom domain, and make major changes to the site content and structure. however, the website would not address his fourth goal to publish creative commons licensed microbiological images for teaching. to address this gap, we decided to invest in a profile on the photo sharing website flickr because it allows him to create a profile for the rubin lab (https://www.flickr.com/photos/therubinlab/), upload descriptive metadata, easily apply a creative commons licence, and track how many times his images have been accessed. uploading images to flickr also made them easier to embed in his website. this was an ambitious plan to implement, but one that could be done in stages and continue to be built up over time. creating a website was time-consuming, but it was the right choice to meet these goals and the balance between cost and reward was worth it for joe. a scalable alternative to an entire website is to create a landing page or researcher profile.   profile landing page a landing page is simply a single-page website that lists contact information, a brief description of the author and their research interests, and may link to their institution or other sites. shannon currently maintains a landing page at www.shannonlucky.com/ that lists her current position and research interests, contact information, and links to her academic cv, orcid account, google scholar profile, profile on the u of s website, linkedin, and twitter. this design rarely needs to be updated and creates a centralized hub for her professional information, making it clear that these are the platforms to look for her on. this still requires some customization of a website building tool, but you may be able to achieve the same result using an institutionally supported researcher profile.   researcher profiles   researcher profiles are becoming more common at post-secondary institutions. if your institution offers them, they are a great tool to build your online identity. researcher profiles are typically web pages that list your contact information, essential cv information (grants, publications, awards), and descriptions of your research and scholarly experience. they may link to your publications and scholarly output held in an institutional repository as well. researcher profiles are usually part of the institution’s website and their connection to an official, highly trafficked website translates to a higher ranking in google search results. depending on the level of control you have over your personal profile, you may be able to link to your other online profiles, boosting their visibility in search results as well.   while an institutional researcher profile is a great resource, there are reasons why you might want to create an independent website or profile. researcher profiles and institutional websites tend to have limited flexibility of both content and style. additionally, these pages belong to the institution. if you leave your position, that profile will likely disappear. maintaining a separate website maximizes both portability and the level of personal control over the content you share.   create great content   now that you have decided where you will cultivate your online presence, you need to populate that space with information. high-quality content is critical for the development of an online presence that will effectively meet your goals. fortunately, in joe’s case, he had been planning to put together a website for several years and was diligent about taking interesting photos of his work, and had lay summaries of his research on hand. you may have already done some of this writing and not realize it. grant applications, research summaries, abstracts, cv updates, and tenure and promotion packages are great sources to work from. remember to modify text borrowed from these sources for a general audience; look for information on writing for the web, which is very different than for an academic audience.    creating quality text and images will go a long way to making your online presence useful, engaging, and authoritative. start by deciding what ideas you want to share, and then map out how you want to present it online. take inspiration from other websites and profiles. replicating the same types of pages (contact, publications, projects) or layouts you find useful on other researchers’ sites is a good way to anticipate what is important to include, and what may not be worth your time to maintain. don’t get hung up on a particular platform or stylistic elements. if something you try isn’t working, don’t be afraid to change direction.   while developing your online presence, consider the shareability of the content you create. most online sharing uses social media, and designing your message to be easily contextualized in a few sentences, an image or infographic, or a short video dramatically increases the odds that someone will tweet it, share it on facebook, or otherwise pass it along to their online community.   how do i know this is worth it? remember to periodically return to your goals and critically assess whether what you are doing is working for you.  for joe, the creative process of writing content for his site ended up being beneficial in itself, becoming a generative process to define his program of research. now he has a place where his entire research enterprise is summarized with writing and visual/multimedia elements creating something more holistic, illustrative, and engaging than a traditional cv. the website and flickr account now represent the breadth of the things he does in efficient and creative ways, and he can see how it has helped define and demonstrate the research culture of his lab. in addition to these goals being met, he describes some of the objective results he has seen.    in the first four months of my website, i have already reaped some rewards from this project. using google analytics, i have been able to track views and see where visitors are coming from. my website has already had over 1,400 views from more than 850 unique users originating from 34 countries. i was particularly interested to see that 36 unique users were referred to my website from my institutional profile, suggesting that people were looking for more information than was provided on my department’s site. having a description of my research program online led to an invitation to speak at an international conference. in this case, the conference organizer was referred to my website by a colleague, but the information he found there helped him decide that my research would be a good fit for the program.  the site has also facilitated information sharing with my colleagues. requests for teaching materials, project abstracts, or my academic bio can now be referred to the website, saving time composing new text or hunting for the information on my computer.   finally, i’ve found that blog posts on my website are easily shared on social media. between my department’s facebook page, and those of my lab members and students, my website has acquired almost 600 page views. notably, one post described a bonus assignment i gave to an undergraduate class asking them to rewrite lyrics for a song using information they learned in my course. i linked to youtube videos of their performances and it attracted hundreds of views through facebook. this post was also seen by a local radio host, leading to a broadcast interview on cbc radio and the opportunity to share some fun teaching moments with the public.   joe has already experienced some concrete outcomes from investing time in his online presence. not everyone will have the time to engage so deeply in developing a fully realized website, but the reward is likely to be commensurate with effort. for an early-career researcher, making your information easy to find and professional in appearance may create opportunities you could not have predicted. keep it current once you have your strategy defined and your online presence set up and working for you, the work is not done. your research is always progressing and so should your online presence. it is critical to regularly revisit your online profiles and sites to keep them up to date. keeping a list of all of your profiles is the first step, then schedule time into your calendar to review them. you can use tools like a google scholar profile to automate some things like a list of your publications, but any site more complex than the most basic landing page requires some attention from time to time.   future directions the process of developing our respective online profiles as researchers has been a learning process and we have had to negotiate how much we are willing to invest. joe has had great success taking the time to keep his site up to date with content that is accessible and interesting to a range of audiences. more importantly, he enjoys working on the site and directly benefits from those efforts. our experiences have been entirely anecdotal and our advice comes from our own conversations, trial, and error. in the future, we would like to take an evidence based approach to understanding the value that maintaining an online presence has for scholarly researchers, why academics do or do not choose to use online platforms, and if institutions can support researcher success by providing infrastructure or services for developing a strong online presence.       evidence summary   health sciences patrons use electronic books more than print books   a review of: li, j. (2016). is it cost-effective to purchase print books when the equivalent e-book is available? journal of hospital librarianship, 16(1), 40-48. http://dx.doi.org/10.1080/15323269.2016.1118288   reviewed by: robin e. miller associate professor and research & instruction librarian mcintyre library university of wisconsin-eau claire eau claire, wisconsin, united states of america  email: millerob@uwec.edu   received: 22 may 2017  accepted: 11 july 2017      2017 miller. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to compare use of books held simultaneously in print and electronic formats.   design – case study.   setting – a health sciences library at a public comprehensive university with a medical college in the southern united states of america.   subjects – usage data for 60 books held by the library simultaneously in print and electronically. the titles were on standing order in print and considered “core” texts for clinical, instructional, or reference for health sciences faculty, students, and medical residents.   methods – researchers collected usage data for 60 print titles from the integrated library system and compared the data to counter reports for electronic versions of the same titles, for the period spanning 2010-2014.   main results – overall, the 60 e-book titles were used more than the print versions, with the electronic versions used a total of 370,695 times while the print versions were used 93 times during the time period being examined.   conclusion – the use of electronic books outnumbers the use of print books of the same title.   commentary   the title of the article suggests that the author may offer a method for determining whether it is “cost-effective” to purchase both the print and electronic versions of the same book title. the sole method employed is to compare raw usage statistics between print and electronic versions of the same 60 titles. the author does not discuss other factors that may influence “cost-effectiveness,” including cost of print titles versus cost to license electronic titles, which is often substantially greater. the article does not include a discussion of other aspects of e-book costs, including platform fees, total cost of vendor-selected e-book packages, or terms and conditions that limit simultaneous use or that require repurchase of an electronic title after a defined number of uses.   comparing the use of the electronic and print versions of a collection of books may demonstrate that patrons of this health sciences library prefer electronic formats as compared to print. however, the researcher does not define “use” for electronic or print content. electronic use may constitute full downloads, chapter downloads, printing, online reading, or another form of use often articulated by vendor usage reports. the article also provides no evidence that comparing electronic usage and print usage is a valid method. the reported statistics do not control for variables that may influence the user’s choice to access e-books, such as instant access to the full-text in a discovery layer, direct links to electronic content in syllabi, or library location.   computational errors compromise the conclusions of this article. the author states, “[t]otal electronic book usage was 99.7% versus 0.03% print book usage” (p. 44), though these percentages do not add up to 100% and what they relate to is not stated clearly. the article reports 93 uses of print titles and 370,695 uses of e-book titles. total use of the print and electronic subjects equals 370,788; however, 0.03% of 370,788 does not equal 93 and 99.7% does not equal 370,695.   the author argues that libraries must respond to the rise of devices like kindle and e-book applications offered by various publishers and database vendors, which have rendered e-books more attractive to library users. library practitioners may agree with this argument; however, this article does not provide evidence to support that assertion because the results do not delineate e-book usage by mode or device of access.   for the collection development practitioner, this study may offer insight about whether it is necessary to provide print and electronic access to material in core instructional or reference collections. however, the author compared usage of 60 titles that have clearly been identified by librarians and faculty as valuable to the patron population. in contrast, 250,000 electronic books were available in the author’s library at the time of publication. the results of this study would be more compelling if the research included a larger sample of titles held in both formats, and if the analysis also incorporated other factors that librarians typically use to make collection development decisions, particularly cost.   research article   it takes a researcher to know a researcher: academic librarian perspectives regarding skills and training for research data support in canada   alisa b. rod research data management specialist mcgill university montreal, quebec, canada email: alisa.rod@mcgill.ca   received: 4 jan. 2023                                                                      accepted: 7 feb. 2023      2023 rod. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30297     abstract   objective – this empirical study aims to contribute qualitative evidence on the perspectives of data-related librarians regarding the necessary skills, education, and training for these roles in the context of canadian academic libraries. a second aim of this study is to understand the perspectives of data-related librarians regarding the specific role of the mlis in providing relevant training and education. the definition of a data-related librarian in this study includes any librarian or professional who has a conventional title related to a field of data librarianship (i.e., research data management, data services, gis, data visualization, data science) or any other librarian or professional whose duties include providing data-related services within an academic institution.   methods – this study incorporates in-depth qualitative empirical evidence in the form of 12 semi-structured interviews of data-related librarians to investigate first-hand perspectives on the necessary skills required for such positions and the mechanisms for acquiring and maintaining such skills.   results – the interviews identified four major themes related to the skills required for library-related data services positions, including the perceived importance of experience conducting original research, proficiency in computational coding and quantitative methods, mlis-related skills such as understanding metadata, and the ability to learn new skills quickly on the job. overall, the implication of this study regarding the training from mlis programs concerning data-related librarianship is that although expertise in metadata, documentation, and information management are vital skills for data-related librarians, the mlis is increasingly less competitive compared with degree programs that offer a greater emphasis on practical experience working with different types of data in a research context and implementing a variety of methodological approaches.   conclusion – this study demonstrates that an in-depth qualitative portrait of data-related librarians within a national academic ecosystem provides valuable new insights regarding the perceived importance of conducting original empirical research to succeed in these roles.     introduction   in the context of academic libraries, there has been increasing demand from researchers for data-related services over the past ten years (barsky, 2019; barsky et al., 2017; cox et al., 2017; cox et al., 2019a; steeleworthy, 2014). this shift is projected to continue a growth path, which has implications for increased capacity and infrastructure needs at all levels of service provision (briney et al., 2015; khair et al., 2020). academic libraries have responded to increasing demand for research data support by developing capacity in this regard and positioning the library as the appropriate centralized support resource (ashiq & warraich, 2022).   in this way, academic libraries have spearheaded service models for the provision of data-related research services (radecki & springer, 2020). in the academic library context more generally, new positions focused on data have emerged over the past ten years, including positions focusing on research data services (rds), data curation, and research data management (rdm). rdm, while closely related to rds, is more focused on organizing and managing research data over a lifecycle rather than providing reference support related to finding sources for data or accessing data held in controlled collections, which is more the purview of rds. obscuring the notion of a clearcut definition of these roles is the lack of standardized titles and the conflation of research data support with liaison duties at many institutions. for example, a recent study by theilen and neeser (2020) identified 119 discrete job titles for academic library job postings, of which the majority did not include a reference to librarianship, although the most frequently occurring librarian job titles reflect the four most common roles, including data services librarian, data curation librarian, research data management librarian, and data librarian. thus, for the purposes of this study, the broader term “data-related librarian” is used to represent any librarians or professionals who offer support for research data within an academic context.   there are several additional studies that have investigated the competencies, skills, and educational background required in job advertisements for these and related roles in academic libraries both in north american and globally (acrl research planning and review committee, 2020; cox et al., 2017; federer, 2018; fuhr, 2019; goben & sapp-nelson, 2018a; goben & sapp-nelson, 2018b; thielen & neeser, 2020; xia & wang, 2017). however, it is unclear as to whether job postings are indicative of the profiles of individuals who are performing these roles. in addition, there are also existing studies that have investigated the self-perceptions of librarians regarding technical readiness and confidence in assuming these roles (chiware, 2020; ducas et al., 2020; joo & schmidt, 2021; tang & hu, 2019; thomas & urban, 2018). however, these studies typically rely on survey methodology, which is useful in identifying broad trends but less capable of identifying nuance or providing entirely novel insights.   aims   this study aims to contribute a qualitative analysis of the perspectives of data-related librarians regarding the necessary skills, experience, and training for these roles. this study is organized around the following two research questions:   rq1: what are data-related librarians’ perspectives regarding the skills required to work effectively as academic data professionals?   rq2: what are data-related librarians’ perspectives regarding the role of the mlis in providing relevant training and education to prepare academic data professionals?   this study incorporates in-depth qualitative empirical evidence in the form of 12 semi-structured interviews of data-related librarians to investigate first-hand perspectives on the necessary skills and expertise required for such positions and the mechanisms for acquiring and maintaining such skills.   literature review   over the past two decades, technological advances and institutional investments related to digital research infrastructures have rendered the collection of larger and larger quantities of data as ubiquitous to many academic disciplines and research processes (briney et al., 2015). thus, research communities and public funding agencies have articulated and socialized best practices for handling data throughout a research projects’ lifecycle, culminating with the development and adoption of the fair principles as the primary guiding framework for research data (force11, 2014; tang & hu, 2019; wilkenson et al., 2016). fair refers to data that are findable, accessible, interoperable, and re-usable, which may be implemented primarily through infrastructure, such as digital data repositories, designed for enhancing the discoverability, preservation, and sharing of research data. following a consultation period lasting almost 10 years, the three major funding agencies in canada, the tri-agencies, recently published a policy on rdm highlighting the fair principles and focusing on the proper handling of data as a matter of integrity and ethics (government of canada, 2021). these large-scale shifts in norms and policies governing accountability and reproducibility regarding research data have necessitated parallel shifts among higher education institutions related to the provision of support and the investment in staffing capacity as a crucial pillar of a modern academic digital research infrastructure program (baxter et al., 2021; federer et al., 2020).   in most cases, the library is the campus unit that has been designated with a mandate to provide data-related support services, training, and guidance (radecki & springer, 2020). indeed, the library is the natural home for data-related digital research infrastructure management as this emerging function intersects with other key contemporary academic library roles, including scholarly communications, digital collections more generally, digital preservation, and digital scholarship (acrl research planning and review committee, 2020; steeleworthy, 2014). in addition, the liaison model provides a support system that is embedded within conventional research processes and offers a communication channel by which to facilitate awareness and outreach campaigns (steeleworthy, 2014). thus, academic libraries have spent the past decade investing in capacity-building in terms of technical infrastructure and highly qualified personnel (hqp) in the form of data-related librarian positions or units such as rds that incorporate several data-related librarian positions.   several previous studies have mapped both the change in academic library positions regarding data services and the skills required for these and adjacent positions over the past decade (acrl research planning and review committee, 2020; goben & sapp-nelson, 2018a; goben & sapp-nelson, 2018b; tenopir et al., 2015; tenopir et al., 2019; thielen & neeser, 2020). in the past few years, a shift has occurred regarding the educational requirements for data-related librarian positions (chen & zhang, 2017; theilen & neeser, 2020). a study by chen and zhang (2017) found that fewer than 30% of data management academic library job postings between january and april 2015 required the mlis and that an alternative relevant advanced degree would be accepted. a more recent study by theilen and neeser (2020) reviewed postings between 2013 and 2018 for rds positions within academic libraries. in general, theilen and neeser (2020) claim that the requirements and preferred qualifications for data professional job advertisements favors the education and experience of candidates with a non-lis background. a recent study by fuhr (2022) also found that hiring for early career rds librarians is trending toward candidates who do not hold an mlis. chen and zhang (2017) similarly speculated that lis programs may not be preparing students with expertise in data management or that the relevance of lis curriculum to data management professions, even in the context of academic libraries, is becoming less apparent. in general, continuing education programs and professional development offerings have emerged to supplement formal lis education (davis & cross, 2015; read et al., 2019).   in addition to educational requirements, several studies have also focused on the skills and/or competencies required for data-related librarian positions. a recent study by federer (2018) detailed 47 skills and competencies organized according to a taxonomy of nine categories. the overarching categories of skills, knowledge, and competencies required for data librarianship, as identified by federer (2018), included data management, technology and information technology, evaluation and assessment, teaching and instruction, marketing and outreach, library skills, professional involvement, skills and personal attributes, and education and training. many of these areas of skills and competencies overlap with other functional or subject-specific librarian roles. the data-specific skills, knowledge, and competencies cover a range of applications of technological and functional abilities and training, including data visualization, scientific programming, geographic information systems (gis) data and programs, discipline-specific data management skills, and knowledge of best practices regarding data sources, finding data, sharing data, curating data, and developing service models for data support.   in a more recent review, fuhr (2019) synthesizes the literature in terms of suggested or required skills for rdm-related information professionals, finding that previous research identified skills related to the full research data lifecycle, including planning for the collection and use of data, actively managing data in terms of storage and security, documenting data and incorporating relevant metadata at all phases of a research project, and planning for the dissemination or archiving of data in the long-term. in addition, several studies also identify “soft skills” such as effective communication, team-oriented professionalism, and relationship building to be cited across job postings or identified by practitioners (chen & zhang, 2017; federer, 2018). other research has also found an increasing emphasis on familiarity or expertise in a programming language or statistical software, gis, or data visualization (fuhr, 2022).   in this way, the technological and functionally specific skills required to acquire and perform a data-related role within an academic library context may not map onto the conventional lis curriculum. indeed, several recent studies claim that these competencies are not easily acquired through lis curricula (chen & zhang, 2017; si et al., 2013; thielen & neeser, 2020). as rds was emerging as a support area within academic libraries, a handful of mlis-granting institutions developed pathways or certificates in data curation or related topics embedded within the broader lis curriculum (corrall, 2012). currently, courses in rds, rdm, and related topics are still not widely integrated within lis programs, although a systematic review of curricula is lacking (chen & zhang, 2017). however, a recent study by wang and lin (2019) provided an empirical review of the 2018-2019 academic year course offerings of 48 ischools across the us and found that only 35% of programs offered any courses related to rds, of which fewer than 10% of the total course offerings were related to rds.   there are also several recent studies incorporating methodologies to directly investigate the self-perceptions of rds and rdm librarians regarding the skills and qualifications necessary to perform their roles (chiware, 2020; ducas et al., 2020; joo & schmidt, 2021; tang & hu, 2019; thomas & urban, 2018). thomas and urban (2018) surveyed 105 rds professionals and asked directly for participants to indicate the extent to which their mlis degree program prepared them for their position. the primary takeaway from the thomas and urban (2018) study is that rds librarians learn about topics that are less conventionally covered in lis curricula, such as data management, through hands on experience and thus perceive mlis degree programs to be ill-equipped at preparing professionals for data-intensive librarian roles. a more recent study by ducas et al. (2020) surveyed 205 librarians in canada about a range of emerging roles for academic libraries including data management and data curation. ducas et al. (2020) found that most participants reported needing additional training in several emerging librarian job functions, including data curation, statistical methods and programs, and data management.   although several recent studies have surveyed librarians in rds positions, most studies on the background and skills required for data-related librarian positions tend to focus on job postings rather than individuals hired into the positions. although surveys do provide some insight into the disconnect between educational pathways, job postings, and the actual demands of job duties related to research data, they typically rely on sample sizes that are not adequate for representing larger populations. in general, there has not been an in-depth investigation into the perceptions of information professionals regarding the necessary skills to perform these data-related librarian roles, broadly defined, in the canadian context, which relies on a nationally integrated infrastructure and support network including federal stakeholder organizations, such as borealis, the digital research alliance of canada (the alliance), and the canadian research knowledge network (ducas et al., 2020). thus, a qualitative in-depth investigation of individuals holding data-related librarian positions in the context of canadian academic libraries would be a major contribution to this body of literature.   methodology   the data collection for this study involved conducting 12 semi-structured interviews with data-related librarians at canadian higher education academic institutions. the interview research described in this article was approved by mcgill university’s research ethics board (reb file #: 22-01-077). one major methodological challenge of this study was determining an operationalization for the construct of data-related librarian. previous studies found that data-related librarians have a range of unique titles. thus, the operationalization of data-related librarian includes any librarian or professional who has a conventional title related to a field of data librarianship (i.e., research data management, data services, gis, data visualization, data science) or any other librarian or professional whose duties align with librarian job descriptions of data-related services within an academic institution (springer, 2019). i randomly selected 13 individuals from a list of 253 data-related librarians and professionals compiled in accordance with this conceptualization (rod, 2022).   i chose to use random selection to minimize the chances of participant re-identification and to be able to provide participants, if requested, with the precise statistical likelihood of their selection. i intended to sample between 10 and 15 in stages in anticipation that i would need to send multiple batches of invitations. however, 12 of the 13 invited participants from the initial sample agreed to the interview. the interview participants represent a range of roles (see table 1 for proportions of titles of interview participants and the appendix for examples of specific job titles for each aggregated category). in addition, the interview participants were located across several provinces, including ontario, québec, one prairie province, and one atlantic province.     table 1 aggregated titles of interview participants aggregated titles frequency percent of total title categories data or gis librarian 4 33% advisors or directors 2 16% liaison librarian 3 25% rdm librarian or specialist 3 25% total 12 100%     the 30-minute interviews were booked via microsoft bookings and took place virtually via microsoft teams in the summer of 2022, between june 22 and july 28. the interview questions focused on participants’ experience, education, and their perspectives on relevant training and expertise required for these positions. for a full list of interview questions and codebook, see the deposited dataset (rod, 2023). interviews were audio recorded and auto-transcribed in microsoft word. i reviewed and manually corrected each interview transcription. overall, the auto-transcriptions were fairly accurate.   the interview transcripts were analyzed using qualitative content analysis, which is an established qualitative methodology for summarizing a series of observations drawn from unstructured data either by human coding or using a text analysis software or program (bernard et al., 2016). qualitative content analysis is an iterative process in which one or more coders read through the textual data, usually more than one time to reconcile disagreements and to create and refine a codebook (creswell, 1994). once a codebook is developed, the coder(s) re-read the text data and apply the codes to specific words, sentences, or paragraphs. the unit of analysis for this study is the individual participant, meaning that i am comparing across participants. thus, after collecting all references to each code within a single interview transcript, i aggregated unique codes by each interview.   following the completion of the interviews, i read a sample of 2 full transcripts to develop an initial codebook. i then read 5 different interview transcripts to adjust and refine the codebook. finally, i read all interview transcripts, including those that had been previously analyzed, to develop a final version of the codebook. once a finalized codebook was established, i re-read and coded all interview transcripts using taguette, a desktop application for tagging textual data with codes.   these interviews were conducted as part of a larger project on the skills, education, and the responsibilities of data-related librarians in the canadian context (rod, 2022). the results reported in the following section reflect an analysis of a portion of the information collected from the interviews that focused on data-related librarians’ perspectives of relevant skills and experience for these roles.   results   to address the first research question, interview participants were asked to reflect on the most essential skills required for roles or positions like theirs at academic libraries. to maintain participants’ privacy, pseudonyms are employed to describe results. interestingly, only two participants indicated that their job duties are clearly defined. most interview participants indicated that they are, to varying degrees, unclear where the boundaries are between their job and other data-related roles at their library. as summarized by victoria, an rdm librarian or specialist at a large research university:   what is the difference between the [number] of us? we have no idea. it's never been defined or clarified with us and no matter how many times we ask we never get an answer, so i couldn't tell you what technically the differences are between our roles and what we're supposed to do, and not supposed to do other than what our titles are.   relatedly, one-third of the interview participants indicated that their data-related duties are done “off the side of their desk” and are split between another position type, such as liaison/subject librarian. in terms of the specific tasks that are associated with library-related data services positions, most interview participants mentioned that they are not an rdm librarian, but that they are responsible for rdm-related tasks as part of their job. the most reported job tasks across all interview participants included data-related reference with individuals, teaching/instruction quantitative or numerical data analysis (e.g., with excel, r, python, etc.), facilitating data sharing and deposit for researchers, and consulting or training on writing data management plans (dmps). interestingly, many participants are also involved in rdm-related initiatives at the strategic planning level of their institution. this is due to the recent tri-agency rdm policy, which requires institutions administering public research funds in canada to develop and share an institutional rdm strategy by march 1, 2023 (government of canada, 2021). less commonly, interview participants mentioned that, as part of their job, they are involved as principal investigator (pi) or related staff at the project-level of sponsored research. in addition, fewer than one-third of participants indicated that they perform tasks related to text analysis, outreach, and/or qualitative analysis.   overall, the findings of the interviews identified four major themes related to the skills required for library-related data services positions, including:   ·         experience conducting original research; ·         proficiency in computational coding and quantitative methods; ·         mlis-related skills such as understanding metadata, documentation, preservation, and curation; ·         the ability to learn new skills quickly on the job.   first, the interviews highlighted the perceived importance of conducting original research. ten of the twelve interview participants explicitly discussed the importance of conducting original research as a qualification for a data-related librarian position. across these ten interview participants, several participants noted that conducting research helps to bridge the gap between researchers/faculty and librarians/library-related data services staff. as anna, a data or gis librarian at a large research university summarized “if you've done some research in the past, i think that's extremely helpful. if you've analyzed data in some way, [if] you've written a paper, then you've done something that's going to help you in this job.”   relatedly, the second theme identified across the interviews is proficiency in computational methods (e.g., coding), quantitative methods, and data literacy. most interview participants discussed the importance of working with various types of data, various quantitative methodologies, and having an advanced understanding of the research data lifecycle. more than half of the interview participants mentioned the importance of technical skills related to different types of data and familiarity with a wide range of data types and tools.   the third theme that emerged across most interview participants relates to skills that are developed through training in mlis programs and coursework. participants consistently indicated that skills and expertise acquired through the standard mlis curriculum, such as understanding metadata, documentation, preservation, and curation, are the bedrock of succeeding at data-related librarian positions. for example, lily, a data or gis librarian, discussed the value of having expertise in thinking critically about empirical information divorced of disciplinary biases and argued that acquiring this expertise is unique to mlis programs.   the fourth theme that emerged across interview participants relates to the importance of the meta-skill of acquiring new skills and expertise quickly on the job. interestingly, interview participants weighed the soft skill of gaining new expertise quickly as relatively equally important as hard skills such as technical proficiency in data analysis programs. for example, claudia, an rdm librarian or specialist, discussed a project where they were asked to provide advice on data sharing for a medical research project. although they were not familiar with medical research, they were able to learn quickly and apply research principles that are cross-cutting. other participants defined learning quickly on the job in terms of the importance of understanding the high-level principles of coding and data analysis so that when trends regarding specific tools (e.g., spss, r, python, etc.) change, it is easier to adapt and evolve. jake, a liaison librarian with data responsibilities, provided an analogy to explain that data-related librarian positions are not inherently or necessarily different from other types of librarian roles:   i'm going to pick on [law librarians] specifically 'cause i know they tend to have this idea of: “no, you should not be a law librarian unless you are a lawyer, and this is very much a world that nobody else can ever understand unless you have this training.” i go in the exact opposite direction. you could come into data librarianship from anywhere and really, it's more about having the fluency of knowing how to understand different ways of talking about things.   indeed, five interview participants expressed the view that having specific skills and expertise, including the ability to learn new skills quickly, is more important than any specific disciplinary background, work experience, or formal academic training.   to address the second research question regarding the importance of the mlis in providing relevant training and education for these positions, interview participants were asked about their educational background and previous work experience. seven of the interview participants mentioned that their work experience as a librarian translated directly into their acquisition of relevant skills and expertise. these seven participants indicated that they had worked as data-related librarians in a previous position(s) or have already been working in the same data-related position for a significant number of years. in addition, half of the participants explicitly mentioned that obtaining an mlis degree facilitated the acquisition of many relevant skills and expertise.   when asked to reflect on the ideal educational background for information professionals working in data-related librarian positions, nine of the interview participants discussed the importance of the mlis degree. overall, the interview participants were highly supportive of the mlis, while acknowledging that this perspective is self-serving. however, another consistent theme that emerged across the participants was that the mlis does not incorporate enough data-specific courses to facilitate gaining advanced proficiency in the relevant skills required for data-related librarians. as articulated by lily, one issue identified is that:   people don't understand what librarians do and there's also just a huge variety of things that librarians do, so i think people sometimes wonder…why couldn't someone with a phd in that field do that better than you? my response to that is… just because someone has, for example, like in my field in [stem], someone who has a [stem] phd isn't necessarily going to know how databases work within the sciences…or how to manage that data in ways that preserve that data for the future.   on the other hand, three participants argued that a master’s degree in a quantitative field, or a phd, should substitute or augment an mlis specifically for data-related librarian positions. as claudia argued, “in my opinion it's not possible to respond to faculty needs without an advanced degree at the level of a phd.” in this way, the interviews offer views of how information professionals working in data-related librarian positions in canada have acquired the relevant skills that are mostly reflective of the interviewee’s own experiences. however, the interview participants were generally still divided in terms of their perspectives regarding the relevance of specific educational backgrounds or prior work experience.   discussion   the findings of this qualitative study reinforce several emerging trends identified within recent literature. for example, recent research has echoed the finding that there is increasingly less of an emphasis on the mlis and that data-related librarians offer higher levels of technical assistance relative to the state of the field ten years ago (fuhr, 2022; plassche, 2022). overall, the themes that emerged from the interviews provides evidence that there are at least three key implications for data-related librarianship training and skills acquisition.   first, interview participants identified the importance of data-related librarians to have first-hand experience conducting empirical research. several participants argued that research experience should be a requirement for data-related librarian positions compared with liaison/subject positions. the overall rationale for this perspective is that academic researchers (e.g., non-librarian faculty members, professors, etc.), whether warranted or not, do not respect the advice of colleagues who do not “understand” the full research project lifecycle and all that it entails (e.g., writing grants and data management plans, collecting data in compliance with institutional and national privacy policies, storing data, sharing data among collaborators, analyzing empirical data, and writing up findings based on their analysis). according to this perspective, the only way to gain this understanding is to go through the process of conducting research. this perspective in many ways is antithetical to the discipline and conventional professional practice of librarianship, where meta-knowledge of the organization of information can be applied across domains without necessarily having experience in each domain or in-depth disciplinary expertise.   of course, several participants argued that it is possible to gain this experience through an mlis program, either through conducting a short original research project within the context of a course or through a multi-term thesis or project. however, other participants argued that professors and researchers who have gone through the process of acquiring doctoral degrees are biased regarding the importance of conducting original research at that level. in addition, participants also argued that the methodological and research training in mlis programs are insufficient and are not often required as part of a core curriculum. as roberta, an advisor or director who previously worked as a librarian supporting rdm and rds services, reflected,   if they said, “you need a phd or you need a background exactly in data science or something like that,” then i would never have gotten this position. maybe that's okay too. maybe they would have gotten someone who has all of that and it would be even better for the community.   a second key finding of this study is that although there is empirical evidence, in addition to the findings from other recent studies, which depicts a general slight decline in the requirement of an mlis for data-related librarian positions, data-related librarians in canada generally still view the mlis as valuable. this is consistent with a recent study by plassche (2022) finding that map and geospatial library positions are increasingly requiring domain-specific degrees (e.g., in geography, gis, or a related field) and technical backgrounds, but still strongly favor an mlis. interview participants were highly supportive of the mlis even while acknowledging that the ability to pursue data-related courses within mlis programs greatly varies and curricula are generally currently misaligned to the actual demands of data-related librarian positions. overall, the implication of this study regarding the training from mlis programs for data-related librarianship is that although expertise in metadata, documentation, and information management are vital skills for data-related librarians, the mlis is increasingly less competitive compared with degree programs that offer a greater emphasis on practical experience working with different types of data in a research context and implementing a variety of methodological approaches (e.g., domain-specific empirical master’s degrees or doctoral degrees).   a final key finding from this study aligns with previous research identifying the particular importance of soft skills (e.g., communication, learning quickly, networking, leadership, etc.) for data-related librarian positions (chen & zhang, 2017; harp & ogborn, 2019; federer, 2018). interview participants in this study generally reported that learning quickly on the job is a top skill for data-related librarians, given that educational pathways to these positions may be diverse and the landscape defining vital technical skills changes rapidly.   in addition, as reinforced by cox et al. (2019b), the rise of funder mandates requiring better data management across disciplines has carved a role for data-related librarians regarding strategic planning at the institutional level. for example, in 2021, the three major public funding agencies in canada, or the tri-agency, released a harmonized policy on rdm for which one requirement is that higher education institutions that administer tri-agency funds develop and publicly post an institutional rdm strategy by march 1, 2023. the development of these institutional rdm strategies has facilitated cross-functional collaborations and discussions among research administration, the library, it departments, and other campus units relevant to the management of research data. in this way, data-related librarians in canada have increased opportunities to employ networking skills in developing and formalizing relationships with various campus stakeholders (see also pinfield et al., 2014; harp & ogborn, 2019).   a key limitation of this study is that it is focused on canadian data-related librarians, and thus may not reflect the perspectives of data-related librarians in other geographic communities. future research could expand this methodology to conduct qualitative research across a variety of geographic regions and contexts. a second limitation involves reliance on a single coder, thus potentially introducing measurement bias. however, this is mitigated by the subject and methodological expertise and experience of the coder, in addition to the number of iterations of coding, and the qualitative nature of the study, which is not inherently designed to maximize replicability, but rather to uncover new information that would not be easily surfaced using other methods or approaches.    conclusion   this study demonstrates that an in-depth qualitative portrait of data-related librarians within a national academic ecosystem provides valuable new insights regarding the perceived importance of the specific training and skills required to succeed in these roles. the interviews identified four major themes related to the skills required for library-related data services positions in canada, including the perceived importance of experience conducting original research, proficiency in computational coding and quantitative methods, mlis-related skills such as understanding metadata, and the ability to learn new skills quickly on the job. overall, the implication of this study regarding the training from mlis programs regarding data-related librarianship is that although expertise in metadata, documentation, and information management are vital skills for data-related librarians, the mlis is increasingly less competitive compared with more technical (e.g., programs in stem fields) or research-oriented degrees (e.g., empirical phd programs). a potential future line of inquiry could involve investigating the opportunities that exist within mlis programs for conducting original research and what those research projects or course components entail.   in general, this study contributes to the literature on data-related librarianship in terms of providing new qualitative evidence on perspectives regarding the skills, education, and training required for these roles. in addition, this study provides a valuable methodological approach for conceptualizing and operationalizing data-related librarians according to inclusive parameters (e.g., operationalizing data librarian to include any librarian with responsibilities related to rdm, rds, data visualization, data science, gis, etc. in a single analysis) and can be applied to other contexts or geographical regions.   acknowledgement   this manuscript draws on data collected as part of a master's degree research project for the mcgill university master of information studies program. the author wishes to thank her project supervisor, dr. rebekah willson, for providing valuable feedback on previous versions of this manuscript.   references   acrl research planning and review committee. (2020). 2020 top trends in academic libraries: a review of the trends and issues affecting academic libraries in higher education. college & research libraries news, 81(6), 270-278. https://doi.org/10.5860/crln.81.6.270   ashiq, m., & warraich, n. f. (2022). a systematized review on data librarianship literature: current services, challenges, skills, and motivational factors. journal of librarianship and information science. advance online publication. https://doi.org/10.1177/09610006221083675   barsky, e. (2019, june 20). research data management: a view from canada [presentation]. http://dx.doi.org/10.14288/1.0379514   barsky, e., laliberté l., leahey, a., and trimble, l. (2017). collaborative research data curation services: a view from canada. in l. r. johnston (ed.), curating research data, volume one: practical strategies for your digital repository (pp. 79-101). association of college and research libraries.   baxter, g., beard, l., beattie, g., blake, m., greenhall, m., lingstadt, k., nixon, w., & reimer, t. (2021). covid-19 and the future of the digital shift amongst research libraries: an rluk perspective in context. new review of academic librarianship. advance online publication. https://doi.org/10.1080/13614533.2021.1976232   bernard, h. r., wutich, a., & ryan, g. w. (2016). analyzing qualitative data: systematic approaches. sage publications.   briney, k. & goben, a. & zilinski, l. (2015). do you have an institutional data policy? a review of the current landscape of library data services and institutional data policies. journal of librarianship and scholarly communication 3(2),  ep1232. https://doi.org/10.7710/2162-3309.1232   chen, h. l., & zhang, y. (2017). educating data management professionals: a content analysis of job descriptions. the journal of academic librarianship, 43(1), 18-24. https://doi.org/10.1016/j.acalib.2016.11.002   chiware, e. r. t. (2020). data librarianship in south african academic and research libraries: a survey. library management, 41(6/7), 401-416. https://doi.org/10.1108/lm-03-2020-0045   corrall, s. (2012). roles and responsibilities: libraries, librarians, and data. in g. pryor (ed.), managing research data. (pp. 105-133). facet.   cox, a. m., kennan, m. a., lyon, l., & pinfield, s. (2017). developments in research data management in academic libraries: towards an understanding of research data service maturity. journal of the association for information science and technology, 68(9), 2182-2200. https://doi.org/10.1002/asi.23781   cox, a. m., kennan, m. a., lyon, l., pinfield, s., & sbaffi, l. (2019a). maturing research data services and the transformation of academic libraries. journal of documentation, 75(6), 1432-1462. https://doi.org/10.1108/jd-12-2018-0211   cox, a. m., kennan, m. a., lyon, l., pinfield, s., & sbaffi, l. (2019b). progress in research data services: an international survey of university libraries. international journal of digital curation, 14(1), 126-135. https://doi.org/10.2218/ijdc.v14i1.595   creswell, j. w. (1994). research design: qualitative and quantitative approaches. sage.   davis, h. m., & cross, w. m. (2015). using a data management plan review service as a training ground for librarians. journal of librarianship and scholarly communication, 3(2), ep1243. https://doi.org/10.7710/2162-3309.1243   ducas, a., michaud-oystryk, n., & speare, m. (2020). reinventing ourselves: new and emerging roles of academic librarians in canadian research-intensive universities. college & research libraries, 81(1), 43. https://doi.org/10.5860/crl.81.1.43   federer, l. (2018). defining data librarianship: a survey of competencies, skills, and training. journal of the medical library association, 106(3), 294-303. https://doi.org/10.5195/jmla.2018.306   federer, l., clarke, s. c., & zaringhalam, m. (2020, january 16). developing the librarian workforce for data science and open science. osf preprints.  https://doi.org/10.31219/osf.io/uycax    force11. (2014). guiding principles for findable, accessible, interoperable and re‐usable data publishing version b1.0. https://www.force11.org/fairprinciples   fuhr, j. (2019). “how do i do that?” a literature review of research data management skill gaps of canadian health sciences information professionals. journal of the canadian health libraries association/journal de l'association des bibliothèques de la santé du canada, 40(2), 51-60. https://doi.org/10.29173/jchla29371   fuhr, j. (2022). developing data services skills in academic libraries. college & research libraries, 83(3), 474-502. https://doi.org/10.5860/crl.83.3.474   goben, a., & sapp nelson, m. (2018a). the data engagement opportunities scaffold: development and implementation. journal of escience librarianship, 7(2), e1128. https://doi.org/10.7191/jeslib.2018.1128.   goben, a., & sapp nelson, m. (2018b). teaching librarians about data: the acrl research data management roadshow. college & research libraries news, 79(7), 354-357. https://doi.org/10.5860/crln.79.7.354   government of canada. (2021). tri-agency research data management policy. https://www.science.gc.ca/eic/site/063.nsf/eng/h_97610.html   harp, m. r. & ogborn, m. (2019). collaborating externally and training internally to support research data services. journal of escience librarianship, 8(2), e1165. https://doi.org/10.7191/jeslib.2019.1165   joo, s., & schmidt, g. m. (2021). research data services from the perspective of academic librarians. digital library perspectives, 37(3), 242-256. https://doi.org/10.1108/dlp-10-2020-0106   khair, s., dara, r., haigh, s., leggott, m., milligan, i., moon, j., payne, k., portales-casamar, e., roquet, g., & wilson, l. (2020). the current state of research data management in canada: an update to the lcdri data management position paper. new digital research infrastructure organization (ndrio). https://engagedri.ca/assets/documents/ndrio_rdm_positionpaper.pdf   pinfield, s., cox, a. m., & smith, j. (2014). research data management and libraries: relationships, activities, drivers and influences. plos one, 9(12), e114734. https://doi.org/10.1371/journal.pone.0114734   plassche, k. a. (2022). evaluating map and geospatial academic library position descriptions. college & research libraries, 83(4), 631-645. https://doi.org/10.5860/crl.83.4.631   radecki, j. & springer, r. (2020, november 18). research data services in us higher education [research report]. ithaka s+r. https://sr.ithaka.org/publications/research-data-services-in-us-higher-education/   rod, a.b. (2022). a mixed-methods study of data professionals in the canadian academic library ecosystem [unpublished manuscript]. school of information studies, mcgill university.   rod, a.b. (2023). a qualitative study of academic data-related librarians in canada (version 1) [data set]. borealis dataverse. https://doi.org/10.5683/sp3/weogmv   read, k., larson, c., gillespie, c., oh, s., & surkis, a. (2019). a two-tiered curriculum to improve data management practices for researchers. plos one, 14(5), e0215509. https://doi.org/10.1371/journal.pone.0215509   si, l., zhuang, x., xing, w., & guo, w. (2013). the cultivation of scientific data specialists: development of lis education oriented to e-science service requirements. library hi tech, 31(4), 700-724. https://doi.org/10.1108/lht-06-2013-0070 steeleworthy, m. (2014). research data management and the canadian academic library: an organizational consideration of data management and data stewardship. partnership: the canadian journal of library and information practice and research, 9(1). https://journal.lib.uoguelph.ca/index.php/perj/article/view/2990   tang, r., & hu, z. (2019). providing research data management (rdm) services in libraries: preparedness, roles, challenges, and training for rdm practice. data and information management, 3(2), 84-101. https://doi.org/10.2478/dim-2019-0009   tenopir, c., hughes, d., allard, s., frame, m., birch, b., baird, l., sandusky, r., langseth, m., & lundeen, a. (2015). research data services in academic libraries: data intensive roles for the future? journal of escience librarianship, 4(2), e1085. https://doi.org/10.7191/jeslib.2015.1085   tenopir, c., kaufman, j., sandusky, r., & pollock, d. (2019). research data services in academic libraries: where are we today? [white paper]. acrl/choice 360. https://www.choice360.org/research/research-data-services-in-academic-libraries-where-are-we-today/   thielen, j., & neeser, a. (2020). making job postings more equitable: evidence based recommendations from an analysis of data professionals job postings between 2013-2018. evidence based library and information practice, 15(3), 103-156. https://doi.org/10.18438/eblip29674   thomas, c. v., & urban, r. j. (2017). what do data librarians think of the mlis? professionals' perceptions of knowledge transfer, trends and challenges. college and research libraries. 79(3), 401-423. https://doi.org/10.5860/crl.79.3.401   wang, y.-y., & lin, c.-s. (2019). a survey of data science programs and courses in the ischools. proceedings of the association for information science and technology, 56(1), 801-802. https://doi.org/10.1002/pra2.184   wilkinson, m. d., dumontier, m., aalbersberg, i. j., appleton, g., axton, m., baak, a., blomberg, n., boiten, j., bonino da silva santos, l.o., bourne, p.e., bouwman, j., brookes, a.j., clark, t., crosas, m., dillo, i., dumon, o., edmunds, s., evelo, c.t., finkers, r., ... mons, b. (2016). the fair guiding principles for scientific data management and stewardship. scientific data, 3(1), 160018. https://doi.org/10.1038/sdata.2016.18   xia, j., & wang, m. (2017). competencies and responsibilities of social science data librarians: an analysis of job descriptions. college & research libraries, 75(3), 362-388 https://doi.org/10.5860/crl13-435     appendix examples of aggregated title categories   examples of the advisors or directors category include high-level administrators, research officers, unit heads, associate university librarians, directors, or other managers. examples of the data or gis librarian category include titles containing the terms data and data services, gis or geospatial, data visualization, data curation, or data steward. examples of the other functional librarian category include systems, metadata, scholarly communications, digital scholarship, digital curation, or digital preservation. examples of the liaison librarian category include stem librarian, academic librarian, business librarian, or reference librarian.   evidence summary   a mapping review of poster presentation publications across time and academic disciplines   a review of: rowe, n. (2017). tracing the 'grey literature' of poster presentations: a mapping review. health information & libraries journal, 34(2), 106-124.   reviewed by: ruby warren user experience librarian university of manitoba libraries winnipeg, manitoba, canada email: ruby.warren@umanitoba.ca   received: 3 dec. 2017     accepted: 2 feb. 2018      2018 warren. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29383       abstract   objective – to map the development and use of poster presentations in order to determine disciplines in which they are particularly prolific and how their popularity or presence may have changed over time.  this will potentially assist health and interdisciplinary librarians asked to search for poster presentations in systematic reviews.   design – informetric mapping review.   setting – an unnamed uk university library search facility which processes 249 international databases and research publications. databases and publications range across 37 research disciplines, including literature, medicine, and engineering.   subjects – published literature connected to poster presentations – the authors state that this could be poster presentations themselves, abstracts, title listings in conference proceedings, or any variety of materials. they also state that over 99% of the results of this review were title citations or abstracts of conference poster presentations.   methods – an informetric mapping review was conducted via a uk university library search facility by searching for the term “poster presentation” in 249 databases spanning 37 research areas. an index of databases used is provided as an appendix to the article. results were not connected by the search facility to an individual database. search results were categorized by discipline and decade of publication. scholarly and peer-reviewed search limiters were used to obtain an idea of the themes and contributions to what could be considered core literature, and the search was also run in google scholar to provide a comparator. duplications across databases were removed by the search service, although several results appeared both in aggregate (for example, conference proceedings encompassing all poster presentations) and individual form. review of results took an informetrics approach, concerned with quantitative analysis (number of publications over time, number of publications in specific areas or by certain authors, etc.) of production, publication, and use of information, and not with its origins or quality.   main results – even with limiters for peer reviewed or scholarly sources applied, over 99% of returns were abstract or title citations for conference poster presentations – sources which by themselves may not meet the requirements for being scholarly information. from 1937-1969, results only uncovered references to poster use in an educational context. from 1970-1979, the researchers found that poster presentations became a common conference feature, although a less prestigious one than papers. 1980-1989 reiterated the commonality of academic posters, and saw publication of works to advise poster preparation and running poster sessions. during the years 1990-1999, health related disciplines became the main users of posters as an academic medium with 68% of search returns being in health care disciplines. the prominence of posters in health and medicine increased over time. from 2000-2009, search returns in this study show an increase of 360% from those located in 1990-1999. this could indicate an increase in poster sessions, an increase in search accuracy and online availability of material, or both. health care and medical disciplines have demonstrated the most prominent use of poster sessions since the 1990s, although all disciplines have visible poster presentation activity.   conclusions – the author concludes that consistently increasing levels of return for poster abstracts indicate that poster presentations are a fulfilling and popular activity that will continue to be practiced by academics worldwide, but that literature in this review raises issues with the effectiveness of posters as ways to disseminate and discuss research. locating and acquiring conference poster content, not just abstracts or titles, has been a recognized issue in libraries for many years. the authors conclude that the increasing number of poster presentations over time makes it more urgent that we determine what the personal and objective needs of poster users are and ensure that we are meeting them.   commentary   although there is a substantial body of literature on conducting poster sessions and creating poster presentations, rowe’s work provides an insightful overview of the proliferation of the poster presentation in academic literature over time. observations regarding the growth of poster sessions as a form of academic dissemination, and the lack of significant change observed across the author’s search returns in poster presentations as a medium, should prove interesting for librarians and conference organizers.   this study does not achieve validity according to glynn’s ebl critical appraisal checklist (2006), receiving 70% when overall validity is calculated with relevant sections (data collection, study design, results). an appraisal tool more targeted to informetrics mapping, if it exists, might produce a more nuanced understanding of the study’s validity. a lack of clarity regarding specific research questions and variables of interest damage the study’s face validity, and the author does not strongly connect study results to their initial research concerns or objectives. the study also does not screen returned search results for relevance or validity, and the use of a single search string across all databases would likely achieve uneven returns and/or omit relevant results. the author does note alternate explanations or possible confounding variables for the drastic increase in returns during 2000-2009. the author also acknowledges that study replication could be hindered by changes to database algorithms or indexed journals, but indicates that their general observed patterns in the numbers, discipline areas, and content of returns should remain consistent. this seems to be supported by consistent increase patterns viewed between the author’s review of selected databases and their comparator review of google scholar.   even if they do not completely address stated study objectives, authorial conclusions follow logically from study results. in particular, the observation that over 99% of returns consisted of poster titles or abstracts, which by themselves do not reliably convey the full breadth and depth of information necessary from a scholarly source. in light of this result, the author’s suggestions for further research into the needs and motivations of poster users of all kinds seems a necessary next step to determine how libraries and academic communities may better provide access to the information contained in poster presentations.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi:10.1108/07378830610692154 academic librarians’ educational factors and perceptions of teaching transformation: an exploratory examination   amanda nichols hess elearning, instructional technology, and education librarian oakland university libraries rochester, michigan, united states of america email: nichols@oakland.edu   received: 18 nov. 2018                                                                  accepted: 12 july 2019      2019 hess. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29526     abstract   objective – as information literacy instruction is an increasingly important function of academic librarianship, it is relevant to consider librarians’ attitudes about their teaching. more specifically, it can be instructive to consider how academic librarians with different educational backgrounds have developed their thinking about themselves as educators. understanding the influences in how these shifts have happened can help librarians to explore the different supports and structures that enable them to experience such perspective transformation.   methods – the author electronically distributed a modified version of king’s (2009) learning activities survey to academic librarians on three instruction-focused electronic mail lists. this instrument collected information on participants’ demographics, occurrence of perspective transformation around teaching, and perception of the factors that influenced said perspective transformation (if applicable). the author analyzed the data for those academic librarians who had experienced perspective transformation around their teaching identities to determine if statistically significant relationships existed between their education and the factors they reported as influencing this transformation.   results – results demonstrated several statistically significant relationships and differences in the factors that academic librarians with different educational backgrounds cited as influential in their teaching-focused perspective transformation.   conclusion – this research offers a starting point for considering how to support different groups of librarians as they engage in information literacy instruction. the findings suggest that addressing academic librarians’ needs based on their educational levels (e.g., additional master’s degrees, phds, or professional degrees) may help develop productive professional learning around instruction.     introduction   in the shifting higher education environment, academic libraries continually work to serve students and faculty in meaningful, responsive ways. library instruction represents one area where intentional evolution has occurred: while librarians once focused on systematically presenting information on library resources, or bibliographic instruction, their instructional area has changed with the information landscape. as information resources emerged in new formats and finding sources grew more multifaceted, academic librarians shifted into information literacy instruction. rather than focusing on presenting library resources, information literacy is grounded in developing learners’ capacities to “recognize when information is needed and … locate, evaluate, and use effectively the needed information” (american library association [ala], 1989, paragraph 3). the association of college and research libraries (acrl) supported this kind of instruction by developing the information literacy competency standards for higher education (2000). this resource provided academic librarians with information literacy outcomes they could apply across varied instructional environments as the information age emerged in the early 21st century. however, learning needs have continued to shift since that time.   the prescriptive guidelines set forth by the standards did not reflect the information ecosystem where understanding information access, value, and power structures became more crucial and where academic librarians’ instruction was situated. in 2016, acrl sought to address these emerging needs through the framework for information literacy for higher education, which focused information literacy instruction on facilitating deeper learning. this document provided threshold concepts learners need to grasp, rather than performance outcomes they can attain, to be information literate lifelong learners. in reframing instruction, the framework encourages academic librarians to consider their roles as educators in more holistic ways. while the acrl framework may aim to present a new—or perhaps more nuanced—approach to information literacy, it also raised challenges for librarians. even if the framework more fully represented 21st century information dynamics, this approach was a departure from library instruction as set forth in the acrl standards. academic librarians may need to consider how they think of themselves as educators, in response to these changes; scott walter (2008) referred to this self-concept as a teacher identity.   this research considered academic librarians’ teacher identity and, more specifically, whether there are relationships between the experiences that shape this self-concept and their educational background. i used transformative learning theory as a framework with the learning activities survey (king, 2009) to collect librarians’ perception data about their experiences developing teaching identities. i conducted cross-tab and one-way analysis of variance (anova) tests to identify statistically significant relationships between librarians’ education and relational, experiential, and work-related factors in developing these identities. the results show that interactions or experiences impacted academic librarians’ teacher identity development differently, depending on individuals’ education levels.   other research has established that academic librarians can develop teacher identities (walter, 2008) and that this self-concept may emerge from a perspective transformation process (nichols hess, 2018). this scholarship offers a way to advance this scholarly agenda by more deeply understanding the inputs academic librarians believe have influenced this component of their professional identities. beginning to establish such understandings can help librarianship more effectively support information literacy instructors and instruction.   literature review   first, it is important to operationalize the idea of a teacher or teaching identity. in the most practical sense, these terms represent an individual’s self-perception about his or her work as an educator (beauchamp & thomas, 2009). walter (2008) applied this notion more specifically to academic librarians, identifying that their teacher identities center on how they consider their educational roles at their institutions. however, this professional self-concept is not limited to libraries; teaching identities have been explored in the literature around teacher education and preparation (agee, 2004; friesen & besley, 2013; rahmawati & taylor, 2018, smagorisnky, cook, jackson, fry, & moore, 2004; stillwaggon, 2008). in the existing research, scholars have established teaching or teacher identities as multifaceted, dynamic ideas that evolve throughout an individual’s career.   since teaching identities are fluid, it is useful to consider how they may develop with a theoretical framework focused on personal evolution and development. jack mezirow’s (1978, 1981, 1994, 1997, 2000) transformative learning theory offers such a starting point. his work is built on the idea that adults use their experiences to make meaning of the world around them but that they can fall back onto ideas or schema adopted from others (e.g., authority figures, perceived experts, family, friends) and not personally evaluated (mezirow, 1997). transformation, then, happens when adults consider the environment in which they exist and establish their own beliefs and values based on biographical, social, and cultural experiences. more specifically, “perspective transformation” happens. adults have internal cognitive “frames of reference” they use to make sense of the world, and these frames are composed of “habits of mind” (mezirow, 1978). while frames of reference are broader ways adults view situations, groups, and interactions, habits of mind are more specifically grounded in the snap judgments or interpretations adults make (mezirow, 1997). from these frames of reference and habits of mind, adults then present external-facing points of view (mezirow, 1997). these frames of reference and habits of mind may change with inputs from individuals’ experiences in the world (mezirow, 1978, 1994, 2000); in such instances, external-facing points of view also shift. having these transformative experiences leads adults to develop more authentic senses of selves.   researchers have applied transformative learning theory to understand how disciplinary faculty in higher education engage in developing teaching identities (balmer & richards, 2012; cranton & carusetta, 2004; post, 2011). neither this scholarship nor the research on k-12 teacher identities can be applied wholesale to academic librarians, though. academic librarians’ teaching practices are considerably different from either k-12 educators or subject-area faculty; thus, they may have unique needs or experiences in forming, or transforming, how they see themselves as educators. therefore, the scholarship on academic librarians’ educational experiences influences how these transformational experiences can be understood. these may be related to their library-focused graduate education, the informal or professional learning they engage in within the field, or any other formal degree-granting programs pursued outside of librarianship. this focus area does not exist in the research literature on disciplinary faculty’s teaching identity development and transformative learning, since they generally hold doctoral degrees in their subject area. the research from the library literature in this area can both consider academic librarians’ unique experiences as educators and provide important context.   researchers have established that academic librarians generally experience limited or inadequate exposure to information literacy in library school (bailey, jr., 2010; corral, 2010; sproles, johnson, & farison, 2008). as such, academic librarians may engage in post-graduate training or education around instructional practices and educational identity development. in fact, scholars have demonstrated that new professionals enter the field expecting to engage in this kind of job-specific training that offer opportunities to enhance their skills and gain knowledge not addressed in their academic experiences (sare, bales & neville, 2012). moreover, researchers conducting a study of 788 canadian library staff with instructional responsibilities found that many used self-directed or self-selected postgraduate professional learning experiences (e.g., attending workshops, reviewing the literature) or informal job-based learning offerings to prepare for their teaching responsibilities (julien & genuis, 2011). other scholars have focused on how librarians have used such resources, including job-embedded professional learning (click & walker, 2010; nichols hess, 2016; shamchuk, 2015; walter, 2006), instruction-centric institutional offerings (hoseth, 2009; otto, 2014), and a variety of professional mentorship relationships (james, rayner, & bruno, 2015; lorenzetti & powelson, 2015; mavrinac, 2005) to support their own teaching identity development. these researchers’ works emphasize that academic librarians only begin to learn the pedagogical essentials after they earn master’s of library or information science (mlis) degrees.   while some academic librarians pursue ongoing informal professional development, others elect more formal educational options. librarians who have in-depth liaison relationships with academic units may find that additional degrees—master’s, professional (e.g., jd, specialist certificates), or phds—offer opportunities to deepen subject knowledge and develop pedagogical competencies. while this route is not uncommon, there is not broad agreement on whether such education is necessary—or helpful—to the profession (crowley, 2004; ferguson, 2016; mayer & terrill, 2005). researchers have demonstrated that those who had attained doctorates in subject areas felt this experience gave them credibility with faculty, expertise in their instructional disciplines, and deep research experience they could use to connect with students (gilman & lindquist, 2010). however, these librarians indicated that additional education was not the only route to gain advanced subject knowledge; they cited on-the-job experience and other learning undertakings as real difference-makers in developing their disciplinary understandings, not credentials or degrees.   aims   using the teaching identity concept, transformative learning theory, and the existing research on how academic librarians’ educational experiences impact their professional identity, i investigated the following question: how do academic librarians’ educational experiences (i.e., education level, additional degrees) interact with external inputs (e.g., relationships, professional experiences) to influence their teaching identity development?   this inquiry builds on research establishing that academic librarians can experience perspective transformation around their teaching identities and that different types of hands-on experiences as educators may shape these identities in different ways (nichols hess, 2018). this existing scholarship identified an area for inquiry around academic librarians’ teaching identities and perspective transformation. this research, then, sought to advance this topic by considering whether academic attainment influenced how academic librarians’ teaching identity development happened.   methods   research approach   i used an exploratory perspective to further develop this research area in the library literature. i used a modified version of kathleen p. king’s (1997, 2009) learning activities survey (las; see appendix a) to solicit a voluntary sample from academic librarians engaged in instruction. the las is grounded in transformative learning theory. respondents reflect on whether they believe they have experienced perspective transformation and indicate which inputs they believe have influenced such experiences. although other researchers have explored librarians’ teaching-based perspective transformation in qualitative ways (walter, 2008), i chose a survey instrument to collect deductive data from a large group of academic librarians. while the exploratory study design did not generate generalizable data, it does establish a foundation on which other researchers can construct related scholarship. all appropriate regulatory approvals from my university research board were received before data collection began.   survey modification, distribution, and data collection   king (2009) developed, copyrighted, demonstrated the reliability of, and validated the las. she encouraged researchers to use or modify her instrument, gratis, so long as she was credited; other researchers have used king’s las to examine specific populations’ cognitive and behavioral transformations (see, for example, brock, 2010; kitchenham, 2006; kumi-yeboah & james, 2014). king provided specific modification guidelines to preserve the instrument’s integrity (king, 2009, pp. 36-44). in this research context, i modified the las per king’s directions to ground librarians’ transformative experiences around their teaching in the broader body of research while maintaining the instrument’s reliability.   any version of the las has three types of questions:   demographic items; items that ask respondents to indicate whether perspective transformation has occurred; and items about what inputs impacted an individual’s perspective transformation process.   questions related to whether individuals have experienced perspective transformation should not be altered except to provide relevant contextual information. researchers must review participants’ responses to generate perspective transformation index (pt-index) groupings using a standard set of procedures (king, 2009). this baseline metric determines whether individuals report experiencing perspective transformation, and it establishes a sub-group of participants that the researcher can use for subsequent analyses. i adhered to these guidelines when examining academic librarians’ experiences with perspective transformation around their teaching identities.   i built the new version of the las in qualtrics and distributed the survey instrument via email to three information literacy-focused electronic mailing lists (acrlframe-l, infolit-l, and lirt-l) to recruit a voluntary sample; 501 individuals responded. at the time of distribution, this figure represented between a 5.9% (total overlap in list membership) and 8.1% (no overlap in list membership) response rate. while anyone could participate in the survey, those who indicated that library instruction or information literacy was not part of their job responsibilities were automatically directed to the end of the instrument. the survey was open from february 6 to april 6, 2017; all incomplete responses were automatically recorded when the survey closed.    preparatory procedures: identifying perspective transformation   per king’s (2009) directions, all respondents were assigned to a pt-index designation. this information reflects participants’ responses to four items on the las, and it “indicates whether [learners] have experienced a perspective transformation” (king, 2009, p. 38). on this version of the las, those four questions were:   ●        item 14: think about your professional experiences in teaching—check off any of the following statements that apply. ●        item 15: since you have been providing information literacy instruction, do you believe you experienced a time when you realized that your values, beliefs, opinions, or expectations (for example, how you viewed your work responsibilities or roles as an academic librarian) changed? ●        item 16: describe what happened when you realized your values, beliefs, opinions, or expectations about your instructional responsibilities had changed. ●        item 20: think back to when you first realized that your views or perspective had changed. what did your professional life have to do with the experience of change?   to identify the pt-index designations of all 501 participants, i first identified individuals who had checked at least one of the affirmative statements in item 14 or who had indicated “yes” or “i’m not sure” in response to item 15. these individuals were initially classified in a yes pt-index group. individuals who had not selected any of the affirmative statements about transformation in item 14 or had indicated “no” to item 15 were categorized into a no pt-index group. i then reviewed respondents’ free-text comments for items 16 and 20 to affirm or modify these group assignments as needed.   while a total of 501 individuals responded, 353 survey participants were ultimately classified as yes pt-index group members, or as individuals who had reported experiencing perspective transformation around their teaching identities in some way. those in the no pt-index group were excluded from all additional analyses.   preparatory procedures: identifying transformative constructs   the next goal was to understand what factors had influenced the respondents’ perspective transformation. on items 17 to 19 of the las, participants identified the relationships, experiences, or resources, and professional events they believed had influenced their teaching identity development. there were 41 potential inputs across these three items, and participants could select all that applied. analyses between demographic categories and each of the inputs individually would not provide meaningful data. instead, i used spss to conduct a principal component analysis using varimax (orthogonal) rotation followed by a subsequent confirmatory factor analysis on participants’ responses to each item to identify transformative constructs for relationship-, experience-, and professionally centric inputs. in this type of analysis, statistical tests were used to examine where participants selected common variables in response to each question separately; this process helped to identify where links existed across participants’ responses to a single question.   the principal component analysis reduced 41 variables from three items into 12 transformative constructs that participants indicated had influenced their teaching identity development process. the resulting confirmatory factor analysis was used to identify the connections and build these constructs; they each had eigenvalues greater than 1.0 and significant factor criterion of at least 0.4. i used the inputs within each construct to determine the terms used to describe each construct’s core ideas.   in response to item 17 on the las, the relationship-centric constructs that influenced participants’ teaching identity development were:   ●        supportive interpersonal relationships, which was comprised of six inputs related to the positive relationships participants developed laterally—such as with colleagues and disciplinary faculty—as well as their interactions with students ●        motivating leaders, which was comprised of four inputs related to the relationships participants had with their work mentors, supervisors, and administrators in more of a top-down structure ●        challenging colleagues, which was comprised of three inputs related to participants’ negative interactions (e.g., criticism, negative feedback, comments on issues with instruction) with colleagues, other librarians, and disciplinary faculty ●        other important relationships, which was comprised of other relationship-centric inputs participants could include   in response to item 18 on the las, the experience-centric constructs that influenced teaching participants’ identity development were:   ●        professional learning, which was comprised of seven inputs related to participants engaging with diverse readings on teaching, attending professional development workshops, and observing other librarians’ instruction ●        writing and technology-rich teaching, which was comprised of four inputs related to participants’ experiences teaching online or in hybrid environments and writing about teaching practices for publication ●        external feedback, which was comprised of three inputs related to participants’ experiences observing disciplinary faculty’s teaching, receiving comments from students, and getting feedback from disciplinary faculty ●        library-centric input, which was comprised of three experiential inputs related to participants’ library school coursework, engaging in discussion with other librarians about their instructional practices, and completing teaching self-reflections ●        self-reflection and other experiences, which was comprised of two inputs related to participants’ use of reflection journals, and other experience-centric inputs participants could include    in response to item 19 on the las, professionally centric constructs that influenced teaching participants’ identity development were:   ·         completing graduate education, which was comprised of two inputs related to participants’ library and non-library program graduation (that is, not their education level itself—but that the experience of completing an educational program had impacted these participants’ senses of themselves as educators) ·         changing job statuses, which was comprised of three inputs related to participants’ first professional job, changes in professional jobs, or job losses ·         other shifting responsibilities, which was comprised of two inputs related to participants’ changing work duties and other work-centric inputs participants could include   because there is overlap between relationship-, experience-, and professionally centric inputs, there are some similarities between the resulting constructs. however, each of the original inputs aligned with only one transformative construct. one experience-centric input—teaching face-to-face—did not align with a specific transformative construct. this outlier existed because 179 respondents (of the 353 individuals in the yes pt-index group) selected this input as an influence in shaping their teaching identity. face-to-face teaching, then, influenced teaching identity transformation across participants’ other experiences rather than aligning as part of a particular construct. this input was maintained in subsequent data analysis.   preparatory procedures: transforming participants’ responses to z-scores   i transformed participants’ (n = 353) combined responses for the inputs in each of the 12 transformative constructs into composite scores. this data transformation allowed for the analysis of perceptions of how the 12 constructs had influenced perspective transformation around teaching identities. spss was used to generate these responses into standardized z-scores, and this process allowed for comparison of how constructs composed of diverse numbers of inputs influenced participants across demographic items. in these z-scores, 0 is the mean, and one unit indicates a standard deviation in the sample. the probability of a score occurring within a normal distribution from these standard scores could then be calculated.   the preparatory procedures involved considerable data-related work, but the values generated in these processes (i.e., eigenvalues/factors associated with 12 transformative constructs) were not used in any subsequent analyses. rather, these steps allowed for cleaning the data as a prerequisite step to examining whether differences existed among how librarians across educational experiences experienced perspective transformation related to their teaching. these distinctions between the preparatory procedures and data analysis process are represented in figure 1.   data analysis: crosstab analysis and one-way anova   after establishing the following:   ●        which participants believed they had experienced perspective transformation around their teaching (n = 353), ●        the 12 transformative constructs and one input that impacted these transformative processes, and ●        participants’ composite z-scores that reflected their responses to the 12 transformative constructs,   i analyzed whether different constructs or one input affected participants’ teaching identity transformation processes in relation to their education levels.   in the instance of the one remaining input—teaching face-to-face—i used spss to run cross-tabulation analysis with a chi-square test statistic to consider its relationship to librarians’ teaching identity development. this type of analysis determines whether statistically significant relationships exist between categorical independent variables (e.g., education level, education beyond an mlis) and categorical dependent variables (i.e., whether teaching face-to-face had influenced perspective transformation). librarians’ responses to this item were analyzed this way because this input did not align with a single transformative construct. the standard alpha level of .05 was used to argue for significance for this analysis.     figure 1 a visual representation of the data preparatory and analysis processes.     spss was used to conduct anova tests and explore whether there were statistically significant relationships between librarians’ education levels and the 12 transformative constructs. when comparing multiple groups within a population, the one-way anova compares means in relation to a single variable (e.g., a transformative construct). one-way anova is appropriate when the independent variable is categorical (i.e., mutually exclusive options) and the dependent variables are continuous (i.e., points on a fixed scale). in this research, participants’ responses to the demographic questions about their education levels and additional education beyond the mlis—the independent variables—were categorical. the compiled data for the 12 transformative constructs are continuous data because participants’ responses were transformed into z-scores. one-way anova, then, is the most useful way to examine whether librarians with different educational or work-related backgrounds felt that different transformative constructs influenced their teaching identity development. since one-way anova only identifies whether differences exist between groups, fisher’s least significant distance (lsd) post-hoc comparison tests were used to examine where those differences existed between groups to more fully understand the statistical results. i used the standard alpha level of .05 to argue for significance for this analysis.   results   overall education level   participants who had experienced perspective transformation around their teaching identities (n = 353) were largely homogeneous in their overall education level. of these respondents, 324 held master’s degrees, followed by 16 who had earned doctorate degrees. seven participants held professional degrees (e.g., mba, jd), while four held bachelor’s degrees and two respondents had some other level of education (see figure 2).     figure 2 participants’ (n = 353) highest education levels.     no statistically significant differences existed between participants’ highest education levels and whether they believed any of the 12 transformative constructs or teaching face-to-face had influenced their teaching identity development. these constructs and input, then, seemed to similarly impact librarians’ teaching identities across overall education levels. however, participants’ overall education level did not represent the granularity of their graduate learning experiences—for instance, the master’s degree demographic group included those with an mlis, those with additional master’s degrees in other subject areas, and potentially those currently in graduate programs (master’s, professional, or doctorate). therefore, i considered these components in greater detail to more fully understand academic librarians’ educational experiences and the impacts that affected their teaching identity development.     figure 3 participants’ (n = 352) additional degrees.     additional education beyond the mlis   while the mlis is considered the terminal degree in the field (association of college and research libraries [acrl], 2018), participants who had experienced perspective transformation around their teaching identities also shared information about additional graduate experiences. of the participant sub-group who responded to this item (n = 352; one person did not respond), 195 had no additional degree. those participants who already held additional degrees included 106 with additional master’s degrees, 14 with doctorates, and eight with professional degrees. some participants had degrees in process: 16 respondents were working to complete additional master’s degrees, while nine were completing doctorates and three were completing professional degrees (see figure 3).   a chi-square test of independence was used to examine whether there were statistically significant relationships between respondents’ additional education and the impact of teaching face-to-face on teaching identity transformation. the relation between these variables was not significant, x2 (7, n = 352) = 3.40, p > .05. these data suggest that the impact of teaching face-to-face does not influence librarians’ teaching identity transformation differently across additional degree levels.     table 1 impact of the motivating leaders construct on teaching identity transformation for academic librarians with education beyond an mlis additional education  significantly different from: mean as a z-score standard deviation professional degree (n = 8) professional degree in process* -0.34 0.56 additional master’s (n = 106) professional degree in process* doctorate in process* -0.20 0.85 doctorate (n = 14) professional degree in process* -0.19 0.77 no additional degree (n = 195) professional degree in process* 0.06 1.30 additional master’s in process (n = 6) no other educational level 0.28 1.27 doctorate in process (n = 9) additional master’s* 0.71 1.82 professional degree in process (n = 3) no additional education* additional master’s* professional degree* doctorate* 1.44 2.72 *p < .05     based on librarians’ reported education in addition to an mlis, i observed differences in the role that motivating leaders (f [6, 344] = 2.214, p = .041), writing and technology-rich teaching (f [6, 344] = 4.219, p < .001), and library-centric input (f [6, 344] = 4.184, p = .005) played in their perspective transformation around teaching identities. tables 1 to 3 illustrate the differences observed for these three components. the first column lists participants’ education levels; in the second column, the groups where differences occurred are presented, along with the appropriate p values. the third column presents the means (represented as z-scores) organized in ascending order, and the fourth column contains standard deviations.   in the case of motivating leaders, those librarians pursuing doctorate and professional degrees were more likely to cite this construct as a component in their teaching identity transformation 0.71 and 1.44 standard deviations above the mean, respectively (see table 1). in contrast, those respondents with professional, additional master’s, and doctorate degrees cited motivating supervisors 0.34, 0.20, and 0.19 standard deviations below the mean, respectively. these data suggest that those participants with additional graduate degrees did not believe that motivation from supervisors had influenced their perspective transformation around their teaching identities, while those with degrees in process may have held different perceptions.     table 2 impact of the writing and technology-rich teaching construct on teaching identity transformation for academic librarians with education beyond an mlis additional education significantly different from: mean as a z-score standard deviation professional degree (n = 8) professional degree in process* doctorate in process* -0.24 0.87 no additional degree (n = 195) professional degree in process* doctorate in process** -0.01 1.03 additional master’s in process (n = 6) professional degree in process* doctorate in process* -0.01 1.03 additional master’s (n = 106) professional degree in process* doctorate in process** 0.16 1.16 doctorate (n = 14) doctorate in process* 0.26 1.42 professional degree in process (n = 3) no additional education* additional master’s in process* additional master’s* professional degree*   1.61   2.01 doctorate in process (n = 9) additional master’s in process* additional master’s** professional degree* doctorate* 1.62 1.78  *p < .05 **p < .001     similarly, those respondents pursuing professional or doctorate degrees were more likely to indicate that writing and technology-rich teaching had influenced their transformation around their teaching identities (see table 2). these individuals cited the influence of writing and technology-rich teaching in their teaching identity development processes 1.61 (professional degree in process) and 1.62 (doctorate in process) standard deviations above the mean. these results suggest that these groups of academic librarians may be more likely to report having experienced teaching-related perspective transformation because of writing and technology-rich teaching than their colleagues with different educational backgrounds.   those individuals who held professional degrees or were earning doctorates were more likely to report having experienced a shift in their perspectives based on library-centric input rather than external feedback (see table 3). individuals with these degrees reported that this construct had influenced their teaching identity development 1.12 and 1.33 standard deviations above the mean, respectively. interestingly, though, respondents with doctorates were less likely—0.32 standard deviations below the mean—to cite library-centric input as having played a role in their perspective transformation. these data suggest there are differences in how library-centric feedback impacts librarians’ teaching identity development across educational backgrounds.     table 3 impact of the library-centric input construct on teaching identity transformation for academic librarians with education beyond an mlis additional education statistically different from: mean as a z-score standard deviation doctorate (n = 14) professional degree* doctorate in process* -0.32 0.96 professional degree in process (n = 3) no other educational level -0.22 0.43 additional master’s (n = 106) professional degree* doctorate in process* -0.09 1.02 additional master’s in process (n = 16) professional degree* doctorate in process*   0.03 1.01 no additional degree (n = 195) professional degree* doctorate in process* 0.15 1.19 professional degree (n = 8) no additional education* additional master’s in process* additional master’s* doctorate* 1.12 1.76 doctorate in process (n = 9) additional master’s in process** additional master’s** doctorate* 1.33 1.77  *p < .05 **p < .001     discussion   when viewed through the transformative learning theoretical framework as well as existing literature on academic librarians’ educational experiences, these results suggest several relevant, practical takeaways. while elsewhere i have established that academic librarians believe they experience perspective transformation around their teaching identities (nichols hess, 2018), these data suggest how education-related inputs differently impact academic librarians’ experiences in forming teaching identities. furthermore, they build on other teaching identity-related research to better understand how academic librarians develop this facet of their self-concept (julien & genuis, 2011; shamchuk, 2015; walter, 2006, 2008). these findings also reinforce mezirow’s (1994, 1997, 2000) assertion that external experiences, relationships, and environments affect individuals’ self-concepts in different ways. while this study’s conclusions are exploratory and suggestive, the statistically significant differences present ideas for individual librarians and library leaders to consider for ongoing teaching identity development.   there were several areas where academic librarians’ educational experiences influenced the transformative constructs important to their teaching identities. for example, the author’s data analysis suggested that those with education beyond an mlis experienced shifts in their thinking about their teaching in different ways from their peers who held the terminal degree. individuals who pursued professional or doctorate degrees indicated that transformation around their teaching identities had been influenced more by motivating leaders. perhaps this top-down motivation for instructional identity development came from supervisors’ beliefs that these academic librarians were instructional leaders or their desire to see these individuals act as pedagogical champions. academic librarians’ professional or doctoral education, then, may have outwardly manifested their developing teaching identities to those in leadership roles in different ways.   in addition to leadership’s top-down influence in shaping their teaching identities, those academic librarians with doctorates indicated that library-based feedback was less influential in their transformative experiences than many of their peers. this kind of feedback included comments from other librarians—both at and outside of their institutions—and from library school faculty. this demographic group was relatively small, but they may have also interacted with both colleagues and faculty outside of librarianship in different ways. as such, it makes sense that those instructional librarians with doctorate-level education would find instructional communities outside of the library, including with disciplinary faculty, to be useful in developing their teaching identities.   moreover, several groups of librarians with education beyond an mlis, including those with doctorates and additional master’s degrees, indicated that writing and technology-rich teaching had positive impacts on their teaching-related perspective transformations. librarians with these educational backgrounds may find it useful to pursue these kinds of experiences more intentionally as they seek to further hone their instructional identities. for instance, academic librarians may find it instructive to embed in online or hybrid courses more intentionally. there are myriad ways to make such connections, including being embedded in a learning management system, offering synchronous online instructional support, and developing freestanding e-learning modules. librarians with doctorates or additional master’s degrees may find these experiences help them consider their teaching identities in new ways. also, librarians with experience with data collection and analysis in master’s or doctoral programs should seek opportunities to apply these experiences to writing about their instructional practices. such additions to information literacy-centric scholarship would deepen the field’s research corpus, could inspire other academic librarians to develop their teaching identities, and may engage those librarians with additional master’s or doctorates in more fully considering their educational expertise.   similarly, library leaders may also want to investigate how they can provide these kinds of opportunities to their academic librarians with additional degrees. at a broader level, though, it may be worth considering whether these academic librarians feel more equipped or have more frequent opportunities to engage in writing and technology-rich teaching. if so, academic librarians may find it useful to consider what experiences from these kinds of degree-granting programs could benefit individuals in mlis programs or on-the-job learning experiences.   limitations   while this research identified statistically significant differences in academic librarians’ education, work experiences, and transformative inputs in developing teaching identities, there are several important limitations to consider. this research is suggestive only; it does not present, or attempt to present, any causal relationships. moreover, the size of these groups may have impacted the effect size. and it is important to consider when individuals earned any additional graduate training. the timing of additional degrees (e.g., before or after an mlis, earned well before working as an academic librarian) may influence individuals’ experiences. however, this version of the las did not ask participants for such information. future research that can mitigate these constraints would help to better contextualize the author’s findings in this study.   implications for instructional practice   there are several practical takeaways for academic librarians and library leaders who support teaching identity development. individuals’ responses to this survey highlighted that librarians with different academic backgrounds may find different supports beneficial for perspective transformation around teaching. since instruction librarians have a variety of educational backgrounds—including additional master’s degrees, professional degrees, and doctorates—it is useful for the profession to acknowledge that learning, development, and motivation experiences impact academic librarians in different ways. acknowledging these differences is the first step to providing the appropriate support for academic librarians’ teaching identity development, and it can help librarians, supervisors, and library administrators to develop personalized or focused plans for individuals’ professional development. for example, those with advanced education may find supervisor-based mentorship useful, either within the library or at their institution more broadly. these librarians may also find it helpful to pursue supportive interpersonal relationships outside of librarianship, whether at their institutions (e.g., workshops at teaching and learning centers) or in other environments (e.g., teaching conferences, social media, teaching-focused electronic mailing lists). conversely, those academic librarians who hold an mlis may find it most beneficial to develop library-centric relationships, both within their own institutions and across the profession, that focus on teaching approaches, instructional practices, and education-centered reflection. these kinds of experiences may help these professionals to more intentionally develop their teaching identities.   more intentional research with those academic librarians who hold doctorates, work in instruction, and have experienced transformation around their teaching identities may be useful in this case. and more broadly, academic library administrators who work with librarians who hold education beyond an mlis should investigate how they can support these individuals’ perspective transformation around teaching. doing so may benefit both those librarians’ practices and the libraries’ broader information literacy instruction programs.   conclusion   in this study, i analyzed data from academic librarians who indicated they had experienced perspective transformation around their teaching identity to determine if there were relationships to individuals’ educational backgrounds and transformative inputs. i used one-way anova with 12 transformative constructs and cross-tab analysis with one categorical input to identify where differences existed between these demographic categories. the results show that there are some statistically significant differences between academic librarians’ educational levels and the inputs they believe have influenced their perspective transformation processes.   researchers can conduct additional, focused scholarship to determine how to best understand and act on these relationships. for example, survey research with librarians with additional master’s, professional, or doctorate degrees may help frame how they experience shifts in their thinking and practices around their instructional identities. moreover, interviews with an intentional sampling of librarians from these groups may provide more in-depth insight into how librarians’ educational backgrounds influence the effects of different transformative inputs on their senses of themselves as educators. these kinds of follow-up studies may help us to both better understand different academic librarians’ instruction-driven perspective transformation experiences and provide opportunities that promote such shifts in thinking.    references   agee, j. (2004). negotiating a teaching identity: an african american teacher's struggle to teach in test-driven contexts. teachers college record, 106(4), 747-774. https://doi.org/10.1111/j.1467-9620.2004.00357.x   american library association. (1989, january 10). presidential committee on information literacy: final report. retrieved from  http://www.ala.org/acrl/publications/whitepapers/presidential   association of college and research libraries. (2018, april). statement on the terminal professional degree for academic librarians. retrieved from http://www.ala.org/acrl/standards/statementterminal   association of college and research libraries. (2016). framework for information literacy for higher education. retrieved from http://www.ala.org/acrl/standards/ilframework   association of college and research libraries. (2000). information literacy competency standards for higher education. retrieved from https://alair.ala.org/bitstream/handle/11213/7668/acrl%20information%20literacy%20competency%20standards%20for%20higher%20education.pdf?sequence=1&isallowed=y   bailey, jr., e. c. (2010). educating future academic librarians: an analysis of courses in academic librarianship. journal of education for library and information science, 51(1), 30-42.   balmer, d. f., & richards, b. f. (2012). faculty development as transformation: lessons learned from a process-oriented program. teaching and learning in medicine, 24(3), 242-247. https://doi.org/10.1080/10401334.2012.692275   beauchamp, c., & thomas, l. (2009). understanding teacher identity: an overview of issues in the literature and implications for teacher education. cambridge journal of education, 39(2), 175-189. https://doi.org/10.1080/03057640902902252   brock. s. e. (2010). measuring the importance of precursor steps to transformative learning. adult education quarterly, 60(2), 122-142. https://doi.org/10.1177/0741713609333084   click, a., & walker. c. (2010). life after library school: on-the-job training for new instruction librarians. endnotes: the journal of the new members round table, 1(1), 1-14. retrieved from http://www.ala.org/rt/sites/ala.org.rt/files/content/oversightgroups/comm/schres/endnotesvol1is1/2lifeafterlibrarysch.pdf   cranton, p., & carusetta, e. (2004). perspectives on authenticity in teaching. adult education quarterly, 55(1), 5-22. https://doi.org/10.1177/0741713604268894   crowley, b. (2004). just another field? lis programs can, and should, reclaim the education of academic librarians. library journal, 129(18), 44-46.   ferguson, j. (2016). additional degree required? advanced subject knowledge and academic librarianship. portal: libraries and the academy, 16(4), 721-736. https://doi.org/10.1353/pla.2016.0049   friesen, m. d., & besley, s. c. (2013). teacher identity development in the first year of teacher education: a developmental and social psychological perspective. teaching and teacher education, 36, 23-32. https://doi.org/10.1016/j.tate.2013.06.005   gilman, t., & lindquist, t. (2010) academic/research librarians with subject doctorates: experiences and perceptions, 1965-2006. portal: libraries and the academy, 10(4), 399-412. https://doi.org/10.1353/pla.2010.0007   hoseth, a. (2009). library participation in a campus-wide teaching program. reference services review, 37(4), 371-385. https://doi.org/10.1108/00907320911006985   julien, h., & genuis, s. k. (2011). librarians’ experiences of the teaching role: a national survey of librarians. library & information science research, 33(2), 103-111. https://doi.org/10.1016/j.lisr.2010.09.005   king, k. p. (1997). examining activities that promote perspective transformation among adult learners in higher education. international journal of university adult education, 36(3), 23-37.   king, k. p. (2009). the handbook of the evolving research of transformative learning based on the learning activities survey. charlotte, nc: information age publishing.   kitchenham, a. (2006). teachers and technology: a transformative journey. journal of transformative education, 4(3), 202-225. https://doi.org/10.1177/1541344606290947   kumi-yeboah, a., & james, w. (2014). transformative learning experiences of international graduate students from asian countries. journal of transformative education, 12(1), 25-53. https://doi.org/10.1177/1541344614538120   lorenzetti, d. l., & powelson, s. e. (2015). a scoping review of mentoring programs for academic librarians. journal of academic librarianship, 41(2), 186-196. https://doi.org/10.1016/j.acalib.2014.12.001   mavrinac, m. a. (2005). transformational leadership: peer mentoring as values-based learning process. portal: libraries and the academy, 5(3), 391-404. https://doi.org/10.1353/pla.2005.0037   mayer, j., & terrill, l. j. (2005). academic librarians’ attitudes about advanced-subject degrees. college & research libraries, 66(1), 59-73. https://doi.org/10.5860/crl.66.1.59   mezirow, j. (1978). perspective transformation. adult education quarterly, 28(2), 100-110.   mezirow, j. (1981). a critical theory of adult learning and education. adult education quarterly, 32(1), 3-24.   mezirow, j. (1994). understanding transformation theory. adult education quarterly, 44(4), 222-232.   mezirow, j. (1997). transformative learning: theory to practice. new directions for adult & continuing education, 74, 5-12   mezirow, j. (2000). learning to think like an adult. in j. mezirow and associates (eds.), learning as transformation: critical perspectives on a theory in progress (pp. 3-33). san francisco: jossey-bass, 2000.   nichols hess, a. (2018). trasnforming academic library instruction: shifting teaching practices to reflect changed perspectives. new york: rowman & littlefield.    nichols hess, a. (2016). a case study of job-embedded learning. portal: libraries and the academy, 16(2), 327-347. https://doi.org/10.1353/pla.2016.0021   otto, p. (2014). librarians, libraries, and the scholarship of teaching and learning. new directions for teaching and learning, 139, 77-93. https://doi.org/10.1002/tl.20106   post, p. a. (2011). trial by hire: the seven stages of learning to teach in higher education. contemporary issues in education research, 4(12), 25-35. https://doi.org/10.19030/cier.v4i12.6659   rahmawati, y., & taylor, p. (2018). “the fish becomes aware of the water in which it swims”: revealing the power of culture in shaping teaching identity. cultural studies of science education, 13(2), 525-537. https://doi.org/10.1007/s11422-016-9801-1   sare, l., bales, s., & neville, b. (2012). new academic librarians and their perceptions of the profession. portal: libraries and the academy, 12(2), 179-203. https://doi.org/10.1353/pla.2012.0017     shamchuk, l. (2015). professional development on a budget: facilitating learning opportunities for information literacy instructors. partnership: the canadian journal of library and information practice and research, 10(1), 1-14. https://doi.org/10.21083/partnership.v10i1.3437   smagorinsky, p., cook, l. s., jackson, a. y., fry, p. g., & moore, c. (2004). tensions in learning to teach: accommodation and the development of a teaching identity. journal of teacher education, 55(1), 8-24. https://doi.org/10.1177/0022487103260067   sproles, c., johnson, a. m., & farison, l. (2008). what the teachers are teaching: how mlis programs are preparing academic librarians for instructional roles. journal of education for library and information science, 49(3), 195-209.   stillwaggon, j. (2008). performing for the students: teaching identity and the pedagogical relationship. journal of philosophy of education, 42(1), 67-83. https://doi.org/10.1111/j.1467-9752.2008.00603.x   walter, s. (2008). librarians as teachers: a qualitative inquiry into professional identity. college & research libraries, 69(1), 51-71. https://doi.org/10.5860/crl.69.1.51   walter, s. (2006). instructional improvement: building capacity for the professional development of librarians as teachers. reference and user services quarterly, 45(3), 213-218.     appendix a survey instrument   1. do you agree to participate in this study? yes, i agree to participate in this study. no, i do not agree to participate in this study.   2. is information literacy instruction part of your current work responsibilities? yes no   3. gender prefer not to say male female   4. ethnicity white / caucasian hispanic or latinx black or african american native american or american indian asian / pacific islander other multiracial prefer not to answer   5. age group under 25 25-34 35-44 45-54 55-64 65-74 75 or over   6. what is the highest level of education you have completed? bachelor's degree master's degree professional degree doctorate degree other   7. have you completed a graduate degree in addition to a master's degree in library/information science? no no, but i am in the process of completing an additional master's degree no, but i am in the process of completing a professional degree no, but i am in the process of completing a doctoral degree yes, i have an additional master’s degree yes, i have a professional degree yes, i have a doctoral degree other   8. when did you graduate from library school? i did not attend library school i am currently in library school within the last year 1-3 years ago 4-6 years ago 7-9 years ago 10+ years ago   9. at what kind of institution do you work? i am not currently employed community or junior college four-year college master's-granting university doctoral/research university other   10. how long have you worked at your current institution? less than one year 1-3 years 4-6 years 7-9 years 10+ years   11. how long has instruction been a part of your work responsibilities? less than one year 1-3 years 4-6 years 7-9 years 10+ years   12. what kinds of instruction are part of your work responsibilities? select all that apply. face-to-face instruction online instruction blended / hybrid instruction   13. on average, how frequently do you engage in classroom instruction? once a year 1-3 times a semester 4-6 times a semester 7-9 times a semester 10+ times a semester   14. think about your professional experiences in teaching—check off any of the following statements that apply. i had an experience that caused me to question the way i normally teach. i had an experience that caused me to question my ideas about professional roles (examples of professional roles include the kinds of instructional responsibilities an academic librarian should take on.) as i questioned my ideas, i realized i no longer agreed with my some or all of my previous beliefs or role expectations. as i questioned my ideas, i realized i still agreed with some or all of my beliefs or role expectations. i realized that other people also questioned their beliefs about their instructional roles or responsibilities. i thought about acting in a different way from my usual teaching beliefs and roles. i felt uncomfortable with professional expectations (for example, what my job responsibilities or work roles were) around teaching and instruction. i tried out new teaching roles so i would become more comfortable and confident in them. i tried to figure out a way to adopt these new ways of acting. i gathered the information i needed to adopt these new ways of acting. i began to think about the reactions and feedback from my new professional behavior. i took action and adopted these new ways of acting. i do not identify with any of the statements above.   15. since you have been providing information literacy instruction, do you believe you experienced a time when you realized that your values, beliefs, opinions, or expectations (for example, how you viewed your work responsibilities or roles as an academic librarian) changed? yes no i'm not sure   16. describe what happened when you realized your values, beliefs, opinions, or expectations about your instructional responsibilities had changed.   17. did any of the following individuals influence this change? check all that apply. interaction with a student or students support from a colleague a challenge from a colleague support from another librarian a challenge from another librarian support from a subject area faculty member a challenge from a subject area faculty member support from a mentor a challenge from a mentor support from a supervisor a challenge from a supervisor support from my library/institution’s administration a challenge from my library/institution’s administration other: ________________________________________________ no individual influenced my experience of change   18. did any specific learning experience or resource influence this change? if so, check all that apply. taking a class or classes in library school taking a class or classes in another graduate program teaching in a face-to-face course teaching in an online course teaching in a blended/hybrid course observing other academic librarians’ instructional practices receiving feedback from other academic librarians on your teaching practices observing subject area faculty’s instructional practices receiving feedback from subject area faculty on your teaching practices receiving feedback from students who participated in your instruction completing a self-assessment of your teaching practices writing about your teaching practices in a reflection journal or other personal format writing about your teaching practices for publication attending meetings, workshops, or trainings within your normal working environment attending professional meetings, conferences, or workshops outside of your normal working environment participating in online webinars or seminars reviewing guidelines, standards, or other documents from professional organizations reading scholarly literature on information literacy instruction reading scholarly literature on the scholarship of teaching and learning other ________________________________________________ no experience influenced the change i experienced   19. did any significant professional event influence the change? if so, check all that apply. completion of library graduate program completion of other graduate program first professional job after graduate school change of job loss of job change in job responsibility or duties other ________________________________________________ no professional event influenced the change i experienced   20. think back to when you first realized that your views or perspective had changed. what did your professional life have to do with the experience of change? [free response]   21. would you characterize yourself as someone who usually thinks back over previous decisions or past behavior? yes no   22. would you characterize yourself as someone who reflects upon the meaning of your professional experiences for your own purposes? yes no   23. which of the following factors have been a part of your instructional work as an academic librarian? please select all that apply. interaction with a student or students support from a colleague  a challenge from a colleague support from another librarian a challenge from another librarian support from a subject area faculty member a challenge from a faculty member support from a mentor a challenge from a mentor support from a supervisor a challenge from a supervisor  taking a class or classes in library school taking a class or classes in another graduate program teaching a face-to-face class session teaching or providing instruction for an online course observing other academic librarians’ instructional practices receiving feedback from other academic librarians on your teaching practices observing subject area faculty’s instructional practices receiving feedback from subject area faculty on your teaching practices receiving feedback from students who participated in your instruction completing a self-assessment of your teaching practices writing about your teaching practices in a reflection journal or other personal format writing about your teaching practices for publication attending professional meetings, conferences, or workshops outside of your normal working environment attending meetings, workshops, or trainings within your normal working environment participating in online webinars or seminars reviewing guidelines, standards, or other documents from professional organizations reading the scholarly literature on information literacy instruction reading the scholarly literature on the scholarship of teaching and learning other ________________________________________________ none of these have been factors of my instructional work as a librarian   complete this survey   thank you for completing this survey! would you be willing to participate in a virtual follow-up interview? if so, please include your first and last name as well as an email address where you can be reached during the summer months.   name ________________________________________________   email address ________________________________________________   individuals who qualify to participate in the follow-up interviews will be selected at random.     this survey instrument was published in:   hess, a. n. (2018) transforming academic library instruction: shifting teaching practices to reflect changed perspectives. lantham, md: rowman & littlefield.   king (1997, 2009) retains the copyright to the original learning activities survey.     article   e-journal metrics for collection management: exploring disciplinary usage differences in scopus and web of science   katherine chew research/outreach services librarian for the health sciences libraries university of minnesota libraries minneapolis, minnesota, united states of america email: chewx002@umn.edu   mary schoenborn liaison, humphrey school of public affairs & carlson school of management university of minnesota libraries minneapolis, minnesota, united states of america email: hawki003@umn.edu   james stemper organizational data strategist university of minnesota minneapolis, minnesota, united states of america email: stemp003@umn.edu   caroline lilyard business, economics, & global studies librarian university of minnesota libraries minneapolis, minnesota, united states of america email: lily@umn.edu   received: 5 feb. 2016     accepted: 26 apr. 2016       2016 chew, schoenborn, stemper, and lilyard. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – the purpose was to determine whether a relationship exists between journal downloads and either faculty authoring venue or citations to these faculty, or whether a relationship exists between journal rankings and local authoring venues or citations. a related purpose was to determine if any such relationship varied between or within disciplines. a final purpose was to determine if specific tools for ranking journals or indexing authorship and citation were demonstrably better than alternatives.   methods – multiple years of journal usage, ranking, and citation data for twelve disciplines were combined in excel, and the strength of relationships were determined using rank correlation coefficients.   results – the results illustrated marked disciplinary variation as to the degree that faculty decisions to download a journal article can be used as a proxy to predict which journals they will publish in or which journals will cite faculty’s work. while journal access requests show moderate to strong relationships with the journals in which faculty publish, as well as journals whose articles cite local faculty, the data suggest that scopus may be the better resource to find such information for these journals in the health sciences and web of science may be the better resource for all other disciplines analyzed. the same can be said for the ability of external ranking mechanisms to predict faculty publishing behaviours. eigenfactor is more predictive for both authoring and citing-by-others across most of the representative disciplines in the social sciences as well as the physical and natural sciences. with the health sciences, no clear pattern emerges.   conclusion – collecting and correlating authorship and citation data allows patterns of use to emerge, resulting in a more accurate picture of use activity than the commonly used cost-per-use method. to find the best information on authoring activity by local faculty for subscribed journals, use scopus. to find the best information on citing activity by faculty peers for subscribed titles use thomson reuters’ customized local journal use reports (ljur), or limit a web of science search to local institution. the eigenfactor and snip journal quality metrics results can better inform selection decisions, and are publicly available. given the trend toward more centralized collection development, it is still critical to obtain liaison input no matter what datasets are used for decision making. this evidence of value can be used to defend any local library “tax” that academic departments pay as well as promote services to help faculty demonstrate their research impact.   introduction   for years, academic librarians faced with static or reduced collection budgets have searched for e-journal usage metrics that would best inform difficult retention decisions. download statistics do not tell the whole story; an article download does not indicate whether it is later read or cited. the cost-per-use figures derived from them rarely resonate with faculty when it is “their” journal on the chopping block. further, each usage metric has unique limitations. login data from openurl link resolvers lose track of the user when the user reaches the publisher site, and thus may not capture all of the eventual downloads. counter-compliant download data is not available from all publishers, especially small societies. journal rankings such as impact factor are based on a short time interval that does not necessarily reflect the citation or publishing patterns of all disciplines. such rankings are also not available for many social sciences or arts and humanities journals, and can be manipulated to some extent. ideally, librarians would like to connect available usage measures to research outcomes in a valid and meaningful way.   the authors sought to compare the available metrics and determine the value users assign to a collection through their decisions about the journal articles they download and the journals they publish in, as well as the value inherent in their peers’ decisions to cite faculty journal articles.   literature review   the centre for information behaviour and the evaluation of research (ciber) at university college london studied publishing patterns of researchers in six disciplines at eight uk universities. they found a strong positive correlation between the use of e-journals and successful research performance. institutions varied in use more than disciplines but they discovered that the journals accounting for the top five percent of use could vary by as much as 20% between the six disciplines (jubb, rowlands, & nicholas, 2010). regardless of disciplinary and institutional variance, electronic journal usage had positive outcomes.   an ongoing issue in collection analysis is knowing which metrics to use to evaluate electronic journal usage and value. the california digital library’s weighted value algorithm (cdl-wva) put into practice the ideals underlying the results of their white paper (university of california libraries’ collection development committee, 2007). anderson (2011) demonstrated a tool in which the selector can determine which publishers offer the highest value for money to the academic department, and also how that publisher’s demonstrated value changes on the value scale. this kind of dashboard gives selectors a tool customized to their subject areas. customized, easily used tools such as this are increasingly important to ensure broad adoption of metrics evaluations.   the university of memphis adapted california digital library’s journal value metrics and compared them with faculty decisions on which journals to cancel. knowlton, sales, and merriman (2014) found that faculty selection of journals differed significantly from bibliometric valuation, and that “higher cdl-wva scores are highly associated with faculty decisions to retain a title, but lower cdl-wva scores are not highly associated with decisions to cancel” (p. 35). to explain the difference, they suggested that special faculty research needs, institutional pressure to retain titles for accreditation, and a focus on teaching over research by faculty all lead to certain journals selected for retention while not frequently being downloaded or cited. these findings echo the authors’ findings here in that the metrics valued by faculty are not always those used by librarians.   are metrics different when assessing consortial package deals? do limitations surface when assessing the value of “big deals”? the canadian research knowledge network (crkn) also adapted the cdl approach. crkn assessed whether the value of a consortial package of journals stayed the same despite variation in institutional characteristics. they found the top quartile was largely composed of the same journals, regardless of the individual characteristics of the institutions. the overlap of journal titles was around 90%. similarly, the bottom quartile for each school had an overlap of titles around 90%. consortia could move from a big deal to a smaller core package and still meet the needs of most members (jurczyk & jacobs, 2014). appavoo (2013) also found that when comparing traditional cost-per-use against the cdl approach for the top 100 journals, “the journal value metric method returned a wider variety of disciplines in the results, while the use-based metric returned primarily journals in the stm disciplines” (slide 11, notes). schools using cost-per-use to reduce costs of journal packages need to be careful not to disadvantage users in the social sciences and humanities.   the carolina consortium also analyzed its big deals, and found that the utility of cost-per-use metrics is mitigated by the fluid nature of the industry (e.g., title changes, publisher mergers, etc.). this should be just one of a suite of decision tools (bucknall, bernhardt, & johnson, 2014). this reinforces our findings that various metric analyses must be employed for meaningful results.   several studies document differences among subject disciplines as to how closely download and citation behaviours are related. the university of mississippi examined publications by the business school faculty to see what they cited. the conclusion was that local citation patterns vary widely, even among departments in one discipline, thus necessitating analysis at the local level (dewland, 2011). variations exist among departments, let alone disciplines.   in the health sciences, a comparison of vendor, link resolver, and local citation statistics revealed a high positive correlation between the three data sets (de groote, blecic & martin, 2013). in another study, physicians from norway examined the 50 most viewed articles from five open access oncology journals, and concluded that more downloads do not always lead to more citations (neider, dalhaug & aandahl, 2013).   fields in which faculty publish in multidisciplinary journals, such as public administration and public policy, provide additional challenges. in these cases, how are “good journals” defined? the authors discuss measuring and ranking article output in the discipline and the effect on analysis (van de walle & van delft, 2015). the complexity of arrangements, e.g., single purchase electronic journals, big deal packages, and interdisciplinary journals and fields, necessitate a more thorough approach and point to a variety of metric analysis methods being more useful than a simple cost-per-use model.   another complicating factor for this endeavour is open access. the university of illinois at chicago noted that a focus on article downloads is indeed complicated by open access. subject repositories such as arxiv, pubmed central, repec, and ssrn can draw users without leaving a counter trail (blecic, wiberley, fiscella, bahnmaier-blaszczak & lowery. 2013), skewing analysis results. in the future, other metrics may become more significant. a 2012 study sampled 24,331 articles published by the public library of science (plos) and tracked their appearance in tools such as web of science citation counts, mendeley saves, and html page views. as an indicator of how open access is not only changing how researchers read and cite but how they share articles, the authors found that 20% of the articles were both read and cited, while 21% were read, saved, and shared (priem, piwowar, & hemminger, 2012).   the pareto principle is often mentioned in journal usage studies. this is also known as the 80/20 rule, and states that, for many events, roughly 80% of the effects come from 20% of the causes (nisonger, 2008). an example is a citation analysis of atmospheric science faculty publications at texas a&m university. it found 80% of cited journal articles were from just 8% of the journal titles (kimball, stephens, hubbard & pickett, 2013). ten years earlier, one of the authors of this study and a colleague found a larger percentage of titles, roughly 30%, comprising 80% of downloads when analyzing use of all subjects in five large journal publishers (stemper & jaguszewski, 2004). a small percentage of the total journals were most heavily cited and downloaded in both instances. taken together, one could conclude that online journals lead users to read more but not necessarily to cite more journals. the variety of metrics cited here reflect our findings that collecting and correlating authorship and citation data allows patterns of use to emerge, resulting in a more accurate picture of activity. the development of more complex analysis will inform collection development in meaningful ways in the future of academic libraries.   aims   the authors had collected traditional link resolver and publisher statistics for years, and to facilitate a study on e-journal metrics the created a comprehensive “uber file”, one which combined all selected subjects and publishers and allowed sorting by title or subject fund number. they purchased and included a customized dataset from thomson reuters’ web of science, the local journal use reports (ljur), that showed which journals were cited by university of minnesota (u-mn) authors, the journals in which they published, and which journal venues with u-mn authors were cited by others. the authors felt they could no longer rely solely on download statistics, which while convenient and comprehensive, are unfavourable to disciplines that do not depend heavily on journal articles and are much more favourable to disciplines that do, as shown in table 1. plus, as noted, there was a wish to focus more on user outcomes, which made citation data attractive.     table 1 top twenty journals accessed by university of minnesota students, staff and faculty 20092012 journal title number of downloads science 44,114 nature 39,407 jama 33,126 the new england journal of medicine 32,467 the lancet 22,676 harvard business review 21,275 journal of the american chemical society 16,838 proceedings of the national academy of sciences of the united states of america 14,666 pediatrics 12,485 scientific american 12,213 health affairs 10,897 annals of internal medicine 10,757 neurology 10,406 american journal of public health 8,937 journal of personality and social psychology 8,515 child development 8,011 critical care medicine 7,622 ecology 7,534 medicine and science in sports and exercise 7,263 journal of clinical oncology 7,153     the jumping-off point was an unpublished study by the wendt engineering library at the university of wisconsin-madison, a peer institution, which surveyed faculty to gauge the importance of various criteria in journal cancellations. the journals that engineering faculty cited the most in their articles were ranked as most important, followed by journals that they published in, then in decreasing order, usage statistics, impact factors, citation by peers, ending with the metric of cost-per-use, the one most used by librarians (helman, 2008). due to perceived survey fatigue by u-mn faculty, a different approach was developed using university of wisconsin-madison’s findings to guide the investigation. the first phase of the authors’ investigation addressed u-mn faculty’s citation patterns (chew, stemper, lilyard, & schoenborn, 2013). this second phase addresses their choice of publication venue and external citations to their articles in these journals.   methods   the design of the study was heavily based on the california digital library’s (cdl) weighted value algorithm framework project, which assesses user value in three overall categories:   1.       utility: usage statistics and citations 2.       quality: represented by impact factor and source normalized impact per paper (snip) 3.       cost effectiveness: cost-per-use and the cost-per-snip   the weighted value algorithm combines these aspects of use when assessing the journal’s value in the institutional context while also factoring in disciplinary differences (wilson & li, 2012). adapting this approach, each journal’s value would be assessed by a) local author decisions to publish there, b) external citations to institutional authors, and c) cost effectiveness (via downloads and citations). in addition to cdl’s categories of user-defined value-based metrics, data was added on u-mn users’ departmental affiliations to assess any disciplinary differences. these “affinity strings,” attached to a user’s resource login, are generated by u-mn university’s office of information technology with information from the university’s human resources management system. all u-mn students, staff, and faculty are assigned affinity strings that are based on his or her area of work or study.   the study framed the following questions to try to ascertain what faculty actually value with regards to the journal collection:   1.       utility or reading value: does locally-gathered openurl click data combined with affinity string data provide a “good enough” departmental view of user activities, such that counter-compliant publisher download data is expendable? 2.       quality or citing value: is eigenfactor or snip an adequate substitute for impact factor as a measure of faculty citation patterns? 3.       cost effectiveness or cost value: how should these reading and citing values be combined with cost data to create a “cost-per-activity” metric that meaningfully informs collection management decisions? 4.       lastly, to what extent could unique local usage data be leveraged: do departments vary greatly in their journal downloading, and do any of the measures predict which journals u-mn faculty publish in and which of these articles will get cited by their peers?     table 2 subjects included in study major discipline department/school number of subscribed titles arts & humanities history 48       social sciences accounting 14 finance 22 management 29 marketing 15 public affairs 40       physical sciences chemistry 160       life sciences forestry 51       health sciences hematology 34 pediatrics 64 pharmacy 99 nursing 115     the data   the data for the project was collected from nearly 700 e-journals that were licensed for system-wide use, owned by, or accessible to the u-mn libraries users. in order to discover whether or not there may be any disciplinary differences in local faculty download or authorship behaviours, or patterns of external citing by their peers, 12 subjects were chosen from four major disciplinary areas as defined by u-mn libraries’ organizational structure. in all but three cases, the authors were either departmental liaisons or the previous subject coordinator. table 2 lists the subjects and the number of subscribed titles that were funded for that subject. subject relevant titles that were excluded from this project included those that were part of consortia purchases or centrally-funded full-text databases such as ebsco’s academic search premier, where resource costs could not be parsed out to individual titles.   to gain an understanding into a journal’s usage patterns, researchers used four years of usage data spanning from 2009 through 2012, along with 2-4 years of citation data, and journal impact metrics from 2012. these were and analyzed by individual subject, and then combined in a single spreadsheet for comparative analysis. the data variables collected, sorted into cdl-wva categories, are shown in table 3.   the median figures were calculated for each metric in order to reduce the influence of outlier results, except for impact factor where 5-year scores that were available for the project titles were used. data could not be collected for all of the variables for every title, as not every publisher is counter-compliant, nor are there impact factor, eigenfactor, ljur, or snip data available for every title. for the sciences (with the exception of nursing) at least three-quarters of the titles had journal ranking metrics available; eigenfactor scores were available for 84-94% of the titles, snip scores for 92-100% and impact factor for 78-94% of the titles, where the 5-year impact factor was the least available. the difference was only significant in nursing, where the gap was counted across 13 titles. conversely, the social sciences had a much lower comparative journal ranking metrics. eigenfactor scores were available for only 36-64% of the titles and impact factor for 31-55% of the titles. on the other hand, the social sciences did well with snip, ranging from 70-95% of the titles available (table 4).     table 3 e-journal metrics collected cdl-wva category metric years utility: usage article view requests, as reported by the library’s openlink resolver sfx 2009-2012 utility: usage article downloads, as reported by publisher countercompliant reports 2009-2012 utility: citation university of minnesota a) authorship and b) citations to these locally authored articles, from thomson reuter’s local journal use reports (ljur) 2009-2010 utility: citation university of minnesota a) authorship and b) citations to these locally authored articles, elsevier’s scival/scopus (2009-2012) 2009-2012 quality ● journal citation reports (jcr) five year impact factor (if) ● eigenfactor scores ● elsevier’s source normalized impact per paper (snip) 2012 cost effectiveness ● via cost per download ● via cost per ranking (ebsco subscription price divided by sfx /counter and impact factor/eigenfactor/snip as appropriate for each subject) 2013 note. due to the significant yearly cost of a purchase of the ljur dataset only the 2009-2010 dataset was available. elsevier’s scival is an institutional level research tool that provides a snapshot of institutional research performance at the institutional and departmental level. information provided by scival is drawn from the scopus dataset.     a pearson’s correlation analysis was chosen to examine if there was any relationship, positive or negative, between selected journal metrics, whether or not there were any disciplinary differences between the various metrics, and the potential significance or strength of those relationships. the goal was to find which correlations, and thus which metrics, provided the best “goodness of fit,” i.e., which best explained past patron use of e-journals as well as best predicted their future use.   data analysis was done using excel’s correl function. in conjunction with the correlation coefficient, “r”, the coefficient of determination, which is the square of r and is reported as r-squared, was calculated. all of the correlations’ f-test p-values were less than 2.2e-16 (2 x 10-16), therefore statistically significant. r-squared is often expressed as a percentage when discussing the proportion variance explained by the correlation. though there can be a range of interpretation depending on the discipline, it is generally accepted that within the social sciences, or when looking at correlations based on human behaviour, an r<0.3 is considered a low or weak correlation, 0.3-0.5 modest or moderate, 0.5-1.0 strong or high correlations, with anything over 0.90 a very high correlation (table 5), and r2 values anywhere between 30-50% are considered meaningful (meyer, et. al., 2001). a wide variety of correlations were run to provide comparison data points.     table 4 percentage of subscribed journal titles that have impact factors, eigenfactors or snip department / school no. of subscribed titles % 5-year impact factor % eigenfactor % snip hematology 34 94% 94% 100% pharmacy 99 91% 92% 95% pediatrics 64 80% 86% 92% nursing 115 44% 56% 86% chemistry 160 91% 91% 94% forestry 51 78% 84% 96% history 48 31% 38% 83% marketing 15 40% 53% 80% management 29 55% 55% 93% finance 22 45% 64% 95% accounting 14 36% 36% 93% public affairs 40 55% 60% 70%   table 5 range of pearson values for study correlation negative positive none -0.09 to 0.00 0.0 to 0.09 low or weak -0.3 to -0.1 0.1 to 0.3 moderate or modest -0.5 to -0.3 0.3 to 0.5 strong -1.0 to -0.5 0.5 to 1.0   table 6 comparison of indexing of locally-held titles in web of science and scopus department / school no. of subscribed titles no. of titles indexed in scopus % of titles indexed in scopus no. of titles indexed in web of science % of titles indexed in web of science nursing 115 111 97% 54 47% pharmacy 99 98 99% 92 93% pediatrics 64 64 100% 56 86% hematology 34 34 100% 31 91% chemistry 160 156 98% 154 96% forestry 51 49 96% 49 96% history 48 44 92% 41 85% finance 22 19 86% 13 59% accounting 14 13 93% 4 29% public affairs 40 31 78% 22 55% marketing 15 13 87% 5 33% management 29 28 97% 17 59%   table 7 comparison of citing of u of m authors in locally-held titles in web of science and scopus department / school no. of subscribed titles scopus: u of m authors cited % of titles cited in scopus web of science: u of m authors cited % of titles cited in web of science nursing 115 26 23% 66 57% pharmacy 99 71 72% 94 95% pediatrics 64 45 69% 55 85% hematology 34 21 62% 33 97% chemistry 160 87 54% 144 90% forestry 51 22 43% 43 84% history 48 4 8% 23 48% finance 22 7 32% 15 68% accounting 14 3 21% 4 29% public affairs 40 15 38% 23 58% marketing 15 6 40% 7 47% management 29 8 28% 21 72%   in order to determine “utility”, sfx link resolver and counter data were correlated with both the ljur for local authorship and local citing patterns and scival/scopus data for local authorship and local citing patterns. for “quality”, ljur authoring/citing and scival/scopus authoring/citing data were correlated with impact factors, eigenfactors, and snip. the r² values that resulted from the correlations were then inserted into bar charts for subject comparisons.   indexing selections by publishers.   the two primary indexes used as a basis for the “utility” and “quality” analysis, web of science and scopus, were also analyzed. the question was whether web of science or scopus fared better in tracking the publishing activity of u-mn faculty. the surprising discovery was that neither scopus nor web of science could function as a single data source (harzing, 2010). in answering the question of which database was the better metric data source, it turned out that scopus provided better authoring data, because it indexed more of u-mn subscribed titles than web of science, ranging from a low of   78% for public affairs titles to a high of 100% of pediatrics titles, compared to web of science, with a low of 47% for nursing titles to a high of 95% for chemistry titles (table 6). on the other hand, web of science provided better citing data, because it contains citation data dating back to the 1900s and includes citation data from journals that they do not regularly index, whereas the majority of scopus citing data only goes back to 1996 and only includes titles that they index. web of science ranged from a low of 29% for accounting titles to a high of 97% for hematology titles, compared to scopus, with a very low 8% for history titles and a modest highest 72% for pharmacy titles (table 7).   results   authorship decisions by u of minnesota authors, or, “where do i publish my article?”   the first question to answer was whether the journals in which u-mn faculty choose to publish are also the journals that are most downloaded by u-mn users. overall, the social sciences and humanities had several moderate to strong positive correlations between downloads and where faculty chose to publish. journals for finance and accounting were found to have a strong relationship in both web of science and scopus. history shows the greatest variation between downloads and choice of authoring venue, with web of science at about 75%, compared to scopus at 8% predictive (see figure 1). pediatrics shows the greatest variation in the health sciences between downloads and choice of authoring venue, with scopus at about 65% and web of science at about 5% predictive.       figure 1 downloads and authorship choice based on sfx title clicks correlated with u-mn authors titles in web of science or scopus.     the next question to answer was whether the journals in which u-mn faculty chose to publish are also the journals that external rating services consider being of the highest quality. using scopus authoring, figure 2 illustrates initial results. in the social sciences and humanities subjects, the data show that no one impact measure stood out as most predictive overall. accounting and management both show strong correlations for all three measures, while, interestingly, a weak negative relationship was found for marketing.   in the physical and health sciences, multiple weak or negative relationships are evident. the negative correlations, while low, may suggest there is close to no correlation between those journals that faculty in nursing, hematology and pharmacy chose to publish in and their value rankings. on the other hand, in pediatrics all of the value metrics correlations are either moderate or strong, suggesting that impact factors or similar value measures may play a role in faculty publishing decisions.     figure 2 journal ranking and authorship choice using snip, eigenfactor, and impact factor score correlated with u-mn authored titles in scopus.     comparatively, in the web of science authoring results shown in figure 3, the social sciences and humanities impact factor rankings overall were weak to moderate predictors, except for history where impact factor is a strong predictor. eigenfactor on the other hand, was the overall stronger predictor, in the subjects of history, finance, accounting, and very strong in marketing.   finally, snip proved to be a better predictor only for finance.   the web of science authoring results in the physical and health sciences subjects illustrate a very different, far less stable pattern of correlations. here eigenfactor is most predictive only for chemistry, and all other web of science authoring relationships are moderate at best, but mostly weak or negative. in summary, the data comparing a discipline’s impact measure and its faculty journal authoring choices suggests that impact factor rankings are weak predictors about half the time, but the strongest predictors are in the humanities and social sciences where eigenfactor may be “good enough”.   citing decisions by peers: is this u-mn article worth citing?   how are u-mn faculty researchers viewed by their peers? to put it another way, were the journals that cited u-mn faculty’s research also the most downloaded journals by u-mn users? among the disciplines analyzed, the external citing patterns for many disciplines, including public affairs, accounting, finance, management, hematology, pediatrics, forestry, chemistry all showed strong relationships with either scopus or web of science, and as noted in figure 4, a few instances of disciplinary relationship strength in both tools. conversely, history, marketing, and pharmacy had weak-to-moderate citing correlations in both web of science and scopus. finally, nursing results show the greatest variability, where web of science is strong and scopus is a negative relationship. the results show web of science citing correlated stronger in the majority of disciplines except for hematology, pediatrics, and accounting, fields where scopus is a stronger predictor.       figure 3 journal ranking and authorship choice using snip, eigenfactor, and impact factor score correlated with u-mn authored titles in web of science.     also analyzed were citing decisions by external authors and impact measures in web of science. were the journals that cited u-mn faculty’s research also the journals that external rating services consider to be of the highest quality? as figure 5 illustrates, the social sciences and humanities results present multiple strong correlations in management, accounting, and finance. public affairs and marketing each have one strongly predictive value measure, snip and eigenfactor respectively. overall, the value metrics that are most predictive are snip and eigenfactor.   in the natural, physical, and health sciences, common patterns are far less pronounced, though for forestry, chemistry, and pediatrics, eigenfactor is strongest. beyond these subjects, web of science citing shows moderate, weak or negative relationships to the three impact value metrics.       figure 4 downloads and others citing u-mn based on sfx title clicks correlated with cites to u-mn authored titles in web of science or scopus.   figure 5 snip, eigenfactor, and impact factor scores correlated with cites to u-mn authors titles in web of science.   figure 6 snip, eigenfactor, and impact factor score correlated with cites to u-mn authored titles in scopus.     using scopus citing data, almost all disciplines have at least one impact measure with strong correlation, but no one measure stands out as most predictive overall. figure 6 shows multiple negative or weak relationships are evident when looking at peer citing decisions in finance, pharmacy, and nursing. and some of the strongest relationships are found with impact factor for both public affairs and marketing. on the other hand, eigenfactor is strongly predictive with peer citing in history, accounting, management, forestry, chemistry, and pediatrics. meanwhile, snip shows strong relationships in hematology, pediatrics, management, and accounting.   finally, these results provide evidence to answer the question of comparative impact measure at the journal discipline level. while many disciplines have multiple strong correlations, many also have weak or negative relationships. thus, discipline does matter in terms of overall impact measure decisions, though patterns do emerge for some fields where the discipline result may be sufficient for a group of subjects, such as business, as we found for eigenfactor in web of science. the same though cannot be said for health subjects where a far more nuanced approach may be required.   discipline usage behaviour   what could be the possible explanation behind low to barely moderate, or even the negative correlations with regards to authorship, citing behaviour, or relationships with value metrics such as impact factor? is there something in the usage behaviour of discipline specific users that can provide insight? one way to understand these differences is to look at u-mn’s affinity string data. affinity strings provide some insight into usage patterns at college or school level, as well as degree or subject discipline level. affinity string data reveals who is accessing u-mn electronic resources without identifying a specific person.     table 8 affinity string usage of harvard business review 2009-2012 affinity string status college department /school no. logins 2009-2012 tc.grad.csom.bus_adm.emba graduate student carlson school of management business admin 2734 tc.grad.gs.humrsrc_ir.ma graduate student graduate school human resource development 507 tc.grad.gs. graduate student graduate school general 485 ahc.pubh.hcadm.mha graduate student academic health center public health 338 tc.grad.csom.bus_adm.dmba graduate student carlson school of management business admin 323 ahc.grad.nurs.d_n_p graduate student academic health center nursing 212 tc.grad.cehd.humrsrcdev.m_ed graduate student education & human develop human resource development 156 tc.grad.csom.humrsrc_ir.ma graduate student carlson school of management human resources 156 tc.grad.gs.strat_comm.ma graduate student graduate school strategic communication 133 tc.grad.gs.workhumres.phd graduate student graduate school work & human resources education 106 ahc.staff.pubh staff academic health center public health 93 tc.grad.cehd.humresdev.humresd_gr graduate student education & human develop human resource development 93 tc.grad.gs.mgmt_tech.ms_m_t graduate student graduate school management of technology 87 tc.ugrd.csom.mktg.bs_b.cl2011 undergraduate student carlson school of management marketing 86 tc.ugrd.fans.env_scienc.bs.nas undergraduate student food, agricultural & natural resource sciences environmental sciences 84 ahc.staff.med staff academic health center medicine 81 tc.grad.gs.humrsrcdev.m_ed graduate student graduate school human resource development 81 tc.grad.csom.bus_adm.cemba graduate student carlson school of management business admin 73 tc.grad.gs.publ_pol.m_p_p graduate student graduate school public policy 57 tc.grad.cehd.workhumres.phd graduate student education & human develop work & human resources education 57     sometimes this data reveals rather surprising things. for instance, table 8 shows that among the top twenty users of the harvard business review are graduate school nursing students, as well as public health and medical school staff. so decisions about the harvard business review would not only impact the academic business community, but the health sciences as well.     figure 7 nursing staff download activity versus nursing faculty download activity   figure 8 pharmacy staff download activity versus pharmacy faculty download activity.     within a particular school, there can be differences in what e-journals are accessed. nursing or pharmacy staff and faculty (which includes research assistants, fellows, and phd candidates) access a wide variety of journals outside of their immediate disciplines. research staff download to a much greater extent than faculty, possibly because they are the ones doing the bulk of the background work for grants, publications, or curriculum instruction (figures 7 & 8). so decisions about any health sciences/bio-sciences titles could impact how the nursing school or college of pharmacy would be able to conduct research, apply for grants, or build curriculum content.   publication practices   when looking at where nursing or pharmacy authors chose to publish, the vast majority publish within their disciplinary journals. however, when looking through a list of articles that have the highest citation counts that include nursing or pharmacy authors, the top journals are not nursing or pharmacy journals, but well-known medical titles, such as the new england journal of medicine or circulation. examining the author list from these articles reveals the increasing interdisciplinary nature of research, where the nursing or pharmacy author is one member of a team.   selected disciplinary evidence: visualizing the data at the discipline level   the results illustrate that disciplinary trends exist. can a more careful look at specific funds determine how these data actually may impact librarian selection decisions, or certainly the discussions that surround selection/deselection? to draw out patterns in the data, and hopefully tease out a more meaningful story, the data was visualized using tableau software.     figure 9 journals selected using the finance fund.     figure 9 represents titles selected using the finance fund. as noted above, web of science was found to be a strong predictor for both authoring in and peer citing for this subject. the addition of the authoring data shows journal of banking & finance and journal of financial intermediation as titles with comparatively few downloads but marked faculty authoring activity. couple this with the additional peer citing data from the last column and it becomes clear that the journal of empirical finance, journal of financial markets, and the journal of financial intermediation are titles with higher local impact to u-mn faculty than downloads alone would suggest.   figure 10 highlights titles selected using the public affairs fund. seen here are international public management and journal of transport geography as titles with a lower level of downloads, but marked authoring and peer citing activity. these additional journal level views provide a richer set of data from which to analyze collections.       figure 10 journals selected using the public affairs fund.     it is enlightening to consider how “weak” “moderate” and “strong” correlations play out in practice. through comparison, the next couple of figures offer some insight. for example, data for authorship and downloads in history are presented in figure 11 because of the previously noted gap between web of science authoring (at 75% correlation), shown in the first column, and scopus authoring (at only 8%) shown in the second column.   comparing the ljur and scopus columns for journals where data exists for both, web of science results are often higher than scopus, but not always. noticeable outliers include radical history review and historical methods with stronger scopus authorship.   forestry presents another view of variability (figure 12). presented are downloads in relation to citing by peers of u-mn authored works. the findings show web of science is the better predictor at 75% to scopus ‘moderate 35%.   while many of the same journals are represented as having been cited by peers based on data for both web of science and scopus, what is remarkable is the degree of variation. certainly, web of science tells a strong story. scopus tells a story too, just not as compelling.   the final case study looks at the relationship between journal ranking measures and scopus authoring in public affairs, the tools with the more predictive authoring result. figure 13 shows impact results ranked by snip, the most predictive of the three measures in scopus authoring. ranked in descending order of snip values, scopus does consistently provide comparatively stronger authoring relationships than either impact factor or eigenfactor.       figure 11 journals selected using the history fund.   figure 12 journals selected using the forestry fund.   figure 13 public affairs fund titles and impact measures.     discussion   as both login demographics and interdisciplinary use are collected, correlated evidence of patterns of use emerge, resulting in a more accurate picture of activity. the results suggest practical ways to inform selection decisions. web of science provides more complete information on citing activity by faculty peers for subscribed titles, while scopus provides better information on authoring activity by local faculty for subscribed journals. one solution is to use both the web of science local journal use reports and scopus tools. if ljur is too pricey but one subscribes to web of science, the latter can be searched by institutional affiliation (though this can be labour-intensive).   given the trend toward more centralized collection development, it is still critical to obtain liaison/subject coordinator input no matter what datasets are used for decision making. not only do liaisons have the deepest understanding of disciplinary level use and quality, but as this research demonstrates, the “best fit” metric may vary both within a broad discipline category as well as between disciplinary categories.   such analysis also provides proactive evidence of value to the academy. the process of looking at impact provides the same frame or structure across disciplines, often with very different outcomes. furthermore, this evidence of value can be used to defend any local library “tax” that academic departments pay, as well as to promote services that help faculty demonstrate their research impact, e.g., for tenure portfolios.   conclusion   collecting and correlating authorship and citation data allows patterns of use to emerge, resulting in a more accurate picture of activity than the more often used cost-per-use. to find the best information on authoring activity by local faculty for subscribed journals, use scopus. to find the best information on citing activity by faculty peers for subscribed titles, use thomson reuters’ customized ljur report, or limit a web of science search to local institution. the eigenfactor and snip journal quality metrics results can better inform selection decisions, and are publicly available. given the trend toward more centralized collection development, it is still critical to obtain liaison input no matter what datasets are used for decision making. this evidence of value can be used to defend any local library “tax” that academic departments pay as well as promote services to help faculty demonstrate their research impact.   references   anderson, i. (2011, october). shaking up the norm: journal pricing and valuation. paper presented at the meeting of the arl-cni 21st century collections forum, washington, d.c. retrieved from http://www.arl.org/storage/documents/publications/ff11-anderson-ivy.pdf appavoo, c. (2013). what’s the big deal? collection evaluation at the national level [powerpoint slides]. retrieved from http://connect.ala.org/files/crkn_ala_2013%20bullets.ppt   blecic, d. d., wiberley, s. e., fiscella, j. b., bahnmaier-blaszczak, s., & lowery, r. (2013). deal or no deal? evaluating big deals and their journals. college & research libraries, 74(2), 178-194. http://dx.doi.org/10.5860/crl-300 bucknall, t., bernhardt, b., & johnson, a. (2014). using cost per use to assess big seals. serials review, 40(3), 194-196. http://doi.org/10.1080/00987913.2014.949398   chew, k., stemper, j., lilyard, c., & schoenborn, m. (2013, october). user-defined value metrics for electronic journals. paper presented at the 2012 library assessment conference: building effective, sustainable, practical assessment, charlottesville, va. washington, dc: association of research libraries. retrieved from http://purl.umn.edu/144461   de groote, s. l., blecic, d. d., & martin, k. (2013). measures of health sciences journal use: a comparison of vendor, link-resolver, and local citation statistics. journal of the medical library association,101(2), 110-119. retrieved from http://dx.doi.org/10.3163/1536-5050.101.2.006 dewland, j. c. (2011). a local citation analysis of a business school faculty: a comparison of the who, what, where, and when of their citations. journal of business & finance librarianship, 16(2), 145-158. http://dx.doi.org/10.1080/08963568.2011.554740   harzing, a. w. (2010). citation analysis across disciplines: the impact of different data sources and citation metrics [white paper]. retrieved from http://www.harzing.com/data_metrics_comparison.htm helman, d. (2008). wendt library – faculty collections survey 2008. unpublished manuscript, university of wisconsin-madison, madison, wi.   jubb, m., rowlands, i., & nicholas, d. (2013). value of libraries: relationships between provision, usage, and research outcomes. evidence based library and information practice, 8(2), 139-152. http://dx.doi.org/10.18438/b8kg7h jurczyk, e., & jacobs, p. (2014). what's the big deal? collection evaluation at the national level. portal: libraries and the academy, 14(4), 617-631. http://dx.doi.org/10.1353/pla.2014.0029   kimball, r., stephens, j., hubbard, d., & pickett, c. (2013). a citation analysis of atmospheric science publications by faculty at texas a&m university. college & research libraries, 74(4), 356-367. http://dx.doi.org/10.5860/crl-351 knowlton, s. a., sales, a. c., & merriman, k. w. (2014). a comparison of faculty and bibliometric valuation of serials subscriptions at an academic research library. serials review, 40(1), 28-39. http://dx.doi.org/10.1080/00987913.2014.897174   meyer, g. j., finn, s. e., eyde, l. d., kay, g. g., moreland, k. l., dies, r. r., ... & reed, g. m. (2001). psychological testing and psychological assessment: a review of evidence and issues. american psychologist, 56(2), 128-165. http://dx.doi.org/10.1037/0003-066x.56.2.128   nieder, c., dalhaug, a., & aandahl, g. (2013). correlation between article download and citation figures for highly accessed articles from five open access oncology journals. springerplus, 2(1), 1-5. http://dx.doi.org/10.1186/2193-1801-2-261   nisonger, t. e. (2008). the “80/20 rule” and core journals. the serials librarian, 55(1-2), 62-84. http://dx.doi.org/10.1080/03615260801970774 priem, j., piwowar, h. a., & hemminger, b. m. (2012). altmetrics in the wild: using social media to explore scholarly impact [arxiv preprint]. retrieved from http://arxiv.org/html/1203.4745v1   stemper, j. a., & jaguszewski, j. m. (2004). usage statistics for electronic journals: an analysis of local and vendor counts. collection management, 28(4), 3-22. http://dx.doi.org/10.1300/j105v28n04_02 university of california libraries’ collection development committee. (2007). the promise of value-based journal prices and negotiation: a uc report and view forward [white paper]. retrieved from http://libraries.universityofcalifornia.edu/groups/files/cdc/docs/valuebasedprices.pdf van de walle, s., & van delft, r. (2015). publishing in public administration: issues with defining, comparing, and ranking the output of universities. international public management journal, 18(1), 87-107. http://dx.doi.org/10.1080/10967494.2014.972482   wilson, j., & li, c. (2012, february 13). calculating scholarly journal value through objective metrics. cdlinfo news. retrieved from http://www.cdlib.org/cdlinfo/2012/02/13/calculating-scholarly-journal-value-through-objective-metrics/   microsoft word news4415 evidence based library and information practice 2008, 3:4 92 evidence based library and information practice news i³ conference preliminary announcement © 2008. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. mark your calendars for the 2nd annual canadian network for innovation in education (cnie) international conference 2009, to be held in ottawa, on, from 10 may 2009 – 13 may 2009. this exciting event will be hosted by the canadian healthcare association (cha). theme learning and technology: a capital idea! conference streams • building effective learning environments: innovations in instructional design and course development • the prescription for learning: integrating educational technology into the health system • breaking learning barriers: new technology and media applications to create accessible learning • k to 12 innovation: engaging the digital learner • transforming education: constructing a student centered world with open learning and distance education • leading the way: effective policies, partnerships and administrative practices with an expected attendance of over 400 national and international delegates working in the fields of educational technology, health education, k-12 education, multi-media design, and distance learning, the 2009 cnie international conference offers a unique opportunity for learning, networking and idea exchange. for more information, please visit the 2009 cnie international conference website <http://www.learningconference.ca/>. article   iterative chat transcript analysis: making meaning from existing data   steven baumgart head of memorial library public services university of wisconsin-madison libraries madison, wisconsin, united states of america email: steven.baumgart@wisc.edu   erin carillo information services librarian university of wisconsin-madison libraries madison, wisconsin, united states of america email: erin.carrillo@wisc.edu    laura schmidli information services librarian university of wisconsin-madison libraries madison, wisconsin, united states of america email: laura.schmidli@wisc.edu   received: 12 feb. 2016   accepted: 11 mar. 2016       2016 baumgart, carillo, and schmidli. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – in order to better contextualize library data about patron satisfaction with reference services, we analyzed an existing corpus of chat transcripts. having conducted a similar analysis in 2010, we also compared librarian behaviors over time.   methods – drawing from the library literature, we identified a set of librarian behaviors closely associated with patron satisfaction. these behaviors include listening to and understanding patrons’ needs, inviting patrons to use the service again, and providing instruction or completing a search for patrons. analysis of the chat transcripts included establishing a coding schema, applying these codes to individual chat transcripts, and analyzing these codes across the corpus of transcripts for frequency and correlation with other codes. the currently presented analysis used chat transcripts from the fall of 2013 and seeks changes in librarian behavior over time in order to gauge the success of establishing best practices and improving training standardization over the last three years.   results – the analysis shows that librarian behaviors have changed over time, pointing to what campus librarians are doing well, and that implementation of best practices at a campus level after the 2010 analysis may have increased these positive behaviors. the analysis also shows opportunities for further standardization and reinforcement of best practices.   conclusion – qualitative analysis of already-collected data serves as a model for other units and suggests areas for process improvement, including enhanced coder training and code schema design. further analysis of chat patrons’ questions is also warranted, including investigation of the relationship between subjectand location-specific questions and referrals.     introduction   twice each year, university of wisconsin-madison campus libraries participate in a public service data gathering week, during which each library is encouraged to record all public service interactions. these sweeps weeks occur during the tenth week of the fall semester and the seventh week of the spring semester. they generate a corpus of chat interactions that are recorded and retained. in 2010, the library’s reference assessment working group decided to analyze this data set to assess the quality of our campus reference service.   the reference assessment working group is composed of three to six librarians from different libraries on campus and is charged with coordinating each sweeps week and reporting about this data twice per year. this group decided to analyze chat transcripts in order to better contextualize and add qualitative data to this report. for the analysis, the group used chat transcripts from the general campus queue, which is the main point of entry into chat for uw-madison users. the main goal of this analysis was to discover patterns of librarian and patron behavior, particularly as our chat reference service had become increasingly busy over the previous years.   while this first analysis using 2010 chat transcripts included 28 codes, indicating a variety of behaviors, the main focus was to identify and measure librarian behaviors associated with patron satisfaction as identified at the university of south florida (kwon and gregory, 2007.) this focus was retained even as the coding schema was simplified for the 2013 iteration.   methods   text transcripts of chat interactions from the general campus library chat queue that occurred in the tenth week of the fall semester between november 4 and 10, 2013, were used in this analysis. a similar analysis was conducted in 2010 that also used general queue chat transcripts from the same week of the fall semester, from november 7 through november 13, 2010 (reference assessment working group, 2010).   preparing transcripts for analysis involved downloading transcripts from our chat software, converting transcripts to text files, and stripping transcripts of any identifying information. the transcripts were then individually imported into r using the rqda package, an open-source statistical analysis software program that was pre-loaded with all codes to be used in the analysis.   the analysis was conducted by four graduate students in the school of library and information studies who worked at three different campus library locations. prior to beginning to apply codes, these students participated in a one-hour group training and calibration session with the three librarians leading the analysis. student coders also had access to a screencast tutorial and were oriented to the software and process at their individual library.   in order to establish inter-rater reliability scores for each code, one of the principal investigators separately coded 10% of the transcripts, which were compared to those coded by students. cohen’s kappa (landis and koch, 1977; banerjee et al., 1999) was used to establish levels of reliability at the file and code levels in both the 2010 and 2013 analyses. the file level cohen’s kappa values ignore the frequency of codes and views a transcript as either tagged or not tagged with a specific code. the code level cohen’s kappa values take the frequency of codes into account, but it was not used in this analysis.   as in our previous analysis, we used common thresholds for cohen’s kappa to interpret the meaning of magnitude, establishing a four-part scale including poor (cohen’s kappa < 0.40), moderate (between 0.41 and 0.60), good (between 0.61 and 0.80), and very good (> 0.80). these values are represented in figure 1 using orange dots, where dots higher on the y-axis represent a higher level of agreement.   codes applied in the analysis were based upon those used in our previous 2010 analysis. twenty-eight codes were used in the previous iteration, which seemed to be overly complicated based on relatively low levels of inter-rater reliability. for the 2013 analysis, the principal investigators decided to simplify the coding schema. first, codes that correlated strongly with user satisfaction, according to both the rusa guidelines (reference and user services association, 2004) and the kwon and gregory study (2007), were retained. remaining codes with the lowest levels of reliability in the 2010 analysis were then examined, and either scope notes were improved, or the codes were combined into larger, simplified categories. finally, codes that were no longer relevant were eliminated. this process resulted in 14 codes that were applied to our 2013 chat transcripts. the codes are outlined in detail in “appendix a coding scope notes.”     figure 1 all codes by percent occurrence for 2013.     results   in total, 403 chat transcripts were analyzed, with a confidence level of 95%. fourteen codes were applied to these transcripts in the 2013 iteration. all codes are shown in figure 1.   codes were organized into four categories, based on their inter-rater reliability scores: very good, good, moderate, and poor. codes classified within the poor category, with cohen’s kappa scoring of less than 0.40, were not considered usable in this study.   codes with very good reliability, shown in figure 2, indicated that librarians greeted the patron (greeting), gave their name (name_librarian), gave the name of their library (name_library), and asked patrons to use the chat service again (comeback_again). the code that identified problem transcripts also had very good reliability between coders. this included transcripts that indicated technical difficulties, were incomplete, or included inappropriate patron behavior.   codes with good reliability, shown in figure 3, indicated that librarians listened to patrons, asked clarifying questions and generally checked to make sure they understood the patron question (listening_and_questioning), and referred the patron to a different service point (referral_services). the code initial_question also had good reliability. this code was used to mark the patron’s initiating question or problem that prompted the chat interaction.     figure 2 percent occurrence of codes with very good inter-rater reliability in 2013.   figure 3 percent occurrence of codes with good inter-rater reliability in 2013.     codes with moderate reliability, seen in figure 4, indicate that librarians provided instruction to patrons on how to complete a task (instruction) and searched for patrons (searching_for_patron). the code library_specific also had moderate reliability and was used to mark patron questions requiring specific knowledge from a subject specialist or specific library.     figure 4 percent occurrence of codes with moderate inter-rater reliability in 2013.     codes with poor reliability cannot be used to draw meaningful conclusions and are shown in figure 5. these codes were applied inconsistently between coders and include those that designate that librarians checked on a patron’s progress or acknowledged their own progress toward answering a question (maintain_contact) or referred patrons to another mode of reference service, such as email or in-person services (referral_mode). the code explicit_compliment also had poor reliability, though this is of less concern as it was primarily intended to flag patron comments to be used in marketing.   figure 5 percent occurrence of codes with poor inter-rater reliability in 2013.     codes that are highly correlated with user satisfaction and have acceptable levels of reliability were also separated out and are shown in figure 6. these include codes that indicate that librarians listened to patrons, asked clarifying questions and generally checked to make sure they understood the patrons’ question (listening_and_questioning), asked patrons to use the chat service again (comeback_again), provided instruction on how to complete a task (instruction), and searched for patrons (searching_for_patron).     figure 6 percent occurrence of codes with acceptable inter-rater reliability that influence user satisfaction in 2013.     finally, only one code associated with user satisfaction—maintain_contact—had poor reliability and could not be included in this analysis. this code indicates that a librarian checked on a patron’s progress or acknowledged their own progress toward answering a question. this code will need to be improved in order to be used in future analyses.   discussion   the purpose of this analysis was to build upon the previous analysis, examining how the 2010 analysis and accompanying report may have changed librarian behaviors. we are specifically interested in charting those behaviors over time that correlate with user satisfaction, examining how often subject-specific questions occur over chat, and discovering how often chat questions are referred to other service points and modes of contact. our focus in identifying these behaviors is to improve training and update best practices, as needed, to ensure user satisfaction in the future. finally, we also had an interest in improving our coding process in terms of efficiency and inter-rater reliability, possibly serving as an example to other groups on campus interested in qualitative analysis.   codes eliminated for the 2013 analysis   in 2010, we analyzed both how often patrons gave their names and how often librarians used patrons’ names. we chose not to track this behavior in the current analysis as this behavior is relatively rare and not correlated with increased user satisfaction.   the prior analysis also coded transcripts that contained questions of a general nature that can be answered by a majority of librarians in order to identify questions that were appropriate for our general chat queue. in 2010, over 83% of transcripts received this tag. for the 2013 analysis, we decided it was more important to mark only questions that, inversely, required specific subject-area knowledge or knowledge specific to a library location. our main interest lay in charting how often these questions requiring specialized knowledge occur and how often they are referred from our main service point. in the 2013 iteration, this was indicated by the code library_specific.   the 2010 analysis also recorded transcripts in which the librarian was polite or encouraging, the librarian ended the chat with a closing other than inviting the patron to chat again, the patron thanked the librarian, and the patron was dissatisfied or the patron’s question was unanswered. these four codes all had relatively low inter-rater reliability in the 2010 analysis and were not correlated with patron satisfaction. all four were eliminated from the 2013 iteration.   codes added for the 2013 analysis   only one entirely new code was added for this analysis. the code initial_question was added to the schema in order to mark patrons’ initial questions or the problems that prompted them to contact the chat reference service. we anticipate doing further analysis on these initial questions separately to identify common problems and questions, or pain points. knowledge of the specific issues for which patrons contact us may help to improve services in other areas, for example, improving instructions available on our website.   analysis of code frequency   for each code applied to the transcripts, we calculated inter-rater reliability scores and also the frequency with which it was applied to our transcripts. within the subset of codes with acceptable levels of reliability (cohen’s kappa > 0.40), five codes were applied to more than half of the transcripts as seen in figure 7. these represent the five most common desirable behaviors exhibited by librarians via chat. librarians greeted patrons in 87% of interactions, searched on behalf of patrons in 72% of interactions, engaged in listening and questioning behaviors in 64% of interactions, and stated their name and their library’s name in 59% of interactions. three out of five of these behaviors occurred more often than in our previous analysis. the remaining two codes are unfortunately not directly comparable with our 2013 codes as these two codes consolidated codes used in the 2010 analysis. the 2013 code listening_and_questioning combined the 2010 codes check_on_success, open_ended_questions, rephrasing, and clarifying_or_closed_questions. the 2013 code searching_for_patron combined this same code in 2010 with url_other. for a full list of codes used and comparison to codes used in 2010 see “appendix a coding scope notes.”   most notable in these commonly applied codes from 2013 is that librarians identified both themselves and their library far more frequently than in 2010. this is also one of four librarian behaviors that is highly correlated with patron satisfaction. the increase in this behavior demonstrates that emphasis placed on this identified best practice through training and documentation after the 2010 analysis has had a positive impact on librarian behaviors. however, as best practices, these behaviors should ideally be occurring in more than 59% of interactions. there is still room for improvement.     figure 7 percent occurrence of codes applied to more than half of transcripts in 2013.     the code instruction in the 2013 analysis combined two codes from the 2010 analysis: instruction and url_jing. as jing is inherently instructional in nature, these two codes were combined for the 2013 analysis. similarly, use of non-jing urls by librarians was no longer explicitly tracked, but it often occurred in conjunction with searching for a patron (coded searching_for_patron) that includes librarians providing information directly to patrons. the latter still happens in a majority (72%) of interactions. in contrast, instruction occurred within 36% of interactions. similar to the 2010 results, this indicates that librarians are still more likely to provide patrons with information directly over chat rather than teaching patrons how to obtain that information, which is likely a result of the chat medium. this relationship can be seen in figure 8, which shows the breakdown between these two codes. though a significant number of interactions were coded with both codes (35%), an additional 42% of interactions were coded with searching_for_patron and not with instruction. while both of these librarian behaviors are correlated with user satisfaction, they do represent different philosophies of reference service. this may be an area for future analysis.   in 2013, there was an increase of over 8% in librarians encouraging patrons to use the service again, denoted by the code comeback_again as seen in figure 9. however, this code was present only in 21% of all transcripts. as this librarian behavior is highly correlated with patron satisfaction, there is room for improvement. while there are specific situations where this is difficult, for example if a patron leaves the conversation abruptly, in many chat interactions it can be added to librarians’ typical chat closing.     figure 8 breakdown of percent occurrence of instruction and searching_for_patron in 2013.   figure 9 percent occurrence of comeback_again in 2010 and 2013.     finally, in the 2013 analysis as compared to 2010, approximately the same percentage of referrals to other service points were recorded. there were approximately 5% fewer transcripts coded in 2013 than in 2010 as being best answered by specific libraries or subject specialists. this data is shown in figure 10. this indicates a decrease of over 5% in questions marked as library_specific (or best answered by subject specialists) and no decrease in referrals. the decrease in library-specific chats may be related to the establishment of additional subject-specific chat queues between the 2010 and 2013 analysis. the fact that referrals have remained constant despite a decrease in library-specific questions may indicate an increase in collaborative work among librarians at different libraries. refining our coding in the future may be needed to more accurately analyze these behaviors.   as seen in figure 11, only 15 transcripts were coded as library_specific (3%), with only five (1%) of those also coded as referral_services. though these numbers are relatively small, this does bring into question how many subjector library-specific questions are being referred appropriately. we reviewed these individual transcripts a second time to look for situations where a referral was appropriate but not made. in almost all cases, the specific question was adequately answered by the librarian on chat and thus not referred. in a few cases, the chat was incorrectly tagged. while we did not uncover missed opportunities for referrals, we did find some ways to refine our coding schema in the future. namely, we need to explicitly determine appropriate coding for the following situations: a patron asks for a librarian by name, a librarian refers a patron to an entity outside of campus, and a librarian is testing or demonstrating chat services.   finally, it is important to recognize that while our total sample gives a confidence of 95%, both of these codes have only moderate inter-rater reliability. by improving our coding definitions, we hope to improve the reliability of these codes in future analyses.     figure 10 percent occurrence of referral and subject-specific codes in 2010 and 2013.   figure 11 breakdown of percent occurrence of referral and subject-specific codes in 2013.     analysis of inter-rater reliability   overall, inter-rater reliability for the 2013 analysis has improved from 2010. five out of 14 codes (36%) exhibited very good agreement, three out of 14 (21%) exhibited good agreement and three out of 14 (21%) exhibited moderate agreement. overall, 11 out of 14 codes (79%) in the 2013 analysis were of moderate, good, or very good reliability. only 75% of codes were of the same reliability in the 2010 analysis as seen in figure 12.     figure 12 inter-rater reliability seen as cohen's kappa values over time.     only one code that correlated to user satisfaction had poor agreement and was unusable. two additional codes exhibited poor agreement but are not correlated to user satisfaction and thus not considered critical codes. this code comparison can be seen in figure 13.   we attribute the overall improvement in inter-rater reliability of the 2013 codes to several factors. first, we conducted more comprehensive training and held a group session with all student coders in order to ensure everyone understood and was able to apply our codes. this session resulted in some minor adjustment of coding scope notes in order to make them more sensible for students to apply. we also used fewer individual student coders for the 2013 analysis, and we chose graduate students from the slis program in paid library positions with the rationale that these students would have an improved work ethic and commitment to the analysis. finally, we drastically simplified the codes used by combining, simplifying, and eliminating codes from the 2010 analysis.   however, even with these improvements, three codes out of 14 had low levels of reliability. the code explicit_compliment is intended to mark out patron comments that may be useful in future marketing or promotional materials, and thus reliability is not extremely important for this code.   the code referral_mode is not correlated with patron satisfaction, but it is important in order to know the frequency with which our librarians refer patrons to alternate forms of communication with a librarian (e.g., phone, email, and in person.) during our one-hour training session, our coding group discussed this code and decided that it should be used to identify situations where chat doesn’t work to answer a question. the group decided it should be used in cases where supplementary material is provided through another mode (e.g., when an article is delivered via email in conjunction with chat instruction.) one coder noted that, “most librarians were able to use jing or guided instruction, giving the students lots of time and patience even when questions were more challenging. i felt that this reflected that librarians are more comfortable with online interfaces and are able to give quality reference via chat.” this is a positive observation, but further analysis should be done to determine why the inter-rater reliability is so low for this code.     figure 13 inter-rater reliability of directly comparable codes in 2010 and 2013.     the code maintain_contact is correlated with patron satisfaction and exhibited poor levels of inter-rater reliability. one factor noted by coders that made this code difficult to apply is that timestamps were not included in the chat transcripts. this reduced the context coders had in deciding to apply this code. one possible solution would be to include timestamps in future analyses. another is to separate out the two parts of this code, using one code to indicate when librarians check on patron progress and a second code to indicate when librarians update patrons on their own progress. however, this code was also problematic in our 2010 analysis and, at that point, solely indicated when librarians updated patrons on their own progress.   these latter two codes, referral_mode and maintain_contact, should be improved upon in the future. we intend to work further with student coders to re-examine our scope notes and training examples.   finally, we intend to have a principal investigator code a larger portion of the transcripts in future analyses in order to more accurately gauge inter-rater reliability. in our small sample size, we found that reliability was easily skewed with our current practice of coding 10% of transcripts for comparison.   conclusions   the 2013 analysis again focused on evaluating the frequency of best practices in providing chat reference services. librarian behaviors have improved, likely in response to improved training and awareness as a result of the 2010 analysis. however, there is still room for improvement, specifically regarding librarians providing their name and the name of their library, providing instruction in conjunction with searching for patrons, and inviting patrons to come back to use the service again.   additionally, the investigators have improved upon the analysis process and have identified further areas for improvement including coder training and coding schema design. the methods outlined in this report may serve as an example to other units interested in conducting qualitative analysis in the future.   finally, we plan to conduct a further analysis in the future based on the initial_question code, as outlined in the discussion section of this report. this will identify difficulties that most commonly prompt patrons to initiate chat interactions. we also plan to further investigate the correlation between codes related to subjectand library-specific questions and referrals.   references   banerjee, m., capozzoli, m., mcsweeney, l., & sinha, d. (1999). beyond kappa: a review of interrater agreement measures. the canadian journal of statistics/la revue canadienne de statistique, 27(1), 3–23. http://doi.org/10.2307/3315487   kwon, n., & gregory, v. l. (2007). the effects of librarians’ behavioral performance on user satisfaction in chat reference services. reference & user services quarterly, 47(2), 137–148. retrieved from http://www.jstor.org/stable/20864841   landis, j. r., & koch, g. g. (1977). the measurement of observer agreement for categorical data. biometrics, 33(1), 159–174. http://doi.org/10.2307/2529310   reference assessment working group. (2010). fall 2010 qualitative analysis of chat transcripts. unpublished manuscript, university of wisconsin, madison.   reference and user services association. (2004). guidelines for behavioral performance of reference and information service providers. retrieved from http://www.ala.org/rusa/resources/guidelines/guidelinesbehavioral     appendix a coding schema name memo changes from 2010 notes/comments comeback_ again scope: librarian   use: times when the librarian invites the patron to return.   examples: if you have any further questions, please let us know.     correlates with rusa guideline 5. follow-up. influences patron satisfaction (kwon & gregory, 2007) explicit_ compliment scope: patron   use: when the patron provides a compliment to the service after they have received a response from the librarian. this goes beyond the normal politeness that may occur during transactions.   example: you rock! great service!   changed name (was compliment) tracked for marketing       greeting scope: librarian   use: when a librarian greets the guest at the start of a chat interaction.   examples: hi hello how can i help?   correlates with rusa guideline 1. approachability       initial_ question scope: patron   use: mark the patron’s initial question that prompted the chat interaction.   examples: i’m having trouble finding this journal article   what are your hours today?   new for 2013 for later coding, looking for pain points       instruction scope: librarian   use: when the librarian gives the guest information on how to do a task. if more than one direction is given in sequence, highlight the entire sequence and count it as a single instance. this even includes if the sequence is contained in more than one line or response. this includes when the librarian supplies a video or screenshot for a guest or indicates that they are walking the guest through the steps of searching while simultaneously searching with the guest. in this case, also use searching_for_patron.   examples: librarian: click on the findit button.   librarian: i'm going to the database tab and search for academic search.   guest: let me go where you are. librarian: once you are there click on the database name and then search for: clowns and noses guest: great i'm there. librarian: do you see the 3rd article down guest: yes!   combines instruction, searching_with_patron, and url_jing correlates with rusa guideline 4. searching. influences patron satisfaction (kwon & gregory, 2007) library_ specific scope: patron   use: when a question asked by a patron requires specific knowledge likely better answered by a subject specialist or a specific library. these will be highly technical questions or involve specialized literature types or software (e.g., laboratory protocols, patents, standards).   examples: do you have ascme standard 1234?   someone is making too much noise on the second floor of steenbock.   i have to find articles related to marketing data for these new widgets.       listening _and_ questioning scope: librarian   use: librarian checks on whether they have sufficiently helped the patron, asks clarifying questions, or rephrases the question or request and asks for confirmation to ensure that it is understood.   examples: did this answer your question? what type of information do you need (books, articles, etc.)? so you are looking for articles on the gestation period of tibetan yaks?   combines check_on _success, clarifying_or _closed _question, open_ended _questions, and rephrasing correlates with rusa guideline 3. listening/inquiring. influences patron satisfaction (kwon & gregory, 2007) maintain _contact scope: librarian   use: when the librarian leaves for a time and then returns acknowledging that they are working on the question or are back and when the librarian indicates to the guest that they are still working on a question or thinking about the question. this may also be used when librarian checks in with the patron’s progress. this differs from listening_and_questioning, which is used when the librarian is trying to clarify the patron’s needs.   examples: i'm back. i'm still working on it.   i'll be back in a second. how are you doing?   combines focus_on _patron and maintain _contact   name _librarian scope: librarian   use: when librarian gives their name. usually this will be indicated in the chat as [name omitted]   examples:   hi this is [name omitted] at [library omitted] library       name _library scope: librarian   use: when librarian gives the name of their library. usually this will be indicated in the chat as [library omitted]   examples:   hi this is [name omitted] at [library omitted] library       problem   scope: applies to entire transaction   use: if the transaction ended abruptly, indicating technical difficulties. tag the last word in the document.   or   scope: patron   use: when the patron asks an inappropriate question or makes a crude or rude remark.   examples: will you go out with me later?   combines abrupt and inappropriate   referral _mode scope: librarian   use: when the librarian refers the patron to another mode of communication in order to better serve them.   examples: i think that you should come into the library where we can better serve you.   it would be better if you call us at xxx-xxxx.   i can reply by email more easily.     correlates with rusa guideline 5. follow-up     referral _services scope: librarian   use: when the librarian refers the guest to another service point in order to better serve them. don’t use if the patron directly asks about a particular library. in that case, use searching_for_patron.   examples: i think that you will better if you contact the business library directly.   wendt library will be able to better help. here is their contact information.   please call the circulation office at xxx-xxxx.   ill is on chat, i will transfer you to them now.     correlates with rusa guideline 5. follow-up searching _for_patron scope: librarian   use: librarian gives the answer to the patron or indicates they are searching for them. this may be used in conjunction with instruction if instruction is given before or afterwards. also use with instruction if patron indicates they are following along.    examples: hang on. let me check on that.   i found this: http://someurl.com.   combines searching_for_patron and url_other correlates with rusa guideline 4. searching. influences patron satisfaction (kwon & gregory, 2007)   evidence summary   students who used the library in their first year of university are more likely to graduate or still be enrolled after four years   a review of: soria, k. m., fransen, j., & nackerud, s. (2017). the impact of academic library resources on undergraduates’ degree completion. college & research libraries, 78(6), 812–823. https://doi.org/10.5860/crl.78.6.812   reviewed by: judith logan user services librarian university of toronto libraries toronto, ontario, canada email: judith.logan@utoronto.ca   received: 1 aug. 2018                                                                   accepted: 6 sept. 2018      2018 logan. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29477     abstract   objective – the researchers sought to measure the effect of accessing library resources on academic retention and graduation after four years while accounting for external factors that may influence academic outcomes.   design – quasi-experimental observational study.   setting – a large public university in the midwestern united states of america.   subjects – 5,368 first-year, non-transfer undergraduates; an entire freshman class. methods – using already collected student and library records data, the researchers grouped the population into those that had accessed one of five library resources at least once (treatment) and those who had not (control). the five treatment variables studied were circulation use, electronic resource or website access, library computer workstation logins, enrollment in open registration or course-embedded library instruction, and use of two reference services (online chat and peer research consultations).   the researchers then performed a series of propensity score matching and regression analyses to compare the treatment and control groups’ outcome measures—graduation or continued enrollment after four years. these statistical models controlled for ten covariate measures that included sat scores, first generation status, on campus residency, college of enrollment (e.g., business, engineering, education, biological sciences, design, or food, agriculture, and natural sciences), and demographic profiles. the regressions included subset analyses of the treatment group to determine if some treatment variables were associated with better outcomes than others.   main results – the researchers found that students in the treatment group (n = 4,415) were 1.441 times more likely to graduate and 1.389 times more likely to still be enrolled after four years than those in the control group (n = 953). both results were statistically significant at p < 0.01 and p < 0.001 respectively. the subset regression analyses revealed that accessing an electronic resource at least once was associated with the best graduation odds at 1.924 times (p < 0.001) and the best continued enrollment odds at 1.450 times (p < 0.001). students who had accessed computer workstations and either of the two reference services studied were no more likely to have graduated or still been enrolled after four years than those who had not (p < 0.001 and p < 0.05).   conclusion – accessing library services during the first year of university is associated with improved academic outcomes after four years. more research is needed to accurately measure this impact for methodological reasons. libraries should document contact with students as much as possible for later assessment.   commentary   quantifying the library’s impact on student success has continued to grow as a field since oakleaf’s (2010) influential work. soria, fransen, and nackerud (2013, 2014, 2017) are active contributors to this field of study with three previous studies examining the same first-year undergraduate class. the current study represents a departure from the methodology of these previous works in that it uses propensity score matching in addition to regression analyses following chiteng kot and jones’ (2015) study.   for practical reasons, the authors could only include treatment variables that had an accessible paper trail. for example, at their institution students are tracked when registering for information literacy sessions, but not when attending the sessions. therefore, it is conceivable that some of the treatment group may belong in the control group if they did not actually attend the session. the authors were transparent about what treatment variables entailed, however, and have done an impressive job of gathering and processing variables from as many trackable library activities as they could.   the researchers accounted for factors that would influence students’ use of the library and their academic outcome by including ten covariate variables that attempt to create a profile of the student at the time they begin college. as with the treatment variables, the researchers were limited to data sources available, which may not fully capture the external factors that would cause a student to succeed. they acknowledge this possibility in their limitations section and cite academic motivation as a key unmeasured variable. while it is not feasible to account for all possible covariates, their use of propensity score matching allows them to compare the treatment and control groups using the data available.   the researchers used several sensitive data sources for this work that represent serious risk to students’ privacy and confidentiality. presumably it was gathered with the cooperation of university administration since both the covariate measures and the outcome measures would have originated at least in part from the university’s registration services, but this was not made explicit. the researchers also did not state whether they had an ethics approval nor how they obtained consent from the population as glynn (2006) recommends.   practitioners can make use of this work as an advocacy tool. although more research is needed before we can confidently claim a causal relationship, this study adds to the growing body of evidence that quantifies the benefits students reap from library use. this study design should be replicated at other institutions as the covariate and treatment variables available will differ. libraries may want to consider tracking students at more points of contact so they can participate in studies such as these. in particular, the inclusion of in-person reference, use of physical study spaces, and attendance at information literacy sessions would be interesting to study.   references   chiteng kot, f., & jones, j. l. (2015). the impact of library resource utilization on undergraduate students’ academic performance: a propensity score matching design. college & research libraries, 76(5). https://doi.org/10.5860/crl.76.5.566   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387–399. https://doi.org/10.1108/07378830610692154   oakleaf, m. (2010). the value of academic libraries: a comprehensive research review and report. chicago, il: association of college & research libraries. retrieved from http://www.acrl.ala.org/value/?page_id=21   soria, k. m., fransen, j., & nackerud, s. (2013). library use and undergraduate student outcomes: new evidence for students’ retention and academic success. portal: libraries & the academy, 13(2), 147–164. https://doi.org/10.1353/pla.2013.0010   soria, k. m., fransen, j., & nackerud, s. (2014). stacks, serials, search engines, and students’ success: first-year undergraduate students’ library use, academic achievement, and retention. journal of academic librarianship, 40(1), 84–91. https://doi.org/10.1016/j.acalib.2013.12.002   soria, k. m., fransen, j., & nackerud, s. (2017). beyond books: the extended academic benefits of library use for first-year college students. college & research libraries, 78(1), 8–22. https://doi.org/10.5860/crl.v78i1.16564   evidence based library and information practice evidence based library and information practice 2009, 4:3  evidence based library and information practice       evidence summary    scholars in international relations cite books more frequently than journals: more  research is needed to better understand research behaviour and use    a review of:   zhang, li. ʺcitation analysis for collection development: a study of international relations journal  literature.ʺ library collections, acquisitions, and technical services 31.3‐4 (2007): 195‐207.    reviewed by:  megan von isenburg   associate director of public services, duke medical center library  durham, nc, united states of america   e‐mail: megan.vonisenburg@duke.edu     received: 14 may 2009          accepted: 14 july 2009      © 2009 von isenburg. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution,  and reproduction in any medium, provided the original work is properly cited.    abstract    objective – to determine primary type,  format, language and subject category of  research materials used by u.s. scholars of  international relations. also, to investigate  whether research method, qualitative or  quantitative, can be correlated with the type  and age of sources that scholars use.     design – citation analysis.    setting – research articles published in three  journals on international relations with high  impact factors: international organization,  international studies quarterly, and world  politics.    subjects – a random sample of cited  references taken from the 410 full‐length  research articles published in these journals  from 2000 to2005. cited references of articles  written by authors of foreign institutions (i.e.,  non‐american institutions), as well as cited  references of editorial and research notes,  comments, responses, and review essays were  excluded.     methods – cited references were exported  from isi’s social science citation index (ssci)  to ms excel spreadsheets for analysis. data  was verified against original reference lists.  citations were numbered and identified by  source format, place of publication (foreign or  domestic), age, and language used, if other  than english. the author used a random  number generator to select a random sample  of 651 from a total of 29,862 citations. citations  were randomly drawn from each journal  according to the proportion of the journals’  citations to the total. these citations were  analyzed by material type and language.    52 mailto:megan.vonisenburg@duke.edu http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2009, 4:3  the author also used the library of congress  classification outline to identify the subject  category of each book and journal citation in  the sample.    a separate sampling method was used to  investigate if there is a relationship between  research methodology and citation behaviour.  each of the original 410 articles was  categorized according to research method:  quantitative, qualitative or a combination of  the two. two articles representing qualitative  research and two representing quantitative  research were randomly selected from each of  the three journals for each of the six years.  subsequently, five citations from each of the  resulting pool of 72 articles were randomly  selected to create a sample of 360 citations.  these citations were analyzed by material type  and age of source.    main results – analysis of the citation data  showed that books (including monographs,  edited books, book chapters and dictionaries)  made up 48.2% of the total citations; journals  (including scholarly and non‐scholarly titles)  made up 38.4% of the citations; and  government publications made up 4.5% of the  citations. electronic resources, which  primarily refer to web sites and digital  collections in this study, represented 1.7% of  the citations. other sources of citations  included magazines (1.1%), newspapers  (1.1%), working papers (1.1%), theses (0.9%),  conference papers not yet published as articles  (0.6%), and a miscellaneous category, which  included items such as committee minutes,  radio broadcasts, unpublished materials and  personal communications (2.5%).    the average age of book citations was 14.3  years and the median age was 8 years. foreign  language citations represented 3.7% of the 651  total citations. the top ranked foreign  languages were german (7), french (5),  russian (4), spanish (3), korean (2) and  swedish (number not given    subject analysis of the citations revealed that  38% of all citations were from international  relations and two related disciplines, political  science, political theory, and public  administration. subject areas outside  international relations included social sciences  (23.4% ‐ including economics, commerce,  industries and finance), history (16.3%),  sociology (6.2%), and law (5.9%). citations  from philosophy, psychology, military science  and general works together made up 7.3% of  the total citations. citations from science,  linguistics, literature, geography and medicine  made up less than 2% of the total.    authors of qualitative research articles were  more likely to cite books (56.7%) than journals  (29.4%) while authors of quantitative research  articles were more likely to cite journals  (58.3%) than books (28.9%). authors of  qualitative research articles were also more  likely to cite government publications and  electronic resources than those of quantitative  articles. however, authors of quantitative  research articles were more likely to cite other  materials, such as dissertations, conference  papers, working papers and unpublished  materials.    the age of cited materials for both qualitative  and quantitative research articles is similar.  citations to recent materials up to 5 years old  were most frequent, followed by materials 6  to10 years old, materials 11 to15 years old, and  those 26 or more years old. the least  frequently cited materials were 16 to 20 and 21  to25 years old.     conclusion – scholars in international  relations primarily cite books, followed by  journals and government publications.  citations to electronic resources such as web  sites and digital collections, and to other  materials are far less common. scholars  primarily cite english‐language materials on  international relations and related subjects.  authors of qualitative research articles are  more likely to cite books than journals, while  authors of quantitative research articles are  more likely to cite journals than books. recent  materials are more frequently cited than older  materials, though materials that are more than  26 years old are still being cited regularly.    53 evidence based library and information practice 2009, 4:3  commentary    this study offers librarians with collection  development responsibilities in international  relations materials some insight into the  citation behaviour of scholars in the field.  however, the study uses standard bibliometric  techniques and is subject to many of the  limitations common to the practice.    first, the study relies on data extracted from  social science citation index (ssci). zhang  acknowledges that “analyzing journals and  excluding books may not completely reveal  the citation patterns of intellectual  communication in the field of international  relations,” especially given the high reliance  on monographs that this study itself  demonstrates (206). if monographs are  important and frequently used sources in  international relations, then a citation study  that looks only at journal references is only  going to tell part of the story.    another problem is the discrepancy between  ssci and each journal’s methodology for  categorizing journal content. for example,  what one journal may consider an article, ssci  may designate a review article. the author  deals with discrepancies by stating the study’s  inclusion and exclusion criteria explicitly, but  does not explore them further.    a third problem is the issue of self‐citations.   since the article does not address these, one  can presume that they were not treated  differently from other citations.     with regard to scholars’ use of electronic  resources, the limitations of using ssci data  raise questions about the finding that scholars  in international relations only cite electronic  sources 1.7% of the time and zhang’s  conclusion that “electronic resources have  relatively low impact on formal scholarly  communication” (200). if an author cites a  journal or book that is accessible in print and  in digital format, isi will strip out any data  that would indicate that the source was  electronic, such as access date and  url(warwick). additionally, while most style  manuals require authors to note the method of  access to digital content in their citations,  many may not regularly follow this practice,  preferring to note down the citation as if it  were from a print journal or book. it is  therefore at best an overstatement to imply  that scholars are not using electronic resources  and books and to suggest that scholars may  “not prefer to adopt electronic resources”  (200).    another weakness in the study stems from the  author’s decision to exclude articles by  scholars at foreign institutions. given the  global nature of international relations as a  discipline and of academia in general, this  surprising decision most likely led to an  inaccurate and too low percentage of foreign  language materials cited. while this finding  could be applicable to collection development  librarians at u.s. institutions, it does not  accurately describe the international relations  literature as a whole and does not take into  account the fluidity of scholars’ academic  institutional affiliations    other findings support more reasonable  conclusions that can assist librarians seeking  to make data‐informed collection decisions.  for example, zhang notes that information  about the age of cited materials could help  librarians decide whether or not to store older  or less‐used materials (205). another useful  recommendation is that librarians should  explore the types of research done at their  institutions so that they can use the study’s  conclusions to collect more heavily in the  appropriate source material ‐ books for  qualitative research and journals for  quantitative research (206). this  recommendation would be easier to  implement if the author had clearly stated  how qualitative, quantitative and qualitative‐ quantitative methodologies were defined in  the context of this study.    more research is needed to understand  scholarly activity in international relations. in  addition to investigating citation practices in  monographs, it would be of interest to  investigate them in dissertations and theses.  54 evidence based library and information practice 2009, 4:3  55 these student publications could shed light on  new developments in the field, especially as  universities initiate new interdisciplinary  programs in global health, environmental  studies, and other areas related to  international relations.    in order to understand fully what kinds of  sources scholars are using, it is essential to  complement a citation study with an  observational or behavioural survey. the 2007  study of student research behaviours at the  university of rochester has demonstrated the  relevance and applicability of various  ethnographic research methods to library  research (foster and gibbons). future  explorations of scholarly research behaviours  should incorporate these more robust  inquiries to create a better understanding of  how scholars in international relations  approach research.      works cited    foster, nancy fried, and susan gibbons.  studying students: the undergraduate  research project at the university of  rochester. chicago: association of  college and research libraries, 2007.    warwick, marylou (thomson reuters  technical support). phone call. 8 may  2009.  evidence summary   social scientists’ data reuse principally influenced by disciplinary norms, attitude, and perceived effort   a review of: yoon, a. & kim, y. (2017). social scientists’ data reuse behaviors: exploring the roles of attitudinal beliefs, attitudes, norms, and data repositories. library & information science research, 39(3), 224–233. https://doi.org/10.1016/j.lisr.2017.07.008   reviewed by: scott goldstein web librarian appalachian state university libraries boone, north carolina, united states of america email: goldsteinsl@appstate.edu   received: 26 feb. 2018   accepted: 21 may 2018      2018 goldstein. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29415       abstract   objective – to propose and test a model grounded in constructs from psychology and information systems to explain data reuse behaviours and practices in the social sciences.   design – electronic survey.   setting – proquest’s community of science scholars database.   subjects – included 2,193 randomly selected social scientists associated with u.s. academic institutions.   methods – an electronic survey was distributed to a random sample of u.s.-based social science scholars from proquest’s community of science scholars database. the survey adapted 21 measurement items for constructs taken from the theory of planned behaviour (tpb) and the technology acceptance model (tam), including perceived usefulness, perceived effort, and the subjective norm surrounding data reuse.   main results – there were 292 valid responses received, giving a response rate of 14.91%. survey data largely validated the authors’ theoretical model. attitudinal, normative, and resource factors all influence social scientists’ intended data reuse. in particular, perceived usefulness of reusing data and subjective norms surrounding data reuse in one’s discipline positively correlate with intentions to reuse data, and perceived concern of reusing data negatively correlate with intentions to reuse data.   conclusion – data reuse in the social sciences is influenced by the perceptions and beliefs held by social scientists. social scientists reuse others’ data when they perceive that doing so would improve their research productivity and when their discipline has strong norms of data reuse. they avoid reusing others’ data when they believe that doing so is problematic (e.g., if they believe reusing infringes on copyright). supporters of data sharing, including librarians, are encouraged to apply these findings by proactively educating researchers on the benefits, potential obstacles, and methods of data reuse.   commentary   this study adds to the literature on data reuse practices in the social sciences. unlike in the “harder” sciences, social science data may contain qualitative and highly contextual information about human subjects, thereby demanding a higher level of ethical consideration. previous studies have been primarily exploratory, looking at behaviours and concerns raised by sharing this kind of data. the authors build on this by developing a theoretical model using constructs from the theory of planned behaviour (tpb) from social psychology (ajzen, 1991) and the technology acceptance model (tam) from information systems (davis, 1989).   this evidence summary relies on glynn’s (2006) critical appraisal checklist to determine the validity of the study. a major strength of the study is its data collection methodology. the procedure is fully described and the authors’ instrument and data are publically available. furthermore, the items in the survey were adapted from prior studies and displayed good reliability and convergent and discriminating validity. there are some concerns with the sampling frame, however. the random sample of scholars was obtained from proquest’s pivot database, which is populated via web harvesting with some unspecified amount of manual correction by pivot’s profile editing team. (this database was formerly referred to as community of science, and that is the name the authors use throughout the article.) it is not clear if the process used to harvest publicly available profile information introduces any biases into the collection criteria. for example, the demographic breakdown of survey respondents looks as if it might be skewed towards senior, established academics, but no mention of this is made in the text. it is also worth pointing out that 234 invitation emails went undelivered, which is over 10 percent of the total sample. this is a notably high percentage, especially if this is almost entirely due to invalid email addresses, further raising concerns about the original sampling frame.   the study’s implications for library and information professionals reinforce what many in the practice are already doing: talking with users about what data is available, addressing copyright and other potential limitations to reusing data, and marketing and providing support for relevant data repositories. the authors suggest research libraries should be more proactive in informing and educating researchers. librarians may wish to include information on finding and searching data repositories in their instruction, especially in disciplines with a strong norm of data sharing. they may also wish to advocate in favour of open data practices beyond simply what may be required of researchers in some data management plans. librarians are well-suited to contribute to a culture of data reuse at their institutions.     references   ajzen, i. (1991). the theory of planned behavior. organizational behavior and human decision process, 50, 179–211.   davis, f. d. (1989). perceived usefulness, perceived ease of use, and user acceptance in information technology. mis quarterly, 13(3), 319–340.   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387–399. https://doi.org/10.1108/07378830610692154     evidence summary   some lis faculty indicate reservations about open access   a review of: peekhaus, w., & proferes, n. (2015). how library and information science faculty perceive and engage with open access. journal of information science, 41(5), 640-661. http://dx.doi.org/10.1177/0165551515587855   reviewed by: richard hayman assistant professor & digital initiatives librarian mount royal university calgary, alberta, canada email: rhayman@mtroyal.ca   received: 29 may 2016  accepted: 6 aug. 2016      2016 hayman. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract    objective – to examine the awareness of, attitudes toward, and engagement with open access (oa) publishing, based on rank and tenure status among library and information science (lis) faculty in north america.   design – web-based survey distributed via email.   setting – accredited library and information science (lis) programs in north america.   subjects – 276 professors and professors emeriti.   methods – researchers collected email addresses for 1,017 tenure-track, tenured, and emeriti professors from the public websites of the lis programs. researchers sent an email invitation to participate in the survey by accessing a url, with the survey itself delivered using qualtrics software. the survey included 51 total questions, some with additional sub-questions, and most items used likert-type rating scale. the researchers analysed the data using spss software, and indicated using chi-square tests to measure significance, with a stated intent to get beyond the descriptive statistics commonly seen in other publications.    main results – this study’s results draw on 276 completed responses, for a response rate of 27%. researchers reported that 53% of respondents had some experience with publishing in a peer-reviewed oa format. when asked whether they agreed that scholarly articles should be free to access for everyone, pre-tenure assistant professors were most likely to agree (74%), followed by tenured associate professors (62%), full professors (59%) and then emeriti professors (8%). however, they found less likelihood that associate professors would have actually published in an oa format, highlighting a “disconnect between beliefs about accessibility of research and actual practice with open access” (p. 646). researchers also discovered a connection between faculty awareness of institutional and disciplinary repositories and faculty publishing in oa journals, though a relatively low number (35%) had deposited their output in a repository within the previous year. that increases to 50% of respondents when timeframe is ignored.   faculty who had never published in oa journals ranked several barriers to doing so, barriers common across disciplinary boundaries. these include objections to paying oa fees; perceptions of slow time to publish, low research impact, and venue prestige when compared to traditional subscription journals; an inability to identify an appropriate oa journal; and an inability to pay oa fees. however, the researchers note that a majority of these respondents who had never published in an oa format would do so if these barriers were removed. those participants who had some previous experience with oa were more likely to have positive perceptions of oa journal quality and impact, as well as the overall publishing experience, as compared to publishing in traditional journals.   as in other disciplines, lis faculty are conscious of the connection between oa and tenure and promotion processes. for example, this study reveals that non-tenured faculty are more likely to agree that publishing in oa venues may affect their career progress. researchers report uncertainty about oa even among tenured lis faculty. of all respondents, only 34% agreed that a tenure or promotion committee might consider an oa publication on par with a traditional publication, while 44% of respondents were of the opinion that an oa publication would be treated less favourably than a traditional journal. a mere 1% of respondents believed that an oa publication would be treated more favourably within the tenure and promotion process. despite this unfavourable perception of oa, the researchers report that 38% of respondents planned to publish in an oa journal regardless of whether their tenure and promotion committees might treat that oa publication unfavourably.   conclusion – the researchers report a connection between publishing in an oa journal and academic rank, with full professors more likely to publish oa or to have previous experience in publishing in an oa journal as compared to assistant professor colleagues, who perceive publishing in oa as a potential impediment to career progress. the researchers note that there is significant opportunity for lis faculty involved in tenure and promotion committees to consider and clarify how oa publications are treated, and the impact of oa publishing with regard to career progress. moreover, given the levels of uncertainty and equivocacy among faculty respondents as a whole regarding certain aspects of oa, the perceptions around quality and rigour, there is room for further research into lis professors’ perceptions and attitudes toward open access, and how these change over time.   commentary   as a descriptive analysis, this study adds new knowledge to the conversation about oa engagement. by including their survey instrument as an appendix, alongside their coding key for collapsing certain likert-like response categories, the authors have contributed a new tool for measuring oa engagement by faculty that can be further adapted for future research. the tool could be improved by providing justification or rationale for collapsing the likert scale in the manner as done in this study. to aid instrument validity (glynn, 2006), the authors have pre-trialled and adjusted their survey tool before distribution to participants, and the tool itself was adapted from an instrument used in previous research. however, the researchers do not describe if or how the instrument may have been tested for reliability or internal consistency (e.g., cronbach’s alpha).   the researchers state their intent to move beyond descriptive statistics toward inferential analysis, but their design and analysis may be problematic. they acknowledge using a non-random (i.e., non-probability) sample, but the authors do not define their intended sampling approach except to state they surveyed “all north american faculty members (excluding puerto rico)” and excluding adjunct professors (p. 644). while they discuss the representativeness of participant characteristics within the response rate and their “overall sample” (p. 647), it is unclear whether they intended to achieve a census given the population of lis faculty invited to participate, and it appears that this research instead draws on a non-probability sample (e.g., convenience sampling). ultimately the researchers do not clearly identify the survey sampling method, and identifying the specific type of probability or non-probability sample used would have been valuable.   moving into results analysis, while there is still debate in the literature regarding best practices, it is generally understood that inferential analysis requires a random sample to ensure that the population being studied is properly represented (lópez, x., valenzuela, j., nussbaum, m., & tsai, 2015). further, though the researchers indicate that the distribution of participants from the subgroups of assistant, associate, full professors matches “fairly closely” to their distribution among the larger population, there are discrepancies here that require more detail. as lopez et al note, “if the conclusions of the study involve generalising for subgroups, then the sample size should be representative at the subgroup level”, and also that confidence intervals for calculating these should be explicitly stated (p. 107). given the low response rate, and the lack of probability sampling, this study is open to self-selection sampling bias and should not be treated as generalizable. thus, while the descriptive analysis provided is interesting, the inferential results are problematic, leading this author to focus solely on the implications of the descriptive rather than the inferential results.   takeaways from this study highlight that lis faculty who participated in the survey reflect similar attitudes toward and engagement with open access as their faculty colleagues from other disciplines. this is important as these attitudes and perceived constraints may affect uptake of oa within the lis discipline and beyond. the participants from ala-accredited graduate programs in north america are the educators responsible for instructing new librarians on the fundamental principles and practices of information access and availability. if these educators have reservations about the benefits, challenges, and impacts of open access, this could affect those responsible for encouraging oa practice for colleagues both within and beyond lis. this suggests that ongoing study of researchers’ attitudes and perceptions toward open access, both within and beyond the discipline, is necessary.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154   lópez, x., valenzuela, j., nussbaum, m., & tsai, c. (2015). some recommendations for the reporting of quantitative studies. computers & education, 91, 106-110. http://dx.doi.org/10.1016/j.compedu.2015.09.010 research in practice   the reciprocal benefits of library researcher-in-residence programs   virginia wilson director, centre for evidence based library and information practice (c-eblip) university library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   selinda berg department head, information services leddy library university of windsor windsor, ontario, canada email: sberg@uwindsor.ca​   received: 13 may 2016  accepted: 20 may 2016      2016 wilson and berg. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   traditionally, libraries have offered scholar-in-residence programs to encourage researchers to visit libraries and to make use of their collections and materials[1]. new forms of residence programs, designed to encourage the development of research culture within the library, are emerging and gaining popularity at canadian academic libraries. while these programs are often developed “to help enrich and develop the research culture at the university library” (researcher-in-residence program, 2015), the benefits of these researcher-in-residence programs extend beyond the walls of the university library. as the director of the centre for evidence based library and information practice (c-eblip) at the university of saskatchewan (wilson) and the 2015-2016 researcher-in-residence at the university of saskatchewan (berg), we want to share how we have experienced the benefits of the program, as well as consider how researcher-in-residence programs can contribute to the research capacity of an institution and individuals. the university of saskatchewan (u of s) library’s researcher-in-residence pursues their own research agenda while in residence. there is no expectation to conduct research relevant to the university library. however, researchers make a commitment to undertake one or more activities designed to engage the u of s librarians, which may include the following:   ·         presenting a research seminar or outlining the research being undertaken. ·         conducting seminars or workshops on selected aspects of research methodologies relevant to library and information science research. ·         advising individual librarians about their research in progress or about the development of research skills and/or interests. ·         mentoring librarians in aspects of research work.   the u of s’s program provides dedicated office space, access to computing facilities, and other supports-in-kind for a visiting professional (educator or practitioner) to spend a period of time at the university library. in addition to the researcher-in-residence program at the university of saskatchewan, new researcher-in-residence programs have been developed in canada. each program has a slightly different focus. in may 2016, concordia university in montreal, quebec launched a researcher-in-residence program. concordia’s program invites doctoral students, professors, and librarians to propose and complete a research project that addresses pre-defined themes that align with the library’s strategic direction. in addition to funding for the research project, the program “offers the opportunity for scholars or doctoral students to focus on an area of inquiry in a supportive and enriching environment, and to interact with concordia library staff and its resources” (researcher-in-residence program, 2016).  the program aspires to encourage both the scholar and the library staff to “become engaged in the use of research in library practice and the concept of evidence-based librarianship” (concordia libraries, 2016). in 2013, mcmaster university library in hamilton, ontario hosted its first faculty member in residence, brian deltor. deltor and vivien lewis (mcmaster university librarian) have published about their experience with the program in the january 2015 issue of the journal of academic librarianship. mcmaster’s ‘faculty member in residence’ program invites faculty members to spend a year in the library encouraging and mentoring librarians in their research endeavours, while conducting a research project related to library and information studies and library culture. like the concordia program, mcmaster’s resident is encouraged to “become actively involved in at least one library project” either as a research study or as a member of the work team. (libplessla, 2014). the program hopes to advance and enhance the research agenda of the faculty member while providing research mentorship and modeling for practicing librarians. the focus and execution of each program are slightly different, but the goals are much the same: to enhance the research culture in the academic library and to help build research capacity amongst practicing librarians.  the experience and benefits for the resident (from selinda’s perspective)   i began my residency at c-eblip at the university of saskatchewan in august 2015, and completed the residency in june 2016. i recognize that moving across the country for a sabbatical is not possible for everyone, but for me, the experience was well worth the effort that the move required. i have grown and evolved as a researcher and as a professional as a result of the residency. the researcher-in-residence program provided me with a very productive and gratifying sabbatical filled with new colleagues, new experiences, and new learnings. access to resources: don’t underestimate the photocopier   resources-in-kind seems like a vague and rather insignificant element of the remuneration across researcher-in-residence programs. however, this is also the element of the program that i am most thankful for on a daily basis. one cannot underestimate the value of an ergonomic chair, two computer screens, and a photocopier/printer that will print on both sides expeditiously. the productivity of my sabbatical would have decreased significantly had i been limited to a folding chair, a laptop on a small desk, and an inkjet printer. these amenities are also not free, so in addition to the comfort, there is additional monetary benefit. further, i had access to fantastic it support and services, reliable phone and teleconferencing services, statistical and data assistance, and of course, library materials. i made use of these services regularly throughout the year. as academic librarians we are privileged to have access to abundant and rich resources and services at our disposal and maintaining access to these amenities during a sabbatical away from home cannot be underestimated. immersion in a new culture   i have such an appreciation for the chance to be a part of another institutional culture, even if just for a year. this immersion has provided me with an opportunity to see different ways to be a professional, a leader, and a researcher, as well as to see the different ways a library can function. we are all too often immersed in our own world to see different ways of being, and the sabbatical is the perfect time to consider these different ways. an immersion into another institution is a great catalyst for this process.   the university of saskatchewan has a rich and ever-evolving culture of research. being a part of that culture and contributing to making it stronger and richer was a very rewarding experience. the university of saskatchewan libraries, in particular working closely with virginia, has inspired me and allowed me to see new ways that i can contribute to the health and productivity of the research culture within and outside of my own institution. not only in the development of research culture, but across the library as a whole, there is reassurance in witnessing that most institutions are struggling with the same issues, all the while being inspired to see new ways of dealing with such issues.  a place to go each day (if i choose)   for me, the process of getting up, having a shower, dropping the child off at school, and ultimately arriving at the office to work is a pattern that i like and within which i thrive. while i did not want to spend my sabbatical in the same office that i have spent the last 7 years, i knew that i would benefit from having a place to go. i knew that i need to be away from the demands and distractions of my home. having access to new and uncluttered space to work provided me with a retreat-like space.   in addition to the physical space, the office also provided me with an opportunity to connect with (new) people. colleagues are wonderful for mentorship and professional camaraderie, but are also essential for asking for advice on places to eat in the city, for grabbing a coffee or lunch, or for talking about the weather or state of the winter roads. the colleagues that i connected with at the university of saskatchewan also provided inspiration and support for my research, especially for my dissertation. colleagues are very important to cheer you on from the sidelines and the researcher-in-residence program did provide me with a wonderful cheer team!   new fish in an old pond   it is a harsh reality but we often do not see the expertise that we are surrounded by each day in the same light that we see the new, the distant, and the fresh. talking with people outside of your everyday world seems glamorous, and perhaps even more valuable. during my residency, i got to be the person who offered the fresh and novel voice to conversations. i had discussions with university of saskatchewan colleagues about research projects, research programs, and research culture that normally i do not have, or that i have so often at my own institution that no one wants to hear any more. i was able to provide a new voice to their conversations and that felt wonderful. similarly, i likely listened a little more carefully to the voices around me. i did not assume that i already knew what my colleagues would say. instead, i hung on every word hoping to take away some of the great wisdom and perspective that these librarians were sharing with me. being the new fish in an old pond was a unique experience where i listened more intently and was listened to more intently.   the experience and benefits for the institution (virginia’s perspective)   the university library has hosted two previous researchers in residence. each has been a very different experience but all have enriched our library research culture in some way. selinda berg’s residency has been particularly fruitful. she is working on her phd in library and information studies (lis), has extensive experience in research and publishing, and has an open and curious personality. all combine to make selinda’s residency beneficial for all concerned.   new fish in an old pond   i will start my section where selinda left off. our old pond is quite lively in terms of research, research supports, and research outputs. however, there is always something to be learned from a new fish, and the university library certainly took advantage of selinda’s time with us. most notably, the drop-in visits to selinda’s office provided librarians with new ways of looking at their own research. librarians took the time to come and meet selinda, and to speak to her about their research projects, their programs of research, and specific issues they might be dealing with. having a fresh perspective to bounce things off of is helpful to librarians at all levels of their research endeavours. selinda brought in previous experience from her work at western and windsor, and the experience of working on her phd. her reputation as it pertains to the canadian association of research libraries (carl) librarians’ research institute gave her words and advice weight. our librarians were thrilled to be able to interact with selinda on a day to day basis, and much was learned from these one on one encounters. on a personal note, selinda fully engaged with the c-eblip and was integral to the development of the c-eblip research network. i truly appreciate my relationship with selinda, our friendship, and our collegial conversations.   sharing of skills and knowledge   this is slightly different from the section above in that the u of s library researcher-in-residence program requires that our researcher makes a commitment to formally engage with the librarians in a structured way. the first such activity during selinda’s residency not only engaged our own librarians, it also engaged librarians from across canada. selinda presented the very first pre-symposium workshop the day prior to the 2nd annual c-eblip fall symposium: librarians as researchers. her workshop topic focused on how to turn a research idea into a research question. formal feedback received from workshop participants indicated that the workshop was “very good” and “excellent,” while informal feedback gave the impression of enthusiasm and inspiration. early in 2016, selinda met with c-eblip members for a facilitated discussion on how we might best navigate and lead ourselves through the exciting and ever-evolving landscape of research dissemination. we talked about blogs and twitter, comfort zones and new vistas, and maximizing benefits of engagement in social media. prior to selinda’s departure, she will be presenting a talk to u of s library employees about embarking on a phd. our group looks forward to selinda’s thoughts and insight as she comes to the end of her doctoral journey.   a focus on research   the identity of the researcher-in-residence aside, just the fact that we have a researcher-in-residence program serves to announce to the wider campus community that our librarians are faculty members and research is part of our assignment of duties. it also helps to remind us that research is part of our jobs. the university library has a service mission and an academic mission, and so do librarians. the presence of the program and of the researcher-in-residence puts research front and centre for us. with busy schedules and a major focus on the work of the library, research can sometimes be relegated to a secondary task. however, research is important not only for academic considerations such as tenure and promotion, but also as a means to progress the discipline of librarianship. practicing librarians conducting and disseminating research enhances the lis evidence base. it provides a much needed perspective from a practical viewpoint. the more we are able to engage with our practice issues in a rigorous and robust manner, the better we will be at making timely and appropriate decisions and moving our practice and service forward. the researcher-in-residence program is a focal point, for ourselves and for those outside of our library.   closing   a well-run researcher-in-residence program can be an asset to any academic library where research is a focus. the program also enriches the resident’s experience and knowledge as a researcher. extending these programs beyond canada and even beyond academic institutions can provide opportunities for librarian researchers to be new fish in old ponds, and can provide new energy and fresh ideas for the host institution. communication and collaboration are important catalysts in developing an environment which supports and nurtures research. the more opportunities we have to trade knowledge and ideas, share skills and philosophies, and experience research life outside the boundaries of our home institutions, the richer and more vibrant our own research will be.   references   libplessla. (2014, october 16). open call – university library “faculty-member-in-residence” july 1, 2015 – june 30, 2016. message posted to https://library.mcmaster.ca/news/26598    researcher-in-residence program. (2015). retrieved from http://library.usask.ca/ceblip/activities/researcher-in-residence-program.php   researcher-in-residence program. (2016). retrieved from http://library.concordia.ca/about/researcher-in-residence/index.php   [1] see for example: the dibner library resident scholar program at the smithsonian libraries, the scholars-in-residence program at the burke library (columbia university libraries), and the schomburg center scholars-in-residence program at the new york public library. microsoft word es_1802_2 evidence based library and information practice 2008, 3:3 80 evidence based library and information practice evidence summary a faceted catalogue aids doctoral-level searchers a review of: olson, tod a. “utility of a faceted catalog for scholarly research” library hi tech 25.4 (2007): 550-61. reviewed by: kurt blythe head of serials cataloging and binding, e-resources & serials management department the university of north carolina at chapel hill chapel hill, north carolina, united states of america e-mail: kcblythe@email.unc.edu received: 06 june 2008 accepted: 10 august 2008 © 2008 blythe. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – to learn whether a faceted catalogue and word cloud aids in the discovery process. design – user study. setting – large academic research library in the united states. subjects – twelve phd candidates in the humanities, the majority of whom are engaged with researching, proposing, or writing their dissertations. methods – the library’s entire catalogue of 5.2 million records was loaded into the aquabrowser opac search interface. a pilot study was conducted using three humanities graduate students employed by the library. following the pilot, the main study was conducted using graduate students in the humanities. graduate students in the social sciences were desired for the study, but were not able to be contacted due to time constraints. once selected, the test subjects were asked to use an interface that offered both facets and tag clouds for enhanced search quality. test subjects were allowed to choose the topic they would like to research; all chose to research their dissertation topic. a moderator and recorder facilitated research conducted with the faceted catalogue. the moderator ensured that students commented on their findings, cleared up any confusion with using the interface, and evidence based library and information practice 2008, 3:3 81 kept the students on task. only when students remarked that a new discovery had been made were those discoveries noted. the impact to the discovery process of faceted navigation and aquabrowser’s word cloud was studied while the impact of relevance ranking was not. main results – the article asserts that results from both the pilot and main study were sufficiently similar to justify combining them for the paper, but the advantage that students employed by the library might have over other students is not discussed. nine of the twelve students used in the study found new results using the faceted catalogue and word cloud. the responses of the user group to the faceted catalogue and word cloud were “overwhelmingly positive” (555). however, since students were allowed to move freely between the word cloud and faceted navigation tool, it is difficult to attribute new discoveries solely to one or the other. however, when a new discovery could be “attributed primarily to one factor or another” (555) it was noted. the faceted navigation tool aided discovery at least four times and the word cloud aided discovery at least six. conclusion – a faceted catalogue interface with a word cloud feature clearly aids in the discovery process for more advanced researchers—those with specialized subject knowledge, familiarity with their library’s collection, and experience in researching their area. however, facets and word clouds have limitations: records with limited cataloguing have little to offer faceted navigation; catalogue records from diverse providers introduce controlled vocabularies beyond lcsh and mesh into search returns, resulting in the same word potentially appearing multiple times in the same return albeit with different meanings; the word cloud may contain certain words that researchers feel to be irrelevant. despite these issues, the use of word clouds and faceted navigation (and relevance ranking) appears to be beneficial to research conducted by experienced subject searchers in the humanities. commentary although the study is well-explained, questions remain unanswered. neither the number of new discoveries per student nor the significance of the new discovery was reported. the role of moderators and their potential to influence results is difficult to discern: “the facilitator … intervened when subjects became confused or needed coaxing” (554), but the extent of the intervention is not revealed. finally, the study does not show the impact of relevance rankings (although this omission is noted). however, these issues are acknowledged in a discussion of whether the artificiality of testing may have impacted the results. an explanation of the decision to limit test participants to dissertation-writing graduate students in the humanities (and social sciences, ideally) is useful. humanities and social sciences are two areas of study that make extensive use of the library’s print collection. moreover, it may reasonably be assumed that graduate students engaged in dissertation research are familiar with library collections and traditional tools for discovery, thus making the test subjects better equipped to judge whether a new discovery tool contributes to their research process. therefore, to answer the question of whether extensive users of print collections will benefit from a faceted interface to the library’s catalogue, the test subjects were appropriately chosen despite their limited numbers. the question of how best to serve the needs of users of print collections is timely. print collections are increasingly moving off-site due to limited on-site shelf space. therefore, access to print literature becomes evidence based library and information practice 2008, 3:3 82 increasingly difficult. at the same time, other studies have shown that library users respond well to faceted searching. this study illustrates the way user studies inform the decision-making process in libraries, while also contributing an analysis of a specific user group’s experience with a faceted interface to the larger discussion of revamping traditional discovery tools. research article   investigating the persistence of federal government publications in academic former full depository libraries in canada   graeme campbell open government librarian queen’s university library kingston, ontario, canada email: graeme.campbell@queensu.ca    michelle lake government publications librarian concordia university library montreal, quebec, canada email: michelle.lake@concordia.ca    catherine mcgoveran head, research support (education, law, management, social sciences)  university of ottawa library ottawa, ontario, canada email: catherine.mcgoveran@uottawa.ca    received: 12 july 2022                                                               accepted: 28 oct. 2022      2023 campbell, lake, and mcgoveran. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   data availability: campbell, g., lake, m., & mcgoveran, c. (2023). persistence of federal government publications in academic former full depository libraries in canada (v1) [data]. borealis: the canadian dataverse repository. https://doi.org/10.5683/sp3/bds0y6   doi: 10.18438/eblip30203     abstract   objective – the depository services program (dsp) provided printed government of canada publications to libraries until the termination of its distribution program in 2013. full depository libraries (fdls) received all eligible publications distributed by the dsp automatically. this study endeavours to determine whether academic library members of the canadian association of research libraries (carl) that were formerly fdls have maintained their print, federal government holdings since 2013; and what the results of the data collected in this study reveal about access to government information in canada more broadly.   methods – the study identified a sample of 100 monographs distributed to fdls via the dsp between 1979 and 2009. each monograph was then searched for in the public catalogues of former fdl carl member libraries to determine current holdings.    results – most libraries included in the sample did not have records of all 100 publications, but every publication was located in at least 5 libraries and 12 publications were found in all libraries included in the study. of the libraries in our sample, 1/3 had retained more than 90 of 100 publications, and 3/4 had retained at least 80.   conclusion – the redundancy that was a cornerstone of the dsp network still exists to a certain extent and should be leveraged to ensure retention and access to these essential materials for years to come. existing collaborations and partnerships are well positioned to support a pan-canadian discussion about preservation of and access to historical federal government information in canadian libraries and library networks.     introduction   wakaruk and li (2019) state in the introduction to government information in canada: access and stewardship, “government information has enduring value ... [and is] used by academics and social commentators in all areas of intellectual output” (p. xv). it is also “precarious and requires stewardship” (p. xvi). government information in canada exists in a complex, difficult to navigate web of authors, publishers, jurisdictions, and aggregators. for most of the 20th century, public access to government of canada information was shaped by the depository services program (dsp), created by order in council p.c. 1471 in 1927 (dolan, 1989, pp. 55-56). the dsp had a mandate to distribute official print publications from federal government departments free of charge to any library registered as a selective or full depository. from 1928 onwards, a catalogue or checklist of items was distributed to depository libraries at varying frequencies, settling on weekly distribution from 1978 until the conclusion of its distribution program in 2013 (canada. public works and government services canada, 2013, p. vi; dolan, 1989, p.106). full depository libraries (fdls) automatically received copies of eligible items from the dsp weekly checklist of canadian government publications, while selective depository libraries (sdls) had to actively select items (canada. depository services program, 2010, pp. 4-5). while there were circumstances under which fdls could withdraw depository items from their collection, fdls were generally expected to retain items indefinitely (canada. dsp, 2010, pp. 6, 17-19). consequently, the network of dsp fdls formed a decentralized collection of government publications, with redundancy built in as a safeguard against loss of access.   in 2006, the government of canada introduced a new economic plan, which directed departments to “assess their programs regularly and to identify areas where they could contain or eliminate ongoing costs” in what would become the deficit reduction action plan (drap) (paterson, 2021, pp. 28-29). in 2012, the dsp responded to the drap, by announcing that, “as of april 2014, they would no longer produce, print, or warehouse hard copies of publications” (wakaruk, 2014, p. 17). part of the justification for this change was that as of 2013, “over 90% of the publications listed in the weekly checklist were in downloadable electronic formats” (canada. government of canada publications, 2014). there was discussion among depository libraries at the time (depository services program library advisory committee, 2012), and some concern voiced by librarians related to ongoing access and preservation of government of canada publications (kaufman & moon, 2013; paterson, 2021, p.29).  nevertheless, all agreements between the dsp and depository libraries were terminated in december 2013, when the dsp ended its distribution program (canada. government of canada publications, 2014). the end of these agreements meant that the guidelines requiring fdls to retain deposited publications expired and the dsp-provided collections that remained in fdls could be weeded. this study explores whether the depository collections received by former fdls have changed since 2013.   literature review   printed government of canada information resources   our study expands on the literature from the 1990s that discusses the availability, persistence, and vulnerability of canadian federal government publications. morton and zink (1992) characterized the dsp as an important “safety net” securing citizen access to government of canada publications but noted “considerable concern” for its continuation during a time of government downsizing (p. 387). a 1991 task group made recommendations concerning dsp operations, including the loosening of retention requirements for sdls (canada. task group on depository program review, 1991, p. 45), and by 2001, an exemption for sdls allowing the removal of any publication after holding it for five years had been added to the guidelines (canada. communication canada, 2001). as a result, when the dsp’s print distribution program ended in 2013, sdls had already had several years to discard the vast majority of what they had received from the dsp with little restriction. for this reason, we focused on examining the collections of former fdls only.   there is also evidence that the redundancy originally provided by the collections of federal departmental libraries began to weaken earlier than that of the dsp network. in 2005, monty asserted that “...most documents have been preserved by libraries outside of government and not necessarily by the government itself” (p. 14). almost a decade later, wakaruk’s (2014) assessment of the government information landscape reported reductions in the federal government’s national and departmental library services with the closure of 30 departmental libraries, which was occurring at the same time as the dsp’s termination of print publication distribution (pp. 16-17). the loss of collection redundancy inherent in these library closures increases support for the idea that the network of former fdls remains an important location of federal publication access.   cross and lafortune unearthed early evidence of weeding within the collections of former depository libraries in their 2017-18 survey of canadian academic libraries. they also reported that some libraries were choosing to retain smaller print government publication collections with many preferring to focus on municipal and provincial publications (cross & lafortune, 2019, p.8). our study flows directly from these findings by examining the currently discoverable holdings of federal publications originally intended to reside permanently at fdls.   collection assessment   our approach was to combine attributes of the traditional method of assessment as described by goldhor (1973), checking a list of publications against a library catalogue, and an overlap study, as outlined in potter (1982) and kairis (2003). instead of carrying out an inductive assessment using multiple checklists like goldhor, our traditional assessment uses a single checklist: a subset of items from the dsp’s weekly checklist. this checklist acts as a shelf list for what items should have been available in a participating library’s collection at the end of 2013.   in assessing the persistence of an item across many libraries, our study is procedurally similar to an overlap study, though it differs in its ultimate goals. potter (1982) divided overlap studies into four categories based on their objectives: optimizing access to materials within a group of libraries, assessing the advantages of centralized processing, investigating the benefits of collaborative collection development, and examining the phenomenon of collection overlap itself (p. 4). while our study most closely sits within potter’s collaborative collection development category, our aim is to assess overlap to ensure that duplication continues, rather than avoid it in future acquisitions (p. 14). as such, we would add an additional category to this list: collaborative collection retention.   kairis’ (2003) overlap study created a sample set of publications which were then manually searched in a central consortial library catalogue using their isbns. kairis also repeated data collection six months later through computational automation (pp. 320-321). our study followed a method similar to kairis’, as we generated a sample set of publications from several weekly checklists and compared them against library catalogues. however, the searches in our study were carried out manually by our research assistant in over 20 different systems and the use of computational automation was out of scope for our project.   shortly after our study was underway, a concurrent study with similar aims was launched by the canadian collective print strategy working group (ccpswg). the ccpswg was formed in 2018 through a collaboration between library and archives canada (lac) and the canadian association of research libraries (carl), with the goal of investigating the potential for canadian shared print initiatives. their work included a pilot overlap study that examined the collections of canadian federal government publications at 26 libraries (ccpswg, 2020, pp. 5-6). the ccpswg pilot used an automated checking system with oclc’s greenglass application to assess the overlap of federal publications at participating institutions, as represented by oclc-held data, to identify rare or scarce items (ccpswg, 2020, pp. i-ii). the ccpswg selected their list of serial and monograph publications based on what federal materials were held by the participants at the time of data collection (ccpswg, 2020, p. viii). compared to the ccpswg pilot, the set of publications considered by our study was smaller and limited to monograph publications that were listed in the weekly checklist as having been distributed to all libraries in our study, meaning that the titles should still have been held in each library’s collection in 2013. as such, in addition to identifying which items remained, our approach also provided the opportunity to identify items that had disappeared entirely from the collections of the libraries in our study and calculate a rate of retention for federal government publications in our sample among former fdls.   aims   now that the dsp’s print distribution program has been concluded for several years, it is an ideal time to examine the ways in which libraries are adapting their print government publication holdings. there is a gap in the literature related to how these distributed print holdings have evolved since the end of the program, one that we hope this paper will help to fill. an analysis of the current print federal government information landscape could do well to inform the advocacy efforts of carl and other groups, particularly in relation to government publishing, information access, preservation, and transparency. our study endeavours to estimate the extent to which former fdl academic carl libraries have maintained their dsp-distributed print holdings.   methods   for this study, we extracted from the weekly checklists a sample of 100 publications distributed to fdls between 1979 and 2009 and searched for them in the library catalogues of former fdls.   library selection   we narrowed our focus to the collections of former fdl carl members, as carl works to “ensure long-term access and preservation in a dynamic digital environment” for academic researchers in its member libraries (canadian association of research libraries, 2021). as such, carl academic library members seemed most likely to have developed comprehensive collections of historical value to support research activities.   we further restricted the scope of our study to carl academic library members that had been dsp fdls. while we did attempt to identify the dates at which carl libraries became fdls, it became clear early on that this information was not easily discoverable. for that reason, we relied on four lists of dsp fdls published between 1989-2011 to select the pool of libraries for our investigation (canada. depository services program, 1999; canada. depository services program, 2005; canada. government of canada publications, 2011; statistics canada, 1989, pp. 221-223;). there were 22 carl academic libraries listed in all four sources as fdls, and 25 listed as such in at least two of the sources. we decided to include all 25 carl libraries that had been fdls during at least some of the period under study.   the university of waterloo and the university of guelph, both of which are carl members and former fdls, also represent two of three members in the triuniversity group of libraries (tug), which has been in existence since 1995. tug institutions have shared a storage facility since 1996, an integrated library system (ils) since 1998, and a preservation of last copy agreement since 2006 (triuniversity group of libraries, 2000; triuniversity group of libraries, 2018). the last copy agreement specifically recognizes the duplication of materials received via the dsp amongst tug institutions (tug, 2018). given this context, we decided to treat tug as a single library when performing our analysis. we considered an item found in any of the three members of tug to be a positive result. this included situations where items were found only at wilfrid laurier university, the third institutional member of tug, even though laurier’s library is not a member of carl. laurier is also a former sdl, not a former fdl. as such, our analysis and discussion refers to 24 libraries, even though our searches within the tug system as a whole meant that we actually looked for each publication in the collections of 26 libraries. a list of all libraries included in this study, and their relevant consortial and partnership affiliations, is included in appendix a.   publication selection    the publication inclusion and exclusion criteria were based largely on the dsp’s guidelines for retaining and disposing of depository publications (2010). a total of 100 publications were identified for this study, with 25 publications from each of the following years: 1979, 1989, 1999, and 2009. these years were chosen to include a range of dates in order to consider whether date of distribution affected retention. publications were selected from weekly checklist issues from the identified years or successive years when our selection method necessitated it. a list of all checklists consulted for the development of the sample list is provided in appendix b. the criteria for selecting publications were designed to maximize our chances of identifying them in library catalogues and to increase the likelihood that all fdls should have received the publications in our sample list and had them as part of their holdings as of december 2013. as such, publications selected for inclusion had to be published in print, issued as monographs or standalone volumes, distributed to depository libraries, and available in both english and french or bilingually. the research team carried out the work of selecting publications for inclusion in the study, while the research assistant (ra) completed the catalogue searches and collected relevant metadata.   for exclusion criteria, our decision to exclude ephemera, serials, and electronic media was based on the guidelines’ allowance for the disposal of certain items in these categories under defined circumstances. our exclusion of serials was also based on the lack of detail found in many government publication serial records in library catalogues, making it difficult to confirm the scope of holdings. the study focuses on monographs because we believe they provide the clearest paths to identification in library catalogues and significantly reduce challenges confirming holdings while searching. a detailed description of the publication selection process is described in appendix c. a list of the 100 publications selected using this process is found in appendix d.   search strategy and data collection   searches were performed using the library catalogue’s advanced search. where consortial union catalogues were in use, we confirmed whether the catalogue indicated that the physical items were located at the specific libraries that were part of this study.   we developed a four-step search process, outlined in detail in appendix e, that involved searching for the title, isbn, and department or agency from each publication. the search process was carried out for all publications in the sample by the ra using each library’s catalogue. a set of five key metadata fields were used to confirm a successful match: title, date, author department or agency, format, and language. if all fields matched, the search was deemed successful. if all steps in the search process were attempted without resulting in a successful match, the publication was marked as not found at that library. for each successful identification, a limited amount of metadata was collected from the catalogue record so that comparisons could be made during our analysis (see appendix e).   results   overall retention   searches for the 100 publications in our sample were performed at each of the 24 libraries in our study. if the dsp guidelines were still in force, we would expect to have found all publications, a total of 2,400 instances. instead, we found only 1,981. every title in the sample was located in at least 5 libraries, and 12 titles were found in all 24 libraries.   the low end of retention at a single library was 22 of 100 publications, while the high end was 100 (see figure 1). average retention of the 100 publications was 82.5 and the median retention was 85. of the libraries in our sample, 1/3 had retained over 90 of 100 publications, and 3/4 had retained at least 80.   figure 1 titles found, by institution.   retention by year   since our sample publications were equally distributed into four groups of 25, separated by distribution year by 10-year intervals, we could compare retention across all library systems for items distributed during different time periods. overall retention was higher for items distributed in 1989 and 1999 compared to those distributed in 1979 and 2009. as illustrated in figure 2, retention across the four decades covered in the study was as follows: 1979: 80%, 1989: 89%, 1999: 87%, 2009: 74%.   figure 2 percentage of total expected items found, by year of distribution.   retention by region   since the libraries in our study span multiple regions across canada, we can compare retention regionally within academic carl-member libraries. we have done this by grouping each of the 24 libraries into their respective regional associations: council of prairie and pacific university libraries (coppul), ontario council of university libraries (ocul), bureau de coopération interuniversitaire (bci), and council of atlantic university libraries (caul). while our study did not investigate the collections of all library members of these regional associations, this view of our data does provide one perspective on the overall retention of research libraries in the different regions.   figure 3 demonstrates retention among libraries included in our study by membership in regional associations: coppul: 90%, ocul: 88%, bci: 64%, caul: 89%.   figure 3 percentage of total expected items found within regional associations.   coverage within selected partnerships   consortia and partnerships are becoming increasingly important for, and directly involved in, the management of government publications at academic libraries. as such, we wanted to consider membership in selected collaborations as another lens for our analysis. the coppul shared print archive network (coppul-span) is a collaboration between 21 coppul members that implicates government of canada publications, with one aim to preserve and provide access to shared print resources (bigelow et al., 2017). keep@downsview (k@d) is an initiative of five partner institutions, all included in this study, that aims to “preserve the scholarly record in ontario in a shared high-density storage and preservation facility” (keep@downsview, 2020). although publicly available documentation for k@d does not currently indicate that government publications are in scope, we believe there is potential for them to be included in the future (keep@downsview, 2020).   additionally, more than half of the libraries in our study migrated their bibliographic information to one of two shared library management systems following the completion of our data collection, with the promise of significant future collaboration on print resource management and sharing. we thought it important to view our results through the lens of the following additional groupings: ocul’s collaborative futures project (ocul-cf) and bci’s quebec university libraries partnership (bci-qulp).   since our study did not include all members of these partnerships, apart from k@d, we can only provide an incomplete picture of the estimated level of coverage for each of them. however, this view of our data helps provide an idea of potential collection management collaborations for redundancy across the country.   figure 4 shows the number of titles for which at least one copy was found in one member of the four groups. considering only the 24 libraries included in our study, all 100 titles were found within the members of coppul-span, ocul-cf and k@d, while 97 of the titles were found within the members of bci-qulp.   figure 4 titles found in at least one member of selected regional partnerships.   discussion   overall retention   while an overall estimation of retention across all 24 libraries of 82.5% seems positive, it does imply that there has been some evaluation and weeding of dsp collections at the libraries included in this study. based on the original requirements of the dsp retention guidelines, and the way the publication selection criteria for this study were designed, there is a very high likelihood that all libraries in our study held all publications in the sample at the end of 2013. from this perspective, our results suggest that the pan-canadian redundancy provided by the network of fdls that was fundamental to the program and reflective of the value of government information decreased significantly in the 5 years following the termination of the dsp retention requirements.   retention by year   overall retention was noticeably greater for items distributed in 1989 and 1999, compared to those distributed in 1979 and 2009. the significantly lower retention for items distributed in 2009 may be due to the availability of born-digital versions of the same printed publications. our ra reported that in several instances where no print versions of publications distributed in 2009 were found, the library catalogues did provide metadata for electronic versions of the same documents. the lower retention of 1979 items was more of a surprise. it is possible that older government of canada monographs were more likely to be weeded after the expiration of the dsp retention guidelines due to a perceived lack of currency, especially since as standalone items there may not have been relationships to other publications (e.g. part of a series) supporting their retention.   consortial and collaborative opportunities   we also examined how retention could be viewed through membership of the libraries in our study in existing regional associations, partnerships and consortia. while we did not search the catalogues of all members of these groupings, besides k@d, our view of their membership includes those former fdl academic libraries who are members of carl. as such, we believe they represent key stakeholders within these groups in securing long-term access to research materials such as government documents.   viewing retention through the lens of regional association membership revealed that three of the regions have above-average retention levels within our subset of their membership. on the other hand, overall retention within academic carl library members of bci was noticeably lower than for the other regions. for this reason, we suggest that considering the future of access and preservation for these collections within the province of quebec might be of more urgent concern.   however, maintaining a high level of retention at all libraries regionally is not necessarily the most efficient way of ensuring long-term access and preservation. in situations where multiple libraries agree to share resources and responsibility for their preservation, fewer copies per library could provide adequate security for items that, while historically significant, have individually low rates of use. for this reason, we also looked at coverage--whether at least one copy could be found--within existing consortia and preservation-related partnerships.   clearly these existing collaborations provide high levels of collective coverage, with at least one copy of all 100 publications found in three of the four partnerships examined. assuming coverage has not changed significantly since we completed our data collection in 2019, these partnerships appear to provide ample opportunity to proactively secure the future of former federal depository collections before retention drops to problematic levels.   scarcity and retention   the ccpswg overlap study published their final report in september 2020, while we were performing the analysis for our study. the results of their study found that canadian federal government publications had “...limited holdings coverage within the 26 participant libraries. the total number of titles identified was 748,000, and of these, 509,000 (68%) showed only one library holding in the participant group” (ccpswg, 2020, p. ii). the ccpswg also applied a threshold of “...three or fewer title-holdings…” to denote scarcity in their analysis (ccpswg, 2020, p. iv).   while the ccpswg overlap study brings forward important information to consider when examining collection redundancy, the ways in which its methodology and scope differed from ours were significant, meaning that the findings of both studies should also be interpreted differently.   the libraries included in the ccpswg and our study were different, but there was cross-participation of 17 libraries in both studies, with the ccpswg also categorizing tug as a single library in their analysis (ccpswg, 2020, p. vii). our study focused on the collections of former fdls that were also academic library carl members, while the ccpswg included government and public libraries in their study in addition to some carl member libraries (ccpswg, 2020, p. vii). the ccpswg study’s data was collected and overlap analysis conducted computationally with oclc’s greenglass application, which meant that a much larger set of publications was analyzed (748,000), while our study performed these activities manually on a sample of 100 publications. this meant that we could troubleshoot during the search and verification process to ensure matches were accurate even when encountering minor differences in bibliographic information. another important difference is that the ccpswg study gathered data about publications that were still held by libraries participating in their study at the time of data collection, whereas our study looked at what should have been in fdl collections, according to the weekly checklist, as of 2013. the publications our study was searching for should have been found in all the libraries in our study if the intended redundancy of the dsp distribution network had been maintained, meaning that when titles were not found, it likely represented weeding locally, and collectively the beginnings of a dismantling of the distributed network of dsp publications.   using ccpswg’s definition of scarcity (three or fewer holdings) for the full sample of libraries included in our study, we identified no scarce titles, since all 100 of our titles were found in at least 5 libraries. that said, the methodological differences in the two studies means that a different perspective on scarcity could be considered. we used the lens of the partnerships and consortia identified above to examine potential scarcity within their collective holdings.   figure 5 potentially scarce titles within selected regional partnerships.   figure 5 shows the number of titles found in three or fewer members of selected consortia and partnerships that were also libraries included in our study. since we did not investigate the collections of all members of coppul-span, k@d, ocul-cf, and bci-qulp, we cannot be certain that items held at three or fewer of their members also included in our study indicates scarcity within the entire partnership, which is why we have characterized these items as potentially scarce. accordingly, we found 3, 6, 19, and 38 potentially scarce titles in coppul-span, ocul-cf, k@d, and bci-qulp respectively. while it is concerning that we identified several potentially scarce titles within these existing collaborations, it is important to recall that at least 1 copy of all 100 publications was found among the k@d, ocul-cf and coppul-span libraries included in our study, and at least 1 copy of 97 of the 100 publications was found in bci-qulp. while items in the dsp-distributed collections are starting to become scarcer, we believe that the existing partnerships highlighted here have an opportunity to ensure long-term access and discoverability for a large component of the federal depository collection.   based on their findings, the ccpswg recommended “the formation of a national shared print network that coordinates the activities of existing regional shared print initiatives and provides a path to participation for other interested libraries not yet in a shared print program” (ccpswg, 2020, p. 8). in 2021, north: the canadian shared print network was announced as a jointly-funded partnership between multiple canadian organizations and consortia “working towards stable, retained and identified collections in canada,” including government of canada publications (brigham & hafner, 2021).   while the ccpswg overlap study used different methodology, had different participant libraries, and assessed more than just monographs, the recommendations of our study are much the same: now is the ideal moment to develop consortial and shared print collections in canadian academic libraries, to ensure consistent future access to canadian government information. there is evidence that existing regional associations and partnerships have maintained reasonable levels of collection redundancy. we support developing retention strategies and scarcity thresholds within these existing partnerships, with an overall strategy for preservation coordinated nationally. an approach, such as that of north: the canadian shared print network, could represent a key path forward for ensuring long-term access and preservation for historical, print government of canada information.   while electronic government of canada publications were not the focus of this study, it bears mentioning that the canadian government information digital preservation network (cgi dpn), has been working to preserve such content since 2012. the cgi dpn is a lockss (lots of copies keeps stuff safe) network that replicates copies of electronic government of canada publications at multiple locations in an effort to preserve and keep them safe for future generations. the first collection ingested by the cgi dpn included over 100,000 electronic publications collected by the dsp (wakaruk & marks, 2019, pp. 279-280). this archive is publicly accessible through archive-it.   limitations   there are limitations to our study’s research methodology and replicability. firstly, our sample set of titles was created using inclusion and exclusion criteria designed to maximize findability, based on professional experience, printed monographs would likely show less variability in their indications of holdings in their publicly viewable metadata than other publications. manually identifying publications at each institution meant a limited number could be included in the sample to keep it manageable. it would be difficult to predict the difference in results had we included more publication types and a larger set of publications.   secondly, institution-specific approaches to cataloguing, description, and discovery across the 24 library catalogues searched posed some problems for confirming successful matches. as cross and lafortune (2019) found, while there had been many projects to integrate government publications into the general collections of academic libraries, it sometimes made the collections harder to navigate due to the way in which the titles were catalogued (p.8). for example, there may have been different approaches in how to catalogue something as either a monograph or a serial. it is also possible that subsets of some federal government collections may not have been catalogued at some libraries.   thirdly, exact replication of this study would pose a challenge as the data was collected ahead of the rollout of two province-wide academic consortial projects, ocul-cf and bci-qulp. the data collection for this study took place from april 2019 to september 2019. the initial implementation of the ocul-cf shared catalogue, omni, went live in december 2019 (ontario council of university libraries, 2020), and the bci-qulp catalogue, sofia, went live in the summer of 2020 (bureau de coopération interuniversitaire, 2022). before these consortial projects, most academic libraries in these two provinces had their own ils and library catalogues. metadata migration to the consortial systems likely resulted in changes that would make exact replication of this study difficult, though it might also provide opportunities to perform larger-scale analyses within the smaller scope of these individual partnerships. additionally, it is possible that in the three years since our data collection, the government documents collections of one or more of the 24 libraries in our study have changed significantly.   lastly, our focus in this study was on assessing the persistence of the printed publications that were originally distributed by the dsp. we did not attempt to match or gather data related to alternative formats, including electronic versions. our study relied on the publicly viewable catalogue metadata of former fdls to estimate the retention of dsp-distributed materials in 2019. our assumption was that if an item was not found in the catalogue, it was not retained by the library. it was beyond the scope of this project to determine why or when any given item was weeded.   conclusion   in 2013, the dsp ended its print distribution services and terminated the agreements requiring fdls to retain indefinitely most publications received through the program. without the informal, redundant network of publications formed by the fdls, dsp publications became vulnerable to loss. we set out to determine if these federal print collections were still intact and discoverable at academic libraries across the country.   our study searched the collections of 24 former fdl carl academic libraries for a sample of 100 publications distributed to fdls between 1979 and 2009. every publication in our sample was found in at least 5 of the 24 libraries, indicating a degree of redundancy remaining among the collections of carl academic libraries. however, an overall retention rate of 82.5% for the 2400 publication instances expected to be found by our study suggests that weeding of these collections has likely taken place in the five years following the end of dsp distribution.   we hope that our study has generated valuable information about government of canada publications in canadian academic libraries that could be used to support future decision making. the redundancy that was a cornerstone of the dsp network still exists, to a certain extent, and should be leveraged alongside existing collaborations and partnerships to support a pan-canadian discussion about preservation of and access to federal government information in canadian libraries and library networks. coordination and collaboration at the pan-canadian level could help to secure these collections into the future. this could be accomplished through a new initiative, or by supporting or advocating for the work of longstanding collaborations, such as the cgi dpn, or of recently created partnerships, such as north: the canadian shared print network.   author contributions   graeme campbell: conceptualization, methodology, validation, formal analysis, investigation, data curation, writing original draft, writing review & editing, visualization, supervision, project administration, funding acquisition michelle lake: conceptualization, methodology, validation, investigation, data curation, writing original draft, writing review & editing, visualization, supervision (lead), project administration (lead), funding acquisition catherine mcgoveran: conceptualization, methodology, validation, formal analysis, data curation, writing original draft, writing review & editing, visualization, project administration, funding acquisition   acknowledgements   this research project was generously supported by the carl 2018 research in librarianship grant and the 2019 concordia library research grant. we would like to thank our research assistant, dr. claire burrows, for her diligent work and contributions to the project.   references   bigelow, i., bird, g., blackburn, j., nelke, b., brigham, d., & mcdavid, k. (2017). span 5 year report and recommendations. coppul: the council of prairie and pacific university libraries. https://drive.google.com/file/d/1ay8k3pjhxy___p8wbwrrdgtyuoqqltmk/view   brigham, d., & hafner, h. (2021, december 14-15). announcement: newly-formed national shared print program, north [conference presentation]. government information day(s) 2021 conference (online), canada. https://nextcloud.scholarsportal.info/s/asystag48ocb7b6   bureau de coopération interuniversitaire. (2022).  partenariat des bibliothèques universitaires québec: plateforme partagée de services (pps). https://web.archive.org/web/20220222165807/https://biblios-uni-qc.org/nos-projets/plateforme-partagee-de-services-pps-2/   canada. communication canada. (2001). depository services program: retention/disposition guidelines for canadian government depository publications. government of canada. https://web.archive.org/web/20011217203626/http://dsp-psd.pwgsc.gc.ca/info/retention-e.html   canada. depository services program. (1999). full depository libraries. government of canada. https://web.archive.org/web/19990503083525/http://dsp-psd.pwgsc.gc.ca/dsp-psd/aboutdsp/deponew/full-e.html   canada. depository services program. (2005). full depository libraries. government of canada. https://web.archive.org/web/20050422075128/http://dsp-psd.pwgsc.gc.ca/depo/full-e.html   canada. depository services program. (2010). quick reference guide for depository libraries. depository services program. public works and government services canada. https://web.archive.org/web/20130421140839/https://publications.gc.ca/collections/collection_2010/tpsgc-pwgsc/p109-6-2010-eng.pdf   canada. government of canada publications. (2011). full depository libraries. government of canada. http://web.archive.org/web/20111210164444/http://publications.gc.ca/site/eng/locatingourpublications/depositorylibraries/fulldepositories.html   canada. government of canada publications. (2014). about the depository services program. government of canada. https://web.archive.org/web/20140805201507/http://www.publications.gc.ca/site/eng/programs/aboutdsp.html   canada. public works and government services canada. (2013). canadian government publications catalogue: commemorative weekly checklist 13-49. http://publications.gc.ca/collections/collection_2014/tpsgc-pwgsc/p107-1-2013-49.pdf     canada. task group on depository program review. (1991). partners in access. minister of supply and services canada.   canadian association of research libraries. (2021). strategic directions may 2019-may 2022. carl-abrc. https://web.archive.org/web/20220125153703/https://www.carl-abrc.ca/about-carl/strategic-directions/   canadian collective print strategy working group. (2020). final report of the canadian collective print strategy working group. carl-abrc/library and archives canada-bibliotheque et archives canada. https://web.archive.org/web/20220125200207/https://sharedprint.org/wp-content/uploads/2020/11/ccpswg_final_report_eng_complete.pdf   cross, e., & lafortune, s. (2019). government information in canadian academic libraries, 2017-2018: survey of academic librarians. dttp: documents to the people, 47(3), 6-14. https://doi.org/10.5860/dttp.v47i3.7121   depository services program library advisory committee. (2012, august 28). minutes of the depository services program library advisory committee (dsp-lac) teleconference, august 28, 2012. https://web.archive.org/web/20140305162446/https://publications.gc.ca/site/eng/depositorylibraries/dsp-lac/meetingminutes20120828.html   dolan, e. (1989). the depository dilemma: a study of the free distribution of canadian federal government publications to depository libraries in canada. canadian library association.   goldhor, h. (1973). analysis of an inductive method of evaluating the book collection of a public library. libri, 23(1), 6-17. https://doi.org/10.1515/libr.1973.23.1.6    kairis, r. (2003). consortium level collection development: a duplication study of the ohiolink central catalogue. library collections, acquisitions and technical services, 27(3), 317-326. https://doi.org/10.1080/14649055.2003.10765934   kaufman, a., & moon, j. (2013, november 4). farewell to depository services, a building block of democracy [op-ed]. ottawa citizen. https://web.archive.org/web/20131205194722/http://www.ottawacitizen.com/opinion/op-ed/farewell+depository+services+building+block+democracy/9121876/story.html   keep@downsview. (2020). welcome/about keep@downsview. https://downsviewkeep.org/#about   monty, v. (2005). news from the north: canada is a cold place for government publications. dttp: documents to the people, 33(2), 12-14. http://wikis.ala.org/godort/images/c/c4/dttp_v33n2.pdf    morton, b., & zink, s. d. (1992). the dissemination and accessibility of canadian government information. government publications review, 19(4), 385-396. https://doi.org/10.1016/0277-9390(92)90030-f       ontario council of university libraries. (2020). ocul collaborative futures. https://web.archive.org/web/20200619031025/https://ocul.on.ca/projects/collaborative-futures   paterson, s. (2021). the government information landscape in canada. in k. cassell, j. church, k. tallman & c, church (eds.), the government information landscape and libraries. ifla professional report no. 137 (pp.27-47). ifla. https://repository.ifla.org/bitstream/123456789/842/1/ifla-professional-reports-nr-137-en.pdf    potter, w. g. (1982). studies of collection overlap: a literature review. library research 4, 3-21.   statistics canada. (1989). general review of the 1986 census: reference. minister of supply and services canada. https://archive.org/details/198699137e1989eng/page/n1/mode/2up   triuniversity group of libraries. (2000). the first five years: reports from the triuniversity group of libraries. https://web.archive.org/web/20141215034215/https://www.tug-libraries.on.ca/sites/default/files/the_first_five_years-reports_from_the_triuniversity_group_of_libraries.pdf   triuniversity group of libraries. (2018). tug last copy policy. https://web.archive.org/web/20201027173152/https://www.tug-libraries.on.ca/policies/tug-last-copy-policy   wakaruk, a. (2014). what the heck is happening up north? canadian federal government information, circa 2014. dttp: documents to the people, 42(1), 15-20. https://journals.ala.org/index.php/dttp/issue/viewissue/598/354   wakaruk, a., & li, s. (2019). introduction: the evolution of government information services and stewardship in canada. in a. wakaruk & s. li (eds.), government information in canada: access and stewardship. (pp. xiii-xxxi). university of alberta press. https://www.uap.ualberta.ca/book-images/open%20access/9781772124446_web.pdf   wakaruk, a., & marks, s. (2019). the canadian government information digital preservation network: a collective response to a national crisis. in a. wakaruk & s. li (eds.), government information in canada: access and stewardship. (pp. 275-294). university of alberta press. https://www.uap.ualberta.ca/book-images/open%20access/9781772124446_web.pdf   appendix a consortial and partnership memberships of former fdl* academic library carl members   university province coppul span ocul ocul cf k@d bci qulp caul 1.university of alberta ab 🗸           2. university of calgary ab 🗸           3. simon fraser university bc 🗸           4. university of british columbia bc 🗸           5. university of victoria bc 🗸           6. university of saskatchewan sk 🗸           7. university of manitoba mb 🗸           8. mcmaster university on   🗸   🗸     9. queen’s university on   🗸 🗸 🗸     10. tug** on   🗸 🗸       11. university of ottawa on   🗸 🗸 🗸     12. university of toronto on   🗸   🗸     13. university of windsor on   🗸 🗸       14. western university on   🗸 🗸 🗸     15. york university on   🗸 🗸       16. concordia university qc         🗸   17. mcgill university qc         🗸   18. université de montréal qc         🗸   19. université de sherbrooke qc         🗸   20. université du québec à montréal qc         🗸   21. université laval qc         🗸   22. university of new brunswick nb           🗸 23. dalhousie university ns           🗸 24. memorial university of newfoundland nl           🗸   * we classified the library as a former full depository library if it was described as such in at least two of the following sources (statistics canada, 1989, pp. 221-223; canada. depository services program, 1999; canada. depository services program, 2005; canada. government of canada publications, 2011)   ** triuniversities group (guelph, waterloo, wilfrid laurier). waterloo and guelph, are carl members and former fdls, wilfrid laurier is not a carl member and was an sdl, but all three libraries make up the triuniversity group of libraries (tug), which has been in existence since 1995. tug institutions have shared a storage facility since 1996, an ils since 1998, and have had a preservation of last copy agreement since 2006 (triuniversity group of libraries, 2000; triuniversity group of libraries, 2018).   membership in the library partnerships and consortia examined is listed as of 2020.   appendix b list of weekly checklists used to select sample publications   canada. depository services program. (1978-2015). weekly checklist of canadian government publications. public works and government services canada. https://publications.gc.ca/pub?id=9.500735&sl=0 79-10: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1979-10.pdf  79-11: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1979-11.pdf 79-20: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1979-20.pdf 79-21: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1979-21.pdf 79-22: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1979-22.pdf 79-23: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1979-23.pdf 79-30: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1979-30.pdf 79-31: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1979-31.pdf 79-32: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1979-32.pdf 79-33: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1979-33.pdf 79-40: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1979-40.pdf 79-41: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1979-41.pdf 79-42: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1979-42.pdf 79-50: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1979-50.pdf 79-51: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1979-51.pdf 79-52: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1979-52.pdf 80-10: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1980-10.pdf 89-10: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1989-10.pdf 89-11: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1989-11.pdf 89-20: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1989-20.pdf 89-21: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1989-21.pdf 89-22: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1989-22.pdf 89-23: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1989-23.pdf 89-30: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1989-30.pdf 89-31: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1989-31.pdf 89-32: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1989-32.pdf 89-33: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1989-33.pdf 89-34: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1989-34.pdf 89-40: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1989-40.pdf 89-41: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1989-41.pdf 89-42: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1989-42.pdf 89-43: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1989-43.pdf 89-44: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1989-44.pdf 89-50: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1989-50.pdf 89-51: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1989-51.pdf 89-52: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1989-52.pdf 90-10: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1990-10.pdf 90-11: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1990-11.pdf 99-10: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1999-10.pdf 99-11: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1999-11.pdf 99-12: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1999-12.pdf 99-20: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1999-20.pdf 99-30: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1999-30.pdf 99-31: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1999-31.pdf 99-32: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1999-32.pdf 99-40: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1999-40.pdf 99-41: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1999-41.pdf 99-42: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1999-42.pdf 99-50: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1999-50.pdf 99-51: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-1999-51.pdf 2000-10: https://publications.gc.ca/collections/collection_2016/spac-pspc/p107-1-2000-10.pdf 2009-10: https://publications.gc.ca/collections/collection_2017/spac-pspc/p107-1/p107-1-2009-10.pdf 2009-11: https://publications.gc.ca/collections/collection_2017/spac-pspc/p107-1/p107-1-2009-12.pdf 2009-12: https://publications.gc.ca/collections/collection_2017/spac-pspc/p107-1/p107-1-2009-12.pdf 2009-20: https://publications.gc.ca/collections/collection_2017/spac-pspc/p107-1/p107-1-2009-20.pdf 2009-21: https://publications.gc.ca/collections/collection_2017/spac-pspc/p107-1/p107-1-2009-21.pdf 2009-22: https://publications.gc.ca/collections/collection_2017/spac-pspc/p107-1/p107-1-2009-22.pdf 2009-30: https://publications.gc.ca/collections/collection_2017/spac-pspc/p107-1/p107-1-2009-30.pdf 2009-31: https://publications.gc.ca/collections/collection_2017/spac-pspc/p107-1/p107-1-2009-31.pdf 2009-32: https://publications.gc.ca/collections/collection_2017/spac-pspc/p107-1/p107-1-2009-32.pdf 2009-40: https://publications.gc.ca/collections/collection_2017/spac-pspc/p107-1/p107-1-2009-40.pdf 2009-41: https://publications.gc.ca/collections/collection_2017/spac-pspc/p107-1/p107-1-2009-41.pdf 2009-42: https://publications.gc.ca/collections/collection_2017/spac-pspc/p107-1/p107-1-2009-42.pdf 2009-43: https://publications.gc.ca/collections/collection_2017/spac-pspc/p107-1/p107-1-2009-43.pdf 2009-50: https://publications.gc.ca/collections/collection_2017/spac-pspc/p107-1/p107-1-2009-50.pdf 2009-51: https://publications.gc.ca/collections/collection_2017/spac-pspc/p107-1/p107-1-2009-51.pdf 2009-52: https://publications.gc.ca/collections/collection_2017/spac-pspc/p107-1/p107-1-2009-52.pdf   appendix c selection criteria for publication sample set items   publication selection process for each of 1979, 1989, 1999, and 2009, five publications were selected starting from each of the 10th, 20th, 30th, 40th and 50th issues of the weekly checklist. the following steps were undertaken to select publications from the checklists:   1.      a random checklist starting page for selecting publications was generated using the rand function in google sheets. 2.      publication descriptions in the checklist were reviewed in the order they appeared until one was found matching the “checklist item selection criteria” (see below). the matching publication was added to the sample list. 3.      following the selection of a publication, review of the checklist descriptions continued after skipping ahead to the next department or organization in order to select a set of publications from a diverse set of author organizations. new author organizations are defined in the weekly checklist by the use of all caps in the headings. branches or subunits within author organizations are indicated by title case headings, and did not constitute changes in author organization for the purposes of this step. 4.      steps 2 and 3 were repeated until 5 publications were selected. if the end of the checklist was reached before 5 items could be selected using our criteria and method, the process was started again at page 1 of the checklist and steps 2 and 3 were repeated until a total of 5 publications were identified.   if the steps above did not result in 5 publications being selected from the checklist issue, the team moved to the next checklist issue numerically (e.g. from 79-10 to 79-11) and repeated these steps until a total of 5 publications were selected. this occasionally necessitated the inclusion of checklists from subsequent years (e.g. 80-10).   publications included in the sample for this study were added to a spreadsheet for tracking purposes. for each publication in the sample, the following metadata was collected from the weekly checklists:   ○       full title  ○       date ○       author department or agency ○       format ○       language ○       checklist number ○       isbn ○       government of canada classification number ○       distribution information   checklist item inclusion and exclusion criteria inclusion criteria the following criteria had to be met for a publication to be included in this study:   1.      materials published in print format. this does not mean that materials could not also be available electronically, but that only print records were selected from the weekly checklist for inclusion in the sample. 2.      materials published as monographs or standalone volumes. monographic series issues were not included if it seemed likely that they could have been catalogued solely under their series title. 3.      materials that were distributed in one of the following ways (distribution statements provided in the weekly checklist varied in wording): ●       one free copy to depository libraries ●       available to full depository libraries only ●       one free copy available to selective depositories 4.      materials that were listed in both english and french, or as bilingual, in the same checklist. exclusion criteria the following materials were excluded from this study, even if they met the inclusion criteria above:   1.      materials where the publication author or type is one of the following: ●       house of commons ●       senate ●       bills / committees ●       library of parliament ●       budget / economic action plan / estimates / public accounts / reports on plans and priorities ●       all annual financial cycle publications 2.      materials in the following formats: ●       cd-rom ●       kits ●       microfilm / microfiche ●       electronic ●       loose leaf ●       vhs ●       dvd ●       folder ●       pamphlets / ephemera ●       processed 3.      serial publications, indicated by the presence of an issn, or materials published on a periodic basis, typically indicated by the following words: ●       irregular, annual, quarterly, monthly, weekly, etc. ●       bulletin 4.      materials that were not distributed through the dsp typically indicated by the following statement “available only from issuing agency”. 5.      materials that were a specific edition of an item, as the discarding of older editions by fdls was permitted through the dsp agreement.   appendix d titles and author organizations, as listed in the weekly checklist, of the 100 publications used as the sample set in our study.   publication title author organization checklist a study of profits and profit margins in the food industry / étude des bénéfices et des marges bénéficiaires dans l'industrie alimentaire anti-inflation board / commission de lutte contre l'inflation 79-10 people planning: a guide for employers / planification du personnel: guide à l'intention des employeurs employment and immigration / emploi et immigration 79-10 report on the cheque: some modernization / rapport sur le chèque: un peu plus moderne law reform commission of canada / commission de réforme du droit du canada 79-10 home-made dairy products / produits laitiers maison agriculture canada / agriculture canada 79-11 the basics of oil spill cleanup, with particular reference to southern canada / principes fondamentaux du nettoyage des déversements d'hydrocarbures, compte tenu spécialement du sud du canada fisheries and environment canada / pêches et environnement canada 79-11 agriculture is food / agriculture à votre table agriculture canada / agriculture canada 79-20 partners in pursuit of excellence: a nati[o]nal policy on amateur sport. a white paper on sport. / ensemble vers l'excellence: politique nationale sur the sport amateur. un livre blanc sur le sport. health and welfare canada / santé et bien-être social canada 79-20 parks canada policy / politique de parcs canada indian and northern affairs / affaires indiennes et du nord 79-20 interdependence and inflation / interdépendance et inflation anti-inflation board / commission de lutte contre l'inflation 79-22 national library of canada: bibliography / bibliothèque nationale du canada: une bibliographie national library / bibliothèque nationale 79-23 composting / le compostage agriculture canada / agriculture canada 79-30 telidon / télidon communications canada / communications canada 79-30 meaning of dismissal: the meaning of dismissals under division v.7 of part iii of the canada labour code / le congédiement: le congédiement, aux termes de la division v.7 de la partie iii du code canadien du travail labour canada / travail canada 79-32 the riel rebellions: a cartographic history / le récit cartographique des affaires riel public archives canada / archives publiques canada 79-32 access to information: independent administrative agencies / l'accès à l'information: organismes administratifs autonomes law reform commission of canada / commission de réforme du droit du canada 79-33 the future of the automobile in canada: growth, usage, energy, technology, safety, environment, other modes, and urban and inter-city aspects. / l'avenir de l'automobile au canada: croissance, utilisation, énergie, technologie, sécurité, environnement, autres modes de transport, perspective urbaine et interurbaine. transport canada / transports canada 79-40 the canadian crown / la monarchie au canada government house / résidence du gouverneur général 79-40 report of the commission of inquiry into redundancies and lay-offs / rapport de la commission d'enquête sur les excédents de main-d'oeuvre et les mises à pied labour canada / travail canada 79-41 aphids infesting potatoes in canada: a field guide / les pucerons nuisibles de la pomme de terre au canada: guide d'identification sur le terrain agriculture canada / agriculture canada 79-42 report of the special committee on the review of personnel management and the merit principle in the public service / rapport du comité spécial sur la gestion du personnel et le principe du mérite public service commission / commission de la fonction publique 79-42 checklist of canadian directories, 1790-1950 / répertoire des annuaires canadiens, 1790-1950 national library / bibliothèque nationale 79-50 synopsis and recommendations from responsible regulation: an interim report by the economic council of canada / résumé et recommandations: rationalisation de la réglementation publique, un rapport provisoire du conseil économique du canada economic council of canada / conseil économique du canada 79-50 education and working canadians: report of the commission of inquiry on educational leave and productivity / l'éducation et le travailleur canadien: rapport de la commission d'enquête sur le congé-éducation et la productivité labour canada / travail canada 79-51 chronicles of the anti-inflation board / historique de la commission de lutte contre l'inflation anti-inflation board / commission de lutte contre l'inflation 79-51 the canadian military experience, 1867-1967: a bibliography / bibliographie de la vie militaire au canada, 1867-1967 national defence / défense nationale 80-10 how we work for you / à votre service labour canada / travail canada 89-10 safety handbook for the inshore fisherman / guide de sécurité pour pêcheurs côtiers transport canada / transports canada 89-10 canada's food aid: helping africans to help themselves / l'aide alimentaire canadienne: aider les africains à s'aider euxmêmes. canada international development agency / agence canadienne de développement international 89-11 smoke in the workplace: an evaluation of smoking restrictions / l'usage du tabac en milieu de travail: évaluation de la politique concernant l'usage du tabac health and welfare canada / santé et bien-être social canada 89-11 libraries for all: guidelines for library services for disabled canadians / des bibliothèques pour tous: lignes directrices pour les services de bibliothèque offerts aux canadiens handicapés national library / bibliothèque nationale 89-11 child sexual abuse overview: a summary of 26 literature reviews and special projects / aperçu général sur les agressions sexuelles contre les enfants: résumé de 26 analyses de documentation et de projets spéciaux health and welfare canada / santé et bien-être social canada 89-20 victims: questions & answers on parole / victimes: questions & réponses sur les libérations conditionnelles solicitor general canada / solliciteur général canada 89-20 report on a review concerning coated broadwoven polyester fabrics for use in the manufacture of textile covers / rapport du réexamen des tissus larges de polyester enduits servant à la fabrication de couvertures de textile. industry, science and technology / industrie, sciences et technologie 89-21 the active health report on alcohol, tobacco and marijuana: what we think, what we know, what we do / action santé: alcool, tabac et marijuana: le rapport action santé: nos conceptions et nos actions. health and welfare canada / santé et bien-être social canada 89-22 a new frontier: globalization and canada's financial markets / le nouvel espace financier: les marchés canadiens et la mondialisation economic council of canada / conseil économique du canada 89-23 canada and the changing atmosphere / le canada et l'atmosphère en évolution environment canada / environnement canada 89-31 main findings report of the canadian blood pressure survey / principaux résultats de l'étude sur l'hypertension artérielle au canada health and welfare canada / santé et bien-être social canada 89-32 goods and services tax: an overview / taxe sur les produits et services: vue d'ensemble finance canada / finances canada 89-32 development day activities / activités pour la journée du développement. canada international development agency / agence canadienne de développement international 89-33 ten years to 2000: a strategy document / dans dix ans, l'an 2000: un document stratégique natural sciences and engineering research council / conseil de recherches en sciences naturelles et en génie 89-34 canada youth & aids study / étude sur les jeunes canadiens face au sida health and welfare canada / santé et bien-être social canada 89-40 the importance of wildlife to canadians in 1987: highlights of a national survey / l'importance de la faune pour les canadiens en 1987: rapport sommaire de l'enquête nationale. environment canada / environnement canada 89-41 international development video library / vidéothèque du développement international canada international development agency / agence canadienne de développement international 89-43 handbook on nursing / guide sur les soins infirmiers health and welfare canada / santé et bien-être social canada 89-43 a decade of achievement: environment and energy research and development / dix années de succès: recherche et développement dans le domaine de l'environnement et de l'énergie. environment canada / environnement canada 89-44 charting canada's future: a report of the demographic review / esquisse du canada de demain: rapport de l'étude démographique health and welfare canada / santé et bien-être social canada 89-50 commission of inquiry into the air ontario crash at dryden, ontario: interim report / commission d'enquête sur l'écrasement d'un avion d'air ontario à dryden (ontario): rapport provisoire privy council / conseil privé 89-50 a management model / un modèle de gestion canadian centre for management development / centre canadien de gestion 89-52 an evaluation of the beaufort sea environmental assessment panel review / évaluation de l'examen mené par la commission d'évaluation environmentale du projet de la mer de beaufort   environment canada / environnement canada 90-11 action towards healthy eating... canada's guidelines for healthy eating and recommended strategies for implementation: the report of the communications/implementation committee / action concertée pour une saine alimentation... recommandations alimentaires pour la santé des canadiens et canadiennes et stratégies recommandées pour la mise en application: rapport du comité des communications et de la mise en application health and welfare canada / santé et bien-être social canada 90-11 the health of our air: toward sustainable agriculture in canada / la santé de l'air que nous respirons: vers une agriculture durable au canada agriculture and agri-food canada / agriculture et agroalimentaire canada 99-10 the live-in caregiver program: information for employers and live-in caregivers from abroad / le programme concernant les aides familiaux residants: renseignements à l'intention des employeurs et des aides familiaux résidants étrangers citizenship and immigration canada / citoyenneté et immigration canada 99-10 motivations for school choices by eligible parents outside quebec / motivations en ce qui a trait aux choix scolaires chez les parents ayants droit hors québec commissioner of official languages / commissaire aux langues officiellles 99-10 highlights: 1998 edition of the canadians std guidelines. / l'essentiel des lignes directrices canadiennes pour les mts, édition de 1998. health canada / santé canada 99-10 sida's policy on gender equality / politique de l'acdi en matière d'égalité entre les sexes. canadian international development agency / agence canadienne de développement international 99-12 the ozone primer / la couche d'ozone environment canada / environnement canada 99-20 a study of resiliency in communities. / enquête sur le ressort psychologique des communautés. health canada / santé canada 99-20 sustaining growth, human development, and social cohesion in a global world: a report prepared for the policy research initiative, february, 1999 / soutenir la croissance, le développement humain et la cohésion sociale dans un contexte de mondialisation. un rapport préparé pour le projet de recherche sur les politiques, février 1999 privy council / conseil privé 99-20 setting judicial compensation: multidisciplinary perspectives / établir la rémunération des juges: perspectives multidisciplinaires law commission of canada / commission du droit du canada 99-20 canada's military air forces, 1914-1999 / l'aviation militaire canadienne 1914-1999 national defence / défense nationale 99-20 workplace literacy pilot projects: a discussion paper / les projets-pilotes en alphabétisation en milieu de travail: document de discussion human resources development canada / développement des ressources humaines canada 99-31 atlantic canada: opportunities / la région de l'atlantique: perspectives atlantic canada opportunities agency / agence de promotion économique du canada atlantique 99-31 how are health program reforms affecting seniors? a participatory evaluation guide / la réforme des programmes de santé... quels effets sur les aînés: guide d'évaluation participative health canada / santé canada 99-32 the state of small business and entrepreneurship in northern ontario. / situation des petites entreprises et de l'entrepreneuriat dans le nord de l'ontario. industry canada / industrie canada 99-32 the way forward: action plan for the office of the ombudsman, submitted to the honourable art eggleton, by andré marin, ombudsman. / allons de l'avant: plan directeur du bureau de l'ombudsman. présenté à l'honorable art eggleton, par andré marin.   national defence / défense nationale 99-32 reforming canada's financial services sector: a framework for the future. / la réforme du secteur des services financiers canadien: un cadre pour l'avenir finance canada / finances canada 99-40 intergovernmental collaboration on hiv/aids: a discussion paper / la collaboration intergouvernementale dans la lutte contre le sida: document de travail health canada / santé canada 99-40 ready for year 2000 -a practical guide for small and medium-sized businesses / objectif 2000 -un guide pratique pour les pme [petites et moyennes entreprises] industry canada / industrie canada 99-40 official languages: words in deeds: official languages best practices compendium / les langages officielles: des mots en action: recueil de pratiques exemplaires en langues officielles. treasury board / conseil du trésor 99-40 health canada progress report, 1999 / rapport d'étape de santé canada, 1999 health canada / santé canada 99-42 canadian research on immigration and health: an overview. / recherche sur l'immigration et la santé au canada: un aperçu. health canada / santé canada 99-50 year 2000 readiness: canada prepares / l'état de préparation pour l'an 2000: le canada se prépare foreign affairs and international trade / affaires étrangères et commerce international 99-51 your internet business: earning consumer trust: a guide to consumer protection for on-line merchants / votre commerce dans internet: gagner la confiance des consommateurs: un guide pour la protection des consommateurs à l'intention des commerces en direct industry canada / industrie canada 99-51 shaping the future of canadian defence: a strategy for 2020 / façonner l'avenir de la défense canadienne: une stratégie pour l'an 2020 national defence / défense nationale 99-51 evaluation of the urban social housing programs / évaluation des programmes de logement social en milieu urbain canada mortgage and housing corporation / société canadienne d'hypothèques et de logement 2000-10 talking with your teen about drugs / aborder le sujet des drogues avec son adolescent health canada / santé canada 2009-10 small business financing profiles borrowers under the canada small business financing program / profils de financement des petites entreprises emprunteurs dans le cadre du programme de financement des petites entreprises du canada industry canada / industrie canada 2009-10 liquefied natural gas: a canadian perspective / gaz naturel liquéfié: perspective canadienne national energy board / office national de l'énergie 2009-10 handbook on sensitive practice for health care practitioners: lessons from adult survivors of childhood sexual abuse / manuel de pratique sensible à l'intention des professionnels de la santé: leçons tirées des personnes qui ont été victimes de violence sexuelle durant l'enfance public health agency of canada / agence de santé publique du canada 2009-10 a status report on the national pharmaceuticals strategy: a prescription unfilled / rapport d'étape sur la stratégie nationale relative aux produits pharmaceutiques: une ordonnance non remplie health council of canada / conseil canadien de la santé 2009-12 gender matters institute of gender and health strategic plan 2009-2012. / le genre a de l'importance : institut de la santé des femmes et des hommes plan stratégique 2009-2012. canadian institutes of health research / instituts de recherche en santé du canada 2009-20 2009-2012 employment equity and diversity plan [at head of title: implementation of the employment equity act] / plan d'équité en matière d'emploi et de diversité 2009-2012 [en tête du titre: mise en oeuvre de la loi sur l'équité en matière d'emploi]   canada economic development for quebec regions / développement économique canada pour les régions du québec 2009-21 communities at risk : security infrastructure pilot program: application guide / programme pilote de financement des projets d'infrastructure de sécurité pour les collectivités à risque: guide de demande public safety canada / sécurité publique canada 2009-21 up against a wall coping with becoming a teen when you have been maltreated as a child: substance use among adolescents in child welfare versus adolescents in the general population : a comparison of the maltreatment and adolescent pathways (map) longitudinal study and the ontario student drug use survey (osdus) datasets / au pied du mur faire face à l'adolescence après avoir été maltraité pendant l'enfance: consommation d'alcool et de drogues chez les adolescents suivis par les services de protection de la jeunesse par rapport aux adolescents dans la population générale : comparaison des ensembles de données issues de l'étude longitudinale sur les mauvais traitements et le cheminement des adolescents (mca) et du sondage sur la consommation de drogues des élèves de l'ontario (scdeo) public health agency of canada / agence de santé publique du canada 2009-22 audit of the federal student work experience program and subsequent appointments through bridging mechanisms / vérification du programme fédéral d'expérience de travail étudiant et des nominations subséquentes effectuées au moyen des mécanismes d'intégration public service commission / commission de la fonction publique 2009-22 atlantic integrated commercial fisheries initiative (aicfi): a reporting procedures handbook for contribution agreement recipients / initiative des pêches commerciales intégrées de l'atlantique (ipcia): guide sur la présentation de rapports à l'intention des bénéficiaires d'accord de contribution fisheries and oceans canada / pêches et océans canada 2009-30 environmental code of practice for metal mines 2009 / code de pratiques écologiques pour les mines de métaux, 2009 environment canada / environnement canada 2009-31 federal framework for aboriginal economic development / cadre fédéral pour le développement économique des autochtones indian and northern affairs / affaires indiennes et du nord 2009-31 roadmap for canada's linguistic duality 2008-2013: acting for the future / feuille de route pour la dualité linguistique canadienne 2008-2013: agir pour l'avenir canadian heritage / patrimoine canadien 2009-32 contribution funds for non-governmental organizations a handbook / contributions à l'intention des organisations non gouvernementales guide justice canada / justice canada 2009-32 inuit relations secretariat progress report [june 2007 december 2008] / secrétariat des relations avec les inuit rapport d'étape [2007 juin 2008 décembre] indian and northern affairs / affaires indiennes et du nord 2009-41 halifax citadel, georges island, fort mcnab, prince of wales tower and york redoubt national historic sites of canada management plan / lieux historiques nationaux du canada de la citadelle-d'halifax, de l'île-georges, du fort-mcnab, de la tour-prince-de-galles et de la redoute-york plan directeur parks canada / parcs canada 2009-42 audit of the immigration and refugee board of canada: a report by the public service commission of canada / vérification de la commission de l'immigration et du statut de réfugié du canada: rapport de la commission de la fonction publique du canada public service commission / commission de la fonction publique 2009-42 mentally healthy communities aboriginal perspectives / des collectivités en bonne santé mentale points de vue autochtones canadian institute for health information / institut canadien d'information sur la santé 2009-42 the naval service of canada, 1910-2010: the centennial story / le service naval du canada, 1910-2010: cent ans d'histoire national defence / défense nationale 2009-43 canadian addiction survey (cas): focus on gender / enquête sur les toxicomanies au canada (etc): tendances selon le sexe health canada / santé canada 2009-50 quttinirpaaq national park of canada management plan / parc national du canada quttinirpaaq, plan directeur parks canada / parcs canada 2009-50 canada remembers the burma campaign / le canada se souvient la campagne de birmanie veterans affairs / affaires des anciens combattants 2009-50 information on mammography for women aged 40 and older: a decision aid for breast cancer screening in canada / renseignements sur la mammographie à l'intention des femmes de 40 ans et plus: un outil d'aide à la prise de décision pour le dépistage du cancer du sein au canada public health agency of canada / agence de santé publique du canada 2009-52 coding rules for static-2002 / règles de codage pour l'échelle statique-2002 public safety canada / sécurité publique canada 2009-52   appendix e research data collection workflow   publication search process our search process included searching and matching. for each publication, and in each library’s catalogue, a four-step search process was followed. if after any step a potential match was found, it was verified against the required matching fields. a search was deemed successful if it matched all five required matching fields.   if all four steps of the search process were completed and no verified matching results were identified, the search was deemed failed and the publication was marked as not found at that library.   search process:   1.      search for the exact, full title 2.      search for the title, but with punctuation and subtitles removed 3.      search for isbn 4.      search for author department or agency, in combination with 2-4 keywords from the title. if this step is reached, ensure that the keywords used are the same for each library’s catalogue.   required matching fields:   1.      title (formatting may vary) 2.      date 3.      author department or agency 4.      format 5.      language   data collection for each successful search in each library catalogue, the following metadata was collected from the catalogue record.   ○       full title  ○       author department or agency ○       status ○       record url ○       location(s) ○       call number(s)   searches for the 100 titles in our sample were performed at each of the 24 libraries in our study.   replacement publications   some challenges were encountered in the search process that resulted in having to identify replacement publications for the sample list. for example, upon viewing the full bibliographic descriptions of some publications originally included in our sample set, we found they had attributes that should have excluded them from our study based on our selection criteria.   a total of 10 publications were replaced in the sample list. all replacement publications were labelled with an “a” after the id number in our data collection spreadsheet.     news   call for papers: charleston conference (south carolina november 5–9, 2018)      2018. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29436       call for papers charleston conference  november 5-9, 2018 http://www.charlestonlibraryconference.com/call-for-papers/ we are excited to announce that the 2018 call for papers, ideas, panels, debates, diatribes, speakers, poster sessions, etc. is now open. the proposal deadline is friday, july 13. submit your proposal now!   please read the program proposal instructions before you begin. do you have ideas, challenges, solutions, or information to share? we’re seeking proposals on topics related to collection development and acquisitions, including, but not limited to:   analysis and assessment, budgeting, discovery, digital scholarship, end users/use statistics, management/leadership, library entrepreneurship, professional development, scholarly communication, and technology and trends. another area of focus is our “up and comer” session thread. this is for topics relating to, or presenters who are new to the profession, new to the charleston conference, mlis students or recent graduates, etc. it is related to the up and comer award, which will open for nominations in june. 2017 award winners are listed at http://www.charlestonlibraryconference.com/2017-up-comer-award-winners/. the charleston conference wishes to provide opportunities for everyone to be heard. we encourage proposals that include representation from the different viewpoints and stakeholders in the scholarly communications process, so please consider the diversity of your panel. also, please note that conference sessions are meant for librarians, publishers, and vendors to discuss issues of interest to them all. they are not an opportunity for marketing products or services. 2018 theme and important dates 2018 registration rates and information questions should be directed to beth bernhardt at brbernha@uncg.edu or leah hinds at leah@charlestonlibraryconference.com. requirements charleston conference presenters will be required to: register for and attend the conference; and grant permission for possible video and/or audio recording and broadcast of their presentation. it is strongly encouraged, but not required, that you submit a paper for the conference proceedings. if you elect to do so, we ask that you: assign the charleston conference first publication rights, and submit your paper through our digital commons portal on the purdue university press website using the guidelines and the deadline provided. slides and rough outlines will not be accepted. selection criteria the charleston conference planning committee will evaluate the content of your proposal for relevance to the conference theme and threads, originality, and timeliness. special attention will be given to proposals that: are oriented toward providing practical, concrete information for practicing professionals; demonstrate innovative or entrepreneurial thinking; that include a diverse representation from the different viewpoints and stakeholders in the scholarly communications process; generate ideas or report research that contribute to ongoing discussion about the future of the library and information industry; present strategies for effectively implementing new ideas and technology; and encourage active learning among conference attendees.   please note: conference sessions are meant for librarians, publishers, and vendors to discuss issues of interest to them all. they are not an opportunity for marketing products or services. proposal notifications will be sent out in early august.     research article   learning about student research practices through an ethnographic investigation: insights into contact with librarians and use of library space   eamon tewell reference & instruction librarian long island university brooklyn brooklyn, new york, united states of america email: eamon.tewell@liu.edu   kimberly mullins instructional design librarian long island university post brookville, new york, united states of america email: kimberly.mullins@liu.edu   natalia tomlin technical services librarian long island university post brookville, new york, united states of america email: natalia.tomlin@liu.edu   valeda dent dean of libraries st. john’s university jamaica, new york, united states of america email: dentv@stjohns.edu   received: 28 july 2017    accepted: 23 oct. 2017         2017 tewell, mullins, tomlin, and dent. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – student research habits and expectations continue to change, complicating the design of library spaces and the provision of research support. this study’s intent was to explore undergraduate and graduate student research and study needs at a mid-sized university’s two campuses in the northeastern united states, and to improve librarians’ understandings of these practices so that more appropriate services and spaces may be developed to support student learning.   methods – the research project utilized a primarily qualitative design for data collection that spanned from fall 2012 to summer 2013, consisting of an online questionnaire, unobtrusive observations, and in-depth semi-structured interviews. data collection commenced with a questionnaire consisting of 51 items, distributed through campus email to all students and receiving 1182 responses. second, 32 hours of unobtrusive observations were carried out by librarians, who took ethnographic “field notes” in a variety of library locations during different times and days of the week. the final method was in-depth interviews conducted with 30 undergraduate and graduate students. the qualitative data were analyzed through the application of a codebook consisting of 459 codes, developed by a data analysis team of 4 librarians.   results – the results address topical areas of student interactions with librarians, contact preferences, and use of library space. of the interviewees, 60% contacted a librarian at least once, with texting being the most popular method of contact (27%). in being contacted by the library, students preferred a range of methods and generally indicated interest in learning about library news and events through posters and signage. participants were less interested in receiving library contact via social media, such as facebook or twitter. regarding student use of and preference for library space, prominent themes were students creating their own spaces for individual study by moving furniture, leaving personal items unattended, the presence of unwanted noise, and a general preference for either working nearby other students in groups or in carrels to facilitate individual study.   conclusions – being aware of student research processes and preferences can result in the ability to design learning environments and research services that are more responsive to their needs. ethnographic research methods, as part of an ongoing research process, are recommended as a means to better understand library user practices and expectations.     introduction   academic librarians have increasingly implemented ethnographic approaches to understanding how patrons utilize library spaces, resources, and services, due to the unique contextual insights that can be revealed. as noted in a recent review of the literature, ramsden (2016) observes that the use of ethnographic methods by librarians has increased greatly since the mid-2000s. broadly defined, ethnographic research in libraries takes the form of exploratory investigations into how a library is used or conceived of. instead of seeking to predict student behaviours or measure library use, these studies aim to cultivate a greater understanding of what patrons do in actuality, with an emphasis on their motivations or reasoning for doing so.   using an ethnographic approach, the long island university (liu) libraries in brookville, ny and brooklyn, ny conducted a four-year research project to better understand undergraduate and graduate student help-seeking and study habits at its suburban residential and urban commuter campuses. this project’s intent was to improve liu librarians’ understandings of students’ research and study needs, and used the methods of in-depth interviews, unobtrusive observations, and a survey questionnaire to do so. the ethnographic framework was adopted in order to better consider students’ practices from their own perspectives, and to situate research and study habits within the complex social settings that they take place.   literature review   representing a range of qualitative research methods and based in the field of anthropology, ethnography seeks to understand the thoughts, experiences, and/or actions of a given culture through observation and interpretation. ethnographic research necessarily involves the contextualization of practices and activities, and through a longitudinal and iterative process of information gathering, can allow for the detailed description and understanding of a subject under study. because of its focus upon social behaviours, ethnography is particularly useful for developing insights into people’s experiences and expectations.   in libraries, ethnographic research can contribute to the essential tasks of “understanding users, the way they work, and the various challenges they face when trying to locate, retrieve and use information” (dent goodman, 2011, p. 1). through an analysis of the library and information studies literature, khoo, rozaklis, and hall (2012) identified five primary types of ethnographic research methods employed by researchers in library settings: observations, interviews, fieldwork, focus groups, and cultural probes (p. 84). many researchers acknowledge that, like other qualitative methods, ethnography is a process requiring considerable time and resources to conduct. yet lanclos and asher (2016) point out that as a practice ethnography holds significant advantages, including potentially “profound implications for the nature of libraries, for definitions of work and practice, for imagining the connections that libraries have within their larger contexts, for holistic considerations of student and faculty experiences, actions, and priorities.”   the field of academic librarianship has seen several particularly influential ethnographic studies, beginning with the university of rochester’s study that culminated in foster and gibbons’ 2007 book studying students: the undergraduate research project at the university of rochester. fresno state (delcore, mullooly, & scroggins, 2009) and mit libraries (gabridge, gaskell, & stout, 2008) also conducted large-scale studies using a combination of participant observation, interviews, mapping, and photo diaries around this time. two recent studies of major significance are the ethnographic research in illinois academic libraries (erial) project conducted at five universities illinois representing both public and private institutions (duke & asher, 2012) and the city university of new york’s undergraduate scholarly habits ethnography project, which explored student research and technology use at six public commuter colleges (regalado & smale, 2015; smale & regalado, 2017).   in a review of ethnographic methods in libraries, ramsden (2016) describes the considerable range of subjects this approach has been applied to: “ethnography has been utilised to learn more about collection management, use of library materials or technology, information seeking behaviours, reference desk use, student behaviour, space organisation and wayfinding, and to analyse (and even as a student task in) library inductions and teaching” (p. 256). researchers continue to adopt and develop inventive uses of ethnography in library settings, whether as a method, as in dunne’s (2016) shadowing of several students during the final weeks of their undergraduate studies and kinsley, schoonover, and spitler’s (2016) use of gopro cameras to learn about students’ processes of finding books in library stacks, or as pedagogical inspiration, as in pashia and critten’s (2015) use of mapping and observation in library orientation sessions. recent studies with implications for the research at hand include holder and lange’s (2014) mixed-methods examination of library space and patron satisfaction, allan’s (2016) analysis of student awareness of librarians’ roles within a learning commons setting, and khoo, rozaklis, hall, and kusunoki’s (2016) surveys of student perception and usage of library space. these implications will be addressed in the discussion section.   aims   the purpose of this study was to better understand undergraduate and graduate students’ research and study needs at long island university libraries in order to inform the design of library services and environments. by studying the local culture of student research practices through self-reported behaviours and unobtrusive observations, the researchers sought to increase their comprehension of what research and study habits students are actually engaged in, and to use this information to identify ways to create a library more responsive to and reflective of students’ expressed needs.   the research began as an initiative of the dean of libraries, whose background in anthropology was invaluable as inspiration for the project and in training librarians regarding data collection procedures. the project initially intended to learn more about how students were using the library website and electronic devices for their academic work. when it quickly became clear how closely intertwined the use of electronic devices, academic work, and library resources and space were, the project’s scope was expanded to encompass the additional areas. the study did not begin with predetermined research questions in order to remain open to possibilities during data collection and analysis, but instead focused upon the intersection of student research and study habits and library use.   this study holds potential significance in terms of both methodology and findings. while the aim of the research is not to provide replicable or generalizable findings, the project presents a methodology that examines a topic from multiple perspectives and allows for the triangulation of results. though studies that examine people’s behaviours describe results that can and will change over time, this investigation’s aims encourage a greater understanding on part of the researchers that will ideally inspire future research and additional understanding, all necessarily rooted in different times and places. this study contributes to the evidence base in that it examines the findings in relation to other studies and compares where they meet or diverge. in adopting a combination of qualitative and quantitative methods using an ethnographic framework, the research methods also represent a contribution to the literature.   methods   data collection   this project utilized a primarily qualitative design but drew upon survey data to formulate and revise the in-depth interview questions. the data collection methods consisted of unobtrusive observations, interviews, and a survey questionnaire, and involved a total of 16 librarians and staff members across 2 campuses. each research team member underwent ethics training in research involving human participants. data collection occurred from fall 2012 to summer 2013, while the coding and analysis of interview transcripts and observations began in spring 2014 and concluded in spring 2016. table 1 provides a summary of the project’s timeline.     table 1. data collection and analysis timeline for the study conducted time period action summer 2012 irb approval received fall 2012 survey distributed spring 2013 observations conducted spring and summer 2013 interviews conducted spring 2014 coding process started summer 2015 coding process completed fall 2015 data analysis started spring 2016 data analysis completed     the first step of data collection was the development and distribution of a survey questionnaire consisting of 51 multiple choice and open-ended items. the primary uses of the survey were to inform the development of interview questions and recruit interview participants, and as such, it represents a separate phase of the research in terms of findings. the questionnaire was based on a survey investigating library website use that the principal investigator utilized in a study at a prior institution (au, boyle, & mcdonald, 2009). the survey was created using proprietary university software, distributed through a university email listserv for all undergraduate and graduate students, and remained open for two weeks. 1,182 responses were received, for a response rate of 13.6%. at the conclusion of the survey participants could indicate whether they were interested in taking part in an interview. appendix a contains the full survey questionnaire.   the second method of unobtrusive observations was conducted by research team members by taking notes in a variety of campus library locations on both campuses in half-hour increments, including hallways, book stacks, computer labs, quiet study rooms, and near reference desks. these observations were conducted during different times and days of the week. the notes included what was observed using the doblin group’s aeiou framework as well as the researcher’s interpretation (ethnohub, 2017). a total of 32 hours of observations were completed, and the notes were compiled for future analysis. the research team used the observation data to create interview questions. appendix b contains a sample observation sheet.   the final data collection method was semi-structured in-depth interviews with 20 undergraduate and 10 graduate students, representing different majors and class levels. 15 students from each campus were randomly selected from the pool of survey participants for a total of 30 interview participants. for each interview, one librarian acted as the interviewer and one librarian operated a camera to video record the discussion. sample interview questions are included as appendix c. interview durations ranged between 40 and 60 minutes, and participants were compensated for their time with a $30 gift card for a large online retailer. the audio files were professionally transcribed and made available to the team of librarians performing coding and data analysis.   data analysis   four librarians representing both campuses volunteered to act as data analysts for the study. after survey responses were collected, a word count of the 185 observations and 15 randomly selected interview transcripts served as the basis for developing a codebook, to later guide the coding of observations and interviews. the observations and interviews were coded in teams of two, with one librarian representing each campus. each coder read a given observation sheet or transcript and inserted codes into the text as applicable. after completing a transcript or set of observations, the pairs met to reconcile their coding and agree upon a final version. the teams met periodically as a group to report their progress and compare themes. an interrater agreement of 85% was established between group members and between teams through double-coding 20% (6) of the total number of transcripts.   the initial version of the codebook was devised through a review of prominent keywords from the observation word count, along with an analysis of the 15 random interview transcripts for repeated themes that was conducted in pairs and then as a group of four. a total of 6 iterations of the codebook were devised during the process, and the final codebook contained a total of 459 codes at the question, unit, and thematic levels. sample thematic codes from the final codebook are included as appendix d. the survey data were entered into spss and analyzed using inferential and descriptive statistics, and the interview and observation data were analyzed using descriptive statistics in addition to coding.   limitations   in terms of limitations regarding data analysis, each campus library offers different services and has different spatial configurations, making direct comparisons across campuses difficult. as a data collection method, unobtrusive observations are subject to the observer’s biases, and thus have limited reliability when considered alone. interviews were semi-structured and conducted by multiple interviewers at two campuses, potentially resulting in differences in how the interviews were conducted or the interview questions posed. although the precautions of calculating interrater reliability and working in pairs were taken to limit coder bias, it is also possible for errors to have occurred during the coding process, as coders’ biases could potentially lead them to focus on some findings while unconsciously ignoring others. the qualitative orientation of this study resulted in a long data analysis process, and as such, the data has aged significantly since its collection. because ethnographic research is highly contextual and dependent on many unique variables, the study’s results are not generalizable to other settings. the findings should not automatically be assumed of one’s own library users, or applied directly to one’s services or space without first conducting research into the needs of a student population. however, the methods and data analysis process may be of significant interest and use to other researchers seeking to conduct a study of their own.   results   seven major themes were identified through data analysis, including: 1) student interactions with librarians and contact preferences, 2) access services (such as interlibrary loan and reserves), 3) use of online library sources, 4) use of non-library information sources, 5) use of technology for academic work, 6) use of library space, and, 7) research and study habits. because the full results from this study are not possible to describe within one article, the results at hand will focus upon two themes with potential implications for academic library service and space planning: participant interactions with librarians and contact preferences, and participant use of library space. these two themes were selected in order to provide an illustration of relatively distinct but potentially overlapping areas related to library services and use, reflecting the study’s aims of examining the library holistically while keeping the results to a manageable scope.   student interactions with librarians and contact preferences   the three data collection methods each furnished different perspectives on student interactions with library staff. among interviewees (n=30), 60% had interacted with a librarian for academic purposes one or more times. of these interactions, students reported using or preferring various modes, including text message (27%), research appointment scheduled in advance (13%), ask-a-librarian desk (13%), phone call (13%), and instant message chat (10%). students who interacted with a librarian typically expressed favorable comments, as one sophomore described after a recommendation from her professor prompted a visit to the ask-a-librarian desk: “i didn’t know how to go about finding information. the librarian helped me. she showed me how to do things online, very helpful, a very good experience.” of all the survey respondents (n=1,072) 45% considered contacting a librarian through the libraries’ website as either “extremely important” or “very important.” first-year students and sophomores were most likely to rate contacting a librarian through the website as “extremely important” or “very important,” at 54% and 53% respectively. this importance decreased as levels of study increased, with graduate students least likely to select “extremely important” or “very important.” among the 185 unobtrusive observations, 20 in-person interactions with librarians were recorded.   during interviews, students were asked their contact preferences for library information and whether they would welcome contact from the library on social media. responses varied widely, but email, print (such as signage and flyers), and social media (including facebook and twitter) were discussed most often. more than one-quarter of interview participants (27%) were interested in learning about library services, events, new acquisitions, or general library news by signage and posters. only a handful of students reported regularly checking their university email account, instead relying on a personal email address. “i never checked my liu email until this year,” one junior stated, who did so only “when my professors said i can’t use my personal email but need to strictly use my liu email. i didn’t know about that until this semester.”   although social media was widely used by both undergraduate and graduate students, many interview participants favored social media to interact with friends and family instead of purposes related to academic work, and drew a strong distinction between the two. when asked if they were aware of the library’s facebook page and if they would be interested in “liking” it if they had not done so already, 67% of students were either unaware or uninterested. several students stated they would “like” the page if it was convenient to do so, or if they received some incentive, such as the opportunity to win a prize. as one graduate student described, “it’d have to pop up and be like, ‘like this and be entered to win a contest.’ it has to be convenient and welcoming. i wouldn’t go out of my way to search for the library to become friends.” twitter was used less frequently than facebook by interviewees, with 30% using the website in some manner and 13% of twitter users uninterested in following a library twitter account. using the platform to keep up to date with personal or professional interests were the most common reasons for not wanting to follow a library account.   student use of library space   concerning student use of library space, individual study was a prominent theme across the data. by moving furniture, occupying nearby chairs and table space with their belongings, stacking books around themselves, and other means, students would “cocoon” themselves to focus their attention on studying or signal that they did not want to be disturbed. this practice was described in 12 observations and 57% of interviews. one graduate student related the importance of having everything spread out and “just so” in order to concentrate: “i make myself at home when i put myself down [to study]. my laptop here, my water here, so that everything’s there, out in the open.” studying as part of a group was another recurrent theme, with 7 observations and 30% of interviewees describing meeting with classmates to study for a test, prepare for a presentation, conduct research, or other academic tasks.   librarians at both campuses had observed that students frequently left their laptops, phones, or other personal items unattended. to determine why this happened, interviewees were asked whether they had left personal items unattended for any length of time in the library. more than half of the interviewees (57%) stated that they had, whether to use a restroom, leave a quiet space to take a phone call, or purchase a snack, and 14 separate observations confirmed the interview data. no participants reported having items stolen, and they left their belongings because they felt the library was a “safe place” where theft was not an issue, or they assumed the library had video surveillance cameras that would record a theft.   along with creating individual study spaces, studying in groups, and leaving items unattended, another prominent theme included student preferences in seating. students were generally open to different types of seating arrangements and furniture types, and their preferences largely depended on the activity they engaged in. tables or study carrels were equally preferred by 80% of interviewees, followed by any type of seating as long as outlets were located nearby (33%), and soft seating such as couches (20%). unobtrusive observations confirmed the popularity of individual study carrels, with 30 students noted at carrels, 16 at tables, and 12 at soft seating locations. finally, the problem of unwanted noise was a prevalent theme, with 10 observations and 60% of interviewees referencing excessive noise in the library. students addressed this issue in various ways, including using earplugs while studying, wearing earphones but not playing music, and one senior who took substantial measures, stating, “i use those big headphones that cancel out the noise.”   discussion   the following discussion addresses the two primary areas of interest described in the findings: student interactions with librarians and contact preferences, and student use of library space. the findings will be further contextualized and compared to the evidence presented in related research studies.   student interactions with librarians and contact preferences   comparing interactions with a librarian to students’ level of study, interactions appeared to be less important to students as they progressed in their studies. interviews suggested this trend was due to three factors among upper-level undergraduates and graduate students: a) an increased confidence in conducting work independently, b) an increased familiarity with academic information resources, and, c) a greater reliance on professors for assistance due to having developed closer relationships. contacting a librarian by text message was rated or described more favorably than any other online or in-person method, which contrasts with carey and prathak’s (2017) study based on in-person questionnaires that found nearly 75% of respondents prefer face-to-face reference. rather than seek help with in-depth research strategies or their academic work not related to the library, students of all levels and academic subjects tended to seek librarian assistance for primarily library-specific tasks, such as to locate information resources provided through the library, including books or journal articles, or to find an answer to a directional question. one student’s remarks, echoed by a number of other interviewees, revealed that their only interactions with a librarian were “when i couldn’t find a book on a shelf or when i get lost and i can’t find the room i’m supposed to go to.”   findings concerning the type of assistance sought from librarians are in agreement with the evidence from a large set of interviews conducted as part of the erial project, involving 91 undergraduates and 45 teaching faculty. miller and murillo (2012) reported that undergraduates typically seek directional or library-specific assistance from librarians, and identified a lack of relationships or connections with librarians as resulting in students’ frequent consultation of instructors, peers, or family instead. considering pellegrino’s (2012) survey findings that telling students to ask for help from librarians is effective only when teaching faculty are telling them, it is apparent that more effectively communicating librarian roles as well as closer collaboration with faculty are important efforts to undertake. although few participants described library instruction sessions as a place of interaction with librarians, it remains a site of significant contact with students. more so than many other roles librarians have, library instruction has a great deal of opportunity for collaboration with both faculty and students through assignment design, pedagogical collaboration, and research assistance.   student contact preferences concerning the library, including contact initiated by the student and by the library, included strong opinions on keeping certain social media platforms for certain purposes. many participants used facebook for non-academic personal activities, and expressed a general lack of interest in library news on other platforms such as twitter. this response to using social media for receiving library information, which ranged from rejection to indifference to tepid interest, was surprising considering an extensive body of publications suggests ways to adopt social media for library marketing and outreach. of the studies that account for students’ reception to library social media, some find receptivity to facebook as a marketing tool (connell, 2009; sachs, eckel, & langan, 2011), while other research casts doubt on student interest in social media for academic purposes, and asks that librarians consider questions of patron privacy and the mining of personal data (bodnar & doshi, 2011; epperson & leffler, 2009). due to students’ reported lack of interest as well as the rapid rate at which the popularity of social media platforms waxes and wanes, the findings may warrant caution and consideration of sustainability before devoting significant time and resources to social media outreach (gaha & hall, 2015).   printed signage and posters were not mentioned in interview questions, yet were brought up favorably as a means of learning about library news, services, or collections. for students who were on-campus or at the library with some regularity, they wished to be notified of the same information they might otherwise be online. in light of the conflicting information received from interview participants, who preferred to contact librarians through text and online chat, compared to upper-division survey respondents, who rated the importance of contacting a librarian through the website as less significant, demographic factors and various channels of communication, such as posters, text updates, and email, should all be considered when appraising the contact preferences of one’s student population.   student use of library space   the findings noted that “cocooning” (defined by locating a preferred study space and making it into one’s temporary “home” for studying) was relatively common, practiced by more than half of the interviewees. these interviewees created their own space through moving library furniture, stacking books, or otherwise blocking off a space of their own to focus or seek privacy. these behaviours were observed in various areas of the libraries. many students sought proximity to certain areas, such as natural light, away from distractions, or in areas where groups can work comfortably. in particular, students frequently sought out the limited electrical power sources to charge their devices, even waiting their turn to sit near outlets. while some students did not move furniture or create their own space, it was clear that many valued the ability to form a space of their own, or to have the flexibility to do so. modular furniture that can be configured for group or individual study, as well as study areas that create or accent a pleasant environment, could benefit students in this manner.   students leaving personal items unattended was perceived by librarians to be a problem in terms of potential theft. the findings confirmed this to be a common practice, as 20 out of 30 interviewees indicated they leave behind items to do other tasks. this was particularly common among students who lived on campus, who likely feel they are in a familiar or friendly environment. creating more awareness of the risk involved in leaving one’s items behind for any amount of time, through signage or other means, would be one way to potentially reduce this problem.   noise within the libraries was mentioned across interviews, observations, and survey responses, and in all cases was something students wanted to change about the library. this dislike of noise was shared across student academic levels and disciplines. reported use of the libraries’ rooms designated for quiet study was not as prominent. only eight interviewees used the quiet rooms, although observations indicated that these rooms are filled to capacity during busy times of the semester. some students, including seniors, were unaware the libraries had quiet rooms. this underscores the necessity of communicating the different purposes of library space to students through formal and informal cues, particularly considering the implementation of a noise-monitoring device at one academic library had no impact upon noise levels (lange, miller-nesbitt, & severson, 2016).   in general, students expressed the need for more comfortable or functional spaces and extended hours. many interviewees wanted the library to be open earlier, later, or 24 hours, due to personal, work, and academic obligations that made it difficult to visit. observations indicated students using the library until closing and waiting for the library to open, particularly during limited weekend hours. other items discussed by students as key to improving the library were to increase the number of electrical outlets, to improve the wi-fi signal throughout the entire building, and to offer wireless and free printing. hall and kapa (2015) found similar requests from library users for larger table space, additional comfortable furniture, and an increased number of desktop computers. other recent qualitative and mixed-methods assessments of library space use have come to similar conclusions regarding the need for access to electrical power, sufficient group study space, and flexible seating (asher, 2017; dominguez, 2016; mccrary, 2017). these basic features that tend to be overlooked can very much determine the quality of students’ library experiences.   conclusion   drawing upon data from in-depth semi-structured interviews, unobtrusive observations, and an online questionnaire, this project explored undergraduate and graduate students’ library and research experiences at two campuses. after an extensive coding process, a thematic analysis uncovered a number of findings relating to library services, space, and student research habits. future research could pursue a number of different directions, including focusing upon or incorporating additional ethnographic methods, such as participant-driven photo elicitation methods (bedi & webb, 2017), or cognitive mapping exercises. collecting data longitudinally, whether over several years or at intervals of two or more years, would contribute an important dimension to the study of research habits and library use.   the authors’ libraries have undertaken several actions based on the findings, from instructional efforts to the redesign of library space. to increase student contact with librarians across all types of communication, librarians made efforts in the area of library instruction so that students would be comfortable approaching librarians for help later on. to help accomplish this goal, first-year students and an information literacy module were integrated within the curriculum at one campus. additionally, librarians have become increasingly involved in learning communities, which are interdisciplinary courses that span a student’s first academic year, to communicate more directly with students early on. in terms of digital initiatives, the library implemented a library app for mobile devices, “book a librarian” research consultations conducted via skype, and redesigned the library website. participants noted overwhelmingly that comfortable furniture and pleasant spaces to study were a priority, and both libraries have undergone renovations that include soft seating, additional group study tables, and natural light. concerning long term goals of this study, such as making ethnographic research on students’ library use and academic practices a continual process and an ingrained part of the culture among long island university library staff, the outcomes have yet to be seen. for the time being, the results have been disseminated among members of the university, and they will continue to inform decision making while another research project is developed and conducted.   while ethnographic research demands a substantial contribution of time, exercising patience, and potentially learning new data collection and analysis skills, the benefits lie in developing a detailed and contextualized understanding of one’s topic. while there are many methods of conducting research to better understand and assist library users, lanclos and asher (2016) compellingly argue that “ethnography can serve as an effective antidote for the problematic reliance in higher education (including libraries) on analytics and quantitative measures of effectiveness.” as academic libraries continue to seek ways to meet the needs of their campuses, ethnographic research holds potential for doing so in a way that accounts for the complexity of libraries, learning, and people’s lives.   acknowledgements   we sincerely thank the large group of library staff, graduate student assistants, and librarians at long island university libraries who participated in conducting this study, whether through attending meetings, brainstorming ideas, collecting data, or making the data ready for analysis.   references   allan, e. g. (2016). ethnographic perspectives on student-centeredness in academic libraries. college & undergraduate libraries, 23(2), 111-129. http://doi.org/10.1080/10691316.2014.965374   asher, a. (2017). space use in the commons: evaluating a flexible library environment. evidence based library and information practice, 12(2), 68-89. http://doi.org/10.18438/b8m659   au, k., boyle, j., & mcdonald, s. (2009). studying students to enhance library services at rutgers university: an ethnographic research project. retrieved from https://www.libraries.rutgers.edu/rul/staff/admin/state_of_the_libraries/state_of_the_libraries_09_erp.pdf   bedi, s., & webb, j. (2017). through the students’ lens: photographic methods for research in library spaces. evidence based library and information practice, 12(2), 15-35. http://doi.org/10.18438/b8fh33   bodnar, j., & doshi, a. (2011). asking the right questions: a critique of facebook, social media, and libraries. public services quarterly, 7(3-4), 102-110. http://doi.org/10.1080/15228959.2011.623594   carey, j., & pathak, a. (2017). reference mode preferences of community college (two-year) and four-year college students: a comparison study. evidence based library and information practice, 12(1), 50-71. http://doi.org/10.18438/b8vw40   connell, r. s. (2009). academic libraries, facebook and myspace, and student outreach: a survey of student opinion. portal: libraries and the academy, 9(1), 25-36. http://doi.org/10.1353/pla.0.0036   delcore, h. d., mullooly, j., & scroggins, m. (2009). the library study at fresno state. retrieved from institute of public anthropology, csu fresno website: http://fresnostate.edu/socialsciences/anthropology/documents/ipa/thelibrarystudy(delcoremulloolyscroggins).pdf   dent goodman, v. (2011). applying ethnographic research methods in library and information settings. libri, 61(1), 1-11. https://doi.org/10.1515/libr.2011.001   dominguez, g. (2016). beyond gate counts: seating studies and observations to assess library space usage. new library world, 117(5-6), 321-328. http://doi.org/10.1108/nlw-08-2015-0058   duke, l. m., & asher, a. d. (2012). college libraries and student culture: what we now know. chicago, il: american library association.   dunne, s. (2016). how do they research? an ethnographic study of final year undergraduate research behaviour in an irish university. new review of academic librarianship, 22(4), 410-429. http://doi.org/10.1080/13614533.2016.1168747   epperson, a., & leffler, j. j. (2009). social software programs: student preferences of librarian use. new library world, 110(7-8), 366-372. http://doi.org/10.1108/03074800910975188   ethnohub. (2017). aeiou framework. retrieved from https://help.ethnohub.com/guide/aeiouframework   foster, n. f., & gibbons, s. l. (2007). studying students: the undergraduate research project at the university of rochester. chicago, il: association of college & research libraries.   gabridge, t., gaskell, m., & stout, a. (2008). information seeking through students’ eyes: the mit photo diary study. college & research libraries, 69(6), 510-522. http://doi.org/10.5860/crl.69.6.510   gaha, u., & hall, s. (2015). sustainable use of social media in libraries. codex: the journal of the louisiana chapter of the acrl, 3(2), 47-67. retrieved from http://journal.acrlla.org/index.php/codex/article/view/106   hall, k., & kapa, d. (2015). silent and independent: student use of academic library study space. partnership: the canadian journal of library and information practice and research, 10(1), 1-38. http://doi.org/10.21083/partnership.v10i1.3338   holder, s., & lange, j. (2014). looking and listening: a mixed-methods study of space use and user satisfaction. evidence based library and information practice, 9(3), 4-27. http://dx.doi.org/10.18438/b8303t   khoo, m., rozaklis, l., & hall, c. (2012). a survey of the use of ethnographic methods in the study of libraries and library users. library & information science research, 34(2), 82-91. http://doi.org/10.1016/j.lisr.2011.07.010   khoo, m. j., rozaklis, l., hall, c., & kusunoki, d. (2016). “a really nice spot”: evaluating place, space, and technology in academic libraries. college & research libraries, 77(1), 51-70. http://doi.org/10.5860/crl.77.1.51   kinsley, k. m., schoonover, d., & spitler, j. (2016). gopro as an ethnographic tool: a wayfinding study in an academic library. journal of access services, 13(1), 7-23. http://doi.org/10.1080/15367967.2016.1154465   lanclos, d., & asher, a. d. (2016). “ethnographish”: the state of ethnography in libraries. weave: journal of library user experience, 1(5). http://doi.org/10.3998/weave.12535642.0001.503   lange, j., miller-nesbitt, a., & severson, s. (2016). reducing noise in the academic library: the effectiveness of installing noise meters. library hi tech, 34(1), 45-63. http://doi.org/10.1108/lht-04-2015-0034   mccrary, q. (2017). small library research: using qualitative and user-oriented research to transform a traditional library into an information commons. evidence based library and information practice, 12(1), 34-49. http://doi.org/10.18438/b8863f   miller, s., & murillo, n. (2012). why don’t students ask librarians for help? undergraduate help-seeking behavior in three academic libraries. in l. m. duke & a. d. asher (eds.), college libraries and student culture: what we now know (pp. 49-70). chicago: american library association.   pashia, a., & critten, j. (2015). ethnography as pedagogy in library orientations. journal of information literacy, 9(2), 84-93. http://doi.org/10.11645/9.2.2028   pellegrino, c. (2012). does telling them to ask for help work? investigating library help-seeking behaviors in college undergraduates. reference & user services quarterly, 51(3), 272-277. http://doi.org/10.5860/rusq.51n3.272   ramsden, b. (2016). ethnographic methods in academic libraries: a review. new review of academic librarianship, 22(4), 355-369. http://doi.org/10.1080/13614533.2016.1231696   regalado, m., & smale, m. (2015). “i am more productive in the library because it’s quiet”: commuter students in the college library. college & research libraries, 76(7), 899-913. http://doi.org/10.5860/crl.76.7.899   sachs, d. e., eckel, e. j., & langan, k. a. (2011). striking a balance: effective use of facebook in an academic library. internet reference services quarterly, 16(1-2), 35-54. http://doi.org/10.1080/10875301.2011.572457   smale, m., & regalado, m. (2017). digital technology as affordance and barrier in higher education. cham, switzerland: springer.     appendix a: survey questionnaire   q1. what discipline are you affiliated with at long island university?           social sciences science technology engineering medicine arts and humanities business education   q2. what is your home campus at long island university?                [campus 1] [campus 2] [campus 3] [campus 4] [campus 5] [campus 6]   q3. what is your current status at long island university?                 first year student sophomore junior senior graduate student                q5. do you live on or off campus?                 on campus off campus   q6. as of december 31, 2012, how many semesters have you been at long island university? (count fall, spring, and summer terms)      1-2 semesters 3-4 semesters 5-6 semesters 7-8 semesters 9-10 semesters 11-12 semesters more than 12 semesters   q7. have you had any library instruction while at long island university?                  yes no   q8. how many research-based papers, articles, presentations, or projects did you produce in this past school year (including high school if you are a first year student)?                0 1 2-4 5-10 more than 10   q9. during the last academic year, how often was the libraries' website a basic part of your research process?               always usually sometimes seldom   q10. how often do you use the libraries' website in a week?              1-4 times 5-9 times 10-14 times 15 times or more never   q11. how often did you use the libraries' website last week?             1-4 times 5-9 times 10-14 times 15 times or more never   q12. how do you usually access the libraries’ website?   bookmark            search for it using a search engine such as google or yahoo!            type in the url or web address                  link from blackboard      link from another long island university page link from google scholar              do not access the long island university libraries' website                              other (please specify)        q13. which of these resources do you us most often?   interlibrary loan               google or another search engine  google scholar   libguides/subject research guides           long island university libraries' website                liucat  (the library catalog)           online databases (e.g. academic search premier and jstor)          wikipedia            worldcat             not applicable                   other (please specify)        q14-q24. below are several activities that you can engage in using the libraries' website. how important are each of these activities to you? (each q below is rated using following scale: extremely important; very important; moderately important; not very important; not at all important; no basis to judge).   q14. finding books q15. finding articles or journals q16. requesting books or articles from another library q17. contacting a librarian q18. finding course reserves materials q19. consulting libguides/subject research guides q21. looking up library hours, directions, and/or phone numbers q22. accessing your library account q23. reading library news or finding library events q24. finding media such as dvds and videos        q25. when you use the libraries' website, do you tend to . . . ?            know exactly where things are figure out where things are by browsing get confused and ask for help get confused and give up   q26. in general, how satisfied are you with the libraries' website?   very satisfied moderately satisfied neither satisfied nor dissatisfied moderately dissatisfied very dissatisfied   q27. do you ever access the libraries' web site using your web-enabled cell phone?                  yes no   q28. what information were you seeking on the libraries' website through your cell phone? (check all that apply)   hours    databases            library catalog   contact information         my library account           directions            other (please specify)        q29. would you use text messaging to get an answer to a reference or research question from the long island university libraries?               yes no other (please specify)   q30. are you aware that the library has a new general information app available through the libraries' web site?       yes no other (please specify)   q31. which devices do you use to regularly browse the web?   smartphone        ipad       laptop  desktop                other tablet device (samsung galaxy, etc.)               other (please specify)        q32. which web browsers do you regularly use?   firefox   internet explorer               safari     chrome                other (please specify)        q33-q37. on average, how many hours do you spend on the web (using a web browser) each day for each of the following activities? (each q below is answered using one of the following responses: under 1 hour, 1-5 hours; 6-10 hours; 11 hours or more; none).   q33. research for school q34. other research q35. reading news          q36. online gaming q37. social networking (i.e., facebook, twitter)   q38. which social networking sites do you use most frequently?   twitter  facebook              tumblr google+                none                     other (please specify)        q39. please respond to the following: do you have a desktop at home?           yes no   q40. please respond to the following: do you own a laptop?               yes no   q41. please respond to the following: do you have access to a computer at work?      yes no   q42. please respond to the following: do you have a tablet computer?            yes no   q43. please respond to the following: do you own a mobile/cell phone?         yes no   q44. which devices do you currently own?   iphone  android               blackberry           cell phone without internet access             other (please specify)        q45. do you ever use your cell phone to do the following?   send or receive e-mail      send or receive text messages       take a picture     play music          play games          watch videos      record a video   access the internet            keep a calendar do a research paper or assignment             catch-up on assigned readings for class   take notes           use apps             video conference (i.e., skype, facetime)    other (please specify)        q46. where do you access the internet the majority of the time?                         laptop at home laptop at school laptop at work desktop at home desktop at school other (please specify)   q47. how satisfied are you with the wifi connection on your campus?           very satisfied moderately satisfied neither satisfied nor dissatisfied moderately dissatisfied very dissatisfied   q48. where are you most often when you use your web-enabled cell phone?                home school work out socializing with friends commuting (i.e., car, bus, train) other (please specify)   q49. have you used or do you use your ipad in class for class-related work?                yes no not applicable   q50. if you have apps on your mobile device or other device (such as an ipad), which apps do you use most often?   q51. do you have any ideas for tools that you would like to see the library develop for mobile devices? please describe them in the space provided and explain why:   q52. do you have any other comments, questions, or concerns?   q53. may we contact you to participate in a voluntary in-person interview? (this interview will take approximately 60 minutes. if you are selected, for your participation, you will receive a $30 gift card.)   q54. would you like to enter yourself into a drawing for a chance to win a macbook air, tickets to barclays center events, or a gift card? please note that your survey responses will be stored in a database separate from your personal information for the drawing.     appendix b: sample observation recording sheet   location:              periodicals reading room lower level hallway lower level date:                      02/25/2013 time started:       10am time ended:        10:30am   a activities are goal directed sets of actions-things which people want to accomplish e environments include the entire arena where activities take place i interactions are between a person and someone or something else, and are the building blocks of activities o objects are building blocks of the environment, key elements sometimes put to complex or unintended uses, changing their function, meaning and context u users are the consumers, the people providing the behaviors, preferences and needs   what i saw/raw data (a, e, i, o, u/spradley) what i thought/interpretation the hallway area was empty during the entire time of observation (except for the normal walking-through traffic). there were two groups of students in the periodical reading room area. one group consisted of three students. they were sitting at the large table by the windows. students had ipads, laptops, smartphones, food, and water on the table. they also talked in full voice. the second group was consisted of two students sitting at the table close to the wall by the technical services area. they had food, water, and laptops on the table. there was very little interaction between those two students. they were reading and using laptops. at one point, one of the two students got up and left the area with her iphone in hand. previously she was trying to make a phone call and could not get a reception. besides those two groups one student was sitting by himself at the empty computer carrel and was reading. the student sitting by himself (reading) was there long before the observation began. i saw him at 8am in the morning on exactly the same spot.                   appendix c: sample in-depth interview questions   ●        when you study in the library (if you do), do you prefer to be around other students, or have more of your own personal space? ○         if you prefer to have more of your own space, where do you go in the library? ○         do you ever have to “create” your own space?   ●        when you study, do you have more than one electronic device in use? ○         do you ever listen to audio such as music, tutorials, etc. on headphones while you are studying? ○         if you do, can you describe what you typically listen to?   ●        do you come to the library when you are on campus? ○         if yes, do you tend to come to the library alone or with friends and classmates? ○         if you come to the library alone or as a group, what are some of your typical activities?   ●        are you interested in receiving information about the library’s services and programs via social media? ○         for instance, would you “like” the library on facebook or follow us on twitter?   ●         if you use the library to study, do you bring a laptop with you? ○                     where in the library to you tend to study? ○                     do you use different areas of the library at different times?   ●        do you seek help from library personnel? ○                     if yes, please describe. ○                     if not, when you have questions regarding your assignments or research, where do you turn for assistance?   ●        have you ever used the libraries’ website to help you with an assignment? ○                     if you did, how did you find the libraries’ website/homepage? ○                     can you show me how you used the website and how you found your way to the things you used?   ●        do you access the library from home? ○                     if you do, can you give me an example of what you did or what you were looking for? ○                     did you ever need help connecting to the library from off-campus? how often do you access the library’s website and for how long?     appendix d: sample thematic codes from codebook   major theme code subtheme code student research strategies resstrat first source consulted firstsrc   resstrat search engine use searcheng   resstrat evaluating sources evalsrc   resstrat shelf browsing shelfbrw   resstrat catalogue browsing catbrw student use of library libsvuse librarian/staff interaction libint   libsvuse chat message chat   libsvuse remote access remacc   libsvuse reserves reserves   libsvuse citation help cithelp student use of library space libspa stacks stacks   libspa leave personal items leavepit   libspa food food   libspa no available computer stations nocomp   libspa move furniture movefurn   libspa social gathering space socspa student study habits studyhab time spent in library timeinlib2   studyhab print print   studyhab saving save   studyhab playing music playmus   research article   e-preferred approval books at the university of manitoba: a comparison of print and ebook usage   jan c horner senior scholar, collections management university of manitoba libraries winnipeg, manitoba, canada email: j.horner@umanitoba.ca   received: 23 aug. 2016  accepted: 13 feb. 2017      2017 horner. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to compare the usage of print and ebooks received on university of manitoba’s e-preferred ybp approval plan as well as to examine cost per use for the approval print books and ebooks. methods – usage data was compiled for books received on approval in 2012/2013 to december 31, 2014. counter reports were used to determine use and non-use of ebooks, while vendor reports from ebl and ebrary were used for the cost per use analysis. print usage information was drawn from sirsi and then alma when uml switched systems at the beginning of 2014. results – ebooks received more use than p-books overall, but when examined by subject discipline, significant differences could not be found for the “stm” and “other” categories. with ebooks, university press books tended to be used more than those from other publishers, but the same result was not found for print books. ebrary ebooks tended to be used more often than ebl, ebsco, and wiley ebooks, and single-licence books tended to be somewhat more used than multi-user ones. cost-per-use data was much lower for print books, though the comparison did not look at staffing costs for each medium. conclusions – this study finds that of approval books matching the same profile, ebooks are used more, but print books receive more substantial use. both formats are needed in a library’s collection. future comparisons of cost per use should take into account hidden labour costs associated with each medium. usage studies provide evidence for librarians refining approval plan profiles and for budget managers considering changes to monographic acquisition methods and allocations.   introduction   libraries are changing the way they acquire monographs. demand-driven and publisher front-list acquisition options are competing with traditional methods of firm ordering and approval plan delivery.1 approval plans are intended to save staff time and can operate despite staff absences or changes. but libraries are looking more critically at approval plans since purchasing a publisher’s front list or using demand-driven acquisition can also save staff time.    this study looks at the performance of the ybp e-preferred approval plan at the university of manitoba libraries (uml). uml is an arl and carl member with an enrollment of approximately 30,000 and faculties of agricultural & food sciences, architecture, arts, engineering, health sciences, law, management, music, and sciences. it is the major research library for the province.   uml is facing budget challenges similar to those at other arl libraries where monograph budget funds have decreased on average 4% over the last three years reported. uml’s monograph budget has decreased 8.5%.2 during the same period the financial commitment to demand-driven ebook acquisitions has remained steady and uml has purchased the springer ebook collection and subscribed to other ebook collections (e.g. ebsco, knovel). budget challenges were thus a strong motive to review the performance of uml’s approval plan.   literature review   this study is quantitative; therefore, qualitative studies of ebook versus print use, although of interest and value, were not included in the review.   ebook v print comparisons   most quantitative studies of ebook and print monograph usage have examined the same titles in both formats.  in addition, most of the early studies (e.g., littman & connaway, 2004) looked at netlibrary titles as a basis of comparison with print. after 16 months of use the littman & connaway study at duke university found 40% of the ebook versions had been used and 36% of the print (p. 259).   kimball, ives, and jackson, in their study (2009), identified 4,288 netlibrary books in the sciences that were also available in print. although the ebooks were used more times per book, 24% of the ebooks were accessed while 23% of the print versions circulated (p. 23-24).   more recently levine-clark & brown (2013) compared the use of duke university press titles online and in print published between 2009 and 2012. a greater number of print titles had been acquired before the ebook version, but there were 1,150 titles held in both formats. of those titles, 54% of the print circulated and 39% of the ebooks had been used, leading the authors to conclude that when both formats were available, users preferred the print (slide 25). goodwin (2014) looked at the 2011 collection from duke university press, 285 ebooks and 275 print (10 matching print titles had not been received by the study start). usage was counted to october 2013 (two years, five months) and resulted in 73% of the ebooks being used and only 29% of the print, although only 12% of the ebooks received “substantive use.”   a kent state study (downey et al., 2014) did not compare the same titles in print and ebook. rather it looked at the performance of 20,000 of the most recently acquired print books (up to january 2012) added between july 2009 and january 1, 2012 in relation to 20,018 discovery records which were loaded in their catalogue for a patron-driven acquisition (pda) project that ran january to december 2012 using ebrary ebooks. by the end of 2012, 8% of the books in the pda discovery pool had been used (p. 148) with only 2% triggering a purchase (p. 154), but a relatively high number (62.5 %) (p. 149) of the purchased print books had circulated.   ebook usage   levine-clark (2014), working with major ebook vendors ebl and ebrary, examined data for about 625,000 ebooks across roughly 800 academic libraries. this study could not be used to predict rates of use in a single library, since the titles had a massive pool of potential readers. it found that usage of social science ebooks was higher than that for humanities and science, technology & medicine (stm) titles in terms of percentage of titles used and average amount of use (slides 31-36). patrons spent more time in humanities ebooks per online session than for ebooks in the other disciplines. stm ebooks involved more downloading. ebl and ebrary titles were examined separately. in general humanities ebooks were used more than stm titles on ebrary, and stm titles were used more than humanities ones on ebl (slides 31-32).   comparisons of ebook types of acquisition   many authors have published reports on patron-driven or demand-driven acquisition (pda, dda) programs at academic libraries. however, because pdas/ddas count usage of never purchased titles as well, they are not relevant to the present study of titles acquired by approval in print and electronic format.   carrico et al. (2015, pp. 106-107) compared the usage of ebooks acquired in large publisher packages with those that were firm-ordered by librarians. in terms of the packages, 50% were used compared to 52% of the firm-ordered titles. this figure was somewhat skewed by the use of the medical titles in which 63% of the package titles and 84% of the firm-ordered titles were used. for the humanities/social sciences, package titles and firm orders were used 47% and 45% respectively. for science & technology package titles and firm orders were used 49% and 57% respectively.   print usage and substantive use of ebooks   a number of studies have shown that the percentage of print collections that circulate is low. rose-wiles (2013) examined the circulation for 2005 to 2009 of the entire collection of 443,577 print books at seton hall university and found 21.5% had circulated (p. 137). however, only 17.7% had been published in the 2000s. she examined the subset of science books published since 2000 and found 34.5% circulated 2005 to 2009 (p. 141).  a cornell university study (2010, p. 2) looked at the circulation of its collection published between 1990 and 2010 (1.6 million titles) and found that 45% had circulated.   a 2009 study (alan et al.) looked at the usage of print titles acquired through approval plans, examining books received at university of illinois at urbana-champaign (uiuc) and pennsylvania state university (penn state) july 2004 to june 2005 where usage was gathered from july 2004 to march 2007. it found that 69% of penn state’s and 60% of uiuc’s approval books had circulated at least once (p. 70). it was suggested that the higher percentage of use at penn state was based on the larger user population, 98,000 versus 45,000 at uiuc. no breakdown of use/non-use was provided by subject.   most studies and guidelines (niso, 2014, p.34) have exercised caution in drawing conclusions about the higher number of uses/transactions/sessions of ebooks in comparison with print usage. many libraries do not capture in-house use of their print collections, and some ebook accesses may be equivalent to the act of taking a book off the shelf, glancing through it, and then replacing it.   some authors have looked at measures of substantive ebook use as a means of comparison with print circulation. rose-wiles used viewing of 10+ pages in an ebook as equivalent to a print check-out (p. 146). goodwin used viewing 11+ pages as a measure of substantive use (p. 103). ahmad & brogan (2012, p. 198) in their case study of ebl ebooks usage, set the standard of more than 9 minutes spent in a book as an indication of reading, since the ebl platform uses that time to differentiate reading from browsing. in their study 29.12% of transactions were spent in reading under this definition and 70.88% in browsing.   none of the studies mentioned downloading as an equivalent to print check-outs. however, download could provide a more equivalent measure of comparison to the print check-out. some effort is required to download the book. though shorter than most print loan periods, a patron can use the ebook offline for one to three days. with both ebook downloads and print check-outs, there is no way of knowing how much a book is used while ‘out’ or in ‘offline’ mode.   aims   this study assumes that a comparison of print book (p-book) and ebook usage is possible, especially when substantive use of ebooks is compared with print checkouts.   given that the ybp approval plan will deliver a greater proportion of ebooks over time, this analysis attempts to answer these questions:   are approval ebooks being used at an equal or higher rate than print approval books? is the usage or non-usage of approval ebooks and print books affected by their broad subject discipline? is the usage/non-usage the same or different for the two media? are university press books used more than non-university press books, and is usage the same for print and ebook? did the vendor or type of license affect the usage of ebooks? what is the cost-per-use of an approval ebook and the cost-per checkout for a print approval book?   method   most comparisons of the performance of p-books and ebooks have been done comparing the same titles in both formats. unless an analysis looks at the same titles purchased at the same time in the two formats, it is challenging to compare performance over a similar time period. differing time periods for access may have affected the results of earlier studies (levine-clark & brown, 2013, slide 3; downey et al., 2014, p.145).   although the titles are different in the uml comparison of approval books, approval books, no matter what format, are purchased over the same time period, match the same subject profile, and come from a similar set of publishers with similar publication dates. there is a wide variety of ebook platforms on the market, but the uml ebooks come on four platforms prescribed by the approval profiles: ebl, ebrary, ebsco, and wiley.   in 2012/2013, all subject profiles within the ybp approval plan were converted to e-preferred with the exception of pharmacy. the e-preferred mode meant that if an ebook were published within eight weeks of the print version, an ebook would be received by uml.  in 2012/13, 35% of the approval receipts were ebooks. usage of the approval books received in 2012/13 was counted to december 31, 2014, so both pand ebooks were available for use between 20 and 32 months.     table 1 overall comparisons of printand ebook usage   no. of titles no. used % used print books 5,237 2,003 38% ebooks 1,855 872 47% p-value two-sided proportion of usage different between p& ebooks 0.0011 p-value one-sided proportion of usage greater for ebooks 0.00056 note: p = < 0.05         source of data   lists of ebooks and p-books received on approval were generated from sirsi’s director’s station. some call number and publisher data was retrieved from ybp’s gobi database.   while the counter book report 2 was only available from ebrary, only the counter book report 1 was produced for ebsco, wiley, and ebl ebooks. the counter book report 2 records the number of uses of a section or sections of a book by month and title, whereas the counter book report 1 counts only the uses of a title by month, without considering the use of chapters within it. the counter 2 book report will therefore yield higher uses per title since it counts every section or chapter of a single title as a separate use, whether they were viewed in separate sessions or not. because of this discrepancy, this study does not attempt to measure rates of use based on counter reports, only use and non-use. more detailed information about use was obtained from vendor reports for ebl and ebrary ebooks for cost per use analysis; ebsco and wiley did not provide detailed information for their ebooks so were not included. cost per use was calculated per book and then averaged, rather than the collection cost being divided by overall use as in other studies (bucknell, 2010, p. 133; bucknell, 2012, p. 53).   for p-books, circulation data was derived from sirsi to the end of 2013, and then when uml changed systems, from alma for 2014. uml does not have consistent data on in-house use of its print collections. therefore usage of print books was based simply on number of check-outs.   2012/2013 approval results   in this fiscal year uml received 5,237 p-books and 1,855 ebooks on approval. of those, 38% of the p-books (2,003) circulated and 47% of the ebooks (872) were used. a pearson’s chi-square test for comparing two proportions showed that there was sufficient statistical evidence to conclude the usage of print and ebooks was different and that ebook use was greater than print.   a further analysis was done by broad subject discipline based on the following lc call number ranges, using the same subject breakdown as levine-clark (2014, slide 30).   ·         arts & humanities: b-bd, bh-bx, c-f, m-p ·         social sciences: bf, h-l, u-v ·         science, technology & medicine (stm): q-t   use of ebooks by broad discipline   this analysis indicates that usage of ebooks in the broad disciplines is remarkably similar.   table 2 ebook usage by broad subject discipline   no. ebooks no. used % used proportion of total ebooks proportion of total uses arts & humanities 635 300 47% 34.2% 34.4% social sciences 669 306 46% 36% 35% science, technology & medicine (stm) 494 236 48% 26.6% 27.2% other 57 30 53% 3.1% 3.4% total 1,855 872 47%         use of p-books by broad discipline   uml received many more “arts & humanities” print books through the approval plan, but their usage was lower, compared to the size of the collection, while use of “stm” print books appeared to be higher relative to the size of its collection.   the results of the chi-square tests (table 4 below) provided sufficient statistical evidence to find that usage of print and ebooks was different and higher for ebooks in the “arts & humanities” and “social sciences” categories. however, there was not statistical evidence to claim the same for the “stm” and “other” categories.     table 3 print book usage by broad subject discipline   no. p-books no. used % used proportion of total p-books proportion of total uses arts & humanities 2,263 807 36% 43% 40% social sciences 1,740 657 38% 33% 33% science, technology & medicine (stm) 1,063 469 44% 21% 24% other 171 70 41% 3% 3% total 5,237 2003 38%         figure 1 comparison of print and ebook usage by broad subject.     table 4 pand ebook usage comparison chi-square test subject proportion of usage ebook proportion of usage p-book p-value two-sided p-value one-sided decision arts & humanities 0.472 0.357 <0.0001 <0.0001 significant difference social sciences 0.457 0.379 0.0004 0.0002 significant difference stm 0.478 0.439 0.1474 0.07372 insufficient stat. evidence other 0.526 0.41 0.1262 0.06309 insufficient stat. evidence note: p = < 0.05     ebooks use by type of publisher   for ebooks, usage of books published by university presses was higher, especially considering use relative to the collection size.   table 5 ebook usage by type of publisher   no. of ebooks no. used % used proportion of total ebooks proportion of total uses university presses 878 449 51% 47% 51% other publishers 977 423 43% 53% 49% total 1,855 872 47%     p-value two-sided     0.0007 proportion used is different between university presses & other publishers p-value one-sided     0.0004 proportion used is greater for university presses than other publishers note: p = < 0.05             p-books use by type of publisher   in comparison with ebooks, usage of print books from university presses did not appear different from that of print books from trade or other publishers. proportionally uml received more university press books (47%) as ebooks while university press books accounted for only 39% of the print titles received.   table 6 print book usage by type of publisher   no. of p-books no. used % used proportion of total p-books proportion of total uses university presses 2,055 788 38% 39% 39% other publishers 3,182 1,215 38% 61% 61% total 5,237 2,003 38%     p-value two-sided     0.4205 proportion used is not significantly different between publisher type p-value one-sided     0.2103 proportion used is not significantly greater note: p = < 0.05             ebooks use by vendor platform   the analysis by vendor platform suggested that ebrary was the preferred platform.     table 7 ebook usage by vendor platform   no. of ebooks no. used % used proportion of total ebooks proportion of total uses ebl 867 371 43% 47% 43% ebrary 820 434 53% 44% 50% ebsco 77 28 36% 4% 3% wiley 91 39 43% 5% 4% total 1,855 872 47%     p-value two-sided     <0.0001 proportion of usage differs significantly among vendors note: p = < 0.05             table 8 ebook usage by type of license   no. of ebooks no. used % used proportion of total ebooks proportion of total uses single user  (ebrary) 483 250 52% 26% 29% 1 user  (ebsco) 77 28 36% 5% 4% non-linear lending  (ebl) 863 368 43% 46% 52% multi-user (ebl) 4 3 75% .02% .03% multi-user (wiley) 91 39 43% 5% 4% multi-user (ebrary) 337 184 55% 18% 21% total 1,855 872 47%     p-value two-sided     <0.0001 proportion of usage differs significantly among the different vendor licenses note: p = < 0.05             figure 2 comparison of usage of single user and multi-user licences.     ebooks use by license   ebrary ebooks seemed to receive higher usage independent of license type.   when results were separated into single-user and multi-user licenses, single-user licensed books were used somewhat more.   substantive use of ebl and ebrary ebooks   both ebl and ebrary provided detailed reports on usage, allowing a better comparison between p-books and ebooks. the two providers accounted for 91% of the ebooks received on approval in 2012/2013. wiley and ebsco ebooks were excluded from cost and cost-per-use calculations. for the purpose of this study, substantive use was determined to have occurred either when a title had been downloaded at least once, or when 11+ pages had been viewed or read. the level of substantive use of ebooks (32%) is lower than the overall mean usage (38%) of approval print books based on check-outs.   table 9 substantive use of ebl and ebrary ebooks   no. ebooks no. used no. of sessions/ transactions mean session per book mode session per book no.  of ebooks downloaded no. of ebooks with substantive use % ebooks with substantive use ebl 868 371 1,519 4 2 149 309 35.6% ebrary 819 434 1,610 3.7 2 81 238 29% total 1,687 805         547 32%                       mean costs for ebl and ebrary ebooks   the overall cost of the ebl collections is slightly higher than that of ebrary’s because it has about 40% more “stm” books, which tend to be more expensive. in addition, pricing of an ebl non-linear lending ebook tends to be higher than a single-user title from ebrary.   a further analysis was done by broad discipline on the two platforms. the results showed that in general the ebl books were more expensive than ebrary’s in all disciplines except “social sciences”.     table 10 mean cost of ebl and ebrary ebooks   no. of ebooks cost of all ebooks mean cost of all ebooks no. of ebooks used cost of ebooks used mean cost of ebooks used ebl 868 $87,737* $101 371 $36,826 $99 ebrary 819 $70,556 $86 434 $36,063 $83 total 1,687 $158,293 $94 805 $72,889 $91 * all costs given are in canadian dollars   table 11a ebl ebooks mean cost by discipline   no. of ebooks cost of all e books mean cost of all ebooks no. of ebooks used cost of ebooks used mean cost of ebooks used arts & humanities 284 $25,168 $88 116 $10,792 $93 social sciences 313 $24,801 $79 123 $9,726 $79 stm 245 $31,800 $130 119 $14,863 $125 other 26 $3,043 $117 13 $1,446 $111 total 868 $84,812 $98 371 $36,827 $99   table 11b ebrary ebooks mean cost by discipline   no. of ebooks cost of all ebooks mean cost of all ebooks no. of ebooks used cost of ebooks used mean cost of ebooks used arts & humanities 322 $24,426 $76 173 $12,944 $75 social sciences 322 $27,574 $86 174 $15,062 $87 stm 148 $17,059 $115 70 $7,133 $102 other 27 $2,310 $86 17 $1499 $88 total 819 $71,369 $87 434 $36,638 $84     mean print book costs   the mean purchase price of all ebl and ebrary ebooks ($94) is much higher than the mean cost of all print books ($55).   a further breakdown of this information was done by broad discipline.     table 12 mean cost of print books no. of p-books cost of all p-books mean cost all no. of p-books used cost of p-books used mean cost p-books used 5,237 $289,672 $55 2,003 $112,318 $56     table 13 mean cost of print books by discipline   no. of pbooks cost. of all p-books mean cost all pbooks no. of pbooks used cost of pbooks used mean cost (pbooks used) arts & humanities 2,263 $109,462 $48 807 $39,735 $49 social sciences 1,740 $90,654 $52 657 $33,383 $51 stm 1,063 $79,681 $75 469 $34,774 $74 other 171 $9,875 $58 70 $4,426 $63 total 5,237 $289,672 $55 2003 $112,318 $56     cost per use comparison   cost per use for ebl ebooks, ebrary ebooks, and print books ($57, $51, and $34) was about 40% less than the mean purchase cost per book ($101, $86, and $56 respectively).   the cost-per-use data has a non-normal or right-skewed distribution for pand ebooks and for all disciplines. therefore, the nonparametric kruskal-wallis test was used to determine where cost-per-use differs among the three groups: p-books, ebl ebooks, and ebrary ebooks.   as an alternative strategy, the square-root transformation was applied to the cost-per-use figures in all groups in order to reduce the right-skewedness in the data. a t-test (to compare two groups at a time) and the anova (to compare all three groups) were conducted on the transformed data and the results were compared with those from the kruskal-wallis test.  the reported conclusion is based on the kruskal-wallis test.   ebl and ebrary ebooks are not significantly different in terms of their cost-per-use.  the mean cost per use is definitely lower for print books with the exception of books in the “other” discipline, where the small sample size fails to lead to a conclusion.     table 14 ebrary, ebl, and print comparison – cost per use/checkout   arts & humanities social sciences stm other all ebl no. used 116 123 119 13 371        cost $10,792 $9,726 $14,863 $1,446 $36,827        uses 396 522 521 82 1,521        cpu $57 $45 $72 $38 $57 ebrary no. used 173 174 70 17 434        cost $12,944 $15,062 $7,133 $1,499 $36,638        uses 676 641 230 63 1,610        cpu $47 $50 $64 $39 $51 print no. used 807 657 469 70 2,003       cost $39,735 $33,383 $34,774 $4,426 $112,318       uses 1,857 1,725 1,374 173 5,129       cpu $32 $31 $42 $38 $34     discussion   in terms of the comparison of print and ebook usage, although the overall results favoured ebooks, in the “stm” and “other” categories the results were inconclusive. the sample size was too small for the other category, and usage of stm books in the two formats was not significantly different. the sciences and technology libraries at uml have always had a comparatively small print book budget, and their smaller collections may account for greater print usage than in other categories.   usage of print approval books was low, but a number of libraries have indicated equally low results (e.g., rose-wiles, cornell university). these libraries studied a larger number of books over a longer time period, but nonetheless their findings point to a similar trend. a number of factors may be affecting print use: changes in teaching requiring fewer research papers, the availability of information sources outside of libraries, changing reading habits, an increased ability of students to buy books, and student reluctance to use the physical library.   as others have pointed out, it is difficult to compare pand ebook usage, especially if the print usage does not include in-house use.  although the comparison of substantive use of ebooks to check-outs of print books is an attempt to mitigate the lack of in-house information, ‘substantive use’ is not an objective measure. the results do, however, seem to agree with the often cited belief that print and ebooks are used for different purposes; ebooks for quick searches and reading shorter texts, and print books for more extensive reading.   in addition, the cost per use data for both formats in the study does not consider the labour cost of the processing and maintenance of print books versus those for ebooks. during the time studied, uml paid $14.87 per p-book for outsourced cataloguing and processing and nothing for ebook catalogue records. full consideration of any such ‘hidden’ costs is beyond the scope of this study, but nonetheless deserves further inquiry.   the ebook results by broad subject are similar to those for package titles in the “humanities/social sciences” (47%) and “science/technology” (49%) areas in the carrico study (2015, p. 106). carrico found much higher use of medical books; however, the only categories of medicine covered by the uml’s approval plan were nursing and public health. levine-clark’s (2014, slides 31-36) meta-analysis of ebl and ebrary ebooks found that social sciences titles were used more than “stm” ones, but in the uml study of approval ebooks, the usage was quite similar for the three main categories.    the higher usage of university press ebooks at uml is in agreement with levine-clark’s (2014, slides 23-27) study. however, type of publisher did not seem to affect print book usage at uml, perhaps reflecting the higher number of print approval books coming from important, non-university press publishers (e.g., wiley, routledge, palgrave macmillan).    the greater usage for ebrary ebooks was somewhat surprising given that ebl is the preferred format for many users and librarians at uml (warren, 2015, p. 15). however, during the study uml’s proxy server was dropping users of ebl ebooks. these access problems have since been remedied but could have skewed these ‘historical’ results. in addition, this result may reflect a different database structure and method of counting use.     table 15 p-value: ebl, ebrary, and print comparison – cost per use/checkout discipline comparison average difference t-test anova kruskal-wallis conclusion arts & humanities ebl to ebrary 9.30 0.163   0.322 not sig. different ebl to print 23.27 < 0.0001   < 0.0001 significantly different ebrary to print 14.97 < 0.0001   < 0.0001 significantly different ebl-ebrary-print     < 0.0001 < 0.0001 significantly different social sciences ebl to ebrary -4.28 0.170   0.068 not sig. different ebl to print 14.76 < 0.0001   0.024 significantly different ebrary to print 19.04 < 0.0001   < 0.0001 significantly different ebl-ebrary-print     < 0.0001 < 0.0001 significantly different stm ebl to ebrary 7.55 0.326   0.281 not sig. different ebl to print 29.92 < 0.0001   < 0.0001 significantly different ebrary to print 22.37 < 0.0001   0.005 significantly different ebl-ebrary-print     < 0.0001 < 0.0001 significantly different other ebl to ebrary -0.89 0.811   0.802 not sig. different ebl to print 0.35 0.822   1.00 not sig. different ebrary to print 1.24 0.899   0.931 not sig. different ebl-ebrary-print     0.965 0.931 not sig. different overall ebl to ebrary 6.46 0.201   0.474 not sig. different ebl to print 22.93 < 0.0001   < 0.0001 significantly different ebrary to print 16.47 < 0.0001   < 0.0001 significantly different ebl-ebrary-print     < 0.0001 < 0.0001 significantly different note: p = <0.05     conclusions   of approval books that matched the same subject profile, this study finds that ebooks are used more, but print books receive more substantial use. when the data is broken down by broad discipline, the greater usage for ebooks is significant for the “arts & humanities” and “social sciences” category. however, results for the “stm” and “other” categories were inconclusive. clearly both formats of books are needed in a library’s collection.   in terms of type of publisher, university press ebooks were more highly used than ebooks from other publishers, while for print books there appeared to be little difference in usage of the two publisher types. this difference may have occurred because a greater number of important non-university press books arrived in print. cost per use for print books ($34) was lower than for ebl and ebrary ebooks3 ($53), in part because ebooks are generally more expensive; the mean purchase cost of ebrary and ebl titles was $91, but $55 for print books. these differences may not tell the full story since print usage did not include in-house browsing or reading. on the other hand, turnstile counts at uml, as at other academic libraries, have decreased over the last decade suggesting that in-house use may not be as significant as it once was. finally, cost per use in this study did not reflect the cost of processing and maintenance of the different media, but the cost is likely higher for print.   it is recommended that future usage studies work with a larger sample of ebooks relative to print and take into account the hidden costs of access and maintenance of the two formats in order to produce more compelling results. many books are published in the arts & humanities and social sciences. for a library with limited resources and similar low use of print in these areas, it may be more cost-effective to acquire monographs in these disciplines, through publisher front-lists or pdas/ddas than through an approval plan.   usage patterns take time to unfold, and libraries like the uml often have to make abrupt changes to monographic acquisition methods and allocations for budgetary reasons. thus uml   altered and reduced its ybp approval plan before the results of this study were determined. nonetheless usage studies can provide evidence for librarians refining approval plan profiles and for budget managers considering changes to monographic acquisition methods and allocations.   endnotes   1 for a discussion of demand-driven acquisition options, see niso’s “demand driven acquisition: a recommended practice” (9-13) and for a discussion of ebook acquisition methods see kumbhar (pp. 786-787, 2012) and bucknell (2012).   2 arl statistics 2011/2012 to 2013/2014 provide statistics on amounts spent on one-time purchases, of which monographs would be a large portion.   3although ebsco (77) and wiley (91) ebooks came on approval, neither vendor supplied detailed usage reports that could be used for cost-per-use comparisons.   references   ahmad, p., & brogan, m. (2012). scholarly use of e-books in a virtual academic environment: a case study.  australian academic & research libraries, 43(3), 189-213. http://dx.doi.org/10.1080/00048623.2012.10722277   alan, r., chrzastowski, t. e., german, l., & wiley, l. (2010). approval plan profile assessment in two large arl libraries, university of illinois at urbana-champaign and pennsylvania state university.  library resources & technical services, 54(2), 64-76.   association of research libraries. (2013). arl statistics 2011-2012. retrieved from: http://publications.arl.org/arl_statistics   association of research libraries. (2014). arl statistics 2012-2013. retrieved from: http://publications.arl.org/arl_statistics   association of research libraries. (2015). arl statistics 2013-2014. retrieved from: http://publications.arl.org/arl_statistics   bucknell, t. (2012). buying by the bucketful: a comparative study of e-book acquisition strategies. insights, 25(1), 51-60. http://dx.doi.org/10.1629/2048-7754.25.1.51   bucknell, t. (2010). the ‘big deal’ approach to acquiring e-books: a usage-based study. serials, 23(2), 126-134. http://dx.doi.org/10.1629/23126   carrico, s. b., cataldo, t. t., botero, c., & shelton, t. (2015). what cost and usage data reveals about e-book acquisitions: ramifications for collection development. library resources & technical services, 59(3), 102-111. http://dx.doi.org/10.5860/lrts.59n3.102   cornell university library.  (2010). report of the collection development executive committee task force on print collection usage. retrieved from http://staffweb.library.cornell.edu/system/files/collectionsusagetf_reportfinal11-22-10.pdf and http://staffweb.library.cornell.edu/node/1781   downey, k., zhang, y., urbano, c., & klinger, t. (2014). a comparative study of print book and dda ebook acquisition and use. technical services quarterly, 31(2), 139-160. http://dx.doi.org/10.1080/07317131.2014.875379   goodwin, c. (2014). the e-duke scholarly collection: e-book v. print use. collection building, 33(4), 101-105. http://dx.doi.org/10.1108/cb-05-2014-0024   kimball, r., ives, g., and jackson, k. (2009). comparative usage of science e-book and print collections at texas a&m university libraries. collection management, 35(1), 15-28. http://dx.doi.org/10.1080/01462670903386182   kumbhar, r. (2012). e-books: review of research and writing during 2010. the electronic library, 30(1), 777-795. http://dx.doi.org/10.1108/02640471211282109   levine-clark, m. (2014, june 27). diving into ebook usage: an ala update. american library association annual meeting, las vegas, nv, usa. retrieved from: http://www.slideshare.net/michaellevineclark   levine-clark, m., & brown, c. (2013, march 19). e or p? a comparative analysis of electronic and print book usage. electronic resources & libraries conference, austin, tx, usa. retrieved from: http://www.slideshare.net/michaellevineclark/erl-e-books-2013   littman, j., & connaway, l. s. (2004). a circulation analysis of print books and e-books in an academic research library. library resources & technical services, 48(4), 256-262. retrieved from: http://www.oclc.org/content/dam/research/publications/library/2004/littman-connaway-duke.pdf   national information standards organization. (2014). demand-driven acquisition of monographs: a recommended practice (niso rp-20-2014). retrieved from: http://www.niso.org/apps/group_public/download.php/13373/rp-20-2014_dda.pdf   rose-wiles, l. m. (2013). are print books dead? an investigation of book circulation at a mid-sized academic library. technical services quarterly, 30(2), 29-152.   warren, r. (2015). e-book assessment report [for usability]. university of manitoba libraries.   evidence summary   chat transcript analysis reveals that undergraduate students are open to instruction, while instructors and librarians care about supporting student learning   a review of: jacoby, j., ward, d., avery, s., & marcyk, e. (2016). the value of chat reference services: a pilot study. portal: libraries and the academy, 16(1), 109-129. https://doi.org/10.1353/pla.2016.0013   reviewed by: elaine sullo coordinator, information and instructional services himmelfarb health sciences library the george washington university washington, district of columbia, united states of america email: elainej@gwu.edu   received: 30 nov. 2016 accepted: 1 feb. 2017      2017 sullo. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to investigate student, instructor, and librarian perspectives of chat reference service in the context of first-year undergraduate students conducting research for an introductory composition course.   design – focus groups, individual interviews, and surveys.   setting – a large, public university in the united states of america.   subjects – 57 library reference providers, 36 instructors of an introductory composition course, and approximately 936 undergraduate students in certain sections of the introductory composition course who were assigned a specific research project.   methods – in spring of 2014, all participants were invited via email to respond to an anonymous chat transcript of a librarian interacting with a student working on his or her research project. study participants could participate via a brief survey or by taking part in a focus group or individual interview. the invited instructors were asked to forward the invitation to the students in their sections, and reminder emails were sent two weeks after the initial email.   main results – nine instructors, 24 students, and 25 library reference providers participated in the study, representing a response rate of 25%, 3% (estimated), and 44%, respectively. the authors conducted a qualitative analysis of key themes that were derived from both the focus groups or individual interviews and the survey questions. the themes were: students as novice researchers, question negotiation, open and closed questions, instruction, speed and convenience, customer service, and referrals. the theme of “students as novice researchers” is based on student comments related to their frustrations of being inexperienced researchers, as well as librarian comments on strategies for helping these students. opinions regarding the traditional reference interview, including specific techniques that made the interaction successful, were categorized as “question negotiation.” the “open and closed questions” theme focused on feedback on the types of questions used by librarians in the reference interview. several components related to chat and instruction were encompassed within the “instruction” theme, including whether those participating in the study were conscious of librarians providing instructions via chat and whether it was deemed valuable; the impact of a library instruction session in which students participated; and identification of missed teachable moments during the chat. the “speed and convenience” theme represented thoughts regarding the balance of instruction and librarian support of news skills, with the student expectation of having their question answered quickly and efficiently. the “customer service” theme focused on the service quality of the reference transaction, while the “referrals” theme encompassed thoughts related to whether students were referred to subject specialists, writing specialists, instructors, or if there was a lack of a referral altogether.   conclusion – based on the research results, the authors highlighted the importance of the interconnectedness of teaching that is done in the classroom, in library instruction sessions, and on the reference desk, as all three types of instruction should align. furthermore, because students are open to instruction via the chat service when they are creating and revising their research question and delving into subject research, chat can be viewed as a key teaching and learning opportunity. additionally, study results led the authors to conclude that chat reference services could be better marketed; some students were unaware of the extent of the chat service or that it existed at all.   commentary   in this article, the authors address a topic that has been well researched, however they use a unique methodology in that they examine chat reference service from three perspectives – the student, the instructor, and the librarian. furthermore, while prior studies focus on the quality of chat transactions or user satisfaction with the service, this study, in contrast, considers a specific context for the chat, which is the student and instructor perspective related to using chat to assist undergraduate students develop research strategies for a particular assignment. the authors state that the patron perspective regarding chat transcript analysis provides librarians with a unique opportunity to understand what users value and hope that this study builds upon the existing literature that uses more traditional research methodologies.   the study was evaluated using the cristal checklist for appraising a user study (cristal, n.d.). the study addresses a clearly focused issue in terms of the population studied and the qualitative outcomes: the analysis of focus groups, individual interviews, and open-ended survey questions. the survey instrument and focus group questions are listed as appendices to the article, which makes for a study that could be easily replicated, although readers do not have access to the actual chat transcripts that were analyzed. while the questions were created for the specific study population, they could be used to survey a different student and faculty group without difficulty. the authors did not mention if the survey and focus group questions were pre-tested or piloted, so the researchers did not have the opportunity to ensure that the questions made sense to participants.   the primary limitation of this study was the small sample size and low response rate; this limitation contributed to the difficulty in performing statistical analysis of the survey responses. the authors suggest that a follow-up study could be considered that would focus on only one transcript, compared to multiple transcripts in the current study. a single transcript, along with a larger sample size, would allow the researchers to conduct a comparative analysis between the study groups and better understand the themes that emerged.   another probable limitation was the way the “librarians” group was defined. this category of participants was comprised of professional librarians, preprofessional graduate assistants, and a small number of paraprofessional staff. professional librarians who have extensive experience may have different perspectives related to chat transcripts than those who are non-librarians, or those with little experience.   while this research focused on undergraduate students and instructors in an introductory composition course, study data can be used by both instructors and librarians in other disciplines as well. specifically, the article provides insight into what students’ experience is like as “novice researchers.” instructors can use the information to better understand how librarians interact with students via chat and perhaps create plans to work in tandem with librarians so the information being provided is uniform. librarians providing chat service can examine and make changes to their service based on the themes and responses elicited in this research.   references   cristal checklist for appraising a user study. (n.d.) in nettingtheevidence.pbwiki.com. retrieved from http://nettingtheevidence.pbwiki.com/f/use.doc     evidence based library and information practice 2009, 4:2 3 evidence based library and information practice editorial responsibilites © 2009. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. editor-in-chief: denise koufogiannakis associate editor (articles): alison brettle associate editor (evidence summaries): lorie kloda associate editor (classics, using evidence in practice): scott walter associate editor (features, commentaries, ebl101, news): denise koufogiannakis production editor: katrine mallan copyeditors: priscilla stephenson (lead copyeditor), lisa shen, dale storie, mary virginia taylor, elizabeth zeeuw indexing support: pam morgan microsoft word comm_booth.doc evidence based library and information practice 2006, 1:2  51 evidence based library and information practice     commentary    the unteachable in pursuit of the unreadable?       andrew booth  reader in evidence based information practice  school of health and related research (scharr), university of sheffield  sheffield, united kingdom   email: a.booth@sheffield.ac.uk       © 2006 booth. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.  introduction  advocates of evidence based library and  information practice (eblip) are typically  enthusiastic of temperament and  beguiling in argument.  it is all too rare to  encounter either outright criticism of, or  even measured caution towards, the  paradigm.  reasons for this are clear;  eblip appeals to the scientific rationality  that underpins much of our day‐to‐day  work.  it imposes a semblance of order on  what is otherwise chaotic, and is  exemplified when we (only half‐jokingly)  say, “of course it would all work perfectly,  if it weren’t for our users.”  however, the  real world is messy and uncertain, and  problems proliferate.    for some professions, medicine being the  foremost example (hayward), uncertainty  is a natural state of affairs. “is this drug,  which has been shown to benefit patients  on average, likely to be beneficial in my  patient who is older, has a more advanced  stage of disease, and has an additional co‐ existing condition?”  library and  information practice is not so comfortable  with uncertainty.  as a profession we tend  to be optimistic, clinging to a linear and  mechanical model of information transfer  between research and practice, by which  knowledge is simply moved from one  place to another.  our underlying  assumption is that if an idea is good  enough, it will be used (nutley, walter,  davies). a corollary to this is that when  we ourselves face a knowledge gap, we  believe we can address this simply by  acquiring more information – the so‐called  information deficit model (marteau,  sowden, armstrong).  the unteachable  this editorial deliberately evokes oscar  wilde’s description of fox‐hunting, what  he terms “the unspeakable in full pursuit  of the uneatable” (wilde). it highlights  two overwhelming barriers to eblip ‐ one  at the consumption end and the other at  the production end of the evidence chain;  namely that librarians are ‘unteachable’,  and systematic reviews are ‘unreadable’.   whether eblip proves as futile a pursuit  as fox‐hunting depends on whether these  barriers can be overcome.    of course there is an element of licence to  both appellations.  librarians are not  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:2  52 inherently ‘unteachable’; as a profession  we show ourselves ever ready to adapt  and to respond to new organisational  imperatives.  however, as hinted above,  we do this by addressing an information  deficit, by absorbing new data like a  sponge.    critical appraisal, “the process of  assessing and interpreting evidence by  systematically considering its validity,  results, and relevance to an individual’s  work” (last), is a building block of  evidence based practice.  frequently,  when taught such skills for the first time,  healthcare or information practitioners  expect to arrive at some authoritative  answer – life, the universe, and everything  equals 42 (adams).  when, in contrast, the  session concludes with limited certainty –  with, perhaps, more questions than  answers – most healthcare professionals  can handle this.  this is where  ‘unteachable’ comes into the picture.   library and information practitioners  respond to the challenge of critical  appraisal in their time‐honoured fashion –  by seeking to learn more about research  design, by becoming more familiar with  statistics, and by attempting to learn the  terminology and concepts associated with  the specific context in which the research  has been conducted.  this quest, this  appetite for knowledge, is potentially  insatiable.  as a consequence, most library  and information professionals never feel  fully equipped to meet the technical  challenges of evidence based practice.  in  fact the more they learn, the more they  recognise that there is much more yet to  learn.   acquiring self‐efficacy  so what is the solution? certainly the  answer is not to equip librarians with all  they need to know.  if we add formal  training in research design and statistics,  and possibly sector‐specific terminology,  to the array of professional skills we  already require, we will produce a  profession that enjoys only a handful of  years of competence immediately prior to  retirement!  instead, librarians need to  acquire self‐efficacy, defined as “the belief  that one can perform successfully the  behaviour required to produce designated  types of performance” (sadri and  robertson 139).  the emphasis is therefore  not on what we know, but on equipping  ourselves with as many strategies as  possible to handle not knowing.  such  strategies may indeed include different  ways of research and discovery. however,  they will also comprise different ways of  acknowledging our ignorance and  techniques for making decisions when  faced with limited information.  these  might include tools for formal decision‐ making and informal sharing of ‘rules of  thumb’, grounded in a practical working  knowledge of the quality and quantity of  our knowledge base.      thus, library and information  practitioners are ‘unteachable’ to the  degree that they rely too much on a single  strategy for addressing an ‘information  deficit’.  only by acquiring strategies for  handling uncertainty ‐ an uncertainty  endemic to a field where the evidence base  is of poor quality, where researchers do  not answer the questions that practitioners  ask, and where insights are scattered  across the literature of numerous  disciplines – will they be able to fully  engage with evidence based library and  information practice.  the unreadable?     while the accusation that library and  information practitioners are ‘unteachable’  requires brief justification, describing  systematic reviews as ‘unreadable’ may  appear downright perverse. after all, are  not most systematic reviews structured  documents with clear statements of  implications for research and practice?  furthermore, as an erstwhile champion of  systematic reviews in general, and  specifically within library and information  evidence based library and information practice 2006, 1:2  53 practice, i represent an unexpected source  of criticism.  the systematic review model  does have limitations – a refreshing aspect  of the systematic review community is its  facility for self‐scrutiny – but it is  fundamentally sound.   however,  systematic reviews expend extensive  resources and should be pursued only if  there is a reasonable expectation of  furnishing some ‘answer’ and hence of  achieving ‘closure’.    several factors reduce the likelihood of a  useable answer in the field of library and  information practice when compared with  other disciplines. as previously remarked,  our evidence base is poor with  comparatively few experimental studies.  as a consequence, systematic reviews  steer between a scylla of very precise  reviews addressing tightly focused  questions (by which we learn more and  more about less and less, until we know  everything about nothing!) and a  charybdis of broad overviews containing  studies which are so dissimilar as to make  comparison and synthesis almost  impracticable.  similarly, the diffusion of  our potential evidence base across many  disciplines tends towards fragmentation,  rather than consolidation.     the real drawback of systematic reviews,  paradoxically the source of their  methodological strength, is the quality  assurance offered by the review process,  regardless of outcome. imagine having the  same quality assurance mechanisms in  place for automobile manufacture  regardless of whether the resultant car  will participate in the paris‐dakar rally or  never leave the showroom.  reviewers are  required to follow the same phases of the  review process whether they retrieve and  synthesise numerous high quality primary  research studies, whether they identify  only poor quality research, or whether  they find no eligible studies whatsoever!   herein lies a paradox; we need a fairly  good idea about whether a review topic is  viable and cost‐effective before we  undertake our review.  at the same time  we must not have any a priori suspicion of  what the findings will be, otherwise we  are likely to introduce bias.     as a consequence, there is a very strong  likelihood that we will continue to witness  the production of well‐conducted, well‐ written systematic reviews where the  bottom line is that there is no bottom line.   such systematic reviews are readable in  the technical sense but are ‘unreadable’ to  the extent that they offer little or no  pragmatic guidance for library and  information practice. practitioners will feel  correspondingly little allegiance to an  otherwise high‐quality ‘academic’ product  and will likely concentrate their precious  professional reading time on material that  is more immediately rewarding.   diorama, not microscope?  what, then, is the alternative? as a  fundamentally pragmatic profession we  continually seek answers to practical  questions to benefit our users. we require  breadth of vision (a holistic view that  embraces multiple perspectives), rather  than depth (as offered by systematic  reviews). we need a diorama,* not a  microscope!    how might this work? the library and  information practitioner works with  stakeholders to identify a fully‐rounded  view of a real‐life problem. this might  involve a genuinely consultative process  such as a focus group or group interview.  alternatively the practitioner may piece  together a series of individual two‐ dimensional viewpoints (e.g., the reader  and the professional, or the manager and  the funder) to create a semblance of the  entire problem. the aim is to identify  issues, potential causes, interrelationships,  and solutions to model – perhaps using  such techniques as concept mapping or  mind mapping (ferrario). they then  explore the literature to populate their  basic models with evidence from research  evidence based library and information practice 2006, 1:2  54 and other sources such as good practice,  surveys, and service evaluations. they  employ a best available evidence approach  (eldredge). lower quality evidence is  continually overwritten by higher quality  evidence when it is identified, or when  new evidence appears. they do not  employ a comprehensive and ultimately  fruitless attempt to gather high‐quality  evidence on a single topic. instead, they  try to gain as many insights as are feasible  for the entire problem. beginning with a  series of rapid searches around possible  causes, there follows a series of searches  around likely interventions and  alternatives, perhaps culminating in brief  searches on how impact has been  evaluated. in short, they pursue related  approaches variously anchored in the  population, interventions/comparisons, or  outcomes separately, instead of seeking  some ‘holy grail’ document that contains  exactly what they want. boosted by this  multiple array of relevant evidence, the  practitioner gains a broader  understanding of the issues and options  involved and thus makes a better,  evidence‐informed decision.     to illustrate, we may be considering  introduction of a virtual reference desk  alongside (or even as an alternative to)  traditional face‐to‐face reference services.  we could conduct a series of rapid  searches to look at the context within  which such services are being considered  together with their claimed advantages  and limitations. we could seek to identify  a range of proposed models, such as e‐ mail or chat‐based reference service, in  parallel to existing services or as a means  of extending availability across time or  geographic space. we could look at  qualitative data reporting staff or reader  views of such services. we may identify  published evaluations that have examined  implementation of such services. while  recognising that results may be context‐ specific, we may learn a considerable  amount about pertinent issues and  possible metrics for success. of course if  the fruits of our labours are particularly  productive, we will naturally encounter  any rigorous studies with the potential to  answer that important ‘effectiveness’  question. more importantly, however, if  we do not find any sufficiently rigorous  studies, we still have a considerable body  of evidence to inform our pragmatic  decision. we have not allowed the ideal to  be the enemy of the good.     these route maps for evidence based  problem solving (tentatively assigned the  working title “resolve”) become living  organic documents to be shared, edited,  and augmented as other colleagues bring  additional insights to bear. unlike  systematic reviews, they require little  technical knowledge either to interpret or  to use in practice. they can be locally  owned, adapted and modified. where  rigorous studies or systematic reviews do  exist, these underpin specific nodes on the  mind map or concept map that address  very focused questions.  the way forward?  cynics may identify this suggestion as yet  another attempt to breathe life into a  paradigm that continues to fit  uncomfortably within library and  information practice. evidence products  such as guidelines (booth “from ebm to  ebl”), critically appraised topics (booth  “research”), and systematic reviews  (booth “health librarians”) have been  proposed as evidence based tools to  address the research‐practice gap within  our profession. while the visual format of  these route maps reflects established  techniques of concept mapping, their  theoretical base draws on realist synthesis  (pawson et al.). however, the context for  their use is far more pragmatic and ‘front‐ line’ than the originators of realist  synthesis ever envisaged. producing an  accumulating body of knowledge that  requires little technical knowledge,  acknowledges uncertainty, and presents  key messages in an easily navigable  evidence based library and information practice 2006, 1:2  55 format offers the potential to make library  and information professionals ‘teachable’  and the products of their evidence  syntheses ‘readable’!     note  * in this context a diorama is “a model  which shows a situation . . . in a way that  looks real, because the height, length, and  width of what is being shown are  accurately represented in comparison with  each other.”  works cited  adams, douglas. the hitchhikerʹs guide  to the galaxy.  chapter 27.  <http://www.globusz.com/ebooks/hit chhikers/00000038.htm> .    booth, andrew.  ʺfrom ebm to ebl: two  steps forward or one step back?ʺ  medical reference services quarterly  21.3 (2002): 51‐64.     ‐‐‐.  “research.”health libraries  review 17.4 (2000): 232‐5.     ‐‐‐. “will health librarians and related                  information workers ever work   together to create an international  network?” health information and  libraries journal 18.1 (2001): 60‐3.     eldredge, jonathan d. “evidence‐based  librarianship: what might we expect  in the years ahead?” health  information and libraries journal 19.2  (2002): 71‐7.    ferrario, catherine g. “developing  nursesʹ critical thinking skills with  concept mapping.” journal for  nurses in staff development – jnsd  20.6 (2004): 261‐7.     hayward, richard d. “balancing  certainty and uncertainty in clinical  medicine.”  developmental medicine  & child neurology  48.1 (2006): 74‐7.  last, john m., ed.. “critical appraisal.” a  dictionary of epidemiology. 2nd ed.  new york: oxford university press,  1988.    marteau, theresa m, amanda sowden,  and david armstrong. “implementing  research findings into practice:  beyond the information deficit  model.”  getting research findings  into practice. eds. andrew haines and  anna donald. london: bmj books,  1998. 68‐76.      nutley, sandra, isobel walter, and huw  davies. “from knowing to doing: a  framework for understanding the  evidence‐into‐practice agenda.” mar.  2002.  esrc research unit for  research utilisation discussion paper  1. department of management,  university of st andrews. 28 mar.  2006 <http://www.st‐ andrews.ac.uk/~ruru/knowdo%20  paper.pdf>.     pawson, ray, trisha greenhalgh, gill  harvey, and kieran walshe. realist  synthesis: an introduction. aug. 2004.  esrc research methods programme,  university of manchester. rmp  methods paper. 27 mar. 2006  <http://www.ccsr.ac.uk/methods/publi cations/documents/  rmpmethods2.pdf>.       sadri, golnaz, and ian t robertson. “self‐ efficacy and work‐related behaviour:  a review and meta‐analysis”  applied psychology: an international  review 42.2 (1993): 139‐52.    walter, elizabeth. ed. “diorama”.  cambridge advanced learners  dictionary. 2nd ed. cambridge  university press, 2005.  <http://dictionary.cambridge.org/defin e.asp?key=21869&dict=cald>.        http://www.globusz.com/ebooks/hit http://www.st%e2%80%90andrews%00 http://www.ccsr.ac.uk/methods/publi http://dictionary.cambridge.org/defin evidence based library and information practice 2006, 1:2  56 wilde, oscar. a woman of no importance,  act 1. encyclopedia of self‐ knowledge. 27 mar. 2006  <http://emotionalliteracyeducation.co m/classic_books_online/awoni10.htm>.      http://emotionalliteracyeducation.co evidence summary   academic librarians in spanish-speaking latin america see a growing need for librarian role in scholarly publishing literacy   a review of: buitrago ciro, j. (2022). how are academic libraries in spanish-speaking latin america responding to new models of scholarly communication and predatory publishing? journal of librarianship and information science, 54(3), 373–388. https://doi.org/10.1177/09610006211016533   reviewed by: abbey lewis stem engagement librarian university of colorado boulder boulder, colorado, united states of america email: abbey.b.lewis@colorado.edu   received: 20 jan. 2023                                                                   accepted:  4 apr. 2023      2023 lewis. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30308     abstract   objective – to examine the current state of knowledge among academic librarians in spanish-speaking countries in latin america regarding open access and predatory publishing. furthermore, the study sought to explore actions or potential plans among this group that could allow them to better assist researchers with these aspects of scholarly publishing literacy.   design – an online survey, followed by interviews with self-selected participants.   setting – academic libraries in spanish-speaking countries in latin america.   subjects – 104 librarians in the online survey and seven librarians in the interviews, each holding a position in an academic library in spanish-speaking latin america.   methods – an anonymous survey in spanish consisting of 34 required questions was built with surveymonkey and distributed to participants between 30 july 2020 and 20 september 2020. survey respondents were directed to contact the author via email if they wished to participate in follow-up interviews. interviews were then conducted in spanish over zoom between 10 october 2020 and 10 november 2020, and the transcripts were coded and analyzed with nvivo.   main results – of the survey’s respondents, 31.73% indicated that their library has a scholarly communication librarian, 29.81% noted that the library receives requests for assistance with finding journals in which to publish at least once per week, and 16.35% reported this type of request at least 1-2 times per month. in specific regard to predatory publishing, almost 52.88% of respondents indicated that they never receive questions on this topic, and 25.00% answered that users seek this kind of information less than once per month. additionally, 31.73% responded that information on predatory publishing is not available at their library, and 42.31% rated their own knowledge of predatory publishing as fair to very poor. finally, 44.23% reported plans to develop training or information to educate their institution’s research communities about predatory publishing, and 2.88% planned to recruit a scholarly communications specialist. in interviews, all participants agreed that exchanging information with other institutions with more knowledge and services related to predatory publishing would be beneficial. three interviewees saw a general role for the library in providing information on predatory publishing, and three others indicated that this should be a leadership role when working in conjunction with other departments. all participants saw a lack of knowledge about predatory publishing as one of the primary difficulties for academic libraries, and three reported that the survey itself had led them to recognize the importance of the library’s role in scholarly communication literacy and create plans for their library to assist researchers in regard to predatory publishing.   conclusion – academic libraries in spanish-speaking countries in latin america are not yet offering robust support for researchers concerning predatory publishing, but there is growing recognition of the issue, as well as interest in further developing expertise among librarians. collaboration and exchanges of information with other institutions with more developed resources related to predatory publishing is viewed positively and may build upon existing knowledge of open access publishing.   commentary   as the global south expands contributions to the open access information landscape and infrastructure, predatory publishing impedes progress in research achievement and dissemination of information (heredia, 2022; jain & singh, 2019). more guidance and training on the risks of predatory publishing is needed to help researchers successfully navigate the publication process (otike et al., 2022). this study turns attention to the role that academic libraries in latin america might play in directing researchers to appropriate publication venues and fostering broader scholarly communication literacy (buitrago ciro, 2022).   the study is assessed here using boynton and greenhalgh’s (2004) guide. the survey was conducted in spanish, which is appropriate given the geographic locations of the respondents. however, the study’s publication in an english-language journal inhibits the utility of reproducing the exact instrument used. translations or paraphrasing are instead provided for almost all questions in the survey. the choice of topics is well-supported by a thorough literature review covering open access and predatory publishing, laying the groundwork for understanding latin american librarians' approaches to these issues.   the author acknowledges recruitment from his own personal network from employment as an academic library director in colombia likely led to an overrepresentation of colombian librarians, limiting the potential for generalizing results. however, each of the 19 countries that comprise spanish-speaking latin america are represented in survey responses and six countries were represented in the follow-up interviews. during interviews, some participants noted they were selected to answer the survey on their institution’s behalf. it is unclear whether this selection was part of the survey’s instructions. to safeguard the anonymity of respondents, the survey did not ask for institution names, so there may have been instances of multiple responses for one institution, leading to overrepresentation.   methods, themes, and analytic tools are adequately detailed for the study’s interview phase. this method of semi-structured questioning allowed for further exploration of survey topics and additional confirmation of the survey’s findings. participants sometimes described themselves as the person at their institution with the most knowledge about open access and predatory publishing practices. while this positioning allowed for substantial insight on the issues each university confronted, it also reveals that responses regarding self-assessed knowledge of scholarly communications may not be representative of most librarians at these institutions.   the study offers recommendations that could enhance support for assisting researchers with identifying reputable journals for publication through librarian training in the form of workshops and information sharing collaborations with other institutions. additional research should be used to investigate the effectiveness of these kinds of training and resulting changes in researcher support. the methodology employed here may also be useful when exploring the scholarly communication knowledge and practices of academic librarians in other geographic areas where open access publishing practices are increasing in popularity.   references   boynton, p. m., & greenhalgh, t. (2004). selecting, designing, and developing your questionnaire. bmj, 328(7451), 1312–1315. https://doi.org/10.1136/bmj.328.7451.1312   buitrago ciro, j. (2022). how are academic libraries in spanish-speaking latin america responding to new models of scholarly communication and predatory publishing? journal of librarianship and information science, 54(3), 373–388. https://doi.org/10.1177/09610006211016533   heredia, a. (2022). a tradition of open, academy-owned, and non-profit research infrastructure in latin america. information services & use, 42(3–4), 447–452. https://doi.org/10.3233/isu-220177   jain, n., & singh, m. (2019). the evolving ecosystem of predatory journals: a case study in indian perspective. in a. jatowt, a. maeda, & s. y. syn (eds.), digital libraries at the crossroads of digital information for the future: 21st international conference on asia-pacific digital libraries, icadl 2019, kuala lumpur, malaysia, november 4–7, 2019, proceedings (pp. 78–92). springer. https://doi.org/10.1007/978-3-030-34058-2_9   otike, f., bouaamri, a., & hajdu barát, á. (2022). predatory publishing: a catalyst of misinformation and disinformation amongst academicians and learners in developing countries. the serials librarian, 83(1), 81–98. https://doi.org/10.1080/0361526x.2022.2078924 classics   library anxiety impedes college students’ library use, but may be alleviated through improved bibliographic instruction   a review of: jiao, q. g., onwuegbuzie, a. j., & lichtenstein, a. a. (1996). library anxiety: characteristics of ‘at-risk’ college students. library & information science research, 18(2), 151-163. https://doi.org/10.1016/s0740-8188(96)90017-1   reviewed by: barbara wildemuth professor emeritus school of information and library science university of north carolina chapel hill, north carolina, united states of america email: wildemuth@unc.edu   received: 3 oct. 2017     accepted: 31 oct. 2017      2017 wildemuth. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to identify the characteristics of college students that are related to their experiences of library anxiety.   design – survey, analyzed with multiple regression.   setting – two universities, one in the mid-south and one in the northeastern united states.   subjects – 493 students from those two universities.   methods – the students responded to two questionnaires: the library anxiety scale developed by bostick (1992), and a demographic information form that included questions about students’ gender, age, native language, academic standing and study habits, library instruction received, and library use. spearman’s rank correlation was used to identify those demographic characteristics that were correlated with library anxiety. multiple regression analysis was used to develop a model for predicting library anxiety.   main results – the study found that age, sex, native language, grade point average, employment status, frequency of library visits, and reasons for using the library contributed significantly to predicting library anxiety. library anxiety was highest among young male students who did not speak english as their native language, had high levels of academic achievement, were employed while in school, and infrequently visited the library. while the overall regression model was statistically significant and explained 21% of the variability in library anxiety, the individual correlations with library anxiety were generally weak (the strongest was a -0.21 correlation with frequency of library visits).   conclusion – the authors conclude that many students experience library anxiety, and recommend that libraries make every effort to be welcoming. in addition, they recommend that library instruction should be introduced at the high school level and, in college, incorporated into the classes that require library research. in this setting, library anxiety should be addressed during the instruction, and classroom teachers should plan to assist students in the early stages of their research.   commentary   while the results from this study continue to be both interesting and useful to practitioners, its primary contribution was to further develop the concept of library anxiety. work on defining and measuring library anxiety began with mellon’s (1986/2015) grounded theory study of it. eblip published a classic review of this work in 2008 (bailey, 2008). jiao and onwuegbuzie began publishing a number of studies on library anxiety in 1996, with this article; published their book with sharon bostick in 2004; and continued to publish on library anxiety through 2008 (jiao, onwuegbuzie, & waytowich, 2008). key reviews on library anxiety include those by cleveland (2004) and carlile (2007).   in the article reviewed here, jiao, onwuegbuzie, and lichtenstein (1996) define library anxiety as “an uncomfortable feeling or emotional disposition, experienced in a library setting, which has cognitive, affective, physiological, and behavioral ramifications. it is characterized by ruminations, tension, fear, feelings of uncertainty and helplessness, negative self-defeating thoughts, and mental disorganization” (p.152). in onwuegbuzie and jiao’s later (2004) book, co-authored with bostick, they clarify further, noting that library anxiety is not an enduring personal trait, but is a state experienced in particular situations.   in their work on library anxiety, jiao and onwuegbuzie have consistently used bostick’s (1992) library anxiety scale, which consists of 43 likert-scale items (rated on a 5-point scale from strongly agree to strongly disagree). the items are organized into five factors: barriers with staff, affective barriers, comfort with the library, knowledge of the library, and mechanical barriers. this scale continues to be the basis for most studies of library anxiety. for example, it was used in mcpherson’s (2015) study of library anxiety among college students in the west indies.   as noted above, jiao and onwuegbuzie conducted a number of studies after publishing this initial study; these are reviewed by cleveland (2004). these studies considered the relationship between level of library anxiety and students’ demographic characteristics, such as race (jiao, onwuegbuzie & bostick, 2004); the causes or antecedents of library anxiety, including students’ computer attitudes, their reading ability, and their learning environment preferences, among others (e.g., jiao & onwuegbuzie, 2004); the relationship between library anxiety and other psychological characteristics, such as social interdependence (jiao & onwuegbuzie, 2002), academic procrastination (onwuegbuzie & jiao, 2000), and perfectionism (jiao & onwuegbuzie, 1998); and the effects of library anxiety on study habits (jiao & onwuegbuzie, 2001) and citation errors (jiao, onwuegbuzie, & waytowich, 2008). while many of the student characteristics that have been associated with library anxiety are not amenable to change (such as race or social interdependence), some of them may be targets for intervention by librarians or college instructors. for example, an instructor’s structuring of a library research assignment or a librarian’s discussion of a research plan, including a schedule with intermediate deadlines, may address the role of procrastination in library anxiety. thus far, few intervention studies building directly on these findings have included an evaluation of the success of the intervention.   studies by other researchers have paralleled the work of jiao and onwuegbuzie. they have studied the prevalence of library anxiety among students in particular countries including pakistan (naveed, 2016) or specific disciplines such as nursing (still, 2015); possible causes or antecedents of library anxiety (karim & ansari, 2013); the relationship between library anxiety and other psychological characteristics, such as foreign language anxiety (sinnasamy & karim, 2014); and the effects of library anxiety, such as the adoption of online library resources (booker, detlor, & serenko, 2012). these studies augment the research program of jiao and onwuegbuzie, in terms of adding more depth to our understanding of these aspects of library anxiety.   in addition, several researchers have taken the next step, to develop approaches to alleviating library anxiety and evaluating the effectiveness of those approaches. several of these focused on new approaches to bibliographic instruction. for example, fleming-may, mays, and radom (2015) developed a three-workshop model specifically targeting at-risk students; bell (2011) developed a one-credit information literacy course; and van scoyoc (2003) compared a computer-assisted instruction approach with face-to-face instruction led by a librarian. these and other studies have taken to heart some of the recommendations made by jiao and onwuegbuzie, to develop bibliographic instruction programs that are intended to alleviate library anxiety, then went on to evaluate the effectiveness of those programs. in addition, a few studies have taken a broader look at improvement of library services. for example, ott and chhiu (2007) evaluated the effectiveness of providing “deskless” services, and dugdale (2000) evaluated the impact of electronic reserves. these studies serve as models for the ways in which findings about library anxiety can be used to improve the practice of reference services generally, and bibliographic instruction specifically. even now, over twenty years after jiao and onwuegbuzie published their first study of library anxiety, their results can be mined for additional ways in which library services might be improved.   in addition, more research in this area should be conducted, particularly because the technological infrastructure of academic libraries has evolved since 1996. college students now routinely use hybrid collections: a physical collection in the library itself, as well as the online resources provided through the library. two examples will illustrate some of the possibilities for future research. first, the role of technology might be considered in relation to library use. as kohrman (2003) points out, “the library’s old wooden card catalog is now a computerized online catalog greeting students” (p.1), so computer anxiety must be considered in combination with library anxiety. while still widely used, bostick’s library anxiety scale would be more valuable for current research if it were updated to take this change into account. second, the situation in which students experience anxiety might be broadened slightly, to consider information seeking more broadly, rather than focusing on library use. as our field has made the turn to user-centered studies of information behaviors and practices, we might also make this turn in the measures we use. for example, naveed (2016) has recently taken the first steps toward developing a measure of information seeking anxiety. such a measurement instrument may also lead to studies of the anxiety of information seekers outside the academic library setting.   it is true that the number of studies on library anxiety, particularly those conducted in north america and europe, has decreased quite a bit since the publication of the onwuegbuzie, jiao, and bostick (2004) book. it’s not as clear that there is less library anxiety being experienced by college students or other library users. thus, this continues to be an area in which important research – both applied and basic – can be fruitfully conducted.   references   bailey, e. (2008). constance mellon demonstrated that college freshmen are afraid of academic libraries. evidence based library and information practice, 3(3), 94-97. http://dx.doi.org/10.18438/b8fs60   bell, j. c. (2011). student affect regarding library-based and web-based research before and after an information literacy course. journal of librarianship and information science, 43(2), 120-130. https://doi.org/10.1177/0961000610383634   booker, l. d., detlor, b., & serenko, a. (2012). factors affecting the adoption of online library resources by business students. journal of the american society for information science and technology, 63(12), 2503-2520. https://doi.org/10.1002/asi.22723   bostick, s. l. (1992). the development and validation of the library anxiety scale (doctoral dissertation). wayne state university, michigan.   carlile, h. (2007). the implications of library anxiety for academic reference services: a review of the literature. australian academic & research libraries, 38(2), 129-147. https://doi.org/10.1080/00048623.2007.10721282   cleveland, a. (2004). library anxiety: a decade of empirical research. library review, 53(3), 177-185. https://doi.org/10.1108/00242530410526583   dugdale, c. (2000). a new uk university library service to meet the needs of today’s students and tomorrow’s lifelong learners. information services and use, 20(1), 1-8.   fleming-may, r. a., mays, r., & radom, r. (2015). “i never had to use the library in high school”: a library instruction program for at-risk students. portal:libraries and the academy, 15(3), 433-456. https://doi.org/10.1353/pla.2015.0038   jiao, q. g., & onwuegbuzie, a. j. (1998). perfectionism and library anxiety among graduate students. journal of academic librarianship, 24(5), 365-371. https://doi.org/10.1016/s0099-1333(98)90073-8   jiao, q. g., & onwuegbuzie, a. j. (2001). library anxiety and characteristic strengths and weaknesses of graduate students’ study habits. library review, 50(2), 73-80. https://doi.org/10.1108/00242530110381118   jiao, q. g., & onwuegbuzie, a. j. (2002). dimensions of library anxiety and social interdependence: implications for library services. library review, 51(2), 71-78. https://doi.org/10.1108/00242530210418837   jiao, q. g., & onwuegbuzie, a. j. (2004). the impact of information technology on library anxiety: the role of computer attitudes. information technology and libraries, 23(4), 138-144. https://doi.org/10.6017/ital.v23i4.9655   jiao, q. g., onwuegbuzie, a. j., & bostick, s. l. (2004). racial differences in library anxiety among graduate students. library review, 53(4), 228-235. https://doi.org/10.1108/00242530410531857   jiao, q. g., onwuebguzie, a. j., & waytowich, v. l. (2008). the relationship between citation errors and library anxiety: an empirical study of doctoral students in education. information processing & management, 44(2), 948-956. https://doi.org/10.1016/j.ipm.2007.05.007   karim, n. h. a., & ansari, n. a. (2013). investigating the effects of students' major and bibliographic instruction programme on library anxiety sub-scale, 'barriers with staff'. malaysian journal of library & information science, 18(3), 39-47.   kohrman, r. (2003). computer anxiety in the 21st century: when you are not in kansas any more. paper presented at the association of college and research libraries 11th national conference, charlotte, nc. retrieved from https://pdfs.semanticscholar.org/d7cc/ac7c1ab4af5918e0e01829582714e7e893d3.pdf   mcpherson, m. a. (2015). library anxiety among university students: a survey. ifla journal, 41(4), 317-325. https://doi.org/10.1177/0340035215603993   mellon, c. a. (2015). library anxiety: a grounded theory and its development. college & research libraries, 76(3), 276-282. (original work published 1986). https://doi.org/10.5860/crl.76.3.268   naveed, m. a. (2016). exploring information seeking anxiety among research students in pakistan. libri, 66(1), 73-82. https://doi.org/10.1515/libri-2015-0047   onwuegbuzie, a. j., & jiao, q. g. (2000). i'll go to the library later: the relationship between academic procrastination and library anxiety. college & research libraries, 61(1), 45-54. https://doi.org/10.5860/crl.61.1.45   onwuegbuzie, a. j., jiao, q. g., & bostick, s. l. (2004). library anxiety: theory, research, and applications.. lanham, md: scarecrow press.   ott, k., & chhiu, s. (2007). the first wave: floating in the florida state university strozier library, florida state university, tallahassee, florida. new library world, 108(3-4), 165-176. https://doi.org/10.1108/03074800710735366   sinnasamy, j., karim, n. h. a. (2014). a correlational study of foreign language anxiety and library anxiety among non-native speakers of english: a case study in a malaysian public university. journal of academic librarianship, 40(5), 431-435. https://doi.org/10.1016/j.acalib.2014.07.010   still, m. (2015). addressing library anxiety (la) in student nurses: a study in an nhs foundation trust hospital library and information service. health information and libraries journal, 32(4), 322-325. https://doi.org/10.1111/hir.12124   van scoyoc, a. m. (2003). reducing library anxiety in first-year students: the impact of computer-assisted instruction and bibliographic instruction. reference & user services quarterly, 42(4), 329-341.   research article   visualization of the scholarly output on evidence based librarianship: a social network analysis   nafiseh vahed msc student  department of medical library and information sciences  school of management and medical informatics tabriz university of medical sciences tabriz, iran  email: vahedn66@gmail.com   vahideh zarea gavgani associate professor  department of medical library and information science, school of management and medical informatics tabriz university of medical sciences tabriz, iran email: vgavgani@gmail.com   rashid jafarzadeh phd candidate  department of knowledge and information science tehran university tehran, iran email: rashid.jafarzadeh@gmail.com   ziba tusi scientometrics librarian department of knowledge and information science   yazd university yazd, iran email: zbtusi@yahoo.com   mohammadamin erfanmanesh senior lecturer department of library & information science, faculty of computer science & information technology university of malaya kuala lumpur, selangor, malaysia email: amin.erfanmanesh@gmail.com   received: 23 dec. 2017                                                                   accepted: 14 oct. 2018      2018 vahed, gavgani, jafarzadeh, tusi, and erfanmanesh. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29396     abstract   objective – this paper aimed to analyze worldwide research on evidence based librarianship (ebl) using social network analysis (sna).   methods – this descriptive study has been conducted using scientometrics and a sna approach. the researchers identified 523 publications on ebl, as indexed by scopus and web of science with no date limitation. a range of software tools (ravar premap, netdraw, ucinet and vosviewer) were utilized for data visualization and analysis.   results – results of the study revealed that the united kingdom (uk) and the united states (us) occupied the topmost positions regarding centrality measures, clearly indicating their important structural roles in ebl research. the network of ebl research in terms of the degree of connectedness showed low density in the co-authorship networks of both authors (0.013) and countries (0.214). seven subject clusters were identified in the ebl research network, four of which related to health and medicine. the occurrence of the keywords related to these four subject clusters suggested that ebl research had a greater association with the setting of health and medicine than with traditional librarianship elements such as human resources or library collection management.   conclusion – this study provided a systematic understanding of topics, research, and researchers in ebl by visualizing the networks and may thus inform the development of future aspects of ebl research and education.     introduction   evidence based practitioners consider evidence based librarianship (ebl) a trusted and robust approach to improve professional decisions and service, based on the collection, interpretation and integration of valid, significant and applicable user-reported, librarian-observed and research-derived evidence (eldredge, 2006). since this approach first emerged in the library science literature in 1995 (haines, 1995; booth & brice, 2004), numerous research outputs have been published. however, limited research has been conducted on the characteristics and properties of the ebl research network. in the research field of complex networks and bibliometrics, co-authorship network analysis represents an important area of study. co-authorship networks are a kind of social network which are constructed by connecting two or more authors together if they have co-authored at-least one article. this method has been utilized in a number of fields, such as library and information science (lis) (erfanmanesh & hosseini, 2015; hu, hu, gao, & zhang, 2011; larivière, sugimoto & cronin, 2012; liu, bollen, nelson, & van de sompel, 2005; shu, larivière, mongeon, julien, & piper, 2016; yan, ding, & zhu, 2010), scientometrics (erfanmanesh, rohani, & abrizah, 2012), medicine (gonzalez-alcaide, park, huamaní, gascón, & ramos, 2012), general practitioner research (hong et al., 2016), cardiology (yu, shao, & duan, 2013) and computer science and information systems (bazzan & argenta, 2011; takeda, 2010). the aforementioned researchers have pointed out the functionality and importance of co-authorship networks in the field of complex networks. however, there is still a gap in the literature of librarianship with regard to analysis of scientific collaboration networks, determination of subject clusters, and visualization of the scientific map of ebl. therefore, comprehensive studies are required to understand the characteristics of co-authorship and keyword co-occurrence networks in ebl research. social network analysis (sna), as a process of investigating social structures through the use of networks and graph theory (otte & rousseau, 2002), is particularly suited to this purpose. using sna by visualizing the study of ebl as a scientific map could provide us with a greater understanding of the status of research in this field. the findings of this research could be used to inform future educational policies and research development plans.   literature review   a number of previous studies have investigated social network measures in the area of librarianship and information science. ding, chowdhury, and foo (2001) mapped the intellectual structure of the field of information retrieval during the period of 1987-1997 using co-word analysis. the results suggested the information retrieval field had some established research themes and had also changed rapidly to embrace new themes. erfanmanesh et al. (2012) studied the co-authorship networks of 3024 authors, 1207 institutions, and 68 countries in the field of scientometrics. results revealed that the scientometrician’s collaboration network forms a small-world topology in which authors are typically separated by a short path. zong et al. (2013) visualized the subject clusters of doctoral dissertations in lis in china using co-word analysis. the results revealed 15 clusters with 51 keywords. the highest number of occurrences related to knowledge management. the researchers concluded that topics of lis doctoral dissertations in china were varied, with many immature research fields and a few well-developed, core research fields.   erfanmanesh and hosseini (2015) analyzed international co-authorship networks in lis research across a time span of 50 years (1963-2012). findings showed that the co-authorship network of countries in lis research contained 151 vertices which connected together through 3121 links (co-authorships). moreover, the results of clustering analysis revealed that this network comprised 39 clusters. the greatest density of research was found in those clusters relating to the us and the uk. velmurugan and radhakrishnan (2015) studied the authorship trend and network of papers published in the desidoc journal of library and information technology. results of the study revealed that the degree centrality of authors’ collaboration networks ranged from 0.36 to 0.77. moreover, the average degree of collaboration was found to be 0.5, which indicated a high level of collaboration amongst researchers. shu et al. (2016) studied the evolution of research topics in lis doctoral dissertations in north america during 1960-2013. findings showed that the dominant research interest in doctoral dissertations was in information science rather than library science. however, the mapping of doctoral dissertations’ topic co-assignments revealed a strong relationship between library science and information science. moghadami, hassanzadeh, and shokreian (2016) studied ebl research outputs with quantitative methods in scientometrics. they aimed to identify the most productive authors, countries, institutions and journals in ebl over 30 years. findings showed: the us, england, and canada were leading countries; booth was the most prolific author; and the health information and libraries journal was the core journal in publishing ebl papers (moghadami et al., 2016).   our review showed that whilst mapping and visualization have been investigated in the field of library and information sciences, there is a gap in the use of scientometrics or sna to demonstrate collaboration networks, interdisciplinary relations, and subject clusters in ebl.   aims   this study aimed to investigate the scientific output of ebl using scientometric and sna methods. detailed objectives of the study were to:   ·         visualize the collaboration network of authors in ebl research. ·         visualize the collaboration network of countries in ebl research. ·         visualize subject clusters in ebl research.   methods   the researchers used scientometrics and sna methods to visualize and analyze research collaboration networks and subject clusters in ebl.   we aimed to identify all publications on ebl indexed by scopus and web of science. the selection of appropriate key words was critical to building a robust search. the identified words and phrases were converted into key word validity ratio (kwvr) format. a panel of experts, including five search experts and subject specialists, were invited to vote on the validity of presence of terms in the search strategy, according to a scale of three criteria (essential, related but needs modification, nonessential). the votes were matched with lawshe’s table, which is used for content validity ratio (cvr) (gavgani & vahed, 2017). terms scoring 0.99 or higher were included in the search strategy. the search strategy included the following terms relevant to ebl: evidence based, systematic review, meta-analysis, journal club, librarianship, information practice, library and information sciences, library sciences, and eblip. we limited the search to the title, abstract, and keyword fields and applied no date limitation. we conducted the search on october 1st, 2017. a total of 523 unique publications (398 from scopus and 125 from web of science) were identified which constituted the sample of the study.   to visualize and analyze the collaboration networks, we used the following software tools: ravar premap, netdraw, ucinet and vosviewer. first, bibliographic data were retrieved from the scopus and web of science databases. data cleansing was conducted to detect and remove errors and duplicates. this also included disambiguation of author and country names which were written in different forms (e.g., eldredge j.d. and eldredge j., or uk and united kingdom). to prepare the adjacency matrixes of authors, countries, and subjects, the data were separately imported to the ravar premap software. the ravar premap software generated the matrixes and produced data in .txt format. the ucinet software program was used to convert the data to a format which could be used within netdraw to visualize the collaboration network of authors and countries. moreover, the vosviewer software was used to visualize subject clusters and the co-occurrence analysis of the keywords. to identify the main topics in ebl research outputs, we used a co-word occurrence approach. described by son, jeong, kang, kim, and lee (2015), this approach assumed that terms which frequently co-occur in a body of text tend to be more closely related. this approach is therefore a good indicator of the terminology used within a subject area. in other words, the more frequent the co-occurrence of a pair of words in the literature, the more similar the themes they indicate (hong et al., 2016).   to recognize the co-occurrence of the words and determine subject clusters in ebl, five further tasks were conducted to prepare and cleanse data:   the single and plural forms of keywords were merged,  all abbreviations were written in their full format, ambiguous keywords such as location and date were deleted from the list, synonyms were combined, and the threshold for the frequency of occurrences was agreed as three. as a result, keywords with one or two occurrences were excluded.   we then conducted the sna to describe collaboration and subject cluster networks. we utilized both macro and micro-level metrics. macro-level metrics concentrated on the topology of a network as a whole, with the aim of capturing the overall structure of a network; while micro-level metrics focused on the evaluation of individual actors with the aim of capturing the features of each actor in a network (yan et al., 2010). in this study, one macro-level indicator, namely density, and three micro-level indicators, namely degree centrality, betweenness centrality and closeness centrality, were investigated. the density was scored as a number between zero and one. a number close to one meant that the network was consistent and the relationships between nodes showed greater density. the density indicated direct relations between agents in a network and a high proportion of possible relations (ergün & usluel, 2016). in other words, when the number of links is less than the number of nodes, the density of network is low (scott & carrington, 2011).   results   visualizing the collaboration network of authors in ebl research   the co-authorship network of authors in ebl research was mapped and analyzed. the results showed that the authors’ co-authorship network consists of 73 nodes and 384 links. centrality of the nodes was analyzed using three measures: degree, closeness, and betweenness. in a co-authorship network, the degree centrality of a node showed the total number of co-authorships that a node had with other researchers. in this network, a. booth, with the normalized degree centrality of 1.385, had the highest number of co-authorships, followed by j.d. eldredge (1.293) and a. brice (1.108). a network member with a high degree centrality can be considered to be an active member, indicated by a higher number of partnerships with other members in the network. closeness centrality can be defined as how close an author was on average to all others in the network (hanneman, & riddle, 2005). it is the mean length of all shortest paths from a node to all other nodes in the network. as indicated in table 1, all of the top 30 authors in ebl research had the same normalized closeness centrality (0.099). these authors were closest, or more central, actors of the network, because the sum of their geodesic distances to other actors was among the lowest. another centrality measure that depicted the importance of a particular node was betweenness centrality. the betweenness centrality was defined as the number of shortest paths that pass through a node divided by all shortest paths in the network (guns, liu, & mahbuba, 2011). in regard to normalized betweenness centrality, the most influential authors in the network were a. booth (0.278), j.d. eldredge (0.164) and a. brice (0.116). table 1 shows the most important authors based on centrality measures. additionally, the network had very low density (0.013), which indicated only 1.3% of all possible links being present. this could suggest that the network was still relatively developmental, characterized by limited and weak relationships between nodes.     table 1 top 10 authors in centrality measures rank authors degree centrality closeness centrality betweenness centrality 1 booth a. 1.385 0.099 0.278 2 eldredge j.d. 1.293 0.099 0.164 3 brice a. 1.108 0.099 0.116 4 koufogiannakis d. 0.923 0.099 0.094 5 cleyle s. 0.831 0.099 0.162 6 partridge h. 0.831 0.099 0.099 7 lewis s. 0.739 0.099 0.055 8 hallam g. 0.739 0.099 0.029 9 howlett a. 0.554 0.099 0.024 10 grant m.j. 0.185 0.099 0.012     table 2 top 10 countries in centrality measures rank country degree centrality closeness centrality betweenness centrality 1 uk 53.488 5.250 12.373 2 us 44.186 5.225 5.747 3 germany 39.535 5.212 0.542 4 denmark 37.209 5.206 1.199 5 australia 37.209 5.206 0.945 6 portugal 34.884 5.200 0.313 7 spain 34.884 5.200 0.259 8 netherlands 34.884 5.200 0.259 9 norway 34.884 5.200 0.259 10 canada 25.581 5.174 0.369     figure 1 the co-authorship network of authors in ebl research.     figure 2 the co-authorship network of countries in ebl research.     figure 1 visually depicts the collaboration network of authors in ebl. this network consisted of the nodes and links: nodes represented authors, while links connected vertices in the form of co-authorships.   there was a link between two nodes if they had co-authored at least one paper. the size of a node was proportional to its betweenness centrality. the nodes with high betweenness played a significant role in bridging sub-networks and controlling the flow of information in the network. the ebl network would display greater fragmentation into separate unconnected components without these key brokers with high betweenness centrality (figure 1).   visualizing the collaboration network of countries in ebl research   the co-authorship network between countries in ebl research was also visualized. this network contains 44 nodes (countries) and 268 links (co-authorships) (figure 2). there were 12 isolated nodes such as greece, india, and turkey which represented countries that had not collaborated with others. moreover, three components with only two countries (dyads) existed, including iran and malaysia, israel and mexico, as well as ireland and croatia. table 2 indicates the top 10 countries contributing to the ebl research based on centrality measures. among all countries within the sample, the uk occupied the topmost positions in normalized betweenness centrality (12.373), normalized closeness centrality (5.250), and normalized degree centrality (53.488), which indicated its central role in collaboration network of countries in this field.   the second country with the highest betweenness centrality was the us (5.747), followed by denmark (1.199), australia (0.945), and germany (0.542). these countries featured prominently in the network, appearing on the geodesic paths between other pairs of countries, suggesting a crucial role in knowledge flow in ebl research. countries with the highest normalized degree centrality included the uk (53.488), us (44.186), germany (39.535), denmark (37.209), and australia (37.209). in regard to closeness centrality, the uk was very close to the other nodes, suggesting that the uk had been very successful in establishing collaborations with other countries compared with other countries.   the collaboration network of countries in ebl research is presented in figure 2. in this network, each node stands for a country and there was a link between two nodes if the authors affiliated with those countries had co-authored at least one paper. the size of a node was proportional to its normalized betweenness centrality. the density of the collaboration network of countries was found to be equal to 0.223, which indicated only 22.3% of all possible links being presented and the network had a low overall cohesion.   identifying and visualizing subject clusters in ebl research   co-word analysis is a content analysis technique for discovering the linkages and associations among subjects through the analysis of the co-occurrence frequency of pairs of words or noun phrases (he, 1999, p.134). to map subject clusters, keywords were extracted from the title and keywords sections of papers identified in the search. a total of 1901 keywords were identified. applying a threshold of three word co-occurrences, 134 keywords were subsequently identified. analysis and visualization of keywords was conducted using vosviewer (figure 3).   table 3 shows the top 20 keywords which occurred most frequently, with scores for closeness centrality, betweenness centrality and degree centrality. results showed that “library science”, “evidence based medicine”, “evidence based practice”, “systematic review” and “education” were the most frequent keywords in ebl research.     table 3 top 20 keywords with the highest frequency in ebl research betweenness centrality closeness centrality degree centrality frequency of keyword keyword rank 8.699 90.476 89.474 316 library science 1 5.501 84.177 81.203 194 evidence based medicine 2 4.701 77.778 71.429 148 evidence based practice 3 3.021 70.370 57.895 116 systematic review 4 3.179 76.000 68.421 114 education 5 2.754 73.077 63.158 112 review 6 2.543 76.000 68.421 110 medical science 7 3.162 76.879 69.925 102 information storage and retrieval 8 3.252 75.141 66.917 96 librarians 9 1.044 62.150 39.098 84 meta-analysis 10 0.908 60.455 34.586 72 evidence based librarianship 11 1.484 65.517 47.368 70 research 12 0.604 63.942 43.609 54 medical libraries 13 1.361 68.557 54.135 54 organization and management 14 1.538 67.172 51.128 52 statistics 15 1.654 70.370 57.895 46 health services 16 1.513 66.834 50.376 46 library services 17 0.770 63.033 41.353 46 publication 18 0.608 57.576 26.613 44 academic libraries 19 1.863 67.172 51.128 42 information services 20     figure 3 co-occurrence network of keywords in ebl research.     table 4 subject clusters in ebl research keywords within the cluster number of keywords name of the cluster no. academic libraries, bibliometric, citation analysis, clinical librarianship, collaborative information activities, consumer health information, digital libraries, evidence based librarianship, evidence based practice, health science, health information literacy, health information needs, health librarianship, health science libraries, higher education, information communication technology, information literacy, information practice, information professional, information science, information services, information sources, information systems, information technology, knowledge transfer, knowledge translation, learning, library instruction, library research, library services, management, medical librarianship, professional development, professional development, public libraries, research, school libraries, special libraries, user study 39 library and information science 1 access to information, biomedical research, clinical practice, data mining, decision support systems, financial management, health care delivery, health education, information processing, medical documentation, medical education, medical informatics, medical information, medical information systems, medical research, needs assessment, patient education, patient information, practice guidelines, quality assurance, risk assessment, search engine, university teaching hospitals 23  health and medical  information 2 clinical competence, computer science, consumer satisfaction, continuing education, cooperative behavior, curriculum, education, evidence based nursing, health knowledge, health services, information dissemination, information seeking behavior, internet, nursing, nursing education, nursing research, program development, program evaluation, public relations, quality of health care, research design, teaching 22 nursing 3 clinical effectiveness, clinical research, decision making, drug efficacy, drug safety, health care system, hospitalization, meta-analysis, outcome assessment, patient care, psychology, publication, quality control, quality of life, systematic review, treatment outcome, urology 18 clinical outcome 4 in-service training, library collection development, library science, medical libraries, medical science, organization and management, periodicals, personnel management, professional management, professional standard, staff development 12 management 5 clinical decision support systems, communication, emergency medicine, evidence based medicine, health science, hospital emergency service, hospital libraries, information management, interpersonal communication, knowledge management, national health services, ontology 12 evidence based medicine 6 epidemiology, health care policy, information storage and retrieval, publishing, social science, sociology, statistics, technology 8 information storage and retrieval 7     figure 4 cluster density map of keywords in ebl research.     seven subject clusters were identified as sub-networks in the field of ebl (table 4). cluster analysis was performed to categorize key words appearing in the ebl network and the primary cluster. a cluster is a set of closely related nodes with each node in a network assigned to exactly one cluster (waltman, van eck, & noyons, 2010, p.7). the leading and the biggest cluster, which is shown with red color (figure 3), was formed of 39 keywords (table 4). in this cluster, “evidence based practice” and “evidence based librarianship” were the keywords with the highest number of occurrences. the second cluster (colored in pale green) consisted of 23 terms, in which the terms “medical informatics” and “medical information”, with a frequency of 36, were the most frequently co-occurring terms. the third cluster (colored in dark blue) was comprised of 22 keywords; “education” with a frequency of 114 was found to be the most frequent key word. the fourth cluster (colored in yellow) contains 18 terms; “systematic review” and “review”, with frequencies of 116 and 112 respectively, were the most frequently occurring keywords. the fifth cluster (colored in violet) included 12 terms; “library science”, with a frequency of 316, was the most frequently occurring keyword. the sixth cluster (colored in pale blue) comprised 12 terms, of which “evidence based medicine” with a frequency of 194, was the most prevalent keyword. finally, the seventh cluster (colored in dark green) consisted of 8 keywords, of which “information storage and retrieval” with a frequency of 102, was the most frequently occurring term. to better illustrate the clusters, we named them based on the most leading words based on frequency (table 4).   a density view of the network was visualized to provide a high level overview of the network structure and to highlight areas of particular density (figure 4). in this map, each node had a color that depended on the weight of the vertex in the network, the number of vertices in the neighborhood, and the importance of the neighboring vertices. the highest number of vertices in the neighborhood of a vertex and the higher weights of the vertices were indicated by the color red. conversely, fewest vertices in the neighborhood of a point and the lower weights of the vertices were indicated by the color blue. it can be argued that subjects located in dense areas of the map (shown in red) indicated importance in ebl research (figure 4).   discussion    for this study, the researchers examined the collaboration network of authors and countries as well as subject clusters in ebl research. we used sna to investigate the research collaboration network, allowing for analysis which was both rigorous and highly visual. we analyzed the network at the micro-level, using degree, closeness, and betweenness centrality measures, and at the macro-level, exploring density.   results showed that the share of co-authored papers represented a high percentage of the total number of publications in ebl research. our findings resonated with other studies of research and authorship networks. pecaric and tudman (2017) revealed similar patterns, finding that multiple authorship grew during the three years of the research. our results also revealed that the share of multiple authorship in information science and librarianship was greater than in other disciplines.   the distribution of degree centrality in the collaboration network of ebl research can be said to follow a power-law distribution. as rastogi (2016) described, power law refers to “a relationship between two quantities in a way that, a relative change in one quantity would be reflective of relative change in the other quantity, but it is not necessarily representative of changes in initial value of the quantities” (p.7) and can be used in sna to show changes over time. our findings were consistent with earlier studies in lis, such as erfanmanesh and hosseini (2015) which revealed that a high number of authors with small degrees of centrality and a low number with large degrees of centrality were seen in the network analysis. the co-authorship network of authors in ebl research consisted of 73 nodes and 384 links with 10 authors (see table 1) positioned as central to the collaboration network.   additionally, through our findings we suggest that the ebl co-authorship network was weak in term of density (0.013), with only 1.3% of all possible links being present. this weak density implied the network had limited cohesion with the potential for more relationships between nodes. other studies have found similarly low density in lis authorship networks, including fahimifar and sahli (2015), hariri and nikzad (2011), and soheili & osareh (2014). it can be inferred that the ebl authorship network followed the same characteristics of the broader lis network in terms of density.   regarding the co-authorship network of countries, we found the uk, us, germany, denmark, and australia scored highest in terms of centrality. the uk and the us had the strongest collaborative ties together as well as the highest “betweenness” with other countries in the network, consistent with findings from the broader lis field (erfanmanesh & hosseini, 2015). results showed that the highest number of links within the ebl network were direct with no mediation. according to otte and rousseau (2002), “a co-authorship network is an example of an undirected graph: if author a co-authored an article with author b, automatically author b co-authored an article with a therefore the links (or edges) are bidirectional. an undirected graph can be represented by a symmetrical matrix m = (mij), where mij is equal to 1 if there is an edge between nodes i and j, and mij is equal to 0 if there is no direct link between nodes i and j" (p. 442). this issue was not a favorable sign in analyzing the connections in a network, as it was expected that the nodes in the network facilitated the connections among themselves and empowered the network as well.   this finding was consistent with a former study of iranian lis journals in web of science (soheili & osareh, 2014). those countries with a high betweenness centrality were of particular importance in connecting different nodes and promoting the unity of the network; they occupied a central position in the network and played an important role in knowledge flow. our study revealed the ebl co-authorship network was of low density (0.223). a similarly low density (0.082) was found in erfanmanesh and hosseini’s (2015) study of co-authorship within lis research across countries, indicating only 8.2% of all possible links as present.   we identified seven subject clusters in the ebl research network (see table 4). four out of seven clusters, more than half of all identified subject themes, were related to health, for example: evidence based medicine, education and nursing, clinical outcome, health and medical informatics. this seems to resonate with the results of ankem’s (2008) evaluation of systematic reviews and meta-analyses in lis literature, which found all published systematic reviews in lis at the time related to medical library or medical information settings. exploring lis specialties, chang and lin (2015) identified information systems, information retrieval, and bibliometrics as the dominant themes in a study exploring the broader lis literature. similar findings were evident in studies of dissertations, which can be a useful indicator of research activity within a subject field. zong et al.’s (2013) investigation of chinese doctoral dissertations in lis found the leading topics to be information resource, ontology, semantic web, semantic search, electronic government, information resource management, knowledge management, knowledge innovation, knowledge sharing, knowledge organization, network, information service, information need, and digital library. shu et al.’s (2016) analysis of the us lis doctoral dissertations also revealed a similar shift from traditional library science toward information science. mostafavi, osareh, and  tavakolizadeh-ravari’s (2018) content analysis and sna in the field of knowledge and information science (kis) highlighted the following main subjects clusters: “teaching and learning of kis; information literacy”, “knowledge & information organization”, “web resources and social networks”, “professional ethics in information science”, “informatics, communication and health information services”, “information management; information systems; knowledge management and innovation", and “indicators of informetrics and scientometrics.”  based on these findings, it can be inferred that ebl had stronger roots in the health and medical library community and was less associated with traditional library science subjects like collection development, acquisition, and organization.     conclusion   through this study, the researchers investigated the research outputs of ebl by social network analysis to explore collaboration between authors and between countries and to visualize research topics.   results highlighted a core role for the uk and us in the international ebl research network, both playing a leading role in research and engaging in collaboration with authors around the world. the study also revealed a number of countries with limited relationships within the network, suggesting an opportunity to develop their research collaboration with the leaders or other countries in the main network. based on the findings of this study and results of previous research, we concluded that ebl is more deeply rooted within health and medical settings,   our findings provided evidence about the current status of ebl research around the world, thus helping researchers and policy makers better understand the nature of ebl research and predict the future dimension of research.   acknowledgements and appreciations   this study was ethically approved and financially supported by the tabriz university of medical sciences, iran and the ethical committee of research department, with the reference number: ir-tbzmed-rec-1395-1262. we are grateful to dr. mohammad tavakkolizadeh, designer and developer of ravar premap software, for his instructions and guidances on using the ravar premap.       references   ankem k. (2008). evaluation of method in systematic reviews and meta-analyses published in lis. library and information research, 32(101), 91-104. https://doi.org/10.1.1.870.3067   bazzan, a. l., & argenta, v. f. (2011). network of collaboration among pc members of brazilian computer science conferences. journal of the brazilian computer society, 17(2), 133-139. https://doi.org/10.1007/s13173-011-0033-7   booth, a., & brice, a. (2004). evidence-based practice for information professionals: a handbook. london: facet publishing.   chang, y. w., huang, m. h., & lin, c. w. (2015). evolution of research subjects in library and information science based on keyword, bibliographical coupling, and co-citation analyses. scientometrics, 105(3), 2071-2087. https://doi.org/10.1007/s11192-015-1762-8   ding, y., chowdhury, g., & foo, s. (2001). bibliometric cartography of information retrieval research by using co-word analysis. information processing & management, 37(6), 817-842. https://doi.org/10.1016/s0306-4573(00)00051-0   eldredge, j. d. (2006). evidence-based librarianship: the ebl process. library hi tech, 24(3), 341-354. https://doi.org/10.1108/07378830610692118   erfanmanesh, m., & hosseini, e. (2015). cross-time analysis of countries co-authorship networks in library & information science research. library philosophy and practice. retrieved from http://digitalcommons.unl.edu/libphilprac/1316/   erfanmanesh, m., rohani, v. a., & abrizah, a. (2012). co-authorship network of scientometrics research collaboration. malaysian journal of library & information science, 17(3), 73-93. retrieved from http://ejum.fsktm.um.edu.my/article/1298.pdf   ergün, e., & usluel, y. k. (2016). an analysis of density and degree-centrality according to the social networking structure formed in an online learning environment. journal of educational technology & society, 19(4), 34-46. retrieved from https://www.jstor.org/stable/pdf/jeductechsoci.19.4.34.pdf   fahimifar, s., & sahli, f. (2015). investigation of the co-authorship network of farsi scientific journals in the field of information science. research on information science and public libraries, 21(1), 127-151.   gavgani, v. z., & vahed, n. (2017). 184: keyword validity ratio (kwvr); a tool for validity of keywords for building an accurate literature search strategy. bmj open, 7. https://doi.org/10.1136/bmjopen-2016-015415.184   gonzalez-alcaide, g., park, j., huamaní, c., gascón, j., & ramos, j. m. (2012). scientific authorships and collaboration network analysis on chagas disease: papers indexed in pubmed (1940-2009). revista do instituto de medicina tropical de são paulo, 54(4), 219-228. https://doi.org/10.1590/s0036-46652012000400007   guns, r., liu, y. x., & mahbuba, d. (2011). q-measures and betweenness centrality in a collaboration network: a case study of the field of informetrics. scientometrics, 87(1), 133-147. https://doi.org/10.1007/s11192-010-0332-3   haines, m. (1995). librarians and evidence-based purchasing. evidence-based purchasing, 8(1).   hanneman, r., & riddle, m. (2005). introduction to social network analysis. riverside ca: university of california. (published in digital form at http://faculty.ucr.edu/~hanneman/.   hariri, n., & nikzad, m. (2011). co-authorship networks of iranian articles in library and information science, psychology, management and economics in isi during 2000-2009. iranian journal of information processing management, 26(4), 825-844. retrieved from http://jipm.irandoc.ac.ir/   he, q. (1999). knowledge discovery through co-word analysis. library trends, 48(1), 133-159. retrieved from: https://www.ideals.illinois.edu/bitstream/handle/2142/8267/librarytrendsv48i1i_opt.pdf?sequence=1   hong, y., yao, q., yang, y., feng, j. j., ji, w. x., yao, l., & liu, z. y. (2016). knowledge structure and theme trends analysis on general practitioner research: a co-word perspective. bmc family practice, 17(1), 10. https://doi.org/10.1186/s12875-016-0403-5   hu, c. p., hu, j. m., gao, y., & zhang, y. k. (2011). a journal co-citation analysis of library and information science in china. scientometrics, 86(3), 657-670. https://doi.org/10.1007/s11192-010-0313-6   larivière, v., sugimoto, c. r., & cronin, b. (2012). a bibliometric chronicling of library and information science's first hundred years. journal of the association for information science and technology, 63(5), 997-1016. https://doi.org/10.1002/asi.22645   liu, x., bollen, j., nelson, m. l., & van de sompel, h. (2005). co-authorship networks in the digital library research community. information processing & management, 41(6), 1462-1480. https://doi.org/10.1016/j.ipm.2005.03.012   moghadami, m., hassanzadeh, m., & shokreian, s. (2016). thirty years of evidence-based librarianship (ebl): citation analysis and review. collnet journal of scientometrics and information management, 10(2), 209-218. https://doi.org/10.1080/09737766.2016.1213965   mostafavi, i., osareh, f., & tavakolizadeh-ravari, m. (2018). identifying content structure of “knowledge and information science (kis)” studies based on co-word analysis of articles in “web of science (wos)” database (2009-2013). iranian journal of information processing and management, 33(3), 1285-1314. retrieved from http://jipm.irandoc.ac.ir/   otte, e., & rousseau, r. (2002). social network analysis: a powerful strategy, also for the information sciences. journal of information science, 28(6), 441-453. https://doi.org/10.1177/016555150202800601   rastogi, t. (2016). a power law approach to estimating fake social network accounts. retrieved from https://arxiv.org/abs/1605.07984   pecaric, d., & tudman, m. (2017). co-authorship networks of scientific elite: case study of information science in croatia. qualitative and quantitative methods in libraries, 5(1), 49-59. retrieved from http://www.qqml-journal.net/index.php/qqml/article/view/28   scott, j., & carrington, p. j. (2011). the sage handbook of social network analysis. london, u.k.: sage publications.   shu, f., larivière, v., mongeon, p., julien, c. a., & piper, a. (2016). on the evolution of library and information science doctoral dissertation topics in north america (1960-2013). journal of education for library & information science, 57(2), 131-142.  retrieved from https://utpjournals.press/doi/abs/10.3138/jelis.57.2.131   soheili, f., & osareh, f. (2014). a survey on density and size of co-authorship networks in information science journals. iranian journal of information processing management, 29(2), 351-372. retrieved from http://jipm.irandoc.ac.ir/   son, y. j., jeong, s., kang, b. g., kim, s. h., & lee, s. k. (2015). visualization of e-health research topics and current trends using social network analysis. telemedicine journal and e-health, 21(5), 436-442. https://doi.org/10.1089/tmj.2014.0172   takeda, h. (2010). a social network analysis of the is field: a co-authorship network study. proceedings of the southern association for information systems conference, 25. retrieved from https://aisel.aisnet.org/   velmurugan, c., & radhakrishnan, n. (2015). scientometric observations of authorship trends and collaborative research on desidoc journal of library and information technology. collnet journal of scientometrics & information management, 9(2), 193-204. https://doi.org/10.1080/09737766.2015.1069957   yan, e., ding, y., & zhu, q. (2010). mapping library and information science in china: a coauthorship network analysis. scientometrics, 83(1), 115-131. https://doi.org/10.1007/s11192-009-0027-9   yu, q., shao, h., & duan, z. (2013). the research collaboration in chinese cardiology and cardiovasology field. international journal of cardiology, 167(3), 786-791. https://doi.org/10.1016/j.ijcard.2012.03.019   zong, q. j., shen, h. z., yuan, q. j., hu, x. w., hou, z. p. & deng, s. g. (2013). doctoral dissertations of library and information science in china: a co-word analysis. scientometrics, 94(2), 781-799. https://doi.org/10.1007/s11192-012-0799-1     microsoft word es_needham.doc evidence based library and information practice 2006, 1:2      30 evidence based library and information practice       evidence summary    a web‐based tutorial may produce the same cognitive outcomes as face to face  instruction   a review of:  beile, penny m. and david n. boote. “does the medium matter?: a comparison of a web‐based  tutorial with face‐to‐face library instruction on education students’ self‐efficacy  levels and learning outcomes.”  research strategies 20 (2004): 57‐68.    reviewed by:  gill needham  head of strategic and service development, the open university library and learning  resources centre  walton hall, milton keynes, united kingdom  e‐mail: g.needham@open.ac.uk    received: 28 february 2006  accepted: 13 april 2006      © 2006 needham. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to determine whether library  skills self‐efficacy levels and learning  outcomes of postgraduate education  students varied with different instructional  delivery methods, specifically web‐based or  face to face.    design – pre‐ and post‐intervention survey  comparing three groups receiving different  types of instruction.    setting – department of educational  studies at a large u.s. urban university.    subjects – forty‐nine masters, doctoral, and  certificate‐seeking education students  enrolled in one of three sections of a  research methods course. there were 40  female and 9 male students.    methods – immediately before receiving  library instruction, the three student groups  were asked to complete a library skills self‐ efficacy questionnaire, comprising 30 items  designed to measure students’ perceptions  of their ability to successfully perform  library research. they also completed a  library skills test, consisting of 20 multiple  choice questions, designed to assess  conceptual knowledge, knowledge of  database searching, and institution‐specific  knowledge.  the intervention groups were:    • group 1 (sixteen students)  –  an on‐ campus class that received a face to  face instruction session comprised  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:2      31 of a 70‐minute demonstration of key  library databases followed by an  activity that allowed students to  practice their skills.    • group 2 (nineteen students) – an  on‐campus class that received a  web‐based tutorial comprised of  four interactive modules, requiring  an average 80 minutes to complete.     • group 3 (nineteen students) ‐ a  web‐based class that received the  same web‐based tutorial as group 2.    the survey and test were repeated six weeks  after the instruction.    main results – both self‐efficacy scores and  library skills test scores increased for all  three groups post‐intervention. average  self‐efficacy levels increased from a mean of  68.88 (sd=19.92) to a mean of 91.90  (sd=16.24); library skills scores increased  from an average score of 58.78 (sd=13.80) to  an average of 73.16 (sd=12.65). there was  no statistically significant difference  between the post‐ intervention scores of the  three groups on the library skills test.   however, the web‐based students in group  3 showed a statistically significant greater  increase in self‐efficacy score (78.86 to 102.36)  when compared with group 2 participants  (64.74 to 83.68).    conclusion – the study provides evidence  that library instruction is effective in  increasing both skill levels and self‐efficacy  levels. it does not give a clear indication of  the relative value of different modes of  delivery, but it does support the contention  that web‐based tutorials are at least as  effective as face to face sessions.      commentary    this study is ambitious in attempting to  compare three different combinations of  settings and methods for delivering  information skills instruction.  the three  groups are fairly small (16, 19, and 14  participants respectively), and there is no  way to determine the extent to which they  are representative of the population of  postgraduate educational studies students.  participants were not randomly assigned to  the three groups. participants in group 1  and group 2 had chosen to attend a campus  class, and members of group 3 had selected  a web‐based class.     the study does not provide details as to  how students were allocated between  group 1 (campus class with face to face  instruction) and group 2 (campus class with  web‐based instruction).  group 3 students  were however self‐selected, having chosen  to receive web‐based instruction, which  suggests that they may have been more  familiar with that mode of delivery. it  would therefore be potentially misleading to  conclude from the results that web‐based  tutorials are more effective in a web‐based  classroom than in a campus classroom.  another issue is that a large number of  statistical procedures are reported in the  findings without a great deal of explanation  for those with limited knowledge of  statistics.    a major strength of the study is that the  post‐intervention tests were conducted six  weeks after the library instruction. we do,  however, know very little about the two  instruments themselves; it would have been  helpful to include them as an appendix.  the  authors themselves draw attention to the  fact that neither instrument had been  validated with the sample population,  although they had been tested in advance  with a different sample. the authors also  note that the multiple choice quiz used to  test knowledge and skills may be of limited  value and suggest that empirical  observation of performance and/or  analyzing the quality of references in a  bibliography might yield more authentic  outcomes. it would also have been helpful,  from a practitioner perspective, to have  more information about the content and  evidence based library and information practice 2006, 1:2      32 delivery of the face to face instruction. the  brief description provided (“a 70 minute  demonstration of relevant library databases,  followed by an activity to allow the students  to immediately apply the lesson” 61)  suggests a far less engaging student  experience than the interactive web‐based  tutorial (<http://library.ucf.edu/cmc/edtut>),  but this may underestimate the quality of  the face to face teaching. one of the  potential weaknesses of studies of this kind  is that there is rarely any attempt to  independently assess the comparability of  the quality of the educational interventions,  in order to isolate the influence of the mode  of delivery.                                                                      this study is helpful in that it serves to  support the hypothesis that library  instruction has a positive influence on both  library skills self‐efficacy levels and learning  outcomes.  it also suggests that web‐based  tutorials are at least as effective as face to  face teaching sessions and that these may be  successfully delivered either in the  classroom or via the web.        http://library.ucf.edu/cmc/edtut review article   wayfinding research in library and information studies: state of the field   lauren h. mandel assistant professor graduate school of library and information studies university of rhode island kingston, rhode island, united states of america email: lauren_mandel@uri.edu   received: 1 nov. 2016    accepted: 3 apr. 2017      2017 mandel. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – often people enter libraries focused on their primary information needs and haven't considered their need for spatial information to find their way to what they need. this presents unique wayfinding information challenges for libraries. papers on library wayfinding often include some discussion of the lack of wayfinding research in libraries, but apparently there has been no comprehensive review of the lis literature on wayfinding.    methods – this paper is a comprehensive review of library wayfinding literature, using the library, information science & technology abstracts with full text (via ebscohost) database to collect the dataset.   results – findings indicate a small collection of library wayfinding research, primarily focused on academic libraries.   conclusion – empirical research in this area is limited. suggestions for future research on library wayfinding, including potential foci for that research, are presented.   introduction   we live in a physical world, and in order to navigate it successfully, we need wayfinding skills. this is true whether one is navigating a forest, city, college campus, or a complex building, such as a library. wayfinding is the process by which humans orient themselves in and navigate a space. wayfinding can be facilitated through the provision of spatial information that guides people in orienting and navigating a space. in many ways, libraries provide a unique challenge for wayfinding. when patrons walk into a library with a need for information, they need to orient themselves to the library space, navigate to a source of assistance (person, catalogue, and others), and then navigate to the resource(s) that fulfills their information need. so patrons have both their unique information needs and wayfinding information needs, the latter of which patrons are less likely to acknowledge consciously.   while there is a large focus in the lis literature on understanding information needs and how to help patrons solve those needs, there is considerably less focus on the wayfinding information that patrons need.  but how much less focus?  it is common for people writing on wayfinding in libraries to lament the dearth of literature on the subject, but there has been no recent comprehensive review of the lis research literature on wayfinding. without that, it is difficult to ascertain where the field of library wayfinding is, what research is being done, in which types of libraries, and what work remains to be undertaken. in short, what is the state of the field of library wayfinding research?  this paper reviews library wayfinding research literature to address that question.   aims   conducting a literature review as a way to assess the state of a field is not uncommon. three such studies have been completed in a longitudinal series on the state of information behaviour research (julien, 1996; julien & duggan, 2000; julien, pecoskie, & reed, 2011). others have looked at topics such as use of geographic information systems (gis) in lis research (bishop & mandel, 2010), emotions in human-computer interaction (hci) (lopatovska & arapakis, 2011), emotions in librarianship (matteson & miller, 2012), and libraries’ use of open access software (palmer & choi, 2014). as yet, there is apparently no such review of literature on wayfinding in lis or wayfinding in libraries.   how well people are able to find their way in libraries has an impact on their ability to successfully use library facilities to fulfill information needs. wayfinding can be guided through spatial information systems, including architectural legibility, signage, and people. anecdotal evidence indicates there is a growing body of literature on wayfinding issues in libraries. a literature review of the research area would allow identification of trends, topics of interest, commonly used research methods, and assessment of “the scholarly maturity of the area” (julien et al., 2011, p. 19).   the literature review reported here is guided by three research questions. rq1: what is being published in lis journals about wayfinding?  rq2: how much of this is about wayfinding in libraries vs. wayfinding elsewhere?  rq3: for articles that are about wayfinding in libraries, how many are research, and in which library types?   literature review   research on wayfinding in the built environment is conducted in a wide variety of settings, including both cities and buildings. as the focus of this paper is on library wayfinding, this literature review is limited to discussion of wayfinding in buildings. best’s (1970) pioneering research occurred in a municipal facility, manchester (uk) town hall. later work has often focused on medical facilities (baskaya, wilson, & özcan, 2004; kaya, ileri, & yuceler, 2016; morag, heylighen, & pintelon, 2016), transportation facilities (hafiz & zohdy, 2016; shiwakoti, tay, stasinopoulos, & woolley, 2016), and shopping malls (chebat, gélinas-chebat, & therrien, 2005; dogu & erkip, 2000; tüzün, telli, & alır, 2016). there has also been work in educational facilities other than libraries (kanakri, schott, mitchell, mohammad, etters, & palme, 2016; stoffell, schoder, & ohlbach, 2008). more recently, there is increased focus on wayfinding for people with varying degrees of physical limitations, including cognitively impaired persons (davis & ohman, 2016; torrado, montoro, & gomez, 2016), visually impaired persons (lee, li, & lin, 2015; nguyen, vu, tran, & nguyen, 2017), and even firefighters whose senses are impaired when operating in a smoke-filled building (hsiao, tang, huang, & lin, 2016).   these studies employ a variety of methodologies, including spatial analysis (dogu & erkip, 2000), interviews (best, 1970), surveys/questionnaires (baskaya et al., 2004; dogu & erkip, 2000; kaya et al., 2016; morag et al., 2016; shiwakoti et al., 2016), and sketch mapping (baskaya et al., 2004). one of the most commonly used methods is a task completion experiment, employed by best (1970), chebat et al. (2005), lee et al. (2015), davis and ohman (2016), hsiao et al. (2016), kanakri et al. (2016), nguyen et al. (2017), torrado et al. (2016), and tüzün et al. (2016). in these experiments, participants are given predefined tasks to complete. while they complete the tasks, they might be video recorded, audio recorded, accompanied by a note-taker, asked to think aloud, or otherwise observed. most of these experiments are conducted in physical settings, but some have been in virtual reality simulations (davis & ohman, 2016; tang, wu, & lin, 2009).   papers on library wayfinding cover a variety of topics, from overview of the literature and work in the field (beck, 1996) to informal assessment of library facilities for their wayfinding ease (dempsey, 2006) to formal assessment and research on library wayfinding, which sometimes take the form of institutional reports that are not published in the peer-reviewed literature like the burke library wayfinding study (baker, bakkalbasi, call, & kamsler, 2015). most research seems to be reported in peer-reviewed journals. some comes from dissertations (beecher, 2004; mandel, 2012), which are not indexed in library, information science & technology abstracts with full text (lista).   there is documented need for research on wayfinding in libraries given the challenge of wayfinding in complex buildings like libraries and the concerns over library anxiety (eaton, 1991; eaton, vocino, & taylor, 1992; hahn & zitron, 2011; li & klippel, 2012; mandel, 2013; schoonover & kinsley, 2014). libraries have been called mazes (li & klippel, 2012) and labyrinths (mandel, 2012; zaugg et al., 2016), both terms suggesting the complexity of library facility layouts. all library users live in a physical world and need to navigate the physical library space to solve their information problems. this problem has been noted by eaton (1991), hahn and zitron (2011), mandel (2013), schoonover and kinsley (2014), zaugg and child (2016), and zaugg et al. (2016).   method   following the models of prior literature reviews (bishop & mandel, 2010; julien, 1996; julien & duggan, 2000; julien et al., 2011), this study focuses on literature indexed in a specific database, in this case lista (via ebscohost). lista indexes over 600 journals, as well as research reports, conference proceedings, and books and is considered a standard lis database. lista and library literature & information science retrospective are the only lis-specific databases available at the author’s institutional library. a search of library literature & information science retrospective for wayfind* in all text returned only 30 records. two records were for the book sign systems for libraries: solving the wayfinding problem, a signage guidelines text published in 1979 that doesn’t report research. twenty records were chapters in the book, and the remaining eight records were reviews of the book. given this finding in library literature & information science retrospective, the research reported here focused entirely on the literature indexed in lista. the database was searched on august 17, 2016, for wayfind* in all text, with results limited to peer-reviewed items. this search returned 136 papers.   all citations were downloaded, and then each article was coded using content analysis. julien et al. (2011) noted the value of content analysis for a literature review as being systematic, rigorous, and flexible. in this case, the full content of articles was assessed to make the most accurate determination of the degree to which the article was about wayfinding research in lis.   coding occurred in a microsoft excel spreadsheet. first each article was assessed as to whether it was about wayfinding or not. for those articles about wayfinding, they were further coded for whether they reported research or not and which library type was researched (academic, public, school, other, any/all types, and not library). articles that were not about wayfinding were coded for whether they were about something else, but mentioned wayfinding once or twice, or whether they cited a source that was about wayfinding. the following types of articles were removed from the dataset: duplicates; reviews of books, articles, and digital applications; and editorials that listed the articles appearing in the issue of a journal. after these removals, the dataset comprised 103 items.   results   rq1: what is being published in lis journals about wayfinding?   less than half of the 103 articles in the dataset were found to be actually about wayfinding (n=43; 41.7%). the largest group comprised articles about something else that mentioned wayfinding once or twice (n=46; 44.7%), and a few articles cited a source that had wayfinding in its title (n=14; 13.6%). the majority of the articles that were about wayfinding were research-based (n=39; 90.7% of articles about wayfinding and 37.9% of all articles in the dataset).   the articles that were not about wayfinding but mentioned it a couple of times covered a wide range of topics. some focused on library space (n=9), and the same number focused on information architecture. other topics of papers that mentioned wayfinding once or twice were cataloguing and classification (n=2), collection management (n=1), service orientation (n=1), graphic design (n=1), historical collections (n=2), information kiosks and electronic signage (n=2), information seeking and retrieval (n=4), map collections (n=1), marketing (n=1), mobile devices and mobile apps (n=4), qr codes as information tools (n=2), reference services (n=4), sense-making (n=1), and spatial literacy (n=2).   a wide array of journals is covering wayfinding to some extent, with 60 journals represented in the entire body of 103 articles. less than a quarter of these journals included articles actually about wayfinding (n=24; 23.3%). behavior & information technology (n=5) and information design journal (n=4) included the most articles about wayfinding with code4lib, information -wissenschaft und praxis, and library and information science research each including three articles on wayfinding. together, these five journals included 34.9% of all the articles on wayfinding (n=15). see table 1 for a breakdown of each journal’s coverage of wayfinding.     table 1 wayfinding coverage in lis journals by journal title journal title # articles about wayfinding behaviour & information technology 5 information design journal 4 code4lib journal 3 information - wissenschaft und praxis 3 library and information science research 3 journal of the association for information science & technology 2 journal of academic librarianship 2 journal of access services 2 public services quarterly 2 reference services review 2 universal access in the information society 2 cataloging & classification quarterly 1 collection management 1 evidence based library & information practice 1 information processing & management 1 journal of map & geography libraries 1 journal of web librarianship 1 library hi tech 1 oclc systems & services 1 performance measurement & metrics 1 reference librarian 1 reference quarterly 1 reference & user services quarterly 1 school libraries worldwide 1     rq2: how much of this is about wayfinding in libraries vs. wayfinding elsewhere?   the majority of articles about wayfinding indexed in lista (not including those articles that mentioned it once or twice or cited a source with wayfinding in the title) did focus on wayfinding in libraries. about 40% of the articles about wayfinding were about non-library sites (n=17; 39.5%), with the rest about wayfinding in academic, public, school, and other libraries. see figure 1 for a visual representation of the breakdown of wayfinding articles by not-library and library types.   wayfinding in academic libraries comprised the same number of articles as wayfinding in non-library sites (n=17 each; 39.5%). very few articles reported on wayfinding in public (n=2; 4.7%) or school libraries (n=2; 4.7%), a few articles reported on wayfinding in any or all types of libraries (n=4; 9.3%), and one article reported on wayfinding in a digital library (2.3%). the non-library sites included city streets (n=10), digital and virtual environments (n=3), other types of complex facilities (n=3) such as hospitals and airports, and a university campus (n=1).     figure 1   wayfinding articles in lista by institutional focus.     rq3: for articles that are about wayfinding in libraries, how many are research, and in which library types?   of the articles actually about wayfinding, the vast majority reported research (n=39; 90.7%). four articles were about something else related to wayfinding. there was one article each on the development of a wayfinding app for libraries (hahn, 2011), the development of a navigation system for cities (allard, 2008), design principles (fendley, 2009), and wayfinding in the digital age (fox, 2015).   twenty-four articles (23.3% of all 103 articles) reported wayfinding research in libraries. see table 2 for a list of articles on wayfinding research in libraries by library type. the majority of this library wayfinding research comes from academic libraries (n=17; 70.8% of articles reporting wayfinding research in libraries). the research on academic library wayfinding covers the following topics:   ·         development of a wayfinding system (agarwal & lawrence, 2014; orphanides, 2011); ·         reference transactions that include wayfinding questions (bishop, 2012); ·         signage assessment (eaton et al., 1992; polger & stempler, 2014; stempler, 2013; stempler & polger, 2013); ·         app development (hahn & morales, 2011; hahn & ryckman, 2012; hahn, ryckman, & lux, 2015; hahn, twidale, gutierrez, & farivar, 2011); ·         assessing user wayfinding (hahn & zitron, 2011; kinsley, schoonover, & spitler, 2016; li & klippel, 2012; schoonover & kinsley, 2014; zaugg et al., 2016); and ·         wayfinding as a component of a marketing and assessment project (porat, 2013).   as compared to wayfinding research in academic libraries, there is less wayfinding research in lista about any or all library types (n=2; 8.3%), public (n=2; 8.3%), school (n=2; 8.3%), or digital (n=1; 4.2%) libraries. both public library wayfinding research articles (mandel, 2010, 2013), both school library wayfinding research articles (eaton, 1991; johnston & mandel, 2014), and the digital library wayfinding research article (kerr, 1990) are about assessing user wayfinding. one of the articles about wayfinding research in any/all library types is a comparison of existing research on wayfinding, familiarity, and sketch maps to give libraries a way to assess patron familiarity with and anxiety about the facility (horan, 1999). the other is about wayfinding as a problem-solving strategy for navigating cataloguing rules (normore, 2012).     table 2 articles on wayfinding research in libraries by library type library type citations academic agarwal, n. k., & lawrence, h. (2014). office location map of individuals in the library and other college campus buildings: a proof-of-concept wayfinding system. journal of web librarianship, 8(3), 305-323. http://dx.doi.org/10.1080/19322909.2014.927744 bishop, b. w. (2012). analysis of reference transactions to inform library applications (apps). library & information science research, 34(4), 265-270. http://dx.doi.org/10.1016/j.lisr.2012.06.001 eaton, g., vocino, m. & taylor, m. (1992). evaluating signs in a university library. collection management, 16(3), 81-102. http://dx.doi.org.uri.idm.oclc.org/10.1300/j105v16n03_06 hahn, j., & morales, a. (2011). rapid prototyping a collections-based mobile wayfinding application. journal of academic librarianship, 37(5), 416-422. http://dx.doi.org/10.1016/j.acalib.2011.06.001 hahn, j., & ryckman, n. (2012). modular mobile application design. code4lib journal, (18), n.p. retrieved from http://journal.code4lib.org hahn, j., ryckman, b., & lux, m. (2015). topic space: rapid prototyping a mobile augmented reality recommendation app. code4lib journal, (30), n.p. retrieved from http://journal.code4lib.org hahn, j., twidale, m., gutierrez, a., & farivar, r. (2011). methods for applied mobile digital library research: a framework for extensible wayfinding systems. reference librarian, 52(1/2), 106-116. http://dx.doi.org/10.1080/02763877.2011.527600 hahn, j., & zitron, l. (2011). how first-year students navigate the stacks: implications for improving wayfinding. reference & user services quarterly, 51(1), 28-35. retrieved from https://journals.ala.org/index.php/rusq kinsley, k. m., schoonover, d., & spitler, j. (2016). gopro as an ethnographic tool: a wayfinding study in an academic library. journal of access services, 13(1), 7-23. http://dx.doi.org/10.1080/15367967.2016.1154465 li, r., & klippel, a. (2012). wayfinding in libraries: can problems be predicted? journal of map & geography libraries, 8(1), 21-38. http://dx.doi.org/10.1080/15420353.2011.622456 orphanides, a. k. (2011). lessons in public touchscreen development. code4lib journal, (15), n.p. retrieved from http://journal.code4lib.org polger, m. a., & stempler, a. f. (2014). out with the old, in with the new: best practices for replacing library signage. public services quarterly, 10(2), 67-95. http://dx.doi.org/10.1080/15228959.2014.904210 porat, l. (2013). marketing and assessment in academic libraries: a marriage of convenience or true love? evidence based library & information practice, 8(2), 60-67. http://dx.doi.org/10.18438/b8fs5m schoonover, d., & kinsley, k. m. (2014). stories from the stacks: students lost in the labyrinth. journal of access services, 11(3), 175-188. http://dx.doi.org/10.1080/15367967.2014.914426 stempler, a. f. (2013). navigating circular library stacks: a case study on signage. reference services review, 41(3), 503-513. http://dx.doi.org/10.1108/rsr-02-2013-0006 stempler, a. f., & polger, m. a. (2013). do you see the signs? evaluating language, branding, and design in a library signage audit. public services quarterly, 9(2), 121-135. http://dx.doi.org/10.1080/15228959.2013.785881 zaugg, h., child, c., bennett, d., brown, j., alcaraz, m., allred, a., & ... lee, s. (2016). comparing library wayfinding among novices and experts. performance measurement & metrics, 17(1), 70-82. http://dx.doi.org/10.1108/pmm-12-2015-0041 public mandel, l. h. (2010). toward an understanding of library patron wayfinding: observing patrons' entry routes in a public library. library & information science research, 32(2), 116-130. http://dx.doi.org/10.1016/j.lisr.2009.12.004 mandel, l. h. (2013). finding their way: how public library users wayfind. library & information science research, 35(4), 264-271. http://dx.doi.org/10.1016/j.lisr.2013.04.003   school eaton, g. (1991). wayfinding in the library: book searches and route uncertainty. rq, 30(4), 519-527. johnston, m. p., & mandel, l. h. (2014). are we leaving them lost in the woods with no breadcrumbs to follow? assessing signage systems in school libraries. school libraries worldwide, 20(2), 38-53. retrieved from http://www.iasl-online.org/publications/slw/index.html digital kerr, s. t. (1990). wayfinding in an electronic database: the relative importance of navigational cues vs. mental models. information processing & management, 26(4), 511-523. https://doi.org/10.1016/0306-4573(90)90071-9 any/all horan, m. (1999). what students see: sketch maps as tools for assessing knowledge of libraries. journal of academic librarianship, 25(3), 187-201. https://doi.org/10.1016/s0099-1333(99)80198-0 normore, l. f. (2012). “here be dragons”: a wayfinding approach to teaching cataloguing. cataloging & classification quarterly, 50(2/3), 172-188. http://dx.doi.org/10.1080/01639374.2011.651192     discussion: implications for future research on library wayfinding   the papers found in lista that report research in libraries all suggest a need for more wayfinding research to be conducted in libraries. there is great concern about how to make the physical spaces of libraries more navigable for users (agarwal & lawrence, 2014; eaton, 1991; eaton et al., 1992; hahn & zitron, 2011; hahn et al., 2015; hahn et al., 2011; johnston & mandel, 2014; kinsley et al., 2016; li & klippel, 2012; mandel, 2010, 2013; orphanides, 2011; schoonover & kinsley, 2014; stempler, 2013; zaugg et al., 2016). this is especially true in places where staffing is limited. this can be a physical location that poses a staffing challenge, with stempler (2013) noting that “open stacks are traditionally not staffed, therefore efficient signage is vital to ensure these collections are accessible to users” (p. 503). or it can be more general, based on either users’ preferences for self-sufficiency or the necessity of self-sufficiency brought on by reduced staffing. orphanides’ (2011) development of a self-service kiosk came out of a confluence of factors, including “a perceived need for improved user access to unmediated wayfinding information” (motivations, item 1).   many arguments focus on the challenge of wayfinding in library spaces. there are general statements like “wayfinding in libraries can be a difficult and challenging task” (eaton et al., 1992, p. 81) and “research shows that academic libraries can be difficult to navigate and that students are often frustrated with not being able to find the right materials” (schoonover & kinsley, 2014, p. 175). there are also more critical statements about specific spaces, such as “anecdotally referred to as a maze, the interior spaces of the main library on the authors’ university campus […] bring a challenge for patrons to navigate” (li & klippel, 2012, p. 22).   but it’s not just that wayfinding is a challenge, it is also that service provision is a central tenet of librarianship and wayfinding is a means to improve provision of library services. hahn and zitron (2011) asserted, “providing assistance in the search for books and other resources in the physical library space is a foundational library service. this search can be fostered or hindered by library layout” (p. 28). zaugg et al. (2016) said that “wayfinding tools are the means to meet ranganathan’s (1931) third and fourth laws of library science, namely, to help patrons find their book (or needed service) and to save the time of the reader” (p. 70).   if wayfinding is so important to the provision of library services, then why is it such a small area of focus in lis research?  out of tens of thousands of articles indexed in lista, only 103 include the keyword wayfind*. this shows wayfinding to be a small subset of the lis literature, despite the known importance of user services for librarianship. there is more work being done on wayfinding than what appears in lista, including dissertations, such as beecher (2004) and mandel (2012), papers published in journals not indexed in lista, and institutional reports like the burke library wayfinding study report (baker et al., 2015). but, it does seem like wayfinding is not experiencing the level of research focus that its importance demands.   a search of information behaviour research indexed in lista for 1999-2008 found 749 articles (julien et al., 2011). as compared to this study, which found 103 articles over the entire time period indexed in lista, that is seven times as many articles in a much shorter time period. effective wayfinding requires spatial information, yet it receives very little research attention in lis compared to information behaviour, a central topic in the field (bawden & robinson, 2012). and, despite the focus on development of models and frameworks of information behaviour, there is hardly any work on theoretical development for library wayfinding with most studies forgoing a theoretical model entirely.   the research indexed in lista includes calls for more wayfinding research on a variety of topics. some call for further development and testing of wayfinding tools (agarwal & lawrence, 2014; hahn & ryckman, 2012; hahn & zitron, 2011; hahn et al., 2015; oprhanides, 2011), as well as development of best practices (mandel, 2013; stempler & polger, 2013). others recommend modifications to existing tools with follow-up research to assess the new tools (eaton et al., 1992). there are calls for replication studies (kinsley et al., 2016; mandel, 2010; polger & stempler, 2013; schoonover & kinsely, 2014) and expansion of research methods used in wayfinding research (johnston & mandel, 2014; li & klippel, 2012; mandel, 2010; schoonover & kinsely, 2014; stempler, 2013). there are even calls for tools that facilitate serendipitous information seeking behaviour (hahn & morales, 2011) and research to develop theories related to library user wayfinding (mandel, 2013).   there is need for wayfinding research in libraries   libraries are physical spaces that users must navigate in order to satisfy their information needs. over 20 years ago, there was a call for wayfinding research in libraries (eaton, 1991), and there has been growth in library wayfinding research since that time, but it is still a very small segment of lis research compared to other information seeking research. if we accept that “people come to libraries with cognitive information needs, and library use—especially first-time library use--adds to those needs before it meets them” (eaton, 1991, p. 520) and that “there is little point in having a resource or service if locating it is a tiresome and frustrating experience for patrons” (zaugg et al., 2016, p. 80), then libraries cannot afford to neglect the needs of physical space navigation. there is already pressure on many libraries, of all types, to cede space to other uses. it doesn’t help if library spaces are hard to use; that makes people even less likely to support library requests to maintain or expand their space. wayfinding research that helps libraries make their spaces more usable is of extreme importance, requiring more research and subsequent publication of the findings of that research in the lis literature.   there is need for library wayfinding research in public, school, and other libraries    over 70% of library wayfinding research indexed in lista is about academic libraries and based on academic library users. academic libraries serve a specific population that differs greatly from the user populations of public, school, and special libraries. it cannot be assumed that research based on undergraduate students (the most common population studied in the academic library wayfinding research) is generalizable to k-12 students and teachers, digital and other special library users, or the broad populations that comprise public library users. there is a strong need for empirical wayfinding research in all types of libraries. perhaps once there is a large body of wayfinding research in a wide variety of library types, it might be determined that wayfinding needs are similar across library types, but we cannot safely assume that without any empirical evidence on which to support such an assertion. so, we first need a larger body of literature on wayfinding research in public libraries, school libraries, and special libraries before we can determine the generalizability of wayfinding research in one library type to all library types.   there is need for more research on library wayfinding that is empirical, guided by theoretical models, and in various types of libraries   such research might replicate existing studies, expand studies from academic libraries to other types of libraries, or ask new questions. for example, to what degree does familiarity play a role in successful library patron wayfinding and how can we support that for new patrons?  libraries focus much of their wayfinding improvement efforts on signage, but how much can signage overcome problems of physical layout and architectural legibility?  how can we use new technologies to support and facilitate wayfinding in libraries?  given the nascent nature of library wayfinding research, there are countless questions to be asked. this is a rich area for future research that offers many opportunities for researchers willing to take up the challenge. it is hoped that this article will serve as an impetus for more researchers to investigate library wayfinding in their research and to consider researching library wayfinding in a variety of library types.   conclusion   reviewing the literature indexed in lista showed that research on library wayfinding is a nascent field with much work to be done. there are fewer than 40 articles in lista that report research on wayfinding in libraries, an extremely small number compared to the research articles on other topics with lis, such as information behaviour research. the majority of that research on library wayfinding is based on studies conducted in academic libraries, so we know even less about wayfinding in school, public, special, and digital libraries. in a field like librarianship that prides itself on a service orientation, research on critical issues for customer service should be prevalent. wayfinding is one such critical issue, but empirical research in this area is limited.   references   allard, j. (2008). coping with complexity: reconfiguring the navigation system for santiago's new transportation plan. information design journal, 16(3), 163-177. http://dx.doi.org/10.1075/idj.16.3.01all   agarwal, n. k., & lawrence, h. (2014). office location map of individuals in the library and other college campus buildings: a proof-of-concept wayfinding system. journal of web librarianship, 8(3), 305-323. http://dx.doi.org/10.1080/19322909.2014.927744   baker, m., bakkalbasi, n., call, e., & kamsler, b. (2015). burke library wayfinding study report. new york, ny: columbia university. http://dx.doi.org/10.7916/d8kh0mg6   baskaya, a., wilson, c., & özcan, y. (2004). wayfinding in an unfamiliar environment: different spatial settings of two polyclinics. environment and behavior, 36(6), 839-867. http://dx.doi.org/10.1177/0013916504265445   bawden, d., & robinson, l. (2013). introduction to information science. chicago, il: neal-schuman.   beck, s. g. (1996). wayfinding in libraries. library hi tech, 14(1), 27-36. http://dx.doi.org.uri.idm.oclc.org/10.1108/eb047977   beecher, a. b. (2004). wayfinding tools in public library buildings: a multiple case study. retrieved from proquest digital dissertations. (umi #3126555).   best, g. (1970). direction-finding in large buildings. in d.v. canter (ed.), architectural psychology: proceedings of the conference held at dalandhui university of strathclyde, 28 february – 2 march 1969 (pp. 72-75). cambridge, uk: w. heffer and sons.   bishop, b. w. (2012). analysis of reference transactions to inform library applications (apps). library & information science research, 34(4), 265-270. http://dx.doi.org/10.1016/j.lisr.2012.06.001   bishop, b. w., & mandel, l. h. (2010). utilizing geographic information systems (gis) in library research. library hi tech, 28(4), 536-547. http://dx.doi.org/10.1108/07378831011096213   chebat, j.-c., gélinas-chebat, c, & therrien, k. (2005). lost in a mall, the effects of gender, familiarity with the shopping mall and the shopping values on shoppers’ way finding processes. journal of business research, 58(11), 1590-1598. http://dx.doi.org/10.1016/j.jbusres.2004.02.006   davis, r., & ohman, j. (2016). wayfinding in ageing and alzheimer’s disease with a virtual senior residence: study protocol. journal of advanced nursing, 72(7), 1677-1688. http://dx.doi.org/10.1111/jan.12945   dempsey, b. (2006). wayfinding in action: patrons get to what they want fast at these four libraries. library journal, 131(spring2006 design supplement), 14-15. retrieved from http://lj.libraryjournal.com   dogu, u., & erkip, f. (2000). spatial factors affecting wayfinding and orientation: a case study in a shopping mall. environment and behavior, 32(6), 731-755. retrieved from http://journals.sagepub.com/home/eab   eaton, g. (1991). wayfinding in the library: book searches and route uncertainty. rq, 30(4), 519-527.   eaton, g., vocino, m. & taylor, m. (1992). evaluating signs in a university library. collection management, 16(3), 81-102. http://dx.doi.org.uri.idm.oclc.org/10.1300/j105v16n03_06   ebsco industries. (2016). library, information science & technology abstractstm with full text [web page]. retrieved from https://www.ebscohost.com/public/library-information-science-and-technology-abstracts-with-full-text   fendley, t. (2009). making sense of the city: a collection of design principles for urban wayfinding. information design journal, 17(2), 91-108. http://dx.doi.org/10.1075/idj.17.2.03fen   fox, r. (2015). paths of the mind. oclc systems & services, 31(4), 154-157. http://dx.doi.org/10.1108/oclc-07-2015-0006   hafiz, d. o., & zohdy, i. h. (2016). wayfinding improvement in metro stations using dynamic lighting environment. in transportation research board 95th annual meeting (no. 16-5600). retrieved from https://trid.trb.org/view.aspx?id=1394065   hahn, j. (2011). location-based recommendation services in library book stacks. reference services review, 39(4), 654-674. http://dx.doi.org/10.1108/00907321111186677   hahn, j., & morales, a. (2011). rapid prototyping a collections-based mobile wayfinding application. journal of academic librarianship, 37(5), 416-422. http://dx.doi.org/10.1016/j.acalib.2011.06.001   hahn, j., & ryckman, n. (2012). modular mobile application design. code4lib journal, (18), n.p. retrieved from http://journal.code4lib.org   hahn, j., ryckman, b., & lux, m. (2015). topic space: rapid prototyping a mobile augmented reality recommendation app. code4lib journal, (30), n.p. retrieved from http://journal.code4lib.org   hahn, j., twidale, m., gutierrez, a., & farivar, r. (2011). methods for applied mobile digital library research: a framework for extensible wayfinding systems. reference librarian, 52(1/2), 106-116. http://dx.doi.org/10.1080/02763877.2011.527600   hahn, j., & zitron, l. (2011). how first-year students navigate the stacks: implications for improving wayfinding. reference & user services quarterly, 51(1), 28-35. retrieved from https://journals.ala.org/index.php/rusq   horan, m. (1999). what students see: sketch maps as tools for assessing knowledge of libraries. journal of academic librarianship, 25(3), 187-201. https://doi.org/10.1016/s0099-1333(99)80198-0   hsiao, g. l.-k., tang, c.-h., huang, t.-c., & lin, c.-y. (2016). firefighter wayfinding in dark environments monitored by rfid. fire technology, 52(1), 273-279. http://dx.doi.org/10.1007/s10694-015-0477-y   johnston, m. p., & mandel, l. h. (2014). are we leaving them lost in the woods with no breadcrumbs to follow? assessing signage systems in school libraries. school libraries worldwide, 20(2), 38-53. retrieved from http://www.iasl-online.org/publications/slw/index.html   julien, h. (1996). a content analysis of the recent information needs and uses literature. library and information science research, 18(1), 53-65. http://dx.doi.org/10.1016/s0740-8188(96)90030-4   julien, h., & duggan, l. j. (2000). a longitudinal analysis of the information needs and uses literature. library and information science research, 22(3), 291-309. http://dx.doi.org/10.1016/s0740-8188(99)00057-2   julien, h., pecoskie, j. l., & reed, k. (2011). trends in information behavior research, 1999–2008: a content analysis. library and information science research, 33(1), 19-24. http://dx.doi.org/10.1016/j.lisr.2010.07.014   kanakri, s., schott, m., mitchell, a., mohammad, h., etters, m., & palme, n. (2016). wayfinding systems in educational environments. environment and ecology research, 4(5), 251-256. http://dx.doi.org/10.13189/eer.2016.040503   kaya, s. d., ileri, y. y., & yuceler, a. (2016). importance of hospital way-finding system on patient satisfaction. in m. h. bilgin, h. danis, e. demir, & u. can (eds.), business challenges in the changing economic landscape vol. 2 (pp. 33-40). cham, switzerland: springer international.   kerr, s. t. (1990). wayfinding in an electronic database: the relative importance of navigational cues vs. mental models. information processing & management, 26(4), 511-523. https://doi.org/10.1016/0306-4573(90)90071-9   kinsley, k. m., schoonover, d., & spitler, j. (2016). gopro as an ethnographic tool: a wayfinding study in an academic library. journal of access services, 13(1), 7-23. http://dx.doi.org/10.1080/15367967.2016.1154465   lee, k. m., li, m., & lin, c. y. (2015, july). a novel way-finding method based on geomagnetic field effects and magnetic tensor measurements for visually impaired users. in 2015 ieee international conference on advanced intelligent mechatronics (aim) (pp. 232-237). ieee. http://dx.doi.org/10.1109/aim.2015.7222537    li, r., & klippel, a. (2012). wayfinding in libraries: can problems be predicted? journal of map & geography libraries, 8(1), 21-38. http://dx.doi.org/10.1080/15420353.2011.622456   lopatovska, i., & arapakis, i. (2011). theories, methods and current research on emotions in library and information science, information retrieval and human–computer interaction. information processing and management, 47(4), 575-592. http://dx.doi.org/10.1016/j.ipm.2010.09.001   mandel, l. h. (2010). toward an understanding of library patron wayfinding: observing patrons' entry routes in a public library. library & information science research, 32(2), 116-130. http://dx.doi.org/10.1016/j.lisr.2009.12.004   mandel, l. h. (2013). finding their way: how public library users wayfind. library & information science research, 35(4), 264-271. http://dx.doi.org/10.1016/j.lisr.2013.04.003   matteson, m. l., & miller, s. s. (2012). emotional labor in librarianship: a research agenda. library and information science research, 34(3), 176-183. http://dx.doi.org/10.1016/j.lisr.2012.02.003   morag, i., heylighen, a., & pintelon, l. (2016). evaluating the inclusivity of hospital wayfinding systems for people with diverse needs and abilities. journal of health services research & policy, 21(4) 243-248. http://dx.doi.org/10.1177/1355819616642257   normore, l. f. (2012). “here be dragons”: a wayfinding approach to teaching cataloguing. cataloging & classification quarterly, 50(2/3), 172-188. http://dx.doi.org/10.1080/01639374.2011.651192   nguyen, q. h., vu, h., tran, t. h., & nguyen, q. h. (2017). developing a way-finding system on mobile robot assisting visually impaired people in an indoor environment. multimedia tools and applications, 76(2), 2645-2669. http://dx.doi.org/10.1007/s11042-015-3204-2   orphanides, a. k. (2011). lessons in public touchscreen development. code4lib journal, (15), n.p. retrieved from http://journal.code4lib.org   palmer, a., & choi, n. (2014). the current state of library open source software research. library hi tech, 32(1), 11-27. http://dx.doi.org/10.1108/lht-05-2013-0056   polger, m. a., & stempler, a. f. (2014). out with the old, in with the new: best practices for replacing library signage. public services quarterly, 10(2), 67-95. http://dx.doi.org/10.1080/15228959.2014.904210   porat, l. (2013). marketing and assessment in academic libraries: a marriage of convenience or true love? evidence based library & information practice, 8(2), 60-67. http://dx.doi.org/10.18438/b8fs5m   schoonover, d., & kinsley, k. m. (2014). stories from the stacks: students lost in the labyrinth. journal of access services, 11(3), 175-188. http://dx.doi.org/10.1080/15367967.2014.914426   shiwakoti, n., tay, r., stasinopoulos, p., & woolley, p. j. (2016). passengers’ awareness and perceptions of way finding tools in a train station. safety science, 87, 179-185. http://dx.doi.org/10.1016/j.ssci.2016.04.004   stempler, a. f. (2013). navigating circular library stacks: a case study on signage. reference services review, 41(3), 503-513. http://dx.doi.org/10.1108/rsr-02-2013-0006   stempler, a. f., & polger, m. a. (2013). do you see the signs? evaluating language, branding, and design in a library signage audit. public services quarterly, 9(2), 121-135. http://dx.doi.org/10.1080/15228959.2013.785881   stoffell, e.-p., schoder, k., & ohlbach, h. j. (2008). applying hierarchical graphs to pedestrian indoor navigation. in w. g. aref, m. f. mokbel, h. samet, m. schneider, c. shahabi, & o. wolfson (eds.), proceedings of the 16th acm sigspatial international conference on advances in geographic information systems (acm gis 2008) (pp 419-422). http://dx.doi.org/10.1145/1463434.1463499   tang, c. h., wu, w. t., & lin, c. y. (2009). using virtual reality to determine how emergency signs facilitate way-finding. applied ergonomics, 40(4), 722-730. http://dx.doi.org/10.1016/j.apergo.2008.06.009   torrado, j. c., montoro, g., & gomez, j. (2016). easing the integration: a feasible indoor wayfinding system for cognitive impaired people. pervasive and mobile computing, 31, 137-146. http://dx.doi.org/10.1016/j.pmcj.2016.02.003   tüzün, h., telli, e., & alır, a. (2016). usability testing of a 3d touch screen kiosk system for way-finding. computers in human behavior, 61, 73-79. http://dx.doi.org/10.1016/j.chb.2016.03.006   zaugg, h., & child, c. (2016). collaborating with nonlibrary faculty for assessment and improved instruction. journal of library administration, 56(7), 823-844, http://dx.doi.org/10.1080/01930826.2015.1124704   zaugg, h., child, c., bennett, d., brown, j., alcaraz, m., allred, a., & ... lee, s. (2016). comparing library wayfinding among novices and experts. performance measurement & metrics, 17(1), 70-82. http://dx.doi.org/10.1108/pmm-12-2015-0041   evidence summary   a study of a sample of facebook users finds they do not seek political news through facebook but are exposed to political news through this medium   a review of: schaferm, s., sulflow, m., & muller, p. (2017). the special taste of snack news: an application of niche theory to understand the appeal of facebook as a source for political news. first monday, 22(4-3). http://dx.doi.org/10.5210/fm.v22i4.7431   reviewed by: elizabeth stovold information specialist, cochrane airways group st george’s, university of london tooting, london, united kingdom email: estovold@sgul.ac.uk   received: 4 dec. 2017     accepted: 22 feb. 2018      2018 stovold. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29384       abstract   objective – to investigate facebook as a source of exposure to political news stories and to compare the reasons for using facebook as a news source and the gratifications obtained, compared with other news sources. design – survey questionnaire. setting – facebook. subjects – 422 german facebook users. methods – an online survey was developed to investigate the use of facebook as a news source compared with other sources. specific research questions were informed by the ‘theory of niche’ (dimmick, 2003) which examines the coexistence and competition between different media outlets by examining the breadth, overlap and superiority of one platform over another. the survey was distributed using a ‘snowball’ technique between july and august 2015. the survey was shared by 52 student research assistants on their facebook profiles. they asked their friends to complete the survey and share it with their own networks. main results – the mean (m) age of the 422 respondents was 23.5 years (sd=8.25). the majority were female (61%) with a high school degree (89%). tv news and news websites were the most frequently used sources of political news. facebook ranked third, ahead of newspapers, search engines, magazines, email provider websites, and twitter. the mean score for the importance of facebook as a news sources was 2.46 (sd=1.13) on a scale of 1 to 5, where 1 is low and 5 is high. this fell in the middle of the range when compared with the top ranked source assessed by importance (tv news, m 4.40, sd=0.88) and the lowest (email providers, m 1.92, sd=0.97). users rarely visited facebook with the purpose of finding news (m 1.59, sd=0.73). however, they estimated around 24% of the posts they see were concerned with political news, and when encountered, these stories are frequently read (m 3.53, sd=1.18). however, the level of interaction as measured by liking, commenting, sharing or status updates was low (m 1.94 sd=1.09; m 1.37, sd=0.79; m 1.51, sd=0.85 and m 1.4, sd=0.78 respectively).   the ‘gratification’ categories where facebook as a news source scored the highest were for killing time (m 2.97, sd=1.29), entertainment (m 2.92, sd=1.05), and surveillance (m 2.77, sd=1.01). when compared to newspapers and tv news, it was found that facebook has a lower score for niche breadth, meaning that it serves a specific rather than general news function. facebook also had a lower overlap score when compared with the other media, thereby performing a complementary function, while tv news and newspapers perform similarly. tv news scored better for providing balanced information, surveillance and social utility while facebook scored highest for killing time. there was no difference in the category of entertainment. there was a similar picture when comparing facebook with newspapers. conclusion – the authors conclude that while users do not actively seek political news through facebook, they are exposed to political news through this medium. respondents did not consider the news to be well balanced, and that currently facebooks’ niche is restricted to entertainment and killing time. the authors note that this may be disappointing for news organisations, but there is potential to expose large audiences to political news when they are not actively seeking it. the findings represent a specific time point in a changing landscape and future research will need to take these changes into account. comparisons with other online news sources and the use of objective measures to validate self-reported data would be valuable areas for future research.   commentary   the first part of the study reviews research around the notion of ‘snack news’, where news is presented in a compact form containing only essential headlines and passively consumed rather than actively sought and read in depth. the authors then discuss the ‘theory of niche’ as applied to media outlets, and provide a good overview of the research in both these areas. the authors developed their research questions in the context of the ‘theory of niche’ which allowed them to compare the perceived gratifications users derive from traditional news media with facebook.   the study was evaluated with the glynn (2006) critical appraisal tool and was found to be valid, with an overall score of 75% for validity. the survey questions were described in the text as part of the reported results. this enabled the authors to show how the specific questions related to the research objectives, however it would have been helpful to have the survey presented in its entirety to help readers wishing to replicate the study. the results are described in detail and there is a thorough discussion and interpretation of the findings, together with a discussion of the limitations of the study and suggestions for future research.   where the study lost points was for the population sample. this is very small sample as a proportion of all facebook users, and represents a very young demographic. because of this, it is not clear how generalizable the findings of this study are, particularly in other age brackets. the authors acknowledge this limitation, as well as their reliance on self-reported data, which is open to bias and both underor over-estimations by the respondents.   what the study of this sample of facebook users shows is that this small group still regards traditional media such as newspapers and tv news programs as reliable sources of news. however they do not consider that facebook provides balanced information. this is encouraging for library and information professionals who seek to educate their patrons to evaluate and critically appraise their information sources. the study focused on the motivations behind using facebook as a news source and the perceived gratifications of using this platform. it did not examine the origins of the news stories, or which news sites and other facebook pages the respondents followed and if this has any impact on users’ perceptions. this would be an interesting avenue for future research, together with the author’s recommendations.   references   dimmick, j.w. (2003). media competition and coexistence: the theory of the niche. mahwah, n.j.: l. erlbaum associates.   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692145 evidence based library and information practice    evidence based library and information practice       commentary   a big step forward: it’s time for a database of evidence summaries in library and information practice   denise koufogiannakis collections and acquisitions coordinator university of alberta libraries edmonton, alberta, canada email: dak@ualberta.ca   lorie kloda editor-in-chief associate university librarian, planning & community relations concordia university montreal, quebec, canada email: lorie.kloda@concordia.ca   heather j. pretty cataloguing librarian memorial university st. john’s, newfoundland & labrador, canada email: hjpretty@mun.ca   received: 13 feb. 2016                                                                  accepted: 14 feb. 2016      2016 koufogiannakis, kloda, and pretty. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     since evidence based library and information practice (eblip) began 10 years ago, evidence summaries have been an integral part of the journal. evidence summaries present readers with a brief overview of an original research article in structured abstract form, followed by a commentary that critically appraises the research. a standardized format and guidelines are used to ensure consistency amongst writers. in our first 10 volumes, eblip has published 349 evidence summaries, averaging 9 per issue. they cover a wide variety of topics and touch on all domains of librarianship. evidence summaries are written by members of a writing team selected by the editorial board, and research articles are assigned by the associate editor. all published evidence summaries have undergone peer review.   evidence summaries were designed to overcome some of the barriers to evidence based practice in librarianship and information fields by bringing awareness of previously published research to practitioners, and providing objective critiques of these. there was also the hope that summaries would help librarians be critical readers of the research literature (koufogiannakis, 2006), and to inspire more critical thinking, as well as more rigorous research.   the editorial board has always been committed to making evidence summaries a useful tool for practitioners, and as such, some members conducted research on evidence summaries in order to learn more about the summaries and their impact. kloda, koufogiannakis, and mallan (2011) undertook a study examining the content of evidence summaries from the first three volumes of the journal, with a focus on the commentary, or critical appraisal, component. this study revealed both strengths and weaknesses within library and information research, and also identified the aspects most likely to be critiqued (and overlooked) by evidence summary writers. based on findings from the study, in 2012, the journal implemented changes to the guidelines for evidence summary writers. these changes were designed to improve consistency and focus, such as ensuring a descriptive title, a more concise abstract, a commentary not exceeding 450 words, and inclusion of a statement of significance and practical application of the research (kloda, 2012).    to determine if evidence summaries had any impact upon readers, members of the editorial board embarked on another study, this time investigating the ways in which reading a specific evidence summary influenced a librarian’s knowledge, their individual practice or workplace practices, or possibly had an impact on library users (kloda, koufogiannakis, brettle, 2014). the findings demonstrated the numerous ways in which evidence summaries not only provide new knowledge, but assist in decision making at the individual and organizational level. it is now not only our opinion that evidence summaries are valuable aids in a librarian’s ability to understand and access the research literature, but that they are instrumental resources for translating research to inform knowledge and practice. in this 11th volume of the journal, as we publish the 350th evidence summary, with no end in sight, it’s time to harness the value that evidence summaries hold: librarianship needs a database of evidence summaries.   we can look to medicine for a way to potentially create such a database and make it useful to those searching for synopses of research evidence. shurtz and foster developed a rubric for evaluating evidence based medicine (ebm) point-of-care tools with five general evaluation criteria: compatibility and access, content, search options and results, quality control, and evidence (2011). these general evaluation criteria could also be applied to the development of a database of eblip evidence summaries. content (e.g., types of summaries) and quality control (e.g., peer-review, author affiliation, editing process) are already inherent in the evidence summaries that would be the basis of a searchable database. the editorial board fully supports compatibility and accessibility of eblip evidence summaries such that any database developed should be open access and should link outward to open access versions of the original articles where possible (otherwise linking to abstracts). while we do not wish to compare a database of eblip summaries with ebm point-of-care tools, there are many concepts we can draw from looking at fields that have already developed such tools. library and information research and practice would dictate how the product is used and organized, and we see overall value in having a one-stop shop for pre-evaluated analyses of research. at one point, a group of australian librarians categorized each evidence summary published in eblip according to domain, but as the number of evidence summarize expanded this effort was not sustainable and has since ended. however, it was useful for readers to focus on evidence summaries by broad general topic area, such as collections or management, for example. ultimately, we would like to see the evidence summaries being housed in a searchable database, specific to this type of scholarship. at the moment, eblip is indexed in several sources, but evidence summaries are treated in the same way as original research articles, which sometimes leads to confusion. a specialized database that allows for searching by specific fields such as librarianship domain and sector as well as topic area, would allow practitioners to easily find relevant evidence. each evidence summary allows a practitioner to get an overview of the article prior to actually reading the original.   based on shurtz and foster’s criteria, the real opportunities to improve upon eblip evidence summaries for use within a database are firstly in what search options and results will be available to those using the database as discussed above. this may require some additional work with each evidence summary to classify it based on domain, sector, and topic area to make these concepts more easily searchable. the second opportunity to improve upon eblip evidence summaries would be to assign a category to indicate the strength of the evidence of each original research article. shurtz and foster define the characteristics of their evidence criteria as “standard of grading” and “clarity of levels.” most eblip evidence summaries are critically appraised using a tool (e.g., the ebl checklist and the reliant instrument). as part of preparing evidence summaries for a searchable database, the scores from these tools could be made more evident.       our call now is to the wider library and information studies community for assistance to develop a database that could house and treat evidence summaries in a way that would make them more discoverable and useful than is possible within the confines of a journal. we know that the expertise exists within our community, as do open source tools that could facilitate the development of such a home. if an institution is willing to host and help develop the underlying structure of the database, we know there will be overwhelming support from the eblip community to help with determining processes for the needed structure of content, metadata, editing, and other workflows. we know this because after 10 years of building a successful journal based on the will of the eblip community, the same effort will certainly be extended to a logical next step in the process. will you join us in moving eblip forward by helping to build a database of evidence summaries?   references   kloda, l. (2012). improvements to evidence summaries: an evidence based approach. evidence based library and information practice, 7(3): 71-72. http://dx.doi.org/http://dx.doi.org/10.18438/b87s5r   kloda, l. a., koufogiannakis, d., & brettle, a. (2014). assessing the impact of evidence summaries in library and information practice. library & information research, 38(119): 29-46. retrieved from http://www.lirgjournal.org.uk/lir/ojs/index.php/lir/article/view/644   kloda, l. a., koufogiannakis, d., & mallan, k. (2011). transferring evidence into practice: what evidence summaries of library and information studies research tell practitioners. information research, 6(1). retrieved from http://informationr.net/ir/16-1/paper465.html koufogiannakis, d. (2006). small steps forward through critical appraisal. evidence based library and information practice, 1(1): 81-2. http://dx.doi.org/10.18438/b8z59k    shurtz, s., & foster, m. j. (2011). developing and using a rubric for evaluating evidence-based medicine point-of-care tools. journal of the medical library association, 99(3): 247-254. http://dx.doi/org/10.3163/1536-5050.99.3.012   evidence summary   early career researchers demand full-text and rely on google to find scholarly sources   a review of: nicholas, d., boukacem-zeghmouri, c., rodríguez-bravo, b., xu, j., watkinson, a., abrizah, a., herman, e., & świgoń, m. (2017). where and how early career researchers find scholarly information. learned publishing, 30(1), 19-29. http://dx.doi.org/10.1002/leap.1087   reviewed by: richard hayman associate professor & digital initiatives librarian mount royal university calgary, alberta, canada email: rhayman@mtroyal.ca   received: 5 may 2017    accepted: 5 december 2017      2017 hayman. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract    objective – to examine the attitudes and information behaviours of early career researchers (ecrs) when locating scholarly information.   design – qualitative longitudinal study.   setting – research participants from the united kingdom, united states of america, china, france, malaysia, poland, and spain.   subjects – a total 116 participants from various disciplines, aged 35 and younger, who were holding or had previously held a research position, but not in a tenured position. all participants held a doctorate or were in the process of earning one.   methods – using structured interviews of 60-90 minutes, researchers asked 60 questions of each participant via face-to-face, skype, or telephone interviews. the interview format and questions were formed via focus groups.   main results – as part of a longitudinal project, results reported are limited to the first year of the study, and focused on three primary questions identified by the authors: where do ecrs find scholarly information, whether they use their smartphones to locate and read scholarly information, and what social media do they use to find scholarly information. researchers describe how ecrs themselves interpreted the phrase scholarly information to primarily mean journal articles, while the researchers themselves had a much expanded definition to include professional and “scholarly contacts, ideas, and data” (p. 22).   this research shows that google and google scholar are widely used by ecrs for locating scholarly information regardless of discipline, language, or geography. their analysis by country points to currency and the combined breadth-and-depth search experience that google provides as prime reasons for the popularity of google and google scholar. of particular interest is the popularity and use of google scholar in china, where it is officially blocked but accessed by ecrs via proxy services. other general indexes, such as web of science and scopus, are also popular but not universally used by ecrs, and regional differences again point to pros and cons of these services. some specialized services are emphasized, including regional tools such as the china national knowledge infrastructure, as well as certain broad disciplinary resources, such as pubmed for its coverage of sciences and biomedical information.    researchers report that ecrs participating in this study were less concerned about how they gained access to full-text scholarly information, only that they could access full-text sources. in particular, ecrs do not take much notice of libraries and their platforms, seemingly unaware of the steps libraries take to acquire and ensure access to scholarly information, while viewing physical libraries themselves primarily as study spaces for undergraduate students and not places for the ecr to visit or work. while ecrs occasionally acknowledge library portals and login interfaces, researchers found that these participants mostly ignored these, and that they found discovery services to be confusing or difficult.   concerning social media use, participants identified 11 different platforms used but only researchgate was mentioned and used by participants from all seven countries represented. social media tends to be used directly for keeping track of research trends and opinions and also the work specific researchers are publishing, and indirectly when referred to sites such as researchgate to find full-text of a specific article. facebook, twitter, and linkedin are used occasionally or moderately, but not universally. researchers highlight regional differences of social media use in china, where ecrs are more likely to connect with other researchers and receive notifications when those researchers publish.   the study reports limited information ecrs’ use of smartphones for information seeking. about half of ecr participants reported use of their smartphone for discovering scholarly sources. the advantage smartphones provide includes near-ubiquitous internet access and therefore the ability to access scholarly materials on the go, though ecrs are less likely to download or read full-text articles via their smartphones. the rate of adoption of smartphone use for scholarly materials varies by country.   conclusion – early career researchers access scholarly information in a wide variety of ways, with google and google scholar as the preferred starting location, and with social media also proving useful. ease-of-use and full-text availability are paramount concerns; the spread of open access materials helps fuel the availability of materials, and google makes these easy to find. though physical libraries are perceived to be of limited use, the digital access they provide to full-text scholarly sources is still vital even if ecrs do not make the connection between having that important access and the fact that libraries act as buyers and providers of access   commentary   this early report on a broad longitudinal study provides some insights to the information behaviours of early career researchers. in particular, it confirms results from other studies showing that ecr behaviours mirror recognized information seeking behaviours among researchers who use google/google scholar, pubmed, researchgate, and similar tools, as well as regular updates from ones’ professional and social networks, for finding relevant and timely scholarly information (pontis, blandford, greifeneder, attalla, & neal, 2015).   this study relied on convenience and snowball recruitment, and uses small sample sizes, both of which are acceptable within the scope of qualitative research. the intentional selection of most participants from the sciences (two-thirds) over the social sciences (one-third), at the behest of the funding agency, and seemingly excluding the arts, humanities, or business disciplinary categories altogether, and the reliance on different types of materials these disciplines have (e.g., monographs vs. journals) presents challenges for drawing generalizable conclusions about ecrs. the researchers do acknowledge that small participant numbers limit the generalizability of their findings. despite their assurances that such “limitations were compensated for by using personal interview techniques and asking in-depth questions” (p. 28), this kind of qualitative research cannot be generalized beyond the immediate participant pool. recruitment occurred from within publisher lists and society memberships loosely connected to the study’s sponsoring body, in combination with the skewed disciplinary representation addressed above, lends further weight to the criticism that that these findings cannot be treated as representative of ecr behaviours, and raises the possibility of conflict of interest. for findings found to be universal across all ecrs who participated, the use of multiple geographies with multiple participants from each location mitigates to some degree these shortcomings, and may serve to help reduce some bias introduced during recruitment (glynn, 2006).   though they are not all plainly stated, this study points to a number of implications for information practice in academic libraries. the most obvious is the need for university libraries and liaison librarians to improve ecrs’ understanding of the connections between the library purchasing subscriptions and full-text access, and ecrs’ demand for such access. this research study may further confuse already complex categorizations by making distinctions between general databases (e.g., web of science and scopus), specialized databases (e.g., sciencedirect, springerlink), and “libraries and their platforms”, despite the fact that in most cases these all fundamentally depend on subscription-driven resources purchased by the library. since ecrs and other researchers are dedicated google users for seeking scholarly information, libraries and their vendors must be prepared to work toward improving their resources and services to mimic the google/google scholar experience as much as possible, or to better integrate those google services into their offerings.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154   pontis, s., blandford, a., greifeneder, e., attalla, h., & neal, d. (2015). keeping up to date: an academic researcher's information journey. journal of the association for information science and technology, 68(1), 22-35. http://dx.doi.org/10.1002/asi.23623   editorial   integrating 2014 library assessment conference proceedings and peer review   martha kyrillidou principal, qualitymetrics llc research associate, university of illinois at urbana-champaign part-time instructor, kent state university email: marthakyrillidou@gmail.com      2016 kyrillidou. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     this is the third editorial i am writing for eblip.  in the prior two occasions i focused my editorial only on those articles from the 2010 and the 2012 library assessment conferences that we invited and published through eblip.  in 2010 i co-authored the editorial with damon jaggars as the effort we had initiated was going to be a special issue of the journal of library administration originally but with debates over the way open access was treated we decided to move the corpus of articles to a venue that had a strong open access mission.    since then i continued to work with eblip in featuring articles from the 2012 conference. this issue features work from the 2014 conference.  in this issue my editorial will cover not only the two commentaries (carlsson, and doshi, scharf, and fox) and the four feature articles that originally appeared in the 2014 conference proceedings (neurohr and bailey; reed, mcfarland and croft; baumgart, carillo, and schmidli; and, ziegenfuss and borrelli) but also the two original submissions to eblip by applegate as well as chew, schoenborn, stemper and lilyard.    it is really important and would like to encourage all authors submitting conference proceedings to consider publishing their work in the formal journal literature beyond the proceedings.  going through the peer review submission workflow allows further refinement of ideas and growth of our own professional thinking.  publishing in an open access venue is also an important consideration.  over the years of working for the association of research libraries (arl), i often felt the tension of publishing in open access platforms while also gaining the reputation necessary that the peer review process can afford us and this is typically for work that was above and beyond what i produced for arl as work for hire.  our field has an increasing number of venues that are open access, so i am hoping that most of our literature will be available as open access in the future.  it is an important consideration for the growth of library assessment and evidence based library and information practice that journals like eblip have facilitated and enabled.   the richness of our library assessment work is emerging through four key themes in the current issue: assessment across boundaries (interand intrainstitutional), organizational improvements based on data, innovative qualitative methods that develop emergent understanding of key aspects of our environment, and articles that focus on the scholarly communication cycle with its emerging emphasis on online profiles for researchers and disciplinary differences of e-journals.   assessment across boundaries: collaborating across institutions and within the parent institution   two of the pieces presented in this issue discuss strategic assessment elements crossing organizational boundaries.  focusing on student learning outcomes assessment, ziegenfuss and borrelli describe an impressive effort within the greater western library alliance and focusing on a more than a decade long strategy development effort, carlsson describes the strategy and assessment integration at gothenburg university in sweden.   “exploring the complexity of student learning outcome assessment practices across multiple libraries” is a collaborative qualitative research project, initiated by the greater western library alliance (gwla), to explore how librarians were involved in the designing, implementing, assessing, and disseminating student learning outcomes (slos) in gwla member academic libraries. the original objective of the research was to identify library evaluation/assessment practices at the different libraries to share and discuss by consortia members at a gwla-sponsored student learning assessment symposium in 2013. however, findings raised new questions and areas to explore beyond student learning assessment, and additional research was continued by two of the gwla collaborators after the symposium. the purpose of this second phase of research was to explore the intersection of library and institutional contexts and academic library assessment practices.   “library assessment and quality assurance creating a staff-driven and user-focused development process” describes the ways assessment and strategy is linked at gothenburg university. “the process has both bottom-up and top-down features designed to generate strong staff involvement and long-term strategic stability.” dating back to 2001 the quality cycle initiated by the university filtered into the library by 2003.the paper describes how the library operates with the quality cycle building feedback mechanisms for improving activities. the quality cycle is viewed as part of the strategic cycle that is an annual process of visioning and refreshing strategy at the library level.  an environmental scan provides new intelligence every year on the main areas that need to be the focus of the strategy. “the definition of a yearly process, into which data and previous findings can be funneled, has shown to be a powerful driving force for implementing meaningful change.”   organizational improvements based on data   whether it’s about improving reference services or developing advisory structures that help advocate sound improvements in the library, the next two articles have important organizational improvement lessons to share with us.   “the role of student advisory boards in assessment” describes the successful deployment of student advisory boards at the university of central florida, georgia tech and the university of louisville.  these institutions received valuable assessment information through the student advisory board. “the input has been used to trigger additional evaluation and assessment of programs, services, or resources as an indicator of areas that need a quick fix or a longer-term solution, and to focus on trends in campus life that affect use of the library. in all three libraries, insight gained from board members has resulted in positive improvements. board members can be surveyed between meetings. they can act as a test group for a survey or for questions and topics for focus group and other qualitative research. minutes and feedback to board members ensure their continued engagement with the library.” board members become active advocates for the library on campus who can help close the assessment loop by supporting the programs and enhancements … “through invaluable word-of-mouth publicity and through support of campus funding initiatives.” as the authors caution us “boards are not a shortcut to obtaining qualitative assessment information. if done well, everything takes time: recruiting the right students, preparing agendas, distributing minutes. it is only one tool in an assessment portfolio.”   “iterative chat transcript analysis: making meaning from existing data” examines patron satisfaction with reference services analyzing an existing corpus of chat transcripts. having conducted a similar analysis in 2010, the authors also compared librarian behaviors over time. drawing from the library literature, the authors identified a set of librarian behaviors closely associated with patron satisfaction. these behaviors include listening to and understanding patrons’ needs, inviting patrons to use the service again, and providing instruction or completing a search for patrons. the analysis shows that librarian behaviors have changed over time, pointing to what campus librarians are doing well, and that implementation of best practices at a campus level after the 2010 analysis may have increased these positive behaviors.   emergent understandings   two of the pieces focus on implementation of interesting qualitative approaches – deploying photovoice and content analysis of a variety of evidence sources help us gain emergent understanding of the needs of special user groups and special skills we need to develop as assessment professionals.   “using photo-elicitation with native american students to explore perceptions of the physical library” describes native american students’ perceptions of the edmon low library at oklahoma state university (osu). the study sought to understand how native american students perceived the role of the academic library in their lives, and which elements of the library students depicted and described as holding meaning for them. photo-elicitation, a form of visual research and a participatory research method, was the primary method chosen to explore students’ perceptions of the library. students followed a photo prompt for taking at least fifteen pictures of the library, then participated in two separate interviews with the primary researcher. participants also completed a demographic/questionnaire form, answered semi-structured questions, and ranked the photos they took. exploring how individual students who identify as native american perceive the university library enhanced our understanding of how libraries in predominantly white institutions (pwis) can best serve and support students. this study provided insight into the method of photo-elicitation interviews. this research also provided practical benefits for student participants through increased library knowledge.   “educating assessors: preparing librarians with micro and macro skills” examined the fit between libraries’ needs for evaluation skills, and library education and professional development opportunities. many library position descriptions and many areas of library science education focus on professional skills and activities, such as delivering information literacy, designing programs, and managing resources. only some positions, some parts of positions, and some areas of education specifically address assessment/evaluation skills. the growth of the library assessment conference, the establishment of the arl-assess listserv, and other evidence indicates that assessment skills are increasingly important. examining different bodies of evidence such as core competencies, training and course requirements, the authors found that while one-third of job postings made some mention of evaluation responsibilities, less than 10% of conference or continuing education offerings addressed assessment skills. in addition, management as a topic is a widespread requirement in mls programs (78%), while research (58%) and assessment (15%) far less common. overall, there seems to be more need for assessment/evaluation skills than there are structured offerings to educate people in developing those skills.   serving scholarship   the last two pieces focus on serving the scholarly needs of faculty and students through the development of online profiles and altmetrics and through the use of e-journal metrics.   “laying the groundwork for a new library service: scholar-practitioner & graduate student attitudes toward altmetrics and the curation of online profiles” assesses the knowledge base and needs of our academic communities in order to support the creation and maintenance of scholarly online profiles. participants were queried about use, issues, and attitudes toward scholarly profile and altmetric tools, as well as the role librarians could play in assisting with the curation of online reputation.  while all participants had googled themselves, few were strategic about their online scholarly identity. participants affirmed the perception that altmetrics can be of value in helping to craft a story of the value of their research and its diverse outputs. librarians are well-placed to assist scholar-practitioners who wish to curate an online profile or use altmetrics tools. areas of assistance include: personalized support, establishment of goals, orientation to specific tools, orientation to altmetrics and scholarly promotion landscape, preparing users for potential difficulties, discussing copyright implications, open access education, and guidance with packaging content for different venues and audiences.   “e-journal metrics for collection management: exploring disciplinary usage differences in scopus and web of science” shares a few important conclusions with us: collecting and correlating authorship and citation data allows patterns of use to emerge, resulting in a more accurate picture of use activity than the commonly used cost-per-use method. to find the best information on authoring activity by local faculty for subscribed journals, use scopus. to find the best information on citing activity by faculty peers for subscribed titles use thomson reuters’ customized local journal use reports (ljur), or limit a web of science search to local institution. the eigenfactor and snip journal quality metrics results can better inform selection decisions, and are publicly available. given the trend toward more centralized collection development, it is still critical to obtain liaison input no matter what datasets are used for decision making. this evidence of value can be used to defend any local library “tax” that academic departments pay as well as promote services to help faculty demonstrate their research impact.   i hope this corpus of articles serves as an inspiration to all of you to continue to innovate in library assessment and evidence based library and information practice and demonstrate the value of our services and libraries to our users through tangible contributions to their improved outcomes and increased impact.   reference   hiller, s., kyrillidou, m., and oakleaf, m. (2014). the library assessment conference – past, present and near future! journal of academic librarianship, 3(4), 410–412.   evidence summary   engineering faculty indicate high levels of awareness and use of the library but tend to consult google and google scholar first for research resources   a review of: zhang, l. (2015). use of library services by engineering faculty at mississippi state university, a large land grant institution. science & technology libraries, 34(3), 272-286. http://dx.doi.org/10.1080/0194262x.2015.1090941   reviewed by: elaine sullo coordinator, information and instructional services himmelfarb health sciences library the george washington university washington, district of columbia, united states of america email: elainej@gwu.edu   received: 31 may 2016  accepted: 26 july 2016      2016 sullo. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to investigate the engineering faculty’s information-seeking behaviour, experiences, awareness, and use of the university library.   design – web-based survey questionnaire.   setting – the main campus of a state university in the united states of america.   subjects – 119 faculty members within 8 engineering departments.   methods – an email invitation to participate in a 16-item electronic survey questionnaire, with questions related to library use, was sent in the spring of 2015 to 119 engineering faculty members. faculty were given 24 days to complete the survey, and a reminder email was sent 10 days after the original survey invitation.   main results – thirty-eight faculty members responded to the survey, representing a response rate of 32%. overall, faculty had a high level of use and awareness of both online and physical library resources and services, although their awareness of certain scholarly communication services, such as data archiving and copyright advisory, was significantly lower. faculty tend to turn to google and google scholar when searching for information rather than turning to library databases. faculty do not use social media to keep up with library news and updates. the library website, as well as liaison librarians, were cited as the primary sources for this type of information.   conclusions – the researcher concludes that librarians need to do a better job of marketing library resources, such as discipline-specific databases, as well as other library search tools. because faculty use web search engines as a significant source of information, the author proposes further research on this behaviour, and suggests more action to educate faculty on different search tools, their limitations, and effective use.   as faculty indicated a general lack of interest in integrating information literacy into their classes, the researcher notes that librarians need to find ways to persuade faculty that this type of integrated instruction is beneficial for students’ learning and research needs. faculty were aware of the library liaison program, so this baseline relationship between faculty and librarian can serve as an opportunity to build upon current liaison services and responsibilities.   commentary   the research addresses a clearly focused issue in terms of a specific population and important, well-defined library resources and services being studied. a number of previous studies have examined engineers’ information-seeking behaviour both in the industry and in the academy. most of the literature referenced by the author cites research that included students and faculty, faculty among several disciplines, or research that included faculty at multiple institutions. the author states an interest in examining the topic based on observations, library statistics, and conversations among librarians that suggest library services are underutilized. as such, this study is an attempt to examine more deeply the issues described in other studies, while at the same time, to try to understand specific library use patterns and faculty awareness of the engineering faculty at mississippi state university.   the study was evaluated using the cristal checklist (n.d.) for appraising a user study. the study questionnaire is available in the supplemental material on the publisher’s website, and therefore makes for a study that could be easily replicated. while the questions were created specifically for engineering faculty, they address library resources and services generally and could be used to survey non-engineering faculty without difficulty. however, the author did not mention if the survey was pre-tested or piloted, so the researcher did not have the opportunity to ensure that the questions made sense to faculty.   although the survey was sent to 119 faculty members, only 38 responded, representing a response rate of 32%. the respondents were rather evenly distributed among the eight engineering departments and they varied in age, providing a sample of respondents that most likely represented the entirety of the engineering faculty population. several of the survey questions, as well as the findings derived from the research, appear, at times, to provide conflicting or incomplete evidence. for example, faculty indicate that google and google scholar are the main resources consulted for research, but in response to a separate question indicate that online databases are important and rank them as the second-most essential resource. furthermore, while 89% of faculty consider library instruction to students as important or very important, the research does not indicate how often faculty actually schedule instruction sessions. the study may have benefited from the use of statistical tests to see whether the survey is uncovering the information that it was intended to discover. the research findings are described in the text and visually represented in graphs, tables, and charts. the author makes several conclusions based on the data, but notes that she hopes to explore the study findings further by conducting focus groups or interviews.   while this research is specific to engineering faculty and their use of library services, the data from the survey may apply to faculty in other disciplines, and therefore could be valuable to librarians who serve faculty in other academic library settings. this research provides a snapshot of a select group of faculty, but the results can be considered and perhaps even acted upon by librarians in general.   reference   cristal checklist for appraising a user study. (n.d.). retrieved may 26, 2016 from http://nettingtheevidence.pbwiki.com/f/use.doc     evidence summary   european academic libraries offer or plan to offer research data services   a review of: tenopir, c., talja, s., horstmann, w., late, e., hughes, d., pollock, d., … allard, s. (2017). research data services in european academic research libraries. liber quarterly, 27(1), 23-44. https://doi.org/10.18352/lq.10180   reviewed by: jennifer kaari medical librarian mount sinai health system new york, new york, united states of america email: jennifer.kaari@mountsinai.org   received: 2 feb. 2018     accepted: 4 apr. 2018      2018 kaari. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29416       abstract   objective – to investigate the current state of research data services (rds) in european academic libraries by determining the types of rds being currently implemented and planned by these institutions.   design – email survey.   setting – european academic research libraries.   subjects – 333 directors of the association of european research libraries (liber) academic member libraries.   methods – the researchers revised a survey instrument previously used for the dataone survey of north american research libraries and conducted pilot testing with european academic library directors. the survey instrument was created using the qualtrics software. the revised survey was distributed by email to liber institutions identified as academic libraries by the researchers and remained open for 6 weeks. question topics included demographics, rds currently offered, rds planned, staffing considerations, and the director’s opinions on rds. libraries from 22 countries participated and libraries were grouped into 4 regions in order to compare regional differences. data analysis was conducted using excel, spss or r software university of tennessee, university of tampere, and university of göttingen.   main results – 119 library directors responded to more than one question beyond basic demographics, for a response rate of 35.7%. among the libraries surveyed, more libraries offer consultative services than offered technical support for rds, although a majority planned to offer technical services in the future. geographically, libraries in western europe offer more rds compared with other regions. more libraries have reassigned or plan to reassign current staff to support rds services, rather than hire new staff for these roles. regardless of whether or not they currently offer rds, library directors surveyed strongly agree that libraries need to offer rds to remain relevant.   conclusion – the authors determine that a majority of library directors recognize that data management is increasingly important and many libraries are responding to this by implementing rds and collaborating across their institutions and beyond to help meet these needs. future research is suggested to track how these services develop over time, how libraries respond to the staffing challenges of rds, and whether consultative rather than technical services continue to be primary forms of rds offered.   commentary   data management is increasingly a concern for academics and researchers, and therefore to academic librarians. in 2016, the association of college & research libraries (acrl) identified rds, data policies and data management plans, and professional development for librarians in rds as three of the top trends in academic libraries (acrl, 2016). the current literature on rds in academic libraries focuses primarily on providing benchmarks for the current state of rds as well as librarians’ plans for future services. past surveys have been geographic, including studies of rds in north america, the uk, and australasia (tenopir et al., 2015; cox, kennan, lyon, & pinfield, 2017). this study builds on that literature by extending this inquiry to a european context.   this study was found to have a 90% validity rating overall based on glynn’s critical appraisal tool for library and information research (2006).  the study is well-designed, reflecting the authors’ previous experience studying this topic in other geographic areas. the research methodology is thoroughly described and the outcomes are clearly stated and discussed with respect to the data collection. the questions on the survey instrument are well-designed to glean precise answers that directly address the outcomes that the researchers intended to measure. the full data set and survey instrument are available online, which is extremely valuable both for the purposes of critically appraising the study and for potential future replication.   a potential weakness lies in the study population, which consisted of the 333 liber member institutions identified as university libraries by the investigators. the authors do not discuss the degree to which including only liber member institutions may or may not have been limiting to the study. according to the european bureau of library, information and documentation associations (2015), there are at least 5,974 academic libraries in europe,.  it is worth considering whether the 333 institutions offered the survey and the 119 respondents are truly representative of the entire field, particularly in such a nationally, culturally and linguistically varied region such as europe.   the results of this study were in keeping with past studies in other geographic regions, which similarly found that libraries are likely to focus on the advisory rather than technical aspects of rds (cox, kennan, lyon, & pinfield, 2017). rds implementation also presents many challenges to academic libraries, and future research is vital to determining how rds develops overtime and how librarians adapt to these challenges. ultimately, this study has the greatest value as a baseline assessment of the current state of rds in europe that can be used to inform future research and map the development of this trend. the authors suggest many future areas for research and with this well-documented study, they provide a vital foundation for future investigation.   references   acrl research planning and review committee. (2016). 2016 top trends in academic libraries: a review of the trends and issues affecting academic libraries in higher education. college & research libraries news, 77(6), 274-281. https://doi.org/10.5860/crln.77.6.9505   cox, a. m., kennan, m. a., lyon, l., & pinfield, s. (2017). developments in research data management in academic libraries: towards an understanding of research data service maturity. journal of the association for information science and technology, 68(9), 2182–2200. https://doi.org/10.1002/asi.23781   european bureau of library, information and documentation associations. (2015). [infographic]. academic libraries – statistics. retrieved  from http://www.eblida.org/activities/kic/academic-libraries-statistics.html   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387–399. https://doi.org/10.1108/07378830610692154   tenopir, c., hughes, d., allard, s., frame, m., birch, b., baird, l., … lundeen, a. (2015). research data services in academic libraries: data intensive roles for the future? journal of escience librarianship, 4(2). https://doi.org/10.7191/jeslib.2015.1085     evidence summary   education and criminal justice faculty value electronic serials over print to support professional activities   a review of: jones, g. f., cassidy, e. d., mcmain, l., strickland, s. d., thompson, m., & valdes, z. (2015). are serials worth their weight in knowledge? a value study. the journal of academic librarianship, 41(5), 578-582. http://dx.doi.org/10.1016/j.acalib.2015.07.004   reviewed by: sue f. phelps health sciences and outreach services librarian washington state university vancouver library vancouver, washington, united states of america email: asphelps@vancouver.wsu.edu   received: 17 may 2016  accepted: 15 jul. 2016      2016 phelps. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine the faculty assessed value of print and electronic serials.   design – qualitative survey.   setting – doctoral research institution in the southern united states of america.   subjects – 122 tenured or tenure-track faculty from the school of criminal justice and the school of education.   methods – a survey was designed to measure the value of online and print serials for key faculty activities: research, publishing, course preparation and development, service, and personal interests. measures included: recentness of use, the extent to which library journals supported work in the key activities (minor, moderate, or major), requirement of students to use online or print journals in their courses, cancellations of personal journal subscriptions in favor of library subscriptions, and travel to other libraries to use library journals.   main results – twenty-seven faculty responded to the survey (22%). two of the respondents (7%) had never used the library journals, though the majority (93%) had. of those who used library journals, the most recent use was of online over print publications.   for each key activity, 40-87% of the respondents reported they had never used print journals, and those who did use print reported that it supported their work only to a minor extent, primarily in the area of research. respondents noted they used online journals most frequently for research (92%), publishing (83%), and course preparation and development (76%). service is the least supported by journal use in either print or online, with 87% of the respondents never using print and 50% never using online journals for service.   the respondents who taught undergraduates required the use of online journals over print journals at a ratio of 3:1 for assigned readings, course activities, and writing assignments. the ratio increased to a range of 4.5:1 to 8.5:1 across activities for graduate students. respondents indicated that print (22%) and online journals (72%) had the highest use in assigned readings. the majority of respondents required graduate students to use online journals in all activities and less than a quarter required the use of print.   twenty respondents (80%) had not dropped personal subscriptions, but among those who did, print subscriptions were more likely to be dropped than online. if institutional access were available, 55% indicated that they would drop a personal subscription for online access, and only 27% indicated they would cancel personal subscriptions for print access. those who did drop subscriptions cited cost, storage space, and ease of access to library journals as their motivation. faculty comments praised the serials holdings, especially the holdings of back issues.   finally, the majority of respondents (74%) reported not having traveled to another library for journal access, but those who did, accessed materials for research, class preparation, and publishing. many of those who went to other libraries did so because they were closer to their residence or they needed to access original manuscripts.   conclusion – participants used journal subscriptions for all of the key activities surveyed, with research and publishing the top reasons for use and service the lowest. both undergraduate and graduate students were required to use both print and online journals, with graduate student use being greater for online access. faculty acknowledged their use of print and online journals for key activities to a major extent, with a strong preference for online journals.   commentary   the published literature includes both qualitative and quantitative research methods to determine the value of serials collections. some of the quantitative methods included in the literature review for this article are the cost per download model, the return on investment model, and citation analysis. the authors also indicate some of the potential complications presented by each method. they cite two qualitative studies that used interviews and surveys to assess the value of serials, both of which indicate a high value placed on journal collections, especially those that are available electronically. the authors used surveys to assess the value of the journal collection on a more granular level by asking the degree to which the library’s journal publications affect the teaching faculty’s achievement with regard to research, publishing, course preparation and development, service, and their personal interests.   using glynn’s critical appraisal checklist (2006), the study scored high in the subsections for study design (80%) and results (80%), but not as high in the subsections for population (60%) and data collection (28%). the overall validity calculation (59%) is based on the 4 sub scores, and falls below the 75% score that indicates validity.   the selected population for this survey was 122 faculty in the education and criminal justice programs. the researchers did not offer an explanation for why they limited the study to these two colleges. however, they acknowledged that the low number of participants and the focus on only two disciplines do not allow generalization to all shsu faculty. this does not discount the usefulness of the results to those librarians who are responsible for serials decisions for this population, nor does it invalidate the use of the methodology.   the data collection process was well described and could be replicated by others, though they would need to design their own survey questions. because the survey instrument was not included in the article and because the authors did not indicate that they had validated their survey, mention the number of questions on the survey, or specify the means of survey distribution, the score for data collection was at a level considered invalid. however, outcomes were clearly communicated through text and tables, as were the results of the study, with conclusions in line with the data analysis.   the authors identify areas for further research, including administering the study to a broader pool and to faculty across various disciplines. because their survey responses came from faculty with 15 years or less experience, they would like to solicit responses from longer tenured faculty. additionally, they suggest asking whether the comprehensiveness of a subject area correlates to the perceived value of the serials, and whether providing serials in one format influences the frequency of use in another format.   the design of the study is strong. the authors have given the library community a new application for the tried and true survey method, the results of which can make a statement about library value in the critical area of serials subscriptions. regardless of why this particular study was limited to two disciplines, it may illustrate that the best use of the method is in the departmental level of specificity.   reference   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org /10.1108/07378830610692154 news   call for papers: library assessment conference 2018 (houston, texas december 2018)      2018. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29437     the association of research libraries (arl), the university of washington libraries, and the conference steering committee invite paper and poster proposals for the 2018 library assessment conference: building effective, sustainable, practical assessment. this seventh biennial conference will be held in houston, texas, december 5–7, 2018, and is co-hosted by the university of houston libraries and texas a&m libraries.   the conference goal is to build and further a vibrant library assessment community by bringing together interested practitioners and researchers who have responsibility or interest in the broad field of library assessment. the event provides a mix of invited speakers, contributed papers, posters, and pre-conference workshops that stimulate discussion and provide workable ideas for effective, sustainable, and practical library assessment.   proposal topics   paper and poster proposals that cover any aspect of library assessment in any type of library are invited. the conference steering committee especially encourages assessment-related proposals in the following areas:   ·         collaborative assessments ·         data management and visualization ·         digital libraries and repositories ·         diversity, equity, and inclusion ·         measurement and measures/indicators ·         methods and tools ·         organizational issues ·         research and scholarly life cycle ·         services, collections (including specialized collections), and programs ·         space and facilities planning and use ·         teaching and learning ·         usability ·         user experience ·         user needs ·         value and impact   presentation formats   proposals are invited as papers and posters. brief descriptions of the various formats are provided on the conference website. proposals are required to include a title, author names, format, and abstract (maximum 500 words) describing the paper or poster. papers will be included in the conference proceedings and are due by january 15, 2019.   proposal submission, evaluation, notification, publication   to submit a proposal, please visit the proposal submission site. the primary author will be required to create a profile. one author will complete the form submission and enter co-author information. proposal submissions are due by monday, may 7, 2018, at 11:59 p.m. pacific daylight time.   presenters who are not able to attend in-person due to a prohibition on using state travel funds to texas will be able to present their papers virtually.   the library assessment conference steering committee will evaluate proposals based on:   ·         relevance to effective, sustainable, and practical library assessment ·         significance of contribution to the body of work associated with library assessment ·         clarity of expression ·         status of research (for paper proposals, are the results in hand? when appropriate, please include the timeline for completion of research.) ·         results/findings that can be used to enact change   those submitting proposals will be notified of their status in june 2018. drafts of papers will be due by november 1 and final papers for the proceedings will be due by january 15, 2019.   each accepted presenter will be guaranteed a conference registration place and will be expected to pay the registration fee by september 1, 2018. additional registration information will be available in june 2018.   papers will be published in the conference proceedings, which will be freely and openly available via the conference website. poster abstracts and pdf versions of posters will also be available on the conference website. authors retain the copyright to their original work and are encouraged to publish their work in other established venues and professional journals.   more information   for additional information, visit the library assessment conference website or send email to laconf@arl.org.   about the association of research libraries   the association of research libraries (arl) is a nonprofit organization of 125 research libraries in the us and canada. arl’s mission is to influence the changing environment of scholarly communication and the public policies that affect research libraries and the diverse communities they serve. arl pursues this mission by advancing the goals of its member research libraries, providing leadership in public and information policy to the scholarly and higher education communities, fostering the exchange of ideas and expertise, facilitating the emergence of new roles for research libraries, and shaping a future environment that leverages its interests with those of allied organizations. arl is on the web at arl.org.   about the university of washington libraries   the university of washington (uw) libraries, located in seattle, is the largest library in the pacific northwest. the uw libraries is well-known for its innovative programs and services in assessment, organizational development, user spaces, and institutional collaboration. its print and digital collections support world-class research and scholarship in such areas as health sciences, environmental sciences, area and language studies, and the pacific northwest.   uw libraries is on the web at www.lib.washington.edu.   about the university of houston libraries   the university of houston libraries advances student success, knowledge creation and preservation, and globally competitive research. uh libraries comprises the md anderson library, the william r. jenkins architecture and art library, the weston a. pettey optometry library, and the music library, providing high-impact collections, spaces, and tools that spark opportunities for transformational learning, discovery, and scholarship.   about the texas a&m university libraries   the texas a&m university libraries, located in college station, serve 67,000 students, 3,700 instructional faculty, and 19 colleges and schools. the libraries are driven by: information literacy programs to support university strategic missions, digitization and conservation efforts to protect collections, scholarly communications programs to enhance scholarly visibility, open access projects to reduce student costs, and re-imagining of library spaces to invite collaboration.     evidence summary   lexicomp provides more comprehensive drug information than wikipedia in small sample comparison   a review of: hunter, j. a., lee, t., & persaud, n. (2018). a comparison of the content and primary literature support for online medication information provided by lexicomp and wikipedia. journal of the medical library association: jmla, 106(3), 352-360. http://dx.doi.org/10.5195/jmla.2018.256   reviewed by: lindsay alcock head, public services health sciences library memorial university of newfoundland st. john’s, newfoundland, canada email: lalcock@mun.ca   received: 6 dec. 2018                                                                    accepted: 16 jan. 2019      2019 alcock. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29541     abstract    objective – to compare the content veracity and comprehensiveness of lexicomp and wikipedia with respect to drug information.   design – comparative study.   subjects – lexicomp and wikipedia.   methods – five of the six most commonly prescribed medications in canada were selected for content comparison in both lexicomp and wikipedia (levothyroxine, atorvastatin, pantoprazole, acetylsalicylic acid, and metformin). three categories compared included dose and instructions, uses, and adverse effects or warnings; sixteen subcategories were identified to provide further comparative detail. five outcomes were assessed using a rating scale to identify the presence or absence of each subcategory for each drug entry: present in neither source, present in wikipedia but not lexicomp, present in lexicomp but not in wikipedia, present in both without discrepancies, and present in both with discrepancies. the only subcategory meeting the criteria for “present in both with discrepancies” for all five medications was adverse reactions, indicating that the information in each resource differed. a “fact-checking literature search” in medline and embase as well as searches in the usfda prescribing information (supplemental index) (fda pis) and the fda adverse events reporting systems (fdaers) were used to determine the veracity of the discrepancies. quantitative assessment was used to determine how comprehensive the entries were in terms of the number of times in which each resource provided subcategory information. adverse reaction information was expressed as a percentage based on the number of adverse reactions identified in the sources.   main results – overall, lexicomp was shown to provide more comprehensive information than wikipedia. in the subheading analysis, there was no instance in which wikipedia contained information while lexicomp did not, while in over half of instances lexicomp only contained the information. 18% of subheading information was found in both with discrepancies and 20% was found in both without discrepancies. only 10% of instances were not present in lexicomp or wikipedia. detailed dosing information was consistently present in lexicomp for all five medications while only general dosage information was present in just two instances in wikipedia.   of all the subcategory comparisons, adverse reactions was the only one identified as “present with discrepancies” for all medications being compared; medline, embase, fda pis and the faers dashboard searches were performed for a total of 309 discrepant adverse reactions. 63% (191/302) of the adverse reactions listed in lexicomp were supported by the literature retrieved from medline and embase compared to 100% (7/7) of those listed in wikipedia. of the lexicomp adverse reactions unsupported by the peer-reviewed literature, 17% were supported from information found in fda pis and 90% supported from information found in the faers dashboard. a “substantial proportion” of adverse events listed in lexicomp were not supported in any retrieved literature.   conclusion – based on the comparative criteria, drug information in lexicomp for the five medications was found to be more comprehensive than wikipedia. adverse effects listed in lexicomp did not always have corresponding support in the published peer-reviewed literature.   commentary   the authors note that comparing information for five medications in two resources limits the scope of the study making it less generalizable. therefore, it is difficult to deduct that lexicomp is more comprehensive overall or that wikipedia is less exhaustive in its provision of drug information from this study alone. that said, the methodology is structured and straight-forward and the study reproducible thus providing a mechanism for further research with a more extensive scope. this study adds to the significant body of literature studying the currency, accuracy, and comprehensiveness of medical information found in wikipedia compared to other resources including textbooks (kräenbring et al., 2014), micromedex (reilly, jackson, berger, & candelario, 2016) and medication guides (candelario, vazquez, jackson, & reilly, 2017).   the additional step of fact-checking listed adverse reactions with peer-reviewed literature and fda information added weight to the study while also presenting an interesting avenue for additional research. that a high percentage of adverse reactions were not supported by the literature would be of interest to those studying publication and reporting bias. it highlights the challenge of accuracy and transparency regarding adverse event knowledge transfer. the authors contacted lexicomp regarding their process for adding adverse effects and were told that “they are working to provide better referencing for adverse drug reactions.” a follow-up study would determine the impact of such improvements.   while the process for fact-checking was described, it would have been helpful for the authors to include, as an appendix, the literature search strategy and criteria for eligibility. a limitation of the study, which the authors note, is that screening and fact-checking was done by one person rather than two.   for health information specialists, the results of this study are not especially surprising and serve as a reminder that drug information resources, regardless of the source, require questioning in terms of the sources of the content and exhaustiveness. to recognize the limitations of a source such as lexicomp allows librarians to more effectively utilize and transfer knowledge. but what does it mean for consumers, clinicians, or students studying in the health professions? results can be used to further inform consumers on the importance, or dare i say necessity, of approaching wikipedia with a healthy skepticism while introducing alternatives. for healthcare clinicians and students, the results demonstrate that even licensed resources need to be critically appraised and called to account.   references   candelario, d. m., vazquez, v., jackson, w., & reilly, t. (2017). completeness, accuracy, and readability of wikipedia as a reference for patient medication information. journal of the american pharmacists association, 57(2), 197–200.e1. https://doi.org/10.1016/j.japh.2016.12.063   kräenbring, j., monzon penza, t., gutmann, j., muehlich, s., zolk, o., wojnowski, l. … sarikas, a. (2014). accuracy and completeness of drug information in wikipedia: a comparison with standard textbooks of pharmacology. plos one, 9(9), e106930. https://doi.org/10.1371/journal.pone.0106930   reilly, t., jackson, w., berger, v., & candelario, d. (2016). accuracy and completeness of drug information in wikipedia medication monographs. journal of the american pharmacists association, 57(2), 193–196.e1. https://doi.org/10.1016/j.japh.2016.10.007   the effects of counterproductive workplace behaviors on academic lis professionals’ health and well-being research article   the effects of counterproductive workplace behaviors on academic lis professionals’ health and well-being   christy fic formerly archivist & special collections librarian shippensburg university of pennsylvania shippensburg, pennsylvania, united states of america   maggie albro agriculture & natural resources librarian university of tennessee, knoxville knoxville, tennessee, united states of america email: malbro@utk.edu   received: 21 apr. 2022                                                              accepted: 8 july 2022      2022 fic and albro. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30153     abstract   objective – this study seeks to investigate the degree of counterproductive workplace behaviors (cwb) experienced by library and information science (lis) professionals and how these behaviors contribute to physical, mental, and chronic health outcomes. while health outcomes may be present independent of cwb, this study seeks to explore the relationship between the two to provide context to the growing incidence of burnout among academic lis professionals.   methods – this quantitative study analyzed 327 responses to a survey about colleague behavior and health sent to lis professionals through library community electronic mailing lists. the survey contained demographic questions, questions about cwb, questions about health experiences, and questions about the perceived relationship between work and health. counterproductive workplace behaviors were rated on a seven-point likert scale. a behavior score was calculated by adding the likert values of the 12 behavior questions. this score was used when comparisons about cwb were compared by demographics and health responses. statistical analysis of survey results was performed using rstudio.   results – the mean total behavior score was 39. 107 respondents’ total behavior scores fell in the low range, 202 in the moderate range, and 18 in the high range. there was no significant relationship found between demographic factors and behavior score. a negative relationship was observed between duration of employment in an academic library and presence of mental health issues (f(5, 310) = 10.114, p = 5.5e-09). a similar relationship was observed between duration of employment in the respondents’ current library and presence of mental health issues (f(5, 311) = 9.748, p = 1.15e-08). level of cwb experienced was found to have a relationship with the perceived ability to maintain good mental (f(2, 324) = 36.34, p = 5.75e-15), physical (f(2, 324) = 23.82, p = 2.24e-10), and chronic health (f(2, 323) = 13.04, p = 3.57e-06). generally speaking, lower levels of cwb were associated with fewer challenges maintaining health.   conclusion – low to moderate levels of cwb are common in academic libraries. these behavior levels are associated with an increase in health challenges. lis professionals perceive work as being a factor that contributes to having trouble maintaining good mental and physical health and toward successfully managing chronic health conditions. further study is needed to determine the degree to which experiencing cwb in the workplace affects health. further study is also needed to determine if certain behaviors impact health outcomes more than others.     introduction   counterproductive workplace behaviors (cwb) encompass a wide range of detrimental employee actions. cwb are commonly understood to include employee behaviors that “harm their organization or organization members, such as theft, sabotage, interpersonal aggression, work slowdowns, wasting time and/or materials, and spreading rumors” (penney & spector, 2002, p. 126). this study explores the degree of counterproductive workplace behaviors experienced by academic librarians and archivists and how these behaviors contribute to physical, mental, and chronic health outcomes. while library and information science (lis) professionals may have pre-existing health conditions independent of facing counterproductive workplace behaviors, this study seeks to investigate the relationship between the two in order to provide context to the growing incidence of burnout among academic librarians and archivists.   to examine this issue, the researchers disseminated a survey about colleague behavior and health to lis professionals through library community electronic mailing lists. this study will provide insight into trends that contribute to burnout among academic information professionals. the interprofessional dynamics of librarians and archivists have not been thoroughly studied. knowledge of the types of counterproductive workplace behaviors coworkers engage in will allow academic libraries to mitigate their occurrence, and this may enhance morale and work satisfaction. preventing negative behaviors that cause lis professionals stress may reduce burnout.   literature review   librarians and library leaders have shown great interest in burnout’s impact on the lis profession. alves et al. (2019) noted that burnout is negatively associated with academics’ quality of life, including their physical and mental health. when examining the issues that contribute to librarian burnout, previous studies have focused on external factors including funding streams, salaries, sabbatical leaves, work-life balance, patron interactions, understaffing, heavy workloads, management decisions, and the demands of the tenure clock (badia, 2018; flaspohler, 2009; galbraith et al., 2006; heady et al., 2020; howlett, 2019; johnson, 2018; kennedy & garewal, 2020; nardine, 2019; petek, 2018; sheesley, 2001; shupe et al., 2015; spires, 2007). an important area that researchers have not explored is the connection between academic librarians’ and archivists’ interprofessional relationships and their well-being. several contributing factors related to burnout served as foundational elements of this study: workplace climate and organizational culture, counterproductive workplace behaviors, and the role of library management.   the trouble with the existing lis literature on burnout is that it focuses on issues individual librarians and archivists cannot control. many studies explore the impact of workplace climate and organizational culture (affleck, 1996; ajala, 2011; akakandelwa & jain, 2013; albanese, 2008; bartlett, 2014; blessinger & hrycaj, 2013; mccormack & cotter, 2013; hall, 2015; heady et al., 2020; henry et al., 2018; mchone-chase, 2020; nardine, 2019; oyintola et al., 2014; spires, 2007; steiner, 2018). additionally, while lis research is deeply concerned with how to improve the experience of library patrons, there is little attention given to making libraries better workplaces for employees (blessinger & hrycaj, 2013).   organizational culture creates the systemic factors that contribute to employee burnout. according to a pivotal study on the topic, howard schein (1984) defines organizational culture as “the pattern of basic assumptions that a given group has invented, discovered or developed in learning to cope with its problems of external adaptation and internal integration, and that have . . . to be taught to new members as the correct way to perceive, think and feel in relation to those problems” (p. 3). in academic libraries, many of the “problems” schein describes are related to the work environment. more recently, heady et al. (2020) found that academic librarians were most dissatisfied with their work environment followed by other factors including compensation, professional responsibilities, and personal issues. they discovered that unsupportive organizational cultures and poor management contributed to low morale, which caused librarians to leave their current positions.   library management has the power to shape the work environment in academic libraries. unfortunately, library managers often fail to address counterproductive workplace behaviors such as incivility and bullying. freedman and vreven’s 2016 survey of association of college & research libraries (acrl) members revealed that when library administrators noticed bullying behaviors such as coworkers withholding information or excluding colleagues, they failed to stop it. while 53% of librarians surveyed witnessed bullying in the workplace, only 46% of administrators reported they had witnessed bullying. freedman and vreven (2016) noted that “these findings suggest a gap in bullying perception between library administrators and librarians [and] . . . this result is a clear example of how library leadership and, in particular, avoiding confrontation are motivating structures for bullying” (p. 740). a survey of american library association (ala) members, conducted in 2018, found that the vast majority of librarians (91%) have experienced incivility at work (henry et. al.). ala’s respondents suggested that libraries could create a more civil workplace through stronger leadership and setting clear expectations regarding behavior.   although lis scholars have begun to explore counterproductive workplace behaviors in academic libraries (bartlett, 2016; freedman & vreven, 2016) and librarians’ mental health (burns & green, 2019), additional research needs to be conducted to understand the variety of coworker behaviors that can lead to stress and burnout. while kendrick (2017) found that low morale can be triggered by abuse from coworkers, the current literature is limited in that it does not explore the less extreme counterproductive workplace behaviors that may lead to stress, burnout, and other health conditions. while these actions may not be as severe as bullying or harassment, they may impact librarians’ mental and physical health. much of the current literature on this subject takes the form of personal essays. in her column about how new librarians can deal with difficult coworkers, stephanie walker (2011) acknowledges that colleagues may be rude, insensitive, manipulative, lazy, incompetent, and deceitful. she recommends that librarians “try to see the reason behind the behavior” (p. 182), “talk to the coworker . . . talk to your supervisor . . . [and] talk to a unit designated to help with employee complaints” (p. 183). however, those suggestions place the burden of change on newer, often younger, less experienced librarians with lower organizational status. these types of personal experience think pieces do not address the need for systemic change within academic libraries.   aims   the purpose of this study was to expand upon the research focused on burnout among academic librarians and archivists. specifically, this article explores the connection between challenging coworkers and librarian and archivist health. two research questions guided this study:   to what degree are counterproductive workplace behaviors occurring in academic libraries? how does the degree of counterproductive workplace behaviors experienced contribute to physical, mental, and chronic health outcomes among lis professionals?   methods   a survey (see appendix) was used to collect data about colleague behavior and mental and physical health of academic librarians in the united states. the survey contained demographic information, questions about specific counterproductive workplace behaviors, and questions about health status and maintenance. cwb were selected for inclusion based on anecdotal discussions among the authors, discussion with librarians they spoke to about their study, and other surveys and validated measures that featured negative acts (freedman & vreven, 2016; heady et al., 2020; henry et al., 2018; spector & jex, 1998).   the survey, available through qualtrics, was distributed widely to the academic library community through professional electronic mailing lists in late october 2020. librarians over the age of 18 who were employed in an academic library at the time of the survey distribution were eligible to participate, and respondents self-selected for participation. the survey received 356 responses. there were 29 responses discarded due to incompleteness, leaving 327 responses for analysis.   counterproductive workplace behaviors were rated on a seven-point likert scale, and the likert responses were transformed into numerical values with “strongly agree” receiving the value of 1 and “strongly disagree” the value of 7. a behavior score was calculated by adding the likert values of the 12 colleague behavior questions. this score was used when comparisons about cwb were compared by demographics and health responses. scores less than 31 were considered low. scores between 31 and 67 were considered moderate. scores greater than 67 were considered high. statistical analyses of survey results (one-way anova, ancova, and tukey-hsd, as well as summary statistics) were performed using rstudio (2020).   results   demographics   327 survey responses were used for analysis. respondents worked across several areas of the library, with some respondents working in multiple areas (see table 1). 64 (19.6%) respondents worked at a college, 39 (11.9%) at a community college, 219 (67.0%) at a university, and 5 (1.5%) at some other type of institution.     table 1 areas of the library in which survey participants work area of the library number of participants access services 45 technical services 52 collection management 74 reference 169 instruction 165 archives and/or special collections 65 management 91 other 38     the duration of time respondents have worked in academic libraries ranged from zero to 31+ years (see table 2). the same range of time was again seen when participants were asked how long they have worked in their current library (see table 2).   table 2  duration of time employed in academic libraries and in current library duration of time employment in academic libraries employment at current library 0-5 years 67 154 6-10 years 90 70 11-15 years 52 37 16-20 years 43 27 21-30 years 50 27 31+ years 25 12     respondents' ages varied as follows: 31.8% were between 22 and 37 years of age, 42.2% were between 38 and 53 years of age, 25.7% fell between the ages of 54 and 72, and 0.3% were between the ages of 73 and 90.   counterproductive workplace behaviors   respondents reported varied experiences with counterproductive workplace behaviors. the cwb reported as most common were gossip about colleagues (48.9%), requests for last-minute coverage of duties (43.4%), lack of appropriate record keeping (42.5%), lack of initiative (40.1%), and refusal to take responsibility for one’s own actions (36.7%). behavior scores ranged from 12 to 80 out of a possible total of 84. the mean total behavior score was 39. there were 107 respondents with total behavior scores in the low range, 202 in the moderate range, and 18 in the high range (see table 3).     table 3 behavior score outcomes score range number of respondents low (<31) 107 medium (31-67) 202 high (>67) 18     behavior scores were examined in the context of respondents’ demographic factors. one-way anovas found no significant relationship between behavior score and type of institution (f(3,323) = 2.453, p = .0633), duration of employment in an academic library setting (f(5,321) = 2.036, p = .0732), duration of employment at current library (f(5,321) = 0.662, p = 0.653), or age (f(3,323) = 0.26, p = 0.854).   health   respondents were asked if they had any mental, physical, or chronic health issues. the results were varied: 58.72% of respondents agreed in some way to experiencing mental health issues, 40.98% agreed in some way to having issues with physical health, and 39.45% indicated they experience a chronic health condition.   mental health   responses to the presence of mental health issues were examined in the context of respondents' demographic factors. an ancova showed a significant relationship between duration of employment in an academic library and presence of mental health issues (f(5, 310) = 10.114, p = 5.5e-09). this relationship was not observed with age (f(3, 310) = 1.023, p = 0.383), and there was no combined effect of age and duration of employment (f(7, 310) = 1.302, p = 0.249). a post-hoc tukey-hsd showed different durations of employment saw different overall levels of the presence of mental health issues (see table 4). specifically, the longer someone is employed in an academic library setting, the lower the level of mental health issues seen.     table 4 differences in the level of mental health issues present based on duration of employment in an academic library setting duration of employment mean level of mental health issues present standard deviation group 0-5 years 4.85 1.59 a 6-10 years 4.74 1.76 a 11-15 years 4.18 1.66 ab 16-20 years 4.12 1.58 ab 21-30 years 3.66 1.97 b 31+ years 2.44 1.56 c note. group a is significantly different from groups b and c. group b is significantly different from groups a and c. group c is significantly different from groups a, ab, and b.     a similar relationship was observed by ancova between duration of employment in the current library and presence of mental health issues (f(5, 311) = 9.748, p = 1.15e-08). again, the relationship was not observed with age (f(3, 311) = 2.051, p = 0.107) or as a combined effect of age and duration of employment in the current library (f(6, 311) = 0.947, p = 0.462). post-hoc tukey-hsd showed different durations of employment at the current library had different overall levels of the presence of mental health issues (see table 5). similar to overall duration of employment, the longer someone is employed in their current library, the lower the level of mental health issues seen.     table 5 differences in the level of mental health issues present based on duration of employment in current library duration of employment mean level of mental health issues present standard deviation group 0-5 years 4.75 1.63 a 6-10 years 4.35 1.79 ab 11-15 years 3.62 1.91 bc 16-20 years 4.56 1.65 ab 21-30 years 3.07 1.84 c 31+ years 2.25 1.54 c note. group a is significantly different from groups bc and c. group c is significantly different from groups a and ab.     physical health   responses to the presence of physical health issues were examined in the context of respondents’ demographic factors. an ancova showed no significant relationship between duration of employment in an academic library and presence of mental health issues (f(5, 310) = 0.810, p = 0.543). this relationship was not observed with age (f(3, 310) = 0.337, p = 0.798), and there was no combined effect of age and duration of employment (f(7, 310) = 1.031, p = 0.409).   similarly, no significant relationship was observed by ancova between duration of employment in the current library and presence of physical health issues (f(5, 311) = 0.388, p = 0.857). again, the relationship was not observed with age (f(3, 311) = 0.287, p = 0.834) or as a combined effect of age and duration of employment in the current library (f(6, 311) = 0.801, p = 0.570).   chronic health   responses to the presence of a chronic health condition were examined in the context of respondents’ demographic factors. an ancova showed no significant relationship between duration of employment in an academic library and presence of a chronic health condition (f(5, 310) = 0.923, p = 0.466). this relationship was not observed with age (f(2, 310) = 0.324, p = 0.723), and there was no combined effect of age and duration of employment (f(7, 310) = 0.631, p = 0.730).   similarly, no significant relationship was observed by ancova between duration of employment in the current library and presence of a chronic health condition (f(5, 311) = 0.417, p = 0.837). again, the relationship was not observed with age (f(2, 311) = 0.471, p = 0.625) or as a combined effect of age and duration of employment in the current library (f(6, 311) = 0.880, p = 0.510).   health and work   respondents were asked if work made it difficult to maintain mental or physical health or manage chronic health conditions. the results were varied: 54.43% of respondents indicated that work makes it difficult to maintain good mental health, 44.34% indicated work makes it difficult to maintain good physical health, and 21.71% indicated work makes it difficult to manage a chronic health condition.   mental health    level of cwb experienced was found to have an effect on whether or not it was difficult to maintain good mental health due to work by one-way anova (f(2, 324) = 36.34, p = 5.75e-15). a post-hoc tukey-hsd test found that those who experienced low levels of cwb (m = 3.07, sd = 1.68) experienced a statistically significant difference in their perception of work’s effect on maintaining good mental health as compared to those who experienced moderate (m = 4.57, sd = 1.60) and high (m = 5.50, sd = 1.62) levels of cwb. this implies that once a threshold of cwb is reached, there is a greater toll on mental health.   physical health   level of cwb experienced was found to have an effect on whether or not it was difficult to maintain good physical health due to work by one-way anova (f(2, 324) = 23.82, p = 2.24e-10). a post-hoc turkey-hsd test found that those who experienced low levels of cwb (m = 2.82, sd = 1.62) experienced a statistically significant difference in their perception of work’s effect on maintaining good physical health as compared to those who experienced moderate (m = 4.04, sd = 1.62) and high (m = 4.78, sd = 1.90) levels of cwb. this implies that once a threshold of cwb is reached, there is a greater toll on physical health.   chronic health   level of cwb experienced was found to have an effect on whether or not it was difficult to manage a chronic health condition due to work by one-way anova (f(2, 323) = 13.04, p = 3.57e-06). a post-hoc tukey-hsd test found that there were differences in perception of work’s effect on the ability to manage a chronic health condition between all three levels of cwb experienced (see table 6). this implies that as more cwb is experienced, it becomes a greater challenge to manage chronic health conditions.     table 6 differences in perception of work’s effect on ability to manage chronic health condition(s) based on level of counterproductive workplace behavior experienced behavior score level mean perception of effect on chronic health management standard deviation group low 2.49 1.60 c moderate 3.23 1.71 b high 4.39 1.75 a     discussion   counterproductive workplace behaviors appear to be pervasive in academic libraries at a low to moderate level. exposure to, and experience with, these behaviors is not dependent upon demographic factors such as type of institution, duration of employment, or age. this suggests a level of consistency to these behaviors that is not necessarily a product of place or experience. since participants across the board reported experiencing cwb, the organizational culture and expectations within academic libraries must permit these behaviors to proliferate. this is a systemic issue in academic libraries that may impact the well-being and effectiveness of lis professionals more broadly than this survey was able to capture.   health issues were present at a moderate level among survey respondents. when these health issues were examined in the context of demographic factors, connections were only observed in the area of mental health. a significant relationship was observed between duration of employment in an academic library and presence of mental health issues. librarians who had shorter durations of employment either in an academic library setting, or in their current library, experienced a higher level of mental health issues. these findings may be related to the uncertainty many librarians experience early in their careers, especially if they are in term or contract positions and their future employment is unknown. additionally, newer hires must adjust to their job, which is typical of any workplace. lis professionals who have worked in academic librarianship, or at their current library, for a shorter duration are also most likely to be facing the demands of tenure and promotion processes, while those who have been with the institution longer may have already achieved tenure and promotion. these factors could all contribute to higher stress levels and pronounced mental health conditions in librarians and archivists who are either new to the profession or to their current library.   the percentage of respondents who indicated work made it difficult to manage health conditions was similar to the percentage of respondents who experienced health conditions in the first place, suggesting the work experience may differ depending on pre-existing conditions. this corresponds to burns & green’s (2019) finding that many academic librarians with invisible disabilities, including mental illness, believe their disorder negatively impacted their work, and that stress at work exacerbated their health condition. further research is needed to verify the connection between pre-existing conditions and library workplace stressors.   level of cwb experienced was found to have a relationship with the perceived ability to maintain good mental, physical, and chronic health. generally speaking, as the level of cwb increased, a greater level of mental, physical, and chronic health difficulties was experienced. previous research indicates that librarians will leave, or consider leaving, workplaces that permit cwb. heady et al. (2020) found that although librarians’ reasons for leaving an institution vary, librarians “are not fleeing their positions, they are fleeing work environments they feel are toxic” (p. 591). low morale and burnout, which can be caused by toxic work environments, leads to higher turnover (kennedy & garewal, 2020). other studies (mchone-chase, 2020) have found that individuals coped with toxic work environments by practicing self-care. this method puts the burden of mitigating systemic problems on individual librarians and archivists. library leaders need to ensure safe and healthy working environments so that their employees can focus on providing the services and resources patrons expect.   limitations   this study faced several key limitations. the sampling method used for survey distribution did not provide for a comprehensive sampling of all academic librarians and archivists. recruitment was conducted via email notification to nine local and national electronic mailing lists the authors had access to, as well as through posts on two national facebook groups for librarians and archivists. those who participated in the survey self-selected to be included in the sample, meaning their experiences may not be representative of the entire population of academic lis professionals.   the survey itself was delivered without definitions. this allowed for participants to interpret the questions, which may have led to some inconsistencies in responses. additionally, the survey did not account for demographic factors, such as race or gender, that could have an impact on the amount of conflict experienced. future studies are needed to explore the intersections of demographic factors on experience with cwb, health consequences, and burnout.   conclusion   based on the findings of this study, library managers and lis professionals considering moving into leadership positions should seek additional education and training on change management to learn how to reduce counterproductive workplace behaviors. colleen harris-keith’s (2015) doctoral dissertation on academic library work experience and leadership development showed that academic librarians lacked opportunities to develop measurable leadership skills. they only had the chance to grow as leaders once they were promoted to a top position such as library director. jennifer bartlett (2014) argues that for librarians “leadership does not come naturally . . . we enter leadership roles from other specialties with no formal management training” (p. 5). library deans and directors are often librarians who get promoted with limited management experience, which explains why they may struggle to be effective and supportive managers. in their study of the relationships between leadership, interpersonal conflict, and counterproductive workplace behavior, kessler et al. (2013) found that transformational leadership was associated with a decrease in conflict, while passive or avoidant leadership correlated with negative emotions and counterproductive workplace behavior. it is management’s responsibility to enforce consequences for counterproductive workplace behavior, take reports of these behaviors seriously, and resolve reported issues.   managers operating in a unionized environment should consult relevant collective bargaining agreements and determine what methods they can use to minimize employees’ counterproductive workplace behaviors. these managers may be able to develop a performance improvement plan for an employee with a history of engaging in such behaviors. alternatively, managers may need to be more involved in training employees on appropriate workplace conduct and make themselves more visible.                additionally, academic library leaders should build a workplace culture that reduces mental and physical challenges on librarians and archivists’ health:   ●       establish a community of practice centered on emotional intelligence (gola & martin, 2020) or other issues personnel face. ●       conduct regular stress assessments of employees to spot negative trends and react in a timely manner to mitigate stressors. ●       determine strategies for clear and effective communication between management and personnel, and between coworkers. ●       review job descriptions and assess library needs to align individual duties with what needs to be done, resulting in reasonably dispersed workloads. ●       establish behavioral expectations for all employees, focusing on healthy workplace culture.   by making changes that prioritize the mental and physical well-being of employees, academic libraries can reduce the occurrence of burnout among librarians and archivists, ultimately creating a stronger, more empowered workforce. when employees do not have to worry about support structures for facing challenging coworkers, dealing with burnout, or coping with mental and physical health challenges, they are able to perform to the best of their potential. supporting initiatives that reduce these stressors on employees can go a long way toward transforming the library workforce.   notes   the methodology for this study was approved by the institutional review board at both shippensburg university of pennsylvania and clemson university.   author contributions   christy fic: conceptualization, investigation, methodology, project administration, writing – original draft, writing – review & editing maggie albro: conceptualization, data curation, formal analysis, investigation, methodology, project administration, writing – original draft   references   affleck, m. a. (1996). burnout among bibliographic instruction librarians. library & information science research, 18(2), 165–183. https://doi.org/10.1016/s0740-8188(96)90018-3   ajala, e. b. (2011). work-related stress among librarians and information professionals in a nigerian university. library philosophy & practice, 21–31.   akakandelwa, a., & jain, p. (2013). a comparative study of perceived work related stress among library staff in two academic libraries in southern africa. library management, 34(8/9), 569–584. https://doi.org/10.1108/lm-03-2013-0018   albanese, a. r. (2008). take this job and love it. library journal, 133(2), 36–39.   alves, p. c., oliveira, a. d. f., & paro, h. b. m. d. s. (2019). quality of life and burnout among faculty members: how much does the field of knowledge matter? plos one, 14(3). https://doi.org/10.1371/journal.pone.0214217   badia, g. (2018). forty ways to survive il instruction overload; or, how to avoid teacher burnout. college & undergraduate libraries, 25(1), 65-71. https://doi.org/10.1080/10691316.2017.1364077   bartlett, j. a. (2014). “that’s how we do things here:” organizational culture in libraries. library leadership & management, 28(3), 1-6.   bartlett, j. a. (2016). workplace bullying: a silent epidemic. library leadership & management, 31(1), 1–4.   blessinger, k., & hrycaj, p. l. (2013). workplace culture in academic libraries: the early 21st century. oxford: chandos publishing.   burns, e., & green, k. e. c. (2019). academic librarians’ experiences and perceptions on mental illness stigma and the workplace. college & research libraries, 80(5), 638–657. https://doi.org/10.5860/crl.80.5.638   flaspohler, m. r. (2009). librarian sabbatical leaves: do we need to get out more? journal of academic librarianship, 35(2), 152–161. https://doi.org/10.1016/j.acalib.2009.01.009   freedman, s., & vreven, d. (2016). workplace incivility and bullying in the library: perception or reality? college & research libraries, 77(6), 727–748. https://doi.org/10.5860/crl.77.6.727   galbraith, q., fry, l., & garrison, m. (2016). the impact of faculty status and gender on employee well-being in academic libraries. college & research libraries, 77(1), 71–86. https://doi.org/10.5860/crl.77.1.71   gola, c. h., & martin, l. (2020). creating an emotional intelligence community of practice: a case study for academic libraries. journal of library administration, 60(7), 752–761. https://doi.org/10.1080/01930826.2020.1786982   hall, l. w. (2015). changing the workplace culture at flinders university library: from pragmatism to professional reflection. australian academic & research libraries, 46(1), 29–38. https://doi.org/10.1080/00048623.2014.985773   harris-keith, c. s. (2015). an exploratory study of the relationship between academic library work experience and perceptions of leadership skill development relevant to academic library directorship [doctoral dissertation, university of tennessee at chattanooga]. eric.   heady, c., fyn, a. f., foster kaufman, a., hosier, a., & weber, m. (2020). contributory factors to academic librarian turnover: a mixed-methods study. journal of library administration, 60(6), 579–599. https://doi.org/10.1080/01930826.2020.1748425   henry, j., eshleman, j., croxton, r., & moniz, r. (2018). incivility and dysfunction in the library workplace: perceptions and feedback from the field. journal of library administration, 58(2), 128–152. https://doi.org/10.1080/01930826.2017.1412708 howlett, a. (2019). differences in work/life balance and stress at work between male and female academic librarians. evidence based library & information practice, 14(4), 188–190. https://doi.org/10.18438/eblip29649   johnson, a. m. (2018). librarian sabbaticals: overcoming the hurdles and realizing the benefits. college & research libraries news, 79(11), 607–624. https://doi.org/10.5860/crln.79.11.607   kendrick, k. d. (2017). the low morale experience of academic librarians: a phenomenological study. journal of library administration, 57(8), 846–878. https://doi.org/10.1080/01930826.2017.1368325   kennedy, s. p., & garewal, k. r. (2020). quantitative analysis of workplace morale in academic librarians and the impact of direct supervisors on workplace morale. journal of academic librarianship, 46(5). https://doi.org/10.1016/j.acalib.2020.102191   kessler, s. r., bruursema, k., rodopman, b., & spector, p. e. (2013). leadership, interpersonal conflict, and counterproductive work behavior: an examination of the stressor–strain process. negotiation and conflict management research, 6(3), 180–190. https://doi.org/10.1111/ncmr.12009   mccormack, n., & cotter, c. (2013). managing burnout in the workplace: a guide for information professionals. oxford: chandos publishing.   mchone-chase, s. (2020). library burnout: it’s common and okay to admit! ila reporter, 38(2), 24-28.   nardine, j. (2019). the state of academic liaison librarian burnout in arl libraries in the united states. college & research libraries, 80(4), 508–524. https://doi.org/10.5860/crl.80.4.508   oyintola, i. a., abdsalam, a. s., & ajani, f. o. (2014). occupational frustration variables of the librarians in public universities in south-west nigeria. the information manager, 14(1&2), 38-45.   penney, l. m., & spector, p. e. (2002). narcissism and counterproductive work behavior: do bigger egos mean bigger problems? international journal of selection and assessment, 10(1-2), 126–134. https://doi.org/10.1111/1468-2389.00199   petek, m. (2018). stress among reference library staff in academic and public libraries. reference services review, 46(1), 128–145. https://doi.org/10.1108/rsr-01-2017-0002   rstudio team. (2020). rstudio: integrated development for r. boston, ma: rstudio, pbc.   schein, e. h. (1984). coming to a new awareness of organizational culture. sloan management review (pre-1986), 25(2), 3-16.   sheesley, d. f. (2001). burnout and the academic teaching librarian: an examination of the problem and suggested solutions. journal of academic librarianship, 27(6), 447–451. https://doi.org/10.1016/s0099-1333(01)00264-6   shupe, e. i., wambaugh, s. k., & bramble, r. j. (2015). role-related stress experienced by academic librarians. journal of academic librarianship, 41(3), 264–269. https://doi.org/10.1016/j.acalib.2015.03.016   spector, p. e., & jex, s. m. (1998). development of four self-report measures of job stressors and strain: interpersonal conflict at work scale, organizational constraints scale, quantitative workload inventory, and physical symptoms inventory. journal of occupational health psychology, 3(4), 356-367. https://doi.org/10.1037/1076-8998.3.4.356   spires, t. (2007). the busy librarian: prioritizing tenure and dealing with stress for academic library professionals. illinois libraries, 86(4), 101–108.   steiner, s. (2018). burnout culture shift: strategies and techniques for preventing and addressing library worker fatigue and demotivation. international information & library review, 50(4), 319-327. https://doi.org/10.1080/10572317.2018.1526832   walker, s. (2011). dealing with difficult colleagues when you’re the “new kid.” bottom line: managing library finances, 24(3), 180–184. https://doi.org/10.1108/08880451111186026     appendix librarian professional relationships and well-being survey   demographics in which area of the library do you work? (select all that apply.) access services technical services collection management reference instruction archives and/or special collections management other:  _______________________ at what type of institution do you work? community college college university other:  _______________________ how long have you worked in an academic library setting? 0-5 years 6-10 years 11-15 years 16-20 years 21-30 years 31+ years how long have you worked at your current library? 0-5 years 6-10 years 11-15 years 16-20 years 21-30 years 31+ years what is your age? 21 or younger 22-37 38-53 54-72 73-90 91+   coworker behaviors rate each of the following items from strongly disagree to strongly agree. coworkers ask you to cover their responsibilities, or fill in, at the last minute. coworkers show lack of initiative. coworkers do not contribute to group projects or discussions coworkers do not respect your time. coworkers do not record data that is necessary to assess unit goals (example: reference transitions, instruction statistics, etc.) coworkers treat you like a child. coworkers tell you how (or how not) to do your job. coworkers talk down to you. coworkers stifle your creativity by saying things like “that’s not how we do things here.” coworkers limit your ability to pursue new initiatives. coworkers do not take responsibility for their actions. coworkers gossip about you or other colleagues.   health and work please rate each of the following items from strongly disagree to strongly agree. i experience mental health issues. work makes it difficult to maintain good mental health. i have issues with my physical health. work makes it difficult to maintain good physical health. i experience a chronic health condition. work makes it difficult to manage my chronic health condition(s).   news/announcements   call for proposals: identity, agency, and culture in academic libraries conference      2016. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     the identity, agency, and culture in academic libraries conference will take place may 22nd and 23rd, 2017 in los angeles, california at the university of southern california. please see our call for proposals below. deadline for proposals is 5pm (pst), october 1, 2016. proposals must be submitted using the submissions form (http://bit.ly/2bkll1l). more information will be shared soon on the conference website (http://iacal2016.wixsite.com/iacal2017). we hope you can join us!   lists of trends in academic libraries and higher education do not always make explicit connections to the changing roles of librarians, the shifting identity required to tackle these new trends and roles, or the agency that librarians may or may not experience as they work at the vanguard of these transformations. in a 2014 article (http://www.cais-acsi.ca/ojs/index.php/cais/article/viewfile/888/808), deborah hicks remarked, “by focusing on how librarians describe their profession, attention can be drawn to how librarians themselves construct librarianship, and how this construction shapes their interactions with patrons, their local communities, other professions, and society at large.” this conference will extend ongoing conversations aimed at understanding how issues of identity and agency contribute to access, inclusivity, and diversity in academic libraries. in the current information and higher education landscapes, we cannot succeed by only performing the roles and responsibilities laid out in our job descriptions. this conference will also raise awareness of, begin to reflect on, and in some cases interrogate, how our roles and identities affect interactions with stakeholders, as well as our perceptions of each other and of libraries and librarians in society at large.   this conference will include keynotes, workshops, and discussions spread across the two days. we invite proposals that address questions of identity, agency, and culture in academic libraries for sessions that will take place on may 22nd and 23rd, 2017 in any of the following formats:   presentations and/or panels, 50 minutes facilitated roundtable discussions, 50 minutes workshops, 50 minutes lightning talk sessions, 10 minutes per person   library students and library employees of all kinds are invited to submit proposals. there will be student and staff scholarships to help with travel and lodging costs. the application for these scholarships will be available in december on the conference website.   topics may include, but are not limited to:   queering the workplace for people and resources technology and professional identity developing a professional identity or focus, and the fear of getting pigeonholed, and/or coping with impostor syndrome how librarians’ official status (e.g., as faculty) affects how we interact with teaching faculty, students, and staff agency in collections decisions, negotiations, and the importance of considering that the systems we use are profit-driven valuing library work while maintaining, or deconstructing, job titles generational and cultural differences among library employees how intersectionality affects workplace culture differences (and similarities) between library science and information science degrees peer mentoring and other mentoring models impact of professional recognition and awards on the kind of work we value how cultivating professional skills impacts collections and resources how the language we use on our websites and in our buildings includes and excludes maintaining professional identity with diverse professional backgrounds (for example, phds with no mlis hired as librarians, or librarians working in the academy outside the library) academic culture and the effect on agency   please submit proposals by october 1, 2016 at 5pm pst using the submissions form. proposals must include presenters’ name(s) and institutional affiliation(s), a working title, and a description of your session no more than 300 words long. proposals will be evaluated based on adherence to the theme of identity, agency, and culture in academic libraries and on their appeal to a broad audience. an effort will be made to represent a wide variety of viewpoints, roles, and institutions. decisions and notifications of acceptance will be made by december 1, 2016.   questions, comments, or concerns can be sent to iaclatusc@gmail.com   evidence summary   do systemic inequities lead to differences between information behaviors of older adults in the usa and india during the covid-19 pandemic?   a review of: lund, b. d., & maurya, s. k. (2022). how older adults in the usa and india seek information during the covid-19 pandemic: a comparative study of information behavior. ifla journal, 48(1), 205–215. https://doi.org/10.1177/03400352211024675     reviewed by: christine fena undergraduate success librarian stony brook university libraries stony brook, new york, united states of america email: christine.fena@stonybrook.edu   received: 25 nov. 2022                                                             accepted:  26 jan. 2023      2023 fena. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30257     abstract   objective – to investigate and compare the information-seeking behaviors of older adults in one developing and one developed country during the covid-19 pandemic.   design – structured interviews via zoom (video), telephone, or email.   setting – two towns with moderately large populations (about 300,000), one in eastern india and one in the midwest of the usa.   subjects – sixty adults ages 65 and older, 35 in the india cohort and 25 in the usa cohort.   methods – the researchers recruited participants from the communities in which their respective institutions are located by using online advertisements in facebook groups, local (print) advertisements/flyers, and word of mouth. the ten interview questions were informed by dervin’s (1998) sense-making methodology and sought to identify a specific information need, behavior to address the need, and the influences on and outcomes of the behavior. they conducted the interviews in july and august of 2020, translated the questions into hindi for hindi-speaking participants, and analyzed responses using qualitative content analysis. within each of the resulting themes and categories, the researchers compared the responses of american and indian participants.   main results – the researchers found many significant differences between the information behaviors of indian and american participants. some of the biggest differences were in the information needs expressed by the participants, as well as the sources consulted and the reasons for consulting those sources. for example, when asked about the types of information needed, 77% of indians focused on a “covid and health-related” information need, as opposed to only 33% of americans. and 37% of americans indicated information needs related to “political and economic issues,” especially the upcoming 2020 election, as opposed to only 3% of indians. when asked about sources, 28% of indians consulted television, compared to only 6% of americans. web-based sources were generally used more by americans, with 31% of americans consulting websites, compared to 13% of indians. in regard to their reasons for consulting a source, 28% of indians chose a source based on availability, compared to only 9% of americans. and 32% and 36% of americans chose information based on ease and familiarity (“i know how to find it”), compared to only 18% and 13% of indians, respectively. only 3% of indians met all their information needs, as opposed to 43% of americans, and indians were more likely to stop searching after encountering barriers. americans had more confidence in their information behavior overall, and only 32% of americans were interested in taking a class on how to find information, as opposed to 97% of indians.   conclusion – older adults in developing and developed countries described very different information-seeking experiences. the disparities between the types of information sought, sources consulted, and barriers encountered highlight not only cultural differences, but also systemic inequities that exist between the information infrastructure of the two countries, especially as concerns access to computers and the internet. the study points to areas for future improvement, including the need for interventions such as information literacy instruction.   commentary   many research areas contextualize this study, including the digital divide, the impact of socioeconomic status, the issues facing older adults as a population, and the role of information access in mitigating a global pandemic and creating communities that are health literate and achieve mental wellness. xie et al. (2020) recognized the interconnected relationship between information crises and global health crises. those most vulnerable to a lack of information access also become vulnerable to the health crisis. research on information behaviors of older adults demonstrates a unique set of challenges, including willingness to adapt to new technologies (berkowsky et al., 2017) and the role of internet use in reducing depression (cotten et al., 2012). furthermore, hargittai and dobransky (2017) point to the role of socioeconomic status in one’s web-using skills.   this study was appraised using “the cat: a generic critical appraisal tool” (perryman & rathbun-grubb, 2014), and has many strengths. the authors have expertise in information behavior and are affiliated with the university of north texas and brm government model college, respectively. the methodology and objective are well matched. the methods used – dervin’s (1998) sense-making approach, interview questions, and content analysis – address the complexity of the diverse research contexts well, since they are flexible ways to qualitatively investigate information behavior and identify gaps in individual information seeking. another strength of the study is that the authors are transparent in locating themselves within the towns from which the participants were recruited; one of the authors lives in the american town and the other lives in the indian town. finally, the authors adequately represent the hindi-speaking population by ensuring the interview questions were translated into hindi. they included the interview questions in both english and hindi within an appendix.   as the authors point out in their discussion of the study’s limitations, however, the sample size was not broad-based or the results statistically strong. although the authors do list their methods of recruitment, they do not include their selection criteria for the 60 participants. they also leave out more detailed information about preparatory actions related to the study and development and execution of the interviews, such as whether they obtained irb approval, methods of writing and piloting the interview questions, whether they considered the cultural relevance of the interview questions for both locations, and how many participants were interviewed within each modality (video, telephone, email). in the reporting of the results, they leave out demographic information beyond whether the participants were indian or american. additionally, as information and conditions change throughout the covid-19 pandemic one might expect responses to the interview questions would also change, and thus these results only represent a “snapshot in time” (lund & maurya, 2022, p. 213).   despite these weaknesses, the study demonstrates the extent to which information access impacted two groups of older adults differently in india and the u.s. at a specific historical moment. practice implications include the potential desire and need for information literacy instruction within the indian community studied, and the need for further research to determine the desire for instruction in similar communities. finally, the authors point to the importance of library administrators’ understanding of how cultural differences and infrastructure constraints impact the delivery of services and resources.   references   berkowsky, r., sharit, j., & czaja, s. (2017). factors predicting decisions about technology adoption among older adults. innovation in aging, 1(3). https://doi.org/10.1093/geroni/igy002   cotten, s., ford, g., ford, s., & hale, t. (2012). internet use and depression among older adults. computers in human behavior, 28(2): 496–499. https://doi.org/10.1016/j.chb.2011.10.021   dervin, b. (1998) sense-making theory and practice: an overview of user interests in knowledge seeking and use. journal of knowledge management 2(2): 36–46. https://doi.org/10.1108/13673279810249369   hargittai, e., & dobransky, k. (2017). old dogs, new clicks: digital inequality in skills and uses among older adults. canadian journal of communication, 42(2): 195–212. https://doi.org/10.22230/cjc.2017v42n2a3176   lund, b. d., & maurya, s. k. (2022). how older adults in the usa and india seek information during the covid-19 pandemic: a comparative study of information behavior. ifla journal, 48(1), 205–215. https://doi.org/10.1177/03400352211024675   perryman, c., & rathbun-grubb, s. (2014). the cat: a generic critical appraisal tool. http://www.jotform.us/cp1757/thecat   xie, b., he, d., mercer, t., want, y., wu, d., fleischmann, k., zhang, y., yoder, l., stephens, k., mackert, m., & kyung lee, m. (2020). global health crises are also information crises: a call to action. journal of the association for information science and technology, 71(12): 1419–1423. https://doi.org/10.1002/asi.24357   research article   a systematic review of information literacy programs in higher education: effects of face-to-face, online, and blended formats on student skills and views   alison l. weightman director, specialist unit for review evidence (sure) university library service cardiff university cardiff, united kingdom email:  weightmanal@cardiff.ac.uk   damian j. j. farnell lecturer in medical statistics school of dentistry cardiff university cardiff, united kingdom email: farnelld@cardiff.ac.uk   delyth morris subject librarian university library service cardiff university cardiff, united kingdom email: morrisd13@cardiff.ac.uk   heather strange research associate se wales trials unit cardiff university, united kingdom email: strangehr1@cardiff.ac.uk   gillian hallam information literacy project manager university of queensland brisbane, australia email: g.hallam@library.uq.edu.au   received: 8 feb. 2017      accepted: 2 aug. 2017         2017 weightman, farnell, morris, strange, and hallam. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – evidence from systematic reviews a decade ago suggested that face-to-face and online methods to provide information literacy training in universities were equally effective in terms of skills learnt, but there was a lack of robust comparative research. the objectives of this review were (1) to update these findings with the inclusion of more recent primary research; (2) to further enhance the summary of existing evidence by including studies of blended formats (with components of both online and face-to-face teaching) compared to single format education; and (3) to explore student views on the various formats employed.   methods – authors searched seven databases along with a range of supplementary search methods to identify comparative research studies, dated january 1995 to october 2016, exploring skill outcomes for students enrolled in higher education programs. there were 33 studies included, of which 19 also contained comparative data on student views. where feasible, meta-analyses were carried out to provide summary estimates of skills development and a thematic analysis was completed to identify student views across the different formats.   results – a large majority of studies (27 of 33; 82%) found no statistically significant difference between formats in skills outcomes for students. of 13 studies that could be included in a meta-analysis, the standardized mean difference (smd) between skill test results for face-to-face versus online formats was -0.01 (95% confidence interval -0.28 to 0.26). of ten studies comparing blended to single delivery format, seven (70%) found no statistically significant difference between formats, and the remaining studies had mixed outcomes. from the limited evidence available across all studies, there is a potential dichotomy between outcomes measured via skill test and assignment (course work) which is worthy of further investigation. the thematic analysis of student views found no preference in relation to format on a range of measures in 14 of 19 studies (74%). the remainder identified that students perceived advantages and disadvantages for each format but had no overall preference.   conclusions – there is compelling evidence that information literacy training is effective and well received across a range of delivery formats. further research looking at blended versus single format methods, and the time implications for each, as well as comparing assignment to skill test outcomes would be valuable. future studies should adopt a methodologically robust design (such as the randomized controlled trial) with a large student population and validated outcome measures.      introduction   the provision of information literacy (il) education for students is an established and valued role within university libraries. there are many definitions of il but this can be broadly described as, “knowing when and why you need information, where to find it, and how to evaluate, use and communicate it in an ethical manner” (cilip, 2017). il training has been shown to result in an increase in student skills and understanding compared to no instruction (koufogiannakis & wiebe, 2006; weightman, farnell, morris & strange, 2015).   around a decade ago, two systematic reviews of il interventions in higher education looked at the specific question of online versus face-to-face instruction in academic libraries (koufogiannakis & wiebe, 2006; zhang, watson & banfield, 2007). both reviews concluded that online provision was as effective as face-to-face training in terms of skills learned but noted the lack of robust comparative studies.   since the reviews were published, further studies of ‘taught’ student il provision comparing traditional versus online delivery have been completed, including studies looking at blended (with components of both online and face-to-face teaching) compared to single format delivery. there are suggestions from the library setting of theoretical benefits to a blended approach (such as the ‘flipped classroom’ where students study online in advance of the face to face session), particularly for the more technical and practical skills involved in information literacy (arnold-garza, 2014). the potential benefits of blended teaching include the effective use of class time, more active learning, allowance of individual learning styles, and speed (arnold-garza 2014). such techniques are increasingly being used across academic settings, suggesting that these will become the ‘new traditional model[s]’ (brown, 2016).   a recent meta-analysis of 45 studies of online and face-to-face learning across the education and subject spectrum, from secondary to higher education, concluded that students in online learning conditions performed modestly better than those receiving face-to-face instruction. however, this analysis indicated a significant difference only for the blended versus face-to-face and not the online versus face-to-face   conditions (means, toyama, murphy & baki, 2013). the authors noted that blended formats tended to involve additional learning time and resources which could explain the findings. a further systematic review and meta-analysis of 44 studies exploring knowledge acquisition in health education (liu et al., 2016) concluded that blended learning was more effective, or at least as effective, as single format learning but that the result should be treated with caution given the huge variation between studies.   we could not identify any review level evidence from the il literature on blended versus other learning formats with similar curricula/contact times and ‘hard’ outcomes such as skills acquisition. neither was there a systematic summary of student views on the different formats.   thus, the aims of this research study were to carry out an up-to-date systematic review of research into il programs in higher education to:   (i)                  confirm or refute the findings of the earlier reviews in terms of the relative effectiveness of traditional (face-to-face) and online (web or computer based) educational provision by the inclusion of more recent studies; (ii)                expand the scope of the review to include comparative studies of blended versus single format delivery; and (iii)              systematically explore the views of research participants from each study on their perceptions of the differing formats.   methods   we undertook a systematic review of controlled studies to summarize the findings of comparative research studies using both quantitative and qualitative methods. we extracted data on student skills as assessed after exposure to each delivery format and completed a thematic analysis of student views identified within the research.   studies were identified via a comprehensive search for published and unpublished papers comparing face-to-face and online information literacy programs using database searching and supplementary search methods.    search strategy    we searched seven relevant databases for formally published research publications or ‘grey literature’ in higher education or libraries in october 2016:  british education index; eric; proquest dissertations and theses (index to theses); librarians’ information literacy annual conference (lilac) abstracts; library, information science & technology abstracts (lista); loex conference abstracts; open grey; scopus.   text words and phrases were identified from the authors’ knowledge of the subject area and existing known literature. text mining for common words and phrases using the free software, termine (national centre for text mining 2012) was also used to identify the most relevant search terms to use in text word searching. this software used the titles and abstracts from a set of 42 papers that explored information literacy education taught to students in universities. a set of search terms and associated subject headings were developed for lista (table 1) and then adapted for each database.   we sought recent studies (from january 1995 onwards) to assure relevance to the modern and higher speed internet architecture, and the wide-scale adoption of database searching in libraries.   in addition, the extensive use of supplementary search methods increased the sensitivity of the search (i.e., the ability to identify the vast majority of relevant papers). these methods included reference list follow up, unpicking of related systematic reviews for primary research studies, citation tracking (via scopus and google scholar), expert contact and hand searching of the 2016 editions of a number of journals: college and research libraries; communications in information literacy; evidence based library and information practice; health information & libraries journal; journal of academic librarianship; journal of information literacy; journal of the medical library association; portal: libraries & the academy.     table 1 search terms for lista s1 and s2 and s3 (1995-2016) s3 ti (test score or learning outcome or effective* or student performance or control group or randomised or pretest or pre-test or posttest or post-test or randomized or trial or controlled or efficacy or impact or evaluat*) or ab (test score or learning outcome or effective* or student performance or control group or randomised or pretest or pre-test or posttest or post-test or randomized or trial or controlled or efficacy or impact or evaluat*)  s2 (de "college students" or de "college freshmen" or de "college seniors" or de "college transfer students" or de "first generation college students" or de "graduate students" or de "in state students" or de "on campus students" or de "out of state students" or de "preservice teachers" or de "two year college students" or de "undergraduate students" ) or ( ti ( college student* or freshman or first-year or undergrad* or freshmen or sophomore* or universit* or higher education or academic or taught postgraduate*) or ab ( college student* or freshman or first-year or undergrad* or freshmen or sophomore* or universit* or higher education or academic or taught postgraduate*) )  s1 de information literacy or ti ( (information litera* or library instruct* or library skill* or acrl il standard or information competen* or bibliographic instruct* or library research or il concept or instruction librarian) or ((research skill* or electronic information or information retrieval or ebm skill or electronic resource* or instructional method or user train* or user education or literacy instruct* or hands-on instruction or research strateg* or evidence-based or print workbook or instructional format or social medi* learning or online tutor*) and librar*)  ab: word(s) in the abstract; de: descriptor (assigned by indexer); s: set of terms; ti: word(s) in the title; *= truncation term.       inclusion and exclusion criteria   the criteria for selection of studies are provided in table 2. the training had to be described as information literacy or library skills, with a statement that equivalent content was covered within each format to avoid any potential for bias as a result of differing curricula.    study selection   after removing duplicates and clearly irrelevant citations (unrelated to library-based training), study selection at both title/abstract and full-text stages was undertaken independently by two authors. any disagreements at either stage were resolved by recourse to a third reviewer.   quality assessment and data extraction  two authors independently appraised each included study using criteria specifically developed for educational interventions. we used the glasgow checklist for educational interventions (morrison, sullivan, murray & jolly, 1999), adapted to include the questions from the reliant checklist for library based educational interventions (koufogiannakis, booth & brettle, 2005). a quality commentary for each paper was agreed by discussion and these commentaries, along with summary data from each study on skill related outcomes and any student views, were extracted by one author and checked by another. the study detail, including the il content of each intervention, was summarized in the detailed data extraction table (see appendix) with summary data provided in table 3.   data synthesis  we carried out a synthesis of the findings across the body of evidence on skills outcomes and student views.   we combined the study findings for skills outcomes by meta-analysis when studies provided means, sample sizes, and standard deviations for the outcomes. meta-analysis forms a pooled result based on all studies by finding an average of the outcomes from each study. for fixed-effects meta-analysis, the results of each study are “weighted” by the variance (i.e., the overall standard error squared) for the difference in means for each study when forming this average. thus, those studies that are more accurate (often those studies with larger sample sizes) make a greater contribution to the result. a similar weighting occurs for random effects meta-analysis, except that heterogeneity (in variances and effects sizes) is accounted for also in the weighting process. the included studies used different types of tests (and thus had different maximum possible test scores) so a standardized mean difference (smd = difference in means divided by the standard deviation) was employed.     table 2 inclusion/exclusion criteria population ·         undergraduates and postgraduates enrolled in higher education coursework programs intervention ·         an information literacy intervention comparing face-to-face and online delivery formats with a formal assessment of student skills (via a test, diagnostic essay, or end-of-course exam) comparators 1.       face-to-face 2.       online 3.       ‘blended’ (with face-to-face and online components) outcomes primary outcome ·         change in information literacy skills secondary outcomes ·         student views on the educational format(s) limits studies published since january 1995 types of evidence  included randomized and non-randomized controlled studies exclusions ·         sessions for research postgraduates, unless as part of a formal ‘taught’ program, such as a research methods course ·         sessions for professional trainees, not based at the university (e.g. junior health professionals based in hospital or primary care sites)  ·         comparisons involving differing face-to-face formats only, or differing online formats only ·         different curricula for each learning format ·         students not from the same cohort (e.g. different year groups for different formats)       a forest plot (lewis & clark, 2001) shows both the results of each individual study and the pooled results of meta-analysis. the pooled results are identified by the diamonds within the forest plot, where the middle of the diamond gives the pooled point-value estimate for the smd and its edges give the associated 95% confidence interval (ci). for specific studies, the point-value estimate of the smd is indicated by the central symbol and the associated 95% ci for the smd is indicated by the horizontal line. an overall meta-analysis that included all studies, irrespective of subgroup, was carried out using standard statistical software (stata v13). when the number of studies included in meta-analysis was large enough (i.e., equal to or greater than about 10 studies), any evidence of bias was assessed by funnel plots, egger’s and begg’s test of small sample size effects.   heterogeneity was assessed by i2 scores and p < 0.05 from a chi-squared test of heterogeneity before deciding whether to carry out a random-effects or fixed effects meta-analysis. random-effects meta-analysis takes into account both the variability within each individual study (shown by the confidence intervals for each study) and variability between the different studies (i.e., variability of the point-estimates of the smd). this approach tends to lead to larger confidence intervals than fixed-effects meta-analysis, which includes only variability within each individual study.   (1)     we also carried out a thematic analysis of information on student views, where available within the comparative studies, using methods described by braun and clarke (2006) to generate descriptive themes. initially, each paper was examined line by line, by two authors independently. codes (features of the options expressed) were assigned to relevant sentences and paragraphs. these codes were then organized, via discussion, into related areas to construct descriptive themes that best reflected students’ views on the different teaching formats. all data on student views from each paper were then imported into nvivo 10 software (qsr international pty ltd., 2012) for analysis.   results   of 5,313 records identified via the various search strategies employed (figure 1), 33 studies met the inclusion criteria for providing a direct comparison between traditional and online il education, and these studies were included in the review. summary data from all studies are provided in table 3. detailed information on study characteristics and the results of skills assessments is available (see appendix).   study quality   of the 33 studies, 11 were randomized controlled trials (brettle & raynor, 2013; churkovich & oughtred, 2002; goates et al., 2016; greer et al., 2016; koenig & novotny, 2001; kraemer et al., 2007; lechner, 2007; schilling, 2012; shaffer, 2011; swain et al., unpub; vander meer & rike, 1996), whereas the remaining studies were (non-randomized) controlled before and after studies.    the vast majority of research was carried out in the u.s. (26 studies; 79%). of the remaining seven studies, three were based in the u.k. (brettle & raynor, 2013; walton & hepworth, 2012; swain et al., 2015 unpub.), two in australia (churkovich & oughtred, 2002; salisbury & ellis, 2003), one in canada (bordignon et al., 2016) and one in the czech republic (kratochvil, 2014).   the 11 studies that used a randomized controlled design were less prone to bias since the study design increased the likelihood that the student groups were well matched.  however, most of the studies had some methodological limitations (table 3).    of the 33 studies, 25 did not pilot or validate the test instrument. only two studies carried out formal validity testing (brettle & raynor, 2013; mery et al., 2012a) with a further five piloting the test before use (bordignon et al. 2016; burhanna et al., 2008; churkovich & oughtred, 2002; kratochvil, 2014; swain et al., 2015 unpub.). finally, one study used a predetermined rubric for marking (goates et al., 2016).   of the 33 studies, 17 included mean il test scores with standard deviations and could be included in the meta-analyses (alexander & smith, 2001; anderson & may, 2010; beile & boote, 2005; brettle & raynor, 2013; churkovich & oughtred, 2002; germain, jacobson & kaczor, 2000; goates, nelson & frost, 2016; greer, hess & kraemer, 2016; lantzy, 2016; mery, newby & peng, 2012a; shaffer, 2011; silk, perrault, ladenson & nazione, 2015; swain, weightman, farnell & mogg unpub.; vander meer & rike, 1996; walton & hepworth, 2012; wilcox brooks, 2014).   the results from the studies were ‘heterogeneous’ (i.e., effect sizes or variances varied considerably) and so a random-effects meta-analysis was used. a sensitivity analysis was carried out in order to study the effects of heterogeneity that was here driven by just one or two "outlying" studies in each comparison. these studies were systematically removed from the meta-analyses. this process did not change the overall results of meta-analysis very greatly: i.e., effect sizes and associated 95% confidence intervals remained broadly constant and the statistical significance (or not) of all two-group comparisons remained unchanged. clearly though, caution should be exercised when interpreting pooled results of meta-analysis when the heterogeneity is high.       figure 1 flow diagram (‘n’ indicates the number of studies).       of the 33 studies, 21 provided data on participants’ views (anderson & may, 2010; beile & boote 2005; burhanna, eschedor voelker & gedeon, 2008; byerley, 2005; churkovich & oughtred, 2002; gall, 2014; goates et al., 2016; holman, 2000; kaplowitz & contini, 1998; koenig & novotny, 2001; kraemer, lombardo & lepkowski, 2007; lantzy, 2016; nichols, shaffer & shockey, 2003; nichols hess, 2014; schilling, 2012; shaffer, 2011; silk et al., 2015; silver & nickel, 2007; swain et al., unpub; vander meer & rike, 1996; wilhite, 2004). in all cases this information related to views expressed by students rather than the library staff delivering the interventions (table 3).     table 3 summary of included studies study details population and setting methods outcomes: skills outcomes: views   limitations first author and year: alexander 2001   study design: cba, posttest only   delivered by: graduate student (ftf); course coordinator (online)   setting: western kentucky university, u.s.   participants: 88 undergraduates on library media course interventions: (1)     face-to-face (2)     online   hours of contact time: 14x 1h course (face-to-face) vs. self-paced (online) neutral no pretest. mean scores posttest for skill levels: 82.6 (traditional) and 85 (online).   follow-up period: n/s   favoured online preference for the online course in terms of: ·         perceived benefits/effectiveness of course (p<0.05) ·         comfort in doing library research (p<0.01). researcher was both teacher and investigator. students self-selected for online course. no pretest. no piloting or validation of test. no information on participant loss.   first author and year: anderson 2010   study design: cba   delivered by: librarian setting: university of north texas, u.s.   participants: 103 undergraduates on introduction to communication course interventions: (1)     face-to-face (2)     blended (3)     online   hours of contact time: entire course: 3 x 50 minute sessions neutral skills increased with no significant differences between formats (p>0.1) other than research assignment (persuasive presentation) scores higher for online (p=0.000).   follow-up period: 5 weeks   teaching content, student characteristics & treatment may have varied between groups. no information on characteristics. no validation of tests. pretest scores high so difficult to assess any benefit. first author and year: beile 2005   study design: cba   delivered by: librarian setting: university of central florida, u.s.   participants: 49 postgraduates on research methods course interventions: (2)     face-to-face (3)     blended (4)     online   hours of contact time: ftf 70 min. online ~80 min   neutral skills increased with no significant differences between formats.   follow-up period: n/s   neutral confidence/self-efficacy levels increased in all groups with no significant differences between formats.   teaching content, student characteristics & treatment may have varied between groups. no information on characteristics. no validation of tests. response rates varied. first author and year: bordignon 2016   study design: cba   delivered by: librarian setting: seneca college, toronto, canada   participants: 110 undergraduates on foundation english composition course   interventions: (1)     online videos (2)     ftf   hours of contact time: not stated neutral skills increased in both formats with no clear differences between them.     follow-up period: immediately post-training no information on student characteristics. participation was optional and students self-selected.  mcqs changed for the two groups. no overall test results. first author and year: brettle 2013   study design: rct   delivered by: librarian setting: university of salford, u.k.   participants: 77 undergraduate nursing students interventions: (1)     face-to-face (2)     online   hours of contact time: 1 hour neutral skills increased (p=0.001) with no significant differences between formats (p=0.263).   follow-up period: 1 month   loss of participants was explained but only 71% completion and no intention to treat analysis.     first author and year: burhanna 2008   study design: cba   delivered by: librarian setting: kent state university, ohio u.s.   participants: 313 undergraduates on orientation program   interventions: library tour (1)     face-to-face (2)     online   hours of contact time: 0.5h   no pretest. neutral greater understanding of library services in online group (92% compared with 82.6%; no significance levels) although no difference in knowledge gained. follow-up period: n/s neutral the majority of students in both formats agreed that ·         the course was effective/beneficial and they were ·         comfortable in asking for help from library staff ·         more comfortable in doing library research ·         more likely to use the library       students self-selected type of course, and whether they participated in survey. over half of in-person participants selected by instructor. no pretest. no validation of test.   first author and year: byerley 2005   study design: cba   delivered by: librarian setting: university of colorado, u.s.   participants: 141 undergraduates in english 141 course     interventions: (1)     face-to-face (2)     blended – ftf with online (3)     online   hours of contact time: not stated neutral skills increased slightly in each group. the mean score for the blended group was significantly different from the ftf although not the online group.   follow-up period: ~8 weeks unclear no useable data – views of online groups only were sought. ftf course introduced three databases while online course introduced only one. different numbers for each format and no information on characteristics. test not piloted or validated. first author and year: churkovich 2002   study design: crct     delivered by: librarian setting: deakin university, geelong, australia   participants: 174 undergraduate sociology students interventions: (1)     face-to-face (2)     blended (3)     online   hours of contact time: unclear favoured face-to-face skills increased in each group with a greater improvement in ftf compared to other formats (statistically significant).   follow-up period: n/s favoured face-to-face there was no difference in confidence/self-efficacy levels of the ftf and blended classes although a significant improvement in both compared to the online only course. there was a clear preference for the class compared to the online course with 14/15 positive comments versus 3/9 positive comments.   group sizes and student origins varied and no information on characteristics. test trialed although only with secondary students & comments from academic staff. no data on statistical significance. first author and year: gall 2014   study design: cba   delivered by: librarian setting: university of iowa, u.s.   participants: 27 postgraduates in social work on campus (numbers off campus unclear) interventions: library induction (1)     face-to-face (2)     online (3)     no instruction   hours of contact time: ftf 50 mins. online self-paced neutral skills increased in each group although no significant differences between groups.   follow-up period: n/s   favoured online? online orientation ‘seemed to’ increase confidence/self-efficacy in choosing databases (awareness of library resources). small sample size. no useable posttests for no instruction (off campus) group. no information on characteristics.  loss of participants not discussed. test not piloted or validated. no confidence intervals or statistical tests. first author and year: germain 2000   study design: cba   delivered by: librarian   setting: university at albany, new york, u.s.   participants: 303 undergraduate on gen. education program   interventions: (1)     face-to-face (2)     online   hours of contact time: ftf 55 mins online 15-55 mins   neutral skills increased in each group with no differences between formats.   follow-up period: 1.5 to 6 weeks     numbers varied between groups and no information on student characteristics. tests not validated. first author and year: goates 2016   study design: rct   delivered by: librarian   setting: brigham young university, utah, u.s.   participants: 122 undergraduates (primarily life sciences) on advanced writing course.   interventions: (1)     face-to-face (2)     blended   hours of contact time: 50 mins no pretest favoured face-to-face assignment scores (a rubric graded search strategy) were higher for students receiving ftf format (p<0.01)   follow-up period: immediately after training neutral positive comments on perceived effectiveness of skills development similar for both formats   randomization method not described. no information on student characteristics. first author and year: greer 2016 linked to kraemer 2007   study design: crct   delivered by: librarian   setting: oakland university, michigan, u.s.   participants: 257 undergraduates on writing & rhetoric course interventions: (1) online (2) blended   hours of contact time: online self-paced? blended self-paced? plus 1h instruction no pretest neutral the exam scores of the two groups were nearly identical.   follow-up period: unstated but short-term no information on student characteristics or drop outs.  test not validated.     first author and year: holman 2000   study design: cba   delivered by: librarian setting: university of north carolina at chapel hill, u.s.   participants: 125 undergraduates on english composition and rhetoric course interventions: (1)     face-to-face (2)     online (cai) (3)     no instruction   hours of contact time: ftf: 40 or 60 mins. cai 30 45 mins neutral skills increased in each group with no statistically significant difference between formats.   follow-up period: n/s   neutral no perceived differences in effectiveness/benefits.  pace of online course and clarity of ftf course preferred. low completion rate online. length/intensity of formats varied. posttest timing varied.  groups were different sizes and minimal information on characteristics. no piloting or validation of test. first author and year: kaplowitz 1998   study design: cba   delivered by: teaching assistants   setting: ucla, u.s.   participants: 423 biology undergraduates   interventions: (1) face-to-face (lecture) (2) online (cai)   hours of contact time: 50 minutes (lecture), 45-60 minutes (cai) neutral skills increased in each group with no differences between formats.   follow-up period: ~12 months   unclear no useable data – views of online group only were sought. no information on group characteristics. no content info/validation of test. only those completing pre/posttests evaluated. no confidence intervals or p values. first author and year: koenig 2001   study design: rct   delivered by: librarian setting: university of illinois at chicago, u.s.   participants: undergraduates (number unstated) on a communication course       interventions: (1)     fact to face (2)     online   hours of contact time: ftf unclear online 50 mins   neutral skills increased in each group with no differences between formats.   follow-up period: n/s (‘end of module’)   neutral confidence/self-efficacy increased in both groups although no difference between groups. information lacking on timing/mode of ftf session. students self-selected for format. tests not validated.  drop outs noted although numbers on the course not stated. first author and year: kraemer 2007   linked to greer 2016 study design: crct   delivered by: librarian setting: oakland university, michigan, u.s.   participants: 224 undergraduates on rhetoric composition class   interventions: (1)     face-to-face (2)     blended online plus ftf (3)     online (webct) hours of contact time: ftf 3h. blended self-paced plus 2h. online self-paced favoured blended skills increased in each group (p<0.0000) with a significantly greater pre-post improvement in the blended compared to the online only group (p=0.023). follow-up period: n/s     neutral similar levels of satisfaction (perceived effectiveness/benefits) across groups. high pretest scores (~70%) limited value of test scores. lack of information on student characteristics. test not piloted or validated.   first author and year: kratochvil 2014   study design: cba   delivered by: librarian setting: masaryk university, czech republic   participants: 251 medicine undergraduates & postgraduates interventions: (1)     face-to-face (2)     online   hours of contact time: unclear. possibly 3x2.5h sessions for ftf unclear skills increased in each group although unclear if any differences between groups. follow-up period: n/s     unsuitable question construction in test and not validated. different student groups for each format. no information on numbers or characteristics. could have been major differences in treatment. first author and year: lantzy 2016   study design: cba   delivered by: librarian   setting: california state university, u.s.   participants: 64 undergraduates in a kinesiology course   interventions: (1)     face-to-face (2)     online   hours of contact time: 1.25 hours neutral both groups showed highly significant pre-post test score increases (p<0.0001) but there were no significant differences between groups.   follow-up period: immediately after training neutral no significant differences across formats in views re: ·         confidence/self-efficacy ·         clarity of presentation ·         responsiveness of instructor no information on student characteristics. tests were not piloted or validated.   first author and year: lechner 2007   study design: rct   delivered by: librarian setting: richard stockton college of new jersey, u.s.   participants: 27 occupational/physical therapy postgraduates   interventions: (1)     face-to-face (2)     online   hours of contact time: not stated. online probably self-paced. favoured face-to-face % change pre to post = 8.1% for the online group and 18.1% for the ftf group. follow-up period: n/s (probably same day)           different sized groups and no information on characteristics. only 63% completed both tests. much higher pretest scores in online group. no confidence intervals or p values. first author and year: mery 2012a, 2012b   study design: cba   delivered by: ftf: tutor (1); librarian (2); online: librarian     setting: university of arizona, u.s.   participants: 660 undergraduates on english compositional course interventions: (1)     face-to-face (tutor) (2)     face-to-face (librarian) (3)     online   hours of contact time: ftf 50 mins. online over 10 weeks favoured online skills increased significantly in the ftf librarian and online groups but not in the tutor group. the online group performed better than ftf groups in both skills test (mery 2012a) and assignment scores (bibliography quality) (mery 2012b). follow-up period: n/s content and delivery varied between formats. no student characteristics and some selection by instructors. much larger online group (570 students compared to circa 30 in other groups). no discussion of participant loss.   first author and year: nichols 2003   study design: cba   delivered by: librarian setting: state university of new york (suny), u.s.   participants: 64 undergraduates on english composition course   interventions: (1)     face-to-face (2)     online   hours of contact time: ftf 50 mins. online unclear   neutral skills increased slightly in each group although no difference between groups.   follow-up period: n/s   neutral no differences between groups re: ·         perceived benefits/effectiveness ·         satisfaction ·         confidence levels ·         preference for format   no information on the characteristics of each group. test not described or validated.  no information on loss of participants.   first author and year: nichols hess 2014   study design: cba   delivered by: librarian   setting: oakland university, rochester, u.s.   participants: 31 undergraduate sociology students   interventions: (1)     face-to-face (2)     online   hours of contact time: ftf not stated online self paced neutral skills increased in each group with no difference between groups.   follow-up period: up to two months neutral no significant differences between formats in: ·         comfort in asking for help ·         using library resources students receiving ftf instruction valued the personal connection and responsiveness of instructor. those receiving online instruction valued the convenience and ability to repeat sections. very little methodological information. different numbers in each group and no information on student characteristics. test not piloted or validated. only completers analyzed. not possible to assess statistical significance of results.   first author and year: orme 2004   study design: cba   delivered by: librarian   setting: indiana university, u.s.   participants: 128 business undergraduates   interventions: (1)     face-to-face (2)     blended online (tilt) plus ftf (3)     online only (tilt)   hours of contact time: unstated   neutral no pretest. no statistically significant difference between groups.   follow-up period: ~10 weeks (next semester)   only students designated as ‘successful’ (passing tilt quizzes or seminar) were included in the study. exact content, length and intensity of teaching for each cohort not clear. test not validated. no pretest. first author and year: salisbury 2003   study design: cba   delivered by: information specialist setting: university of melbourne, australia   participants: 282 history/film undergraduates   interventions: (1)     face-to-face (lecture) (2)     face-to-face (hands on) (3)     online   hours of contact time: 1 hour   neutral skills increased in each group although no clear differences between groups.   follow-up period:  n/s   no detail on content, length or intensity of each mode of delivery. no student characteristics. no validation of test. no confidence intervals or p values. first author and year: schilling 2012   study design: rct   delivered by: librarian   setting: indiana university, u.s.   participants: 128 medical undergraduates   interventions: (1)     face-to-face (2)     online   hours of contact time: 1.5 h neutral no statistically significant difference between groups in medline searching score.   follow-up period: two weeks for skills test: 15 weeks for attitudes survey   no pretest neutral no significant differences between formats in terms of: ·         perceived effectiveness ·         likelihood of using library (more) no information on student characteristics. no validation of test. no confidence intervals with results.   first author and year: shaffer 2011   study design: rct   delivered by: librarian setting: university of new york at oswego, u.s.   participants: 59 postgraduates on a research methods course interventions: (1)     face-to-face (2)     online   hours of contact time: ~2 hours neutral skills increased in each group although no difference between groups.   follow-up period: n/s   favoured face-to-face* the ftf group had higher satisfaction scores on the 5-point likert scale (4.03 viz 3.41).   tests were not validated. *online group experienced technical difficulties.   first author and year: silk 2015   study design: cba   delivered by: librarian setting: midwestern university, u.s.   participants: 232 undergraduates on an organization communication course   interventions: (1)     face-to-face (2)     online   hours of contact time: 1 hour   neutral skills increased in each group with no significant difference between groups. the online group was more successful in finding research articles (87.4% vs. 78.0%, p=0.063).   follow-up period: 4 weeks neutral no significant differences in: ·         confidence/self-efficacy ·         engagement/dynamism of instruction.   no information on student characteristics. tests not piloted or validated. only those who completed post and delayed posttest were included ca 50% attrition in ftf and 59% in online. first author and year: silver 2007   study design: cba   delivered by: librarian setting: university of south florida, u.s.   participants: 295 psychology undergraduates interventions: (1)     face-to-face (2)     online   hours of contact time: ftf not stated. online self-paced (allowed one week ) neutral no pretest. no posttest difference between groups.   follow-up period: n/s   unclear marginally greater number in online group saying they were more confident or much more confident after instruction (88.4% vs. 78.3% for ftf).   students allowed to self-select group. student characteristics varied (and different year groups were used). test was not validated. no pretest. first author and year: swain 2015   study design: rct   delivered by: librarian setting: cardiff university, u.k.   participants: 58 dental undergraduates   interventions: (1)     face-to-face (2)     online   hours of contact time: ftf 50 mins. online:  self-paced within 50 min slot neutral skills increased in each group although no significant difference between groups.   follow-up period: 5 days   neutral overall no significant differences in ·         comfort in asking for library assistance ·         preference for format other than tendency to favour of the format allocated.    limited information on characteristics. test was piloted although not validated. only 58 students attended training but 60 claimed training received at posttest.   first author and year: vander meer 1996   study design: rct   delivered by: librarian setting: western michigan university, u.s.   participants: 186 undergraduates on high school/university transition course interventions: (1)     face-to-face (2)     online   hours of contact time: not stated. neutral skills increased in each group although no significant difference between groups (p<0.05).   follow-up period: ~10 weeks (end of semester) neutral no difference in perceived: ·         confidence/self-efficacy ·         clarity ·         interest online group perceived greater enjoyment (p=0.05) all students had access to tutorial. test not piloted or validated. only 53% completion of posttest. no characteristics although large samples with similar baseline skill and survey results. first author and year: walton 2012   study design: cba   delivered by: librarian   setting: staffordshire university, u.k.   participants: 35 sport and exercise undergraduates   interventions: (1)     blended (2)     intermediate: ftf plus access to online materials (3)     face-to-face   hours of contact time: blended 4x50 mins others 50 mins   no pretest neutral students in the blended group made greater use of evaluative criteria than those in the intermediate or ftf groups but this was not statistically significant.   follow-up period: not stated, possibly at end of 5 week intervention period groups different sizes and no student characteristics.  assessors not blinded to group. evaluation criteria not validated. small sample size.  four times as much contact time for the blended vs. ftf and intermediate formats. first author and year: wilcox brooks 2014   study design: cba   delivered by: librarian   setting: northern kentucky university, u.s.   participants: 38 undergraduates in advanced composition courses interventions: (1)     blended (2)     face-to-face   hours of contact time: not stated neutral no significant differences between groups in bibliographic analysis of final course paper.   follow-up period: not stated unclear no useable data – views of the blended group only were sought. hours of contact time not stated. no information on student characteristics. outcome measures not piloted or validated   first author and year: wilhite 2004   study design: cba   delivered by: librarian setting: university of oklahoma, u.s.   participants: 44 business undergraduates interventions: (1)     face-to-face (2)     online (3)     no instruction   hours of contact time: not stated. 45 min video neutral skills increased in each group when compared to control (p=0.010) although no significant difference between intervention groups (p=0.75).   follow-up period: n/s   favoured face-to-face general preference for ftf with higher scores from ftf group for satisfaction clarity length of course slightly different numbers in groups and pretest scores are very different suggesting characteristics varied across groups. test not piloted or validated. issues for online group. cba:  controlled before and after study; crct: cluster randomized controlled trial; ftf: face-to-face; n/s: not stated; rct: randomized controlled trial shaded rows are papers included in the meta-analysis.   skills   of the 33 studies, 8 did not include a pretest (alexander & smith, 2001; burhanna et al., 2008; goates et al., 2016; greer et al., 2016; orme, 2004; schilling, 2012; silver & nickel, 2007; walton & hepworth, 2012). the remaining 25 studies all noted an increase in skills from pretest to posttest across delivery formats.   a total of 12 studies could be included in a meta-analysis, which indicated that a significant increase in skills occurred from preto posttest. the overall result from meta-analysis for the smd change was 1.02 (95% confidence interval [ci]: 0.75 to 1.29) for face-to-face delivery (figure 2) and 0.92 (95% ci: 0.57 to 1.26) for online delivery (figure 3).   overall, and as suggested by the preto post-results, there was compelling evidence that skills acquired through il teaching are comparable for face-to-face and online delivery methods. of the 33 studies, 27 (82%) reported that there was no statistically significant difference in skills learned via face-to-face and online delivery formats. for one study the results were unclear because of analysis weaknesses (kratochvil, 2014), two favoured online delivery (lechner, 2007; mery et al., 2012a), two favoured face-to-face delivery (churkovich & oughtred, 2002; goates et al., 2016) and one favoured the blended delivery option (kraemer et al., 2007).      figure 2 change in information literacy skills preto post-instruction face-to-face.     figure 3 change in information literacy skills preto post-instruction online       for the 13 studies that could be included in a meta-analysis the smd (95% ci) for face-to-face compared to online instruction was -0.01 (-0.28 to 0.26) (figure 4).   there was not enough data to assess whether there was any difference between skills outcomes and contact time, time to follow-up, delivery method (librarian or non-librarian) or study design. however, there appeared to be no obvious associations from looking at the data.   findings were mixed for the ten studies that included a blended delivery arm (anderson & may, 2010; beile & boote, 2005; byerley, 2005; churkovich & oughtred, 2002; goates et al., 2016; greer et al., 2016; kraemer et al., 2007; orme, 2004; walton & hepworth, 2012; wilcox brooks, 2014), although seven of these studies (70%) found no statistically significant difference between blended and other formats in terms of test or assignment outcomes. of the ten, one study (byerley, 2005) noted that the blended method provided greater skill development than the face-to-face provision, although this was not significant compared to online provision. another study (goates et al., 2016) noted higher posttest scores for students receiving a face-to-face versus blended format (p<0.01). a further study (kraemer et al., 2007) found a significantly greater pre-post improvement in the blended learning compared to the online learning group.       figure 4 comparison of information literacy skills for face-to-face vs. online instruction.       for those studies that could be included in a meta-analysis, there was no statistically significant difference between blended and single format training in terms of skills learnt. the smd comparing blended to online or face-to-face instruction were 0.15 (95% ci, -0.03 to 0.34; 4 studies) and 0.36 (-0.03 to 0.75; 3 studies) respectively (figure 5).    based on the studies that could be included in a meta-analysis, the single format training appeared to be more effective than blended training when skills were measured via a specific assignment such as a piece of persuasive presentation research (anderson & may, 2010) or a rubric graded search strategy (goates et al., 2016). (figure 5)   three further studies looked at specific assignments; two via bibliography assessment within a piece of course work (mery et al., 2012b; wilcox brooks, 2014) and one by a search strategy assessment (schilling, 2012).  mery et al. (2012b) observed a statistically significant improvement in the online compared to the face-to-face group but the other two studies found no difference between face-to-face and blended (wilcox brooks, 2014) or online vs. face-to-face groups (schilling, 2012). no conclusions can be based on this limited evidence.      figure 5 comparison of information literacy skills for online or face-to-face instruction versus blended instruction.     student views overall there was evidence that students felt that the different delivery methods had their advantages and disadvantages. however, the findings are mixed with no clear preference for one method over another. of the 22 studies gathering information on student views, 3 collected data from students exposed to the online (byerley, 2005; kaplowitz & contini, 1998) or blended (wilcox brooks, 2014) training only.   from the 19 studies gathering views on both types of format, 14 (74%) found that students expressed no preference at all in relation to format (table 3). in the five studies finding variations in student views between formats, two studies found that the online course was favoured in terms of perceived benefits, attitudes to the course, and comfort in carrying out library research (alexander & smith, 2001) or increased self-efficacy (a belief in one’s ability to succeed) in choosing databases to search (gall, 2014). three studies identified a preference for face-to-face delivery in terms of greater confidence following training (churkovich & oughtred, 2002; shaffer, 2011) or higher satisfaction in general and around the clarity and length of training (wilhite, 2004). the online group experienced technical difficulties in the studies by shaffer (2011) and wilhite (2004). findings from the themes identified in intervention studies analyzing student views on face-to-face versus online formats are summarized in figure 6. where the findings for a particular measure are neutral, this shows that there was no clear preference from students concerning the online and face-to-face formats.    there were not enough data to guide conclusions concerning perceptions of blended versus single format. however from three studies comparing all three types of format, two found that the views of students across formats were neutral (beile 2005, kraemer 2007) while one noted a preference for the face-to-face format in terms of confidence/self-efficacy (churkovich 2002). a study comparing face-to-face and blended formats found no differences in perceived skills (goates 2016).   study design features the interventions in 30 of the 33 studies were delivered by librarians. face-to-face teaching was delivered by graduate students (alexander & smith, 2001) or teaching assistants (kaplowitz & contini, 1998) in two studies. there was no difference in skills between the face-to-face and online groups at posttest in both studies. only the study by alexander and smith (2001) included comparative information on student views and they found a preference for the online option. mery et al. (2012a) provided the only direct comparison between the deliverers of the intervention, with two face-to-face groups; one trained by librarians and the other by course tutors. the researchers found that skills increased significantly in the librarian and online groups, but not in the tutor group.   of 21 studies providing information on face-to-face contact time, the typical time period was 50-60 minutes (12 studies, see table 3). the longest contact time was for the study by alexander (2001) where graduate students delivered 14 one-hour sessions. the results for the skills test (posttest only) were neutral, but students voiced a preference for the online training. the shortest contact time was 0.5 hour (burhanna et al., 2008), where the researchers reported a trend towards greater skills development in the online group but no difference in student views.      figure 6 analysis of student views on face-to-face versus online formats [numbers of studies].     only 14 studies provided information on the follow-up period between training and the skills test, where the range of follow-up periods was immediately post-training to 12 months (see table 3). there was no statistically significant difference between the two formats in terms of skills retained in 13 studies. there was a statistically significant improvement in the face-to-face group in goates et al. (2016), where skills were measured immediately post-training.   for the 11 randomized controlled trials, 7 studies (64%) found no difference in skills between the formats tested (brettle & raynor, 2013; greer et al., 2016; koenig & novotny, 2001; schilling, 2012; shaffer, 2011; swain et al., unpub; vander meer & rike, 1996), 3 favoured face-to-face training (churkovich & oughtred, 2002; goates et al., 2016; lechner, 2007) and 1 favoured the blended approach (kraemer et al., 2007).     of the 11 randomized controlled trials, 8 explored student views, with 2 favouring the face-to-face format (churkovich & oughtred, 2002; shaffer, 2011) and 6 (75%) with neutral findings (goates et al., 2016; koenig & novotny, 2001; kraemer et al., 2007; schilling, 2012; swain et al., unpub; vander meer & rike, 1996).   discussion   despite the methodological shortcomings of many of the studies included in this review, there is consistent evidence across the body of comparative studies that:   •       face-to-face (traditional) teaching strongly increases information literacy (il) skills when assessed directly preand post-teaching. •       online (web-based) teaching strongly increases il skills when assessed directly preand post-teaching. •       the increase in skills as a result of teaching is broadly comparable for face-to-face and online teaching methods. •       students do not express a clear preference for one format over another although they perceive some differences in the delivery methods (and advantages and disadvantages of each).   the findings from our review of student skills are in keeping with a systematic review evaluating the impact of online or blended and face-to-face learning of clinical skills in undergraduate nurse education (mccutcheon, lohan, traynor & martin, 2015). on the basis of 19 published papers, the authors concluded that online teaching of clinical skills was no less effective than traditional means.   definitive evidence on the effectiveness of blended learning methods compared to single format teaching is limited although it appears that test score outcomes for single and blended format teaching are similar. the potential differences between outcomes, as measured by assignment and test performance, is intriguing and worthy of further study. one might identify test scores and assignment scores as measuring the different outcomes of cognitive (factual knowledge) and behavioural (skills needed to complete a task) aspects of information literacy, respectively.   while the majority of studies that had a potentially more reliable methodology (i.e. the 11 randomized controlled trials) demonstrated neutral findings, four of the studies favoured face-to-face or blended approaches. many of the studies had some methodological shortcomings however.   across the full body of the 33 studies reviewed here, it seems that the choice of format can be left to the educator. given our awareness of the increase in the use of online and blended formats for il teaching, from personal experience and the published literature, this confirmation is welcome. both the student context (e.g., campus-based or distance learners) and cohort sizes are likely to be decisive factors.  blended learning is perceived by academic staff as being more time consuming (brown, 2016), although we could not find any empirical evidence to confirm or refute this perception; nor were any studies identified comparing preparation time for single format face-to-face vs. online sessions.   one development opportunity for the online context is the personalized online learning environment using adaptive learning software (nguyen, 2015). this is an exciting prospect for enhancing student learning in the increasingly online arena of information searching that remains to be explored.   limitations   the authors cannot guarantee that all relevant studies were identified although this review is based on an extensive search for published and unpublished research studies. the quality of the included studies is moderate at best. only 11 studies adopted the randomized controlled trial design, which should minimize the potential for bias, and only 7 piloted or validated the skills tests used. heterogeneity across studies was high so the meta-analysis results should be interpreted with caution. there is also relatively little evidence from outside the u.s.   conclusions and implications for practice   the body of research evidence suggests that information literacy training is equally effective, and well received, across a range of delivery methods. the format can vary to suit the requirements of the student population and the educational situation. in the light of these findings, in our institutions we are confident in moving towards a greater use of online options, particularly for routine il sessions such as library orientations for new students and for access by individuals at ‘point of need’.    future comparative studies should aim to minimize the potential for bias, perhaps by adopting a randomized controlled design. these studies should also employ a large population and they should use validated test instrument(s). more high quality research comparing blended and single format delivery methods will be valuable, along with exploration to unravel the potential dichotomies in outcomes from specific assignments (marked course work) as opposed to il skills tests. further research into the time and resource implications for educators in delivering teaching via these different methods would also be useful.    once these studies have been completed it should be possible to provide clearer guidance to educators, perhaps along the lines of a ‘decision aid’ to guide the choice of teaching format for particular contexts and student groups.   acknowledgements   the authors acknowledge, with thanks, the following specialists for reading and commenting on two drafts of this manuscript: alison brettle (professor in health information and evidence based practice, university of salford u.k.), cecily gilbert (research librarian, barwon health library, victoria australia) and erica swain (subject librarian, cardiff university, u.k.). references   alexander, l.b. & smith, r.c. (2001). research findings of a library skills instruction web course. portal: libraries and the academy, 1(3), 309-328. https://doi.org/10.1353/pla.2001.0033   anderson, k. & may, f.a. (2010). does the method of instruction matter? an experimental examination of information literacy instruction in the online, blended, and face-to-face classrooms.  journal of academic librarianship, 36(6), 495-500. https://doi.org/10.1016/j.acalib.2010.08.005   arnold-garza, s. (2014). the flipped classroom teaching model and its use for information literacy instruction. communications in information literacy, 8(1), 7-22. http://files.eric.ed.gov/fulltext/ej1089137.pdf   beile, p.m. & boote, d.n. (2005). does the medium matter? a comparison of a web-based tutorial with face-to-face library instruction on education students’ self-efficacy levels and learning outcomes. research strategies, 20, 57-68. https://doi.org/10.1016/j.resstr.2005.07.002   bordignon, m., strachan, g., peters, j., muller, j., otis, a., georgievski. a., & tamin, r. (2016). assessment of online information literacy learning objects for first year community college english composition. evidence based library & information practice, 11(2), 50-57. https://doi.org/10.18438/b8t922   braun, v. & clarke, v. (2006). using thematic analysis in psychology. qualitative research in psychology, 3(2), 77-101. https://doi.org/10.1191/1478088706qp063oa   brettle, a. & raynor, m. (2013). developing information literacy skills in pre-registration nurses: an experimental study of teaching methods. nurse education today, 33(2), 103-109. https://doi.org/10.1016/j.nedt.2011.12.003   brown, m.g. (2016). blended instructional practice: a review of the empirical literature on instructors’ adoption and use of online tools in face-to-face teaching.  internet and higher education, 31, 1-10 https://doi.org/10.1016/j.iheduc.2016.05.001   burhanna, k.j., eschedor voelker, t.j. & gedeon, j.a. (2008). virtually the same: comparing the effectiveness of online versus in-person library tours. public services quarterly, 4(4), 317-338. https://doi.org/10.1080/15228950802461616   byerley, s.l. (2005). library instruction: online or in the classroom? academic exchange, 9(4), 193-197. https://www.questia.com/library/journal/1g1-142636415/library-instruction-online-or-in-the-classroom   churkovich, m. & oughtred, c. (2002). can an online tutorial pass the test for library instruction? an evaluation and comparison of library skills instruction methods for first year students at deakin university. australian academic & research libraries, 33, 25-38. https://doi.org/10.1080/00048623.2002.10755177   cilip. (2017). information literacy. the library and information association. web page. https://www.cilip.org.uk/research/topics/information-literacy   gall, d. (2014). facing off: comparing an in-person library orientation lecture with an asynchronous online library orientation. journal of library & information services in distance learning, 8(3/4), 275-287. https://doi.org/10.1080/1533290x.2014.945873   germain, c.a., jacobson, t.e. & kaczor, s.a.a. (2000). comparison of the effectiveness of presentation formats for instruction: teaching first-year students. college & research libraries, 61(1), 65-72. https://doi.org/10.5860/crl.61.1.65   goates, m.c., nelson, g.m. & frost, m. (2016). search strategy development in a flipped library classroom: a student-focused assessment. college & research libraries, anticipated publication date 1 may 2017.   greer, k., hess, a.n. & kraemer, e.w. (2016). the librarian leading the machine: a reassessment of library instruction methods. college & research libraries, 77(3), 286-301. https://doi.org/10.5860/crl.77.3.286   holman, l. (2000). a comparison of computer-assisted instruction and classroom bibliographic instruction. reference & user services quarterly, 40(1), 53-60. https://www.jstor.org/stable/20863900   kaplowitz, j. & contini, j. (1998). computer-assisted instruction: is it an option for bibliographic instruction in large undergraduate survey classes? college & research libraries, 59(1), 19-27. https://doi.org/10.5860/crl.59.1.19   koenig, m. & novotny, e. (2001). on-line course integrated library instruction modules as an alternative delivery method. chapter 26, pp.200-208 in: dewey bi, editor. library user education. lanham, maryland and london: scarecrow press; 2001.   koufogiannakis, d., booth, a. & brettle, a. (2005). reliant: reader's guide to the literature on interventions addressing the need for education and training.  library and information research, 30(94), 8. http://eprints.rclis.org/8082/1/reliant__final_.pdf   koufogiannakis, d. & wiebe, n. (2006). effective methods for teaching information literacy skills to undergraduate students: a systematic review and meta-analysis. evidence based library and information practice, 1(3), 3-43. https://doi.org/10.18438/b8ms3d   kraemer, e.w., lombardo, s.v. & lepkowski, f.j. (2007). the librarian, the machine, or a little of both: a comparative study of three information literacy pedagogies at oakland university. college & research libraries, 68(4), 330-342. https://doi.org/10.5860/crl.68.4.330   kratochvil, j. (2014). measuring the impact of information literacy e-learning and in-class courses via pre-tests and post-test at the faculty of medicine, masaryk university. mefanet j, 2(2), 41-50. https://is.muni.cz/repo/1214193/en/kratochvil/measuring-the-impact-of-information-literacy-e-learning-and-in-class-courses-via-pre-tests-and-post-test-at-the-faculty-of-medicine-masaryk-university?lang=en   lantzy, t. (2016). health literacy education: the impact of synchronous instruction. reference services review, 44(2), 100-21. https://doi.org/10.1108/rsr-02-2016-0007   lechner, d.l. (2007). graduate student research instruction: testing an interactive web-based library tutorial for a health sciences database. research strategies, 20, 469-481. https://doi.org/10.1016/j.resstr.2006.12.017   lewis, s. & clarke, m. (2001). forest plots: trying to see the wood and the trees. bmj, 322(7300), 1479-1480. https://doi.org/10.1136/bmj.322.7300.1479   light, r.j. & pillemer, d.b. (1984). summing up: the science of reviewing research.  cambridge, massachusetts: harvard university press, 1984   liu, q., peng, w., zhang, f., hu, r., yingzue, l. & yan, w. (2016). the effectiveness of blended learning in health professions: systematic review and meta-analysis. journal of medical internet research, 18(1), e2. https://doi.org/10.2196/jmir.4807   mccutcheon, k., lohan, m., traynor, m. & martin, d. (2015). a systematic review evaluating the impact of online or blended learning vs. face-to-face learning of clinical skills in undergraduate nurse education.  journal of advanced nursing, 71(2), 255-270 https://doi.org/10.1111/jan.12509   means, b., toyama, y., murphy, r., baki, m. (2013). the effectiveness of online and blended learning: a meta-analysis of the empirical literature. teachers college record 115, 1-47 https://www.sri.com/sites/default/files/publications/effectiveness_of_online_and_blended_learning.pdf   mery, y., newby, j. & peng, k. (2012a). why one-shot information literacy sessions are not the future of instruction: a case for online credit courses. college & research libraries, 73(4), 366-377. https://doi.org/10.5860/crl-271   mery, y., newby, j. & peng, k. (2012b).  performance-based assessment in an online course: comparing different types of information literacy instruction. portal: libraries and the academy, 12(3), 283-298 https://doi.org/10.1353/pla.2012.0029   morrison, j.m., sullivan, f., murray, e. & jolly, b. (1999). evidence-based education: development of an instrument to critically appraise reports of educational interventions. medical education, 33(12), 890-893. https://doi.org/10.1046/j.1365-2923.1999.00479.x   national centre for text mining. termine web demonstration, 2012. web page.  http://www.nactem.ac.uk/software/termine/   nguyen, t. (2015). the effectiveness of online learning: beyond no significant difference and future horizons. merlot journal of online learning & teaching, 11(2), 309-319. http://jolt.merlot.org/vol11no2/nguyen_0615.pdf   nichols, j., shaffer, b. & shockey, k. (2003). changing the face of instruction: is online or in-class more effective? college & research libraries, 64(5), 378-388. https://doi.org/10.5860/crl.64.5.378   nichols hess, a.n. (2014). online and face-to-face library instruction: assessing the impact on upper-level sociology undergraduates. behavioral & social sciences librarian, 3, 132-147. https://doi.org/10.1080/01639269.2014.934122   orme, w.a. (2004). a study of the residual impact of the texas information literacy tutorial on the information-seeking ability of first year college students. college & research libraries, 65(3), 205-215. https://doi.org/10.5860/crl.65.3.205   qsr international pty ltd. nvivo 10 software. released 2012. http://www.qsrinternational.com/nvivo-support/downloads   salisbury, f. & ellis, j. (2003). online and face-to-face: evaluating methods for teaching information literacy skills to undergraduate arts students. library review, 52(5), 209-217. https://doi.org/10.1108/00242530310476715   schilling, k. (2012). the efficacy of elearning for information-retrieval skills in medical education. european conference on e-learning, october 2012.   shaffer, b.a. (2011). graduate student library research skills: is online instruction effective?  journal of library & information services in distance learning, 5(1/2), 35-55. https://doi.org/10.1080/1533290x.2011.570546   silk, k.j., perrault, e.k., ladenson, s. & nazione, s.a. (2015). the effectiveness of online versus in-person library instruction on finding empirical communication research. the journal of academic librarianship, 41,149-154. https://doi.org/10.1016/j.acalib.2014.12.007   silver, s.l. & nickel, l.t. (2007). are online tutorials effective? a comparison of online and classroom library instruction methods. research strategies, 20(4), 389-396. https://doi.org/10.1016/j.resstr.2006.12.012   swain, e., weightman, a.l., farnell, d.j.j & mogg, r. (2016). an experimental study of online versus face-to-face student induction at a university library: both formats are equally effective and well received. unpublished.   vander meer, p.f. and rike, g.e. (1996). multimedia: meeting the demand for user education with a self-instructional tutorial. research strategies, 14(3), 145-158. https://www.learntechlib.org/p/82581   walton, g. & hepworth, m. (2012). using assignment data to analyse a blended information literacy intervention: a quantitative approach. journal of librarianship & information science, 45(1), 53-63. https://doi.org/10.1177/0961000611434999   weightman, a.l., farnell, d.j.j., morris, d. & strange, h. (2015). information literacy teaching in universities – a systematic review of evaluation studies. preliminary findings for online versus traditional methods. [poster] eighth evidence based library & information practice conference (eblip8), brisbane, queensland, australia, 6-8 july 2015.   wilcox brooks, a.w. (2014). information literacy and the flipped classroom: examining the impact of a one-shot flipped class on student learning and perceptions. communications in information literacy, 8(2), 225-235. http://files.eric.ed.gov/fulltext/ej1089274.pdf   wilhite, j.m. (2004). internet versus live: assessment of government documents bibliographic instruction. journal of government information, 30(5/6), 561-574. https://doi.org/10.1016/j.jgi.2004.10.002   zhang, l., watson, e.m. & banfield. l. (2007). the efficacy of computer-assisted instruction versus face-to-face instruction in academic libraries: a systematic review. journal of academic librarianship, 33(4), 478-484. https://doi.org/10.1016/j.acalib.2007.03.006     appendix   additional file: evidence table: effectiveness       editorial evidence based library and information practice 2009, 4:2 133 evidence based library and information practice article study in grey and white: measuring the impact of the 8rs canadian library human resources study allison sivak assessment librarian university of alberta libraries university of alberta edmonton, alberta, canada asivak@ualberta.ca received: 21 february 2009 accepted: 8 may 2009 © 2009 sivak. this is an open access article distributed under the terms of the creative commons attribution license, <http://creativecommons.org/licenses/by/2.0>, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – to use the 8rs canadian library human resources study (the 8rs study) as a test case to develop a model for assessing research impact in lis. methods – three different methods of citation analysis which take into account the changing environment of scholarly communications. these include a ‚manual‛ method of locating citations to the 8rs study through a major lis database, an enhanced-citation tool google scholar, and a general google search to locate study references in non-scholarly documents results – the majority of references (82%) were found using google or google scholar; the remainder were located via lisa. each method had strengths and limitations. conclusion in-depth citation analysis provides a promising method of understanding the reach of published research. this investigation’s findings suggest the need for improvements in lis citation tools, as well as digital archiving practices to improve the accessibility of references for measuring research impact. the findings also suggest the merit of researchers and mailto:asivak@ualberta.ca evidence based library and information practice 2009, 4:2 134 practitioners defining levels of research impact, which will assist researchers in the dissemination of their work. introduction the question of research impact is one of concern to scholars, universities, and research funding bodies. researchers and the organizations or associations which support their work face increasing demands to demonstrate the impact of their research as a measure of accountability, particularly when that work is supported through public funds. with respect to professional achievement, the impact factor of the journals in which pre-tenured academics publish has influence in their application for tenure (monastersky 2005). further, leaders within the open access movement promote freely-available online scholarly content as a method of ‚seamlessly linking data, knowledge, and scholars‛ which will ‚stimulate and accelerate discovery — and ultimately to fuel advances beyond the realm of scholarship‛ (association of research libraries 2009). the 8rs canadian library human resources study (the 8rs study) is a research project that can serve as a test case to develop a model for assessing research impact in library and information studies (lis). the study has a number of characteristics that make it uniquely suited to this purpose. first, the study was designed to answer emerging concerns from practitioners regarding retirements, recruitment, and other aspects of human resources. second, the 8rs research team conceived the study as a tool which could inform library administrators’ and managers’ decision-making regarding human resource issues; therefore, the report presented not just major findings, but also presented recommendations for specific stakeholders within the library field, i.e., administrators, practitioners, educators, and associations. third, the work was financially supported by the library community and both the federal and provincial government bodies, which has implications for accountability, although certainly not identical implications to those of researchers funded by formal funding agencies. last, study results have been disseminated in the form of grey literature, including reports freely available online via the study’s website at <http://www.ls.ualberta.ca/8rs/> and distributed in print and conference presentations for which slides were sometimes posted. the implications of the study’s characteristics are that its relevance to the professional library community should be high and the results are accessible; both these implications would ostensibly increase the study’s impact. 8rs study context the 8rs canadian library human resources study has been a major research project in library and information studies. it was conducted by a research team at the university of alberta libraries, and was designed at a time of great anecdotal discussion in the library community regarding anticipated ‚mass retirements‛ of senior librarians. the work was inspired in part by the previous research of stanley wilder (2003) on the demographics of librarians at member institutions of the association of research libraries (arl). a notable finding in wilder’s study was that academic librarians, as a profession, were older than those in other fields, noting that 48% of north american librarians in arl libraries would be at retirement age by 2015. wilder’s statistics gave an even more pessimistic picture for the canadian library community, suggesting that a greater percentage of canadian academic librarians http://www.ls.ualberta.ca/8rs/ evidence based library and information practice 2009, 4:2 135 were older, on average (2003). where wilder noted that 23% of american research librarians were under age 40, only 16% of canadian research librarians were in this age category (2003). an article published around the same time in the book trade publication quill & quire, reviewed published labour force statistics from statistics canada to project retirement figures for the industry. the article’s title was alarmist in nature: ‚libraries face ‘skills gap’: 48% of librarians could retire by 2005‛ (weiler 2000). from 2002 to 2005 the 8rs research team conducted a study of human resources in canadian libraries across all sectors (academic, public, and special). after the initial development of the research proposal, the research team concluded that the study required several types of support from the library community: financial support, inkind support (e.g., membership lists for the sampling frame), and community ‚buy-in‛ to encourage survey response. research team chair ernie ingles made a number of presentations on the proposed work at conferences and association meetings. data collection began in 2003, and several methods were used: interviews with 17 library administrators, focus groups with administrators of research libraries and large urban public libraries, and most substantially, major surveys of library administrators and library staff (both librarians and paraprofessionals). response rates overall were approximately 36%, resulting in data from 461 administrators and 4,693 library staff. the 8rs report, the future of human resources in canadian libraries, was published in the spring of 2005, by the university of alberta libraries, in both french and english. this report falls into the category of grey literature. rabina (2008) reviews several definitions of grey literature, noting that ‚there is agreement on the main characteristics of grey literature: they are materials that are published by not for profit institutions, and, as a result, typically are not marketed or distributed by commercial publishing organizations‛ (34). this above definition of grey literature is appropriate with respect to the character of the study’s publisher, the university of alberta libraries, as a non-profit organization for which publishing is an atypical activity. however, the 8rs study is an unusual instance of grey literature publication because the report was widely marketed and distributed. it was made freely available in pdf format on the 8rs website <http://www.ls.ualberta.ca/8rs/reports>, and some print copies were produced in order to facilitate ease of reading; at 275 pages, the team believed that print would be the preferred format for some readers. the publication was formally launched at the annual conference of the canadian library association in june 2005; print copies of the report were available for purchase, and the 8rs research team made a presentation on the study’s major findings. in july 2005, over 300 print reports were distributed to members of the canadian association of research libraries, the canadian association of large urban public libraries (now the canadian urban libraries’ council), the provincial and territorial library directors council, the canadian masters of library and information studies (mlis) programs, and to all other organizations and associations that supported the study through financial and in-kind assistance. additionally, copies were mailed to the membership of the association of research libraries (arl) and the association for library and information science education. this resulted in further promotion regarding the 8rs study through third http://www.ls.ualberta.ca/8rs/reports evidence based library and information practice 2009, 4:2 136 parties. shortly after the report’s publication, the canadian library association (cla), in conjunction with the 8rs research team, announced the formation of a president’s council on the 8rs to further investigate implications of the study’s findings and recommendations. one of the president’s council actions was to promote the study; to this end, the group embarked on several different initiatives to present relevant findings to different stakeholders in the canadian library community. for example, while many statistics in the report were broken down by library sector and sub-sector, the sheer size of the document was presumed to be daunting. therefore, the group determined a need for ‚executive summaries‛ of findings for different library sectors such as research libraries, other academic libraries, special libraries, urban public libraries, other public libraries, etc. the report’s publication was also noted in the june 2005 issue of the arl bimonthly report. over the following months, members of the research team presented the study’s major findings to a number of audiences, predominantly at library association conferences. these presentations were tailored towards their audiences, in that rather than presenting the aggregate statistics for the canadian library community as a whole, the presentation focused on those statistics more closely related to the sector hosting the conference. a measure of the success agreed upon by the 8rs study research team would be its ‚use‛ by library staff and scholars in workforce planning, as well as in understanding the current and predicted workforce. the study described below endeavors to measure this success. aims this paper investigates the impact of the 8rs study through three different methods of citation analysis. it attempts to take into account the changing environment of scholarly communications by using a ‚manual‛ method of locating citations to the 8rs study through a major lis database, through the enhanced-citation tool google scholar, and through a general google search to locate 8rs study references in nonscholarly documents. through the analysis, it attempts to develop recommendations for understanding the impact of lis research in the field. methods a number of parameters were defined prior to beginning the search: references to the study in any article or presentation given by the 8rs research team were not counted. references by other library scholars who had connection to the 8rs study (members of the cla president’s council, for example) were counted. the reason for this was that presentations given by the 8rs research team served two pre-defined functions: to both raise awareness of the 8rs study, and to disseminate the results for relevant audiences. for the purposes of this study ‚impact‛ refers to the utilization of the 8rs published statistics, report recommendations, or original data to support their research questions. citation analysis was selected as the most appropriate method for conducting this investigation. prior to the development of commercial indexing and abstracting vendors, such as thomson scientific, chemical abstracts, mathscinet, psychinfo, and google scholar, citation analysis was a laborious and timeconsuming process, involving the textual review of publications in periodicals and books to map the frequency of citation of particular authors or papers. electronic databases provide a vastly-improved process; however, neuhaus and daniel’s review notes the importance of critically assessing these databases for their potential with respect to citation analysis as well as evidence based library and information practice 2009, 4:2 137 for their coverage limitations (2008). this study also attempts to identify some relevant sources for the purpose of citation analysis in the field of lis. with this aim, it was important to begin with the library and information science abstracts (lisa) database, the most credible source for the scholarly literature. in their content analysis of lis research databases, koufogiannakis, crumley, and slater concluded that lisa provides the best coverage of the ten most research-intensive peer-reviewed journals in the field of lis (2004). however, aina’s work on grey literature in lis stated that although some coverage of grey literature was present in lisa, it did not serve as a comprehensive source (2001). currently, there exists no lisspecific citation-enhanced database, meaning that citation analysis is laborious and time-consuming in the lis databases, leaving google scholar as the sole automated citation-assessment tool. therefore, both these tools were selected to identify citations of the 8rs study. in order to understand the scope of the 8rs study published in the journal literature, lisa was searched both by keyword ‚8rs,‛ and by the controlled vocabulary subject headings ‚library staff‛ and ‚leadership,‛ limited to the dates 2005-2009. references were checked in all the resulting articles which were accessible to the researcher to see if the articles cited the 8rs study. both peer-reviewed and non-refereed citations were counted. the database searches were complemented by searches in google and google scholar for the combined terms ‚8rs,‛ ‚library,‛ and ‚human resources.‛ the addition of the search in google sought to identify any references to the study in the grey literature. results the search located 480 citations in the lisa database, 3140 in google, and 161 in google scholar. of the lisa citations, 118 articles could not be obtained, representing 28 individual journal titles. the summary of relevant search findings is shown in table 1. in all, 68 relevant references were located, with about onethird (22) occurring in articles published in peer-reviewed journals, 19 found in conference presentations online, 6 in nonpeer-reviewed journals, 6 in blogs, and 5 in reports. four references to the publication were found on websites in bibliographies or as ‚suggested resources‛, and four were found related to libraries’ internal planning documents such as strategic plans. two references to the study were located on mlis course syllabi. the majority of references (82%) were found using the google or google scholar search engines. the remaining 18% of the references were located using the lisa database. it should be noted that each article was counted only once; the lisa database was the first method of searching employed, and articles located using lisa were not counted in the results for google or google scholar, if found again. each source had different limitations. although google resulted in the greatest number of results, those results needed to be assessed individually for categorization purposes. however, the google and google scholar searches took much less time to conduct than the searches in lisa, as the lisa database does not have an enhanced-citation tool. two categories of ‚formal‛ publications were not found in lisa: conference presentations and reports. indeed, the original 8rs report, the future of human resources in canadian libraries, is not indexed in the lisa database. while conference presentations would not normally be expected to be found in a evidence based library and information practice 2009, 4:2 138 database, reports are at times indexed in commercial databases. one type of ‚informal‛ publication that can be interpreted as a strong indicator of research impact is the internal planning document for an individual library or library association. four such publications were found via google: two strategic planning documents, minutes from a board of directors meeting, and one memo to support an institutional human resources initiative. as an original goal for the 8rs study was to present findings in a way that allowed institutions to make use of the statistics in their decision-making process, this finding suggests some success. table 2 analyzes the search findings by year and type of publication. the highest number of references to or citations of the 8rs study (29) took place in 2007, which is also when the greater number of articles published (13) in peer-reviewed journals appeared. this was an increase from the number of references found for the years 2005 and 2006 (6 and 18 respectively). in 2008 the number dropped to 15. the peer-reviewed articles tended to be published 18-24 months after the report was released; which seems intuitive, suggesting that some period of time is necessary for the report to be disseminated and further analyzed before other scholars are likely to cite it. location/type conference non peerreviewed journal peerreviewed journal report blog course syllabus resource on website internal planning document total google 13 1 6 3 6 2 4 4 39 google scholar 6 9 2 17 lisa 5 7 12 total 19 6 22 5 6 2 4 4 68 table 1: number of references to 8rs study, by search method and publication type type 2005 2006 2007 2008 total blog 2 4 6 conference 7 10 2 19 non peer-reviewed journal 2 1 3 6 peer-reviewed journal 1 1 13 7 22 report 1 2 1 1 5 course syllabus 2 2 resource on website 1 3 4 internal planning document 3 1 4 grand total 6 18 29 15 68 table 2: frequency of reference by type of publication and year published evidence based library and information practice 2009, 4:2 139 while this is not a study of the comprehensiveness of the citation sources, it is notable that the search results did not include some conference presentations which cited the 8rs study, of which the author was aware. several explanations are possible here, the most likely being that the presentations may not have been uploaded post-conference. conference presentations are increasingly being hosted on conference websites, but this practice is clearly not a standard within the field. table 3 summarizes the type of reference made to the 8rs study. the study’s published statistics were the primary reference point, with 52 of 68 publications citing some of the study’s key findings. twelve of the publications made reference to the study’s publication in general; this type of reference tended to remark on implications of an aging workforce or comment on major study findings without specific reference to statistics. three references specifically discussed an aspect of the work underway by the cla president’s council on the 8rs, and in two publications the authors completed their own data analysis, having requested data files from the 8rs research team. type of reference total original data analysis 2 reference to president's council work 3 reference to published statistics 51 reference to study publication 12 total 68 table 3: type of reference made to 8rs study type original data analysis reference to president's council work reference to published statistics reference to study publication total blog 3 3 6 conference 18 1 19 course syllabus 2 2 internal planning document 2 2 4 non peer-reviewed journal 6 6 peer-reviewed journal 2 20 22 report 4 1 5 resource on website 1 3 4 total 2 3 51 12 68 table 4: frequency of type of reference by type of publication evidence based library and information practice 2009, 4:2 140 the frequency of type of reference cross tabulated with the type of publication found shows that references to the study’s publication, and more specifically to the published statistics, were generally distributed across several types of publications. original data analyses were only found in peer-reviewed articles, and references to further work on the study were only found on blogs discussion the methods employed in this study had some limitations. the search resulted in english-language results only, and the keyword search was not replicated using ‚bibliothèque‛ in place of ‚library.‛ although the majority of library literature is in the english language, a review of the study references in french might add useful information, as this was a canadian study. in addition, the data was not analyzed for overlap among the lisa database, google, and google scholar, since this study did not intend to formally compare the coverage between different citation sources. however, future analysis of duplication between these sources, in particular between google scholar and lisa, would be useful in determining which tool is a more effective resource for citation analysis. the databases isi/thomson web of knowledge and scopus were not included as sources for this study, although both have been frequently used in other disciplines for citation analysis. these two databases were excluded from the study due to the limited coverage of lis journals; while lisa indexes 414 journal titles, web of knowledge indexes 62 titles and scopus indexes 79 titles. a large proportion of scholarly communication within the field of lis takes place in the form of conference presentations which may not be subsequently published in peer-reviewed journals. although presentations are certainly less-formal publications as compared to peer-reviewed literature, it can be argued that they are important aspects of scholarly communication; presentations to peers create a forum in which scholars can receive useful responses to their work, which may very well develop or solidify their thinking on the subject. a conference may also serve as an arena in which to test theories or research findings, to determine the feasibility of writing an article for submission to a journal, and may mark an important point in the development of the research. an increasing number of lis conference presentations are accessible online, via conference websites; most likely in response to demand from delegates, so that they may refer to the work presented in the future. the question remains as to the need for an aggregator of conference content for the field of library and information studies, in order to capture subject matter of concern to the profession. librarians, other researchers, and associations should consider the consistent archiving of presentation material in digital repositories for this purpose. that the number of references to the 8rs study decreased in 2008 may allude to the ‚shelf life‛ of a publication or of the perceived timeliness of the statistics. researchers have established that grey literature citation tends to peak in the first five years after publication, whereas journal literature has a longer ‚citation window‛ of seven to ten years (di cesare et al. 2008). this study should therefore be repeated at least six years after the 8rs study release. while small in number, the publications that employed original analysis or ‚data repurposing of the 8rs data represent an important and novel approach to use of the study. data repurposing is an emerging issue in contemporary scholarship across many disciplines. ponti (2008), in her proposal of a virtual research evidence based library and information practice 2009, 4:2 141 ‚collaboratory‛ for lis, suggests that ‚collaboratories have the potential to expand the participation of practitioners in research contexts,‛ and proposes data repurposing as a method that could close the education-practice gap identified by the library literature (271). the issues and questions regarding data repurposing are complex and will not be addressed in this paper. however, given that scholars in other disciplines are raising questions about the benefits, concerns, and logistics of data repurposing (silva 2007, smith 2008, sales et al. 2006, morrison et al. 2009), lis scholars should investigate the implications of this practice within the field. researchers have found that the ability of google scholar to retrieve different versions of grey literature allows for the reconstruction of the document life cycle; this can provide insight into the path of research publication between grey literature and journal literature (di cesare et al. 2008). further, the google searches also resulted in some references that allowed for an understanding of the way the 8rs study is informing literature outside the peerreviewed journals, as well as how it is informing practice as noted in organizational documentation. impact on disciplinary knowledge and future work are general goals of all scholarly research; however, lis research is integral to the professional work of librarianship, and so impact cannot strictly be measured through the peer-reviewed literature—particularly considering the nature of the study and the research team’s goals in completing this work. conclusion the research-practice gap is of concern to scholars and practitioners in the field of library and information studies. as many practitioners increasingly aim to base their professional work on evidence-based decision-making, many researchers strive to connect their research to the professional sphere. in-depth citation analysis provides a promising method of understanding the reach of published research. this investigation’s findings suggest the need for improvements in lis citation tools, as well as digital archiving practices to improve the accessibility of references for measuring research impact. the findings also suggest the merit of researchers and practitioners defining levels of research impact, which will assist researchers in the dissemination of their work. works cited 8rs research team. the future of human resources in canadian libraries. edmonton: univ. of alberta, 2005. aina, l. o. ‚grey literature and library and information studies: a global perspective.‛ international journal on grey literature 1.4 (2000): 179182. di cesare, rosa, and daniela luzi, roberta ruggieri. ‚the impact of grey literature in the web environment: a citation analysis using google scholar.‛ grey journal 4.2 (2008): 83-96. koufogiannakis, denise, linda slater, and ellen crumley. ‚a content analysis of librarianship research.‛ journal of information science 30 (2004): 227-239. ponti, m. ‚a lis collaboratory to bridge the research-practice gap.‛ library management 29.4/5 (2008): 265-277. association of research libraries and scholarly publishing and academic evidence based library and information practice 2009, 4:2 142 resources coalition. create change <http://www.createchange.org/schol arship/index.shtml> 12 feb. 2009. monastersky, richard. ‚the number that’s devouring science.‛ the chronicle of higher education 52.8 (2005): a12. morrison, frances p., and li li, albert m. lai, george hripcsak. repurposing the clinical record: can an existing natural language processing system deidentify clinical notes? journal of the american medical informatics association 16.1 (2009): 37-39. neuhaus, christoph and hans-dieter daniel. “data sources for performing citation analysis. an overview.‛ journal of documentation 64.2 (2008): 193-210. rabina, debbie l. ‚a scholarly communication perspective of grey literature in library and information science education.‛ the grey journal 4.1 (2008): 34-40. sales, esther, sara lichtenwalter, and antonio fevola. "secondary analysis in social work research education: past, present, and future promise." journal of social work education 42.3 (fall 2006): 543-558. silva, elizabeth b. ‚what’s *yet+ to be seen? re-using qualitative data.‛ sociological research online 12.3 (2007). 10 feb. 2009 <http://www.socresonline.org.uk/12 /3/4.html>. smith, emma. "pitfalls and promises: the use of secondary data analysis in educational research." british journal of educational studies 56.3 (sep. 2008): 323-339. 18 feb. 2009 weiler, derek. "libraries face "skills gap": 48 % of librarians could retire by 2005." quill & quire 66.1 (jan. 2000): 10-11. wilder, stanley. demographic change in academic librarianship. washington, dc: association of research libraries, 2003. evidence summary   web-scale discovery services retrieve relevant results in health sciences topics including medline content   a review of: hanneke, r., & o’brien, k. k. (2016). comparison of three web-scale discovery services for health sciences research. journal of the medical library association, 104(2), 109-117. http://dx.doi.org/10.3163/1536-5050.104.2.004   reviewed by: elizabeth stovold information specialist, cochrane airways group st george’s, university of london tooting, london, united kingdom email: estovold@sgul.ac.uk   received: 3 mar. 2017    accepted: 21 apr. 2017      2017 stovold. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to compare the results of health sciences search queries in three web-scale discovery (wsd) services for relevance, duplicate detection, and retrieval of medline content. design – comparative evaluation and bibliometric study. setting – six university libraries in the united states of america. subjects – three commercial wsd services: primo, summon, and ebsco discovery service (eds). methods – the authors collected data at six universities, including their own. they tested each of the three wsds at two data collection sites. however, since one of the sites was using a legacy version of summon that was due to be upgraded, data collected for summon at this site were considered obsolete and excluded from the analysis.   the authors generated three questions for each of six major health disciplines, then designed simple keyword searches to mimic typical student search behaviours. they captured the first 20 results from each query run at each test site, to represent the first “page” of results, giving a total of 2,086 total search results. these were independently assessed for relevance to the topic. authors resolved disagreements by discussion, and calculated a kappa inter-observer score. they retained duplicate records within the results so that the duplicate detection by the wsds could be compared.   they assessed medline coverage by the wsds in several ways. using precise strategies to generate a relevant set of articles, they conducted one search from each of the six disciplines in pubmed so that they could compare retrieval of medline content. these results were cross-checked against the first 20 results from the corresponding query in the wsds. to aid investigation of overall coverage of medline, they recorded the first 50 results from each of the 6 pubmed searches in a spreadsheet. during data collection at the wsd sites, they searched for these references to discover if the wsd tool at each site indexed these known items.   authors adopted measures to control for any customisation of the product setup at each data collection site. in particular, they excluded local holdings from the results by limiting the searches to scholarly, peer-reviewed articles. main results – authors reported results for 5 of the 6 sites. all of the wsd tools retrieved between 50-60% relevant results. eds retrieved the highest number of relevant records (195/360 and 216/360), while primo retrieved the lowest (167/328 and 169/325). there was good observer agreement (k=0.725) for the relevance assessment. the duplicate detection rate was similar in eds and summon (between 96-97% unique articles), while the primo searches returned 82.9-84.9% unique articles.   all three tools retrieved relevant results that were not indexed in medline, and retrieved relevant material indexed in medline that was not retrieved in the pubmed searches. eds and summon retrieved more non-medline material than primo. eds performed best in the known-item searches, with 300/300 and 299/300 items retrieved, while primo performed worst with 230/300 and 267/300 items retrieved.   the summon platform features an “automated query expansion” search function, where user-entered keywords are matched to related search terms and these are automatically searched along with the original keyword. the authors observed that this function resulted in a wholly relevant first page of results for one of the search questions tested in summon. conclusion – while eds performed slightly better overall, the difference was not great enough in this small sample of test sites to recommend eds over the other tools being tested. the automated query expansion found in summon is a useful function that is worthy of further investigation by the wsd vendors. the ability of the wsds to retrieve medline content through simple keyword searches demonstrates the potential value of using a wsd tool in health sciences research, particularly for inexpert searchers.   commentary   previous studies such as ketterman and inman (2014) have sought to compare wsds directly with traditional bibliographic databases. however the authors of this study highlight research into typical library user behaviour that shows a preference for google-style searching over traditional methods due to ease, efficiency, and relevance ranking. an assessment of wsd system performance using relevance of the results as an indicator is therefore warranted.   this study was evaluated using perryman’s (2009) critical appraisal tool for bibliometric studies. the objectives are clearly stated and the methodology is described in detail for each aspect of the study. the chosen search questions are based on real life examples, and the retrieval methods are designed to reflect common user behaviours, and therefore both are appropriate for the stated aims of the study. all of the search strategies are included in the online appendices, and the processes for data collection and handling are well documented. overall the methods section of this paper is strong and the authors provide an equally robust discussion of the limitations of their study, together with the controls they put in place to help mitigate these, such as duplicate screening of the results when assessing for relevance.   results from each strand of the study are clearly presented, however it would be helpful to see the tabulated results in percentages as well as absolute numbers so that the reader is able compare the performance of each wsd more easily. the authors collected a large amount of data and it would be interesting to see more reporting of this information, particularly the relevance assessments per search query, as the authors noted in their discussion section that relevance was often a function of the topic.   although the authors were not able to recommend one wsd tool over the other, this study is a good starting point for library professionals considering promoting one of these tools to their library users or implementing one of these products in their library. there are many other issues to consider when evaluating a wsd, such as usability and compatibility with other library tools, and these are recognised by the authors. deodato’s (2015) comprehensive guide to conducting a full evaluation of wsds is a useful resource.   the key finding of this study is the ability of wsd products to retrieve medline content with simple searches representative of typical student search behaviours. this has implications for health sciences librarians who are involved in the training and education of library users and the selection of library resources. there are opportunities for further research to see if the findings of this study are consistent across other test sites and in different health science disciplines, and more studies designed to directly compare the performance of wsds with medline are needed.                                                       references   deodato, j. (2015). evaluating web-scale discovery services: a step-by-step guide. information technology and libraries, 34(2), 19-75. http://dx.doi.org/10.6017/ital.v34i2.5745   ketterman, e., inman, m. e. (2014). discovery tool vs. pubmed: a health sciences literature comparison analysis. journal of electronic resources in medical libraries, 11(3), 115-123. http://dx.doi.org/10.1080/15424065.2014.938999   perryman, c. (2009). evaluation tool for bibliometric studies. retrieved from carol perryman website: https://www.dropbox.com/l/scl/aaal7luzple90fxfnbv5hcnoz0ctlh6rqrs         news   call for applications: evidence summaries writers for evidence based library and information practice    2018. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     eblip seeks to add five writers to the evidence summaries team. evidence summaries are critical appraisal syntheses, which provide analysis regarding the validity and reliability of the methodology used in an original research article. as such, they are a key component of eblip to aid readers in making informed decisions in their local practice. evidence summaries team members are required to write two evidence summaries per year, with a two year commitment to the journal. evidence summaries cover all areas of library and information studies and we encourage applications from information professionals in areas such as school, public, and special libraries, as well as academic settings. interested persons should send a cover letter, indicating areas of strength they would bring to the role, and resume/cv as a single pdf file to heather pretty (associate editor, evidence summaries) hjpretty@mun.ca by january 15, 2019. applicants who are shortlisted will be asked to submit a sample evidence summary. **please note that evidence based library and information practice is a non-profit, open access journal and all positions are voluntary and unpaid. the positions are an excellent opportunity for continuing professional development and gaining experience in reviewing or critically appraising library-related research. **only those applicants who are selected or shortlisted will be contacted by the editors. about the journal:   published quarterly, and hosted by the university of alberta, this peer-reviewed, open access journal is targeted at all library and information professionals interested in an evidence based model of practice. by facilitating access to librarianship research via original research articles and evidence summaries of relevant research from the library literature, evidence based library and information practice will enable librarians to practice their profession in an evidence based manner.   please visit the evidence based library and information practice web site (http://journals.library.ualberta.ca/index.php/eblip) for further information about the journal. research article   using ethnographic methods to explore how international business students approach their academic assignments and their experiences of the spaces they use for studying   kathrine s. h. jensen research assistant university of huddersfield huddersfield, west yorkshire, united kingdom email: kathrineshjensen@gmail.com   bryony ramsden, phd subject librarian university of huddersfield huddersfield, west yorkshire, united kingdom email: b.j.ramsden@hud.ac.uk   jess haigh subject librarian university of huddersfield huddersfield, west yorkshire, united kingdom email: j.m.haigh@hud.ac.uk   alison sharman academic librarian university of huddersfield huddersfield, west yorkshire, united kingdom email: a.sharman@hud.ac.uk     received: 24 sept. 2018                                                                  accepted: 14 july 2019      2019 jensen, ramsden, haigh, and sharman. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29509     abstract   objective – understanding students’ approaches to studying and their experiences of library spaces and other learning spaces are central to developing library spaces, policies, resources and support services that fit with and meet students’ evolving needs. the aim of the research was to explore how international students approach academic assignments and how they experience the spaces they use for studying to determine what constituted enablers or barriers to study. the paper focuses on how the two ethnographic methods of retrospective interviewing and cognitive mapping produce rich qualitative data that puts the students’ lived experience at the centre and allows us a better understanding of where study practices and study spaces fit into their lives. methods – the study used a qualitative ethnographic approach for data collection which took place in april 2016. we used two innovative interview activities, the retrospective process interview and a cognitive mapping activity, to elicit student practices in relation to how they approach an assignment and which spaces they use for study. we conducted eight interviews with international students in the business school, produced interview notes with transcribed excerpts, and developed a themed coding frame.   results – the retrospective process interview offered a way of gathering detailed information about the resources students draw on when working on academic assignments, including library provided resources and personal social networks. the cognitive mapping activity enabled us to develop a better understanding of where students go to study and what they find enabling or disruptive about different types of spaces. the combination of the two methods gave students the opportunity to discuss how their study practices changed over time and provided insight into their student journeys, both in how their requirements for and knowledge of spaces, and their use of resources, were evolving.   conclusion – the study shows how ethnographic methods can be used to develop a greater understanding of study practices inside and outside library spaces, how students use and feel about library spaces, and where the library fits into the students’ lives and journey. this can be beneficial for universities and other institutions, and their stakeholders, looking to make significant changes to library buildings and/or campus environments.     introduction   user research in academic libraries in the past has often focused on quantitative data to learn about their users, utilising results from the national student survey and statistics such as gate entries and book borrowing data, providing a limited understanding of library use. however, library staff have increasingly used ethnographic methods to develop a richer understanding of students’ usage patterns and needs within and outside of their study spaces. this contextual qualitative data about students’ lived experience is not easily available through other methods and can be of great use in developing library staff’s understanding of the preferences and practices of [potential] users. the research presented in this paper was inspired by a large scale quantitative piece of research which identified some groups of students as ‘low users’ of the library spaces and resources (i.e. those who rarely or never visited the library, and rarely or never accessed library subscribed resources) (stone & ramsden, 2013; collins & stone, 2014; stone, sharman, dunn & woods, 2015; sharman, 2017). one low user group identified in the quantitative research was international students. the quantitative study did not have any information about spaces students might go to study, what they thought about library spaces or which resources they might access, if not library resources. the study offered a very limited view of student practices and we wanted to understand more about how students access resources and support for their studies. also, the term ‘low user’ is conceptualised primarily from the perspective of the service goals of an academic library and as such can be seen to have negative connotations by classifying students as somehow deficient. this was not how we approached or perceived the students and the focus on international students were therefore formed in response to them being a group that we seemed to know very little about (although this could be said to be true of most of the student groups identified in the quantitative study). our approach was exploratory and aimed at developing a more holistic view of student practices and experiences. we turned to the business school for participant recruitment as they have the highest percentage of international students. the students recruited were not specifically identified as ‘low users’, or part of the previous quantitative research study, but volunteered as participants in our qualitative study.   in this method-focused paper we argue that, in order to develop a better understanding of the students’ study practices, we need to focus on gathering data about the contexts and processes that students are situated within, and engage in: the purpose of this paper is to demonstrate the effectiveness of the methods used in gathering richer data that can improve our understanding of student practices. in terms of study practices, our focus is on how they approach academic assignments and considering how their study practices are enabled or disrupted by the spaces they use for studying.   we gathered data regarding these practices by utilizing retrospective interviews to explore their assignment processes and employed cognitive mapping within interviews to explore their experiences of and attitudes towards spaces where they studied.   the retrospective interview technique involves asking the participant to draw and then explain how they go about an activity in order to understand their process, the resources they make use of on and their decision making. the cognitive mapping technique also involves the participant spending a short time drawing a map based on a theme, in this case study spaces. the map is then labelled by the participant with explanatory details and discussed with the interviewer. the details of the map are discussed within the interview and form part of the transcript that is then coded and analyzed. you could potentially carry out a separate analysis of the maps, but in this study they were analyzed as part of the interview discussion.    both techniques produce rich data that can be used to provide prompts for discussion and to explore experiences, concepts and perceptions in more detail. these methods put the experiences of the participant at the centre of the research process and can therefore yield the kind of qualitative data that is a crucial part of understanding how complex the everyday lives of students are. the maps are a great way to showcase the interrelatedness of studying, and can also be used to complement, critique and contextualise patterns and issues identified by quantitative studies. library practitioners and other professionals can adapt the methods discussed in this paper, as well as the coding themes identified, to their own contexts to focus on different user groups. the details of the implementation and analysis of the ethnographic data can act as a framework for staff in other academic libraries to explore their students’ academic practices, develop their understanding of the student journey, and gather details of students’ experiences of, and preferences in relation to, learning spaces.     literature review: ethnographic methods in academic libraries   incorporating and adapting ethnographic methods to explore the user experience (ux) of libraries is an evolving field (gibbons, 2013; goodman, 2011; lanclos & asher, 2016; priestner & borg, 2016; ramsden, 2016) and driven by a recognition that such a qualitative approach offers opportunities to gather meaningful data about the ‘how’ and ‘why’ of student behaviour. this literature review focuses on the use of ethnographic methods in library research, providing an overview of techniques as a background and foundation for understanding the context of our own research intentions. the papers included demonstrate and discuss the importance of utilizing ethnographic methods and how the data can be more beneficial to service (non)users. for a more detailed, comprehensive review of the use of ethnographic methods in libraries, refer to khoo, rozakilis and hall (2012), who provide extensive information on both the variety of methods employed and the purpose of their use.   using ethnographic methods enables us to gain an insight into the complexity of students’ everyday lives, which can inform and improve the design of study spaces, library signposting practise, library policies and other support services. the use of ethnography is frequently identified as central in library research to aid in understanding student practises. tewell, mullins, tomlin and dent (2017) highlight that as social behaviours are central to ethnography it is “particularly useful for developing insights into people’s experiences and expectations” (p. 80). these insights into the everyday lived experience of students can then feed into the way spaces and services are organised: “being aware of student research processes and preferences can result in the ability to design learning environments and research services that are more responsive to their needs” (tewell, mullins, tomlin, & dent, 2017, p.79). lanclos and asher (2016) argue that ethnography as an approach enables a holistic focus for the research in order to consider contexts and connections outside of or beyond the students’ engagement with the library. in a study of students’ research processes at an irish university, dunne (2016) considers that “…gathering data about individual student interactions through ethnographic study captures the physical use of space and the emotional experience of students, in a way that surveys and interviews cannot” (p. 412).   in research carried out at the university of rochester in the usa to explore student practices, ethnographic methods were used in order to “…learn more about where students like to study and why, with whom, and when” (gibbons & foster, 2007, p. 20). this research also involved several other methods, including retrospective interviews. the interviews required students to draw the process of completing an assignment, while describing each step as they drew. students were also asked to keep a “mapping diary” detailing where they went throughout the day (briden, 2007, p. 40). influenced by the work by foster and gibbons (2007), the erial project (asher & miller, 2011) was a two-year ethnographic study of the student research process at five universities in illinois, usa. the researchers adopted multiple methods, including the cognitive mapping method developed by mark horan (1999) to explore students’ knowledge of libraries. horan (1999) describes the output as a ‘sketch map’ and claims they “can give patrons an opportunity to express things for which they perhaps do not have the words” (p. 194). the erial project (asher & miller, 2011) also utilized retrospective research interviews when researching student practices at the university of rochester. the combination of visual data alongside qualitative interviews that explained the images in their context proved a particularly powerful research tool in discovering student practices (asher & miller, 2011).   ethnographic approaches have also been used to explore the experiences of specific groups of students. in regalado and smale’s (2015; n.d.) extensive research of commuter students at the city university of new york (cuny) a key finding was that students “valued the library as a distraction-free place for academic work, in contrast to the constraints they experienced in other places including in their homes and on the commute” (regalado & smale, 2015, p. 899). the data from this large scale study includes students’ maps of their daily routes, photographed items related to their academic lives, and representations of their research processes (regalado & smale, 2015; smale & regalado, n.d.).   the benefit of focusing on the complexity of student lives is further reinforced by a recent u.s. study focusing on how users experience the library in the context of their lives. ‘a day in the life’ of over 200 students’ focused on students’ lives, and the library’s place in it, undertaking collaborative ethnographic research as part of a mixed methods approach (asher, amaral, couture, fister, lanclos, lowe, regalado, & smale, 2017). in a presentation at the association of college & research libraries conference on ‘the topography of learning: using cognitive mapping to evolve and innovate in the academic library’, the benefit of getting participants to produce and discuss maps of their practises is that they can contribute to revealing the unrevealed and offers a way to ‘provide narrative to accompany statistics’ (lanclos, smale, asher, regalado, & gourlay, 2015).   qualitative research designed with students at the centre, particularly utilising ethnographic or ux based designs, is clearly a key route to developing understanding of library users and the lives of students more generally. our intention when carrying out our own research was to do just that, learning about our students and what was important to them in their study practises. additionally, our research gave us an opportunity to further our understanding of ethnographic practise in library user research.   methods   in order to explore the reasons why international students may be low users of the library and library resources, we designed a study to gather more contextual knowledge about their academic practises, where they study and how they approach academic assignments. we already knew that we were looking to learn more about where students liked to study (and why), as well as to explore what resources the students used for their studies, including how they accessed the resources and who they worked with in this process, e.g. tutors, librarians, peers, friends, etc. these were the initial parameters for designing the study and guided the analysis of the data later on.   we chose the methods of cognitive mapping (as per asher and miller (2011) above) and retrospective process interviews because they involve the participants in producing something which is not driven by questions from the researcher. what the participant produces can then form the basis of the subsequent conversation. these methods ensure that the participants’ experience and meaning making is at the centre of the research and frames the data produced in the interview.   the instructions were given in a short verbal explanation to the students and also written down. this served as a useful reminder, but also helped to communicate the various steps involved in the exercise to international students. we stressed there was no right way go about doing these exercises and if, for example, the students didn’t feel comfortable drawing the spaces they frequented, they could produce a mind map or simply write down keywords.    we believe these methods are well suited to developing meaningful discussions with international students who may not have english as a first language. the method of cognitive mapping had previously been employed as part of a research project about academic study practises and more general use of campus spaces at the university of huddersfield (ramsden, jensen, & beech, 2015). the research highlighted how useful the maps were in getting details about the complex reasoning behind students’ choice of study spaces. the research also indicated that the maps were very useful as interview talking prompts and we therefore saw the benefit of using this approach in getting students to tell us about the characteristics of the spaces they choose to study in.   following the instructions for the activity from asher and miller (2011), the mapping activity was carried out with three differently coloured pens, with the interviewee changing pens every two minutes (3x2 minutes). the idea behind the mapping activity is that the participant will first draw what is the main or most important area for them. we asked students to draw a map of where they go to learn or study, and gave them six minutes in total to complete the exercise. following the drawing exercise, we asked participants to label the spaces and add details as we talked through their maps as part of a recorded interview. the prompt for the mapping activity was:   you will be given six minutes to draw from memory a map of where you go to learn or study (your learning spaces). every two minutes you will be asked to change the colour of your pen in the following order: 1. blue. 2. green. 3. red. after the six minutes are completed, please label the features on your map. please try and be as complete as possible, and don’t worry about the quality of the drawing.   the second method, retrospective process interviews, was utilized to learn about students’ approach to writing and researching assignments, and to explore what resources, from online databases and search engines to their peers or academic colleagues, they use in this process. our use of the method drew from foster and gibbons (2007) and asher and miller (2011), who recommend this method for ‘step-by-step processes’ that require students to recall how they did a particular activity. in contrast to the cognitive mapping, there was no time limit or requirement to swap pen colours. the prompt for the retrospective process interview was:   please describe how you did your last assignment. begin with when you first got the assignment brief/title, how/where you looked for information, how you wrote it and end with when you submitted it on turnitin. please draw each step below.   in order to get some ideas about the different factors and decision making that came into play in their study processes, we then asked follow-up questions. for example, where did the students seek help, did they rely on reading list items or engage in wider reading and did they use the specialist resources purchased for their subject area. it was felt to be a particularly effective and simple method to use with international students to facilitate useful conversations to find out more about their study habits and how they differed to that of their u.k. peers.   recruiting students rather than recruit international students from all the disciplines, we concentrated on the business school. we chose students from the business school partly because they have the largest population of international students and partly because they had compulsory classes, where we could ask for volunteers to participate in the study. we recognize that this recruitment process may represent some limitations for the study in terms of constituting a convenient sample and capturing only subject specific practices. this research is a snapshot of a particular set of students, from varying backgrounds and cultures, studying in the uk for a limited time. this methodology allows researchers to understand the practises of students who have had to adapt their own norms of studying to those of the university they find themselves in. readers and practitioners should be aware that cultural differences may have informed previous, and current, study practices of the students involved.    tutors from the business school’s international learning development group recruited the students. the students were a mixture of undergraduate and postgraduate; four participants were chinese, while others came from iraq, thailand, vietnam, and morocco. the students received a £10 voucher as an incentive to participate and to reimburse them for their time. the interviews were recorded and lasted from 30 minutes to about an hour. the interviews were carried out by all the team members. following the interviews, we produced notes with transcribed excerpts.   thematic analysis the qualitative data from the interviews was analyzed by identifying patterns of meaning across the data in order to develop themes (braun & clarke, 2006). our focus was on the experiences and the reality of the participants in relation to the topic areas of study practises and study spaces that we identified at the beginning.   although we were building on existing research, the team decided to do the initial coding of the interviews without a predetermined framework of themes. this was to allow for any unexpected topic areas that participants might focus on or highlight as being of specific importance for their experiences and practises.   developing consistent coding in order to code the interviews consistently, all the team members first coded the same interview and then met to discuss the codes assigned. this initial coding of the data aimed to develop themes in an inductive way, to be as close to the data as possible and therefore produce themes mostly descriptive in nature. however, it is important to recognize that we were building on previous research, which meant some of the initial coding was more interpretive. one example of interpretive coding is that we decided to code whether something was a “study enabler” or “study barrier” for the students’ practises. codes were subdivided again into comments that expressed positive or negative attitudes towards the theme in terms of how they affected their study and labelled as such to distinguish how the same things can have different resonance to different people. for example, some people find group work enables their study, whereas some find other people a distraction.   following this initial phase, a detailed list of codes and sub-codes was developed to code the interviews. we then stripped the interviews of their codes, swapped amongst team members, and re-coded. all the team met to compare the re-coded interviews and amalgamate or refine codes as needed. one team member created a spreadsheet with all the coding incidences across the eight interviews, and this formed the basis for the team to write up the findings across the themes that were most prevalent. the codes are outlined in appendix 1, and can be used as a starting point for anyone planning research into this area. they have already informed further research into the user experience at the university.   second stage data analysis consisted of revisiting the interviews to flesh out the selected coding themes. initial coding of the eight interviews included ten themes. when the team reviewed the coding themes in all the interviews, four areas were identified as common across the interviews and were therefore explored in more detail. this doesn’t mean that themes that were identified in perhaps just one interview were not taken into account in the presentation of the findings. we recognize that the commonality of themes was also produced in part because we were asking interviewees about specific aspects of their practise.     figure 1 a cognitive map of the student’s study spaces.     results: where students go to study and why   the cognitive mapping activity helped us to identify the types of spaces where students went to study and what made some spaces better suited for studying than others. the maps visualized the components of the different spaces, and the subsequent discussion enabled the interviewer to ask follow-up questions as to what was positive and negative in terms of studying in each of the spaces drawn. the maps varied considerably in terms of the details drawn but they were excellent for pointing us towards what the students found enabling or disruptive for study purposes, and to their requirements and expectations of the different types of spaces. this information on the varying requirements of students is useful in supporting a university in making decisions on what types of spaces meet student needs at different times.    figure 1 is an example of one of the cognitive maps produced during the interviews, and here we can see the first space mapped in blue pen as depicting a single desk with a computer and a chair. importantly, this space has then been labelled as a specific floor in the library to identify the space in more detail. the second space in green pen is a communal space where the key items are a large table with a laptop, and this is also later labelled with the name of a central student cafeteria area on campus: the student likes the option to eat and drink while studying, which library rules prevent. and finally, the last space in red pen is the home environment where a bed, desk, music, food, and friends are drawn in detail. in the interview, the student explained that not much studying happens when they meet up with friends at home. the map is a useful visual prompt for the interviewer (and student) to discuss what is enabling and what is disrupting about the different spaces, as well as valuable in identifying the resources they use and their study practises.     figure 2 retrospective interview map from interview eight.     results: retrospective process maps of approaches to academic assignments the retrospective process maps were valuable as a tool to discover what resources students used in planning their assignments, and what different people they involved in this process (classroom peers, lecturer, tutor, support staff, library staff, etc.).   figure 2 shows an example of an output from the retrospective process interview demonstrating a starting point for researching an assignment, and then subsequent key points in the student’s planning process including the resources they are drawing on. on the map, the student indicates that their first action after getting the assignment title is to ask classmates for help, but if they require additional support they email the tutor.   in the example, the student also indicates the different practises they have used from the first to their third year, giving the interviewer a useful prompt for further discussion, as well as producing valuable data on their student journey. the discussion covers the development of academic skills, from using google as a first source in the first year to a growing understanding of the benefit of using academic references in the third year, which is prompted by a discussion with his tutor following a poorly graded assignment.    i got 45% and my tutor said you have to paraphrase the paragraph and change your structure and you have to put the name, author’s name and the year phew! (interview eight.)   discussion: space preferences and student needs   our analysis of the discussion of the maps highlighted that students change where they go depending on what they need to get done and what their current preferences or needs are, and they plan this according to their evolving knowledge about the spaces that work for them. the picture this rich data presents challenges simplistic assumptions about the library as a study space and foregrounds the complexity of student decision making around studying. the theme of convenience and the link to current location and schedule emerged strongly in the multi-sited u.s. study mapping student days. they found that:   though students across the board were most likely to report a feeling of happiness when they were at home, the choices they made for studying depended on convenience (such as proximity to their next destination) and on surroundings that encouraged them to do academic work (which could be a designated space at home or could be table or carrel in a library where being in the company of other students encouraged focus).  (asher, amaral, couture, fister, lanclos, lowe, regalado, & smale, 2017, p. 310)   for example, a student discussed going from one floor of the library to another floor as a result of discovering a space that matched their needs to discuss their work with fellow students.   i found the room here [indicates the room opposite the interview space, which is an open area] you can talk with your classmates, because we are discussing. because if you work alone you will just sleep, you will not get your work done this way … if you go to silent area, to quiet area you can’t talk, you can’t discuss. (interview seven.)   understanding the changing and different space requirements students have is important for staff working in libraries if they want to play a key role in enabling the study practises of their students.   another student talked about the library space as being part of their social network, and this is demonstrated in the maps where social and academic activities can be seen to overlap. the details of the maps also underline the importance for students to be able to easily access food and drink in order to carry out their academic work. when students talked about using non-campus spaces like cafes, the benefits they mentioned were their atmospheres, such as lighting, music, and the smell of coffee.   some of the students preferred to work at home because they had more autonomy over what they could do within the space, including eating and drinking. being able to eat and drink is a large factor in how they think of spaces, though it is possible that this reflects a general wish to have more control over the environments they work in. one student mentioned that in their personal room, they can de-stress and do what they want as they study. the positives of working in the ‘home space’ give us some ideas about why the library spaces might not be the students’ first choice.   we did find that students appreciated the bookable group areas in the library, as being able to work with their peers at times that are best for them was important. some students found working with friends a distraction, but still used the group work areas whilst being aware that they do their best work solo in their own space. for these students, the library was therefore a social study space, rather than a solo study space. allan (2016) found similar student behaviours in terms of adapting the space for personal needs.   in relation to exploring low usage of the library, the methods helped us to discover some reasons why home is a preferred study space, to learn that the international students were unfamiliar with the support they could get from librarians, and that navigating the library classification system remains a challenge for most.   student confusion about the role of librarians, as well as which staff are librarians, is also highlighted in a u.s. ethnographic study using observation and interviews with library users in an academic library recently relabelled as an ‘information commons’ (allan, 2016). that students do not make use of librarians in relation to support with their studies, but only for more ‘directional support’ such as asking where a book is located is also reported in another recent u.s. study (tewell, mullins, tomlin, & dent, 2017).   the analysis of the discussion of the maps from the retrospective interviews enabled us to identify student support networks as the students talked about making use of friends, peers or classmates, tutors, and other support services. for example, it became clear that the international learning development group (ildg) played a central role in supporting international students as students referred to ildg as being how they had become aware of resources to use and learned about referencing, how to check their work for plagiarism, and was also somewhere they could book appointments to discuss their drafted work. in contrast, the students did not mention the librarian in relation to information searching and referencing, which has led us to consider ways of raising awareness of the librarian role and to consider better signposting for the librarian help desk in the physical library space.   the cognitive map activity allowed us to better ask questions about what students liked or did not like about the spaces they had drawn and how this was connected to enabling or disrupting their studying. it also enabled us to collect rich data about the different ambiances of the spaces, including how others contribute towards that ambiance, and making connections as to how this contributed to the feelings of students towards the space and their use of it.   we believe one reason students expressed a preference for studying in ‘home spaces’ is that in these spaces students have more control over their environment, such as access to food and drink, noise levels, and soft furnishings. conversely, some students recognized that this meant that the benefits of the home space could also turn into distractions, which were barriers to getting studying done.   the need to adapt space to different individual requirements is reported by tewell, mullins, tomlin, and dent (2017) as they found students attempted to create their own temporary ‘home spaces’ for studying within the library by moving furniture and books.   the research findings led us to recommend that regulations regarding the consumption of food and drink within the library should be reconsidered, especially concerning access to hot drinks as this is clearly an issue of great concern to students and central to their study practises. the students’ attitude to working in library spaces is negatively impacted by the enforcement of a no-hot-drinks policy, which is a policy that appears incongruent to the development of the library as being open 24/7. the library has since relaxed the rules on allowing hot drinks, as long as they are brought into the library in a travel mug, and provides a space for students to go relax and eat snacks should they want to take a break without leaving the library.   the rich detail from the mapping approaches is a reminder of the need for a holistic approach that takes into account the complexity of student lives when looking at study practises. we are reminded of the embodied and embedded nature of any activity.   conclusion   we have made a case for the value of ethnographic methods in exploring the contexts and processes of students’ study practises, as these methods allow us to gather rich qualitative data that offer a more holistic representation of the students’ lived experience. understanding our students’ lives and needs are important, and is the key to optimally developing and evaluating library spaces, policies, resources, and support services. the retrospective process interview offered a way to gather details and develop knowledge of student use of resources (including those provided by the library) and their social networks to aid their studying. the cognitive mapping activity provided us with a better understanding of where students go to study and what they find enabling or disruptive about different types of spaces. the combination of the two methods allowed the students to talk about changes over time to the way they did academic work and gave us insight into their student journeys; their requirements for and knowledge of spaces, as well as their use of resources, were evolving. the data collected helped to highlight the international student experience and study culture. further research has been planned to use the same research techniques with u.k. students to find out whether they experience similar issues and engage in similar practises.   the study shows how ethnographic methods can be used to develop a greater understanding of how students use and feel about library spaces and where library staff, resources, and spaces fit into the students’ lives and journeys. this can be beneficial for universities, other institutions, and their stakeholders looking to make significant changes to library buildings or campus environments. the student practises and preferences revealed by the maps also formed the basis for practical recommendations, such as the necessity of ensuring the library opening hour policy is congruent with the facilities for refreshments and storage available to students. although this was small scale research, it demonstrates the value of mini projects to develop targeted data collection to contextualize and develop our understanding of quantitative and survey data. such projects enable the development of an analytical framework that can then be built on, and allow for accumulation of evidence that can impact practise.   references   allan, e. g. (2016). ethnographic perspectives on student-centeredness in an academic library. college and undergraduate libraries, 23(2), 111–129. https://doi.org/10.1080/10691316.2014.965374   asher, a., amaral, j., couture, j., fister, b., lanclos, d., lowe, m. s., regalado, m, & smale, m. a. (2017). mapping student days: collaborative ethnography and the student experience. collaborative librarianship, 9(4), 293–317. retrieved from https://digitalcommons.du.edu/collaborativelibrarianship/vol9/iss4/7   asher, a., & miller, s. (2011). so you want to do anthropology in your library? or a practical guide to ethnographic research in academic libraries. retrieved from http://www.erialproject.org/wp-content/uploads/2011/03/toolkit-3.22.11.pdf   braun, v., & clarke, v. (2006). using thematic analysis in psychology. qualitative research in psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa   briden, j. (2007). photo surveys: eliciting more than you knew to ask for. in n. f. foster & s. gibbons (eds.), studying students: the undergraduate research project (pp. 40–47). chicago: association of college & research libraries.   collins, e., & stone, g. (2014). understanding patterns of library use among undergraduate students from different disciplines. evidence based library and information practice, 9(3), 51-67. https://doi.org/10.18438/b8930k   dunne, s. (2016). how do they research? an ethnographic study of final year undergraduate research behavior in an irish university. new review of academic librarianship, 22(4), 410–429. https://doi.org/10.1080/13614533.2016.1168747   foster, n. f., & gibbons, s. (2007). studying students: the undergraduate research project at the university of rochester. chicago: association of college & research libraries. retrieved from http://www.ala.org/acrl/sites/ala.org.acrl/files/content/publications/booksanddigitalresources/digital/foster-gibbons_cmpd.pdf   gibbons, s. (2013). techniques to understand the changing needs of library users. ifla journal, 39(2), 162–167. https://doi.org/10.1177/0340035212472846   gibbons, s., & foster, n. f. (2007). library design and ethnography. in n. f. foster & s. gibbons (eds.), studying students: the undergraduate research project (pp. 20–29). chicago: association of college & research libraries. retrieved from http://www.ala.org/acrl/sites/ala.org.acrl/files/content/publications/booksanddigitalresources/digital/foster-gibbons_cmpd.pdf   goodman, v. d. (2011). applying ethnographic research methods in library and information settings. libri, 61(1), 1-11. https://doi.org/10.1515/libr.2011.001   horan, m. (1999). what students see: sketch maps as tools for assessing knowledge of libraries. journal of academic librarianship, 25(3), 187–201. https://doi.org/10.1016/s0099-1333(99)80198-0   khoo, m., rozaklis, l., & hall, c. (2012). a survey of the use of ethnographic methods in the study of libraries and library users. library & information science research, 34(2), 82–91. https://doi.org/10.1016/j.lisr.2011.07.010   lanclos, d., & asher, a. d. (2016). “ethnographish”: the state of the ethnography in libraries. weave: journal of library user experience, 1(5). https://doi.org/10.3998/weave.12535642.0001.503   lanclos, d., smale, m. a., asher, a., regalado, m., & gourlay, l. (2015). the topography of learning: using cognitive mapping to evolve and innovate in the academic library. in association of college & research libraries conference, march 27. retrieved from https://prezi.com/qvhdcuiikine/the-topography-of-learning-using-cognitive-mapping-to-evolve-and-innovate-in-the-academic-library/   priestner, a., & borg, m. (2016). user experience in libraries: applying ethnography and human-centred design. new york, ny: routledge. retrieved from https://www.routledge.com/user-experience-in-libraries-applying-ethnography-and-human-centred-design/priestner-borg/p/book/9781472484727   ramsden, b. (2016). ethnographic methods in academic libraries: a review. new review of academic librarianship, 22(4), 355–369. https://doi.org/10.1080/13614533.2016.1231696   ramsden, b., jensen, k., & beech, m. (2015). cognitive mapping and collaborating. ukanthrolib blog, november. retrieved from https://ukanthrolib.wordpress.com/2015/11/   regalado, m., & smale, m. a. (2015). “i am more productive in the library because it’s quiet:” commuter students in the college library. college & research libraries, 76(7), 899–913. https://doi.org/10.5860/crl.76.7.899   sharman, a. (2017) using ethnographic research techniques to find out the story behind international student library usage in the library impact data project. library management, 38(1), 2-10. https://doi.org/10.1108/lm-08-2016-0061   smale, m. a., & regalado, m. (n.d.). finding places, making spaces. retrieved february 22, 2019, from https://ushep.net/   stone, g., & ramsden, b. (2013) library impact data project: looking for the link between library usage and student attainment. college & research libraries, 74(6), 546-559. https://doi.org/10.5860/crl12-406   stone, g., sharman, a., dunn, p., & woods, l. (2015). increasing the impact: building on the library impact data project. journal of academic librarianship, 41(4), 517-520. https://doi.org/10.1016/j.acalib.2015.06.003   tewell, e., mullins, k., tomlin, n., & dent, v. (2017). learning about student research practices through an ethnographic investigation: insights into contact with librarians and use of library space. evidence based library and information practice, 12(4), 78–101. https://doi.org/10.18438/b8mw9q     appendix 1 interview codes and sub-codes   1. resources 1.1 discovery systems/search tools 1.1.1 library, summon 1.1.2 search engine (google, bing, yahoo etc) 1.1.3 academic search engine (google scholar, academia.edu) 1.2 library physical resources (books) 1.3 library subscription resources (databases, journals, mintel etc) 1.4 internally monitored non-subscription resources (unilearn, myreading) 1.5 non-subscription external non-monitored resources (wikipedia, websites) 1.6 personally owned resources (books) 1.7 ebooks 2. help/support 2.1 peer 2.1.1 peer face to face 2.1.2 peer online (facebook etc) 2.2 tutor 2.2.1 tutor face to face 2.2.2 tutor online (email) 2.3 librarian 2.3.1 librarian face to face (one to one, help centre) 2.3.1.1 librarian teaching session/induction 2.3.2 librarian online (question point, email) 2.4 ildg 2.5 it support 2.5.1 it support within the library 2.5.2 it support remotely 2.6 library support staff (student helpers) 2.7 interlibrary loans staff 3. time 3.1 management of time 3.2 time of day specific activities 3.2.1 morning 3.2.2 afternoon 3.2.3 evening 3.2.4 night 4. space/use of space 4.1 library 4.1.1 floor specific 4.1.1.1 floor 2 4.1.1.2 floor 3 4.1.1.3 floor 4 4.1.1.4 floor 5 4.1.1.5 floor 6 4.1.2 bookable group area 4.1.3 silent working area 4.1.4 wayfinding 4.2 home (bedroom, halls) 4.3 external multi-use environments (coffee shop etc) 4.4 internal multi-use environments 4.4.1 student central 4.4.2 business school 4.5 non-study 4.5.1 home 4.5.2 library 4.5.3 external multi-use environments 4.5.5 internal multi-use environments 5. student journey (changes between years) 6. country differences 6.1 language 6.2 structure of course 6.3 culture 6.4 academic expectations 6.5 library 7. food and drink 7.1 food 7.1.1 snacks 7.1.2 main meals 7.2 drink 7.2.1 drink-hot 7.2.1 drink-cold 8. study style 8.1 noise/quiet (preferred noise level important) 9. technology use 9.1 laptops (personal) 9.2 library technology 9.2.1 laptops (borrowed) 9.2.2 library printers 9.2.3 computers 10. academic skills 10.1 grades 10.2 critical thinking 10.3 exams 10.4 structure 10.5 references 10.6 search skills (truncations etc.) 10.7 plagiarism (turnitin etc.)     evidence based library and information practice editorial responsibilities      2017. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     editor-in-chief: lorie kloda   associate editor (evidence summaries): heather pretty   associate editor (research articles): rebekah (becky) willson, lisl zach   associate editor (classics & reviews): i. diane cooper   associate editor (commentaries, using evidence in practice, research in practice): lorie kloda   associate editor (feature): lisl zach   communications officer (news): tatiana bryant   production editor: michelle dunaway   editorial intern: rachel hinrichs   editorial advisors: lindsay alcock, alison brettle, jonathan eldredge, denise koufogiannakis   copyeditors: richard hayman (lead copyeditor), georgianne bordner, julie evener, heather healy, julie james, jane morgan-daniel, stacey penney, elizabeth stregger,  nikki tummon, alison yeoman   indexing support: kate shore   evidence summary   scale evaluating the information literacy self-efficacy of medical students created and tested in a six-year belgian medical program   a review of: de meulemeester, a., buysse, h., & peleman, r. (2018). development and validation of an information literacy self-efficacy scale for medical students. journal of information literacy, 12(1), 27-47. retrieved from https://ojs.lboro.ac.uk/jil/article/view/pra-v12-i1-2     reviewed by: brittany richardson web services librarian, assistant professor university of tennessee at chattanooga library chattanooga, tn, united states of america email: brittany-richardson01@utc.edu   received: 1 mar. 2019                                                                    accepted: 5 apr. 2019      2019 richardson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29564     abstract   objective – to create and validate a scale evaluating the information literacy (il) self-efficacy beliefs of medical students.   design – scale development.   setting – large, public research university in belgium.   subjects – 1,252 medical students enrolled in a six-year medical program in the 2013-2014 academic year.   methods – ten medical-specific il self-efficacy questions were developed to expand a 28-item information literacy self-efficacy scale (ilses) (kurbanoglu, akkoyunlu, & umay, 2006). medical students in years 1 – 5 completed the questionnaire (in english) in the first two weeks of the academic year, with students in year 6 completing after final exams. respondents rated their confidence with each item 0 (‘i do not feel confident at all’) to 100 (‘i feel 100% confident’). principal axis factoring analysis was conducted on all 38 items to identify subscales. responses were found suitable for factor analysis using bartlett’s test of sphericity and the kaiser-meyer-olkin measure (kmo). factors were extracted using the kaiser-gutmann rule with varimax rotation applied. cronbach’s alpha was used to test the internal consistency of each identified subscale. following a one-way-anova testing for significant differences, a tamhane t2 post-hoc test obtained a pairwise comparison between mean responses for each student year.   main results – five subscales with a total of 35 items were validated for inclusion in the information literacy self-efficacy scale for medicine (ilses-m) and found to have a high reliability (chronbach’s alpha scores greater than .70). subscales were labelled by concept, including “evaluating and processing information” (11 items), “medical information literacy skills” (10 items), “searching and finding information” (6 items), “using the library” (4 items), and “bibliography” (4 items). the factor loading of non-medical subscales closely reflected studies validating the original ilses (kurbanoglu, akkoyunla, & umay, 2006; usluel, 2007), suggesting consistency in varying contexts and across time. although overall subscale means were relatively low, immediate findings among medical students at ghent university demonstrated an increase in the il self-efficacy of students as they advance through the 6-year medical program. students revealed the least confidence in “using the library.”   conclusions – the self-efficacy of individuals in approaching il tasks has an impact on self-motivation and lifelong learning. the authors developed the ilses-m as part of a longitudinal study protocol appraising the il self-efficacy beliefs of students in a six-year medical curriculum (de meulemeester, peleman, & buysse, 2018). the ilses-m “…could give a clear idea about the evolution of perceived il and the related need for support and training” (p. 43). further research could evaluate the scale’s impact on curriculum and, conversely, the impact of curricular changes on ilse. qualitative research may afford additional context for scale interpretation. the scale may also provide opportunities to assess the confidence levels of incoming students throughout time. the authors suggested further research should apply the ilses-m in diverse cultural and curricular settings.   commentary   the impact of self-efficacy beliefs on il behaviors and lifelong learning was first studied by kurbanoglu (2003), who suggested “perceived self-efficacy can be accepted as one of the psychological factors which has an impact on information literacy” (p. 637). through a study of teachers in turkey, kurbanoglu, akkoyunlu, and umay (2006) developed and validated a 28-item information literacy self-efficacy scale (ilses) in both turkish and english. the scale was further tested for factor loading and reliability by usluel (2007). the ilses has been applied in the medical discipline as a measure of the il self-efficacy of nursing students (özbıçakçı, gezer, & bilik, 2015; robertson & felicilda-reynaldo, 2015). as kurbanoglu (2003) suggested, self-efficacy is “domain-specific” (p. 636). the authors’ study contributed to the literature by expanding the scale through the inclusion of medical discipline-specific items.   the study demonstrated strengths when evaluated using glynn’s (2006) ebl critical appraisal checklist: the response rate was high; participants represented students from all years of a medical school curriculum; the instrument was included in the publication; and suggestions were provided for further application of the ilses-m scale. results of the factor analysis were meaningfully presented, including comparisons of scale factor loadings with those of previous studies.   there were also several areas for study improvement. first, further information on questionnaire administration and the development of the ten medical-specific scale items would have benefited practitioners interested in undertaking similar efforts. additionally, the use of acronyms (e.g. pico, mesh) and potentially vague definitions (e.g. “use a factual database,” “evaluate bias”) in some scale items may have impacted response accuracy. review of scale items by content experts and members of the target population could have tested content validity and highlighted any needed improvements. overall, the study methodology focused heavily on factor analysis and scale reliability. boateng, neilands, frongillo, melgar-quiñonez, and young (2018) suggested scale creation additionally requires item validation and pre-testing, as well as tests for item reduction analysis, dimensionality, and validity after initial responses are gathered. although the reported methodology for subscale creation was robust, applying additional steps would have ensured a thoroughly validated ilses-m scale.   the use of the ilses-m in a longitudinal study may provide further insights into its validity and applicability (de meulemeester, peleman, & buysse, 2018). ultimately, the newly created ilses-m scale contributes meaningfully to il self-efficacy measurement efforts through the creation of a discipline-specific tool tested in a large population. its potential applicability in other medical settings should be considered and may afford opportunities to further validate the scale.   references   boateng, g. o., neilands, t. b., frongillo, e. a., melgar-quiñonez, h. r., & young, s. l. (2018). best practices for developing and validating scales for health, social, and behavioral research: a primer. frontiers in public health, 6, 1-18. https://doi.org/10.3389/fpubh.2018.00149   de meulemeester, a., peleman, r., & buysse, h. (2018). medical students’ information literacy self-efficacy: longitudinal study-protocol covering a whole medical curriculum. in s. kurbanoglu, s., boustany, j.,  špiranec, e., grassian, e., mizrachi, d., & l. roy (eds.). ecil2017: information literacy in the workplace (pp. 419–429). switzerland: springer international publishing. https://doi.org/10.1007/978-3-319-74334-9_44   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154   kurbanoglu, s. s. (2003). self-effica­cy: a concept closely linked to information literacy and lifelong learning. journal of documentation, 59(6), 635-646. https://doi.org/10.1108/00220410310506295   kurbanoglu, s. s., akkoyunlu, b., & a. umay. (2006). developing the information literacy self-efficacy scale. journal of documentation, 62(6), 730-743. https://doi.org/10.1108/00220410610714949   özbıçakçı, ş., gezer, n.& bilik, ö. (2015). comparison of effects of training programs for final year nursing students in turkey: differences in self-efficacy with regard to information literacy. nurse education today, 35(2), 73–77. https://doi.org/10.1016/j.nedt.2014.10.008   robertson, d. s., & felicilda-reynaldo, r. (2015). evaluation of graduate nursing students' information literacy self-efficacy and applied skills. journal of nursing education, 54(3), s26-30. https://doi.org/10.3928/01484834-20150218-03   usluel, y. k. (2007). can ict usage make a difference on student teachers' information literacy self-efficacy? library & information science research, 29(1), 92–102. https://doi.org/10.1016/j.lisr.2007.01.003   evidence summary   embeddedness creates opportunities for enhanced library liaison services and relationships   a review of: o’toole, e., barham, r., & monahan, j. (2016). the impact of physically embedded librarianship on academic departments. portal: libraries and the academy, 16(3), 529-556. http://dx.doi.org/10.1353/pla.2016.0032   reviewed by: richard hayman assistant professor & digital initiatives librarian mount royal university calgary, alberta, canada email: rhayman@mtroyal.ca   received: 2 nov. 2016   accepted: 2 jan. 2017      2017 hayman. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to examine whether liaison librarian interactions increase when librarians are physically embedded in their liaison areas.   design – natural experiment using quantitative measures.   setting – a large, public university in the united states of america.   subjects – liaison librarian reference interactions.   methods – this research is organized around four primary research questions that examine the effect of liaison librarian physical, co-located embeddedness on the following: 1) the frequency of walk-up reference transactions of the embedded location versus the service desk; 2) the frequency of reference and instructional transactions with liaison areas after the implementation of embedded services; 3) the frequency of walk-up transactions at embedded sites compared to the number of reference and instructional transactions after embeddedness began; and 4) liaison librarian participation in new collaborative or integrative activities with their liaison areas. researchers used data collected between fall 2012 and spring 2014 and compared this to data collected in the pre-embedded period for fall 2010 to fall 2011. data sources included the library’s locally developed reference services statistics tracking tool, individual librarians’ calendar appointment records, and librarian performance agreements. the analysis uses descriptive statistics.   main results – researchers discovered a decrease in the frequency of liaison librarians’ walk-up reference transactions at the service desk, as tracked by transactions per hour, occurring before the transition, during the transition, and after the transition to embedded librarianship. they note a decrease of 45% in the number of walk-up interactions at service points for the three librarians involved in the study from the pre-embeddedness service period during fall 2010 as compared to spring 2012. the data show this decline through spring 2013 before rebounding in fall 2013 and spring 2014. they identified a median decline of three transactions per hour at the service desk from the pre-embeddedness to post-embeddedness periods.   they identified an increase of 371% in the number of email transactions following the implementation of embedded librarianship as compared to the pre-embeddedness period. telephone interactions declined overall during the research period, though they were already in decline before the transition to embeddedness began. the overall number of face-to-face reference appointments increased during the transition to embeddedness and continued to rise during the post-embeddedness period, with a 275% increase in the median number of appointments between preand post-embeddedness periods. the new embeddedness service did not have as significant an impact on the frequency of information literacy instruction sessions, with a small increase of 11.5% between the preand post-embeddedness periods, but it did spur the creation of online course research guides, which saw an increase of 54%.   regarding the third research question, researchers totalled the combined numbers of reference transactions by phone, email, and appointment, and compared those against walk-up interactions and also against instruction activities. in both cases, they did not discover any apparent impact of embeddedness and the frequency of these activities.   the final research question addressed whether embeddedness led to liaison librarians having new collaborative and integrative activities with their subject areas. the researchers indicate that the liaison librarians “indeed experienced novel interactions with their assigned departments that fall into both categories” (p. 547). they highlight several types of activities experienced by the liaison librarians in the study, such as participating in the grant proposal process, assisting department projects, and involvement in student activities.   conclusion – this library’s expanded embedded library services led to an increased frequency of reference interactions, instruction opportunities, and opportunities for new collaborative and integrative activities between the liaison librarian and their subject area. this study reveals several opportunities for future research around embedded services as well as models of embeddedness, including opportunities to address impact and benefits of such services on the liaison areas.   commentary   this study reflects findings in other recent studies showing that embedded academic librarianship leads to increased interactions between librarians and students and faculty. for example, freiburger, martin, and nuñez (2016) highlight the benefits to instructional and collaborative interactions after eight years of embedded practice in the health sciences, while connolly-brown, mears, and johnson (2016) reveal the value of embeddedness for faculty and students in virtual environments, with a focus on remote library users.   this research provides a unique perspective on three academic librarians’ experiences with embedded librarianship. using glynn’s (2006) critical appraisal tool, the study is internally valid despite a number of weaknesses. there is limited generalizability of the methods or results beyond the specific liaison librarians at this particular library, as this study draws upon measures (e.g., an in-house reference statistics tool) and circumstances (e.g., having a librarian’s primary office located in the department they serve) that are unique to this institution. the small sample size is questionable, though internally the results do show that their local model of embeddedness has led to an increased frequency of librarian-to-liaison area interactions across the three disparate liaison areas examined.   the study benefits from the decision to use only quantitative measures, as this helps limit the influence of researcher bias within this self-study. however, this study would be enhanced by drawing on qualitative methods of self-study to examine the librarians’ experiences in their own words. such methods might allow this research to properly address the relationship between any benefits of moving to an embedded service and to further explore the impact of new collaborative and integrative activities examined in the final research question. instead, this research only highlights a correlation between changing the embedded model and the impact of liaison interactions, without establishing a causal relationship.   this research has practical implications for academic librarians considering or engaged in physical embedded librarian services. physical embeddedness is no longer new or unique to academic library settings, and this study provides further evidence those unique opportunities for embeddedness can be leveraged to help enhance relationships to one’s liaison area. one area for future research not identified is how physical embeddedness, such as being located in one’s liaison area, would operate for those liaison librarians serving multiple departments. finally, as the authors note, this research suggests there are opportunities to link expanded library services to impact and success measures at the institutional level, such as academic success, student retention, and successful research collaborations and funding applications.   references   connolly-brown, m., mears, k., & johnson, m. e. (2016). reference for the remote user through embedded librarianship. the reference librarian, 57(3), 165-181. http://dx.doi.org/10.1080/02763877.2015.1131658   freiburger, g., martin, j. r., & nuñez, a. v. (2016). an embedded librarian program: eight years on. medical reference services quarterly, 35(4), 388-396. http://dx.doi.org/10.1080/02763869.2016.1220756   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154     evidence summary   the professional identity experiences of lis graduates in non-library roles can be described by the theory of personalizing professionalism   a review of: fraser-arnott, m. (2017, may 17). personalizing professionalism: the professional identity experiences of lis graduates in non-library roles. journal of librarianship and information science. https://doi.org/10.1177/0961000617709062   reviewed by: melissa goertzen information management consultant halifax, nova scotia, canada email: mjg2227@columbia.edu   received: 2 oct. 2017     accepted: 2 feb. 2018      2018 goertzen. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29356       abstract   objective – to produce a theory that captures the professional identity experiences of library and information science (lis) graduates employed in non-library roles.    design – data collection and analysis were conducted using semi-structured interviews and grounded theory methodology.   setting – a variety of industry sectors located in the united states or canada.    subjects – twenty professionals with master's degrees in lis employed in non-library roles.   method – this study used the glaserian grounded theory methodology, which requires constant theoretical sampling and comparison until no new data is found in any coding category. the researcher utilized two types of sampling in this study: snowball or chain referral sampling, and theoretical sampling. these techniques allowed the researcher to build a potential list of participants from a difficult to reach population. study data was collected through semi-structured interviews divided into three sections: 1) participants were asked to describe their career experiences from their decision to attend library school to the present; 2) follow-up questions by the researcher in response to comments made by participants during the first phase; 3) questions listed in an interview guide that examined educational experiences, communities, and how participants identify themselves as professionals.    main results – study results produced the theory of personalizing professionalism, which suggests that each individual possess two identities that interact with one another throughout the course of one's career. the first is an internal appraisal of self that represents who one is as a professional. it is developed as a result of socialization in the profession and an understanding of personal motivations and interests. the second is an externally expressed identity that represents how an individual presents himself or herself to achieve professional goals. this can include self-imposed labels, such as "librarian", or strategies used to find a path within the profession. this process involves reflections and actions aimed at identifying what type of professional to be and steps required to achieve goals.   the results also indicate that interactions with others impact an individual’s internal appraisal of self and externally expressed identity. areas of conflict were identified when commonly held views of how a professional identity should be expressed did not match the identity that an individual developed or displayed to others. when conflicts arose, individuals used a variety of strategies to resolve the discrepancy between internal and external identities: assimilation, attempting to influence or change the perceptions of the group, or withdrawal.    in terms of self-identifying as a librarian, the study found that participants who chose the term as a professional label believed that the work they do in non-library settings was still compatible with their definition of what it meant to be a librarian. participants who identified as librarians some of the time and by their job title at other times did so based on an evaluation of which label would best advance their position with a given audience. finally, participants who chose not to use the label of librarian had never internally associated with the role or job title; these individuals completed a lis program to gain transferable skills or qualify for a wide variety of employment opportunities.   conclusion – the theory of personalizing professionalism provides insight into the development and expression of professional identity experiences when lis graduates work in non-library roles. the results have value to practitioners and educators who market lis programs or develop course content. for instance, in the future greater emphasis could be placed on transferable skill sets that are of value to roles outside of traditional library settings. many participants described potential or actual conflicts when trying to place themselves within the lis community because new ideas of what it means to be a “librarian” were rejected, leading to feelings of exclusion. over time, this could lead to a detrimental loss of innovation and ideas.    commentary   to date, researchers have not examined how the professional identities of lis graduates working in non-library settings are developed or expressed. the study at hand fills this research gap and supports what is known regarding the development of professional identity: it is shaped by values and attitudes held by others of that affiliation, alongside perceptions of context within a professional community (billot, 2010; gibson, 2010). because the study is limited to lis graduates, it provides value to the information management profession in particular.    the researcher introduces the theory of personalizing professionalism, which was developed over the course of the study. this theory states that lis graduates in non-library roles develop an "internal appraisal of self which represents an individual's assessment of who they ‘really’ are as a professional" along with an externally expressed identity, which represents "who that individual presents him or herself to be" (p. 5). the theory brings valuable insight to conversations regarding transferable skill sets, socialization within professional communities, and the career decision-making process.   while the ideas expressed in this research, particularly those related to the theory of personalizing professionalism, no doubt have value to the information management community, there is an unfortunate lack of discussion regarding the methodology. for instance, although the researcher describes the method used to generate a list of potential participants, she does not discuss the recruitment process or the response rate to requests for participation. also, there is no mention of how many participants completed all three phases of the interview process. a detailed discussion on methodology would add value to the paper and allow for reproducibility of the study.    in regard to data collection and analysis, it is not clear how the interviews were conducted (e.g., by phone or in person), the length of each session, and whether the sessions were recorded and transcribed. also, the researcher stated that qualitative data was coded to identify themes, yet there was no discussion regarding coding techniques, software used to conduct analysis, or trends that emerged in the data set. finally, since this is the first study of its kind, it would have been useful to include the interview guide in an appendix. because of the omissions regarding the general methodology of the study, it will be difficult for other researchers to replicate the study or build on the work that has been done. this is truly unfortunate as it would be valuable to explore the topic in greater detail by expanding the sample size or comparing the results of different demographic groups.    despite these limitations, the research introduces an important topic to the information management community. as the job market continues to evolve, it is essential to understand how graduates craft a professional identity and whether or not they continue to self-identify as librarians over the course of their careers. such insights provide all information managers with greater knowledge regarding the strengths and limitations of the professional community and how to foster feelings of membership in the community going forward.    references   billot, j. (2010). the imagined and the real: identifying the tensions for academic identity. higher education research & development, 29(6), 709-721. https://doi.org/10.1080/07294360.2010.487201   gibson, d. m., dollarhide, c. t., & moss, j. m. (2010). professional identity development: a grounded theory of transformational tasks of new counsellors. counsellor education & supervision, 50(1), 21-38. https://doi.org/10.1002/j.1556-6978.2010.tb00106.x microsoft word es 4041 evidence based library and information practice 2008, 3:4 65 evidence based library and information practice evidence summary researchers’ attitudes and behaviour to data archiving policies and practice is typified by low awareness and little planning a review of: perry, carol marie. “archiving of publicly funded research data: a survey of canadian researchers.” government information quarterly 25 (2008): 133-48. reviewed by: gaby haddow lecturer, school of media, culture & creative arts curtin university of technology perth, western australia e-mail: g.haddow@curtin.edu.au received: 23 august 2008 accepted: 25 september 2008 © 2008 haddow. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – to assess researchers’ attitudes and behaviours in relation to archiving research data and to determine researchers’ views about policies relating to data archiving. design – survey. setting – canadian universities. subjects – social sciences and humanities researchers from universities across canada. methods – a questionnaire comprising 15 questions was mailed to 175 researchers randomly sampled from a publicly available list of 5,821 individuals who had received grants and awards from the social sciences and humanities research council of canada (sshrc). from this sample, 75 (43.4%) responded within the five week time-frame stipulated. the questionnaire was constructed using four existing surveys and asked researchers for information about: geographical location, years of research experience, research funding sources, current plans to archive research data, awareness of archiving policies, attitude to mandated research data archiving, effect of mandatory data archiving policies on grant-seeking, attitude to making archived research data accessible, evidence based library and information practice 2008, 3:4 66 and use of research data collected by others. the questionnaire also included space for respondents to make comments. responses to each question were analyzed, followed by a series of cross-analyses to investigate associations between findings. these cross-analyses include: whether attitudes to making data accessible differed according to length of experience, support for a national archive initiative, or agreement with a mandatory policy; and whether plans to archive research data were associated with awareness of policy, agreement with mandatory policy, or funding from government agencies other than sshrc. some cross-analyses were conducted between responses to questions and the comments provided by respondents. most of the questionnaire responses were analyzed as percentages in two categories only. for example, length of service responses were presented as “up to 20 years” and “more than 20 years,” and for responses relating to agreement with mandatory archiving “no” and “undecided” responses were combined into one category and compared with “yes” responses. main results plans to archive research data only 41.3% of the respondent sample had current plans to archive their research data; plans that included filing materials in their office and destroying materials. a small proportion of this group (18.7%) planned to archive their research data in an established repository. respondents who planned to archive their research data were more likely to have received funding from other government agencies (64.7%) than respondents who had not received other government funding (23.5%). examination of the comments made by respondents indicated those who planned to archive their data were more likely to make comments on this issue. knowledge of data archiving policies respondents’ awareness of data archiving policies, particularly that of the sshrc, an organisation for economic co-operation and development (oecd) declaration, and that of their own university, was low. less than a third of the sample (28%) was aware of the sshrc policy and less than 10% of the oecd declaration or their own university’s policy. a sizeable proportion of the sample (65.4%) stated they were “uncertain” whether their university had a data archiving policy. of the respondents who planned to archive their data only 42.4% were aware of the sshrc policy, and a high percentage of respondents with no plans to archive their data (83.3%) were also unaware of this policy. when asked if they agreed with a mandatory policy of data archiving being linked to grant funding the sample group was almost evenly split into agree (33.3%), don’t agree (28%), and uncertain (36%). only 46.9% of the sample who planned to archive their data agreed with the policy. support for data archives there was strong support for a data archive at the home institution by respondents who agreed with mandatory archiving of research data (84%). however, when asked about allowing public access to this data the results were evenly divided across most of the sample group (66 respondents). full open access to data was supported by 44% of this group and 44% thought access should be at the discretion of the depositing researcher. agreement with open access to data was also associated with researchers who agreed with mandatory archiving policy (70.8%) and researchers with over 20 evidence based library and information practice 2008, 3:4 67 years experience (68%). the enforcement of a mandatory archiving policy would not alter most respondents’ future grant-seeking behaviour (86.7%). support for a national data archive was solid across the sample group, with only 17.3% indicating they did not support an initiative of this type. almost all respondents who agreed with public access to research data supported a national data archive (96.5%) and of the respondents who did not support a national archive 90.9% were undecided about data access or wanted restrictions in place. when asked about use of research data collected by others, 48% had used such data in the past and 49.3% had not analyzed research data collected by others. conclusion – while support for a national data archives strategy is strong, researchers’ attitudes and behaviour towards data archiving is less supportive and their plans to archive research data would not, in most cases, meet standards of archiving practice. the notion of open access to research data has less support, with researchers noting barriers relating to confidentiality, ethics, effect on academic career, and cost of archiving. funding of data archiving raises particular concerns and the author recommends that sshrc policy is clarified in this regard. further, the author notes that differences between institutional requirements and national policies relating to research data require attention so that archiving policy objectives are coordinated. the decisions made in moving towards a national data archiving policy will enable canada to contribute to the wider international discussion of standards for research data archives. commentary at a time of increased interest in research data archiving and the issues this topic raises, this is a timely paper reporting on a study of canadian researchers’ attitudes, awareness and behaviour in relation to data archiving. since the publication of this paper, a report commissioned by the research information network in the uk and a survey of researchers in australia (henty et al.) has been released, each arriving at conclusions similar to those discussed here. it is clear from these and other sources, that policies and standards for data archiving are critical to ensure research data is not lost. equally important is support for researchers engaged in data archiving activities and education programs to promote the long term benefits of archiving research data. perry’s findings exemplify why these issues must be addressed and the paper is a useful starting point for those unfamiliar with them. further research to consider would be a study of science researchers’ attitudes, awareness and behaviour in relation to data archiving which would facilitate comparisons across the social science/science spectrum. the study’s sample was randomly drawn from a list of researchers who had received grants from the canadian social sciences and humanities research council and the paper provides a clear description of how the sampling was carried out. also clear are the tables used to present the study’s findings. from these tables, it might be assumed that advanced statistical analyses were not carried out for the data, as most results are shown as percentages. however, the analysis of length of research experience and external funding sources refers to expected values and probability, and the analysis of receipt of corporate funding and support for a home institution repository notes there is no correlation between the questions. these differences instil some uncertainty about data analysis processes carried out across the study. it is also clear from the tables that not all respondents evidence based library and information practice 2008, 3:4 68 answered all questions, but this is not discussed by the author. most confusing though, is the status of the sshrc policy on research data archiving. early in the paper, perry refers to a report recommending the implementation of policies that would require researchers to deposit research data on completion of their research, noting “[m]ovement on this recommendation appears to have stalled”, but later writes “archiving of research data is a mandatory component of its [sshrc] granting program.” of less significance, but vexing nevertheless, was the range of terminology used to describe granting agencies, which included “internal,” “external,” “internal grants from their institution,” “corporate,” “other government,” “other than sshrc,” and “non-sshrc.” publication of the survey instrument as an appendix to the paper may have been useful in clarifying this aspect of the study. while clarification of these points would assist readers, they are not critical to comprehending the results of the study and its value to those making future decision about research data archiving. what is evident from this paper, and the reports mentioned above, is the need for a coordinated and long term commitment to data archiving as well as sound policy and support mechanisms for researchers. library practitioners working in research institutions could play an important role in data archiving activities. in many cases, librarians have gained expertise in research archiving through responsibilities relating to institutional repositories. this experience indicates that librarians have a unique place in supporting researchers to meet institutional and national requirements through individual and group training, membership of research committees, and establishment of large information systems. further, librarians’ expertise should not be viewed as a resource to draw on after policy decisions have been made; rather, librarians’ knowledge and expertise should contribute to the making of research archiving policy and implementation decisions. works cited henty, margaret, belinda weaver, stephanie bradbury, and simon porter. investigating data management practices in australian universities. australian partnership for sustainable repositories, 2008. <http://www.apsr.edu.au/orca/investi gating_data_management.pdf> research information network. to share or not to share: publication and quality assurance of research data outputs. 2008. <http://www.rin.ac.uk/datapublication> evidence summary   nurses need training and policies to address barriers to use of mobile devices and apps for direct patient care in hospital settings   a review of: giles-smith, l., spencer, a., shaw, c., porter, c., & lobchuk, m. (2017). a study of the impact of an educational intervention on nurse attitudes and behaviours toward mobile device and application use in hospital settings. journal of the canadian health libraries association/journal de l'association des bibliothèques de la santé du canada, 38(1), 12-29. doi: 10.5596/c17-003   reviewed by: kelley wadson library information specialist bow valley college calgary, alberta, canada email: kwadson@bowvalleycollege.ca   received: 4 dec. 2017     accepted: 21 feb. 2018      2018 wadson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29385       abstract   objective to describe nurses’ usage of and attitudes toward mobile devices and apps and assess the impact of an educational intervention by hospital librarians and educators   design descriptive, cross-sectional survey, one-group preand post-test, and post-intervention focus group   setting one 251-bed community hospital and one 554-bed tertiary care hospital in winnipeg, canada   subjects 348 inpatient medical and surgical nurses   methods – the study had two phases. in phase i, respondents completed a survey of 21 fixed and open-ended questions offered online or in print to a convenience sample from the community hospital and a random sample of medical and surgical units from the tertiary hospital. the survey collected demographic data and included questions about mobile devices and apps covering current awareness of hospital policy, ownership, internet access, usage patterns, concerns, and attitudes toward their use for direct patient care. it also included information to recruit volunteers for phase ii. in phase ii, participants attended four 30-minute educational sessions facilitated by the researchers. the first session addressed the regional health authority’s policies, personal health information act, and infection control practices. subsequent sessions covered relevance, features, and training exercises for one or more selected apps. participants installed five free or low-cost apps, which were chosen by the librarians and nurse educators, on their mobile devices: medscape, lab tests online, lexicomp, twitter, and evernote. participants were then given a two-month period to use the apps for patient care. afterward, they completed the same survey from phase i and their preand post-intervention responses were matched for comparative analysis. phase ii concluded with a one-hour audio-recorded focus group using ten open-ended questions to gather feedback on the impact of the educational sessions.   main results – 94 nurses completed the phase i survey for a response rate of 27%. although 89 respondents reported owning a mobile device, less than half used them for patient care. just under half the respondents were unsure if they were allowed to use mobile devices at work and a similar number answered that devices were not allowed. two-thirds of respondents were unsure whether any institutional policies existed regarding mobile device use. of the 16 participants that volunteered for phase ii, 14 completed the post-intervention survey and 6 attended the focus group. in comparison to the phase i survey, post-intervention survey responses showed more awareness of institutional policies and increased concern about mobile devices causing distraction. in the phase i survey, just over half of the nurses expressed a desire to use mobile devices in patient care.   four themes emerged from the survey’s qualitative responses in phase i:   (1) policy: nurses were unsure of institutional policy or experienced either disapproval or bans on mobile device use from management; (2) barriers to use, namely cost, potential damage to or loss of devices, infection control, and lack of familiarity with technology; (3) patient perceptions, including generational differences with younger patients seen as more accepting than older patients; and (4) nurse perceptions: most valued access to information but expressed concerns about distraction, undermining of professionalism, and use of technology.   qualitative responses in the phase ii survey and focus group also revealed four themes:   (1) barriers: participants did not cite loss of device or infection control as concerns as in phase i; (2) patient acceptance and non-acceptance: education and familiarity with mobile devices were noted as positive influential factors; (3) information need, accessibility, and convenience: nurses reported needing easy-to-use apps, particularly lexicomp, and appreciated improved access to information; and (4) nurse behaviour and attitude: participants reported more time would be needed for changes to occur in these areas.   conclusion – the study found that although most nurses own mobile devices and express strong interest in using them for patient care, there are significant barriers including lack of clarity about institutional policies and concerns about infection control, risk of damage to personal devices, costs, lack of experience with the technology, distraction, and negative patient perceptions. to address these concerns, the authors recommend that hospital librarians and educators work together to offer training and advocate for improved communication and policies regarding use of mobile devices in hospital settings. moreover, the study affirmed the benefits of using mobile devices and apps to support evidence-based practice, for example by providing access to reliable drug information. the authors conclude that additional research is needed to inform policy and develop strategies that hospital librarians and nurse educators can use to promote the most effective application of mobile technologies for patient care.   commentary   as noted by the authors, the study contributes to the growing literature addressing the use of mobile technologies in the nursing workplace. another recent and larger study of registered nurses in hospitals in the united states drew similar results, including concerns about distraction, safety, and policy as well as generational differences in the perception and use of mobile devices (mcbride & levasseur, 2017). the study thus affirms many of the findings of previous and more recent studies on the topic (mcbride & levasseur, 2017; grabowsky, 2015; planitz, sanderson, kipps, & driver, 2013).   the study was evaluated using koufogiannakis, booth, and brettle’s (2006) reliant instrument. several flaws were found in the areas of study design and educational context.   in terms of study design, the study’s objectives, methods, and instruments are well explained but as the authors recognize in the discussion section, there are significant limitations due to the sample size and method that compromise its validity and introduce the risk of bias. the use of convenience sampling at the community hospital, low response rate in phase i, and small sample size in phase ii mean the results lack external validity and cannot be generalized to the broader population or beyond. this is reinforced by the composition of the phase i respondents; almost half (48%) identified as having less than five years of experience and a significant portion (35%) were aged 24 to 34, which suggests the sample was not representative of the target population. moreover, a calculation of the margin of error was not provided. lastly, while inclusion of the research instruments in the appendices is helpful, the authors do not indicate if these were piloted or validated.   additional flaws are found in the study’s educational context. although the content of the educational sessions is clearly outlined, the study lacks description of the teaching method and learning objectives against which the intervention could have been evaluated. while the objective of the study suggests the intervention aimed to invoke affective and behavioural changes, there is no explanation of how or if it purposively intended to do so through instructional design. the study consequently missed an opportunity to extend its applicability into the realm of nursing education curriculum and nursing competencies maintained by professional associations, which would be relevant for librarians working in higher education.   the study’s conclusions are most useful for librarians and nurse educators and managers working in hospital settings. specifically, its description of the professional concerns, behaviours, and barriers to mobile device use encountered by nurses can be used to direct and inform policy and training interventions in similar contexts. however, given the deficiencies found in the study design, the study’s findings should be corroborated by other sources of published evidence to affirm their external validity and relevance for practice.   references   grabowsky a. (2015). smartphone use to answer clinical questions: a descriptive study of apns. medical reference services quarterly, 34(2), 135-148. http://dx.doi.org/10.1080/02763869.2015.1019320   koufogiannakis, d., booth, a., & brettle, a. (2006). reliant: reader’s guide to the literature on interventions addressing the need for education and training. library & information research, 30(94), 44-51. retrieved from http://www.lirgjournal.org.uk     mcbride, d. l., & levasseur, s. a. (2017). personal communication device use by nurses providing in-patient care: survey of prevalence, patterns, and distraction potential. jmir human factors, 4(2), e10. http://doi.org/10.2196/humanfactors.5110   planitz, b., sanderson, p., kipps, t., & driver, c. (2013). nurses' self-reported smartphone use during clinical care. proceedings of the human factors and ergonomics society annual meeting, 57(1), 738-742. https://doi.org/10.1177/1541931213571161 evidence summary   completion of an online library module improves engineering student performance on information literacy skills tests   a review of: zhang, q., goodman, m., & xie, s. (2015). integrating library instruction into the course management system for a first-year engineering class: an evidence-based study measuring the effectiveness of blended learning on students’ information literacy levels. college & research libraries, 76(7), 934-958. http://dx.doi.org/10.5860/crl.76.7.934   reviewed by: rachel e. scott integrated library systems librarian university libraries university of memphis memphis, tennessee, united states of america email: rescott3@memphis.edu   received: 31 may 2016  accepted: 14 oct. 2016      2016 scott. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to assess the efficacy of an online library module and of blended learning methods on students’ information literacy skills.   design – multi-modal, preand posttests, survey questionnaire, and focus groups.   setting – public research university in london, ontario, canada.   subjects – first-year engineering students.   methods – of 413 students enrolled in engineering science (es) 1050, 252 volunteered to participate in the study. participants were asked to complete the online module, a pretest, a posttest, an online follow-up survey, and to take part in a focus group.   researchers generated a pretest and a posttest, each comprised of 15 questions: multiple choice, true or false, and matching questions which tested students’ general and engineering-specific information literacy skills. the pretest and posttest had different, but similarly challenging, questions to ensure that students involved in the study would not have an advantage over those who had opted out. while all components of the study were voluntary, the posttest was a graded course assignment.  in-person tutorials were offered on 4 occasions, with only 15 students participating. both tutorial and module content were designed to cover all questions and competencies tested in the pretest and the posttest, including boolean operators, peer review, identifying plagiarism, engineering standards, engineering handbooks, search strategies, patents, article citations, identifying reliable sources, and how to read journal articles.   the posttest survey was delivered in the cms immediately after the posttest was completed. it measured self-reported student behaviours and preferences concerning the online modules. two focus groups were convened after all posttest surveys were completed to gather qualitative data about student preferences.   main results – of the 252 volunteers, 239 students (57.9% of enrolled students) completed both the pretest and the posttest, 89 filled out the follow-up survey, and 7 students participated in a focus group. students used the online module content differently; accordingly those numbers were not reported. researchers compared pretest and posttest scores to find that the posttest scores were significantly higher than the pretest scores (p < 0.001). of 239 pretest and posttest pairs evaluated, the mean pretest score was 10.456 and the mean posttest score was 13.843. a t-test survey and focus group data evaluated student perceptions of the module. students reported a slight preference for online instruction.   conclusion – after completing an online library module, students’ performance on information literacy skills tests improved from the pretest to the posttest. focus group and survey data indicate a slight student preference for online tutorials over in-person instruction. although intended as a blended approach to library instruction, the voluntary in-person instruction was not well attended and has subsequently been changed to mandatory in-class instruction. the authors recommend further research to evaluate how the medium and format of instruction impacts student learning outcomes.   commentary   instruction librarians continuously evaluate how to efficiently and effectively deliver instruction to various user groups. a body of literature supports the convenience and efficacy of embedding online tutorials in course management systems (cms), both to save the time of the librarian and to meaningfully contextualize and teach information literacy skills (mery et al., 2012; henrich & attebury, 2012). these studies have found that embedded online tutorials have a positive impact on student learning. here, by focusing on first-year engineering students, the researchers targeted a unique population.     the reliant (reader’s guide to the literature on interventions addressing the need for education and training) instrument provides librarians the means to critically assess educational and training interventions in library and information science (koufogiannakis, booth, & brettle, 2006). this instrument was employed by the reviewer to evaluate the design, educational context, results, and relevance of the study at hand.   the authors do a good job of answering the first research question. however, the second research question is not adequately answered due to the absence of well-attended, in-person instruction sessions, and this presents a substantial design flaw. with only approximately 15 students receiving in-person instruction, this study does not assess blended learning methods. some other elements of the methodology are problematic as well. no demographic information was collected and it is unknown if subjects were representative of the engineering department or the university’s undergraduate population. students were asked to complete a written consent form and a pretest during class, but posttests and surveys were conducted online and outside of class. the authors do not state whether the survey and the pretest/posttest content had been piloted. had the content been piloted, pretest questions regarding the appearance of print publications and the difficulty level may have been reconsidered.   the educational context and instructional topics are clearly defined and described. however the teaching methods employed and the mode of delivery are clouded by the authors’ claim of measuring the effectiveness of “blended learning” methods. additionally, the study neglects to offer any detail regarding how the module was integrated into the cms and indicates that students did not consistently view all tutorial videos. in order to ensure equal instructional contact time, researchers might instead have worked with content creators to ensure that all tutorial content was mandatory.                 the results of the study are well-explained with the data clearly presented and analyzed using spss. the resulting improvement in student information literacy levels cannot be attributed to blended learning methods. the authors acknowledge that they must incorporate more in-person instruction to create a truly blended approach. they describe how in subsequent semesters they have “flipped the classroom” and require students to complete modules before in-person classroom sessions.   this article’s positive contribution to the literature lies in its evidence for collaborating with faculty, it, and other stakeholders to create multimedia online content that can be conveniently accessed and integrated into a cms. instead of consulting this article for best practices on blended learning, librarians would do well to consult the practical sections on collaborating to develop and embed effective library instruction.   references   henrich, k. j., & attebury, r. i. (2012). using blackboard to assess course-specific asynchronous library instruction. internet reference services quarterly, 17(3-4), 167-179. http://dx.doi.org/10.1080/10875301.2013.772930   koufogiannakis, d., booth, a., & brettle, a. (2006). reliant: reader’s guide to the literature on interventions addressing the need for education and training. library & information research, 30(94), 44-51. retrieved from http://www.lirgjournal.org.uk/lir/ojs/index.php/lir/article/view/271/318   mery, y., newby, j., & peng, k. (2012). why one-shot information literacy sessions are not the future of instruction: a case for online credit courses. college & research libraries, 73(4), 366–377. retrieved from http://crl.acrl.org/content/73/4/366     evidence summary   title, description, and subject are the most important metadata fields for keyword discoverability   a review of: yang, l. (2016). metadata effectiveness in internet discovery: an analysis of digital collection metadata elements and internet search engine keywords. college & research libraries, 77(1), 7-19. http://doi.org/10.5860/crl.77.1.7   reviewed by: laura costello head of research & emerging technologies stony brook university libraries stony brook, new york, united states of america email: laura.costello@stonybrook.edu   received: 1 june 2016    accepted: 15 july 2016      2016 costello. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine which metadata elements best facilitate discovery of digital collections.   design – case study.   setting – a public research university serving over 32,000 graduate and undergraduate students in the southwestern united states of america.   subjects – a sample of 22,559 keyword searches leading to the institution’s digital repository between august 1, 2013, and july 31, 2014.    methods – the author used google analytics to analyze 73,341 visits to the institution’s digital repository. he determined that 22,559 of these visits were due to keyword searches. using random integer generator, the author identified a random sample of 378 keyword searches. the author then matched the keywords with the dublin core and vra core metadata elements on the landing page in the digital repository to determine which metadata field had drawn the keyword searcher to that particular page. many of these keywords matched to more than one metadata field, so the author also analyzed the metadata elements that generated unique keyword hits and those fields that were frequently matched together.   main results – title was the most matched metadata field with 279 matched keywords from searches. description and subject were also significant fields with 208 and 79 matches respectively. slightly more than half of the results, 195 keywords, matched the institutional repository in one field only. both title and description had significant match rates both independently and in conjunction with other elements, but subject keywords were the sole match in only three of the sampled cases.   conclusion – the dublin core elements of title, description, and subject were the most frequently matched fields in keyword searches. academic librarians should focus on these elements when creating records in digital repositories to optimize traffic to their site from search engines.    commentary   this study examines common digital repository metadata fields by looking critically at successful keyword searches and provides context for the way these records are discovered organically through search engine traffic. though both of these topics have been explored independently, the latter largely outside of library literature, the study represents a unique illumination of library metadata through the lens of general searching. a few studies have examined the frequency of dublin core elements on websites (phelps, 2012; windnagel, 2014), though this study is unique in its consideration of these elements through external search engines. though projects like linked open data and current metadata schema development deeply consider the impact of digital searching, the results of this study could potentially lead to search-oriented workflow optimization in existing collections. the study’s focus on keywords for searching is particularly helpful for libraries struggling to make in-house digital collections more visible in discovery layers and through organic searches from outside the library.   the author chose an appropriate sample size for a 95% confidence level and a ±5% margin of error, and samples were selected randomly over the course of one year of data. the sample selected seems likely to be representative of the types of searches that are regularly performed by users when accessing the digital repository. the major limitation of this study is that it examines only one digital repository. more research is needed to determine whether the results are generalizable to other repositories with different collections.   broadening this type of research to other collections is particularly important for studying search because much of the strategy and success of search practice is unique to the file type and format type of the material being searched. though this study focused on a large digital repository of 29,705 items and included many of the common file formats and types found in digital repositories, such as digitized images and text, dissertations, and research papers, there is much to gain from testing the results against other collections.    digital libraries have struggled with crafting metadata that accommodates and supports searches conducted within library catalogues and resources while providing enough information for non-library search engines. this study highlights the essential points of metadata creation from the perspective of outside searching but has the potential to reflect back on the way libraries internally evaluate appropriate and essential metadata for digital materials. as library searching becomes more keyword-based, it will be important to continue to study the way keyword searches interact with digital metadata.   references   phelps, t. e. (2012). an evaluation of metadata and dublin core use in web-based resources. libri: international journal of libraries & information services, 62(4), 326-335. http://doi.org/10.1515/libri-2012-0025   windnagel, a. (2014). the usage of simple dublin core metadata in digital math and science repositories. journal of library metadata, 14(2), 77-102. http://doi.org/10.1080/19386389.2014.909677 evidence summary   libraries may teach some skills through mobile application games   a review of: kaneko, k., saito, y., nohara, y., kudo, e., & yamada, m. (2018). does physical activity enhance learning performance? learning effectiveness of game-based experiential learning for university library instruction. journal of academic librarianship, 44(5), 569-581. https://doi.org/10.1016/j.acalib.2018.06.002     reviewed by: robin e. miller assessment and instruction librarian mcintyre library university of wisconsin eau claire eau claire, wisconsin, united states of america email: millerob@uwec.edu   received: 29 may 2019                                                                  accepted:  14 july 2019      2019 miller. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29587     abstract   objective – to understand the impact of a mobile application game for library knowledge acquisition, task performance, and the process of learning.   design – the main experiment included a pretest, learning experience, post-test, and a questionnaire. one month later, a post-experiment was conducted, including a test of “declarative knowledge” and a behavioural test.   setting – kyushu university in fukuoka, japan.   subjects – 36 first-year undergraduate students, of which 25 were female and 11 were male. students were divided into experimental and control groups. 32 students completed the study.   methods – in the main experiment, students responded to the same 20 question pre-test on library use, and then both groups participated in learning experiences designed to convey knowledge about using the library. the control group’s learning setting was a web-based tutorial about the library. the experimental group’s learning setting was “library adventures: unveil the hidden mysteries!” a “game-based learning environment” developed by the researchers (kaneko, saito, nohara, kudo, & yamada, 2015, p. 404), which required students to complete activities by physically moving through the library. for both groups, learning content related to local library procedures, like hours, arrangement of collections, and methods for locating books and articles. the game collected data that the authors analyzed using statistical methods in an attempt to validate quizzes that were embedded in the game. after finishing the learning experience, all students completed the 20-question post-test, and then responded to the instructional materials motivation survey (imms), a questionnaire designed to gauge learning motivation using the attention, relevance, confidence, and satisfaction (arcs) model. one month following the main experiment, all students took a test of declarative knowledge and completed a skills test.   main results – experimental and control group students gained about the same level of declarative knowledge. all students lost some knowledge in the one-month gap between the main and post-experiment. students who had learned through library adventure were able to borrow a journal and locate a newspaper article more effectively than the control group. in contrast, tutorial users made study room reservations more quickly than the experimental group. more significantly, the imms instrument demonstrated that game-based learners scored higher in attention, relevance, and satisfaction than tutorial-based learners. experimental and control group participants demonstrated the same level of confidence.   conclusion – while inconclusive about the effectiveness of games versus tutorials for acquisition and retention of knowledge, the authors concluded that game-based instructional content may foster greater learner engagement, aiding some students in understanding how to use the library in a manner superior to web-based tutorials. librarians and instructional designers developing game-based learning experiences for novice library users may find this research informative.   commentary   the authors describe a multi-part experiment intended to demonstrate the extent of student learning from a game versus a tutorial. this study is notable in the library and information science literature because it compares the effectiveness of a game and a tutorial, rather than simply reporting the outcome of a single intervention. furthermore, the library adventure game requires students to move around the library and complete physical tasks in order to play. as librarians implement instructional design principles in daily practice, this study is a practical example of how libraries can engage students in active learning even when instructional content is delivered digitally.   the critical appraisal tool developed by glynn (2006) was used to evaluate this study. while 32 students completed the study, the authors do not describe informed consent procedures or recruitment methods, and, although participants were first-year students, the authors do not describe additional inclusion/exclusion criteria.  additionally, the research methodology and results are not always clearly stated, which may confuse some readers.   for librarians exploring “gamification,” this research has potential to inform the efficacy of video game-based instruction. the authors applied a modified version of the instructional materials motivation survey (keller, 2009), a validated tool that examines four elements of learning motivation: attention, relevance, confidence, and satisfaction (arcs). applying imms to library learning offers library practitioners method of formative assessment by focusing on student motivation to learn. while this concept is likely familiar to practitioners in the instructional design domain, instruction librarians may value gauging learner motivation through the process of developing and refining educational content.   a substantial portion of the article is devoted to analysis of each game player’s performance in library adventure. using this data, the authors identified “stages” of the game that were easy or difficult. instructional design practitioners may value data like this in order to understand where students failed or succeeded. unfortunately, the authors did not report comparable data for the web-based tutorial, so the two learning modes cannot be compared on this point. in addition, the authors do not relate this analysis to the outcomes of the pre-/post-tests and imms instrument. the authors have an opportunity to expand on these points to extend their research in future publications.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154   kaneko, k., saito, y., nohara, y., kudo, e., & yamada, m. (2015). a game-based learning environment using the arcs model at a university library. in t. matsuo, k. hashimoto, t. mine, & s. hirokawa (eds.), the proceedings of iiai 4th international congress on advanced applied informatics (pp. 403–408). okayama, japan: institute of electrical and electronics engineers. https://doi.org/10.1109/iiai-aai.2015.285   keller, j. m. (2009). motivational design for learning and performance: the arcs model approach. new york, ny: springer science & business media.     microsoft word comm_morrison.doc evidence based library and information practice 2006, 1:2  46 evidence based library and information practice     commentary    evidence based librarianship and open access      heather morrison  project coordinator, british columbia electronic library network  burnaby, british columbia, canada  e‐mail: heatherm@eln.bc.ca      © 2006 morrison. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      introduction    booth defines evidence based librarianship  as “an approach to information science that  promotes the collection, interpretation, and  integration of valid, important and  applicable user‐reported, librarian‐observed,  and research‐derived evidence. the best  available evidence, moderated by user  needs and preferences, is applied to  improve the quality of professional  judgments” (6).    evidence based practice, in librarianship or  any other profession, depends upon access –  access to the evidence gathered by others  (published or unpublished literature), and  access to opportunities to publish the results  of one’s own research.  there are several  aspects that make access an important issue  for library and information science (lis)  literature:    • inequities in access that were the  norm until fairly recently;    • expansion of access  made possible  by the internet; and     • optimum access  ‐ represented by  open access.    suber draws on the three major defining  statements of open access (the budapest  open access initiative, the bethesda  statement on open access publishing, and  the berlin declaration on open access to  knowledge in the sciences and humanities),  in his definition of open access (oa)  literature as “digital, online, free of charge,  and free of most copyright and licensing  restrictions.” the focus of the open access  movement is the literature that scholars  have traditionally given away without  expectation of reward, particularly in peer‐ reviewed journal literature.    there are two major approaches to open  access:  open access publishing, also called  the ‘gold road’, and open access archiving,  known as the “green road”. whatever road  one chooses to travel, lis literature is well  on its way.  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:2  47 access to the lis literature     print collections of lis literature have never  resulted in equitable access.  the largest and  best collections of lis literature are found in  the largest research libraries, especially  those attached to universities with lis  programs.  the smallest libraries, by  contrast, might be fortunate to have a  subscription to a single lis journal.  it  follows that access to the lis literature is far  from equitable, with the best access enjoyed  by librarians at the libraries with the largest  collections.    even at the largest research libraries, access  has never been perfect, or equitable, despite  the best efforts of the libraries involved.   when materials are available only in print,  they are not always there when we would  like to access them.  they may be out at the  bindery, lost, sitting in the in‐basket of a  colleague, misshelved, or sitting in a stack of  materials to be shelved.      even when the item is right there on the  shelves, it is not equally accessible to  everyone.  many universities have always  welcomed the public as walk‐in users, and  have offered generous access to colleagues  through interlibrary loans programs.  walk‐ in use of print materials offers a means to  keep up with professional reading, which is  a great deal more accessible for those who  live and work close to the large university  libraries than for   those in more remote  locations.   even amongst the large  university library’s own staff, access is not  quite equitable.  the librarian who works in  the branch that houses the lis literature can  easily slip to the stacks to keep up with  reading during a lunch hour.  this is not  true for colleagues in the branch library  across campus, never mind the branch  library in another community.    access has also varied depending on the  type, size, and location of the library.   academic libraries are more likely to  subscribe to more lis journals than public,  special or school libraries.  large libraries  are more likely to be able to support lis  subscriptions than are small libraries.   wealthier libraries are more likely to be able  to afford lis journals then poor libraries.  rural libraries and libraries in developing  countries, tend to be smaller, less wealthy,  less likely to be situated near large research  libraries, and less likely to have financial  support for lis journals in their collections.    libraries vary in their ability to support  interlibrary loans programs; some libraries  are able to offer patrons, including librarians,  free interlibrary loans, while others are  obligated to charge patrons for this service.   even where interlibrary loans are free, my  experience is that librarians are very  sensitive to the costs of providing this  service, and hence are somewhat reluctant  to make full use of this service to meet their  own needs.      with a move toward online access, the  electronic medium has increased equity in  access.  general, aggregated journal  packages such as those offered by gale  group and ebsco, include a selection of  lis journals.  for example, as of march 25,  2006, there are 15 lis journals in gale’s  expanded academic index, many of which are  full‐text, and a total of 84 lis journals  indexed in ebsco’s academic search elite, of  which 55 are full‐text.  for many smaller  libraries, subscribing to one of these  packages means a huge increase in access to  lis journals.    this increase in electronic access, however,  does not equal open access.  many of the  journals in these packages come with  embargoes or delayed access, and not all  libraries can afford these packages.     one example from my own experience may  help to illustrate just how much difference  evidence based library and information practice 2006, 1:2  48 open access makes.  i am a project  coordinator for the british columbia  electronic library network (bc eln), a  consortium of publicly funded post‐ secondary libraries across british columbia  and the yukon.  bc eln partner libraries  span a wide range of sizes – from the one‐ person library to the university of british  columbia, types – from vocational /  technical colleges to research libraries, and  locations, from vancouver to the arctic.     in the summer of 2005, i was writing an  article for bc eln connect, the quarterly  communiqué of the bc eln, explaining a  new approach to communications at bc  eln, involving qualitative research   (stories).  an article by jinx watson,  published in the journal of education for  library and information science, explained  this approach beautifully.  a search of eln  serials, the province’s union serials list,  showed that 2 of bc eln’s partner libraries  owned this title.  i contacted the author and  asked if she would consider depositing her  article in e‐lis, the open archive for  library and information science, which she  did – with the result that the bc eln  connect article included a citation to an  article equally accessible to everyone.      in contrast with the embargoed and limited‐ to‐subscribers access of traditional journals  and aggregated packages, the 59 lis  journals listed in the directory of open  access journals (doaj)  http://www.doaj.org/ljbs?cpid=129 (as of  april 10, 2006), are peer‐reviewed and fully  open access, with articles immediately  available to everyone, as soon as they are  published.       a quick glance at the titles available  provides an indication of the quality and  relevance of what is available in doaj.   along with traditional journals such as  bulletin of the american society for  information science and technology, many  of the new open access journal titles are  focused on emerging areas of librarianship,  such as d‐lib magazine, first monday, and,  of course, evidence based library and  information practice.    the doaj titles not only provide increased  access, but they also present a more  inclusive, global collection, than can be  found in our aggregated collections, which  often reflect an english‐language and  western civilization bias.  while many of  the doaj titles are in english, some are in  spanish, french, portuguese, german, and  italian.  browsing through doaj not only  provides us with a great deal of information  about lis, but also brings a greater diversity  of viewpoints than is available through our  aggregated packages.    this diversity of perspectives and languages  is even more striking in e‐lis, the open  archive for library and information science.   e‐lis supports documents in more than 20  languages and is coordinated by an all‐ volunteer editorial board composed of  editors from more than 50 countries.    open access archives make it possible to  share not only peer‐reviewed literature  published in subscription journals, but also  forms of literature that, until recently, were  relatively inaccessible, often even at the very  largest research libraries.      technical reports, theses, and other  publications are all forms of literature once  readily accessible only to the few, and are  now often widely and freely available.  for  example, the proceedings of the 68th annual  meeting of the american society for  information science and technology (asist)  can be found in e‐lis.   the council of  prairie and pacific university libraries  (coppul) is using an institutional  repository approach to share animated  tutorials.    http://www.doaj.org/ljbs?cpid=129 evidence based library and information practice 2006, 1:2  49 open access journals and archives are  creating enhanced opportunities for  librarians to share their own work and  research, enabling more to become  participants – not merely readers ‐ of  research.      it is important to note that, while open  access and electronic access in general mean  very greatly expanded access to research,  this is dependent on internet access, which  is still an issue in many parts of the world,  including regions of canada.  according to  the telus / british columbia partnership site,  89 % of british columbia’s citizens live in  communities that are connected to high‐ speed internet, yet citizens in 151  communities in british columbia lack any  internet access.  of these remote  communities, 76 are first nations  communities. many of the communities  without high‐speed internet access also lack  basic telephone service.    conclusion    open access and evidence‐based  librarianship are a natural combination.  we  can enhance our practice by taking  advantage of research that is already  available to us, whether it be found in the  lis oa journals in the directory of open  access journals, or the self‐archived articles  in open access institutional archives or  disciplinary repositories.  let us share what  we have, and as we practice evidence based  librarianship and expand our research  literature, let us openly share all of what we  create in the future as well.    works cited    booth, andrew.  “exceeding expectations:  achieving professional excellence by  getting research into practice”  lianza 2000 conference, 15‐18 oct.  2000, christchurch, new zealand. 26  mar. 2006  <http://www.shef.ac.uk/scharr/eblib/  exceed.pdf>    coppul, council of prairie and pacific  university libraries. coppul  animated tutorial sharing project.  brandon university, manitoba, canada.  26 mar. 2006         <http://www.brandonu.ca/library/c  oppul/>.    doaj, directory of open access journals.   social sciences ‐ library and  information science titles ‐ 59 titles. 26  mar. 2006  <http://www.doaj.org/ljbs?cpid=129>.    ebsco. title lists ‐ academic search elite.  26 mar. 2006 <http://www.epnet.com/>.    “welcome to e‐lis, the open archive for   library and information science.”   e‐lis, e‐prints in library and  information science.26 mar. 2006  <http://eprints.rclis.org/>.    morrison, heather. “bc eln  communications framework 2005.”  eln connect 4.2   (aug. 2005). 26 mar. 2006  <http://www.eln.bc.ca/view.php?id=120 4>.    suber, peter.  “open access overview:    focusing on open access to peer  reviewed research articles and their  preprints.”  10 mar. 2006  <http://www.earlham.edu/~peters/os/ov erview.htm>.    telus british columbia.  connecting b.c.  communities. 26 mar. 2006  <http://about.telus.com/digitaldivide/in dex.html>.    thomson gale. database title lists. 26 mar.  2006 <http://www.gale.com/title_lists/>.    http://www.shef.ac.uk/scharr/eblib/ http://www.brandonu.ca/library/c http://www.doaj.org/ljbs?cpid=129 http://www.epnet.com/ http://eprints.rclis.org/ http://www.eln.bc.ca/view.php?id=120 http://www.earlham.edu/~peters/os/ov http://about.telus.com/digitaldivide/in http://www.gale.com/title_lists/ evidence based library and information practice 2006, 1:2  50 watson, jinx stapleton.  “making sense of  the stories of experience:  methodology  for research and teaching.” journal of  education for library and information  science 2001; 42.2 (2001): 137‐48. e‐lis:  26 mar. 2006  <http://eprints.rclis.org/archive/0000457 8/>.                http://eprints.rclis.org/archive/0000457 evidence summary   the role of reading classic fiction in book groups for people with dementia is better understood through use of a qualitative feasibility study   a review of: rimkeit, b.s. and claridge, g. (2017). peer reviewed: literary alzheimer’s, a qualitative feasibility study of dementia-friendly book groups. new zealand library & information management journal, 56(2), 14-22. https://figshare.com/articles/literary_alzheimer_s_a_qualitative_feasibility_study_of_dementia-friendly_book_groups/5715052/1   reviewed by: joanne muellenbach director, jay sexter library touro university nevada 874 american pacific drive henderson, nevada, united states of america e-mail: joanne.muellenbach@tun.touro.edu   received: 2 feb. 2018     accepted: 3 may 2018      2018 muellenbach. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29417       abstract   objective – to explore how people living with dementia experience reading classic fiction in book groups and what benefits this intervention provides.    design – qualitative feasibility study.   setting – day centre within a care home in the north island of new zealand.   subjects – eight participants with a medical diagnosis of dementia – four community dwellers who attend day centers, and four residents of a secure dementia unit in a care home.   methods – investigators used surveys, focus groups, and interpretative phenomenological analysis (ipa), for ideographic analysis of the data.    main results – following analysis of the focus book group data, three superordinate, with related subordinate, themes were found: 1) the participant as a lively reader. the participants shared childhood memories of reading and when they became adults, how they encouraged reading within the household and with their own children. subordinate themes included: recall, liveliness of discussion, and interest in reading and book clubs; 2) the participant as guardian of the voice of dickens. participants believed that, when the language is simplified, the beauty and rich imagery of dickens is lost. subordinate themes included: oversimplifying “loses the voice of dickens”, familiarity, and continued play on words; and 3) the participant as a discerning book reviewer. the participants offered a number of ‘dementia-friendly’ suggestions, including the use of memory aids and simplifying text. subordinate themes were expressed as four recommendations: use cast of characters; illustrations pick up the energy of the story, but balance quantity with risk of being childish; the physical quality of the text and paper; and chunk quantity of text while keeping the style of the original author. the choice of using classic fiction that was already well known was validated by the participants, who had some preconceptions about ebenezer scrooge, and described him by using epithets such as mean, an old bastard, and ugly. the participants found the investigators’ adapted version to be oversimplified, as short excerpts of the original dickens seemed to evoke emotional and aesthetic responses of appreciation. therefore, when creating adaptations, it is important to preserve the beauty of the original writing as much as possible.   conclusion – this qualitative feasibility study has provided a better understanding of how people living with dementia experience classic fiction in shared book groups. for individuals with alzheimer’s disease, language skills may be well-preserved until later in the disease course. for example, the focus group participants demonstrated an appreciation and command of language, as well as enthusiasm and excitement in the sharing of the original dickens with others. they suggested the use of memory aids, such as including a cast of characters, and repeating the referent newly on each page. participants also suggested that the adapted version be shortened, to use a large font, and to include plenty of pictures. the choice of using classic fiction was validated by the participants, as they found these tales comforting and familiar, particularly when they included such colorful characters as ebenezer scrooge. finally, people living with dementia should be encouraged to enjoy books for the same reason other adults love to read – primarily for the creative process. classic fiction may be adapted to enhance readability, but the adaptation must be done in a thoughtful manner. while memory deficits occur in alzheimer’s disease, an appreciation of complex language may be preserved until the later disease stages.    commentary   this study highlights how people living with dementia experience classic fiction in shared book groups. the findings build upon the understanding of experiences identified by billington (2013), who found that engagement in reading-group activity appeared to produce a significant reduction in dementia symptom severity. her findings suggested three themes for effective reading group engagement: its components; enjoyment, and a renewed sense of personal identity; and enhanced listening and memory. billington suggested that a randomized controlled trial (rct), along with a qualitative component, could further research in this area, by addressing issues of individual differences amongst participants. in fact, the investigators for this study have published five adaptations, based on classic fiction, drama, and poetry, for a pilot rct and linguistic analysis of dementia-friendly book groups, in preparation for a 2018 multicenter, international rct. the findings of a systematic review by latchem (2014) on the role of reading on the health and well-being of people with neurological conditions concluded that, while research in this area is limited, a number of studies reported reduced agitation, and increased engagement, in people with dementia. latchem also suggested that rcts, along with qualitative research, are needed to provide an in-depth and rich understanding of the effect of reading interventions for people living with dementia.    this study had specific shortcomings – it explored the experiences of just eight participants with a medical diagnosis of dementia, in one location, the north island of new zealand, and at one particular point in time. o’cathain (2015) published a guide for researchers who are assessing qualitative research within feasibility studies for rcts. she recommended that further qualitative, and quantitative, research on how dementia is experienced by this population, in other health care settings and countries, involving a larger, more diverse, sample, and over a longer period of time, would strengthen the key findings and applicability of the study. she also stressed the importance of having an experienced qualitative researcher as a part of the feasibility study design team.   the findings of this research study have the potential to assist programs in psychology, geriatrics, linguistics, and library and information science to develop curricula on research study design. study results may also assist associations and organizations involved in educating, preparing, and supporting health professionals at every career stage to develop the actions, mindsets, and skills needed to facilitate and advance knowledge about how people with dementia experience reading classic fiction. in addition, this study could influence writers to create new adaptations of classic fiction, and market it to day centres and care homes. health sciences or public librarians may be inspired to start a book group for people with dementia. the findings may also serve to inform the design preferences in printed education materials for people with dementia.   references   billington, j., carroll, j., davis, p., healey, c., & kinderman, p. (2013). a literature-based intervention for older people living with dementia. perspectives in public health, 133(3), 165-173. doi: 10.1177/1757913912470052   latchem, j., & greenhalgh, j. (2014). the role of reading on the health and well-being of people with neurological conditions: a systematic review. aging & mental health, 18(6), 731-744, doi: 10.1080/13607863.2013.875125   o’cathain, a., hoddinott, p., lewin, s., thomas, k., young, b., adamson, j., jansen, y., mills, n., moore, g., and donovan, j. (2015). maximizing the impact of qualitative research in feasibility studies for randomized controlled trials: guidance for researchers. pilot and feasibility studies, 1(32). https://doe.org/10.1186/s40814-015-0026-y.     microsoft word ed_5132 1 evidence based library and information practice editorial looking forward denise koufogiannakis editor-in-chief collections & acquisitions coordinator, university of alberta libraries edmonton, alberta, canada e-mail: denise.koufogiannakis@ualberta.ca © 2009 koufogiannakis. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. this issue marks the start of eblip’s fourth year of publication. i am excited to begin my new role as editor-in-chief at a time when the journal is functioning well and the editorial team has a sufficient amount of experience to confidently manage the publishing process, but is young enough to be open to new ideas and make changes without too much angst. mostly, i hope that the journal can quickly respond to the needs of our readers, authors, and everyone who works to make the journal a success. i am always pleasantly surprised when we put out calls for volunteer help and are overwhelmed with the number of people who are interested in contributing to the journal. i definitely believe that the more people who are involved, the better the end result will be, since more ideas and points of view only strengthen the quality of our endeavour. to kick off this first issue of 2009, we debut a new section called using evidence in practice. this section features brief reports of lis practitioners' use of evidence to assist with decision making. we hope that this section will attract all the practitioners out there who are trying to incorporate evidence into their daily practice, and want to share some of their successes or struggles with the evidence-based process. the section follows a structured format, so that readers can always expect to be provided with information on a project that includes an overview of the setting, problem, evidence that was used, how change was implemented, the outcome of those changes, and the author’s reflections on trying to implement change by using evidence. i hope these brief reports of “real-life” stories inspire and provide ideas for our readers. our first article in this section looks at one library’s exploration of the use of folksonomy tags and what the evidence they gathered might mean for the cataloguing of their electronic dissertations and theses collection. i am pleased to announce that we have a new member joining the editorial team. scott walter is taking on the role of 2 associate editor responsible for classics and the new using evidence in practice section. scott is the associate university librarian for services and associate dean of libraries at the university of illinois at urbanachampaign. he brings a wealth of experience to our team, having served on the boards of several lis journals. he is actively involved in professional activities, notably with the association of college & research libraries, and he has an extensive record of publishing his own research. scott also has more degrees than i can count, so i know that he will be a vast source of knowledge upon which the entire team will draw. welcome, scott! most of the members of the editorial team will be attending the eblip5 conference in stockholm, sweden this june/july, and are planning a journal user group session during the conference. in addition to providing an update about the journal, our goal is to solicit feedback from journal readers, authors, and anyone who cares about the future of our publication. we are looking for ideas about how to make the journal better, attract more article submissions, and determine what areas may be lacking that are of interest to our readership. if you can attend the conference, we would love to speak with you and meet you in person. if you’re not lucky enough to make it to the conference, please feel free to send me or any of the associate editors an email with your ideas! research in practice   networks: making connections and sharing information    virginia wilson director, centre for evidence based library and information practice (c-eblip) university library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   received: 29 aug. 2016 accepted: 1 sept. 2016      2016 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   this latest installment of the research in practice column is an exploration of networks: groups or systems of interconnected people or things (network, 2016). i would like to admit a particular interest in networks, as the centre for evidence based library and information practice (c-eblip) has launched the c-eblip research network – an international affiliation of institutions that support librarians as researchers or that are interested in evidence based library and information practice. since the end of april 2016, 17 institutional members have joined the c-eblip research network. these members include institutions and groups from canada, the united kingdom, australia, and hong kong. the c-eblip research network, though an international affiliation of institutions, is particularly for librarians who are interested in research, are conducting research, or who are interested in evidence based library and information practice. the advantage of an institutional membership is that it becomes a public declaration that the institution supports its librarians in these endeavors. however, while i do have a vested interest in the success of the c-eblip research network, i am mindful that networks can come in all shapes and sizes to serve all types of functions.   networks come in many forms: computer networks, telecommunication networks, business networks, and social networks, to name a few. there are laws that pertain to the functionality and impact of networks. for example, there’s metcalfe’s law: “a network's impact is the square of the number of nodes in the network” (metcalfe’s law, 2016). basically, if the number of nodes in the network is 10, the impact is 100 (10 x 10). the more nodes you add to a network, the greater the number of possible cross-connections. and then there’s reed’s law: “the utility of large networks, particularly social networks, can scale exponentially with the size of the network” (reed’s law, 2016). the bigger it gets, the more useful it is. there’s a mathematical formula involved in that one, too. it seems like a reasonable assumption: the more members you have in the network, the more potential there is to make connections. networks all have one thing in common: information sharing. whether we’re talking computer networks, your central nervous system, or television station affiliates, networks are designed to facilitate the sharing of information by making connections.   setting up groups is not unusual. if we think about library associations, special interest groups, support groups, or any other groupings of people around a central theme or themes, we see that they are pretty ubiquitous. in terms of research groups for librarians, i know of two off the top of my head: the libraries nova scotia research support group (lns rsg) in canada and lark library applied research kollektive in australia. the goal of lns rsg is “to increase the amount and quality of library research in nova scotia” (n.d.). rsg members “can learn about research, communicate research projects to each other, pose questions to the group, practice presentations, and create constructive feedback drawing on the strengths of the group to raise the level of research for all” (n.d.). this group aspires to share information by making connections. lark “fosters evidence-based practice and applied research in library and information studies” (2016). lark provides and promotes research support events, is actively involved in social media, and is a member of the c-eblip research network.   okay, so we set up networks to share information by making connections. that’s pretty simple, but it makes for a short column if that’s all there is. what is the value of this? what are the potential impacts that can be made by making connections and sharing information? and why is such a structure needed? so many questions, so i’ll start with the last one first: why is a formal structure important? if you google “networking” you’ll see plenty of tips and advice for how to do this activity. networking is getting out there, putting yourself forward, and making connections, and no one seems to doubt the benefits you can derive from such an activity. but let’s face it: networking is hard! and for the more introverted among us, it’s really hard. and we’re busy. our work involves many components, and while we know that we could benefit from making connections and sharing information, sometimes it’s all we can do to get the daily tasks completed. if left to its own devices, networking drops down the priority list. so, something that is more formal in structure that focuses on networking, that brings that activity to the forefront, and that not only provides opportunities but also places the opportunities directly in front of you almost makes networking a “can’t fail” proposition.   if the purpose of networks is to share information by making connections, what can come of that activity? i believe that several powerful things can happen once people start to make connections and share information:   collaboration   early career librarians, solo librarians, or librarians working in highly specialized areas can particularly benefit from collaborative research relationships. additionally, a particular project may benefit from having more than one co-investigator. sharing a research project, co-presenting a conference paper, or even co-planning a conference allows for the exchange of vital, useful, and transferable information between colleagues and beyond.   mentorship   the nice thing about forming a mentorship relationship (whether it be formal mentorship, peer mentorship, or reverse mentorship) is that very often, both parties benefit. what we’ve discovered at the university of saskatchewan is that our librarians have different kinds of research experience and expertise, and in one instance, one librarian serves in the role of the mentor while in the next instance, the other librarians may well serve that function. the one-to-one relationship between mentor and protégé (or one-to-many relationship of a mentorship team) functions as an information transfer node which can grow roots in the fertile soil of a network.   peer support   i see peer support as being different than mentorship. sometimes it’s not the expertise or the knowledge that is needed – it’s the support. it’s commiserating, listening, coaching, or cheer leading if necessary. it’s finding someone who has walked in your shoes and who can provide impactful empathy to get you over the occasional bumps in the research road.   motivation   the collaborative, mentoring, and supportive connections that can result from participation in a network can assist with motivation in the form of accountability, inspiration, and encouragement. often just having someone to report back to, however informally, is enough to keep a timeline on track and deadlines met. if a project hits a roadblock, the act of being able to brainstorm ideas or discuss methods or topics with someone else can provide needed inspiration. and sometimes all that is needed is a pep talk or a pick-me-up – some encouragement to see you through the revisions, the last bit of data analysis, or the final stretch of a literature review.   career advice/opportunities   traditionally, networking is related to career advice and advancement, and this can still be the case. increasing your professional circle increases the number of new and interesting job opportunities you will be exposed to and also provides a wider range of people who could offer valuable career advice. in turn, if your organization is on the searching end of the job opportunity, belonging to a wide-reaching network expands the pool significantly.   i am in favour of all kinds of groups that get librarians together to communicate and share. the more we are exposed in meaningful ways to what’s happening in the wider world of librarianship, the more progress we can make in whatever area we are working. when it comes to research or evidence based practice, getting together with like-minded others can get the creative juices flowing. connecting with different-minded others can be helpful as well. it’s difficult to generate new ideas while working in a vacuum. it’s always useful to get a second opinion, a second set of eyes, or a viewpoint from a different perspective. if you or your institution is interested in joining the c-eblip research network, you can find more information on the c-eblip website (http://library.usask.ca/ceblip/c-eblip-research-network/about.php) or you can contact me directly.   references   lark library applied research kollektive. (2016). retrieved from http://lark-kollektive.blogspot.ca/p/about-lark.html   libraries nova scotia research support group. (n.d.). retrieved from https://librariesns.ca/content/research-support-group   metcalfe’s law. (2016). in techopedia. retrieved from https://www.techopedia.com/definition/29066/metcalfes-law   network. (2016). in oxford dictionaries. oxford: oxford university press. retrieved from http://www.oxforddictionaries.com/definition/english/network   reed’s law. (2016). in wikipedia. retrieved from https://en.wikipedia.org/wiki/reed%27s_law   research article   embracing the generalized propensity score method: measuring the effect of library usage on first-time-in-college student academic success   jingying mao department of statistics florida state university tallahassee, florida, united states of america email: mjy_jean@hotmail.com    kirsten kinsley assessment librarian florida state university libraries tallahassee, florida, united states of america email: kkinsley@fsu.edu    received: 2 aug. 2017    accepted: 9 nov. 2017        2017 mao and kinsley. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – this research focuses on first-time-in-college (ftic) student library usage during the first academic year as number of visits (frequency) and length of stay (duration) and how that might affect first-term grade point average (gpa) and first-year retention using the generalized propensity score (gps). we also want to demonstrate that gps is a proper tool that researchers in libraries can use to make causal inferences about the effects of library usage on student academic success outcomes in observation studies.   methods – the sample for this study includes 6,380 ftic students who matriculated in the fall 2014 and fall 2015 semesters at a large southeastern university. students’ library usage (frequency and duration), background characteristics, and academic records were collected. the generalized propensity score method was used to estimate the effects of frequency and duration of ftic library visits. this method minimizes self-selection bias and allows researchers to control for   demographic, pre-college, and collegiate variables. four dose-response functions were estimated for each treatment (frequency and duration) and outcome variable (gpa and retention).   results – the estimated dose-response function plots for first-term gpa and first-year retention rate have similar shapes, which initially decrease to the minimum values then gradually increase as the treatment level increases. specifically, the estimated average first-term gpa is minimized when the ftic student only visits the library three times or spends one hour in the library during his/her first semester. the threshold for first-year retention occurs when students visit the library 15 times or spend 21 hours in the library during their first semester. after those thresholds, an increase in students’ library usage is related to an increase in their academic success.   conclusions – the generalized propensity score method gives the library researcher a scientifically rigorous methodological means to make causal inferences in an observational study (imai & van dyk, 2004). using this methodological approach demonstrates that increasing library usage is likely to increase ftic students’ first-term gpa and first-year retention rates past a certain threshold of frequency and duration.     introduction   the collegiate experience often includes a diversity of opportunities and experiences to foster student development and engagement affecting the retention and academic success of the first-time-in-college (ftic) student. according to astin’s input-environment-output (i-e-o) model of student involvement, student inputs—such as high school grade point average (gpa), act scores, and gender—are often associated predictors of first-year student success outputs (or outcomes), such as grades and retention (strauss, 2014; astin, 1997). the collegiate environment, including a student’s major, enrolled credit hours, involvement in athletics, living in learning communities, and employment is also an important influence on student outputs. another potential environmental factor that may affect student success outputs is time spent in the library.   the research study presented in this article attempts to isolate the treatment variables of number of library visits (frequency) and total hours of stay (duration) during the first year of college while controlling for other potential predictors of college success, such as student input and other collegiate environmental variables, by measuring the effects of frequency and duration of library visits on retention and gpa. since randomizing a control group of students who do not use the library and those who do is ethically impossible, how do we measure ftic students’ success and the effects of library usage while also controlling for student inputs and other non-library environmental impacts?    we decided to apply the generalized propensity score (gps) method for a number of reasons. using gps in addition to the i-e-o design gives a more rigorous approach to measuring library impact on student academic success because we attempt to control for as many inputs and other environmental collegiate variables as possible. in addition, it allows us to “make causal inferences from correlational data” and to “minimize the chances that our inferences are wrong” (astin & antonio, 2012, p. 31). as astin & antonio (2012) emphatically state, “although we can never be sure that we have controlled all such variables, the more we control, the greater confidence we can have in our causal inferences” (p. 31). furthermore, using the gps method reduces the effects of self-selection bias (astin & antonio, 2012, p. 31). the bias may be caused because students who have certain characteristics, such as higher act scores and higher high school gpa, may self-select to use the library frequently and for long durations. this may cause an overestimation of the treatment effect of library usage. gps also allows us to measure the effect of continuous library usage variables over time by frequency and duration. moreover, we can predict that with each treatment or dose of library time, retention and gpa for ftic students will increase. if more library visits and duration of stay are related to increasing retention rates and higher grades, we will have more confidence to say that as library visits increase so do the student success variables of first-year retention and gpa.   literature review   according to astin’s input-environment-output (i-e-o) model of student involvement (1970, 1990, 1993), both student inputs and the college environment influence student outputs (arrows b and c on figure 1). (please note: the terms output and outcomes will be used interchangeably throughout this paper as they relate to astin’s theory, even though outputs are typically defined differently than outcomes.) at the same time, student inputs (arrow a on figure 1) affect how students experience the college environment.   according to the model, input variables such as pre-college high school grades and college entrance exam scores (e.g., sat scores) collectively impact whether a student succeeds in college. higher education research has been exploring the environmental and engagement variables that contribute to student academic success or outputs. these variables may includes student engagement, investment in “educationally purposeful activities” (kuh, 2001, p. 12), involvement in student organizations, social interactions, and engagement with faculty (braxton, hirschy, & mcclendon, 2004; kuh, cruce, shoup, kinzie, & gonyea, 2008; roksa & whitley, 2017). “without knowing how students spend their time, it’s almost impossible to link student learning outcomes to the educational activities and processes associated with them” (kuh, 2001, p. 15).   librarians who research what factors the library contributes to student success would benefit from applying astin’s model since it offers a practical, holistic theoretical approach to looking at the interaction between student attributes and their environment and can easily incorporate library activities as part of the environmental variables. it acknowledges what academic librarians already know—that “many other factors besides the library contribute to students’ academic success . . .” (jantii & cox, 2012, p. 4). even so, libraries provide many services and resources that help to engage students in “educationally purposeful activities” that contribute to student success. “students engage in a wider variety of interactions with their libraries and it is important to examine the differences those interactions can have on student outcomes” (soria, fransen, & nackerud, 2013, p. 149).     figure 1 astin’s input-environment-output model.     in 2003, kuh and gonyea stated that “relatively little is known about what and how students’ academic library experiences contribute to desired outcomes of college . . .” (p. 258). over 15 years later, soria et al. (2017a, 2017b) report a similar dearth of research in this area, though more and more research is rapidly being published on this topic. almost 50 years ago, kramer and kramer (1968) looked at the retention rates of freshman who used the library and found that borrowing library books was associated with retention. mezick (2007) found a significant positive association between library expenditures and student persistence for all carnegie classifications and between retention and the “number of library professional staff . . . at doctoral granting institutions” (p. 564).   although other studies have looked at student outcomes and library use, it was not until the value of academic libraries’ initiative of the association of college & research libraries (acrl) that a collective, concentrated effort was made to create a body of research demonstrating academic library value and impact related to student success measures (oakleaf, 2010). following the commencement of the value of academic libraries initiative, current library research demonstrates connections between ftic student library usage and its impact on gpa and retention outcomes. emmons & wilkinson (2011) found that library input variables (e.g., wages, library volumes, and expenditures) had an effect on student retention. using a linear regression model while controlling for socioeconomic status, race, and ethnicity, they discovered that an increase in the ratio of professional library staff to students had a positive effect on both student retention (measured by students returning for their second year) and six-year graduation rates. interestingly, stemmer and mahan (2016) found that the ways that freshman used the library (outputs) were associated with gpa and retention. using the library for academic purposes like checking out books or using online resources were associated with gpa and retention, but using the library computers for personal use and the late-night study rooms for cramming sessions was negatively associated with success outcomes.   nine recent studies examined by the authors found that a combination of library space, instruction, and resource usage by ftic students was positively associated with retention, gpa, or both (kot & jones, 2015; soria et al., 2013, 2014, 2017a, 2017b; haddow, 2013; murray, ireland, & hackathorn, 2016; stemmer & mahan, 2016; stone & ramsden, 2012). note that of the studies examined, most focused on library space and resource usage effects on student outcomes which included workstation logins, study room usage, e-resources and print books usage, interactions with library personnel, use of ill and reference, and other similar resources. kot & jones (2015), soria et al. (2017b), and murray et al. (2016) also included library instruction in their list of environmental variables. some of the studies controlled for other input and environmental variables that may impact student success (kot & jones, 2015; soria et al., 2013, 2014, 2017a, 2017b). some used the propensity score matching methodology (kot & jones, 2015; soria et al. (2017b) and some studies applied astin’s i-e-o model as their conceptual framework (kot & jones, 2015; soria et al., 2014, 2017a, 2017b; stemmer & mahan, 2016).   another study, conducted by masters of economics students at florida state university using our local library turnstile data, found that students who had low gpas showed “larger academic gains from additional library usage than their high-gpa library user counterparts” (holcombe, lukashevich, & alvarez (2016, p. 14). note that though this study examined undergraduate student library usage and gpa, it was not limited to the ftic population. the use of the gps methodology is unique to this library study since we were predicting outcomes based on continuous variables of library usage over time from actual turnstile data. it is interesting to note that the two outcomes measured in this study, gpa and retention, have been correlated: higher individual gpas “may well be the single best predictors of student persistence . . .” (pascarella & terenzini 2005, p. 396). in addition, scholarship that focuses exclusively on the critical role of library instruction and its effect on first-year retention and gpa is not reviewed here.   aims   this study aims to evaluate the effect of library usage (frequency of visits and duration of stay) over the course of a semester on ftic student academic success measured in first-term gpa and first-year retention rate. in our study, student outputs or dependent variables are first-term gpa and first-year retention rate. the independent variables include the environmental variables of library usage (library visit frequency and duration) while controlling for other non-library related college environment variables. other controlled variables include student inputs, such as demographic characteristics and other pre-college academic variables. by studying first-year students we by default control for the effects “of later collegiate experiences that may also influence students’ outcomes . . .” (soria et al., 2017a, p. 10).   this is an observational study where we could not randomly assign students to different amounts of library visit treatment during their first year. as a result, students have self-selected themselves into different levels of treatment because of their different input variables, such as gender, class, major etc. so we also tried to find a statistical method to minimize the self-selection bias in our sample.   specifically, the research questions for this study are:   1)       does library usage measured in frequency (visits per semester) and duration (length of stay per semester) impact student academic success in terms of first-term gpa and first-year retention rate? 2)       are these impacts still observed after controlling for other input and environmental variables? and 3)       does embracing generalized propensity scoring give librarians more rigorous research results?   methods   data   the sample for this study includes 6,380 ftic students who matriculated in the fall 2014 and fall 2015 semesters at a large southeastern university. here ftic refers to an entering freshman or a first-year student attending college for the first time at the undergraduate level. this includes students who attended college for the first time in the prior summer term and are also enrolled in the fall term. also included are students who entered with advanced standing (having earned college credits before graduation from high school). for the purposes of this paper, retention is measured for ftic students by their “persistence between the first and second year at college” (kuh, et al., 2008, p. 555).   data in the study comes from two sources: the c-cure system (card swipe system) and the office of institutional research. the campus has two major libraries and these were chosen sites for the study because they have turnstiles that could provide primary data for our study. each library has six turnstiles, including two entrances, two exits, and a handicap entrance and exit. both libraries require students to swipe student ids at the turnstiles to enter or exit libraries. the c-cure system collects card-swipe data that includes student identification information, time that students enter or exit the library, direction (in or out), and which turnstile they use. by matching swipe-in and swipe-out records, we extracted frequency and duration of individual library usage for each semester.   at our request, the office of institutional research provided all other student background characteristics and academic records for all ftic students. by merging card-swipe data and student information data, the final data set was ready for analysis. this data was coded to keep student information anonymous. the output (dependent) variables of interest were first-term gpa and first-year retention rate.   the environment (treatment) variables of interest were library usage measures, defined as first-term library visit frequency and duration (measured in hours). other environment variables that we controlled for include major (college), class (freshman, sophomore, junior, senior or non-degree), military status, participation in athletics or sports, current load (credit hours enrolled in the first term), matriculation year (2014 or 2015), housing status (whether living on or off campus), and participation in the center for academic retention and enhancement program (provides transition support for minority students).   the input variables for the study included students’ demographic characteristics and pre-college academic variables. demographic characteristics included the student’s gender, race, citizenship, age at matriculation, parent income level, and education levels of students’ mothers and fathers. pre-college academic variables included the student’s high school gpa, act scores, and transfer credits. some of students were admitted with sat or act scores only. to compare those two measures, we transferred sat scores into corresponding act scores using an sat/act concordance/comparison chart. for those students who had both test scores, only the act scores were used. table a1 in the appendix presents summary statistics for all variables.   generalized propensity score method   to adjust for self-selection bias and control for the inputs and other environmental variables in a scientifically rigorous way, we use the gps method developed by hirano and imbens (2004). this method is a generalization of the binary treatment propensity score matching method (rosenbaum & rubin, 1983) and is used to make causal inference in the observational studies (imai & dyk, 2004).   in this study, the treatment variables (library visit frequency and duration per student) are continuous measurements that can take the value of all positive integers. so, we decided to use the gps method instead of the binary propensity score matching method to estimate the effects of continuous treatments—that is, the number of library visits and the number of hours spent in the library over time on student grades and retention.   following hirano and imbens (2004), we have random samples of ftic students indexed by . for each sample , there is a set of potential outcomes,  (i.e. first-term gpa, first-year retention rate) with a given level of treatment , referred to as the unit-level dose-response function. in our study, treatment  is the first-term library visit frequency and duration and  is an interval . for each sample , we observed a vector of covariates, , its actual treatment received, , and actual outcome corresponding to the actual treatment received, . our goal was to estimate the average dose-response function: . hereafter, we will omit  to simplify the notation.   the key assumption for the gps method is weak unconfoundedness introduced by hirano and imbens (2004):   .   we assumed that the level of treatment received is independent of the potential outcome given observed covariates. this assumption requires us to get a rich set of covariates including all possible variables that may influence selection into different levels of treatment.   based on this assumption, we were able to estimate the gps. if we write the conditional density of the treatment given the covariates as , then the gps is defined as:   .   if the gps is correctly estimated, then it has a balance property as the binary propensity score:   .   hirano and imbens (2004) mentioned that this property does not require unconfoundedness. in combination with weak unconfoundedness, it implies that the level of treatment received is unconfounded given the gps as well.   given this result, gps can be used to remove bias caused by difference in covariates in the following two steps. first, we estimated the conditional expectation of potential outcome as a function of the treatment level and estimated gps:   .   second, we estimated the dose-response function at each treatment level by taking the average of this conditional expectation over the gps evaluated at that particular treatment level:   .   implementation   the first step is to estimate the gps. since our treatment variables (frequency and duration) are counts and highly skewed with a large amount of zero values, a negative binomial generalized linear model with log link function is used to model the conditional distribution:   .   then the gps is estimated via the following:   .   there are many other ways to specify the distribution and estimate the gps. as long as the balance of covariates is achieved after adjusting for the gps, the model specification is not the key point here.   the second step is to specify the conditional expectation of potential outcome given the treatment level and estimated gps using ols. in our study, a quadratic approximation including the interaction term was used when the outcome variable is first-year gpa:   .   when the outcome is first-year retention rate, we used a logistic regression model to estimate the conditional expectation of potential outcome because retention is a binary outcome with value 0 as not being retained and 1 as being retained:   .   however, there is no direct causal interpretation of those estimated coefficients (hirano & imbens, 2004).   the final step was to estimate the average dose-response function at treatment levels of interest given the estimated parameters in the last step. in the case of first-term gpa, the dose-response function was estimated as the following:   .   and in the case of first-year retention, the dose-response function is estimated as the following:   .   we also computed the 95% confidence bands for the dose-response function based on 1,000 bootstrap replications, considering all estimation steps including gps and -parameters.   common support condition and balancing of covariates   as in the standard propensity score matching method, we needed to check the common support condition. we adapted the approach from kluve, schneider, uhlendorff, & zhao (2012). first, we divided the sample into three groups by the 30th and 70th quartiles of the treatment. for each group, we evaluated the gps for the whole sample at the group mean of the treatment. then we plotted the distribution of the evaluated gps for that group against the distribution of the evaluated gps for the rest of the sample. the overlap of those two distributions is the common support. we repeated the above procedures for all three groups. finally, we restricted our final sample to individuals who are comparable across all three groups simultaneously. in other words, we deleted individuals whose gps fell out of any common support of the three groups.   besides assessing the common support condition, balancing of covariates is also very important to the gps method. we regressed each covariate on the treatment with and without conditioning on the predicted level of treatment  (imai & van dyk, 2004).  if there was no correlation between treatment and any covariate after conditioning on the predicted treatment, then we concluded that the covariate balance is achieved after adjusting for the gps.   results   first-term gpa   all tables and figures regarding the process of implementing the gps method are included in the appendix. as previously noted, table a1 provides summary statistics. table a2 provides the estimated coefficients from the negative binomial generalized linear models using the first-term gpa as the outcome variable. both models showed that age, participation in athletics, act scores, college attended, current academic load, matriculation year, and race had influence on student library usage.   we assessed the common support condition using the method we described in the methodology section. figures a1 and a2 in the appendix illustrate the distribution of the evaluated gps before and after deleting the non-overlap for the treatment variables of frequency and duration, respectively. after imposition of common support for the frequency treatment, we deleted only 0.4% of our original sample. for the duration treatment, we deleted 0.3% of our original sample.   then we checked the balancing properties of the gps using the method proposed by imai & van dyk (2004). table a3 presents the coefficient and its standard error for each covariate with and without conditioning on . table a3 clearly demonstrates that before we conditioned on  multiple covariates were significant. after we conditioned on, no significant covariate was observed. for example, participation in athletics had a high positive correlation with both treatments (frequency and duration). however, once we conditioned on the predicted level of treatment, athletic participation was not significant in either case. so, we concluded that the balancing properties of the gps were achieved in both treatment cases.     figure 2 the dose-response function of first-term library usage frequency vs. first-term gpa.     figure 3 the dose-response function of first-term library usage duration vs. first-term gpa.     the final step of our study was to estimate the dose-response function. we regressed the outcome: first-term gpa on the treatment variable and the gps. the estimated coefficients are listed in table a4. as was mentioned before, the estimated coefficients did not have any direct causal interpretation.   the dose-response function was estimated for each treatment level of interest by averaging the estimated regression function over the gps evaluated at the desired treatment level. figures 2 and 3 present the dose-response function of first-term gpa for the treatment variables of frequency and duration, respectively. the dotted lines were 95% confidence bands based on 1,000 bootstrap replications that accounted for all estimation steps.   figures 2 and 3 show the dose-response functions for frequency and duration have similar shapes. first-term gpa first decreased and reached its minimum value, then gradually increased when the library usage frequency and duration increased.   for frequency, first-term gpa was minimized at 3.19066 when the ftic student only visited the library three times in their first semester. once the student visited the library over three times, library usage had a continued positive relationship with their first-term gpa.   similarly, for duration, first-term gpa was minimized at 3.177407 when the ftic student only spent one hour in the library during their first semester. when the student spent an hour or longer in the library there were gains in first-term gpa. the longer the time spent in the library, the larger the increase in first-term gpa.   first-year retention rate analysis procedures for first-year retention rate were almost the same as the procedures for first-term gpa, except that we included first-term gpa as a covariate when the outcome variable was retention rate. we then used a logistic regression model in order to estimate the conditional expectation of outcome.   in the appendix, table a5 presents the estimated coefficients from the gps estimation step.  figures a3 and a4 and table a6 (see the appendix) verified the common support condition and the balancing properties. the estimated coefficients from the logistic regression model are presented in table a7.   the dose-response functions were finally estimated at each treatment level of interest. figures 4 and 5 present the dose-response function of first-year retention rate for the treatment variables of frequency and duration, respectively. the dotted lines are 95% confidence bands based on 1,000 bootstrap replications that accounted for all estimation steps.     figure 4 the dose-response function of first-term library usage frequency vs. first-year retention.     figure 5 the dose-response function of first-term library usage duration vs. first-year retention.     both dose-response functions have a shape similar to figures 2 and 3. both plots indicate that first-year retention rate first declined to its minimum value within the lower value of the treatment and then gradually increased as the treatment increased.   for frequency, when students visited the library only fifteen times in their first semester, they had the lowest first-term retention rate at 93.89%. for duration, the minimum retention rate was achieved at 93.84% when ftic students spent only twenty-one hours in the library during their first semester. after that, further increases in first-term library usage frequency and duration both resulted in higher first-year retention rate.   the estimated dose-response function plots for first-term gpa and first-year retention rate have similar shapes, which initially decrease to minimum values and then gradually increase as the treatment levels increase. in other words, there was a threshold of frequency and duration of library visits where an increase of students’ library usage had a negative effect on their first-term gpa and retention rates. specifically, the estimated average first-term gpa was minimized when ftic students visited the library only three times or spent only one hour in the library during their first semester. the threshold for measurable increases in first-year retention occurred when students visited the library fifteen times or spent twenty-one hours in the library during their first semester.   as the estimated dose-response functions reveal, increasing library usage was likely to increase ftic students’ first-term gpa and first-year retention rates past a certain threshold of frequency and duration. when ftic students visited more than three times or spent more than two hours in the library during their first semester, library usage positively affected students’ first-term gpas. after ftic students crossed the threshold of visiting the library more than fifteen times or spending more than twenty-one hours there in their first semester, students with higher library usage had higher first-year retention rates.   discussion   the small drop of both first-term gpa and retention rate before reaching the thresholds for frequency and duration may be explained in several possible ways. first, we did not account for those ftic students who may go to other libraries on campus other than the two major libraries included in this study. for example, engineering majors may not choose to come to the two on-campus libraries because their department and library are located off-campus. some students may only come to the libraries at the beginning of the semester or during finals. holcombe et al. (2016), using the same cohort and data set, found that those students who come to the library only to cram during finals week do not seem to benefit from low frequency, high duration library usage per semester.     the study has several limitations. the definition of library usage used here (total frequency and duration in one semester) may be too broad. we consider only when and how long the students entered the building, ignoring what they might be doing while in the building such as using other   library services, collections, and spaces (such use of study rooms)  (soria et al. 2017a; 2017b). furthermore, we cannot presume that students are studying when they visit the library. we can only assume they are doing some form of “educationally purposeful activities” that include using databases to conduct research and studying (kuh, 2001, p. 12; kuh & gonyea, 2003). in one recent survey by cengage, results showed that student library users spend their time studying alone, using the databases and reference materials, and meeting study groups (strang, 2015). in a fall 2016 survey, the activities our students reported coming to the library for were to 1) work on a paper, project, or homework; 2) study for an exam; 3) print something; or 4) wait between classes (dawson, 2016). another limitation of this study is that it is not possible to control or account for all possible covariates that may influence the student success outcomes of gpa and first-year retention rates. especially difficult to measure are intangible, intrinsic, and individual student inputs. for example, one study found that a student’s “grit” or “mindset,” which is the “willingness to work hard for an extended period in search of a long-term goal,” was a key factor in college student success (barton, 2015, para. 9).   conclusion   our results indicate that increasing library usage contributes to higher ftic students’ first-term gpas and first-year retention rates past a certain threshold of frequency and duration. in addition, gps is a valid methodology to use because it minimizes self-selection bias and estimates the potential outcome, gpa and retention rate, at every possible value of library usage (frequency and duration).   using the gps method, future studies could build on the findings of this study by looking at library usage and the relative impact on student four-to-six-year graduation rates, library usage across different academic disciplines, and other populations of library users, such as faculty and graduate students. furthermore, future analyses could triangulate these results by analyzing the effects of library e-resource and equipment usage, instruction, and participation in library outreach and engagement activities to gain a more comprehensive understanding of how the academic library services, spaces, and resources collectively impact student success.   references   astin, a. w. (1970). the methodology of research on college impact, part one. sociology of education, 43(3), 223–254. https://dx.doi.org/10.2307/2112065   astin, a. w. (1990). assessment for excellence: the philosophy and practice of assessment and evaluation in higher education. new york: maxwell macmillan international.   astin, a. w. (1993). what matters in college?: four critical years revisited (1st ed.). san francisco: jossey-bass.   astin, a. w. (1997). how “good” is your institution’s retention rate? research in higher education, 38(6), pp. 647-658. https://dx.doi.org/10.1023/a:1024903702810     astin, a. w., & antonio, a. l. (2012). assessment for excellence: the philosophy and practice of assessment and evaluation in higher education. lanham, md: rowman & littlefield publishers.   barton, d. (2015, september 16). the most important factor in a college student’s success [blog post]. retrieved from https://blogs.wsj.com/experts/2015/09/16/the-most-important-factor-in-a-college-students-success/   braxton, j. m., hirschy, a. s., & mcclendon, s. a. (2004). understanding and reducing college student departure. ashe-eric higher education report, volume 30, issue 3. indianapolis: jossey-bass, an imprint of wiley.   cox, b. l., & jantti, m. (2012, july 17). discovering the impact of library use and student performance. educause review. retrieved from http://er.educause.edu/articles/2012/7/discovering-the-impact-of-library-use-and-student-performance   dawson, a. (2016). strozier daytime visit feedback fall 2016 results. retrieved from https://docs.google.com/document/d/1y8rjdkgnvuvvwms6dtnhwsofkqm1iuonq2qvb-enrgo/edit?usp=drive_web&usp=embed_facebook   donnelly, p. j. (2010). examining pre-college academic variables: investigating future college success. retrieved from proquest dissertations & theses global. (3398089).   haddow, g. (2013). academic library use and student retention: a qualitative analysis. library & information science research, 35(2), 127-136. https://dx.doi.org/10.1016/j.lisr.2012.12.002   hirano, k., & imbens, g. w. (2004). the propensity score with continuous treatments. in a. gelman & x. l. meng (eds.), applied bayesian modeling and causal inference from incomplete-data perspectives (pp. 73–84). john wiley & sons, ltd.   holcombe, c., lukashevich, i., & alvarez, j. (2016, july 29). measuring the effects of increased library use on gpa outcomes of fsu undergraduates. symposium on applied economics 2016: the final presentations for the m.s. in applied economics. florida state university. retrieved from https://www.lib.fsu.edu/sites/default/files/sites/default/files/upload/executive_summary.pdf   imai, k., & van dyk, d. a. (2004). causal inference with general treatment regimes. journal of the american statistical association, 99(467), 854–866. https://dx.doi.org/10.1198/016214504000001187   kot, f. c., & jones, j. l. (2014). the impact of library resource utilization on undergraduate students’ academic performance: a propensity score matching design. college & research libraries, 76(5), 566-586. https://dx.doi.org/10.5860/crl.76.5.566   kramer, l. a., & kramer, m. b. (1968). the college library and the drop-out. college & research libraries, 29(4), 310–312.     kuh, g. d. (2001). assessing what really matters to student learning: inside the national survey of student engagement. change: the magazine of higher learning, 33(3), 10–17. https://dx.doi.org/10.1080/00091380109601795   kuh, g. d., & gonyea, r. m. (2003). the role of the academic library in promoting student engagement in learning. college & research libraries, 64(4), 256–282. https://dx.doi.org/10.5860/crl.64.4.256   kuh, g. d., cruce, t. m., shoup, r., kinzie, j., & gonyea, r. m. (2008). unmasking the effects of student engagement on first-year college grades and persistence. the journal of higher education, 79(5), 540–563. https://doi.org/10.1080/00221546.2008.11772116   kluve, j., schneider, h., uhlendorff, a., & zhao, z. (2012). evaluating continuous training programmes by using the generalized propensity score. journal of the royal statistical society: series a (statistics in society), 175(2), 587–617. https://dx.doi.org/10.1111/j.1467-985x.2011.01000.x   mezick, e. m. (2007). return on investment: libraries and student retention. the journal of academic librarianship, 33(5), 561–566. https://dx.doi.org/10.1016/j.acalib.2007.05.002   murray, a., ireland, a., & hackathorn, j. (2016). the value of academic libraries: library services as a predictor of student retention. college & research libraries, 77(5), 631–642. https://dx.doi.org/10.5860/crl.77.5.631   oakleaf, m. (2010). the value of academic libraries: a comprehensive research review and report. chicago: association of college and research libraries. http://www.ala.org/acrl/sites/ala.org.acrl/files/content/issues/value/val_report.pdf   pascarella, e. t., & terenzini, p. t.  (1991). how college affects students: findings and insights from twenty years of research. san francisco: jossey-bass publishers.   roksa, j., & whitley, s. e. (2017). fostering academic success of first-year students: exploring the roles of motivation, race, and faculty. journal of college student development, 58(3), 333–348. https://dx.doi.org/10.1353/csd.2017.0026   rosenbaum, p. r., & rubin, d. b. (1983). the central role of the propensity score in observational studies for causal effects. biometrika, 70(1), 41–55. https://dx.doi.org/10.2307/2335942                 soria, k. m., fransen, j., & nackerud, s. (2013). library use and undergraduate student outcomes: new evidence for students’ retention and academic success. portal: libraries and the academy, 13(2), 147–164. https://dx.doi.org/10.1353/pla.2013.0010   soria, k. m., fransen, j., & nackerud, s. (2014). stacks, serials, search engines, and students’ success: first-year undergraduate students’ library use, academic achievement, and retention. the journal of academic librarianship, 40(1), 84–91. https://dx.doi.org/10.1016/j.acalib.2013.12.002   soria, k. m., fransen, j., & nackerud, s. (2017a). beyond books: the extended academic benefits of library use for first-year college students. college & research libraries, 78(1), 8–22. https://dx.doi.org/10.5860/crl.v78i1.16564   soria, k. m., fransen, j, & nackerud, s. (2017b). the impact of academic library resources on undergraduates’ degree completion. college & research libraries, 78(6), 812-823.   https://dx.doi.org/10.5860/crl.78.6.812   stemmer, j. k., & mahan, d. m. (2016). investigating the relationship of library usage to student outcomes. college & research libraries, 77(3), 359-375. https://dx.doi.org/10.5860/crl.77.3.359    stone, g., & ramsden, b. (2013). library impact data project: looking for the link between library usage and student attainment. college and research libraries, 74(6), 546-559. https://dx.doi.org/10.5860/crl12-406   strang, t. (2015, july 2). top four reasons students use their college library [blog post]. retrieved from https://blog.cengage.com/top-four-reasons-students-use-their-college-library     appendix   table a1 summary statistics variables mean standard deviation output variables gpa 3.278 0.690 retention 0.957 0.204 environment (treatment) variables frequency 35.066 39.705 duration 56.019 74.300 other environment variables military 0.026 0.160 athlete 0.018 0.134 housing 0.821 0.384 care 0.000 0.022 current load 12.869 1.842 class freshman 0.711 0.453 sophomore 0.253 0.435 junior 0.036 0.186 senior 0.001 0.025 non-degree 0.000 0.013 college applied studies 0.000 0.018 arts & sciences 0.301 0.459 business 0.150 0.357 communication & information 0.046 0.210 criminology 0.029 0.167 education 0.021 0.143 engineering 0.070 0.255 film school 0.005 0.071 fine arts 0.006 0.075 human sciences 0.072 0.259 music 0.027 0.163 nursing 0.025 0.157 registrar 0.000 0.013 social sciences 0.071 0.257 social work 0.006 0.078 undergraduate studies 0.146 0.353 visual arts, theatre, & dance 0.024 0.153 matriculation year 2014 0.453 0.498 2015 0.547 0.498 input variables age 20.749 0.776 us citizen 0.978 0.146 hs gpa 4.045 0.340 act 27.145 2.740 transfer or exam credit 21.679 16.793 race white 0.683 0.465 hispanic/latino 0.177 0.382 black/african american 0.046 0.210 asian 0.031 0.174 american indian/alaska native 0.002 0.041 native hawaiian/other pacific islands 0.002 0.040 two or more races 0.041 0.199 not specified 0.018 0.131 gender female 0.593 0.491 male 0.407 0.491 father's education level college 0.057 0.231 high school 0.028 0.165 middle school 0.001 0.028 unknown 0.914 0.280 mother's education level college 0.058 0.235 high school 0.024 0.153 middle school 0.002 0.040 unknown 0.916 0.277 parent income level < $1000 0.008 0.091 $1000-$40000 0.018 0.132 $40000-$75000 0.017 0.130 $75000-$100000 0.013 0.114 $100000+ 0.036 0.187 unknown 0.907 0.290     table a2 estimated coefficients from the gps estimation   treatment: frequency treatment: duration covariates estimate std. error estimate std. error   military -0.0713 0.1049 -0.0987 0.1201   athlete -0.5749a 0.1277 -0.6429a 0.1459   housing 0.0723 0.0444 0.1100c 0.0509   care 0.2593 0.7719 -0.2987 0.8878   current load 0.0319a 0.0096 0.0239c 0.0110   class.freshman 2.3445 1.5296 1.6594 1.6231   class.sophomore 2.3328 1.5314 1.6293 1.6253   class.junior 2.3246 1.5368 1.7034 1.6319   class.senior 2.5368 1.6819 1.5132 1.8105   college.applied.studies -2.1336c 1.0327 -0.9900 1.1066   college.arts & sciences 0.2616c 0.1132 0.5256a 0.1298   college.business 0.0681 0.1180 0.3541b 0.1352   college.communication & information 0.0556 0.1338 0.3242c 0.1533   college.criminology 0.0034 0.1471 0.1712 0.1686   college.education -0.1176 0.1593 -0.0022 0.1824   college.engineering 0.3619b 0.1272 0.6368a 0.1459   college.film.school -0.0923 0.2603 -0.2208 0.2986   college.fine.arts -0.1341 0.2564 -0.1448 0.2934   college.human.sciences 0.2856c 0.1257 0.6087a 0.1440   college.music -0.2808d 0.1488 -0.5593b 0.1707   college.nursing 0.2225 0.1511 0.5199b 0.1731   college.social.sciences 0.2755c 0.1260 0.5308a 0.1444   college.social.work 0.2448 0.2405 0.3673 0.2756   college.undergraduate.studies 0.0885 0.1178 0.3144c 0.1350   matriculationyearter.20149 -0.1387b 0.0427 -0.1155c 0.0489   age 0.0755b 0.0276 0.0652c 0.0317   us citizen -0.1027 0.1189 -0.0344 0.1363   hs gpa 0.0655 0.0591 0.0191 0.0677   act -0.0139c 0.0070 -0.0236b 0.0080   transfer or exam credit -0.0009 0.0019 -0.0014 0.0022   race.white -0.1113 0.1281 -0.0075 0.1468   race.hispanic.latino -0.0377 0.1327 0.0865 0.1522   race.black.african.american 0.0161 0.1490 0.0765 0.1709   race.asian 0.2804d 0.1585 0.3924c 0.1817   race.american.indian.alaska 0.1095 0.4228 0.1406 0.4849   race.native.hawaiian.oth.pa 0.2246 0.4402 0.0388 0.5055   race.two.or.more.races -0.0897 0.1509 0.0016 0.1730   gender.male 0.1047b 0.0368 -0.0265 0.0422   educationfather.college -0.2234 0.2676 -0.3814 0.3063   educationfather.high.school -0.1018 0.2706 -0.3427 0.3098   educationfather.middle.school -0.6790 0.5771 -1.3476c 0.6611   educationmother.college 0.1792 0.2459 0.1591 0.2815   educationmother.high.school 0.0914 0.2560 0.0494 0.2930   educationmother.middle.school -0.0591 0.4932 -0.2111 0.5648   parentincome....1000 0.0774 0.2275 0.2417 0.2605   parentincome..1000..40000 -0.2691 0.2604 -0.1351 0.2981   parentincome..40000..75000 -0.0937 0.2729 0.0781 0.3124   parentincome..75000.100000 -0.3024 0.2875 -0.0555 0.3290   parentincome..100000 -0.2199 0.2579 -0.0599 0.2952   asignificant at the 0.1% level bsignificant at the 1% level csignificant at the 5% level dsignificant at the 10% level       figure a1 common support condition for frequency.     figure a2 common support condition for duration.     table a3 covariate balance with and without conditioning on treatment: frequency treatment: duration   without condition condition without condition condition covariates est std. error est std. error est std. error est std. error military -1.936 3.108 0.526 3.057 -2.772 5.784 1.440 5.688 athlete -14.716a 3.801 -0.301 3.864 -25.067a 6.985 -0.703 7.060 housing 1.317 1.301 -0.655 1.285 4.120d 2.422 -0.504 2.399 care 11.519 22.951 -2.977 22.564 -17.749 42.721 -1.992 41.973 currentload 1.118a 0.270 0.075 0.275 0.892d 0.503 0.028 0.498 class.freshman -0.359 1.100 -0.267 1.080 1.943 2.046 -0.724 2.017 class.sophomore 0.657 1.147 0.267 1.127 -1.968 2.134 0.912 2.105 class.junior -1.365 2.692 0.260 2.646 -0.424 5.000 -0.542 4.911 class.senior -5.406 19.878 -7.218 19.525 -20.587 37.000 -7.299 36.352 college.arts & sciences 4.786a 1.085 0.087 1.115 8.660a 2.018 0.145 2.068 college.business -3.644b 1.394 -0.120 1.390 -4.658d 2.594 -0.519 2.563 college.communication & infor -4.302d 2.369 0.145 2.346 -4.382 4.410 -0.331 4.340 college.criminology -6.015c 2.996 -0.233 2.969 -11.474c 5.562 0.612 5.523 college.education -9.548b 3.481 0.706 3.489 -19.332b 6.455 0.973 6.487 college.engineering 8.641a 1.951 -0.073 2.010 11.458b 3.637 -0.691 3.666 college.film.school -6.655 7.043 -0.487 6.930 -25.044d 13.107 -0.126 12.981 college.fine.arts -15.940c 7.153 0.055 7.108 -33.430b 12.358 -1.315 12.331 college.human.sciences 3.163 1.926 -0.034 1.903 13.329a 3.585 1.253 3.617 college.music -11.205a 3.087 0.574 3.137 -33.874a 5.820 1.200 6.247 college.nursing 0.914 3.163 -0.293 3.108 6.410 5.887 -1.413 5.806 college.social.sciences 3.658d 1.939 0.037 1.920 5.109 3.606 -0.348 3.560 college.social.work -1.366 6.384 -0.835 6.270 -6.215 11.882 -0.654 11.676 college.undergraduate.studies -3.875b 1.410 0.001 1.409 -6.168c 2.622 0.457 2.613 matriculationyearter.20149 -1.728d 1.002 0.089 0.991 -2.460 1.863 -0.307 1.836 age 1.165 d 0.649 0.013 0.643 1.267 1.200 -0.483 1.184 uscitizen -6.396 d 3.444 0.910 3.418 -9.033 6.389 -1.147 6.297 hsgpa 1.525 1.468 -0.031 1.445 -1.398 2.734 -0.209 2.686 act -0.168 0.182 -0.045 0.179 -1.129a 0.339 -0.029 0.341 transfer or exam credit 0.005 0.030 0.006 0.029 -0.054 0.055 0.016 0.054 race.white -5.577a 1.069 -0.653 1.107 -9.178a 1.990 -0.937 2.038 race.hispanic.latino 2.554d 1.305 0.497 1.289 5.730c 2.427 0.340 2.412 race.black.african.american 4.788c 2.377 0.488 2.352 7.144 4.417 1.146 4.357 race.asian 16.637a 2.869 0.482 3.049 26.383a 5.344 0.283 5.561 race.american.indian.alaska 8.863 11.993 0.214 11.794 7.120 22.325 -0.085 21.932 race.native.hawaiian.oth.pa 8.661 12.578 -2.597 12.377 -6.284 23.412 -3.421 22.996 race.two.or.more.races 0.594 2.503 0.275 2.459 0.310 4.651 0.560 4.568 gender.m 3.301b 1.01 -0.371 1.027 -1.505 1.888 -0.548 1.856 educationfather.college -9.825a 2.176 -0.668 2.231 -16.136a 4.041 -1.312 4.099 educationfather.high.school -4.732 3.063 1.153 3.034 -11.430c 5.653 1.782 5.622 educationfather.middle.school -18.915 19.877 -5.248 19.546 -31.344 36.999 3.270 36.412 educationmother.college -8.686a 2.148 -0.640 2.183 -14.549a 3.989 -1.151 4.025 educationmother.high.school -7.704c 3.304 0.271 3.290 -14.884c 6.090 0.268 6.068 educationmother.middle.scho -7.497 13.257 1.780 13.036 -11.964 26.172 7.415 25.736 parentincome....1000 -0.193 5.534 -2.504 5.438 5.859 10.301 -0.247 10.125 parentincome..1000..40000 -8.768c 3.821 -0.093 3.799 -14.877c 7.113 1.502 7.072 parentincome..40000..75000 -4.059 3.822 0.459 3.766 -7.888 7.147 0.072 7.039 parentincome..75000.100000 -10.535 4.470c 0.485 4.453 -17.606c 8.220 -1.511 8.146 parentincome..100000. -9.337a 2.719 -0.525 2.738 -16.642a 5.028 -1.055 5.052 asignificant at the 0.1% level bsignificant at the 1% level csignificant at the 5% level dsignificant at the 10% level     table a4 estimated coefficients of conditional distribution of gpa given treatment and gps treatment: frequency treatment: duration estimate std. error estimate std. error intercept 3.0990a 0.1311 intercept 3.2390a 0.0880 frequency 0.0039a 0.0010 duration 0.0008c 0.0003 frequency^2 0.0000b 0.0000 duration^2 0.0000 0.0000 gps 1.9740 3.2350 gps -2.1390 1.7650 gps^2 -7.1340 20.2600 gps^2 15.7100d 9.0180 frequency*gps 0.1875 0.3512 duration*gps 0.1173 0.3676 asignificant at the 0.1% level bsignificant at the 1% level csignificant at the 5% level dsignificant at the 10% level     table a5 estimated coefficients from the gps estimation treatment: frequency                            treatment: duration covariates estimate std. error estimate std. error gpa 0.2140a 0.0283 0.2084a 0.0324 military -0.0586 0.1045 -0.0911 0.1199 athlete -0.6102a 0.1273 -0.6777a 0.1457 housing 0.0482 0.0442 0.0838 0.0507 care 0.2962 0.7687 -0.2671 0.8852 current load 0.0069 0.0099 -0.0008 0.0113 class.freshman 2.3856 1.5245 1.6851 1.6187 class.sophomore 2.3646 1.5263 1.6456 1.6209 class.junior 2.3548 1.5316 1.7060 1.6276 class.senior 2.7508 1.6760 1.6585 1.8056 college.applied.studies -2.1790c 1.0298 -1.0332 1.1037 college.arts & sciences 0.3227b 0.1131 0.5905a 0.1298 college.business 0.1013 0.1176 0.3939b 0.1350 college.communication & information 0.0601 0.1333 0.3383c 0.1529 college.criminology 0.0301 0.1466 0.2083 0.1682 college.education -0.1111 0.1586 0.0170 0.1819 college.engineering 0.4516a 0.1274 0.7308a 0.1463 college.film.school -0.0267 0.2593 -0.1574 0.2978 college.fine.arts -0.2152 0.2556 -0.2062 0.2927 college.human.sciences 0.3389b 0.1254 0.6642a 0.1439 college.music -0.2335 0.1483 -0.4992b 0.1703 college.nursing 0.2451 0.1506 0.5535b 0.1727 college.social.sciences 0.2956 0.1255 0.5550a 0.1440 college.social.work 0.3005 0.2394c 0.4239 0.2748 college.undergraduate.studies 0.1287 0.1175 0.3610b 0.1348 matriculationyearter.20149 -0.1213 0.0425 -0.1020c 0.0487 age 0.0595c 0.0276 0.0515 0.0316 us citizen -0.0978 0.1184 -0.0327 0.1359 hs gpa -0.0619 0.0616b -0.1012 0.0707 act -0.0163c 0.0070 -0.0264a 0.0080 transfer or exam credit -0.0003 0.0019 -0.0007 0.0022 race.white -0.1151 0.1275 -0.0031 0.1464 race.hispanic.latino -0.0488 0.1322 0.0879d 0.1517 race.black.african.american 0.0074d 0.1484 0.0770 0.1704 race.asian 0.2748 0.1578 0.3960c 0.1812 race.american.indian.alaska 0.1277 0.4212 0.1675 0.4837 race.native.hawaiian.oth.pa 0.2161 0.4384 0.0228 0.5042 race.two.or.more.races -0.0783 0.1503 0.0170 0.1725 gender.male 0.1198 0.0368 -0.0140 0.0422 educationfather.college -0.2083 0.2665 -0.3705 0.3054 educationfather.high.school -0.0887 0.2695 -0.3393 0.3089 educationfather.middle.school -0.5736 0.5743 -1.2376d 0.6588 educationmother.college 0.1774 0.2449 0.1567 0.2807 educationmother.high.school 0.0653 0.2549 0.0233 0.2922 educationmother.middle.school -0.0353 0.4912 -0.1828 0.5631 parentincome....1000 0.1116 0.2267 0.2785 0.2598 parentincome..1000..40000 -0.2009 0.2596b -0.0745 0.2975 parentincome..40000..75000 -0.0625 0.2719 0.1074 0.3116 parentincome..75000.100000 -0.3075 0.2864 -0.0456 0.3281 parentincome..100000. -0.1718 0.2570 -0.0019 0.2945 asignificant at the 0.1% level bsignificant at the 1% level csignificant at the 5% level dsignificant at the 10% level       figure a3 common support condition for frequency.     figure a4 common support condition for duration.     table a6 covariate balance with and without conditioning on treatment: frequency treatment: duration without condition condition without condition condition covariates est std. error est std. error est std. error est std. error   gpa 5.674a 0.727 0.327 0.781 7.768a 1.350 0.429 1.395   military -1.898 3.102 0.923 3.026 -2.804 5.793 2.214 5.666   athlete -15.467a 3.713 -0.152 3.721 -25.607a 6.935 -0.445 6.940   housing 1.262 1.299 -0.087 1.268 4.324d 2.423 0.579 2.378   care 11.556 22.913 -3.566 22.338 -17.780 42.785 -0.885 41.806   currentload 1.053a 0.275 0.284 0.271 0.887d 0.506 0.443 0.495   class.freshman -0.397 1.098 -0.143 1.069 1.966 2.050 -0.482 2.007   class.sophomore 0.649 1.144 0.075 1.115 -1.988 2.138 0.522 2.094   class.junior -1.110 2.699 0.708 2.631 -0.456 5.008 0.236 4.892   class.senior -5.369 19.845 -14.297 19.339 -20.618 37.056 -12.298 36.201   college.arts...sciences 4.909a 1.082 -0.303 1.096 8.608a 2.022 -0.461 2.049   college.business -3.680b 1.391 0.252 1.373 -4.695d 2.598 -0.191 2.552   college.communication...infor -4.263d 2.365 0.465 2.319 -4.415 4.417 -0.438 4.321   college.criminology -5.976c 2.991 0.432 2.936 -11.501c 5.570 1.372 5.493   college.education -9.678b 3.462 1.356 3.429 -19.364b 6.464 1.681 6.436   college.engineering 8.373a 1.952 -0.750 1.971 11.757b 3.647 -0.627 3.637   college.film.school -6.618 7.031 -0.666 6.858 -25.075d 13.127 0.707 12.912   college.fine.arts -17.297b 6.628 0.784 6.536 -33.462b 12.376 0.447 12.255   college.human.sciences 3.219d 1.922 -0.502 1.884 12.929a 3.587 0.253 3.585   college.music -11.371a 3.064 0.921 3.064 -33.435a 5.918 2.781 6.196   college.nursing 0.952 3.158 0.182 3.077 6.378 5.896 -1.268 5.777   college.social.sciences 3.795c 1.934 -0.128 1.897 5.076 3.612 -0.439 3.543   college.social.work -1.329 6.373 -2.496 6.209 -6.246 11.900 -2.610 11.627   college.undergraduate.studies -3.782b 1.409 0.352 1.392 -6.205c 2.626 0.820 2.598   college.visual.arts..theatre. -5.690d 3.246 -0.013 3.178 -20.066a 6.058 -0.168 6.035   matriculationyearter.20149 -1.807d 0.999 0.134 0.979 -1.994 1.867 0.121 1.828   age 1.308c 0.644 0.053 0.631 1.649 1.201 -0.155 1.178   uscitizen -6.704c 3.414 2.121 3.361 -8.449 6.398 0.951 6.274   hsgpa 1.396 1.465 -0.311 1.430 -1.224 2.737 -0.355 2.674   act -0.183 0.182 -0.015 0.177 -1.161a 0.339 0.058 0.339   transferorexamcredit 0.004 0.030 -0.002 0.029 -0.057 0.055 0.002 0.054   race.white -5.593a 1.067 -0.339 1.083 -9.155a 1.994 -0.598 2.016   race.hispanic.latino 2.671c 1.303 0.303 1.276 5.693c 2.431 -0.392 2.402   race.black.african.american 4.739c 2.369 0.602 2.319 6.910 4.417 1.316 4.327   race.asian 16.431a 2.871 -0.979 2.975 27.114a 5.392 0.059 5.520   race.american.indian.alaska 8.900 11.973 1.262 11.673 7.089 22.358 1.818 21.843   race.native.hawaiian.oth.pa 8.698 12.556 -3.485 12.251 -6.315 23.448 -2.985 22.905   race.two.or.more.races 0.508 2.494 0.468 2.430 0.277 4.658 1.287 4.550   gender.m 3.468a 1.012 -0.257 1.008 -1.556 1.892 0.040 1.851   educationfather.college -9.672a 2.170 -0.011 2.185 -16.042a 4.053 -0.649 4.066   educationfather.high.school -5.033d 3.032 1.256 2.975 -11.238c 5.678 2.428 5.603   educationfather.middle.school -18.878 19.843 -5.654 19.347 -31.375 37.055 3.601 36.254   educationmother.college -8.656a 2.136 -0.218 2.136 -14.566a 3.995 -0.715 3.990   educationmother.high.school -7.534c 3.299 0.959 3.248 -14.423c 6.139 1.349 6.069   educationmother.middle.scho -7.460 13.235 1.873 12.904 -11.995 26.211 7.913 25.629   parentincome....1000 0.682 5.473 -2.517 5.335 4.650 10.220 -2.945 9.992   parentincome..1000..40000 -8.851c 3.815 0.328 3.752 -14.909c 7.124 2.560 7.033   parentincome..40000..75000 -4.022 3.816 0.834 3.727 -7.920 7.157 0.953 7.011   parentincome..75000.100000 -10.584c 4.435 1.780 4.375 -17.638c 8.232 -0.373 8.106   parentincome..100000. -9.479a 2.702 -0.412 2.682 -16.324b 5.058 -1.122 5.023   asignificant at the 0.1% level bsignificant at the 1% level csignificant at the 5% level dsignificant at the 10% level     table a7 estimated coefficients of conditional distribution of gpa given treatment and gps treatment: frequency treatment: duration     estimate std. error estimate std. error intercept 5.2350a 0.9633 intercept 3.0600a 0.6626 frequency 0.0127 0.0083 duration 0.0148a 0.0028 frequency^2 0.0000 0.0000 duration^2 0.0000b 0.0000 gps -53.1700c 24.3100 gps 17.2900 12.1200 gps^2 366.7000c 161.7000 gps^2 -123.1000c 55.9100 frequency*gps -8.7860a 2.6010 duration*gps -4.1490 2.9280 asignificant at the 0.1% level bsignificant at the 1% level csignificant at the 5% level dsignificant at the 10% level     research article   connecting music and place: exploring library collection data using geo-visualizations   carolyn doi music & education librarian university library university of saskatchewan saskatoon, saskatchewan, canada email: carolyn.doi@usask.ca   received: 23 jan. 2017    accepted: 26 mar. 2017      2017 doi. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objectives – this project had two stated objectives: 1) to compare the location and concentration of saskatchewan-based large ensembles (bands, orchestras, choirs) within the province, with the intention to draw conclusions about the history of community-based musical activity within the province; and 2) to enable location-based browsing of saskatchewan music materials through an interactive search interface.   methods – data was harvested from marc metadata found in the library catalogue for a special collection of saskatchewan music at the university of saskatchewan. microsoft excel and openrefine were used to screen, clean, and enhance the dataset. data was imported into arcgis software, where it was plotted using a geo-visualization showing location and concentrations of musical activity by large ensembles within the province. the geo-visualization also allows users to filter results based on the ensemble type (band, orchestra, or choir).   results – the geo-visualization shows that albums from large community ensembles appear across the province, in cities and towns of all sizes. the ensembles are concentrated in the southern portion of the province and there is a correlation between population density and ensemble location. choral ensembles are more prevalent than bands and orchestras, and appear more widely across the province, whereas bands and orchestras are concentrated around larger centres.   conclusions – library catalogue data contains unique information for research based on special collections, though additional cleaning is needed. using geospatial visualizations to navigate collections allows for more intuitive searching by location, and allow users to compare facets. while not appropriate for all kinds of searching, maps are useful for browsing and for location-based searches. information is displayed in a visual way that allows users to explore and connect with other platforms for more information.   introduction   local music collections are a type of library collection composed of music or musical materials that are representative of an identified place or location. previous approaches for searching library or archival holdings of local music in online environments has primarily involved use of public access catalogues, finding aids, or digital asset management systems. these approaches to searching are useful for text-based queries, but may hinder understanding of the connection to place, which is the unique and defining aspect of local music. this paper illustrates the potential for analysis of local music materials using a geo-visualization tool based on a case study of large ensemble sound recordings from a local music collection at the university of saskatchewan. results are presented and analyzed using a map developed on the arcgis platform. this approach uses identification by ensemble location and categorization by ensemble type, plotted using clusters and points on a topographic map.   the dataset used for this research was built from library catalogue records of sound recordings in the saskatchewan music collection (smc), a historical local music collection at the university library, university of saskatchewan. the collection aims to preserve the musical heritage of the province through collection of local music materials, containing representative items from the early 1900s to the current day. items meet collection criteria when they are considered local by demonstrating a connection to an artist, composer, music label, or topic relating to places within the canadian province of saskatchewan.   this is a growing collection that, as of january 2017, contains 3202 items, the majority (90.9%) of which are sound recordings. these items are physically held in the education & music library at the university of saskatchewan, and are fully catalogued and findable in the library catalogue under the series title “saskatchewan music collection.” approximately a third of the smc has been digitized and is available online through sask history online (sho), an initiative that aims to bring together digital collections from galleries, archives, museums, and libraries within the province (http://saskhistoryonline.ca/). sho is built on the islandora platform, an open-source software framework for digital asset management (https://islandora.ca/).    project background   an overview of the origins of this project will help to contextualize the analysis presented in this paper. the music represented in the saskatchewan music collection is not limited by format or genre, aiming to preserve and make available music pertaining to all aspects of the province’s history. one of the defining features of this musical history is a prevalence of music from community-based large ensembles, which have often been established by and connected to community organizations such as schools, service clubs, churches, or arts organizations.   during an exhibit of smc materials at the university library in early 2015, it was the albums of community bands, orchestras, and choirs connected with rural saskatchewan towns and cities that especially engaged visitors. the album covers provided a unique gateway into stories and memories connected with the musical activities of these places. this experience prompted me to consider how the digitized albums from the smc might be presented in a similar way, where one might have the opportunity to learn more about the connection between the music and geography of the province.   in the spring of 2016, the engineering and gis librarian (kristin lee) and i (carolyn doi) launched a project to develop a geo-visualization using the local music collection data. the goals of this project were to create a dataset from the smc catalogue data, and to present this dataset online for public use in a geo-visualization tool. i managed the musicological and music collection management aspects of the project, while lee oversaw the aspects related to data management and map development. staff in the education & music library, library systems & it, and the collections unit also offered additional support for the data collection, cleaning, and implementation of the map.   this paper shows the process for collecting the data, cleaning the data, and building the geo-visualization. this is followed by a presentation of findings based on the data analysis, conducted in late 2016.   literature review   history of music in saskatchewan   saskatchewan is a prairie province located at the geographic centre of canada. settled in 1774 and then created in 1905, the musical life within the province has been defined in many ways by geography, climate, settlement, and industrial history. its musical origins are tied to a diversity of contributors including: “military and police bands, church choirs, immigrant teachers, enthusiastic residents, and . . . many travelling groups of instrumentalists and singers who came by rail across the continent from the earliest days of settlement” (swales, 2006b).   literature on the history of artistic development of the province is sparse, often found primarily in encyclopedias, general histories of canadian music, or in more anecdotal sources such as newspapers and reports. an early account of this history by robert solem, as summarized by lewis, recounts the quick development in the early years of the province as especially vibrant:   . . . a period of rapid growth accompanied by a spirit of musical optimism. by 1912 the city of saskatoon, for example, could boast of four good bands, two good orchestras, a conservatory of music, and a philharmonic society. this society was established by a large and energetic group of music teachers (1973, p. 6).   the history of bands, orchestras, and choirs in the province has been documented intermittently, primarily through narrative summaries. despite this, there is evidence that the province was flourishing with artistic activity.   orchestras   although fewer in number than other types of ensembles, saskatchewan has maintained two notable orchestras since the early 1900s. the regina symphony orchestra (rso) was founded in 1908 and is known as one of canada's oldest symphony orchestras. originally known as the regina orchestral society, its first concert was given on december 3, 1908, under the direction of its founder, l. frank laubach (wall, 2006). meanwhile, in 1927, 40 amateur members came together to form an early iteration of the saskatoon symphony orchestra (sso). the only symphony in the northern part of the province, the sso is one of the only ensembles in the region that performs major symphonic repertoire on a regular basis (conway, 2006). the presence of these ensembles is significant since they act to encourage the overall health of the musical community. as “magnets for orchestral performers,” who by extension “enrich the musical life of the community as soloists, as players in . . . other musical groups . . . and as music teachers” (conway, 2012a).   bands   wasiak’s paper “school bands in saskatchewan, canada: a history” outlines the history of school band development in the province, including an overview of curricular development, repertoire, historical impacts, challenges, and teaching methods. he states that the implementation of band programs in the schools occurred “comparatively late” in the province and it was not until the 1950s that bands “gained a foothold in the province's schools” (2000, p. 113). development was influenced both by “british bands, [which] were the prototypes for the first military, civilian, and (later) community bands in the province” and by “widespread and profound” american influences, which included teaching methods, repertoire, and expertise (p. 125).   choirs and choir music   the development of the saskatchewan arts board, choral societies, and arts festivals were influential in the development and support for the many community choirs, which were prominent in local churches, schools, and communities (swales, 2006b). church, school, and community choirs were at the core of the saskatchewan music festivals in the early 1900s and were known for their large numbers of participants. “many church choirs exceeded fifty members, and in the 1950s the bishop’s school for choristers in the diocese of qu’appelle had over 100 boys annually each summer” (swales, 2006a). in the paper “what about choral art music from the prairies?,” morgan identifies five features of choral music from the prairies including: 1) the christian church, 2) prairie landscapes, 3) secondary and post-secondary education, 4) the arrival of new immigrants, and 5) aboriginal culture (n.d, p. 3). these themes, morgan states, have had a profound influence on the history of choral music (and by extension, other musical genres) from the prairies, which is made up of a “rich artistic community that is well endowed with musicians who create and develop choral art” (p. 19).   geo-visualizations and digital libraries   within the realm of computer science, there has been extensive literature on the topic of geo-visualizations as they relate to digital libraries. one of the earliest and most widely cited pieces of literature introducing the concept of data visualization comes from ahlberg and shneiderman’s chapter titled “visual information seeking: tight coupling of dynamic query filters with starfield displays” from 1993, which discusses the new concepts of “visual information seeking” applications and methods. the authors pose several principles unique to visual search such as: “dynamic query filters [that] are rapidly adjusted with sliders, buttons, etc.,” “starfield display . . . result sets [that] are continuously available and support viewing of hundreds or thousands of items,” and “tight coupling . . . query components [that] are interrelated in ways that preserve display invariants and support progressive refinement” (pp. 313-14). these principles set the foundation for many of the visualization case studies being implemented today.   the literature on using geo-visualizations to understand music collections has focused on personal libraries (leitich & topf, 2007; torrens, hertzog, & arcos, 2004) or commercial applications (pampalk, rauber, & merkl, 2002). the literature on data visualizations for use in cultural heritage institutions has been less saturated, though it is growing in recent years. much of this research comes in the form of case studies that focus on how visualizations can: assist librarians in collection assessment activities (kilb & jansen, 2016; lowery, 2011), facilitate user understandings of print library collections (finch & flenner, 2016), or improve access to digital collections (deal, 2014; shiri, 2008; hoek & mayr, 2014).    music collections have often presented challenges for libraries and archives when it comes to information retrieval and organization. typically, music collections are represented with textual metadata and searchable using text queries, though these are not ideal for browsing and serendipitous searching. as deal identifies, “browsing features, if they are offered, are often limited to a list of metadata vocabulary terms or a search that displays the entire contents of a collection, one page of 10-20 items at a time” (2014, p. 15). additionally, finding a way to manage non-textual information such as sound files or images has posed challenges in the library systems environment. speaking specifically on the topic of ethnomusicological collections, magas and proutskova identify both metadata and content management as challenging areas in the design of music collection information retrieval, especially in relation to the audio content (2013, p. 152).   benefits are found when visualizations are used to engage with spatial representations of data. it is this interaction that has an “important role in sense-making activities involving space. it allows users to restructure and modify a space's content according to their perceptual and cognitive needs. for example, interaction with the geographic space may support understanding of differences among various types of locations and their associations with collections.” (buchel & sedig, 2014, para. 3). while literature on geo-visualizations of music collections in academic settings is limited, there are many examples of projects within academic, commercial, and public domains that serve as examples of the potential for such work. notable examples include maps representing locations of music library collections (gonzález ribot, 2015; ward, 2015), maps documenting musical events in time such as the travelling through time with sounds project (europeana, n.d.), and maps created to link users with items from a music collection, such as the smithsonian folkways holiday music map (2013).   aims   this project had two identified aims:   1)       to understand the number, location, and concentration of community-based large ensembles in saskatchewan and 2)       to enable location-based searching and browsing of the saskatchewan music collection.   these aims support a longer-term goal to increase the visibility of the saskatchewan music collection and to increase visibility of saskatchewan music by making it more accessible to the public and scholarly community.   methods   data collection and screening   marc catalogue records from the saskatchewan music collection were used as the basis for analysis in this project. the catalogue data was exported in a .csv file format and initially contained records for all items from the smc (3202 items). screening criteria were established to narrow the dataset. items were included if they met the following screening conditions:   1)       sound recordings in any format. 2)       recordings by large ensembles that were in an identifiable place (city or town) in saskatchewan. large ensembles were defined as musical groups that self-identified as choirs (including children’s choirs, church choirs, and community choirs), bands (including concert bands, wind bands, and pipe bands), or orchestras (symphony orchestras, string orchestras, and fiddle orchestras).    items were excluded if they met the following criteria:   1)       sound recordings containing reductions of choir, band, or orchestra music but not performed by a large grouping of musicians (e.g., a quartet playing string orchestra parts). 2)       ensembles that were not associated with a place within the province. for example, a group might be based in another province, but have recorded the music in saskatchewan. 3)       items that had not been digitized and uploaded to the islandora digital asset repository system at the university of saskatchewan.   the screening was conducted by applying filters using the “find” feature in microsoft excel for bands, orchestras, choirs, and related subject terms. no date filters were applied, as all eras were of interest. after the screening process, 119 results remained.   data cleaning   the dataset was cleaned first using microsoft excel and then using openrefine software. when the data was first received, each record contained 113 fields. the empty fields were removed (many were empty marc fields) as were any fields determined to be irrelevant to this project (e.g., the marc field 90 containing the library call number). marc field numbers were then translated to natural language headings. at this point in the process, a field containing the ensemble location was added. in some cases, this information was found in the dataset, album cover, or liner notes, but when the information was not clearly indicated, additional research was done to identify the location of the ensemble.   in the second phase of cleaning, the dataset was imported into openrefine, an open source application that can be used for data cleanup processes (http://openrefine.org/). most of this stage of cleaning involved adding new fields and standardizing the language within the records. standard openrefine commands such as “find and replace,” “slice,” “chomp,” and “trim” were employed to standardize language and organize data. information on how to apply these commands in openrefine was aided by use of the openrefine github wiki page (wicentowski, 2015). records were enhanced by adding new fields and a unique identifier for each record. the ensemble categories (choir, band, or orchestra) were added, and in cases where an album included more than one type of ensemble, multiple categories were applied. three types of urls were added to each record: a link to the item record in the university library catalogue, a link to the digital album in the islandora repository, and a link to the location of the cover image of the album located on the islandora server.   existing information in the records also needed to be standardized, because of variations in the catalogue language practices. the information in the 300 field (description) was used to add a new format field in the record added to indicate the type of sound recording (formats included 33⅓ rpm, 45 rpm, 78 rpm, and compact disc). the information found in the 260 field (publication) was extracted to identify a place of publication. place names needed to be standardized, as variations in cataloguing practices have resulted in the same types of information being represented in multiple ways. for example, the name “saskatchewan” appears in a variety of ways, including “sk,” “sask,” and “saskatchewan.” using openrefine facet filters and reconciling category function was useful in grouping variations of the same word for editing. using the “cluster” feature allows the user to group information that might be related and the “merge” option provides an easy way to reconcile some of the values.     table 1 comparison of terminology allowed in rda 3.4.1.3 extent (“300 $a,” yale university) rda 3.3.1.3 alternative: common usage aacr2 audio disc cd or compact disc sound disc audiocassette cassette or cassette tape sound cassette audio cartridge minidisc sound disc cartridge     subject headings found in the 650 field were cleaned using the openrefine “slice” function to remove unnecessary characters, words, and duplication. for example, “|achoruses with band|zsaskatchewan|aband music|zsaskatchewan” was updated to “choruses with band; band music” in the final dataset. in this case, the word “saskatchewan” was removed because it is assumed all items in the dataset are from saskatchewan. semi-colons were added instead of pipes, in order to clean the appearance of this information in the public-facing map.   based on the location of the ensemble, latitude and longitude fields were added to the dataset. this was done using the google geocoding api, a service that converts addresses into geographic coordinates that can then be used to place markers on a map (google developers, 2017). instructions for how to translate addresses to latitude and longitude data were located in the openrefine github wiki (moscovitz, 2015). the finalized dataset was published through the university of alberta dataverse project (doi & lee, 2017b).   challenges   using library catalogue metadata to build the dataset had both advantages and disadvantages. even though using tools such as openrefine simplified the process, it still took considerable time and several members of staff to clean and finalize the dataset. since the marc catalogue records had been created by a variety of means, including outsourced copy cataloguing and internal original cataloguing, the initial dataset was inconsistent. this was particularly apparent in the way that musical formats are described in the marc records. this information is usually found in the 300 field for physical description, though standards for the type of information to include in this field varies from aacr2 (anglo-american cataloguing rules, 2nd edition) to rda (resource description and access). these may include extent, duration, sound characteristics, dimensions, content type, media type, and carrier type (yale university library, n.d.a). this information is not always included consistently, and when it is, variant forms of spelling or terminology may be used. rda 3.4.1.3 allows use of either an authorized term from the list at 3.3.1.3, or, as an alternative, a term in common usage (see table 1).   the original marc record includes a place of publication, but this information was not always consistently formatted. in some cases, this is because publication information on sound recordings has not been regulated in the same way as monographs; in other cases this is because some sound recordings were produced independently by the artist, resulting in inconsistent formatting on the albums.   while these challenges with the catalogue data slowed our processes, it was by reducing the size of the dataset that the project remained manageable. the same work might be more challenging with a larger or more diverse set of records.   data visualization   the dataset was exported into .csv file format and imported into arcgis, a mapping and analytics platform that provides “contextual tools for mapping and spatial reasoning . . . to explore data and share location-based insights” (esri, n.d.). arcgis was chosen in part because there is internal support for the platform at the university of saskatchewan, and more importantly because it provided a stable platform on which to present the data to meet the research aims for this project. for this project, we implemented two instances of arcgis: a privately managed project where data can be imported and visualization aspects can be managed, and a public facing map developed using the web app building provided by arcgis (doi & lee, 2017b).    the records in the dataset are plotted on the map per ensemble location. this is defined as the place(s) where the ensemble was or is primarily located or with which it is associated. in some instances, an ensemble is associated with more than one location and appears more than once on the map. the map contains several layers, which can be switched on or off by the user. all ensembles can be displayed on the map using the “all ensembles – location points” layer, which pinpoints the locations (see figure 1).   using the “all ensembles – location clusters” layer displays the same information using a heat map, where concentration of ensembles is defined by variations in the colour (see figure 2).   figure 2 all ensembles – location clusters layer enabled.   information about the ensembles can also be displayed by ensemble type, by switching the orchestra, choir, or band layers, which displays the representation of each using coloured pins (see figure 3).   more detailed information about each album can be displayed by clicking on specific locations to reveal an information box or by opening an attribute table below the map. the displayed fields include album title, ensemble name, ensemble location, format, publication location, publication date, latitude and longitude coordinates, a url linking to the catalogue record, a url linking to the digitized album, and an image of the front cover. if more than one ensemble exists at the same coordinate, this is indicated in the upper portion of the information box. a search box is present in the mapping interface, which allows users to search by location (e.g., saskatoon, prince albert, or others) or place (e.g., university of saskatchewan, kinsmen hall, or others).     figure 3 choirs layer enabled and information box.     results   concentrations of musical activity   comparison of data from the arcgis world population estimate map and the smc heat map reveals a direct correlation between the number of albums produced and the size of a town or city (esri, 2016). the area measurement tool was used to determine the reach and concentration of the plotted musical recordings, showing that almost all the recordings are in the southern half of the province. this accounts for approximately 202,063 square kilometers (34.35%) of the total 588,239.21 square kilometers occupied by the province (statistics canada, 2015).   these findings also show clusters around larger centres in the province, such as saskatoon and regina, demonstrating a connection between city size and number of ensembles (see table 2).   enabling layers within the arcgis map shows the location of ensembles filtered by ensemble type. these findings show that choirs make up the largest percentage of ensemble recordings within the province, representing 77.31% of the total albums. bands make up the second largest percentage, representing 21.85%, and orchestras make up the lowest, representing 0.07%. in some of these cases, an album might contain music from more than one ensemble type, explaining the higher total number of ensemble representations overall than the total number of albums included in the dataset. when comparing the number of ensemble types to the location data, findings show that more concentrated centres such as regina and saskatoon are likely to contain all three ensemble types, while the choirs are more likely to appear in rural locations (see figure 4).     table 2 saskatchewan populations compared with number of albums per location city name city size (2011 census) (statistics canada, “saskatchewan population report”) number of albums per location saskatoon 222,189 41 regina 193,100 18 moose jaw 33,274 5 outlook 2,204 9 swift current 15,503 5     figure 4  concentration of large ensembles by genre: orchestras, bands, and choirs.     when ensemble types are compared alongside ensemble names using the table feature in arcgis, we can see trends between ensemble location and the responsible community organization. while orchestras are more likely to be associated with a city or university (e.g., university of saskatchewan wind orchestra and saskatoon junior symphony orchestra), bands were more often associated with a school, community organization, or military affiliate (e.g., aden bowman collegiate band, melfort kinsmen band, and royal canadian air force wing auxiliary band,). finally, choirs were associated with a wider variety of organizations, including schools, churches, universities, and community organizations (e.g., college park school choirs and rosthern mennonite school choir). the correlation between population size, available services or community organizations, and appearance of musical ensembles demonstrates a strong link between community development and musical activity within the province.   search interface   comparison of the three interfaces where the smc materials are now available for searching shows both the benefits and drawbacks of searching within the mapping tool. most notably, the map allows for non-text based interaction with the collection, while text searching remains more effective in either the catalogue or islandora, except in the case of searching by ensemble location (see table 3).   discussion   the findings of this paper provide an as-yet unseen picture into the history of large ensembles within saskatchewan. while the existing narrative histories provide more anecdotal perspectives on the musical activities, the visualization allows us to see the big picture of development and its reach within the province.   ties to local history   the ability to track the development of these three large ensemble groups within the province offers insight into the similarities and differences between them. the establishment of orchestras and bands typically relies on access to a variety of resources including specific instruments, instruction, rehearsal space, and musicians, which may have limited their overall numbers within the province. bands and orchestras may have needed community funding or financial support from community organizations, which could have limited their numbers in comparison to community choirs. meanwhile, choirs were well established in the tradition of church and school activities, and would have required fewer resources. as churches and schools had a welldeveloped and pervasive network within both urban and rural locations in the province, the development of community choirs throughout the province was inevitable.   musical activity within the province was one indicator of development, which can be tied to many others. comparing the data of musical recordings to other data related to community development or local history would provide a more complete picture of the establishment of such community organizations and individuals within them. church records, local histories, or provincial archives might have additional information on the formation and activity of musical ensembles.   while musical recordings are just one type of evidence of musical development and activity, it is nonetheless an important consideration of how music was produced, consumed and distributed. access to the album covers and liner notes often provides insight into the activities of the ensembles, including rehearsal schedules, tour dates and locations, individual members, and supporting members of the community. it is these connections to community and place that make these materials such an insightful window into local history.   testing a proof of concept   the geo-visualization was initially developed as a proof of concept. this varied slightly from the final design presented in this paper, as it used all the smc items found in the islandora platform (over 1000) plotted per publication location. the data analysis and visualization were realized using carto, a software that provides gis and web mapping tools for display in a web browser (https://carto.com/). the proof of concept was presented during the mountain plains chapter meeting of the music library association (bogdan & doi, 2016), and feedback was gathered on the initial design.     table 3 comparison of search interfaces: library catalogue, islandora digital repository, and arcgis map criteria catalogue islandora arcgis map text searching author, title, subject, keyword, call number keyword location visual representation and searching no no yes portion of the collection complete collection, 3000+ albums digitized items, 1000 albums digitized albums by large ensembles, 119 albums digital object included no yes – images and sound files yes – cover image information in record standard marc fields: 001, 003, 005, 007, 008, 028, 040, 090, 245, 260, 300, 500, 505, 590, 650, 700, 710, 740, 830, 850, 902 selected mods fields: artist, title, item type, table of contents, subjects, bib number, publisher, place of publication, item description selected marc and added fields: album title, ensemble name, ensemble location, format, publication location, publication date, decade, latitude, longitude, link to catalogue record, link to islandora record. link to other representations of the collection no no yes – links to both catalogue and islandora     the feedback revealed that the geo-visualization had provided an effective overview of the collection while also compelling users to explore selected albums. however, using the place of publication as the primary location for the recordings presented a challenge for users. while in some cases this geographic information was relevant to the artist or recording (e.g., the album was published in the same city as where the artist resided), in many others, the location of the music label was tangentially related, and provided only supplementary information as to how place relates to the artist or music. based on this feedback, the dataset was reconceived from june to august, 2016. these changes ultimately led to the decision to further limit the inclusion criteria of the dataset, and to incorporate information about the location of each ensemble.   challenges   using catalogue data as a dataset was challenging because historical cataloguing practices have led to inconsistencies due to changes in practice, variations in acceptable language, and human error. this inconsistency was not unique to this collection, and using library collection data will always require cleaning before use.   in this case, the music materials themselves also presented additional challenges. historical sound recordings offer inconsistent information on basic elements of the publication, and this inconsistency is amplified when many of the albums are independently-published as in this case. examples of this include missing publication information, or variations in spellings of titles or artists. while catalogue records are designed to accommodate for variations in spelling or missing information, these variations are more challenging when trying to achieve consistency in the dataset.   limitations   as the process for adding data to the map is not automated, it does not incorporate newly added digital items from the smc. as such, the map represents the collection at a moment in time, and significant staff time would be needed to add new content.   the dataset used for this project does not fully represent the recorded history of musical ensembles in saskatchewan. there are more known items located in the smc that were excluded because they have not yet been digitized. to achieve a more complete representation of the musical recording history, workflows for adding new content will need to be established.   future work   there are many possibilities for expanding the work initiated in this research, including expanding the dataset or adding additional types of information. there is also potential to realize other aspects of the dataset within the visualization, including the correlation between date and location.   while the data currently included is drawn from the smc, there are many other collections with materials related to the history of music and large ensembles in saskatchewan. special collections holdings at the university of saskatchewan or items in the saskatchewan history online project would provide a good starting place for locating related materials. such materials might extend the depth and breadth of the story of this history and would also make good candidates for inclusion in the project. saskatchewan’s citizens have produced more than 2000 local history books, which discuss the histories, lives, and activities of the town or city, chronicling the “rise (and in some cases, the decline and demise) of communities, families, churches, and businesses large and small across the province.” (massie, 2006). linking the information in these sources might serve to build more detailed narratives about the musical histories presented.   while the work to curate the dataset for this project was done by staff at the university, there may be opportunities to draw on local knowledge to enhance the data. since there are many aspects of local history that are missing from more traditional information sources, implementing a system for collecting music histories from the public might be a worthwhile option for developing the map further. by implementing a crowdsourcing information gathering model, this project would further serve to engage the public, to use the knowledge of the community, and to add value to the data (holley, 2010).   this project focusses on mapping large ensemble items from the saskatchewan music collection, but there is potential to extend the mapping template beyond this collection. while not all collections will be ideally represented in a geo-spatial environment, there are certainly others to consider that might provide insight when seen through a geographic lens.   conclusions   this research presents a picture of the history of musical life in the province of saskatchewan and its connection to historical social and economic development. findings show that there is a correlation between the population density of the province and the prevalence of large ensembles within communities. it provides evidence that these ensembles are often linked to community organizations and establishments such as schools, churches, service organizations, and arts organizations. as the province continues to develop we will certainly continue to see these connections between music and place continue to grow and change.   geo-spatial visualizations have potential as a tool to draw new research conclusions about the contents of library collections. researchers who are considering using catalogue data will find that it is a rich source of untapped data, which has potential for exploring local histories.   using an online mapping tool such as arcgis provides an alternative way to explore special collections, using parameters that are not accessible in interfaces such as the library catalogue or digital content repositories. these tools demonstrate potential to attract and engage a wide audience to explore library special collections and audiovisual materials.   acknowledgements   aspects of this work were originally presented at the 2016 mountain plains chapter meeting of the music library association in greeley, co and later at the centre for evidence based library and information practice symposium in saskatoon, sk in 2016. my thanks to k. lee who collaborated with me on the implementation of this project. thank you to my colleagues at the university of saskatchewan: c. sjoberg for her assistance with cleaning and enhancing the dataset and j. mclean who offered helpful feedback on openrefine and cleaning library catalogue data.   references   ahlberg, c., & shneiderman, b. (1993). visual information seeking: tight coupling of dynamic query filters with starfield displays. in s. k. card, j. d. mackinlay, & b. shneiderman (eds.), readings in information visualization: using vision to think. (pp. 244–250). san francisco, ca: academic press.   bogdan, k. & doi, c. (2016, may). music in space and time: using data visualization to understand a local music collection. paper presented at the mountain plains chapter meeting of the music library association in greeley, co.   buchel, o. & sedig, k. (2011). extending map-based visualizations to support visual tasks: the role of ontological properties. knowledge organization, 38(3), 204-229.   buchel, o., & sedig, k. (2014). making sense of document collections with map-based visualisations: the role of interaction with representations. information research, 19(3), 29-55. retrieved from http://www.informationr.net/ir/19-3/paper631.html#.wdxnufkrk00   conway, k. (2006). saskatoon symphony orchestra. in the encyclopedia of saskatchewan. retrieved from http://esask.uregina.ca/entry/saskatoon_symphony_orchestra.html   crampton, j. w. (2002). interactivity types in geographic visualization. cartography and geographic information science, 29(2), 85-98. http://dx.doi.org/10.1559/152304002782053314   deal, l. (2014). visualizing digital collections. technical services quarterly, 32(1), 14-34. http://dx.doi.org/10.1080/07317131.2015.972871   department of music. (2015). about ensembles. in university of saskatchewan, college of arts & science. retrieved from http://artsandscience.usask.ca/music/ensembles/   doi, c. (2015). local music collections: strategies for digital access, presentation, and preservation—a case study. new review of academic librarianship, 21(2), 256-263. http://dx.doi.org/10.1080/13614533.2015.1022663   doi, c. and lee, k. (2017a). choirs, bands, and orchestras of saskatchewan v1. retrieved from https://spatialsk.maps.arcgis.com/apps/webappviewer/index.html?id=de483307f29641a2b7682c3f09d517b8   doi, c. and lee, k. (2017b). saskatchewan music large ensembles. in university of alberta libraries. http://dx.doi.org/10.7939/dvn/10974   esri. (2016). world population estimates. retrieved from: http://arcg.is/2jf68ql   esri. (n.d.). about arcgis. retrieved from http://www.esri.com/arcgis/about-arcgis   europeana. (n.d.). travelling through time with sounds. retrieved from http://www.europeana.eu/portal/en/collections/music/travelling-through-time-with-sounds.html   finch, j. l., & flenner, a. r. (2016). using data visualization to examine an academic library collection. college and research libraries, 77(6), 765-778.  http://dx.doi.org/10.5860/crl.77.6.765 gonzález ribot, m. j. (2015, june). map of spanish musical heritage. presented during the iaml/ims annual meeting, music research in the digital age, 21-26. june 2015, juilliard school, new york, ny. retrieved from http://www.iaml.info/sites/default/files/pdf/map_of_spanish_musical_heritage_gonzalez_ribot_iaml_2015.pdf   google developers. (2017). geocoding api: getting started. retrieved from https://developers.google.com/maps/documentation/geocoding/start?csw=1   hilliges, o., holzer, p., klüber, r., and butz, a. (2006). audioradar: a metaphorical visualization for the navigation of large music collections. in a. butz, b. fisher, a. krüger, & p. olivier (eds). smart graphics, sg 2006. lecture notes in computer science, vol. 4073. (pp. 82-92). http://dx.doi.org/10.1007/11795018_8   hoek, w. van, & mayr, p. (2014). assessing visualization techniques for the search process in digital libraries. in s. a. keller, r. schneider, & b. volk (eds.), wissensorganisation und-repräsentation mit digitalen technologien. (pp. 63–85). berlin: de gruyter saur. retrieved from http://nbn-resolving.de/urn:nbn:de:0168-ssoar-463733   holley, r. (2010). crowdsourcing: how and why should libraries do it? d-lib magazine, 16(3/4). http://dx.doi.org/10.1045/march2010-holley   islandora (n.d.). about. retrieved from https://islandora.ca/about   kilb, m., & jansen, m. (2016). visualizing collections data: why pie charts aren't always the answer. serials review, 42(3), 192-200. http://dx.doi.org/10.1080/00987913.2016.1207479   krygier, j. b. (1994). sound and geographic visualization. in a. m. maceachren and d. r. f. taylor (eds). visualization in modern cartography. (pp. 149-166). new york, ny: elsevier. http://dx.doi.org/10.1016/b978-0-08-042415-6.50015-6   leitich, s., & topf, m. (2007). globe of music: music library visualization using geosom. retrieved from http://ismir2007.ismir.net/proceedings/ismir2007_p167_leitich.pdf   lewis, m. (1973). music in saskatchewan. caml review, 2(1), 6-7. http://caml.journals.yorku.ca/index.php/caml/article/view/4148/3347   lowery, r. (2011). a visualization tool for atlas collection assessment. journal of map & geography libraries, 7(2), 138-153. http://dx.doi.org/10.1080/15420353.2011.566837   magas, m. & p. proutskova. (2013). a location-tracking interface for ethnomusicological collections. journal of new music research, 42(2), 151-160. http://dx.doi.org/10.1080/09298215.2013.821999   massie, m. (2006). local history. in the encyclopedia of saskatchewan. retrieved from http://esask.uregina.ca/entry/local_history.html morgan, m. (n.d.). what about choral art music from the prairies?: an introduction to canadian prairie composers and their compositions for satb choir. retrieved from https://drive.google.com/file/d/0b1njvb97apfglwthemw1eli3m3m/view?usp=sharing   moscovitz, m. (2015). openrefine: geocoding. retrieved from https://github.com/openrefine/openrefine/wiki/geocoding   pampalk, e., rauber, a., & merkl, d. (2002). content-based organization and visualization of music archives. in multimedia ’02: proceedings of the tenth acm international conference on multimedia. (pp. 570-579). new york, ny: acm press. http://dx.doi.org/10.1145/641007.641121   sask history online. (n.d.). about the project. retrieved from http://saskhistoryonline.ca/about   saskatchewan bureau of statistics. (2012). saskatchewan population report: 2011 census of canada. retrieved from http://www.stats.gov.sk.ca/stats/population/censuspop2011.pdf   shiri, a. (2008). metadata-enhanced visual interfaces to digital libraries. journal of information science, 34(6), 763-775. https://doi.org/10.1177/0165551507087711   schumiatcher, m. & luko, a. r. (2016). saskatchewan arts board. in the canadian encyclopedia. retrieved from http://www.thecanadianencyclopedia.ca/en/article/saskatchewan-arts-board-emc/   smithsonian folkways holiday music map. (2013). retrieved from http://arcg.is/2jhgrke   statistics canada. (2015). focus on geography series, 2011 census. retrieved from http://www12.statcan.gc.ca/census-recensement/2011/as-sa/fogs-spg/facts-pr-eng.cfm?lang=eng&gc=47   swales, r. (2006a). choirs. in the encyclopedia of saskatchewan. retrieved from  http://esask.uregina.ca/entry/choirs.html   swales, r. (2006b). music. in the encyclopedia of saskatchewan. retrieved from http://esask.uregina.ca/entry/music.html   symes, c. (2004). setting the record straight: a material history of classical recording. middletown, ct: wesleyan university press.   torrens, m., hertzog, p., and arcos, j. l. (2004, oct.). visualizing and exploring personal music libraries. paper presented at the 5th international conference on music information retrieval, barcelona, spain, 10-14 october 2004. retrieved from http://www.ee.columbia.edu/~dpwe/ismir2004/crfiles/paper214.pdf   wall, k. (2006). regina symphony orchestra. in the encyclopedia of saskatchewan. retrieved from http://esask.uregina.ca/entry/regina_symphony_orchestra.html    ward, j. (2015). an interactive map of musical institutions in belgium. retrieved from http://www.iaml.info/news/interactive-map-musical-institutions-belgium    ward, n. (2009). saskatchewan (province). in the canadian encyclopedia. retrieved from http://www.thecanadianencyclopedia.ca/en/article/saskatchewan/   wasiak, e. b. (2000). school bands in saskatchewan, canada: a history. journal of historical research in music education, 21(2), 112-131. retrieved from http://www.jstor.org/stable/40215213   wicentowski, j. (2015). grel functions. retrieved from https://github.com/openrefine/openrefine/wiki/grel-functions   yale university library. (n.d.a). rda descriptive examples (1). retrieved from http://web.library.yale.edu/cataloging/sound-recordings/music-popular/examples   yale university library. (n.d.b). 300 $a (sound recordings, rda). retrieved from http://web.library.yale.edu/cataloging/sound-recordings/non-music-checklist/300   evidence summary   usage volume and trends indicate academic library online learning objects and tutorials are being used   a review of: hess, a. n., & hristova, m. (2016). to search or to browse: how users navigate a new interface for online library tutorials. college & undergraduate libraries, 23(2), 168-183. http://dx.doi.org/10.1080/10691316.2014.963274   reviewed by: ruby warren user experience librarian university of manitoba libraries winnipeg, manitoba, canada email: ruby.warren@umanitoba.ca   received: 1 dec. 2016    accepted: 17 jan. 2017      2017 warren. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to discover how users interact with a new online interface for learning objects, user preferences for types of access when given both browsing and searching options, and user needs for tutorial subject matter.   design – mixed methods, with quantitative analysis of web traffic and qualitative analysis of recorded search terms through grounded textual theory.   setting – an academic library in the western united states of america.   subjects – users of the libraries’ online tutorials and learning objects.   methods – the researchers collected web traffic statistics and organically occurring searches from the libraries’ tutorial access interface. they defined the collection period as the 2013/2014 academic year, with collection beginning in september 2013 and ending in april 2014. web traffic for organic searches, facilitated searches (search results accessed through clicking on particular words in a tag cloud), and categorical browsing was collected via google analytics. they categorized other interaction types (accessing featured content, leaving the page, etc.) under an umbrella term of “other.” their analysis of web traffic was limited to unique page views, with unique page views defined as views registered to different browser sessions. unique page views were analyzed to determine which types of interface interaction occurred most frequently, both on-campus and off-campus, and whether there were differences in types of interaction preferred over time or by users with different points of origin. individual organic search keywords and phrases, and the dates and times of those searches, were separately collected and recorded. one of the researchers coded the recorded organic search terms using grounded textual theory analysis, and the researcher formed generalized categories. they sent these categories and a random sample of 10% of the recorded search terms to librarians unaffiliated with the study, and used their categorizations of the search term samples to validate the initial researcher’s textual analysis.   main results – after analyzing the 5,638 unique page views recorded, researchers found that categorical browsing was used more frequently than facilitated searching throughout the year, and more frequently than organic searching for 6 of the 8 recorded months. organic searching was used more frequently than facilitated searching during most months, while both organic and facilitated searching were less likely to be engaged in by users working on saturday or sunday. they found that interactions in the “other” category were quite high, and the researchers attributed this to featured videos on the interface homepage being required for a number of classes. the researchers discovered that patterns in interface use were similar between on-campus and off-campus users, and that most traffic to the interface was through referral from other websites (such as the library homepage). direct traffic (from urls manually typed in or in documents) was the second most frequent point of access, while users arriving at the interface from a search engine interaction was a distant third. grounded textual theory analysis of the 14,428 collected organic searches achieved a 92% consensus in coding, and showed a user focus in searching for specific resources, tasks, and knowledge, rather than broader conceptual searches. additionally, researchers noticed that a significant number of users performed organic searches for videos that were featured on the front page, possibly indicating that certain users engage with search functions before viewing page content.   conclusions – the researchers concluded that despite the limitations of the study, the usage volume and trends identified indicate that the libraries’ online learning objects and tutorials are being used. they also concluded that the categorization and labelling of these learning objects has been successful because the categorical browsing function is used more than the other search functionalities. the researchers determined that they should consider the non-user in the future, and examine the barriers that students, faculty, and staff encounter when attempting to use online learning content. they affirm a need to develop, via further studies, a more thorough understanding of the motivations behind user interactions.   commentary   there is extensive literature available on web interfaces, usability, and online learning. the authors reference literature in each of these areas, much of it informing the way their online learning materials were created or organized. library and information science and user experience literature also contains a broad knowledge base of the areas of particular concern in this article: user interface usability and user needs. although studies of particular web interfaces and their relative effectiveness are fairly common, the questions posed in this study could have broad applicability to any library or learning institution providing online, self-serve instructional materials.   analyzing recorded organic search terms with grounded textual theory analysis was an appropriate method to draw initial conclusions about user preferences and needs regarding learning object and tutorial subject matter. the methods used in this study to evaluate usage statistics provided an overall picture of the ways users choose to interact with the interface, although a fuller understanding could have been obtained with alternate research methods. however, the authors do not justify or explain how usage statistics regarding page access and methods of interaction could reliably indicate the success of an interface design, and their conclusions in these areas do not logically follow from the study results. the fact that features like categorical browsing interfaces are being used does not necessarily prove that these features have been designed well or intuitively. alternate methods, such as interviews, usability studies, or heuristic evaluations, would better serve to answer these study objectives.   this study scored a 75% overall validity rating using glynn’s critical appraisal tool for library and information science research (2006), chiefly due to section c study design (rated 25%) and section d results (60%). these low rankings were due to aforementioned problems with the chosen methodology matching study conclusions. additionally, the authors also do not provide detailed information regarding the final categorizations used in their analysis of search keywords and phrases, which limits our understanding of those results.   the stated objectives of this study are met, and readers interested in examining how users at this institution have used the interfaces available to them will find interesting and relevant information, but researchers interested in the conclusions the authors attempt to draw would be better served by work using a different methodology.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154   evidence summary   academic medical library services contribute to scholarship in medical faculty and residents   a review of: quesenberry, a. c., oelschlegel, s., earl, m., leonard, k., & vaughn, c. j. (2016). the impact of library resources and services on the scholarly activity of medical faculty and residents. medical reference services quarterly, 35(3), 259-265. http://dx.doi.org/10.1080/02763869.2016.1189778   reviewed by: peace ossom williamson director for research data services university of texas at arlington libraries arlington, texas, united states of america email: peace@uta.edu   received: 10 mar. 2017  accepted: 25 apr. 2017      2017 williamson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to assess the impact of academic medical library services and resources on information-seeking behaviours during the academic efforts of medical faculty and residents.   design – value study derived from a 23-item survey.   setting – public medical residency program and training hospital in tennessee, united states of america.   subjects – 433 faculty and residents currently employed by or completing residency in an academic medical centre.   methods – respondents completed a 23-question survey about their use of library resources and services in preparation for publishing, presenting, and teaching. the library services in the survey included literature searches completed by librarians and document delivery for preparation of publications, presentations, and lecture material. the survey also included questions about how resources were being accessed in preparation for scholarship. the survey sought information on whether respondents published articles or chapters or presented papers or posters in the previous three years. if respondents answered in the affirmative to one of the aforementioned methods of scholarship, they were provided with further questions about how they access library resources and whether they sought mediated literature search and document delivery services in preparation for their recent presentations and publications. the survey also included questions concerning what types of scholarly activity prompt faculty and residents to use online library resources.   main results – the study was provided to 433 subjects, including 220 faculty and 213 residents, contacted through an email distribution list. the response rate to the survey was 15% (n=65). residents comprised 35% of the respondents, and faculty at each of the three levels of tenure comprised 60%. the remaining 5% of respondents included phd and non-clinical faculty within the graduate school. over 50% of respondents reported use of library services in preparation for publishing and presenting. these library services were literature searches, document delivery, and accessing online resources. faculty and residents reported use of pubmed first (71%) and most often, with 56% of respondents reporting weekly use, followed by google or google scholar, with 20% of respondents reporting its use first and 23% of respondents reporting weekly use.   however, regarding responses to the question concerning how journal articles are accessed, “using a search engine” was chosen most often, at almost 65%, followed by (in order) clicking library links in a database, contacting the library directly, searching the list of library e-journals, clicking publisher links in a database, using personal subscriptions, searching the library catalog, and using bookmarks saved in a web browser. based on survey responses, faculty reported higher use of library services and resources than residents; however, residents reported higher use of library services and resources when preparing posters and papers for conferences and professional meetings. in addition, several comments spoke to the importance of the library for scholarly activity, many indicating the critical role of library assistance or resources in their academic accomplishments.   conclusion – this study provides evidence in support of library resources and services for medical faculty and residents, which contributes to discussions of the contributions of medical libraries. as hospital libraries close and academic medical libraries see reductions in budgets, this study contributes to the value of a library’s presence, as well as the role of the health sciences librarian in medical research and scholarly communication. this academic medical library was reported to be first and most often used, in comparison with other resources or none, in preparation for publication and presenting. the results of this and similar studies can contribute to the generalizability of its findings relating to the value of medical libraries. in addition, pubmed, uptodate, and google were the resources used most often by respondents, along with search engines and library links in databases. these findings can be incorporated into future outreach, marketing, and instructional curriculum for this library’s users. the survey results also provide additional support for the library’s role in the academic research lifecycle, and free-text comments about the critical role of library services furthered those findings. the authors state that further research is necessary for improving awareness of library resources and services in the role of scholarship at institutions.   commentary   the assessment of library integration in key areas of academic medical centres is integral to exhibiting ongoing value and to determining areas of potential growth. assessment of library use varies largely in methods of research on this topic. in other research, libraries are analyzing patron data (nackerud, fransen, peterson, & mastel, 2013) and patrons’ journaling of the resources used over a period of time (brennan et al., 2014), as well as using many other methods. in the case of this study, user response to survey questions provided the data used to analyze library impact.   the study was evaluated using the cristal checklist for appraising a user study (n.d.). strengths of the study include the use of a clearly focused issue (i.e., the evaluation of library services and resources use by medical faculty and residents at an academic medical centre). the authors of this study also used the rochester study (marshall, 1992) as a benchmark to which they compared methods and results. data collection was described in detail along with findings; however the questions in the survey show signs of bias. for example, library resources at that particular campus were described, but not the resources of other libraries to which their users may have access.   in this study, there were no interventions implemented, but there were clear outcomes defined, including the frequency, prevalence, and ranking of online resources, the use of literature searches by librarians, and document delivery. however, conclusions from self-reported survey responses may be limited in usefulness, as gross and latham (2009) found that undergraduate students demonstrating poor information literacy skills greatly overestimated their proficiency before and after being tested. therefore, the research would benefit from follow-up studies evaluating library interactions through patron data, including information from the library management system, interlibrary loan, proxy system reports, and other data points with patron characteristics tied directly to library services.   the results of the study were described in detail and are likely repeatable, but the study did not mention any bias, limitations, or further analyses that could be completed. the study had a low response rate; incentives may be of benefit for future studies to encourage broader participation. the use of incentives may also reduce submission bias, or a higher rate of responses from heavy library users than non-library users. further follow-up of the data in this study could include quantitative analysis of the relationship between the independent and dependent variables as well as statistical significance. follow-up involving the integration of other data points, as previously mentioned, would also strengthen and help verify the results of this study.   references   brennan, n., edwards, s., kelly, n., miller, a., harrower, l., & mattick, k. (2014). qualified doctor and medical students’ use of resources for accessing information: what is used and why? health information & libraries journal, 31(3), 204-214. http://dx.doi.org/10.1111/hir.12072   cristal checklist for appraising a user study. (n.d.) in nettingtheevidence.pbwiki.com. retrieved from http://nettingtheevidence.pbwiki.com/f/use.doc   gross, m., & latham, d. (2009). undergraduate perceptions of information literacy: defining, attaining, and self-assessing skills. college & research libraries, 70(4), 336-350. http://dx.doi.org/10.5860/crl.70.4.336   marshall, j. g. (1992). the impact of the hospital library on clinical decision making: the rochester study. bulletin of the medical library association, 80(2), 169-178. retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/pmc225641/   nackerud, s., fransen, j., peterson, k., & mastel, k. (2013). analyzing demographics: assessing library use across the institution. portal: libraries and the academy, 13(2), 131-145. http://dx.doi.org/10.1353/pla.2013.0017 microsoft word art_mcknight.doc evidence based library and information practice 2007, 2:4  22 evidence based library and information practice     article    acquisition and cataloguing processes: changes as a result of customer value  discovery research      sue mcknight  director of libraries and knowledge resources  boots library, nottingham trent university  nottingham, united kingdom  e‐mail: sue.mcknight@ntu.ac.uk      received: 29 july 2007    accepted: 05 november 2007      © 2007 mcknight. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    abstract    objective ‐ this study seeks to highlight the profound effect of customer value discovery  research on the internal business processes of two university libraries in the areas of  cataloguing and acquisitions.     methods ‐ in this project, “customer discovery workshops” with academic staff, students,  and university stakeholders provided library managers and staff with information on what  services and resources were of value to customers. the workshops also aimed to discover  what features of existing library services and resources irritated the students, staff, and  faculty.   a student satisfaction survey assessed longer‐term impact of library changes to  students in one university.    results ‐ the findings resulted in significant changes to collection development,  acquisitions, and cataloguing processes. a number of value added services were introduced  for the customer. the project also resulted in greater speed and efficiency in dealing with  collection development, acquisitions, and cataloguing by the introduction of more  technology‐enhanced services. overall customer satisfaction was improved during the  project period.      conclusion ‐ the changes to services introduced as a result of customer feedback also  improved relationships between librarians and their university community, through the  introduction of a more proactive and supportive service.    mailto:mcknight@ntu.ac.uk http://creativecommons.org/licenses/by/2.0 http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:4  23   introduction  the role of a library director is to establish a  vision, based on knowing where a service  ought to be heading, and then to allow staff  to make the vision become a reality. this is  easily said, but quite a bit harder to  implement. what should our vision be?   this paper describes how customer  feedback has been collected and used as the  basis for establishing a vision of excellence  for the library.    this paper describes a case study where  customer feedback was used to effect  changes in acquisitions and cataloguing  processes in academic libraries. the case  study draws on the results and changes  made over a number of years following  customer value discovery research  undertaken in two universities (deakin  university library, australia and  nottingham trent university, uk). the  paper builds on a presentation to the  ‘exploring acquisitions conference’ held in  cambridge in april 2007, and it focuses on  those actions that resulted from the research  that impacted on acquisitions and  cataloguing.     enzyme international  (<http://www.enzymeinternational.com.au/>)  served as an external facilitator and  provided analysis of data gathered from the  customer value discovery (cvd) research  used to drive change at both universities.    objectives  the aim of the research was to gather data  that would be used by library management  to ensure that service and resource delivery  within the organisation are aligned to the  actual needs of the customer. evidence from  the research regarding perceptions of  current value delivered, when compared to  what customers desire, provides a gap  analysis.      by ascertaining hierarchies of ‘value’ and  ‘irritation,’ priorities for action can then be  developed to inform operational planning,  service standards, key performance  indicators, and individual work objectives.    over time, after implementing changes  based on results of the customer value  discovery research, it will be possible to re‐ test to determine whether the service is  increasing customer value, reducing  customer irritation, and closing the gap  between actual and desired performance  against the original datasets.     methods  the customer value discovery process  researchers used customer value  discovery (cvd) methods to gather data on  customer needs.  cvd is based on a model  known as the hierarchy of value (albrecht,  2000). its continuum describes the potential  experiences of interactions between  customers and service providers.     in a service industry, such as a library, all  interactions, either with a staff member, a  resource (e.g., book), or a service (e.g., web  page) can both satisfy and irritate a  customer at the same time. the aim is to  consistently deliver on the basic and expected  services, and to deliver desired and unexpected  services (but not at the expense of neglecting  the basic/expected services) and doing these  without causing irritation to the customer.  while simple to explain, it is not so easy to  implement on a consistent basis, for all  services, at all campuses.        http://www.enzymeinternational.com.au http://www.enzymeinternational.com.au evidence based library and information practice 2007, 2:4  24     hierarchy of value unanticipated desired expected basic frustrated irritated angry     figure 1. hierarchy of value ©: enzyme international (aust.)    customer discovery workshops  the process of customer value discovery  research involves holding ‘customer  discovery’ workshops with  customer/market segments (e.g.  undergraduate on‐campus students, post‐ graduate researchers, distance learners,  mature students, or academic staff). the  goal of the workshops is first to ascertain  what irritates these customer groups about  existing services and resources and, second,  to determine how they would define an  excellent service.  the participants in the  workshops are then asked to rate their  perceptions of the current services against  the ideal service levels.     analysis and consolidation  following the customer discovery workshop,  which is held in the morning, a facilitated  analysis workshop is held in the afternoon  to thematically arrange the irritants that  were identified at the customer value  discovery workshop. once all customer  value discovery workshops have been held,  a consolidation workshop is conducted  with all the staff who observed the  workshops, plus library managers. at both  the analysis and consolidation workshops,  staff delve into the individual customer  comments that have been thematically  arranged and assigned a topic classification.  this process allows everyone to be clear of  the meanings and intentions, even if they  did not personally attend a particular  workshop.    interactive value modelling  through a process called interactive value  modelling, all the staff observers and  manager participants use mathematical  modelling software to model how value (or  satisfaction) is improved and irritations  reduced if certain actions are taken.  by  using the interactive value modelling  technique, priorities for further action can be  established, based on top‐level assumptions  about how to respond to customer feedback.   mcknight describes further details about  interactive value modelling.  evidence based library and information practice 2007, 2:4  25 customer value discovery preparation segmentation & study design customer discovery workshops per segment analysis workshops build value model consolidation interactive value modelling & action planning “customer” refers to a target group quick wins ongoing measurement   figure 2. customer value discovery process (enzyme international)    this research was conducted between 1996  and 2003 at deakin university, a multi‐ campus higher education institution with  libraries in melbourne, geelong and  warrnambool, australia with a number of  different customer segments. it was  repeated in 2005 at nottingham trent  university (ntu) in the u.k. with  undergraduate on‐campus students and  academic staff. over 70 students and 50  academic staff participated in the  workshops held on ntu’s three campuses.  one‐on‐one interviews with academic staff  were conducted at the smallest campus,  using the same workshop methodology and  workbook.    longer‐term impact was assessed at ntu  using a student satisfaction survey. in 2005  3,492 students completed the survey; and  another group of 5,611 students completed  the survey in 2007. these surveys, the first  held two months after the customer value  discovery research was undertaken, enabled  comparison of the data from the cvd  process with the formal survey. there was  close alignment in results between the two  groups; however, the cvd process provided  more detailed data on which to act. the  ntu student satisfaction surveys also  enabled a longitudinal study to ascertain  whether satisfaction had improved as a  result of initiatives introduced after the cvd.    results   results of the workshops held at both  universities are combined in the following  tables in order to illustrate the type of  evidence obtained and how it was used.      values and irritations  the customer value discovery research  process provides hierarchies of students’  values and irritations to help with decision  making. below are examples, however, the  identification of the particular library is not  given.  the highest value or irritation is given a  value of 100%, and the others are expressed  as percentages of the highest ranked item.  graphical examples of the hierarchy of   evidence based library and information practice 2007, 2:4  26     figure 3. hierarchy of irritants, students         figure 4. hierarchy of irritants, students, gaps    hierarchy of irritants students low relative importance high 0 % 50 % 100 % poor staff service poor physical environment unsatisfactory loan rules poor communication and feedback inadequate library skills training difficulties with printing and photocopying problems with the catalogue service delays limited hours of access difficulties with technology and electronic access collection inadequacies materials not where they're supposed to be hierarchy of irritants students gaps poor staff service poor physical environment unsatisfactory loan rules poor communication and feedback inadequate library skills training difficulties with printing and photocopying problems with the catalogue service delays limited hours of access difficulties with technology and electronic access collection inadequacies materials not where they're supposed to be low relative importance high 0 % 50 % 100 % observers students evidence based library and information practice 2007, 2:4  27   values  irritations  easy access to materials where and when i  need them.  cannot find materials i need.   problems with the catalogue.     resources are comprehensive, available, and  relevant.  inadequacy of the collection and its  management.    academic staff and librarians have a  proactive partnership.   inadequate academic liaison and  communication.    lack of training and support when i need it.    reliable up‐to‐date technology and facilities  are available.  unreliable, limited information technology  facilities.    library hours of operation meet user needs  (physical access and availability of electronic  services).    library operating hours are inadequate.     user friendly loans policies and procedures.  restrictive and difficult loan policies and  practices.    table 1. consolidated values and irritations    irritations and the hierarchy of irritation –  gaps are presented in figures 3 and 4.   figure 3 shows that the lowest rated  irritation for students was poor staff service,  and the highest rated irritation resulted  from materials not being where they were  supposed to be. the gaps charts show the  differences between how customers voted  and how the staff assumed the customers  would vote.      figure 4 illustrates that the views of the  librarian observers differed from the views  of the students in a majority of areas.  for  example, library staff believed that poor  staff service and difficulties with printing,  photocopying, and electronic access would  be more irritating than students actually  perceived them to be.  in contrast, students  were most irritated by materials not being  where they were supposed to be.  changing acquisitions and related  processes  with regard to the changing acquisition and  cataloguing processes from the two  university libraries, table 1 contains the  consolidated list of relevant values and  irritations    values and irritations related to loan  policies and physical library opening hours  are related to acquisitions and cataloguing  issues. this is because restrictive loans of  short loan and reserve collection items and  decisions about whether to acquire multiple  copies of in‐demand books, or whether to  provide digital access to resources are  determined by acquisitions and cataloguing  policies and practices.    basic transactions  the responses to this customer feedback are  also a combination of actions undertaken (or  evidence based library and information practice 2007, 2:4  28 planned) as a result of the research. not  every action mentioned can be attributed to  both libraries.    referring to the diagram of hierarchy of  value (figure 1), staff identified the need to  provide a consistent level of service for all  customers on all campuses. it was evident  that some inconsistency in service was due  to the availability of staff on a particular  campus. irritations were tied to problems of  maintaining workflows at busy times, and  failure to do so resulted in backlogs (e.g.  orders not processed; reading lists not  checked) or failure to deliver services (e.g.  alerts for new books in a given discipline). it  was decided to automate certain basic  transactions that would provide valuable  staff time to undertake planning and  delivery of value‐added services.  automating certain transactions also helped  to reduce identified irritations.     automation of basic transactions  table 2 summarizes the automated services  implemented following the cvd process.  it  was estimated at one of the libraries that the  new services replaced 70%‐80% of the work  in a traditional library ‘technical services’  department.     by changing the way these transactions and  services were delivered, the library service  was able to deliver a consistent level of      initiative  outcome  online selection basket for  collection development  this enabled academic staff to use online selection  services and to easily identify items for purchase  consideration. the request lists were automatically sent  to acquisitions, saving time and providing accurate  machine‐readable records for creation of orders.    electronic data interchange (edi)  for ordering, invoicing, and  payments  the introduction of edi enabled significant efficiency  gains. the elimination of: paper records, postage and  handling costs, time delays, and manual payments of  orders benefited the library, the finance department,  and the vendor. it also sped up the acquisitions process,  reducing the time from order to shelf.    copy cataloguing  accepting copy cataloguing records with book items  reduced time in the cataloguing department, sped up  delivery of new items to the shelves, and freed  cataloguers’ time for other work. cataloguers were able  to use the extra time to focus on cataloguing grey  literature, thus making more items accessible to  customers.    shelf‐ready books  transferring this end‐processing activity to the supplier  eliminated a basic, but non‐value added activity from  the technical services operations.      exception claiming rather than  manually checking everything  with a highly automated acquisitions system, it is  possible to let the system do the order checking.  this  evidence based library and information practice 2007, 2:4  29 required a modicum of cultural change, as the past  practices of checking were entrenched, and staff had to  learn to ‘let go.’  however, the standard of service did  not slip.    system reporting and statistics   cessation of maintaining manual spreadsheets freed up  time. system generated reports and graphs were  regarded as trusted.    vendor records used as order slips  academic staff did not have to complete order forms;  the order slips had the needed information; and  acquisitions staff no longer had to decipher difficult  handwriting.    system alerts for new publications  academic staff and liaison librarians were pushed  information on new items, thus removing their need to  schedule dedicated time for checking.    filters used to create individual  staff profiles  academic staff and liaison librarians were pushed  information on new items, thus removing their need to  schedule dedicated time for checking.    approval plans; blanket orders;  standing orders  these provide quick ways to ensure ordering of  important new works. they eliminate the chance of  missing a ‘must have’ publication that matches the  library’s profile, and they save significant selection time  when the library collects ‘everything’ in a specific  discipline/genre/category.    ability to track orders online  academic staff and liaison librarians can check the  status of their orders without requiring acquisitions  staff intervention.    new order lists/books just received  lists  academic staff and liaison librarians can be advised of  new receipts without requiring acquisitions staff  intervention.    table 2. initiatives implemented following the cvd process    service, with reduced timeframes, without  requiring additional staffing resources.  many of the irritations that had been  identified were eliminated, thus improving  customer satisfaction.  some innovations,  such as electronic data interchange (used  for processing digital orders, invoices, and  payments), revolutionised the handling of  what had previously been manual and time‐ consuming transactions.  however, a  number of changes had to be negotiated  with other parts of the organisation (e.g., in  the case of edi, the library had to work with  the university finance department and the  internal auditor, to ensure appropriate  checks and balances and an audit trail  evidence based library and information practice 2007, 2:4  30 would be maintained) and with library staff.  cataloguers, in particular, were involved in  decision‐making regarding copy  cataloguing, and liaison and acquisition  librarians were involved with changes to  blanket orders and approval plans.      value adding ‘wow’ services  implementing many automated backroom  processes freed professional time both for  technical services staff and for academic  liaison. as a result, value added services  were implemented without requiring  additional resources.     new value added services resulted in the  implementation of a more proactive  collection development programme. vendor  lists were utilized to aid recommendations  for new orders, and vendor alert messages  helped staff learn about new publications.  this reduced an irritant of ‘not having time  to search for new items’ and also made  collection development more efficient.  filters that delivered advice on new  publications in identified disciplines,  combined with greater use of approval  plans and blanket orders, meant that the  collection was enhanced with little  additional effort for the academic staff or  liaison librarians.    a major thrust in both libraries concerned  changes in handling short‐term loan items  and recommended reading lists. resulting  innovations focused on digitising short loan  items (while conforming to relevant  copyright laws), thus making these items  available 24/7 to students and removing  their need to visit the library in person. the  students were delighted with the improved  service, with links imbedded from the  virtual learning environment (vle) as well  as from the library catalogue and reading  list module. the changes also provided  liaison librarians with more time in their  working days, and enabled them to conceive  and implement projects relating to  improving access to reading list material. in  addition, the physical space of the short loan  collection was significantly reduced, freeing  space for other purposes. the labour  requirement for managing the reserve items  was significantly reduced, freeing staff  resources for other value added endeavours.     students, in particular, were demanding  more digital resources in response to  making information resources available  ‘when and from wherever i need them.’   undergraduate students identified the need  for more full‐text journal articles, rather than  additional digital bibliographic citation  services. they also wanted to make the  searching of the libraries’ myriad e‐journal  collections more user‐friendly. the result  was not only more full‐text e‐journals, but  also the implementation of federated  searching solutions that enabled the  majority of digital resources to be cross‐ searched with a single search strategy.  further, linking the full‐text search results  with licensed resources provided by the  library significantly increased access to  valued information, thus adding value for  customers (and reducing irritation). adding  links to full‐text resources also maximized  asset utilization, in accounting terms, by  increasing the use of resources already  available to library customers.    the implementation of e‐book platforms  and services has also resulted in a value  added service, in that books, as well as  journal articles, can be available 24/7. ntu  is participating in a trial (uk national e‐ books observatory project) that provides a  number of online textbooks to the uk  higher education community and provides  significantly improved access to the chosen  textbook resources in four nominated  disciplines for the duration of the trial (jisc).       another value added service introduced  several years ago was the introduction of  enhanced cataloguing records with table of  evidence based library and information practice 2007, 2:4  31 contents information and thumbnail images  of book dust jackets. this innovation  resulted in contents of books and conference  proceedings, in particular, being more  accessible to customers, because catalogue  access had been expanded to include  chapter headings and chapter authors.      copy cataloguing originally reduced work  for librarians, but additional value added  services have since been identified that use  librarians’ new‐found time. instead of  focusing expertise on aacrii cataloguing  rules, librarians’ skills were broadened to  incorporate the application of metadata  standards for digital object repositories, a  new value added service that fulfills an  unmet demand of academic staff customers,  in particular. in addition, as mentioned in  the table of basic services, leveraging  original cataloguing expertise to make  special collection resources and grey  literature accessible provided a value added  service that previously could not be readily  afforded. not only did customers gain  access to valuable resources via institutional  digital repositories and special collections,  the institution increased its reputation  through its scholarly, unique, and research‐ based collections.    a final value added service related to the  impact of actions resulting from the  customer value discovery research, has been  in the area of knowledge and information  management. from the experience of  listening to customers’ desires and needs,  and providing space for, and encouraging,  strategic thinking, the role of the librarian in  facilitating debate and action on  institutional practices regarding knowledge  and information management was an added  bonus. in many ways, identifying the need  to provide digital object repositories for a  variety of content types fuelled thinking  about overall knowledge management.  information links were easily identified:  customer relationship management (crm)  needs of the institution and the relationship  with virtual library reference services; help  desk services, crms and library inquiry  services and virtual reference services;  protecting institutional intellectual property  and the link to library managed copyright  compliance systems. all these are examples  of the inter‐connectedness of library services  and expertise with wider institutional issues  and contexts.    longer term impact and follow‐up  all these changes did not happen  immediately, but there were several quick  successes.  however, the sustained cultural  change that resulted from listening to  customers meant that innovation flourished  and customer satisfaction improved.     table 3 uses results from nottingham trent  university’s student satisfaction survey,  conducted in 2007 to demonstrate that the  library’s satisfaction rating shows a marked  increase in satisfaction levels.  this can be  attributed to the initiatives undertaken as a  result of the detailed 2005 research.     discussion    value/uniqueness of the cvd process   there are four major aspects to the  customer value discovery research that set  it apart from other quality and customer  satisfaction surveys.  first, library staff  participate in the workshops as silent  observers, but they vote as they assume the  customers will vote regarding irritations,  frequency of irritations, values, and current  performance.    hearing customers’ comments directly and  listening to their interactions with the  facilitator, can be a powerful experience for  library staff. this is an intensely personal,  and at times emotional, experience,  especially when staff members are  confronted by adverse comments on    evidence based library and information practice 2007, 2:4  32     library services/resources  2005  % actively  satisfied  2007  % actively  satisfied  change  library website  73%  81%  +8%  opening hours  70%  76%  +6%  helpful and accessible staff   67%  74%  +7%  access to printers/ photocopiers  55%  73%  +18%  availability of study places  64%  73%  +9%  availability of e‐journals and  databases  66%  70%  +4%  range of e‐journals and databases  61%  69%  +8%  range of books  61%  65%  +4%  range of printed journals  61%  64%  +3%  availability of pcs in libraries  48%  63%  +15%  noise levels  52%  62%  +10%  training and support for students in  using library facilities  57%  61%  +4%  range of audio and visual materials  61%  59%  ‐2%  availability of books and materials  from reading lists  47%  53%  +6%  loan periods  52%  57%  +5%    table 3. 2005‐2007 nottingham trent university library student satisfaction survey comparison    services for which they are responsible.  compare this to receiving results from the  university‐wide student satisfaction surveys.  although such surveys provide feedback on  specific library services, the data is devoid  of any immediate, personal impact upon  library staff.  it is far removed from their  experience of actually hearing what students  were thinking when they commented in the  customer value discovery workshop.    a second unique aspect of the cvd process  is that feedback is generated in terms used  by customers, because the facilitator starts  with a blank piece of paper. the process  involves completing workbook exercises  that use open‐ended questions, visioning a  future that is successful, and then  prioritising the major values and irritants  into thematic sets.  this allows customers to  decide which services, resources, values,     and irritants they want to consider; they are  not influenced by librarians’ assumptions  about which library services they consider  important.    third, by using workbooks, every  participant in the customer value discovery  workshops has an equal voice. the  participants work in silence in their  workbooks and prioritize their own  comments, which are then transferred to  sticky notes and used to create thematic sets  of issues. this way, no one individual is able  to dominate the process and discussions, as  sometimes happens in focus groups and  other discussion forums.  the fourth significant aspect of cvd  research is that the process of analyzing and  consolidating feedback from all workshops  provides another opportunity to engage  staff in the change process.  all librarian  evidence based library and information practice 2007, 2:4  33 observers and their managers participate in  the analysis to ensure understanding about  customer perceptions, and to gain buy‐in to  follow‐up actions.  it becomes a change  process that is influenced by customers and  led by staff responsible for delivering  services and resources.  it is not a  management‐directed approach to change,  but a staff initiated change. as such, it has a  greater chance of success because of the  personal and local commitment to changes  being suggested.     it is the combination of these factors that  constitute the building blocks for delivering  customer value. change management is  more easily facilitated because staff are  given the opportunity to understand the  needs of their customers, and to realize that  their assumptions about what customers  require or how customers rate the  performance of the library is not always  accurate. also, library staff can influence  how the library responds to the customer  value discovery research through active  engagement and project work.  as a result  of the process, staff members are more likely  to be committed to making changes in their  own work practices to reduce or eliminate  irritants and to implement value added  services and resources.    cultural and organisational change  by automating many basic transactions and  introducing services now possible due to the  power of technology (such as new title  alerts), many irritations were eliminated and,  in some cases, the resultant service was  initially considered novel or ‘wow’. the  problem, if it can be called that, of  consistently delivering value added services,  is that these, over time, are no longer  considered new or exciting and become  expected, basic services.  so the bar is  always being lifted on delivering value  added and unexpected service delivery  delights.     the process of engaging staff as observers in  the customer discovery workshops, and  involving an even larger number of staff in  deciding what actions might reduce  irritation and add value as a result of  customer feedback, has the impact of  encouraging ‘change seeking’ (wilson;  pieters and young) behaviour. this effort at  continuous quality improvement requires  staff to be constantly on the lookout for new  or better ways of delivering value to the  customer.  this is important because  customer expectations are always changing  in response to the wider environment, and  their expectations of service delivery  increases as they experience other, but  related services. what is value adding today  will be basic and expected service in the  near future.    had implementation of changes and  efficiencies meant an immediate reduction  in staffing levels, the process would likely  have failed, as there would be too much fear  and distrust to engage in the process of  customer consultation.  therefore, it was  important from the outset to advise staff  about the parameters of the exercise. in both  cases, staff members were reassured that no  redundancies would result, but that, over  time, it was anticipated that staffing levels  would either reduce or that the focus of  staffing would change. when vacancies  arose, these were to be assessed in regard to  the implementation of the overall plan of  action that had been developed in light of  the customer value research. for instance,  efficiencies in technical services meant  savings could be realigned to providing  more value added services in areas dealing  directly with library customers. the  implementation of self‐service technology  enabled the staffing budget to move from  circulation attendants to liaison librarians.    the need to let go of some long‐undertaken  practices, as identified before, was  challenging.  however, before decisions  evidence based library and information practice 2007, 2:4  34 were finalized, site visits to other libraries  that had changed similar practices reassured  staff that dire outcomes were unlikely, for  instance, if manual checks for outstanding  orders were not undertaken.  it was found  that most items were received from the  supplier within a reasonable time and  automatic alerts at, say 60 days overdue,  were sufficient to identify a problem item.    similarly, different work, and often more  professionally stimulating work,  materialised to fill newly created gaps in the  working day.  for example, cataloguers  were professionally challenged with  metadata as well as traditional cataloguing  rules.    some people are adverse to change and staff  development training was needed,  especially in regard to the psychology of  change management (coping with change),  customer service (ensuring the needs of the  customer are foremost), and team work  (participating fully in project work, often  across organisational boundaries to identify  ways to add value and reduce irritation).  most staff welcomed the opportunities  provided to gain additional skills.    breaking down organisational silos was  another outcome of the customer value  discovery research.  the staff and students  involved in the customer value discovery  workshops are not aware of the discrete  services of the various providers involved in  the service chain. they do not see academic  staff requesting a book, a liaison librarian  approving the order, an acquisitions  assistant creating the order, the clerk who  receives items delivered by the supplier, the  cataloguer, end‐processors or shelvers in the  value added chain of events needed to place  a new book on the shelves. they simply see  ‘the library’ or more likely ‘the university.’   therefore, the importance of teamwork in  facilitating services that do not irritate and  that add value is a key to the library’s  success. proactive partnerships developed  across the library’s different departments,  and within the academic community, as  sharing of knowledge, expertise and  understanding increased as a result of  project work that resulted from customer  value discovery research exercises.    an unanticipated result of these exercises  has been the identification of new ways of  working that have meant more  organisational change, as isolated  workflows, when taken as a combined  process map of service delivery, identified  better ways of working. for instance,  separate teams of staff working on books  and journal orders were merged into a  multi‐skilled team in one library.    conclusions  the examples given in this paper represent  evidence based practice at work. the cvd  process was conducted between 1996 and  2003 at deakin university with different  customer segments, and in 2005 at ntu  with undergraduate on‐campus students  and academic staff. the paper illustrates  how information was obtained from  customers and how it was used to  implement change in cataloguing and  acquisitions.  other customer service  changes that resulted from the same  research will be the subject of a future paper.     would these changes have happened  without customer value discovery research?  in some instances, the answer would have to  be ‘yes.’ however, at what pace would  these changes have been achieved? the  contention in this paper is that the process  of engaging customers and library staff in  focused activities aimed at delivering  customer value considerably speeds the  process of change.    in other instances, the answer is perhaps a  ‘maybe’ or more likely a ‘no.’ the pain of  listening to customers describe a less than  evidence based library and information practice 2007, 2:4  35 optimal service, and customers who  describe situations that, when explained  from a customer perspective, make no sense  at all but fall into the category of ‘we have  always done it this way’ are profound  experiences that challenge every practice,  not just the ones critically described. ‘sacred  cows’ can be challenged.    success breeds success. in the united  kingdom national student (satisfaction)  survey, ntu improved its overall ranking  in the library‐related section of the survey  from 47th overall in 2006 to 34th in 2007, with  a score of 83% satisfaction, against the 2007  sector average of 78%. receiving  significantly improved customer satisfaction  ratings are very powerful tools to encourage  more innovation in service delivery. staff  who receive tangible and positive feedback  on the impact of their change strategies are  more likely to continue to strive for service  improvement.    the process of measuring customer  satisfaction and gauging customer  perceptions of value and irritation is an  ongoing process. as already stated, what  once were ‘wow’ services become basic  services, so the bar continues to rise as time  goes on. further research to define the  values and irritations for specific customer  segments (e.g., off‐campus students, part‐ time students, research staff) as opposed to  broad customer groups (e.g., undergraduate  on‐campus students, academic staff) has led  to more detailed information on which to  base decisions regarding future service  improvements.     the techniques of customer value  discovery described are unusual in the  library and service environments, as the  process was initially envisaged for  commercial, profit‐generating organizations  (austin). the great majority of organisations  using this technique are large financial  institutions and the retail sector. therefore,  the use of this in a not‐for‐profit and service  organisation has been groundbreaking.     the optimal situation for any library  director, any director in fact, is to have staff  members who actively seek change, not for  the sake of change, but to add value to  customers and to reduce irritations.  there  will always be irritations, and as the  environment changes, services have to  respond to changing needs, expectations,  and possibilities. customer value discovery  research can be used as a tool for  transformational leaders, as it enables a  vision of excellence to be articulated, and  engages staff to respond to feedback and  work towards creating the future vision of  excellence.    works cited    albrecht, karl. corporate radar : tracking  the forces that are shaping your business.  amacom, american management  association, new york, ny. 2000.    austin, kevin. e‐mail.  25 sept. 2006.    joint information systems committee (jisc).  “jisc national e‐books observatory  project.” 6 june 2007.  <http://www.jiscebooksproject.org>.    mcknight, susan. “customer value  research.” management, marketing and  promotion of library services based on  statistics, analyses and evaluation. ed.  trine kolderup flaten.  munich: k.g. saur,  ifla publications 120/121, 2006. 206‐16.    pieters. gerald r., and doyle w. young. the  ever‐changing organization: creating the  capacity for continuous change, learning,  and improvement. boca raton, fl: st. lucie  press, 1999.     wilson, frankie.  e‐mail interview. 7 june  2007.   http://www.jiscebooksproject.org evidence based library and information practice 2007, 2:4  36       classics   curry’s study on the quality of public library reference service to lgbtq youth   a review of: curry, a. (2005). if i ask, will they answer? evaluating public library reference service to gay and lesbian youth. reference & user services quarterly, 45(1), 65-75. retrieved from https://www.jstor.org/journal/refuseserq   reviewed by: gregg a. stevens health sciences librarian health sciences library stony brook university stony brook, new york, united states of america email: gregg.stevens@stonybrook.edu   received: 9 jan. 2018      accepted: 30 jan. 2018      2018 stevens. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29399     abstract   objective to assess the quality of service provided by reference staff in public libraries when presented with a request for lgbtq information by a young person.   design unobtrusive observation without informed consent.   setting public library branches in the greater vancouver area, british columbia, canada.   subjects – reference librarians.   methods a 19-year-old posing as a high school student approached reference desk staff at 20 public library branches. the student proxy, “angela”, was instructed to ask for books on forming a gay-straight alliance at her school and, if there was a full reference interview, to also ask for recommendations of novels that the group might read. she recorded the reactions, both verbal and nonverbal, using reference and user services association guidelines as a template. library administrators were aware of the potential visits and permitted the research, but the reference desk staff were not aware of a potential visit by the student proxy. the researcher claimed that her method, while deceptive, was necessary to obtain authentic reactions from the library staff.     main results most reference librarians approached by angela made adequate attempts to assist her, although a few library staff reacted negatively to her query. half of the librarians reacted positively to the patron’s request, with most of the others providing neutral responses. very few of the librarians actually taught the patron how to use the library’s catalog to search for materials, and most of the librarians were unable to find appropriate materials due to not knowing the appropriate search terms. only three library staff showed overt disapproval of the search topic, such as frowning or rushing to finish the reference interview quickly, with most remaining objective or supportive. because of the service she received, angela stated that eight of the 20 libraries were welcoming enough that she thought she would return.   conclusion the wide range of responses received by angela indicated that there was room for improvement in educating public library staff on gay and lesbian issues and materials, especially for gay and lesbian youth.   commentary   library services for members of the lgbtq (lesbian, gay, bisexual, transgender, and queer) community have not been studied extensively in the library literature. many of the existing research studies on libraries and lgbtq patrons have been written from the academic library perspective. studies on lgbtq information seeking behavior, the composition of collections, and the discoverability of lgbtq materials through appropriate subject headings are common themes in the existing body of literature for all types of libraries. however, there have been fewer studies on lgbtq patrons and public services such as reference. in this context, curry’s study was significant for several reasons. her study was one of a very small number of research studies on providing reference services to the lgbtq community within a public library. it was the first study to consider public library reference services exclusive of other library services and collections. perhaps most significantly, her study was focused on gay and lesbian youth, a group whose experiences in public libraries had not been studied previously.   background   a few research studies in the 1990s focused on how well lgbtq patron needs were being met through public library collections and services. creelman and harris (1990) and whitt (1993) both studied the information needs of lesbian patrons. creelman and harris focused on how well public library collections met the information needs of a small cohort of women during the coming out process, but there was no mention in their findings of the women using the services of library reference staff to find information (creelman & harris, 1990). in her larger study of lesbian information needs, whitt mentioned that the women in her survey were generally dissatisfied with the services in their public libraries. a variety of factors played into their responses, such as embarrassment or fear for one’s safety in a small community, to a perceived lack of training for library staff on gay and lesbian needs and issues (whitt, 1993). joyce and schrader (1997) studied the satisfaction of gay men with their public library, and the men generally were satisfied with the services they received when seeking gay-related information. between these studies, it is difficult to come to any conclusions about the quality of public services in the 1990s. however, none of these earlier studies were dedicated specifically to the quality of public services offered by library staff. instead, the focus of all three was the quality of the collections available to the patrons and how well these materials met the information needs of the survey respondents.   none of these studies from the 1990s specifically addressed the needs of lgbtq youth. lgbtq youth are arguably the subgroup in greatest need for accurate information on lgbtq issues, especially during the coming out process (mehra & braquet, 2007). libraries are a reliable source of information on sexuality and can provide a supportive environment for young people (siegel, 2007). all of the studies discussed above did include younger people, but the age ranges were varied. the men in the joyce and schrader (1997) study were generally young, with a mean age of 20, but their ages ranged from teens to late 40s; for that reason it cannot truly be considered a youth study. furthermore, it cannot be considered a full representation of the lgbtq community because all of the respondents were gay men. similarly, the lesbians in the creelman and harris (1990) and whitt (1993) studies also represented a range of ages with a mean of about 29 and 34 years old respectively, so they are of limited use when considering the needs and views of lgbtq youth.   the curry study   the three studies mentioned above from the 1990s all used surveys in order to determine the satisfaction of gay and lesbian patrons in public libraries. while surveys can provide insight into how information needs are being met, they are also based on the retrospective perceptions of library users. they do not necessarily provide an objective picture of the overall quality of public services being offered to patrons.   in order to determine the quality of services that a young lgbtq person might receive at a public library reference desk, curry designed an unusual study. she employed a university student to pose as a 15-year-old high school student. this student was to approach the reference desk in branches of the vancouver public library system and ask for books on starting a gay-straight alliance at her high school, as well as some suggestions of novels for the group to read. the student proxy, given the pseudonym angela, would record the responses she received to her query, both the librarian’s actions as well as the books and resources recommended, immediately after the reference interview. curry referred to this method as “unobtrusive observation without informed consent” (curry, 2005, p. 67). the reference and user services association (rusa) guidelines were used as the template for evaluating the quality of the individual reference interviews, including assessment of objectivity, verbal and nonverbal communication, and the parts of a reference interview.   the institutional review board (irb) at curry’s university (university of british columbia) scrutinized her study because the project involved intentional deception of the library staff. however, curry maintained that deception was crucial to obtaining authentic responses by the library staff. other potential methods of studying how library staff would react, such as self-reported surveys or mock reference interviews with informed consent, would lead to more guarded responses. therefore, she argued that the potential benefit was greater than any harm that might be suffered by staff members who were deceived. as a compromise, the chief librarians of all 11 library systems in the service area were consulted regarding the project and their concerns were assuaged by the researcher. all of the administrators decided not to inform the reference desk staff in their respective libraries, so the staff were not aware that the interaction might take place.   angela visited 20 library branches, representing all the area library systems. she was instructed to visit the reference desk for young adult materials, which was generally the adult reference desk. she recorded the physical and verbal manners the library staff person used when greeting her, as well as the reactions when she asked for books that could help her in starting a gay-straight alliance club. the library staff reactions ranged widely, with half receiving positive scores from angela on their general attitude toward her request. only 3 of the 20 responded negatively to her request for help. angela described negative non-verbal cues such as raised eyebrows, frowns, and lip-biting, as well as remarks such as referring to gay and lesbian fiction as “weird fiction” and a perceived desire to finish the reference interview as soon as possible (curry, 2005, p. 70).   despite offering pleasant greetings and body language, many of the library staff were unable to provide responses to angela’s request that curry deemed acceptable. most of the librarians had difficulty formulating the correct keywords for a catalog search, with three-fourths of them only using the word gay as a search term. even more disappointing for curry was the lack of instruction provided to angela by the library staff: only 3 out of 20 showed angela how to use the library’s online resources, and one of the other 17 just told her to “look it up on the computer” herself despite angela claiming ignorance on how to use the catalog (curry, 2005, p. 71). one staff person was reported to have started a reference interview and then disappeared, leaving angela abandoned at the desk. despite a mixed range of attitudes and reference interview results, angela stated that she would have returned to 8 of the 20 again. the ignorance, indifference, and, in three cases, negativity, she received from staff would have deterred her from returning to the majority of the reference desks she visited.   curry concluded that there was “room for improvement” in most of the reference interviews (p. 73). she speculated on how actual lgbtq teens would have reacted to one of the poorer reference interviews. she also recommended that awareness of lgbtq issues and materials, especially for younger patrons, be added to library school curricula.   subsequent research   it has been more than a decade since curry conducted her study. during that time many significant changes have occurred, both in librarianship and in the lgbtq community. while there remain challenges in attaining full equality for the lgbtq community, there has been an overall increase in acceptance of gay and lesbian people in the united states over the past few decades (keleher & smith, 2012). in 2005, the year curry’s study was published, the civil marriage act became law in canada, allowing same-sex couples to marry. ten years later, the us supreme court ruling in obergefell v. hodges guaranteed the same right to marry in all states. increasing numbers of celebrities and other notables are coming out, often to less fanfare than before. despite these improvements, there are still challenges for the lgbtq community, some of which adversely affect younger lgbtq people. lgbtq teens remain at a greater risk for suicide than other teens (caputi, smith, & ayers, 2017). another great challenge for younger people is that there are no explicit protections in 34 states to protect lgbtq students in elementary and high schools (human rights campaign, 2017). because of this, lgbtq youth still may fear coming out because of the potential for unchecked bullying and discrimination. both of these factors also illustrate why lgbtq teens need reliable information and that a public library, unlike a school library, may be a safer place for a teenager to search for lgbtq materials.   just as there have been societal changes over the past 15 years that impact the lgbtq community, there have also been significant changes in librarianship. public library websites have become more robust since 2005, allowing patrons to access databases and e-resources remotely. a library patron worried about judgement or embarrassment could conceivably locate a book in the library’s catalog, request it to be held for them, and then use self-checkout, circumventing any awkward interactions with library staff and ensuring their privacy. alternatively, the patron could also choose from an increasing number of e-books, which could be virtually checked out and downloaded without a visit to the local library.   virtual reference services are also more prevalent, which could potentially allow a young lgbtq person to ask questions of the reference staff through online chat instead of asking face-to-face. however, there has not been any definitive research regarding online reference services and lgbtq patrons of any age. matteson, salamon, and brewster (2011) conducted a systematic review on synchronous chat services, but could not draw any certain conclusions regarding patron preferences and satisfaction. morris and roberto (2016) discovered in their study of lgbtq healthcare professionals’ information seeking that there is not necessarily a preference for virtual services over in-person reference. in that study, they determined that it was more important to the healthcare professionals to find librarians who were either lgbtq themselves or at least knowledgeable on lgbtq issues and needs (morris & roberto, 2016). while the use of virtual reference may be a solution for any fear or embarrassment in asking sensitive questions, further research may be needed to determine if lgbtq youth would actually embrace this technology.   all of this leads to the ultimate question: have public services in libraries improved since curry’s study?   despite curry’s call for greater awareness of lgbtq issues and materials through increased training, both in lis education and in the workplace, a significant change in lgbtq awareness in public libraries is not apparent. in their survey on lgbtq materials and services in public libraries, hart and mfazo (2010) found that less than one-third (29.4%) of respondents to their survey of library staff indicated that they had received an lgbtq-related question within the past year. it is unclear why so few questions were presented, but it can be speculated that many lgbtq patrons do not feel comfortable asking such questions in the public library.   most “lgbtq” library studies are actually discussing the two most visible groups under the lgbtq umbrella: gay men and lesbians. in 2017, drake and bielefield (2017) conducted a significant survey on transgender usage of library collections and services. they noted that most respondents to their survey did not use reference services for fear of discrimination, often based on past experiences, or for the library staff’s lack of knowledge on transgender issues and resources (drake & bielefield, 2017). this study is also notable because of the ages of the respondents. while it is not specifically a study of transgender youth, nearly half (42%) of the participants were under 25 (drake & bielefield, 2017). this study can be viewed as continuing curry’s work; however, it is disappointing to think that there remains a great need for both diversity training for staff and outreach for lgbtq patrons.   all of these more recent studies point to a continuing need for libraries to be more proactive in addressing the needs of lgbtq youth. mehra and braquet (2007) presented this need as an opportunity for libraries to assist young people during their coming out, through services such as referrals to community resources and better lgbtq collections. lgbtq youth want librarians they can trust, and who will work with them to develop relationships (hawkins, morris, nguyen, siegel, & vardell, 2017). if a reference librarian can work on establishing that relationship with a younger lgbtq patron, then that relationship would be meaningful for both parties, ultimately helping the young person beyond the basic reference transaction.   curry’s study remains a crucial work. it is one of a handful of research articles on reference services to lgbtq patrons in the last two decades, and practically the only study to focus exclusively on lgbtq youth in public libraries. as nearly every other available research study on services to lgbtq patrons in libraries has been a survey, curry’s method of using unobtrusive observation with a proxy teen is unique. it provided an honest snapshot of how a teen would be treated at the reference desk in a given situation. the lack of newer research in this area is an opportunity to determine if libraries are providing better public services to lgbtq youth. until newer research occurs, the curry study will remain the cornerstone study on lgbtq youth in public libraries.   references   caputi, t. l., smith, d., & ayers, j. w. (2017). suicide risk behaviors among sexual minority adolescents in the united states, 2015. jama, 318(23), 2349-2351. https://doi.org/10.1001/jama.2017.16908   creelman, j. a. e., & harris, r. m. (1990). coming out: the information needs of lesbians. collection building, 10(3/4), 37-41. https://doi.org/10.1108/eb023281   curry, a. (2005). if i ask, will they answer? evaluating public library reference service to gay and lesbian youth. reference & user services quarterly, 45(1), 65-75. retrieved from https://www.jstor.org/journal/refuseserq   drake, a. a., & bielefield, a. (2017). equitable access: information seeking behavior, information needs, and necessary library accommodations for transgender patrons. library & information science research, 39(3), 160-168. https://doi.org/10.1016/j.lisr.2017.06.002   hart, g., & mfazo, n. (2010). places for all? cape town’s public library services to gays and lesbians. south african journal of libraries and information science, 76(2), 98-108. retrieved from http://sajlis.journals.ac.za/pub/index   hawkins, b. w., morris, m., nguyen, t., siegel, j., & vardell, e. (2017). advancing the conversation: next steps for lesbian, gay, bisexual, trans, and queer (lgbtq) health sciences librarianship. journal of the medical library association : jmla, 105(4), 316-327. https://doi.org/10.5195/jmla.2017.206   human rights campaign. (2017, december 1). state maps of laws & policies: education. retrieved 29 december 2017 from http://www.hrc.org/state-maps/education/   joyce, s., & schrader, a. m. (1997). hidden perceptions: edmonton gay males and the edmonton public library. canadian journal of information & library sciences, 22(1), 19-37.   keleher, a., & smith, e. r. a. n. (2012). growing support for gay and lesbian equality since 1990. journal of homosexuality, 59(9), 1307-1326. https://doi.org/10.1080/00918369.2012.720540   matteson, m. l., salamon, j., & brewster, l. (2011). a systematic review of research on live chat service. reference & user services quarterly, 51(2), 172-189. retrieved from https://journals.ala.org/index.php/rusq/    mehra, b., & braquet, d. (2007). process of information seeking during “queer” youth coming-out experiences. in m.k. chelton & c. cool (eds.), youth information-seeking behavior ii: contexts, theories, models, and issues. (pp. 93-131). lanham, md: scarecrow press.   morris, m., & roberto, k. r. (2016). information-seeking behaviour and information needs of lgbtq health professionals: a follow-up study. health information and libraries journal, 33(3), 204-221. https://doi.org/10.1111/hir.12139   siegel, j. (2007). let’s talk about sex: the librarians’ response. journal of hospital librarianship, 7(2), 1-15. https://doi.org/10.1300/j186v07n02_01   whitt, a. j. (1993). the information needs of lesbians. library & information science research, 15(3), 275-288.   evidence based library and information practice using evidence in practice   engaging students who have insufficient library resources: a case of a secondary school in lahore, pakistan   abdul jabbar librarian department of political science and visiting lecturer institute of information management university of the punjab lahore, pakistan email: jabbar.polsc@pu.edu.pk   nosheen fatima warraich director institute of information management university of the punjab lahore, pakistan email: nosheen.im@pu.edu.pk     received: 12 oct. 2022                                                              accepted: 8 dec. 2022      2023 jabbar and warraich. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30263     setting   the majority of schools in pakistan have libraries that are nonexistent or in poor condition. where school libraries exist, students are not interested in visiting them due to a lack of interest from principals and teachers, inadequate curriculum structure, and outdated and irrelevant resources available in these libraries. government pilot higher secondary school is situated in wahdat colony, lahore city, the provincial capital of punjab province, pakistan. it is one of the oldest and biggest schools in lahore, covering 25 acres and including 41 classrooms, one library, one staff room, two canteens, and a large playground. it was initially started as a high school in 1958 and then upgraded to a secondary school in 2017. it consists of a qualified and well-experienced principal and a total of 80 teachers, including 70 males and 10 females.   approximately 2,000 students from grades 1 through 12 are studying in the school. the school has a library room with a collection of approximately 5,000 books. there are a total of 28 iron shelves on which books are arranged by subject. the majority of the books are related to religion and history. there is no spine labelling of books. the borrowing facility is available for the students and teachers and is maintained through a register. initially, books are issued for seven working days but can be extended on demand. there is no specified budget for the purchase of books; rather, the government provides books for the library. there is no professional librarian appointed; rather, a teacher who has a m.phil. in urdu is voluntarily assisting with the library, and he is gaining no extra financial benefits for this service.   problem   according to the rules, it is compulsory for academic institutions in pakistan, whether public or private, to have a library to support teaching and learning. as per the federal board of intermediate and secondary education (fbise) act of 1975, a school must have a well-stocked library building to be affiliated with the board. in the case of the establishment of a new school library, rs. 25000 must be spent in the first year, with the provision of rs. 5000 in subsequent years’ budgets. there should also be an adequate number of newspapers and periodicals. here, newspapers are considered as printed publications issued daily containing news, articles, and advertisements whereas periodicals indicate monthly magazines, journals, and newsletters. the school must appoint a library assistant having a certificate or diploma in library science.    we have observed in the case of pilot higher secondary school that there is an inappropriate library building, non-provision of budget for collection development, and no appointment of a library professional, resulting in minimal use of the school library. scanning of the currently available library collection revealed that the majority of books were not written at the appropriate level for students or are outdated or irrelevant. although there is a provision of a library period every day, equal to one class time, students have not utilized that period due to irrelevant books, unavailability of a library professional, lack of interest by the principal and school teachers, and student and teacher misperceptions regarding leisure reading habits and their benefits.   evidence   we established a team of six graduate students studying at the institute of information management to visit the school. we planned to conduct face-to-face interviews with the principal, schoolteachers, and students to better understand the status of the school library and to explore why there was minimal use of the school library. to devise interview questions, we consulted tabassum et al.’s (2019) study, which explored the status of school libraries and their developmental issues in a pakistani context.   the interview questions were categorized in terms of school heads/principals, librarians/teacher-librarians, teachers, and students. the principal was asked about his educational profile, role in extending school library services, and future library development plans. the teacher assisting with the library was asked about his perceptions of the school library, the library management system, and library facilities and services. schoolteachers were asked about their own use of the school library, perceptions about library benefits, and their strategies to promote library use among students. students were asked about their library perceptions, visitation practices, and their satisfaction with the library services. our team was trained to conduct and record in-depth interviews.    the face-to-face interviews were conducted with the principal, five schoolteachers, the teacher assisting with the library, and 15 students from different grades in august and september 2022. all interviews were recorded and transcribed. each interview took an average of 15 minutes to complete. this evidence helped us to understand the situation and perceptions of the principal, teachers, and students regarding the school library.   views of school library stakeholders   although there was a view that the school library had a significant place in engaging students towards leisure reading and lifelong learning, interviewees expressed frustration with the lack of support from the government for the provision of infrastructure, finance, staff, and required resources for the library. they stated that the staff member working in the library was unqualified to provide professional library services, and interviewees mentioned that there was almost no use of the school library for library-related activities. the school library opens every day during the library period, but a low number of students use it due to the non-availability of relevant sources and facilities, and a lack of motivation.   schoolteachers mentioned that there was almost no use of the school library; rather, the library room was used for other activities. they pointed out that the irrelevant and outdated library stock, lack of budget, and non-library-oriented curriculum structure, which focuses on passing tests and examinations rather than the development of voluntary reading, were major obstacles to extensive use of the school library. the interview data also revealed that schoolteachers did not consider the library to be an important place because the emphasis at the school is on the completion of coursework.   views of students   the majority of the students (n = 9, 60%) claimed they rarely visited the library (once a month). they disclosed that their teachers did not give any assignments that required a visit to the library. because there was a peaceful environment for reading but unavailability of relevant and interesting reading materials, they did not like to visit the library. the students were of the view that the structure of their curriculum did not support voluntary reading, but rather encouraged assigned or examination related-readings. a 12th grade student stated: “unfortunately, we have to spend the library period out of the school library.” this expression showed the ineffectiveness of library periods for the students due to the lack of resources and services, and the teachers’ and administration interests in developing and promoting the school library.     implementation   the evidence helped us to understand the situation that although there was a lack of resources and staffing in the school library, we should work on changing the perceptions of the principal, schoolteachers, and students regarding the importance and significance of the use of the school library. we were also planning to work on how and what activities were possible to be conducted through available resources to engage students in leisure reading.   first, we conducted an audit of the available library collection to find relevant and interesting reading materials. we separated a total of 250 books that were up to the level of the students and might interest them.   second, we planned three activities to conduct with the students in the presence of their teachers and the principal in the library during library periods. the activities—quiz competition, urdu poetry competition, and book reviews—were planned according to available resources, and with the guidance of the principal, the teacher assisting with the library, and schoolteachers. a flyer was designed to advertise the library activities and was placed on notice boards and near the entrance of the school. three days before the program, our team visited each classroom to prepare students for participation. the details of the conducted activities are described below.   quiz competition   the program took place on october 3, 2022, with 10 5th grade students who voluntarily agreed to participate, with their remaining classmates as an audience. the quiz contained questions related to general knowledge, about which students were informed during the advertisement of the activity. each student was asked five questions and upon failure to answer, they were dropped from the competition. after two rounds of competition, two students remained, with the final winner announced by the chief guest.    poetry competition   this competition of poetry in urdu language was held among students from upper grades: two students from each class from 6th to 12th grade. the competitors were given letters from the alphabet (i.e., ا، ج، ل  ) and were asked to read a verse starting with that letter. three students were announced as winners of the competition.    book reviews   the students of 2nd, 3rd, and 4th grade were provided books according to their level to read silently for half an hour. these books were selected by our team with the consultation of a teacher, the principal, and a library committee member. the library committee consisted of three schoolteachers from science and arts subjects and the principal. the students were requested to provide a review of the books they read. interesting and analytical reviews were received. at the end of these activities, prizes were distributed among the winners.    outcome   the outcome of the project was wonderful. the students showed surprising interest and enthusiasm to be part of these activities. we observed that these activities stimulated their interest in leisure reading and in visiting the school library during library periods. they urged for a continuation of such activities in the future.   the principal, teacher assisting with the library, and schoolteachers learned a lot and they demonstrated that despite minimal resources it was wonderful to conduct these activities to stimulate students’ reading motivation and literacy engagement. they asked us to serve voluntarily as consultants for the school library, which we accepted. the principal committed to focusing on the development of the library collection by establishing an active library committee. he was also in support of students utilizing the library period in the library in the presence of a teacher who would assist with the library.    reflection   the process was long and required continuous focus. however, cooperative behavior, specifically from the principal and the teacher assisting with the library, helped us a lot in planning and conducting these activities. as there was no funding for this project, it was difficult for us to manage traveling, printing, and prize purchasing. overall, it was a wonderful opportunity for us to learn new things, and we felt satisfied to be able to engage students who previously had minimal chances to use the library. in the future, we are planning to visit other public schools situated in the geographical boundaries of lahore and adjacent rural areas. collecting evidence from these school libraries, we plan to conduct varied innovative reading promotion activities to engage school students.    acknowledgments   we acknowledge extensive support for data collection by the graduate students of the institute of information management, university of the punjab, lahore. the welcoming behavior of the principal, person serving as a librarian, and other teachers of government pilot higher secondary school is highly acknowledged.   author contributions   abdul jabbar: conceptualization, investigation, data curation, methodology, writing – original draft nosheen fatima warraich: conceptualization, project administration, formal analysis, writing – review & editing   references   tabassum, f., batool, s. h., ameen, k., & hassan, m. (2019). status of school libraries and developmental issues in pakistan: a case study of public high schools. global knowledge, memory, and communication, 68(4/5), 377-391. https://doi.org/10.1108/gkmc-08-2018-0070   evidence summary   usability study identifies vocabulary, facets, and education as primary primo discovery system interface problems   a review of: brett, k. r., lierman, a., & turner, c. (2016). lessons learned: a primo usability study. information technology and libraries, 35(1), 7-25. https://doi.org/10.6017/ital.v35i1.8965   reviewed by: ruby warren user experience librarian university of manitoba libraries winnipeg, manitoba, canada email: ruby.warren@umanitoba.ca   received: 1 june 2017    accepted: 11 aug. 2017      2017 warren. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to discover whether users can effectively complete common research tasks in a modified primo discovery system interface.   design – usability testing.   setting – university of houston libraries.   subjects – users of the university of houston libraries ex libris primo discovery system interface.   methods – the researchers used a think aloud usability test methodology, with participants asked to verbalize their thought processes as they completed a set of tasks. four tasks were developed and divided into two task sets (test 1 and test 2), with session facilitators alternating sets for each participant. tasks were as follows: locating a known article, finding a peer reviewed article on a requested subject, locating a book, and finding a newspaper article on a topic. tests were conducted in front of the library entrance using a laptop equipped with morae (screen and audio recording software), and participants were recruited via an assigned “caller” at the table offering library merchandise and food as a research incentive. users could opt out of having their session recorded, resulting in a total of fifteen sessions completed with fourteen recorded. thirteen of the 15 participants were undergraduate students, one was a graduate student, one was a post-baccalaureate student, and there were no faculty participants. facilitators completed notes on a standard rubric, coding participant responses into successes or failures and noting participant feedback.   main results – all eight participants assigned test 1 successfully completed test 1, task 1: locating a known article. participants expressed a need for an author limiter in advanced search, and had difficulty using the citation formatted information to locate materials efficiently. again, all eight participants found an article on the requested subject in test 1, task 2, but two were unable to determine if the article met peer review requirements. one participant used the peer-reviewed journals facet, while the rest attempted to determine this using the item record or with facilitator help. all seven participants in test 2 were able to locate the book requested in task 1 via title search, but most had difficulty determining what steps to take to check that book out. five participants completed test 2, task 2 (finding a newspaper article on a topic) unassisted, one completed it with assistance, and one could not complete it at all. five users did not notice the newspaper articles facet, and no participants noticed resource type icons without facilitator prompting.   conclusions – the researchers, while noting that there were few experienced researchers and a narrow scope of disciplines in their sample, concluded that there were a number of clear barriers to successful research in the primo interface. participants rarely used post-search facets, although they used pre-search filtering when possible, and ignored links and tabs within search results in favour of clicking on the material’s title. this led to users missing helpful tools and features. they conclude that a number of the usability problems with primo’s interface are standard discovery systems usability problems, and express concern that this has been inadequately addressed by vendors. they also note that a number of usability issues stemmed from misunderstandings of terminology, such as “peer-reviewed” or “citation.” they conclude that while they have been able to make several improvements to their primo interface, such as adding an author limiter and changing “peer-reviewed journals” to “peer-reviewed articles,” further education of users will be the only way to solve many of these usability problems.   commentary   there is, as the authors of this study note, substantial literature available on the usability of discovery systems, and on the primo interface in particular. this study, while not precisely replicating any previously published usability studies of the primo interface, does not seek out or fill any gaps in the literature available; however, it is important to conduct usability studies periodically to identify needs or issues unique to an institution’s local context, a purpose this study ultimately serves.   this study scored an 88% overall validity rating in glynn’s critical appraisal tool for library and information science research (2006), with points deducted for the lack of representative diversity in the study participant population (as noted by the study authors), and for the impact of observer bias and observer influence on the results. the authors note that usability study facilitators provided participants with guidance and prompting to use certain features, which negatively impacts the face validity of the study – completions obtained with facilitator assistance can’t tell us if the user would have been ultimately successful independently navigating the primo interface, and should have been recorded as incomplete tasks or invalid results.   although study participants do not completely represent the spectrum of library users at the university of houston libraries, adequate information was collected from undergraduate students to inform design decisions that would impact them. although the number of respondents might seem low for other types of research, for insight gathering usability studies a total of 13 participants is quite high and more than enough to inform design decisions (nielsen, 2012).   the authors confirm a number of standard usability findings as valid, including reducing the amount of jargon and unclear terminology used in web interfaces. should they decide to pursue further research to confirm their hypothesis that instruction is the only way to reduce interface difficulties caused by a lack of understanding of research components, reducing observer influence on study results should be a top priority.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi:10.1108/07378830610692154   nielsen, j. (2012). how many test users in a usability study? retrieved from nielsen norman group articles: https://www.nngroup.com/articles/how-many-test-users/ microsoft word art_zhang.doc evidence based library and information practice 2006, 1:4  3 evidence based library and information practice     article    reporting of the role of the expert searcher in cochrane reviews       li zhang  information services librarian   university of saskatchewan library  saskatoon, saskatchewan, canada  e‐mail: li.zhang@usask.ca    margaret sampson  senior information specialist   chalmers research group  children’s hospital of eastern ontario research institute  ottawa, ontario, canada  e‐mail: msampson@cheo.on.ca     jessie mcgowan  senior information scientist  institute of population health  university of ottawa  ottawa, ontario, canada  e‐mail: jmcgowan@uottawa.ca      received: 29 may 2006    accepted: 14 july 2006      © 2006 zhang, sampson, and mcgowan. this is an open access article distributed under the terms of the  creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits  unrestricted use, distribution, and reproduction in any medium, provided the original work is properly  cited.    abstract    introduction ‐ this study applied the principles of evidence based information practice to  clarify the role of information specialists and librarians in the preparation of cochrane  systematic reviews and to determine whether information specialists impact the quality of  searching in cochrane systematic reviews.   objectives ‐ this research project sought to determine how the contribution of the person  responsible for searching in the preparation of cochrane systematic reviews was reported;  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:4  4 whether the contribution was recognized through authorship or acknowledgement; the  qualifications of the searcher; and the association between the type of contributorship and  characteristics of the search strategy, assessability, and the presence of certain types of  errors.    methods     data sources: the cochrane database of systematic reviews, the cochrane library 3 (2002).     inclusion criteria: the study included systematic reviews that met the following criteria: one  or more sections of the cochrane highly sensitive search strategy were utilised, primary  studies were either randomised controlled trials (rcts) or quasi‐rcts, and included and  excluded studies were clearly identified.     data extraction: two librarians assessed the searches for errors, establishing consensus on  discordant ratings.     results ‐ of the 169 reviews screened for this project, 105 met all eligibility criteria. authors  fulfilled the searching role in 41.9% of reviews studied, acknowledged persons or groups in  13.3%, a combination in 9.5%, and the role was not reported in 35.2% of reviews. for the 78  reviews in which meta‐analyses were performed, the positions of those responsible for  statistical decisions were examined for comparative purposes. the statistical role was  performed by an author in 47.4% of cases and unreported in the same number of cases.  insufficient analyzable data was obtained regarding professional qualifications (3/105 for  searching, 2/78 for statistical decisions).     search quality was assessed for 66 searches across 74 reviews. in general, it was more  possible to assess the search quality when the searcher role was reported ( χ 2=7.41, df=1,  p<.01). an association was found between the reporting of searcher role and the presence of  a consequential error (fisher’s exact ptwo‐sided=.003). there was no association between the  number of consequential errors and how the contribution of the searcher was reported  ( χ 2= 15.536, df=16, p=.486).     conclusions ‐ qualifications of the persons responsible for searching and statistical  decision‐making were poorly reported in cochrane reviews, but more complete role  reporting is associated with greater assessability of searches and fewer substantive errors in  search strategies.       introduction    the systematic review is one of the most  important tools used in the evidence based  medicine movement. the cochrane database  of systematic reviews (cdsr) is the main  product of the cochrane collaboration, an  international non‐profit organization  dedicated to promoting evidence based  medicine. the reviews in cdsr are  conducted primarily by health care  professionals who volunteer for one of the  cochrane review groups (the cochrane  collaboration). a systematic review  involves a comprehensive search for  relevant studies on a specific topic, and  those identified are then assessed and  synthesized according to a predetermined  and explicit method (rychetnik et al. 542).  search quality is the foundation of a  systematic review, and it is anecdotally  known that information specialists and  evidence based library and information practice 2006, 1:4  5 librarians may play an important role in the  systematic reviewing process because of  their expertise in information retrieval.     however, whether the involvement of an  expert searcher in systematic reviews  improves the quality of the search is  unknown. evidence based librarianship and  information practice has gained momentum  in the past few years. its ultimate goal is to  improve the quality of professional  judgment and to develop higher quality  information support by utilizing the best  available evidence (booth, “bridging”). this  study applied the principles of evidence  based information practice in order to find  evidence from the systematic reviews in  cdsr to clarify the role of information  specialists and librarians in the development  of cochrane systematic reviews and to  determine whether information specialists  have an impact on the quality of searching  in cochrane systematic reviews.    literature review    until quite recently, the methods and  reporting of searching have been given little  attention in systematic reviews. quorom  reporting standards, which have been  endorsed by numerous journal editors as a  guideline for reporting systematic reviews  (international committee of medical journal  editors), call only for reporting information  sources searched and restrictions such as  publication year and language (moher et al.,  “quorom” 1897). the oxman and guyatt  quality scale for systematic reviews requires  only two or more searching methods, main  terms, and dates for electronic searching  (1274). although the cochrane  collaboration gives guidelines on how to  report the literature search process, the  structure of cochrane systematic reviews  places the search outside of the methods  section of the report (alderson, green, and  higgins 17). newer guidance requires more  complete reporting (sampson et al., “search  reporting”), but only two studies advocate  reporting the qualifications of the person  doing the search (jensen and ket; stroup et  al. 2009).     complete reporting of literature searching  for systematic reviews would allow readers  to judge the breadth and depth of the search,  thus they could make informed decisions  about the applicability and credibility of the  systematic review results based on the  evidence. however, the bibliographic  database search methods used in systematic  reviews are often not reported in sufficient  detail to allow replication, and some have  argued that journal formats are unlikely to  support full reporting due to space  constraints (chalmers and haynes).  a  study that assessed the quality of reporting  of systematic reviews in pediatric  complementary and alternative medicine  found that half of the reviews reported the  search terms used, but that very few (8.5%)  actually listed the search strings (moher et  al., “assessing” 5‐6). patrick and colleagues  found that only 6.7% of a random sample of  meta‐analyses had both reported the search  strategies in sufficient detail and had  provided evidence of the effectiveness of the  search strategy (198).     further, errors in the search strategies of  systematic reviews are common, even when  the search strategy is reported in enough  detail to allow replication (sampson and  mcgowan). the rationale for choices such as  selection of databases, years searched, or  material to be hand searched, is often not  defended (weller 163‐4).  more recently,  proposed starlite criteria suggest greater  detail for reporting search strategies for  systematic reviews (booth, “starlite”).  these criteria include reporting sampling  strategy, type of study, approaches, range of  year, language, inclusions and exclusions,  terms used, and electronic sources.     evidence based library and information practice 2006, 1:4  6 weller called for the involvement of a  librarian in the search process in order to  improve the quality of conducting and  documenting search strategies (163). the  cochrane reviewers’ handbook also advocates  librarian involvement (higgins and green  71‐72). however, whether librarian  involvement improves reporting of search  methods, improves retrieval, or limits bias  of systematic reviews is unknown. for  example, a study in 1985 comparing the  performance of search strategies developed  by librarians and non‐librarians found that  the librarians did not systematically  outperform the non‐librarians (crumley et  al.; kirpalani et al. 544). listing who is  responsible for searching in a systematic  review delivers important information,  because it not only establishes credit but  also indicates who is responsible for the  quality of the search (rennie, yank, and  emanuel). knowing the credentials of those  undertaking the search could be a surrogate  for determining the likely quality of the  evidence base in the absence of full  reporting of search details.  problems in  conducting and reporting the search are  common in systematic reviews.  in order to  find ways to improve the search quality, it is  necessary to explore the relationship  between the qualifications of the searcher  and the assessability of the search strategies.  in order to find ways to improve the search  quality, it is also necessary to determine,  whether errors were apparent in the  electronic search strategies. the research  results will provide evidence to help  systematic reviewers decide whether a  librarian should be involved.    objectives    1. this research will determine the  contribution of the person responsible for  searching and will examine the statistician’s  role for comparison. specifically, the study  will examine the following questions:    a. how many reviews reported the  contribution of the person responsible  for searching?  b. if the contribution of the searcher is  recognized, is it through authorship or  acknowledgement?  c. what is the qualification of the person  responsible for searching? the term  ‘qualification’ used in this study refers  to job title, credential, or professional  training, and explains why the person is  qualified to perform the job.  d. how many reviews reported the  contribution of the person responsible  for statistical decisions?  e. if the contribution of statistical decision  is recognized, is it through authorship  or acknowledgement?  f. what is the qualification of the person  responsible for statistical decisions?  g. this research report will compare the  reporting of searchers and statisticians.    2. a further purpose of this research is to  determine whether there is an association  between the type of contributorship and  problems in reporting and conducting the  search. specifically, the following questions  will be answered:    a. what is the relationship, if any, between  the search role reporting and the  assessability of the search?   b. what is the relationship, if any, between  the type of contributorship (author,  acknowledged contributor, unknown)  and the presence of certain types of  search errors?    methods    sample: a sample of 169 cochrane reviews  from cdsr, the cochrane library, 3 (2002)  was analysed (representing approximately  10% of reviews published in cdsr at that  time).  this collection was originally  developed for a larger study “finding  evidence to inform the conduct, reporting,  evidence based library and information practice 2006, 1:4  7 and dissemination of systematic reviews”  (abbreviated as “findev” hereafter). the  method of how this collection was identified  was reported elsewhere (sampson et al.,  “search engines”; zhang, ajiferuke, and  sampson).  the structured and relatively  more detailed reporting found in cochrane  reviews, compared to journal‐published  reviews, makes the cochrane library an  efficient lab for studying some aspects of  review methodology. cochrane reviews  usually report electronic search strategies  and include a section of the review  specifically for reporting contributorship  (higgins and green 38), thus providing data  for this study.    eligibility criteria: to be included in this  study (and the “findev” study), each  systematic review had to meet all three of  the following criteria: it had to use at least  one phase of the cochrane highly sensitive  search strategy (hsss), report the citations  for included and excluded studies, and  indicate if primary studies were either rcts  or quasi‐rcts.  of the original 169 reviews  screened for the study, 28 were excluded  because they did not use at least one phase  of hsss; 29 did not report the citations for  included or excluded studies; and 7 did not  have ‘rct’ or ‘quasi‐rct’ as an inclusion  criterion.    there were 105 that met all three screening  criteria, and of those there were 74 studies  that reported the search in sufficient detail  to allow assessment for search errors. eight  searches were duplicates of those used in  other included reviews and were eliminated,  leaving 66 unique searches in the final  assessment to identify search errors.    for the analysis of contribution to the  statistical decision, 78 reviews that pooled 2  or more primary studies in a meta‐analysis  were examined to determine the  contributorship of the person(s) responsible  for the statistical decision. (the screening  and inclusion/exclusion process is  illustrated in figure 1).                                             figure 1. flow of systematic reviews through the phase of screening and eligibility evaluation.  * only the primary exclusion criteria are listed.  169 systematic reviews identified in cdsr 64 systematic reviews failed to meet inclusion criteria *    28 did not use at least one phase of hsss    29 did not report the citations for included and excluded studies    7 rct or quasi‐rct not an inclusion criteria  105 systematic reviews included: examined contributorship of searching 78 included meta‐analytic pooling: examined contributorship of statistical decision  74 assessable for search errors  66  assessed for search errors    8  assessable, but duplicates of searches  previously assessed  evidence based library and information practice 2006, 1:4  8 contributorship reporting: two reviewers (ms  and jm) read each included systematic  review to determine how the contribution of  the bibliographic searcher(s) was reported  (i.e., as author, acknowledged, or not  reported). the qualifications of the  searcher(s) were also examined. for those  systematic reviews that included at least one  pooled meta‐analysis, the contribution and  the qualification of the person(s) responsible  for statistical decisions was examined for  comparison.     search accuracy assessment: each included  systematic review was further screened to  determine if the subject search was  presented in enough detail to permit  assessment (lz). the search accuracy for  this set of systematic reviews was  independently assessed by two reviewers  (ms and jm), with discrepancies resolved  through consensus. the detailed criteria of  search accuracy are reported elsewhere  (sampson and mcgowan). mainly, the types  of search errors were classified into  consequential error and inconsequential  error. the types of search errors that could  potentially have negative consequences on  the evidence base of the review (meaning  they could affect recall of relevant studies)  were defined as consequential errors, and  were considered to be the following:  spelling errors, missed spelling variants,  truncation errors, logical operator errors,  missed mesh, wrong line numbers, search  not tailored for other databases, and  combining mesh and free text terms in the  same line. other types of errors such as  redundancies without rationale, explosion  when no narrower terms existed, and the  inclusion of irrelevant terms were  considered as inconsequential errors.    data analysis: study characteristics were  extracted by a single reviewer (ms). srs  software (<www.trialstat.com>) was used  for all data management including screening  of items for eligibility and extraction of all  data. spss (<http://www.spss.com>) was  used for statistical analysis and table  preparation. analysis was planned by ms,  jm and ig, and was conducted by ms and  ig.    results    the characteristics of the systematic reviews  included in this study are listed in table 1.       n (total=105)  %  year of publication or substantive update      2002  22  21.0  2001  35  33.3  2000  22  21.0  1999  18  17.1  1998  5  4.8  1997  3  2.9        focus of the review      treatment  89  84.8  prevention  14  13.3  diagnosis  1  1.0  other  1  1.0        study designs included      rct only  55  52.4  http://www.spss.com evidence based library and information practice 2006, 1:4  9 rct and quasi‐rct  39  37.1  rct and other controlled trials  11  10.5        meta‐analysis done  78  74.3        country of first author      uk   48  45.7  australia   13  12.4  usa   12  11.4  netherlands   4  3.8  finland   4  3.8  canada   4  3.8  new zealand   4  3.8  germany   3  2.9  hong kong   2  1.9  norway   2  1.9  other  9  8.6        review group      musculoskeletal injuries  21  20.0  eyes and vision  12  11.4  renal  9  8.6  prostatic diseases and urologic cancers  8  7.6  schizophrenia  6  5.7  back  6  5.7  depression, anxiety and neurosis  5  4.8  upper gastrointestinal and pancreatic diseases  4  3.8  infectious diseases  4  3.8  skin  3  2.9  menstrual disorders and subfertility  3  2.9  other  24  22.9  table 1. characteristics of included studies.  the contribution of the person responsible  for searching was reported in 68/105 reviews  (64.8%); authors fulfilled the searching role  in 44/105 reviews (41.9%), acknowledged  persons or groups (including trial search  coordinators of cochrane review groups)  in 14/105 (13.3%), a combination in 10/105  (9.5%), and the role was not reported in  37/105 reviews (35.2%). only 3/105 (2.9%)  reported that the search was conducted by a  librarian and in most cases searcher  qualifications were not reported. there was  insufficient data therefore, to analyze  whether the quality of the searches carried  out by librarians was better than the quality  of those performed by non‐librarians. (table  2).          evidence based library and information practice 2006, 1:4  10   searching    n   %  role fulfilled by:      author  44  41.9  acknowledged named person(s)  8  7.6  acknowledged group  6  5.7  combination of the above  10  9.5  not reported  37  35.2  total  105    was the person fulfilling the role a librarian?                yes  3  2.9  no  2  1.9  can’t tell  63  60.0  n/a (role not reported)  37  35.2  total  105      table 2. search role reporting (n=105)    comparison of reporting contributorship of  searchers and persons responsible for statistics    in 78 reviews a meta‐analysis was  performed, allowing a comparison of the  reporting of contributorship of bibliographic  searchers and persons responsible for  statistics. 57/78 (73.1%) reviews had a  statement of contributorship. for both  searching and statistical components, when  a statement of contributorship was present,  role reporting was more likely to be  complete (search role reported pearson χ 2=  21.914, 1 df, pone‐sided <.001; statistical role  reported pearson χ 2= 21.35, 1df, pone‐sided  <.001).     the person responsible for statistical  decisions was an author in 37/78 (47.4%)  cases, and unreported in the same number  of cases. there were 51/78 (65.3%) reviews  that reported the types of contributions of  the person(s) responsible for searching,  while 41/78 (52.5%) reported the types of  contributions of the person(s) responsible  for statistical analysis (table 3). of the 78  reviewed meta‐analyses, the qualifications   of the person(s) responsible for searching or  statistical decisions were not reported or  were unclear in all but 3 that indicated  library science training and 2 that reported  statistical qualifications (table 4).        searching  statistics  role fulfilled by:  n   %  n  %  author  32  41.0  37  47.4  acknowledged named person(s)  6  7.7  3  3.8  acknowledged group  4  5.1  0  0.0  combination of the above  9  11.5  1  1.3  not reported  27  34.6  37  47.4  total  78    78      table 3. role reporting for the search and the statistical analysis for reviews that had both a search and a  meta‐analysis (n=78)  evidence based library and information practice 2006, 1:4  11     searching  statistics  was the person fulfilling the role a librarian  or statistician?  n  %  n  %  yes  3  3.8  2  2.6  no  1  1.3  0  0.0  can’t tell  47  60.2  39  50.0  n/a (role not reported)  27  34.6  37  47.4  total  78    78    table 4. comparison of qualification reporting of librarian and statistician (when a meta‐analysis was  performed, n = 78)  54 20 14 17 0 10 20 30 40 50 60 search role reported search role not reported n o f c oc hr an e r ev ie w s search assessable search not assessable   figure 2. search role reporting and assessability of the search (n=105)    search quality    it was possible to assess 74 of 105 (70.5%)  reviews for search quality. of the 74 reviews,  54 reported the searcher role, 20 did not. it  was possible to assess the search accuracy in  54 of 68 (79.4%) reviews where the searching  role was reported.  where there was no  attribution of the searching role (37 reviews),  it was only possible to assess the search  strategies in 20 reviews (54.1%)  (figure 2).     this result suggests that it was more likely  to be possible to assess search quality when  searcher role was reported ( χ 2=7.41, df=1,  p<.01).     search quality was assessed for 66 searches  across 74 reviews.  consequential error  presence rate was 32/47 (68.1%) when the  searching role was reported, and 19/19  (100%) when not reported (table 5). an  association between the reporting of   evidence based library and information practice 2006, 1:4  12   number of consequential errors      0  1≥   total  searching role reported  15  32  47  searching role not reported  0  19  19  total  15  51  66    table 5. presence of consequential errors and reporting of searching role (total searches: 66)        number of consequential errors   0  1  2  3  4  total  author  8  8  7  4  1  28  acknowledged named  person(s)  2  2  2  1  0  7  acknowledged group  3  2  1  0  0  6  combination of the  above  2  3  0  1  0  6  was the  searching  role  fulfilled by  an:  not reported  0  9  7  1  2  19  total  15  24  17  7  3  66    table 6.  number of consequential errors and types of contributorship reporting (total searches: 66)      searcher role and the presence of a  consequential error (fisher’s exact  ptwo‐ sided=.003) was found. when the role of the  searcher was reported, a consequential error  was less likely to be present. there was no  statistical association between the number of  consequential errors and how the  contributorship (author, acknowledged  person, acknowledged group, combination  of the above, or not reported) of the searcher  was reported ( χ 2= 15.536, df=16, p=.486,  table 6). as a result, it was not possible to  determine whether fewer consequential  errors appeared in searches when search  authors were recognized or acknowledged.    discussion     in general, more than half of the reviews  studied here reported the contribution of the  person responsible for the critical tasks of  searching and statistical analysis. librarians  who participate in or support systematic  reviews must understand the issues of  authorship and contributorship and  advocate for the recognition of their  professional contribution. listing  contributions imparts important  information and establishes accountability,  responsibility, and credits (rennie, yank,  and emanuel 579). the findings in this  study provide superficial evidence to  support this statement. search accuracy  could be assessed more frequently when the  searching role was reported. further, more  complete role reporting is associated with  fewer consequential errors in search  strategies. these findings indicate that the  disclosure of who performed the search  improves the quality of conducting and  reporting of searches for systematic reviews,  possibly because the named contributor  holds responsibility for the work performed.  therefore, it is suggested that all reviews  should describe who performed the  searching role in the list of contributors,  evidence based library and information practice 2006, 1:4  13 because high quality searches are the  foundation of systematic reviews.     there was inadequate reporting of the  qualifications of the searchers to permit an  assessment of the association between the  involvement of librarians and search quality.  it was difficult to assess the professional  qualifications of authors and contributors  because of the cochrane collaboration’s  practice of not publishing credentials or job  titles. the moose statement and a newly  developed checklist for reporting literature  searches require that authors of meta‐ analyses report the qualifications of the  searchers (stroup et al. 2009; jensen and ket),  because this may provide indirect evidence  of the quality of the searches (patrick et al.  199).  only 3 of 105 reviews indicated that  the searches were performed by librarians,  and most did not report the qualifications of  the searchers. hence, from this limited data,  it cannot be concluded that qualifications  are a proxy of the search quality in the  absence of full reporting of the search  methods. more research is needed to explore  whether the involvement of a librarian is  associated with better conducting and  reporting of searches for systematic reviews.  golder and colleagues, for example, recently  reported that few searches in systematic  reviews of adverse effects were reported in  enough detail to allow replication, but that  nearly half of those that could be replicated  were known to have been conducted by a  qualified information specialist (golder,  loke, and mcintosh 4). for those reviews  where a meta‐analysis was completed, the  qualifications of the person responsible for  statistical decisions were also not well  reported.     the cochrane collaboration has called for  improved reporting of contributorship in its  reviews. the cochrane reviewers’ handbook  has suggested since 1999 that each review  include a list of contributors on the cover  sheet (clarke and oxman 99). it also  provides a list of descriptions of potential  contributions. the contributions that are  relevant to searching include developing  search strategy, undertaking searches, and  organizing retrieval of articles. however,  this study showed that 24/105 reviews (23%)  reported the searching role in the  acknowledgement section or in a  combination of author and  acknowledgement sections. the overlap  between reporting of searcher role on the  lists of contributors and acknowledgements  suggests that the distinction between  contributors and acknowledgements is  unclear. it has been suggested that a  research paper should have only one list of  contributors naming and describing the  contributions of all participants, and that the  use of an acknowledgements list should be  restricted to recognition of funding agencies  or corporate bodies (yank and rennie 668).  this approach would increase the integrity  of the research publications.     conclusions  this study showed that more complete  reporting of the searching role is associated  with greater assessability of searches and  fewer substantive errors in search strategies.  this finding provides evidence to confirm  the importance of complete reporting of  literature searching for systematic reviews,  including reporting who performed the  searching role. this evidence should be  incorporated into new guidelines for  reporting of searching for systematic  reviews to improve the quality of the  literature search.    due to a lack of analyzable data, there was  insufficient evidence to conclude whether  the involvement of a librarian improved the  quality of the literature searches reviewed in  this study. further investigation needs to be  conducted on this subject. a  recommendation for obtaining additional  supportive data (e.g., who undertook the  evidence based library and information practice 2006, 1:4  14 literature search and the qualifications of the  searcher) would be to send questionnaires to  the corresponding authors of published  systematic reviews, asking for their input.  analysis of this information might provide  more evidence to determine whether  librarian involvement improves the quality  of literature searches in systematic reviews,  thus ensuring that our information practice  is evidence based.     acknowledgements  the data used in this study is partly based  on a project, “finding evidence to inform  the conduct, reporting, and dissemination  of systematic reviews,” funded by the  canadian institutes of health research  (cihr grant mop57880).    we thank isabelle gaboury, msc for  statistical advice.  works cited     alderson, phil, sally green, and julian  higgins, eds. cochrane reviewers’  handbook 4.2.2 [updated march 2004]  in: the cochrane library 1(2004).  chichester, uk: john wiley & sons,  ltd., 2004.     booth, andrew. ʺbridging the research‐ practice gap? the role of evidence  based librarianship.ʺ the new review  of information and library research 9  (2003): 3‐23.     booth, andrew. ʺbrimful of starlite”:  toward standards for reporting  literature searches.ʺ journal of the  medical library association 94.4  (2006): 421‐9.  chalmers, iain, and brian haynes.  ʺreporting, updating, and correcting  systematic reviews of the effects of  health care.ʺ bmj 309.6958 (1994): 862‐ 5.   clarke, mike, and andrew d. oxman, eds.  cochrane reviewersʹ handbook 4.1  [updated july 1999] in reviewer  manager (revman) [computer  program]. version 4.0. oxford, england:  the cochrane collaboration, 1999.   crumley, ellen t., natasha wiebe , kristie  cramer , terry p. klassen,  and lisa  hartling. ʺwhich resources should be  used to identify rct/ccts for  systematic reviews: a systematic  review.ʺ bmc medical research  methodology 5 (2005): 24.   golder, su, yoon loke, and heather m.  mcintosh. ʺroom for improvement? a  survey of the methods used in  systematic reviews of adverse  effects.ʺ bmc medical research  methodology 6 (2006): 3.   higgins, julian pt, and sally green, eds.  cochrane handbook for systematic  reviews of interventions 4.2.5  [updated may 2005]. chichester, uk:  john wiley & sons, ltd., 2005.   international committee of medical journal  editors. “uniform requirements for  manuscripts submitted to biomedical  journals: writing and editing for  biomedical publication.” updated feb.  2006. 6 nov. 2006 <  http://www.icmje.org>.  jensen, fabricius m., and hans ket. ʺcheck  it out! a check‐list for evaluating the  reporting of literature search  methodology in htas and cpgs.ʺ 2nd  annual htai meeting. rome, italy, 20‐ 22 june 2005.   kirpalani, haresh, barbara schmidt, k. ann  mckibbon, r. brian haynes, and john  http://www.icmje.org evidence based library and information practice 2006, 1:4  15 c. sinclair. ʺsearching medline for  randomized clinical trials involving  care of the newborn.ʺ pediatrics 83.4  (1989): 543‐6.   moher, david, deborah j. cook, susan  eastwood, ingram olkin, drummond  rennie, and donna f. stroup,  ʺimproving the quality of reports of  meta‐analyses of randomised  controlled trials: the quorom  statement. quality of reporting of  meta‐analyses.ʺ lancet 354.9193 (1999):  1896‐900.     moher, david, karen soeken, margaret  sampson, leah ben‐porat, and brian  berman. “assessing the quality of  reports of systematic reviews in  pediatric complementary and  alternative medicine.ʺ bmc pediatrics  2 (2002): 3.   oxman, andrew d., and gordon h. guyatt.  ʺvalidation of an index of the quality  of review articles.ʺ journal of clinical  epidemiology 44.11 (1991): 1271‐8.   patrick, timothy b., george demiris, lillian  c. folk, david e. moxley, joyce a.  mitchell, and donghua tao. ʺevidence‐ based retrieval in evidence‐based  medicine.ʺ journal of the medical  library association 92.2 (2004): 196‐9.   rennie, drummond, veronica yank, and  linda emanuel. ʺwhen authorship  fails. a proposal to make contributors  accountable.ʺ jama 278.7 (1997): 579‐ 85.   rychetnik, lucie, penelope hawe, elizabeth  waters, alexandra barratt, and michael  frommer. ʺa glossary for evidence  based public health.ʺ journal of  epidemiology & community health  58.7 (2004): 538‐45.   sampson, margaret, jennifer tetzlaff , jessie  mcgowan, elise cogo, and david  moher. ʺsearch reporting: current  practice and recommendations from 11  scales.ʺ xiv  cochrane colloquium.  dublin, ireland, october 24, 2006.   sampson, margaret, and jessie mcgowan.  ʺerrors in electronic search strategies  of cochrane reviews were identified  by type and frequency.ʺ journal of  clinical epidemiology 59.10  (2006):  1057‐64.  sampson, margaret, nicholas j. barrowman,  david moher, tammy j. clifford,  robert w. platt, andra morrison, terry  p. klassen, and li zhang. ʺcan  electronic search engines optimize  screening of search results in  systematic reviews: an empirical  study.ʺ bmc medical research  methodology 6 (2006): 7.   stroup, donna f., jesse a. berlin, sally c.  morton, ingram olkin, g. david  williamson, drummond rennie, david  moher, betsy j. becker, theresa a. sipe,  and stephen thacker. ʺmeta‐analysis  of observational studies in  epidemiology: a proposal for  reporting. meta‐analysis of  observational studies in epidemiology  (moose) group.ʺ jama 283.15 (2000):  2008‐12.   the cochrane collaboration. ʺwhat is the  cochrane collaboration?ʺ 20 nov. 2006.  <http://www.cochrane.org/docs/descrip .htm>.  weller, ann c. ʺmounting evidence that  librarians are essential for  comprehensive literature searches for  meta‐analyses and cochrane reports.ʺ  journal of the medical library  association 92.2 (2004): 163‐4.   http://www.cochrane.org/docs/descrip evidence based library and information practice 2006, 1:4  16   yank, veronica, and drummond rennie.  ʺdisclosure of researcher  contributions: a study of original  research articles in the lancet.ʺ  annals of internal medicine 130.8  (1999): 661‐70.   zhang, li, isola ajiferuke, and margaret  sampson. ʺoptimizing search  strategies to identify randomized  controlled trials in medline.ʺ bmc  medical research methodology  6  (2006): 23.                                         research article   effect of undergraduate research output on faculty scholarly research impact   adriana popescu interim dean of library services robert e. kennedy library cal poly san luis obispo, california, united states of america email: popescu@calpoly.edu    radu popescu lecturer, architectural engineering college of architecture and environmental design cal poly san luis obispo, california, united states of america email: rpopescu@calpoly.edu    received: 04 july 2017    accepted: 24 oct. 2017         2017 popescu and popescu. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – in the context of the ongoing discourse about the role of institutional repositories (irs), the objective of the study is to investigate if there is any evidence of a relation between undergraduate student activity in an ir and the impact of faculty research.   methods – the data used for the study is representative of six academic departments of the college of science and mathematics (csm) at california polytechnic state university (cal poly). digital commons@cal poly (dc) is the ir supported by the library. regression analysis was used to investigate the interdependence between faculty research impact (dependent variable) and undergraduate student repository activity (independent variable). for each department, faculty research impact was quantified as a measure of the citation counts for all faculty publications indexed in web of science (wos) between january 2008 and may 2017. student repository activity was quantified for each department in two ways: (1) total number of student projects deposited in dc since 2008 (sp) and (2) total number of student project downloads from dc (sd). the dependent variable was regressed against each of the two elements of student repository activity (sp and sd), and the resulting statistics (sample correlation coefficients, coefficients of determination, and linear regression coefficients) were calculated and checked for statistical significance.   results – the statistical analysis showed that both components of student repository activity are positively and significantly correlated with the impact of faculty research quantified by a measure of the citation counts. it was also found that faculty repository activity, although positively correlated with faculty research impact, has no significant effect on the correlation between student repository activity and faculty research impact.   conclusion – the analysis considers two distinct groups of publications: one group of student publications (senior projects) from six academic departments, which are deposited in an open repository (dc), and one group of publications (not necessarily represented in dc) of faculty affiliated with the same six departments and whose citation impact is believed to be affected by the first group. the statistical correlation between student repository activity and faculty research impact can be seen as an indication that an active, open ir centered on collecting, preserving, and making discoverable student research output has a positive impact on faculty’s research impact. more research that includes additional factors and uses a larger data set is necessary to arrive at a definitive conclusion.       introduction   cal poly is a nationally ranked public university and part of the california state university (csu) system, the largest public university system in united states. the school’s motto is “learn by doing,” which translates into a pedagogical focus on project-based curriculum. throughout their cal poly experience, students actively engage in research, experimentation, studio work, and design, and the outcomes of their class experience and learning are reflected in the senior project submissions.   in 2008, the library launched digital commons@cal poly (dc), which serves as the institutional repository (ir). dc is powered by bepress (https://www.bepress.com), which is used by over 500 educational institutions to preserve and showcase their scholarly output and special collections. the mission of cal poly’s ir is to collect, preserve, and make visible all institutional intellectual output, including pre-prints, working papers, journal articles, senior projects, master's theses, conference proceedings, presentations, images, and a wide variety of other content types. although the library’s dc is an open access (oa) repository and is available for submission of student and faculty work, not all departments actively deposit to dc. the library’s ongoing efforts to promote the benefits of contributing scholarly and creative works to dc had mixed results, with some colleges (and departments) being more active participants than others.   recent discussions about the purpose of irs and a call to “disconnect them from the oa agenda for journal articles” and reposition them “in the broader context of managing and preserving institutional community assets” (lynch, 2017, p. 127) triggered the interest to investigate whether community assets (e.g., student senior projects) preserved and exposed in irs can have a positive influence on the overall faculty research impact. senior projects, electronic theses, and dissertations represent a significant part of the institutional intellectual output. by exposing this output in irs, libraries not only fulfill their mission to curate, archive, and preserve but by developing irs centered on student work, they also facilitate the advancement of the faculty research agenda and profile.   many studies have investigated the effect of open access (oa) on the research impact of publications. the general conclusion was that oa offers clear advantages over paid access with respect to accessibility and therefore visibility of published research and has a significant effect on the overall research impact expressed as a function of citation count (brody, harnad, & carr, 2006; gargouri et al., 2010). the novelty of the present study is that it aims to investigate using statistical methods whether an active, open ir centered on student work has a positive impact on faculty’s research impact independent of faculty’s participation in the ir. the study analyzes two distinct samples of publications:   1.       a group of faculty publications from the six csm departments selected for the study for which research impact is calculated based on web of science (wos) citation data 2.       a group of csm student publications (senior projects) from the same six csm departments that are deposited in dc   the first sample of publications consists of articles published between january 2008 and may 2017 by the faculty in the six departments of csm at cal poly and indexed by wos. only articles published under the cal poly affiliation are included in the study. incites (http://clarivate.com/?product=incites), a customized, web-based research evaluation tool that uses wos data to generate institutional reports to showcase strengths and identify potential areas for growth, was used to acquire values for journal expected citations (jec) and journal normalized citation impact (jnci) indicators.   the second sample of publications originates from dc. three major categories of scholarly output are deposited in dc (among others): (1) faculty works (e.g., voluntarily deposited scholarly output), (2) undergraduate student senior projects, and (3) master’s theses. deposit and download metrics for the first two categories were obtained from institutional activity reports for dc and were used in the study.   the goal was to identify a possible correlation between the scholarly impact of faculty research and undergraduate student repository activity. the faculty activity in dc is also included in the analysis to verify whether it affects the direct correlation between faculty research impact and student repository activity.   csm at cal poly has a strong record of faculty and undergraduate research, which is also reflected in an active participation and submissions of student works to dc. six departments were selected for the study: biological sciences, chemistry & biochemistry, kinesiology, mathematics, physics, and statistics. two departments have been omitted for the following reasons: (1) faculty in the liberal studies department have dual departmental affiliations (csm and the college of liberal arts), and it was not possible to isolate the research contributions of the faculty specific to csm; and (2) the school of education does not offer undergraduate programs. the six selected departments were randomly assigned numbers, and the departments are identified in the study only by these numbers.   one limitation of the study is the small sample of academic units (n = 6), which may affect credibility of the conclusions on the grounds that some results could have been obtained by chance. to overcome this issue, all results were thoroughly checked for statistical significance.   literature review   role of institutional repositories   the report of the coalition for networked information (cni) executive roundtable on “rethinking institutional repository strategies” held during the spring 2017 cni meeting in albuquerque, new mexico, highlighted the challenges faced by existing irs (cni, 2017). it is increasingly difficult to justify why faculty should place materials in an ir when other options, such as disciplinary repositories that meet funders open access mandates are available, or when commercially run systems, such as academia.edu or researchgate that offer networking and analytics features, are available (cni, 2017). given perennial competing funding priorities, academic libraries are faced with the task of demonstrating value and return on investment for continuing to support and maintain irs, which have been implemented, developed, and maintained since the early 2000s at significant costs and mostly supported by libraries. one way to demonstrate and make the case for the viability of an ir is to demonstrate that the undergraduate research output deposited in ir is reflected in the overall faculty research impact.   undergraduate research   the positive benefits of exposing and encouraging undergraduate research experiences have been studied and reported in the literature. undergraduate research experiences translate into personal and professional gains for students and are reflected in elucidation of career paths and enhanced graduate school preparation (seymour, hunter, laursen & deantoni, 2004). positive impact on student retention (gregerman, lerner, von hippel & nagda, 1998) has also been reported. a limited number of studies (lei & chuang, 2009) show that faculty benefit indirectly because students who have gained publishing and practical original research experience while working on faculty research projects become contributors to scholarly publications. by generating publishable results from undergraduate research projects, faculty may have established a valuable future research collaboration with these students. however, no studies have been identified that attempt to demonstrate a direct influence of undergraduate research activity on faculty research impact.   research impact   when assessing research performance, it is important to take into account both the volume and the quality of research output. citations are widely recognized as a proxy for quality. the citation impact quantifies the citation usage of scholarly works. eugene garfield, the creator of web of science, states that “citation frequency is a measure of research activity . . .” (garfield, 1973), and that frequency of citations is an “indicator of quality . . . of productivity as well as impact” (garfield, 1988). moed (2005) discusses in detail the relationship between intellectual influence or research impact and citation impact. he shows that “even if one assumes that citations measure intellectual influence . . . intellectual influence needs to be valued in a wider cognitive framework” and that there are some factors that affect in a different manner intellectual influence and citation impact (p. 223). there are possible biases and errors in the interpretation of citation impact, and therefore, empirical analyses do not result in perfect correlations. moed (2005, p. 224) concludes, however, that the fact that these correlations are positive provides an empirical justification for relating citation impact to intellectual influence (or research impact—as it is termed in this study). he further shows that analysis bias may be reduced to a considerable extent when analyzing aggregates of entities that have some aspects in common rather than analyzing individual units (p. 225).   citation counts, or times cited (tc), were first used to evaluate importance of scientific work by gross and gross (1927) and since then have remained the main means to characterize research impact. while tc is a meaningful and accessible way to reflect scholarly output and measure the impact of an individual researcher, a group, or an institution, garfield (1972) warned that tc is a function of many other variables besides scientific impact. bornmann and daniel (2008) list and discuss some of those factors: (1) time of publication, with more citations to recent than to older publications; (2) field of research, with the citation potential varying significantly from one field to another; (3) journal frequency of publication and journal impact factor; (4) article type (e.g., review, research, letter, note), language, and length; (5) number of coauthors; and (6) accessibility (i.e., oa or paid access).   to alleviate the effect of some factors, one can look at how the citation count (tc) compares with expected citation count for a field or discipline or for a specific journal. the expected citation count is available for most of the journals indexed in wos as the journal expected citations (jec) indicator (clarivate analytics, 2017). the ratio between tc and jec, for example, becomes a qualitative measure of the research impact that can be compared across various publications and even various disciplines.   data used in the study   independent variables   two categories of dc repository activity were included in the study as described below. the data were obtained from cal poly's dc institutional activity reports.   1.       undergraduate student activity consisted of senior projects and was quantified for each department through the project counts (sp) and the project download counts (sd). as of may 2017, dc contained 263 undergraduate student projects totaling about 276,000 downloads for the six csm departments considered. these data were normalized by the number of faculty (nf) in each department and are listed in columns 2 and 3 of table 1. these data represent the independent variables for the correlation sought after in this study. 2.       faculty activity consisted of research articles deposited in dc and was quantified for each department through the paper counts (fp) and the paper download counts (fd). this activity was included in the study to investigate if the correlation between student activity in dc and faculty research impact is in fact a result of the faculty repository activity in dc. these data were normalized by nf and are listed in columns 4 and 5 of table 1.   the indicators of student repository activity (sp and sd) and faculty repository activity (fp and fd) were normalized by the department size expressed as number of faculty (nf). given that all departments discussed here belong to the same academic unit, and most likely have comparable resources, the size of each department can be expressed as a function of either nf or the number of students. as the number of students in a department may vary significantly from one academic year to another, nf was selected as a measure of the department size.   some of the csm departments also offer graduate programs, and master’s theses are usually deposited in dc. the effect of graduate student repository activity on faculty research impact will be analyzed in a future phase of the study.   dependent variable   the dependent variable in this study is a measure of the scientific impact of all faculty in each department quantified by a measure of the citation count of their publications. as discussed in the literature review, faculty research impact or performance (in short research impact) can be quantified by a measure of the citation count from faculty publications. raw citation counts are affected by other factors besides research performance. the measure of citation count used here aims to eliminate most of these factors. in this respect, incites provides the journal normalized citation impact (jnci) indicator for each publication. the jnci is the total number of citations per paper (tc) “normalized for journal, year and document type subject” (clarivate analytics, 2017, p. 18). the normalizing factor is the journal expected citations (jec) indicator defined as the “average number of citations to articles of the same document type from the same journal in the same database year” (clarivate analytics, 2017, p. 18).     table 1 data used in the study department no. repository activity in dc (all values are divided by nf) faculty research impact indicator (jnciav) undergraduate student projects faculty papers project count (sp/nf) download count (sd/nf) paper count (fp/nf) download count (fd/nf) 1 4.46 3,863.0 14.91 5,033.0 2.012 2 2.67 5,038.7 3.05 996.9 1.269 3 0.20 155.2 1.57 762.3 0.765 4 0.22 272.8 3.56 930.1 0.882 5 1.10 522.6 16.17 8,698.7 1.145 6 1.36 1,286.6 11.55 9,209.5 1.374     in reference to the list of factors affecting tc discussed in the literature review, use of jec as a normalizing factor eliminates the influence of the first three factors in the list (time of publication, research field, and journal impact factor). given the relatively large groups of papers analyzed here, the elements characterizing the other three factors can be considered to be roughly similar for all departments. based on these considerations, the jnci indicator is used to assess the scientific impact of each individual paper. each individual value of jnci shows if the paper has been cited more than expected (jnci > 1) or less than expected (jnci < 1).   the research impact, denoted as jnciav, is a qualitative measure of the impact of the faculty publications, is defined for an entire department, and is calculated here as the average of all jnci values for all papers indexed by wos published by the faculty in each department between january 2008 and may 2017 (a total of 871 articles for the six departments). only active faculty as of may 2017 (according to departmental directory listings) have been considered in the study. incites was used to extract and process wos data used to calculate the research impact indicator.   a series of issues exist when using this research impact indicator:   1.       for some journals, the jec value listed by incites is zero or is not available; therefore, jnci cannot be calculated. 2.       if the value of jec is very small, one single citation would result in unusually large values of jnci that may bias the resulting average value for some departments.   these limitations were addressed as follows:   1.       the papers where jec is not available or zero were not included in the jnciav indicator calculation. these papers represent 14% of all papers considered in this study. 2.       the papers with jec lower than a given threshold were also eliminated from the research impact calculation. the threshold selected was jec = 0.1. an additional 10% of all papers considered were eliminated due to this filter.   thus, the indicator used to characterize the research impact in each department is the average of jnci for all papers that have jec ≥ 0.1. this indicator is referred to as research impact and denoted by the symbol jnciav. the values of jnciav for the six departments considered are listed in the last column of table 1.   tests for normality   this study used linear regression analysis between the independent variables (various aspects of student repository activity) and the dependent variable (faculty research impact indicator). though there is no general requirement for the input data in a regression analysis to be normally distributed, certain statistical tests used in the next section require normality, especially for small samples (devore, 2000, p. 533). therefore, the data used here is first checked for normality and transformed if necessary to achieve normality.   the test for normality is in general easily met for very small samples such as those in this study. one way to qualitatively assess the goodness of fit with the normal distribution is to visually compare the quantile-quantile plots (or qq-plots) of the sample versus theoretical quantiles from the normal distribution. as the sample is closer to normal, the qq-plot is closer to a straight line. qq-plots for the quantities used here are presented in figure 1 and are used to estimate whether original sample data or logarithm of sample data is closer to a normal distribution. based on visual comparison, it appears that logarithms of the values in table 1 are closer to the normal distribution for normalized student project downloads, sd/nf, and for research impact indicator, jnciav. no conclusion could be obtained from the plots regarding the normalized student project counts, sp/nf.   statistical quantitative assessments for goodness of fit are also available. the most popular test for assessing normality of a sample is the chi-square test, but the sample size used here is too small to provide reliable results. two other tests are used that accept small sample sizes, namely kolmogorov-smirnov (massey, 1951) and ryan-joiner (devore, 2000, p. 634). based on these two statistical tests, all data sets fit the normal distribution at the 5% level of significance, but the log-value sets are closer to a normal distribution than the original values for all sets listed in table 1. therefore, to obtain samples closer to the normal distribution, logarithm of all values listed in table 1 (independent and dependent variables) are used in the regression analyses. the statistical level of significance is briefly discussed in the next section.   analysis and results   correlation between faculty research impact and undergraduate student activity in dc   regression analyses   regression analysis explores the relationship between two or more variables related in a nondeterministic fashion (devore, 2000, p. 489). more specifically, a regression analysis between two sets of measured quantities, the dependent variable denoted by y and the independent variable denoted by x, explains how y changes as a function of the changes in x, or, in other words, it expresses y as a function of x. this function,  = f(x), is called regression function or regression model. note that, for any value of x, the result of f(x) is not necessarily equal to the corresponding measured value of y but to a predicted value, . linear regression seeks to find a linear functional relationship between y and x. in simple linear regression, as described here, there is only one independent variable. in multiple linear regression analysis, as described later in the section titled effect of faculty activity in dc on research impact, the analysis includes more than one independent variable.     figure 1 quantile-quantile plots for assessing normality of the data samples used in the study.       the strength of the relation between y and x is measured through a series of quantities obtained from the regression analysis, such as the coefficient of determination and the sample correlation coefficient. the coefficient of determination, r2 (or r-squared), is defined as the proportion of the variance in the dependent variable that can be explained by the linear regression model (devore, 2000, p. 506). in simple linear regression, r2 is equal to the square of the sample correlation coefficient between the independent and the dependent variables. this correlation coefficient, denoted here by r, is a measure of the strength of the linear association between the two quantities. the functional relationship between the dependent and the independent variables is expressed in simple linear regression as  = a0 + a1x. the parameters a0 and a1 are the intercept and the slope of the regression line and are also a result of the regression analysis.   the regression function in excel has been used to perform two linear regression analyses: (1) between faculty research impact indicator, log(jnciav), and the normalized student project counts, log(sp/nf), and (2) between log(jnciav) and the normalized student project downloads, log(sd/nf). the results are shown in figure 2. the dependent variable in these analyses is the research impact indicator, represented by a set of six observed values—the log of values shown in the last column of table 1. the independent variable in each regression analysis, either log(sp/nf) or log(sd/nf), is also represented by a set of six observed values. figure 2 shows scatter plots of the data along with the regression line (predicted values) that is estimated using the least squares method. the markers represent observed values, namely log of the values in table 1. the other results discussed previously (r2 and the regression line parameters) are also shown in figure 2.      figure 2 results of regression analysis between undergraduate student repository activity and research impact.       the predictive linear equations are of the form  = a0 + a1x, where  = log() is the predicted log-value of research impact and x = log(sp/nf) or x = log(sd/nf). these equations can be written as power equations in terms of the original data from table 1 as  = b0*vb1, where  =  and v = sp/nf or v = sd/nf. with the values of a0 and a1 shown in figure 2, the predictive equations become:                               and          discussion of regression analysis results   at this juncture, two questions still need to be addressed: (1) how significant is the linear dependence between research impact and student repository activity? and (2) how significant are the calculated regression line parameters a0 and a1? this significance is investigated here by means of statistical hypothesis testing that is used to check the validity of a result at a certain level of significance, α. a commonly accepted significance level, also selected here, is α = 5%. a simple interpretation of the level of significance in statistical testing can be stated as follows: when accepting the hypothesis that a certain quantity is statistically significant at the α = 5% level of significance there is still a 5% chance that the hypothesis is false. (note: for brevity, the ad-hoc definition of significance level stated here is based on the alternate hypothesis, h1, rather than on the null hypothesis, h0.) the significance of regression analysis results was investigated using three statistical tests.   the strength of the linear dependence between faculty research impact and student repository activity was first verified through the p value of the observed relationship. this p value is an output of the regression function in excel that directly indicates the level of statistical significance of the relationship between the dependent and independent variables (see devore, 2000, p. 394 for more details on p value). for the level of significance selected, α = 5%, a calculated p value < 0.05 indicates that the observed relationship is significant at least at the 5% level (i.e., there is less than 5% probability that this relationship resulted by chance). the calculated p values for the two regression analyses are listed in table 2.   the strength of the linear dependence between research impact and student activity was also assessed by comparing the calculated sample correlation coefficients, , with the minimum significant value of r at the α level of significance:                     where z is the standardized normal random variable and n = 6. for α = 5%, . the test, described in detail by bendat and piersol (2010, pp. 99-101), states that there is evidence of statistical correlation at the α level of significance if the absolute value of the sample correlation coefficient is . the resulting sample correlation coefficients are compared with  in table 2.   one common type of statistical hypothesis testing uses t statistics (devore, 2000, pp. 296-301). the t statistic of a certain result to be tested for significance is compared with the critical value from t distribution. the critical value depends on the number of degrees of freedom, n, and on the level of significance, α. critical values of t distribution are tabulated in any statistics textbook. the critical t distribution value corresponding to the regression analyses performed here, with n = 4 degrees of freedom (n = n − 2 for simple linear regression, with n = 6, the sample size) and level of significance α = 5%, is tn(α)/2 = 2.776. if the absolute value of the t statistic for a certain parameter is larger than or equal to 2.776, the respective parameter is considered statistically significant at the 5% level. the regression function in excel provides t statistic values for the regression parameters, a0 and a1. these t statistics are compared in table 2 with the critical value from t distribution, t4,2.5% = 2.776.   from comparing the values in table 2, it is concluded that all parameters considered here meet all statistical tests at the 5% level of significance. therefore, there is significant linear dependence between student repository activity and faculty research impact, and the calculated linear regression coefficients can be used with confidence in a predictive model.   effect of faculty activity in dc on research impact   as inferred from several previous studies on the effect of oa on research impact (brody, harnard, & carr, 2006; gargouri et al., 2010), faculty repository activity (self-archiving of faculty papers and download counts) in dc is expected to be correlated with faculty research impact. even in the presence of significant correlation between student repository activity in dc and faculty research impact, a question arises: could this correlation be a result only of the two variables (student repository activity and faculty research impact) each being strongly correlated to faculty repository activity? if so, then faculty repository activity may be the determining factor for research impact. two variables being strongly correlated to a third variable is known as severe multicollinearity. the following analysis answers the question noted and determines whether severe multicollinearity exists in this situation.     table 2 hypothesis testing of regression analysis results at 5% level of significance statistics from regression analysis regression between log(jnciav) and log(sp/nf) regression between log(jnciav) and log(sd/nf) critical values strength of linear relationship p value 0.006 < 0.05 0.029 < 0.05 pmax = 0.05 sample correlation coefficient, r 0.937 > 0.812 0.859 > 0.812 confidence in regression parameters t statistics for a0 3.215 > 2.776 |–2.906| > 2.776 t4,2.5% = 2.776 t statistics for a1 5.346 > 2.776 3.351 > 2.776     table 3 sample correlation coefficients between various pairs of data used in this study data pairs sample correlation coefficient student project count, log(sp/nf), and research impact, log(jnciav) 0.937 faculty paper counts in dc, log(fp/nf), and research impact, log(jnciav) 0.741 student project count, log(sp/nf), and faculty paper count in dc, log(fp/nf) 0.632 student project downloads, log(sd/nf), and research impact, log(jnciav) 0.859 faculty paper downloads in dc, log(fd/nf), and research impact, log(jnciav) 0.625 student project downloads, log(sd/nf), and faculty paper downloads, log(fd/nf) 0.290     sample correlation coefficients   significant correlation indicates strong linear dependence. as discussed earlier and as shown in table 3, significant correlation exists between faculty research impact and student activity in dc (both student project counts and student project downloads) with values of the sample correlation coefficients r = 0.937 between log(jnciav) and log(sp/nf) and r = 0.859 between log(jnciav) and log(sd/nf), which are both larger than the critical value, .    to investigate the effect of faculty repository activity in dc on research impact, sample correlation coefficients between other pairs of data have been calculated using the correlation function in excel and are listed in table 3. the sample correlation coefficient between log(jnciav) and log(fp/nf) is r = 0.741 and between log(jnciav) and log(fd/nf) is r = 0.625. both values are smaller than , meaning that they do not pass the statistical test discussed before. this indicates that the correlation between faculty repository activity and research impact is not statistically significant at the 5% level, and therefore the dependence is not as strong as the one between research impact and student activity in dc.   the sample correlation coefficients between the two types of independent variables resulted as follows:   ·         between log(sp/nf) and log(fp/nf): r = 0.632, which is smaller than the corresponding correlation coefficients between each independent variable and the dependent variable, or 0.937 and 0.741 ·         between log(sd/nf) and log(fd/nf): r = 0.29, which is smaller than 0.859 and 0.625   lower correlation between the independent variables than between each independent variable and the dependent variable (research impact) indicates that there is no severe multicollinearity.     table 4 adjusted r2 between research impact indicator and repository activity in dc regression analysis adjr2 effect of adding factor 1. between log(sp/nf) and log(jnciav) 85% 86% − 44% = 42% 2. between log(fp/nf) and log(jnciav) 44% 86% − 85% = 1% 3. between log(sp/nf) & log(fp/nf), the independent variables, and log(jnciav), the dependent variable 86%   4. between log(sd/nf) and log(jnciav) 67% 82% − 24% = 58% 5. between log(fd/nf) and log(jnciav) 24% 82% − 67% = 15% 6. between log(sd/nf) & log(fd/nf), the independent variables, and log(jnciav), the dependent variable 82%         adjusted r2   the adjusted r2 (adjr2) is a modified version of r2 that is adjusted for the number of independent variables in the model and is always lower than r2. adjr2 is one of the results of the regression analysis in excel and is useful in multilinear regression analysis. the difference between adjr2 of a bilinear regression analysis with independent variables x1 and x2 and the adjr2 of a simple linear regression using only x1 indicates by how much the regression model is improved by adding the variable x2.   the resulting values of adjr2 from the simple linear regression analyses discussed in the previous subsection are included in the second column of table 4 (analyses 1 and 4). two additional simple linear regression analyses were performed between the components of faculty activity in dc (independent variables) and the research impact (dependent variable). the resulting adjr2 values are listed in table 4 (analyses 2 and 5). two bilinear regression analyses were also performed, and the resulting adjr2 is listed in table 4:   ·         log(sp/nf) and log(fp/nf) as independent variables versus log(jnciav); see analysis 3 ·         log(sd/nf) and log(fd/nf) as independent variables versus log(jnciav); see analysis 6   finally, the third column of table 4 shows by how much each independent variable would improve a linear regression model between another independent variable and the research impact. for example, a linear model linking log(fp/nf) and log(jnciav) is improved by 42% (86% − 44%) by adding log(sp/nf) in the model, while a linear model linking log(sp/nf) and log(jnciav)is improved by only 1% (86% − 85%) by adding log(fp/nf) in the model. from these results, it is clear that the student paper downloads (sd) and student paper counts (sp) contribute more significantly to the bilinear regression model for predicting research impact than the corresponding quantities from faculty papers deposited in the dc.   it is therefore safe to consider that, for the data analyzed here for the six csm departments, the impact of faculty research can be correlated with the student research activity in digitalcommons@cal poly with little interference from the csm faculty deposits in dc. note that this conclusion does not imply that the open availability of faculty works in dc has little influence on the faculty research impact. in this study, the correlation between faculty repository activity and research impact resulted weaker than the correlation between student repository activity and research impact this is probably due to the fact that faculty also participate and deposit oa publications in other repositories (disciplinary or commercial).   conclusion   in the context of the ongoing conversation surrounding the role of irs, this study investigates statistically if an ir focused on stewarding, preserving, and disseminating materials created by the student community has a positive impact on the visibility and performance of faculty scholarship, independent of faculty’s participation in the ir. this is done by analyzing two distinct samples of publications:   1.       a group of faculty publications from six csm academic departments for which research impact is calculated based on wos citation data 2.       a group of csm student publications (senior projects) from the same six csm departments that are in dc   the main conclusion of the statistical analysis is that student repository activity, quantified through undergraduate senior student projects deposited in an open ir and the download counts of these projects, is significantly correlated with the research impact of faculty publications, expressed as a measure of the citation counts. the authors postulate two factors that may contribute to this strong dependence.   the first factor is that undergraduate student senior projects follow (and sometimes anticipate) the topics of faculty research. having student work deposited in an open ir, where it is easily discovered and accessed may constitute an effective conduit for promoting faculty research.   the second factor is rooted in the causality between student research quality and faculty research quality. for the departments analyzed, the results may indicate that the student research quality, quantified through download counts, reflects the quality of faculty research. it can be argued that the number of project downloads may not reflect quality of scholarly output on the same level as citations; however, downloads are still considered a significant quality indicator (haustein, 2014). haustein’s study surveyed bibliometricians to assess their opinions on the potential of alternative metrics (altmetrics). while the bibliometrics experts surveyed expressed mixed opinions on the value of altmetrics, 72% still valued download counts as a valuable source of impact data. moreover, student project citations are not easily tracked; therefore, no other indicator was available for this study to infer student research quality besides ir downloads. faculty repository activity in dc, while also positively correlated with the faculty research impact, had no significant effect on the correlation between student repository activity and faculty research impact.   to maintain some uniformity in the data, the study was performed on a coherent group of departments from the same college (csm). this resulted in a relatively small sample of data (n = 6), which may be regarded as a limitation of the study. to overcome this issue, all results were thoroughly checked for statistical significance.   though no definitive conclusion can be drawn based on the analysis of only six academic departments, the present study can be viewed as a first step in a broader research process that can be extended to investigate, among other factors, the effect of master’s theses ir exposure, direct correlation between individual faculty research impact and student advisees’ ir activity, and differences in scholarly communication practices across disciplines.   references   bendat, j. s. & piersol, a. g. (2010). random data, analysis and measurement procedures. (4th ed.). hoboken, nj: wiley.   bornmann, l. & daniel, h. d. (2008). what do citation counts measure? a review of studies on citing behavior. journal of documentation, 64(1), 45-80. http://dx.doi.org/10.1108/00220410810844150    brody, t., harnad, s. & carr, l. (2006). earlier web usage statistics as predictors of later citation impact. journal of the american association for information science and technology (jasist), 57(8), 1060-1072.  http://dx.doi.org/10.1002/asi.20373   clarivate analytics (2017). incites indicators handbook. retrieved from http://researchanalytics.thomsonreuters.com/m/pdfs/indicators-handbook.pdf   cni (2017). rethinking institutional repository strategies: executive roundtable report, may 2017. retrieved from http://www.cni.org/go/rethinking-irs-cni-er-s17   devore, j.l. (2000). probability and statistics for engineering and the sciences (5th ed.). pacific grove, ca: duxbury.   garfield, e. (1972). citation analysis as a tool in journal evaluation. science, 178, 471-479. retrieved from http://www.garfield.library.upenn.edu/essays/v1p527y1962-73.pdf   garfield, e. (1973). citation frequency as a measure of research activity and performance. essays of an information scientist, 1, 406-408. retrieved from http://www.garfield.library.upenn.edu/essays/v1p406y1962-73.pdf   garfield, e. (1988). can researchers bank on citation analysis? essays of an information scientist, 11, 354-356. retrieved from  http://www.garfield.library.upenn.edu/essays/v11p354y1988.pdf   gargouri, y., hajjem, c., larivière, v., gingras, y., carr, l., brody, t. & harnad, s. (2010). self-selected or mandated, open access increases citation impact for higher quality research. plos one, 5(10): e13636. https://doi.org/10.1371/journal.pone.0013636   gregerman, s. e., lerner, s. j., von hippel, w. & nagda, b. a. (1998). undergraduate student faculty research partnerships affect student retention. the review of higher education, 22(1), 55-72. http://dx.doi.org/10.1353/rhe.1998.0016    gross, p. l. k. & gross, e. m. (1927). college libraries and chemical education. science, 66, 385-389. retrieved from  http://www.garfield.library.upenn.edu/papers/grossandgross_science1927.pdf   haustein, s., peters, i., bar-ilan, j., priem, j., shema, h., & terliesner, j. (2014). coverage and adoption of altmetrics sources in the bibliometric community. scientometrics, 1-19. http://dx.doi.org/10.1007/s11192-013-1221-3   lei, s. a. & chuang, n. (2009). undergraduate research assistantship: a comparison of benefits and costs from faculty and students' perspectives. education, 130(2), 232-240.   lynch, c. (2017). updating the agenda for academic libraries and scholarly communications. college & research libraries, 78(2):126-130. http://dx.doi.org/10.5860/crl.78.2.16577   massey, f. j. (1951). the kolmogorov-smirnov test for goodness of fit. journal of the american statistical association, 46(253), 68-78.  http://dx.doi.org/10.1080/01621459.1951.10500769   moed, h.f. (2005). citation analysis in research evaluation. dordrecht: springer netherlands.   seymour, e., hunter, a-b., laursen, s. l. & deantoni, t. (2004). establishing the benefits of research experiences for undergraduates in the sciences: first findings from a three-year study. science education, 88(4), 493-534.  http://dx.doi.org/10.1002/sce.10131     appendix notations symbol description α level of significance a0 intercept of the regression line a1 slope of the regression line adjr2 adjusted r2 cni coalition for networked information csm college of science and mathematics at cal poly fd number of faculty papers downloads in dc fp number of faculty papers deposited in dc ir institutional repository isi institute for scientific information jec journal expected citations jnci journal normalized citation impact jnciav average of jnci for all faculty publications in one department n number of degrees of freedom n sample size nf number of faculty in a department oa open access publication qq-plot quantile-quantile plot r sample correlation coefficient r2 coefficient of determination sd number of undergraduate student project downloads from dc sp number of undergraduate student projects deposited in dc tc times cited (or citation count for a given paper) wos web of science x, x1, x2 independent variable y dependent variable predicted dependent variable z standardized normal random variable   evidence summary   blind user experiences of us academic libraries can be improved by more proactive reference service delivery   a review of: mulliken, a. (2017). there is nothing inherently mysterious about assistive technology: a qualitative study about blind user experiences in us academic libraries. reference & user services quarterly, 57(2), 115-126. https://doi.org/10.5860/rusq.57.2.6528     reviewed by: alisa howlett coordinator, evidence based practice university of southern queensland library springfield, queensland, australia email: alisa.howlett@usq.edu.au   received: 1 mar. 2019                                                                    accepted: 25 apr. 2019      2019 howlett. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29565     abstract   objective – to explore blind users’ experiences with academic libraries.   design – qualitative questionnaire.   setting – academic libraries within the united states of america.   subjects – 18 individuals who are legally blind, have experience relying on a screen reader to access the internet, and have used an academic library either online or in person within the previous two years.   methods – an open-ended questionnaire was administered via telephone interview. interviews were recorded, transcribed and analysed using an inductive approach to identify themes using hill et al.’s (2005) approach.   main results – the author found seven themes in the interview data: experiences working with reference librarians in person, difficulty with library websites, screen reader use during reference transactions, preferences for independence, using chat, interactions with disability officers, and challenges of working with citation styles.    conclusion – the study concluded that academic libraries and librarians should be more proactive when approaching reference services for blind users. the author offered suggestions for practice about how to improve blind user experiences of academic libraries.    commentary   accessibility of websites and electronic resources, particularly for visually impaired users, is an ongoing concern for libraries. a number of investigations in recent years have highlighted a need to continually improve online service usability and for librarians to advocate to the publishers the rights of visually impaired persons (byerley & chambers, 2002; haanperä & nieminen, 2013; sahib, tombros, & stockman, 2011; yoon, newberry, hulscher, & dols, 2014). however, as mulliken (2017) notes, these studies tend to focus on what the library provides, such as databases and indexing services, or observation of user behaviours, rather than investigating user’s needs (byerley & chambers, 2002; haanperä & nieminen, 2013). mulliken’s study addresses this knowledge gap by exploring the user experience of us academic library reference services and websites by blind users.    a critical review form developed by letts et al. (2007) was used to identify strengths and weaknesses of mulliken’s study. the study’s approach was clearly articulated and explored the topic from a user experience perspective, rather than the service provider’s, alternative to previous studies. however, the study was not as transparent as it could have been: the questionnaire was not published with this article, and the author also did not indicate whether the sampling continued until data saturation. with regards to the study’s findings, presentation of themes arising from the data would have benefited from additional structure as well as a discussion of how the themes potentially impact the overall user experience. this would have allowed the author to construct a more coherent narrative of the data.    this study concluded that librarians responsible for providing academic library reference services need to be more proactive in their approach to blind user experience. the author outlined the following suggestions for implementation: firstly, librarians need to build their understanding of using screen readers. secondly, the author recommends a local expert model of service delivery — similar to the way that copyright services is supported in many academic libraries — whereby a librarian who is experienced with and keeps up to date with screen readers and related technology and accessibility issues shares this knowledge with others. thirdly, the author suggests that library teams ought to engage in discussion about the accessibility and usability of library reference services and electronic resources. furthermore, collaboration opportunities may also be explored between library services and other areas of the university, such as the disability office, to coordinate accessibility more effectively. all of these suggestions are worth exploring by practitioners and library leaders in order to raise awareness of accessibility issues with their staff and identify areas for service improvement. though the author does not offer direction for future research, the study itself highlights an opportunity to build upon its findings.   references   byerley, s., & chambers, m. b. (2002). accessibility and usability of web-based library databases for non-visual users. library hi tech, 20(2), 169-178. https://doi.org/10.1108/07378830220432534   haanperä t., & nieminen m. (2013). usability of web search interfaces for blind users – a review of digital academic library user interfaces. in c. stephanidis & m. antona (eds.), universal access in human-computer interaction. applications and services for quality of life (pp. 321-330). berlin: springer. https://doi.org/10.1007/978-3-642-39194-1_38   hill, c. e., knox, s., thompson, b. j., williams, e. n., hess, s. a., & ladany, n. (2005). consensual qualitative research: an update. journal of counseling psychology, 52(2), 196–205. https://doi.org/10.1037/0022-0167.52.2.196   letts, l., wilkins, s., law, m., stewart, d., bosch, j., & westmorland, m. (2007). critical review form – qualitative studies (version 2.0). retrieved from http://www.peelregion.ca/health/library/eidmtools/qualreview_version2_0.pdf   sahib, n. g., tombros, a., & stockman, t. (2012). a comparative analysis of the information‐seeking behavior of visually impaired and sighted searchers. journal of the american society for information science and technology, 63(2), 377-391. https://doi.org/10.1002/asi.21696   yoon, k., newberry, t., hulscher, l., & dols, r. (2013). call for library websites with a separate information architecture for visually impaired users. proceedings of the american society for information science and technology, 50(1), 1-3. https://doi.org/10.1002/meet.14505001100     evidence summary   use of diverse online resources amongst politically active university students fosters civic knowledge integration   a review of: soe, y. (2018) understanding politics more thoroughly: how highly engaged young citizens use the internet for civic knowledge integration. first monday, 23(6), 1-17. http://doi.org/10.5210/fm.v23i6.7923     reviewed by: joanne muellenbach director, jay sexter library touro university nevada henderson, nevada, united states of america e-mail: joanne.muellenbach@tun.touro.edu   received: 20 nov. 2018                                                                 accepted: 27 feb. 2019      2019 muellenbach. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29529     abstract   objective – to examine the process by which university students with a high interest in politics and public affairs incorporate new information into their understanding of politics and public affairs, a process referred to as civic knowledge integration.   design – this study utilized a qualitative research design that consisted of focus group interviews and essay questions.   setting – two large four-year midwestern public universities and two four-year east coast private universities in the united states of america in 2008 and 2010.   subjects – a total of 65 undergraduate and graduate (masters) students participated in the focus group interviews and answered essay questions by e-mail.   methods – in 2008, the researcher conducted 11 focus groups consisting of 5 to 7 participants per group. in 2010, additional data were collected from students at another large four-year midwestern public university who responded by e-mail to essay questions that were adapted from those used in the focus groups. recruitment of participants was achieved by contacting professors of media and political science at the universities and targeting students with interest in media, politics, and public affairs, and who were politically active. course credit or a small monetary incentive was offered to students as compensation. data resulting from the focus group and essay responses were combined and imported into the qda miner software. data analysis, which used some techniques of grounded theory, was conducted in two phases. in the first phase, 120 analyzable subsets were identified, and open coding of 36 subsets was performed to determine themes. these themes were then modified and renamed using an axial and selective coding process. examples of resulting topics included collaborative layering of ideas, comparison of differing viewpoints, and monitorial scanning. the second phase involved coding of the 120 subsets, and 65 subgroups that focused on civic knowledge integration were identified. ultimately, open, thematic coding of the 65 subsets was performed to identify comments that contained the most common themes.   main results – an analysis of the data revealed that participants used the resulting themes as self-guided learning strategies when searching the internet for civic knowledge integration, the process by which university students with a high interest in politics and public affairs incorporate new information into their understanding of these areas. one of the strategies used was a two-step process of monitorial scanning and opinion sampling. monitorial scanning involves the careful selection of search engines in order to scan the news and determine their potential levels of interest, and the use of online encyclopedias such as wikipedia to locate background information and other details. opinion sampling involves the process of sorting the sources found, such as blogs and candidates’ web pages. another strategy used was verification (cross-checking), which consisted of checking multiple sources to find more information on a particular news item or news show, such as those watched on cnn.com. comparison of differing and opposing viewpoints was another strategy used, that involved the comparing of information about political candidates' perspectives or views to justify their own opinions. finally, collaborative layering of ideas was a strategy that involved participation in online forums, such as facebook. this strategy provided participants with the opportunity to express their thoughts and opinions globally, and to contribute to a change in a set practice.     conclusion – through the use of strategies for self-guided learning, participants were able to add new information to their knowledge base and to develop new points of view. these students developed advanced search strategies and took pleasure in finding opposing perspectives, and as a result, enhanced their critical thinking skills. the conclusions also increased general knowledge of why young people used specific online platforms, information resources, or social media sites to enhance their understanding of politics and public affairs. these findings may also challenge media and political science to investigate the long-term effects of self-guided learning strategies for civic knowledge integration practiced by some young people.   commentary   this study examined how university students with a high interest in politics and media use the internet to learn about politics and public affairs, a process referred to as civic knowledge integration. the findings build upon experiences reported in a study by moeller, kühne, and de vreese (2018) that surveyed dutch youth who had not yet reached voting age. moeller et al. found that exposure to digital news influenced their political participation, and developed their sense of civic duty and information efficacy. in another study, vromen, loader, xenos, and bailo (2016) surveyed young people in australia, the united kingdom, and the united states of america, and found that social media had become a regular source of information on politics and that youth engagement through social media is a form of political participation worthy of our attention.      glynn’s (2006) “critical appraisal checklist” confirmed that this study was valid in several areas: institutional review board approval was received, and focus group interview and essay questions were provided. also, results provided a descriptive summary, with applications and areas for further research. however, concerning the study population, it’s unclear if there was a total of 65 participants for the focus group interviews and online essays, or just for the focus groups. knowing the exact number of participants in the focus groups and responders to essay questions could increase confidence in the study design overall. also, the homogeneity of the study participants may have been a limitation. knowing whether the survey tools were pre-tested would also ensure other researchers that the questions were understandable and designed to elicit the desired outcomes. the results of the data analysis described two types of coding used – axial coding and open-thematic coding – both of which are techniques of grounded theory design. a study by moghaddan (2006) describes axial and open thematic coding as two different approaches, and since it is unlikely that both were used, the coding process is unclear. in terms of the results, considering the ten-year gap between collecting and publishing the data, and given the degree to which the internet and the social media landscape has changed during this period, one may question how relevant these findings are to how today’s young people acquire, evaluate, and share political and public affairs news.    there is a significant focus on “fake news” these days, and while this may not have been a big issue in 2008 when the study began, these results could inspire librarians who are interested in this topic. librarians could incorporate “fake news” into their curricula. they could also highlight the self-guided learning strategies used by study participants, and create exciting and engaging active learning sessions in the areas of monitorial scanning, opinion sampling, verification, comparing viewpoints, and collaborative layering of ideas.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154   moeller, j., kühne, r., & de vreese, c. (2018). mobilizing youth in the 21st century: how digital media use fosters civic duty, information efficacy, and political participation. journal of broadcasting & electronic media, 62(3), 445-460. https://doi.org/10.1080/08838151.2018.1451866   moghaddam, a. (2006). coding issues in grounded theory. issues in educational research, 16(1), 52-66. retrieved from http://www.iier.org.au/iier16/moghaddam.html      vromen, a., loader, b. d., xenos, m. a., & bailo, f. (2016). everyday making through facebook engagement: young citizens' political interactions in australia, the united kingdom and the united states. political studies, 64(3), 513-533.                https://doi.org/10.1177/0032321715614012     research article   research support priorities of and relationships among librarians and research administrators: a content analysis of the professional literature   cara bradley research & scholarship librarian university of regina regina, saskatchewan, canada email: cara.bradley@uregina.ca   received: 1 aug. 2018    accepted: 1 oct. 2018      2018 bradley. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29478     abstract   objective this research studied the recent literature of two professions, library and information studies (lis) and research administration (ra), to map the priorities and concerns of each with regard to research support. specifically, the research sought to answer these research questions: (1) what are the similarities and differences emerging from the lis and ra literatures on research support? (2) how do librarians and research administrators understand and engage with each other’s activities through their professional literatures? (3) do whitchurch’s (2008a, 2008b, 2015) concepts of bounded-cross-boundary-unbounded professionals and theory of the “third space” provide a useful framework for understanding research support?   methods the research method was a content analysis of journal articles on research-related topics published in select journals in the lis (n = 195) and ra (n = 95) fields from 2012-2017. the titles and abstracts of articles to be included were reviewed to guide the creation of thematic coding categories. the coded articles were then analyzed to characterize and compare the topics and concerns addressed by the literature of each profession.     results only two (2.2%) ra articles referred to librarians and libraries in their exploration of research support topics, while six (3.1%) lis articles referred to the research office or research administrators in a meaningful way. of these six, two focused on undergraduate research programs, two on research data management, and two on scholarly communications. thematic coding revealed five broad topics that appeared repeatedly in both bodies of literature: research funding, research impact, research methodologies, research infrastructure, and use of research. however, within these broad categories, the focus varied widely between the professions. there were also several topics that received considerable attention in the literature of one field without a major presence in that of the other, including research collaboration in the ra literature, and institutional repositories, research data management, citation analysis or bibliometrics, scholarly communication, and open access in the lis literature.   conclusion this content analysis of the lis and ra literature provided insight into the priorities and concerns of each profession with respect to research support. it found that, even in instances where the professions engaged on the same broad topics, they largely focused on different aspects of issues. the literature of each profession demonstrated little awareness of the activities and concerns of the other. in whitchurch’s (2008a) taxonomy, librarians and research administrators are largely working as “bounded” professionals, with occasional forays into “cross-boundary” activities (p. 377). there is not yet evidence of “unbounded” professionalism or a move to a “third space” of research support activity involving these professions (whitchurch, 2015, p. 85). librarians and research administrators will benefit from a better understanding of the current research support landscape and new modes of working, like the third space, that could prove transformative.     introduction   support for teaching and research is the core mission of academic libraries worldwide, and services offered to further these activities should be based on sound evidence. a significant number of studies have been conducted on library support for faculty teaching and student learning, and hundreds (if not thousands) of research papers have been published about information literacy. as well, the importance of collaborating with others on campus (units, students, and faculty) in developing and delivering support for student learning has been well-documented (sproles, detmering, & johnson, 2013).   there has been less evidence collected about how academic libraries can best support campus research. major library organizations worldwide have authored reports—most notably the research libraries uk’s re-skilling for research and the association of research libraries’ new roles for new times—that suggested a range of research support services that libraries could be offering. these reports noted a need to increase research support capacity and services, as well as the benefits of increased collaboration. new roles for new times reported that “many large and complex initiatives require collaboration between the library and other campus units. for example, one interviewee noted that a ‘library-centric approach to e-science is doomed to fail,’ citing the need for the library, information technology, the university’s office of research, and other campus units to define their respective roles and work together toward mutual goals” (association of research libraries, 2013, p. 13).   the present study picked up on this call for collaboration with university research offices by exploring the priorities and concerns of the research administration (ra) literature and the library and information studies (lis) literature, and analyzing the relationship between the two. in the context of this study, research administration is used as a broad term to describe staff working in campus research offices who play a role in facilitating academic research. specific position titles vary, but include research office directors, research funding officers, research facilitators, research contracts officers, research compliance officers, and research communication staff, among others. their work involves “formulating, developing, supporting, monitoring, evaluating and promoting the research and research-degree activity of their universities” (hockey & allen-collinson, 2009, p. 142), giving them a holistic view of system-wide campus research support needs.   both librarians and research administrators regard themselves as integral to the research mission of the university, but there has been little documented dialogue between the two professions in the literature of either field. this study analyzed the recent literature of each profession to map the priorities and concerns of each with regard to research support. it also explored the extent to which these two professions are collaborating or at least referencing one another’s activities within their professional literatures. ultimately, the goal is to provide a picture of the research support landscape and to suggest potential approaches to library support for academic research into the future.   literature review   research support relationships among librarians and research administration professionals   both librarians and research administrators have explored research support issues within their own professional bodies of literature, but the relationships between the two professions in supporting institutional research are only infrequently addressed. in their 2010 report, research support services in uk universities, the uk research information network (rin) reported that in the four universities they surveyed, the library and the research office “provide services to researchers from very different perspectives” (p. 18), with the research office perceived as actively engaging with researchers early in the research process, in a way that librarians do not. one of the report’s recommendations is that “libraries should work together with research offices to review their provision of support for researchers” (rin, 2010, p. 18). a year later, in a report commissioned by oclc that incorporated the rin data and other sources, maccoll and jubb (2011) were blunt in their assessment of the situation, writing that “it is hard to avoid the conclusion that libraries in recent years have been struggling to make a positive impact on the scholarly work of researchers, but having relatively little effect” (p. 5). they went on to report that institutional repositories are a major example of a library-initiated research support service that has garnered little faculty interest and support. they pointed to the need for “mutual reinforcement” (p. 10) among research support services, especially the research office, as essential for sustaining and advancing the service.   corrall’s study (2014) appeared three years after the oclc report and found evidence that some libraries were heeding maccoll and jubb’s (2011) suggestion. corrall studied the websites and public documents of uk academic libraries to explore how they are organizing research support resources and services. one of the themes emerging from the study was that of “boundary-spanning activities” (p. 35), which involves stakeholders from across the institution (including the library and the research office) working on large research support projects or policy issues. corrall reported that “operational convergence,” in which academic units collaborate to achieve larger goals, are “more prevalent than ever, with libraries extending and deepening their collaborations and partnerships beyond it and educational development colleagues to other professional services, such as research offices” (p. 37). such collaboration, as evidenced by the literature, seems thus far to be centred around research data management activities (antell, foote, turner, & shults, 2014; cox & pinfield, 2013; o’brien & richardson, 2015; verbaan & cox, 2014).   cox and verbaan (2016) conducted the most targeted study to date on librarians and research administrator relationships in supporting university research. they interviewed librarians, it staff, and research administrators to investigate how each group sees research and their role in supporting it. they found significant differences in how each profession viewed research, concluding that “in most respects what is apparent is the lack of common ground between professional services” in how they conceive of research and their role in it (p. 324). they noted that additional investigation is needed to determine how these professions conceptualize research and how this evolves over time. in the context of research data management (rdm), they encouraged further exploration of “how these differing perspectives shape collaborations between professional services” (p. 324). the present study undertook this challenge in the broader context of research support, as viewed through the literature of the two professions.   whitchurch’s theoretical model   whitchurch developed a theoretical model to explain professional roles in higher education that provides a useful framework for exploring librarians’ professional culture and the relationship between librarians and research administrators. she observed that higher education professionals are “not only interpreting their given roles more actively . . . but are also moving laterally across functional and organizational boundaries to create new professional spaces, knowledges and relationships” (whitchurch, 2008b, p. 379). through her interviews with higher education professionals, she developed a taxonomy to describe variations in the approaches to the fixed structural conditions attached to a profession and the resulting degree of engagement across organizational boundaries. whitchurch (2008a) described three categories:   ·         “bounded professionals” are those firmly embedded “within the boundaries of a functional or organizational location that they had either constructed for themselves, or which had been imposed upon them” (p. 377). bounded professionals work in prescribed roles, and are guided by set standards, rules, and structures. ·         “cross-boundary professionals,” as the name suggests, are open to working beyond the boundaries of their profession. boundaries are still a “defining mechanism for them,” much as for bounded professionals, but they navigate the boundaries of more than one profession, “recogniz[ing], and actively us[ing] boundaries to build strategic advantage and institutional capacity” (p. 377). they use their knowledge of more than one bounded space (or profession) to “construct their identity, [are] likely to display negotiating and political skills, and also to interact with the external environment” (p. 377). ·         “unbounded professionals” are not constricted by boundaries and “have a more open-ended and exploratory approach to the broadly based projects with which they were involved” (p. 377). whitchurch (2008b) went on to theorize that “cross-boundary” and “unbounded” professionals create a “third space” that resides outside of existing boundaries (p. 386). the notion of third space has its origins in the work of post-colonial theorist homi bhabha (1994),  who used the term to describe the “boundary zone in which two cultures meet, hybrid identities take shape, and new discourses are created” (verbaan & cox, 2014, p. 212). whitchurch (2015) picked up on this concept as “a way of problematising binary approaches to higher education communities, and a lens through which to view the roles, identities and working practices of staff in contemporary institutions” (p. 96).   in whitchurch’s conception, the third space transcends employment category, discipline or field, and organizational structure, and is instead a multi-professional space for work that reaches beyond boundaries. cross-boundary professionals dip in and out of the third space as needed to achieve goals, whereas unbounded professionals consistently work and thrive “beyond functional and disciplinary boundaries” in the third space (whitchurch, 2015, p. 85). in higher education, the third space is often the arena where new strategies and directions are formed, and where projects that transcend the scope or capacity of a single unit are found.   whitchurch’s taxonomy of bounded, cross-boundary, and unbounded professionals, and the third space has only rarely been applied to lis professionals. ferguson and metz (2003) used the third space model to study the relationship between library and it services. verbaan and cox (2014) applied the third space framework to rdm, positing that it could be viewed as “an unclaimed ‘third space’ . . . where staff from different professional cultures and departments meet,” and where “the actors in this rdm space would need to invent new hybrid identities, rather than be able to stay within relatively clear, familiar professional structures” (p. 218). the present research built on verbaan and cox’s application of whitchurch’s theory to rdm, investigating the extent to which it can be applied to the study of the research support relationship of lis and ra professionals, as evidenced through the professional literature of these fields. aims   this study analyzed the recent literature of two professions, lis and ra, in order to better understand the priorities and concerns of each with regard to research support. it also explored the relationship between these two professional groups and considered the usefulness of whitchurch’s theories of professional roles in higher education in understanding this relationship. the questions guiding the research were:   1. what are the similarities and differences emerging from the lis and ra literatures on research support?  2. how do librarians and research administrators understand and engage with each other’s activities through their professional literatures? 3. do whitchurch’s concepts of bounded-cross-boundary-unbounded professionals and her theory of third space provide a useful framework for understanding these research support relationships?   ultimately, this research aimed to suggest new ways for academic librarians to support campus research and to work with a key stakeholder, research administration.   methods   this study was conducted using a content analysis of journal articles published in select journals in the lis and ra fields. as there are only three main ra journals (journal of research administration, research management review, and journal of research practice), they are all included in the study. the lis journals included are the major journals publishing about academic librarianship in north america (journal of academic librarianship and college & research libraries), and two canadian journals that publish across library sectors, but include academic libraries (evidence based library and information practice as well as partnership: the canadian journal of library and information practice), in an attempt to increase canadian content included in the analysis. only articles addressing research support topics were included in the analysis (i.e., articles on topics such as undergraduate instructional techniques were excluded). the content analysis included articles published in the five years prior to this study (july 2012-june 2017); this limitation created a manageable project and, given the speed with which technology, research, and libraries have changed in recent years, generated the most relevant results. the final analysis included 195 articles from the lis literature and 95 articles from the ra literature.   a student research assistant was hired to create a project in nvivo 11 pro containing pdf versions of all articles from the journal issues under consideration. the researcher reviewed the titles and abstracts of all the articles in the database and removed those from the lis literature that clearly addressed topics other than research support. this preliminary review of the articles also guided the creation of initial coding categories, with the understanding that more categories would be added as needed when the articles were analyzed in more detail. the parent/child node feature of nvivo was used to capture hierarchical relationships (e.g., coding to child node “open access” was also captured and counted under parent node “scholarly communication”). the goal of the coding activity was to identify major topics addressed in each article, rather than every topic mentioned in passing. thus, the bulk of the coding was performed on the abstracts, but every paper was skimmed in its entirety to ensure that other major topics were not missed. this approach was productive, as there were several instances where major topics emerged that were not highlighted in the abstract. it was also useful in instances where the abstract did not provide sufficient clarity to allow placement of a topic at the appropriate place in a hierarchical relationship, for example, whether a topic should be coded as “research grants” or the more specific subnode “requirements.”   the student and the researcher independently coded 10 articles and then compared results. there were three inconsistencies in this initial round of coding, all related to coding within hierarchical relationships, that is, when to prefer specific (child) rather than broad (parent) nodes. after discussion and an agreement to code to the most specific subnode possible, a further 10 articles were coded and consistency was reached. after this, the researcher and student assistant coded independently, but consulted frequently. additional coding categories were added as needed during the coding process in the rare instances where topics emerged that had not been foreseen in the initial coding category construction. the need to add additional coding categories emerged for two reasons: (1) a topic emerged in the ra literature that the researcher had not anticipated because of a lack of familiarity with this body of literature (e.g., research clusters); (2) the need for subnodes was not clear from the initial screening of abstracts (e.g., data management plans, rather than just research data management).   the coded articles were then analyzed to ascertain and compare the topics and concerns addressed by the literature of each profession. they were also studied to determine whether research administrators are aware of library expertise and resources related to their concerns (and vice versa), as well as to suggest potential areas for further library activity to support the research enterprise.   this coding work was supplemented by word frequency searches, in which nvivo was used to search the pdfs of all included articles to calculate how often terms occurred within the articles of each profession, providing another basis for comparison. word frequency searches did not provide the sophisticated analysis achieved by careful coding of the data, but they did offer another way of understanding similarities and differences in the attention each body of professional literature devotes to various research support topics.  results   references to each profession in the literature of the other   one question of primary interest was the extent to which the ra literature refers to librarians and libraries in its exploration of research support topics. this question could not be satisfactorily addressed by word frequency searches due to the number of false positives and was instead addressed in the coding. results revealed that only two of the 93 ra articles (2.2%) analyzed included libraries or librarians as a major topic. one article described a fee-based library editing service provided as part of an institution’s writing support for faculty (russell-simmons et al., 2016). the reference was relatively brief, and was an outlier in that it highlighted a relatively unique service, as compared with services more commonly found in academic libraries (rdm, open access publishing, impact or metrics support, and others). the other article provided a broad overview of library resources and facilities as evidence of institutional commitment to and support of research (masango, 2015). it was, however, consistent with the article by russell-simmons et al. in that it did not make any reference to research support services that dominated the lis literature on the topic, instead focusing exclusively on physical library space and collections (both print and electronic).   also of interest was the question of how often the library literature refers to the research office or research administrators as providers of services needed by researchers, or as potential partners for library research support service provision. of the 195 lis articles included in the study, six (3.1%) referred to the research office or research administrators in a meaningful way. two of the six focused on undergraduate research programs, with one reporting on a survey of research coordinators and attempts to bridge the gap between research offices and libraries (hensley, shreeves, & davis-kahl, 2015), and the other describing a study of library support for undergraduate research programs (hensley, shreeves, & davis-kahl, 2014). in the latter, some librarian survey respondents noted communication failures between the library and the research office: “one library cited lack of communication between the library and the undergraduate research office, one had attempted to convince the undergraduate research office of the value of library-specific support only to have failed in the endeavor” (hensley et al., 2014, p. 431).   two of the other lis articles that highlighted the role of the research office or research administrator focus on rdm. antell et al. (2014) surveyed arl science librarians and found that some respondents reported working in institutions where rdm services were provided by the campus research office. many respondents were uncertain about the role of various units on their campus in providing rdm support, leading antell et al. (2014) to conclude that “efforts to increase communication among campus offices and library departments might well be beneficial in reducing librarians’ uncertainty and, more important, in promoting more efficient coordination of data management initiatives” (p. 571). verbaan and cox (2014) theorize about the roles of various professions—librarians, research administration, and it—in offering rdm services and suggest that rdm might emerge as a “new intra-professional space” (p. 211).   the remaining two lis articles writing about the role of research offices or administrators focused on scholarly communications issues. nariani’s (2013) study of the role of academic librarians in promoting deposit in open access (oa) repositories noted in passing that “a more concerted effort amongst librarians and research officers is required to convey information on scholarly communication issues” (p. 81). in their survey of academic library support for scholarly publishing, hansson and johannesson (2013) reported that one of their librarian interviewees highlighted bibliometrics as a way for the library to connect with the research office: table 1 terms appearing in 100 most frequent word lists of both lis and ra journal articles term position in top 100 lis terms number of occurrences in lis journal articles (n = 195) position in top 100 ra terms number of occurrences in ra journal articles (n = 95) journal 3 6463 4 1605 data 4 6112 24 848 student 9 3609 13 1120 publish 16 2925 18 981 public 22 2486 37 728 review 27 2120 19 973 community 45 1503 5 1526 policy 50 1454 71 516 fund 64 1184 20 959 impact 82 1019 49 612     “the research administrators were less knowledgeable about how to measure publication quality, so the issue was given to the library. . . . that is a bit how we got that relation to them [the research administrators], that we are helping them” (p. 237). thus, scholarly communication, rdm, and undergraduate research programs were the few topics for which the lis literature comments on the role of the research office or administrator in research support.   common topics   the most basic level of investigation in a content analysis study is word frequency. searches were run to include stem words (e.g., “publish” included “publish,” “publisher, “publishers,” and “publishing”). there were several terms that appeared with similar frequency in the literatures of the two disciplines (see table 1).    these common keywords provided an initial snapshot of the frequency of topics occurring in the literature of the two professions, but lacked the nuance and depth achieved by careful reading and coding of the journal articles (see table 2 for coding summary). the coding process revealed five broad topics that appeared repeatedly in both bodies of literature and provided the opportunity to characterize and compare the references found within each field.   research funding was a frequently occurring topic in both the lis and ra literatures; there was however virtually no overlap in the emphasis of research funding articles between the two fields. the lis literature was concerned almost exclusively with funding issues related to oa of both publications and research data, including funding agency requirements, article processing charges, and author funds. the few exceptions to this oa focus were two articles on the role of librarians in grant-funded research projects, and one about librarians offering research funding workshops for graduate students. the ra literature was, by contrast, much more diverse. oa funding issues were never mentioned in the ra journal articles; instead, topics of interest included collaborative grant writing with community partners (two articles); internal funding programs to build capacity for larger external applications (three articles); institutional readiness or support for grant applications (two articles); researcher readiness/support (two articles); and one article each on return on investment for grants, estimating chances of grant success, role of table 2 coding summary—number of articles coded to each topic topic library journal articles (n = 195) research administration journal articles (n = 95) author identifiers 2 0      orcid 2 0 citation managers 4 0 copyright 11 0 data mining 8 0 digital humanities 7 0 gis-geographic information systems 4 0 growing research university 3 3      research clusters 0 2 information needs of researchers 78 6 institutional research activity or profile 7 2 institutional research strategy or culture 13 18 librarian role in institutional research 97 0 library collections 67 0      digital collections 30 0 mentoring 2 5 peer review 8 0 repositories 37 0      institutional 27 0      subject or discipline specific 8 0 research administrator or office role in institutional research 3 14 research collaboration 10 19      interdisciplinary collaboration 3 6      international collaboration 0 2      other collaboration 2 5      private and public collaboration 0 4 research costs 7 1      indirect costs of research 6 1 research data management 27 1      data management plans 3 0 research ethics 4 39      conflict of interest 1 2      plagiarism 1 2      research ethics board 0 1 research funding 18 12      research grants 15 9           application for 2 1           evaluation of 0 0           requirements 6 2 research impact 46 12      citation analysis or bibliometrics 33 3      research metrics-other 9 4 research infrastructure 45 16 research methodologies 20 12 researcher type 70 11      faculty member 43 2      graduate students 27 8      other 2 2 scholarly communication 138 2      open access 56 0      publication type 54 2           journal articles 33 1           other publication types 5 0           patents 1 1           theses and dissertations 14 1 social media 7 0 undergraduate research experience or program 3 2 use of research 8 9      knowledge management 5 0      knowledge mobilization 2 5      knowledge transfer 1 1      translational research 0 4     innovation in grants, and international funding opportunities.   research impact was another topic prevalent in both the ra and lis journal articles included in this study, but again, the range of subjects covered under this umbrella category differed by profession. in lis, the vast majority of articles in this category were citation analyses or bibliographic studies assessing the literature of specific disciplines, and often including collection development implications. other articles included citation analysis studies of the relationship between various library services (including interlibrary loan use, institutional repository use, and availability of oa publication funds) and research impact. altmetrics (three articles), issues around journal impact factors (two articles), and faculty understanding of research metrics, academic library website content on research impact, and effectiveness of various search tools (one article each) rounded out the lis literature on research impact. the ra literature included two articles on bibliometrics/citation analysis, with one using these methods to assess level of international collaboration, and the other exploring their utility as predictors of future grant success. other ra articles on research impact focused on social/community impact of research (six articles), the complexities of assessing impact (three articles), and return on investment on grant-funded projects (one article).   both the lis and ra journal articles also took research methodologies as their focus, but again their approaches were quite different. more than three-quarters of lis research methodology articles were reviews of various research methodologies than can be employed by lis researchers. this number was skewed by the “research methods” column that ran in one of the journals under study, evidence based library and information practice, though similar articles did appear in other lis journals in smaller numbers. outliers included one article on web search strategies for systematic reviews, and one on librarian contributions to bibliometric research projects. the ra articles all explored various research methodologies but, in contrast to the lis articles which were overviews without contextualization, did so through having researchers report on their use of specific methodologies and associated pitfalls.   research infrastructure was another recurring broad topic in the literature of both fields. the lis literature focused largely on what could be called “digital research infrastructure” (28 of 41 articles), of which 10 articles explored specific search tools (e.g., google vs. pubmed, and others), 10 articles emphasized scholarly communications infrastructure (primarily oa and institutional repository systems), and seven articles focused on rdm infrastructure. a few articles also addressed space and other physical infrastructure, especially in terms of meeting graduate student needs. rdm infrastructure was the only real overlap between lis and ra, as one ra article considered rdm-related infrastructure needs. six ra articles focused on research management and administration infrastructure (research office space, staffing, and others), but the rest of the ra articles started with a specific need (research funding or grants, ethical conduct of research, research communication) and then explored solutions for addressing this particular need.   the final major area of overlap was the broad topic of the use of academic research, but again the focus differed significantly. four lis articles described knowledge mobilization issues and practices within academic libraries themselves, and the remaining four addressed the role of oa in increasing use of academic research. the ra literature focused on knowledge transfer (one article), translational research (two articles), and community engagement (two articles). each ra article emphasized the importance of spreading academic research beyond a traditional academic environment—to industry, to policy makers, and to the public.   topics unique to each body of literature   despite the overlap in references to these topics, there were also several subjects that received considerable attention in the literature of one field without a major presence in that of the other. some terms that were important in the ra literature did not occur within the most 100 frequently occurring terms in the lis journals (see table 3). again, though, it was the analysis and coding of the articles that produced the richest results. the appearance of institutional research culture or strategy as well as research ethics in the ra literature but not the lis articles, was unsurprising, given that responsibility for these issues usually resides firmly within the institutional research office. more curious (and perhaps indicating missed opportunity) was the absence of content on research collaboration, culture, and international topics in the lis articles.   conversely, several of the most frequently appearing terms in the lis literature did not appear within the 100 most frequent terms in the table 3 terms appearing in ra top 100 but not appearing in lis top 100 term position in top 100 ra terms number of occurrences in ra journal articles (n = 95) grant 29 810 collaborator 31 792 team 61 575 international 84 480 culture 100 425     table 4 terms appearing in lis top 100 but not appearing in ra top 100 term position in top 100 lis terms number of occurrences in lis journal articles (n = 195) access 7 4213 open 19 2688 citation 24 2398 digitize 41 1581 repository 44 1531 search 53 1425 copyright 63 1197 technology 81 1021     ra articles (see table 4). with the possible exception of technology, this was not surprising, as most of these are issues of more concern to librarians than others. the content analysis and coding similarly identified foci that were unsurprisingly unique to the lis articles, including information needs of researchers, librarian role in institutional research, library collections, and digital collections. again, though, there were omissions from the ra literature that are perhaps indicative of opportunities for librarian advocacy or collaboration, including institutional repositories, rdm, citation analysis or bibliometrics, scholarly communication, and oa.   discussion   differences in the topics and approaches of the lis and ra literatures with respect to research support were expected and healthy. however, considering that the professions are working toward similar goals of supporting institutional research, it was surprising that the literatures diverged to the extent that they did. at the very least, they suggested new or different avenues and approaches that each profession might want to consider in supporting campus research, as well as potential connection points. for example, the complete lack of references to institutional repositories and oa in the ra literature, despite a concern with research impact and use of research, suggests that research administrators might be receptive to, and could possibly become allies in, advocacy efforts in these areas. it also suggests that librarians may need to rethink their messaging on these topics, as current approaches have clearly not engaged research administrators.   the coding analysis found five topics that the literatures had in common: research funding, research impact, research methodologies, research infrastructure, and use of research. there were, however, significant differences in the aspects of the topics that generated interest, and these differences are instructive for librarians to consider. research funding was a common broad topic, but while the lis literature focused heavily on oa and scholarly publishing as a funding issues, this aspect of research funding was not present in the ra literature. this presents an opportunity for librarians to educate and collaborate with research administrators, who often administer research support funds intended to cover indirect costs of research and may also be aware of other potential funding sources. the lis literature on research impact relied heavily on citation analysis, often in the context of the literature of specific disciplines. expanding citation analysis to broader institutional uses might provide useful information for research administrators. the ra literature also provided examples of research impact methodologies beyond citation analysis that could provide new tools for librarians to consider employing in their studies of research impact. a related common topic was use of research. the ra literature focused heavily on encouraging the use of research outside of academia (by the public, policy makers, and others) but never mentioned oa, which has wide dissemination of information as its goal. this suggests that there may be an untapped appetite among research administrators for oa, and an advocacy and education role for librarians. there are undoubtedly other contributions that librarians could make to efforts to expand the use of research beyond traditional academic boundaries as well.   instructive articles on research methodologies also appeared in the literature of both professions, but the approach to exploring these methodologies differed. while the ra literature provided examples of use and commentaries by researchers with experience employing specific methodologies, the lis literature tended to provide overviews of methodologies, divorced from specific examples of application. the ra approach may suggest an additional way to increase librarians’ comfort levels with research methodologies that are new to them. research infrastructure, another topic common to both literatures, was also addressed quite differently. the ra literature tended to start with a specific need, while the lis literature was very focused on tools, and therefore often read like a literature of solutions in search of problems. this is another example where librarians could potentially learn from the ra approach, and might also suggest ways for librarians to reframe discussions with research administrators.   the infrequency with which the literature of each profession referred to the research support work of the other is also telling. the picture that emerged was very much one of librarians and research administrators working as what whitchurch (2008a) calls “bounded” professionals, “within the boundaries of a functional or organizational location that they had either constructed for themselves, or which had been imposed upon them” (p. 377). no ra articles suggested working with or partnering with librarians in a meaningful way, and the lis literature only infrequently raised the possibility. in the small pool of six articles that did mention working together (at least to some small extent), three arenas of activity emerged: undergraduate research programs, rdm, and scholarly communications. this finding, along with other articles on rdm identified in the literature review of this paper, suggest that these might be areas where there is some movement into “cross-boundary” work, with librarians “recogniz[ing], and actively us[ing] boundaries to build strategic advantage and institutional capacity” (whitchurch, 2008a, p. 377).   this study of the literature found little evidence of a third space yet emerging in research support work involving librarians and research administrators. there has been some trepidation expressed about third space, where “[s]tatus is uncertain, career paths are complex, and relationships may be challenging” (verbaan & cox, 2014, p. 212), but it is generally regarded as key to moving ahead with large projects and making progress on issues that extend beyond the capacity and bounds of a single profession. as mcalpine and hopwood (2009) observed, “these new constellations of people, and the common motive they share, offer degrees of freedom to explore new possibilities outside the constraints of established modes of working which shape interactions in the various contexts from which people come” (p. 159). librarians and research administrators may want to consider the potential of the “unbounding” of professional roles and the potential offered by a third space for research support.   the limited number of research administration journals currently published resulted in fewer ra articles being included in the analysis (n = 95) than lis articles (n = 195) and readers should keep this in mind when comparing occurrences of themes and keyword counts. this research was limited to a select range of journals and a specific timeframe, and other choices in these regards may have led to different results or conclusions. north american journals were chosen for the analysis both to create a data set that could be managed with the resources available and because this is the environment the author most needs to understand for her own practice. similar research conducted in other settings (asia, south america, etc.) in the future could yield valuable comparative data and provide insight into differences in research support environments in other parts of the world. it would be particularly interesting to re-run this study in five years’ time to see what changes have taken shape in the research support landscape. it is also possible that discussions and collaborations between librarians and research administrators are documented in other venues (e.g., blogs), or are taking place but not yet being documented. this limitation could be overcome by broadening the range of included sources beyond journal articles, or by interviewing librarians and research administrators to obtain information about research support work and collaborations not documented in the professional literature.   conclusion   this content analysis of the lis and ra literatures provided insight into the priorities and concerns of each profession with respect to research support. it found that, even in instances where the professions engaged on the same broad topics, they largely focused on different aspects of issues. the literature of each profession demonstrated little awareness of the activities and concerns of the other. in whitchurch’s (2008a) taxonomy, librarians and research administrators are largely working as bounded professionals, with occasional forays into cross-boundary activities. there is not yet evidence of unbounded professionalism or a move to a third space of research support activity involving these professions (whitchurch, 2015). librarians and research administrators will benefit from a better understanding of the current research support landscape and new modes of working, like the third space, that could prove transformative.   acknowledgements   the author gratefully acknowledges the canadian association of research libraries for providing a carl research grant for practicing librarians, which made this research possible. she would also like to acknowledge the coding work of student assistant john kapp. a preliminary version of this research was presented at the research libraries uk conference in march 2018.   references   antell, k., foote, j. b., turner, j., & shults, b. (2014). dealing with data: science librarians’ participation in data management at association of research libraries institutions. college & research libraries, 75, 557-574. https://doi.org/10.5860/crl.75.4.557 association of research libraries. (2013). new roles for new times: transforming liaison roles in research libraries.. retrieved from http://www.arl.org/storage/documents/publications/nrnt-liaison-roles-revised.pdf   bhabha, h. k. (1994). the location of culture. london: routledge.   corrall, s. (2014). designing libraries for research collaboration in the network world: an exploratory study. liber quarterly, 24(1), 17-48. https://doi.org/10.18352/lq.9525   cox, a. m., & pinfield, s. (2013). research data management and libraries: current activities and future priorities. journal of librarianship and information science, 46(4), 299-316. https://doi.org/10.1177/0961000613492542   cox, a. m., & verbaan, e. (2016). how academic librarians, it staff, and research administrators perceive and relate to research. library & information science research, 38, 319-326. https://doi.org/10.1016/j.lisr.2016.11.004   ferguson, c., & metz, t. (2003). finding the third space: on leadership issues related to the integration of library and computing. in c. e. regenstein, & b. i. dewey (eds.), leadership, higher education, and the information age: a new era for information technology and libraries (pp. 95–112). new york, ny: neal-schuman.   hansson, j., & johannesson, k. (2013). librarians' views of academic library support for scholarly publishing: an every-day perspective. journal of academic librarianship, 39, 232-240. https://doi.org/10.1016/j.acalib.2013.02.002 hensley, m. k., shreeves, s. l., & davis-kahl, s. (2014). a survey of library support for formal undergraduate research programs. college & research libraries, 75, 422-441. https://doi.org/10.5860/crl.75.4.422   hensley, m. k., shreeves, s. l., & davis-kahl, s. (2015). a survey of campus coordinators of undergraduate research programs. college & research libraries, 76, 975-995. https://doi.org/10.5860/crl.76.7.975   hockey, j., & allen-collinson, j. (2009). occupational knowledge and practice amongst uk university research administrators. higher education quarterly, 63(2), 141-159. https://doi.org/10.1111/j.1468-2273.2008.00409.x   maccoll, j. & jubb, m. (2011). supporting research: environments, administration and libraries. retrieved from: www.oclc.org/research/publications/library/2011/2011‐10.pdf    masango, c. a. (2015). combating inhibitors of quality research outputs at the university of cape town. journal of research administration, 46(1): 11-24. retrieved from http://www.journalra.org/   mcalpine, l., & hopwood, n. (2009). ‘third spaces’: a useful developmental lens? international journal for academic development, 14, 159–162. https://doi.org/10.1080/13601440902970072   nariani, r. (2013). pubmed central canada: beyond an open access repository. journal of academic librarianship, 39, 76-83. https://doi.org/10.1016/j.acalib.2012.11.005   o’brien, l., & richardson, j. (2015). supporting research through partnership. in b. l. eden (ed.), partnerships and new roles in the 21st-century academic library: collaborating, embedding, and cross-training for the future (pp. 191-212). lanham, md: rowman & littlefield.   research information network. (2010). research support services in uk libraries. london: rin. retrieved from: http://www.rin.ac.uk/system/files/attachments/research_support_services_in_uk_universities_report_for_screen.pdf   research libraries uk. (2012). re-skilling for research. retrieved from www.rluk.ac.uk/wp-content/uploads/2014/02/rluk-re-skilling.pdf   russell-simmons, h. n., anthony, c., ballard, m., coffman, j., gilbreath, d., keys, t. l.,. . . vanderford, n. l. (2016). enhancing faculty productivity through a centralized communications and project management infrastructure: a case study at the university of kentucky markey cancer center. journal of research administration, 47(2), 68-79. retrieved from http://www.journalra.org/   sproles, c., detmering, r., & johnson, a. m. (2013). trends in the literature on library instruction and information literacy, 2001‐2010. reference services review, 41, 395-412. https://doi.org/10.1108/rsr-03-2013-0014 verbaan, e., & cox, a. m. (2014). occupational sub-cultures, jurisdictional struggle and third space: theorising professional service responses to research data management. journal of academic librarianship, 40, 211-219. https://doi.org/10.1016/j.acalib.2014.02.008   whitchurch, c. (2008a). beyond administration and management: reconstructing the identities of professional staff in uk higher education. journal of higher education policy and management, 30, 375-386. https://doi.org/10.1080/13600800802383042   whitchurch, c. (2008b). shifting identities and blurring boundaries: the emergence of third space professionals in uk higher education. higher education quarterly, 62, 377–396. https://doi.org/10.1111/j.1468-2273.2008.00387.x   whitchurch, c. (2015). the rise of third space professionals: paradoxes and dilemmas. in: u. teichler, & w. cummings (eds), forming, recruiting and managing the academic profession (pp. 79-99). cham: springer international publishing. https://doi.org/10.1007/978-3-319-16080-1_5              microsoft word let_10pt font evidence based library and information practice 2008, 3:3 101 evidence based library and information practice letters to the editor commentary on a library non-use study judith siess president, information bridges international champaign, illinois, united states of america e-mail: jsiess@ibi-opl.com received: 13 april 2008 accepted: 17 june 2008 © 2008 siess. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. when i read non-use of library services by students in a uk academic library, by lisa toner in eblip 3:2, 2008, i just had to write down the following comments on this wonderful article and study. i was impressed that not only did toner study why people did not use the library, she actually implemented the results (“the results were then fed into the library’s future strategic planning cycle”). she also looked at specific possible reasons for nonuses: inconvenient opening hours, use of electronic resources, lack of or poor introduction to the library, poor perception of library service. the methodology was excellent. the study period was long enough (three months), they did a pilot survey, used two method of distribution (mail and in classrooms) and got a quite acceptable response rate (14 percent). many people are quick to jump to conclusions based on their survey results, but toner was more cautious: “…the incidence of non-use of library services does increase with age. however, these results could also indicate that older people are more diligent in returning their survey forms.” “[after a] further analysis of the age ranges…, mode of study and age do seem to be predictors of use of the library services”. her report even included results that were not favorable: “the results [on information sources used] highlighted that non-users not only made no use of traditional library services but only made limited use of the electronic resources provided by the library. just under a third of non-users did not make any use of the college’s electronic resources.... in contrast to this, the majority (88 percent) use the internet as a source of information for their assignments.” i was a bit surprised that in this day and age students could respond that they “did not evidence based library and information practice 2008, 3:3 102 need” libraries (four percent) or “do not like libraries at all” (one percent). however, those that say they do not need libraries probably just don’t know that they need them or what libraries have to offer. one very insightful and important conclusion that toner made was, “this [respondents who buy their own books or borrow them from their friends instead of borrowing from the library] demonstrates a significant shift away from the concept of the library being the provider of all the research and information needs of students…. students who do not borrow books are on the whole not using other library services either. those [students who use the library via computer] who do visit the library building do so mainly for photocopying services (25 percent) or private study.” these indicate the changing nature of the library–from warehouse of information to social networking workplace. additionally, not only has the library taken specific steps to remedy the shortcoming revealed from the study, but they recognize that they need to repeat the evaluation cycle, as shown by the following: “the results of the survey have shown that they library needs to become much more creative in its marketing, publicity, and promotion to nonuser groups, particularly part-time and distance learners. as a consequence, the faculty liaison teams have been strengthened and library staff work more closely with course and program leaders to ensure that all students receive meaningful induction and subsequent training on databases, information searching, and critical thinking skills. an increasing number of students are taught not to rely totally on the internet for their information needs. work has been done to embed this training into the curriculum wherever possible and the library staff are increasingly included in the training of students at remote locations. further research is needed to analyse the effectiveness of our actions in 2-3 years time when it is hoped that much of the impact of increased training and promotion will have had a positive effect on usage figures.” this is a wonderful study–a model for others to follow. judith siess evidence summary   students use library resources but are unlikely to consult with librarians during the early research process   a review of: thomas, s., tewell, e., & wilson, g. (2017). where students start and what they do when they get stuck: a qualitative inquiry into academic information-seeking and help-seeking practices. journal of academic librarianship, 43(3), 224-231. http://dx.doi.org/10.1016/j.acalib.2017.02.016   reviewed by: kimberly miller learning technologies librarian albert s. cook library towson university towson, maryland, united states of america email: kimberlymiller@towson.edu   received: 11 nov. 2017 accepted: 6 dec. 2017      2017 miller. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to investigate where students start their research, what resources they use, and when they may consult with a librarian.   design – ethnographic, semi-structured interviews.   setting – a mid-sized, private university located in the northeastern united states of america.   subjects – 15 students; 7 undergraduate students and 8 graduate students.   methods – researchers gathered data as part of a larger ethnographic study conducted at the university.  interview participants were selected from among respondents to an email survey sent to all university students. interview participants were purposefully selected to represent the student population with regards to their status (undergraduate or graduate), progress through their programs, and their majors. the semi-structured interviews focused primarily on how students approached the beginning stages of research and the types of resources used.   the authors read each interview transcript to identify possible research questions, then re-read transcripts to identify codes and potential themes related to the selected research questions. finally, they analyzed the transcripts to determine where essential themes and keywords appeared, while highlighting relevant passages and finalizing themes.   main results – students were more likely to seek research help from faculty members and their peers than from librarians. graduate student interviewees were more likely to report consulting with librarians than undergraduate students. interview themes suggest that students may not consult with librarians because they do not perceive librarians as having the subject knowledge or “insider” status (p. 227) of their professors and peers. few students articulated an understanding of the expertise librarians could bring to a research project.   when starting a research project, students were more likely to report beginning with library databases than they were google or other open web sources. while many students also shared that they used multiple different resources in their initial stages, most also reported that they ultimately narrowed their search focus to a specific database. students also discussed struggling with their database searching.   conclusion – the authors suggest that future research should focus on understanding the types of resources that faculty members recommend to their students, which could inform how librarians approach their work with students. additional research related to how faculty members and students perceive librarians may also clarify the role these groups expect librarians to fill during the research process. although results cannot be generalized to all student populations, the authors call for librarians to further explore assumptions about how students begin their research and the work academic librarians do to support students’ natural behaviours and preferences.   commentary   academic reference and instruction librarians have a strong interest in understanding the decisions students make about conducting research. the current study adds to a line of research that explores the types of resources, both human and informational, that students consult during their research processes. while the study’s findings largely confirm previous research demonstrating that students are unlikely to consult with librarians, and that students use a range of information sources to begin their work, the emergent themes shed additional light on how a particular student population works through early stages of research.   appraising the study design and evidence using letts et al.’s (2007) critical review instrument suggests the study’s strengths lie in using the qualitative data collection and analysis approach to discover key components of students’ early research process. relying on questions and themes that emerge from the data allowed the authors to explore students’ ordinary experiences. the team-based approach to identifying questions, codes, and themes increases the evidence’s reliability. the article provides a relatively clear trail through the coding and decision making process; however, it is somewhat unclear whether codes were entirely derived from the initial codebook that was shared with the researchers or if new codes emerged during the current study’s data analysis. relying on data from a larger study also means that the authors had little control over data collection and design, including whether the selected student sample led to saturation in their themes. while the article notes limitations regarding the generalizability of findings, there is no discussion of whether using data collected in a different context may have an influence on the results. given that findings of a small qualitative study may not be generalizable to other populations, including the interview protocol within the article could help other librarians confirm the study’s findings with their own students.   academic librarians whose responsibilities include student research support may be most interested in exploring how they can gain “insider” status with student and faculty researchers. the current study suggests that while students regularly rely on library resources, students see their peers and their professors as more useful sources of help because these groups understand either the assignment or the subject of interest. librarians must demonstrate a similar niche within the research process, where students start to assume librarians also innately understand research or subject-specific needs. this can be a challenge when, for example, librarians begin research consultations with an extensive reference interview that students may perceive as extra questions that other students or faculty members do not need to ask in order to be helpful. librarians may consider how to reframe this approach to questioning as their value-added expertise in an inquiry-based research process.   references   letts, l., wilkins, s., law, m., stewart, d., bosch, j., & westmorland, m. (2007) critical review form – qualitative studies (version 2.0). in region of peel. retrieved from http://www.peelregion.ca/health/library/eidmtools/qualreview_version2_0.pdf   evidence summary   users engage more with interface than materials at welsh newspapers online website   a review of: gooding, p. (2016). exploring the information behaviour of users of welsh newspapers online through web log analysis. journal of documentation, 72(2), 232-246. doi:10.1108/jd-10-2014-0149   reviewed by: kathleen reed assessment & data librarian vancouver island university nanaimo, british columbia, canada email: kathleen.reed@viu.ca   received: 1 june 2016    accepted: 4 aug. 2016      2016 reed. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – this study has two specific objectives: to learn about the behaviours of visitors to the welsh newspapers online (wno) website, and to explore how the identified behaviours are different from those common to information-seeking in a physical archive.   design – analysis of google analytics and web server content logs.   setting – welsh newspapers online website: http://newspapers.library.wales   subjects – wno had 19,805 unique visitors from 12 march 2013 to 30 june 2013, who made 52,767 visits to the site.   methods – gooding accessed the wno google analytics account, which provided visitor numbers, user engagement by page visit and visit duration, bounce rate, and mobile and social media usage. using anonymized processed content logs provided by the national library of wales, he then explored searches undertaken by users on the website; instances where users browsed, filtered, or otherwise interacted with search results; and instances where users viewed content.   main results – google analytics statistics showed users of wno demonstrate behaviour that is “deeper and more sustained than general web browsing” (p. 237). the number of visitors who only viewed one page and then left the site (bounce rate) was low, while page views and time spent on the site were higher than considered standard on general websites.  mobile users made up 11% of visits, although on average they viewed fewer pages and stayed for less time than non-mobile users. screen size was directly correlated to the level of engagement. there were 9% of visitors referred via social media, but generally showed a low engagement rate similar to that of mobile users; the exception was users who were directed to wno via blogging platforms.   web log analysis showed visitors most frequently accessed newspapers from the 1840s and 1850s. they viewed the title page much more frequently than any other page in the newspapers, likely reflecting that the title page is default when users access a paper via browsing. a correlation between time spent on the site and searching versus engaging with content was found: the longer a visitor was on wno, the less time they spent searching, and the more time spent engaging with content. still, as gooding reports, “over half of all pageviews are dedicated to interacting with the web interface rather than the historical sources” (p. 240).   conclusion – wno visitors spend more of their time interacting with the site’s interface than with digitized content, making it important that interface design be a high priority when designing online archives. gooding concludes that despite a focus on interface, visitors are still engaged in a research process similar to that found in an offline archive and that “a differently remediated experience is not necessarily any less rich” (p. 242).   commentary   this well-written article will be of interest to library and information professionals and researchers who work in areas related to webometrics, information behaviour, electronic resources, and user experience. although the data used is highly technical, the clearly-articulated process, results, and thoughtful conclusions are well translated for those without backgrounds in webometrics.    the evidence presented makes a compelling case that user behaviour at the wno site is significantly different and more engaged than would be seen among users of general websites. although this study does not test the same group of individuals in the two settings, studies of general surfing behaviour are well established in the scholarly literature and appropriately cited in this paper.    a particularly enjoyable aspect of this article is gooding’s entry into the theoretical landscape of debates related to print versus electronic texts, and the notion of readers versus users. he responds with an excellent critique of the unhelpful binary nature of these debates, concluding that people accessing online and offline archives have more in common than not.   gooding is upfront about the limitations of his study: the data analyzed tells us how people use the site, not why; and this research is based on only one site over a period of approximately three months. additionally, several times throughout the article gooding refers to a rising problem for both researchers and users of websites: proprietary blackbox technologies that provide results without being transparent about either the data or algorithm used. as a researcher, he refers to google analytics disallowing him view of the raw data and algorithm on which he relies in this study, leaving him forced to trust the reliability and reproducibility of google’s results. from a user perspective, gooding discusses how relying on digital archival database interfaces means placing trust in search and optical character recognition algorithms. in our inability to see into the blackbox, users “place trust in algorithmic discovery, metadata production and digitization technologies to ensure the quality of the resources they discover” (p. 241).   while this particular study would be difficult to reproduce due to the proprietary nature of the data and algorithms used to analyze it, gooding details his methodology clearly enough that it could be utilized to analyze datasets from different websites.    evidence based library and information practice evidence based library and information practice 2010, 5.1  evidence based library and information practice       article     addressing the challenge: cataloguing electronic books in academic libraries    shuzhen zhao  bibliographic services librarian  leddy library, university of windsor  windsor, ontario, canada  email: zhaoszf@uwindsor.ca    wei zhao  metadata librarian   ontario council of university libraries / ocul  toronto, ontario, canada  email: w.zhao@utoronto.ca     received: 14 may 2009          accepted: 30 jan. 2010       2010 s. zhao and w. zhao. this is an open access article distributed under the terms of the creative  commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐ nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the  original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is  redistributed under the same or similar license to this one.    abstract    objective ‐ this paper explores the various issues and challenges arising from e‐book  cataloguing experienced at the university of windsor’s leddy library and the ontario  council of university libraries (ocul).  this discussion uses an evidence based  approach to identify and resolve issues relevant to academic libraries as well as to  consortia.  with the ever rising popularity of e‐books within academic libraries,  cataloguing librarians are actively seeking more effective methods of managing this  new electronic medium, including the development of new cataloguing policies and  procedures.  this paper will explore the various issues and challenges surrounding e‐ book cataloguing and processing within academic libraries, and will identify new  policies and procedures that may be used to effectively assist in e‐book management.     methods ‐ this paper presents a case study of e‐book cataloguing practices  undertaken by a canadian academic library and the consortium with which it is  affiliated. towards this end, the university of windsor’s leddy library will be the  prime focus of this study, with its establishment of a new e‐book marc records  database.  the research is based on the results of the e‐book marc project undertaken  by the leddy library and the ontario council of university libraries (ocul).   93 evidence based library and information practice 2010, 5.1  94 through analysis of various suppliers’ marc records and the actual implementation  of the e‐book marc project, the authors developed and evaluated a new approach to  e‐book cataloguing for use in academic libraries.      results ‐ this practice‐based approach towards the development of a new method of  e‐book cataloguing required continual modification and examination of e‐book marc  records within the target library.  the leddy library’s e‐book marc project provided  an excellent opportunity to test the library’s existing cataloguing standards and  procedures for print format, while at the same time, identifying related e‐books issues.   the new policies and procedures for e‐book cataloguing that followed were  developed as a direct result of the authors’ cataloguing experiences and the  information gained by examination of other academic libraries’ e‐book cataloguing  processes.  this paper also provides an evaluation regarding the quality of suppliers’  marc records. results of this study should serve to increase the efficiency and  effectiveness of cataloguing in academic libraries.    conclusions ‐  this paper identifies key issues regarding the cataloguing of e‐books in  academic libraries. throughout, the authors have provided an evidence based  approach.  the hope is that the results will provide a useful framework for other  academic libraries to build upon when developing their respective e‐book cataloguing  databases.  e‐books are effective resources, and academic libraries need to adapt to  this new electronic medium in order to assist patrons in their discovery and usage.    introduction    academic libraries have experienced a  dramatic increase in electronic book (e‐book)  purchases since the beginning of the new  millennium. in canada, this was due in large  part to the formation of consortia, such as the  canadian research knowledge network  (crkn) and the ontario council of university  libraries (ocul).  in addition, the direct  availability of e‐books through book vendors  such as netlibrary, ebrary, and coutts now  makes e‐book purchase possible in the form of  “packages.”      an electronic book, or “e‐book,” is the  presentation of electronic files via digital  displays.  although the term ʺe‐bookʺ implies  “book‐oriented” information, other types of  formats are also displayed on these devices.   e‐book files can come in the form of recorded  units (disks), or they can be downloaded from  digital repositories (including web sites) to  computers, portable digital assistants (pdas),  smartphones, and other handheld computers  (romano, 2009, p. 2).  most libraries today use  remote access to retrieve the content of e‐ books.  “remote access involves a connection  to the internet and an internet browser to  access the content of an e‐book from a local  area network (lan) or a remote server”  (bothmann, 2004, p. 13).  many publishers and  vendors offer this type of e‐book access using  platforms such as ebrary, myilibrary and  knovel.    the traditional focus of library collections has  been on the acquisition and processing of print  materials, so the transition to e‐books has led  to a new set of challenges for academic  libraries. this includes issues relating to  licensing, purchasing and ownership,  downloading of records, as well as  cataloguing policies and procedures.  furthermore, e‐book collections are  challenging long‐established library practices  of collecting, organizing, and preserving  information in print format. worldwide, large  print collections and storage facilities have  been a source of great pride for libraries, and  the size of a library’s collection has been a  mark of a library’s status.  as a result,  cataloguing standards and procedures have  focused primarily on print collections.   evidence based library and information practice 2010, 5.1  these long‐established standards and  procedures do not conform to the unique  characteristics of e‐books, and academic  libraries need to adapt to the new format.    during the past twenty years, digital formats,  such as dvds and cd‐roms have made the  greatest impact on library collections.  “the  reverberations of this impact are still being  felt, and the long‐term consequences for  traditional print book collections are not yet  determined” (lavoie, 2007, p. 106).  the e‐ book is in its early stages of integration into  academic libraries.  with its advantages of  lower costs, remote access, and ease of use, the  e‐book now plays a more prominent role  within academic environments as a whole.     increasingly, over the past decade, publishers  and vendors have made their books available  in electronic format.  publishers often  encourage libraries to buy e‐books by  providing only the electronic format.  downloading bibliographic records for e‐ books, is therefore a matter of great  importance for academic libraries.  however,  issues have been raised regarding the quality  and number of records necessary for e‐book  format, bibliographic levels, inventory control,  as well as the stability of the links.  these  issues need to be addressed before records are  downloaded into cataloguing systems, since  they affect how academic libraries will  manage their e‐book collections.  furthermore,  as discovered by the author, academic libraries  currently have inadequate policies for  managing their e‐books.  this article,  therefore, explores these issues while  providing an evidence based approach to e‐ book management.               literature review    during the past decade, the information  science literature has focused on the quantity  of electronic materials, their usage,  cataloguing policies, as well as the sharing of  these resources.  several studies have shown  that the quantity of e‐books within their  collections is very important to academic  libraries. the primary research group’s 2008  study of academic libraries noted a sharp rise   in libraries’ e‐book acquisitions.    institutions that gave data on e‐book  spending from 2006 to 2008 showed  an increase in spending between 2006  and 2007 from a mean of $19,340 to  $26,290, a 36% increase, while in 2008  mean spending rose to $29,861, a  13.6% increase from 2007. (p. 18)      at the university of texas, once records were  added to the online catalogue, usage increased  immediately by about 50% over three months’  time (dillon, 2001, p. 116), green noted that  after the university of surrey began e‐book  purchases in 2001, “feedback from students  was positive, and they reported that they were  keen to use e‐books more.” (2003, p. 55)  cataloguing policies of academic libraries,  therefore, are instrumental in providing access  to e‐book collections for library users.       circulation studies of e‐books have focused on  comparisons of usage with their print  counterparts, and also on their usage in  various subject areas.  dillon at the university  of texas at austin, and ramirez and gyeszly  from texas a&m university, examined usage  reports for e‐books at their respective  universities as well as those of larger consortia  collections; namely, the amigos netlibrary  and the texas state library collections.  both  studies found that e‐books in the subject areas  of computer science, economics, and business  received higher usage than those of other  subject areas.  their research also discovered  that these results were consistent across the  various collections (williams & best, 2006, p.  475).      in 2006, the university of bedfordshire  surveyed the cataloguing of e‐books in the  u.k.’s higher education libraries. in belanger’s  report of that survey she noted that there was  “widespread consensus that bibliographic  records for e‐books should be integrated into  library opacs to assist users in resources  discovery and access” (2007, p. 204).  “it is  essential to include the e‐book details and  url links in the library catalogue to make it  easy for students to identify which titles are   95 evidence based library and information practice 2010, 5.1  available as e‐books” (belanger, 2007, p. 204).      as academic libraries spend most of their  acquisition budgets on electronic resources, it  is understandable that the focus of literature  has been on the evaluation of their usage and  implementation. electronic resource  management is also important, however,  because it impacts on libraries’ processing  activities and workflow.  the resulting new  policies affect library services, such as opac  displays.  librarians currently have  insufficient access to bibliographic standards  to support e‐book cataloguing in comparison  with standards and guidelines available for  materials in their print collections. the anglo‐ american cataloguing rules’  (http://www.aacr2.org/) treatment of e‐ resources is still in need of revision (jisc,  2003).  this is further supported by evidence  from ocul’s experience, which suggests e‐ book cataloguing is still a nascent field.  there  remain several issues to be considered, such as  cataloguing rules, implementation of separate  catalogue records, and the quality of  suppliers’ records.  the standards for  managing electronic collections appear to be  evolving more slowly than the resources  themselves.  due to the ever‐changing format  of e‐books, it is not surprising that their  bibliographic implications have not yet been  fully examined.  in lieu of formal standards for  cataloguing e‐books, it is essential to provide  some guidelines.         aims    this paper investigates current issues  regarding vendor‐supplied marc records  and the cataloguing of e‐books at the  university of windsor’s leddy library and at  ocul. through the authors’ analysis of these  records, they explore various policies and  procedures for e‐book cataloguing within  academic libraries. the purpose of this study  is to develop recommendations to improve the  cataloguing of e‐books in order to assist in  their management. it is through improved e‐ book cataloguing, that librarians can begin to  resolve many of the problems arising from this  new resource.  this paper emphasizes the  importance of e‐book cataloguing and its  management.    context    ocul is a consortium comprised of 21  university libraries within the province of  ontario, canada. as a consortium, ocul is a  recognized leader provincially, nationally, and  internationally, in the collaborative  development and delivery of outstanding and  innovative library services.  the member  libraries cooperate to enhance information  services through resource sharing, collective  purchasing, and document delivery, as well as  many other related activities (ocul, 2010).  ocul has recently developed its own e‐book  platform for use by member libraries in order  to provide more efficient access to e‐books.  since 2007 ocul has licensed e‐books for  uploading to the e‐book project platform.  the  objective of this project is to implement a  locally installed, managed, and operated e‐ book system. on january 9, 2008, ocul  announced the selection of ebrary as the  official platform for e‐books, available for use  by all ontario universities and colleges.  later  that same year, on september 18, 2008, ocul  announced the “preview release” of this new  e‐book service.  this platform includes over  130,000 digitized books from various  publishers, including: elsevier, cambridge,  oxford, taylor and francis, gibson, and the  internet archive collection.     the university of windsor’s leddy library is  one of ocul’s 21 member libraries that  actively participate in ocul’s licensing of e‐ books. the leddy library uses ocul’s  platform, in conjunction with its own local e‐ book database, to access e‐book content from  such publishers as cambridge university  press, oxford university press, and taylor  and francis.  in the future, the leddy library  will use ocul’s wiki to retrieve these marc  records, eliminating duplication in its  cataloguing database.  furthermore, the leddy  library hopes to use ocul’s e‐book records,  along with those of other publishers, in its  local cataloguing database on a permanent  basis. this would eliminate the need for  96 evidence based library and information practice 2010, 5.1  cataloguing e‐books individually and provide  a more efficient method of e‐book processing.    the leddy library also participates in ocul’s  e‐book forum where staff members discuss  issues regarding e‐book cataloguing, metadata  standards, policies, and procedures. the  library created a link on its web site for  ocul’s e‐book packages, and the ocul  scholars portal web site added a link in  october, 2009.     methods     this paper presents a case study of e‐book  cataloguing practices undertaken by one  canadian academic library in conjunction with  the consortium with which it is affiliated.   towards this end, the university of windsor’s  leddy library is the prime focus of this study,  with a project to establish an e‐book marc  records database.  this paper is based on the  results of the e‐book marc projects  undertaken by the leddy library and ocul  respectively.  through their analysis of  various suppliers’ marc records and their  involvement with the actual implementation  of the e‐book marc project, the authors  attempted to discover a more effective method  of e‐book cataloguing.  because creating e‐ book marc records within academic libraries  is a relatively new practice, and in order to  assist them in building up their e‐book  cataloguing database, the authors used an  interpretive, participative approach so as to  provide evidence for e‐book cataloguing.  due  to the lack of e‐book cataloguing guidelines,  the project utilized a comparative approach  when analyzing the unique fields in e‐book  marc records.  the method selected for  cataloguing e‐books directly impacts library  patrons’ ability to discover and access e‐books.   therefore, the authors relied on their  knowledge, as well as the unique  characteristics of e‐books, to identify issues  surrounding the cataloguing of e‐books, and  to develop policies and procedures for future  practice.            e‐book cataloging practices at the university  of windsor’s leddy library    the university of windsor’s leddy library  began purchasing e‐books at the beginning of  the new millennium.  the library has since  expanded its collection, utilizing platforms  such as ebrary and myilibrary, while also  acquiring packages from springer, oxford  university press, cambridge university press,  taylor and francis, and eebo (early english  books online).  currently, the leddy library  provides access to over 110,000 e‐books that  patrons can readily access through the  library’s catalogue.  future plans involve  purchasing more e‐books through consortia  arrangements. according to a survey  conducted in 2007 (zhao), increasing numbers  of students and faculty members prefer using  e‐books over their print counterparts. in spite  of the increase in e‐book collections and usage,  there are no current guidelines regarding the  management of e‐book collections in academic  libraries.     e‐books are available through a variety of  packages that can be purchased or licensed  from various suppliers using different  platforms.  each package has different  standards for its marc records, affecting the  quality of overall cataloguing.  in 2007, the  leddy library initially downloaded  individual title records into its voyager  catalogue database for every e‐book title  within the various packages.  prior to  downloading these records, there had been  considerable discussion regarding the  “establishment” of e‐book marc records in  the local catalogue database.  questions arose  regarding whether to use single or multiple  records when a print record for the electronic  title already existed, and whether to download  records in batches or individually. there were  also questions about which unique marc  fields should be used for e‐books.  all of these  issues have a direct impact on the  establishment of an e‐book database and are  discussed below.        97 evidence based library and information practice 2010, 5.1  problems and issues     quality of suppliers’ marc records     the quality of suppliers’ marc records is the  most important issue to consider when  establishing a cataloguing database.  to obtain  marc records, there are two sources:  the  publishers and the vendors.  every e‐book  package is unique, and each comes with its  own licensing terms and viewing restrictions.   for example, myilibrary restricts the number  of simultaneous users for e‐books, while  ebrary does not.  the quality, format, and  functionality of marc records also varies  from package to package.  there are a number  of issues that need to be addressed regarding  suppliers’ e‐book marc records including lc  subjects, url stability, added entry fields, and  punctuation.  listed below are the various  suppliers of marc records and the issues  affecting the quality of their records.    springer:       the springer marc21 records of  today are basic records that are  generated using springer metadata.  as a result, the tags or fields within  each record can be populated with  accurate information directly from the  publisher. . . . field 650 contains  internal springer subject  classifications, the same exact  classifications used by  springerlink.com . . . we have also  included the library of congress  subject headings in our 650 field.  (springer link)      between the years 2005 and 2008, springer  made ʺbriefʺ marc records available for their  e‐books through their web site. the leddy  library decided to download the brief e‐book  records as a cost saving measure to avoid  record usage fees.  however, librarians  discovered those brief records contained urls  pointing to springerlink (dx.doi.org), and that  those marc records were not updated  regularly.  therefore, while it was beneficial  for their patrons to have linked access to these  e‐books, these brief records were inadequate  for leddy library’s cataloguing database.   appendix 1 provides an example of a  springerlink e‐book marc record. one major  problem encountered was the lack of  authorized subject headings, the essential  finding aids for users searching an opac.   below is a summary of problems discovered  while analyzing a sample of ten random  marc records from springerlink’s web site.      • the 008 tag was incomplete (e.g.,  missing publication location or  bibliographic reference).  • all nouns were capitalized in the 245,  440, and 490 fields of all 10 records.  • punctuation did not conform to the  international standard bibliographic  description (isbd).   • name authorities were in german for  4 of the 10 records.  • the 300 fields were incomplete.  • subject headings were general, with  variations in spelling. sometimes the  headings used illegal terms as subject  headings (when compared to oclc  records which had more complete  subject headings).     • springer marc records used utf‐8  as their character coding scheme.    myilibrary:  myilibrary is the platform used  by coutts library services.  as a vendor,  coutts provides e‐books with their marc  records to subscribing libraries.  the authors’  analysis of coutts’ marc records revealed  that the quality of these records is superior to  that of other publishers.  this is because  myilibrary contains comprehensive  bibliographic information for every e‐book  available on its platform.  data elements  include:  access fields, publication information,  description, subject, and call number. the  university of windsor’s leddy library  purchases e‐books from myilibrary as  individual orders without extra charges for the  marc records.  these records contain urls  pointing to myilibrary  (http://www.myilibrary.com/). the records  require only minor adjustments to the  subfields to reflect leddy’s local information.  98 evidence based library and information practice 2010, 5.1  appendix 2 provides an example of a  myilibrary e‐book marc record.    ebrary:   on january 6, 2009, ebrary announced,  “customers can now immediately upload free  marc records for individual titles they  purchase as well as e‐books.” (wagner, 2009)   ebrary offers an “on‐demand” feature which  allows its subscribers to download marc  records from all packages purchased.   subscribers can make additions to their e‐book  collections from previously purchased  packages, with no restrictions at any time, by  providing their marc records in both  increments and complete record sets.   furthermore, to simplify downloading,  complete marc records are now available  from ebrary’s extranet site in batches of 5,000  in a single zip file.  these new marc features  make it easier for subscribers to manage their  marc records, while enabling them to  download large batches at the same time.  as a  result of these enhanced features, ocul  selected the ebrary platform to host over  100,000 e‐books locally.  complete academic  packages, such as those offered by taylor and  francis or oxford university press will soon  become available on ebrary’s platform,  allowing the leddy library’s patrons to access   these ebrary full‐text e‐books via ocul.              alexander street press and adam matthew press  digital: alexander street press is one of the  leading publishers of scholarly databases in  the arts, humanities, and social sciences.  the  adam matthew digital collections offer rare  sources for the humanities and social sciences.   the records of alexander street press and  adam matthew press digital contain urls  pointing to their own respective web sites.   due to electronic reproduction, the marc  records from these two publishers are very  general, especially for description fields,  although the urls of both are quite stable.  the university of windsor’s leddy library  has had access to both of these publishers’ e‐ books since 2008.    implementation of separate catalogue records    the initial problem encountered when   cataloguing e‐books is to decide whether to  create separate records for each format of the  same title.  for example, when the library  already has the print copy of the electronic  title, should it simply add the url for the  electronic version to their existing record for  the print title?  the guidelines of both oclc  and the library of congress suggest the use of  separate records, but stress that their  recommendations are not binding (weitz,  2009).  “practice for this has not yet been  widely established by academic libraries and  the situation at the time was that libraries  made their own local choices in this matter”  (gravett, 2006, p. 203).  there are a number of  factors to consider when making the choice  between single and separate records.    single records      • have the advantage of reducing  the number of places in the  opac where users have to look  when a book exists in various  formats  • can be more cost‐effective, by  adding detailed information for  e‐books to existing records when  no vendor‐supplied records are  available  • are a disadvantage if changes  need to be made to the records   (e.g., the library’s subscription to  an electronic title is cancelled). it  can be more time consuming to  delete the relevant fields from a  single record rather than  removing an entire record from  cataloguing.  • are a disadvantage if users wish  to limit their search by format    separate records    • enable users to retrieve only e‐books, if  they wish  • can save libraries a great deal of time  when dealing with large collections from  packages such as eebo, ebrary, and other  products, because they can do “batch”  downloading of marc records to their  99 evidence based library and information practice 2010, 5.1  catalogues without consideration as to  whether there is duplication of titles  within the existing database  • can ensure more timely access to various  collections  • are easier to maintain when updating  information    after careful examination and discussion of all  of these issues, as well as reviewing the  procedures used by other academic libraries,  the leddy library chose to use the “separate  record” rule for its e‐books.     bibliographic level    the leddy library purchases e‐book packages  through consortia such as the canadian  research knowledge network (crkn) and  ocul, or directly from publishers and book  venders.  during the cataloguing process, the  authors discovered that many suppliers’  records did not use the same bibliographic  level as the leddy library. for example,  myilibrary records included a higher level of  detail, as compared to those of springerlink,  whose records contained only “basic level”  bibliographic information.  the bibliographic  level used in marc records is important, as  the type of information contained within these  records directs patrons to the resources they  need when researching and referencing their  research papers.  therefore, the higher the  level of bibliographic detail, the easier it is for  patrons to utilize the cataloguing database as a  research tool.  the bibliographic level used in  the catalog must be standardized and based  on cataloguing standards and policies.  if  different bibliographic levels are used within  the same cataloguing database, the results will  be confusing for both patrons as well as  library staff.      in light of the backlog of e‐books, and after  exploring the cataloguing practices used by  other universities, the leddy library decided  to use suppliers’ records for their e‐book  database, which they then adjusted by adding  the necessary fields.  in conclusion, if suppliers  offered records with higher levels of  bibliographic information, it would enable  libraries to maintain consistency in their  cataloguing more efficiently and effectively.   this would not only be beneficial to the  library’s users, but also to the publishers and  vendors with regards to the sale of their  packages.        individual purchases vs. package purchases     while most of their e‐books were purchased as  “packages,” the leddy library often found it  necessary to order e‐books individually, as  certain titles were not available within these  packages.  however, this often caused  duplication of electronic records when  individual titles already held at the leddy  library were found within these package  purchases.  as a result of this duplication of  electronic records, it was necessary to delete  many titles of the same format.      the leddy library had originally chosen to  use “single records” as its policy for  individual purchases of e‐books, by simply  updating the existing print record and adding  the necessary information to allow for  electronic format. after discovering that this  single record rule was not the norm among  other libraries, and because this decision  violated the “separate record” rule established  later in their library, leddy decided to limit  the use of “separate records” for their future  individual purchases of e‐books.       inventory control for e‐books     in january, 2009, the canadian research  knowledge network (crkn) announced to  its subscribers, that taylor and francis was  eliminating 279 of the e‐books from crkn’s  licensed collection, because the authors of  those works had withdrawn electronic rights.   to compensate for this, taylor and francis  replaced these titles with 349 new e‐books;  representing 25% more content than was  originally lost.  in addition, the canadian  reference centre removed 679 e‐books from  its cataloguing database, and other suppliers  disconnected their e‐books from their  subscribers’ opacs without explanation.   adding to the problem was that many of the  100 evidence based library and information practice 2010, 5.1  libraries’ licensed collections had to be  withdrawn due to budget constraints and  cutbacks. all of these issues directly impacted  e‐book inventory, their stability, and  ultimately, their usage.    inventory control for e‐books remains a critical  issue for both libraries and publishers alike.   other inventory related issues such as  duplication, withdrawing of records, url  status, and holdings, also need to be  addressed.  however, since many activities  associated with acquiring and activating e‐ books are substantially different from those  for print materials, inventory of e‐books is  difficult to manage.      cataloguing policy and procedures      policy    the leddy library has established several  new policies to assist patrons in using the  library’s e‐book collections, while at the same  time improving its cataloguing, processing,  and management. initially, the standard fields  used for monographs were utilized for e‐book  marc records.  special fields were added to  reflect the unique characteristics of e‐books,  such as searchable fields 099 (value=online),  710 (value=name of platform or publisher), 538  (system details note), and 856 (electronic  location and access)  $uhttp://ezproxy.uwindsor.ca/login?url=...    these changes made it possible for e‐book  titles, with their respective publishers, to be  retrieved automatically by users and library  staff alike.      after the integration of e‐books into the  voyager cataloguing system, many similarities  were discovered within the electronic  packages and the traditional collections. for  example, some of the packages, such as eco  (early canadiana online) and eebo were  simply electronic reproductions of titles that  already existed in the cataloguing database in  other formats.  when establishing new policies  and procedures for the cataloguing of e‐books,  consideration was given regarding the  decision to create single or separate records for  these diverse formats.  after much  consideration given to the issues already  discussed above, as well as those issues  relating to time, cost and stability of the e‐ book records, the leddy library decided to  use separate records for each format, as noted  earlier.  due to the nature of e‐books, the  library staff decided that no item records were  necessary, and that only bibliographic and  holdings records were needed for the  cataloguing database.            procedures      the knowledge gained by the e‐book project  and observations of other libraries’ practices  resulted in the development of these  procedures for importing e‐books:    1. before importing    a. batch edit the marc record using  marcedit software.  b. batch process records using the  marc functions mnemonic  format and marc.   c. check marc fields and their  character sets using cataloguing  standards.  d. split files, if they are too large  (more than 10,000 records in a  file).  e. add and delete fields as  necessary.     2. importing    a. check all marc fields, especially  1xx, 650, 7xx, as well as   punctuation and indicators.    b. create authority records, if  necessary.   c. always test the url link field.  d. establish holdings records.  e. do not create item records     3. after importing    a. check these records for  accessibility.   101 evidence based library and information practice 2010, 5.1  b. contact platforms or publishers, if  the record has problems with the  url.      as mentioned earlier, many suppliers provide  marc records for their e‐book packages.  this  is beneficial for libraries, as it saves them time  when editing and upgrading records, rather  than cataloguing these records manually.   however the provision of “basic level” records  from a number of suppliers leads to a range of  problems that these new policies and  procedures mitigate.     conclusions and recommendations    academic libraries have been purchasing  more electronic material than traditional print  resources.  as a result, electronic resources  have received priority within academic  libraries during the past decade.  however,  the move to electronic collections has raised a  number of issues for those involved in  cataloguing and importing e‐books.  by  tackling the challenges discussed in this paper,  academic libraries can assist in the promotion  of e‐books within the academic environment  as tools for teaching and learning.  they can  also facilitate their discovery and delivery  within library systems and services. hence,  establishing standard practices for e‐book  cataloguing is essential for academic libraries  in order to direct patrons to their electronic  collections.      establishing e‐book cataloguing policies and  procedures is an important aspect of  providing better services to patrons. libraries  need to create e‐book cataloguing standards in  order to keep records consistent, as their  adherence ensures uniform description of  resources. this will assist their patrons in  using e‐books, while at the same time helping  academic libraries manage their cataloguing  databases.     e‐books within academic libraries still pose  many challenges, such as availability, quality,  downloading, and cataloguing.  as a result of  the e‐book projects undertaken by the  university of windsor’s leddy library and  ocul, each institution established its own set  of e‐book cataloguing policies and procedures.  together they have established the foundation  for other academic libraries to build upon  when developing their own e‐book  cataloguing databases, and increasing the  usefulness of e‐books within these academic  libraries.    references     bothmann, r. (2004). cataloging electronic  books. library resources & technical  services, 48(1), 12‐19.    belanger, j. (2007). cataloguing e‐books in uk  higher education libraries: report of a  survey. program electronic library and  information systems 41(3), 204.    dillon, d. (2001). e‐books: the university of  texas experience, part 1. library hi‐tech,  19(2), 113–124.    green, k. (2003). introducing e‐books at the  university of surrey. sconul newsletter,  29, 54‐56.    gravett, k. (2006). the cataloguing of e‐books  at the university of surrey. serials, 19(3),  202‐207.    joint information systems committee. (2003).  promoting the uptake of e‐books in higher and  further education. retrieved 23 feb. 2010  from   http://www.jisc.ac.uk/uploaded_docume nts/promotingebooksreportb.pdf    lavoie, b. f., connaway, l. s., and o’neill, e.  i. (2007). mapping worldcat’s digital  landscape. library resources & technical  services, 51(2), 106‐115.    myilibrary. (n.d.)  e‐content for libraries.  retrieved 23 feb. 2010 from  http://www.myilibrary.com/?404=y    ocul, ontario council of university  libraries. (2010). what is ocul?  102 http://www.jisc.ac.uk/uploaded_documents/promotingebooksreportb.pdf http://www.jisc.ac.uk/uploaded_documents/promotingebooksreportb.pdf http://www.myilibrary.com/?404=y evidence based library and information practice 2010, 5.1  103 retrieved 23 feb. 2010 from  http://www.ocul.on.ca    primary research group inc. (2008). library  use of e‐books (2008‐09 ed.).  new york:  primary research group inc.    romano, frank. (2009). e‐books and the  challenge of preservation. retrieved 23 feb.  2010 from    http://www.digitalpreservation.gov/librar y/docs/es_e‐books.pdf      springerlink. marc record. retrieved 9 jan.  2009 from      http://www.springer.com/e‐ content/ebooks    wagner, t. (6 jan. 2009). ebrary announces on‐ demand marc records.  retrieved 23 feb.  2010 from  http://www.ebrary.com/corp/newspdf/eb rary_marc.pdf;jsessionid=miiaainak lad    weitz, j. (11 july 2006). cataloguing electronic  resources: oclc‐marc coding guidelines.  retrieved 23 feb. 2010 from  http://www.oclc.org/support/documentat ion/worldcat/cataloging/electronicresourc es/    williams, k. c., & best, r. (2006). e‐book  usage and the choice outstanding  academic book list: is there a correlation?  journal of academic librarianship, 32(5),  474‐478.    zhao, s. (2007). influence of new reading  behaviour for academic libraries: a  survey of electronic materials and print  materials for library users. iasl 2007  conference proceedings. iasl conference,  16th ‐20th july 2007. national taiwan  normal university, taipei, taiwan.              acknowledgements:    the authors would like to acknowledge the  contributions of victoria kosnik, whose  editing skills and knowledge assisted greatly  in the publication of this paper.  victoria  works with government documents at the  university of windsor’s leddy library. http://www.ocul.on.ca/ http://www.digitalpreservation.gov/library/docs/es_e-books.pdf http://www.digitalpreservation.gov/library/docs/es_e-books.pdf http://www.springer.com/e-content/ebooks http://www.springer.com/e-content/ebooks http://www.ebrary.com/corp/newspdf/ebrary_marc.pdf;jsessionid=miiaainaklad http://www.ebrary.com/corp/newspdf/ebrary_marc.pdf;jsessionid=miiaainaklad http://www.ebrary.com/corp/newspdf/ebrary_marc.pdf;jsessionid=miiaainaklad http://www.oclc.org/support/documentation/worldcat/cataloging/electronicresources/ http://www.oclc.org/support/documentation/worldcat/cataloging/electronicresources/ http://www.oclc.org/support/documentation/worldcat/cataloging/electronicresources/ evidence summary   medical librarians may be underutilised in ebm training within pediatric resident programs   a review of: boykan, r., & jacobson, r. m. (2017). the role of librarians in teaching evidence-based medicine to pediatric residents. journal of the medical library association, 105(4), 355-360. https://doi.org/10.5195/jmla.2017.178     reviewed by: alisa howlett coordinator, evidence based practice university of southern queensland library springfield, queensland, australia email: alisa.howlett@usq.edu.au   received: 2 feb. 2018                                                                     accepted: 9 aug. 2018      2018 howlett. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29418     abstract   objective – to identify the use and role of medical librarians in pediatric residency training, specifically in the teaching of evidence-based medicine (ebm) to medical residents. this research also aims to describe current strategies used for teaching evidence-based medicine in pediatric residency training programs.   design – web-based survey.   setting – pediatric residency programs within the united states of america. subjects – 200 members of the association of pediatric program directors (appd).   methods – the 13-question, web-based survey used multiple choice and short answer questions to ask how pediatric residency programs used medical librarians. the survey collected demographic information such as program name, geographic region, and program size. where respondents indicated their programs utilised librarians, the survey asked about their specific role, including involvement in ebm curricula. for respondents who indicated their programs did not use librarians, the survey asked about their reasons for not doing so, and to describe their ebm curricula. researchers used spss software to analyse the quantitative data.   main results – overall 91 (46%) appd-member program directors responded to the online survey. of these, 76% of program directors indicated a formal ebm curriculum in their residency programs. medical librarians were responsible for teaching ebm in 37% of responding pediatric programs. however, only 17% of responding program directors stated that medical librarians were involved in teaching ebm on a regular basis. the ebm skills most commonly taught within the pediatric residency programs included framing questions using pico (population, intervention, comparator, outcome), searching for relevant research literature, and critical appraisal of studies. the strategies reported as most effective for teaching ebm in pediatric residency training programs were journal clubs, regular ebm conferences or seminars, and ‘morning reports.’   conclusion – the study concluded that medical librarians may be important in the teaching of ebm in pediatric residency programs, but are likely underutilised. the librarian might not be seen has having a significant role in forums such as journal clubs, despite these being a predominant venue for ebm teaching. the authors recommend that program directors and faculty work together to better integrate medical librarians’ expertise into clinical teaching of ebm.   commentary   there have been several investigations in different types of residency program into ebm curricula and their effectiveness over the last two decades (burneo, jenkins, & bussière, 2006; green, 2000; kuhn, wyer, cordell, & rowe, 2005). however, only more recently has an understanding of the role of the medical librarian in ebm teaching emerged. zeblisky, birr, and sjursen guerrero (2015) demonstrated positive improvements to the learning experience where medical librarians have been involved in improving an ebm curriculum by being a part of an ebm sub-committee. the authors of this study further highlight the opportunity to involve medical librarians in the design and implementation of ebm teaching and to understand librarians’ role in pediatric residency programs.   an appraisal of the article using glynn’s (2006) critical appraisal checklist identified areas of concern about the study’s validity, specifically around population bias and the methods used. the study presents only one perspective related to the topic: that of program directors. the authors do not discuss how the study’s design limits or eliminates bias, nor how alternative populations are considered. however, the authors do acknowledge possible reporter bias in the responses. the authors acknowledge the use of a non-validated survey tool, though its development involved various stakeholders. the survey tool was not published with the article.   this study acknowledges a role for medical librarians’ involvement in teaching ebm curriculum in pediatric residency programs. by doing so, the authors identify a potential gap (and opportunity) for medical librarians to assist program directors to enhance the teaching and learning experience. the study also provides an update on ebm teaching methods and the skills taught in pediatric residency programs. although the survey had a 46% response rate, the population sample did represent a spread of varying program sizes, based on the number of residents.   the main finding of this study was that medical librarians are likely underutilised in ebm training. what is significant about this study is that it provides evidence of the skills most commonly taught in pediatric residency programs, and venues perceived by program directors to be most effective in teaching ebm. these skills and venues show where medical librarians can further develop their expertise and knowledge and become involved in ebm teaching in residency programs.   references   burneo, j. g., jenkins, m. e., & bussière, m. (2006). evaluating a formal evidence-based clinical practice curriculum in a neurology residency program. journal of the neurological sciences, 250(1-2), 10-19. https://doi.org/10.1016/j.jns.2006.06.013   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154   green, m. l. (2000). evidence-based medicine training in internal medicine residency programs. journal of general internal medicine, 15(2), 129-133. https://doi.org/10.1046/j.1525-1497.2000.03119.x   kuhn, g. j., wyer, p. c., cordell, w. h., & rowe, b. h. (2005). a survey to determine the prevalence and characteristics of training in evidence-based medicine in emergency medicine residency programs. the journal of emergency medicine, 28(3), 353-359. https://doi.org/10.1016/j.jemermed.2004.09.015   zeblisky, k., birr, r. a., & sjursen guerrero, a. m. (2015). effecting change in an evidence-based medicine curriculum: librarians’ role in a pediatric residency program. medical reference services quarterly, 34(3), 370-381. https://doi.org/10.1080/02763869.2015.1052702     evidence summary   health centre staff are satisfied with librarian-mediated search services, especially when librarians follow up   a review of: mckeown, s., konrad, s.-l., mctavish, j., & boyce, e. (2017). evaluation of hospital staff’s perceived quality of librarian-mediated literature searching services. journal of the medical library association, 105(2), 120-131. http://dx.doi.org/10.5195/jmla.2017.201   reviewed by: peace ossom williamson director for research data services                                                                                    university of texas at arlington libraries arlington, texas, united states of america email: peace@uta.edu   received: 11 dec. 2017  accepted: 21 feb. 2018      2018 ossom williamson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29387       abstract   objective – to determine the effects of the professional designation and communication method on clinical, educational, and research activities and related users’ reported satisfaction with and perceived quality of a librarian-mediated literature searching service.   design – online survey.   setting – a large teaching hospital in ontario, canada.   subjects – 237 health sciences centre staff who were requesting librarian-mediated literature searching over a one-year period.   methods – from february 1, 2014 to january 31, 2015, one-third of the health centre staff members requesting searching services, representing a systematic sample of the user group, were invited to participate in the survey. the survey centred on questioning participants on a critical incident, which, according to the critical incident technique, is an actual event upon which recollections are made, rather than hypothetical situations. in the case of this study, the critical incident was the service they received upon requesting literature searching by a librarian who was blinded concerning the originator of the request. with a 71% response rate, the researchers received 137 responses to the survey by health sciences staff.   participants were asked how many literature searches they had requested in the previous year, the reason they requested the service, how they submitted the request, and whether the librarian followed up for further clarification of their need. they also reported on the relevance of the results and their method of delivery, along with their perceptions of the overall quality of the service.   main results – the results came from 137 completed surveys, for a 71% response rate. physicians, nurses, and allied health professionals comprised 85% of the responses, at 35%, 27%, and 23% respectively. scientists, researchers, research coordinators, and other staff made up the remainder of responses. responses indicated frequent search requests, with the average number of searches being five, and 68% of respondents reported searching for the information themselves before contacting the library for assistance. most searches were for research/publishing (34%) and teaching/training (20%). requests were submitted via email (44%), online form (32%), in person (17.5%), and phone (6.5%), and most respondents rated themselves extremely satisfied (54%) or very satisfied (42%). most respondents (72%) reported that the librarian followed up for further clarification of the request, and staff who received follow-up rated themselves extremely satisfied at a significantly higher rate than those who did not (p=0.002). respondents whose request was submitted verbally (i.e., by phone or in person), in comparison with those whose request was submitted by email or online form, rated themselves extremely satisfied at a significantly higher rate (p=0.004) and rated the quality of results as excellent at a significantly higher rate (p=0.005).   conclusion – the need for comprehensive and expert searching when publishing or completing research and the availability of easy to use point-of-care resources may be why librarian-mediated literature searching was used for research and publishing at a rate much higher than for patient care. in addition, the fact that the institution was also engaged in efforts toward evidence-based standardization of care and electronic health records during that year may have also affected results.   while satisfaction with the service was higher for those communicating verbally with a librarian, it is unclear whether this was caused by other factors or differences between staff members who engage in phone or in-person communication and those who submit forms and online requests. because following up was correlated with higher satisfaction, adjustments in service encouraging librarians to follow up are recommended. following up in person and via phone may help further.   commentary   while some studies have investigated the sources health professionals use for finding information, little research evaluating users’ views on the features of librarian-mediated literature searching services exists. the majority of published research that has explored these services measured the number of individuals utilizing the service or the response time (brettle et al., 2011). therefore, this article evaluates elements that are understudied. complementary to research studying librarians’ perspectives on literature searching services (lasserre, 2012; mctavish, 2015), the study investigated the perspectives of health staff, including the professions of respondents, the reasons for requests, number of requests submitted, how requests were submitted, accuracy of results, and respondents’ satisfaction with the results and the service.   according to criteria from the critical appraisal tool developed by glynn (2006), the study was comprehensive and well-designed, with a number of strengths. the survey went through a pilot and multi-prong revision process to ensure its quality. the researchers also used the critical incident technique to determine an appropriate sample size, and they provided incentives for participation in order to reduce response bias. furthermore, the methods were clearly described and appropriate for the outcomes being measured, the survey questionnaire was provided in the appendix, and each outcome was measured independently and relationships between outcomes were measured using chi squared testing, which explores correlation between variables. the study design also was approved by the institutional research ethics board. the 71% response rate was also high, according to standards for social research (babbie, 2004).   the use of figures could be stronger. in figure 1, the bar chart provided displays the primary purpose of requests, but it is posited by profession, rather than by the request purpose, and a simplified accompanying chart with totals is not provided. that made it difficult to compare the various purposes on the whole. the same is true for figure 2, which displays the most important aspects of the search requests by profession. figure 3 compares the varying ratings of search results’ quality by how requests were submitted. the purpose is obscured by the chart because of the drastic difference in number of responses for the various categories. pie charts or a 100% stacked column chart would have been more effective at communicating the differences in perceived quality across mediums.   social desirability bias may have affected results as respondents may have provided answers they thought researchers would like to hear, especially since many of these respondents have ongoing contact with librarians, as evidenced by their reports of frequently submitting requests. also, some responses may have been affected by the prompt for explanation if respondents chose the negative of two options; some may have chosen the affirmative for questions like “were you satisfied with the layout/format of your search results?” in order to avoid having to type. it also was not mentioned by researchers whether the searching service had a cost, a factor which can influence users’ expectations. a consideration for further study is the inclusion of non-requesters, or staff who did not submit a request for searching service. also, in order to further explore the survey findings, a focus group or targeted interviews with some of the survey respondents could confirm, refute, or further clarify some of the conclusions the authors provided in the discussion of results.   this study provides insight into a number of outcomes. health sciences librarians may want to consider the inclusion of following up, additional fields on forms, and other aspects in their search service workflows. personal communication through direct contact should also be explored as a preferred alternative to email for communicating with users. in addition, the study provides numerous opportunities for researchers to study librarian-mediated search services, including methods of follow-up by a librarian and the comparison of the information provided during verbal and written requests.   references   babbie, e. (2004). the practice of social research. belmont, ca: thomson learning.   brettle, a., maden-jenkins, m., anderson, l., mcnally, r., pratchett, t., tancock, j., thornton, d., & webb, a. (2011). evaluating clinical librarian services: a systematic review. health information and libraries journal, 28(1), 3-22. http://dx.doi.org/10.1111/j.1471-1842.2010.00925.x   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154   lasserre, k. (2012). expert searching in health librarianship: a literature review to identify international issues and australian concerns. health information and libraries journal, 29(1), 3-15. http://dx.doi.org/10.1111/j.1471-1842.2011.00974.x     mctavish, j. (2015). creative and imaginative searching: health science librarians' strategies and barriers to providing good service. journal of the canadian health libraries association (jchla), 36(2), 69. retrieved from https://journals.library.ualberta.ca/jchla/index.php/jchla/article/view/25332/18737 evidence based library and information practice 2009, 4:2 158 evidence based library and information practice evidence summary information professional job advertisements in the u.k. indicate professional experience is the most required skill a review of: orme, verity. “you will be…: a study of job advertisements to determine employers’ requirements for lis professionals in the uk in 2007.” library review 57.8 (2008): 619-33. reviewed by: stephanie j. schulte education & reference services coordinator, assistant professor prior health sciences library and center for knowledge management ohio state university, columbus, oh, united states of america email: schulte.109@osu.edu received: 23 february 2009 accepted: 02 april 2009 © 2009 schulte. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. objective –to determine what skills employers in the united kingdom (u.k.) want from information professionals as revealed through their job advertisements. design – content analysis, combining elements of both quantitative and qualitative content analysis. orme describes it as “a descriptive non-experimental approach of content analysis” (62). setting – data for this study were obtained from job advertisements in the chartered institute of library and information professional’s (cilip) library and information gazette published from june 2006 through may 2007. subjects – a total of 180 job advertisements. methods – job advertisements were selected using a random number generator, purposely selecting only 15 advertisements per first issue of each month of the library and information gazette (published every two weeks). the author used several sources to create an initial list of skills required by information professionals, using such sources as prior studies that examined this topic, the library and information science abstracts (lisa) database thesaurus, and personal knowledge. synonyms for the skills were then added to the framework for coding. skills that were coded had to be noted in such a way that the employer plainly stated the employee would be a certain skill or attribute or they were seeking a skill or a particular skill was essential or desirable. skills mailto:schulte.109@osu.edu evidence based library and information practice 2009, 4:2 159 that were stated in synonymous ways within the same advertisement were counted as two incidences of that skill. duties for the position were not counted unless they were listed as a specific skill. data were all coded by hand and then tallied. the author claims to have triangulated the results of this study with the literature review, the synonym ring used to prepare the coding framework, and a few notable studies. main results – a wide variety of job titles was observed, including “copyright clearance officer,” “electronic resources and training librarian,” and “assistant information advisor.” employers represented private, school, and university libraries, as well as legal firms and prisons. fifty-nine skills were found a total of 1,021 times across all of the advertisements. each advertisement averaged 5.67 requirements. these skills were classified in four categories: professional, generic, personal, and experience. the most highly noted requirement was professional experience, noted 129 times, followed by interpersonal/communication skills (94), general computing skills (63), enthusiasm (48), and team-working skills (39). professional skills were noted just slightly more than generic and personal skills in the top twenty skills found. other professional skills that were highly noted were customer service skills (34), chartership (30), cataloguing/classification/metadata skills (25), and information retrieval skills (20). some notable skills that occurred rarely included web design and development skills (6), application of information technology in the library (5), and knowledge management skills (3). conclusion – professional, generic, and personal qualities were all important to employers in the u.k.; however, without experience, possessing these qualities may not be enough for new professionals in the field. commentary this study used a content analysis method that appeared to combine both quantitative and qualitative characteristics. according to white and marsh (36), qualitative content analysis requires a strong grounding in the data itself. orme took the necessary steps to prepare for coding and accurate assessment of the data by thoroughly reviewing the literature surrounding content analysis of job advertisements, looking for patterns of skills that became apparent from their results, and including the coding framework in the paper. however, closer examination of the framework seems to confound the findings of the study slightly. one could argue that the categories where certain skills were placed changed the outcome of the study, except with regard to experience, which is clearly the most frequently requested quality. for example, customer service skill is coded as a professional skill. many might consider this skill to fall in the generic category rather than professional skill. another potential coding problem is that some categories are not mutually exclusive. this can be seen in the professional skill “knowledge of the sector/subject area” and in the generic skill “subject knowledge.” we do not have the full context in which these skills were derived or classified, so it is difficult to determine if this is truly an overlap area or not. mutually exclusive categories are important in quantitative content analysis (white and marsh 32), therefore, the validity of these results could be somewhat questioned. these issues raise questions about whether the study could be replicated by another researcher using the same parameters and result in the same conclusions. this study had a lengthy literature review that adds a lot of value to the article. orme claims to use the literature review as one way to substantiate the findings of the study. however, what discussion there is about the results appears to simply be a comparison of the results rather than a true triangulation, where a researcher would use other research perspectives to re-interpret the results. evidence based library and information practice 2009, 4:2 160 additionally, the dominance of skills classified as generic and personal was not discussed by the author in relation to the debate surrounding the obsolescence of the master’s degree for information professionals, though mentioned in the literature review. on a practical note, this article is not divided into the typical sections of a research article (i.e., methods, results, discussion), making it a bit difficult to read. orme does acknowledge the two biggest limitations of this study: limiting data collection to only one source (the library and information gazette), and assuming that job advertisements sufficiently and correctly described the skills and qualities desired from information professionals. this study focused on the u.k., therefore, its generalizability is somewhat limited. the emphasis on experience found in this study was unexpected and had not been seen as the dominant factor in previous similar studies. but, this finding confirms the anecdotal experiences of many newer north american graduates who are unable to obtain professional positions partly because of their lack of prior experience (orbanus 46; holt and strock 44). findings such as this are significant and should grab the attention of both library students and library schools everywhere, reemphasizing the importance of practical and service learning while earning the professional master’s degree and also calling attention the need to produce current research to evaluate the impact of such experiential learning experiences (ball 77). works cited ball, mary alice. "practicums and service learning in lis education." journal of education for library & information science 49.1 (2008): 70-82. holt, rachel, and adrienne l. strock "the entrylevel gap, revisited." library journal. 132.16 (oct. 2007): 44. orbanus, christen. "where are all the jobs?" library journal 132.11 (15 june 2007): 46. white, marilyn domas, and emily e. marsh. “content analysis: a flexible methodology.” library trends 55.1 (2006): 22-45. evidence summary   australian academic librarians’ experience of evidence based practice involves empowering, intuiting, affirming, connecting, noticing, and impacting   a review of: miller, f., partridge, h., bruce, c., yates, c., & howlett, a. (2017). how academic librarians experience evidence-based practice: a grounded theory model. library & information science research, 39(2), 124-130. http://dx.doi.org/10.1016/j.lisr.2017.04.003   reviewed by: joanne muellenbach founding director, health sciences library university of nevada, las vegas las vegas, nevada, united states of america e-mail : joanne.muellenbach@unlv.edu   received: 8 aug. 2017    accepted: 8 dec. 2017      2017 muellenbach. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to explore and enhance the understanding of how australian library and information science (lis) practitioners experience or understand evidence based practice (ebp) within the context of their day-to-day professional work.   design – constructivist grounded theory methodology.   setting – university libraries in queensland, australia.   subjects – 13 academic librarians.   methods – researchers contacted academic librarians by email and invited each participant to take part in a 30-60 minute, semi-structured interview. they designed interview questions to allow participants to explain their process and experience of ebp.   main results – this study identified six categories of experience of ebp using a constructivist grounded theory analysis process. the categories are: empowering; intuiting; affirming; connecting; noticing; and impacting. briefly, empowering includes being empowered, or empowering clients, colleagues, and institutions through improved practice or performance. intuiting includes being intuitive, or using one’s own intuition, wisdom, and understanding, of colleagues and clients’ behaviours to solve problems and redesign services. affirming includes being affirmed through sharing feedback and using affirmation to strengthen support for action. connecting includes being connected, and building connections, with clients, colleagues, and institutions. noticing includes being actively aware of, observing, and reflecting on clients, colleagues, and literature within and outside of one’s own university, and noticing patterns in data to inform decision-making. impacting includes being impactful, or having a visible impact, on clients, colleagues, and institutions. together, these categories represent a model that explains the nature of academic librarians’ experiences of ebp. the theory describes academic librarians' experiences as complex and highly contextualized phenomena. there is no clear relationship between these categories, as data analysis did not generate a specific hierarchy of categories.   conclusion – based on the research findings the authors hypothesize that their study is one of a growing number of studies that has begun to establish an empirical basis for ebp in the lis profession.   commentary   this study highlights how academic librarians experience ebp in the workplace. the findings build upon the understanding of experiences identified in a study by gillespie (2017) that revealed similarities in the way that academic and public library professionals experienced evidence. gillespie concluded that library professionals were able to draw upon more than one source of evidence and apply their professional knowledge and experiences. librarian experiences also match the recommended practices of ebp, identified by mckibbon and wilczynski (2009), which include defining the question, finding evidence, critical appraisal, applying evidence, and evaluating the process. no divergence from recommended practices was noted.   this research study explored the lived experiences of 13 academic librarians from queensland universities at a particular point in time. further qualitative research of how ebp is experienced by lis professionals from other work settings, on different groups of participants, such as public librarians, special librarians, or academic librarians in other countries, and over a longer period of time, would strengthen the key findings and applicability of the study. the constructivist grounded theory methodology discussed by charmaz (2006) succeeded in enhancing awareness of the various experiences of ebp, how it is used, and how ebp actually happens in terms of emotional, mental, and physical experiences in a workplace setting. however, the study methods did have some weaknesses. researchers report recruiting participants via a purposive sampling approach, yet the details of this approach were not explained. while those selected were employed in a variety of roles, the results would be enhanced if the specific selection criteria had been revealed. authors also make a reference to memos, but their purpose is not clear. in addition, the study would have been more impressive if it had also identified the researchers who conducted the interviews, carried out the initial line-by-line coding of interview transcripts and memos, and conducted the data analysis.   this study has succeeded in providing new insights into how the realities of ebp are enacted, experienced, and understood for a particular group of lis professionals, at one point in time. the findings of this research study have the potential to assist library schools, associations, and others involved in educating, preparing, and supporting lis professionals at every career stage to develop the actions, knowledge, mindsets, and skills needed to facilitate and advance ebp. for example, this study could influence library schools to design and deliver new ebp curricula, or lis professionals may be inspired to start a discussion group, organize a journal club, or conduct further research on this topic. the findings may also serve to inform the design of professional development programs in evidence based library and information practice, as well as leadership in academic libraries and more broadly, within the information sector.   references   charmaz, k. (2006). constructing grounded theory: a practical guide through qualitative analysis. london, uk: sage publishing.   gillespie, a., miller, f., partridge, h., bruce, c., & howlett, a. (2017). what do australian library and information professionals experience as evidence? evidence based library and information practice, 12(1), 97-108. http://dx.doi.org/10.18438/b8r645   mckibbon, a., & wilczynski, n. (2009). pdq evidence-based principles and practice (2nd ed.).  shelton, ct: people’s medical publishing house.   evidence summary   situated cognition principles increase students’ likelihood of knowledge transfer in an online information literacy course   a review of: catalano, a. (2015). the effect of a situated learning environment in a distance education information literacy course. the journal of academic librarianship, 41(5), 653-659. http://dx.doi:org/10.1016/j.acalib.2015.06.008     reviewed by: peace ossom williamson director for research data services university of texas at arlington libraries arlington, texas, united states of america email: peace@uta.edu   received: 24 june 2016  accepted: 14 oct. 2016      2016 ossom williamson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to assess the efficacy of the application of situated cognition principles in education students’ transfer of knowledge to practice in an online information literacy course. “situation cognition” refers to a theory in which expert behaviour-modeling, authentic activity and apprenticeship, and learning environment are integral in learning.   design – a randomized controlled trial.   setting – a small private university in new york state.   subjects – 85 education college students in 7 sections of a 1-credit online course titled introduction to library research and technology.   methods – each course section was randomly assigned via cluster sampling to “situated cognition” treatment (n = 48) or control conditions (n = 37). the treatment sections provided students with expert modeling, scaffolding, authentic activity, and problem-based assessments according to the principles of situated learning and teaching for transfer; while the control sections provided students with traditional instruction of lectures and handouts.   a pretest and posttest were given to students in order to assess improvement in knowledge of the five categories for evaluating resources: currency, relevance, authority, credibility, and audience. the pretest was a survey administered during the first week of the course, and the posttest questions were included in the final exam. the researcher also used a final assignment to evaluate students’ improvement on the far transfer task (i.e., their ability to transfer a skill learned in one context for use and applicability in other contexts). the task was to create a resolution for one of three concerns raised to a school board, to write a statement addressing the received concern, and to articulate a recommended solution. two raters independently graded tasks on a scale using a rubric based on information literacy principles, including incorporating some corroborating sources, evaluating of source information, and determining a solution based on source credibility.   main results – the researcher compared the impact of the situated cognition teaching on retention using the posttest and on far transfer task of the 85 students in both treatment and control groups using multivariate analysis of variance (mancova). the mancova analysis found no significant difference between scores based on belonging to treatment or control groups. however, because far transfer occurred in 59 cases based on scoring of the tests, the researchers performed a logistic regression analysis and found the group variable (i.e., belonging to the situated cognition treatment group or control group) provided a significant prediction of transfer (p < .05), when controlling for engagement, self-regulation, and motivation. potential confounding due to variation in motivation, self-regulation, and engagement were ruled out by using the motivated strategy for learning questionnaire (mslq) and by recording of participation in weekly assignments. furthermore, presence in the treatment group increased the odds for the incidence of far transfer by a factor of 2.90.   conclusion – when testing the use of principles of situated learning (e.g., problem-based learning and cognitive apprenticeship in library instruction), the study results indicated an increase in the likelihood of students being able to practise knowledge in various situations and apply what they have learned in real-life examples. while the majority of students in both treatment and control groups demonstrated the skills of far transfer, the study provides evidence to support situated cognition, an understudied set of principles, and the study also utilizes quantitative methods to further strengthen this support.   commentary   the article largely pioneers the concept of integrating situated learning in library instruction. nichols (2009) discussed the teaching of information literacy behaviours through situated cognition strategies and identified themes in students’ transitions from action to cognition and participation in which actions (e.g., searching and taking notes) progress to cognition direction (i.e., becoming more knowledgeable about a subject) and leading to participation in the academic community and engaging in successful scholarly communication. this article furthers the aims of the proposed model nichols (2009) presented because it provides more concrete evidence due to research design moving from case studies to a randomized controlled trial.   strengths of the study include its detailed background and investigation into current research and practice around implementation of situated cognition principles. the author’s use of concepts from problem-based learning and cognitive apprenticeship reveal the ability to integrate various models in order to establish an intervention meaningful for the population being studied. in addition, the use of all sections of a course prevents bias due to self-selection, which often occurs when students choose whether to participate in a research activity. a limitation of the study is that it has limited generalizability due to its occurring within one course and with a smaller sample size.   furthermore, data collection, analysis, and coding methods were clearly defined. the author also provides the details of the assignment to allow for replication. the logic behind the assignment, its coding, and the analysis tools being used were well described, and evidence documenting the support for these practices was included; however, data loss is a possibility when the assignment submissions were coded into binomial scores. confounding variables, including engagement assessed by weekly participation and motivation assessed using the mslq, were accounted for, strengthening the validity of the outcome. based upon calculations from the ebl critical appraisal checklist (glynn, 2006), validity overall and for each section was over 85%, clearly meeting the established 75% threshold for validity in study methods and reporting.   the study contributes to evidence supporting situated learning and problem-based learning as effective methods for teaching information literacy skills in education students. one outcome of the research is that it provides new methods for developing curriculum for librarians who teach information literacy. the study also presents several effective strategies for providing students with skills applicable for work within their disciplines and the ability to build upon these skills using methods taught by frameworks and models arising out of the field of education. most evident is the clear need for continued research in this realm.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154   nichols, j. t. (2009). the 3 directions: situated information literacy. college & research libraries, 70(6), 515-530. http://dx.doi.org/10.5860/crl.70.6.515     evidence summary   without library resources and services, the scholarly activity of medical faculty and residents would register a code blue   a review of: quesenberry, a. c., oelschlegel, s., earl, m., leonard, k., & vaughn, c. j. (2016). the impact of library resources and services on the scholarly activity of medical faculty and residents. medical reference services quarterly, 35(3), 259-265. http://dx.doi:org/10.1080/02763869.2016.1189778   reviewed by: kathleen reed assessment & data librarian vancouver island university nanaimo, british columbia, canada email: kathleen.reed@viu.ca     received: 1 dec. 2016    accepted: 8 feb. 2017      2017 reed. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – this study aimed to determine the use of three library services – literature search service, article delivery service, and library resources – among medical faculty and residents with regard to scholarly activity.   design – survey.    setting – medical library and health information centre at a large university in the united states of america.   subjects – 65 medical faculty and residents.   methods – the authors sent out 433 invitations to participate in a 23-question survey via an email distribution list. a total of 65 individuals participated, for a response rate of 15%. questions related to the use of library services for scholarly activity, patterns of information-seeking behaviour, and instructional needs. comments were allowed on several questions, and a final open-ended question was included.   main results – all respondents used pubmed at least a few times a year, with 71% selecting it as their first choice to search for articles. only 20% prioritized google or google scholar above pubmed as the first place to begin a search. the most popular reasons for using library resources were “lectures, papers, research, and patient care” (p.262). the first three of these activities are types of scholarly activity.   of the 65 respondents, 46% published article(s) or book chapter(s). within this group of authors, 67% of residents undertaking scholarly activity requested a literature review, 100% accessed online material themselves, and 67% requested articles. faculty placed similar importance on these services, with 71% having requested a literature review, 87% having accessed materials themselves, and 75% having requested articles. among those respondents who presented posters or papers, there was high use of library services, ranging from 59% of faculty requesting a literature review to 98% of faculty accessing online material themselves.   conclusion – the library is a key resource for faculty and residents undertaking scholarly activity. however, faculty members use the library’s services and resources for publishing articles and book chapters more than residents do. this may be because of “publish or perish” pressure, or because faculty have less time to locate research by themselves. surveys are useful to ensure the library’s resources and services align with the needs of the user community. inclusion of free-text comment boxes in the survey allowed users to put a “personal face” (p. 264) to their comments that would have otherwise not been captured.   commentary   as academic institutions place more value on quantifying the effect of dollars spent, it becomes increasingly important for libraries to demonstrate that the resources and services they offer are needed and used by the communities they serve. this is particularly important when personnel comprise a large portion of a library budget, as it may mean the difference between keeping or losing an individual’s job. in the case of this library, in which personnel comprise 50% of the library budget, this type of value survey research is vital.    an additional benefit of the study is that it works to test librarian assumptions. in this case, it would be unsurprising to learn that many librarians would likely assume that pubmed, being a key medical database, would be highly used in a medical library. another assumption that our profession may hold is that access to medical information will assist in patient care. the rochester study (dunn, brewer, marshall, and sollenberger, 2009) confirmed that 75% of medical professionals definitely or probably handled some aspect of patient care differently as a result of access to information via a library. as demonstrated in this article, testing assumptions that librarians hold is critical, so that we are able to make decisions from evidence and not guesses.   in order to enhance these findings overall, the authors might consider bolstering their survey results with database usage numbers and trends. while database numbers cannot indicate the intended use of the articles accessed, they can show trends in access and cost-per-use data which over time may contribute to the case for keeping or cancelling particular products.    another way in which the authors could strengthen this article is a more fulsome discussion of their assertion of the importance of including several opportunities for survey participants to write in free-text comments. the authors make this claim, but do not give readers any examples or analysis. the answers received via these comment boxes could bolster the evidence that library services and resources are important, or give the library feedback on how services and resources could be improved.   while the authors acknowledge that the small sample size (n=65) and low response rate (15%) limit the transferability of results to other academic medical libraries, this article is a fine example of how a simple survey can provide valuable information to librarians and administrators. as the authors mention, the results of this survey show that the library is making a positive difference to the research activity of faculty and residents, thus aligning the library with the university’s strategic plan. it also provided librarians with comments from users related to their issues and priorities, which is a helpful feedback tool to inform future service and resource decisions.    references   dunn, k., brewer, k., marshall, j. g., & sollenberger, j. (2009). measuring the value and impact of health sciences libraries: planning an update and replication of the rochester study. journal of the medical library association, 97(4), 308-312. http://dx.doi:org/10.3163/1536-5050.97.4.016   evidence summary   promoting the library to distance education students and faculty can increase use and awareness, but libraries should assess their efforts   a review of: bonella, l., pitts, j., & coleman, j. (2017). how do we market to distance populations, and does it work?: results from a longitudinal study and a survey of the profession. journal of library administration, 57(1), 69–86. https://doi.org/10.1080/01930826.2016.1202720    reviewed by: judith logan assistant head, user services john p. robarts library university of toronto libraries toronto, ontario, canada email: judith.logan@utoronto.ca   received: 13 aug. 2019                                                                 accepted:  18 oct. 2019      2019 logan. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29637     abstract   objective – to determine if library promotion efforts targeted at distance education students and instructors were successful and in line with similar activities at other institutions   design – mixed: longitudinal and survey questionnaire   setting – large publicly-funded, doctoral-granting university in the midwestern united states   subjects – 494 distance education students and instructors in 2014 compared to 544 in 2011 and “more than 300” (bonella, pitts, & coleman, 2017, p. 77) professionals at american academic libraries.   methods – in the longitudinal study, the researchers invited all distance education students and instructors who were active in the 2010-2011 academic year (n = 8,793) and the spring 2014 semester (n = 4,922) to complete an online questionnaire about their awareness and use of library’s services. questions were formatted as multiple choice or likert scale with optional qualitative comments. the researchers used descriptive statistics to compare the responses.   then, the researchers invited library professionals via relevant distance-education and academic library listservs to complete an online questionnaire about how distance education is supported, promoted, and assessed. free text questions comprised the majority of the questionnaire.  the researchers categorized these and summarized them textually. the researchers used descriptive statistics to collate the responses to the multiple-choice questions.   main results – the researchers observed an increase in awareness of all the library services about which they asked undergraduates. off campus access to databases (92%, n = 55), an online course in the learning management system (78%, n = 47), and online help pages (71%, n = 43) had the highest awareness in 2014 as compared to 2011 when off campus access to databases (73%, n = 74), research guides (43%, n = 44), and online help pages (42%, n = 43) were the top three most visible items. fewer undergraduates said they do not use the library at all between 2011 (54%, n = 56) and 2014 (30%, n = 18).   more graduate students reported that they were very satisfied with the library in 2014 (45%, n = 12) than in 2011 (27%, n = 10). faculty members were more aware of library services, especially research guides, which had 79% awareness in 2014 (n = 56) up from 60% (n = 55) in 2011. almost half (46%) of faculty member respondents had recommended them to students in 2014 as compared to 27% in 2011.   the library professionals who responded indicated that their institutions did not evaluate the success of distance educators and students’ awareness of the library’s services and resources (54%, n = 97) nor the success of any promotional campaigns they may have undertaken (84%, n = 151). both the respondents (37%, n = 54) and the authors recommended partnering with faculty members as a best practice to promote the library. conclusion – more libraries should be marketing specifically and regularly to distance education students by leveraging existing communication and organizational structures. assessing these efforts is important to understanding their effectiveness.   commentary   the association of college & research libraries’ (acrl) standards for distance learning library services begin by espousing that everyone at an academic institution is “entitled to the library services and resources of that institution…regardless of ...the modality by which they take courses” (para. 1). this frames services and resources for distance education students and instructors as an equity matter. promotional efforts targeting this community like email campaigns or embedded content within the institution’s learning management system (lms) are designed to increase awareness and use of the library, but how do we know if they are effective? as the results of the authors’ longitudinal survey indicate, it is rare for libraries to assess their promotional efforts, but those promotions could be an individual’s only contact with their library.   the results of the longitudinal study suggest that the promotional interventions employed at the institution were successful. however, the authors rely heavily on face validity and miss the opportunity to perform a more thorough analysis (glynn, 2006). the descriptive statistics presented in this survey are interesting and suggestive, but without statistical analyses we are unable to determine if they are significant. similarly designed studies have used t-tests for this purpose (england, lo, & breaux, 2018).   in surveying library professionals, the authors added value to their longitudinal study by contextualizing their promotional interventions and assessment efforts in common practice. these responses and the authors’ experiences were used to create a list of “best practices” in supporting distance learners and instructors. these suggestions were all sensible and helpful, though it seems curious that the authors chose to call them best practices considering their survey showed that little formal assessment had been done. more research like the longitudinal survey presented could help validate these suggestions as evidence-based best practice.   practitioners can make use of this mixed methods study in two ways. firstly, the longitudinal survey presented a realistic and replicable model for assessing the effectiveness of library promotions. it could be implemented with distance learners or any other distinct population within the library community. analyzed statistically, results could indicate where gains have been made. secondly, the emerging best practices presented provide library professionals working with distance populations some suggested activities and approaches to service and promotion design.   references   association of college & research libraries (acrl). (2016, june). standards for distance learning library services. retrieved from http://www.ala.org/acrl/standards/guidelinesdistancelearning   england, e., lo, l. s., & breaux, a. p. (2018). the librarian bff: a case study of a cohort-based personal librarian program. journal of library & information services in distance learning, 12(1-2), 3-12. https://doi.org/10.1080/1533290x.2018.1467810    glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387–399. https://doi.org/10.1108/07378830610692154      evidence summary   comparison of print monograph acquisitions strategies finds circulation advantage to firm orders   a review of: ke, i., gao, w., & bronicki, j. (2017). does title-by-title selection make a difference? a usage title analysis on print monograph purchasing. collection management, 42(1), 34-47. http://dx.doi.org/10.1080/01462679.2016.1249040   reviewed by: laura costello head of research and emerging technologies stony brook university libraries stony brook, new york, united states of america email: laura.costello@stonybrook.edu   received: 1 june 2017    accepted: 25 july 2017      2017 costello. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to compare usage of print monographs acquired through firm order to those acquired through approval plans.   design – quantitative study.   setting – a public research university serving an annual enrollment of over 43,500 students and employing more than 2,600 faculty members in the south central united states.   subjects – circulation and call number data from 21,356 print books acquired through approval plans, and 23,920 print books acquired through firm orders.   methods – item records for print materials purchased between january 1, 2011 and december 31, 2014 were extracted from the catalog and separated by acquisitions strategy into firm order and approval plan lists. items without call numbers and materials that had been placed on course reserves were removed from the lists. the authors examined accumulated circulation counts and conducted trend analyses to examine year-to-year usage. the authors also measured circulation performance in each library of congress call number class; they grouped these classes into science, social science, and humanities titles.   main results – the authors found that 31% of approval plan books and 39% of firm order books had circulated at least once. the firm order books that had circulated were used an average of 1.87 times, compared to approval plan books which were used an average of 1.47 times. the year-to-year analysis showed that the initial circulation rate for approval plan books decreased from 42% in 2011 to 14% in 2014, and from 46% to 24% for firm order books. subject area analysis showed that medicine and military science had the highest circulation rates at over 45%, and that agriculture and bibliography titles had the lowest circulation rates. subject area groups showed the same pattern, with books in the social sciences and sciences experiencing more significant circulation benefits to firm order purchasing.   conclusion – monographs acquired through firm orders circulated at a slightly higher rate than those acquired through approval plans.   commentary   this study centers on print collection development practices. the authors quickly and correctly identify that there is conflict in this genre, alongside a lack of generalizability because of differences in scope, sample size, and methodology. in accordance with this, the authors cite two studies that found circulation advantages for titles purchased via approval plan (ellis, ghouse, claassen-wilson, stratton, & clement, 2009; tucker, 2009) and another that found advantages to firm orders (tyler, falci, melvin, epp, & kreps, 2013). though the findings may vary for any given library, there is plenty of food for thought in this article. the study examined all of the public research university’s print monograph circulation data within a relatively narrow range of years, so the findings are able to avoid some of the complications of assessing collections that have accumulated slowly over a long period of time (fry, 2015).   the sample pulled out approval plan purchases with a note on the item record and firm orders represented the other print purchases, so it is difficult to tell exactly how the firm orders were collected. for example, some of the disciplines may have had more faculty involvement in requests, while others were selected entirely by the librarian. the study did find a particular advantage to firm orders that represented purchases outside of the disciplines offered by the university or spanning multiple disciplines; this could suggest that the flexibility of human intervention over the fixed approval plan was part of the reason for its advantage. the authors also point out that librarians seemed to alter their firm orders in response to changes in their academic programs, while leaving their approval plans static.   beyond the scope of the research question, the authors identify a low percentage of total circulations to purchased materials in both firm and approval orders, and a steeply declining circulation rate for all print monographs over the study period. the sample excluded serials and electronic materials, so it is not clear whether usage is migrating to non-print platforms, moving outside the library, or disappearing altogether.   this study is clearly outlined and very replicable as described in glynn’s critical appraisal tool (2006). it would be useful to replicate it in similar and different institutions to establish better baselines for print circulation statistics, especially since this study flagged a decline in print circulation rates over the past few years. if this is true of other institutions, deep changes should be in the works for collection development processes. this study only used descriptive statistics, so this type of work would help determine standards for the significance of collections data, which could help add meaning to statistics like these. there are many other methods for providing access to scholarly content; libraries would benefit from reputable collection usage benchmarks to indicate when change should occur in collection development strategies.   references   ellis, e. l., ghouse, n. j., claassen-wilson, m., stratton, j. m., & clement, s. k. (2010). comparing approval and librarian-selected monographs: an analysis of use. in d. orcutt (ed.), library data: empowering practice and persuasion (pp. 53-68). santa barbara, ca: libraries unlimited. fry, a. (2015). conventional wisdom or faulty logic? the recent literature on monograph use and e-book acquisition. library philosophy and practice, (1), 1-27. retrieved from http://digitalcommons.unl.edu/libphilprac/1307   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154   tucker, j. c. (2009). collection assessment of monograph purchases at the university of nevada, las vegas libraries. collection management, 34(3), 157-181. https://doi.org/10.1080/01462670902962959   tyler, d. c., falci, c., melvin, j. c., epp, m., & kreps, a. m. (2013). patron-driven acquisition and circulation at an academic library: interaction effects and circulation performance of print books acquired via librarians’ orders, approval plans, and patrons’ interlibrary loan requests. collection management, 38(1), 3-32. https://doi.org/10.1080/01462679.2012.730494   research article   an analysis of student performance at the intersection of diversity and information literacy   nastasha e. johnson assistant professor of library science physical and mathematical sciences information specialist purdue university libraries and school of information studies purdue university west lafayette, indiana, united states of america nejohnson@purdue.edu   nathan mentzer associate professor engineering/technology teacher education dept of technology leadership & innovation dept of curriculum and instruction purdue university west lafayette, indiana, united states of america nmentzer@purdue.edu   received: 26 apr. 2018                                                                  accepted: 26 apr. 2019      2019 johnson and mentzer. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29438     abstract   objective – when teaching information literacy (il) concepts, instructors often have no knowledge about the background or previous il exposure of the students they are teaching. this study aims to create a holistic picture of the students at a large midwestern united states university in a first year introductory course on the design process for solving engineering problems. methods – institutional data and course level data were traced and linked to individual students in an introduction to design thinking first year course. this course is at a major high research activity institution in the midwestern united states. from a total course size of 650, institutional and course level data of 127 students were selected randomly and analyzed. some data points are self-reported and some data points are performance-based.   results – underrepresented minorities (urms) had a higher increase in il score from assignment 1 to assignment 3 than non-urm students. however, non-urms performed higher on both the first and the last assignments. students in concurrent il designated courses had a higher increase from assignments 1 to 3 than those not in simultaneous il designated courses. black and international students had the highest increases from assignments 1 to 3 of any demographic. regarding il, the fact that none of the students had been exposed to much il instruction justified continued collaboration in the course between the instructor of record and the il specialist. there were significantly negative correlations between the final grade and first-generation status. legacy students also performed more poorly from assignments 1 to 3.   conclusion – students are more diverse in a single classroom setting than presumed prior to research; therefore, our instructional practices should be diverse and inclusive, as well. more preparation work and fact finding should be conducted by library faculty and instructors to facilitate the learning of the students, and not just the act of teaching. librarians could ask for more information about the course demographics and respond accordingly. librarians should also be properly trained in instructional practices to be better equipped to meet the expectations and challenges of teaching a diverse class.     introduction   in higher education, library faculty and instructors rarely know about the backgrounds of the students we teach. we are often encouraged to teach to a variety of types of students, not completely understanding what that means or looks like in the context of our individual classes. furthermore, librarians may be at a disadvantage, not knowing the information literacy (il) skills of the students that we teach. we may be invited to speak on a topic, with little to no knowledge about what they are preparing for and how much they may already know. information literacy instruction sometimes occurs as if in a vacuum, with little knowledge about the background and il exposure of the students taught. this same lack of knowledge often applies when we design our own courses. we often teach what we want them to know, but not what they are prepared to understand.    not only are past il experiences ignored, so are students’ concurrent research and il experiences. we know that preparedness gained in high school can have an impact on performance in college, including research practices (mccarron & inkelas, 2006). preparedness can sometimes be related to the rigor and resources of their high schools (roderick, coca, & nagaoka, 2011) but also to family finances (bettinger, 2004; castleman & long, 2016) or not having exposure to those who have attended college before, i.e., first-generation college students (bui, 2002; pike & kuh, 2005; stroud, 2017). there have been many studies on the intersection of gender and grade performance, especially in stem disciplines (hubbard, 2005; severiens & ten dam, 2012). furthermore, information literacy and library usage are positively correlated with student matriculation (soria, fransen, & nackerud, 2013).   in the lis literature, the relationship between student success and information literacy has been well documented. in 2014, soria, fransen, and nackerud conducted a series of regression analyses of over 5000 students and found that students who had used the library resources and services at least once in the first year had a higher gpa than those who did not. though several data points were collected, they did not specifically report about the performance differences of students along ethnicities, income, and other “pre-college” data. there was an opportunity to explore the library use and exposure along with a variety of data points, which this study will do.   the lis profession is continuously challenged to think beyond race and ethnicity to include other diversity measures. specifically, we were challenged to expand our definition of diversity to include “underrepresented, disadvantaged, and underserved in terms of information” (jaeger, subramaniam, jones, & bertot, 2011, p. 11).  based on their definitions, diversity expands to include any people who may not have the best access to information, whether it is because of language barriers, access to technology, or statistical status as a minority. according to fabbi’s (2015) research on the use of the iskills assessment of information and computer testing, she found that there are four predictive variables to a high school student’s success: student’s best language, race, cumulative gpa score and honors/non-honors curricular paths. this was supported by huerta and watt’s (2015) work that also said that gpa and ap courses in high school predicted college success. after high school, more research is needed to explore those predictive variables over time. conversely, lanning and mallek (2017) found that students’ high school performance and demographics had no influence on their information literacy performance. they collected demographic and high school information, along with admission test scores. only their current gpa and act were relevant in their post-test regression analysis of il performance.    library instruction and cultural competence is an emerging area of interest for researchers. understanding diversity is quite different than being culturally competent and adept when working with people who are different than you, especially in an instructional setting. lori s. mestre’s research (2009) has been looking at cultural competence in k-12 and college environments. in 2009, she published a work that found a significant gap in the cultural competence training of librarians before professional positions (mestre, 2009). she found that such training would help librarians modify their instruction to be more culturally non-offensive (mestre, 2009). in 2010, in the book librarians serving diverse populations (mestre, 2010) she expanded on her earlier research to suggest how librarians could be trained in intercultural competence, as well as in strategies for library administration and library school curriculum development to effect positive change for professionals and pre-professionals. some of the efforts include strategic assessment and ongoing training on incorporating multiculturally sensitive stories in the lesson planning (mestre, 2010, pp. 100-101).    in 1991, marilyn loden and judy rosener published pioneering work on the dimensions of diversity (loden & rosener, 1991). in their book, they introduced the diversity wheel, with primary and secondary levels of diversity of individuals and institutions. the first level of diversity represents the internal dimensions of diversity, characteristics that influence self-identity. the six dimensions on the first level are age, gender, sexual orientation, physical ability, ethnicity, and race. the second level of diversity represents external characteristics that influence social identity. the 10 dimensions on the second level are: marital/family status, parental status, geographic location, income, personal habits, recreational habits, first language, work experience, educational background, and work experience. the original dimensions were expanded in 2010 to include income, class, and spiritual beliefs. these dimensions and characteristics of diversity can influence how people value themselves and those around them. because of the value placed on these dimensions, individually and collectively, the dimensions of diversity can positively or negatively influence interpersonal interactions in the classroom (milem, chang, & antonio, 2005). understanding these dimensions and where students appear within the social construct of the classroom is within the realm of responsibility for teaching faculty who are interested in effectively teaching to all walks of students (milem et al., 2005). for this study, we will relate these dimensions of diversity with student performance on assignments to understand more about their performance along those dimensions. the dimensions are many of the data points collected by the university or self-reported by students at admission. we will collate those variables to create a holistic picture of the students in the course studied.   background of the course   according to the association of american colleges and universities (aacu) information literacy value rubric, information literacy is “the ability to know when there is a need for information, to be able to identify, locate, evaluate, and effectively and responsibly use and share that information for the problem at hand” (association of american college and universities, 2019). in the classroom setting it may be manifest as written assignments, projects, or other learning objects that require research and producing an assignment or experience. according to criterion 3 of the aacu rubric, the student should be able to evaluate information and its sources critically. additionally, in criterion 5, students should access and use information ethically and legally. the course studied, tech 120, is a first-year gateway course which introduces students to design thinking for solving problems. the steps of the design thinking include utilizing available information at each step, including defining the problem, brainstorming solutions, and developing and testing a prototype. the aacu il standards, not the current or previous acrl il standards, are the approved definitions used to create the “information literacy” core curriculum designation by university administration at the institution where the study was held. the learning objectives for this course, and others that are considered core curriculum il designated courses at the institution, are created using the aacu il standards.   tech 120, technology and the individual, an introduction to technology design, typically enrolls approximately 650 students each year, most of whom go on to pursue majors in science, technology, engineering and mathematics (stem) disciplines. it is the gateway course to the college of technology and is a required course for all of the majors in that college. most of the students are first-year students. tech 120 also fulfills the information literacy course category of the general education core curriculum requirements that all students must complete before graduating. a librarian has been an integral part of the course design and has contributed information literacy-related content through the entire length of the semester-long course, including assistance with rubric design and assignments.    student assignments are produced along the design thinking continuum of designing prototypes, including three il-specific assignments that were analyzed as a part of this study. the first assignment is a bibliography created by the students on pedestrian safety at crosswalks, after watching a librarian-created video on keyword selection and the basic use of the databases google scholar, engineering village, and academic search premier. the second assignment is a repeat of the first assignment, after a librarian in-class visit to review the databases and answer questions about their experiences. at the end of the semester, as a final project and the third assignment of this study, students produce academic-style poster presentations about a technological problem and solution within their college. a bibliography section is included in the rubric for this assignment and is also a part of the optional templates provided. the self-selected problems vary from mechanical/facilities problems to student time-management problems.    aims   this study aims first to create a holistic picture of the lives of the students in a single technology course; capturing demographic data, high school rank, pell grant eligibility, college transcript, and other institutional data and assessments. we want to investigate which demographics and common categories of diversity, i.e., underrepresented minorities (urms), first-generation and legacy students (relatives, usually children, of a graduate of a school), and family financial contribution, correlate to their il performance in a first-year course. we also investigate how students perform who have taken il courses before or concurrently.     methods   two separate irb approvals were granted for this study. the first irb protocol enabled the ethical use of the student assignments for citation analysis. the second irb protocol granted consent to engage in data agreements with financial aid, the registrar, and admissions for the ethical use of the institutional data of the students studied in the first irb. the institutional data was paired with the citation scores of the assignments completed by the students.   variables   there are 11 non-il independent variables in this project, along with 3 il dependent variables. the variables are defined in table 1.   table 1 project variable definitions high school name & location secondary institution listed on the student’s transcripts, geographic location course grade data final grade letter & value the final grade submitted to the registrar’s office, and its weight major selected course of undergraduate study gender commonly referred to as “sex”, self-identified biological and physiological characteristics that denote male and female, as defined by the world health organization ethnicity identified as having the physical characteristics of a particular ethnic or cultural group; one of 6 options: 2+ races, asian, black/african american, hispanic/latinx, international, white; includes non-domestically/foreign born underrepresented minority (urm) status university assigned; denotation of the student as an underrepresented racial minority, such as latinx, african american/black, or asian american.  does not include non-domestically/foreign born semester gpa cumulative grade point average with all coursework in the semester studied overall gpa cumulative grade point average with all coursework in entire college career first generation status whether a student’s parents have not attended or graduated from a higher education institution legacy status whether a student’s parents or other immediate family members have attended the institution where the study was held birthdate the date when a student was born pell grant eligibility whether a student’s financial contribution or family’s contribution makes them eligible for need-based federal financial aid, i.e. how much of the cost of education can be provided by the student and/or the parents. assignment one an annotated bibliography collected and analyzed using the rubric in the appendix, before librarian-led instruction assignment three a bibliography collected and analyzed using the rubric in the appendix at the end of the course average (avg) il difference average difference in the citation scores between assignment 1 and assignment 3 concurrent il status whether student is enrolled in another il designated core curriculum course during the same semester such as freshmen english   citation analysis process   a 127-student sample population was randomly selected from a total 650-student course, across 17 sections. each of the students was assigned a number, and numbers were selected using an online randomizer, www.randomizer.org. student assignments were collected and analyzed using a customized three-point scale rubric based on the craap test (meriam library, 2010) on the elements of currency, relevance, authority, accuracy, and purpose of the citations rendered. the author created a 3-point scale to measure the merit of each criterion, from low (1) to high (3). three separate assignments were collected: 1) a bibliography after watching an online il video, 2) a bibliography created after a librarian-facilitated face-to-face session, and 3) an end-of-course project bibliography. however, for this study, we evaluated the difference in il performance from assignment 1 and assignment 3. that is, we evaluated the difference between an assignment early in the semester with an assignment at the end of the semester. those il results were then paired with institutional data about each student. four librarians, in two teams of two, normed the citation scores of the students’ assignments to establish inter-rater reliability. the librarians randomly selected 10 assignments, measured them individually, and then discussed them to normalize the scores given. librarians met three times to discuss the scores due to the number of assignments and to ensure consistency over time. the librarians were from different disciplines or departments, in order to minimize the subjective bias inherent with being familiar or unfamiliar with the disciplines that the students cited.   institutional data collection   data sources include the campus learning management system, the campus office of institutional research, assessment and effectiveness, and the financial aid office. some data points, such as the first-generation status, are self-reported, and other data points are performance based. data were retrieved via ibm cognos analytics, which is web modeling and analysis software. all identifiable data were anonymized by a campus data analyst prior to being shared with the other authors. project metadata was kept via a shared google document.   data were stored using excel spreadsheets. data analysis was conducted using minitab, excel, and tableau. we used descriptive and inferential statistics to determine how different demographics and preparedness affect performance. minitab was used to calculate pearson correlations to determine whether or not there were any associations between variables. next, excel was used to test for significance. one-sided t-tests and anova t-tests were conducted to determine p values. cohen’s d was used to determine effect size. a t-test’s statistical significance indicates whether or not the difference between groups’ means most likely reflects a real difference in the population from which the groups were sampled. finally, tableau was used to create data visualizations to get a view of the demographic breakdowns.   results   after compiling the data, we found the overall demographics of the class. students from outside of the college of technology comprised 9% of those enrolled, which means that 91% were college of technology majors. the class consisted of 81% that were self-identified as male, 69% white, and 13% underrepresented minorities. a total of 76% of the students were freshmen.    concurrent enrollment in an il course   a total of 62% of tech 120 students were not concurrently enrolled in another information literacy core curriculum course. however, 32.28% (41) students were also enrolled in engl 106, the cornerstone english course required by all freshmen. another 5.51% (7) were enrolled in stat 301, which also fulfilled the il requirements as required by the university.    figure 1 percentage of students in concurrent il course.   figure 2 citation score difference of students in concurrent il course.   prior enrollment in an il course   most students, 83.46%, had not completed an il categorized core curriculum course prior to enrollment in this course. however, one student who scored considerably worse on both assignments had taken stat 301 previously and had the largest difference between assignments.   figure 3 percentage of students who previously enrolled in il course.   figure 4 citation score difference of students who previously enrolled in il course.   ethnicity   in terms of ethnicity, 68.50% (87) of the students identified as white and 8.66% (11) of the students identified as asian or asian american. in terms of nationality, 7.87% (10) students were identified as international students or non-domestically born. nearly 10% of the students were identified as an underrepresented minority, with 4.72% (6) black/african american, 5.51% (7) latinx, and 5 (3.94%) that identified themselves as multiracial.    figure 5 ethnic backgrounds of the students.   figure 6 citation score difference by ethnic background.   first generation status   first-generation college students made up 20% (26) of the sample. there were 101 students (79.53%) who reported that at least one parent or both parents had attended a higher education institution. interestingly, the difference in il performance was greater with non-first generation students than first-generation students. that is, students who were exposed to family members who had a college education had a greater performance gap than those who did not have a family member who had attended college. figure 7 first generation status of students.   figure 8 citation score difference based on first generation status.   gender status   only 24 students (18%) were female, while the remaining 81% were male. the difference in il performance was greater among male students than female students.   figure 9 gender of the students.   figure 10 citation score difference by gender.   legacy status   a total of 63.78% (81) of the students were not the immediate family members of university alumni. the remaining students had a parent, sibling, or another relative that attended the university. most students with relatives who attended the university had a positive il performance difference from assignment 1 to assignment 3. however, students whose parents attended the university had a negative il performance difference, which means that they actually did worse on the final assignment than the first assignment.   figure 11 legacy status of the students.   figure 12 citation score difference by legacy status.   pell grant eligibility   although 89 students were not eligible for the pell grant, 30% (38) of students were eligible for the grant. those who were eligible for a pell grant had a greater il difference, denoting a larger improvement from assignment 1 to assignment 3.   figure 13 pell grant eligibility by student.   figure 14 citation score difference by pell grant eligibility status.   underrepresented minority status   a significant majority of students, 87% (111), were not underrepresented minorities. a count of 16 students identified as urm; 13% of students were identified as black/african american, asian american, or latinx. based on the ethnicity data, multi-racial students may also be grouped with underrepresented minorities. this is unclear, but the data supports this as a possibility. students who were urm had a larger il difference from assignment 1 to assignment 3 than those who were non-urm.   figure 15 underrepresented minority status.   figure 16 citation score difference by underrepresented minority status.   information literacy   overall with all variables controlled, the average citation score for all students was 2.289 on the first il assignment, on a scale of 1 to 3, with 30 students scoring below 2. the overall average citation score on the second assignment was 2.532, with 3 students (8.66%) scoring below 2. this suggests growth in overall il performance for the entire sample of 127 students from assignment 1 to assignment 3.   table 2 average citation score, cumulatively   assignment #1 assignment #3 average il score 2.289940031 2.532168551   discussion   diversity   diversity within higher education can be defined along many variables; including, but not limited to, gender, ethnicity, urm status, and economic contributions. research has shown that gender (moss-racusin, dovidio, brescoll, graham, & handelsman, 2012)) does influence the performance of females in stem. there are inherent and explicit biases in the classroom that can dictate the success of a diverse group of students (greenwald & krieger, 2006; gregory, skiba, & noguera, 2010; hill, corbett, & st rose, 2010; jacoby-senghor, sinclair, & shelton, 2016; staats, 2015). according to the findings, there was a significantly negative correlation between the il performance of urm students and their course grade, suggesting the grades of urms decreased in relation to minority status. there were no other significant correlations found between those variables identified as pertaining to diversity, gender, and pell grant eligibility. our findings are supported by the literature that urms can perform more poorly academically in certain settings. however, our study found no significant differences along gender lines nor economic status.    table 3 t-tests scores comparing urm status, gender, and pell grant eligibility status with il performance   urm (y=1) gender (m=1) pell-eligible (y=1) il 1 -0.093 -0.072 -0.062 il 3 0.05 -0.017 0.004 change in il 0.12 0.063 0.064 tech 120 grade -0.209* -0.087 -0.109 f14 term gpa -0.127 -0.142 -0.1 *statistically significant at p<.05 exposure   an important interest in the study was to find the influences of pre-college and concurrent college experiences on the performance of il related assignments. one of those influences is exposure to formal il instruction in other courses. we ran pearson’s correlations to determine the relationship between exposure to concurrent and prior courses. there was a significantly negative correlation between the performances on assignment 1 and being concurrently enrolled in another il course. that is, students who were in two il designated courses simultaneously performed poorer on assignment 1 than those enrolled in the single course. different il topics, techniques, and course elements could be the reason for the difference in performance. librarian involvement could also be a contributing factor, as il is a significant portion of the learning outcomes for the course and the librarian was significantly involved with the course design of the studied course. it is unknown whether a librarian was involved with the design of other courses.     preparedness   college readiness can be an accurate measure of performance capability at the college level. college readiness is often denoted with the academic rigor of the courses offered and taken in high school (roderick et al., 2011). however, college readiness can also be attributed to exposure through social networks like family and fellow students (bui, 2002). we found that there was a significant negative correlation between the il performance of first-generation students and their gpa during the term of the study. that finding suggests that first-generation students perform less than their counterparts both in il performance and overall for the course and the term studied. this is consistent with existing literature regarding the performance of first-generation students, justifying the need for support interventions. interestingly, legacy students had a negative average change in il score, meaning that their il scores decreased over the course of the semester. high school was not a significant correlation, though considered as a preparedness factor. only those students who attended high schools in the same state of the study were included.   table 4 t-test scores comparing first-generation status, legacy status, and high school rank first generation (y=1) legacy (y=1) hs rank (n=79) il 1 0.04 0.135 0.093 il 3 0 -0.165 -0.072 change in il -0.04 -0.224* -0.138 tech 120 grade -0.254** -0.06 -0.029 f14 term gpa -0.185* -0.087 -0.011 *statistically significant at p<.05 **statistically significant at p<.01                  conclusion   students within a single course are more diverse than the eye can see. they have complicated upbringings and have followed different paths to arrive in the college classroom. from this study, we investigated the diversity of the backgrounds of the students and aspects of their social network contributions, tangible and intangible. we learned that having concurrent or prior il instruction may compromise the integrity of the il instruction that took place in this course because students who took prior or concurrent il courses did not perform as well as those who had not. this contradicts the study done by soria et al. about the use of the library databases (2014). more research is needed to explore what happens when students take more than one research heavy or il related course, especially in their first or second years. perhaps further work can be done to understand why more il instruction did not lead to a stronger performance in this study. additionally, we learned that the impact of the urm and international experience on their overall performance cannot be overlooked in the il instructional setting. international and urm students experienced lower il performance gains. more il related research and inclusive il instructional practices should be explored to engage traditionally underserved students, like urm and international students. perhaps considerations should be made for lower-income students, in regard to the use of technology and prior exposure to il that may have been limited prior to their university arrival. that is to say, we can question whether every student has every app or cool new technology device to adequately engage with some course materials. similar considerations may apply to first-generation students understanding the nuances of navigating the academic setting, including il instruction and course and library materials. this study demonstrates that in some instances instructor assumptions may not be supported by data, and we instructors should make efforts to understand and teach the whole student with equity, not equality..   references   association of american colleges & universities. (2019). information literacy value rubric. retrieved from https://www.aacu.org/value/rubrics/information-literacy   bettinger, e. (2004). how financial aid affects persistence. in c. m. hoxby (ed.), college choices: the economics of where to go, when to go, and how to pay for it. (pp. 207-238). chicago, il: university of chicago press.   bui, k. v. t. (2002). first-generation college students at a four-year university: background characteristics, reasons for pursuing higher education, and first-year experiences. college student journal, 36(1), 3-12.   castleman, b. l., & long, b. t. (2016). looking beyond enrollment: the causal effect of need-based grants on college access, persistence, and graduation. journal of labor economics, 34(4), 1023-1073. https://doi.org/10.1086/686643   fabbi, j. l. (2015). fortifying the pipeline: a quantitative exploration of high school factors impacting the information literacy of first-year college students. college & research libraries, 76(1), 31-42. https://doi.org/10.5860/crl.76.1.31   greenwald, a. g., & krieger, l. h. (2006). implicit bias: scientific foundations. california law review, 94(4), 945-967.   gregory, a., skiba, r. j., & noguera, p. a. (2010). the achievement gap and the discipline gap: two sides of the same coin? educational researcher, 39(1), 59-68. https://doi.org/10.3102/0013189x09357621   hill, c., corbett, c., & st. rose, a. (2010). why so few? women in science, technology, engineering, and mathematics. washington, dc: american association of university women.   hubbard, l. (2005). the role of gender in academic achievement. international journal of qualitative studies in education, 18(5), 605-623. https://doi.org/10.1080/09518390500224887   huerta, j., & watt, k. m. (2015). examining the college preparation and intermediate outcomes of college success of avid graduates enrolled in universities and community colleges. american secondary education, 43(3), 20-35.   jacoby-senghor, d. s., sinclair, s., & shelton, j. n. (2016). a lesson in bias: the relationship between implicit racial bias and performance in pedagogical contexts. journal of experimental social psychology, 63, 50-55. https://doi.org/10.1016/j.jesp.2015.10.010   jaeger, p. t., subramaniam, m. m., jones, c. b., & bertot, j. c. (2011). diversity and lis education: inclusion and the age of information. journal of education for library and information science, 52(3), 166-183.   lanning, s., & mallek, j. (2017). factors influencing information literacy competency of college students. the journal of academic librarianship, 43(5), 443-450. https://doi.org/10.1016/j.acalib.2017.07.005   loden, m., & rosener, j. b. (1991). workforce america!: managing employee diversity as a vital resource. new york, ny: mcgraw-hill.   mccarron, g. p., & inkelas, k. k. (2006). the gap between educational aspirations and attainment for first-generation college students and the role of parental involvement. journal of college student development, 47(5), 534-549. https://doi.org/10.1353/csd.2006.0059   meriam library, california state university-chico. (2010). evaluating information – applying the craap test. retrieved from https://library.csuchico.edu/sites/default/files/craap-test.pdf   mestre, l. (2009). culturally responsive instruction for teacher-librarians. teacher librarian, 36(3), 8-12.   mestre, l. (2010). librarians serving diverse populations: challenges and opportunities. (acrl publications in librarianship, 62). chicago, il: association of college & research libraries.   milem, j. f., chang, m. j., & antonio, a. l. (2005). making diversity work on campus: a research-based perspective. washington, dc: association of american colleges and universities.   moss-racusin, c. a., dovidio, j. f., brescoll, v. l., graham, m. j., & handelsman, j. (2012). science faculty’s subtle gender biases favor male students. proceedings of the national academy of sciences, 109(41), 16474-16479. https://doi.org/10.1073/pnas.1211286109   osa, j. o., nyana, s. a., & ogbaa, c. a. (2006). effective cross-cultural communication to enhance reference transactions: training guidelines and tips. knowledge quest, 35(2), 22-24.   pike, g. r., & kuh, g. d. (2005). firstand second-generation college students: a comparison of their engagement and intellectual development. the journal of higher education, 76(3), 276-300. https://doi.org/10.1080/00221546.2005.11772283   roderick, m., coca, v., & nagaoka, j. (2011). potholes on the road to college: high school effects in shaping urban students’ participation in college application, four-year college enrollment, and college match. sociology of education, 84(3), 178-211. https://doi.org/10.1177/0038040711411280   severiens, s., & ten dam, g. (2012). leaving college: a gender comparison in male and female-dominated programs. research in higher education, 53(4), 453-470. https://doi.org/10.1007/s11162-011-9237-0   soria, k. m., fransen, j., & nackerud, s. (2013). library use and undergraduate student outcomes: new evidence for students' retention and academic success. portal: libraries and the academy, 13(2), 147-164. https://doi.org/10.1353/pla.2013.0010   soria, k. m., fransen, j., & nackerud, s. (2014). stacks, serials, search engines, and students' success: first-year undergraduate students' library use, academic achievement, and retention. the journal of academic librarianship, 40(1), 84-91. https://doi.org/10.1016/j.acalib.2013.12.002   staats, c. (2015). understanding implicit bias: what educators should know. american educator, 39(4), 29-44. retrieved from: https://eric.ed.gov/?id=ej1086492   stroud, d. i. (2017). a quantitative exploration of the educational paths to completion taken by first generation college students and students who have a parent with a four-year college degree (doctoral dissertation, university of missouri--kansas city). proquest dissertations and theses global, 1927182146.   appendix rubric – citation analysis, based on craap test   currency: the timeliness of the information how old is the information and is that important for your topic? does it report facts from the actual time of the event or issue? is it retrospective, providing some review or analysis of previous research? relevancy: the importance of the information for your needs ·         does the information relate to your topic or answer your question? ·         who is the intended audience? ·         is the information at an appropriate level (i.e., not too elementary)? ·         have you looked at a variety of sources before determining the appropriateness of this source? authority: the source of the information ·         who is the author/creator of the information? is it a person, group of people, an organization? ·         is he/she the original author/creator? ·         is the person qualified? what are his/her credentials? what is his/her occupation? ·         is the source sponsored or endorsed by an institution or organization? ·         is there a potential for bias? accuracy: the reliability, truthfulness, and correctness of the content ·         is the bias of the author/creator obvious? is the source trying to convince you of a point of view? ·         where does the information come from? is it supported by evidence? ·         is the publication in which the item appears published, sponsored, or endorsed by a political or other special interest group? ·         does the language or tone seem unbiased or free of emotion? ·         are there typos, spelling errors, or grammatical errors? purpose: the reason the information exists what is the intended purpose of the information: inform, teach, sale? is the information fact, opinion, propaganda? does the point of view appear objective and impartial? are there political, ideological, culture, religious, institutional leanings presented?   considerations for evaluators scale –  low (1) to high (3) currency: timeliness 1-       not acceptable: no date indicated, inappropriate, obsolete, or outdated for paper topic/assignment 2-       acceptable: should be used with sources from other dates 3-       completely appropriate: most timely for paper topic/assignment relevancy: importance of the information to the topic/assignment 1-       not at all relevant/partially relevant to topic: show to minimal understanding of the relation between the source and the paper topic/assignment; not appropriate for academic level & audience 2-       relevant to topic: information relates to the topic; shows some understanding of the relation between the source and the paper topic/assignment; fairly appropriate for academic level & audience 3-       completely relevant: information relates to the topic; clear relation between the source and the paper topic/assignment; appropriate for academic level & audience authority/accuracy: source of the information 1-       not accurate/no authority: unedited/unverifiable; no to little accountability of the author; no author identified, potentially biased 2-       some accuracy/some authority: popular or unscholarly source; demonstrates some understanding of the information 3-       authoritative/accurate: verifiable content, demonstrate thorough understanding of the information, scholarly source purpose: reason the information exists (inform, sell, persuade) 1-       no understanding/minimal understanding of the purpose of the information 2-       adequate understanding of the purpose of the information 3-       expert understanding of the purpose of the source understanding difference between fact and opinion; recognizing bias or misinformation   evidence summary   free access to point of care resource results in increased use and satisfaction by rural healthcare providers   a review of: eldredge, j. d., hall, l. j., mcelfresh, k. r., warner, t. d., stromberg, t. l., trost, j. t., & jelinek, d. a. (2016). rural providers’ access to online resources: a randomized controlled trial. journal of the medical library association, 104(1), 33-41. http://dx.doi.org/10.3163/1536-5050.104.1.005   reviewed by: lindsay alcock head, public services health sciences library memorial university of newfoundland st. john’s, newfoundland, canada email: lalcock@mun.ca   received: 1 sep. 2016     accepted: 19 oct. 2016      2016 alcock. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract    objective – to determine whether free access to the point of care (poc) resource dynamed or the electronic book collection accessmedicine was more useful to rural health care providers in answering clinical questions in terms of usage and satisfaction.   design – randomized controlled trial.   setting – rural new mexico.   subjects – twenty-eight health care providers (physicians, nurses, physician assistants, and pharmacists) with no reported access to poc resources, (specifically dynamed and accessmedicine) or electronic textbook collections prior to enrollment.   methods – study participants from a previously identified underserved rural area were selected and contacted by email. interested participants were able to enroll through a link in the email invitation and then contacted by a member of the research team. study participants were stratified by geographic region and occupation, then randomized and allocated to receive free access to either dynamed or accessmedicine for six months. usage and satisfaction were determined prior to intervention and after six months of use for the allocated resource through survey data. other survey data collected included demographic information, how long participants took on average to locate clinical information, what participants’ preferred information sources were for clinical information and patient information, willingness to pay for access to information, and usage and satisfaction of other resources including free medical websites, fee-based websites, print scientific journals, pubmed or medline, general web resources, uptodate, etc. participation was voluntary and those enrolled were able to withdraw at any time. data related to the subjects/topics searched in the intervention resources were not collected and all identifying participant information was removed following the linkage of the pre-intervention survey, the resource access data, and the post-intervention survey.   during the intervention period medical students on the research team provided technical support and training to study participants including phone and email support and in-house training videos.   preand post-intervention user satisfaction and frequency of use of 13 health resources were compared with doubly repeated anova measures, adjusted using huynh-feldt to reduce type 1 error rate. cohen’s d-statistic was used to determine the effect size difference.   main results – the authors hypothesized that clinicians would prefer and be more satisfied with the clinically oriented dynamed rather than the textbook based accessmedicine, and that these two resources would be preferred over other resources normally utilized by participants. participants in the dynamed arm reported an increase in the use of dynamed, but no significant change in the use of accessmedicine. participants in the accessmedicine arm reported an increase in use of accessmedicine, but no increase in the use of uptodate or dynamed, despite the fact that these participants did not report access to uptodate upon study enrollment. reported usage of the other 13 resources varied across time indicating a highly significant resource main effect. that is, the effect of the intervention, regardless of the study arm and the time of assessment, was statistically significant. reported use of the 13 resources was higher in the dynamed arm, though it is important to note that reported use and level of satisfaction was higher at baseline and posttest for the dynamed arm indicating a potential randomization error. an increase in satisfaction with only accessmedicine was reported in the accessmedicine arm while an increase in satisfaction with uptodate, dynamed, and accessmedicine was reported in the dynamed arm. in terms of reported use, cohen’s d indicated an increase of +1.50 for dynamed users compared to 0.82 for accessmedicine users. both arms reported an increase in the number of searches, the success of searches and satisfaction with the level of information obtained from searches. neither intervention resulted in a change from baseline related to participants’ willingness to pay for regular access to an online health information resource.   conclusion – free access to online health information resources is a potential benefit to health professionals in terms of usage and satisfaction, and participants utilized point of care tools more heavily than the textbook-based resource thus supporting the authors’ hypothesis.   commentary   one hundred and fourteen articles were reviewed to inform the study, however only six were cited in the introduction and literature review. the paper would benefit from a more extensive presentation of literature findings.   the authors noted limitations of the study particularly related to the impact of the small sample size and difficulties communicating regularly with participants. given the small sample size, external validity is low and therefore the results cannot be generalized to a larger population. this also affects the statistical power of the study. it would be interesting to see analysis based on profession, though authors indicate that the sample size was too small for subgroup analysis.   twenty-three of the enrolled 28 participants completed the study. the authors claim an “analysis of the urban-rural distribution” indicated a respective 55% and 45% rate (p. 35). given that the population was defined as rural, it is unclear why urban is identified in the analysis.   training availability for study participants was quite extensive, however it may not be indicative of standard training opportunities for this population. it may have biased results since the participants were provided with both free access to the resource as well as consistent focused and individual training. the fact that usage increased may have been influenced simply through study participation, as that brought the resource to the forefront and also provided the option for personal training and help. this is a potential bias that could be addressed in future studies with the addition of a control group.   online resources provided to participants were paid for by a grant and therefore there is no appearance of conflict of interest. it is interesting that some participants in the dynamed arm appeared to have access to both dynamed and uptodate as evidenced in the pre-intervention survey, but that no participants in the accessmedicine arm appeared to have access to either. the authors state that participants did not initially indicate that they had access to uptodate upon study enrollment. interestingly, use of uptodate was also reported to increase during the intervention period.   this is a well-designed and well-written, transparent study that provides a good grounding for future research. were the study to be replicated, it would be important to utilize a larger sample size and a comparator group with normal access to the selected resources. further, this article provides a validated methodology for subsequent research on the correlation between free access and usage of informational resources.     microsoft word es_herron.doc evidence based library and information practice 2007, 2:4  73 evidence based library and information practice       evidence summary    mismatch between the demands for tenure and those of public services is creating a  crossroads in academic librarianship      a review of:  johnson, c.m. and e.b. lindsay. “why we do what we do: exploring priorities within public  services librarianship.” portal: libraries and the academy 6.3 (2006): 347‐69.      reviewed by:   david herron  scholarly developer, university library, karolinska institutet  stockholm, sweden  e‐mail: david.herron@ki.se    lotta haglund  head of information and public relations, university library, karolinska institutet  stockholm, sweden  e‐mail: lotta.haglund@ki.se        received: 30 august 2007      accepted: 19 october 2007      © 2007 herron and haglund. this is an open access article distributed under the terms of the creative  commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to explore how public services  librarians in academic libraries perceive  their duties and experience job satisfaction  especially in relation to faculty status  (tenure track or non‐tenure).    design – a multi‐site study.    setting – academic libraries in the united  states.    subjects – academic public service  librarians.    methods – in march 2004, an online survey  containing 25 questions was sent out to  1,510 academic librarians across the united  states. the survey included demographic  and professional work‐related questions,  but mainly focused on retrieving  information about librarian feelings,  thoughts, and perceptions in relation to six  aspects of the role of the public service  librarian, namely: reference, collections,  publishing, teaching, committee, and  mailto:herron@ki.se mailto:haglund@ki.se http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:4  74 management activities. the survey data was  stored in an sql database and analyzed in  various ways to try to see if there were  correlations in the replies. in some cases,  chi‐square was used to test for statistical  significance.    main results – 328 (22%) out of 1,510  potential replies were received. 74% came  from public universities and 26% from  private institutions. 23% of the respondent  librarians had received their mls before  1980, 34% between 1980 and 1994, and 43%  1995 or later. about 50% had worked as an  academic librarian fewer than 10 years and  50% for 10 or more years. about 50% had  worked at their current institution five years  or fewer and the other 50% for six years or  more. 49% had job descriptions which  clearly indicated the relative emphasis of  each of the main areas of their job. 50% came  from tenure‐track and non‐tenure‐track  institutions alike.    the librarians were asked to reply to “most”  and “least” questions. the following lists  show the most chosen responses (in %) to  the questions.    “most” questions:    80% felt that they were most prepared for  reference activities by their library school  training.    50% felt most satisfied with doing reference  activities.    25% felt that administrators viewed  reference activities as being most important.    41% thought that reference activities had  most service impact on users.    24% found teaching activities the most  challenging aspect of their job.    49% had attended most training in the area  of reference activities.    31% thought that reference activities would  be most important for them 5 years later on  (that is, at the date of publication of this  review, 2009).    for those with a position description, 49%  had descriptions that most emphasized  reference activity.    for those in tenure‐track positions, 78 %  replied that publishing activities were most  important for tenure.    “least” questions:    28% felt that they were least prepared for  teaching activities by their library school  training.    44% felt that committee activities were the  least satisfactory part of their job.    33% felt that administrators regarded  publishing as the least important aspect of  the job.    49% thought that publishing activities  would have the least service impact on users.    46% thought that committee activities were  the least challenging.    31% had attended least training in  publishing activities.    35% thought that publishing activities  would be least important to them 5 years  later.    for those with a position description, 33%  had descriptions that least emphasized  publishing activities.    evidence based library and information practice 2007, 2:4  75 for those in tenure‐track positions, 31%  replied that management activities were  least important for tenure.    there was a correlation at an individual  level between reference activities considered  as having the most service impact on users  and giving the most job satisfaction; and  what the librarian thought was the most  important to administrators.    there was an overall correlation between  what librarians found most challenging and  the type of in‐service training they had  attended, with the notable exception of  publishing.    the area of scholarly publishing was  perceived generally as the second most  challenging area (18%) after teaching.  however, very few (2%) replied that they  had (recently) attended in‐service training in  this area.    librarians with publishing as a clear  component of their job description were  more likely to find publishing challenging  and to say that it was an important  component for their administrators.  publishing was clearly considered  important for tenure.    there were some statistically significant  differences between more‐ and less‐ experienced librarians and more‐ and less‐ recently qualified librarians. in general  (according to the authors), less experienced  librarians tended to place more emphasis on  teaching and committee activities, whereas  more experienced librarians tended to  emphasize collection development and  management.    conclusions – generally, the authors  conclude that “librarians feel that they are in  tune with their library administrators” (360).  the authors found a clear link between in‐ service training opportunities chosen by  librarians and the areas that the librarians  thought that the administration regarded as  important.    however, there seems to be discord between  the requirements for tenure (the majority  ranked publishing as most important for  tenure) and the service needs of users,  librarian job satisfaction, and the perceived  demands of administrators. the authors  conclude that academic librarianship is at a  “crossroads” where “librarians need to  decide how and if publishing activities can  be successfully balanced with other job  components that are arguably more central  to the library’s mission” (363).    commentary    generally, the study casts a broad (perhaps  too broad) net on the sea of librarian feelings,  thoughts, and perceptions concerning duties,  job satisfaction, and administrator  expectations. unfortunately, after reading  the article one ends up feeling more  confused than enlightened by the  complexity of the results. in contrast, the  mismatch between the demands for tenure  and those of public services librarianship  seems clear. the study is probably most  useful to help management understand  training requirements in connection with  strategic library development and to  enlighten administration about the  mismatch.    the authors themselves point out a number  of areas for concern in the study. firstly, a  random survey pool was not created, but  the survey was sent to public service  librarians as far as they could be identified  from the web. secondly, the idea of tenure  was unclear to many respondents. thirdly,  many participants found it difficult to  classify their duties into the six suggested  public services work categories.    evidence based library and information practice 2007, 2:4  76 it is a pity that the results concerning how  much time librarians spent on the six  activities were unavailable. for example,  50% replied that they found reference  activities most satisfying, but it is unknown  how much time was actually being spent on  such activities.    many of the tables seemed unnecessary, as  the information was adequately presented  in the text.  however, a table would have  been helpful to clearly denote the often  statistically significant responses between  more‐ and less‐experienced librarians and  more‐ and less‐recently qualified librarians.  readability may also have been improved  by using more sub‐titles in the results  section.    generally speaking, the study seems fairly  valid and some comparison is made to the  published literature. it is unclear how  representative the study is, as the total  population of academic public services  librarians in the united states is both  unclear and probably unknown. the results  of the study, especially when it comes to the  general popularity of reference activities  and unpopularity of academic publishing  activities, seems applicable to many library  environments; although other results of the  study may be culturally and institutionally  dependent.    the benefits of academic publishing ‐ such  as providing insight into the nature and  opus of academia, giving insight into the  research process, improving critical analysis  skills, increasing understanding of the needs  of academic writers, mentally preparing for  an increasingly complex work environment,  and considering long‐term library  development and the further development  of the library profession ‐ need to be more  skillfully argued. discussion regarding these  benefits would steer discussion away from  johnson and lindsay’s simplistic  “crossroads” metaphor at which, if the  “wrong turn” were made, would be the  demise of eblip. others are better skilled at  arguing these points (neal) and providing  tips (kraemer; tysick and babb).    finally (in order to finish on a high note), a  quote from the johnson and lindsay’s  article: “both newer and more experienced  librarians tend to be more satisfied with job  areas they also find challenging, which is  certainly a positive finding for the  profession” (362).      works cited    kraemer, elizabeth w. ʺkeeping up with  the journals: a library journal club at  oakland university.ʺ journal of  academic librarianship 33.1 (2007):  136‐7.    neal, james g. ʺthe research and  development imperative in the  academic library: path to the future.ʺ  portal: libraries and the academy 6.1  (2006): 1‐3.    tysick, cynthia, and nancy babb. ʺwriting  support for junior faculty librarians:  a case study.ʺ journal of academic  librarianship 32.1 (2006): 94‐100.    commentary   quantitative methods and inferential statistics: capacity and development for librarians   lise doucette assessment librarian western libraries western university london, ontario, canada email: ldoucet@uwo.ca   received: 23 jan. 2017    accepted: 26 mar. 2017       2017 doucette. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   introduction   “librarianship and statistics have always had an uneasy relationship.” so begins dilevko’s (2007, p. 209) article investigating the extent to which inferential statistics are used in journals read by academic and public librarians. uneasiness is an interesting and apt word. in informal conversations locally and at conferences, i have heard librarians acknowledge that statistics may in fact be useful, but then liberally use adjectives such as intimidating and boring. personally, i love math and statistics, perhaps to what others might consider an unreasonable or evangelical degree. i do not expect all librarians to become devoted mathophiles (though i would welcome that situation). however, i do contend that increasing our understanding of statistics individually and collectively will lead to better research, better evidence, better assessment, and better library and information practice.   in this paper, i will discuss my observations of the current relationship between academic librarianship and statistics, the utility of and case for statistics, and a number of different ways to learn more about statistics. my presentation at the 2016 c-eblip fall symposium (upon which this paper is based) was descriptive and subjective in nature, and i did not discuss the wide range of research that relates to the use of statistics by librarians. to round out my personal reflections with more academic work, i will add an analysis of that research in the next section of this paper.   i would like to introduce a few terms and definitions. quantitative methods are used to analyze phenomena that can be measured and expressed in numerical format. examples of quantitative data are type of student or researcher (nominal data), a ranking of library services by students (ordinal data), a rating of satisfaction with the library on a scale from 1-10 (interval data), and a student’s score on an assignment (ratio data). descriptive statistics describe or summarize the data and could include a simple table of numbers of each type of student or researcher, the number of students ranking each library service most highly, range of satisfaction scores, and average of students’ assignment scores. inferential statistics go much further, and they allow researchers to test hypotheses about relationships among the data and to make conclusions based on statistical evidence. questions that inferential statistics might help answer include: do students who receive information literacy sessions score higher on assignments and by how much? do graduate students who publish in open access journals have supervisors who also publish in open access journals?   current relationship between librarianship and statistics   a number of factors contribute to the uneasy relationship between librarianship and statistics. students arriving in mlis programs are generally coming from a humanities background and have little previous experience with statistics. lis researchers (stephenson, 1990; dilevko, 2000; park, 2003) have found that while the majority of mlis programs do have required research methods courses, most of those courses do not involve a strong statistical analysis component or an applied research project. statistics anxiety is also an issue and has been extensively studied in psychology, education, and statistics. a study of graduate students in an educational research methods course found that statistics anxiety is one of four key forms of anxiety experienced during the research proposal writing process (onwuegbuzie, 1997). although the author does not make any explicit connection to students in mlis programs, the background of the graduate students and the types of projects undertaken in the educational research methods course are quite similar to those of mlis students or librarians.   published research in librarianship does not use inferential statistics to a great degree, and a number of studies address differences based on type of statistics (inferential or descriptive) and role (lis faculty and practising librarians). dilevko (2007) found that there was an increase in the use of inferential statistics in his study period of 2001-2005 (14.5% of articles in the journals he studied) as compared to earlier studies from the 1970s and 1980s, where researchers found that 0.5-13.3% of articles used inferential statistics. dilevko also found that 38.5% of articles use only descriptive statistics, with 46.3% of articles using no statistics.   in 1999, hernon (then editor of the journal of academic librarianship) reflected on research in lis based on the manuscripts he receives and published literature generally. among the concerns he expressed about librarianship research are failure to select a confidence interval to guide data interpretation for inferential statistics methods, misuse or limited use of statistics, and inappropriate or incorrect use of statistical language (in particular, significance). he also quotes an earlier article (hernon, smith, & coxen, 1993) reviewing ten years of college & research libraries manuscripts that names “poor use of statistical methods” as one issue with submitted and published research.   i also see these issues in published papers. given the uneasiness regarding statistics in our discipline, i wonder if reviewers of papers are themselves knowledgeable enough about statistical methods such that they can critically review papers that use inferential statistics. within librarianship, there is neither a strong culture of critical discussion of research nor a culture of reproducibility and replicability. as an example of a journal in a discipline with such a culture, the american journal of political science requires that authors of quantitative papers submit their data as well as the code used to analyze the data. after the regular peer review process, the analysis and results of each conditionally accepted paper are then independently verified before publication. developing a strong culture of critically evaluating others’ statistical work will strengthen our research and our ability to have good research conversations.   why quantitative methods and inferential statistics?   quantitative methods have long been essential to social sciences research. research by librarians uses a variety of humanities and social sciences methodologies, and the evidence used in evidence based practice takes many forms. quantitative methods often complement qualitative methods. understanding quantitative methods allows librarians to expand their capacity to develop and answer research questions and develop evidence for informing practice, and it also allows them to read, understand, and critically evaluate research results and evidence created by others.   curiosity is key to evidence based practice and research, and learning a different way of understanding and measuring phenomena can expand your ability to think about all of the interactions in the world around you. inferential statistics are used to study differences or variance and to explore factors causing that variance. do students who take a library module on academic integrity change their citing behaviours in future essays? how do their behaviours differ from those who do not take the library module? what factors influence physical library usage? (qualitative methods might help answer more of the why questions and allow for a deeper understanding of, for example, why faculty publish in open access journals.) inferential statistics also allow for exploration of the degree of difference, the confidence that there is in fact a difference (from a mathematical not a personal perspective), the factors that might be influencing the measurement (such as interactions between different variables), and the ability to which a generalization (inference) or prediction can be made about certain research results. descriptive statistics provide a useful overview of your data but can only summarize your results.   quantitative methods also give you a new language (shared with researchers around the world) to be able to describe phenomena appropriately and to draw appropriate conclusions. byrne (2007) highlights an example of an apparent difference between two groups when looking at descriptive statistics and then shows that when an inferential statistics test is applied, there is in fact no statistically significant difference. hernon (1999) highlights librarians’ tendencies to use the statistical terms significant and not significant without applying statistical tests.   generally, rigorous quantitative methods should be both reliable (consistently reproducible within the sample you choose) and valid (measuring what you say you are measuring). these standards require a great deal of critical thinking and planning and are reliant as much on good research design as they are on appropriate statistical analysis.   learning about statistics   “we know accurately only when we know little; doubt grows with knowledge.” —goethe   the above quote captures the joy and frustration of learning for me—the more i learn about statistics, the more i realize how much i really don’t know, and the more motivated i am to keep learning. a recent article (berg and banks, 2016) highlighted librarians’ capacity to grow and evolve as researchers, advocating for a shift away from identifying and attempting to achieve specific research competencies. this resonated with me, as i do not think it would be helpful to have a list of specific statistical tests or statistical knowledge that all librarians should know. i believe librarians will explore statistics as interest, research responsibilities, and professional practice requires, and i certainly agree with berg and banks that librarians have a great capacity for learning.   it can be difficult to identify where your current knowledge of quantitate methods fits in when the landscape of the topic is difficult to identify. additionally, statistics anxiety is very real and can be a barrier to learning. in my experience, there is no shortcut for understanding quantitative methods. i know of a number of librarian researchers who are currently undertaking research projects that require quantitative methods—some have been learning how do to so on their own, and others are working with librarian or other university colleagues who already have this knowledge. (in 2014, i worked with an educational researcher at my university to help me refresh my statistics knowledge.) there are many different ways to learn depending on your existing knowledge, your available time, and what you want to learn. i’ll also point out a few examples of what i’ve done to further my knowledge.   self-directed learning   if you are interested in learning on your own, these options may work well for you:   read research articles (you are already doing this!) and deliberately include articles that use quantitative methods. read their research questions and methodologies carefully, and look up new terms. read introductory articles, such as gillian byrne’s introduction to statistics (https://ejournals.library.ualberta.ca/index.php/eblip/article/view/168). many universities (and of course wikipedia!) also have good explanations of many statistical topics. try a quick search for introduction regression and add in video if that is your preferred learning method. read (or skim) a research methods or statistics text, such as the following: o    statistics for people who (think they) hate statistics by neil salkind (includes the chapter “statistics or sadistics? it’s up to you!”) o    statistical methods for the information professional: a practical, painless approach to understanding, using, and interpreting statistics by liwen vaughan o    research design: qualitative, quantitative, and mixed methods approaches by john w. creswell   structured courses   if you are looking for more structure, some of these options may work well:   look at your university’s undergraduate statistics courses in disciplines such as psychology and sociology. talk to the professor about sitting in on lectures, auditing, or taking the course for credit. (i sat in on the lectures for western’s psyc3800: psychological statistics using computers). consider short courses on statistics or statistical software. your university’s statistics department may offer some of these or may be willing to if you express interest. (i took a full-day course in the statistical software r and a half-day review called “crash course in introductory inferential statistics.”) look at individual courses or sequences of online courses. for example, coursera has many courses and certificates in inferential statistics, research methods, and data science. consider longer, focused programs (try searching for summer institute statistics), such as these courses: o    the inter-university consortium for political and social research (icpsr) has a summer program in quantitative methods for social research, which is a fouror eight-week program that includes introductory and advanced statistics courses, computer software and math courses, and evening research lectures. (i attended the 2016 eight-week program with courses in regression, categorical data analysis, data management, and various software applications.) o    icpsr and many universities offer one-week introductory and advanced courses.   the important thing with any of this is to apply what you are learning. think of related research applications and try analysing some of your own data. if possible, try this with colleagues; the mutual support and ability to discuss and ask questions will be beneficial. there may be times when you need more substantial support. find a colleague who knows more about the topic, look at consultancy options at your university (many statistics departments offer this service), or search for a published paper that uses a similar method and contact the authors. remember that you are the person who cares the most about your data and your research; external support is great but at the same time, you want to ensure that you understand the analysis and would be able to answer questions at a conference presentation.   conclusions   we all have a limited amount of time in our professional lives, with different priorities and areas of focus. i certainly understand that increasing knowledge of quantitative methods will not be of interest to everyone. however, i would challenge you to consider the benefits of including or increasing quantitative methods in your own research and practice and to deliberately take on one small learning opportunity (personally or perhaps with colleagues).   by collectively broadening our knowledge of certain types of methodologies, we broaden the types of research questions we can conceive of and address. while there are methods to increase your own knowledge, there may also be larger systemic structures or solutions within mlis programs or for practising librarians. our profession has more exploring to do of how and why librarians do not often use inferential statistics; if this is a priority for our community, we can investigate ways to enact change.   references   american journal of political science. (n.d.). guidelines for accepted articles. in american journal of political science. retrieved 22 january 2016 from https://ajps.org/guidelines-for-accepted-articles/   berg, s. a., & banks, m. (2016). beyond competencies: naming librarians’ capacity for research. the journal of academic librarianship, 42(4), 469-471.  http://dx.doi.org/10.1016/j.acalib.2016.06.002   byrne, g. (2007). a statistical primer: understanding descriptive and inferential statistics. evidence based library and information practice, 2(1), 32-47.  http://dx.doi.org/10.18438/b8fw2h   dilevko, j. (2000). a new approach to teaching research methods courses in lis programs. journal of education for library and information science, 41(4), 307-329.  http://dx.doi.org/10.2307/40324048   dilevko, j. (2007). inferential statistics and librarianship. library & information science research, 29(2), 209-229.  http://dx.doi.org/10.1016/j.lisr.2007.04.003   hernon, p. (1999). editorial: research in library and information science—reflections on the journal literature. journal of academic librarianship, 25(4), 263-266. http://dx.doi.org/10.1016/s0099-1333(99)80025-1   hernon, p., smith, a., & croxen, m. b. (1993). publication in college & research libraries: accepted, rejected and published papers, 1980-1991. college & research libraries, 54(4), 303-321. http://dx.doi.org/10.5860/crl_54_04_303   onwuegbuzie, a. j. (1997). writing a research proposal: the role of library anxiety, statistics anxiety, and composition anxiety. library & information science research, 19(1), 5-33. http://dx.doi.org/10.1016/s0740-8188(97)90003-7   park, s. (2003). research methods as a core competency. journal of education for library and information science, 44(1), 17-25. http://dx.doi.org/10.2307/40323939   stephenson, m. s. (1990). teaching research methods in library and information studies programs. journal of education for library and information science, 31(1), 49-65. http://dx.doi.org/10.2307/40323727   news   call for submissions: “new research in collection management and development” at the 2018 american library association annual conference      2017. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     the publications committee of the collection management section of alcts is sponsoring the program “new research in collection management and development” (previously known as the annual collection management & development research forum) at the 2018 american library association annual conference held in new orleans, la from june 21-26, 2018.   this is an opportunity to present and discuss your research. both completed research and research in progress will be considered. all researchers, including collection practitioners from all types of libraries, library school faculty and students, and other interested individuals, are encouraged to submit a proposal. the committee will use a blind review process to select two projects. the selected researchers are required to present their papers in person at the forum. each researcher should plan for a 20 minute presentation, with a 10 minute open discussion following each presentation.   criteria for selection:   significance of the study for improving collection management and development practices potential for research to fill a gap in collections scholarship or to build on previous studies quality and creativity of the methodology previously published research or research accepted for publication prior to december 13, 2016, will not be accepted.   application:   the submission must consist of no more than two pages. on the first page, please list your name(s), title(s), institutional affiliation(s), and contact information (including your mailing address, telephone number, fax number, and email address). the second page should be a one-page proposal, and it should not show your name or any personal information. instead, it must include only:   the title of your project a clear statement of the research problem a description of the research methodology used results of the project, if any   the deadline for proposals is december 18, 2017.   notification of acceptance will be made by february 28, 2018.     alcts, in its bylaws, claims the right of first refusal for publication of any work emanating from an alcts body or program.   please send submissions by email to: jennifer bazeley, co-chair, cms publications committee bazelejw@miamioh.edu   evidence summary   religious studies scholarship is not widely available via open access, but some authors share their work through institutional repositories or social networking sites   a review of: avery, j. m. (2018). the open access availability of articles from highly ranked religious studies journals: a study of ten journals. theological librarianship, 11(1), 12-17. retrieved from https://theolib.atla.com/theolib/index       reviewed by: elaine sullo coordinator, information and instructional services himmelfarb health sciences library the george washington university washington, district of columbia, united states of america email: elainej@gwu.edu   received: 10 aug. 2018                                                                 accepted: 7 sept. 2018      2018 sullo. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29483     abstract   objective – to examine the current state of open access scholarship among the most highly rated religious studies journals.   design – quantitative analysis.   setting – research articles published in 2014, in the ten most highly rated religious studies journals.   subjects – 377 peer-reviewed articles.   methods – using the scimago journal & country rank, the researcher identified the top ten most cited religious studies journals from mid-2015. articles published in these journals during 2014 were evaluated. the researcher identified 377 research articles through online databases and journal websites. the researcher then used both google and google scholar to search for these articles using titles and authors. if the article was not found, other search strategies were employed, such as the use of additional search terms, limits, and quotes, as well as other search engines.   main results – open access (oa) versions were found for 132 of the 377 articles (35%), and the percent of oa articles by journal ranged from 5% to 100%. the researcher found 70 oa articles in institutional repositories (53%), 70 in academia.edu or researchgate.net (53%), 19 from organizational websites (14.4%), 13 on personal websites (9.8%), and 4 on other sites (3%). the researcher found 44 articles in more than one location (33.3%). of the 132 oa articles found, 87 (65.9%) were found by both google and google scholar, and 43 (32.6%) articles were found by either google or google scholar, but not both.      conclusion – overall, the research results reveal that finding oa content can be done via google and google scholar. while articles in religious studies journals are not typically accessible through oa, authors who tend to publish in these journals who support oa may use institutional repositories or social networking sites to make their work available.   commentary    as noted by the author, the growth rate of oa journals has far exceeded the growth of journal publishing overall, with the most web-accessible oa articles coming from the sciences, such as medicine, physics, social science, biology, chemistry and math (avery, 2018). studies have also shown that open access articles related to the humanities were few, representing only 4% of the total journal output (avery, 2018). among the published articles that explore open access growth among certain disciplines, there has been little investigation into oa for religious studies journals.   the study was evaluated using the cristal checklist for appraising a user study (n.d.). the research is focused on a specific topic: the oa availability of articles published in religious studies journals. based upon the literature review and lack of evidence available related to the author’s specific interests, the author provided a clear justification for this study. the study methodology was clearly described and a list of the top religious studies journals was included, making this research easy to replicate.   while the author used google and google scholar to find 132 oa versions of the 377 identified articles, perhaps there are other open access articles that could be discovered by using additional search engines. for a broader perspective, the author could continue this study by researching the journals in more recent years, as the articles analyzed here were published in 2014. as the oa movement continues to grow, the number of oa religious studies articles is most likely increasing with each year that passes.    a drawback of this study is that the author did not differentiate green open access from other types of open access, such as posting full-text on academic social networks or personal websites. the latter does not comply with open access requirements of most funding agencies. only gold open access (oa through publishing in oa journals or hybrid journals) and green open access (depositing in institutional or subject repositories) comply with the oa requirements (avery, 2018). further, it did not provide oa information on subject repositories.   the study findings are relevant to theology librarians and academic librarians, especially those working as liaisons to religious studies departments. the evidence suggests that while some religious studies scholarly articles are available via oa, the bulk of the literature is still behind a pay wall. furthermore, the data provides support for librarians to educate those faculty who are likely to publish in religious studies journals about oa and ways to share their scholarship, such as institutional repositories and social media. because some faculty may not be familiar with oa or have hesitancies about making their work freely accessible, librarians should also use the opportunity to educate faculty about copyright and how it relates to oa. finally, because the study found that other types of oa (e.g., academic social networks and personal websites) were used more often than green open access, librarians can also educate users on the types of oa and how to comply with oa requirements.   reference   cristal checklist for appraising a user study. (n.d.). in nettingtheevidence.pbwiki.com. retrieved august 10, 2018 from http://nettingtheevidence.pbwiki.com/f/use.doc   evidence based library and information practice evidence in practice   transparency and tiers: restructuring a publisher deal with a modified decision matrix   denise pan associate dean, collections and content university libraries university of washington seattle, washington, united states of america email: dpan@uw.edu   gabrielle wiersma head, collection development university libraries university of colorado boulder boulder, colorado, united states of america email: gabrielle.wiersma@colorado.edu   received: 16 june 2016  accepted: 19 july 2016      2017 pan and wiersma. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   setting   the university of colorado (cu) system consists of five separately administered libraries, located at four campuses and at three institutions (boulder, denver, and colorado springs). each campus has grown substantially over time, and the budgets for library materials have not kept pace with increased demand for resources or the cost of serials inflation. the cu libraries are members of several consortial groups that negotiate competitive pricing and facilitate cooperative purchasing. cu deans and directors convened the cu libraries electronic   resources team (clert), a representative group of librarians with acquisitions and collection development responsibilities, and charged them to negotiate consistent access, at the best possible system-wide pricing, for common needs. in addition to clert, most of the cu libraries also participate in the colorado alliance of research libraries (alliance), a regional consortium of thirteen academic and research libraries. like many consortia, participating member institutions benefit from sharing resources through cooperative purchasing and lending. the alliance has successfully negotiated several license agreements for member libraries including major databases and journal packages.   problem   the costs to participate in most consortial deals are typically based on publisher pricing and are generally lower than list price or the cost for a single library to participate on its own. one particular big deal journal package was called into question when eleven alliance member libraries participated in the purchase; however, the four cu libraries paid for nearly half of the costs. the cu libraries questioned why they were paying such a large proportion and whether clert could negotiate a better deal for the cu system.   the historical cost distribution for this particular package was based on print journal expenditures. overtime the burden of publisher increases were experienced disproportionately for the cu system who subscribed to nearly half of the titles in print when the package was converted to an online package. most alliance libraries shared a longstanding belief that cu boulder possessed the greatest capacity for absorbing high inflation costs for serials because it had one of the largest materials budgets. initially, this understanding possessed some truth, and cu boulder traditionally subsidized the costs of shared resources by paying a larger amount.   however, cu boulder experienced three rounds of budget cuts resulting in serials cancellations after the initial deal was negotiated and absorbing inflation became increasingly difficult. other alliance libraries felt similar economic pressures and could not absorb increased subscription costs either. the deal was in jeopardy of breaking apart. an alliance task force was convened to find out what needed to be negotiated in the new contract and to recommend a more sustainable cost distribution.   evidence   to demonstrate that the original distribution needed to be reevaluated, the task force gathered qualitative and quantitative data from participating libraries. specifically, their first survey focused on two themes: 1) satisfaction with the current deal and 2) priorities for the new contract. the second survey included questions regarding each library’s budget and willingness to renew. (both surveys are available in the appendix.)   for a meeting with representatives from participating libraries, the authors of this article prepared a presentation to establish shared understanding, facilitate discussion, and gather feedback for negotiation with the publisher. talking points included explanation of the history of the deal, overview of license agreement terms, and highlights from the survey. by anonymizing the libraries and their data presented in tables 1 through 4, this case study retains focus on methodology.   table 1 includes qualitative survey data from each library about their total materials budget, the approximate percentage of their materials budgets that is dedicated to serials or ongoing costs, and the percentage of the entire materials budget and serials budget that is spent on this journal package. this illustrated the significant impact of the costs of this journal package for each library in the consortium.   table 2 shows two different methods of calculating the collective benefit or cost avoidance from participating in the journal package. both results demonstrate that libraries are saving money by participating in this package, but savings varies by institution. even though the costs of the journal package are supposed be distributed based on historical spending, table 3 illustrates how cost distributions are no longer aligned with the number of subscribed titles at each library. for example, library k has 26.61% of the subscribed titles but pays for 33% of the costs for the package.   knowing that a cost distribution based solely on subscription costs produced inequitable results, the task force calculated costs using other variables that are commonly used to determine pricing for academic journals, such as full time enrollment (fte) and usage statistics. table 4 includes data about fte and usage at each library to calculate costs based on either of those variables compared to the current distribution. this yielded similar results for library k, which would pay less than their initial contribution if costs were based on fte or usage. however, this analysis also revealed that using either fte or usage alone could drastically impact a few of the libraries in the deal. for example, library f has a very large fte but relatively low usage, while library g is just the opposite situation with a relatively small fte but substantial usage.     table 1 impact on materials and serials budgets     table 2 cost avoidance     table 3 cost comparison   table 4 other factors for determining costs     using conditional formatting features available in microsoft excel, the authors demonstrated disparities between libraries both numerically and visually. overlaying data bars to represent the proportion of cost that each library bears quickly illustrated discrepancies. adding red, yellow, and green icons to variables, such as cost per use, indicated the relative performance of the journal package at each library (e.g., green indicated low cost per use and red indicated high cost per use). color-coded icons were also used to highlight the differences between the costs for each model.   the analysis confirmed that the original distribution gave some libraries significantly more benefits or less costs than others. it also demonstrated that relying on a single factor to determine cost would result in similarly inequitable results. sharing the results with all of the participating libraries produced mutual understanding regarding the collective unsustainability of the package and created the impetus to redefine the allocation model.     implementation   in business management literature, a classical decision matrix has “options on one axis and criteria on the other.” see an example in table 5. when used as an evaluation, the decision matrix can help leaders make better strategic decisions by extending the “decision frame beyond the obvious options and criteria” (enders, könig, and barsoux, 2016, p. 63).   the authors proposed and the task force agreed to redistribute costs among academic libraries based on multiple criteria including fte, usage, and materials budget. in their modified matrix the decision criteria are presented on one axis and academic libraries on the other. they determined a percentage of the total (or weight) for each criteria and used the library’s data to calculate a percentage of the total for each criteria. the costs are distributed by these percentages, and the sum determines a library’s share of the total. once they identified comparison variables for a weighted decision matrix, they calculated what a library should pay (table 6). to ensure that all libraries remained in the package, they tiered cost distributions into three levels of more gradual price increases based on what libraries could reasonably pay (table 7).     table 5 classic decision matrix     table 6 multi-factor cost distribution using a modified decision matrix     table 7 tiered cost distribution     outcome   presenting the evidence in a modified decision matrix expanded the framework for decision making. this process encourages visualization of options, criteria, and trade-offs, which can help leaders clarify thinking, engage colleagues, and promote buy-in from the larger organization (enders, könig, and barsoux, 2016, p. 68). in the colorado case study, the modified decision matrix encouraged libraries to recognize that the historical cost distribution model was no longer accurate and over time had resulted in some libraries paying a disproportionately high portion of cost. the alliance libraries agreed to a new cost distribution model, and the consortia signed a multi-year journal package with the publisher. for the cu system, the new cost distribution resulted in cost savings for boulder but increases for auraria and colorado springs. however, the tiered approach kept the costs affordable for all of the cu libraries, and the net result made the shared purchase viable for the near future.   reflection   consortia activities are most beneficial when costs and benefits are understood and shared among all member libraries. the process of surveying participating libraries, reviewing criteria and variables, and developing cost distributions should be conducted on a routine basis. the alliance plans to update the variables in the decision matrix on a regular basis so that the costs will be transparent and reflect changes in fte, usage, or budget for each library.   conclusion   to ensure better decision making and timely implementation, strategic business leaders utilize evaluation tools, such as a decision matrices, to explore options, make choices, and communicate decisions to stakeholders. in this case study, the authors developed a modified decision matrix with multiple weighted criteria to redistribute the costs of a purchase that is shared among consortia of academic libraries. this methodology could be applied to other scenarios when complex problems require systematic consideration of multiple criteria and various stakeholders.   references   colorado alliance of research libraries. (2016). about. retrieved 19 march 2017 from https://www.coalliance.org/about   enders, a., könig, a., & barsoux, j. (2016). stop jumping to solutions! mit sloan management review, 57(4), 63-70. retrieved from http://sloanreview.mit.edu/article/stop-jumping-to-solutions/     appendix survey questions   please note the name of the publisher of the big deal journal package has been removed from the survey. minor edits have been made to the surveys for clarity because it appears here in a different format from the original. renewal survey, part 1   section 1 please provide feedback about the existing package and license agreement.   1.       which alliance library do you represent?   2.       how satisfied are you with the existing journal package? on a scale between 1 (dissatisfied) and 5 (very satisfied) check one.   3.       how do the following factors impact your satisfaction with the existing package? check one type of impact per factor: negatively impact | no impact | positively impact a.       amount of content b.       faculty feedback c.        student feedback d.       librarian feedback e.       usability of the platform f.        access issues g.       usage statistics h.       license terms i.         cost j.         are there any other factors that impact your satisfaction that were not included above?   4.       how likely are you to renew if a new contract was negotiated with the same or similar terms? on a scale between 1 (very unlikely) and 5 (definitely) check one for each scenario. a.       3 year deal; set at set percent increase b.       price based on historical title list and transfer titles c.        access to unsubscribed titles for an additional cost   section 2 please provide feedback to help the alliance negotiate the 2015-2020 contract.   5.       what are some of your priorities for a new contract? check one priority level per contract term: not a priority | low priority | medium priority | high priority a.       reduce our costs b.       cap annual increases c.        increase amount of content d.       create a new title list e.       break up the big deal and subscribe to individual titles f.        create uniform access across alliance libraries g.       secure perpetual rights/post-cancellation access h.       add license terms for ada compliance   6.       how do the following factors influence your decision to participate in the alliance package? check one level of importance per factor: not important | somewhat important | very important a.       current budget b.       projected/expected annual budget increases c.        impact of cancellation on ill d.       impact of cancellation on collection size   7.       what factors should be considered to determine our costs for the new alliance package? the following factors have traditionally been used to determine prices for other journal packages or databases. check all that apply to your library. a.       fte b.       carnegie classification c.        number of faculty (total or within certain departments) d.       usage e.       historical spend for titles f.        list price g.       other: please provide details.   8.       are there other factors to determine price that were not included above?   9.       would you be interested in exploring different acquisition models for this content during the negotiations? check one level of interest per acquisition model: not interested | somewhat interested | very interested a.       evidence-based model b.       token-based model c.        pay-per-view model d.       database subscription model (no title lists, no perpetual access) e.       one-time purchase model f.        other model, to be determined   10.    are there other acquisition models that should be explored but were not mentioned above?   11.    are you interested in adding/integrating other publisher products into this renewal? check one level of interest per product: not interested | somewhat interested | very interested a.       product a b.       product b c.        product c   12.    are there other publisher products that should be considered during this negotiation?   13.    do you have questions for the publisher?   14.    do you have any comments or questions for the negotiating team?   renewal survey, part 2   1.       which alliance library do you represent?   2.       what is your fy2014-15 budgeted appropriation for library materials (excluding gifts, grants, external funding)?   3.       what percent of your library materials budget is typically allocated to serials/ongoing costs? a.       30-40% b.       40-50% c.        50-60% d.       60-70% e.       70-80% f.        80-90% g.       90-100% h.       other: please provide details.   4.       if we negotiated another 3-year deal with capped annual increases, at what percentage increase would you renew? check all that apply a.       below 3.0% b.       3.5% c.        4.0% d.       4.5% e.       5.0% f.        other: please provide details.       article   exploring the complexity of student learning outcome assessment practices across multiple libraries   donna harp ziegenfuss interim head of scholarship and education services university of utah salt lake city, utah, united states of america email: donna.ziegenfuss@utah.edu   stephen borrelli head of library assessment penn state university libraries university park, pennsylvania, united states of america email: sborrelli@psu.edu   received: 15 feb. 2016   accepted: 26 apr. 2016       2016 ziegenfuss and borrelli. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objectives – the purpose of this collaborative qualitative research project, initiated by the greater western library alliance (gwla), was to explore how librarians were involved in the designing, implementing, assessing, and disseminating student learning outcomes (slos) in gwla member academic libraries. the original objective of the research was to identify library evaluation/assessment practices at the different libraries to share and discuss by consortia members at a gwla-sponsored student learning assessment symposium in 2013. however, findings raised new questions and areas to explore beyond student learning assessment, and additional research was continued by two of the gwla collaborators after the symposium. the purpose of this second phase of research was to explore the intersection of library and institutional contexts and academic library assessment practices.   methods – this qualitative research study involved a survey of librarians at 23 gwla member libraries, about student learning assessment practices at their institutions. twenty follow-up interviews were also conducted to further describe and detail the assessment practices identified in the survey. librarians with expertise in library instruction, assessment, and evaluation, either volunteered or were designated by their dean or director, to respond to the survey and participate in the interviews. interview data were analyzed by seven librarians, across six different gwla libraries, using constant comparison methods (strauss & corbin, 2014). emerging themes were used to plan a gwla member symposium. based on unexpected findings, after the symposium, two gwla researchers continued the analysis using a grounded theory methodology to re-examine the data and uncover categorical relationships and conceptual coding, and to explore data alignment to theoretical possibilities.   results – seventeen categories and five themes emerged from the interview data and were used to create a 3-part framework for describing and explaining library slo assessment practices. the themes were used to plan the gwla assessment symposium. through additional qualitative grounded theory data analysis, researchers also identified a core variable, and data were re-evaluated to verify an alignment to engeström’s activity and expansion theories (engeström, 2001, 2004).   conclusions – the findings of this multi-phased qualitative study discovered how contextual, structural, and organizational factors can influence how libraries interact and communicate with college departments, and the larger institution about student learning outcomes and assessment. viewing library and campus interaction through the activity theory lens can demonstrate how particular factors might influence library collaboration and interaction on campuses. institutional contexts and cultures, campus-wide academic priorities, leadership at the library level, and changing roles of librarians were all themes that emerged from this study that are important factors to consider when planning the design, implementation, assessment and dissemination of library slos.   introduction   the purpose of this research was to uncover the various types of student learning outcomes and assessment practices at gwla member academic libraries. themes identified from research data were then used to organize and plan a symposium focused on student learning outcomes assessment. questions that emerged from the survey and interview findings also prompted a need for additional research to uncover relationships between institutional and library culture with assessment practices. therefore, this paper will present the methods, results and findings from phase 1 of the research, the pre-symposium survey and interview data analysis, as well as the additional phase 2 post-symposium grounded theory analysis. finally, findings from the grounded theory analysis will be used to present three different institutional profile vignettes as examples of how activity theory might be utilized to rethink library-institutional interactions.   a charge from the greater western library alliance (gwla) formed the student learning outcomes (slo) taskforce committee to investigate learning assessment practices at gwla member libraries and how academic libraries are impacting student learning outcomes assessment. a qualitative research approach was selected for this study because it was the best method for gathering rich and descriptive information about student learning outcome assessment practices. members of the taskforce worked on subcommittees to create the survey, implement the survey, design the interview protocol, conduct the interviews and analyze the interview data. the taskforce membership included library representatives from eleven institutions: arizona state university, brigham young university, texas tech university, university of arizona, university of colorado-boulder, university of houston, university of illinois chicago, university of kansas, university of missouri, university of nevada las vegas, and university of utah.   this paper outlines the processes and findings for this collaborative qualitative research project. initially, representatives from 23 institutions were surveyed. from the survey, representatives from 20 gwla academic libraries volunteered to be interviewed about the assessment practices in their library, as well as, campus assessment practices at the institution and department/college levels. analysis of interviews resulted in themes and related categories and in the development of a conceptual framework. this framework was used to design a three-day gwla student learning assessment symposium (gwla, 2013). going beyond that analysis, two librarians continued to re-examine data using a more rigorous grounded theory process to uncover a core variable and generate theory that can be used to guide library reflection and analysis at any institution.   literature review   the library value movement posits that in the current environment, connecting library services with institutional priorities to demonstrate library impacts results in increasing library relevancy (kaufman & watstein, 2008; menchaca, 2014; oakleaf, 2010; pritchard, 1996). in the seminal work on library value, multiple approaches for academic libraries to develop institutional relevance are identified. developing and assessing student learning outcomes is just one option identified for demonstrating library value in relation to student learning (oakleaf, 2010; hiller, kyrillidou & self, 2006; pan, ignacio, ferrer-vinent & bruehl, 2014). published evidence of library impact on student learning has been historically disconnected from institutional outcomes, and generally focuses on individual librarian/faculty collaboration, rather than programmatic approaches (oakleaf, 2011). hufford (2013) contends, in a 2005-2011 review of the library assessment literature, that, while traditional library inputs and output measurements remain valuable, libraries are increasingly focusing on institutional priorities and assessing student-learning outcomes programmatically, to uncover institutional impacts.   as the library value literature indicates, it is also important to investigate higher education change and organizational development issues more broadly (barth, 2013; kezar, 2009). economic, social, technological, and cultural issues are currently emerging and driving change in new directions on many campuses (altbach, gumport & berdahl, 2011; kezar & eckel, 2002; kyrillidou, 2005). there are calls for transformational change (eckel & kezar, 2003), encouragement for ‘disruptive’ education tools (christensen & eyring, 2011), and a reinvention of the college experience (hu, scheuch, schwartz, gayles, & li, 2008). in addition, findings from phase 1 of this study identify a need for investigating how higher education contextual and organizational structures are influencing how libraries are changing and functioning on campuses. one theory, activity theory (engeström, 2001, 2004; engeström, miettinen, & punamäki, 1999), aligns well with the emerging library literature and the higher education change literature, as well as the results from this study. this activity theory framework, grounded in the seminal constructivist theory of vygotsky (roth & lee, 2007; vygotsky, 1980) has been utilized in many studies to theorize and describe a variety of work and learning environments or systems through the structure of goals and objects that include activity towards an object, tools, community structures, and rules (figure 1). a second theory, expansive learning (engeström, 2001, 2004), that is an extension of activity theory, focuses on the interactions and change between multiple activity systems. as libraries transform and become more embedded in the institutional structure and culture, awareness of other campus activity systems and interactions will only become more critical to demonstrating library value and success.     figure 1 activity theory model diagram (bury, 2012).     phase 1 pre-symposium: survey and interview methodology, results and findings   phase 1: methods   this qualitative study was conducted in two phases. in the first phase, gwla librarians collaborated to conduct a survey and interviews to identify the slo assessment practices of gwla academic librarians (see appendix a and b for the survey questions and interview script). the survey was designed and distributed electronically. a taskforce sub-committee of librarians worked to design, implement, and evaluate the survey responses. another sub-committee of librarians designed the interview protocol and conducted the interviews. audio interview files for 20 interviews were transcribed by an external transcription service. the follow-up interviews, which further described and informed the survey responses where then coded and analyzed by a third sub-committee of seven librarians, using a grounded theory approach (strauss & corbin, 2014). librarians worked in pairs to triangulate coding results. all of this research was conducted with a purpose of planning for a gwla assessment symposium and prior to the 2013 symposium. in the second phase of the research, 2 of the original 7 librarians who helped to code and analyze the 20 interviews continued to analyze data and took the analysis to the next step of theory generation. this data analysis took place after the symposium.   an electronic survey was distributed to the gwla membership, and 23 gwla libraries (72%) responded to the survey. survey respondents were either selected by their library dean or director, or volunteered to take the survey because they were a librarian with assessment or instruction expertise and aware of student learning outcomes assessment practices on their campuses. the selection and position of the interviewee(s) varied based on the organizational structure of each of the academic libraries and that decision was left up to the survey respondents on who should represent the library and participate in the survey and interviews. however, all interviewees were instructed to respond to questions on behalf of their library and not on their individual projects. the respondents answered a series of questions about the presence and assessment of slos on their campuses. the purpose of the survey was to uncover which libraries had established slos and were using information literacy (il) slos, and at what levels of the institution the slos existed or were being assessed. librarians were also asked if they were assessing their faculty/librarian collaborations.   from the 23 survey respondents, 20 people either volunteered for a follow-up interview, or appointed someone else as the designated library representative. librarians were invited to participate based on their role in the library, either with assessment or library instruction, and also for their ability to discuss the status of assessment at other levels of their institution. in 65% of the cases, the same person who responded to the survey also consented to the interview. in 35% of the cases, several people participated in the interview to speak to multiple aspects of the instruction and assessment topics. for example, an instruction librarian and an assessment librarian were interviewed together in some cases at institutions. interview participants were encouraged to extend invitations to other assessment and instruction librarians or staff to participate in the interview if one person might not be able to answer all of the questions.   a plan was also established for conducting interviews and collaborative qualitative analysis of the interviews. follow-up interviews began in spring 2012 and were completed in december 2012. data analysis was ongoing during the interview process and completed in spring 2013. an additional bibliography was also compiled on published reports of assessment evidence, practices, and innovations, which were gleaned from topics raised in the 20 interview transcripts. this data was used to recruit presenters for the gwla november 2013 librarians partnering for student learning symposium that was held at the university of nevada, las vegas. throughout the spring, summer and winter of 2012 interviews of the follow-up contacts were conducted by telephone via interview teams. from these interviews, written summaries were created, interviews were audio-recorded, and the audio files were transcribed. the transcribed transcripts were then submitted to the qualitative analysis team, where pairs of researchers analyzed and triangulated the interview data and compiled the findings. the transcripts were analyzed using grounded theory qualitative methodologies using open and axial coding strategies (strauss & corbin, 2014). for the open coding analysis, the researchers individually read the transcripts and created preliminary codes to describe the text. then the research pairs compared their coding and moved into the axial coding stage where they looked for relationships and connections between the codes and created larger categories. to begin the interview analysis, pairs of researchers coded four interview transcripts, one pair for each interview. each researcher coded his/her interview independently first and then member-checked coding with his/her partner who had also coded the same interview. each pair submitted a single set of coding. the coding from the four interviews was then compiled and analyzed for themes. since not all institutions had a qualitative analysis package like nvivo or atlas.ti to conduct qualitative analysis, the research team used microsoft excel to conduct the qualitative data analysis and to compile the results of the survey and interviews into themes and topics for further study. tutorials on using excel to do qualitative analysis were provided to researchers. from the first set of four interviews, a preliminary set of 17 categories was uncovered and used to define the codebook for the rest of the research process. the 17 categories were consolidated and re-evaluated to create a set of 5 major themes. a framework was developed from the themes and used to plan the gwla student learning outcome symposium in 2013 (gwla, 2013).   phase 1: survey results   the survey results demonstrated that the presence and assessment of information literacy slos at gwla institutions occurs at a variety of levels. fifty-seven percent of the 23 institutions that responded to the survey reported that they have campus-level slos, but only 26% reported that those campus-level slos were assessed. a similar disparity was identified at the college/department level between the presence and assessment of slos with 61% reporting the presence of slos but only 26% reporting assessment of the slos. however, at the library level, 65% of institutions reported the presence of slos, and 48% reported that the slos were assessed. in addition, when institutions were asked if librarian/classroom faculty interactions were assessed, 61% (14 of the 23 institutions) reported that they do assess these types of collaborations and 35% (8 institutions) reported they do not assess these collaborations, one institution reported that they do not know if these types of collaborations were assessed. the gap between what institutions reported about the presence of slos, and the actual assessment of slos, drove the question formation for follow up interviews with a purpose of trying to identify how slos are assessed.   phase 1: interview results   audio-recorded interviews were conducted and transcribed. analysis of the first 4 interviews resulted in the identification of 484 codes, organized into 71 categories. these categories were analyzed using a recursive process of recoding, collapsing and combining codes, and renaming of categories until the remaining categories were deemed to be unique. from this process, 17 unique core categories were identified and defined. the 17 original categories were: 1) strategies for planning, implementing & integrating slos; 2) roles/responsibilities for assessment of slos; 3) collaboration; 4) communication issues; 5) tools-instruments-resources for slos; 6) accountability & reporting of slos; 7) curriculum & instruction; 8) departmental relationships; 9) culture and priorities issues; 10) structures, policies, and administration; 11) professional development; 12) challenges; 13) leadership; 14) change related; 15) opportunities; 16) general (slo catch-all); and 17) information literacy topics. these categories were then used to code the remaining interviews. no new categories emerged from the remaining 16 interviews indicating data saturation.   during the second round of coding, the 17 categories of codes were collapsed and refined into 5 main themes. the five themes were: 1) curriculum and instruction; 2) strategies for planning, implementing and integrating slos, 3) collaboration and communications issues, 4) roles/responsibilities for assessment & slos; and 5) slos structures, policies, and administration. these five themes were returned to the researchers for confirmation; each researcher taking one or two themes, to verify that no additional themes had emerged. using the five themes and code frequency data, a conceptual framework was constructed to relate and explain the themes. for triangulation and confirmation purposes, another gwla taskforce member, who had not been involved in the coding process, reviewed and refined the framework. the resulting framework (figure 2) consists of three main parts: deconstructing the process of slo assessment, building partnerships, and embracing change and opportunities. since the main focus of the interviews was to uncover slo practices and processes across gwla institutions it is not surprising that 55% of the coding resides in the first column of the matrix that includes two of the five themes and coding about slo design, implementation, assessment and dissemination. the framework structure, across the rows, aligns the slo design and assessment process to other cultural, contextual, and organizational institutional factors. the codes and themes in the second and third columns, although smaller in number, were consistently present and related back to the main slo assessment theme.     figure 2 conceptual framework for designing, implementing, assessing, and disseminating slos.     phase 1: survey and interview discussion   the conceptual framework for designing, implementing, assessing, and disseminating slos (figure 2), developed from the consolidated gwla data in the first phase of this study, can provide guidance for individual libraries as they work to evaluate their own contributions to campus efforts related to articulating, embedding, and assessing of slos. this conceptual framework emphasizes the importance of building relationships, embracing change and opportunities, and considering contextual and organizational structures when planning or sustaining successful slo design and implementation projects. these findings are in line with current library research that focuses specifically on developing strategies for building library-faculty collaboration and trust and consideration for the complex set of contextual factors that can vary widely across institutions (phelps & campbell, 2012; oakleaf, 2010). these factors may become critical or pivotal barriers or possible opportunities related to successful slo implementation and dissemination. findings from this study indicate there is no one magic bullet method for integration of library il slo or successful slo assessment implementation. the themes of communication, collaborations/partnerships, embracing opportunities, addressing challenges, and the rethinking of roles and responsibilities were evident across all institutions that participated in the study. however, the variation in contextual/cultural factors, organizational structures, internal and external drivers, as well as, leadership and levels of librarian proactivity also appear to result in very different practices and outcomes. one librarian stated, “i think the library’s leadership needs to be more proactive in promoting the library’s role as an information literacy agency on campus.” therefore, the conceptual framework can be used as a roadmap to establish a process for developing library awareness, and establishing priorities for libraries to take leadership roles. findings from this study suggest that institutions reflect on their own institutional context and therefore tackle their unique complex situation in their own way. best practices or assessment strategies successful at one institution may not always be easily replicated at other institutions. in addition, since each institution and library may be at a very different place related to the articulation and implementation of slos, this framework may provide a more flexible and holistic option for reflection and strategic decision-making than a step-by-step assessment implementation procedure or checklist approach to assessing slos.   data from the study also indicate that the planning process for campus-wide slos is often a top-down or administrative initiative, resulting from accreditation concerns, or an institutional focus on evidence based decision-making or assessment. one example of how a university librarian described the assessment support structure at the administrative level drove slo assessment is,   “our institution is very, very driven by the evidence-base learning outcomes of students. we don’t just call them student-learning outcomes. the office of the provost for the past five years has made it very clear that every school has to have evidence based learning outcome. and that of course does include information literacy at the departmental level.  so we are very much embedded in this kind of approach.”   it was also noted during analysis of this study data, at both the campus and library levels, considerable efforts are being made to standardize assessment efforts. libraries and institutions are investing in the assessment effort, creating assessment and planning librarians, or instruction and assessment positions to focus efforts and provide accountability. one librarian discussed reactions to accreditation needs and remarked, “one of the things that’s happening in response to our last accreditation visit is that we have developed this office of assessment of teaching and learning. and they are responsible for conducting undergraduate assessment.” many member institutions indicate that they are in the process of learning to assess. instruction librarians are applying many approaches and instruments in their assessments, using qualitative and quantitative methods often modeled after national tools like the association of american colleges & universities (aacu) valid assessment of learning in undergraduate education (value) rubrics (aacu, n.d.); tool for real-time assessment of information literacy skills (trails) (kent state university libraries, 2016); rubric assessment of information literacy skills (rails) (oakleaf, n.d.); and standardized assessment of information literacy skills (sails) (kent state university libraries, 2016).   phase 2: grounded theory methodology, results, and findings   phase 2: methods   in phase 2 of the study, two of the original gwla researchers continued the search for a core variable and theoretical grounding, and continued to recode and reevaluate data. the purpose of this phase of the research was to go beyond description and uncover a theory or conceptual framework that would help institutions analyze their own institutional context so they could better integrate the academic library into their own institutional and contextual processes. the 17 categories and 5 themes from phase 1 of the research created the foundation for further analysis. the researchers returned to the literature to uncover theoretical connections by recoding and categorizing through a process outlined by glaser & holton, (2004). the research process included numerous coding iterations, constant comparative analysis as well as member checking and collaborative discussions and memoing about the data. after many iterations of coding and recoding, the data and categories from phase 1 of the research were used to generate the theoretical construct discussed in this paper.      figure 3 a library level activity system model (diagram created by spencer, 2014).     phase 2: grounded theory results   the purpose of this additional analysis was to take the study to the stage of theory identification. interview data were recoded and reanalyzed with a more conceptual focus examining the three different institution levels of campus, department/college, and library. recoding resulted in a more detailed and conceptual description of the slo contextual factors and uncovered how opportunities and challenges of the design, implementation and distribution of slos are mediated at the different institutions. the six conceptual themes that emerged from this additional coding process were building awareness, power and ownership, embedded in or on the fringe of culture, opportunity advantages, organizational structure, and strategic leveraging. taking a grounded theory approach and revisiting the literature after revised conceptual coding provided a broader lens of perspective and yielded an identification of activity theory as a possible theoretical frame for understanding slo development and implementation as well as campus interactions. (chaiklin, hedegaard, jensen & aarhus, 1999; engeström, miettinen, & punamäki, 1999).   after theory identification, the data were recoded once again to confirm alignment of the data to the main components of the activity theory model which consist of 4 components: 1) rules and policies, 2) community, 3) division of labor (roles and responsibilities), and 4) mediating tools and artifacts. both researchers recoded data again using these four components as codes and all data could be aligned directly to these activity theory components. this confirmed the suitability of this activity systems theory as a lens for understanding the research data. figure 3 demonstrates the alignment of previous codes, categories, and themes from the conceptual framework analysis of phase 1, to the library activity theory model of the phase 2 research.   further literature searching exposed a related theory, expansive learning (engeström & sannino, 2010) which is called third-generation activity theory, that offer explanations for contextual factor interaction inherent in multiple systems. figure 4 demonstrates the alignment of the phase 2 conceptual themes analysis across the three different levels of an institution and at the intersection of the three different activity systems. the interaction of all three systems or what engestrom calls “co-configuration” (engeström, 2004), focuses on the theme of strategic leveraging of opportunities, challenges and needs.     figure 4 intersections of three institutional activity systems.     phase 2: grounded theory discussion   as related to a finding from phase 2 of this study, the researchers contend that the activity of planning, designing, assessing, and disseminating slos is mediated through tools, processes, rules, and community interactions. re-examination of the data using the activity theory model focused our analysis more on the contextual factors influencing slos and less on the actual slo assessment. the highest occurrence of coding related to the rules and policies component of the activity theory model, which was located in the institutional level data. although this may not be a surprising finding, it is important to be aware of this when trying to work within an institutional context. in this study, library/faculty interactions were influenced by socio-cultural factors and library/campus level interactions were driven by organizational structure and policy, as well as, by change. themes such as accreditation as driver, change, leadership, organizational structure, institutional culture, and getting a place at the decision making table can now be connected to the institutional level structures and culture. as librarians, it appears to be critical to get plugged into the institutional culture. one librarian expressed concern about this and stated,    … there is also the idea that on this campus, and i think probably we’re not unique at all, people … still think of libraries as the place that has the stuff and … they don’t necessarily look [at] the librarians as partners in their teaching … and we don't have faculty status here and so we’re not at the table.    librarians should consider taking a more proactive approach for inclusion in reform and change initiatives, as well as employing routine operating procedures at their institution, which may vary depending on the culture, leadership and engagement of administrative units with assessment. one institution discussed the challenges, but also the opportunities, when librarians take on new roles:   i would say the biggest challenge that we've had is the fact that we have kind of taken on … being experts on course design and so we have had pockets of faculties who sort of questioned that or why our librarians doing this, they don't teach.  so, it has been a big kind of image remake and marketing opportunity for us.   this concept of librarians as change agents is an emerging theme in the library (pham & tanner, 2014; travis, 2008).   three institutional vignettes   of the 20 institutions analyzed, institutional coding profiles varied which was evident in the frequency of coding and categories. by exploring the data using the components of activity theory, different priorities, foci, and initiatives at different institutions were uncovered. three different institutional profile vignettes are presented below as examples to demonstrate the alignment of interview coding, categories and themes at three different levels of the institution (campus or institutional, department or college, and library) coded at the four different components of activity theory.   vignette 1: the bigger picture   the first vignette is an institution that stood out with exceptionally high coding frequency numbers at the institutional and library levels and very low levels at the college/department level (table 1). this large public research institution, reported slos at all three levels of the institution. assessment is reportedly driven by accreditation and there is a central assessment office, which may account for the larger number of codes around the structure and process of community at the institutional level. library slos are aligned to the institutional slos and there are assessment representatives in each unit. slos developed out of the faculty senate with no library involvement but there is evidence in the library of assessment professional development. at the library level the high coding frequency for division of labor (roles and responsibilities) is attributed to instances of discussion about the roles librarians play in faculty collaboration and assessment of slos by designing assignments, slos, collecting data, and disseminating slos. at the college/department level however, there is a very low number of codes and the discussion in the interview was only focused on the variation of assessment and culture across departments.     table 1 the bigger picture university focused profile     vignette 2: the community in the library   the second institution, also a large public university, has the largest concentration of code frequencies at the library level, and specifically in the community component of the library level (table 2). this is a decentralized institution, and a centralized slos assessment culture is a challenge, but locally in the library there is a strong culture of assessment. historically it appears this institution has focused on library as place and collections, and less on measuring student learning across the different levels of the institution.     table 2 the library community profile     vignette 3: a lot of teaching responsibilities   in this last vignette the high coding frequencies at the college/department and library level are attributed to a high percentage of the discussion focused on discussing specific teaching projects in departments by librarians in the interview. this research institution is in the process of moving to a liaison model approach with faculty and therefore this may account for the higher department/college coding frequency numbers (table 3). the interviewee reported that there is a good balance at this institution between research and teaching but reports challenges of time constraints that dictate preparation issues. there is more of a focus on curriculum development than assessment.     table 3 the library focused on teaching university profile     as you can see from the brief vignettes of these three different gwla institutional libraries, each institution has slightly different priorities, and in the interviews discussion was focused on different issues. unique situational factors and cultures can impact institutions differently. findings from this study emphasize the importance of developing awareness of your institutional culture, organizational structure, and academic priorities. by being aware of the environment and also tuned into emerging priorities and initiatives, librarians will have opportunities to be proactive and step-up and engage with their academic community. libraries are positioned to increase their organizational value by drawing on internal teaching expertise, developing new skill sets in instructional design or other areas, and taking a proactive stance where leadership or expertise is desired.   conclusions and recommendations   the impact that a unique institutional culture and context has on the ability of an organization to come together around designing, assessing and disseminating slos was the most interesting finding in our data at both phases of the research study. some institutional efforts are bolstered through an institutional commitment to evidence-based decision making while other institutions reported that a decentralized organization, lacking a culture of assessment, or lack of leadership could deter success in developing and implementing slos. other related limitations identified were academic freedom issues, fear of negative impact on the tenure and promotion, and the location and status of the library staff within the institutional structure. many libraries reported that they are actively building a culture of assessment and creating positions to support slo efforts. additionally, information from the interviews suggests that planning the process for slos is often a top down initiative, resulting from accreditation drivers, or a presence or lack of presence of an institutional focus on evidence or assessment. this is an area that might merit further exploration and research in conjunction with the emerging economic and political issues in higher education, which impact the ability to staff and fund assessment efforts.   another interesting aspect of the data analysis evolution centered on differentiating between collaboration and campus-wide partnerships. as the analysis progressed, the researchers saw collaborations as more related to individuals working together, whereas partnerships focused more on developing alliances or more long-term working partnerships with other campus units. these are two very different things. it appears from the data that partnerships could have a broader and more powerful impact on the work done in the library when integrated with the opportunities for librarians interacting at different campus levels, as compared to collaborations, which, focused on one-on-one interactions with faculty. therefore, one recommendation for future research is to focus on studying how the presence of partnerships, as compared to collaborations, specifically might impact the process of designing, implementing, assessing and disseminating slos at various levels of the institution.   data indicated that curriculum development might be an area fruitful for more study. as one interviewee in this study noted, “often the process of curriculum development does not include incorporating assessment. instead, assessment of learning is considered something to be addressed separately, after the curriculum is developed.” this practice seems to run counter to the current practice of ‘backward design’ (fink, 2013), which was a successful strategy used by one of the gwla partners, and includes the sequential steps of outcomes, assessment and then curriculum development. additional research in this curriculum design area could shed light on how libraries are integrating assessment into curriculum level.   for the second phase of this study that linked the study data to engestrom’s activity theory, there are many implications and recommendations for library practice. analyzing the library landscape and how the library interacts, interfaces and embeds within both the campus and departmental level could benefit from strategic planning. this study data indicates the importance of considering the broader aspects of interaction and partnership when designing, implementing and disseminating il slos. awareness of the larger institution culture and what initiatives are ‘hot’ and being funded will provide opportunities for being proactive and engaging with the campus community. awareness of new initiatives might also provide opportunities to extend library roles or take on new roles.    even though each gwla institution reported on a variety of methods, strategies, and organizational approaches based on their unique contextual and cultural structures for designing, implementing, assessing, and dissemination of slos, there are however, commonalities in the motivators and drivers for assessment across institutions, such as accreditation reviews, program redesigns, and a desire to move to a more evidence-based driven culture. institutional contexts and cultures, campus academic priorities and initiatives, leadership at both the institutional and library levels, and changing roles of librarians; themes that emerged from this study are important factors to consider when planning the design, implementation, assessment and dissemination of il slos.   limitations of the study   as with any research project, there are process and methodology limitations in this study. not all gwla member institutions participated in the study; this was a purposive sample of volunteers interested in slos. therefore, since participants self-selected, participation may not be a true representation of the consortium. in addition, although interviewees were librarians selected to or volunteered to represent each institution, and selected by the role they played at their institution, there may be other people at their institution that could speak better to the institutional view of assessing slos. therefore, the information they provided may be limited by their own personal library role and experience or limited by their personal knowledge about the larger institution.   the data analysis in this study was done in excel due to the lack of access to expensive qualitative analysis software by the participating researchers. using qualitative software like nvivo or atlas.ti would have enabled a more comprehensive and accurate method for coding data and drawing conclusions. in order to understand the study findings, it is also important to take into account that the qualitative analysis part of this study was explorative in nature with a purpose to identify possible topics or gaps for future gwla sponsored research study. it should also be noted that the negative and positive coding instances of themes are not teased apart to isolate negative and positive coding separately; they are combined together under the major category/theme frequency numbers to demonstrate the need for exploration in the most commonly described topics/issues area.   finally, taskforce researchers with a variety of levels of qualitative expertise conducted the research. despite this limitation, the taskforce was able to set up an effective process for collaborative research and triangulate coding with partners. now that the process is defined, it will be easier to replicate this process and use this method as a possible model for conducting gwla collaborative qualitative research in the future; however, we did experience some accuracy and logistical issues in this first attempt at collaborative qualitative research using excel.   references   altbach, p. g., gumport, p. j., & berdahl, r. o. (eds.). (2011). american higher education in the twenty-first century: social, political, and economic challenges. baltimore, md: jhu press.   association of american colleges & universities. (n.d.) value rubric development project. retrieved from: https://www.aacu.org/value/rubrics   barth, m. (2013). many roads lead to sustainability: a process-oriented analysis of change in higher education. international journal of sustainability in higher education, 14(2), 160-175. http://dx.doi.org/10.1108/14676371311312879   bury, m. (2012) activity system. retrieved from: http://commons.wikimedia.org/wiki/file:activity_system.png.   chaiklin, s., hedegaard, m., jensen, u. j., aarhus. n (eds.) (1999). activity theory and social practice. denmark: aarhus university press.   christensen, c. m., & eyring, h. j. (2011). the innovative university: changing the dna of higher education from the inside out. hobocken, nj: john wiley & sons.   eckel, p. d., & kezar, a. j. (2003). taking the reins: institutional transformation in higher education. westport, ct: greenwood publishing group.   engeström, y. (2004). new forms of learning in co-configuration work. journal of workplace learning, 16(1/2), 11-21. http://dx.doi.org/10.1108/13665620410521477   engeström, y. (2001). expansive learning at work: toward an activity theoretical reconceptualization. journal of education and work, 14(1), 133-156. 31(4), 293-302. http://dx.doi.org/10.1080/13639080020028747   engeström, y., miettinen, r., & punamäki, r. l. (1999). perspectives on activity theory. boston, ma: cambridge university press.   engeström, y., & sannino, a. (2010). studies of expansive learning: foundations, findings and future challenges. educational research review, 5(1), 1-24. http://dx.doi.org/doi:10.1016/j.edurev.2009.12.002   fink, l. d. (2013). creating significant learning experiences: an integrated approach to designing college courses. hobocken, nj: john wiley & sons.   glaser, b. g., & holton, j. (2004, may). remodeling grounded theory. in forum qualitative social research, 5(2). retrieved at: http://www.qualitative-research.net/index.php/fqs/article/view/607/1315volume   greater western library alliance student learning outcome task force (gwla). (2013, november). librarians partnering for student learning: leadership, practice and culture assessment symposium, information available at: https://sites.google.com/a/gwla.org/greater-western-library-alliance/committees/slo.   hiller, s., kyrillidou, m., & self, j. (2006). assessment in north american research libraries: a preliminary report card. performance measurement and metrics, 7(2), 100-106. http://dx.doi.org/10.1108/14678040610679489   hu, s., scheuch, k., schwartz, r., gayles, j. g., & li, s. (2008). reinventing undergraduate education: engaging college students in research and creative activities. ashe higher education report, volume 33, number 4. ashe higher education report, 33(4), 1-103. http://dx.doi.org/10.1002/aehe.3304   hufford, j. r. (2013). a review of the literature on assessment in academic and research libraries, 2005 to august 2011. portal: libraries and the academy, 13(1), 5-35. http://dx.doi.org/10.1353/pla.2013.0005    kaufman, p., & watstein, s. (2008). library value (return on investment, roi) and the challenge of placing a value on public services. reference services review, 36(3), 226-231. http://dx.doi.org/10.1108/00907320810895314   kent state university libraries. (2016). rails: standardized assessment of information literacy skills. retrieved from: https://www.projectsails.org/   kent state university libraries. (2016). trails: tool for real-time assessment of information literacy skills. retrieved from: http://www.trails-9.org/   kezar, a. j., & eckel, p. d. (2002). the effect of institutional culture on change strategies in higher education: universal principles or culturally responsive concepts? the journal of higher education, 73(4), 435-460.  http://dx.doi.org/10.1002/aehe.2804   kezar, a. (2009). change in higher education: not enough, or too much?. change: the magazine of higher learning, 41(6), 18-23. http://dx.doi.org/10.1080/00091380903270110   kyrillidou, m. (2005). library assessment as a collaborative enterprise. resource sharing & information networks, 18(1-2), 73-87. http://dx.doi.org/10.1300/j121v18n01_07   menchaca, f. (2014). start a new fire: measuring the value of academic libraries in undergraduate learning. portal: libraries and the academy, 14(3), 353-367. http://dx.doi.org/10.1353/pla.2014.0020     oakleaf, m. (n.d.), rubric assessment of information literacy skills (rails). retrieved from: http://railsontrack.info/   oakleaf, m. (2010). the value of academic libraries: a comprehensive research review and report. chicago, il: association of college and research libraries. retrieved from: http://www.ala.org/acrl/sites/ala.org.acrl/files/content/issues/value/val_report.pdf   oakleaf, m. (2011, march). what’s the value of an academic library? the development of the acrl value of academic libraries comprehensive research review and report. australian academic & research libraries, 42(1), 1-13. http://dx.doi.org/10.1080/00048623.2011.10722200   pan, d., ferrer-vinent, i. j. & bruehl, m. (2014). library value in the classroom: assessing student learning outcomes from instruction and collections. the journal of academic librarianship, 40(3), 332-338. http://dx.doi.org/10.1016/j.acalib.2014.04.011   pham, h. t., & tanner, k. (2014). collaboration between academics and librarians: a literature review and framework for analysis. library review, 63(1/2), 15-45. http://dx.doi.org/10.1108/lr-06-2013-0064   phelps, s. f., & campbell, n. (2012). commitment and trust in librarian–faculty relationships: a systematic review of the literature. the journal of academic librarianship, 38(1), 13-19. http://dx.doi.org/10.1016/j.acalib.2011.11.003   pritchard, s. m. (1996). determining quality in academic libraries. library trends, 44(3), 572-595.   roth, w. m., & lee, y. j. (2007). “vygotsky’s neglected legacy”: cultural-historical activity theory. review of educational research, 77(2), 186-232 http://dx.doi.org/10.3102/0034654306298273   spencer, s. [graphic] (2014). a library level activity system model.   travis, t. a. (2008). librarians as agents of change: working with curriculum committees using change agency theory. new directions for teaching and learning, 2008(114), 17-33. http://dx.doi.org/10.1002/tl.314   vygotsky, l. s. (1980). mind in society: the development of higher psychological processes. boston, ma: harvard university press.     appendix a gwla survey questions   1.       does your institution have slo's that address information literacy (i.e., critical thinking, evaluation and synthesis of information) at any of the following levels? yes, no, don’t know responses for the: a.       campus level b.       college/department level c.        library level 2.       does the library assess information literacy slo's at any of the following levels? yes, no, don’t know responses for the: a.       campus level b.       college/department level c.        library level 3.       does the library measure the impact of its collaborations with classroom faculty and other academic partners? (yes, no, don’t know responses) 4.       would you, or someone else at your institution, be available to answer more in-depth questions about student learning outcomes and assessment at your institution? place to provide contact information     appendix b gwla interview script   the purpose of the interviews was to get more detailed information about the survey responses and therefore the possible interview prompt script was developed from that structure.   1. does your institution have slo’s that address information literacy (i.e. critical thinking, evaluation and synthesis of information) at any of the following levels (campus, college/department, or library levels) ? if yes:   o  please describe. o  is there a central slo organization (taskforce, department, committee etc.) on your campus that oversees college/unit student learning assessment? o  are they posted on a publicly accessible website? if yes, please provide the url. o  was it a cooperative effort to develop them?  if yes, was the library involved? o  if they exist but not formally stated, are they cultural? how do faculty know about them? o  did the slos have an impact? what programs have changed as a result of the slos? o  how have libraries built partnerships on campus that have led to the development of slos within courses and programs?   if no:   o  are there future plans to develop slos? o  is there a lack of resources or personnel to develop slos o  what actions should the library be taking? received comment that this question may be too leading.   what is limiting the institution in creating slos> what role does the library have in creating slos?   2.  does the library assess information literacy slo’s at any of the following levels (campus, college/department, library levels)?   if yes:   o  how are they assessed? o  how often are they assessed? o  in what venues? o  are the results shared with the wider academic campus? o  is it a cooperative effort with faculty?   if no:   o  are there future plans to develop slos? o  what actions should the library be taking? o  is there a lack of resources or personnel to develop slos? o  is there campus support for developing slos? o  will academic faculty be involved in their development? why or why not?   3. does the library measure the impact of its collaborations with classroom faculty and other academic partners?   if yes:   o  which collaborations does it measure? o  how? when? how often? o  are academic faculty included in the assessment? o  are the results shared with academic partners?   if no   o  any future plans to assess them? o  is there campus support for assessment? o  are there venues on your campus for people interested in discussing, sharing, or collaborating on institutional data or assessment? o  will academic faculty be involved in the assessment? why or why not?   4. is there anything further you want to add or discuss?         evidence summary   collaborative and interactive teaching approaches have a positive impact on information literacy instruction supporting evidence based practice in work placements   a review of: kolstad, a. (2017). students’ learning outcomes from cross-collaborative supervision in information seeking processes during work placements. nordic journal of information literacy in higher education, 9(1), 2-20. https://doi.org/10.15845/noril.v9i1.231     reviewed by: kelley wadson library information specialist bow valley college calgary, alberta, canada email: kwadson@bowvalleycollege.ca   received: 20 nov. 2018                                                                 accepted: 8 jan. 2019      2019 wadson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29530     abstract   objective – to analyze the effect of collaborative interdisciplinary teaching and supervision using physical and digital tools on students’ information literacy (il) and evidence based practice (ebp) abilities.   design – qualitative and quantitative text analysis.   setting – learning centre at oslo university college and student work placements in oslo, norway.   subjects – approximately 400 students enrolled in the undergraduate nursing degree programme.   methods – the author is a librarian and project manager of the langerud project, an initiative wherein nursing students were jointly trained and supervised by nurse educators, nurse supervisors, and librarians in preparation for and during work placements over an eight-week period. in this role, the librarian author collected 36 student group assignments, 285 blog/wiki comments from students, nurse educators, nurse supervisors, and librarians, and 102 individual student logs written during six work placements between spring 2010 and spring 2012, which were posted in a learning management system (lms), as well as in an evaluation form from spring 2010. the unstructured text is analyzed according to how the students fulfilled the learning outcome of integrating steps zero to four of the seven-step ebp model: (1) cultivate a spirit of inquiry; (2) ask clinical questions in the pico format; (3) search for the best evidence; (4) critically appraise the evidence; and (5) integrate the evidence with clinical expertise and patient preferences and values. the logs are also analyzed quantitatively to measure if and how many students combined the three aspects of ebp defined as being the practitioner’s individual expertise, best research evidence, and client values and expectations. lastly, the author seeks to evaluate the role of the lms as a mediating tool.   main results – the author found that the majority (83%) of students successfully met the learning outcome, particularly for steps 1, 2, and 5. for step three, the author observed that some students did not apply pico in the information-seeking process and were thus not sufficiently thorough in their searching. for step four, the author found that most students failed to demonstrate critical appraisal of the evidence and that many struggled to find up-to-date research findings. the author noted that the results for both steps three and four could be attributed to the students finding international databases and english-language research articles too challenging, given the language barrier. the author’s analysis of the logs reveals that two-thirds of the students combined the 3 aspects of ebp and that 39% described 1 or 2 aspects, of which most described user-based knowledge and experience-based knowledge. one department produced twice as many log entries as the other seven departments; in this department, students were able to choose what aspect of ebp to focus on and the librarian had a co-teaching role in that learning group. overall, 60% of all students described research-based knowledge, which increased over time from 46% in spring 2011, to 60% in autumn 2011, and 83% in spring 2012. on the evaluation form from spring 2010, most students rated the supervision by and satisfaction with the nurse educator, nurse supervisor, and librarian as good, very good, or excellent, and many commented that the lms was a useful learning platform.   conclusion – the author concludes that the project had a positive impact on students’ preparedness for work placements and that the early educational intervention improved il and ebp competencies. furthermore, the working relationship between the nursing department and library was strengthened. after the langerud project ended, the curriculum was revised to add more searching for research-based information in written assignments. additionally, a lecture on ebp was developed based on real-life experiences from the project and delivered collaboratively by the project’s manager, a nurse educator, and a librarian.   commentary   the study contributes to research supporting collaborative efforts to integrate il into nursing education and professional development (plaice, lloyd, & shaw, 2017; beck, blake-campbell, & mckay, 2012). although recent literature does discuss the information needs of healthcare students on work placements (plaice, lloyd, & shaw, 2017) and the importance of workplace il (lloyd, 2013), this study addresses the lack of research on the combination of these elements in the context of library instruction for nursing students.   the study was evaluated using koufogiannakis, booth, and brettle’s (2006) reliant instrument. while the objective, context, results, and relevance are well-explained and discernable to the reader, there are several flaws in the study design, results, and educational context.   the most notable flaw in the study design is the absence of appendices in relation to the research instruments, such as the evaluation form and assignment questions. although described in the results, the inclusion of these would allow explicit mapping of learning outcomes to teaching methods and assessments. as such, it is difficult to precisely determine if and how the latter were effective interventions for addressing the author’s research question examining the development of students’ il abilities and understanding. in addition, the author uses data from an evaluation form distributed in one session, spring 2010, but does not explain why this form was not used to collect data from subsequent sessions. lastly, as acknowledged within the paper, there is risk of bias in the study due to the author’s role as both researcher and practitioner.   in terms of educational context, the author provides a clear overview of the value of a collaborative approach for the delivery of il instruction. however, the study would benefit from a more explicit exploration of the modes of assessment used to build students’ il skills and abilities, particularly regarding the use of reflective thinking and writing as described in the results. this aspect of the study provides useful evidence to practitioners in the domain of affective learning, as the author traces the development of positive attitudes and behaviours toward il in the data.   although the study focused primarily on nursing students, it is relevant for all educators involved in il instruction intended to be relevant to workplace contexts and support evidence based practice. as mentioned above, it supports collaborative teaching approaches and the use of constructivist, interactive, and metacognitive teaching philosophies. this is consistent with a recent review recommending interactive and clinically integrated strategies, as well as collaboration with librarians for teaching evidence based practice (horntvedt, nordsteien, fermann, & severinsson, 2018). in concrete terms, the study provides evidence demonstrating that librarians should collaborate with other stakeholders when delivering il instruction, and that meaningful, interactive guidance is effective for improving il skills and abilities, particularly in the affective domain.   references   beck, s., blake-campbell, b., & mckay, d. (2012). partnership for the advancement of information literacy in a nursing program. community & junior college libraries, 18(1), 3–11. https://doi.org/10.1080/02763915.2012.651957   horntvedt, m. t., nordsteien, a., fermann, t., & severinsson, e. (2018). strategies for teaching evidence-based practice in nursing education: a thematic literature review. bmc medical education, 18(172), 1-11. https://doi.org/10.1186/s12909-018-1278-z   koufogiannakis, d., booth, a., & brettle, a. (2006). reliant: reader’s guide to the literature on interventions addressing the need for education and training. library & information research, 30(94), 44-51. https://doi.org/10.29173/lirg271   lloyd, a. (2013). building information resilient workers: the critical ground of workplace information literacy. what have we learnt? in s. kurbanoğlu, e. grassian, d. mizrachi, r. catts, & s. špiranec (eds.), worldwide commonalities and challenges in information literacy research and practice: european conference on information literacy, ecil 2013, istanbul, turkey (pp. 219-228). heidelberg, germany: springer. https://doi.org/10.1007/978-3-319-03919-0_28   plaice, c., lloyd, j., & shaw, p. (2017). supporting the library and information needs of uwe health and social care students on placement. health information & libraries journal, 34(1), 32–44. https://doi.org/10.1111/hir.12171   editorial   journal update from eblip9   lorie kloda editor-in-chief associate university librarian, planning and community relations concordia university montreal, quebec, canada email: lorie.kloda@concordia.ca      2017 kloda. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     this past june, the 9th international on evidence based library and information practice conference (eblip9) was held at drexel university, in philadephia (see the conference website http://eblip9.org/). this issue of the journal contains a feature highlighting some of the presentations and posters from the conference, as well as an editorial from the two conference co-chairs, danita nitecki and john wiggins.   at the conference, it’s been a long-held tradition for a member of the editorial board to give a presentation on evidence based library and information practice (eblip). the short presentation offers an overview of the journal’s editorial team, provides some usage and acceptance rate figures, and highlights recent achievement and upcoming changes. for readers who were not in attendance, i’m providing a summary of the presentation here. in addition to the journal’s editorial board members, eblip has 4 editorial advisors, 65 peer reviewers, 10 copyeditors, 14 evidence summary writers, and 4 writing assistants. the journal is in its 12 year of publications (with this issue, that will make a total of 48 published issues). the journal publishes several types of publications, including editorials, research articles, review articles, evidence summaries, classics, using evidence in practice, commentaries, news, and features.   an average issue includes 16 peer reviewed articles. research articles, review articles, evidence summaries and classics all undergo double-blind peer review. the acceptance rate for research and review articles in 2016 was 39%. in 2016 it took an average of 73 days to peer review a submission, and 187 days from submission to publication for those that were accepted.   eblip is indexed in lisa, library literature, lista, google scholar, scopus, the informed librarian online, and the directory of open access journals (doaj). as i reported in my editorial in 12(2), eblip recently was approved in the reapplication process of the doaj and received the doaj seal, which recognizes a journal’s commitment to open access best practices.   as of june 2017, there were 4,487 registered readers of eblip. the journal’s pages were viewed a total of 273,128 times between may 1, 2016 and april 30, 2017. at that time our twitter account (@eblip) had 1,180 followers. the eblip website visits come mainly from the united states (34%), followed by the united kingdom (13%), canada (12%), and australia (7%). the remaining 18% of visits are from people located in india, the philippines, nigeria, malaysia, sweden, and south africa.   the journal has several initiatives underway, which you can expect to hear more about in the coming year. these are: a data sharing policy, a controlled vocabulary for indexing, and peer reviewer and editor recognition through a partnership with publons (https://publons.com). i look forward to sharing the outcomes of these initiatives, as well as hearing from you about what you think the journal should be doing.   evidence summary   academic librarians perceive duration and social interaction as important elements for professional development   a review of: attebury, r. i. (2017). professional development: a qualitative study of high impact characteristics affecting meaningful and transformational learning. the journal of academic librarianship, 43(3), 232-241. http://dx.doi.org//10.1016/j.acalib.2017.02.015   reviewed by: hilary bussell assistant professor/social sciences librarian the ohio state university libraries columbus, ohio, united states of america email: bussell.21@osu.edu   received: 2 feb. 2018     accepted: 27 apr. 2018      2018 bussell. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29419       abstract   objective – to understand the characteristics of meaningful and transformational professional development experiences of academic librarians.   design – qualitative analysis of in-depth interviews using a hermeneutic phenomenological approach.   setting – public and private colleges and universities in the united states of america.   subjects – 10 academic librarians.   methods – the researcher selected 10 participants using an initial survey distributed through national library electronic mail lists. two rounds of semi-structured in-depth interviews were conducted over skype during fall 2014 and spring 2015. the first round of interviews began with background questions about participants’ careers, then moved on to questions about professional development experiences that were meaningful and/or transformational. the responses from this first round of interviews were used to develop questions for a second round of interviews with the same participants. after completing the interviews, the researcher sent follow-up emails to participants in order to gather feedback on summaries and interpretations of interviews. the transcribed interviews were used to create an initial set of codes and then imported into nvivo for analysis using a hermeneutic phenomenological approach.   main results – all participants reported on professional development experiences that they found to be meaningful. half of the participants discussed professional development experiences that were transformational for their perceptions and practice of librarianship. the themes of duration and interaction were identified in every participant’s discussions of meaningful or transformational professional development. reflection, discomfort, and self-awareness were also identified as prominent themes.   conclusion – the study found that two of the most important ingredients for meaningful and transformational professional development are activities that are sustained over time and that include social interaction. the participants perceived long-term interactive professional development activities as opportunities to identify and address gaps in their professional knowledge, which benefits themselves and their organizations. on-the-job learning, single-theme workshops or institutes, and professional committee work were particularly promising forms of meaningful professional development.   the author recommends that academic librarians who are interested in meaningful or transformational professional development look for activities that are sustained and interactive, that promote reflection, and that provide opportunities to increase self-awareness of gaps in knowledge. facilitators of professional development activities should include interactive components and ensure that participants have a chance to stay in contact after the event in order to encourage long-term interaction and reflection.    commentary   librarian professional development has garnered significant interest in recent years. with rapid changes in the library landscape, continuing education and skill development will continue to be essential for librarians (acrl professional development committee, 2000). a number of recent studies explore professional development for specific areas of academic librarianship, such as information literacy (shamchuk, 2015) and data librarianship (conrad, shorish, whitmore, & hswe, 2017). building on previous work (attebury, 2015), this study uses adult educational theory to identify characteristics of meaningful and transformational professional development across academic librarianship.   the form developed by letts et al. (2007) was used to evaluate the strengths and weaknesses of this study. the purpose and justification for this study are clearly articulated, and the author grounds her research question and findings in adult education theory. hermeneutical phenomenology is an appropriate research design, as it can be used to interpret and develop practices and policies around a social phenomenon based on participants’ first-hand descriptions. the author describes how hermeneutical phenomenological techniques were used to enhance the analytical rigor of the study, and she includes a number of transcript excerpts in order to more fully illustrate the importance of the themes. the fact that the descriptions and interpretations of the interviews were provided to participants for feedback increases the credibility of the results.   the main weakness of this paper lies in the lack of descriptive clarity of the sampling and data collection methods. although the author mentions that the interview participants were purposively selected, she does not provide the criteria for selection. further, she does not indicate whether 10 participants was determined to be an appropriate sample size in advance, or whether sampling was done until redundancy in data was reached. more discussion of the purpose and development of the second round of interviews would also be helpful. specifically, it is not clear whether these interviews were conducted to address gaps and misunderstandings of the content covered in the first round of interviews or to cover additional content. it is also unclear whether individual sets of follow-up questions were constructed for each participant. finally, the reader’s overall understanding of how the findings follow from the data would be enhanced if the interview questions were included as appendices at the end of the paper. the lack of transparency of methods may cause problems for anyone wanting to reproduce this study.   this study is significant in its focus on adult educational theory, particularly the constructs of meaningfulness and transformational learning, in exploring the professional development experiences of academic librarians. with limited time and money, librarians face a daunting task in choosing among the plethora of professional development activities available. this study will help librarians and their organizations to know how to identify opportunities that are likely to be the most impactful for their practice.   references   acrl professional development committee. (2000). acrl statement on professional development. retrieved from http://www.ala.org/acrl/publications/whitepapers/acrlstatement    attebury, r. i. (2015). adult education concepts in library professional development activities. new library world, 116(5/6), 302–315. http://dx.doi.org/10.1108/nlw-08-2014-0100   conrad, s., shorish, y., whitmire, a. l., & hswe, p. (2017). building professional development opportunities in data services for academic librarians. ifla journal, 43(1), 65-80. http://dx.doi.org/10.1177/0340035216678237   letts, l., wilkins, s., law, m., stewart, d., bosch, j., & westmorland, m. (2007). critical review form qualitative studies (version 2.0). retrieved from https://srs-mcmaster.ca/wp-content/uploads/2015/04/critical-review-form-qualitative-studies-version-2-english.doc    shamchuk, l. (2015). professional development on a budget: facilitating learning opportunities for information literacy instructors. partnership: the canadian journal of library & information practice & research, 10(1), 1-14. http://dx.doi.org/10.21083/partnership.v10i1.3437     evidence based library and information practice evidence based library and information practice 2009, 4:2  evidence based library and information practice       editorial  weaving evidence, reflection, and action into the fabric of school librarianship   carol a. gordon  associate professor  school of communication, information and library studies  rutgers, the state university of new jersey  new brunswick, new jersey, united states of america  e‐mail: cgordon@scils.rutgers.edu    ross j. todd  associate professor  school of communication, information and library studies  rutgers, the state university of new jersey  new brunswick, new jersey, united states of america  e‐mail: rtodd@scils.rutgers.ed    © 2009 gordon and todd. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      this issue of evidence based library and  information practice is the first devoted to  evidence based practice (ebp) in school  librarianship. the focus on ebp in school  librarianship has a shorter history than its  counterparts in evidence based education and  evidence based librarianship.  that history  began with a speech given by todd at a  conference of the international association of  school librarianship in 2001. in this speech the  convergence of two critical concepts defined a  21st century mission for school libraries. the  first was an extension of the definition of  information literacy:     in order for school libraries to play a key  role in the information age school, i  believe   there needs to be a fundamental shift  from thinking about the movement and  management of information resources  through structures and networks, and  from information skills and information  literacy, to a key focus on knowledge  construction and human   understanding, implemented through a  constructivist, inquiry‐based  framework.ʺ (todd “transitions”)     this statement complicates the teaching of  information literacy. it is no longer enough to  teach location and evaluation of resources;  information literacy now encompasses the full  utilization of information in the process of  constructing new knowledge and connecting to  4 mailto:cgordon@scils.rutgers.edu mailto:rtodd@scils.rutgers.ed evidence based library and information practice 2009, 4:2  existing knowledge. the second critical concept  addressed evidence‐based practice:    information is the heartbeat of  meaningful learning in schools. but it is  not the hallmark   of the 21st century school. the hallmark  of a school library in the 21st century is  not its   collections, its systems, its technology,  its staffing, its buildings, but its actions  and evidences that show that it makes a  real difference to student learning, that  it contributes in tangible and significant  ways to the development of human  understanding, meaning making and  constructing knowledge. the school  library is about empowerment,  connectivity, engagement, interactivity,  and its outcome is knowledge  construction. (todd “transitions”)     these paradigm shifts prepared school  librarians to look at their practice differently. it  is with this new mission that school libraries  embrace evidence based practice ʺ...where day‐ by‐day professional work is directed toward  demonstrating the tangible impact and  outcomes of sound decisions making and  implementation of organizational goals and  objectives.” (todd “learning” 7).   this laid the groundwork for evidence based  practice to become an organizing principle for  the continuous improvement of school  librarianship. school librarians had traditionally  taken a holistic view of continuous program  improvement through the collection of evidence  that measures the status of facilities, collections,  access to information, staffing and funding.  typically, these measures are quantitative; they  document size, age, capacity, and amount. this  statistical approach supports evidence based  advocacy in terms of inputs, but is not  particularly effective in assessing the value that  the school library adds to the school’s mission,  which focuses on targets learning outcomes. in  recent years there has been a shift to a more  qualitative approach to program evaluation. for  example, school librarians incorporate questions  about perceptions of school library efficacy  when surveying students and staff. with a more  qualitative perspective, instruction emerges as  an element of the school library that could be  evaluated in terms of what worked and what  didnʹt.  since its emergence in 2001, a community of  researchers and practitioners has contributed to  a growing body of literature that is represented  in this landmark edition of eblip.  the article by  ballard, march & sand describes two action  research projects on plagiarism. one project is  set in an american high school, the other in an  elementary school. the two school librarians  who conducted the action research applied skills  acquired through the use of gordon’s three  dimensional action research training model to  assess whether their students were ethical users  of ideas and information. “the use of action  research as a means to provide a rich and  meaningful professional development  opportunity for school library media specialists  and teachers has transformed instructional  practice in the londonderry school district… a  collaborative research culture has been created  in which all those responsible for student  learning engage in meaningful reflection on  instructional and program practices at the local  level through an examination of evidence  provided through the process..”    farmer examines the field of special education  and its relationship to educational librarianship.  a review of studies about school library media  specialist practices relative to special education  personnel was conducted, and cross‐study  factors were identified that facilitate  collaboration between school library media  specialists and special educators in support of  student achievement. the paper concludes,  “school library media specialists and special  education personnel have much expertise to  share with each other and with others in the  5 evidence based library and information practice 2009, 4:2  6 school community. increased research and  documentation of those efforts can strengthen  their collaboration and contribute significantly  to student achievement.”    part i of a paper by gordon develops the  concept of action research as a tool of evidence‐ based practice in the context of inquiry learning  in the school library. it traces the evolution of  evidence based teaching, a pedagogy that  triangulates inquiry learning, the information  search process, and evidence based practice.  action research emerges as a viable tool that  facilitates the collection of evidence that is basic  to the continuous improvement of student  performance as information searchers and users.  part 1 lays the groundwork for an emerging  theory for evidence‐based teaching. “the  emerging theory indicates that a model for a  culture of inquiry in the learning environment of  the school library is interdisciplinary, bringing  together user‐centered information searching  and learner‐centered educational theory. it  points to a research agenda and informs the  methods appropriate for the study of evidence  based information literacy instruction in school  libraries.” part 2 of this paper will be published  in the fall edition of eblip.      todd provides an overview of progress and  developments in evidence‐based practice in  school librarianship, and provides a picture of  current thinking about evidence‐based practice  as it relates to the field. he notes, “a focus on  evidence based school librarianship shifts the  basis on which value statements can be made  about school libraries if they are to be perceived  as playing a strong role in the school. by placing  emphasis on measured outcomes, evidence  based practice in school libraries shifts the focus  from the medium to the message and  articulating what school librarians do in their  day‐to‐day work, to articulating what students  become.  by placing emphasis on systematically  gathered evidence, it moves school library  advocacy from a “tell me” framework to a  “show me” framework.”    yukawa and harada analyzed the effects of a  practice‐based model of professional  development on the teaching and collaborative  practices of nine teams of librarians and teachers  who created and implemented units of inquiry‐ focused study with k‐12 students during a  yearlong course. the authors describe how the  collection and analysis of evidence guided the  development team in the formative and  summative evaluations of the outcomes of the  professional development, as well as the long‐ term results of participation in this initiative.  “the practice‐based model of professional  development described in this study appears to  be effective and sustainable.”    it is with great pleasure that we bring you the  work of this community of formal researchers  and practitioner‐researchers who are pioneering  the application and study of evidence based  practice in school libraries.       works cited  gordon, carol a. “a study of a three‐ dimensional action research training  model for school library programs.”  school library media research 9 (2006).  19 may 2009  <http://www.ala.org/ala/mgrps/divs/aas l/aaslpubsandjournals/slmrb/slmrconten ts/volume09/gordon_study3daction.cfm >.  todd, ross j. “learning in the information age  school: opportunities, outcomes, and  options.” keynote paper. international  association of school librarianship  conference, durbin, south africa, 7‐11  july 2003.  19 may 2009  <http://www.iasl‐ online.org/events/conf/conference2003‐ virtualpap.html>.  todd, ross j. “transitions for preferred futures  of school libraries: knowledge space,  http://www.ala.org/ala/mgrps/divs/aasl/aaslpubsandjournals/slmrb/slmrcontents/volume09/gordon_study3daction.cfm http://www.ala.org/ala/mgrps/divs/aasl/aaslpubsandjournals/slmrb/slmrcontents/volume09/gordon_study3daction.cfm http://www.ala.org/ala/mgrps/divs/aasl/aaslpubsandjournals/slmrb/slmrcontents/volume09/gordon_study3daction.cfm http://www.iasl-online.org/events/conf/conference2003-virtualpap.html http://www.iasl-online.org/events/conf/conference2003-virtualpap.html http://www.iasl-online.org/events/conf/conference2003-virtualpap.html evidence based library and information practice 2009, 4:2  not information place. connections,  not collections. actions, not positions.  evidence, not advocacy.”  keynote  address. international association of  school librarianship conference,  auckland, new zealand, 9‐12 july  2001. 19 may 2009 <http://www.iasl‐ online.org/events/conf/virtualpaper200 1.html>.  7 http://www.iasl-online.org/events/conf/virtualpaper2001.html http://www.iasl-online.org/events/conf/virtualpaper2001.html http://www.iasl-online.org/events/conf/virtualpaper2001.html these paradigm shifts prepared school librarians to look at their practice differently. it is with this new mission that school libraries embrace evidence based practice "...where day-by-day professional work is directed toward demonstrating the tangible impact and outcomes of sound decisions making and implementation of organizational goals and objectives.” (todd “learning” 7). evidence summary   library impact on student retention is often not well documented or communicated   a review of: murray, a. l., & ireland, a. p. (2017). communicating library impact on retention: a framework for developing reciprocal value propositions. journal of library administration, 57(3), 311-326. http://www.tandfonline.com/doi/full/10.1080/01930826.2016.1243425   reviewed by: heather macdonald health and biosciences librarian macodrum library carleton university ottawa, ontario, canada email: heather.macdonald@carleton.ca   received: 16 aug. 2017 accepted: 5 dec. 2017      2017 macdonald. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract    objective – identification of trends in documenting and communicating library impact on student retention.  presentation of a framework of library stakeholders with examples of how libraries can communicate their value to each stakeholder group.   design – survey and presentation of framework.   setting – comprehensive universities in the usa.   subjects – 68 academic library deans/directors.   methods – a survey on current methods of documenting and communicating library impact on student retention was sent to all 271 comprehensive universities with a carnegie classification of master’s level.  the response rate was 25%.  emergent themes were identified using nvivo for the qualitative data analysis.    the six markets model was presented as a framework for identifying library stakeholder groups.  examples of reciprocal value propositions (rvp) for each stakeholder group were provided.   main results – analysis of the survey results identified a number of themes about documenting library impact on student retention: use of information literacy assessment, use of satisfaction or feedback instruments (eg: survey, focus group), library-use data, and lack of knowledge of methods.  several responses indicated the methods used for information literacy assessment were not a direct measure for documenting impact on retention.  a few institutions piloted more direct methods by combining library use data and student success metrics.  a number of institutions said they struggled with how to use library-use data to calculate library impact on retention.    methods for communicating library impact on retention included formal presentations, annual reports, annual assessment reports, informal communication, and none. communication was often tied to documentation; if a library did not collect or document impact on retention, they were not able to communicate anything.  the authors noted communication tended to be unidirectional rather than being a multidirectional discussion between the library and its stakeholders.   based on the six markets model, the authors identified six library stakeholder groups that would benefit from understanding library impact on student retention.  the authors postulated that identifying these markets would allow the library to define value propositions for each market.  the value propositions for each market would be reciprocal because value would be co-created when the library engages with each stakeholder group to fill a service need. the authors proposed that identifying and engaging with stakeholders, and defining reciprocal value propositions for each, would provide the library with an opportunity to advocate for itself.   conclusion – some libraries are documenting and communicating library impact on student retention but many are not.  there is a lack of knowledge of how to document impact.  the authors suggest more direct methods of measuring library impact are needed, as are more deliberate approaches to communicating impact.    commentary – this commentary uses the cat critical appraisal tool (perryman & rathbun-grubb, 2014) to guide the appraisal. the authors situated this study by referring to the 2010 report by oakleaf, value of academic libraries: a comprehensive research review and report.  according to the authors, this report is a call to academic libraries to promote the library’s value to their stakeholders.  the authors provided background on studies investigating library impact on student retention including a 2016 study by the first author. they identified a gap in the literature around using these findings for library advocacy. the authors asked how library value is currently being documented and communicated by library deans. the authors do not explicitly state the connection between library advocacy and communication.    the authors conducted a survey to assess how library leaders are currently assessing and conveying library impact on student retention to stakeholders and in particular to institutional administration.  the survey tool was not published.  the authors provided minimal information about their survey population.  with only a 25% rate response rate, it would have been helpful to know the characteristics of the non-responders.  there was no quantitative analysis of the themes provided, rather the authors used “highly prevalent”, “less prevalent,” or “prevailing” to describe the frequency of the themes.  in particular seeing how many libraries did not document or communicate impact on retention would have been revealing. a visual aid would have illustrated this at a glance.    the authors provided detailed information on the origin of the six markets model from the marketing literature and explained what the six markets are in terms of library stakeholders.  they clearly explained the reciprocal nature of a value proposition and gave many examples for each stakeholder group including some based on the survey findings.  the authors suggested library leaders could use these tools to strengthen their advocacy role when communicating with senior administration.    the authors discussed future research topics but did not discuss the limitations for the current study. there was no discussion of why only comprehensive universities with a carnegie classification of master’s level were included nor what impact the 25% response rate could have had on the results.  the analysis of the data was sparse in this paper; it would have benefited from a more granular analysis and visual presentations. the paper offers a unique contribution to the library literature with the introduction of the six markets model and reciprocal value propositions as tools to help with communicating library value.  according to the authors, reciprocal value propositions can help with “altering traditional mindsets – an important activity for libraries struggling to be viewed as a service provider with value beyond their collections.”  therein lies the real value of this paper.   references           perryman, c. & rathbun-grubb, s. (2014). the cat: a generic critical appraisal tool. in jotform – form builder. retrieved from http://www.jotform.us/cp1757/thecat   oakleaf, m. (2010). the value of academic libraries: a comprehensive research review and report. chicago: american library association.  retrieved from http://www.ala.org/acrl/sites/ala.org.acrl/files/content/issues/value/val_report.pdf   microsoft word es_hannigan.doc evidence based library and information practice 2007, 2:4  61 evidence based library and information practice       evidence summary    risk profile may affect search process but not results       a review of:  mckibbon, k. ann, douglas b. fridsma, and rebecca s. crowley. “how primary care  physicians’ attitudes toward risk and uncertainty affect their use of electronic  information resources.” journal of the medical library association 95.2 (2007): 138‐46,  e49‐50. 10 nov. 2007  <http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=1852627>.       reviewed by:   gale g. hannigan  professor, texas a&m medical sciences library  college station, texas, united states of america  e‐mail: g‐hannigan@tamu.edu        received: 01 september 2007    accepted: 14 october 2007      © 2007 hannigan. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to compare the use, in terms of  process and outcomes, of electronic  information resources by primary care  physicians with different risk profiles and  comfort with uncertainty.    design – survey, and observation using  “think‐aloud” method.    setting – physicians’ offices.    subjects – canadian and u.s. primary care  physicians who report seeing patients in  clinic settings.    methods – volunteers were recruited from  personal contacts and the list of physicians  who rate current studies for the mcmaster  online rating of evidence (more) project.  physicians completed the pearson scale to  measure attitude toward risk and the  gerrity scale to measure comfort with  uncertainty, and those who scored at the  extremes of each of these two scales were  included in the study (n=25), resulting in  four groups (risk‐seeking, risk‐avoiding,  uncertainty‐stressed, uncertainty‐ unstressed). one researcher observed each  of these physicians in their offices for an  hour during which they completed  questionnaires about their computer skills  and familiarity with resources, answered  http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=1852627 mailto:hannigan@tamu.edu http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:4  62 multiple‐choice clinical questions, and  indicated level of certainty with regard to  those answers (scale of 0 to 100%).  physicians also chose two of the clinical  questions to answer using their own  resources. the think‐aloud method was  employed, and transcripts were coded and  analyzed.    main results – the study analysis included  two comparisons: risk‐seeking (11 subjects)  versus risk‐avoiding (11 subjects) physicians,  and uncertainty‐stressed (11 subjects) versus  uncertainty‐unstressed (10 subjects)  physicians. most physicians were included  in both sets of analyses. the researchers  found no association of risk attitude and  uncertainty stress with computer skills nor  with familiarity and use of specific  information resources (internet, medline,  pier, clinical evidence, and uptodate). no  differences were found for the following  outcomes: time spent searching, answers  correct before searching, answers correct  after searching, and certainty of answer if  answer is right, certainty of answer if  answer is wrong. there was a statistically  significant association  of participants’  indicating certainty for answers that were  correct versus those that were not correct  (p<0.03). across all groups, the percentage  of correct answers after searching was low,  with the uncertainty‐stressed group  achieving the highest at 55.5%. there were  differences observed in search process,  including a trend toward a statistically  significant difference in the number and  proportion of search methods used across  groups between the risk‐seeking versus the  risk‐avoiding groups. risk‐seekers and  those not stressed by uncertainty used more  search heuristics (rules of thumb or guides  that work in most circumstances) than their  counterparts when searching.  those  stressed by uncertainty tended to use  medline less than those not stressed by  uncertainty.     conclusion – based on a small sample, it  appears that primary care physicians who  are risk‐seeking and/or less stressed by  uncertainty have the same (poor) search  results as their more risk‐averse and  stressed‐by‐uncertainty colleagues although  their search processes may differ.     commentary     mckibbon, fridsma, and crowley present a  thoughtfully designed study based on  previous research identifying risk profile  characteristics of physicians. they employ  validated scales to categorize risk‐ seeking/risk‐avoiding and uncertainty‐ stressed/uncertainty‐unstressed physicians,  and select only those who fall into the  extremes to include in the study.  appendices in print and online provide  additional information about the study and  its tools. it is interesting that their work  reinforces previous findings, cited by the  authors, that processes but not outcomes  may differ between those who can be  categorized as risk‐averse and those who are  risk‐seeking. the authors identify the  study’s limitations including small sample  size, its lack of statistical power to detect a  difference, and the use of volunteers. this  reviewer would add concerns about the lack  of blinding – was the observer aware of the  categorization of the physician whose office  she visited and whom she observed  searching?  also, it would have  strengthened the study to use an  independent observer, rather than someone  who was probably known to many of the  subjects. in two places, the text indicates  that risk avoiders tended toward less use of  systems that provided answers and  summaries of evidence, but the data indicate  otherwise (table 4, 143).     on the one hand, this research prompts  questions for further investigation into the  relationship between risk profile (or other  personality traits, like patience or self‐ evidence based library and information practice 2007, 2:4  63 confidence) and information retrieval to  inform systems design and training. on the  other hand, one wonders about the  applicability of this research for those of us  who are not in a position to assess the risk  profiles of those we teach and who would  predictably be faced with a heterogeneous  audience in any case. the most troubling  result of this study is the overall lack of  success – even by physicians involved in  rating evidence and assumed to be more  information savvy – in finding correct  answers to clinical questions, independent  of risk profile. this failure to easily and  correctly answer clinical questions from the  literature has been a recurrent finding, and a  reason why evidence summary tools were  developed in the first place. it is reassuring  that those systems are being used, though  distressing that their use doesn’t guarantee  correct answers to clinical questions.  mckibbon and her colleagues are on to  something: we need to build better systems  for retrieval as well as train physicians to be  more effective searchers for clinical  information, although the strategy for doing  this may need to be less refined than  identifying individuals’ risk profiles.     evidence based library and information practice editorial responsibilities      2016. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     editor-in-chief: lorie kloda   associate editor (evidence summaries): heather pretty   associate editor (research articles): rebekah (becky) willson, lisl zach   associate editor (classics & reviews): i. diane cooper   associate editor (commentaries, using evidence in practice, research in practice, news): lorie kloda   production editor: michelle dunaway   editorial intern: melissa griffiths   editorial advisors: lindsay alcock, alison brettle, jonathan eldredge, denise koufogiannakis, katrine mallan, pam ryan   copyeditors: richard hayman (lead copyeditor), georgianne bordner, julie evener, heather healy, julie james, stacey penney, maria tan, nikki tummon, alison yeoman   indexing support: meral alakus   research article   the library’s impact on university students’ academic success and learning   jung mi scoulas clinical assistant professor and assessment coordinator university of illinois at chicago chicago, illinois, united states of america email: jscoul2@uic.edu   sandra l. de groote professor and head of assessment and scholarly communications university of illinois at chicago chicago, illinois, united states of america email: sgroote@uic.edu   received: 7 jan. 2019                                                                      accepted: 18 july 2019      2019 scoulas and groote. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29547     abstract   objective – the purpose of this study was to examine relationships among student library visits, library resource use, library space satisfaction (e.g., quiet study space), and students’ academic performance (i.e., grade point average or gpa) using quantitative data and to better understand how the academic library has an impact on students’ learning from students’ perspectives using qualitative data.   methods – a survey was distributed during the spring 2018 semester to graduate and undergraduate students at a large public research institution. survey responses consisted of two types of data: (1) quantitative data pertaining to multiple choice questions related to the student library experience, and (2) qualitative data, including open-ended questions, regarding students’ perceptions of the library’s impact on their learning. quantitative data was analyzed using spearman’s rank correlations between students’ library experience and their gpas, whereas qualitative data was analyzed employing thematic analysis.   results – the key findings from the quantitative data show that student library visits and library space satisfaction were negatively associated with their gpa, whereas most students’ use of library resources (e.g., journal articles and databases) was positively associated with their gpas. the primary findings from the qualitative data reveal that students perceived the library as a place where they can concentrate and complete their work. additionally, the students reported that they utilize both the quiet and collaborative study spaces interchangeably depending on their academic needs, and expressed that the library provides them with invaluable resources that enhance their coursework and research.   conclusions – while the findings show that the student library experience was associated with their academic achievements, there were mixed findings in the study. the findings suggest that as a student’s gpa increases, their in-person library visits and library space satisfaction decrease. on the other hand, as a student’s gpa increases, their library resource usage increases. further investigation is needed to better understand the negative relationship between students’ library visits, library space satisfaction, and their gpas.     introduction   academic libraries exert great effort to demonstrate with empirical evidence that library use has an impact on students’ academic success and learning. our public research university library is no exception and considers this an important task. we conducted a locally developed survey to examine how students’ library experiences (e.g., frequency of library visits, library resource use, and satisfaction with library spaces) are associated with their academic achievement. at the same time, we further asked how they perceived the library’s impact on their learning, using an open-ended question. by integrating and comparing quantitative data with qualitative data, we aim to gain a broader understanding of the influence of students’ library experiences on academic achievement and learning. in this manner, this study aspires to provide a deeper understanding of how an academic library demonstrates the impact of the library on students’ learning, using their self-reported data and institutional data (e.g., grade point average or gpa).    literature review   academic libraries are under constant pressure to prove their value. to demonstrate their value in a “clear, measurable and [meaningful]” way using existing information, the association of college & research libraries’ (acrl) value of academic libraries initiatives issued value of academic libraries: comprehensive review and report (oakleaf, 2010, p.8) suggesting several possible correlations between students’ academic success (e.g., gpa) and their library data (e.g., checkouts, database use, and library instructions). since then, college and university libraries have increasingly dedicated resources to reveal the library’s impact on students’ academic success using various assessment measurements.   in an effort to demonstrate the library’s impact on students’ learning and academic achievement, many researchers have used their library data such as online library resource use and library instructional workshops. more than 200 higher educational institutions, from community colleges to doctorate/research universities, participated in the assessment in action (aia) project between 2013 and 2016 and provided compelling evidence regarding the relationship between library use and students’ learning and outcomes (brown, 2018). for example, eastern kentucky university, one of the participants in the aia project, examined undergraduate students’ online library resource use and their gpas. the results showed that students who logged into online library resources had a higher gpa than their peers who did not access its online resources (brown & malenfant, 2018a). in addition to the eastern kentucky university study, similar results from other institutions in the aia project (e.g., murray state university, york university, california state university-east bay) showed a positive relationship between students’ library use and their gpas (brown & malenfant, 2018a, 2018b).   a study undertaken at the university of minnesota by soria, fransen, and nackerud (2013) measured whether the use of library resources and services (e.g., databases and library workshops) was associated with student retention and gpa for first-year undergraduate students. using student database login information and student instructional and reference interactions, they demonstrated that students’ gpas were positively associated with students’ library use (e.g., database use, electronic journals, and book loans), but students’ gpas were negatively associated with course-integrated instruction sessions. a two-year study at the university of nebraska-lincoln examined whether use of library resources activities (e.g., circulation checkouts and off-campus access to databases) were related to both undergraduate and graduate students’ gpas (allison, 2015). data was employed from student information systems, including students’ identification number, gpa, and class information (e.g., undergraduate and graduate students). a positive correlation between variations of library use and changes in gpa was found for two years for both undergraduate and graduate students. while soria and colleagues (2013) focused only on first-year undergraduate students and used internal data from their institutions and one year of library data, allison included undergraduate students beyond the first year and examined two years of library data.   the studies above (allison, 2015; soria et al., 2013) focused on the correlations between students’ use of library resources and their academic achievements to examine academic libraries’ impact on students’ academic success. other academic libraries have examined other types of library usage data, including use of library space and technology (e.g., checking out laptops). for instance, the illinois institute of technology (iit) from the aia project, examined the relationships between library use (e.g., entering the library, checking out laptops, and using study rooms) and students’ gpas. the results showed that students who used these library resources and services had a higher gpa than their peers (brown & malenfant, 2018a). massengale, piotrowski, and savage (2016) used library usage data such as entry into the library by collecting students’ id numbers at the building entrance and when reserving study rooms, and manually collected data (e.g., borrowing laptops) in addition to institutional data (e.g., gpa and students’ participation in extracurricular activities). they found that students who reserved group study rooms had the highest gpas, followed by those who used their laptop in the library.   other institutions also used library and institutional data to explore the impact of libraries not only on students’ gpas but also their learning. at the university of minnesota, soria, fransen and nackerud (2017) used the student experience in the research university (seru) survey, a campus-wide survey measuring the students’ educational experience and academic engagement. the survey data was used in addition to library data (e.g., students’ library use in collections, web-services, and online chat with librarian) and institutional data (students’ demographic information and gpa) to examine whether first-year college students’ use of academic libraries was associated with students’ academic outcomes (e.g., students’ academic engagement, academic skills and gpa). soria and colleagues found that students who consulted or met with reference librarians had higher academic engagement and academic skills. in addition, they found that students who used web-based services (e.g., databases or library websites) and engaged in library instruction (e.g., workshops or course integrated instruction) had a higher gpa in comparison to students who did not use these services (soria et al., 2017).   while the above-mentioned studies demonstrated a correlation between library use and students’ academic success by utilizing a wide range of data sources collected from libraries and institutions, academic libraries face challenges to gathering and accessing this data (oakleaf, 2016; oakleaf, whyte, lynema & brown, 2017). depending on the academic institution’s policies (e.g., data privacy), capability (e.g., data analysis), or capacity (e.g., recording data), not all academic institutions provide access to this data. given these limitations, what types of assessment methods could institutions use to measure the library’s value in students’ academic success when the data is not accessible, or the data is accessible but cannot be used for research studies?   to get around these difficulties, some libraries have developed their own survey instruments and examined whether or not students’ library space usage has an impact on their learning needs and their academic achievements. for instance, montgomery (2014) created an ethnographic survey regarding library space to examine whether or not students’ behaviours using library spaces changed (e.g., spaces for working alone or working with other students), and how these decisions impacted students’ learning before and after renovation. she found that students’ use of library spaces for working with others before and after the renovation were unchanged, while students using library spaces to work alone increased after the renovation. while montgomery’s study contributed to measuring how students perceived the library space, and how this perception affected learning behaviours, this study did not attempt to directly link these decisions to students’ academic achievement, such as gpa. other researchers attempted to examine how students used the library space, and the relationships between the students’ library space use and their learning. using survey questions including self-reported gpa and observational sweep seating, a method observing students’ use of the library space, may and swabey (2015) found that students used the library space to do their academic work. however, self-reported gpas were not correlated with their number of visits to the library nor the amount of academic work completed in the library.   aims   by considering what types of data were available to our institution, and our university library’s desire to measure the impact of the library on students’ success, the current study used survey questions locally developed by the university’s library and examined how students’ library visits, use of library resources, and library space satisfaction are associated with students’ gpas. unlike previous studies assessing only the correlations of limited resources (e.g., books or multimedia checkout and gpa; library instruction and gpa), the current study aims to examine the comprehensive picture of the student library experience (e.g., library visits, use of library resources, and library space satisfaction) and its influence on gpa. furthermore, students’ open-ended responses to the one survey question were further analyzed to better understand how students perceived that the academic library has an impact on their learning. open-ended survey responses give respondents an opportunity to explain and express themselves in a narrative form, which is very helpful for organizations to gain insight and better understand users’ needs (jackson & trochim, 2002). additionally, having a key open-ended question in online surveys can increase the richness of responses, especially for respondents interested in the question topic (holland & christian, 2009). as such, this study aims to address the following questions:   question 1 from the quantitative data: what are the relationships among student library visits, resource use, library space satisfaction, and their academic achievement (gpa)? question 2 from the qualitative data (open-ended): how do students perceive the library’s impact on their learning experiences?   using both quantitative and qualitative data can assist library staff with developing a better understanding of how students consider the benefits of using the library for their own learning, and what types of resources are associated with their learning.   methods   in this study, a survey was distributed collecting both quantitative data and qualitative data through the use of an open-ended question. after analyzing the quantitative and qualitative data separately, the results were used to address the two research questions above.    institutional setting   the institution is a public research university in the midwest serving approximately 29,000 students who are enrolled in 15 different colleges. it offers hundreds of degree programs at a bachelor’s, master’s and doctoral degree level, and has more than 80 undergraduate majors and 60 minors. at this academic institution, a total of 5 libraries are located across multiple campuses.   measures   the student survey was developed in 2015 by   the library’s assessment advisory committee (aac) in collaboration with an outside research consultant with expertise in library assessment and experience in developing library surveys. the aac consisted of seven library faculty who represented various units at the university library. hiring a research consultant ensured that the type of survey and content were appropriate to guiding the university library in the right direction. the role of the aac was to ensure that all of the survey questions pertain to our organization’s information needs by focusing on users’ satisfaction and library usage. the first locally developed survey was distributed to the institutions’ students in 2016. for the 2018 student survey, aac reviewed the 2016 student survey questions and findings, and revised some scales (converting dichotomous to interval scales) and questions focusing on five areas: (1) frequency of in-person or online library visits; (2) frequency of library resource use; (3) what the library is doing well; (4) identifying areas to improve students’ experiences; and (5) how the library might contribute to student success (see the appendix for the survey questions). to address the research questions for this study, four variables were selected from the 2018 student survey questions: multiple choice questions related to student library visits, either in-person or online; use of resources; student library space satisfaction; and an open-ended question regarding students’ perceptions of the library’s impact on their learning. in both 2016 and 2018, the surveys were piloted with six to eight students who read the questions and shared their thoughts aloud to allow the research team to observe if there were any issues of interpretation with the questions. adjustments to the questions were made accordingly. to avoid any confusion of the library terms, we also provided links to these within the survey. to measure the library’s impact on students’ academic achievements, students’ cumulative gpas were provided by the office of institutional research.   quantitative data   student library visits either in person or online   the frequency of student library visits, either in-person or online, was assessed by the following question: “last semester, how often did you visit the library (1) in-person and (2) online.” the responses were coded as: 0 (never), 1 (once a month), 2 (once a week), 3 (multiple days in a week), and 4 (daily).   student library resource use   students’ use of library resources was measured by 11 items through the following question: “last semester, how often did you use each of the library resources? (1) journal articles, (2) subject specific databases, (3) print books from the stacks, (4) textbooks on reserve, (5) electronic books, (6) library subject and course guides, (7) special collections and university archives, (8) digital images, (9) streaming media, (10) dvds on reserve, and (11) patient care tools.” the answers were coded as: 0 (never), 1 (once a month), 2 (once a week), 3 (multiple days in a week), and 4 (daily).   student library space satisfaction   to measure the extent that students are satisfied with library spaces, the four items were included in the following question: “how satisfied are you with the library spaces below at your library? (1) quiet study spaces, (2) collaborative study spaces, (3) group study rooms, and (4) computer areas.” the answer was coded as: 0 (i don’t use this space in the library), 1 (very dissatisfied), 2 (dissatisfied), 3 (satisfied), and 4 (very satisfied).   student grade point average (gpa)   students’ cumulative gpas were taken from fall 2017 institutional records. the cumulative gpa refers to overall gpa, including all of the grades students earned from the beginning of the program to the end of the term. the range of gpas was from 0.00 to 4.00.   qualitative data   students’ perceptions of the library’s impact on their learning was assessed with the following open-ended question: “thinking about your overall library experience at the university, please tell us about your experiences with the library that positively impacted your coursework or research.”   participants   quantitative data   in spring 2018, a total of 28,725 undergraduate and graduate students were invited to participate in an online survey. a total of 2,277 students completed the multiple-choice survey for an overall response rate of 8%. while students across five library locations participated in the survey, for this study we did not closely examine responses by location. the majority of the respondents (about 92%) were enrolled in a program located in a large urban location, where there are two libraries. because students were not asked which library they used, it is not possible to definitively determine which location the students visited. while assumptions could be made that students enrolled in the health sciences programs used the library of health sciences, some of these programs are located geographically closer to the other library. students might choose a library based on reasons outside of their program, such as proximity to their home. it was also clear based on the open-ended questions that some students were familiar with both urban library locations. all of those students would have used the same university library website and had access to the same virtual resources.   as shown in table 1, it appears that the ratio of respondents from the student survey was similar to the ratio of the university population (a difference of less than 5%) in all of the demographic variables, except for two categories: female respondents (approximately 11%) and graduate students (about 9%) were overrepresented in the survey. we can claim that the survey respondents were representative of the university’s population. undergraduate students made up 57% of the total, and 86% were commuters. with respect to the range of students’ ages, 69.4% of respondents were between 16 and 25, followed by respondents between 26 and 35 (22.4%), and over 35 years old (8.2%). with respect to students’ ethnicity, 36.4% of the respondents were white, followed by hispanic (21.3%), asian american (18.6%), international (13%), and african american (7.2%).   table 1 students’ demographic information: sample from quantitative and qualitative data and university population note: in this institution, “international status” was included in the race/ethnicity category. “international” refers to all international students, regardless of their race or ethnicity, who were not born in the u.s. and are not permanent residents or u.s. citizens.   qualitative data   while 2,277 students completed the multiple-choice survey, a total of 992 students completed the open-ended question at the very end. as with the quantitative data, the qualitative data sample for the student survey is representative of the university population, except for gender and class: female students (more than 12%) and graduate students (8%) were overrepresented in the qualitative data sample (see table 1). with respect to students’ ethnicity, 38% of the respondents were white, followed by hispanic (21.7%), asian american (17.1%), international (11.4%), and african american (7.6%). students’ demographic information is displayed in detail in table 1.    procedures   this study was approved by the institutional review board at the institution. the list of potential students, including students’ demographic information and their cumulative gpas, was obtained from the office of institutional research (oir). the demographic information requested from the oir for this study includes gender, race/ethnicity, class standing, academic program, campus location, commuter/resident/online, age, and first-generation (table 1). all of the university students’ email addresses and the data listed above, including gpas, were uploaded as a “panel” in qualtrics (2018 version). the email address and the data including gpa of each student were linked in qualtrics to a unique url created by qualtrics. after distributing the survey, qualtrics was able to track who responded to the survey and provided de-identified survey responses as well as the demographic data and gpas for only those students who completed the survey. the survey was administered during the spring semester of 2018 and the survey reminder email, which included a drawing to win an ipad, was sent to students four times in order to increase the number of responses.   data analysis   quantitative data analysis   quantitative data was analyzed using quantitative data analysis software spss 25. prior to analyzing correlations, descriptive statistics using frequencies were used to demonstrate how responses were distributed. the median was also used for the variables (library visit, library resource use, and library space satisfaction) and organized by class level in order to demonstrate where the center of the data is located. the median is appropriate for data that has the character of an ordinal scale, and is generally used for skewed number distribution—that is, the majority of scores tend to accumulate either at the high or low end of the distribution (king & minium, 2007). to examine whether the library visits (in-person or online), library resource use, and library space satisfaction were associated with students’ gpas, a two-tailed spearman’s rank correlation was used. given that most of the variables, except for students’ gpas, are ordinal scale (e.g., from 0 [never] to 4 [daily]), the spearman’s rank correlation coefficient, also called spearman rho (coefficient) is appropriate. this type of correlation is used when testing non-linear correlations and measuring the strength of the relationship between variables on a scale that is at least ordinal (gust & d’journo, 2015; king & minium, 2007; pallant, 2011). spearman rank correlation is commonly used in the field of psychology research and medical literature (pallant, 2011).   qualitative data analysis   prior to analyzing the data, students’ responses were screened to verify that their answers were complete. during this process, 14 responses were excluded because their responses were recorded as “n/a.” a total of 992 responses were reviewed several times. qualitative thematic analysis was employed using qualitative data analysis software (nvivo 12) in order to have an in-depth understanding of the impact of the academic library on students’ learning. thematic analysis is a method for identifying, analyzing, describing, organizing, and reporting themes that emerge from a data set, and is widely used for a qualitative research method in epistemology and psychology (braun & clarke, 2006). codes were developed mainly from guided research questions (provisional coding) and, at the same time, they emerged from respondents’ quotations (inductive coding). after initial coding, codes were reviewed and revised by the first author. afterwards, the pattern coding technique was utilized in order to condense the existing codes into a small number of themes (creswell, 2009; miles, huberman, & saldaña, 2014). themes for coding were organized by the main question, “tell us about your experiences with the library that positively impacted your coursework or research.” the final codes and themes were reviewed by the second author, who designed the student survey. at this stage, no codes and themes were changed.   results   quantitative findings   this study aims to examine whether students’ library visits, library resource use, and library space satisfaction are related to their academic achievements. table 2 shows descriptive statistics for each measure of student library visits and student library resource usage. as shown in table 2, about half of the students responded that they visited the library “multiple days in a week” or “daily,” whereas 33% of the students replied that they visited the library “once a month” or “never.” on the other hand, only 24% of the students responded that they used the online library “multiple days in a week” or “daily,” whereas about 45% of the students replied that they used the online library “once a month” or “never.”   table 2 quantitative data: descriptive statistics on students’ library visits and library resource use the frequency of students’ library resource use is also displayed in table 2. the most common response for students’ use of any type of library resource fell into “never.” the top five resources that were marked as “never”: dvds on reserve were ranked first for “never” (91.1%), patient care tools were ranked second (88%), followed by digital images (83%), special collections and university archives (82.7%), and streaming media (81.7%). the patterns of library resource use for students who replied “once a month” to “daily” were consistent. the top three resources that were used “once a month” or more include journal articles, subject specific databases, and electronic books.   with respect to student library space satisfaction (quiet and collaborative study spaces, group study rooms, and computer areas), quiet study room satisfaction (satisfied and very satisfied) was the highest (69.6%), whereas group study room satisfaction was the lowest (44.6%) (see figure 1). among library spaces, group study rooms were ranked the least used (31.8%).   figure 1 students’ library space satisfaction.   taken together, students’ library usage (students’ library visits, library resource use, and space satisfaction) are further shown in table 3 using median values organized by class level. as shown in table 3, the median values of undergraduate students’ library visits in person and their online library use were 3 (multiple days in a week) and 1 (once a month). however, the pattern of library visits for doctoral students was the opposite: the median values of doctoral students’ library visits in person and their online library use were 1 (once a month) and 3 (multiple days in a week). in terms of student library resource use, the results showed that the median values of undergraduate students using both journal articles and databases were 1 (once a month). however, the median values of doctoral students using journal articles and databases were 3 (multiple days in a week) and 2 (once a week). regarding student library space satisfaction, the median values were the same across class level, except for doctoral students. specifically, the median values of doctoral students’ satisfaction with collaborative spaces and group study rooms were 0 (i don’t use this space in the library).   table 3 quantitative data: median values for students’ library visits, library resource use, and library space satisfaction, by class level note: variables for library visit and library resource use are ordinal scale: 0 (never), 1 (once a month), 2 (once a week), 3 (multiple days in a week), and 4 (daily). the library space satisfaction variable is ordinal scale: 0 (i don’t use this space in the library), 1 (very dissatisfied), 2 (dissatisfied), 3 (satisfied), and 4 (very satisfied).   we further examined the relationships between students’ library visits, library resource use, library space satisfaction, and their gpas. a spearman’s rank correlation was employed, and the correlation matrix is shown in table 4. the results indicate that student library visits in person (rs[2,086] = -.24, p < .01) and online (rs[1,794] = .27, p < .01) were associated with their gpa. however, when looking closely at the directions of the relationships, the results show that students’ in-person library visits were negatively associated with their gpas, whereas a positive relationship was observed between students’ gpas and their online library use. in other words, the higher the number of student library visits in person, the lower the student’s gpa. on the other hand, the higher the student’s online use, the higher their gpa.   the results of the relationship between library resources and gpa revealed that all of the library resources, except for dvds on reserve (rs [2,062] = -.03, p =.126), were correlated with students’ gpas (see table 4). most of the library resources (e.g., journal articles, databases, print books, electronic books, subject course guides, and special collections) are positively associated with student gpas, suggesting that as a student’s gpa increases, their use of resources such as journal articles and databases also increases. on the other hand, some resources (e.g., textbooks on reserve, digital images, streaming media, and patient care tools) were negatively associated with student gpas, indicating that as a student’s gpa increases, their use of some resources such as digital images and streaming media decreases.   in terms of the relationship between students’ library space satisfaction and their gpas, as shown in table 4, the results revealed that students’ satisfaction with all library spaces (quiet study spaces, rs [1,896] = -.11; collaborative study spaces, rs[1,893] = -.11; group study rooms, rs[1,889] = -.11; and computer areas, rs[1,889] = -.09; all p’s < .01), were negatively correlated with students’ gpas. this finding suggests that as a student’s gpa increases, their library space satisfaction decreases.   while most of the variables (students’ library visits, resources use, and library space satisfaction) were found to be statistically correlated with students’ gpas at either p < .05 or p < .01, the strengths of relationships across variables were weak (all rss < |.30|), as indicated by dancey and reidy (2011). the highest rs value is .29 for journal articles (see table 4).   table 4 quantitative data: correlations among student library visits, library resource use, library space satisfaction, and gpa * indicates p < .05.; ** indicates p < .01 qualitative findings   the second goal of the current study is to better understand how libraries have an impact on students’ learning from the students’ perspective. by utilizing an open-ended question, four overarching themes relating to students’ perceptions of the library’s impact on their learning emerged from the thematic analysis (table 5). those four themes include: (1) primary activities in the library; (2) choosing an appropriate space for academic and learning needs; (3) types of resources used for course work and research; and (4) library staff and librarians assisting students.   primary activities in the library   primary activities in the library represent the reasons why students visited the library. the majority of students responded that they went to the library for studying (n = 190), utilizing library resources for research (n = 111), or working on their coursework/homework (n = 77). the balance of the students used the library to prepare for an exam (n = 30), or work with their peers on group projects (n = 18).    choosing an appropriate space for academic and learning needs   a great number of students reported that they considered the library as a place where they can stay focused without distractions and get work done (n = 104). specifically, students commented that when they needed to concentrate, or had to finish coursework or a project, they utilized a quiet space.” one representative comment included the following: i spend time in the library to focus on homework and to study efficiently on the quiet floors. the quiet environment influences me to focus and get all of my work done (criminal law & justice, freshman, female).   of those students who perceived quiet spaces as an essential place for their learning, some responded that the use of the quiet space enabled them to increase their academic grades (n = 10). by way of example, one student commented:    my scores get lot more improved after studying in quiet library space (biological sciences, sophomore, male).   while many students reported that they were able to focus on their academic work and finish their work in the library quiet space, other students stated that they appreciated the library providing both a quiet space and a collaborative space (n = 26). in other words, students are likely to choose the appropriate library space depending on their academic needs, studying independently, or working with their peers. for example, a graduate student responded:   i love that both of the libraries have options for people who need all different kinds of environments to study (i.e. quiet, shared space, etc.). some days i need absolute silence, other days i like to energy of being around people talking. i recently learned about the study corrals, and i wish that was something i knew about in undergrad, though. as a graduate student, that would have been especially helpful to know (public health, graduate student, female).   in addition to the themes mentioned above, a few students reported that the library environment motivated them to work on their studies (n = 7).   table 5 qualitative data: themes and codes for students’ perceptions of the library’s impact on their learning   types of resources used for course work and research   students expressed that it is critical for them to access library resources in order to successfully complete their coursework and research. the resources students used for their coursework and research include journal articles (n = 140), books (n = 89), databases (n = 76), computers (n = 50), interlibrary loan, and reciprocal borrowing (n = 35). in many classes, students were asked to write papers using citations. students perceived accessing library resources as the key to completing their coursework or research. one student stated:   the online library databases and journals helped me find scholarly articles to aid in my research papers for my english and art history courses (graphic design, sophomore, female).     library staff and librarians assisting students   some students admitted that they have difficulty navigating resources in the library and online. when they encountered this challenge, they sought help from a librarian through a one-on-one consultation, online chat, and asking questions in person (n = 135). overall, students who received assistance from librarians and staff stated that they appreciated receiving immediate help from librarians and staff. in additional, students found it helpful when the librarians provided instructions on how to search for resources on the library website to use in their coursework and research. a master’s student acknowledged: the ability to meet with a librarian for individualized help was extremely helpful and we wouldn't have been able to successfully complete our projects without it (public health, masters student, female).   because the open-ended question was asking about the library’s “positive impact” on students’ learning from the students’ perspective, the themes from students’ responses mainly focused on the positive impact. while the majority of students responded with their positive perceptions of the library’s impact on their learning, some students expressed negative feedback on the library’s facilities. for example, some students wanted a microwave to reheat food, while others wanted more food vending machine options, and others wanted more quiet study spaces.   discussion   the first goal from the quantitative data was to learn how students’ library experiences (library visits, library resource use, and library space satisfaction) were associated with students’ gpas. the results revealed several important patterns. of interesting, the direction of the relationships between students’ library in-person visits and students’ gpas showed a negative correlation. that is, the frequency of a student’s library visits in person is negatively associated with student’s gpa. this finding is contrary to the earlier studies indicating that students’ library visits were positively associated with students’ academic success (brown & malenfant, 2018a; massengale et al., 2016). it is possible that students’ previous academic success may influence their intention to visit the library. in other words, students with a lower gpa may need a place where they can concentrate on studying, which results in an increase in their visits to the library. on the other hand, students with a higher gpa may not think it is necessary to go to the library because of their self-discipline. another possible explanation is that, due to the advantages of technology (e.g., easy access to internet), students may shift their library use from a physical place of study to a place for accessing resources online. however, it is not easy to clearly explain this negative relationship. given that this is merely a correlation, the results should not be interpreted that one necessarily causes the other. as such, further research is needed to investigate how this negative relationship occurs.   most of the students’ library resource use was shown to be positively associated with their gpas. this finding is consistent with previous studies indicating that library resource use (e.g., databases) was positively correlated with students’ gpas (brown & malenfant, 2018a; soria et al., 2013, 2017). it is important to note that some resources (textbooks on reserve, digital images, and streaming media) are negatively associated with students’ gpas. in the case of streaming media, it is possible that students with a low gpa need access to online tutorials in order to catch up on their studies, or the content of the streaming media may not be related to their course work or research, which results in the negative relationship. it will be useful to further investigate how specific resources are negatively associated with students’ gpas.   the results also revealed that there are correlations between all types of library space satisfaction and students’ gpas. however, it is important to note that the directions of relationships were negative. that is, the lower a student’s gpa, the higher their library space satisfaction. given that there was a negative relationship between students’ library in-person visits and their gpas in the current study, this finding can be accounted for by the same explanation. students with lower gpas are likely to use the library spaces and are satisfied with the library spaces. again, this correlation does not indicate causation; further research is required to examine the relationship between students’ library space satisfaction and their gpas.   while the strengths of the correlations across all variables were weak, the interpretation of the strengths of correlations should be made with caution. for example, the rs value for journal articles is .29. while this may appear weak, the r square value for journal articles is .08, meaning that journal use accounted for 8% of variability in gpa. given all the potential variables that could impact gpa, it is not insignificant to see the potential impact of journal article use on gpa. use of databases accounted for 4% variability with gpa. again, with many factors likely impacting gpa, there is a potential for a lower grade without use of the library and its resources. on the other hand, some rs values were less than |.10| (e.g., special collections, digital images, steaming media, and patient care tools), meaning that the variability related to the use of these and gpa is very small. this is not surprising for some library resources like special collections. it is likely that only students in specific programs would use some of these resources, such as special collections or patient care tools. for these, it might be more meaningful to look at their use and relationship with students’ gpas within specific programs. in spite of the findings of the weak relationships in the current study, it is important to note that the findings of the correlations (e.g., resource use) were supported by the literature discussed above (allison, 2015; soria et al., 2013, 2017). weak correlations between undergraduate students’ library use (e.g., checkouts and databases) and gpas also were found in allison’s study.   through quantitative data, the results provided evidence that the library has an impact on students’ academic achievements. however, it is not clear whether the library’s measurement of the relationship between students’ library experiences and their academic achievements are aligned with students’ perceptions of the library’s impact on their learning. utilizing qualitative data provided a deeper understanding of how students perceived the impact of library use on their learning and, furthermore, the association between students’ library use and their learning outcomes.   findings from the qualitative data uncovered how the library spaces and library resources impact students’ learning. the key findings related to library spaces showed that students were able to increase their concentration and productivity in a quiet space, and they selectively chose the library space depending on their learning needs, studying independently or working with their peers. this finding provides evidence that at least some of the students from the study institution found that library spaces promote students’ learning behaviours within that space. additionally, providing both quiet and collaborative spaces in the library allows students to select the space they need to achieve their academic goals. in particular, students choosing and utilizing both kinds of spaces (quiet and collaborative spaces) provides important evidence that the university library promotes students’ choice of learning venue. this finding is supported by the previous articles which argue how library space plays a role, which impacts students’ learning. that is, the library is regarded as a place that nurtures both “self-directed learning and the creation of new knowledge” (nitecki, 2011, p. 31), as well as “social learning” by providing an environment where students talk and collaborate (montgomery, 2014, p. 72). these findings help library staff make decisions on how best to use library spaces to meet students’ needs.   the qualitative findings further revealed which library resources have a positive impact on students’ learning from their perspective. students reported that they valued library resources (journal articles, books, and databases) and believed that those library resources have a positive impact on their coursework and research. students commented that interlibrary loan and reciprocal borrowing are critical for them to complete their coursework and research. moreover, students mentioned that library instruction and research consultations (learning how to search references and navigate the library website), as well as library staff’s support when looking for assistance either in person or online, were valuable resources in order to successfully complete their course work and research. this finding is also supported by the results of the quantitative data indicating that students’ library resource use was positively associated with their gpas. other students have also reported similar findings related to the positive impact that library resources and services (e.g., database use, interlibrary loan, library instruction, research consultations) may have to promote students’ learning (allison, 2015; brown & malenfant, 2018a, 2018b; gaha, hinnefeld & pellegrino, 2018; soria et al., 2013, 2017).   taken together, while the results from the quantitative and qualitative data may look different, in fact, the findings from the qualitative data complement the findings of the quantitative data. the quantitative data results showed that students’ library visits in person and library space satisfaction were negatively correlated with their gpas. however, the findings from the qualitative data revealed that students used the library for studying and preparing for exams. in addition, findings from the qualitative data indicated that respondents considered library space as a valuable space to accomplish their academic goals. therefore, we speculate that students with a lower gpa tend to use the library to study and prepare for exams in order to improve their gpa, and those with a lower gpa are likely to be satisfied with the library spaces. admittedly, the main focus of the current study weighs toward the findings from the quantitative data, and the open-ended question from the qualitative data are also largely presented from a quantitative perspective. nevertheless, when looking at these findings together, the findings from the qualitative data may provide a better understanding of the results of the quantitative data. while we cannot confirm the causation without knowing students’ previous gpas, it is worth noting that through the qualitative data at least we are able to better understand why and how students used the physical library, and their perceptions of library use for their learning.   limitations and future directions   a significant contribution to this study is the use of the survey developed by the university library staff in an effort to measure the impact of the library on university students’ academic success. in spite of this contribution, there are some limitations to be addressed in this study. the open-ended question, “…please tell us about your experiences with the library that positively impacted your coursework or research,” asked only about the positive impact on students’ coursework or research. how a question is written might produce biased results. however, while it was written to produce positive comments, based on the comments it would appear that those who wanted to share negative feedback did so. while we do intend to use this question again in a future survey because we are interested specifically in understanding how students view the library in contributing to their success, we may add another open-ended question asking them to let us know whether there is anything additional they wish to share to provide the opportunity to give other feedback or comments, including negative ones. another limitation of the current study is that it only focused on analyzing the relationship between students’ overall library experience and their gpas, rather than examining other factors such as degree sought, because the correlation of students’ library use and gpas may be influenced by those factors. future research is needed to further examine whether there are correlations with other factors (e.g., class level or degree sought). nevertheless, the finding of the negative relationship between in-person library visits, library space satisfaction, and students’ gpas needs further investigation. the university library has already begun designing a follow up study to further explore this finding by using the qualitative approach.   conclusion   the purpose of the current study was to examine whether students’ library experiences (student library visits, library resource use, and library space satisfaction) were associated with students’ gpas, and explore how students perceived the library’s impact on their learning. the current study provided evidence that, overall, the student library experience was associated with their academic achievement even though the strength of correlations was weak. when looking closely at the directions of the associations between students’ library experiences and their gpas, there were mixed findings: students’ library in-person visits and library space satisfaction were negatively associated with their gpas, whereas most students’ library resource usage was positively associated with their gpas. the qualitative analysis demonstrated students’ perspectives about how they benefited from using the university library. the findings indicate that students’ primary activity in the library was studying. furthermore, students utilized both quiet and collective study spaces, depending on their learning purpose, and valued library resources (e.g., journal articles and interlibrary loan) for their coursework and research. last, students appreciated the assistance provided by the librarians and library staff with explaining how best to utilize library resources.     acknowledgements   our thanks to the university of illinois at chicago library assessment advisory committee for the development of the surveys, and our colleagues paula dempsey and glenda insua for their helpful internal review of this article.   references   allison, d. (2015). measuring the academic impact of libraries. portal: libraries and the academy, 15(1), 29-40. https://doi.org/10.1353/pla.2015.0001   braun, v., & clarke, v. (2006). using thematic analysis in psychology. qualitative research in psychology, 3, 77-101. https://doi.org/10.1191/1478088706qp063oa   brown, k. (2018). evidence of academic library impact on student learning and success: advancing library leadership and advocacy with assessment in action. in k. e. brown, d. l. gilchrist, s. goek, l. j. hinchliffe, k. j. malenfant, c. ollis, & a. payne (eds.), shaping the campus conversation on student learning and experience: activating the results of assessment in action (pp. 9-22). chicago, il: association of college & research libraries.   brown, k., & malenfant, k. j. (2018a). documented library contributions to student learning and success: building evidence with team-based assessment in action campus projects. in k. e. brown, d. l. gilchrist, s. goek, l. j. hinchliffe, k. j. malenfant, c. ollis, & a. payne (eds.), shaping the campus conversation on student learning and experience: activating the results of assessment in action (pp.79-104). chicago, il: association of college & research libraries.   brown, k., & malenfant, k. j. (2018b). academic library impact on student learning and success: findings from assessment in action team projects. in k. e. brown, d. l. gilchrist, s. goek, l. j. hinchliffe, k. j. malenfant, c. ollis, & a. payne (eds.), shaping the campus conversation on student learning and experience: activating the results of assessment in action (pp. 105-132). chicago, il: association of college & research libraries.   creswell, j. w. (2009). research design: qualitative, quantitative, and mixed methods approaches (3rd ed.). thousand oaks, ca: sage publications.   dancey, c. p., & reidy, j. (2011). statistics without maths for psychology (5th ed). prentice hall/pearson, harlow, england; new york.   gaha, u., hinnefeld, s., & pellegrino, c. (2018). the academic library’s contribution to student success: library instruction and gpa. college & research libraries, 79(6), 737-746. https://doi.org/10.5860/crl.79.6.737   gust, l., & d’journo, x. b. (2015). the use of correlation functions in thoracic surgery research. journal of thoracic disease, 7(3), e11-e15. https://doi.org/10.3978/j.issn.2072-1439.2015.01.54   holland, j. l., & christian, l. m. (2009). the influence of topic interest and interactive probing on responses to open-ended questions in web surveys. social science computer review, 27(2), 196-212. https://doi.org/10.1177/0894439308327481   jackson, k. m., & trochim, w. m. k. (2002). concept mapping as an alternative approach for the analysis of open-ended survey responses. organizational research methods, 5(4), 307-336. https://doi.org/10.1177/109442802237114   king, b. m., & minium, e. w. (2007). statistical reasoning in the behavioral sciences (5th ed.). hoboken, nj: john wiley & sons, inc.   massengale, l., piotrowski, p., & savage, d. (2016). identifying and articulating library connections to student success. college & research libraries, 77(2), 227-235. https://doi.org/10.5860/crl.77.2.227   may, f., & swabey, a. (2015). using and experiencing the academic library: a multisite observational study of space and place. college & research libraries, 76(6), 771-795. https://doi.org/10.5860/crl.76.6.771   miles, m. b., huberman, a. m., & saldaña, j. (2014). qualitative data analysis: a methods sourcebook (3rd ed.). thousand oaks, ca: sage publications, inc.   montgomery, s. e. (2014). library space assessment: user learning behaviors in the library. journal of academic librarianship, 40(1), 70-75. https://doi.org/10.1016/j.acalib.2013.11.003   nitecki, d. a. (2011). space assessment as a venue for defining the academic library. library quarterly, 81(1), 27-59.  https://doi.org/10.1086/657446   oakleaf, m. (2010). the value of academic libraries: a comprehensive research review and report. chicago, il: association of college and research libraries.   oakleaf, m. (2016). getting ready & getting started: academic librarian involvement in institutional learning analytics initiatives. journal of academic librarianship, 42(4), 472-475. https://doi.org/10.1016/j.acalib.2016.05.013   oakleaf, m., whyte, a., lynema, e., & brown, m. (2017). academic libraries & institutional learning analytics: one path to integration. journal of academic librarianship, 43(5), 454-461. https://doi.org/10.1016/j.acalib.2017.08.008   pallant, j. f. (2011) spss survival manual: a step by step guide to data analysis using the spss program (4th ed.). crows nest, nsw: allen & unwin book publishers.   soria, k. m., fransen, j., & nackerud, s. (2013). library use and undergraduate student outcomes: new evidence for students’ retention and academic success. portal: libraries and the academy, 13(2), 147-164. https://doi.org/10.1353/pla.2013.0010   soria, k. m., fransen, j., & nackerud, s. (2017). beyond books: the extended academic benefits of library use for first-year college students. college & research libraries, 78(1), 8-22. https://doi.org/10.5860/crl.v78i1.16564     appendix student experience survey   i have read the “agreement to participate” document and agree to participate in this research.   o  yes o  no   last semester, how often did you visit the university library?     daily multiple days in a week once a week once a month never in person o    o    o    o    o    online o    o    o    o    o      how satisfied are you with the library spaces below at your library?     very satisfied satisfied dissatisfied very dissatisfied i don’t use this space in the library quiet study spaces o    o    o    o    o    collaborative study spaces o    o    o    o    o    group study rooms o    o    o    o    o    computer areas o    o    o    o    o      if you study in places other than the university library, what do you like about those spaces? [check all that apply]   o  more study space o  quieter study space o  food/drink availability o  software availability o  equipment (e.g., computer, printer, scanner, etc.) availability o  longer hours o  more comfortable furniture o  i can find a seat o  i prefer to study at home o  other (please specify) [                                 ]   last semester, how often did you use each of the library resources below?     daily multiple days in a week once a week once a month never journal articles o    o    o    o    o    subject specific databases o    o    o    o    o    print books from the stacks o    o    o    o    o    textbooks on reserve o    o    o    o    o    electronic books o    o    o    o    o    library subject & course guides o    o    o    o    o    special collections & university archives o    o    o    o    o    digital images o    o    o    o    o    streaming media o    o    o    o    o    dvds on reserve o    o    o    o    o    patient care tools o    o    o    o    o      please indicate the relative importance of each of the library resources/services for your research or coursework.     very important important somewhat important not at all important i don’t use this tool/service journal articles o    o    o    o    o    subject specific databases o    o    o    o    o    print books from the stacks o    o    o    o    o    textbooks on reserve o    o    o    o    o    electronic books o    o    o    o    o    library subject & course guides o    o    o    o    o    special collections & university archives o    o    o    o    o    digital images o    o    o    o    o    streaming media o    o    o    o    o    dvds on reserve o    o    o    o    o    patient care tools o    o    o    o    o    library instruction arranged by your professor o    o    o    o    o    library workshops that you self-selected to attend o    o    o    o    o    other (please specify) o    o    o    o    o      how easy is to use the university library website for the services below?     very easy easy difficult very difficult i don’t use this service finding journal articles using the search box on the library home page o    o    o    o    o    finding an e-book using the search box on the library home page o    o    o    o    o    finding a print book using the search box on the library home page o    o    o    o    o    accessing a database to search for articles and other scholarly materials o    o    o    o    o    requesting a print book from another library o    o    o    o    o    requesting an article from another library o    o    o    o    o    logging into my library account to renew a book o    o    o    o    o    asking for help from a librarian by im/chat o    o    o    o    o    using library subject & course guides to access materials by subject o    o    o    o    o    finding media (e.g., films, videos, online images, etc.) o    o    o    o    o    booking a group study room online o    o    o    o    o    other (please specify) o    o    o    o    o      how likely are you to recommend the following library services to another student?     very likely likely unlikely very unlikely i don’t use this service one on one research consultation with a librarian o    o    o    o    o    library workshops about library research (e.g., finding resources, requesting materials, etc.) o    o    o    o    o    im/online chat research help o    o    o    o    o    e-mail research help o    o    o    o    o      suppose you had funding to improve the university library. please select up to three of your most important priorities from the list below.   o  access to more online journals o  access to more books (e-books, print books, textbooks) o  more computers o  more quiet study space o  more group study space o  more electronical outlets o  more white boards o  more drink/food options o  additional comfortable furniture o  other (please list) [                              ]   think about your overall library experience at the institution, please tell us about your experiences with the library that positively impacted your coursework or research.                     evidence based library and information practice evidence based library and information practice 2009, 4:3  evidence based library and information practice       news    the library and information research group announces a new research prize      © 2009. this is an open access article distributed under the terms of the creative commons attribution   license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and   reproduction in any medium, provided the original work is properly cited.      for u.k. librarians, the library and  information research group (lirg) has  recently announced:    a new research prize for £2,500,  sponsored by ex libris, for library  systems‐based research.   a new lirg researcher directory  listing library research practitioners  in the u.k. the directory is  keyword searchable and can be  browsed by research topic.  researchers can enter their own  name, under multiple topics if need  be, and are encouraged to do so if  they have a work‐based interest in  research.  workshops on writing a research  proposal are currently being  arranged in three u.k. cities for  autumn 2009 (london, birmingham  and glasgow).    further information is available on the lirg  website:  http://www.cilip.org.u.k./specialinterestgro ups/bysubject/research/activities   90 http://www.cilip.org.uk/specialinterestgroups/bysubject/research/publications http://www.cilip.org.uk/specialinterestgroups/bysubject/research/activities http://www.cilip.org.uk/specialinterestgroups/bysubject/research/activities evidence summary   mixed method study examines undergraduate student researchers’ knowledge and perceptions about scholarly communication practices   a review of: riehle, c. f., & hensley, m. k. (2017). what do undergraduate students know about scholarly communication?: a mixed methods study. portal: libraries and the academy, 17(1), 145–178. http://dx.doi.org/10.1353/pla.2017.0009    reviewed by: melissa goertzen collection development analysis & support librarian columbia university libraries new york, new york, united states of america email: mjg2227@columbia.edu   received: 1 june 2017    accepted: 25 july 2017      2017 goertzen. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to examine undergraduate student researchers’ perception and understanding of scholarly communication practices and issues.   design – mixed method study involving a survey and semi-structured interviews.   setting – two major undergraduate universities in the midwest region of the united states of america.   subjects – undergraduate students who participated in or had completed undergraduate research experiences with faculty mentors.   method – the method was first approved by institutional review board offices at both campuses involved in the study. then, students received invitations to participate in a survey via email (campus 1 = 221 students; campus 2 = 345 students). identical online surveys ran separately on each campus; both remained open for a period of three weeks. all respondents received a reminder email one week before the survey closed.   participants answered 12 questions related to demographics and scholarly communication practices. the survey examined knowledge and experience across five areas: the peer review process, author and publisher rights, publication and access models, impact of research, and data management. all students who completed the survey were entered in a drawing for a $50 amazon card. the response rates were 34.8% (campus 1) and 18.6% (campus 2).   surveys on both campuses were administered using different software: campus 1 utilized qualtrics survey software while campus 2 used an institution-specific survey software. data sets were normed and merged later in the study to enable comparison and identify broad themes.   survey respondents were also invited to participate in a 15 to 20 minute follow-up interview and were compensated with a $20 amazon gift card. the interviews consisted of four open-ended questions that further examined students’ knowledge of scholarly communication practices. the researchers coded interview transcripts and identified themes. qualitative software was used to analyze the surveys and assess coder agreement. finally, connections and anomalies between survey and interview results were explored.   main results – quantitative and qualitative data collected during the study indicate that students were most confident in their understanding of the peer-review process and data management but felt less confident in their knowledge of author and publisher rights, publication and access models, and determining the impact of scholarly research publication. in addition, they value instruction related to scholarly communication topics like the peer-review process, publication models, and data management. however, few students feel confident in their current level of knowledge or ability surrounding the previously mentioned topics. study findings suggest that this knowledge gap is based on a lack of training or discussion of scholarly communication topics in relation to students’ research activities.   results also suggest that undergraduate students have difficulty articulating their rights as authors and their scholarly communication practices. in many cases, skill sets like data management are learned through trial and error while students progress through the research process. in some cases, faculty mentors have misperceptions and assumptions about undergraduate students’ knowledge and abilities regarding scholarly communication practices. this can create challenges for undergraduate students as they attempt to make informed decisions about research activities based on a limited foundation of experience or information.   finally, results indicate that undergraduate student researchers do not currently view the library as a place to learn about scholarly communication practices. the authors suggest that by forming strategic relationships with undergraduate research program directors, faculty, and graduate student mentors, librarians are in a prime position to incorporate scholarly communication practices into information literacy sessions or provide point-of-need coaching.   conclusion – the researchers conclude that academic libraries are in a unique position to support overarching research, teaching, and learning goals within the academic community. by developing programs that support information literacy and scholarly communication, libraries demonstrate value and align goals with teaching and learning priorities within the higher education community as a whole. through this work, librarians support students as knowledge creators and advocate for training that emphasizes data literacy, copyright and authors’ rights, and the impact of research within specific disciplines.   commentary   within the academic community, undergraduate student researchers are gaining credit as knowledge creators. in response, information professionals are transforming information literacy programs to include scholarly communication practices. the study at hand supports what was previously known—that students have a greater role in the creation of knowledge (acrl, 2013) and that traditional information literacy training provides guidance on scholarly communication practices (hensley, 2015) – and includes a unique perspective that brings value to the information profession. that perspective is an exploration of students’ perception and understanding of scholarly communication.   the researchers suggest that librarians support “undergraduate students as they move beyond their role as knowledge consumers, encouraging them to become skillful knowledge creators” (p. 146). to achieve this goal, they designed a mixed methods study that documents students’ application of scholarly communication practices. detailed descriptions of the research design, recruitment methods, and data analysis techniques provide a detailed roadmap that can be adopted by other librarians who wish to conduct similar investigations.   the paper was enhanced by the organization and presentation of results. it was interesting to read about the methodology and results of the survey and interview sessions separately, followed by a discussion of observed trends across the complete body of quantitative and qualitative data collected during the study. the conclusions offered viable strategies to merge information literacy sessions with scholarly communication training. for instance, the researchers stress the importance of strategic relationships between the library and undergraduate program directors to facilitate mentorship and training opportunities.   since the participant group largely conducted research in scientific fields, it would be interesting to discover if observed trends apply to students working in the humanities or social sciences. with that being said, the researchers note this limitation and suggest it as an area for future study. additionally, the researchers mentioned that participants had only experienced small portions of the research process and did not have a chance to see the “big picture” of research projects. running the study again when participants are involved in the full research process would be useful.    because the value of the study rests on the methodology and ability to document local gaps in knowledge or training, it was unfortunate that the research tools (i.e., survey and interview questions) were not included as an appendix to the study. these tools may have served as a starting point for similar studies within the information management profession.    the paper successfully demonstrates how a mixed methods study can be used to understand students’ perception and knowledge of scholarly communication practices. repeating the study in several years could help to determine how the establishment of strategic relationships between the library and the campus community influences students’ skill sets and comfort level with scholarly communication practices.   references   association of college and research libraries (acrl). (2013). intersections of scholarly communication and information literacy. working group on intersections of scholarly communication and information literacy. retrieved from http://acrl.ala.org/intersections/   hensley, m. k. (2015). a survey of instructional support for undergraduate research programs. libraries and the academy, 15(4), 719–62. evidence summary   national differences in perceived benefit of libraries may be due to their investments in libraries, library supply, and cultural factors   a review of: vakkari, p., aabø, s., audunson, r., huysmans, f, kwon, n., oomes, m., & sin, s. (2016). patterns of perceived public library outcomes in five countries. journal of documentation, 72(2), 342–361. http://dx.doi:org/10.1108/jd-08-2015-0103   reviewed by: ann glusker reference/consumer health librarian business, science and technology department the seattle public library seattle, washington, united states of america email: ann.glusker@spl.org   received: 1 sep. 2016     accepted: 19 oct. 2016      2016 glusker. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to compare citizens' perceptions of the benefits of libraries in five culturally diverse countries.   design – postal survey to a random stratified sample and web surveys (some with a sampling plan).   setting – surveys were administered in finland (by post), norway, the netherlands, the united states of america, and south korea (online).   subjects – selected or self-selected members of the general adult population in the specified countries who had used a public library within the past year.   methods – surveys were administered and data were collected in each of the five countries. a dependent variable representing perceived outcomes was calculated from 19 outcome measures (related to life experiences). within this, 4 indices were calculated from subsets of the 19 measures, relating to work, education, everyday activities, and leisure activities. five independent variables were used: frequency of library use, number of services used, gender, age, and education level. respondent country was also entered into analyses. descriptive statistics and analysis of covariance results were presented.   main results – it was noted that each country's sample was skewed in some way towards one or more of the variables of gender, age, and education, and some statistical corrections were employed. while patterns within countries are similar, library users from finland, the united states of america, and south korea reported higher levels of benefits overall. "fun in reading" and "self-education" were the two outcomes with the highest scores by respondents. higher numbers of visits and greater use of services may account for the higher perceived benefits in the three countries reporting them. in fact, these two factors appear to explain a substantial portion of the variance in perceptions of benefits between countries, meaning that between-country variation in library resources and supply plays a role in perception of benefit. there were varied rather than linear patterns of benefit reporting along age and education continua, with those at the lowest education levels deriving the most perceived benefits in all spheres. by gender, women derived fewer perceived benefits in the work sphere than men.   conclusions – there is variation across countries in the level of public library benefits reported, as well as variation across individual measures, creating different profiles of response by country. even when respondent demographic characteristics and library usage are controlled for, country differences remain. these may be explained by the differences in investment in – and hence supply of – libraries by country, types of investment (e.g., according to the authors, finland invests in services, norway in collections, and the usa in staffing), and cultural factors such as the propensity of usa respondents to have a more extreme response style. future research may profitably concentrate on policy contexts of libraries in each country. in the nineteenth century libraries provided social welfare services and in the twentieth they provided human rights through equitable access to information, so research should focus, by country, on what libraries will provide in the twenty-first century. future studies might also address how differences in demographic patterns among respondents play out in benefit perceptions between countries.   commentary   library valuation is an essential tool for political advocacy, as libraries are vulnerable to threats from budget reductions in times of financial downturn. in her much-cited meta-analysis, aabø (2009) showed that $1 invested in a public library brings 4 to 5 times that in societal returns, but library valuation research is moving from a focus on performance measures such as circulation to impact measures such as behavior change attributable to libraries (micka, 2013; streatfield, 2012). there is fertile ground for considering cross-cultural differences and similarities in perceptions of library benefits.    for this evidence summary, methodologies were systematically assessed using glynn’s critical appraisal checklist (2006). a wide range of issues undermines the usefulness of this work as an unbiased source of evidence about populations. important concerns include: only people who had used a library in the last year were sampled, rather than the general public, and in a way (in four of five countries) that may have excluded non-internet users; survey questions were not identical in each of the five countries; countries were chosen on the basis of interest from researchers from the nations studied (except the united states of america) without an advance strategy for representation, with choices defended after the fact; responses were collected over a wide time period; random stratification was used for respondent selection in only one of the countries; and samples showed skewedness on several dimensions, especially favoring more highly educated respondents. in addition, there was important information that the authors did not provide, including: the survey questionnaires; information about how the finnish survey was modified for use in other countries (so, it is unknown how concepts were understood across languages); the specifics about how web panels were recruited and selected in the four countries in which they were used; and information about informed consent and ethics approvals. the omission of the survey questionnaires is notable; the authors cite the work of harzing (2006) on cross-country differences in response styles, and it is hoped that they took her suggestions for employing careful instrument design, as "response bias is a serious threat to valid comparisons across countries" (p. 27). without the inclusion of the instruments, there is no way to tell.   the authors conclude that there are cross-country variations even when all other factors are controlled for, and surmise that these may be the result of differences in investments in libraries, supply of libraries and staff availability. if this study had been more rigorously constructed and hypothesis-driven, these findings would have more power and interest. as it is, they are inconclusive; it is to be hoped that as the authors pursue their future research agenda of examining policy contexts, that these issues will be addressed.   references   aabø, s. (2009). libraries and return on investment (roi): a meta-analysis. new library world, 110(7/8), 311-324. http://dx.doi.org/10.1108/03074800910975142   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi:org/10.1108/07378830610692154   harzing, a.-w. (2006). response styles in cross-national survey research: a 26-country study. retrieved from http://www.harzing.com/download/respstyles.pdf     micka, t. l. (2013). demonstrating the value of the public library: economic valuation and the advocacy imperative. student research journal, 3(1), 1-19. retrieved fromhttp://scholarworks.sjsu.edu/slissrj/vol3/iss1/4     streatfield, d. (2012). impact planning and assessment of public libraries: a country level perspective. performance measurement and metrics, 13(1), 8-14. http://dx.doi.org/10.1108/14678041211228535       research article   determining gate count reliability in a library setting   jeffrey phillips student success librarian robert manning strozier library florida state university tallahassee, fl, usa email: jbphillips@fsu.edu   received: 6 june 2016     accepted: 5 aug. 2016         2016 phillips. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – patron counts are a common form of measurement for library assessment. to develop accurate library statistics, it is necessary to determine any differences between various counting devices. a yearlong comparison between card reader turnstiles and laser gate counters in a university library sought to offer a standard percentage of variance and provide suggestions to increase the precision of counts.    methods – the collection of library exit counts identified the differences between turnstile and laser gate counter data. statistical software helped to eliminate any inaccuracies in the collection of turnstile data, allowing this data set to be the base for comparison. collection intervals were randomly determined and demonstrated periods of slow, average, and heavy traffic.    results – after analyzing 1,039,766 patron visits throughout a year, the final totals only showed a difference of .43% (.0043) between the two devices. the majority of collection periods did not exceed a difference of 3% between the counting instruments.   conclusion – turnstiles card readers and laser gate counters provide similar levels of reliability when measuring patron activity. each system has potential counting inaccuracies, but several methods exist to create more precise totals. turnstile card readers are capable of offering greater detail involving patron identity, but their high cost makes them inaccessible for libraries with lower budgets. this makes laser gate counters an affordable alternative for reliable patron counting in an academic library.   introduction   gate counts are a common tool for the assessment of libraries, correlating patron visits with the use of library facilities (hernon, dugan, matthews, & thornton, 2014). this form of analysis associates the value of a library with its popularity of in-person patronage, but requires a thorough collection of quantitative data for a true justification of expediency. libraries use various methods for counting patrons, including laser counters, turnstiles, or designated employees who physically count the individuals visiting the library. as libraries attempt to maximize funding, many have implemented counting devices in place of a dedicated census employee. these devices add numerous advantages beyond monetary frugality, including both accuracy and security.   with the significantly lower cost of theft-deterrent beam counters, many libraries purchase these devices for inventory security but remain concerned about the accuracy of their counting-ability. libraries who insist on more accurate counts may choose to implement both theft deterrent beam counters and turnstiles, with each device maintaining its individual purpose of either assessment or security. however, with the strict budget concerns that many libraries are facing, it is of interest to determine if the less expensive laser beam counters are also reliable assessment tools.   literature review   libraries offer numerous services outside of book collections, and gate counts are capable of showing the in-person usage of a library in its entirety (dotson & garris, 2008). although the increase of online resources causes concern about the viability of physical libraries (hiller, 2004), universities need library-like places for student interaction, peer learning, tutoring, collaboration, and other in-person functions (hurlbert, 2008). incorporating these services presents an opportunity to influence patrons to visit the library for reasons beyond the collection (hiller, 2004), and patron counts help determine if certain events or workshops lead to an increase library popularity.   by collecting information that tracks the habits, movements, and patterns of patrons, institutions can identify trends in traffic over both daily and weekly time frames (zhu, aghdasi, millar, & mitchell, 2014). this data promotes informed staffing decisions that efficiently match the amount of employees with the patron population. patron counts also help evaluate the effectiveness of outreach promotions and activities.   turnstiles are more mechanically reliable and provide better patron security when compared to laser counters (boss, 1999). these devices offer correct counts from contact-based functionality, necessitating a physical interaction between the user and machine (hashimoto, kawaguchi, matsueda, morinaka, & yoshiike, 1998). the design of the turnstile only permits the passage of one person at a time and counts remain more accurate and consistent by forcing the user to personally engage the machine’s counting mechanism. in addition to statistical reliability, unique card swipes also promote better security, forcing patrons to authenticate their identity to gain clearance through the machine.   infrared (horizontal) beam counters do not require physical contact to operate but instead calculate patron visits by counting the amount of times breaks occur in the laser beam. the beam transmits to a reflector across the desired path of measurement, and whenever the connection is broken, the counter records a new visitor. this is a popular method of counting but accuracy suffers when more than one person passes through the laser at the same time (riachi, karam, & greige, 2014). additionally, problems occur from obstructions (dotson & garris, 2008) and the inability to distinguish between objects and humans, thus mistakenly counting shopping carts, luggage, and other objects as patrons (kryjak & komorkiewicz, 2013). although concerns of counting reliability loom around the functionality of beam counters, their affordable price influences their popularity. these devices are capable of offering both counting and theft detection simultaneously, allowing libraries to use one device to fulfill multiple needs. theft deterrent gate systems suggest a reduction of loses from 70%-80%, which not only may pay for itself with two years, but also helps prevent the loss of high-demand literature from the collection (boss, 1999).   aims   the goal of this research is to promote the collection of accurate patron traffic counts in a library setting. this study compares over one million unique library exits from laser gate counters and card swipe turnstiles, revealing any dissimilarities in their totals. finalized results aim to define an average variation between both devices and offer approaches to enhance the precision of collecting patron counts.   methods   variations between turnstile and gate counter data were determined by inspecting library exit data in a large academic library. patrons were required to pass through both devices to exit the building, and each system individually counted the patron. users first crossed through a theft deterrent gate system that contained the laser counter. several feet afterwards, patrons approached the turnstiles and were required to swipe their official university identification card to exit the library. a wall and rope barricade discouraged patrons from altering the suggested pathway between counting devices and the entrance contained a separate group of turnstiles that patrons could not use to exit the library. the entrance turnstiles were located abreast of the exit turnstiles, prohibiting patrons altering the explicit traffic pattern.   the collection period consisted of 26 random intervals throughout 1 year, with dates ranging from 1 to 35 days. the frequency of these periods targeted dates that represented busy, average, and slow foot traffic periods. the computerized record of the individual swipe interactions determined the turnstile total, while the gate count numbers required a manual monitoring of a built-in digital gate counter. turnstile totals calculated the records of an entire day ending at midnight during the weekdays and 6pm during the weekends, which required a simultaneous visual confirmation of the laser counter total to guarantee a precise comparison. this influenced several of the collection dates, requiring the periods to correspond with the researcher’s availability. the original research design also focused on variances in collection intervals to determine if inaccuracies developed from specific days or patron counts, which encouraged sporadic collection periods.   to create a reliable comparison between the patron counters, it was necessary to first analyze the turnstile results and eliminate any errors in their collection. the turnstile totals consisted of all successful card swipes that occurred throughout the designated date range. however, these outcomes often included multiple successful swipe acknowledgements for the same person upon one exit.   each time the turnstiles encountered a sequence of simultaneous rapid card swipes, inaccuracies occurred. by default, all swipe processes contained a one second buffer but the results included any interactions that occurred from the same patron after the one second delay. even though the user received approval to exit through the turnstiles, these swipes also registered as unique patron exits in the total.   the use of ibm spss software corrected these miscalculations. sequentially organizing the exit logs allowed for the identification and elimination of additional successful swipes occurring from the same user within the same minute.   results   the total number of laser gate counts (n = 1,035,327) differed from the total number of turnstile swipes (n = 1,039,766) by -4,439, or -.43%. although the laser counter totals were often times greater than the turnstile counts, the extreme variance from the longest interval of days (interval 35) made the final tally of turnstiles exceed the total results of the laser counter.     table 1 collection of patron counts interval in days percentage difference actual turnstile count laser gate count 1 9.02% 474 521 1 6.95% 589 633 3 4.73% 1,471 1,544 5 4.73% 1,350 1,417 6 -0.58% 38,385 38,162 6 1.09% 20,344 20,569 7 2.59% 10,670 10,954 7 2.79% 18,989 19,533 7 1.33% 16,884 17,111 7 1.78% 28,582 29,101 7 -0.12% 9,710 9,698 7 2.73% 9,046 9,300 12 1.31% 40,642 41,182 14 1.83% 61,008 62,143 14 1.75% 57,974 59,006 19 23.22% 248 323 20 1.13% 78,024 78,918 21 1.62% 56,918 57,857 21 2.44% 22,975 23,549 21 3.72% 25,079 26,047 21 1.43% 80,867 82,037 22 1.42% 85,854 87,090 24 2.69% 10,437 10,725 28 0.50% 89,606 90,052 33 1.47% 164,359 166,803 35 -20.02% 109,281 (possible error) 91,052     as table 1 displays, the results often remained within 1% to 3% of one another. the most common period of data collection took place at seven days, in which the difference never exceeded 2.79% throughout all six collection periods. information gathered from a period of less than a week was the most inconsistent, ranging from 1.09% to 9.02%.   the results note a range of possibly distorted data due to an error in the gate counter’s functionality. interference caused the theft deterrent system to stop operating and required a system reboot to continue proper functionality. on the readout, an error code replaced the count listing, making it unknown if this error also affected the counting ability of the machine. this data period (interval 35) was grossly different from the other periods, and eliminating this information from the total would change the total difference by 2%, with the laser counters yielding a 1.5% higher result than the turnstiles.   figure 1 shows that the largest variances between the counters resulted from interval 19 (23.22%) and interval 35 (-20.02%). conversely, the difference of 23.22% was only comprised of 75 patrons, whereas the difference of -20.02% involved 18,229 users. the collection period with the largest quantity of patrons (n = 164,359) only showed a difference of 1.47% between both systems.     figure 1 percentage of difference between the counters and the interval of collection days     discussion   the single-user multiple-swipe theory appeared to be a significant factor in distorting finalized turnstile counts. to gain an accurate result of patron activity through the turnstile card readers it was necessary to export the turnstile totals into spss software. analyzing all sequential swipes from the same user concluded that the data consisted of 65,475 duplicated swipes, or -5.92% of the yearly turnstile total. the turnstile totals removed all instances of these duplicated entries before the comparison analysis began. future studies may determine the reasons for patrons to perform a rapid succession of swipes in the reader upon exit. for example, this behavior could be a result of swipe anxiety, a feeling of impatience, or psychological mimicry.   gate counters have difficulty providing an accurate assessment when multiple patrons exit in a staggered or side by side formation. this results in totals that are less than the turnstile count, with multiple patrons registering as a single person. however, the majority of collection periods had the beam counter producing a larger number of patron visits than the turnstiles. a possible factor contributing to this increase resulted from the theft deterrent feature of the gate system.   the theft deterrent gate system alarm notifies patrons with sensitized materials to return to the circulation desk for desensitization of those items. while the beam counter has processed a successful exit, the patron is required to return to the circulation desk before reaching the turnstiles. when returning to the circulation desk, the patron will cross through the theft deterrent gate system again, creating a second count for the same exiting patron. after desensitization of their materials at the desk, the patron returns through the beam counter for a third count of their same attempt to exit the library, finally the turnstile for the first time. in this situation, the turnstile count system would only register this as one event, whereas the beam counter assumes three separate exits have occurred.   whereas the largest discrepancy in data (interval 35) could be the result of an equipment error, a possible outlier occurred from an interval of 19 days, where the totals varied 23.22%. fortunately, this was also the period with the lowest total of patron visits, and failed to create a significant variance in the yearly total.   conclusion   turnstile card readers and laser gate counters provide similar reliability as counting devices in an academic library setting. the totals of both devices in a one year study shows a difference of less than half of a percent (.43%) and the majority of collection periods did not exceed a difference of 3% between the devices.   turnstile readers may encounter a multiple swipe dilemma, counting the same patron several times for one particular exit. it is necessary to inspect and edit these records for an accurate portrayal of library visits in a turnstile environment. alternatively, multiple users simultaneously exiting the library threaten the reliability of laser gate counters. these devices may have difficulty in distinguishing the difference between individual patrons and multiple users walking side-by-side, providing less results than actually occurred. however, the occasional patron who must return to the circulation desk to desensitize materials before exiting the library appears to balance this divergence.   both turnstiles and laser gate counters offer additional functionality beyond basic counting. turnstiles offer better physical security and the ability to record individual patron statistics, but their higher cost may dissuade potential buyers. alternatively, laser gate counters do not offer the same level of physical security, but can provide product security and decent dependability at a lower cost. when evaluating both systems as instruments for collecting patron activity, they generate similar results in reliability. therefore, the accuracy of patron counts are comparable between turnstiles and laser gate counters in an academic library settin   references   boss, r. w. (1999). security technologies for libraries: policy concerns and a survey of available products. library technology reports, 35(3), 271.   dotson, d. s., & garris, j. b. (2008). counting more than the gate: developing building use statistics to create better facilities for today's academic library users. library philosophy and practice.   hashimoto, k., kawaguchi, c., matsueda, s., morinaka, k., & yoshiike, n. (1998). people-counting system using multisensing application. sensors & actuators a: physical, 66(1-3), 50-55.   hernon, p., dugan, r. e., matthews, j. r., & thornton, k. (2014). getting started with evaluation. chicago, il: ala editions.   hiller, s. (2004). measure by measure: assessing the viability of the physical library. the bottom line, 17(4), 126-131.   hurlbert, j. m. (2008). defining relevancy: managing the new academic library. westport, ct: libraries unlimited.   kryjak, t., & komorkiewicz, m. (2013). hardware-software vision system for moving people counting based on 3d information. image processing & communications, 18(2-3), 71-81. http://dx.doi.org/10.2478/v10248-012-0081-4   riachi, s., karam, w., & greige, h. (2014). an improved real-time method for counting people in crowded scenes based on a statistical approach. paper presented at the 11th international conference on informatics in control, automation and robotics, vienna, austria.   zhu, h., aghdasi, f., millar, g. m., & mitchell, s. j. (2014). online learning method for people detection and counting for retail stores. u.s. patent no. 20140270358 a1. google patents.   research article   an analysis of digital library publishing services in ukrainian universities   tetiana kolesnykova director of the scientific and technical library dnipro national university of railway transport named after academician v. lazaryan dnipro, ukraine email: chief.library@gmail.com    olena matveyeva head of the electronic library sector of the scientific and technical library dnipro national university of railway transport named after academician v. lazaryan dnipro, ukraine email: diit.media@gmail.com   received: 24 sep. 2018                                                                   accepted: 3 sep. 2019          2019 kolesnykova and matveyeva. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29510     abstract   objective – the objective of this study was to assess the current state of digital library publishing (dlp) in university libraries in the ukraine. the study was conducted in the hopes of gaining a better understanding of the dlp landscape, namely institutional operations, as well as their varying publishing initiatives, processes, and scope.   methods – the current study was conducted from january to june 2017 using a mixed methods approach, involving semi-structured interviews and an online questionnaire. semi-structured interviews were conducted (n = 11) to gain insight into participants’ experiences with dlp. the interviews helped in the creation of the questions included in our online questionnaire. the questionnaire was distributed to 195 representatives (directors and leading specialists) of university libraries in the ukraine. replies were received from 111 of those institutions. the questionnaire consisted of 11 openand closed-ended questions to allow the researchers to obtain a holistic picture of the process under investigation.   results – analysis of the 111 questionnaires showed that for 26 libraries, dlp services were performed by employees of a separate structural unit of the library. for 34 libraries, employees of various departments were involved in performing certain types of services. the other 40 respondents’ libraries were planning to do this in the near future. only 11 respondents replied that they did provide dlp services now nor planned to in the future. among the libraries providing dlp services, the following results were observed: 54 of 60 work with digital repositories, 47 provide digital publishing platforms for journals, 26 provide digital publishing platforms for books, and 23 provide digital publishing platforms for conferences.   conclusions – the results obtained indicate a growing trend of expanding digital services in university libraries to support study, teaching, and research. despite the still spontaneous, chaotic, and poorly explored nature of the development of the library publishing movement in the university libraries of the ukraine, the readiness of librarians to implement publishing activities is notable. at the same time, the survey results point to specific aspects, such as organizational, economic, personnel, and motivational, that require further study.     introduction   over the past 10 years, digital library publishing (dlp) services have become increasingly popular in many countries around the world. to date, it is impossible to specify their exact number. but it is possible to judge the popularity of library publishing. for example, according to the data of the library publishing coalition (lpc), the number of libraries registered with the lpc that provided library publishing services increased from 120 in 2015 to 156 in 2018, an increase of 30% (lpc, 2019). at the same time, dlp is a narrower concept than the broad library publishing offered by the lpc (lpc, n.d.). for the purposes of our study, we defined dlp as a set of activities conducted by college and university libraries to support institutional communities in developing, managing, and distributing online publications, including journals, conference materials, monographs, and other scholarly content. dlp services help scholars share their research through new and emerging publishing models in a constantly changing academic communication environment, and help control and solve the problems and issues related to electronic publishing (bains, 2017; ginther, lackner, & kaier, 2017; raju, & pietersen, 2017; tracy, 2017). this trend becomes evident when examining how localized digital publishing services are in high demand by researchers in the united states (usa), canada, australia, germany, the united kingdom, austria, sweden and other countries (bonn, & furlough, 2015; depping, 2014;). this demand has placed increasing importance on the library and the services it provides, making libraries partners in the production of knowledge (perry et al., 2011).   while this phenomenon has gained traction in north america and western europe, the notion of library as a publisher is relatively new in eastern european countries. over the last decade, dlp services in the ukraine have evolved from the development of institutional repositories, to the establishment and maintenance of electronic academic journals, conference proceedings, and monographs. the current state of dlp in the ukraine is understudied. as of 2017, no comprehensive study or environmental scan had been done. this study was conducted in the hopes of gaining a better understanding of the dlp landscape in the ukraine, namely institutional operations, including their varying publishing initiatives, processes, and scope.   literature review   growth of library publishing services   there is growing evidence that many university libraries have expanded their activities with dlp as one of the new models of scholarly communication. publications and research support services are areas with some of the greatest potential for the future development of academic libraries (simser, stockham, & turtle, 2015; watkinson, 2016; wolff-eisenberg, rod, & schonfeld, 2016). okerson and holzman (2015) asserted that the role of the academic library in the scientific work of an institution was changing and becoming an active player—promising, inquisitive, and ready to experiment—through in library publishing. libraries, having decided to strengthen their positions in universities, are not afraid to experiment, challenge the status quo, and put new services into practice in accordance with new users’ requests and new development strategies for their universities (kolesnykova, 2017; lippincott, 2016; okerson, & holzman, 2015). being interested and active in comprehending modern digital opportunities, as well as having systematic and diverse skills in working with publications, librarians hope for the success and sustainability of dlp initiatives (calarco, shearer, schmidt, & tate, 2016; ginther et al., 2017; lippincott, schlosser, ballard, & maron, 2018).   library publishing efforts in the usa began as early as the 1900s, but the form of dlp that we recognize today did not begin to take shape until the end of the century (bonn & furlough, 2015). by the 1990s there was a technological boom that brought about new and innovative partnerships, emphasizing the digitization of older publications. by the end of the decade, the collaboration environment shifted to an emphasis on advocating for open access (oa) as a result of the increase in subscription costs of materials as well as the need to increase publishing services for informal scholarly outputs traditionally referred to as gray literature (newman, blecic & armstrong, 2007; watkinson, 2014).    the early part of the twenty-first century was marked by a series of innovative publishing initiatives at usa university libraries with the creation of institutional repositories and electronic academic journals, specifically the development and deployment of dspace and open journal systems (ojs) (bonn & furlough, 2015). these developments allowed libraries to meet the publishing and research needs of scientists and researchers (hahn, 2008). through these developments, academic libraries acquired a new paradigm, allowing them to evolve “… from a focus on reader services to a focus on author services” (borgman, 2010, p. 13).   for almost 20 years, we have witnessed the continued growth and strengthening of the synergy between publishing and librarianship, a fact two association of research libraries studies, conducted in 2007 and 2012, demonstrated. the data showed the growth in the number of libraries providing services in scholarly communication, which rose from 75% in 2007 (newman et al., 2007) to 93% in 2012 (radom, feltner-reichert, & stringer-stanback, 2012). a significant part of these services relates to the dlp field. for example, libraries indicated that they helped scholars manage their scholarly identities, understand the intricacies of copyright, study and publish materials in the public domain, and create their own online journals. analysis of data on lpc libraries for 2016 and 2017 further demonstrated this growth. the libraries reported a steady increase in the number of articles they published in 2017, including 436 faculty journals for campuses (compared with 404 for 2016), 905 monographs (compared with 773), and 65 textbooks (compared with 58). in addition, the number of publications issued by library publishers for external groups increased from 189 in 2016 to 249 in 2017. data were analyzed for 118 institutions in the usa, canada, brazil, the united kingdom, germany, and australia (skinner et al., 2017). almost all libraries in the study indicated that they were seeking to provide oa to the results of research from their institutions that were previously invisible to outsiders.   the topic of diversity and features of publishing services has been studied quite often (bonn & furlough, 2015; depping, 2014; mullins et al., 2012; nazarovets, 2012; perry et al., 2011). for example, ginther, lackner, and kaier (2017) emphasized that library publishing support includes the provision of infrastructure, university press, and institutional repository, as well as the dissemination and evaluation of information. agreeing with this, lippincott, schlosser, ballard, and maron (2018) clarified that there is still no complete list of services, and their diversity depends on the skills and abilities of library staff and the specific needs of teachers and students.   library activities for the direct publication of books, journals, conference proceedings, theses and dissertations, technical reports, and other works are fueled by the oa movement. libraries, while striving to provide high-quality oa content, at the same time extend scientists’ capabilities to exert influence in the publishing process (bonn & furlough, 2015; chadwell & sutton, 2014). traditional commercial publishers confer on authors the limited rights to their work and limited control over how to distribute it. giving authors greater control over their work, including distribution, is a common promise of libraries as publishers.   the evolution of dlp has some distinguishing characteristics. libraries, having become knowledge production partners (perry et al., 2011), actively cooperate with faculty, various campus organizations, it support services, and university presses (kolesnykova & kliushnyk, 2015; perry et al., 2011). gradually, there is an increased interest in the idea of such cooperation from academic libraries and university presses, which should be natural allies in an effort to create a more equitable scientific publishing system (okerson & holzman, 2015). for example, in the aaup biennial reporting structure survey of 2016 revealed that 30 of the 133 members of the association of american university presses reported to libraries, which is a doubling over 5 years (watkinson, 2016). at the same time, important developments in the cooperation between the university press and the library offer economic benefits and technological opportunities.   another feature in the development of dlp is its business model, which does not concentrate on making a profit (bains, 2017; raju & pietersen, 2017; skinner, lippincott, sper, & walters, 2014). this was evident in the description of the publishing services offered at the virginia technical libraries: “library publishing services are free” (mcmillan & lawrence, 2013, p. 28). the number of library publishing programs of any size which relied entirely on library budgets to fund their operations has ranged from 50%-56%. according to the library publishing coalition, “in the 2016 directory, 56% of programs relied entirely on the library’s operations budget; in 2017, the percentage had fallen to 48%; this year, it settled in the middle at 50%” (schlosser, hamilton, neds-fox, bielavitz, & hoff, 2018, p. ix). the reason for this fluctuation is not yet clear. it is likely that further full dlp financing from the library operational budget can no longer be guaranteed in the long run. in any case, libraries need to search for new models of financial stability for development and experimentation.   the changing landscape that has put university presses under the auspices of library administrators has also helped transform the library into a modern service-oriented model (bonn & furlough, 2015; ginther et al., 2017; kolesnykova, 2017; radom et al., 2012; watkinson, 2014). at the same time, there are various types of activities performed by libraries with different types of publishing arrangements, such as having work performed by one librarian, an entire department, or different librarians working in different departments (tracy, 2017). organizational changes aimed at improving library services may include the creation of a new library or administrative position, a new department or the restructuring of an existing one, the creation of new working groups or the reorganization and/or integration of several university departments, including university presses and it services. publishing services at the university of graz in austria are an example of how the library’s publishing initiative, the needs of scholars, and the willingness of other administrative departments to cooperate helped create an inter-department group (ginther et al., 2017). in this new organizational form, the new community of practitioners with an expanded service portfolio, a library assumes the role of a service provider, partner, and leader. at the same time, the philosophy of library science is expanding, which is manifested in the philosophy of library publication (kolesnykova, 2017), including the provision of dlp services based on a core value of libraries—the open dissemination of information and knowledge. often, dlp services come about as a result of private initiatives by individual librarians for the public good (hahn, 2008; raju & pietersen, 2017). for example, librarians at universities in south africa developed programs and their own skills to provide reliable publishing services and free access to information/knowledge to any member of their academic communities. these librarians often provided these services on their own time and developed curricula to help educate their colleagues about these issues (raju & pietersen, 2017).   of course, the changes taking place require new competencies from librarians (calarco et al., 2016; keller, 2015; lippincott et al., 2018; wesolek et al., 2017). the need to disseminate and evaluate information actualizes the services of an expert librarian to help university publishers make their content accessible for viewing. library research support services suggest that competent librarians themselves must become active researchers who know and understand the entire life cycle of the research process in addition to issues relating to metadata standards, scientific communication, copyright, and oa (ginther et al., 2017; mullins et al., 2012; perry et al., 2011).   digital publishing services in the ukraine   higher education in the ukraine is made up of institutions that are governed at either state or municipal levels, and private forms of ownership. in total, there are about 300 universities and academies that train specialists possessing educational qualifications no less than a master’s degree (ministerstvo osvity i nauky ukrainy, 2017). the network of libraries at state-owned institutions (hereafter, referred to as university libraries) numbered 195 as of 2017 (yakunina, 2017). libraries at institutions of higher education that are not state owned, as a rule, have very small staffs (1-3 people) and do not deal with dlp.   the protracted economic crisis in the ukraine and the actuality of being in a state of war have an increasing negative impact on the development of ukrainian science. the possibilities of librarians to support scholars are also limited due to the meager funding for acquiring resources from the leading publishers. in addition, the salary of a ukrainian librarian in 2018 was $144 per month (nazarovets, 2019). at the same time, the average monthly salary of a full-time employee was $318 (serednia zarplata v ukraini u dolarakh dosiahla rivnia 2013 roku (infohrafika), 2019). the minimum wage in ukraine in january 2018 was $142 per month (harkusha, 2019). but the desire of ukrainian librarians to improve the reputational value both of their universities and libraries encourages them to search for and introduce new solutions in the provision of digital services.   the development of institutional repositories in the ukraine has become more prevalent over the past few years. the beginning of this activity occurred as a result of the initiative of the libraries themselves. the first university library that began in 2007 to work with an institutional repository was the national university of kyiv-mohyla academy library. by 2012, there were 23 repositories (nazarovets, 2012). in 2018, this initiative received support at the state level. at that time, there were already 99 digital archives of institutions, most of which were supervised by the university libraries. the repository services are part of the library publishing services. special research on this topic in the ukraine has not been conducted. but on the basis of theoretical and practical experience, we are confident that the repositories were the first stage of dlp in the universities of the ukraine, and all the libraries with journal support services, conference material publishing, and book publishing started from the repositories.   the digital repositories in the western regions of the ukraine managed by university libraries archive dissertations, journal articles of teachers and students, conference materials, educational literature, and research reports (lutsyshyna, 2015). in addition to discussing the varying functions of the digital archives, levchenko (2018) also noted the role that library support of digital repositories plays in increasing the prestige (ranking) of the university and its library. additionally, dlp is an attractive alternative to the traditional subscription-based access models, especially as the availability of quality oa publications continues to rise. this is particularly true in conditions of a protracted economic crisis in the ukraine (nazarovets, 2019).   in the ukraine, library services for digital publishing of journals, as in the case of institutional repositories, are an initiative of the libraries themselves. the first example (2011) of the university library as a digital publisher of academic journals belongs to the scientific and technical library of the dnipropetrovsk national university of railway transport (kolesnykova & kliushnyk, 2015; kolesnykova & myrhorodska, 2015). all ukrainian university libraries providing journal publishing services use the ojs software. since 2016, the scientific and technical library of the dnipropetrovsk national university of railway transport (http://conflib.diit.edu.ua/bun_16) has also initiated conference support services using the open conference systems software (ocs). participation in the publication of books (monographs, textbooks) has begun to be included as an area of libraries` interests, and the open monograph press (omp) is often used. despite the significant increase in scientific publications at ukrainian universities, the main issue facing the future of dlp is that there is no direction in the system of training and further education of librarians.   aims   currently, no in-depth study has been conducted examining and evaluating the dlp landscape, as it exists in the ukrainian context. we hoped to fill this gap in the literature, breaking out of the embrace of scientific provincialism and isolation of ukrainian library and information science. this study sought to explore the current state of the dlp movement as a new tool for scholarly communication in higher education institutions. this study analyzed the number of libraries providing publishing services and the extent of the services they provide.   in the course of the study, we tried to answer the following questions:   (1)    how many ukrainian university libraries provide dlp services or plan to do so in the future? (2)    what types of dlp services are ukrainian university libraries providing and what types of publications are they supporting?   methods   a mixed methods approach was used in carrying out this study. eleven semi-structured interviews were conducted with library directors and leading specialists overseeing established digital publishing services at academic institutions. the selection of respondents was carried out according to the following main criteria: level of competence, work experience, position, and participation in professional events. the interviews were conducted in an effort to gain a greater understanding of the dlp landscape prior to formulating the questions for the online questionnaire. two questions were asked:   (1)    does your library provide dlp services, in addition to supporting institutional repositories? (2)    what types of materials/publications are included?   the interviews lasted up to 20 minutes and were conducted both personally (n = 9) and by telephone (n = 2). the average interview duration was 14 minutes. personal interviews were held at the seminar “professionals competences of libraries in the terms of media reality: media culture and copyright” (dnipro, january 2017) (n = 6) and during the scientific сommunication in the digital age сonference (kyiv, march 2017) (n = 3). audio recordings of the opinions and comments of all participants were made; audio to text were translated using the express scribe program. the most important passages in the interviews relating to the research questions were coded. then the codes were recorded and analyzed using microsoft excel.   an online questionnaire (see appendix a) was created in ukrainian using google forms and distributed to 195 library directors and leading specialists working in academic settings in the ukraine, via email, facebook, and the “library synergy: support of scientific research” website of the section of university libraries of the ukrainian library association (http: //bibliosynergy.ula.org.ua/). the first series of questions were created to ascertain the extent of dlp services currently being provided at their institutions and plans for the creation or expansion of existing services. the questionnaire remained open for a period of 49 days in the spring of 2017. participants were asked to indicate their educational institution, followed by six questions about the degree of participation of the university in the provision of dlp and the specific available services. of these questions, three were semi-closed and three closed. in the three semi-closed questions (b3.2.1, b3.2.2, and b3.3), respondents could add their own answers. certain types of services were specified in the response options. respondents who selected the “other” option could add their own answers.   the questionnaire received responses from 115 institutions. after eliminating incomplete answers, we were left with 111 respondents, giving us a 60% response rate. the majority of respondents (75%, n = 83) were affiliated with academic libraries in regional centres of the ukraine. the remainder of respondents were affiliated with academic libraries in kyiv (capital of the ukraine) (14%, n = 16) and academic libraries in small towns (11%, n = 12).   an additional questionnaire was also issued (see appendix b) to study the distribution of publications and their indexing in databases. it contained four questions: one closed and three open. the checklist was distributed using the same methods described above. the additional questionnaire received responses from only 81 institutions. this questionnaire was distributed 14 days after the first. from 81 respondents, six did not answer the first questionnaire, while 75 responded to both.   we assumed that it would take up to 20 minutes to complete each questionnaire. the results were analyzed in tabular form using google forms and excel.   results   semi-structured interviews   all 11 respondents said that their libraries provide other types of dlp services in addition to supporting repositories. the following services were indicated:   ·         monitoring, analysis, and visibility (n = 9) ·         supporting the journal archives on their websites (including digitizing the retrospective editions) (n = 6) ·         providing training on open science principles and on copyright and licensing (n = 6) ·         providing training for the editors of publications (n = 5) ·         registration of publications (issn, isbn, doi) and distribution of doi among university editorial boards (n = 5) ·         support for publishing scientific journals through supporting the individual editorial processes (metadata, plagiarism checks, coordinating of manuscripts) (n = 4) ·         hosting and administering the websites of journals (n = 4) ·         registration of publications, transfer of metadata or full texts to databases (national, thematic, international) (n = 4) ·         support for open conference sites (n = 3) ·         trainings for conference organizing committees (n = 1)   it can be seen from the list that in the field of publishing infrastructure, librarians want to play the role of partners by providing technical services and content placement services. also, thanks to their communication skills, librarians have good opportunities to conduct interviews, consultations, and trainings. the variety of dlp services mentioned confirmed that researchers with a new worldview, requiring fundamentally new services, have knowledgeable and reliable partners in the theoretical and practical development of these issues through libraries’ assistance. the last item was mentioned in only one interview: “by telling and teaching the conference organizers, answering their questions, you realize that it is you who enhances the reputation of the library.” while this concept was only stated once, we assumed that this activity was promising and could be practiced by other university libraries.   respondents mentioned the following types of publications that their libraries supported: journals (n = 8), conference materials (n = 5), and books (n = 4).   during the interviews, it became clear that respondents did not always know what their role will be in supporting journals, conference materials, and books in the future and whether they will be promising. for example, if at the initial stage technical support for the ojs system was provided, then, with the gaining of practical experience and authority, support for the publishing of academic journals could grow through the support of separate editorial processes (e.g., metadata, plagiarism check, coordination of manuscripts). if at the initial stage advocacy practices and technical support of the ojs system was provided, then, with the obtaining of practical experience and authority, support for the publication of scholarly journals could also expand due to the support of individual editing processes (e.g., metadata, plagiarism verification, manuscript approval). but the negative aspects associated with insufficient funding, staff reduction, and the drift of highly qualified library staff due to low salaries, did not give confidence in the guaranteed long-term perspective.   at the same time, unexpectedly for the authors, there was a constant comparison with repository support services, which ukrainian libraries have practiced for more than 10 years. analysis of the interview transcripts allowed us to identify the publication types that are supported and the types of services provided. analysis also revealed that there are six common objectives associated with library-managed repositories:   ·         to preserve scientific works of university scientists (n = 11) ·         to comply with oa (policies, features, motivations, etc.) (n = 11) ·         to fulfill institutional missions for research (n = 10) ·         to increase external visibility of institution (n = 8) ·         to manage institutional archives and digital collections (n = 6) ·         to fulfill institutional missions for education (n = 4)   during the interviews, there was no intention of discussing the dlp’s impact on library staff, but the issue came up occasionally. for example, one library director stated: “in a short time, to switch to new technologies, to the field of scientific publishing is psychologically and physically very difficult.” another stated: “communication with authors is different from communication with scientists who just need books, journals, electronic information. it is much more difficult . . . more disturbing.” the reasons for this concern with the impact on staff could involve overload of information, adaptation to new systems, concern about performance, poor motivation, and lack of computer experts.   according to some respondents (n = 4), it is advisable to conduct a study of techno-stress as a negative psychological connection between people and the introduction of new technologies in university libraries in the future. we think that this is very important, since the facts of dependence of the psychological status and productivity of librarians on new computer information technologies are obvious.   online questionnaire   some respondents, 54% (n = 60), stated that they currently provide digital library services, while 36% (n = 40) responded that while they currently do not provide dlp, they plan to in the future. only 10% (n = 11) of respondents stated that they have no plans to develop services now or in the future. in 26 libraries, dlp services were a special area of work, performed by employees of a separate structural unit of the library. in 34 libraries, employees of various departments are involved in performing certain types of services (e.g., registration of publications [issn, isbn, doi], distribution of dois among university editorial boards, metadata creation, plagiarism check), because it is difficult to immediately organize a new formal structure in ukrainian libraries. the 36% who responded positively to the creation of dlp in the future identified the need to build infrastructure and conduct preparatory activities such as staff training, negotiations with university management and editorial boards, improvement of technical facilities, etc.   the 60 respondents who answered affirmatively to providing dlp services were asked to answer additional questions detailing the scope of the services they provide. some responding institutions, 90% (n = 54), supported an institutional repository. another 78.3% (n = 47) provided support for journal publishing, with 38.3% (n = 23) providing support for conferences, and 43.3% (n = 26) providing support for books.   when asked to describe the types of services they provide in relation to various publication types (i.e., journals, conference proceedings, and online books) associated with their dlp services, the results show that they are quite varied in scope and practice. of all respondents, 78.3% (n = 47) stated that they support the publishing of academic journals through the support of separate editorial processes (e.g., metadata, checking for plagiarism, coordination of manuscripts), the support of journal archives on their websites (including digitizing retrospective issues), software training, and the hosting and administration of journal websites (see table 1).   table 1 library support of journals (n = 60) services number of libraries percentage support for digital archives of the editions 26 43.3% support of separate editorial processes 26 43.3% trainings for editorial staff 20 33.3% hosting and administration of journal websites 15 25.0% other services 5 8.3% not practiced or not answered 13 21.7%   when asked to provide descriptions of other services they provide, the respondents indicated that they provided support for indexing in scholarly databases and applying appropriate identifiers for published content including issns and dois. in addition, publications are checked for compliance with international publishing standards and ethic guidelines of publications.   when asked to describe the services they provide relating to conferences, the majority of respondents (61.7%, n = 37) stated that they did not provide conference support services at all, while 33% (n = 20) provided hosting support and digital publishing of conference proceedings and reports (see table 2).   table 2 library support of conference publishing (n = 60) services number of libraries percentage support of the sites of open conferences 20 33.3% posting conference proceedings in the repository 2 3.3% trainings for organizational committees 1 1.7% not practiced or not answered 37 61.7%   when asked whether or not book publishing is part of their dlp service, only 43.3% responded in the affirmative, while the majority (56.7%) did not support this mode of publishing.   when asked to list the dlp services they offered in addition to the online publication of materials, respondents stated that they advised on issues relating to oa publishing (81.7%); the monitoring and analysis of efficiency, visibility, and impact of publications (45.0%); intellectual property issues (41.7%); and the registration of publications (31.7%) (see table 3).   table 3 aspects of common services in dlp (n = 60) services number of libraries percentage advising on oa publishing specifics 49 81.7% monitoring and analysis of efficiency, visibility, and impact of publications 27 45.0% advising on intellectual property issues 25 41.7% registration of publications (issn, isbn, doi), distribution of dois to university editorial boards 19 31.7% other 4 6.7% not answered 6 10.0%   thus, among the libraries providing dlp (n = 60), the following support results were observed: institutional repositories, 90% (n = 54); journals, 78.3% (n = 47); conferences, 38.3% (n = 23); books, 43.3% (n = 26).   of the 111 responding institutions, only 75 libraries (67.6%) answered questions regarding the practice of registration of publications and transferring metadata or full texts to databases (national, thematic, international) (see appendix b). of the respondents (n = 81), 28.4% (n = 23) answered in the affirmative, while 71.6% (n = 58) responded in the negative. of those respondents not currently providing these services, 54.3% stated that it was part of their future plans. tables 4, 5, and 6 show the databases to which the information is being transferred.   table 4 national information systems (n = 23) (see appendix b, b4.2.1) national information systems/databases number of libraries percentage “scientific periodicals of ukraine” (polythematical repository of the vernadsky national library of ukraine) 18 78.3% bibliographic database “ukrainika naukova” 7 30.4% bibliographic database “dzherelo” 5 21.7% ukrainian research and academic network (uran) (journals on the ojs-platform) 4 17.4% information portal “science of ukraine: access to knowledge” 4 17.4% ukrainian scientific journals (usj) 2 8.7% others 3 13.0%   table 5 thematic databases (n = 23) (see appendix b, b4.2.2) thematic databases number of libraries percentage abstract database “dzherelo” by themes series: 1. natural sciences; 2. engineering; industry; agriculture; 3. social sciences and humanities; arts; 4. medicine; medical sciences 9 39.1% “consolidated database of theses on education, pedagogy and psychology” (v.o. sukhomlynsky sspl) 2 8.7% business source international platformed by ebsco 1 4.3% chemical abstracts service (cas) 1 4.3% iconda®bibliographic (the international construction database) 1 4.3% repec (research papers in economics) 1 4.3% legal education. legal culture: a consolidated e-catalog 1 4.3% social communications: a consolidated e-catalog 1 4.3% no or not answered 12 52.2%   table 6 international databases (n = 23) (see appendix b, b4.2.3) international scientific information systems/databases number of libraries percentage index copernicus 8 34.8% google scholar (bibliometrics of ukrainian science) 8 34.8% worldcat 7 30.4% bielefeld academic search engine (base) 7 30.4% ulrich’s periodicals directory (ulrichsweb™ global serials directory) 5 21.7% web of science 4 17.4% researchbib 4 17.4% directory of oa journals (doaj) 3 13.0% opendoar 3 13.0% directory of research journal indexing (drji) 3 13.0% openaire 3 13.0% crossref (2) + cited by linking (1) 3 13.0% ebsco 2 8.7% scopus 2 8.7% universal impact factor 2 8.7% journaltocs 2 8.7% open academic journals index (oaji) 2 8.7% no or not answered 3 13.0%   of the respondents, 23 libraries were involved in registration of publications, or transfer of metadata or full texts to databases. the most popular is the work with the national polythematical repository of the vernadsky national library of ukraine “scientific periodicals of ukraine.” among thematic databases, dzherelo (bibliographic database of ukraine) is more popular than other databases. index copernicus and google scholar are the most popular among international information systems.   discussion   university libraries in ukraine are actively involved in the creation of a new infrastructure for scholarly communication, which is particularly evident in dlp. like university libraries from around the world, academic libraries in the ukraine have dealt with issues associated with dlp, including economic viability, technological opportunities, potential business models, and oa. the results of the research demonstrated the level of interest that librarians have in the development of dlp. this is demonstrated by the fact that dlp is happening across the country. the intention of ukrainian university libraries to focus on maximizing the openness and accessibility of publications within the international academic field and promoting the free flow of knowledge, as well as increasing activity in internal communications with researchers, is the main driving force of the acceptance and development of dlp services. however, these concepts are not always clearly defined by institutions. this vagueness was demonstrated among some of those interviewed by concerns for the uncertainty and stability in long-term dlp development, partly because of the negative aspects of the economic situation in the ukraine.   while the potential for expansion and future development of these services is important, we must acknowledge the existing issues that may hinder its ongoing progress, primarily insufficient funding, which can lead to a reduction in staff. it also does not allow for updating of computers, purchasing new licensed software, etc. the loss of qualified personnel due to low salaries has particularly negative effects. in addition, the question of how to strengthen dlp by strengthening the infrastructure that supports it, whether that infrastructure be workflows, production support, or platforms, has not yet been answered. therefore, we can consider these topics to be particularly interesting and requiring further study.   only 11% (n = 12) of the first questionnaire participants were representatives of small cities, while in the country as a whole, 17.4% (34 from 195) of the population were from small cities. the remaining respondents (n = 99) were from the capital and regional centers (administrative centers of oblasts). considering that a lack of financial resources is a common problem for all libraries in the ukraine, we assumed that the heightened interest in creating, providing and supporting dlp services in the capital and regional centers is the result of increased and continuous training activities for librarians. these opportunities include access to guest speakers from germany, the u.s., and the united kingdom, sessions that are held mainly in kiev and occasionally in large regional centers, but not in small towns. although the protracted economic crisis reduced the opportunities for the professional development of librarians throughout the ukraine, in small towns it is felt more acutely. therefore, librarians traveling to other cities for conferences, seminars, and workshops was difficult due to insufficient financial support from their home institutions. unfortunately, dlp webinars in ukrainian or russian are non-existent, which hinders a librarian’s ability to further develop their competencies. without opportunities for professional development, dlp may not develop fast enough to serve as a viable alternative to expensive traditionally published titles, which might have resulted in significant savings.   integrating institutional research into international databases will continue to be a challenge. traditionally, it has been widely believed that the more databases a journal is indexed in, the greater its impact will be. the problem is the growing number of misleading metrics and the general misunderstanding of what they represent. we assume that working with some dubious companies is a situational and temporary measure to increase the reach of the studies conducted by ukrainian scholars and make them more visible. but journal editors and librarians in ukrainian universities are becoming more and more selective, choosing which databases to cooperate with.   this study may be interesting to librarians from different countries for at least two main reasons. first, ukrainian librarians have already created and continue to create a large and diverse array of oa information that may be of interest not only for researchers living in the ukraine, but also for ukrainian scientists and students living and working in different countries of the world. according to the analytical center cedos in 2016-2017 (stadny, 2019), 77,424 students with ukrainian citizenship were trained outside the ukraine: 33,370 in poland, 11,440 in russia, 9,638 in germany, 3,425 in canada, and 2,471 in the czech republic, as well as in austria, italy, spain, and bulgaria. in addition, in many countries there are large ukrainian diasporas. in particular, according to the 2016 census, 1,359,665 canadians indicated their ukrainian ethnic origin (statistics canada, 2019). literature from ukrainian repositories, electronic journals, and other oa resources can be used directly by researchers as well as by librarians seeking to satisfy the interests of their readers as much as possible.   the second reason for possible interest of librarians from other countries in the ukrainian dlp experience is the widespread introduction of these services throughout the country in the presence of formidable financial barriers. by creating dlp, ukrainian librarians are trying to minimize the negative attitude of researchers towards libraries due to the lack of a sufficient range of modern publications and access to information resources.   in difficult economic conditions, university libraries have taken the initiative to support the development, management, and distribution of reliable scholarly content created in their institutions. the model of library as a publisher is developing in the ukraine, overcoming a number of difficulties. in addition to economic difficulties, one can also name the linguistic ones (poor knowledge of english), lack of methodological assistance and educational programs, the deficit of highly qualified staff, and difficulties with updating computer equipment. therefore, it seems to us that this movement is slow compared to the university libraries of the developed countries. but it does not stop attracting an increasing number of enthusiastic librarians as its allies. this study can provide an inspiring example to developing country libraries for planning and implementing new dlp services.   conclusions   this study was conducted in the hopes of gaining a greater understanding of the current state of dlp services offered at university libraries in the ukraine. the results of the questionnaire demonstrated that the current state of dlp is strong, with 54% of responding libraries already providing the service. the results show that future growth is inevitable, with 36% of respondents stating that while they currently do not provide dlp services, there are plans for program development in the future. the results from the interviews showed that the six most common objectives for providing digital library services are to fulfill the institutional missions for research and education, to increase the external visibility of the institution and its research output, to preserve scientific works of university scientists, to manage institutional archives and digital collections, and to ensure oa compliance. the main challenges participants identified related mainly to institutional repositories: their installation, maintenance, and the submission processes. of the barriers to service implementation and growth, respondents identified the need to build infrastructure and to improve training and awareness for staff, librarians, researchers, and university administrators. the authors consider it necessary to build the capacity of librarians to support the new roles of libraries in the field of scientific communications and electronic research, including dlp.   acknowledgments   the authors of the article are sincerely grateful to o. serbin, the director of the scientific library of the taras shevchenko national university of kyiv, and the staff of the scientific and methodical department of this library for the organizational assistance in conducting the online survey, as well as the staff of the ukrainian library association. the authors also thank kealin mccabe (geoffrey r. weller library, university of northern british columbia, canada) for her invaluable assistance in working on the manuscript.   references   bains, s. (2017). the role of the library in scholarly publishing: the university of manchester experience. insights, 30(3), 70-77. https://doi.org/10.1629/uksg.380   bonn, m., & furlough, m. (eds.). (2015). getting the word out: academic libraries as scholarly publishers. chicago, il: association of college & research libraries. retrieved from http://www.ala.org/acrl/sites/ala.org.acrl/files/content/publications/booksanddigitalresources/digital/9780838986981_getting_oa.pdf   borgman, c. l. (2010). research data: who will share what, with whom, when, and why? the fifth china-north america library conference, beijing, china. retrieved from http://works.bepress.com/borgman/238/   calarco, p., shearer, k., schmidt, b., & tate, d. (2016). librarians’ competencies profile for scholarly communication and open access. retrieved from https://www.coar-repositories.org/files/competencies-for-scholcomm-and-oa_june-2016.pdf   chadwell, f., & sutton, s. c. (2014). the future of open access and library publishing. new library world, 115(5/6), 225-236. https://doi.org/10.1108/nlw-05-2014-0049   depping, r. (2014). publikationsservices im dienstleistungsportfolio von hochschulbibliotheken. eine (neu-)verortung in der wissenschaftlichen publikationskette. o-bib. das offene bibliotheksjournal / herausgegeben vom vdb, 1(1), 71-91. https://doi.org/10.5282/o-bib/2014h1s71-91 (in german)   ginther, c., lackner, k., & kaier, c. (2017). publication services at the university library graz: a new venture, a new role. new review of academic librarianship, 23(2-3), 136-147. https://doi.org/10.1080/13614533.2017.1324802    hahn, k. (2008). research library publishing services: new options for university publishing. washington, dc: association of research libraries (arl). retrieved from https://www.arl.org/resources/research-library-publishing-services-new-options-for-university-publishing/   harkusha, t. (2019) minimalna zarplata v ukraini v 2019 rotsi: rozmir i prohnozy shchodo pidvyshchennia. unian. retrieved from https://maanimo.com/ua/indexes/142788-minimalna-zarplata-2019 (in ukrainian)   keller, a. (2015). publikationskompetenz als neues aufgabengebiet für bibliotheken: eine australische fallstudie. bibliothek forschung und praxis, 39(2), 158-162. https://doi.org/10.1515/bfp-2015-0019 (in german)   kolesnykova, t. o. (2017). “i light my candle from yours…”: anthropological aspects of modern library services for scientists. anthropological measurements of philosophical research, 11, 49-62. https://doi.org/10.15802/ampr.v0i11.105478   kolesnykova, t. o., & kliushnyk, i. a. (2015). publication of scientific periodicals at universities: new challenges, participants, technology. science and transport progress, 6(60), 183-197. https://doi.org/10.15802/stp2015/57105 (in ukrainian)   kolesnykova, t. o., & myrhorodska, a. i. (2015). scientific and publishing model “library publishing” in university libraries of ukraine and the world. visnyk knyzhkovoi palaty, 3, 24-28. retrieved from http://eadnurt.diit.edu.ua/jspui/handle/123456789/3811 (in ukrainian)   levchenko, n. (2018). vidkrytyi elektronnyi arkhiv – vazhlyvyi chynnyk publikatsiinoi aktyvnosti naukovtsiv. bibliotechnyi visnyk, 5, 3-7. retrieved from http://bv.nbuv.gov.ua/doc/bv_2018_5_3 (in ukrainian)   library publishing coalition. (2019). library publishing directory. retrieved 3 december 2018 from https://librarypublishing.org/directory/   lippincott, s. k. (2016). the library publishing coalition: organizing libraries to enhance scholarly publishing. insights, 29(2), 186-191. https://doi.org/10.1629/uksg.296   lippincott, s., schlosser, m., ballard, h., maron, n. (eds). (2018). library publishing curriculum. retrieved from https://educopia.org/deliverables/library-publishing-curriculum   lutsyshyna, t. (2015). instytutsiinyi repozytarii yak perspektyvna forma naukovoi ta osvitnoi komunikatsii u vyshchomu navchalnomu zakladi. naukovi pratsi natsionalnoi biblioteky ukrainy imeni v. i. vernadskoho, 42, 567-579. retrieved from http://nbuv.gov.ua/ujrn/npnbuimviv_2015_42_47 (in ukranian)   mcmillan, g., & lawrence, a. (2013). publishing services from the center for digital research and scholarship. retrieved from https://vtechworks.lib.vt.edu/bitstream/handle/10919/23917/libpubng4oaw2013nlicombined.pdf?sequence=5   ministerstvo osvity i nauky ukrainy. (2017). reiestr vyshchykh navchalnykh zakladiv. retrieved 12 december 2017 from https://mon.gov.ua/ua/ministerstvo/diyalnist/vidkriti-dani (in ukrainian)   mullins, j. l., murray-rust, c., ogburn, j. l., crow, r., ivins, o., mower, a., … & watkinson, c. (2012). library publishing services: strategies for success. washington, dc: sparc. retrieved from https://docs.lib.purdue.edu/purduepress_ebooks/24/   nazarovets, s. (2012). repozytarii vyshchykh navchalnykh zakladiv ukrainy u systemi naukovoi komunikatsii. visnyk knyzhkovoi palaty, 8, 1-5. retrieved from http://eprints.rclis.org/17738/ (in ukrainian)   nazarovets, s., teixeira da silva, j. a., & nazarovets, m. (2019) challenge of ukrainian academic librarians in an evolving scholarly publishing landscape. journal of academic librarianship, 45(1), 9-14. https://doi.org/10.1016/j.acalib.2018.11.001   newman, k., blecic, d., & armstrong, k. (2007). scholarly communication education initiatives: spec kit 299. washington, dc: association of research libraries. retrieved from http://publications.arl.org/scholarly-communication-spec-kit-299   okerson, a., & holzman, a. (2015). the once and future publishing library. washington, dc: council on library and information resources. retrieved from  https://www.clir.org/pubs/reports/pub166/   perry, a. m., borchert, c. a., deliyannides, t. s., kosavic, a., kennison, r., & dyas-correia, s. (2011). the balance point: libraries as journal publishers. serials review, 37(3), 196-204. retrieved from http://scholarcommons.usf.edu/tlar_pub/2   radom, r., feltner-reichert, m., & stringer-stanback, k. (2012). organization of scholarly communication services: spec kit 332. washington, dc: association of research libraries. retrieved from http://publications.arl.org/organization-of-scholarly-communication-services-spec-kit-332/   raju, r., & pietersen, j. (2017). library as publisher: from an african lens. journal of electronic publishing, 20(2). https://doi.org/10.3998/3336451.0020.203   serednia zarplata v ukraini u dolarakh dosiahla rivnia 2013 roku (infohrafika). (2019). unian. retrieved from: https://www.unian.ua/economics/finance/10541199-serednya-zarplata-v-ukrajini-u-dolarah-dosyagla-rivnya-2013-roku-infografika.html (in ukranian)   simser, c. n., stockham, m. g., & turtle, e. (2015). libraries as publishers: a winning combination, oclc systems & services: international digital library perspectives 31(2), 69-75. https://doi.org/10.1108/oclc-01-2014-0006   skinner, k., lippincott, s., sper, j., & walters, t. (2014). library-as-publisher: capacity building for the library publishing subfield. the journal of electronic publishing, 17(2). https://doi.org/10.3998/3336451.0017.207   skinner, k., neds-fox, j., hamilton, l., simser, c., owen,t., & purple, k. (2017). introduction. in library publishing directory 2017 (pp. vi-ix). atlanta, ga: library publishing coalition. retrieved from https://librarypublishing.org/wp-content/uploads/2017/03/lpc_lpdirectory2017.pdf   schlosser, m., hamilton, l., neds-fox, j., bielavitz, t., & hoff, a. (2018). introduction. in library publishing directory 2018. atlanta, ga: library publishing coalition. retrieved from https://librarypublishing.org/wp-content/uploads/2017/03/lpc_lpdirectory2018.pdf   stadny, y. (2019). ukrainske studentstvo za kordonom: dani do 2017/18 navchalnoho roku. cedos think tank. retrieved from https://cedos.org.ua/en/articles/ukrainske-studentstvo-za-kordonom-dani-do-201718-navchalnoho-roku (in ukranian)   statistics canada. (2019). census profile, 2016 census. retrieved from https://www12.statcan.gc.ca/census-recensement/2016/dp-pd/prof/details/page.cfm?lang=e&geo1=pr&code1=01&geo2=&code2=&data=count&searchtext=canada&searchtype=begins&searchpr=01&b1=all&tabid=1#fnb103   tracy, d. g. (2017). libraries as content producers: how library publishing services address the reading experience. college and research libraries, 78(2), 219-240. https://doi.org/10.5860/crl.78.2.219   watkinson, c. (2014). the university as publisher revisited. insights, 27(2), 181-185. https://doi.org/10.1629/2048-7754.131   watkinson, c. (2016). why marriage matters: a north american perspective on press/library partnerships. learned publishing, 29, 342-347. https://doi.org/10.1002/leap.1044   wesolek, a., thomas, j. w., dresselhaus, a., fielding, j., simser, c., sutton, s. … & spratt, s. (2017). nasig core competencies for scholarly communications librarians. retrieved from http://www.nasig.org/site_page.cfm?pk_association_webpage_menu=310&pk_association_webpage=9435   wolff-eisenberg, c., rod, a. b., & schonfeld, r. c. (2016). ithaka s+r us faculty survey 2015. https://doi.org/10.18665/sr.277685   yakunina, n. (2017). merezha bibliotek vnz ukrainy derzhavnoi formy vlasnosti. ii mizhnarodna naukovo-praktychna konferentsiia “biblioteky vyshchykh navchalnykh zakladiv: dosvid ta perspektyvy.” retrieved from http://www.library.univ.kiev.ua/ukr/for_lib/konf-2017-1.php3 (in ukrainian)   appendix a   library publishing services at higher education institutions of ukraine questionnaire   b1.           your higher education institution    b2.           library publishing services at your university are (choose one of the options): a)       a special area of work b)      a certain type of service c)       not practiced but planned d)      neither practiced nor planned   b3.           if you chose answer a) or b) in block 2, please specify the services.   b3.1.          support of the institutional repository ("yes" or "no")   b3.2.          scientific publishing   b3.2.1.      journals (choose one or more options or add your own) ·         trainings for editorial staff ·         hosting and administration of journal websites ·         support of separate editorial processes ·         support for digital archives of the editions ·         not practiced or not answered ·         other services   b3.2.2.      conferences (choose one or more options or add your own) ·         support of the sites of open conferences, posting reports ·         not practiced ·         other   b3.2.3.      online books (monographs, textbooks) ("yes" or "no")   b3.3.          general lp services (choose one or more options or add your own) ·         registration of publications (issn, isbn, doi) and distribution of dois among university editorial boards ·         advising on intellectual property issues ·         advising on open access publishing specifics ·         monitoring and analysis of efficiency, visibility, and impact of publications ·         other   appendix b   library publishing services at higher education institutions of ukraine addition to the questionnaire   b4.           your higher education institution   b4.1.                      does your library practice the services of integration of university research results into international and national information systems / databases? (choose one of the options) ·         yes ·         no, but we are planning to ·         no, and we are not planning to   b4.2.                      please indicate geography and systems b4.2.1.                 indicate national information systems b4.2.2.                 indicate thematic or industry databases b4.2.3.                 indicate international databases     evidence summary   a survey of provosts indicates that academic libraries should connect outcomes to university goals   a review of: murray, a. & ireland, a. (2018). provosts’ perceptions of academic library value and preferences for communication: a national study. college & research libraries, 79(3), 336-365. https://doi.org/10.5860/crl.79.3.336     reviewed by: laura costello virtual reference librarian rutgers university libraries rutgers, the state university of new jersey new brunswick, new jersey, united states of america email: laura.costello@rutgers.edu   received: 10 aug. 2018 accepted: 8 sept. 2018      2018 costello. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29484     abstract   objective – to understand how public and private university provosts understand and interpret the value of academic libraries.   design – electronic survey.   setting – public and private colleges and universities in the united states with carnegie classifications of master’s (small), master’s (medium), master’s (large), doctoral/research (dru), research (ru/h), and research very high (ru/vh). subjects – 209 provosts and chief academic officers.   methods – the authors distributed the survey to a pool of 935 provosts and chief academic officers in academic institutions. questions were organized toward understanding participants’ perceptions of their libraries’ involvement with issues of institutional importance inspired by the association of college & research libraries’ value of academic libraries: a comprehensive research review and report, and high impact educational practices (hips) based on the work of george kuh (2008). the survey also asked participants to select their data preferences when making library funding allocation decisions and their library communication preferences when making funding decisions. the authors received 209 responses and analyzed the content using qualtrics to determine the highest and lowest ranked responses to each question. in addition, responses for specific survey questions were cross tabulated with demographic information about the institution to identify any potential trends that conformed to or deviated from the overall set of responses. chi squares were then calculated to determine potential significance.   main results – in terms of involvement with university initiatives, almost all of the 209 provosts and chief academic officers who responded to the survey had the perception that their respective libraries are either very involved or somewhat involved. the highest areas of involvement included: faculty research productivity (85.02%), accreditation (82.15%), student academic success (75%). and undergraduate retention (67.26%). of note, only 9% of provosts indicated their libraries were very involved with enrollment. the authors found a trend that suggests that higher-enrollment institutions with a carnegie ranking of doctoral/research, research, or research very high, increased provosts’ perceptions of their institutions’ libraries involvement in retention initiatives, student academic success, and faculty research productivity. a significant point of note: when asked why provosts did not view their institutions’ academic libraries as being involved in undergraduate retention initiatives, a significant number (76.12%) of respondents indicated that it was because the campuses overall did not recognize the role the libraries could play in retention initiatives. this position co-exists in an environment where the demographic, economic, and cultural transitions taking place in the united states are continuing to have a disruptive impact on higher education. library directors need to make these connections much more tangible.   utilizing kuh’s (2008) 10 high-impact educational practices, the authors gauged the participants’ perception of their libraries’ involvement in educationally purposeful activities. they found that 84.43% of provosts perceived their libraries as highly involved with undergraduate research, 78.39% with first-year seminars/experiences, 77.38% with collaborative assignments and projects, 75.76% with writing-intensive courses, 71.34% with common intellectual experiences, and 69.64% with capstone courses/projects. fewer provosts indicated that their libraries were involved in diversity and global learning, learning communities, service learning/community-based learning, or internships. a significant point of note: when asked if their institution’s library had an impact on students’ decisions to continue enrollment, opinion was divided. of the total respondents, a combined total of 91 indicated yes, based on demonstrated evidence or anecdotal or suspected evidence, while 81 respondents indicated unclear or no. this suggests further work is required for libraries in terms of investigating the potential role they might play in enrollment and how to demonstrate such.   the authors also asked participants to indicate their opinion on the level of influence 11 different data types would have on a moderate (non-capital) funding request for the library. in terms of highest influence, 72.02% indicated they would like to see correlations linking the use of library services/resources with student success, 66.07% with undergraduate retention, and 56.55% with enrollment. of moderate influence, 57.14% indicated they would like to see library usage data, 55.36% user satisfaction data, and 50% focus groups or other qualitative data. a total of 60% of the provosts also indicated that anecdotal evidence had a low influence on their funding allocations. most provosts preferred the information to be communicated in a formal annual report, and indicated that the report should include information literacy student learning outcomes (slos) (50.9%), user satisfaction data (46.11%), correlations with faculty productivity (45.45%), correlations with student success (44.91%), correlations with undergraduate retention (43.11%), correlations with enrollment (42.51%), basic use data (40.12%), and faculty feedback (39.1%).   conclusion – most provosts have an understanding that their libraries play an important role on campus, but demonstrating a strong connection to university goals and outcomes is essential. when seeking funding, academic library administrators should focus on projects or initiatives that support the priorities of the institution as a whole, and work to communicate evidence of the value of library services and resources within this context. this is achieved through communication channels that are both timely and relevant, and include a formal annual report or a dedicated budget meeting.   commentary   positioning libraries as an asset to academic administration is an increasingly essential mandate for library administrators. recent research has explored this area, including a recent article in the new review of academic librarianship by john cox (2018) and the previous work of the authors of the topic article, adam murray and ashley ireland, which examined the strategies of library directors in communicating value to university leadership (2017). the authors make clear the importance of communicating library impact in ways and areas that university leaders understand and value.   this study focused on the outcomes provosts and chief academic officers expect from libraries and the data they are looking for to demonstrate this impact. the survey addressed a significant number of provosts and chief academic officers and provided a representative sample when analyzed according to criteria in the glynn critical appraisal tool (2006). the population of provosts and chief academic officers was selected from institutions with relevant carnegie classifications. population collection relied on publicly available email data, but for this population of academic administrators at graduate-level carnegie-classified institutions, this method of selection does not exclude a significant number of participants and can be considered representative. there were 935 provosts contacted, and the authors received 209 responses, representing a 22% response rate.   the authors used likert-type scales to gauge provosts’ impressions of library involvement in the activities of institutional importance and high-impact educational activities they had defined, but did not provide the participants with the option to indicate other areas of importance that they might consider. this represents a potential weakness in the study, as provosts may have identified different areas of importance if they had been given more freedom in the survey. in-depth interviews with this population could augment this research: since the issues of institutional importance were partially defined by a library organization, there is a possibility that provosts’ perceptions of these goals might differ from the options that they were allowed to choose in the survey. this information will be relevant to academic librarians and administrators, because these activities are likely already included in the institutional goals of their libraries. this is also true of the data definitions in the funding section of the survey. because most libraries gather and use the defined data categories, it is important to understand the value of each of these data to academic administrators.   this article is particularly effective for librarians and library administrators in the target group of institutions because the data is cross tabulated to reveal provost responses for particular carnegie classifications, enrollments, and types of institutions (public/private). this organization will help readers analyze the data in the context of their own institution, though the authors also analyze and explain the general trends in responses. understanding how current provosts and chief academic officers understand the work of libraries and how we can approach these administrators with data that is capable of changing and improving their understanding of our work is an important goal. this article provides an important context for an evolving work in libraries and a foundation for developing funding and promoting the work of libraries to stakeholders in academic administration.   references   cox, j. (2018). positioning the academic library within the institution: a literature review. new review of academic librarianship. https://doi.org/10.1080/13614533.2018.1466342   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154   kuh, g. d. (2008). high-impact educational practices: what they are, who has access to them, and why they matter. washington, dc: association of american colleges & universities.   murray, a. l., & ireland, a. p. (2017). communicating library impact on retention: a framework for developing reciprocal value propositions. journal of library administration, 57(3), 311-326. https://doi.org/10.1080/01930826.2016.1243425     evidence based library and information practice commentary   making meaningful connections and relationships in cataloguing practices: the decolonizing description project at university of alberta libraries   sheila laroque digital discovery librarian edmonton public library edmonton, alberta, canada email: sheila.laroque@epl.ca   received: 1 may 2018                                                                   accepted: 24 oct. 2018      2018 laroque. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29440     overview this paper seeks to examine some of the history behind the work that led to the decolonizing description working group (ddwg) and the efforts that have come from the further decolonizing description project at the university of alberta libraries (ual). within universities and a variety of memory institutions, there has been a shift since the calls to action of the truth and reconciliation commission (trc) report (2015). this paper seeks to give those who are interested in this type of work some insight into the processes that have been underway at the ual, and into ways that this could be replicated within their own institutions.   truth and reconciliation in libraries the decolonizing description project at the ual flows from the recommendations of the final report of the ddwg. the work of this group seeks to examine our classification systems and explores new approaches of creating new, more accurate and appropriate subject headings within our classification schemes. it is the emphasis on the ways of how to go about investigating more respectful ways of building relationships with indigenous communities that will make this type of work successful. it is not enough to simply put in additional efforts for “including” indigenous peoples, but rather to build relationships with indigenous peoples. the release of the calls to action of the truth and reconciliation commission allows us to reimagine the ways that we work alongside indigenous peoples. in order to have truly reciprocal and respectful relationships we need to allow as much time and space as is required for this to happen. in the words of justice murray sinclair, “it was education that got us into this, and it will be education that gets us out” (anderson, 2016). this same sentiment can be applied to the ways that libraries use broad and inappropriate subject headings to identify vast groups of people, such as ‘indians of north america’. librarian of the past misclassified and misunderstood these seemingly innocuous ramifications, which has led to this erasure and exclusion for future generations of scholars. in order to better understand how best to work with indigenous communities and knowledge, librarians will need to begin a time of education on these matters.     decolonizing description project background as i began my work in my position as an academic librarian resident, with a focus on the decolonizing description project, it became quite apparent that i would need to centre the relationships that ual already has with indigenous communities throughout alberta, as well as with other librarians who had taken on a similar project. to begin, it was critical to reach out to other librarians who have been working with indigenous knowledge and subject headings. deborah lee from the university of saskatchewan libraries was one of those people, and she gave me the idea of hosting the making meaning symposium at ual. many librarians, scholars and students have been having conversations about the ways that our library classification systems do not correctly represent or reflect indigenous knowledge, and this makes accessing these resources through the library very difficult. the intention was to gather together many people who would be affected by changes to classifying indigenous knowledge, and to begin having the necessary first conversations to begin this work.   in recent years, there has been a shift across many libraries and library systems to change the ways that librarianship is practiced, with a greater emphasis on community-based processes. while it is one thing to be able to reach out to and build a greater sense of community among people who already use the library, it is another thing altogether to reach out to those who have traditionally been excluded. this omission and failure to take into account different ways of knowing has been entrenched in library classification systems. this is why relationship-building is a critical aspect of a technical project. it reaffirms that librarians are not experts when it comes to indigenous knowledge, and that we have much to learn when it comes to the ways that we serve indigenous communities. in the beginning, it may seem like a significant gap that needs to be bridged, but the payoffs from respectfully moving forward with indigenous communities will only ever change libraries for the better.   the ddwg was formed in 2016, made of several members of ual representing different aspects of librarianship. there were people involved in public service, as well as metadata and cataloguing specialists. the ddwg spent a year investigating the ways that descriptive standards can be applied in ways that more accurately and respectfully reflect indigenous peoples and indigenous knowledges. more has been written about the decolonizing description project in other venues (farnel, koufogiannakis, laroque, bigelow, carr-wiggin, feisst, & lar-son, 2018) and the project seeks to continue to build relationships between indigenous communities and the ual. by viewing these communities as equal partners within the project, a relationship built upon mutual respect will be beneficial for users of libraries for years to come.   first steps there have been many lessons learned in the brief time that the decolonizing description project has been underway. the first and foremost would be to start with building upon the connections we already have, and to do everything we can to incorporate what has already been done while respecting our relationships. the ual is in a unique position as a leader in moving forward with these project goals, due to the relationships that it has with affiliated tribal college libraries through the first nations information connection (fnic) as well as through the neos library consortium. interestingly, neos is not only an acronym that can mean networking edmonton’s online systems, but also a nod to the greek word neos, meaning new (neos faqs). it is within these new and connected resources that we hope to be able to continue this work as it gains momentum. libraries are known for sharing resources, and while we hope to be able to accomplish this with our current partners, there is always potential for expanding our circles and working toward building new partnerships. it is also essential to reflect this type of outwards work for other institutions that would be interested in achieving similar goals within their own unique local contexts.   integrating community knowledge there are many steps that can be taken for the consultation process for this project. it is best to be able to take the time at the beginning of the project, so that we are going forward with communities in a good way. in thinking about who is included in ‘indigenous community members,’ it is important for us to recognize that we have a unique and dynamic indigenous community on our campus. many indigenous people come from a variety of locations, not just across alberta but from across the world, to study and work here. these indigenous students, staff and faculty are experts in their own communities, and can have much to teach us about how we can better shape library services with them in mind. the people who are using our library are also experts in the ways that our systems have either helped or hindered their research processes. by spending the time to have more meaningful relationships with these people, we will also gain a better understanding of the more nuanced ways that classification can be done more meaningfully. in addition, the consultation process will also seek to understand the best term to replace ‘indians of north america’ with, and to continue to have this conversation in the future. relationships are not stuck in time, and neither should the subject headings and classification systems that we use be. by understanding the areas of where we need to make improvements as an institution, we will only ever help our researchers do better work, and with less frustrations. these are the first steps towards creating a more effective working library for indigenous people, where respect and reciprocity are inherent.   it should also be noted that while ual are beginning with investigating what the people within our immediate campus community have to say, we will also be looking to bolster our existing relationships with fnic partners and beyond. indeed, these partner institutions also represent some of the same communities that our on-campus students come from. if the ual and university of alberta more broadly are committing to working with indigenous communities, then it only makes sense to honour this commitment by being present in communities, wherever that happens to be logistically. these future consultations, whether they appear in a survey format or otherwise, can be embedded as part of the library services that are shared throughout the consortium. likewise, these future consultations can appear on the library websites and remain up, as a means of ongoing consultation and service improvement. we expect that the number of responses from these surveys will change throughout the year, for example when information literacy classes make the intentions of these surveys known, or when more community visits are conducted. it should also be stated that the intention of these surveys is never going to be to make more work for the institution or individual instructors. rather, this is part of an ongoing benefit of sharing resources in a consortium environment.   these consultations will be building upon the work that has been started by the manitoba archival information network (main) in their project to create more accurate archival subject headings. other institutions have been engaging in similar work, notably the university of british columbia. other libraries are at the beginning phases of investigating what this type of project might look like, and there is much to be gained from working together and sharing our processes and lessons learned as we are all seeking to move forward. while we are building relationships at our own institution, it is also critical to build relationships with others who have taken on similar projects. while we are undergoing an education process to understand the ways that we can learn from indigenous communities, we can also engage in teaching and learning processes from other institutions. this learning goes beyond trying to establish ways of creating new subject headings and extends to shaping engagement and library services as well. this is not to say that efforts lobbying for systematic changes are not welcome in future processes of this project, but rather, a diversity of tactics grounded in local contexts will be the most effective way of creating long-lasting change across the library landscape.     making meaning symposium in starting the decolonizing description project, it was important to be able to begin this work grounded in and incorporating as many voices of the people who will be most affected by the project. gathering many people, both who work in communities as well as in libraries, quickly became the best way that we saw to move forward with this project. while at the surface, the decolonizing description project can be seen as a technical, cataloguing project, it is much more than that. the intention of the project is to change the ways that indigenous knowledge is represented, reflected and respected within libraries. our social biases have been reflected within classification systems, and we cannot only rely on our technical skills to solve these social problems that libraries have helped to reinforce in trying to achieve a balance between interests of community members, as well as with the more technical aspects of what will be needed in order to implement this project. based on the feedback that we have received, it was surprising to see that people were interested in more of the technical details and in really discussing what and more importantly how these changes will be made possible. the time for a project like this has come, and many people within libraries have known about the problems around classification for a long time. the focus now can be on how we will actually implement these changes, and to go forward with community partners.   it was the integration of a sharing circle format that really made the symposium a meaningful event. in a sharing circle, people are given the opportunity to speak in a safe, but open and facilitated discussion. there were three different circles throughout the two days, in order to capture people’s thoughts and perceptions throughout the event. we were interested in understanding how people were identifying the communities they come from, including but not limited to an indigenous identity or occupation-related identity, but also how these perceptions may have changed as the conversations continued. another aspect that stood out for many people was the indigenous student/scholar panel on the second day. the chance to hear from students themselves about how subject headings and classification have affected and hindered their academic work was very eye-opening for many people. we are aware that our current subject headings are inadequate, but having an understanding of exactly how they reflect on the academic work of our indigenous students is very powerful. it is well known in different learning institutions that indigenous students do not feel supported or reflected, and we can see that lack of representation reflected in every aspect of learning institutions, including in the way we organize information in our libraries.   the next steps of this project will be to continue to have a presence and build a culture of respect and learning in our partner communities. this includes the creation of additional partnerships as well as respecting where these new directions of service may take us. this also involves a great deal of transparency in every step of the way, in order to avoid mistakes such as those that have been created through unequal research relationships in the past. we are also hoping that by seeking to continue all of the conversations that have been started with the making meaning symposium, that we are able to spark interest in a variety of libraries. what works in one particular context might not necessarily work in others, but the intention of creating new, systematic examinations of how we can engage with indigenous people and knowledges better in the future is something that all memory institutions are being called to do. gaining momentum within learning institutions such as libraries can only seek to create further change in a variety of institutions, as we will be showing students that new ways of creating change is indeed possible.   going forward, the relationships that we build with other libraries will also help to bolster the new relationships that we will foster with our existing vendors. being able to understand the potential problematic subject headings across a variety of local contexts will help us to prevent materials from needing to be reclassified in the future, as will having strong connections with other libraries as we go forward with newer understandings of how we want our classification systems to be. these vendor relationships are also an integral part of how people will be able to interact with materials within the library, and therefore these relationships will also need to be examined. by creating new workflows and relationships with our vendors, these new workflows will be presented in a more streamlined manner, as integrating indigenous knowledge into the library becomes a more critical part of what we do.   references   anderson, s. (2016, june 9). murray sinclair: education is key to reconciliation. rabble. retrieved from http://rabble.ca/blogs/bloggers/kairos-canada/2016/06/murray-sinclair-education-key-to-reconciliation     farnel, s., koufogiannakis, d., laroque, s., bigelow, i., carr-wiggin, a., feisst, d., & lar-son, k. (2018).  rethinking representation: indigenous peoples and contexts at the university of alberta  libraries. the international journal of information, diversity, & inclusion, 2(3).   neos library consortium. faqs. retrieved from https://www.neoslibraries.ca/wordpress/wp-content/uploads/2016/04/neosfaqandanswers.pdf   truth and reconciliation canada. (2015). honouring the truth, reconciling for the future: summary of the final report of the truth and reconciliation commission of canada. winnipeg: truth and reconciliation commission of canada.   microsoft word comm_helliwell.doc evidence based library and information practice 2007, 2:2    120 evidence based library and information practice     commentary    reflections of a practitioner in an evidence based world: 4th international evidence  based library & information practice conference      michelle helliwell  knowledge services librarian  library and knowledge management services  shared services (dalhousie school of nursing – yarmouth site, annapolis valley health, south  shore health, south west health)  wolfville, nova scotia, canada  e‐mail: mhelliwell@avdha.nshealth.ca       © 2007 helliwell. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.        the 4th international evidence based  library & information practice conference  (eblip4) was a wonderful experience on  many fronts. our hosts, joanne gard  marshall and school of information and  library science (slis) at the university of  north carolina chapel hill, were friendly  and gracious. the setting was lush and  green (appreciated by this canadian who, at  the time of the conference, was still waiting  for the leaves to come out); the scale  intimate and conducive to building  relationships; the content thought‐ provoking, and the debate friendly. it was  gratifying to see so many professional  librarians in one room who were interested  and dedicated to evidence based library and  information practice in a variety of settings.    i work in a non‐academic, non‐traditional  library environment in a library service  (emphasis on service) for three district  health authorities in rural nova scotia.  evidence based practice is something that  my colleagues and i try to do. my entire  purpose for coming to this conference was  to see what other practitioners were doing,  and what could be translated to my own  practice setting.    the conference is biased towards academic  health science library settings for a host of  reasons. this is not a criticism. evidence  based practice started in medicine, and it  was perhaps natural that health sciences  librarians would be exposed to its ideas and  rationales given the librarian’s role in that  movement. that being said, there was  plenty in the program for those working in  other contexts. the report of a journal club  in a swedish library, evidence based project  management tools, the investigation of the  mailto:mhelliwell@avdha.nshealth.ca http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:2    121 usage of library databases offsite, and the  use of rubrics as a tool for assessing  information literacy were just some of the  posters and presentations that could be  applied in a variety of library settings, both  traditional and non‐traditional.    from my perspective, one of the highlights  of this conference was the opening speaker,  margaret haines. her “three r’s” of  professional practice should be some sort of  mantra at lis schools: reflective practice, or  learning from one’s own practice; retrieving  evidence in order to learn from others; and  researching key questions to learn about  changes in practice (haines). i was  particularly gratified by the first of the “r’s”.  among the charges laid at the door of  evidence based practice is that it ignores  both experience and tacit knowledge of  one’s environment. the act of reflection,  which was highlighted repeatedly by  andrew booth and others, is a key  cornerstone of professional practice. i would  suggest, very humbly, that it is also a  marker of wisdom. it highlights not only  what one knows, but acknowledges gaps in  one’s knowledge as well.    another high point was andrew booth’s  passionate defense of evidence based  practice, both in his keynote as well as in his  excellent debate with t. scott plutchak.   acknowledging that eblip will have a  different research base from the ebm gold  standard and different types of evidence  available does not excuse it from having  none at all. his work with anne brice on  qualitative systematic reviews and the  spice (setting, population, intervention,  comparison and evaluation) model  provides useful tools that practitioners in a  wide variety of settings, from public  libraries to corporate intranets, can harness  to sharpen their practice.(booth and brice)      if reflective practice is a cornerstone of  professional practice, eblip4 was a great  platform for opening participants up to  introspection. i found this in a number of  unexpected places, and frankly, it was  refreshing. there are times when i feel that  librarians have bought and nurtured the  stereotypes of selflessness and giving to the  user/patron without truly looking at what is  actually happening. are we as a profession  as user‐focused as we like to think?    this idea that librarians are not as patron‐ centred as we perhaps assume came up  several times in the course of the conference.  another highlight of the conference was jon  eldredge’s paper “cognitive biases as an  obstacle to effective decision making in  eblip.” his results, although preliminary,  suggest that librarians have two primary  cognitive biases in their practice: status quo,  preferring things as they are, and  deformation professionalle, or seeing the  world through the eyes and biases of one’s  profession (eldredge). this theme was also  underscored by several speakers in the final  lunch panel, “library leaders discuss the  pros and cons of evidence‐based practice.”  jean wilkins, the former head of the illinois  state library, and patricia thibodeau of  duke university medical center were  particularly candid about the challenges  associated with changing attitudes about  practicing by actually listening to what our  patrons are telling us they want from us  (thibodeau et al.). in a small discussion over  lunch with colleagues at my table, this  subject came up again: when patrons are  going around, over and through us, creating  their own information products and using  them, what does that say about our  relevance or our ability to listen to what our  clients want from us? eblip from a user‐ centered approach is vital to our relevance  as professional librarians.    the schism between research and practice  was well noted at the conference. as a  practitioner not attached to an academic  institution, i was in the minority at eblip4.  evidence based library and information practice 2007, 2:2    122 further complicating eblip is that when  research is done, it is usually done in  academic settings, which is of limited use to  someone in my particular practice. bridging  this gap between research and practice –  something i noted very early in my grad  school days – is going to take some  interesting partnerships between academic  and non‐academic institutions, and better  support and continuing education  to build  the skills required to do eblip.     another issue that arose during the  conference was the lack of strong, accessible  professional literature. there was some  contention at the conference about whether  eblip is actually new, or an old tie in a new  box. i think that if it is something librarians  have been doing all along, the profession  has done a remarkably poor job translating  research into practice. in many ways the key  here is accessibility. as a librarian who is  not attached to a research institute of any  kind, i have no access to the literature save  what a search of pubmed or google might  bring. this is, i think, an issue that lis  schools and associations must consider to  better support practitioners.       if we are going take margaret haines’  advice and reflect on our practice, we must  acknowledge these shortcomings. if eblip  will improve our practice in a way that is  genuinely useful to our clients, regardless of  the setting, we should welcome it. eblip4  was an excellent platform to look at these  important professional issues, and it is my  hope that in the coming years professional  librarians from a broader field of practice  settings will see the relevance in attending  future eblip conferences. this would make  the conference an even richer experience for  everyone, and give further life to eblip.        works cited    booth, andrew, and anne brice. “the  neglected voice: is there a role for  qualitative systematic reviews in  eblip?” 4th international evidence  based library & information practice  conference. sheraton imperial hotel  and convention center, durham. 9  may 2007.    booth, andrew, and t. scott plutchak.  “eblip: clear, simple – and wrong? a  friendly debate.” 4th international  evidence based library & information  practice conference. sheraton imperial  hotel and convention center, durham.  9 may 2007.    eldredge, jonathan. “cognitive biases as  obstacles to effective decision making  in eblip.” 4th international evidence  based library & information practice  conference. sheraton imperial hotel  and convention center, durham. 7  may 2007.     haines, margaret. “professionalism and  evidence‐based practice: reflections of  a university librarian.” 4th  international evidence based library  & information practice conference.  sheraton imperial hotel and  convention center, durham. 7 may  2007.      thibodeau, patricia l., jean wilkins, susan  k. nutter, lian ruan, and sandra  hughes‐hassell. “library leaders  discuss the pros and cons of evidence‐ based practice.” 4th international  evidence based library & information  practice conference. sheraton imperial  hotel and convention center, durham.  9 may 2007.   microsoft word class_hunsu.doc evidence based library and information practice 2007, 2:4  95 evidence based library and information practice       classic    more appropriate information systems and services for the social scientist: time to  put our findings to work       a review of:  line, maurice b. “the information uses and needs of social scientists: an overview of  infross.” aslib proceedings 23.8 (1971): 412‐34. rpt. in lines of thought: selected  papers. ed. l.j. anthony. london: bingley, 1988. 45‐66.  .     reviewed by:   r. laval hunsucker  vakreferent / collectie‐ en informatiespecialist, wetenschappelijke informatie  geesteswetenschappen, universiteitsbibliotheek, universiteit van amsterdam  amsterdam, the netherlands  e‐mail: r.l.hunsucker@uva.nl      received: 17 august 2007    accepted: 04 november 2007      © 2007 hunsucker. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – the study reported in this  article was conceived in order to answer a  question of very large scope: what are the  information systems and services  requirements of social scientists? inherent in  this question was the correlative question:  how do social scientists tend to use such  systems and services, and what resources  and information access approaches do they  by choice employ? the choice for such an  approach was well‐considered, given that 1)  there were at the time almost no research  results available in this area; 2) the  investigators feared that approaches  developed earlier for the natural sciences  and technology would be uncritically  adopted for the social sciences as well; and 3)  “the social science information system was  developing anyway, and if it was to develop  in appropriate ways, some guidance had to  be provided quickly” (412). the  investigation into information requirements  of the social sciences (infross) project  team believed that there was “no point” (412)  in embarking first on a series of more  narrowly focused studies. the express  intention was to derive findings that would  be usable “for the improvement of  information systems, or for the design of  new ones” (414). for more on the projectʹs  conceptual underpinnings, see line’s  “information requirements.”    mailto:hunsucker@uva.nl http://creativecommons.org/licenses/by/2.0 http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:4  96 design – exploratory study employing both  quantitative and qualitative approaches  over a period of three and a half years,  beginning in the autumn of 1967.    setting – the whole of the united kingdom.  the project was funded by that country’s  office for scientific and technical  information (osti), which had been  established in 1965.    subjects – almost 1,100 randomly selected  academic social science researchers, plus a  substantial number of government social  science researchers and social science  “practitioners” (“college of education  lecturers, schoolteachers, and individuals in  social work and welfare” [413]). for the  purposes of the study, the social sciences  included anthropology, economics,  education, geography, political science,  psychology and sociology, but numerous  historians and statisticians ultimately  participated.    methods – three methods were employed:  surveys, interviews, and direct observation.  a “very long” (413) questionnaire was sent  to 2,602 of the identified ca. 9,100 social  science researchers in the united kingdom,  with 1,089 (41.8%) completed questionnaires  returned. two pilots were conducted with  the questionnaire before a definitive version  was finalized for the study. seventy‐five  interviews were conducted (individually or  in groups) with researchers, some of whom  had received but not responded to the  questionnaire, and some of whom were not  included in the questionnaire sample. the  interviews with non‐responding persons in  the sample were for purposes of  determining “whether they were non‐ typical” (413). fifty additional interviews  were conducted (individually or in groups)  with practitioners. day‐to‐day observation  of a small number of social scientists was  undertaken in the context of a two and a  half year‐long experimental information  service at bath university – the first time  any uk university had employed  information officers for the social sciences.     main results – the results showed a  pronounced perception among social  scientists that informal “methods of locating  references to relevant published  information” (416‐8, 426‐7, 431) are more  useful than formal methods (such as  consulting the library catalogue, searching  library shelves, or searching in indexing and  abstracting publications), and an even more  pronounced inclination to actually use such  informal methods – something of a  revelation at the time. less than one sixth of  all sociologists, for example, made use of  sociological abstracts. on both counts,  “consulting librarian” (418) scored worse  than all the other ten options. forty‐eight  percent of respondents never did it, and  only 8% perceived it as a “very useful” (418)  method. nonetheless, 88% of respondents  were in principle prepared to delegate at  least some of their literature searching, and  approximately 45% all of it, “to a  hypothetical information officer” (425).  more than 75% of the experimental service  clients also responded affirmatively to the  question: “should a social science  information officer be a high priority,” given  limited available resources? (line,  cunningham, and evans 73‐5). most  subjects found, in any case, that their major  “information problems” (427‐8) lay not in  discovering what relevant documents might  exist, but rather in actually getting their  hands on them. in only around 20% of the  cases were they ultimately successful in  doing so. the younger the researcher, the  greater the dissatisfaction with her/his own  institution’s collection. this study also  revealed that academic social scientists drew  little distinction between information needs  for their research and those for their  teaching.    evidence based library and information practice 2007, 2:4  97 there was one social science discipline  which clearly stood out from the rest:  psychology. psychologists were the heaviest  users of abstracting and indexing (a&i)  publications, as well as of the journal  literature, published conference proceedings,  and research reports. they were also the  least tolerant of time lags in the a&i  services’ coverage of new publications.    further significant findings were:    • a librarian’s way of categorizing  research materials was not very  meaningful to the researchers  themselves.  • a&i services were generally used more  often for ‘keeping up’ than for  retrospective searching.  • consultation with librarians was more  common in the less scholarly and more  intimate college environment than at  research institutions.  • a large percentage found library  cataloguing insufficiently detailed. the  same was true for book indexes.  • there was considerable enthusiasm for  the idea of a citation index for the social  sciences. (n.b.: the ssci began  publication two years after the  appearance of this article.)  • among informal methods of scholarly  communication and information  transfer, conferences (to the  investigators’ surprise) rated  remarkably low.  • researchers with large personal  collections made more use of the library  and its services than those with small  collections.  • social scientists had little interest in  non‐english‐language materials. line  speaks of “a serious foreign language  problem” (424).    the infross study produced an enormous  amount of data. only 384 of the computer  tables produced were made available in 4  separate reports to osti. only 3 tables, 2 of  which were abbreviated, appeared in this  article. the further raw data were available  on request.    conclusion – line himself was exceedingly  cautious in drawing explicit positive  conclusions from the infross results. he  even stated that, “no major patterns were  detected which could be of use for  information system design purposes” (430).  he was freer with his negative and  provisional assessments. two years earlier  he had written: “it still remains to be  established that there is an information  problem in the social sciences, or that, if  there is, it is of any magnitude”  (“information requirements” 3). however,  it was now clear to line that information  services and systems for the social scientist  were indeed quite inadequate, and that  (potential) users were not satisfied.     he was, furthermore, prepared to go out on  a limb with the following assertions and  inferences:      1) it was a great strength of infross that it  had – in marked contrast to previous science  user studies – generated “a mass of  comparable [his italics] data within a very  broad field, so that every finding can be  related to other findings” (430).     2) there are discernable – and exploitable –  differences in the information needs and use  patterns among the different social science  disciplines (which he often also refers to as  the different “subjects”).     3) infross had likewise made more  evident the nature of similarities across  disciplines.     4) there is indeed, from an  information/library perspective, a  continuum from the ‘harder’ to the ‘softer’  social sciences.   evidence based library and information practice 2007, 2:4  98 5) social scientists showed too little  awareness, made too little use, and even  displayed “insufficient motivation” (431) to  make use of available information  systems/services. he elsewhere (“secondary  services” 269, 272) characterizes them as  “remarkably complacent,” “even apathetic.”     6) there is good reason to doubt the wisdom  of libraries’ investing in user education,  since it is bound to have little effect (for  further discussion of this matter, one can  consult his “the case for” 385‐6 and  “ignoring the user” 86).     7) user‐friendly systems amount inevitably  to underdeveloped and ineffective systems –  and therefore “personal intermediaries,” in  sufficient numbers, will remain essential if  we wish to offer social scientists really good  information services (426, 431).    line believed that infross was only a  beginning, and he had already, even before  writing this article, begun follow‐up  research aimed at attaining results really of  use for information system design purposes  (e.g., the disiss project). he complained  many years later, however, that all this  research “indicated means of improvement,  but led to no action” (“social science  information” 131). in any case, “bath” (the  common shorthand subsequently used to  refer to all this research) became, and has  remained, the starting point for all  subsequent discussions of social science  information problems. several years ago,  there was a well‐argued international call  for “a new and updated version of the  infross study” – with an eye to finally  using the findings for practical purposes,  and aiming “to extend and follow up the  research agenda set by the original study”  (janes “time to take”).        commentary    the caution with which line apprized the  reliability and generalizability of his  project’s results was decidedly exaggerated.  rather, it has the appearance now, with the  benefit of hindsight, of being decidedly  exaggerated. such reserve was, in the 1971  context, perhaps less out of place, as even  line himself later (e.g., “social science  information”) realized that his judgment  had proven too cautious. that is largely  because so much of what the infross data  was – or seemed to be – suggesting has,  since then, been time and again confirmed  by subsequent studies in the uk and  elsewhere. even in its own time, i would  guess that there was an intuitive feeling in  our profession that infross was very  much on the money, whether we liked to  admit it or not. this, together with the fact  that it was the first large‐scale information  needs, seeking and use (“insu”) study  outside of the natural sciences and  technology, could help explain why the  impression it made was immediate and  widespread.    yet, what i find so noteworthy about line’s  article of thirty‐six years ago is not only that  he offers us a succinct overview of a very  ambitious, carefully executed and  compelling study, but that he along the way  addresses some of the important  methodological challenges for our field’s  research in general. line makes cogent  observations on some of the most  fundamental quandaries of our professional  practice, ones which are still very much with  us. one paradoxical example of the latter is  (dis)intermediation. most users would have  liked, infross found, to delegate their  searching – but didn’t see the librarian as an  appropriate intermediary. for line, this  meant a tragic impediment to developing  “far more efficient and effective retrieval  systems” (426). he concluded, “indeed, i  would go so far as to say that until this  evidence based library and information practice 2007, 2:4  99 question of the intermediary is settled one  way or the other, it is extremely hard to  know where to go in the development of  information systems” (426). the technology  is now vastly different, but we are still not  much closer to a solution, although line’s  own proposed solution could at the same  time alleviate the greatest information  problem of all: the “paradox of the active  user” (carroll & rosson) – and perhaps even  largely abrogate our classic  “information/instruction” dilemma (e.g.,  rettig). today’s practitioner might do very  well to honestly reflect upon what line’s  reasoning should mean for current service  configurations.    line’s insights do not end there. he  emphasizes the potentially critical  significance of the factor personality in  information systems use, the phenomenon  of the “power user” (without employing  that term), the vital function of content as  “stimulus for ideas” (424) rather than as  “information,” and the importance of  “accidental discovery” for system design.  he discusses at some length the serious  problem of the research/practice gap in the  social sciences. perhaps most striking is his  advice that our profession should design  information services that incorporate the  virtues of both the informal communication  channels which users prefer, and the  powerful formalized kind of information  systems which we have traditionally  espoused (see also his “information  requirements” 14). he makes, moreover,  perspicacious comparisons of humanities  and social science research, disarming the  cliché that the latter is in general ‘harder’  (i.e., more like science and technology) than  the former.    in the methodological department, he  acknowledges that user studies such as his  can tend to measure the level of awareness of,  rather than the extent of the existence of,  information problems; that respondents  usually have no standard against which  usefully to critique existing systems; and  that expressions of desiderata tend to be  conditioned by expectations. revealing as  they may be, their predictive value is  relatively limited. he stresses also the  dangers of not combining qualitative with  quantitative methodologies in lis research,  the difficulty of achieving meaningful meta‐ analyses, and the importance of sufficient  project funding. this article, the entire study,  and in fact much of line’s long career as  prolific lis scholar and distinguished lis  practitioner breathe abundantly the spirit of  an evidence‐based approach to library and  information work (in this regard, see also  his eloquent avowal at “le métier” 48).    summarizing in a single journal article (the  planned “book presenting the results of  infross in extended form” (432) never  materialized) such an enormous, pioneering  research project and its findings must have  been quite a tall order, even for maurice  line – and some criticism of this resulting  publication is not out of place. line states,  for example, that the investigators started  out with certain hypotheses in mind that  they wanted to test – but fails to make clear  just what they were. the tables, being  compressed versions of those appearing in  the full reports, are not always sufficiently  easy to interpret. also unfortunate is that  line speaks of “information uses” (e.g., 412,  415, 427) and the “information user” (e.g.,  430) without making perfectly clear what he  has in mind. what the study addressed is  the use of systems, methods, publications  and services for discovering and accessing  potentially useful information – not what the  social scientist then actually does with the  information she or he has found (when, how,  in what combinations or synthesis, and to  what effect or for what purpose). this  terminology has (unfortunately) become  common shorthand in our field, but we  might have expected more from someone  who was not only very conscious of the  evidence based library and information practice 2007, 2:4  100 importance of this distinction, but had  already (“ends and means”) made a striking,  even radical, case for librarians’ developing  more insight into how their clients really use  information. this is what blagden (27) calls  the “expanded” definition of “use,” and he  reemphasizes that it rarely plays a role in  user studies. that was still true in 1980, and  has (again, unfortunately) been true ever  since. as a fourth point of critique, we might  observe that in our article line likewise  gives insufficient explicit attention to the  concept of actual information needs as  opposed to wishes, requests or expectations.  we know from his other writings that he  was much occupied with the implications of  these distinctions and held that librarians  should often base what they do on their  perception of user need even when that is in  conflict with user wishes or requests.    the infross program itself displayed  certain shortcomings. the practitioner (as  opposed to the researcher) samples were not  random, social scientists in the commercial  sector were not even included, and the  investigation of informal channels was less  thorough than that of formal ones. the  attempt to shed light on the subjects’  research processes (how they were “going  about” their research) was also largely  unsuccessful, not least because the  investigation’s ethnographic component  was too limited. line was aware of these  drawbacks, but pleaded – not unjustifiably –  a shortage of time and funding.    but such critiques have little relevance for  the heart of the matter. what is the  importance, then, of line’s article for library  and information services (lis) practice today?  i would submit that its importance lies  especially in its being such a panoramic  digest – perhaps the most panoramic  available digest based on actual empirical  research findings ‐ of the many respects in  which we can perceive that the kind of lis  that probably would be most appropriate  and beneficial for the social scientist (and  many other classes of user?) is not the kind  of lis that most of us have been offering –  or willing or equipped to offer, even to this  very day. the article, moreover, is sprinkled  with still very pertinent methodological  caveats, and served up with the  characteristic linean modesty,  understanding of human nature, and  waggish humour. this practical significance  is increased by the fact that the publication  in question dates back to the early seventies,  and was widely noted from the beginning.  we cited above line’s 1999 observation that  infross should have but didn’t lead to any  positive action, janes’ 2005 proposal for  “finally using the findings for practical  purposes,” and noted that infross’  messages have in the meantime been  individually reinforced by numerous  narrower studies. altogether, this amounts  to eloquent testimony to the validity of the  following remark by line:    however, even when several surveys,  carried out at different places and  times, point clearly in the same  direction, librarians appear to be either  unaware of the findings or reluctant to  believe them or unwilling to act upon  them. (“ignoring the user” 83)    a more fundamentally practical point is  hardly imaginable. there comes a time  when the courage (call it what you like) has  to be found really to use such findings,  finally, as janes suggests, for practical  purposes. the integrity and legitimacy of  our profession ultimately depends on doing  so, i dare say. if this message is not yet  forceful enough, we can call to mind some  conclusions from the investigations reported  by swift and his colleagues (215‐6) that the  systems librarians traditionally have been  offering are “inappropriate for social  scientists,” “do not take social science needs  into account,” and “interfere with the  knowledge generation process” – whence  evidence based library and information practice 2007, 2:4  101 the “widespread unwillingness” to use them,  as they are more apt to “impede the user”  than to help.    one does not come forward with a heavy‐ duty proposal such as janes’ without good  reason. since 1971, no study has been  conducted with anything like the scope and  thoroughness of infross. an “updated  version” – as recommended to ifla (janes,  “time to take”) – would be invaluable if it  reinforced the original findings so that  practitioners could then confidently accept  them as at least provisionally axiomatic, and  at last begin to act upon them. it could also  suggest to us how the tremendous technical  developments of the last forty years have  genuinely affected social scientists’  behaviour and requirements. it could also,  of course, indicate where line’s results and  inferences are not yet adequately shown to  be reliable for modification of practice, and  where further targeted research is therefore  required (an almost ideal practical point of  departure, one is tempted to suggest).    the line of reasoning presented here gains  even more cogency when we consider that  broad‐scale user studies have been lacking  in our field now for a long time. janes  (“time to take”) notes this fact, and wilson  (“revisiting user studies” 680‐1) recently  remarked that “there is a need for research  programmes lasting for several years, rather  than numerous one‐off projects.” “[s]mall‐ scale” studies, wilson adds, have worked  against pursuing “in‐depth research”  (“revisiting” 680‐1). he also rightly  observes that (non)user motivation is “an  area barren of research” (“revisiting” 682).  from 1966 through 1974 (excepting 1973),  the annual review of information science and  technology published in each installment a  comprehensive review chapter on  information needs and uses studies.  thereafter, this summary appeared once  every 4 to 6 years, until the series dried up  with the 1990 volume, whereafter only more  narrowly directed reviews appeared (none  for the social sciences). there was  furthermore a shift to the concept  “information behavior” – a concept which  the editor apparently considers to be  narrower, given that case’s 2006 review  “information behavior” is grouped with  other (limited‐scope) reviews under the  section heading “information needs and  use.” those interested in a recent classified  bibliography of social science information  needs and uses studies may refer to janes,  “ifla bibliography.”    obviously, an updated study would have to  be quite carefully designed, as it would  necessarily amount to more than a  replication. even apart from the questions of  technological advancements and altered  types of working environments since 1971,  appropriate (meta)theoretical approaches to  insu research developed since then must  also be taken into account. to my mind,  there should also be more extensive  ethnographic and participatory‐research  elements. one must avoid any return to a  situation of “conceptual impoverishment”  (dervin & nilan 3), from which user studies  too often have suffered, or of a lack of  “theoretical underpinning” (wilson,  “information needs”). most importantly,  the research results ought all to be  interpretable in the context of practical  application to service improvement.    characterizing the state of affairs before the  infross study, line had written:  “information requirements in the social  sciences are almost entirely unexplored”  (“information requirements” 1). his  landmark study here reviewed was soon  profoundly and conspicuously to alter that  situation for good: a ’classic’ study if there  ever was one.       evidence based library and information practice 2007, 2:4  102 works cited    blagden, john. do we really need  libraries : an assessment of  approaches to the  evaluation of the  performance of libraries. new york:  saur; london: bingley, 1980.    carroll, john m., and mary beth rosson.  “paradox of the active user.”  interfacing thought: cognitive aspects  of human‐computer interaction. ed.  john m. carroll. cambridge, ma: mit  press, 1987. 80‐111.    case, donald o. “information behavior.”  annual review of information science  and technology 40 (2006): 293‐327.    dervin, brenda, and michael nilan.  “information needs and uses.”  annual review of information science  and technology 21 (1986): 3‐33.    janes, mark. “ifla bibliography on social  science information needs and uses  studies.” iflanet ‐ social science  libraries section. 12 jan. 2005.  international federation of library  associations and institutions. 31 oct.  2007  <http://www.ifla.org/vii/s5/project/pro ject_info‐needs.htm>.    ‐‐‐. “time to take another bath? a  preliminary report on the feasibility of   repeating the infross study.”  iflanet ‐ social science libraries  section. 12 jan. 2005. international  federation of library associations and  institutions. 31 oct. 2007   <http://www.ifla.org/vii/s5/project/pro ject_infross.htm>.    line, maurice b. “the case for information  officers.” educating the library user.  ed. john lubans, jr. new york: bowker,  1974. 383‐91.    ‐‐‐. “ends and means – librarianship as a  social science.” the library world:   a medium of intercommunication for  librarians 66.779 (1965): 270‐5.    ‐‐‐. “ignoring the user: how, when and  why.” the nationwide provision and  use of information : proceedings:  aslib/iis/la joint conference, 15‐19  september 1980 sheffield. london:  library association, 1981. 80‐8.    ‐‐‐. “information requirements in the social  sciences: some preliminary   considerations.” journal of  librarianship 1.1 (1969): 1‐19.    ‐‐‐. “le métier de bibliothécaire: un  ensemble de pratiques confuses et  discontinues.” bulletin des  bibliothèques de france 43.2 (1998): 44‐ 8.    ‐‐‐. “secondary services in the social  sciences: the need for improvement  and the role of librarians.” behavioral  & social sciences librarian 1.4 (1980):  263‐73.    ‐‐‐. “social science information ‐ the poor  relation.” inspel 33.3 (1999): 131‐6.    line, maurice b., dawn cunningham, and  susan evans. experimental  information service in     the social sciences, 1969‐1971: final  report. [bath]: bath university library,  1972.    rettig, james. “the convergence of the  twain or titanic collision? bi and  reference in the 1990s’ sea of change.”  change in reference and bi: how  much help and how? :   http://www.ifla.org/vii/s5/project/project_info%e2%80%90needs.htm%00 http://www.ifla.org/vii/s5/project/project_info%e2%80%90needs.htm%00 http://www.ifla.org/vii/s5/project/project_infross.htm http://www.ifla.org/vii/s5/project/project_infross.htm http://www.ifla.org/vii/s5/project/project_info%e2%80%90needs.htm%00%00 http://www.ifla.org/vii/s5/project/project_info%e2%80%90needs.htm%00%00 evidence based library and information practice 2007, 2:4  103 papers and session materials  presented at the twenty‐second  national loex library instruction  conference held in ypsilanti,  michigan, 13 to 14 may 1994. ed. linda  shirato and rhonda fowler. ann  arbor, mi: pieran press, 1996. 1‐14.    swift, donald f., viola a. winn, and dawn  a. bramer. “a sociological approach  to the design of information systems.”  journal of the american society for  information science 30.4 (1979): 215‐23.    wilson, t.d. “information needs and uses:  fifty years of progress?” fifty years of   information progress: a journal of  documentation review. ed. brian c.  vickery. london: aslib, 1994. 15‐51.    ‐‐‐. “revisiting user studies and  information needs.” journal of  documentation 62.6 (2006): 680‐4.            microsoft word art_pearce.doc evidence based library and information practice 2006, 1:3  44 evidence based library and information practice     article    a  randomised  controlled  trial  comparing  the  effect  of  e‐learning,  with  a  taught  workshop, on the knowledge and search skills of health professionals       nicola pearce‐smith  department of knowledge and information science  badenoch building  old road campus  headington  oxford   e‐mail: nicola.pearce‐smith@dphpc.ox.ac.uk      received: 10 may 2006    accepted: 15 august 2006      © 2006 pearce‐smith. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    abstract    objective ‐ the aim of the trial was to establish whether there is a significant difference in  terms of knowledge and skills, between self‐directed learning using a web‐based resource,  compared with a classroom based interactive workshop, for teaching health professionals  how to search. the outcomes measured were knowledge of databases and study designs,  and search skills.    methods ‐ the study design was a randomised controlled trial (rct). 17 health  professionals were randomised into one of two groups –  one group (eg) received access to  a search‐skills web resource, and the other group received a search workshop (wg) taught  by a librarian. participants completed pre‐ and post‐intervention tests involving multiple  choice questions and practical searching using clinical scenarios.    results ‐ 9 wg and 6 eg participants completed both pre‐and post‐intervention tests. the  test results were blindly marked using a score chart developed with two other librarians.  for question formulation and devising a search strategy, all participants obtained a score  that was the same or better after receiving the intervention (both wg and eg), but statistical  analysis showed that the only significant outcomes were for the wg devising a search  strategy (p=0.01) and preferring to search using mesh after receiving the taught workshop  (p=0.02). the mann‐whitney test showed there were no significant differences in any of the  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:3  45 outcomes (p>0.05), between the wg and the eg. the statistical analyses must be viewed  with caution due to the small sample size.    conclusion ‐ there were no significant differences in knowledge of databases and study  design, or search skills, when the wg and the eg were compared. although many  participants obtained a score that was higher post‐intervention, only devising a search  strategy and preferring to search using mesh were statistically significant for the wg. the  question of whether a taught workshop and an e‐learning module are of equal effectiveness  in teaching search skills, is an important one for health librarians involved in user education,  and was a justifiable topic to propose and conduct research. the fact that the results are  mainly inconclusive due to the small sample size is disappointing, but does not diminish  the importance of conducting the study.     background    searching for evidence is an essential skill  for any health professional wishing to apply  evidence to practice. training health  professionals to search databases is thought  to improve their skills and knowledge, but  evidence of this in the literature is not  extensive. a systematic review in early 2003  found that there is some evidence of a  positive impact of search skills training for  health professionals (garg and turtle 33‐41);  however, the studies included were small or  methodologically poor.     a randomised trial published after this  review found that a 3 hour educational  workshop was more effective than no  training in improving question formulation  and use of databases by doctors (cheng 22‐ 33). more recently, a study measuring the  effect of an evidence‐based practice  workshop on the knowledge, skills,  behaviour and attitudes of occupational  therapists (published in dec 2005 after  completion of this trial), also found  improvements in knowledge of, attitudes to,  and confidence with searching and  appraisal (mccluskey and lovarini).   health librarians regularly teach directional  search workshops, but do not know whether  these social cognitive learning methods are  more effective than other educational  methods. self‐directed online, or e‐learning,  is becoming more widespread, but there is  little evidence to show whether this is an  effective training method. indeed, very few  studies were found examining the use of e‐ learning as a method of teaching health  professionals in any subject areas. a review  of e‐learning in continuing medical  education found that 6 out of 16 studies  involving internet‐based education of health  professionals showed a positive change in  participant knowledge when compared with  traditional teaching formats; the remaining  studies showed no difference in levels of  knowledge(wutoh, boren, and balas 20‐30).  in 2005, a qualitative study looking at a  randomised trial comparing directed  learning and self‐directed learning, found  no statistically significant differences  between the two groups in terms of  evidence‐based knowledge, skills, and  attitudes of medical students (bradley et al.  149‐77).    as the literature on this subject is not  extensive, the author designed and  conducted a research study between sept  2004 and sept 2005. a research proposal was  developed and a grant received from the  helicon research in the workplace award  2003/04  (http://www.ifmh.org.uk/riwa2003‐ 4.html). this paper presents the methods  and results of a rct comparing two  different educational interventions for  http://www.ifmh.org.uk/riwa2003%e2%80%90 evidence based library and information practice 2006, 1:3  46 teaching search skills to health  professionals.    objectives    the aim of the trial was to establish whether  there is a significant difference in terms of  knowledge and skills, between self‐directed  learning using a web‐based resource  directed by participants, compared with a  classroom based interactive workshop  directed by a librarian, for teaching health  professionals how to search.     a common method of proceeding is to test a  hypothesis, often called the null hypothesis.  the null hypothesis is that there is no  difference between the two interventions  being compared (i below). it was also  relevant to have a second null hypothesis –  that there would be no difference before and  after either educational intervention (ii  below)    i) there will be no difference between those health  professionals receiving an online learning  intervention using a web‐based resource, and  those receiving a classroom based workshop  intervention, in terms of knowledge of databases  and study designs, and the skills of formulating  a question, designing a search strategy and  selecting appropriate citations    ii) there will be no difference in terms of a health  professionals’ knowledge of databases and study  designs, and the skills of formulating a question,  designing a search strategy and selecting  appropriate citations, before and after an  educational intervention    testing these hypotheses involves a  comparison of the effectiveness of two  different interventions, so a rct was chosen  as the most appropriate study design. the  outcomes measured were knowledge of  databases and study designs, and search  skills. ethical approval was obtained from  the relevant research ethics committee.  methods    the study population was a convenience  sample – health professionals working for  the oxfordshire radcliffe hospitals nhs  trust (orht). recruitment was by  invitation letter, sent by post to departments  and/or by email. contact details were  obtained from the trust intranet and the  email lists provided by trust librarians.  posters and leaflets about the trial were  displayed in the hospital.     recruitment was self‐selecting – participants  registered for the trial after responding to an  email, letter or poster. these participants  may have had more interest in the topic of  search skills than your average health  professional – this could be because they  have no searching skills and wish to learn,  or because they have attended similar  sessions before and are more experienced  searchers.     inclusion criteria were very broad –  participants must work within the orht,  and have access to the internet either at  work or home.     the trial    on arrival all participants completed a  “search exercise” using a computer. the  search exercise had two sections – the first  contained multiple choice questions (mcqs)  and was designed to test knowledge about  databases and study designs. the second  section involved a practical search using  pubmed (www.pubmed.gov) to test  participants’ skills at structuring a search on  a given scenario (see appendix 1).  participants were randomly assigned to one  of two groups, using computer generated  random numbers.   evidence based library and information practice 2006, 1:3  47     figure 1: flow chart showing trial stages and numbers of participants      those participants allocated to the  workshop group (wg) then attended a 2  hour search skills workshop taught by a  librarian (the author did not teach the  workshop session in order to prevent any  possible bias). those participants allocated  to the e‐learning group (eg) were shown  how to access the online learning module by  the author, and asked to complete as much  of the module as they could, in one week.  one week later, participants were sent (via  post and/or email) the second “search  exercise” containing a different scenario,  and asked to complete and return.     in order to minimise the possibility that one  scenario was easier to search for than the  other, the order in which participants  completed the search exercises was assigned       randomly, again by computer generated  random numbers.    interventions in more detail    the online learning resource was designed  by the author. it was made available on the  web, but was password protected so that  only eg participants could access it. the  content included question formulation,  study design, free text, thesaurus and  boolean searching, and examples of  searching pubmed and the cochrane  library.    before the trial, the online learning module  was piloted on a general practitioner and a  nurse. they were asked to comment on the  usability, the time taken to complete the eligible  participants (17)  randomised into two groups  ‐ randomisation done before attendance  search skills group workshop  taught by a librarian (9)  search skills electronic‐learning  module accessed via the internet (7)  completed second  search exercise (9)  completed second  search exercise (6)  complete participant characteristics form  complete search exercise 1 (randomised to  e‐group) did not  attend    module completed  individually at home or  work (about 1 hour over  one week)  1 loss to  follow‐up  group  workshop  lasts 2 hours  evidence based library and information practice 2006, 1:3  48                                       figure 2: screen picture showing the online learning module      module, and the content covered. these  suggestions were then incorporated into the  module ready for use in the trial.     the wg was taught by an experienced  librarian, who used methods such as  presentations, live internet demonstrations  and interactive group work. the workshop  was carefully planned by the librarian and  author to deliver content on exactly the  same topics covered by the online learning  resource. teaching examples were also the  same as those contained in the online  learning module. participants were able to  practice their own searches during the  session, with help and advice from the  librarian.        table 1: baseline participant characteristics        wg (n=9)  eg (n=6)  gender:          female        male  6  3  4  2  age (years):         21 ‐ 30       31 – 40       41 ‐ 50       51 ‐ 60  0  1  6  2  4  2  0  0  job:         doctor       nurse       allied health       manager  3  4  0  2  3  2  1  0  attended previous  search session?       yes in last 3 months     yes > 1 year ago      no  1  2  6  1  1  4  evidence based library and information practice 2006, 1:3  49 results    despite the randomisation procedure, the  ages of participants in the two groups  differed – those in the online learning group  were a lot younger than those in the  workshop group. stratified randomisation  could have been used to help keep the  characteristics of participants (e.g. age) the  same acrosss both groups, but this was not  undertaken – stratified randomisation is  quite complicated and requires certainty  about which participant characteristics  might influence the intervention outcome  (roberts and torgerson 1301).    marking of the search exercises    the author was ‘blind’ as to which search  exercises had been completed by a  particular person or to which group they  had been allocated. there was no indication  on the search exercises as to which had been  completed before or after the interventions,  to reduce any inadvertent bias. existing  scenarios/score charts for testing search  skills were available, but at the time of  planning, no one tool appeared to fit the  study requirements exactly (vogel, block,  and wallingford 327‐30;rosenberg et al. 557‐ 63;burrows and tylman 471‐76;bradley et al.  194‐201;dorsch, aiyer, and meyer 397‐406).    the author collaborated with two other  librarians to develop a ‘gold standard’ for  formulating a question from a scenario,  developing a search strategy and selecting  appropriate citations. this information was  used in conjunction with the tools in the  literature to develop a score chart (see  appendix 2); the exercises were graded in  each of the categories above according to a  scale excellent, good, fair, poor. each  exercise received a grading for question  formulation, search strategy and citation  selection, plus a mark for the number of  correct multiple choice questions (see tables  2‐7). a random selection of the search  exercises were marked by another librarian  in order to validate the marking; no  differences in scores were observed.    summary of scores    question formulation  • all participants received a score that  was the same or better after  receiving the intervention (both wg  and eg)    search strategy  • all participants received a score that  was the same or better after  receiving the interventions (both  wg and eg)  • the majority of wg increased their  score by at least 1 after receiving the  intervention    • most of the eg scores were the  same before and after receiving the  intervention    citation selection  • the majority of participants  received a score that was the same  or better after receiving the  intervention (both wg and eg)  • 1 participant in each group received  a worse score after receiving the  intervention    statistical analysis    analysis was carried out using a statistical  software package. the mann‐whitney test (a  nonparametric method testing the  significance of the difference between the  distributions of two independent samples)  showed there were no significant differences  in any of the outcomes (p>0.05), between the  wg and the eg. thus, the null hypothesis ii)  is accepted, although due to the small  numbers of participants in this trial, the  statistical analyses have limited value.   evidence based library and information practice 2006, 1:3  50               workshop  wilcoxon test ns/r <5          e‐group    wilcoxon test ns/r <5                participant no.  before  after  difference  participant no.  before  after  difference  001  3  3  0  003  4  4  0  002  3  3  0  004  3  4  1  005  4  4  0  006  3  4  1  007  3  4  1  010  4  4  0  008  2  3  1  014  3  4  1  009  4  4  0  015  3  4  1  011  3  4  1          012  4  4  0          013  4  4  0  mann‐whitney test z = ‐1, p = 0.32    table 2: question formulation  (excellent = 4   good = 3    fair = 2    poor = 1)          workshop  wilcoxon test w= ‐36, ns/r = 8, p=0.01        e‐group    wilcoxon test ns/r <5          participant no.  before  after  difference  participant no.  before  after  difference  001  2  3  1  003  2  3  1  002  1  3  2  004  2  4  2  005  2  2  0  006  2  2  0  007  1  3  2  010  2  2  0  008  2  3  1  014  2  2  0  009  1  2  1  015  2  2  0  011  1  3  2          012  1  2  1          013  2  4  2  mann‐whitney test z = 1.71, p = 0.09    table 3: search strategy    (excellent = 4   good = 3    fair = 2    poor = 1)      evidence based library and information practice 2006, 1:3  51       workshop  wilcoxon test w= ‐11, ns/r = 5, p>0.05        e‐group wilcoxon test  w= ‐8, ns/r = 5, p>0.05    participant no.  before  after  difference  participant no.  before  after  difference  001  3  4  1  003  3  4  1  002  2  2  0  004  1  3  2  005  3  3  0  006  4  4  0  007  3  4  1  010  4  2  ‐2  008  3  3  0  014  1  4  3  009  3  3  0  015  1  2  1  011  1  4  3          012  1  4  3          013  4  3  ‐1  mann‐whitney test z = ‐0.29, p = 0.77    table 4: citation selection    (excellent = 4   good = 3    fair = 2    poor = 1)        workshop  paired t‐test  t= =2.29, df= 8, p=0.05          e‐group paired t‐test  t= ‐2.08, df=5, p=0.09    participant  no.  mark before   mark after   difference  participant no.  mark before   mark after   difference  001  2  2  0  003  1  4  3  002  3  1  ‐2  004  3  3  0  005  2  3  1  006  1  2  1  007  2  3  1  010  1  5  4  008  1  3  2  014  0  3  3  009  2  4  2  015  2  1  ‐1  011  0  3  3          012  0  2  2          013  4  5  1  t‐test for independent samples t  = ‐0.06, df = 13, p = 0.95    table 5: multiple choice correct answers (maximum 6)    evidence based library and information practice 2006, 1:3  52     workshop                  e‐group    participant no.  study limits  before?  study limits after?    participant no.  study limits before?  study limits after?  001  no  no    003  no  no  002  no  no    004  yes  yes  005  no  no    006  no  yes  007  no  yes    010  no  no  008  yes  yes    014  no  no  009  no  no    015  no  yes  011  no  yes          012  no  yes          013  yes  yes            table 6: did the participant use appropriate study design limits? (yes or no)        workshop  wilcoxon test w= ‐28, ns/r = 7, p = 0.02          e‐group wilcoxon test ns/r <5    participant  no  prefer mesh  (before)  prefer mesh  (after)  difference  participant no  prefer mesh  (before)  prefer mesh  (after)  difference  001  2  3  1  003  2  1  ‐1  002  1  3  2  004  2  3  1  005  1  3  2  006  3  3  0  007  1  3  2  010  1  3  2  008  1  3  2  014  1  3  2  009  1  2  1  015  1  1  0  011  2  2  0          012  2  3  1          013  3  3  0  mann‐whitney test z = 0.82, p = 0.41    table 7: searching preference ‐ when i search medline, i prefer to use mesh (thesaurus) rather than keywords (yes = 3     neutral = 2     no = 1 )    evidence based library and information practice 2006, 1:3  53 this shows that the wg and the eg performed  equally as well for all outcomes in question.  however, were these interventions effective, or  were the groups performing equally as badly as  each other? to test if either of these educational  interventions were actually effective in teaching  health professionals to search, the before/after  results were analysed using the wilcoxon signed‐ ranks test (a nonparametric test for the  significance of the difference between the  distributions of two non‐independent samples  involving matched pairs).    for devising a search strategy, the wg performed  significantly better (p=0.01) after receiving the  librarian‐taught workshop. this group also  preferred to search using mesh after receiving the  workshop (p=0.02). there was no significant  difference in citation selection or knowledge of  databases and study designs before and after  receiving either intervention. the null hypothesis i)  is therefore accepted, except for the outcome of  devising a search strategy, where the wg  performed significantly better after the  intervention. again, these statistical analyses only  have limited value owing to the small sample size.    discussion    the results show that knowledge and skills were  increased in most outcomes, but not significantly.  perhaps a single workshop or a brief e‐learning  module is not long enough to learn how to search  effectively; learning search skills may take time to  sink in, and should be practiced regularly to have  an effect – the follow up time of a week may have  been too short for participants to have digested and  used their knowledge and skills effectively.  evidence‐based skills have been shown to improve  and to be retained in a clinically integrated  situation (bradley et al. 194‐201;coomarasamy and  khan 1017), so setting the study in an  academic/learning situation may lead to less  knowledge and skill retention.    this study had some limitations. recruitment was  by self‐selection, but participants were being  measured against their own baseline skill level  (however good or poor that was) in a before/after  study, so this should not have influenced the  results unduly. two participants dropped out of  the online learning group – the first did not turn up  and later phoned to say they had a clinical  emergency (randomisation was done before the  participants arrived, so they did not know they had  been allocated to the elearning group). the second  drop‐out did not complete the second search  exercise, even after repeated contacting, citing that  they were too busy (it was not clear if they were  too busy to use the online learning module at all, or  just too busy to complete the exercise afterwards).     clinical staff proved difficult to recruit to the  research study resulting in a small sample size  which limits the results.  health professionals may  have been encouraged to participate by offering  incentives such as prizes (book tokens, wine etc),  but funding for this in a small study is problematic.   recruitment difficulties may be one reason why a  large number of studies comparing the  effectiveness of search skills interventions for  healthcare professionals are not available.  using  medical students is much easier and large  established groups are available for use in research  studies. however the information needs, skills and  knowledge of medical students and health  professionals are different, and must be examined  separately.  issues such as the difficulty of obtaining up‐to‐date  contact lists for staff (various staff lists were  considerably out of date), plus the perceived  importance of the outcomes by the health  professional can affect the recruitment (rahman et  al. 38‐40;traynor et al. 790‐95;veitch et al. 399‐406).  consequently, the group of people of particular  interest to health librarians is actually one of the  most difficult to study.  recruitment numbers for  this study were very low leading to frustration  when time has been spent developing the  methodology for a trial.  only 17 participants  registered and were deemed eligible for inclusion.  the reasons for low recruitment in this trial were  specifically:  evidence based library and information practice 2006, 1:3  54 • many staff names and contact details were  incorrect, out‐of‐date or unavailable  • email addresses of hospital staff were not  readily available   • clinical staff reported that they found it  difficult to obtain permission to attend  non‐mandatory training   • some staff did not have the time or interest  to participate in a trial  • the ethical approval process reduced time  available to recruit participants  • training rooms were not always available  on the hospital sites  it is significant that learning how to search and use  good quality health information, in order to assist  and improve patient care, is not considered  mandatory!  published studies with small sample  sizes can be easily criticised due to their lack of  statistical power, but the reasons why recruitment  is so difficult became apparent during this trial.  as this was an educational trial recruiting only  health professionals, with no obvious risks  associated, ethical approval was expected to be  easy to obtain. however, due to recent changes in  the ethical application process in the uk, all health  related research requires the completion of a long  (68 page) form (http://www.corec.org.uk/), and the  provision of additional documentation including a  protocol and peer review of the proposal. to a  novice researcher some sections seemed irrelevant  or were difficult to complete as the form did not  differentiate between a multi‐million pound drug  trial involving patients and an international team  of investigators, and a local educational trial  involving health professionals.  further  management approval (research governance) was  also required from the hospital where the research  took place, involving completion of additional  forms and letters. obtaining ethical approval  therefore took almost 4 months and delayed  recruitment for the trial.     articles published in the british medical journal  suggest this is not an isolated experience (mayor  1258‐5f;wald 282‐84;jamrozik 286‐87;bentley and  enderby 361;elwyn et al. 847); clinical researchers  have reported that the ethical committee  application procedure can impede, delay or distort  their research, or prevent them from starting  altogether. one editorial from a director for public  health states:    “…bureaucratic barriers, such as the length  and complexity of the corec forms and the  dual hoops of research ethics and governance,  are putting off those considering starting  smaller locally based studies.” (bentley and  enderby 361)    the research question and multiple outcomes may  have been a bit too ambitious for a small research  study, and concentrating solely on development of  a search strategy might have been easier. future  research questions could be limited to the  effectiveness of interventions for search strategy  development.     to a novice researcher, and a practicing librarian, a  number of issues became apparent during the  course of the study.  this led to the development of  a list of tips (box 1) that researchers may want to  consider when planning a research study.    conclusion    this paper has presented the methods and results  of a randomized trial seeking to compare two  interventions for teaching search skills to health  professionals. for question formulation and  devising a search strategy, all participants received  a score that was the same or better after receiving  the intervention (both wg and eg). however,  statistical analysis showed that the only significant  outcomes were for the wg devising a search  strategy and preferring to search using mesh after  receiving the workshop intervention. there were  no significant differences in any of the outcomes  when the wg and the eg were compared.     the question of whether a taught workshop and an  e‐learning module are of equal effectiveness in  teaching search skills, is an important one for  health librarians involved in user education, and   http://www.corec.org.uk/ evidence based library and information practice 2006, 1:3  55 • conduct a thorough literature search; your question may already have been answered  • read books on research methods to try and get an overview of the subject  • apply for ethical and management approval as soon as the study is initiated, and try to obtain help in  completing it; this will reduce delays in the process  • the population that practitioners’ would like to study are often the most difficult to engage or recruit;  this does not mean they should be ignored  • recruitment of staff, especially clinical staff, may be difficult, but start the recruitment process early,  use different contact methods and make the inclusion criteria as wide as possible  • collaborate with other librarians or researchers where possible; conducting a research study  individually while also maintaining a full‐time job is very time consuming  • don’t feel inadequate when things go wrong; often you have no control over them  • don’t give up; without research into the issues that matter, evidence‐based librarianship would not be  possible    box 1: tips for planning research      was a justifiable topic to propose and conduct  research. the fact that the results are mainly  inconclusive due to the small sample size is  disappointing, but does not diminish the  importance of conducting the study.    by discussing the main issues and problems  experienced during this study, it is hoped other  practitioners’ intending to undertake research will  have a clearer picture of the processes involved, be  able to learn from any mistakes made in this study,  and have a realistic idea of the problems that may  be encountered, some of which may be insoluble.  additionally, it is hoped other researchers’ will  build on the methodology used in this study to  conduct further research into the effectiveness of  interventions for teaching search skills to health  professionals.       works cited    bentley, chris and pam enderby. ʺacademic  medicine: who is it for?: applications to local  research ethics committees are down by  around 40%.ʺ british medical journal 330.7487  (2005): 361.    bradley, d. r. et al. ʺreal‐time, evidence‐based  medicine instruction: a randomized  controlled trial in a neonatal intensive care  unit.ʺ journal medical library association  90.2 (2002): 194‐201.    bradley, p. et al. ʺmedical studentsʹ and tutorsʹ  experiences of directed and self‐directed  learning programs in evidence‐based  medicine: a qualitative evaluation  accompanying a randomized controlled trial.ʺ  evaluation review 29.2 (2005): 149‐77.    burrows, s. c. and v. tylman. ʺevaluating medical  student searches of medline for evidence‐ based information: process and application of  results.ʺ bulletin medical library association  87.4 (1999): 471‐76.    cheng, g. y. ʺeducational workshop improved  information‐seeking skills, knowledge,  attitudes and the search outcome of hospital  clinicians: a randomised controlled trial.ʺ  health information and libraries journal 20  suppl 1 (2003): 22‐33.    coomarasamy, a. and k. s. khan. ʺwhat is the  evidence that postgraduate teaching in  evidence based medicine changes anything?  a systematic review.ʺ british medical journal  329.7473 (2004): 1017.    dorsch, j. l., m. k. aiyer, and l. e. meyer. ʺimpact  of an evidence‐based medicine curriculum on  evidence based library and information practice 2006, 1:3  56 medical studentsʹ attitudes and skills.ʺ j med  libr assoc 92.4 (2004): 397‐406.    elwyn, glyn et al. ʺethics and research governance  in a multicentre study: add 150 days to your  study protocol.ʺ british medical journal  330.7495 (2005): 847.    garg, a. and k. m. turtle. ʺeffectiveness of  training health professionals in literature  search skills using electronic health databases‐ ‐a critical appraisal.ʺ health information and  libraries journal 20.1 (2003): 33‐41.    jamrozik, konrad. ʺresearch ethics paperwork:  what is the plot we seem to have lost?ʺ british  medical journal 329.7460 (2004): 286‐87.    mayor, susan. ʺadvisory group to review nhs  research ethics committees.ʺ british medical  journal 329.7477 (2004): 1258‐5f.    mccluskey, a and m lovarini. ʺproviding  education on evidence‐based practice  improved knowledge but did not change  behaviour: a before and after study.ʺ bmc  medical education 5.40 (2005).    rahman, m. et al. ʺphysiciansʹ attitudes towards  and reasons for participation in the  candesartan antihypertensive survival  evaluation in japan (case‐j) trial.ʺ journal of  epidemiology 15.2 (2005): 38‐40.  roberts, c. and d. torgerson. ʺrandomisation  methods in controlled trials.ʺ british medical  journal 317.7168 (1998): 1301.    rosenberg, w. m. et al. ʺimproving searching skills  and evidence retrieval.ʺ journal of the royal  college of physicians of london 32.6 (1998):  557‐63.    traynor, v. et al. ʺrecruiting general practitioners  for survey research.ʺ australian family  physician 22.5 (1993): 790‐95.    veitch, c. et al. ʺgeneral practice research.  problems and solutions in participant  recruitment and retention.ʺ australian family  physician 30.4 (2001): 399‐406.    vogel, e. w., k. r. block, and k. t. wallingford.  ʺfinding the evidence: teaching medical  residents to search medline.ʺ journal  medical library association 90.3 (2002): 327‐ 30.    wald, david s. ʺbureaucracy of ethics  applications.ʺ british medical journal 329.7460  (2004): 282‐84.    wutoh, r., s. a. boren, and e. a. balas. ʺelearning:  a review of internet‐based continuing medical  education.ʺ journal of continuing education  in the health professions 24.1 (2004): 20‐30.      evidence based library and information practice 2009, 4:2 164 evidence based library and information practice evidence summary citations to conference papers indicate they are declining in importance across all discipline areas a review of: lisée, cynthia, vincent larivière and eric archambault. ‚conference proceedings as a source of scientific information: a bibliometric analysis.‛ journal of the american society for information science and technology 59.11 (2008): 1776-84. reviewed by: gaby haddow lecturer, school of media, culture & creative arts curtin university of technology perth, western australia e-mail: g.haddow@curtin.edu.au received: 01 march 2009 accepted: 06 april 2009 © 2009 haddow. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – to compare the impact and ageing of conference proceedings with that of scientific literature in general, as reflected in citation characteristics. design – citation analysis. setting – thomson’s science citation index, social science citation index, and arts and humanities citation index (cd-rom version). subjects – conference proceedings citations. methods – the thomson citation indexes were searched to identify all citations to conference proceedings in natural sciences and engineering (nse) and social sciences and humanities (ssh) from 1980 to 2005. keywords in english, spanish, italian and german, truncated terms (such as ‘bienn’), single letters (such as p), and numbers were combined to retrieve all possible citations. additional filters to exclude citations to publications other than proceedings were applied to the p search results, which had accounted for 75% of the total results. the references remaining in the p search set were validated using google scholar and worldcat. finally, two random samples of 1,000 references were checked manually to determine the extent mailto:g.haddow@curtin.edu.au evidence based library and information practice 2009, 4:2 165 of false positives and false negatives in the results. main results – the study’s findings are presented for nse and ssh separately, with 1.7% of nse citations and 2.5% of ssh citations referring to conference proceedings. the total number of citations to proceedings has increased over the period 1980-2005, however, citations to proceedings in nse and ssh as a proportion of all citations decreased during this time. a small increase in the average number of proceedings citations per paper was found for nse and ssh. when this increase is compared to the overall increase in references per paper over this period, the share of proceedings citations per paper has decreased. of all fields in nse and ssh, only engineering has increased the proportion of proceedings citations, rising from 7% to 10% in the period studied. in 2005, the share of proceedings citations in nse (excluding engineering) was below 3%, and for ssh it was below 1.5%. the share of proceedings citations varies across different fields within nse and ssh. engineering fields and computer science range from around 5% (general engineering) to 19.6% (computers) in the share of proceedings citations, with only five of the 109 nse fields having 10% or more as a share of proceedings citations. ssh has only one field (ergonomics, 7.6%) with a share of proceedings citations over 5%. transport studies has a share of proceedings just under 5%, followed by the field information science & library science with proceedings citations at 3.3%. in relation to the ageing characteristics of proceedings citations overall, the findings show a median age of 4.0 years compared with 6.1 years for citations to literature in general. the difference between the age of nse cited proceedings and nse cited literature in general had decreased during the period specified. in 1980, the median age of nse cited proceedings was 6.3 years compared with 9.3 years for nse citations to literature in general. in 2005, the median ages were 8.4 years and 10.1 years, respectively. the median age of ssh cited proceedings in 2005 was 10.3 years, compared with 14.2 years for all ssh cited literature. cited literature in general is older for ssh (14.2 years) than nse (10.1 years), but the age difference between proceedings cited in the two discipline areas is almost half this. a number of fields in nse (such as physics, chemistry, and engineering) indicate a greater difference between the age of cited proceedings and literature in general, while for others (such as biology and biomedical research) the ageing characteristics are similar. in ssh, the difference between age of cited proceedings and literature in general is greater. fine arts and psychology proceedings citations are 43% younger than citations to literature in general; literature cited proceedings are 42% younger, and social sciences 31% younger. humanities are an exception, with cited proceedings only 11% younger than citations to literature in general. conclusion – only 2% of all citations are to conference proceedings in nse and ssh combined; a proportion that has declined over the 25-year period studied. while there was an increase in the average number of (all) citations per paper during this time, proceedings citations per paper have seen only a very slight increase. these findings are true of all fields studied, with the exception of engineering-related fields which have enjoyed an increase of over 2% in proceedings citations in the period studied. the results also indicate the importance of proceedings in the field of computers. the authors speculate that in these fields, proceedings are regarded as ‚more than just prototypes, but rather as the final products of scientific research.‛ due to the higher proportion of proceedings citations in engineering and computer science fields, they should be considered for analysis in bibliometric studies. despite arriving at this conclusion, the authors suggest that computer scientists might consider publishing their papers through channels other evidence based library and information practice 2009, 4:2 166 than conference proceedings to ‚maximize their scientific impact [original italics].‛ they support this statement by noting that although proceedings citations in computer science represent 20% of total citations, a study of australian computer science research output (butler) found proceedings comprise over 60% of all publications in computer science. the authors suggest that the difference between the proportion of proceeding published and the proportion of proceedings cited indicate that their ‚scientific impact does not seem to be all that important.‛ in all fields, proceedings are cited sooner after publication and they cease to be cited earlier than literature in general. these results indicate that proceedings deliver more current information and cutting edge research findings than literature in general. the differences between ageing of proceedings citations and of literature in general lead the authors to conclude that ‚conference proceedings serve different functions and have different life cycles depending on the community they serve.‛ commentary conference proceedings play an important role in some fields, yet intuitively, perhaps stemming from our own citation practices, they seem less important as a source of information to reference. this study confirms that perception. it used exhaustive and sophisticated search strategies to identify citations and is the first study to examine the share of citations that proceedings receive across all fields. researchers wishing to conduct a similar study would be well-served to note the methods used. the findings indicate that proceedings’ citations differ between fields, but overall, the proportion is low and has declined during the period 1980 to 2005. interestingly, the study also found that cited proceedings have increased in age over this period. a number of factors which may have influenced the study’s findings are not addressed and the paper makes some unsupported statements. clarification is required as to whether the relative growth in journal publications and conferences has differed over the period studied, as this may explain the decreased proportion of proceedings’ citations (despite the increase in the total number of proceedings’ citations). importantly, the authors have not acknowledged the limitations of using the thomson citation indexes which have variable coverage of subject fields, potentially affecting results. an alternative to the thomson indexes (scopus, for example, launched in 2004) includes a wider range of conference proceedings and may have produced quite different results. it is also debatable that the thomson citation indexes include all core papers, as suggested in the hypothesis. for a study involving such extensive and complex methods of data collection, it may have little impact on researchers. in applying these methods to determine the ‘importance’ of proceedings, the researchers imply that citations are indeed a useful measure. however, the importance of conferences as ‚diffusion media‛ (acknowledged by the authors) may be better suited to alternative methods of measurement. the authors’ conclusion that ‚the extent to which conference proceedings are later converted into scientific articles ... is independent of the percentage of references that are made to conference proceedings,‛ (1783) is unexplained and requires supporting evidence. taken as a whole, the findings may deter library practitioners from acquiring proceedings for collections if they are perceived as less ‘important’ than other publication forms. however, in fields of engineering and computer science proceedings are relatively highly cited, which suggests that clients in those fields will benefit from access to proceedings. in our own field of library and information science, proceedings are cited at a relatively high rate compared with others in ssh. this, as well as personal experience, indicates that proceedings evidence based library and information practice 2009, 4:2 167 are a valuable source of information. and while proceedings may not be cited at the rate of other publication forms, our practices may improve as a consequence of attending them. works cited butler, linda. ‚ict assessment: moving beyond journal outputs.‛ scientometrics 74.1 (2008): 3955. evidence summary   norwegian public library language cafés facilitate discourse between immigrants and norwegian-born citizens   a review of: johnston, j., & audunson, r. (2017). supporting immigrants’ political integration through discussion and debate in public libraries. journal of librarianship and information science, 29 may, 1-15. https://dx.doi.org/10.1177/0961000617709056    reviewed by: stephanie krueger head, office of specialized academic services czech national library of technology prague, czech republic email: stephanie.krueger@techlib.cz   received: 12 sept. 2017 accepted: 5 dec. 2017      2018 krueger. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29330       abstract   objective – to investigate if conversation-based language cafés in norway provide a platform for improving communication between immigrants and norwegian-born citizens, potentially setting the stage for better participation by immigrants in civic dialogues.   design – multi-site case study.   setting – public libraries in oslo, moss, and horten, norway.   subjects – language café participants (immigrants and norwegian-born volunteers).   methods – participant observation and questionnaires for immigrants (norwegian, english, somali, and arabic language versions) and volunteers (norwegian language only) who took part in café activities.   main results – 64 immigrants (21 in oslo, 30 in moss, 13 in horten) and 31 volunteers (7 in oslo, 14 in moss, 10 in horton) completed questionnaires. language cafés at all three sites led to informal, respectful discursive interaction between participants. though each café had a unique set of participants and conversational topics, all cafés enabled immigrants to improve their norwegian language skills while providing all participants with a place to meet new people, exchange information, and discuss political issues.   conclusion – having attended the cafés and improved their knowledge of norwegian language and culture, immigrants at all three sites were potentially better equipped for future participation in the norwegian public sphere.   commentary   while public libraries in many countries still debate the extent to which they should participate in the political process (jaeger, bertot, & gorham, 2013), norway’s 2014 library act provides a mandate for public libraries to actively foster public discussion and debate. conversation-based programs in public libraries, including language cafés, support this mandate by providing neutral fora for participation in the so-called habermasian public sphere. this study contributes qualitative richness to prior examinations of the public sphere and public libraries summarized by widdersheim & masanori (2016). tangentially, the study also adds to the broader corpus of social scientific investigations about immigration and multicultural societies. for example, the study describes specific aspects of norway’s approach to social integration of immigrants that may be useful to researchers who study how national integration policies differ (alba & foner, 2014).   the study achieves the three primary aims of case study research—generality, accuracy, and complexity/coverage (woodside, 2010, pp. 19-20)—but only in relation to the specific question do language cafés facilitate discourse in settings where immigrants must learn a new language? the study is not longitudinal in scope and does not investigate whether or not immigrants, armed with new language and cultural skills, actually do participate in the political process following participation in a language café. future researchers may wish to pursue this question.   at the theoretical level, generality is strengthened with the universally applicable, in-depth discussion of the concept of the public sphere applied to multicultural settings, including descriptions of so-called weak/strong publics and social capital (putnam, 2005). this supranational “theoretical anchoring” provides any public library, anywhere, with a conceptual roadmap for advocating conversation-based programming.   in terms of the case study itself, conducting research at multiple sites and gathering data with three mechanisms were strategies employed for ensuring adequate accuracy and complexity/coverage. in future presentations of this research, it would be helpful to include the questionnaire for volunteers as an appendix (even in the original norwegian), which was missing in this study. additionally, while the discussion of findings includes highly relevant questionnaire responses and field note excerpts, numeric data about questionnaire responses would enable readers to understand the study more completely. as an illustration, the authors note “the vast majority of participants reported they think attending the language café has improved their language skills somewhat or a lot” (p. 7), but the phrase “vast majority” is open to interpretation by different readers. summarizing this statement with a specific percentage would clarify meaning.   overall, this study provides public libraries with examples of how conversation-based programs can enrich civic dialogues. rich in theory, the study also provides conceptual ammunition for public libraries wishing to engage more actively in political discourse for use in defending democratic discussion in hostile environments.   references   alba, r., & foner, n. (2014). comparing immigrant integration in north america and western europe: how much do the grand narratives tell us? international migration review, 48(s1). s262–s290. https://dx.doi.org/10.1111/imre.12134   jaeger, p.t., bertot, j.c., & gorham, j. (2013). wake up the nation: public libraries, policy making, and political discourse. the library quarterly, 83(1), 61–72. https://dx.doi.org/10.1086/668582   putnam, r. d. (2007). e pluribus unum: diversity and community in the twenty-first century the 2006 johan skytte prize lecture. scandinavian political studies, 30(2), 137–174. https://dx.doi.org/10.1111/j.1467-9477.2007.00176.x   widdersheim, m. m., & masanori, k. (2016). conceptual modelling of the public sphere in public libraries. journal of documentation, 72(3), 591-610. https://dx.doi.org/10.1108/jd-06-2015-0079   woodside, a.g. (2010). case study research: theory, methods and practice. bingley, uk: emerald group publishing limited.     news/announcements   call for proposals for capal17      2016. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     we are pleased to announce the call for proposals (cfp) for capal17: foundations & futures: critical reflections on the pasts, presents, and possibilities of academic librarianship. this will be the fourth annual conference of the canadian association of professional academic librarians (capal) to be held may 30th – june 1st as part of congress of the humanities and social sciences 2017 in toronto, ontario which lies in the territory of the haudenosaunee and the mississaugas of the new credit river. this conference provides an opportunity for the academic library community to critically examine and discuss together the ways in which our profession is influenced by its social, political, and economic environments. by considering academic librarianship within its historical contexts, its presents, and its possible futures, and by situating it within evolving cultural frameworks and structures of power, we can better understand the ways in which academic librarianship may reflect, reinforce, or challenge these contexts both positively and negatively. the deadline for proposals is the 23rd of december, 2016. see the full call for proposals including instructions for submissions at: http://capalibrarians.org/2016/11/cfp-for-capal17-foundations-futures-critical-reflections-on-the-pasts-presents-and-possibilities-of-academic-librarianship/ courtney waugh, program chair, cwaugh5@uwo.ca on behalf of the capal17 program committee      evidence based library and information practice       research article   mixed methods not mixed messages: improving libguides with student usability data   nora almeida instruction librarian, assistant professor ursula schwerin library new york city college of technology, city university of new york brooklyn, new york, united states of america email: nalmeida@citytech.cuny.edu   junior tidal web services librarian, associate professor ursula schwerin library new york city college of technology, city university of new york brooklyn, new york, united states of america email: jtidal@citytech.cuny.edu   received: 5 july 2017                                                                      accepted: 23 oct. 2017         2017 almeida and tidal. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – this article describes a mixed methods usability study of research guides created using the libguides 2.0 platform conducted in 2016 at an urban, public university library. the goal of the study was to translate user design and learning modality preferences into executable design principles, and ultimately to improve the design and usage of libguides at the new york city college of technology library.   methods – user-centred design demands that stakeholders participate in each stage of an application’s development and that assumptions about user design preferences are validated through testing. methods used for this usability study include: a task analysis on paper prototypes with a think aloud protocol (tap), an advanced scribbling technique modeled on the work of linek and tochtermann (2015), and semi-structured interviews. the authors introduce specifics of each protocol in addition to data collection and analysis methods.   results – the authors present quantitative and qualitative student feedback on navigation layouts, terminology, and design elements and discuss concrete institutional and technical measures they will take to implement best practices. additionally, the authors discuss students’ impressions of multimedia, text-based, and interactive instructional content in relation to specific research scenarios defined during the usability test.   conclusion – the authors translate study findings into best practices that can be incorporated into custom user-centric libguide templates and assets. the authors also discuss relevant correlations between students’ learning modality preferences and design feedback, and identify several areas that warrant further research. the authors believe this study will spark a larger discussion about relationships between instructional design, learning modalities, and research guide use contexts.      introduction   subject and course specific research guides created on the popular springshare platform, libguides, have become ubiquitous in academic library environments. in spite of this, there has been little published research on the pedagogical efficacy, use contexts, and design of research guides. the libguides platform, which purports to be the “easiest to use web publishing and content creation platform for libraries,” allows librarians to remix and reuse content across guides and institutions, and caters to libraries who want a flexible tool for curating library content that can accommodate librarians with little technological literacy or experience with digital instructional design (springshare, 2017). however, just because librarians have subject expertise and knowledge of specialized research practices does not necessarily mean they can create digital resources that will be easy for students to use or that will address the information needs students have in different contexts. as academic libraries increasingly rely on digital platforms like libguides to reach students conducting research off campus and to supplement or replace face-to-face instruction, they should consider whether the subject and course specific research guides they create reflect user preferences and research behaviours. librarians must also consider the various use contexts for online research guides when making decisions about language, layout, navigation, and interactivity.   this article describes a mixed methods usability study the authors conducted in 2016 to learn more about student design preferences and learning styles, and to improve subject specific research guides at the new york city college of technology’s ursula schwerin library. the goal of the study was to translate user design and learning modality preferences into executable design principles. after introducing the project methodology and presenting study findings, the authors will discuss discrepancies and study limitations, and will outline several areas for future research. while focused on a specific institutional context, the methodology and results of this study will be of interest to librarians at other institutions who want to ensure that research guides and other educational technology platforms deploy user-centric design principles.    institution and platform context   the new york city college of technology, colloquially known as city tech, is one of the city university of new york (cuny) system’s 24 colleges. the campus is located in downtown brooklyn and is a commuter school primarily serving residents of new york city’s five boroughs. offering 2-year associate and 4-year baccalaureate programs, city tech is known for technical and professional programs such as nursing, hospitality management, and vision care technology. the institution is demographically diverse and enrolls over 17,000 students. while the city tech population is unique, enrollment trends and student demographics reflect patterns at colleges and universities across the united states (u.s. department of education, 2014).    libguides is one of a number of educational platforms used by city tech librarians for information literacy instruction and outreach. libguides 2.0 was acquired through a consortial cuny-wide license and rolled out at the city tech campus in fall 2015. the platform replaced previous research guides housed on mediawiki, the same open-source software that powers wikipedia. prior to the 2015 roll-out, a majority of faculty librarians at city tech had no experience creating guides on either the libguides 2.0 or 1.0 platforms. a project to migrate existing mediawiki guides to libguides 2.0 revealed that the guides lacked consistency in terms of overall design, navigation, extent of content, and interactivity. while the library employs user testing to inform the design of the library website, subject expert librarians had autonomously developed subject specific research guides without soliciting user feedback. the authors realized that usability testing might provide insight into student preferences and help improve the design of libguides while still giving subject expert librarians the autonomy to curate disciplinary content.   literature review   in spite of its ubiquity, the libguides platform is currently underexplored in lis literature and some librarians have expressed concern that “there is sparse research on how university students use libguides and what benefits it affords them” (bowen, 2014, p.152; hicks, 2015; thorngate & hoden, 2017). recently, the libguides platform has received some critical attention from user experience librarians but published case studies infrequently address connections with regards to pedagogy, student learning modality preferences, social dimensions of library technology, or sociocultural contexts of research (brumfield, 2010, p.64; hicks, 2015). an exception to this is a forthcoming study by thorngate and hoden (2017), who discuss the importance of “the connection between research guide usability and student learning” and explicitly connect design features with cognitive practices. the few existing case studies that discuss libguides, user experience, and pedagogy point to the necessity of qualitative and task based user testing approaches in order to understand student learning styles, research behaviors, and design preferences (thorngate & hoden, 2017; bowen, 2014).   user testing protocols   literature on user testing protocols reveals that qualitative methods like interviews and tap can be combined with more traditional user experience protocols to generate substantive, qualitative feedback. a mixed methods approach allows experimenters to gain pointed feedback about specific design elements that can then be analyzed alongside subjective learning modality preferences and user behaviors (linek & tochtermann, 2015). the testing protocol initially considered for the city tech study included an a/b test combined with semi-structured interviews in order to compare design variants and capture user preferences for different layouts, page elements, and navigation schemas (young, 2014; martin & hanington, 2012). however, since a/b tests are most effective with fully executed designs and a large number of study participants, the authors concluded that paper prototyping was the most appropriate method for testing interface variations using low-fidelity mock-ups (nielsen, 2005). paper prototyping allows for an analysis of “realistic tasks” as study participants “interac[t] with a paper version of the interface” (snyder, 2003). this protocol is flexible enough to be used with more than one interface variation and, unlike a/b testing, only requires 5 participants to identify most usability issues (snyder, 2003; nielsen, 2012). while some literature indicates that users prefer computer prototypes in task based protocols, the quantity and quality of feedback generated by paper versus computer prototype testing is comparable (sefelin, tscheligi, & giller, 2003; tohidi, buxton, baecker, & sellen, 2006). task analyses on paper prototypes are frequently combined with tap to capture subjective feedback and to identify “concrete obstacles” participants encounter (linek & tochtermann, 2015). the tap protocol has been used for numerous usability studies involving libguides (thorngate & hoden, 2017; yoon, dols, hulscher, & newberry, t., 2016; sonsteby, a., & dejonghe, 2013).   numerous usability studies have pointed to the “reluctance of people to express critique and to verbalize negative thoughts during user testing” (linek & tochtermann, 2015; sonsteby & dejonghe, 2013; tohidi et al., 2006). to invite more critical responses from study participants and to “receive more informal, creative feedback,” experimenters can combine alternative methods with traditional protocols and present “multiple alternative designs” to participants (linek & tochtermann, 2015; tohidi et al., 2006). one such alternative protocol is the advanced scribbling technique, which can be combined with traditional paper prototype task analyses and interviews. during advanced scribbling, participants annotate paper prototypes with colored highlighters in order to identify important, confusing, and unnecessary design elements (linek & tochtermann, 2015).  linek and tochtermann (2015) describe this protocol as a “systematic way of receiving feedback and avoiding ambiguity” and also note this method may reduce barriers to critique because it “enables the evaluation of single design elements without pressuring the participant to express explicitly negative comments.”       libguide templates and design elements   libraries cite user studies, case studies, or best practices documentation on the springshare libguides website as the basis for local design decisions (bowen, 2014; desimio & chrisagis, 2014; dumuhosky, rath, & wierzbowski, 2015; duncan, lucky, & mcclean, 2015; thorngate & hoden, 2017). however, it is important to note that many institutions use libguides without conducting any user testing or surveying published case studies to inform design. as a result, many libguides are problematically “library-centric” in terms of how information is presented and organized (hicks, 2015). hicks (2015) argues that such unreflective design practices can undercut critical pedagogical models and “marginalize the student voice from the very academic conversations” that most concern them. consequently, user testing is not only important in terms of defining design decisions but may also be a critical imperative if such interactions yield important insights into how students learn.   libguides are most successful if they focus on student information needs and reflect student research behaviors. researchers have found that students will abandon guides if they are confusing, cluttered, or if the purpose of the guide is not immediate apparent (gimenez, grimm, & parker, 2015). some institutions have opted to replace librarian-centric terms such as “articles and journals” with natural language terms such as “magazines” or “news” after conducting user testing (sonsteby & dejonghe, 2013). additionally, several studies specifically looked at how students respond to the use of columns on libguides and introduced strategies to reduce “noise” and clutter (gimenez et al., 2015; thorngate & hoden, 2017).   many libraries make use of libguides templates to hard code design elements and ensure design consistency across research guides (duncan, lucky, & mcclean, 2015). templates not only make guides more useful to students but also allow subject selectors to focus on content instead of technical aspects of the libguides platform (duncan et al., 2015). while specific template recommendations are helpful in that they identify concrete design elements on the libguides platform, it is essential that libraries consider specific user populations, institutional cultures, and use contexts when designing libguide templates.   aims   the primary objective of this mixed methods user study was to understand city tech students’ design and learning modality preferences and to improve subject specific research guides. the study was designed to capture students’ impressions of multimedia content such as videos, images, embedded presentations, text-based content, and interactive instructional content like search boxes or quizzes. this was in the context of a specific research scenario and in relation to a specific platform interface. the authors used study findings to document best practices for design, and plan to translate this document into a new hard-coded libguides template that includes a standard navigational schema and layout. for optimal features that may not be relevant for every guide or subject area, the authors plan to create a series of custom libguides assets that librarians without technological expertise can easily pull into research guides.     methodology   before beginning user testing, the authors worked with an instructional design intern to conduct a brief content audit of the subject specific guides migrated from mediawiki. this audit revealed that some guide content was out of date, content was duplicated, most guides were heavily text-based, the migrated guides used inconsistent database linking protocols, and most guides did not have embedded search opportunities or interactive features. in the fall 2016 semester database assets were loaded into libguides in order to resolve linking issues. a handful of guides were revised to mitigate duplication and to remove some out of date content before user testing began.   after conducting a literature review, the authors determined that a mixed methods approach would yield the most robust data about student design and learning modality preferences. the methods ultimately used for the study included a combination of paper prototyping, advanced scribbling, task analysis, tap, and semi-structured interviews. below, each protocol is described along with how data was compiled and analyzed. after refining the project methodology, the authors worked with the instructional design intern to develop two paper prototypes: a control prototype that mirrored a typical subject guide, and a revised prototype that used a simplified navigation schema and included more multimedia elements. ten student participants were recruited through flyers and email blasts, and were compensated $5.00 for 30 minute individual test sessions. the authors and the instructional design intern conducted testing in december 2016 with one experimenter serving as proctor, a second experimenter as note taker, and a third experimenter as a human “web server” who supplied access to interior pages of the prototype after study participants “clicked” on features or menu items during the task analysis phase. results were analyzed and shared with the city tech library department in spring 2017 and the authors plan to complete the template and library faculty training in fall 2017.    paper prototyping              student participants were given two paper prototypes emulating two variations of the landing page of an english research guide. the control prototype had a two-tiered primary navigation menu, a static welcome image, and contained few linked elements with the exception of a list of recommended databases. alternatively, the revised prototype contained a minimal primary navigation menu with labels containing action verbs like “find” or “search”, introduced more opportunities for interaction with search boxes and collapsible info boxes, and contained multimedia features including a book gallery and video (see appendix b and c). participants were asked to annotate both prototypes following the advanced scribbling protocol guidelines (see below). subsequently, participants selected their preferred layout, answered questions about their modality and design preferences in a semi-structured interview (see below), and completed a research task on the paper prototype that they selected (see below).   advanced scribbling   participants were given green, yellow, and red highlighters and instructed to color code design elements on both the control and revised prototypes. students marked elements they deemed important in green, confusing in yellow, and unnecessary in red and used a pen for substantive annotations. data for advanced scribbling was collected by tabulating how users color coded each design element on each prototype. since not all participants marked every element, percentages of elements coded as important, confusing, and unnecessary on the two prototypes were calculated by assessing the color coded elements in relation to the total number of participants who marked that element.    interviews   prior to scribbling, participants were asked contextual questions about their learning preferences and research experience (see appendix a). after completing the advanced scribbling task on both prototypes, students were asked questions about the prototypes including which interface they preferred and why. students were asked to expound upon their scribbles and to provide feedback on the navigation labels and the extent of content displayed on each guide. during these semi-structured interviews, a note taker recorded student feedback. responses were tabulated for yes / no and either / or questions. during the data analysis phase, the authors identified keywords and mapped thematic patterns for qualitative feedback that could not be easily tabulated. for example, in response to the question “have you used the library to conduct research?” many students mentioned that they had borrowed books or used library databases or articles. the authors identified books, databases, and articles as keywords and were thus able to identify patterns about the types of research materials study participants had previously used.   task analysis and tap   after the interviews, participants were given two hypothetical research scenarios (see table 2 below) to test how well the prototype interfaces supported the completion of these tasks. during the task scenarios, participants were told to “think-aloud,” verbally expressing their thought processes. metrics were recorded for each task scenario, including fail/success rate and the number of “clicks” that users would need to complete the task. tap response data was combined and analyzed with qualitative interview feedback using keyword analysis and thematic mapping.   findings   participant profiles   most student study participants indicated that they had some experience in academic research environments. participants characterized themselves as either beginner or intermediate level researchers. the majority of participants (50%) indicated they regularly used a library other than the city tech library, 20% were not library users, and the remaining 30% of participants were city tech library users (see figure 1). only 4 out of 10 participants had ever attended a library instruction session. none of the participants in this study had previously used an online library research guide. figure 1 participant interview responses about library usage.     figure 2 participants’ preferred types of content.       during interviews, student participants expressed a slight preference for multimedia over text-based content, interactive content, and the combination of all types of content (see figure 2).  student media preferences were corroborated by analysis of advanced scribbling protocols and qualitative feedback compiled during user testing.  figure 3 participants’ advanced scribbling data for paper prototype a.     design elements   fifty percent of study participants preferred the control prototype (prototype a) and 50% of users preferred the revised prototype (prototype b). since no best overall design emerged from this study, the authors will highlight specific feedback on individual design elements across both prototypes in their analysis. the advanced scribbling protocol yielded some contradictory data with some students marking elements as important and others marking those same elements as either confusing or unnecessary. some contradictory data is the result of variations in student design or modality preferences, and other disparities were clarified in semi-structured interviews.   the highest ranking elements (see figure 3) of the control prototype (prototype a) were the primary navigation menu, the “ask a librarian” box, and the welcome message providing context for the guide. a large majority of users marked the class exercise element as either unnecessary or confusing, and three users indicated that the welcome image was confusing or unnecessary.                   there was less clear consensus about design elements on the revised prototype (prototype b), although several elements received positive rankings from users (see figure 4). high ranking elements included the primary navigation menu, although seven out of ten users found specific tab labels within the main menu confusing or unnecessary. while qualitative feedback suggested that language was important to users, none of the student participants provided feedback on the use of action verbs versus nouns in labels (e.g. “find articles” vs. “articles”). other high ranking elements on prototype b were the book carousel, the search box, and the “ask a librarian” box. several elements received mixed rankings including an instructional video which four users marked as important but three users found unnecessary, perhaps reflecting variations in modality preferences. the “your librarian” box figure 4 participant advanced scribbling data for paper prototype b.     table 1 user feedback about template elements "begin search" should be "home" there are two "search" "should not be 2 search options" ask a librarian + contact [your librarian] are redundant combine "ask a librarian" and "your librarian" boxes     was ranked as important by six users, but three marked this feature as unnecessary, and qualitative feedback revealed that some students found this feature redundant.   qualitative feedback   in interview questions and during the tap, users offered some concrete suggestions for the guide layout and navigation labels that could be incorporated into the libguides template (see table 1). several users pointed out redundant features that should be consolidated, including the “ask a librarian” and “your librarian” boxes, and the search box widget and the “begin search” menu item on prototype b. participants also noted that the purpose of the guide should be explicit and they wanted to be able to conduct research without navigating away from the guide or being redirected to the library website. table 2 tasks that users were asked to attempt on the paper prototypes task 1: where would you go on this guide to find scholarly articles? task 2: where would you go on this guide to search for books in the city tech library’s collection?     task analysis   participants were given a hypothetical research scenario during which they were asked to conduct scholarly research for a paper on post-colonial caribbean literature. students were then given two tasks to complete on their preferred prototype in relation to the scenario (see table 2).   task completion success rates were high (above 80%) on both prototypes with a slightly higher failure rate for prototype a. the lack of a search box on prototype a accounted for most failures since users would have to navigate away from the guide to search the library’s electronic and print holdings. additionally, several participants had difficulty interpreting certain elements on the paper prototypes and in some cases assumed that certain features were hyperlinked. this resulted in a small percentage of false positives where users incorrectly believed they had successfully completed a task. this data was useful in that it revealed student expectations for linked and dynamic elements.   navigation the other metric recorded during the task analysis phase was the number of “clicks” needed to complete each task. a click was recorded whenever a participant indicated they would use the search box, follow a link, or go back a page. the click averages were low for both prototypes and tasks, with less than three clicks performed by all users per task. click averages for task two were substantially lower for both prototypes. the authors speculate that this is a result of learnability as users became   more comfortable with the prototype layouts after completing task one.  discussion   implications for design   no “best” layout emerged as a result of the usability study and thus, the authors will focus on design elements ranked favorably across prototypes in developing a new user-centric template. based on participant feedback, the authors plan to maintain the labels and navigation schema used in prototype a but will incorporate more multimedia and interactive content to ensure libguides can accommodate different learning modality preferences. study findings indicate that students should be able to complete basic research tasks from within research guides, and a search widget for the city tech library’s discovery layer will be hardcoded into the final template. based on user suggestions, the “your librarian” and “ask a librarian” boxes will be combined. since there was some ambiguous feedback on multimedia features, the video and book carousel elements will be made available as optional assets that can be integrated into guides where appropriate.   in addition to producing a new template and multimedia assets, the authors plan to provide training for city tech librarians and to discuss strategies to use libguides as part of the library’s instruction program. while creating a standardized template and faculty librarian training can make research guides more intuitive and easy to use, guides must be analyzed in relation to one-shot instruction sessions and reference desk interactions to ensure they align with pedagogical models. defining concrete usage contexts for guides will help city tech librarians tailor guides to meet student needs and help ensure that digital tools enhance information literacy outcomes.  learning modalities, research experience, and design correlation analysis   this study revealed a positive correlation between learning modality and design preferences. students who selected prototype b were more likely to indicate that they favour multimedia as their preferred learning modality during the interview phase of the study, and were more likely to mark interactive and multimedia elements as “important” during the advanced scribbling protocol phase (see figures 3 and 4, above). students who selected prototype a were more likely to indicate that they learn best by reading, and were more likely to mark text-based contextual elements as “important” during the advanced scribbling protocol.   the authors found no comparable correlation between participants’ level of research experience and design preferences but believe a larger sample size and more diverse participant pool may be needed to measure whether research experience is predictive of design preference. if such a correlation were found, librarians could more effectively customize guides for different research levels.    disparities and study limitations   while analyzing the study data, the authors identified some disparities and study protocol limitations. in some cases, student participants had difficulty reading design cues on the low-fidelity mock-ups, which is a known limitation of paper prototypes (sefelin, et. al., 2003). in particular, users were confused about image placeholders and assumed that certain static elements were hyperlinked. these limitations did not significantly skew data because of the mixed methods approach but may have impacted overall preferences for one prototype over another. disparities arose in instances where individual design elements were liked by some students but marked as confusing or unnecessary by others. these contradictory findings likely have more to do with personal design and learning modality preferences than a misreading of the prototypes, and illustrate that there is no single design approach that will please every user. lastly, we found that some of the advanced scribbling data on navigation is misleading since qualitative feedback illustrates that the problem is the navigation labels not the menu design. this may be an inherent limitation of the advanced scribbling protocol that researchers can mitigate by encouraging marginal annotation in addition to color-coding, and by combining this protocol with semi-structured interviews.                  areas for future research   this mixed methods user study raised some questions and illuminated several areas that require additional research. the authors would like to further explore relationships between user research experience, learning modality preferences, and design, perhaps by diversifying the study sample to include more advanced level researchers and different kinds of learners. while beyond the scope of this study, the authors acknowledge that different use contexts such as use by librarians in instruction sessions versus independent use by student researchers for subject specific libguides may impact user design preferences and influence what features they deem important. the authors question whether guides should place emphasis on discovery, information literacy, or resource curation, and how these decisions relate to theoretical and political conversations about the purpose of online instructional tools. are the guides intended to be used by students working independently on research assignments, as a supplement or replacement for face-to-face instruction? is the intended audience for research guides students, classroom faculty, or librarians? can a libguide serve all of these various purposes and audiences at the same time? how do these shifting contexts resolve themselves in design? do libguides and other digital instructional objects have a measurable effect on student research outcomes and achievement, and could they have more of an impact if they were deployed or designed differently?   the authors also hope to conduct more usability testing once the new libguides template is live since computer-based task analysis might present a clearer picture of how users interact with the libguides interface. alternative protocols, such as mobile and remote usability testing, should also be deployed after template implementation to assess whether students have issues with access or navigation on different devices. additionally, cuny is in the process of acquiring a consortial license for the libguides cms package, which includes an analytics package. implementation of the cms package will introduce more options for quantitative analysis of usage. although libguides analytics cannot capture the kind of granular data a usability study can, analytical data can provide insight into what areas of a guide are used frequently.   conclusion   this mixed methods user study yielded important insights into student design and modality preferences at city tech. while there is no single design approach that can satisfy all users’ preferences, there was consensus surrounding several concrete design features that can be hard-coded into a libguides template. the authors believe that the implementation of a new template and the creation of custom multimedia assets will make libguides more intuitive and accessible. additionally, librarian training sessions and institutional efforts to align research guides with library instruction and reference services may ultimately enhance pedagogical outcomes and start an important dialogue about instructional design. while the concrete design outcomes generated from this study may not be translatable to other institutional contexts, academic librarians can adopt the methodology articulated here to create effective libguides templates at their own institutions. additionally, the correlation between learner preferences and design principles identified here is applicable to other institutional and platform contexts, and should be studied further. ultimately, the authors hope this study will encourage other libraries to focus on user-centric design and spark a larger discussion about relationships between instructional design, learning modalities, and research guide use contexts.   references   bowen, a. (2014). libguides and web-based library guides in comparison: is there a pedagogical advantage? journal of web librarianship, 8(2), 147-171. http://dx.doi.org/10.1080/19322909.2014.903709   brumfield, e. j. (2010). applying the critical theory of library technology to distance library services. journal of library & information services in distance learning, 4(1-2), 63-71. http://dx.doi.org/10.1080/15332901003765795   city university of new york. (2017). online instruction at cuny. retrieved from http://www2.cuny.edu/academics/current-initiatives/academic-technology/online-instruction/   desimio, t., & chrisagis, x. (2014). rethinking our libguides to engage our students: easy diy libguides usability testing and redesign that works [pdf document]. retrieved from http://corescholar.libraries.wright.edu/ul_pub/168   dumuhosky, l., rath, l. t., & wierzbowski, k. r. (2015). libguides guided: how research and collaboration leads to success [pdf document]. retrieved from http://digitalcommons.brockport.edu/lib_presentations/2   duncan, v., lucky, s., & mclean, j. (2015). implementing libguides 2: an academic case study. journal of electronic resources librarianship, 27(4), 248-258. http://dx.doi.org/10.1080/1941126x.2015.1092351   gimenez, p., grimm, s., & parker, k. (2015). testing and templates: building effective research guides [pdf document]. paper presented at the georgia international conference on information literacy. retrieved from http://digitalcommons.georgiasouthern.edu/gaintlit/2015/2015/37   held, t., & gil-trejo, l. (2016). students weigh in: usability test of online library tutorials. internet reference services quarterly, 21(1-2), 1-21. http://dx.doi.org/10.1080/10875301.2016.1164786   hertzum, m., borlund, p., & kristoffersen, k. b. (2015). what do thinking-aloud participants say? a comparison of moderated and unmoderated usability sessions. international journal of human-computer interaction, 31(9), 557-570. http://dx.doi.org/10.1080/10447318.2015.1065691   hicks, a. (2015). libguides: pedagogy to oppress hybrid pedagogy. retrieved from http://www.digitalpedagogylab.com/hybridped/libguides-pedagogy-to-oppress/   linek, s. b., & tochtermann, k. (2015). paper prototyping: the surplus merit of a multi-method approach. forum: qualitative social research, 16(3). retrieved from http://www.qualitative-research.net/index.php/fqs/article/view/2236/3836   martin, b., & hanington, b. m. (2012). a/b testing. in universal methods of design: 100 ways to research complex problems, develop innovative ideas, and design effective solutions. (pp. 8-9). beverly, ma: rockport. nielsen, j. (2005). putting a/b testing in its place. in nielsen norman group. retrieved from https://www.nngroup.com/articles/putting-ab-testing-in-its-place/   nielsen, j. (2012). how many test users in a usability study? in nielsen norman group. retrieved from https://www.nngroup.com/articles/how-many-test-users/   sefelin, r., tscheligi, m., & giller, v. (2003, april). paper prototyping what is it good for? a comparison of paper-and computer-based low-fidelity prototyping. in chi '03 extended abstracts on human factors in computing systems. (pp. 778-779). acm. retrieved from http://mcom.cit.ie/staff/computing/prothwell/hci/papers/prototyping-sefelin.pdf   snyder, c. (2003). paper prototyping: the fast and easy way to design and refine user interfaces. san francisco: morgan kaufmann.   sonsteby, a., & dejonghe, j. (2013). usability testing, user-centered design, and libguides subject guides: a case study. journal of web librarianship, 7(1), 83-94. http://dx.doi.org/10.1080/19322909.2013.747366   spingshare. (2017). libguides. retrieved from https://www.springshare.com/libguides/   thorngate, s., & hoden, a. (2017). exploratory usability testing of user interface options in libguides 2. college & research libraries. retrieved from http://crl.acrl.org/index.php/crl/article/view/16739   tohidi, m., buxton, w., baecker, r., & sellen, a. (2006). getting the right design and the design right: testing many is better than one. in ch 2006 proceedings, april 22-27, 2006, montreal, quebec, canada. (pp.1243-1252). retrieved from https://www.billbuxton.com/rightdesign.pdf   u.s. department of education, national center for education statistics, higher education general information survey (hegis) (2014). total undergraduate fall enrollment in degree-granting postsecondary institutions, by attendance status, sex of student, and control and level of institution: selected years, 1970 through 2023 [data file]. retrieved from http://nces.ed.gov/programs/digest/d13/tables/dt13_303.70.asp   yoon, k., dols, r., hulscher, l., & newberry, t. (2016). an exploratory study of library website accessibility for visually impaired users. library & information science research, 38(3), 250-258.   young, s. w. h. (2014). improving library user experience with a/b testing: principles and process. weave: journal of library user experience, 1(1). http://dx.doi.org/10.3998/weave.12535642.0001.101     appendix a interview questions   1.       which research guide do you prefer? 2.       what don’t you like about the guide you didn’t select? 3.       which features of the research guide made you select this guide? 4.       how much content do you think should be included on a research guide? 5.       what labels or features did you find confusing?   6.       do the menu labels on this guide make sense to you?     appendix b prototype a ⇦ appendix c prototype b               research article   changes in the library landscape regarding visible minority librarians in canada   yanli li business and economics librarian wilfrid laurier university library waterloo, ontario, canada email: yli@wlu.ca   maha kumaran librarian education and music library university of saskatchewan saskatoon, saskatchewan, canada email: maha.kumaran@usask.ca   allan cho community engagement librarian (program services) university of british columbia library vancouver, british columbia, canada email: allan.cho@ubc.ca   valentina ly research librarian university of ottawa library ottawa, ontario, canada email: vly@uottawa.ca   suzanne fernando senior services specialist toronto public library toronto, ontario, canada email: sfernando@tpl.ca   michael david miller associate librarian & liaison librarian for french literature, economics and gender studies mcgill university library montréal, québec, canada email: michael.david.miller@mcgill.ca   received: 16 apr. 2022                                                                  accepted: 25 july 2022    2022 li, kumaran, cho, ly, fernando, & miller. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30151     abstract   objective – as a follow-up to the first 2013 survey, the visible minority librarians of canada (vimloc) network conducted its second comprehensive survey in 2021. the 2021 survey gathered detailed information about the demography, education, and employment of visible minority librarians (vmls) working in canadian institutions. data from the 2021 survey and the analysis presented in this paper help us better understand the current library landscape, presented alongside findings from the 2013 survey. the research results will be helpful for professional associations and library administrators to develop initiatives to support vmls.   methods – researchers created online survey questionnaires using qualtrics xm in english and translated them into french. we distributed the survey invitation through relevant library association electronic mail lists and posted on vimloc’s website, social networking platforms, and through their electronic mail list. the survey asked if the participant was a visible minority librarian. if the response was “no,” the survey closed. respondents indicating "yes" were asked 36 personal and professional questions of three types: multiple-choice, yes/no, and open-ended questions.   results – one hundred and sixty-two vmls completed the 2021 survey. chinese remained the largest ethnic identity, but their proportion in the survey decreased from 36% in 2013 to 24% in 2021. 65% were aged between 26 and 45 years old. more than half received their library degree during the 2010s. 89% completed their library degree in canada, a 5% increase from 2013. the majority of librarians had graduated from university of toronto (25%), followed closely by university of british columbia (23%), and western university (22%). only 3% received their library degree from a library school outside north america. 34% of librarians earned a second master’s degree and 5% had a phd. 60% of librarians had less than 11 years of experience. nearly half worked in academic libraries. most were located in ontario and british columbia. 69% of librarians were in non-management positions with 5% being senior administrators. 25% reported a salary above $100,000. in terms of job categories, the largest group worked in reference/information services (45%), followed by instruction services (32%), and as liaison librarians (31%). those working in acquisitions/collection development saw the biggest jump from 1% in 2013 to 28% in 2021. 58% of librarians sought mentoring support, of whom 54% participated in formal mentorship programs, and 48% had a visible minority mentor.   conclusion – 35% more vmls responded to the 2021 survey compared to the 2013 survey. changes occurred in ethnic identity, generation, where vmls earned a master of library and information science (mlis) or equivalent degree, library type, geographic location, and job responsibilities. the 2021 survey also explored other aspects of the vmls not covered in the 2013 survey, such as librarian experience, salary, management positions, and mentorship experience. the findings suggested that the professional associations and library administrators would need collaborative efforts to support vmls.     introduction   for decades there has been an awareness and recognition that the library workforce does not reflect the diversity of the population in canada (capal, 2019; jennings & kinzer, 2022). the visible minority librarians of canada (vimloc) network formed in december 2011 to connect, engage, and support racialized librarians in the country (vimloc, n.d.). in 2013, on behalf of vimloc, maha kumaran and heather cai (2015) conducted the first survey of its kind (referred to as the “2013 survey”) to gather statistical information on canadian visible minority librarians (vmls). the 2013 survey assessed vmls' educational qualifications and employment details to help identify their needs, challenges, and barriers within the profession. besides this foundational survey, there remains very little information of this kind on racialized librarians across different institutions in canada. when implementing the 2013 survey, vimloc had planned to repeat its survey to better understand the changes in the library workforce over time. as such, the authors designed and administered a redux survey in english and in french between january and march 2021 (referred to as the “2021 survey”). the 2021 survey investigated two aspects: 1) similar demographic questions as the previous survey; and 2) additional questions that sought to explore the experiences of vmls in the workplace.   this paper focuses on the results from 162 respondents of the 2021 survey and compares results, where applicable, to the 2013 survey (kumaran & cai, 2015). for questions that are not covered in the 2013 survey, findings will be compared to other studies as appropriate. this research will help vmls understand how their position in the library landscape has changed over the years. recommendations provided will help professional associations and library administrators continue to develop initiatives to advocate for vmls, which in turn will contribute to the promotion of equity, diversity, and inclusion (edi).   literature review   the literature review for this article focuses on many topics in librarianship such as racial diversity, lack of minorities in leadership positions, salary inconsistencies, and mentorship.  while the major focus of the literature review is from a canadian context, some papers from the united states are cited to provide a broader context for racial diversity in librarianship. in the literature, the terms “visible minority librarians”, “librarians of colour”, “racialized librarians”, and “ethnic minority librarians” are one of many terms to refer to the population of interest (kandiuk, 2014; kumaran, 2012; kumaran & cai, 2015; kung et al., 2020). a scarcity of professional literature on and by vmls in a canadian context continues to persist, even years after the initial vimloc study, with a notable absence of publications in french.   demographics   the data on racialized library professionals has been sparse for much of the history of canadian libraries, although it has gradually increased. kumaran and cai’s (2015) “identifying the visible minority librarians in canada: a national survey” is one of the most comprehensive studies of diversity in canadian libraries. more research from census surveys has been released since then. the canadian association of professional academic librarians (capal) conducted censuses in 2016 and 2018. though limited to academic libraries in canada, the capal censuses built a comprehensive demographic picture of the profession of academic librarianship by collecting data about librarians working in colleges and university libraries. in 2021, the canadian association of research libraries (carl) (n.d.) released a diversity and inclusion survey to gather baseline data on the composition of personnel in 21 carl libraries, to gauge employee feedback on current edi initiatives, and to establish a set of benchmarks against which to evaluate and measure the impact of their edi strategies and practices.   recruitment and retention   beyond just the demographics of diverse librarians in the field is an emerging body of literature about diversity within the scope of racism in libraries. in kung, fraser, and winn’s (2020) systematic review, the authors found that despite a number of approaches used to recruit minorities in academic libraries, the number of visible minorities in the field has remained stagnant for decades. their research indicated an established body of literature that defined diversity, with race, ethnicity, gender, and class identified as the most frequently used dimensions. the authors found an increased number of publications on diversity in librarianship in the 2000s, drawing more attention to the topic over the past twenty years than prior to that time. in particular, residencies, internships, and mentorship were the major interventions for recruitment and retention, but residency programs existed in an american context, and not so much within the canadian context. the papers analyzed in the study mostly focused on recruitment and retention for early career librarians, but less so about advancement of more senior librarians.    leadership   although the literature is somewhat limited, there are three areas to highlight from the literature that relate to our paper: leadership, mentorship, and salary. the recently published carl diversity census and inclusion survey revealed that racialized library staff were underrepresented in senior leader and other managerial roles (ccdi, 2022). kumaran’s (2012) leadership in libraries focused on ethnic-minority librarians and is one of the more comprehensive texts targeting strategies for success. written primarily for first-generation immigrant librarians, kumaran explored the major cultural differences affecting leadership from mainly asian and african cultures in the context of white mainstream libraries, including cultural adaptation and language issues. hines’ (2019) research focused on academic librarians and further explored how current leadership development opportunities reinforced the existing biased structures within libraries. using the lens of critical race theory, hines offered tools to better describe and understand the problems so that they could be addressed meaningfully, chiefly through a restructuring of both the mechanics and the curriculum of leadership development training.   salaries   salary has been understudied for library professionals in canada, not to mention for vmls. while capal’s 2016 and 2018 censuses of academic librarians gathered visible minority status, ethnic identity, and salary information, their summary reports did not include any analysis of the relationship between salary and race (capal, 2016; capal, 2019). the survey conducted by the canadian association of university teachers (2019) provided average academic librarian salaries by gender, age, region, and institution, but not by race. the 8rs practitioner survey in 2014 had microdata on salary and visible minority status for carl librarians (delong et al., 2015a). using a subset of those microdata, li (2021) used multiple regression models to study demographic, job, and labour market factors that affected carl librarians’ salaries. a significant pay gap was identified between vmls and non-vmls.   mentorship   mentorship has proven to be one of the most significant factors contributing to a librarian’s career success in canada (harrington & marshall, 2014; law, 2001; oud, 2008). mentoring can be provided in formal and informal formats (damasco & hodges, 2012; mackinnon & shepley, 2014). particularly, for vmls who face challenges entering the canadian job market and adapting to the workplace climate and culture, getting mentoring support is essential (kandiuk, 2014; kumaran & cai, 2015). kung, fraser, and winn (2020) found that mentorship was used at academic libraries for retention of vmls, but there was an overall lack of focus on midto latecareer vmls.   as mentioned in kumaran & cai’s (2015) study, this absence of professional literature by minority librarians could have many causes:   [librarians] are in positions that do not require them to publish; lack of training in writing academic papers, especially if they are first generation minority librarians; lack of support for writing for publication; lack of time or funding; not having a dedicated minority-focused canadian library journal that allows them to voice their thoughts; and perhaps fear of bringing attention to themselves” (p.111)   being in a position that does not require publication is particularly true for quebec academic librarians in francophone colleges and universities where they do not yet hold academic status or an equivalent/parallel to academic status. this does not mean discussions like these are not happening in french; they simply are not being distributed via professional and academic publications. there is also a gap in the literature where librarians are not writing about minority librarians. based on data from the 2021 survey focusing on vmls, this research will contribute to filling the gap. a future study on the motivations for such lack of writing could be extremely beneficial to the current body of canadian minority librarian research.   methods   building on the 12 questions from the 2013 survey, wording for seven of the original questions was updated for clarity or to reflect changes to the profession, and 24 additional questions were added to the 2021 survey. an online survey questionnaire was created using qualtrics xm (see appendix). the entire survey was sent to the 2021 survey research team and vimloc committee members for a pilot test before they were released to the target audience. after ethics approvals from the authors’ respective institutions, the english language survey was made available between january 21, 2021 and february 28, 2021. it was a nation-wide survey with participation from vmls working in canadian institutions. the online survey invitation was sent to vmls through relevant library association electronic mail lists, such as carl, capal, and provincial library associations. the invitation was also posted on vimloc’s website, three social networking platforms (twitter, facebook, and linkedin), and through their electronic mail list. when the english survey was distributed, the research team received inquiries about the availability of a french version of the survey. to help get information from french racialized minority librarians and to fill the gap in the work of vimloc, the research team decided to do a french version. one team member translated the english survey to french and circulated it amongst québecois library associations and other networks between march 1, 2021 and march 31, 2021.   the 2021 survey provided a definition of visible minorities from the canadian employment equity act: “persons, other than aboriginal peoples, who are non-caucasian in race or non-white in colour” (government of canada, 2021).  the participants were asked to identify if they were a visible minority librarian. if the response was “no,” the survey closed. the rest of the survey consisted of personal and professional questions of three types: multiple-choice, yes/no, and open-ended. specifically, there were six questions about demographic information, 10 questions about education, and 20 questions about employment. for details of the questionnaire, see the appendix. for the purpose of this research, we analyzed microdata on these librarians. we conducted cross-tabulation and chi-square analyses for in-depth examinations of the relationships between some variables, including employment type and career stage, type of mentorship and perceptions of the helpfulness of mentorship in librarians’ career development, and the visible minority status of mentors and perceptions of their helpfulness for mentees.   results and discussion   of the 294 librarians that attempted the 2021 survey, 162 respondents who identified themselves as vmls were permitted to complete it, representing a 35% increase from the 120 participants in the 2013 survey. there were 138 librarians who completed the english survey and 24 who completed the french survey.   demography   the questions in this section focused on ethnic identity, generation status, disability status, gender, and age.   ethnic identity   respondents were asked to self-identify their ethnic group. without accounting for mixed races, the largest ethnic identity represented among the 159 respondents was chinese (24%, n=38), compared to 36% (n=43) in the 2013 survey, followed by south asian (15%, n=24), and black (12%, n=19) (figure 1). the percentage of black librarians remained unchanged between the two surveys at 12%. the proportions of latin american, korean, filipino, southeast asian, and arab librarians increased slightly, whereas those of south asian, west asian, and japanese librarians decreased slightly. in 2021 there were 13% (n=21) respondents that identified as a mixture of white and visible minorities and 9% (n=14) that identified as multiple visible minorities.   generation status   participants were asked about their generation status in canada. first generation visible minorities refer to those who were born elsewhere and moved to canada at some point during their lives. second generation visible minorities are those who were born in canada to one or more immigrant parents. third generation or more refers to visible minorities who were born in canada, with both parents who were also born in canada (statistics canada, 2021). as shown in figure 2, of the 157 respondents in the 2021 survey, 56% (n=88) identified themselves as first generation, compared to 63% (n=76) in 2013 (kumaran & cai, 2015, p.113). 40% (n=63) identified themselves as second generation, compared to 28% (n=33) from the 2013 survey (28%, n=33). the portion of third generation or more librarians was 4% (n=6) in 2021 versus 9% (n=11) in 2013.   figure 1 ethnic identity.   figure 2 generation status.   disability status, gender, and age   of the 159 respondents in the 2021 survey, 8% (n=12) identified themselves as a person with a disability. ten respondents provided details of their disability conditions that included physical and mental disabilities, and chronic illnesses. librarians were predominantly female (81%, n=130) with 16% (n=25) males, and nearly 4% (n=6) had other gender identifications or preferred not to answer. as shown in figure 3, those aged 31-35 accounted for 22% (n=36), followed by 36-40 (16%, n=26), 41-45 (15%, n=25), and 46-50 (13%, n=21). only 10% (n=16) were over the age of 55, which suggested that the respondents were younger than the overall visible minority workforce, of which 15% were over the age of 55 (statistics canada, 2017).   figure 3 age group.     education   library degree   the questions in this section focused on when and where participants received their professional library degree – master of library & information science (mlis) or equivalent – and whether or not it was american library association (ala) accredited. figure 4 shows that more than half (53%, n=86) received their library degree during the 2010s, doubling the percentage of those receiving their library degree during the 2000s (24%, n=39). seven librarians received their degree before the 1990s and eight librarians received it after 2019.     when asked where they completed their library degree, 89% (n=142) of the 160 respondents indicated “from an ala-accredited canadian library school,” as compared to 84% (n=101) in 2013 (kumaran & cai, 2015, p.113). 5% (n=8) received their library degree from an ala-accredited american library school. only 3% (n=4) mentioned getting their library professional degrees from outside north america (figure 5). this small proportion could be because ala accreditation has been an impediment for librarians with foreign credentials (taleban, 2016). many immigrant librarians received an additional library degree through an ala-accredited program after moving to canada.    for the 142 respondents who completed their library degree in canada, they were asked to indicate the university that granted their library degree. the top three institutions were the university of toronto (25%, n=35), the university of british columbia (23%, n=32), and western university (22%, n=31) (figure 6). in comparison, in the 2013 survey, of the 101 respondents, about 15% received their library degree from the university of toronto, almost 40% from the university of british columbia, and 31% from western university (kumaran & cai, 2015, p.113). it is worth noting that 14% (n=20) of the respondents graduated from université de montréal. this data was not collected in 2013 when the survey was not conducted in french.   seventeen respondents provided the name of the country outside of canada where they received their library education. almost half (n=8) indicated the united states; other countries mentioned included brazil, france, ghana, india, iran, singapore, and the united kingdom. of the nine respondents who had an international non-ala accredited library degree, four stated that their degree was recognized for their current employment at public or academic libraries, whereas the other five had different experiences. two librarians felt compelled to complete a canadian mlis degree to secure a job; another received a library technician diploma in canada and ended up with librarian status after years of doing non-librarian jobs. the other two did not get their foreign degree recognized and were working in a non-library setting or in a position relying more on their non-librarian experience and skills.   figure 4 year library degree was received.   figure 5 where library degree was received.   figure 6 university where library degree was received.   additional education   in addition to an mlis degree, the respondents have attained professional degrees, additional certificates, diplomas, or advanced degrees. in table 1, of the 152 respondents, 21% (n=32) earned professional degrees, and 34% (n=51) had their second master’s degree. this finding was close to the carl’s 8rs redux survey which reported 32% with a second master’s (delong et al., 2015b, p.100) but lower than 57% in the 2018 capal census (capal, 2019, p.45). 2% (n=3) of the respondents in the 2021 survey reported having a third master’s degree, as compared to 3% in the 2018 capal census. 5% (n=8) held a phd which was in line with the carl’s 8rs redux survey result but much lower than nearly 11% among academic librarians in the 2018 capal census. 38% (n=58) indicated they had additional degrees, certificates or diplomas, specifically 34 bachelor’s degrees, 25 certificates, seven diplomas, and 10 other education attainments. these research results suggested evidence of a trend of increasing professional and graduate education among librarians. this may be attributable to an increased demand for librarians to perform specialist roles that require additional credentials after they have entered the librarian profession. as revealed in ferguson (2016), 26% of the 800 academic library job postings preferred a second advanced degree and 7% required one. the most frequent functional areas asking for advanced subject knowledge were subject specialists. librarians pursuing additional education may also be due to personal interest or the possibility of support from their current institutions with funds and time for studying. it is also possible that some vmls have earned non-mlis degrees in their home country before pursuing librarianship in canada, or that they feel the need to upgrade themselves with additional qualifications to sustain their professional positions here in canada.   table 1 non-mlis education attained education     count percentage professional degree 32 21% second master’s degree 51 34% third master’s degree 3 2% phd 8 5% additional degrees, certificates, or diplomas 58 38%   employment   the questions in this section focused on librarian experience, library type, geographic distribution, type of employment, leadership positions, salary, job categories, and mentorship experience. the librarians were also surveyed about their experiences in workplaces such as microaggressions and job satisfaction. these questions deserve an in-depth study and will be published in a separate paper.   librarian experience   librarians with less than six years of experience made up 35% (n=56), followed by 6-10 years (25%, n=41), 11-15 years (13%, n=21), 16-20 years (12%, n=20), and 21-25 years of experience (9%, n=14). 6% (n=9) have been a librarian for more than 25 years (figure 7).     figure 7 librarian experience.   library type   from 2013 to 2021, the most noticeable change was the increase of respondents working in academic libraries. as shown in figure 8, 48% (n=78) of vmls in the 2021 survey identified themselves as working in academic libraries, compared to 38% (n=45) in the 2013 survey. conversely, the spread of respondents employed at public, special, and school libraries was lower than previously captured. the increase of respondents from academic libraries could be due to the retirement wave hitting academic librarianship. the carl 8rs 2014 practitioner survey revealed that 34% of all carl librarians expected to retire within the next 10 years (delong et al., 2015b, p.47). many studies have examined succession planning at canadian academic libraries when bracing the reality of baby boomer librarians retiring (guise, 2015; harrington & marshall, 2014; popowich, 2011). kumaran (2015) pointed out the importance of including vmls in the succession planning process. another explanation for the increased academic librarian participation could be due to the more effective application of policies towards edi in universities in recent years. in addition to having general employment equity policies in place, some universities have set goals to increase hiring of visible minority staff (university of victoria, 2015). as an example of professional library associations, carl (2020) has realized the significance of edi in academic libraries and published a guide to aid recruitment and retention of diverse talent. in such contexts, visible minorities may have more opportunities to enter academic librarianship compared to nearly a decade ago. it is noteworthy that studies on librarian turnover at non-academic libraries are very limited. further research is needed to explore how employment of vmls has changed at those libraries and how that change may have affected employment at academic libraries.   figure 8 employment by library type.   geographic distribution   as in the 2013 survey, vmls were widely spread across canada, with respondents from prince edward island and yukon participating in the 2021 survey (table 2). a vast majority of respondents continued to be in british columbia and ontario; however, british columbia comprised 22% (n=35) of these employed librarians, compared to 40% (n=48) in 2013, whereas respondents from ontario accounted for 45% (n=72) in 2021 versus 27% (n=32) in 2013. due to adding a french iteration of the survey in 2021, librarians in quebec made up 14% (n=22), compared to only 4% (n=5) in 2013. the geographic distribution of vmls reflected similar patterns of visible minority populations across canada. the 2016 census data showed that ontario, british columbia, and quebec were the top three most populous provinces for visible minorities (statistics canada, 2018b).   table 2 geographic distribution province or territory 2013 vimloc survey 2021 vimloc survey alberta 8% (n=10) 7% (n=11) british columbia 40% (n=48) 22% (n=35) manitoba 6% (n=7) 2% (n=3) new brunswick 2% (n=2) 1% (n=1) newfoundland and labrador 1% (n=1) 1% (n=2) nova scotia 5% (n=6) 3% (n=4) nunavut 1% (n=1) 0% (n=0) ontario 27% (n=32) 45% (n=72) prince edward island 0% (n=0) 1% (n=1) quebec 4% (n=5) 14% (n=22) saskatchewan 6% (n=7) 3% (n=4) yukon 0% (n=0) 2% (n=3)   type of employment   in 2021, an overwhelming 85% (n=137) of respondents were working in permanent positions and 11% (n=18) were in temporary positions (e.g., contract, limited-term). a larger proportion of librarians were working full-time (30 or more hours/week) (90%, n=143 in 2021 versus 82%, n=99 in 2013). these findings seemed to be more positive compared with other studies which reported that precarious employment was on the rise in libraries (henninger et al., 2019; o’reilly, 2015; the canadian press, 2016) and that minorities were disproportionately affected (cupe, 2017). henninger et al. (2020) analyzed job postings and found that employees with managerial positions, advanced degrees, or more experience, and positions requiring an mlis were least likely to be precarious. in the 2021 survey, a vast majority of the librarians had an mlis or equivalent, one of five librarians had professional degrees, and two of five librarians had at least two graduate degrees. also, 65% had more than five years of experience as a librarian. these factors might have helped them secure stable jobs. given the small sample size of this survey, it might also be possible that those working on part-time or temporary jobs were not participating in the 2021 survey; hence their proportions might be underestimated.   this research further examined employment of vmls by career stage. career stage consists of early career, mid-career, and late career. however, the concept of each career stage is open to interpretation. it is either based on year of graduation from an mlis program (delong et al., 2015b), or post-mlis experience in a librarian role and the number of years the librarian has worked for their current employer (tucker, 2008). the data collected in the 2021 survey did not align with these studies. for instance, the survey asked about years of librarian experience, but did not ask how many years the librarians worked for their current employer. therefore, it was not possible to analyze the career stage as defined in the above studies. instead, the authors referred to sullivan (2011) and morison et al. (2006), who defined career stage by age. this research broke down the sample into three subgroups: early career (ages 26-35, 34%, n=56), mid-career (ages 36-55, 56%, n=89) and late career (ages 55+, 10%, n=16). as illustrated in figure 9, of all librarians in permanent or full-time positions, around one third were at early career stage and over half were at mid-career stage. on the contrary, early career librarians comprised over a half of those in temporary or part-time positions. early career minority librarians seemed to have more challenges securing a full-time or permanent job than their mid-career peers, as manifested through the experiences of some new librarians (ford, 2021; lee, 2020). however, the results from a chi-square analysis of this research indicated that there was no statistically significant relationship between career stage and full-time or part-time employment, x2 (4, n = 159) = 3.86, p = .426, or between career stage and permanency of the employment, x2 (4, n = 161) = 5.22, p = .266.   figure 9 employment by career stage.   leadership positions   in terms of their current position, 69% (n=110) of the 160 respondents were not in managerial positions. an equal share (13%, n=21 each) were supervisors and middle managers (e.g., branch head, department head). only 5% (n=8) were senior administrators (e.g., head/chief librarian, director, or deputy/assistant head, chief, director). in comparison, carl’s 2021 survey revealed 14.4% of senior leaders identified as racialized, compared to 20% of all library staff (ccdi, 2022). furthermore, 8rs 2014 practitioner survey reported that 55% of carl librarians were not in management positions and 15% were senior administrators (delong et al., 2015b, p.18). participants expressed their frustration by lack of “racial diversity among librarians” and being “flat out dismissed” for expressing such concerns. in addition, they were accused of being frustrated that they could not find jobs. in one situation where the respondent wrote:   i was the most qualified and experienced person applying to run my library and acted successfully in the job for over 8 months, i failed the standardized leadership tests and was screened out of the competition and a non-mlis candidate with no library experience was placed in the leadership positions.   the findings from this research supported that vmls were less likely to be working in senior administration positions for various reasons including lack of leadership training. respondents mentioned attending mentorship programs such as the arl leadership and career development program (arl. n.d.-a), certificate program on public library leadership (ontario library service, n.d.), and public library interlink (n.d.)’s llead program, all of which have a focus on developing leadership skills. it should be noted that there are no leadership programs that aim to develop minority library leaders in canada. some libraries have tailored leadership programs for their employees; however, none focuses on visible minority employees and their leadership skills development.   salary   respondents were asked about their gross yearly salary. about 17% (n=26) of the 158 respondents reported a salary at $60,000 or less, 55% (n=86) reported a salary between $60,001 and $100,000, 25% (n=40) reported a salary above $100,000, and 4% (n=6) preferred not to answer. according to the 2016 canadian census data (statistics canada, 2018a), the median employment income for vmls was $59,710, meaning half of the total 1,055 vmls had an employment income above this amount, and half had an employment income below this amount. hence, using $60,000 as an approximate benchmark, 80% of vmls in the 2021survey earned more than this amount compared to 50% of the general visible minority librarian population in the 2016 census.   job categories   respondents were asked to select as many of the job categories that match their current job responsibilities. as shown in figure 10, the majority worked in reference/information services (45%, n=72), followed by instruction services (32%, n=52), and as liaison librarians (31%, n=50). their proportions were 38% (n=46), 14% (n=17), and 17% (n=20), respectively, in 2013. those working in acquisitions/collection development accounted for 28% (n=45) in 2021 compared to only 1% (n=1) in 2013. the rate of librarians working in cataloguing/metadata management was 7% (n=12) in 2021 versus 11% (n=13) in 2013. there were no changes in the proportions of respondents working in automation/systems/it services, licensing, and school/teacher librarian jobs. moreover, additional job categories were added in the 2021 survey to reflect recent trends in librarian responsibilities, including public services (29%, n=47), research services (27%, n=43), user experience (14%, n=23), project management (12%, n=19), data management and curation (8%, n=13), bibliometrics (7%, n=11), copyright (7%, n=12), and scholarly communications (6%, n=10).   mentorship experience   in the 2021 survey, of the 160 respondents, 58% (n=93) indicated that they sought support from mentors throughout their library career. 54% (n=50) of those respondents participated in formal mentorship programs and nearly half (48%, n=45) had a visible minority mentor. these figures were much higher than those reported in kandiuk’s (2014) study, which found that 32% (n=18) of vmls had been mentored, with only 22% (n=4) of them engaged in a formal mentoring relationship, and the same number had a visible minority mentor.   forty-three respondents participated in formal mentorship programs that were offered in workplaces (n=6), library schools (n=7), or professional associations (n=30). thirteen librarians mentioned participating in more than one formal mentorship program. the most cited professional association offering a mentorship program was vimloc (n=15), followed by ontario library association (n=8), british columbia library association (n=6), among others.   figure 10 job categories.   when asked how mentors were helpful in supporting them, 26% (n=24) of the respondents indicated “extremely helpful”, 32% (n=30) indicated “very helpful”, 30% (n=28) indicated “moderately helpful”, 9% (n=8) indicated “slightly helpful”, and 3% (n=3) indicated “not at all helpful”. we further separately examined the librarians who engaged in formal and informal mentorship (table 3). respondents engaging in formal and informal mentorship were nearly equivalent in their rates of feeling the mentors were extremely helpful, very helpful, and moderately helpful. there was a divergence of more negative mentorship experiences from formal mentorship programs. 6% of the librarians in formal mentorship did not find their mentor helpful at all, whereas no respondents in informal mentorship thought so. these findings reflected those of damasco and hodges (2012), where academic librarians of colour were more likely to cite informal mentoring as an effective form of professional development than formal mentoring and perceive formal mentoring as an ineffective form compared to informal mentorship (p. 293). we performed a chi-square test to further examine the relationship between the type of mentorship and participants’ perception of the helpfulness of mentorship. the relation was not statistically significant, x2 (4, n = 93) = 3.44, p = .488 (table 3).   we thought it would also be useful to examine whether these librarians perceived the value of mentorship differently if their mentor was a visible minority. in table 3, the librarians who had a visible minority mentor were more likely to feel that their mentors were extremely helpful (33% versus 19%) or very helpful (38% versus 27%) as compared to those who had a non-visible minority mentor. conversely, the librarians who did not have a visible minority mentor were more likely to find their mentors were moderately helpful (40% versus 20%), slightly helpful (10% versus 7%), and not helpful at all (4% versus 2%). however, the chi-square test indicated that there was no significant relationship between visible minority status of the mentor and mentees’ perception of the helpfulness of mentorship, x2 (4, n = 93) = 6.35, p = .175 (table 3).    table 3 perceptions of the helpfulness of mentorship type of mentorship extremely helpful very helpful moderately helpful slightly helpful not helpful chi-square value p value formal mentorship 13  (26%) 16 (32%) 15  (30%) 3 (6%) 3 (6%) 3.44 0.488 informal   mentorship 11  (26%) 14 (33%) 13 (30%) 5 (12%) 0  (0%) minority mentor 15  (33%) 17 (38%) 9 (20%) 3  (7%) 1 (2%) 6.35 0.175 nonminority mentor 9  (19%) 13 (27%) 19 (40%) 5 (10%) 2 (4%)   there could be a couple of reasons why there was no difference in their perceptions between having or not having a minority mentor, for example, lack of visible minorities in managerial or senior administrative roles, or lack of public knowledge of their racial identity. however, two of the respondents highlighted the importance of having a minority mentor:    mentors are so important for bipoc [black, indigenous, and people of colour] librarians. most of the things i know about librarianship, about the unsaid things i should know (knowledge sharing) or how to navigate this field, come from wonderful and talented bipoc librarians, who deserve their flowers and increased pay! honestly, i think mentors are the primary reason most of us early career bipoc librarians stay in this field. so, please continue the program or create a space to informally or formally interact more. bipoc/visible minority librarians are often alone at their jobs and having support from others is uplifting and empowering.   in formal mentorship programs, i have not had a mentor from a visible minority group. however, i've had informal mentors who are people of colour who have been generous with sharing their experience and perspectives about issues like feeling tokenized, moving to a new city that is less diverse, the dynamics of edi committees, and the lack of movement on issues around social justice (racial and otherwise) in libraries.   findings and recommendations   in light of the research findings, the authors would like to make the following recommendations from the perspectives of the canadian library profession, library administrators, vmls, and vimloc network.   canadian library profession   there is fluctuation in which groups of racialized minority librarians increased or decreased with the two surveys. the change in numbers may be connected to the immigration patterns in canada. however, the truth remains that the vmls continue to grow in numbers and are looking for opportunities. although many are still in career exploration stages, 38% of librarians have 6-15 years of work experience. however, 67% of first generation or immigrant librarians and 73% of second generation librarians are not in leadership positions. divided by ethnicity, 64% of black librarians, 79% of chinese librarians, and 58% of south asian librarians are not in such leadership roles.   recommendation: regarding vmls, there is a huge market that the canadian library profession has left untapped for future leadership positions, thus depriving the profession of diverse talents and perspectives. the library profession would benefit from a program that helps vmls visualize themselves in leadership positions and succeed in their leadership undertaking.   as hines (2019) mentioned, leadership programs continue to reinforce biased structures. most canadian librarians attend leadership programs from the united states such as the leadership and career development program or the mosaic program (arl, n.d.-a., n.d.-b). while these programs are immensely helpful, they have an american focus; costs are in american dollars and there is a requirement to travel to the united states. due to costs and travel needs outside of canada, this could be an impediment to vmls in their leadership development.   recommendation: the canadian library profession needs a library leadership program that focuses on racialized minorities, offered in both english and french and through a hybrid model that includes both online and in-person formats.    there are non-ala accredited librarians who are already in canada waiting to gain employment. however, there are no pathways for them to assess their education and experience and compete equally in the job market. canadian librarianship could design pathways to evaluate international librarian credentials through programs such as the international qualifications assessment service (government of alberta, 2022), or other similar assessment bodies within their campuses. for example, the british columbia institute of technology (n.d.) has an international credential evaluation service dedicated to their employees and students. canadian librarianship can work with the canadian information centre for international credentials (n.d.) or the alliance of credential evaluation services of canada (n.d.) to create a standardized evaluation process for library courses and programs. certificate programs to upgrade skills could be an option, however, this would still be a cost impediment for many first-generation immigrant librarians who already have a degree, experience, and are new to the country. universities that offer professional library degrees could consider offering certification programs to these librarians so they do not have to pay the full fees to receive a degree. the canadian library profession, library associations, and academic library standards on hiring should mandate that such programs, along with previous experience in the profession be accepted for hiring these librarians as equals to ala-accredited librarians. their additional master’s and doctorate degrees also need to be taken into consideration.   recommendation: canadian librarianship could design pathways to assess, and if necessary, upgrade international librarian qualifications.   library administrators   as the results of this research show, there is a conspicuous lack of racialized librarians in management or leadership positions. as one respondent pointed out, participants were tested for leadership skills through a test. the respondent considered this “a barrier to racialized employees…seeking higher leadership opportunities,” but their feedback of the issue of equity in such tests were not addressed by their administration. more than one respondent mentioned feeling isolated or experiencing racial microaggressions in their current positions and were disappointed that their concerns were not being heard by their leadership. library administrators could encourage and coach these minority librarians about their career goals to include them in the succession plan and groom them for future leadership opportunities. this would also be one way for library administrators to sustain their edi commitments and strengthen their retention practices.   recommendation: all library leaders need to undergo ongoing training through their institutions, library and national associations, so they understand the perspectives of minority librarians. leaders should also consider mentoring and preparing bipoc in their institutions for future leadership positions.   the survey result shows that four of the nine participants with international non-ala accredited degrees are currently employed. this may suggest that there exists a possibility for canadian libraries to recognize non-ala accredited degrees. in fact, many american academic library job postings are broadening the eligibility criteria by changing the education requirements from ala accredited degrees to be more inclusive of foreign lis education (burtis et al., 2010). in carl’s guidelines on hiring and retaining diverse talent, one of the recommended strategies is to “consider broadening the eligibility for librarian positions to be inclusive of lis accredited degrees beyond ala-accreditation” (carl, 2020). another important question to be considered is whether the work experience of these four librarians would benefit their future career paths in librarianship. would their experience give them an equal standing along with librarians who have an ala-accredited degree, or are they destined to stay in their current positions and status for fear of losing what they currently have?   recommendation: library administrators who are not hiring librarians with non-ala accredited degrees could examine the reasons why and the processes of how some libraries are able to do so.   visible minority librarians   an increased number of first generation librarians responded to the 2021 survey than the previous one. for visible minorities who want to enter canadian librarianship, it is important to consider additional training or knowledge to avoid culture shock while working in canadian libraries. it is essential to know how to work with patrons, their rights, the design and delivery of library policies, and their impact on practice and patrons. it is important to understand the structure of libraries. for example, what does it mean when a library is considered regional versus public, or college versus university, especially in terms of its structure, governance, and funding? it is also important for these librarians to find a network to get support in their job search, in creating their resumes or building cvs, and practice interview skills with trusted colleagues. additionally, librarians can take advantage of many learning opportunities outside of library schools. for example, the library juice academy (2022) offers regular courses on many current and relevant library topics. librarians can also join many library association listservs to receive updated information on library happenings. if small costs are not an issue, librarians can join library associations for a small membership fee. such membership may provide access to knowledge resources, webinars, and other informational materials, which may be valuable when preparing for interviews.   recommendation: vmls need to be proactive in finding learning, network, and job opportunities in canada.   vimloc   the 2013 survey identified the importance of mentoring support for visible minority librarians. thereafter, vimloc developed its own mentorship program and ran the first session during 2013-2014. this program was reinitiated in 2018 and continued to run on an annual basis. the findings from the 2021 survey showed that 35% (n=15) of the librarians participated in the vimloc mentorship program and two out of three librarians found the mentors were extremely helpful or very helpful. vimloc can support vmls through other efforts, for example, organizing panel presentations that host minority library leaders and highlight their pathways into leadership. such presentations will empower vmls and help them design their own career pathways. vimloc can collaborate with other associations as well. while vimloc currently does not have the resources to host conferences similar to the joint conference of librarians of color (2022), it could connect with canadian health libraries association (chla), capal, congrès des profesionnel·le·s de l’information (cpi) or ontario library association (ola) conferences and add a minority focused session/stream in those conferences. the canadian journal of academic librarianship (2019) hosted a special issue with a focus on diversity. in addition, vimloc has partnered with university of toronto libraries to organize the navigating the field workshop series targeted for those new to applying for academic jobs (vimloc, 2021).   recommendation: vimloc should continue to implement the mentorship program and pursue more collaborative opportunities to support vmls.    limitations   first, 162 visible minority librarians completed the 2021 survey, representing approximately 15% of the visible minority librarian population (1,055) based on the 2016 census data (statistics canada, 2018a). this means that the findings from this research may not provide a complete picture of the visible minority librarian population in canada.   second, data collection errors might occur in the survey and affect analysis results in this research. representation data for members of visible minorities was based on voluntary self-identification. the information on librarians was self-reported such as disability status, salary, years of librarian experience, and respondent’s perception of how mentorship was helpful.  regarding gender identity, the 2021 survey used the binary biological terms, male/female, which are normally used for gender and not interchangeable with gender identity. in addition, transgender identity was recorded separately from male/female identity, which might have resulted in reporting errors.   third, there is a limitation of using age to assume a career stage. some people pursue librarianship as a second or third career. hence, they may be older in age, but are still in an early career stage. someone may enter the librarian profession at a very early age, and possibly reach a mid-career stage before the age of 36.   finally, the 2021 survey comprised 36 questions, three times the number of questions covered in the 2013 survey. wording for seven out of 12 original questions were updated when it was determined that they would result in an improvement to the data. however, this limited the authors’ ability to compare results between the two surveys.   conclusion   with more visible minority librarians participating in the 2021 survey, this larger snapshot provided a more robust and updated perspective of potential changes in the population compared to the 2013 survey. differences were identified with regard to ethnic identity, generation classification, where their mlis or equivalent degree was received, type of library, geographic location, and job responsibilities. the 2021 survey also explored other aspects of these librarians not covered in the previous survey, such as age, disability status, non-mlis education, librarian experience, salary, management positions, and mentorship experience. based on the survey findings, the profession needs to create pathways for vmls to explore leadership positions. mentorship and leadership opportunities offer such librarians a sense of belonging and sense of possibility for their own future. canadian librarianship could design pathways for non-ala accredited librarians with expertise and experience from their home country to secure employment in canada. professional associations and library administrators also need to make continued efforts to support these librarians to create an inclusive space in their libraries.   author contributions   yanli li: conceptualization, data curation, analysis, investigation, methodology, project administration, supervision, visualization, writing – original draft, writing – review & editing maha kumaran:  conceptualization, investigation, qualitative analysis, methodology, writing – original draft, writing – review & editing allan cho: conceptualization, investigation, methodology, writing – original draft, writing – review & editing valentina ly: conceptualization, investigation, methodology, visualization, writing – original draft, writing – review & editing suzanne fernando: conceptualization, investigation, methodology, writing – original draft, writing – review & editing michael david miller: conceptualization, investigation, methodology, translation, writing – original draft, writing – review & editing   references   alliance of credential evaluation services of canada. (n.d.). about the acesc. https://canalliance.org/en/ association of research libraries (arl). (n.d.-a). leadership and career development program. https://www.arl.org/category/our-priorities/diversity-equity-inclusion/leadership-and-career-development-program/   association of research libraries (arl). (n.d.-b). mosaic program. https://www.arl.org/category/our-priorities/diversity-equity-inclusion/arl-saa-mosaic-program/ british columbia institute of technology. (n.d.). international credential evaluation service. https://www.bcit.ca/ices/ burtis, a.t., hubbard, m.a., & lotts, m.c. (2010). foreign lis degrees in contemporary u.s. academic libraries. new library world, 111(9/10), 399-412. https://doi.org/10.1108/03074801011089323 canadian association of professional academic librarians (capal). (2016). 2016 census of canadian academic librarians user guide and results summary. https://capalibrarians.org/wp/wp-content/ uploads/2016/12/census_summary_and_user_guide_december_16_2016.pdf canadian association of professional academic librarians (capal). (2019). 2018 census of canadian academic librarians user guide and results summary. https://capalibrarians.org/wp/wp-content/uploads/2019/03/2018_census_march_24_2019.pdf canadian association of research libraries (carl). (n.d.). diversity and inclusion survey. https://www.carl-abrc.ca/diversity-and-inclusion-survey/ canadian association of research libraries (carl). (2020). strategies and practices for hiring and retaining diverse talent. https://www.carl-abrc.ca/wp-content/uploads/2020/12/carl-strategies-and-practices-for-hiring-and-retaining-diverse-talent-1.pdf canadian association of university teachers (caut). (2019). almanac of post-secondary education 2019 academic staff: table 3.8. https://www.caut.ca/resources/almanac/3-academic-staff canadian journal of academic librarianship. (2019). special focus on diversity. https://cjal.ca/index.php/capal/catalog/category/diversity canadian union of public employees (cupe). (2017, october 24). employment increasingly precarious in public libraries, survey finds. https://cupe.ca/employment-increasingly-precarious-public-libraries-survey-finds ccdi consulting. (2022, may). diversity census and inclusion survey: insights report. https://www.carl-abrc.ca/wp-content/uploads/2022/06/carl-diversity-meter-insights-report-final.pdf damasco, i. t., & hodges, d. (2012). tenure and promotion experiences of academic librarians of color. college & research libraries, 73(3), 279-301. https://doi.org/10.5860/crl-244 delong, k., sorenson, m., & williamson, v. (2015a). 8rs practitioner survey data. https://doi.org/10.7939/dvn/10459 delong, k., sorensen, m., & williamson, v. (2015b). 8rs redux carl libraries human resources study. http://www.carl-abrc.ca/wp-content/uploads/docs/8rs_redux_final_report_oct2015.pdf ferguson, j. (2016). additional degree required? advanced subject knowledge and academic librarianship. portal: libraries and the academy, 16(4), 721-736. http://doi.org/10.1353/pla.2016.0049 ford, a. (2021). the library employment landscape: job seekers navigate uncertain terrain. american libraries, 52(5), 34-37. https://americanlibrariesmagazine.org/2021/05/03/library-employment-landscape/ government of alberta. (2022). international qualifications assessment service. https://www.alberta.ca/international-qualifications-assessment.aspx government of canada. (2021). employment equity act. https://laws-lois.justice.gc.ca/eng/acts/e-5.401/page-1.html#h-215195 guise, j. (2015). succession planning in canadian academic libraries. chandos publishing. harrington, m. r., & marshall, e. (2014). analyses of mentoring expectations, activities, and support in canadian academic libraries. college & research libraries, 75(6), 763. https://doi.org/10.5860/crl.75.6.763 henninger, e., brons, a., riley, c., & yin, c. (2019). perceptions and experiences of precarious employment in canadian libraries: an exploratory study. partnership: the canadian journal of library and information practice and research, 14(2), 1-22. https://doi.org/10.21083/partnership.v14i2.5169 henninger, e., brons, a., riley, c., & yin, c. (2020). factors associated with the prevalence of precarious positions in canadian libraries: statistical analysis of a national job board. evidence based library and information practice, 15(3), 78-102. https://doi.org/10.18438/eblip29783 hines, s. (2019). leadership development for academic librarians: maintaining the status quo? canadian journal of academic librarianship, 4, 1-19. https://doi.org/10.33137/cjal-rcbu.v4.29311 jennings, a., & kinzer, k. (2022). whiteness from top down: systemic change as antiracist action in lis. reference services review, 50(1), 64-80.  https://doi.org/10.1108/rsr-07-2021-0027 joint conference of librarians of color. (2022). jclc 2022. https://www.jclcinc.org/ kandiuk, m. (2014). promoting racial and ethnic diversity among canadian academic librarians. college & research libraries, 75(4), 492–556. https://doi.org/10.5860/crl.75.4.492 kumaran, m. (2012). leadership in libraries: a focus on ethnic-minority librarians. chandos publishing. kumaran, m. (2015). succession planning process that includes visible minority librarians. library management, 36(6/7), 434-447. https://doi.org/10.1108/lm-12-2014-0138 kumaran, m., & cai, h. (2015). identifying the visible minority librarians in canada: a national survey. evidence based library and information practice, 10(2), 108-126. https://doi.org/10.18438/b8zc88 kung, j. y., fraser, k. l., & winn, d. (2020). diversity initiatives to recruit and retain academic librarians: a systematic review. college & research libraries, 81(1), 96-108. https://doi.org/10.5860/crl.81.1.96 law, m. (2001). mentoring programs: in search of the perfect model. feliciter, 47(3), 146-148. lee, y. (2020). bumpy inroads: graduates navigate a precarious job landscape. american libraries, 51(5), 51.  https://americanlibrariesmagazine.org/2020/05/01/bumpy-inroads-library-job-precarity/ li, y. (2021). racial pay gap: an analysis of carl libraries. college & research libraries, 82(3), 436-454. https://doi.org/10.5860/crl.82.3.436 library juice academy. (2022). all courses. https://libraryjuiceacademy.com/all-courses/ mackinnon, c., & shepley, s. (2014). stories of informal mentorship: recognizing the voices of mentees in academic libraries. partnership: the canadian journal of library and information practice and research, 9(1), 1-9. https://doi.org/10.21083/partnership.v9i1.3000 morison, r., erickson, t., & dychtwald, k. (2006, march). managing middlescence. harvard business review, 84(3), 78-86. https://hbr.org/2006/03/managing-middlescence ontario library service. (n.d.). advancing public library leadership institute. https://www.olservice.ca/consulting-training/apll o’reilly, m. (2015, january 29). the impact of precarious work in canada’s libraries. presentation to the ontario library association, toronto, on. oud, j. (2008). adjusting to the workplace: transitions faced by new academic librarians. college & research libraries, 69(3), 252-266. https://doi.org/10.5860/crl.69.3.252 popowich, e. (2011). a forecast for librarians facing retirement. feliciter, 57(6), 220-221. public library interlink. (n.d.). project llead. https://www.interlinklibraries.ca/services/project-llead/ statistics canada. (2017, november 29). labour force status (8), visible minority (15), immigrant status and period of immigration (11), highest certificate, diploma or degree (7), age (13a) and sex (3) for the population aged 15 years and over in private households of canada, provinces and territories, census metropolitan areas and census agglomerations, 2016 census 25% sample data. https://www12.statcan.gc.ca/census-recensement/2016/dp-pd/dt-td/rp-eng.cfm?tabid=2&lang=e&a=r&apath=3&detail=0&dim=0&fl=a&free=0&gc=01&gl=1&gid=1341679&gk=1&grp=1&o=d&pid=110692&prid=10&ptype=109445&s=0&showall=0&sub=0&temporal=2017&theme=124&vid=0&vnamee=&vnamef=&d1=0&d2=0&d3=0&d4=0&d5=0&d6=0 statistics canada. (2018a, march 28). occupation—national occupational classification (noc) 2016 (691), employment income statistics (3), highest certificate, diploma or degree (7), visible minority (15), work activity during the reference year (4), age (4d) and sex (3) for the population aged 15 years and over who worked in 2015 and reported employment income in 2015, in private households of canada, provinces and territories and census metropolitan areas, 2016 census—25% sample data. https://www12.statcan.gc.ca/global/urlredirect.cfm?lang=e&ips=98-400-x2016356 statistics canada. (2018b, may 31). census indicator profile, based on the 2016 census long-form questionnaire, canada, provinces and territories, and health regions (2017 boundaries). https://doi.org/10.25318/1710012301-eng statistics canada. (2021, september 30). classification of generation status. https://www23.statcan.gc.ca/imdb/p3vd.pl?function=getvd&tvd=117200&cvd=117200&clv=0&mlv=1&d=1 sullivan, d. (2011). work envy, workhorses and the mid-career librarian. in d. lowe-wincentsen & l. crook (eds.), mid-career library and information professionals: a leadership primer (pp. 113-135). chandos publishing. taleban, s. (2016). the journey of a non-ala-accredited librarian in canada. british columbia library association. https://bclaconnect.ca/perspectives/2016/05/01/an-iranian-librarian-in-canada-on-attaining-an-ala-accredited-mlis/ the canadian information centre for international credentials. (n.d.). assessor portal. https://www.cicic.ca/853/assessor.canada.canada the canadian press. (2016, march 27). librarians fight rise of precarious work. cbc news. https://www.cbc.ca/news/canada/toronto/precarious-work-librarians-1.3508778 tucker, j. c. (2008). development of midcareer librarians. journal of business & finance librarianship, 13(3), 241-248. https://doi.org/10.1080/08963560802183146 university of victoria. (2015). employment equity plan 2015 – 2020. https://www.uvic.ca/equity/assets/docs/eep2015.pdf visible minority librarians of canada (vimloc). (n.d.). vimloc operating values. https://vimloc.wordpress.com/ visible minority librarians of canada (vimloc). (2021). a utl & vimloc workshop series: navigating the field: finding that first academic librarian position in canada. https://vimloc.wordpress.com/workshop-events-series/   appendix survey questionnaire section one: demographic information  1. the canadian employment equity act defines visible minorities as “persons, other than aboriginal peoples, who are non-caucasian in race or non-white in colour.” the visible minority population consists mainly of the following groups: chinese, south asian, black, arab, west asian, filipino, southeast asian, latin american, japanese and korean. are you a visible minority librarian currently working in canada?     o yes o no 2. what group do you belong to or which group fits you the best? o arab only (includes egyptian, kuwaiti and libyan) o black only o chinese only o  filipino only o  japanese only o  korean only o  latin american only o south asian only (includes bangladeshi, indian, pakistani, and sri lankan) o  southeast asian only (includes vietnamese, cambodian, malaysian, and laotian) o  west asian only (includes afghan, assyrian, and iranian) o white and arab o white and black o white and chinese o white and filipino o white and japanese o white and korean o white and latin american o white and south asian o white and southeast asian o white and west asian o white and multiple visible minorities o multiple visible minorities o other (please specify) _________________ 3. tell us if you are a first generation minority librarian or not. first generation would mean that you were born elsewhere but moved to canada at some point in your life. second generation would mean you were born in canada to immigrant parents. if you would like to add an explanation about this, please use the text box below, such as your age or the year when you came to canada.       o first generation ___________________ o second generation ________________ o other ___________________________ 4. do you consider yourself to have a disability? o yes (please elaborate if you wish) ___________________ o no 5. what is your age? o 20-25 o 26-30 o 31-35 o 36-40 o 41-45 o 46-50 o 51-55 o 56-60 o 61-65 o 65+  6. what is your gender identity? o female o male o transgender o two spirit o other (please elaborate if you wish) ________________ o prefer not to answer section two: education 7. when did you receive your mlis / mls degree or equivalent? o before or during 1980 o between 1981 and 1989 o between 1990 and 1999 o between 2000 and 2009 o between 2010 and 2019 o after 2019 8. where did you receive your mlis / mls degree or equivalent? answer q9 if option one is selected, otherwise skip to q10-15. o from an ala-accredited canadian library school o from an ala-accredited american library school o from a library school outside north america o other (please specify) _______________  9. please select the university that you received your degree.  o university of british columbia o university of alberta o university of western ontario / western university o university of toronto o university of ottawa o université de montréal o dalhousie university o mcgill university 10. please specify the country where you received your library degree:____________ 11. please provide the name of your institution:_____________ 12. does your current employer recognize your professional library degree in terms of your position?        o yes o no 13. have you taken any courses of study or programs in canada to supplement your library degree? o yes o no 14. please provide the name of the course or program:______________ 15. how, if at all, has this made a difference to how your employer and the library community recognize your credentials?__________________ 16. in addition to your mlis / mls degree or equivalent, please indicate other education you attained. select all that apply. o professional degree (what degree? e.g. law)__________________ o second master’s degree (what discipline?) ________________ o third master’s degree (what discipline?)________________ o phd (what discipline?) ___________________ o additional degrees, certificates, or diplomas (what type?) ________________ o none of the above  section three: employment 17. how many total years have you worked as a librarian? o 0-5 o 6-10 o 11-15 o 16-20 o 21-25 o 25+ 18. what inspired you to enter the library profession? select all that apply. ▢ i was inspired by a family member or friend that worked in the profession ▢ i got an entry level job in a library ▢ library role models influenced me ▢ i thought it would be an interesting profession ▢ i thought it would be a well paying job ▢ i thought it would be a rewarding job because i would have the opportunity to help others ▢ i liked the work environment in a library ▢ i had the expertise and skills fit for the library job ▢ i enjoyed books and reading ▢ other (please elaborate) ___________________  19. which province / territory do you currently work in? o alberta o british columbia o manitoba o new brunswick o newfoundland and labrador o northwest territories o nova scotia o nunavut o ontario o prince edward island o quebec o saskatchewan o yukon o other (if you are working for a canadian library outside of canada) _________ 20. what type of library are you currently working at? o public library o regional library o academic library o college library o special library (what type? e.g., government, religious organization) _______ o school library  o other (please specify) __________ 21. please select the job category(ies) that matches your current job responsibilities. select all that apply. o  acquisitions / collection development o administration o adult services o  archives o  assessment o  automation / systems / it services o  bibliometrics o  cataloguing / metadata management o  children’s services o  circulation o  consultant / knowledge management / researchers o  copyright o data management and curation o digitization and preservation o  e-resources and serials o  government documents o  instruction services o  interlibrary loan services o  liaison librarian o  licensing o marketing / outreach / community services o  media specialist o  project management o  public services o  rare books and special collections o  reference / information services o  research services o  school / teacher librarian o  scholarly communications o  user experience o  web services o  youth services o  other (please specify) _______  _________  22. do you work part-time (less than 30 hours/week), full-time (30 or more hours/week), or casual hours? o part-time o full-time o casual hours o other _____________________ 23. approximately how many hours are you expected to work per week? o 20 hours or less o 21-25 o 26-30 o 31-35 o 36-40 o 40+ o no hours specified (until the job is done) 24. what type of appointment do you have? o permanent o temporary (e.g., contract, limited-term) o other (please specify) ________________  25. what is your gross (before taxes) yearly salary from your employing library? o $10,000 or less o $10,001 – 20,000 o $20,001 –30,000 o $30,001-40,000 o $40,001-50,000 o $50,001-60,000 o $60,001-70,000 o $70,001-80,000 o $80,001-90,000 o $90,001-100,000 o $100,001-110,000 o $110,001-120,000 o $120,001-130,000 o $130,001-140,000 o $140,001-150,000 o $150,001+ o prefer not to answer 26. please describe your experience with respect to the following statements:    (1) i am treated with respect and accepted as an equal member by colleagues in my department. o strongly agree o agree o neither agree or disagree o disagree o strongly disagree (2) my knowledge and work contributions are valued by colleagues in my department. o strongly agree o agree o neither agree or disagree o disagree o strongly disagree  (3) i feel free to speak my mind and express my views openly amongst colleagues in my department.       o strongly agree o agree o neither agree or disagree o disagree o strongly disagree 27. racial microaggressions are subtle hostile, derogatory, or negative slights and insults (verbal, nonverbal, and/or visual) directed toward people of colour, whether intentional or unintentional. please rate how frequently you have experienced each of the following forms of racial microaggressions throughout your career (never, rarely, sometimes, often, always) (1) i was told that people should not think about race anymore. (2) i was told that people of all racial groups face the same barriers in employment or promotion. (3) i was told that people of colour do not experience racism anymore. (4) i was told that i was overly sensitive about issues of race. (5) i was told that all people in my racial group are all the same. (6) my opinion was ignored in a group discussion because of my race. (7) a colleague assumed that i would have a lower english proficiency because of my race. (8) a colleague claimed that he/she felt threatened because of my race. (9) a colleague showed surprise at my professional success because of my race. (10) i was told that i was hired because of my race. 28. other forms of racial microaggressions you have experienced throughout your career: ________________________________________________________________ 29. overall, how satisfied are you with your current job?  o very satisfied o satisfied o neither satisfied nor dissatisfied o dissatisfied o very dissatisfied 30. what level is your current position? o non-management o supervisor o middle management (e.g., branch head, department head) o senior administrator (e.g., head / chief librarian, director, or deputy / assistant head, chief, director) 31. please indicate the extent to which race or ethnicity is a barrier to your library career aspirations? o not at all o to a small extent o to some extent o to a moderate extent o to a great extent o to a very great extent  32. have you ever sought support from a mentor? if “no” is selected, skip to q36. o yes o no 33. have you participated in any formal mentorship program(s)? o yes  (what program?) ______________________ o no 34. have you ever had a mentor who is a member of visible minority group? o yes o no o don’t know 35. how helpful do you think the mentors have been in your career development? o extremely helpful o very helpful o moderately helpful o slightly helpful o not at all helpful  36. please use the box below to add anything else that we may have missed asking you or that you would like to share: ______________________________________________       evidence summary   older adults’ internet use is varied, suggesting the need for targeted rather than broadly focused outreach   a review of: van boekel, l.c., peek, s. t., & luijkx, k.g. (2017). diversity in older adults’ use of the internet: identifying subgroups through latent class analysis. journal of medical internet research, 19(5:e180), 1-10. doi: 10.2196/jmir.6853   reviewed by: ann glusker research & data coordinator national network of libraries of medicine, pacific northwest region university of washington health sciences library seattle, washington, united states of america email: glusker@uw.edu   received: 28 feb. 2018   accepted: 24 apr. 2018      2018 glusker. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29420       abstract   objective – to determine the amount and types of variation in internet use among older adults, and to test its relationship to social and health factors.   design – representative longitudinal survey panel of households   setting – the netherlands   subjects – a panel with 1,418 members who were over 65 years of age had answered the survey questionnaire that included internet use questions, and who reported access to and use of the internet.   methods – using information about the internet activities the respondents reported, the authors conducted latent class analysis and extracted a best-fitting model including four clusters of respondent internet use types.  the four groups were analyzed using descriptive statistics and compared using anova and chi-square tests.  analysis and comparisons were conducted both between groups, and on the relationship of the groups with a range of social and health variables.   main results – the four clusters identified included: 1) practical users using the internet for practical purposes such as financial transactions; 2) social users using the internet for activities such as social media and gaming; 3) minimizers, who spent the least time on the internet and were the oldest group; and 4) maximizers, who used the internet for the widest range of purposes, for the most time, and who were the youngest group.  once the clusters were delineated, social and health factors were examined (specifically social and emotional loneliness, psychological well-being, and two activities of daily living (adl) measures).  there were significant differences between groups, but the effect sizes were small.  practical users had higher psychological well-being, whereas minimizers had the lowest scores related to adls and overall health (however, they were also the oldest group).   conclusions – the establishment of four clusters of internet use types demonstrates that older adults are not homogeneous in their internet practices.  however, there were no marked findings showing differences between the clusters in social and health-related variables (the minimizers reported lower health status, but they were also the oldest group).  nevertheless, the finding of internet use heterogeneity is an important one for those who wish to connect with older adults through internet-based programming.  the different patterns evidenced in each cluster will require differing outreach strategies. it also highlights the need for ongoing longitudinal research, to determine whether those who are currently younger and more technologically savvy will age into similar patterns that these authors found, or whether a new set of older adult internet use profiles will emerge as younger generations with more internet experience and affinity become older.   commentary – for this evidence summary, methodologies were systematically assessed using glynn’s critical appraisal checklist (2006).  the authors used secondary data from a large randomized sample, collected in a rigorous manner.  along with their appropriate use of methodologies and proportionate statements of findings relative to effect sizes, there are few concerns about this study’s data quality.   nevertheless, there are limitations of note, several of which (mostly technical) were mentioned by the authors.  among those they did not mention was the question of whether the researchers only including respondents who had internet access and who also had completed the “social integration and leisure” questionnaire may have introduced bias. also, while they note a survey drawn from dutch citizens is “considered to be comparable to other western populations in terms of internet use”, they cite information that internet use in the united states is 14 percentage points lower than that in the netherlands, and some internet activities among older adults are higher in the netherlands than in the rest of europe, leaving some question of the representativeness of the population studied.  there is also no mention of the potentially lower percentages of internet use in households with low income and disabled older adults (choi & dinitto, 2013).   most importantly, however, the authors note that there was no information available about the supports available to the respondents for using technology in general and the internet in particular.  information about whether older adults were living alone (which relates to lower percentage uptake of the internet (van deursen & helsper, 2015)), whether they had ever had jobs requiring internet use (chang et al., 2014; hargittai & dobransky, 2017), and what their cognitive status was (freese et al., 2006), could be very illuminating to their results.   at the same time, the potential for future studies on the topic of heterogeneity of older adult internet use is vast and fascinating, since the topic is such an important one for those who wish to engage older adults in order to promote programs and activities such as those related to ehealth.  the authors call for a longitudinally-focused replication of their study, which would demonstrate whether currently younger adults will age into a similar profile to that now seen, or into a more internet-intensive use profile, given their deeper internet experience.  additionally the authors suggest research on how older adults overcome physical and mental (specifically, cognitive) barriers to internet use, whether there are any direct associations between declining health and internet use, and how older adults expand and contract their choices of activities in general.  hargittai and dobransky (2017) also suggest that research include the internet use skill levels of older adults: “understanding how the internet and online services work is something that can be taught and is thus open to intervention, [so] it is an important factor to examine in work on digital inequality”(p. 208).   finally, the phenomenon of ehealth is referred to repeatedly in this article, and it would help the reader to have a clearer definition of how that is experienced in the dutch context.   references   chang, j., mcallister,c., & mccaslin, r. (2015). correlates of, and barriers to, internet use among older adults. journal of gerontological social work, 58(1), 66-85. doi:10.1080/01634372.2014.913754   choi, n.g. & dinitto, d.m. (2013). the digital divide among low-income homebound older adults: internet use patterns, ehealth literacy, and attitudes toward computer/internet use. journal of medical internet research, 15(5:e93), 1-25. doi:  10.2196/jmir.2645   freese, j., rivas, s., & hargittai, e. (2006). cognitive ability and internet use among older adults. poetics, 34(4-5), 236-249. doi: 10.1016/j.poetic.2006.05.008   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399.   hargittai, e. & dobransky, k. (2017). old dogs, new clicks: digital inequality in skills and uses among older adults. canadian journal of communication, 42(2), 195-212. doi: 10.22230/cjc.2017v42n2a3176   van deursen, a.j. & helsper, e.j. (2015). a nuanced understanding of internet use and non-use among the elderly. european journal of communication, 30 (2), 171-187. doi: 10.1177/0267323115578059   evidence based library and information practice evidence based library and information practice 2009, 4.4     evidence based library and information practice         evidence summary    rejuvenation and networking motivates librarians to attend conferences    a review of:  vega, robert d. and ruth s. connell. “librarians’ attitudes toward conferences: a study.” college  & research libraries 68.6 (2007):  503‐15.     reviewed by:   yvonne hultman özek  senior librarian, lund university, faculty of medicine   lund, sweden  email: yvonne.hultman_ozek@med.lu.se     received: 23 may 2009          accepted: 19 october 2009      © 2009 özek. this is an open access article distributed under the terms of the creative commons attribution   license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and   reproduction in any medium, provided the original work is properly cited.      objectives – to determine why professional  librarians attend or do not attend conferences  and to consider their attitudes toward the  various conference offerings and whether  conference attendance is important to their  careers.     design – an electronic mail list survey  designed to collect both quantitative and  qualitative data through closed and open‐ ended questions.     setting – mainly academic and public libraries  in the u.s.a.    subjects – a total of 794 librarians  participated in the study.    methods – the quantitative and qualitative  data gathered from the survey were analyzed  using statistical software. the quantitative  items on the questionnaire regarded the  following: years working as a librarian, age,  gender, average number of conferences  attended each year, amount of financial  support provided, and the importance of  attending conferences. for qualitative analysis,  the instrument included open‐ended questions  to gather comments about issues related to  conferences. electronic mail list subscriptions  were used to reach approximately 17,000  potential respondents of whom 794 completed  the survey.     main results – the study found the main  reason the respondents attended conferences  was to achieve professional rejuvenation  (56%), and that this finding was associated  with age: the older the librarian, the higher he  or she would rate the need to stay updated in  librarianship. the study also found that men  as compared to women ranked rejuvenation as  less important. respondents who visited more  conferences tended to rate rejuvenation as  more important. networking was also highly  78 mailto:yvonne.hultman_ozek@med.lu.se evidence based library and information practice 2009, 4.4  valued (40%), although was not of significant  importance among reference librarians.     exhibits were considered to be very useful  among older respondents, particularly older  public librarians and older administrators  from all types of libraries, but were regarded  as less valuable among younger academic  librarians. . invited speakers were rated highly  by academic librarians, but less highly by  public librarians. general sessions primarily  attracted newer and younger librarians, and  poster sessions primarily appealed to  academic librarians. being able to add  conference participation to the curriculum  vitae was primarily of interest to younger  academic librarians. academic librarians rated  roundtables as important.     conclusion – respondents primarily attended  conferences for professional rejuvenation and  networking, though felt these benefits were  not necessarily related to conference content.  those who had worked longer as librarians  valued exhibits more than those who were  new in the field. based on the results, further  investigation is needed to answer the  following questions:     1) why do public librarians find poster  sessions of low interest?  2) why do men compared to women  find professional rejuvenation less important?  3) have more experienced librarians  learned to navigate exhibits better?    the findings of this study would be useful to  conference organizers.      commentary     as pointed out in the article, investigators in  previous studies have often allowed their own  personal experiences to intervene in their  evaluations. by performing a survey in the  library community, the present authors have  offered a broader perspective on this topic.  nevertheless, some comments are needed  with regard to certain aspects of the  methodology that was used.  the authors state that, in this survey, the total  potential audience on electronic discussion  lists consisted of 17,000 librarians, but that  may be an overstatement due to the likely  overlap of names on the targeted lists. in all  794 people took part in the study, mainly  academic and public librarians. however, the  reader is faced with inconsistent terminology  in the article text and the questionnaire. 16% of  the respondents identified themselves as being  primarily library instructors, a category not  included in the questionnaire, so must have  selected the category “other” instead. library  instruction is important and well established  in the library profession, and hence it should  have been included as a separate category on  the survey instrument. without explanation  that it is not included in the questionnaire,  “instruction” is listed as a category in table  1in the article (508).  furthermore, the authors  reported that the electronic mail lists they  used were selected to reach a large and  diverse audience. the results regarding  diversity of the respondents could have been  strengthened by including a question about  “academic degree” in the survey. the authors  also mentioned that the electronic mail lists  were targeted to capture a variety of positions  and settings (505), but a query about  “position” was not included in the survey  instrument. it would have been interesting if  the authors had gathered information on how  many respondents were in a managerial  position and then correlated the results with  the findings in table 2 about preferred  activities when attending conferences (509).    professional rejuvenation received the highest  rating for why respondents attended  conferences, but it can be difficult to draw  conclusions about that finding due to  significant overlap between the  conferences/activities chosen and the reasons  for attending. did the respondents acquire the  most rejuvenation from general sessions or  from any of the other categories? also, as  noted by the authors, “cv padding” has a  negative connotation that might have affected  the responses.    79 evidence based library and information practice 2009, 4.4  80 the authors further maintained that a free‐ form response is needed for the question  concerning the amount of financial support  provided for attending conferences. however,  they did not present any conclusions or  correlations regarding the impact of full or  partial funding on conference participation.    this research contributes to general insight  into what motivates librarians in different  settings to attend conferences. the authors  rightly acknowledge that further study is  needed, which might be directed toward new  research avenues aimed at determining why  public librarians consider poster sessions to be  of little interest, why men as compared to  women rank professional rejuvenation as less  important, and whether more experienced  librarians have learned to navigate exhibits  better.     another aspect that should be considered, as  stated in part in the conclusion section of the  article, is that such research would contribute  information that can be useful in the areas of  psychology and sociology, as well as in the  practical implementation handled by  conference organizers. conference organizers  would especially benefit from research results  if the survey instrument is changed to include  a question about whether respondents would  be willing to evaluate future conferences. in  addition, although not mentioned in the  article, it might be of interest to design an  investigation so the results could aid library  managers in decision‐making processes  related to professional staff development.     <html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/tr/rec-html40"> <head> <meta http-equiv=content-type content="text/html; charset=unicode"> <meta name=progid content=word.document> <meta name=generator content="microsoft word 15"> <meta name=originator content="microsoft word 15"> <link rel=file-list href="edresp_29566_files/filelist.xml"> <link rel=edit-time-data href="edresp_29566_files/editdata.mso"> <!--[if !mso]> <style> v\:* {behavior:url(#default#vml);} o\:* {behavior:url(#default#vml);} w\:* {behavior:url(#default#vml);} .shape {behavior:url(#default#vml);} </style> <![endif]--> <title>evidence based library and information practice</title> <!--[if gte mso 9]><xml> <o:documentproperties> <o:author>mallan, katrine</o:author> <o:template>normal</o:template> <o:lastauthor>hinrichs, rachel</o:lastauthor> <o:revision>3</o:revision> <o:totaltime>2</o:totaltime> <o:lastprinted>2012-06-12t11:55:00z</o:lastprinted> <o:created>2019-03-14t14:39:00z</o:created> <o:lastsaved>2019-03-14t14:42:00z</o:lastsaved> <o:pages>1</o:pages> <o:words>196</o:words> <o:characters>1123</o:characters> <o:company>microsoft</o:company> <o:lines>9</o:lines> <o:paragraphs>2</o:paragraphs> <o:characterswithspaces>1317</o:characterswithspaces> <o:version>16.00</o:version> </o:documentproperties> <o:officedocumentsettings> <o:allowpng/> </o:officedocumentsettings> </xml><![endif]--> <link rel=themedata href="edresp_29566_files/themedata.thmx"> <link rel=colorschememapping href="edresp_29566_files/colorschememapping.xml"> <!--[if gte mso 9]><xml> <w:worddocument> <w:trackmoves>false</w:trackmoves> <w:trackformatting/> <w:hyphenationzone>21</w:hyphenationzone> <w:donothyphenatecaps/> <w:punctuationkerning/> <w:validateagainstschemas>false</w:validateagainstschemas> <w:saveifxmlinvalid>false</w:saveifxmlinvalid> <w:ignoremixedcontent>false</w:ignoremixedcontent> <w:alwaysshowplaceholdertext>false</w:alwaysshowplaceholdertext> <w:donotunderlineinvalidxml/> <w:donotpromoteqf/> <w:lidthemeother>en-gb</w:lidthemeother> <w:lidthemeasian>x-none</w:lidthemeasian> <w:lidthemecomplexscript>x-none</w:lidthemecomplexscript> <w:compatibility> <w:breakwrappedtables/> <w:snaptogridincell/> <w:wraptextwithpunct/> <w:useasianbreakrules/> <w:useword2010tablestylerules/> <w:dontgrowautofit/> <w:splitpgbreakandparamark/> <w:enableopentypekerning/> <w:dontflipmirrorindents/> <w:overridetablestylehps/> </w:compatibility> <w:browserlevel>microsoftinternetexplorer4</w:browserlevel> <m:mathpr> <m:mathfont m:val="cambria math"/> <m:brkbin m:val="before"/> <m:brkbinsub m:val="&#45;-"/> <m:smallfrac m:val="off"/> <m:dispdef/> <m:lmargin m:val="0"/> <m:rmargin m:val="0"/> <m:defjc m:val="centergroup"/> <m:wrapindent m:val="1440"/> <m:intlim m:val="subsup"/> <m:narylim m:val="undovr"/> </m:mathpr></w:worddocument> </xml><![endif]--><!--[if gte mso 9]><xml> <w:latentstyles deflockedstate="false" defunhidewhenused="false" defsemihidden="false" defqformat="false" defpriority="99" latentstylecount="371"> <w:lsdexception locked="true" priority="0" qformat="true" name="normal"/> <w:lsdexception locked="true" priority="0" qformat="true" name="heading 1"/> <w:lsdexception locked="true" priority="0" semihidden="true" unhidewhenused="true" qformat="true" name="heading 2"/> <w:lsdexception locked="true" priority="0" semihidden="true" unhidewhenused="true" qformat="true" name="heading 3"/> <w:lsdexception locked="true" priority="0" semihidden="true" unhidewhenused="true" qformat="true" name="heading 4"/> <w:lsdexception locked="true" priority="0" semihidden="true" unhidewhenused="true" qformat="true" name="heading 5"/> <w:lsdexception locked="true" priority="0" semihidden="true" unhidewhenused="true" qformat="true" name="heading 6"/> <w:lsdexception locked="true" priority="0" semihidden="true" unhidewhenused="true" qformat="true" name="heading 7"/> <w:lsdexception locked="true" priority="0" semihidden="true" unhidewhenused="true" qformat="true" name="heading 8"/> <w:lsdexception locked="true" priority="0" semihidden="true" unhidewhenused="true" qformat="true" name="heading 9"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 6"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 7"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 8"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 9"/> <w:lsdexception locked="true" priority="0" name="toc 1"/> <w:lsdexception locked="true" priority="0" name="toc 2"/> <w:lsdexception locked="true" priority="0" name="toc 3"/> <w:lsdexception locked="true" priority="0" name="toc 4"/> <w:lsdexception locked="true" priority="0" name="toc 5"/> <w:lsdexception locked="true" priority="0" name="toc 6"/> <w:lsdexception locked="true" priority="0" name="toc 7"/> <w:lsdexception locked="true" priority="0" name="toc 8"/> <w:lsdexception locked="true" priority="0" name="toc 9"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="normal indent"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="footnote text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="annotation text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="header"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="footer"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index heading"/> <w:lsdexception locked="true" priority="0" semihidden="true" unhidewhenused="true" qformat="true" name="caption"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table of figures"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="envelope address"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="envelope return"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="footnote reference"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="annotation reference"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="line number"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="page number"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="endnote reference"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="endnote text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table of authorities"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="macro"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="toa heading"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number 5"/> <w:lsdexception locked="true" priority="0" qformat="true" name="title"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="closing"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="signature"/> <w:lsdexception locked="true" priority="1" name="default paragraph font"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text indent"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="message header"/> <w:lsdexception locked="true" priority="0" qformat="true" name="subtitle"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="salutation"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="date"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text first indent"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text first indent 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="note heading"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text indent 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text indent 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="block text"/> <w:lsdexception locked="true" name="hyperlink"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="followedhyperlink"/> <w:lsdexception locked="true" priority="0" qformat="true" name="strong"/> <w:lsdexception locked="true" priority="0" qformat="true" name="emphasis"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="document map"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="plain text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="e-mail signature"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html top of form"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html bottom of form"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="normal (web)"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html acronym"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html address"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html cite"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html code"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html definition"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html keyboard"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html preformatted"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html sample"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html typewriter"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html variable"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="normal table"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="annotation subject"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="no list"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="outline list 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="outline list 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="outline list 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table simple 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table simple 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table simple 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table classic 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table classic 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table classic 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table classic 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table colorful 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table colorful 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table colorful 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 6"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 7"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 8"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 6"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 7"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 8"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table 3d effects 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table 3d effects 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table 3d effects 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table contemporary"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table elegant"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table professional"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table subtle 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table subtle 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table web 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table web 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table web 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="balloon text"/> <w:lsdexception locked="true" priority="0" name="table grid"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table theme"/> <w:lsdexception locked="false" semihidden="true" name="placeholder text"/> <w:lsdexception locked="false" priority="1" qformat="true" name="no spacing"/> <w:lsdexception locked="false" priority="60" name="light shading"/> <w:lsdexception locked="false" priority="61" name="light list"/> <w:lsdexception locked="false" priority="62" name="light grid"/> <w:lsdexception locked="false" priority="63" name="medium shading 1"/> <w:lsdexception locked="false" priority="64" name="medium shading 2"/> <w:lsdexception locked="false" priority="65" name="medium list 1"/> <w:lsdexception locked="false" priority="66" name="medium list 2"/> <w:lsdexception locked="false" priority="67" name="medium grid 1"/> <w:lsdexception locked="false" priority="68" name="medium grid 2"/> <w:lsdexception locked="false" priority="69" name="medium grid 3"/> <w:lsdexception locked="false" priority="70" name="dark list"/> <w:lsdexception locked="false" priority="71" name="colorful shading"/> <w:lsdexception locked="false" priority="72" name="colorful list"/> <w:lsdexception locked="false" priority="73" name="colorful grid"/> <w:lsdexception locked="false" priority="60" name="light shading accent 1"/> <w:lsdexception locked="false" priority="61" name="light list accent 1"/> <w:lsdexception locked="false" priority="62" name="light grid accent 1"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 1"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 1"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 1"/> <w:lsdexception locked="false" semihidden="true" name="revision"/> <w:lsdexception locked="false" priority="34" qformat="true" name="list paragraph"/> <w:lsdexception locked="false" priority="29" qformat="true" name="quote"/> <w:lsdexception locked="false" priority="30" qformat="true" name="intense quote"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 1"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 1"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 1"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 1"/> <w:lsdexception locked="false" priority="70" name="dark list accent 1"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 1"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 1"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 1"/> <w:lsdexception locked="false" priority="60" name="light shading accent 2"/> <w:lsdexception locked="false" priority="61" name="light list accent 2"/> <w:lsdexception locked="false" priority="62" name="light grid accent 2"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 2"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 2"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 2"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 2"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 2"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 2"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 2"/> <w:lsdexception locked="false" priority="70" name="dark list accent 2"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 2"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 2"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 2"/> <w:lsdexception locked="false" priority="60" name="light shading accent 3"/> <w:lsdexception locked="false" priority="61" name="light list accent 3"/> <w:lsdexception locked="false" priority="62" name="light grid accent 3"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 3"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 3"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 3"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 3"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 3"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 3"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 3"/> <w:lsdexception locked="false" priority="70" name="dark list accent 3"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 3"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 3"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 3"/> <w:lsdexception locked="false" priority="60" name="light shading accent 4"/> <w:lsdexception locked="false" priority="61" name="light list accent 4"/> <w:lsdexception locked="false" priority="62" name="light grid accent 4"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 4"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 4"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 4"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 4"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 4"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 4"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 4"/> <w:lsdexception locked="false" priority="70" name="dark list accent 4"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 4"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 4"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 4"/> <w:lsdexception locked="false" priority="60" name="light shading accent 5"/> <w:lsdexception locked="false" priority="61" name="light list accent 5"/> <w:lsdexception locked="false" priority="62" name="light grid accent 5"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 5"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 5"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 5"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 5"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 5"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 5"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 5"/> <w:lsdexception locked="false" priority="70" name="dark list accent 5"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 5"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 5"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 5"/> <w:lsdexception locked="false" priority="60" name="light shading accent 6"/> <w:lsdexception locked="false" priority="61" name="light list accent 6"/> <w:lsdexception locked="false" priority="62" name="light grid accent 6"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 6"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 6"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 6"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 6"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 6"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 6"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 6"/> <w:lsdexception locked="false" priority="70" name="dark list accent 6"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 6"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 6"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 6"/> <w:lsdexception locked="false" priority="19" qformat="true" name="subtle emphasis"/> <w:lsdexception locked="false" priority="21" qformat="true" name="intense emphasis"/> <w:lsdexception locked="false" priority="31" qformat="true" name="subtle reference"/> <w:lsdexception locked="false" priority="32" qformat="true" name="intense reference"/> <w:lsdexception locked="false" priority="33" qformat="true" name="book title"/> <w:lsdexception locked="false" priority="37" semihidden="true" unhidewhenused="true" name="bibliography"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" qformat="true" name="toc heading"/> <w:lsdexception locked="false" priority="41" name="plain table 1"/> <w:lsdexception locked="false" priority="42" name="plain table 2"/> <w:lsdexception locked="false" priority="43" name="plain table 3"/> <w:lsdexception locked="false" priority="44" name="plain table 4"/> <w:lsdexception locked="false" priority="45" name="plain table 5"/> <w:lsdexception locked="false" priority="40" name="grid table light"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light"/> <w:lsdexception locked="false" priority="47" name="grid table 2"/> <w:lsdexception locked="false" priority="48" name="grid table 3"/> <w:lsdexception locked="false" priority="49" name="grid table 4"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 1"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 1"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 1"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 1"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 1"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 1"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 1"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 2"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 2"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 2"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 2"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 2"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 2"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 2"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 3"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 3"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 3"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 3"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 3"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 3"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 3"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 4"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 4"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 4"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 4"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 4"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 4"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 4"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 5"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 5"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 5"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 5"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 5"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 5"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 5"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 6"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 6"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 6"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 6"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 6"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 6"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 6"/> <w:lsdexception locked="false" priority="46" name="list table 1 light"/> <w:lsdexception locked="false" priority="47" name="list table 2"/> <w:lsdexception locked="false" priority="48" name="list table 3"/> <w:lsdexception locked="false" priority="49" name="list table 4"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 1"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 1"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 1"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 1"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 1"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 1"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 1"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 2"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 2"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 2"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 2"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 2"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 2"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 2"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 3"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 3"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 3"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 3"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 3"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 3"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 3"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 4"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 4"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 4"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 4"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 4"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 4"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 4"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 5"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 5"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 5"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 5"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 5"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 5"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 5"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 6"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 6"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 6"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 6"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 6"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 6"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 6"/> </w:latentstyles> </xml><![endif]--> <style> <!- /* font definitions */ @font-face {font-family:"ms mincho"; panose-1:2 2 6 9 4 2 5 8 3 4; mso-font-alt:"ｍｓ 明朝"; mso-font-charset:128; mso-generic-font-family:roman; mso-font-pitch:fixed; mso-font-signature:1 134676480 16 0 131072 0;} @font-face {font-family:"cambria math"; panose-1:2 4 5 3 5 4 6 3 2 4; mso-font-charset:1; mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:0 0 0 0 0 0;} @font-face {font-family:calibri; panose-1:2 15 5 2 2 2 4 3 2 4; mso-font-charset:0; mso-generic-font-family:swiss; mso-font-pitch:variable; mso-font-signature:-536859905 -1073732485 9 0 511 0;} @font-face {font-family:tahoma; panose-1:2 11 6 4 3 5 4 4 2 4; mso-font-charset:0; mso-generic-font-family:swiss; mso-font-pitch:variable; mso-font-signature:-520081665 -1073717157 41 0 66047 0;} @font-face {font-family:"palatino linotype"; panose-1:2 4 5 2 5 5 5 3 3 4; mso-font-charset:0; mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:-536870265 1073741843 0 0 415 0;} @font-face {font-family:"\@ms mincho"; panose-1:2 2 6 9 4 2 5 8 3 4; mso-font-charset:128; mso-generic-font-family:modern; mso-font-pitch:fixed; mso-font-signature:-536870145 1791491579 18 0 131231 0;} /* style definitions */ p.msonormal, li.msonormal, div.msonormal {mso-style-unhide:no; mso-style-qformat:yes; mso-style-parent:""; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca;} p.msoheader, li.msoheader, div.msoheader {mso-style-priority:99; mso-style-link:"header char"; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; tab-stops:center 3.15in right 6.3in; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca;} p.msofooter, li.msofooter, div.msofooter {mso-style-priority:99; mso-style-link:"footer char"; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; tab-stops:center 3.15in right 6.3in; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca;} p.msobodytext, li.msobodytext, div.msobodytext {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"body text char"; margin:0in; margin-bottom:.0001pt; mso-pagination:none; mso-layout-grid-align:none; text-autospace:none; font-size:12.0pt; font-family:"arial",sans-serif; mso-fareast-font-family:calibri;} a:link, span.msohyperlink {mso-style-priority:99; mso-style-unhide:no; mso-style-parent:""; color:blue; text-decoration:underline; text-underline:single;} a:visited, span.msohyperlinkfollowed {mso-style-noshow:yes; mso-style-priority:99; color:purple; mso-themecolor:followedhyperlink; text-decoration:underline; text-underline:single;} p.msoacetate, li.msoacetate, div.msoacetate {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"balloon text char"; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:8.0pt; font-family:"tahoma",sans-serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca;} p.msonormal0, li.msonormal0, div.msonormal0 {mso-style-name:msonormal; mso-style-unhide:no; mso-margin-top-alt:auto; margin-right:0in; mso-margin-bottom-alt:auto; margin-left:0in; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast;} span.headerchar {mso-style-name:"header char"; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-parent:""; mso-style-link:header; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; mso-ansi-language:en-ca;} span.footerchar {mso-style-name:"footer char"; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-parent:""; mso-style-link:footer; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; mso-ansi-language:en-ca;} span.bodytextchar {mso-style-name:"body text char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-parent:""; mso-style-link:"body text"; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"arial",sans-serif; mso-ascii-font-family:arial; mso-hansi-font-family:arial; mso-bidi-font-family:arial; mso-ansi-language:en-us;} span.balloontextchar {mso-style-name:"balloon text char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-parent:""; mso-style-link:"balloon text"; mso-ansi-font-size:8.0pt; mso-bidi-font-size:8.0pt; font-family:"tahoma",sans-serif; mso-ascii-font-family:tahoma; mso-hansi-font-family:tahoma; mso-bidi-font-family:tahoma; mso-ansi-language:en-ca;} p.default, li.default, div.default {mso-style-name:default; mso-style-priority:99; mso-style-unhide:no; mso-style-parent:""; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; mso-layout-grid-align:none; text-autospace:none; font-size:12.0pt; font-family:"palatino linotype",serif; mso-fareast-font-family:"ms mincho"; mso-bidi-font-family:"palatino linotype"; color:black; mso-ansi-language:en-ca; mso-fareast-language:ja;} span.normalchar1 {mso-style-name:normal__char1; mso-style-priority:99; mso-style-unhide:no; mso-style-parent:""; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman";} span.shorttext {mso-style-name:short_text; mso-style-priority:99; mso-style-unhide:no; mso-style-parent:""; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman";} span.heading2main {mso-style-name:"heading 2 main"; mso-style-priority:99; mso-style-unhide:no; mso-style-parent:""; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"arial",sans-serif; mso-ascii-font-family:arial; mso-hansi-font-family:arial; mso-bidi-font-family:arial; color:black; font-weight:bold; text-underline:black;} .msochpdefault {mso-style-type:export-only; mso-default-props:yes; font-size:10.0pt; mso-ansi-font-size:10.0pt; mso-bidi-font-size:10.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-fareast-font-family:calibri; mso-hansi-font-family:calibri; mso-bidi-font-family:calibri;} /* page definitions */ @page {mso-footnote-separator:url("edresp_29566_files/header.htm") fs; mso-footnote-continuation-separator:url("edresp_29566_files/header.htm") fcs; mso-endnote-separator:url("edresp_29566_files/header.htm") es; mso-endnote-continuation-separator:url("edresp_29566_files/header.htm") ecs;} @page wordsection1 {size:595.3pt 841.9pt; margin:70.85pt 70.85pt 56.7pt 70.85pt; mso-header-margin:35.4pt; mso-footer-margin:35.4pt; mso-page-numbers:1; mso-header:url("edresp_29566_files/header.htm") h1; mso-footer:url("edresp_29566_files/header.htm") f1; mso-paper-source:0; mso-gutter-direction:rtl;} div.wordsection1 {page:wordsection1;} --> </style> <!--[if gte mso 10]> <style> /* style definitions */ table.msonormaltable {mso-style-name:"table normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"calibri",sans-serif;} </style> <![endif]--><!--[if gte mso 9]><xml> <o:shapedefaults v:ext="edit" spidmax="2049"/> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext="edit"> <o:idmap v:ext="edit" data="1"/> </o:shapelayout></xml><![endif]--> </head> <body lang=en-us link=blue vlink=purple style='tab-interval:35.4pt'> <div class=wordsection1> <p class=msonormal><i><span lang=en-ca style='font-family:"palatino linotype",serif; mso-bidi-font-family:"palatino linotype"'>editorial responsibilities </span></i><span style='mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-family:"palatino linotype",serif; mso-bidi-font-family:"palatino linotype"'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-family:"palatino linotype",serif; mso-bidi-font-family:"palatino linotype"'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"palatino linotype";mso-ansi-language:en-us;mso-no-proof: yes'><!--[if gte vml 1]><v:shapetype id="_x0000_t75" coordsize="21600,21600" o:spt="75" o:preferrelative="t" path="m@4@5l@4@11@9@11@9@5xe" filled="f" stroked="f"> <v:stroke joinstyle="miter"/> <v:formulas> <v:f eqn="if linedrawn pixellinewidth 0"/> <v:f eqn="sum @0 1 0"/> <v:f eqn="sum 0 0 @1"/> <v:f eqn="prod @2 1 2"/> <v:f eqn="prod @3 21600 pixelwidth"/> <v:f eqn="prod @3 21600 pixelheight"/> <v:f eqn="sum @0 0 1"/> <v:f eqn="prod @6 1 2"/> <v:f eqn="prod @7 21600 pixelwidth"/> <v:f eqn="sum @8 21600 0"/> <v:f eqn="prod @7 21600 pixelheight"/> <v:f eqn="sum @10 21600 0"/> </v:formulas> <v:path o:extrusionok="f" gradientshapeok="t" o:connecttype="rect"/> <o:lock v:ext="edit" aspectratio="t"/> </v:shapetype><v:shape id="picture_x0020_5" o:spid="_x0000_i1025" type="#_x0000_t75" alt="cc-ca_logo_xl" style='width:8.25pt;height:8.25pt;visibility:visible; mso-wrap-style:square'> <v:imagedata src="edresp_29566_files/image003.png" o:title="cc-ca_logo_xl" cropright="49569f"/> </v:shape><![endif]--><![if !vml]><img width=11 height=11 src="edresp_29566_files/image005.png" alt="cc-ca_logo_xl" v:shapes="picture_x0020_5"><![endif]></span><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"palatino linotype"'><span style='mso-spacerun:yes'> </span>2019. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (<span class=msohyperlink><a href="http://creativecommons.org/licenses/by-nc-sa/4.0/">http://creativecommons.org/licenses/by-nc-sa/4.0/</a></span>), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.<o:p></o:p></span></p> <div style='mso-element:para-border-div;border:none;border-bottom:solid black 1.0pt; mso-border-bottom-alt:solid black .25pt;padding:0in 0in 1.0pt 0in'> <p class=msobodytext style='border:none;mso-border-bottom-alt:solid black .25pt; padding:0in;mso-padding-alt:0in 0in 1.0pt 0in'><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: "palatino linotype";mso-ansi-language:en-ca'><o:p>&nbsp;</o:p></span></p> </div> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"palatino linotype";mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"palatino linotype";mso-ansi-language:en-us'>editor-in-chief: lorie kloda <o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"palatino linotype";mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"palatino linotype";mso-ansi-language:en-us'>associate editor (evidence summaries): heather pretty, christina wissinger<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"palatino linotype";mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"palatino linotype";mso-ansi-language:en-us'>associate editor (research articles): ann medaille, lisl zach<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"palatino linotype";mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"palatino linotype";mso-ansi-language:en-us'>associate editor (classics &amp; reviews): lorie kloda<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"palatino linotype";mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"palatino linotype";mso-ansi-language:en-us'>associate editor (commentaries, feature, using evidence in practice): lorie kloda <o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"palatino linotype";mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"palatino linotype";mso-ansi-language:en-us'>communications officer (news): tatiana bryant<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"palatino linotype";mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"palatino linotype";mso-ansi-language:en-us'>production editor: rachel hinrichs<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"palatino linotype";mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"palatino linotype";mso-ansi-language:en-us'>editorial intern: kimberly mackenzie<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"palatino linotype";mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"palatino linotype";mso-ansi-language:en-us'>editorial advisors: lindsay alcock, alison brettle, jonathan eldredge, denise koufogiannakis<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"palatino linotype";mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"palatino linotype";mso-ansi-language:en-us'>copyeditors: richard hayman (lead copyeditor), georgianne bordner, julie evener, heather healy, julie james, emily kingsland, alison moore, jane morgan-daniel, elizabeth stregger, elaina vitale<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"palatino linotype";mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"palatino linotype";mso-ansi-language:en-us'>indexing support: kate shore<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca><o:p>&nbsp;</o:p></span></p> </div> </body> </html> evidence summary   assisting with systematic reviews can be associated with job-related burnout in information professionals   a review of: demetres, m. r., wright, d. n., & derosa, a. p. (2020). burnout among medical and health sciences information professionals who support systematic reviews: an exploratory study. journal of the medical library association, 108(1), 89–97. https://doi.org/10.5195/jmla.2020.665 reviewed by: kimberly mackenzie research data and scholarly communications librarian lamar soutter library university of massachusetts medical school worcester, massachusetts, united states of america email: kimberly.mackenzie@umassmed.edu    received: 2 june 2020                                                               accepted:  20 july 2020      2020 mackenzie. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29791     abstract   objective – this study explored reports of burnout among librarians who assist with systematic review preparation.   design – electronic survey (copenhagen burnout inventory).   setting – the survey was advertised via three email discussion lists based in the united states of america.   subjects – the study surveyed 198 librarians and information specialists who support the systematic review process. of these, 166 completed the personal burnout scale, 159 completed the work burnout scale, and 151 completed the client burnout scale.   methods – the copenhagen burnout inventory (cbi) is a validated survey that includes three separate scales: personal burnout, work-related burnout, and client-related burnout. the end of the survey addressed demographics, including questions on the respondents’ involvement with systematic reviews. survey questions use a 0 to 100 rating scale, with 0 indicating never/to a low degree and 100 indicating always/to a high degree. the researchers shared the survey to the email discussion lists medlib-l and docline and advertised it on the medical library association (mla) news. survey answers were collected using qualtrics survey software. once emailed, the survey remained open for one month. data was coded in excel and analysis included scoring following the cbi metrics, as well as tukeyhsd and kruskal-wallis tests to determine differences in demographic groups.   main results – reported burnout levels were significantly lower for those who spend more than 80% of their time helping with systematic reviews compared to those who spend less than 10%. the consistent use of a systematic review support tool was also associated with significantly lower burnout levels. other comparisons were not significant. the average overall response score for personal burnout was 48.6. the average score for work-related burnout was 46.4 and the average score for client-related burnout was 32.5. reference librarians reported the highest average total burnout scores (47.1), while research librarians had the lowest (37.7).   conclusion – consistency, either in time spent dedicated to systematic reviews or in the use of a support tool, was associated with lower levels of burnout among librarians and information specialists. the authors suggest that these results could inform ways of improving burnout among those assisting with systematic reviews.   commentary   research has examined job-related burnout in multiple areas of librarianship, including public and academic liaison librarians (nardine, 2019; smith, bazalar, & wheeler, 2020). burnout related to a specific job role, such as supporting systematic reviews, has not been a focus. however, it is clear that supporting the systematic review process can be a time consuming and potentially stressful endeavor for a librarian (bullers et al., 2018).   this review consulted boynton and greenhalgh’s (2004) critical survey appraisal checklist, which makes a clear suggestion to use an already written and validated questionnaire when one is available. the cbi is a validated questionnaire used to measure burnout in a variety of professions. the survey was advertised via appropriate mailing lists and was available for a sufficient time for participants to respond. however, the method of recruiting respondents, by sending the requests via professional email lists, may have introduced a source of bias. as the authors themselves point out in their discussion, those experiencing feelings of burnout may have been more likely to respond to a survey questioning that topic than those not experiencing burnout. another source of potential bias comes from the cross-contamination of personal burnout affecting feelings of work-related burnout and vice versa. survey instructions directed respondents to answer based on work-related feelings, but as the authors explain, it is “difficult to compartmentalize” (p. 96).   the researchers asked respondents a number of demographic questions within the survey, which helped them compare burnout scores across different groups. it is unclear, however, if they planned these comparisons. the majority did not yield significant differences, but there were a number of demographics that were not included in the reported results. these demographic questions focused on the stages of systematic review a respondent was involved with, the type of users, and the number of information professionals that worked together on a review. it would be interesting to know whether these had an association with burnout, as would further analysis into whether interactions between demographics (such as percentage of job duties versus use of support tool) were correlated with burnout. also, a meaningful analysis of burnout between different job titles was not possible, due to the low response rates and the wide variety in job title and duties. future studies could first analyze job titles and duties for those who report working with systematic reviews, followed by a closer look at burnout in those different roles.   the results of this study do give libraries some potential ideas for mitigating burnout in those supporting systematic reviews. it appeared that a lack of consistency, whether in time spent on systematic reviews within the job role or with use of a systematic review assistance tool, led to higher rates of burnout. while not all libraries can afford or need to dedicate staff to systematic reviews as a primary job role, a clear library policy defining systematic review assistance, including the use of a review tool, could help to offset potential burnout.   references   boynton, p.m., & greenhalgh, t. (2004). hands-on guide to questionnaire research: selecting, designing, and developing your questionnaire. british medical journal, 328(7451): 1312–1315. https://doi.org/10.1136/bmj.328.7451.1312   bullers, k., howard, a. m., hanson, a., kearns, w. d., orriola, j. j., polo, r. l., & sakmar, k. a. (2018). it takes longer than you think: librarian time spent on systematic review tasks. journal of the medical library association, 106(2): 198–207. https://doi.org/10.5195/jmla.2018.323   nardine, j. (2019). the state of academic liaison librarian burnout in arl libraries in the united states. college & research libraries, 80(4): 508–524. https://doi.org/10.5860/crl.80.4.508   smith, d. l., bazalar, b., & wheeler, m. (2020). public librarian job stressors and burnout predictors. journal of library administration, 60(4): 412–429. https://doi.org/10.1080/01930826.2020.1733347   evidence summary   disadvantaged youth in southern scotland experience greater barriers to information access resulting from poor technology skills, information literacy, and social structures and norms   a review of: buchanan, s., & tuckerman, l. (2016). the information behaviours of disadvantaged and disengaged adolescents. journal of documentation, 72(3), 527-548. http://dx.doi.org/10.1108/jd-05-2015-0060   reviewed by: heather coates digital scholarship & data management librarian university library indiana university-purdue university indianapolis (iupui) indianapolis, indiana, united states of america email: hcoates@iupui.edu   received: 1 sept. 2016   accepted: 14 oct. 2016      2016 coates. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to describe the information behaviours of adolescents living in disadvantaged and disengaged circumstances in relation to social integration and self-efficacy.   design – mixed methods design using observation, semi-structured interviews, and focus groups.   setting – public agencies offering drop-in support services for employment and training at three locations in south ayrshire, scotland.                 subjects – adolescents aged 16-20 not in education, employment, or training (neet status) seeking public agency support services in south ayrshire, scotland; and public agency support workers.   methods – the south ayrshire region was identified using the scottish index of multiple deprivation (2012) and selected because nearly 20% of the region fell into the most deprived decile. participants were recruited using purposive sampling of neet youth attending drop-in support groups. over a period of 4 weeks, the 36 neet youth attending these sessions were observed regarding their information behaviours, social interactions, and experiences with support workers. following observation, the investigator used the critical incident technique during interviews to discuss real life scenarios and needs with 15 participants. one focus group of four interview participants was held to explore how groups discuss information behaviours. an information resource sorting activity was used as a discussion prompt. to gain further insight into the information needs, sources used, and barriers experienced by the adolescents who sought support, six support workers were also interviewed. these data were analyzed using iterative deductive coding and thematic analysis.   main results – internet usage of participants is similar to general population peers, with general browsing, social networking, and music and gaming usage as the most popular activities. information needs included employment, education, and training such as finding appropriate job vacancies, identifying employer addresses and contact details, and accessing public transportation schedules. other personal needs such as financial, health, and housing were identified. some of these information needs were unmet, which could have implications for their wellbeing. approximately one third of the interview participants described challenges identifying or meeting needs.   teens frequently rely on people as sources of information, preferring face-to-face interactions. they turn to family, friends, and support workers for information as important members of their social network who listen and “usually have the answer.” other frequently used sources include the internet and television news. a third of the interview participants described scenarios in which they used multiple sources – the internet, family and friends, books and leaflets, and local youth clubs and health clinics. a majority of interview participants (73%) did not use the public library because they were not interested (“i don’t read books”) or thought there was nothing there for them. of the four who used public libraries, none considered themselves regular users. focus group participants rated support workers and television news as having high credibility, friends and family as having low credibility, and the internet as having medium credibility.   the information seeking behaviours of neet teens are often passive and non-motivated, with abandonment and incompletion occurring frequently. many observation participants appeared visibly withdrawn and were reluctant or unable to engage in discussion with support workers. throughout the study, participants demonstrated an unwillingness to engage in lengthy discussions. support workers noted that attendees rely on them to find the information, evaluate it, and give the teens advice about what to do next. participants exhibited bonding social capital with family and friends, though evidence for bridging social capital was found only with support workers. these adolescents appear to be relatively isolated and have more inward-facing social capital, which puts them at risk for social exclusion.   though nearly two thirds of interview participants reported no barriers to finding information, evidence from observations and support worker interviews suggest that deficits in technology and literacy skills are substantial. despite assistance from support workers, neet youth often abandoned tasks before completing them due to literacy and technological barriers. support workers described a variety of barriers to meeting information needs: home life, isolation, literacy and access, institutional regulations, motivation, lack of confidence, and poor social skills. common circumstances relating to motivation are familial unemployment, substance abuse issues, and inability to focus.   conclusion – neet youth faced increased access and behavioural barriers beyond those of typical adolescents. access barriers are influenced by technology and literacy skills deficits, while behavioural barriers are influenced by social structures. considering the range of personal needs expressed to support workers in combination with pervasive barriers to information, further study of everyday information needs is warranted. public and third sector (e.g., typically voluntary and community organizations, associations, etc.) agencies should make remedial literacy education a priority. since the perception of public libraries is poor, librarians should collaborate with third sector agencies to develop tailored outreach and services to build sustained relationships with these disengaged adolescents. generally, these adolescents exhibit self-protective behaviours (e.g., deception, risk-taking, secrecy) in seeking out information as well as situation relevance of information sources and inward-facing social networks, all of which suggest they live in an impoverished information environment.   commentary   this study examines an unstudied population of users – adolescents who are not currently students. as the first study, the authors use concepts from social network theory, social capital, and information behaviours to develop a theoretical framework that describes the factors influencing the information seeking behaviours of teens in scotland. additionally, it places information behaviours and information literacy within the context of authentic social, personal, and technological barriers. this is an important early exploration of a complex but everyday issue.   the consolidated criteria for reporting qualitative research (coreq), a list of criteria for reporting qualitative interviews, was used to evaluate this study (tong, sainsbury, & craig, 2007). the strength of this study lies in the design. the use of three methods to gather data representing three perspectives (adolescent, support worker, and investigator) on the information seeking behaviours of neet adolescents offers triangulation. although there are gaps in the reporting, the design is thoughtful and takes into account the particular challenges in studying this population.   the reporting is less robust for the description of the research team (domain 1) and analysis and findings (domain 3; tong et al., 2007). the specific roles of each author and their training and credentials are not provided. it is also unclear whether the investigators established relationships with the participants and shared information about the study’s purpose. although the methods for developing the theoretical framework and designing the study are clearly described, procedures for recruitment and analysis are not. further description of the coding scheme along with examples would be valuable for evaluation and replication studies. as a qualitative study, the findings may not generalize to adolescents in other regions or countries or teens who do not seek out public services.   despite some reporting gaps, the authors provide compelling evidence to suggest that these teens are living in an impoverished information environment. it is an excellent first step towards documenting the information needs and behaviours of adolescents seeking employment and training outside the traditional educational system. we simply do not have an adequate understanding of the information seeking needs and behaviours, as well as the interplay between social networks, in teens. future studies could adapt this study model for examining local populations of teens receiving support services. it would be particularly interesting to combine this rich qualitative approach with a broader survey of teens across europe and north america to compare information needs, sources, and perceptions of public libraries. the value of public libraries in fostering community engagement is not yet well documented. neither do we understand how to recognize and prevent disengagement, which is often reliant on apparent failure to meet societal expectations, such as dropping out of school or being unemployed. considering the participants’ perception of public libraries, there is tremendous potential value in situating information literacy training within the context of other job and life skills training opportunities. as public libraries increasingly serve as community anchors, it is also worth considering how programs can foster community engagement of disadvantaged youth as much as they develop their digital and information literacy skills.   references   tong, a., sainsbury, p., & craig, j. (2007). consolidated criteria for reporting qualitative research (coreq): a 32-item checklist for interviews and focus groups. international journal of quality in health care, 19(6), 349-357 http://dx.doi.org/10.1093/intqhc/mzm042 research article   expressing intellectual freedom: a content analysis of catholic library world from 1980 to 2015   megan e. welsh assistant professor, interdisciplinary arts & humanities librarian norlin library university of colorado boulder boulder, colorado, united states of america email: megan.welsh@colorado.edu   received: 24 june 2016   accepted: 22 oct. 2016         2016 welsh. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – professional librarians have varying values relating to the topic of intellectual freedom that may or may not align with the american library association’s (ala) policies defining professional expectations on the topic. the personally held values and beliefs of roman catholic librarians and those working in libraries affiliated with roman catholicism are worthy of study to determine how personal religious values may translate into professional practice. the objective of this paper is to ascertain how frequently and in what context the topics of intellectual freedom and censorship were expressed in articles published in catholic library world (clw), the professional journal of the catholic library association (cla) from 1980 to 2015. published content on these topics can be used as evidence to determine how this population discusses the concept of intellectual freedom.   methods – articles relevant to these topics were retrieved from the american theological library association catholic periodical and literature index (atla cpli) and library, information science & technology abstracts (lista) databases by conducting keyword searches using the terms “intellectual freedom” and censorship. each retrieved publication was analyzed by counting the number of times the phrase “intellectual freedom” and the root censor* occurred. through a deep reading of each publication, statements containing these search terms were then coded as positive, negative, or neutral, establishing a context for each occurrence.   results – the majority of published content supported intellectual freedom and opposed censorship. negative content typically occurred in publications about children or school libraries. additionally, clw contributors did express a certain level of conflict between personally held religious values and professional values.   conclusions – this study adds to the limited research available on the intersection of personally held religious values and professional values. further research is needed to gain a better understanding of the conflict between values amongst catholic librarians, librarians of other faith traditions, and librarians in general.     introduction   the american library association (ala) has published a code of ethics (coe) for professional librarians, as well as many other policy documents and interpretations of these policies, enumerating the role of librarians and the rights of library users, which librarians are entrusted to support. the various documents that outline professional standards for librarians, including the coe, can be found in the ala’s intellectual freedom manual (ifm). this corpus establishes the concept of intellectual freedom (if) as a central theme in the profession and promotes the protection of if as a primary role for librarians, stating that “[t]he freedom of expression guaranteed by the first amendment and the corollary to that freedom, the freedom to read, are uniquely fulfilled by the library” (office for intellectual freedom of the american library association, 1974, p. viii). in addition to the professional ethics outlined in these documents, librarians bring their own personal experiences and values to work. celeste west, a key figure of the “social responsibility movement” in american libraries in the 1960s “argued that it was not possible for librarians to take apolitical, nonaligned positions in their professional work” (samek, 2001, p. 1). in other words, librarians cannot always be objective professionally. personal experiences, values, and beliefs influence a librarian’s professional work. the question this paper raises is: in relation to the concept of if, how does a librarian’s personal bias present itself in their professional work?   the personally held values and beliefs of roman catholic librarians and those working in libraries affiliated with catholicism are worthy of study to determine how personal religious values may translate into professional practice. roman catholic librarians may be influenced by the church to which they belong, especially based on a history of institutionalized support of censorship. this support is exemplified by the ecclesiastically sanctioned index librorum prohibitorum (index of forbidden books) which banned the reading of certain works without approval from the church and through publication practices that require authors to get permission from the church before publishing books (burke, 1952; betten, 1932). one way in which this population and their attitudes towards if can be studied is by analyzing their expression of personal and professional values through their writing. this study is a content analysis of articles published in catholic library world (clw), the journal of the catholic library association (cla), between 1980 and 2015, focusing on the concept of if, a topic influenced by personal values and rife with various levels of support among professional librarians.   literature review   although the ala has not officially accepted a definition of “intellectual freedom” (office for intellectual freedom of the american library association, magi, t. j., & garnar, m., 2015b), the unofficial definition described in the first edition of the ifm published in 1974 states that if is “the right of any person to believe whatever he wants on any subject, and to express his beliefs or ideas in whatever way he thinks appropriate” and also includes “the right of unrestricted access to all information and ideas regardless of the medium of communication used” (p. vii). this definition identifies two major principles inherent to the concept of if: the freedom of expression and the freedom to access information. ala policy documents, especially the library bill of rights (lbr) and the freedom to read statement, explain the nuances of these freedoms, emphasizing the importance of accessing materials that provide a variety of viewpoints on all sides of an issue and protecting information from the threat of censorship, especially based on “doctrinal disapproval” (office for intellectual freedom of the american library association, magi, t. j., & garnar, m., 2015a, p. 15). censorship, which can be defined as “the suppression of ideas and information that certain persons—individuals, groups or government officials—find objectionable or dangerous,” (ala, 2016a) directly inhibits access to information and, therefore, intellectual freedom.   although these association-wide documents exist to promote if, the ala itself and not all librarians have defended the principles of if over the years. a history of ala policy on intellectual freedom (2015) details the variety of instances where the association and the profession in general wavered on its position on censorship (see “chapter 1 ala and intellectual freedom: a historical overview”). for example, the same year that the lbr was adopted in 1939, john steinbeck’s grapes of wrath was banned in libraries throughout the country (office for intellectual freedom of the american library association et al., 2015b). additionally, only two library systems, chicago public library and des moines public library, had policies protecting if before the lbr was adopted (latham, 2009; office for intellectual freedom of the american library association et al., 2015b). the 1960s pushed the role of librarians as advocates for social justice, instead of simply maintaining a neutral position on social issues (samek, 2001; robbins, 1996). the role of religiously affiliated librarians is currently addressed on a webpage entitled “religion in america” (ala, 2016c). in this document, the ala (2016c) clearly states that the rights of library employees “to self expression including religious expression are more restricted than those of the general public for the simple reason that they are employed for a purpose” (n.p.). it also refers to the coe and states that librarians cannot “refuse to answer questions on the basis of individual conscience” (ala, 2016c, n.p.). this discussion about the role of librarians persists, emphasizing the various degrees of support for if that exist across the profession and which are based on various personal and professional values.   the intersection of personally held religious values and professional values relating to the topic of if has been the subject of a small number of previous studies. the majority of these studies examine christian values broadly without focusing on any single christian denomination, nor do they cover non-christian faith traditions. for example, gregory a. smith edited a book of essays entitled christian librarianship: essays on the integration of faith and profession (2002) in addition to writing an article about if and bible college libraries (2004) in which he states that if “is probably the greatest point of tension for evangelical christians in the library profession” (p. 242). craighton hippenhammer summarized the findings of a survey in which he asked evangelical christian college librarians about policies, the lbr, and challenges to collections. the majority of respondents fully or mostly supported the lbr, and responses showed that libraries with policies retain more challenged materials than those without: however, only half of all respondents had a written policy regarding challenged materials. scott kaihoi (2015) conducted an online survey of 123 christian librarians regarding their viewpoints on professional ethical standards represented in the lbr in an effort to “discover the extent to which christian librarians perceive conflict between the ethics of their christian worldview and the ethics that underlie the lbr” (p. 52). kaihoi (2015) concluded that the majority of christian librarians generally support the lbr, and, therefore, its ethical propositions, but the majority of respondents also indicated that they did not always adhere to it, most notably when viewpoints expressed by authors conflicted with values of the library community. additionally, “only a minority of respondents indicated that their personal views and professional decisions conflicted with the principles found in the lbr” (kaihoi, 2015, p. 49) while the majority of respondents also approved of limiting access to some materials, such as pornographic content, thus indicating a contradiction between stated values and actual practice. kaihoi’s (2015) study indicates general, yet not absolute, support for principles of if amongst christian librarians.     the attitudes toward if held by roman catholic librarians in the united states (as opposed to those of the broader christian librarian community) are specifically examined in only one published study. working as a part of an ad hoc committee of the midwest unit of the catholic library association (cla) approved to investigate censorship and published as her graduate thesis, sr. maureen kehoe (1977) conducted a study “to determine the attitudes of members of the midwest unit of catholic library association toward intellectual freedom and censorship” (p. 3). cla did not have an official policy on if, and kehoe concluded that measuring the views of cla members, especially in light of the degree to which members professed support for ala policy documents, could help the organization develop an if policy (kehoe, 1977, p. 9). the constitutions and bylaws of cla at the time (as cited in kehoe, 1977, p.7) stated that the “object of the organization (cla) is the promotion of catholic principles.” this emphasis on the promotion of catholic principles coupled with the lack of official cla policy and contrasted with the concept of if outlined in the, then recently published, ifm created a potential climate of conflicting ethical standards.   kehoe (1977) distributed questionnaires asking participants about their familiarity with and the degree to which they agreed with the lbr, freedom to read statement, and the school library bill of rights. kehoe also asked participants if they had collection development and challenged materials policies, attempting to ascertain the degree to which participants had applied principles of if in a practical manner. kehoe (1977) concluded that most respondents “agreed with the basic principle to include in libraries materials on all sides of controversial issues . . .” (p. 3). additionally, the majority of respondents also agreed that materials should not be “removed from libraries because of partisan or doctrinal disapproval” (p. 138). overall, cla members of the midwest unit were supportive of principles of if, especially the principle of access.   in addition to the self-reported data gathered through the studies cited above detailing what librarians think about if and how they say they act, evidence of librarian attitudes toward if can be gathered in additional ways. these include 1) collecting anecdotal evidence of the practice of librarianship (how do librarians act in specific situations?), 2) reviewing institutional policies (what policies or statements of faith have been developed to guide the practice of librarians?), and 3) analyzing publications (what are librarians saying when they write about their personal and professional values?). in this final method, a content analysis, the researcher identifies specific words relevant to the construct of intellectual freedom that provide contextual evidence and insight into what librarians feel value in expressing. the remainder of this paper focuses on the third method through an analysis of content published in catholic library world (clw) over a 35-year timeframe. clw is the official publication of the cla, the national professional association that represents those affiliated with catholic libraries. the first issue of clw was published in 1929 and continues to be published currently as a quarterly journal containing articles “focusing on all aspects of librarianship, especially as it relates to catholicism and catholic studies” (catholic library association, n.d.) and book reviews. clw is ideal for analysis based on its prominence as the singular official journal representing this demographic. it is also ideal due to the lack of previous research about the demographic of catholic librarians and those affiliated with catholic libraries who may express both personally held religious values and professional values of librarianship through the written contributions published through the journal itself.   aims   this study attempts to ascertain how clw contributors have expressed their views on if through articles published in the journal. two research questions guided the course of this study:   what are the frequency and context of the terms “intellectual freedom” and “censorship” in articles about those topics published in catholic library world? what do these patterns indicate about the attitudes of clw authors towards intellectual freedom and censorship?   research methods   clw articles published from 1980 to 2015 were retrieved by searching for the keywords “intellectual freedom” and censorship separately in the american theological library association catholic periodical and literature index (atla cpli) and library, information science & technology abstracts (lista) databases. the phrase “intellectual freedom” is the overarching principle that is analyzed. censorship was also chosen as a keyword term since the reaction to censorship is the foundation of the concept of if (office for intellectual freedom of the american library association, 1974), and it is also a term that frequently appears across library science literature. this set of articles represents what is readily available for those looking for information in clw on if and the related concept of censorship. although using these terms does not cover all aspects of if, it is a logical place to start a review of how clw contributors express their views on these issues.   the terms “intellectual freedom” and censorship were searched without selecting a specified field to search (e.g. subject term or author-supplied keywords), representing a broad keyword search. the date range for this study was chosen based on accessible journal issues in each database. atla-cpli indexes clw from 1980 to the present and, while lista indexes some articles from 1966 to the present, its indexing is more limited between 1966 and 1980. searching was limited to the same timeframe, 1980 through 2015, in each database for the purpose of being consistent. the author searched these databases in february 2016 and pdfs of the articles were acquired either through the databases or via interlibrary loan.   retrievals from both databases contained some overlap as well as some articles unique to only one of the databases. in order to have as comprehensive of a view of the journal as possible, articles retrieved from each database were analyzed. figure 1 details the number of total publications retrieved. the keyword censorship retrieved 28 publications initially, almost twice as many as the 15 publications retrieved by searching “intellectual freedom.”     figure 1 number of publications retrieved with the search terms "intellectual freedom," censorship in clw, 1980–2015     figure 2 number of instances of "intellectual freedom" or censor* found in retrieved clw publications between 1980–2015     the sample of publications analyzed further in this study includes 24 from atla-cpli and 14 from lista. both articles (constituting full length journal articles and short columns) and book reviews (since some reviews offered opinions about if) were retrieved. there were 23 articles and 15 reviews, totaling 38 publications in all. duplicated articles (numbering six in all) retrieved from searches in both databases were not counted twice. upon retrieval, two full-text word searches were conducted within each publication. first, the publications were searched for the phrase “intellectual freedom” and then each was searched for the root word censor* (allowing for variations such as “censorship” and “censors” to be retrieved). to determine the frequency of the occurrence of these terms, some publications were searched electronically and the occurrence tallied and others, without optical character recognition (ocr), were searched manually and terms were counted as the articles were read. typically publications searched manually were those published prior to 2004 and requested through interlibrary loan. in addition to the title of the publication, date of publication, and some additional metadata, the frequency of each term was recorded in a spreadsheet. occurrences of the search terms found in the title of the article, works cited, or urls were not included in the recorded count. figure 2 indicates the total number of coded statements containing “intellectual freedom” or censor* found in the clw publications retrieved for this content analysis.   the sentence in which the term occurred was transcribed and recorded in the spreadsheet, chronicling the context in which the term was found. the sentence was coded as either positive (in support of if or opposing censorship), negative (opposing if or supporting censorship), or neutral (neither supporting nor opposing if or censorship). an example of a statement coded as positive is “intellectual freedom is not only a value but a right” (ladwig & archer, 2010, p. 284). this next example proposes a certain degree of censorship, therefore, negatively reflecting if: “what we need, then, is ‘responsible censorship’ which allows for the needs and rights of the young adult and yet allows for the rights and responsibilities of the adult” (hodges, 1982, p. 399). neutral statements typically state facts instead of judgment and represent organization names. this is a fact-based example: “there are some books which are provided that parents are apt to censor” (ballard, 1983, p. 321). a deep reading of each article generated an understanding of broader themes within the article and also allowed individual sentences to be placed in a broader context to be coded more accurately.       figure 3 context of statements including the term "intellectual freedom" in clw, 1980–2015.     figure 4 context of statements including the term censor* in clw, 1980–2015.     results   as seen in figure 1, few articles or book reviews were published on the topic of if and censorship in the 1990s. almost an equal number were published in the 1980s (20 publications) and from 2002 to 2015 (22 publications); however, only one publication, a book review of the ifm was published in the 1990s. more research is needed to determine a possible cause for this lull in publishing on these topics. also evident is the fact that more publications treated the subject of censorship than intellectual freedom.   the full-text search of each publication indicated a total of 94 instances of the phrase “intellectual freedom” and 202 instances of censor*. figures 3 and 4 indicate the context of the statements in which the terms “intellectual freedom” and censor* occurred, respectively. in each case, the total number of negative representations of if and pro-censorship statements were the minority viewpoint expressed in these clw articles and reviews.   figure 3 indicates that, when intellectual freedom was discussed by name, the majority of the time (60%) the author indicated support for the concept. only 3% of statements were unsupportive of intellectual freedom; however, figure 4 indicates that 15% of the statements containing the word censor* promoted censorship, a principle antithetical to the concept of intellectual freedom. this higher percentage of negative statements could support kaihoi’s (2015) findings that, in theory, christian librarians are supportive of intellectual freedom in general, yet, in practice, librarians are willing to limit access to some controversial materials.   figures 5 and 6 highlight the context of statements over time, demonstrating the expression of more support for intellectual freedom and less support for censorship over the course of the timeframe of this study. all three statements containing “intellectual freedom” coded as negative were published in the 1980s. however, almost an equal number of positive statements were published between 1980 and 1990 (27 positive statements) and also over a longer, more recent, timeframe between 2000 and 2015 (28 positive statements). graphing statements containing censor* over time shows that clw contributors have also expressed less support of censorship between 1980 and 2015. there were 22 pro-censorship statements in the 1980s involving the term censor* compared with only 8 statements supporting censorship from 2000 to 2010 (in fact, throughout all of the 21st century). although the language surrounding censorship is less negative, it has also become more neutral. about double the number of neutral statements (47) compared to positive statements (23) were published from 2000 to 2015.     figure 5 context of statements including the term "intellectual freedom" in clw, 1980–2015.     figure 6 context of statements including the term censor* in clw, 1980–2015.     in addition to the coded data gathered, some noteworthy themes emerged through a deep reading of the content. especially of note are the clw articles that helped to facilitate a discussion between the clw authors and their audience. in the 1980s, pre-press copies of articles on controversial topics were distributed to select readers, whose responses were published alongside the main article, thus encouraging discourse around the topic. there were five of these articles published from 1983 to 1989 relating to if which were analyzed in this study. themes occurring in them were the difference between censorship and selection, professional responsibility, self-censorship, policy development, the idea that the best offense against censorship is a good defense, censorship online, personal values, and ethics in general. the majority of statements in these articles were positive, though some comments from select readers were negative.   across all articles and reviews retrieved, statements about intellectual freedom and censorship occurred most in articles discussing school libraries (see figure 7). in fact, of the statements coded by setting, most statements in general dealt with school libraries. the popularity of discussing if, censorship, and their relationship to school libraries in clw aligns with the fact that, according to statistics published by the office for intellectual freedom of the american library association (2016b), more challenges to materials occurred in a school or school library setting compared with any other setting, including public and academic libraries between 2000 and 2009 (n.p.). it is not surprising that, since the majority of challenges occur in school or school library settings, content published in clw would mirror that trend.   the three most frequently discussed themes occurring across clw articles dealing with intellectual freedom and censorship were 1) the importance of policy development, 2) the use and recommendation of professional guides (such as the ifm), and 3) the influence of personal values on professional work.     figure 7 context of statements based on library setting discussed in each article in clw, 1980–2015.     thirteen publications focused primarily on policy development. the majority of these articles highlighted the need for selection and challenged materials policies, the problems that can arise when policies are not implemented, and suggested resources to help develop policies. kamm’s (1988) article detailed a program to defuse censorship by prioritizing the development of if related policies, emphasizing their importance in combating challenges, and developing a pr campaign to educate stakeholders. the author also suggested relying on a professional network for support when challenges do arise. her well-articulated argument in support of policy development, operating under the guiding philosophy of “the best offense is a good defense” (p. 176), garnered much praise and agreement among advanced copy readers.   the use of professional guides and resources was promoted in both articles and book reviews. three reviews (weathers, 1993; cuseo, 2002; manz, 2011) and one column detailing new ala publications (“from ala editions,” 2006) recommended the use of the ifm, and five additional reviews referred readers to books dealing with censorship and intellectual freedom in school (cuseo, 2003; manz, 2009a; sirvint, 2015), academic (manz, 2009b), and public libraries (bagley, 2012). many articles cited advice published by the office for intellectual freedom and some even reprinted recommendations from organizations that promote if. for example, to accompany the text of her article, hunter (1982) reprinted the “checklist for survival against censorship” (p. 288)  which had been compiled by the intellectual freedom committee of the michigan association for media in education and “guidelines for writing a materials selection policy” (p. 289) prepared by the intellectual freedom committee of ohio association of school libraries. the inclusion of these recommendations not only encourage the use of these aids, but also indicates just how many professional organizations and texts exist to serve as professional resources.   discussion   the most interesting theme to note is the indication of ethical conflict between personally held values and professional values that some clw contributors expressed. one author stated that it can be difficult to develop fail-safe guidelines to prevent censorship “[b]ecause individual attitudes, beliefs, and values guide actions . . .” (rehring, 1982, p. 277). in another article, a clw contributor stated that fulfilling her professional commitment is a “difficult matter” because she has problems “melding together the christian philosophy of our school and the intellectual freedom of our students” (hodges, 1982, p. 399). this testifies to the struggle some librarians face in balancing values contrary to those held in the library profession. adams (2010) acknowledged this potential conflict in a more recently published article, stating: “it may be difficult for catholic school librarians with strong personal religious beliefs to set aside their religious convictions while selecting books” (p. 114). she stresses the importance of the coe in guiding professional practice, especially in avoiding self-censorship due to personal bias. self-reflection can also be used as a way to minimize bias. pierson (1987) promotes thinking critically and asking “ourselves whether we are indeed as objective as we like to think!" (pierson, 1987, p. 120).   the variety of opinions about how librarians should act according to their values is exemplified in one article in which the author raises a hypothetical case about a young patron requesting a copy of a book about how to kill oneself (swan, 1988). reaction from librarians varied from providing the book as requested without any other course of action to providing the book with provisions including taking the patron’s information to contact his parents, violating his intellectual freedom by not valuing his privacy. through his example, swan (1988) demonstrates that, despite a straightforward coe, librarians also have “decision-making roles as ethical, sympathetic human beings” (p. 272), and our professional goals are realized in the "real world of compromise and ambiguity" (p. 273). personal values may inspire concern and a need to protect, motivating that librarian to refuse access to the title, withholding information and, therefore, compromising the patron’s intellectual freedom. sr. therese marie echoed this motivation to guide and protect library users, and she encouraged all librarians to be “thusly motivated, not just those of us who happen to be catholic, or a religious” (ballard, 1983, p. 322). even as some clw contributors express support for if and against censorship, there is still an undertone of a desire to guide users and base intellectual freedom decisions on their own personal judgment. for example, sr. joanne korn wrote first that “intellectual freedom is so important to our basic freedoms,” indicating support for the concept (catano & hsu, 1989, p. 267). however, she goes on to say that “it is vital that we librarians allow ourselves the freedom to provide materials for our students which we, according to our consciences, feel should be made available to them,” indicating a level of subjectivity in providing access (catano & hsu, 1989, p. 267).   in addition to individual librarians functioning as a protector and guide in the acquisition of information, the roman catholic church can also be seen as a protector more generally. in an article discussing the moral authority of the roman catholic church manifested through the index librorum prohibitorum (a list of banned books, authors, and topics maintained by the church from the 16th century until 1966), j. parker ladwig, one of the two authors of the article, rationalized the history of church censorship. ladwig (2010) defended the index librorum prohibitorum stating that since books and the ideas within them may be harmful to morals and truth, it is the role of the church, having moral authority, to protect and guide readers (p. 281). the moral authority of the church trumps the constitution (including the first amendment) and the un’s universal declaration of human rights, two documents frequently cited as ensuring intellectual freedom as a fundamental american right specifically and a global human right in general. if the church is viewed as having ultimate moral authority, this adds another level of complexity to the struggle some catholic librarians face in both practicing their religion and their profession. catholic librarians may minimize the importance of professional documents such as the lbr and coe, in protecting the intellectual freedom of library users and, instead, prioritize church teaching antithetical to intellectual freedom as defined by the profession. whether because of the moral authority of the church, the philosophy of a religious school, or the personal values of individual librarians, the content analyzed in this study indicates that the path to supporting intellectual freedom is not always clear for some librarians.   limitations and future research   intellectual freedom is a broad concept, with access and expression as two of its main components and many related principles (including privacy and an opposition to censorship). the words identified as the subject of this content analysis (“intellectual freedom” and censor*) adequately represent the construct of expressions of intellectual freedom, making this a valid method of inquiry to evaluate clw contributors’ written values through their expressions on the subject. yet, this study focused on if in a broad sense and only one of its related principles—the opposition to censorship. yet, finding support or disagreement with the concept of if goes beyond searching only two phrases. many other words representative of the concept could be used to gather additional information about the values expressed by clw contributors. such keywords could include access, expression, ban*, prohibit*, privacy, challenge, and obscen*. the need to search for other words in the text is reinforced by the fact that not all retrieved content contained the words “intellectual freedom” or censor*, despite still being about these topics. in these articles, browne & hanson (1982) and fein (1985) used words such as challenge and confidentiality to express their positions on if. other articles that do not contain those terms and that were not retrieved based on the database search may also include a discussion of if and censorship and could provide a more comprehensive view on how those topics are represented in clw. additionally, the coding of the statements containing “intellectual freedom” and censor* was completed solely by the author. although this was conducted as objectively as possible, other researchers may code some statements differently.   it is significant to note that not all articles that are potentially relevant are indexed and accessible electronically in the two databases used in this study. analog research led to the discovery of an article entitled “censorship: a selective bibliography” published in april 1980 by donald f. schubert. the article itself contains both “intellectual freedom” and censor*. the topic of the article is certainly relevant to this study, yet it was not indexed and was therefore not accessible in either database. this discovery suggests that deep reading of clw articles is necessary for future content analyses.   further study could include a content analysis of other journals in the field of librarianship during this same timeframe. although the findings of this content analysis are consistent with the findings of the survey that kehoe (1977) conducted, indicating general support among catholic librarians for intellectual freedom, more research must be conducted to determine if the ethical concerns raised through clw contributor comments represent issues specific to catholic librarians or if they are representative of librarians more broadly. frequency and context of intellectual freedom and its related principles could be compared to content published in clw during this same period, especially to investigate how catholic writings compare to writings of those who are not catholic. additionally, since the majority of research has been conducted on christian librarians in general, further research should be conducted on the intersection of professional values and the values of specific christian denominations and other faith traditions. also interesting to research would be the distribution of publications on the topic in other journals, especially to determine if fewer retrievals occurred in the 1990s as they did in clw.   conclusion   this content analysis based on the research interest of how personal and professional values intersect finds that positive, anti-censorship, and neutral statements about intellectual freedom and censorship greatly outnumber the negative, pro-censorship ones, indicating a general expression of support for intellectual freedom amongst clw contributors between 1980 and 2015. more research is needed to determine 1) why so few articles were published on if and censorship in the 1990s, 2) the frequency and context of the occurrence of other terms besides “intellectual freedom” and censor* related to the concept of intellectual freedom, 3) the history of how if and its related principles have been represented in clw since its first publication in 1929, and 4) how the content of clw relating to intellectual freedom mirrors the content found on the topic in other professional journals. finally, the published evidence of the struggle between balancing personal and professional values that some clw authors expressed confirms the need for further research on how personally held religious values of librarians influence their professional values and practice. a librarian is not only composed of their professional identity and ethics but also a combination of all of their personal experiences, values, and beliefs, making the practical application of intellectual freedom an interesting topic of study.   acknowledgements   the author wishes to thank ian burke for his contributions and advice in the development and review of this manuscript.   references   american library association. (2016a). intellectual freedom and censorship q & a. retrieved from http://www.ala.org/advocacy/intfreedom/censorshipfirstamendmentissues/ifcensorshipqanda   american library association. (2016b). number of challenges by reasons, initiator, & institution, 2000-2009. retrieved from http://www.ala.org/bbooks/frequentlychallengedbooks/statistics/2000-09#institution2000   american library association. (2016c). religion in america. retrieved from http://www.ala.org/advocacy/intfreedom/librarybill/interpretations/meetingrooms/religion-q-a   adams, h. r. (2010). preparing for and facing a challenge in a school library. catholic library world, 81(2), 113-117.   bagley, c. (2012). protecting intellectual freedom in your public library [review of the book protecting intellectual freedom in your public library, by j. pinnell-stephens]. catholic library world, 83(1), 60.   ballard, t. (1983). intellectual freedom and representative government. catholic library world, 54(8), 320-322.   betten, f. s. (1932). the roman index of forbidden books: briefly explained. chicago, il: loyola university press.   browne, b. g., & hanson, d. (1982). technique of using role play for a challenged book. the catholic library world, 54(2), 54-55.   burke, r. a. (1952). what is the index? milwaukee, wi: the bruce publishing company.   catholic library association. (n.d.). catholic library world author guidelines. retrieved from http://cathla.org/documents/clwauthorguidelines.pdf   cuseo, a. o. (2002). intellectual freedom manual, 6th ed. [review of the book intellectual freedom manual, 6th ed., compiled by the office for intellectual freedom of the american library association]. catholic library world, 72(3), 187.   cuseo, a. o. (2003). censorship and the school library media center [review of the book censorship and the school library media center, by n. kravitz]. catholic library world, 74(1), 62.   catano, p., & hsu, b. (1989). online searching in school libraries: censorship considerations. catholic library world, 60(6), 265-275.   fein, l. a. (1985). children and confidentiality. catholic library world, 56(9), 365.   from ala editions. (2006). catholic library world, 76(3), 194.   hippenhammer, c. (1993). patron objections to library materials: a survey of christian college libraries part i. the christian librarian, 37(1), 12-17.   hodges, g. g. (1982) library instruction programs: development, implementation, and evaluation. catholic library world, 53(9), 398-400   hunter, d. (1982). accessibility to media in the school library. catholic library world, 53(7), 288-289.   kaihoi, s. (2015). christian librarians and the ethics of the library bill of rights. theological librarianship, 8(1), 42-60.   kamm, s. (1988). defusing censorship. catholic library world, 59(4), 176-178.   kehoe, s. m. (1977). attitudes of members of the midwest unit of catholic library association toward recent revisions of the documents on intellectual freedom adopted by the american library association. retrieved from eric (ed143363)   ladwig, j. p., & archer, j. d. (2010). index of prohibited books: liberty, license and the common good. catholic library world, 80(4), 279-288.   latham, j. m. (2009). wheat and chaff: carl roden, abe korman, and the definitions of intellectual freedom in the chicago public library. libraries & the cultural record, 44(3), 279-298.   manz, w. h. (2009). ensuring intellectual freedom and access to information in the school library media program [review of the book ensuring intellectual freedom and access to information in the school library media program, by h. r. adams]. catholic library world, 79(4), 310.   manz, w. h. (2009). protecting intellectual freedom in your academic library: scenarios from the front lines [review of the book protecting intellectual freedom in your academic library: scenarios from the front lines, by b. m. jones]. catholic library world, 80(2), 141.   manz, w. h. (2011). intellectual freedom manual, 8th ed. [review of the book intellectual freedom manual, 8th ed., edited by c. d. morgan]. catholic library world, 82(1), 59-60.   office for intellectual freedom of the american library association. (1974). intellectual freedom manual. chicago, il: american library association.   office for intellectual freedom of the american library association, magi, t. j., & garnar, m. (eds.). (2015a). intellectual freedom manual (9th ed.). chicago, il: ala editions.   office for intellectual freedom of the american library association, magi, t. j., & garnar, m. (eds.). (2015b). a history of ala policy on intellectual freedom: a supplement to the intellectual freedom manual, ninth edition. chicago, il: ala editions.   pierson, r. (1987). access to information: opinions and speculations. catholic library world, 59(3), 118-121.   rehring, m. c. (1982). some thoughts about controversial materials: school libraries. catholic library world, 53(7), 277-278.   robbins, l. s. (1996). censorship and the american library: the american library association's response to threats to intellectual freedom, 1939-1969. westport, ct: greenwood press.   samek, t. (2001). intellectual freedom and social responsibility in american librarianship, 1967-1974. jefferson, nc: mcfarland & company.   sirvint, r. (2015). intellectual freedom for teens: a practical guide for ya & school librarians [review of the book intellectual freedom for teens: a practical guide for ya & school librarians, edited by k. fletcher-spear & k. tyler]. catholic library world, 85(3), 209.   smith, g. a. (2002). christian librarianship: essays on the integration of faith and profession. jefferson, nc: mcfarland & company.   smith, g. a. (2004). intellectual freedom and the bible college library. christian higher education, 3(3), 241-259. http://dx.doi.org/10.1080/15363750490433269   swan, j. (1988). speaking out: helpful librarians and hurtful books. catholic library world, 59(6), 271-274.   weathers, b. (1993). intellectual freedom manual, 4th ed. [review of the book intellectual freedom manual, 4th ed., by the american library association]. catholic library world, 64(1), 30.   research article   women in adult education program for sustainable development: challenges and implications for library and information services   ngozi perpetua osuchukwu department of library and information science madonna university, nigeria (okija campus) anambra state, nigeria email: ngostary2k@yahoo.com   ndidiamaka lucy nebolise academic librarian national open university of nigeria (noun) awka study center anambra state, nigeria email: ndynebe@yahoo.com   received: 9 nov. 2017                                                                    accepted: 24 oct. 2018      2019 osuchukwu and nebolise. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29366     abstract   objective – education offers advances in human and social development. it provides knowledge and resources that hold the potential for economic empowerment, resulting in a better livelihood. hence, women need access to education with library services, if they are to have a voice, participate in sustainable development, and take care of their own health, as well as that of their children and members of their households. this paper examines the challenges women encounter in onitsha metropolis, anambra state, nigeria. this study seeks to gain insight into the resources used to enhance learning, as well as the students’ perceptions and satisfaction with their learning experience.   methods – a descriptive survey research design was used. the study was carried out in five adult education centres in the city with a sample size of 120 women, randomly selected for collection of data. questionnaire, interview, and observation methods were employed. the physical assessment of the centres was done for an evidence based report and to assess the real situations of the centres.   results – the findings show the challenges faced by nigerian women in their pursuit of ongoing adult education included: time for the classes which are usually held in the evening, poor financial status, lack of encouragement from spouses and relations, poor learning environments, and stress. the data were analyzed using percentages and frequency counts. they are presented in tables and figures.   conclusions – it is recommended that education and library management should assess these centres for program improvements like providing more enabling environments and learning facilities. the implication of the study is that library and information services should be extended to these women to stimulate and support learning with the right attitude for active involvement in the educational activities and for enhancement of social inclusion.     introduction   educational attainment is arguably the most empowering strategy that has the potential to change lifestyles and increase earnings, as well as improve one’s standard of living. education is vital to everyone and has the potential to drive creativity and productivity and close the gap for socially marginalized women in achieving freedom from poverty, allowing them to fully develop and participate as citizens (sanchez moretti & frandell, 2013). libraries also play an important role in promoting education and engaging women in information services, through the provision of resources like information communication technology and other literacy activities. the overarching goal is to reduce illiteracy rates affecting 493 million women worldwide (kelleher, 2014; gillard, 2014). improved literacy rates will allow for increased participation and contributions by women for sustainable development.   adult education is classified as any form of education or learning programs in which adults participate (kwapong, 2005). research has shown that education helps in assisting adults with basic knowledge, reading and writing skills, and use of numbers, which are all critical in the fight against poverty and inequality (nhamo & nhamo, 2006). this is especially important for women, as adult education creates spaces for them to earn their school equivalency or to gain further knowledge in their chosen profession. in nigeria, an illiterate woman is placed in the background and without adult education programs, she cannot understand the framework of today’s societal standards or participate in nigerian politics. participating in the adult evening education program may not be an easy venture to pursue given their responsibilities of cooking, child rearing, marriage, and other wifely roles in the home; geographical locations; social-cultural background; religion; health; or economic status. the multiple roles women play in the household can be overwhelming in accommodating other valuable social literacy activities (cleveland clinic, 2018; wanyama, 2014).   literacy programs in the south eastern region of nigeria have had a tremendous impact on the lives of women. here, the value of woman’s education has been underestimated but is beginning to be seen as a vital priority to a more functional existence (mbah, 2018). the adult evening classes were restructured to enable the women to acquire a quality education for the actualization of sustainability development goals (sdg). sustainable development goals have been set by the united nations to end hunger and poverty and improve access to education, while promoting gender equality. the sdgs also seek to improve a person’s overall well-being through improved access to healthcare, clean water, and sanitation initiatives, as well as economic development (united nations, 2015). giving women a second chance at literacy through adult education will increase their earning power and give their children a brighter future (kelleher, 2014). this study identifies the resources and challenges of women in adult education in attaining their goals for improvement of quality of life.   statement of the problem   low literacy levels for women can be attributed to inequality, cultural bias, unequal opportunities, gender disparity, poverty, and lack of decision-making (alexander & robinson, 2014). this is true in the case of nigeria and is the reason adult education centres have been created, being mindful that every woman can learn if given an enabling environment and the right resources, like library services. library services are vital because they supplement and support both formal and informal education, while promoting education at all levels. however, what is not clear is what constitutes adequate resources for the adult education program. this study addresses and assesses the availability of resources and the challenges women face in their pursuit of adult education, as well as the implications for library and information services.   literature review   education is recognized as a source of empowerment for women throughout the world and has become one of the foremost topical issues throughout africa (farrell, 2016). adult education is an inalienable human right that is enshrined constitutionally by all countries to ensure its effectiveness and practicability (nyagoha, 2016; sanchez moretti & frandell, 2013). this is the reason adult education centres are established and why their existence is vital in the fight against multi-generational illiteracy. libraries play an important role in ensuring a woman’s right to education, in that libraries ensure the creation of safe spaces that allow women to realize their full potential through effective adult education programs (sciabica, 2014).the value of literacy should never be underestimated as it is a vital priority to a more functional existence.    existing literature indicates that the major barriers women face in their pursuit of education include family demands and the stress of upholding the family structure. women fulfil multiple roles; mother, wife, daughter, and worker within the economy. each of these roles affects their ability to improve their literacy. this is consistent with the findings of the study conducted by moyo (2014), which states that women cannot easily pursue adult education because priority is given to looking for food and meeting other basic requirements to sustain life and the well-being of their families. it is these basic requirements that outweigh their intellectual needs.   arguably, a woman with family challenges cannot afford education, especially where poverty and lack of supports are experienced. in nigeria, literacy rates vary from state to state and from urban to rural areas, but what is clear is that there is a gender gap. according to the national bureau of statistics (nbs) (2010), the overall literacy (in any language) was 71.6%; 79.3% for males and 63.7% for women. the report also shows that a desire to be literate exists with 47.4%of illiterate persons (both genders). the challenge is in creating awareness of the availability of programs. only 32.5%of surveyed adults were aware of literacy centres and the services they provide (nbs, 2010).   adult education for women should be made more accessible. this calls for the restructuring of processes and the need for assessment of resources in the adult education centres in order to ensure willingness and effective learning experiences for women. it also demonstrates a need to improve general awareness of program availability throughout nigeria. libraries are essential in achieving this, as they provide useful services to the community by fostering education and disseminating information to all sections of the society (laila, 2010). as partners in the education system and as an empowerment hub, libraries have applied strategies in integrating women in adult reading as well as providing night reading opportunities (farrell, 2016; katz, 2012). information and communication technology (ict) has also been integrated into library services and other learning activities. women in adult education should have access to adequate representation of ict-based resources, equipped libraries, and other instructional media resources (mamman, 2015; jiyane & mostert, 2010; oyelude & bamigbola, 2012). the simple reason, according to osuchukwu (2016), is that ict has come to stay and has become “a must-have” if the society and the people therein do not want to be left behind in a fast-changing world. access to ict will ensure that no woman is left behind in the ever-changing world.   studies have recorded satisfaction of women participating in adult education programs. mbah (2018) in enugu state, nigeria found that women whose formal education was stopped at the primary level were satisfied with their adult education experiences and their new literacy levels. even a limited increase in a woman’s education improves her knowledge, boosts her wages, and enhances social inclusion, all of which have the potential to foster economic empowerment, better livelihood, and social development (egenti & omoruyi, 2011). education can provide a woman with the choices she needs to improve her overall health and to participate and contribute to a country’s sustainable development (organization for economic cooperation and development, 2008; moyo, 2014).   aims of the study   this study seeks to gain an understanding of women’s experiences of attending adult education programs in onitsha, anambra state, nigeria. the study seeks to explore the resources available for the program, the challenges women face in seeking to further their educational goals, and their perceptions, as well as the satisfaction derived from the adult education program.   research questions   the following research questions were asked:   what challenges do women face in acquiring education in the adult learning centres? what are their perceptions of the challenges they encounter? which resources do they use in enhancing learning in the adult education centres? are they satisfied with the learning taking place at the centres?   methods   this is a descriptive survey research design carried out in five adult education centres in two local government areas (lgas) of onitsha north and south metropolis in anambra state, nigeria. random sampling was employed to select five adult education centres in the lgas: holy trinity adult education center, holy spirit adult education, sacred heart adult education, st. john adult school and st. john college. two centres were selected from onitsha north and three from onitsha south. all centres are government approved and use the government stipulated curriculum for education in nigeria. the qualitative data were obtained through a self-designed questionnaire designed for this study by the authors. the copies of questionnaires were randomly distributed to women enrolled in the adult education classes at each of the five centres. in total, 120 copies of the questionnaire were distributed, 24 copies in each centre. questionnaires were collected back each night, thus making a 100% rate of duly filled completed and returned questionnaires. the questionnaires were distributed over a two week period to enable the physical distribution of questionnaires to the women and the assessment of facilities in the adult education centres.   interviews were also carried out. the researchers randomly selected two women class captains in each centre as participants. this was done to validate the claims in the questionnaire. also used was an observation checklist, which was self-designed by the authors of this study to assess the availability of facilities and resources for evidence based research of the real situations of the adult education centres. results were collated and analyzed using percentages, frequency counts, and figures. a benchmark of 50% was taken for responses rated on percentages.   results   a summary of the results of the study are shown below.   table 1 biodata of women participants in the adult education centres understudy s/n age f % class f % occupation f % 1 below 18 24 20 primary 24 20 trading 42 35 2 19-29 36 30 junior secondary school (jss) 36 30 stay at home mums 12 10 3 30-40 42 35 senior secondary school (sss) 60 50 artisan 36 30 4 50 & above 18 15       house help 18 15 5             public servant 12 10   total 120 100   120 100   120 100     table 2 observation checklist of the resources/facilities seen in all the five education centres s/n items holy trinity adult education center sacred heart adult education holy spirit adult education st john adult school st john college 1 library x x x x x 2 board ✓ ✓ ✓ ✓ ✓ 3 classroom ✓ ✓ ✓ ✓ ✓ 4 chairs and desks ✓ ✓ ✓ ✓ ✓ 5 instructional aids x x x x x 6 computer room x x x x x     the analysis in table 1 shows that the highest number of adult education students fall between the ages of 30 and 40 years (35%). this is followed by the age group of 19-29 years (30%) and below 18 years (20%), while 50 years and above represents 15%. it was also shown that 50% of the students are in the senior secondary class, while others are at the junior secondary class (30%) and primary school (20%) credentials levels. the cross sections of occupations identified by the women are traders (35%) and artisans (30%). others identified as stay-at-home mothers, household helpers, and public servants. the results demonstrate that women in every sector seek education and are determined to upgrade their literacy levels in the information society.   challenges women face in acquiring education in the adult learning centres   a majority of the women (75%, n= 90) indicated that they faced challenges during their time pursuing adult education, while 25% (n= 30) stated they had not faced any barriers. of the challenges experienced, the participants identified the following: stress (79.16%), family demands (58%), financial issues (54.16%), lack of resources to enhance learning (50%), duration of classes and time constraints (35%), and poor learning environments (20.83%).   the interviews allowed for further exploration of these challenges. one class captain revealed that some women and girls, especially the household helpers, do not pay school fees and other dues until they are stopped from attending school by the management. also disclosed was that some women drop out of school due to family demands and stress.   table 2 disclosed the researchers’ physical assessment of the five adult education centres. the observation of the adult education centres revealed that the students in all adult education centres have access to classroom, board, and chairs and desks, but cannot access the school library and the computer room where media instructional resources are housed. students in adult education centres are denied access to these facilities. the researchers also observed that the chairs and desks were chained, making sitting arrangements uncomfortable for reading and learning.   the interview process allowed for the exploration into the restriction of resources, the reasoning for it, and student perceptions. when asked about the reasons for chained chairs and desks as well as restricted access to school libraries and computer rooms, one teacher stated that classroom environments and facility access is determined by the management of the day school section. management dictates the availability of facilities for the adult learning program. when asked about the classroom environment and the restrictions, the students commented that not having access to the library and computer rooms negatively impacted their education. they indicated that access would have helped them in acquiring a higher level of knowledge, since many of them have never used a computer or entered a library.   when asked to comment on the chained chairs and desks, the women complained that they make sitting very uncomfortable, affecting their attention span because their legs were cramped and relaxation could not be achieved in the classroom. the women also added that the young students in the morning sections probably chained their chairs to their desks to avoid people moving them to other classes and for fear of loss. the adult education program has no power over the chairs. they only access what they see when they come to school in the evening.   satisfaction with the learning taking place at the adult education centres   the women were asked to evaluate their overall satisfaction with their educational experience, specifically their acquisition of reading and writing skills. overall, 75% of participants were satisfied with their experiences in the adult education program. the 25% of the participants who expressed dissatisfaction with the adult education program attributed it to lack of access to educational resources in the centres. thus, the satisfaction suggests that the basic acquisition of knowledge and literacy mean a great deal to the women in spite of all the challenges encountered in the adult education program.   perceptions of the challenges women encounter in the adult education centres   half of the women (50%) indicated that they expected to encounter challenges while participating in the adult education program because of added responsibilities as adults. another 35% described themselves as adjusting, while 15% disclosed that they did not expect to encounter challenges before they started and that they are coping very well.   the interviews allowed for the participants to indicate the specific challenges they faced. interviewees revealed that they felt overwhelmed and were exhausted when trying to find work-life balance. the stress experienced by the women related to the management of busy schedules in pursuit of their daily economic activities, while seeking opportunities for higher education. their desire for ongoing education far outweighed their need for rest; hence, the desire to continue to achieve their goals. the interviews also revealed that those who were coping well had already possessed the opinion that education was the only way to improve their social and economic status.   discussion   the age differences of women in the adult education centres indicate that women of all ages still seek basic education opportunities to live functionally. the number of women enrolled in senior secondary school classes (50%) indicates the women’s level of determination to upgrade their literacy levels and to improve their chances of having a voice in decisions that affect them. besides, the fact that women traders, artisans, stay-at-home mothers, household helpers, and public servants struggle to acquire education shows that women in every facet of life have come to realize that they have more to offer in their homes and societies when equipped with education. this result is consistent with existing studies exploring adult education and the livelihood of women as agents of change. studies conducted by both the millennium challenge corporation (2016) and egenti and omoruyi (2011) demonstrate that basic literacy skills help women to acquire both the knowledge and skills required to improve and develop in all fields, as well as empower them socially and politically.   the study shows that many women face challenges, while others report no challenges (75% affirmative and 25% negative). the responses are likely to vary, as some women are still young mothers with children and husbands and multiple demands that divide their time, especially in the evenings when lessons take place. in addition to the challenges presented by family demands, other challenges including the time and duration of classes, financial issues, lack of support at home, and the stress associated with housekeeping and employment were mentioned as the leading reason for them to drop out, which the interviews confirmed. these results affirm the findings that the roles women play as wives, mothers, learners, community members, and workers could seriously affect learning for real academic achievement and adult development (harp, 2010).   the confirmation that the adult education participants have classrooms, desks, and chairs in the local government areas studied indicate availability of basic learning facilities. the fact that they do not have access to the school libraries and ict rooms located in their learning centres shows an uncomfortable gap in today’s learning experiences, which sustainable development goal (sdg) 4 emphasized for inclusion for equitable quality education. this opposes the sdg’s mantra of “leave no one behind” in today’s learning resources in information society, which is characterized as one of the prerequisites for a 21st century learner (osuchukwu, 2016).   chained chairs and desks affect sitting positions and can result in ineffective learning. it takes an enabling environment for desired learning to take place. the interviews confirmed this result and highlighted the fact that adult evening school students are restricted in their use of the facilities as dictated by the morning school staff. these restrictions could have impacted the women’s satisfaction with their learning experience (25% not satisfied). some of the women stated that they have never touched a computer nor been in a library, but would love to access these resources, potentially impacting their satisfaction levels. the implication is that the developmental benefits of education and information, which could have added values to their learning, will elude them in achieving all around quality education as sdg4 proposes. the theory on women’s adult learning needs to be expanded to consider gender sensitivity in the planning of curriculum and programs in order to make it more accessible and satisfactory. hence, adjusting resources for learning can help facilitate learning and active participation and reduce the incidences of overwhelming situations.   the responses on the perception of challenges women face in adult learning centres established the fact that missing the opportunity of basic education at a prime age can pose serious difficulties in trying to regain that opportunity as an adult. nonetheless, the women still derive satisfaction and broaden their knowledge through adult education programs. this participation shows that all hope is not lost for these women. they can still attain their goals and improve their lives through adult education.   conclusion   the need to acquire basic education and access to information acts as a bridge from misery to hope. being literate in today’s information society will help in improving knowledge and eliminating social exclusion. it is critical that women are supported to seek education irrespective of the difficulties in order to be part of sustainable development. literacy will enable them to have a voice, participate, and contribute as first class citizens in decision-making processes. hence, this study discovered that:   many women participate in adult education programs, from the stay-at-home mothers to the traders, and from various age brackets. even though some of them are still in the primary level, they all look forward to achieving success and writing certificate examinations. many women face undeniable challenges, which include family demands, stress, finances, time, and the duration of classes, among others. although the availability of classrooms, chairs, and desks is appreciated, the resources for learning should include accessibility of library services and ict lessons, which these women agreed would add value to their learning. the majority of women anticipated that they would encounter challenges, but they persevered despite these overwhelming challenges because they knew it would improve their social and political standing in society.   implication of the study to library and information services   when women cannot attain basic education, marginalization and social exclusion continue to relegate them to the background, perpetuating the gendered economic and educational gap for women and their children. this gap can be closed through the various stimulating learning experiences library engagement can offer. this study found that adult women learners desire access to libraries, with their resources and technology, in order to enhance their education and their educational experience. this study revealed that not having access to these resources affected the women’s perception of the adult learning programs. enhancing education with proper learning resources will improve learning. therefore, library and information services should be extended to the women in adult education centres to create the right attitude for active involvement in society and the economy.   recommendations   based on the findings, the following recommendations were made:   the government, policy makers, and other stakeholders in education should regularly assess the adult education program for improvement and inclusion of all necessary resources. there is a need for women’s groups like national association of women in academics (nawacs), association of women librarians in nigeria (awlin), and other highly respected groups to form partnerships with ngos and agencies for the purposes of empowering women and advocating for better learning experiences for them. it is important that libraries and other information centres extend their services to the adult learning centres for successful integration of equitable quality education. more needs analysis should be carried out in order to come up with solutions that are relevant to the needs of women in adult education programs in order to guard against unnecessary difficulties in participating.   references   alexander, j. & robinson, j. p. (2014, 7 march). education for women and girls is progress for all on international women’s day [blog post]. retrieved 2 november 2018 from https://www.brookings.edu/blog/education-plus-development/2014/03/07/education-for-women-and-girls-is-progress-for-all-on-international-womens-day/   cleveland clinic. (2018). women: work, home, multiple roles, & stress. retrieved 7 november 2018 from https://my.clevelandclinic.org/health/articles/5545  egenti, m. n. & omoruyi, p. f. (2011). challenges of women participation in continuing higher education programme: implications for adult women counselling and education. edo journal of counselling 4(1&2):130-143. retrieved from https://www.ajol.info/index.php/ejc/article/view/72730   farrell, l. (2016). the importance of education for women and girls. retrieved 3 march 2018 from http://farmingtonlibraries.org/the-importance-of-education-for-women-and-girls/   gillard, j. (2014). #girlsedu: the value of girls’ education in 2014 and beyond. retrieved 21 february 2018 from https://www.brookings.edu/blog/education-plus-development/2014/06/10/girlsedu-the-value-of-girls-education-in-2014-and-beyond/   harp, n. f. (2010). women’s issues in adult education [blog post]. retrieved 5 november 2018 from http://sites.psu.edu/natalieharp/writings/womens-issues-in-adult-education/   jiyane, v. & mostert, j. (2010). use of information and communication technologies by women hawkers and vendors in south africa. african journal of library archive and information science 20(1), 53-61. retrieved from https://www.researchgate.net/publication/285865114_use_of_information_and_communication_technologies_by_women_hawkers_and_vendors_in_south_africa   katz, a. (2012). how libraries empower women, strengthen healthcare, and bridge the education gap. retrieved 3 march 2017 from https://www.impatientoptimists.org/posts/2012/08/how-libraries-empower-women-strengthen-healthcare-and-bridge-the-education-gap#.xik1vchki70   kelleher, f. (2014). the literacy injustice: 493 million women still can't read. the guardian. retrieved 7 november 2018 from  https://www.theguardian.com/global-development-professionals-network/2014/jun/17/literacy-women-illiteracy-development   kwapong, o. t. (2005). training and empowerment/using adult education for empowerment of rural women. journal aed-adult education and development. edition 65. retrieved 9 november 2017 from https://www.dvv-international.de/en/adult-education-and-development/editions/   laila, t. a. (2010). role of public libraries in non-formal education in kerala. kelpro bulletin, 14(2), 60-67. retrieved from http://eprints.rclis.org/15208/   mamman, e. s. (2015). utilization of information and communication technologies (icts) in public library services in nigeria (unpublished doctoral dissertation). university of nigeria, nsukka, nigeria.   mbah, c. (2018). adult literacy/basic education for the adult women. retrieved 3 march 2017 from http://decnigeria.com/dp.php?view=3   millennium challenge corporation (mcc). (2016). gender and social inclusion. retrieved 6 november 2018 from https://www.mcc.gov/initiatives/initiative/gender   moyo, c. (2014). is adult basic education the answer to rural women’s poverty? journal of human ecology, 45(2), 111-118. https://doi.org/10.1080/09709274.2014.11906684   national bureau of statistics (2010). the national literacy survey. retrieved 7 november 2018 from http://www.nigerianstat.gov.ng   nhamo, s. & nhamo, g. (2006). macroeconomics, adult education, and poverty eradication in southern africa. international review of education, 52(3-4), 305-322.   nyagoha, d. (2016). factors influencing access to adult and continuing education programmes in kenya: case of bumula sub county, bungoma county (master’s thesis, university of nairobi, kenya). retrieved from http://erepository.uonbi.ac.ke/   organization for economic cooperation and development (oecd). (2008). gender and sustainable development: maximizing the economic, social and environmental role of women. retrieved 7 november 2018 from https://www.oecd.org/social/40881538.pdf   osuchukwu, n. p. (2016). ict and school media centre: platforms for sustainable quality education. paper presented at a two day workshop for teacher-librarians in all secondary schools in anambra state, nigeria at post primary schools service commission (ppssc) headquarters, awka, 2-3rd february.   oyelude, a. a. & bamigbola, a. a. (2012). women empowerment through access to information (ati): the strategic roles of non-governmental organizations in nigeria. paper presented at the world library and information congress/78th ifla general congress and assembly at helsinki.   sanchez moretti, g. a. & frandell, t. (2013). literacy from a right to education perspective (unesco document 221427). retrieved 8 november 2018 from https://unesdoc.unesco.org/ark:/48223/pf0000221427   sciabica, t. (2014, 1 december). the case for a safe space: libraries empower women in asia [blog post]. retrieved 7 november 2018 from https://www.impatientoptimists.org/posts/2014/12/the-case-for-a-safe-space-libraries-empower-women-in-asia#.xik9gchki70   united nations (2015). transforming our world: the 2030 agenda for sustainable development. retrieved 8 november, 2018 from https://sustainabledevelopment.un.org/post2015/transformingourworld/publication   wanyama, i. k. (2014). challenges facing the sustainability of adult and continuing education programmes in kenya. mediterranean journal of social sciences, 5(5), 159-166. https://doi.org/10.5901/mjss.2014.v5n5p159 microsoft word news_glynn.doc evidence based library and information practice 2006, 1:2  57  evidence based library and information practice     news    4th international conference on evidence based library and information practice          © 2006 eblip. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    the university of north carolina will host  the 4th international conference on evidence  based library and information practice on  may 6‐9, 2007, in chapel hill, nc.  the  conference will be followed by two days of  continuing education.      topics to be addressed include building the  knowledge base of the profession, various  roles in promoting evidence based practice,  setting evidence based practice standards,  and key practice areas requiring systematic  review of evidence.  workshops include  how to apply the evidence based model of  practice in different settings and how to  assess library literature.        further information, including the official  press release, is available on the conference  web site at: http://www.eblip4.unc.edu/                  http://creativecommons.org/licenses/by/2.0 http://www.eblip4.unc.edu/ research article   using information practices of nurses to reform information literacy instruction in baccalaureate nursing programs   anne r. diekema assistant professor, department chair, and instruction librarian gerald r. sherratt library southern utah university cedar city, utah, united states of america email: annediekema@suu.edu   elizabeth (betsy) s. hopkins nursing and communication disorders librarian harold b. lee library  brigham young university provo, utah, united states of america email: betsy_hopkins@byu.edu   brandon patterson technology engagement librarian spencer s. eccles health sciences library university of utah salt lake city, utah, united states of america email: b.patterson@utah.edu   nena schvaneveldt assistant librarian, education librarian spencer s. eccles health sciences library university of utah salt lake city, utah, united states of america email: nena.schvanevelt@utah.edu   received: 30 may 2019                                                                  accepted: 12 oct. 2019      2019 diekema, hopkins, patterson, and schvaneveldt. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29588     abstract   objective seeking information is a key element of evidence based practice and successful healthcare delivery. significant literature exists on both the information seeking behaviour of professional nurses and information literacy teaching methods, but scarce evidence connects nurses’ information behaviour and environments with their education. this study sought to use data from nursing alumni to answer the following research questions: what are the current information practices of professional bachelor’s-prepared nurses? how do recently-graduated nurses suggest that their education could have better prepared them to find and evaluate information in the workplace?   methods the researchers conducted a descriptive study using a 59-item survey instrument with a variety of question formats including short-answer, multiple choice, likert, and open response. the researchers distributed the survey to baccalaureate nursing alumni who graduated in 2012-2017 from four universities in the state of utah in the united states.   results nurses seek practical information primarily to provide informed patient care, while also clarifying medical situations and expanding their health care knowledge. they frequently consult nursing colleagues and physicians when seeking information. the majority of nurses consult electronic health records daily. respondents described time as the biggest barrier to accessing information. they requested authentic, clinically-focused scenarios, training on freely-accessible resources, and more explicit teaching of lifelong learning skills, such as critical thinking.   conclusion information literacy education should prepare student nurses for the fast-paced information environment they will face in the workplace. this means incorporating more patient-focused scenarios, freely available quality resources, and time-based activities in their education. the researchers suggest areas to prepare nurses for information seeking, including problem-based clinical scenarios, building guides with databases accessible for free or little cost, and added emphasis on critical thinking and self-motivated learning.     introduction   preparing nurses for evidence based practice involves incorporating clinical expertise, the patient’s preferences, and the “best available external evidence” (sackett, rosenberg, gray, haynes, & richardson, 1996, p. 72). perhaps the most difficult challenge in incorporating evidence into practice is the exponential growth of health care knowledge. an increase in published knowledge means more information to sort through, and when nurses cannot find reliable information quickly, evidence based patient care is diminished (clarke et al., 2013). information literacy is a critical part of evidence based practice in a progressively more complex information environment, as it provides a framework to manage, retrieve, evaluate, and use information effectively. the association for college & research libraries (2015) defines information literacy as “the set of integrated abilities encompassing the reflective discovery of information, the understanding of how information is produced and valued, and the use of information in creating new knowledge and participating ethically in communities of learning.” not surprisingly, the american association of colleges of nursing (2008) recognizes information literacy as a component of liberal education of baccalaureate-level nurses. given the great importance of nurses’ ability to obtain evidence based information, and a lack of research connecting information behaviour with information literacy instruction, librarians from four universities in the state of utah in the united states aimed to better understand information practices of professional nurses to improve information literacy instruction in baccalaureate nursing programs. the researchers wanted to place instruction in the context of evidence based practice to prepare nursing students for professional information seeking.   literature review   prior to conducting the study, the researchers reviewed the literature for recent data on information literacy education of nursing students, as well as the information practices of nurses.   information literacy is being taught to nursing students with varying levels of success. while information literacy is important, it is also difficult to teach and assess effectively, and at times outcomes are only moderately improved (farrell, goosney, & hutchens, 2013; moreton & conklin, 2015). according to a survey of instruction librarians, information literacy programs in higher education are often limited to first year instruction, although discipline-specific instruction for undergraduates is also available (julien, gross, & latham, 2018). the standards and frameworks available to librarians to teach information literacy fall short in teaching clinical information seeking, as they fail to capture the social and transformative aspects of information literacy (lupton & bruce, 2010). for example, the standards for nursing appear to consider extensive library research as the norm while practicing nurses often have more immediate information needs that require quick turnaround times. further, nursing students fail to see the connection between evidence based practice and their clinical experience (aglen, 2016). one way to reinforce information literacy skills is to incorporate them throughout the curriculum as it may better prepare students for professional practice (flood, gasiewicz, & delpier, 2010; phelps & hyde, 2018). nursing students are eager to gain more information on patient care, database training, and computer skills, especially the primary database in nursing, cinahl (dee & stanley, 2005; duncan & holtslander, 2012).   as nursing students transition to work in a clinical setting, they mimic the practices they will likely take on as practicing nurses namely, they rely heavily on peers and electronic materials, although newer nurses tend to rely more heavily on print resources (dee & stanley, 2005; wahoush & banfield, 2013). nurses seek information most frequently for patient care and in novel situations, such as a patient presenting with an atypical symptom (newman & doran, 2012; o’leary & ni mhaolrúnaigh, 2012; younger, 2010). practicing nurses’ reliance on colleagues and general internet searches as their first and second sources of information, respectively, are well documented in the literature (alving, christensen, & thrysøe, 2018; argyri, kostagiolas, & diomidous, 2014; baro & ebhomeya, 2013; clarke et al., 2013; majid et al., 2011; marshall, west, & aitken, 2011; newman & doran, 2012; o’leary & ni mhaolrúnaigh, 2012). familiarity and proximity may play a large role in these preferences: using proprietary databases is time consuming and requires access and training, whereas speaking with colleagues is a normal occurrence (gilmour, huntington, broadbent, strong, & hawkins, 2012; marshall et al., 2011).   however, nurses and nursing students experience several barriers and frustrations to seeking and accessing information: a lack of computer skills or, more specifically, database searching skills (argyri et al., 2014; dee & stanley, 2005; intas et al., 2016); frustration with choosing the right words or phrases to query information resources (duncan & holtslander, 2012); lack of access to computers or internet connections (raj, sharma, singh, & goel, 2015; sarbaz, kimiafar, sheikhtaheri, taherzadeh, & eslami, 2016); and resistance to evidence based practice (kahouei, babamohamadi, panahi, & zadeh, 2013). nurses and students need better training using mobile technology, including apps, in clinical, classroom, and laboratory settings (baro & ebhomeya, 2013; majid et al., 2011; raman, 2015). a lack of time was the barrier most frequently reported by nurses (argyri et al., 2014; baro & ebhomeya, 2013; kumaran & chipanshi, 2015; majid et al., 2011). these barriers are not unique to nursing; they are commonly reported in literature for many health sciences professions. lack of knowledge and time are compounding barriers: when a nurse is ignorant about or resistant to evidence based practice, they are unlikely to seek evidence to improve their practice. a lack of time would also make learning new skills or technologies more difficult. if a nurse fails to see the connection between accessing information and patient care, they are less likely to invest the time to change their thinking. finally, when searching for evidence is not built into the professional nursing workflow, searching falls by the wayside (alving et al., 2018).   no study connected information seeking behaviour to the teaching of information literacy skills explicitly by asking nurses how well their nursing education prepared them for the professional information environment.   aims   in an effort to improve information literacy education of nursing students, this study answers the following research questions: what are the current information practices of professional bachelor’s-prepared nurses? how do recently-graduated nurses suggest that their education could have better prepared them to find and evaluate information in the workplace?   methods   study concepts and scope   for the purposes of this study, nurses are defined as practicing registered nurses (rns) with at least a bachelor’s degree in nursing (bsn). the authors defined information literacy instruction as teaching provided by academic librarians designed to teach students how to find, evaluate, and use information resources; however, understanding that nurses learn information literacy skills from non-librarians, the researchers included a question (see question 53 in appendix a) to elucidate the importance of a variety of sources in learning these skills.   the study was limited to surveying the perceptions of nursing alumni who 1) graduated with baccalaureate degrees in nursing (i.e. bachelor of science in nursing) between 2012-2017 from one of four utah-based institutions, and 2) were employed as a nurse while participating in the study. the researchers elected to do a cross-institutional study in order to study patterns of nurses educated statewide, as well as to increase the sample size. the four represented institutions include two public (southern utah university and the university of utah) and two private universities (brigham young university and roseman university of health sciences), two of which are small (i.e., an enrollment of fewer than 10,000 students) and two are large (i.e., enroll at least 20,000 students).   data collection and analysis   a 59-item survey instrument was developed, tested, and administered using qualtrics to capture data from the large group of nursing alumni (see appendix a). the researchers chose the survey as a data collection tool to reach a large number of geographically dispersed respondents with a set of standardized questions. survey question topics and themes were informed by questionnaire-based research studies on nurses’ and nursing students’ information behaviour (argyri, kostagiolas, diomidous, 2014; baro & ebhomeya, 2013; intas et al., 2016; kumaran & chipanshi, 2015; majid et al., 2011; o’leary & ni mhaolrúnaigh, 2012; sarbaz,et al., 2016; wahoush & banfield, 2014), specifically those studies which included their survey instruments (dee & stanley, 2005; farrell, goosney, & hutchens, 2013; pravikoff, tanner, & pierce, 2005). the survey questions, in a variety of formats including multiple choice and open-ended questions, were pre-tested and revised where necessary to prevent misinterpretation and bias. the survey was sent to librarians, nursing faculty, and practicing nurses, and testers’ feedback was incorporated into the survey prior to its deployment. the survey consisted of eight sections: introduction and consent; demographics and current employment situation; information needs; information sources; information environment at workplace; barriers to finding, accessing, and evaluating information; education & instruction; and exiting the survey. approval of the study was obtained from all four institutional review boards.   an email from each nursing dean, chair, or alumni office was sent to their respective nursing alumni asking them to participate in the upcoming survey. within a week of this message, each nursing program graduate received an email link to the survey, which opened on november 13, 2017. a total of 1,926 invitations were distributed. alumni received up to four reminders if their survey response had not been received within a specific timeframe. respondents who did not graduate from one of the four institutions (e.g., because they were given the survey link through a forwarded email) or did not currently work as a nurse were routed to exit the survey. no questions were required, so response rates fluctuated from question to question. after completing the survey, respondents had the option to leave the survey and go to another page to leave their contact information to enter one of ten $20 amazon gift cards. gift cards were funded by a research award from the nursing and allied health resources section of the medical library association.   preceding data analysis, all identifying information including names and contact information were removed to ensure anonymity of respondents. the results of the open-ended questions were analyzed for emergent themes using the qualitative method of analytical coding, where codes emerged from the data (richards, 2009). researchers did an initial round of coding for all responses to an open-ended question. in a second pass through these responses, more detailed codes were added and codes were normalized where needed. in a final round of coding, the codes were grouped into themes.   results   what are the information practices of professional bachelor’s-prepared nurses?   to answer the first research question, the researchers examined demographics of the group, reasons nurses seek information, source types used, and information access and barriers.   demographics   out of the 1,926 total invitations sent, 349 respondents completed the survey, resulting in a response rate of 18.1%. most respondents graduated within two years of the study with at least half having two or more years of employment as a nurse. ninety-six percent of respondents’ highest degree completed was a bachelor’s in nursing and 14% were currently enrolled in a graduate nursing degree program (see table 1). while this study focused on nurses trained at a baccalaureate level, fourteen respondents earned a master’s degree, doctorate or other degree. it is likely that they earned these other degrees in the time since completing their bsn. these nurses, who only made up only 4% of all respondents, were included in the analysis as it did not appear education level made for drastically different responses. eighty-eight percent of respondents worked in a hospital setting of some kind. other settings, such as dialysis, endoscopy clinics, and home health agencies were also represented. most respondents described their work setting as critical care nursing or medical-surgical nursing. other specialties, such as pediatrics, oncology, endoscopy, hospice, and emergency nursing were also common. a majority of respondents are located in the mountain west (66%), though there were a significant number from other states across the united states (33%) and two international respondents.   table 1 demographic characteristics of graduated nurses who responded to survey   n percent nurses completed survey 349 --------- graduation year         2012 36 10.32     2013 60 17.19     2014 58 16.62     2015 74 21.20     2016 81 23.21     2017 39 11.17 years of employment as a nurse         0 39 11.17     1 83 23.78     2 75 21.49     3 71 20.34     4+ 81 23.21 highest nursing degree obtained         bachelor’s 334 95.70     master’s 8 02.29     doctorate 5 01.43     other 1 00.28   reasons nurses seek information   the top reasons survey respondents reported seeking information were: (a) “to answer patient questions” (85%), (b) “to determine best practices for patient care” (70%), (c) “to consult clinical practice guidelines” (70%), (d) “to check clinical procedures” (70%), and (e) “to interpret laboratory data” (67%).   respondents answered an open-ended question about how they used information to move beyond a recent moment at work where they struggled to make sense of a situation. from 131 responses to the question, the researchers identified seven core themes. these were: seeking information to help or educate patients (40%), clarifying medical situations (19%), expanding health care knowledge (13%), verifying medications (11%), clarifying hospital safety and policy procedures (8%), checking measurements, data, and labs (5%), and clarifying treatment plans (4%).   source types   only two print sources were used at least daily by more than 10% of the nurse participants: policy & procedures manuals (12%) and paper fact sheets / handouts (15%). when asked about information sources, respondents reported most frequently using electronic health records, which may include links to external information sources, with daily usage reported in almost three out of four cases (72%). far fewer use general search engines daily (39%) and websites with medical information (23%). respondents reported a number of other electronic sources, including mobile apps (e.g., uptodate, lexicomp, micromedex, and epocrates), specific hospital resources (e.g., hospital handbooks, policies, and/or guidelines), and other websites (e.g., center for disease control (cdc), association of perioperative registered nurses (aorn), associated regional and university pathologists (arup)). as interpersonal sources, nursing colleagues and physicians are used most frequently, with at least daily use for 70% and 62% of respondents, respectively. table 2 shows survey respondent data for selected sources based on a 6-point likert scale on frequency of electronic and interpersonal source use.   table 2 selected sources based on frequency of use values: 1 = never, 2 = rarely, 3 = monthly, 4 = weekly, 5 = daily, 6 = multiple times a day grouping source type 1 2 3 4 5 6 electronic                 emr 13 11 24 43 64 166   general search 21 30 53 92 75 48   uptodate 62 54 74 83 29 19   medical websites 21 46 73 107 55 17   pubmed 64 79 92 65 15 5   google scholar 114 77 71 38 18 1   cinahl 151 100 47 18 2 0   e-books 163 87 47 14 4 0 interpersonal                 nursing colleagues 8 13 24 53 103 112   physicians 3 13 31 77 117 83   pharmacists 7 25 43 101 94 53   allied health workers 29 55 62 79 71 27   social workers 47 71 65 77 42 21   medical librarians 214 71 21 12 0 0   access and barriers   a series of survey questions addressed access to information sources, as well as barriers that prevent efficient and effective access. several questions allowed respondents to check all options that apply; percentages add up to more than 100% in those cases. in total, 64% of respondents have access to information sources through their employers, either because the employers provided the sources themselves or could get them upon request. other sources included local libraries (21%), personal membership(s) in professional organizations or societies (17%), personal purchases (11%), and professional contacts outside of the workplace (10%). the great majority of nurses used work computers (89%) and personal phones (58%) to access clinical information at work.   respondents reported that they needed information in a number of clinical environments, including team meetings (48%), clinical rounds (45%), but mostly chairside or bedside (86%). respondents are generally able to access information where they need it, reporting access as follows: team meetings (46%), clinical rounds (38%), and chairside or bedside (78%). some of the other situations where respondents said they needed information included educating other nurses, during phone calls with patients, or when verifying the purpose of prescribed medications.   most respondents (59%) reported that time constraints were a moderate or substantial barrier to accessing information. other moderate to substantial barriers included lack of training on using information sources (24%), cost (21%), lack of expertise or training in accessing information (19%), lack of access to technology (12%) and lack of training in technology (13%). in the open-ended responses, time and access emerged as the most common barriers to seeking information. for example, one respondent stated: “on our unit, time constraints are a huge barrier.” this response was typical of the comments on access: “my worksite does not provide access to databases which require subscriptions, so i find myself mainly utilizing free resources such as pubmed [sic] and google scholar [sic].”   how do recently-graduated nurses suggest that their education could have better prepared them to find and evaluate information in the workplace?   to answer the second research question, the researchers looked at respondents’ answers to how they find and evaluate information generally, how they do this in the course of their work, confidence level in search and evaluation skills, and how their schooling could have better prepared them for information seeking at their workplace.   how nurses learned to find and evaluate information   the researchers asked respondents to rank the importance of where they learned to find information on a five-point scale (important, somewhat important, neutral, somewhat unimportant, unimportant). ninety-three percent of respondents ranked self-taught skills as either important or somewhat important. similarly, on-the-job training was deemed at least somewhat important by 88% of respondents. nursing school faculty also had a strong impact on teaching respondents to find information, with 84% reporting that it was important or somewhat important, and nursing school librarians were found important by 53% of respondents.   finding and evaluating information at work   when nurses need help searching for information at work, proximity seemed to be a major factor. eighty-nine percent of the respondents answering this question consulted their nursing colleagues within the workplace, while 59% consulted other healthcare colleagues in their workplace. similarly, respondents sought help for evaluating sources from their immediate nursing colleagues (90%), or from other healthcare professionals in their workplace (63%).   confidence in search and evaluation skills   a series of questions asked respondents to evaluate their confidence in finding, evaluating, sharing and explaining information based on a five-point scale (completely confident=4, very confident=3, moderately confident=2, somewhat confident=1, not at all confident=0). when averaging the values, most were moderately confident in their ability to share information with others (x̄=3.26), evaluate the quality of the information (x̄=3.21), find information (x̄=3.16) and explain information needs (x̄=3.07). interestingly, respondents’ confidence levels declined with their ability to understand the issues surrounding information, such as ethical use and copyright (x̄=2.70).   how nursing education could have better prepared nurses for information seeking   when the survey asked how their education could have better prepared respondents for information seeking, respondents provided a wealth of information. of the 151 responses on this question, 17 had no suggestions and didn’t explicitly mention whether the respondent felt prepared or not. of the 135 remaining responses, 50 respondents (37%) mentioned they were either satisfied or happy with their preparation in this area. a thematic analysis of all 91 responses with suggestions revealed suggestions for improvement in four different categories: specific information literacy skills (41%), teaching methodology (24%), resources (35%), and miscellaneous (16%). note that some responses fit multiple categories.   the researchers categorized the 37 suggested information literacy skills based on the acrl information literacy standards for nursing (association of college & research libraries, 2013): information need (3%), information access (62%), evaluation (24%), information use (27%), ethics and legal issues (0%). in addition, the researchers added a miscellaneous category (22%). information access skills were by far the largest category. many respondents requested instruction in resources outside of typical nursing and medical databases, specifically quality sources that are freely available on the internet. a representative comment was: “teach us more places to get free, good information.” on evaluation, respondents specifically wanted more practice, as well as more instruction on evaluating non-academic sources. respondents also suggested additional instruction on information use. some mentioned wanting instruction on reading and skimming research; others wanted more practice using sources to find answers and to solve problems. interestingly, no respondents suggested more preparation on using information legally and ethically, an area in which they reported having lower confidence. skills mentioned in the miscellaneous category were better knowledge of statistics, building critical thinking and lifelong learning skills.   over two-thirds of the 22 suggestions to improve approaches to teaching information skills spoke clearly for using real-life scenarios. for example, one respondent stated: “giving students a scenario "patient" with a complicated diagnosis/history/problem and having them look up evidence, even in a made-up evidence-based-practice manual of a made-up facility could be good practice.” respondents also suggested factoring in other real-life aspects such as time constraints and realistic assignments, like creating care plans. several comments recommended teaching information seeking and research skills in tandem with clinical experiences rather than in isolation in the classroom.   a third of comments about resources addressed the issue of losing access to subscription databases after graduation, as in this comment: “more in depth training of resources available outside of school.” others also stressed that they don’t have time to do actual research but need to have access to free online (24%) and vetted (18%) resources that allow them to quickly (15%) look up information they need. respondents also suggested teaching a variety of sources (18%) including proprietary and other commercial sources (15%) they would be likely to use when practicing: “more education on apps and tools on the internet that you can use, and also ones typically used by hospitals through the ehr.”   the remaining comments included suggestions for more and earlier instruction in the course of study (27%), more one-on-one time with the librarian (13%), and more organized and interesting library sessions (13%).   discussion   nurses’ information use patterns   the information use patterns of the surveyed nursing alumni were mostly in line with findings in the literature (newman & doran, 2012), with two exceptions. the use of print sources was much lower in the current study than previously reported by dee and stanley (2005). the drop in the use of print resources is likely explained by the fourteen-year time difference between the studies. in addition, the electronic health record as a frequently used source has not been reported elsewhere; this may reflect the recent adoption of electronic health records across healthcare systems.   because they are motivated to provide the best patient care, nurses use information to question orders, ensure procedures are correct, seek clarification on diagnoses and treatments, and answer patient questions. when they are unsure, nurses speak to another medical professional (likely a nurse) and/or look up the answer. nurses use readily available tools, including electronic medical records, local colleagues, and general internet searches. time is of the essence and a lack of access to information can create a frustrating barrier to success. though most of the nurses in this study had some access to databases through work, many wanted access to the same resources available during nursing school. to many, losing their information privilege (hare & evanson, 2018) was not something they had anticipated but a common frustration among college graduates (head, 2016). a recent hashtag on twitter, #nojournalsnoebp, has highlighted this situation for nurses and other health professionals (cogan, 2019).   to respond to these factors, educators should prepare nurses for this information environment and teach student nurses to leverage what sources are available. faculty and librarians should collaborate to ensure that librarian instruction is authentic and effective at building information literacy skills: subject faculty may not feel comfortable teaching skills and resources alone, while librarians may lack the critical subject context that  nursing faculty provide (brasley, 2008; saines et al., 2019). collaborative partnerships tend to ensure the best information literacy education for students (booth, lowe, tagge, & stone, 2015; varlejs & stec, 2014).   implications for information literacy instruction   in an ever-evolving clinical environment, it is important that nurses’ information skills match their patient-focused care preferences (forster, 2015). nursing students would benefit from developing critical thinking skills in order to quickly locate and evaluate information based on novel situations that prompt information seeking in their work.   figure 1. researchers’ suggested areas for preparing nurses for information seeking.   this study has significant implications for nursing information literacy instruction to facilitate knowledge transfer to the clinical setting. conforming to the problematic research assignments described in head and eisenberg (2010), information literacy education in nursing is often focused on subscription-based article databases and finding sources for research paper assignments that span several weeks. while some respondents indicated that research-focused coursework was valuable to learn the research process, this survey showed that this type of information seeking is rare in the workplace. nurses need to be prepared to access reliable information quickly during the course of their daily work. the researchers propose three new areas of emphasis: clinical scenarios, teaching beyond academic sources, and lifelong learning.   first, respondents indicated a desire for more active learning and clinical situations in information literacy sessions: they want instruction to mirror real life experience more closely. hands-on, active exercises lead to improved student learning compared to delivering information through lecture (maybee, bruce, lupton, & rebmann, 2017). faculty and librarians should collaborate on creating appropriate use cases and other clinical questions in assignments and instruction sessions. educators can incorporate time constraints into learning activities in order to simulate the clinical information environment. for example, the researchers suggest integrating timed exercises where students seek information in a few minutes, then report if what they found was sufficient to inform their work. while this is a classroom-based activity, it can also be performed in patient simulation exercises to help students transfer their information skills to a clinical environment.   second, teaching should not focus solely on traditional academic sources. nursing students need to know what sources will be available to them after graduation; instruction sessions near the end of the program and research guides for alumni will provide resources and guidance to new nurses. students should be familiar with freely-available resources that provide reliable information (e.g., pubmed, google scholar). in addition, information literacy skills need to be taught in a way that encourages transferability from the academic environment to the professional one (kuglitsch, 2015).   third, it is important to teach and stimulate critical thinking skills and the importance of lifelong learning. generic evaluation skills are often taught by librarians in general university curricula. nursing students need further instruction on evaluating information in their professional context (sleutel, bullion, & sullivan, 2018), especially because many of them turn to general internet searches instead of academic sources. in addition, self-taught skills and on-the-job mentoring were very important to nurses. educators can empower nursing students to teach themselves and each other when they encounter a new resource or interface. educators should also introduce nursing students to colleagues as potential information sources and increase awareness of helpful co-workers such as hospital librarians. this type of interpersonal information seeking should not be considered a least-effort approach but rather quality-driven, seeking out trusted experienced colleagues (xu, tan, & yang, 2006; hertzum, 2014).   implications for partnering with nursing faculty   as the findings show, nursing faculty are important in teaching students to both find and evaluate information. faculty subject expertise is essential to developing students’ skills and is complemented by librarians’ expertise in teaching information literacy. faculty wishing to implement more authentic information seeking into their curricula partner with librarians on their campus to deliver expert instruction in this area, especially targeting using information in clinical scenarios (haines & horrocks, 2006). because information-seeking behaviours are built and reinforced over time, librarian involvement in scaffolding information literacy throughout curricula is more effective and engaging than having librarians present only at orientation or in a research-focused course (johnson-grau, archambault, acosta & mclean, 2016).   librarians, while experts in information literacy, are not necessarily adept at developing authentic cases for either classroom or simulation instruction. the researchers encourage librarians to collaborate with nursing faculty to ensure their examples are specific and authentic to increase student engagement and learning (carder, willingham, & bibb, 2001). librarian involvement goes beyond research assignments; teaching students to seek information to inform care plans, answer patient questions, or resolve conflicting medical information would be useful in developing the skills the students will need in patient care. in involving nursing faculty in the creation of guides to curate resources in support of these activities, librarians can enhance the quality of their resources.   since interpersonal sources are often consulted, it is important to help students learn how to critically evaluate information they learn from colleagues (hertzum, 2014). bachelor’s-prepared nurses may be new to healthcare and orienting them to the wide variety of colleagues they will join, along with the information they can provide, is important. new nurses may be unaware that many hospitals have librarians who can assist with medical information searches (dee & stanley, 2005). educating new nurses about the availability of librarians who can perform expert searches rapidly can free up nurses’ time to provide patient care. the respondents had no problem contacting other health professionals for answers to their discipline-specific questions, e.g., asking pharmacists for drug information, medical laboratory technicians for questions about lab tests, and physicians for diagnostic and treatment information; however, a lack of awareness of librarians may hinder their ability to provide evidence based care.   study limitations   all participating nurses received their education in the state of utah and the majority were employed within the region. this limited geographic range might have affected the study results. it also should be noted that the response rate for this study was low, perhaps because busy nurses were intimidated by the estimated time commitment. in addition, the survey was long and not all participants completed all questions. regrettably, the survey did not specifically ask nurses about their use of online or mobile apps as an information resource.   conclusion   through a survey, the researchers elucidated nurses’ information-seeking behaviour and elicited suggestions for better preparation for finding and evaluating information. the results indicate discrepancies between the information environments of nursing school and professional nursing.  correspondingly, changes are necessary in nursing information literacy instruction to prepare nurses for the professional information environment. emphasizing clinical scenarios, teaching beyond traditional academic sources, and encouraging lifelong learning skills will ensure students learn relevant, transferable skills in authentic settings. lives depend on nurses delivering the best healthcare possible; it is imperative that librarians and educators enable nursing students to use information in the most effective way.   acknowledgements   this work was supported in part by an award from the nursing and allied health section of the medical library association. the authors acknowledge phil roché, erin wimmer, kyle o’sullivan, and the harold b. lee library assessment & usability department, specifically holt zaugg and brian roberts, and the nursing administration and alumni from our four institutions.   references   aglen, b. (2016). pedagogical strategies to teach bachelor students evidence-based practice: a systematic review. nurse education today, 36, 255–263. https://doi.org/10.1016/j.nedt.2015.08.025   alving, b. e., christiansen, j. b., & thrysøe, l. (2018). hospital nurses’ information retrieval behaviours in relation to evidence based nursing: a literature review. health information & libraries journal, 35(1), 3–23. https://doi.org/10.1111/hir.12204   american association of colleges of nursing. (2008). the essentials of baccalaureate education for professional nursing practice. retrieved from http://www.aacnnursing.org/portals/42/publications/baccessentials08.pdf   argyri, p., kostagiolas, p., & diomidous, m. (2014). a survey on information seeking behaviour of nurses at a private hospital in greece. studies in health technology and informatics, 202, 127-130. https://doi.org/10.3233/978-1-61499-423-7-127   association of college & research libraries. (2013). information literacy competency standards for nursing. retrieved from http://www.ala.org/acrl/standards/nursing   association of college & research libraries. (2015). framework for information literacy for higher education. retrieved from http://www.ala.org/acrl/standards/ilframework   baro, e. e., & ebhomeya, l. (2013). information needs and seeking behaviours of nurses: a survey of two hospitals in bayelsa state, nigeria. health education, 113(3), 183–195. https://doi.org/10.1108/09654281311309828       booth, c., lowe, m. s., tagge, n., & stone, s. m. (2015). degrees of impact: analyzing the effects of progressive librarian course collaborations on student performance. college & research libraries, 76(5), 623–651. https://doi.org/10.5860/crl.76.5.623     brasley, s. s. (2008). effective librarian and discipline faculty collaboration models for integrating information literacy into the fabric of an academic institution. new directions for teaching and learning, 2008(114), 71-88. https://doi.org/10.1002/tl.318   carder, l., willingham, p., & bibb, d. (2001). case-based, problem-based learning: information literacy for the real world. research strategies, 18(3), 181-190. https://doi.org/10.1016/s0734-3310(02)00087-3   clarke, m. a., belden, j. l., koopman, r. j., steege, l. m., moore, j. l., canfield, s. m., & kim, m. s. (2013). information needs and information-seeking behaviour analysis of primary care physicians and nurses: a literature review. health information and libraries journal, 30(3), 178–190. https://doi.org/10.1111/hir.12036   cogan, r. (2019, april 17). the relentless school nurse: #nojournalsnoebp update #3 – where the conversation has gone [blog post]. retrieved from https://relentlessschoolnurse.com/2019/04/17/the-relentless-school-nurse-nojournalsnoebp-update-3-where-the-conversation-has-gone/   dee, c., & stanley, e. e. (2005). information-seeking behavior of nursing students and clinical nurses: implications for health sciences librarians. journal of the medical library association, 93(2), 213–222. retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/pmc1082938/   duncan, v., & holtslander, l. (2012). utilizing grounded theory to explore the information-seeking behavior of senior nursing students. journal of the medical library association, 100(1), 20–27. https://doi.org/10.3163/1536-5050.100.1.005   farrell, a., goosney, j., & hutchens, k. (2013). evaluation of the effectiveness of course integrated library instruction in an undergraduate nursing program. journal of the canadian health libraries association / journal de l’association des bibliothèques de la santé du canada, 34(3), 164–171. https://doi.org/10.5596/c13-061   flood, l. s., gasiewicz, n., & delpier, t. (2010). integrating information literacy across a bsn curriculum. journal of nursing education, 49(2), 101–104. https://doi.org/10.3928/01484834-20091023-01   forster, m. (2015). six ways of experiencing information literacy in nursing: the findings of a phenomenographic study. nurse education today, 35(1), 195-200. https://doi.org/10.1016/j.nedt.2014.06.005   gilmour, j. a., huntington, a., broadbent, r., strong, a., & hawkins, m. (2012). nurses’ use of online health information in medical wards. journal of advanced nursing, 68(6), 1349–1358. https://doi.org/10.1111/j.1365-2648.2011.05845.x   head, a. j. & eisenberg, m. b. (2010). assigning inquiry: how handouts for research assignments guide today’s college students. retrieved from https://www.projectinfolit.org/uploads/2/7/5/4/27541717/pil_handout_study_finalvjuly_2010.pdf   head, a. j. (2016). staying smart: how today’s graduates continue to learn once they complete college. retrieved from www.projectinfolit.org/uploads/2/7/5/4/27541717/staying_smart_pil_1_5_2016b_fullreport.pdf   haines, m., & horrocks, g. (2006). health information literacy and higher education: the king's college london approach. library review, 55(1), 8-19. https://doi.org/10.1108/00242530610641754   hare, s., & evanson, c. (2018). information privilege outreach for undergraduate students. college & research libraries, 79(6), 726-736. https://doi.org/10.5860/crl.79.6.726   hertzum, m. (2014). expertise seeking: a review. information processing & management, 50(5), 775-795. https://doi.org/10.1016/j.ipm.2014.04.003   intas, g., kostagiolas, p., zavras, d., chalari, e., stergiannis, p., toylia, g., & niakas, d. (2016). information-seeking behavior of greek nursing students: a questionnaire study. cin: computers, informatics, nursing, 35(2), 109–114. https://doi.org/10.1097/cin.0000000000000294   johnson-grau, g., archambault, s. g., acosta, e. s., & mclean, l. (2016). patience, persistence, and process: embedding a campus-wide information literacy program across the curriculum. the journal of academic librarianship, 42(6), 750-756. https://doi.org/10.1016/j.acalib.2016.10.013   julien, h., gross, m., & latham, d. (2018). survey of information literacy instructional practices in u.s. academic libraries. college & research libraries, 79(2), 179–199. https://doi.org/10.5860/crl.79.2.179   kahouei, m., babamohamadi, h., panahi, s. s. g. s., & zadeh, j. m. (2013). the impact of it infrastructures on iranian nurses’ and students’ health information-seeking strategies. program: electronic library and information systems, 47(4), 369–383. https://doi.org/10.1108/prog-05-2012-0021   kuglitsch, r. z. (2015). teaching for transfer: reconciling the framework with disciplinary information literacy. portal: libraries and the academy, 15(3), 457-470. https://doi.org/10.1353/pla.2015.0040   kumaran, m., & chipanshi, m. (2015). exploring the information-seeking behaviour of internationally educated nurses (iens) in saskatchewan. journal of the canadian health libraries association / journal de l’association des bibliothèques de la santé du canada, 36, 45–53. https://doi.org/10.5596/c15-013   lupton, m., & bruce, c. (2010). windows on information literacy worlds: generic, situated and transformative perspectives. in a. lloyd & s. talja (eds.), practising information literacy: bringing theories of learning, practice and information literacy together (pp. 3–27). cambridge: woodhead publishing limited. https://doi.org/10.1016/b978-1-876938-79-6.50001-7   majid, s., foo, s., luyt, b., zhang, x., theng, y.-l., chang, y.-k., & mokhtar, i. a. (2011). adopting evidence-based practice in clinical decision making: nurses’ perceptions, knowledge, and barriers. journal of the medical library association, 99(3), 229–236. https://doi.org/10.3163/1536-5050.99.3.010   marshall, a. p., west, s. h., & aitken, l. m. (2011). preferred information sources for clinical decision making: critical care nurses’ perceptions of information accessibility and usefulness. worldviews on evidence-based nursing, 8(4), 224–235. https://doi.org/10.1111/j.1741-6787.2011.00221.x   maybee, c., bruce, c. s., lupton, m., & rebmann, k. (2017). designing rich information experiences to shape learning outcomes. studies in higher education, 42(12), 2373-2388. https://doi.org/10.1080/03075079.2016.1148684   moreton, e. o., & conklin, j. l. (2015). closing the loop on nursing library instruction: using student performance to improve outcomes. medical reference services quarterly, 34(1), 113–121. https://doi.org/10.1080/02763869.2015.986805   newman, k. m., & doran, d. (2012). critical care nurses’ information-seeking behaviour during an unfamiliar patient care task. dynamics: journal of the canadian association of critical care nurses, 23(1), 12–17. retrieved from https://www.caccn.ca/canadian-journal-of-critical-care-nursing/   o’leary, d. f., & ni mhaolrúnaigh, s. (2012). information-seeking behaviour of nurses: where is information sought and what processes are followed? journal of advanced nursing, 68(2), 379–390. https://doi.org/10.1111/j.1365-2648.2011.05750.x   phelps, s. f., & hyde, l. (2018). integrating information literacy into the academic curriculum and nursing practice. in s. phelps, l. hyde, & j. p. wolf (eds.), the intersection: where evidence based nursing and information literacy meet (pp. 95–113). cambridge: elsevier ltd. https://doi.org/10.1016/b978-0-08-101282-6.00008-6   pravikoff, d. s., tanner, a. b., & pierce, s. t. (2005). readiness of u.s. nurses for evidence-based practice: many don’t understand or value research and have had little or no training to help them find evidence on which to base their practice. ajn: the american journal of nursing, 105(9), 40-51. retrieved from https://journals.lww.com/ajnonline/abstract/2005/09000/readiness_of_u_s__nurses_for_evidence_based.25.aspx   raj, s., sharma, v. l., singh, a., & goel, s. (2015). the health information seeking behaviour and needs of community health workers in chandigarh in northern india. health information and libraries journal, 32(2), 143–149. https://doi.org/10.1111/hir.12104   raman, j. (2015). mobile technology in nursing education: where do we go from here? a review of the literature. nurse education today, 35(5), 663–672. https://doi.org/10.1016/j.nedt.2015.01.018   richards, l. (2009). handling qualitative data : a practical guide. thousand oaks, ca: sage foundation.   sackett, d. l., rosenberg, w. m. c., gray, j. a. m., haynes, r. b., & richardson, w. s. (1996). evidence based medicine: what it is and what it isn’t. british medical journal, 312(7023), 71–72. https://doi.org/10.1136/bmj.312.7023.71   saines, s., harrington, s., boeninger, c., campbell, p., canter, j., & mcgeary, b. (2019). reimagining the research assignment: faculty-librarian collaborations that increase student learning. college & research libraries news, 80(1), 14-17, 41. https://doi.org/10.5860/crln.80.1.14   sarbaz, m., kimiafar, k., sheikhtaheri, a., taherzadeh, z., & eslami, s. (2016). nurses’ information seeking behavior for clinical practice: a case study in a developing country. studies in health technology and informatics, 225, 23-27. https://doi.org/10.3233/978-1-61499-658-3-23   sleutel, m. r., bullion, j. w., & sullivan, r. (2018). tools of the trade: improving nurses’ ability to access and evaluate research. journal of nursing management, 26(2), 167-171. https://doi.org/10.1111/jonm.12529       varlejs, j., stec, e., & kwon, h. (2014). factors affecting students’ information literacy as they transition from high school to college. school library research, 17, 1–23. retrieved from: https://files.eric.ed.gov/fulltext/ej1022549.pdf   wahoush, o., & banfield, l. (2014). information literacy during entry to practice: information-seeking behaviors in student nurses and recent nurse graduates. nurse education today, 34(2), 208–213. https://doi.org/10.1016/j.nedt.2013.04.009 . xu, y., tan, b. c., & yang, l. (2006). who will you ask? an empirical study of interpersonal task information seeking. journal of the american society for information science and technology, 57(12), 1666-1677. https://doi.org/10.1002/asi.20339   younger, p. (2010). internet-based information-seeking behaviour amongst doctors and nurses: a short review of the literature. health information and libraries journal, 27(1), 2–10. https://doi.org/10.1111/j.1471-1842.2010.00883.x     appendix a survey instrument   1. about this survey 2. do you want to take the survey? □        yes, i have a bachelor of science in nursing graduate from one of the following institutions: brigham young university, roseman university of health sciences, southern utah university, or the university of utah □        no thanks! 3. demographics 4. are you currently employed as a nurse? □        yes □        no 5. where did you obtain your bsn degree? □        brigham young university □        roseman university □        southern utah university □        university of utah 6. what year did you graduate with your bsn? □        2012 – 2017 (drop-down) 7. what is the highest nursing degree that you have completed? □        bachelor’s in nursing □        master’s in nursing (including education, informatics, etc.) □        doctorate in nursing (dnp, fnp, nursing education, etc.) □        phd in nursing □        other, please specify: (fill in text) 8. are you currently enrolled in a graduate nursing degree program? □        yes □        no 9. since graduating with your bsn, how many years have you been employed as a nurse? □        0  □        1 □        2 □        3 □        4+ 10. describe your employer or work setting. please check all that apply.  □        academic setting □        community clinic □        doctor’s office □        government/military □        k-12 school □        nursing home/long-term care facility □        non-hospital ambulatory care □        general hospital □        research hospital □        specialty hospital □        teaching hospital □        other, please specify (fill in text) 11. which of the following best describes the type of nursing work that you do. please check all that apply. □        advanced practice nursing, please specify (fill in text) □        critical care nursing □        emergency nursing □        home health nursing □        labor and delivery nursing □        maternity nursing □        medical surgical nursing □        neonatal/pediatric nursing □        oncology nursing □        orthopedic nursing □        perioperative nursing □        psychiatric/mental health nursing □        public health nursing □        school nursing □        other, please specify (fill in text) 12. information needs this section contains questions about information needs that might arise during your work day. for example, when a patient asks you whether her headache could be caused by the interaction of two medications she is currently taking, you might need information to be able to answer the question. 13. why do you look for information during your work?  please check all that apply. □        to answer administrative or policy questions □        to answer patient questions □        to clarify conflicting medical evidence □        to consult clinical practice guidelines □        to check clinical procedures □        to determine best practices for patient care □        to solve a clinical problem □        to consult equipment manuals/processes □        to keep up with the latest professional trends and news □        to interpret laboratory data   □        to consult local practice data □        to locate patient education materials or consumer health information □        to interpret physical exam results □        to solve problems you haven't seen before □        to locate evidence about treatment modalities □        to consult population statistics □        to consult public health data □        other, please specify (fill in text) *14. how often do you seek information to answer administrative or policy questions? □        never □        rarely □        monthly □        weekly □        daily □        multiple times a day *15. how often do you seek information to answer patient questions? □        never □        rarely □        monthly □        weekly □        daily □        multiple times a day *16. how often do you seek information to clarify conflicting medical evidence? □        never □        rarely □        monthly □        weekly □        daily □        multiple times a day *17. how often do you seek information to consult clinical practice guidelines? □        never □        rarely □        monthly □        weekly □        daily □        multiple times a day *18. how often do you seek information to check clinical procedures? □        never □        rarely □        monthly □        weekly □        daily □        multiple times a day *19. how often do you seek information to determine best practices for patient care? □        never □        rarely □        monthly □        weekly □        daily □        multiple times a day *20. how often do you seek information to solve a clinical problem? □        never □        rarely □        monthly □        weekly □        daily □        multiple times a day *21. how often do you seek information to consult equipment manuals and/or processes? □        never □        rarely □        monthly □        weekly □        daily □        multiple times a day *22. how often do you seek information to keep up with the latest professional trends and news? □        never □        rarely □        monthly □        weekly □        daily □        multiple times a day *23. how often do you seek information to interpret laboratory data? □        never □        rarely □        monthly □        weekly □        daily □        multiple times a day *24. how often do you seek information about local practice data? □        never □        rarely □        monthly □        weekly □        daily □        multiple times a day *25. how often do you seek information to locate patient education materials or consumer health information? □        never □        rarely □        monthly □        weekly □        daily □        multiple times a day *26. how often do you seek information to interpret physical exam results? □        never □        rarely □        monthly □        weekly □        daily □        multiple times a day *27. how often do you seek information to solve problems you haven't seen before? □        never □        rarely □        monthly □        weekly □        daily □        multiple times a day *28. how often do you seek information to locate evidence about treatment modalities? □        never □        rarely □        monthly □        weekly □        daily □        multiple times a day *29. how often do you seek information to consult population statistics? □        never □        rarely □        monthly □        weekly □        daily □        multiple times a day *30. how often do you seek information to consult public health data? □        never □        rarely □        monthly □        weekly □        daily □        multiple times a day *31. how often do you seek information about the other information need(s) you specified earlier? □        never □        rarely □        monthly □        weekly □        daily □        multiple times a day 32. briefly describe a recent moment at work where you struggled to make sense of a situation. how did you use information to move beyond that situation? 33. information sources this section contains questions about sources you use to get your information. the word “source” is used broadly; a source can be a research article or one of your colleagues.    34. how often do you use information from the following print sources?   never rarely monthly weekly daily multiple times a day handbooks (e.g. drug information handbook)             local education materials             manuals (e.g. merck manual)             medical reference books (e.g. dictionaries)             paper fact sheets / handouts             print journal             textbooks               35. how often do you use information from the following electronic sources?   never rarely monthly weekly daily multiple times a day bmj clinical evidence             cinahl             clinical key             cochrane library             dynamed             e-books             electronic patient record / electronic health record / electronic medical record             google scholar             national guideline clearinghouse             pubmed / medline             general search engines like google/bing/etc.             uptodate             websites with medical information             other point-of-care tool             other electronic source, please explain               36. how often do you use information from the following interpersonal sources?   never rarely monthly weekly daily multiple times a day medical librarians             nursing colleagues             pharmacists             physicians             allied health professionals (e.g. physical therapists, occupational therapists, etc.)             social workers             other health professionals, please explain               37. how often do you use information from the following educational sources?   never rarely monthly weekly daily multiple times a day conferences (conference presentations, posters, and/or proceedings)             courses             in-service training             webinars               38. which options describe your access to full text copies of journal articles, papers, books, or other information sources? please check all that apply. □        i don't know □        employer-provided access to journals, electronic databases, apps, and/or print materials □        employer-provided access to articles, guidelines, books, etc, upon request □        personal membership(s) in professional organizations or societies □        purchase with personal funds □        local academic library □        local public library □        professional contacts outside of my workplace □        other, please specify (fill in text) 39. information environment this section contains questions about the information environment in your workplace. 40. in which of these clinical environments do you need information? please check all that apply. □        bedside □        chairside □        clinical rounds □        healthcare team meetings □        other, please specify (fill in text) 41. in which of these clinical environments can you access information? please check all that apply. □        bedside □        chairside □        clinical rounds □        healthcare team meetings other, please specify (fill in text) 42. does your workplace have a library, or other centralized information resource center? □        yes □        no □        unsure **43. who has access to the library or similar space? please check all that apply. □        all employees/personnel □        certain employees/personnel □        patients □        general public □        unsure **44. does a librarian or other information professional work in this space? □        yes □        no □        unsure **45. is there a librarian or other information professional available for you to contact? □        yes □        no □        unsure 46. what kind of device(s) do you typically use for finding clinical information at work? please check all that apply. □        employer supplied computer □        employer supplied phone □        employer supplied tablet □        personal computer □        personal phone □        personal tablet □        other, please specify (fill in text) 47. do you have access to the internet at work? please check all that apply. □        yes, on a shared computer □        yes, on a personal computer □        yes, on a mobile device □        yes, but internet access is restricted (e.g. with a firewall or other security measures)  □        no □        other, please specify (fill in text) 48. barriers to finding, accessing, and evaluating information this section contains questions about problems you may face as you navigate your information needs.   49. which barriers, if any, keep you from accessing information efficiently and effectively?   not at all a barrier minimal barrier moderate barrier substantial barrier expertise or training in accessing information         access to technology         cost         time constraints         training on technology         training on using information sources         other, please specify (fill in text)           50. which barriers, if any, keep you from evaluating information efficiently and effectively?   not at all a barrier minimal barrier moderate barrier substantial barrier expertise or training in accessing information         access to technology         cost         time constraints         training on technology         training on using information sources         other, please specify (fill in text)           51. do you have any other comments about barriers or constraints to accessing and evaluating information that you would like to share? 52. education/instruction  this section contains questions about how you learned and developed the skills to access and evaluate information at your workplace. the instruction may have happened inside or outside of a classroom, and may have been formal or informal.   53. how important were the following in learning to find information?   important somewhat important neutral somewhat unimportant unimportant n/a self-taught skills             on-the-job mentoring/training             nursing school faculty             nursing school librarian             other higher education             high school or earlier               54. how important were the following in learning to evaluate information?   important somewhat important neutral somewhat unimportant unimportant n/a self-taught skills             on-the-job mentoring/training             nursing school faculty             nursing school librarian             other higher education             high school or earlier               55. if you seek help searching for information (e.g. tips on specific search strategies, tools, or sources), where do you get that help? please check all that apply. □        healthcare colleagues other than nurses in my workplace □        healthcare colleagues other than nurses outside my workplace □        librarian or other information professional in my workplace □        librarian or other information professional at a public, academic, or other library in my community □        nursing colleagues in my workplace □        nursing colleagues outside my workplace □        online social networks, such as twitter or facebook □        other, please specify (enter text) 56. if you seek help evaluating information, where do you get that help? please check all that apply. □        healthcare colleagues other than nurses in my workplace □        healthcare colleagues other than nurses outside my workplace □        librarian or other information professional in my workplace □        librarian or other information professional at a public, academic, or other library in my community □        nursing colleagues in my workplace □        nursing colleagues outside my workplace □        online social networks, such as twitter or facebook □        other, please specify (enter text) 57. what are some of the things you look for when evaluating information? for example, if you were searching for evidence-based articles, how would you determine their suitability to answer your clinical question? 58. how do you stay up to date with information in your field? please check all that apply. □        conferences □        continuing education (ceus) □        interacting with colleagues who are up to date □        listservs □        personal reading □        professional organizations □        social media □        webinars □        other, please specify (enter text) 59. how could your nursing education have better prepared you to find and evaluate information? 60. how confident are you at explaining what kind of information you need (e.g. what sources to use)? □        not at all confident □        somewhat confident □        moderately confident □        very confident □        completely confident 61. how confident are you with finding information (e.g. selecting a database, creating and refining a search strategy, and managing the information you get back)? □        not at all confident □        somewhat confident □        moderately confident □        very confident □        completely confident 62. how confident are you with evaluating the quality and relevance of the information you find? □        not at all confident □        somewhat confident □        moderately confident □        very confident □        completely confident 63. how confident are you with applying the information you find to a problem or creating a product (memo, presentation, paper) to share what you learned with others? □        not at all confident □        somewhat confident □        moderately confident □        very confident □        completely confident 64. how confident are you with understanding the economic, legal, and social issues surrounding information (e.g. ethical use of information, familiarity with laws, regulations, and institutional policies about using and sharing information, citing your sources)? □        not at all confident □        somewhat confident □        moderately confident □        very confident □        completely confident 65. please provide any additional comments that you have about your information needs and barriers. 66. thank you for taking the survey! would you like to enter your name and email for the gift card drawing? □        yes □        no   * questions 14 – 31 only displayed if the respondent checked the option that they sought information for that particular reason in question 13. ** questions 43 & 44 displayed if the answer to question 42 was yes; question 45 displayed if the answer to question 42 was no.   evidence based library and information practice evidence based library and information practice 2009, 4.4  evidence based library and information practice       evidence summary    undergraduate students with strong tendencies towards critical thinking experience  less library anxiety    a review of:  kwon, nahyun. “a mixed‐methods investigation of the relationship between critical thinking and  library anxiety among undergraduate students in their information search process.”  college & research libraries 69.2 (2008): 117‐31.c    reviewed by:  cari merkley  librarian, mount royal university  calgary, alberta, canada  email: cmerkley@mtroyal.ca    received: 01 september 2009        accepted: 30 october 2009      © 2009 merkley. this is an open access article distributed under the terms of the creative commons attribution  license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and  reproduction in any medium, provided the original work is properly cited.    objective – to investigate the nature of the  association between a student’s critical  thinking disposition and the extent to which  they suffer from library anxiety.    design – standardized quantitative survey  instruments and a qualitative content analysis  of student essays.    setting – a state (publically funded) research  university located in the southeast united  states.    subjects – 137 undergraduate students  enrolled in the library and research skills  course.    methods – undergraduate students enrolled  in the three‐credit course library and research  skills during the spring 2006 semester were  invited to participate in the study. of 180  students registered in the course, 137  volunteered to take part. data collection took  place in the first two weeks of the semester.     participants were asked to complete two  standardized survey instruments: the  california critical thinking disposition  inventory (cctdi) and the library anxiety  scale (las). the purpose of the cctdi is to  “measure a person’s disposition to use critical  thinking” (119). the instrument consists of  seven scales: “truth‐seeking”; “open‐ mindedness”; “analyticity”; “systematicity”;  “critical thinking self‐confidence”;  “inquisitiveness”; and “maturity” (119).  “truth‐seeking” is a commitment to seeking  answers even if the process proves difficult or  reveals information outside of one’s belief  system, “systematicity” is defined as an  organized approach to problem solving, and  “maturity” is the ability to make “reflective  63 mailto:cmerkley@mtroyal.ca evidence based library and information practice 2009, 4.4  decisions when facing ill‐structured problem  situations” (119). “analyticity” refers to a  subject’s ability to anticipate possible  outcomes, “open‐mindedness” to being open  to different points of view, “critical thinking  self‐ confidence” to a belief in one’s own  critical thinking skills, and “inquisitiveness” to  “intellectual curiosity” (119). participants  scored 75 items using a six‐point likert‐type  scale.     the las measures levels of library anxiety by  asking students to respond to 43 statements  using a five‐point point likert‐type scale. the  las is designed to identify perceived  roadblocks to their students’ use of the library,  including “barriers with staff” or staff who are  not helpful, “affective barriers” or a lack of  confidence in one’s research skills, “comfort  with the library,” “knowledge of the library,”  and “mechanical barriers” such as equipment  that is difficult to use (119).    in addition, participants were asked to write a  500‐1,000 word essay about their “most recent  or most memorable experience of using the  library and its resources to write a research a  paper” (120). quantitative data collected from  the cctdi and las was analyzed using  statistical software and the content of the  qualitative data generated by the student  essays was analyzed to identify common  critical thinking and library anxiety themes.    main results – only a small percentage (6%)  of participants in the study were freshman  (i.e., in their first year of study). the largest  group was comprised of third year students or  juniors (41.8%), followed by sophomores  (27.6%) and seniors (21.6%). the participants  ranged in age from 18 to 60, with an average  age of 22.9 years. over 68% percent were  female.    overall, a higher percentage of study  participants scored lower on the cctdi across  all seven scales than a normative sample of  undergraduate students. a score below 40 on  a particular scale is considered by the  instrument developers to be an indication of  weakness in that particular dimension of  critical thinking. the participants’ mean score  for each of the seven scales fell below this  threshold. areas of particular weakness were  truth‐seeking (82% of students scored below  40), systematicity (63% scored below 40), and  maturity (55% below 40).    the researcher ranked the students by their  total cctdi scores, and then divided the  subjects into three equal groups. the 37  students with the highest overall cctdi  scores were labelled the strong critical  thinking dispositions (ctd) group. the 37  students with the lowest overall cctdi scores  formed the weak ctd group. the mean las  scores of participants in each group were then  compared. a higher las score is indicative of  a higher level of library anxiety. students with  strong ctd demonstrated significantly less  library anxiety than those with weak critical  thinking dispositions (an overall mean score of  93.03 versus 111.13). when it came to the five  dimensions of library anxiety, the difference in  the mean scores between the two groups was  greatest for staff barriers (30.88 for participants  with strong ctd versus 38.20 for those weak  ctd) and affective barriers (27.24 versus  32.94). the difference in scores for anxiety  arising from mechanical barriers was lower  (0.83), but still statistically significant (p<.05).     according to kwon, the analysis of the  student essays uncovered widespread library  anxiety among students regardless of their  cctdi scores, with many reporting that they  felt lost when first approaching library  research. particular sources of anxiety were  affective barriers (e.g., lack of confidence) and  staff barriers. students also reported that their  anxiety made it difficult to think clearly about  their search. students with strong critical  thinking dispositions in the areas of  systematicity, critical thinking self‐confidence,  and inquisitiveness were able to mobilize  these skills to overcome their library anxiety  and move forward with their research. those  who were able to move past their discomfort  and activate their critical thinking skills  reported a reduction in their overall anxiety.  in some cases, the essays of students who had  scored low on the cctdi demonstrated  64 evidence based library and information practice 2009, 4.4  increasing levels of anxiety as their search  progressed and a failure to use critical  thinking to overcome the challenges  encountered during the research process. the  researcher expressed these findings in “the  interactive model of critical thinking and  library anxiety” (125). in the model, students  initiating a research project (stage 1) move  into a state of library anxiety (stage 2) that  impedes their cognitive skills (stage 3). stage 4  is the critical juncture at which the student’s  critical thinking disposition is activated or not.  if it is activated, students are able to access  their critical thinking skills (stage 5), lessening  their anxiety (stage 6) and resulting in the  successful completion of work relating to their  task (stage 7). if the disposition is not  activated, their critical thinking skills remain  hampered (stage 5) and anxiety increases  (stage 6), preventing the successful  performance of their task (stage 7).    students also reported in their essays that  their critical thinking skills improved as they  gained more experience with the library  research process, and that positive encounters  with library staff resulted in lower levels of  library anxiety.       conclusion – the quantitative analysis of the  cctdi and las results suggest that there is a  negative association between critical thinking  dispositions and library anxiety. the  qualitative data also seems to imply that those  with strong critical thinking dispositions are  able to reduce their levels of library anxiety  through their ability to work through  problems in a persistent and methodical way,  although further research is required to  validate all the steps in the proposed model.     the evidence also suggests that a student’s  emotional state plays a key role in his or her  ability to think critically and problem solve,  and demonstrates the importance information  literacy instructors should place on cultivating  students’ confidence in their own skills when  preparing them for research success.         commentary    the concept of critical thinking dispositions is  an interesting addition to the library literature,  although it comes late to our field after years  of discussion and application in other  academic circles. kwon clearly sees it as  pivotal, writing that “it can also be contended  that critical thinking disposition is a catalyst  that can change the information search process  from frustration to hope” (128). the  developers of the cctdi instrument also  argue “common sense tells us that a strong  overall disposition toward [critical thinking] is  integral to insuring the use of ct skills outside  the narrow instructional setting” (facione,  giancarlo, facione, and gainen 3). it would be  interesting to know how the concept of critical  thinking lines up with self‐efficacy, which has  seen more play among information  researchers.    this study builds on kwon’s earlier work that  examined critical thinking dispositions and  library anxiety among graduate students with  similar results (kwon, onwuegbuzie, and  alexander). in this current study, kwon  complemented the quantitative data with the  voices of students themselves. however, the  methodology possesses several weaknesses  that may limit the utility of the results. kwon  acknowledges that the library anxiety scale is  dated, reflecting a library environment of 1992  rather than one with an increasingly virtual  presence. while modified versions of the las  do exist, kwon argues that “none of these  newly developed scales have demonstrated  general applicability yet, while the las still is  a highly robust, validated scale that can  measure library anxiety of the undergraduate  students in the united states”(120). kwon also  admits that the sample of students may also be  somewhat suspect. high levels of existing  library anxiety and low predispositions  towards critical thinking may have been what  drove those students not required to take the  library and research skills to enrol in it as an  elective. kwon writes, “this fact might have  skewed the study results” (128). she does not,  however, reveal how many students fell into  this category, or how their scores may have  65 evidence based library and information practice 2009, 4.4  66 compared with those who were required to  enrol. the presentation of scores could have  also been improved. while providing the  standard deviation for the las and cctdi  scores is helpful, the median scores for all  categories would have provided readers with  a clearer picture as mean scores are so  sensitive to outliers in small groupings of  subjects (jargowsky and yang).    finally, the model constructed by kwon is not  fully corroborated by the evidence collected.  the researcher acknowledges that the essay  documents student perceptions of their own  skills, and as such provides no objective data  that the task was successfully completed (step  7). more information about how the study was  presented to the students and the process the  author went through in analyzing the essays  would have also been useful. for example, did  the students receive any sort of compensation  for completing the essays? if library anxiety  was a “common experience to most students,”  exactly how many of the essays demonstrated  that (kwon, 123)? did the researcher know  which of the essays were written by the  “strong” ctd group when the essays were  coded? more information on the methodology  or theoretical framework employed in the  analysis of the essays would have lent weight  to conclusions drawn from the data.     library anxiety is alive and well among the  college populace, at least among the students  who formed part of this study. in spite of its  limitations, kwon’s work makes a case for  considering critical thinking dispositions  when seeking solutions to the problem.  academic librarians shouldn’t go wrong if  they accept kwon’s recommendation to  consider the impact of students’ affective  states in all library staff/student interactions  and to build an environment that encourages  and supports intellectual curiosity. kwon  clearly acknowledges the link between her  findings and the emphasis on affective states  in kuhlthau’s existing information‐search  process model (127). however, kwon’s  assumption that library instruction is part of  that solution may be premature. assessing the  impact of our work on students’ affective and  cognitive states remains critical. for example,  participants in this study were assessed in the  first two weeks of a course explicitly intended  to bolster their ability to employ critical  thinking in the information search process  (kwon 118). the obvious question is whether  or not the course had any impact on either  their critical thinking dispositions or library  anxiety, so it is surprising that the author does  not identify this as an area for future study. in  fact, how would one isolate any changes in  ctd due to library instruction, when other  factors may be at play including the  maturation of student minds, instruction  received in other venues, and efforts to build  critical thinking taking place at an institutional  level? many of the students were in their  second or third year of study, but there was no  information about how this may have played  into their levels of critical thinking or library  anxiety. assessing the efficacy of information  literacy instruction in this regard may  illuminate areas of weakness, but more  research needs to be done before any  recommendations can be fully accepted.       works cited    facione, peter a., carol a. giancarlo, noreen  c. facione and joanne gainen. “the  disposition toward critical  thinking.” 1995. 28 aug. 2009  <http://www.insightassessment.com/  pdf_files/disposition_to_ct_1995_jg e.pdf>    jargowsky, paul a. and rebecca yang.  “descriptive and inferential  statistics.” encyclopedia of social  measurement. ed. kimberly kempf‐ leonard. 31 mar. 2005. sciencedirect.  26 oct. 2009.     kwon, nahyun, anthony j. onwuegbuzie and   linda alexander. “critical thinking  disposition and library anxiety:  affective domains on the space of  information seeking and use in  academic libraries.” college and  research libraries 68.3 (2007): 268‐78.  microsoft word es_haddow.doc evidence based library and information practice 2007, 2:4  57 evidence based library and information practice       evidence summary    open access pricing models would reduce journal expenditure at most colleges and  universities    a review of:  walters, william h. “institutional journal costs in an open access environment.” journal of the  american society for information science and technology 58.1 (2007): 108‐20.      reviewed by:  gaby haddow  divisional librarian, humanities, university library, curtin university of technology  perth, western australia, australia  e‐mail: g.haddow@curtin.edu.au      received: 31 august 2007      accepted: 19 october 2007      © 2007 haddow. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to determine journal  expenditure at nine colleges and universities  using three pricing models, and to  investigate the impact of an open access  pricing model on journal costs according to  institution type.    design – cost analysis of three pricing  models for journals.    setting – nine college and university  libraries in the united states.    subjects – 349 journal titles indexed by the  science citation index from four subject fields  (general biology, cell biology, organic  chemistry, and applied physics).     methods – institutions included in the  study were selected from america’s best  colleges (2004) on the basis of type of  institution and quality ratings. stratified  sampling was used to randomly select an  institution from the top 25% (tier 1) and  bottom 25% (tier 4) from four institutional  categories (doctoral national universities,  master’s universities, bachelor’s liberal arts  colleges, and bachelor’s comprehensive  colleges). an additional institution was  selected from tier 1 of the doctoral  universities category to ensure both a large  and small institution was represented in the  sample.    institutional access to the 349 journals was  determined by searching the nine  institutions’ library catalogues between  august and october 2004. print, online and  mailto:haddow@curtin.edu.au http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:4  58 combined print/online access was identified  for each title at each institution.    a component of the pricing models (below)  included costs incurred when academics at  the institutions published articles in some  journals. to arrive at an estimate of these  fees, articles published by authors at each  institution were identified in the journal  titles. all articles in the 349 titles by staff  (first author only) at the nine institutions  were costed at the lowest rate listed and  assigned as journal expenditure for the  institution.    the cost analysis of institutional journal  expenditure at each institution was  conducted using three pricing models:    1. a conventional subscription model.  journal expenditure was calculated from  four elements: individual title subscriptions;  online journal collection costs; online journal  aggregator costs; and  submission/publication fees. these costs  were determined from: online information  about journal subscription prices; estimating  a proportion of journal collection and  aggregator databases costs; and page  charges, publication and submission fees for  authors from the institution.    2. an open access publishing model based on the  public library of science (plos) pricing model.  two costs comprised the open access  publishing model: publication fees and print  title subscriptions. publishing fees were  calculated from determining the average  number of articles published by authors  from an institution in one year. this number  was then multiplied by $1,500 – the rate  charged by plos in 2005. the second  component was the cost of existing  subscriptions to print titles, on the  assumption that libraries will continue these  subscriptions despite receiving online access  to them. print titles were costed at $160 per  subscription.  3. an equal revenue open access model (designed  by the researcher to allow for the assumption  that publishers considering taking up the plos  model would not accept a reduction in revenue).  if the plos model was implemented across  the 349 titles, a substantial reduction in  revenue to publishers would occur.  therefore, a multiplier was calculated for  each subject area by dividing the aggregate  amount (in the conventional model) paid for  journals in a subject area by the amount  paid under the plos model. the multiplier  was then applied to the plos title cost. an  adjustment was also made to reduce the  effects of the non‐representative sample of  institutions.     main results – the conventional pricing  model calculations indicate that the  association between journal expenditure  and library size is stronger than the  association between journal expenditure  and type of institution. however, type of  institution is a factor in journal expenditure  with doctoral universities’ expenditure  much higher than that of other types of  institutions. no associations or trends were  found when comparing journal expenditure  across institutions for the four subject fields.     under the plos pricing model, journal  expenditure at all nine institutions would be  substantially reduced. if this model was  adopted by all publishers they would  receive only 15% of the total amount  expended on journals by the institutions  under the conventional pricing model. the  plos model would affect institutions’  proportion of total expenditure, with  doctoral universities paying 96% of journal  expenditure compared with 86% under the  conventional model. this result is directly  related to publishing activities at institutions.  large research universities, where staff  publish at higher rates, would pay a  proportionally higher amount of the total  expenditure than other types of institutions,  where publishing activity is low.   evidence based library and information practice 2007, 2:4  59 the equal revenue model, when compared  with the conventional model, would  provide substantial reductions in journal  expenditure for seven of the institutions.  however, under this model the largest  doctoral university would see an increase in  journal expenditure of 337% from  conventional model expenditure: an  increase primarily due to the higher  publishing activity at the university.  institutions with low publishing activity and  least print subscriptions would see the  greatest reductions in expenditure if  publishers moved from the conventional  model to the equal revenue model.    up to 90% of american colleges would see a  reduction in journal expenditure if  publishers adopted the plos or the equal  revenue pricing models.    conclusion – most colleges and universities  are consumers rather than producers of  scholarly literature and, therefore, a shift  from the conventional subscription model to  a model based on publication activity will  reduce their proportion of total expenditure.  on the other hand, large research  universities will see an increase in their  expenditure as a proportion of total journal  costs.    given the cost savings and the access to  journals, a pricing model such as the plos  would benefit most colleges and universities.  in turn, under this model revenues to  publishers would be reduced substantially.    the equal revenue model would benefit the  majority of colleges and universities by  increasing their access to journal titles while  reducing their journal expenditure. for  example, a master’s university journal  holdings could increase up to ten times with  expenditure reduced by between 20–60%.  large research universities, however, would  see an increase in journal expenditure for  very few additional journal titles.   commentary    this study investigates the impact of open  access journal pricing models on journal  expenditure at nine american colleges and  universities. open access pricing models are  currently operating for a number of journal  collections, including public library of science  and biomed central, and shift the cost of  institutional journal access from one based  on subscriptions to an author‐pays model.  the findings of the study will provide some  useful information for college and  university librarians considering taking up  an open access journal collection.    to conduct the study, the researcher  required a defined journal set and  institutions for the sample. while the journal  sample is clearly described and justified, the  sample size is small as a proportion of all  colleges and universities in the united states.  only nine, from over 1,300, colleges and  universities were included in the sample.  although the equal revenue pricing model  applies a multiplier that adjusts for  institutional representation in the sample,  the researcher acknowledges sample size as  a limitation to making generalisations.  however, given the extensive data  collection activities involved in the study,  the researcher’s decision to limit the sample  to nine institutions is understandable;  although it means associations and trends  are discussed, rather than statistically  significant findings.    the methods used to calculate journal  expenditure at the nine institutions, across  four subject fields, and for three pricing  models are highly complex. this is  particularly true for the equal revenue  model. it should be noted that since the  study was conducted plos has substantially  increased its fees, which in turn will affect  results for the plos and equal revenue  models. however, the assumptions made in  designing the pricing models are sound and  evidence based library and information practice 2007, 2:4  60 the study’s results assist the reader  comprehend the methodological  complexities.    an earlier study at cornell university  (davis) found the majority of arl  institutions would pay more in journal  expenditure under an open access model  than the conventional subscription model.  this study was criticised on a number of  fronts, including: setting the lower level  publishing fee too high; and assuming  science and humanities journals would  incur the same average costs. walters’ set  lower publishing fees when calculating for  the plos model and included some print  subscription costs. these adjustments, and  the pragmatic approach taken by walters in  his equal revenue model, contribute to the  argument that open access publishing  models are advantageous for many  institutions and result in an alternative  conclusion to the cornell study. clearly, the  methods used to measure journal  expenditure in an open access publishing  environment can greatly affect results.  unless there is a consistent application of  methods, the profession will find it difficult  to draw conclusions from study findings.    several suggestions for further research are  made by the author, including research  using a larger sample of institutions and an  examination of the sustainability of different  pricing models from a publisher’s  perspective. a study of publishers may raise  some interesting issues for discussion, but  based on the current range of pricing  models used by publishers, it is doubtful  whether the findings would provide greater  clarity about the future. the author also  notes a number of broader questions to  address. these range from scholarly  communication issues, such as the effect of  pricing models on publishers’ acceptance  rates, to budgetary considerations, such as  shifts in institutional expenditure on  journals.   this study contributes to our knowledge of  how open access pricing models might  impact on institutional expenditure on  journals and raises important issues to  consider as consequences of open access  publishing. while the methods used in the  study are complex and involve extensive  data collection activities, further research  incorporating more institutions would be of  great value. however, at the heart of the  debate about open access publishing is an  important question – which pricing models  are publishers seriously thinking about?       works cited    davis, philip m. calculating the cost per  article in the current subscription  model. 2004.  13 nov. 2007  <http://hdl.handle.net/1813/236>.    http://hdl.handle.net/1813/236 research article   gathering evidence of learning in library curriculum center spaces with web gis   rick stoddart library assessment coordinator university of oregon libraries eugene, oregon, united states of america email: ricks@uoregon.edu   bruce godfrey gis librarian university of idaho library moscow, idaho, united states of america email: bgodfrey@uidaho.edu   received: 29 jan. 2020                                                               accepted: 18 may 2020      2020 stoddart and godfrey. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29721     abstract   objective –this article reports on a pilot research project that gathered usage statistics in specifically designated library learning spaces using a web-based geographic information system (gis). these learning spaces were then mapped to expected learning activities that would occur in these areas based on its intention or design. in this way, the library could begin to associate the usage of a space with different types of learning. the researchers then mapped these learning activities to campus learning outcomes to create learning impact statements.   methods – the researchers used observation data gathered with a web gis tool to examine space usage within the library’s curriculum center.   results – the pilot study found that student usage of the curriculum center was mainly associated with two campus learning outcomes: (1) communicate and (2) learning and integrate. the evidence also indicated possible design improvements that may make the curriculum centers spaces more functional for students.   conclusions – the web gis tool proved to be a useful tool to gather evidence of student space usage within the library environment. the mapping of individual spaces to learning activities further enhanced the usefulness in interpreting how students are using library spaces. leveraging the space usage data within learning outcomes statements created another means for the library to communicate its learning impact with campus stakeholders.     introduction   academic libraries offer diverse learning spaces for students and researchers. these spaces can range from the traditional quiet study areas to more dynamic technology-infused spaces such as data visualization labs or makerspaces. while this variety of library spaces demonstrate the evolving efforts academic libraries have undertaken to be responsive to student and researcher needs and expectations, the core mission of academic libraries remains the same – to support the learning needs of the communities they serve. academic libraries advance their learning mission through the development of resource collections that support their institution's curriculum and research needs. libraries also offer library instruction, workshops, and tutorials that aid in research productivity, information literacy, and workplace skills. in addition, libraries create spaces that are conducive to student learning and engagement.   this pilot research project focused on one aspect of the library’s learning mission – learning spaces. as van note chism (2006) suggested about learning spaces:   institutions of higher education are charged with fostering specific kinds of learning: higher-order thinking abilities, communication skills, and knowledge of the ways of disciplinary experts, to name a few. educators must create structures that support this learning. space can have a powerful impact on learning; we cannot overlook space in our attempts to accomplish our goals. (p. 2.2)   academic libraries are prime locations for the types of learning van note chism highlighted by offering dedicated spaces for students to gather, study, and learn. as a result, libraries have been taking a more active role in designing and thoughtfully thinking about the physical environment they provide for their patrons. van note chism pointed to monahan's (2002) idea of a "built pedagogy" as one way to think about this idea of intentionally designed learning space.   however, as libraries create and cultivate these diverse learning environments, they remain challenged to assess and evaluate what types of learning activities occur in these spaces – especially if the learning activity, such as studying, is self-directed by students. this article reports on a research project for which researchers gathered usage statistics in specifically designated library learning spaces using a web-based geographic information system (gis). these learning spaces were then mapped to expected learning activities that would occur in these areas based on its intention or design. in this way, the library staff could begin to associate the usage of a space with different types of learning.   this work builds on previous research by the authors that detailed the technical, technology, and some methodological aspects of this project that focused primarily on data gathered in the main library (godfrey & stoddart, 2018). this current article reports on a different dataset concentrating on one specific library space, the curriculum center, which is embedded in the university's college of education. in addition, this article focuses more directly on mapping usage data to campus learning objectives.   objective   documenting learning in libraries has always been a challenge. gate counts capture the number of patrons who walk through the door but do not illuminate where in the library patrons go, or what learning activities patrons undertake when in the library. traditionally, libraries are viewed as a space for students to study, which can also be seen as a form of self-directed learning. self-directed learning is an essential form of learning that often occurs outside of the classroom. many libraries now offer additional spaces such as computer labs where students apply, create, and integrate knowledge through completing homework assignments, writing papers, or interacting with online learning management systems. libraries also often offer group spaces where students can collaborate in teams to complete projects or study. additionally, libraries have begun to build dynamic spaces such as audio/visual labs or makerspaces where students can create or apply knowledge in a hands-on technology-rich environment. within all these possible library spaces, simple gate counts are an insufficient measure to adequately express how learners interact with the library and leverage these spaces for learning or other activities.   recently, the university of idaho completely remodeled the college of education building, including its curriculum center, which is staffed by library personnel. the new curriculum center space includes a collection area of five shelving units for materials, a service point for circulation and research assistance, a bank of five computers for printing and writing, a group table for study and collaboration, and various soft seating elements for study, relaxation, and gathering.   this article focuses specifically on data and observations captured at the curriculum center in these newly designed spaces using gis, and how this data might inform the reporting of the library’s contributions to campus learning outcomes. the intended goal of this research is to be able to gather evidence that would support statements connected to relevant campus learning outcomes similar to the one articulated below:   curriculum center learning outcome statement: communicate the curriculum center supports the campus learning outcome of communicate by offering spaces, such as computer stations and a public demonstration space, that encourage acquiring, articulating, creating, and conveying meaning. in 2017, the curriculum center recorded x# interactions in these communicate supporting spaces and observed x# patrons using these resources.   method: evidence-based research   koufogiannakis and brettle (2016) outlined an evidence-based framework based on booth’s collaborative model (2009) to guide researchers and practitioners. this evidence-based cycle is as follows: articulate. assemble. assess. agree. adapt. this model is used by researchers and practitioners to assist in developing their evidence-gathering for research projects and decision-making. the evidence-based framework also helped construct the pilot project methodology detailed below and was embedded within the traditional research paper structure of introduction, objective, literature review, methods, results, discussion, and conclusion.   articulate the question   the development of guiding questions for this study was intended to determine if students are using the new furniture and spaces in the curriculum center, as well as an attempt to map this usage to related campus student learning outcomes. the research questions were as follows:   what is the student usage of the new curriculum center spaces/furniture?   can this usage data be mapped to campus learning outcomes?   assemble the evidence   the evidence gathered for this pilot project consists of internal evidence available from local data sources, external evidence available from the literature, and evidence gathered from research associated with the web gis pilot project. in combination, these sources of evidence informed the research direction for this pilot project.   external evidence (literature review)   libraries have been quick to embrace exploring various design elements to expand the learning opportunities available to their patrons. this is evidenced by the rise in redesigned library spaces such as learning commons, makerspaces, and ideation rooms that allow for flexible interaction with design elements and technology. evaluation of these spaces has been a challenge for some libraries. ferria et al. (2017) noted "there is a growing concern for universities to evaluate their library facilities, services, technology, and information resources to determine the impact on student learning and how library supports the research and public service mission of the institution” (p. 20).   one significant development for library space evaluation has been the work undertaken by casden et al. (2020), researchers at north carolina state university. using their suma tablet-based space assessment tool, they investigated library space usage, activities, and transactions. this mobility to gather evidence using a tablet and manipulate the data into visualizations or dashboards was an inspiration for the authors to undertake their own research in this area. however, this particular study leveraged locally available gis expertise and web gis instead of deploying a suma software application installation or building other technology evaluation options from scratch. the study was a continuation of the work undertaken previously by the authors (godfrey & stoddart, 2018) that demonstrated the feasibility of using web gis as a means to capture and articulate library space usage. the research is also built upon the previous gis library space work such as that by bishop and mandel (2010), coyle (2011), elliott (2014), given and archibald (2015), mandel (2010), and xia (2004, 2005) that all explored the connections of gis and library space evaluation.   in addition to gathering data on space usage, the authors were interested in exploring the possibility of more direct ways to connect activities that occur in library spaces to campus learning outcomes. after all, these newly reimagined spaces were intentionally designed to facilitate certain types of activities associated with learning. for example, learning commons areas where students research and write papers is associated with knowledge creation; or collaboration spaces, such as group study rooms, contribute to communication and teamwork development. monahan (2002) suggested the term built pedagogy as the way the design of learning environments influences what forms of learning might be accessible to students interacting in such spaces. an example of this built pedagogy would be a room of unmovable study carrels which would convey a pedagogy of conformity. this sort of design has built-in parameters on the types of learning that could effectively occur in such a space constraining learning actions to individualized studying and limited interaction from peers. conversely, a learning environment that has moveable tables and chairs suggests a pedagogy of freedom, collaboration, and discovery. scholars such as jonassen and land (2000), oblinger (2006), and savin-baden (2008) also put together works examining the theoretical and functional aspects of learning spaces that influenced the thinking behind this pilot research project. in particular, mathews and soistmann's recent work (2016) about responsive, flexible design concepts and learning environments inspired the research as exemplified by their suggestion that "space imparts action" (p. 30). we also believed that thoughtful library designed space might influence the act of learning in beneficial ways.   evidence-based librarianship offers a useful framework to begin connecting space usage data to campus learning outcomes. evaluation of library spaces and evidence-based research are not strangers to each other. recent examples include evidence-based library space research undertaken by asher (2017), who examined a library's learning commons, as well as ferria et al. (2017) who investigated in what ways students are using library spaces for learning and social engagement. the researchers for both of these studies used a mixed-methods approach. still, other evidence-based methodologies for library space evaluation have included photographic research methods (bedi & webb, 2017), longitudinal observation (fox & doshi, 2013), and ethnographic investigation (tewell et al., 2017).   in summary, this study leveraged the methodology processes and research assembled by the research base of evidence-based librarianship, gis library space evaluation, and purposely designed learning spaces.   internal evidence   while library staff can capture gate count data from the main library’s electronic security gates, they do not have that option for the curriculum center because of its open design. circulation data on collection use was available, but it only reflected usage in one area of the curriculum center and did not take into account browsing or other activities. to remedy this lack of space usage data, the curriculum center staff began keeping observational statistics in an online spreadsheet. however, this spreadsheet was not designed or intended as a sophisticated data instrument. spreadsheet usage data was organized simply by observed activity such as browsing or studying and did not include elements such as location or time of day. therefore, inferences can only be made about how patrons were using the curriculum center but not where within the multiple potential learning spaces available such activities were occurring. in order to begin gathering this different level of detail, a new method was needed to fully capture learning activities within the curriculum center.   a good starting point to begin to understand how the curriculum center spaces might impact student learning was to examine the center's layout to reveal the intentionality of its design more clearly. a map of the curriculum center was created based on various attributes such as seating, study tables, or collections – as well as available technology such as computer terminals (see figure 1).   once a detailed map of the curriculum center was created, the next step was to connect this map to potential learning outcomes or activities that might occur within these areas. the learning space taxonomy, part of the learning space toolkit (https://learningspacetoolkit.org/space-types/learning-space-mind-map/index.html), was used to map activities associated with the curriculum center (see figure 2). data collectors were asked to record comments if observations were outside expected space activities. the taxonomy groups activities into five broad categories:   ·        focus (listening, studying, meditating, viewing, etc.) ·        create (designing, editing, writing, producing, etc.) ·        collaborate (brainstorming, demonstrating, discussing, meeting, presenting, etc.) ·        share (assessing, teaching, tutoring, advising, etc.) ·        socialize (eating/drinking, gaming, networking, etc.)     figure 1 curriculum center spaces.     these learning taxonomy categories are then mapped to corresponding university learning outcomes (see table 1).   once spaces on the map of the curriculum center spaces were assigned, the authors met and discussed the types of taxonomy activities that would most likely occur in these areas (see table 1 and figure 2). for the seating areas identified in the curriculum center, it was thought that studying would be the most appropriate activity, so the researchers associated this area with focus. the curriculum center collections area, which encompassed the available stacks of books, was also associated with focus as patrons used this area to view or browse materials. for the computer area, the authors associated this with create as this was the place where writing and editing most likely would occur. the study table was associated with collaborate, as this was the most prominent group space in the curriculum center. while the front desk in the curriculum center is a service point to check out materials, it is also a place to ask questions or seek assistance. with this in mind, the front desk was associated with share for the teaching and advising aspect that occurs there. share was also associated with the rug space as this area is envisioned as a place where story times or informal teaching opportunities might happen. based on the learning space taxonomy that was being used for the study and the associated spaces in the curriculum center, the researchers were able to crosswalk these space usage criteria to the campus learning goals (see table 1).     figure 2 curriculum center learning taxonomy.     table 1 curriculum center learning spaces taxonomy learning taxonomy learning outcome activity space focus learn & integrate listening, studying, meditating, viewing seating, collections create communicate designing, editing, writing, producing computers collaborate learn & integrate brainstorming, demonstrating, discussing, meeting, presenting study table share learn & integrate assessing, teaching, tutoring, advising, front desk socialize communicate eating/drinking, gaming, networking, rug space,     this mapping did not indicate that curriculum center space usage was a direct measure of a particular learning outcome, rather this research was intended to gather evidence to better communicate to stakeholders how libraries contribute to supporting learning on campus. additionally, every single campus learning outcome would not be captured by the learning taxonomy assigned within the curriculum center spaces (see https://www.uidaho.edu/learningoutcomes for a full list of learning outcomes). similarly, there might be multiple learning outcomes associated with activities occurring in library spaces. for the simplicity of this research project, a primary learning outcome was assigned to each taxonomy. thus, this study created an indirect assessment of potential learning activities that might be occurring in curriculum center spaces. this study relied heavily on the assumption that the spaces were designed appropriately to facilitate and enhance specific types of learning (i.e., built pedagogy). ultimately, the data gathered was intended for the curriculum center to begin to evaluate the effectiveness of these intentionally designed spaces. also, this evidence acts as another data point from which to construct new narratives on campus regarding the role libraries and the curriculum play in contributing to student success and learning.    capturing data with the gis pilot project   these maps articulating the learning spaces within the curriculum center were useful to understand how patrons might operationalize these areas for their learning development. however, without capturing the actual usage of the spaces, it was uncertain if and when these learning spaces were utilized for their intended purposes or even other activities not envisioned by the designers. because of their previous familiarity with this technology, the authors wondered if a web-based gis application would be an appropriate tool to capture detailed patron space usage data in the curriculum center. in addition to obtaining location-specific information, a web gis data-gathering tool would afford a flexible means to begin gathering usage data without a significant expenditure of library resources or technical training.   after some preliminary investigation of web gis tools, the researchers selected collector for arcgis (https://www.esri.com/en-us/arcgis/products/collector-for-arcgis/overview) as the most appropriate application to gather data via a desktop computer that directly observed the curriculum center. there were many advantages of using web gis for this research project. a support network and infrastructure was already in place to work with web gis on campus. there was direct access to expertise from a dedicated gis librarian as part of the research team. the researchers were able to use off-the-shelf technology and web gis software that was already available on campus and relatively easily installed on library computers. additionally, there was the potential to engage student workers with web gis as an experiential learning activity they might be able to put on their resume. there was also the future potential to collaborate on similar library spaces projects with regional libraries that already had gis expertise on their campuses. the authors' previous study (godfrey & stoddart, 2018) outlined in more detail the technology and technical specifics of using web gis for space assessment. the researchers and trained staff gathered data by observation during the operating hours of the curriculum center. observation data were inputted into collector for arcgis, a mobile-data collection application installed on the desktop computer used at the curriculum center circulation desk service point. data-gathering occurred when observed as opposed to randomly assigned times or via a specified schedule. staff were instructed to input the number of patrons in predefined areas and to include written notes in an open data field regarding activities occurring. during an observation, a patron might be seen traversing between different curriculum center spaces. for example, a patron might be seen looking for books in the curriculum center stacks and then taking these items to sit down and read. in such instances, when multiple actions were occurring by the same patron(s) across different learning spaces within the curriculum center, each item was recorded as a distinct observation in collector for arcgis. as such, the data recorded is more concerned with activities occurring within spaces rather than patrons themselves.   limitations with the gis pilot project   while collector for arcgis offered a ready-made tool for gathering data about space usage in the curriculum center, the project did encounter some issues as it rolled out. one of the first issues encountered was accessibility issues with the gis application related to campus computers. collector for arcgis needed to be installed on the curriculum service point computer, as well as be made accessible via all workplace computer curriculum center accounts when they signed in. this required campus it staff to become involved in installation and access of the software but also to resolve staff access issues when computer updates created unexpected problems. the it staff response time to resolve application and account issues often resulted in delays in data-gathering. while campus it delays were at times a limiting factor, collector by arcgis was already a campus approved form of software, which meant that other software might have taken even longer to support and install.   additionally, data fidelity issues arose from staff interrater reliability complications associated with the first iterations of the gis survey instrument. earlier iterations of this pilot project asked data collectors to not only indicate the number of users in a specific place but to select from a list of patron activities observed. for example, for a patron viewed in the computer area working on a homework assignment, the data collector might select composing a paper from a list of activities provided in the survey instrument. activities in the list were then mapped to official university learning outcomes (https://www.uidaho.edu/learningoutcomes) such as communicate, think and create, and others. however, based on feedback from the curriculum center staff participating in the pilot project, inputting this extra datapoint from the dropdown list was too burdensome to gather within the time constraints of recording each observation. staff also admitted confusion between learning activity items like using library computers and individual studying as being similar. in some cases, these learning activities were too subjective for accurate interpretation without being overly intrusive to the patrons (e.g., looking over a patron's shoulder). with this constructive feedback in mind, the methodology associated with data-gathering was refined and simplified to only capture usage in designated spaces. the gis observation form only asked the observer to indicate the number of people in a given area and to make a notation if the observed activity was not congruent with the learning intention behind the space design. for example, if staff observed a single student by the library computers working on a paper, they would note on the gis form that one student was in the library computer area and nothing else because the student was using the space as intended. however, if a group of students was observed around a computer, the staff person might indicate on the form the number of patrons by the computer area, but also include a notation that the observation was more akin to group study. this approach simplified data entry for staff and also captured if spaces were being used as intended or in unintended ways.   thus, the dataset reported in this article for the curriculum center did not require data collectors to assign observed patron activities but instead pre-assigned learning taxonomy activities to spaces (see figure 2 and table 1).   results: assess evidence   once the evidence has been assembled, it must be assessed within the context of the research question(s) as they have been articulated. this pilot project gathered evidence to help answer what is the student usage of the new curriculum center spaces/furniture? and can this usage data be mapped to campus learning outcomes?   what is the student usage of the new curriculum center spaces/furniture?   the gis space assessment tool was successful in documenting space usage in the curriculum center and offered evidence about how patrons were using this new space. staff recorded 1,235 observations using the gis instrument during the fall 2017 data collection period (see figure 3). in total, there were 1,837 patrons observed using the curriculum center, either individually or in groups.     figure 3 curriculum space usage – fall 2017.     the data indicated that the area patrons used most in the curriculum center was the bank of computers. staff observed 584 interactions in this space, comprised of 818 patrons. these results were not surprising as many students stopped by to print course materials or homework assignments before class. however, what is striking about usage in this space is that these computer stations were designed for individual usage. still, data and staff observations indicated that many students gathered around these computers in groups. this is supported by the evidence that 818 patrons were observed using this space, while there were only 584 interactions, which indicates that 234 students were gathering in groups. this finding suggests that this computer space might need to be re-envisioned to be more conducive for group work.    conversely, the large study table area, which was intended as a student collaboration space, only saw limited group usage as evidenced by 53 patrons using this space during 37 interactions, indicating that only 16 students used this space for group work. compare this data with the seating area, which saw 377 patrons using this space from 232 observations, suggesting that 45 students gathered in groups in this seating space, which is an intended function of this flexible area. these group table findings are open to further study or interpretation as they might suggest that patrons are uncomfortable sharing table space or that further promotion of these spaces as collaboration areas might be needed.   the curriculum center collection was another area that saw students gathering in groups, which might be unexpected to some. however, this can be attributed to the various library instruction classes and education courses that required usage of the curriculum center collections for class assignments. students were often given class time to visit the curriculum center in groups to locate materials.     figure 4 curriculum center learning taxonomy – fall 2017.     figure 5 curriculum center campus learning outcomes – fall 2017.     can this usage data be mapped to campus learning outcomes?   beyond capturing a snapshot of how the curriculum center spaces were used by students during the term, this research project also wanted to explore how such evidence might be mapped to campus learning outcomes. as previously discussed, the curriculum center spaces were assigned to a learning space taxonomy and then cross-walked further to campus learning outcomes. this mapping to outcomes allows for viewing library spaces not only in terms of usage but also within the greater overall context of a learning environment.   the most used space in the curriculum center was the computers, which were associated with the learning space taxonomy descriptor create (47.3%) (see figure 4). this was followed by focus (24.7%) as mapped to the curriculum center seating area and collections and then share (23.8%), which was assigned to the combined circulation/reference help service point. mapping and attaching usage data in this manner allow the curriculum center to not only state that they offer computers, a study table, seating, and computers, but also spaces that promote creating, sharing, focus, and collaboration.   additional mapping to the campus learning outcomes is another way to indicate to campus stakeholders how the curriculum center supports learning on campus. based on this mapping, the pilot study space usage observation data suggested that the curriculum center spaces mainly supported two campus learning outcomes with its spaces, collections, service point, and technology (see figure 5). the campus learning outcomes that the curriculum center supported are learning/integrate (51.5%) and communicate (48.5%).   the campus learning outcomes associated with the curriculum center are stated in more detail below and are intended for "students to be able to…":   learn and integrate through independent learning and collaborative study, attain, use, and develop knowledge in the arts, humanities, sciences, and social sciences, with disciplinary specialization and the ability to integrate information across disciplines.   communicate acquire, articulate, create and convey intended meaning using verbal and non-verbal methods of communication that demonstrate respect and understanding in a complex society.   these two learning outcomes align with the information literacy, research, and educational mission of both the library and curriculum center at the university of idaho. therefore, it was not surprising that the curriculum center spaces aided in supporting these learning outcomes. however, what has not always been available is usage statistics and evidence that demonstrates the ongoing contribution libraries and curriculum centers make in supporting such outcomes.     figure 6 observations/patrons to learning outcomes – fall 2017.     creating learning outcomes statements   by leveraging the space usage data and mapping to campus learning outcomes, it is now possible for the curriculum center to make stronger statements about how library spaces support learning on campus. using both observation and patron count data, the curriculum center can create prepared statements suitable for stakeholder reporting and public promotion the campus-wide learning outcomes of communication and learning and integration (see figure 6).  examples of such learning outcomes-oriented statements for the curriculum center are shared below:   curriculum center learning outcome statement: communicate the curriculum center supports the campus learning outcome of communicate by offering spaces, such as computer stations and a public demonstration space, that encourage acquiring, articulating, creating, and conveying meaning. in 2017, the curriculum center recorded 599 interactions in these communicate supporting spaces and observed 854 patrons using these resources.   curriculum center learning outcome statement: learning and integrate the curriculum center supports the campus learning outcome of learn and integrate by offering spaces, such as discipline-specific collections, flexible seating, and a group study table that encourage independent learning and collaborative study to develop knowledge and integrate information across disciplines. in 2017, the curriculum center recorded 636 interactions in these learn and integrate supporting spaces and observed 983 patrons using these resources.   once learning outcomes specific statements are created, they can be used as templates for future reporting needs, enhanced with periodically updated data as necessary. it also may be possible to create a dynamic real-time dashboard using the gis application to fill in data fields automatically.   discussion   agree   the authors agree that the web gis pilot study proved useful in gathering data to articulate space usage and map data to learning outcomes. the collector for arcgis application demonstrated its utility as an instrument for library space assessment. the mapping of specific library spaces to learning outcomes also showed merit in conveying library value beyond simple gate-counts. if viewed as a proof-of-concept methodology from both a technological and data reporting viewpoint, this research project was successful.   the authors admit there are both technological and training improvements that are required to strengthen the data collection aspects of this research. some changes would be needed if this project were to move from a pilot to a more formal assessment undertaking. first, there needs to be ongoing refinement of the gis instrument to ease and clarify aspects of data collection by staff. the application is being updated quarterly and suggested enhancements can be contributed to arcgis ideas. while there remains a slight learning curve associated with data collection for users, this was not insurmountable and remained no different than using most any other new piece of software. however, anything that could be done through technological design to make data-gathering smoother for staff would be welcomed. second, the continued and ongoing reinforcement of data collection training would be necessary to increase the interrater reliability of the observations captured. overall, the pilot project was successful in achieving its objectives to demonstrate how an off-the-shelf application could capture space usage evidence and map these to campus learning outcomes.   adapt   lessons learned from this project included the need to consider adding collaborative computer seating in the curriculum center to accommodate students who work in groups. additional promotion of curriculum center spaces such as the group study table might also be necessary.   typically, the authors would list the next steps to transition this pilot project toward a more established library assessment program. some ideas have included a real-time data dashboard, adding more descriptive survey questions to the gis instrument to capture student activities, partnering with other institutions to gather similar space usage data for peer comparison, and leveraging the location-specific aspects of gis to pinpoint which areas within library spaces are preferred by students. however, despite agreement by the researchers about the positive outcomes and potential of this pilot project, library administration did not see a suitable venue to report out the project data and felt that resources and staff time would be better spent elsewhere. despite this, there remains untapped potential for web gis applications such as collector for arcgis to assist with capturing student usage in library spaces.   conclusion   while the pilot project did not capture direct measures of learning within the curriculum center, the evidence demonstrated active student engagement within these learner-centric design spaces. additionally, these data suggested potential design improvements that might be needed in such areas to make them more functional to students. the research indicated that web gis applications, such as collector for arcgis, offer a practical and flexible tool for library space assessment. the mapping of specific library areas with a learning space taxonomy provided an opportunity to more clearly connect library efforts to learning outcomes that might more strongly resonate with stakeholders compared to traditional library usage statistics. articulating the learning value of library spaces to stakeholders demonstrated that money is not wasted and that libraries have a positive impact supporting student success.   references   asher, a. (2017). space use in the commons: evaluating a flexible library environment. evidence based library and information practice, 12(2), 68-89. https://doi.org/10.18438/b8m659   bedi, s., & webb, j. (2017). through the students’ lens: photographic methods for research in library spaces. evidence based library and information practice, 12(2), 15-35. https://doi.org/10.18438/b8fh33   bishop, b. w., & mandel, l. h. (2010). utilizing geographic information systems (gis) in library research. library hi tech, 28(4), 536-547. https://doi.org/10.1108/07378831011096213   booth, a. (2009). eblip five-point-zero: towards a collaborative model of evidence-based practice. health information and libraries journal, 26(4), 341-344. https://doi.org/10.1111/j.1471-1842.2009.00867.x   casden, j., rucker, r., aeschleman, l., davidson, b., & beswick, k. (2020). suma. in north carolina state university libraries. retrieved 17 october 2017 from https://www.lib.ncsu.edu/projects/suma   coyle, a. (2011). interior library gis. library hi tech, 29(3), 529-549. https://doi.org/10.1108/07378831111174468   elliott, r. (2014). geographic information systems (gis) and libraries: concepts, services and resources. library hi tech news, 31(8), 8-11. https://doi.org/10.1108/lhtn-07-2014-0054   ferria, a., gallagher, b., izenstark, a., larsen, p., lemeur, k., mccarthy, c., & mongeau, d. (2017). what are they doing anyway?: library as place and student use of a university library. evidence based library and information practice, 12(1), 18-33. https://doi.org/10.18438/b83d0t   fox, r., & doshi, a. (2013). longitudinal assessment of “user-driven” library commons spaces. evidence based library and information practice, 8(2), 85-95. https://doi.org/10.18438/b8761c   given, l., & archibald, h. (2015). visual traffic sweeps (vts): a research method for mapping user activities in the library space. library and information science research, 37(2), 100-108. https://doi.org/10.1016/j.lisr.2015.02.005   godfrey, b., & stoddart, r. (2018). managing in-library use data: putting a web geographic information systems platform through its paces. information technology and libraries, 37(2), 34-49. https://doi.org/10.6017/ital.v37i2.10208   jonassen, d. h., & land, s. m. (2000). theoretical foundations of learning environments. mahwah, n.j: lawrence erlbaum associates.   koufogiannakis, d., & brettle, a. (2016). being evidence based in library and information practice. chicago, il: neal-schuman.   mandel, l. h. (2010). geographic information systems: tools for displaying in-library use data. information technology and libraries, 29(1), 47-52. https://doi.org/10.6017/ital.v29i1.3158   mathews, b. & soistmann, l. a. (2016). encoding space: shaping learning environments that unlock human potential. chicago, il: acrl.   monahan, t. (2002). flexible space & built pedagogy: emerging it embodiments. inventio, 4(1), 1-19.   oblinger, d. (2006). learning spaces. washington, dc: educause.   savin-baden, m. (2003). facilitating problem-based learning: illuminating perspectives. maidenhead: society for research into higher education.   tewell, e., mullins, k., tomlin, n., & dent, v. (2017). learning about student research practices through an ethnographic investigation: insights into contact with librarians and use of library space. evidence based library and information practice, 12(4), 78-101. https://doi.org/10.18438/b8mw9q   van note chism, n. (2006). challenging traditional assumptions and rethinking learning spaces.  in d. oblinger (ed.)  learning spaces. washington, dc: educause.   xia, j. (2004). library space management: a gis proposal. library hi tech, 22(4), 375-382. https://doi.org/10.1108/07378830410570476   xia, j. (2005). visualizing occupancy of library study space with gis maps. new library world, 106(5/6), 219-233. https://doi.org/10.1108/03074800510595832 microsoft word art_shrimplin.doc evidence based library and information practice 2007, 2:4  4 evidence based library and information practice     article    a virtual standoff – using q methodology to analyze virtual reference      aaron shrimplin  207 king library  miami university  oxford, ohio, usa  email: shrimpak@muohio.edu    susan hurst  203 king library  miami university  oxford, ohio, usa  email: hurstsj@muohio.edu      received: 18 may 2007    accepted: 12 november 2007      © 2007 shrimplin and hurst. this is an open access article distributed under the terms of the creative  commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    abstract    objective ‐ to develop an exploratory understanding of reference librarians’ perceptions of  virtual reference.    methods ‐ q methodology was used to uncover points of view about virtual reference.  thirty‐four librarians sorted 28 statements covering a wide range of opinions about virtual  reference. factor analysis was used to analyze the q‐sorts and factor scores were calculated  to aid the task of understanding and interpretation.    results ‐ the factor analysis revealed three attitudinal typologies: technophiles,  traditionalists, and pragmatists. each factor represents a group of reference librarians who  think similarly about virtual reference.     conclusion ‐ this type of analysis provides data on the actual range of feelings and  attitudes about providing virtual reference services. the factor analysis demonstrates that  there are still a variety of strongly held viewpoints concerning virtual reference.  convergence towards either acceptance or rejection does not appear to be forthcoming.   mailto:shrimpak@muohio.edu mailto:hurstsj@muohio.edu http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:4  5 by using this type of analysis and the resulting data as a basis for decision making,  administrators could staff services more efficiently and with the resulting better fit between  librarians and their positions, possibly increase morale.      introduction  virtual reference is a service through which  patron and librarian communicate in real  time using a computer and communications  technology. this research focuses its  investigation on virtual reference from the  perspective of academic reference librarians  who provide this service as a part of their  job.  is there broad consensus among  academic reference librarians with regards  to their attitudes and beliefs toward virtual  reference? or are there diverse opinions and  perceptions of this relatively new service? if  there are divergent points of view, what are  they? there is an expectation that there are  competing viewpoints or perspectives  toward virtual reference and that they need  to be revealed and better understood. as we  seek to understand the long term viability of  virtual reference and, for that matter, other  technology‐enhanced services, it is also  necessary to understand librarians’ attitudes  toward this service.  if there is a contingent  of reference librarians that do not find value  in virtual reference, what does this say  about the future development of this service?  how important are consensus and shared  values among reference librarians as they  examine ways to exploit information  technologies in their traditional work?  this  research is a step towards developing a  better understanding of how reference  librarians think about virtual reference. the  findings may also be of more general  interest as the roles of librarians continue to  be shaped by information technologies.     literature review  as digital reference services become more  ubiquitous, practitioner reports have  proliferated.  this can be seen through a  search of the literature.  bernie sloan’s  “digital reference services bibliography,”  last updated in 2004, includes over 700  entries on digital reference services.   searching just on the subject term  “electronic reference servicesʺ in the library,  information sciences and technology  abstracts (lista) database returned over  390 articles, and the numbers continue to  grow as more is written on the topic.   however, despite the abundance of articles,  empirical scholarly research on the subject  has lagged. of those 394 articles, less than  9% (34) used quantitative research based on  an analysis of their abstracts.     this study is grounded in two bodies of  literature.  the first consists of articles  describing how reference librarians interact  with digital and networked technologies.   this has been an issue since the late 1990s  when digital reference services were  becoming more common.  a survey of 68  academic libraries in 1998 found that  librarians at that time reported increased  satisfaction from providing digital reference  services but were also overwhelmed by the  constant changes and felt that they needed  more instruction to be effective  (tenopir  and ennis).  support staff echoed these  sentiments in a study on perceptions and  opinions of technological change in 1999  (jones).  they felt the need to keep up with  change was challenging, and desired more  training.  however, they also felt that  technology gave them more control over  their workday, which is especially  important in a support staff position.      as more and more libraries began to  provide virtual reference, librarians  reported feeling that the context of reference  has changed (wilson). library users now  evidence based library and information practice 2007, 2:4  6 have different values, attitudes and beliefs  regarding information, and reference service  providers will have to change to meet the  users’ new needs and expectations.   one  aspect of these changes is the perceived loss  of the reference interview.  janes and hill  reported that although librarians were  generally happy with the questions they  received through digital reference, they felt  the evolving practice of reference involved  trade‐offs between traditional reference  (including a face to face reference interview)  and the new practices online.  overall,  studies continued to find librarians  generally positive about the impact and  reach of digital reference.  it is felt that  reference is now more accessible and also  more interesting and challenging to the  profession (janes).  based on these findings,  there seems to be a good amount of research  citing librarians that are positive about  digital reference, but what about those who  are not?    the second body of literature on which this  research is based is more normative,  examining the nature of reference in this  increasingly digital world?  these studies  also look to the future and consider how  reference is evolving.  here more  dissatisfaction with digital reference services  emerges.  in 2000, campbell reported that  librarians were in denial about the decline of  reference, particularly face to face reference  services.  the trend toward digital reference  is seen as another fad that is adopted, rather  than insisting on good service through face  to face reference (bridges).  bridges also  compared contemporary university library  policies with current consumer values, (i.e.  giving customers what they want).  anhang  and coffman went so far as to debate the  future of traditional reference librarians in  their article in american libraries.  while  anhang cited statistics showing a  continuing decline of in‐person reference,  coupled with little success in online  reference (at that time), coffman countered  with arguments about the strengths of  librarians in providing free reference  services and access to materials not available  online.  this debate has continued  throughout the literature, including a  “manifesto” written by jackson in 2003.  he  contends that librarianship is still a vital  profession and has much to contribute.  the  fact that instruction sessions have continued  to increase at many us association of  research libraries (arl) libraries and that  question complexity is also on the rise  indicates a need for librarians’ services in his  view.      coffman & arret wrote a two‐part series on  virtual reference in 2003.  the first  installment discussed the history of virtual  reference and how the services had changed  since their introduction in the late 1990s. the  demise of commercial on‐line reference  services is also discussed as a menace that  came and went “virtually” unnoticed.  the  second article in the series examined the  state of current virtual reference services  and made suggestions for their  improvement, including some radical ideas  such as eliminating chat reference altogether  and emphasizing phone and email reference  as alternatives.  the difficulty of getting  good usage statistics is also mentioned,  although it is agreed by all that in‐person  usage seems to be declining, however it is  measured.  they go onto discuss online  reference services operated by library  consortia, some of which have been  successful (“knowitnow” in ohio), and  others less so.     so what does this tell us?  virtual or digital  reference seems to be a part of the modern  academic library, regardless of its usage or  popularity.  while some librarians have  been largely positive about it since the  beginning, others still bemoan the loss of the  reference interview and traditional face‐to‐ face reference service.  studies of librarians’  evidence based library and information practice 2007, 2:4  7 views about online reference after providing  the service for a few years are lacking.      q‐methodology  q‐methodology is a fully developed method  for the systematic investigation of human  subjectivity (brown, “q technique and  method” 57). subjectivity in this context  refers to the communication of a person’s  own point of view about some topic. as  such, q is particularly well suited to  developing an exploratory understanding of  librarians’ perceptions regarding virtual  reference. most typically, a q study involves  three basic procedures.  first, a set of  opinion statements about some topic of  interest are collected. next, individuals read  the statements, react to them, and sort them  along a continuum of preference (usually  from “agree” to “disagree”). this operation  is known as a q sort. it is the ranking of the  statements from an individual’s own point  of view that brings subjectivity into the  picture.  lastly, once viewpoints are  modeled in q‐sorts, data are analyzed, most  often using a statistical technique called  factor analysis.  unlike survey research  which is concerned with patterns across  variables, q is interested in patterns across  individuals.  factors that emerge from the  analyzed q sorts indicate segments of  subjectivity and represent points of view.   factor scores are calculated to aid the task of  understanding.      q methodology was introduced in 1935 in a  letter to nature written by william  stevenson, a british physicist and  psychologist.  today, q methodology is  being widely adopted in the social sciences,  most notably in the fields of communication,  political science, and health sciences.   according to the international society for  the scientific study of subjectivity, a q  bibliographic database maintained at q‐ method, a website devoted to the practice of  q, has over 2500 entries. in recent years, the  q technique and its methodology has  broadened its appeal and been applied in a  number of intellectual fields, including  marketing, religion, and women’s studies  (brown, “history and principles of q”).  in  the field of academic librarianship, however,  q methodology is relatively unknown,  except for one study that reports how a q  sort was used as a technique to prioritize  journal titles as candidates for possible  cancellation (dick and edelman).     with the small sample sizes required and  equally small costs associated with it, q  methodology is well suited to the field of  library research.   the ability to get at  subjective opinions but yet still analyze  them in a quantitative manner also uniquely  suits the field, where anecdotal evidence is  more often obtained rather than hard data.   while it is often assumed that subjectivity –  a person’s point of view – is impossible to  study systematically and with any degree of  precision, q methodology provides  researchers with a systematic and  quantitative means for examining  subjectivity. this study uses q methodology  to address the following questions. is there  broad consensus among academic reference  librarians with regards to their attitudes and  beliefs toward virtual reference? or are  there diverse opinions and perceptions of  this relatively new service? if there are  divergent points of view, what are they?  once this information is obtained, it can be  used for practical purposes such as  improving staffing decisions and utilizing  staff more effectively, while at the same time  improving morale and perhaps even  increasing staff retention.      methods  the statements selected for a q sort are  drawn from what is called a “concourse.”  a  concourse can be understood as the volume  of discussion on any topic. it is the task of q  methodology to reveal the form and  structure of a concourse.  there are a  evidence based library and information practice 2007, 2:4  8 number of ways to capture a concourse.   typically, interviews are undertaken to  collect views on a topic.  this study took a  slightly different approach: gathering close  to 100 statements from a number of different  sources including essays, discussion groups,  blogs, and conference presentations where  people expressed opinions about virtual  reference. statements were not edited.  from  this collection of statements about virtual  reference service, 28 were chosen that  sample the diversity of opinions on virtual  reference.  there are two basic techniques  for selecting statements to be used in a q  sort, structured and unstructured sampling  (mckeown and thomas 28). this study  selected statements based on unstructured  sampling, a process where statements  presumed to be relevant to the topic of  investigation are chosen without undue  concern that coverage of all possible sub‐ issues are exposed.  from the initial sample  of just under 100 statements, the researchers  determined that 28 statements adequately  represented the diversity of opinions about  virtual reference without favoring some to  the exclusion of others. additional  statements could have been added to the q‐ sample, but the researchers were concerned  that a larger number of statements would  deter participants from completing the q‐ sort.    q is an intensive form of analysis and  involves small numbers of subjects.  as such,  it makes no claims about being statistically  representative of some larger population. its  purpose is to present a picture of the types  of thinking that exist in relation to a specific  topic or issue.  it does not tell what  proportion of the population subscribes to a  particular way of thinking. to answer this  question, a traditional, large n‐sample  survey would need to be administered  (mckeown and thomas 37).    the intent of this research was to draw a  sample that represents a cross‐section of  academic reference librarians from three  sizes of libraries: large, medium, and small.  to this end, data from the 1998 integrated  postsecondary education data system  (ipeds) survey of academic libraries was  used to create a list of libraries ranked by  the size of their professional staff (national  center for education statistics, usa).  from  this list of over 3,000 libraries, 37 libraries  were selected. the researchers initially  selected 42 libraries (12 from each size of  library), but it was determined via an  investigation of the libraries websites that  not all libraries selected for the study  provided a virtual reference service.    in the spring of 2005, reference librarians  from the 37 libraries were contacted by mail  to participate in the q study. librarians  were encouraged to give the q study to  another colleague if they were unable or  unwilling to participate in the study.  included in the mailing was a letter  describing the study, a consent form to be  signed and returned, and a deck of 28  statements about virtual reference (see  appendix 1). also included in the packet  was a step‐by‐step guide for how to sort the  statements (known as a “condition of  instruction”) and a score sheet to record the  order of the statements. (see   appendices 2 and 3, respectively). they also  completed a short questionnaire about their  demographic information and time spent  doing reference.    data analysis  using pqmethod, a statistical program  tailored to the requirements of q studies,  each q‐sort was intercorrelated with the  others and a 34 x 34 correlation matrix was  factor analyzed using the principal  component method (schmolck).  three  unrotated factors were extracted and rotated  using a varimax rotation. mckeown and  thomas provide an excellent description of  the statistical procedures used in q  methodology.  evidence based library and information practice 2007, 2:4  9 factor scores were then computed for all  three factors to reveal clusters of opinion.  in  this context, a factor represents a group of  academic reference librarians who have q‐ sorted the 28 statements essentially the same,  thus demonstrating a distinct point of view  toward virtual reference.      results  a total of 34 librarians sorted the 28  statements according to their degree of  agreement or disagreement into a forced  distribution grid that resembles a normal  bell‐shaped curve. in some cases, more than  one librarian per institution completed the  q sort. librarians were encouraged to share  the study with a colleague; in some cases  this happened, while in other institutions no  librarians completed the survey.    rotated factor matrix  table 1 presents the rotated factor matrix  and suggests that the three factor solution is  adequate given the fact that 29 of the 34 q‐ sorts loaded significantly on only one factor.  a factor loading is a measure of how  saturated a subject is on a given factor.   loadings in excess of ±.49 are significant at  the .01 level and are placed in parentheses  for convenience.  while the authors can  make no claim that the three factors brought  to light here are exhaustive of all possible  points of view, they do represent three  distinctive ways of thinking that exist  among reference librarians. as presented in  table 1, 14 librarians loaded significantly  upon factor 1, nine upon factor 2, and six  upon factor 3. five librarians did not load  significantly on any factor. as table 1  reveals, four confounded sorts were  identified, which loaded significantly upon  two sets of factors. these sorts were not  used to define the factors, but their sorts  were viewed by the authors (alongside each  idealized q‐sort) as part of the interpretative  process in describing the factors.  factor analysis  the factor analysis of the 34 reference  librarians revealed three factors or  attitudinal typologies: technophiles (factor  1), traditionalists (factor 2), and  pragmatists (factor 3). labels are attached to  the factors to enhance understanding of  each groups’ attitudes toward virtual  reference.  a description of each group is  given below. each factor represents a group  of reference librarians who think similarly  about virtual reference. these descriptions  and their labels were derived by looking at  the q sorts that help define each factor. to  further aid in the interpretative process, an  idealized q‐sort can be computed for each  factor that represents how a hypothetical  individual loading 100% on a factor would  order the 28 statements.  in table 2, the  authors report the scores of all 28 statements  in the idealized q sort for each of the three  factors (factor 1, factor 2, and factor 3). this  table reveals in a general way how reference  librarians of that type think about virtual  reference. an idealized q sort is used in the  interpretative process to help compare and  describe the distinctive ways of thinking  about virtual reference; it allows researchers  to view how q‐items are distinguished from  each other across the factors. in interpreting  the three factors and their respective points‐ of‐view, it is important to understand that  this examination is not done simply by  pulling out statements with extreme scores  on each factor.  researchers need to take  into account how the statements are placed  in relation to one another in each factor and  the comparative placement of statements in  different factors. when analyzing the data,  the researcher “listens to the data” and tells  the story using a qualitative process. some  items do not add up at the individual  statement level, but when looked at more  holistically, they do make sense. the  narrative below aims to capture this.   evidence based library and information practice 2007, 2:4  10     factor loadings  participant  gender  year mls or  equivalent  degree  hours per  typical  week doing  reference  questions  factor 1  factor 2  factor 3  1  female  1996  7  .41  (.69)  .03  2  male  1993  6  .32  (.55)  .21  3  female  1996  10  .35  (.62)  .42  4  nr  nr  nr  (.73)  .04  .39  5  female  1998  10  (.56)  .34  (.50)  6  male  1979  12  (.53)  .05  (.62)  7  female  2004  10  (.69)  .23  .30  8  male  2001  5  .41  ‐.03  .48  9  female  1973  10  (.73)  .02  .11  10  female  1982  5  (.62)  .08  .07  11  female  2002  12  .44  (.71)  .11  12  male  2004  15  (.82)  .15  .31  13  female  1978  2  .48  .25  (.51)  14  female  1976  6  .06  .34  (.64)  15  female  1977  12  ‐.37  (.73)  .13  16  female  1994  8  ‐.23  (.73)  ‐.03  17  female  2001  12  .29  (.57)  .26  18  female  1998  8  (.76)  .07  .43  19  female  1987  40  .46  .12  (.59)  20  male  2004  8  (.72)  .25  .40  21  female  1999  4  (.64)  .26  ‐.05  22  male  1974  12  ‐.30  .28  (.55)  23  female  2002  10  (.77)  .18  .17  24  female  2004  10  (.83)  ‐.20  .01  25  female  1995  5  (.50)  (.53)  .21  26  female  1989  12  (.57)  .45  .34  27  female  2004  10  (.73)  .20  .46  28  female  1991  6  .24  ‐.05  (.74)  29  male  1973  10  .23  .16  (.73)  30  female  1996  12  (.71)  ‐.13  .48  31  male  1991  12  ‐.42  (.74)  .12  32  female  1992  4  (.61)  (.55)  .19  33  female  1984  11  .15  (.84)  ‐.02  34  female  1976  14  (.79)  ‐.01  .46    table 1. rotated factor matrix    note: loadings placed in parentheses are significant (p < .01).      evidence based library and information practice 2007, 2:4  11 factor 1: technophiles   technophiles believe that virtual reference is  a necessary and valuable new service. to  their thinking, “reference is reference,  regardless of format.” there is no one way to  do reference, and perhaps more importantly,  one format is not necessarily any better than  another. they perceive virtual reference as a  creative solution to putting librarians in  touch with their users for whom  communicating online is a way of life.  it is  their opinion that for many students the  choice is between asking a question online  and not asking a question at all, and  reference librarians have a professional  responsibility to find ways to do their jobs in  this kind of environment.      factor 2: traditionalists   traditionalists believe that traditional  reference, that is to say, face‐to‐face and  over the phone, is the best kind of service  libraries can offer to their patrons.  they  view virtual reference as a mistake because  it fails to offer a quality reference service – it  is impersonal, inefficient, and sub par.   traditionalists view virtual reference as an  expensive, trendy service that is pushed by  individuals who do not understand  reference. from their perspective, virtual  reference is fast‐food reference promoted in  the name of good customer service.    factor 3: pragmatists   pragmatists believe that virtual reference  will never equal the potency and  effectiveness of on‐site, face‐to‐face  reference.  in this regard, they have common  ground with traditionalists.  they feel  strongly that face‐to‐face reference is the  most effective way to meet users’  information needs.  but unlike the  traditionalists, they believe virtual reference  is a valuable service. like the technophiles,  pragmatists are keenly aware that younger  people habitually communicate online.  and  unlike the traditionalists, they are not  willing to penalize those who use online  communication tools just because there is a  better, more effective reference service  model (i.e., face‐to‐face).  taken as a whole,  pragmatists are less likely to do something  just because it is easier for library users.  in  this regard, they are somewhere in between  the “give ‘em what they want” approach  supported by technophiles and the “we  know what is best” approach supported by  traditionalist. while all three groups agree  that not every librarian has to do it all, it is  clear that traditionalists believe that some  librarians should not be required to staff  virtual reference services, while pragmatists  are less likely to support a service model  that splits virtual reference among those  librarians that enjoy it and those that do not.  factors   1  technophiles  2  traditionalist  3  pragmatists  1.  i like providing digital reference service more  than other kinds of reference.  ‐1  ‐3  ‐3  2.  i have found myself debating whether to tell  someone how to do something online, which  would mean a lot of typing, or just pushing  the results of what i’ve done for them.   0  +1  ‐1  3.  we now have an entire generation of young  people for whom text messaging is a primary  means of communication. if we ignore that,  we risk marginalizing ourselves.  +2  ‐2  +2  4.  the idea for real‐time, computer based  ‐3  +2  ‐3  evidence based library and information practice 2007, 2:4  12 reference assistance rarely comes from  reference librarians who must deliver this  service.  5.  librarians should spend their resources on  making it easier for patrons to find  information on their own.  +1  ‐1  0  6.  i have anxiety over whether i am spending  too much time or not getting answers back  fast enough.  ‐1  ‐2  0  7.  everyone has different learning styles, so we  shouldn’t try to force one way of doing  reference on every student.  +2  +1  +1  8.  virtual reference is just another way to  communicate with human reference  librarians, along with email, phones, and  paper mail.  +3  +1  +3  9.  instead of spending time and money  developing and running chat services, we  should spend it on improving our websites  and information systems.  ‐1  0  ‐2  10.  for all of the hype about reaching out in  extraordinary ways and in unusual times,  virtual reference fails our users. it doesn’t  meet their information needs efficiently, and  it doesn’t deepen their research capacities.  ‐3  +2  0  11.  add it all up and you can see that virtual  reference built on chat technology is a pretty  expensive proposition, especially if we ever  came close to getting the thousands and  thousands of questions many of us expected.  ‐2  +2  ‐1  12.  quick factual questions are the best because  we don’t have the body; anything that has  depth should be a consultation.  0  0  0  13.  i worry about a new breed being great online  but having no face‐to‐face skills at all.  ‐2  ‐1  ‐2  14.  we need to meet the patron where they are  and with the methods they use.  +3  +1  ‐2  15.  most people don’t like chat for serious  information because they have other and less  costly reference options, namely email and  telephone.  ‐1  ‐1  ‐1  16.  for many students the choice may be  between asking a question virtually or not  asking it at all.  +2  0  +1  17.  only occasionally is it necessary to engage in  back in forth to get a good idea of the  question.  ‐2  ‐2  ‐2  evidence based library and information practice 2007, 2:4  13 18.  yes, i give better service face‐to‐face, and i  dislike the limitations of chat.  but time and  convenience is very important to the students  and i’d rather provide half‐a‐loaf than  nothing at all.  +1  0  0  19.  the reference interview is not as necessary in  the digital reference setting.  ‐3  ‐3  ‐3  20.  in‐person, genuine real time reference  involves moral and emotional elements that  are essentially unattainable through  disembodied online interaction.  ‐2  0  ‐1  21.  i have absolutely no regrets about enhancing  our reference services through chat.  +1  ‐3  +2  22.  virtual reference can never equal the potency  and effectiveness of on‐site, in‐house, in‐ place, and wholly interactive traditional  reference practice.  0  +2  +3  23.  traditional reference – face‐to‐face interaction  and over‐the‐phone discussion – still  provides the best all around service to our  patrons?  0  +3  +2  24.  why do we wish to penalize those who have  adapted to the online world and its  technologies by lamenting the fact that they  don’t use the old technologies?  0  ‐1  +1  25.  i’m glad to have the opportunity to assist our  students in whatever manner they choose to  contact us. i’m here to provide reference –  and reference is reference – regardless of  format.  +3  0  +3  26.  one of the problems with trying to conduct a  good reference interview in a chat session is  that the patron is often in a great hurry.  0  +3  0  27.  a basic knowledge of how to instant message  or chat is an essential skill held by and  needed in our librarians.  +1  ‐2  +2  28.  the institution should offer the gamut of  reference service/contact – i’m not sure that  means every staff member has to be able to  do it all.  +2  +3  +1    table 2. statement scores for each factor      from the demographic profiles of the  respondents (table 1), it can be seen that  over three‐quarters (n=26) of the librarians  involved in the study were female. the  length of time since the respondents had  received their degrees in librarianship  evidence based library and information practice 2007, 2:4  14 varied from just a few years to over 30, so a  wide range of experience was included.    there was also a wide ranging response to  the questions regarding how much time  each respondent typically spent answering  reference questions in all formats (in‐person,  virtual, telephone, etc.).  this varied from 2  hours per week to 40.  taking these aspects  into account, the results show the strength  of the differing viewpoints and how they  continue to differ even at this point in the  process.  thus, not all newly credentialed  librarians think one way versus those with  more experience automatically fitting into  another factor.  caution should be taken to  avoid reading too much into the  demographic profile, however.  as an  intensive form of analysis, q helps present a  picture of the types of thinking that exist  about a given issue; it does not tell us  though what proportion of the population  subscribes to a particular way of thinking. a  suggestive pattern might be seen that may  make sense from a theoretical standpoint,  but further study would be required.    virtual reference has been around in some  form or another in many institutions for 10  years or more, making it a standard part of  many library positions.  however, despite  (or perhaps because of) the length of time  virtual reference has been in existence, it can  be seen that there are still definite feelings  and arguments for and against this service.   this range of attitudes from technophiles to  traditionalists, shows no signs of narrowing;  in fact, it may be growing, as librarians  spend more time providing this service and  observing its successes and failures.    the common wisdom is that when a new  service like this is started that the “early  adopters” will get it going and then they  will win over the majority of their  colleagues, eventually achieving a critical  mass or at the very least, “buy in.”  this  may not be the case.  not all reference  librarians think alike.  however, the fact that  all participants in the study provide virtual  reference services, despite their feelings or  attitudes says something as well.  even  traditionalists have been required or at least  strongly encouraged to staff virtual  reference, despite their feelings to the  contrary.  this may or may not be a good  idea.  making square pegs fit into round  holes is inefficient and can result in poor  service and poor morale and high staff  turnover.    discussion   most of the research on virtual reference  services has focused on the patron end and  their level of satisfaction.  few researchers  have looked at those performing the service,  the librarians themselves.  a recent study by  steiner and long did look at librarians’  opinions about using instant messaging (im)  to provide virtual reference and found  mixed opinions on the subject.   other  research found that reference librarians  were optimistic and positive about the  impact and reach of digital reference overall  (tenopir and ennis 84).   however, our  study reveals more of a diversity of opinion.   while this can be seen as a natural outcome  of surveying a variety of individuals, this  type of analysis goes beyond the reporting  of attitudes and provides data on the range  of subjective feelings and opinions about  providing digital reference service.     based on our findings, we can surmise that  some librarians (the traditionalists) are  performing a balancing act (one of many  performed each day, probably) between  their own opinions about virtual reference  and the need to be a team player and assist  their colleagues with staffing the service.  in  other cases, technophiles may feel unduly  constrained by those around them that do  not share their enthusiasm for this latest  piece of technology.  by providing data that  can be used as a basis for making staffing  decisions, this type of analysis could go a  long way toward relieving librarians of at  evidence based library and information practice 2007, 2:4  15 least some aspects of their balancing acts.    more research into librarians’ opinions and  attitudes concerning virtual reference would  be helpful to add to the findings.    by using tools such as q methodology, this  data can be utilized by supervisors or  library administers to make the most  efficient and effective use of their staff.    once the factors are known and identified,  staff can be queried as to their own opinions  about the various forms of reference they  provide.  these can then be used to  determine where they fit into the three  groups (technophiles, pragmatists or  traditionalists) and these factors can in turn  be taken into account when parsing out  duties and responsibilities.  after all, as  statement 28 puts it, “the institution should  offer the gamut of reference service/contact  – i’m not sure that means every staff  member has to be able to do it all.” an end  result might include some form of job  sculpting, where individuals are allowed to  focus on their strengths and minimize their  activities in areas where they feel less  comfortable.  this in turn could improve  morale and perhaps even improve  productivity by allowing persons to perform  primarily those duties at which they are the  most skilled.    the data can also be used to assess training  needs and who would benefit most from  training in specific areas.  again, the more  supervisors or administrators can learn  about their staff and their respective  strengths and weakness (and likes and  dislikes), the more effective staffing can be  and the more smoothly the entire facility can  be run.    playing to people’s strengths is the  ideal, but that can only be done once those  strengths are known.    specific ideas for further research include  performing a similar study looking at  groups of librarians within an institution  and determining the range of attitudes  among colleagues.  this same type of  research could also be carried out at  institutions not currently involved with  virtual reference and seeing if their feelings  fell into different categories or were more  skewed one direction or another.  yet  another idea would be to consider other  types of technology (e‐books, for example)  and see if similar ranges of opinions exist  and more interestingly to see if the same  respondents as participated in this study  were consistent in their opinions.  is a  technophile always a technophile?  is it  more of a personality trait or is it situational  in nature?  is there a progression from one  perspective to the next?  if so, does it only  go in one direction?      knowing more about people’s attitudes and  opinions can be a step towards an improved  working environment and improved patron  service.  librarians are unlikely to always  think or feel in the same way, but  technology and new ways of doing things  are certainties.  the better able we are to  understand the range of feelings  surrounding these changes; the better able  we will be to work with them, no matter  which q factor we fall into.    conclusion  this study used q methodology to develop  an exploratory understanding of how  reference librarians perceive or think about  virtual reference.  while previous research  found librarians generally positive about  virtual reference, the results presented here  suggest that there are competing viewpoints  toward virtual reference. this in turn raises  the following questions.  how important is  consensus and shared values among  librarians as they examine ways to exploit  information technologies in their traditional  work?  must everyone share the same  values and more importantly the same roles?   and lastly, can we use these findings to  encourage administrators and librarians to  work together to find the mix of roles and  evidence based library and information practice 2007, 2:4  16 responsibilities that work best for each  individual?      works cited    anhang, abe, and steve coffman. “the  great reference debate.” american  libraries 33.3 (2002): 50‐4.    bridges, karl. “why traditional  librarianship matters.” american  libraries 32.10 (2001): 52‐4.    brown, steven r. the history and principles  of q methodology in psychology and  the social sciences. 24 jan. 2007  <http://facstaff.uww.edu/cottlec/qarch ive/bps.htm>.    ‐‐‐. “a primer on q methodology.” operant  subjectivity 16 (1993): 91‐138.    ‐‐‐. “q technique and method: principles  and procedures.” new tools for social  scientists. ed. william berry and  michael lewis‐beck. beverly hills, ca:  sage publications, inc., 1986. 57‐76.    campbell, jerry d. “clinging to traditional  reference services: an open invitation  to libref.com.” reference and user  services quarterly 39 (2000): 223‐7.    coffman, steve, and linda arret. “to chat  or not to chat – taking another look  at virtual reference.” searcher 12.7  (2004): 38‐46.    ‐‐‐. “to chat or not to chat.” searcher 12.8  (2004): 49‐56.    dick, margaret jorgensen and marla  edelman. “consequences of the budget  crunch: using q methodology to  prioritize subscription cancellations.”  journal of nursing education 32 (1993):  181‐2.  international society for the scientific study  of subjectivity. 22 jan. 2007  <http://www.qmethod.org/>.    jackson, michael g. “the great reference  debate continued – with a manifesto.”  american libraries 34.5 (2003): 50‐2.    janes, joseph. “digital reference: reference  librarians’ experiences and attitudes.”  journal of the american society for  information science and technology 53  (2002): 549‐566.    janes, joseph, and chrystie hill. “finger on  the pulse.” reference & user services  quarterly 42 (2002): 54‐65.    jones, dorothy e. “ten years later: support  staff perceptions and opinions on  technology in the workplace.” library  trends 47 (1999): 711‐40.    mckeown, bruce, and dan thomas. q  methodology. newbury park,  california: sage publications, inc., 1998.     national center for education statistics.  ipeds dataset cutting tool.  3 feb.  2005  <http://nces.ed.gov/ipedspas/dct/index. asp>.    schmolck, peter. qmethod page. 30 dec.  2004  <http://www.lrzmuenchen.de/~schmol ck/qmethod/>.    sloan, bernie. digital reference services  bibliography. 5 jan. 2007  <http://people.lis.uiuc.edu/~b‐ sloan/digiref.html>.    steiner, sarah and casey long.  “what are  we afraid of? a survey of librarian  opinions and misconceptions  regarding instant messenger”.   reference librarian 47 (2007): 31‐50.  http://facstaff.uww.edu/cottlec/qarchive/bps.htm http://facstaff.uww.edu/cottlec/qarchive/bps.htm http://www.qmethod.org http://nces.ed.gov/ipedspas/dct/index.asp http://nces.ed.gov/ipedspas/dct/index.asp http://www.lrzmuenchen.de/~schmolck/qmethod http://www.lrzmuenchen.de/~schmolck/qmethod http://people.lis.uiuc.edu/%7eb%e2%80%90sloan/digiref.html%00%00 http://people.lis.uiuc.edu/%7eb%e2%80%90sloan/digiref.html%00%00 http://people.lis.uiuc.edu/%7eb%e2%80%90sloan/digiref.html%00%00 evidence based library and information practice 2007, 2:4  17   stephenson, william. “technique of factory  analysis.” nature 136 (1935): 297.    tenopir, carol, and lisa ennis. “the impact  of digital reference on librarians and  library users.” online 22 (1998): 84‐7.    wilson, myoung c. “evolution or entropy.”  reference & user services quarterly 34  (2000): 387‐400.    evidence based library and information practice 2007, 2:4  18 appendix 1 – q sample     1. i like providing digital reference service more than other kinds of reference.    2. i have found myself debating whether to tell someone how to do something online,  which would mean a lot of typing, or just pushing the results of what i’ve done for them.    3. we now have an entire generation of young people for whom text messaging is a  primary means of communication. if we ignore that, we risk marginalizing ourselves.    4. the idea for real‐time, computer based reference assistance rarely comes from reference  librarians who must deliver this service.    5. librarians should spend their resources on making it easier for patrons to find  information on their own.    6. i have anxiety over whether i am spending too much time or not getting answers back  fast enough.    7. everyone has different learning styles, so we shouldn’t try to force one way of doing  reference on every student.    8. virtual reference is just another way to communicate with human reference librarians,  along with email, phones, and paper mail.    9. instead of spending time and money developing and running chat services, we should  spend it on improving our websites and information systems.    10. for all of the hype about reaching out in extraordinary ways and in unusual times,  virtual reference fails our users. it doesn’t meet their information needs efficiently, and it  doesn’t deepen their research capacities.    11. add it all up and you can see that virtual reference built on chat technology is a pretty  expensive proposition, especially if we ever came close to getting the thousands and  thousands of questions many of us expected.    12. quick factual questions are the best because we don’t have the body; anything that has  depth should be a consultation.    13. i worry about a new breed being great online but having no face‐to‐face skills at all.    14. we need to meet the patron where they are and with the methods they use.      15. most people don’t like chat for serious information because they have other and less  costly reference options, namely email and telephone.    evidence based library and information practice 2007, 2:4  19 16. for many students the choice may be between asking a question virtually or not asking it  at all.    17. only occasionally is it necessary to engage in back in forth to get a good idea of the  question.    18. yes, i give better service face‐to‐face, and i dislike the limitations of chat.  but time and  convenience is very important to the students and i’d rather provide half‐a‐loaf than  nothing at all.    19. the reference interview is not as necessary in the digital reference setting.    20. in‐person, genuine real time reference involves moral and emotional elements that are  essentially unattainable through disembodied online interaction.    21. i have absolutely no regrets about enhancing our reference services through chat.    22. virtual reference can never equal the potency and effectiveness of on‐site, in‐house, in‐ place, and wholly interactive traditional reference practice.    23. traditional reference – face‐to‐face interaction and over‐the‐phone discussion – still  provides the best all around service to our patrons?    24. why do we wish to penalize those who have adapted to the online world and its  technologies by lamenting the fact that they don’t use the old technologies?    25. i’m glad to have the opportunity to assist our students in whatever manner they choose  to contact us. i’m here to provide reference – and reference is reference – regardless of  format.    26. one of the problems with trying to conduct a good reference interview in a chat session  is that the patron is often in a great hurry.    27. a basic knowledge of how to instant message or chat is an essential skill held by and  needed in our librarians.    28. the institution should offer the gamut of reference service/contact – i’m not sure that  means every staff member has to be able to do it all.                      evidence based library and information practice 2007, 2:4  20 appendix 2 – sorting instructions      the objective here is to sort the statements along the continuum from the ones that you most  disagree with to the ones that you most agree with.    1. look at all the opinion statements to familiarize yourself with the range of issues.  2. sort the issues into 2 piles. one should contain the statements that you agree with ‐‐ for  any reason.  the other pile contains those statements that you do not agree with ‐‐ for any  reason.  the piles do not have to contain equal number of statements.  3. from the pile of statements you agree with, select the three items (only three) that you  most agree with. place them in a three‐item column at the extreme right of your  workspace.  4. from the remaining agree pile, select four more issues that are now more agreeable to  you than the others in the pile.  place these four statements in another column just to the  left of the three already selected in step 3 above.  5. next, select from the remaining agree pile the four statements that you now agree with  the most.  place these four statements in another column just to the left of the four  already selected in step 4 above.  6. next, select from the remainder of the agree pile the six statements that you agree with  the most.  place these six statements in another column just to the left of the four already  selected in step 5 above. if you have run out of statements in the agree pile and cannot  finish step 6, proceed immediately to the next step. if you have extra unsorted statements  at the end of this step, combine the extras with the not agree pile and go on to the next  step.  7. now, work with the pile of statements you feel you do not agree with.  begin by  selecting the three statements you find most disagreeable.  place them in a three‐item  column on the far‐left side of your work area.  8. from the remaining most disagree pile, select four more statements that are now more  disagreeable to you than the others in the file. place these four statements in another  column just to the right of the three already selected in step 7 above.  9. next, select from the remaining disagree pile the four statements that you disagree with  the most.  place these four statements in another column just to the right of the four  already selected in step 8 above.  10. place any remaining statements in the middle of your grid.  11. now, look at your arrangement. feel free to move issues around to make sure that you  opinion is reflected correctly.  12. when everything is sorted as you want it to be, write the statement numbers in the blank  boxes in the grid on your answer sheet and answer the remaining questions of the form.                    evidence based library and information practice 2007, 2:4  21   appendix 3 – q‐sort grid    most disagree                     most agree    ‐3  ‐2  ‐1  0  +1  +2  +3                                                                                                                    research article   library supported open access funds: criteria, impact, and viability   amanda b. click business librarian bender library american university washington, district of columbia, united states of america email: aclick@american.edu   rachel borchardt associate director, research and instructional services bender library american university washington, district of columbia, united states of america email: borchard@american.edu   received: 13 aug. 2019                                                                  accepted: 12 oct. 2019      2019 click and borchardt. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29623     abstract   objective – this study analyzes scholarly publications supported by library open access funds, including author demographics, journal trends, and article impact. it also identifies and summarizes open access fund criteria and viability. the goal is to better understand the sustainability of open access funds, as well as identify potential best practices for institutions with open access funds.   methods – publication data was solicited from universities with open access (oa) funds, and supplemented with publication and author metrics, including journal impact factor, altmetric attention score, and author h-index. additionally, data was collected from oa fund websites, including fund criteria and guidelines.   results – library oa funds tend to support faculty in science and medical fields. impact varied widely, especially between disciplines, but a limited measurement indicated an overall smaller relative impact of publications funded by library oa funds. many open access funds operate using similar criteria related to author and publication eligibility, which seem to be largely successful at avoiding the funding of articles published in predatory journals.   conclusions – libraries have successfully funded many publications using criteria that could constitute best practices in this area. however, institutions with oa funds may need to identify opportunities to increase support for high-impact publications, as well as consider the financial stability of these funds. alternative models for oa support are discussed in the context of an ever-changing open access landscape.     introduction   libraries have been supporting open access (oa) publishing for more than a decade, often by administering funds dedicated to paying article processing charges (apcs). the literature provides some insight into the design, implementation, and evaluation of library oa funds, but no study has collected and analyzed the scholarship published using these funds. this study involved building a dataset of almost 1,200 publications funded by library oa funds collected from 16 universities. the authors compiled descriptive statistics and conducted an analysis of the research impact of a subset of the publications. in addition, the details and criteria of 55 active library oa funds were collected in order to better contextualize impact and identify trends in funding models.   the scholarly communications landscape is currently in a state of flux. plan s was rolled out in the fall of 2018, with the goal of “making full and immediate open access a reality” (coalition s, n.d.). the university of california system has made headlines by canceling access to elsevier after failing to agree on funding for oa publications (kell, 2019). librarians are exploring options and deciding how to best support oa efforts, and this research will inform these efforts. those considering the implementation of a new fund, thinking about making changes to funding support for oa, or designing marketing and outreach plans around oa may find the results of this study to be useful.   literature review   in knowledge unbound, suber (2016) defines the apc in this way:   a fee charged by some oa journals when accepting an article for publication, in order to cover the costs of production. it’s one way to cover production costs without charging readers and erecting access barriers. while the invoice goes to the author, the fee is usually paid by the author's funder or employer rather than by the author out of pocket. (p. 413).   university of california berkeley librarians laid out their argument for institutional open access funds as early as 2010 (eckman & weil, 2010). that same year, however, an opinion piece in d-lib magazine argued against institutional funds for paying gold oa apcs in favor of green oa self-archiving mandates (harnad, 2010). regardless, north american libraries have been providing oa funds to pay apcs since 2008, according to sparc’s (2018) open access funds in action report. often these funds combine gold oa with green oa by paying apcs but also requiring authors to deposit manuscripts in the institutional repository.   the research on open access funds is sparse, and generally focuses on surveying librarians about perspectives on oa, or collecting feedback from fund recipients. there are also a number of case studies describing the implementation of specific oa funds (pinfield, 2010; price, engelson, vance, richardson, & henry, 2017; sinn, woodson, & cyzyk, 2017; zuniga & hoffecker, 2016), which will not be discussed in this review of the literature. similarly, while concerns about the rise of so-called predatory publishing have been well documented, their implications for open access funds have not been well researched (berger, 2017).   an international survey of libraries published in 2015 showed that almost one quarter of the respondents offered oa funding to authors provided by the institutional administration, library or academic departments (lara, 2015). librarians surveyed about their libraries’ funds all used these funds to promote oa on their campuses to some degree. monson, highby, and rathe (2014) found that some were “ambitious advocates” who hoped for “significant changes in campus culture,” while others simply hoped to convince faculty to consider oa publishing a viable option (p. 317-318). a survey of faculty at large public universities that explored opinions about and behaviors toward oa demonstrated that respondents had varying expectations of library oa funding. around 30% of total respondents felt that the library should not be expected to pay apcs, while half of the life sciences or medical faculty felt that it was appropriate for the library to contribute from $500 to $4,000 for apcs (tenopir et al., 2017).   in 2015, librarians at grand valley state university surveyed the 50 recipients who received funds to pay oa article processing charges over the 4 years that the fund had been active. most faculty indicated that they chose to publish oa in order to increase the visibility of their work. many expressed support for the oa movement, and noted that they would not have been able to pay the apc without the library oa fund (beaubien, garrison, & way, 2016). university of california berkeley librarians also surveyed the 138 recipients of apc funding from the berkeley research impact initiative (brii). funding recipients felt that “that their articles received more attention and had a greater impact that they might have had in a subscription journal” (teplitzky & phillips, 2016).    aims   this study was designed to explore the impact of the literature supported by library oa funds, as well as summarize fund guidelines and criteria. our research questions include: what types of authors and publications are libraries supporting with oa funds? what is the research impact of these publications? how are library oa funds structured and maintained? answering these questions allowed us to consider of future viability of oa funds in academia, as well as identify trends and potential best practices for institutions looking to establish or evaluate an oa fund.   methods   dataset collection   using sparc’s 2016 list of library oa funds, we contacted 63 college and university libraries to request data on funded oa publications (scholarly publishing and academic resources coalition [sparc], 2018). we provided a spreadsheet template (see appendix a for included fields) with instructions to either send existing data or complete as much of the template as possible. the 16 libraries listed in table 1 responded. from these responses we built a dataset of almost 1,200 articles, including data on discipline, authorship, journal, publisher and doi. we chose a subset of 453 articles – those published in 2014 and 2016 – for additional impact analysis.   table 1 list of universities that contributed funded article information to the study dataset george mason university university of massachusetts amherst johns hopkins university university of north carolina at greensboro university of california, irvine university of oklahoma university of california, san francisco university of pennsylvania university of california, santa barbara university of pittsburgh university of california, santa clara university of rhode island university of colorado boulder virginia tech university of iowa wake forest university   impact analysis   in march 2019, we collected citation counts and altmetric attention scores for each article published in 2014 and 2016 using the dimensions database (digital science, n.d.-b). we also collected journal impact factors (jif) from journal citation reports and scimago journal ranking (sjr) from scimagojr for each journal, along with their inclusion status in the directory of open access journals (doaj). finally, we used web of science to identify the higher h-index between the first and last author of each article for 450 of 453 publications. we were unable to find author information in web of science for three articles.   to compare the relative impact of the articles in our dataset to that of similar publications, we measured the average weighted relative citation ratio of all 2014/2016 plos publications in our dataset as compared to all plos articles published in the middle (late june/early july) of the same year (“relative citation ratio,” 2017).   fund identification and criteria analysis   the november 2018 version of the sparc open access funds in action sheet listed 64 current and former college and university oa funds (scholarly publishing and academic resources coalition [sparc], 2018). to update this list, we searched google for additional funds, using the search statement “site:.edu ‘open access fund.’” we found an additional 23 oa funds, for a total of 87 identified funds. note that the sparc list is based on self-reported data, and thus its accuracy depends on librarians knowing that it exists and also sending fund information annually. only 55 of the 87 funds appeared to be currently active the remaining 32 funds had either indicated a cease in operations on their website or on the sparc list, or no longer maintained a discoverable website. in july 2019, we collected information from these 55 websites regarding the funds and their criteria, using google to identify each individual fund website. we entered information regarding each fund’s guidelines and criteria into a google form (see appendix b).   findings   funded article dataset   the average number of funded articles per oa fund per year ranged from 3 to more than 46, with an average of 21 and median of 16 articles.   nearly ¾ of funded applicants were classified as faculty. seven of the responding institutions tracked faculty status, and in those institutions, 56% of funded articles were published by faculty classified as “tenure,” including tenure-track faculty. authors were predominantly affiliated with either medicine/health, or science institutions or departments, with 69% of articles in the dataset published in these combined categories. similarly, ⅔ of the journals in which funded articles appeared were classified as science or medicine. articles were published in plos one more than any other journal, representing 19% of total funded publications.   the dataset included payment data for 885 articles, demonstrating that these 16 libraries had paid more than 1.2 million usd for apcs between 2009 and 2018. note that some of these funds had been in existence for close to a decade, and some for just a couple of years. a few funding programs had ended by the time we requested data on the supported publications.   for additional demographic information and descriptive findings from the initial dataset, please refer to slides from a 2016 presentation (click & borchardt, 2017).   impact   to better understand the impact of library funded oa publications, we analyzed several metrics at the article, journal, and author level for articles published in 2014 and 2016. additionally, in order to better contextualize some of these citation counts, we compared citation ratios from plos articles in our dataset with all plos articles published mid-year in the same years.   article-level metrics   article citation counts varied widely, with a range from 0 to 194 for the combined 2014 and 2016 article dataset. the average citation count was 8.9, while the median was five. the altmetric attention scores for our article subset ranged from 0 to 685. the average score was 15.8, and the median was 2. the altmetric attention score is “a weighted count of all of the mentions altmetric has tracked for an individual research output, and is designed as an indicator of the amount and reach of the attention an item has received” (williams, 2016). it includes mentions in policy documents, blogs, tweets, course syllabi, reddit and more (digital science, 2015). figure 1 directly compares the citation count and altmetric attention score for all articles.   figure 1 comparison of citation counts and altmetric attention scores for all articles in the 2014/2016 publication dataset.   breaking down articles by journal subject category, we found a range of average citation counts and altmetric attention scores for each discipline. the highest average citation count was for articles published in engineering journals, at 11.66 average citations, while articles in science journals had the highest average altmetric attention score with 20.01, as shown in table 2.   table 2 disciplinary breakdown of average citation count and altmetric attention scores in the 2014/2016 publication dataset   agriculture engineering humanities medicine/ health sciences social sciences average citation count 9.22 11.66 1.67 8.88 8.77 3.58 average altmetric attention score 10.61 8.72 0.33 14.95 20.01 11.25   journal-level metrics   the majority of the articles (65%) in the 2014 and 2016 dataset were published in journals that had journal impact factors (jif), ranging from .451 to 40.137, with an average jif of 3.7 and median of 3.234. for context, the mean 2016 jifs for social science journals was 1.199, engineering and technology 1.989, and clinical medicine 2.976, although a direct comparison with our data is not appropriate as the subject categories are not necessarily defined in the same way (larivière & sugimoto, 2019). by contrast, 90% of the articles in the subset were indexed by scimago and had scimago journal rank (sjr) scores. the sjr scores ranged from 0.106 to 18.389, with an average of 1.75 and median of 1.455. see table 3 for average jif and sjr by discipline. the range of jifs and sjrs for all articles are displayed in figure 2.   table 3 disciplinary breakdown of average journal impact factor (jif) and scimago journal rankings (sjr) for journals in the 2014/2016 publication dataset academic discipline average jif average sjr agriculture 3.129 1.509 engineering 3.101 1.323 humanities 2.441 1.013 medicine/health 3.761 1.675 science 4.002 2.061 social science 2.933 1.036     figure 2 comparison of journal impact factor (jif) and scimago journal rankings (sjr) for journals in the 2014/2016 publication dataset.     author-level metrics   h-indices were found for all but three publications in the 2014 and 2016 dataset. the h-index is an “author-level metric calculated from the count of citations to an author’s set of publications” (“h-index,” 2017). if an author’s h-index is seven, this means that the author has published at least seven articles and each of them have been cited at least seven times. in this study, we looked up the h-index for the first and last author of each paper in the subset of articles, and used the higher numbers. we looked at both because in some disciplines the lead author is first and in others last. h-indices ranged from 0 to 108, with an average of 25.3 and median of 22.     oa funds and criteria   of the 87 funds identified, only 55 (63%) were active as of july 2019. we collected and summarized fund guidelines and evaluative criteria related to author eligibility, publication eligibility, and funding details.   author eligibility   nearly all of the funds analyzed listed faculty as eligible fund recipients, with the majority (50 out of 55) listing all faculty, with another four specifying tenure-track or non-tenured faculty. graduate students were the next most common group, listed by 48 of the 55 funds (including 1 fund specifically for graduate students), followed by staff and post-docs. undergraduate students and researchers were also listed at lower rates, with a few other groups, such as emeriti and fellows, selectively mentioned. several libraries give priority to graduate students, early career faculty, and applicants who have not previously received oa funding. some require that the corresponding or lead author apply for funding.   in total, 36% of funds had some form of policy dealing with multiple authors. often, these policies indicated that the level of funding would be prorated by the number of authors, and funding would only be given proportionately to the percentage of authors associated with the institution.   most of the funds also specified that the funds only be used when the author had exhausted other sources of funding, though this criteria was variously worded. while most stipulated that library funds be considered “last resort,” some specifically excluded researchers with grant funds, such as those with an nih grant.   38% of the funds either requested or mandated that a version of the article be placed in the institution’s repository. the wording often indicated that this step was automated, usually by the library, as part of the funding process.   publication eligibility   every one of the funds covered journal articles, though their journal inclusion criteria differed as discussed below. it was found that 15explicitly cover monographs, 12 cover book chapters, 4 cover conference proceedings, and 3 cover datasets. however, in the vast majority of cases these other publication types are not specifically excluded but neither are they mentioned leaving their final eligibility unknown (or perhaps simply untested).   every fund listed criteria the publication must meet in order to be eligible for funding, though in many cases, several criteria were used in conjunction to determine eligibility. the most common criterion mentioned was inclusion in the directory of open access journals (doaj), followed by open access scholarly publishers association (oaspa) membership or compliance with oaspa membership criteria. see figure 3 for the most common publication criteria. although we did not track this specifically, we noticed that many funds require authors to include an acknowledgement statement with their articles, such as “publication of this article was funded by the abc university libraries open access publishing fund.”   figure 3 most commonly-mentioned journal, article, and author criteria present on oa fund websites.   hybrid publications, or journals which require a subscription but make individual articles open access for an additional fee, were excluded by 50 of the 55 funds. of the remaining five, two explicitly allowed for hybrid publication funding, one evaluated hybrid journals on a case-by-case basis, and two were unknown based on the listed criteria. one fund that allows hybrid publications offers a higher pay rate for fully oa versus hybrid. in a previous survey with a smaller sample, 6 out of 10 libraries declined to provide oa funds for hybrid publications (monson et al., 2014).    funding   for 43 out of 55 funds, a definitive source or sources of funding were identified. of those, 93% indicated that funding came from the library, while 14% listed the provost’s office. also listed were offices of research, vice provost or vice chancellor’s offices, individual schools or colleges, office of academic affairs, faculty senate, and an emeriti association. a small survey of 10 universities published in 2014 also found the provost's office and the office of research to be common funding partners for oa funds (monson et al., 2014).   most of the library funds (87%) have a maximum reimbursement per article, ranging from 750 cdn (570 usd as of 5 august 2019) to 4,000 usd. the most common reimbursement maximums are 1,500 usd and 3,000 usd (see figure 4 for more detail). the few funds that specifically address monographs commonly have a 5000 usd limit, although one offered 7,500 usd. in addition, ⅔ of the funds have a maximum reimbursement per author per year, most commonly 3,000 usd. interestingly, two funds require that authors first request a waiver or reduction of publishing charges prior to applying for library oa funds.     figure 4 distribution of maximum reimbursement per article amounts present in oa fund criteria.   discussion   impact   we observed that science and medicine largely dominated both the overall funded publication output as well as impact metrics, which is generally consistent with disciplinary trends in higher education (clarivate analytics, n.d.; digital science, n.d.-a).   looking at the impact metrics, both the range of citation counts and h-indexes were broader than we had anticipated. clearly, some high-impact research is being funded with library oa funds, despite two common fund restrictions that could limit impact: the “last resort” requirement makes it less likely that a grant-funded project would be funded (on the assumption that grant-funded projects have a higher likelihood of being high-impact research), and the near-universal limit of hybrid publication funding mostly eliminates the ability to fund articles for publication in many of the highest-impact subscription model journals. these high-impact publications confirm that faculty’s self-reported interest in oa publishing to increase their visibility discussed earlier is legitimate, and can result in not only a high citation count but also in a high altmetric attention score (beaubien et al., 2016; teplitzky & phillips, 2016).   however, the rcr comparisons for the plos articles indicate that, based on the limited comparison, these funded articles have a slightly lower impact based on their citation counts as compared to similarly published research outside the dataset. this could be due to the two limiting criteria for funds described above. regardless, it represents an opportunity for libraries with oa funds to increase outreach efforts to researchers and labs considered to be high-impact at their institution. while we see some mixed results from overall relative impact and attention of this dataset, messaging around visibility remains a viable selling point to faculty considering oa publication, with plenty of examples of high-visibility work being funded.    effectiveness of oa fund criteria   in a 2015 study, only ⅓ of the libraries that provide oa funding indicated that they had evaluative criteria in place for funding requests. some respondents noted that funded articles must be published in fully oa journals and hybrid journals do not qualify, with 35% requiring listing in the doaj. this study found that 27% of the libraries simply provided funding on faculty request (lara, 2015). our study observed a much higher rate of evaluative criteria, with virtually every oa fund listing guidelines and requirements on their websites, indicating a large trend toward the development of criteria in the past several years.   we were interested to explore the effectiveness of these criteria, and did so by checking the journals in our sample for predatory publishers. predatory publishers – sometimes called deceptive publishers – charge publication fees but make false claims about their publication practices. these publishers, which tend to be oa, may accept and publish articles with little to no peer review or editing, falsely list scholars as editorial board members, and/or fail to be transparent regarding apcs. identifying predatory publishers can be a challenge. jeffrey beall ran a popular website tracking predatory publishers, which was deactivated in 2017 (basken, 2017). currently, cabell’s provides a blacklist of deceptive and predatory journals, using a list of criteria that are categorized as severe (e.g., the journal gives a fake issn, the journal includes scholars on an editorial board without their knowledge or permission), moderate (e.g., the journal’s website does not have a clearly stated peer review policy), and minor (e.g., the publisher or its journals are not listed in standard periodical directories or are not widely catalogued in library databases) (toutloff, 2019). we used a different tool, however, to evaluate journals in our sample.   we identified 20 journals in our 2014/2016 sample that were not indexed by scimagojr. we used a list of questions from think. check. submit to evaluate those 20 journals (e.g., is the journal clear about the type of peer review it uses?) and found 4 did not “pass” this checklist (think. check. submit., n.d.). however, we could not determine whether these four journals were predatory, or simply struggling publications with unclear or incomplete information on their websites. for example, one of the four journals is a sage publication, but does not provide apc information or discuss adherence to or compliance with any open access initiatives such as cope, oaspa, or doaj. the lack of clarity for these four journals mirrors jain and singh’s (2019) findings that predatory publishers are ‘evolving’ with criteria checklists, making these kinds of evaluations more difficult, though they base their findings on beall’s criteria rather than think. check. submit.   a 2017 commentary in nature human behavior discussing stakeholders affected by predatory journals suggests explicit exclusion of predatory journals in oa fund criteria as one mechanism for deterring researchers from predatory publication (lalu, shamseer, cobey, & moher, 2017). two older papers that surveyed librarians also mentioned using beall’s list in oa fund criteria to identify predatory or low quality journals (lara, 2015; monson et al., 2014). however, 2 of the 55 oa funds we examined still mentioned beall’s list a sign that libraries have not entirely kept current with oa journal evaluation practices (or, at the very least, that their websites are no longer accurate reflections of current practice). librarians and other oa funders must continue to monitor evolving practices for evaluation of predatory publications, such as cabell’s and think. check. submit, in order to maintain the effectiveness of oa fund criteria.   viability of oa funds   37%t of the oa funds that we identified via our data collection, sparc’s oa funds in action list, and google searching are no longer active as of summer 2019. given the relatively short time that oa funds have been in existence, this rate of default points to a potentially troubling viability for oa funds. whether oa funds will continue to be funded may largely depend on other concurrent oa and library initiatives, such as big deal cancellations and plan s compliance, which could help determine the future oa landscape and more sustainable funding models.   funding sources could also play a critical role in the future viability of these funds. in a 2015 survey of libraries that provide oa funding, 70% stated that oa funds came from the existing materials budget, and 24% indicated that they came from a new budget allotment unrelated to materials (lara, 2015). we posit that, in the age of uncertain library budgets for many libraries, identification of non-library campus partners may be critical for the long-term continuation of these funds. examples of distributed funding includeiupui’s fund, which lists no less than 13 campus partners contributing to the fund;and wake forest, which cost-shares publication fees equally between the library, office of research and sponsored programs, and the author’s department (iupui university library, n.d.; wake forest university library, n.d.).   we observed several cost-saving measures employed by oa funds, including maximum article and author fees, as well as article funding at less than 100%, all of which may also help contribute to the sustainability of these funds. in the 2015 survey, “about 80% of respondents were unsure or stated that there is no established maximum, 19% stated that there is a maximum fee in place. nearly all of the respondents whose institutions have an established ceiling for funding placed the maximum price in the range of $2,000–3,000” (lara, 2015, p. 7). this shift from 19% in 2015 to the 87% of funds in 2019 with price capping suggests that future viability may be dependent on limiting these funds, at least for now. one of the more innovative approaches to price capping we observed was university of massachusetts amherst’s oa fund, which started at 50% fee coverage, with increased coverage earned through additional criteria, such as early-career authors, first-time applicants, a non-profit or society publisher, and having an orcid (umass amherst libraries, n.d.).   future research & directions   we see an opportunity to further investigate oa funds in order to establish more concrete best practices. we have seen shifts in criteria models used by funds but have these shifts contributed to the success or failure of individual funds? are funds with more distributed funding models more sustainable? our findings hint at these possibilities, but more research would help clarify these potential best practices. we also see value in continuing to monitor institutional funding for oa as the scholarly communications landscape continues to change. many possibilities for oa rely on financial support from libraries, and a coordinated approach toward funding models may be the key to the success or failure of broad oa adoption.   alternative oa support models are already emerging. for example, reinsfelder and pike (2018) urge a shift away from libraries spending funds on apcs and towards crowdfunded models like knowledge unlatched, scoap3, and unglue.it. they argue that $25,000 would pay approximately 12.5 journal apcs, but would fund 471 new oa books through a knowledge unlatched pledge. likewise, berger (2017) argues that advocacy by libraries for different funding models de-commodifies scholarship, and will also “mortally wound” predatory publishers’ viability. some universities in the u.s. are starting to make this shift. in 2019, the university of arizona libraries transitioned away from their open access publishing fund, establishing an open access investment fund. instead of paying individual apcs for oa publications, the libraries will now pay for institutional memberships with specific publishers that include apc discounts, as well as initiatives with “wide potential global impact” like arxiv and the open textbook network (university of arizona university libraries, 2019).   conclusion   libraries in north america are clearly dedicated to supporting the oa movement, and in recent years this has meant providing authors with funds to pay apcs. this study explores the articles published via library oa funds at 16 universities and their impact, as well as the guidelines and criteria set forth in 55 funds. findings indicate that research impact is a useful tool for increasing faculty support of oa and that existing fund criteria have been refined over recent years to encourage publication in mostly high-quality journals. oa funds have supported researchers in a wide range of disciplines and career stages, with stem fields and researchers being the most frequently-supported by these funds. however, there is some evidence to suggest that these funds may not be supporting the highest impact research, possibly as a result of fund criteria restrictions. the overall oa landscape is shifting, and the apc model may not prove to be viable. price capping of funds and distributed funding models may increase the sustainability of these funds in the future. regardless of the administrative details behind funding, the ways that institutions choose to financially support oa will continue to evolve as the oa movement develops.   references   basken, p. (2017, september 12). why beall’s list died—and what it left unresolved about open access. the chronicle of higher education. retrieved from https://www.chronicle.com/article/why-beall-s-list-died-/241171   beaubien, s., garrison, j., & way, d. (2016). evaluating an open access publishing fund at a comprehensive university. journal of librarianship and scholarly communication, 3(3), ep1204. https://doi.org/10.7710/2162-3309.1204   berger, m. (2017). everything you ever wanted to know about predatory publishing but were afraid to ask. association of college & research libraries conference proceedings, pp. 206–217. retrieved from http://www.ala.org/acrl/sites/ala.org.acrl/files/content/conferences/confsandpreconfs/2017/everythingyoueverwantedtoknowaboutpredatorypublishing.pdf   clarivate analytics. (n.d.). field baselines, incites essential science indicators. retrieved 13 august 2019 from https://esi.clarivate.com/baselineaction.action   click, a., & borchardt, r. (2017). follow the money: an exploratory study of open access publishing funds’ impact. retrieved from https://www.slideshare.net/amandaclick/follow-the-money-an-exploratory-study-of-open-access-publishing-funds-impact   coalition s. (n.d.). ’plan s’—making full and immediate open access a reality. retrieved 9 august 2019 from https://www.coalition-s.org/   digital science. (2015, july 9). the donut and altmetric attention score. retrieved 12 august 2019 from https://www.altmetric.com/about-our-data/the-donut-and-score/   digital science. (n.d.-a). the 2018 altmetric top 100. retrieved 13 august 13 2019 from http://www.altmetric.com/top100/2018/   digital science. (n.d.-b). dimensions. retrieved 5 august 2019 from https://app.dimensions.ai/discover/publication   eckman, c. d., & weil, b. t. (2010). institutional open access funds: now is the time. plos biology, 8(5), e1000375. https://doi.org/10.1371/journal.pbio.1000375   harnad, s. (2010). no-fault peer review charges: the price of selectivity need not be access denied or delayed. d-lib magazine, 16(7/8). https://doi.org/10.1045/july2010-harnad   h-index. (2017, july 23). retrieved 12 august 2019 from https://www.metrics-toolkit.org/h-index/   iupui university library. (n.d.). iupui open access fund. retrieved 13 august 2019 from http://www.ulib.iupui.edu/digitalscholarship/openaccess/oafund   jain, n., & singh, m. (2019). the evolving ecosystem of predatory journals: a case study in indian perspective. arxiv:1906.06856 [cs.dl]. retrieved from http://arxiv.org/abs/1906.06856   kell, g. (2019, march 6). why uc split with publishing giant elsevier. retrieved from https://www.universityofcalifornia.edu/news/why-uc-split-publishing-giant-elsevier   .lalu, m. m., shamseer, l., cobey, k. d., & moher, d. (2017). how stakeholders can respond to the rise of predatory journals. nature human behaviour, 1(12), 852–855. https://doi.org/10.1038/s41562-017-0257-4   lara, k. (2015). the library’s role in the management and funding of open access publishing. learned publishing, 28(1), 4–8. https://doi.org/10.1087/20150102   larivière, v., & sugimoto, c. (2019). the journal impact factor: a brief history, critique, and discussion of adverse effects. in w. glänzel, h. f. moed, u. schmoch, & m. thelwall (eds.), springer handbook of science and technology indicators. retrieved from https://arxiv.org/abs/1801.08992   monson, j., highby, w., & rathe, b. (2014). library involvement in faculty publication funds. college & undergraduate libraries, 21(3–4), 308–329. https://doi.org/10.1080/10691316.2014.933088   pinfield, s. (2010). paying for open access? institutional funding streams and oa publication charges. learned publishing, 23(1), 39–52. https://doi.org/10.1087/20100108   price, e., engelson, l., vance, c. k., richardson, r., & henry, j. (2017). open access and closed minds? collaborating across campus to help faculty understand changing scholarly communication models. in k. smith & k. dickson (ed.), open access and the future of scholarly communication (pp. 67–84). lanham, md: rowman & littlefield.   reinsfelder, t. l., & pike, c. a. (2018). using library funds to support open access publishing through crowdfunding: going beyond article processing charges. collection management, 43(2), 138–149. https://doi.org/10.1080/01462679.2017.1415826   relative citation ratio. (2017, july 24). retrieved 5 august 2019 from https://www.metrics-toolkit.org/relative-citation-ratio/   scholarly publishing and academic resources coalition [sparc]. (2018, november). campus open access funds. retrieved 1 july 2019 from https://sparcopen.org/our-work/oa-funds   sinn, r. n., woodson, s. m., & cyzyk, m. (2017). the johns hopkins libraries open access promotion fund: an open and shut case study. college & research libraries news, 78(1), 32–35. retrieved from https://crln.acrl.org/index.php/crlnews/article/view/9605/10998   suber, p. (2016). knowledge unbound: selected writings on open access, 2002–2011. cambridge, massachusetts: mit press. retrieved from https://www.dropbox.com/s/ege0me3s7y24jno/8479.pdf?dl=0   tenopir, c., dalton, e., christian, l., jones, m., mccabe, m., smith, m., & fish, a. (2017). imagining a gold open access future: attitudes, behaviors, and funding scenarios among authors of academic scholarship. college & research libraries, 78(6), 824–843. https://doi.org/10.5860/crl.78.6.824   teplitzky, s., & phillips, m. (2016). evaluating the impact of open access at berkeley: results from the 2015 survey of berkeley research impact initiative (brii) funding recipients. college & research libraries, 77(5), 568–581. https://doi.org/10.5860/crl.77.5.568   think. check. submit. (n.d.). check. retrieved 12 august 2019 from https://thinkchecksubmit.org/check/   toutloff, l. (2019, march 20). cabells blacklist criteria v 1.1. retrieved 14 october 2019 from https://blog.cabells.com/2019/03/20/blacklist-criteria-v1-1/   umass amherst libraries. (n.d.). soar fund guidelines. retrieved 12 august 2019 from https://www.library.umass.edu/soar-fund/soar-fund-guidelines/   university of arizona university libraries. (2019). open access investment fund. retrieved 9 august 2019 from https://new.library.arizona.edu/about/awards/oa-fund   wake forest university library. (n.d.). open access publishing fund. retrieved 13 august 2019 from https://zsr.wfu.edu/digital-scholarship/open-access-publishing-fund/   williams, c. (2016, june 30). the altmetric score is now the altmetric attention score. retrieved 12 august 2019 from https://www.altmetric.com/blog/the-altmetric-score-is-now-the-altmetric-attention-score/   zuniga, h., & hoffecker, l. (2016). managing an open access fund: tips from the trenches and questions for the future. journal of copyright in education and librarianship, 1(1). 1-13. https://doi.org/10.17161/jcel.v1i1.5920     appendix a library fund data collection fields institutional details publication details institution name journal title private or public indexed in doaj (y/n) carnegie classification (e.g., r2) hybrid (y/n) author details journal impact factor discipline journal publisher author name article details co-authors (y/n) article title international collaborators (y/n) reimbursement amount status (e.g., faculty, grad student) reimbursement year tenure (y/n) publication year email doi h-index       appendix b oa fund criteria data collection form   name of university: ______________________________________   who is eligible for these funds? (check all that apply) faculty (all) faculty (tenure track specified) staff undergraduate students graduate students postdocs researchers other: ________________________________   what types of publications are eligible? (check all that apply) journal articles book chapters monographs other: ________________________________   which criteria must the publication meet? (check all that apply) peer reviewed listed in doaj listed in doab oaspa member or compliant immediate open access published fee schedule policy for economic hardship not on beall’s list no predatory publishers agree to put in repository oa fund is last resort apc only (e.g., no submission fees) other: ________________________________   hybrid allowed? yes no case-by-case other: ________________________________   is there a maximum reimbursement per article? yes no   what is the maximum reimbursement per article? _____________________   is there a maximum reimbursement per author per year? yes no   what is the maximum reimbursement per author per year? ______________________   limited to 1 publication per author per year? yes no   multiple author policy? yes no   source of funds? (check all that apply) provost’s office library other: ________________________________   notes: ______________________________________       evidence summary   literature suggests information professionals have adopted new roles   a review of: vassilakaki, e. & moniarou-papaconstantinou, v. (2015). a systematic literature review informing library and information professionals’ emerging roles. new library world, 116(1/2), 37-66. http://dx.doi.org/10.1108/nlw-05-2014-0060   reviewed by: robin e. miller associate professor and research & instruction librarian mcintyre library university of wisconsin-eau claire eau claire, wisconsin, united states of america  email: millerob@uwec.edu   received: 14 dec. 2016  accepted: 17 jan. 2017      2017 miller. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to provide a systematic review of the emerging or newly adopted roles of information professionals, over the past 14 years, as described in the library and information science (lis) professional literature.   design – systematic review of the literature.   setting – databases featuring information science content, including acm digital library, library, information science and technology abstracts (lista), library and information science abstracts (lisa), citeseer, google scholar, e-prints in library and information science (e-lis), digital library of information science and technology (dlist), scopus, and science direct. the database library literature & information science index was not included.   subjects – through a systematic literature search, the authors identified 114 peer-reviewed studies published between 2000-2014.   methods – the authors searched selected databases using the terms “librarian/s role” and “information professional/s role” to collect literature about the roles of information professionals. the authors searched the selected databases in two phases. the initial search yielded 600 search results and the authors included 100 articles about “roles” information professionals have adopted. the authors excluded articles focused on specific positions, health and medical libraries, librarians’ professional skills, and development of specific programs or initiatives within libraries. in the second phase of searching, the authors refined search terms to include phrases specifically related to the roles identified in the 100 articles initially included in the review. there were 48 articles identified in the second search and 14 were included in the final pool of articles. the authors also cross-checked the references of all included literature.   main results – the authors identified six roles of information professionals described in the literature during the review period. the role of “embedded librarian” was described in the largest number of articles (42%), followed by “librarian as teacher” (20%), “knowledge manager” (20%), “technology specialist” (9%), “subject librarian” (6%), and “information consultant” (3%).   the study did not identify a dominant journal title or professional conference publishing research on information professionals’ roles. some included literature reported a specific method for investigation, including questionnaires, content analyses, and mixed methods. however, the researchers report that the majority of articles represented personal views or perceptions of the authors.   conclusion – the roles of information professionals are continually changing, both in practice and in description. in particular, information professionals expanded their roles in teaching during the review period, shedding light on institutional and professional priorities.   commentary   the authors identified information professional roles that may have emerged or evolved during the review period. in synthesizing the reviewed literature’s discussion of each role, the authors offer a detailed view of the scholarly conversation about the evolution of the information profession. this research may indicate that information professionals have assumed new roles as embedded librarians, particularly in light of changing pedagogical resources and tools that foster new methods of patron interaction. however, the authors’ discussion of the other roles – librarian as teacher, knowledge manager, technology specialist, subject librarian, and information consultant – seem more indicative of changing rhetoric than the emergence of new conceptual roles for information professionals. a longer review period might alter this perspective.   the systematic review method is a promising means of quantifying professional discussions on changing roles in the information field. the evidence presented in this article may largely apply to academic libraries because, as the authors note, the majority of articles included in the review were about academic libraries. the authors searched a wide range of databases, following the guidance of hemingway and brereton (2009). the focus on peer-reviewed literature discovered through database searches may be the reason that the majority of included articles address academic libraries. this limitation was acknowledged by the authors. while trade and grey literature published by professional associations and consortia may be more difficult to locate in databases, such literature might expand the subject matter to information professionals working outside of higher education.   the inclusion and exclusion criteria employed by the authors reveal ambiguities in lis professional rhetoric that this research may be unable to overcome. the search terms “librarian/s roles” and “information professional/s roles” were employed to search several databases. however, the authors do not define “role,” nor do they identify other terms they considered and discarded. research about the “skills” of librarians was also excluded, though the 114 articles ultimately included do discuss librarian “competencies,” “expertise,” “knowledge,” and “training.” “papers referring to specific positions or specific occupational groups” (p. 39) were excluded, though the article does not explain the distinction between a librarian’s “role” and “professional responsibilities.” for example, the authors indicate that the role of “embedded librarian” dominated the scholarly conversation in the included literature, but the exclusion criteria indicate that “embedded librarianship” is a “type” of librarianship that was excluded from systematic review (2015). new positions and titles in information organizations may well represent emerging roles in areas like scholarly communication, assessment, outreach, emerging technology, and instructional design. are these roles, professional responsibilities, skills, competencies, or something else?   how information professionals describe their work is a useful inquiry. as a survey of articles about the work information professionals do, this research may inform hiring managers or other librarians who seek to redefine existing roles or create new roles to fill needs within their libraries. this research could be expanded to include trade and grey literature. practical applications of this research would be enhanced with refined search terms and terminology that distinguishes between librarian position titles, responsibilities, and competencies.   references   hemingway, p. & brereton, n. (2009). what is a systematic review? what is...? series. retrieved from http://www.bandolier.org.uk/painres/download/whatis/syst-review.pdf     editorial   scholarly publishing during a pandemic   lorie kloda editor-in-chief associate university librarian, planning and community relations concordia university montreal, quebec, canada email: lorie.kloda@concordia.ca      2020 kloda. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29792     the past few months have been a first for everyone. no one has experienced a pandemic of this scale, and we have all had to make significant adjustments to our work lives and personal lives that we never imagined having to make. for those of us fortunate enough to be in good health and to remain employed, we have had to alter our schedules, routines, and habits. despite this enormous disruption in our day-to-day lives, the members of the evidence based library and information practice editorial team managed to publish our previous issue in the middle of march, only a few days behind schedule.   over the last three months, the editorial team, all volunteers with day jobs and lives disrupted by the pandemic, have continued to work to produce the latest issue of the journal before you. as the editor-in-chief, i am of course very proud of our team. i am also impressed by the contributions of the evidence summary writers, peer reviewers, and copyeditors who have continued to move the editorial process forward and to meet tight deadlines. these individuals increase the caliber of the journal submissions and continue to do so while handling current world events. the pandemic would have been an acceptable reason to step away from these additional volunteer responsibilities and yet no one did.   i want to thank all the people who are involved with the journal for their dedication, at all times but especially now.   news/announcements   evidence based library and information practice (eblip) is seeking an editorial intern      2016. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     we are looking for a new editorial intern to assist our editorial team. the ideal candidate will be in place by november 2016 and will commit to serving a two year term.    the role of the intern is to:    provide a final editorial check of proofed copy before publication, using the pre-developed checklist  check and edit the item metadata on the eblip website to ensure title, author, and abstract correspond to the submitted manuscript  assist the editor-in-chief and communications officer with calls for papers or calls for volunteers  assist with investigation and implementation of new projects related to the journal  participate in monthly editorial team meetings    the ideal candidate will be an mlis (or equivalent) student interested in evidence based practice and research methods, and who possesses strong attention to detail. the position requires dedicated time on a regular basis, and it is therefore essential that interested persons ensure available time to devote to this position prior to applying. it is estimated that the workload is approximately 10 hours per month.    interested persons should send a statement of interest, indicating the areas of strength they would bring to the role, as well as a brief resume to lorie kloda at lorie.kloda@concordia.ca by october 14, 2016.   specific queries about the role should be addressed to michelle dunaway at mdunawa@gmail.com.    evidence based library and information practice using evidence in practice   a mixed methods approach to iterative service design of an in-person reference service point   kyla everall user services librarian university of toronto libraries toronto, ontario, canada email: kyla.everall@utoronto.ca   judith logan user services librarian university of toronto libraries toronto, ontario, canada email: judith.logan@utoronto.ca   received: 10 july 2017    accepted: 6 oct. 2017      2017 everall and logan. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     setting   the university of toronto is canada’s largest university, with an enrolment of over 88,000 students (university of toronto, n.d.). currently ranked fourth by the association of research libraries, with an annual total expenditure of $72 million usd (arl, 2016), we have 44 libraries spread over three campuses.   problem   our largest library is a 14-floor building located in the center of our downtown campus.   when john p. robarts library was first opened in 1973, the reference desk on the fourth floor was conveniently located next to the card catalogues and near the main entry point to the book stacks. the card catalogues are long gone and the ground floor is now open for stacks access, but the reference desk has remained nearly unchanged physically. likewise, our service model remained stable for decades. two or three library staff members sat behind a large wooden desk from mid-morning to early evening. shifts were primarily staffed by librarians with support from student librarians trained in the department. our schedule called on librarians regardless of specialized expertise; when our department incorporated government information librarians in 2009, they, too, served on the reference desk.     figure 1 “john p. robarts library” by jeff hitchcock on flickr. retrieved from https://www.flickr.com/photos/arbron/29821692490       over the last ten years, student feedback collected through libqual+ and other means has identified that robarts is a confusing and intimidating library to navigate. service provided by staff throughout the building did not always meet students’ expectations. we partially addressed this longstanding problem in 2015 when we added a roving service of student library assistants whose primary roles were to direct users to the correct location within the building and to troubleshoot problems with self-service machines such as printers. the usage statistics for this service showed the most traffic on the second floor of the building. this made sense considering that there were no library service points or staff members on this floor besides one entrance monitor who monitors the security gates and enforces our food policy.   we hoped that redesigning our outdated reference desk could also address gaps in robarts’ perceived user-friendliness. evidence   we approached this problem within a service design framework. a relatively new method in libraries, service design is a “holistic, co-creative, and user-centered approach to understanding customer behavior for the creation or refining of services” (marquez & downey, 2015, para. 7). the aim is to use lightweight research methods to gain insights and make decisions, rather than conduct exhaustive studies of the problem (polaine, løvlie, & reason, 2013). service design also encourages prototyping, allowing service providers to test their ideas without a large investment of resources (polaine et al., 2013).   looking at our reference desk statistics emphasized just how much usage patterns have changed over time. our interactions declined by more than 50% since the 2009-2010 academic year (figure 2). demand for the service was clearly changing. we asked ourselves, if in-depth reference expertise was only needed in about a third of the interactions currently on the desk, was there a better way to deploy our staff without sacrificing quality of service?       figure 2 reference desk interactions from 2009-2016.     figure 3 average estimated cost of reference desk interactions per month.       to investigate the effectiveness of our service model’s staffing mix, we did a rough cost analysis to estimate the cost per interaction (figure 3). this analysis indicated that there were times of the year when our service model was more cost effective than others. we planned to use this information when designing the schedule of our redesigned service point.     figure 4 an example of a completed journey map for a student who wanted to use scanners.       further probing this issue, we conducted focus groups with our student librarians to assess their training and identify gaps in their knowledge. they reported high levels of confidence in answering general research questions. referrals and questions that require organizational knowledge were their biggest challenges. this indicated to us that, once trained, our student librarians might be able to provide the bulk of staffing at our new service point if properly supported.   we also conducted a journey mapping research project to help us understand how our users were moving through the building. we surveyed users as they left the library about their goals during their visit. we then asked for a step by step breakdown of how they tried to achieve their goals and how they felt during each step (figure 4).   we found only 35% of users came to the library with scholarly goals. this category included using print resources and working with librarians. respondents reported two pain points that had previously surfaced in libqual+: navigating the building and unhelpful interactions with staff.   implementation   we moved our reference desk to the second floor of the building and rebranded it the “ask us! desk” with a large sign placed next to it. all staff wore blue vests emblazoned with an information symbol while working on the service desk for the first eight months.   we purchased a variable height standing desk on castors with the intention of moving it around the floor to identify the ideal location, but we discovered that the wireless service was too unreliable. instead, staff connected the desk’s laptops to a nearby lan network. happily, this was near the entrance to the cafeteria and a very busy coffee shop. we forwarded our reference desk’s extension to a cell phone so we could continue to offer telephone service, and issued earbuds to staff so they could be hands free while serving users on the phone.   a team collaboration and chat tool called slack (https://slack.com/) was already in use in other units of our library, so we used it to facilitate communication between staff members at the ask us! desk and the rest of the department. we created a slack “team,” which is similar to a message board, and invited as many public service staff members in the building as we could in hopes of increasing communication between service points.   once we were physically set up on the second floor, we began to adjust our staffing mix. we stopped scheduling our government information specialist librarians on the desk as their expertise is in high demand on other reference channels. we also stopped scheduling two librarians at the same time; the maximum complement became one student librarian and one librarian. once our student librarians were trained, on boarded, and sufficiently mentored, we moved to a backup model where the librarian assigned to the shift remained in their office and stayed in communication with the student librarian on shift over slack. if the shift became busy or the librarian's expertise was needed, the student librarian would ask the librarian to come down to help over slack.   outcome   to see if the new model was working for our users, we conducted exit surveys in the fall and winter semesters. respondents reported high levels of satisfaction with the service that they received at the desk (figure 5), and 87% of respondents liked the location (figure 6). only 4% of respondents had a negative impression of the location, and the remaining 9% were either neutral or responded in a way that was not readily classifiable (e.g., “i wish we had a desk like this on every floor”). we also discovered that the blue vests were not significantly more important to our visibility than signage or staff demeanor (figure 7).     figure 5 were you satisfied with the help that you received today? (n=77)     figure 6 what do you think of the current ask us! desk location? (n=77)   figure 7 what was the main thing that made you think that you could get help at this desk? (n=77)     figure 8 reference desk/ask us! desk interactions, 2009-2017.       we monitored our usage statistics to see what impact the new location would have. while we saw an overall increase in usage, the bulk of the questions were directional (figure 8). brief and in-depth reference questions declined more than in past years, suggesting that we need to reassess the new model to ensure that it is attracting in-depth reference appropriately.   we also repeated the journey mapping exercise to gauge if the new desk had any impact on users’ holistic experience of the library. there was almost no change in the majority of pain points reported, but this time there were no complaints about library staff. there are too many variables to attribute this positive outcome to solely our service, but it is an encouraging sign.   throughout the implementation process, we consulted the staff working on the desk about the new model. some staff voiced concerns about the blue vests. they felt strongly that wearing the vest made them look unprofessional, and reported that staff from other units had made comments to this effect. since we learned from the exit surveys that the vests were not significantly more important to identifiability than other factors, we replaced the vests with lanyards.   librarians reported noticeable improvements in their mentoring relationships with student librarians. due to the layout of the new desk, we worked shoulder to shoulder with them, creating more feelings of teamwork and more collegiality than in previous years. we also noticed that the student librarians’ skills developed significantly faster under the new model.   reflection   by considering a variety of data sources when redesigning our service point, we developed a model that better fit our users. our intention was always to do just enough research to allow us to produce a prototype which we could refine after the initial implementation. when we encountered unforeseen obstacles, such as poor wi-fi, we reconfigured the service. rather than derailing the project, we were prepared to address issues as they arose, ultimately resulting in a more responsive, flexible service.   prior to this project, the prevailing organizational narrative was that service changes involving staff would be met with overwhelming resistance. a benefit of launching our new service model as a work in progress was more staff engagement and less resistance than anticipated. this prototyping approach signaled to staff that their feedback was not only tolerated, but necessary to the success of the project.     references   association of research libraries. (2016, august 14). spending by university research libraries, 2014-15. retrieved from http://www.chronicle.com/interactives/almanac-2016?#id=65_416   marquez, j., & downey, a. (2015). service design: an introduction to a holistic assessment methodology of library services. weave: journal of library user experience, 1(2). http://dx.doi.org/10.3998/weave.12535642.0001.201   polaine, a., løvlie, l., & reason, b. (2013). service design: from insight to implementation. brooklyn, n.y.: rosenfeld media.   university of toronto. (n.d.). quick facts. retrieved from https://www.utoronto.ca/about-u-of-t/quick-facts     evidence summary   students may demonstrate information literacy skills following library instruction   a review of: luetkenhaus, h., hvizdak, e., johnson, c., & schiller, n. (2017). measuring library impacts through first year course assessment. communications in information literacy, 11(2), 339-353. http://comminfolit.org/index.php   reviewed by: robin e. miller associate professor and assessment & instruction librarian mcintyre library university of wisconsin-eau claire eau claire, wisconsin, united states of america  email: millerob@uwec.edu   received: 1 mar. 2018    accepted: 3 may 2018      2018 miller. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29422       abstract   objective – to determine whether there is a correlation between information literacy skill development and participation in one or more library instruction sessions.   design – learning outcomes assessment.   setting – a public research institution with multiple campuses.   subjects – 244 first-year undergraduates enrolled in a compulsory general education course during the 2014-2015 academic year. all subjects completed a series of library research assignments, followed by a final research paper. 65% of subjects participated in at least one library instruction session as part of the course, and 35% did not.   methods – the researchers convened six librarians and six instructors/faculty to score 244 research papers using a rubric designed to measure six possible information literacy learning outcomes. evaluators established inter-rater reliability through a norming session, and each artifact was scored twice. the authors analyzed rubric scores using ordinary least squares regression modeling.   main results – participation in a library instruction session correlated with higher rubric scores in three information literacy learning outcomes: argument building; source type integration; and ethical source citation.   conclusion – students may achieve greater information literacy learning outcomes when they participate in course-integrated library instruction.   commentary   this research presents a thoughtful attempt to measure the impact of library instruction integrated into a compulsory course for undergraduates at a large institution. during the 2014-2015 academic year, more than 4,600 students enrolled in this course completed a common set of library research assignments, and all were expected to write a final research paper. the researchers recruited 12 evaluators to score research papers authored by 5% of students enrolled in the course. a weighted sample was necessary to adequately represent students from a smaller campus. the validity of the sample cannot be determined because the authors do not disclose the populations of students representing each of the campuses involved in the study.   each research paper was scored with a six-point rubric featuring five levels of achievement. the rubric aligned with the common research assignment, general education goals of the institution, and value rubrics published by the association of american colleges & universities (aac&u). the authors indicate they convened a norming session to finalize the rubric, implying that the instrument was validated at that time. this study aligns with, and is similar to, current research about the impact of information literacy instruction on the products of student research (jastram, leebaw & tomkins, 2014; lowe, booth, stone & tagge, 2015).   students who had participated in 1-4 library instruction sessions authored a majority of the artifacts scored in this research project. the authors identified six possible information literacy learning outcomes, and report that participation in library instruction positively correlated with 3 of the 6 learning outcomes. for information literacy instruction practitioners, this finding seems encouraging. however, in discussion of the study’s limitations, the authors note that their analysis does not address variables such as the number of library instruction sessions attended by students in the sample, or other forms of assistance included in individual courses. in a contrasting study of librarian engagement in first year courses, booth, lowe, tagge, and stone (2015) found that greater degrees of librarian involvement in a course improved student information literacy learning outcomes. analysis using ordinary least squares regression modeling enabled the investigators to control for variations in library instruction attendance on different campuses. the same model could also have been used to control for the variation in number of sessions attended.   the researchers report that papers authored by students who had participated in library instruction received higher scores in three areas: argument building (p<.05), source type integration (p<.05), and ethical source citation (p<.01). they offer a general description of a library lesson in the discussion, making a logical argument about the lesson’s connection to higher scores in source type integration. the authors state that one of the campuses involved delivered “standardized” library instruction, but it remains unclear if the majority of the sample was exposed to the same interventions. similarly, ethical source citation was not explicitly covered by the library instruction lesson. while scores appear to be higher, this research does not identify an intervention that helped students to perform well in ethical source citation. these ambiguities point to a general problem with assessment of library instruction due to variations in scope, content, and course implementation (ackermann, 2007).   the authors suggest that this article “bolsters” the field of research about information literacy learning outcomes. oakleaf (2007) has observed that a challenge to rubric-based assessment may be that rubrics are difficult to transfer from one university setting to another. while this research may indicate that learning outcomes are improved by course-integrated library instruction, the findings are preliminary because the tool has not been generalized beyond a single academic year or to another institutional setting.   references   ackermann, e. (2007). program assessment in academic libraries: an introduction for assessment practitioners. research and practice in assessment, 2, 18-23. http://www.rpajournal.com/   booth, c., lowe, m. s., tagge, n., & stone, s. m. (2015). degrees of impact: analyzing the effects of progressive librarian course collaborations on student performance. college & research libraries, 76(5), 623-651. https://doi.org/10.5860/crl.76.5.623   jastram, i., leebaw, d., & tomkins, h. (2014). situating information literacy within the curriculum: using a rubric to shape a program. portal: libraries and the academy, 14(2), 165-186. https://doi.org/10.1353/pla.2014.0011   lowe, m. s., booth, c., stone, s., & tagge, n. (2015). impacting information literacy learning in first-year seminars: a rubric-based evaluation. portal: libraries and the academy, 15(3), 489-512. https://doi.org/10.1353/pla.2015.0030   oakleaf, m. (2007). using rubrics to collect evidence for decision-making: what do librarians need to learn? evidence based library and information practice, 2(3), 27-42. https://doi.org/10.18438/b8ws3w     evidence based library and information practice using evidence in practice   why are they not visiting the library? understanding political science postgraduate students   abdul jabbar librarian, department of political science phd scholar, department of information management university of the punjab lahore, pakistan email: jabbar.polsc@pu.edu.pk   received: 16 mar. 2019                                                                  accepted: 7 may 2019      2019 jabbar. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29569   setting   the department of political science is one of the oldest departments of university of the punjab, lahore, pakistan. the university of the punjab is one of the oldest and renowned public sector universities of the subcontinent. currently, the department is offering three graduate and four post-graduate programs in political science, international relations, diplomacy and strategic studies, and a post-graduate diploma program in international affairs.   the departmental library supports the curricular needs and promotes research culture among the students and faculty members of the department. the library also serves as an archive for the research heritage of the department. one librarian with an mphil degree in library and information science serves in the library, alongside two clerks and two library attendants.   problem   the librarian joined the library almost three years ago.   the work assigned to undergraduate students was usually done through the help of available books in the library. however, the graduate students who had chosen thesis in their second year were visiting library and the librarian with research-related queries, that is, database selection, query formulation, advanced searching techniques and help related to references management and formatting style.   however, the librarian was curious about the needs of the post-graduate students who were conducting mphil and phd level research, but were neither visiting the library or consulting the librarian frequently. some post-graduate students visited the library once or twice a month for required books without consulting the librarian for any research-related help or queries. the librarian wanted to understand why postgraduate students were not visiting the library frequently.   evidence   it was therefore considered appropriate for the librarian to conduct in-depth, face-to-face interviews with the students. in mid-december 2015, a total of 25 post-graduate students who had completed their coursework and were carrying out their theses or research work were invited for interview through email and telephone. interviews were conducted with 15 students, including 8 female and 7 male, from 4 post-graduate programs after acquiring their prior consent for participation. the students were asked to discuss the reasons why they were not consulting the librarian for help with research-related issues. the interviews were conducted by the librarian and notes were taken during interview. the data were analyzed by employing thematic analysis. a number of reasons were identified through thematic analysis.   unawareness and misconceptions regarding the librarian’s professional skills   the students (n=9) had never considered that the librarian would be able to resolve research problems, because they were unfamiliar with the job role of the librarian. some (n=4) said that they had never met the librarian, or felt the need to meet them. the students felt that the support staff conducted the core work of the library, while the librarian’s role was to control them.   multiple previous bad experiences   some participants did not expect that any librarian could help them, or that any librarian would offer to help. hence, they did not consider the librarian as a source of help for research assignments or theses.   assignments and theses did not require exploration   the most interesting inferred reason was that most students claimed they were working on a thesis topic that was descriptive, meaning they thought that they did not require any primary sources like research articles or reports. instead, they focused on books related to the topic, as well as some websites and newspapers.   unawareness of relevant databases offered by higher education commission   the majority of the students mentioned that they were only familiar with google, google scholar, and jstor as sources to consult for assignments and research work. they had not heard of other relevant databases available through the hec digital library, such as emerald, science direct (elsevier), and springerlink.   misconceptions regarding use of journal articles   the post-graduate students considered that the nature of the field (political science and international relations) did not require the incorporation of journal articles. instead, they wrongly thought that secondary sources like books were the most appropriate.   lack of scholarship and interpretation in research projects and theses   through interviewing, it had found that postgraduate students had chosen those topics of their theses that based on descriptive research rather analytical research. they were dividing their dissertations into five or six chapters mainly focused on fulfilling the minimum requirement of words count for a dissertation.   implementation   after data collection, the results were discussed with the post-graduate program coordinator. a number of steps were taken to address the issues identified. a series of seminars were arranged with the help of the course coordinator, covering topics such as avoiding plagiarism, reference management styles and software, advanced searching techniques, and how to do literature review. through these seminars, the students were encouraged to visit the librarian with research-related queries.   with the help of the main library at the university of the punjab, a hands-on practice session was arranged on the hec digital library and on zotero reference management software. program teachers were also requested to assign topics which required extensive use of published and unpublished literature.   outcome   the post-graduate students contacted the librarian through email and telephone with research-related queries. these queries were mostly related to checking similarity through ‘turnitin’ as a way to avoid plagiarism, the formatting of theses according to the doctoral program coordination committee (dpcc), and for guidance on reference management. the frequency of visits to the library, as well as to the librarian increased. this data was recorded through logging issues, returns, and the number of post-graduate students’ visits to the librarian. the information literacy services offered by the librarian positively affected the students’ perceptions; they now considered the librarian to be instrumental in their research assignments and theses.   reflection   although data collection was limited by the fact that it was very difficult to convince the students to participate in the study, the results were very encouraging and positive. tangible as well as moral support from faculty members and program coordinator was appreciable. future plans include expanding the study to undergraduate students.   evidence based library and information practice editorial responsibilities      2019. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     editor-in-chief: lorie kloda   associate editor (evidence summaries): heather pretty, christina wissinger   associate editor (research articles): ann medaille, lisl zach   associate editor (classics & reviews): lorie kloda   associate editor (commentaries, feature, using evidence in practice): lorie kloda   communications officer (news): tatiana bryant   production editor: rachel hinrichs   editorial intern: kimberly mackenzie   editorial advisors: lindsay alcock, alison brettle, michelle dunaway, jonathan eldredge, denise koufogiannakis   copyeditors: richard hayman (lead copyeditor), georgianne bordner, julie evener, heather healy, julie james, emily kingsland, alison moore, jane morgan-daniel, elizabeth stregger, elaina vitale   indexing support: kate shore   microsoft word news5612 evidence based library and information practice 2009, 4:1 evidence based library and information practice news invitation to the 5th international evidence based library and information practice conference (eblip5) in stockholm, sweden © 2009. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. the 2009 conference theme is bridging the gap and five key note speakers will expand on different aspects of this theme, among them the well-known dr. jonathan eldredge and professor sue mcknight. in addition to the keynote speakers, eblip5 will offer participants themed sessions on an array of research papers, user studies and innovative solutions from a wide range of local and international presenters from all library types. five continuing education workshops will be offered, before and after the conference, as well as visits to noteworthy libraries in stockholm. a visit to sweden in june also means the opportunity to enjoy the “white nights” of the nordic summer. as a tourist city, stockholm offers many possibilities for cultural as well as outdoor excursions. situated where lake mälaren meets the baltic sea, stockholm is built on several islands, and is often called the venice of the north. the social program of eblip5 includes a reception in the famous blue hall of the city hall in stockholm, where the nobel prize banquet is held every year in december. the conference dinner location – the restaurant solliden – has one of the best views of the waters and islands of stockholm. you will also have the chance, as part of the social program, to explore the world’s oldest open air museum, skansen! the conference will take place in central stockholm from june 29th to july 3rd. the registration form, along with further information about eblip5 and stockholm, is available at http://eblip5.kib.ki.se. early bird registration ends on april 26th, and registration closes on june 1st. evidence based library and information practice evidence based library and information practice 2009, 4:4  evidence based library and information practice       article      the perception and practice of evidence based library and information practice  among iranian medical librarians      vahideh zarea gavgani  librarian  tabriz university of medical science and education  tabriz, iran  email: vgavgani@gmail.com    received: 14 august 2009        accepted: 13 november 2009      © 2009 gavgani. this is an open access article distributed under the terms of the creative commons attribution  license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and  reproduction in any medium, provided the original work is properly cited.    abstract    objective – evidence based library and information services help to link best evidence  with decision making in library practice.  current library and information science  practice operates in both a knowledge and evidence‐based environment. health  service librarians provide information services in an evidence based health care  context to improve patient care. but the evidence based practice movement has  influenced many fields of human knowledge, including librarianship. therefore, this  study seeks to answer the following questions: 1) what are the perceptions of iranian  medical librarians regarding the use of an evidence based approach in their decision  making processes? 2) do iranian medical librarians apply an evidence based approach  in their professional work? 3) how do iranian medical librarians practice an evidence  based approach? 4) what are the barriers and limitations for iranian medical librarians  who engage in evidence based library and information practice (eblip)?    methods – this study utilized a survey to discover medical librarians’ attitudes and  perceptions towards the use of an evidence based approach to library practice in iran.  data was collected using a structured questionnaire to identify medical librarians’  attitudes toward eblip.    results – the findings of the study indicate that iranian medical librarians are aware  of eblip and that they utilize an evidence based approach towards their lis work.  they practice the five steps of an evidence based answering cycle in formulating,  locating, assessing, applying, and redefining questions. however, they have less  knowledge about levels of evidence, research methodologies, and critical appraisal.     37 mailto:vgavgani@gmail.com http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2009, 4:4  38 conclusions – medical librarians in iran are familiar with the concept of an evidence  based approach. more training is needed in some elements of evidence based practice  to improve their approach to evidence based library and information practice.    introduction    while there are various definitions of evidence  based library and information practice  (eblip), it is generally understood to be the  pragmatic use of the best available evidence  from relevant research in conjunction with  professional experience and users’ preference  in decision making in library and information  science practice.     it might be suggested that the core element of  an evidence based approach in a health care  context originated in the principles set out in  archie cochrane’s 1972 work, effectiveness and  efficiency: random reflections on health services.  cochrane stressed the importance of using  evidence from randomized controlled trials  (rcts), because rcts are likely to provide  more reliable information than other sources  of evidence. although sackett noted that the  ʺphilosophical origins [of evidence based  practice] extend back to mid‐19th century paris  and earlierʺ (71), it was not until the 1980s that  the evidence based medicine approach  emerged as “a system for using medical  information to improve everyday health care  decisions.ʺ(jordan 22).      today the importance of the ʺexplicit and  judicious use of best available evidenceʺ  (sackett 72)  in the everyday practice of  many  fields of human knowledge has led to the use  of an evidence based approach in professional  decision making and practice. evidence based  law, evidence based engineering, and  evidence based librarianship are examples of  the development and growing influence of  this approach in diverse fields and specialties,  beyond the health and medical arena.     the research policy statement of the medical  library association (mla) takes the position  that scientific evidence is the basis for  improving the quality of library and  information sciences now and in the future.  research activity is seen as the foundation of  an evolving knowledge base for the  profession‐‐a knowledge base that will set  health sciences librarians apart from others in  an increasingly competitive world of  information service providers (bradley and  marshall 147).    since the first discussion of an evidence based  approach for library and information science  by eldredge (“commentary”), various authors  have provided different definitions of eblip,  including booth (”exceeding”), eldredge  (“overview”), and crumley and  koufogiannakis . each of these authors has  discussed how an evidence based approach  works in library and information science and  practice. though the definitions differ, they  share four common criteria that help to  explain how library and information practice  can be evidence based:  • user centricity   • research driven  • pragmatic  • integrated with professional  experience and observation  these four basic elements and many of the  same research methods used in ebm are  incorporated by ebl to improve library  practice.  however, ʺto adapt core  characteristics from ebm does not imply that  ebl imitates ebm, or even ebhc, blindlyʺ  (eldredge,”overview” 290).     since the emergence of ebm concepts in the  information service of medical libraries,  librarians have been involved in ebm  processes in theory and practice, training  clinicians to locate evidence in the literature  (hill; klem and weiss; mccarthy; and  crumley) and providing access to the best  available evidence and reliable information  (marshall; williams and zipperer; lucas et    al.).      evidence based library and information practice 2009, 4:4  partridge and colleagues noted that  ʺincreasingly, the library and information  science (lis) practitioner is being challenged  to incorporate evidence based practice into the  context of their [sic] professional work.ʺ(2) a  review of farsi language literature indicated  that although the evidence based approach  has not been formally included in the  curriculum for medical librarianship in iran, a  medical librarian delivered the first formal  lecture concerning the topic of ebm in a  presentation to iranian medical school faculty   in 2004.  (gavgani, “qualified”). gavgani also  authored a paper researching the use of ebm  among clinical faculty in iran (“approach”).  the latter paper concluded that clinical faculty  members have little practical knowledge about  ebm or sources of evidence based  information. the study also reported that  iranian clinical faculty lacked knowledge  about methods of searching and retrieving  evidence.  gavgani concluded that training in  searching and retrieving evidence based  information was needed. this led to the  formation of the center for evidence based  medicine at the tabriz university of medicine.  the center recruited a number of medical  librarians to collaborate in its training  programmes.     aims and objectives    the question remains, however, as to whether  librarians incorporate an evidence based  approach in their practice more for their own  professional development rather than as a  service for users. thus, the present study aims  to determine iranian medical librarians’  perceptions and attitudes towards eblip, in  addition to investigating how eblip is  practiced in iran.    the study sought to address the following  questions:  • what is the perception of iranian  medical librarians about using an  evidence based approach in their  decision making process?  • do iranian medical librarians apply an  evidence based approach in their daily  library practice?   • how do iranian medical librarians  practice an evidence based approach  in their daily library practice and  services?  • do iranian medical librarians face  barriers and limitations in practicing  eblip?    literature review    there are few studies about the actual practice  of eblip in libraries. a presentation by brice  et al. at the 2005 ifla conference provided a  practical introduction to evidence based  information practice. their paper, “evidence  based librarianship: a case study in the  social sciences” sought to equip participants  with the skills required for evidence based  practice in the workplace. a social sciences  scenario demonstrated how evidence can be  used to support library management  decisions. concepts such as formulating  focused questions, understanding research  design, and critical appraisal were applied. in  conclusion, the study suggested that library  and information staff need to consider, and  plan for, practical steps to introduce the  concept of evidence based practice in their  workplaces. lewis and cotter examined the  similarities and differences between research  questions asked by librarians in 2001 to those  posed in 2006.  they also explored the extent  to which the published research supports the  questions being asked. they found that in  2001 and 2006 the most commonly asked  questions were regarding management and  education issues.     westwood examined one librarian’s  experience in applying an evidence based  librarianship model to her practice as a  humanities librarian. she concluded the paper  by expressing her success and proposing the  self evidence based model as a way forward.    the literature review highlighted a lack of  studies examining medical librarians’ practices  and their perceptions of eblip. this study  sought to understand iranian medical  librarians’ perceptions of eblip and to  39 evidence based library and information practice 2009, 4:4  40 examine how eblip is used in everyday  decision making processes.     methods    the study was carried out utilizing a survey  approach. a random sample of 100 librarians  was selected from the current membership list  of the iranian medical library association.  the total number of members was 600 at the  time. a structured questionnaire with fixed  responses and a set of attitude statements  using likert scales formed the basis of the  questionnaire (see appendix).  open‐ended  questions were also included to capture  respondent’s opinions. the questions were  derived by extracting the factors and criteria  for practicing eblip, levels of evidence, and  the generally accepted five steps of evidence  based practice from relevant evidence based  practice literature. for example, the evidence  based cycle that has been recognized by a  number of authors in the ebm/eblip  literature (rosenberg et al.; cook, jaeschke,  and guyatt; epling et al.; eldredge  “formulating”) was used to formulate  questions 14‐17.  open‐ended questions  helped determine which methods of  evaluation and appraisal of evidence were  used by these librarians and which sources of  evidence they most frequently consulted.  responding librarians used likert scale  response options (ʺstrongly agree,ʺ ʺagree,ʺ  ʺneutral,ʺ “disagree,ʺ and ʺstrongly disagreeʺ)  to respond to statements regarding their  perceptions about eblip and its importance in  their daily work.   electronic mail was used to collect data from  the sample. email was also used to clarify   opinions and responses where necessary.   results    there were 100 questionnaires distributed  among medical librarians in iran, and 63 valid  responses were returned and analyzed. the  findings of the study are presented below.    what are iranian medical librarians’ perceptions of  eblip?    familiarity with the eblip concept  respondents were asked if they knew what  eblip was and, if so, from which sources they  obtained information about eblip. a total of  63 librarians answered this question.  of these  n=32 (50.7%) stated that they knew about  eblip, and n=31 (49.2%) stated that they had  not heard the term. those familiar with the  term eblip (n=32) also responded to the  question, ʺfrom which sources did you receive  information about eblip?ʺ. respondents were  able to select more than one response; n=17  (53.1%) said they had discovered eblip from  the ʺliterature.ʺ other responses included  ʺfriendsʺ n=9 (28.1%), ʺdiscussion groupsʺ n=6  (18.75%), and ʺother mediaʺ n=9 (28.1% ).    the librarians were asked whether they  searched for evidence in the published  literature for everyday decision making. of  the 49 respondents to this question, 79.1%  stated that they search for evidence when they  face questions in daily lis practice.     why engage in eblip?  to determine the librarians’ views about the  importance of eblip in their daily practice,  they were asked the extent to which they  agreed with the following statements.                        evidence based library and information practice 2009, 4:4  table 1   the importance of eblip for iranian medical librarians  strongly  agree  agree    neutral    disagree    strongly  disagree  respondents attitude   statement    #  %  #  %  #  %  #  %  #  %  #  eblip  ensures risk  free decision  making.  16  25.4  40  63.5  5  7.9  2  3.2  0  0  63  practicing  eblip  improves the  lis  profession  and its  practice.   12  20  36  60  10  16.7  2  3.3  0  0  60  i practice  eblip to  demonstrate  the correct  practice for  my  organization.  24  38.1  30  47.6  5  7.9  4  6.3  0  0  63  i don’t  practice  eblip,  because i am  not a policy  maker or  decision  maker.  7  11.1  3  4.7  35  55.5  16  25.3  2  3.1  63      the majority of respondents “strongly agree”  or “agree” that eblip ensures risk free  decision making, improves the lis profession  and practice, and demonstrates correct  practice. a negative statement asked whether  the respondents do not practice eblip because  they are not library managers. just over half of  respondents (55.5%) were “neutral” about the  statement ʺi don’t practice eblip because i am  not a policy maker or decision maker.ʺ  another 28.4% responded “strongly disagree”  or “disagree”     with the statement, while 15.8% of the  respondents said they “strongly agree” or  “agree” with the statement.     formulating questions and accessing evidence   to ascertain whether librarians consider the  ʺfive steps of evidence based practiceʺ when  they engage in eblip, participants were asked  to answer a set of statements by selecting one  of five likert scale responses (“strongly  agree,” “agree,” “neutral, “disagree,” or  “strongly disagree”).               41 evidence based library and information practice 2009, 4:4  table 2  do librarians consider the ʺfive steps of evidence based practice” in their daily practice?  strongly  agree  agree  neutral  disagree  strongly  disagree  respond ents  attitude   statement    #  %  #  %  #  %  #  %  #  %  #  i formulate  questions in an  answerable  form.  22  34.9  35  55.6  5  7.9  0  0  1  1.6  63  i access research   evidence for  daily lis  practice.  3  4.9  35  57.4  7  11.5  16  26.2  0  0  61  i assess the  accuracy   of evidence.  28  45.2  31  50  1  1.6  1  1.6  1  1.6  62  i rely on the  evidence and  apply it in daily  lis practice.  18  29  34  54.8  8  12.9  1  1.6  1  1.6  62  i’m usually  satisfied with  whatever  evidence i find,  and then i stop  searching.  1  2  6  11.5  6  11.5  30  57.7  9  17.3  52  the majority of respondents were familiar  with the five steps of ebp, and they follow  these procedures regularly (table 2). the  majority a) formulate questions in an  answerable form; b) access research evidence  in their daily decision making; c) assess the  accuracy of the research evidence;  d) apply  evidence discovered in the literature in their  lis work; and e) continue to search for more  evidence when the evidence is not conclusive.     do iranian medical librarians apply an evidence  based approach to their daily library practice?     evidence based information needs of medical  librarians in daily lis practice   crumley and koufogiannakis articulated six  domains for ebl as ʺmajor areas under which  questions can be grouped.ʺ (63). they  suggested that every lis practice question  falls within one or more of the following  domains: collections, education, management,  professional issues, information access and  retrieval, and reference questions.   in this  study, the questionnaire categorized the lis  practice questions in the following domains,  typical of the lis curricula and daily practice  in iran: management, collection development,  organization (e.g., classification, cataloging,  indexing), standards, user studies, and new  services and trends (question 2).  librarians  were asked to specify in which fields they  needed more evidence to make correct  decisions.     the majority of the respondents (66.7%) stated  that they primarily search for evidence about  lis standards, followed closely by searches  regarding collection development (63.5%).  other reasons cited for locating research  evidence were to help with organization of  library materials (52.4%) and to support  library management (50.80%). “user studies”  and news about “new services and trends”  were each selected as reasons for research by  47.6% of the respondents.  42 evidence based library and information practice 2009, 4:4    fig. 1. evidence based information needs of medical librarians in daily lis practice    to determine which sources librarians consult  most often for lis decision making and  whether they value the combination of their  experience and users’ preference with the  results of research studies, a question with  pre‐determined responses asked about the  sources they consult for evidence in lis  decision making (figure 2).    the majority of respondents (65.1%) stated  that they consult the “literature” as a first  option. other options in descending order  were “libraries with similar experiences”  (39.7%), “senior colleagues” (36.5%), and  “personal experience” (31.7%). the lowest  percentage (30.2%) referred to “lis professors  or faculty members” as a source of evidence.  none of the respondents specified other  sources such as user feedback, perspectives, or  opinions.     the survey asked the librarians to specify  which sources they used most often to locate  evidence based information relevant to their  eblip practice.      fig. 2. what sources do librarians prefer to consult?  43 evidence based library and information practice 2009, 4:4    fig. 3. sources most widely used by librarians to search for evidence      the results indicated that the majority of  respondents (58.7%) search for evidence on the  internet (e.g., google, yahoo!). the next most  popular sources of evidence were electronic or  print journals (34.9%).  books were a source of  evidence for 20.6% of respondents, and 15.9%  stated they consult blogs to answer lis  practice questions. the lowest percentage of  respondents (14.3%) stated that they use lis  databases for their ebl research.    how do iranian medical librarians use an  evidence based approach?    critical appraisal     an open‐ended question sought to learn how  librarians appraise the level of evidence  extracted from relevant literature. the survey  asked, “how do you evaluate the accuracy  and reliability of the evidence gained from  research studies?” only 31 of the 63 total  respondents (49.2%) answered this question.  of these respondents, 20.6% referred to  ʺcitation analysis,ʺ which they explained as the  number of articles cited by a particular study  and the reputation of the articles that, in turn,  cited the original source documents. other  factors the librarians considered included an  evaluation of the “reputation of the sources”  (11.1%), the “reputation of the author of the     article” (4.8%), “research methods” used in the  study (3.2%), “comparative methods” (7.9%),  and “statistical methods” (1.6%).     levels of evidence    eldredge and colleagues noted that levels of  evidence in lis research influence decision  making in librarianship. (“weeding”)  this  study sought to examine iranian medical  librarians’ attitudes towards levels of evidence  and to determine whether and how these  librarians link the evaluation of evidence to  the level of evidence.  respondents who  answered the earlier question (number 6)  about evaluating evidence in eblip, were then  asked about levels of evidence (figure 4).    most of these respondents (40%) stated that  they use evidence based information extracted  using various research methods or from the  literature, without considering the type of  research method used. the survey method  ranked second with 20% of the responses.  next were systematic reviews and case  studies, each listed by 13.3% of the  respondents. the lowest percentage  mentioned using quantitative (6.7%) and  qualitative studies (6.7%). none of the  44 evidence based library and information practice 2009, 4:4    fig. 4. in which level of evidence do librarians find research evidence for their decision making?      respondents referred to using randomized  control trials (rcts), and none referred to any  other research methodology used in lis  practice.    has evidence based information changed iranian  medical librarians’ practice?      respondents were asked if evidence based  information had changed their daily lis  practice and, if so, in which areas. the  majority (67%; n=42) said that the evidence  they find from the literature impacts their  practice and creates change in their decision  making. for 20% of these respondents, there  was no change in lis practice based on  evidence based information.     in terms of the areas where evidence based  practice has led to changes, 40.5% stated that  eblip leads to changes in lis service, 33.3%  saw changes in the organization of library       collections, 21.4%  reported changes in library  management, and 4.8% noted changes in  technical areas.    to better understand whether eblip has had a  creative or reformative role in lis practice,  respondents were asked how eblip impacts  in their decision making and lis practice. the  majority of respondents (66.6%) stated that  eblip impacts their lis practice by creating  change in existing procedures and practices,  and 60.3% stated that it leads to new decisions  in library and information practice. this  suggests that eblip is useful for both existing  lis practice and for the development of new  services.    are there any barriers and limitations to eblip in  iran?    a range of questions was posed to discover if  there are barriers for iranian medical librarians  in practicing eblip (table 3).    45 evidence based library and information practice 2009, 4:4    fig. 5. in which areas of lis practice has the eblip movement led to change in iranian medical  libraries?    table 3   barriers and limitations librarians face in practicing eblip and accessing evidence  strongly  agree  agree  neutral  disagree  strongly  disagree  respond ents  attitude   statement    #  %  #  %  #  %  #  %  #  %  #  in some  instances there is  no relevant  evidence in the  literature.  6  9.8  20  32.8  10    16.4    23    37.7    2    3.3    61    as there is less  research  evidence in the  library and  information  science  literature, i’m  satisfied with  any evidence i  find.  1  1.9  6  11.5  6  11.5  30  57.7  9  17  52  there is  abundant lis  evidence based  information in  farsi literature.   0  0  4  6.6  9  14.7  40  65.6  8  13.1  61  it does not  matter in which  language the  relevant evidence  or information  has been  published, but it  is important to  find the  evidence.  15  24.6  17  27.7  3  4.9  23  37.7  3  4.9  61  46 evidence based library and information practice 2009, 4:4  less than half of the respondents agreed with  the statement, ʺin some instances there is no  relevant evidence in the research,ʺ and the  majority of respondents (74.7%) disagreed that  ʺthere is less research evidence in the library  and information science literature.” at the  same time, the majority (78.6%) disagree or  strongly disagree with the statement, ʺthere is  abundant evidence based information in lis in  farsi literature.ʺ just over half of the  respondents (52.4%) agreed with the  statement, “it does not matter in which  language the relevant evidence or information  has been published, but it is important to find  the evidence.” this suggests that iranian  medical librarians do not face problems in  finding evidence based information from the  literature due to a lack of research evidence,  nor do they see the lack of evidence in the  farsi literature as problematic.      discussion     a total of 63 iranian medical librarians  responded to the questionnaire. almost half of  these understood the concept of eblip and  reported they gained their knowledge about  the recent eblip movement through relevant  literature. the majority of these librarians  search for evidence from relevant research and  professional literature to support their daily  library and information work when they face  uncertainty.     the majority of librarians believe that  evidence based library and information  practice ensures risk free decision making,  improves their practice and profession, and  can be cited as a proof of best practice or  correct decision making to upper level  managers in their organizations, because  evidence based decisions are reliable and  usually risk free. however, the majority were  neutral about the statement, ʺi do not practice  eblip because i am not a policy maker or a  decision maker.ʺ  this suggests that although  librarians are knowledgeable about eblip,  believe that it is an essential part of their  practice, and that it provides advantages for  their profession, they do not have strong  convictions about their right to make decisions  regarding their lis practice. it indicates that  paternal, uni‐directional, and top‐ down  decision making methods remain dominant  within iran’s library systems and  management.     the majority of iranian medical librarians in  the study are familiar with the five steps of  ebp (formulating an answerable question,  accessing, assessing, applying evidence, and  redefining the question to the best extent  possible), and they regularly practice the  process. this suggests that medical librarians  in iran understand the concept of eblip and  its importance in their daily lis practice.  although this study did not include tests to  determine the representativeness of the  sample, it documents that these iranian  medical librarians were attentive to a critical  and essential task of the lis profession in their  search for evidence, and that they practiced  eblip.     the aim of evidence based practice is to  overcome uncertainty in decision making and  to improve the outcomes of practice in any  profession. patient safety is the ultimate aim of  ebm in the practice of medicine, thus ebp  should improve patient safety and reduce  uncertainty and risks in treatment, care, and  diagnosis. in the same manner, for library and  information science there may be uncertainty  about new information services, the selection  of new library materials, new methods and  technologies in information service, or the  collection development decisions to purchase  new titles or weed others. the iranian  librarians in this study paid less attention to  new services and trends and more to  standards, management, and organization.     management and organization represent  strategic and risky decision making, but  standards usually represent stability, certainty,  and background knowledge.  evidence based  practice is a ʺpractice in the context of rigorous  research, it is framework for decision making  and supports lifelong learning. it is not  empiric, based on tradition, cook book,  perspective limiting, or static” (hannigan and  pokala).  searching for evidence based  48 evidence based library and information practice 2009, 4:4  information about standards indicates that  librarians in iran are concerned about  principles, rather than trends. developed  countries like the usa (gluck; hassig et al.),  canada (chla), and germany (ahrens et al.)  have already developed their own standards  for health science libraries, and they regularly  update and revise their standards. in iran,  however, there are no standards for health  science libraries. in his primary conceptual  framework for evidence based librarianship,  eldredge stated that ebl supports the  adoption of practice guidelines and standards  developed by expert committees based upon  the best available evidence, but not as  endorsement of adhering to rigid protocols  (“overview” 291).     ʺcollection management is a further area that  most libraries would see as core business,  whether it be book acquisition or withdrawal  or management of the serial collection.ʺ  (booth, ʺusingʺ 82 ). there are several  examples of small‐scale projects that would  allow a library manager to demonstrate the  application of evidence based practice. these  include selective weeding of sections of the  collection (eldredge, mondragon, and fierro)  and examining the use of the reference  collection. (booth, ʺ on the shelfʺ 154 ).  similarly, annual performance appraisals or  staff development reviews should include one  specific and measurable objective that relates  to achievement of an agreed task in an  evidence based way (booth ʺusingʺ 83). this  study also revealed that these librarians search  for evidence based information in all areas of  library operations, including management,  organization, user studies, and new services.     ʺevidence based practice is the integration of  research with professional experience and  consumer valueʺ, eldredge stated that ʺebl  seeks to improve library practice by utilising  the best available evidence combined with a  pragmatic perspective developed from a  working experience in librarianship.ʺ  (“overview” 291) the data in this study shows  that iranian medical librarians are  knowledgeable about sources of decision  making that can support the successful  practice of eblip. the author’s anecdotal  experiences as a medical school librarian  suggests that librarians traditionally value  usersʹ preferences and choices in their decision  making, especially when it comes to service,  organization, and collection development.     in this study the survey instrument was  designed to avoid giving respondents an  explanation of the eblip process to ensure  their objective responses regarding the basic  components of the process, especially the  inclusion of users’ preferences and feedback as  sources of decision making. however, none of  the respondent librarians entered any data in  this option. if eblip is a combination of  research evidence, personal experiences, and  user preferences (booth “exceeding”; crumley  and koufogiannakis; eldredge, mondragon,  and fierro; wilson), librarians in iran are  missing one of the basic components of eblip‐ ‐that of users’ preferences‐‐in their decision  making. rigid rules and regulations still  dominate lis practice in iran, and in spite of  having the knowledge and ability to practice  eblip, librarians are either not able to do so or  are not fully authorized to incorporate  evidence into their practice.      a further issue is the reliability of the evidence  on which iranian librarians base their  decisions.  more than half of the responding  librarians stated they search for evidence on  the internet using google or another browser.  although google is a powerful search engine,  it cannot provide full‐text access to most  scholarly journals or databases, and relying on  google alone will lead to missing most of the  best evidence available. another resource  cited by librarians in this study are blogs,  which record the personal opinions of their  authors. as the purpose of eblip is making  correct, unbiased, strong, risk free, and cost  effective decisions based on the results of up‐ to‐date and rigorous research, blogs cannot be  considered as a qualified and reliable source  for evidence based practice. there are also  problems in relying on books for evidence to  support eblip.     49 evidence based library and information practice 2009, 4:4  books as a source of information are more  useful for learning background  knowledge such as principles,  fundamentals, and history rather than  foreground knowledge such as trends,  latest approaches and on‐going research.  since the evidence based decision making  process needs foreground knowledge,  books should be considered only as  secondary sources for extracting  information. (gavgani and mohan  7)      furthermore, less than half (n=31) responded  to the open‐ended question about methods  used by librarians to evaluate the accuracy,  reliability, and applicability of evidence.  critical appraisal uses intrinsic factors (design,  etc.), rather than extrinsic factors (author,  journal, institution) to determine the quality of  an article (booth and brice).  intrinsic factors  such as research methods (3.2%) and statistical  methods (1.6%) were stated by less  respondents compared to extrinsic factors  such as “reputation of the sources” (11.1%)  and “reputation of the author of the article”  (4.8%) as methods of critical appraisal. none  of the librarians referred to critical appraisal  tools such as cristal checklists to evaluate  the validity, reliability, and applicability of  evidence found from the literature.  however,  a small percentage of librarians (7.9%) did  refer to comparative methods that might be  considered as one of the five steps in the  spice process (booth and brice; perryman).  this suggests that librarians have gained  knowledge in formulating answerable  questions, accessing evidence from the  resources by providing information, reference,  and ebl services for clinicians and other  health professionals, but they have not  acquired the skills for evidence based practice  through education. theoretically they mix  reference service, information literacy, and the  traditional evaluation of reference materials  with evidence based information or levels of  evidence.  the longstanding use of authority  and reputation of publisher and author, date  of publication or copyright, and references or  citations were mentioned by the majority of  respondents in assessing the reliability and  validity of evidence. the findings of the study  in this regard substantiate eldredgeʹs concept  that ʺreference librarians are experts at helping  library users articulate and refine their  research questions. reference skills can be  helpful in formulating ebl questionsʺ  (eldredge, “formulating” 74).  however, it  does not mean that librariansʹ acquired  knowledge is efficient for successful evidence  based library and information practice.   librarians need to be trained in research  methodologies to not only conduct rigorous  research but also to be able to judge the  validity, reliability and applicability of the  evidence and information they retrieve for  their users and themselves.    more than 60% of the responding librarians  stated that eblip has either changed their  practice or it has led to new decisions and  practices. this indicates that eblip has both a  creative and a reformative role in the lis  profession and in librarians’ daily decision  making. it can either develop new methods,  services, and strategies, or it can reform  already existing rules, regulations, and  strategies.    in terms of barriers to practicing eblip in iran,  respondents are neutral regarding a potential  lack of evidence in the lis literature and also  believe that there is an insufficient amount of  research evidence in farsi language. however  this does not act as a barrier, as iranian  medical librarians continue searching until  they find suitable evidence in any language.     conclusions and recommendations    this is a significant study about the practice of  eblip among medical librarians in iran. it  revealed that medical librarians in iran have  knowledge of the evidence based approach,  that they know how to search for evidence,  and that they are fully aware of the five steps  of evidence based practice. however, they  have less practical knowledge about using  eblip in their daily practice, and they are  more concerned about standards and library  organization rather than trends and new  services. they do not pay much attention to  the quality and reliability of evidence or to the  50 evidence based library and information practice 2009, 4:4  sources of evidence. furthermore, few are  familiar with critical appraisal methods.  ʺwithin the last decade critical appraisal has  been added as a topic to many medical school  and uk royal college curricula, and several  continuing professional development ventures  have been funded to provide further trainingʺ  (parkes et al. 1). for this reason iranian lis  curricula should also expand to include critical  appraisal instruction. the new curricula of  health science lis programs must include  eblip, critical appraisal, and rigorous research  methods. librarians and lis researchers  should be encouraged to conduct rigorous  studies with reliable research methods to  investigate and discuss libraries’ current  conditions, problems, and prospects. results  of these studies need to be published in both  local and international journals.  the  professional library associations‐‐imla  (iranian medical library association) and  ilisa (iranian library and information  science association)—should take action to  empower librarians through training  programmes and to reform and enrich lis  curricula by providing structured and  systematic programmes to the ministry of  health and medical science education and  ministry of science and technology. in this  way iranian medical librarianship might be  put on the path to a more research‐based  profession and be able to attain the medical  library association’s vision of research for our  profession.    …a foundation for excellence in health  information practice, for new and  expanded roles for health sciences  librarians, and for attracting excellent  people to the profession.  (medical  library association)        works cited    ahrens, peter, frank baumann, dorothee  boeckh, diana klein, petra  riethmuller, and ilona rohde.  ʺstandards for hospital libraries in  germany,ʺ german medical libraries  association (agmb), nov. 2004. 16  nov. 2009  <www.agmb.de/papoopro/index.php? menuid=12&downloadid=18&reporei d=0>.    booth andrew. “exceeding expectations:  achieving professional excellence by  getting research into practice,”  lianza, 15‐18 oct. 2000.  christchurch, new zealand. 22 nov.  2009  <http://www.shef.ac.uk/~scharr/eblib/ exceed.pdf >.    ‐‐‐. ʺfrom ebm to ebl: two steps forward or  one step back?ʺ medical reference  services quarterly 21.3 (2002): 51‐64.     ‐‐‐. ʺusing evidence in practice: eleven steps  to eblip service.ʺ health information  and libraries journal 26.1(2009): 81‐4.  16 nov. 2009.    ‐‐‐. ʺusing evidence in practice: on the shelf?:  how unused is your reference  collection.ʺ health information and  libraries journal 23 (2006):154‐ 157. 16  nov. 2009‐12‐04 <  http://www3.interscience.wiley.com/c gi‐bin/fulltext/118585536/pdfstart>     booth, andrew, and anne brice. “evidence  based librarianship: the first steps.”  second international evidence based  librarianship conference, 6 june 2003.  lister hall, university of alberta,  edmonton, ab, canada. 20 nov. 2009  <http://www.shef.ac.uk/scharr/eblib/c onf2003.htm>.     bradley, jana, and joanne gard marshall.  ʺusing scientific evidence to improve  information practiceʺ health libraries  review 12.3(sept. 1995):147‐57    brice, anne, andrew booth, and nicola bexon.  ʺevidence based librarianship: a  case study in the social sciences.  “libraries ‐ a voyage of discovery,ʺ  world library and information  congress: seventy‐first ifla general  51 http://www3.interscience.wiley.com/journal/122208156/issue evidence based library and information practice 2009, 4:4  conference and council, 14‐18 aug.  2005, oslo, norway. 12 nov. 2009  <http://www.ifla.org/iv/ifla71/progra mme.htm>.    canadian health libraries association /  association des bibliothèques de la  santé du canada (chla/absc).  “standards for libraries and  information services in canadian  healthcare facilities.” 24 may 2006.   17 nov. 2009 <http://www.chla‐ absc.ca/documents/standards_2006.pd f >.    cochrane, archie l. effectiveness and  efficiency. random reflections on  health services. london: nuffield  provincial hospitals trust, 1972.     cook, deborah j., roman jaeschke, and  gordon h. guyatt. ʺcritical appraisal  of therapeutic interventions in the  intensive care unit: human  monoclonal antibody treatment in  sepsis. journal club of the hamilton  regional critical care group.ʺ journal  of intensive care medicine 7.6 (nov.‐ dec.1992):275‐82.     crumley, ellen t. ʺexploring the roles of  librarians and health care  professionals involved with  complementary and alternative  medicine.ʺ journal of medical library  association, 94.1(jan. 2006):81‐9    crumley, ellen t., and denise  koufogiannakis. ʺdeveloping  evidence based librarianship:  practical steps for implementation,ʺ  health information and libraries  journal 19.2 (june 2002):61‐70.    eldredge, jonathan d. ʺevidence based  librarianship: a commentary for  hypothesisʺ hypothesis 11.3(fall  1997): 4‐7.    ‐‐‐. ʺevidence based librarianship: an  overview,ʺ bulletin of the medical  library association 88.4(oct. 2000):  289‐302.    ‐‐‐. ʺevidence based librarianship  formulating the ebl question,ʺ  bibliotheca medica canadiana  22.2(win. 2000):74‐7.    eldredge, jonathan d., kathryn l.  mondragon, and carol l. fierro.  ʺdoes weeding a monographs  collection increase subsequent usage  of unweeded titles? a randomized  controlled trial.ʺ medical library  association annual meeting, dallas,  tx, usa. 17–23 may 2002.    epling, john, jason smueny, aparna patil, and  fred tudiver. ʺteaching evidence‐ based medicine skills through a  residency‐developed guideline.ʺ  family medicine 34.9 (oct. 2002): 646‐ 8.     gavgani, zarea vahideh, “evidence‐based  medicine approach among clinical  faculty members.” medical journal of  tabriz university of medicine and  health sciences 85(spr. 2006). 22 nov.  2009  <http://medicaljournal.tbzmed.ac.ir/p df/bahar%2085/ae.htm>.    ‐‐‐.  “evidence based medicine and qualified  information.” proceedings of the sixth  national congress of medical library  and information sciences, 2004.  isfahan, iran: isfahan university of  medical sciences. 52.      gavgani, zarea vahideh, and vangari vishwa  mohan. ʺphysicians’ attitude  towards evidence‐based medical  practice and health science library  services.ʺ libres library and  information science research  electronic journal. 18.1(mar. 2008):1‐ 10 18 nov. 2009  <http://libres.curtin.edu.au/libres18n1/ gavgani_final_rev.pdf>.    52 evidence based library and information practice 2009, 4:4  gluck, jeannine cyr, robin ackley hassig,  leeni balogh, margaret bandy,  jacqueline donaldson doyle, michael  r. kronenfeld, katherine lois  lindner, kathleen murray, joan  petersen, and debra c. rand.  ʺstandards for hospital libraries  2002.ʺ journal of the medical library  association 90.4 (oct. 2002): 465–72. 15  nov. 2009  <http://www.pubmedcentral.nih.gov/ articlerender.fcgi?artid=1082947>.    hassig, robin ackley, leeni balogh, margaret  bandy, jacqueline donaldson doyle,  jeannine cyr gluck, katherine lois  lindner, barbara reich, and douglas  varner. ʺstandards for hospital  libraries 2002 with 2004 revisions.ʺ  journal of the medical library  association 93.2 (apr. 2005): 282–3. 17  nov. 2009  <http://www.pubmedcentral.nih.gov/ articlerender.fcgi?artid=1082947>.    hannigan, gale g., pokala, suma. “evidence  based librarianship: evolution,  principle and practice.” atlis annual  lecture. osmania university,  hyderabad, india. 19 feb. 2009.    hill, thomas. ʺfear, concern, fate, and hope:  survival of hospital libraries,ʺ  journal of   the medical library association  95.4(2007): 371–3.    iranian medical library and information  science association (imla). 22 nov.  2009 <http://www.imla.ir/>.    jordan, theresa j.  understanding medical   information: a user’s guide to  informaticsand decision‐making.  new york, mcgraw‐hill, 2001.    klem, mary l., weiss, patricia m., ʺevidence‐ based resources and the role of  librarians in developing evidence‐ based practice curricula.ʺ journal of  professional nursing 21.6 (nov. 2005):  380‐7.    koufogiannakis, denise. ʺestablishing a model  for evidence based collection  management.ʺ fourth international  evidence based library and  information practice conference,  durham, nc, usa, 6–11 may 2007.  8  nov. 2009  <http://www.eblip4.unc.edu/papers/k oufogiannakis.pdf>.    kronenfeld, michael, priscilla l. stephenson,  barbara nail‐chiwetalu, elizabeth m.  tweed, eric l. sauers, tamara c.  valovich mcleod, ruiling guo, henry  trahan, kristine m. alpi, beth hill,  pamela sherwill‐navarro, margaret p.  allen, linda m. hartman, judy  burnham, dennis fell, raymond  pavlick, ellen w. macnaughton, and  nan bernstein ratner. “review for  librarians of evidence‐based practice  in nursing and the allied health  professions in the united states.ʺ   journal of the medical library  association  95.4(oct. 2007):349‐407.  20 nov. 2009  <http://www.pubmedcentral.nih.gov/ picrender.fcgi?artid=2000778&blobtyp e=pdf >.    lewis, suzanne, and lisa cotter. ʺhave the  most relevant and answerable  research questions facing librarians  changed between 2001 and 2006?ʺ  evidence based library and  information practice 2.1(2007):107‐20.  22 nov. 2009  <http://ejournals.library.ualberta.ca/in dex.php/eblip/article/view/98/243>.    lucas, brian. p., arthur. t. evans, brendan m.  reilly, yuri v. khodakov, kalyani  perumal, louis g. rohr, joseph a.  akamah, tunji m. alausa,  christopher a. smith, and jeremy p.  smith. ʺthe impact of evidence on  physiciansʹ inpatient treatment  53 evidence based library and information practice 2009, 4:4  decisions.ʺ journal of general internal  medicine 19.5 pt. 1(may 2004): 402‐9.    marshall, joanne gard. ʺthe impact of the   hospital library on clinical decision  making: the rochester study.ʺ  bulletin of the medical library  association, 80.2(april 1992):169 78.  19 nov. 2009  <http://www.ncbi.nlm.nih.gov/pmc/ar ticles/pmc225641/pdf/mlab00115‐ 0079.pdf>.    mccarthy, laine h. ʺevidence‐based  medicine: an opportunity for health  science librarians. medical reference  service quarterly, 15.4 (win 1996): 63‐ 71.    medical library association. ʺusing scientific  evidence to improve information  practice. the research policy  statement of the medical library  association.” chicago, il: the  association; 1995. 17 may 2009 22  nov. 2009  <http://www.mlanet.org/research/scie nce1.html>.     parkes, julie, chris hyde, jonathan j. deeks,  and ruairidh milne. ʺteaching critical  appraisal skills in health care  settings.ʺ cochrane database of  systematic reviews 23 july 2001  (3):cd001270; amended 2 nov. 2008.  19 nov. 2009  <http://www.ncbi.nlm.nih.gov/pubme d/11686986>.    partridge, helen l., clare e. thorpe, sylvia l.  edwards, and gillian hallam. ʺthe  practitioner’s experience and  conception of evidence based library  and information practice: an  exploratory analysis.ʺ fourth  international evidence based library  and information practice conference,  chapel hill‐durham, nc, usa.  6‐11  may 2007. 21 nov. 2009  <http://eprints.qut.edu.au/9946/1/9946. pdf>.  perryman, carol. “the process of evidence‐ based library and information  practice: the well‐built question.”  16  apr. 2006. 19 nov. 2009     <http://www.unc.edu/~cperryma/teac hing/wellbuilt%20question%20lesson %20plan.htm>.    richardson, w. scott, mark c. wilson, junichi  nishikawa, and robert s.a. hayward.  ʺthe well‐built clinical question: a  key to evidence‐based decisions.ʺ  acp journal club 123.3 (nov.‐ dec.1995): a12‐3.     rosenberg, william m.c., jon deeks, anne  lusher, robin snowball, gordon  dooley, and david l.  sackett.ʺimproving searching skills  and evidence retrieval.ʺ journal of the  royal college of physicians of  london, 32(1998):557‐63.  http://www.biomedcentral.com/pub med/9881313     sackett, david l., william m.c.  rosenberg,  j.a. muir gray, r. brian haynes, and  w. scott richardson.  “evidence based  medicine: what it is and what it  isn’t.” british medical journal  312.7023 (13 jan. 1996):71‐2. 19 nov.  2009  <http://www.bmj.com/cgi/content/full/ 312/7023/71>.    westwood, glenna. ʺevidence based  librarianship in the humanities: an  application for second language  learners.ʺthird international  evidence based librarianship  conference. brisbane, australia. 16‐19  oct. 2005. 18 nov. 2009  <http://conferences.alia.org.au/ebl2005 /westwood.html>.  williams, linda, and lorri zipperer.   ʺimproving access to information:  librarians and nurses team up for  patient safety.ʺ  nursing economic$  21.4 (july‐aug.2003):199‐201.    54 http://www.biomedcentral.com/sfx_links.asp?ui=1472-6920-5-1&bibl=b38� http://www.biomedcentral.com/sfx_links.asp?ui=1472-6920-5-1&bibl=b39� http://www.biomedcentral.com/pubmed/9881313 http://www.biomedcentral.com/pubmed/9881313 evidence based library and information practice 2009, 4:4  wilson, virginia. “evidence based  librarianship.” canadian association  of law libraries conference.  university of saskatchewan,  saskatoon, canada. 26 may 2008. 22  nov. 2009   <www.callacbd.ca/conferences/2008/p resentations/wilson.ppt>.  55 evidence based library and information practice 2009, 4:4  56 appendix  questionnaire sent to iranian medical librarians (translated from the farsi original)    a  name_______________ degree_____________ address ___________________  b  qualification ‐ in which library section or department do you work?  _______________________________________________________________  1  do you know about eblip? if ‘yes,’ from which of the following sources have you  learned about eblip?     literature                                 yes/no           friends                             yes/no           discussion groups                   yes/no           educational curriculum    yes/no    others                                      yes/no  2  in which fields do you need evidence based information to make right decisions?    management (e.g., quality performance, employee issues)                     yes/no  collection development (e.g., vendor choice, journal subscriptions)     yes/no   organization (e.g., classification, cataloging, indexing)                          yes/no  standards                                                                                                  yes/no   user studies                                                                                              yes/no  new services and trends                                                                         yes/no  3  do you use the results of previous research studies in making                yes/no  decisions about your library practice?  4  which sources do you consult and use to access evidence for decisions in your lis  practice?    relevant literature       yes/no          libraries with similar experiences  yes/no  senior colleagues        yes/no         personal experience                          yes/no   lis professors/faculty members      yes/no     other (please specify )                      yes/no___________________________   5  which of these widely used sources do you use to search for evidence?    internet (e.g., google, yahoo!)    yes/no                       lis databases                               yes/no  journals (electronic or print)        yes/no                       books                                            yes/no  blogs                                             yes/no  6  how do you evaluate the accuracy and reliability of the evidence you access from  research studies? ________________________________________  7  has evidence based information changed your library and information (lis) practice?   ____________________________________________  if ‘yes,’ in which areas has the change occurred? ___________________________  8  how does eblip impact your decision making and lis practice?    it has led to development of new services or /methods                        yes/no   it has led to the creation of change and reformation in lis practice    yes/no  9  in which language do you search for evidence based information in lis?    english language                             yes/no                      farsi                                               yes/no  both farsi and english language  yes/no                 evidence based library and information practice 2009, 4:4  other (please specify)                    yes/no ____________________________  please respond to the following statements by selecting the appropriate responses from the  scale below:             1=strongly agree 2=agree 3=uncertain 4=disagree 5=strongly disagree    10  eblip ensures risk free decision making.  1,2,3,4,5  11  practicing eblip improves lis profession and practice.  1,2,3,4,5  12  i practice eblip as a proof of right practice for my organization.  1,2,3,4,5  13  i don’t practice eblip, because i am not policy maker or decision maker.  1,2,3,4,5  14  i formulate questions in an answerable form.  1,2,3,4,5  15  i access research evidence for daily lis practice.  1,2,3,4,5  16  i assess the accuracy of evidence.  1,2,3,4,5  17  i rely on the evidence and apply it in my lis practice.  1,2,3,4,5  18  in some situations there is no evidence in the professional literature.  1,2,3,4,5  19  as there is less research evidence in library and information science  literature, i am satisfied with even one piece of evidence.  1,2,3,4,5  20  i am usually satisfied with whatever evidence i find, and i do not continue  to search.  1,2,3,4,5  21  there is abundant lis evidence based information written in farsi.  1,2,3,4,5  22  it does not matter in which language the relevant evidence or information  has been published, but it is important to find the evidence.  1,2,3,4,5                57 booth, andrew, and anne brice. “evidence based librarianship: the first steps.” second international evidence based librarianship conference, 6 june 2003. lister hall, university of alberta, edmonton, ab, canada. 20 nov. 2009 <http://www.shef.ac.uk/scharr/eblib/conf2003.htm>. microsoft word es_loy.doc evidence based library and information practice 2006, 1:3      80 evidence based library and information practice       evidence summary    information needs of cancer patients are influenced by time since diagnosis, stage of  cancer, patients’ age, and preferred role in treatment‐related decisions   a review of:  kalyani, ankem. “factors influencing information needs among cancer patients: a meta‐ analysis.” library & information science research; 28.1 (2006) 7‐23.     reviewed by:  john loy   learning resources manager, avon & wiltshire mental health partnership nhs trust  callington road hospital  brislington, bristol, uk.   e‐mail: john.loy@awp.nhs.uk    received: 26 may 2006            accepted: 07 july 2006      © 2006 loy. this is an open access article distributed under the terms of the creative commons attribution  license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and  reproduction in any medium, provided the original work is properly cited.      abstract    objective – the author aims to study the  aggregate influence of demographic and  situational variables on the information  needs of cancer patients, in order to inform  the provision of information to those  patients.    design – meta‐analysis.    setting – research articles published in the  medline and cinahl databases.    subjects – english language studies  published between 1993 and 2003. an initial  search set of 196 studies from medline and  283 studies from cinahl were identified.  following rigorous assessment, 12 studies  met the inclusion criteria.  methods – a comprehensive search of the  databases was conducted, initially  combining “neoplasm” with “cancer  patients” using the boolean “or”. these  results were then combined with five  separate searches using the following terms;  information need(s), information seeking,  information seeking behaviour, information  source(s) and information resource(s). this  identified in total 479 english language  articles. based on a review of titles and  abstracts, 110 articles were found covering  information resources or the information  needs of cancer patients. these articles were  then subjected to the further inclusion  criteria and limited to studies which  included: analysis of information needs  and/or information sources of cancer  patients; adults as subjects of the research;  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:3      81 and application of quantitative research  methods and relevant statistics.     this eliminated a further 35 papers. twelve  of the remaining 75 studies were selected for  meta‐analysis based on their use of the same  variables measured consistently in  comparable units. the final 12 studies  included various forms of cancer, and no  distinction was made among them. all 12  studies appeared in peer‐reviewed journals.    main results – the meta‐analysis found  there was consistently no difference  between the information needs of men and  women. five subsets were identified within  the meta‐analysis, and findings for each can  be stated as follows:  • the younger the age of the patient,  the greater their overall need for  information was likely to be.    • during treatment, the time elapsed  from the diagnosis to the  information need was not  significant. once identified, the  information need remained constant.    • during treatment and post‐ treatment phases, the time elapsed  from the diagnosis to the  information need made no  significant difference, with the  information need remaining  constant and continuing into the  post‐treatment phase.    • the stage of cancer made no  difference to the need for  information. those patients in the  advanced stages of cancer required  an equal amount of information to  those in the early stages of cancer.    • the individual patient’s preferred  role in treatment‐related decisions  made a difference to the information  need. patients who took an active  role in treatment‐related decisions  had a greater need for information  than those who did not take an  active role.    conclusion – findings from this meta‐ analysis can be used to guide information  provision to cancer patients, specifically  taking patient age and preferred role in  treatment decision‐making into  consideration. further research into the  reasons behind the lower information needs  among older patients is called for by the  author.    commentary    this is a timely piece, at least in the uk, as  the consideration of the information needs  of patients and greater patient involvement  in healthcare libraries is very much on the  agenda. many libraries, traditionally  involved in the business of information for  providers rather than consumers of  healthcare, are now finding themselves  wrestling with the challenges of providing  patient information. for many this is  uncharted territory, and ankem’s meta‐ analysis provides us with an excellent  navigation tool.    meta‐analysis is a statistical technique to  combine the results of different research  studies in order to ascertain the overall  effectiveness of particular procedures or  interventions. most commonly used in  healthcare, this study is one of the first  examples of meta‐analysis being used in  library and information science, and is  perhaps a baptism of fire for newcomers to  the technique, assuming levels of  knowledge which may not yet be there.  ankem has published a related piece  “approaches to meta‐analysis: a guide for  lis researchers”, which discusses the  methodology specifically in relation to this  article. that guide complements this well,  providing explanatory background reading,  evidence based library and information practice 2006, 1:3      82 and having access to it puts the reader at a  distinct advantage.    a common criticism of published research is  that at times there just isn’t enough detail,  and we are left trying to fill in gaps where  further explanation would have avoided  such frustration. that is certainly not the  case here, where the reader can at times feel  overwhelmed with detail. as such, this is  heaven for those familiar with research  methodology, but for the librarian with a  less confident grasp of statistical techniques  this may be more of a challenge and  consequently of more limited use.     as meta‐analysis is dependent on the  quality of the systematic review upon which  it is based, the author’s reasons for going  into such detail throughout are necessary  and understandable. it is laudable that the  researcher doesn’t just focus his search on  medline, but also broadens scope to  include cinahl, the cumulative index of  nursing and allied health literature. inclusion  of embase would have been even better, as  this would have broadened the scope  further with the potential to identify  research not included in medline and  cinahl. it is difficult to fault the  explanation of the methodology as we know  exactly which search terms were used,  which fields were searched, and how  boolean operators were employed. it is a  delight to read the documented detail of  such a thorough literature search. initial  search results of 196 studies in medline  and 283 in cinahl were ultimately reduced  to 12 studies which meet the final inclusion  criteria. once again, the detailed  methodology leaves no room for doubt as to  how this process evolved. the decision to  limit the search to english language material  could possibly be seen as a shortcoming as it  may not have identified all relevant studies.  however, this really is a tiny quibble and of  no great concern as healthcare research has  demonstrated that excluding languages  other than english from meta‐analyses  makes no difference to overall results  (moher).    the greatest frustration with this study is  that is it seems to ignore the various types  and formats of information. we have no  idea what information was delivered to the  patients, how they received that information,  how it was used, or the format in which it  appeared. is the information face‐to‐face  delivery from the caregiver, patient  information leaflets, digital media, generic,  personalised, or some other form? however,  the author again sidesteps any criticism as,  in fact, the original systematic review has  provided material for three articles.    [this] article published in library &  information science research covers a  meta‐analysis of the level (low versus  high) of information need among cancer  patients. the second article [ankem  2005] published in libres is a  systematic review of the kinds of  information these patients needed. the  third article [ankem 2006], the one most  recently published in information  research, is a systematic review of the  information sources cancer patients  used. (ankem)    in essence, different data sets within a large  pool have been analyzed to explore different  but related themes. the author could  perhaps aid the reader by making it clearer  in this piece that it is one of a triptych of  articles. while this is not an evidence  summary of the other two articles, it is  foolish to ignore them and treat this one in  isolation as the three together give the  reader a much richer picture. for example,  we don’t learn about the types of  information needed by cancer patients here,  as that is a main focus of the article  published in libres (ankem). interested  readers should most certainly seek out the  evidence based library and information practice 2006, 1:3      83 additional two to thoroughly immerse  themselves in the data.    for the librarian working in research this is  16 pages of paradise, 10 pages of which are  packed with meticulously detailed  methodology and results. those working on  the hospital enquiry desk may be frustrated  that, with less than 2 pages of discussion,  the possible implications of the results seem  almost an afterthought. how we should  apply ankem’s findings to plan the strategic  delivery of services is far from clear, and to  get the full picture the reader really does  need to obtain and digest the additional two  equally weighty articles. those less‐inclined  to digest the three would do well to seek out  a systematic review covering very similar  subject matter (rutten) conducted over the  longer time‐scale of 1980‐2003. in addition  to the databases searched for this meta‐ analysis, the rutten systematic review also  includes articles from social science citation  index (ssci) and psycinfo. rutten and  colleagues do not nearly include the amount  of methodological detail, nor go as far as the  meta‐analysis we have here; but they do  discuss type and source of information, and  suggest practice implications. the busy  librarian may find this “one‐stop shop” as  useful in actually planning services as  ankem’s thorough three‐part exploration.    in conclusion, this meta‐analysis is a terrific  journey with a great deal to look at along  the way, but is the destination worth it? yes,  without question. some may feel that this is  a long climb up a hill to find a view, which  they were fairly sure was likely to be there  anyway, but they would be hard‐pressed to  find a more meticulous and organised tour  guide. the conclusion that younger cancer  patients and those actively involved in  treatment‐related decisions have a greater  need for information than others is unlikely  to be a great surprise to anyone working in  the field of cancer care. what we now  unquestionably have is a high‐quality  examination of the literature to support that  model.      works cited    ankem, kalyani. “approaches to meta‐ analysis: a guide for lis  researchers.” library and information  science research 27.2 (2005) :164‐176.    ankem, kalyani. “types of information  needs among cancer patients.”  libres 15.2 (2005): 1.    ankem, kalyani. “use of information  sources by cancer patients: results of  a systematic review of the research  literature.” information research 11.3  (april 2006): 7. ankem, kalyani. e‐mail  to the author. may 2006.    moher, d., et al. “what contributions do  languages other than english make  on the results of meta‐analyses?”  journal of clinical epidemiology 53.9  (2000): 964‐972.    rutten, l.j., et al. “information needs and  sources of information among cancer  patients: a systematic review of  research (1980‐2003).” patient  education and counselling 57.3  (2005):250‐261.        evidence summary   academic library department experience fosters the development of leadership skills relevant to academic library directorship   a review of: harris-keith, colleen s. (2015). the relationship between academic library department experience and perceptions of leadership skill development relevant to academic library directorship. the journal of academic librarianship, 41(3), 246-263. doi:10.1016/j.acalib.2015.03.017   reviewed by: joanne m. muellenbach founding director, health sciences library university of nevada, las vegas libraries las vegas, nevada, united states of america email: joanne.muellenbach@unlv.edu   received: 28 june 2016  accepted: 01 feb. 2017      2017 muellenbach. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – this study sought to identify if the perception of library leadership skill and quality development is equal across departmental experience, and what are the leadership skills and qualities most commonly perceived to be used in each department.   design – quantitative online survey instrument.   setting – master’s colleges and universities from 728 institutions in the united states of america, as classified by the carnegie foundation.   subjects – 666 academic library directors.   methods – selected participants, representing academic library administrative leadership, were contacted by email a maximum of four times and were invited to complete an online survey instrument composed of six sections. the first three sections contained the purpose and confidentiality statements, demographic information, and data on the past five positions held by respondents prior to their current directorship. the next two sections each had 25 statements on a 5-point likert scale, to collect data on perceived leadership skills and qualities exercised by respondents in their most recent three positions. the final section had four open-ended questions to help explain the academic library directors’ responses and provide context for the ratings in previous sections of the instrument.   main results – a total of 296 responses were received, for a 40.66% response rate, which was representative of the institution type demographics, including private non-profit, public, and private for-profit.   the first research question asked: is the perception of library leadership skill and quality development equal across departmental experience? the data used for this question involved all library departments: access services, administration, collection development, digital library services, information technology, reference and instruction, and technical services. when departments were compared pairwise on composite leadership skill scores, administration was significantly higher than another department. results showed that perceptions of leadership quality development appeared to be equal across departments, but leadership skill development was not, and in fact, there was a significant difference between the variances of the composite scores in the population.   the second research question asked: what are the leadership skills and qualities most commonly perceived to be used in each department? results revealed that every leadership skill score except for time management was significant, indicating a difference among library departments based on individual leadership skill scores. respondents perceived that there was a difference in leadership skill (but not leadership quality) development opportunity by department.   departments were also compared pairwise on offering a greater opportunity to develop leadership skills, and overall, academic library directors perceived that there were significant differences in skill development by department. furthermore, respondents overwhelmingly indicated that administration was where they perceived the most leadership skill development opportunities. there was no perceived difference in leadership quality development by department. as well, some departments were reported to provide targeted, department-specific leadership skills, such as resource allocation and budget management.   conclusion – this study offers strong evidence that development of many of the leadership skills necessary for success as an academic library director only present themselves to professionals once they enter the administration department, the library director position, or the assistant director position.   commentary   this study highlights the academic library directors’ role as visional leader, how their role is essential to on-going development of libraries, and the best path for leadership development as they gain expertise within particular departments. the study was incorporated into the author’s dissertation (harris-keith, 2015) and targeted academic library directors at carnegie-designated (2010) master’s-granting institutions.   application of glynn’s (2006) ebl critical appraisal checklist to the study resulted in a valid study. however, it would have been more impressive if the study had expanded its population to include leadership at additional institutions, including shared or outsourced library operations. glynn’s tool determined that the study would have been enhanced if the survey instrument had undergone more rigorous content validity testing, and if field testing of the pilot survey had been expanded, to include academic library directors in newly organized libraries, incorporating new department naming conventions.   study results noted that leadership qualities explored did not demonstrate statistically significant relationships to academic library director work experience. while leadership skills were easily conceptualized as competencies, leadership qualities were more broadly conceptualized, possibly making them more challenging for respondents to measure empirically in terms of their progress or ability. although the instrument was validated, the definition of leadership skills is highly subjective. additionally, the data strongly indicated that neither graduate school nor professional experience outside of the academic library directorship prepared the academic library director for library directorship. this suggests that most first-time academic library directors rarely exercised leadership skills prior to becoming an academic library director. however, just over 40% of respondents had been in a leadership position for four years or less. therefore, the study would have been more revealing if it had looked at data from their most recent five, rather than three, positions. these findings have serious implications for organizational succession planning, and for professional organizations involved in leadership development.   through collaboration with curriculum leaders, practicing academic librarians, and their professional organizations, this report is an excellent starting point from which further research could provide greater insights into the development of leadership skills in academic libraries. in addition, the results could enhance the content within leadership development opportunities available to academic and research librarians, such as those summarized by herold (2015). as well, professional associations serving academic librarians should focus not only on helping individuals learn leadership behaviours but also on helping them learn where in their professional work participants expect to practice those skills. finally, there should also be a call for action amongst academic library directors, encouraging them to collaborate on the design and delivery of leadership opportunities, and to develop work responsibilities that would allow aspiring directors to gain the necessary experience for success in academic library directorship.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154   harris-keith, c.s. (2015). an exploratory study of the relationship between academic library experience and perceptions of leadership skill development relevant to the academic library directorship. (unpublished doctoral dissertation).university of tennessee at chattanooga, chattanooga, tennessee. retrieved 9 march, 2016 from eric database (ed555658) http://files.eric.ed.gov/fulltext/ed555658.pdf   herold, i.m.h. (2015). creating leaders: an examination of academic and research library leadership institutes (pil #69). chicago, illinois: american library association.   research article   maintaining quality while expanding our reach: using online information literacy tutorials in the sciences and health sciences   talitha matlin stem librarian university library california state university san marcos san marcos, california, united states of america email: tmatlin@csusm.edu   tricia lantzy health sciences & human services librarian university library california state university san marcos san marcos, california, united states of america email: plantzy@csusm.edu   received: 20 mar. 2017  accepted: 21 july 2017        2017 matlin and lantzy. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – this article aims to assess student achievement of higher-order information literacy learning outcomes from online tutorials as compared to in-person instruction in science and health science courses.   methods – information literacy instruction via online tutorials or an in-person one-shot session was implemented in multiple sections of a biology (n=100) and a kinesiology course (n=54). after instruction, students in both instructional environments completed an identical library assignment to measure the achievement of higher-order learning outcomes and an anonymous student survey to measure the student experience of instruction.   results – the data collected from library assignments revealed no statistically significant differences between the two instructional groups in total assignment scores or scores on specific questions related to higher-order learning outcomes. student survey results indicated the student experience is comparable between instruction groups in terms of clarity of instruction, student confidence in completing the course assignment after library instruction, and comfort in asking a librarian for help after instruction.   conclusions – this study demonstrates that it is possible to replace one-shot information literacy instruction sessions with asynchronous online tutorials with no significant reduction in student learning in undergraduate science and health science courses. replacing in-person instruction with online tutorials will allow librarians at this university to reach a greater number of students and maintain contact with certain courses that are transitioning to completely online environments. while the creation of online tutorials is initially time-intensive, over time implementing online instruction could free up librarian time to allow for the strategic integration of information literacy instruction into other courses. additional time savings could be realized by incorporating auto-grading into the online tutorials.   introduction   much of the recent literature on incorporating online teaching methods in information literacy instruction (ili) has focused on “flipped” and hybrid settings. however, the effectiveness of purely online ili needs to be examined within the context of higher education, particularly when it is used to replace one-shot il sessions. at california state university san marcos (csusm), two librarians replaced in-person il sessions with online tutorials in order to more easily reach a large number of students in critical major courses while still maintaining high levels of student learning. by making the strategic decision to spend less in-person time with students in lower-level courses, the librarians were then able to spend more time on in-person instruction in research-intensive upper-division courses. this study goes beyond examining student perceptions of online versus in-person instruction and focuses on achievement of higher-order student learning outcomes via these two teaching modalities.   csusm is a master’s-granting institution with approximately 14,000 students (csusm, 2015). from 2012-2015, the student population saw a large increase of 32% (csusm, 2015). tenure-track faculty hiring is not increasing at the same rate as the student population, thereby prompting the library to include in its strategic plan a call for the investigation of more scalable methods of instruction. this issue of scalability is not unique to csusm (bracke & dickstein, 2002; nichols, shaffer, & shockey, 2003; kraemer, lombardo, & lepkowski, 2007; greer, hess, & kraemer, 2016), making the development of online learning objects to replace in-person instruction an important area of research in librarianship.   within the csusm library, the y unit undertook a curriculum-mapping project in the 2015-2016 academic year. curriculum-mapping allowed the librarians to make strategic and informed decisions about which courses needed the most library-related instruction, and which type of instruction would be most appropriate for the identified courses. the stem librarian and the health sciences & human services (hshs) librarian took this opportunity to embark on a pilot project comparing the effectiveness of online il tutorials with in-person instruction in required major courses in biology and kinesiology, the 6th and 7th most popular majors at csusm, with over 850 students each (csusm, 2015). in fall 2016, there were four sections of biology 212: evolution (approximately 140 students total) and two sections of kinesiology 306: exercise fitness and health (approximately 65 students total).   traditionally, it has been difficult for librarians to “take” devoted class time for ili within the sciences and the health sciences due to the courses’ tightly controlled schedules (gregory, 2013). additionally, there are an increasing number of online-only and hybrid classes in the sciences and health sciences at csusm, requiring alternative methods of library instruction. despite these challenges, the stem and hshs librarians had previously worked with many classes in these subjects (including the courses being used in this study). however, the curriculum-mapping project identified additional courses in each subject that would benefit from library instruction and with which the librarians had not yet worked. the librarians’ working hypothesis was that if they were able to demonstrate that students who received asynchronous online ili (which wouldn’t require disciplinary faculty to give up any lecture time and could easily be incorporated into online-only and hybrid courses) learned as much as those who received in-person ili, it would be easier to integrate library instruction into additional critical science and health science courses with which they had not worked previously.    literature review   traditional ili in the sciences and health sciences has been based on the information literacy competency standards for higher education (the “standards”) (association of college & research libraries [acrl], 2000). after the rollout of the standards, the science and technology section and the nursing section of acrl adapted them to better suit the needs of their disciplinary populations (association of college & research libraries, 2006; association of college & research libraries, 2013). however, the standards were recently replaced with the more flexible framework for information literacy for higher education (the “framework”) (acrl, 2016). a good amount of research has been done to evaluate online ili in the sciences and health sciences (li, 2011; schimming, 2008; tierney & stefanie, 2013; weiner, pelaez, chang, & weiner, 2012), but (due to the very recent rescinding of the standards) none of this research examines online ili based on the newly adopted framework. greer et al. emphasize how well-suited an online format is to providing framework-based instruction, due to the fact that it can “allow for more exploration and feedback than what may be possible in a more traditional face-to-face instructional setting” (2016, p. 296).   online instruction, a term that is often used interchangeably with “computer aided/assisted” instruction and “computer aided learning,” is instruction that is delivered via the internet (allen & seaman, 2013). for this project, the authors decided that asynchronous online tutorials would best meet their students’ needs. asynchronous instruction occurs “…among geographically separated learners, independent of time or place” (mayadas, 1997, p. 2). in other words, students are able to complete coursework without engaging in a lesson in real-time. in deciding which modality to adopt for this project, the authors consulted the literature on the benefits and drawbacks of different types of online instruction. some of the reported drawbacks of asynchronous online instruction include the expense and time needed to develop and maintain online learning objects (joint, 2003; zhang, watson, & banfield, 2007), the lack of personal interaction between students and instructors (gall, 2014), and the difficulty of incorporating active learning (li, 2011). however, although difficult, it is possible to include active learning into online learning objects (dewald, 1999; nichols et al., 2003; zhang et al., 2007), which is one of the necessary components of effective ili in general (drueke, 1992).   one of the main reported benefits of asynchronous instruction is the scalability, since instructors can design learning objects once and then continue to use these same objects to reach a (hypothetically) unlimited number of students an unlimited number of times until the content becomes outdated (grassian & kaplowitz, 2009; joint, 2003; mestre, 2012; zhang et al., 2007). for students with various learning styles and abilities, asynchronous online tutorials can be more accessible (bowles-terry, hensley, & hinchliffe, 2010; webb & hoover, 2015). additionally, tutorials can be repeated multiple times (bowles-terry et al., 2010) and are self-paced (mestre, 2012; schimming, 2008; zhang et al., 2007). the authors decided that these benefits, in particular the scalability, outweighed the potential drawbacks of asynchronous instruction.   much research has been done to compare in-person to online library instruction. “flipped” or hybrid methods can be used effectively to provide interactive instruction (mestre, 2012; walton & hepworth, 2013) and can allow instructors to focus on higher-order skills and concepts in-class since students are responsible for learning the more basic skills and concepts prior to any face-to-face instruction (gilboy, heinerichs, & pazzaglia, 2015). however, although librarians can cover a greater amount of content in a flipped class, they require the same amount of in-person time (plus the additional prep time to create the pre-class instruction/assignments), thereby negating the potential scalability benefits of purely asynchronous online instruction.   prior research on faculty/student satisfaction with online learning has produced mixed results. schimming (2008) examined medical students’ reactions to online and in-person learning, and found that online students were more satisfied with the instruction, possibly because they were able to control the pacing of the lessons. other studies have also used post-surveys to determine that students experience high levels of satisfaction with online learning (nichols et al., 2003; weiner et al., 2012). however, there are also numerous examples of studies that found lower levels of student satisfaction with online versus in-person instruction (shaffer, 2011; summers, waigandt, & whittaker, 2005). johnson, aragon, and shaik (2000) found that graduate students had slightly more positive reactions to in-person learning, possibly due to the fact that they developed deeper social ties to their instructor and fellow students, and reported a higher level of instructor support.   in addition to the conflicting evidence of student and faculty satisfaction regarding online versus in-person learning, there is also conflicting or inconclusive evidence regarding student achievement of learning outcomes in these different formats. gall (2014) compared in-person and online library orientations and found that all student groups improved their research skills, but the study could not determine whether online students learned as much as or more than in-person students. kraemer et al. (2007) compared in-person, hybrid, and online library instruction and found that “…[both] groups that had contact with a librarian … scored higher on the final exam than the online group…” (p. 337). the authors concluded that “…contact with a librarian is an important component of student learning” (kraemer et al., 2007, p. 339).   however, numerous studies have found that students learn as much as or even more through online instruction as they do in person. silk, perrault, ladenson, and nazione (2015) state that “whether or not student learning occurs likely has more to do with the quality of the material and teaching rather than the type of modality” (p. 154). johnson et al. (2000) found that although students tend to prefer face-to-face over online instruction, there was “no difference in the quality of the learning that takes place” (p. 44). this finding was confirmed by other research in this area (anderson & may, 2010; beile & boote, 2004; greer et al., 2016; nichols et al., 2003; zhang et al., 2007). silk et al. (2015) compared modalities when providing ili to undergraduate business students and found that students performed the same in-person and online on knowledge and attitudinal measures, but online students were actually 10% more successful at finding an empirical article. the authors hypothesize that “…because students were instructed to find research articles online for their projects, maybe library instruction works best when the medium by which the instruction is delivered matches the behavior desired…” (silk et al., 2015, p. 153).   due to the mixed results of many studies in regards to both student achievement of learning outcomes and student/faculty satisfaction with online and in-person learning, additional research needs to be conducted. furthermore, there is a gap in the literature regarding assessment of student achievement of higher-order learning outcomes. joint (2003) notes that it is difficult to teach higher-order il concepts and skills (such as topic development, advanced database searching, dispositions, and “knowledge practices” that require significant critical thinking) through asynchronous online instruction, especially when the learning modules are not integrated into disciplinary coursework. in their systematic review of the efficacy of in-person and computer assisted library instruction, zhang et al. (2007) found that both modalities were equally effective in helping students achieve the learning outcomes, but noted that the majority of studies “focused on teaching of basic library skills, such as use of the library catalog and keyword searching of databases, and of knowledge of library services such as interlibrary loan (and placed less emphasis on teaching more advanced skills)” (p. 483).   in order to address this gap, this study aims to assess higher-order student learning outcomes using online tutorials to replace one-shot ili in the sciences and health sciences. in this instance, the authors used bloom’s “taxonomy of educational objectives” to define “lower-order” learning outcomes as the first three levels in the hierarchy (knowledge, comprehension, application) and “higher-order” learning outcomes as the last three levels (analysis, synthesis, evaluation) (1956). although student satisfaction data was collected regarding the method of instruction, the main focus of this study was the evaluation of student learning. rather than using survey data alone, the librarians evaluated students’ post-instruction assignments in order to assess student achievement of higher-order learning outcomes, a topic that has typically been addressed in the literature by instructors who are providing online instruction through entire courses (lalonde, 2011) or in flipped classrooms (gilboy, heinerichs, & pazzaglia, 2015; walton & hepworth, 2013).   methods   courses   the stem and hshs librarians chose courses with multiple sections in their respective subject areas to compare the efficacy of library instruction delivered in-person and through online tutorials. these courses were selected as a result of a curriculum mapping project that revealed that more scalable library instruction was needed due to either section growth or because the course was transitioning to an online environment. prior to delivery of instruction using two different teaching modalities and collection of student work, the authors obtained approval from the campus institutional review board to embark on the study.   in fall 2016, the stem librarian taught four sections of biology 212 “evolution”, two of which participated in 50 minutes of in-person library instruction while the other two completed online tutorials focused on the same learning outcomes. two biology instructors participated and had one section in each instructional group (in-person and online) to control for any differences due to the instructor. the purpose of librarian-led instruction in biology 212 is to prepare students to conduct research for multiple papers that require them to find, use, and cite both scholarly and popular information. the stem librarian has worked with this course (although with many different professors) for the last four years; this was the first year to incorporate online instruction.   also in fall 2016, the hshs librarian collected similar data in two sections of kinesiology 306 “exercise health and fitness”. as in the biology course, one group participated in 50 minutes of in-person instruction while the other completed online tutorials. both sections had the same course instructor. in kinesiology 306, students must find several types of information on a controversial health topic to demonstrate how information from these sources can vary depending on the audience and the purpose. these source types include non-scholarly popular sources (e.g., blog posts, news articles, and message boards), non-scholarly authoritative sources, and peer-reviewed research articles. the hshs librarian has worked with the primary instructor for several years and began investigating online methods of instruction when the course first began transitioning into hybrid and totally online sections. although finding student learning to be comparable between in-person and online synchronous instruction offered through web conferencing (lantzy, 2016), obstacles to this type of online instruction can be burdensome for course faculty. for these reasons, the hshs librarian decided asynchronous online instruction might be a better alternative for this course.   participants   all participants were undergraduate students enrolled in either biology 212 or kinesiology 306. the authors used a quasi-experimental design and assigned students to instructional conditions based on their section enrollment. a total of 100 students across 4 sections of biology 212 (total enrollment for the 4 biology sections: 120 students) and 54 students across 2 sections of kinesiology 306 completed the assignments and participated in the study (total enrollment for the 2 kinesiology sections: 64 students).   instructional content   recognizing that instructional materials used for in-person library classes would not be appropriate for online asynchronous tutorials, the authors used the backwards instructional design process described by wiggins & mactighe (2006) to develop both the online and in-person sections. after articulating the desired learning outcomes, the authors developed a library assignment to measure the achievement of those learning outcomes. teaching/learning activities were then created that directly addressed the learning outcomes and prepared students to complete the assignments in ways appropriate to each learning environment.   the authors chose to use adobe captivate based on its ability to incorporate active learning components and knowledge checks. the instruction for both the in-person and online classes was developed in alignment with specific course assignments, and therefore reflected the unique il needs of the course. blummer and kristskaya (2009) outlined five best practices for the development of tutorials: identify the objectives of the tutorial, align content with the appropriate guiding standards, collaborate, increase user engagement with active learning, and evaluate. the authors incorporated these along with other accepted practices (such as speaking slowly during recordings and including closed captions) to develop the tutorials.   the biology 212 tutorials (https://microsites.csusm.edu/wp-content/tutorials/biol212-citingtutorial; https://microsites.csusm.edu/wp-content/tutorials/biol212-findingarticlestutorial) provided instruction on the peer review process, search strategies for finding peer-reviewed journal articles, and cse citations. two tutorials (https://microsites.csusm.edu/wp-content/tutorials/kine306-authoritative; https://microsites.csusm.edu/wp-content/tutorials/kine306-scholarly) were developed for kinesiology 306 and included instruction on evaluating online non-scholarly health information and recognizing and finding peer-reviewed journal articles. both sections of the kinesiology class also received a handout (physical or electronic) to assist with developing citations for both the library and course assignment.   tools   all four biology sections completed an identical library assignment (appendix a). the stem librarian graded the assignments together and sorted by section afterwards to eliminate grading bias by instructional group. both sections of the kinesiology course completed a library assignment (appendix b) that was graded by the hshs librarian before being sorted by instructional group. both the stem and hshs librarians created grading rubrics to be used in the evaluation of the completed assignments; each assignment had a total possible point value of 10. for the reflection questions, the authors awarded full points if students mentioned particular key words/phrases, and if they provided enough complexity in the response to demonstrate understanding of the concepts being evaluated.   library assignment data was analyzed using spss statistical software. each assignment measured student learning outcomes related to both basic and higher-order il skills, although they varied greatly in content. therefore grades were compared only within each subject area in order to control for potential bias introduced by the differing content. unpaired sample t-tests were run to determine whether statistically significant differences existed between total library assignment scores in the two instructional groups for each course (see table 1). however, both assignments asked students to critically reflect on how scholarly sources differ from non-scholarly sources – a higher-order il skill. to measure differences in the achievement of higher-order learning outcomes, unpaired sample t-tests were run on question 6a (appendix a biology) and question b5 (appendix b kinesiology).   to supplement the assessment of student learning through library assignments, students in all sections completed an anonymous student survey (appendix c) that measured student attitudes to instruction and provided some indication of their experience. the survey gathered information on the perceived clarity of library instruction, confidence levels after instruction, comfort in asking a librarian for help in the future, and other open-ended feedback.     table 1 unpaired sample t-tests results: comparison of library assignment grades by instructional format in biology 212 & kinesiology 306 course library instruction format n mean sd t p biology 212         1.16 0.25   in-person 55 9.16 0.94       online tutorials 45 9.36 0.65     kinesiology 306         0.47 0.64   in-person 27 8.22 1.09       online tutorials 27 8.07 1.21         results   library assignments   in biology 212, a comparison of library assignment scores in the two instructional groups did not show a significant difference between in-person (m=9.16, sd=0.94) and online library instruction (m=9.36, sd=0.64), t(100)=1.16, p=0.25. however, the mean of the online group was 0.2 points higher than the in-person group. in kinesiology 306, the in-person average was slightly higher than the online group by 0.15 points. the differences between the in-person (m=8.22, sd=1.09) and online section (m=8.07, sd=1.21) scores were not statistically significant, t(54)=0.47, p=0.64.   higher-order student learning outcomes   both library assignments asked students to articulate the differences between non-scholarly and scholarly sources. this task aligns with two framework frames: “authority is constructed and contextual,” and “information creation as a process.” the goal of this question was for students to consider how authority is defined in the academic community and how the peer-review process sets these types of information resources apart from non-scholarly sources. this learning outcome was addressed in question 6a of the biology assignment and question b5 of the kinesiology assignment. a comparison of scores for question 6a in the biology sections showed no statistically significant differences between the in-person class (m=0.96, sd=0.15) and the online class (m=0.98, sd=0.10), t(100)=0.90, p=0.37. the kinesiology scores also showed no statistically significant differences for question b5 between the in-person (m=1.56, sd=0.64) and the online class (m=1.74, sd=0.59), t(54)=1.10, p=0.28. although no significant differences were found between scores on this question, it is interesting to note that the online groups in both courses outperformed the in-person groups on this higher-order critical thinking question (by 2% in the biology course and 9% in the kinesiology course).     table 2 weighted average responses (scale 1-4) course library instruction format n the subject matter of the library instruction was clear and understandable. i feel confident completing the assignment for my class as a result of this library instruction. i feel more comfortable asking for help from a librarian as a result of this library instruction. biology 212             in-person 74* 3.68 3.51 3.60   online tutorials 43 3.58 3.40 3.53 kinesiology 306             in-person 27 3.78 3.63 3.70   online tutorials 16 3.75 3.50 3.63     student surveys   the quantitative results from the student surveys demonstrated a comparable experience in terms of the clarity of instruction, student confidence in completing the course assignment, and comfort in asking for help from a librarian after instruction (see table 2). in each category, the weighted average responses were marginally higher in the in-person environment than the online environment. the largest difference between the two instructional groups was seen in student confidence levels in the kinesiology sections (0.13 higher for the in-person section).   the two open-ended questions on the student surveys asked “what did you find most helpful about the library instruction?” and “what is still confusing for you after the library instruction?” responses to these questions revealed a different set of themes between instructional groups.   positive student comments from the in-person groups were heavily content-oriented. several students in the in-person kinesiology section commented on the helpfulness of learning how to differentiate between the three categories of information resources. many in-person biology students reported the clarity of instruction and the ability to ask questions of the librarian was particularly helpful. one biology student noted, “the information was directly connected to our assignment making it reliant [sic] and any questions that came up were easily answered.”   responses to the question “what is still confusing?” in the in-person groups reflected concepts that are generally difficult for students or concepts that were given less in-class time during the session. students in both courses mentioned citing in apa/cse as confusing, and a few biology students cited differentiating between scholarly and non-scholarly articles as challenging. one kinesiology student also mentioned feeling rushed during the session, writing “[t]his was a lot of information that was introduced in a really short period of time. i would feel more confident if it wasn’t such a rush to get everything done in 50 minutes.”   students in the online groups who completed the tutorials often mentioned the structure, clarity, and active learning activities as the most helpful parts of the tutorials. for example, one kinesiology student stated that the most helpful aspect was “[t]he simple breakdown of topics and the knowledge check in certain areas to make sure i was understanding the material that was being taught.” positive responses from the online biology students highlighted the clarity, interactivity, and pace of the tutorials. one student described the biology tutorials as “…very clear and concise. they tried to answer all of your potential questions before there was time to let you get confused about searching for a topic or how to properly site in cse format.” another biology student found it helpful that they were able to go at their own pace and rewatch the tutorials to ensure they understood the content.   there were some technical glitches in the biology tutorials that survey responses helped to uncover. for example, one student reported “the little hot spot buttons didn’t always work and it wouldn’t let me move on in some sections because even though i was clicking on what it was asking for (i.e. editor names) it wouldn’t let me continue.” another student mentioned the navigation as problematic: “i could not navigate back to a page after completing it, so i found it very difficult to use the instruction for the assignment afterward.” lastly, one student brought up the fact that there was no librarian immediately available to answer questions. the student explained, “throughout the lecture, if i had a question, i was unable to ask anyone so i would just google [sic] it and try to find the answer that way.”   discussion   the online tutorials developed by librarians in this study proved to be as effective as in-person instruction in supporting student learning. while previous studies have shown that online library instruction through tutorials can lead to the same learning outcomes as in-person instruction (anderson & may, 2010; beile & boote, 2004; greer et al., 2016; johnson et al., 2000; nichols et al., 2003; silk et al., 2015; zhang et al., 2007), none of these studies focused specifically on undergraduate-level science and health sciences courses. it is difficult to generalize from non-science-based library instruction because il in the sciences tends to focus on discipline-specific learning goals that can depart from basic library instruction goals. in the kinesiology course, for example, library instruction centred on evaluating different forms of health information based on authority, purpose, and audience. library instruction in the biology course explained the peer review process in the sciences, explored unique features of searching scientific databases, and provided guidance on developing cse citations.   this study also aimed to assess student achievement of higher-order learning outcomes in online and in-person settings by assessing student work on specific library assignment questions that required critical thinking and a deep understanding of information processes and authority. much of the current library research comparing student learning from one-shot in-person and online asynchronous environments focuses on basic library skills such as general catalog use and requesting library materials (zhang et al., 2007) rather than higher-order il skills that involve critical thinking. to address this gap in the literature, the authors identified a common question between the library assignments for the biology and kinesiology courses that measured a higher-order concept that aligns with the framework for information literacy for higher education (acrl, 2016). while the authors found no significant differences in analyzing these answers, it is interesting to note that in both courses, students who took the online tutorials performed slightly better than the in-person group on these higher-order questions. it is possible that this slight advantage is the result of students being able to rewatch tutorials as they work on the assignments. the differences may also be a result of the more rigidly structured nature of tutorials. when explaining peer review (a concept that can be difficult for many students to understand) the organization of the material explaining the process (and how this process changes the way the final information “product” is perceived) may have been more beneficial for students than the more conversational nature of in-person classes.   at the start of this project, the authors decided it was important to require a library assignment for assessment purposes. unfortunately, providing this individualized feedback was extremely time consuming. the authors will likely modify the assignments to be at least partially auto-graded online. in addition to reducing the workload for the librarian, this change will allow for future growth and will reduce the turnaround time for students receiving feedback. developing and assessing these tutorials was labour intensive as well. the librarians had to learn the software and spent eight to twelve hours creating each tutorial. fortunately, any updates or minor changes to the tutorials can be made relatively quickly moving forward. for librarians interested in undertaking a similar project, the authors recommend ensuring administrative support of the project and putting aside an appropriate amount of time for its completion.   overall, the student surveys showed that the student experience in both instructional environments was positive. survey responses also brought technical issues to the attention of the librarians quickly, making this type of feedback a useful indicator of potential problems with the online instruction. the average weighted responses for the first three survey questions demonstrated a very slight but consistently higher average for students in the in-person sections. the open-ended survey results allowed the authors some room to speculate about the causes of this difference in the weighted averages. while it would be imprudent to overstate the importance of non-statistically significant differences, these small distinctions may reflect the impact of in-person librarian-student interaction or the importance of receiving immediate answers to questions that arise when students are learning higher-order il concepts. future research exploring the reasons behind these differences should aim to uncover ways to improve the student experience of online tutorials.   the positive feedback received from the online groups supports the continued use of tutorials in these courses. these students were able to revisit important concepts as they completed their library assignment, leading to a slight advantage in averages for questions measuring higher-order learning outcomes. many students appreciated the structure and clarity of the tutorials, a feature that may be hard to duplicate in the classroom. finally, many students described the active learning components as being valuable. in large classroom environments, students may disengage from group discussions and other active learning activities if they are uncomfortable sharing or worried about offering an incorrect answer. all students participate in the activities on their own terms when completing online tutorials, making the experience more consistent.   the findings of this assessment affirm that it is possible to replace an in-person one-shot library instruction session with asynchronous online tutorials without any significant detriment to student learning in science and health science courses. in the long-term, this could result in a significant savings of instructional hours and the ability to effectively reach a greater number of students in these disciplines. the authors are now redirecting time generally spent in these classes to other upper-level courses in need of in-person instruction and to developing online instructional materials for other courses. pairing this assessment of library instruction with curriculum maps to identify classes that require librarian intervention has made instruction in these programs more strategic and thoughtful.   although the authors controlled for librarian and course instructor, there exist some limitations to this study based on the population and university setting. the results of this study cannot be directly applied to non-university settings, although the findings may be of interest to public or special librarians planning on developing tutorials for instruction or outreach purposes. also, as a result of this article’s focus on student learning and the student experience, various other factors (such as the preferences of course instructors and students) that should be involved in determining which instructional format is best suited for a particular course have not been included. additionally, although the subject areas of biology and kinesiology can be generalized to a certain extent across the science and health sciences disciplines, future studies that measure student learning from online tutorials in different subjects and courses within those disciplinary groups would expand the generalizability of these results. lastly, although the authors determined in this study that it is possible to achieve similar levels of learning through both online and in-person delivery of instruction, these findings do not necessitate that this would always be the case. indeed, this result depends upon the development of high-quality online tutorials – any instructor hoping to achieve a similar result would need to invest time and energy into developing a level of expertise in learning theory and online tutorial development. additional studies comparing achievement of student learning across instructional modalities would add to the generalizability of this finding.   references   allen, i. e., & seaman, j. (2013). changing course: ten years of tracking online education in       the united states. sloan consortium. retrieved from http://eric.ed.gov/?id=ed541571   anderson, k., & may, f. a. (2010). does the method of instruction matter? an experimental examination of information literacy instruction in the online, blended, and face-to-face classrooms. the journal of academic librarianship, 36(6), 495-500. https://doi.org/10.1016/j.acalib.2010.08.005   association of college & research libraries. (2000). information literacy competency standards for higher education. retrieved from http://www.ala.org/acrl/standards/informationliteracycompetency   association of college & research libraries. (2006). information literacy standards for science and engineering/technology. retrieved from http://www.ala.org/acrl/standards/infolitscitech   association of college & research libraries (2013). information literacy competency standards              for nursing. retrieved from http://www.ala.org/acrl/standards/nursing   association of college & research libraries. (2016). framework for information literacy for higher education. retrieved from http://www.ala.org/acrl/standards/ilframework   beile, p. m., & boote, d. n. (2004). does the medium matter? a comparison of a web-based tutorial with face-to-face library instruction on education students’ self-efficacy levels and learning outcomes. research strategies, 20(1), 57-68.   bloom, b. s. (ed.). (1956). taxonomy of educational objectives. new york: david mckay.   blummer, b. a., & kritskaya, o. (2009). best practices for creating an online tutorial: a literature review. journal of web librarianship, 3(3), 199-216. https://doi.org/10.1080/19322900903050799   bowles-terry, m., hensley, m. k., & hinchliffe, l. j. (2010). best practices for online video tutorials: a study of student preferences and understanding. communications in information literacy, 4(1), 17-28. retrieved from http://repository.uwyo.edu/cgi/viewcontent.cgi?article=1008&context=libraries_facpub     bracke, p. j., & dickstein, r. (2002). web tutorials and scalable instruction: testing the waters. reference services review, 30(4), 330-337. https://doi.org/10.1108/00907320210451321   csusm. (2013). campus master plan. retrieved from https://www.csusm.edu/pdc/campus-master-plan/index.html   csusm. (2015). fast facts. retrieved from http://news.csusm.edu/fast-facts/   dewald, n. h. (1999). transporting good library instruction practices into the web environment: an analysis of online tutorials. the journal of academic librarianship, 25(1), 26-31. https://doi.org/10.1016/s0099-1333(99)80172-4   drueke, j. (1992). active learning in the university library instruction classroom. research strategies, 10(2), 77-83.   gall, d. (2014). facing off: comparing an in-person library orientation lecture with an asynchronous online library orientation. journal of library & information services in distance learning, 8(3-4), 275-287. https://doi.org/10.1080/1533290x.2014.945873   gilboy, m. b., heinerichs, s., & pazzaglia, g. (2015). enhancing student engagement using the flipped classroom. journal of nutrition education and behavior, 47(1), 109-114. https://doi.org/10.1016/j.jneb.2014.08.008 grassian, e. s., & kaplowitz, j. r. (2009). information literacy instruction: theory and practice (2nd ed.). new york: neal-schuman publishers.   greer, k., hess, a. n., & kraemer, e. w. (2016). the librarian leading the machine: a reassessment of library instruction methods. college & research libraries, 77(3), 286-301. https://doi.org/10.5860/crl.77.3.286   gregory, k. (2013). laboratory logistics: strategies for integrating information literacy instruction into science laboratory classes. issues in science and technology librarianship, 74. https://doi.org/10.5062/f49g5jsj   johnson, s. d., aragon, s. r., shaik, n., & palma-rivas, n. (2000). comparative analysis of learner satisfaction and learning outcomes in online and face-to-face learning environments. journal of interactive learning research, 11(1), 29-49.   joint, n. (2003). information literacy evaluation: moving towards virtual learning environments. the electronic library, 21(4), 322-334. https://doi.org/10.1108/02640470310491559   kraemer, e. w., lombardo, s. v., & lepkowski, f. j. (2007). the librarian, the machine, or a little of both: a comparative study of three information literacy pedagogies at oakland university. college & research libraries, 68(4), 330-342. https://doi.org/10.5860/crl.68.4.330   lantzy, t. (2016). health literacy education: the impact of synchronous instruction. reference services review, 44(2), 100-121. https://doi.org/10.1108/rsr-02-2016-0007 lalonde, c. (2011). courses that deliver: reflecting on constructivist critical pedagogical approaches to teaching online and on-site foundations courses. international journal of teaching and learning in higher education, 23(3), 408-423. retrieved from http://www.isetl.org/ijtlhe/pdf/ijtlhe1070.pdf   li, p. (2011). science information literacy tutorials and pedagogy. evidence based library and information practice, 6(2), 5-18. https://doi.org/10.18438/b81k8z   mayadas, f. (1997). asynchronous learning networks: a sloan foundation perspective. journal of asynchronous learning networks, 1(1), 1-16.   mestre, l. (2012). designing effective library tutorials: a guide for accommodating multiple learning styles. oxford: chandos publishing.   nichols, j., shaffer, b., & shockey, k. (2003). changing the face of instruction: is online or in-class more effective? college & research libraries, 64(5), 378-388. https://doi.org/ 10.5860/crl.64.5.378   schimming, l. m. (2008). measuring medical student preference: a comparison of classroom versus online instruction for teaching pubmed. journal of the medical library association, 96(3), 217-222. retrieved from https://www.ncbi.nlm.nih.gov/pmc/ articles/pmc2479068/   shaffer, b. a. (2011). graduate student library research skills: is online instruction effective? journal of library & information services in distance learning, 5(1-2), 35-55. https://doi.org/10.1080/1533290x.2011.570546   silk, k. j., perrault, e. k., ladenson, s., & nazione, s. a. (2015). the effectiveness of online versus in-person library instruction on finding empirical communication research. the journal of academic librarianship, 41(2), 149-154. https://doi.org/10.1016/ j.acalib.2014.12.007   summers, j. j., waigandt, a., & whittaker, t. a. (2005). a comparison of student achievement and satisfaction in an online versus a traditional face-to-face statistics class. innovative higher education, 29(3), 233-250. https://doi.org/10.1007/s10755-005-1938-x   tierney, l., & stefanie, w. (2013). health sciences information literacy in cms environments: learning from our peers. the electronic library, 31(6), 770–780. https://doi.org/10.1108/el-06-2012-0063   walton, g., & hepworth, m. (2013). using assignment data to analyse a blended information literacy intervention: a quantitative approach. journal of librarianship and information science, 45(1), 53-63. https://doi.org/10.1177/0961000611434999   webb, k. k., & hoover, j. (2015). universal design for learning (udl) in the academic library: a methodology for mapping multiple means of representation in library tutorials. college & research libraries, 76(4), 537-553. https://doi.org/10.5860/crl.76.4.537   weiner, s. a., pelaez, n., chang, k., & weiner, j. m. (2012). biology and nursing students’ perceptions of a web-based information literacy tutorial. communications in information literacy. retrieved from http://docs.lib.purdue.edu/lib_fsdocs/4   wiggins, g. p., & mctighe, j. (2006). understanding by design (2nd ed.). upper saddle river, n.j: pearson prentice hall international.   zhang, l., watson, e. m., & banfield, l. (2007). the efficacy of computer-assisted instruction versus face-to-face instruction in academic libraries: a systematic review. the journal of academic librarianship, 33(4), 478-484. https://doi.org/10.1016/j.acalib.2007.03.006     appendix a library assignment biology 212   biology 212 – library assignment for this assignment, you will find two sources for your biol 212 writing assignments on the topic of climate change.   1.       write one or two sentences describing your topic and what you’ll be searching for: 2.       now, break this up into keywords that you can use to search the databases and online. combine different concepts with and.   and 3.       find one scholarly article on your topic using one of the recommended databases for this class. author(s):_________________________________________________________ article title:_______________________________________________________ __________________________________________________________________ journal title abbreviation:____________________________________________ year: ____________                         volume: ____________  issue:_____________ pages:____________                        url:_____________________________________ doi:_____________________________________________________________ 4.       find one non-scholarly source on your topic. this could be a webpage, newspaper article, or magazine article. author(s) – if any:___________________________________________________ article/webpage title: _______________________________________________ __________________________________________________________________ magazine/newspaper/website title:____________________________________ __________________________________________________________________ date of publication: _________________________________________________ url:_____________________________________________________________ 5.       write your citations in cse format: a.       journal article citation: b.       popular source citation:   6.       reflection a.       compare and contrast scholarly and popular sources. how are they similar and how are they different? b.       provide an example of how you, as a student, would use the scholarly source you found in step #3 for this class (or another biology class). in other words, what is the best way for you to use a scholarly article to support your argument/thesis/hypothesis? c.        provide an example of how you, as a student, would use the popular source you found in step #4 for this class (or another biology class). in other words, what is the best way for you to use a popular source to support your argument/thesis/hypothesis? d.       who do you think are the intended audiences for scholarly and popular sources? are   appendix b library assignment kinesiology 306   kine 306 – library assignment top of form bottom of form part a:  compare the two non-scholarly online health resources provided below by filling out the table.   source a source b http://www.healthcentral.com/copd/c/215658/167244/cigarettes-cigarettes https://www.drugabuse.gov/publications/drugfacts/electronic-cigarettes-e-cigarettes 1.       when was it created/last updated? 2.       is the information available in languages other than english? 3.       who created the content? do they have expertise on this topic? 4.       who sponsors the site? how is the site funded? (the "about us" section may help.) 5.       is there advertising? if so, what are they advertising? 6.       what is the goal or purpose of this website? (ex. entertain, inform, sell, educate, convince, etc.) 7.       would you consider this to be an authoritative health resource for the topic? why or why not? (think about what you learned in responding to each of the previous questions.) at least 3 sentences. 8.       write an apa citation for one of these two sources. (use the apa handout for help.)     part b:  find a scholarly article on your topic   1.       write one or two sentences describing your controversial health topic:     2.       list at least 5 keywords to help you in your search. they could be synonyms, related terms, or broader terms than your main topic:   3.       find one scholarly article on your topic using one of the recommended databases for this class. fill in the table below with the article information. author(s)   article title   journal title   year   volume   issue (if applicable)   page range   url   doi   database used   4.       write the citation for your scholarly article in apa format:   5.       what makes this source different from the non-scholarly sources you evaluated in part a of this assignment? at least 3 sentences. appendix c student survey used for biology 212 & kinesiology 306     evidence summary   analysis of question type can help inform chat staffing decisions   a review of: meert-williston, d., & sandieson, r. (2019). online chat reference: question type and the implication for staffing in a large academic library. the reference librarian, 60(1), 51-61. http://www.tandfonline.com/doi/full/10.1080/02763877.2018.1515688     reviewed by: heather macdonald health and biosciences librarian macodrum library carleton university ottawa, ontario, canada email: heather.macdonald@carleton.ca   received: 7 feb. 2020                                                                accepted:  30 mar. 2020      2020 macdonald. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29727     abstract   objective – determine the type of online chat questions to help inform staffing decisions for chat reference service considering their library’s service mandate.   design – content analysis of consortial online chat questions.   setting – large academic library in canada.   subjects – analysis included 2,734 chat question transcripts.   methods – the authors analyzed chat question transcripts from patrons at the institution for the period of time from september 2013 to august 2014.  the authors coded transcripts by question type using a coding tool created by the authors. for transcripts that fit more than one question type, the authors chose the most prominent type.   main results – the authors coded the chat questions as follows: service (51%), reference (25%), citation (9%), technology (7%), and miscellaneous (8%). the majority of service questions were informational, followed by account related questions.  most of the reference chat questions were ready reference with only 16% (4% of the total number of chat questions) being in-depth. after removing miscellaneous questions, those that required a high level of expertise (in-depth reference, instructional, copyright, or citation) equaled 19%.   conclusion – at this institution, one in five chat questions needed a high level of expertise.  library assistants with sufficient expertise could effectively answer circulation and general reference questions.  with training they could triage complex questions.   commentary   this evidence summary used the cat critical appraisal tool (perryman & rathbun-grubb, 2014).  the authors clearly state the objectives for this study.  however, the mandate for the library is not stated explicitly. as the authors’ conclusion takes into account their service mandate, it would be helpful to have a clear statement of the mandate. it appears that the service mandate is to provide as high a level of expertise as possible (complete reference service) rather than simply directing users to resources. the literature review provides adequate background on staffing the reference desk, staffing chat reference, and whether question type should impact staffing regardless of medium.    the data collected is for an entire year which provides a broad view of the types of questions asked at the institution. the authors developed a comprehensive coding scheme to evaluate the questions which they provide in an appendix. the authors do not discuss the development of the coding schema, whether they did pilot testing to test reliability, or if the coding was done in duplicate.  percentages and raw data are provided in tabular and graphic representations.  they are easy to read and present the results clearly.  in order to calculate the number of questions requiring a high level of expertise, the authors remove the miscellaneous questions resulting in a percentage of 19%.  it is not clear why the miscellaneous questions should be removed.  when left in, the percentage drops to 17%.    one potential limitation of this article is the absence of analysis by student status.  however, this information may not be collected automatically.  the authors note a potential critique of their study: lack of a comparison of virtual and in-person questions.  they suggest this would be an interesting study on its own.  the bishop and bartlett (2015) study that the authors cite analyzed question type in a variety of media (chat, email, phone and in-person). the authors also note that the types of questions asked may be influenced by the medium itself.  fennewald (2006) found that question type distribution differed between in-person and online questions.   the authors state an institution should consider cost vs. outcome when making staffing decisions for chat.  however, the authors do not articulate what their cost and outcome variables are (presumably staffing and service quality respectively).  including a statement such as the following would have summed up their study nicely: with less than 20% of questions requiring a high level of expertise, the library can maintain high quality chat service by staffing with trained library assistants rather than librarians.  the authors do discuss other factors that could influence staffing decisions in addition to question type: total staff, staff expertise levels, library service mandate, and patron expectations.  this is noteworthy as studies mentioned in the literature review found similar question type distributions but had different staffing models.   this paper adds a comprehensive analysis of chat question type to the growing body of literature.  question type can be helpful in determining staffing for chat but other factors should also be considered.    references          bishop, b. w., & bartlett, j. a. (2013). where do we go from here? informing academic library staffing through reference transaction analysis. college & research libraries, 74(5), 489-500. https://doi.org/10.5860/crl-365   fennewald, j. (2006). same questions, different venue: an analysis of in-person and online questions. reference librarian, 46(95/96), 20–35. https://doi.org/10.1300/j120v46n95_03   perryman, c., & rathbun-grubb, s. (2014). the cat: a generic critical appraisal tool. in jotform – form builder. retrieved from http://www.jotform.us/cp1757/thecat commentary   impacts and reflections on the making meaning symposium for small, independent libraries   eric t. leonhardt public service assistant university of alberta libraries edmonton, alberta, canada email: leonhard@ualberta.ca   received: 1 may 2018                                                                     accepted: 24 oct. 2018      2018 leonhardt. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29441     abstract   this paper discusses how a small volunteer library changed after attending a symposium on indigenous librarianship and metadata. it provides an introduction to the aboriginal teacher education program library, how it was formed, and the decision making process behind it. the paper goes on to summarize key points from speakers at the making meaning symposium, and how they challenge the choices made by volunteers. the results have all been positive, and continue to shape discussions about how to best implement what was learned.   when we think we know, we think we own something, and we take away from what it represents. -ambrose cardinal, personal communication     author statement   when reading this document, it is important to note that i have specific backgrounds and biases that have likely affected this paper. part of this is my heritage as a settler colonial in drumheller, ab, treaty 7 territory, métis region 3. i currently live and work as a guest in amiskwaciwâskahikan, treaty 6 territory, métis region 4.   wherever possible, those mentioned in the paper have been contacted and asked to provide the introductions that they would prefer for themselves. these introductions highlight the importance that land, family, and community has, which is no less important than academic credentials. for those i was unable to contact, i have provided an introduction based on their speaker biography from making meaning symposium.   ambrose cardinal, who is quoted at the beginning of this paper, chose to introduce himself in his own words:   nipâhkwêsiminowin pâhpimoteht nitsiyihkâson. i am a member of the mètis nation of alberta but my ancestors and relatives come from the road allowances of what is now known as prince albert. indigenous people within the fabrication of academia wear many hats as we simultaneously represent the nation in which we come from while jumping through the hoops of academia and existing in ‘the politics of recognition.’ i myself refer to myself as an oskâpewis, a helper for the people. a helper to deflect the heavy burden of colonialism that effects my people on a daily basis. i do this in ceremony, i do this politically, i do this academically, i do this resiliently. i carry the prayers of my people in ceremony and through the work that i do. êkosi mâka. (a. cardinal, personal communication, august 14, 2018)   introduction   located at the university of alberta, the aboriginal teacher education program (atep) “is a teacher education program working to improve the success of aboriginal children” (“aboriginal teacher education program: about”, 2018, para. 1). the atep library is a small classroom lending library designed to reflect indigenous ways of knowing. it has given those working with it a chance to re-examine many of our relationships: relationships of materials, of metadata, and of “parent communities” and how libraries serve them.   the making meaning symposium drastically improved those relationships. held in february of 2018, the symposium brought together “indigenous and non-indigenous peoples interested in indigenous librarianship, community, and metadata” (“making meaning symposium: home”, 2018, para. 1) to discuss the process of decolonizing libraries. by challenging decisions that were increasingly isolating atep library from indigenous communities, the making meaning symposium affirmed that we must all engage in broader discussions surrounding the indigenization of librarianship.   this document specifies three volunteer communities. library volunteers are those with a background in library studies. staff volunteers are those who work for atep in any capacity. future volunteers are for those who will run atep library in the years to come.   before making meaning   while atep library has a tentative start date of july 2017, this collaborative effort has its roots in conversations and interactions built over many years. atep staff have been receptive to questions from the university of alberta libraries (ual) for a long time and met requests for support with passion. they are also enthusiastic about the library, advocating for us while challenging existing barriers to indigenous peoples in academic libraries.   when i was asked to create a display for national first peoples day in 2017, debbie feisst, acting library head at h.t. coutts library and metis librarian from the red river settlement area (winnipeg), was the one to suggest contacting atep. because the display would coincide with canada 150, a celebration of the 150th anniversary of canadian confederation, feisst hinted that staff there might be able to provide some perspective on the concept of a canadian national holiday—there are strong criticisms of the entire affair, including the fact that the celebration is happening at all (marsh & karabit, 2017).   angela wolfe, associate director of atep, did more than just provide perspective. she found articles, newspaper clippings, and helped select the posters we would display. she even volunteered her own time: helping to set up, arrange artwork, and providing resources from atep’s student lounge.   less than a month later wolfe and trish collins, administrator for atep with a bachelor of arts in native studies, decided to approach ual with a project of their own. atep had accumulated a considerable variety of resources over the years, mostly in the form of donations and gifts. these could be useful for students of the program, but there was no way to circulate them or record which students had checked items out.   ual agreed to help by providing library volunteers. they were tanya ball, academic librarian resident and métis librarian from winnipeg, treaty 1 territory, feisst, and the author of this paper. additional meetings were scheduled between atep and library volunteers, leading to consultations on what criteria were needed to ensure correct material handling.   system-wide collaboration   wolfe and collins wanted to develop a system that would mirror the best parts of a lending library while avoiding barriers for indigenous patrons. this system included a need for revised subject headings, a system which would allow them to circulate materials, and cataloguing the collection. it also needed to be flexible on what counted as a patron in case whole cohorts of students asked to borrow a section of the library for use in a community outside of amiskwaciwâskahikan.   to meet these needs, atep library realized it would need experts from other parts of the library system. feisst suggested they connect with sharon farnel, chair of the decolonizing description working group (ddwg) and metadata coordinator for ual. ball also invited sheila laroque, academic librarian resident and métis librarian from saskatoon, treaty 6 territory, to share her experience working with metadata and decolonizing descriptions.   laroque and farnel were able to identify resources and provide direction for atep library, like inviting library and atep volunteers to participate in the making meaning symposium. the ddwg was interested in seeing how the volunteers would utilize the research they had already been doing on a university-wide project, and what challenges or pitfalls we might discover along the way. their help was critical for the project, providing valuable knowledge that library volunteers missed.   challenges   for atep library to succeed, volunteers needed to find or create new solutions to long-standing issues with racism and current models of librarianship. the following is a broad generalization of the decisions made before attending the making meaning symposium.   relationships of materials the first challenge staff volunteers tackled was our classification schema. the 21 categories of the library of congress classification system (lcc), and the 10 classes of the dewey decimal classification system (ddc), are the two most prominent systems across canada – they are also problematic, because of how “european partitions within the thinking space […] are not generally present in such traditional systems” (cherry & mukunda, 2015, p. 550). items would need to be shelved and housed beside resources in a way that made sense to atep and its community, preserving the relationship that materials should have to one another. while volunteer staff briefly considered updating lcc instead of a new system, findings from an extensive survey conducted between 2009 and 2011 were ample evidence that there was no consensus on how best to modify it, confirming the choice of an alternative (lee, 2011). the brian deer cataloguing system (bdcs), created by a. brian deer, a kahnawake librarian in the 1970’s, was put forward by laroque as a solution. citing swanson’s (2015) article on work done for the aanischaaukamikw cree cultural institute: it [bdcs] was not intended to be a universal solution for indigenous materials; however, it has provided a foundation for other institutions to create their own tailored classification systems and for the possible development of a more universally used classification system (p. 571). it also solved another major issue: racist descriptions. instead of modifying settler colonial schema we would start with respect. bdcs is not an adaptation of other cataloguing systems, but an arrangement that respected how indigenous ways of knowing should be better represented. unfortunately, it brought its own well documented trials. as swanson (2015) said, we found that our challenges were similar to those who had adopted this system in the past. lack of resources, notably case studies, a lack of dialogue between small libraries that use non-traditional classification systems, and practical guides to implementation, caused difficulty (pp. 575-576). relationships of metadata   discoverability became the central challenge for atep library. library volunteers were unable to find any practical resources that addressed indigenous worldviews without adapting the library of congress subject headings (lchs). library volunteers, hoping to find something rooted in indigenous ways of knowing like the bdcs, decided to continue searching for alternatives. this decision meant subject headings would not exist when items were ready to circulate, requiring a temporary remedy: web 2.0 functionality.   laroque, while running a comprehensive search for alternative subject headings, recommended an article on the formation of the national research centre for truth and reconciliation. in it were descriptions of a “participatory archive” in which community comments, stories, and contributed headings would be treated with equal weight to “professional” metadata (lougheed, moran, & callison, 2015, p. 607-608). the need for web 2.0 functionality became a critical requirement of the online public access catalogue (opac).   the opac needed to offer friendly, flexible support for users with little to no library experience. omeka, librika, evergreen, koha, and librarything’s tinycat were all evaluated, as each offered free or low-cost opportunities for classroom libraries. while many provided web 2.0 options and allowed for copy cataloguing, few of them were deemed easy to use. both the front and back-end of these options needed to be designed with users in mind, while avoiding excessive references to marc tags or library jargon.   after several discussions, library volunteers selected tinycat. tinycat is an opac offered by the creators of librarything, which began as a free, browser-based cataloging service for private individuals and small libraries. its cataloguing system (librarything) was user-friendly although not without technical issues. librarything treats empty fields for subject headings and classification systems as auto-generating fields, surprising library volunteers who thought they had eliminated racist terminology. tinycat’s design and functions proved less demanding; the layout of records and how to manage the opac were clear to volunteers without library experience.   relationships of parent communities a final challenge for library volunteers was the longevity of the project— administrators and students in atep will operate and maintain the library exclusively, without support from ual. library volunteers looked at the corley smith library in the galapagos islands and illinois state university for guidance on future volunteers. corley smith library, when migrating to a new web-based opac, had been staffed almost entirely by local and international volunteers; while some had a background in library studies, few if any had catalogued since graduating. to ameliorate these concerns, corley smith library built guides and recorded their workflows, which they revised with input from new staff (taylor, jacobi, & foster, 2013). in 2007, illinois state university developed a volunteer coordinator program, which focused on policies for how to train, retain, and evaluate staff performance (schobernd, tucker, & wetzel, 2009). although atep library would have a smaller future volunteer population then either library, they recognized a need for clear and consistent procedures. library volunteers began creating guidelines for librarything cataloguing, utilizing some of the language and techniques from corley smith library. other staff began discussing and drafting possible policies surrounding the library, using illinois state university as a template for successful practice. discussions about creating the role of volunteer coordinator were considered but did not produce any results.   after making meaning   the making meaning symposium influenced nearly all aspects of atep library. it was both eye-opening and humbling to receive direction from professionals whose articles informed the atep library, and from community members for whom racism and settler colonialism in the library has a direct, negative impact on library services. the following paragraphs discuss how the symposium corrected many decisions by library volunteers who would have damaged our relationships with materials, metadata, and parent communities.   relationships of materials   while our choice of using bdcs fit the frame of many previous libraries, it was clear that adapting the cataloging system would require far more work then atep library could manage alone. kim lawson, a member of the heiltsuk nation of bella bella bc, librarian at xwi7xwa library at ubc, and former librarian/ archivist at the union of bc indian chiefs resource centre, mlis, spoke on behalf of xwi7xwa library. she shared the challenges and opportunities that existed when changing or developing cataloguing systems. for xwi7xwa library it required building lifelong relationships with elders and their communities, with the expectation that these lines of communication be maintained and welcomed by xwi7xwa staff. anything less would guarantee that the library becomes out of date, and no longer represent authentic indigenous ways of knowing (k. lawson, personal communication, february 8, 2018).   library volunteers also learned our decision to eradicate all traces of racist or discriminatory classification systems was not ideal. molly swain, metis from calgary, and a master’s student in the faculty of native studies invited library staff to be aware that this would sweep a long history of colonial violence out of sight, which is akin to censorship. she utilized racist terminology as a way to hunt for resources that challenged existing settler colonial structures (m. swain, personal communication, february 9, 2018). in atep library, that would become impossible under our current classification. instead of removing these terms or allowing them to appear for everyone, swain suggested that libraries consider technical solutions, which could mask specific results before they arise.   deborah lee, indigenous studies and community initiatives librarian, provided some in-depth context for understanding the spirit of indigenous librarianship. she displayed how the university of saskatchewan had made indigenous worldviews part of not just the classification, but also the entire website. after consulting with local elders and participants at five indigenous academic gatherings, lee was able to glean insight into the complexity of determining “preferred” terminology by a diverse representation of 50+ indigenous community members (lee, 2011).   these insights included discussions about essential components for designing a decolonized and culturally significant website and virtual library that indigenous peoples are proud to call their own (i.e., the indigenous studies portal) (d. lee, personal communication, february 8, 2018). until this point, volunteers had imagined atep library as something which, under the surface, was indigenized and appropriate, but there had been no discussion how the website or look of the physical collection might prove a barrier to users.   relationships of metadata   jessie loyer, cree-métis and a member of michel nation, also a librarian at mount royal university, made library volunteers aware that their lack of community engagement would also reflect in work surrounding subject headings. pointing out that the “way we refer to indigenous peoples has always been shifting” she argued that this needed to be an ongoing learning opportunity, not a problem (j. loyer, personal communication, february 9, 2018). while atep library had not yet found a source material for subject headings, some library volunteers had envisioned a single, finished work that would require little in the way of updating. if that happened, atep library would become just as outdated as the systems we were seeking to avoid.   dr. chris andersen, a professor and the dean of the faculty of native studies at the university of alberta, called on library sciences to start anticipating “categories that did not yet exist” (c. andersen, personal communication, february 9, 2018). not only, as others had pointed out, were the terms likely to change, but the very concept of change required systems and policies that were ready for flexibility. no fixed, finished system was possible, and libraries that intended to decolonize and create spaces for the indigenization of their resources would have to embrace a continual cycle of revision and change (c. andersen, personal communication, february 9, 2018).   relationships of parent communities   gwichyà gwich'in historian crystal gail fraser revealed a flaw in the plans surrounding the atep library regarding policies. it was more than just the cataloguing language that required community oversight, but also what materials could be displayed, and how the organization behaved internally. fraser also pointed to the university of alberta’s adoption and promotion of “pan-indigenous” narratives and symbology, found on the university of alberta’s library website (c. fraser, personal communication, february 8, 2018.) without engaging those represented in atep library resources on its design and structure, volunteer staff ran the risk of failing to properly develop and display resources in a way that respected the differences between peoples.   while others reinforced that atep library overlooked the challenges of engaging in communities, camille callison[1] opened the doors to collaboration between those in library studies as well. using her previous work across canada as examples, she invited indigenous librarians and allies to join national action being done through the canadian federation of library associations indigenous matters committee, regarding the decolonization and indigenization of information services. collaborating meant more than local interactions, but work with governments and communities in areas outside of traditional library spaces (c. callison, personal communication, february 9, 2018).   without joining cfla-fcab imc, atep library would be unable to connect with communities across canada effectively. it will also allow future volunteers to direct their activity in ways that are meaningful, avoiding having to duplicate work already done by those with more extensive resources and staff complements.   moving forward   after the making meaning symposium, the following concerns are now high priorities for the atep library:   ·         a lack of mechanisms for strong community voice and oversight in current design plans. ·         how to best preserve racist or outdated terminology while minimizing harm to users. ·         evaluation of how the collection is accessed, with special consideration for website design that preserves material relationships throughout atep library. ·         flexible systems and policies regarding future developments. ·         staff engagement with larger metadata initiatives.   the atep library still has a great deal to review after the symposium. while finishing the initial cataloguing, library volunteers are reconsidering the relationships of materials, metadata, and parent communities so that these connections are valid and form the core of atep library. in the words of the symposium itself, “we cannot begin the process of ‘decolonizing’ our practices without bringing in our relatives and community partners” (“making meaning symposium: home”, 2018, para. 2.)   references   aboriginal teacher education program. (2018). retrieved from https://www.ualberta.ca/education/programs/undergraduate-admissions/aboriginal-teacher-education-program   cherry, a., & mukunda, k. (2015). a case study in indigenous classification: revisiting and reviving the brian deer scheme. cataloging & classification quarterly, 53(5-6), 548-567. https://doi.org/10.1080/01639374.2015.1008717       lee, d. (2011). indigenous knowledge organization: a study of concepts, terminology, structure and (mostly) indigenous voices. partnership: the canadian journal of library and information practice and research, 6(1). https://doi.org/10.21083/partnership.v6i1.1427    lougheed, b., moran, r., & callison, c. (2015). reconciliation through description: using metadata to realize the vision of the national research centre for truth and reconciliation. cataloging & classification quarterly, 53(5-6), 596-614. https://doi.org/10.1080/01639374.2015.1008718   making meaning symposium. (2018). retrieved from https://sites.google.com/ualberta.ca/makingmeaningsymposium/home   marsh, k., & karabit, m. (2017). reflections on reconciliation after 150 years since confederation – an interview with dr. cindy blackstock. ottawa law review, 49(1), 15-26. retrieved from https://commonlaw.uottawa.ca/ottawa-law-review/reflections-reconciliation-after-150-years-confederation-interview-dr-cindy-blackstock   schobernd, e., tucker, t., & wetzel, s. (2009). closing the gap: use of student volunteers in an academic library. technical services quarterly, 26(3), 194-198. https://doi.org/10.1080/07317130802520021    swanson, r. (2015). adapting the brian deer classification system for aanischaaukamikw cree cultural institute. cataloging & classification quarterly, 53(5-6), 568-579. https://doi.org/10.1080/01639374.2015.1009669    taylor, s., jacobi, k., knight, e., & foster, d. (2013). cataloging in a remote location: a case study of international collaboration in the galapagos islands. cataloging & classification quarterly, 51(1-3), 168-178. https://doi.org/10.1080/01639374.2012.729552     [1] camille callison is a member of the tsesk iye (crow) clan of the tahltan nation and the indigenous services librarian/liaison librarian for anthropology, native studies & social work at the university of manitoba. she holds a b.a. anthropology, m.l.i.s. first nations concentration and is currently a phd student in anthropology. camille is the vice-chair and indigenous representative of the canadian federation of library associations/fédération canadienne des associations de bibliothèques (cfla-fcab) where she is chair of the indigenous matters committee and a copyright committee member.  she is a member of ifla indigenous matters standing committee, the national film board (nfb) indigenous advisory board and the canadian commission for unesco sector commission on culture, communications & information and vice-chair, canada memory of the world committee. microsoft word es_brown.doc evidence based library and information practice 2006, 1:3      60 evidence based library and information practice       evidence summary    searching for adverse effects in medline and embase requires a combined  approach for efficient retrieval    a review of:  golder, su, heather m. mcintosh, steve duffy, and julie glanville. “developing efficient search  strategies to identify reports of adverse effects in medline and embase.” health  information & libraries journal 23.1 (mar. 2006): 3‐12.    reviewed by:  marcy l. brown  clinical medical librarian  the western pennsylvania hospital, forbes regional campus  monroeville, pennsylvania, united states of america  e‐mail: wordsmith@alltel.net     received: 29 may 2006            accepted: 30 june 2006      © 2006 brown. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to assess the sensitivity and  precision of various search strategies for  retrieving adverse effects studies from the  medline and embase databases.    design – analytical survey.    subjects – a case study using a recently  published systematic review of the  effectiveness and adverse effects of seven  new anti‐epileptic drugs.    setting – medline and embase searches  performed by researchers at the centre for  reviews and dissemination and the uk  cochrane centre search filters design  group at the university of york, uk.    methods – five key approaches to searching  were defined. the first approach used either  text words or controlled vocabulary to  search for specific adverse effects. the  second used subheadings or qualifiers either  attached to drug names found in the  controlled vocabulary (approach 2a) or  ‘floating’ without drug names (approach 2b).  the third approach used text words as  synonyms for the phrase ‘adverse effects.’  the fourth used controlled indexing terms  for adverse effects. the fifth and final  approach used two published search  strategies incorporating study design  (badgett et al., loke et al.).   http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:3      61 these five approaches were used to search  for studies of the adverse effects of seven  new anti‐epileptic drugs. 5,011 unique  papers were retrieved. of these, 236 were  judged potentially relevant and 225 full text  articles were obtained. the inclusion criteria  from a previously published systematic  review (wilby et al.) were applied to the  papers, and 79 met the criteria. five papers  were added to the set after being identified  from reference lists, clinical experts, and  other sources. this new set of 84 studies was  used as a quasi gold standard (qgs) against  which more than 300 combinations of the  five approaches could be tested. to create  the set of possible approaches, the  researchers combined search strategies one  through four in all possible ways, and used  all available subheading combinations from  2a and 2b. the badgett and loke searches  were tested separately.     main results – sensitivity and precision  were determined for each combination.  formulas used to calculate sensitivity and  precision were provided. in medline,  search strategies using floating subheadings  achieved the highest sensitivity. the most  useful single subheading in both medline  and embase was “adverse effects,” with  79.1% and 79.5% sensitivity respectively. of  the more than 300 combinations tested, the  most sensitive combination in medline  included specified adverse effects in  combination with the floating subheadings  “adverse effects,” “complications,” and  “drug effects,” together with text words for  adverse effects. this strategy had 97.0%  sensitivity, but low precision at 2.8%. the  highest precision was achieved by using  subheadings attached to drug indexing  terms. in embase, the strategy of loke et al.  provided the highest sensitivity at 86.3%  and precision of 2.0%. since researchers are  not likely to know in advance all of the  reported adverse effects of a particular drug  therapy, the most sensitive strategies  without specific adverse events were also  identified. the search with the highest  sensitivity in medline had 95.5%  sensitivity, and 97.3% sensitivity in  embase.    conclusion – searching for adverse effects  requires a combination of approaches in  both medline and embase. in medline,  the most sensitive combination yielded  97.0% sensitivity. regardless of the  approach used, precision remains low. an  effective generic search filter for adverse  effects searches may not yet be feasible.  more research is needed on search strategies,  as well as more consistent methods of  reporting and indexing adverse effects.    commentary    locating reports of adverse events is one of  the most difficult evidence‐based clinical  searches to undertake. because adverse  effects are reported a number of ways in the  literature, and because newer and  investigational drugs often have not been  incorporated into a controlled vocabulary,  finding these papers becomes as much an  art as a science. the evaluation of over 300  multiple‐term search strategies in this  research study is an important step toward a  generic clinical search filter such as the ones  that currently exist to retrieve studies of  diagnosis, therapy, prognosis, or etiology.    the authors describe in detail their five  varied search approaches as well as their  methods for creating the quasi gold  standard (qgs) and search combinations.  the researchers admit that additional  adverse effects “were later identified by  browsing the qgs set of records” (9).  adding these additional effects may have  changed the percentage of relevant papers  retrieved.     to create the qgs, the researchers applied  the inclusion criteria developed by a recent  systematic review to the combined results of  evidence based library and information practice 2006, 1:3      62 all search approaches. it is impossible to tell  whether or not these criteria were  appropriate without evaluating the  methodological quality of that particular  review. also, testing search results by  comparing them to the same search results  seems somewhat redundant. testing the  search approaches against a set of externally  developed records would go a long way  toward eliminating any redundancy.    that the authors tested their strategies in  both medline and embase is important,  since the degree of overlap in the two  databases varies. embase indexes  pharmaceutical literature not found in  medline, and medline indexes clinical  information not found in embase. the  difference in the two databases is apparent  in the controlled vocabulary: only one of the  seven drugs has a mesh term and is  indexed in medline, while all seven drugs  are indexed in embase. the implication for  the health sciences librarian is that a truly  thorough adverse effects search should  never use just one database, just as no  librarian should ever base a thorough search  in any discipline on the results from a single  source.    the authors provide useful information on  combinations of search approaches that  yield high sensitivity. although not quite  ready for incorporation into a generic search  filter, the most sensitive search strategy  combining approaches 1, 2b, and 3  (medline) or approaches 1 and 3  (embase) can currently be utilized by  health sciences librarians searching for drug  adverse effects.    precision was low in all search approaches  and strategies, indicating that searchers  must be prepared to filter a high number of  irrelevant papers from any given results set.  this finding points to the need for librarians  to sharpen their critical appraisal and  filtering skills, regardless of the subject of  the search.    works cited    badgett, robert, elaine chiquette, betsy  anagnostelis, and cindy mulrow.  “locating reports of serious adverse  drug reactions.” 7th annual cochrane  colloquium abstracts (october 1999):  pb54. 28 may 2006.  <http://www.cochrane.org/colloquia/ab stracts/rome/romepb54.htm>.    loke, yoon k., sheena derry, and angharad  pritchard‐copley. “appetite  suppressants and valvular heart  disease – a systematic review.” bmc  clinical pharmacology 2 (23 august  2002): 6. 28 may 2006.  <http://www.biomedcentral.com/1472‐ 6904/2/6>.    wilby, jennifer, et al. “clinical effectiveness,  tolerability and cost‐effectiveness of  newer drugs for epilepsy in adults: a  systematic review and economic  evaluation.” health technology  assessment 9.15 (april 2005): 1‐157, iii‐ iv. 28 may  2006.<http://www.hta.nhsweb.nhs.uk/e xecsumm/summ915.htm>.    http://www.cochrane.org/colloquia/ab http://www.biomedcentral.com/1472%e2%80%90 http://www.hta.nhsweb.nhs.uk/e <html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/tr/rec-html40"> <head> <meta http-equiv=content-type content="text/html; charset=unicode"> <meta name=progid content=word.document> <meta name=generator content="microsoft word 15"> <meta name=originator content="microsoft word 15"> <link rel=file-list href="es_scott_29531_files/filelist.xml"> <link rel=edit-time-data href="es_scott_29531_files/editdata.mso"> <!--[if !mso]> <style> v\:* {behavior:url(#default#vml);} o\:* {behavior:url(#default#vml);} w\:* {behavior:url(#default#vml);} .shape {behavior:url(#default#vml);} </style> <![endif]--><!--[if gte mso 9]><xml> <o:documentproperties> <o:author>mallan, katrine</o:author> <o:template>normal</o:template> <o:lastauthor>hinrichs, rachel</o:lastauthor> <o:revision>5</o:revision> <o:totaltime>2</o:totaltime> <o:created>2019-03-14t14:25:00z</o:created> <o:lastsaved>2019-03-19t19:47:00z</o:lastsaved> <o:pages>1</o:pages> <o:words>1196</o:words> <o:characters>7135</o:characters> <o:company>microsoft</o:company> <o:lines>246</o:lines> <o:paragraphs>43</o:paragraphs> <o:characterswithspaces>8288</o:characterswithspaces> <o:version>16.00</o:version> </o:documentproperties> <o:officedocumentsettings> <o:allowpng/> </o:officedocumentsettings> </xml><![endif]--> <link rel=datastoreitem href="es_scott_29531_files/item0009.xml" target="es_scott_29531_files/props010.xml"> <link rel=themedata href="es_scott_29531_files/themedata.thmx"> <link rel=colorschememapping href="es_scott_29531_files/colorschememapping.xml"> <!--[if gte mso 9]><xml> <w:worddocument> <w:trackmoves>false</w:trackmoves> <w:trackformatting/> <w:hyphenationzone>21</w:hyphenationzone> <w:punctuationkerning/> <w:validateagainstschemas/> <w:saveifxmlinvalid>false</w:saveifxmlinvalid> <w:ignoremixedcontent>false</w:ignoremixedcontent> <w:alwaysshowplaceholdertext>false</w:alwaysshowplaceholdertext> <w:donotpromoteqf/> <w:lidthemeother>de</w:lidthemeother> <w:lidthemeasian>x-none</w:lidthemeasian> <w:lidthemecomplexscript>x-none</w:lidthemecomplexscript> <w:compatibility> <w:breakwrappedtables/> <w:snaptogridincell/> <w:wraptextwithpunct/> <w:useasianbreakrules/> <w:dontgrowautofit/> <w:splitpgbreakandparamark/> <w:enableopentypekerning/> <w:dontflipmirrorindents/> <w:overridetablestylehps/> </w:compatibility> <w:browserlevel>microsoftinternetexplorer4</w:browserlevel> <m:mathpr> <m:mathfont m:val="cambria math"/> <m:brkbin m:val="before"/> <m:brkbinsub m:val="&#45;-"/> <m:smallfrac m:val="off"/> <m:dispdef/> <m:lmargin m:val="0"/> <m:rmargin m:val="0"/> <m:defjc m:val="centergroup"/> <m:wrapindent m:val="1440"/> <m:intlim m:val="subsup"/> <m:narylim m:val="undovr"/> </m:mathpr></w:worddocument> </xml><![endif]--><!--[if gte mso 9]><xml> <w:latentstyles deflockedstate="false" defunhidewhenused="false" defsemihidden="false" defqformat="false" defpriority="99" latentstylecount="371"> <w:lsdexception locked="false" priority="0" qformat="true" name="normal"/> <w:lsdexception locked="false" priority="9" qformat="true" name="heading 1"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 2"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 3"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 4"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 5"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 6"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 7"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 8"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 9"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 6"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 7"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 8"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 9"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 1"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 2"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 3"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 4"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 5"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 6"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 7"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 8"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 9"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="normal indent"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="footnote text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="annotation text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="header"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="footer"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index heading"/> <w:lsdexception locked="false" priority="35" semihidden="true" unhidewhenused="true" qformat="true" name="caption"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table of figures"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="envelope address"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="envelope return"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="footnote reference"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="annotation reference"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="line number"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="page number"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="endnote reference"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="endnote text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table of authorities"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="macro"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="toa heading"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number 5"/> <w:lsdexception locked="false" priority="10" qformat="true" name="title"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="closing"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="signature"/> <w:lsdexception locked="false" priority="1" semihidden="true" unhidewhenused="true" name="default paragraph font"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text indent"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="message header"/> <w:lsdexception locked="false" priority="11" qformat="true" name="subtitle"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="salutation"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="date"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text first indent"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text first indent 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="note heading"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text indent 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text indent 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="block text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="hyperlink"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="followedhyperlink"/> <w:lsdexception locked="false" priority="22" qformat="true" name="strong"/> <w:lsdexception locked="false" priority="20" qformat="true" name="emphasis"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="document map"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="plain text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="e-mail signature"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html top of form"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html bottom of form"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="normal (web)"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html acronym"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html address"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html cite"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html code"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html definition"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html keyboard"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html preformatted"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html sample"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html typewriter"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html variable"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="normal table"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="annotation subject"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="no list"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="outline list 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="outline list 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="outline list 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table simple 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table simple 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table simple 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table classic 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table classic 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table classic 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table classic 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table colorful 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table colorful 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table colorful 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 6"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 7"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 8"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 6"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 7"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 8"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table 3d effects 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table 3d effects 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table 3d effects 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table contemporary"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table elegant"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table professional"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table subtle 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table subtle 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table web 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table web 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table web 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="balloon text"/> <w:lsdexception locked="false" priority="59" name="table grid"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table theme"/> <w:lsdexception locked="false" semihidden="true" name="placeholder text"/> <w:lsdexception locked="false" priority="1" qformat="true" name="no spacing"/> <w:lsdexception locked="false" priority="60" name="light shading"/> <w:lsdexception locked="false" priority="61" name="light list"/> <w:lsdexception locked="false" priority="62" name="light grid"/> <w:lsdexception locked="false" priority="63" name="medium shading 1"/> <w:lsdexception locked="false" priority="64" name="medium shading 2"/> <w:lsdexception locked="false" priority="65" name="medium list 1"/> <w:lsdexception locked="false" priority="66" name="medium list 2"/> <w:lsdexception locked="false" priority="67" name="medium grid 1"/> <w:lsdexception locked="false" priority="68" name="medium grid 2"/> <w:lsdexception locked="false" priority="69" name="medium grid 3"/> <w:lsdexception locked="false" priority="70" name="dark list"/> <w:lsdexception locked="false" priority="71" name="colorful shading"/> <w:lsdexception locked="false" priority="72" name="colorful list"/> <w:lsdexception locked="false" priority="73" name="colorful grid"/> <w:lsdexception locked="false" priority="60" name="light shading accent 1"/> <w:lsdexception locked="false" priority="61" name="light list accent 1"/> <w:lsdexception locked="false" priority="62" name="light grid accent 1"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 1"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 1"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 1"/> <w:lsdexception locked="false" semihidden="true" name="revision"/> <w:lsdexception locked="false" priority="34" qformat="true" name="list paragraph"/> <w:lsdexception locked="false" priority="29" qformat="true" name="quote"/> <w:lsdexception locked="false" priority="30" qformat="true" name="intense quote"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 1"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 1"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 1"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 1"/> <w:lsdexception locked="false" priority="70" name="dark list accent 1"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 1"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 1"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 1"/> <w:lsdexception locked="false" priority="60" name="light shading accent 2"/> <w:lsdexception locked="false" priority="61" name="light list accent 2"/> <w:lsdexception locked="false" priority="62" name="light grid accent 2"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 2"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 2"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 2"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 2"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 2"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 2"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 2"/> <w:lsdexception locked="false" priority="70" name="dark list accent 2"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 2"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 2"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 2"/> <w:lsdexception locked="false" priority="60" name="light shading accent 3"/> <w:lsdexception locked="false" priority="61" name="light list accent 3"/> <w:lsdexception locked="false" priority="62" name="light grid accent 3"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 3"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 3"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 3"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 3"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 3"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 3"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 3"/> <w:lsdexception locked="false" priority="70" name="dark list accent 3"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 3"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 3"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 3"/> <w:lsdexception locked="false" priority="60" name="light shading accent 4"/> <w:lsdexception locked="false" priority="61" name="light list accent 4"/> <w:lsdexception locked="false" priority="62" name="light grid accent 4"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 4"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 4"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 4"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 4"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 4"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 4"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 4"/> <w:lsdexception locked="false" priority="70" name="dark list accent 4"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 4"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 4"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 4"/> <w:lsdexception locked="false" priority="60" name="light shading accent 5"/> <w:lsdexception locked="false" priority="61" name="light list accent 5"/> <w:lsdexception locked="false" priority="62" name="light grid accent 5"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 5"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 5"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 5"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 5"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 5"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 5"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 5"/> <w:lsdexception locked="false" priority="70" name="dark list accent 5"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 5"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 5"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 5"/> <w:lsdexception locked="false" priority="60" name="light shading accent 6"/> <w:lsdexception locked="false" priority="61" name="light list accent 6"/> <w:lsdexception locked="false" priority="62" name="light grid accent 6"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 6"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 6"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 6"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 6"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 6"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 6"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 6"/> <w:lsdexception locked="false" priority="70" name="dark list accent 6"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 6"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 6"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 6"/> <w:lsdexception locked="false" priority="19" qformat="true" name="subtle emphasis"/> <w:lsdexception locked="false" priority="21" qformat="true" name="intense emphasis"/> <w:lsdexception locked="false" priority="31" qformat="true" name="subtle reference"/> <w:lsdexception locked="false" priority="32" qformat="true" name="intense reference"/> <w:lsdexception locked="false" priority="33" qformat="true" name="book title"/> <w:lsdexception locked="false" priority="37" semihidden="true" unhidewhenused="true" name="bibliography"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" qformat="true" name="toc heading"/> <w:lsdexception locked="false" priority="41" name="plain table 1"/> <w:lsdexception locked="false" priority="42" name="plain table 2"/> <w:lsdexception locked="false" priority="43" name="plain table 3"/> <w:lsdexception locked="false" priority="44" name="plain table 4"/> <w:lsdexception locked="false" priority="45" name="plain table 5"/> <w:lsdexception locked="false" priority="40" name="grid table light"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light"/> <w:lsdexception locked="false" priority="47" name="grid table 2"/> <w:lsdexception locked="false" priority="48" name="grid table 3"/> <w:lsdexception locked="false" priority="49" name="grid table 4"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 1"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 1"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 1"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 1"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 1"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 1"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 1"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 2"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 2"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 2"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 2"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 2"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 2"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 2"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 3"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 3"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 3"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 3"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 3"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 3"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 3"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 4"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 4"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 4"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 4"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 4"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 4"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 4"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 5"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 5"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 5"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 5"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 5"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 5"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 5"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 6"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 6"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 6"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 6"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 6"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 6"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 6"/> <w:lsdexception locked="false" priority="46" name="list table 1 light"/> <w:lsdexception locked="false" priority="47" name="list table 2"/> <w:lsdexception locked="false" priority="48" name="list table 3"/> <w:lsdexception locked="false" priority="49" name="list table 4"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 1"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 1"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 1"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 1"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 1"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 1"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 1"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 2"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 2"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 2"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 2"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 2"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 2"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 2"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 3"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 3"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 3"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 3"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 3"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 3"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 3"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 4"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 4"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 4"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 4"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 4"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 4"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 4"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 5"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 5"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 5"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 5"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 5"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 5"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 5"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 6"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 6"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 6"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 6"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 6"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 6"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 6"/> </w:latentstyles> </xml><![endif]--> <style> <!- /* font definitions */ @font-face {font-family:"ms mincho"; panose-1:2 2 6 9 4 2 5 8 3 4; mso-font-alt:"yu gothic ui"; mso-font-charset:128; mso-generic-font-family:roman; mso-font-format:other; mso-font-pitch:fixed; mso-font-signature:1 134676480 16 0 131072 0;} @font-face {font-family:latha; panose-1:2 0 4 0 0 0 0 0 0 0; mso-font-charset:1; mso-generic-font-family:roman; mso-font-format:other; mso-font-pitch:variable; mso-font-signature:262144 0 0 0 0 0;} @font-face {font-family:"cambria math"; panose-1:2 4 5 3 5 4 6 3 2 4; mso-font-charset:0; mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:536871559 0 0 0 415 0;} @font-face {font-family:calibri; panose-1:2 15 5 2 2 2 4 3 2 4; mso-font-charset:0; mso-generic-font-family:swiss; mso-font-pitch:variable; mso-font-signature:-536859905 -1073732485 9 0 511 0;} @font-face {font-family:tahoma; panose-1:2 11 6 4 3 5 4 4 2 4; mso-font-charset:0; mso-generic-font-family:swiss; mso-font-pitch:variable; mso-font-signature:-520081665 -1073717157 41 0 66047 0;} @font-face {font-family:"palatino linotype"; panose-1:2 4 5 2 5 5 5 3 3 4; mso-font-charset:0; mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:-536870265 1073741843 0 0 415 0;} @font-face {font-family:"\@ms mincho"; panose-1:2 2 6 9 4 2 5 8 3 4; mso-font-charset:128; mso-generic-font-family:roman; mso-font-format:other; mso-font-pitch:fixed; mso-font-signature:1 134676480 16 0 131072 0;} /* style definitions */ p.msonormal, li.msonormal, div.msonormal {mso-style-unhide:no; mso-style-qformat:yes; mso-style-parent:""; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca;} p.msocommenttext, li.msocommenttext, div.msocommenttext {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"comment text char"; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca;} p.msoheader, li.msoheader, div.msoheader {mso-style-priority:99; mso-style-link:"header char"; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; tab-stops:center 3.15in right 6.3in; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca;} p.msofooter, li.msofooter, div.msofooter {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"footer char"; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; tab-stops:center 3.15in right 6.3in; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca;} span.msocommentreference {mso-style-noshow:yes; mso-style-priority:99; mso-ansi-font-size:8.0pt; mso-bidi-font-size:8.0pt;} p.msobodytext, li.msobodytext, div.msobodytext {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"body text char"; margin:0in; margin-bottom:.0001pt; mso-pagination:none; mso-layout-grid-align:none; text-autospace:none; font-size:12.0pt; font-family:"arial",sans-serif; mso-fareast-font-family:calibri; mso-bidi-font-family:latha;} a:link, span.msohyperlink {mso-style-noshow:yes; mso-style-priority:99; color:blue; text-decoration:underline; text-underline:single;} a:visited, span.msohyperlinkfollowed {mso-style-noshow:yes; mso-style-priority:99; color:purple; mso-themecolor:followedhyperlink; text-decoration:underline; text-underline:single;} p.msocommentsubject, li.msocommentsubject, div.msocommentsubject {mso-style-noshow:yes; mso-style-priority:99; mso-style-parent:"comment text"; mso-style-link:"comment subject char"; mso-style-next:"comment text"; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca; font-weight:bold;} p.msoacetate, li.msoacetate, div.msoacetate {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"balloon text char"; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:8.0pt; font-family:"tahoma",sans-serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca;} p.msobibliography, li.msobibliography, div.msobibliography {mso-style-noshow:yes; mso-style-priority:37; mso-style-next:normal; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca;} p.msonormal0, li.msonormal0, div.msonormal0 {mso-style-name:msonormal; mso-style-unhide:no; mso-margin-top-alt:auto; margin-right:0in; mso-margin-bottom-alt:auto; margin-left:0in; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast;} span.commenttextchar {mso-style-name:"comment text char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"comment text"; mso-ansi-font-size:10.0pt; mso-bidi-font-size:10.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; mso-ansi-language:en-ca;} span.headerchar {mso-style-name:"header char"; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:header; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; mso-ansi-language:en-ca;} span.footerchar {mso-style-name:"footer char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:footer; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; mso-ansi-language:en-ca;} span.bodytextchar {mso-style-name:"body text char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"body text"; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"arial",sans-serif; mso-ascii-font-family:arial; mso-fareast-font-family:calibri; mso-hansi-font-family:arial; mso-bidi-font-family:latha; mso-ansi-language:en-us;} span.commentsubjectchar {mso-style-name:"comment subject char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-parent:"comment text char"; mso-style-link:"comment subject"; mso-ansi-font-size:10.0pt; mso-bidi-font-size:10.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; mso-ansi-language:en-ca; font-weight:bold;} span.balloontextchar {mso-style-name:"balloon text char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"balloon text"; mso-ansi-font-size:8.0pt; mso-bidi-font-size:8.0pt; font-family:"tahoma",sans-serif; mso-ascii-font-family:tahoma; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:tahoma; mso-bidi-font-family:tahoma; mso-ansi-language:en-ca;} p.default, li.default, div.default {mso-style-name:default; mso-style-unhide:no; mso-style-parent:""; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; mso-layout-grid-align:none; text-autospace:none; font-size:12.0pt; font-family:"palatino linotype",serif; mso-fareast-font-family:"ms mincho"; mso-bidi-font-family:"palatino linotype"; color:black; mso-ansi-language:en-ca; mso-fareast-language:ja;} span.normalchar1 {mso-style-name:normal__char1; mso-style-priority:99; mso-style-unhide:no; mso-style-parent:""; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman";} span.shorttext {mso-style-name:short_text; mso-style-priority:99; mso-style-unhide:no; mso-style-parent:""; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman";} span.heading2main {mso-style-name:"heading 2 main"; mso-style-unhide:no; mso-style-parent:""; mso-ansi-font-size:12.0pt; font-family:"arial",sans-serif; mso-ascii-font-family:arial; mso-fareast-font-family:arial; mso-hansi-font-family:arial; mso-bidi-font-family:arial; color:black; font-weight:bold; mso-bidi-font-weight:normal; font-style:normal; text-underline:black;} span.mention1 {mso-style-name:mention1; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; color:#2b579a; background:#e6e6e6;} span.unresolvedmention {mso-style-name:"unresolved mention"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; color:#605e5c; background:#e1dfdd;} .msochpdefault {mso-style-type:export-only; mso-default-props:yes; font-size:10.0pt; mso-ansi-font-size:10.0pt; mso-bidi-font-size:10.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:calibri; mso-fareast-theme-font:minor-latin; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; mso-ansi-language:de;} /* page definitions */ @page {mso-footnote-separator:url("es_scott_29531_files/header.htm") fs; mso-footnote-continuation-separator:url("es_scott_29531_files/header.htm") fcs; mso-endnote-separator:url("es_scott_29531_files/header.htm") es; mso-endnote-continuation-separator:url("es_scott_29531_files/header.htm") ecs;} @page wordsection1 {size:595.3pt 841.9pt; margin:70.85pt 70.85pt 56.7pt 70.85pt; mso-header-margin:35.4pt; mso-footer-margin:35.4pt; mso-columns:2 even 35.4pt; mso-header:url("es_scott_29531_files/header.htm") h1; mso-paper-source:0;} div.wordsection1 {page:wordsection1;} --> </style> <!--[if gte mso 10]> <style> /* style definitions */ table.msonormaltable {mso-style-name:"table normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; mso-ansi-language:de;} </style> <![endif]--><!--[if gte mso 9]><xml> <o:shapedefaults v:ext="edit" spidmax="1026"/> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext="edit"> <o:idmap v:ext="edit" data="1"/> </o:shapelayout></xml><![endif]--> </head> <body lang=en-us link=blue vlink=purple style='tab-interval:35.4pt'> <div class=wordsection1> <p class=msonormal><i style='mso-bidi-font-style:normal'><span lang=en-ca style='font-family:"palatino linotype",serif'>evidence summary<o:p></o:p></span></i></p> <p class=msonormal><i style='mso-bidi-font-style:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></i></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:11.0pt;font-family:"palatino linotype",serif'>much library and information science research on open access is available in open access, but there is still room to grow<o:p></o:p></span></b></p> <p class=msonormal><span lang=en-ca style='font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>a review of:<o:p></o:p></span></b></p> <p class=msonormal style='margin-left:35.3pt;text-indent:-35.3pt'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>chilimo, w. l., &amp; onyancha, o. b. (2018). how open is open access research in library and information science? <i style='mso-bidi-font-style:normal'>south african journal of libraries &amp; information science, 84</i>(1), 11-19. <span class=msohyperlink><a href="https://doi.org/10.7553/84-1-1710">https://doi.org/10.7553/84-1-1710</a></span> <o:p></o:p></span></p> <p class=msonormal style='margin-left:35.3pt;text-indent:-35.3pt'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>reviewed by:<o:p></o:p></span></b></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>rachel elizabeth scott<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>interim coordinator, cataloging, collection management, and library information systems &amp; integrated library systems librarian<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>university libraries<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>university of memphis<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>memphis, tennessee, united states of america<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>email: <span class=msohyperlink><a href="mailto:rescott3@memphis.edu">rescott3@memphis.edu</a></span><o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>received: </span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>20 nov. 2018</span><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><span style='mso-tab-count:5'>                                                                </span>accepted: </span></b><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>7 jan. 2019<b style='mso-bidi-font-weight:normal'><o:p></o:p></b></span></p> <p class=msonormal><span lang=en-ca style='font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:9.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:arial;mso-ansi-language:en-us;mso-no-proof:yes'><!--[if gte vml 1]><v:shapetype id="_x0000_t75" coordsize="21600,21600" o:spt="75" o:preferrelative="t" path="m@4@5l@4@11@9@11@9@5xe" filled="f" stroked="f"> <v:stroke joinstyle="miter"/> <v:formulas> <v:f eqn="if linedrawn pixellinewidth 0"/> <v:f eqn="sum @0 1 0"/> <v:f eqn="sum 0 0 @1"/> <v:f eqn="prod @2 1 2"/> <v:f eqn="prod @3 21600 pixelwidth"/> <v:f eqn="prod @3 21600 pixelheight"/> <v:f eqn="sum @0 0 1"/> <v:f eqn="prod @6 1 2"/> <v:f eqn="prod @7 21600 pixelwidth"/> <v:f eqn="sum @8 21600 0"/> <v:f eqn="prod @7 21600 pixelheight"/> <v:f eqn="sum @10 21600 0"/> </v:formulas> <v:path o:extrusionok="f" gradientshapeok="t" o:connecttype="rect"/> <o:lock v:ext="edit" aspectratio="t"/> </v:shapetype><v:shape id="picture_x0020_4" o:spid="_x0000_i1025" type="#_x0000_t75" alt="cc-ca_logo_xl" style='width:7.5pt;height:7.5pt;visibility:visible; mso-wrap-style:square'> <v:imagedata src="es_scott_29531_files/image003.png" o:title="cc-ca_logo_xl" cropright="49941f"/> </v:shape><![endif]--><![if !vml]><img border=0 width=10 height=10 src="es_scott_29531_files/image002.png" alt="cc-ca_logo_xl" v:shapes="picture_x0020_4"><![endif]></span><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:arial'><span style='mso-spacerun:yes'> </span>2019 </span><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif'>scott. this is an open access article distributed under the terms of the creative commons</span><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"cambria math"'>‐</span><span lang=en-ca style='font-size: 9.0pt;font-family:"palatino linotype",serif'>attribution</span><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"cambria math"'>‐</span><span lang=en-ca style='font-size: 9.0pt;font-family:"palatino linotype",serif'>noncommercial</span><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"cambria math"'>‐</span><span lang=en-ca style='font-size: 9.0pt;font-family:"palatino linotype",serif'>share alike license 4.0 international (<span class=msohyperlink><a href="http://creativecommons.org/licenses/by-nc-sa/4.0/">http://creativecommons.org/licenses/by-nc-sa/4.0/</a></span>), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.<o:p></o:p></span></p> <div style='mso-element:para-border-div;border:none;border-bottom:solid black 1.0pt; mso-border-bottom-alt:solid black .25pt;padding:0in 0in 1.0pt 0in'> <p class=msobodytext style='border:none;mso-border-bottom-alt:solid black .25pt; padding:0in;mso-padding-alt:0in 0in 1.0pt 0in'><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-ca'><o:p>&nbsp;</o:p></span></p> <p class=msobodytext style='border:none;mso-border-bottom-alt:solid black .25pt; padding:0in;mso-padding-alt:0in 0in 1.0pt 0in'><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-ca'><o:p>&nbsp;</o:p></span></p> <p class=msobodytext style='border:none;mso-border-bottom-alt:solid black .25pt; padding:0in;mso-padding-alt:0in 0in 1.0pt 0in'><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif;color:blue; mso-ansi-language:en-ca'>doi</span><span lang=en-ca style='font-size:9.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-ca'>: 10.18438/eblip29531<o:p></o:p></span></p> <p class=msobodytext style='border:none;mso-border-bottom-alt:solid black .25pt; padding:0in;mso-padding-alt:0in 0in 1.0pt 0in'><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-ca'><o:p>&nbsp;</o:p></span></p> </div> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>abstract <o:p></o:p></span></b></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>objective</span></b><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'> – to investigate the open access (oa) availability of library and information science (lis) research on the topic of oa, the relative openness of the journals in which this research is published, and the degree to which the oa policies of lis journals facilitate free access.<b style='mso-bidi-font-weight: normal'><o:p></o:p></b></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>design</span></b><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'> – bibliometric, quantitative dataset analysis.<o:p></o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>setting</span></b><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'> – african academic library and information science department.<o:p></o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>subjects</span></b><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'> – 1,185 english-language, peer-reviewed articles published between 2003 and 2013 on oa and published in journals indexed by three major lis databases, of which 909 articles in the top 56 journals received further analysis.<o:p></o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>methods</span></b><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'> – authors first searched lis indexes to compile a dataset of published articles focusing on oa. they then manually identified and evaluated the oa policies of the top 56 journals in which these articles were found. the openness of these journals was scored according to a rubric modified from the scholarly publishing and academic resources coalition’s (sparc’s) 2013 oa spectrum. finally, authors manually searched google scholar to determine the oa availability of the articles from the dataset.<o:p></o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>main results</span></b><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'> – of the 909 articles published in the top 56 journals, 602 were available in some form of oa. of these, 431 were available as gold copies and 171 were available as green copies. of the 56 journals evaluated for openness, 13 were considered oa, 3 delayed oa, 27 hybrid/unconditional post-print, 2 hybrid/conditional post-print, and 11 had unrecognized oa policies.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>conclusion</span></b><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'> – the increasing amount and significance of lis research on oa has not directly translated to the comprehensive adoption of oa publishing. although a majority of the articles in the dataset were available in oa, the authors indicate that some measures of oa adoption and growth assessed in this study are only somewhat higher than in other disciplines. the authors call upon lis professionals to become more conversant with journals’ oa policies. an acknowledgement that not all lis scholars researching oa are necessarily advocates thereof led the authors of this study to recommend further investigation of oa research not available in oa to shed light on those scholars’ perceptions and preferences.<b style='mso-bidi-font-weight:normal'><o:p></o:p></b></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>commentary<o:p></o:p></span></b></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>the study at hand builds on existing oa analysis of lis publications such as vandegrift &amp; bowley (2014) and grandbois &amp; beheshti (2014). it is unique in its analysis of lis articles on the topic of oa, the quantification of their oa availability, and the relative openness of the journals in which most are published.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>throughout this commentary, perryman’s tool for bibliometric studies was used to evaluate the rigor of the research (2009). this tool was selected for its focus on the systematic construction of bibliometric studies.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>there is no discrete literature review, however the authors make use of current and relevant published literature to support their objectives and methodology, and to delineate the gap that their research will address. the authors cite bibliometric studies to discuss the limitations of their data sources and to provide a rationale for the indexes they selected and excluded.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>they compiled their dataset by hand by searching the following databases: ebsco library &amp; information science source, ebsco library, information science and technology abstracts, and proquest library and information science abstracts. the methodology includes imprecise search procedures. the authors used the “advanced search” to conduct a subject keyword search, but the criteria in place beyond the initial search are ambiguous. in order to be included in the dataset, articles must be peer reviewed, english-language, published between 2003 and 2013, and discuss open access. these inclusion criteria are relevant to the research question, but the authors mentioned expanding the search to keywords such as “institutional repository,” which have potentially little to do with oa. the authors claim to have conducted a “thorough check” (p. 12) to remove irrelevant and duplicated records, the details of which are not defined.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>overall, the evaluation methods were appropriate to the objectives, but there are some discrepancies in the data. table 2 is labeled “top 50 journals…” where the text indicates “table 2 shows the fifty-six journals” (p. 15). the list of “journals with unrecognized oa policies” included several major titles, such as <i style='mso-bidi-font-style:normal'>american archivist</i> and <i style='mso-bidi-font-style:normal'>portal</i>. although it is difficult to confirm that all policies were not readily available when the authors searched in june 2015, a december 28, 2014 version of the <i style='mso-bidi-font-style: normal'>american archivist</i>’s website indicates that its journal content “is licensed under a creative commons attribution non-commercial 3.0 united states license” (society of american archivists, 2014). that so many processes were executed manually and individually renders replication both challenging and time consuming without defined supporting data. what may simply be a lack of clarity also casts a shadow on data collection and analysis, both of which would benefit from additional supporting data.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><span style='mso-spacerun:yes'> </span><o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>the study achieves its original objective of measuring the degree of openness of oa scholarship in lis. like prior research, the present study shows that both self-archiving and journal-based oa are not yet widely established practices. this article’s contribution is in showing that despite increased lis literature and advocacy on the topic of oa, lis scholars and journals have plenty of room to grow in their adoption of oa. although the data is not sufficiently strong to serve as a benchmark for future measurement, the article in its current form is a strong piece of advocacy.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>references<o:p></o:p></span></b></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:35.3pt;text-indent:-35.3pt'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>grandbois, j., and beheshti, j. (2014). a bibliometric study of scholarly articles published by library and information science authors about open access. <i style='mso-bidi-font-style:normal'>information research: an international electronic journal, 19</i>(4). <span class=msohyperlink><a href="http://www.informationr.net/ir/19-4/paper648.html#.xif69yhkhpy">http://www.informationr.net/ir/19-4/paper648.html#.xif69yhkhpy</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:35.3pt;text-indent:-35.3pt'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:35.3pt;text-indent:-35.3pt'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>perryman, c. (2009). <i style='mso-bidi-font-style:normal'>evaluation tool for bibliometric studies</i>. retrieved from <span class=msohyperlink><a href="https://www.dropbox.com/l/scl/aaal7luzple90fxfnbv5hcnoz0ctlh6rqrs">https://www.dropbox.com/l/scl/aaal7luzple90fxfnbv5hcnoz0ctlh6rqrs</a><o:p></o:p></span></span></p> <p class=msonormal style='margin-left:35.3pt;text-indent:-35.3pt'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:35.3pt;text-indent:-35.3pt'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>society of american archivists. (2014, december 28). <i style='mso-bidi-font-style: normal'>american archivist editorial policy</i>. retrieved from <span class=msohyperlink><a href="https://web.archive.org/web/20141228053114/http:/www2.archivists.org:80/american-archivist/editorialpolicy">https://web.archive.org/web/20141228053114/http://www2.archivists.org:80/american-archivist/editorialpolicy</a></span> <o:p></o:p></span></p> <p class=msonormal style='margin-left:35.3pt;text-indent:-35.3pt'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:35.3pt;text-indent:-35.3pt'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>vandegrift, m., &amp; bowley, c. (2014 april 23). librarian, heal thyself: a scholarly communication analysis of lis journals. <i style='mso-bidi-font-style:normal'>in the library with the lead pipe</i>, 2-18. <span class=msohyperlink><a href="http://www.inthelibrarywiththeleadpipe.org/2014/healthyself/">http://www.inthelibrarywiththeleadpipe.org/2014/healthyself/</a></span><o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> </div> </body> </html> news   eblip panel presentation at 2018 american library association annual conference      2018. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29425     four professionals engaged in eblip in school, public, and academic library sectors will present “evidence based library and information practice is . . . “as part of a panel at the 2018 ala annual conference to be held in new orleans, louisiana, united states.   this presentation will introduce the basic techniques of evidence based library and information practice (eblip) and also offer examples from libraries and librarians that have already embraced the practice. the panel of librarians, who have adopted eblip concepts and principles, will share how evidence-based librarianship has impacted their libraries, library practice, research agendas, and patron interactions. participants will receive an overview of evidence based library and information practice theory and research; learn about evidence-based frameworks; and discover how evidence-based decision making improves libraries. the presentation will draw heavily from recent research by koufogiannakis & brettle (2016) and their book being evidence based in library and information practice.   participants will learn what evidence-based librarianship is, what it looks like, and suggestions on how to begin incorporating eblip into their own day-to-day practice.   date and time: monday 25 june 2018, 9 a.m.-10 a.m.   moderator/panelist: trish chatterley, collections strategies coordinator, university of alberta   panelists: lee ann fullington, health & environmental sciences librarian, brooklyn college; hannah byrd little, director of library and archives, webb school; pam ryan, director, service development & innovation, toronto public library   location: consult the conference schedule when available.   evidence based library and information practice commentary   gathering evidence for routine decision-making   alison brettle professor of health information and evidence based practice school of health and society university of salford salford, manchester, united kingdom email: a.brettle@salford.ac.uk   received: 17 nov. 2017 accepted: 23 nov. 2017      2017 brettle. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     this paper is based on the opening keynote address at the 9th international evidence based library and information practice conference, philadelphia, 18-21 june 2017.   introduction   discussions about evidence based library and information practice (eblip) often focus on the use of research evidence in decision making. however, eblip can be an approach to professional practice that is about being evidence based, rather than just a one-off event or a restriction to decision-making alone. this involves:   ·         questioning our practice ·         gathering or creating the evidence through research and evaluation ·         using information or evidence wisely to: make decisions about our practice; improve our practice; make decisions about our services; help others make decisions about our services (by demonstrating our effectiveness, impact, value, or worth); and using our professional skills to help others make their own evidence-based decisions (koufogiannakis & brettle, 2016).   using examples from the united kingdom (uk), this paper examines the wider range of evidence that librarians can gather or create to make decisions about their practice and services. these examples also demonstrate how librarians can use this evidence in terms of advocacy, to help others make decisions about their services. in this paper eblip is considered holistically; research evidence, local evidence, and  professional knowledge are all taken into account (koufogiannakis, 2011). a wide range of different types of evidence may also be used (table 1).     table 1 different types of research evidencea research local professional quantitative statistics professional expertise qualitative assessment/evaluation tacit knowledge mixed documents input from colleagues secondary librarian observation what other libraries do user feedback non-research literature anecdotal evidence organizational realities aadapted from (koufogiannakis & brettle, 2016).     gathering research evidence   the chartered institute of library and information professionals (cilip) is keen to support its members in advocacy. high quality research evidence of the value of library and information professionals is therefore needed.  to this end cilip commissioned a systematic scoping review of evidence that collated evidence on the value and impact of professionally trained library, information and knowledge workers (brettle & maden, 2016). this evidence is summarised below and can be used by the professional body to advocate on behalf of its members, and by library and information professionals themselves to demonstrate value to their stakeholders.   when trying to demonstrate impact or value, outcomes or outcome measures are often used. outcomes are “the consequences of deploying services on the people who encounter them or the communities served” (markless & streatfield, 2006). however, for libraries these outcomes or consequences are difficult to capture, because they may be quite intangible or the library may only make a contribution to an outcome rather than a whole consequence. according to oakleaf “libraries need to define outcomes relevant to their institution and assess the extent to which they are met”. this is easier said than done, but it was the approach taken within this review.   in brief, the review used a comprehensive search to locate research evidence on the value of any type of library, information, or knowledge worker. only studies that provided evidence of librarians contributing to clear outcomes were included. evidence was found for the following four sectors: health, academic, public, and school. each sector favoured particular types of study designs; this included return on investment studies (public libraries), correlational designs (school and academic libraries), critical incident technique (school and health), surveys (school and health), and mixed methods, quasi experiments, and randomised controlled trials (rcts) (academic and health). although some designs are suited to particular sectors, such as the return on investment (roi) for public libraries, all sectors have much to learn from each other. for example, academic libraries could make better use of more rigorous designs such as rcts to evaluate information literacy, and other methods could be used alongside correlational designs to strengthen the evidence found.   the review concluded that library and information professionals contribute to a wide range of outcomes in their sectors. these contributions are summarised in table 2.     table 2 contributions of librariansb health librarians contribute to… academic librarians contribute to… public librarians contribute to… school librarians contribute to… improving the quality of patient care better research, researchers, and research achievement helping people to feel a sense of belonging in their community improving student achievement improving clinical decision-making better grades or degrees improving attitudes to reading improving reading skills improving patient centred care a good return on investment for the university a good return on investment facilitating student learning aiding risk management and safety improved retention helping people improve education and employment prospects positive pupil engagement helping to demonstrate efficiency and cost effectiveness   helping people improve their health   health service development and delivery       assisting health professionals to pursue continuing professional development       b(brettle & maden, 2016)     creating the evidence   one of the recommendations from the above scoping review (brettle & maden, 2016) recommended that health libraries should improve standards for reporting impact studies. within the uk, the knowledge for health quality and impact group have established a project across all english hospital library services that seeks to do this. all libraries working within the english national health service (nhs) are part of the knowledge for healthcare framework, which sets service standards and monitors them regularly using an nhs library quality assurance framework (lqaf) (http://www.libraryservices.nhs.uk/forlibrarystaff/lqaf/lqaf.html).   in relation to demonstrating impact, the framework requires “evidence that a variety of methods have been used to systematically gather information about the impact of library services and that the information has been used to demonstrate the impact of services”. libraries use a wide range of methods to do this, and guidance has been developed to help them provide high quality evidence (weightman et al., 2009). a survey showed that this guidance is not widely used and that most libraries develop their own questionnaires.  this means that there is little rigour within each questionnaire, and that an opportunity has been missed to compile results across the whole english hospital library service using the same tools. to address these issues a toolkit has been developed that provides access to guidance on measuring impact, as well as a suite of simple, generic tools that librarians can routinely use to measure impact and disseminate evidence about their services (http://kfh.libraryservices.nhs.uk/value-and-impact-toolkit/). these tools use an outcomes approach to collecting evidence.   a pilot of one of the tools (a simple generic questionnaire) provided evidence of impact that could be used by a range of stakeholders. for example, responses to one question provide evidence of how the library is being used (what services), which is likely to be of use to library managers. the highest uses of the library were literature search services, study space, article or book supply, and training. in contrast, use of current awareness services was low. this evidence can help a manager decide where best to direct resources within the service. in relation to how the information from the library was used, the pilot showed that information from the library is being used for direct patient care (40%), to provide help to patients and families (27%), for organizational development (15%), and for legal and ethical questions (9%). this shows that the library clearly contributes in a wide number of ways to its parent organization.  this type of evidence could be crucial to keeping the library open in times of financial constraint and budgetary cuts.   an interview template is also provided as part of the toolkit, to enable libraries to collect evidence of more detailed outcomes and to explain how some of the contributions are really made by libraries. this evidence can be disseminated using a case study template, and case studies are being collated at a national level (http://kfh.libraryservices.nhs.uk/value-and-impact-toolkit/kfh-impact-tools/impact-case-studies/). these can be used in a range of ways to demonstrate the value and impact of libraries.   evidence for advocacy   the case studies described above are being used as part of a high level social media campaign to demonstrate the value and impact of health librarians. the campaign is called #amilliondecisions and it uses twitter to promote the evidence provided by health librarians to support healthcare decision-making. one example highlighted how evidence from health librarians contributed to a change in practice that reduced “do not attends” by 2% at clinics and reduced clinic waiting times by two weeks (http://kfh.libraryservices.nhs.uk/wp-content/uploads/2016/08/dna.jpg).   at the university of salford, uk, staff are currently taking part in a project to improve skills in analyzing data from social media. using tableau software, staff tracked the #amilliondecisions to provide evidence of who tweeted the most, what tweets had the most impact, as well as the overall activity of the hashtag. figure 1 clearly shows peaks and troughs in activity, including when all tweets  had to be stopped due to the uk general election campaign. this is a simple means of collecting evidence about a campaign that can be used by those running the campaign to see its value and where best to target their resources.     figure 1 evidence on the value of social media campaign     conclusion   librarians make a wide range of contributions to the organisations they serve, but it is often difficult to articulate these and demonstrate their impact and value. using evidence about the outcomes to which libraries or librarians contribute is one way forward. this paper highlights the different types of evidence that librarians can gather or use to demonstrate their impact or value; this may be research evidence or evidence that has been generated locally through evaluation. within the u.k. health library sector a number of initiatives are taking place to help libraries collect impact data that can be used on a local or national level to demonstrate impact to a wide range of stakeholders. by doing this, u.k. health libraries are becoming evidence based. although these examples are uk based and within the health sector, this approach can be easily adapted by libraries within other sectors.   references   brettle, a., & maden, m. (2016). what evidence is there to support the employment of trained and professionally registered library, information and knowledge workers? a systematic scoping review of the evidence. london: cilip. https://archive.cilip.org.uk/sites/default/files/media/document/2017/value_of_trained_lik_workers_final_211215_0.pdf   koufogiannakis, d., & brettle, a. (eds.). (2016). being evidence based in library and information practice. london: facet publishing.   koufogiannakis, d. (2011). considering the place of practice-based evidence within evidence based library and information practice (eblip). library and information research, 35(111), 41-58. http://www.lirgjournal.org.uk/lir/ojs/index.php/lir/article/view/486/527   markless, s., & streatfield, d. (2006). evaluating the impact of your library. london: facet publishing.   oakleaf, m. (2010). the value of academic libraries: a comprehensive research review and report (0004-8623). chicago, il: association of college and research libraries. http://www.ala.org/acrl/sites/ala.org.acrl/files/content/issues/value/val_report.pdf   weightman, a., urquhart, c., spink, s., thomas, r., and on behalf of the national library for health library services development group. (2009). the value and impact of information provided through library services for patient care: developing guidance for best practice. health information & libraries journal, 26(1), 63-71. http://doi.org/10.1111/j.1471-1842.2008.00782.x research article   testing a warmth-based instruction intervention for reducing library anxiety in first-year undergraduate students   cecelia parks research and instruction librarian and assistant professor university of mississippi libraries oxford, mississippi, united states of america email: cparks@olemiss.edu   received: 11 jan. 2019                                                                    accepted: 26 apr. 2019      2019 parks. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29548     abstract   objective – this study aimed to test the efficacy of a warmth-based library instruction intervention in reducing rates of library anxiety in first-year undergraduate students. "warmth" is a concept that is commonly discussed within literature on library anxiety, but to date no studies have explicitly tested the application of a warmth-based instruction intervention. first-year students are ideal targets for this intervention because they are the most likely to experience library anxiety.   methods – a quasi-experiment was conducted examining library anxiety rates in first-year undergraduate students at a public research university in the u.s. south. a one-shot warmth-based instruction session focusing on the emotional dimensions of library use was compared to a standard one-shot instruction session. library anxiety was measured using a modified version of bostick's library anxiety scale as a pretest and posttest.   results – results indicated that both warmth-based and standard library instruction were associated with a decrease in participants' library anxiety rates without significant differences between the types of instruction. however, warmth-based instruction was correlated with greater reductions in areas of library anxiety related to interactions with library workers. though library anxiety rates decreased significantly after experiencing library instruction, participants exhibited low levels of library anxiety before their library instruction session occurred.   conclusion – though warmth-based instruction did not have a significantly different impact than standard library instruction on general library anxiety, the intervention tested in this study suggests strategies that could be used to increase student comfort with library workers. this study also demonstrates a successful method to include emotional factors such as library anxiety in academic libraries' regular assessment programs. focusing assessment on students' skills and knowledge alone risks ignoring an important aspect of student engagement and missing opportunities for academic libraries to connect with students. assessment of emotional components of library instruction initiatives is especially crucial to ensure and demonstrate that libraries are using their resources effectively to maximize student success.     introduction   library anxiety is a demonstrated phenomenon that presents real barriers to academic success in higher education (mellon, 1986). students with library anxiety feel nervous or uncomfortable engaging with library resources, including simple acts such as entering the library building, and are therefore less likely to do so. this lack of engagement can be detrimental to student success, as the use of library resources has been tied to increased academic success and retention rates (gaha, hinnefeld, & pellegrino, 2018; soria, fransen, & nackerud, 2017). the negative effects of library anxiety may be particularly acute for students early in their academic careers, for example first-years and sophomores have the highest reported levels of library anxiety (jiao, onwuegbuzie, & lichtenstein, 1996; mech & brooks, 1997).   studies show that interventions by librarians such as library instruction sessions can decrease library anxiety. the literature also demonstrates that it is critical to target interventions towards the most vulnerable students, like first-years, in order to maximize the impact of the intervention (brown, weigart, johnson, & dance, 2004; chiman, nwajei, & akpom, 2015; muszkiewicz, 2017; onwuegbuzie, jiao, & bostick, 2004; van scoyoc, 2003). this study further tests these findings by conducting a quasi-experiment comparing two different types of library instruction, with the purpose of determining whether warmth-based instruction is more effective than standard library instruction, for lowering rates of library anxiety among first-year students at a public research university in the south.   assessment of library instruction is particularly important for institutions that devote considerable resources to library instruction for first-year students, such as the university under consideration in this study. assessment of information literacy skills is common and is often used to justify large first-year instruction initiatives, but assessment of first-year students’ emotional responses to the library is equally important, yet often neglected (cook, 2014; gilbert, 2009; lowe, booth, stone & tagge, 2015; luetkenhaus, hvizdak, johnson & schiller, 2017; shao & purpur, 2016). ideally, library instruction for first-year students provides not only a base of information literacy skills to build on, but also an understanding of the library as a resource that they can return to for help throughout their academic careers. as librarians cannot work with every single class or individual student, it becomes the students’ responsibility to seek out help when needed; feelings of library anxiety make it less likely that they will do so. therefore, understanding the emotional impact of library instruction for first-year students is a necessary component of assessment, to ensure that library instruction resources are being used effectively.   literature review   constance mellon coined the term “library anxiety” in her 1986 article titled “library anxiety: a grounded theory and its development,” naming what many academic librarians had noticed for years in their interactions with students. mellon found that students commonly expressed feelings of fear and anxiety about using the library in general, rather than frustration with specific aspects of library use or deficits in certain skills. this anxiety often stemmed from a sense of perceived inadequacy, or feelings that everyone but them already possessed the skills and knowledge necessary to use the library to conduct research, and that library workers would judge them for asking questions. mellon (1986, p. 163) noted that students became so anxious that they were “unable to approach the problem logically or effectively,” preventing the students from seeking help. sharon bostick (1992) developed the library anxiety scale to measure library anxiety in college students. although other scales have been created, such as the aqak (anwar, al-qallaf, al-kandari, & al-ansari, 2011) and an information anxiety scale (blundell & lambert, 2014), bostick’s library anxiety scale remains the basis for most assessments of library anxiety today.   many studies on this subject have focused on understanding factors that contribute to students’ experiences of library anxiety, such as gender, past academic experiences, race, and socioeconomic status, though no consensus has emerged around any predictor. jiao & onwuegbuzie (1999) studied library anxiety in graduate students and found that students with lower self-perceptions of competence exhibited the highest rates of library anxiety, while jiao, onwuegbuzie, & lichtenstein (1996) demonstrated that firstand second-year undergraduates, men, and those who infrequently used the library had higher rates of library anxiety (though bostick’s 1992 dissertation showed no differences in library anxiety between genders and blundell and lambert’s 2014 study indicated that female students experience higher rates of library anxiety). sinnasamy and abdul karim (2015) demonstrated that communication anxiety and english language anxiety influence library anxiety. blundell and lambert (2014) examined the relationship between information anxiety, race, gender, and prior library experience, whereas wildemuth (2017) noted that male students who spoke english as a second language experienced the highest rates of library anxiety.   other studies have concentrated on how to mitigate the effects of library anxiety, regardless of who experiences the highest levels of such anxiety. mellon (1986) focused on the importance of “warmth” in interactions between library workers and students as a way to decrease library anxiety. fiske, cuddy, and glick (2006, p. 77) noted that warmth, or traits related to perceived intent, such as “friendliness, helpfulness, sincerity, trustworthiness, and morality”, is the first dimension of personality on which humans judge one another. this finding suggests that warmth is crucial for interactions with first-year students, particularly because the interaction may be the student's first experience with an academic library. other researchers have tested various methods of incorporating warmth into library services and interactions (though the impact of librarian personality on students' library anxiety has not been tested) and one of the most common venues for intervention is new student orientation, due to the high percentage of incoming students that can be reached through orientation events. orientation activities described in the literature include traditional information sessions (brown, weingart, johnson, & dance, 2006), distribution of “library survival guides” (diprince, wilson, karafit, bryant, & springer, 2016), and fun, casual, non-skills focused activities (muszkiewicz, 2017). each of these studies demonstrated decreased levels of students’ library anxiety after participating in the orientation activity.   library instruction has also been commonly used to decrease library anxiety. mellon (1986) pointed to library instruction as a way to project warmth and minimize library anxiety and that recommendation has been echoed by others (chimah, nwajei, & akpom, 2015; collins, mellon, & young, 1987; naveed, 2017; wildemuth, 2017). studies such as those by van scoyoc (2003) and fleming-may, mays, and radom (2015) demonstrated that library instruction was correlated with decreased library anxiety, though van scoyoc focused on one-shot skills-based instruction sessions while fleming-may, mays, and radom explored the impact of a three workshop series for at-risk students. similarly, kracker (2002) found that presentations on kuhlthau’s information search process model (kuhlthau, 1993) correlated with a decrease in research anxiety among their participants. these studies demonstrate that library instruction can have an impact on students’ library anxiety, but no study has specifically explored the implementation of warmth in instruction sessions, even though the importance of warmth is widely recognized.   first-year students are often the targets of library instruction initiatives. jiao, onwuegbuzie, and lichtenstein (1996) and mech and brooks (1997) demonstrated that first-year students exhibited the highest rates of library anxiety, and students new to higher education naturally lack some of the skills and knowledge necessary to utilize academic libraries. library instruction has been shown to increase first-year students’ information literacy skills, whether the instruction takes the form of a one-shot session (gilbert, 2009; luetkenhaus, hvizdak, johnson & schiller, 2017), a for-credit course (cook, 2014), or general interactions with librarians (lowe, booth, stone & tagge, 2015). van scoyoc (2003) found that instruction reduced first-year students’ library anxiety rates, but that study is over fifteen years old (as are other important library anxiety studies), and first-year students have changed significantly as today’s students have come of age regularly using new and powerful information tools. these changes do not eliminate first-year students’ library anxiety and information literacy challenges, but they indicate that libraries’ approaches to addressing these issues need to be updated.   aims   the aims of this study are to:   test the efficacy of explicitly warmth-based instruction on reducing first-year students’ library anxiety update previous studies such as van scoyoc’s (2003) that examine the impact of library instruction sessions on first-year students’ library anxiety   methods   this study used a quasi-experimental method to compare the effects of standard library instruction (the control) to warmth-based library instruction (the intervention) on library anxiety in first-year students as measured by pretest and posttest results. as in a fully randomized experiment, a researcher using quasi-experimental methods "manipulate[s] presumed causes to discover their effects, but the researcher does not assign units to conditions randomly. quasi-experiments are necessary because it is not always possible to randomize" (lewis-beck, bryman, & futing liao, 2004, p. 888). quasi-experiments are often used in educational settings because these settings often do not allow for true randomization. the quasi-experimental method was ideal for this study because it allowed for an intervention to be tested on a population that could not be randomized, due to the practical infeasibility of breaking up pre-determined classes to create truly randomized testing groups.   a modified version of bostick’s (1992) library anxiety scale was developed for use as the pretest and posttest. modifications were necessary to shorten the original 43-question scale to one that could be administered in a shorter time period and to update some of the more dated items. bostick's (1992) modified library anxiety scale was more suitable to the purposes of this study than newer adaptations, such as the aqak (anwar, al-qallaf, al-kandari, & al-ansari, 2011), which were too specific to a particular time and place to be useful. a 24-question version of the modified library anxiety scale was piloted with a group of approximately 60 first-year students, and factor analysis was used to eliminate an additional 6 items for a total of 18 items on the final scale. see appendix a for the full instrument and appendix b for the factor analysis table.   sections of the “edhe 105: freshman year experience course”, which is intended to help first-time, first-year students adjust to college, were targeted for this study due to their consistent first-year student population and history of cooperation with the library. new transfer students were not included in this study. eight instructors teaching a total of 12 sections and 262 enrolled students were recruited to participate in the study. sections were assigned to the control (standard instruction) or intervention (warmth-based instruction) group. if an instructor taught more than one section, half of their sections were assigned to the control group and half were assigned to the intervention group to minimize instructor influence on the results. all participating sections were pre-scheduled for a library instruction session and all sessions were taught by the same librarian instructor. pretests were administered by the “edhe 105” instructor approximately one week before each section’s library instruction session, roughly early september through early october 2018. each student participant was randomly assigned a unique identifier, which was destroyed before analysis, to allow direct comparison between the pretests and posttests.   the “edhe 105” sections in the control group participated in a standard library instruction session, in which a librarian instructor introduced basic information literacy skills by guiding students through an online worksheet to learn about the library website, discovery service, and online catalog, culminating in finding a book in the stacks. no emphasis was placed on the emotional dimensions of using the library or any anxiety the students may have been experiencing. the “edhe 105” sections in the intervention group participated in a warmth-based library instruction session, which emphasized the emotional dimensions of library use rather than any specific skills, and was intended to make students feel comfortable working with library workers and using library resources. activities for the warmth-based session were drawn from the instructor's own experiences observing what makes students feel comfortable in the classroom and in the library. activities were also pulled from the literature, such as sharing information about library anxiety (kracker, 2002), incorporating elements of active learning that require students to leave the library classroom (brown, weingart, johnson, & dance, 2006; muszkiewicz, 2017), and mellon's (1986) and fiske, cuddy, and glick's (2006) discussions of what "warmth" looks like in personal interactions.   the warmth-based instruction session used the same online worksheet as the standard session, as seen in appendix c. it began with a think-pair-share activity about how the university library differs from students’ high school libraries; this was intended to connect students with one another and to help them think critically about what an academic library is, while acknowledging the anxiety that may arise from the transition to using one. this activity was followed by a brief discussion of what library anxiety is, accompanied by reassurance from the librarian instructor that if students are experiencing library anxiety, they are not alone and that library workers are there to help. the session then proceeded to cover similar information literacy skills content as the standard session, with the librarian instructor sharing their personal experiences with library anxiety throughout the remainder of the session to normalize library anxiety and to model ways to deal with it. posttests were administered approximately one week after each section’s library instruction session, roughly mid-september through mid-october 2018.   data analysis   after the pretests and posttests were collected and the data was entered, all incomplete responses and responses from participants who did not take both the pretest and posttest were removed. there was a total of 161 complete responses, 76 in the control group and 85 in the intervention group (as noted by onwuegbuzie, jiao, and bostick (2004, p. 101) a minimum of 64 participants per group is necessary for a quasi-experiment). one hundred and twelve participants self-identified as female and 49 self-identified as male.   after reverse-coding items 7, 8, 9, 11, 12, and 13 and calculating each participant’s mean and total scores on pretests and posttests, paired t-tests were conducted using r software to compare the following results:   overall pretest and posttest means item-level pretest and posttest scores overall pretest and posttest means broken out by intervention item-level pretest and posttest scores broken out by intervention   results   regardless of whether or not participants experienced the standard instruction (control) or warmth-based instruction (intervention), overall mean scores on the posttest were significantly lower than those on the pretest, with a medium-large effect size, indicating a reduction in library anxiety levels (table 1). at the item-level, item one (“i’m embarrassed that i don’t know how to use the library”) displayed a particularly significant change from pretest to posttest, as did item five (“the librarians don’t have time to help me because they’re always busy doing something else").     table 1 pretest and posttest mean scores for control and intervention groups combined a item pretest mean score posttest mean score t-value p-value r-value overall total scores 2.39 2.13 t(160) = 7.12 p < 0.05 0.49             item 1 scores 2.29 1.71 t(160) = 6.41 p < 0.05 0.45 item 3 scores 2.14 1.70 t(160) = 5.12 p < 0.05 0.38 item 5 scores 2.25 1.83 t(160) = 4.73 p < 0.05 0.41 item 8 scores 2.26 1.94 t(160) = 4.63 p < 0.05 0.34 item 9 scores 2.15 1.86 t(160) = 4.25 p < 0.05 0.32 item 11 scores 2.41 2.04 t(160) = 4.44 p < 0.05 0.33 item 15 scores 2.30 1.92 t(160) = 5.38 p < 0.05 0.37 item 18 scores 2.78 2.35 t(160) = 4.64 p < 0.05 0.34 a only items that displayed a medium or medium-large effect size were included in the table     table 2 pretest and posttest mean scores broken out by control and intervention a item pretest mean score posttest mean score t-value p-value r-value control total scores 2.38 2.09 t(75) = 4.80 p < 0.05 0.48 intervention total scores 2.39 2.16 t(84) = 5.37 p < 0.05 0.51             item 9 control scores 2.14 1.89 t(75) = 2.35 p > 0.05 0.26 item 9 intervention scores 2.15 1.82 t(84) = 3.69 p < 0.05 0.37 item 11 control scores 2.36 2.09 t(75) = 1.92 p > 0.05 0.22 item 11 intervention scores 2.25 1.84 t(84) = 4.73 p < 0.05 0.46 item 18 control scores 2.71 2.49 t(75) = 1.71 p > 0.05 0.19 item 18 intervention scores 2.84 2.22 t(84) = 4.79 p < 0.05 0.46 a only items that displayed a medium or medium-large effect size in the intervention group but not the control group were included in the table.     though both the intervention group and the control group demonstrated reductions in library anxiety scores from the pretest to the posttest, the intervention group had a slightly larger effect size than the control group (table 2). the intervention had an impact on several specific items on the scale; items 9, 11, and 18 (see appendix a for the full instrument). all of these items had larger effect sizes in the intervention group, and almost all asked about participants' comfort with library workers. the intervention had a particularly strong effect on item 11 (“people who work at the circulation desk are helpful)." other item-level effect sizes were essentially the same between the intervention and control groups.   discussion   the results of previous studies that demonstrated a correlation between library instruction and reduced library anxiety rates for first-year students were supported by this study, as posttest scores dropped for most participants regardless of the specific library instruction session they experienced. however, results of this study indicate that the warmth-based instruction advocated for by mellon (1986) may not be as crucial as mellon argued it was; in other words, any purposefully-designed library instruction may be as effective in reducing levels of library anxiety as specifically warmth-based instruction. warmth-based instruction did correlate with decreases in library anxiety scores on items related to interactions with library workers that were not seen with the standard instruction alone, indicating that such instruction may be useful in achieving specific goals around increasing student comfort with library workers and willingness to ask for help.   it is important to note, however, that the warmth-based instruction session did not differ dramatically from the standard instruction session. both were based on the same online worksheet; the warmth-based instruction simply took a different approach to guiding students through the worksheet and added an activity and some discussion about library anxiety. developing basic library and information literacy skills, such as using the library’s online discovery service or finding books in the stacks, is important for reducing library anxiety, but there may be a way to design a more wholly warmth-based session that does not include a significant information literacy component and that leads to a more significant reduction in library anxiety when compared with standard instruction.   the reduction in library anxiety rates associated with library instruction, though important, is not the only result worth discussing. participants demonstrated low overall rates of library anxiety in the pretest scores, with scores less than three defined as “low”, indicating that perhaps this group of first-year students is entering the university without significant library anxiety to contend with. however, the low levels of initial library anxiety could be a result of the early stage in students’ university careers, because participants took the pretest when they had only been at the university for a few weeks. as kelly (2017, p. 167) noted, this result may be a manifestation of the dunning-kruger effect, in which students overestimate their ability to complete tasks because “first-year students’ lack of experience with academic research skills also [makes] them unable to accurately assess their own competence in that domain”. this result indicates that it is important for library instructors to recognize that students may come into library sessions lacking context for library skills and with an unrealistic assessment of their own competence, and that library instructors may need to provide that context for students. however, despite the low levels of library anxiety initially displayed, scores still decreased significantly after participating in library instruction, suggesting that library instruction may still have an effect.   one limitation of this study is the unequal distribution of participants’ gender, with 70% of participants self-identifying as female. the cause of this gender imbalance is unclear; more female students than male students enroll in “edhe 105” classes, but not at the level seen here (59% of the total “edhe 105” students were female and 41% were male in fall 2018). previous studies have shown varying levels of influence of gender on library anxiety rates, with no clear indication that gender plays a role or which gender experiences library anxiety more (bostick, 1992; blundell & lambert, 2014; jiao, onwuegbuzie, & lichtenstein, 1996; wildemuth, 2017). however, overall rates of library anxiety may be different with a more gender-balanced sample, which may affect the impact of library instruction. another limitation of this study is that all participants experienced a library instruction session, so the impact of library instruction could not be compared with individuals who did not participate in a session. such a comparison would help to determine if the decrease in library anxiety rates was truly due to the library instruction session or if the weeks between when participants completed the pretest and posttest allowed students to become more familiar with the library on their own, decreasing their anxiety without library instruction. additionally, data on previous library experience or participation in other library instruction sessions was not collected or controlled for, though it is unlikely that participants had experienced other library instruction sessions or had a reason to use university library resources, due to the timing of the “edhe 105” library instruction sessions early in the participants’ first semester at the university.   areas for future research   this study presents multiple areas for future research. the finding that first-year students exhibited low initial levels of library anxiety merits further examination, as this information is key to understanding, targeting, and creating library programming for today’s first-year students. to test this finding, researchers could update jiao, onwuegbuzie, and lichtenstein’s 1996 study that first indicated that first-year students demonstrated the highest levels of library anxiety. alternately, first-year students could be surveyed about their library anxiety at multiple points throughout their first year to see if their early feelings of confidence about the library turned out to be a manifestation of the dunning-kruger effect and diminished as students gained more context or if they continued unchanged.   the method of intervention itself should also be tested further. though the warmth-based instruction in this study did not lead to significantly different outcomes than the standard instruction, this study does not present the only possible method of warmth-based instruction, and other methods should be tested. such methods could include instruction sessions that do not have a significant information literacy skills component, or less formal instruction settings such as pop-up library events around campus that allow students to interact with the library in a fun, casual setting. additionally, comparison between orientation events and class-based instruction sessions should be conducted to determine if one broad category of intervention is more effective in reducing rates of library anxiety. if possible, credit-bearing information literacy courses should also be compared, as should the effect of not participating in any formal library instruction.   conclusion   this study used a quasi-experimental method to investigate the impact of warmth-based library instruction compared to standard instruction on library anxiety levels of first-year undergraduate students. results indicated that instruction was associated with an overall decrease in library anxiety rates (aligning with prior studies done on this topic), though there was not a significant difference between the warmth-based and standard instruction. however, warmth-based instruction was correlated with gains in specific scale items related to interactions with library workers, indicating that warmth-based interventions may be effective in increasing students’ positive perceptions of workers and thus their willingness to ask for help.   "warmth" has long been described as an important feature of library anxiety interventions but has not been rigorously tested. this study attempted to fill that gap by implementing and assessing warmth in library instruction. though warmth-based instruction was not found to be significantly more impactful in decreasing general library anxiety, the intervention tested here points to possibilities for intervention strategies targeted at increasing students' comfort with library workers. it also indicates areas for future research on more wholly warmth-based instruction interventions.   in order to appropriately target and design library instruction, demonstrate the full value of instruction, and ensure that library resources are being used effectively, assessment must address the emotional state in which students utilize information literacy skills and further their knowledge, as well as the content in which they learn. students bring their entire selves into the classroom, including all of their anxieties, and assessment must take this into account. failure to do so can result not only in missed opportunities to connect with students, but also in missed ways to show the impact that library instruction has on student success beyond the acquisition of specific skills.   acknowledgements   the author would like to thank catherine hensly and savannah kelly for their help with the statistical analysis in this study.   references   anwar, a., al-qallaf, c. l, al-kandari, n. m., & al-ansari, h. a. (2011). aqak: a library anxiety scale for undergraduate students. journal of librarianship and information science, 44(1), 36-46. https://doi.org/10.1177/0961000611425568   bostick, s. l. (1992). the development and validation of the library anxiety scale (unpublished doctoral dissertation). wayne state university, detroit.   blundell, s., & lambert, f. (2014). information anxiety from the undergraduate student perspective: a pilot study of second-semester freshmen. journal of education for library & information science, 55(4), 261–273. retrieved from https://files.eric.ed.gov/fulltext/ej1074306.pdf   brown, a. g., weingart, s., johnson, j. r. j., & dance, b. (2004). librarians don’t bite: assessing library orientation for freshmen. reference services review, 32(4), 394-403. https://doi.org/10.1108/00907320410569752   chimah, j. n., nwajei, m., & akpom, c. (2015). library anxiety and intervention strategies: review of conceptualized antecedents in public service librarianship. british journal of education, society, & behavioural science, 10(1), 1-8. https://doi.org/10.9734/bjesbs/2015/18178   collins, b. l., mellon, c. a., & young, s. b. (1987). the needs and feelings of beginning researchers. in c. a. mellon (ed.), bibliographic instruction: the second generation (pp. 73-84). littleton, co: libraries unlimited, inc.   cook, j. m. (2014). a library credit course and student success rates: a longitudinal study. college & research libraries, 75(3), 272-283. https://doi.org/10.5860/crl12-424   diprince, e., wilson, a., karafit, c., bryant, a., & springer, c. (2016). don’t panic! managing library anxiety with a library survival guide. reference & user services quarterly, 55(4), 283–292. https://doi.org/10.5860/rusq.55n4.283   fiske, s. t., cuddy, a. j. c., & glick, p. (2006). universal dimensions of social cognition: warmth and competence. trends in cognitive science, 11(2), 77-83. https://doi.org/10.1016/j.tics.2006.11.005   fleming-may, r. a. & mays, r. & radom, r. (2015). “i never had to use the library in high school”: a library instruction program for at-risk students. portal: libraries and the academy, 15(3), 433-456. https://doi.org/10.1353/pla.2015.0038   gaha, u., hinnefeld, s., & pellegrino, c. (2018). the academic library’s contribution to student success: library instruction and gpa. college & research libraries, 79(6), 737-746. https://doi.org/10.5860/crl.79.6.737   gilbert, j. k. (2009). using assessment data to investigate library instruction for first year students. communications in information literacy, 3(2), 181–192. https://doi.org/10.15760/comminfolit.2010.3.2.80   jiao, q. g., & onwuegbuzie, a. j. (1999). self-perception and library anxiety: an empirical study. library review, 48(3), 140-147. https://doi.org/10.1108/00242539910270312   jiao, q. g., onwuegbuzie, a. j., & lichtenstein, a. a. (1996). library anxiety: characteristics of "at-risk" college students. library and information science research, 18(2), 151-163. https://doi.org/10.1016/s0740-8188(96)90017-1   kelly, s. (2017). first-year students’ research challenges: does watching videos on common struggles affect students’ research self-efficacy? evidence based library and information practice, 12(4), 158-172. https://doi.org/10.18438/b8qq28   kracker, j. (2002). research anxiety and students’ perceptions of research: an experiment, part i: effect of teaching kuhlthau’s isp model. journal of the american society for information science and technology, 53(4), 282-294. https://doi.org./10.1002/asi.10040   kuhlthau, c. c. (1993). seeking meaning: a process approach to library and information services. norwood, nj: ablex.   lewis-beck, m. s., bryman, a., & futing liao, t. (2004). the sage encyclopedia of social science research methods. thousand oaks, ca: sage publications, inc. https://doi.org/10.4135/9781412950589   lowe, m. s., booth, c., stone, s., & tagge, n. (2015). impacting information literacy learning in first-year seminars: a rubric-based evaluation. portal: libraries and the academy, 15(3), 489-512. https://doi.org/10.1353/pla.2015.0030   luetkenhaus, h., hvizdak, e., johnson, c., & schiller, n. (2017). measuring library impacts through first year course assessment. communications in information literacy, 11(2), 339–353. https://doi.org/10.15760/comminfolit.2017.11.2.6   mech, t. f., & brooks, c. i. (1997). anxiety and confidence in using a library by college freshmen and seniors. psychological reports, 81(3), 929-930. https://doi.org/10.2466%2fpr0.1997.81.3.929   mellon, c. a. (1986). library anxiety: a grounded theory and its development. college & research libraries, 47(2), 160-165. https://doi.org/10.5860/crl.76.3.276   muszkiewicz, r. (2017). get to know your librarian: how a simple orientation program helped alleviate library anxiety. public services quarterly, 13(4), 223-240. https://doi.org/10.1080/15228959.2017.1319780   naveed, m. (2017). information seeking anxiety: background, research, and implications. international information & library review, 49(4), 266-273. https://doi.org/10.1080/10572317.2017.1319713   onwuegbuzie, a. j., jiao, q. g., & bostick, s. l. (2004). library anxiety: theory, research, and applications. lanham, md: scarecrow press.   shao, x., & purpur, g. (2016). effects of information literacy skills on student writing and course performance. journal of academic librarianship, 42(6), 670-678. https://doi.org/10.1016/j.acalib.2016.08.006   sinnasamy, j., & abdul karim, n. h. (2015). academic related anxieties: a case study investigating the relationships among library, communication and language anxieties among non-native speakers of english. malaysian journal of library & information science, 20(2), 1–12. retrieved from https://mjlis.um.edu.my/article/view/1763   soria, k. m., fransen, j. l., & nackerud, s. (2017). beyond books: the extended academic benefits of library use for first-year college students. college & research libraries, 78(1), 8-22. https://doi.org/10.5860/crl.78.1.8   van scoyoc, a. m. (2003). reducing library anxiety in first-year students: the impact of computer-assisted instruction and bibliographic instruction. reference and user services quarterly, 42(4), 329-341. retrieved from https://www.jstor.org/stable/20864059   wildemuth, b. (2017). library anxiety impedes college students’ library use, but may be alleviated through improved bibliographic instruction. evidence based library & information practice, 12(4), 275–280. https://doi.org/10.18438/b8k082     appendix a modified library anxiety scale instrument   pretest[1]   you are being asked to respond to statements concerning your feelings about college or university libraries. please mark the number which most closely matches your feelings about the statement. the numbers range from:   1 = strongly disagree  2 = disagree  3 = undecided  4 = agree  5 = strongly agree   1. i'm embarrassed that i don't know how to use the library.                                           1     2     3     4     5   2. a lot of the university is confusing to me.                                                                           1     2     3     4     5   3. the librarians are unapproachable.                                                                                       1     2     3     4     5   4. i can't get help in the library at the times i need it.                                                          1     2     3     4     5   5. the librarians don't have time to help me because they're                                            1     2     3     4     5 always busy doing something else.   6. i get confused trying to find my way around the library.                                              1     2     3     4     5   7. i enjoy learning new things about the library.                                                                   1     2     3     4     5   8. if i can't find a book on the shelf, the library staff will help me.                                 1     2     3     4     5   9. i can always ask a librarian if i don't know how to work a                                           1     2     3     4     5 piece of equipment in the library.   10. the library never has the materials i need.                                                                       1     2     3     4     5   11. the people who work at the circulation desk are helpful.                                          1     2     3     4     5   12. the library is an important part of my school.                                                                1     2     3     4     5   13. i want to learn to do my own research.                                                                              1     2     3     4     5   14. the library does not have good wireless internet.                                                          1     2     3     4     5   15. librarians don't have time to help me.                                                                               1     2     3     4     5                 16. the printers are often out of paper.                                                                                     1     2     3     4     5   17. there are not enough electrical outlets available in the library.                                1     2     3     4     5   18. i can't find enough space in the library to study.                                                            1     2     3     4     5   what is your gender identity? ____ female                ____ male    ____ non-binary / third gender ____ prefer not to say ____ prefer to self-describe: ___________________________     appendix b factor analysis table   factor 1 factor 2 factor 3 factor 4 factor 5 factor 6 i'm embarrassed that i don't know how to use the library       0.473     a lot of the university is confusing to me.       0.986     the librarians are unapproachable.     0.753         i can't get help in the library at the times i need it.   0.654         the librarians don't have time to help me because they're always busy doing something else. 0.384 0.626     0.300   i get confused trying to find my way around the library.   0.500   0.309     i enjoy learning new things about the library.     0.672       if i can't find a book on the shelf, the library staff will help me. 0.743       0.337   i can always ask a librarian if i don't know how to work a piece of equipment in the library. 0.792 0.345         the library never has the materials i need. 0.345   0.361       the people who work at the circulation desk are helpful. 0.436           the library is an important part of my school.     0.543       i want to learn to do my own research.     0.432       the library does not have good wireless internet.           0.956 librarians don't have time to help me.   0.377     0.843   the printers are often out of paper.     -0.544       there are not enough electrical outlets available in the library.   0.432         i can't find enough space in the library to study. -0.403 0.478 0.461     0.463   these items were removed from the final scale because they did not fit in to one of the six identified factors: ·         i am unsure about how to begin my research. ·         i feel safe in the library. ·         the library is a comfortable place to study. ·         i don't understand the library's overdue fines. ·         the library's rules are too restrictive. ·         i don't know what resources are available in the library.     appendix c warmth-based 50-minute instruction session lesson plan   items in bold are unique to the warmth-based session.   i.                    introduction – name, role, welcome (5 minutes) ii.                 think-pair-share and class-wide discussion of how our library is different from your high school library (10 minutes) iii.               lead into how differences can cause anxiety (5 minutes) a.       library anxiety as a demonstrated phenomenon (though not an anxiety disorder) b.       some people may feel it; some people might not -both is fine but you are not alone c.        no one expects you to know how to use our resources and no one expects you to be an expert after today iv.                online worksheet:  tinyurl.com/libraryedhe105 v.                  brief overview of library website (10 minutes) a.       demonstrate one search                                                                i.      select a few sources to discuss differences between popular and scholarly sources b.       give students time to fill in worksheet vi.                brief overview of library catalog (10 minutes total) a.       library organization/library of congress call numbers (5 minutes)                                                               i.      why do we talk about this? share personal experience with library anxiety and confusion when first using an academic library. b.       give students time to fill in worksheet vii.             use catalog to look up book related to major or career (10 minutes) a.       leave classroom, find book on shelf, and take a “shelfie”   [1] the posttest was exactly the same as the pretest without the question about gender identity. research article   what is the best way to develop information literacy and academic skills of first year health science students? a systematic review   joanne munn lecturer centre for teaching and learning southern cross university coffs harbour, new south wales, australia email: joanne.munn@scu.edu.au    jann small health and human sciences liaison librarian southern cross university coffs harbour, new south wales, australia email: jannsmall56@gmail.com   received: 3 nov. 2016     accepted: 21 july 2017         2017 munn and small. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – this systematic review sought to identify evidence for best practice to support the development of information literacy and academic skills of first year undergraduate health science students.   methods – a range of electronic databases were searched and hand searches conducted. initial results were screened using explicit inclusion and exclusion criteria to identify 53 relevant articles. data on study design, student cohort, support strategy, and learning outcomes were extracted from each article. quality of individual studies was considered and described narratively. articles were classified and findings synthesized according to the mode of delivery of the intervention (embedded, integrated, or adjunct) and classification of the study’s learning evaluation outcome (organizational change, behaviour, learning, or reaction).   results – studies included in this review provide information on academic skills and information literacy support strategies offered to over 12,000 first year health science students. courses targeted were varied but most commonly involved nursing, followed by psychology. embedded strategies were adopted in 21 studies with integrated and adjunct strategies covered in 14 and 16 studies respectively. across all modes of delivery, intervention formats included face-to-face, peer mentoring, online, and print based approaches, either solely or in combination. most studies provided some outcomes at a level higher than student reaction to the intervention. overall, irrespective of mode of delivery, positive learning outcomes were generally reported. typically, findings of individual studies were confounded by the absence of suitable control groups, students self-selecting support and analysis of outcomes not accounting for these issues. as a result, there is very little unbiased, evaluative evidence for the best approach to supporting students. nonetheless, our findings did identify poor student uptake of strategies when they are not interwoven into the curriculum, even when students were encouraged to attend on the basis that they had been identified at academic risk.   conclusions – the majority of studies included have reported positive learning outcomes following the implementation of academic skills and information literacy support strategies, irrespective of their mode of delivery (embedded, integrated, or adjunct). clear, rigorous evidence that embedded strategies offer superior learning outcomes compared to other delivery modes is lacking. however, because of poor student uptake of strategies offered outside curricula, embedded modes of academic and information literacy support are recommended for first year health science courses.     introduction   information literacy and other generic academic skills are generally considered essential for successful student transition at university (fergy, heatley, morgan, & hodgson, 2008; goldfinch & hughes, 2007; hafford-letchfield, 2007; moore, brewster, dorroh, & moreau, 2002; ooms, fergy, marks-maran, burke, & sheehy, 2013). information literacy is defined as “an understanding and set of abilities enabling individuals to recognize when information is needed and have the capacity to locate, evaluate, and use effectively the needed information” (bundy, 2004, p. 3). academic skills are a broader domain, focused on generic educational abilities such as reading, writing, citation skills (hitch et al., 2012), critical thinking, problem solving and information literacy (gunn, hearne, & sibthorpe, 2011), technology skills, time management, communication, and working with others (goldfinch & hughes, 2007). on many levels, information literacy and academic skills are not discrete domains (for example, effective use of information and citation skills) and development of such skills are frequently targeted together (bailey et al., 2007; cassar, funk, hutchings, henderson, & pancini, 2012; pryjmachuk, gill, wood, olleveant, & keeley, 2012).   data has shown first year heath science students are underprepared for university study in the areas of information literacy (birmingham et al., 2008; dubicki, 2013; moore et al., 2002) and academic skill (palmer, levett-jones, smith, & mcmillan, 2014; sacre & nash, 2010; stevens & miretzky, 2012). recent papers suggest students have the greatest chance of acquiring such skills when strategies are embedded into course curricula (boruff & thomas, 2011; chanock, horton, reedman, & stephenson, 2012; gunn et al., 2011; mcwilliams & allan, 2014; shorten, wallace, & crookes, 2001), as embedding allows for the delivery in a meaningful context, thereby providing opportunity for students to develop the skills in the course of their discipline specific study program (chanock, 2013; chanock et al., 2012). increasingly, embedded strategies are developed collaboratively with academic language and learning and information literacy specialists working together with discipline academics (ambery, manners, & smith, 2005; einfalt & turley, 2009; rae & hunn, 2015; wilkes, godwin, & gurney, 2015).   previous systematic reviews on the teaching of information literacy and academic skills to undergraduate students, particularly in health science disciplines, are limited. koufogiannakis and wiebe (2006) investigated the effectiveness of teaching methods for delivering information literacy to undergraduate students. it was concluded that computer assisted instruction is as effective as traditional methods, and self-directed independent learning and traditional instruction are better than no instruction. while not limited to undergraduate students, similarly, zhang, watson and banfield (2007) showed that face-to-face delivery was equally as effective as computer assisted instruction for developing the information literacy skills of academic library patrons. more specifically, for health and medical students and clinicians there is limited evidence that information skills training improves skills, and insufficient evidence to determine the most effective training methods (brettle, 2003). for academic skills, oermann et al. (2015) found that despite a wide range of educational strategies for developing nursing students’ and practitioners’ writing, the majority of individual studies reporting on such strategies do not provide estimates of effectiveness, with the authors concluding that the best strategies to improve writing ability could not be identified.   the view that the best approach to develop information literacy and academic skills is to embed these within curricula is often based on expert opinion or descriptive accounts of strategies rather than empirical research, and where available, empirical evidence is often subject to methodological bias (pryjmachuk et al., 2012). furthermore, despite the notion that embedded strategies are preferred practice, strategies adjunct to the curricula continue to be frequently offered (for example, see bailey et al., 2007; edwards & o'connor, 2011; fenton-smith & frohman, 2013). additionally, it has been specifically identified that there is a knowledge gap in relation to academic language and learning support for the health science disciplines (fenton-smith & frohman, 2013). in this context, the following paper is a review of the current literature evaluating information literacy and academic skills support strategies for first year health science students.   objective   the specific objective of this review is to identify evidence for best practice to support the development of first year university health science students’ information literacy and academic skills using a systematic review framework. here, health science students include those studying health professions, excluding medicine. while systematic reviews are not common place in higher education (bearman et al., 2012), this approach is beneficial as it provides evidence via a logical, comprehensive synthesis and appraisal of literature on the topic (bearman et al., 2012; hammick, dornan, & steinert, 2010).   methods   the current systematic review was planned and conducted using a protocol developed by the two authors based on guidelines and previous work involving systematic review methodology in health and higher education (bearman et al., 2012; hammick et al., 2010; steinert et al., 2006). the planned protocol was undertaken without significant variation and is represented by the methods reported here.   information sources   the following electronic databases were searched: academic search premier, cinahl plus with full text, eric, education research complete, psycinfo, sportdiscus with full text, the psychology and behavioral sciences collection and proquest. in addition, hand searching of the journal of academic language and learning, international journal of the first year in higher education, the journal of english for academic purposes, and papers from the first year in higher education conferences was performed, as these journals were not indexed in any of the institution’s subscribed databases. manual checking of the reference list of included articles for relevant papers was also performed.   search strategy   the reviewers had identified a selection of relevant articles which were used as a search strategy validity tool. the initial search strategy failed to locate two papers identified in the pilot phase of the search. the search was therefore modified and repeated in the databases listed above. the modified search was also structured to exclude articles related to children and filtering strategies applied to limit results to a date range from 2000 to 2014, and to english language articles (see appendix a). results were exported to an endnote library.   study selection   exclusion criteria   articles not written in english and not within the date range specified had already been excluded in the databases by the search criteria. newspaper articles, theses, and book reviews were also excluded. reviews, conference papers not written in full, short opinion, discussion, or descriptive pieces were also excluded. remaining articles were initially screened independently by the authors by reading the article abstract. where there was not initial consensus between reviewers, the full text was retrieved and further discussion ensued until consensus was reached.   inclusion criteria   following exclusion based on the above criteria, the full text of the remaining articles was screened independently by each author. to be included, articles had to describe original research, evaluating the effect of a clearly identified support strategy to improve academic or information literacy skills development of first year undergraduate students studying health science professions (excluding medicine) at university. in addition, learning outcomes for first year health student cohorts had to be clearly identified. again, where there were discrepancies between authors regarding an article’s inclusion, discussion ensued until consensus was reached.   data collection and synthesis   relevant information and data were extracted from included papers into tables on each study’s aims, design, cohort, support strategy characteristics (including mode of delivery), outcome measures, analysis, and findings. authors independently evaluated individual papers to identify and record potential methodological limitations, predominantly associated with selection bias. these limitations were then considered collaboratively and comments recorded about the possible risk of such biases on interpreting the reported outcome of individual studies. a narrative synthesis of potential risks and their possible influence on findings is reported.   our approach to critically synthesize the literature in this review was based around two frameworks. first, for classifying the support strategy’s mode of delivery and second, for classifying learning outcomes.    categories for mode of delivery were adapted from the criteria outlined by bonanno (2002) as follows:   embedded model: this model moves the development of information literacy and academic skills into the mainstream unit curriculum.   integrated resources, modules or workshops: these resources, modules or workshops are offered within a unit timetable or unit learning site. they are discipline and/or assessment task specific.   adjunct resources, modules or workshops: these resources, modules or workshops are offered outside of timetabled unit sessions, either lectures or tutorials. they are generic, and may be delivered online, in an electronic format or face-to-face. there is no consistency in the literature regarding such terminology, and terms are used inconsistently by academics, information literacy specialists, and language and learning specialists. this can lead to ambiguity when attempting to make sense of the literature. as such, our categorization may not align with terminology used in the individual papers themselves (betts, bostoek, elder, & trueman, 2012; hendricks, andrew, & fowler, 2014; wray, aspland, taghzouit, & pace, 2013), nor with other frameworks used (bundy, 2004, p. 6; harris & ashton, 2011, p. a-81).   within the categories classifying mode of delivery, an outcome classification was assigned based on the study’s outcome measure. this classification was adapted from kirkpatrick’s model of four levels of evaluation (kirkpatrick, 1996) for training programs. the evaluation levels used, from lowest to highest level of the hierarchy, are:   reaction: a measure of how students thought or felt about the intervention. for example, agreement or helpfulness likert questionnaires.   learning: a measure of the skills or knowledge students have gained as a result of the intervention. for example, a change in score on skill quiz.   behaviour: a measure of how the student’s behaviour has changed as a result of the intervention. for example, applied learning where assessment grades, or retention rates have changed.   organizational change [listed as results by kirkpatrick (1996)]: a measure of change that has occurred at the institutional level as a result of the intervention; generally involving a curriculum change. for example, following the study, the intervention was adopted at an organizational level.   a potential limitation of applying such a model in this systematic review is that the kirkpatrick model was designed for training in industry as opposed to hierarchical application as a critical appraisal tool (yardley & dornan, 2012). despite this, the use of this tool as a conceptual framework is consistent with guidelines (hammick et al., 2010) and previous systematic reviews in health education (steinert et al., 2006).   data was synthesized for analysis by categorizing studies according to the mode of delivery of the intervention (embedded, integrated, or adjunct) and, within these classification, for outcome (organizational change, behaviour, learning, or reaction).   results   fifty-three papers were identified for review. the prisma flow chart (figure 1) details the number of articles identified for inclusion and excluded during phases of the screening process. from these 53 papers, 51 independent studies were identified. it should be noted that each of the two paired papers by wallace, shorten, and crookes (2000) and shorten et al. (2001); chester, burton, xenos, elgar, & denny (2013a) and chester, burton, xenos, & elgar (2013b); salamonson, koch, weaver, everett, & jackson (2010) and weaver and jackson (2011), report on findings from the same study and cohort of students. when summarizing these paired papers in this review, extracted data on cohort and types of interventions have only been counted once (that is, per study). where the individual papers discussed different aspects of data, those findings have been counted independently and discussed accordingly. furthermore, the paper by cranney, morris, spehar and scoufis (2008) reports on two independent studies in the one research publication; one targeting information literacy and one the academic skill of working with others.   from the 51 included studies, over 12,000 health science students were provided an opportunity to access support strategies (see appendices b-d). participant numbers across studies varied from 8 to 1,841 students. while the majority of studies did not focus on specific equity groupings, five reported on at risk students (n = 345); two on indigenous students (n = 47); two on students with low english language levels (n = 235) and one on students with learning disability (n = 384). courses targeted were nursing (25); psychology (11); health sciences (9); occupational/physical therapies (3); and bioscience and midwifery (1 each); over half of the targeted students were enrolled in a single discipline. studies were carried out in australia (18); england (12); the united states (10); ireland (3); new zealand (2); and 1 each in south africa, thailand, canada, and indonesia. interventions in 10 of the studies targeted both information literacy and academic skills, 11 targeted information literacy, and 30 targeted academic skills. for embedded, integrated and adjunct interventions, formats included face-to-face, peer mentoring, online, and print.   embedded strategies   an embedded mode of delivery was used in 21 studies (23 papers) (table 1). a number of formats were used to embed support strategies in the curriculum, either singularly or in combination, including workshops, online learning modules, and resources. these were often devised or delivered collaboratively by discipline academics and information literacy and academic language and learning specialists. the majority (11 studies) targeted academic skills, with 5 each addressing information literacy or academic skills and information literacy in combination.   organizational change   corbin and karasmanis (2010) effected change at an organizational level through an iterative process of implementing an online information literacy support strategy and response to stakeholder feedback. while improvement in information literacy scores was reported, these scores remained poor post-intervention, with only 2 of 11 categories improved to the extent that over half of the respondents were correct. despite this strategy being institution wide, data on learning indicates limited success.   behaviour   eight studies (nine papers) assessed behavioural outcomes, providing evidence of the impact on application of learning, often in the context of curriculum assessment (table 1 and appendix b). a control or pseudo–control group featured in three studies that targeted a broad spectrum of academic skills (chester et al., 2013a; chester et al., 2013b; walker et al., 2010; winstone & millward, 2012). winstone and millward (2012) showed a statistically significant improvement in essay grades and positive student reaction with scaffolded academic skills tutorials; however, this small (2%) increase may not be practically significant (kirk, 1996). in comparison to their peers, students receiving learning strategies embedded into the curriculum, including formative assessments, were shown to adopt a deeper approach to learning, although surprisingly, higher exam scores were correlated with surface approaches to learning (walker et al., 2010). the utilization of peer mentoring, to support academic and psychosocial adjustment to university, resulted in improved grades and positive student reaction but did not improve academic progress (chester et al., 2013a; chester et al., 2013b).   several studies investigated student outcomes following embedded support without comparison to a control cohort, with most assessing behaviour based on academic writing outcomes. with a focus on information literacy skills, boruff and thomas’ (2011) strategy, involving a lecture, workshops, and a specific assessment, was deemed successful, with 97% of students scoring at least 80% on the assessment. cassar et al. (2012) investigated a multifaceted academic skills and information literacy support strategy. outcomes showed improvement in writing grade levels as the semester progressed and pre/posttest numeracy significantly improved. improved writing outcomes, for consecutive written assessments, were also demonstrated by fallahi, wood, austad, and fallahi, (2006) with following in-class lessons targeting academic writing and providing peer editing and extensive feedback. while improvement for referencing was immediate, other writing skills did not improve until the fourth assessment (fallahi et al., 2006). targeting indigenous australian health science students, rose, rose, farrington, and page (2008) showed scaffolding of academic writing skills significantly improved academic literacy skills. finally, with a specific focus on teamwork, cranney et al. (2008) looked at the development of this skill through a strategy built around a group assessment reporting positive outcomes based on mean assessment grades (87%) and a strong relationship between higher marks and group cohesiveness.         figure 1 prisma flow chart for identifying and selecting articles to be included in the systematic review. reprinted from "preferred reporting items for systematic reviews and meta-analyses: the prisma statement,” by d. moher, a. liberati, j. tetzlaff, d. g. altman, & the prisma group (2009). plos medicine 6(6): e1000097 p. 3. http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1000097. copyright 2009 by prisma. reprinted with permission under the creative commons attributions licence.     learning   learning outcomes were the highest level of evaluation in four studies (reported in five papers, table 1 and appendix b). several of these papers, investigating information literacy, used strategies employing a control or pseudo-control. in a study published by both wallace et al. (2000) and shorten et al. (2001), within group improvement for searching and locating resources, as well as interpretation of bibliographic citations, were identified following a structured information literacy strategy. while these improvements were superior to non-program students’ scores for locating and interpreting resources, this was not the case for citation skills. additionally, students reported higher levels of skill confidence than non-program students. van moorsel (2005) reported that a computer literacy strategy significantly improved computer and information literacy of those students in the intervention group, whereas scores for controls remained unchanged. despite the use of a control, however, between-group statistical comparisons were not reported. using the data from van moorsel (2005, table 4, column 2 & 3), we calculated 95% confidence intervals for between-group differences (appendix b), demonstrating the intervention group performed better than controls at both immediate and 5-week follow-up periods, supporting the reported finding.   in the absence of a control group, pryjmachuk et al. (2012), using a combined information literacy and academic skills strategy, reported significant (13%) improvement in knowledge and in confidence. likewise, mandleco, bohn, callister, lassetter, and carlton (2012) reported within group improvement in 12 of 26 categories of grammar as well as improvement in writing confidence following the provision of three embedded writing modules.   reaction   eight papers (table 1) evaluated outcomes based only on reaction. consistently, these papers showed that overall, embedded academic skills strategies are viewed positively by students for providing practical tools for improving skills, confidence, and being useful or helpful.   integrated strategies   integrated strategies were delivered by 14 studies, with the majority using module style interventions (table 1 and appendix c). most commonly, the focus was on information literacy skills (brettle & raynor, 2013; craig & corrall, 2007; cranney et al., 2008 (study 1); lalor, clarke, & sheaf, 2012; weiner, pelaez, chang, & weiner, 2011; xiao, 2010), followed by academic skills (betts et al., 2012; elander, pittam, lusher, fox, & payne, 2010; griffiths & nicolls, 2010; wray et al., 2013) and then academic and information literacy skills in combination (hendricks et al., 2014; hooley, morrison, thomas, & marrs, 2011; turnbull, royal, & purnell, 2011). modules were made available to students online, although face-to-face delivery was also utilized (brettle & raynor, 2013; craig & corrall, 2007; lalor et al., 2012). in the case of xiao (2010), face-to-face delivery was blended with self-paced resources and online support.   organizational change   a pilot study by hooley et al. (2011), to normalize library and academic skills support access, led to extension of the strategies within the organization. this was despite ambiguous evidence on student success. while there were fewer failures (3%) and similar mean essay grades during the pilot phase compared to a previous cohort, retention was 10% lower in the intervention group. as noted by the authors, the impact on student performance is tentative because of the possible influence of many uncontrolled variables between the two comparison groups; however, authors suggest that the aims of raising the profile of support services and normalizing access to support was achieved.     table 1. summary of studies included in the review based on their mode of delivery and outcome measure. mode of deliverya study outcome measureb     embedded organizational change behaviour   learning   reaction arpanantikul et al. (2006)       · beatty et al. (2014)       · beccaria et al. (2014)       · boruff et al. (2011)   ·     cassar et al. (2012)   ·     chester et al. (2013a); chester et al. (2013b)   ·     corbin et al. (2010) ·       cranney et al. (2008)   ·     fallahi et al. (2006)   ·     hegarty et al. (2010)       · mandleco et al. (2012)     ·   mcmillan et al. (2011)       · pryjmachuk et al. (2012)     ·   rose et al. (2008)   ·     san miguel et al. (2013)       · thies et al. (2014)       · van moorsel (2005)     ·   walker et al. (2010)   ·     wallace et al. (2000); shorten et al. (2001)     ·   webster et al. (2014)       · winstone et al. (2012)   ·     integrated betts et al. (2012)   ·     brettle et al. (2013)     ·   craig et al. (2007)     ·   cranney et al. (2008) [study 1]   ·     elander et al. (2010)       · griffiths et al. (2010)   ·     hendricks et al. (2014)     ·   hooley et al. (2011) ·       lalor et al. (2012)     ·   rolfe (2011)   ·     turnbull et al. (2011)       · weiner et al. (2011)       · wray et al. (2013)   ·     xiao (2010)     ·   adjunct bailey et al. (2007)   ·     balch (2001)   ·     brown et al. (2008)     ·   edwards et al. (2011)     ·   fleming et al. (2005)   ·     golding et al. (2012)   ·     hammond et al. (2010)       · hoyne et al. (2013) ·       igbo et al. (2011)       · jorgensen et al. (2013)     ·   kartika (2008)     ·   palmer et al. (2014)   ·     salamonson et al. (2010); weaver et al. (2011)   ·     sikhwari et al. (2012)   ·       silburn et al. (2012)   ·       sopoaga et al. (2011)   ·           behaviour   in five papers, behaviour was the highest outcome assessed (table 1 and appendix c). studies by betts et al. (2012) and rolfe (2011), involving turnitin (an originality checking software), compared student receiving the strategy to controls. betts et al. found a significant reduction in plagiarism incidents compared with the previous cohort (1 compared to 7), whilst in contrast, rolfe noted no overall reduction in plagiarism even though the incidents of poor paraphrasing were decreased (22 compared to 7 incidents). student cohorts in each of these studies reacted positively to the strategy. wray et al. (2013) studied students with specific learning difficulties (spld) and, following study skills sessions, compared outcomes to students with spld from previous cohorts and to peers without identified learning difficulties. findings showed higher rates of progression (87% of students) compared to spld students from previous cohorts who did not receive the study sessions (62%) and progression rates were comparable to student peers. furthermore, student reactions to sessions were positive.   following access to online information literacy modules, cranney et al. (2008 [study 1]) showed statistically significant, within group, improvements for students’ pre/posttest scores specific to information literacy skills and positive student reaction. furthermore, specific to behaviour, knowledge application showed significant positive correlation of posttest scores with assignment grades. online learning support was also provided by griffiths and nicolls (2010) where relevant e-tivities and scaffolded phases of academic writing were investigated. here, 100% of students passed the reflective essay assignment and all responses were positive for online academic support. neither of these papers had the benefit of a comparison cohort to evaluate the direct effect of the online modular study support intervention on student learning.   learning   learning was the highest level of outcome assessed in five papers (table 1 and appendix c). brettle and raynor (2013), in a well-designed paper, compared student learning following an online information literacy tutorial with a control group who participated in a face-to-face tutorial. a validated pre/posttesting of scored search histories was used and found no between group difference in scores post-intervention, but statistically significant within group improvements for both groups. despite significant improvements, mean scores for literature searching were still poor for both groups (less than 25%).   four non-controlled studies assessed information literacy strategies. one delivered the strategy face-to-face in small groups (craig & corrall, 2007) and two were computer based (lalor et al., 2012; xiao, 2010), all demonstrating a positive impact on learning based on pre/posttesting. the final study, with a broader focus, combined academic and information literacies in a 10 module strategy and showed student knowledge improved, on average, by 7.6% (hendricks et al., 2014).   reaction   the effectiveness of support strategies based solely on reaction outcomes was assessed in three papers (elander et al., 2010; turnbull et al., 2011; weiner et al., 2011; table 1 and appendix c). while these studies provide evidence that integrated information literacy and academic skills support strategies are viewed positively by students, they do not contribute directly to evidence about improved student learning outcomes.   adjunct strategies   adjunct support strategies (table 1) were used in 17 papers (16 studies), with 14 targeting academic skills and 2 targeting academic skills in combination with information literacy skills (appendix d). the papers by salamonson et al. (2010) and weaver and jackson (2011) both reported on the same intervention implemented with the same cohort, but each reported on different outcomes (behaviour and reaction). the majority of adjunct strategies were optional for students, with several using diagnostic screening to refer students for academic skills support (bailey et al., 2007; hoyne & mcnaught, 2013; palmer et al., 2014). in this context, problematically, outcomes were often compared between students who self-selected to attend strategies and those who did not (cook & beckman, 2010). the majority of studies, with the exception of salamonson et al. (2010) and weaver and jackson (2011), did not use a formal control group. the format of these strategies included face-to-face workshops focused on study skill development (bailey et al., 2007; hoyne & mcnaught, 2013; palmer et al., 2014; salamonson et al., 2010; sikhwari, selepe, & maluleke, 2012; sopoaga & van der meer, 2011), peer-assisted learning (hammond, bithell, jones, & bidgood, 2010), and synchronous (silburn, flack, bridgeman, & warwick, 2012) or asynchronous online workshops (brown, dickson, humphreys, mcquillan, & smears, 2008).   organizational change   hoyne and mcnaught (2013) investigated a reading and writing program delivered by academic support specialists for students not meeting post-entrance literacy assessment benchmarks. this initiated graduated change in policy at an organizational level, making the program compulsory for at risk students. once the program was compulsory, there was a 50% reduction in fail rates in a core literacy unit for at risk students.   behaviour   behaviour was the highest level of outcome for 10 papers (bailey et al., 2007; balch, 2001; fleming & mckee, 2005; golding, wasarhaley, & fletcher, 2012; palmer et al., 2014; salamonson et al., 2010; sikhwari et al., 2012; silburn et al., 2012; sopoaga & van der meer, 2011; weaver & jackson, 2011; appendix d). provision of learning support was offered to students where poor academic skills were evident (bailey et al., 2007; palmer et al., 2014). at risk students were identified by bailey et al. (2007) using a diagnostic essay, and palmer et al. (2014) via a diagnostic academic literacy tool (measuring the academic skills of university students [masus]). palmer et al. (2014) showed that following early feedback and compulsory learning support, 73% of at risk students achieved a pass grade, with 77% of students, who were in the lowest band on the masus (< 9), improving their score, with 40% moving to the highest band. the analysis by bailey et al. reported students who attended at least one of four academic skills or information literacy workshops improved their grade. the size of the improvement was not reported in the bailey study, nor was a control comparison provided. of importance, however, when considering the value of add-on support, the authors reported that only 50% of those identified as needing additional support accepted it.   face-to-face workshops on varied academic skills content were the focus of several studies (fleming & mckee, 2005; salamonson et al., 2010; sikhwari et al., 2012; sopoaga & van der meer, 2011; weaver & jackson, 2011). generally, outcomes were reported as positive, based on reported improvements in participants’ behaviour. neither fleming and mckee (2005), nor sikhwari et al. (2012) made use of a control group, and whilst sopoaga and van der meer (2011) compared the academic results of those students who attended with those who did not, the students self-selected to participate in the intervention. while a key strength of the study reported by salamonson et al. (2010), and weaver and jackson (2011) was random allocation of participants into an intervention and control group, unfortunately, participants that were allocated to the intervention and did not attend were then evaluated in the control group.   an external, synchronous academic skills strategy was implemented by silburn et al. (2012). students self-selected into the study strategy and results showed these students had a higher mean assessment mark of 8% or greater (adjusted for baseline scores) than students in any of the 3 comparison groups. furthermore, 90% of students viewed this strategy positively and most of the few negative responses were related to technology.   print-style interventions for developing academic skills were investigated in the studies by golding et al. (2012) and balch (2001). golding found students who used flash cards more frequently performed significantly better on exams than students who used the cards less. balch, however, found no correlation between overall degree-of-use ratings and course performance, despite a significant positive correlation between degree-of-use and helpfulness.   learning   four papers used learning as the highest level of evaluation. three of these delivered academic skills strategies (brown et al., 2008; jorgensen & marek, 2013; kartika, 2008) and demonstrate mixed results for the skills sets tested (appendix d). while the study by jorgensen and marek (2013) had students self-select attendance at workshops, findings showed that those attending had significantly greater improvement for recognizing academic writing errors compared to non-attending controls. in the absence of a control for comparison, study skills delivered as part of an orientation program were shown to significantly improve general study, concentration, and exam preparation skills, but had no benefit on time management, writing, and note taking based on a within group pre/posttest study skills inventory (kartika, 2008). a non-compulsory online learning intervention to develop referencing skills resulted in no significant change in pre/posttesting but a significant increase in skill confidence (reaction) on four of seven items (brown et al., 2008). also of note here was the poor uptake of this non-compulsory strategy, with less than 36% of students accessing resources. edwards and o'connor, (2011) investigated a compulsory combined online learning intervention, involving computer literacy, research skills, and referencing, finding a 17% improvement in skill pre/post-intervention. furthermore, students also responded positively to the value of the strategy.   reaction   two papers based their evaluation on reaction (hammond et al., 2010; igbo et al., 2011; table 1 and appendix d). both papers offered face-to-face academic skills interventions on a self-selection basis, delivered to three separate cohorts over a three year period. in the study by hammond et al. (2010), while students agreed peer assisted learning improved social aspects of learning, they felt it did not improve study skills. additionally, there was generally low attendance for the program (appendix d). a multifaceted, face-to-face program was reported by igbo et al. (2011) to be helpful and appreciated.   discussion   this systematic review has identified very little unbiased, evaluative evidence on the best approach for developing either the academic or information literacy skills of first year health science undergraduates. this is despite the widely reported benefits of study skill support from studies included in this review, irrespective of mode of delivery. while some studies compared formats of intervention, for example, handout compared to workshop and handout (jorgensen & marek, 2013); online to face-to-face (brettle & raynor, 2013); online, synchronous to asynchronous (silburn et al., 2012), none of the studies aimed to compare the effect of the support strategy under different modes of delivery (embedded, integrated, or adjunct).   embedded strategies were investigated most frequently (21 studies), with more than half of these (13) delivering outcomes at the level of learning or higher, thereby providing evidence based on student learning rather than the student’s perception of the strategy on their learning. of these 13 studies, all viewed the strategies positively. this included embedded strategies being adopted at an organizational level, even in the absence of empirical findings demonstrating the strategy’s efficacy (corbin & karasmanis, 2010). others report outcomes such as assessment results (for example, boruff & thomas, 2011; cassar et al., 2012; chester et al., 2013a; chester et al., 2013b) and improvement in skill domains (for example, fallahi et al., 2006; mandleco et al., 2012; rose et al., 2008) without comparison to a control group. this is problematic because without a control group, this improvement could be attributed to other variables, such as natural progression (cook & beckman, 2010), rather than the interventions themselves. where between group comparisons were made, studies used previous or similar student cohorts who had not undertaken the intervention (for example, chester et al., 2013a; chester et al., 2013b; shorten et al., 2001; van moorsel, 2005; walker et al., 2010; wallace et al., 2000; winstone & millward, 2012). while the reported findings in such studies were positive, due to inherent biases associated with non-random allocation of participants (cook & beckman, 2010), such comparisons can be limited. overall, from the studies included in this review, learning outcomes associated with embedded strategies are positive; however, the effect on learning cannot be clearly attributed to the intervention strategies themselves.   overall, integrated strategies are also viewed positively for supporting the development of first year health science students’ academic and information literacy skills. consistent with embedded delivery, one study showed organizational uptake of a strategy with ambiguous quantitative evidence of a beneficial impact on student success and retention (hooley et al., 2011). eleven studies reported that learning (via behavioural or learning outcomes) occurred subsequent to integrated support strategies. again, the majority of studies only looked at outcomes for an intervention group, or provided comparison to a pseudo-control group, limiting the capacity to conclude about the learning effects of the intervention directly (cook & beckman, 2010). one study did randomly allocate participants (brettle & raynor, 2013), providing strong evidence that the delivery of integrated information literacy support was equally as effective via either face-to-face or online formats.   for adjunct strategies, again, positive learning and behaviour outcomes are typically reported. similar to other modes of delivery, findings here are commonly based on outcomes from an intervention cohort alone (for example, bailey et al., 2007; balch, 2001; golding et al., 2012; palmer et al., 2014; sikhwari et al., 2012). where comparison to other student groups is provided, this is usually a cohort of students who chose not to attend the intervention (for example, fleming & mckee, 2005; jorgensen & marek, 2013; silburn et al., 2012; sopoaga & van der meer, 2011) and thereby, such findings are subject to selection bias (cook & beckman, 2010). while, by nature, adjunct strategies usually depend on students self-selecting support, there may be possible systematic differences between students who self-select support and those that do not. factors related to self-selection may result in differences in learning outcomes regardless of the intervention strategy (cook & beckman, 2010). nonetheless, the studies investigating adjunct strategies included in this review provide insight into their potential limitation of low uptake, even where a need for skill development is identified (bailey et al., 2007; brown et al., 2008). of practical relevance, where adjunct strategies were made compulsory for at risk students, the beneficial impact on student success was evident (hoyne & mcnaught, 2013).   a major limitation to finding unbiased evidence on best practice for supporting student skill development is that the majority of studies’ findings, despite reporting positive outcomes, are confounded by the absence of a control. only 11 of the 50 studies included in this review provided evidence on student outcomes in comparison to a “control” or alternative intervention. most frequently, studies provided only a pseudo-control whereby students self-selected into an intervention or alternative group (for example, fleming & mckee, 2005; golding et al., 2012; jorgensen & marek, 2013; silburn et al., 2012) or a previous cohort was used (betts et al., 2012; hoyne & mcnaught, 2013; walker et al., 2010). while it is not uncommon for educational research to lack rigorous design, unfortunately, where students self-select into groups, selection bias is likely to confound results, or comparison to previous cohorts may also be confounded by other uncontrolled or systematic variables unrelated to the intervention (cook & beckman, 2010).   two studies in this review attempted to control for biases related to group allocation by randomizing participants. brettle and raynor (2013) compared the effectiveness of integrated support strategies via either online tutorials or face-to-face delivery for developing information literacy skills. as previously discussed, this well-designed study found no between group differences in learning scores post-intervention, with both groups, although still scoring poorly, improving. similarly, random allocation was adopted in the study reported in the papers by salamonson et al. (2010) and weaver and jackson (2011). here, esl students were randomly allocated to an adjunct intervention targeting academic learning and writing support or to a usual support that included an invitation to attend a generic skills program offered by the university. assignment scores for those attending the intervention were significantly higher than controls. while a strength of this study design was the random allocation of participants, only data for the 47% of the participants who were allocated to the intervention and actually attended were included in the analysis, thereby confounding the study’s findings on the basis that the data was not analyzed on intention to treat principles (sainani, 2010).   reaction was the only level of evaluation for 13 studies in this review. eight of these investigated embedded strategies. evaluating the effectiveness of an intervention only on outcomes at the level of reaction is problematic. while the overall positive reactions to the majority of interventions is commendable, liking a strategy, or reporting an increase in confidence after completion does not necessarily translate to learning or increases in skill level, or improved grades and retention. for example, brown et al. (2008) showed no significant change in referencing performance but a significant increase in skill confidence. likewise, there was a notable mismatch between student success and reaction where hooley et al. (2011) showed positive reaction outcomes, yet a 10% decline in retention. in further support of this inconsistency, sikhwari et al. (2012) found a one-day study skills workshop resulted in improved academic achievement but a reduction in scores on a study skills inventory test.   where individual studies use surveys to evaluate student reaction, response rates are frequently low. poor response rates were noted in several studies included in this review (for example see, beccaria et al., 2014 [34%]; craig and corrall, 2007 [38%]; elander et al., 2010 [30%]; turnbull et al., 2011 [19%]). systematic non-response can, in part, contribute to non-response bias (nishimura, wagner, & elliott, 2016). it is possible that findings on reaction outcomes, measured via surveys, are subject to such bias and this may also contribute to incongruence of reaction with actual learning and student success.   low uptake of non-compulsory interventions external to the curriculum is another issue highlighted by the review. in health science disciplines, such interventions are reported as the most common type of support strategies implemented in practice, as they do not encroach on “credit-bearing class time” (fenton-smith & frohman, 2013, p. a-61). providing evidence to support poor uptake of non-compulsory strategies, less than 36% of health science students accessed online referencing modules (brown et al., 2008) and similarly, only 47% of students allocated to a targeted academic learning and writing program attended (salamonson et al., 2010; weaver & jackson, 2011). the literature reports that time demands (may, hodgson, & marks-maran, 2005) and lack of relevance and misperception of skill levels (kimmins & stagg, 2009), as well as negative stigma (goldingay et al., 2014), are associated with students not attending adjunct support strategies. in this review, where support programs were made compulsory for all students, completion rates were high (for example, weiner et al., 2011) and this has significant impact, particularly for students academically at risk (hoyne & mcnaught, 2013).   students identified as being academically at risk yet failing to engage with support offered was an issue identified in this review. wray et al. (2013) found that 48% of students identified as at risk of having a specific learning difficulty did not pursue further support on offer. likewise, only 50% of nursing students identified on a diagnostic essay accepted additional academic support (bailey et al., 2007). even lower uptake was identified by beatty, collins, and buckingham (2014), where only 20% of at risk students engaged with available support. making attendance at support programs for those identified as at risk compulsory, however, was found to significantly reduce failure rates (hoyne & mcnaught, 2013). while this may be perceived as ethically inappropriate, evidence from this review suggests that for students to access support, interventions either need to be compulsory or embedded as a way of increasing the likelihood of engagement.   limitations   a key limitation to this review is that it has not been able to clearly achieve our aim. results of studies included are not based on stringent methodology and quantitative analysis; therefore, unbiased evidence about the best approach to support the academic and information literacy needs of first year health science students cannot be provided. this issue has previously been acknowledged by pryjmachuk et al. (2012) regarding effectiveness of study skills support and commenting on the lack of evaluative evidence in the literature and the need for more robust research. it is important to acknowledge, however, that frequently, individual papers are written as practice reports and are not intended to be experimentally designed research reports.   the currency of the literature search may also be viewed as problematic. in considering the absence of the most current literature (beyond august 2014) as a potential limitation, a further search was undertaken to determine the impact on findings. the search was re-run in selected databases (cinahl, academic search premier, and psycinfo) to account for the most recent literature. after duplicates, theses, and book reviews were removed, 62 articles remained and were checked for relevance. four articles (kavšek, peklaj, & žugelj, 2016; lin, 2015; moreton & conklin, 2015; sego & stuart, 2016) were retrieved in full for further consideration. the study by kavšek et al. (2016) was found to be relevant to the review criteria; however, whilst the study used a control group, it was not randomized. it therefore provided no additional, higher quality evidence and the updated search has not altered the overall findings of this systematic review.   conclusion   this review has synthesized evidence on academic and information literacy support strategies for first year health science students. the majority of the studies included have reported positive outcomes following the implementation of such strategies, irrespective of their mode of delivery (embedded, integrated or adjunct). despite the contemporary view and rationalization for delivery of support strategies within curricula, approaches frequently continue to require extracurricular engagement. in terms of identifying the best practice for developing academic and information literacy skills, a major limitation to reaching an unbiased conclusion is that, typically, findings of individual studies are confounded by the absence of suitable control groups. without suitable control groups for comparison, learning outcomes for the student cohort of interest cannot necessarily be attributed to the intervention itself. this is not necessarily meant to be a criticism of individual papers, as frequently they are written as practice reports. of further note, articles in this review were not aimed at comparing different modes of delivery, and therefore, do not provide direct evidence on what mode is best. in this context, higher quality research is required to provide increased certainty on what strategies are most effective for developing information literacy and academic skills of first year health science students.   the problem of low student uptake by first year health science students with non-compulsory interventions, particularly when they are targeting those at risk, has also been identified. with this considered, and despite the absence of clear, unbiased evidence of superior learning outcomes for embedded support, there is a strong rationale for academic and information literacy support strategies for first year health science students to be fully embedded into the curriculum. strategies need to be fully inter-woven in a disciplinary context as a way of maximizing student uptake of the support strategy in a meaningful way to the discipline, and thereby provide an opportunity to impact learning.   due to lack of suitably designed research providing evidence on which mode of support delivery is most effective, this review has not been able to clearly achieve its aim of identifying the best practice for developing first year health science students’ information literacy and academic skills. however, when considering the nature of students accessing support, embedding strategies into the curriculum is recommended.   references   ambery, d., manners, j., & smith, k. (2005, november). embedded information literacy: a collaborative approach. paper presented at the 7th biennial conference of the association for academic language and learning, critiquing and reflecting: las profession and practice, australian national university, canberra. retrieved from http://www.aall.org.au/sites/default/files/las2005/ambery_manners_%26_smith.pdf   arpanantikul, m., thanooruk, r., & chanpuelksa, p. (2006). self-directed learning readiness, critical thinking skill, and self esteem in nursing students studying through problem based learning. thai journal of nursing research, 10(1), 59-72.    bailey, p., derbyshire, j., harding, a., middleton, a., rayson, k., & syson, l. (2007). assessing the impact of a study skills programme on the academic development of nursing diploma students at northumbria university, uk. health information and libraries journal, 24(s1), 77-85. http://dx.doi.org/10.1111/j.1471-1842.2007.00741.x   balch, w. r. (2001). study tips: how helpful do introductory psychology students find them?   teaching of psychology, 28(4), 272-291. http://dx.doi.org/10.1207/s15328023top2804_09   bearman, m., smith, c. d., carbone, a., slade, s., baik, c., hughes-warrington, m., & neumann, d. l. (2012). systematic review methodology in higher education. higher education research & development, 31(5), 625-640. http://dx.doi.org/10.1080/07294360.2012.702735   beatty, s., collins, a., & buckingham, m. (2014). embedding academic socialisation within a language support program: an australian case study. the international journal of the first year in higher education, 5(1), 9-18. http://dx.doi.org/10.5204/intjfyhe.v5i1.180   beccaria, l., kek, m., huijser, h., rose, j., & kimmins, l. (2014). the interrelationships between student approaches to learning and group work. nurse education today, 34(7), 1094-1103. http://dx.doi.org/10.1016/j.nedt.2014.02.006   betts, l. r., bostoek, s. j., elder, t. j., & trueman, m. (2012). encouraging good writing practice in first-year psychology students: an intervention using turnitin. psychology teaching review, 18(2), 74-81. retrieved from http://files.eric.ed.gov/fulltext/ej991412.pdf   birmingham, e., chinwongs, l., flaspohler, m. r., hearn, c., kvanvig, d., & portmann, r. (2008). first-year writing teachers, perceptions of students' information literacy competencies, and a call for a collaborative approach. communications in information literacy, 2(1), 6-24. retrieved from http://www.comminfolit.org/index.php?journal=cil&page=article&op=view&path[]=spring2008ar1&path[]=64   bonanno, h. (2002, july). standing the test of time: revisiting a first year diagnostic procedure. paper presented at the 6th pacific rim conference: first year in higher education, university of canterbury, christchurch, new zealand. retrieved from http://fyhe.com.au/past_papers/papers02/bonannopaper.doc   boruff, j. t., & thomas, a. (2011). integrating evidence-based practice and information literacy skills in teaching physical and occupational therapy students. health information and libraries journal, 28(4), 264-272. http://dx.doi.org/10.1111/j.1471-1842.2011.00953.x   brettle, a. (2003). information skills training: a systematic review of the literature. health information and libraries journal, 20(suppl. s1), 3-9. http://dx.doi.org/10.1046/j.1365-2532.20.s1.3.x   brettle, a., & raynor, m. (2013). developing information literacy skills in pre-registration nurses: an experimental study of teaching methods. nurse education today, 33(2), 103-109. http://dx.doi.org/10.1016/j.nedt.2011.12.003   brown, c. a., dickson, r., humphreys, a.-l., mcquillan, v., & smears, e. (2008). promoting academic writing/referencing skills: outcome of an undergraduate e-learning pilot project. british journal of educational technology, 39(1), 140-156. http://dx.doi.org/10.1111/j.1467-8535.2007.00735.x   bundy, a. (ed.). (2004). australian and new zealand information literacy framework: principles, standards and practice (2nd ed.). adelaide: australian and new zealand institute for information literacy. retrieved from http://www.caul.edu.au/content/upload/files/info-literacy/infoliteracyframework.pdf   cassar, a., funk, r., hutchings, d., henderson, f., & pancini, g. (2012). student transitions: evaluation of an embedded skills approach to scaffolded learning in the nursing curriculum. international journal of the first year in higher education, 3(1), 35-47. http://dx.doi.org/10.5204/intjfyhe.v3i1.102   chanock, k. (2013). teaching subject literacies through blended learning: reflections on a collaboration between academic learning staff and teachers in the disciplines. journal of academic language and learning, 7(2), a106-a119. retrieved from http://journal.aall.org.au/index.php/jall/article/viewarticle/256   chanock, k., horton, c., reedman, m., & stephenson, b. (2012). collaborating to embed academic literacies and personal support in first year discipline subjects. journal of university teaching & learning practice, 9(3), 1-13. retrieved from http://ro.uow.edu.au/cgi/viewcontent.cgi?article=1356&context=jutlp   chester, a., burton, l., xenos, s., elgar, k., & denny, b. (2013a). transition in, transition out: a sustainable model to engage first year students in learning. a practice report. the international journal of the first year in higher education, 4(2), 125-130. http://dx.doi.org/10.5204/intjfyhe.v4i2.174   chester, a., burton, l. j., xenos, s., & elgar, k. (2013b). peer mentoring: supporting successful transition for first year undergraduate psychology students. australian journal of psychology, 65(1), 30-37. http://dx.doi.org/10.1111/ajpy.12006   cook, d. a., & beckman, t. j. (2010). reflections on experimental research in medical education. advances in health sciences education, 15(3), 455-464. http://dx.doi.org/10.1007/s10459-008-9117-3   corbin, j., & karasmanis, s. (2010, june). evidence into practice: evaluation of an innovative information literacy model for first year health sciences students. paper presented at the 13th pacific rim first year in higher education conference, adelaide, australia. retrieved from http://hdl.handle.net/1959.9/526237   craig, a., & corrall, s. (2007). making a difference? measuring the impact of an information literacy programme for pre-registration nursing students in the uk. health information and libraries journal, 24(2), 118-127. http://dx.doi.org/10.1111/j.1471-1842.2007.00688.x   cranney, j., morris, s., spehar, b., & scoufis, m. (2008). helping first year students think like psychologists: supporting information literacy and teamwork skill development. psychology learning & teaching, 7(1), 28-36. http://dx.doi.org/10.2304/plat.2008.7.1.28   dubicki, e. (2013). faculty perceptions of students' information literacy skills competencies. journal of information literacy, 7(2), 97-125. http://dx.doi.org/10.11645/7.2.1852   edwards, j., & o'connor, p. a. (2011). improving technological competency in nursing students: the passport project. journal of educators online, 8(2), 1-20. retrieved from https://www.thejeo.com/archive/archive/2011_82/edwardsandoconnorpaperpdf   einfalt, j., & turley, j. (2009). developing a three-way collaborative model to promote first year student engagement and skill support. e-journal of business education & scholarship of teaching 3(2), 41-48. retrieved from http://www.ejbest.org/upload/ejbest_einfaltturley2009.pdf   elander, j., pittam, g., lusher, j., fox, p., & payne, n. (2010). evaluation of an intervention to help students avoid unintentional plagiarism by improving their authorial identity. assessment & evaluation in higher education, 35(2), 157-171. http://dx.doi.org/10.1080/02602930802687745   fallahi, c. r., wood, r. m., austad, c. s., & fallahi, h. (2006). a program for improving undergraduate psychology students' basic writing skills. teaching of psychology, 33(3), 171-175. http://dx.doi.org/10.1207/s15328023top3303_3   fenton-smith, b., & frohman, r. (2013). a scoping study of academic language and learning in the health sciences at australian universities. journal of academic language and learning, 7(1), a61-a79. retrieved from http://journal.aall.org.au/index.php/jall/article/view/253/164 fergy, s., heatley, s., morgan, g., & hodgson, d. (2008). the impact of pre-entry study skills training programmes on students' first year experience in health and social care programmes. nurse education in practice, 8(1), 20-30. http://dx.doi.org/10.1016/j.nepr.2007.02.003   fleming, s., & mckee, g. (2005). the mature student question. nurse education today, 25(3), 230-237. http://dx.doi.org/10.1016/j.nedt.2005.01.006   goldfinch, j., & hughes, m. (2007). skills, learning styles and success of first-year undergraduates. active learning in higher education, 8(3), 259-273. http://dx.doi.org/10.1177/1469787407081881   golding, j. m., wasarhaley, n. e., & fletcher, b. (2012). the use of flashcards in an introduction to psychology class. teaching of psychology, 39(3), 199-202. http://dx.doi.org/10.1177/0098628312450436   goldingay, s., hitch, d., ryan, j., farrugia, d., hosken, n., lamaro, g., nihill, c., & macfarlane, s. (2014). "the university didn't actually tell us this is what you have to do": social inclusion through embedding of academic skills in first year professional courses. the international journal of the first year in higher education, 5(1), 43-53. http://dx.doi.org/10.5204/intjfyhe.v5i1.194   griffiths, l., & nicolls, b. (2010). e-support4u: an evaluation of academic writing skills support in practice. nurse education in practice, 10(6), 341-348. http://dx.doi.org/10.1016/j.nepr.2010.02.005 gunn, c., hearne, s., & sibthorpe, j. (2011). right from the start: a rationale for embedding academic literacy skills in university courses. journal of university teaching & learning practice, 8(1), 1-10. retrieved from http://ro.uow.edu.au/jutlp/vol8/iss1/6   hafford-letchfield, t. (2007). factors affecting the retention of learners following the degree in social work at a university in the south-east of england. learning in health and social care, 6(3), 170-184. http://dx.doi.org/10.1111/j.1473-6861.2007.00159.x   hammick, m., dornan, t., & steinert, y. (2010). conducting a best evidence systematic review. part 1: from idea to data coding. beme guide no. 13. medical teacher, 32(1), 3-15. http://dx.doi.org/10.3109/01421590903414245   hammond, j. a., bithell, c. p., jones, l., & bidgood, p. (2010). a first year experience of student-directed peer-assisted learning. active learning in higher education, 11(3), 201-212. http://dx.doi.org/10.1177/1469787410379683   harris, a., & ashton, j. (2011). embedding and integrating language and academic skills: an innovative approach. journal of academic language and learning, 5(2), a73-a87. retrieved from http://www.journal.aall.org.au/index.php/jall/article/view/158   hegarty, n., & carbery, a. (2010). piloting a dedicated information literacy programme for nursing students at waterford institute of technology libraries. library review, 59(8), 606-614. http://dx.doi:10.1108/00242531011073137   hendricks, j., andrew, l., & fowler, a. c. (2014). the piloting of an academic literary education course (alec) to improve academic literacy of first semester undergraduate students in a western australian university. journal of nursing education and practice, 4(4), 19-27. http://dx.doi.org/10.5430/jnep.v4n4p19   hitch, d., goldingay, s., hosken, n., lamaro, g., macfarlane, s., nihill, c., ryan, j., & farrugia, d. (2012). academic skills and beyond: a resource based approach to support student success in higher education. journal of academic language and learning, 6(2), a29-a41. retrieved from http://www.journal.aall.org.au/index.php/jall/article/viewarticle/197   hooley, m., morrison, l., thomas, m., & marrs, l. (2011, june). enhancing the fyhe experience through professional partnership: embedding academic and library skills in a first-year psychology unit. paper presented at the first year in higher education conference, fremantle. retrieved from http://fyhe.com.au/past_papers/papers11/fyhe-2011/content/pdf/11e.pdf   hoyne, g. f., & mcnaught, k. (2013). understanding the psychology of seeking support to increase health science student engagement in academic support services. a practice report. the international journal of the first year in higher education, 4(1), 109-116. http://dx.doi.org/10.5204/intjfyhe.v4i1.149   igbo, i. n., straker, k. c., landson, m. j., symes, l., bernard, l. f., hughes, l. a., & carroll, t. l. (2011). an innovative, multidisciplinary strategy to improve retention of nursing students from disadvantaged backgrounds. nursing education perspectives, 32(6), 375-379. retrieved from http://journals.lww.com/neponline/abstract/2011/11000/an_innovative,_multidisciplinary_strategy_to.7.aspx   jorgensen, t. d., & marek, p. (2013). workshops increase students’ proficiency at identifying general and apa-style writing errors. teaching of psychology, 40(4), 294-299. http://dx.doi.org/10.1177/0098628313501037   kartika, a. (2008). study skills training: is it an answer to the lack of college students' study skills? international journal of learning, 14(9), 35-43. http://dx.doi.org/10.18848/1447-9494/cgp/v14i09/45480   kavšek, t., peklaj, c., & žugelj, u. (2016). information literacy training evaluation: the case of first year psychology students. journal of academic librarianship, 42(4), 293-299. http://dx.doi.org/10.1016/j.acalib.2016.06.008   kimmins, l., & stagg, a. (2009, 28–30 september). creating confidence: developing academic skills and information literacy behaviours to support the precepts of tertiary academic performance. paper presented at the 4th asia pacific conference on educational integrity (4apcei), wollongong, australia. retrieved from https://eprints.usq.edu.au/6150/1/kimmins_stagg_av.pdf   kirk, r. e. (1996). practical significance: a concept whose time has come. educational and psychological measurement, 56(5), 746-759. http://dx.doi.org/10.1177/0013164496056005002   kirkpatrick, d. (1996). great ideas revisited. training and development, 50(1), 54-59. retrieved from http://www.astd.org/td/ koufogiannakis, d., & wiebe, n. (2006). effective methods for teaching information literacy skills to undergraduate students: a systematic review and meta-analysis. evidence based library and information practice, 1(3), 3-43. http://dx.doi.org/10.18438/b8ms3d   lalor, j. g., clarke, m., & sheaf, g. (2012). an evaluation of the effectiveness of information literacy training for undergraduate midwives to improve their ability to access evidence for practice. nurse education in practice, 12(5), 269-272. http://dx.doi.org/10.1016/j.nepr.2012.06.005   lin, k. (2015). evaluating the effect of a clicker in an information literacy course for college nursing students in taiwan. cin: computers, informatics, nursing, 33(3), 115-121. http://dx.doi.org/10.1097/cin.0000000000000129   mandleco, b. l., bohn, c., callister, l. c., lassetter, j., & carlton, t. (2012). integrating advanced writing content into a scholarly inquiry in nursing course. international journal of nursing education scholarship, 9(1), 1-16. http://dx.doi.org/10.1515/1548-923x.2213   may, s., hodgson, d., & marks-maran, d. (2005, december). feet under the table: students' perceptions of the effectiveness of learning support provided during their first year of study on health and social care programmes. paper presented at the society for research into higher education conference, university of edinburgh, edinburgh. retrieved from http://www.leeds.ac.uk/educol/documents/151488.htm   mcmillan, l. r., & raines, k. (2011). using the "write" resources: nursing student evaluation of an interdisciplinary collaboration using a professional writing assignment. journal of nursing education, 50(12), 697-702. http://dx.doi:10.3928/01484834-20110930-01   mcwilliams, r., & allan, q. (2014). embedding academic literacy skills: towards a best practice model. journal of university teaching & learning practice, 11(3), 1-20. retrieved from http://ro.uow.edu.au/jutlp/vol11/iss3/8   moore, d., brewster, s., dorroh, c., & moreau, m. (2002). information competency instruction in a two-year college: one size does not fit all. reference services review, 30(4), 300-306. http://dx.doi.org/10.1108/00907320210451286   moreton, e. o., & conklin, j. l. (2015). closing the loop on nursing library instruction: using student performance to improve outcomes. medical reference services quarterly, 34(1), 113-121. http://dx.doi.org/10.1080/02763869.2015.986805   nishimura, r., wagner, j., & elliott, m. (2016). alternative indicators for the risk of non-response bias: a simulation study. international statistical review, 84(1), 43-62. http://dx.doi.org/10.1111/insr.12100   oermann, m. h., leonardelli, a. k., turner, k. m., hawks, s. j., derouin, a. l., & hueckel, r. m. (2015). systematic review of educational programs and strategies for developing students' and nurses' writing skills. journal of nursing education, 54(1), 28-34. http://dx.doi.org/10.3928/01484834-20141224-01   ooms, a., fergy, s., marks-maran, d., burke, l., & sheehy, k. (2013). providing learning support to nursing students: a study of two universities. nurse education in practice, 13(2), 89-95. http://dx.doi.org/10.1016/j.nepr.2012.07.011   palmer, l., levett-jones, t., smith, r., & mcmillan, m. (2014). academic literacy diagnostic assessment in the first semester of first year at university. the international journal of the first year in higher education, 5(1), 67-78. http://dx.doi.org/10.5204/intjfyhe.v5i1.201   pryjmachuk, s., gill, a., wood, p., olleveant, n., & keeley, p. (2012). evaluation of an online study skills course. active learning in higher education, 13(2), 155-168. http://dx.doi.org/10.1177/1469787412441298   rae, s., & hunn, m. (2015). assessing the impact of embedding online academic and information literacy resources into a first year business course. evidence based library and information practice, 10(4), 95-112. http://dx.doi.org/10.18438/b80c76   rolfe, v. (2011). can turnitin be used to provide instant formative feedback? british journal of educational technology, 42(4), 701-710. http://dx.doi.org/10.1111/j.1467-8535.2010.01091.x   rose, d., rose, m., farrington, s., & page, s. (2008). scaffolding academic literacy with indigenous health sciences students: an evaluative study. journal of english for academic purposes, 7(3), 165-179. http://dx.doi.org/10.1016/j.jeap.2008.05.004   sacre, s., & nash, r. (2010). assessing and developing academic literacy in first year health undergraduates. the international journal of learning, 17(9), 189-196. retrieved from http://ijl.cgpublisher.com/product/pub.30/prod.2933   sainani, k. l. (2010). making sense of intention-to-treat. pm & r, 2(3), 209-213. http://dx.doi.org/10.1016/j.pmrj.2010.01.004   salamonson, y., koch, j., weaver, r., everett, b., & jackson, d. (2010). embedded academic writing support for nursing students with english as a second language. journal of advanced nursing, 66(2), 413-421. http://dx.doi.org/10.1111/j.1365-2648.2009.05158.x   sego, s. a., & stuart, a. e. (2016). learning to read empirical articles in general psychology. teaching of psychology, 43(1), 38-42. http://dx.doi.org/10.1177/0098628315620875   shorten, a., wallace, c., & crookes, p. a. (2001). developing information literacy: a key to evidence-based nursing. international nursing review, 48(2), 86-92. http://dx.doi.org/10.1046/j.1466-7657.2001.00045.x   sikhwari, t. d., selepe, m. c., & maluleke, m. (2012). assessing the impact of a study skills workshop on the academic achievement of first-year nursing students at the university of venda, south africa. african journal for physical activity and health sciences, 18(supplement 2), 83-94. retrieved from https://www.ajol.info/index.php/ajpherd/article/view/83918   silburn, j., flack, m., bridgeman, s., & warwick, r. (2012). embedded academic language and learning support via an e-learning tool. journal of academic language & learning, 6(3), a36-a47. retrieved from http://journal.aall.org.au/index.php/jall/article/view/191/153   sopoaga, f., & van der meer, j. (2011). building a pacific health workforce in new zealand: initial findings from a transition project in first year health sciences at university. a practice report. the international journal of the first year in higher education, 2(2), 61-68. http://dx.doi.org/10.5204/intjfyhe.v2i2.88   steinert, y., mann, k., centeno, a., dolmans, d., spencer, j., gelula, m., & prideaux, d. (2006). a systematic review of faculty development initiatives designed to improve teaching effectiveness in medical education: beme guide no. 8. medical teacher, 28(6), 497-526. http://dx.doi.org/10.1080/01421590600902976   stevens, s., & miretzky, d. (2012). faculty's perceptions of students' characteristics: a for effort please. current issues in education, 15(2), 1-17. retrieved from https://cie.asu.edu/ojs/index.php/cieatasu/article/viewfile/875/330   turnbull, b., royal, b., & purnell, m. (2011). using an interdisciplinary partnership to develop nursing students' information literacy skills: an evaluation. contemporary nurse, 38(1/2), 122-129. http://dx.doi.org/10.5172/conu.2011.38.1-2.122   thies, l. c., wallis, a., turner, a., & wishart, l. (2014). embedded academic literacies curricula: the challenges of measuring success. journal of academic language and learning, 8(2), a43-a59. retrieved from http://journal.aall.org.au/index.php/jall/article/view/301/197   van moorsel, g. (2005). library-sponsored instruction improves core informatics competencies among allied health students: a research-based case study. journal of allied health, 34(3), 145-152.   walker, r., spronken-smith, r., bond, c., mcdonald, f., reynolds, j., & mcmartin, a. (2010). the impact of curriculum change on health sciences first year students' approaches to learning. instructional science: an international journal of the learning sciences, 38(6), 707-722. http://dx.doi.org/10.1007/s11251-009-9092-y   wallace, m. c., shorten, a., & crookes, p. a. (2000). teaching information literacy skills: an evaluation. nurse education today, 20(6), 485-489. http://dx.doi.org/10.1054/nedt.1999.0439   weaver, r., & jackson, d. (2011). evaluating an academic writing program for nursing students who have english as a second language. contemporary nurse, 38(1/2), 130-138. http://dx.doi.org/10.5172/conu.2011.38.1-2.130   webster, s., harding, t., robinson, j., yeboah, c., hutchison, c., & mountain, d. (2014). improving academic success: evaluation of the bn transition to clinical practice module. journal of nursing education and practice, 4(5), 95-106. http://dx.doi:10.5430/jnep.v4n5p95   weiner, s. a., pelaez, n., chang, k., & weiner, j. (2011). biology and nursing students' perceptions of a web-based information literacy tutorial. communications in information literacy, 5(2), 187-201. retrieved from http://www.comminfolit.org/index.php?journal=cil&page=article&op=view&path[]=v5i2p   wilkes, j., godwin, j., & gurney, l. j. (2015). developing information literacy and academic writing skills through the collaborative design of an assessment task for first year engineering students. australian academic & research libraries, 46(3), 164-175. http://dx.doi.org/10.1080/00048623.2015.1062260   winstone, n., & millward, l. (2012). the value of peers and support from scaffolding: applying constructivist principles to the teaching of psychology. psychology teaching review, 18(2), 59-67. retrieved from https://shop.bps.org.uk/psychology-teaching-review-vol-18-no-2-autumn-2012.html   wray, j., aspland, j., taghzouit, j., & pace, k. (2013). making the nursing curriculum more inclusive for students with specific learning difficulties (spld): embedding specialist study skills into a core module. nurse education today, 33(6), 602-607. http://dx.doi.org/10.1016/j.nedt.2012.07.009   xiao, j. (2010). integrating information literacy into blackboard: librarian-faculty collaboration for successful student learning. library management, 31(8/9), 654-668. http://dx.doi.org/10.1108/01435121011093423   yardley, s., & dornan, t. (2012). kirkpatrick's levels and education 'evidence'. medical education, 46(1), 97-106. http://dx.doi.org/10.1111/j.1365-2923.2011.04076.x   zhang, l., watson, e. m., & banfield, l. (2007). the efficacy of computer-assisted instruction versus face-to-face instruction in academic libraries: a systematic review. the journal of academic librarianship, 33(4), 478-484. http://dx.doi.org/10.1016/j.acalib.2007.03.006     appendix a examples of search strategies for two databases   ebsco databases:   search 1 “academic skills” or “academic literacy” or “academic writing” or “academic language” or “study skills” or “writing skills” or “learning skills” or “information literacy” or “graduate attributes” and “allied health” or nurs* or “psychology students” or physiotherapy or podiatry or midwifery or “occupational therapy” or “speech therapy” and student* or undergraduate* limiters: date range: 2000-current language: english   search 2 child* or preschool limiters: date range: 2000-current language: english   search 3 s1 not s2 search repeated for all listed ebsco databases, including the psychology and behavioral sciences collection.   proquest 5000: “academic skills” or “academic literacy” or “academic writing” or “academic language” or “study skills” or “writing skills” or “learning skills” or “information literacy” or “graduate attributes” and “allied health” or nurs* or “psychology students” or physiotherapy or podiatry or midwifery or “occupational therapy” or “speech therapy” and student* or undergraduate* not child* or preschool limiters: date range: 2000-current language: english appendix b studies reporting on academic and information literacy skill development strategies embedded into curriculuma authors cohort intervention outcome measure general findings organizational change corbin et al. (2010)   health science; australia. approximately 1700 students, 1000 usable pre-experience survey responses; 1085 post-experience surveys. online il modules offered in a core unit. pre/post difference on literacy quizzes; student and staff feedback on: modules; use of library services and resources; and library discussion board. improvement in il skills (mean score preto posttest 26% to 37%); just over ¾ responses were positive – positive student reaction to modules; positive staff feedback; organizational change, responding to stakeholder feedback, with the program implemented faculty-wide. behaviour boruff et al. (2011)   physical and occupational therapy; canada. 104 students. lecture, workshop, targeted assessment targeting il and evidence based practice. evaluation of learning via curriculum assessment task. 97% of students scored at least 80% on assessment. cassar et al. (2012)   nursing; australia. number of enrolled students not reported. data for 2009 & 2010 students, interviews: 35 students; focus groups: 11 students; surveys: 544 students. scaffolded learning approaches in a core, foundational unit including an online learning directory, a numeracy package, and il and essay writing sessions aligned to assessments. mixed methods evaluation for two cohorts in consecutive years. results on numeracy and writing tasks over time; program evaluation via semi-structured interviews/focus groups; usefulness surveys. writing: grade levels improved over the semester; numeracy improved; majority (85%) of students agreed embedding skills was essential or useful. chester et al. (2013a); chester et al. (2013b) psychology; australia. 241 students, 231 provided data. peer mentoring program, transition in transition out model (into first year; out of university), supporting academic (particularly learning approaches) and psychosocial adjustment. focus of paper on on-campus fy students at a one university. academic performance (compared to previous cohort; pre/post self-report measures for students’ learning approach (approaches and study skills inventory for students [assist]); program evaluation questionnaire. higher percentage of students achieved grades > 60%; learning approaches significantly changed (p < .001) to more strategic, deeper (moderate effect size) and less surface based (small effect size); majority enjoyed program (70%) and perceived it to positively influence their academic work (59%). cranney et al. (2008)   psychology; australia study 2: 1st implementation, 533 students (383 analyzed); 2nd implementation 561 students. (study 2): team work skill development program built around a group assessment task. upon reflection, specific strategies were implemented in the 2nd roll out to improve the program. student rating of group process (productivity and cohesiveness); evaluation of the program by survey and focus group; group project assessment mark. marks indicated that groups functioned effectively (mean score 87%); higher marks were predictive of group cohesiveness (β = .159, p < .05), students responded positively to the program. fallahi et al. (2006) psychology; usa. 109 students, 78 at follow-up. lessons involving writing, peer editing, and extensive feedback. 5 assignments graded and separately rated for basic writing skills (grammar, writing style, writing mechanics, and referencing) by blind assessor. improvement on 4 writing skill domains (p < .001), immediately for referencing and by the fourth paper for other writing skills. rose et al. (2008) health science; australia. 8 (fy) indigenous students (also, 25 other students in higher years of study not considered in this review). scaffolding of as in the curriculum where teachers guide students through reading, critical understanding, and note taking. pre/posttesting of as. significant overall improvements in academic literacy skills (for fy bachelor cohort). walker et al. (2010)   health science; new zealand. 1,841 students enrolled in the subject, 705 included completing both pre/posttests; 599 subjects from a previous cohort [control group] in equivalent subject who completed test. study and learning approaches in a new curriculum including self-directed learning modules; more formative assessment tools; website for the discussion of difficult concepts. assist administered pre/post-intervention and compared to a control group. correlations for exam scores and learning approaches determined. post-assessment, students were taking a deep and more strategic approach to their studies (p < .001) and surface approach to a lesser extent; students adopted a deep approach, to a greater extent, compared to controls (p < .001); high performance on final exam was significantly correlated with a surface approach (r = .16, p < .0001). winstone et al. (2012) psychology; england. 125 students (47 in the intervention group; 78 from a prior (control) cohort). 20 as tutorials delivered using a scaffolded approach (compared to a control group who did not receive a scaffolded approach to support as). improvement in essay grades compared between groups; student feedback rating usefulness. increase in average essay grades for the scaffolded group was higher (2%) than for the non-scaffolded group (p = .02); the scaffolded approach was rated significantly more useful than the non-scaffolded approach. learning mandleco et al. (2012)   nursing; australia. 176 students (82 providing data for learning and 47 on confidence). unit consisting of 3 modules embedded in the curriculum on writing in the disciplines; writing to learn and writing across the curriculum. grammar test (clips) pre/post-intervention; writing confidence rated monthly for 4 months. significant improvement in 12 of 26 categories on the clips test; mean scores improved each month for writing confidence from a mean score of 3.48 to 4.02 out of 5 over the 4 month period. pryjmachuk et al. (2012) nursing; england. 260 students (63 complete sets of survey data; 12 interviewees). online, blended learning unit delivering 8 as/il skills topics incorporating compulsory learning tasks. pre/post-surveys evaluating knowledge, skills (via confidence), and attitude; focus groups interviews. improvement in knowledge (median score 58% compared to 71%, p < .001) and confidence for a variety of study related skills; the strategy was reported to be “fit for purpose.” van moorsel (2005)   occupational, physical and respiratory therapy; usa. 189 students (179 usable data pairings); control group of 64 physician assistant students (48 usable data pairings). computer literacy for healthcare professionals (3 hrs/wk for 7 wks.).   pre/posttest (post-intervention and at 5 wk follow-up) measuring acquisition of literature searching skills; change in student confidence for literature searching. difference in skill between groups, mean (95% ci): post = 3.42 (2.27 to 4.56), post at 5wks = 4.93 (3.79 to 6.07) (calculated from mean (sd); van moorsel, 2005, table 4, column 2 & 3). within group improvement in skill at both posttest intervals (p < .001) and improved confidence for literature searching (p < .001). wallace et al. (2000); shorten et al. (2001)   nursing; australia. 138 interventions (55 sets of complete pre/post data); control (non-program) 88, 200-level health and behavioural science (including nursing) students.   structured il program involving lecture and laboratory/tutorial sessions taught collaboratively with three learning activities and three related assessment tasks in the context of nursing.   pre/post-program questionnaires assessing application of il and citation skills; confidence with il skills.   post-program, student scores superior to non-program students for locating and interpreting resources (68% compared to 27%, p < .001) but not for bibliographic citation skills (93% compared to 90%, p = .70); within group improvement for searching and locating resources and interpreting bibliographic citations (p < .001). students’ il confidence higher than “non-program” for 7 of 10 skills (p < .001). reaction         arpanantikul et al. (2006) nursing; thailand. 136 students (129 qualified to participate and 124 with usable data pairings). problem based learning (pbl) method for course delivery (utilizing pbl skills, searching skills, concept mapping, and learning plans). pre/post-program surveys: self-directed learning readiness scale; self esteem scale; critical thinking. pre-program > 95% scored in the high category for self-directed readiness and self-esteem; significant improvement in self-directed readiness (mean increase 4%, p < .01); post-program, 88% scored moderate and 11% low for critical thinking. beatty et al. (2014) health science; australia. 111 students (51 students identified at academic risk and targeted for support). english language and as embedded in a core unit; scaffolded assessment; contextualized examples; in-class collaborations between discipline staff and learning advisers; opportunity for support. at risk students recommended to seek help. generic unit feedback form; number of students who accessed support following referral. > 79% agreement that the unit had improved communication and writing skills; 89% reported having a clear understanding of what was required in the unit; limited success in encouraging at risk students to seek additional support (10 of 51 at risk students sought support). beccaria et al. (2014)   nursing; australia. 301 students at 2 campuses (92 responses for pre-survey 102 responses for post-survey). 3 targeted learning and teaching activities embedded in a core unit, focusing on group work skills. pre/post-surveys; students’: (1) perception of group work; (2) approaches to learning (two-factor study process questionnaire). significant increase to both surface (p = .02, effect size = .04); and deep (p = .04, effect size = .2) approaches to learning. a surface learning approach more likely to be associated with a discomfort for group work. hegarty et al. (2010)   nursing; ireland. 350 students (number in intervention for fys not specified). il training workshops across the 4 year degree. 2 hour session for fys delivered in a research skill module. online survey on reaction to the program. fy: 98% agreement that workshop was practical and useful; 100% valued the program as good or above. mcmillan et al. (2011)   nursing; usa. 46 students. 1 hour il in-class session for: writing tutorials; classroom peers for feedback; student tutors at the writing centre for editing and proofreading assignment.   writing assignment resource evaluation (author developed tool) for effectiveness of activities to improve student writing, learning and quality of work. ≥ 70% agreed that il was instructive; time with tutors valuable; class time for writing assignment & peer review beneficial; ≥ 50% agreed draft revisions with tutor contributed to learning; working with writing centre improved understanding of writing. san miguel et al. (2013) nursing; australia. 176 students with low english language proficiency (2 cohorts over consecutive years). diagnostic screening and specific tutorial program with collaboratively developed materials and assessment tasks to respond to student diversity targeting students identified. focus group: themes identified & clustered into 5 major categories. students felt more comfortable and confident; deeper explanations of information; tutor is very important; learning about reading and writing; mixed responses regarding how helpful this was; moving on helped them to adjust. thies et al. (2014)   health science; australia. 1,152 students.   various as embedded across 3 core health units involving: online modules to improve study techniques; learning resources for researching and report writing; use of feedback to help students be reflective learners. mixed methods approach looking at reaction to program. student surveys and focus groups; staff interviews and surveys; use of a student as reflective tool. findings for 1 of the 3 units (hbs109): 82% accessed at least one module; 76% found them very helpful or quite helpful; staff report students having good or excellent understanding of as. webster et al. (2014)   nursing; australia. 400 students (380 at follow-up). transition to clinical practice module program that includes a 5 week compulsory transition module in a core fy unit that includes english literacy, library research skills, and clinical competencies. pre/post-intervention questionnaire to evaluate knowledge and confidence for transition into clinical practice. 82% rated program better than expected; 96% indicated it had helped to develop skills; significant increase (p < .01) in knowledge and confidence ratings post-program in most areas except confidence with as and accessing support. astudies categorized according to highest level of learning outcome (kirkpatrick, 1996). key: as = academic skills; ci = confidence interval; fy = first year; il = information literacy; sd = standard deviation appendix c studies reporting on academic and information literacy skill development strategies integrated into curriculuma authors cohort intervention outcome measure general findings organizational change hooley et al. (2011)   psychology; australia. 80 (off campus) students (20 completed the questionnaires). as and library staff resided as guest lecturers in the lms answering questions; help sheets provided. organizational change; student academic performance and retention; students surveyed on usage, intention to use, attitudes, and awareness. students reported: increased awareness of resources and likelihood of use (100%); skills support more accessible (90%); helpful (85%). 10% decline in retention (95% vs. 85%); fewer failures (1.4% vs. 4.6%); essay grades unchanged. extension of program in organization including: earlier presence of advisors on lms; online module developed as an early assessment. behaviour betts et al. (2012)   psychology; england. 116 students; 71 completed the evaluation. lecture on academic writing; presentation on referencing conventions; report submitted via turnitin with assistance provided on interpreting the originality report. plagiarism rates in subsequent assessments; online survey on effectiveness and learning experience. reduction in occurrences of plagiarism (7) compared with the previous cohort (1, no p value reported); majority agree turnitin reassured them their work was their own (68%) and that it helped to understand plagiarism (58%). cranney et al. (2008) (study 1) psychology; australia. 752 psychology students. five il skills modules (via lms platform). (study 1): pre/post il tests; questionnaire and focus groups.   within group improvements for students’ pre/posttest il scores (p < .05). higher average posttest performance significantly associated with higher grades (p < .01). program viewed positively for attitudes, usefulness and being liked. griffiths et al. (2010) nursing; england. 17 students. e-support4u: a series of relevant e-tivities, closely related to the students’ academic course, scaffolding phases of academic writing. reflective-practice assignment results; online self-assessment quiz; lms access data. 100% pass rate in assignment; 100% positive response to online academic support; perceived barriers related to: accessibility, finding time for computer use on wards; limited it skills; the placing of the e-support4u as a stand-alone module rather than having it embedded. rolfe (2011)   bioscience; england. 76 students (80 control subjects from previous cohort).   using turnitin; instruction on interpreting originality reports. student draft and final essays analyzed against criteria; staff/student questionnaires; small sub-group interviews.   no significant between group difference in overall plagiarism. significant: reduction in plagiarism due to poor paraphrasing (22 incidences compared to 7, p < .05); increase in students not providing in-text citations (25 to 45 students, p < .05); poorer essay performance (62 compared to 53%, p < .001). turnitin use rated positively; staff reported it raised plagiarism awareness. wray et al. (2013) nursing; england. 384 students (2 consecutive cohorts) with spld, 300 completed the evaluation survey); (control group: previous cohort). nine study skills sessions delivered in a core unit. progression data; feedback questionnaire; time to disability registration. progression rates 25% higher than previous cohort with spld and comparable to peers with no learning difficulty; sessions viewed positively; students contacted disability services 4–6 weeks earlier than previous cohort. learning brettle et al. (2013)   nursing; england. 93 students with 77 randomized attending program (intervention n = 40; comparison group, n=37 and 55 students at follow-up). randomized controlled trial comparing an online il tutorial to a f2f il tutorial. search histories scored using a validated checklist; skill retention measured at 1 month. no between group differences for posttest scores (p < .05); both groups improved (ol mean scores from 3% to 18%, p < .001; f2f 4% to 22% p < .001); skills retained 1 month later. craig et al. (2007) nursing; england. 76 students (29 completing both pre/posttest; 9 interviewed). il program 3 x 3 hr f2f practical sessions to 8 small groups – whole group induction and information searching paired pre/posttest for il skills and confidence; interviews on confidence outcomes. 72% improved their skill scores; 97% self-rated confidence as “fairly” or “very” confident compared with 76% at pretest. hendricks et al. (2014)   nursing, paramedicine; australia. 214 nursing (n = 143) and paramedicine (n = 71) students. academic literacy information course: ten learning modules related to referencing, sourcing information, essay writing, and paraphrasing. paired pre/posttesting related to understanding of academic literacy concepts. significant improvement (7.6%, p < .001) in understanding of academic literacy concepts. lalor et al. (2012)   midwifery; ireland. 63 fy (from a total of 108) students (49 fy students with complete data). in the first year: 4 hrs of computer based sessions focussed on il skills. pre/posttest on search histories rated by researchers as poor, fair or good. improvement in il skills with only 3% (from 79% at baseline) remaining “poor” posttest. xiao (2010)   nursing; usa. 356 students from 2005 – 2008 fy cohorts. il integrated into unit via: 1 hr library orientation; self-paced online tutorials; online resources; and librarian support on discussion forum in lms. 2008 pre/post true/false test on il and apa referencing knowledge. 2005-2008 post-intervention evaluation survey.  pre/posttest (2008): students improved understanding for some il concepts; greater confidence in completing research assignment. high levels of agreement that course was helpful and improved skills. reaction elander et al. (2010)   psychology; england. 364 students (111 completed survey). education regarding authorship, writing, and avoidance of plagiarism integrated into existing modules. delivered at various institutions prior to assignment submission. student authorship questionnaire (18 item likert response); questionnaire re: usefulness of the intervention; focus groups. confidence in writing, understanding of authorship, knowledge to avoid plagiarism, and top-down approaches to writing increased significantly, with greatest improvements for fy undergraduates; 86% believed it helped them avoid plagiarism; 66% believed it helped them write better; changed understanding about authorial identity and academic writing. turnbull et al. (2011) nursing; australia. 174 students. online tutorials – 6 modules covering il, academic integrity, and referencing. online survey. 83% agreed to being more confident in using library resources post-tutorial. weiner et al. (2011)   nursing; usa. 60 students (48 fy students completed the modules & questionnaire). (biology student outcomes not reported in this review). compulsory online il tutorial for nursing students that formed an assignment in their course. module completion; self-report questionnaires. nursing students: 97% completed all modules; 75% liked the intervention as they learned important information; 91% of these indicated they did not know the information prior (p = .005). astudies categorized according to highest level of learning outcome (kirkpatrick, 1996). key: as = academic skills; f2f = face-to-face; fy = first year; il = information literacy; lms = learning management system; spld = specific learning difficulties.     appendix d studies reporting on academic and information literacy skill development strategies adjunct to the curriculuma authors cohort intervention outcome measure general findings organization hoyne et al. (2013)   health science; australia. 76 students identified as “at risk” using a post-entrance literacy assessment. support program, for students diagnosed at risk, in reading and writing delivered by the academic enabling support centre. pass/fail rates for a core literacy unit. reduction in fail rate by 50% for at risk students; (10% compared to 21%) when support was made compulsory. behaviour bailey et al. (2007)   nursing; england. 46 students identified from a diagnostic essay (at risk). four remedial workshops: 2 on il skills; 1 on essay writing; 1 on referencing outside of class tutorial times. focus groups; questionnaire; assignment grade. 50% of students identified as needing additional support accepted it (only 1 student attended all 4 sessions). students attending at least one workshop improved their academic grades in next assignment. students increased their confidence and perceived an improvement in il and referencing. balch (2001)   psychology; usa. 114 students. series of study tips sheets. students rated each tip for helpfulness and amount of use on a scale from 0-10. most helpful tips were for lecture notes; degree-of-use ratings were correlated with overall helpfulness r(113) = .42, p < .001 ratings, but not with course performance; no correlation between overall degree-of-use ratings and course performance, r(113) =  –.05, p > .10). fleming et al. (2005)   nursing; ireland. 67 mature age students invited with 44 attending the intervention and 33 returning questionnaires. one week, 2 part, pre-course program: 1) socialization to university life; 2) study skills. questionnaires about program; progression rates. better progression through course of mature students who attended compared to those who did not p < .05; high agreement for: program achieving objectives; successful and very helpful in preparing for the course; giving confidence and information needed to start course. golding et al. (2012)   psychology; usa. 415 psychology students with 60% fys. flash cards (both written and computer generated) to prepare for exams. flashcard survey; exam results. students who used flashcards on all exams performed better than other students (mean ±sd) (41.34±5.16) compared to those who used flashcards on one exam (38.67±4.86, p = .0001); and two exams (38.57±5.21, p = .002); and no exams (40.03±5.23, p = .043). palmer et al. (2014)   nursing; australia. 569 students (513 completed preliteracy screening test – 92 targeted (at risk) based on test results). early feedback on academic literacy skill levels from unit tutors and provision of non-compulsory learning support (from the learning development unit in the university) targeted at students who scored in the lowest band (<9) on measuring the academic skills of university students (masus). all students could attend. masus scores classified into 3 bands; course grade. of those in the lowest band (<9): 77% improved their masus score; 40% shifted to the highest band (>12); and 73% achieved a pass grade. salamonson et al. (2010); weaver et al. (2011)   nursing; australia. 106 esl students (with low to medium elas score <19) randomly allocated into intervention group: n = 59 (28 attending); and usual support n = 47). 4-day targeted academic learning and writing support workshops; one-on-one sessions providing individual feedback on academic writing.   assignment results; open-ended questionnaire (pre/post) about assistance they wanted and perceptions of the program and support; informal feedback sought by group discussion in final intervention session. better assignment scores for intervention group (mean ±sd = 70.8±6.1) compared to control group (58.4±3.4, p = .002) and to non-attendees (48.5±5.5, p = .001). provision of individual feedback identified as a key benefit. sikhwari et al. (2012) nursing; south africa. 33 students. one day study skills workshop developed by the student counselling and career development unit, focused on motivation, time management, learning skills, concentration, exam techniques. the learning and study strategies inventory (lassi) as a pre/posttest; academic results between semesters and years. on average, lassi scores poorer at follow-up (p < .025); increase in academic achievement between semesters (mean increased scores ±sd) (7.10±4.21, p = .000 and between years 5.53± 4.57, p = .000). silburn et al. (2012)   health science; australia. 86 (29 internal, 57) students. self-selected into 1 of 4 study groups. external, synchronous academic language and learning support (4 1 hour online workshops in lms) on essay writing and referencing compared to internal – no academic language and learning support; external – no online academic language and learning support; external asynchronous. online survey; assessment marks pre/postintervention. synchronous online academic language and learning students: 90% of positive responses, most negative responses related to technology; highest mean grades (8% or greater) post-intervention compared to other 3 groups (adjusted for baseline scores, p < .05). sopoaga et al. (2011) health science; new zealand. 39 pacific islander students. 6 week structured program: peer educators met with students weekly; general guidance provided, academic support, information about support services and university systems. comparison of academic results for attending and non-attending students. superior grades for attending students: a or b grades for 39% compared to 0% of non-attending students; fail grades for 36% compared to 44% of non-attending students. learning brown et al. (2008)   health science; england. 57 students; 20 accessed intervention. learning to reference project (6 online, archived audio-visual learning presentations). pre/posttesting of knowledge related to referencing, attitudes, and computer skills irrespective of module completion (n = 52 at follow-up); paired data analysis. no change in performance on a referencing quiz; significant increase in referencing skill confidence on 4 of 7 items (p < .005). < 36% accessed the resources. edwards et al. (2011) nursing; usa. 90 students.   passport project for nursing success: 7 online learning modules in the lms to improve student orientation, computer literacy, research and apa format knowledge. pre/posttest computer literacy survey; qualitative evaluation. mean computer literacy score increase on posttest of 17%; positive responses in terms of value of modules, some negative responses relating to access to academic advisor and lack of time to complete passport. jorgensen et al. (2013)   psychology; usa. 105 students; 58 students attended workshops and 47 students acted as controls. students self-selected into groups. participant groups attended 1 of 3, 20-30 minute workshop topics on grammar, mechanics, or referencing. control group attended one discussion session about apa style rules and received a handout. paired pre/posttesting; pretest, items relating to error recognition on topics delivered. follow up at 2 and 7 days and 2-4 weeks.   workshop participants improved more (mean difference±se = 9.91±0.69) than handout-only (control) participants (3.19±1.17, p < .0001). workshop participants improved for all topics (grammar, p < .0001; mechanics, p < .0002; references p < .0001) and retained their proficiency in follow-up tests. control group participants did not improve (p = .15). kartika (2008)   psychology; indonesia. 155 students. study skills training embedded in the university’s orientation program involving 6 x 3 hr sessions during a 14 week semester. pre/posttesting using the study skills inventory (ssi). significant improvement in ssi item scores for concentration (p < .05); exam preparation (p < .001). no improvement for time management and writing/note taking skills. reaction         hammond et al. (2010) physiotherapy; england. 90 students (3 cohorts over 3 years (26 in 2003; 39 in 2004; 25 in 2005). pal – voluntary, timetabled sessions, encouraging cooperation, team work, and active problem solving through student-directed activities around class content. questionnaire evaluating student perception and satisfaction with pal sessions. overall student agreement that pal improves social aspects of learning but does not improve study skills or assignment preparation. generally low attendance with at >3 sessions 20 – 59% (2003-5). igbo et al. (2011) nursing; usa. 105 (76% high risk) students (3 cohorts over 3 years (27 in 2004; 39 in 2005; 39 in 2006). f2f study skills, critical thinking, communication, professional socialization, medical terminology, and career coaching activities 2 hours, 1 afternoon/week for the first academic year. progression/retention rates of students. student feedback; reporting of grade point average increase and confidence levels medical terminology pre/posttests. 76.8% average retention rate over the 3 years. feedback indicates that the program was helpful and students were appreciative of it. astudies categorized according to highest level of learning outcome (kirkpatrick, 1996). key: esl = english as second language; f2f = face-to-face; fy = first year; il= information literacy; lms = learning management system; pal = peer assisted learning.       research article   “don’t make me feel dumb”:  transfer students, the library, and acclimating to a new campus   matthew harrick outreach and education librarian brooklyn college library brooklyn college – cuny brooklyn, new york, united states of america email: mharrick@brooklyn.cuny.edu   lee ann fullington health and environmental sciences librarian brooklyn college library brooklyn college – cuny brooklyn, new york, united states of america email: lafullington@brooklyn.cuny.edu   received:  11 oct. 2018                                                                  accepted: 10 may 2019      2019 harrick and fullington. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29512     abstract   objective – this qualitative study sought to delineate and understand the role of the library in addressing the barriers transfer students experience upon acclimating to their new campus.   methods – a screening survey was used to recruit transfer students in their first semester at brooklyn college (bc) to participate in focus groups. the participants discussed the issues they encountered by answering open-ended questions about their experiences on campus, and with the library specifically.   results – transfer students desired current information about campus procedures, services, and academic support. they often had to find this information on their own, wasting valuable time. students felt confused and stressed by this process; however, strategic library involvement can help alleviate this stress.   conclusion – involving the library more fully in orientations could ease students’ confusion in their transitional semester. students desired local knowledge, and the library is in a key position to disseminate this information.     introduction   transfer students are a distinct population of a college’s student body. at brooklyn college (bc), we define a transfer student as “an applicant who has attended any college, university, vocational, or religious institution after graduating from high school or receiving a ged” (bc, 2019a). a 2017 national student clearinghouse research center report found that “out of 852,439 students who first enrolled at a community college, 31.5 percent (268,749) transferred to a four-year institution within six years” (p. 9).   in our university system, the city university of new york (cuny), which is the largest urban university in the country, transfers account for 35.3 percent of new students in fall 2017: 25,879 out of 73,375 (city university of new york, 2018). bc, like all of the 24 colleges in the cuny system, is a commuter institution, with a large population of first-generation students. at bc, transfer students are the largest group of new students for the same period, 43 percent, or 2,096 out of 4,883 (bc, 2018), and they feature prominently in the college’s strategic plan (bc, 2019b).   this growing population of students is unique from traditional first-year students in many ways, not least of which in the challenges and barriers they experience when transferring from one institution to another.  a commonly experienced phenomenon is “transfer shock”, where transfer students “suffer a severe drop in performance upon transfer” (hills, 1965, p. 202), and during which they “obtain lower average grades immediately after they transfer than they received in junior college” (hills, 1965, p. 204). hills’ research set the tone for transfer student research (cejda 1994; cejda, kaylor, & rewey, 1998; ishitani, 2008).   in addition to transfer shock, researchers have identified other transfer student barriers and challenges in recent studies. some common findings include: poor communication between community colleges and four-year schools (chin-newman & shaw, 2013; owens, 2010); the lack of transfer-specific orientations (owens, 2010; chin-newman & shaw, 2013; townsend & wilson, 2006; townsend, 2008); transfer students do not attend orientations (grites, 2013); weak transfer student advisement (chin-newman & shaw, 2013; owens, 2010); difficulty with academic and social integration (grites, 2013; townsend, 2008; townsend & wilson, 2006); and weak personal and institutional supports (chin-newman & shaw, 2013; grites, 2013; owens, 2010). grites (2013) believed transfer shock would continue alongside these other challenges.   there are many academic and social groups on college and university campuses invested in identifying and easing transfer student barriers and challenges. not surprisingly, academic libraries are also investigating ways they can help, too. furthermore, at a commuter campus like ours, in a system of non-residential colleges, the library becomes a place to meet, to work, or to socialize between classes, and is not just the place to check out books. as students do not live in campus housing, transfer students at a commuter college will not have the same opportunities or mechanisms for acclimating to a new college as those at a residential campus. this research study is significant as it listened directly to transfer students to incorporate their voices in the library’s mission to provide effective services and programming to ease student transition. our findings could help other academic librarians see the importance of student voices, and offers ideas for successful intervention.   literature review   research interest in this population from the academic library perspective is growing. staines (1996) discovered that many transfer students at four-year institutions returned to use their community college libraries, as they felt more comfortable with the resources and the space. whang et al. (2017) highlighted transfer student specific orientations involving multiple departments.   another focus—library outreach efforts for transfer students—also identifies orientations as important (cox & johnson, 1992; kraemer et al., 2004), as well as a few more strategies: a collaboration between an academic library and its local writing center in the creation of a research and writing course for transfer students (tipton & bender, 2006) and personal librarian programs (coats & pemberton, 2017; lafrance & kealey, 2017; macdonald & mohanty, 2017). mcbride, gregor, and mccallister (2017) and sandelli (2017) reinforced the importance of librarians’ commitment to working with other groups on campus, participating in orientations, establishing relationships with feeder school librarians, and providing resources tailored for transfer students.   a common thread running through research regarding transfer students is that academic libraries recognize their position on campus as possibly strategic to easing or eliminating some of the barriers and challenges transfer students encounter. however, many academic librarians developed programming and outreach without direct input from transfer students themselves. or, if they had transfer student input, their programming was limited in scope or scalability. our current research confirmed that we at bc were on the right track in our initial efforts to connect with transfer students, and brought to light issues we had not yet considered. the past research also showed that it is of the utmost importance to talk with transfer students directly, and to apply their comments as holistically as possible to library outreach and programming, involving as many campus constituents as possible, and looking for longevity. our project complements and extends recent research. first, it finds its place among other projects, such as richter-weikum and seeber’s (2018) study, which used focus groups and interviews to talk with transfer students at urban institutions; roberts, welsh, and dudek’s (2019) statewide survey of academic librarians about their perceptions of academic library outreach and instruction for transfer students; and heinbach, fiedler, mitola, and pattni’s (2019) mixed-method research that reoriented librarians’ approaches to supporting transfer students by focusing on their strengths. second, our research differentiates itself from these studies, and expands them, in that we build on the knowledge that transfer students need support from the library, and use focus groups and interviews to collect data that would show us specifically how the library can better integrate itself into the transfer student experience.   aims   this article describes an exploratory qualitative study that seeks to uncover and understand the issues facing transfer students as they acclimate to bc. bc is a public commuter college that is part of a large university system with mechanisms already in place for transferring within the system: students either earn credits at the community colleges to fulfill requirements prior to transfer for completion of bachelor’s degrees, or complete associate’s degrees and subsequently enroll for bachelor’s programs. thus, this study also attempts to pinpoint what the library can do to assist transfer students as they navigate these processes. the data we gathered and subsequently analyzed addressed the following research questions:   what barriers do transfer students encounter in their transition to brooklyn college? what services do transfer students desire most? how can the library support transfer student acclimation to campus?   situating the study   our study took place during the 2016-2017 academic year. of the 4,699 new students in fall 2016, 2,169 were new transfer students (bc, 2016). the following spring, 1,300 of the total 1,367 new students were transfers (bc, 2017).   several offices on our campus, including the transfer evaluations office, are dedicated to improving the transfer student experience (including retention and graduation rates). another example is transfernation, which seeks to ameliorate some of the aforementioned issues by providing a semester-long onboarding program for a select, voluntary group of transfer students. at past transfer orientations, librarians promoted library services and resources, and distributed transfer student library bulletins, our publication detailing important library information.   in 2014, the college revamped transfer orientation programs, and the library’s presence was reduced to tabling at campus resource fairs. though the library had an interest in conducting research to better understand the needs of our transfer students, with this shift in orientation programming, we needed to upgrade this interest to a priority. we wanted to reestablish and subsequently build a more comprehensive library presence at orientations. to do this, it was necessary to collect evidence to share with other campus groups that transfer students would benefit by a more robust library presence at orientation. the best way to gather this evidence was to talk directly with transfer students: learn what they need, what they already know, and how the library can help. hence, the heightened need for our study.   methods   we chose focus groups as our data collection method to bring together first semester transfer students in small groups to encourage interaction between the participants, with the intention that the students would share their stories and comment on each other’s experiences (glitz, 1997; von seggern & young, 2003; widdows, hensler, & wyncott, 1991). we conducted focus groups in both fall and spring semesters to capture any differences in the overall experiences transfer students had depending on which semester they entered the college.   to recruit students for the study, we created a short screening survey using surveymonkey and emailed students the survey via the transfer student electronic mailing list. we also hung posters with the survey link around campus, and distributed flyers at transfer student events. we wanted to recruit both cuny and non-cuny transfer students in the focus groups to potentially compare their experiences. the survey was open for one month. we received 77 responses in the fall and 62 responses in the spring semester.     we selected participants using convenience sampling and contacted all survey respondents who met the inclusion criteria of being a first semester undergraduate transfer student. we offered $25 target gift cards as participation incentive. we emailed qualified survey respondents a doodle poll to select dates to meet for the focus groups. we offered 5-8 sessions each semester, at times when students would likely be available, and aimed to have groups of 4-6 students. we then assigned respondents to focus groups and sent confirmation and reminder emails to mitigate any drop outs (billups, 2012). ultimately, we scheduled ten focus groups (five per semester).   we identified an alternative data collection method in case only one participant showed up on any particular date. turning someone away simply to adhere to our research design meant we would miss an opportunity to talk to a student. so, we would use an in-depth interview as a backup data collection method. though we would miss the opportunity for interaction effects that focus groups afford, an interview would still allow us to include these student perspectives.    our diverse pool of respondents reflected our student population, and we captured a range of experiences from students of varied ethnic and racial backgrounds, genders, and ages, as shown in table 1. we had 21 participants in total, between all the focus groups.   using braun and clarke’s (2014) notion of crafting questions that “open out” (p. 117) and encourage discussion that allows for agreement and disagreement, we created a set of open-ended questions for our semi-structured focus group guide (see appendix). our questions were sequential, moving from broader transfer process experiences to their current bc experiences. as librarians interested in hearing about the transfer experience, and developing services to assist transfers with acclimation, we created several questions about library specific experiences. we wanted to keep the questions open-ended and avoid leading questions (such as, “would a tour or orientation help?”) as we did not want to introduce confirmation bias into the study. thus, we phrased our library-related focus group questions to allow for more organic discussion and consideration. the broader purpose of our study was to explore their experiences rather than confirm any notions we held about the library’s role in their acclimation. though our study did have a confirmatory element, because we needed evidence to argue for reintroducing the library to orientation, it was not the focus of the research. seeing as we are librarians conducting research about the library, we could not however, avoid researcher bias as we both moderated and administered the focus groups.   for the actual focus groups, with participant consent, the co-investigator took notes, handled the consent forms, and recorded each session for transcription purposes. the primary investigator (pi) moderated all sessions. dividing the labor ensured consistency: each instance of participant intake and its associated tasks was uniform, and one moderator prevented deviation from the question guide.                 we analyzed our transcripts using the thematic analysis method (braun & clarke, 2006), which provided flexibility to allow us to understand students’ stories. using an inductive coding approach, we identified themes and patterns in the data. we outsourced the transcription of our recordings. while we awaited the transcriptions, we reviewed our notes from the focus groups to start brainstorming potential themes. we also identified common themes to use to inform our initial codes. to test these codes, we each coded one transcript separately, then compared the application of our codes to this transcript. we had near perfect agreement in how we determined and applied our codes and felt confident coding the remaining transcripts independently. we then each hand-coded a set of the transcripts. after this first round of coding, we discussed sections in our respective transcripts that were unclear or difficult to categorize and coded these sections together. the co-investigator then used nvivo to code all hand-coded transcripts in order to break our transcripts up into data extracts, which we then grouped into themes. for this article, we analyzed only data excerpts related to the library and its role in the transfer process.   table 1 screening survey data group 1 (fall) race/ethnicity gender age previous institution/(associate’s degree) student 1a hispanic male 21-29 cunyaa student 1b black female 21-29 cuny 2yr student 1c white female 21-29 cuny student 1d middle eastern/north african female 18-20 cuny 2yr group 2 (fall)         student 2a black female 18-20 non-cuny student 2b asian male 21-29 non-cuny student 2c black female 18-20 non-cuny student 2d other female 30-39 cuny 2yr student 2e indian female 18-20 non-cuny student 2f white female 18-20 non-cuny group 3 (fall)         student 3a hispanic and african genderqueer 21-29 cuny group 4 (fall)         student 4a black female 30-39 cuny aa student 4b white female 18-20 non-cuny group 5 (spring)         student 5a hispanic female 18-20 cunyaa group 6 (spring)         student 6a hispanic male 21-29 cuny aa student 6b black female 21-29 non-cuny group 7 (spring)         student 7a black female 18-20 non-cuny group 8 (spring)         student 8a white female 21-29 cunyaa student 8b not disclosed genderqueer 18-20 cuny aa group 9 (spring)         student 9a black male 21-29 cuny group 10 (spring)         student 10a hispanic female 21-29 non-cuny     results   our participants discussed early decision-making steps at their previous institutions, contacting offices at our campus before registering, their first day on campus, and beyond. our data analysis identified patterns that coalesced in two major themes: 1) transfer process bureaucracy, and 2) acclimating to a new campus. the first theme refers to the steps the student takes: deciding to transfer, applying for admission, credit evaluation, and the processes they go through once they have enrolled and arrived on campus. the second theme encapsulates students’ experiences, and the steps they take to get used to the new campus (e.g., programs they may attend, including orientation). it is beyond the scope of our article to discuss the entire transfer process, so we focused on library related sub-themes. patterns we identified from the data show that the library plays two key roles for transfer students: library as a provider of local, campus specific knowledge and library as place.   library as place   because bc is a commuter school, and our students do not live in dormitories, they do not experience immersive campus life, including opportunities to learn about local culture. the library is a space that is neither home nor dorm (places that may be full of distractions or offer no study spaces), but a place where students can be productive and collaborative (regalado & smale, 2015). the library was not the top reason to transfer to bc (cost and proximity to home were top two); however, several students factored the library into their decisions to enroll at bc. cuny libraries have reciprocal access agreements, and some students had already used the bc library before transferring, and envisioned themselves being productive there. student 9a summarized his impression of the library as a place: "i could really get some work done here. it's clean and it's quiet. you're near people. i'm going to say it was a factor.”   according to student 8b:   for me, it's really about spots. that's why i picked bc in the first place. i went to the library with my friend. we were like we need to come here. it is a big place where i study. i could live in the library.   student 3a visited our library when they interned at a nearby high school, and already felt a familiarity when considering transferring here:   i love the space. it's quiet... that's a big thing for me, having quiet spaces, having endless amount of resources both digital and print and computers… i felt accommodated by the librarians. the library was a big reason why i moved to transfer. i saw and i felt that i would do very well here just based on the library, the space.   the library as space is also a consideration for transfer students without opportunities to come to the library prior to transferring. for these students, being overwhelmed and confused pointed to a need for the library to intervene positively in their acclimation. for example, student 1b commented that “certain areas [of the library] i kind of had to learn on my own. sometimes i would want to be seated at a certain area, so i wouldn't get distracted by what was going on in the library.” student 5a also felt lost, offering: “if there was a tour to the library, then i wouldn't have been confused. i literally roamed around the second floor for a good half an hour looking for a space.” the library does not offer tours, so students must discover our spaces on their own, possibly adding stress to their acclimation.   our focus group participants told us tales of frustration, bewilderment, confusion, and a lack of information. this gap in local knowledge can be visualized as transfer fog.  though most of the students figured out processes and asked clarifying questions, the fog could lift sooner if we disseminate core information earlier in their transition to campus. staines (1996) found that students returned to previous college libraries because they were familiar with them. however, when students in our system draw on the knowledge they have of how things work at their previous institutions, this may complicate their acclimation because the way things work at a different library (cuny or non-cuny) is not necessarily indicative of how things work at bc.   rather than relying on students’ self-efficacy, we can intervene in the transfer process by participating in orientation and finding alternative ways to reach them prior to their first time in the library.   acquiring local knowledge   many students expressed anxiety, unease, and frustration with feeling uninformed about campus services and procedures. at bc, the library houses the college’s main computer labs. the it-related issues transfer students encounter often come to the attention of the staff at the library’s service desks. our reference desk is split into two service points: reference and computer sign-in. though we have tried to differentiate the service points with signage, the blending of services at service points confuses students.   of particular confusion is our idiosyncratic computer sign-in system. at other cuny campuses, students choose their own computer and log in with college credentials.  our system requires our staff to assign computers to users when they hand in their ids, and for student 2b, this was confusing: “my first day i just sat-i didn't know the policy. i just sat at a random computer on the first floor...then someone was like, ‘oh, you're in my seat.’” student 7a was confused, too: "why do i have to sit at a specific computer? i had free choice at [my previous institution] of computers for was whichever one was open.”   student 1d got help:   [i]n (sic) [my previous institution] sometimes you don't need to sign in, you can just go and sit and sign in if there's not a lot of people. i thought that was how it was here. then the [librarian] was like, "you have to give your id--" he was very nice about it. i felt really dumb.   as previously noted, some students rely on their experiences from previous institutions as a problem-solving technique to use services at our library, but this technique is not necessarily sustainable. student 3a was familiar with the cuny interlibrary loan system, and applied that knowledge at bc, stating, “for a while, i actually used [my previous] library’s database for books and sent it here or picked it up there.”   library service desk staff can ease student transition. staff members at our circulation and reserves desks are often the first people students ask for help, even if the question is not library related. student 4a related that the library was the first place on campus where she got answers to her questions about college procedures:   when i first got [to bc] with my id card, it was one of the guys from the circulation desk who explained what the card gets you. he went into explaining to me about the validation. being new, i didn't know what it meant. he kind of explained certain things to me, gave me a little good direction.   simultaneously, she learned about the need for memorizing her student id number, as students use this number to log in to numerous library services. however, at her previous cuny institution, this number was printed on her id card, and the college used a different system for logging into services.   our campus receives many transfer students in both fall and spring semesters. new students will have similar questions in both semesters, and staff may (erroneously) assume that spring students are up to speed on how things work, not realizing they could be new first-year or transfer students. student 5a recounted:   i didn't know where all the computers were. so i asked one of the librarians at the reference desk, and she just handed me this big laminated sheet. they didn't explain anything, so i had to ask them, "can i use any of these computers just to do homework or not?"   students mentioned that an informational tour or talk at orientation would have been helpful. student 1d notes, “i kind of wish i went to the orientation. i feel like it would've been way more helpful.” other participants agreed:   student 2e: they could just talk about it. because i didn't know about the library until one of my friends brought me in and told me about it. i think if they spoke about it at orientation, it would kind of give the students a heads up.   student 2f: because everybody kind of walks into the library… not really knowing how to use the library resources, yet knowing that they're there. so maybe having an information session at orientation…maybe the library could get up there and talk a little bit about the process of getting integrated into the library.   students would also like to meet librarians at orientation. student 10a felt that a librarian should be present at each orientation tour because she “would have wanted to see a familiar face. then you could feel comfortable going back to them for any more questions.”   for students unconvinced that orientation or a tour would be beneficial, framing either as a moment to get insider or local knowledge may be effective to entice students to participate:   student 9a: i'm kind of iffy about [orientation] because as a student, you know that the library is an essential part in your education. you have to take that step and walk in the library and talk to the librarians. you don't have to have people tell you, "this is a librarian. this is what they do."   student 8b: i do these things because...i hate feeling like i don’t know what’s going on. i think the basics are where you start. once you know the basics, it's easier to explore the college and the library. but all that stuff comes from knowing the basics and being able to walk into the library and not feel like, where the hell am i going?       discussion                 the library is a central building on our campus and serves a role similar to a student center as a place to be between classes, where students come to work and socialize. the library is also the college’s largest computer lab. addressing, during orientation, the myriad roles the library plays would help alleviate transfer students’ frustration and cut through the transfer fog. students who used our library prior to transferring noted they either felt a connection with the library space, or with librarians and library staff. these students knew the spaces they wanted to use and felt that they would work well here. highlighting library spaces could promote the library to transfer students who did not have the opportunity to use the bc library while attending their previous college. a tour during orientation could also address local knowledge deficiencies.   students’ near universal concerns were about the lack of local knowledge: using the library’s computer labs, printing, study space locations, checking out books, using group study rooms, and the importance of the student id. the library could transmit this local knowledge to students if we were to participate more fully in transfer student orientations. by addressing seemingly little issues, we can save students time and ease their transition, confusion, and frustration.    though students indicated they want librarians to present at orientation, providing an overview of library resources and services in print form, and having informal conversations with students at resource fair events, can be enough to get students started. the main issue is to make sure all transfer students receive at least the core information about library and it services.       conclusion/recommendations   the academic library is strategically situated to play a prominent role in easing transfer student acclimation to a new campus. however, because each group of transfer students is unique, not all outreach programs and services will work for all groups. there is no one common transfer experience, and therefore no one-size-fits-all solution. it is incumbent upon librarians to meet with and interview transfer students at their institutions to discover their specific and unique needs, challenges, and successes; to share the results of these discussions with constituents on campus invested in easing the transfer experience; and to develop holistic and diverse programming and partnerships to engage with transfer students at multiple points. we found through our focus group discussions that there are particular needs the library could address, as evidenced in the discussion section. although we found no single solution to address transfer student needs, what is evident is that using focus groups or interviews is useful to uncover unmet needs, and to use students’ feedback in creating new or improved library outreach initiatives to support their acclimation. however, we can suggest using such methods to collect information from students in order to create supports that address the idiosyncrasies of our institution.    libraries are in prime positions to connect transfer students with library services and resources at their new institutions, and with other campus academic and social services, if librarians know what the specific issues are.  the library must complement other services and offices on campus, such as the transfer student center, registration and advisement, and peer mentoring/college transfer group, in its outreach and programming for transfer students.   as with richter-weikum and seeber (2018), townsend (2008), and townsend and wilson (2006), and, where researchers conducted interviews with students, our research confirmed that the value of actually talking to the students cannot be underestimated. engaging with the students allowed us to delve deeper into understanding their needs and determine ways to meet them. a related, unexpected outcome from the focus groups was the students’ enthusiasm and desire to help future transfer students, to share their experiences to try to improve the process. student 4b proclaimed: “i'm glad i got to come and have somebody hear about my experience.... i was talking about what will help prevent some people from having the same bad experiences that we've been having.”   this passion to help would not have been nearly as evident had we relied on collecting survey responses. this is one of the benefits of qualitative research: from the tones of their voices, and from their facial expressions, we could see that our participants were angry about their transfer experiences, and they wanted to participate in our research as a way to help future students.  when we deviated from our method and conducted three in-depth interviews (some students did not show up to their scheduled focus group), our nimble design allowed us to talk to these students who wanted to share their stories, rather than turning them away because they had missed their focus group appointments. the positive feedback we received, and the trust we gained from students because we were willing to listen, showed that our engagement with these students was meaningful, and our attempts to improve their experiences were appreciated.   as cox and johnson (1992) found, their library orientation workshops for transfer students were deemed useful by participants. building on this, our participants also spoke of a desire for better library representation at orientations. thus, our first next steps are local to our campus. first, we will approach student affairs to re-embed the library into orientation so we can impart local knowledge to incoming transfer students. we will offer a library tour to introduce students to several librarians, friendly faces they could feel comfortable approaching for help. as noted earlier, framing the tour as a way to learn insider tips for success, and about library resources and services, may increase participation. we will also modify our virtual tour, the library online orientation program, also known as the loop (georgas, 2014), and its attendant quiz, to deploy to new transfer students each semester. perhaps most importantly, library managers need to remind public services staff to treat every semester as each student’s first, and to be gentle and helpful to everyone to alleviate the transfer fog that happens in both fall and spring.   when the above are in motion, we will look to partner with other librarians in our university system to find collaborative ways to make the entire transfer process smoother. bc is a popular transfer destination from 2-year cuny schools, and our university system-wide library association is our mechanism to work with librarians from these 2-year schools. we can use this association to begin a wider conversation about easing transfer student stress with library support. we could consider coordinating instruction between the junior and senior colleges by creating a new roundtable for interested librarians or as part of the current instruction roundtable. ultimately, we are in a unique, and uniquely strong position, in that instead of one library tackling the transfer student problem once they arrive here, we can intervene earlier by collaborating with librarians at schools where transfer students transfer from, so that, when they arrive here, a solid foundation for transfer student success has already been laid.   references   billups, f. d. (2012). conducting focus groups with college students: strategies to ensure success. retrieved from eric database. (ed573089)   braun, v., clarke, v., & terry, g. (2014). thematic analysis. in p. rohleder & a. c. lyons (eds.), qualitative research in clinical and health psychology (pp. 95-114). new york, ny: palgrave macmillan.   braun, v., & clarke, v. (2006). using thematic analysis in psychology. qualitative research in psychology, 3, 77-101. https://doi.org/10.1191/1478088706qp063oa   brooklyn college. (2019a). transfer students faq. retrieved 1 january 2019 from http://www.brooklyn.cuny.edu/web/admissions/transfer/faq.php   brooklyn college. (2019b. strategic plan 2018-2023. retrieved 1 january 2019 from http://www.brooklyn.cuny.edu/web/about/administration/president/strategicplan/2018-2023.php   brooklyn college. (2018). fall 2017 enrollment snapshot. retrieved 1 october 2018 from http://www.brooklyn.cuny.edu/web/abo_misc/171120_enrollment_snapshot_fall_2017.pdf   brooklyn college. (2017). spring 2017 enrollment snapshot. retrieved 1 october 2018 from http://www.brooklyn.cuny.edu/bc/offices/avpbandp/ipra/enrollment/spring%202017%20enrollment%20snapshot.pdf   brooklyn college. (2016). fall 2016 enrollment snapshot. retrieved 1 october 2018 from http://www.brooklyn.cuny.edu/bc/offices/avpbandp/ipra/enrollment/fall%202016%20enrollment%20snapshot.pdf   cejda, b. (1994). reducing transfer shock through faculty collaboration: a case study. community college journal of research and practice, 18, 189-199. https://doi.org/10.1080/1066892940180207   cejda, b., kaylor, a., & rewey, k. (1998). transfer shock in an academic discipline: the relationship between students’ majors and their academic performance. community college review, 26(3), 1-13. https://doi.org/10.1177/009155219802600301   chin-newman, c., & shaw, s. (2013). the anxiety of change: how new transfer students overcome challenges. journal of college admission, 221, 14-21. retrieved from https://www.nacacnet.org/news--publications/journal-of-college-admission/   city university of new york. (2018). total new undergraduate and graduate students by full-time/part-time attendance, gender and college: fall 2017. retrieved 1 october 2018 from https://www2.cuny.edu/about/administration/offices/oira/institutional/data/historical-student-data-book-tables-all-semesters/   coats, l. r., & pemberton, a. e. (2017). transforming for our transfers: the creation of a transfer student services librarian. reference services review, 45(3), 485-497. https://doi.org/10.1108/rsr-11-2016-0079   cox, j., & johnson, r. (1992). transfer students in the library: the forgotten population. research strategies, 10(2), 88-91.   georgas, h. (2014). the implementation of an independent and self-paced online library orientation for freshman students and the use of sakai as a quiz management system (qms). college & undergraduate libraries, 21(1), 56-75. https://doi.org/10.1080/10691316.2014.877737   glitz, b. (1997). the focus group technique in library research: an introduction. bulletin of the medical library association, 85(4), 385-390. retrieved from https://www.ncbi.nlm.nih.gov/pmc/journals/72/   grites, t. (2013). successful transitions from two-year to four-year institutions. new directions for higher education, 162, 61-68. https://doi.org/10.1002/he.20057   heinbach, c., fiedler, b., mitola, r., & pattni, e. (2019). dismantling deficit thinking: a strengths-based inquiry into the experiences of transfer students in and out of academic libraries. in the library with the lead pipe. retrieved from http://www.inthelibrarywiththeleadpipe.org   hills, j. (1965). transfer shock: the academic performance of the junior college transfer. the journal of experimental education, 33(3), 201-215. https://doi.org/10.1080/00220973.1965.11010875   ishitani, t. (2008). how do transfers survive after “transfer shock”?: a longitudinal study of transfer student departure at a four-year institution. research in higher education, 49(5), 403-419. https://doi.org/10.1007/s11162-008-9091-x    kraemer, e. w., keyse, d. j., & lombardo, s. v. (2004). beyond these walls: building a library outreach program at oakland university. the reference librarian, 39(82), 5-17. https://doi.org/10.1300/j120v39n82_02   lafrance, h., & kealey, s. b. (2017). a boutique personal librarian program for transfer students. reference services review, 45(2), 332-345. https://doi.org/10.1108/rsr-10-2016-0066   macdonald, a., & mohanty, s. (2017). personal librarian program for transfer students: an overview. reference services review, 45(2), 346-354. https://doi.org/10.1108/rsr-10-2016-0071    mcbride, k. r., gregor, m. n., & mccallister, k. c. (2017). bridging the gap: developing library services and instructional programs for transfer students at appalachian state university. reference services review, 45(3), 498-510. https://doi.org/10.1108/rsr-10-2016-0067   national student clearinghouse research center. (2017). tracking transfer: measures of effectiveness in helping community college students to complete bachelor’s degrees. retrieved from https://nscresearchcenter.org/signaturereport13/   owens, k. (2010). community college transfer students’ adjustment to a four-year institution: a qualitative analysis. journal of the first-year experience & students in transition, 22(1), 87-128. retrieved from https://sc.edu/about/offices_and_divisions/national_resource_center/publications/journal/index.php   regalado, m., & smale, m. a. (2015). “i am more productive in the library because it’s quiet”: commuter students in the college library. college & research libraries, 76(7), 899-913. https://doi.org/10.5860/crl.76.7.899   richter-weikum, e., & seeber, k. (2018). library experiences of transfer students at an urban campus. student success, 9(2), 63-71. https://doi.org/10.5204/ssj.v9i2.411   roberts, l., welsh, m., & dudek, b. (2019). instruction and outreach for transfer students: a colorado case study. college & research libraries, 80(1), 94-122. retrieved from https://crl.acrl.org   sandelli, a. (2017). through three lenses: transfer students and the library. reference services review, 45(3), 400-414. https://doi.org/10.1108/rsr-10-2016-0074   staines, g. m. (1996). moving beyond institutional boundaries: perceptions toward bi for transfer students. research strategies, 14(2), 93-107.   tipton, r. l., & bender, p. (2006). from failure to success: working with under-prepared transfer students. reference services review, 34(3), 389-404. https://doi.org/10.1108/00907320610685337    townsend, b. k., & wilson, k. (2006). "a hand hold for a little bit": factors facilitating the success of community college transfer students to a large research university. journal of college student development, 47(4), 439-456. https://doi.org/10.1353/csd.2006.0052   townsend, b. k. (2008). “feeling like a freshman again”: the transfer student transition. new directions for higher education, 2008(144), 69-77. https://doi.org/10.1002/he.327   von seggern, m., & young, n. (2003). the focus group method in libraries: issues relating to process and data analysis. reference services review, 31(3), 272-284. https://doi.org/10.1108/00907320310486872   whang, l., tawatao, c., danneker, j., belanger, j., edward weber, s., garcia, l., & klaus, a. (2017). understanding the transfer student experience using design thinking. reference services review, 45(2), 298-313. https://doi.org/10.1108/rsr-10-2016-0073   widdows, r., hensler, t., & wyncott, m. (1991). the focus group interview: a method for assessing users’ evaluation of library service. college and research libraries 52, 352-359. https://doi.org/10.5860/crl_52_04_352    appendix focus group questions   introduction:  you’ve all been invited here today for this focus group to discuss your experiences transferring into bc.    what was the most difficult part of transferring into bc? what made this frustrating or difficult? what was the easiest part of transferring? what did you enjoy about the process? think about the offices and services you may have used throughout the transfer process. which service/office was most useful to you? a.       which service/office was least useful or most frustrating? b.       how would you improve this service? as librarians, we want to be of service to students and improve your experiences at bc. have you had any interactions with librarians, or have you taken a class that came to the library? please elaborate.      a.       can you think of (or describe the) ways the librarians can assist with the transfer process? b.       what role does the library play, if any, in your transfer experience? is there anything you wish you knew before you came to campus and started your bc career? is there anything else you’d like to add about your transfer experience?     editorial   journal update from eblip10   lorie kloda editor-in-chief associate university librarian, planning and community relations concordia university montreal, quebec, canada email: lorie.kloda@concordia.ca      2019 kloda. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29591     in this issue, i would like to welcome three new members of the eblip editorial team. jane schmidt was recently appointed as associate editor (classics & reviews). stephanie krueger is now one of two associate editors (evidence summaries) along with christine wissinger. and heather healy is the new lead copyeditor, taking over from richard hayman, who served in the role for several years and is stepping down at the end of his second term. also stepping down is heather pretty, who served as associate editor (evidence summaries) for the past six years.   in a few days, the 10th international evidence based library and information conference (eblip10) will be held at the university of strathclyde in glasgow (see the conference website https://eblip10.org/). at the conference, as is the tradition, i will present an update on the journal to interested conference delegates about the journal’s readership, usage, and impact, as well as some recent developments. here are some of the highlights.   the eblip editorial team is composed of the editor-in-chief, five associate editors, the production editor, the indexing coordinator, and the communications officer. in addition, the journal benefits from a group of 5 editorial advisors, all previous members of the editorial team, 70 peer reviewers, 11 copyeditors, 20 evidence summary writers, and 4 writing assistants. the journal is in its 14th year of publication (this issue marks the 54th issue). the journal publishes several types of papers, including editorials, research articles, review articles, evidence summaries, classics, using evidence in practice, commentaries, news, and features.   the average issue of eblip includes 11 peer reviewed articles. the acceptance rate for research and review articles between 2016 and 2018 was 44%. in that time period the average duration from submission to acceptance was two months, and the average time to publication, four months.   eblip has always been indexed in several librarianship and general databases, and was recently included in emerging sources citation index. as an open access journal, eblip is indexed in the directory of open access journals at the article level.   as of june 2019, the journal has 6,914 registered users. the journal sees an average of 265 visits (sessions) per day and an average of 8,057 visitors per month (5,882 unique visitors). visits to the journal’s website mainly come from the united states (34%), followed by the united kingdom (13%), canada (11%), and australia (7%). to view the journal’s most cited papers, see the google scholar page for evidence based library and information practice at http://bit.ly/ebliptopcited.   recently, the journal migrated to the online journal system (ojs) 3 platform and underwent a redesign. more improvements and initiatives are underway. to stay informed about the journal, register on the journal website as a reader, and follow the @eblip twitter account.     research article   a comparative analysis of the use of github by librarians and non-librarians   mark e. eaton assistant professor / reader services librarian kingsborough community college city university of new york brooklyn, new york, united states of america email: mark.eaton@kbcc.cuny.edu   received: 12 june 2017                                                              accepted: 6 apr. 2018      2018 eaton. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29291     abstract   objective – github is a popular tool that allows software developers to collaborate and share their code on the web. librarians have adopted github to support their own work, sharing code in support of their libraries. this paper asks: how does librarians’ use of github compare to that of other users?   method – to retrieve quantitative data on github users, we queried the github apis (application programming interfaces). by assembling data on librarians’ use of github, as well as on a comparison group, we provided preliminary comparisons of these two samples. we analyzed and visualized this data across a number of variables to offer salient insights as to how librarians compare to randomly selected github users.   results – librarians regularly use a more diverse range of programming languages than the comparison group, hinting at a broad range of possible uses of code in libraries. while the librarians’ sample group did not demonstrate statistically significant differences from the comparison group on most measures of activity and popularity, they scored significantly higher in reach and productivity than the comparison group. this could be due to librarians’ greater longevity on github, as well as their greater investment in github as a tool for sharing.   conclusion – our data suggest that librarians are actively building their libraries with code and sharing the results. while it was unclear whether librarians were more active or popular on github than the comparison group, it was clear that they demonstrated statistically significant outperformance in terms of reach and productivity. to explain these findings, we hypothesized that librarians’ embrace of github is in line with widely held values of “openness” in the library profession.     github is “the biggest revelation in my workflow … since i started writing code.” (falster, as qtd. in perkel, 2016, p. 127)   introduction   github is a well-known web-based code sharing platform that has recently exploded in popularity. its functionality underpins cooperation by developers on many software projects by allowing programmers to share and promote their work. for scholars, github is an interesting space to examine web collaboration and cooperative coding.   github is built on git, a prominent software version control system that allows many geographically dispersed contributors to collaborate on a project asynchronously. git is a command line tool that, among other things, allows for versioning, branching, and merging of projects’ histories. github adds value to git by providing a web interface to a great deal of git’s functionality, and by adding additional social and workflow features, thereby making git-based projects much more accessible to the public and to other programmers.   librarians who code presumably benefit from github in many of the same ways as other programmers. it makes librarians’ programming work accessible and open to the public, and it improves workflows via robust web tools. we can postulate that github is an effective tool for collaboration amongst coding librarians in much the same way as it is for other users.   aims   the goal of this paper is to answer the following question: how does librarians’ use of github compare to that of other users?   this question leads to more specific inquiries about librarians’ use of github. are librarians more or less active on github than other users? are they more or less popular? are they prolific producers of code? are they more or less connected with other developers on the site? do they tend to use the same programming languages as the larger community, or is their code clustered idiosyncratically in specific languages? are their repositories well regarded?   we can begin to work toward answers to these questions using data from the github apis, or application programming interfaces. this paper describes how we queried the github apis to assemble quantitative data about github use. we describe how we handled the data to make useful visualizations. our analyses offer preliminary conclusions focused on our research question.   our aim is to provide evidence-based insights about how librarians use github. this is important, as librarians’ software development work arguably plays a key role in the future of the profession. this paper makes a small step toward providing these important insights.                this paper does not explain how to use git, github, or the github apis. some understanding of github and apis is presumed. for those looking for an introduction to git, blischak, davenport, and wilson (2016), perez-riverol et al. (2016), and bell and beer (2015) have provided detailed how-tos. there are also plenty of interactive tutorials online that teach how to use git and github, such as lord (2014), lord (2015), and github (2017a). others resources provide detailed introductions on how to use the github apis (dataquest, 2017; github, 2017b). this paper does not review these topics.   literature review   scholarly literature on the use of github in librarianship is almost nonexistent. if we broaden our focus beyond github, we find that the scholarly literature on librarians’ coding practices is only slightly less sparse. de la cruz and hogan (2016) describe it as a “limited literature” (p. 251). one might assume that this is because there are not many librarians writing code. however, the existence of prominent librarian professional associations that focus on technology skills—such as code4lib and the library and information technology association (lita)—suggests otherwise.                unfortunately, this “limited literature” does not offer many insights about why librarians are writing and using code. some authors have speculated. marshall (2015) suggests that librarians may use code “to solve problems, to improve or enhance existing applications, or to create new ones to meet specialized needs” (p. 25). de la cruz and hogan (2016) describe several use cases, including facilitating librarians’ collaboration with it departments, building websites, and wrangling data. yelton (2015) suggests that “data import, export, and cleanup; expanded reporting capability; and patron facing services” (p. 2) are possible reasons to write code in libraries. others suggest new, underexplored opportunities for code in librarianship, such as organizing hackathons (davis, 2016). yet these suggestions do not exhaust the possibilities. somewhat more cynically, stuart (2011) describes programming as an “obligation” (p. 43) for contemporary librarians; de la cruz and hogan (2016) imply the same when they point out that programming skills are often requirements of candidates looking for library jobs.   some of the best insights into why librarians use code can be gleaned from andromeda yelton’s (2015) extensive report on learning to code for librarians. her interviews with over 50 librarians who have written code for their work give us important perspectives on librarians’ coding practices. upon reading through her examples, we get a sense of the preoccupations and interests of librarians who are either learning to program or who are actively using code. likewise, enis (2013) helpfully provides real world examples of individual librarians’ coding practices. while the samples of librarians assembled by yelton and enis do not appear to be controlled or random, they do provide useful and interesting narrative examples of why librarians might use code.   apart from librarianship, scholarship on github is a new and growing field. in 2015, grier argued that “scholars have drawn only modest conclusions from their study of the github database” (p. 116). longo and kelley identified a similar lack of scholarship in 2016. however, the study of github is changing quickly. grier’s description may have been accurate at the time he was writing; however, git and github scholarship have developed substantially since then.   much of the existing work on github is concentrated in the field of computer science. given that git and github are often taken for granted among professional software developers, it is perhaps unsurprising that computer science is well ahead of other disciplines in producing scholarship on github. there are many examples of this type of literature: arora, goel, and mittal (2016) analyze syntactic and semantic conflicts that arise when code is being written by multiple developers; blincoe, sheoran, goggins, petakovic, and damian (2016) measure the level of influence of popular users on github; hu, zhang, bai, yu, and yang (2016) also attempt to measure influence; jiang et al. (2017) look at forking practices; kalliamvakou et al. (2016) examine the quality of data that can be gathered from the github apis; lima, rossi, and musolesi (2014) parse interactions on github to analyze social activity; mcdonald, blincoe, petakovic, and goggins (2014) consider factors associated with successful open source projects on github; and yan, wei, han, and wang (2017) analyze the use of github for blogging.   it is clear that github is important to software developers. however, davis (2015) argues that these tools are also useful and important to librarians. while github was not designed explicitly for librarians, it is a tool that has increasingly proved useful in many disciplines, including librarianship. longo and kelley (2016) argue that github’s popularity has spread to a range of diverse fields. as github spreads across disciplinary boundaries, scholarship on github begins to appear in many fields.   in other words, we are not alone in attempting to broaden the scholarly conversation on github beyond computer science. the current growth of scholarly literature on github mirrors the ongoing growth and expansion of github’s user base. while git has existed since 2005, the subsequent growth of github has fueled the expansion of git-based workflows beyond git’s core developer community: “as the uses and users of github move beyond its original core community of software developers, the present and potential impact on fields such as social knowledge creation, open science, open collaboration, and open governance warrants consideration” (longo & kelley, 2016, p. 617). recent work has been done on the use of github in fields as diverse as computational biology (blischak et al., 2016; perez-riverol et al., 2016) and public administration (longo & kelley, 2016; mergel, 2015), among others.   while much of this scholarship is fairly recent, version control and code sharing did not begin with github. christopher kelty (2008) demonstrates that a culture of sharing code has a very long history in computing. a number of version control tools have been used in open source projects since the early 1990s, and many of these were popular in their respective day. however, unlike github, many of these early tools were only suitable for small development teams (hu et al., 2016). moreover, they have largely waned in popularity recently as git and github have become ascendant.   our methodological approach—gathering quantitative data from the github api—is not entirely unique. other scholars have done api-based work on github, including some large scale data gathering by jiang et al. (2017); as well some interesting work by authors who have supplemented their api-based work with qualitative surveys (blincoe et al., 2016; mergel, 2015); and in several interesting projects presented at conferences, described by hu et al. (2016).   beyond these nearest methodological cousins, others do similar work while opting not to use the github api. these latter studies instead rely on third party data-gathering tools or collections to access github data. for example, mcdonald et al. (2014) draw from the api using a tool called gitminer; lima et al. (2014) do similar data mining using a tool called the github archive; and a number of projects use ghtorrent, which is a mirror of the github api (blincoe, 2016; kalliamvakou et al., 2016; miller, 2016). while these studies are interesting and inform our present work, to our knowledge there has not been a study that directly examines the use of github by librarians with quantitative methods. this study was intended to fill this gap by offering some preliminary quantitative analysis.   method   this study compares two distinct samples of github users across a number of variables, namely: programming language choice, number of followers, number of following, number of public repositories, number of repository stars, gh index, and account creation date. we have gathered data related to these variables from the github api.   our methodology focused in part on exploratory data visualization. many scholars who have studied github have made a point of presenting their findings visually (blincoe et al., 2016; hu et al., 2016; jiang et al., 2017; kalliamvakou et al., 2016; lima et al., 2014; mcdonald et al., 2014; mergel, 2015; yan, wei, han, and wang, 2017). while there is a longstanding tradition of exploratory data visualization in scholarly literatures, stretching at least as far back at john tukey (1977) and edward tufte (1983), a robust popular visualization literature has only developed more recently. this latter body of work is often aimed at professional practitioners rather than scholars (cairo, 2013; few, 2012; knaflic, 2015; wong, 2010; yau, 2011, 2013). this paper draws on the insights of both of these traditions, insofar as they have taught us to think rigorously about effective visualization and to use data as an exploratory tool.   in his foundational text exploratory data analysis, tukey (1977) distinguishes between “exploratory” and “confirmatory” data analysis (p. 3). subsequently, he argues that exploratory data analysis is necessary to successfully implement a confirmatory analysis (1980). in his later work with hoaglin et al. (1983), he goes on to say that “exploratory data analysis emphasizes flexible searching for clues and evidence, whereas confirmatory data analysis stresses evaluating the availability of evidence” (p. 2). tukey’s encouragements toward exploratory analyses are taken up much later in the popular and technical literature on data visualization. for example, nathan yau (2011, 2013) follows in tukey’s footsteps in his work on data visualization.   exploratory data analysis, in the tradition of tukey and yau, provided us with a methodological starting point. we also supplemented this with confirmatory analysis in the form of t-tests and chi-square tests to ensure that our exploratory work was on the right track and to evaluate the significance of our findings.   we gathered data from the github apis, and then manually and programmatically filtered the data to produce a useful, workable dataset. we then applied scientific python tools to our data. python is a popular programming language with well-regarded data science libraries. our work used the numpy, scipy and pandas libraries to structure our data and run statistical tests, matplotlib and seaborn to plot charts, and other very common python libraries like requests and the python standard library. we made the code we wrote publicly available on github (eaton, 2016b).   we wrote code that retrieves and processes data from the github apis. understanding how apis work is helpful in understanding the methodology described below. while many apis have a very wide range of possible use cases, our approach ignored most of these, and instead focused on how we could harvest quantitative data from the github apis. below is a brief summary of how the code we wrote assembled data from github.   1.      we began by using the github search api to identify librarians on github. we did this to establish the primary sample of subjects to study. we queried the search api for the terms “library,” “librarian,” “libraries,” “bibliothèque,” and “bibliothecaire.” (we included the french terms to increase our sample size.) to see an example of captured search data, please consult appendix a, which shows the json for an individual record as it appears when retrieved from the search api. json, or javascript object notation, is a very common data serialization format (json, 2016).   a unique request was constructed for each of the search terms mentioned above. overall, this approach worked very well. however, the github search api returns a maximum of 100 results per query. therefore, if there were more than 100 results, our initial api call would not retrieve all of the available data. for this reason, it was preferable not to use stemming. instead, we searched each term individually because this maximized the number of results we retrieved.   we were able to partly work around the 100-result limitation by applying both ascending and descending sort order to the resulting data. this technique allowed us to get at both ends of the search results, meaning that we could capture up to 200 results per search, rather than just 100. this was sufficient to retrieve all of the results for the keywords “librarian,” “biblothèque,” and “bibliothecaire.” however, the searches for “library” and “libraries” yielded more than 200 hits. for these larger results sets, the total number of results was in the low four figures. as a result, in these cases, our script only captured the first 100 and the last 100 results. this situation was not ideal, but we decided this was acceptable for our purposes, since our searches provided us with sufficient data to conduct statistically significant analysis.   the data produced by these searches was deduplicated, concatenated, and saved as a json document.   2.      alongside the list of librarians and libraries that were generated, we also wrote a function that generated a random comparison group of github users. this did not use the search api, but rather generated random integers that could be passed to the user api as user id numbers. at the time of our analysis, the range of valid github user ids went up to around 20,000,000. our method involved randomly generating user ids to create a randomized sample of non-librarians.   3.      to maintain data quality, it was also important for us to distinguish between libraries in the traditional sense—as institutions, physical spaces, and collections—from libraries in the programming sense—as software packages. for our librarians sample, we were only interested in the former. for our comparison group sample, we were interested in excluding librarians. we considered using a natural language processing approach to make these distinctions; however, we ultimately decided that our dataset was small enough to sort through our results manually. this process allowed us to weed out software libraries from the librarians dataset. our manual approach allowed us to maintain a high degree of accuracy.   4.      we also needed to ensure that no librarians were placed in the comparison group by the function that generates the randomly selected comparison users. for this reason, we also manually reviewed each record in the comparison group to confirm that they were not librarians. we are confident that our manual verification of both the librarians group and comparison group was effective; nonetheless, any potential error introduced by this process is a limitation of this study.   5.      for both groups we retrieved data on each individual user from the github user api. this returned json data about that user. we captured all of the user data provided by the api; however, we used only a small subset of that data in our subsequent analysis. to see an example of captured user data, please consult appendix b, which shows user api json for an individual user.   6.      we then retrieved repository-level data for both groups of users, using the github repo api. this returned json that describes that user’s repositories. again, we captured all of the repository data provided by the api; however, we used only a small subset of that data in our subsequent analysis. to see an example of captured repository data, please consult appendix c, which shows some repository api json for an individual user.   7.      to improve the quality of the data ultimately used for analysis, it was necessary to filter the data about our two sample groups across a number of criteria. there are numerous reasons to carefully filter github data, some of which are described by kalliamvakou et al. (2016). many of the techniques they describe for mitigating the “perils” of using github data are reflected in the approaches to filtering data that we used in our scripts. the filtering techniques that we used are described in table 1.   once we had gathered and processed our data, the final samples used in our analyses consisted of 112 librarians and 112 comparison group subjects.   results and discussion   the most obvious place for us to begin our analysis was to compare the programming languages used by librarians to those used by the comparison group of github users. figures 1 and 2 show the top 15 languages for both groups. in total, there were 1,433 repositories for librarians (average of 12.79 per librarian) and 1,075 repositories for the comparison group (average of 9.60 per comparison group user). the same scale is used in figures 1 and 2 for easy comparison. a chi-square test was run on the top 12 languages that are common to both librarians and the control group. the most striking aspect of the results is the highly significant difference between the two groups (x2(11, n = 1840) = 282.70; p < .001). the p-value for this chi-square test is < .001, far below the conventional threshold for statistical significance of p < .050.   table 1 criteria for selection and their justification our criteria reasons for the criteria the user must have been active during the last 90 days. we wanted a contemporary picture of github, so our focus was on current users. to this end, we excluded users whose updated_at date was more than 90 days old. the user account must be more than 30 days old. we deliberately excluded very new sign-ups to focus on those users who had an established presence on github. we did this by excluding users whose created_at date was less than 30 days old. the user must have contributed to at least one public repository. it is important to note that this contribution can be to someone else’s repository; in other words, it is not necessarily their own repository. a significant number of users sign up for a github account but contribute nothing. these abandoned accounts would have produced almost no interesting data and would have crowded out accounts that have data that is interesting and useful. for these reasons, we excluded these users. the user must have a bio. because querying the search api for keywords (such as “librarian”) favours those profiles that have bios, we required that all users included in the study have bios. this made the comparison group and the librarians group more directly comparable. the list of users must be deduplicated. for obvious reasons we did not want to count the same user twice. the librarians dataset and the comparison dataset need to be the same size. this was done to make for an easy comparison of the two groups being studied.   the comparison group’s language choices are concentrated specifically in javascript and java, while the librarians’ language choices are more evenly distributed over a wider range of programming languages. the librarians’ wide variety of language choices may reflect the many different possible uses of code in libraries. libraries are home to many diverse activities, possibly resulting in many varied reasons for adopting different programming languages.   interestingly, the most popular languages differ as well. javascript, the preeminent language of the web, perhaps unsurprisingly dominates both groups. but from there the two samples diverge. while java features so prominently in the comparison group’s preferred choices, it comes in at a distant seventh among librarians. ruby is strongly favoured by librarians, while c# is strongly favoured by the comparsion group. while we will forgo discussing the merits of various languages, it is clear that librarians are choosing noticeably different languages for their projects. what we learned from these charts is that librarians are using a broad range of programming options, suggesting a range of possible use cases for code in libraries.   figure 1 language choice among librarians   figure 2 language choice among comparison group   next, we turned our attention to user metrics such as number of followers, number of following, and number of public repositories for users in both groups. “followers” and “following” should be familiar concepts to users of most social media. in the github context, if i “follow” someone, the result is that their activity (such as creating, forking, or starring repositories) will appear in a timeline on my github home screen. public repositories are projects that a user has shared on github. a repository can be either created from scratch by a user or derived from another user’s work (“forked,” in github terms).   we applied two-tailed t-tests to these variables for confirmatory purposes. because of the very high variance of the data, we ran a log transformation on the original data before doing the t-tests on the transformed data. from these tests, we found that there is not a statistically significant difference in terms of number of followers (t(222) = 1.62, p  = .107); number of following (t(222) = 0.91, p  = .363); and number of public repositories (t(222) = 1.52, p  = .131) between the two groups.   stars are another github concept that should be familiar to users of other social media. however, stars are used somewhat differently in github than in other social media. github users star repositories, rather than individual messages or posts. in this respect, starring is more of an endorsement of a project, rather than a reaction to a specific message or post from another user.   interestingly, there is a statistically significant difference in the number of repository stars (t(222) = 2.00, p  = .048) for the two sample groups. librarians have significantly more repository stars than the comparison group, according to a two-tailed t-test. because of the high variance in the data, we first ran a log transformation on the original number of stars data, and then did our t-test on the transformed data.   to summarize, the results of t-tests thus far can be seen in table 2.   table 2 summary of results from t-tests variable p-value of log(variable) mean, librarians mean, comparison group number of followers .107 107.19 12.61 number of following .363 13.24 8.85 number of public repositories .180 19.05 15.47 number of repository stars .048 41.96 9.0 p-value threshold used for statistical significance: < .050   to use a more exploratory, data visualization approach—in the spirit of tukey (1977) and yau (2011, 2013) —it was also interesting to visually group these datasets into those that measure activity (number of public repositories and number of following) and those that measure popularity (number of repository stars and number of followers) via scatterplots. this approach produced the charts in figure 3 and figure 4, respectively.   we used a log scale for these plots for increased readability and for consistency with our confirmatory analysis. visually, a log scale accommodates outlying users who were disproportionately more active or more popular than most of the subjects. also, we added one (1) to each value, so that all values would be displayed on the scatterplots, as a log scale cannot display zero values. it should be noted that this causes some distortion in the lower left-hand corner of the scatterplots.   to facilitate comparison of the two samples in aggregate, we included the averages for both groups, displayed on the scatterplots as a star. while the number of repository stars is the only statistically significant variable in these calculations, it is interesting to note that the mean for librarians tended to be higher than the mean for the comparison group for all of the other variables as well. this relationship is especially pronounced in the popularity scatterplot and can be seen in table 2 as well.   given the high variance found in all of these variables, it would be useful to conduct further analysis with a larger study sample to demonstrate (or alternately disprove) a statistically significant relationship between these factors. in this way, our exploratory visual analysis suggests directions for future, larger-scale confirmatory analyses.   figure 3 activity   figure 4 popularity   figure 5 distribution of gh index   it is also interesting to apply a more sophisticated measure to evaluate the number of repository stars. elsewhere, i have devised a measure called gh index, which measures the reach and productivity of github users (eaton, 2016a). gh index uses the same math as the widely known h-index measure, which measures the reach and productivity of a scholar. the innovation of gh index is that it applies the logic of h-index to github stars rather than academic citations. to our knowledge, gh index is a novel measure of contributions to open sources projects. miller (2016) later adopted this same measure and further popularized it, naming it gh impact.   when we chart our subject groups according to gh index score, we get the histogram shown in figure 5.   while the two groups are fairly similar according to gh index, librarians do have an edge, implying greater productivity and greater reach for their projects. we applied a t-test to this relationship, as statistical confirmation. because of much lower variance of the gh index data, we did not apply a log transformation when doing this particular test.  thus, we can see that librarians (m = 1.12) score significantly higher than the comparison group (m = 0.70) in productivity and reach (t(222) = 2.22, p  = .027), which is statistically significant at the conventional p < .050 level.   considering our findings on number of repository stars and gh index, we could potentially argue, following hu et al. (2016), that “popularity and quality [of github repositories] are strong indicators of their owners’ capability” (p. 5). however, because hu et al. do not adequately support this claim, we are reluctant to state the case as strongly as they do. nonetheless, our measures of repository stars and gh index lead us to suggest that our librarians’ repositories are making a greater impact than those of our comparison group.   this exploratory and confirmatory analysis of librarians’ popularity, activity, productivity, and reach leads us to ask a more specific follow-up question: what is librarians’ level of influence on github? blincoe et al. (2016) have measured the effect of various metrics on the level of influence that a user has on github. they point out “that popular users often attract their followers to new projects.” moreover, they argue that “users who are both very popular and very active influence their followers” (p. 31). if blincoe et al. are correct about this, popular librarians might similarly benefit from the added influence of their github reputation. this is an interesting suggestion; however, two issues prevent us from generalizing blincoe et al.’s conclusions to our dataset. first, only two librarians in our study reach the popularity threshold of 500 followers that blincoe et al. require to be included in their analysis of influence. second, while our exploratory analysis suggests that librarians may possibly be more active and popular than our comparison sample, this is not confirmed by t-tests. our statistically significant variables, number of repository stars and gh index, are not sufficient for an analysis along the lines of blincoe et al. because of these factors, blincoe’s conclusions are not generalizable to our dataset.   one factor that may help to explain our findings is that librarians can be shown to be early adopters of github. specifically, if librarians have been on github longer than the average user, it can explain their relative productivity and reach, as shown in the data on repository stars and gh index. it is therefore interesting to plot a histogram that shows the account creation date of our samples’ user accounts, showing how long they have been on github (see figure 6). this relationship, when tested with a two-tailed t-test, yields a highly significant p-value (t(222) = 2.64, p = .009). we can therefore confidently say that librarians (m = 1227.21 days) tend to have been on github longer than our comparison group users (m = 928.21 days).   figure 6 account creation date   these analyses provide many interesting insights to consider. we hypothesize that librarians’ measurable and statistically significant involvement with github is the result of their profession’s embrace of github as an “open” platform. sharing code publicly often presupposes a certain commitment to openness. oftentimes this openness is a legal category, assigned by the programmer or the institution as a software license. kelty (2008) describes the history of openness in software communities. openness may also be a powerful motivator for some librarians who share a commitment to the value of openly sharing information in all formats (puckett, 2012; fernandez, 2012). github provides a way for librarians to put their commitments to openness into action by providing a highly visible way to share code freely.   conclusion   code sharing is an important topic in librarianship because librarians shape their libraries and communities through the software they build. by programming for libraries, librarians are directly contributing to what their libraries will be in the future. librarians’ work builds their institutions with code. in this sense, librarians create software tools that produce “actually existing alternatives” (kelty, 2008, p. 3) for libraries.   we have preliminarily established that the librarians in our study demonstrate statistically significant outperformance in reach and productivity on github. they also closely mirror the comparison group on measures of activity and popularity. we have pointed out that librarians use diverse programming languages, perhaps as a result of their diverse librarianship practices. moreover, we hypothesize that librarians’ embrace of github is rooted in values of openness. hopefully this study has demonstrated that librarians are indeed significant users of github and that further confirmatory study of these topics is warranted.   references   arora, r., goel, s., & mittal, r. k. (2016). supporting collaborative software development over github. software: practice and experience, 47, 1393–1416. https://doi.org/10.1002/spe.2468   bell, p., & beer, b. (2015). introducing github: a non-technical guide. beijing: o’reilly.   blincoe, k., sheoran, j., goggins, s., petakovic, e., & damian, d. (2016). understanding the popular users: following, affiliation influence and leadership on github. information and software technology, 70, 30–39. https://doi.org/10.1016/j.infsof.2015.10.002   blischak, j. d., davenport, e. r., & wilson, g. (2016). a quick introduction to version control with git and github. plos computational biology, 12(1). https://doi.org/10.1371/journal.pcbi.1004668   breeding, m. (2015). more than a hack: empowering your library through coding. computers in libraries, 35(6), 25–27.   cairo, a. (2013). the functional art: an introduction to information graphics and visualization. berkeley, ca: new riders.   dataquest. (2015, september 8). python api tutorial: an introduction to using apis. retrieved may 24, 2017, from https://www.dataquest.io/blog/python-api-tutorial/   davis, r. c. (2015). git and github for librarians. behavioral & social sciences librarian, 34(3), 159–164. https://doi.org/10.1080/01639269.2015.1062586   davis, r. c. (2016). hackathons for libraries and librarians. behavioral & social sciences librarian, 35(2), 87–91. https://doi.org/10.1080/01639269.2016.1208561   de la cruz, j., & hogan, j. (2016). “hello, world!”: starting a coding group for librarians. public services quarterly, 12(3), 249-256. https://doi.org/10.1080/15228959.2016.1197082     eaton, m. (2016a). i made my own altmetric. retrieved april 12, 2017, from https://kingsboroughlibtech.commons.gc.cuny.edu/2016/06/14/i-made-up-an-altmetric/   eaton, m. (2016b). github-study. github. retrieved april 12, 2017, from https://github.com/markeeaton/github-study   enis, m. (2013). cracking the code. library journal, 138(4), 24.   fernandez, p. (2012). library values that interface with technology: public service information professionals, zotero and open source decision making. library philosophy and practice, 1–11. https://digitalcommons.unl.edu/libphilprac/803/   few, s. (2012). show me the numbers: designing tables and graphs to enlighten (2nd ed.). burlingame, ca: analytics press.   github (2017a). git and github learning resources: user documentation. retrieved april 10, 2017, from https://help.github.com/articles/git-and-github-learning-resources/   github (2017b). github api v3: github developer guide. retrieved april 11, 2017, from https://developer.github.com/v3/   grier, d. a. (2015). the github effect. computer, 48(5), 116. https://doi.org/10.1109/mc.2015.146    hoaglin, d. c., mosteller, f., & tukey, j. w. (1983). understanding robust and exploratory data analysis. new york: john wiley & sons.   hu, y., zhang, j., bai, x., yu, s., & yang, z. (2016). influence analysis of github repositories. springerplus, 5, 1–19. https://doi.org/10.1186/s40064-016-2897-7   jiang, j., lo, d., he, j., xia, x., kochhar, p. s., & zhang, l. (2017). why and how developers fork what from whom in github. empirical software engineering, (22), 547–578. https://doi.org/10.1007/s10664-016-9436-6   json. (2016). retrieved february 23, 2016, from http://www.json.org/   kalliamvakou, e., gousios, g., blincoe, k., singer, l., german, d. m., & damian, d. (2016). an in-depth study of the promises and perils of mining github. empirical software engineering, 21, 2035–2071. https://doi.org/10.1007/s10664-015-9393-5   kelty, c. m. (2008). two bits: the cultural significance of free software. durham, nc: duke up.   knaflic, c. n. (2015). storytelling with data: a data visualization guide for business professionals. hoboken, nj: john wiley and sons, inc.   lima, a., rossi, l., & musolesi, m. (2014). coding together at scale: github as a collaborative social network. arxiv (1407.2535). https://arxiv.org/abs/1407.2535   longo, j., & kelley, t. m. (2016). github use in public administration in canada: early experience with a new collaboration tool. canadian public administration, 59(4), 598–623. https://doi.org/10.1111/capa.12192   lord, j. (2014). jlord/git-it. retrieved april 10, 2017, from https://github.com/jlord/git-it   lord, j. (2015). jlord/git-it-electron. retrieved april 10, 2017, from https://github.com/jlord/git-it-electron   mcdonald, n., blincoe, k., petakovic, e., & goggins, s. (2014). modeling distributed collaboration on github. advances in complex systems, 17(7-8). https://doi.org/10.1142/s0219525914500246   mergel, i. (2015). open collaboration in the public sector: the case of social coding on github. government information quarterly, 32, 464–472. https://doi.org/10.1016/j.giq.2015.09.004   miller, i. d. (2016). gh-impact measures open source influence. retrieved april 10, 2017, from http://www.gh-impact.com   perez-riverol, y., gatto, l., wang, r., sachsenberg, t., uszkoreit, j., da veiga leprevost, f., fufezan, c., ternent, t., eglen, s. j., … vizcaino, j. a. (2016). ten simple rules for taking advantage of git and github. plos computational biology, 12(7). https://doi.org/10.1371/journal.pcbi.1004947   perkel, j. (2016). democratic databases: science on github. nature, 538, 127–128.   puckett, j. (2012). open source software and librarian values. georgia library quarterly, 49(3), 30–34. https://scholarworks.gsu.edu/univ_lib_facpub/95/   stuart, d. (2011). programming librarians in the web of data. online, 35(2), 42–44.     tufte, e. r. (1983). the visual display of quantitative information. cheshire, ct: graphics press.   tukey, j. w. (1977). exploratory data analysis. reading, ma: addison-wesley.   tukey, j. w. (1980). we need both exploratory and confirmatory. the american statistician, 34(1), 23–25. https://doi.org/10.1080/00031305.1980.10482706   wong, d. m. (2010). the wall street journal guide to information graphics: the do’s and don’ts of presenting data, facts, and figures. new york: w.w. norton & co.   yan, d.-c., wei, z.-w., han, x.-p., & wang, b.-h. (2017). empirical analysis on the human dynamics of blogging behavior on github. physica a, 465, 775–781. https://doi.org/10.1016/j.physa.2016.08.054   yau, n. (2011). visualize this: the flowingdata guide to design, visualization and statistics. indianapolis, in: wiley.   yau, n. (2013). data points: visualization that means something. somerset, nj: john wiley & sons.   yelton, a. (2015). coding for librarians: learning by example (library technology reports no. 51(3)). chicago, il: american library association.     appendix a data from the github search api {   "total_count": 322,   "incomplete_results": false,   "items": [     {       "login": "octocat",       "id": 583231,       "avatar_url": "https://avatars3.githubusercontent.com/u/583231?v=4",       "gravatar_id": "",       "url": "https://api.github.com/users/octocat",       "html_url": "https://github.com/octocat",       "followers_url": "https://api.github.com/users/octocat/followers",       "following_url": "https://api.github.com/users/octocat/following{/other_user}",       "gists_url": "https://api.github.com/users/octocat/gists{/gist_id}",       "starred_url": "https://api.github.com/users/octocat/starred{/owner}{/repo}",       "subscriptions_url": "https://api.github.com/users/octocat/subscriptions",       "organizations_url": "https://api.github.com/users/octocat/orgs",       "repos_url": "https://api.github.com/users/octocat/repos",       "events_url": "https://api.github.com/users/octocat/events{/privacy}",       "received_events_url": "https://api.github.com/users/octocat/received_events",       "type": "user",       "site_admin": false,       "score": 114.60762     },     ...   ] }     appendix b data from the github user api {   "login": "octocat",   "id": 583231,   "avatar_url": "https://avatars3.githubusercontent.com/u/583231?v=4",   "gravatar_id": "",   "url": "https://api.github.com/users/octocat",   "html_url": "https://github.com/octocat",   "followers_url": "https://api.github.com/users/octocat/followers",   "following_url": "https://api.github.com/users/octocat/following{/other_user}",   "gists_url": "https://api.github.com/users/octocat/gists{/gist_id}",   "starred_url": "https://api.github.com/users/octocat/starred{/owner}{/repo}",   "subscriptions_url": "https://api.github.com/users/octocat/subscriptions",   "organizations_url": "https://api.github.com/users/octocat/orgs",   "repos_url": "https://api.github.com/users/octocat/repos",   "events_url": "https://api.github.com/users/octocat/events{/privacy}",   "received_events_url": "https://api.github.com/users/octocat/received_events",   "type": "user",   "site_admin": false,   "name": "the octocat",   "company": "github",   "blog": "http://www.github.com/blog",   "location": "san francisco",   "email": null,   "hireable": null,   "bio": null,   "public_repos": 7,   "public_gists": 8,   "followers": 2070,   "following": 5,   "created_at": "2011-01-25t18:44:36z",   "updated_at": "2018-01-01t12:31:09z" }     appendix c data from the github repo api [   {     "id": 18221276,     "name": "git-consortium",     "full_name": "octocat/git-consortium",     "owner": {       "login": "octocat",       "id": 583231,       "avatar_url": "https://avatars3.githubusercontent.com/u/583231?v=4",       "gravatar_id": "",       "url": "https://api.github.com/users/octocat",       "html_url": "https://github.com/octocat",       "followers_url": "https://api.github.com/users/octocat/followers",       "following_url": "https://api.github.com/users/octocat/following{/other_user}",       "gists_url": "https://api.github.com/users/octocat/gists{/gist_id}",       "starred_url": "https://api.github.com/users/octocat/starred{/owner}{/repo}",       "subscriptions_url": "https://api.github.com/users/octocat/subscriptions",       "organizations_url": "https://api.github.com/users/octocat/orgs",       "repos_url": "https://api.github.com/users/octocat/repos",       "events_url": "https://api.github.com/users/octocat/events{/privacy}",       "received_events_url": "https://api.github.com/users/octocat/received_events",       "type": "user",       "site_admin": false     },     "private": false,     "html_url": "https://github.com/octocat/git-consortium",     "description": "this repo is for demonstration purposes only.",     "fork": false,     "url": "https://api.github.com/repos/octocat/git-consortium",     "forks_url": "https://api.github.com/repos/octocat/git-consortium/forks",     "keys_url": "https://api.github.com/repos/octocat/git-consortium/keys{/key_id}",     "collaborators_url": "https://api.github.com/repos/octocat/git-consortium/collaborators{/collaborator}",     "teams_url": "https://api.github.com/repos/octocat/git-consortium/teams",     "hooks_url": "https://api.github.com/repos/octocat/git-consortium/hooks",     "issue_events_url": "https://api.github.com/repos/octocat/git-consortium/issues/events{/number}",     "events_url": "https://api.github.com/repos/octocat/git-consortium/events",     "assignees_url": "https://api.github.com/repos/octocat/git-consortium/assignees{/user}",     "branches_url": "https://api.github.com/repos/octocat/git-consortium/branches{/branch}",     "tags_url": "https://api.github.com/repos/octocat/git-consortium/tags",     "blobs_url": "https://api.github.com/repos/octocat/git-consortium/git/blobs{/sha}",     "git_tags_url": "https://api.github.com/repos/octocat/git-consortium/git/tags{/sha}",     "git_refs_url": "https://api.github.com/repos/octocat/git-consortium/git/refs{/sha}",     "trees_url": "https://api.github.com/repos/octocat/git-consortium/git/trees{/sha}",     "statuses_url": "https://api.github.com/repos/octocat/git-consortium/statuses/{sha}",     "languages_url": "https://api.github.com/repos/octocat/git-consortium/languages",     "stargazers_url": "https://api.github.com/repos/octocat/git-consortium/stargazers",     "contributors_url": "https://api.github.com/repos/octocat/git-consortium/contributors",     "subscribers_url": "https://api.github.com/repos/octocat/git-consortium/subscribers",     "subscription_url": "https://api.github.com/repos/octocat/git-consortium/subscription",     "commits_url": "https://api.github.com/repos/octocat/git-consortium/commits{/sha}",     "git_commits_url": "https://api.github.com/repos/octocat/git-consortium/git/commits{/sha}",     "comments_url": "https://api.github.com/repos/octocat/git-consortium/comments{/number}",     "issue_comment_url": "https://api.github.com/repos/octocat/git-consortium/issues/comments{/number}",     "contents_url": "https://api.github.com/repos/octocat/git-consortium/contents/{+path}",     "compare_url": "https://api.github.com/repos/octocat/git-consortium/compare/{base}...{head}",     "merges_url": "https://api.github.com/repos/octocat/git-consortium/merges",     "archive_url": "https://api.github.com/repos/octocat/git-consortium/{archive_format}{/ref}",     "downloads_url": "https://api.github.com/repos/octocat/git-consortium/downloads",     "issues_url": "https://api.github.com/repos/octocat/git-consortium/issues{/number}",     "pulls_url": "https://api.github.com/repos/octocat/git-consortium/pulls{/number}",     "milestones_url": "https://api.github.com/repos/octocat/git-consortium/milestones{/number}",     "notifications_url": "https://api.github.com/repos/octocat/git-consortium/notifications{?since,all,participating}",     "labels_url": "https://api.github.com/repos/octocat/git-consortium/labels{/name}",     "releases_url": "https://api.github.com/repos/octocat/git-consortium/releases{/id}",     "deployments_url": "https://api.github.com/repos/octocat/git-consortium/deployments",     "created_at": "2014-03-28t17:55:38z",     "updated_at": "2017-12-06t01:15:32z",     "pushed_at": "2016-10-30t13:43:30z",     "git_url": "git://github.com/octocat/git-consortium.git",     "ssh_url": "git@github.com:octocat/git-consortium.git",     "clone_url": "https://github.com/octocat/git-consortium.git",     "svn_url": "https://github.com/octocat/git-consortium",     "homepage": null,     "size": 190,     "stargazers_count": 8,     "watchers_count": 8,     "language": null,     "has_issues": true,     "has_projects": true,     "has_downloads": true,     "has_wiki": true,     "has_pages": false,     "forks_count": 27,     "mirror_url": null,     "archived": false,     "open_issues_count": 4,     "license": {       "key": "mit",       "name": "mit license",       "spdx_id": "mit",       "url": "https://api.github.com/licenses/mit"     },     "forks": 27,     "open_issues": 4,     "watchers": 8,     "default_branch": "master"   },   ... ]       <html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/tr/rec-html40"> <head> <meta http-equiv=content-type content="text/html; charset=unicode"> <meta name=progid content=word.document> <meta name=generator content="microsoft word 15"> <meta name=originator content="microsoft word 15"> <link rel=file-list href="art_grabowsky_29657_files/filelist.xml"> <link rel=edit-time-data href="art_grabowsky_29657_files/editdata.mso"> <!--[if !mso]> <style> v\:* {behavior:url(#default#vml);} o\:* {behavior:url(#default#vml);} w\:* {behavior:url(#default#vml);} .shape {behavior:url(#default#vml);} </style> <![endif]--><!--[if gte mso 9]><xml> <o:documentproperties> <o:author>mallan, katrine</o:author> <o:template>normal</o:template> <o:lastauthor>hinrichs, rachel</o:lastauthor> <o:revision>4</o:revision> <o:totaltime>16</o:totaltime> <o:lastprinted>2020-06-12t14:46:00z</o:lastprinted> <o:created>2020-06-12t14:47:00z</o:created> <o:lastsaved>2020-06-12t15:08:00z</o:lastsaved> <o:pages>39</o:pages> <o:words>10981</o:words> <o:characters>62598</o:characters> <o:company>microsoft</o:company> <o:lines>521</o:lines> <o:paragraphs>146</o:paragraphs> <o:characterswithspaces>73433</o:characterswithspaces> <o:version>16.00</o:version> </o:documentproperties> <o:officedocumentsettings> <o:allowpng/> </o:officedocumentsettings> </xml><![endif]--> <link rel=datastoreitem href="art_grabowsky_29657_files/item0017.xml" target="art_grabowsky_29657_files/props018.xml"> <link rel=themedata href="art_grabowsky_29657_files/themedata.thmx"> <link rel=colorschememapping href="art_grabowsky_29657_files/colorschememapping.xml"> <!--[if gte mso 9]><xml> <w:worddocument> <w:view>print</w:view> <w:spellingstate>clean</w:spellingstate> <w:grammarstate>clean</w:grammarstate> <w:trackmoves>false</w:trackmoves> <w:trackformatting/> <w:punctuationkerning/> <w:validateagainstschemas/> <w:saveifxmlinvalid>false</w:saveifxmlinvalid> <w:ignoremixedcontent>false</w:ignoremixedcontent> <w:alwaysshowplaceholdertext>false</w:alwaysshowplaceholdertext> <w:donotpromoteqf/> <w:lidthemeother>en-ca</w:lidthemeother> <w:lidthemeasian>x-none</w:lidthemeasian> <w:lidthemecomplexscript>x-none</w:lidthemecomplexscript> <w:compatibility> <w:breakwrappedtables/> <w:snaptogridincell/> <w:wraptextwithpunct/> <w:useasianbreakrules/> <w:dontgrowautofit/> <w:splitpgbreakandparamark/> <w:enableopentypekerning/> <w:dontflipmirrorindents/> <w:overridetablestylehps/> </w:compatibility> <w:browserlevel>microsoftinternetexplorer4</w:browserlevel> <m:mathpr> <m:mathfont m:val="cambria math"/> <m:brkbin m:val="before"/> <m:brkbinsub m:val="&#45;-"/> <m:smallfrac m:val="off"/> <m:dispdef/> <m:lmargin m:val="0"/> <m:rmargin m:val="0"/> <m:defjc m:val="centergroup"/> <m:wrapindent m:val="1440"/> <m:intlim m:val="subsup"/> <m:narylim m:val="undovr"/> </m:mathpr></w:worddocument> </xml><![endif]--><!--[if gte mso 9]><xml> <w:latentstyles deflockedstate="false" defunhidewhenused="false" defsemihidden="false" defqformat="false" defpriority="99" latentstylecount="375"> <w:lsdexception locked="false" priority="0" qformat="true" name="normal"/> <w:lsdexception locked="false" priority="0" qformat="true" name="heading 1"/> <w:lsdexception locked="false" priority="0" semihidden="true" unhidewhenused="true" qformat="true" name="heading 2"/> <w:lsdexception locked="false" priority="0" semihidden="true" unhidewhenused="true" qformat="true" name="heading 3"/> <w:lsdexception locked="false" priority="0" semihidden="true" unhidewhenused="true" qformat="true" name="heading 4"/> <w:lsdexception locked="false" priority="0" semihidden="true" unhidewhenused="true" qformat="true" name="heading 5"/> <w:lsdexception locked="false" priority="0" semihidden="true" unhidewhenused="true" qformat="true" name="heading 6"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 7"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 8"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 9"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 6"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 7"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 8"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 9"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 1"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 2"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 3"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 4"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 5"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 6"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 7"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 8"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 9"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="normal indent"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="footnote text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="annotation text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="header"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="footer"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index heading"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" qformat="true" name="caption"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table of figures"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="envelope address"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="envelope return"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="footnote reference"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="annotation reference"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="line number"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="page number"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="endnote reference"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="endnote text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table of authorities"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="macro"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="toa heading"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number 5"/> <w:lsdexception locked="false" qformat="true" name="title"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="closing"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="signature"/> <w:lsdexception locked="false" priority="1" semihidden="true" unhidewhenused="true" name="default paragraph font"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text indent"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="message header"/> <w:lsdexception locked="false" qformat="true" name="subtitle"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="salutation"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="date"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text first indent"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text first indent 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="note heading"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text indent 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text indent 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="block text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="hyperlink"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="followedhyperlink"/> <w:lsdexception locked="false" priority="22" qformat="true" name="strong"/> <w:lsdexception locked="false" priority="20" qformat="true" name="emphasis"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="document map"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="plain text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="e-mail signature"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html top of form"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html bottom of form"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="normal (web)"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html acronym"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html address"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html cite"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html code"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html definition"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html keyboard"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html preformatted"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html sample"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html typewriter"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html variable"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="normal table"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="annotation subject"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="no list"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="outline list 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="outline list 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="outline list 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table simple 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table simple 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table simple 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table classic 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table classic 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table classic 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table classic 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table colorful 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table colorful 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table colorful 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 6"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 7"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 8"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 6"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 7"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 8"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table 3d effects 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table 3d effects 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table 3d effects 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table contemporary"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table elegant"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table professional"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table subtle 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table subtle 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table web 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table web 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table web 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="balloon text"/> <w:lsdexception locked="false" priority="39" name="table grid"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table theme"/> <w:lsdexception locked="false" semihidden="true" name="placeholder text"/> <w:lsdexception locked="false" priority="1" qformat="true" name="no spacing"/> <w:lsdexception locked="false" priority="60" name="light shading"/> <w:lsdexception locked="false" priority="61" name="light list"/> <w:lsdexception locked="false" priority="62" name="light grid"/> <w:lsdexception locked="false" priority="63" name="medium shading 1"/> <w:lsdexception locked="false" priority="64" name="medium shading 2"/> <w:lsdexception locked="false" priority="65" name="medium list 1"/> <w:lsdexception locked="false" priority="66" name="medium list 2"/> <w:lsdexception locked="false" priority="67" name="medium grid 1"/> <w:lsdexception locked="false" priority="68" name="medium grid 2"/> <w:lsdexception locked="false" priority="69" name="medium grid 3"/> <w:lsdexception locked="false" priority="70" name="dark list"/> <w:lsdexception locked="false" priority="71" name="colorful shading"/> <w:lsdexception locked="false" priority="72" name="colorful list"/> <w:lsdexception locked="false" priority="73" name="colorful grid"/> <w:lsdexception locked="false" priority="60" name="light shading accent 1"/> <w:lsdexception locked="false" priority="61" name="light list accent 1"/> <w:lsdexception locked="false" priority="62" name="light grid accent 1"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 1"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 1"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 1"/> <w:lsdexception locked="false" semihidden="true" name="revision"/> <w:lsdexception locked="false" priority="34" qformat="true" name="list paragraph"/> <w:lsdexception locked="false" priority="29" qformat="true" name="quote"/> <w:lsdexception locked="false" priority="30" qformat="true" name="intense quote"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 1"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 1"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 1"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 1"/> <w:lsdexception locked="false" priority="70" name="dark list accent 1"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 1"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 1"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 1"/> <w:lsdexception locked="false" priority="60" name="light shading accent 2"/> <w:lsdexception locked="false" priority="61" name="light list accent 2"/> <w:lsdexception locked="false" priority="62" name="light grid accent 2"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 2"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 2"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 2"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 2"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 2"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 2"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 2"/> <w:lsdexception locked="false" priority="70" name="dark list accent 2"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 2"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 2"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 2"/> <w:lsdexception locked="false" priority="60" name="light shading accent 3"/> <w:lsdexception locked="false" priority="61" name="light list accent 3"/> <w:lsdexception locked="false" priority="62" name="light grid accent 3"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 3"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 3"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 3"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 3"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 3"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 3"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 3"/> <w:lsdexception locked="false" priority="70" name="dark list accent 3"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 3"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 3"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 3"/> <w:lsdexception locked="false" priority="60" name="light shading accent 4"/> <w:lsdexception locked="false" priority="61" name="light list accent 4"/> <w:lsdexception locked="false" priority="62" name="light grid accent 4"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 4"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 4"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 4"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 4"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 4"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 4"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 4"/> <w:lsdexception locked="false" priority="70" name="dark list accent 4"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 4"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 4"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 4"/> <w:lsdexception locked="false" priority="60" name="light shading accent 5"/> <w:lsdexception locked="false" priority="61" name="light list accent 5"/> <w:lsdexception locked="false" priority="62" name="light grid accent 5"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 5"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 5"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 5"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 5"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 5"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 5"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 5"/> <w:lsdexception locked="false" priority="70" name="dark list accent 5"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 5"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 5"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 5"/> <w:lsdexception locked="false" priority="60" name="light shading accent 6"/> <w:lsdexception locked="false" priority="61" name="light list accent 6"/> <w:lsdexception locked="false" priority="62" name="light grid accent 6"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 6"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 6"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 6"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 6"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 6"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 6"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 6"/> <w:lsdexception locked="false" priority="70" name="dark list accent 6"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 6"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 6"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 6"/> <w:lsdexception locked="false" priority="19" qformat="true" name="subtle emphasis"/> <w:lsdexception locked="false" priority="21" qformat="true" name="intense emphasis"/> <w:lsdexception locked="false" priority="31" qformat="true" name="subtle reference"/> <w:lsdexception locked="false" priority="32" qformat="true" name="intense reference"/> <w:lsdexception locked="false" priority="33" qformat="true" name="book title"/> <w:lsdexception locked="false" priority="37" semihidden="true" unhidewhenused="true" name="bibliography"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" qformat="true" name="toc heading"/> <w:lsdexception locked="false" priority="41" name="plain table 1"/> <w:lsdexception locked="false" priority="42" name="plain table 2"/> <w:lsdexception locked="false" priority="43" name="plain table 3"/> <w:lsdexception locked="false" priority="44" name="plain table 4"/> <w:lsdexception locked="false" priority="45" name="plain table 5"/> <w:lsdexception locked="false" priority="40" name="grid table light"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light"/> <w:lsdexception locked="false" priority="47" name="grid table 2"/> <w:lsdexception locked="false" priority="48" name="grid table 3"/> <w:lsdexception locked="false" priority="49" name="grid table 4"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 1"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 1"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 1"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 1"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 1"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 1"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 1"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 2"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 2"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 2"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 2"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 2"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 2"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 2"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 3"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 3"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 3"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 3"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 3"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 3"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 3"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 4"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 4"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 4"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 4"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 4"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 4"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 4"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 5"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 5"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 5"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 5"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 5"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 5"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 5"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 6"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 6"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 6"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 6"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 6"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 6"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 6"/> <w:lsdexception locked="false" priority="46" name="list table 1 light"/> <w:lsdexception locked="false" priority="47" name="list table 2"/> <w:lsdexception locked="false" priority="48" name="list table 3"/> <w:lsdexception locked="false" priority="49" name="list table 4"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 1"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 1"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 1"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 1"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 1"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 1"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 1"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 2"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 2"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 2"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 2"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 2"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 2"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 2"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 3"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 3"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 3"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 3"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 3"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 3"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 3"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 4"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 4"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 4"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 4"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 4"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 4"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 4"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 5"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 5"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 5"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 5"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 5"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 5"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 5"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 6"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 6"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 6"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 6"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 6"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 6"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 6"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="mention"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="smart hyperlink"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="hashtag"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="unresolved mention"/> </w:latentstyles> </xml><![endif]--> <style> <!- /* font definitions */ @font-face {font-family:wingdings; panose-1:5 0 0 0 0 0 0 0 0 0; mso-font-charset:2; mso-generic-font-family:auto; mso-font-pitch:variable; mso-font-signature:0 268435456 0 0 -2147483648 0;} @font-face {font-family:simsun; panose-1:2 1 6 0 3 1 1 1 1 1; mso-font-alt:宋体; mso-font-charset:134; mso-generic-font-family:auto; mso-font-pitch:variable; mso-font-signature:3 680460288 22 0 262145 0;} @font-face {font-family:latha; panose-1:2 0 4 0 0 0 0 0 0 0; mso-font-charset:0; mso-generic-font-family:swiss; mso-font-pitch:variable; mso-font-signature:1048579 0 0 0 1 0;} @font-face {font-family:"cambria math"; panose-1:2 4 5 3 5 4 6 3 2 4; mso-font-charset:0; mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:-536869121 1107305727 33554432 0 415 0;} @font-face {font-family:calibri; panose-1:2 15 5 2 2 2 4 3 2 4; mso-font-charset:0; mso-generic-font-family:swiss; mso-font-pitch:variable; mso-font-signature:-536858881 -1073732485 9 0 511 0;} @font-face {font-family:"palatino linotype"; panose-1:2 4 5 2 5 5 5 3 3 4; mso-font-charset:0; mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:-536870265 1073741843 0 0 415 0;} @font-face {font-family:cambria; panose-1:2 4 5 3 5 4 6 3 2 4; mso-font-charset:0; mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:-536869121 1107305727 33554432 0 415 0;} @font-face {font-family:tahoma; panose-1:2 11 6 4 3 5 4 4 2 4; mso-font-charset:0; mso-generic-font-family:swiss; mso-font-pitch:variable; mso-font-signature:-520081665 -1073717157 41 0 66047 0;} @font-face {font-family:georgia; panose-1:2 4 5 2 5 4 5 2 3 3; mso-font-charset:0; mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:647 0 0 0 159 0;} @font-face {font-family:"\@simsun"; panose-1:2 1 6 0 3 1 1 1 1 1; mso-font-charset:134; mso-generic-font-family:auto; mso-font-pitch:variable; mso-font-signature:3 680460288 22 0 262145 0;} /* style definitions */ p.msonormal, li.msonormal, div.msonormal {mso-style-unhide:no; mso-style-qformat:yes; mso-style-parent:""; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca;} h1 {mso-style-unhide:no; mso-style-qformat:yes; mso-style-link:"heading 1 char"; mso-style-next:normal; margin-top:12.0pt; margin-right:0in; margin-bottom:0in; margin-left:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan lines-together; page-break-after:avoid; mso-outline-level:1; font-size:14.0pt; font-family:"cambria",serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:#365f91; mso-font-kerning:0pt; mso-ansi-language:en-ca; font-weight:bold;} h2 {mso-style-noshow:yes; mso-style-qformat:yes; mso-style-link:"heading 2 char"; mso-style-next:normal; margin-top:10.0pt; margin-right:0in; margin-bottom:0in; margin-left:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan lines-together; page-break-after:avoid; mso-outline-level:2; font-size:13.0pt; font-family:"cambria",serif; mso-ascii-font-family:cambria; mso-ascii-theme-font:major-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:major-fareast; mso-hansi-font-family:cambria; mso-hansi-theme-font:major-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:major-bidi; color:#4f81bd; mso-themecolor:accent1; mso-ansi-language:en-ca; font-weight:bold;} h3 {mso-style-noshow:yes; mso-style-qformat:yes; mso-style-link:"heading 3 char"; mso-margin-top-alt:auto; margin-right:0in; mso-margin-bottom-alt:auto; margin-left:0in; mso-pagination:widow-orphan; mso-outline-level:3; font-size:13.5pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; font-weight:bold;} h4 {mso-style-noshow:yes; mso-style-qformat:yes; mso-style-link:"heading 4 char"; mso-style-next:normal; margin-top:14.0pt; margin-right:0in; margin-bottom:4.0pt; margin-left:0in; mso-pagination:widow-orphan lines-together; page-break-after:avoid; mso-outline-level:4; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:arial; color:#666666; font-weight:normal;} h5 {mso-style-noshow:yes; mso-style-qformat:yes; mso-style-link:"heading 5 char"; mso-style-next:normal; margin-top:12.0pt; margin-right:0in; margin-bottom:4.0pt; margin-left:0in; mso-pagination:widow-orphan lines-together; page-break-after:avoid; mso-outline-level:5; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:arial; color:#666666; font-weight:normal;} h6 {mso-style-noshow:yes; mso-style-qformat:yes; mso-style-link:"heading 6 char"; mso-style-next:normal; margin-top:12.0pt; margin-right:0in; margin-bottom:4.0pt; margin-left:0in; mso-pagination:widow-orphan lines-together; page-break-after:avoid; mso-outline-level:6; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:arial; color:#666666; font-weight:normal; font-style:italic; mso-bidi-font-style:normal;} p.msofootnotetext, li.msofootnotetext, div.msofootnotetext {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"footnote text char"; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:calibri; mso-fareast-theme-font:minor-latin; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi;} p.msocommenttext, li.msocommenttext, div.msocommenttext {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"comment text char"; margin-top:0in; margin-right:0in; margin-bottom:10.0pt; margin-left:0in; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman";} p.msoheader, li.msoheader, div.msoheader {mso-style-priority:99; mso-style-link:"header char"; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; tab-stops:center 3.25in right 6.5in; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca;} p.msofooter, li.msofooter, div.msofooter {mso-style-priority:99; mso-style-link:"footer char"; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; tab-stops:center 3.25in right 6.5in; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca;} p.msocaption, li.msocaption, div.msocaption {mso-style-noshow:yes; mso-style-priority:99; mso-style-qformat:yes; mso-style-next:normal; margin-top:.25in; margin-right:0in; margin-bottom:0in; margin-left:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; tab-stops:255.15pt; font-size:11.0pt; mso-bidi-font-size:12.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:#3e9ed0;} span.msofootnotereference {mso-style-noshow:yes; mso-style-priority:99; vertical-align:super;} span.msocommentreference {mso-style-noshow:yes; mso-style-priority:99; mso-ansi-font-size:8.0pt; mso-bidi-font-size:8.0pt;} p.msolistbullet, li.msolistbullet, div.msolistbullet {mso-style-noshow:yes; mso-style-priority:99; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.25in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:-.25in; mso-pagination:widow-orphan; mso-list:l0 level1 lfo1; font-size:11.0pt; mso-bidi-font-size:12.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman";} p.msolistbulletcxspfirst, li.msolistbulletcxspfirst, div.msolistbulletcxspfirst {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.25in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:-.25in; mso-pagination:widow-orphan; mso-list:l0 level1 lfo1; font-size:11.0pt; mso-bidi-font-size:12.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman";} p.msolistbulletcxspmiddle, li.msolistbulletcxspmiddle, div.msolistbulletcxspmiddle {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.25in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:-.25in; mso-pagination:widow-orphan; mso-list:l0 level1 lfo1; font-size:11.0pt; mso-bidi-font-size:12.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman";} p.msolistbulletcxsplast, li.msolistbulletcxsplast, div.msolistbulletcxsplast {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.25in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:-.25in; mso-pagination:widow-orphan; mso-list:l0 level1 lfo1; font-size:11.0pt; mso-bidi-font-size:12.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman";} p.msotitle, li.msotitle, div.msotitle {mso-style-priority:99; mso-style-unhide:no; mso-style-qformat:yes; mso-style-link:"title char"; mso-style-next:normal; margin-top:0in; margin-right:0in; margin-bottom:3.0pt; margin-left:0in; mso-pagination:widow-orphan lines-together; page-break-after:avoid; font-size:26.0pt; font-family:"times new roman",serif; mso-fareast-font-family:arial;} p.msobodytext, li.msobodytext, div.msobodytext {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"body text char"; margin:0in; margin-bottom:.0001pt; mso-pagination:none; mso-layout-grid-align:none; text-autospace:none; font-size:12.0pt; font-family:"arial",sans-serif; mso-fareast-font-family:calibri; mso-bidi-font-family:latha;} p.msobodytextindent, li.msobodytextindent, div.msobodytextindent {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"body text indent char"; margin-top:0in; margin-right:0in; margin-bottom:6.0pt; margin-left:.25in; line-height:115%; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman";} p.msosubtitle, li.msosubtitle, div.msosubtitle {mso-style-priority:99; mso-style-unhide:no; mso-style-qformat:yes; mso-style-link:"subtitle char"; mso-style-next:normal; margin-top:0in; margin-right:0in; margin-bottom:16.0pt; margin-left:0in; mso-pagination:widow-orphan lines-together; page-break-after:avoid; font-size:15.0pt; font-family:"times new roman",serif; mso-fareast-font-family:arial; color:#666666;} p.msobodytextfirstindent2, li.msobodytextfirstindent2, div.msobodytextfirstindent2 {mso-style-noshow:yes; mso-style-priority:99; mso-style-parent:"body text indent"; mso-style-link:"body text first indent 2 char"; margin-top:0in; margin-right:0in; margin-bottom:6.0pt; margin-left:.25in; text-indent:10.5pt; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman";} a:link, span.msohyperlink {mso-style-noshow:yes; mso-style-priority:99; color:blue; text-decoration:underline; text-underline:single;} a:visited, span.msohyperlinkfollowed {mso-style-noshow:yes; mso-style-priority:99; color:purple; mso-themecolor:followedhyperlink; text-decoration:underline; text-underline:single;} p {mso-style-noshow:yes; mso-style-priority:99; mso-margin-top-alt:auto; margin-right:0in; mso-margin-bottom-alt:auto; margin-left:0in; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:arial;} p.msocommentsubject, li.msocommentsubject, div.msocommentsubject {mso-style-noshow:yes; mso-style-priority:99; mso-style-parent:"comment text"; mso-style-link:"comment subject char"; mso-style-next:"comment text"; margin-top:0in; margin-right:0in; margin-bottom:10.0pt; margin-left:0in; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; font-weight:bold;} p.msoacetate, li.msoacetate, div.msoacetate {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"balloon text char"; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:8.0pt; font-family:"tahoma",sans-serif; mso-fareast-font-family:"times new roman"; mso-ansi-language:en-ca;} span.msoplaceholdertext {mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; color:gray;} p.msonospacing, li.msonospacing, div.msonospacing {mso-style-priority:1; mso-style-unhide:no; mso-style-qformat:yes; mso-style-parent:""; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:calibri; mso-fareast-theme-font:minor-latin; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi;} p.msormpane, li.msormpane, div.msormpane {mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-parent:""; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi;} p.msolistparagraph, li.msolistparagraph, div.msolistparagraph {mso-style-priority:34; mso-style-unhide:no; mso-style-qformat:yes; mso-style-link:"list paragraph char"; margin-top:0in; margin-right:0in; margin-bottom:10.0pt; margin-left:.5in; mso-add-space:auto; line-height:115%; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman";} p.msolistparagraphcxspfirst, li.msolistparagraphcxspfirst, div.msolistparagraphcxspfirst {mso-style-priority:34; mso-style-unhide:no; mso-style-qformat:yes; mso-style-link:"list paragraph char"; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.5in; margin-bottom:.0001pt; mso-add-space:auto; line-height:115%; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman";} p.msolistparagraphcxspmiddle, li.msolistparagraphcxspmiddle, div.msolistparagraphcxspmiddle {mso-style-priority:34; mso-style-unhide:no; mso-style-qformat:yes; mso-style-link:"list paragraph char"; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.5in; margin-bottom:.0001pt; mso-add-space:auto; line-height:115%; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman";} p.msolistparagraphcxsplast, li.msolistparagraphcxsplast, div.msolistparagraphcxsplast {mso-style-priority:34; mso-style-unhide:no; mso-style-qformat:yes; mso-style-link:"list paragraph char"; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:10.0pt; margin-left:.5in; mso-add-space:auto; line-height:115%; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman";} p.msobibliography, li.msobibliography, div.msobibliography {mso-style-noshow:yes; mso-style-priority:37; mso-style-next:normal; margin-top:0in; margin-right:0in; margin-bottom:10.0pt; margin-left:0in; line-height:115%; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman";} span.heading1char {mso-style-name:"heading 1 char"; mso-style-priority:9; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"heading 1"; mso-ansi-font-size:14.0pt; mso-bidi-font-size:14.0pt; font-family:"cambria",serif; mso-ascii-font-family:cambria; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:cambria; mso-bidi-font-family:"times new roman"; color:#365f91; font-weight:bold;} span.heading2char {mso-style-name:"heading 2 char"; mso-style-noshow:yes; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"heading 2"; mso-ansi-font-size:13.0pt; mso-bidi-font-size:13.0pt; font-family:"cambria",serif; mso-ascii-font-family:cambria; mso-ascii-theme-font:major-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:major-fareast; mso-hansi-font-family:cambria; mso-hansi-theme-font:major-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:major-bidi; color:#4f81bd; mso-themecolor:accent1; font-weight:bold;} span.heading3char {mso-style-name:"heading 3 char"; mso-style-noshow:yes; mso-style-priority:9; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"heading 3"; mso-ansi-font-size:13.5pt; mso-bidi-font-size:13.5pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; mso-ansi-language:en-us; font-weight:bold;} span.heading4char {mso-style-name:"heading 4 char"; mso-style-noshow:yes; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"heading 4"; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:arial; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:#666666; mso-ansi-language:en-us;} span.heading5char {mso-style-name:"heading 5 char"; mso-style-noshow:yes; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"heading 5"; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:arial; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:#666666; mso-ansi-language:en-us;} span.heading6char {mso-style-name:"heading 6 char"; mso-style-noshow:yes; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"heading 6"; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:arial; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:#666666; mso-ansi-language:en-us; font-style:italic; mso-bidi-font-style:normal;} p.msonormal0, li.msonormal0, div.msonormal0 {mso-style-name:msonormal; mso-style-priority:99; mso-style-unhide:no; mso-margin-top-alt:auto; margin-right:0in; mso-margin-bottom-alt:auto; margin-left:0in; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:arial;} span.footnotetextchar {mso-style-name:"footnote text char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"footnote text"; mso-ansi-font-size:10.0pt; mso-bidi-font-size:10.0pt; mso-ansi-language:en-us;} span.commenttextchar {mso-style-name:"comment text char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"comment text"; mso-ansi-font-size:10.0pt; mso-bidi-font-size:10.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:calibri; mso-bidi-font-family:"times new roman"; mso-ansi-language:en-us;} span.headerchar {mso-style-name:"header char"; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:header; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman";} span.footerchar {mso-style-name:"footer char"; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:footer; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman";} span.titlechar {mso-style-name:"title char"; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:title; mso-ansi-font-size:26.0pt; mso-bidi-font-size:26.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:arial; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; mso-ansi-language:en-us;} span.bodytextchar {mso-style-name:"body text char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"body text"; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"arial",sans-serif; mso-ascii-font-family:arial; mso-fareast-font-family:calibri; mso-hansi-font-family:arial; mso-bidi-font-family:latha; mso-ansi-language:en-us;} span.bodytextindentchar {mso-style-name:"body text indent char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"body text indent"; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:calibri; mso-bidi-font-family:"times new roman"; mso-ansi-language:en-us;} span.subtitlechar {mso-style-name:"subtitle char"; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:subtitle; mso-ansi-font-size:15.0pt; mso-bidi-font-size:15.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:arial; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:#666666; mso-ansi-language:en-us;} span.bodytextfirstindent2char {mso-style-name:"body text first indent 2 char"; mso-style-noshow:yes; mso-style-unhide:no; mso-style-locked:yes; mso-style-parent:"body text indent char"; mso-style-link:"body text first indent 2"; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; mso-ansi-language:en-us;} span.commentsubjectchar {mso-style-name:"comment subject char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-parent:"comment text char"; mso-style-link:"comment subject"; mso-ansi-font-size:10.0pt; mso-bidi-font-size:10.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:calibri; mso-bidi-font-family:"times new roman"; mso-ansi-language:en-us; font-weight:bold;} span.balloontextchar {mso-style-name:"balloon text char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"balloon text"; mso-ansi-font-size:8.0pt; mso-bidi-font-size:8.0pt; font-family:"tahoma",sans-serif; mso-ascii-font-family:tahoma; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:tahoma; mso-bidi-font-family:tahoma;} span.listparagraphchar {mso-style-name:"list paragraph char"; mso-style-priority:34; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"list paragraph"; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:calibri; mso-bidi-font-family:"times new roman"; mso-ansi-language:en-us;} p.heading11, li.heading11, div.heading11 {mso-style-name:"heading 11"; mso-style-priority:9; mso-style-unhide:no; mso-style-qformat:yes; mso-style-next:normal; margin-top:24.0pt; margin-right:0in; margin-bottom:0in; margin-left:0in; margin-bottom:.0001pt; line-height:115%; mso-pagination:widow-orphan lines-together; page-break-after:avoid; mso-outline-level:1; font-size:14.0pt; font-family:"cambria",serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:#365f91; font-weight:bold;} p.titlemain, li.titlemain, div.titlemain {mso-style-name:"title main"; mso-style-priority:99; mso-style-unhide:no; mso-style-qformat:yes; margin:0in; margin-bottom:.0001pt; line-height:80%; mso-pagination:widow-orphan; font-size:24.0pt; mso-bidi-font-size:12.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:black; font-weight:bold; mso-bidi-font-weight:normal;} p.titlesubtitle, li.titlesubtitle, div.titlesubtitle {mso-style-name:"title subtitle"; mso-style-priority:99; mso-style-unhide:no; mso-style-qformat:yes; margin-top:0in; margin-right:0in; margin-bottom:30.0pt; margin-left:0in; line-height:110%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"georgia",serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:black; font-style:italic; mso-bidi-font-style:normal;} p.titleauthors, li.titleauthors, div.titleauthors {mso-style-name:"title authors"; mso-style-priority:99; mso-style-unhide:no; mso-style-qformat:yes; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; mso-bidi-font-size:12.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:black; font-weight:bold; mso-bidi-font-weight:normal;} p.references, li.references, div.references {mso-style-name:references; mso-style-priority:99; mso-style-unhide:no; mso-style-qformat:yes; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.5in; margin-bottom:.0001pt; text-indent:-.5in; mso-pagination:widow-orphan; font-size:11.0pt; mso-bidi-font-size:12.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman";} p.heading1nonumber, li.heading1nonumber, div.heading1nonumber {mso-style-name:"heading 1 \(no number\)"; mso-style-priority:99; mso-style-unhide:no; mso-style-qformat:yes; mso-style-parent:"heading 1"; mso-style-next:"heading 1"; margin-top:12.0pt; margin-right:0in; margin-bottom:0in; margin-left:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan lines-together; page-break-after:avoid; mso-outline-level:1; font-size:14.0pt; font-family:"cambria",serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:#365f91; mso-ansi-language:en-ca; font-weight:bold;} p.listbulletspaces, li.listbulletspaces, div.listbulletspaces {mso-style-name:"list bullet \(+spaces\)"; mso-style-priority:99; mso-style-unhide:no; mso-style-qformat:yes; mso-style-parent:"list bullet"; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.25in; margin-bottom:.0001pt; text-indent:-.25in; mso-pagination:widow-orphan; mso-list:l0 level1 lfo1; font-size:11.0pt; mso-bidi-font-size:12.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman";} span.st {mso-style-name:st; mso-style-unhide:no;} span.apple-converted-space {mso-style-name:apple-converted-space; mso-style-unhide:no;} span.slug-doi {mso-style-name:slug-doi; mso-style-unhide:no;} span.heading1char1 {mso-style-name:"heading 1 char1"; mso-style-priority:9; mso-style-unhide:no; mso-ansi-font-size:16.0pt; mso-bidi-font-size:16.0pt; font-family:"cambria",serif; mso-ascii-font-family:cambria; mso-ascii-theme-font:major-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:major-fareast; mso-hansi-font-family:cambria; mso-hansi-theme-font:major-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:major-bidi; color:#365f91; mso-themecolor:accent1; mso-themeshade:191;} span.mention1 {mso-style-name:mention1; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; color:#2b579a; background:#e6e6e6;} span.unresolvedmention1 {mso-style-name:"unresolved mention1"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; color:#605e5c; background:#e1dfdd;} span.unresolvedmention2 {mso-style-name:"unresolved mention2"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; color:#605e5c; background:#e1dfdd;} span.spelle {mso-style-name:""; mso-spl-e:yes;} span.grame {mso-style-name:""; mso-gram-e:yes;} .msochpdefault {mso-style-type:export-only; mso-default-props:yes; font-size:10.0pt; mso-ansi-font-size:10.0pt; mso-bidi-font-size:10.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:calibri; mso-fareast-theme-font:minor-latin; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; mso-ansi-language:en-ca;} /* page definitions */ @page {mso-footnote-separator:url("art_grabowsky_29657_files/header.htm") fs; mso-footnote-continuation-separator:url("art_grabowsky_29657_files/header.htm") fcs; mso-endnote-separator:url("art_grabowsky_29657_files/header.htm") es; mso-endnote-continuation-separator:url("art_grabowsky_29657_files/header.htm") ecs;} @page wordsection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in; mso-header-margin:.5in; mso-footer-margin:.5in; mso-header:url("art_grabowsky_29657_files/header.htm") h1; mso-footer:url("art_grabowsky_29657_files/header.htm") f1; mso-paper-source:0;} div.wordsection1 {page:wordsection1;} /* list definitions */ @list l0 {mso-list-id:-119; mso-list-type:simple; mso-list-template-ids:877445814;} @list l0:level1 {mso-level-number-format:bullet; mso-level-style-link:"list bullet"; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.25in; text-indent:-.25in; font-family:wingdings;} @list l1 {mso-list-id:128286151; mso-list-type:hybrid; mso-list-template-ids:831183430 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;} @list l1:level1 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.25in; text-indent:-.25in; font-family:symbol;} @list l1:level2 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.75in; text-indent:-.25in; font-family:"courier new";} @list l1:level3 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.25in; text-indent:-.25in; font-family:wingdings;} @list l1:level4 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.75in; text-indent:-.25in; font-family:symbol;} @list l1:level5 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.25in; text-indent:-.25in; font-family:"courier new";} @list l1:level6 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.75in; text-indent:-.25in; font-family:wingdings;} @list l1:level7 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.25in; text-indent:-.25in; font-family:symbol;} @list l1:level8 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.75in; text-indent:-.25in; font-family:"courier new";} @list l1:level9 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.25in; text-indent:-.25in; font-family:wingdings;} @list l2 {mso-list-id:389504314; mso-list-type:hybrid; mso-list-template-ids:2007165398 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;} @list l2:level1 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:symbol;} @list l2:level2 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:"courier new";} @list l2:level3 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:wingdings;} @list l2:level4 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:symbol;} @list l2:level5 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:"courier new";} @list l2:level6 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:wingdings;} @list l2:level7 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:symbol;} @list l2:level8 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:"courier new";} @list l2:level9 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:wingdings;} @list l3 {mso-list-id:474566917; mso-list-type:hybrid; mso-list-template-ids:-665388430 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;} @list l3:level1 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.25in; text-indent:-.25in; font-family:symbol;} @list l3:level2 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.75in; text-indent:-.25in; font-family:"courier new";} @list l3:level3 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.25in; text-indent:-.25in; font-family:wingdings;} @list l3:level4 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.75in; text-indent:-.25in; font-family:symbol;} @list l3:level5 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.25in; text-indent:-.25in; font-family:"courier new";} @list l3:level6 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.75in; text-indent:-.25in; font-family:wingdings;} @list l3:level7 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.25in; text-indent:-.25in; font-family:symbol;} @list l3:level8 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.75in; text-indent:-.25in; font-family:"courier new";} @list l3:level9 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.25in; text-indent:-.25in; font-family:wingdings;} @list l4 {mso-list-id:487938832; mso-list-type:hybrid; mso-list-template-ids:1201437324 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;} @list l4:level1 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.25in; text-indent:-.25in; font-family:symbol;} @list l4:level2 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.75in; text-indent:-.25in; font-family:"courier new";} @list l4:level3 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.25in; text-indent:-.25in; font-family:wingdings;} @list l4:level4 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.75in; text-indent:-.25in; font-family:symbol;} @list l4:level5 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.25in; text-indent:-.25in; font-family:"courier new";} @list l4:level6 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.75in; text-indent:-.25in; font-family:wingdings;} @list l4:level7 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.25in; text-indent:-.25in; font-family:symbol;} @list l4:level8 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.75in; text-indent:-.25in; font-family:"courier new";} @list l4:level9 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.25in; text-indent:-.25in; font-family:wingdings;} @list l5 {mso-list-id:541671380; mso-list-type:hybrid; mso-list-template-ids:-1755406670 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;} @list l5:level1 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.25in; text-indent:-.25in; font-family:symbol;} @list l5:level2 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.75in; text-indent:-.25in; font-family:"courier new";} @list l5:level3 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.25in; text-indent:-.25in; font-family:wingdings;} @list l5:level4 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.75in; text-indent:-.25in; font-family:symbol;} @list l5:level5 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.25in; text-indent:-.25in; font-family:"courier new";} @list l5:level6 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.75in; text-indent:-.25in; font-family:wingdings;} @list l5:level7 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.25in; text-indent:-.25in; font-family:symbol;} @list l5:level8 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.75in; text-indent:-.25in; font-family:"courier new";} @list l5:level9 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.25in; text-indent:-.25in; font-family:wingdings;} @list l6 {mso-list-id:638413621; mso-list-type:hybrid; mso-list-template-ids:1628452398 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;} @list l6:level1 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.25in; text-indent:-.25in; font-family:symbol;} @list l6:level2 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.75in; text-indent:-.25in; font-family:"courier new";} @list l6:level3 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.25in; text-indent:-.25in; font-family:wingdings;} @list l6:level4 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.75in; text-indent:-.25in; font-family:symbol;} @list l6:level5 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.25in; text-indent:-.25in; font-family:"courier new";} @list l6:level6 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.75in; text-indent:-.25in; font-family:wingdings;} @list l6:level7 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.25in; text-indent:-.25in; font-family:symbol;} @list l6:level8 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.75in; text-indent:-.25in; font-family:"courier new";} @list l6:level9 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.25in; text-indent:-.25in; font-family:wingdings;} @list l7 {mso-list-id:640888003; mso-list-type:hybrid; mso-list-template-ids:-797904790 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;} @list l7:level1 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.25in; text-indent:-.25in; font-family:symbol;} @list l7:level2 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.75in; text-indent:-.25in; font-family:"courier new";} @list l7:level3 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.25in; text-indent:-.25in; font-family:wingdings;} @list l7:level4 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.75in; text-indent:-.25in; font-family:symbol;} @list l7:level5 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.25in; text-indent:-.25in; font-family:"courier new";} @list l7:level6 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.75in; text-indent:-.25in; font-family:wingdings;} @list l7:level7 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.25in; text-indent:-.25in; font-family:symbol;} @list l7:level8 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.75in; text-indent:-.25in; font-family:"courier new";} @list l7:level9 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.25in; text-indent:-.25in; font-family:wingdings;} @list l8 {mso-list-id:677778210; mso-list-type:hybrid; mso-list-template-ids:1296730746 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;} @list l8:level1 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.25in; text-indent:-.25in; font-family:symbol;} @list l8:level2 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.75in; text-indent:-.25in; font-family:"courier new";} @list l8:level3 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.25in; text-indent:-.25in; font-family:wingdings;} @list l8:level4 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.75in; text-indent:-.25in; font-family:symbol;} @list l8:level5 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.25in; text-indent:-.25in; font-family:"courier new";} @list l8:level6 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.75in; text-indent:-.25in; font-family:wingdings;} @list l8:level7 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.25in; text-indent:-.25in; font-family:symbol;} @list l8:level8 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.75in; text-indent:-.25in; font-family:"courier new";} @list l8:level9 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.25in; text-indent:-.25in; font-family:wingdings;} @list l9 {mso-list-id:725957192; mso-list-type:hybrid; mso-list-template-ids:1502931906 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;} @list l9:level1 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.25in; text-indent:-.25in; font-family:symbol;} @list l9:level2 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.75in; text-indent:-.25in; font-family:"courier new";} @list l9:level3 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.25in; text-indent:-.25in; font-family:wingdings;} @list l9:level4 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.75in; text-indent:-.25in; font-family:symbol;} @list l9:level5 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.25in; text-indent:-.25in; font-family:"courier new";} @list l9:level6 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.75in; text-indent:-.25in; font-family:wingdings;} @list l9:level7 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.25in; text-indent:-.25in; font-family:symbol;} @list l9:level8 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.75in; text-indent:-.25in; font-family:"courier new";} @list l9:level9 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.25in; text-indent:-.25in; font-family:wingdings;} @list l10 {mso-list-id:830870097; mso-list-type:hybrid; mso-list-template-ids:-1557993474 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;} @list l10:level1 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.25in; text-indent:-.25in; font-family:symbol;} @list l10:level2 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.75in; text-indent:-.25in; font-family:"courier new";} @list l10:level3 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.25in; text-indent:-.25in; font-family:wingdings;} @list l10:level4 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.75in; text-indent:-.25in; font-family:symbol;} @list l10:level5 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.25in; text-indent:-.25in; font-family:"courier new";} @list l10:level6 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.75in; text-indent:-.25in; font-family:wingdings;} @list l10:level7 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.25in; text-indent:-.25in; font-family:symbol;} @list l10:level8 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.75in; text-indent:-.25in; font-family:"courier new";} @list l10:level9 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.25in; text-indent:-.25in; font-family:wingdings;} @list l11 {mso-list-id:966004581; mso-list-type:hybrid; mso-list-template-ids:-1710086584 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;} @list l11:level1 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.25in; text-indent:-.25in; font-family:symbol;} @list l11:level2 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.75in; text-indent:-.25in; font-family:"courier new";} @list l11:level3 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.25in; text-indent:-.25in; font-family:wingdings;} @list l11:level4 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.75in; text-indent:-.25in; font-family:symbol;} @list l11:level5 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.25in; text-indent:-.25in; font-family:"courier new";} @list l11:level6 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.75in; text-indent:-.25in; font-family:wingdings;} @list l11:level7 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.25in; text-indent:-.25in; font-family:symbol;} @list l11:level8 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.75in; text-indent:-.25in; font-family:"courier new";} @list l11:level9 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.25in; text-indent:-.25in; font-family:wingdings;} @list l12 {mso-list-id:983436621; mso-list-type:hybrid; mso-list-template-ids:-868817466 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;} @list l12:level1 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.25in; text-indent:-.25in; font-family:symbol;} @list l12:level2 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.75in; text-indent:-.25in; font-family:"courier new";} @list l12:level3 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.25in; text-indent:-.25in; font-family:wingdings;} @list l12:level4 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.75in; text-indent:-.25in; font-family:symbol;} @list l12:level5 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.25in; text-indent:-.25in; font-family:"courier new";} @list l12:level6 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.75in; text-indent:-.25in; font-family:wingdings;} @list l12:level7 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.25in; text-indent:-.25in; font-family:symbol;} @list l12:level8 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.75in; text-indent:-.25in; font-family:"courier new";} @list l12:level9 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.25in; text-indent:-.25in; font-family:wingdings;} @list l13 {mso-list-id:1086652996; mso-list-type:hybrid; mso-list-template-ids:-1656746534 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;} @list l13:level1 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:symbol;} @list l13:level2 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:"courier new";} @list l13:level3 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:wingdings;} @list l13:level4 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:symbol;} @list l13:level5 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:"courier new";} @list l13:level6 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:wingdings;} @list l13:level7 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:symbol;} @list l13:level8 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:"courier new";} @list l13:level9 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:wingdings;} @list l14 {mso-list-id:1299846132; mso-list-type:hybrid; mso-list-template-ids:1347606912 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;} @list l14:level1 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.25in; text-indent:-.25in; font-family:symbol;} @list l14:level2 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.75in; text-indent:-.25in; font-family:"courier new";} @list l14:level3 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.25in; text-indent:-.25in; font-family:wingdings;} @list l14:level4 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.75in; text-indent:-.25in; font-family:symbol;} @list l14:level5 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.25in; text-indent:-.25in; font-family:"courier new";} @list l14:level6 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.75in; text-indent:-.25in; font-family:wingdings;} @list l14:level7 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.25in; text-indent:-.25in; font-family:symbol;} @list l14:level8 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.75in; text-indent:-.25in; font-family:"courier new";} @list l14:level9 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.25in; text-indent:-.25in; font-family:wingdings;} @list l15 {mso-list-id:1387292979; mso-list-type:hybrid; mso-list-template-ids:758174862 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;} @list l15:level1 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.25in; text-indent:-.25in; font-family:symbol;} @list l15:level2 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.75in; text-indent:-.25in; font-family:"courier new";} @list l15:level3 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.25in; text-indent:-.25in; font-family:wingdings;} @list l15:level4 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.75in; text-indent:-.25in; font-family:symbol;} @list l15:level5 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.25in; text-indent:-.25in; font-family:"courier new";} @list l15:level6 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.75in; text-indent:-.25in; font-family:wingdings;} @list l15:level7 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.25in; text-indent:-.25in; font-family:symbol;} @list l15:level8 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.75in; text-indent:-.25in; font-family:"courier new";} @list l15:level9 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.25in; text-indent:-.25in; font-family:wingdings;} @list l16 {mso-list-id:1463187088; mso-list-type:hybrid; mso-list-template-ids:2022352640 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;} @list l16:level1 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.25in; text-indent:-.25in; font-family:symbol;} @list l16:level2 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.75in; text-indent:-.25in; font-family:"courier new";} @list l16:level3 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.25in; text-indent:-.25in; font-family:wingdings;} @list l16:level4 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.75in; text-indent:-.25in; font-family:symbol;} @list l16:level5 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.25in; text-indent:-.25in; font-family:"courier new";} @list l16:level6 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.75in; text-indent:-.25in; font-family:wingdings;} @list l16:level7 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.25in; text-indent:-.25in; font-family:symbol;} @list l16:level8 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.75in; text-indent:-.25in; font-family:"courier new";} @list l16:level9 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.25in; text-indent:-.25in; font-family:wingdings;} @list l17 {mso-list-id:1526140453; mso-list-type:hybrid; mso-list-template-ids:-1393943542 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;} @list l17:level1 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:symbol;} @list l17:level2 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:"courier new";} @list l17:level3 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:wingdings;} @list l17:level4 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:symbol;} @list l17:level5 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:"courier new";} @list l17:level6 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:wingdings;} @list l17:level7 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:symbol;} @list l17:level8 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:"courier new";} @list l17:level9 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:wingdings;} @list l18 {mso-list-id:1621954721; mso-list-type:hybrid; mso-list-template-ids:-225674740 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;} @list l18:level1 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.25in; text-indent:-.25in; font-family:symbol;} @list l18:level2 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.75in; text-indent:-.25in; font-family:"courier new";} @list l18:level3 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.25in; text-indent:-.25in; font-family:wingdings;} @list l18:level4 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.75in; text-indent:-.25in; font-family:symbol;} @list l18:level5 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.25in; text-indent:-.25in; font-family:"courier new";} @list l18:level6 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.75in; text-indent:-.25in; font-family:wingdings;} @list l18:level7 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.25in; text-indent:-.25in; font-family:symbol;} @list l18:level8 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.75in; text-indent:-.25in; font-family:"courier new";} @list l18:level9 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.25in; text-indent:-.25in; font-family:wingdings;} @list l19 {mso-list-id:1638563738; mso-list-type:hybrid; mso-list-template-ids:317856006 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;} @list l19:level1 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.25in; text-indent:-.25in; font-family:symbol;} @list l19:level2 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.75in; text-indent:-.25in; font-family:"courier new";} @list l19:level3 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.25in; text-indent:-.25in; font-family:wingdings;} @list l19:level4 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.75in; text-indent:-.25in; font-family:symbol;} @list l19:level5 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.25in; text-indent:-.25in; font-family:"courier new";} @list l19:level6 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.75in; text-indent:-.25in; font-family:wingdings;} @list l19:level7 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.25in; text-indent:-.25in; font-family:symbol;} @list l19:level8 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.75in; text-indent:-.25in; font-family:"courier new";} @list l19:level9 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.25in; text-indent:-.25in; font-family:wingdings;} @list l20 {mso-list-id:1727143364; mso-list-type:hybrid; mso-list-template-ids:504651632 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;} @list l20:level1 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.25in; text-indent:-.25in; font-family:symbol;} @list l20:level2 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.75in; text-indent:-.25in; font-family:"courier new";} @list l20:level3 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.25in; text-indent:-.25in; font-family:wingdings;} @list l20:level4 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.75in; text-indent:-.25in; font-family:symbol;} @list l20:level5 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.25in; text-indent:-.25in; font-family:"courier new";} @list l20:level6 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.75in; text-indent:-.25in; font-family:wingdings;} @list l20:level7 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.25in; text-indent:-.25in; font-family:symbol;} @list l20:level8 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.75in; text-indent:-.25in; font-family:"courier new";} @list l20:level9 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.25in; text-indent:-.25in; font-family:wingdings;} @list l21 {mso-list-id:1848204328; mso-list-type:hybrid; mso-list-template-ids:1402252996 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;} @list l21:level1 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.0in; text-indent:-.25in; font-family:symbol;} @list l21:level2 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.5in; text-indent:-.25in; font-family:"courier new";} @list l21:level3 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.0in; text-indent:-.25in; font-family:wingdings;} @list l21:level4 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.5in; text-indent:-.25in; font-family:symbol;} @list l21:level5 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.0in; text-indent:-.25in; font-family:"courier new";} @list l21:level6 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.5in; text-indent:-.25in; font-family:wingdings;} @list l21:level7 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.0in; text-indent:-.25in; font-family:symbol;} @list l21:level8 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.5in; text-indent:-.25in; font-family:"courier new";} @list l21:level9 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:5.0in; text-indent:-.25in; font-family:wingdings;} @list l22 {mso-list-id:2026663980; mso-list-type:hybrid; mso-list-template-ids:326023790 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;} @list l22:level1 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:symbol;} @list l22:level2 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:"courier new";} @list l22:level3 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:wingdings;} @list l22:level4 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:symbol;} @list l22:level5 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:"courier new";} @list l22:level6 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:wingdings;} @list l22:level7 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:symbol;} @list l22:level8 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:"courier new";} @list l22:level9 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:wingdings;} @list l23 {mso-list-id:2058121355; mso-list-type:hybrid; mso-list-template-ids:-879695872 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;} @list l23:level1 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.25in; text-indent:-.25in; font-family:symbol;} @list l23:level2 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.75in; text-indent:-.25in; font-family:"courier new";} @list l23:level3 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.25in; text-indent:-.25in; font-family:wingdings;} @list l23:level4 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.75in; text-indent:-.25in; font-family:symbol;} @list l23:level5 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.25in; text-indent:-.25in; font-family:"courier new";} @list l23:level6 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.75in; text-indent:-.25in; font-family:wingdings;} @list l23:level7 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.25in; text-indent:-.25in; font-family:symbol;} @list l23:level8 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.75in; text-indent:-.25in; font-family:"courier new";} @list l23:level9 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.25in; text-indent:-.25in; font-family:wingdings;} ol {margin-bottom:0in;} ul {margin-bottom:0in;} --> </style> <!--[if gte mso 10]> <style> /* style definitions */ table.msonormaltable {mso-style-name:"table normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; mso-ansi-language:en-ca;} table.msotablegrid {mso-style-name:"table grid"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-priority:39; mso-style-unhide:no; border:solid windowtext 1.0pt; mso-border-alt:solid windowtext .5pt; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-border-insideh:.5pt solid windowtext; mso-border-insidev:.5pt solid windowtext; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:simsun; mso-bidi-font-family:arial; mso-ansi-language:en-ca; mso-fareast-language:zh-cn; mso-bidi-language:hi;} table.msotable15plain1 {mso-style-name:"plain table 1"; mso-tstyle-rowband-size:1; mso-tstyle-colband-size:1; mso-style-priority:41; mso-style-unhide:no; border:solid #bfbfbf 1.0pt; mso-border-themecolor:background1; mso-border-themeshade:191; mso-border-alt:solid #bfbfbf .5pt; mso-border-themecolor:background1; mso-border-themeshade:191; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-border-insideh:.5pt solid #bfbfbf; mso-border-insideh-themecolor:background1; mso-border-insideh-themeshade:191; mso-border-insidev:.5pt solid #bfbfbf; mso-border-insidev-themecolor:background1; mso-border-insidev-themeshade:191; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; mso-ansi-language:en-ca;} table.msotable15plain1firstrow {mso-style-name:"plain table 1"; mso-table-condition:first-row; mso-style-priority:41; mso-style-unhide:no; mso-ansi-font-weight:bold; mso-bidi-font-weight:bold;} table.msotable15plain1lastrow {mso-style-name:"plain table 1"; mso-table-condition:last-row; mso-style-priority:41; mso-style-unhide:no; mso-tstyle-border-top:1.5pt double #bfbfbf; mso-tstyle-border-top-themecolor:background1; mso-tstyle-border-top-themeshade:191; mso-ansi-font-weight:bold; mso-bidi-font-weight:bold;} table.msotable15plain1firstcol {mso-style-name:"plain table 1"; mso-table-condition:first-column; mso-style-priority:41; mso-style-unhide:no; mso-ansi-font-weight:bold; mso-bidi-font-weight:bold;} table.msotable15plain1lastcol {mso-style-name:"plain table 1"; mso-table-condition:last-column; mso-style-priority:41; mso-style-unhide:no; mso-ansi-font-weight:bold; mso-bidi-font-weight:bold;} table.msotable15plain1oddcolumn {mso-style-name:"plain table 1"; mso-table-condition:odd-column; mso-style-priority:41; mso-style-unhide:no; mso-tstyle-shading:#f2f2f2; mso-tstyle-shading-themecolor:background1; mso-tstyle-shading-themeshade:242;} table.msotable15plain1oddrow {mso-style-name:"plain table 1"; mso-table-condition:odd-row; mso-style-priority:41; mso-style-unhide:no; mso-tstyle-shading:#f2f2f2; mso-tstyle-shading-themecolor:background1; mso-tstyle-shading-themeshade:242;} table.msotable15plain3 {mso-style-name:"plain table 3"; mso-tstyle-rowband-size:1; mso-tstyle-colband-size:1; mso-style-priority:43; mso-style-unhide:no; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; mso-ansi-language:en-ca;} table.msotable15plain3firstrow {mso-style-name:"plain table 3"; mso-table-condition:first-row; mso-style-priority:43; mso-style-unhide:no; mso-tstyle-border-bottom:.5pt solid #7f7f7f; mso-tstyle-border-bottom-themecolor:text1; mso-tstyle-border-bottom-themetint:128; text-transform:uppercase; mso-ansi-font-weight:bold; mso-bidi-font-weight:bold;} table.msotable15plain3lastrow {mso-style-name:"plain table 3"; mso-table-condition:last-row; mso-style-priority:43; mso-style-unhide:no; mso-tstyle-border-top:cell-none; text-transform:uppercase; mso-ansi-font-weight:bold; mso-bidi-font-weight:bold;} table.msotable15plain3firstcol {mso-style-name:"plain table 3"; mso-table-condition:first-column; mso-style-priority:43; mso-style-unhide:no; mso-tstyle-border-right:.5pt solid #7f7f7f; mso-tstyle-border-right-themecolor:text1; mso-tstyle-border-right-themetint:128; text-transform:uppercase; mso-ansi-font-weight:bold; mso-bidi-font-weight:bold;} table.msotable15plain3lastcol {mso-style-name:"plain table 3"; mso-table-condition:last-column; mso-style-priority:43; mso-style-unhide:no; mso-tstyle-border-left:cell-none; text-transform:uppercase; mso-ansi-font-weight:bold; mso-bidi-font-weight:bold;} table.msotable15plain3oddcolumn {mso-style-name:"plain table 3"; mso-table-condition:odd-column; mso-style-priority:43; mso-style-unhide:no; mso-tstyle-shading:#f2f2f2; mso-tstyle-shading-themecolor:background1; mso-tstyle-shading-themeshade:242;} table.msotable15plain3oddrow {mso-style-name:"plain table 3"; mso-table-condition:odd-row; mso-style-priority:43; mso-style-unhide:no; mso-tstyle-shading:#f2f2f2; mso-tstyle-shading-themecolor:background1; mso-tstyle-shading-themeshade:242;} table.msotable15plain3necell {mso-style-name:"plain table 3"; mso-table-condition:ne-cell; mso-style-priority:43; mso-style-unhide:no; mso-tstyle-border-left:cell-none;} table.msotable15plain3nwcell {mso-style-name:"plain table 3"; mso-table-condition:nw-cell; mso-style-priority:43; mso-style-unhide:no; mso-tstyle-border-right:cell-none;} table.msotable15plain4 {mso-style-name:"plain table 4"; mso-tstyle-rowband-size:1; mso-tstyle-colband-size:1; mso-style-priority:44; mso-style-unhide:no; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; mso-ansi-language:en-ca;} table.msotable15plain4firstrow {mso-style-name:"plain table 4"; mso-table-condition:first-row; mso-style-priority:44; mso-style-unhide:no; mso-ansi-font-weight:bold; mso-bidi-font-weight:bold;} table.msotable15plain4lastrow {mso-style-name:"plain table 4"; mso-table-condition:last-row; mso-style-priority:44; mso-style-unhide:no; mso-ansi-font-weight:bold; mso-bidi-font-weight:bold;} table.msotable15plain4firstcol {mso-style-name:"plain table 4"; mso-table-condition:first-column; mso-style-priority:44; mso-style-unhide:no; mso-ansi-font-weight:bold; mso-bidi-font-weight:bold;} table.msotable15plain4lastcol {mso-style-name:"plain table 4"; mso-table-condition:last-column; mso-style-priority:44; mso-style-unhide:no; mso-ansi-font-weight:bold; mso-bidi-font-weight:bold;} table.msotable15plain4oddcolumn {mso-style-name:"plain table 4"; mso-table-condition:odd-column; mso-style-priority:44; mso-style-unhide:no; mso-tstyle-shading:#f2f2f2; mso-tstyle-shading-themecolor:background1; mso-tstyle-shading-themeshade:242;} table.msotable15plain4oddrow {mso-style-name:"plain table 4"; mso-table-condition:odd-row; mso-style-priority:44; mso-style-unhide:no; mso-tstyle-shading:#f2f2f2; mso-tstyle-shading-themecolor:background1; mso-tstyle-shading-themeshade:242;} table.msotablegridlight {mso-style-name:"table grid light"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-priority:40; mso-style-unhide:no; border:solid #bfbfbf 1.0pt; mso-border-themecolor:background1; mso-border-themeshade:191; mso-border-alt:solid #bfbfbf .5pt; mso-border-themecolor:background1; mso-border-themeshade:191; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-border-insideh:.5pt solid #bfbfbf; mso-border-insideh-themecolor:background1; mso-border-insideh-themeshade:191; mso-border-insidev:.5pt solid #bfbfbf; mso-border-insidev-themecolor:background1; mso-border-insidev-themeshade:191; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; mso-ansi-language:en-ca;} </style> <![endif]--><!--[if gte mso 9]><xml> <o:shapedefaults v:ext="edit" spidmax="1026"/> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext="edit"> <o:idmap v:ext="edit" data="1"/> </o:shapelayout></xml><![endif]--> </head> <body lang=en-us link=blue vlink=purple style='tab-interval:.5in'> <div class=wordsection1> <p class=msonormal style='mso-outline-level:1'><i style='mso-bidi-font-style: normal'><span lang=en-ca style='font-family:"palatino linotype",serif; mso-bidi-font-weight:bold'>review article<o:p></o:p></span></i></p> <p class=msonormal><b><span lang=en-ca style='font-size:10.0pt;font-family: "palatino linotype",serif'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal><b><span style='font-size:11.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>the effectiveness of library instruction for graduate/professional students: a systematic review and meta-analysis<o:p></o:p></span></b></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-weight:bold'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-bidi-font-weight:bold'>adelia <span class=spelle>grabowsky</span><o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-bidi-font-weight:bold'>research &amp; instruction librarian<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-bidi-font-weight:bold'>auburn university libraries<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-bidi-font-weight:bold'>auburn, alabama, united states of america <o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-bidi-font-weight:bold'>email: </span><span lang=en-ca><a href="mailto:abg0011@auburn.edu"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us;mso-bidi-font-weight:bold'>abg0011@auburn.edu</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-bidi-font-weight:bold'> </span><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us;mso-bidi-font-weight:bold'><o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-bidi-font-weight:bold'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-bidi-font-weight:bold'>liza <span class=spelle>weisbrod</span><o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-bidi-font-weight:bold'>research &amp; instruction librarian<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-bidi-font-weight:bold'>auburn university libraries<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-bidi-font-weight:bold'>auburn, alabama, united states of america <o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-bidi-font-weight:bold'>email: </span><span lang=en-ca><a href="mailto:weisbel@auburn.edu"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us;mso-bidi-font-weight:bold'>weisbel@auburn.edu</span></a></span><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us;mso-bidi-font-weight:bold'><o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-bidi-font-weight:bold'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b><span lang=en-ca style='font-size:10.0pt;font-family: "palatino linotype",serif'>received</span></b><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-weight: bold'>: </span><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-bidi-font-weight:bold'>1 oct. 2019</span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-weight:bold'><span style='mso-tab-count:5'>                                                                 </span><b>accepted</b>: </span><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-bidi-font-weight:bold'>20 dec. 2019</span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-weight:bold'><o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:9.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:arial;mso-ansi-language:en-us;mso-no-proof:yes'><!--[if gte vml 1]><v:shapetype id="_x0000_t75" coordsize="21600,21600" o:spt="75" o:preferrelative="t" path="m@4@5l@4@11@9@11@9@5xe" filled="f" stroked="f"> <v:stroke joinstyle="miter"/> <v:formulas> <v:f eqn="if linedrawn pixellinewidth 0"/> <v:f eqn="sum @0 1 0"/> <v:f eqn="sum 0 0 @1"/> <v:f eqn="prod @2 1 2"/> <v:f eqn="prod @3 21600 pixelwidth"/> <v:f eqn="prod @3 21600 pixelheight"/> <v:f eqn="sum @0 0 1"/> <v:f eqn="prod @6 1 2"/> <v:f eqn="prod @7 21600 pixelwidth"/> <v:f eqn="sum @8 21600 0"/> <v:f eqn="prod @7 21600 pixelheight"/> <v:f eqn="sum @10 21600 0"/> </v:formulas> <v:path o:extrusionok="f" gradientshapeok="t" o:connecttype="rect"/> <o:lock v:ext="edit" aspectratio="t"/> </v:shapetype><v:shape id="picture_x0020_4" o:spid="_x0000_i1029" type="#_x0000_t75" alt="cc-ca_logo_xl" style='width:7.2pt;height:7.2pt;visibility:visible; mso-wrap-style:square'> <v:imagedata src="art_grabowsky_29657_files/image003.png" o:title="cc-ca_logo_xl" cropright="49941f"/> </v:shape><![endif]--><![if !vml]><img border=0 width=10 height=10 src="art_grabowsky_29657_files/image001.png" alt="cc-ca_logo_xl" v:shapes="picture_x0020_4"><![endif]></span><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:arial'><span style='mso-spacerun:yes'> </span>2020 </span><span class=spelle><span style='font-size:9.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-bidi-font-weight:bold'>grabowsky</span></span><span style='font-size:9.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us;mso-bidi-font-weight:bold'> and <span class=spelle>weisbrod</span></span><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif'>. this is an open access article distributed under the terms of the creative commons</span><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"cambria math"'>‐</span><span lang=en-ca style='font-size: 9.0pt;font-family:"palatino linotype",serif'>attribution</span><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"cambria math"'>‐</span><span class=spelle><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif'>noncommercial</span></span><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"cambria math"'>‐</span><span lang=en-ca style='font-size: 9.0pt;font-family:"palatino linotype",serif'>share alike license 4.0 international (</span><span lang=en-ca><a href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><span style='font-size:9.0pt;font-family:"palatino linotype",serif'>http://creativecommons.org/licenses/by-nc-sa/4.0/</span></a></span><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif'>), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.<o:p></o:p></span></p> <div style='mso-element:para-border-div;border:none;border-bottom:solid black 1.0pt; mso-border-bottom-alt:solid black .25pt;padding:0in 0in 1.0pt 0in'> <p class=msobodytext style='border:none;mso-border-bottom-alt:solid black .25pt; padding:0in;mso-padding-alt:0in 0in 1.0pt 0in'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-ca'><o:p>&nbsp;</o:p></span></p> <p class=msobodytext style='border:none;mso-border-bottom-alt:solid black .25pt; padding:0in;mso-padding-alt:0in 0in 1.0pt 0in'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-ca'><o:p>&nbsp;</o:p></span></p> <p class=msobodytext style='border:none;mso-border-bottom-alt:solid black .25pt; padding:0in;mso-padding-alt:0in 0in 1.0pt 0in'><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif;color:blue; mso-ansi-language:en-ca'>doi</span><span lang=en-ca style='font-size:9.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-ca'>: 10.18438/eblip29657<o:p></o:p></span></p> <p class=msobodytext style='border:none;mso-border-bottom-alt:solid black .25pt; padding:0in;mso-padding-alt:0in 0in 1.0pt 0in'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-ca'><o:p>&nbsp;</o:p></span></p> </div> <p class=msonormal style='margin-left:.5in'><b style='mso-bidi-font-weight: normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal style='margin-left:.5in'><b style='mso-bidi-font-weight: normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>abstract<o:p></o:p></span></b></p> <p class=msonormal style='margin-left:.5in'><b style='mso-bidi-font-weight: normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal style='margin-left:.5in'><b style='mso-bidi-font-weight: normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>objective </span></b><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>this study sought to assess the effectiveness of library instruction for increasing information literacy skills and/or knowledge among graduate and professional students. <o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in'><b style='mso-bidi-font-weight: normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal style='margin-left:.5in'><b style='mso-bidi-font-weight: normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>methods -</span></b><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'> a search was conducted in library literature and information science index (h. w. wilson); library, information science &amp; technology abstracts; medline; cinahl; eric; library and information science abstracts (lisa); and proquest dissertations and theses global. studies were included if they were published between 2000 and 2019, in english, reported on library instruction for graduate or professional students, and objectively measured change in information literacy knowledge/skills.<o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in'><span lang=en-ca style='font-size: 10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in'><b style='mso-bidi-font-weight: normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>results </span></b><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>sixteen studies were included in the systematic review; 12 of the 16 studies included sufficient information to be included in the meta-analysis. the overall effect of library instruction was significant [smd = 1.03, se=0.19, <i style='mso-bidi-font-style: normal'>z</i>=5.49, <i style='mso-bidi-font-style:normal'>p</i>&lt;.0001, 95% ci=0.66-1.40], meaning that on average, a student scored about one standard deviation higher on an information literacy assessment after library instruction. high heterogeneity indicated a need for subgroup analysis, which showed a significant moderation of effect by discipline of students, but none by format of instruction. however, subgroup analysis must be viewed with caution due to the small number of studies in several of the subgroups.<o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in'><b style='mso-bidi-font-weight: normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal style='margin-left:.5in'><b style='mso-bidi-font-weight: normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>conclusions </span></b><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>this meta-analysis indicates that library instruction for graduate students is effective in increasing information literacy knowledge and/or skills. however, to strengthen the accuracy of results of future meta-analyses, there is a need for more precise descriptions of instructional sessions as well as more complete data reporting by authors of primary studies. there is also a need for the publication of more studies, particularly studies of hybrid and online instruction.<o:p></o:p></span></p> <div style='mso-element:para-border-div;border:none;border-bottom:solid windowtext 1.0pt; mso-border-bottom-alt:solid windowtext .5pt;padding:0in 0in 1.0pt 0in'> <p class=msobodytext style='border:none;mso-border-bottom-alt:solid windowtext .5pt; padding:0in;mso-padding-alt:0in 0in 1.0pt 0in'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-fareast-font-family: "times new roman";mso-bidi-font-family:"times new roman";mso-ansi-language: en-ca'><o:p>&nbsp;</o:p></span></p> </div> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'>introduction <o:p></o:p></span></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>regional accrediting standards for colleges and universities emphasize the need for institutions to engage in effective assessment of desired student learning outcomes to substantiate results (baker, 2002). one common learning outcome for university students is the ability to locate, evaluate, and manage information (i.e., to be information literate) (markle, brenneman, jackson, burrus, &amp; robbins, 2013). although information literacy (il) instruction should be interwoven throughout the curriculum, most academic librarians are invested in collaborating with subject faculty to provide library specific instruction to improve the il skills of students (mcgowan, gonzalez, &amp; <span class=spelle>stanny</span>, 2016) and are interested in assessing the value of that instruction. library instruction to improve il is often seen as essential only for undergraduates (<span class=spelle>blummer</span>, 2009). however, students in graduate/professional studies do not always have the requisite skills needed for graduate level study and research (conway, 2011), which suggests they may also benefit from library instruction targeted specifically to graduate students. for example, <span class=spelle>o’clair</span> (2013) found that graduate students felt more prepared to tackle thesis research after taking a for-credit information literacy course.<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'>aims<o:p></o:p></span></b></p> <p class=msonospacing><span style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonospacing><span style='font-size:10.0pt;font-family:"palatino linotype",serif'>this study includes both a systematic review and meta-analysis. the systematic review examines the current state of library instruction for graduate students and seeks to determine what formats of instruction are used, the content of instructional sessions, and how instruction is assessed. one issue with assessment of library instruction is that small sample sizes may limit the ability to identify actual change (coe, 2002; higgins, 2019).<span style='mso-spacerun:yes'>  </span>meta-analysis is one way to combine the results of multiple studies to improve the statistical power and lessen the possibility of failing to identify a true difference (<span class=spelle>shinogle</span>, 2012; thornton &amp; lee, 2000). although a meta-analysis has been completed on the effectiveness of library instruction for undergraduates (<span class=spelle>koufogiannakis</span> &amp; wiebe, 2006), none was found for graduate/professional students. this study looks at the effectiveness of library instruction for graduate and/or professional students and if that effectiveness varies by discipline, format, or duration of instruction. specific research questions include:<o:p></o:p></span></p> <p class=msonospacing><span style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b><i><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>systematic review<o:p></o:p></span></i></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b><i><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></i></b></p> <p class=msonormal style='margin-left:.5in;text-indent:-.25in;page-break-after: avoid;mso-outline-level:2;mso-list:l21 level1 lfo2'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol;mso-ansi-language:en-us'><span style='mso-list: ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>what formats are used to provide library instruction to graduate/professional students?<o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.25in;page-break-after: avoid;mso-outline-level:2;mso-list:l21 level1 lfo2'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol;mso-ansi-language:en-us'><span style='mso-list: ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>what content is covered in instruction sessions for graduate/professional students?<o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.25in;page-break-after: avoid;mso-outline-level:2;mso-list:l21 level1 lfo2'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol;mso-ansi-language:en-us'><span style='mso-list: ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>how is instruction for graduate/professional students assessed?<o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;page-break-after:avoid;mso-outline-level: 2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b><i><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>meta-analysis<o:p></o:p></span></i></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b><i><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></i></b></p> <p class=msonormal style='margin-left:.5in;text-indent:-.25in;page-break-after: avoid;mso-outline-level:2;mso-list:l21 level1 lfo2'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol;mso-ansi-language:en-us'><span style='mso-list: ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>does library instruction for graduate/professional students result in improved information literacy knowledge and/or skill?<o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.25in;page-break-after: avoid;mso-outline-level:2;mso-list:l21 level1 lfo2'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol;mso-ansi-language:en-us'><span style='mso-list: ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>does effectiveness of library instruction for graduate/professional students vary by format, duration, or discipline?<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'>methods<o:p></o:p></span></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>this study was conducted using the guidelines established in the prisma (preferred reporting items for systematic reviews and meta-analyses) statement (moher et al., 2009). concepts for library instruction, graduate/professional students, and study type along with synonyms and subject headings were searched on 11 march 2019 in library literature and information science index (h. w. wilson); library, information science &amp; technology abstracts; medline; cumulative index of nursing and allied health literature (cinahl); education resources information center (eric); library and information science abstracts (lisa); and proquest dissertations and theses global (see appendix a for search strategies). academic libraries underwent significant changes in the late 1990’s with the advent of personal computers and electronic access to journal articles. since those changes also affected library instruction, searches were limited to a date range of 2000 to 2019. literature searches were also limited to english language, but no restrictions were placed on type of publication.<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>eligibility criteria<o:p></o:p></span></i></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></i></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>to be eligible for inclusion in this review, studies had to include instruction for graduate or professional students related to information literacy (il) knowledge and/or skills. the instruction had to be provided wholly or in part by one or more librarians, and studies had to include a measure of change in il knowledge/skills. finally, studies had to include either one or more groups with a preand post-measure of il knowledge/skills or both a treatment and control group with a post-assessment of il knowledge/skills. graduate students included students studying for a master’s or phd in any subject area (other than library science), while professional students included any health science student working on a clinical doctorate, including medical, dental, pharmacy, veterinary, nursing, and audiology students. synthesis studies, studies written in a language other than english, and studies involving medical residents or library science students were excluded. additionally, studies were excluded if the measure of change in il skills/knowledge was self-reported by students.<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>study selection<o:p></o:p></span></i></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></i></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>the number of studies examined at each stage of the review process are shown in figure 1. both authors independently examined each source, first at the title and abstract stage, then later at the full text stage. after each screening level, the authors compared individual decisions for congruence; conflicting decisions were resolved by discussion. <o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='text-indent:.5in;line-height:200%'><span lang=en-ca style='font-size:10.0pt;line-height:200%;font-family:"palatino linotype",serif; mso-no-proof:yes'><!--[if gte vml 1]><v:shape id="picture_x0020_7" o:spid="_x0000_i1028" type="#_x0000_t75" style='width:456pt;height:515.4pt;visibility:visible; mso-wrap-style:square'> <v:imagedata src="art_grabowsky_29657_files/image005.png" o:title=""/> </v:shape><![endif]--><![if !vml]><img border=0 width=608 height=687 src="art_grabowsky_29657_files/image002.png" v:shapes="picture_x0020_7"><![endif]></span><span lang=en-ca style='font-size:10.0pt;line-height:200%;font-family:"palatino linotype",serif'><o:p></o:p></span></p> <p class=msonormal style='margin-right:-59.25pt'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>figure 1 <br> prisma flow diagram (moher, <span class=spelle>liberati</span>, <span class=spelle>tetzlaff</span>, &amp; altman, 2009)<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>data extraction<o:p></o:p></span></i></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></i></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>each author extracted data from half the studies to an excel spreadsheet, and then checked data extracted by the other author for accuracy and completeness. data collected included information about participants (level of study, discipline, and geographic location), the intervention (description, duration, format, content taught, and content assessed), the assessment/test (timing, validity, and availability), and study statistics (sample size, mean, and standard deviation). some studies did not include standard deviation but did provide individual scores. in those cases, standard deviations were calculated using excel. seven authors were emailed for additional data, and three replied with the requested information.<o:p></o:p></span></p> <p class=msonormal style='margin-right:-59.25pt'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>quality assessment<o:p></o:p></span></i></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><i style='mso-bidi-font-style:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></i></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>quality of each included study was assessed using an instrument developed to critically appraise educational interventions (morrison, sullivan, murray, &amp; jolly, 1999).<b style='mso-bidi-font-weight:normal'> </b><span style='mso-bidi-font-weight:bold'>the checklist includes nine questions addressing content, context, outcomes, study design, and methods. </span>both authors independently answered the nine questions for each study with ‘yes’, ‘no’, or ‘can’t tell’ and then met to compare results. differences in answers to individual questions were settled by discussion and reference back to the article. the authors then voted to include or exclude the article based on preponderance of ‘yes’ answers with more weight given to questions 5 and 6. those two questions addressed whether the study design was able to answer the posed question and whether the methods used were appropriately measuring the phenomena of interest. all articles except for one received ‘yes’ answers for both question 5 and 6 from both authors. that article received ‘no’ to both questions from both authors and was discarded due to quality concerns. <o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>data synthesis<o:p></o:p></span></i></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></i></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>analysis was carried out with r [version 3.5.0 (23 april 2018)] (r core team, 2018) using the <span class=spelle><i style='mso-bidi-font-style:normal'>metafor</i></span> package (<span class=spelle>viechtbauer</span>, 2010) (see appendix b for data). standardized mean difference (smd) was the chosen effect size. smd represents the difference in the preand post-intervention means divided by the pooled standard deviation (<span class=spelle>borenstein</span>, hedges, higgins, &amp; rothstein, 2009). when a study includes a small number of samples, smd may be biased; therefore, smd with a correction factor (hedge’s <i style='mso-bidi-font-style:normal'>g</i>) was used (<span class=spelle>borenstein</span> et al., 2009). hedge’s <i style='mso-bidi-font-style:normal'>g</i> is computed in r using the smdh (<i style='mso-bidi-font-style:normal'>standardized mean difference</i> with heteroscedastic population variances in the two groups) function; smdh requires input of the sample size, preand post-mean (m) scores, and preand post-standard deviations (sd) (see appendix c for sample r code). when sd was not provided and could not be calculated from available data, an estimate based on the average sd of all other studies was used (furukawa, <span class=spelle>barbui</span>, cipriani, brambilla, &amp; watanabe, 2006). all meta-analyses were conducted using a random-effects model, which assumes that there is not one “true” value that all studies are seeking, but instead that values may vary among studies due to differences in how the studies are carried out (<span class=spelle>bown</span> &amp; sutton, 2010). a random-effects model is recommended when there is assumed to be heterogeneity in outcome estimates (<span class=spelle>bown</span> &amp; sutton, 2010), which was the case in this meta-analysis. the <i style='mso-bidi-font-style:normal'>i</i><sup>2</sup> statistic was used to quantify the heterogeneity of effect sizes. <i style='mso-bidi-font-style:normal'>i</i><sup>2 </sup>ranges from 0 to 100%; <span class=spelle>borenstein</span> et al. (2009) suggest that a small <i style='mso-bidi-font-style:normal'>i</i><sup>2</sup> (close to 0) indicates that only a small part of the observed variance reflects actual differences in effect size. however, larger numbers indicate a larger proportion of the observed variance is real and suggest a need to carry out subgroup analysis or meta-regression in order to explain the heterogeneity (<span class=spelle>borenstein</span> et al., 2009).<span style='mso-spacerun:yes'>  </span><o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>publication bias<o:p></o:p></span></i></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></i></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>publication bias can arise through several means, for example, authors may decide not to report non-significant findings, or journals may refuse to publish negative studies. since meta-analysis depends on finding all studies that answer a specific research question, publication bias has the potential to distort findings (song, hooper, &amp; loke, 2013; thornton &amp; lee, 2000). this study followed the recommendations of song et al. (2013), using a comprehensive search that did not limit results to only journal articles. in addition, publication bias was assessed with both a funnel plot and through use of rosenthal’s fail-safe number. the fail-safe number is an estimate of the number of non-significant studies required to nullify the results of the meta-analysis; rosenthal (1979) suggests a fail-safe number greater than 5<i style='mso-bidi-font-style:normal'>n</i> + 10 (where n is the number of studies) is sufficient to consider publication bias inconsequential (see appendix c for sample code to calculate rosenthal’s fail-safe number in r).<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'>results<o:p></o:p></span></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>description of studies<o:p></o:p></span></i></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></i></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>the final 16 studies included one dissertation and 15 journal articles, with publication years ranging from 2004 to 2018 (see appendix d for list of studies and table 1 for characteristics of studies). the majority of studies took place in the united states (n=12), but there was one study each from canada, the united kingdom, australia, and tanzania. a variety of disciplines were represented (see figure 2), with the largest number of studies including medical students (n=4) followed closely by students in education (n=3). overall, health science students were included more often, with 10 of 16 studies involving students from some area of health sciences. the 16 studies included 12 studies that were preand post-assessments of one or more groups (repeated measures); the remaining four were post-assessment of a treatment and control group (independent groups). sample size of the repeated measures (rm) studies ranged from 10 to 61 students, while sample size of the independent group (ig) studies ranged from 37 to 300 students. the most common format of instruction was face-to-face (f2f). six studies included only f2f instruction, five included only hybrid instruction (a combination of face-to-face and some sort of online instruction), and two studies examined only online instruction. an additional two studies compared f2f to online, and one study compared all three formats--f2f, hybrid, and online. duration of instruction was not reported for every study. durations that were reported varied widely; for example, for library instruction provided within a subject class, time of instruction ranged from one 70-minute session to two 3-hour sessions (see table 2).<span style='mso-spacerun:yes'>  </span><o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><i style='mso-bidi-font-style:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'>content of instruction<o:p></o:p></span></i></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><i style='mso-bidi-font-style:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></i></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us;mso-bidi-font-weight:bold'>while not every study included a detailed description of instructional content, certain themes emerged in the studies (see table 1). all classes taught database searching strategies (n=16). the classes for health sciences students (n=10) provided instruction on biomedical databases (pubmed, international pharmaceutical abstracts, medline, cinahl, and others) while classes for non-health science students (n=6) taught a variety of resources including chemistry and education databases and sources of data from united states government agencies. search strategies taught included boolean logic (n=6), limiters (n=5), and <span class=spelle>mesh</span> (medical subject headings) vocabulary (n=5). other topics included critical appraisal skills (n=6), citation styles and citation managers (n=4), ethical use of information (including plagiarism) (n=3), and library-specific resources and services (n=4). <o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us;mso-bidi-font-weight:bold'><o:p>&nbsp;</o:p></span></p> <p class=msonospacing><i style='mso-bidi-font-style:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif'>assessment <o:p></o:p></span></i></p> <p class=msonospacing><span style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonospacing><span style='font-size:10.0pt;font-family:"palatino linotype",serif'>six of the 16 studies employed a validated assessment tool (see table 3). of those six, two used an instrument based on the fresno test (ramos, schafer, &amp; <span class=spelle>tracz</span>, 2003), one used the rrsa (research readiness self-assessment) (<span class=spelle>ivanitskaya</span>, <span class=spelle>laus</span>, &amp; casey, 2004), and two studies used rubrics validated in-house. the remaining study used backwards design and the information literacy competency standards for higher education (association of college and research libraries [acrl], 2000) to develop a validated assessment tool. nine studies provided the full questionnaire or assessment in the article. <o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us;mso-bidi-font-weight:bold'>seven of the 16 studies referenced the <i style='mso-bidi-font-style:normal'>information literacy competency standards for higher education</i> </span><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>(association of college and research libraries, 2000)<span style='mso-bidi-font-weight:bold'> with one of the seven also referencing the <i style='mso-bidi-font-style:normal'>framework for information literacy for higher education</i> (</span>association of college and research libraries<span style='mso-bidi-font-weight:bold'>, 2016). of the nine remaining studies that did not reference acrl standards, three referenced other standards; for example, the proposed core competencies for data information literacy (carlson, <span class=spelle>fosmire</span>, miller, &amp; nelson, 2011). <o:p></o:p></span></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>most studies (n=9) used objective tests such as multiple choice and true/false questions to measure recall of knowledge, while five studies measured the application of knowledge by evaluating search strategies or scenario responses. two studies measured the application of knowledge through short answer and multiple-choice questions that required hands-on use of databases. <o:p></o:p></span></p> <p class=msonospacing><span style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>table 1<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>characteristics of studies<o:p></o:p></span></p> <table class=msotablegridlight border=1 cellspacing=0 cellpadding=0 style='border-collapse:collapse;border:none;mso-border-alt:solid #bfbfbf .5pt; mso-border-themecolor:background1;mso-border-themeshade:191;mso-yfti-tbllook: 1184;mso-padding-alt:0in 5.4pt 0in 5.4pt'> <thead> <tr style='mso-yfti-irow:0;mso-yfti-firstrow:yes;height:75.1pt'> <td valign=top style='border:solid #bfbfbf 1.0pt;mso-border-themecolor:background1; mso-border-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt; height:75.1pt'> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>shortened citation<o:p></o:p></span></b></p> </td> <td valign=top style='border:solid #bfbfbf 1.0pt;mso-border-themecolor:background1; mso-border-themeshade:191;border-left:none;mso-border-left-alt:solid #bfbfbf .5pt; mso-border-left-themecolor:background1;mso-border-left-themeshade:191; mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor:background1; mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height:75.1pt'> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>participants<br> level<br> discipline<br> location<o:p></o:p></span></b></p> </td> <td valign=top style='border:solid #bfbfbf 1.0pt;mso-border-themecolor:background1; mso-border-themeshade:191;border-left:none;mso-border-left-alt:solid #bfbfbf .5pt; mso-border-left-themecolor:background1;mso-border-left-themeshade:191; mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor:background1; mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height:75.1pt'> <p class=msonospacing><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: "times new roman"'>design<o:p></o:p></span></b></p> <p class=msonospacing><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"times new roman"'>repeated measures=rm<o:p></o:p></span></p> <p class=msonospacing><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"times new roman"'>independent groups=ig<o:p></o:p></span></p> </td> <td valign=top style='border:solid #bfbfbf 1.0pt;mso-border-themecolor:background1; mso-border-themeshade:191;border-left:none;mso-border-left-alt:solid #bfbfbf .5pt; mso-border-left-themecolor:background1;mso-border-left-themeshade:191; mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor:background1; mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height:75.1pt'> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>format<o:p></o:p></span></b></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>f2f (face to face)<br> online<br> hybrid<o:p></o:p></span></p> </td> <td valign=top style='border:solid #bfbfbf 1.0pt;mso-border-themecolor:background1; mso-border-themeshade:191;border-left:none;mso-border-left-alt:solid #bfbfbf .5pt; mso-border-left-themecolor:background1;mso-border-left-themeshade:191; mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor:background1; mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height:75.1pt'> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>description of intervention<o:p></o:p></span></b></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>duration of intervention<o:p></o:p></span></b></p> </td> <td valign=top style='border:solid #bfbfbf 1.0pt;mso-border-themecolor:background1; mso-border-themeshade:191;border-left:none;mso-border-left-alt:solid #bfbfbf .5pt; mso-border-left-themecolor:background1;mso-border-left-themeshade:191; mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor:background1; mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height:75.1pt'> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>content taught<o:p></o:p></span></b></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>content assessed<o:p></o:p></span></b></p> </td> </tr> </thead> <tr style='mso-yfti-irow:1;height:167.35pt'> <td valign=top style='border:solid #bfbfbf 1.0pt;mso-border-themecolor:background1; mso-border-themeshade:191;border-top:none;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor:background1; mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height:167.35pt'> <p class=msonormal><span class=spelle><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us'>aronoff</span></span><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>, 2017<br> teaching evidence-based practice…<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 167.35pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>students from 8 health profession programs (medical, dental, pharmacy, occupational therapy (ot), physical therapy (pt), social work, speech language pathology, dietetics)<br> usa<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 167.35pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>rm<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>pre/post-assessment<br> 1 tx group<br> (n=39)<br style='mso-special-character:line-break'> <![if !supportlinebreaknewline]><br style='mso-special-character:line-break'> <![endif]><o:p></o:p></span></p> </td> <td nowrap valign=top style='border-top:none;border-left:none;border-bottom: solid #bfbfbf 1.0pt;mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade: 191;border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 167.35pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>hybrid<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 167.35pt'> <p class=msolistparagraphcxspfirst style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l20 level1 lfo3'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>2 online <span class=grame>evidence based</span> practice (ebp) learning modules hosted on the learning management system.<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l20 level1 lfo3'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>participation in a facilitated in-person interprofessional small group learning experience.<o:p></o:p></span></p> <p class=msolistparagraphcxsplast style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l20 level1 lfo3'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>each module 1 hour long.<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 167.35pt'> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>taught:</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> module 1: ebp principles, critical appraisal strategies. module 2: pubmed instruction, medical subject headings (<span class=spelle>mesh</span>) terms.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>assessed:</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> module 1: knowledge of ebp components; development of patient/population, intervention, comparison, and outcome (pico) questions; study designs; critical appraisal strategies. module 2: pubmed searching strategies; using <span class=spelle>mesh</span> terms; limiting with pubmed filters. clinical scenario: creation of a pico question, utilization of information resources, study design, search characteristics, and critical appraisal.<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:2;height:39.15pt'> <td rowspan=4 valign=top style='border:solid #bfbfbf 1.0pt;mso-border-themecolor: background1;mso-border-themeshade:191;border-top:none;mso-border-top-alt: solid #bfbfbf .5pt;mso-border-top-themecolor:background1;mso-border-top-themeshade: 191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor:background1; mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height:39.15pt'> <p class=msonormal><span class=spelle><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us'>beile</span></span><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>, 2004<br> does the medium matter?<o:p></o:p></span></p> </td> <td rowspan=4 valign=top style='border-top:none;border-left:none;border-bottom: solid #bfbfbf 1.0pt;mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade: 191;border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 39.15pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>master’s, doctoral, and certificate-seeking education students<br> usa<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 39.15pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>rm<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>pre/post-test<br> 3 tx groups<br style='mso-special-character:line-break'> <![if !supportlinebreaknewline]><br style='mso-special-character:line-break'> <![endif]><o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 39.15pt'></td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 39.15pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>study compared 3 modes of delivery.<br style='mso-special-character:line-break'> <![if !supportlinebreaknewline]><br style='mso-special-character:line-break'> <![endif]><o:p></o:p></span></p> </td> <td rowspan=4 valign=top style='border-top:none;border-left:none;border-bottom: solid #bfbfbf 1.0pt;mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade: 191;border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 39.15pt'> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>taught</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>: f2f: demonstration of relevant library databases followed by an activity to allow the students to apply the lesson.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>tutorial: 4 interactive modules. principles of library and information research, navigation, and search techniques, practical application of search techniques, locating, evaluating, and citing information.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>assessed:</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> conceptual knowledge (how information is produced and organized), knowledge of database-searching skills (identifying databases and using boolean logic), knowledge of institution-specific information (accessing databases and awareness of services).<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:3;height:39.1pt'> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 39.1pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>group 1 (f2f, on-campus)<br> (n=16)<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 39.1pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>f2f<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 39.1pt'> <p class=msolistparagraphcxspfirst style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l12 level1 lfo4'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>an on-campus class with face-to-face library instruction.<o:p></o:p></span></p> <p class=msolistparagraphcxsplast style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l12 level1 lfo4'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>70-minute demo followed by application activity.<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:4;height:45.0pt'> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 45.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>group 2 (web tutorial, on-campus)<br> (n=19)<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 45.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>hybrid<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 45.0pt'> <p class=msolistparagraphcxspfirst style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l15 level1 lfo5'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>an on-campus class with web-based library tutorial consisting of 4 interactive modules.<o:p></o:p></span></p> <p class=msolistparagraphcxsplast style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l15 level1 lfo5'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>participants spent an average of 80 minutes on modules.<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:5;height:45.0pt'> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 45.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>group 3 (web tutorial, web-based class)<br> (n=14)<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 45.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>online<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 45.0pt'> <p class=msolistparagraphcxspfirst style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l3 level1 lfo6'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>a web-based class with a web-based library tutorial consisting of 4 interactive modules.<o:p></o:p></span></p> <p class=msolistparagraphcxsplast style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l3 level1 lfo6'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>participants spent an average of 80 minutes on modules.<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:6;height:105.0pt'> <td valign=top style='border:solid #bfbfbf 1.0pt;mso-border-themecolor:background1; mso-border-themeshade:191;border-top:none;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor:background1; mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height:105.0pt'> <p class=msonormal><span class=spelle><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us'>chiarella</span></span><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>, 2014<br> information literacy skills retention…<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 105.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>pharmacy students<br> 1st year<br> usa<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 105.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>rm<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>pre/post-test<br> 1 tx group<br> (n=61)<o:p></o:p></span></p> </td> <td nowrap valign=top style='border-top:none;border-left:none;border-bottom: solid #bfbfbf 1.0pt;mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade: 191;border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 105.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>f2f<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 105.0pt'> <p class=msolistparagraphcxspfirst style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l11 level1 lfo7'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>librarian presented library skills material 4 times during the fall semester of p1 year.<o:p></o:p></span></p> <p class=msolistparagraphcxsplast style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l11 level1 lfo7'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>no indication of length of session.<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 105.0pt'> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>taught:</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> basic database search strategies; google searching versus biomedical databases; pubmed, embase, and medline; endnote.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>assessed</span></b><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us'>: <span class=spelle>mesh</span> subject heading searches, boolean operators, and limits.<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:7;height:165.0pt'> <td valign=top style='border:solid #bfbfbf 1.0pt;mso-border-themecolor:background1; mso-border-themeshade:191;border-top:none;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor:background1; mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height:165.0pt'> <p class=msonormal><span class=spelle><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us'>dorsch</span></span><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>, 2004<br> impact of an evidence-based…<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 165.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>medical students <br> 3rd year<br> usa<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 165.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>rm<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>pre/post-skills assessment<br> 1 tx group<br> (n=33, pre)<br> (n=30, post)<o:p></o:p></span></p> </td> <td nowrap valign=top style='border-top:none;border-left:none;border-bottom: solid #bfbfbf 1.0pt;mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade: 191;border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 165.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>f2f<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 165.0pt'> <p class=msolistparagraphcxspfirst style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l1 level1 lfo8'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>8 1-hour weekly seminars.<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l1 level1 lfo8'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>weeks 1-2 taught by librarian.<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l1 level1 lfo8'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>weeks 3-5 taught by medical school faculty.<o:p></o:p></span></p> <p class=msolistparagraphcxsplast style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l1 level1 lfo8'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>weeks 6-8 practice sessions.<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 165.0pt'> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>taught:</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> librarians: to define <span class=grame>evidence based</span> medicine (ebm), formulate clinical questions based on a standardized case scenario; identify and review ebm search strategies and resources.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>assessed: </span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>formulating a clinical question, using effective strategies to identify the best clinical literature to answer the question, analyzing the relevance and validity of the retrieved article.<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:8;height:60.0pt'> <td valign=top style='border:solid #bfbfbf 1.0pt;mso-border-themecolor:background1; mso-border-themeshade:191;border-top:none;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor:background1; mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height:60.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>emmett, 2007<br> assessing information literacy skills…<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 60.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>1st, 2nd year phd chemistry students<br> usa<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 60.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>rm<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>pre/post-test<br> 1 tx group<br> (n=16)<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>(using 2006 data)<o:p></o:p></span></p> </td> <td nowrap valign=top style='border-top:none;border-left:none;border-bottom: solid #bfbfbf 1.0pt;mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade: 191;border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 60.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>f2f<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 60.0pt'> <p class=msolistparagraphcxspfirst style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l8 level1 lfo9'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>1-hour credit course taught by librarian.<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l8 level1 lfo9'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>75 minutes per week for one semester.<o:p></o:p></span></p> <p class=msolistparagraphcxsplast style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l8 level1 lfo9'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>chem 720, “bibliography of chemistry.”<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 60.0pt'> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>taught:</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> major resources in the chemical and biomedical literature, research strategies, bibliographic management, ethical use of information.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>assessed:</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> searching, citation style, databases, plagiarism.<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:9;height:62.5pt'> <td valign=top style='border:solid #bfbfbf 1.0pt;mso-border-themecolor:background1; mso-border-themeshade:191;border-top:none;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor:background1; mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height:62.5pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>grant, 2006<br> developing and evaluating an interactive…<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 62.5pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>master’s, phd students (nursing, ot, pt)<br> united kingdom<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 62.5pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>rm<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>pre/post-assessment<br> 1 tx group<br> (n=13, pre)<br> (n=11, post)<br> (n=10, extended)<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 62.5pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>hybrid<br> (online tutorial within ebp module)<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 62.5pt'> <p class=msolistparagraphcxspfirst style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l18 level1 lfo10'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>during a 12-week ebp module, 2 sessions (3 hours each) were allocated to information skills development.<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l18 level1 lfo10'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>an online tutorial was used in-class for both sessions, and students were asked to complete between-session exercises using the tutorial.<o:p></o:p></span></p> <p class=msolistparagraphcxsplast style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l18 level1 lfo10'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>no indication of length of tutorial.<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 62.5pt'> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>taught</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>: tutorial: the rationale for a literature search; how a database works; seven search steps covering clarifying a search question, breaking down the question, <span class=spelle>mesh</span>, free text searching, boolean operators, refining the search; final tips.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>lecture: formulating a search question; selecting search terms; building up a search strategy; limiting searches.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>assessed:</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> short-term, a literature search; longer term, systematic literature search on a topic of choice, describing the literature search process and providing search strategies, then selecting and critically appraising two papers. both assessed by skills checklist such as boolean operators, use of <span class=spelle>mesh</span>/indexing terms, application of limits, and whether a manageable and relevant number references were retrieved.<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:10;height:1.25in'> <td valign=top style='border:solid #bfbfbf 1.0pt;mso-border-themecolor:background1; mso-border-themeshade:191;border-top:none;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor:background1; mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height:1.25in'> <p class=msonormal><span class=spelle><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us'>ilic</span></span><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>, 2012<br> teaching evidence-based medicine…<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 1.25in'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>medical students<br> 3rd year<br> australia<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 1.25in'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>ig<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>1 tx group<br> (n= 60)<br> 1 control group<br> (n=37)<br> block randomization (groups of 4)<o:p></o:p></span></p> </td> <td nowrap valign=top style='border-top:none;border-left:none;border-bottom: solid #bfbfbf 1.0pt;mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade: 191;border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 1.25in'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>hybrid<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 1.25in'> <p class=msolistparagraphcxspfirst style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l7 level1 lfo11'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>ebm literature searching skills workshop (intervention group attended workshop, control group did not).<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l7 level1 lfo11'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>workshop consisted of formal presentation by librarian followed by an interactive, computer-based searching session and self-directed learning exercises with support provided by librarian if needed.<o:p></o:p></span></p> <p class=msolistparagraphcxsplast style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l7 level1 lfo11'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>workshop 2 hours long.<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 1.25in'> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>taught:</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> how to construct an answerable question from the clinical environment, major sources of medical information, how to effectively and efficiently search the medical literature to identify the best available evidence to answer the question.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>assessed:</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> writing a clinical question, identifying information sources, identifying appropriate study types, performing an effective literature search.<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:11;height:67.0pt'> <td valign=top style='border:solid #bfbfbf 1.0pt;mso-border-themecolor:background1; mso-border-themeshade:191;border-top:none;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor:background1; mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height:67.0pt'> <p class=msonormal><span class=spelle><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us'>ivanitskaya</span></span><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>, 2008<br> how does a pre-assessment of off-campus…<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 67.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>master of science in administration students<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>usa<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 67.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>rm<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>pre/post-test<br> 1 tx group<br> (n=14)<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 67.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>f2f<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 67.0pt'> <p class=msolistparagraphcxspfirst style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l19 level1 lfo12'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>library instruction during a class session at the beginning of the course.<o:p></o:p></span></p> <p class=msolistparagraphcxsplast style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l19 level1 lfo12'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>class sessions were from 5:30 to 10 pm but amount of time given to library instruction was not specified.<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 67.0pt'> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>taught: s</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>earch strategies (keywords, subject headings, and boolean operators), how to find journal articles, identifying and searching for scholarly journals, searching for articles using the appropriate journal database for the topic, refining the search, evaluating the article, and downloading or ordering the full-text of the article.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>assessed</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>: ability to find information, ability to evaluate information, and understanding of plagiarism.<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:12;height:150.0pt'> <td valign=top style='border:solid #bfbfbf 1.0pt;mso-border-themecolor:background1; mso-border-themeshade:191;border-top:none;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor:background1; mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height:150.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>lapidus, 2012<br> combined use of online tutorials…<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 150.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>pharmacy students<br> 2nd year<br> usa<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 150.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>ig<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>1 tx group (2010)<br> (n=299)<br> 1 control group (2008)<br> (n=300)<o:p></o:p></span></p> </td> <td nowrap valign=top style='border-top:none;border-left:none;border-bottom: solid #bfbfbf 1.0pt;mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade: 191;border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 150.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>hybrid<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 150.0pt'> <p class=msolistparagraphcxspfirst style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l5 level1 lfo13'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>control group (2008) received library instruction using lecture and demo.<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l5 level1 lfo13'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>intervention group (2010) used blended learning with online tutorials, brief demo, in-class hands-on exercises, group discussion.<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l5 level1 lfo13'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>5 to 6 class sessions taught by librarians during a fall semester.<o:p></o:p></span></p> <p class=msolistparagraphcxsplast style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l5 level1 lfo13'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>no indication of length of individual sessions.<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 150.0pt'> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>taught:</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> searching secondary databases (ovid medline, <span class=spelle>mesh</span>, boolean operators, scopus, ovid international pharmaceutical abstracts [ipa]); using tertiary computerized databases (micromedex, clinical pharmacology, <span class=spelle><span class=grame>stat!ref</span></span>, clinical reference library, clin-<span class=spelle>eguide</span>, natural medicines, natural standard), pubmed.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>assessed:</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> answering drug information questions using tertiary print and electronic resources; searching medline and ipa.<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:13;height:78.6pt'> <td rowspan=3 valign=top style='border:solid #bfbfbf 1.0pt;mso-border-themecolor: background1;mso-border-themeshade:191;border-top:none;mso-border-top-alt: solid #bfbfbf .5pt;mso-border-top-themecolor:background1;mso-border-top-themeshade: 191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor:background1; mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height:78.6pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>lechner, 2005<br> graduate student research instruction…<o:p></o:p></span></p> </td> <td rowspan=3 valign=top style='border-top:none;border-left:none;border-bottom: solid #bfbfbf 1.0pt;mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade: 191;border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 78.6pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>master of science in occupational therapy and master of physical therapy<br> usa<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 78.6pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>rm<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>pre/post-test<br> 2 tx groups<br> randomized into online or f2f<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 78.6pt'></td> <td rowspan=3 valign=top style='border-top:none;border-left:none;border-bottom: solid #bfbfbf 1.0pt;mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade: 191;border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 78.6pt'> <p class=msolistparagraphcxspfirst style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l10 level1 lfo14'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>online tutorial that provided live results in response to students’ actions.<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l10 level1 lfo14'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>each class (ot and pt) was randomized into 2 groups; one group went to another room to complete the online tutorial while the remaining students attended a lecture covering the same material. students in the lecture group could choose to watch only or could follow along on computers.<o:p></o:p></span></p> <p class=msolistparagraphcxsplast style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l10 level1 lfo14'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>no information about length of class.<o:p></o:p></span></p> </td> <td rowspan=3 valign=top style='border-top:none;border-left:none;border-bottom: solid #bfbfbf 1.0pt;mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade: 191;border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 78.6pt'> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>taught:</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> searching cinahl database including controlled vocabulary, functions of various indexes, using limits to filter and focus results.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>assessed:</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> basic information literacy (e.g., definition of peer-reviewed), basic cinahl characteristics (e.g., target audience), basic cinahl skills (e.g., combining searches), advanced cinahl skills (e.g., interpreting hierarchy of subject headings), advanced cinahl characteristics (e.g., using account to store results).<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:14;height:39.15pt'> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 39.15pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>online group<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>(n=17)<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 39.15pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>online<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:15;height:39.1pt'> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 39.1pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>f2f group<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>(n=10)<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 39.1pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>f2f<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:16;height:120.0pt'> <td valign=top style='border:solid #bfbfbf 1.0pt;mso-border-themecolor:background1; mso-border-themeshade:191;border-top:none;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor:background1; mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height:120.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>maranda, 2016<br> evaluation of the long-term impact…<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 120.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>medical students<br> 1st year<br> canada<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 120.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>rm<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>pre/post-test<br> 1 tx group<br> (n=100, pre)<br> (n=59, post)<o:p></o:p></span></p> </td> <td nowrap valign=top style='border-top:none;border-left:none;border-bottom: solid #bfbfbf 1.0pt;mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade: 191;border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 120.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>hybrid<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 120.0pt'> <p class=msolistparagraphcxspfirst style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l23 level1 lfo15'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>library instruction in year 1 consisted of 3 online modules and 3 in-person sessions.<o:p></o:p></span></p> <p class=msolistparagraphcxsplast style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l23 level1 lfo15'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>no indication of length of sessions or online modules.<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 120.0pt'> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>taught:</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> e-books, poc (point of care) tools, medline/pubmed searching, drug information resources.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>assessed:</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> only 2 questions (knowledge of boolean logic, choice of resource for clinical scenario) were consistent across the preand post-tests, and the survey.<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:17;height:49.0pt'> <td valign=top style='border:solid #bfbfbf 1.0pt;mso-border-themecolor:background1; mso-border-themeshade:191;border-top:none;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor:background1; mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height:49.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>otto, 2012<br> assessing and improving data literacy…<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 49.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>master’s students<br> urban regional planning<br> college of business and public administration<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>usa<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 49.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>rm<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>pre/post-text<br> 1 tx group<br> (n=13)<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 49.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>f2f<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 49.0pt'> <p class=msolistparagraphcxspfirst style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l4 level1 lfo16'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>2 sessions, each a combination of lecture and hands-on teaching.<o:p></o:p></span></p> <p class=msolistparagraphcxsplast style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l4 level1 lfo16'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>each session was 90 minutes, first session was about 1/3 through semester, second was about 2/3 through semester.<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 49.0pt'> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>taught: </span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>selecting and using appropriate data sources, retrieving needed data. 1<sup>st</sup> session focused on demographic and population data, 2<sup>nd</sup> session on economic data.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>assessed: </span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>knowledge of trusted government sources, conceptual understanding of how to search for data, what kinds of web-based sources are considered trustworthy.<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:18;height:1.25in'> <td valign=top style='border:solid #bfbfbf 1.0pt;mso-border-themecolor:background1; mso-border-themeshade:191;border-top:none;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor:background1; mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height:1.25in'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>schilling, 2006<br> an interactive web-based curriculum…<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 1.25in'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>medical students<br> 3rd year<br> usa<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 1.25in'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>ig<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>1 tx group<br> (n=74)<br> 1 control group<br> (n=58)<br> block randomization<o:p></o:p></span></p> </td> <td nowrap valign=top style='border-top:none;border-left:none;border-bottom: solid #bfbfbf 1.0pt;mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade: 191;border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 1.25in'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>online<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 1.25in'> <p class=msolistparagraphcxspfirst style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l9 level1 lfo17'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>during weeks 1 &amp; 2 of a 6-week clerkship, students used 2 course-integrated, web-based learning modules designed by health science librarians.<o:p></o:p></span></p> <p class=msolistparagraphcxsplast style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l9 level1 lfo17'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>modules required 40 to 60 minutes to complete.<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 1.25in'> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>taught</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>: basic medline searching; using <span class=spelle>mesh</span>, boolean operators; finding randomized controlled trials, meta-analyses, &amp; gold standard literature; searching the cochrane database; information found in different types of research.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>assessed:</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> ability to formulate a clinical question, develop an effective search strategy, id and use correct <span class=spelle>mesh</span> terms, use boolean operators, use appropriate limits, restrict search results to randomized controlled trials or meta-analyses.<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:19;height:135.0pt'> <td valign=top style='border:solid #bfbfbf 1.0pt;mso-border-themecolor:background1; mso-border-themeshade:191;border-top:none;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor:background1; mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height:135.0pt'> <p class=msonormal><span class=spelle><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us'>schweikhard</span></span><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>, 2018<br> the impact of library tutorials…<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 135.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>master of occupational therapy &amp; doctor of physical therapy students<br> usa<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 135.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>ig<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><br> 1 tx group<br> (n=90)<br> 1 control group<br> (n=90)<br> participants randomly selected from a larger pool<o:p></o:p></span></p> </td> <td nowrap valign=top style='border-top:none;border-left:none;border-bottom: solid #bfbfbf 1.0pt;mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade: 191;border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 135.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>online<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 135.0pt'> <p class=msolistparagraphcxspfirst style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l6 level1 lfo18'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>9 online library instructional tutorials created with guide on the side and embedded in the online course platform.<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l6 level1 lfo18'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>tutorials were not required but “strongly recommended” by instructor.<o:p></o:p></span></p> <p class=msolistparagraphcxsplast style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l6 level1 lfo18'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>1 tutorial was created for each of 9 class sessions, no indication of length of tutorials.<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 135.0pt'> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>taught:</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> overview of library; using appropriate databases; using <span class=spelle>mesh</span>; searching for different types of evidence (randomized controlled trials, systematic reviews and meta-analyses, cohort &amp; case-control studies, diagnostic tests, qualitative research, practice guidelines).<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>assessed:</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> use of databases; use of search terms and <span class=spelle>mesh</span>/subject headings; use of limits; level of evidence for each cited study.<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:20;height:67.2pt'> <td rowspan=3 valign=top style='border:solid #bfbfbf 1.0pt;mso-border-themecolor: background1;mso-border-themeshade:191;border-top:none;mso-border-top-alt: solid #bfbfbf .5pt;mso-border-top-themecolor:background1;mso-border-top-themeshade: 191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor:background1; mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height:67.2pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>shaffer, 2011<br> graduate student library research…<o:p></o:p></span></p> </td> <td rowspan=3 valign=top style='border-top:none;border-left:none;border-bottom: solid #bfbfbf 1.0pt;mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade: 191;border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 67.2pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>curriculum and instruction dept., college of education<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>usa<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 67.2pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>rm<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>pre/post-test<br> 2 tx groups<br style='mso-special-character:line-break'> <![if !supportlinebreaknewline]><br style='mso-special-character:line-break'> <![endif]><o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 67.2pt'></td> <td rowspan=3 valign=top style='border-top:none;border-left:none;border-bottom: solid #bfbfbf 1.0pt;mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade: 191;border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 67.2pt'> <p class=msolistparagraphcxspfirst style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l14 level1 lfo19'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>online tutorial consisted of 8 mini-tutorials.<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l14 level1 lfo19'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>both face-to-face and online tutorial sessions took place on-campus.<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l14 level1 lfo19'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>the 4 lit 530 sections were randomized to 2 online and 2 f2f, the edu 504 section was randomized with half of students in f2f and the other half in a different room for the online tutorial.<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l14 level1 lfo19'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>both f2f and online tutorial took place during a 3-hour class; students had instruction either f2f or online then used remaining time for independent research.<o:p></o:p></span></p> <p class=msolistparagraphcxsplast style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l14 level1 lfo19'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>online session averaged less than 2 hours, f2f instruction averaged 2 hours.<o:p></o:p></span></p> </td> <td rowspan=3 valign=top style='border-top:none;border-left:none;border-bottom: solid #bfbfbf 1.0pt;mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade: 191;border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 67.2pt'> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>taught:</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> sources for quality education research; scholarly and primary research; choosing search terms; searching effectively and efficiently; finding full-text, apa citation; other database features.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>assessed:</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> sources for quality education research; scholarly and primary research; choosing search terms; searching effectively and efficiently; finding full-text, apa citation; other database features.<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:21;height:67.2pt'> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 67.2pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>online group<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>(n=29)<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 67.2pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>online<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:22;height:67.2pt'> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 67.2pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>f2f group<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>(n=30)<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 67.2pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>f2f<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:23;mso-yfti-lastrow:yes;height:150.0pt'> <td valign=top style='border:solid #bfbfbf 1.0pt;mso-border-themecolor:background1; mso-border-themeshade:191;border-top:none;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor:background1; mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height:150.0pt'> <p class=msonormal><span class=spelle><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us'>wema</span></span><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>, 2006<br> developing information literacy <span class=spelle>programmes</span>…<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 150.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>master of education students<br> tanzania<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 150.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>rm<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>pre/post-test<br> 1 tx group<br> (n=12)<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 150.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>f2f<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 150.0pt'> <p class=msolistparagraphcxspfirst style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l16 level1 lfo20'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>a combination of lectures and hands-on activities.<o:p></o:p></span></p> <p class=msolistparagraphcxsplast style='margin-top:0in;margin-right:0in; margin-bottom:0in;margin-left:.25in;margin-bottom:.0001pt;mso-add-space:auto; text-indent:-.25in;line-height:normal;mso-list:l16 level1 lfo20'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family: "palatino linotype",serif'>7-day course, each day began at 8 am and lasted to approximately 5 pm.<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid #bfbfbf 1.0pt; mso-border-bottom-themecolor:background1;mso-border-bottom-themeshade:191; border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor:background1; mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 150.0pt'> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>taught:</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> formulating a question; defining information needs; organizing ideas for information need; categories and structure of information sources; developing search strategies, how to modify search; capturing and synthesizing information from sources; evaluating sources; presenting information; referencing and citing; ethical and legal issues in using information.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>assessed:</span></b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> defining a problem or research topic; information sources; internet sources; internet search; library and database searching; evaluating information and sources; referencing; synthesizing information; presenting information.<o:p></o:p></span></p> </td> </tr> </table> <p class=msonospacing><span style='font-size:10.0pt;font-family:"palatino linotype",serif; color:#7030a0'><o:p>&nbsp;</o:p></span></p> <p class=msonospacing><span style='font-size:10.0pt;font-family:"palatino linotype",serif; color:#7030a0'><o:p>&nbsp;</o:p></span></p> <p class=msonospacing><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-no-proof:yes'><!--[if gte vml 1]><v:shape id="picture_x0020_8" o:spid="_x0000_i1027" type="#_x0000_t75" style='width:462.6pt;height:301.8pt;visibility:visible; mso-wrap-style:square' o:bordertopcolor="black" o:borderleftcolor="black" o:borderbottomcolor="black" o:borderrightcolor="black"> <v:imagedata src="art_grabowsky_29657_files/image007.png" o:title=""/> <w:bordertop type="single" width="6"/> <w:borderleft type="single" width="6"/> <w:borderbottom type="single" width="6"/> <w:borderright type="single" width="6"/> </v:shape><![endif]--><![if !vml]><img border=0 width=618 height=404 src="art_grabowsky_29657_files/image004.png" v:shapes="picture_x0020_8"><![endif]></span><span style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>figure 2<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>number of studies by discipline.<o:p></o:p></span></p> <p class=msonospacing><span style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonospacing><span style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>table 2<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>duration of instruction<o:p></o:p></span></p> <table class=msotablegridlight border=1 cellspacing=0 cellpadding=0 style='border-collapse:collapse;border:none;mso-border-alt:solid #bfbfbf .5pt; mso-border-themecolor:background1;mso-border-themeshade:191;mso-yfti-tbllook: 1184;mso-padding-alt:0in 5.4pt 0in 5.4pt'> <tr style='mso-yfti-irow:0;mso-yfti-firstrow:yes;height:.4in'> <td width=162 valign=top style='width:121.25pt;border:solid #bfbfbf 1.0pt; mso-border-themecolor:background1;mso-border-themeshade:191;mso-border-alt: solid #bfbfbf .5pt;mso-border-themecolor:background1;mso-border-themeshade: 191;padding:0in 5.4pt 0in 5.4pt;height:.4in'> <p class=msonormal align=center style='text-align:center'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'>not mentioned<o:p></o:p></span></b></p> <p class=msonormal align=center style='text-align:center'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>(n=4)<o:p></o:p></span></p> </td> <td width=150 valign=top style='width:112.5pt;border:solid #bfbfbf 1.0pt; mso-border-themecolor:background1;mso-border-themeshade:191;border-left:none; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: .4in'> <p class=msonormal align=center style='text-align:center'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'>online tutorials<o:p></o:p></span></b></p> <p class=msonormal align=center style='text-align:center'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>(n=3)<o:p></o:p></span></p> </td> <td width=168 valign=top style='width:1.75in;border:solid #bfbfbf 1.0pt; mso-border-themecolor:background1;mso-border-themeshade:191;border-left:none; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: .4in'> <p class=msonormal align=center style='text-align:center'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'>stand-alone classes<o:p></o:p></span></b></p> <p class=msonormal align=center style='text-align:center'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>(n=2)<o:p></o:p></span></p> </td> <td width=132 valign=top style='width:98.75pt;border:solid #bfbfbf 1.0pt; mso-border-themecolor:background1;mso-border-themeshade:191;border-left:none; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: .4in'> <p class=msonormal align=center style='text-align:center'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'>sessions within subject classes<o:p></o:p></span></b></p> <p class=msonormal align=center style='text-align:center'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>(n=7)<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:1;height:11.45pt'> <td width=162 valign=top style='width:121.25pt;border:solid #bfbfbf 1.0pt; mso-border-themecolor:background1;mso-border-themeshade:191;border-top:none; mso-border-top-alt:solid #bfbfbf .5pt;mso-border-top-themecolor:background1; mso-border-top-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 11.45pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>no mention of duration (n=2)<o:p></o:p></span></p> </td> <td width=150 valign=top style='width:112.5pt;border-top:none;border-left: none;border-bottom:solid #bfbfbf 1.0pt;mso-border-bottom-themecolor:background1; mso-border-bottom-themeshade:191;border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor: background1;mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 11.45pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>2 modules, 50 minutes total<o:p></o:p></span></p> </td> <td width=168 valign=top style='width:1.75in;border-top:none;border-left: none;border-bottom:solid #bfbfbf 1.0pt;mso-border-bottom-themecolor:background1; mso-border-bottom-themeshade:191;border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor: background1;mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 11.45pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>75 min/week for one semester (n=1)<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>[for credit class]<o:p></o:p></span></p> </td> <td width=132 valign=top style='width:98.75pt;border-top:none;border-left: none;border-bottom:solid #bfbfbf 1.0pt;mso-border-bottom-themecolor:background1; mso-border-bottom-themeshade:191;border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor: background1;mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 11.45pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>1 session @ 70 min (n=1)<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:2;height:10.85pt'> <td width=162 valign=top style='width:121.25pt;border:solid #bfbfbf 1.0pt; mso-border-themecolor:background1;mso-border-themeshade:191;border-top:none; mso-border-top-alt:solid #bfbfbf .5pt;mso-border-top-themecolor:background1; mso-border-top-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 10.85pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>mentioned # of sessions but not length of sessions (n=2)<o:p></o:p></span></p> </td> <td width=150 valign=top style='width:112.5pt;border-top:none;border-left: none;border-bottom:solid #bfbfbf 1.0pt;mso-border-bottom-themecolor:background1; mso-border-bottom-themeshade:191;border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor: background1;mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 10.85pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>2 modules, 120 minutes total<o:p></o:p></span></p> </td> <td width=168 valign=top style='width:1.75in;border-top:none;border-left: none;border-bottom:solid #bfbfbf 1.0pt;mso-border-bottom-themecolor:background1; mso-border-bottom-themeshade:191;border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor: background1;mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 10.85pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>each day for 1 week (n=1)<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>[seminar]<o:p></o:p></span></p> </td> <td width=132 valign=top style='width:98.75pt;border-top:none;border-left: none;border-bottom:solid #bfbfbf 1.0pt;mso-border-bottom-themecolor:background1; mso-border-bottom-themeshade:191;border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor: background1;mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 10.85pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>1 session @ 120 minutes (n=2)<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:3;height:11.45pt'> <td width=162 valign=top style='width:121.25pt;border:solid #bfbfbf 1.0pt; mso-border-themecolor:background1;mso-border-themeshade:191;border-top:none; mso-border-top-alt:solid #bfbfbf .5pt;mso-border-top-themecolor:background1; mso-border-top-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 11.45pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> </td> <td width=150 valign=top style='width:112.5pt;border-top:none;border-left: none;border-bottom:solid #bfbfbf 1.0pt;mso-border-bottom-themecolor:background1; mso-border-bottom-themeshade:191;border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor: background1;mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 11.45pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>4 modules, 80 minutes total<o:p></o:p></span></p> </td> <td width=168 valign=top style='width:1.75in;border-top:none;border-left: none;border-bottom:solid #bfbfbf 1.0pt;mso-border-bottom-themecolor:background1; mso-border-bottom-themeshade:191;border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor: background1;mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 11.45pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> </td> <td width=132 valign=top style='width:98.75pt;border-top:none;border-left: none;border-bottom:solid #bfbfbf 1.0pt;mso-border-bottom-themecolor:background1; mso-border-bottom-themeshade:191;border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor: background1;mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 11.45pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>1 session @ 180 minutes (n=2)<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:4;height:11.45pt'> <td width=162 valign=top style='width:121.25pt;border:solid #bfbfbf 1.0pt; mso-border-themecolor:background1;mso-border-themeshade:191;border-top:none; mso-border-top-alt:solid #bfbfbf .5pt;mso-border-top-themecolor:background1; mso-border-top-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 11.45pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> </td> <td width=150 valign=top style='width:112.5pt;border-top:none;border-left: none;border-bottom:solid #bfbfbf 1.0pt;mso-border-bottom-themecolor:background1; mso-border-bottom-themeshade:191;border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor: background1;mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 11.45pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> </td> <td width=168 valign=top style='width:1.75in;border-top:none;border-left: none;border-bottom:solid #bfbfbf 1.0pt;mso-border-bottom-themecolor:background1; mso-border-bottom-themeshade:191;border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor: background1;mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 11.45pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> </td> <td width=132 valign=top style='width:98.75pt;border-top:none;border-left: none;border-bottom:solid #bfbfbf 1.0pt;mso-border-bottom-themecolor:background1; mso-border-bottom-themeshade:191;border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor: background1;mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 11.45pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>2 sessions @ 90 minutes each (n=1)<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:5;mso-yfti-lastrow:yes;height:10.85pt'> <td width=162 valign=top style='width:121.25pt;border:solid #bfbfbf 1.0pt; mso-border-themecolor:background1;mso-border-themeshade:191;border-top:none; mso-border-top-alt:solid #bfbfbf .5pt;mso-border-top-themecolor:background1; mso-border-top-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 10.85pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> </td> <td width=150 valign=top style='width:112.5pt;border-top:none;border-left: none;border-bottom:solid #bfbfbf 1.0pt;mso-border-bottom-themecolor:background1; mso-border-bottom-themeshade:191;border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor: background1;mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 10.85pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> </td> <td width=168 valign=top style='width:1.75in;border-top:none;border-left: none;border-bottom:solid #bfbfbf 1.0pt;mso-border-bottom-themecolor:background1; mso-border-bottom-themeshade:191;border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor: background1;mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 10.85pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> </td> <td width=132 valign=top style='width:98.75pt;border-top:none;border-left: none;border-bottom:solid #bfbfbf 1.0pt;mso-border-bottom-themecolor:background1; mso-border-bottom-themeshade:191;border-right:solid #bfbfbf 1.0pt;mso-border-right-themecolor: background1;mso-border-right-themeshade:191;mso-border-top-alt:solid #bfbfbf .5pt; mso-border-top-themecolor:background1;mso-border-top-themeshade:191; mso-border-left-alt:solid #bfbfbf .5pt;mso-border-left-themecolor:background1; mso-border-left-themeshade:191;mso-border-alt:solid #bfbfbf .5pt;mso-border-themecolor: background1;mso-border-themeshade:191;padding:0in 5.4pt 0in 5.4pt;height: 10.85pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>2 sessions @ 180 minutes each (n=1)<o:p></o:p></span></p> </td> </tr> </table> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></i></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>meta-analysis<o:p></o:p></span></i></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></i></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>meta-analysis often involves examination of experimental studies involving independent groups (ig), for example treatment and control groups; however, meta-analysis is also possible with repeated measures designs (rm). rm studies involve one or more groups; individuals within the groups are assessed both before and after an intervention. these two types of studies differ in the type of research question involved, with igs interested in group differences while rms explore change at the individual level (morris &amp; <span class=spelle>deshon</span>, 2002). morris and <span class=spelle>deshon</span> (2002) point out that combining ig and rm studies may be done, but only if effect sizes are transformed to account for differences in how standard deviations are calculated. the ig studies found in this systematic review did not include the information required to transform the effect sizes to equivalent rm effect sizes as recommended (morris and <span class=spelle>deshon</span>, 2002).<span style='mso-spacerun:yes'>  </span>in addition, the small number of ig studies was considered insufficient to complete a separate meta-analysis, therefore only the rm studies (preand post-assessment of one or more groups) were included in the meta-analyses. when an rm study included multiple groups, for example, a comparison of online versus face-to-face instruction, each group was considered separately in the meta-analysis. therefore, for the 12 rm studies there were 16 associated effect sizes. nine of the rm effect sizes involved face-to-face instruction (f2f) by a librarian, three were online modules only, and four were hybrid sessions, involving f2f instruction supplemented with online modules.<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><i style='mso-bidi-font-style:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'>effectiveness of library instruction for graduate students<o:p></o:p></span></i></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>a meta-analysis run on all rm groups (16 effect sizes from 12 studies) produced an overall standardized mean difference (smd) of 1.03 [se=0.19, <i style='mso-bidi-font-style:normal'>z</i>=5.49, <i style='mso-bidi-font-style: normal'>p</i>&lt;.0001, 95% ci=0.66-1.40], (see figure 3), which is considered a large effect size (cohen, 1988). another way to state the result is that graduate students scored slightly more than one standard deviation higher on a measure of il skills after receiving library instruction. the <i style='mso-bidi-font-style:normal'>i</i><sup>2</sup> statistic for the meta-analysis was 81.47%, indicating a large amount of heterogeneity (higgins, thompson, <span class=spelle>deeks</span>, &amp; altman, 2003) and a need for subgroup analysis. one possibility is that results could have been influenced by the estimation of standard deviation (sd) for two studies (3 associated effect sizes) (appendix b); however, a test of estimation as a moderator revealed no significant difference between studies with estimated variables and those without (qm (df=1) =0.24, <i style='mso-bidi-font-style:normal'>p</i>=.63).<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><i style='mso-bidi-font-style:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></i></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><i style='mso-bidi-font-style:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'>effectiveness by format, duration, or discipline<o:p></o:p></span></i></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>sub-group analysis was completed in an attempt to explain the large amount of heterogeneity in the overall meta-analysis. format of instruction as a moderator was considered first by comparing the three types of format: face-to-face (nine effect sizes), online (three effect sizes), and hybrid (four effect sizes). results of the analysis were not significant (qm (df=2) =0.77, <i style='mso-bidi-font-style:normal'>p</i>=.68) indicating there was only random variation in effect sizes between format types. <o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>discipline of students was also considered as a potential moderator. other than medicine and education, no discipline had more than two associated studies, and many had only one (see figure 2). however, studies were almost equally divided between those involving health science students (seven effect sizes from six studies) and non-health science students (nine effect sizes from six studies), so those two groups were compared. there was a significant difference in effect size based on discipline as a moderator (qm (df=1) =6.54, <i style='mso-bidi-font-style:normal'>p</i>=.01), therefore, two additional meta-analyses were run. for studies involving only health science students there was a lower smd of 0.60 [se=0.17, <i style='mso-bidi-font-style:normal'>z</i>=3.32, <i style='mso-bidi-font-style:normal'>p</i>=.009, 95% ci=0.23-0.88] while for non-health science students the smd increased to 1.43 [se=0.30, <i style='mso-bidi-font-style:normal'>z</i>=4.83, <i style='mso-bidi-font-style: normal'>p</i>&lt;.001, 95% ci=0.85-2.00]. <o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>in a model including both moderators (format plus discipline), the test for residual heterogeneity was significant (qe (df=12) = 40.23, <i style='mso-bidi-font-style:normal'>p</i>&lt;.0001) indicating that other moderators, not included in the analysis, are potentially influencing the effectiveness of instruction (<span class=spelle>viechtbauer</span>, 2010). duration of instruction could be expected to influence effectiveness; however, several studies failed to include duration of instruction. when information about duration was provided, length varied widely. for example, for sessions provided within a subject class, duration ranged from one 70-minute session to two 3-hour sessions. the small number of studies in each duration length precluded completing a subgroup analysis of duration.<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><i style='mso-bidi-font-style:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'>publication bias<o:p></o:p></span></i></p> <p class=msonospacing><span style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonospacing><span style='font-size:10.0pt;font-family:"palatino linotype",serif'>the funnel plot for the meta-analysis of all rm studies is shown in figure 4. studies seem to be evenly distributed at the top of the funnel but lacking toward the bottom. however, the fail-safe number was calculated to be 750, considerably larger than the minimum of 90 suggested by rosenthal (5<i style='mso-bidi-font-style:normal'>n</i> + 10 =5(16) +10).<span style='mso-spacerun:yes'>  </span>since 750 non-significant studies would be required to reduce the overall effect size to zero, publication bias was not considered an issue.<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>table 3<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>characteristics of assessment<o:p></o:p></span></p> <table class=msotablegrid border=1 cellspacing=0 cellpadding=0 width=876 style='width:656.75pt;border-collapse:collapse;border:none;mso-border-alt: solid windowtext .5pt;mso-yfti-tbllook:1184;mso-padding-alt:0in 5.4pt 0in 5.4pt'> <thead> <tr style='mso-yfti-irow:0;mso-yfti-firstrow:yes;height:75.0pt'> <td width=101 style='width:75.5pt;border:solid windowtext 1.0pt;mso-border-alt: solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:75.0pt'> <p class=msonormal align=center style='text-align:center'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us;mso-fareast-language: zh-cn;mso-bidi-language:hi'>shortened citation<o:p></o:p></span></b></p> </td> <td width=223 style='width:167.25pt;border:solid windowtext 1.0pt; border-left:none;mso-border-left-alt:solid windowtext .5pt;mso-border-alt: solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:75.0pt'> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>a) questionnaire/assessment notes<o:p></o:p></span></b></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>b) timing<o:p></o:p></span></b></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>c) tested recall of knowledge or application of knowledge?<o:p></o:p></span></b></p> </td> <td width=228 style='width:171.0pt;border:solid windowtext 1.0pt;border-left: none;mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:0in 5.4pt 0in 5.4pt;height:75.0pt'> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>questionnaire/assessment<o:p></o:p></span></b></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>a) validity/reliability addressed?<br> b) used existing or created for this study?<o:p></o:p></span></b></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>c) author(s) referenced the acrl standards or framework?<o:p></o:p></span></b></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>d) full questionnaire/assessment available?<o:p></o:p></span></b></p> </td> <td width=156 style='width:117.0pt;border:solid windowtext 1.0pt;border-left: none;mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:0in 5.4pt 0in 5.4pt;height:75.0pt'> <p class=msonormal align=center style='text-align:center'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us;mso-fareast-language: zh-cn;mso-bidi-language:hi'>findings<o:p></o:p></span></b></p> </td> <td width=168 style='width:1.75in;border:solid windowtext 1.0pt;border-left: none;mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:0in 5.4pt 0in 5.4pt;height:75.0pt'> <p class=msonormal align=center style='text-align:center'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us;mso-fareast-language: zh-cn;mso-bidi-language:hi'>additional assessments?<o:p></o:p></span></b></p> </td> </tr> </thead> <tr style='mso-yfti-irow:1;height:88.15pt'> <td width=101 valign=top style='width:75.5pt;border:solid windowtext 1.0pt; border-top:none;mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:0in 5.4pt 0in 5.4pt;height:88.15pt'> <p class=msonormal><span class=spelle><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us;mso-fareast-language: zh-cn;mso-bidi-language:hi'>aronoff</span></span><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us;mso-fareast-language: zh-cn;mso-bidi-language:hi'>, 2017<br> teaching evidence-based practice…<o:p></o:p></span></p> </td> <td width=223 valign=top style='width:167.25pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:88.15pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>a) aft (adapted fresno test).<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>b) given before students had access to the online modules then again after they completed the online modules, then a 3rd time after they participated in the small group learning experience.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>c) application.<o:p></o:p></span></p> </td> <td width=228 valign=top style='width:171.0pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:88.15pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>a) yes.<br> b) used existing adapted fresno test.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>c) no.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>d) no.<o:p></o:p></span></p> </td> <td width=156 valign=top style='width:117.0pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:88.15pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>scores on the aft increased significantly post-modules, but decreased post-small group experience.<o:p></o:p></span></p> </td> <td width=168 valign=top style='width:1.75in;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:88.15pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>students took a quiz after each of the 2 modules.<br> students completed an anonymous evaluation of the modules.<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:2;height:88.6pt'> <td width=101 valign=top style='width:75.5pt;border:solid windowtext 1.0pt; border-top:none;mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:0in 5.4pt 0in 5.4pt;height:88.6pt'> <p class=msonormal><span class=spelle><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us;mso-fareast-language: zh-cn;mso-bidi-language:hi'>beile</span></span><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us;mso-fareast-language: zh-cn;mso-bidi-language:hi'>, 2004<br> does the medium matter?<o:p></o:p></span></p> </td> <td width=223 valign=top style='width:167.25pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:88.6pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>a) 20 multiple choice questions.<br> b) pre-test given immediately before the instruction session began.<br> post-test given ~ 6 weeks after the instruction session.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>c) recall.<o:p></o:p></span></p> </td> <td width=228 valign=top style='width:171.0pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:88.6pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>a) no.<br> b) created for this study, written by faculty who teach library instruction sessions. <o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>c) yes, assessment based on acrl il standards.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>d) no<o:p></o:p></span></p> </td> <td width=156 valign=top style='width:117.0pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:88.6pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>significant increase in post-instruction scores; no significant difference in scores by format of instruction.<o:p></o:p></span></p> </td> <td width=168 valign=top style='width:1.75in;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:88.6pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>self-reported perceptions of efficacy.<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:3;height:105.0pt'> <td width=101 valign=top style='width:75.5pt;border:solid windowtext 1.0pt; border-top:none;mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:0in 5.4pt 0in 5.4pt;height:105.0pt'> <p class=msonormal><span class=spelle><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us;mso-fareast-language: zh-cn;mso-bidi-language:hi'>chiarella</span></span><span style='font-size: 10.0pt;font-family:"palatino linotype",serif;mso-ansi-language:en-us; mso-fareast-language:zh-cn;mso-bidi-language:hi'>, 2014<br> information literacy skills retention…<o:p></o:p></span></p> </td> <td width=223 valign=top style='width:167.25pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:105.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>a) 7-item multiple choice quiz.<br> b) pre-test administered before the 1st library instruction of p1 fall semester,<br> post-test administered at end of p1 spring semester.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>c) recall.<o:p></o:p></span></p> </td> <td width=228 valign=top style='width:171.0pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:105.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>a) no.<br> b) created for this study.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>c) no.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>d) yes.<o:p></o:p></span></p> </td> <td width=156 valign=top style='width:117.0pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:105.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>no significant difference between preand postscores.<o:p></o:p></span></p> </td> <td width=168 nowrap valign=top style='width:1.75in;border-top:none; border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:105.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>no<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:4;height:109.75pt'> <td width=101 valign=top style='width:75.5pt;border:solid windowtext 1.0pt; border-top:none;mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:0in 5.4pt 0in 5.4pt;height:109.75pt'> <p class=msonormal><span class=spelle><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us;mso-fareast-language: zh-cn;mso-bidi-language:hi'>dorsch</span></span><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us;mso-fareast-language: zh-cn;mso-bidi-language:hi'>, 2004<br> impact of an evidence-based….<o:p></o:p></span></p> </td> <td width=223 valign=top style='width:167.25pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:109.75pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>a) students given simulated case scenarios, which were different for preand post-tests. scenarios evaluated by both a librarian and a faculty member using a competency-based instrument with 15 items, each scored from 1 to 7.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>b) assessment given at beginning and end of seminar series.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>c) application.<o:p></o:p></span></p> </td> <td width=228 valign=top style='width:171.0pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:109.75pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>a) no.<br> b) created for this study.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>c) no (referenced medical school objectives).<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>d) yes.<o:p></o:p></span></p> </td> <td width=156 valign=top style='width:117.0pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:109.75pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>statistically significant improvement occurred in creating a pico question; using <span class=spelle>mesh</span>, boolean, and limits; assessing articles. <o:p></o:p></span></p> </td> <td width=168 valign=top style='width:1.75in;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:109.75pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>pre and post-survey to assess students’ self-perception of change in ebm skills.<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:5;height:60.0pt'> <td width=101 valign=top style='width:75.5pt;border:solid windowtext 1.0pt; border-top:none;mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:0in 5.4pt 0in 5.4pt;height:60.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>emmett, 2007<br> assessing information literacy skills…<o:p></o:p></span></p> </td> <td width=223 valign=top style='width:167.25pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:60.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>a) 29 multiple choice/short answer questions.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>b) pre-test given at beginning of semester, post-test given at end of semester. <o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>c) recall and application.<o:p></o:p></span></p> </td> <td width=228 valign=top style='width:171.0pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:60.0pt'> <p class=msonormal style='tab-stops:58.05pt'><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us;mso-fareast-language: zh-cn;mso-bidi-language:hi'>a) yes.<br> b) created for this study.<o:p></o:p></span></p> <p class=msonormal style='tab-stops:58.05pt'><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us;mso-fareast-language: zh-cn;mso-bidi-language:hi'>c) yes, assessment created using acrl il standards and backward design.<o:p></o:p></span></p> <p class=msonormal style='tab-stops:58.05pt'><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us;mso-fareast-language: zh-cn;mso-bidi-language:hi'>d) yes.<o:p></o:p></span></p> </td> <td width=156 valign=top style='width:117.0pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:60.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>57% increase in post-test scores, no statistical analysis provided.<o:p></o:p></span></p> </td> <td width=168 valign=top style='width:1.75in;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:60.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>assessment related to class itself, including in-class exercises, final project, and final exam.<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:6;height:1.25in'> <td width=101 valign=top style='width:75.5pt;border:solid windowtext 1.0pt; border-top:none;mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:0in 5.4pt 0in 5.4pt;height:1.25in'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>grant, 2006<br> developing and evaluating an interactive…<o:p></o:p></span></p> </td> <td width=223 valign=top style='width:167.25pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:1.25in'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>a) assessment of searches.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>b) assessed search done at the beginning of the first session (pre), search done at the end of the second session (post), and search done at the end of the <span class=grame>12 week</span> class (extended).<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>c) application.<o:p></o:p></span></p> </td> <td width=228 valign=top style='width:171.0pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:1.25in'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>a) no.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>b) used existing assessment tool modified from rosenberg et al. (1998).<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>c) no.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>d) yes.<br style='mso-special-character:line-break'> <![if !supportlinebreaknewline]><br style='mso-special-character:line-break'> <![endif]><span style='color:red'><o:p></o:p></span></span></p> </td> <td width=156 valign=top style='width:117.0pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:1.25in'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>statistically significant difference between pre-and post-scores; and between postand extended scores.<o:p></o:p></span></p> </td> <td width=168 valign=top style='width:1.75in;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:1.25in'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>subjective evaluation of students' perceptions of learning.<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:7;height:1.25in'> <td width=101 valign=top style='width:75.5pt;border:solid windowtext 1.0pt; border-top:none;mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:0in 5.4pt 0in 5.4pt;height:1.25in'> <p class=msonormal><span class=spelle><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us;mso-fareast-language: zh-cn;mso-bidi-language:hi'>ilic</span></span><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us;mso-fareast-language: zh-cn;mso-bidi-language:hi'>, 2012<br> teaching evidence-based medicine…<o:p></o:p></span></p> </td> <td width=223 valign=top style='width:167.25pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:1.25in'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>a) fresno test.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>b) post-test assessment done at <span class=grame>1 week</span> post-implementation of intervention.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>c) application.<o:p></o:p></span></p> </td> <td width=228 valign=top style='width:171.0pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:1.25in'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>a) yes.<br> b) used existing fresno test.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>c) no.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>d) no.<o:p></o:p></span></p> </td> <td width=156 valign=top style='width:117.0pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:1.25in'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>no statistically significant difference in scores between the treatment and control group.<o:p></o:p></span></p> </td> <td width=168 valign=top style='width:1.75in;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:1.25in'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>clinical effectiveness and evidence-based practice questionnaire (ebpq) used to assess students' self-perceived competency in ebm literature searching.<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:8;height:1.25in'> <td width=101 valign=top style='width:75.5pt;border:solid windowtext 1.0pt; border-top:none;mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:0in 5.4pt 0in 5.4pt;height:1.25in'> <p class=msonormal><span class=spelle><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us;mso-fareast-language: zh-cn;mso-bidi-language:hi'>ivanitskaya</span></span><span style='font-size: 10.0pt;font-family:"palatino linotype",serif;mso-ansi-language:en-us; mso-fareast-language:zh-cn;mso-bidi-language:hi'>, 2008<br> how does a pre-assessment of off-campus…<o:p></o:p></span></p> </td> <td width=223 valign=top style='width:167.25pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:1.25in'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>a) research readiness self-assessment (rrsa).<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>b) pre-test completed before library instruction, post-test completed after instruction.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>c) recall and application.<o:p></o:p></span></p> </td> <td width=228 valign=top style='width:171.0pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:1.25in'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>a) yes.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>b) used existing research readiness self-assessment (rrsa).<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>c) yes, rrsa is based on acrl il standards.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>d) no.<o:p></o:p></span></p> </td> <td width=156 valign=top style='width:117.0pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:1.25in'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>statistically significant difference in preand post-test scores.<o:p></o:p></span></p> </td> <td width=168 valign=top style='width:1.75in;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:1.25in'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>rrsa also includes subjective measures that ask for students’ perceptions of research skills and previous library/research experience.<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:9;height:90.85pt'> <td width=101 valign=top style='width:75.5pt;border:solid windowtext 1.0pt; border-top:none;mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:0in 5.4pt 0in 5.4pt;height:90.85pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>lapidus, 2012<br> combined use of online tutorials…<o:p></o:p></span></p> </td> <td width=223 valign=top style='width:167.25pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:90.85pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>a) grades earned on a homework assignment related to secondary databases (medline &amp; international pharmaceutical abstracts).<br> b) timing not specified.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>c) application.<o:p></o:p></span></p> </td> <td width=228 valign=top style='width:171.0pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:90.85pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>a) no.<br> b) created rubric for this class to grade assignments.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>c) no (referenced american association of colleges of pharmacy standards).<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>d) no.<o:p></o:p></span></p> </td> <td width=156 valign=top style='width:117.0pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:90.85pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>no difference in students’ scores when comparing hybrid instruction to traditional lecture-based instruction.<o:p></o:p></span></p> </td> <td width=168 valign=top style='width:1.75in;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:90.85pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>additional homework assignment covering tertiary resources; <br> course evaluations with students’ perceptions of course design and teaching methods.<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:10;height:120.0pt'> <td width=101 valign=top style='width:75.5pt;border:solid windowtext 1.0pt; border-top:none;mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:0in 5.4pt 0in 5.4pt;height:120.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>lechner, 2005<br> graduate student research instruction…<o:p></o:p></span></p> </td> <td width=223 valign=top style='width:167.25pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:120.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>a) 20 multiple choice questions delivered using webct.<br> b) pre-test given immediately before instruction began. post-test given at a later unspecified date.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>c) recall.<o:p></o:p></span></p> </td> <td width=228 nowrap valign=top style='width:171.0pt;border-top:none; border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:120.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>a) no.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>b) no indication of origin of questionnaire.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>c) no.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>d) no.<o:p></o:p></span></p> </td> <td width=156 valign=top style='width:117.0pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:120.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>average scores increased after instruction, no statistical analysis provided.<o:p></o:p></span></p> </td> <td width=168 nowrap valign=top style='width:1.75in;border-top:none; border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:120.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>2 additional questions asked about students’ prior use of cinahl.<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:11;height:120.0pt'> <td width=101 valign=top style='width:75.5pt;border:solid windowtext 1.0pt; border-top:none;mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:0in 5.4pt 0in 5.4pt;height:120.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>maranda, 2016<br> evaluation of the long-term impact…<o:p></o:p></span></p> </td> <td width=223 valign=top style='width:167.25pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:120.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>a) 5-item multiple choice test.<br> b) pre-test completed online in the first few days of medical school.<br> post-test administered at end of first year. questions asked again at the end of fourth year.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>c) recall.<o:p></o:p></span></p> </td> <td width=228 valign=top style='width:171.0pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:120.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>a) mentioned validity/reliability of assessments considered but not used, however validity/reliability of their questionnaire was not addressed. <o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>b) created for this study, piloted with 4 medical students and 5 librarians and changes made based on feedback.<br> c) no.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>d) yes, in supplementary material.<o:p></o:p></span></p> </td> <td width=156 valign=top style='width:117.0pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:120.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>statistically significant increase in scores between pre-test and post-test; increase in scores at end of 4<sup>th</sup> year but no statistical analysis provided. <o:p></o:p></span></p> </td> <td width=168 valign=top style='width:1.75in;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:120.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>post-program survey of attitudes and behaviors, and confidence in ebm tasks.<br> results of preand post-tests compared to post-program survey.<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:12;height:105.0pt'> <td width=101 valign=top style='width:75.5pt;border:solid windowtext 1.0pt; border-top:none;mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:0in 5.4pt 0in 5.4pt;height:105.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>otto, 2012<br> assessing and improving data literacy…<o:p></o:p></span></p> </td> <td width=223 valign=top style='width:167.25pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:105.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>a) 12 multiple choice/matching questions; identical pre/post-tests.<br> b) pre-test given before 1st session. post-test given at end of quarter.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>c) recall.<o:p></o:p></span></p> </td> <td width=228 valign=top style='width:171.0pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:105.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>a) no. feedback on the questionnaire was solicited from library colleagues; the course instructor vetted the final questionnaire. there was no trial run before use.<br> b) created for this study. <o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>c) no (referenced core competencies for data information literacy).<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>d) yes, in appendix.<o:p></o:p></span></p> </td> <td width=156 valign=top style='width:117.0pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:105.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>average scores increased from pre-test to post-test, no statistical analysis provided.<o:p></o:p></span></p> </td> <td width=168 valign=top style='width:1.75in;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:105.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>examination of student assignments.<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:13;height:1.25in'> <td width=101 valign=top style='width:75.5pt;border:solid windowtext 1.0pt; border-top:none;mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:0in 5.4pt 0in 5.4pt;height:1.25in'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>schilling, 2006<br> an interactive web-based curriculum…<o:p></o:p></span></p> </td> <td width=223 valign=top style='width:167.25pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:1.25in'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>a) analysis of students’ medline search strategies.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>b) final (6<sup>th</sup>) week of rotation.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>c) application.<o:p></o:p></span></p> </td> <td width=228 valign=top style='width:171.0pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:1.25in'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>a) yes.<span style='color:red'><o:p></o:p></span></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>b) evaluation criteria developed in previous research. interrater reliability assessed on evaluations of search strategy.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>c) yes (acrl il standards).<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>d) no.<o:p></o:p></span></p> </td> <td width=156 valign=top style='width:117.0pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:1.25in'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>scores for treatment group were significantly greater than the control group.<o:p></o:p></span></p> </td> <td width=168 valign=top style='width:1.75in;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:1.25in'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>pre and post-clerkship survey (self-report); post-clerkship nnt (number needed to treat) test (self-report); analysis of articles identified as best evidence.<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:14;height:109.75pt'> <td width=101 valign=top style='width:75.5pt;border:solid windowtext 1.0pt; border-top:none;mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:0in 5.4pt 0in 5.4pt;height:109.75pt'> <p class=msonormal><span class=spelle><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us;mso-fareast-language: zh-cn;mso-bidi-language:hi'>schweikhard</span></span><span style='font-size: 10.0pt;font-family:"palatino linotype",serif;mso-ansi-language:en-us; mso-fareast-language:zh-cn;mso-bidi-language:hi'>, 2018<br> the impact of library tutorials…<o:p></o:p></span></p> </td> <td width=223 valign=top style='width:167.25pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:109.75pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>a) final course papers scored by 2 independent reviewers using a rubric.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>b) end of course.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>c) application.<o:p></o:p></span></p> </td> <td width=228 valign=top style='width:171.0pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:109.75pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>a) yes, two reviewers for each paper. reviewers practiced scoring papers not selected for the assessment sample to support interrater reliability.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>b) scoring rubric created for this study.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>c) yes (referenced both acrl il standards and the acrl framework)<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>d) yes.<o:p></o:p></span></p> </td> <td width=156 valign=top style='width:117.0pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:109.75pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>statistically significant increase in post-tutorial students’ use of search terms, <span class=spelle>mesh</span> headings, limits, and level of evidence of cited studies. there was no increase in post-tutorial students’ use of databases.<o:p></o:p></span></p> </td> <td width=168 valign=top style='width:1.75in;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:109.75pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>no.<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:15;height:185.1pt'> <td width=101 valign=top style='width:75.5pt;border:solid windowtext 1.0pt; border-top:none;mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:0in 5.4pt 0in 5.4pt;height:185.1pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>shaffer, 2011<br> graduate student library research…<o:p></o:p></span></p> </td> <td width=223 valign=top style='width:167.25pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:185.1pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>a) 20 multiple choice questions <br> b) pre-test given immediately before instruction. post-test varied according to instructors’ assignment schedule, given after the first iteration of the “works cited list” was due.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>c) recall.<o:p></o:p></span></p> </td> <td width=228 nowrap valign=top style='width:171.0pt;border-top:none; border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:185.1pt'> <p class=msonormal style='tab-stops:104.35pt'><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us;mso-fareast-language: zh-cn;mso-bidi-language:hi'>a) no.<o:p></o:p></span></p> <p class=msonormal style='tab-stops:104.35pt'><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us;mso-fareast-language: zh-cn;mso-bidi-language:hi'>b) some questions adapted from a validated test (<span class=spelle>beile</span> test of information literacy in education); <o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>questions addressed learning outcomes.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>c) yes (acrl il standards).<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>d) yes.<o:p></o:p></span></p> </td> <td width=156 valign=top style='width:117.0pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:185.1pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>statistically significant improvement in scores after instruction for both groups (f2f &amp; online); no significant difference in scores between the f2f and online students. <o:p></o:p></span></p> </td> <td width=168 valign=top style='width:1.75in;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:185.1pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>citation analysis; five questions to determine students’ general level of confidence in key library research skills;<br> students using online tutorial also completed quizzes after each module and a survey asking about tutorial design and comfort with technology used.<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:16;mso-yfti-lastrow:yes;height:150.0pt'> <td width=101 valign=top style='width:75.5pt;border:solid windowtext 1.0pt; border-top:none;mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:0in 5.4pt 0in 5.4pt;height:150.0pt'> <p class=msonormal><span class=spelle><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us;mso-fareast-language: zh-cn;mso-bidi-language:hi'>wema</span></span><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us;mso-fareast-language: zh-cn;mso-bidi-language:hi'>, 2006<br> developing information literacy <span class=spelle>programmes</span>…<o:p></o:p></span></p> </td> <td width=223 nowrap valign=top style='width:167.25pt;border-top:none; border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:150.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>a) 9 sets of questions, all questions were true/false/no comment. <o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>sections with number of questions:<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>defining information problem or research question – 5 <o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>information sources – 10 <o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>internet sources – 8 <o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>internet searching – 12 <o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>library and database searching – 8 <o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>evaluating information and sources – 13 <o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>referencing – 10 <o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>synthesizing information – 6 <o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>presenting information – 8 <o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>b) preand post-test were given during the 7-day training session but no mention of exact timing.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>c) recall.<o:p></o:p></span></p> </td> <td width=228 nowrap valign=top style='width:171.0pt;border-top:none; border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:150.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>a) yes, instruments were tested prior to use. program and assessments were piloted with a group of librarians from the same institution before use in the study.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>b) instrument was based on a questionnaire by andretta (2005) plus others not specified, with adjustments made to reflect needs of setting and participants. <o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>c) yes (acrl il standards plus il standards from other countries).<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>d) yes.<o:p></o:p></span></p> </td> <td width=156 valign=top style='width:117.0pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:150.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>students’ scores increased on average about 30 points, but no statistical analysis provided. <o:p></o:p></span></p> </td> <td width=168 valign=top style='width:1.75in;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:0in 5.4pt 0in 5.4pt;height:150.0pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>quizzes to encourage reflection &amp; test understanding; <o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-fareast-language:zh-cn;mso-bidi-language:hi'>assessment of presentations (at the end of each module and on the last day of the program) to determine strengths and weaknesses in applying what was learned.<o:p></o:p></span></p> </td> </tr> </table> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;color:red; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;color:red; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;color:#7030a0; mso-ansi-language:en-us;mso-no-proof:yes'><!--[if gte vml 1]><v:shape id="picture_x0020_10" o:spid="_x0000_i1026" type="#_x0000_t75" style='width:400.8pt;height:318.6pt; visibility:visible;mso-wrap-style:square' o:bordertopcolor="black" o:borderleftcolor="black" o:borderbottomcolor="black" o:borderrightcolor="black"> <v:imagedata src="art_grabowsky_29657_files/image006.wmz" o:title="" croptop="11977f" cropbottom="2682f" cropleft="1421f" cropright="782f"/> <w:bordertop type="single" width="6"/> <w:borderleft type="single" width="6"/> <w:borderbottom type="single" width="6"/> <w:borderright type="single" width="6"/> </v:shape><![endif]--><![if !vml]><img border=0 width=536 height=426 src="art_grabowsky_29657_files/image008.png" v:shapes="picture_x0020_10"><![endif]></span><span style='font-size:10.0pt;font-family:"palatino linotype",serif;color:#7030a0; mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>figure 3<br> forest plot of meta-analysis of repeated measures studies.<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='line-height:200%'><span lang=en-ca style='font-size: 10.0pt;line-height:200%;font-family:"palatino linotype",serif;mso-no-proof: yes'><!--[if gte vml 1]><v:shape id="picture_x0020_13" o:spid="_x0000_i1025" type="#_x0000_t75" style='width:341.4pt;height:302.4pt;visibility:visible; mso-wrap-style:square' o:bordertopcolor="black" o:borderleftcolor="black" o:borderbottomcolor="black" o:borderrightcolor="black"> <v:imagedata src="art_grabowsky_29657_files/image010.wmz" o:title="" croptop="7926f" cropbottom="2207f" cropright="2988f"/> <w:bordertop type="single" width="6"/> <w:borderleft type="single" width="6"/> <w:borderbottom type="single" width="6"/> <w:borderright type="single" width="6"/> </v:shape><![endif]--><![if !vml]><img border=0 width=457 height=405 src="art_grabowsky_29657_files/image012.png" v:shapes="picture_x0020_13"><![endif]></span><span lang=en-ca style='font-size:10.0pt;line-height:200%;font-family:"palatino linotype",serif'><o:p></o:p></span></p> <p class=msonospacing><span style='font-size:10.0pt;font-family:"palatino linotype",serif'>figure 4<o:p></o:p></span></p> <p class=msonospacing><span style='font-size:10.0pt;font-family:"palatino linotype",serif'>funnel plot of meta-analysis of repeated measures studies. <o:p></o:p></span></p> <p class=msonospacing><span style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'>discussion<o:p></o:p></span></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>there was a positive overall smd, which suggests that library instruction does increase information literacy knowledge and/or skills in graduate students, and that the average increase in score is about one standard deviation. although this appears to be the first systematic review and meta-analysis involving library instruction for graduate students, there is a previous meta-analysis of library instruction for undergraduates, which found similar results (<span class=spelle>koufogiannakis</span> &amp; wiebe, 2006). like this study, <span class=spelle>koufogiannakis</span> and wiebe (2006) found a positive effect when comparing library instruction to no instruction, but the effect was much smaller, about one-third of a standard deviation (smd=0.36, 95% ci=0.14-0.50). the smaller effect may be explained by the fact that <span class=spelle>koufogiannakis</span> and wiebe (2006) were comparing only traditional (passive) instruction to no instruction, while this study compared all types of instruction to no instruction. <span class=spelle>koufogiannakis</span> and wiebe (2006) also compared traditional instruction to computer-aided (online) instruction, and like this study, found no difference in the effectiveness of the two formats. however, findings about hybrid instruction from this study differ from those of another meta-analysis of blended (hybrid) learning in health professions (liu et al., 2016). while liu et al. (2016) concluded that blended (hybrid) instruction was more effective than non-blended instruction (smd=0.81, 95% ci= 0.57-1.05), this study found that there was no statistically significant difference between effect sizes for different formats of instruction, including hybrid, face-to-face, and online. one difference between the two studies is sample size; the small number of studies involving hybrid instruction in this meta-analysis limits the robustness of those results.<span style='mso-spacerun:yes'>   </span><o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>small numbers of studies also impacted the ability to look at effect of instruction by discipline of students. two broad categories (health science students and non-health science students) were examined rather than individual disciplines. findings indicated a significant difference in effect size between instruction for health science and non-health science students, with library instruction for health science students slightly less effective (average increase of about two-thirds of a standard deviation) than library instruction for non-health science students (average increase of almost 1.5 standard deviations). this result may be explained in part by the likelihood that assessing the ability to apply knowledge results in smaller changes than simply testing students’ recall of information. more than 40% of the studies that included health science students assessed application of knowledge. in contrast, the studies involving non-health students all assessed recall of knowledge, although two of them did also include a few questions that required students to apply knowledge in order to answer multiple-choice questions.<span style='mso-spacerun:yes'>  </span><o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><span style='mso-spacerun:yes'> </span><o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>limitations<o:p></o:p></span></i></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></i></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>one limitation for the overall meta-analysis was the lack of required information from studies, resulting in the need to contact authors and if that failed, to estimate standard deviation for some studies. as pointed out by gerstner et al. (2017), effective meta-analyses rely on complete data reporting in primary studies. to ensure more complete and accurate meta-analysis of results, studies reporting educational interventions with preand postassessments should either include preand postmeans and standard deviations or provide raw data so that those statistics can be calculated. <o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>a second limitation in the subgroup analyses was the small number of studies in some categories. duration of intervention, which might be expected to affect effectiveness, was not considered for subgroup analysis because of the lack of information in some studies and lack of uniformity of duration in the remaining studies. in addition, when examining format of instruction, there were only three studies involving online instruction and four with hybrid instruction. <span class=spelle>borenstein</span> et al. (2009) point out that in a random effects model, small numbers of studies make it more difficult to estimate error and increase the possibility of not only an inaccurate effect, but also an inaccurate range of effect. therefore, results of any subgroup analysis with a small number of studies must be regarded with caution.<span style='mso-spacerun:yes'>  </span>small numbers of studies may have also affected subgroup analysis of instruction by discipline since no discipline had more than four studies, and studies had to be combined into much broader categories of health science and non-health science students. <o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b><i><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>implications for practice<o:p></o:p></span></i></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b><i><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></i></b></p> <p class=msonospacing style='margin-left:.5in;text-indent:-.25in;mso-list:l17 level1 lfo21'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif'>library instruction for graduate students seems to be effective in increasing students’ knowledge and skills.<o:p></o:p></span></p> <p class=msonospacing style='margin-left:.5in;text-indent:-.25in;mso-list:l17 level1 lfo21'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif'>there was no significant difference in effectiveness of face-to-face, online, or hybrid formats of instruction.<o:p></o:p></span></p> <p class=msonospacing style='margin-left:.5in;text-indent:-.25in;mso-list:l17 level1 lfo21'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif'>content varied but information about searching effectively was present in all studies.<o:p></o:p></span></p> <p class=msonospacing style='margin-left:.5in;text-indent:-.25in;mso-list:l17 level1 lfo21'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif'>evaluating students’ ability to apply what they learned rather than testing recall of facts may be a more accurate evaluation of instructional impact.<o:p></o:p></span></p> <p class=msonospacing style='margin-left:.5in;text-indent:-.25in;mso-list:l17 level1 lfo21'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif'>most researchers created their own evaluation instrument. using existing validated instruments would allow more robust comparisons.<o:p></o:p></span></p> <p class=msonospacing style='margin-left:.5in;text-indent:-.25in;mso-list:l17 level1 lfo21'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif'>there is a need for more published studies (particularly for non-health science disciplines) and for more complete reporting of study design including information about timing, duration, and content.<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'>conclusion<span style='mso-tab-count:1'>        </span><o:p></o:p></span></b></p> <p class=msonospacing><span style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonospacing><span style='font-size:10.0pt;font-family:"palatino linotype",serif'>in the current climate of accountability in higher education, it is important to know whether the time and effort spent on providing library instruction for graduate students is effective in producing an increase in information literacy knowledge and skills. however, studies involving library instruction often lack power due to small sample sizes; combining studies in a meta-analysis to determine an overall effect size can overcome that problem. this review found 12 repeated measures studies and four independent group studies that tested the impact of library instruction. meta-analysis of the 12 repeated measures studies indicate that library instruction for graduate students was effective in increasing information literacy knowledge and/or skills on average by about one standard deviation. subgroup analysis found a significant moderation of effect between two broad categories of health science and non-health science students. studies involving health science students resulted in a smaller increase of almost two-thirds of a standard deviation, while studies of non-health science students had an increase of almost 1.5 standard deviations. the difference in the two groups may be the result of a difference in assessment, with health science studies more likely to assess application of knowledge rather than recall of information. results of subgroup analyses must be viewed with caution due to small numbers of studies in most subgroups. to strengthen the accuracy of future meta-analyses, there is a need for larger numbers of studies that measure the impact of library instruction, particularly instruction provided in an online or hybrid format. there is also a need for precise description of instructional sessions and more robust data reporting by authors of primary studies.</span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'>acknowledgements<o:p></o:p></span></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>the authors acknowledge with grateful appreciation the assistance of dr. alan wilson of the school of fisheries, aquaculture and aquatic sciences at auburn university in completing this meta-analysis and manuscript. <o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'>references<o:p></o:p></span></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>andretta, s. (2005). <i style='mso-bidi-font-style: normal'>information literacy: a practitioner’s guide</i>. oxford: <span class=spelle>chandos</span>.<o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>association of college and research libraries. (2016). <i style='mso-bidi-font-style:normal'>framework for information literacy for higher education</i>. retrieved from </span><span lang=en-ca><a href="http://www.ala.org/acrl/sites/ala.org.acrl/files/content/issues/infolit/framework_ilhe.pdf"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>http://www.ala.org/acrl/sites/ala.org.acrl/files/content/issues/infolit/framework_ilhe.pdf</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonospacing style='margin-left:.5in;text-indent:-.5in'><span style='font-size:10.0pt;font-family:"palatino linotype",serif'>association of college and research libraries. (2000). <i style='mso-bidi-font-style:normal'>information literacy competency standards for higher education</i>. retrieved from </span><a href="https://alair.ala.org/bitstream/handle/11213/7668/acrl%20information%20literacy%20competency%20standards%20for%20higher%20education.pdf?sequence=1&amp;isallowed=y"><span style='font-size:10.0pt;font-family:"palatino linotype",serif'>https://alair.ala.org/bitstream/handle/11213/7668/acrl%20information%20literacy%20competency%20standards%20for%20higher%20education.pdf?sequence=1&amp;isallowed=y</span></a><span style='font-size:10.0pt;font-family:"palatino linotype",serif'> <o:p></o:p></span></p> <p class=msonospacing style='margin-left:.5in;text-indent:-.5in'><span style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonospacing style='margin-left:.5in;text-indent:-.5in'><span style='font-size:10.0pt;font-family:"palatino linotype",serif'>baker, r. l. (2002). evaluating quality and effectiveness: regional accreditation principles and practices. <i style='mso-bidi-font-style:normal'>the journal of academic librarianship, 28</i>(1-2), 3-7. </span><a href="https://doi.org/10.1016/s0099-1333(01)00279-8"><span style='font-size: 10.0pt;font-family:"palatino linotype",serif'>https://doi.org/10.1016/s0099-1333(01)00279-8</span></a><span style='font-size:10.0pt;font-family:"palatino linotype",serif'> <o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span class=spelle><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us'>blummer</span></span><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>, b. (2009). providing library instruction to graduate students: a review of the literature. <i>public services quarterly</i>, <i>5</i>(1), 15–39. </span><span lang=en-ca><a href="https://doi.org/10.1080/15228950802507525"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>https://doi.org/10.1080/15228950802507525</span></a></span><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span class=spelle><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us'>borenstein</span></span><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>, m., hedges, l. v., higgins, j. p. t., &amp; rothstein, h. r. (2009). <i style='mso-bidi-font-style:normal'>introduction to meta-analysis</i>. west sussex, uk: wiley.<o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span class=spelle><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us'>bown</span></span><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>, m. j., &amp; sutton, a. j. (2010). quality control in systematic reviews and meta-analyses.<span style='mso-spacerun:yes'>  </span><i style='mso-bidi-font-style:normal'>european journal of vascular and endovascular surgery, 40</i>(5), 669-677. </span><span lang=en-ca><a href="https://doi.org/10.1016/j.ejvs.2010.07.011"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>https://doi.org/10.1016/j.ejvs.2010.07.011</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>carlson, j., <span class=spelle>fosmire</span>, m., miller, c. c., &amp; nelson, m. s. (2011). determining data information literacy needs: a study of students and research faculty. <i style='mso-bidi-font-style: normal'>portal: libraries and the academy, 11</i>(2), 629-657. </span><span lang=en-ca><a href="https://doi.org/10.1353/pla.2011.0022"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>https://doi.org/10.1353/pla.2011.0022</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'> </span><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>coe, r. (2002). it’s the effect size, stupid: what effect size is and why it is important. paper presented at the meeting of the british educational research association, university of exeter, england. </span><span lang=en-ca><a href="https://www.leeds.ac.uk/educol/documents/00002182.htm"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>https://www.leeds.ac.uk/educol/documents/00002182.htm</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>cohen, j. (1988). <i style='mso-bidi-font-style:normal'>statistical power analysis for the behavioral sciences </i>(2<sup>nd</sup> ed.).<i style='mso-bidi-font-style:normal'> </i>mahwah, nj: lawrence erlbaum associates.<o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>conway, k. (2011). how prepared are students for postgraduate study? a comparison of the information literacy skills of commencing undergraduate and postgraduate information studies students at curtin university. <i style='mso-bidi-font-style:normal'>australian academic &amp; research libraries, 42</i>(2), 121-135. </span><span lang=en-ca><a href="https://doi.org/10.1080/00048623.2011.10722218"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>https://doi.org/10.1080/00048623.2011.10722218</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>furukawa, t. a., <span class=spelle>barbui</span>, c., cipriani, a., brambilla, p., &amp; watanabe, n. (2006). imputing missing standard deviations in meta-analyses can provide accurate results<i style='mso-bidi-font-style:normal'>. journal of clinical epidemiology, 59</i>(1), 7-10. </span><span lang=en-ca><a href="https://doi.org/10.1016/j.jclinepi.2005.06.006"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>https://doi.org/10.1016/j.jclinepi.2005.06.006</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>gerstner, k., moreno-<span class=spelle>mateos</span>, d., <span class=spelle>gurevitch</span>, j., beckmann, m., <span class=spelle>kambach</span>, s., jones, h. p. &amp; <span class=spelle>seppelt</span>, r. (2017). will your paper be used in a meta-analysis? make the reach of your research broader and longer lasting. <i style='mso-bidi-font-style:normal'>methods in ecology and evolution, 8</i>(6), 777-784. </span><span lang=en-ca><a href="https://doi.org/10.1111/2041-210x.12758"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>https://doi.org/10.1111/2041-210x.12758</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>higgins, j. p. t., thompson, s. g., <span class=spelle>deeks</span>, j. j., &amp; altman, d. g. (2003). measuring inconsistency in meta-analyses. <i style='mso-bidi-font-style:normal'>bmj: british medical journal</i>, <i style='mso-bidi-font-style:normal'>327</i>(7414), 557-560. </span><span lang=en-ca><a href="https://doi.org/10.1136/bmj.327.7414.557"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>https://doi.org/10.1136/bmj.327.7414.557</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>higgins, s. (2019). <i>improving learning: meta-analysis of intervention research in education.</i> cambridge, uk: cambridge university press. <o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span class=spelle><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us'>ivanitskaya</span></span><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>, l., <span class=spelle>laus</span>, r., &amp; casey, a. m. (2004). research readiness self-assessment: assessing students’ research skills and attitudes. <i>journal of library administration</i>, 41(1-2), 167-183. </span><span lang=en-ca><a href="https://doi.org/10.1300/j111v41n01_13"><span lang=en-us style='font-size: 10.0pt;font-family:"palatino linotype",serif;mso-ansi-language:en-us'>https://doi.org/10.1300/j111v41n01_13</span></a></span><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span class=spelle><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us'>koufogiannakis</span></span><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>, d. &amp; wiebe, n. (2006). effective methods for teaching information literacy skills to undergraduate students: a systematic review and meta-analysis. <i style='mso-bidi-font-style:normal'>evidence based library and information practice, 1</i>(3), 3-43. </span><span lang=en-ca><a href="https://doi.org/10.18438/b8ms3d"><span lang=en-us style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us'>https://doi.org/10.18438/b8ms3d</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>liu, q., peng, w., zhang, f., hu, r., li, y., &amp; yan, w. (2016). the effectiveness of blended learning in health professions: systematic review and meta-analysis.<i style='mso-bidi-font-style:normal'> journal of medical internet research, 18</i>(1), e2. </span><span lang=en-ca><a href="https://doi.org/10.2196/jmir.4807"><span lang=en-us style='font-size: 10.0pt;font-family:"palatino linotype",serif;mso-ansi-language:en-us'>https://doi.org/10.2196/jmir.4807</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>markle, r., brenneman, m., jackson, t., burrus, j., &amp; robbins, s. (2013). <i style='mso-bidi-font-style:normal'>synthesizing frameworks of higher education student learning outcomes</i>. (research report ets rr-13-22). retrieved from </span><span lang=en-ca><a href="https://files.eric.ed.gov/fulltext/ej1109931.pdf"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>https://files.eric.ed.gov/fulltext/ej1109931.pdf</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>mcgowan, b., gonzalez, m., &amp; <span class=spelle>stanny</span>, c. j. (2016). what do undergraduate course syllabi say about information literacy? <i style='mso-bidi-font-style:normal'>portal: libraries and the academy, 16</i>(3), 599-617. </span><span lang=en-ca><a href="https://doi.org/10.1353/pla.2016.0040"><span lang=en-us style='font-size: 10.0pt;font-family:"palatino linotype",serif;mso-ansi-language:en-us'>https://doi.org/10.1353/pla.2016.0040</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>moher, d., <span class=spelle>liberati</span>, a., <span class=spelle>tetzlaff</span>, j., altman, d. g., &amp; the prisma group. (2009). preferred reporting items for systematic reviews and meta-analyses: the prisma statement. <span class=spelle><i style='mso-bidi-font-style:normal'>plos</i></span><i style='mso-bidi-font-style:normal'> medicine, 6</i>(7), e1000097. </span><span lang=en-ca><a href="https://doi.org/10.1371/journal.pmed.1000097"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>https://doi.org/10.1371/journal.pmed.1000097</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>morris, s. b, &amp; <span class=spelle>deshon</span>, r. p. (2002). combining effect size estimates in meta-analysis with repeated measures and independent-group designs. <i style='mso-bidi-font-style:normal'>psychological methods, 7</i>(1), 105-125. </span><span lang=en-ca><a href="https://doi.org/10.1037/1082-989x.7.1.105"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>https://doi.org/10.1037//1082-989x.7.1.105</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>morrison, j. m., sullivan, f., murray, e., &amp; jolly, b. (1999). evidence-based education: development of an instrument to critically appraise reports of educational interventions. <i style='mso-bidi-font-style: normal'>medical education, 33</i>(12), 890-893. </span><span lang=en-ca><a href="https://doi.org/10.1046/j.1365-2923.1999.00479.x"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>https://doi.org/10.1046/j.1365-2923.1999.00479.x</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span class=spelle><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us'>o’clair</span></span><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>, k. (2013). preparing graduate students for graduate-level study and research. <i style='mso-bidi-font-style:normal'>reference services review, 41</i>(2), 336-350. </span><span lang=en-ca><a href="https://doi.org/10.1108/00907321311326255"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>https://doi.org/10.1108/00907321311326255</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>r core team (2018). r: a language and environment for statistical computing. r foundation for statistical computing, vienna, austria. retrieved from </span><span lang=en-ca><a href="https://www.r-project.org/"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>https://www.r-project.org/</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>ramos, k. d., schafer, s., &amp; <span class=spelle>tracz</span>, s. m. (2003). validation of the fresno test of competence in <span class=grame>evidence based</span> medicine. <i>bmj: british medical journal</i>, <i style='mso-bidi-font-style:normal'>326</i>(7384), 319–321. </span><span lang=en-ca><a href="https://doi.org/10.1136/bmj.326.7384.319"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>https://doi.org/10.1136/bmj.326.7384.319</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>rosenberg, w. m., <span class=spelle>deeks</span>, j., lusher, a., snowball, r., dooley, g. &amp; sackett, d. (1998). improving searching skills and evidence retrieval. <i style='mso-bidi-font-style:normal'>journal of the royal college of physicians of london, 32</i>(6), 557-563. <o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>rosenthal, r. (1979). the “file drawer problem” and tolerance for null results. <i style='mso-bidi-font-style:normal'>psychological bulletin, 86</i>(3), 638-641. </span><span lang=en-ca><a href="https://doi.org/10.1037/0033-2909.86.3.638"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>https://doi.org/10.1037//0033-2909.86.3.638</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span class=spelle><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us'>shinogle</span></span><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>, j. (2012). <i style='mso-bidi-font-style:normal'>methodological challenges associated with meta-analyses in health care and behavioral health research</i>. retrieved from </span><span lang=en-ca><a href="https://www.bio.org/sites/default/files/meta%20analyses.pdf"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>https://www.bio.org/sites/default/files/meta%20analyses.pdf</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonospacing style='margin-left:.5in;text-indent:-.5in'><span style='font-size:10.0pt;font-family:"palatino linotype",serif'>song, f., hooper, l., &amp; loke, y. k. (2013). publication bias: what is it? how do we measure it? how do we avoid it? <i style='mso-bidi-font-style:normal'>open access</i> <i style='mso-bidi-font-style:normal'>journal of clinical trials, 2013</i>(5), 71-81. </span><a href="https://doi.org/10.2147/oajct.s34419"><span style='font-size:10.0pt;font-family:"palatino linotype",serif'>https://doi.org/10.2147/oajct.s34419</span></a><span style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>thornton, a. &amp; lee, p. (2000). publication bias in meta-analysis: its causes and consequences. <i style='mso-bidi-font-style:normal'>journal of clinical epidemiology, 53</i>(2), 207-216. </span><span lang=en-ca><a href="https://doi.org/10.1016/s0895-4356(99)00161-4"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>https://doi.org/10.1016/s0895-4356(99)00161-4</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span class=spelle><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us'>viechtbauer</span></span><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>, w. (2010). conducting meta-analyses in r with the <span class=spelle>metafor</span> package. <i style='mso-bidi-font-style:normal'>journal of statistical software, 36</i>(3), 1-48. retrieved from </span><span lang=en-ca><a href="http://www.jstatsoft.org/v36/i03/"><span lang=en-us style='font-size: 10.0pt;font-family:"palatino linotype",serif;mso-ansi-language:en-us'>http://www.jstatsoft.org/v36/i03/</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'>appendix a<o:p></o:p></span></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'>search strategies<o:p></o:p></span></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>all searches were run on 11 march 2019 and were limited to english language and a date range of 2000-2019.<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>1. the following five databases were searched concurrently through the ebsco interface with the “select a field” option<a style='mso-footnote-id:ftn1' href="#_ftn1" name="_ftnref1" title=""><sup><span style='mso-special-character: footnote'><![if !supportfootnotes]><sup><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman";mso-ansi-language:en-us;mso-fareast-language: en-us;mso-bidi-language:ar-sa'>[1]</span></sup><![endif]></span></sup></a>:<o:p></o:p></span></p> <ul style='margin-top:0in' type=disc> <li class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-list:l2 level1 lfo22'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>library literature and information science index (h. w. wilson)<o:p></o:p></span></li> <li class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-list:l2 level1 lfo22'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>library, information science &amp; technology abstracts<o:p></o:p></span></li> <li class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-list:l2 level1 lfo22'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>medline<o:p></o:p></span></li> <li class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-list:l2 level1 lfo22'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>cinahl<o:p></o:p></span></li> <li class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-list:l2 level1 lfo22'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>eric<o:p></o:p></span></li> </ul> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>using this search:<o:p></o:p></span></p> <ul style='margin-top:0in' type=disc> <li class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-list:l13 level1 lfo23'><span class=spelle><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us'>librar</span></span><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>* and (information literacy or instruct* or train* or orient* or <span class=spelle>educat</span>* or library user education or library instruction or library orientation) and (random* or rct or (pre and post) or (before and after)) and <a name="_hlk26962461">(</a><span class=grame><span style='mso-bookmark:_hlk26962461'>(( graduate</span></span><span style='mso-bookmark:_hlk26962461'> or masters or doctoral or phd) and (student* or study or studies or program* or degree or education))</span> or medical student* or dental student* or professional student*)<o:p></o:p></span></li> </ul> <p class=msonormal style='margin-left:.5in;page-break-after:avoid;mso-outline-level: 2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>2. library and information science abstracts (lisa) was searched through the proquest interface using this search:<o:p></o:p></span></p> <ul style='margin-top:0in' type=disc> <li class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-list:l22 level1 lfo24'><span class=spelle><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us'>librar</span></span><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>? and (information literacy or instruct? or train? or orient? or <span class=spelle>educat</span>? or library user education or library instruction or library orientation) and (random? or rct or (“pre” and post) or (before and after)) and ((graduate or masters or doctoral or phd) and (student? or study or studies or program? or degree or education) or medical student? or professional student? or dental student?)<o:p></o:p></span></li> </ul> <p class=msonormal style='margin-left:.5in;page-break-after:avoid;mso-outline-level: 2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>3. proquest dissertations and theses global was searched using this search:<o:p></o:p></span></p> <ul style='margin-top:0in' type=disc> <li class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-list:l22 level1 lfo24'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>ab((<span class=spelle>librar</span>?) and (information literacy or instruct? or train? or orient? or <span class=spelle>educat</span>? or library user education or library instruction or library orientation) and (random? or rct or (&quot;pre&quot; and post) or (before and after)) and ((graduate or masters or doctoral or phd) and (student? or study or studies or program? or degree or education))) or ti((<span class=spelle>librar</span>?) and (information literacy or instruct? or train? or orient? or <span class=spelle>educat</span>? or library user education or library instruction or library orientation) and (random? or rct or (&quot;pre&quot; and post) or (before and after)) and ((graduate or masters or doctoral or phd) and (student? or study or studies or program? or degree or education) or medical student? or dental student? or professional student?))<b style='mso-bidi-font-weight:normal'><o:p></o:p></b></span></li> </ul> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'>appendix b<o:p></o:p></span></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'>data used in meta-analyses<o:p></o:p></span></b></p> <table class=msotable15plain3 border=0 cellspacing=0 cellpadding=0 width=624 style='width:6.5in;border-collapse:collapse;mso-yfti-tbllook:1184;mso-padding-alt: 0in 5.4pt 0in 5.4pt'> <tr style='mso-yfti-irow:-1;mso-yfti-firstrow:yes;mso-yfti-lastfirstrow:yes; height:14.75pt'> <td width=187 nowrap valign=top style='width:140.0pt;border:none;border-bottom: solid #7f7f7f 1.0pt;mso-border-bottom-themecolor:text1;mso-border-bottom-themetint: 128;mso-border-bottom-alt:solid #7f7f7f .5pt;mso-border-bottom-themecolor: text1;mso-border-bottom-themetint:128;padding:0in 5.4pt 0in 5.4pt;height: 14.75pt'></td> <td width=72 nowrap colspan=2 valign=top style='width:53.7pt;border:none; border-bottom:solid #7f7f7f 1.0pt;mso-border-bottom-themecolor:text1; mso-border-bottom-themetint:128;mso-border-bottom-alt:solid #7f7f7f .5pt; mso-border-bottom-themecolor:text1;mso-border-bottom-themetint:128; padding:0in 5.4pt 0in 5.4pt;height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:1'><b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; text-transform:uppercase;mso-ansi-language:en-us'>n_pre<o:p></o:p></span></b></p> </td> <td width=75 nowrap valign=top style='width:56.3pt;border:none;border-bottom: solid #7f7f7f 1.0pt;mso-border-bottom-themecolor:text1;mso-border-bottom-themetint: 128;mso-border-bottom-alt:solid #7f7f7f .5pt;mso-border-bottom-themecolor: text1;mso-border-bottom-themetint:128;padding:0in 5.4pt 0in 5.4pt;height: 14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:1'><b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; text-transform:uppercase;mso-ansi-language:en-us'>m_pre<o:p></o:p></span></b></p> </td> <td width=71 nowrap valign=top style='width:53.3pt;border:none;border-bottom: solid #7f7f7f 1.0pt;mso-border-bottom-themecolor:text1;mso-border-bottom-themetint: 128;mso-border-bottom-alt:solid #7f7f7f .5pt;mso-border-bottom-themecolor: text1;mso-border-bottom-themetint:128;padding:0in 5.4pt 0in 5.4pt;height: 14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:1'><b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; text-transform:uppercase;mso-ansi-language:en-us'>sd_pre<o:p></o:p></span></b></p> </td> <td width=71 nowrap valign=top style='width:53.4pt;border:none;border-bottom: solid #7f7f7f 1.0pt;mso-border-bottom-themecolor:text1;mso-border-bottom-themetint: 128;mso-border-bottom-alt:solid #7f7f7f .5pt;mso-border-bottom-themecolor: text1;mso-border-bottom-themetint:128;padding:0in 5.4pt 0in 5.4pt;height: 14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:1'><b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; text-transform:uppercase;mso-ansi-language:en-us'>n_post<o:p></o:p></span></b></p> </td> <td width=72 nowrap valign=top style='width:53.9pt;border:none;border-bottom: solid #7f7f7f 1.0pt;mso-border-bottom-themecolor:text1;mso-border-bottom-themetint: 128;mso-border-bottom-alt:solid #7f7f7f .5pt;mso-border-bottom-themecolor: text1;mso-border-bottom-themetint:128;padding:0in 5.4pt 0in 5.4pt;height: 14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:1'><b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; text-transform:uppercase;mso-ansi-language:en-us'>m_post<o:p></o:p></span></b></p> </td> <td width=77 nowrap valign=top style='width:57.4pt;border:none;border-bottom: solid #7f7f7f 1.0pt;mso-border-bottom-themecolor:text1;mso-border-bottom-themetint: 128;mso-border-bottom-alt:solid #7f7f7f .5pt;mso-border-bottom-themecolor: text1;mso-border-bottom-themetint:128;padding:0in 5.4pt 0in 5.4pt;height: 14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:1'><b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; text-transform:uppercase;mso-ansi-language:en-us'>sd_post<o:p></o:p></span></b></p> </td> </tr> <tr style='mso-yfti-irow:0;height:14.75pt'> <td width=187 valign=top style='width:140.0pt;border:none;border-right:solid #7f7f7f 1.0pt; mso-border-right-themecolor:text1;mso-border-right-themetint:128;mso-border-right-alt: solid #7f7f7f .5pt;mso-border-right-themecolor:text1;mso-border-right-themetint: 128;background:#f2f2f2;mso-background-themecolor:background1;mso-background-themeshade: 242;padding:0in 5.4pt 0in 5.4pt;height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:68'><b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; text-transform:uppercase;mso-ansi-language:en-us'>aronoff et al., 2017<o:p></o:p></span></b></p> </td> <td width=57 valign=top style='width:42.9pt;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>40<o:p></o:p></span></p> </td> <td width=89 colspan=2 valign=top style='width:67.1pt;background:#f2f2f2; mso-background-themecolor:background1;mso-background-themeshade:242; padding:0in 5.4pt 0in 5.4pt;height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>64<o:p></o:p></span></p> </td> <td width=71 valign=top style='width:53.3pt;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>16<sup>1</sup><o:p></o:p></span></p> </td> <td width=71 valign=top style='width:53.4pt;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>40<o:p></o:p></span></p> </td> <td width=72 valign=top style='width:53.9pt;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>73<o:p></o:p></span></p> </td> <td width=77 valign=top style='width:57.4pt;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>16<sup>1</sup><o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:1;height:14.75pt'> <td width=187 valign=top style='width:140.0pt;border:none;border-right:solid #7f7f7f 1.0pt; mso-border-right-themecolor:text1;mso-border-right-themetint:128;mso-border-right-alt: solid #7f7f7f .5pt;mso-border-right-themecolor:text1;mso-border-right-themetint: 128;padding:0in 5.4pt 0in 5.4pt;height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:4'><b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; text-transform:uppercase;mso-ansi-language:en-us'>beile et al., 2004 group 1<o:p></o:p></span></b></p> </td> <td width=57 valign=top style='width:42.9pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>16<o:p></o:p></span></p> </td> <td width=89 colspan=2 valign=top style='width:67.1pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>60<o:p></o:p></span></p> </td> <td width=71 valign=top style='width:53.3pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>9.83<o:p></o:p></span></p> </td> <td width=71 valign=top style='width:53.4pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>16<o:p></o:p></span></p> </td> <td width=72 valign=top style='width:53.9pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>70.63<o:p></o:p></span></p> </td> <td width=77 valign=top style='width:57.4pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>11.53<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:2;height:14.75pt'> <td width=187 valign=top style='width:140.0pt;border:none;border-right:solid #7f7f7f 1.0pt; mso-border-right-themecolor:text1;mso-border-right-themetint:128;mso-border-right-alt: solid #7f7f7f .5pt;mso-border-right-themecolor:text1;mso-border-right-themetint: 128;background:#f2f2f2;mso-background-themecolor:background1;mso-background-themeshade: 242;padding:0in 5.4pt 0in 5.4pt;height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:68'><b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; text-transform:uppercase;mso-ansi-language:en-us'>beile et al., 2004 group 2<o:p></o:p></span></b></p> </td> <td width=57 valign=top style='width:42.9pt;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>19<o:p></o:p></span></p> </td> <td width=89 colspan=2 valign=top style='width:67.1pt;background:#f2f2f2; mso-background-themecolor:background1;mso-background-themeshade:242; padding:0in 5.4pt 0in 5.4pt;height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>54.21<o:p></o:p></span></p> </td> <td width=71 valign=top style='width:53.3pt;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>14.65<o:p></o:p></span></p> </td> <td width=71 valign=top style='width:53.4pt;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>19<o:p></o:p></span></p> </td> <td width=72 valign=top style='width:53.9pt;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>71.32<o:p></o:p></span></p> </td> <td width=77 valign=top style='width:57.4pt;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>12<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:3;height:14.75pt'> <td width=187 valign=top style='width:140.0pt;border:none;border-right:solid #7f7f7f 1.0pt; mso-border-right-themecolor:text1;mso-border-right-themetint:128;mso-border-right-alt: solid #7f7f7f .5pt;mso-border-right-themecolor:text1;mso-border-right-themetint: 128;padding:0in 5.4pt 0in 5.4pt;height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:4'><b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; text-transform:uppercase;mso-ansi-language:en-us'>beile et al., 2004 group 3<o:p></o:p></span></b></p> </td> <td width=57 valign=top style='width:42.9pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>14<o:p></o:p></span></p> </td> <td width=89 colspan=2 valign=top style='width:67.1pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>63.57<o:p></o:p></span></p> </td> <td width=71 valign=top style='width:53.3pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>15.62<o:p></o:p></span></p> </td> <td width=71 valign=top style='width:53.4pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>14<o:p></o:p></span></p> </td> <td width=72 valign=top style='width:53.9pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>78.57<o:p></o:p></span></p> </td> <td width=77 valign=top style='width:57.4pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>13.93<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:4;height:14.75pt'> <td width=187 valign=top style='width:140.0pt;border:none;border-right:solid #7f7f7f 1.0pt; mso-border-right-themecolor:text1;mso-border-right-themetint:128;mso-border-right-alt: solid #7f7f7f .5pt;mso-border-right-themecolor:text1;mso-border-right-themetint: 128;background:#f2f2f2;mso-background-themecolor:background1;mso-background-themeshade: 242;padding:0in 5.4pt 0in 5.4pt;height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:68'><b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; text-transform:uppercase;mso-ansi-language:en-us'>chiarella et al., 2014<o:p></o:p></span></b></p> </td> <td width=57 valign=top style='width:42.9pt;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>61<o:p></o:p></span></p> </td> <td width=89 colspan=2 valign=top style='width:67.1pt;background:#f2f2f2; mso-background-themecolor:background1;mso-background-themeshade:242; padding:0in 5.4pt 0in 5.4pt;height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>78.9<o:p></o:p></span></p> </td> <td width=71 valign=top style='width:53.3pt;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>15.7<o:p></o:p></span></p> </td> <td width=71 valign=top style='width:53.4pt;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>61<o:p></o:p></span></p> </td> <td width=72 valign=top style='width:53.9pt;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>79.9<o:p></o:p></span></p> </td> <td width=77 valign=top style='width:57.4pt;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>15.8<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:5;height:14.75pt'> <td width=187 valign=top style='width:140.0pt;border:none;border-right:solid #7f7f7f 1.0pt; mso-border-right-themecolor:text1;mso-border-right-themetint:128;mso-border-right-alt: solid #7f7f7f .5pt;mso-border-right-themecolor:text1;mso-border-right-themetint: 128;padding:0in 5.4pt 0in 5.4pt;height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:4'><b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; text-transform:uppercase;mso-ansi-language:en-us'>dorsch et al., 2004<o:p></o:p></span></b></p> </td> <td width=57 valign=top style='width:42.9pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>33<o:p></o:p></span></p> </td> <td width=89 colspan=2 valign=top style='width:67.1pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>56.11<o:p></o:p></span></p> </td> <td width=71 valign=top style='width:53.3pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>13.14<sup>2</sup><o:p></o:p></span></p> </td> <td width=71 valign=top style='width:53.4pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>30<o:p></o:p></span></p> </td> <td width=72 valign=top style='width:53.9pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>64.08<o:p></o:p></span></p> </td> <td width=77 valign=top style='width:57.4pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>13.14<sup>2</sup><o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:6;height:14.75pt'> <td width=187 valign=top style='width:140.0pt;border:none;border-right:solid #7f7f7f 1.0pt; mso-border-right-themecolor:text1;mso-border-right-themetint:128;mso-border-right-alt: solid #7f7f7f .5pt;mso-border-right-themecolor:text1;mso-border-right-themetint: 128;background:#f2f2f2;mso-background-themecolor:background1;mso-background-themeshade: 242;padding:0in 5.4pt 0in 5.4pt;height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:68'><b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; text-transform:uppercase;mso-ansi-language:en-us'>emmett et al., 2007<o:p></o:p></span></b></p> </td> <td width=57 valign=top style='width:42.9pt;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>16<o:p></o:p></span></p> </td> <td width=89 colspan=2 valign=top style='width:67.1pt;background:#f2f2f2; mso-background-themecolor:background1;mso-background-themeshade:242; padding:0in 5.4pt 0in 5.4pt;height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>47.5<o:p></o:p></span></p> </td> <td width=71 valign=top style='width:53.3pt;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>15.6<sup>1</sup><o:p></o:p></span></p> </td> <td width=71 valign=top style='width:53.4pt;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>16<o:p></o:p></span></p> </td> <td width=72 valign=top style='width:53.9pt;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>74.1<o:p></o:p></span></p> </td> <td width=77 valign=top style='width:57.4pt;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>5.03<sup>1</sup><o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:7;height:14.75pt'> <td width=187 valign=top style='width:140.0pt;border:none;border-right:solid #7f7f7f 1.0pt; mso-border-right-themecolor:text1;mso-border-right-themetint:128;mso-border-right-alt: solid #7f7f7f .5pt;mso-border-right-themecolor:text1;mso-border-right-themetint: 128;padding:0in 5.4pt 0in 5.4pt;height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:4'><b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; text-transform:uppercase;mso-ansi-language:en-us'>grant et al., 2006<o:p></o:p></span></b></p> </td> <td width=57 valign=top style='width:42.9pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>13<o:p></o:p></span></p> </td> <td width=89 colspan=2 valign=top style='width:67.1pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>4.58<o:p></o:p></span></p> </td> <td width=71 valign=top style='width:53.3pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>1.5<o:p></o:p></span></p> </td> <td width=71 valign=top style='width:53.4pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>11<o:p></o:p></span></p> </td> <td width=72 valign=top style='width:53.9pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>6.45<o:p></o:p></span></p> </td> <td width=77 valign=top style='width:57.4pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>1.46<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:8;height:14.75pt'> <td width=187 valign=top style='width:140.0pt;border:none;border-right:solid #7f7f7f 1.0pt; mso-border-right-themecolor:text1;mso-border-right-themetint:128;mso-border-right-alt: solid #7f7f7f .5pt;mso-border-right-themecolor:text1;mso-border-right-themetint: 128;background:#f2f2f2;mso-background-themecolor:background1;mso-background-themeshade: 242;padding:0in 5.4pt 0in 5.4pt;height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:68'><b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; text-transform:uppercase;mso-ansi-language:en-us'>ivanitskaya et al., 2008<o:p></o:p></span></b></p> </td> <td width=57 valign=top style='width:42.9pt;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>14<o:p></o:p></span></p> </td> <td width=89 colspan=2 valign=top style='width:67.1pt;background:#f2f2f2; mso-background-themecolor:background1;mso-background-themeshade:242; padding:0in 5.4pt 0in 5.4pt;height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>39<o:p></o:p></span></p> </td> <td width=71 valign=top style='width:53.3pt;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>6.31<o:p></o:p></span></p> </td> <td width=71 valign=top style='width:53.4pt;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>14<o:p></o:p></span></p> </td> <td width=72 valign=top style='width:53.9pt;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>41.36<o:p></o:p></span></p> </td> <td width=77 valign=top style='width:57.4pt;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>6.33<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:9;height:14.75pt'> <td width=187 valign=top style='width:140.0pt;border:none;border-right:solid #7f7f7f 1.0pt; mso-border-right-themecolor:text1;mso-border-right-themetint:128;mso-border-right-alt: solid #7f7f7f .5pt;mso-border-right-themecolor:text1;mso-border-right-themetint: 128;padding:0in 5.4pt 0in 5.4pt;height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:4'><b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; text-transform:uppercase;mso-ansi-language:en-us'>lechner, 2005 group 1<o:p></o:p></span></b></p> </td> <td width=57 valign=top style='width:42.9pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>17<o:p></o:p></span></p> </td> <td width=89 colspan=2 valign=top style='width:67.1pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>48.9<o:p></o:p></span></p> </td> <td width=71 valign=top style='width:53.3pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>13.14<sup>2</sup><o:p></o:p></span></p> </td> <td width=71 valign=top style='width:53.4pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>17<o:p></o:p></span></p> </td> <td width=72 valign=top style='width:53.9pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>67<o:p></o:p></span></p> </td> <td width=77 valign=top style='width:57.4pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>13.14<sup>2</sup><o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:10;height:14.75pt'> <td width=187 valign=top style='width:140.0pt;border:none;border-right:solid #7f7f7f 1.0pt; mso-border-right-themecolor:text1;mso-border-right-themetint:128;mso-border-right-alt: solid #7f7f7f .5pt;mso-border-right-themecolor:text1;mso-border-right-themetint: 128;background:#f2f2f2;mso-background-themecolor:background1;mso-background-themeshade: 242;padding:0in 5.4pt 0in 5.4pt;height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:68'><b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; text-transform:uppercase;mso-ansi-language:en-us'>lechner, 2005 group 2<o:p></o:p></span></b></p> </td> <td width=57 valign=top style='width:42.9pt;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>10<o:p></o:p></span></p> </td> <td width=89 colspan=2 valign=top style='width:67.1pt;background:#f2f2f2; mso-background-themecolor:background1;mso-background-themeshade:242; padding:0in 5.4pt 0in 5.4pt;height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>54.5<o:p></o:p></span></p> </td> <td width=71 valign=top style='width:53.3pt;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>13.14<sup>2</sup><o:p></o:p></span></p> </td> <td width=71 valign=top style='width:53.4pt;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>10<o:p></o:p></span></p> </td> <td width=72 valign=top style='width:53.9pt;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>62.6<o:p></o:p></span></p> </td> <td width=77 valign=top style='width:57.4pt;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>13.14<sup>2</sup><o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:11;height:14.75pt'> <td width=187 nowrap valign=top style='width:140.0pt;border:none;border-right: solid #7f7f7f 1.0pt;mso-border-right-themecolor:text1;mso-border-right-themetint: 128;mso-border-right-alt:solid #7f7f7f .5pt;mso-border-right-themecolor:text1; mso-border-right-themetint:128;padding:0in 5.4pt 0in 5.4pt;height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:4'><b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; text-transform:uppercase;mso-ansi-language:en-us'>maranda et al., 2016<o:p></o:p></span></b></p> </td> <td width=57 valign=top style='width:42.9pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>60<o:p></o:p></span></p> </td> <td width=89 colspan=2 valign=top style='width:67.1pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>43.33<o:p></o:p></span></p> </td> <td width=71 valign=top style='width:53.3pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>35.01<o:p></o:p></span></p> </td> <td width=71 valign=top style='width:53.4pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>60<o:p></o:p></span></p> </td> <td width=72 valign=top style='width:53.9pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>52.5<o:p></o:p></span></p> </td> <td width=77 valign=top style='width:57.4pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>38.43<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:12;height:14.75pt'> <td width=187 nowrap valign=top style='width:140.0pt;border:none;border-right: solid #7f7f7f 1.0pt;mso-border-right-themecolor:text1;mso-border-right-themetint: 128;mso-border-right-alt:solid #7f7f7f .5pt;mso-border-right-themecolor:text1; mso-border-right-themetint:128;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:68'><b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; text-transform:uppercase;mso-ansi-language:en-us'>otto, 2012<o:p></o:p></span></b></p> </td> <td width=57 nowrap valign=top style='width:42.9pt;background:#f2f2f2; mso-background-themecolor:background1;mso-background-themeshade:242; padding:0in 5.4pt 0in 5.4pt;height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>13<o:p></o:p></span></p> </td> <td width=89 nowrap colspan=2 valign=top style='width:67.1pt;background:#f2f2f2; mso-background-themecolor:background1;mso-background-themeshade:242; padding:0in 5.4pt 0in 5.4pt;height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>5.77<o:p></o:p></span></p> </td> <td width=71 nowrap valign=top style='width:53.3pt;background:#f2f2f2; mso-background-themecolor:background1;mso-background-themeshade:242; padding:0in 5.4pt 0in 5.4pt;height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>1.74<sup>3</sup><o:p></o:p></span></p> </td> <td width=71 valign=top style='width:53.4pt;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>13<o:p></o:p></span></p> </td> <td width=72 nowrap valign=top style='width:53.9pt;background:#f2f2f2; mso-background-themecolor:background1;mso-background-themeshade:242; padding:0in 5.4pt 0in 5.4pt;height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>7.62<o:p></o:p></span></p> </td> <td width=77 nowrap valign=top style='width:57.4pt;background:#f2f2f2; mso-background-themecolor:background1;mso-background-themeshade:242; padding:0in 5.4pt 0in 5.4pt;height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>2.1<sup>3</sup><o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:13;height:14.75pt'> <td width=187 nowrap valign=top style='width:140.0pt;border:none;border-right: solid #7f7f7f 1.0pt;mso-border-right-themecolor:text1;mso-border-right-themetint: 128;mso-border-right-alt:solid #7f7f7f .5pt;mso-border-right-themecolor:text1; mso-border-right-themetint:128;padding:0in 5.4pt 0in 5.4pt;height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:4'><b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; text-transform:uppercase;mso-ansi-language:en-us'>shaffer, 2011 group 1<o:p></o:p></span></b></p> </td> <td width=57 valign=top style='width:42.9pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>29<o:p></o:p></span></p> </td> <td width=89 colspan=2 valign=top style='width:67.1pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>39.14<o:p></o:p></span></p> </td> <td width=71 nowrap valign=top style='width:53.3pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>17.631<o:p></o:p></span></p> </td> <td width=71 valign=top style='width:53.4pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>29<o:p></o:p></span></p> </td> <td width=72 valign=top style='width:53.9pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>62.76<o:p></o:p></span></p> </td> <td width=77 valign=top style='width:57.4pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>13.535<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:14;height:14.75pt'> <td width=187 nowrap valign=top style='width:140.0pt;border:none;border-right: solid #7f7f7f 1.0pt;mso-border-right-themecolor:text1;mso-border-right-themetint: 128;mso-border-right-alt:solid #7f7f7f .5pt;mso-border-right-themecolor:text1; mso-border-right-themetint:128;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:68'><b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; text-transform:uppercase;mso-ansi-language:en-us'>shaffer, 2011 group 2<o:p></o:p></span></b></p> </td> <td width=57 nowrap valign=top style='width:42.9pt;background:#f2f2f2; mso-background-themecolor:background1;mso-background-themeshade:242; padding:0in 5.4pt 0in 5.4pt;height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>30<o:p></o:p></span></p> </td> <td width=89 nowrap colspan=2 valign=top style='width:67.1pt;background:#f2f2f2; mso-background-themecolor:background1;mso-background-themeshade:242; padding:0in 5.4pt 0in 5.4pt;height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>40.83<o:p></o:p></span></p> </td> <td width=71 nowrap valign=top style='width:53.3pt;background:#f2f2f2; mso-background-themecolor:background1;mso-background-themeshade:242; padding:0in 5.4pt 0in 5.4pt;height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>14.568<o:p></o:p></span></p> </td> <td width=71 valign=top style='width:53.4pt;background:#f2f2f2;mso-background-themecolor: background1;mso-background-themeshade:242;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>30<o:p></o:p></span></p> </td> <td width=72 nowrap valign=top style='width:53.9pt;background:#f2f2f2; mso-background-themecolor:background1;mso-background-themeshade:242; padding:0in 5.4pt 0in 5.4pt;height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>63.33<o:p></o:p></span></p> </td> <td width=77 nowrap valign=top style='width:57.4pt;background:#f2f2f2; mso-background-themecolor:background1;mso-background-themeshade:242; padding:0in 5.4pt 0in 5.4pt;height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:64'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>16.312<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:15;mso-yfti-lastrow:yes;height:14.75pt'> <td width=187 nowrap valign=top style='width:140.0pt;border:none;border-right: solid #7f7f7f 1.0pt;mso-border-right-themecolor:text1;mso-border-right-themetint: 128;mso-border-right-alt:solid #7f7f7f .5pt;mso-border-right-themecolor:text1; mso-border-right-themetint:128;padding:0in 5.4pt 0in 5.4pt;height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2; mso-yfti-cnfc:4'><b><span style='font-size:10.0pt;font-family:"palatino linotype",serif; text-transform:uppercase;mso-ansi-language:en-us'>wema, 2006<o:p></o:p></span></b></p> </td> <td width=57 nowrap valign=top style='width:42.9pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>12<o:p></o:p></span></p> </td> <td width=89 nowrap colspan=2 valign=top style='width:67.1pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>31.8<o:p></o:p></span></p> </td> <td width=71 nowrap valign=top style='width:53.3pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>9.09<sup>3</sup><o:p></o:p></span></p> </td> <td width=71 valign=top style='width:53.4pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>12<o:p></o:p></span></p> </td> <td width=72 nowrap valign=top style='width:53.9pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>68.7<o:p></o:p></span></p> </td> <td width=77 nowrap valign=top style='width:57.4pt;padding:0in 5.4pt 0in 5.4pt; height:14.75pt'> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>6.23<sup>3</sup><o:p></o:p></span></p> </td> </tr> <![if !supportmisalignedcolumns]> <tr height=0> <td width=2800 style='border:none'></td> <td width=858 style='border:none'></td> <td width=216 style='border:none'></td> <td width=1126 style='border:none'></td> <td width=1066 style='border:none'></td> <td width=1068 style='border:none'></td> <td width=1078 style='border:none'></td> <td width=1148 style='border:none'></td> </tr> <![endif]> </table> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>n = number of participants, m = mean, <span class=spelle>sd</span> = standard deviation, _<span class=grame>pre refers</span> to pre-assessment, _post refers to post-assessment<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'>notes:<o:p></o:p></span></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><sup><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>1 </span></sup><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>information not provided in article; author provided data by email<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><sup><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>2</span></sup><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> sd not available, used average of other included studies to estimate sd (see furukawa et al., 2006 for justification)<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><sup><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>3 </span></sup><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>information not provided in article, calculated from available raw scores<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'>appendix c<o:p></o:p></span></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'>sample r code<o:p></o:p></span></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>#meta-analysis using smdh<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>&gt; library(<span class=spelle>metafor</span>)<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>&gt; data=read.csv(<span class=spelle><span class=grame>file.choose</span></span>())<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>&gt; data<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>&gt; <span class=spelle>mymeta</span>&lt;<span class=spelle><span class=grame>escalc</span></span><span class=grame>(</span>measure=&quot;smdh&quot;, n1i=<span class=spelle>n_post</span>, m1i=<span class=spelle>m_post</span>, sd1i=<span class=spelle>sd_post</span>, n2i=<span class=spelle>n_pre</span>, m2i=<span class=spelle>m_pre</span>, sd2i=<span class=spelle>sd_pre</span>, data=data)<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>&gt; <span class=spelle>mymeta</span><o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>&gt; results=<span class=spelle><span class=grame>rma</span></span><span class=grame>(</span><span class=spelle>yi</span>, vi, data=<span class=spelle>mymeta</span>, slab=paste (study), method=&quot;reml&quot;)<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>&gt; results<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>&gt;forest(results)<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>&gt;funnel(results)<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>#effects of moderators<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>&gt; <span class=spelle>resm</span>=<span class=spelle><span class=grame>rma</span></span><span class=grame>(</span><span class=spelle>yi</span>, vi, mods=~factor(<span class=spelle>allocest</span>),data=<span class=spelle>mymeta</span>, method=&quot;reml&quot;)<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>&gt; <span class=spelle>resm</span><o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>&gt; <span class=spelle>resf</span>&lt;-<span class=spelle><span class=grame>rma</span></span><span class=grame>(</span><span class=spelle>yi</span>, vi, mods=~<span class=spelle>allocf</span>, data=<span class=spelle>mymeta</span>)<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>&gt;<span class=spelle>resf</span><o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>&gt;<span class=spelle>resd</span>&lt;-<span class=spelle><span class=grame>rma</span></span><span class=grame>(</span><span class=spelle>yi</span>, vi, mods=~<span class=spelle>allocd</span>, data=<span class=spelle>mymeta</span>)<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>&gt;<span class=spelle>resd</span><o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>#combined effects of moderators to test residual heterogeneity<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>&gt;res&lt;-<span class=spelle><span class=grame>rma</span></span><span class=grame>(</span><span class=spelle>yi</span>, vi, mods=~<span class=spelle>allocd</span> + <span class=spelle>allocf</span>, data=<span class=spelle>mymeta</span>)<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>&gt; res<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>#rosenthal’s fail-safe number<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>&gt;<span class=spelle>fsn</span> (<span class=spelle>yi</span>, vi, data=<span class=spelle>mymeta</span>)<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'>appendix d<o:p></o:p></span></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'>studies included in the systematic review<o:p></o:p></span></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>note that studies with an * were not included in the meta-analysis.<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span class=spelle><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us'>aronoff</span></span><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>, n., <span class=spelle>stellrecht</span>, e., lyons, a. g., <span class=spelle>zafron</span>, m. l., <span class=spelle>glogowski</span>, m., grabowski, j., &amp; <span class=spelle>ohtake</span>, p. j. (2017). teaching evidence-based practice principles to prepare health professions students for an interprofessional learning experience.<i style='mso-bidi-font-style:normal'> journal of the medical library association, 105</i>(4), 376-384. </span><span lang=en-ca><a href="https://doi.org/10.5195/jmla.2017.179"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>https://doi.org/10.5195/jmla.2017.179</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'> </span><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span class=spelle><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us'>beile</span></span><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>, p. m. &amp; <span class=spelle>boote</span>, d. n. (2004). does the medium <span class=grame>matter?:</span> a comparison of a web-based tutorial with face-to-face library instruction on education students’ self-efficacy levels and learning outcomes. <i style='mso-bidi-font-style:normal'>research strategies, 20</i>(1-2), 57-68. </span><span lang=en-ca><a href="https://doi.org/10.1016/j.resstr.2005.07.002"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>https://doi.org/10.1016/j.resstr.2005.07.002</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span class=spelle><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us'>chiarella</span></span><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>, d., <span class=spelle>khadem</span>, t. m., brown, j. e., &amp; wrobel, m. j. (2014). information literacy skills retention over the first professional year of pharmacy school. <i style='mso-bidi-font-style:normal'>medical reference services quarterly, 33</i>(3), 302-312. </span><span lang=en-ca><a href="https://doi.org/10.1080/02763869.2014.925693"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>https://doi.org/10.1080/02763869.2014.925693</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span class=spelle><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us'>dorsch</span></span><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>, j. l., <span class=spelle>aiyer</span>, m. k., &amp; meyer, l. e. (2004). impact of an evidence-based medicine curriculum on medical students’ attitudes and skills. <i style='mso-bidi-font-style:normal'>journal of the medical library association, 92</i>(4), 397406. retrieved from </span><span lang=en-ca><a href="https://www.ncbi.nlm.nih.gov/pmc/articles/pmc521510/"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>https://www.ncbi.nlm.nih.gov/pmc/articles/pmc521510/</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>emmett, a. &amp; <span class=spelle>emde</span>, j. (2007). assessing information literacy skills using the acrl standards as a guide. <i style='mso-bidi-font-style:normal'>reference services review, 35</i>(2), 210-229. </span><span lang=en-ca><a href="https://doi.org/10.1108/00907320710749146"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>https://doi.org/10.1108/00907320710749146</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>grant, m. j., &amp; <span class=spelle>brettle</span>, a. j. (2006). developing and evaluating an interactive information skills tutorial. <i style='mso-bidi-font-style:normal'>health information and libraries journal, 23</i>(2), 79-88. </span><span lang=en-ca><a href="https://doi.org/10.1111/j.1471-1842.2006.00655.x"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>https://doi.org/10.1111/j.1471-1842.2006.00655.x</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>*<span class=spelle>ilic</span>, d., tepper, k., &amp; <span class=spelle>misso</span>, m. (2012). teaching evidence-based medicine literature searching skills to medical students during the clinical years: a randomized controlled trial. <i style='mso-bidi-font-style:normal'>journal of the medical library association, 100</i>(3), 190-196. </span><span lang=en-ca><a href="https://doi.org/10.3163/1536-5050.100.3.009"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>https://doi.org/10.3163/1536-5050.100.3.009</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span class=spelle><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us'>ivanitskaya</span></span><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>, l., <span class=spelle>duford</span>, s., craig, m. &amp; casey, a. m. (2008). how does a pre-assessment of off-campus students’ information literacy affect the effectiveness of library instruction? <i style='mso-bidi-font-style: normal'>journal of library administration, 48</i>(3-4), 509-525. </span><span lang=en-ca><a href="https://doi.org/10.1080/01930820802289649"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>https://doi.org/10.1080/01930820802289649</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>*lapidus, m., mccord, s. k., mccloskey, w. w., &amp; <span class=spelle>kostka-rokosz</span>, m. d. (2012). combined use of online tutorials and hands-on group exercises in bibliographic instruction for pharmacy students. <i style='mso-bidi-font-style:normal'>medical reference services quarterly, 31</i>(4), 383-399. </span><span lang=en-ca><a href="https://doi.org/10.1080/02763869.2012.724277"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>https://doi.org/10.1080/02763869.2012.724277</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>lechner, d. l. (2005). graduate student research instruction: testing an interactive web-based library tutorial for a health sciences database. <i style='mso-bidi-font-style:normal'>research strategies, 20</i>(4), 469-481. </span><span lang=en-ca><a href="https://doi.org/10.1016/j.resstr.2006.12.017"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>https://doi.org/10.1016/j.resstr.2006.12.017</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>maranda, s., harding, b., &amp; <span class=spelle>kinderman</span>, l. (2016). evaluation of the long-term impact of a curriculum-integrated medical information literacy program. <i style='mso-bidi-font-style:normal'>journal of the canadian health libraries association / journal de <span class=spelle>l’association</span> des <span class=spelle>bibliothèques</span> de la santé du canada, 37</i>(3), 109-117. </span><span lang=en-ca><a href="https://doi.org/10.5596/c16-026"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>https://doi.org/10.5596/c16-026</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>otto, j. l. (2012). assessing and improving data literacy: a study with urban and regional planning students. <i style='mso-bidi-font-style:normal'>pnla quarterly, 76</i>(4), 5-23. retrieved from </span><span lang=en-ca><a href="https://pnlaorg.files.wordpress.com/2018/03/volume-76-4.pdf"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>https://pnlaorg.files.wordpress.com/2018/03/volume-76-4.pdf</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>*schilling, k., <span class=spelle>wiecha</span>, j., <span class=spelle>polineni</span>, d., &amp; khalil, s. (2006). an interactive web-based curriculum on evidence-based medicine: design and effectiveness. <i style='mso-bidi-font-style:normal'>family medicine, 38</i>(2), 126-132. retrieved from </span><span lang=en-ca><a href="https://www.stfm.org/familymedicine/vol38issue2/schilling126"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>https://www.stfm.org/familymedicine/vol38issue2/schilling126</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>*<span class=spelle>schweikhard</span>, a. j., <span class=spelle>hoberecht</span>, t., peterson, a., &amp; randall, k. (2018). the impact of library tutorials on the information literacy skills of occupational therapy and physical therapy students in an evidence-based practice course: a rubric assessment. <i style='mso-bidi-font-style:normal'>medical reference services quarterly, 37</i>(1), 43-59. </span><span lang=en-ca><a href="https://doi.org/10.1080/02763869.2018.1404388"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>https://doi.org/10.1080/02763869.2018.1404388</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span lang=en-ca style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><span style='mso-spacerun:yes'> </span></span><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>shaffer, b. a. (2011). graduate student library research skills: is online instruction effective? <i style='mso-bidi-font-style: normal'>journal of library &amp; information services in distance learning, 5</i>(1-2), 35-55. </span><span lang=en-ca><a href="https://doi.org/10.1080/1533290x.2011.570546"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>https://doi.org/10.1080/1533290x.2011.570546</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:.5in;text-indent:-.5in;page-break-after: avoid;mso-outline-level:2'><span class=spelle><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-us'>wema</span></span><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'>, e. f. (2006). <i style='mso-bidi-font-style:normal'>developing information literacy <span class=spelle>programmes</span> for public university libraries in tanzania: a case study of the university of dar es salaam</i>. (doctoral dissertation). retrieved from </span><span lang=en-ca><a href="https://dspace.lboro.ac.uk/dspace-jspui/handle/2134/10918"><span lang=en-us style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'>https://dspace.lboro.ac.uk/dspace-jspui/handle/2134/10918</span></a></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us'> </span><span style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-ansi-language:en-us'><o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> </div> <div style='mso-element:footnote-list'><![if !supportfootnotes]><br clear=all> <hr align=left size=1 width="33%"> <![endif]> <div style='mso-element:footnote' id=ftn1> <p class=msofootnotetext><a style='mso-footnote-id:ftn1' href="#_ftnref1" name="_ftn1" title=""><span class=msofootnotereference><span style='mso-special-character: footnote'><![if !supportfootnotes]><span class=msofootnotereference><span style='font-size:10.0pt;font-family:"calibri",sans-serif;mso-ascii-theme-font: minor-latin;mso-fareast-font-family:calibri;mso-fareast-theme-font:minor-latin; mso-hansi-theme-font:minor-latin;mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi;mso-ansi-language:en-us;mso-fareast-language: en-us;mso-bidi-language:ar-sa'>[1]</span></span><![endif]></span></span></a><span style='font-family:"times new roman",serif'> “select a field” searches the author, subject, keyword, title, and abstract fields. (more information here: </span><a href="https://help.ebsco.com/interfaces/ebsco_guides/general_product_faqs/fields_searched_using_select_a_field_drop_down_list"><span style='font-family:"times new roman",serif'>https://help.ebsco.com/interfaces/ebsco_guides/general_product_faqs/fields_searched_using_select_a_field_drop_down_list</span></a><span style='font-family:"times new roman",serif'>)<o:p></o:p></span></p> </div> </div> </body> </html> evidence based library and information practice using evidence in practice   an evidence based approach to supporting library staff scholarly communication competencies   christie hurrell digital initiatives and scholarly communication librarian libraries and cultural resources, university of calgary calgary, alberta, canada email: christie.hurrell@ucalgary.ca   james e. murphy research and learning librarian libraries and cultural resources, university of calgary calgary, alberta, canada email: james.murphy2@ucalgary.ca    received: 20 mar. 2019                                                                  accepted: 18 apr. 2019      2019 hurrell and murphy. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29570   setting   the university of calgary is a research-intensive university with 14 faculties offering more than 250 academic programs and serving more than 30,000 students. libraries and cultural resources (lcr) delivers front line reference services in a variety of channels (in person, email, chat, and sms) to students, staff and faculty through seven physical libraries and via our website, which is powered by springshare’s libapps platform. library patrons, including faculty, students, staff, alumni, and members of the community, engage with front line reference staff to pose a wide variety of questions, including in the complex and rapidly-changing area of scholarly communication.   problem   the scholarly communication ecosystem has been in a period of disruption for a number of years, leaving both novice and experienced information seekers with unanswered information needs (myers, 2016). as defined by the association of college and research libraries (2006), scholarly communication is “the system through which research outputs are created, evaluated, disseminated, and preserved” (para. 1). much of the research literature to date has focused on training librarians in scholarly communication, even though front line staff may often be the first point of contact. if these staff have not been provided training to respond adequately, the quality of the reference interaction may be negatively impacted.   we were interested in assessing the frequency with which patrons approached front line staff members with questions related to scholarly communication and assessing whether or not staff members had adequate training and support to answer or appropriately refer these questions. to do this, we implemented a project to collect and analyze reference transactions before and after a training program. this allowed us to assess baseline competencies in scholarly communication, as well as the impact of the training program on reference transactions.   evidence   reference transactions were collected by all library staff members in the libapps platform via three channels:   libchat chat reference, which recorded transcripts of chat reference conversations reference analytics, where staff members input queries they received at any service point tickets, which recorded email and sms inquiries that may be answered or referred   for the project, we examined anonymized reference transactions collected through all three channels.   to capture reference transactions relating to scholarly communication, we searched for a variety of keywords. the keywords were selected based on content in a document developed by the joint task force on librarians’ competencies in support of e-­research and scholarly communication (calarco, shearer, schmidt, & tate, 2016), which described competencies in four categories:   scholarly publishing services open access repository services copyright and open access advice assessment of scholarly resources   the document outlines competencies in terms of knowledge, understandings, and abilities. we judged that front line staff members should be able to answer or refer basic questions in the knowledge categories and should be able to appropriately refer more complex questions.   we selected 12 keywords: publishing, open access, prism (our institutional repository), repository, deposit, orcid, impact, copyright, predatory, host, doaj, and creative commons. duplicates, transactions that were not complete enough to categorize, and irrelevant results were manually removed.   we analyzed and compared data between two time periods: (1) september to december 2017, and (2) september to december 2018. we sought to eliminate potential differences between fall and winter terms by comparing two fall term periods. the staff training sessions were held between the two time periods, in summer 2018.   data from 2017—the period before training—revealed 70 unique reference transactions.  questions including the keyword copyright were by far the most common (figure 1), and copyright and open access advice was the most common category as defined by the joint task force profile (figure 2). some keywords garnered no results (deposit, orcid, creative commons). questions on open access were second most frequent, while questions about the institutional repository, prism, were third most frequent.   the range of questions was broad. questions in the copyright and open access advice competency area included course reserve inquiries, use of third party materials (particularly images) in teaching and learning resources, fair dealing, scholarly sharing, and requests for advice around open access publishing. transactions in the open access repository services category included a large number of questions about access to electronic resources for alumni or community members, as well as questions about both the university of calgary’s institutional repository, prism, and third party scholarly sharing sites.    we found that the vast majority of questions were either answered or referred adequately. in the 2017 time period, 96% (n=67/70) of questions were deemed successfully answered or referred.   implementation   due to the diverse and fluid nature of scholarly communication topics, and because lcr did not have any internal training opportunities for current staff, the authors developed a training program. the training program involved two key components:   website updates and improvements   libraries and cultural resources (lcr) used frequently asked questions (faqs) through the springshare platform to both answer queries and direct patrons to more detailed information resources such as research guides. these were popular tools for both patrons and reference staff. in the spring and summer of 2018, we identified, reviewed, and updated or created new faqs relating to scholarly communication. topics included predatory publishers, bibliometrics, embargoes in the institutional repository, tools to legally access open access literature (e.g., browser extensions), and assessing the copyright status of digital images found online for use in teaching and learning. these topics reflected questions that were found in the initial assessment of reference transactions and were not adequately answered by current website content. we promoted library staff awareness of the scholarly communication faqs during the training sessions and using email and word of mouth to maximize impact.   training sessions   two interactive one-hour training sessions were designed and delivered to library staff members to promote knowledge and awareness of common scholarly communication questions. the content for the sessions was based on the joint task force on librarians’ competencies in support of e-­research and scholarly communication. our aim in designing the sessions was to ensure that front line staff members would have the capacity to correctly answer or refer queries in the knowledge categories in all four of the scholarly communication categories as defined by the joint task force document. the sessions included training regarding when to refer more complex questions. training sessions were structured in a series of scenarios, which small groups attempted to resolve and then discussed with the larger group. these were adapted from scenarios developed by the coppul scholarly communication working group (2018).   in total, 15 library staff members attended the two interactive training sessions. staff members represented a mix of academic and support staff. no formal assessment of these sessions was done but participants noted that the sessions were useful and that they planned to reuse the scenarios in their own teaching and reference practice.   outcome   data from 2018—which represented the time period after training—contained 76 unique reference transactions, representing an increase of 9% between time periods. generally, the categories of questions being asked were similar between the two time periods, although questions relating to open access showed a dramatic increase from 0.0 to 28% of total questions between time periods. this is likely due to a change in lcr’s open access authors fund which occurred in october 2018.     figure 1 frequency of transactions based on keywords present in reference transactions.     figure 2 frequency of interaction based on coar task force profile.     figure 3 reference transactions from the 2018 period that incorporated training materials.     in the 2018 data, 97% (n=71/73) questions were successfully answered or referred, representing a 1% increase from the 2017 period. additionally, there were 28 reference questions that directly addressed topics covered in the training. based on our analysis, half (n=14) of these transactions referenced the training in some way, either by directing patrons to a website faq, or rephrasing content from the training in a direct response (figure 3).   reflection   the primary goal of this project was to target the skill development of front-line academic library staff members. it was also important to make use of data that staff members collected daily and use it for evidence based training and quality improvement.   using three communications channels to collect data on scholarly communication transactions proved to be successful and provided a broad overview of the types of interactions the library received through chat, in person, email, or other. a weakness of the data was that it likely did not reflect all interactions. although most lcr staff were encouraged to record all transactions, there may have been transactions that were not entered, and some specialized staff (e.g., those in the copyright office) did not follow this workflow. additionally, reference analytics transactions that staff members manually entered were sometimes lacking details, making them impossible to categorize.   training initiatives were well-received and appear to have had an impact on competencies. however, the number of transactions captured in the second time period specifically related to training was small (n=28). of these, half showed evidence of training effects. more frequent training opportunities, and more outreach and engagement with front-line support staff would be beneficial. the combination of online tools and interactive in-person workshops were well-received by library staff but offering more options and on a more regular basis would be beneficial. the model employed at east carolina university provided a useful template for such initiatives (shirkey, hoover, & webb, 2019).   two resources were invaluable for structuring this project. the profile developed by the joint task force on librarians’ competencies in support of e-­research and scholarly communication (2016) provided scholarly communication categories and competencies. secondly, the training scenarios developed by the coppul scholarly communications working group (2018) provided compelling interactive scenarios for staff to engage and discuss key concepts and challenging questions in scholarly communication. incorporating both resources added to the success of the project.   conclusion   trends and developments in issues of copyright, open access, predatory publishing, and other realms of scholarly communication are continuing to unfold. scholars and students often look to the library to stay current and compliant with regulatory changes and best practices. as the first point of contact for many of these queries, it is essential to ensure library staff are well-equipped to respond and direct patrons toward success. using evidence from routinely-collected library data can assist libraries in continually improving their reference services.   references   association of college and research libraries. (2006). principles and strategies for the reform of scholarly communication 1. retrieved from http://www.ala.org/acrl/publications/whitepapers/principlesstrategies   calarco, p., shearer, k., schmidt, b., & tate, d. (2016, june). librarians’ competencies profile for scholarly communication and open access. retrieved from https://www.coar-repositories.org/files/competencies-for-scholcomm-and-oa_june-2016.pdf   coppul scholarly communications working group. (2018). talking to faculty and students about open access. retrieved from  https://coppulscwg.files.wordpress.com/2018/05/talking-to-faculty-students-about-open-access-1.pdf   myers, k. l. (2016). libraries’ response to scholarly communication in the digital era. endnotes: the journal of the new members round table 7(1), p.13-20. retrieved from http://www.ala.org/rt/sites/ala.org.rt/files/content/oversightgroups/comm/schres/endnotesvol7no1/article_scholarly_communication.pdf   shirkey, c., hoover, j., & webb, k. (2019). establishing a scholarly communication baseline using liaison competencies to design scholarly communication boot camp training sessions. retrieved from http://thescholarship.ecu.edu/handle/10342/7015 news   call for applicants: concordia university library 2018 researcher-in-residence program      2017. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   concordia university is one of canada’s most dynamic universities in one of north america’s most livable cities. it reflects a unique blend of commitments to diversity, accessibility, social responsibility, and innovation. with over 46,000 students (6,300 international) and nearly 1,700 full-time and part-time professors, concordia is recognized for offering a rich academic experience combined with strong community engagement.   this is a time of change for concordia university library. a new library space was recently inaugurated in the grey nuns building. a major renovation and expansion project is transforming the webster library at the sir george williams campus (downtown), and will be complete in the fall of 2017. an innovative technology program is being developed and will contribute to enhance the students experience in the libraries.   concordia library recently developed a strategic plan (https://library.concordia.ca/about/plan/) for 2016-2021. some of the highlights from the plan’s goals and objectives include:   the transformation of service offerings to provide an outstanding user experience the launch of the concordia university press to disseminate engaging research in the form of open access scholarly books the creation of a platform for the preservation and dissemination of oral history material the promotion of research and innovation in library and information studies   in light of this, the concordia library researcher-in-residence program was created to promote research in the library and the use of research by practitioners. the program offers the opportunity for scholars, information professionals, or doctoral students to focus on an area of inquiry in a supportive and enriching environment, and to interact with concordia library staff and its resources.   the presence of the researcher-in-residence in concordia library has as one of its goals to foster a culture of research. it is hoped that throughout the residency, both the researcher and staff will become engaged in the use of research in library practice and the concept of evidence-based librarianship. the residency period is from january 1 to december 1, 2018. proposals which include alternative start dates and those of different duration will also be considered.   the inaugural researcher-in-residence began in january 2017. more information can be found on the concordia website (https://library.concordia.ca/about/researcher-in-residence/emily-kopley.php). for the second researcher-in-residence program, concordia library invites proposals relating to any of the library’s strategic plan goals as well as the following themes:   the role of the library in a next-generation university the role of the library in the digital humanities collection development in an increasingly connected world use of space for learning and research digital libraries communication and outreach to users efficacy of library instruction professional development and capacity building of library staff   researchers and practitioners from library and information studies, as well as other disciplines (e.g., anthropology, computer science, education, informatics, media studies, sociology) are welcome to apply.   scope   as part of the program, concordia library will provide:   ·         dedicated workspace in concordia library ·         access to computing facilities ·         access to library resources, print and electronic collections, and technological infrastructure ·         opportunities for the researcher-in-residence to interact with concordia library staff ·         other types of in-kind support may be made available   in return, the researcher-in-residence is expected to participate in the intellectual life of concordia library. this participation may take various forms, including:   ·         presenting research in progress to staff ·         providing learning opportunities in the form of seminars or workshops for a campus or library audience on topics related to their own expertise ·         participating in selected activities in concordia library, depending upon the relevance to their research, their expertise, and interest. such activities include: o    committee meetings o    special events o    brown bag talks, seminars, round tables, or other information sessions o    concordia library research forum ·         meeting individually with librarians and library staff (e.g., office hour drop-ins, or by appointment)   funding   ·         for a current phd student, postdoctoral fellow, or independent scholar, a stipend of $20,000, plus up to $2,500 for travel/research expenses ·         for a faculty member on sabbatical leave, research funds of up to $20,000 (budget required)   eligibility   phd students, postdoctoral fellows, and full-time members of academic staff are eligible to apply. the proposed research should make a contribution to the knowledge base of librarianship, libraries, or information studies. professors and librarians preparing for a sabbatical are encouraged to apply.   application and instructions   applications should include information about the applicant, including the expertise brought to the program:   ·         letter of intent, including o    a description of study / project proposal o    proposed in-library activities (outline of activities to be undertaken during the period of residency) o    proposed research outputs ·         curriculum vitae ·         short biography (100 words) ·         for all applicants not on sabbatic leave, two letters of recommendation should be supplied. for doctoral students, one of these should be from the applicant’s doctoral supervisor. letters of recommendation should be submitted electronically directly by the referee. ·         for members of academic staff on sabbatic leave, a budget should also be included   selection criteria   submissions will be evaluated by a selection committee composed of the university librarian and four librarians using these criteria:   ·         quality and originality of the proposed research project ·         demonstrated ability to complete the proposed project based on previous accomplishments ·         significance of the proposed research for librarianship, libraries, or information studies   deadline   the application process opens april 3, 2017.   the deadline for receiving the complete application and letters of recommendation is august 4, 2017 by 5:00 pm edt. applications may be submitted in either english or french.   applications should be submitted as a single pdf file to lorie kloda, associate university librarian, planning & community relations, at lib-admin@concordia.ca   notification of the successful candidate will be given in september 2017.   the full details of the program may also be found at: https://library.concordia.ca/about/researcher-in-residence/   editorial evidence based library and information practice 2009, 4:2 120 evidence based library and information practice article understanding the information research process of experienced online information researchers to inform development of a scholars portal martha whitehead associate university librarian queen's university library kingston, ontario, canada email: martha.whitehead@queensu.ca terry costantino principal and information architect usability matters toronto, ontario, canada email: terry@usabilitymatters.com received: 04 march 2009 accepted: 12 may 2009 © 2009 whitehead and costantino. this is an open access article distributed under the terms of the creative commons attribution license, <http://creativecommons.org/licenses/by/2.0>, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective the main purpose of this study was to understand the information research process of experienced online information researchers in a variety of disciplines, gather their ideas for improvement and as part of this to validate a proposed research framework for use in future development of ontario’s scholars portal. methods this was a qualitative research study in which sixty experienced online information researchers participated in face-to-face workshops that included a collaborative design component. the sessions were conducted and recorded by usability specialists who subsequently analyzed the data and identified patterns and themes. results key themes included the similarities of the information research process across all disciplines, the impact of interdisciplinarity, the social aspect of research and opportunities for process improvement. there were many specific mailto:martha.whitehead@queensu.ca mailto:terry@usabilitymatters.com evidence based library and information practice 2009, 4:2 121 observations regarding current and ideal processes. implications for portal development and further research included: supporting a common process while accommodating user-defined differences; supporting citation chaining practices with new opportunities for data linkage and granularity; enhancing keyword searching with various types of intervention; exploring trusted social networks; exploring new mental models for data manipulation while retaining traditional objects; improving citation and document management. conclusion – the majority of researchers in the study had almost no routine in their information research processes, had developed few techniques to assist themselves and had very little awareness of the tools available to help them. there are many opportunities to aid researchers in the research process that can be explored when developing scholarly research portals. that development will be well guided by the framework ‘discover, gather, synthesize, create, share.’ introduction and context for the study the ontario council of university libraries (ocul), a twenty-one member consortium in canada, is at an interesting stage in the development of its scholars portal. the vision for scholars portal is a sophisticated electronic environment that enables easy access to high quality scholarly resources and long term archiving of those resources. one of the benefits is a cost-effective infrastructure for centrally managing systems that libraries routinely purchase or develop to support the use of scholarly information resources. additionally, it is something much more interesting: a vast collection of diverse resources completely under the control and stewardship of the consortium. scholars portal is perfectly positioned for the development of innovative, integrated services to support scholarly information research. since its inception in 2002, scholars portal has had vendor permissions to locally store, permanently, the vast majority of ejournals purchased by ocul members. these were initially stored on a local installation of science server and then moved to a local mark logic platform as xml-encoded files. with 14 million articles in 8400 journals, this is one of the largest ejournal archives in existence and is currently undergoing accreditation as a trusted digital repository. since 2005, scholars portal has been using a local installation of csa’s illumina product to aggregate many of the abstracting and indexing databases licensed by ocul members as well as the metadata of the e-journal archive. this provides the ability to perform a single search across multiple sources. in 2007, work began on two other significant projects: the development of an e-book platform to host content purchased by or digitized by member libraries and the development of odesi (ontario data documentation, extraction service and infrastructure initiative), a platform to provide access to data sets. the potential provided by local control of these various scholarly resources is exciting. the powerful search opportunities afforded by the full-text content and the ease of linkages between citations and sources can easily be imagined. it is interesting to consider the potential of xml-encoded digital objects beyond their traditional narrative linear form, and which familiar mental models may need to be retained as new innovative features are introduced. the development of scholars portal requires an understanding of the features of the ideal online research environment, from the scholars’ perspective. evidence based library and information practice 2009, 4:2 122 in 2008, the scholars portal team initiated a user study to inform development of this online research environment. working with external consultants in toronto canada (usability matters), the team formulated questions regarding users and their contexts, user research tasks, current and potential features of user interfaces and other relevant technologies and services. a research assistant was hired to explore the research literature and compile relevant observations1.. the largest gap in understanding related to the information research processes of experienced online information researchers. the team therefore decided that this would be an appropriate focus of its own user study. one of the most relevant studies the team considered in its literature review was conducted by the university of minnesota, with support from the andrew w. mellon foundation. the goal of this study was to develop a model for bringing greater coherence to wide ranging aspects of support for scholarship in the humanities and social sciences. in a multi-dimensional framework for academic support: a final report, the authors presented a way to structure the analysis of the data they gathered regarding faculty and graduate students’ research needs and to frame possible future directions. they proposed the categories of ‘discover,’ ‘gather,’ ‘create’ and ‘share’ as phases in the research process, noting that these are not discrete or linear, but rather iterative and overlapping in multi-dimensional ways (university of minnesota 38). this concept is rooted in the notion of ‘scholarly primitives’ presented in 2000 by john unsworth, an internationally renowned scholar and leader in the field of digital humanities. considering the work of humanities scholars, and comparing this to the work of other disciplines, unsworth spoke of scholarly primitives as basic functions common to all scholarly activity and proposed several as a starting point for considering the tasks to be supported by our digital tools: discovering, annotating, comparing, referring, sampling, illustrating, representing (unsworth 1). each of unsworth’s primitives can be considered to be elements of the categories proposed by the university of minnesota. although the project team did not set out to seek a framework to inform development, ‘discover, gather, create, share’ seemed to have the potential to guide scholars portal’s development of services. because the framework resonated strongly with the team, it was decided to explore the value of this particular framework in the study rather than seek alternative approaches. the team recognized that the questions raised prior to the research review, and much of the selected research, was centred around the ‘discover’ phase and more needed to be learned about the other phases. other projects have also seen the value in this framework (for example, it is cited on the project bamboo planning wiki) and it is hoped that the scholars portal study provides useful insights for those considering its application. study objectives and methodology the main purpose of this study was to understand the current information research processes of experienced academic researchers in a variety of disciplines and to gather their ideas for improvement. as part of this, the team wanted to explore whether the framework ‘discover, gather, create, share’ (university of minnesota 38) resonated with the participants. they also wanted to determine what tools and techniques the participants were currently using to aid the information research process. through this study, the team sought to improve and enhance the scholars portal suite of tools in the near-term but, more significantly, to gain insights that evidence based library and information practice 2009, 4:2 123 would inform the future vision of scholars portal. based on the objectives, it was decided that a series of face-to-face workshops would provide the most valuable interaction with and amongst the participants. the first half of the workshop was considered as preparation and followed a traditional format with a facilitator asking questions, to which individuals responded. this format was used to discuss the current information research processes of the participants, to validate the university of minnesota framework and to review a search interface prototype. in the second half of the workshop participants worked together in small groups to envision an idealized information research process. because the design component was key, the workshops were referred to as ‘collaborative design sessions’. the term ‘collaborative design’ can be applied to any situation in which two or more people work together to design anything. collaborative design is frequently used in the field of human-computer interaction (hci), specifically in its subdomains of computer supported cooperative work (cscw) and participatory design (pd), to refer specifically to collaborations between ‘designers’ and ‘end-users’ of computer systems and quite often these collaborations extend over time (kyng 66). in this particular case, participants were asked to collaborate with one another on a single occasion to envision an idealized information research process. study participants to recruit study participants, members of the ocul public services advisory group sent an email to faculty and graduate students at three toronto-area universities. in the email, potential participants were asked to self-identify with one of three broad discipline areas: arts and humanities, social sciences and sciences (natural, applied, health, etc.), as well as one of four experience levels for conducting online information research: none, novice, intermediate and advanced. as the study required participants with experience in online information research, those indicating ‘none’ were excluded from the study. potential participants were asked to identify their role at the university (faculty, grad student, post doc, research librarian, other researcher) as well as their age group, to ensure a reasonable mix of participants. a mixture of participants was chosen to meet a range of criteria with eight to ten participants in each of the six sessions. most of the participants were graduate students; however 9 of 60 participants were faculty members. there was a good mix of age and gender within each session. at the end of the session, participants received a small cash incentive for their participation in one 90 minute session. study procedures the six collaborative design sessions were held at the university of toronto in may 2008. two sessions were conducted in each of the three broad discipline areas (arts and humanities, social sciences and science). as they arrived, participants were asked to complete a ‘warm-up’ questionnaire outlining the steps involved in their research process and three things that would make that process easier. participants were encouraged to use this questionnaire as reference during the first part of the session in which the whole group outlined the steps involved in a ‘typical’ research process. the questionnaires were collected at the end of the session1. during the next part of the session, the minnesota framework was introduced and evidence based library and information practice 2009, 4:2 124 participants were asked, in general, if the steps the group had identified in their research process could be loosely organized by this framework and, if so, which steps would fit into each of the stages (i.e. discover, gather, create and share). a few moments of the workshop were spent reviewing the draft search and search results interfaces to aid development of a new scholars portal search interface. in the second half of each session participants envisioned an ‘ideal’ information research process, concentrating on the ‘gather’, ‘create’ and ‘share’ parts of the process. participants were assigned to three small groups of three or four people each and given flip-chart paper and other materials with which to ‘storyboard’ their ideal research process. each of the small groups presented their outcomes to the large group, answering clarification questions only. finally, the entire group discussed similarities and differences in their approaches, elements that surprised them, and other reflections of interest. overall, the sessions ran smoothly and provided the project team with a clearer understanding of the current research process of experienced online information researchers together with the strengths and weaknesses of the minnesota framework. however, the collaborative design sessions did not yield the anticipated types of ‘storyboards’. despite encouragement to the contrary, most of the groups focused on ‘discover’ and left little time for ‘gather’, ‘create’ and ‘share’. very few groups mapped an ‘ideal’ process, continuing to focus on their current process. nonetheless, there were some interesting insights gained from this design exercise and especially from the sessions as a whole. in hindsight, trying to cover the current and ideal research processes in one session was perhaps overambitious, since participants struggled to shift their focus between the two. the rationale had been that discussing the current approach would enable participants to envision improvements. a future approach could include a facilitator/designer in each small group to help the group focus on the specific task, to elicit more ideas and to create more useful ‘storyboards’. data collection and analysis during each session, notes were taken on a laptop and the session was video-recorded. in addition, the pre-questionnaires and flipchart notes were transcribed, as were the ‘storyboards’ produced by the small groups in each session. the session notes were the main basis for the data analysis, along with discussions between the facilitator and notetaker (both of whom are experienced usability specialists). like most qualitative research, the analysis involved combing the data looking for patterns and themes (creswell 203). working together the consultants (i.e. the facilitator and note-taker) looked for similarities, differences and patterns between individuals, between groups and between the discipline areas. in the report, findings were organized task-by-task, in the order that these tasks were undertaken during the collaborative design sessions. the findings within each task were organized by themes that emerged from the data, for the earlier tasks, and organized by the minnesota framework for the tasks that followed its introduction into the workshop. at the end of the report on each task, the consultants provided analysis and recommendations based on the findings and their expertise in interpreting the findings for the specific context of this organization (ocul) and initiative (the scholars portal suite of tools). evidence based library and information practice 2009, 4:2 125 the raw data (notes, videos, etc.) was provided to ocul and further examination of the data was encouraged to better understand the findings, to reveal additional insights but, perhaps most importantly, to inspire design ideas from the scholars portal and ocul team members. results validating the framework the ‘discover, gather, create, share’ framework was proposed by the university of minnesota in the context of a broad range of research-related activities and services, whereas in this study the focus was somewhat narrower, examining ‘the information research process.’ this focus was made clear to the study participants in the opening warm-up task in which they were asked to ‚list the steps you take for doing information research for academic purposes, from when you recognize that you need information to when you use that information in one or more ways.‛ the framework resonated well with the participants in this context. all groups agreed that it provides a useful, high-level picture of the information research process. as noted in the minnesota report, however, participants emphasized that the process is non-linear, that steps rarely happen in a specific order and that they are often repeated with differing levels of specificity at different stages of the process. in some cases, the terms themselves were problematic. for example, many participants felt that ‘discover’ wasn’t quite the right word in relation to the information research process because it relates to the result they are trying to achieve through their primary research. most groups also believed that there was a step missing between ‘gather’ and ‘create,’ related to engaging with the materials and organizing one’s thoughts. within the definitions used by the university of minnesota, this falls within the category of ‘create,’ but most participants were insistent that ‘synthesize’ is distinct and so it is introduced below. overarching observations several themes emerged relating to all phases of the framework: similarities across disciplines the processes described in each discipline group were remarkably similar. different sources were mentioned in the different groups, and in the science sessions there was emphasis on preparation for and validation of bench research, but this did not reveal fundamental differences in the information research process. interdisciplinarity participants talked about the challenge of interdisciplinarity and the need to easily search across disciplines, but also the need for tools that can help them be selective about the disciplines included in a search and/or the ability to narrow the results to their areas of interest. the social aspect of research several of the groups talked about ‚interaction,‛ ‚collaboration‛ or ‚conversation‛ as part of the framework. ultimately they decided that these are not discrete steps in the process but, rather, are overarching throughout all phases. they were adamant, however, about the importance of this aspect of their information research process. room for improvement most participants believed that they should have a better process and evidence based library and information practice 2009, 4:2 126 would like to improve their approach, but taking the time to learn was not a high priority. in answer to the question of ‚what would make your information research process easier,‛ one participant gave a telling response: ‚user friendly search engine; actually attending some of the different seminars on web research.‛ discover the process participants associated many activities with ‘discover’: talk with colleagues, keep up with the field, attend conferences, observe, read, develop questions, consider one’s own personal knowledge and beliefs on a chosen subject, follow known sources, rediscover things you’ve found previously, search for literature. this phase generated the most discussion in all groups and several themes emerged: web search engines and common internet tools appeared in most researchers’ steps, but these same people relied on research databases provided by libraries. one researcher’s succinct summary is representative: ‚google, wikipedia, jstor, scholars portal, lexisnexis.‛ keywords, colleagues and the citation network are all important approaches. most participants said that they start their search broadly and then narrow it, but there is no set routine. the process could involve, at various points: getting ideas for keywords from colleagues or overview sources, searching keywords, discovering known experts and searching for their publications, finding a literature review and following the references. the ultimate goal of the search effort is a resource that can be downloaded, ideally a pdf document. when shown a prototype interface that provided tabs for tables and figures, a few participants were intrigued by the thought that they could easily access these data elements, but they were puzzled by how these elements would be separated from their original context, which was assumed to be an article or book. keeping up in one’s field is accomplished through a combination of methods, such as getting ideas and resources from listservs, rss feeds and email alerts from key journals or news services. often these sources were discovered by chance and the researchers had no memory of how they signed up for them. no one mentioned receiving alerts from their university library. the ideal participants easily articulated elements of their ideal ‘discover’ process, many of which relate to features common in current systems. the following were given the most emphasis: more electronic resources were mentioned in all of the groups. a ‘one-stop shop’ or single interface to search for all relevant material. some participants expressed a need for narrower search engines, including a narrower scholars portal, and in talking about desired features they often identified ‚relevant results‛ and ‚ability to narrow results.‛ assistance generating keywords, evidence based library and information practice 2009, 4:2 127 synonyms or other related terms, to ensure the search is complete. an interesting discussion arose in one session about including one or more definitions of the search term at the top of the results. this would help researchers less familiar with the subject matter get a quick overview without having to click through and would also help all researchers focus their search by choosing the definition that applies to their current search. several groups wanted the ability to find out the history of a topic, for example through a visual representation like a mind map. expert advice. participants wanted to see recommendations from ‚authorities‛ and they wanted to be able to identify ‚classics in the field.‛ it was interesting to note, however, that in the prototype design, use of the term ‚top journals‛ was very contentious because it was not clear to participants how ‚top‛ was derived. more intelligent refinement of results. participants wanted the ability to say ‚don’t show me this item again‛ in subsequent searches together with an indication in subsequent searches of items already marked or downloaded. they also wanted the ability to start a completely new search within search results, for example a search box entitled ‚search within.‛ easier citation. enabling easy cut and paste of citations was more important to most participants than exporting to citation management software. some participants wanted to display the full citation in each result, with the ability to choose citation style (apa, etc). gather the process activities associated with ‘gather’ included: obtaining materials (downloading, printing, photocopying); weeding and sorting based on a brief review of table of contents, abstract or conclusions (often into ‘yes’, ‘maybe’, ‘no’ categories); filing materials; creating a bibliography; reading and annotating lightly. the process of borrowing from the library was mentioned only in terms of frustrations with missing materials. in terms of filing materials, most participants download pdfs and store them on their hard-drive in a self-styled folder/sub-folder system generally based on topics or author name. in addition to, or instead of pdfs, many participants print the papers they intend to use and physically file them, usually by topic. very few participants are consistently using any bibliographic management tools. many seem to use the bibliographies they produce for individual academic papers as their main organizing method, returning to these bibliographies when working on subsequent papers. most participants create some sort of annotated bibliography/citation list, most often in ms word, and one person mentioned creating a handwritten list. about half the participants were aware of the bibliographic management tool provided by scholars portal, however most of these were either not using it at all or not consistently. of those who had tried it, many said they had abandoned it quite quickly, not willing to make the effort to learn how to use it effectively. a few said they were using other bibliographic management software. when asked if they had created any systems to manage their citations, few seemed to feel they have a system and only one person mentioned creating a database. evidence based library and information practice 2009, 4:2 128 the ideal participants had several comments about the ideal ‘gather’ process: more electronic resources was a constant refrain. easy, successive annotation. annotations develop over time, being lighter at the beginning and more detailed in later stages on specific papers of interest, and this evolution should be retained and evident. it should be possible to annotate pdfs with the equivalent of post-it notes. a few participants said they annotate in pdf and then use the search function later to find their specific, pertinent notes. the ability to annotate directly in pdf was a surprise to many in those sessions and some were very intrigued by the possibilities. ability to display, extract, and easily compare relevant sections of each paper, for example conclusions and methods. synthesize the process activities noted in this phase related to organizing thoughts, a process that participants saw as distinct from ‘gather’ and ‘create.’ they included weeding further, validating the quality of sources, organizing and coding sources thematically, annotating further by hand or directly in the pdf, reading for detail, taking notes and extracting quotes. the latter might entail cutting and pasting text from sources into an email, a document, excel, a table or index cards. participants talked about the intellectual acts of summarizing, looking for patterns, mind-mapping (e.g. with mindmeister <http://www.mindmeister.com/>), digesting, fitting data to one’s purpose, formulating the research question, formulating a thesis sentence, determining the theoretical framework, outlining the paper and creating a bibliography. the ideal although this was a process that participants emphasized as important, few expressed any particular ideas for how it could be improved beyond those covered in ‘gather’ above. create the process in some sessions, the term ‘create’ was closely associated with the participants’ original research and less so with research output such as scholarly papers. however, all participants did easily identify activities related to ‘create’: sweat, clarify audience, outline, write findings and ideas, edit, refine, consider reviews and feedback, revise, discuss, collaborate. almost all participants use msword but other tools are used as well, such as latex. writing in groups was mentioned only briefly by faculty members who are working with their research assistants. in this context, a few concerns about tracking changes and version control were raised. there was little discussion of issues regarding illustrating papers and presentations in the sessions other than the sciences. the ideal participants provided a few ideas for their ideal process: one small group dreamed of a personalized online whiteboard or light table, for organizing materials and, ultimately, the paper. it would include templates, the ability to evidence based library and information practice 2009, 4:2 129 export to powerpoint and the ability to attach references, documents and figures. one participant wanted the opportunity to run papers through ‘turn it in’ in advance, so that adjustments could be made to the paper before making the final academic submission. one group suggested a timeline tool that would provide a schedule, tell you it’s time to take a break and prevent use of email if set to do so. share the process the activities associated with ‘share’ included: share with specific individuals (supervisor, colleagues, experts, authors of the papers you used), publish, submit to online archives (mentioned only in a science group), teach, give presentations and participate in seminars, conferences, symposia. the ideal ideas for the ‘share’ process included: submission process improvements, such as providing more standardized and more online processes. tools for facilitating sharing with colleagues, students, advisors. as well as sharing folders and documents, sharing search strategies and results was suggested. graduate students saw the value in identifying and getting in touch with authors and leading researchers, but said they rarely follow through. one suggestion was a network of researchers to facilitate communication between learners and experts. help with identifying potential publishing venues and conferences. it was noted in one of the sessions that these opportunities arise and should be collected throughout the information research process, including during the ‘discover’ phase. alerts regarding who has cited your article and links to those publications, and alerts to new research in your area. discussion the goal of a portal is to provide unified access to diverse resources and services. for ocul, scholars portal is an entry point for the information research process but its services can also be embedded in the learning and research workflow, in web spaces designed for different groups of users and different purposes at different schools. in developing a portal to support scholarship, the minnesota framework appears useful for envisioning the information research process as a whole. it is not suggested that the research framework be made visible to end-users: experienced online information researchers may quibble over the words chosen and perceive it as a rigid categorization of a very fluid intellectual process. however, given researchers’ strong interest in finding better ways to manage the information research process, scholars portal developers will use the framework to consider how best to promote the availability and interrelation of a set of research tools that goes beyond the traditional portal focus of ‘discover.’ considering the opportunities presented by scholars portal’s data repositories, several aspects of the study results are of particular evidence based library and information practice 2009, 4:2 130 interest in informing portal development and future research; these are highlighted below. the overall information research process is similar across disciplines. it appears that scholars portal should support this common process while providing the flexibility necessary to accommodate differences in resources and tools by discipline. the single search interface, with both excellent precision and recall, would be the ideal outcome. however, as well as searching across all disciplines and types of resources it should be possible to search a userdefined subset of disciplines or resources. the relative importance of different search methods (i.e. browsing, citation chaining and directed searching) is known to vary by discipline and depend on various factors (talja 1675). it seems clear from this study, however, that citation chaining is relied upon heavily by at least some researchers, and that the opportunities to provide linkages between the digital objects in scholars portal’s repositories will be of great benefit. it would be useful to turn any citation within a digital object into a link to the cited source, whether it is a journal article, a data set or some other type of resource. all disciplines also routinely rely on keyword searches, as confirmed in this study and by vakkari and talja, who state that ‚keyword searching in journal and reference databases were clearly the most important access methods in all disciplines compared to browsing, chaining or obtaining material from colleagues‛ (vakkari 1). in many cases, however, researchers are not confident about the vocabulary they are using. participants in this study had some suggestions for features that would help with this, but more exploration is required. in a recent summary of research on end user searching, markey describes different types of user searching difficulties that could cause systems to intervene with vocabulary assistance, and suggests a half dozen research questions to inform such development (markey 1126). the potential of user generated content and social search – a search aided by trusted or expert opinion – was not discussed directly in this study, but the participants’ emphasis on the social aspect of research suggests it is relevant. while they stressed their reliance on colleagues and an interest in being able to identify sources recommended by experts, it was not clear what would make them trust others’ evaluations. this is an area that needs to be more clearly understood in the academic context. one of the opportunities provided by scholars portal’s repositories is the flexibility provided by the xml-encoded digital objects. it seems likely that researchers will take advantage of linkages provided between pieces of data, given current habits of following paths to find and verify information. the ability to manipulate particular pieces of data and use it for other purposes was not suggested by study participants, but there was nothing to indicate that they would not respond positively to those opportunities once presented. one clear message is that the mental model of the downloadable pdf replicating the traditional print object will need to be supported for some time to come. participants in this study uniformly felt they could be doing a better job of managing the sources they used in their research. some aspects of that could be helped with a citation management tool, yet few were interested in using one. this lack of uptake is not just a problem of awareness and the scholars portal team plans to investigate this issue in 2009. in general, there appear to be opportunities for improvement in all phases of the information research process, though the ones that engaged participants the most were ‘discover’ and ‘gather.’ participants appeared very interested in developing evidence based library and information practice 2009, 4:2 131 techniques to improve their currently haphazard approaches, and suggested some avenues to explore. in some cases it may be as simple as providing visibility for existing software, such as pdf annotation tools. in all cases, the challenge will be to provide tools with very low barriers and clear advantages over well-established current practices. conclusion this study was intended to help the scholars portal team better understand the information research process of experienced online information researchers, the tools and techniques they currently employ and their vision for an ideal information research process. it validated the idea of approaching development from a framework of ‘discover, gather, synthesize, create, share’ and provided a variety of useful insights. overall, it was apparent that participants have almost no routine in their processes, have developed few techniques to assist themselves and have very little awareness of the tools available to help them. the collaborative design sessions yielded fewer ideas for the ideal information process than hoped, instead focusing on improvements to current processes. a recommended modification to the methodology would be to include a facilitator or designer in each group to help the group focus and react to ideas and thus create more useful storyboards. the sessions yielded a wealth of observations about the information research process and experienced online information researchers’ needs. based on this study, the scholars portal team has begun designing a new interface for the ejournal repository on the marklogic platform and will be conducting iterative usability testing in early 2009. acknowledgements the authors gratefully acknowledge stacy allison-cassin, alan darnell, kate davis, amy greenberg, kathy scardellato, dana thomas and sarah toy for their contributions to the design and execution of this study, patricia lawton for her research assistance and ocul directors for their support. works cited creswell, john w. research design: qualitative, quantitative, and mixed methods approaches. thousand oaks: sage, 2003. kyng, morten. ‚designing for cooperation: cooperating in design.‛ communications of the acm 34. (1991): 64-73. markey, karen. ‚twenty-five years of enduser searching, part 2: future research directions.‛ journal of the american society for information science and technology 58.8. (2007):1123-30. project bamboo. 10 april 2009 <https://wiki.projectbamboo.org/dis play/bpub/discover,+gather,+creat e,+share++a+research+lifecycle+view>. talja, sanna, pertti vakkari, jenny fry and paul wouters. ‚impact of research cultures on the use of digital library resources.‛ journal of the american society for information science and technology 58.11. (2007): 1674-85. university of minnesota libraries. ‚a multidimensional framework for academic support: a final report submitted to the andrew w. mellon foundation, june 2006‛.10 april 2009 < http://search0.scholarsportal.info.myaccess.library.utoronto.ca/ids70/view_record.php?id=1&recnum=13&log=from_res&sid=02cad1cd4298a0079e64d7317d131b0e http://search0.scholarsportal.info.myaccess.library.utoronto.ca/ids70/view_record.php?id=1&recnum=13&log=from_res&sid=02cad1cd4298a0079e64d7317d131b0e http://search0.scholarsportal.info.myaccess.library.utoronto.ca/ids70/view_record.php?id=1&recnum=13&log=from_res&sid=02cad1cd4298a0079e64d7317d131b0e https://wiki.projectbamboo.org/display/bpub/discover,+gather,+create,+share+-+a+research+lifecycle+view https://wiki.projectbamboo.org/display/bpub/discover,+gather,+create,+share+-+a+research+lifecycle+view https://wiki.projectbamboo.org/display/bpub/discover,+gather,+create,+share+-+a+research+lifecycle+view https://wiki.projectbamboo.org/display/bpub/discover,+gather,+create,+share+-+a+research+lifecycle+view https://wiki.projectbamboo.org/display/bpub/discover,+gather,+create,+share+-+a+research+lifecycle+view evidence based library and information practice 2009, 4:2 132 http://www.lib.umn.edu/about/mellon/ umn_multidimensional_framework_final_report. pdf>. unsworth, john. ‚scholarly primitives: what methods do humanities researchers have in common and how might our tools reflect this?‛ humanities computing, formal methods, experimental practice symposium, kings college, london, 13 may 2000. 10 april 2009 <http://www.iath.virginia.edu/~jmu 2m/kings.5-00/primitives.html>. vakkari, pertti and sanna talja. "searching for electronic journal articles to support academic tasks. a case study of the use of the finnish national electronic library (finelib)." information research 12.1 (2006). 10 april 2009 <http://informationr.net/ir/121/paper285.html>. notes 1 documents related to this user study project, including an annotated bibliography compiled by patricia lawton and the full report by usability matters appended with research instruments and the raw data, are available on the project wiki: http://spotdocs.scholarsportal.info/display/p swg/user+study http://www.iath.virginia.edu/~jmu2m/kings.5-00/primitives.html http://www.iath.virginia.edu/~jmu2m/kings.5-00/primitives.html http://informationr.net/ir/12-1/paper285.html http://informationr.net/ir/12-1/paper285.html http://spotdocs.scholarsportal.info/display/pswg/user+study http://spotdocs.scholarsportal.info/display/pswg/user+study research article   the socioeconomic profile of well-funded public libraries: a regression analysis   michael carlozzi library director wareham free library wareham, massachusetts, united states of america email: carlotsee@gmail.com   received: 19 aug. 2017  accepted: 18 apr. 2018      2018 carlozzi. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29332     abstract   objective – this study aimed to explore the well-established link between public library funding and activity, specifically to what extent socioeconomic factors could explain the correlation.   methods – state-level data from the massachusetts board of library commissioners were analyzed for 280 public libraries using two linear regression models. these public libraries were matched with socioeconomic data for their communities.   results – confirming prior research, a library’s municipal funding correlated strongly with its direct circulation. in terms of library outputs, the municipal funding appeared to represent a library’s staffing and number of annual visitations. for socioeconomic factors, the strongest predictor of a library’s municipal appropriation was its “number of educated residents.” other socioeconomic factors were far less important.   conclusion – although education correlated strongly with library activity, variation within the data suggests that public libraries are idiosyncratic and that their funding is not dictated exclusively by the community’s socioeconomic profile. library administrators and advocates can examine what libraries of similar socioeconomic profiles do to receive additional municipal funding.       introduction   i once noticed staggeringly high circulation numbers coming from a particular public library and pointed it out to a senior library director i knew. the notable library served a population almost identical to my own as well as the director’s, roughly 22,000 residents. yet this library circulated over 173 items per hour open in contrast to my library (64) and his (112). i asked the director why he thought this library circulated such volume.   this was his verbatim email reply: “$$$$$$$$$$$$$$$$$$$$$$$$$$”   the light-hearted response turned out to be well-grounded: all three circulation totals corresponded to our ranking in municipal funding. more generally, the pew research center’s survey data suggest that wealth correlates with library usage (rainie, 2016). these data were corroborated by the institute of museum and library services’ (imls) fiscal year 2011 report, which used statistical modeling to show that in “most cases . . . when investment increases, [library] use increases, and when investment decreases, use decreases” (swan et al., 2014, p. 1). a subsequent imls (2016) report drew similar conclusions, supporting what librarians had long suspected: libraries succeed with financial commitment.   but these analyses cannot determine the extent to which financial investment impacts library usage. imls’s multilevel growth models, for instance, showed that library use corresponded to differences in financial investment. yet financial investment might merely measure the size and scope of a library’s service population; larger libraries receive more funding to support larger communities. financial investment also might just reflect a community’s socioeconomic profile. the pew research center’s surveys consistently find that wealthier and more educated people use libraries more often than those with lower income and education levels (geiger, 2017; rainie, 2016). thus, library funding and usage might both be effects of the community’s overall characteristics.   to try to address these concerns, i analyzed library data from 280 public libraries and confirmed that municipal appropriation strongly correlated with direct circulation. i then included socioeconomic factors for the communities of these libraries to find that the number of a community’s “educated residents” significantly affected a library’s municipal appropriation, far more than any other socioeconomic factor. however, enough variation existed within the data to reject any “demographics are destiny” arguments—library funding and library usage are not necessarily governed by uncontrollable, socioeconomic factors.   literature review   around the turn of the century, library researchers sharpened focus on library-based assessments. dugan and hernon (2002) attribute the change in academic libraries to a shift in priorities as the traditional role of libraries was to “meet the needs of the academic community’s information needs” (p. 377). for example, traditional assessment measures (outputs) concerned operating hours and collection space. given the increase in information literacy demands, however, dugan and hernon argue that traditional outputs could not capture the scope, or even existence of, student learning and were even misaligned with assessments; they argue that traditional outputs belong to an evaluative, not assessment, framework. thus were born library-based outcomes, which focused on the measurable results of library-based participation (e.g., information literacy gain scores on a pre/post-test).   public library outcomes tend to focus not so much on learning as on economics. considerable research has attempted to approximate these economic benefits, with consensus reaching a cost-benefit ratio of around $4 to $1 usd (aabø, 2009; bureau of business research, 2017; howard fleeter & associates, 2016; ward, 2008). similar benefits were found internationally as well (bundy, 2009). of course, such a narrow view of “value” cannot capture all of the public library’s benefits. jaeger et al. (2011) summarize several alternative ways to assess value, and mcmenemy (2007) argues that an explicitly economic focus ignores the public library’s other cultural and societal contributions.   public libraries in the united states report data either directly to the imls’s public libraries survey (pls) or to their state agencies, themselves collectors of data in formats very similar to the pls. the pls collects outputs such as a library’s circulation, visitations, reference transactions, computer usage, collection size, staffing levels, financial expenditures, and operating hours. these outputs only indirectly measure value; as holt and elliott (2003) argue, they “do not represent equal consumption of services or equal value to the library customer” (p. 425). nevertheless, as holt and elliott acknowledge, politicians and stakeholders tend to regard libraries with greater numbers of these outputs as “the best libraries” (p. 425). much library research, then, focuses on these outputs. the imls’s own research analyzes circulation, visitations, staffing, financial expenditures, collection size, computer usage, programming, and reference transactions (imls, 2016; swan et al., 2014). economic analyses of public libraries use the same outputs (e.g., bureau of business research, 2017).   some research has established a strong correlation between a library’s activity, as approximated by the above outputs, and a library’s financial investment (swan et al., 2014). although academic researchers avoid inferring causation from correlation, non-researchers might not be so prudent, as in meyer (2016), who argued from an imls report that “if libraries receive more public funds, more people use them. . . . if the public wants to reverse the [downward usage] trend and make the local library more useful, it should do the one thing evidence supports: fund it better” (para. 12). this is a reasonable inference since financial investment facilitates service. as libraries receive more funding they “can have more staff, more classes, more copies of the latest bestseller, and—maybe most importantly—longer hours” (meyers, 2016, para. 14). mcquillan (2003) drew a similar observation: “more money means more librarians, more books, more magazines, and more open hours” (p. 46).   on the other hand, the theory of public choice, especially tiebout’s model, might posit that library funding reflects community demand rather than causal relationships. developed by charles tiebout (1956), this model imagines “consumer-voters” who choose “the community which best satisfies [their] preference pattern for public goods” (p. 418). the model attempts to explain the economics of public goods by arguing that this “preference pattern” leads to people voting with their feet. while little attention has been given to the theory of public choice in the library literature, bryce (2003) describes the tiebout model as allowing for residents to “decide the kind of community they want to live in” (p. 416). residents who want, for example, excellent library services may vote to raise taxes to support such services. research in massachusetts (e.g., snow, gianakis, & haughton, 2015) shows that this effect occurs at the local level. tiebout’s model reflects population shifting; as public expenditure decisions occur, “populations shift and property prices reflect the public choice of the community” (bryce, 2003, p. 416).   in the tiebout model, then, financial investments do not necessarily boost library outputs. instead, higher outputs reflect the desires and voting patterns of specific communities. residents who disagree with raising taxes to support public libraries will, in theory, oppose such raises or, if they occur, move elsewhere. bryce (2003) studied this subject in the context of public libraries, surveying american adults about their attitudes toward public library services and attempting to connect these responses to library funding through respondents’ zip codes. he found “modest levels of association between demand for library services and library funding support” (p. 422) but largely rejected tiebout’s model. despite this rejection, bryce’s research has been used to make bold claims regarding the theory of public choice; based on bryce’s work, stenstrom and haycock (2015) claim that “the theory of public choice has shown increased use does not correlate to increased funding” (para. 6).   one way to further previous research would be to examine community dynamics directly alongside library activity. the imls’s reports omit “population demographics, poverty, and community characteristics” (swan et al., 2013, p. 13). these characteristics might offer insights on library funding and activity. education level, defined often and in this paper as “the percentage of residents with a bachelor’s degree or higher,” shows particular promise. survey data from the pew research center suggest a connection between education and library usage (rainie, 2016); college graduates were significantly more likely to report using libraries than non-college graduates by a difference of 17 percentage points (geiger, 2017).   political affiliation may also be a useful characteristic, but it shares a complicated relationship with wealth. gelman et al.’s (2007) multilevel analysis in america, for example, shows that “richer states” support liberal candidates while “richer voters” support conservative candidates, i.e. wealthier voters within states, regardless of those states, tend to vote conservatively. what about voters within local communities? brett benson (2012) analyzed and collated the voting patterns of every municipality in massachusetts from 2006 to 2012 and generated an average margin of victory for liberal or conservative candidates. a score of zero means that the community demonstrated no preference for liberal or conservative candidates across 2006 to 2012. positive scores indicate a “more liberal” preference and negative scores a “more conservative” preference. in provincetown, for example, the average score of +73% means that, on average, liberal candidates received 73% more of the vote (not 73% of the vote) over conservative candidates. lynnfield, in contrast, scored -28%, indicating that conservative candidates received 28% more of the vote, on average, over liberal candidates.   data provided by a state-level agency can help further current research lines. entering community data for individual states creates both a manageable dataset and a simplified analysis, as multilevel modeling will not be necessary to control for unique statewide dynamics. community data, then, may validate other measures such as the pew research center’s surveys. because state-level library agencies use the imls’s public libraries survey, intrastate analysis may generalize across at least the united states, if not internationally. as holt and elliott (2003) indicate, states hire “staff whose principal tasks . . . are to collect library input and output statistics” (p. 425). the massachusetts board of library commissioners (mblc) is one such state-level agency. turning to the mblc’s dataset, i asked the following research questions:   1)       to what extent does a library’s funding, specifically its municipal appropriation, account for variation among direct circulation after controlling for library-related variables? 2)       to what extent do these library-related variables explain variation among direct circulation? 3)       to what extent do community variables used as proxies of library usage (income, education level, age, and political affiliation) correlate with library activity and funding?   methods   data collection   to analyze the relationship between financial investment and library outputs, i relied on data from the massachusetts board of library commissioners’ fiscal year 2015 report. every year, the mblc releases an extensive report on all massachusetts public libraries. the data come from annual report information surveys (aris), which library directors must submit to qualify for the statewide certification program. for the mblc’s fy 2015 dataset, 369 separate aris reports were released.   based on the imls’s public libraries survey, the mblc’s dataset includes all of the usual outputs, e.g., circulation, visitations, and operating hours. data include financial information such as the library’s total operating income, its expenditures, and its total appropriated municipal income (tami), which is the amount of municipal funding received. overwhelmingly, massachusetts’ public libraries in fy 15 operated from municipal income, as represented by the tami as a percent of total operating income (median = 91.8%; mean = 86.2%). this mean closely resembled the national average of 85.7% as reported in the imls’s fy 13 report.   to represent the library’s financial variable, i chose municipal appropriation over total operating income for several reasons. first, municipal appropriation contains fewer potential errors; it is the amount of funding that a municipality apportions its library, appearing in public documents as the library’s “line-item” funding. total operating income, by contrast, is more of an estimate, meant to include all of a library’s income as generated from small donations to large bequests and requires consideration of all grants, donations, and miscellaneous funds bestowed during the fiscal year. second, within the mblc’s dataset, operating income did not correlate as strongly as municipal appropriation with direct circulation; operating income’s r = .76 whereas municipal appropriation’s r = .93. third, the appropriation represents a municipality’s financial commitment irrespective of a library’s good fortune, i.e. which libraries have generous individual donors, deep endowments, or vigorous fundraising groups. appropriation ostensibly measures overall community support better than total operating income.   not all data reported by the mblc were used in this analysis. roughly 80% of public libraries in massachusetts serve between 2,000 and 99,999 residents. this analysis examined only these libraries because very small and very large libraries skewed results or bore non-generalizable community dynamics. consider that the average municipal allotment in the entire dataset was $707,882 (median = $368,152) and then consider the boston public library’s municipal allotment ($33,416,127). this astronomically high figure would skew the dataset. furthermore, tiny communities may feature high socioeconomic measures because they are populated by wealthy residents ostensibly uninterested in social services. alford’s population of 474, for instance, has a median household income of $95,313, but with a median age of 57 years, alford does not represent a typical community. i removed some other libraries from the original dataset because they were presented as independent libraries in a larger municipality. i also removed one municipality, a college town, for its abnormally low median age. the final number of public libraries (n) was 280.   models   i built two linear regression models to analyze the impacts of (1) library outputs on direct circulation and (2) community variables on municipal funding. regression models are presented alongside their coefficient of determination (r²) and standard error of the estimate. r² refers to the amount of variation within the data explained by the model. all reported r² values are the adjusted figures so as to minimize the impact of adding variables. the standard error of the estimate refers to the average amount a model’s predictions are “off,” or the average distance from an actual value to its estimated value on the regression line.   selecting independent variables for linear regression model 1 (dependent variable = direct circulation) required some consideration. i could not select variables based solely on the strength of correlation because virtually all library outputs correlated strongly with direct circulation (pearson’s zero-order correlations). this was largely because of confounding variables and collinearity. for example, director’s salary correlated with circulation (r = .63) despite having no logical connection to it. when controlling for municipal allotment, i.e. adding it into the model, director’s salary becomes nonsignificant (p = .47), and its partial correlation—so named because the impact of municipal appropriation is “partialled out”—becomes .001.   collinearity refers to the correlation between predictors in a model, not between predictors and dependent variables. with high collinearity between variables, the contribution of each variable becomes unclear. one way to measure collinearity is the variance inflation factor (vif), which estimates the increase in a coefficient’s variance from collinearity, where a vif value of one means “no collinearity.” some collinearity, especially with observational data, is unavoidable. but how much is too much? convention suggests that vif values up to five indicate a small-modest level of collinearity but higher values are more problematic (stine, 1995). given the nature of these data, however, modest-high collinearity is unavoidable; an increase in one measure tends to indicate an increase in another. this makes sense. as libraries receive more funding they add more staff, field more reference questions, circulate more items, pay their directors higher wages—essentially, they do more of everything, as both meyer (2016) and mcquillan (2003) noticed.   i selected variables, then, which were used by the imls and other researchers, were logically linked with circulation, and which had low collinearity. these variables represented activities that might realistically affect circulation. the final list of variables for model 1, which met the above criteria, included programs offered (adult and children, annually), total visitors (annually), staff hours (total annually), and physical holdings (total). i did not include electronic holdings since, in massachusetts, these are often managed at the consortium level.   despite having a logical connection to circulation and being included in previous research, operating hours were excluded from this model because of their non-linear relationship to circulation. the mblc awards state aid partially in proportion to the number of hours opened, but state aid is capped. for example, libraries with service populations between 15,000 and 24,999 must open 50 hours per week for maximum state aid, with additional hours yielding no more aid. libraries lack financial incentive, then, to open more hours than this threshold as suggested by figure 1.   linear regression model 2 examined the impact of community characteristics on municipal appropriation (dependent variable), following swan et al.’s (2014) suggestion that “more could be learned by incorporating other contextual data, such as information on poverty and community characteristics” (p. 13). i added data on these community characteristics based on the latest available census data, either the 2010 u.s. census or the 2011 or later american community survey (acs), from the american fact finder online. age is represented by the community’s median age. population is the latest available estimate from the acs. i estimated political affiliation using benson’s (2012) dataset on municipal massachusetts’ voting trends. i chose median family income over median household income because they measured essentially the same construct but median family income correlated better with both municipal allotment and direct circulation; per capita income correlated poorly with both measures.       figure 1 total operating hours on direct circulation. note the “wall” created as most libraries reach the threshold to receive the maximum amount of state aid.     education level requires some explanation. education level (percentage of residents with a bachelor’s degree or higher) and population shared an interaction effect. a model of just population and education level yielded an r² of .60, with moderate partial correlations to municipal funding (population r = .77 and education r = .32). i suspected, however, that population interacted with education, i.e. gains from population differed depending on education levels. i first centered these two variables around their means and then subtracted the mean from each value to avoid complications from collinearity (afshartous & preston, 2011). i then multiplied population by education level to create the interaction term. with the interaction term in the model, substantially more variance was explained (r² = .82). to simplify model 2, i measured education level by generating a statistic called the “number of educated residents,” calculated by multiplying a community’s estimated population by its estimated educational attainment (percentage of residents with a bachelor’s degree or higher). this statistic alone explained almost as much variance as the above model (r² = 0.80), and i used it for model simplicity.   results   as previous research had suggested might happen, municipal appropriation strongly correlated with direct circulation (r = .93), by far the strongest individual effect of any variable. table 1 presents the results of model 1: library outputs (total visitors, physical holdings, staff hours, number of total programs offered) on direct circulation. table 2 presents a correlation matrix.   this model explained a considerable amount of variance (r² = .87) with a modest standard error of the estimate (69,066). visitors, staff hours, and holdings were all significant predictors. programs offered was the only nonsignificant predictor on circulation (p = .13). it is possible, however, that the effect of programming is so slight that a larger sample size would be required to detect significance. this make sense, as a library’s programs reasonably cannot be expected to influence circulation as much as, say, the number of visitors.   the largest effect on direct circulation was the number of staff hours worked (partial r = .41). the total number of annual visitors came close (partial r = .37). municipal appropriation and total staff hours correlate extremely well and have high collinearity (r = .97; vif = 15.6), suggesting that they measure a similar construct, although when in the same model, municipal appropriation retains a higher partial correlation (r = .48) than staffing (r = .12). that may be because staff hours have an empirical limit whereas appropriation does not; even very large libraries eventually reach a critical mass of staff members.     table 1 output variables on direct circulation   unstandardized b p value 95% confidence interval partial correlation constant -45860 <.01 -62434 – -29286 - visitors .53 <.01 .36 – .71 .37 holdings .28 .03 .03 – .53 .14 programs 35.35 .13 -10.94 – 81.65 .10 staff hours 279.67 <.01 205.88 – 353.46 .44 m = 176,544. n = 236. some libraries were removed for not having submitted data for all included variables.     table 2 correlation matrix of output variables and direct circulation   circulation staff hours programs holdings visitors circulation 1.0 .92 .67 .83 .89 staff hours .92 1.0 .69 .87 .89 programs .67 .69 1.0 .59 .64 holdings .83 .87 .59 1.0 .80 visitors .89 .89 .64 .80 1.0     table 3 socioeconomic variables on a library’s municipal appropriation   unstandardized b p value 95% confidence interval partial correlation constant -23098.23 .23 -607048 – 145093 - family income .44 .52 -.90 – 1.77 .04 education 73.15 <.01 67.72 – 78.57 .85 political 2111.38 .03 213.52 – 4009.24 .13 age 7446.46 .01 765 – 15658 .15 m = $700,428. n = 280.     table 4 correlation matrix of socioeconomic variable and municipal appropriation   tami education family income age political tami 1.0 .89 .23 -.30 .28 education .89 1.0 .26 -.39 .25 family income .23 .26 1.0 .01 -.26 age -.30 -.39 .01 1.0 -.08 political .28 .25 -.26 -.08 1.0       table 3 presents results from model 2, and table 4 presents a correlation matrix on the effects of community dynamics on municipal appropriation. this model explained considerable variance (r² = .85) but contained a relatively high standard error of the estimate ($259,768). the number of educated residents had the strongest impact by far (partial r = .85); for every additional “educated resident,” the model predicted a $73.15 increase in municipal appropriation. the 95% confidence interval was also fairly narrow, ranging from $67.72 to $78.57.   as with population, i suspected that age might have interacted with education level. without the interaction effect, age was negatively correlated with appropriation (r = -.30), suggesting that older communities were not as generous as younger ones. (the effect was nonsignificant with other variables in the model, however.) but with the interaction effect in the model, age retained a significant and positive effect (partial r = .15). this measure was not precise, however, with a very wide 95% ci. income level was insignificant (p = .52) after controlling for education.   political affiliation was also a significant (p = .03) but with a very wide 95% ci. it did not have a clear interaction effect with education or any other variable. such imprecision might suggest problems with the dataset. although benson’s (2012) dataset was extensive, it was not necessarily rigorous; it simply averaged margins of victory across several elections. this might not be a valid way to approximate voting patterns.   discussion   previous research has demonstrated a strong correlation between funding and library activity, at least as measured through the variables of circulation and annual visitations. as swan et al. (2013) found, “[library] revenue was a positive predictor for visitation, circulation, and program attendance” (p. 13). drawing on the mblc’s data, i analyzed library usage statistics, extending previous research by including community characteristics. this analysis aimed to learn what municipal allotment might actually measure, for example, a community’s income or education level.   in terms of library outputs, direct circulation strongly correlated with both staffing and visitations. other variables previously studied by the imls (e.g., reference transactions and programs offered) indicated little to no correlation after controlling for municipal appropriation or other variables. but this insight, unfortunately, lacks utility. the high vif (15.6) between staffing and municipal allotment suggests that they may measure the same construct. advising library administrators to add more staff provides neither clarity nor guidance. we can reasonably infer that libraries hire more staff in reaction to financial increases, something already well known. and, like staffing, visitations are uninformative. we are interested in why people visit libraries not that they do. obviously, visitations correlate with circulation totals—as more people visit libraries, more materials circulate.   as the strongest effect on a library’s activity was its municipal appropriation, it makes sense to determine what affects this appropriation. this analysis suggests that a library’s municipal allotment stems largely from its community’s education level; about 80% of the data’s variation could be explained by the number of a community’s educated residents alone, even after controlling for other influences. model 2 predicted that each additional educated resident might be expected to increase library funding by about $73 while holding other variables constant. interestingly, median family income was found to be nonsignificant when controlling for education level. this may relate to the fact that the examined state was massachusetts, which is historically the highest-ranking state in terms of educational attainment (ogunwole et al., 2012). older or liberal communities were also more likely to receive library funding. these effects were slight, however, and, at least in the case of age, related to education level. political affiliation may also interact with education level, but this analysis may not have been able to pick it up due to methodological issues (e.g., sample size and limitations of benson’s dataset).   that education influences municipal allotment so strongly suggests that municipal allotment reflects the community’s demand for library services, lending indirect and admittedly strictly correlative support for the theory of public choice. had an income measure been the dominant influence instead of education level, then another explanation may have been more plausible, i.e. public libraries simply benefit from the largesse of their communities. yet, when controlling for education, median family income did not predict direct circulation. even without controlling for education, income was a relatively weak predictor (r = .23). many wealthy communities appeared to fund their libraries (relatively) poorly and vice versa. simply put, the more educated people in a community (in this dataset at least), the higher its public library’s funding tended to be, corroborating survey data from the pew research center (geiger, 2017; rainie, 2016).   limitations and future research   it should be noted that this analysis relied exclusively on data from one northeastern, highly educated state. as swan et al. (2013) indicated, interstate analyses should use multilevel models to consider dynamics unique to each state. such dynamics may affect the generalizability of these findings. other researchers could apply socioeconomic analysis to other states and countries. furthermore, this research analyzed correlations and thus cannot establish causation. while the data suggest that educated communities drive library funding, this conclusion cannot be drawn and further research would have to examine its feasibility. previous research by bryce (2003) found a lack of support for the theory of public choice in public libraries, although bryce labels his findings as “too preliminary in nature” (p. 423). to further this research line, one might be interested in examining within-subject funding and circulation levels across several years.   furthermore, the seemingly high r² values in these models obscure the correspondingly high standard errors of the estimate. just because two values correlate does not mean that individual predictions based on the regression line will be accurate. this is a well-documented shortcoming of r²; hahn (1973), for example, noted that “unlike the standard error of the estimate . . . r² alone does not provide direct information as to how well the regression equation can be used for prediction” (p. 611). indeed, when the socioeconomic regression model predicted municipal appropriation, the average estimate was off by $259,768. that is a very high standard error considering that the average value in this dataset was $700,428. circulation values similarly had high standard errors of the estimate; in the model of only library outputs, the error was 69,066. of course, these are average values—some estimates were way off and others were almost perfect—but given that the average circulation total was 176,544, this error comes across as quite high.   however, these high standard errors may matter only insofar as we interpret the data continuously, when perhaps it should be understood as ordinal, similar to a likert scale. in continuous data, all unit increases are treated equally, justifying the calculation of an average. but this approach may be inappropriate here. to illustrate this concern, consider a public library in massachusetts with a service population of 23,000 residents. a funding increase from $200,000 to $400,000 would essentially create a viable public library; $200,000 cannot satisfy statewide certification requirements for a service population of that size. an increase from $400,000 to $600,000, while improving services, would not have the same level of impact as the initial increase from $200,000. and an increase from $1,700,000 to $1,900,000 means even less, given diminishing returns. the high standard errors of the estimate may be deceptive; perhaps what matters is that libraries hit a certain threshold of funding and any variation above that level matters less than variation below that level. therefore, libraries may be better understood as belonging to certain categories. for example, the difference between $676,076 and $2,127,001 is certainly numerically large, but the former library can likely deliver an effective level of public service in a way that even a $400,000 library might not. further research could explore this relationship in detail.   nevertheless, all of the data’s variation demonstrates the idiosyncrasies of public libraries. in spite of the strong correlations found here, these regression models leave considerable “wiggle room” for librarians, administration, and advocates to impact their communities. regarding municipal appropriation, community characteristics could not explain almost 15% of the variance—and that 15% appears significant. swan et al. (2013) reached similar conclusions when arguing that “although revenue is an important piece of the puzzle, it is by no means the only investment that explains changes in library use” (p. 13). these data reaffirm their claim. poorly funded libraries may try comparing their own communities to communities of similar educational levels and reach out to those libraries to understand how they develop, promote, and deliver services. for instance, two libraries in this dataset have an almost identical number of educated residents (16,453 to 16,936) yet extremely divergent municipal appropriations ($676,076 to $2,127,001). the poorer library could try to discover any notable systemic differences (e.g., a form of government), and if the poorer library finds nothing substantive, it could contact the wealthier library to try to understand its good fortune and perhaps implement some of the wealthier library’s services or approaches.   conclusion   municipal allotment appears to operate as a sort of proxy variable, i.e. a variable that approximates some real phenomenon such as a community’s interest in its library. this proxy variable is likely the result of many idiosyncratic factors, but the strongest factor was the number of a community’s educated residents. more educated communities were more likely to have greater municipal allotments and, in turn, to circulate more materials. however, library advocates should take heart knowing that enough variation existed within the data to allow libraries an opportunity to escape any “demographics are destiny” conclusions. financial investment appears to be just one part of a large, mysterious puzzle.   references   aabø, s. (2009). libraries and return on investment (roi): a meta-analysis. new library world, 110 (7/8), 311–324.   afshartous, d., & preston, r. a. (2011). key results of interaction models with centering. journal of statistics education, 19(3). https://dx.doi.org/10.1080/10691898.2011.11889620    benson, b. (2012, november 15). how demographic or republican is my town? a partisan ranking of ma municipalities from p-town to lynnfield. [web log post]. retrieved from http://massnumbers.blogspot.com/2012/11/how-democratic-or-republic-is-my-town.html          bryce, a. (2003). public opinion and the funding of public libraries. library trends, 51(3), 414–423.   bundy, a. (2009). public libraries: it’s their funding, stupid. australian public libraries and information services, 22(3), 95–96.   bureau of business research ic2 institute & university of texas at austin. (2017). texas public libraries: economic benefits and return on investment. retrieved from the texas state library website at https://www.tsl.texas.gov/sites/default/files/public/tslac/pubs/roi_final.pdf   dugan, r. e., & hernon, p. (2002). outcomes assessment: not synonymous with inputs and outputs. the journal of academic librarianship, 28(6), 376–380. https://dx.doi.org/10.1016/s0099-1333(02)00339-7   geiger, a. (2017, june). millennials are the most likely generation of americans to use public libraries. pew research center. retrieved from http://www.pewresearch.org/fact-tank/2017/06/21/millennials-are-the-most-likely-generation-of-americans-to-use-public-libraries/   gelman, a., shor, b., bafumi, j., & park, d. (2008). rich state, poor state, red state, blue state: what’s the matter with connecticut? quarterly journal of political science, 2, 345–367. https://dx.doi.org/10.1561/100.00006026   hahn, g. j. (1973). the coefficient of determination exposed! chemical technology, 3(10), 609–612.   holt, g. e., & elliott, d. (2003). measuring outcomes: applying cost-benefit analysis to middle-sized and smaller public libraries. library trends, 51(3), 424–440.   howard fleeter & associates. (2016). the return on investment of ohio’s public libraries & a comparison with other states. columbus, oh: ohio library council. retrieved from http://olc.org/wp-content/uploads/documents/post-id_2060/2016/04/ohio-public-libraries-roi-report.pdf   institute of museum and library services. (2016). public libraries in the united states survey: fiscal year 2013. retrieved from https://www.imls.gov/sites/default/files/publications/documents/plsfy2013.pdf   jaeger, p. t., bertot, j. c., kodama, c. m., katz, s. m., & decoster, e. j. (2011). describing and measuring the value of public libraries: the growth of the internet and the evolution of library value. first monday: peer reviewed journal on the internet, 16(11).  https://dx.doi.org/10.5210/fm.v16i11.3765    mcmenemy, d. (2007). what is the true value of a public library? library review, 56(4), 273–277. https://dx.doi.org/10.1108/00242530710743471   mcquillan, j. (2003). more money, more librarians, more reading: evidence on funding public libraries. knowledge quest, 31(3), 46.   meyer, r. (2016, april). fewer americans are visiting local libraries—and technology isn’t to blame. the atlantic. retrieved from http://www.theatlantic.com/technology/archive/2016/04/americans-like-their-libraries-but-they-use-them-less-and-less-pew/477336   ogunwole, s. u., drewery, jr., m. p., & rios-vargas, m. (2012). the population with a bachelor’s degree or higher by race and hispanic origin: 2006–2010 (acsbr/10-19). american community survey briefs. retrieved from https://www.census.gov/prod/2012pubs/acsbr10-19.pdf   rainie, l. (2016, april). library users and learning. pew research center. retrieved from http://www.pewinternet.org/2016/04/07/library-users-and-learning   snow, d., gianakis, g., & haughton, j. (2015). the politics of local government stabilization funds. public administration review, 75(2), 304–314.  https://dx.doi.org/10.1111/puar.12317   stenstrom, c., & haycock, k. (2015, september). public library advocacy: an evidence-based perspective on sustainable funding. public libraries online. retrieved from http://publiclibrariesonline.org/2015/09/public-library-advocacy-an-evidence-based-perspective-on-sustainable-funding/     stine, r. a. (1995). graphical interpretation of variance inflation factors. the american statistician, 49(1), 53–56. https://dx.doi.org/10.1080/00031305.1995.10476113   swan, d. w., grimes, j., owens, t., miller, k., arroyo, j., craig, t., dorinski, s., freeman, m., isaac, n., . . . p.& scotto, j. (2014). public libraries in the united states survey: fiscal year 2011 (imls-2014-pls-01). washington, dc: institute of museum and library services. retrieved from https://www.imls.gov/sites/default/files/publications/documents/pls2011.pdf   tiebout, c. m. (1956). a pure theory of local expenditures. the journal of political economy, 64(5), 416–424.     evidence based library and information practice evidence based library and information practice 2009, 4.4  evidence based library and information practice         evidence summary    canadian public library users are unaware of their information literacy deficiencies as  related to internet use and public libraries are challenged to address these needs    a review of:   julien, heidi and cameron hoffman. “information literacy training in canada’s public libraries.”  library quarterly 78.1 (2008): 19‐41.    reviewed by:  martha ingrid preddie  doctoral student, school of information and library science  university of north carolina at chapel hill  chapel hill, north carolina, u.s.a.  email: mipreddie@gmail.com    received: 01 september 2009        accepted: 03 november 2009      © 2009 preddie. this is an open access article distributed under the terms of the creative commons attribution  license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and  reproduction in any medium, provided the original work is properly cited.      objective – to examine the role of canada’s  public libraries in information literacy skills  training, and to ascertain the perspectives of  public library internet users with regard to  their experiences of information literacy.     design – qualitative research using semi‐ structured interviews and observations.     setting – five public libraries in canada.     subjects – twenty‐eight public library staff  members and twenty‐five customers.     methods – this study constituted the second  phase of a detailed examination of information  literacy (il) training in canadian public  libraries. five public libraries located  throughout canada were selected for  participation. these comprised a large central     branch of a public library located in a town  with a population of approximately two  million, a main branch of a public library in an  urban city of about one million people, a  public library in a town with a population of  about 75,000, a library in a town of 900 people  and a public library located in the community  center of a canadian first nations reserve that  housed a population of less than 100 persons.  after notifying customers via signage posted  in the vicinity of computers and internet  access areas, the researchers observed each  patron as they accessed the internet via library  computers. observations focused on the  general physical environment of the internet  access stations, customer activities and use of  the internet, as well as the nature and degree  of customer interactions with each other and  with staff. photographs were also taken and  observations were recorded via field notes.  58 mailto:mipreddie@gmail.com http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2009, 4.4  the former were analyzed via qualitative  content analysis while quantitative analysis  was applied to the observations.     additionally, each observed participant was  interviewed immediately following internet  use. interview questions focused on a range of  issues including the reasons why customers  used the internet in public libraries,  customers’ perceptions about their level of  information literacy and their feelings with  regard to being information literate, the nature  of their exposure to il training, the benefits  they derived from such training, and their  desire for further training. public service  librarians and other staff were also  interviewed in a similar manner. these  questions sought to ascertain staff views on  the role of the public library with regard to il  training; perceptions of the need for and  expected outcomes of such training; as well as  the current situation pertinent to the provision  of il skills training in their respective libraries  in terms of staff competencies, resource  allocation, and the forms of training and  evaluation. interviews were recorded and  transcribed. data were interpreted via  qualitative content analysis through the use of  nvivo software.     main results – men were more frequent users  of public library computers than women,  outnumbering them by a ratio ranging from  2:1 to 3.4:1. customers appeared to be mostly  under the age of 30 and of diverse ethnicities.  the average income of interviewed customers  was less than the canadian average.     the site observations revealed that customers  were seen using the internet mainly for the  purposes of communication (e.g., e‐mail,  instant messaging, online dating services).  such use was observed 78 times in four of the  libraries. entertainment accounted for 43  observations in all five sites and comprised  activities such as online games, music videos,  and movie listings. twenty‐eight observations  involved business/financial uses (e.g., online  shopping, exploration of investment sites,  online banking). the use of search engines (25  observations), news information (23), foreign  language and forum websites (21), and word  processing were less frequently observed.   notably, there were only 20 observed library‐ specific uses (e.g., searching online catalogues,  online database and library websites).  customers reported that they used the  internet mainly for general web searching and  for e‐mail.     it was also observed that in general the  physical environment was not conducive to  computer use due to uncomfortable or absent  seating and a lack of privacy. additionally,  only two sites had areas specifically  designated for il instruction.     of the 25 respondents, 19 reported at least five  years experience with the internet, 9 of whom  cited experience of 10 years or more. self‐ reported confidence with the internet was  high: 16 individuals claimed to be very  confident, 7 somewhat confident, and only 2  lacking in confidence. there was a weak  positive correlation between years of use and  individuals’ reported levels of confidence.   customers reported interest in improving  computer literacy (e.g., keyboarding ability)  and il skills (ability to use more sources of  information). some expressed a desire “to  improve certain personal attitudes” (30), such  as patience when conducting internet  searches. when presented with the  association of college and research libraries’  definition of il, 13 (52%) of those interviewed  claimed to be information literate, 8 were  ambivalent, and 4 admitted to being  information illiterate. those who professed to  be information literate had no particular  feeling about this state of being, however 10  interviewees admitted feeling positive about  being able to use the internet to retrieve  information. most of those interviewed (15)  disagreed that a paucity of il skills is a  deterrent to “accessing online information  efficiently and effectively” (30). eleven  reported development of information skills  through self teaching, while 8 cited secondary  schools or tertiary educational institutions.  however, such training was more in terms of  computer technology education than il.  59 evidence based library and information practice 2009, 4.4  eleven of the participants expressed a desire  for additional il training, 5 of whom indicated  a preference for the public library to supply  such training. customers identified face‐to‐ face, rather than online, as the ideal training  format. four interviewees identified time as  the main barrier to internet use and online  access.   as regards library staff, 22 (78.6%) of those  interviewed posited il training as an  important role for public libraries. many  stated that customers had been asking for  formal il sessions with interest in training  related to use of the catalogue, databases, and  productivity software, as well as searching the  web. two roles were identified in the context  of the public librarian as a provider of il:  “library staff as teachers/agents of  empowerment and library staff as ‘public  parents’” (32). the former was defined as  supporting independent, lifelong learning  through the provision of il skills, and the  latter encompassing assistance, guidance,  problem solving, and filtering of unsuitable  content.   staff identified challenges to il training as  societal challenges (e.g., need for customers to  be able to evaluate information provided by  the media, the public library’s role in reducing  the digital divide), institutional (e.g.,  marketing of il programs, staff constraints,  lack of budget for il training), infrastructural  (e.g., limited space, poor internet access in  library buildings) and pedagogical challenges,  such as differing views pertinent to the  philosophy of il, as well as the low levels of il  training to which canadian students at all  levels had been previously exposed.   despite these challenges library staff  acknowledged positive outcomes resulting  from il training in terms of customers  achieving a higher level of computer literacy,  becoming more skillful at searching, and being  able to use a variety of information sources.  affective benefits were also apparent such as  increased independence and willingness to  learn. library staff also identified life  expanding outcomes, such as the use of il  skills to procure employment.  in contrast to customer self‐perception, library  staff expressed that customers’ il skills were  low, and that this resulted in their avoidance  of “higher‐level online research” and the  inability to “determine appropriate  information sources” (36). several librarians  highlighted customers’ incapacity to perform  simple activities such as opening an email  account. library staff also alluded to  customer’s reluctance to ask them for help.  libraries in the study offered a wide range of  training. all provided informal, personalized  training as needed. formal il sessions on  searching the catalogue, online searching, and  basic computer skills were conducted by the  three bigger libraries. a mix of librarians and  paraprofessional staff provided the training in  these libraries. however, due to a lack of  professional staff, the two smaller libraries  offered periodic workshops facilitated by  regional librarians.   all the libraries lacked a defined training  budget. nonetheless, the largest urban library  was well‐positioned to offer il training as it  had a training coordinator, a training of  trainers program, as well as technologically‐ equipped training spaces. the other libraries  in this study provided no training of trainers  programs and varied in terms of the adequacy  of spaces allocated for the purpose of training.  the libraries also varied in terms of the  importance placed on the evaluation of il  training. at the largest library evaluation  forms were used to improve training  initiatives, while at the small town library  “evaluations were done anecdotally” (38).     conclusion – while internet access is  available and utilized by a wide cross section  of the population, il skills are being  developed informally and not through formal  training offered by public libraries. canadian  public libraries need to work to improve  information literacy skills by offering and  promoting formal il training programs.           60 evidence based library and information practice 2009, 4.4  commentary     this research was critically appraised using  the eblip critical appraisal checklist  developed by lindsay glynn. the study  investigated the role of canada’s public  libraries in information literacy training and  the views of public library internet users about  their information literacy experiences. ethics  approval was obtained for the study and  written consent procured for interviews.  confidentiality was addressed by posting  signs informing customers about observations  and photography, and anonymity was  guaranteed for all participants.     the mixed‐methods approach encompassing  semi‐structured interviews and observations  and was suitable for the research objectives  since it afforded deeper insights into il  training practices, as well as the perspectives  of both customers and staff, than would have  been possible with the use of a single method.  the five participating libraries were chosen  based on their interest in participating in the  study. this resulted in a convenience sample.  this was also the case with the customers who  participated in the study. consequently, while  those studied represented a cross‐section of  canada’s public libraries and population, the  possibility exists that had other libraries and  persons participated the results could have  been different, particularly with regard to  customers’ information literacy experiences.  additionally, the number of customers  interviewed and observed (25) was small. this  limits the generalizability of the study’s  findings, especially when some findings  reported the views of less than 50% of the  respondents.     based on the number of reported incidences of  various internet use activities, and given the  fact that only 25 customers were observed, it is  apparent that library users tended to engage  in several activities once they accessed the  internet. one of the shortcomings of this study  is that it failed to state whether or not an  individual’s engagement in multiple  occurrences of the same activity interspersed  with another activity was recorded more than  one time, and if this was consistently applied  in all libraries. this lack of clarity suggests that  observed activities could have been over‐ or  under‐reported.     despite these limitations, very detailed  questions were posed to customers and library  staff. findings pertinent to customers were  triangulated with that of library staff, and the  outcomes were clearly stated and discussed in  relation to the data collected. the authors  admitted that the outcomes of il training as  expressed by library staff were based on the  views of staff and not on any objective tests or  measures. this limitation raises questions  about the validity of those findings.  nevertheless, this is balanced by the similarity  of some of the findings of the first phase of  this study conducted by julien and breu. in  general, however, the study conclusions  accurately reflect the analysis.     no suggestions were proffered for areas of  further research. nonetheless, the findings  that only one participant stated that “the  possibility of asking staff for help was a  significant reason for going to a public library  to access the internet” (29), and that “none of  the customers interviewed in this study  mentioned seeking the help of library staff”  (39) warrant further investigation. such  investigation would be useful in terms of not  only informing il training initiatives, but also  marketing of such training and approaches to  customer service improvements.     this study is significant since it highlights the  dichotomy between public library users’  mostly favourable perception of their state of  information literacy and the opposing views of  library staff. the results suggest that library  users are unaware of what constitutes  information literacy, and libraries have failed  to nurture sufficient levels of il skills among  their customers. in terms of practice  implications this situation emphasizes the  need for canadian public libraries to increase  information literacy skills by offering and  promoting formal il training programs. this  calls for distinct budgets as well as  improvements in the availability of training  61 evidence based library and information practice 2009, 4.4  62 spaces, appropriate technological  infrastructure, provision for training of  trainers, and the adoption of a unified il  training philosophy within the public library  system. moreover, public libraries need to  lobby decision makers to acquire the requisite  support.         works cited     glynn, lindsay. “a critical appraisal tool for  library and information research.”  library hi tech 24.3 (2006): 387‐99.     julien, heidi and reegan breu. “instructional  practices in canadian public  libraries.” library & information  science research 27.3 (2005): 281‐301.  research article   studying the night shift: a multi-method analysis of overnight academic library users   david schwieder political science liaison and coordinator of humanities and social science data services the george a. smathers libraries university of florida gainesville, florida, united states of america email: dschwieder@uflib.ufl.edu   laura i. spears assessment librarian the george a. smathers libraries university of florida gainesville, florida, united states of america email: laura.spears@ufl.edu   received: 27 feb. 2017    accepted: 14 june 2017        2017 schwieder and spears. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – this paper reports on a study which assessed the preferences and behaviors of overnight library users at a major state university. the findings were used to guide the design and improvement of overnight library resources and services, and the selection of a future overnight library site.   methods – a multi-method design used descriptive and correlational statistics to analyze data produced by a multi-sample survey of overnight library users. these statistical methods included rankings, percentages, and multiple regression.   results – results showed a strong consistency across statistical methods and samples. overnight library users consistently prioritized facilities like power outlets for electronic devices, and group and quiet study spaces, and placed far less emphasis on assistance from library staff.   conclusions – by employing more advanced statistical and sampling procedures than had been found in previous research, this paper strengthens the validity of findings on overnight user preferences and behaviors. the multi-method research design can also serve to guide future work in this area.       introduction   as academic libraries have increasingly prioritized services, they have sought new constituencies and new ways to assist them. one popular initiative has involved extending library hours, most commonly to cover a 24-hour schedule during weekdays. but while such programs represent a major effort, and a major commitment of institutional resources, the scholarly literature on this topic remains small, consisting primarily of descriptive case studies.   this paper extends this body of work in two ways. first, focusing on the key topic of user activities during overnight hours, we conduct a multi-method case study. our survey-based study goes beyond previous studies—which have been wholly descriptive—by using both descriptive and correlational analyses to explore overnight user attitudes and behavior. the results of these analyses “triangulate” to show a consistent pattern; overnight users prioritize study spaces and resources to support study, like power outlets. more traditional library resources and services, i.e. library materials and assistance from library staff, are seen as less important.   second, our approach also provides robust grounds for generalizing our findings. survey respondents were recruited at two different campus libraries, as well as through the media, with each of these three groups of respondents comprising a separate survey sample. results for all three survey samples mirrored the overall results described above. this consistency across multiple samples provides an enhanced basis for generalization; that is, for assuming that the results from our survey accurately represent the overall user population at our university. in sum, then, our approach—a multi-method study conducted across multiple survey samples—extends previous studies, and yields particularly well-founded conclusions about the preferences and behaviors of overnight users. these conclusions then can support effective library decision making and policies.   literature review   over the last several decades, the proportion of american academic libraries offering extended hours has increased significantly (sanders & hodges, 2014). while a few extended hours programs existed as early as the 1980s (bowman, 2013; smith, 2008), overnight hours were uncommon. a 2002 study found that 5 out of 97 association of research libraries (arl) member libraries featured some kind of regular 24 hour access during the week; 2 others offered extended hours access to stacks and circulation services (arant & benefiel, 2002). an association of college & research libraries weblog used data from the 2004 academic libraries survey to calculate that only 24 of roughly 3,700 u.s. academic libraries reported keeping a 24/7 schedule at that time (acrl, 2006). by 2011, a survey of arl library deans and directors found that 71% reported holding either 24/5 or 24/7 hours in some part of their library space (laaker, 2011). thus extended hours have become common at larger academic libraries, although the current prevalence at non-arl academic libraries remains unclear.    an important research focus has involved overnight library users. the studies most relevant for this paper have examined services, resources and activities; what kinds of library services and resources do night patrons want during overnight hours, and what are they doing at the library during this time? lawrence & weber (2012) found that users are engaged in quiet study, using computers and printing, group work, and associating with friends. less common activities involved accessing course reserves or seeking assistance from library staff. engel, womack and ellis found that studying and using a computer were the two most common activities, with assistance from a librarian ranking last among nine options (2002). survey data ranked quiet study, work on projects or papers, group study, and printing as most important (scarletto, burhanna & richardson, 2013). demand for circulation and staff services was consistently low. laaker summed this up by noting that “the majority of late-night users come to [overnight hours] for the space itself—not for the physical collections and access to help desk services (2011, p. 22).”    these user studies have been informative. but the literature remains small—only a handful of studies have been conducted—and it is entirely descriptive (ravenwood, stephens, & walton, 2015, p. 53; scarletto et al., 2013, p. 372). accordingly, it is useful to extend it, as we do here.   background   the george a. smathers libraries at the university of florida consist of six units. the two largest facilities are the humanities and social sciences library, library west, and the marston science library, which serves stem and agriculture-related fields.   in 2013, at the urging of the university’s student government, the libraries inaugurated a “24/5 hours” program where one library remained open, around the clock, during weekdays. the program was located at the humanities and social sciences library for two years, and then switched to the science library in the third year. at this point, library administration sought to devise a longer-term arrangement. in order to collect data on overnight users, and choose a site for future overnight library hours, the administration commissioned a survey.   data and methods   survey instrument   the main body of the survey consisted of three distinct but overlapping sets of items. the first set asked survey respondents to rank 10 overnight library resources and services in order of their perceived importance. second, respondents were asked to indicate which of seven overnight resources and services they actually used. a third set of items asked respondents to evaluate the two candidate libraries on a set of resources and services, and to indicate which library—west or marston—would do a better job of providing each of these. respondents also indicated their preference for the location of the overnight hours program, and basic demographic variables were also collected (the complete survey instrument is provided in the appendix).   the survey instrument underwent pre-testing before going into the field. several volunteers serving in the current student government took the survey, providing think-aloud reactions and subsequent verbal feedback. this allowed the designers to modify any ambiguous or problematic items.   the survey was administered over a one-week period during the spring 2016 semester. responses were solicited in several ways. within each of the two candidate libraries, print surveys were distributed, and promotional signage directed library users to an online qualtrics survey website (each library used a unique website address, which allowed us to determine the origin of all online responses). online survey participation was also solicited through the libraries’ twitter and facebook accounts, and via a short news story in the campus newspaper (again using a unique qualtrics website). this provided three survey samples; respondents solicited within library west, respondents solicited within the marston science library, and respondents solicited outside of the libraries, through traditional and social media channels.   these samples were selected for several reasons. first, we wanted to obtain the opinions and preferences of current overnight users. print surveys were distributed and collected during overnight library hours at the marston science library. because overnight hours were being held only at marston during this time, print surveys were distributed and collected at library west during evening hours—9-11 p.m.—since this provided the closest approximation to overnight hours. second, we also wished to obtain responses from users who might not currently be visiting the libraries during overnight hours. to tap such current daytime users, print surveys were also distributed and collected during daytime hours at both libraries. (and, of course, the in-library signage could be seen at any time, and thus could have solicited responses from both user and non-user groups.) finally, our solicitations in social media and the newspaper were used to reach those overnight library users—intermittent users, or those away from campus—who might not be using the libraries at all during the one-week survey administration period.   since the survey involved choosing a location for overnight library services, it was possible     that some respondents might have had strong preferences on this matter, and thus an incentive to try to sway the outcome. if so, and they submitted multiple responses, this could have undermined the validity of our results. accordingly, we guarded against this possibility. with print surveys, we were careful to distribute only one copy to each library user in our sample. for online surveys, we prevented respondents from easily refreshing screens to submit another survey; instead, they would have had to log out or open multiple browsers. finally, following completion of the survey, we checked the respondent urls included in the qualtrics survey data, looking for repeated patterns. we found nothing that raised our suspicions or concerns.    a total of 2,852 respondents submitted surveys. a screening question revealed that approximately 83.5% of these respondents reported previous use of the library during overnight hours. since we were interested in obtaining feedback for overnight operations, the data analyses presented here were limited to this subset of respondents. this yielded a final sample size of 2,377.    while it would be desirable to report the survey response rate, we are unable to provide a direct figure on this. since our libraries do not specifically collect overnight gate count data, we have no way of determining the size of the overnight user population. however, several other studies do permit a rough estimate. studies from other state university libraries that collect such data have found that overnight users comprised between 17.5% and 22% of the overall student body (sanders & hodges, 2014; scarletto, burhanna & richardson, 2013). if our overnight “usage rate” was similar to these, our estimated overnight user population would range between approximately 9,500 and 12,000. with our sample size of 2,377, this would suggest a response rate of between 20 and 25%.   methods   by providing data on three distinct measures of overnight user preferences and activities, our survey supported a multi-method analysis of these phenomena.  in contrast to the more familiar multimethod approaches, which involve collection and analysis of multiple forms of data, we use multiple methodologies to analyze the same body of data (mingers, 2001; mingers & brocklesby, 1997). despite this difference, however, these two approaches share the same goal: both seek to provide greater analytical traction by approaching a research question from several different directions, and finding consistency across different sets of results.    descriptive statistics were used to analyze overnight users’ rankings of items on the first two scales: first, the perceived importance of 10 overnight library resources and services, and second, how often these were actually used during overnight hours. for importance, average ranking scores were calculated for each of the resources and services. for actual usage, we calculated the percentages of respondents who reported using each resource or service.   the correlational analysis relied on two pieces of data. as previously noted, users were asked to evaluate the two candidate libraries—library west and marston science library—on a set of resources and services, including library services, security inside the library, security outside the library, and a good atmosphere for study—and indicate which library would do the best job of providing each of these. for each attribute, respondents recorded their ratings on a three-point likert scale: better at marston, similar at both libraries, or better at west.   data on these four library attributes were then used as the independent variables in a multiple regression equation. the dependent variable came from the survey question asking respondents to indicate their preferred 24/5 hours site. while a dependent variable with a three-category likert scale might have prompted usage of some form of logistic regression, linear regression is robust for ordinal variables (gertheiss & oehrlein, 2011; winship & mare, 1984), and linear regression has the virtue of providing standardized beta coefficients and a meaningful r-squared statistic. the beta (or standardized regression coefficients) in the regression results will then show which of these four attributes had the strongest impact on users’ site preferences.   finally, a note on terminology. when analyzing such a wide range of library attributes, it is important to be clear about our wording. we use “resources” to refer to elements of the library like power outlets, study spaces, and library materials, and “services” to refer to staff-provided assistance and the starbucks café. the former category also includes “study atmosphere,” and the latter includes “security.” while these two elements do not fit as neatly into this scheme—neither security nor atmosphere represent a clear-cut resource or service—this approach does provide a useful clarity overall. where appropriate, “attributes” is used as a general term to refer to both resources and services.   results   for our core research focus—determining what kinds of library resources and services overnight patrons desire and use—analyses of all three sets of survey items produced highly consistent results.   perceived importance of library resources and services    survey respondents were asked to rank 10 library attributes in terms of their perceived importance. mean rank scores for each of these are reported in table 1 (since the rankings used a 1-10 scale, ranging from most to least important, lower numerical scores indicate higher rankings):   usage of resources and services   a second set of survey items moved beyond attitudes and preferences regarding library resources and services by asking respondents which of these they had actually used during overnight hours. the results in table 2 indicate the percentages of overnight users reporting usage of each resource or service.     table 1 mean importance rankings of library resources and services power outlets   quiet study space group study space café   library tech library materials security inside library security outside library assist from library staff other tech 2.8 2.8 3.5 4.7 4.9 6.3 6.3 6.6 8.1 9.1 n=2312   table 2 percentage reporting usage of library resources and services power outlets quiet study space café   group study space library tech library materials assist from library staff 90 81 81 73 44 19 5 n=2366                                                                across tables 1 and 2, the library attributes differ to some degree. security and the “other technology” responses were omitted from the “usage” section of the survey; security is not an attribute that is personally “used” by patrons, and 3d printing was available only in marston (and thus could not be used in west). but for the seven resources and services that were included in both tables, the orderings were highly consistent; power outlets and quiet study space were viewed as most important, and used most frequently, followed by group study and starbucks, with technology, library materials, and staff assistance seen as least important, in that order.   while it may not seem surprising that patrons’ actual usage of library resources and services would match their views on the importance of these attributes, it is important to note that such consistency is by no means a given. these two sets of items tap two different phenomena—attitudinal preferences and behavior—and they are measured with two different types of scales; one involves ranking alternatives while the other asks users to check boxes indicating their usage of resources and services. and from a more theoretical point of view, a large social psychology literature demonstrates the common lack of linkage between attitudes and behaviors, attesting to the finding that “attitudes and preferences do not always prove to be good predictors of actual behavior” (elen, d’heer, geuens & vermeir, 2013; see ajzen & fishbein, 1977, and ajzen & fishbein, 2005 for general treatments of this topic). accordingly, the consistency seen here is revealing and important.   resources and services -impacts on library preferences   our survey data also allowed us to take a third look at the priorities and preferences of overnight users. survey respondents were asked to indicate their preferred 24/5 hours site; they could choose either marston science library or library west, or indicate that these two libraries would be equally desirable. they were also asked to compare these two libraries on a set of relevant attributes; study atmosphere, library services, and security inside and outside the library.   we used a multiple regression analysis to determine how these four attributes impacted upon users’ overnight library choices. in this situation, impact equates with correlation size, and thus the attribute showing the strongest correlation had the strongest impact on respondents’ library choices, the attribute with the weakest correlation had the weakest effect, and so on.   based on users’ rankings of importance, as seen in table 1, we would expect that study atmosphere and library services would show the strongest correlation with users’ library choices, with security concerns showing the weakest correlation.     table 3 regression results: service ratings and overnight library choice   (constant) 4.159 .040 - 103.652 .000 study atmosphere .466 .018 .444 26.210 .000 library services .409 .019 .381 22.082 .000 security outside .127 .017 .095 7.319 .000 security inside .082 .018 .055 4.437 .000 n=2320      adjusted r2 .722               as table 3 shows, this hypothesis was correct. the size of the standardized regression (beta) coefficients indicate that a good study atmosphere (captured by quiet and group study), and library resources (power outlets, starbucks, and library technology and materials) had a stronger impact on users’ overnight library choices than the security did.   the low importance placed on security might seem to reflect our library’s location in a university community, or “college town.” if this was so, then the importance of security to users could be artificially suppressed. however, this seems unlikely for several reasons. first, the local crime rates are well above the state and national averages. accordingly, security concerns are unlikely to be suppressed by the bucolic nature of our university surroundings. this point is buttressed the overnight library literature; overnight library users have indicated a relative lack of security concern in other studies (lawrence & weber, 2012, p. 538; scarletto et al., 2013, p. 374). thus we conclude that the importance of library factors did outrank the importance of security, just as in table 1.   demonstrating that library factors outranked security represents a rather general test, and thus it would ideally be desirable to further unpack the “study atmosphere” and “library services” categories in order to determine the impacts of individual factors like study space, materials usage, technology and so on. unfortunately, however, this is not possible here. since we were concerned about survey fatigue, we did not repeat the entire list of factors included in table 1, and thus we have measurements only for the general library categories seen in table 3.   still, we can be confident in the general picture here. the table 3 results do reflect those from table 1. moreover, the adjusted r-squared of .722 indicates that the model performed extremely well, explaining almost three-quarters of the variation in user siting preferences, and thus that these four criteria, as a set, strongly shape these preferences.   results across multiple survey samples   overall, then, our multi-method approach produced satisfying results; the findings from these three separate analyses triangulate well, and present a consistent picture. this consistency provides confidence that we have accurately captured the views of our overnight library users.     table 4 importance of library resources and services by sample   power outlets quiet study space group study space café   library tech library materials security inside library security outside library assist from library staff other tech marston 1 2 3 4 5 6 7 8 9 10 media 1 2 3 4 5 6 7 8 9 10 west 2 1 3 4 5 6 7 8 9 10 n = 811 for marston science library; 774 for media; 727 for library     table 5 usage of library resources and services by sample   power outlets quiet study space café   group study space library tech library materials assist from library staff marston 89 73 80 74 40 18 5 media 93 85 83 78 44 17 4 west 88 85 81 68 48 22 7 n = 840 for marston science library; 783 for media; 743 for library west        however, the results presented above are based on a single, combined and aggregated survey sample. accordingly, one must be cautious in generalizing from this. however, our survey method permitted more specific analysis. our aggregated sample was created by combining three separate sub-samples—solicited within marston science library, library west, or via social and newspaper media—and so these can be broken out by sample and analyzed separately. repeating our earlier analyses, by sample, we can see that all three groups of users showed a highly similar ordering of priorities. table 4 summarizes user rankings of the importance of 10 overnight library resources and services across the three samples.   as is evident, the importance rankings, given by users, are strikingly consistent across the three survey samples.   a similar finding is reported for actual usage of resources and services. here too we see that reported patterns of use (measured by percentage of users reporting usage of each type of library resource of service) were broadly similar across the three sets of survey samples (table 5).   thus no matter which campus library they patronize, or how their participation was recruited, overnight users hold similar perceptions and engage in similar types of activities.   finally, we can compare regression results across our three library samples. table 6 summarizes beta coefficients and other key aspects of regression model performance for all three samples.    as is evident, all three samples produced similar results; strong impacts for study atmosphere and library services, with minimal, if any, appreciable substantive effects for security concerns.     table 6 regression results: library attributes and overnight library choice   beta (marston science library sample) beta (media sample) beta (library west sample) study atmosphere .347*** .492*** .480*** library services .414*** .341*** .320*** security outside .088*** .101*** .115*** security inside .101*** .053* .008         sample n 816 782 722 adjusted r2 .617 .764 .588 * = significant at .05; ** = significant at .01; *** = significant at .001     limitations   self-reported usage measures   one limitation involves our procedure for measuring actual usage of library resources and services. these statistics are based on user self-reports, and the accuracy of such reports have been a subject of persistent controversy. self-report studies have most commonly examined health-related topics, including eating habits and body mass (bowman & delucia, 1992), or smoking and drinking alcohol (del boca & darkes, 2003). others have addressed matters as diverse as voting turnout, marital violence, and donations to charitable organizations (abelson, loftus & greenwald, 1992; arias & beach, 1987; bekkers & wiepking, 2011). many of these studies have uncovered systematic divergences between individuals’ self-reports and documented measures of the same phenomena. at the same time, however, other studies have concluded that self-reports are often accurate, and that they provide “a reliable and valid approach” for analyzing behavior (del boca & darkes, 2003, p.1).    for our purposes, this previous work does not permit us to make a well-grounded inference about the accuracy of usage reports by our library users. mixed findings are reported in the literature, and we were unable to identify any studies that had specifically examined self-reports of library usage.   the usual solution would involve obtaining independent measures of these behaviors and comparing them with our user reports. as perusal of table 2 shows, this would require marshaling several types of data. usage measures for library materials and user demand for staff assistance and group study space could draw on existing “transactional” data—that is, data generated by library systems for circulation, reference transactions, and study room reservations. data for usage of quiet study spaces and power outlets would have to be collected; this could be accomplished by observing usage of these at a randomly selected set of times. the same observational approach could be used for starbucks café patronage (or, alternatively, perhaps the company would provide data on business volume).   while all of these steps would be feasible, they would require substantial time and effort. this is the major reason why such checks have not played a larger role in scholarly research studies—with previous overnight library studies included—and, conversely, why researchers have taken such great interest in the accuracy of self-reports.     table 7 sample vs. population demographic percentages   sample population undergraduates 91.0 64.3 graduates 7.9 31.0       male 34.0 45.5 female 64.7 54.4 n = 2,337 (sample)     54,208 (population) percentages total less than 100 due to omission of smaller categories   table 8a mean importance rankings of library resources and services — undergraduates power outlets quiet study space group study space café   library tech security inside library library materials security outside library assist from library staff other tech 2.7 2.8 3.4 4.7 4.9 6.2 6.4 6.5 8.1 9.2 n = 2099   table 8b mean importance rankings of library resources and services — graduate students quiet study space power outlets group study space library tech café   library materials security inside library security outside library assist from library staff other tech 2.7 3.8 4.2 4.4 4.9 5.0 6.6 7.2 7.3 8.7 n = 179     non-random sample   the most significant limitation in this study is common to most survey-based library research; the lack of a randomly-selected sample. this makes it difficult or impossible to generalize research findings. in this paper, however, this problem may be somewhat mitigated by our use of multiple samples, and by the highly consistent findings across these samples. this cross-sample consistency may suggest that our university student population is highly homogenous in its overnight library preferences. if so, then our sample results may be reasonably consistent with this larger population.   still, it would be useful to have a better grasp on this key matter. in order to examine this, we obtained university-level data from our university’s office of institutional planning and research. by comparing our sample with the overall student population on two demographic variables included in our survey—gender and academic status (e.g. undergraduate-or graduate-level standing)—we can estimate the demographic representativeness of our sample (table 7).     clearly, our sample deviates from the demographics of the overall university student population; sharply so for the balance between undergraduate and graduate students. if students in different demographic categories have different perceptions and usage patterns, then this may have induced biased results in our survey.     table 8c mean importance rankings of library resources and services — male respondents quiet study space power outlets group study space library tech café   library materials security inside library assist from library staff security outside library other tech 2.6 3.0 3.3 4.4 5.3 5.6 6.9 7.7 7.7 8.7 n = 777     table 8d mean importance rankings of library resources and services — female respondents power outlets quiet study space group study space café   library tech security inside library security outside library library materials assist from library staff other tech 2.7 2.9 3.6 4.4 5.1 5.9 6.0 6.7 8.3 9.3 n = 1501   perceived importance of library resources and services      in order to examine this, we broke our previous findings down by these demographic categories. tables 8a-8d show the results for mean importance rankings of library resources and services.   when we compare our results across gender and academic status, it is clear that the rankings across groups are broadly similar. there are some differences, of course; for graduate students, power outlets and group study spaces are a bit less important than for undergraduates, with library materials a bit more important. presumably this reflects undergraduates’ habit of using the library primarily as a study facility, and graduates’ increased engagement with scholarly materials. for the second comparison, female students, not surprisingly, show a somewhat heightened concern for security.    overall, however, all the groups are broadly similar in their importance rankings. power outlets, quiet and group study spaces are most important, followed by café services and library technology. library staff assistance and other technology are usually the least important, with the remaining resources and services somewhere in between.   usage of resources and services   our usage percentage results also show significant cross-group similarity. power outlets, café services, and quiet and group study spaces generally vie for the most used library attributes. library technology, library materials, and staff assistance are used least; the last three have an identical ordering across all groups. there are some differences in rate of usage, with graduate students showing less usage than undergraduates, but the relative usage orderings show a broad consistency (tables 9a-9d).   for our third measure, the regression analysis, the overall pattern again is similar to what we have previously seen. while the specific numbers vary, the pattern of beta coefficients are similar across all groups; study atmosphere and library services are the most important, with security least important, just as in tables 3 and 6 (table 10).   in general, then, group breakdowns on all three of our measures show consistent results. what does this tell us? in one sense, this nicely extends our earlier findings, which also identified broad consistency across measures of resources and services. accordingly, our claims about consistency are even more robust than we had supposed.     table 9a percentage reporting usage of library resources and services — undergraduates power outlets café   quiet study space group study space library tech library materials assist from library staff 92 82 81 75 43 18 5 n = 2143     table 9b percentage reporting usage of library resources and services — graduate students quiet study space power outlets café   group study space library tech library materials assist from library staff 75 73 71 52 52 28 7 n = 184     table 9c percentage reporting usage of library resources and services — male respondents power outlets quiet study space group study space café   library tech library materials assist from library staff 85 78 68 67 45 21 7 n = 802     table 10 regression results: library attributes and overnight library choice – all groups   beta undergraduates beta graduate students beta males beta females study atmosphere .445*** .432*** .483*** .425*** library services .388*** .320*** .328*** .405*** security outside .094*** .086 .087*** .099*** security inside .047*** .148** .070** .049**           sample n 2106 182 785 1504 adjusted r2 .728 .644 .699 .733 * = significant at .05; ** = significant at .01; *** = significant at .001       as for the question at hand, the sample-population relationship, do these findings provide evidence for a representative sample? in one sense, yes. the consistency across groups suggests that demographic imbalance in our sample may have only minor effects. however, this still does not mean we can confidently generalize from our sample to our overall university student population. it still is possible that the subjects we sampled—regardless of their demographic categories—might be systematically unrepresentative of the broader population.   a more confident conclusion would require a more advanced methods. a commonly recommended solution would apply a weighting scheme via a “raking” or sample-balancing procedure (anderson & fricker, 2015; battaglia, hoaglin & frankel, 2009). while such an approach is outside of the scope of this paper, it would represent a logical path for future studies in this area.   discussion and conclusion   this paper adds an additional case study to the small existing literature on the attitudes and behaviors of overnight users at academic libraries. and in supplementing previous studies, it has employed a more advanced approach that has extended the literature in two useful ways.   first, our usage of a multi-method research design has allowed us to take a multifaceted look at overnight users. while prior research in this area has relied solely on descriptive approaches, we employ both descriptive and correlational statistics to demonstrate consistent findings across multiple measures of user preferences. second, by replicating these results across multiple survey samples, we provide additional evidence for the veracity of these findings. three different samples produced a highly consistent picture of the preferences and behaviors of overnight library users.   either of these approaches, by itself, would buttress our conclusions. together, these two approaches—and the consistent findings they produce—increase confidence in our results compared with earlier studies. while grounds for generalizing from our sample results to our overall university population remain elusive, we believe that the increased internal validity in this study does advance practice in this area of the library literature, and we would suggest that future studies might consider adopting this type of approach.   references   abelson, r.p., loftus, e. f., & greenwald, a.g. (1992). attempts to improve the accuracy of self-reports of voting. in j. m. tanur (ed.), questions about questions: inquiries into the cognitive bases of surveys (pp. 138-153). new york, ny: sage.   ajzen, i., & fishbein, m. (1977). attitude-behavior relations: a theoretical analysis and review of empirical research. psychological bulletin, 84(5), 888-918. doi: 10.1037/0033-2909.84.5.888   ajzen, i., & fishbein, m. (2005). the influence of attitudes on behavior. in d. albarracin, b.t. johnson, & m.p. zanna (eds.), the handbook of attitudes (pp. 173-221). mahwah, nj: lawrence erlbaum.   anderson, l., & fricker, r.d. (2015). raking: an important and often overlooked survey analysis tool. phalanx, 48(3), 36-42. retrieved from https://www.researchgate.net/publication/305215399_raking_an_important_often_overlooked_survey_analysis_tool   arant, w., & benefiel, c.r. (2002). hours of operation and service in academic libraries: toward a national standard. public services quarterly, 1(1), 71-85. http://dx.doi.org/10.1300/j295v01n01_08   arias, i., & beach, s.r.h. (1987). validity of self-reports of marital violence. journal of family violence, 2(2), 139-149. retrieved from https://link.springer.com/journal/10896/2/2/page/1   association of college and research libraries. (2006). but every other college library is open 24/7. acrlog. retrieved from http://acrlog.org/2006/11/20/ but-every-other-college-library-is-open-247/    battaglia, m.p., hoaglin, d.c., & frankel, m.r. (2009). practical considerations in raking survey data. survey practice, 2(5), 1-10. retrieved from http://surveypractice.org/index.php/surveypractice/issue/view/40   bekkers, r., & wiepking, p. (2011). accuracy of self-reports on donations to charitable organizations. quality & quantity, 45(6), 1369-1383. doi:10.1007/s11135-010-9341-9    bowman, a.c. (2013). 24-hour academic libraries: adjusting to change. journal of access services, 10(4), 217-239. http://dx.doi.org/10.1080/15367967.2013.842342   bowman, r.l., & delucia, j.l. (1992). accuracy of self-reported weight: a meta-analysis. behavioral therapy, 23(4), 637-655. http://doi.org/10.1016/s0005-7894(05)80226-6   del boca, f.k., & darkes, j. (2003). the validity of self-reports of alcohol consumption: state of the science and challenges for research. addiction, 98(2), 1-12. doi:10.1046/j.1359-6357.2003.00586.x   elen, m., d’heer, e., geuens, m., & vermeir, i. (2013). the influence of mood on attitude-behavior consistency. journal of business research, 66(7), 917-923. http://dx.doi.org/10.1016/j.jbusres.2011.12.011   engel, d., womack, k, & ellis, u. (2002). opening a library 24 hours. journal of library administration, 36(4), 95-108. http://dx.doi.org/10.1300/j111v36n04_07   gertheiss, j., & oehrlein, f. (2011). testing linearity and relevance of ordinal predictors. electronic journal of statistics, 5, 1935-1959. http://dx.doi.org/10.1214/11-ejs661   laaker, s. (2011). keeping the doors open: exploring 24-hour library access at washington university in st. louis. research library issues: a quarterly report from arl, cni, and sparc, 277. 15-25. retrieved from http://publications.arl.org/rli277/26?ajax   lawrence, p., & weber, l. (2012). midnight-2 a.m.: what goes on at the library? new library world, 113(11/12), 528-548.  http://dx.doi.org/10.1108/03074801211282911   mingers, j. (2001). combining is research methods: towards a pluralist methodology. information systems research, 12(3), 240-259.  retrieved from http://web.a.ebscohost.com/ehost/pdfviewer/pdfviewer?vid=2&sid=08da5903-54ca-4034-a889-6f47a618d921%40sessionmgr4010&hid=4201   mingers, j., & brocklesby, j. (1997). multimethodology: towards a framework for mixing methodologies. omega, the international journal of management science, 25(5), 489-509. retrieved from https://doi.org/10.1016/s0305-0483(97)00018-2   ravenwood, c., stephens, d., & walton, g. (2015). establishing influential decision making factors for university library opening hours: an exploratory uk regional study. new review of academic librarianship, 21(1), 52-67. http://dx.doi.org/10.1080/13614533.2014.914960     sanders, m., & hodges, c. (2014). an overnight success? usage patterns and demographics of academic library patrons during the overnight period from 11 pm-8am. journal of access services, 11(4), 309-320. http://dx.doi.org/10.1080/15367967.2014.945121   scarletto, e. a., burhanna, k. j., & richardson, e. (2013). wide awake at 4 a.m.: a study of late night user behavior, perceptions, and performance at an academic library. the journal of academic librarianship, 39(5), 371-377. http://dx.doi.org/10.1016/j.acalib.2013.02.006   smith, f. (2008). 24 hour service at georgia southern university: 1989-2007. journal of access services 5(1-2), 69-83. http://dx.doi.org/10.1080/15367960802198226   winship, c., & mare, r.d. (1984). regression models with ordinal variables. american sociological review, 49(4), 512-525. retrieved from http://www.jstor.org/stable/pdf/2095465.pdf       appendix survey instrument   for almost five years, the smathers libraries have offered “overnight library hours” (that is, keeping one library open from 1 a.m. to 8 a.m. sunday through thursday), with funding from student government. the libraries, together with your student government, would like to get your feedback on where we offer overnight library hours during the 2016-2017 academic year. while both library west and marston science library will continue to provide extended hours during reading week and exams week, we are seeking your opinion about where the overnight library hours should be maintained.   you can take this survey anonymously or volunteer to participate in focus groups to discuss future library use studies by providing your email address which will be securely retained by researchers until december 31, 2016.   your participation is completely voluntary and you can withdraw from the study at any time without consequence.   1.       prior to this survey, did you know that library west and marston science         library have been open during overnight hours between 1 a.m. and 8         a.m.? ð     yes ð     no   2.       have you used any either library during overnight hours between 1 a.m.         and 8 a.m.? ð        yes ð        no   3.       during the 2015-2016 academic year, how often have you used the marston science library building during “overnight” hours between 1 a.m. and 8 a.m.? ð        never ð        once ð        a few times ð        often (once a week or so) ð        very often (more than once a week) ð        not applicable (i was not at uf at that time)   4.       during the 2014-2015 academic year, how often did you use the library         west building during “overnight” hours between 1 a.m. and 8 a.m.? ð        never ð        once ð        a few times ð        often (once a week or so) ð        very often (more than once a week) ð        not applicable (i was not at uf at that time)   for each of the following topics, please indicate which library you prefer during overnight hours   5.       what materials or services do you use when you visit the library         overnight? ð        technology (printers, scanners, computers) ð        materials (books, electronic resources, course reserves) ð        group study space (group study rooms, group seating space) ð        quiet study space (silent/quiet floors; graduate study space ð        starbucks ð        power outlets ð        assistance from library staff ð        other   6.       library services (technology, materials, study spaces, starbucks, library         staff assistance) ð        much better at library west ð        better at library west  ð        similar at both libraries ð        better at marston science library    ð        much better at marston science library   7.       good atmosphere for study ð        much better at library west ð        better at library west  ð        similar at both libraries ð        better at marston science library     ð        much better at marston science library   8.       security inside the library building ð        much better at library west     ð        better at library west ð        similar at both libraries ð        better at marston science library ð        much better at marston science library   9.       security outside the library building (parking lots, walkways) ð        much better at library west  ð        better at library west  ð        similar at both libraries ð        better at marston science library     ð        much better at marston science library   10.    parking ð        much better at library west     ð        better at library west ð        similar at both libraries ð        better at marston science library ð        much better at marston science library ð        not applicable – i do not need to park   11.    close to my residence ð        much better at library west     ð        better at library west ð        similar at both libraries ð        better at marston science library ð        much better at marston science library ð        not applicable   12.    now, please rank the importance of these ten areas with 1=most         important, 2=second most important, 3=third most important etc. ð        technology (printers, scanners, computers) ð        materials (books, electronic resources, course reserves) ð        group study space (group study rooms, group seating space) ð        quiet study space (silent/quiet floors; graduate study space) ð        starbucks ð        power outlets ð        assistance from library staff ð        other (3d technology, made@uf) ð        security in the library building ð        security outside the library building   13.    for the next academic year, 2016-2017, would you prefer to have         library overnight hours be held at library west or at the marston science library? ÿ        i would prefer to have overnight hours at library west ÿ        i would prefer to have overnight hours at marston science library  ÿ        i would be equally satisfied to have overnight hours at either library   14.    please provide additional comments you have: ______________________________________________________   please indicate your class standing ÿ        freshman/sophomore ÿ        junior/senior ÿ        masters ÿ        doctoral/professional ÿ        post-baccalaureate ÿ        faculty ÿ        other   please indicate your gender ÿ        man ÿ        woman ÿ        transgender ÿ        other ÿ        prefer not to answer   please provide your primary area of study____________________   if you are interested in participating in a group discussion about planning for future space and new services in the smathers libraries, we would love to include you in student focus groups later this year. please provide an email address at which you can be reached and we will contact you when that study begins.   email address ________________________________   thank you for your feedback!     developing a library association membership survey: challenges and promising themes research article   developing a library association membership survey: challenges and promising themes   mary dunne information specialist health research board dublin, ireland email: mdunne@hrb.ie   received: 25 apr. 2022                                                              accepted: 8 july 2022      2022 dunne. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30157     abstract   objective – many of us involved in the library and information sector are members of associations that represent the interests of our profession. these associations are often key to enabling us to provide evidence based practice by offering opportunities such as professional development. we invest resources in membership so we must be able to inform those in charge about our needs, expectations, and level of satisfaction. governing bodies and committees, therefore, need a method to capture these views and plan strategy accordingly. the committee of the health sciences libraries group (hslg) of the library association of ireland wanted to enable members to give their views on the group, to understand what aspects of a library association are important to librarians in ireland, and to learn about the reasons for and against membership.   methods – surveys are a useful way of obtaining evidence to inform policy and practice. although relatively quick to produce, their design and dissemination can pose challenges. the hslg committee developed an online survey questionnaire for members and non-members (anyone eligible to join our library association). we primarily used multiple choice, matrix, and contextual/demographic questions, with skip logic enabling choices of relevance to respondents. our literature review provided guidance in questionnaire design and suggested four themes that we used to develop options and to analyse results.   results – the survey was made available for two weeks and we received 49 eligible responses. analysis of results and reflection on the process suggested aspects that we would change in terms of the language used in our questionnaire and dissemination methods. there were also aspects that show good potential, including the four themes that were used to understand what matters to members: expertise (professional development), community (connecting and engaging), profession (sustaining and strengthening), and support (financial and organizational supports). overall, our survey provided rich data that met our objectives.   conclusion – it is essential that those who are governing any group make evidence based decisions, and a well-planned survey can support this. our article outlines the elements of our questionnaire and process that didn’t work, and those that show promise. we hope that lessons learned will help anyone planning a survey, particularly associations who wish to ascertain the views of their members and others who are eligible to join. with some proposed modifications, our questionnaire could provide a template for future study in this area.     introduction   library associations are professional organizations formed to bring together those involved in library-related work who share common interests in subjects, types of services, or other factors, such as geographical location (librarianship studies & information technology, 2020). at the local, national, regional, and international levels they play an important role in the development of subject fields; provide opportunities to enhance skills and knowledge, and a platform for discussion; unite and give voice to professionals; and keep members up to date with new developments (dowling & fiels, 2009). to be successful, library associations need to fulfill the goals and expectations of their members, so it is crucial that those managing association strategy and making decisions understand these factors.   the health sciences libraries group (hslg) has been a special interest group of the library association of ireland since 1982, with a recent average of about 50 members. we have an annual conference, annual general meeting, virtual journal club, email discussion list, e-newsletter, website, and hold regular continuing professional development (cpd) and networking events. the committee manages governance and activities on behalf of members. to meet expectations, we needed to obtain their views on the resources and services provided by the group, the aspects that are most important to them, and their reasons for membership. we also wanted to understand why some health librarians in ireland are not members of our group. we conducted a literature search and developed an online survey that was made available in november 2021.   literature review   a search of proquest library science database in september 2021 using the term “library association” gave a useful overview of available literature. this was followed by checking of reference lists, and a search of library association websites. two aspects were of particular interest: the questions used in past survey studies and the themes that emerged from texts. four identified themes related to what members may expect to contribute and receive through association membership: (1) expertise professional development, (2) community connecting and engaging, (3) profession sustaining and strengthening, (4) support financial and organizational supports.   the chartered institute of library and information professionals’ (n.d.-c) five-year action plan has four value propositions—community, expertise, representation, and recognition—that are similar to our first three themes. although “support” may be subsumed within the other themes, for the purpose of examining membership, keeping it separate is useful for highlighting potential barriers or facilitators to joining or engaging in an association.   themes were identified in a range of articles. some were descriptive commentaries or desk research about the value of library associations (broady‐preston, 2006; chase, 2019; dimauro, 2011; joint, 2007; lumpkin, 2016; morrison, 2004; wise, 2012). other articles involved primary research, including studies that indirectly referenced the role of library associations, such as corcoran & mcguinness (2014) who interviewed academic librarians about cpd, and studies that directly researched the subject. for example, in their 2020 study, garrison and cramer (2021) received 140 complete responses when surveying business librarians about what they wanted from their professional associations. henczel (2014) used a phenomenological approach to study the impact of library associations. she conducted 52 semi-structured interviews with members of national library associations, providing a wealth of information. spaulding & maloney (2017) also looked at impact, asking how belonging to and participating in a professional association as a student impacted careers. they reported on 1,869 responses from their online survey. frank (1997) conducted focus groups on the value of being active in professional organizations. in the same year, kamm (1997) received 116 responses to her u.s. survey on how members make decisions about their library association.   identified themes related to library association membership   expertise professional development   one of the common themes in the literature on library associations is the provision of continuing professional development (cpd), including access to training and skills building through attendance at courses, workshops, conferences, and webinars (henczel, 2016b). new knowledge, competencies, and skills gained through this cpd were viewed as a means of boosting resumes (schwartz, 2016). while active participation in associations demonstrated engagement, leading to career enhancement (frank, 1997; garrison & cramer, 2021; spaulding & maloney, 2017) and opportunities for research and publication (chase, 2019; wise, 2012). lachance (2006) remarked that “no library association can survive, sustain, grow, or remain relevant in the modern age if it does not address members' educational needs and provide innovative learning solutions that lower barriers to access” (p. 9).   most associations facilitate professional accreditation pathways that encourage cpd and provide specialist professional competency standards to guide learning. henczel (2014) found that professional registration was regarded by her study participants as a reason for joining associations, retaining membership, and becoming more participative in association activities. registration and certification are available through associations such as the australian library and information association (alia), cilip (uk), and the library and information association of new zealand aotearoa (lianza). these schemes list multiple benefits for participation, including increasing the standing of the profession, recognising professional excellence and cpd, and providing a mechanism for employers to coach and develop staff (lianza, n.d.), increased status, earnings, and recognition of abilities, skills, and experience (cilip, n.d.-e). changes in skills and competences also came about through participation in association activities (henczel, 2014).   community connecting and engaging   studies frequently report it is important for those involved in the library or information sector to have opportunities to connect through networking and collaboration (davidson & middleton, 2006; frank, 1997; kamm, 1997; sauceda, 2018; spaulding & maloney, 2017). garrison & cramer (2021) described networking as vital, saying that healthy organizations must provide ample opportunity for members to share experiences (good and bad), insights, suggestions, and to build friendships and have fun. they assert that library associations should support members through sharing expertise, connecting members in various roles, and “creating a network of supportive colleagues and mentorship” (p. 35).   specific groups of people have been identified as sometimes needing more support in their practice. the ability to participate in an informal network of colleagues can be of enormous benefit, especially for solo or specialist librarians according to chase (2019). bradley et al. (2009) contended that new professionals can benefit from simply observing and interacting with colleagues, and seeing their peers being treated with professional respect. associations have been found to make a difference through their support of members moving across sectors, students and new graduates, those in non-traditional roles, living in rural or geographically isolated areas, and those nearing retirement (henczel, 2014). as spaulding and maloney (2017) assert, we need to connect with people through transitions.   profession sustaining and strengthening   progress and cohesiveness within our profession is being achieved by setting and monitoring of global values and professional standards, accrediting courses and curricula, active recruitment, and disseminating research and professional information that will enhance our reputation as a profession (agee & lillard, 2005). henczel’s (2016a, 2016b) major thesis considers library association impact on individuals, employers, and the profession. her research concluded that five perceived impacts related to the profession: social inclusion and cohesion, information and education, promotion of the profession, and the sustainability of the profession. although much of the literature on the value of associations is based on the personal attitudes of members, some associations have produced literature to demonstrate their impact. for example, researchers streatfield and markless (2019) have worked closely with the international federation of library associations and institutions (ifla) to evaluate the impact of its international programs: freedom of access to information, international advocacy programme, and building strong library associations. the latter program, in turn, focused on helping associations build capacity and meet their goals (ifla, 2016a).   beyond the specific knowledge required to practice, librarians have acknowledged the benefits of being aware of what is happening in the profession and preserving our professional cultural heritage for future generations (henczel, 2014). the altruistic view of contributing to our profession is mentioned throughout the literature (chase, 2019), with many seeing membership as an obligation or “the right thing to do” (kamm, 1997, p. 299) and a way of “giving back to the profession” (henczel, 2014, p. 131). library associations have also been described as a forum to champion our values, such as open access to information (morrison, 2004).   political action, particularly lobbying, has been cited as an important role of our associations (agee & lillard, 2005; kamm, 1997). some librarians have expressed the importance of having a single, united, and strong representative voice (henczel, 2014). ahmadian yazdi and deshpande (2013) viewed it as essential for professionals “to meet and plan their activities to safeguard and promote the interests of their particular profession” (p. 92).   support financial and organizational supports   a fourth theme that was seen as potentially important in relation to library association membership related to costs and employer support. some associations, such as the american library association, have reported declining membership (ala, 2020). one of the main concerns, or reasons for non-membership, has been cited as cost (frank, 1997; kamm, 1997). although financial incentives in terms of grants and member discounts were referred to as a frustration when access is limited (garrison & cramer, 2021), it has also been suggested as a positive reason for joining (schwartz, 2016). the extent of employer support of their activities, either by paying dues or expenses for conferences and meetings, has also been cited as an important factor in the selection of an association (kamm, 1997).   barriers to participating in cpd include time, financial costs, and lack of support from employers (thomas et al., 2010). corcoran & mcguinness (2014) have suggested that professional library organizations must be innovative and consider incentives to participate that resonate with members. this theme of “support,” therefore, involves some of the practical barriers and facilitators to membership that associations must consider.   aims   we began with an iterative process that involved the setting of our aim and objectives, a literature search to assess what was known about the subject, and a review of emerging themes. the hslg committee want to retain current members but also to understand why some of those involved in relevant positions have never joined or have left us. the overall purpose, therefore, was to enable evidence informed decisions by the committee leading to a strategy based on the views and needs of members and that tackles potential barriers to membership. we focused primarily on the views of those involved in health settings but also wanted to be guided by those from other sectors.   survey aim: to gain insight into the issues of relevance to membership of our group and association.   our objectives were to   enable hslg members to give their views on the group, understand what aspects of a library association are important to librarians in ireland, and learn about reasons for and against membership.   methods   questionnaire design   a survey is a quick way of gathering data and allows everyone in a defined population to contribute. online survey providers enable easy creation of various question types and answer options (ball, 2019; nayak & narayan, 2019). for a cost, there are also advanced features such as skip logic (questions offered depend on the previous answers so participants skip irrelevant questions) and crosstab analysis (useful when comparing the answers of participant sub-groups).   however, self-completed surveys do not generally allow for in-depth interrogation or clarification of answers. the wording of questions may also be interpreted differently by participants (particularly if care is not taken during design; french, 2012). where time and costs allow, a qualitative method such as focus groups or interviews would provide additional data and real-life examples to improve understanding (granikov et al., 2020).   in line with good questionnaire design, we only included a question if it could provide important context or useful application (the answers could enable action; national care experience programme, n.d.). for example, new librarians have been identified as potentially having different views and needs to others (chase, 2019; joint, 2007); therefore, a question on length of service was warranted. the number of questions asked depended on the association membership status of participants. questionnaires with more items tend to have a lower return rate (french, 2012), so we asked most questions of those who belonged to our group, as they may be more invested and receive the greatest benefit from providing responses. no personal data (such as age) were necessary.   to facilitate skip logic and analysis by population, we organized our questionnaire into sections. section 1, which was answered by everyone, contained contextual questions based largely on the four variables used by henzcel (2016) in her study on library association impact: association, sector affiliation, career stage, and activity levels. this included information on the work or study status of participants, how long they had worked in the sector, whether they received financial support to join a library association or attend events, whether they had ever been on a committee, and their association membership status. the latter (q7, table 1) was primarily used to direct respondents to further questions. (see appendix a for survey instrument.)   table 1 survey skip logic questionnaire flow a q7. please tick the most appropriate option for you: if yes, then directed to: 1.      i am a hslg member sections 2 and 5 2.      i am a library association of ireland member (but not the hslg) section 5 3.      i belong to another professional library association, instead of the library association of ireland sections 4 and 5 4.      i am a former library association member section 6 5.      i have never belonged to a library association section 6 a section 3 asked why someone working in a health setting was not a member of the hslg. this required additional skip logic in question 1.   closed questions, with options provided, were primarily used for ease and speed of completion, but in case options were not exhaustive, “other, please specify” and open questions were added where appropriate. choices were listed alphabetically to prevent researcher bias in terms of order. only question 1 (on eligibility and status) and 7 (required for skip logic) were mandatory. evaluative questions provide a baseline measure and an opportunity for governing committees to review areas that are working and those that need improvement. these questions can be asked at regular intervals to monitor progress. therefore, we asked participants to rate the value they place on membership, how well we are currently meeting their needs and expectations, and to identify gaps in services. wording of these questions and the options provided were inspired by those used in previous library association survey studies (garrison & cramer, 2021; henczel, 2016a; ifla, 2016b). however, to make items salient to our members and to meet our objectives we developed our own survey tool.   questionnaire testing   questionnaires require testing to assess reliability and validity of questions. reliability refers to how well data can be reproduced, with a reliable survey resulting in consistent information. validity is how well a questionnaire measures what it is intended to measure, with a valid survey producing accurate information (fink & kosecoff, 1998; meadows, 2003). both can be obtained by ensuring that definitions and models used to select questions are grounded in theory or experience (fink & kosecoff, 1998, p. 6), thus underpinning the importance of the literature review and researcher discussions.   using skip-logic requires additional time for testing as each potential option needs to be followed to ensure appropriate flow. one hslg committee member devised the questionnaire and the other five members previewed and filled it in multiple times to check that questions and answer options were appropriate, comprehensive, and made sense (face and content validity).   survey dissemination   we made the survey available online in the first two weeks of november 2021 and sent the link via our group membership list (49 recipients), discussion email list (85 recipients, including members and non-members working in health librarianship), the library association newsletter (approximately 570 personal members), our website, and via three invitations to participate from our twitter account. as an incentive, and a means of thanking participants, we offered eligible respondents the chance to enter a draw for a €50 voucher. to ensure that responses remained anonymous, we set up a separate survey for the draw. those who wanted to participate could click on, or copy, a link to the draw survey and enter their email address at any time during a three-week period. researchers were only allowed access to the one survey to which they were assigned, which also ensured that results could not be connected to individuals.   results   we had 49 valid responses: 21 hslg members (response rate of 46% for the group), 21 other library association members, and 7 non-members of an association (including 6 former members). just two participants worked as an information professional for 0–5 years (4%), 12 (25%) worked 6–11 years, and 35 (71%) worked 12 or more years. as this article focuses on the development of our survey, we primarily present results that highlight issues of importance to design.   to learn about current financial supports, we asked if respondents’ organizations or libraries paid towards membership or attendance at cpd events and courses. considerably more of them paid towards cpd than membership (figure 1).   figure 1 payment towards membership and events by organizations or libraries. (from survey questions 4 & 5.)   themes   having developed new question options, it is usual to look for assurance that these are appropriate and comprehensive. our four themes were useful in setting and analysing two core questions. we asked participants for up to three reasons for their membership, or non-membership, of a library association, then we asked them to rate the importance of 20 options related to membership. asking the open question first allowed participants to provide answers that occurred to them instinctively (before viewing researcher-defined choices).   forty respondents provided one or more reasons why they were a member of a library association (table 2). for non-members, six respondents gave at least one reason why they were not a member. it is difficult to draw conclusions from the small number of responses; however, there appears to be a feeling of disconnect among some of those who are not members of a library association. they were also more unsure of the benefits (see appendix b for responses).   table 2 number of reasons for or against library association membership by theme a theme members (n=40) non-members (n=6) community 47 2 expertise 42 1 profession 23 2 support 3 2 a respondents could give up to three reasons. (from survey questions 14 & 21.)   figure 2 provides results on the importance of membership factors for association-member respondents, coded by theme. all five options for the theme community were in the top half of results and the five options for support were in the lower half. in figure 3, results from non-members show the themes are spread more evenly. again, note the low number of respondents, which restricts our ability to use statistical analysis and to generalize results. figure 2 the importance of factors in terms of membership in a library association. all members, n=40.  (from survey question 17.)   figure 3 the importance of factors in terms of membership of a library association. non-members, n=6. (from survey question 21.)   these results show that there is consistency in responses across our two core questions. for both members and non-members, the reasons for and against membership mirror the subsequent responses for what is important, which provides some confidence in internal consistency for this aspect of the questionnaire. to further check for reliability, we can examine results by subgroup. we might expect more similarity among member subgroups compared to non-members.   looking at the importance of themes, dividing members into hslg members and non-hslg association members shows similarity, and these differ from non-members (figure 4). to visually compare the themes across groups we used the nhs survey programme partial credit scoring system that allows data relating to a question’s options to be summarized by a single number (care quality commission, 2015). the most positive answer option (very important) is scored as 10 and the least positive (not important) is 0. intermediate answer options are scored with intermediate values (somewhat important is scored 5). calculations are then made based on the number of responses. the method has been tested and enables organizational performance on a survey question to be summarized and, when required, compared across organizations.   figure 4 themes ranked by importance and by library association member-status. (from survey questions 17 & 21.)   although not tested for significance, a simple visual examination of results within and across the results of core questions show what we might expect from a reliable questionnaire. factor analysis and significance testing would be useful to confirm these findings. the option of “other” was very rarely used in the survey, which gives us some confidence that we didn’t exclude important options in our questions. this suggests reasonable content validity.   discussion   despite obtaining a relatively small number of responses, our questionnaire performed as expected and enabled us to meet our aim and objectives. we now have a much better understanding of what is important to members and can use this knowledge for planning. in particular, by identifying themes, we understand that our association group members want to be part of a community where they can engage with others as much as they want educational activity. we have already begun to develop a cpd framework that incorporates a more structured approach, but which also focuses on connecting and engaging members.   we learned that financial concerns were not particularly prevalent among members, though it would be interesting to know if this only applied to our respondents. one may speculate that those who take time to complete a survey are more invested and active than others. financial considerations may be more prevalent among those who do not belong to a library association. knowing that most respondents did not have financial support to join an association but did have support to attend events has implications for those deciding on costs. if this is true of the wider library and information service population, it would suggest the importance of keeping costs of joining associations low and recouping costs through events, which are more likely to be subsidized. keeping questions related to financial support is therefore recommended in follow-up surveys.   defining our target population   the language used in surveys is crucial as it determines how results can be interpreted. a challenge in this survey involved defining our population. there were three main cohorts of interest: those involved in library and information services based in ireland who were (1) hslg members, (2) other members of library associations, and (3) non-members (former association members or never joined).   membership in a library association is generally open to a range of people. in ireland, this includes those with or without a professional library qualification who are or have been employed in the field of librarianship; those enrolled on a course leading to a professional qualification in library and information studies; and those with an interest in the work, welfare, and progress of libraries, but who are not employed in the field (library association of ireland, 2012). similarly, the american library association (2021) allows a broad spectrum of membership, which is open to “individuals, organizations, and non-profits, and businesses interested in working together to change the world for the better through libraries and librarians.” and, in the uk professional association, cilip, individual membership is “open to everyone working in knowledge, information, data or librarianship” (cilip, n.d.-b); with those not working in these areas still eligible to join as non-practitioners (cilip, n.d.-d). most associations allow personal and organizational membership.   an openly available online survey needs to clearly describe eligibility to ensure you reach those who you want to include, that you avoid wasting the time of those who you want to exclude, and ultimately, that you get meaningful results. association members may be easily identified through membership lists, but identifying and targeting non-members is difficult. if repeating our survey, we would make significant changes to the language used in our introduction, our questions, and dissemination.   question 1 established the work or study status of respondents. although not intentional, use of the term “librarian / information specialist” in our introduction and in that question is likely to have made some eligible people feel excluded. there has been interest in finding a respectful and inclusive term for those who work in library settings who do not have an accredited professional qualification. “library staff” was the term preferred by respondents in a recent survey aiming to find an agreeable term for staff in non-librarian roles (schilperoort et al., 2021). however, it is difficult to find an encompassing title for those working outside traditional library settings. cilip (n.d.-a) believes that “what makes someone a professional is the knowledge, skills, attitude, behaviours and values that they bring to their work.” to acknowledge the wide-ranging roles and focus of the sector it seems advisable to avoid titles or labels in a survey.   in the future, we may define our population as all current members of our library association, and anyone else working, seeking work, retired from work, or studying for a qualification, in the library or information (knowledge, data) sector in ireland. although this excludes some non-members eligible to join associations, it does include the key groups primarily required for planning purposes. (see appendix c for revised survey instrument.)   the options for question 1 could be the following:   i am currently working in the library or information sector. i am currently seeking work in the library or information sector. i am currently studying on a course leading to a qualification in library or information studies. i have retired from work in the library or information sector. i am a member of a library association and have an interest in the work, welfare and progress of library and information services but have never been employed in the sector.   follow-up questions may be required to establish eligibility or for contextual analysis:   i am based in ireland.     y/n i am working or seeking work in a health-related setting or where health is a significant component of my work.     y/n i have a professional library or information qualification.     y/n (if yes, please specify)   each option needs to have a purpose. if results are going to be used for reporting and planning, then it is necessary to know the status of respondents. for example, the views of those working or seeking work in the sector may be prioritized when planning cpd and other events, and will provide the most meaningful data from non-members. knowing the views of students will be important for future planning and recruitment. for a baseline survey, one may also want to check that the needs and expectations of specific groups, such as those with and without professionally accredited qualifications, are similar. if so, future surveys can omit any distinction. if they provide significantly different responses, then this may have implications for service provision.   clear definitions and appropriate language should help attract those who want to participate in a survey. these are also important for meaningful analysis of responses. the purpose of the survey must guide decisions about who to include. for an openly available survey, which is required to capture non-member views, clear language around eligibility is especially important.   other lessons and limitations   an obvious limitation to the interpretation of our results is the small number of respondents. the use of membership lists by groups and associations for dissemination would enable calculation of response rates. however, using a broad definition for our eligible population and a survey openly promoted through several sources, means that it was not possible to calculate response rates for everyone. attracting participation of non-members would require a more structured approach; for example, contacting a sample of libraries and library schools. there are online listings of libraries by country and sector, such as the ifla (n.d.) library map of the world. although often incomplete, they may be used to increase reach. researchers must decide what is most important when reaching their goals: comprehensiveness (sensitivity) versus precision. narrower definitions and routes may enable more precise and calculable data but also limit the diversity of responses.   social media likes and retweets didn’t necessarily lead to participation, so this method of dissemination cannot be relied upon alone. tagging key groups and individuals and adding a picture may increase interactions, but ensuring eligible populations view individual communications, such as a tweet, is unpredictable. making the survey available for a longer period and sending the link directly to all association member lists should increase response rates.   although the idea of offering a reward for completion is attractive, the openness of social media communication means that it may attract those who are not eligible to take part. in our case, following a tweet that mentioned the draw, we received several (52) inappropriate responses which had to be removed. to ensure transparency, two researchers independently reviewed the spreadsheet of results and highlighted those deemed to be ineligible based on content of answers (such as repeated or inappropriate phrases). agreement was easily reached as the identified responses had been filled consecutively overnight. ensuring inclusion of only valid responses is potentially a problem for all publicly available online surveys. we would not include a reward in the future.   conclusion   our research, including literature review and survey, provides us with information on which to plan strategy. we believe that our questionnaire could be adapted, with relevant elements utilized by other groups and associations. it is important that governing bodies and committees remember that our purpose is to guide and implement activity on behalf of members. we therefore need to understand how well we are doing, and how we should progress, based on the views of members. we also need to understand why people in our profession do not join any association so we can remove barriers and ensure benefits are appropriate, warranted, and clear. six of the seven participants in our survey who weren’t current members were former members. if that represents a broader trend, then we also need to know why people leave their representative associations.   it is also useful for members, and potential members, to consider what they want and expect from their library association. a survey questionnaire can be a useful means of reminding respondents of the range of benefits that is available to them. above all, it should be an impetus for action. our updated questionnaire will be a suitable tool to evaluate how well we are meeting our members’ expectations and provide results that can act as a benchmark for progress. this valuable information will help us plan our activity, set goals, and maintain and grow membership. we are very grateful to those who took part in our survey as they have given us a clear direction and renewed purpose.   we have learned some useful lessons during the research process. key points:   ·        take time to define your population. members of associations are easy to identify, but non-members (including former members, those who never joined, and those who may join in the future—such as students) will provide constructive insight into the value of an organization. ·        use language that is appropriate and inclusive. some terms and titles may alienate potential contributors. it is important that those who you want to include know they are welcome to participate. a clear description of eligibility in the survey introduction and in dissemination channels is required. ·        dissemination requires planning. members can be reached directly through membership lists (enabling response rates to be calculated). reaching non-members requires a targeted approach, which may involve an openly available survey that is promoted through a range of methods including social media and mailing lists, but should include a structured sampling of places where non-members work or study. ·        the four themes identified through the literature and in this survey offer useful categories for assessment and planning. ·        decisions in relation to data collection tools should be based on what you want to achieve in the process.   acknowledgements   the author would like to thank all members of the hslg committee, niamh lucey (chair), linda halton, noreen mchugh, mairea nelson, and miriam williams, who were instrumental in the organization of this research. their work and dedication to evidence based decision making on behalf of those involved in health librarianship demonstrates how library association committees can bring about effective change for our profession.   references   agee, j., & lillard, l. (2005). a global view of library associations for students and new librarians. new library world, 106(11/12), 541–555. https://doi.org/10.1108/03074800510635026 ahmadian yazdi, f., & deshpande, n. j. (2013). evaluation of selected library associations’ web sites. aslib proceedings, 65(2), 92–108. https://doi.org/10.1108/00012531311313952 american library association. (2020, february 14). ala responds to financial situation [press release]. https://www.ala.org/news/press-releases/2020/02/ala-responds-financial-situation  ball, h. l. (2019). conducting online surveys. journal of human lactation, 35(3), 413–417. https://doi.org/10.1177/0890334419848734 bradley, f., dalby, a., & spencer, a. (2009). our space: professional development for new graduates and professionals in australia. ifla journal, 35(3), 232–242. https://doi.org/10.1177/0340035209346211 broady‐preston, j. (2006). cilip: a twenty‐first century association for the information profession? library management, 27(1/2), 48–65. https://doi.org/10.1108/01435120610647947 care quality commission. (2015). nhs patient survey programme: survey scoring method. https://www.cqc.org.uk/sites/default/files/20151125_nhspatientsurveys_scoring_methodology.pdf chase, s. (2019). why join a professional association? public libraries, 58(5), 8–11. chartered institute of library and information professionals. (n.d.-a). a definition of libraries, information and knowledge as a ‘profession.’ retrieved april 24, 2022, from https://www.cilip.org.uk/page/professionalismdefinition  chartered institute of library and information professionals. (n.d.-b). be part of your profession. retrieved april 24, 2022, from https://www.cilip.org.uk/page/joinasamember chartered institute of library and information professionals. (n.d.-c). we are cilip: 5 year action plan. https://cdn.ymaws.com/www.cilip.org.uk/resource/resmgr/cilip/campaigns/we_are_cilip/we_are_cilip_strategy_report.pdf chartered institute of library and information professionals. (n.d.-d). we are here for you. retrieved april 24, 2022, from https://www.cilip.org.uk/page/becomeamember chartered institute of library and information professionals. (n.d.-e). why become professionally registered? retrieved june 5, 2022, from https://www.cilip.org.uk/page/professionalregistration corcoran, m., & mcguinness, c. (2014). keeping ahead of the curve: academic librarians and continuing professional development in ireland. library management, 35(3), 175–198. https://doi.org/10.1108/lm-06-2013-0048 davidson, j. r., & middleton, c. a. (2006). networking, networking, networking: the role of professional association memberships in mentoring and retention of science librarians. science & technology libraries, 27(1–2), 203–224. https://doi.org/10.1300/j122v27n01_14 dimauro, v. (2011). using online communities in professional associations. information outlook, 15(4), 18–20. https://scholarworks.sjsu.edu/sla_io_2011/4/ dowling, m., & fiels, k. m. (2009). global roles of library associations. in i. abdullahi (ed.), global library and information science (pp. 564–577). k. g. saur. https://doi.org/10.1515/9783598441349.564   fink, a., & kosecoff, j. b. (1998). how to conduct surveys: a step by step guide (2nd ed). sage publications. frank, d. g. (1997). activity in professional associations: the positive difference in a librarian’s career. library trends, 46(2), 307–319. https://www.ideals.illinois.edu/handle/2142/8155 french, j. (2012). designing and using surveys as research and evaluation tools. journal of medical imaging and radiation sciences, 43(3), 187–192. https://doi.org/10.1016/j.jmir.2012.06.005 garrison, b., & cramer, s. m. (2021). what librarians say they want from their professional associations: a survey of business librarians. journal of business & finance librarianship, 26(1–2), 81–98. https://doi.org/10.1080/08963568.2020.1819746 granikov, v., hong, q. n., crist, e., & pluye, p. (2020). mixed methods research in library and information science: a methodological review. library & information science research, 42(1), 101003. https://doi.org/10.1016/j.lisr.2020.101003 henczel, s. (2014). the impact of library associations: preliminary findings of a qualitative study. performance measurement and metrics, 15(3), 122–144. https://doi.org/10.1108/pmm-07-2014-0025 henczel, s. (2016a). the impact of national library associations on their members, employing organisations and the profession. [doctoral dissertation, rmit university]. rmit university research repository. https://researchrepository.rmit.edu.au/esploro/outputs/9921863743101341 henczel, s. (2016b). understanding the impact of national library association membership: strengthening the profession for sustainability. qualitative and quantitative methods in libraries, 5(2), 277–285. http://www.qqml-journal.net/index.php/qqml/article/view/340 international federation of library associations and institutions. (n.d.). library map of the world. retrieved june 4, 2022, from https://librarymap.ifla.org/map international federation of library associations and institutions. (2016a). building strong library associations programme manual. https://repository.ifla.org/handle/123456789/311   international federation of library associations and institutions. (2016b). building strong library associations. annex 16: member and non-member survey–start of programme. https://repository.ifla.org/handle/123456789/311 joint, n. (2007). newly qualified librarians and their professional associations: uk and us comparisons: antaeus. library review, 56(9), 767–772. https://doi.org/10.1108/00242530710831202 kamm, s. (1997). to join or not to join: how librarians make membership decisions about their associations. library trends, 46(2), 295–306. https://www.ideals.illinois.edu/handle/2142/8149 lachance, j. r. (2006). learning, community give library and information associations a bright future. library management, 27(1/2), 6–13. https://doi.org/10.1108/01435120610647901 library and information association of new zealand aotearoa. (n.d.). professional registration. retrieved june 4, 2022, from https://lianza.org.nz/professional-development/professional-registration/ library association of ireland. (2012). memorandum and articles of association of cumann leabharlann na héireann (the library association of ireland). https://www.libraryassociation.ie/wp-content/uploads/2018/11/memorandum-and-articles-of-association-of-lai.pdf librarianship studies & information technology. (2020, july 11). library associations. https://www.librarianshipstudies.com/2020/07/library-associations.html lumpkin, j. (2016). #why membership? professional associations in the millennial age: a call to action through mentorship. ola quarterly, 21(3), 5–7. https://journals3.oregondigital.org/olaq/article/view/vol21_iss3_1/1813 meadows, k. a. (2003). so you want to do research? 5: questionnaire design. british journal of community nursing, 8(12), 562–570. https://doi.org/10.12968/bjcn.2003.8.12.11854 morrison, h. (2004). professional library & information associations should rise to the challenge of promoting open access and lead by example. library hi tech news, 21(4), 8–10. https://doi.org/10.1108/07419050410545861 national care experience programme. (n.d.). survey hub. retrieved march 13, 2022, from https://yourexperience.ie/survey-hub/ nayak, m. s. d. p., & narayan, k. a. (2019). strengths and weaknesses of online surveys. iosr journal of humanities and social sciences, 24(5). https://www.iosrjournals.org/iosr-jhss/papers/vol.%2024%20issue5/series-5/e2405053138.pdf sauceda, j. (2018). mla personnel characteristics, 2016: continuity, change, and concerns. notes: the quarterly journal of the music library association, 359–371. https://doi.org/10.7282/t30005gf schilperoort, h., quezada, a., & lezcano, f. (2021). words matter: interpretations and implications of “para” in paraprofessional. journal of the medical library association, 109(1), 13–22. https://doi.org/10.5195/jmla.2021.933 schwartz, a. (2016, october). why join a library association? librarysciencedegree.org. https://web.archive.org/web/20161107013244/https://librarysciencedegree.org/why-join-a-library-association/ spaulding, k., & maloney, a. (2017, june 18–20). the impact of professional associations on the careers of lis professionals [paper presentation]. special libraries association 2017 annual conference, phoenix, az, united states. https://www.sla.org/wp-content/uploads/2017/07/spauldingmaloneythe-impact-of-professional-associations-on-the-careers-of-lis-professionals.pdf streatfield, d., & markless, s. (2019). impact evaluation and ifla: evaluating the impact of three international capacity building initiatives. performance measurement and metrics, 20(2), 105–122. https://doi.org/10.1108/pmm-03-2019-0008 thomas, v. k., satpathi, c., & satpathi, j. n. (2010). emerging challenges in academic librarianship and role of library associations in professional updating. library management, 31(8/9), 594–609. https://doi.org/10.1108/01435121011093379 wise, m. (2012). participation in local library associations: the benefits to participants. pnla quarterly, 77(1), 50–56. https://digitalcommons.cwu.edu/libraryfac/12/ evidence based library and information practice editorial responsibilities      2018. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29424     editor-in-chief: lorie kloda   associate editor (evidence summaries): carol perryman   associate editor (research articles): ann medaille, lisl zach   associate editor (classics & reviews): christina wissinger   associate editor (commentaries, using evidence in practice): lorie kloda   communications officer (news): tatiana bryant   production editor: michelle dunaway   editorial intern: rachel hinrichs   editorial advisors: lindsay alcock, alison brettle, jonathan eldredge, denise koufogiannakis   copyeditors: richard hayman (lead copyeditor), georgianne bordner, julie evener, heather healy, julie james, jane morgan-daniel, stacey penney, elizabeth stregger, nikki tummon   indexing support: kate shore research article   evaluating bibliographic referencing tools for a polytechnic environment   gina brander reference & information services librarian saskatchewan polytechnic library regina, saskatchewan, canada email: gina.brander@saskpolytech.ca   erin langman nursing liaison librarian saskatchewan polytechnic library regina, saskatchewan, canada email: erin.langman@saskpolytech.ca tasha maddison oer & copyright librarian saskatchewan polytechnic library saskatoon, saskatchewan, canada email: maddisont@saskpolytech.ca   jennifer shrubsole learning & teaching librarian saskatchewan polytechnic library moose jaw, saskatchewan, canada email: jennifer.shrubsole@saskpolytech.ca   received: 15 aug. 2018                                                                  accepted: 24 mar. 2019      2019 brander, langman, maddison, and shrubsole. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   data availability:  brander, g., maddison, t., langman, e., & shrubsole, j. (2019). scoring instrument for reference tools. ual dataverse, v1. https://doi.org/10.7939/dvn/7pmkto/ex8vfm   doi: 10.18438/eblip29489     abstract   objective – this paper analyzes the design process for a toolkit for appraising emerging and established bibliographic reference generators and managers for a particular student population. others looking to adapt or draw from the toolkit to meet the needs of users at their own institutions will benefit from this exploration of how one team developed and streamlined the process of assessment.   methods – the authors implemented an extensive initial evaluation using a checklist and comprehensive rubric to review and select reference tools. this work was guided by a matrix of categories from marino (2012), bates (2015), and other literature. as the tools were assessed using the toolkit, the components of the toolkit were evaluated and revised. toolkit revisions were based on evaluators’ feedback and lessons learned during the testing process. results – fifty-three tools were screened using a checklist that reviewed features, including cost and referencing styles. eighteen tools were thoroughly evaluated using the comprehensive rubric by multiple researchers to minimize bias. from this secondary testing, tools were recommended for use within this environment. ultimately the process of creating an assessment toolkit allowed the researchers to develop a streamlined process for further testing. the toolkit includes a checklist to reduce the list of potential tools, a rubric for features, a rubric to evaluate qualitative criteria, and an instrument for scoring.   conclusion – user needs and the campus environment are critical considerations for the selection of reference tools. for this project, researchers developed a comprehensive rubric and testing procedure to ensure consistency and validity of data. the streamlined process in turn enabled library staff to provide evidence based recommendations for the most suitable manager or generator to meet the needs of individual programs.     introduction   saskatchewan polytechnic does not provide access to a subscription-based reference management tool. a task force of four librarians and two library technicians reviewed products for students requiring the use of a generator and/or manager. a key consideration was that training on the product and usability should not be prohibitive, as most students spend a limited amount of time in library instructional sessions. the review also looked at the availability of free basic features, accuracy, technical support, and storage options.   reference managers are robust tools that allow users to save bibliographic metadata and organize sources into folders and subfolders. users can create references for individual citations, or entire bibliographies. many reference managers also include the ability for multiple users to collaborate on a shared library. some managers include web browser plug-ins for adding citations and references quickly and easily, as well as word processing add-ons that allow users to search their citation library and automatically add citations and references. reference generators, on the other hand, are simple tools for creating references or bibliographies. they do not have the ability to save references beyond an individual session, and do not have the additional features of many reference managers. table 1 outlines the main differences between reference managers and generators in greater detail.   table 1 features of reference managers vs. generators features reference managers reference generators create in-text citations ✔ usually create bibliographies ✔ ✔ save and organize references ✔ ✘ collaborate with other users ✔ ✘ browser integration ✔ ✘ word processing add-on usually ✘ save pdfs or other files usually ✘ entirely web-based rarely ✔   the most likely users of reference management tools among saskatchewan polytechnic’s student population include:   ●        students working on a group project who wish to share resources, ●        students working on a capstone project with the expectation of compiling a large number of resources, and ●        students in degree programs with a high expectation of research and writing skills, such as bachelor of science in nursing.   the most likely users of reference generators include:   ●        students in shorter programs, and ●        students with assignments requiring few sources.   due to the range of student needs, both generators and management tools are equally valid in this environment. as a result, both were analyzed as part of this research project.   aims   this paper describes the process of developing a comprehensive rubric and selection process to identify the best reference tool(s) to meet the needs of students at saskatchewan polytechnic. the emphasis of this paper is on the process used to create and revise the assessment toolkit, rather than on the end results of the evaluation. others looking to adapt or draw from the toolkit to meet the needs of students at their own institutions will benefit from this analysis of how the team developed and streamlined the process of assessment.   context   saskatchewan polytechnic offers 12 advanced certificates, 37 certificates, 37 applied certificates, 52 diplomas, three degrees, two post-graduate certificates, and 24 apprenticeship programs (saskatchewan polytechnic, 2017). programs range from two months to four years, and represent most sectors of the economy.   the varying expectations of programs regarding research requirements and the accuracy of citations pose specific challenges for the library. many programs do not prepare students to engage in practice-based research or publishing, and instead focus on teaching students about the ethical use of information by correctly citing their sources. often, assignments for these projects require a low number of sources. since the focus for many programs is on the “why” rather than the “how” of citing sources, many students would benefit from a simple tool that generates citations and references, and which requires less than half an hour to set up and learn.   at the other end of the needs spectrum are the institution’s diploma and degree program students. these programs assign research-intensive, sometimes semester-long projects, which are best managed using tools that offer options for storage and collaboration. the institution’s focus on student participation in applied research also engenders the need for more robust tools.   since the length of programs and student needs vary substantially, the task force entered into the project with the understanding that more than one tool would be selected for recommendation, and that both generators and managers would be investigated. they were also aware that the polytechnic’s librarians do not necessarily have the opportunity to instruct students in the use of reference tools, since many programs have minimal interactions with the library outside of their orientations, and various programs are offered by distance with these students receiving no library instruction. qualitative criteria (i.e., criteria assessed based on individual experience) were thus considered important in determining whether a tool could be easily accessed, learned, and adopted with little support or previous experience.   literature review   it is well established that students have difficulty creating bibliographic references (blicblau, bruwer, & dini, 2016; stevens, 2016). both students and researchers express frustration with creating references and often find the process aggravating and tedious (antonijević & cahoy, 2014; stevens, 2016). evidence indicates that bibliographic referencing tools may alleviate these negative emotions (stevens, 2016).   recommending reference tools is a common task in libraries (childress, 2011). because many people will continue to use the same tool even if not fully satisfied with it, it is important that librarians give advice based on evidence and the users’ needs rather than personal preference (antonijević & cahoy, 2014; blicblau et al., 2016). when matching reference tools to users’ needs, librarians need to consider both management tools that store sources long-term, and generating tools that create a copyable reference without the need for long-term storage (childress, 2011).   relatively few researchers have evaluated reference tools; instead, most authors discuss situations and ways to apply the tools (childress, 2011; lorenzetti & ghali, 2013; stevens, 2016). while there is no common methodology for evaluating reference tools (tramullas et al., 2015), many analyses compare functions and features of the tools (homol, 2014; imperial college london library, 2017; universitätsbibliothek technische universität münchen, 2016). most of the evaluations are based on the needs of university students at the graduate level, or on professional researchers within a particular field (kratochvíl, 2016; lorenzetti & ghali, 2013). even if a user group is defined, most studies do not detail user needs as an initial step (tramullas, sánchez-casabón, & garrido-picazo, 2015).   unlike user groups discussed in the literature, students at polytechnics have a wide range of backgrounds, prior credentials, and work experience (berger, motte, & parkin, 2009; canadian information centre for international credentials and council of ministers of education canada, 2016; polytechnics canada, 2015). polytechnics offer trade and technological education, as well as adult education, health sciences, and business programs (canadian council on learning, 2010; saskatchewan polytechnic, 2017). within polytechnics, there is a focus on applied research within a specific industry (canadian council on learning, 2010). likewise, workplace information literacy that is contextualized to the program is important (bird, crumpton, ozan, & williams, 2012). workplace information literacy, unlike academic information literacy, has collective approaches to information, and supports the use of “noncanonical sources” (inskip, 2014; lloyd, 2011).   evaluations by researchers tend to focus on the literature within a particular field, often in the health sciences (gilmour & cobus-kuo, 2011; kratochvíl, 2016). when testing, researchers often use articles, with books and book chapters as other commonly tested formats (gilmour & cobus-kuo, 2011; homol, 2014; kratochvíl, 2016). testing referencing tools for the polytechnic environment should include articles and books, but also other formats important to, and commonly used within, polytechnics grey literature and web pages (imagine easy solutions & easybib.com, 2014; kelly, 2015; kratochvíl, 2016; marsolek, cooper, farrell, & kelly, 2018; oermann et al., 2008).   most evaluations in the literature use a checklist approach or a list of features. if qualitative comments are included, there is no definition of terms used, e.g. “good” ease of use without explaining what “good” means (universitätsbibliothek technische universität münchen, 2016, p. 15). one framework was found in marino’s (2012) “fore-cite: tactics for evaluating citation management tools.” marino asks the reader to consider the following:   ●        the environment (the tools available), ●        the user in your unique situation, ●        the purpose of the software, such as a generator or a manager, ●        system and browser requirements, ●        accessibility, ●        features important to your users, ●        vendor support available, and ●        true cost of ownership.   complementing marino is bates (2015) sections model for selection of educational media:   ●        students, ●        ease of use, ●        costs, ●        teaching functions, or if the design promotes learning, ●        interaction: of student with the technology, of student to instructor, and of student to student, ●        organisational issues, such as institutional support and barriers, ●        networking with others outside the course, and ●        security and privacy.   both marino and bates take student-centered approaches, making them excellent starting points for developing a research methodology.   methods   with no common testing methodology and a student population quite different from those studied in the literature, the team built a methodology from the ground up. guided by longsight (2013), the team decided to design a rubric that included outcomes and functions, as well as features. including the outcomes and functions in the rubric provided qualitative data on the way people interacted with each reference tool. for this reason, user experience was partially incorporated into the evaluation of the reference tools. a rubric would also improve the decision-making process, since multiple people made the final choices (asq, n.d.).   criteria were gathered from the marino (2012) and bates (2015) frameworks, and other research literature. notable lists of criteria were wikipedia (“comparison,” 2016), universitätsbibliothek technische universität münchen (2016), and the university of toronto libraries (2016). both librarians and faculty members teaching referencing styles provided input on essential tool features. while reviewers did not agree on all points, commonly indicated criteria by librarians and faculty became minimal requirements in the rubric. see appendix a for the final list of criteria.   without a pre-existing rubric, the team adapted a matrix for the evaluation of learning management systems. the longsight (2013) matrix, which shared many similar criteria with the bates and marino models, used a four-point rating scale (with four being the highest rank for a criterion). the researchers wanted to present the final results as a letter grade, and a four-point scale allowed easy calculation of both a percentage and a four-point grade average for conversion into a letter grade. to indicate the minimum requirement of each criterion, a shading system was used (see table 2). shaded areas indicate acceptable levels; lightly shaded areas represent barely acceptable rankings; and no shading indicates unacceptable levels.   evaluators highlighted the words in a cell to indicate their decision (see table 2). if evaluators had additional comments on a criterion and ranking, they were invited to add these to the relevant cell (see table 3). during decision-making, and especially if there were inconsistencies in rankings by reviewers, comments were considered in addition to the numerical values.   the rubric was not normed due to time constraints; however, at least two people tested each reference tool to increase reliability and reduce bias. rankings assigned by testers were fairly consistent, unless browser ad and popup blockers were active. during the testing process, the rubric was revised twice based on feedback from testers. the testing process itself was also refined and developed into an assessment toolkit:   1.       a checklist of basic features that are easy to measure (appendix c) 2.       a rubric capturing more complex features (appendix d) 3.       a rubric to evaluate qualitative criteria (appendix e) 4.       an instrument to score features, an excel file (brander, maddison, langman and shrubsole, 2019)   as the process was refined, the team moved from exclusively focusing on reference managers to also including generators.   based on a list from wikipedia (“comparison,” 2016) and other literature (beel, 2014; “bibliographic,” n.d.; g2 crowd, n.d.), an inventory of tools was compiled (see appendix b). for local reasons, some tools were excluded, such as mobile apps and microsoft word.   the final version of the process started with a checklist to reduce the number of potential tools (appendix c). this checklist included easily assessed “deal-breakers” such as cost, referencing style(s) available, local computing requirements, and type of installation (e.g., all software requiring a user to set up a server was eliminated). since a tool either did or did not have the criteria identified, a single person was adequate to complete the checklist. if there was any uncertainty as to whether a tool met the criterion, it moved to the next stage.   in the second stage, a rubric was used to examine additional features (see appendix d). each evaluator noted the type of device, operating system, and browser used for testing, but this information was only used if there were discrepancies between evaluations. except for the criterion about viewing on mobile devices, evaluators used a desktop or laptop during testing. evaluators used an automatic tool for input, such as a browser extension, for all but two sources. these two sources tested manual input options so as to explore all input methods. with ad and pop-up blockers disabled, evaluators’ results were consistent, so in future only one evaluator could complete this stage.   table 2 example of reviewer’s highlighting, with illustration of shading showing (1) unacceptable, (2) barely acceptable, and (3;4) acceptable levels criterion 1 2 3 4 privacy settings no policy nor statement on privacy privacy policy or statement exists. may involve third parties. privacy policy or statement exists. no third parties involved. server located outside canada privacy policy or statement exists. no third parties involved. server located in canada   table 3 evaluators’ ratings with comments privacy settings mendeley evaluator #2 2. privacy policy or statement exists. may involve third parties not clear where the server is located, but headquarters for elsevier is in the uk. third party information is controlled by individual user. can sign up through facebook, as well as connect scopus author information to social network. privacy settings zotero evaluator #1 2. privacy policy or statement exists. may involve third parties server located outside of canada, site contains links to third-party web sites, but they do not share information to third parties. an open source project, and apps are created by a third party.   during the second stage, a list of the formats supported by each tool was created and compared. the formats were then placed into categories, including books and articles (print), photographs and maps (images), video and music (multimedia), web pages and software (internet and computers), statues (legal), speeches (verbal), and other. online versions of formats were placed into the same category as the hard copy; for instance, ebooks were considered print-based, and streaming videos were placed into multimedia. within the tool, only the main categories of formats were considered, and not subdivisions between hard-copy and electronic versions.   the final stage (appendix e) tested experiential factors, such as ease of set-up. since a person’s previous experience impacted the evaluation, more than one person was needed for this stage. as with the previous stage, browser ad and pop-up blockers were disabled. two librarians and two technicians tested each tool. while the librarians had expertise in bibliographic styles and with reference tools, each technician had experience in one area, but not the other. the team did not include anyone who was inexperienced with both computing and citing. evaluators tested the full process during this stage.   results of the accuracy evaluations were not reliable, since reference tools were not retested after revision of the accuracy evaluation methodology. the instrument used to score the features of individual tools also underwent several revisions. the version that was current at the time of the publication of this article is available (brander, maddison, langman and shrubsole, 2019).   results   results of testing revealed that none of the reference managers available at the time of testing were outstanding choices for saskatchewan polytechnic. despite this temporary setback, the project enabled the team to identify the best tools currently available based on defined institutional and student needs. six tools (three reference managers and three generators) out of the original fifty-three were recommended as a result of testing: refme, zotero, mendeley, citefast, citation machine, and easybib.   given that none of the reference managers were identified as ideal choices for saskatchewan polytechnic, future assessments are likely as the reference tool terrain evolves and new tools become established. the third and final toolkit version (see appendices) was determined by the team to have achieved a better balance of simplification and detail than the first two versions. toolkit components include:   1.       a checklist of basic features that are easy to measure (appendix c) 2.       a rubric capturing more complex features 3.       a rubric to evaluate qualitative criteria (e.g.., one’s experience of the tool) 4.       an instrument to score features   the final toolkit, developed and revised throughout the process of testing, offers an efficient, adaptable, and evidence based method for future testing.   discussion   throughout the process of developing and revising the toolkit, the authors experienced a number of challenges. while some of the challenges were anticipated and unavoidable, others informed toolkit changes and established new evaluator expectations. the lessons learned throughout this process will improve future iterations of testing.   be adaptable   one of the key takeaways for the authors was the need to be flexible. fluid criteria such as institutional software updates, library platform changes, style guide edition and reference tool version updates, and business instability can complicate or prolong the testing process. the sale of the seemingly well-established product refme in the middle of testing illustrates the state of flux testers work within. testing does not occur in a vacuum, and unavoidable challenges may necessitate revisions or a return to the drawing board.   simplify where you can   the several toolkit versions tested ultimately led to a final set of documents that match efficiency with accuracy. key changes included the addition of a checklist, and modification to the number of rubric testers based on the nature of the information being evaluated.   the team determined that the addition of a checklist would speed up the overall process, since some criteria in the rubric did not require a scale to evaluate whether or not tools met basic requirements. the checklist allows a single person to quickly assess descriptive criteria and eliminate tools that do not meet the most basic requirements. using this checklist, the team was able to quickly reduce the number of tools that required in-depth assessment from 53 to 18. the rubric was retained to assess the remaining criteria, which required finer distinctions in evaluation.   the team also decided to split the original rubric into two separate rubrics to improve reliability. individual assessments of qualitative factors were found to vary substantially, since these are influenced by testers’ previous experiences with other citation management tools. for example, the reviewer ratings were found to range widely for the “self-efficacy” criterion, which considered the level of instruction or consultation required for initial setup and use of a tool. to address this inconsistency without requiring more than one tester to evaluate quantitative criteria for 18 tools, the final version of the toolkit placed qualitative criteria in a separate, shorter rubric. separating these criteria allowed two testers to evaluate and compare criteria that were open for interpretation, while requiring only one tester to evaluate yes/no criteria.   limitations   this iteration of testing did not compare the capabilities of standalone browser extensions and applications for mobile devices. investigations into the functionality of tools in mobile environments will gain more relevance as students’ dependence on handheld devices for computing continues to increase.   only a cursory evaluation of user experience and accessibility was performed. user testing would provide rich details about experience and is an additional testing method to be considered in the future. both usability and accessibility testing would require different methodologies than the ones applied in this project.   since reference tools often have issues with languages other than english (libson, 2018), english-language materials alone were tested. datasets were also not tested, despite their importance, due to numerous issues around citing this format (kelly, 2016).   lastly, while the review investigated available privacy policies, input from legal experts is needed for further development of these criteria.   conclusion   much was learned throughout this project as the rubric and testing procedures were refined. without an assessment of local user needs, there is no basis for informed decision making. it is critical, therefore, to understand the users’ needs within a particular institutional environment, and to adapt the toolkit accordingly.   not all criteria required testing by multiple reviewers and reducing the number of people involved expedited the process. descriptive criteria (e.g., cost or support available) yielded generally consistent results across reviewers and did not require multiple evaluators. using a checklist of essential criteria instead of a rubric also accelerated the process. on the other hand, multiple viewpoints were vital when considering experiential criteria (e.g., level of complexity of processes).   saskatchewan polytechnic library has ultimately benefited from the development and application of these assessment tools to help identify the most appropriate bibliographic referencing tools for the student population. the assessment tools have allowed the library to provide evidence based advice that can be tailored to the needs of individual users. as a result, library staff are better equipped to aid students with reference tools within a polytechnic setting.   as reference tools continue to evolve, there is a strong possibility that the toolkit will require revision. in future iterations, an emphasis should be also placed on user experience testing and evaluation of accessibility. for now, however, the toolkit provides a foundation for ongoing testing of reference tools at saskatchewan polytechnic. the toolkit also provides a starting place for others looking to draw from or adapt it to meet students’ needs at their institutions.   references   antonijević, s., & cahoy, e. s. (2014). personal library curation: an ethnographic study of scholars’ information practices. portal: libraries and the academy, 14(2), 287–306. https://doi.org/10.1353/pla.2014.0010   asq. (n.d.). decision matrix. retrieved from http://asq.org/learn-about-quality/decision-making-tools/overview/decision-matrix.html   bates, a. w. (2015). teaching in a digital age: guidelines for designing teaching and learning for a digital age. retrieved from https://opentextbc.ca/teachinginadigitalage   beel, j. (2014, january 15). comprehensive comparison of reference managers: mendeley vs. zotero vs. docear [web log post]. retrieved from http://www.docear.org/2014/01/15/comprehensive-comparison-of-reference-managers-mendeley-vs-zotero-vs-docear   berger, j., motte, a., & parkin, a. (eds.). (2009). the price of knowledge: access and student finance in canada (4th ed.). montreal, qc: the canada millennium scholarship foundation. retrieved from http://www.yorku.ca/pathways/literature/access/the%20price%20of%20knowledge%202009.pdf   bibliographic listings (n.d.). retrieved from http://dirtdirectory.org/   bird, n. j., crumpton, m., ozan, m., & williams, t. (2012). workplace information literacy: a neglected priority for community college libraries. journal of business & finance librarianship, 17(1), 18–33. https://doi.org/10.1080/08963568.2012.630593   blicblau, a. s., bruwer, m., & dini, k. (2016). do engineering students perceive that different learning and teaching modes improve their referencing and citation skills? international journal of mechanical engineering education, 44(1), 3–15. https://doi.org/10.1177/0306419015624186   brander, g., maddison, t., langman, e., & shrubsole, j. (2019). scoring instrument for reference tools. ual dataverse, v1. https://doi.org/10.7939/dvn/7pmkto/ex8vfm   canadian council on learning. (2010). challenges in canadian post-secondary education: navigating post-secondary education in canada: the challenge of a changing landscape. retrieved from eric database (ed525096).canadian information centre for international credentials, and council of ministers of education, canada. (2016). canada’s education systems. retrieved from https://www.cicic.ca/docs/pteducation/canada-s-education-systems-pdf.pdf   childress, d. (2011). citation tools in academic libraries: best practices for reference and instruction. reference & user services quarterly, 51(2), 143–152. https://doi.org/10.5860/rusq.51n2.143   comparison of reference management software. (2016, december 2). in wikipedia. retrieved from https://en.wikipedia.org/wiki/comparison_of_reference_management_software   g2 crowd. (n.d.). best reference management software. retrieved from https://www.g2crowd.com/categories/reference-management   gilmour, r., & cobus-kuo, l. (2011). reference management software: a comparative analysis of four products. issues in science and technology librarianship, 66. https://doi.org/10.5062/f4z60kzf   homol, l. (2014). web-based citation management tools: comparing the accuracy of their electronic journal citations. the journal of academic librarianship, 40(6), 552–557. https://doi.org/10.1016/j.acalib.2014.09.011   imagine easy solutions, & easybib.com. (2014). perspectives on student research skills in k-12 and academic communities. retrieved from http://info.easybib.com/hs-fs/hub/222136/file-1990871677-pdf/infolitreport-part2.pdf?t=1414690132007   imperial college london library. (2017). reference management software comparison. retrieved from http://www.imperial.ac.uk/media/imperial-college/administration-and-support-services/library/public/reference-management-software-comparison.pdf   inskip, c. (2014, june). information literacy is for life, not just for a good degree: a literature review. retrieved from http://discovery.ucl.ac.uk/1448073/   kelly, m. (2015). citation patterns of engineering, statistics, and computer science researchers: an internal and external citation analysis across multiple engineering subfields. college & research libraries, 76(7), 859–882. https://doi.org/10.5860/crl.76.7.859   kratochvíl, j. (2017). comparison of the accuracy of bibliographical references generated for medical citation styles by endnote, mendeley, refworks and zotero. the journal of academic librarianship, 43(1), 57–66. https://doi.org/10.1016/j.acalib.2016.09.001   lisbon, a. h. (2018). multilingual scholarship: non-english sources and reference management software. the journal of academic librarianship, 44(1), 60–65. https://doi.org/10.1016/j.acalib.2017.12.001   lloyd, a. (2011). trapped between a rock and a hard place: what counts as information literacy in the workplace and how is it conceptualized? library trends, 60(2), 277–296. https://doi.org/10.1353/lib.2011.0046   longsight. (2013). criteria for the evaluation of learning management systems. retrieved from https://www.ncat.edu/provost/docs/lms/lms%20evaluation%20tool.pdf   lorenzetti, d. l., & ghali, w. a. (2013). reference management software for systematic reviews and meta-analyses: an exploration of usage and usability. bmc medical research methodology, 13. https://doi.org/10.1186/1471-2288-13-141   marino, w. (2012). fore‐cite: tactics for evaluating citation management tools. reference services review, 40(2), 295–310. https://doi.org/10.1108/00907321211228336   marsolek, w., cooper, k., farrell, s., & kelly, j. (2018). the types, frequencies, and findability of disciplinary grey literature within prominent subject databases and academic institutional repositories. journal of librarianship and scholarly communication, 6(1). https://doi.org/10.7710/2162-3309.2200   oermann, m. h., nordstrom, c. k., wilmes, n. a., denison, d., webb, s. a., featherston, d. e., … striz, p. (2008). information sources for developing the nursing literature. international journal of nursing studies, 45(4), 580–587. https://doi.org/10.1016/j.ijnurstu.2006.10.005   polytechnics canada. (2015). what is a polytechnic. retrieved from http://www.polytechnicscanada.ca/polytechnic-advantage/what-polytechnic   saskatchewan polytechnic. (2017). impact that matters: 2016-2017 annual report. retrieved from https://saskpolytech.ca/about/about-us/reports-and-statistics/annualreport/documents/2016-17%20sask%20polytech%20annual%20report.pdf   stevens, c. r. (2016). citation generators, owl, and the persistence of error-ridden references: an assessment for learning approach to citation errors. the journal of academic librarianship, 42(6), 712–718. https://doi.org/10.1016/j.acalib.2016.07.003   tramullas, j., sánchez-casabón, a. i., & garrido-picazo, p. (2015). studies and analysis of reference management software: a literature review. el profesional de la información, 24(5), 680–688. https://doi.org/10.3145/epi.2015.sep.17   universitätsbibliothek technische universität münchen. (2016). reference management software comparison 6th update (june 2016). retrieved from https://mediatum.ub.tum.de/doc/1320978/1320978.pdf   university of toronto libraries, gerstein science information centre. (2016). what’s the best citation management software for me? retrieved from http://guides.library.utoronto.ca/c.php?g=250610&p=1671260     appendix a list of evaluative criteria   students (including access) ease of use costs teaching (referencing) functions ●     integration with word processors ●     mobile devices (e.g. app) ●     portability ●     syncing   ●     ease of set-up ●     design and layout ●     ease of adding information ●     auto-fill options ●     ease of editing info ●     signaling (e.g. date format, incomplete info) ●     intuitiveness of making references ●     intuitiveness of making citations ●     self-efficacy ●     free or freemium ●     if freemium, extent of use available at no cost ●     if freemium, cost of storage for free ●     accuracy* ●     citation style(s) ●     content stored (e.g. metadata only) ●     formats (e.g. journal articles) ●     generic form for any format ●     importing and exporting files (e.g. ris) ●     pdf compatibility ●     storage available interaction (including with learning materials) organisational (institutional) issues networking (externally, e.g. social media) security and privacy ●     collaborating and sharing ●     manual entry available ●     organization and discovery ●     presence of ads   ●     authentication ●     browser(s) ●     installation requirements ●     operating system(s) ●     support available ●     training materials available [nothing in this category] ●     business stability ●     data security ●     privacy settings   * accuracy includes correct information, correct presentation (e.g. punctuation, capitalization), correct format of source, no empty fields, no fields missing, and variance in accuracy based on method of addition (e.g. browser tool vs. database search within tool).   based on bates (2015)     appendix b tools examined, by category   reference generators ●        apa style wizard* ●        bibme*ǂ ●        citation machine *ǂ ●        citation producer ●        citefast*ǂ ●        citemaker*ǂ ●        citethisforme*ǂ ●        classtools ●        easybib*ǂ ●        google scholar*ǂ ●        knightcite*ǂ ●        make citation ●        ncsu citation builder ●        noodle tools express*ǂ ●        ottobib   reference managers ●        aigaion ●        bebop ●        bibbase ●        bibdesk ●        biblioscape ●        bibserver ●        bibsonomy ●        bibus ●        bookends ●        citavi ●        citelighter ●        citeulike* ●        colwiz* ●        docear ●        endnote ●        endnote basic* ●        ewwoww (wizfolio)* ●        f1000workspace ●        jabref ●        kbibtex ●        mendeley* ●        noodle tools ●        paperpile ●        papers ●        pybliographer ●        qiqqa ●        readcube ●        refbase ●        refdb ●        refme* ●        reference manager ●        referencer ●        refworks ●        sciref ●        sente ●        wikindx zotero*   * tool examined using rubrics. ǂ tested as generator; manager option available.     appendix c step 1: checklist of initial criteria   highlight the features that each tool has. you may highlight more than one option in a category. repeat rows as necessary.   name of tool cost (exclude trial period) type of tool (free version) operating systems citation styles installation   free freemium (free version, pay to upgrade) fee to use generator (cannot save) manager (save references)   windows mac apa (6th ed.) cse chicago mla (8th ed.) ieee desktop (application) desktop (server set up) web-based (online account) browser tool     appendix d step 2: expanded features of reference tools   tool tested: _____________________________________________________   name of reviewer: _____________________________________________________   thank you for helping us determine which generator(s) are best for our students. your input will be very important to making a decision.   instructions ●        use the list of sources attached in the email: o    when manually adding sources, please ignore special information for some items. this is for automatic entry, which may pull this information even if not used. o    please add information by an automatic method (e.g. bookmarklet), except where indicated. ▪          if more than one way of adding information automatically, please try all methods. o    after making notes about the automatic method, please ‘fix’ information before making the reference.   ●        please highlight text that best describes your experience under each criterion. o    if the criterion does not apply to your tool, please skip that line. o    if you need to highlight two squares, please explain why both squares apply to the tool.   ●        there are areas to note observations. o    please feel free to add extra observations, as you feel are needed.   ●        if you test two different citation styles, please fill out two forms (the tool may act differently for each style).   ●        please copy and paste the bibliography into a word document. o    upload bibliography to [] with your initials, tool name, and citation format in filename. o    we will be noting how accurate the tool is.   ●        please upload a copy of this form, with your initials and name of tool in the filename, to […]. ●          please check options, and highlight text or make notes as needed.   style tested   apa   mla: 8th edition   ieee (note version, if applicable)   how i am testing device type make & model (e.g. samsung s7) operating system version   desktop/laptop   chromebook linux mac windows     tablet   android blackberry ios windows     phone   android blackberry ios windows     browsers supported browser full or partial support? version tested chrome     edge     firefox     internet explorer     safari     other:       general overview legend: clear = unacceptable; light grey = barely acceptable; dark grey = acceptable   criterion setting up (generator) setting up (manager) storage (manager) collaborating and sharing (manager) 4 no account and no software installation required to use tool. to use, you set up online personal account. no software installation necessary (may have optional browser tool). no storage limit in free version. can co-create and share bibliography, and can control privacy of bibliography (e.g. can share publicly or privately). 3 no account to use and perform basic functions. additional functions require software installation. to use, you must install a browser tool. syncs with online personal account. free storage limited by # of references (>100), or free storage limited by file size (>100 mb). option to buy additional storage. can co-create bibliography, and can share references. can only share references publicly (except with co-creator). 2 no account to use, but some functions require software installation 5 steps. to use, you must install software. a browser tool is optional. syncs with online personal account. free storage limited by # of references (<100), or free storage limited by file size (<100mb). option to buy additional storage. can co-create bibliography, but can only share with co-creator. 1 before using, you must create an account and/or install software to use, you must install software and a browser tool. syncs with online personal account. must pay to store references (e.g. free generates only); or small free storage (<100 references, <100mb total) without option to buy additional storage. cannot co-create bibliography and cannot share references.     criterion content stored (manager) organization and discovery (manager) ads (disable ad blockers) viewing on mobile devices app 4 stores metadata, and can add files and notes. can annotate pdf files. you can search or filter sources, and use folders or tagging. search includes full-text (e.g. pdfs). no ads. scales to mobile screen. full functionality available in mobile version. app developed by same entity as the original tool. cost is under $5 and available for android and ios. 3 stores metadata, and can add files and notes. cannot annotate pdf files. you can search or filter sources, and use folders or tagging. search does not include full-text (e.g. pdfs). contains ads. ads are static and/or videos that do not play automatically, and are a minority of the interface. scales to mobile screen. basic functionality available, but not all features, in mobile version. app developed by third party (i.e. not by developer of original tool. cost is under $5 and available for android and ios. 2 stores metadata, and can add files or notes, but not both. you can search or filter sources, but folders or tagging requires payment. contains ads. ads are static and/or videos that do not play automatically. ads are a majority of the interface. does not scale to mobile screen. functionality not impaired by lack of mobile version. app over $5 or not available for android and ios. 1 stores only metadata (info about sources). one method is available: searching, filtering, folders, or tagging. contains ads. ads are obtrusive (video that plays automatically, pop-up, animated gif). does not scale to mobile screen. function impaired by lack of mobile version. no app available.   support and business operations   criterion support training materials online help resources privacy settings data security (manager) business stability 4 support available both live and by email. live support available 24/7. print training materials available, with extensive collection of videos/tutorials available. contextually-appropriate help files provide assistance as appropriate.  pop-ups or rollovers provide “just-in-time” information for specific actions. privacy policy or statement exists. no third parties involved. server located in canada. backs up data. allows data retrieval if business suspends operations. will send notification if operations suspended. business has some maturity, and has evidence of continuing investment in product development. 3 support available both live and by email. live support has limited hours. print training materials available, with some videos/tutorials available. help files are accessible at each step of a process. privacy policy or statement exists. no third parties involved. server located outside canada. backs up data. allows data retrieval if business suspends operations. no notification if operations suspended. business has some maturity, but it is unclear if there is continuing investment in product development. 2 email support only. print training materials available, no videos/tutorials available. a users’ manual or space (e.g. blog, user group) is accessible online. privacy policy or statement exists. may involve third parties. backs up data. no policy on data retrieval in event of suspension of business operations. business is a recent start-up that has received attention. 1 no support available, or support only available for a cost. no training materials. no online help resources. no policy nor statement on privacy. unknown information about backing up. no policy on data retrieval in event of suspension of business operations. business is a high-risk organization (e.g. past financial trouble, recent start-up without ‘buzz’).     adding information   criterion method of adding info missing/ required info autofill from searching in tool browser tool (e.g. bookmarklet) importing sources 4 manual entry available. includes multiple automatic methods (database, bookmarklet or other browser tool). warns about missing info. required info indicated. autofill from searching internal database. search has simple interface. can usually find information on sources. browser tool is intuitive, and retrieves metadata fairly accurately. can add import multiple common files types (e.g. ris, bibtex). 3 manual entry available, plus one automatic method (database, bookmarklet or other browser tool). no warning about missing info. required info indicated. autofill from searching internal database. search has simple interface, but usually does not find sources. browser tool is intuitive, but makes frequent errors. can add import files, but limited to one common file type (e.g. ris). 2 only manual entry available. warns about missing info. required info not indicated. autofill from searching internal database. search/database has complicated interface. browser tool is not intuitive to use. can add import files, but not a common file type. 1 only automatic entry available (i.e. no manual entry). no warning about missing info. required info not indicated. no autofill from searching tool’s internal database. no browser tool. cannot import sources.     criterion format of author name corporate authors date format generic form for any format pdfs 4 order of author’s name is obvious. it is obvious how to include multiple authors. corporate author option is obvious, and easy to use. date format does not matter (i.e. tool changes automatically). includes generic format, and considers usability of presentation of options (e.g. ‘chunks’ options). full importing of metadata. 3 order of author’s name is obvious. dealing with multiple authors not obvious. corporate author option is obvious, but difficult to use. date format is obvious. includes generic format, but does not consider usability of presentation of options. limited importing of metadata. importing does not require software or extra steps. 2 clues for order of author’s name are not obvious. corporate author option is hard to find. clues for date format are not obvious. includes generic format, but has very few fields available. limited importing of metadata. importing requires software or extra steps. 1 no clue for order of author’s name. no option for corporate author. no clue for date format. format matters. no option available. no support available – manual entry.     making references and citations   criterion bibliographic references citations word processors order of references indentation (apa, mla) capitalization 4 can make full bibliographic list. bibliography included within full document (i.e. write paper in tool). full citation information provided in-context within full document. can export in various formats, or have plug-ins for various word processors, including ms word. follows style order, and is accurate in order. hanging indentation kept in ms word, and no extra formatting required. changes capitalization to citation style. there are no capitalization errors. 3 can make full bibliographic list. provides full citation information. page number may be prompt only. can export in ms word format, or have ms word plug-in. support for windows and mac. follows style order, but makes errors (e.g. alphabetize the). hanging indentation kept in ms word, but other formatting needed. changes capitalization to citation style, but makes multiple errors. 2 can only make individual references. provides basic citation information (i.e. no page number). can export in ms word format, or have ms word plug-in. support only for windows. does not follow style’s requirements for reference order. hanging indentation in tool, but formatting disappears in ms word. changes capitalization to citation style, but makes 1 error (e.g. does not capitalize subtitle). 1 no references output. no citation output. no special word processor support (e.g. paste text version). not applicable (e.g. does not create list of references). no hanging indentation. keeps capitalization style of input. exporting sources   options for exporting sources (e.g. ris, ms word, rtf).     export individual sources only? yes / no   types of sources supported (e.g. reports, journal articles):   any additional comments that may affect decision:   estimated time to complete evaluation:     appendix e step 3: experiential criteria of reference tools   tool tested: _____________________________________________________   name of reviewer: _____________________________________________________   thank you for helping us determine which generator(s) are best for our students. your input will be very important to making a decision.   instructions ●        use the list of sources attached in the email: o    when manually adding sources, please ignore special information for some items. this is for automatic entry, which may pull this information even if not used. o    please add information by an automatic method (e.g. bookmarklet), except where indicated. ▪          if more than one way of adding information automatically, please try all methods. o    after making notes about the automatic method, please ‘fix’ information before making the reference.   ●        please highlight text that best describes your experience under each criterion. o    if the criterion does not apply to your tool, please skip that line. o    if you need to highlight two squares, make sure to explain why.   ●        there are areas to note observations. o    please feel free to add extra observations, as you feel are needed.   ●        if you test two different citation styles, please fill out two forms (the tool may act differently for each style).   ●        please copy and paste the bibliography into a word document. o    upload word bibliography to [] with your initials, tool name, and citation format in filename. o    we will be noting how accurate the tool is.   ●        please upload a copy of this form, with your initials and name of tool in the filename, to [].   please check options, and highlight text or make notes as needed.   style tested   apa   mla: 8th edition   ieee (note version, if applicable)   how i am testing device type make & model (e.g. samsung s7) operating system version   desktop/laptop   chromebook linux mac windows     tablet   android blackberry ios windows     phone   android blackberry ios windows           browser type version   chrome     edge     firefox     internet explorer     safari     other: name version       using the tool legend: clear = unacceptable; light grey = barely acceptable; dark grey = acceptable   criterion setting up/ starting design and layout adding information (automatically) adding information (manually) editing sources 4 process is clear and has under 5 steps. intuitive interface that can be navigated with little or no training. look is simple and straightforward. process is clear, and does not contain extra screens/windows. process is clear, and does not contain extra screens/windows. easy to change information. few changes needed. 3 process is clear, but has over 5 steps. functional interface that can be navigated with minimal training. look is fairly simple and straightforward. process is clear, but contains extra screens/windows. difficult to find option for format. cannot add all required information. easy to change information. many changes needed. 2 process may be confusing to new users, but has under 5 steps. interface is functional, but some features may be complex and/or confusing. process may be confusing to new users, but does not contain extra screens/windows. easy to find option for format. can add all required information. difficult to change information. few changes needed. 1 process is complicated and requires over 5 steps. complex and potentially confusing. process is complicated, and contains extra screens/windows. difficult to find option for format. cannot add all required information. difficult to change information. many changes needed.   criterion making references self-efficacy 4 process simple and clear. output is clearly identified. most users should be able to set up with help materials alone. minimal time required to maintain skills. 3 process simple and clear, but output may not be clear to new users. most users should be able to set up with help materials alone. requires regular use to maintain skills. 2 process simple, but may be confusing to new users. for most users, requires instruction or consultation for set-up. requires regular use to maintain skills. 1 requires extra clicks during process. process could be simplified. for most users, requires instruction or consultation for set-up. significant time required to maintain skills.     what did you like best about this tool?   what did you find most frustrating about this tool?   any additional comments that may affect decision   estimated time to complete evaluation:     evidence based library and information practice evidence based library and information practice 2010, 5.2  evidence based library and information practice       classic    gratification theory provides a useful framework for understanding the information  seeking behaviours and needs of distinct populations    a review of:  chatman, e. a. (1991). life in a small world: applicability of gratification theory to information‐ seeking behavior. journal of the american society for information science, 42(6), 438‐449.    reviewed by:  kate zoellner  education, human sciences, & psychology librarian   maureen and mike mansfield library, the university of montana  missoula, montana, united states of america  email: kate.zoellner@umontana.edu    received : 10 nov. 2010         accepted : 05 may 2010       2010 zoellner. this is an open access article distributed under the terms of the creative commons‐ attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐ sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original  work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is  redistributed under the same or similar license to this one.    abstract    objective – apply gratification theory to the  information‐seeking behaviours and use of  information by a lower working class  population.      design – an ethnographic study framed by  social stratification literature was utilized to  explore, describe and interpret the everyday  information needs, information‐seeking  behaviours and views of information held by  lower working class individuals.    setting – a major university in the southeast  united states, specifically the physical plant  facilities including classrooms, bathrooms,  janitorial closets, and front steps.    subjects – the participants were 52 lower  working class janitorial staff at a major  university. the majority of subjects were  single black women in their late 30s. the  women had children and were the heads of  their households. the women had not  completed high school and earned minimum  wage; they had been at their jobs for an  average of seven years. the workers’  supervisors, and others at the physical plant,  were also contacted as part of the study.    methods – ethnographic data was collected  over a two‐year period, 1984‐86, through  participation in the setting and interviews. a  28‐item interview guide was used to identify  participants’ job‐search strategies, use of mass  media, television viewing behaviours, and  107 mailto:kate.zoellner@umontana.edu evidence based library and information practice 2010, 5.2  acceptance of information from individuals  and believable sources of information.    main results – chatman confirmed the  usefulness of gratification theory as a  conceptual framework to identify what  defines information problems, motivations,  and information seeking behaviours for an  impoverished population. the results support  the findings of social stratification research on  the parallel between impoverished  individuals’ social life and their orientation  toward gratification. a focus on local present  reality due to pressing economic and  psychological problems orients lower working  class individuals toward immediate  gratification. thus, information sources of  value to the participants were those readily  accessible and easy to use in the moment of  need.    the six theoretical propositions of gratification  theory chatman identified through her  literature review were applied and confirmed  in her analysis of the information behaviours  of janitorial workers:  1. life in a small world. lower working class  individuals have a local worldview and  therefore have limited exposure to job  opportunities compared to other populations.  the majority of chatman’s subjects found out  about their current job through friends and  family employed at the university (51%), or  neighbours employed there (11%).  2. lower expectations and the belief in luck.  individuals of the population have lower  expectations of their success and therefore do  not actively pursue new opportunities; success  is seen as a result of luck. janitorial workers in  the study felt they were lucky to have found  their jobs and that their chances of finding a  better position were slight and based on  “knowing someone” (p. 444).  3. first‐level lifestyle. members of the  population rely on information from members  of their social circles. study participants  sought everyday information from family,  friends, neighbours, local newspapers, and  television. information was considered reliable  in their view if it aligned with their personal  experience, was presented by multiple people,  or if the person sharing the information was  perceived as trustworthy.  4. limited‐time horizon. lower working class  populations experience a time immediacy and  limitation different from those of the middle  and upper classes. study participants  imagined their future job positions and  lifestyles as similar to the ones they currently  held, due to perceptions that opportunities  were not open to them or worth the effort to  pursue. some exceptions were the possibility  of pursuing higher education and having  more leisure time in the future.   5. an “insider” worldview. the worldview of  an insider is focused “on the practical  dimensions of life” (p. 445); information  relevant to lower working class individuals is  that which “solve[s] problematic situations”  (p. 441). study participants’ social  conversations revolved around events that  reinforced their mental models. respondents  relied on themselves and distrusted those  outside their social circles.  6. use of the mass media. mass media is  perceived as a vehicle for passing time, escape,  and entertainment, as well as a reflection of  lived realities for the population. respondents  indicated that they watched television to pass  time, and, secondarily, for practical purposes  (e.g., learning how to be safe).    conclusion – gratification theory provides a  useful framework for library and information  professionals to identify how populations  define information problems and reliable  sources, and their information seeking  behaviours and motivations. chatman’s  analysis indicates that the everyday problems  faced by the lower working class are not, and  will not be, met by traditional sources that  information professionals assume to be of  value for the population. based on these  research results, chatman calls on information  professionals to critically evaluate and  broaden their understanding of how problems  are defined and addressed by the specific  populations they seek to serve – to consider  the relationships between the pressing realities  of their service populations and everyday  information that addresses those realities. this  understanding will enable information  108 evidence based library and information practice 2010, 5.2  professionals to determine if, how, and by  what means, they should develop and  package information to meet the needs of their  service populations and communities.      commentary    chatman’s article confirms the applicability of  gratification theory for understanding the  information‐seeking behaviours of lower  working class populations. the use of an  ethnographic study framed by social  stratification theory was appropriate to  address chatman’s research questions  (creswell, 1998). yet, there are several  shortcomings in the study documentation,  leaving scholars with questions and unable to  replicate the study.    chatman indicated a rationale for the selection  of her subjects and a profile of their  characteristics. she stated that the janitorial  workers “typified in many ways the lower  working class” (p. 442). further discussion of  the rationale for selecting janitorial workers, or  the specific participants of the staff, is not  provided, nor is a definition of the lower  working class for the purpose of the study.    chatman indicated that a 28‐item interview  guide was used in the study. in one instance  she uses the term survey but it is unclear if  this was a separate tool from the interview  guide. while select interview questions are  included in the article, the full instrument is  not. there is neither mention of how the  instrument was developed, tested, or  administered, nor when the interviews took  place during the two‐year time period of the  study. the reader is not told how the  interviews were documented, for example, if  responses were recorded and transcribed, or  dictated from notes or memory.    given the qualitative nature of the study,  numbers and percentages are only used in the  article to represent the number of study  participants. quotations from the janitorial  workers are included in the article. for  example, chatman cites one participant: “’i  made three job trips down here before i got  the job. the one who gave me this job told me  i was lucky. i got here just in time ‘cause they  had only two openings.’” (p. 443). it is not  clear if this, or other, quotations in the article  emerged from chatman’s participant  observations, from a formal interview she  conducted, or via other means.    in addition to missing definitions and  information on the instrument, data  documentation, and the study chronology,  chatman’s article lacks an explanation of how  data was analyzed. scholars are left  questioning how the interviews,  conversations, and observations that  comprised the study came in most cases to  parallel the behaviours and ways of life of the  lower working class as detailed in the social  stratification literature on gratification theory.  it is unknown if chatman coded her  observations and interviews to reveal themes,  traditionally part of a grounded theory study  (creswell, 1998), or if she interpreted her  findings based on the six propositions of  gratification theory identified in her literature  review.    chatman’s work responded to and aligned  with a paradigm shift in information needs  and uses research – from researchers viewing  information as objective and users as “input‐ output processors of information” to a view of  “information as something constructed by  human beings” (dervin & nilan, 1986, p. 16).  while the reliability and validity of chatman’s  study cannot be confirmed due to the absence  of data collection and data analysis  information, her study has been influential.  results from a cited reference search in the isi  web of knowledge database indicate  chatman’s article has been cited 54 times since  its publication in studies of populations  ranging from maori secondary school students  to clergy and canadian immigrants.    interest in the ways in which individuals  conceptualize and seek information in both  everyday life and research‐driven situations is  present in the work of scholars today, and  their research continues to be framed by well‐ 109 evidence based library and information practice 2010, 5.2  110 established theories and to employ qualitative  methodologies. for example, sadler and given  (2007) apply affordance theory from ecological  psychology to the information behaviours of  graduate students in an academic library,  using qualitative interviews, resulting in  recommendations for improving  communication between graduate students  and librarians as well as for the library’s  information literacy instruction. chatman’s  study serves as a critical reminder that in  order to meet the information needs of service  populations, librarians and information  professionals need to first research and  understand the ways in which the social  context of those populations affects their  information definitions, needs, and  behaviours.              references    chatman, e. a. (1991). life in a small world:  applicability of gratification theory to  information‐seeking behavior. journal  of the american society for information  science, 42(6), 438‐449.    creswell, j. w. (1998). qualitative inquiry and  research design: choosing among five  traditions. thousand oaks, ca: sage  publications.    dervin, b., & nilan, m. (1986). information  needs and uses. annual review of  information science and technology, 21,  3‐33.    sadler, e., & given, l. m. (2007). affordance  theory: a framework for graduate  students’ information behavior.  journal of documentation, 63(1), 115‐ 141.  research article   assessing the impact of reference assistance and library instruction on retention and grades using student tracking technology   dennis krieb director, institutional research and library services reid memorial library lewis & clark community college godfrey, illinois, united states of america email: dkrieb@lc.edu   received: 1 jan. 2018                                                                       accepted: 12 apr. 2018        2018 krieb. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29402     abstract   objective – to assess the impact of community college academic librarians upon student retention and grades through reference desk visits and attendance in library instruction classes.     methods – student id data used for this research was collected from students that visited the reference desk to consult about a course-related question or attended a library instruction class for a specific course. after consenting to share their student id number, the students’ ids were scanned and uploaded to a blackboard analytics data warehouse. a pyramid analytics reporting tool was used to query and extract student-level retention and grade data based upon whether the student had visited the reference desk or attended a library instruction class. chi-square and fisher’s exact tests were used to discern any statistical difference in retention rates and grades between students that engaged a librarian through reference or instruction and the general student population.   results – when comparing fall-to-fall retention for all degree-seeking students, students that visited the reference desk or attended a library instruction class had a statistically higher rate of retention. when comparing fall-to-fall retention within low-retention student cohorts, students that visited the reference desk or attended a library instruction class had higher rates of retention. rates of retention in 8 of 10 cohorts were statistically higher for library instruction and in 6 of 10 cohorts were statistically higher for reference visits. with respect to course grades, only one of five high enrollment courses showed a higher grade average for students that attended a library instruction class. none of the differences in average grades between students that attended a library instruction class and all students in the five courses were statistically significant. for the impact of a reference visit upon a course grade, all five courses showed a higher average grade average for students that visited the reference desk for a question related to their course than for all students in the course. four of the five differences were statistically significant.   conclusions – the data collected by systematically tracking students that interact with community college librarians suggests that reference desk visits and attendance of library instruction classes both have a positive statistically significant impact upon student retention. when looking at course grades, the data does not indicate a statistically significant positive or negative impact for library instruction. the impact of visiting the reference desk upon course grades does suggest a strong statistically significant positive correlation.         introduction   lewis & clark community college is a two-year higher education institution located in godfrey, illinois. lewis & clark has multiple campuses, a river research center, a humanities center, a training center, and community education centers located throughout the more than 220,000-person college district that reaches into 7 counties in southwestern illinois. unduplicated, degree-seeking enrollment for academic year 2016-2017 was 7,673 students.   the confluence of reductions in state-level funding and declining student enrollments has generated a sense of urgency upon student retention efforts at lewis & clark community college. in the years from 2006 to 2011, fall-to-fall retention for full-time students dropped from 57% to 52%, and from 42% to 39% for part-time students. these data mirror the low retention rates of all two-year community colleges, where nearly 50% of students leave by the end of their first year of enrollment (hongwei, 2015). within this challenging environment, there began a new emphasis by state-level education agencies and higher education accreditors for evidence based initiatives supporting student success.   to address the demand for more evidence of success, a new approach to leverage data was decided upon by administrators in academic affairs, enrollment services, and institutional research. a campus culture would be cultivated that relied heavily upon quantitative student assessment of innovative practices using predefined measures of success. this approach would also explore student tracking of support services on campus as a means to better understand the impact of these services upon student success measures.   in 2012, the student success team was established to address success initiatives related to grades and retention. members of the student success team included senior level academic administrators and members of the institutional research department. the student success team would act as an academic think tank to investigate, pilot, and assess trends in higher education associated with evidence based practices to improve student success.   the student integration model developed by vincent tinto suggests that supportive social and educational communities outside of the classroom have a positive impact upon student retention (tinto, 2012). it was upon this theoretical framework that the student success team began to investigate the impact of student support services at lewis & clark upon grades and retention.   the first student support service selected by the student success team to investigate was academic tutoring. branded as the student success center, tutoring at lewis & clark is decentralized among various campus locations. reid library also hosts a student success center location that provides assistance for students seeking tutoring in writing and study skills. in 2013, students that were tutored at any student success center location were tracked to discern the impact of tutoring upon retention. the fall-to-fall retention rate for degree-seeking students enrolled in fall 2013 that were tutored was found to be 65.6% (n=640), as compared to the overall retention rate for all degree-seeking students of 51.5% (n=5085).   the student success team decided to expand the research of tinto’s student integration model to reid library in 2014. this decision was supported by research connecting the services and collections of academic libraries to tinto’s student integration model (oakleaf, 2010). correlational evidence linking student retention and academic success with academic libraries published by the university of minnesota (soria, fransen, & nackerud, 2013) was also instrumental in the student success team’s decision to investigate the impact of reid library upon student grades and retention.                              another aspect of the student success team work would be its emphasis on evidence based research using lewis & clark’s technology infrastructure. a data warehouse had recently been implemented, providing the ability to quickly identify calculated success measures such as grades and retention for specific student cohorts. a list of ten student cohorts with retention rates below the overall student retention rate would be used to assess the impact of tinto’s student integration model within reid library.     table 1 student demographics lewis & clark community college, fall 2017 student type percentage white 80.2% african-american 9.8% hispanic 2.3% female 60.5% 18-19 years of age 36.1% 20-24 years of age 33.2% first generation college 24.7% developmental math or english placement 53.0% accepted pell grant 39.2% veteran 3.7%     literature review   there exists a crisis in retention and completion that is unique to community colleges. approximately 51% of all entering community college students will have dropped out within their first year (national student clearinghouse research center, 2016), and only 20% seeking to transfer to a four-year institution will eventually do so (lloyd & eckhardt, 2010). for minority and part-time students, retention and completion are often lower when compared to other community college students (strayhorn, 2012).   in his classic work leaving college, vincent tinto (2012) suggests that student attrition in postsecondary education is related to a student’s immersion within the greater campus community. “institutions of higher education are not unlike other human communities, and the process of educational departure is not substantially different from the other processes of leaving which occur among human communities generally” (p. 204). student support services help foster a campus community of belonging by creating social relationships, clarifying aspirations and enhancing commitment, developing college know-how, and making college life feasible (karp, 2011).   research applying tinto’s theoretical framework of student integration to various student support services has yielded positive correlational relationships between these services and student retention. derby (2006) discovered a significant positive relationship between student club participation and retention and degree completion. the research of grillo and leist (2013) with student academic tutoring also discovered a positive relationship between tutoring and student retention and completion.   research seeking to assess the academic library as a factor for increasing student success metrics such as retention and completion is still a relatively new field of study. early studies often looked at aggregated data sets to discern any correlational relationships between library input and output measures with institutional retention and completion rates. mezick’s (2007) study found a positive relationship between total library expenditures and student retention for postsecondary institutions offering a baccalaureate degree. a positive relationship between library professional staff and student retention was also found in the research of emmons and wilkinson (2011) when analyzing data sets taken from the 2005-2006 annual survey of arl statistics.   with the recent introduction of predictive and learning analytics within higher education, institutions are now seeking more nuanced data to forecast student behaviour to proactively engage students to improve student success measures (lourens & bleazard, 2016). for academic libraries, this new emphasis upon predictive and learning analytics represents a need to rethink how data is collected and how librarians can connect academic library outcomes to institutional outcomes such as retention and graduation (oakleaf, 2010).   one of the first major studies looking into student-level interactions with academic library services and collections was conducted at the university of minnesota. this research involved collecting student-level data from students that interacted with or used a library service or collection and connecting these data to the students’ subsequent enrollment and grade point averages. findings from this research suggested that first-year students that used the library had a higher grade point average and fall-to-spring retention rate than their peers that did not use the library (soria, fransen, & nackerud, 2013). an additional study at the university of minnesota discovered that first-year students that used electronic resources and books had higher odds of graduating over withdrawing (soria, fransen, & nackerud, 2013).   the research shared in this paper applies the student-level approach to tracking student engagement with the library, much like the work published by the university of minnesota. it is hoped that the findings in this research will add to the literature regarding predictive analytics within academic libraries, the technology infrastructure needed to systematically track students that use the library, and the impact of library services – specifically, reference desk encounters and library instruction classes – upon retention and grades.   methods   as previously mentioned in the literature review section, the research of soria, fransen, and nackerud (2013) at the university of minnesota was one of the first published articles to apply student-level tracking data from an academic library to investigate the impact of librarians, services, and collections upon student success measures. this seminal research served as the model for establishing the methods and measures for this paper. by applying the methodology used in the university of minnesota study, a comparison of findings can be made between a two-year community college and major research university.   independent and dependent variables   in 2014, reid library began systematically tracking student use in the library. two library service areas would serve as independent variables: 1) attendance of a library instruction class and 2) visiting the reference desk for assistance. a threshold was established that only reference questions associated with an enrolled course would be tracked; directional and other non-course related questions would not be measured for their impact upon retention and grades. there were two dependent variables used in this research: 1) fall-to-fall retention and 2) student grades for five courses having the highest association with reference desk questions and library instruction. all students in this research had a degree-seeking status.    data collection   a fundamental component of lewis & clark’s evidence based research is its technology infrastructure. central to this architecture is the blackboard analytics data warehouse. the data warehouse serves as a data repository, housing data tables related to student characteristics, enrollments, grades, and completions from the ellucian student information system (sis). the pyramid analytics reporting tool provides the ability to query the data warehouse for calculated student success measures based upon treatments or services the student may have received.   the technology infrastructure showing how student tracking data is merged with student data retrieved from the sis is depicted in figure 1. after a library instruction class or reference visit was completed, the librarian asked for permission from the student to track his or her attendance or visit. this is commonly known as a verbal informed consent. the librarian explained to the student that no content-level information would be recorded, only that they have either attended a library instruction class or visited the reference desk. the student was informed that the data would only be used for research purposes – including the possibility of sharing publicly – to better understand student success metrics, and that no personally identifiable information would ever be shared. since the inception of this pilot in 2014, no student has ever declined to be tracked.       figure 1 technology infrastructure for tracking reference desk visits and library instruction attendance.       if the student agreed to share his or her student id and course information, the librarian used a barcode scanner or manually entered the student id in the tracking software. after entering the student id, a list of the student’s currently enrolled courses was provided by the tracking software.   the librarian then selected the appropriate course with which the library instruction class or reference visit was associated. the tracking software platform used for this initial phase is called sars trak.    after scanning the student id and selecting the associated course, these data were sent to a blackboard analytics data warehouse. student data regarding grades, enrollment, demographics, and other student-level data from the sis were merged with the tracking data imported from sars trak within the data warehouse. a pyramid analytics reporting tool was then used to query the data warehouse for calculated student success measures based upon whether the student had visited the reference desk or attended a library instruction course.   results   tables 2 and 3 compare the fall-to-fall retention rate for all degree-seeking students for academic years 2014/15-2016/17 with the fall-to-fall retention rates of degree-seeking students that attended a library instruction class or visited the reference desk for the same time period. students that attended a library instruction class had a fall-to-fall retention rate of 60.9% (n=1,304), which was higher than the overall retention rate of 48.8% (n=7,319) for all degree-seeking students. students that visited the reference desk had a retention rate of 66.2% (n=215).   to discern the impact of library instruction and reference assistance for students having characteristics associated with lower retention rates, ten student cohorts were identified as having lower retention rates than the retention rate of 48.8% (n=7,319) for all degree-seeking students.   table 4 shows the overall retention rate for each student cohort and the retention rate for the students within each cohort that attended a library instruction class or visited the reference desk. all 10 low retention student cohorts had a higher rate of retention when attending a library instruction class or visiting the reference desk, with 8 cohorts having a statistically significant difference for library instruction and 6 cohorts statistically higher for reference.   tables 5 and 6 compare the course grade success rates with library instruction and reference desk visits. success rates are defined at lewis & clark as a grade of a, b, or c, and failure is a grade of d, f, or w.    the impact of library instruction on grades was minimal, with only one of the five courses having a higher success rate than the overall course success rate. courses selected in table 4 had the highest association of requiring attendance of a library instruction class as part of the course. there was no statistically significant difference in any of the success rates for the five courses.   courses selected in table 6 had the highest association with a reference question relevant to the course. unlike library instruction, students in all five courses that visited the reference desk had a higher success rate than the overall course success rate. four of the five courses had a statistically significant higher success rate for those students that visited the reference desk for assistance with their coursework.   discussion   testing tinto’s student integration model in the context of librarian interactions with students has provided lewis & clark community college with correlational evidence that relationships developed with college personnel outside of the classroom are impactful for student success. with respect to the two independent library variables tested in this research, both library instruction and reference assistance were shown to have a positive statistically significant correlational relationship with student retention. the correlational relationship between library instruction and grades was not established in this research; however, the data did reveal a positive statistically significant correlation between reference assistance and grades.       table 2 fall-to-fall retention rate for all degree-seeking students, academic years 2014/15-2016/17   n fall-to-fall retention rate all degree-seeking students   7,319   48.8% n represents a distinct student count.     table 3 fall-to-fall retention rates for students attending a library instruction class or visiting the reference desk, academic years 2014/15-2016/17 n fall-to-fall retention rate attended a library instruction class   1,304   60.9%** visited the reference desk   215   66.2%** n represents a distinct student count. *   p<.05 ** p<.01     table 4 fall-to-fall retention rates for student cohorts with low retention rates that attended library instruction or visited the reference desk, academic years 2014/15-2016/17     low retention cohort     n retention rate-        all degree-seeking students     n retention rate attended library instruction     n retention rate visited the reference desk cumulative gpa below 2.0 1,508 29.6% 120 34.1% 29 48.3%* africanamerican 787 38.0% 150 46.4%* 37 59.5%** cumulative gpa 2.0 2.29 3,509 40.2% 129 51.9%** 16 68.8% male 3,292 44.7% 543 57.7%** 70 65.7%** part-time 4,963 45.1% 589 52.9%** 102 63.1%** age 20-24 2,569 45.4% 323 54.9%** 46 56.5% first generation 2,164 45.4% 217 54.4%** 49 62.0% developmental english placement 327 45.6% 60 55.0% 16 58.8% pell accepted 3,287 46.6% 660 54.1%** 123 59.7%** developmental math placement 2,337 47.0% 426 58.1%** 79 67.1%** n represents a distinct student count. *   p<.05 ** p<.01     table 5 course success rates for the highest courses associated with a library instruction class, academic years 2014/15-2016/17     n success rate all degree-seeking students     n success rate attended library instruction second semester college english   1,568 70.8% 440 74.2% first semester college english 2,054 71.4% 396 69.6% public and private communication 1,692 80.3% 282 80.1% public speaking 732 81.8% 127 78.9% college reading (developmental) 399 80.3% 93 80.2% n represents an enrolled course count.  *   p<.05 ** p<.01     table 6 course success rates for the highest courses associated with a reference visit, academic years 2014/15-2016/17     n success rate all degree-seeking students     n success rate visited the reference desk english 131 2,054 71.4% 102 85.3%** english 132 1,568 70.8% 99 85.9%** reading 125 399 80.3% 32 96.9%* psychology 131 1,626 63.8% 31 77.4% art 130 597 72.7% 30 90.0%* n represents an enrolled course count. *   p<.05 ** p<.01       in comparison to similar studies that tracked student use of academic libraries to retention and grades, soria, fransen, and nackerud’s (2013) research at the university of minnesota serves as the best example of a study using similar methods when comparing library instruction and reference visits, though it should be noted that the university of minnesota is a selective admissions institution, unlike lewis & clark community college, which is an open admissions institution.   when comparing the impact of library instruction upon grades, findings from the university of minnesota were similar to those discovered at lewis & clark, with both showing a modest positive impact, though neither study found the impact to be statistically significant. with respect to reference visits, the university of minnesota showed a positive correlation with grades, but without statistical significance. findings from lewis & clark showed statistically significant higher grades in four of the five courses measured for students that visited the reference desk.   retention comparisons for both studies found positive statistically significant correlations between library instruction and student retention. reference visits in the university of minnesota study showed a slightly negative relationship with retention, though the data were not statistically significant. reference visits at lewis & clark were found to be positively correlated with retention for all 10 cohorts studied, with 6 cohorts having a statistically higher retention rate.   it is interesting to note that a reference visit had a more significant impact upon students at lewis & clark than for the students in the university of minnesota study. because the setting of a reference visit is a one-on-one encounter, an opportunity exists for the student to establish a relationship with the librarian. for community college students at lewis & clark, 53% of which are not prepared for college-level math or english courses, the need to develop relationships with college personnel outside of the classroom may be more impactful than for those students at a selective admissions university like the university of minnesota.   another observation taken from the findings is the higher impact of a reference visit in comparison to attending a library instruction class for both grades and retention at lewis & clark. though a relationship may be developed between a student and a librarian that teaches a library instruction class of 20 or 30 students, the likelihood of this occurrence is smaller than the opportunity for a student to develop a relationship with a librarian through a reference visit. findings from this research would suggest that student proximity to a librarian is correlated with grades and retention.   in spite of the positive findings discovered in this study, there are limitations. students that visited the reference desk in this study represent a self-selected sample. these students may be more academically motivated to achieve higher grades and graduate than their classroom peers that did not visit the reference desk. future research into the impact of reference visits upon grades and retention should consider propensity score matching of students to reduce the potential for bias associated with student motivation.    another limitation of this study is the presumption that all reference desk visits are equally weighted. the length of time spent during a reference desk visit may also have a correlational relationship with grades and retention. future research should consider grouping reference desk visits by the length of the interview.   conclusion   building on the findings of this research, lewis & clark community college has expanded its tracking of students that interact with support services and co-curricular activities to over 20 points of service, including tutoring, advising, and participation in student life clubs and activities. the library has also expanded its data collection by tracking students that check out items from the collection. as with the current tracking system being used for library instruction and reference assistance, student ids will be used to identify those students that circulate an item from the library. data assessing the correlational relationship between student use of the library’s non-digital collection with grades and retention will be available in the fall semester of 2018. moving forward, a campaign to proactively share the findings of this research with faculty, students, and administrators at lewis & clark is currently being planned in hopes of increasing overall student usage of the library.   incorporating the library as a data resource for institutional research has been a goal for the author of this paper. as a result of the research presented in this article, the library has become a partner with peer divisions and departments on campus with retention initiatives. a recent example is lewis & clark’s requirement for accreditation to complete a four-year quality initiative to improve retention for the higher learning commission. library findings associated with the research in this paper will serve as a data source in the quality initiative that seeks to explore tinto’s student integration theory through student tracking of support services. quality initiative findings for lewis & clark will be presented in 2020 at the higher learning commission’s persistence and completion academy results forum.   it should also be noted that the long-standing legacy of library patron privacy has not been compromised in this research. no personally identifiable information has been disclosed for any student tracked. all data are secured within the institution’s ellucian student information system and are only accessible by the institutional research office at lewis & clark community college.   references   derby, d. c. (2006). student involvement in clubs and organizations: an exploratory study at a community college. journal of applied research in the community college, 14(1), 39-45.   emmons, m., & wilkinson, f. c. (2011). the academic library impact on student persistence. college & research libraries, 72(2), 128-149. https://doi.org/10.5860/crl-74r1   grillo, m. c., & leist, c. w. (2013). academic support as a predictor of retention to graduation: new insights on the role of tutoring, learning assistance, and supplemental instruction. journal of college student retention: research, theory & practice, 15(3), 387-408. http://dx.doi.org/10.2190/cs.15.3.e   hongwei, y. (2015). student retention at two-year community colleges: a structural equation modeling approach. international journal of continuing education & lifelong learning, 8(1), 85-101.   karp, m. m. (2011). how non-academic supports work: four mechanisms for improving student outcomes. ccrc brief. number 54. retrieved from https://ccrc.tc.columbia.edu/media/k2/attachments/how-non-academic-supports-work-brief.pdf lloyd, p. m., & eckhardt, r. a. (2010). strategies for improving retention of community college students in the sciences. science educator, 19(1), 33-41. retrieved from https://files.eric.ed.gov/fulltext/ej874152.pdf lourens, a., & bleazard, d. (2016). applying predictive analytics in identifying students at risk: a case study. south african journal of higher education, 30(2), 129-142. http://dx.doi.org/10.20853/30-2-583   mezick, e. m. (2007). return on investment: libraries and student retention. journal of academic librarianship, 33(5), 561-566. https://doi.org/10.1016/j.acalib.2007.05.002 national student clearinghouse research center. (2016). the role of community colleges in postsecondary success: community colleges outcomes report. retrieved from https://studentclearinghouse.info/onestop/wp-content/uploads/comm-colleges-outcomes-report.pdf oakleaf, m. (2010). the value of academic libraries: a comprehensive research review and report. chicago: association of college and research libraries. retrieved from http://www.ala.org/acrl/sites/ala.org.acrl/files/content/issues/value/val_report.pdf soria, k. m., fransen, j., & nackerud, s. (2013). library use and undergraduate student outcomes: new evidence for students’ retention and academic success. portal: libraries    and the academy, 13(2), 147-164. https://doi.org/10.1353/pla.2013.0010   soria, k. m., fransen, j., & nackerud, s. (2017). the impact of academic library resources on undergraduates' degree completion. college & research libraries, 78(6), 812-823. https://doi.org/10.5860/crl.78.6.812   strayhorn, t. l. (2012). satisfaction and retention among african american men at two-year community colleges. community college journal of research and practice, 36(5), 358-375.  https://doi.org/10.1080/10668920902782508   tinto, v. (2012). leaving college: rethinking the causes and cures of student attrition (2nd ed.). chicago, il: university of chicago press.     evidence summary   recent american library school graduate disciplinary backgrounds are predominantly english and history   a review of: clarke, r. i., & kim, y.-i. (2018). the more things change, the more they stay the same: educational and disciplinary backgrounds of american librarians, 1950-2015. school of information studies: faculty scholarship, 178. https://surface.syr.edu/istpub/178     reviewed by: heather macdonald health and biosciences librarian macodrum library carleton university ottawa, ontario, canada email: heather.macdonald@carleton.ca   received: 1 feb. 2019                                                                     accepted: 18 apr. 2019      2019 macdonald. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29550     abstract   objective – to determine the educational and disciplinary backgrounds of recent library school graduates and compare them to librarians of the past and to the general population.   design – cross-sectional.   setting – 7 library schools in north america.   subjects – 3,191 students and their 4,380 associated degrees.   methods – data was solicited from every ala-accredited master of library science (mls) program in the united states of america, canada, and puerto rico on students enrolled between 2012-2016 about their undergraduate and graduate degrees and areas of study. data was coded and summarized quantitatively. undergraduate degree data were recoded and compared to the undergraduate degree areas of study for the college-educated american population for 2012-2015 using the ipeds classification of instructional programs taxonomic scheme. data were compared to previous studies investigating librarian disciplinary backgrounds.   main results – 12% of schools provided data. recent north american library school graduates have undergraduate and graduate degrees with disciplinary backgrounds in humanities (41%), social sciences (22%), professions (17%), science, technology, engineering and math (stem) (11%), arts (6%), and miscellaneous/interdisciplinary (3%). of the humanities, english (14.68%) and history (10.43%) predominate. comparing undergraduate degrees with the college-educated american population using the integrated postsecondary education data system (ipeds) classification schema, recent library school graduates have a higher percentage of degrees in social sciences and history (21.37% vs. 9.24%), english language and literature/letters (20.33% vs. 2.65%), computer and information science (6.54% vs. 2.96%), and foreign languages, literatures, and linguistics (6.25% vs. 1.1%). compared to librarians in the past, there has been a decline in recent library school graduates with english language and literature/letters, education, biological and physical sciences, and library science undergraduate degrees. there has been an increase in visual and performing arts undergraduate degrees in recent library school graduates.    conclusion – english and history disciplinary backgrounds still predominate in recent library school graduates. this could pose problems for library school students unfamiliar with social science methodologies, both in school and later when doing evidence-based practice in the work place. the disciplinary backgrounds of recent library school graduates were very different from the college-educated american population. an increase in librarians with stem backgrounds may help serve a need for stem support and provide more diverse perspectives. more recent library school graduates have an arts disciplinary background than was seen in previous generations. the creativity and innovation skills that an arts background provides could be an important skill in librarianship.   commentary   this summary uses the cat generic critical appraisal tool (perryman & rathbun-grubb, 2014) to assess the article. the authors of the article clearly stated their study objective and provided a good overview of the literature. in the article, the authors used a longitudinal comparison to highlight three studies, which found that humanities disciplinary backgrounds predominated among mls graduates.   the authors chose to collect data sets from institutions rather than conducting a survey in an effort to have a “more thorough and representative set of data” (methods section). the authors noted that limitations of the study included a small sample size, and although the response rate was not high, this collection method is a strength. of those schools that participated, their entire population was included rather than a selective sample of individual responses. the authors stated that the included schools varied in size, structure, and ranking. including these data would have been useful to compare to non-participating schools to determine if it was a representative sample.   the authors extensively detailed their process for coding data. first, the authors presented the coding of data to compare to the general population. table 2 in the study presented the comparison clearly and concisely. next, the authors recoded the data to allow a comparison with previous papers investigating mls student disciplinary backgrounds. the authors discussed the challenges of this analysis. the presentation of the analysis in figure 2 was awkward to read as not all totals equaled 100%; however, the authors addressed this in the text.   in the “implications and conclusions” section, the authors stated that students with humanities backgrounds struggle to understand expectations because of their unfamiliarity with research methodologies and genre conventions in an mls program. this might be true, but the authors do not present evidence to support this statement. checking the curricula of the participating schools for required research methods courses could provide evidence to support or disprove this statement.   this study is methodologically rigorous. the authors provided an excellent description of their data collection, coding, and analysis. this article will be valuable to researchers doing analyses that require building crosswalks between data sets. overall, the paper points to the continuing skewed educational background of mls graduates. as the authors noted, mls programs should do more to recruit students from stem disciplines. current stem librarians can help promote this by connecting more with their students. each interaction is an opportunity to demonstrate and promote librarianship to stem students, maybe even encouraging some of these students to become librarians.   reference   perryman, c. & rathbun-grubb, s. (2014). the cat: a generic critical appraisal tool. retrieved from http://www.jotform.us/cp1757/thecat   evidence summary   dewey decimal classification trending downward in u.s. academic libraries, but unlikely to disappear completely   a review of: lund, b., & agbaji, d. (2018). use of dewey decimal classification by academic libraries in the united states. cataloging and classification quarterly, 56(7), 653-661. https://doi.org/10.1080/01639374.2018.1517851     reviewed by: jordan patterson cataloguing & metadata librarian memorial university libraries st. john's, newfoundland and labrador, canada email: jpatterson13@mun.ca   received: 11 june 2019                                                                  accepted:  14 july 2019      2019 patterson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29592     abstract   objective – to determine the current use of dewey decimal classification in academic libraries in the united states of america (u.s.).   design – cross-sectional survey using a systematic sampling method.   setting – online academic library catalogues in the u.s.   subjects – 3,973 academic library catalogues.   methods – the researchers identified 3,973 academic libraries affiliated with degree-granting post-secondary institutions in the u.s. the researchers searched each library’s online catalogue for 10 terms from a predetermined list. from the results of each search, the researchers selected at least five titles, noted the classification scheme used to classify each title, and coded the library as using dewey decimal classification (ddc), library of congress classification (lcc), both ddc and lcc, or other classification schemes.   based on the results of their data collection, the researchers calculated totals. the totals of this current study’s data collection were compared to statistics on ddc usage from two previous reports, one published in 1975 and one in 1996. the researchers performed statistical analyses to determine if there were any discernible trends from the earliest reported statistics through to the current study.   main results – collections classified using ddc were present in 717 libraries (18.9%). adjusting for the increase in the number of academic libraries in the u.s. between 1975 and 2017, ddc usage in academic libraries has declined by 56% in that time frame. the number of libraries with only ddc in evidence is unreported.   conclusion – the previous four decades have seen a significant decrease in the use of ddc in u.s. academic libraries in favour of lcc; however, the rate at which ddc has disappeared from academic libraries has slowed dramatically since the 1960s. there is no clear indication that ddc will disappear from academic libraries completely.   commentary   the superiority of classification systems is a topic of perennial debate in library circles, and though opinions abound, there have been surprisingly few recent empirical studies on the subject. shorten, seikel, and ahrberg (2005) asked why some academic libraries have persisted in their use of ddc, while lund and agbaji (2018), the authors of this study, had previously investigated the preference for either ddc or lcc among academic library employees. statistics on the usage of ddc in academic libraries were reported in 1975 and 1996, and this study, undertaken in 2017, closes another 21-year gap in its sampling of u.s. academic libraries. this study helpfully confirmed the trend indicated by the previous two reports: ddc has a diminishing presence in academic libraries.   measured with glynn’s (2006) critical appraisal checklist, this study had some shortcomings. the study would benefit from a more thorough reporting of the results of data collection. the researchers assessed library catalogues for the presence of “ddc, lcc, both, or another system”; yet, despite the four possible categorizations, the authors presented the results in binary form. dewey collections either were or were not present, which cannot convey the nuance of the full data set. for instance, a library using only ddc and a library employing primarily lcc with only their children’s collection in ddc were both counted equally as having ddc collections present, despite the great difference in practical implementation of the classification systems. the number of libraries wherein ddc was found to be the only scheme in evidence was an unreported statistic which would have been a key indicator of ddc’s retreat from academic libraries.   this study also did not maintain a clear distinction between a library’s use of ddc, which was the point of inquiry in the primary research question, and the presence of ddc collections in library catalogues, which was what the collected data measured. while there is little question of the diminished state of ddc in academic libraries, presence of ddc collections does not necessarily equate to an active policy of classifying materials with ddc, which the authors acknowledged, noting that what constitutes “use” varies from library to library. this variable definition resulted in imprecise conclusions.   the cataloguing world is cooperative by necessity, so it is useful to know what our peer libraries are doing. certain libraries may choose to continue using ddc for their own reasons, and what these reasons are (the authors in this study made good conjectures) is worth further study; other libraries continuing to classify their materials with ddc may wish to act on the conclusions of this study and consider reclassifying their collection with the more commonly used lcc scheme.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154   lund, b., & agbaji, d. (2018). what scheme do we prefer? an examination of preference between library of congress and dewey decimal classification among u.s.-based academic library employees. knowledge organization, 45(5), 380-392. https://doi.org/10.5771/0943-7444-2018-5-380   shorten, j., seikel, m., & ahrberg, janet h. (2005). why do you still use dewey? academic libraries that continue with dewey decimal classification. library resources & technical services, 49(2), 123-316. retrieved from https://journals.ala.org/index.php/lrts/article/viewfile/5151/6248 evidence based library and information practice evidence based library and information practice 2009, 4:3  evidence based library and information practice       evidence summary    boys are reading, but their choices are not valued by teachers and librarians    a review of:  mckechnie, lynne (e.f.). “ ‘spiderman is not for babies’ (peter, 4 years): the ‘boys    and reading problem’ from the perspective of the boys themselves.” the    canadian journal of information and library science 30.1/2 (2006): 57‐67.     reviewed by:   virginia wilson  shirp coordinator, health sciences library, university of saskatchewan  saskatoon, saskatchewan, canada  e‐mail: virginia.wilson@usask.ca     received: 1 june 2009        accepted: 30 july 2009      © 2009 wilson. this is an open access article distributed under the terms of the creative commons attribution  license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and  reproduction in any medium, provided the original work is properly cited.    abstract    objective – this study looks at what  constitutes legitimate reading material for  boys and how this material is defined in light  of assessed gender differences in reading, and  is part of a larger, ongoing research project on  the role of public libraries in the development  of youth as readers.    design – semi‐structured, qualitative  interviews and book inventories.    setting – the research originated from the  mlis 566 (literature for children and young  adults) class at the faculty of information and  media studies, university of western ontario,  london, ontario, canada.    subjects – forty‐three boys, ages four through  twelve, were interviewed. most of the boys  lived in ontario, although a few came from  other canadian provinces.    methods – library school students who were  registered in a literature for children and  young adults class interviewed children and  young adults about their reading and  information practice as part of a “book  ownership case study” assignment. the  researcher also interviewed children and  young adults, for a total of 137 case studies.  for the purpose of this article, a data subset  for the 43 boys included in the larger project  was analyzed. the boys ranged in age from  four to twelve years. the mean age was eight  and the median age was nine. the theoretical  perspective of reader response theory was  used to situate the study. this theory has the  relationship between the text and the reader as  its focus, and it suggests that to understand  the reading habits of boys, there needs to be  recognition that the experts about their  46 mailto:virginia.wilson@usask.ca http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2009, 4:3  reading are the boys themselves. the  interviews, which explored reading  preferences and practices, were qualitative,  semi‐structured, and took thirty minutes to  complete. in addition to the interview, each  boy’s personal book and information material  collection was inventoried. the researcher  used a grounded theory approach to analyze  the inventory and interview data to pull out  themes related to the research questions.  grounded theory “uses a prescribed set of  procedures for analyzing data and  constructing a theoretical model” from the  data (leedy and ormrod 154).    main results – the collection inventories  revealed that all 43 study participants had  personal collections of reading materials. the  collections ranged from eight volumes to 398  volumes. there was a mean volume total of  108 and a median of 98 books per boy. in  addition to books, other materials were in the  collections. video recordings were owned by  36 (83.7%) of the boys, 28 (65.1%) of  participants had computer software, 28  (65.1%) owned audio recordings, and 21  (48.8%) of the collections also included  magazines.     in the interview data analysis, a number of  themes were revealed. all of the boys except  one owned fiction. some genres appeared  frequently and were different than the ones  found in the inventories taken of the girls in  the larger study. genres in the boys’  collections included fantasy, science fiction,  sports stories, and humorous stories. the boys  also discussed genres they did not enjoy:  classic children’s fiction, such as the  adventures of robin hood, love stories, and  “books about groups of girls” (61). all but five  boys had series books such as animorphs,  captain underpants, redwall, and magic  treehouse in their collections.     all study participants except for one owned  non‐fiction titles. when asked what their  favourite book was, many of the boys chose a  non‐fiction title. holdings included subjects  such as “jokes, magic, sports, survival guides,  crafts, science, dictionaries, maps, nature, and  dinosaurs” (62).     in addition to books, the boys reported  owning and reading a wide range of other  materials. comics, manga, magazines, pop‐up  and other toy books, sticker books, colouring  books, puzzle books, and catalogues were  among the collection inventories. only one  boy read the newspaper.    another theme that emerged from the  interview data was “gaming as story” (63).  the boys who read video game manuals  reported reading to learn about the game, and  also reading to experience the game’s story.   one boy’s enjoyment of the manual and the  game came from the narrative found within.     various reading practices were explored in the  interviews. formats that featured non‐linear  reading were popular. illustrations were  important. pragmatic reading, done to support  other activities (e.g., pokeman), was “both  useful and pleasurable” (54).     and finally, the issue of what counts as  reading emerged from the data. many boys  discounted the reading that “they liked the  best as not really being reading” (65). some of  the boys felt that reading novels constituted  reading but that the reading of computer  manuals or items such as science fair project  books was “not really reading” (65). a  distinction was made between real books and  information books by the boys.     conclusions – the researcher explored what  has been labelled as the “problem” of boys  reading in this paper. she found that the 43  boys in this study are reading, but what they  are reading has been undervalued by society  and by the boys themselves. collection  inventories found a large number of non‐ fiction books, computer magazines, comic  books, graphic novels, and role‐playing game  manuals—items not necessarily privileged by  libraries, schools, or even by the boys  themselves. the researcher suggests that “part  of the ‘boys and reading problem’ then lies in  what we count as reading” (66). by keeping  47 evidence based library and information practice 2009, 4:3  48 what boys are actually reading in mind when  it comes to collection development and library  programming, children’s librarians can “play  a central role in legitimizing the reading  practices of boys” (66).      commentary    this is a worthwhile study that illuminates the  reading habits and preferences of pre‐teen  boys. the researcher is aware that qualitative  studies are not generalizable and is careful to  indicate that the results of her analysis are  particularly for the boys participating in the  study. the themes that emerged from the data  analysis are useful for public and school  librarians who are interested in learning more  about the preferences of their young male  patrons. the individual collection inventories  revealed some good information about what  the boys value and what they enjoy and find  worthwhile in terms of reading materials. the  researcher’s remarks about the valuation and  privileging of certain types of reading and  information material by the boys themselves  are interesting observations.  they suggest  that perhaps librarians can play a part in  validating the materials chosen by boys, and  that such validation could have an effect on  what is seen as the problem of boys and  reading.     the researcher has indicated that this paper is  part of a larger research study. as such, this  paper is brief and focused. the book inventory  results are interesting and informative, and  the themes that emerge from the qualitative  interviews shed light on boys and reading. if  anything, more examples from the boys’ own  voices would have fleshed this paper out. for  readers who are interested in the mechanics of  such a study, it would have been helpful to  have additional information on how the data  were analyzed into themes, as well as a bit  more about both the reader response  theoretical framework and the grounded  theory approach.     this paper is worthy of note as it allows the  boys to speak for themselves, and it reveals  fascinating information about the habits and  preferences of a patron group which has  offered challenges to many public and school  libraries/librarians. in terms of applying this  research to practice, librarians may consider  expanding collections to reflect the types of  reading materials found in the boys’ personal  collections, starting book clubs that include the  additional reading materials (e.g., video game  manuals, comics), and observing to see if the  results of this study are similar to what they  are seeing in their own library patrons.      works cited    leedy, paul d. and jeanne ellis ormrod.  practical research planning and  design, 8th ed.    upper saddle river, nj: merrill  prentice hall, 2005.  evidence based library and information practice evidence based library and information practice 2009, 4:3  evidence based library and information practice       commentary    being a critical professional: the importance of post‐publication peer review in  evidence based library and information practice     lorie a. kloda  associate editor (evidence summaries)  liaison librarian, life sciences library, mcgill university  montreal, quebec, canada  email: lorie.kloda@mcgill.ca     received : 14 august 2009        accepted : 14 august 2009      © 2009 kloda. this is an open access article distributed under the terms of the creative  license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,   distribution, and reproduction in any medium, provided the original work is properly cited.  commons attribution    last year, stewart brower, co‐editor of the  open access journal communications in  information literacy, defended the peer review  process. he argued that despite its flaws, peer  review helps librarians’ careers. “peer‐ reviewed articles get librarians promoted.  peer‐reviewed articles get librarians tenured.”  while this is true for academic librarians in  institutions that promote librarians and grant  them tenure based on articles published, most  library and information professionals are not  in such positions, so why should library and  information professionals care about peer  review?    lately, it seems as if everyone is talking about  peer review, its benefits and its weaknesses.  by peer review, they usually are referring to  pre‐publication review by experts (“peers”) in  the field, sometimes confidential (“blind”),  sometimes not. recently, nature reported on a  hoax article that passed through peer review  (gilbert) and the new york times featured a  story about ghostwriting and the medical  literature (singer). what’s interesting about  these news items is that they question the  effectiveness of the peer review process.    peer review was never intended to detect  fraud. its original purpose was to evaluate  research in order to assist the author in  improving it and the editor in deciding  whether or not to publish it (zuckerman and  merton 68‐75). although the process has  evolved since its beginnings in the mid‐17th  century, there has been no standardization of  peer review across journals, and its use did  not become widespread until the mid‐19th  century (weller 1‐4). as brower also noted in  his editorial, pre‐publication peer review is  still effective when considered relative to its  original purpose: it assists editors in the  selection of manuscripts and assists authors in  improving their research and writing skills.     post‐publication review    back to why librarians should care about peer  review in our own field. in addition to  concerns about fraud, some question the   72 mailto:lorie.kloda@mcgill.ca evidence based library and information practice 2009, 4:3  effectiveness of peer review in weeding out  substandard or useless research. i would  argue that instead of placing this burden  entirely on the pre‐publication peer review  and editorial processes, we should transfer  some of it to the post‐publication phase. if we  admit that not all original research that gets  published is flawless, then surely we can take  the responsibility of evaluating what we  choose to read. there are two things to  evaluate when considering research: its  validity or accuracy and, when appropriate, its  applicability to current practice.     librarian as critic    librarians and information professionals, who  have a masters degree and sometimes more  advanced education, are equipped to do at  least some evaluation of published research.  we are trained in basic statistical methods  and, in some cases, qualitative design. we also  benefit from our work experience. for those  who lack more advanced critical appraisal  skills and are interested in acquiring them,  online courses, readings, and professional  development workshops are available. all  librarians, as professionals, have a duty not  just to read peer reviewed research, but to  engage in post‐publication review of their  own. not to do so but instead to take  everything one reads at face value, is  irresponsible.    there is a much published researched  available, and scanning recent tables of  content or search results then deciding to read  something is already an onerous task.  choosing to read an article based on its title,  abstract, or author is a first step in evaluating a  study’s relevance, or applicability. but it is  essential to remain critical while reading and  interpreting the report. “critical” in this sense  does not mean negative, but carefully judging,  in the academic sense, as in “critical thinking.”     traditional publishing dictates a linear process  where a manuscript is submitted, reviewed by  peers, revised, edited and eventually  (hopefully) published. this rather rigid format  persists in our field. despite technological  advances in format and speed to improve  access, we still submit manuscripts which  undergo the same linear process, and what  emerges is considered something of a “final  product.” but this does not mean that the so‐ called final product is the last word. librarians  can and should respond to these research  reports, whether by reflection, discussions  with colleagues or in journal clubs. and of  course, we can respond to the original authors.    evidence summaries in eblip    formal critical appraisal is a form of post‐ publication peer review. evidence summaries  published in eblip are structured abstracts  accompanied by critical appraisals of original  research reports. these summaries are written  by peers with expertise in specific domains  and research methods. each evidence  summary is a synopsis of an individual  research article and a commentary on its  validity and applicability. the ultimate goal of  publishing evidence summaries is knowledge  translation, or putting research into practice.  but another goal is for library and information  professionals to think critically and to engage  in a conversation related to the research.    instead of trying to fix pre‐publication peer  review or replace it, we should consider it as  just one aspect of the review process. after an  article is published, professionals and  researchers should be encouraged to engage  with it – to be critical and reflective. evidence  summaries are one way to continue the  conversation begun by the original  publications. the summaries are not the final  word either. it is hoped that evidence  summaries will encourage eblip readers to  think more critically about the original  research and to engage in conversation about  it.     in the coming year, eblip will begin efforts to  inspire these conversations in two ways: first,  by notifying the authors that their publications  have been critically appraised in the journal,  and second, by encouraging these authors and  eblip readers to add their own comments to  evidence summaries using the online journal  73 evidence based library and information practice 2009, 4:3  74 system’s “add comment” feature on each web  page.    peer review is not perfect, and neither is most  research. library and information  professionals can and should evaluate  published research openly and constructively  in order to create conversations and facilitate  putting research into practice. by building on  evidence summaries, eblip hopes to start  these conversations and facilitate the transfer  of evidence into library and information  practice.      works cited    brower, stewart. “why peer review?”  communications in information  literacy 2.2 (fall 2008): 62‐3.    gilbert, natasha. “editor will quit over hoax  paper.” nature news (june 19 2009).  <http://www.nature.com/news/2009/ 090615/full/news.2009.571.html>     singer, natasha. “medical papers by  ghostwriters pushed therapy.” new  york times (august 4, 2009).  <http://www.nytimes.com/2009/08/0 5/health/research/05ghost.html>     weller, ann c. editorial peer review: its  strengths and weaknesses. medford,  nj: information today, 2001.    zuckerman, harriet and robert k. merton.  “patterns of evaluation in science:  institutionalisation, structure and  functions of the referee system.”  minerva 9.1 (1971): 66‐100.   http://www.nature.com/news/2009/090615/full/news.2009.571.html http://www.nature.com/news/2009/090615/full/news.2009.571.html http://www.nytimes.com/2009/08/05/health/research/05ghost.html http://www.nytimes.com/2009/08/05/health/research/05ghost.html news   apply for the research training institute ‘21!    2020. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29880     are you a library professional who wants to take your research skills to the next level? do you have a brilliant idea or research topic that you want to pursue but do not know where to start and need help from your peers and experts to carry it through?   the medical library association (mla) research training institute (rti) is a unique, highly-effective, and collaborative online research training and support program. the rti ‘21 immerses practicing librarians in scholarly research, inquiry, and publishing. librarians of all levels of professional experience and types of work environments who provide health information, services, and support and who have an interest in increasing their research skills and confidence and want to improve library and health care outcomes, are encouraged to apply to the rti program.   the institute is a one-year online program that consists of a series of online modules in advanced research methods, mentoring by faculty experts and peer coaches, preparing and implementing a research project, and an opportunity to present findings at the mla ‘22 virtual conference. rti ‘21 features an expanded research curriculum and greater affordability and flexibility for participants. learn more about rti program details.   rti ‘21 applications will be open until january 4, 2021   applicants are now being accepted through january 4, 2021 for the 2021 cohort of rti research fellows. accepted applicants will be notified in march 2021. see the rti submission process for eligibility requirements and selection criteria. the institute offers many scholarship opportunities, including scholarships for professionals working in small libraries and academic health libraries, engaging in diversity, equity, and inclusion (dei) research, and those with limited financial means.   discover and nurture your research talents! learn how to conduct and lead quality research projects to improve your library and support quality health care. meet like-minded research colleagues, make lifelong friends, gain research skills and confidence, form collaborations with other researchers, and help end-users improve health and wellness! apply today!   read more about the rti success story.   the project is made possible in part by a grant from the u.s. institute of museum and library services (imls). applicants must be u.s. citizens or permanent residents.   if you would like to learn more about the institute, please visit the rti website or contact rti project director susan lessick, ahip, fmla.   evidence based library and information practice using evidence in practice   welcome to the library: data-driven student worker empowerment   elena carrillo clinical assistant professor and head of access services university of illinois at chicago, university library chicago, illinois, united states of america email: elena48@uic.edu    jung mi scoulas clinical assistant professor and assessment coordinator university of illinois at chicago, university library chicago, illinois, united states of america email: jscoul2@uic.edu    received: 18 feb. 2020                                                               accepted: 28 apr. 2020      2020 carrillo and scoulas. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29728   setting   the university of illinois-chicago (uic) is a public research university in the united states. the uic library provides materials (in excess of 2.2 million items) and a wide variety of services in support of instruction and research at its split east and west chicago campuses and at regional health sciences locations in peoria, rockford, and urbana. the collection focus at the richard j. daley library in chicago is humanities, social sciences, and engineering. the university serves more than 33,000 undergraduate and graduate students across sixteen colleges.   the uic library employs 150 faculty and staff members as well as more than 100 student employees. at any given time, 20 to 25 of these student employees serve the daley library access services department, which staff all four key public service points in the building. daley access services is responsible for key in-person, non-virtual functions in the library system including circulation, course reserves, interlibrary loan, and consortial resource sharing, as well as media assistance and other customer-oriented services.   the department also is responsible for two staffed door stations visible to anyone who comes into the building. patrons must pass these stations to enter the library. the additional service desk on the first floor is a combined service point for all circulation activities.   uic boasts a campus on which there is no racial or ethnic majority, but this diversity alone does not determine whether students feel welcomed into campus spaces, including the daley library.   problem   undergraduate student employees responsible for staffing the entry stations at the east and west doors were originally trained to monitor the alarm system and check id cards after hours. their role was primarily security. since 2016, the desire was to transition that role from one of “watchdogging” the doors to one of welcoming patrons in order to encourage library use. this decision was made primarily in conjunction with other efforts outlined in our strategic goal to “create and sustain an inclusive culture and a welcoming environment for all” (uic library, n.d.).   library administration and operations managers had no previous data on patron attitudes about the library, only anecdotal feedback and opinions from staff and patrons raising concerns about their experience entering through doors. we were concerned about the impact of receiving no interaction or negative interaction when entering the library, especially on students who may experience library anxiety.   a study conducted by jiao, onwuegbuzie, and lichtenstein (1996) shows that college students experience library anxiety and that “it is imperative that libraries are made as attractive, as welcoming, and as monocle as possible. in order to achieve this, it is important that all library staff are friendly and professional” (p. 159).   we used data from iterative assessment to develop mission-focused training as part of our effort to create a welcoming library environment toward more positive academic experience. as we stated in an article about this department’s experience with change management, “staff does work simply because they were told to, without any clear sense of why it is necessary or how it figures in the bigger picture…. either they have forgotten the reason or it was never explained to them to begin with” (carrillo & gregory, p. 29).   our goal was to empower student workers to do great work by connecting them to the mission of the library and to our diverse patron population. we also sought to reduce factors that might intimidate or discourage library use. we used data to assess and share the experience of patrons entering the library in order to reinforce a positive customer service attitude and identify areas for improvement.   evidence   the daley library access services department conducted a pilot survey during the fall 2018 midterm exam period to assess how patrons felt about the service they received when engaging with the library service points. the survey was paper based and made available over a two-week period for patrons who visited the library in person. the survey asked if patrons felt they had received what they needed and whether they were made to feel welcome. the survey also asked for any additional feedback about their customer service experience in an open-ended question.   a total of 291 participants completed the survey, and the results show that the majority of them (more than 85%) responded positively (see table 1). we were surprised by the response because of previous mixed anecdotal evidence to the contrary. our desire then became to increase the positive response percentage with a goal in mind of 90% or better.   we then decided to repeat the survey with a specific focus on the welcoming aspect of our customer service at our entryways. the goal was to hone in on first contact with the library as opposed to patrons potentially already familiar with our services. we simplified this new “door survey” to present a single statement: “i felt welcomed when i entered.” patrons were then provided with four options to strongly agree, agree, disagree, or strongly disagree with the statement.   table 1 pilot survey (n = 291) question strongly agree agree disagree strongly disagree total i was greeted with a smile and made to feel welcome 85.91% 11.00% 2.06% 1.03% 291 i got what i needed. 87.97% 8.93% 2.41% 0.69% 291 all of my questions were answered. 89.69% 8.59% 0.69% 1.03% 291   we received the results we expected from the first final exam period survey—that a significant percentage of our patrons (nearly 69%) agreed that they felt welcomed upon entering the library. the numbers could not tell the whole story of why the remaining 31% did not feel welcomed, however.   during the spring of 2019, we created a “secret shopper” type of observation log for staff to provide feedback on behaviour happening at our doors. staff were aware of the observation period and made to understand that the purpose of the log was not disciplinary but to record evidence of patron’s perspectives on our service. we did not differentiate the findings in terms of which door the observed behaviour was recorded at, and no staff were associated with any behaviour or identified by name.   we recruited 10 staff members from a variety of library departments and asked them to record their observations in a google form about their experience when they entered and exited the building for a two-month period (january 21 to march 21, 2019). the survey specifically asked them to record “welcoming” and “negative” behaviours, as well as inviting them to make open-ended comments.   table 2 observation log: top 3 welcoming behaviours top 3 welcoming behaviours (n = 103) % attendant made eye contact 90.30% attendant greeted me 53.40% attendant smiled at me 54.40%   table 3 observation log: top 3 negative behaviours top 3 negative behaviours (n = 71) % checking cell phone 36.60% chatting 21.10% slouching 15.50%   for this observation log, observers recorded over 188 interactions at the doors. the majority of respondents reported welcoming behaviours, and as shown in table 2, the top welcoming behaviour was “attendant made eye contact” (n = 93, 90.3%) followed by “attendant greeted me” (n = 56, 54.4%) and “attendant smiled at me” (n = 55, 53.4%). the findings also indicate that negative behaviours included “checking phone” (n = 26, 36.6%), “chatting” (n = 15, 21.1%), and “slouching” (n = 11, 15.5%), with cell phone usage being the top complaint by observers (table 3). additional comments were reported in the open-ended response including “attendant was reading or working on assignments” (n = 9)” and “looking down” (n = 5).   implementation   as a result of feedback from the pilot survey, the observation log, and subsequent door survey results in 2018, a dedicated program of retraining was introduced. we called it a “student refresher,” referring to the work of student employees who primarily work at the doors.    this student refresher was conducted by the student employee managers in the daley access services department during a half-day session. during the session, the purpose of the assessment, the results, and the mission of the library were emphasized—along with reminders for work expectations and library policies. the training included these elements:   ●       reflection on the observation log comments ●       story-based scenarios to help staff recognize and model exemplary service ●       emphasis on a positive welcoming experience for a diverse patron population ●       goals to increase a welcoming experience for the spring final exam period   while it’s true that “sometimes tasks are handed down through the years to the point at which no one can really remember why they started doing them in the first place” (carrillo & gregory, p.29), it’s often the case with student workers, who have so many competing demands on their attention, that remembering the tasks from semester to semester can be a challenge.   gathering and sharing evidence based data from these various assessment tools was the key to reinforce positive customer service habits, inspire student workers to make connections with patrons, and remind them of their critical role in the library.   outcome   the door survey was conducted again during our final exam period in the spring of 2019. (we have conducted it every final exam period since december 2018). the results appear to corroborate the general positive experience of patrons entering the library.   as shown in table 4, the number of respondents who reported “agree” or “strongly agree” increased 10.6% from fall 2018 to spring 2019. a 15% increase is shown from fall 2018 to fall 2019.   we believe the intervention empowered student workers to empathize with the needs of our patrons and connect their work to the service mission of the library. student workers responding to a survey after the spring 2020 refresher training indicated that they appreciated the evidence of the impact of their work on positive patron response. despite critical feedback pertaining to some of the activities during the refresher, 100% stated that they believed the goal of a 90% welcoming satisfaction response from patrons was achievable. they also reported that they were committed to providing the excellent customer service necessary to reach the goal.   table 4 door surveys: round i (n = 1,282), ii (n = 870), iii (n = 1,002)     strongly agree agree disagree strongly disagree round i dec-18 52.82% 16.15% 6.24% 23.79% round ii may-19 55.98% 23.62% 5.63% 9.77% round iii dec-19 62.67% 21.36% 4.29% 11.68%   reflection   while the process seems like it should have been straightforward, it was a challenge to introduce this work into a department without a strong culture of assessment. after several iterations of completing the survey work and retraining, student workers became more comfortable being evaluated at the service points. the initial skepticism was largely ameliorated by the positive reinforcement of the survey results. through the assessment, students connected their work to the impact it can have on academic success and how it is valued as a cornerstone of the library mission. the future challenge will be to encourage a constructive competitive spirit that will continue the momentum of increasing the percentage of positive results. in order to accomplish this, we will continue to use evidence based practice to reinforce excellent customer service.   references   carrillo, e., & gregory,g. (2019). change management in extremis: a case study. journal of access services, 16(1), 21–33. https://doi.org/10.1080/15367967.2018.1560224   jiao, q. g., onwuegbuzie, a. j., & lichtenstein, a. a. (1996). library anxiety: characteristics of “at-risk” college students. library and information science research, 18(2), 151–163. https://doi.org/10.1016/s0740-8188(96)90017-1   university of illinois at chicago library. (n.d.). strategic plan. https://library.uic.edu/about/strategic-plan editorial   eblip9 notes and highlights   john w. wiggins director of services and quality improvement drexel university libraries philadelphia, pennsylvania, united states of america email: jww27@drexel.edu   danuta a. nitecki dean of libraries, and professor, college of computing & informatics drexel university libraries philadelphia, pennsylvania, united states of america email: dan44@drexel.edu      2017 wiggins and nitecki. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     the ninth international evidence based library and information practice conference (eblip9) was held in philadelphia, pennsylvania, usa on june 18-21, 2017.  the conference theme of embedding and embracing evidence reflected the focus on promoting evidence based practice throughout the information professions. the philadelphia conference followed successful events held previously at sheffield, edmonton, chapel hill-durham, stockholm, salford, saskatoon, and brisbane (once in 2005 and again in 2015).    eblip9 in philadelphia   drexel university was proud to win the competitive selection process to host the eblip9 conference in philadelphia and especially to bring it back to the united states for only the second time. the planners organized content activities, driven by a strong objective of sharing the welcome spirit of this city of brotherly love and creating a memorable conference experience.    keynote addresses, concurrent presentations, and poster sessions were held on the drexel campus in its stunning papadakis integrated sciences building. the convenience of assembly in one area added to building community among participants. john fry, the dynamic drexel university president, gave a thoughtful welcome. many attendees tweeted impressions about the conference throughout and included such comments, as “it is great to listen to a university president who gets libraries.”       figure 1 eblip9’s opening remarks were delivered by drexel university president john fry.     in addition, two off-campus venues for social events drew attendees to the center city district by walking, following guided paths, using city trolleys, or hiring cabs. a sudden downpour on monday did not deter attendees from arriving at the philadelphia free library for the first night reception. with a backdrop view of the city skyline at sunset, the energizing president and director of the free library of philadelphia, siobhan a. reardon, warmly welcomed the attendees. sunshine accompanied conference attendees on wednesday as they made their way to an elegant city venue, the crystal tea room, for the conference gala dinner. as one staff member observed, “the success of the conference was evident when the dance floor was filled with energetic moves and lots of laughs” to the pulse of music.   over 140 attendees registered for eblip9 from seven countries: australia, canada, kenya, norway, sweden, the united kingdom (including the isle of man), and the united states of america. unfortunately, the controversy over travel restrictions to the usa had an impact on attendance from outside the country. some presenters declined invitations to speak at the conference. the conference planners persevered to provide a rich and diverse array of experiences, including three keynote addresses, 41 paper presentations, 13 posters, 1 panel session, 2 sponsor presentations, and 3 workshops. topics were many and varied, with evidence based practice being critically explored in the context of big data, collection management, decision making, information literacy, leadership, open access, and website design and usability. the full conference program is available online (http://easychair.org/smart-program/eblip9/). this eblip conference continued the tradition of providing excellent opportunities to network, and attendees met colleagues from around the world while enjoying the opening reception, the conference dinner, box lunches, and catered breaks throughout the conference.   figure 2 heather holmes, joanne gard marshall, jon eldredge, alison brettle, and denise koufogiannakis gather at the opening reception at the free library of philadelphia.     figure 3 line dancing at the conference gala dinner at the historic crystal tea room!     the afternoon before the conference began, preconference attendees opted to attend one of the workshops on developing eblip skills, centred in archives and special collections libraries, engaging assessment to show value and make decisions, or learning tips to publish research on eblip. many conference attendees arrived early to register and to ask for tips for exploring the city—some finding adventure, discovering local history, or sampling from the vibrant food and brew scenes that philadelphia offers.     figure 3 eblip9 attendees engage with patrick milas and jenifer gundry at their poster.     featured speakers   the conference featured three keynote speakers. alison brettle, professor of health information and evidence based practice in the school of nursing, midwifery & social work at the university of salford, opened the conference with her riveting keynote address, collecting and using evidence routinely for advocacy. dr. brettle closes the special eblip9 feature section of this issue with a thoughtful commentary based on her keynote.   yi deng, dean and isaac l. auerbach professor at the college of computing & informatics at drexel university, spoke on data and the 21st century economy and the changes that might impact the professional work of librarians and information professionals.   pam ryan, director of service development & innovation at toronto public library, delivered the closing keynote address to the philadelphia audience from office in toronto, and addressed eblip and everyday practice for librarians and our libraries.   in addition to the keynotes, papers, and posters, a panel treated conference attendees to a discussion on responding proactively to “fake news,” moderated by jon eldredge with heather holmes, scott walter, and malin ogland responding on the panel.   other crowd pleasers   another tradition, initiated at eblip6 in salford, extended its run at eblip9: poster madness!  in a round of humble and breathless promotion, all poster presenters opted to participate in strict one-minute speed presentations. they certainly delivered, to the delight and support of the audience, and as evidenced by the constant conversations during the two formal poster presentation periods.     figure 4 dr. alison brettle delivering the opening keynote.     figure 5 eblip9 attendees engage with frans albarillo and lee ann fullington at their poster. the biowall in papadakis integrated sciences building is visible in the background.     awards   conference attendees voted to select people’s choice award winners for paper presentations and for posters. the eblip9 winners, three of which have research articles accepted for inclusion in this issue, were:   people's choice awards for best paper   first place: april cunningham & michelle dunaway – analyzing and assessing il in the framework era: a rhetorical analysis followed by exploratory factor analysis second place: kyla everall & judith logan – a mixed methods approach to iterative service design of an in-person reference service point.  (see their related article in this issue). third place: jingying mao & kirsten kinsley – embracing the generalized propensity score method: measuring the effect of library usage on first-time-in-college student academic success.  (see their related article in this issue).   people's choice awards for best posters   first place: savannah kelly – first-year students’ research challenges: does watching videos on common research struggles prior to library instruction increase students’ confidence?  (see her related article in this issue). second place: lee ann fullington & frans albarillo – embedded in technology ecosystems: graduate students, mobile devices, and academic workflows third place: rick stoddart – the library learning narrative: a pilot project to capture learning outcomes at the reference desk   an analysis of conference content   the guest editors of this feature editorial analyzed the abstracts of the eblip9 paper presentations. they determined that the 41 presentations included 24 addressing practice and 17 presenting research. of these and including the poster presentations, ten were accepted for publication in this special issue of eblip journal; three as articles addressing the use of evidence in practice and seven as research articles.     figure 6 people’s choice awards for best papers: ipc chair denise koufogiannakis, kirsten kinsley, jingying mao, judith logan, kyla everall, loc co-chair john wiggins.     figure 7 people’s choice awards for best posters: loc co-chair john wiggins, lee ann fullington, frans albarillo, savannah kelly, rick stoddart, ipc chair denise koufogiannakis.     figure 8 dana thomas presents her paper at eblip9.   conference presentations utilized several different research methods for gathering and analyzing data. among the most prevalent methodologies were surveys (13), interviews, both focus group and individual (12), statistical analysis (8), ethnographic methods (6), literature reviews (5), and content analysis (4).  many combined qualitative and quantitative methodologies to present mixed method designs.    thirty-seven presentations heavily favoured academic library settings, including two describing groups of academic libraries. also included were one presentation addressing academic and medical libraries, and one on academic and public libraries. additionally, there were two presentations addressing public library settings, and two addressing school library settings.   the topics and content of the presentations ran the gamut of library programming. the heaviest concentrations examined included library space, such as space use, needs, and design/redesign, and also library collection development, such as pricing of resources, budget decisions, cooperative collection development, demand driven acquisitions approval plan profiles, inclusion of constituents in budget/selection decisions, and user interactions with collections.   other themes found among conference presentations included library service design, user-focused services, and library use by non-native born patrons and by international graduate students. reference and research support topics included information literacy and program improvement, user knowledge, libguides improvement, creation of online learning objects, reference service design, impact of consultations, and assessment of chat services. student success, faculty productivity and research success, and communication of library impact to stakeholders were also featured, along with library funding and allocations, and library program impact on school readiness of preschoolers. presenters also explored supports for librarians and information professionals, with content covering evidence based practice and embedded culture, staff training library and information practice research skills (including support for working with human research subjects), and conscious reflection to inform one’s own practice.   selected publications   appearing as research articles in this eblip9 conference feature issue are nine papers and one poster presentation.   susan gardner archambault and alexander justice used mixed methods to examine the unexpected ways that students used a six year old information commons at an academic institution. christina r. hillman, kourtney p. blackburn, kaitlyn shamp, and chenisvel nunez collected and analyzed evidence from multiple sources to develop a survey on campus interest in and use of library space. nora almeida and junior tidal addressed the concern that too few librarians consider user design preferences and non-critical approaches when designing tools for academic student use.   eamon tewell, kimberly mullins, natalia tomlin, and valeda dent used ethnographic methods in their pursuit to improve academic library understanding of student research and study needs to improve the student library experience. kristin hoffman, selinda adelle berg, and denise koufogiannakis sought to understand factors that affect academic library and information professionals’ ability to carry out and disseminate research, looking for correlations between research productivity factors and research outputs, including the potential impact of institutional supports.   two of the people’s choice award winners prepared research papers for this issue. jingying “jean” mao and kirsten kinsley explored how one might isolate an independent variable and control for other inputs and environmental variables in a scientifically rigorous way when studying success within academic student populations. savannah lea kelly investigated the impact of providing video tutorials in traditional sessions to see if students who viewed tutorials reported higher research confidence levels than those in control groups.   also included are two papers and one poster presentation focusing on the use evidence in practice in this issue. stacey astill and jess web explored their local practices collecting data on the in-library reference use of materials. people’s choice award winners kyla everall and judith logan pursued the objective of implementing an evidence-based service design framework for a public service point at the university of toronto’s public library. susan breakenridge addressed the use of evidence to address information needs of academic library administrations considering or assessing requests for extending operations overnight.    final words of thanks   reflecting on the success of the eblip9 conference, we share our sincere appreciation for the efforts of the international program committee and our on-site volunteers for their dedicated efforts. particular recognition for our deserving colleagues on the organizing committee is also in order in light of their months of continuous hard work:   stacy stanislaw and darin pfeifer, drexel university denise koufogiannakis (also chair of the international program committee), university of alberta rick stoddart, university of idaho m. j. tooey, university of maryland, baltimore jessica rossi and michael krasulski, community college of philadelphia joel nichols, free library of philadelphia virginia wilson, university of saskatchewan   we also are grateful to our sponsors who helped make this event possible through their generous funding: elsevier; centre for evidence based library and information practice at the university of saskatchewan; association of research libraries; association of college and research libraries; pennsylvania academic library consortium inc.; jstor; school of information science at the university of tennessee at knoxville; medical library association; clarivate analytics; and drexel university libraries.   the call for expressions of interest to host eblip10 went out recently. the selected location will be announced early in 2018. we hope to see you there!     evidence summary   variation among copies of titles catalogued as identical should inform retention decisions   a review of: teper, j. h. (2019). considering “sameness” of monographic holdings in shared print retention decisions. library resources & technical services, 63(1), 29-45.                 https://doi.org/10.5860/lrts.63n1.29   reviewed by: rachel elizabeth scott interim coordinator, cataloguing, collection management, and library information systems & integrated library systems librarian university libraries university of memphis memphis, tennessee, united states of america email: rescott3@memphis.edu   received: 23 nov. 2019                                                                 accepted:  6 jan. 2020      2020 scott. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29663     abstract   objective – to investigate the degree to which books catalogued using the same bibliographic record differ and to consider the implications of these differences for cooperative monographic print retention programs.   design – book condition survey.   setting – academic library consortium in the united states of america.   subjects – 47 monographic titles, publication years 1851-1922, held by all consortium members and catalogued using the same respective oclc record number. 625 out of a possible 705 circulating copies of these titles were available for item-level analysis via interlibrary loan.   methods – book condition surveys were completed for all items and the resulting sets of assessment data points were analyzed to reveal trends.   main results – 3.4% of items analyzed exhibited cataloguing errors (i.e., were catalogued using the wrong oclc records), 56.8% retained their original bindings, 17.8% were marked to show previous ownership, 95.7% were complete with no missing content, 9.8% had no damage, and 18.9% had received identifiable preservation action.   conclusion – books catalogued using the same oclc record demonstrated many differences when compared at the item level. these differences are important in light of shared print retention programs and highlight a need for inquiry into the number of copies that should be retained to minimize the loss of uniqueness in print materials.   commentary   the study at hand responds to digitization initiatives and cooperative print retention programs that ensure the availability of a text to readers, as well as the pressure to use academic library spaces differently and weed non-unique content. previous studies addressing the “sameness” of books have been conducted, notably by stauffer (2016), who compared 10 copies of a single title, and conway (2013), who investigated digital imaging errors in a 1,000-item sample from hathitrust. this study is situated in the context of a large academic consortium with an existing shared print program and differs from studies conducted by the u.s. center for research libraries, for example, in its focus on monographic, not serial, holdings.   glynn’s 2006 critical appraisal tool will be used to evaluate methodology and presentation of the study at hand. the article is clearly written and logically organized. the literature review reveals that the author has engaged thoughtfully with relevant studies and understands complex relationships among cooperative retention programs, digitization, preservation, bibliographic and holdings data, and space reclamation. the survey design and methodology section, however, lacks detail and does not indicate how the survey instrument was conceived (“the author designed a survey,” p. 30). it would seem that the survey was based on the conway (2013) study, which measured several of the same data points, and that the author completed all surveys, but the reader should not have to speculate on either count. although the author refers to “various manipulations of the collected data” (p. 32), the only calculation provided is for the probability of ensuring an item in good condition in this sample. findings are grouped by institution and subject area, but no manipulations or statistical analysis beyond calculating percentages from aggregate data are provided.   the study achieves the objectives of measuring differences among books catalogued using the same bibliographic record and discussing some of the implications of these differences for cooperative print retention programs. this study contributes to library science literature and practice in two significant ways. first, it highlights the risk of relying exclusively on oclc bibliographic and holdings data for retention and deselection decisions by revealing how inadequate they are for capturing item-level uniqueness. second, the calculation for the probability of randomly ensuring an item in good condition, as well as the author’s call for further investigation into and refinement of this calculation, can help counteract the “last copy” approach to weeding and replace it with a more nuanced approach to ensuring that unique content persists in academic library collections. this study clearly illustrates the disconnect between “sameness” in shared bibliographic cataloguing utilities and item-level differences in library stacks.   references   conway, p. (2013). preserving imperfection: assessing the incidence of digital imaging error in hathitrust. preservation, digital technology & culture, 42(1), 17-30. https://doi.org/10.1515/pdtc-2013-0003   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154   stauffer, a. (2016). my old sweethearts: on digitization and the future of the print record. in m. k. gold & l. f. klein (eds.), debates in the digital humanities 2016 (pp. 218-229). minneapolis: university of minnesota press. retrieved from http://dhdebates.gc.cuny.edu/debates/text/70 editorial   the c-eblip fall symposium: librarians as researchers   virginia wilson guest editor director, centre for evidence based library and information practice university library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca    2017 wilson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   i am very pleased to introduce the special feature in this issue of evidence based library and information practice: papers from the 2016 c-eblip fall symposium: librarians as researchers. the c-eblip fall symposium is a day-long gathering of librarians who are interested in or actively conducting research. the symposium is presented by the centre for evidence based library and information practice (c-eblip) at the university of saskatchewan. the first symposium was held in october 2014 with 52 attendees from across canada (and one from the us!). in october 2015, 62 interested library and information professionals attended, and in october 2016, we had about 50 library folks participate.   c-eblip officially opened in july 2013 and has a mandate to support our librarians as researchers and to promote evidence based library and information practice. the idea of an event to gather library and information professionals from far and wide occurred early in c-eblip’s existence. it’s nice to have internal research and eblip support, but it’s even nicer to be able to connect with others outside of our institution so that we’re not working in a vacuum. additionally, the c-eblip fall symposium presents another venue for librarians to disseminate their work.   the day consists of a single-track of sessions kicked off by an opening keynote address. what follows are various types of presentations. some presenters talk about their research projects, current or finished. some discuss why they do research. and there are other sessions about the pitfalls of doing research, or tips and tricks. the day is designed to have as many take-aways as possible for as many attendees as possible. the sessions are quick (20 minutes including questions), and the day is interspersed with plenty of time for networking, connecting, and getting to know one another. as a side note, we have always received happy feedback on the food after the events.   i’m excited that the eblip journal offered us the opportunity for publication. there is a great lineup of articles for you to enjoy from the 2016 edition of the fall symposium. it all kicks off with the keynote from margaret henderson, director, research data management at virginia commonwealth university. margaret’s talk is entitled “collaborating to increase the evidence base in library and information practice.” there are five additional articles for you in this special feature. three are about the nuts and bolts of research and of being an academic from lise doucette, western university, shannon lucky and joe rubin from the university of saskatchewan, and marjorie mitchell from ubc okanagan campus. and there are two articles focusing on research projects – one from shailoo bedi, university of victoria, and janaya webb, university of toronto, and one from carolyn doi, university of saskatchewan.   i would like to thank everyone who has participated in our past three c-eblip fall symposia. the events have been engaging, informative, and uplifting. partnerships have been created, friendships have emerged, and on that day especially, we all feel proud and accomplished. i hope you will consider joining us at a future symposium as a presenter or as an attendee. we are taking a break in 2017 but hope to return in october 2018. please watch for developments on twitter @ceblip or check out the c-eblip website at http://library.usask.ca/ceblip/. in the meantime, please enjoy these fantastic papers!   microsoft word editorial.doc evidence based library and information practice 2007, 2:3  1 evidence based library and information practice     editorial    change is in the air      lindsay glynn  editor‐in‐chief    public services librarian and instruction coordinator, health sciences library  memorial university of newfoundland  st. john’s, newfoundland, canada  e‐mail: lglynn@mun.ca      © 2007 glynn. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      what is it about september that makes me  think about change?  i suppose that it has  something to do with all those years of  going back to school after summer vacation,  signaling a new academic year and the end  of summer vacation.  or perhaps it has to do  with the changing of the seasons (i already  have had to turn my heat on once) as the  weather turns slightly cooler and the air  changes its scent.  my spring cleaning, so to  speak, happens in the fall.  that’s when i  turf out clothing that hasn’t been worn for  over a year as well as all the toys that the  kids have outgrown or outplayed.      change breeds ideas and renewed vigor.   evidence based library and information  practice has established itself in the  information professions as a fine publication  and as a valuable, informative resource.  all  the expected elements are present in every  issue: research articles, commentaries,  evidence summaries and news.  during the  next year we plan to shake things up a bit  through a number of mechanisms.  feature  issues are well received and we plan to  publish one feature issue per year which  will address areas in the evidence based  literature that may be lacking or topics that  are current and contentious. some of these  issues will be presented by guest editors  who are active in the topical areas upon  which the issues are focused.  the december,  2007 issue will feature commentaries on  classic research articles and the impact that  they have had on the profession since first  published.  i guarantee that you will  recognize at least one of these articles, if not  all of them.      we have many plans, some of which are  tentative and unofficial, others that have not  quite solidified.  what we do know,  however, is that ideas are being brought to  our attention on a regular basis and that the  ideas are good.  i regularly receive email  mailto:lglynn@mun.ca http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:3  2 from potential authors with article ideas that  they are thinking about writing, but wanting  to ensure that their approach is conducive to  the journal. some of these show great  promise, and i look forward to the  opportunity to publish them upon  completion. there are follow‐up studies to  articles published in earlier issues on the  horizon. suggestions have been made to  increase the number of sections in each issue  to include book reviews, brief reports of  evidence being used in practice, etc. it is of  utmost importance that this journal  continues to pique the interest of current  readers and to attract new readers. feel free  at anytime to drop us an email with your  ideas and suggestions. we are always happy  to know that we are doing something well  or whether or not something needs to be  looked at more closely.    i anticipate a very exciting year for this  journal and for our readers.  we may not be  cleaning out the closets, but we will be  opening the windows a bit more.                                              evidence summary theme: education    evidence based library and information practice       editorial   evidence summary theme: education   heather macdonald associate editor (evidence summaries) health and biosciences librarian carleton university ottawa, ontario, canada email: heather.macdonald@carleton.ca      2022 macdonald. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30161     the articles critically appraised in the evidence summaries (es) of this issue focus on the domain of education.  the education domain is defined by koufogiannakis et al. (2004) as “incorporating teaching methods and strategies to educate users about library resources and how to improve research skills.”  education includes teaching information literacy as well as what it means to be an educator and developing as an educator.   the es in this issue highlight a cross-section of studies that examine questions related to education.  they bring a variety of different study designs, geographies, and perspectives to bear.  several of the es have an academic setting.  these es critique studies investigating the use of a rubric for experiential learning in the lis classroom, a librarian-led faculty learning community, the role of the librarian in online learning, factors that affect adoption of new teaching practices, and librarian perceptions of teaching identities.  the final es looks at a study on lifelong learning in public libraries.  regardless of the setting, education plays an important role in libraries.   teaching requires constantly changing and adapting to new audiences, new technologies, and new modalities of teaching. this has been particularly true during the pandemic.  research and practice can help practitioners learn and grow as educators.  the es in this issue provide a summary and critique of some of the most recent research related to education in libraries.  we hope you enjoy reading them and that doing so encourages you, the reader, to employ an evidence-based lens in your own teaching practice.   references   koufogiannakis, d., slater, l., & crumley, e. (2004) a content analysis of librarianship research. journal of information science, 30(3), 227–239. https://doi.org/10.1177%2f0165551504044668   research article   local users, consortial providers: seeking points of dissatisfaction with a collaborative virtual reference service   kathryn barrett social sciences liaison librarian university of toronto scarborough library toronto, ontario, canada email: kathryn.barrett@utoronto.ca   sabina pagotto client services and assessment librarian scholars portal, ontario council of university libraries toronto, ontario, canada email: sabina@scholarsportal.info   received: 16 aug. 2019                                                                  accepted: 7 oct. 2019      2019 barrett and pagotto. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29624     abstract   objective – researchers at an academic library consortium examined whether the service model, staffing choices, and policies of its chat reference service were associated with user dissatisfaction, aiming to identify areas where the collaboration is successful and areas which could be improved.   methods – the researchers examined transcripts, metadata, and survey results from 473 chat interactions originating from 13 universities between june and december 2016. transcripts were coded for user, operator, and question type; mismatches between the chat operator and user’s institutions, and reveals of such a mismatch; how busy the shift was; proximity to the end of a shift or service closure; and reveals of such aspects of scheduling. chi-square tests and a binary logistic regression were performed to compare variables to user dissatisfaction.   results – there were no significant relationships between user dissatisfaction and user type, question type, institutional mismatch, busy shifts, chats initiated near the end of a shift or service closure time, or reveals about aspects of scheduling. however, revealing an institutional mismatch was correlated with user dissatisfaction. operator type was also a significant variable; users expressed less dissatisfaction with graduate student staff hired by the consortium.   conclusions – the study largely reaffirmed the consortium’s service model, staffing practices, and policies. users are not dissatisfied with the service received from chat operators at partner institutions, or by service provided by non-librarians. current policies for scheduling, handling shift changes, and service closure are appropriate, but best practices related to disclosing institutional mismatches may need to be changed. this exercise demonstrates that institutions can trust the consortium with their local users’ needs, and underscores the need for periodic service review.     introduction   chat reference has become increasingly common since its inception in the mid-1990s, and is now an integral part of library reference services (radford & kern, 2006). a study by yang and dalal (2015) found that 48% of college and university libraries in north america offer a chat service. almost a quarter of these libraries provide chat service through a consortium, and the trend toward collaboration is increasing (pomerantz, 2006; yang & dalal, 2015).   chat reference is more resource-intensive than traditional in-person service due to labor and software costs (weak & luo, 2014). many institutions find it difficult to launch or maintain a local chat service for budgetary or staffing reasons, especially if usage is low (eakin & pomerantz, 2009; helfer, 2003; radford & kern, 2006). in an effort to make chat reference more cost-efficient and sustainable, many libraries have joined consortial arrangements (coffman & arret, 2004b; peters, 2002; powers, nolen, zhang, xu, & peyton, 2010). by coming together, libraries can mitigate the risks of launching a new service, build a centralized infrastructure, share costs and staffing demands, extend service hours, and tap into a larger target audience to increase service usage (bailey-hainer, 2005; breeding, 2001; coffman & arret, 2004a).   service quality is often a point of concern with consortial chat reference services (meert & given, 2009). many libraries express doubt that staff from outside their institution can respond to their users’ questions effectively, especially queries that are local in nature (berry, casado, & dixon, 2003; bishop, 2011). the appropriate staffing for collaborative chat services is also a matter of debate. approximately 39% of academic libraries rely on paraprofessional staff or library school students to staff a consortial chat reference service (devine, bounds-paladino, & davis, 2011). while expanding the operator pool beyond librarians is a cost-effective way to make up staffing deficits and extend service hours into the evenings and weekends (blonde, 2006), there is some resistance to the practice, as librarians are considered the appropriate staffing level for answering research and reference questions (weak & luo, 2014).   most of the literature about consortial chat services concerning service quality focuses on the completeness and correctness of librarians’ responses and staff members’ adherence to behavioral guidelines. although some studies have reported on user satisfaction, no studies have investigated factors affecting user dissatisfaction in the consortial context. this paper attempts to fill the gap by reporting on an evaluation of an academic library consortium’s chat reference service. using transcript analysis and exit survey responses, the researchers examined whether the consortium’s collaborative service model, staffing choices, and policies contributed to user dissatisfaction.   literature review   effectiveness of the consortial model   location-based questions   collaborative chat reference requires participants to respond to questions concerning unfamiliar libraries or locations. this adds a layer of complexity to the reference transaction, as answering questions from across the consortium may require local knowledge, the practical, collective knowledge that is rooted in a particular place and based on the immediacy of experience (geertz, 1983, p. 75). researchers have tried to estimate the proportion of chat questions that require local knowledge. bishop (2011) refers to these queries as location-based questions, and defines them as questions that concern the geography of a library location or its attributes, such as its policies, services, or collections (bishop, 2012, 2013). eight studies have reported the quantity of location-based questions; they accounted for an average of 35% of total chat volume (berry et al., 2003; bishop, 2011, 2012; bishop & torrence, 2008; coté, kochkina, & mawhinney, 2016; hyde & tucker-raymond, 2006; kwon, 2007; sears, 2001).   evidence regarding consortial partners’ ability to answer location-based questions is mixed. kwon (2007) found that local-specific questions are answered less completely than non-local queries and noted lower user satisfaction among patrons with local-specific questions. bishop (2011) recorded a 45% referral rate for location-based questions, with non-local librarians referring significantly more than local librarians. however, the correctness of responses to location-based questions does not differ greatly between local and non-local librarians (bishop, 2012).   consortial service quality   researchers have also examined the quality of service provided by consortial chat services. meert and given (2009) assessed the chat service of an academic library participating in a 24/7 consortium, comparing local and consortial staff’s adherence to the library’s in-house reference quality standards. adherence was high overall, with local staff meeting standards more often than non-local staff (94% vs. 82%, respectively). consortial staff were less likely to answer questions in real time and made referrals at a higher rate than local staff. similarly, an evaluation of oregon’s statewide chat consortium uncovered that guidelines were met in 62% of interactions, but staff had difficulties working with non-local users, including making referrals (hyde & tucker-raymond, 2006). while consortial operators often rely on referrals as a strategy to handle non-local users’ queries (bishop, sachs-silveira, & avet, 2011), user satisfaction with referrals is significantly lower than for completed chats. referred users experienced the same degree of satisfaction as patrons who received a partial answer or no answer at all (kwon, 2006).   despite these weaknesses, consortial staff are capable of answering users’ questions accurately, although they may take a different approach than local chat operators. brown (2017) examined transcripts at a community college participating in questionpoint’s 24/7 reference. he found that answers from consortial back-up staff were largely correct, but they often provided more information rather than taking on an instructional role. peer-review of transcripts from the statewide ncknows chat consortium found that external staff from the 24/7 reference company received similar scores for skill in research and information use to local librarians, but were rated lower on engagement with the user (pomerantz, luo, & mcclure, 2006).   users are largely satisfied with the service provided by consortial or collaborative chat reference services. for example, the university of maryland university college’s chat service, which partially outsources staffing to provide 24/7 service, has a 90% approval rating (rawson, davis, harding, & miller, 2012). kwon (2007) examined exit survey responses for a large public library system’s chat reference service and found that the results were positive: 65% of users were satisfied with the answer provided, 68% stated that the librarian’s handling of the question was excellent, and 77% of patrons would use the service again. satisfaction did not differ significantly based on the user’s question type.   in addition to overall satisfaction, one study compared satisfaction with different types of staff members within a collaboration. hill, madarash-hill, and allred (2007) compared user satisfaction with local librarians, librarians from partner libraries in the local area, and staff from tutor.com’s librarians by request on southeastern louisiana university’s chat service. local librarians received higher satisfaction scores than external librarians overall, but the partner librarians did receive higher satisfaction scores than local librarians in some categories. notably, satisfaction scores for external librarians concerning the quality of answers, friendliness, overall service, and willingness to return rose over time, indicating that non-local librarians’ performance improves as familiarity with non-local libraries and campuses grows.   appropriateness and effectiveness of student staffing   there has been significant debate about the appropriateness of using student employees to staff in-person and online reference services. several studies have argued that relying on professional librarians alone to staff a reference desk or chat service is cost-ineffective (bracke et al., 2007; bravender, lyon, & molaro, 2011; ryan, 2008). case studies have also reported a high proportion of simple directional or technology questions at the reference desk, suggesting that many transactions do not require the skills of a librarian (bishop & bartlett, 2013; ryan, 2008; stevens, 2013). however, there are conflicting findings about the most common question types on chat. bravender et al. (2011) and cabaniss (2015) reported that reference questions accounted for 17.7% and 23.3% of chats on their respective services, leading them to recommend staffing models in which graduate students or reference assistants handle the majority of chats. however, other researchers have reported that complex research or reference questions occur in 40%–66% of chats, supporting staffing by professional librarians (coté et al., 2016; fuller & dryden, 2015; morais & sampson, 2010).   studies assessing the quality of service provided by student workers have largely been positive. at the reference desk, case studies have shown that student employees receive comparable satisfaction ratings to librarians and score well on measures of approachability and helpfulness (faix, 2014; stevens, 2013). on chat reference, transcript analysis by lux and rich (2016) found that student employees offered quality assistance in 88% of transactions. while the reference librarians outperformed the student workers in most measures of comparison, the margin between them was not large. keyes and dworak (2017) also found that librarians outperformed students in their transcript analysis study. however, there was no significant association between staffing type and patron ratings. both research teams argued that student workers are capable of providing chat reference services and can improve on their weaknesses through training. in particular, many student workers deviate from the reference and user services association’s (rusa) best practices; they often fail to conduct a thorough reference interview and communicate in an overly informal style (barrett & greenberg, 2018; langan, 2012). guiding students through the reference interview to provide appropriate behavioral benchmarks and reviewing transcripts can increase an awareness of reference standards among student workers (langan, 2012; ward, 2003).   aims   the consortial context   the ontario council of university libraries (ocul) is a consortium representing the libraries of all 21 universities in the province of ontario, canada. collectively, these universities have a student population of over 480,000, representing approximately one third of the university population of canada.   ocul leverages collective resources to purchase, manage, and preserve electronic collections, and provides access to them through a digital infrastructure offered by scholars portal (sp), the consortium’s service arm. ocul’s largest member, the university of toronto libraries (utl), acts as the service provider. sp supports a wide range of content repositories, member services, and technical services in the areas of collections, resource sharing, research services, and digital preservation.   ask a librarian is a virtual reference service managed by sp that connects students, faculty members, and researchers from participating university libraries across ontario with real-time library and research assistance through chat. the service launched in 2011 as a partnership among seven ocul libraries and has since expanded to 15 of the 21 ocul members. the service reaches approximately 400,000 full-time equivalent students and handles roughly 25,000 chats per year. since 2014, the service has also been offered in french under the name clavardez avec nos bibliothécaires (“chat with our librarians”) at five libraries.   ask a librarian is open 67 hours per week during the academic year. staffing is managed through a collaborative model in which libraries provide staffing hours relative to their student populations and service usage patterns. during evenings and weekends, staffing is supplemented by part-time virtual reference operators (vros), generally second-year lis students or recent graduates, hired by ocul directly.   consortial analysis   in 2012, one year after the initial implementation of ask a librarian, sp staff conducted a research project investigating the types of questions asked on the service, the academic status and location of users, and overall user satisfaction. in 2017, after an influx of new partners, the introduction of bilingual service, and changes in chat software, a joint research team at sp and utl began another research project, building upon the previous work. this major transcript analysis sought to investigate a wide range of questions about virtual reference.   as one segment of the broader analysis, this paper focuses on the service model, policies, and practices of ask a librarian as a consortial virtual reference service. the aim was to determine whether the current collaborative model is providing appropriate and satisfactory service to local users. since user feedback tends to be very positive overall, the researchers intentionally sought out points of dissatisfaction in order to highlight any weaknesses in the service. to that end, the research questions were:   r1: are dissatisfaction levels higher for some types of users or some categories of questions? r2: do users experience increased levels of dissatisfaction when served by an operator from another institution? do levels of dissatisfaction increase if the user is made aware that the operator is from another institution? r3: do users experience increased levels of dissatisfaction when served by student staff? r4: do busy shifts have an effect on user dissatisfaction? r5: do questions submitted around shift change times or the service’s closure have higher rates of user dissatisfaction? do levels of dissatisfaction increase if the user is told that a shift change or service closure is approaching?   the answers to these questions will help determine if the current collaborative model, as well as our policies and procedures around issues such as staffing levels and instructions to operators for handling events like shift changes, are appropriate and successful.   methods   the researchers received approval for this study from the university of toronto’s research ethics board and ocul’s ask a librarian data working group.   data collection and sampling   the researchers reviewed chats that took place between june 1 and december 1, 2016. during this period, 9,424 chats were submitted to the service. complete chat transcripts, responses to the question initiation form, and chat metadata were available for each interaction through the chat software. of the 9,424 chats that took place during this period, 1,395 interactions (14.8%) had a corresponding completed exit survey.   only chats with completed exit surveys were eligible for sampling. four of the eight exit survey questions assess the user’s satisfaction with the interaction; only responses to these questions were examined in this study. the researchers used an excel spreadsheet to identify chat interactions that had corresponding exit surveys with only satisfied responses, and interactions that had exit surveys with either neutral or dissatisfied responses. the exit survey questions and examples of satisfied, neutral, and dissatisfied responses are listed in the appendix.   a total of 473 chats were sampled according to the following procedures:   a sample of 256 chat interactions with satisfied exit survey responses was randomly selected using excel, representing 18% of all chat interactions with completed exit surveys (n = 1,395). this sample size was chosen because it provides a confidence level of 95%. all 217 chat interactions with corresponding exit surveys reporting anything less than satisfaction were included in the sample. this included any chats with at least one exit survey response that was neutral or dissatisfied. this sampling method was chosen because only 16% of eligible interactions met this criterion. homogenous purposive sampling allowed us to draw on as much data as possible to investigate the experiences of dissatisfied users.   data preparation   the researchers compiled the chat session metadata, responses to the question initiation form, and exit survey responses pulled from the chat software into an excel spreadsheet. chat session metadata included operator type, whether the user and operator were from the same institution, the time the chat was initiated, and whether the shift was busy. the question initiation form included user type and question type. the exit survey responses related to user dissatisfaction.   the researchers anonymized the spreadsheet data according to standards set by the consortium’s data working group. any identifying information, such as the identity of the chat operator, the user, or the institutional affiliation of either individual, was removed. the same process was used to anonymize the corresponding chat transcripts.   study variables   the researchers recorded information related to the study variables in the same spreadsheet containing the data extracted from the software.   user type   users identified their status with the university through a mandatory question initiation form. the options were: undergraduate student, graduate student, faculty, alumni, or other.   operator type   the operator(s) who participated in the chat interaction were listed in the chat metadata. the researchers recorded whether they were librarians, paraprofessionals, part-time virtual reference operators employed by the consortium, students (graduate student workers employed directly by participating libraries), or of different types.   question type   users were asked to provide a detailed description of their question in a mandatory question initiation form. the researchers coded their responses by question type according to a schema that was previously developed by local researchers (maidenberg, greenberg, whyte appleby, logan, & spence, 2012). the question type categories are: accounts, citation, e-resources, facilities, computing, miscellaneous, non-library, policies, research, and writing.   institutional mismatch and institutional mismatch reveal   the institutional affiliation of the operator and user were listed in the software’s chat metadata. the researchers recorded whether the participants in the chat were associated with the same institution or whether there was a mismatch. through transcript analysis, the researchers recorded chats in which the operator disclosed that they did not have the same institutional affiliation or home campus as the user.   busy shift   the chat session metadata listed the time at which the chat was initiated. from this information, the researchers determined the shift during which the chat took place. shifts are an hour in length. the researchers consulted sp’s chat volume statistics to determine how many chats were submitted during that same shift. busyness was determined based on the number of chats submitted during the shift, compared to the number of operators scheduled to be online during the shift. a shift was considered busy if more than three chats were submitted for every available operator.   aspects of scheduling   the chat session metadata recorded the time at which the chat was initiated. the researchers recorded whether the chat began during the last 10 minutes of the shift or within 10 minutes of the time the service was scheduled to close. through transcript analysis, the researchers also noted whether the operator disclosed any information about their shift schedule or about the service’s hours (i.e., whether they were about to go off shift or the service was closing soon).   dissatisfaction   based on the exit survey responses associated with the chat interaction, the researchers recorded whether the user was dissatisfied or not dissatisfied. users were considered dissatisfied if they answered at least one of the four exit survey questions related to satisfaction (appendix) with a neutral or dissatisfied response.   coding   question type   question type was coded by two members of the research team. the researchers coded an initial test set of 42 transcripts and achieved substantial intercoder agreement, as measured by cohen’s kappa, k = 0.794. after discussing discrepancies, the researchers coded a second test set of 44 transcripts. they achieved near perfect agreement, as measured by cohen’s kappa, k = 0.876.   transcripts   as part of a larger service evaluation project, transcripts were coded for 30 variables hypothesized to effect user dissatisfaction, including two variables in the present study: institution mismatch reveal and schedule reveal. the four-member research team coded a test set of 15 transcripts using a draft codebook and coding form to establish intercoder reliability. the team met to discuss discrepancies, refined the definitions and examples in the codebook, and then coded a second test set of 10 transcripts. the researchers assessed intercoder reliability using average pairwise percent agreement, which was set at a threshold of 80%. for the second test set, average pairwise percent agreement was 93.3% for institution mismatch reveal and 95% for schedule reveal.   data compilation and analysis   once transcript coding was completed, the data from the coding form was merged with the spreadsheet containing the chat metadata, survey responses, and information for the other study variables. pearson chi-square tests of independence were conducted in spss to determine if there were significant relationships between variables, with a significance level of p < 0.05 set a priori. the researchers then entered the variables into a binary logistic regression model to determine the strength and directionality of the variables’ effects.   results   the researchers ran eight pearson chi-square tests of independence to determine if there was a significant relationship between user dissatisfaction and aspects of ask a librarian’s service model and staffing and scheduling practices. two variables had a significant relationship with user dissatisfaction at an alpha level of 0.05: operator type, χ2 (4, n = 473) = 25.513, p < 0.001, and institution mismatch reveal, χ2 (1, n = 473) = 4.323, p = 0.038. the remaining variables were not significantly related to dissatisfaction. the results of each chi-square test of independence are available in table 1.   next, we entered the variables into a binary logistic regression, in order to determine how well the variables, taken together, can explain or predict dissatisfaction, as well as to understand the significance, strength, and directionality of the individual variables’ effects. the overall model was statistically significant, χ2 (22, n = 473) = 63.087, p < 0.001, meaning that it was statistically reliable in distinguishing between satisfied and dissatisfied patrons. the model did not have strong predictive power, represented by a nagelkerke r2 of 0.167. nagelkerke’s r2 is a measure relating to the goodness of fit of the model, and can range from 0 to 1. the model was correct in predicting the outcome (i.e., whether the user was dissatisfied) in 64.9% of cases.   in the regression model, there were two significant explanatory variables at the 0.05 alpha level: operator type and institutional mismatch reveal. within the operator type category, the part-time virtual reference operator type was a significant, negative variable within the model (b = -1.065, p = 0.008). this means that dissatisfaction decreased if users were served by graduate student staff or recent graduates hired by the consortium. the other operator types did not significantly contribute to dissatisfaction. institutional mismatch reveal was a positive variable in the model, indicating that users were more likely to be dissatisfied if the operator revealed they were not at the user’s home institution (b = 0.875, p = 0.009).   table 1 summary of one-tailed chi-square tests of independence by variable variable dissatisfied not dissatisfied pearson χ2 df. sig. user type observed expected observed expected 8.010 4 .091 undergraduate student 129 120.7 134 142.3       graduate student 56 55.5 65 65.5       faculty 13 11.9 13 14.1       alumni 7 8.7 12 10.3       other 12 20.2 32 23.8       operator type observed expected observed expected 25.513 4 .000* librarian 80 78.5 91 92.5       paraprofessional 74 60.6 58 71.4       part-time virtual reference operator 25 44 71 52       student 24 24.8 30 29.2       mixed 14 9.2 6 10.8       question type observed expected observed expected 14.714 9 .099 accounts 14 18.8 27 22.2       citation 28 20.6 17 24.4       e-resources 12 16.5 24 19.5       facilities 6 5.5 6 6.5       computing 5 6 8 7       miscellaneous 8 9.6 11.4 5       non-library 2 3.2 5 3.8       policies 16 18.4 24 21.6       research 124 117.4 132 138.6       writing 2 9 0 1.1       institutional mismatch observed expected observed expected 0.073 1 .787 match 84 82.6 96 97.4       mismatch 133 134.4 160 158.6       institutional mismatch reveal observed expected observed expected 4.323 1 .038* revealed 34 26.6 24 31.4       did not reveal 183 190.4 232 224       busy shift observed expected observed expected .745 1 .388 busy 34 30.7 33 36.3       not busy 183 186.3 223 219.7       chat initiated within 10 minutes of end of shift / service closure observed expected observed expected 2.773 1 .096 initiated within 10 minutes 41 34.4 34 40.6       not initiated within 10 minutes 176 182.6 222 215.4       reveal of aspects of scheduling observed expected observed expected 3.202 1 .074 revealed 39 32.1 31 37.9       did not reveal 178 184.9 225 218.1       note. df. = degrees of freedom; sig. = significance. *denotes that relationship is significant at an alpha level of 0.05.   table 2 summary of binary logistic regression variable category b s.e. wald df. sig. exp(b) user type       6.993 4 .136     undergraduate student .558 .579 .930 1 .335 1.747   graduate student .331 .588 .317 1 .573 1.393   faculty .840 .696 1.455 1 .228 2.316   other -.368 .648 .322 1 .570 .692 operator type       26.860 4 .000*     librarian .125 .339 .135 1 .713 1.133   paraprofessional .548 .349 2.459 1 .117 1.730   part-time virtual reference operator -1.065 .400 7.099 1 .008* .345   mixed .777 .595 1.703 1 .192 2.175 question type       12.147 9 .205     accounts -21.661 .000 28257.649 1 .999 .000   citation -20.417 .000 28257.649 1 .999 .000   e-resources -21.657 .000 28257.649 1 .999 .000   facilities -21.012 .000 28257.649 1 .999 .000   computing -21.812 .000 28257.649 1 .999 .000   miscellaneous -21.354 .000 28257.649 1 .999 .000   non-library -22.236 .000 28257.649 1 .999 .000   policies -21.355 .000 28257.649 1 .999 .000   research -21.000 .000 28257.649 1 .999 .000 institutional mismatch   -.299 .225 1.757 1 .185 .742 institutional mismatch reveal   .875 .337 6.750 1 .009* 2.399 busyness of the shift   -.007 .293 .001 1 .981 .993 chat initiated within 10 minutes of end of shift / service closure   .284 .276 1.059 1 .304 1.328 reveal of aspects of scheduling   .363 .297 1.494 1 .222 1.437 note. b = coefficient, s.e. = standard error, wald = wald chi-square test (which tests the null hypothesis); df. = degrees of freedom; sig. = significance; exp(b) = odds ratio. *denotes that relationship is significant at an alpha level of 0.05.   discussion   this analysis did not find a statistically significant relationship between dissatisfaction and user or question type (research question 1), indicating that ask a librarian provides a consistent level of service to all patrons and satisfactorily answers all types of libraryand research-related questions. the results largely reaffirm the consortium’s service model, staffing practices, and policies. dissatisfaction levels did not show relationships with most of the factors examined, indicating that overall service is appropriate and satisfactory. in particular, busy shifts and chats initiated near shift change times or service closure (research questions 4 and 5) had no relationship with dissatisfaction, suggesting that ask a librarian’s scheduling practices and policies for handling shift changes are appropriate.   consortial service quality and institutional mismatch   the analysis found no relationship between institution match and dissatisfaction, indicating that users can be served by operators across the consortium without compromising patron satisfaction. this fits into the literature that finds that users tend to be satisfied with consortial or collaborative chat reference (kwon, 2007; rawson et al., 2012).   the nature of ocul as a purchasing, advocacy, and service-providing consortium means that there are deep levels of collaboration between institutions, which tend to have access to similar resources. this may make it easier for operators from one institution to successfully answer questions from another, consistent with the findings of hill et al. (2007) that satisfaction scores for external librarians in collaborative chat improved as their familiarity with the user’s library increased. therefore, the finding that users are satisfied by service from operators at partner institutions is not necessarily generalizable to all consortia, and particularly large, multi-type consortia such as the one bishop (2011, 2012) found inadequate for answering local questions.   the reveal of an institution mismatch was associated with user dissatisfaction. this is an area that has not been widely studied and the authors were unable to find other literature to help provide context, making this a fruitful area for potential future research. this finding especially requires further investigation to rule out confounding factors. users may simply be more dissatisfied when they learn that they are not being served by their own local library, but the authors’ current hypothesis is that operators are more likely to reveal that they are from another institution if they are unable to answer the user’s question, or if the chat is otherwise going poorly. pending more analysis, sp will consider changing ask a librarian policies to recommend against revealing an institution mismatch unless absolutely necessary.   appropriate and effective student staffing   the results show that users do not express dissatisfaction with the service of non-librarians, and in fact show a slight preference for graduate student staff hired by the consortium. this aligns with earlier literature indicating users find student staff to be approachable and helpful (stevens, 2013) and that they provide high-quality assistance via chat, although not as high-quality as librarians (keyes & dworak, 2017; lux & rich, 2016). however, it is also important to note that the student staff of ask a librarian are all lis graduate students who have taken at least one reference course. as such, they may perform more like librarians than undergraduate students and non-lis graduate students staffing similar services (for example, in terms of following rusa best practices). however, as noted above, this study did not examine response completeness or accuracy as other studies have done.   this finding reinforces ask a librarian’s use of student staff to supplement evening and weekend shifts as an appropriate way to extend reference services beyond the normal working hours of reference librarians.   limitations   beyond the generalizability of specific findings, there are a few limitations to this study. in examining consortial service quality, the researchers did not identify whether the questions required local knowledge, as bishop (2011, 2012) and other researchers have done. satisfaction was reported by users in an exit survey, which was only presented when the operator ended the chat, or when the user clicked an “end chat” button; users who simply closed the window did not see it. self-reported satisfaction scores are also not always reliable measures as they can introduce the user’s bias, and user satisfaction is only one measure of an interaction’s success. this study did not examine other quality metrics, such as response accuracy or completeness or adherence to behavioural standards like rusa guidelines. other factors, including those discussed in the further research section below, influence user satisfaction and therefore may complicate the relationships discussed here. the quantitative analysis for this study did not include any moderating variables that may partially explain relationships.   further research   the research team is already conducting further analysis on the same dataset, building on previous knowledge of what affects dissatisfaction in reference transactions. articles on how operator behaviour and communication styles impact user dissatisfaction are already published (logan & barrett, 2019; logan, barrett, & pagotto, 2019), and work has begun to study instruction and referrals in chat.   more in-depth research is needed to flesh out the nuances of the relationships uncovered in this paper. qualitative research, in particular, could complement these findings by disentangling what leads users to give low scores on the exit survey.    finally, while ask a librarian is a bilingual service, the number of french interactions was so small that it was not feasible to analyze any differences between english and french user satisfaction. this is an area the researchers hope to examine in more depth in the future.   conclusions   as a collaborative chat service, ask a librarian was launched to leverage shared resources and provide cost-effective reference service to ontario university libraries. its service model and policies were developed based on standards and best practices informed by other virtual reference practitioners. now that ask a librarian has grown into a mature service, a review is important to ensure that the model and policies are backed by evidence.   the study largely reaffirmed the consortium’s service model, staffing practices, and policies. users are not dissatisfied with the service received from chat operators at partner institutions or by service provided by non-librarians. current policies for scheduling, service closure, and handling shift changes are appropriate. best practices related to disclosing institutional mismatches may need to be changed, as these reveals were associated with higher levels of dissatisfaction. this is an area that merits further investigation.   no areas of weakness were uncovered, indicating that ask a librarian provides appropriate and satisfactory service to all different user types and for all different question types. overall, this research demonstrates that institutions can trust the consortium with their local users’ virtual reference needs.   acknowledgments   the authors would like to acknowledge the other members of the research team, judith logan (university of toronto libraries) and amy greenberg (scholars portal).   references bailey-hainer, b. (2005). virtual reference: alive & well. library journal, 130(1), 46-47.   barrett, k., & greenberg, a. (2018). student-staffed virtual reference services: how to meet the training challenge. journal of library & information services in distance learning, 12(3-4), 101-119. https://doi.org/10.1080/1533290x.2018.1498620   berry, t. u., casado, m. m., & dixon, l. s. (2003). the local nature of digital reference. southeastern librarian, 51(3), 8-15. retrieved from https://digitalcommons.kennesaw.edu/seln/vol51/iss3/5   bishop, b. w. (2011). location-based questions and local knowledge. journal of the american society for information science and technology, 62(8), 1594-1603. https://doi.org/10.1002/asi.21561   bishop, b. w. (2012). can consortial reference partners answer your local users’ library questions? portal: libraries and the academy, 12(4), 355-370. https://doi.org/10.1353/pla.2012.0036   bishop, b. w. (2013). location-based questions: types and implications for consortial reference services. proceedings of the annual conference of cais. retrieved from https://journals.library.ualberta.ca/ojs.cais-acsi.ca/index.php/cais-asci/article/view/546/496   bishop, b. w., & bartlett, j. a. (2013). where do we go from here? informing academic library staffing through reference transaction analysis. college & research libraries, 74(5), 489-500. https://doi.org/10.5860/crl-365   bishop, b. w., sachs-silveira, d., & avet, t. (2011). populating a knowledge base with local knowledge for florida’s ask a librarian reference consortium. the reference librarian, 52(3), 197-207. https://doi.org/10.1080/02763877.2011.555289   bishop, b. w., & torrence, m. (2008). virtual reference services: consortium versus stand-alone. college & undergraduate libraries, 13(4), 117-127. https://doi.org/10.1300/j106v13n04_08   blonde, j. (2006). staffing for electronic reference: balancing service and sacrifice. in r. d. lankes, m. d. white, e. g. abels, & s. n. haque (eds.), the virtual reference desk: creating a reference future (pp. 75-87). new york, ny: neal-schuman publishers, inc.   bracke, m. s., brewer, m., huff-eibl, r., lee, d. r., mitchell, r., & ray, m. (2007). finding information in a new landscape: developing new service and staffing models for mediated information services. college & research libraries, 68(3), 248-267. https://doi.org/10.5860/crl.68.3.248   bravender, p., lyon, c., & molaro, a. (2011). should chat reference be staffed by librarians? an assessment of chat reference at an academic library using libstats. internet reference services quarterly, 16(3), 111-127. https://doi.org/10.1080/10875301.2011.595255   breeding, m. (2001). providing virtual reference service. information today, 18(4), 42-43.   brown, r. (2017). lifting the veil: analyzing collaborative virtual reference transcripts to demonstrate value and make recommendations for practice. reference & user services quarterly, 57(1), 42-47. https://doi.org/10.5860/rusq.57.1.6441   cabaniss, j. (2015). an assessment of the university of washington’s chat reference services. public library quarterly, 34(1), 85-96. https://doi.org/10.1080/01616846.2015.1000785   coffman, s., & arret, l. (2004a). to chat or not to chat—taking another look at virtual reference: part 1. searcher, 12(7), 38-46.   coffman, s., & arret, l. (2004b). to chat or not to chat: taking yet another look at virtual reference. searcher, 12(8), 49-56.   coté, m., kochkina, s., & mawhinney, t. (2016). do you want to chat? reevaluating organization of virtual reference service at an academic library. reference & user services quarterly, 56(1), 36-46. https://doi.org/10.5860/rusq.56n1.36   devine, c., bounds paladino, e., & davis, j. a. (2011). chat reference training after one decade: the results of a national survey of academic libraries. the journal of academic librarianship, 37(3), 197-206. https://doi.org/10.1016/j.acalib.2011.02.011   eakin, l., & pomerantz, j. (2009). virtual reference, real money: modeling costs in virtual reference services. portal: libraries and the academy, 9(1), 133-164. https://doi.org/10.1353/pla.0.0035   faix, a. (2014). peer reference revisited: evolution of a peer-reference model. reference services review, 42(2), 305-319. https://doi.org/10.1108/rsr-07-2013-0039   fuller, k., & dryden, n. h. (2015). chat reference analysis to determine accuracy and staffing needs at one academic library. internet reference services quarterly, 20(3-4), 163-181. https://doi.org/10.1080/10875301.2015.1106999   geertz, c. (1983). local knowledge: further essays in interpretive anthropology. new york: basic books.   helfer, d. s. (2003). virtual reference in libraries: status and issues. searcher, 11(2), 63-65.   hill, j. b., madarash-hill, c., & allred, a. (2007). outsourcing digital reference: the user perspective. the reference librarian, 47(2), 57-74. https://doi.org/10.1300/j120v47n98_06   hyde, l., & tucker-raymond, c. (2006). benchmarking librarian performance in chat reference. the reference librarian, 46(95-96), 5-19. https://doi.org/10.1300/j120v46n95_02   keyes, k., & dworak, e. (2017). staffing chat reference with undergraduate student assistants at an academic library: a standards-based assessment. the journal of academic librarianship, 43(6), 469-478. https://doi.org/10.1016/j.acalib.2017.09.001   kwon, n. (2006). user satisfaction with referrals at a collaborative virtual reference service. information research, 11(2). retrieved from http://informationr.net/ir/11-2/paper246.html   kwon, n. (2007). public library patrons’ use of collaborative chat reference service: the effectiveness of question answering by question type. library and information science research, 29(1), 70-91. https://doi.org/10.1016/j.lisr.2006.08.012   langan, k. (2012). training millennials: a practical and theoretical approach. reference services review, 40(1), 24-48. https://doi.org/10.1108/00907321211203612   logan, j., barrett, k., & pagotto, s. (2019). dissatisfaction in chat reference users: a transcript analysis study. college & research libraries, 80(7), 925-944. https://doi.org/10.5860/crl.80.7.925   logan, j., & barrett, k. (2019). how important is communication style in chat reference? internet reference services quarterly, 23(1-2), 41-57. https://doi.org/10.1080/10875301.2019.1628157   lux, v. j., & rich, l. (2016). can student assistants effectively provide chat reference services? student transcripts vs. librarian transcripts. internet reference services quarterly, 21(3-4), 115-139. https://doi.org/10.1080/10875301.2016.1248585   maidenberg, k., greenberg, a., whyte-appleby, j., logan, j., & spence, m. (2012). reference query coding key. retrieved from http://hdl.handle.net/1807/94126   meert, d. l., & given, l. m. (2009). measuring quality in chat reference consortia: a comparative analysis of responses to users’ queries. college & research libraries, 70(1), 71-84. https://doi.org/10.5860/0700071   morais, y., & sampson, s. (2010). a content analysis of chat transcripts in the georgetown law library. legal reference services quarterly, 29(3), 165-178. https://doi.org/10.1080/02703191003751289   peters, t. a. (2002). e-reference: how consortia add value. journal of academic librarianship, 28(4), 248-250. https://doi.org/10.1016/s0099-1333(02)00310-5   pomerantz, j. (2006). collaboration as the norm in reference work. reference & user services quarterly, 46(1), 45-55. https://doi.org/10.5860/rusq.46n1.45   pomerantz, j., luo, l., & mcclure, c. r. (2006). peer review of chat reference transcripts: approaches and strategies. library & information science research, 28(1), 24-48. https://doi.org/10.1016/j.lisr.2005.11.004   powers, a. c., nolen, d., zhang, l., xu, y., & peyton, g. (2010). moving from the consortium to the reference desk: keeping chat and improving reference at the msu libraries. internet reference services quarterly, 15(3), 169-188. https://doi.org/10.1080/10875301.2010.500939   radford, m. l., & kern,  m. k. (2006). a multiple-case study investigation of the discontinuation of nine chat reference services. library & information science research, 28(4), 521-547. https://doi.org/10.1016/j.lisr.2006.10.001   rawson, j., davis, m. a., harding, j., & miller, c. (2012). virtual reference at a global university: an analysis of patron and question type. journal of library & information services in distance learning, 7(1-2), 93-97. https://doi.org/10.1080/1533290x.2012.705624   ryan, s. m. (2008). reference transactions analysis: the cost-effectiveness of staffing a traditional academic reference desk. the journal of academic librarianship, 34(5), 389-399. https://doi.org/10.1016/j.acalib.2008.06.002   sears, j. (2001). chat reference service: an analysis of one semester’s data. issues in science and technology librarianship, 32, 200-206. https://doi.org/10.5062/f4cz3545   stevens, c. r. (2013). reference reviewed and re-envisioned: revamping librarian and desk-centric services with libstars and libanswers. the journal of academic librarianship, 39(2), 202-214. https://doi.org/10.1016/j.acalib.2012.11.006   ward, d. (2003). using virtual reference transcripts for staff training. reference services review, 31(1), 46-56. https://doi.org/10.1108/00907320310460915   weak, e., & luo, l. (2014). collaborative virtual reference service: lessons from the past decade. advances in librarianship, 37, 81-112. https://doi.org/10.1108/s0065-2830(2013)0000037008   yang, s. q., & dalal, h. a. (2015). delivering virtual reference services on the web: an investigation into the current practice by academic libraries. journal of academic librarianship, 41(1), 68-86. https://doi.org/10.1016/j.acalib.2014.10.003     appendix exit survey questions assessing user satisfaction   the following questions were included in the current study. responses in bold were identified as dissatisfied, responses in italics were classified as neutral, and those with no text effects were considered satisfied.   1.       the service provided by the librarian was a.       excellent b.       good c.        satisfactory d.       poor e.       very poor   2.       the library provided me with a.       just the right amount of assistance b.       too little assistance c.        too much assistance   3.       this chat service is a.       my preferred way of getting library help b.       a good way of getting library help c.        a satisfactory way of getting library help d.       a poor way of getting library help e.       a last resort for getting library help   4.       would you use this service again? a.       yes b.       no   the following questions also appear on the exit survey, but were not included in this study.   1.       was this your first time using the service? a.       yes b.       no   2.       where were you when you chatted with us today? a.       off campus b.       on campus but not in the library c.        in the library   3.       how did you find out about this service? (users could select more than one response.) a.       library website b.       librarian c.        library instruction session d.       friend e.       professor or ta f.        promotional material (poster, flyer, etc.) g.       social media h.       other (free text response)   4.       other feedback or suggestions (free text response)       news   eblip panel presentation at 2018 american library association annual conference      2018. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29425     four professionals engaged in eblip in school, public, and academic library sectors will present “evidence based library and information practice is . . . “as part of a panel at the 2018 ala annual conference to be held in new orleans, louisiana, united states.   this presentation will introduce the basic techniques of evidence based library and information practice (eblip) and also offer examples from libraries and librarians that have already embraced the practice. the panel of librarians, who have adopted eblip concepts and principles, will share how evidence-based librarianship has impacted their libraries, library practice, research agendas, and patron interactions. participants will receive an overview of evidence based library and information practice theory and research; learn about evidence-based frameworks; and discover how evidence-based decision making improves libraries. the presentation will draw heavily from recent research by koufogiannakis & brettle (2016) and their book being evidence based in library and information practice.   participants will learn what evidence-based librarianship is, what it looks like, and suggestions on how to begin incorporating eblip into their own day-to-day practice.   date and time: monday 25 june 2018, 9 a.m.-10 a.m.   moderator/panelist: trish chatterley, collections strategies coordinator, university of alberta   panelists: lee ann fullington, health & environmental sciences librarian, brooklyn college; hannah byrd little, director of library and archives, webb school; pam ryan, director, service development & innovation, toronto public library   location: consult the conference schedule when available.   research article   beyond reference data: a qualitative analysis of nursing library chats to improve research health science services   samantha harlow online learning librarian university of north carolina greensboro greensboro, north carolina, united states of america email: slharlow@uncg.edu   received: 25 aug. 2020                                                             accepted: 24 jan. 2021      2021 harlow. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29828     abstract   objective the objective of this study was to analyze trends in academic library reference chat transcripts with nursing themes, in order to improve all library services and resources based on the findings.   methods in fall 2018, health science liaison librarians performed a qualitative study by analyzing 60 nursing chat transcripts from libraryh3lp. these chats were tagged, anonymized, coded, and then analyzed in atlas ti to identify patterns and trends.   results chat analysis showed that librarians staffing chat are meeting the research needs of nursing patrons by helping them find full-text articles and suggesting the appropriate library databases. in order to further improve these virtual services, workshops were offered to library and information science (lis) interns and staff who answer reference chats. nursing online tutorials and research guides were also improved based on the results.   conclusion this study will help academic libraries improve and expand services into the virtual realm, to support library employees and patrons during the covid-19 pandemic and beyond. virtual reference chat is not going away; in the current academic environment it is needed more than ever. using these library chats as the basis for additional chat staff training can reduce staff anxiety and prepare them to better serve patrons.     introduction   the university of north carolina greensboro (uncg), a mid-sized public research university, has a nursing program with a strong online presence to accommodate the department’s large population of non-traditional students. non-traditional students are defined as students who fall into any one of the following groups: over 24 years of age; entry to college delayed by at least one year following high school; single parents; employed full-time; attending a postsecondary institution part-time; with dependents; financially independent; or not possessing a high school diploma (choy, 2002). according to the association of college & research libraries (acrl) standards for distance learning library services, academic libraries should provide equitable resources and services for all of their students, including those who learn and study online (2016). in order to meet the needs of all campus researchers and learners, the uncg university libraries provide a variety of information literacy services and instruction through the research, outreach, and instruction (roi) department, which houses academic librarian liaisons. through the roi department, liaisons and staff run a very popular virtual chat service, that receives thousands of chats a year. this department also helps students by providing many other virtual and face-to-face services, including: information literacy and research instruction; one-on-one consultations with students and faculty; research guides through springshare libguides platform; and a variety of canvas (learning management system) integrations, including a libguides lti (learning tools interoperability).   increasing numbers of nursing programs are moving online, and according to the “guide to online schools: accredited online nursing programs by state” website (2019), there are at least 380 online nursing schools in the us. uncg offers many nursing degrees and certificates, and the master's of science (ms) and post-baccalaureate certificate (pbc) in nursing administration and nursing education are offered fully online (university of north carolina greensboro, 2020). uncg has also identified health education as a focus area for the general student body, with “health and wellness” as a theme of the strategic plan; this plan characterizes health and wellness as “broadly defined to encompass the many dimensions necessary for individuals to cope, adapt, grow, and develop” (university of north carolina greensboro, 2013). nursing students are a strong online and non-traditional population, and with health and wellness as target areas for campus learning, it is more important than ever to provide effective and equitable health science library reference services to students. within the libraries, the health science liaison works with a variety of academic health science departments including nursing, while the online learning librarian is liaison to community and therapeutic recreation, kinesiology, and public health education. liaisons provide a variety of services to their departments, and since nursing has always had a strong online presence, the health science librarian offers virtual research consultations, webcasts on a variety of research topics, and online orientations for students.   based on this important student population, the growth of online learning and non-traditional students, and the popularity of the reference chat library service, these two health science liaison librarians performed a qualitative reference analysis on reference chat transcripts from nursing students and instructors. this study was performed in an effort to improve service offerings in several areas, but particularly chat services to nursing students; improving research services to nursing students also helps a variety of other patrons, including other health science departments and all students studying online. this study sought to answer the following research questions: what trends do we see in library chats based on nursing themes, rather than numbers and usage counts? what patterns exist in nursing chats within the library? how can we improve library services and resources based on nursing chat trends?   literature review   a virtual chat service is a vital synchronous online service for library patrons (both face-to-face and distance populations), and there are many studies on providing reference services to researchers through chat. some studies survey academic librarians about their chat reference services and how they train staff to answer chats (devine, paladino, & davis, 2011), while others explore the usefulness of having full-time librarians provide service through chat reference systems (maloney & kemp, 2015). many chat analysis projects take a large-scale qualitative approach by analyzing datasets of academic chat transcripts to show overall improvements over time (baumgart et al., 2016; brown, 2017; dempsey, 2019). mungin (2017) at james madison university analyzed chat transcripts in dedoose over a five-year span in order to improve chat reference; and as recently as 2019, at utah state university (which has a high population of students studying online), a group of librarians and learning technologists looked at chat trends over a year by analyzing 1600 chat transcripts through coding. based on the findings of this analysis, the group made training resources and best practice handouts for answering chats (eastman et al., 2019). in another chat analysis project, logan, barrett, and pagotto (2019) used coding to analyze almost 500 chat transcripts to find behaviors to avoid. with growing student populations and online services and resources, many libraries must rely more heavily on non-librarians and student workers staffing virtual reference systems. barrett and greenberg (2018) conducted a study proving the value of student workers by performing exit interviews with patrons served. in order to help librarians and their non-librarian colleagues better reach distant students, offering professional development through online research guide or courses within the university learning management system is therefore helpful (bliquez & deeken, 2016).   understanding the information and digital literacy needs of nursing undergraduate and graduate students can help improve library reference services for all students, regardless of whether they are studying online or face-to-face. librarians have long understood the need for virtual reference services for nursing students. guillot and stahr (2004) studied the efficiency of a virtual reference desk for nursing patrons at their university and found that distance nursing students valued the online research support system. many qualitative studies have been performed on the research needs of nursing students. interviewing nursing students who may be studying online can help librarians understand the unique life experiences of these students and their information and digital literacy skills (craig & corrall, 2007; duncan & holtslander, 2012; ledwell et al., 2006; reeves & fogg, 2006; stein & reeder, 2009). surveys are another method of understanding the information seeking needs of nursing students; for example, al-gamal and colleagues’ (2018) surveyed nursing students about stress and the coping strategies they used during their clinical rotations.   the increasing shift toward online nursing education means that it is more vital than ever to provide a variety of asynchronous virtual research training for nursing students who prefer this method of help over synchronous chats; research guides such as springshare libguides and online tutorials can help accommodate nursing students on their own time. nursing research guides can serve as portals for accessing virtual and physical collections (johnson & johnson, 2017). stankus and parker (2012) performed a study on nursing libguides across the us from a variety of libraries and found the information on the guides diverse and varied; there were some commonalities, such as inclusion of major medical databases and resources like ebsco’s cinahl, as well as pubmed/medline, and a focus on evidence based practice to inform research. libguides can house online tutorials on a variety of information literacy health science topics. online tutorials can include various multimedia such as videos, pdfs, presentations, and more, and can be created by faculty, instructional designers, or librarians. online tutorials are time consuming to create, but they are powerful tools for asynchronous educational opportunities for nursing students studying online. nurses and librarians have demonstrated in a variety of studies that using online tutorials in a flipped classroom approach or through an online guide is valuable (gilboy et al., 2015; schlairet et al., 2014; schroeder, 2010). lastly, creating online synchronous courses or professional development opportunities can better reach many nursing students (smith & o’hagan, 2014).   librarians have consistently provided information literacy and reference consultations, instruction, and assessment to nursing students, ideally while also integrating the important competency of evidence based practice (ebp) (adams, 2014). librarians are adapting to the demographics of nursing programs and adjusting services for an increasingly online population. when nursing researchers are surveyed or interviewed about their information literacy and evidence based practice essentials, there is a call for more digital support and expanding research services to include grant-writing, scholarly communication, and data management (nierenberg, 2017; wahoush & banfield, 2014). though more and more online library services and tutorials are being offered to nursing students, some of these students are not digital natives and may not possess all of the computer literacy skills needed for researching online (turnbull, royal, & purnell, 2011; brettle & raynor, 2013). virtual chat, library services, and asynchronous information literacy instruction are widely discussed in the literature, but no previous studies combine the research needs of nursing students with an in-depth chat analysis of this population. this study seeks to analyze nursing reference chats to help academic librarians better serve all patrons.   methods   the libraryh3lp system is used to provide virtual chat reference to uncg library patrons. this chat service is available to non-uncg patrons as well, but this study only analyzes transcripts of internal patrons. this chat service is staffed mostly by personnel in the roi department, along with some librarians and staff from the music library and the special collections and university archives (scua) department. these chats are also answered by uncg library and information studies (lis) graduate student interns working in the roi department. during a typical semester, chat service is online and active from 8:00am until 11:45pm monday through thursday, and during the day and into the early evening on friday through sunday. libraryh3lp is heavily used, receiving around 3500 chats a year.   in fall 2018, the two health science liaison librarians downloaded all full chat transcripts from the months august, september, and november 2016, with a total of 1416 chat transcripts. this time frame was selected to avoid singling out any current librarians, staff, or students taking chats because these months provided a sampling of the busiest fall semester months. to find these transcripts, a csv file was created from the backend of libraryh3lp of all chat transactions from fall 2016, and then edited based on the selected months. the csv file was converted into a xls file to perform a “control find” of nursing keywords to identify relevant chats. these keywords were developed by the health science librarian, who is the nursing department liaison, based on instruction sessions and common research services, questions, or issues that arise within the nursing department. the keywords used to identify the chats were: nur*, cumulative index to nursing and allied health literature (cinahl); pubmed; lea (first name of the health science librarian); evidence based practice (ebp); systematic reviews; integrative; health; hospital; patient; clinical; anesthesia; doctor of nursing practice (dnp); bachelor of science in nursing (bsn); master’s of science in nursing (msn); practitioner; geriatric; and patient, population, problem, intervention, comparison, and outcome (pico).   based on this keyword and search method, 60 chat transcripts were identified and pulled from the original 1416 transcripts into box (uncg’s most secure cloud storage system) using the note platform. putting the transcripts within box note allowed each transcript to also have relevant metadata attached, such as date, length of time the patron waited before staff were able to engage with the chat, source of the chat, length of the chat, category of chat (reference, service, technology, library directions, or services), description of chat, and read (reference effort assessment data) scale rating. read scale is a measurement of the difficulty level of the reference transaction of the chat (karr gerlich, n.d.). another reason the transcripts were placed in box notes was to redact any identifying information related to students or instructors, such as names, email addresses, or phone numbers. from the chat data, a master spreadsheet was created in google sheets, allowing librarians to see the overall trends of length of chat, time of chat, read scale, and more.   once the transcripts were anonymized within box notes, one pdf of the selected transcripts was created and read by both librarians. these librarians then created groups of themes and corresponding codes to determine an overall and consistent list to be applied to the transcripts. both librarians coded the chat transcripts based on the following groups: information need, reference interviewing, recommendations, patron emotions, and challenges and barriers. see table 1 for a full list of codes used for this analysis, organized by groups.   both librarians attended assessment workshops on coding qualitative research in atlas ti by uncg oaers (office of assessment and evaluation research services). the research design for this study was informed by creswell and poth’s (2017) guide, which suggests five traditional qualitative research approaches: narrative research, phenomenology, grounded theory, ethnography, and case study. this study used the narrative research approach, considering the virtual chats the narrative to be analyzed. when the codes were finalized, they were input and applied to transcripts using atlas ti. the librarians initially applied the codes on separate sets of transcripts, and then switched transcript sets to ensure that each chat was examined by both parties. to minimize bias and errors when the transcripts were switched, each party checked the other’s codes for consistency and gaps. from atlas ti, the code group could be used to analyze chats based on the individual codes. all forms of qualitative research are subjective and results can shift depending on the individual reading and coding of each transcript. though there were two coders on each transcript, themes could be missed based on the length of chat or state of the reader.   results   information need   the first code group to be analyzed was “information need”, in order to establish trends of research of nursing library patrons. when looking at the coding group of “information need”, the most common themes were students and instructors looking for the full-text of a known item, such as an article or e-book, and searching strategies for research assignments (figure 1). patrons stating their research question, keywords, and assignments were also heavily coded. the code of “space” got no mentions in this set of nursing chats.   table 1 code groups and codes for fall 2016 uncg libraries virtual chat nursing transcripts analysis, inputted into atlas ti code group: codes within group: information need keywords, searching strategies, hours, space, assignment, research question stated, course mentioned, student type dnp, student type bsn, student type masters, student type rn/bsn, student type doctoral, student type dnp anesthesia, resource type peer review, resource type integrative review, resource type systematic review, resource type research article, resource type evidence based, resource type theoretical article, resource type law or court cases, citation, known item journal, known item article, known item database, known item e-book, known item other, currency, source evaluation, non-library resource writing center, non-library resource tutoring, non-library resource technology assistance, non-library resource community partners, other information need reference interviewing request for clarification of research need, ask for course number, navigation of resource, ask if more assistance needed, confirmation need was met, transferred chat, other reference interviewing recommendations database cinahl, database pubmed, database dynamed plus, database healthy people 2020, database cochrane, database scopus, database psycinfo, database academic search complete, database proquest, database other, journal a-z list, assistance library catalogue, referral liaison, referral instructor, filters in catalogue or database, interlibrary loan (ill), citation management zotero, citation management endnote, library tutorials path, library tutorials other, course guide, subject guide, physically come into the library, other recommendation patron emotions frustration, gratitude, stress and anxiety, uncertainty, other patron emotions challenges and barriers full-text, service not working, resource not working, access off campus, access through browser, access database, access catalogue, access e-book, access textbook, permalinks, interdisciplinary research need, misunderstandings assignment, misunderstandings other, business librarian, business of patron, patron disappears, technical issue, too many results, too few results, usability website, usability chat, other challenges and barriers   figure 1 chart depicting the amount of times “information need” was coded, meaning the need of the patron chatting for research help.   patron emotion   findings within the “patron emotion” code group show that people answering chats are doing an effective job of providing permalinks, offering descriptions of navigating to resources, creating keywords, boosting students’ academic confidence, and helping them learn more about the research process. the “gratitude” code was often found within the transcripts. many chats ended with nursing patrons saying “thank you so much! this was so helpful!”, particularly when learning about how to narrow down search results, how to use allied health and nursing library resources, and how to use library databases more efficiently. in many chats nursing students could immediately use the research skills showcased in the chat in their research. for example, one chat patron stated: “i'll try to limit [my search results] down with keywords, but that database has better results!”   the most coded emotion from patrons was “gratitude”, but the second most coded was “uncertainty”, followed by “frustration”. sometimes patrons were unsure of their needs since they were new to the research process. for example, it was not uncommon for nursing patrons to write in messages like, “hey. never done this before but i'm having some difficulty finding articles on my topic and i know there are articles out there. i am just not finding them. can you help?” many times, the chat nursing patrons note being busy working, as well as being a student, so not having time to properly research their assignment. in some cases, the patrons were at work while chatting with librarians, such as in this scenario where the patron writes “currently at work and tried using the library online already and having trouble which is why i want to physically go in.”   figure 2 chart of uncg libraries nursing chats showing the “reference interviewing codes”, showing the interactions of the librarian answering the chat questions.   reference interviewing   reference interactions of the people answering the chats were also coded (figure 2). librarians, students, and staff answering reference chats consistently provided navigation to resources and requested clarification of research needs. there were fewer instances of those staffing chat asking students for the course number for their specific nursing course (only 35% of chats requested a course number). a small number of nursing chats were also transferred to other people staffing chat based on their expertise or availability.   challenges and barriers   the most common challenges and barriers of the chats were also coded. “full-text” was the most common challenge touched upon, but “busyness of librarians” and “busyness of patron” also received many mentions in these chats. for example, librarians would pause and write, “sorry for the delay, i had a patron at the desk while you were chatting.” patrons sometimes mentioned challenges with “access off campus” and “technical issues” with library resources. more patrons had issues with “too many results” when searching for resources than “not enough results.” the overall code group of “challenges and barriers” was the least coded theme.   recommendations   the specific recommendations from people staffing chat were also coded. databases were the most commonly advocated resource for patrons to use to search for research materials (in almost 77% of the chats analyzed), with cinahl recommended the most often. the next most frequently mentioned research technique was for patrons to use filters to narrow down searching in databases or the library catalogue. staff and librarians also encouraged the use of course and subject guides on nursing topics, and in a little over 18% of the chats, a one-on-on meeting with the liaison librarian was endorsed. searching in the library journal finder or a specific nursing journal was never mentioned or promoted in these chats.   figure 3 uncg libraries nursing chat chart depicting the source of the chat.   general trends   the coding groups provided significant insight into nursing research needs. additionally, the overall trends of the nursing chats are useful for improving chat services. the master spreadsheet of nursing chats includes chat source, time of chat, length of chat, whether the chat was transferred, type of patron and course (if mentioned), read scale, and description of the chat. the chats were received evenly throughout the day, with the afternoon being the slightly more popular time for patrons to chat (43%). most chats came in from the libraries ask us homepage, where the chat box is embedded within the libraries home website (figure 3). the second most popular source of chats was oclc worldcat (the library catalogue) and the third was the nursing subject guide. chats that included mentions of a specific course were tagged; this illustrated that 57% of students did not mention a course when chatting in with a research need. people staffing chat can choose to rate the difficulty level of each chat using the read scale. with the chats we analyzed, many did not have read scale ratings; of those that did get a rating, the most common rating was 3. at uncg libraries, read scale 3 means reference interactions such as finding books or dvds in the catalogue by title or author (i.e., “i need to find toni morrison’s beloved”); accessing research guides; accessing subject databases; and basic citation style questions that can be answered using online citation style guides.   discussion   evaluating nursing chat transcripts   reference themes proved valuable in the coding analysis. “busyness of librarians” and “busyness of nursing students” were commonly coded; this shows the importance of offering a variety of virtual reference services, such as online learning objects, chat, and virtual consultations. within the “recommendation” coding groups, not many library staff asked for a course number from chat patrons. while this could mean that a reference need was as simple as asking for a pdf of an article or that the chat staff was pressed for time, clarifying if there is a course number or assignment involved enables staff to better market course-specific research guides with relevant tutorials, links to databases, and applicable contact information. librarians staffing chat should consistently offer detailed resource navigation instructions, as well as follow up information directing patrons to the nursing research guide and librarian. this is a particularly useful approach at a university where health science is in the general education curriculum and is part of the strategic plan, enabling library chat staff to quickly and confidently fulfil the research need and move on to handle the next reference interaction. since librarians working chat are often multi-tasking by handling more than one chat or by talking to patrons at the physical reference desk, it is vital for a library department to also create training to boost confidence with answering nursing research questions.   a frequently coded topic was the need for help with identifying keywords for searching in databases, which confirms the high prevalence of patrons searching for “known items.” highlighting the value of using library nursing and allied health databases and the catalogue is always integral during library information literacy instruction; this data shows the need to better showcase the differences between library resources and tools like google scholar (paywalls, lack of evaluation of quality of journals, insufficient search filters) within nursing and library online learning objects, instruction, consultations, webcasts, and orientations.   when helping patrons look for research articles, the health database cinahl was usually recommended. though cinahl is a great solution for finding health science and nursing articles, there are many other databases that can direct patrons to research resources, including pubmed which has different search functionality. throughout the transcripts, patrons consistently mentioned using pubmed. for example, a patron wrote “i'm looking for a full text scientific article through the journal: current medicinal chemistry. what's the best way to get a full text? i have links to the pubmed and ncbi page but can’t find the pdf of the article.” librarians staffing chat need to be able to quickly navigate between different types of databases to better serve nursing patrons.   training on nursing research   this chat coding project shows that nursing patrons need assistance finding full-text clinical studies and articles, while also understanding the different nursing and allied health library databases. when informally surveying library staff about nursing chat training needs, one library staff chat member stated the desire for training on “more information about pubmed. i often recommend cinahl because it's what i know the most. i've used pubmed in the past and know the overall gist, but some more details about advanced searching, how the database works, would be great.” based on this chat analysis and the needs of librarians working chat, follow up training sessions were created and administered by the health science librarians.   the trainings created were presented to the uncg library and information science (lis) interns, with other chat staff and liaison librarians invited to participate. these training sessions about health science research and resources have now been offered every academic year since 2018, for a total of three workshops. topics covered include recognizing research articles in the context of health sciences (primary research and types of research studies), evidence based practice, pico, health science databases pubmed and cinahl, and chat practice. the workshop generally ends with chat exercises from transcripts pulled from this study, and the attendees answer them on google forms or google docs within a think, pair, share or group discussion format. when these workshops were assessed, lis interns reported that these nursing chat and research workshops were helpful in reducing their anxiety about answering health science reference questions.   reviewing nursing online learning objects   this study created an opportunity for immediate action for chat staff training, which was planned and performed. it also prompted review of the nursing research guides, so that busy nursing patrons and chat staff can more quickly find relevant information and resources. since this study, the libraries roi department has also revamped their suite of information literacy research tutorials.   this in-depth study of the needs of nursing patrons therefore helps to inform the need for more health science-related online learning objects and tutorials, as well as more online support on general topics such as finding full-text, locating permalinks, navigating the library website, and using the library catalogue and databases. since this study, many nursing research online learning objects have been added to help train librarians, faculty, and students on concepts such as advanced searching in cinahl, pubmed basics, the new pubmed interface, evaluating health sources online, and predatory journals.   future directions   qualitative research studies involve limitations, including differing interpretations and errors that can happen with a large amount of data which takes a long time to code and analyze. the methodology of this study was challenging because of its time-consuming nature, but it was useful as an in-depth examination of the reference needs of a specific patron population. a similar study could be designed looking at different academic subjects or themes that feature in the libraries’ general chat interactions, such as e-books, interlibrary loans, and streaming. a variation on this study would be to apply the same methodology to recordings of nursing student consultations. another continuation of this study would be to survey the patron population alongside a chat analysis. since this study was performed, the online learning librarian at uncg has interviewed students studying online about their overall information retrieval and research needs. a similar approach could be taken with nursing students based on this study, to further identify their specific and diverse research needs. another future study could encompass pretesting and posttesting to gauge whether nursing students’ research knowledge improves via long reference chat sessions.   starting in march 2020, due to the covid-19 pandemic many of the library personnel who staff the chat service are working from home. additionally, lis interns no longer staffed the chat in summer 2020. based on this shift in the academic workflow, there have been several virtual professional development workshops. the nursing librarian performed multiple sessions on finding trustworthy health information online during covid-19. this virtual workshop model could be adapted based on assessment from current programming to help lis interns and librarians staffing chat.   conclusion   though time-intensive, this study on library chat transcripts shows the diversity of needs of nursing patrons, which included a large population of non-traditional and distance students. an in-depth examination of nursing chats led to a series of workshops and trainings for library chat staff and lis students on nursing research, while also helping library personnel develop more tutorials and online learning objects on health information literacy. improving the vital online service of chats through training on nursing research, evidence based practice, pico, and specific health databases creates a better research environment for all patrons and librarians. this study will continue to help the libraries improve and expand workshops into the virtual realm, supporting library employees and patrons during the covid-19 pandemic. virtual reference chat is not going away; in the current academic environment it is needed more than ever. studying library chats beyond basic use statistics may reduce library chat staff anxiety and prepare them to better serve patrons.   references   adams, n. e. (2014). a comparison of evidence-based practice and the acrl information literacy standards: implications for information literacy practice. college & research libraries, 75(2), 232–248. https://doi.org/10.5860/crl12-417   al‐gamal, e., alhosain, a., & alsunaye, k. (2018). stress and coping strategies among saudi nursing students during clinical education. perspectives in psychiatric care, 54(2), 198–205. https://doi.org/10.1111/ppc.12223   association of college & research libraries. (2016). standards for distance learning library services. in ala american library association. retrieved 20 september 2020 from http://www.ala.org/acrl/standards/guidelinesdistancelearning   barrett, k., & greenberg, a. (2018). student-staffed virtual reference services: how to meet the training challenge. journal of library & information services in distance learning, 12(3–4), 101–119. https://doi.org/10.1080/1533290x.2018.1498620   baumgart, s., carrillo, e., & schmidli, l. (2016). iterative chat transcript analysis: making meaning from existing data. evidence based library and information practice, 11(2), 39–55. https://doi.org/10.18438/b8x63b   bliquez, r., & deeken, l. (2016). hook, line and canvas: launching a professional development program to help librarians navigate the still and stormy waters of online teaching and learning. journal of library & information services in distance learning, 10(3–4), 101–117. https://doi.org/10.1080/1533290x.2016.1206778   brettle, a., & raynor, m. (2013). developing information literacy skills in pre-registration nurses: an experimental study of teaching methods. nurse education today, 33(2), 103–109. https://doi.org/10.1016/j.nedt.2011.12.003   brown, r. (2017). lifting the veil: analyzing collaborative virtual reference transcripts to demonstrate value and make recommendations for practice. reference and user services quarterly, 57(1), 42–47. https://doi.org/10.5860/rusq.57.1.6441   choy, s. (2002). nontraditional undergraduates, national center for education statistics. in findings from the condition of education. retrieved 20 september 2020 from https://nces.ed.gov/pubs2002/2002012.pdf   craig, a., & corrall, s. (2007). making a difference? measuring the impact of an information literacy programme for pre-registration nursing students in the uk. health information & libraries journal, 24(2), 118–127. https://doi.org/10.1111/j.1471-1842.2007.00688.x   creswell, j. w., & poth, c. n. (2017). qualitative inquiry and research design: choosing among five approaches (4th edition). thousand oaks, ca: sage publications, inc.   dempsey, p. r. (2019). chat reference referral strategies: making a connection, or dropping the ball? college & research libraries, 80(5), 674–689. https://doi.org/10.5860/crl.80.5.674   duncan, v., & holtslander, l. (2012). utilizing grounded theory to explore the information-seeking behavior of senior nursing students. journal of the medical library association, 100(1), 20–27. https://doi.org/10.3163/1536-5050.100.1.005   eastman, t., hyde, m., strand, k., & wishkoski, r. (2019). chatting without borders: assessment as the first step in cultivating an accessible chat reference service. journal of library & information services in distance learning, 13(3), 262–282. https://doi.org/10.1080/1533290x.2019.1577784   gilboy, m. b., heinerichs, s., & pazzaglia, g. (2015). enhancing student engagement using the flipped classroom. journal of nutrition education and behavior, 47(1), 109–114. https://doi.org/10.1016/j.jneb.2014.08.008   guillot, l., & stahr, b. (2004). a tale of two campuses: providing virtual reference to distance nursing students. journal of library administration, 41(1–2), 139–152. https://doi.org/10.1300/j111v41n01_11   johnson, c. v., & johnson, s. y. (2017). an analysis of physician assistant libguides: a tool for collection development. medical reference services quarterly, 36(4), 323–333. https://doi.org/10.1080/02763869.2017.1369241   karr gerlich, b. (n.d.). welcome to the read scale research web site. retrieved from http://www.readscale.org/   ledwell, e. a., andrusyszyn, m. a., & iwasiw, c. l. (2006). nursing students’ empowerment in distance education: testing kanter’s theory. journal of distance education, 21(2), 78–95. retrieved from https://eric.ed.gov/?id=ej807804   logan, j., barrett, k., & pagotto, s. (2019). dissatisfaction in chat reference users: a transcript analysis study. college & research libraries, 80(7), 925–944. https://doi.org/10.5860/crl.80.7.925   maloney, k., & kemp, j. h. (2015). changes in reference question complexity following the implementation of a proactive chat system: implications for practice. college & research libraries, 76(7), 959–974. https://doi.org/10.5860/crl.76.7.959   mungin, m. (2017). stats don’t tell the whole story: using qualitative data analysis of chat reference transcripts to assess and improve services. journal of library & information services in distance learning, 11(1–2), 25–36. https://doi.org/10.1080/1533290x.2016.1223965   reeves, j. s., & fogg, c. (2006). perceptions of graduating nursing students regarding life experiences that promote culturally competent care. journal of transcultural nursing, 17(2), 171–178. https://doi.org/10.1177/1043659605285410   schlairet, m. c., green, r., & benton, m. j. (2014). the flipped classroom: strategies for an undergraduate nursing course. nurse educator, 39(6), 321–325. https://doi.org/10.1097/nne.0000000000000096   schroeder, h. (2010). creating library tutorials for nursing students. medical reference services quarterly, 29(2), 109–120. https://doi.org/10.1080/02763861003723135   smith, s. c., & o’hagan, e. c. (2014). taking library instruction into the online environment: one health sciences library’s experience. journal of the medical library association, 102(3), 196–200. https://doi.org/10.3163/1536-5050.102.3.010   stankus, t., & parker, m. a. (2012). the anatomy of nursing libguides. science & technology libraries, 31(2), 242–255. https://doi.org/10.1080/0194262x.2012.678222   stein, j. v., & reeder, f. (2009). laughing at myself: beginning nursing students’ insight for a professional career. nursing forum, 44(4), 265–276. https://doi.org/10.1111/j.1744-6198.2009.00152.x   university of north carolina greensboro (2013). the university of north carolina at greensboro strategic plan core elements: taking giant steps. in unc greensboro. retrieved 20 september 2020 from https://strategicplan.uncg.edu/core-elements/   university of north carolina greensboro (2020). graduate degrees. in uncg online. retrieved 20 september 2020 from https://online.uncg.edu/graduate-programs/   wahoush, o., & banfield, l. (2014). information literacy during entry to practice: informationseeking behaviors in student nurses and recent nurse graduates. nurse education today, 34(2), 208–213. https://doi.org/10.1016/j.nedt.2013.04.009 research article   user-focused, user-led: space assessment to transform a small academic library   christina hillman assessment & online program librarian lavery library st. john fisher college rochester, new york, united states of america email: chillman@sjfc.edu    kourtney blackburn access services librarian lavery library st. john fisher college rochester, new york, united states of america email: kblackburn@sjfc.edu   kaitlyn shamp student researcher st. john fisher college rochester, new york, united states of america email: kks04047@sjfc.edu   chenisvel nunez student researcher st. john fisher college rochester, new york, united states of america email: cn01722@sjfc.edu     received: 13 july 2017   accepted: 23 oct. 2017        2017 hillman, blackburn, shamp, and nunez. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – by collecting and analyzing evidence from three data points, researchers sought to understand how library spaces are used. researchers have used results for evidence based decision making regarding physical library spaces.  methods – undergraduate researchers, sociology faculty, and librarians used mixed-methods to triangulate findings. seating sweeps were used to map patrons’ activities in the library. student-led focus groups discussed patterns of library use, impressions of facilities, and library features and services. the final step included a campus survey developed from seating sweeps and focus group findings.   results – seating sweeps showed consistent use of the library's main level learning commons and upper level quiet spaces; the library’s multipurpose lower level is under-utilized. students use the main level of the library for collaborative learning, socializing, reading, and computer use. students use the upper level for quiet study and group work in study rooms. focus group findings found library use is task-specific. for example, a student may work with classmates on a project using the main level learning commons during the day, and then come back at night to use the quiet floor for test preparation. survey responses highlighted areas in which the library is deficient. for example, respondents cited crowdedness, noise levels, and temperature concerns.   conclusion – these data offer empirical evidence for library space needs. some data aligns with previous space studies conducted at this library: access to power outlets, lighting, noise, and an outdated environment. evidence also supports anecdotal concerns of crowding, graduate students lacking designated study space, and the need for quiet study space away from group study space.     introduction   established in 1975 as the sole library for the st. john fisher college, lavery library serves a campus of approximately 3800 students, including undergraduate, masters, and doctoral. the college is primarily an undergraduate institution with a growing graduate population. at the same time, the library has witnessed a slow but dramatic shift in the way users work in physical library spaces. the library uses daily headcounts and gate counts to improve library spaces. the library also conducted several space studies over the past decade to inform small-scale physical changes and better accommodate changing user needs. renovations since 2012 include a learning commons, the creation of a multi-purpose space (keating room), a space with cafe-like seating, and additional outlets. through strategic weeding, the library has enlarged study spaces. recent changes include the addition of easily movable tables and soundproofing quiet floor doors. these changes are welcomed by the campus community, but formal and informal feedback from the students provides a clear and consistent message: the library must continue to keep pace with their changing space needs in order to maintain a high standard of service.   the library is three levels, with users entering on the second (main) level. this level houses the keating room and learning commons, which includes group workstations with large monitors, desktop computers, and a variety of tables and chairs for groups and individuals. the lower level includes group work tables and two classrooms, one of which is a computer lab. the upper level is the quiet floor, the only floor with a noise policy. there is a variety of seating, including individual study carrels, small and large tables, individual and group study rooms, and two reservable meeting spaces. the library is also home to other campus departments (e.g., career center, academic opportunities program office, office of information technology, and others), which were not a focus of this study.   literature review   library as place   with a shift from print to electronic collections, libraries have reinvented themselves as flexible learning spaces with a focus on community. the phrase library as place best describes how students use the library as a flexible, dynamic space adaptable for changing needs (freeman, 2005). other studies discuss how students continually remake spaces to fit their needs to support their learning (fallin, 2016; hanson & abresch, 2016). montgomery (2014) refers to the library as a place for informal learning, where students can set their own goals and determine their needs. the library is thought of for its study spaces and less for services and collections (declercq & cranz, 2014; hall and kapa, 2015). a place to gather and have conversations, according to oldenburg (1997), is an important part of learning; the library has begun to be this place. as a result of this flexibility and community building, academic library users, particularly students, see the library as a “third space” (declercq & cranz, 2014)—a place neither classroom nor residence hall. academic work and socializing takes place within third spaces, and “library as place” fills the need for this third space.    space attributes   whether it is quiet study space or an open meeting space, the reasons how and why users select library spaces largely depend on individual needs and activities (cha & kim, 2015; i̇mamoğlu & gürel, 2016; khoo, rozaklis, hall, kusunoki, & rehrig, 2014; montgomery, 2014; vaska, chan, & powelson, 2009). research focusing on students’ requirements of library spaces reveal common themes: more natural light, larger or more tables and chairs, and more outlets (andrews, wright & raskin, 2015; declercq & cranz, 2014; i̇mamoğlu & gürel, 2016; khoo et al., 2014; montgomery, 2014; vaska et al., 2009). library spaces must also accommodate simultaneous device use by students (ellison, 2016; ojennus & watts, 2017). similarly, research indicates the need for collaborative spaces that can accommodate a variety of technologies (andrews et al., 2015; given & archibald, 2015; freeman, 2005; lux, snyder, & boff, 2016). at the same time, goodnight and jeitner (2016) focus on the desire for quiet, because students “come to the library searching for spaces that are quiet, where they can settle down to read and study and write their papers in silence, without distractions . . .” (p. 219) from others. similar research also notes individual study carrels and quiet spaces are valued (hall & kapa, 2015; montgomery & miller, 2011; ojennus & watts, 2017; oliveira, 2016).   group study and non-quiet spaces   non-quiet space in the library—for example, group study rooms and flexible learning spaces—are ideal for many library users, as indicated by freeman (2005). recent literature shows the need for more of these spaces, and that students respond positively to redesigns which provide more flexible learning and group study spaces (cha & kim, 2015; given & archibald, 2015; khoo et al., 2014, montgomery, 2014). studying alongside others provides visual and social pressure for students, furthering the communal space (andrews et al., 2015). there is a need for libraries to create spaces where users can collaborate, socialize, and study alone and alongside others (andrews et al., 2015; declercq & cranz, 2014; freeman, 2005; montgomery, 2014; montgomery & miller, 2011, ojennus and watts, 2017). quiet and individual study   research indicates students use quiet areas to accomplish serious work (e.g., to study for exams or write papers) (cha & kim, 2015; declercq & cranz, 2014; freeman, 2005; khoo et al., 2014). even during individual study, students often indicate their desire to be near others studying (andrews et al., 2015; applegate, 2009; goodnight & jeitner, 2016; hall & kapa, 2014; i̇mamoğlu & gürel, 2016; montgomery, 2014). yet, students still desire ample personal space, feeling a space is full when 40-50% of seats are occupied (applegate, 2009; i̇mamoğlu & gürel, 2016; khoo et al., 2014). physical dividers would allow users to delineate personal space and minimize distractions so that they can work most effectively (i̇mamoğlu & gürel, 2016).   aims   the purpose of this study is to examine and analyze how students use library spaces. collected evidence will be used to plan space renovations, both small and large. additionally, collected evidence will improve understanding of what works, what does not work, and what is needed in the library.   methods   this study used multiple methods to triangulate findings and provide a clearer understanding of how library spaces are used. methods included seating sweeps, focus groups, and survey. research was conducted with institutional review board (irb) approval.                 seating sweeps                 seating sweeps were based on given and leckie’s 2003 study, “’sweeping’ the library: mapping the social activity of the public library.” librarian researchers trained three permanent library staff members to assist with completing sweeps. data was collected floor-by-floor with printed maps and a clipboard (see appendix a). they were conducted three times a day for two non-consecutive weeks during spring 2016. the first sweep took place in february, just before spring break; the second was in april, a few weeks before finals. sweeps were conducted at 9 a.m., 1 p.m., and 8 p.m. to create a snapshot of user behaviours throughout the day, and took between 15 and 60 minutes depending on busyness. staff recorders noted user activities and personal items, such as use of a desktop, laptop, cell phone, tablet, or whiteboard; and if they had food or drink. recorders also marked if users were conducting group work, note-taking, reading, sleeping, talking, or performing other noteworthy activities. for instance, recorders captured when individual users occupied entire tables intended for multiple people, or when users dragged cords across aisle ways to reach outlets. interested in users’ willingness to move larger furniture, librarian researchers purposely left furniture placement off the map in the multi-purpose keating room so recorders would be able to draw changes to configurations of the space. to minimize intrusiveness, recorders maintained a reasonable distance from users. the clipboard also included a sign stating that a library space study was in progress in order to inform users but hopefully not discourage or change user behaviours. data from the coded maps were entered into a google form for analysis.                 focus groups                 after seating sweeps were completed, student researchers and sociology faculty advisers joined the research team. faculty advisers trained student researchers to conduct focus groups.  focus groups were organized by class year (9 freshmen, 9 sophomores, 10 juniors, 8 seniors, 2 masters, and 3 doctoral students) totaling 41 participants. student researchers recruited undergraduate participants by invitation; liaison librarians recruited masters and doctoral participants by emailing targeted classes. participants were offered pizza and the chance to win a prize as an incentive. the research team developed questions based on past local space surveys and sweeps data. librarian researchers and faculty advisers were not present at the focus groups in an effort to minimize their influence on participants’ responses. each undergraduate group was asked the same set of questions; these questions were altered slightly for masters and doctoral students. student researchers took notes of participants’ responses, and after the focus groups were completed, the research team came together to analyze findings. focus group data were reviewed for common themes by each researcher independently, and schemas were developed as a team to help inform survey development.                 survey                 the research team developed questions based on findings from seating sweeps and common themes from library focus group data. qualtrics was used to build and distribute the completed survey (see appendix b). as with many institutions, students have survey fatigue on our campus. in order to keep the survey short and increase response rate, the research team opted not to include demographic information in the survey. prior to distribution, faculty advisers and student researchers piloted the survey with a small group of undergraduates. researchers decided to exclude masters and doctoral students due to their low participation in focus groups and a lack of relevant data.   all undergraduates (n=2948) received the survey via email. to improve response rate, the survey was emailed to students through the well-recognized and respected student government association (sga). respondents completed the survey anonymously, with the caveat that if they wished to enter a drawing for a $100 amazon gift card, they needed to provide their name and email address. a separate survey allowed respondents to enter the drawing, which allowed the research team to maintain confidentiality of responses. the survey ran for three weeks with two reminder emails, sent through the qualtrics platform, to those who had yet to complete the survey. the overall response rate was 12%.   results   seating sweeps   findings from seating sweeps helped visualize occupancy patterns and user behaviours. existing library data shows the busiest time is the 1 p.m. hour monday-friday, which is consistent with seating sweep findings. data from sweeps revealed the main level to be the busiest, followed by the upper level (see table 1). tables meant for 4 people were observed with only 1 person spread over the entire surface 12% of the time, effectively making the space fully occupied. this data is consistent with survey findings regarding crowdedness. at the same time, the lower level occupancy rate was less than 1% during sweeps, despite being a non-quiet space.   behaviours recorded during sweeps indicated the library is a multipurpose, adaptable space, similar to other research. a key finding from the sweeps showed 10% of users were settling in or making themselves at home in their claimed spaces: using bean bag chairs to get comfortable, adjusting lighting, taking off their shoes, sleeping, and abandoning belongings for extended time. findings from sweeps also observed 40% of users eating or drinking, another indicator of the library being a flexible third place. data also showed users crowding around a single computer monitor for collaborative work rather than making use of collaborative group workstations and their larger monitors, with the latter noted only three times. students made frequent use of flexible furniture in the library, especially in the keating room. findings from sweeps showed students use the movable whiteboards for their intended use (studying), but interestingly, also as barriers to create privacy. observed behaviours related to technology confirmed informal feedback regarding the need for more outlets and power.     table 1 combined average occupancy of patrons by floor during seating sweeps   9a.m. 1p.m. 8p.m. lower level 2.5 9 11.6 main level 24 91.1 67.6 upper level 12.3 42.6 33.1     during sweeps, 41.5% of users were recorded simultaneously using at least two electronic devices, creating a higher demand for power and technology options in the library.   focus groups   findings from focus groups provided better understanding of what users think about library spaces, including their intended use and desire for these spaces. common uses for the library included studying, computer use, printing, and working on group projects. these results were common among all focus groups. common responses when asked about well-liked library services and features included: interlibrary loan, librarians and the research help desk, and group workstations for easier collaboration. when asked about services or features they would like to see added, common responses included a stress relief room with nap pods, extended hours, and additional quiet floor study rooms. participants requested smaller, 1-2 person tables for independent work, stating once they set up at larger tables other students appear dissuaded from joining the table. participants suggested extended hours, with a few participants stating the library should stay open 24 hours or at least until 3 a.m.   findings revealed differences in how undergraduate commuters and residents use the library. commuters indicated coming to the library most often between classes to connect   with friends, not to engage in serious work. as with many participants, commuters mentioned choosing somewhere on the quiet level when coming to the library for serious work. residents use dorm lounges or their rooms for work and use the library for printing or socializing. for group work and projects, both commuters and residents commonly use library spaces, but stated the lack of privacy on the main level and the noise policy on the upper level can be frustrating. undergraduate students mentioned the breakout rooms available in other buildings are ideal spaces for this type of work.   focus group questions for masters and doctoral students differed slightly than those asked of undergraduates. these participants’ responses revealed differences in library use, including primarily using the library for research purposes. most stated using librarians as helpful resources when conducting research, and were more emphatic in their responses regarding use of the research help desk. two participants completed undergraduate degrees at st. john fisher college, and indicated their library use as graduate students is much more academically oriented.   survey   the survey provided data for how undergraduates self-reported using library spaces in relation to focus groups and sweeps data. respondents reported using main level – open area and the keating room (tutoring) spaces 45.57% and 8.89%, respectively, “very often”.  respondents self-identified using quiet floor open areas and study rooms “very often” 31.65% and 36.39% of the time, respectively. the main level is the most self-identified used space, with the upper level spaces closely following. survey results find the library’s lower level (basement) is underutilized, with basement – computer lab and basement – tables “never” being used 49.05% and 50.95% of the time, respectively. figure 1 provides a breakdown of library spaces and their frequency of use by respondents. monday-thursday and finals week are the most popular times in the library: 45% of users stated they come to the library “very often” monday-thursday, and 57% of users indicated that they come to the library “very often” during finals week. nearly 50% of respondents think the library needs extended hours, which is similar to findings from focus groups; however, just under 40% of individuals indicated coming to the library “very often” in the evening.     figure 1 responses to survey question: when i go to the library, i go to….     in addition to revealing what spaces respondents reported using most frequently, they also shared which spaces are deficient (see figure 2). as previously noted, the quiet floor and its private rooms are extremely popular, and unsurprisingly, 69% of respondents requested additional private rooms. also unsurprisingly, respondents said the library needs more outlets (60%) and tables (41%) throughout the library. the need for more outlets and tables has a strong relationship to findings of computer use and group work, with 52% of respondents using computers and 64% of respondents “sometimes” conducting group work in the library. overall, respondents are mainly interested in conducting academic-related activities in the library. even so, a high percentage of respondents requested the addition of stress-relief features such as nap pods and massage chairs, as well as grab ‘n go foods.   survey results regarding noise levels and temperature shed light on students’ individual perceptions of spaces. when asked if the library is “too noisy,” 62% of respondents indicated the library was “sometimes” too noisy, which parallels findings of crowdedness, as 64% of respondents indicated the library was “sometimes” too crowded. despite the majority of respondents indicating that the library is “sometimes” too noisy and “sometimes” too crowded, noise and crowdedness may not always be related. this lack of correlation may be due to the time of day a student uses the library. for example, the 1 p.m. hour is extremely crowded and noisy, whereas the 8 p.m. hour might be crowded but relatively quiet. regarding temperature, when responding to the statement “i think the library needs . . .” with a list of options users could check (see figure 2), 58% selected “fans and air conditioning.” there was some relationship between this finding and the library being too hot: 34% felt the library was “very often” too hot and 44% felt the library is “sometimes” too hot, while 64% felt the library is “never” too cold.     figure 2 response to survey question: “i think the library needs . . .”     discussion library as place   common themes from space-related literature are echoed in this study’s findings. as with freeman (2005), lavery library created flexible spaces, providing moveable, lightweight furniture for users to create their ideal study environments. during sweeps users were consistently observed moving tables, chairs, and whiteboards to create such environments, leading researchers to infer users are comfortable enough in the library to make spaces fit their needs (montgomery, 2014). further, observed users exemplified “library as place” by lounging in beanbag chairs, adjusting lighting in study rooms, taking off shoes, sleeping, and using headphones. whether headphones were used as noise dampening or for watching videos was not captured, and focus group participants only mentioned their appreciation of headphones available for checkout and earbuds for purchase at the checkout desk. additionally, observations suggested a high level of comfort in the library and with each other; users frequently abandoned belongings. this may also be a means to save their spaces when the library is crowded.   students make use of flexible learning spaces, moving tables and chairs as needed to accommodate their needs. a good example of this is students consistently moving tables and chairs in the keating room. the maps used for the sweeps purposefully left furniture placement off the map so recorders would be able to draw daily configurations of the space. while the space never changed dramatically, there were small changes, including the rolling white boards. the idea of collaborative, flexible study spaces, where students are able to work together, have been the main focus of recent updates to library spaces over the last 10 years. as other researchers have noted, these spaces support student learning, including collaboration, social learning, and alone-together study (cha & kim, 2015; given & archibald, 2015; khoo et al., 2014; ojennus & watts, 2017). interestingly, focus group participants repeatedly said they like the group work stations for completing group work, yet users were rarely using these tables as intended during sweeps. more often, users at these tables used the integrated outlets to power their laptops, leading the research team to believe students like these tables more for their outlets and less for the ability to share a screen.   demand for a stress relief room and nap pods signals that while users come for serious work, they feel the library should, or could, serve as a comfortable, relaxing environment, indicative of the “third space” discussed by declercq and cranz (2014). focus group and survey results revealed undergraduates frequently come to the library before classes or after dinner for printing and academic work, while they come in between classes for group work and socializing. despite low participation in focus groups, graduate students unsurprisingly indicated their use of library spaces is almost wholly academic, citing a need for quiet and a fondness for academic-oriented library services. students’ motivations for library use need to be considered for any library planning renovations and new services, especially when faced with increasingly diverse student populations. this is something lavery library must take into account given our increasing graduate population.   space attributes   students use library spaces for a variety of reasons; most commonly, data revealed users come to the library for academic work. space needs differ among users and are often task-dependent, with both individual and group work requiring a variety of furniture options. independent of group or individual study spaces, more table and seating options are a common theme within focus groups and survey findings, aligning our students’ desires with other research on space attributes (cha & kim, 2015; i̇mamoğlu & gürel, 2016; khoo et al., 2014; khoo, rozaklis, hall, & kusunoki, 2016; montgomery, 2014; vaska et al., 2009). regardless of space preferences (i.e., quiet vs. non-quiet), users consistently and whenever possible need additional outlets, aligning with research regarding the need for additional power to accommodate technologies (declercq & cranz, 2014; i̇mamoğlu & gürel, 2016; khoo et al., 2014; montgomery, 2014; vaska et al., 2009). the need for more outlets, aside from the building’s age, may stem from multiple, simultaneous device use (i.e., laptop, cell phone, desktop) found in sweeps data. builders in 1974 could not have predicted the pervasiveness of technology today, but future renovations must address power capacity.    the library’s main level is a mix of desktop computer pods, group workstations, lounge furniture, and other flexible spaces, and is frequently abuzz with students working on group projects, studying together, and socializing. it is also where the checkout desk and research help desk are located; these two desks are frequently busy with library users seeking assistance with research, utilizing technology, checking out materials, and performing other activities. the main level is certainly what freeman (2005) would consider “the sound of learning” (p. 5), with sweeps, focus groups, and survey responses indicating the library is used frequently for group work. however, the main level does have its drawbacks for group work. for example, it is possible group workstations are not as frequently used as intended due to a lack of privacy. based on focus group findings, group workspaces should be addressed in library renovations, specifically the addition of break-out rooms or other semi-private spaces with soundproofing.   particularly surprising throughout all phases of research is the under-use of the lower level. this is a mixed-use, flexible space where talking is allowed, but is typically quieter than the main level. occupancy during sweeps was less than 1% and students rarely mentioned the lower level during focus groups. this trend continued in survey responses, with approximately 50% of respondents “never” going to any lower level spaces (i.e., “basementcomputer lab” and “basementtables”). understanding why students are not using this available space would be extremely valuable. as khoo et al.(2016) mentions, spaces without defined use conventions are considered full when they are relatively unoccupied, as individuals are unlikely to join a space already occupied by another individual. in the case of the lower level, this might be doubly true, as the classrooms on this level are not commonly used outside of instruction and students may be unaware of when they are able to, or not able to, use these rooms. other factors contributing to underuse could be the lack of natural lighting, undefined policies regarding noise, and temperature. the only available lighting in the lower level is fluorescent lighting; there are no desk lamps and only one semi-hidden space with windows. while the only designated quiet space in the library is the upper level, the lower level is much quieter than the main level. lastly, underuse may be a result of temperature variance, something noted in the focus groups and survey findings.   as with other research (cha & kim, 2015; declercq & cranz, 2014; freeman, 2005; khoo et al., 2014, khoo et al., 2016), our students are looking for a quiet space to “get serious” (e.g., write research papers). this is especially true for masters and doctoral students, including one doctoral student wishing the library would be more like a neighboring academic library, where the entire space is quiet. this population’s need for quiet space may stem from different academic requirements (e.g., dissertation research), or the need for quiet space outside of home or work. not surprisingly, many undergraduates indicated a desire for quiet space as well, specifically when concentration is required, as the library main level can be noisy. what is particularly interesting, especially in lieu of survey results, is upper level sweeps have only about 20% occupancy, even during peak usage. it is possible students see the space as full at 20% occupancy, rather than the 40-50% reported in other literature (applegate, 2009; i̇mamoğlu & gürel, 2016; khoo et al., 2016). for example, once a study room has one person using the space it is considered full, even though there may be 2-3 available chairs in the room. similarly, as noted in the sweeps and focus groups, a single student may use an entire four-person table, making the space full with only one occupant. i̇mamoğlu and gürel (2016) write about territorial dividers as a way to maintain personal space, and something focus group participants mentioned wanting were smaller, individual work tables in place of the large four-person tables currently available. this follows trends for communal study, or alone-together study, where students seek silence lacking in other areas (e.g., dorm rooms, classrooms, residence hall lounges, and others), but still want to be around others working on similar tasks. it is clear from all three data points that quiet study space is highly valued and sought after on campus, and the library, while providing some quiet, still requires more to meet demand. this is consistent with recent literature about growing demands for quiet spaces, and libraries should consider this growing body of evidence as they plan for renovations.   space-related services   while not solely library-related, participants in all areas of research suggested the library add café and stress-relief services. café service was not surprising given the percentage of people observed during sweeps with food or drink. as the survey found, students frequently visit the library between classes and throughout the day and grab ‘n go foods was rated highly as a need in the survey (see figure 2), having café access would benefit students. this leads researchers to conclude current vending options are inadequate, including the new single-serve coffee machine. out of a specific request for nap pods within the focus groups, student researchers included an option of “stress relief room with nap pods/ massage chairs/ stress relieving activities” for the survey question “i think the library needs…” surprising to librarian researchers, the request for stress relief services came in second to “more quiet rooms.” considering the other spaces on campus in which students elect to study and complete work (e.g., cafés, residence hall lounges, and more), the desire for space-related services, including grab ‘n go foods and stress relief rooms, is very important.   extended hours and interlibrary loan are two other services frequently mentioned in both the focus groups and survey. the request for extended hours has persisted for years, and the library has adjusted hours to open earlier and close later on weekends, including staying open until 2 a.m. during the last two weeks of the semester. the study did not determine what extended hours would mean to users, but existing headcount data does not support a need for extended hours. we acknowledge this could be due to students knowing the library is closing, and therefore moving to an alternate location long before closing. unrelated to library space, praise for interlibrary loan was common throughout all user types in the focus groups. researchers are unsure why this service connects to library spaces for users, though it is possible students have picked up physical interlibrary loan materials at the checkout desk, or focus group questions about space-related library services evoked positive feelings toward this service.   limitations   the researchers acknowledge this research had limitations. multiple recorders’ interpretations during the seating sweeps may influence data. the librarians conducting the research tried to mitigate this by training staff recorders with a shared understanding of what to record.   due to low focus group participation, masters and doctoral students were not surveyed. similarly, a purposeful decision to exclude demographics was made to shorten the survey. therefore, researchers are unable to relate survey responses back to specific demographic traits (e.g., commuter vs. residential, class level), which may have proved valuable for understanding how different student groups use library spaces.   future research   future space studies should investigate students’ need for quiet study spaces, and how libraries may provide these spaces to their students. the need for quiet space may signify a change from previous trends regarding redesigned library spaces. in small academic libraries, would it better serve students to have more quiet spaces than collaborative spaces, since the latter can be found other places on campus? it is also worth exploring students’ use of undefined spaces, which may be common in academic libraries.   acknowledgements   the authors would like to acknowledge others that were instrumental to the success of this library space study: faculty advisers: patricia tweet, phd and david baronov, phd; student researchers: mollie flynn and caroline villa; and library staff members: kate ross, marianne simmons, brian lynch, lynn seavy, stacy celata, and britta stackwick.   references   andrews, c., wright, s. e., & raskin, h. (2015). library learning spaces: investigating libraries and investing in student feedback. journal of library administration, 56, 647-672. http://dx.doi.org/10.1080/01930826.2015.1105556    applegate, r. (2009). the library is for studying: student preference for study space. the journal of academic librarianship, 35, 341-346. http://dx.doi.org/10.1016/j.acalib.2009.04.004   cha, s. h., & kim, t. w. (2015). what matters for students’ use of physical library space? the journal of academic librarianship, 41, 274-279. http://dx.doi.org/10.1016/j.acalib.2015.03.014   declercq, c. p., & cranz, g. (2014). moving beyond seating-centered learning environments: opportunities and challenges identified in a poe of a campus library. the journal of academic librarianship, 40, 574-584. http://dx.doi.org/10.1016/j.acalib.2014.08.005   ellison, w. (2016). designing the learning spaces of a university library. new library world, 117, 294-307. http://dx.doi.org/10.1108/nlw-01-2016-0006   fallin, l. (2016). beyond books: the concept of the academic library as learning space. new library world, 117, 308-320. http://dx.doi.org/10.1108/nlw-10-2015-0079   freeman, g. t. (2005). the library as place: changes in learning patterns, collections, technology, and use. in library as place: rethinking roles, rethinking space (pp. 1-10). council on library resources: washington, dc.   given, l. m., & archibald, h. (2015). visual traffic sweeps (vts): a research method for mapping user activities in the library space. library & information science research, 37, 100-108. http://dx.doi.org/10.1016/j.lisr.2015.02.005   given, l. m., & leckie, g. l. (2003). “sweeping” the library: mapping the social activity space of the public library. library & information science research, 25, 365-385. http://dx.doi.org/10.1016/s0740-8188(03)00049-5   goodnight, c., & jeitner, e. (2016). sending out an sos: being mindful of students’ need for quiet study spaces. in s. s. hines & k. m. crowe (eds.), the future of library spaces (vol. 36, pp. 103-129). emerald group publishing limited. http://dx.doi.org/10.1108/s0732-067120160000036010   hall, k., & kapa, d. (2015). silent and independent: student use of academic library study space. partnership: the canadian journal of library and information practice and research, 10(1), 1-38.   hanson, a., & abresch, j. (2016). socially constructing library as place and space. in s. s. hines & k. m. crowe (eds.), the future of library spaces (vol. 36, pp. 103-129). emerald group publishing limited. http://dx.doi.org/10.1108/s0732-067120160000036004     i̇mamoğlu, ç., & gürel, m. ö. (2016). “good fences make good neighbors”: territorial dividers increase user satisfaction and efficiency in library study spaces. the journal of academic librarianship, 42, 65-73. http://dx.doi.org/10.1016/j.acalib.2015.10.009   khoo, m., rozaklis, l., hall, c., & kusunoki, d. (2016). “a really nice spot”: evaluating place, space, and technology in academic libraries. college & research libraries, 77, 51-70. http://dx.doi.org/10.5860/crl.77.1.51   khoo, m., rozaklis, l., hall, c., kusunoki, d., & rehrig, m. (2014). heat map visualization of seating patterns in an academic library. in iconference 2014 proceedings (p. 612-620). http://dx.doi.org/10.9776/14274   lux, v., snyder, r. j., & boff, c. (2016). why users come to the library: a case study of library and non-library units. the journal of academic librarianship, 42, 109-117. http://dx.doi.org/10.1016/j.acalib.2016.01.004   montgomery, s. e. (2014). library space assessment: user learning behaviors in the library. the journal of academic librarianship, 40, 70-75. http://dx.doi.org/10.1016/j.acalib.2013.11.003   montgomery, s. e., & miller, j. (2011). the third place: the library as collaborative and community space in a time of fiscal restraints. college & undergraduate libraries, 18, 228-238. http://dx.doi.org/10.1080/10691316.2011.577683 ojennus, p., & watts, k. a. (2017). user preferences and library space at whitworth university library. journal of library and information sciences, 49, 320-334. http://dx.doi.org/10.1177/0961000615592947   oldenburg, r. (1997). making college a great place to talk. in g. keller (ed.), the best of planning for higher education (pp. 90-94). ann arbor, mi: society for college and university planning.   oliveira, s. m. (2016). space preference at james white library: what students really want. the journal of academic librarianship, 42, 355-367. http://dx.doi.org/10.1016/j.acalib.2016.05.009   vaska, m., chan, r., & powelson, s. (2009). results of a user survey to determine needs for a health sciences library renovation. new review of academic librarianship, 15, 219-234. http://dx.doi.org/10.1080/13612530903240635   appendix a library maps             appendix b library space assessment survey   q1 when i go to the library, i...   ▢   ▢  very often (1) ▢  sometimes (2) ▢  never (3) ▢  study (1) o   o   o   ▢  use the computers (2) o   o   o   ▢  print/ make copies (3) o   o   o   ▢  do group work (4) o   o   o   ▢  do important projects (5) o   o   o   ▢  write papers (6) o   o   o   ▢  receive/offer tutoring (7) o   o   o   ▢  research (8) o   o   o   ▢  socialize (9) o   o   o   ▢  check out a book (10) o   o   o         q2 i go to the library... ▢   ▢  very often (1) ▢  sometimes (2) ▢  never (3) ▢  monday-thursday (1) o   o   o   ▢  friday (2) o   o   o   ▢  saturday (3) o   o   o   ▢  sunday (4) o   o   o   ▢  morning (5) o   o   o   ▢  between classes (6) o   o   o   ▢  evening hours (7) o   o   o   ▢  during finals week (8) o   o   o       q3 when i go to the library, i go to... ▢   ▢  very often (1) ▢  sometimes (2) ▢  never (3) ▢  rooms on the quiet floor (1) o   o   o   ▢  quiet flooropen area (2) o   o   o   ▢  keating room (tutoring) (3) o   o   o   ▢  main flooropen area (4) o   o   o   ▢  basementcomputer lab (5) o   o   o   ▢  basementtables (6) o   o   o   ▢  other (7) o   o   o       q4 when i go to the library i use... ▢   ▢  very often (1) ▢  sometimes (2) ▢  never (3) ▢  computers and rentable laptops (1) o   o   o   ▢  research help desk (2) o   o   o   ▢  white boards (3) o   o   o   ▢  group tables with tv screens (4) o   o   o   ▢  smart bones (5) o   o   o   ▢  rentable games/movies (6) o   o   o   ▢  interlibrary loan (7) o   o   o   ▢  keurig (8) o   o   o   ▢  rentable  chargers/headphones (9) o   o   o       q5 the library tends to be... ▢   ▢  very often (1) ▢  sometimes (2) ▢  never (3) ▢  too crowded (1) o   o   o   ▢  too noisy (2) o   o   o   ▢  too hot (3) o   o   o   ▢  too cold (4) o   o   o       q6 i think the library needs... (check all that apply) ▢  stress relief room with nap pods/ massage chairs/ stress relieving activities  (1) ▢  fans and air conditioning  (2) ▢  more quiet rooms (soundproof)  (3) ▢  classrooms on main floor  (4) ▢  extended hours  (5) ▢  more white boards  (6) ▢  more tables  (7) ▢  more computers on the main floor  (8) ▢  more computers on the quiet floor  (9) ▢  outlets/tables with outlets  (10) ▢  grab 'n go foods  (11) ▢  lounge chairs  (12) ▢  study chairs  (13) ▢  other  (14) ________________________________________________       q7 other places i study on campus include... (check all that apply) ▢  salerno study labs  (1) ▢  dorm rooms  (2) ▢  student clubs & organizations office  (3) ▢  residence hall lounges  (4) ▢  pioch cafe  (5) ▢  cyber cafe  (6) ▢  classrooms  (7) ▢  nursing common area  (8) ▢  ishs lounge tables  (9) ▢  cop  (10) ▢  cop3  (11) ▢  michaelhouse computer lab  (12) ▢  kearney computer lab  (13) ▢  mainstage  (14) ▢  outside  (15) ▢  other  (16) ________________________________________________       q8 what updated features in the library are most important to you? please rank order of least important to most important ; 1 equals most important. ______ updated lighting (1) ______ updated carpeting (2) ______ more windows (3) ______ updated wall colors (4) ______ more nature (e.g., plants) (5)         evidence summary   low level evidence suggests that librarian-led instruction in evidence based practice is effective regardless of instructional model   a review of: swanberg, s. m., dennison, c. c., farrell, a., machel, v., marton, c., o'brien, k. k., … & holyoke, a. n. (2016). instructional methods used by health sciences librarians to teach evidence based practice (ebp): a systematic review. journal of the medical library association: jmla, 104(3), 197-208. http://dx.doi.org/10.3163/1536-5050.104.3.004   reviewed by: lindsay alcock head, public services health sciences library memorial university of newfoundland st. john’s, newfoundland, canada email: lalcock@mun.ca   received: 27 feb. 2017   accepted: 17 apr. 2017      2017 alcock. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract    objective – to determine both the instructional methods and their effectiveness in teaching evidence based practice (ebp) by librarians in health sciences curricula.   design – systematic review.   setting – a total of 16 databases, google scholar, and mla annual meeting abstracts.   subjects – there were 27 studies identified through a systematic literature search.   methods – an exhaustive list of potential articles was gathered through searching 16 online databases, google scholar, and mla annual conference abstracts. inclusion and exclusion criteria were identified to inform the literature search and determine article eligibility. duplicates were removed and the remaining search results were divided into sets and assigned to two reviewers who screened first by title/abstract and then by full-text. a third reviewer addressed disagreement in article inclusion. data extraction, using a validated method described by koufogiannakis and wiebe (2006), and critical appraisal, using the glasgow checklist (1999), were performed concurrently.   main results – after removal of duplicates 30,043 articles were identified for initial title/abstract screening. of the 637 articles assessed for full-text screening 26 articles and 1 conference proceeding ultimately met all eligibility criteria. there was no meta-analysis included in the synthesis. there were 16 articles published in library and information science journals and 10 in health sciences journals. of those studies, 22 were conducted in the united states. a wide range of user groups was identified as participants in the studies with medical students and residents representing the highest percentage and nursing and other allied health professional programs also included. while there was variation in sample size and group allocation, the authors estimate an average of 50 participants per instructional session. included studies represented research undertaken since the 1990s. all studies addressed at least one of the standard ebp steps including obtaining the best evidence through a literature search (27 studies), developing a clinical question (22 studies), and critical appraisal (12 studies).  there were 11 studies which addressed applying evidence to clinical scenarios, and 1 study which addressed the efficacy and efficiency of the ebp process. the majority of studies indicated that literature searching was the primary focus of ebp instruction with medline being the most utilized database and cochrane second. other resources include databases and clinical decision support tools.   teaching methods, including lecture, small group, computer lab, and online instruction, varied amongst the studies. there were 7 studies which employed 1 instructional method while 20 employed a combination of teaching methods. only one study compared instructional methods and found that students obtained better scores when they received online instruction as compared with face-to-face instruction. the difference, however, was not statistically significant.   skills assessments were conducted in most of the studies utilizing various measurements both validated and not validated. given the variation in measurement tools a cross-study analysis was not possible. the most common assessment methods included self-reporting and preand post-surveys of participants’ attitudes and confidence in ebp skills.   randomization was utilized in 10 studies, and an additional 3 studies had a “clearly defined intervention group.” there were 10 blinded studies and 15 studies utilized cohorts with preand postintervention assessments. there were 25 studies which included descriptive statistics and many also included inferential statistics intended to show significance. differences between groups were assessed with parametric measures in 9 studies and non-parametric measures in 15 studies. good to high statistical significance on at least 1 measurement was achieved in 23 studies. given the absence of effect sizes, the level of differences between study groups could not be determined.    conclusion – numerous pedagogical methods are used in librarian-led instruction in evidence based practice. however, there is a paucity of high level evidence and the literature suggests that no instructional method is demonstrated to be more effective than another.   commentary   as one of the 15 questions/research priorities identified by the mla delphi study (eldredge, ascher, holmes & harris, 2012) this study filled a clear gap in the literature while also addressing the need for more rigorous comparative studies in ebp instruction. with an increasing focus on demonstrating value and impact, the role of the librarian in curricular outcomes – particularly those outcomes as defined by accreditation – needs to be evidence based. more rigorous studies would provide compelling evidence to support the importance of librarian-provided ebp instruction in medical education. the methodology for this systematic review is sound, however the variability of the available studies makes overall comparison and significance difficult to determine.   the search strategy was robust and included 16 databases as well as google scholar. it would be interesting to know which included studies came from which databases. while the choice of databases seems exhaustive it could also be considered excessive. would librarians, for example, expect to see such a list from others in a systematic review? that said, the choice of databases seems to reflect the international collaboration of authors on this project, which is positive and inclusive. the authors chose to search abstracts from only one conference. while the mla annual meeting is large and representative, perhaps eahil, eblip, and chla may have also been considered.   initial screening of the first 500 results from google scholar and the 2009-2014 mla annual meeting abstracts yielded recent results. however, a publication date limit was not utilized in the database searches and therefore the overall results included articles covering two decades. one must consider, for example, that studies from the 1990s are not necessarily comparable to studies from the 2010s given changes in search methods, online advances, assessment methods, etc. how ebp is taught largely depends on how ebp is practiced and one may argue that certainly the literature searching aspect of ebp has changed significantly over time.   the results are presented clearly and the discussion is insightful and provocative. in addition to the results described above, the authors gleaned valuable information from the studies including the fact that librarian authors tended to publish in library and information science journals while medical faculty published in medical journals. this is interesting considering that librarians in all 27 studies held an instructor role with duties that included curriculum development, teaching, and assessment.   references   eldredge, j. d., ascher, m. t., holmes, h. n., & harris, m. r. (2012). the new medical library association research agenda: final results from a three-phase delphi study. journal of the medical library association, 100(3), 214-8. http://dx.doi.org/10.3163/1536-5050.100.3.012   koufogiannakis, d., & wiebe, n. (2006). effective methods for teaching information literacy skills to undergraduate students: a systematic review and meta-analysis. evidence based library and information practice, 1(3), 3-43. http://dx.doi.org/10.18438/b8ms3d   morrison, j. m., sullivan, f., murray, e., & jolly, b. (1999). evidence-based education: development of an instrument to critically appraise reports of educational interventions. medical education, 33(12), 890-3. http://dx.doi.org/10.1046/j.1365-2923.1999.00479.x.     classics   in the growing information mall, some things never change   a review of: fidel, r., davies, r. k., douglass, m. h., holder, j. k., hopkins, c. j., kushner, e. j.,…. toney, c. d. (1999). a visit to the information mall: web searching behavior of high school students. journal of the american society for information science, 50(1), 24-37. http://dx.doi.org/10.1002/(sici)1097-4571(1999)50:1<24::aid-asi5>3.0.co;2-w   reviewed by: hailley fargo student engagement librarian penn state university libraries university park, pennsylvania, united states of america email: hmf14@psu.edu   received: 21 aug. 2017 accepted: 29 sept. 2017      2017 fargo. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – the research study aimed to discover high school students’ information searching behaviour on the web and suggest web changes that would benefit student learning. this study was conducted in 1999, seven years after the internet was publicly available and on the cusp of web 2.0.   design – field study with class observations, students thinking aloud at their terminals, and interviews with the students after their     searching. the study’s duration was three class searching sessions.   setting – west seattle high school in seattle, washington. this school had a diverse population of students, with 50% students of color and many of these students first generation to finish high school. due to a grant from microsoft, west seattle had operational four computer labs.   subjects – eleventh and twelfth graders in a horticulture class. there were eight student participants, six males and two females. five of these students were in 12th grade and three were in 11th grade. the teacher for this class, the school librarian, and the principal of west seattle high school were also interviewed for this project.   methods – qualitative, case-study method was used with controlled comparison. team members observed the students while they searched and wrote down descriptions of the students’ searching methods. after the three observation sessions and interviews with the students, team members wrote up a case study for each student. the students’ think-aloud audio, along with all the interviews conducted, were recorded. this type of method can be considered an early version of usability testing and user experience studies, a field that has grown tremendously since 1999.     main results – while each student observed had a different relationship with the web and training on how to use it, similar searching strategies emerged from all participants. these strategies included focused searching, swift and flexible searching when results were not immediately found, using a webpage as a landmark to return to while searching, starting a new search, and asking for help when needed. it should be noted that focused searching along with the swift, flexible searching were strategies influenced by student motivation to complete their homework assignment as quickly as possible. the team noted exploration of the web was kept to a minimum and this was due to the parameters of the assignment. team members also identified similar frustrations and joys from the students when searching the web. the study identified three steps that should be taken to help students more effectively navigate the web. the steps included an increase in formal teaching on web searching, embedded support in the web to help students search, and relying on graphics to strengthen a web experience.   conclusion – authors noted the possibilities the world wide web has to offer, especially in a school context. however, in order to fully maximize those possibilities, the web needs to take into account user experiences and information seeking behaviour, along with an increase in training on how to use the web.   commentary   although nearing twenty years old, reading “a visit to the information mall” feels like a conversation that librarians and teachers could be having right now. fidel et al.’s work is a classic because it captured a snapshot of teens’ information seeking behaviours, behaviours which are still relevant and being researched by scholars today. while the internet has improved on some of the weaknesses the authors identified, the authors’ discussion still resonates with readers.   today, this article has been cited hundreds of times in a variety of disciplines and publications. we, as researchers, are still curious in the web searching behaviours of elementary, middle, and high school students along with undergraduate students and adults. fidel et al.’s article serves as a touchpoint in many literature reviews and to confirm findings related to desire for immediate results without broken links (ho, lin, & chen, 2012), flexible searching techniques (borlund, 2016), and relying on the use of graphical images (chow, smith, & sun, 2012). the case study, interviews, and think-aloud method used in this study remain a popular way to observe and record information searching behaviours to make conclusions. today, we see this method employed in user experience and web usability testing to help inform the creation and functionality of websites and web design.    this article also captured an important time in the growth of the internet. fidel et al. are writing on the cusp of some changes that have impacted the way we use the web and find information. we can see this edge of change in this article through the suggestions fidel et al. make for improving the user’s experience. one suggestion is the need for support in searching the internet. they suggest including a way to have “easy and immediate access to an encyclopedia” (fidel et al., 1999, p. 34). two years after this article was published, wikipedia was created. as “a multilingual, web-based, free-content encyclopedia project…based on the model of openly editable content” (wikipedia, 2017) wikipedia serves the purpose suggested by fidel et al. today, internet users are familiar with this website and use it for gathering background information. additionally, beyond wikipedia, we also have more traditional, online encyclopedias like gale virtual reference library, credo reference, or britannica student encyclopedia for users to quickly access background content on a topic or idea.     during the time that fidel et al. were working on this study, google was just beginning to gain traction. founders larry page and sergey brin received their first investor check in august 1998 (google, n.d.). with the rise of google, some of the frustrations with internet searching identified in this article were alleviated. as a search engine, google can help automatically correct spelling errors, uses several algorithms to rank results according to relevancy and popularity, provides alternate search suggestions, and employs graphical images to provide context and visual confirmation on a topic. fidel et al.’s article captures the way the internet was before these powerhouses entered the game and serve as a good reminder of the progress and growth of the internet.   one of the biggest reasons this article should be considered a classic is the way it justifies the need to provide adequate training to teachers and students on how to use the web because the authors believe it is “necessary for optimal learning of new topics” (fidel et al., 1999, p. 34). without some sort of formal web training, the students are left to their own devices, coming up with strategies that might be inefficient and frustrating. this creates further tension between the user and the internet when speed declines or the user becomes impatient with poor search results.    even though today some of us might consider that digital natives intuitively know how to use the internet, this article makes a strong case for why training is needed. the authors identify that the students are switching between learning how to use the web and finding the information they need for their class assignment; they are unable to do both tasks well. without any training, students gravitate toward finding the information as quickly as possible to get a good grade. this is then compounded by the fact the students are motivated to find information for their assignment; due to the parameters of the assignment, students then create searches and strategies all based around the ability to find what they need as quickly as possible. with this strategy, they are missing the process of seeking new information, developing search strategies, and making conclusions based on their findings. articles published after fidel et al.’s often point to this article as one where they recognized that students are not efficient in coming up with search strategies or the ability to evaluate information on the web. fidel et al. (1999) capture this best when they say, “with training, teachers could think of ways to use the possibilities of the internet to challenge students to learn, not just to retrieve bits of information for assignments” (p. 34). this reframing really seeks to highlight a transformation in using the internet in a school setting, and is something that current researchers continue to explore. furthermore, as librarians who teach information and digital literacy, we know the benefits of providing a framework for students seeking information or wishing to use technology. when students are equipped with those baseline skills, it provides an opportunity to really dig into more abstract concepts on finding, using, and creating new information.      the world wide web has probably surpassed the size of “a shopping mall the size of seattle” (fidel et al., 1999, p. 24). however, no matter how large the internet has grown, we can see similarities between the eight seattle high school students and the way we use the internet today. fidel et al.’s article is a good reminder of the changes the internet has undergone to improve searching while also pointing to training that still needs to be done with users.   references   borlund, p. (2016) framing of different types of information needs within simulated work task situation: an empirical study in the school context. journal of information science, 43(3), 313-323. https://doi.org/10.1177/0165551515625028   chow, a. s., smith, k. m., & sun, k. (2012). youth as design patterns: age-appropriate websites for middle and high school students. educational technology & society, 15(4), 89-103. retrieved from http://ezaccess.libraries.psu.edu/login?url=http://search.ebscohost.com/login.aspx?direct=true&db=a9h&an=83234621&site=ehost-live&scope=site   fidel, r., davies, r. k., douglass, m. h., holder, j. k., hopkins, c. j., kushner, e. j.,…toney, c. d. (1999). a visit to the information mall: web searching behavior of high school students. journal of the american society for information science, 50(1), 24-37. http://dx.doi.org/10.1002/(sici)1097-4571(1999)50:1<24::aid-asi5>3.0.co;2-w   ho, c., lin, m., & chen, h. (2012). web users’ behavioural patterns of tourism information search: from online to offline. tourism management, 33(6), 1468-1482. https://doi.org/10.1016/j.tourman.2012.01.016   google. (n.d.). our story: from the garage to the googleplex. retrieved from https://www.google.com/intl/en/about/our-story/   wikipedia. (2017). wikipedia:about. retrieved from https://en.wikipedia.org/wiki/wikipedia:about   news   10th international evidence based library and information practice conference awarded to glasgow, scotland for 2019      2018. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29403     glasgow, scotland will host eblip10 in 2019. the tenth in a successful series of international conferences, eblip10 will promote evidence-based approaches to library and information practice.   dr. rebekah willson and dr. diane pennington, both lecturers in information science at university of strathclyde, glasgow, scotland coordinated the glasgow entry into the competitive selection process. professor alison brettle, university of salford, u.k., will chair the international programme committee.   criteria for selection included the location, facilities, regional and national support, the strength of the organizing team and unique selling points. the committee commended the glasgow organizers for the overall quality of the proposal including convenient proximity of conference venues, variety of accommodation selections, local attractions, and excellent support for the eblip movement.   the biennial eblip conference promotes the use of best available evidence to improve library and information practices in all types of libraries. it aims to bring together practitioners, researchers, and students involved in critical and reflective information practice to think through new ways to address contemporary issues of evidence based practice in less isolated, more collaborative, ways. the first ebl(ip) conference took place in sheffield, uk in 2001. subsequent conferences have been hosted in edmonton (canada, 2003), brisbane (australia, 2005 and 2015), chapel hill (usa, 2007), stockholm (sweden, 2009), salford (uk, 2011), saskatoon (canada, 2013), and philadelphia (usa, 2017).   “the university of strathclyde is excited to host the tenth international evidence based library and information practice conference,” said rebekah willson, co-chair of the local organizing committee. “scotland has a vibrant library community and we are looking forward to welcoming practitioners and researchers from around the world to the eblip10 conference, for a chance to make connections, as well as experience all glasgow has to offer.”   queries about eblip10 can be directed to eblip10conference@gmail.com   research article   relevance of a french national database dedicated to infection prevention and control (nosobase®): a three-step quality evaluation of a specialized bibliographic database   kae ting trouilloud infection control pharmacist mobile infection prevention team for nursing homes (emhe) lyon university hospital (hcl) lyon, france email: kae.trouilloud@chu-lyon.fr nathalie sanlaville librarian healthcare-associated infection prevention and control regional coordinating centre for auvergne-rhone-alpes (cpias ara) lyon, france email: nathalie.sanlaville@chu-lyon.fr   sandrine yvars head of documentation center for professionals national police academy (ensp) saint-cyr-au-mont-d’or, france email: yvars.sandrine@wanadoo.fr   anne savey director (ipc) healthcare-associated infection prevention and control regional coordinating centre for auvergne-rhone-alpes (cpias ara) lyon, france email: anne.savey@chu-lyon.fr   received: 15 may 2018                                                                  accepted: 16 jan. 2019      2019 ting trouilloud, sanlaville, yvars, and savey. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29448     abstract   objective – nosobase® is a collection of documentation centres with a national bibliographic database dedicated to infection prevention and control (ipc), with over 20 years of experience in france. as a quality assurance activity, this study was conducted in 2017 with a three-step approach to evaluate the bibliographic database regarding (1) the availability and coverage of citations; (2) the scope and relevance of content; and (3) the quality of the documentation centre services.   methods – the three-step quality approach involved (1) evaluating the availability and coverage of citations in nosobase® by searching for the bibliographic citations of three systematic reviews on hand hygiene practices, published recently in three different peer-reviewed international journals; (2) evaluating the scope and relevance of content in nosobase® by searching for all documents from 2015 indexed in nosobase® under hand hygiene related keywords, and analyzing according to publication language, document type (e.g., legislation, research, or guidelines), and target audience; and 3) evaluating the strengths, weaknesses, and opportunities of the documentation centre services, with interviews involving the librarians.   results – nosobase® contained 70.8%-80.9% of references directly concerning hand hygiene cited by the three systematic reviews. of the 200 articles indexed in nosobase® under hand hygiene related keywords in 2015, 22.5% were french language based, with a significant representation of french non-indexed literature. the analysis of the documentation centre services highlighted future opportunities for growth, building on the strengths of experience and collaborations, to improve marketing and usability, targeting francophone ipc professionals.   conclusion – specialized bibliographic databases may be useful and time efficient for the retrieval of relevant specialized content. nosobase® has significant relevance to french and francophone healthcare professionals in its representation of french documentation and healthcare literature not otherwise indexed internationally. nosobase® needs to highlight its resources and adapt its services to allow easier access to its content.     introduction   with the growing emphasis on evidence based practice, more databases and research content are now being made available. yet it is often difficult and time-consuming for clinicians and researchers to locate relevant literature even if it is more accessible. studies indicate that searches only on google scholar may not be enough (boeker, vach, & motschall, 2013), and that searches on multiple databases are often necessary for finding relevant bibliographic content (rathbone, carter, hoffmann, & glasziou, 2016). one earlier study compared several health-related databases (cinahl, cochrane library, medline, psyclit, sociofile, and social science citation index), and found that psyclit was the most useful database for information on the rehabilitation of people with severe mental illness in terms of search efficiency and relevance in this topic area, highlighting the importance of a specialized bibliographic database (brettle & long, 2001). more recently, rethlefsen, murad, and livingston (2014) proposed that human-indexed databases allow faster, more comprehensive searching in terms of terminology and controlled vocabulary structure than solely computer algorithm-indexed databases such as scopus and google scholar, despite search engines that search full-text articles in the latter. the impact of trained searching and assistance from trained information professionals and librarians has also been underlined (centre for reviews and dissemination, 2009). librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews, leading to a more comprehensive, true systematic review (rethlefsen, farrell, osterhaus trzasko, & brigham, 2015).   as a quality assurance activity, this study evaluated nosobase®. created in 1996, nosobase® is a national project focused on infection prevention and control (ipc) in healthcare settings (savey, sanlaville, & fabry, 2000). it consists of five documentation centres, with trained librarians located in different cities in france (lyon, paris, rennes, bordeaux, and nancy). it hosts a forum for ipc professionals and a website with resources and tools. there are collaborations with national medical or professional ipc learned societies, cooperation with other libraries, and bibliographic support to the national public health agency. based in the french language, it manages a bibliographic database also known as nosobase®, indexing up to 180 journals in multiple languages. this study describes a three-step quality assessment conducted in 2017 to evaluate the content of the database (steps 1 and 2) and the documentation centre services (step 3).   literature review   bibliographic databases came into being in the 19th and 20th centuries in response to the proliferation of journals and other publications, then went online in the late 20th century (glasziou & aronson, 2017). specialized (or subject-specific) bibliographic databases are now available in diverse fields, from chemistry (chemical abstract service) to psychology, psychiatry, and neurology (psycinfo); from nursing and caring sciences (cinahl) to geology (geobase) (gasparyan et al., 2016). many offer access mainly through subscription or through membership in professional associations. in france, bibliographic databases such as pascal & francis and the public health database banque de données en santé publique (bdsp) were developed in the 1970s and 1990s, respectively. it was noted that in the pascal database, around 12% of the journals indexed were french language based, and 55% of the journals indexed were not available in pubmed (dufour, mancini, & fieschi, 2009).   however, bibliographic databases and the documentation centres that maintain them currently face growing costs for maintenance and rising competition from multidisciplinary databases such as pubmed that provide free online access. researchers began to compare recall efficacy between multidisciplinary databases and specialized databases. one case study used a systematic review to investigate the performance of bibliographic databases in identifying the included studies. its results showed that the use of at least two databases and reference checking were required to retrieve all included studies (beyer & wright, 2013). another study examined the yield of medline, embase, and central to find randomized controlled trials within the area of musculoskeletal disorders. it found that searching all three databases was not sufficient for identifying all effect studies on musculoskeletal disorders, though an additional 10 databases did only increase the median recall by 2% (aagaard, lund, & juhl, 2016).   in france, the national conservatory of arts and crafts and national institute for documentation techniques (cnam intd, http://intd.cnam.fr/) developed methodologies to evaluate documentation centre services. these methodologies focused mainly on the organization of resources and processes and evaluating the management of the information systems, with an emphasis on quality assurance activity built into the process. quality indicators and action plans are identified to improve client satisfaction and overall documentation quality and service quality.    in the 1990s, there was also a progression towards practices based on evidence, or evidence based practice, and medicine was at the origin of this movement (goodman, 2002). libraries and documentation centres have joined the movement, examining the management of documents and information based on evidence, an approach known as evidence based librarianship (booth & brice, 2004). an example of this was a systematic review conducted in 2010, which examined models of clinician services and evaluated the value of their service towards clinicians. it described clinician libraries having a positive impact on patient care, resulting in better informed decisions on choices of drugs or therapy, and saving clinicians’ time (brettle et al., 2011).   however, during the same year, the dramatic evolution of information technologies led to a reexamination of the librarian role and a reevaluation of core competences by the canadian association of research libraries (2010). with the rise of google and internet access, studies also described changes in scholar and student information-seeking behavior (jamali & asadi, 2010). a recent study indicated that the coverage of google scholar is improving (gehanno, rollin, & darmoni, 2013). however, the findings of this study have been disputed. guistini and kamel boulos (2013) noted that google scholar was still unreliable for searching systematic reviews due to its constantly changing content, algorithms, and database structure. other studies found that google scholar missed important literature in evidence reviews and grey literature searching (haddaway, collins, coughlin, & kirk, 2015) or that it presented incomplete recall (bramer, giustini, kramer, & anderson, 2013).   in france, websites and portals in the french language dedicated to health literature have been in place for some time, such as cismef, recomedical, biu santé paris, and bdsp, a database for public health hosted by the national public health higher learning institution ecole des hautes etudes en santé publique (ehesp) in rennes. however, funding for bdsp has been greatly limited recently.   aims   after more than 20 years of experience, the librarians of the documentation centre nosobase® wanted to assess the quality and the usefulness of the services and tools provided. this study was conducted in 2017 with a three-step process to evaluate the bibliographic database nosobase® regarding (1) the availability and coverage of citations and (2) the scope and relevance of content, as well as (3) the quality of the nosobase® documentation centre services.   methods   a three-step quality assessment was used in order to evaluate the content of the database and the documentation centre services. step 1 evaluated the availability and coverage of citations in nosobase® through searching for bibliographic citations of three systematic reviews on hand hygiene practices, which were published recently in three different peer-reviewed international journals. step 2 evaluated the scope and relevance of content in nosobase® by searching for all documents from 2015 indexed in nosobase® under hand hygiene related keywords, and analyzing them according to publication language, document type (e.g., legislation, research, or guidelines), and target audience. step 3 evaluated the strengths, weaknesses, and opportunities of the documentation centre services, with input from the librarians.   step 1: availability and coverage of citations in nosobase®   the theme of hand hygiene was selected as it represents an important aspect of ipc. three systematic reviews on hand hygiene practices were selected from a simple straightforward search on pubmed (keywords: “hand hygiene systematic review” and “cochrane and hand hygiene systematic review”) for the most recent publication on january 26, 2017, representing three different peer-reviewed international medical journals. initially, the authors discussed the importance of including a cochrane systematic review, but at the time of searching none were found that fit the criteria of date and theme. we have chosen to leave the keyword search “cochrane” here for transparency to reflect the search process. the three selected reviews are listed below.   review 1: musuuza, j. s., barker, a., ngam, c., vellardita, l., & safdar, n. (2016). assessment of fidelity in interventions to improve hand hygiene of healthcare workers: a systematic review. infection control & hospital epidemiology, 37(5), 567-575.   ·         search: pubmed, cinahl, cochrane, web of science, up to 19 june 2015. ·         the review described limited electronic grey literature searching (p. 568). ·         keywords used: implementation fidelity, intervention fidelity, intervention compliance, hand washing, hand hygiene, hand disinfection. ·         120 citations; author affiliations: united states.   review 2: kingston, l., o’connell, n. h., & dunne, c. p. (2016). hand hygiene-related clinical trials reported since 2010: a systematic review. journal of hospital infection, 92(4), 309-320.   ·         search: pubmed, cinahl. studies from the us and europe, from dec 2009 (after the publication of the who hand hygiene guidelines), up to feb 2014. ·         the contact author confirmed that grey literature and hand searching were not conducted for this study. ·         keywords used: hand hygiene, hand washing, observation, and clinical trial. ·         88 citations; author affiliations: ireland.   review 3: luangasanatip, n., hongsuwan, m., limmathurotsakul, d., lubell, y., lee, a. s., harbarth, s., day, n. p. j., graves, n., & cooper, b. s. (2015). comparative efficacy of interventions to promote hand hygiene in hospital: systematic review and network meta-analysis. bmj, 351, h3728.   ·         search: medline, embase, cinahl, nhs economic evaluation database, nhs centre for reviews & dissemination, cochrane, epoc register, studies from dec 2009 to feb 2014. ·         the review appendix listed only electronic searching. ·         same keywords as former systematic reviews from 1980 to 2009. ·         89 citations; author affiliations: thailand, australia, united kingdom, switzerland. for each systematic review, all bibliographic citations were listed on a spreadsheet indicating availability, whether directly related to hand hygiene practices, and whether indexed under hand hygiene keywords, with a note for comments. their availability in the database of nosobase® was verified during february 2017.   step 2: scope and relevance of content in nosobase®   a search was performed on february 9, 2017, for all indexed documents under hand hygiene related keywords from the database of nosobase®. the year 2015 was chosen since publications from the end of 2016 were not yet completely indexed by february 2017. the documents were analyzed according to the publication language and document type (e.g., legislation, research article, or guidelines), as well as target audience.   step 3: quality review of the documentation centre services   using the national conservatory of arts and crafts and national institute for documentation techniques (cnam intd, http://intd.cnam.fr/) methodologies as models (toneatti, 2008; palisse, 2011), a qualitative descriptive approach was chosen for this study, examining six categories: communication, accessibility, production, management, marketing, and opportunities, through a modified swot analysis (strengths, weaknesses, and opportunities, with threats analyzed under weaknesses).   semi-structured face-to-face discussions using the modified swot analysis and categories were conducted with the on-site librarians (co-authors) separately. semi-structured discussions using the same analysis and categories were conducted by the librarian co-authors with three other librarians from the other nosobase® documentation centres, during regularly scheduled telephone conference meetings between february and july 2017, as part of the meeting agenda, and by electronic mail due to time and geographical constraints. report drafts resulting from input obtained during the above discussions were created. the librarian meetings were not recorded nor transcribed, but minutes were kept and circulated for verification, as per normal practice. ethics approval was not obtained as it was an internal quality assurance exercise.   results   step 1: availability and coverage of citations in nosobase®   there were a variety of citations included in the bibliographies of the three systematic reviews, including articles on methodology and economic impact (review 1); surveillance, good practice, and psychology (review 2); and systematic review methodologies, meta-analysis, statistics, and models (review 3). these and other non-hand hygiene related references were included in the category “other.” hand hygiene references were defined as articles directly related to research in hand hygiene practices. see table 1 and figure 1 for the comparison between the three systematic reviews and the database nosobase®.   from table 1, nosobase® is shown to have indexed 70.8%, 80.7% and 80.9% of hand hygiene citations from each systematic review. articles that were not available in nosobase® were mainly from specialized nursing journals (e.g., plastic surgical nursing, nursing times, critical care nursing quarterly, clinical nursing research, and the american journal of nursing) or regional and national journals not in its repertoire or with a lower scimago ranking (e.g., journal of the medical association of thailand, annals of the royal college of surgeons of england, medical journal of australia, scandinavian journal of infectious diseases, and life science journal acta zhenzhou university overseas edition). see scimago, https://www.scimagojr.com/.     table 1 number of citations (all or hand hygiene/hh), from the three systematic reviews, versus number of citations available in nosobase®   systematic review availability in  nosobase®   all citations hh citations all citations hh citations   number (n) n % n % n % review 1 120 106 88.3 79 65.8 75 70.8 review 2 88 57 64.8 55 62.5 46 80.7 review 3 89 47 52.8 48 53.9 38 80.9                   figure 1 number of citations (all or hand hygiene/hh citations), from the three systematic reviews, versus number of citations available in nosobase®.     the contact author for each systematic review confirmed that the literature search was carried out either by information professionals/librarians, or by researchers previously trained by them. the contact authors for reviews 1 and 3 did not confirm whether grey literature searching was done. the contact author for review 2 confirmed that a grey literature search was undertaken. of the three systematic reviews, all three searched pubmed/medline but none searched google scholar. it was interesting to note that across the three systematic reviews, 10 references were cited by two reviews. only two references were cited by all three systematic reviews.   step 2: scope and relevance of content in nosobase®   of the 200 documents indexed in nosobase® in 2015 with hand hygiene keywords, the majority were research articles. out of 200 items, 45 (22.5%) were published in french. the following french journals were represented: aide-soignante, hygiènes, inter-bloc, noso-info, pratiques psychologiques, risques et qualité en milieu de soins, soins aides-soignantes, soins, and national infection control bulletins. a majority of these journals are not indexed in commercial or other well-known databases.   the documents listed included four legislative texts from the public health council (hcsp), the french national organization for standardization (afnor), and the french republic official journal (journal officiel de la république française). also included were guidelines from the world health organization (who), the cochrane library, the national ipc learned society (sf2h), and the national institute of public health of quebec.    the documents reflected a wide target audience, including health professionals such as doctors, nurses, nursing assistants, surgeons, anesthetists, radiologists, pharmacists, ipc specialists, laboratory technicians, and general practitioners; as well as hospital managers, patients, hospital visitors, medical or nursing students, medical researchers, epidemiologists, the general public, and hospital engineers. corresponding healthcare structures included hospitals, nursing homes, community healthcare centres, and training institutes.   step 3: quality review of the documentation centre services   from the semi-structured discussions with the librarians, the documentation centre services were described and summarized in six categories according to strengths, weaknesses, and opportunities.   communication   regarding strengths in this area, there is a monthly newsletter nosoveille highlighting new arrivals and publications (e.g., legislation) and a trimester thematic publication nosotheme in a french professional ipc journal. also, the brand nosobase® and its documentation services have become well known to ipc professionals for over the past 20 years. however, its current weaknesses include irregular social media presence (e.g., in twitter, facebook, or youtube), despite already having some presence there. its brand is also less well known outside of france. there is therefore an opportunity for growth through increasing social media communication with better frequency in more channels (e.g., linkedin), and exploring communication channels outside of france.   accessibility   in terms of accessibility, there is free online access to the nosobase® database and web resources, with phone and email access to librarians for bibliographic aid and advice. however, the new database system (transitioned in march 2017) required current users to adapt to the new interface. there is also an urgent need for a search engine to access the website resources due to data growth. the opportunities for improvement involve facilitating access through user guides and facilitating website navigation and searching through an efficient search engine.   production   the strengths of nosobase® lie in its rich collection of french and multilingual documents (e.g., english, spanish, and german); diverse ipc related documentation (legislation, guidelines, and toolkits); articles in multiple formats (paper and electronic); and a new generation functional requirements for bibliographic records (frbr) database system. it also has a librarian curated specialized ipc thesaurus. however, there is a lack of standardized internal procedures, and it was significantly time-consuming to adapt and format the new database system. the thesaurus is not updated regularly, and it needs to evolve along mesh terms. there is therefore opportunity for growth through streamlining and standardizing work procedures, through exploiting the functions in the new database system, and through revising the ipc thesaurus to evolve with mesh terms. the development of a bilingual thesaurus would be helpful to bridge the content.   management   there are annual reviews of the documentation centre activities and great strength in a centralized purchasing department and budget, with centralized it services and staff training programs consolidating resources and expertise. however, this also resulted in reduced flexibility and an increase in the time needed for change. the quality assurance activities are sporadic and irregular. there is a need to encourage quality assurance processes to create value.   marketing   nosobase® presents multiple channels established over a significant period with professional learned societies and the french public health agency. it has been involved regularly in professional conferences and in the establishment of national guidelines, and has a strong presence on these websites and professional forums. however, there have been few user satisfaction surveys, and there is an insufficient use of modern user feedback channels, such as through social media. opportunities for growth involve analyzing modern user needs to better adapt services, and growing its online presence through european and international ipc channels (e.g., who and cdc). nosobase® could be marketed towards medical students and non-ipc healthcare professionals, as well as other francophone countries.   discussion   step 1 and step 2 of this study highlighted the potential of a specialized or subject-specific bibliographic database providing literature curated by trained librarians. nosobase® presented good coverage and availability of research articles on the theme of hand hygiene. it is practical and relevant for french users as it contains and regularly updates new french legislation and guidelines linked to ipc in healthcare, as well as international ipc guidelines. it carries good representation and scope of french language health journals in this field, a majority of which are not indexed internationally or are not on pubmed.   however, step 3 of this study underlines the need for more modern approaches in communication and marketing to encourage access by modern users. nosobase® needs to adapt its website and database access accordingly, with better brand presence on social media, and a more user-friendly approach. nosobase® has taken positive steps towards this by transitioning in 2017 to a new frbr database system, enabling documentation to be stored according to modern multiple formats such as electronic or multimedia. earlier studies have highlighted the need to evolve traditional librarian services toward point of care (lamb, jefferson, & white, 1975; reviewed by van kessell, 2012). nosobase® needs to examine such approaches in the future.   this study has limitations. the systematic reviews were selected based on a date cutoff with very simplified keywords. the search was limited to a sample size of three systematic reviews and a single theme of hand hygiene. qualitative discussions were limited to a select number of librarians who were closely involved with nosobase® documentation centres; information obtained from users of the database and services may have provided different perspectives. this is the first quality assurance review of the content of the bibliographic database nosobase®. a previous quality assurance review was based on a user satisfaction survey about the web resources in general (sanlaville, angibaud, girot, lebascle, & yvars, 2011). this study can thus be used as a foundation for future bibliographic content review. hand hygiene may be a theme that naturally interests a wider audience than expected. however, infection control and prevention is in general a multidisciplinary field and has a wide target audience. the next steps proposed by this study are to encourage nosobase® to include more nursing care journals, and to expand this assessment by searching and comparing with other systematic reviews on different ipc themes.       in 2015, a french language based portal, lissa, which stands for scientific literature on health, was created and financed by the national research agency (anr), acknowledging “the shared opinion of the national academy of medicine, that [it was necessary in france] to have a bibliographic database to improve the visibility, access and dynamism of medical and paramedical literature in french” (griffon, scheurs, & darmoni, 2016, p. 956). it is an encouraging development toward which nosobase® hopes to contribute.     conclusion   specialized or subject-specific bibliographic databases came into being due to the growth and proliferation of publications. however, they currently face increasing costs of maintenance, competition from free online access to multidisciplinary databases such as pubmed, and the development of online search engines such as google scholar. evidence based practices in librarianship also led to the development of methodologies to evaluate the relevance of documentation centres.   this study explored the relevance of the national ipc bibliographic database nosobase® and its documentation centre services. the results indicate its relevance, reflected by a good coverage and availability of citations from three systematic reviews based on the theme of hand hygiene, a wide scope of content based on hand hygiene related keywords, and an important listing of french language based publications and grey literature. the qualitative approach through semi-structured discussions with all the librarians in the various documentation centres provided a framework analysis of strengths, weaknesses, and opportunities of the documentation centre services. due to the ever-changing landscape of information services and access, documentation centres need to continuously measure the quality of their contribution, and base their practice on evidence. nosobase® has a rich heritage in france in the specialized multidisciplinary field of infection prevention and control. by adapting to modern user needs and improving communication and access, nosobase® will be able to contribute towards evidence based health practice and evidence based librarianship.      epilogue: due to restructuring of national infection control centres and related budgets, the funding of nosobase® has been dramatically reduced in 2019, and its activities will be taken up by a single different infection control centre, limiting updates mainly to french legislation and recommendations. its bibliographic database has been suspended. the database bdsp faces a similar fate.   references   aagaard, t., lund, h., & juhl, c. (2016). optimizing literature search in systematic reviews – are medline, embase and central enough for identifying effect studies within the area of musculoskeletal disorders? bmc medical research methodology 16, 161. https://doi.org/10.1186/s12874-016-0264-6   beyer, f. r. & wright, k. (2013). can we prioritize which databases to search? a case study using a systematic review of frozen shoulder management. health information and         libraries journal, 30(1), 49-58. https://doi.org/10.1111/hir.12009   boeker, m., vach, w., & motschall, e. (2013). google scholar as replacement for systematic literature searches: good relative recall and precision are not enough. bmc medical research methodology, 13, 131. https://doi.org/10.1186/1471-2288-13-131   booth, a. & brice, a. (2004). evidence-based practice for information professionals: a handbook. london, united kingdom: facet publishing.   bramer, w. m., giustini, d., kramer, b. m. r., & anderson, p. f. (2013). the comparative recall of google scholar versus pubmed in identical searches for biomedical systematic reviews: a review of searches used in systematic reviews. systematic reviews, 2(115). https://doi.org/10.1186/2046-4053-2-115   brettle, a. j. & long, a. f. (2001). comparison of bibliographic databases for information on the rehabilitation of people with severe mental illness. bulletin of the medical library association, 89(4), 353-362. retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/pmc57964/   brettle, a., maden-jenkins, m., anderson, l., mcnally, r., pratchett, t., tancock, j., thornton, d., & webb, a. (2011). evaluating clinical librarian services: a systematic review. health information and libraries journal, 28(1), 3-22. https://doi.org/10.1111/j.1471-1842.2010.00925.x   canadian association of research libraries (2010). core competencies for 21st century carl librarians. ottawa, on: author. retrieved from https://www.carl-abrc.ca/doc/core_comp_profile-e.pdf   centre for reviews and dissemination. (2009). systematic reviews: crd’s guidance for undertaking reviews in health care. york, united kingdom: university of york. retrieved from https://www.york.ac.uk/media/crd/systematic_reviews.pdf   dufour, j.-c., mancini, j., & fieschi, m. (2009). recherche de données factuelles [searching for evidence-based data]. journal de chirurgie 146(4), 355-367. https://doi.org/10.1016/j.jchir.2009.08.025   gasparyan, a. y., yessirkepov, m., voronov, a. a., trukhachev, v. i., kostyukova, e. i., gerasimov, a. n., & kitas, g. d. (2016). specialist bibliographic databases. journal of korean medical science, 31(5), 660-673. https://doi.org/10.3346/jkms.2016.31.5.660   gehanno, j-f., rollin, l., & darmoni, s. (2013). is the coverage of google scholar enough to be used alone for systematic reviews? bmc medical informatics and decision making, 13(7). https://doi.org/10.1186/1472-6947-13-7   glasziou, p. & aronson, j. k. (2017). a brief history of clinical evidence updates and bibliographic databases. jll bulletin: commentaries on the history of treatment evaluation. retrieved from http://www.jameslindlibrary.org/articles/brief-history-clinical-evidence-updates-bibliographic-databases/   goodman, k. w. (2002). ethics and evidence-based medicine: fallibility and responsibility in clinical science. cambridge, united kingdom: cambridge university press.   griffon, n., schuers, m., & darmoni, s. j. (2016). littérature scientifique en santé (lissa): une alternative à l’anglais? [lissa: an alternative in french to browse health scientific literature?]. la presse médicale, 45(11), 955-956. https://doi.org/10.1016/j.lpm.2016.11.001   guistini, d. & kamel boulous, m. n. (2013). google scholar is not enough to be used alone for systematic reviews. online journal of public health informatics, 5(2), 214. http://doi.org/10.5210/ojphi.v5i2.4623   haddaway, n. r., collins, a. m., coughlin, d., & kirk, s. (2015). the role of google scholar in evidence reviews and its applicability to grey literature searching. plos one 10(9), e0138237. https://doi.org/10.1371/journal.pone.0138237   jamali, h. r. & asadi, s. (2010). google and the scholar: the role of google in scientists' information‐seeking behaviour. online information review, 34(2), 282-294. https://doi.org/10.1108/14684521011036990   palisse, v. (2011). valoriser les produits documentaires: quelles méthodes, quel plan d’action? (mémoire). retrieved from https://memsic.ccsd.cnrs.fr/mem_00679869/document   rathbone, j., carter, m., hoffmann, t., & glasziou, p. (2016). a comparison of the performance of seven key bibliographic databases in identifying all relevant systematic reviews of interventions for hypertension. systematic reviews, 5(27). https://doi.org/10.1186/s13643-016-0197-5   rethlefsen, m. l., murad, m. h., & livingston, e. h. (2014). engaging medical librarians to improve the quality of review articles. jama 312(10), 999-1000. https://doi.org/10.1001/jama.2014.9263   rethlefsen, m. l., farrell, a. m., osterhaus trzasko, l. c., & brigham, t. j. (2015). librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. journal of clinical epidemiology, 68(6), 617-626. https://doi.org/10.1016/j.jclinepi.2014.11.025   sanlaville, n., angibaud, m., girot, i., lebascle, k., & yvars, s. (2011). résultats de l’enquête de satisfaction nosobase. unpublished report.   savey, a., sanlaville, n., & fabry, j. (2000). de la documentation à la communication: l’expérience de nosobase®. [from documentation to communication: experiences from nosobase®]. techniques hospitalières, 651, 39-42.   toneatti, v. (2008). valoriser un centre de documentation par une démarche qualité. (mémoire). retrieved from http://bdid-intd.cnam.fr/memoires/2008/toneatti.pdf   van kessel, k. (2012). gertrude lamb’s pioneering concept of the clinical medical librarian. evidence based library and information practice, 7(1), 125-128. https://doi.org/10.18438/b8ns5g   editorial evidence based library and information practice 2009, 4:2 143 evidence summary staffing an academic reference desk with librarians is not cost-effective a review of: ryan, susan m. “reference transactions analysis: the cost-effectiveness of staffing a traditional academic reference desk.” journal of academic librarianship 34.5 (2008): 389-99. reviewed by: cari merkley librarian, mount royal college calgary, alberta, canada e-mail: cmerkley@mtroyal.ca received: 01 march 2009 accepted: 13 april 2009 © 2009 merkley. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. objective – to determine whether it is cost effective to staff an academic reference desk with librarians through an examination of the types of reference questions being asked and the qualifications required to answer them. design – content analysis of reference transaction logs and activity-based costing for reference services based on quantitative data derived from the logs. setting – stetson university, a private institution in the united states with an fte of approximately 2500. subjects – 6959 phone, email, and in-person reference transactions logged at the reference desk by four full-time and two part-time librarians. methods – this study repurposes data originally collected to determine the frequency with which librarians turned to online versus print sources when responding to questions at the reference desk. librarians working at the stetson university library reference desk recorded all reference queries received in person, by phone, or by email for a total of eight months between 2002 and 2006. data collection took place in two month intervals in fall 2002, spring 2003, spring 2006, and fall 2006. each question and the sources used to address it were logged by the librarian. directional questions that were not related to the library’s collections and technical questions dealing with printer or copier mechanical problems were counted, but the specifics of these questions were not recorded. it was felt that these queries would not yield data relevant to the original research question on sources used as they “did not directly relate to an information need” (391). a total of 6959 questions were logged by librarians during the four collection periods. questions were recorded for only 4431 transactions; the remaining 2528 queries related to printer/copier problems or nonlibrary specific directions and were described as “direction and machine: nonmailto:cmerkley@mtroyal.ca evidence based library and information practice 2009, 4:2 144 informational” (394). the 4431 recorded questions were then divided into four categories derived by the researcher: lookup (a search for a known item), directional (library-specific orientation to the space and collections), technology (assistance with using library technology and electronic resources), and reference. the category of reference was further subdivided into eight additional categories: catalogue search, citation help, database help, “guide to correct databases,” “personal knowledge or referral,” “quick internet search,” research, and serials solutions (392). “guide to correct databases” referred to advice on the appropriate database to answer a question and serials solutions included questions that could be answered using the serials solutions product, such as the availability of a particular journal or article in the collection (392). questions were assigned to the single most appropriate category by the researcher. question categories were then mapped to “suggested staffing levels” (396). this determination was made by the researcher, and no details were given as to how the decision was made for each category. the three levels of staffing discussed were librarian, “trained student or staff,” and “well-trained staff/occasional librarian referral” (396). the cost of staffing the reference desk during the eight months captured in this study was calculated by multiplying the hours worked by each librarian by his/her individual average rate of pay across the four data collection periods. indirect staff costs such as benefits were not included in this calculation. the average cost per reference transaction was determined by dividing the total salary costs by the total number of reference queries during the periods of study. costs for those categories of questions best addressed by a librarian could then be determined. the actual number of librarians who participated in the study is unclear. the methodology refers to four full-time and two part-time librarians (391). however, later in the article there is reference to five full-time and three part-time librarians rather than the numbers initially stated (396). this may reflect staffing changes during the study period, with the first set of numbers referring to positions rather than individuals, but this cannot be verified with the evidence presented in the article. main results – it was determined that most questions asked at the reference desk during the study period could have been addressed by trained student and staff member rather than librarians. only 11% (784) of questions logged were deemed sufficiently complex by the researcher to require the attention of a librarian. the remaining 6175 transactions (89% of all those logged) could most likely be handled by a different staffing complement. according to ryan, approximately 74% of the reference transactions, including directional, technology, “quick internet,” and known item searching questions could have been answered by “trained student and staff” (396). questions on catalogue searching, databases, citations, serial solutions, and personal knowledge/referrals, representing approximately 15% of all questions, could have been handled by experienced and knowledgeable staff with limited librarian intervention. the complexity of the question was in part judged by the number of sources required to answer it, with most (75%) answerable with just one source. the total cost of staffing the reference desk with librarians for the eight months studied was approximately us$49,328.00. a total of 6959 questions were logged during this period, resulting in an average cost of us$7.09 per reference transaction. this cost is approximate, as the exact time spent on each question was not recorded. the cost of evidence based library and information practice 2009, 4:2 145 answering “non-informational” directional and technical questions was the most significant (396). this category represented 36.3% of all questions received at the reference desk, with a total staffing cost of $17, 919.41 ($7.09 x 2528). “informationorientated” directional and technology questions followed at 15.4% (us$7,620) and 12.4% (us$6,110.18) respectively (396). according to ryan, questions in all three categories could be addressed by students and staff. the cost of addressing research questions, the only category requiring librarians, was us$5557.29. research transactions were greatly outnumbered by directional and technology related questions. an average of 3.6 research questions were asked at the reference desk during the 12 hours it was open each day, compared to 20.8 directional/technical questions. conclusion – the nature of questions logged at the stetson university library reference desk suggests that it is inefficient to staff the desk with librarians, given the salary costs of such a staffing model and the fact that librarian’s skills may not be required to answer most of the questions posed. since the number of questions that need a librarian is so low, ryan suggests that alternative staffing and service models be considered, so the energies of librarians could be more effectively employed elsewhere in the organization in areas such as information literacy instruction and the development of enhanced web services. it is noted that any reorganization of reference services should be done in concert with user surveys, consultation with staff, and extensive training to prepare staff for new roles. suggested areas for further research identified by the researcher include the quality of reference transactions in an increasingly online environment. commentary this study provides a fascinating look at the types of questions encountered by staff at an academic reference desk. the large number of questions logged relating to computer or copier problems will not be surprising to those currently working in such an environment, but quantifying these queries perhaps lends more weight to shared anecdotal experiences. it would have been interesting to see if the incidence of such questions increased in the time between the first round of data collection in 2002-2003 and the follow-up collection periods in 2006, or if the types of questions and their frequencies remained stable. where ryan departs from previous discussions around what should be done with the reference desk in an increasingly digital environment is in her attempt to calculate staffing costs per transaction as evidence for the need to change an organization’s existing staffing model. what is not clear in the study, however, is how representative stetson’s practice of staffing a reference desk solely with librarians actually is. many academic libraries are already employing a mix of librarians, paraprofessionals and students in their delivery of reference services. banks and pracht recently surveyed a random sample of 191 midsize american academic libraries about their reference desk staffing model. sixty percent of the librarians’ who responded indicated that staff who do not hold a mlis were working at their reference desk (54). reasons given for the use of paraprofessionals and students to cover reference services in the study by banks and pracht include reduced staffing costs and freeing librarians for other activities (56). both of these arguments figure prominently in this article as it attempts to justify a staffing shift at an academic library reference desk by tying staffing costs to the evidence based library and information practice 2009, 4:2 146 types of questions library patrons are asking. the sheer volume of directional and technical questions logged at stetson not specific to the library is daunting, and the message is clear that a librarian performing mundane tasks like fixing printer jams costs the institution as much in salary costs as answering database search questions. however, the process of classification employed by the author accentuates the impression of reference staff functioning largely as tech support by parsing reference questions into ever finer subcategories while technology and directional questions remain largely unpacked. for example, the label of research is essentially a catch-all for questions that do not cleanly fit into the seven other categories available. at times, the number of subcategories seems excessive – for example, having a separate category for serials solutions isolates full text retrieval or known journal searching from database help or catalogue search when the serials solutions tool is linked with both resources. the distinction between database searching and research is also a fine one, as both require the development of complex search strategies. it is also not clear if a transaction that crosses several of these categories is automatically assigned to research or if each part of the transaction is isolated into separate categories. the isolation of research questions in this manner is particularly significant as this is the only category ryan assigns to librarians. ryan acknowledges that this process of categorization may be subjective, but greater discussion of the reasoning behind this home-grown classification scheme would have been welcome given the importance of these categories in later discussions of proposed staffing complements. linking a question’s complexity with the number of sources used to answer it also fails to account for the time, skill, and effort it can sometimes take a librarian to walk a user through a single source. when it comes to questions of cost, time seems to be an important variable that is missing in this discussion. while ryan could refer to the data to find out exactly what sources were used, she could only estimate the time involved with each transaction based on her own experiences. ryan’s use of data that was originally collected for another purpose (to see how often librarians were turning to electronic sources over print in answering reference questions) means that she is limited from the outset in the conclusions she can draw, as she has to build a methodology on someone else’s foundation. even more troubling is the lack of clarity around how decisions were made about the staffing needs assigned to each question category. ryan writes, “although it can be difficult and subjective to determine exactly which reference transactions require the skills of a librarian, more easily addressed are those questions that do not need a librarian” (395). in ryan’s estimation, most questions do not need a librarian, leading to the reader to at times question what the value of an mlis is in the first place if, as ryan suggests, “many librarians would argue that much of the skill set they use to answer reference questions was not learned in a graduate library program”(395). the difference between a “trained” and a “welltrained” staff member is also not clear, even though they are assigned to tasks of differing complexity (396). there is also no acknowledgement of the range of qualifications staff members could possess (e.g., college diplomas, undergraduate degrees, etc.) other than their lack of an mlis. the potential impact of the shift in staffing on the quality of reference transactions is not discussed, although research on the effectiveness of such staffing models exists in the literature. also not addressed in detail are the costs of evidence based library and information practice 2009, 4:2 147 hiring and training other staff to provide reference services. ryan does point out that these costs would need to be considered before any changes to the service model were made, but it is difficult to talk about cost effectiveness of librarians on the desk when the estimated costs of the alternative staffing models proposed are not provided for comparison. also, the article largely speaks of paraprofessionals at the reference desk as a way of freeing up librarian’s time for other duties, but this implies additional staff costs on top of existing librarian salaries to cover reference shifts. this is briefly addressed at the end of the article, but further emphasis on this point is required, particularly as the researcher also makes references to cost savings enjoyed by other institutions who replaced librarians at the reference desk. ryan rightly emphasises that individual libraries should assess their current reference models to see if new staffing complements or even new methods of service delivery would provide more value for their user communities. by placing a dollar value on each reference transaction, however, the activities of the reference desk are framed by default as an expense rather than an investment. the lack of data on the value placed on these services by users at the institution or the impact of these activities in terms of fostering relationships with users makes it difficult to determine the overall cost-effectiveness. ryan acknowledges the need for more research before drastic changes in service delivery are made, but the lack of context provided about how reference fits into stetson’s overall service model takes away from the need to consider these numbers as part of an overall assessment of reference. in particular, academic libraries should consider how reference services may or may not support the librarian’s teaching mandate within an academic institution, and how participating in such interactions potentially inform and enrich a librarian’s understanding of their users’ concerns in a way that remaining “behind the scenes” does not (398). works cited banks, julie, and carl pracht. “reference desk staffing trends: a survey.” reference & user services quarterly 48.1 (2008): 54-9. research article   what happened after the 2012 shift in canadian copyright law? an updated survey on how copyright is managed across canadian universities   rumi graham university copyright advisor & graduate studies librarian university of lethbridge library lethbridge, alberta, canada email: grahry@uleth.ca   christina winter copyright officer university of regina library regina, saskatchewan, canada email: christina.winter@uregina.ca   received: 3 jan. 2017       accepted: 9 june 2017          2017 graham and winter. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – the purpose of this study is to understand the practices and approaches followed by canadian universities in copyright education, permissions clearance, and policy development in light of major changes to canadian copyright law that occurred in mid-2012. the study also seeks to identify aspects of copyright management perceived by the universities to be challenging.   methods – in 2015, an invitation to complete an online survey on institutional copyright practices was sent to the senior administrator at member libraries of canada’s four regional academic library consortia. the invitation requested completion of the survey by the person best suited to respond on behalf of the institution. study methods were largely adapted from those used in a 2008 survey conducted by another researcher who targeted members of same library consortia.   results – while the university library maintained its leadership role in copyright matters across the institution, the majority of responding institutions had delegated responsibility for copyright to a position or office explicitly labeled copyright. in contrast, respondents to the 2008 survey most often held the position of senior library administrator. blanket licensing was an accepted approach to managing copyright across canadian universities in 2008, but by 2015 it had become a live issue, with roughly half of the respondents indicating their institutions had terminated or were planning to terminate their blanket license.   conclusion – in just seven years we have witnessed a significant increase in specialized attention paid to copyright on canadian university campuses and in the breadth of resources dedicated to helping the university community understand, comply with, and exercise various provisions under canadian copyright law, which include rights for creators and users.     introduction   the instrumental role of copyright in canada is to properly balance two competing ends: protection of creators’ private rights to stimulate the creation of new works, and wide dissemination of creative works to advance the public interest in learning, innovation and cultural enrichment (théberge v. galerie d’art, 2002, para. 30). observing that “canadian universities have not generally been proactive in managing copyright and knowledge transfer” (p. 7) while the complexity and contested nature of copyright’s balancing act intensify steadily, horava (2010) conducted a survey in the summer of 2008 to explore how academic libraries and their parent institutions view and manage communication about copyright.   at the time of the 2008 survey, a blanket reprographic licensing regime existed across canada’s publicly funded primary and secondary (k to 12) schools, colleges and institutes, and universities. the regime was formed over the decade following the 1988 copyright act amendments that expanded the scope of managing copyright collectively. one product of that round of statutory reforms was cancopy, a literary works collective now called access copyright (ac), which has operated since 1989 throughout canada except in the province of quebec (friedland, 2007). quebec’s literary works collective, copibec, was formally established in 1998 (soderstrom, 1998). educational institutions entered into blanket licensing primarily due to uncertainty regarding whether classroom copying can qualify as fair dealing under the copyright act (graham, 2016).     within two years of horava’s (2010) survey, discord was palpable in the post-secondary copyright realm as an initial attempt to renew another ac model blanket license agreement was unsuccessful. shortly after negotiations broke down, ac filed its first proposed tariff for post-secondary educational institutions in march 2010, which, to date, has yet to be certified (copyright board of canada, 2010). the decision of some institutions not to renew their ac license after the august 2010 expiry date marked the beginning of a movement away from blanket licensing. unrest was heightened by legislative and judicial proceedings in the last quarter of 2011. parliament embarked on yet another attempt to modernize the copyright act that had better prospects of success due to the majority government, and in an unprecedented two-day period the supreme court of canada heard a total of five copyright cases.   two pivotal events brought matters to a head in mid-2012. first, parliament passed an act to amend the copyright act (2012), which, among other things, expanded the “user’s right” (cch v. lsuc, 2004, para. 12, 48) of fair dealing to include education. second, the supreme court delivered its rulings in the five copyright cases (alberta (education) v. ac, 2012; esa v. socan, 2012; re:sound v. mptac, 2012; rogers v. socan, 2012; socan v. bell, 2012), which have since sparked much legal and academic debate, an example being the collection of essays entitled the copyright pentalogy: how the supreme court of canada shook the foundations of canadian copyright law (geist, 2013a). of critical importance to educators was the pentalogy case in which the court determined that teachers’ copying of short excerpts for classroom use can qualify as fair dealing if, on a properly conducted analysis, the dealings, on the whole, can be shown to be fair (alberta (education) v. ac, 2012).   the subsequent emergence of a “fair dealing consensus” among educators (geist, 2012) prompted many institutions to revise their copyright management approach, taking into account the fair dealing ruling in alberta (education) v. ac (2012) and expanded statutory fair dealing provisions (e.g., noel & snel, 2012; universities canada, 2012). k to 12 schools outside of quebec withdrew from blanket licensing in 2013 (e.g., geist, 2013b), but in 2015 the extent to which universities had followed suit was unclear. factors contributing to a climate of uncertainty were the copyright board’s post-secondary tariff proceedings, copyright lawsuits against two blanket licensing opt-out universities (access copyright, 2013; copibec, 2014), and, for institutions covered by a five-year ac blanket license that became available in spring 2012, the question of whether or not to renew before the license expired in december 2015.   since the authors are responsible for copyright at our respective institutions, we were interested in discovering how the recent major developments in copyright law have affected copyright practices and approaches at other universities. we learned that horava had no plans to update his 2008 survey but received his encouragement to pursue a similar investigation ourselves (personal communication, september 14, 2014). we therefore undertook this study to explore the current state of copyright education, permissions management, and copyright policy development at canadian universities as well as what has changed in these areas over the past five to seven years.    literature review   the two main issues examined by the 2008 survey were the locus of responsibility for copyright within respondents’ library and university, and challenges encountered in educating university community members about copyright (horava, 2010). almost 60% of respondents to the 2008 survey held the senior administrative role in their library while only four respondents (6%) were copyright officers. responsibility for copyright within respondents’ institutions was roughly equally often located in the library, in central administration, or shared by the library and another campus unit (each representing about 30% of all responses). the survey responses thus revealed a wide variety of institutional approaches to managing copyright and educating the university community.   among the challenges identified by the 2008 survey respondents was a lack of institutional coordination in copyright management and education (horava, 2010, p. 10). others were concerned about overlaps between copyright and various kinds of licensing, including blanket licensing and licensing of electronic resources. horava’s (2010) recommendation that library websites should “explain the university licence with copyright collectives” (p. 28) confirms blanket licensing was then the status quo. doubts about its necessity were nonetheless voiced by respondents in comments such as the following: “i suspect we are often licensing and paying for access that is available to us under fair dealing esp. since the cch case. i think an argument could be made that we no longer need part a of the access copyright licence” (horava, 2010, p. 21).   in 2008, few empirical studies were available on academic library perceptions and practices regarding copyright communication (horava, 2010). although they remain relatively scarce, a more recent investigation in this area is a multiple-case study by albitz (2013) on how research universities manage copyright education. using mintzberg’s organizational model as a theoretical lens, albitz (2013) conducted semi-structured interviews with 11 copyright officers at member institutions of the u.s.-based consortia on institutional cooperation. among the topics explored were the locus of copyright education and copyright officers’ responsibilities, credentials, and perceptions of authority. albitz found that it is most important for the copyright officer at research universities to hold a juris doctorate in intellectual property law, and that it is helpful but somewhat less important for the position to be located within the library rather than central administration.   applying a critical theory perspective, di valentino (2013) assessed understandings of copyright law as reflected in fair dealing policies adopted by canadian universities outside of quebec. this inquiry was guided by an interest in “reducing schools’ reliance on private contracts and in promoting awareness of fair dealing rights, and in reversing the trend of basing copyright compliance on the avoidance of liability, which prevents users from taking full advantage of their rights” (di valentino, 2013, pp. 14-15). the study’s examination of institutional copyright websites showed that while most universities had a fair dealing policy or set of guidelines, the presented copyright information was at times explained inconsistently, was inaccurate or unnecessarily restrictive, or was indicative of a strong tendency toward risk aversion.   in another investigation, di valentino (2015) extended horava’s (2010) study by “looking at the issue from the other side” (p. 5). faculty at canadian universities outside of quebec were surveyed on their understanding of institutional copyright policies and services and their practices regarding copyright compliance. di valentino’s (2015) findings established that faculty were broadly aware of institutional copyright policies, but 40% of respondents were unsure about whether copyright training was available. faculty appeared to be comfortable when using publicly accessible internet content in class but much less confident about the permissibility of making an electronic copy of excerpts for course use.   aims   the aim of our 2015 survey was to discover what has changed in the copyright practices and approaches of canadian universities since horava’s 2008 survey. three areas of central interest were:   1.       copyright education, including instructional methods and topics; 2.       copyright policy and the status of blanket licensing; and 3.       permissions management for copied course materials distributed via coursepacks (collections of readings and other course materials selected by instructors), print and electronic reserve (e-reserve), and the institutional learning management system (lms).   within these three areas we sought to identify the locus of responsibility; to find out what changes, if any, had occurred within the past five years; and to identify institutional copyright challenges.   methods   the methods and survey questions used in our study are chiefly adapted from those employed by horava (2010). both researchers obtained approval for the study’s protocols from the research ethics review office at our home institutions. because our investigation, like that of horava’s, was national in scope, the survey and communications to invited respondents were translated into french. our web-based survey was created using an instance of limesurvey hosted by the university of lethbridge library. two parallel versions of the survey were created, affording the option for participants to respond in either english or french. survey responses received in french were translated into english.    because the spotlight on copyright in canada had begun to intensify in 2010, questions about changes in practices asked respondents to reflect on the past five years. unlike horava’s study, our survey was completely anonymous to encourage wide participation. another point of divergence was our decision to focus on institutional approaches and practices rather than those belonging to the university library, since institutions may choose to situate responsibilities for copyright outside of the library. as well, given that di valentino (2013) had recently looked at information about copyright and fair dealing on canadian university websites, we excluded questions about copyright webpages.   we also chose to look at two areas not covered by the 2008 survey—permissions clearance and blanket licensing—as they are pertinent in this time of flux within national and international educational copying contexts (e.g., cambridge v. becker (2012); cambridge v. becker (2016); cambridge v. patton (2014)). a draft version of the survey was pre-tested by two library colleagues at canadian colleges. their feedback is reflected in the final version comprising 18 openand closed-ended questions (see the appendix), which is similar in length to horava’s (2010) 2008 survey containing 19 questions.   applying horava’s approach, an invitation to complete our survey was sent to the university librarian or library director at the member institutions of canada’s four regional academic library consortia: council of atlantic university libraries (caul), bureau de coopération interuniversitaire (bci), ontario council of university libraries (ocul), and council of prairie and pacific university libraries (coppul). recipients were asked to have the survey completed by the institutional staff best suited to do so. the 79 universities invited to participate in our survey is a slightly larger total than the 75 institutions invited to complete horava’s 2008 survey.   our 2015 survey opened for one month in early march 2015, with a reminder issued about one week prior to the closing date. since we desired a response rate comparable to that of the 2008 survey but the initial response rate was low, with the approval of our research ethics offices we re-opened the survey for another month in mid-october 2015. a reminder was sent about three weeks later.   results   respondents   our 2015 survey produced 48 responses: 22 were received in march-april 2015 and a further 26 followed in october-november 2015. the overall 61% response rate fell short of the 84% response rate obtained by horava (2010, p. 9), but represents a sizable improvement over the initial response rate of 28%. all geographic regions of canada were represented in responses to the 2015 survey: 38% were from eastern canada (caul) and quebec (bci), 25% were from ontario (ocul), and 37% were from western canada (coppul). response rates by consortium ranged from a low of 47% to a high of 78% (see table 1).  horava (2010) did not report the consortial distribution of 2008 survey responses.     table 1 survey respondents by consortium, 2015 member libraries 2015 respondents response rate caul 16 9 56% bci 19 9 47% ocul 21 12 57% coppul 23 18 78% totals/average 79 48 61%     figure 1 survey respondents by institutional size (fte), 2015 and 2008.     figure 2 survey respondents by position title, 2015 and 2008.       the 2008 and 2015 surveys both asked about the size of respondents’ institutions, based on full-time equivalent (fte) students. as seen in figure 1, respondents’ institutions in 2008 (horava, 2010, p. 11) and 2015 were proportionally similarly sized. for both surveys, almost half of the respondents were from small institutions, with the other half roughly equally split between medium and large institutions.   one difference between the results of the two surveys is a remarkable growth in the number of institutional positions specifically dedicated to copyright in 2015, as indicated in figure 2. in 2008, 59% of survey respondents held executive positions (university librarian or library director) and only 6% held copyright-specific positions (horava, 2010, p. 11). by 2015, 56% of respondents held copyright positions and only 27% held the senior library executive position. in both 2008 and 2015, about 10% of respondents held second-tier executive (associate university librarian) positions.   locus of responsibility   the 2015 survey questions on the position, department or office responsible for copyright education, policy and permissions were open-ended and did not ask respondents to specify the administrative locus associated with each answer. responses that referenced a copyright office or position within the library were coded under “library.” when two or more positions or campus units were mentioned, the response was coded under “[first-named unit] shared” (e.g., “library shared”). our survey results thus do not allow us to make a clear distinction between copyright offices managed by the library and copyright offices managed by other campus units, as only some responses happened to name the locus of administrative oversight associated with the identified responsible unit or position.     figure 3 responsibility for copyright education, 2015 and 2008.     responsibility for copyright education   in 2008 (horava, 2010, p. 13) and 2015, most respondents (between 50% and 60%) said the locus of institutional responsibility for copyright education resided with the university library acting either alone or with other campus units. figure 3 indicates the campus unit next most often identified as being responsible for copyright education in 2015 (27%) was the copyright office acting alone or in a shared capacity, but in 2008 it was central administration (29%). thus, between 2008 and 2015 some movement is discernable, as responsibility for copyright education formerly located in central administration appears to have been transferred to the copyright office.   responsibility for policy or services for owners of copyrighted materials   both surveys examined the locus of responsibility for matters pertaining to copyright owned by employees and students in works created in the course of employment or academic studies. the questions probing this issue were somewhat different, however. the 2008 survey asked whether a campus unit other than the library was responsible for managing copyright from a rights-holder’s perspective and if they answered “yes,” respondents were asked to specify the unit (horava, 2010, p. 35). the 2015 survey instead asked which campus unit was responsible for developing policies on ownership of copyrighted materials. despite differences in how they are framed, the gist of both questions is the identity of the campus unit responsible for helping authors and other creators understand and protect their copyright interests.     figure 4 responsibility for policy (2015) and service (2008) provisions for copyright owners.     figure 5 responsibility for policy on use of copyrighted works, 2015.         as shown in figure 4, more than 50% of responses to the 2008 survey indicated the library was responsible for helping owners protect their copyright interests, and a further 40% said this responsibility was held by central administration or the office of research (horava, 2010, p. 15). the 2015 responses to the somewhat different question about responsibility for copyright ownership policy indicate the responsible unit was most often central administration alone or in a joint capacity. but in 2015 when responsibility for policy on copyright owners’ rights was situated outside of central administration, the responsible units were roughly equally often the copyright office, the office of research, or the library, each acting alone or with other campus units. responsibility for policy on uses of copyrighted materials   the locus of responsibility for institutional policy relating to copyright compliance and use of copyrighted materials was a question explored only in the 2015 survey. figure 5 indicates that by far the campus unit most often holding this responsibility, alone or in a joint capacity, was the library, followed at a distance by the copyright office, which together account for two-thirds of responses. the extent to which central administration and copyright committees led user-focused policy development is relatively modest as they were identified as the responsible unit by under 20% and under 10% of respondents, respectively.  responsibility for permissions clearance   permissions clearance is the process of first assessing whether a work is protected by copyright and whether permission to use the work is needed, and then, when necessary, obtaining copyright owner consent. the supreme court decisions in cch v. lsuc, 2004 and alberta (education) v. ac, 2012 as well as the 2012 copyright act amendment that added education as a fair dealing purpose together provide educational institutions with a much-enriched understanding of the applicability of statutory user’s rights to educational uses of copyrighted works. institutional responsibility for clearing permissions for course-related use of copyrighted materials was a second issue explored only in the 2015 survey.        figure 6 responsibility for permissions clearance, 2015.     figure 6 shows that the library, acting alone or in a shared capacity, was across the board most often identified as being responsible for permissions clearance for materials distributed via the lms, e-reserve, print reserve, and coursepacks. the unit next most often responsible for permissions clearance was the copyright office, alone or shared, for all distribution modes except coursepacks where the bookstore or commercial copy shop was in second place. lms permissions clearance was the responsibility of the teaching faculty at seven institutions.    responsibility for blanket licensing   a third aspect of copyright responsibility considered only in the 2015 survey pertains to decisions on institutional blanket licensing. as presented in figure 7, 50% of respondents said this responsibility was the purview of central administration alone or in a shared capacity, but more than 30% indicated issues relating to blanket licensing were decided by the library acting alone or jointly with other campus offices. thus, most institutions considered blanket licensing to be a matter for either central administration or the university library. about 10% indicated blanket licensing matters to be the responsibility of the copyright office acting alone or with other offices, or a copyright committee.       figure 7 responsibility for blanket licensing decisions, 2015.     table 2 methods used to educate users of copyrighted works, 2015 and 2008 user education method frequency of response 2015 (n=45) frequency of response 2008 (n=62) individual assistance 33.3% 77.4% information literacy 68.9% 66.1% faculty liaison/outreach 28.9% 64.5% reference service -- 62.9% webpage 86.7% 62.9% printed information 17.8% 50.0% online tutorial 4.4% 19.3% other 24.4% 11.2% none 2.2% 4.8%       copyright education   education for users of copyrighted works   although the 2008 and 2015 surveys both asked about methods used to educate users of copyrighted materials, the question was posed in different ways. the 2008 survey presented a list of education methods and asked respondents to check off all that applied, whereas the 2015 survey question was open-ended, resulting in an inability to make exact comparisons between the two sets of responses. nevertheless, in most cases the categories of education methods arising in 2015 responses were mappable to those used in the 2008 survey (horava, 2010, p. 22), presented in table 2. in 2008, users most often received education via individual assistance, but in 2015 responses the most frequently mentioned education method was webpages. information literacy was the second most frequently used education method in 2008 and 2015. in 2015, no respondents mentioned reference service as an education method, and unlike 2008 respondents, very few said print materials were used. “other” user education methods noted in 2015 were e-mail, guidelines or policy statements, and mailings or newsletters. very few 2015 respondents—about 2%—said no copyright user education was offered.   education for creators of copyrighted works   an open-ended question in the 2015 survey asked about methods used to educate creators about their copyrights.  for ease of comparing user education methods to those used in creator education, table 3 summarizes the latter under the same categories used in table 2, which were borrowed from horava (2010). responses to the 2015 survey indicate the most frequently identified method of providing copyright information to both creators and users of copyrighted works was webpages. “other” means of providing copyright education to creators mentioned by 2015 survey respondents were collective agreements, e-mail, guidelines or policy statements, mailings or newsletters, and university committees. more than 13% of responses indicated copyright education for creators was not offered. copyright education topics   the topics addressed in education directed at users and creators of copyrighted works are another aspect of copyright education considered only in the 2015 survey. table 4 reveals the two most frequently identified copyright user education topics to be copyright infringement exceptions (users’ rights), with strong emphasis placed on the fair dealing provisions of the canadian copyright act. as evidenced in table 5, the three most frequently mentioned topics of education directed at creators of copyrighted works were creators’ or owners’ copyrights, negotiating publisher contracts or addenda, and open access. four topics are common to tables 4 and 5, indicating their relevance to both users and creators of copyrighted works: fair dealing, copyright basics (key provisions of the copyright act), copyright permissions or licensing, and open access.   table 3 methods used to educate creators of copyrighted works, 2015 education methods for creators frequency of response (n=45) individual assistance 24.4% information literacy 37.8% faculty liaison/outreach 22.2% webpage 64.4% printed information 6.7% other 22.2% none 13.3%     table 4 topics addressed in education for users of copyrighted works, 2015 education topics frequency of response (n=45) fair dealing 66.7% exceptions to infringement 28.9% copyright basics 22.2% copyright permissions or licensing 22.2% copyright compliance and ethical use of protected works 20.0% open access 20.0% course-related copying 17.8% images 13.3% lms use 13.3% digital or multimedia works 11.1% how to obtain help with copyright issues 11.1% coursepacks 6.7% motion pictures and videos 6.7% library-licensed resources 6.7% reserve or e-reserve 6.7% materials accessed or streamed from the internet 6.7% attribution or source citation 4.4% public domain 4.4% blanket licensing 2.2% not applicable or in development 4.4%       changes in copyright education   most respondents (77%) said their institution’s approach to copyright education has changed appreciably over the past five years. table 6 summarizes several broad themes identified in respondents’ brief explanations of what has changed.   copyright policy   policy scope and content   the 2008 survey asked whether university copyright policy guided, or was guided by, the library’s provision of copyright information and vice versa (horava, 2010, p. 35). rather than looking at the extent to which university copyright policy and library-provided copyright information were influenced by the other, the 2015 survey sought to determine the prevalence of institutional copyright guidelines or policies as well as the issues they address.   of the 48 respondents to the 2015 survey, 81% confirmed the existence of institutional guidelines or policy pertaining to copyright. just under 70% of respondents who shared the topics of institutional policy mentioned fair dealing, and 41% said policy covered copyright basics. the specificity of the subject matter of institutional copyright policies appears to vary widely as respondents identified topics ranging from the narrow issue of defining the meaning of “short excerpt” to the broad matter of the public domain, as evidenced in table 7.     table 5 topics addressed in education for creators of copyrighted works, 2015 education topics frequency of response (n=43) creators' or owners' copyrights 53.5% negotiating publisher contracts or addenda 37.2% open access 37.2% copyright basics 18.6% creative commons licensing 11.6% fair dealing 11.6% copyright permissions or licensing 11.6% publishing protocols, models, avenues 9.3% predatory publishing 9.3% self-archiving 7.0% granting agency policies 4.7% author-side publication charges (apcs) 2.3% theses and dissertations 2.3% moral rights 2.3% faculty collective agreement 2.3% waiving or sharing copyrights 2.3% not applicable or in development 14.0%     table 6 aspects of copyright education that have changed, 2015 broad themes frequency of response (n=40) education programs launched or intensified 65.0% new copying environment due to terminated blanket license 35.0% education programs moved to copyright office 30.0% new or revised help pages and guidelines 30.0% new case law and statutory amendments 20.0% new administrative structures or processes 10.0%     table 7 topics addressed by institutional copyright guidelines or policy, 2015 (n=39) policy topics frequency of response fair dealing 69.2% copyright basics 41.0% course-related copying 15.4% public performances 5.1% staff policy regarding copyright compliance 5.1% copyright office 2.6% definition of short excerpt 2.6% exhibition rights 2.6% use of images 2.6% moral rights 2.6% copyright permissions 2.6% ownership of works produced by university employees 2.6% public domain 2.6%     table 8 copyright policy year of establishment and last revision, 2015 time period policy established frequency of response (n=48) policy last revised frequency of response (n=48) before 1997 8.3% - between 1997 and 2010 8.3% 4.2% 2011 and after 54.2% 39.6% not applicable/no response 29.2% 56.3%       policy establishment and revision   the 2015 survey asked respondents to identify the date on which their institution’s copyright policy was established as well as the date on which the policy was last revised. table 8 indicates more than half of the institutions (54.2%) had established copyright policies in 2011 or later, which likely accounts for most of the slightly greater proportion of institutions (56.3%) that did not provide a policy revision date.   about 63% of survey respondents were unable to identify the main areas of change in the most recent copyright policy revisions or did not respond to this question. areas of policy revisions mentioned by four to six respondents were copyright act amendments, fair dealing, outcomes of copyright court cases, and educational exceptions to infringement. other policy revision areas identified by one or two respondents were library licenses, blanket licensing, digital copies, a shift to individualresponsibility for copyright compliance, and copyright and teaching.    institutions in 2015 most often used the university’s copyright website to communicate copyright policy to their communities, as shown in table 9. the next most frequently mentioned means of communicating institutional policy on copyright were e-mail and meetings. participation in blanket licensing   although publicly funded canadian post-secondary institutions were blanket licensees from the 1990s to at least 2010, by 2015 some had announced their withdrawal from the blanket licensing regime (katz, 2013). about 44% of respondents to the 2015 survey said their institution had terminated their blanket license and 4% did not answer the question about blanket licensing, leaving just over half of respondents, 52%, whose institutions remained blanket licensees. but an even greater proportion, 62%, said their institution had opted out of blanket licensing in the past.  furthermore, five respondents (10%) at institutions holding an ac blanket license said plans to exit the license were underway.     table 9 methods of communicating copyright policy to university community members, 2015 communication method frequency of response (n=48) copyright website 60.4% e-mail 29.2% meetings 29.2% university news 16.7% workshops 14.6% personal communication by staff specialists 8.3% administrative memos 6.3% newsletters 6.3% posters 4.2% click-through agreement on lms 2.1% checklists 2.1% not applicable/no response 22.9%     table 10 consideration of library licenses as permission sources for course readings, 2015 (n=48) in-house coursepacks copy shop coursepacks lms readings print reserve readings e-reserve readings yes 66.7% 12.5% 52.1% 70.8% 58.3% no 8.3% 18.8% 12.5% 10.4% 6.3% uncertain 10.4% 18.8% 31.3% 10.4% 4.2% not applicable 10.4% 45.8% 4.2% 4.2% 27.1% no response 4.2% 4.2% 0.0% 4.2% 4.2% 100.0% 100.0% 100.0% 100.0% 100.0%     table 11 format of copyright permission tools, 2015 tool format frequency of response (n=23) permissions clearance services and education 47.8% copyright management software 30.4% guide for copyright and permission decisions 21.7% copyright clearance form for instructors 13.0% look-up tool for permitted uses of licensed content 8.7% model permission clearance letters 8.7% tool offered by a copyright collective 8.7% website information 8.7%     table 12 institutional copyright challenges, 2015 challenge area challenge themes frequency of response education communicating copyright information effectively and comprehensively 76.7% (n=43) ensuring copyright/licensing compliance 30.2% overcoming obstacles to compliant practices 18.6% addressing staffing and staff expertise requirements 9.3% dealing with legal and statutory interpretation uncertainties 7.0% evaluating a possible move away from blanket licensing 4.7% helping faculty and students understand their copyrights and publication choices 4.7% policy fostering policy understanding and compliance 57.9% (n=38) applying policies appropriately 21.1% establishing or updating institutional policy 15.8% monitoring copyright and licensing compliance 7.9% achieving appropriate staffing for policy-related education and services 7.9% addressing specific policy-related issues 7.9% permissions managing administrative challenges of permissions clearance service 42.9% (n=35) helping users understand why permissions are important and how to assess them 34.3% acquiring permissions for specific kinds of works 31.4% securing administrative support for permissions staffing, systems or tools 22.9% acquiring permissions generally 5.7%     copyright permissions   applicability of library licenses   the first of two 2015 survey questions on copyright permissions asked if the applicability of library licenses is assessed when permissions are cleared for course readings distributed via coursepacks, the lms, print reserve, or e-reserve. as indicated in table 10, between 52% and 71% of respondents said that library licensing is taken into account when permissions are cleared for readings distributed in all modes except coursepacks produced by commercial copy shops. the greatest degree of uncertainty about whether library licenses are considered in the permissions clearance process pertained to readings distributed via the lms. permission tools   in response to the second question about permissions, 52% of respondents said one or more tools had been developed to help university community members clear copyright permissions, 44% indicated no tools had been developed, and 4% said the question was inapplicable. table 11 summarizes the formats of permission tools developed by universities, as described by respondents. educational, policy, and permissions challenges   in each of the three key areas probed by the 2015 survey, respondents were asked to identify the most significant challenges faced by their institutions. single dominant concerns surfaced within the areas of copyright education and copyright policy. for copyright education, the institutional challenge mentioned in 77% of responses was effective and comprehensive communication of copyright information. for copyright policy, fostering understanding of and compliance with institutional policy was the challenge noted in 58% of responses. no single concern was dominant in the area of permissions challenges, but managing administrative aspects of permissions service and helping users understand and perform permissions clearance were identified in 43% and 34% of responses, respectively. table 12 outlines themes that arose within the challenges identified in all three areas. the following are examples of comments submitted by respondents in the three areas of copyright challenges faced by universities. some of the mentioned challenges pertained to more than one challenge area:   education:   ·         “staffing for an intensive educational effort. entrenched practices of some faculty and staff members.” ·         “meet with lecturers, make existing class notes compliant, ensure that teaching staff comply with institutional policy on fair use.” ·         “reaching everyone. multi-campus environment.” ·         “creating buy-in from faculties and departments, who may simply view copyright clearance and related steps as hindrances or obstacles, rather than as a fundamental component of post-secondary education.”   policy:   ·         “connecting the institutional policy to specific compliance practices/procedures.” ·         “re-writing the fair dealing guidelines to make them more user-friendly, less daunting, shorter while still being useful.” ·         “ensuring compliance with fd policy and identifying individuals who will need assistance transitioning from working under the ac license to the fd policy.” ·         “we do not have a policy or guideline on converting physical av media to formats that allow our institution to stream video content to distance students or web-based courses.”   permissions:   ·         “permission for french documents (especially european, costly and long delays).” ·         “our permissions process is quite labour intensive. no database currently in place to support full workflow process.” ·         “materials in copyright but orphaned. dealing with copyright with regards to music, lyrics, recordings etc.” ·         “reapplying for permissions – keeping track of continuing use and when permissions expire. ensuring instructors and staff are aware of which permissions need reapplication. getting publishers to reply to requests in a timely manner.”   discussion   responsibility for copyright   results of this study evidence several areas of marked change in the copyright practices and approaches of canadian universities since 2008. while the library continued to play a prominent role in copyright education from 2008 to 2015, a shift in the locus of responsibility from central administration to the copyright office is notable. our survey did not reveal reasons for this change, but a possible inference is that it was precipitated by foundational shifts in the copyright landscape that heightened concern about copyright issues across canadian universities and a perceived need for specialized copyright expertise.   a reverse shift took place in the area of copyright ownership policy or advocacy. while the library was most often responsible for promoting and protecting rights-holders’ interests in 2008, the largest proportion of 2015 respondents—about one third—said responsibility for owner-focused policy was most frequently held, alone or jointly, by central administration. at the same time, the proportion of 2015 respondents who said this responsibility belonged to the library or copyright office, alone or with other units—one quarter—was not far behind.    the 2015 survey results show the university library most often played the lead role in permissions clearance for course materials made available to students via four common distribution modes. the unit next most frequently responsible for permissions clearance was once again the copyright office, acting alone or with others, for all distribution modes except coursepacks.    overall, the 2008 and 2015 survey results indicate the library is the primary locus for most matters related to copyright. this attestation of the library’s continued copyright leadership role within canadian universities notwithstanding, in several cases the copyright office served as the institutional lead unit in copyright matters. moreover, the position held by the majority of survey respondents in 2015 specialized in copyright, whereas in 2008 most respondents held the position of university librarian or library director.   educational approaches and topics   as more than three quarters of 2015 survey respondents said their institution’s approach to copyright education had changed, methods used to educate university staff and students about copyright clearly evolved between 2008 and 2015. compared to 2008, copyright user education in 2015 far less often involved individual assistance, faculty outreach, reference service, and print materials, but had become more heavily dependent on copyright webpages. on the other hand, reliance on information literacy remained strong, as about two-thirds of respondents in both 2008 and 2015 said their institution used this approach in copyright user education.   copyright education for creators was not explicitly investigated in the 2008 survey, but in 2015, the most frequently used method of educating creators was making copyright information available via webpages, with information literacy in distant second place. it may be the case that more attention is paid to educating copyright users than copyright creators: while the proportion of respondents who provided no information about their institution’s user education methods was very small, it was six times greater for creator education.   only the 2015 survey explored the topics addressed in copyright education. fair dealing was clearly a central concern, as it was identified as a focus of copyright user education by more than two-thirds of respondents. the next most frequently addressed topic in user-focused education, exceptions to infringement in general, was identified by roughly one-third of respondents. fair dealing was also addressed in copyright education for creators, but not as frequently as creators’ copyrights, negotiating publishing agreements, and open access.   the vast majority of 2015 survey participants’ institutions provided copyright education that was directed most often at copyright users and was somewhat less frequently tailored to copyright creators. tables 4 and 5 indicate fair dealing, copyright basics, copyright permissions or licensing, and open access were addressed in education directed at both groups. this suggests copyright educators recognize the importance of ensuring that their institutional communities understand the fundamental interdependence of provisions for users’ rights and creators’ rights in the copyright act, given that everyone is both a user and creator of copyrighted material.   fair dealing and open access promote the public interest in lawful, wide public use of protected intellectual works. at the same, they also assist creators who wish to build upon prior ideas and knowledge or disseminate their works broadly. similarly, rights granted to copyright owners under the act, including the right to authorize uses of their works, provide a means for creators to control certain uses of their works. when users wish to use protected works in ways not otherwise covered, knowing how to seek permissions properly will help users protect themselves against unintentional infringement.   the broad array of topics addressed in copyright education and the fact that respondents said the greatest change in this area was new or intensified educational programs evidence a serious commitment by canadian universities to enhance awareness and understanding throughout their communities of users’ and creators’ rights under the copyright act.   policy prevalence and focal points   we assume 100% of institutions represented in the 2008 survey results held a blanket copying license, whereas only 52% of 2015 survey respondents said their university was a blanket licensee. this finding aligns with those of di valentino (2013) who reported slightly over half of the institutions in her sample (41 universities outside of quebec) had an ac blanket license. the premise that almost all educational copying requires permission was questioned by a few of horava’s 2008 respondents, but by 2015, more than half of the universities responding to our survey had parted ways with blanket licensing or had definite plans to exit their license in the near future. blanket licensing as a policy approach to copyright compliance is thus a live issue currently trending toward reliance on alternative approaches.   new developments have also unfolded in other aspects of copyright policy. well above three quarters of the 2015 survey respondents said their university had instituted a copyright policy or guidelines. it is notable that more than half of responding institutions initiated copyright policy for the first time in 2011 or later. likely related to this finding is the fact that several respondents said their institution adopted the 2012 revision of the fair dealing policy that was developed and recommended by the association of universities and colleges canada (now universities canada). as well, having a copyright policy in place was possibly a particular concern for institutions who were, or were moving toward, operating without a blanket license. of the topics covered in copyright policy, the one identified in more than two-thirds of responses was fair dealing. the next most common focal point of policy—copyright basics—was mentioned in less than half of responses.    taken together, these developments suggest institutional approaches to copyright policy in 2015 had evolved substantially since 2008. the proportion of universities opting out of blanket licensing was nearing 50% and adoption of institutional copyright policy was prevalent, with a primary focus on the user’s right of fair dealing.   permissions clearance practices   responses to the 2015 survey indicate that library licenses for full-text resources are assessed by most universities during permissions clearance for all modes of distributing course readings except commercially produced coursepacks. all the same, roughly one-third of survey respondents did not know whether clearing permissions for materials distributed via the lms took into account the potential applicability of library licenses. reasons for this relatively high level of uncertainty level are unclear, but one respondent’s comment raises an issue that may be applicable in settings where instructors are responsible for lms permissions clearance: “i have put ‘uncertain’ . . . because we do provide the information and tool so that teaching faculty can check if use is covered by our licenses but we do not have data as to what level it is used.”   more than half of the responding institutions said they had developed tools to help institutional community members clear copyright permissions. the form of those tools was most often permissions clearance services or education, or copyright management software. these findings point to considerable investment of various kinds of resources to enhance permissions assessment and management at canadian universities.   challenges   the challenges in copyright education identified by 2008 and 2015 survey respondents, on the whole, touch on similar themes, the most common in 2015 being effective and comprehensive communication of copyright information to all university community constituencies. this challenge is likely to remain an important and large undertaking because copyright encompasses inherently complex concepts and requirements that are difficult to reduce to simple, memorable ideas while remaining true to how the copyright system actually works.   in 2015, the most frequently identified challenges in the policy and permissions arenas were fostering copyright policy understanding and compliance, and managing administrative aspects of permissions, respectively. the former essentially covers much the same ground as the most frequently identified challenge in the area of copyright education:  communicating copyright information effectively and comprehensively. respondents’ comments indicate administrative aspects of permissions challenges involve issues such as significant delays in securing permissions in a timely fashion and dealing with large volumes of uses needing permissions clearance without adequate staffing levels.   the challenge themes arising from the 2015 survey collectively suggest universities appreciate the importance of ensuring that their communities and operations are properly guided by the provisions of current copyright law and licensing agreements. interestingly, the lack of institutional coordination of copyright matters regarded by many 2008 survey respondents as a major organizational challenge (horava, 2010, p. 27) did not arise as a strongly articulated concern in 2015.   limitations   the overall goal of the 2015 survey was to gain a well-rounded picture of current institutional copyright practices and aspects that may have changed since 2008. a limitation of our study is that its response rate, while strong, was markedly lower than the rate achieved in the 2008 survey. reasons for the lower response rate are unclear, although uncertainty regarding the outcomes of legal and tariff proceedings are likely contributing factors.   another limitation is that our survey necessarily yields only a snapshot of copyright practices and approaches at a time when many canadian universities held a blanket ac license with a december 2015 expiry date. since our study was conducted just prior to this deadline, in several cases the current status of blanket licensing and other aspects of institutional copyright policy and practices may differ from the responses we received to licensing questions in 2015.   a further limitation is the omission of survey questions that might have shed more light on the administrative relationship between the copyright office and the library as well as the specific nature and scope of positions holding responsibility for various aspects of copyright. we decided against their inclusion to contain the length of the survey in order to encourage wide participation.   future research   our study uncovered considerable growth in the number of copyright-specific positions since 2008. because our survey did not ask whether these were new hires or the result of reallocation of existing personnel to a dedicated copyright position, however, this issue is worthy of investigation in a future study. we note that horava (2010) also identified the role of copyright officers as an area for further research. some ground work has already been laid by patterson (2016) in an examination of the role of canadian copyright specialists in universities. additional areas warranting further inquiry include the copyright practices and approaches and role of copyright specialists in other types of educational institutions, such as k to 12 schools, community colleges, and polytechnical institutes.    conclusion   framed as an update to horava’s 2008 survey, our study explored the extent to which canadian universities have responded to several major developments in canadian copyright law by adjusting their copyright practices and approaches. horava (2010) noted new workflows and positions were developed over the 2000s to support a new priority on acquiring licensed digital content that occurred with little attention paid to copyright or the volume of information transfer. in contrast, our study results indicate a much-elevated level of copyright awareness is prevalent at canadian universities, as evidenced by the topics covered in current copyright education and copyright policy and by the variety of tools and resources available to help users clear copyright permissions for teaching and research purposes.   a variety of approaches still exist, but shared practices may now be more common, as several respondents noted their institutional copyright policy was modeled on the 2012 revised universities canada fair dealing policy and di valentino (2013) found this same policy had been “adopted in some way” (p. 17) by many universities in her study. we suggest institutions sharing similar copyright policies are also likely to share broadly aligned copyright practices. a specific change in approach is reflected in the far greater prevalence of copyright-specific positions and offices in 2015. university libraries nevertheless maintain a lead role in the areas of copyright education, copyright use-focused policy, and permissions clearance, with copyright offices often having the distinction of being next most likely to hold the lead role.   universities appear to be paying greater and more nuanced attention to copyright policy and copyright education from users’ and creators’ viewpoints. given that fair dealing was by far the most frequently identified focus of institutional copyright policy and copyright education for users, it is apparent that provisions for user’s rights in the copyright act have achieved a heightened prominence and importance on canadian university campuses.      despite substantive challenges that remain in the copyright realm, over the past several years canadian universities have evidently augmented the attention and resources dedicated to ensuring their communities, on the one hand, understand and comply with canadian copyright law, and on the other hand, are aware of and fully exercise the copyright act’s provisions for user’s rights such as fair dealing.   acknowledgements   this study was partially funded by a 2015 research grant from the canadian library association. the authors thank tony horava for permission to adapt the survey instrument used in his 2008 study.   references   access copyright. (2013). canada's writers and publishers take a stand against damaging interpretations of fair dealing by the education sector [press release]. retrieved from http://accesscopyright.ca/media/35670/2013-04-08_ac_statement.pdf   an act to amend the copyright act (2012, s.c., ch. 20). retrieved from http://www.parl.gc.ca/housepublications/publication.aspx?language=e&mode=1&docid=5697419   alberta (education) v. canadian copyright licensing agency (access copyright), 2012 scc 37 supreme court of canada, (2012). retrieved from https://scc-csc.lexum.com/scc-csc/scc-csc/en/item/9997/index.do   albitz, r. s. (2013). copyright information management and the university library: staffing, organizational placement and authority. journal of academic librarianship, 39(5), 429-435. doi:10.1016/j.acalib.2013.04.002   cambridge university press v. becker, 863 f. supp. 2d 1190 (n.d. ga., 2012). retrieved from http://scholarworks.gsu.edu/cgi/viewcontent.cgi?article=1004&context=univ_lib_copyrightlawsuit   cambridge university press v. becker, 863 f.supp.2d 1190 (n.d. ga., 2016). retrieved from http://policynotes.arl.org/wp-content/uploads/2016/03/dkt-no.-510-order-dated-2016_03_31.pdf   cambridge university press v. patton, 769 f.3d 1232 (11th cir. 2014). retrieved from http://media.ca11.uscourts.gov/opinions/pub/files/201214676.pdf   cch canadian ltd. v. law society of upper canada, 2004 scc 13 supreme court of canada, (2004). retrieved from https://scc-csc.lexum.com/scc-csc/scc-csc/en/item/2125/index.do   copibec. (2014). $4 million class action lawsuit against université laval for copyright infringement [press release]. retrieved from http://www.copibec.qc.ca/portals/0/fichiers_pdf_anglais/news%20release-copibec%20novembre%2010%202014.pdf   copyright board of canada. (2010). statement of proposed royalties to be collected by access copyright for the reprographic reproduction, in canada, of works in its repertoire: post-secondary educational institutions (2011-2013).  ottawa, on: copyright board of canada. retrieved from http://www.cb-cda.gc.ca/tariffs-tarifs/proposed-proposes/2010/2009-06-11-1.pdf.   di valentino, l. (2013, may 9). review of canadian university fair dealing policies. fims working papers.  retrieved from http://ir.lib.uwo.ca/fimswp/2   di valentino, l. (2015). awareness and perception of copyright among teaching faculty at canadian universities. partnership: the canadian journal of library and information practice and research, 10(2), 1-16. doi:10.21083/partnership.v10i2.3556   entertainment software association v. society of composers, authors and music publishers of canada, 2012 scc 34 supreme court of canada, (2012). retrieved from https://scc-csc.lexum.com/scc-csc/scc-csc/en/item/9994/index.do   friedland, m. l. (2007). report to access copyright on distribution of royalties. toronto: access copyright.  retrieved from http://www.accesscopyright.ca/media/8359/access_copyright_report_--_february_15_2007.pdf   geist, m. (2012). fair dealing consensus emerges within canadian educational community. [blog post]. retrieved from http://www.michaelgeist.ca/content/view/6698/125/   geist, m. (2013a). the copyright pentalogy: how the supreme court of canada shook the foundations of canadian copyright law. ottawa: university of ottawa press.  retrieved from http://www.press.uottawa.ca/sites/default/files/9780776620848.pdf   geist, m. (2013b). ontario government emphasizes user rights in its copyright policy for education. [blog post]. retrieved from http://www.michaelgeist.ca/2013/07/ontario-govt-copyright-policy/   graham, r. (2016). an evidence-informed picture of course-related copying. college & research libraries, 77(3), 335-358. doi:10.5860/crl.77.3.335   horava, t. (2010). copyright communication in canadian academic libraries: a national survey. canadian journal of information and library science, 34(1), 1-38. doi:10.1353/ils.0.0002   katz, a. (2013). fair dealing’s halls of fame and shame, 2013 holiday edition. [blog post]. retrieved from http://arielkatz.org/archives/3064   noel, w., & snel, j. (2012). copyright matters! some key questions & answers (3rd ed.). [toronto, on]: council of ministers of education.  retrieved from http://cmec.ca/publications/lists/publications/attachments/291/copyright_matters.pdf   patterson, e. (2016). the university copyright specialist: a cross-canada selfie. paper presented at the abc copyright conference 2016, halifax, ns.  retrieved from http://abccopyright2016.com/wp-content/uploads/2016/07/abc-2016.pdf   re:sound v. motion picture theatre associations of canada, 2012 scc 38 supreme court of canada, (2012). retrieved from https://scc-csc.lexum.com/scc-csc/scc-csc/en/item/9999/index.do   rogers communications inc. v. society of composers, authors and music publishers of canada, 2012 scc 35 supreme court of canada, (2012). retrieved from https://scc-csc.lexum.com/scc-csc/scc-csc/en/item/9995/index.do   society of composers, authors and music publishers of canada v. bell canada, 2012 scc 36 supreme court of canada, (2012). retrieved from http://scc-csc.lexum.com/scc-csc/scc-csc/en/item/9996/index.do   soderstrom, m. (1998). new licencing agency in quebec. quill & quire, 64(3), 13.   théberge v. galerie d’art du petit champlain inc., 2002 scc 34 supreme court of canada, (2002). retrieved from https://scc-csc.lexum.com/scc-csc/scc-csc/en/item/1973/index.do   universities canada. (2012, october 9). fair dealing policy for universities.  retrieved from http://www.univcan.ca/media-room/media-releases/fair-dealing-policy-for-universities/ evidence summary   more dois are accessed through library discovery services than through google   a review of: wang, x., cui, y., & xu, s. (2018). evaluating the impact of web-scale discovery services on scholarly content seeking. the journal of academic librarianship, 44(5), 545-552. https://doi.org/10.1016/j.acalib.2018.05.010     reviewed by: judith logan user services librarian university of toronto libraries toronto, ontario, canada email: judith.logan@utoronto.ca   received: 4 feb. 2019                                                                     accepted: 12 mar. 2019      2019 logan. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29551     abstract   objective – to examine trends in digital object identifier (doi) web referrals and explore the referring domains, especially those originating from web-scale discovery systems like proquest’s summon and primo.   design – log analysis and web traffic analysis.   setting – crossref, a web server that connects dois to the corresponding articles’ landing pages.   subjects – web traffic that passed through crossref between 2011 and 2016.   methods – the researchers collected data from crossref using a web tool called chronograph. the data captured information about the websites users were on when they requested a doi (called the referrer) and about the time and date of each request.   the researchers used time series analysis to discover longitudinal patterns in the data. annual, monthly, and weekly trends were also examined with a seasonal adjustment model, a seasonal trend decomposition, and log transformation. they also isolated traffic from four institutions in australia, japan, sweden, and the united states of america to determine if overall seasonal patterns were reflected locally.   proquest websites were of particular interest to the researchers because they determined that it had the highest market share of discovery services. much of the analysis focused on proquest’s serialsolutions.com, exlibrisgroup.com, and proquest.com website domains.   main results – proquest servers sent over 25 million doi referrals through crossref – more than either web of knowledge (n=24.47 million) or google (n=15.38 million).   referral traffic grew over the period with the sharpest growth rate occurring between 2011 and 2012. of proquest’s domains, serialsolutions.com (summon) had more traffic and more growth over the observation period than exlibrisgroup.com (primo).   in all of the years studied, the busiest months were september to november and january to march, while june to august and december were low points. seasonal fluctuations were attributed to university vacation schedules as demonstrated in the traffic patterns of four proquest-subscribing institutions.   weekly trend analysis showed that monday to thursday had consistently heavy referral traffic. of the remaining days, the fewest referrals were observed on saturdays.   conclusion – doi referrer traffic is closely tied to the university calendar. library discovery products are used more frequently to access dois than google.   commentary   the authors have introduced a novel method of examining scholarly resource usage. log analysis was first adapted for libraries by nicholas, huntington, and watkinson (2005) as a means of dissecting user interactions with an electronic resource or platform. since then, many other researchers have used log analysis to better understand e-resource usage patterns (tripathi & jeevan, 2013). the study at hand is similar in that it uses raw data to examine interactions with scholarly resources, but it also recalls web traffic analysis studies since web domain referrals are the primary focus. web traffic studies are usually performed to provide libraries with actionable insights about their communities’ behaviour using locally owned data sources like google analytics (turner, 2010). the authors were able to perform a non-local analysis, however, thanks to crossref’s statistical openness.   the study presented is tightly focused. the analyses center mainly on the referring web domain and the time and date when the referral occurred. this in itself is a rich source of data, and the authors have clearly taken pains to ensure that the temporal trends are presented accurately, though they failed to mention the application used to process the data once obtained from chronograph. the authors acknowledge that time zones were not considered which could be a factor as referrals were not limited geographically. future research could determine if time zones affect the patterns discovered in the present study.   the choice of delving into proquest domains is fruitful and well-considered. it allows the authors to make sense of the undoubtedly thousands of domains passing through crossref by selecting a highly visible suite of scholarly discovery products. google domains were not included in the study’s temporal analyses, so it might be interesting to compare these with proquest or other library products to see if user behaviour differs.   the findings are more interesting than actionable. the temporal analyses are strikingly similar to patterns in other scholarly resource or service usage studies suggesting strong external validity (glynn, 2006). however, the study’s methodology is its primary practical contribution. researchers wishing to apply this methodology to other open scholarly resources may be limited by data availability. crossref has made their referral traffic publicly available through chronograph and other nonprofit scholarly resources should follow suit. the directory of open access journals and the hathitrust come to mind as scholarly resources that would benefit from a similar study.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154    nicholas, d., huntington, p., & watkinson, a. (2005). scholarly journal usage: the results of deep log analysis. journal of documentation, 61(2), 248-280. https://doi.org/10.1108/00220410510585214    tripathi, m., & jeevan, v. k. j. (2013). a selective review of research on e-resource usage in academic libraries. library review, 62(3), 134-156. https://doi.org/10.1108/00242531311329473   turner, s. j. (2010). website statistics 2.0: using google analytics to measure library website effectiveness. technical services quarterly, 27(3), 261-278. https://doi.org/10.1080/07317131003765910   editorial evidence based library and information practice 2009, 4:2 148 evidence based library and information practice evidence summary a librarian consultation service improves decision-making and saves time for primary care practitioners a review of: mcgowan, jessie, william hogg, craig campbell, and margo rowan. “just-in-time information improved decision-making in primary care: a randomized controlled trial.” plos one 3.11 (2008): e3785. 10 mar 2009 <http://www.plosone.org/article/info%3adoi%2f10.1371%2fjournal.pone.0003785> reviewed by: heather ganshorn librarian health information network calgary calgary, ab, canada e-mail: heather.ganshorn@ucalgary.ca received: 12 march 2009 accepted: 27 april 2009 © 2009 ganshorn. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. objectives – to determine whether a pointof-care librarian consultation service for primary care practitioners (pcps) improves the quality of pcps’ decision-making; saves pcps time; reduces the number of point-ofcare questions that go unanswered due to time constraints; and is cost-effective. overall pcp satisfaction with the service was also assessed. design – randomized controlled trial. setting – four family health networks (fhns) and 14 family health groups (fhgs) in ontario, canada. these represent new models for primary care service delivery in ontario. subjects – pcps working within the selected fhns and fhgs. the majority of these were physicians, but the sample also contained one resident, one nurse, and four nursepractitioners. methods – subjects were trained in the use of a web-based query form or mobile device to submit their point-of-care questions electronically. they were also trained in query formulation using pico (patient, intervention, comparison, and outcome). http://www.plosone.org/article/info%3adoi%2f10.1371%2fjournal.pone.0003785 mailto:heather.ganshorn@ucalgary.ca evidence based library and information practice 2009, 4:2 149 allocation was concealed by an independent company hired to manage data for the project. participants were not randomized; rather the questions were randomized using a random-number generator. to ensure blinding of the librarians, all questions submitted were answered by a librarian. answers to questions in the intervention group were relayed by a third party to the practitioner within minutes. answers to the questions in the control group were not communicated to the physician. blinding of the pcp subjects was not possible, as they either received or did not receive an answer. subjects were asked to respond to a questionnaire 24 hours after submitting their question. if the question was in the control group, subjects were asked to indicate whether they had let the question remain unanswered or pursued an answer on their own. in order to assess cognitive impact of both librarian-provided information and self-sought information, respondents were asked to rate information on a scale from high positive to negative impact on decision making. two linear regression models were run on the data, with participant response time as the dependent variable in the first model, and librarian response time as the dependent variable in the second. main results – the service received a total of 1,889 questions, of which 472 (25%) were randomized to the control group, and 1,417 (75%) to the intervention group. analysis run on both groups found that the types and complexity of questions were similar between the two groups, as was librarian response time. questions were rated for complexity (the rating scale is included in the article), and most (85%) had a level 1 complexity rating, meaning there was only one concept listed for each pico element. the primary outcome measure was the amount of time required to answer the question. average librarian time to respond to questions was 13.68 minutes per question. average pcp time to find answers to their own questions was 20.29 minutes; however, subjects only attempted to answer 40.5% of control-group questions themselves. costeffectiveness analysis was run on these times, and the authors found that the average per-question salary cost for a librarian to answer these questions (based on 15 minutes per question) was $7.15, while average salary cost for a pcp to spend 15 minutes searching for information ranged from $20.75 to $27.69. the results of the questionnaire indicated a significant positive impact of the information on clinician decision-making. approximately 60% of the questions in the control group went unanswered, whereas all of the questions in the intervention group were answered. of the questions answered by the information service, 63.7% of the answers were rated by participants as having a high positive impact on decision-making, versus 14.9% of answers to questions in the control group that practitioners sought out themselves. seventeen percent of the answers were rated as having a moderate positive impact in the intervention group, versus 5.9% in the control group. only 7.8% of answers in the intervention group were rated as having no impact, versus 24.8% of answers in the control group. a negative impact (where practitioners found too much or too little information or information that they disagreed with or felt was harmful) was found for 7.7% of librarian-provided answers, compared with 44.9% of practitioner-sought answers. satisfaction was very high, according to the exit satisfaction survey, with 86% agreeing that the service had a positive impact on decision-making, and 83% stating that evidence based library and information practice 2009, 4:2 150 relevant answers were provided in an appropriate time frame. most participants (72%) would consider using such a service, and 33% indicated they would be willing to pay for this type of service. conclusion – a point-of-care reference service, in which librarians answer primary care practitioners’ questions within minutes, has a very positive impact on clinical decision making and a high rate of client satisfaction. this system saves pcps time, which may allow them to spend more time with patients. in supporting good clinical decision making, the service may also decrease the need for referrals and further tests. the service is cost-effective, as librarians find better quality information than practitioners, and they do it faster, on a lower per-hour salary. commentary this is an interesting study for several reasons. though other studies have evaluated question-and-answer services, this study appears to be unique in looking at a just-in-time service. clinical trials of library services are still rare, so this trial is a useful example for others considering a clinical trial as their research methodology. the randomisation of questions rather than subjects is an interesting twist on the traditional clinical trial methodology. randomising the questions, in combination with using a 3:1 randomisation ratio that allocated most questions to the intervention group, ensured that frequent users of the service had a high probability of having most of their questions answered. the article has the authors’ consolidated standards of reporting trials (consort) checklist attached. the consort statement is “an evidence-based, minimum set of recommendations for reporting rcts” (consort group). this is an excellent template to use when planning and reporting on a trial. the checklist will assist with structuring the article, and may also assist in the planning of a study. despite its strengths, this study does have some limitations. the authors state that their sample was drawn from sites that were geographically convenient to the researchers, so the sample is not representative of all pcps. also, the study was carried out on a relatively small population. although 95 individuals declined to participate in the study, it is unknown why they declined as well as whether they differed from the participant group in any significant ways. the authors ran a pre-intervention simulation on the random number generator. this simulation determined that a sample of 88 physicians with 22 questions each (five of them controls) had a 99% power to distinguish between control and intervention groups. according to this author’s calculations, a total of 1,936 questions, 440 of which would be controls, would be needed. however, the authors indicate that the 88 physicians enrolled in the study generated only 1,889 questions. as the authors do not include their power calculations, it is difficult to tell what impact the lower number of actual questions would have on the statistical significance of the findings. nor do they explain why they chose to use 99% power, rather than a lower number. the authors also do not indicate who did the statistical analysis –one of the authors, or the outside agency that managed the project’s data? the authors state that they used two linear regression models, but these models are not stated in mathematical terms nor are the b weights and significance levels reported. evidence based library and information practice 2009, 4:2 151 it would have been useful to do a preassessment of the participants to discover their existing level of skill in literature searching and evidence based practice. this information might have provided some insight into the reasons for the large satisfaction gap between providers’ own searches and those done by librarians. in addition, it would have been helpful to know which databases were available to the librarians and subjects in the study, and which ones were most commonly consulted by each group. the authors acknowledge limitations to their study including a lack of follow-up with subjects to determine how they used the time saved by the service, or to discover why, in some cases, subjects felt that the information retrieved either by the service or by their own efforts had no impact or a negative impact on clinical decision-making. followup interviews could have been used to elicit this information. the authors also caution that the low enrolment numbers mean that their findings cannot necessarily be generalized. the authors provide several informative graphs and tables with their study, yet they do not include their survey instrument. it would have been useful to see both the question-specific questionnaire that subjects received 24 hours after submitting their question and the final exit survey. overall, this is perhaps the best example of a clinical trial of library services to date. librarians considering similar trials of their own services will find this article invaluable to their planning process. works cited consort group. consort: transparent reporting of trials. 2009. 11 mar 2009 <http://www.consortstatement.org/>. http://www.consort-statement.org/ http://www.consort-statement.org/ evidence summary   emotions expressed in online discussion forums are associated with information poverty and level of information need   a review of: ruthven, i., buchanan, s., & jardine, c. (2018). isolated, overwhelmed, and worried: young first-time mothers asking for information and support online. journal of the association for information science & technology, 69(9), 1073-1083. https://doi.org/10.1002/asi.24037     reviewed by: barbara m. wildemuth professor emeritus, school of information & library science university of north carolina at chapel hill chapel hill, north carolina, united states of america email: wildemuth@unc.edu   received: 11 june 2019                                                                  accepted:  26 july 2019      2019 wildemuth. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29593     abstract   objective – to understand the emotions associated with online forum requests for information from young first-time mothers.   design – naturalistic study of existing online forum postings.   setting – two uk-based online discussion forums intended for use by young mothers.   subjects – two hundred thirty-seven young (aged 14 to 21) first-time mothers, who posted 279 messages in the two forums.   methods – the 279 messages were categorized in terms of 1) the type of emotion expressed, using an inductively developed coding scheme that included interaction emotions, preoccupation emotions, and response emotions; 2) four dimensions of information poverty: secrecy, deception, risk, and situational relevance; and 3) whether the information request expressed a conscious or a formalized information need. in addition to analyzing the frequency with which particular emotions occurred, co-occurrences of emotions with information poverty dimensions and emotions with level of information need were analyzed.   main results – as expected, most of the forum posts included expressions of emotions. interaction emotions relate to the mother’s interactions (or lack of them) with other people and were expressed in 75 of the posts; the most frequently expressed interaction emotions were feelings of isolation and being judged. preoccupation emotions are concerned with states of mental absorption or uncertainty and were expressed in 141 of the posts; the most frequently expressed preoccupation emotions were worry, a feeling of being overwhelmed, and self-doubt. response emotions include reactions to someone else or a situation and were expressed in 45 of the posts; the most frequently expressed response emotion was anger, frustration, or venting (which were handled as one unit by the authors).   dimensions of information poverty were found in 57 of the 279 posts in the sample. situational relevance (i.e., the desire for support or information from someone that is in a very similar situation) accounted for over half of the instances of information poverty. the risks associated with young motherhood were expressed in over a quarter of the instances of information poverty.   emotions were more likely to be expressed when the post included evidence of information poverty. when posts of conscious and formalized needs were compared, emotions were more likely to be expressed in posts of conscious needs (i.e., those which had not yet been formalized).   conclusion – almost all of the 279 posts in the sample included strong emotional content, mostly negative emotions such as worry, isolation, and frustration. these emotions were associated with expressions of information poverty; in particular, feelings of isolation were closely associated with information poverty. in addition, posters at an early stage of problem recognition, expressing a conscious but not yet formalized information need, were more likely to experience these negative emotions. these findings have strong implications for moderators of online forums hoping to provide support to young first-time mothers.   commentary   the emotions associated with various information behaviours have been of interest to the library and information practice community since at least the late 1980s when early versions of kuhlthau’s information seeking process model began to appear (kuhlthau, 1988). studies of the emotions associated with information seeking and related information behaviours have been reviewed in both nahl (2007) and, more recently, lopatovska and arapakis (2011). this study focused on the emotions expressed by users of an online forum while requesting information or support from peers. because existing taxonomies of emotions were not a good fit for the emotions found in these online forums, they developed and applied their own coding scheme, which is clearly defined in the paper and provides a strong basis for classifying the emotions expressed in these discussion forums. while valid for this study, this categorization scheme may not be applicable in similar studies, so a similar process of category definition may be necessary for a replication.   the population of interest in this study, young (aged 14 to 21) first-time mothers, was selected because they face significant challenges in the transition to motherhood. because it was (necessarily) a naturalistic study, the sample was defined using very specific inclusion criteria for the two online forums selected and for the individual posts analyzed. these criteria did a good job of defining a homogeneous sample, avoiding ambiguity about who might be expressing the emotions identified during coding.   to address the study’s research questions, descriptive statistics were reported in addition to findings about the relationships between the expressed emotions and the following two concepts: information poverty and the type of information need being expressed. the paper’s tables report the relevant statistics, though the number of unique posts that contained at least one expression of emotion is not reported. at least one additional potentially confounding factor was not investigated in this study: the topic of the information need (e.g., the advisability of breastfeeding, finding appropriate housing, etc.). it is possible that the topic might be related to the emotions expressed or affect the relationship between the expressed emotions and the other two concepts. it would be appropriate for a future study to investigate these potential interactions.   the findings of this study are of particular importance for moderators of online forums. since emotions are frequently expressed in online forums, we can infer that they are important to the users of those forums. this finding has two specific implications for moderators. first, moderators should look for ways to retain user participation in the forum, thus reducing their feelings of isolation and helping them to link to other situationally relevant users. second, analysis of current postings may allow moderators to identify those users whose information needs are not yet formalized and to provide additional support through some type of online mentoring program or faq services.   references   kuhlthau, c. c. (1988). perceptions of the information search process in libraries: a study of changes from high school through college. information processing & management, 24(4), 419-427. https://doi.org/10.1016/0306-4573(88)90045-3    lopatovska, i., & arapakis, i. (2011). theories, methods and current research on emotions in library and information science, information retrieval and human-computer interaction. information processing & management, 47(4), 575-592. https://doi.org/10.1016/j.ipm.2010.09.001   nahl, d., & bilal, d. (eds.). (2007). information and emotion: the emergent affective paradigm in information behavior research and theory. medford, nj: information today.     review article   syllabus mining for information literacy instruction: a scoping review   kathleen butler health sciences librarian university libraries george mason university fairfax, virginia, united states email: kbutle18@gmu.edu orcid id 0000-0002-9608-4784   theresa calcagno it and engineering librarian university libraries george mason university fairfax, virginia, united states email: tcalcagn@gmu.edu orcid id  0000-0001-7422-1330     we have no conflicts of interest to disclose.   correspondence concerning this article should be addressed to kathleen butler, university libraries, george mason university, 4400 university dr., msn 2fl, fairfax, va 22030. email: kbutle18@gmu.edu   received: 6 july 2020                                                                 accepted: 15 sept. 2020      2020 butler and calcagno. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29800     abstract   background the course syllabus is a roadmap to curriculum development and student learning objectives providing valuable information to assist library instruction. this scoping review examines research that uses syllabus mining to track information literacy concepts and skills in academic settings.   objectives the present study uses a scoping methodology to examine syllabus mining of information literacy with the focus of analysis on the methodologies employed in syllabus review and the recommendations from the studies.   design searches of databases of literature from librarianship and education, as well as a multidisciplinary database, yielded 325 journal articles. inclusion criteria specified peer-reviewed articles from any year, and excluded grey literature. after removing duplicates, 2 reviewers screened titles and abstracts and reviewed full text, yielding 17 studies to analyze.   results characteristics of the included studies, methodology, and recommendations were charted by two reviewers. all studies reported retrieving information that increased opportunities for collaboration with instructors and targeted engagement with students, and seven themes were identified.   conclusions instructional librarians should be encouraged to conduct syllabus studies to increase collaboration with faculty to develop coursework, to meet student information needs in a strategic manner, and to identify discipline-specific information literacy concepts.     introduction   course syllabi provide a roadmap to instructional goals and the development of the student as scholar. although syllabi may present challenges with accessibility and inconsistency, and contain incomplete or vague content, they are one tool instructional librarians can use to coordinate information literacy (il) instruction with a course. student learning objectives (slos) in syllabi show concepts suitable for instruction, helping librarians coordinate the timing of instruction and skill development (miller & neyer, 2016).   one reason for studying research on syllabus mining is to see how il has evolved over time. perceptions of il are still evolving, beginning with bibliographic instruction and moving to il standards, and in 2018, the creation of the il framework. many disciplines and accreditation agencies now incorporate il concepts as part of their professional competencies (aac&u, n.d.; ache healthcare executive competencies assessment tool, 2020). examining the syllabus of a course is an effective way to determine how il is reflected and will help the library instructor put the necessary il skills and concepts into relevant context.   another reason to study syllabus mining is to identify ways library instructors can collaborate with faculty (williams, et al, 2004; dubicki, 2019). by examining syllabus course objectives, the librarian has information to suggest timely and relevant literacy instruction to faculty and create support materials in subject guides or build instructional modules for integration in online learning systems.   williams, cody, and parnell (2004, p.270) sum up the importance of syllabi studies to academic libraries: “key to embedding the library into the student experience is to be an integral part of the course work. the most detailed evidence of what that coursework entails is the syllabus. therefore, obtaining and analyzing syllabi for existing and potential library collaboration are valuable endeavors for librarians.”   historically, syllabus studies in library research examine different outcomes. rambler (1982, p.156) is credited with the first study in library research to examine the syllabi across an academic setting to identify assignments that require library resources and services: “in essence, decisions and actions based at least in part on findings from a syllabus study can facilitate the creation of the ideally responsive and completely curriculum-integrated library.” other studies look at how the syllabus reflected or influenced library usage (dewald, 2003; lauer, 1989) or collection development (lukes et al., 2017). this scoping review aims to systematically search for library research utilizing syllabus studies and information literacy objectives or instruction in academic settings to provide an overview of what research has already been done and help inform new research.   why a scoping review?   a systematic review “uses explicit, systematic methods that are selected with a view to minimizing bias, thus providing reliable findings from which conclusions can be drawn and decisions made” (liberati et al., 2009, p. e2). the strict methods used in a systematic review and a scoping review are designed to minimize bias in study selection and analysis and provide transparent and replicable study design. however, unlike a systematic review, a scoping review is designed to look at literature on a topic without an analysis of quality, so it reveals an overview of all research on a broad topic. the results of a scoping review will not necessarily point to new or best practices or answer a clinical question, but will show the breadth of research conducted on a topic. (arksey & o’malley, 2005; peters et al., 2015).   this scoping review follows the five stages outlined by arksey and o’malley (2005): 1) identifying the research question, 2) identifying relevant studies, 3) study selection, 4) charting the data, and 5) collating, summarizing, and reporting the result. also consulted was the scoping review checklist published by the preferred reporting items for systematic reviews and meta-analyses (prisma) in 2018, which includes 20 essential items. (tricco et al., 2018)   the topics of this scoping review are:   how has syllabus mining or syllabus review been used by librarians to inform information literacy instruction in the academic setting? what methods for analyzing syllabi are described in library and information science literature? what are the conclusions or recommendations of these studies for il instruction?   methods   a protocol established by the authors identified inclusion and exclusion criteria. the protocol is registered at open science framework, february 26, 2020, osf.io/9ur2n, “syllabus mining for information literacy instruction: a scoping review protocol.”   eligible articles for inclusion must be in the english language and peer reviewed. only syllabus studies of college or university classes (undergraduate and graduate levels) in any discipline were included. grey literature was excluded as were studies using syllabi created by librarians for information literacy. no dates were specified, so selected databases were searched without date limits. the search was completed in may 2019.   the searches were performed in the databases that index library research and education research: library and information sciences abstracts, library and information technology abstracts, eric, and education research complete. web of science was searched as the multidisciplinary database available to the reviewers.     figure 1 prisma flowchart     the search strategy used keywords and controlled vocabulary to reflect concepts of “information literacy” and syllabus.   the following search was executed in library literature and information science index full text and library and information science and technology abstracts databases using keywords and descriptors (de):   (syllabus or syllabi)  (“information literacy” or “librar* instruction” or “librar* teach*” or “bibliographic instruction” or “library research”) (de "information literacy" or de "electronic information resource literacy" or de "health literacy" or de "internet literacy" or de "media literacy") 2 or 3 1 and 4   source selection   search results were collected using zotero (https://zotero.org), a citation management software, and duplicates were removed using that software’s feature. two reviewers independently conducted abstract review using rayaan (https://rayyan.qcri.org/) software. the reviews were blinded (reviewer did not know other reviewer’s decision) using rayaan’s feature to help minimize bias. any differences in include/exclude decisions were resolved with discussion. the reviewers piloted a checklist of inclusion/exclusion criteria for full text review in rayaan and differences were resolved with discussion. in addition, the reviewers checked the bibliographies of selected articles and, after a second review, decided to include two studies previously excluded. the prisma flowchart (figure 1) reflects the steps of the selection and review process.   data charting   a preliminary excel sheet with categories was created, and two articles were independently charted by two reviewers. the results were compared, and after discussion the chart was fine-tuned with additional categories before charting all articles. after all articles were charted, reviewers agreed some categories (academic units, class standing, and number of syllabi; methodology and analysis) should be combined.   variables charted for each article included:   ·        purpose or research question(s) ·        academic units ·        graduate or undergraduate classes ·        number of syllabi retrieved ·        methodology and methods of analysis ·        indicators for information literacy instruction ·        il standards used ·        results ·        recommendations ·        limitations ·        themes   results   search results yielded 325 journal articles, and an additional 20 articles were identified through citation analysis of chosen articles. duplicates were removed and 115 articles were eliminated because they did not meet predetermined criteria. the resulting 25 articles were examined in full text, and eight articles were eliminated because information literacy was not the focus of the research or they focused on library usage or collection development. the total number of articles for synthesis was 17.   one article (rambler, 1982) was not included in the final analysis, even though it is cited frequently by research and considered a foundational study. while it is cited as the first syllabus study to address library integration, the outcomes were related to library usage and not specifically information literacy instruction. thus, this study is outside the scope of this syllabus review.     table 1 study characteristics citation purpose academic units # of syllabi undergraduate course syllabi studies alcock & rose., 2016 1) examine difference in disciplinary il & instruction;  2) identify gaps & opportunities to integrate instruction history and chemistry 48 dewald, 2003 1) identify faculty expectations of library use and research for undergraduate business students. school of business not explicitly stated dinkelman, 2010 1) identify research components of il learning outcomes; 2) examining il instruction in a single discipline. biology (majors only) 104 lowry, 2012 1) identify faculty expectations of library use for undergraduate business students business/accounting 66 mcgowan, et al.,2016 1) see how il courses aligned with acrl il standards. multi-disciplinary (all) 1153 miller & neyer, 2016 1) identify research components of il learning outcomes; 2) map nursing curriculum to multiple published il standards. nursing 25 morris, et al., 2014 1) identify expectations of history faculty for archival research skills; 2) create a list of archival research competencies. history 37 o’hanlon, 2007 1) identify relationship of institutional learning outcomes to library research skills instruction; 2) develop a better understanding of faculty implementation of learning outcomes in the classroom. multi-disciplinary 71 smith, et al., 2012 1) identify gaps and opportunities to integrate il instruction; 2) examine differences in disciplinary il instruction; 3) examined library usage expectations multi-disciplinary (all) 144 stanny, et al., 2015 1) review syllabi for best practice components and il was a part of this assessment. multi-disciplinary (all) 1153 vanscoy, & oakleaf, 2008 1) assess research skills needed by incoming college freshman; 2) identify gaps and opportunities for curriculum-based il instruction. multi-disciplinary  (all) 139 undergraduate and graduate course syllabi studies beuoy,& boss, 2019 1) establish methodology for syllabus analysis using acrl framework; 2) identify opportunities for scaffolded/tiered il instruction; 3) examine disciplinary il instruction media, culture & communication;    food studies; & teaching & learning 104 boss, & drabinski, 2014 1) identify opportunities for curriculum-integrated il instruction in school of business classes. school of business 79 dubicki, 2019 1) align il instruction with il in syllabus/curriculum; 2) opportunities for scaffolded/tiered il instruction. multi-disciplinary 180 jeffery, et al., 2017 1) identify library resources and people in syllabus; 2) identify library engagement opportunities 3) establish methodology for syllabus analysis. multi-disciplinary; 1258 maybee, et al., 2015 1) identify expectations for student learning in il and data il nutrition;  political science 88 willingham-mclain, 2011 1) examine articulation of learning outcomes; 2) determine alignment of student learning outcomes in syllabi with institutional outcomes. multi-disciplinary (10 schools) 280     synthesis of results   purpose of research (table 1 characteristics)   the overall purpose of these research studies was to coordinate library il instruction with course and faculty expectations. investigators hoped the results would identify opportunities for, and find gaps in, il instruction (alcock & rose, 2016; dinkleman, 2010; jeffrey et al., 2017; smith et al., 2012) and better collaboration with faculty (dubicki, 2019; lowry, 2012; mcgowan et al., 2016; stanny et al., 2015; dewald, 2003). several articles discussed the place of library il in overall curriculum development, and looked for ways to scaffold, embed, or tier instruction (beuoy & boss, 2019; boss & drabinski, 2014; dinkelman, 2010; vanscoy & oakleaf, 2008; willingham-mcclain, 2011). also, aligning course objectives with il standards or institutional il standards was present in several articles (willingham-mcclain, 2011; beuoy & boss, 2019; mcgowan et al., 2016; miller & neyer, 2016; dubicki, 2019). identifying il components of disciplinary competencies was key in accounting (lowry, 2012), nursing (miller & neyer, 2016), biology (dinkleman, 2010) and other disciplines (maybee et al., 2015; dewald, 2003). developing il competencies specific to archives (morris et al., 2014) and data (maybee et al., 2015) were prominent in two studies. jeffrey et al. (2017) hoped to establish a methodology for syllabus analysis.   academic units, number grad/undergrad (table 1 characteristics)   there was a wide variation in number of syllabi included in research design. a total of 1153 syllabi were retrieved and analyzed in 2 different papers by the same group of researchers. in the first paper, il was a piece of the overall evaluation and the study was a collaboration between the librarian and institutional entities evaluating syllabi for best practices (stanny et al., 2015). the same data set was examined more closely for il outcomes in the second article (mcgowan et al., 2016). the smallest set was 13 syllabi from chemistry courses (alcock & rose, 2016) compared to 35 syllabi retrieved from history courses. the author acknowledged the small set was not generalizable but did offer insight into the instructors’ expectations for il.   six studies looked at a combination of graduate and undergraduate courses (see table 1) and 11 studies examined syllabi from undergraduate courses only. there was not a study that examined graduate course syllabi only. eleven studies analyzed syllabi for a range of disciplines (table 1), but 6 studies focused on individual disciplines, specifically, biology, accounting, nursing, history, and business. two papers provided direct comparisons of two disciplines, contrasting science, humanities, and social sciences—history vs chemistry, and nutrition vs political science.   the indicators for il content reflected tasks, assignments and concepts:   tasks that reflected library resource usage –find articles, find statistics (vanscoy & oakleaf, 2008; lowry, 2012; alcock & rose, 2016; dewald, 2003). assignments including independent use of the library, research papers, annotated bibliographies (mcgowan et al., 2016; dubicki, 2019). statements in the syllabus that reflected il concepts: slos: academic integrity, critical thinking (dubicki, 2019) il competencies from discipline standards—aacn nursing (miller & neyer, 2016); canadian accounting (lowry, 2012) institutional curriculum goals (willingham-mclain, 2011) acrl standards (mcgowan et al., 2016), acrl framework (boss & drabinski, 2014; beuoy & boss, 2019) slos in syllabus compared to il concepts from acrl (stanny et al., 2015; miller & neyer, 2016; mcgowan et al., 2016; lowry, 2012; dubicki, 2019; beouy & boss, 2019) aac&u (miller & neyer, 2016; boss & drabinski, 2014; alcock & rose, 2016), or middle states commission on higher education (willingham-mcclain, 2011).   the lauer/dewald rating scale was used by multiple studies to score the syllabi from 0-4 (used by smith et al., 2012; lowry, 2012; dewald, 2003). a score of zero was assigned if a syllabus showed no research or library use, one point was given to a syllabus with reserve readings, a score of two meant students were required to complete optional readings not on reserve, three points were awarded for shorter writing assignments or presentations, and four points were awarded to a syllabus reflected a significant research project (10 pages or 20% of grade).   boss and drabinski (2014) developed a list of questions, each related to an acrl frame. responses were scored 0-2 for each frame, with a possible total of 12 points (used by alcock & rose, 2016; adapted by beouy & boss, 2019). a list of questions derived from o’hanlon (2007) were used by dinkleman (2010) to evaluate syllabi but not assigned a score.   most studies reported results in terms of percentages of syllabi that contained il concepts or assignments (table 3). percentages varied by discipline (alcock & rose, 2016; dinkelman, 2010; lowry, 2012; morris et al., 2014). science classes showed fewer assignments that required library research (alcock & rose, 2016; dinkelman, 2010). history, as a subject in arts & humanities, required many more library research assignments (alcock & rose, 2016; morris et al., 2014). percentages of il present in syllabi differed by which indicators were used in assessment, and for this reason, comparison of studies is problematic. vanscoy & oakleaf (2008) looked for statements in syllabi that required finding any library material and scored a 97% rate in syllabi. independent research was used as an indicator of il in boss and drabinski, (2014) with a rate 73%, and also in alcock and rose (2016) showing history at 85% and chemistry at 39%.   rubrics, scales, or questions were used by multiple reviewers as evaluation tools for syllabi, so interrater reliability was an important consideration. interrater reliability was calculated for beouy and boss (2019) using cohen’s kappa calculations; krippendorf’s alpha was used in studies by boss and drabinski (2014) and mcgowan et al. (2016). mcgowan et al. (2016) used a random sample of syllabi for norming with reviewers. scores were assigned for the presence of il tasks or concepts in syllabi: 0-4 (smith et al., 2012; lowry, 2012; mcgowan et al., 2016).       table 2 indicators for information literacy in syllabi indicators of il critical thinking evaluating sources integrate multiple viewpoints academic integrity form research question   annotated bib; presentations; book reports                         research paper/project independent research data use reserves; find articles, etc alcock & rose, 2016 x x x x x beuoy & boss, 2019 x x x x x x boss & drabiniski, 2014 x x x x x dewald, 2003           x x     x dinkleman, 2010 x x x x dubicki, 2019 x x x x jeffrey et al., 2017 x x x lowry, 2012 x x x maybee et al., 2015 x x x x mcgowan, 2016 x x x miller & neyer, 2016 x x x x x morris, 2014 x o’hanlon, 2007             x x   x smith et al., 2012 x x x stanny et al., 2015 x x vanscoy & oakleaf, 2008 x x willingham-mclain, 2011 x x x x     table 3 methodology and analysis of studies reviewed citation il standards used in analysis syllabi with il indicators (%) analysis alcock, e., & rose, k., 2016 acrl framework; aac&u history 85% (independent research); chemistry 39% (independent research) used a question list requiring yes/no answers; also searched for keywords in the text. bias was minimized by using yes/no answers and double coding. quantitative methods. beuoy, m., & boss, k., 2019   acrl framework not specifically reported used the boss & drabinski (2014) scale and adapted it to the acrl il framework. normed a randomized set of syllabi; inter-rater reliability was calculated using cohen's kappa.  used nvivo for analysis of syllabi text. used scale to review syllabi for the presence of the 6 il frames.  assigned each syllabi a score of 0-2 based on the frame presence. quantitative methods. boss, k., & drabinski, e., 2014 aac&u 73% developed questions based on the aac&u value rubric to measure presence of il concepts in syllabi. two raters evaluated syllabi independently and compared. normed a set of three unrelated syllabi at beginning and measured inter-rater reliability using krippendorf's alpha and stemler's per cent agreement method.  quantitative methods. dewald, n., 2003 none 52.90% modified the scale developed by lauer et al.  (1989) to include non-library research, e.g. online research or personal contact. analyzed syllabi and scored them according to the scale. quantitative methods.   dinkelman, a. l., 2010 none 25% (biology) combined rubrics/question lists from holiday & martin (2006) and o'hanlon (2007) and then added several other questions.  norming was not done.  used the combined list to evaluate syllabi for the inclusion of il concepts. data were collected on excel sheets. quantitative methods. dubicki, e., 2019 acrl framework 81% syllabi were analyzed to identify slos, research assignments, library services and resources.  data were collected using excel spreadsheets listing courses by level and course codes. columns contained the possible il indicators on a syllabi and their presence was marked with a check. faculty defined learning outcomes were then mapped to the acrl il framework.   jeffery, k. m., et al., 2017 none 21% (research) obtained spreadsheet for syllabi metadata and wrote script to use to download syllabi from the dspace repository. all syllabi (1258) were converted to pdf and imported into qda miner.  metadata were applied to each document.  developed list of keywords related to library, library services, spaces, and research assignments.  similar keywords were grouped to form codes.  coded syllabi (1226 or 17% of classes) were analyzed. used sorenesen's coefficient of similarly to map relationships between codes. lowry, l. 2012 acrl standards; canadian professional accounting competency standards:  canadian institute of chartered accountants; certified management accountants canada and certified general accountants canada 12% (upperclass accounting) collected accounting syllabi for 1 academic year. modified dewald's (2003) revision of the scale developed by lauer et al. (1989). the current revision included questions related to the use of course management systems for linked readings. author coded syllabi using the modified 5 pt scale. evidence-based methods. quantitative methods. maybee, c., et al., 2015 none 13.8% (undergrad political science)  17.9% (undergrad nutrition science)  57.1% (grad political science)         69% (grad nutrition science) used grounded theory approach.  used two teams, one for each subject area.  teams read though syllabi and did initial coding.  syllabi were reviewed a second time by teams and coding results were discussed.  categories of code groups were created and memos discussing each category were written (iterative process). when consensus was reached on the categories, the teams reviewed them to identify the themes. qualitative method. mcgowan, b., et al.,2016 acrl standards 79% developed a rubric to measure if student learning outcomes were aligned with the acrl il standards. rubric was normed using 110 syllabi randomly selected for training with graduate student reviewers.  conducted weekly calibration checks with coder pairs; computed inter-rater agreement. agreement scores improved during the process.   quantitative methods. miller, m., & neyer, l., 2016 acrl il standards; aac&u value rubric;  acrl standards for nursing; aacn standards (nursing); not specifically reported collected all syllabi for nursing classes (n=25) and assignment descriptions. data were transferred to spreadsheets along with keywords from the course description, course objectives, etc. learning outcomes were then mapped to the aac&u value rubric for il and written communication.  mapping was also done to the acrl standards for both il (2000) and for nursing (2013).  a crosswalk was developed with the aacn essentials (2008).  morris, s., et al., 2014 none 60% (primary sources) university archivist developed a list of indicators of archival activities and syllabi were analyzed to identify classes with any of these indicators present. conducted interviews with select history faculty regarding their expectations for student development of archival awareness and research skills. revised list of archival competencies using suggestions from faculty.  after list was revised, sent the list to all history faculty for the feedback. mixed methods. o'hanlon, n. 2007 none not specifically reported conducted web-based survey of faculty focused on writing assignments and research related tasks used in classes as wells as information research skills in students.  reviewed syllabi from interested survey respondents or syllabi that were found on the internet. syllabi came from second writing classes and senior capstone courses. mixed methods. smith, c., et al., 2012 none 57% gathered syllabi from registrar (5173 course sections).  filtered out certain class types: first year composition, graduate, laboratory classes and directed research classes.  randomly sampled the remaining syllabi (n=1496) to get a subset of 200 syllabi.  requested syllabi for these classes from instructor with a return rate of 52% (144 syllabi). used dewald's modified version (2003) of the lauer et al. scale (1989).  syllabi were coded by pairs of reviewers and disagreements were noted.  all disagreements on syllabi were then re-examined and coded by all six team members. quantitative methods. stanny, c., et al., 2015 acrl standards 59.20% developed a rubric and used it to document slos and assignments on the syllabi and their alignment with the acrl il standards.  rubric was normed using 110 syllabi randomly selected for training purposes.  conducted weekly calibration checks with coder pairs and inter-rater agreement (95%) was computed. quantitative methods. vanscoy, a., & oakleaf, m.j., 2008 mentioned acrl il standards          but not used in analysis 97% registrar provided a random sample (n=350) of first semester, freshman containing course information for each.  data were transferred to a relational database for analysis. syllabi and assignment information were collected but complete data were collected for only 139.  the full set of syllabi was analyzed to see if assignments required research tests and yes/no was entered into the database.  willingham-mclain, l., 2011 il components from the middle states commission on higher education; institutional  student learning outcomes 44% created question list from self-study questions list developed by university for accreditation.  created a random sample of syllabi containing 10% of courses in all departments.  solicited syllabi from department and received 68%.  developed a detailed coding sheet and then refined it to be more precise.  three researchers each coded one-third of the syllabi for answers to all the questions.  il was present if one or more of the il indicators from the middle states commission on higher education were found. random, stratified sample used; irr was informally done.     in some studies, scales were used to assign scores to syllabi showing the extent of il activity. higher scores were associated with more demanding il assignments (smith et al., 2012; lowry, 2012; mcgowan et al., 2016). the lauer rating scale gave long research papers/projects more weight than shorter assignments. the syllabi in beouy and boss (2019) were compared to the acrl framework and each frame present in the syllabus was scored between 0-2, with an optimal il score of 12 points for the syllabus. other studies mapped syllabi to il concepts in professional standards, such as aac&u (boss & drabinski, 2014; alcock & rose, 2016); aacn essentials, and the acrl standards for nursing (miller & neyer, 2016). institutional student learning objectives were also used as tool for identifying il concepts (willingham-mcclain, 2011; o’hanlon, 2007).   for many papers, the syllabus studies highlighted classes that were missed by subject librarians or that provided opportunities for il instruction (mcgowan et al., 2016; alcock & rose, 2016; beouy & boss, 2019; boss & drabinski, 2014). the results of syllabus analysis gave librarians new and strategic information for approaching faculty and more opportunities for collaboration (lowry, 2012; boss & drabinski, 2014; beouy & boss, 2019; dewald, 2003).   willingham-mclain (2011) and dinkelman (2010) made recommendations for improving syllabi overall. dinkleman specifically addressed ways to make library resources and services clear to students. for science classes, names of discipline specific databases should be part of the syllabus and the subject librarian and library resources/services should be mentioned. also, wording can be confusing and students may misinterpret directions. if the statement "only 2 resources may be from the internet” is in the syllabus, students may think a scholarly article from a dot com publisher is excluded (dinkleman, 2010).   tailoring il instruction for different disciplines was described in several studies. dinkleman (2010) noted the basics of science literacy and reading a scientific article were indicators for il instruction. alcock & rose (2016) compared syllabi from chemistry and history, and maybee et al. (2015) compared syllabi from nutrition science and political science. both studies noted very different slos and research development. data literacy was included in the maybee et al. (2015) review and they argue that data literacy is a component that should be included when measuring il. morris et al. (2014) looked at archival literacy as a specialized form of il and examined syllabi for use of primary sources.   maybee (2015) found that research assignments increased in syllabi in graduate studies across disciplines. dubicki (2019) found more complex research required in upper level and graduate classes. two studies included graduate and undergraduate syllabi in the same subjects. an increase in emphasis on the research process and data analysis in graduate courses was noted in maybee et al. (2015). beouy and boss (2019) used the acrl frames to identify increased opportunities for research and il intervention in graduate courses. vanscoy and oakleaf (2008) determined that tiered il instruction may not be appropriate because their study of incoming freshman showed il tasks in syllabi from the beginning. however, dubicki (2019) argued strongly for scaffolding instruction and teaching il on a novice to expert searcher path.     table 4 results and recommendations of studies reviewed citation results alcock, e., & rose, k., 2016 comparison of the results revealed that chemistry always lagged history.  in history, 85% of the syllabi required independent research versus 39% of the chemistry syllabi. cumulative projects were required on 72% of the history syllabi versus 8% of the chemistry syllabi. beuoy, m., & boss, k., 2019 analysis of the il presence scores showed in all but one frame, the average score for all disciplines was less than 1. food science had the highest average scores in five of the six frames. media, culture and communication's average score was the highest in one frame (information has value.) boss, k., & drabinski, e., 2014 the data showed that 53% of syllabi required business students to use library resources independently while 64% of syllabi required a cumulative project. dewald, n., 2003 analysis showed that in 2001-2002, 48% of all business classes reviewed did not require library use or research.  significant research projects during the same period were found in only 18.3% of the business classes. dinkelman, a. l., 2010 found that only 18% (average) of the syllabi with il assignments mentioned the library as a resource and that only 10% (average) of the classes required a research paper/project.  recommended a required library course for students dubicki, e., 2019 addressed need to tailor il instruction for specific disciplines; found that tiered il instruction is important for a student's development. jeffery, k. m., et al., 2017 no mention of library related services or spaces or research assignments was found in 54% of the syllabi.  the most popular keyword codes were research paper, apa, and mla. lowry, l. 2012 syllabi covered 100% of accounting classes during study period and found that only 8 of 66 courses (12%; all at the senior level) required outside research or significant research. author suggests that problem-based learning is a good way for students to acquire information competence. maybee, c., et al., 2015 major themes identified in political science were: research inquiry at the undergrad level and research process and critical awareness of aspects of political science research at the grad level.  in nutrition science, the major themes identified included professional identity and scientific practice (undergraduate) and engaging as a scholar including information and data literacy (graduate). mcgowan, b., et al.,2016 inclusion of any acrl il standard by course varied from 53.8% at the junior level to 65% at the senior level and averaged 58.8%. research paper or literature review without data collection varied from 19.6% (sophomores) to 33.1% (freshman). empirical research papers varied from 1.6% of assignments for sophomores to 3.8% for juniors. miller, m., & neyer, l., 2016 discussed il instruction scaffolding within nursing and the importance of tiered il instruction. analysis showed that il outcomes in assignments were explicit 84% of the time. morris, s., et al., 2014 began development of a list of archival literacy competencies. o'hanlon, n. 2007 48% of all syllabi analyzed did not contain any research-related slos. 59% of syllabi described a writing assignment requiring external research. smith, c., et al., 2012 hypothesis that library research would be required by the majority of classes.  the findings did support the hypotheses that the amount and degree of research required would vary by course level and that the amount of research would also vary by subject discipline. stanny, c., et al., 2015 more than half of the syllabi (58.5%) had one or more course slo that aligned with il outcomes.  alignment of assignments with il outcomes was observed in 59.2% of syllabi. online classes described few assignments related to il concepts and the most common assignment was not an il assignment. in both online (17%) and face-to-face classes (27%),  literature reviews were the most common il assignment. vanscoy, a., & oakleaf, m. j., 2008 recommended a re-examination of earlier tiered il instruction recommendations; analysis showed that 97% of the 350 students had assignments that required the use of research resources. for the subset of 139 students, 100% had assignments requiring them to find research resources with the most common being (in rank order) articles, websites, and books for both groups. willingham-mclain, l., 2011 found that 44% of syllabi incorporated any of the middle states commission il indicators.     limitations identified by study authors   several limitations were noted by authors of the syllabus studies. standardized templates for a syllabus can skew results of a syllabus analysis. if the library is mentioned in a syllabus, and that is used as an indicator of il in the course, the researcher needs to know if it is referring to a building, resources, or services (alcock & rose, 2016). “template language and template syllabi can also yield less robust data, as they are a shell for the course” (beouy & boss, 2019; boss & drabinski, 2014). a lack of a thorough norming process for interrater reliability (boss & drabinski, 2014) and the small number of syllabi in sample sets (alcock & rose, 2016; dubicki, 2019) were limitations in some study designs. finally, slos and learning goals can be unique to an institution or department, so generalization of results to other campuses is not possible (dubicki, 2019).   discussion   the questions posed for this scoping review asked how syllabus studies were used to inform il instruction, what methods were used to analyze syllabi, and what recommendations were suggested by researchers for il instruction. seven themes were identified in this scoping review.   universally, syllabus examination gave librarians better insight into collaboration with faculty and student instruction   all studies tried to determine the expectations of instructors for il concepts and tasks through syllabus examination. some found opportunities identified by syllabi to offer il instruction to faculty; others gained an understanding of scaffolding instruction; and others identified specific courses that would benefit from librarian intervention. several studies reported better collaboration with faculty because of the information derived from the syllabus study.   “what emerged were indicators of potential student needs as they conduct research projects, leading to a roadmap of the topics that librarians should include during il instruction at various levels of students' academic careers, as well as services the library can develop to support students' independent study.” (dubicki, 2019, p. 291)   “rather than approaching faculty and administration with the assertion that librarians can add value to their program, the gathered data provide evidence for this claim, as librarians make the case for institutional collaboration and the need for increased resources for the information literacy program.” (boss & drabinski, 2014, p. 274)   disciplines vary in kinds of il instruction needed   differences in il requirements for subjects were illustrated by comparisons between history and chemistry, and food science and political science. several studies approached il with specialized concepts: primary sources, scientific literature, data literacy. also recognized were the range of research assignments that are specific to disciplines: lab reports or field work for the sciences compared to literature reviews or annotated bibliographies for the social sciences and arts & humanities.   “although students are taught basic information regarding research skills and library resources in english composition courses and the required library course, the continued development of these skills, especially as they relate to the discipline, is crucial to their success in college and beyond.” (dinkleman, 2010, n.p.)   numerical scales preferred   the methods used reflected different indicators of il. scales scoring the presence of assignments that would benefit from il instruction were used most widely. library usage (reserve readings, outside readings) was used as an indicator of il content on some measurements. other studies looked for il concepts identified in acrl standards or framework (for example, critical thinking). standards from educational organizations (aac&u) or professional associations (aacn), were also used to identify il concepts.   il reflects changes over time   il instruction has evolved over time, with more emphasis placed on concepts vs tasks. in 2008, vanscoy and oakleaf showed that tasks like “find a book” or “find an article” were required from the beginning coursework, leading them to conclude that freshman need the same il skills as students in advanced classes. eight years later, the acrl framework for il (2016) addressed threshold learning and skill mastery as a student progresses through courses. the assumption is that basic il concepts are taught in undergraduate core classes and advanced research concepts are mastered at higher academic levels. dubicki’s study (2019) argued strongly for building on il instruction through course progression. the conflict between these 2 papers, published 11 years apart, are a reflection of changing views on il instruction.   “this research study revealed that a tiered approach can be used effectively to provide library instruction as students move along the continuum from novice to expert researchers.” (dubicki, 2019, p. 296) “the study results suggest that many early recommendations regarding tiered instructional approaches should be reexamined.” (vanscoy & oakleaf, 2008, p. 572)   mismatch between librarian involvement and il indicators found in syllabi    all studies found elements of information literacy in the syllabus. some studies found il was defined in terms of tasks or activities (find peer-reviewed articles); other studies found il was identified by concepts (plagiarism, critical appraisal of articles) that aligned with student learning objectives of the instructor and institution.   the studies that compared il mentioned in syllabi to librarian involvement in classes showed that librarians were not providing il instruction or librarian presence in a large percentage of courses. results from jeffrey et al., 2017, detailed an 18% gap between library instruction and research assignments in syllabi. several possible reasons were posited by authors: instructors assume students have il competence; are unaware of librarian il instruction or contributions; or instructors don’t have time in the course to use librarian services. smith et al. (2012) and alcock and rose, (2016) found examples that showed faculty are teaching il in courses.    “in the absence of library instruction, 34% of the courses suggested that the professors were taking on a type of library instruction to ensure that students had the skills to successfully complete.” (alcock & rose, 2016, p. 92)   methodology for studies was often unclear   the methodologies used by the different researchers varied and frequently did not provide key details. the number of syllabi used was not always reported. the use of specific scoring tools was reported, but studies using question checklists, searching of keyword lists developed for the study or other methods of analysis were not described sufficiently. however, one study (maybee et al., 2015) used a grounded theory research design, clearly outlined, resulting in a comprehensive examination of differences between political science and nutrition studies at the undergraduate and graduate level. the use of this research design allowed researchers to analyze in depth the il requirements at different stages of curriculum.   studies are replicable but results are not generalizable   although some studies tried to compare results to other published studies, the numerous and inconsistent variables make that difficult to accomplish. too many variables (slos unique to campus, syllabi templates, individual instructors, unique content, number of syllabi, discipline differences) in these studies make the results unique to a campus, instructor, or discipline and not generalizable. “although this project may provide insights on how the framework concepts can be infused into il instruction, the results are unique to the xx curriculum.” (dubicki, 2019, p. 296)   however, the methods of syllabi mining used in these studies are easily replicable and do offer liaison librarians strategic ways to connect to instructors and the curriculum.   limitations   this scoping review includes several limitations. the searches executed excluded grey literature and did not consider poster presentations and conference proceedings. two reviewers charted data independently, but a third reviewer would have helped to limit bias and make decisions on inclusion and exclusion questions. the multidisciplinary database, web of science, was used because the authors had access, but other large databases should be considered.   conclusions   the results of this scoping review show il concepts and assignments are present in approximately half of syllabi examined in these studies. the presence of il competencies in syllabi was more dependent on discipline (arts and humanities vs science) than on class standing (lower vs upper or graduate vs undergraduate). librarian researchers used syllabi studies to examine what kinds of il instruction are needed by students to complete coursework successfully.   the early methods for assessing syllabi for il content looked for any mention of library use, but with the promotion of acrl standards and more recently the il framework, the evaluation of what concepts and assignments meet the criteria for il is better identified. this provides a structure and continuity to the research that will be easier to replicate and interpret. converting the il framework into rubrics or checklists is the method used in many of the existing studies, and provides a blueprint for future research.   il standards from professional associations were central to many studies, and further research aligning acrl frames, discipline standards and course syllabi will help to integrate instruction. three articles mentioned data literacy, which is becoming central to a scholar’s education in the increasingly networked research world (shorish, 2015). more research on inclusion of data literacy as part of il instruction, and as it is reflected in course syllabi, is warranted.   even though the results of a syllabus study are not generalizable, the methods can be consistent and provide valuable information to an il program. overwhelmingly, the studies provided recommendations for using a syllabus study for better collaboration with instructors. the information derived from these studies present talking points to use with instructors and allow the il instruction to be relevant and targeted at the point of need of the student.   liaison librarians should be encouraged to conduct syllabus studies to increase collaboration with instructional faculty, to meet student information needs in a strategic manner, and to identify discipline-specific information literacy concepts.   references   ache healthcare executive competencies assessment tool. (2020). american college of healthcare executives. https://www.ache.org/-/media/ache/career-resource-center/competencies_booklet.pdf   american association of colleges & universities (n.d.) information literacy value rubric. https://www.aacu.org/value/rubrics/information-literacy   arksey, h. and o'malley,l. (2005). scoping studies: towards a methodological framework. international journal of social research methodology, 8(1), 1932. https://doi.org/10.1080/1364557032000119616   albers, c. (2003). using the syllabus to document the scholarship of teaching. teaching sociology, 31(1), 60-72. https://doi.org/10.2307/3211425   dewald, n.h. (2003). anticipating library use by business students: the uses of a syllabus study. research strategies, 19(1), 33–45. https://doi.org/10.1016/j.resstr.2003.09.003   jacobsen, t.e. and mark, b.l. (2001). separating wheat from chaff: helping first-year                students become information savvy. journal of general education, 50(4), 323. https://doi.org/10.1353/jge.2001.0025   lauer, j. d. (1989). what syllabi reveal about library use: a comparative look at two private academic institutions. research strategies, 7(4), 167–174.   liberati, a., altman, d. g., tetzlaff, j., mulrow, c., gøtzsche, p. c., ioannidis, j. p. a., clarke, m., devereaux, p. j., kleijnen, j., & moher, d. (2009). the prisma statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. annals of internal medicine, 151(4), w65–w94. https://doi.org/10.7326/m14-2385   lloyd, a., & williamson, k. (2008). towards an understanding of information literacy in context: implications for research. journal of librarianship and information science, 40(1), 3–12. https://doi.org/10.1177/0961000607086616   lukes, r., thorpe, a., & lesher, m. (2017). using course syllabi to develop collections and assess library service integration. the serials librarian, 72(1-4), 73–76. https://doi.org/10.1080/0361526x.2017.1284492     miller, m., & neyer, l. (2016). mapping information literacy and written communication outcomes in an undergraduate nursing curriculum. pennsylvania libraries: research & practice, 4(1), 22–34. https://doi.org/10.5195/palrap.2016.121   o’hanlon, n. (2007). information literacy in the university curriculum: challenges for outcomes assessment. portal: libraries and the academy, 7(2), 169–189. https://doi.org/10.1353/pla.2007.0021   peters, m.d., godfrey, c.m., khalil, h., mcinerney, p., parker, d., and soares, c.b. (2015). guidance for conducting systematic scoping reviews. international journal of evidence based healthcare, 13(3),141–146. https://doi.org/10.1097/xeb.0000000000000050   rambler, l. k. (1982). syllabus study: key to a responsive academic library. journal of academic librarianship, 8(3), 155–159.   shorish, y. (2015). data information literacy and undergraduates: a critical competency. college & undergraduate libraries, 22(1), 97–106. https://doi.org/10.1080/10691316.2015.1001246   tricco, a. c., lillie, e., zarin, w., o’brien, k. k., colquhoun, h., levac, d., moher, d., peters, m. d. j., horsley, t., weeks, l., hempel, s., akl, e. a., chang, c., mcgowan, j., stewart, l., hartling, l., aldcroft, a., wilson, m. g., garritty, c., straus, s. e. (2018). prisma extension for scoping reviews (prisma-scr): checklist and explanation. annals of internal medicine, 169(7), 467–473. https://doi.org/10.7326/m18-0850     appendix articles included in scoping review   alcock, e., & rose, k. (2016). find the gap: evaluating library instruction reach using syllabi. journal of information literacy, 10(1), 86. https://doi.org/10.11645/10.1.2038   beuoy, m., & boss, k. (2019). revealing instruction opportunities: a framework-based rubric for syllabus analysis. reference services review, 47(2), 151–168. https://doi.org/10.1108/rsr-11-2018-0072   boss, k., & drabinski, e. (2014). evidence-based instruction integration: a syllabus analysis project. reference services review, 42(2), 263–276. https://doi.org/10.1108/rsr-07-2013-0038   dewald, n. h. (2003). anticipating library use by business students: the uses of a syllabus study. research strategies, 19(1), 33–45. https://doi.org/10.1016/j.resstr.2003.09.003   dinkelman, a. l. (2010). using course syllabi to assess research expectations of biology majors: implications for further development of information literacy skills in the curriculum. issues in science & technology librarianship, (60), 8–8.   dubicki, e. (2019). mapping curriculum learning outcomes to acrl’s framework threshold concepts: a syllabus study. the journal of academic librarianship, 45(3), 288–298. https://doi.org/10.1016/j.acalib.2019.04.003   jeffery, k. m., houk, k. m., nielsen, j. m., & wong-welch, j. m. (2017). digging in the mines: mining course syllabi in search of the library. evidence based library and information practice, 12(1), 72. https://doi.org/10.18438/b8gp81   lowry, l. (2012). accounting students, library use, and information competence: evidence from course syllabi and professional accounting association competency maps. journal of business & finance librarianship, 17(2), 117–132. https://doi.org/10.1080/08963568.2012.659238   maybee, c., carlson, j., slebodnik, m., & chapman, b. (2015). “it’s in the syllabus”: identifying information literacy and data information literacy opportunities using a grounded theory approach. the journal of academic librarianship, 41(4), 369–376. https://doi.org/10.1016/j.acalib.2015.05.009   mcgowan, b., gonzalez, m., & stanny, c. j. (2016). what do undergraduate course syllabi say about information literacy portal: libraries and the academy, 16(3), 599–617. https://doi.org/10.1353/pla.2016.0040   miller, m., & neyer, l. (2016). mapping information literacy and written communication outcomes in an undergraduate nursing curriculum. pennsylvania libraries: research & practice, 4(1), 22–34. https://doi.org/10.5195/palrap.2016.121   morris, s., mykytiuk, l. j., & weiner, s. a. (2014). archival literacy for history students: identifying faculty expectations of archival research skills. american archivist, 77(2), 394–424. https://doi.org/10.17723/aarc.77.2.j270637g8q11p460   o’hanlon, n. (2007). information literacy in the university curriculum: challenges for outcomes assessment. portal: libraries and the academy, 7(2), 169–189. https://doi.org/10.1353/pla.2007.0021   smith, c., doversberger, l., jones, s., ladwig, p., parker, j., & pietraszewski, b. (2012). using course syllabi to uncover opportunities for curriculum-integrated instruction. reference & user services quarterly, 51(3), 263–271. https://doi.org/10.5860/rusq.51n3.263   stanny, c., gonzalez, m., & mcgowan, b. (2015). assessing the culture of teaching and learning through a syllabus review. assessment & evaluation in higher education, 40(7), 898–913. https://doi.org/10.1080/02602938.2014.956684   vanscoy, a., & oakleaf, m. j. (2008). evidence vs. anecdote: using syllabi to plan curriculum-integrated information literacy instruction. college & research libraries, 69(6), 566–575. https://doi.org/10.5860/crl.69.6.566   willingham-mclain, l. (2011). using a university-wide syllabus study to examine learning outcomes and assessment. journal of faculty development 25(1), 43-51. https://eric.ed.gov/?id=ej975164       research article   information literacy and retention: a case study of the value of the library   amy catalano associate professor joan and donald axinn library hofstra university hempstead, new york, united states of america email: amy.catalano@hofstra.edu   sharon r. phillips assistant professor department of specialized programs in education hofstra university hempstead, new york, united states of america email: sharon.r.phillips@hofstra.edu   received: 29 jun 2016     accepted: 22 oct. 2016         2016 catalano and phillips. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective the authors investigated the impact of library instruction on information literacy (il) skills as part of acrl’s aia initiative. additionally, the researchers sought to determine whether there was a relationship between il tests scores and research experiences with student success outcomes such as retention.   methods the researchers administered a standardized il test to 455 graduate and undergraduate students in multiple disciplines. they then collected outcome data on gpa, retention, and graduation three years later.   results while there were no significant differences between those students who had instruction and those who did not on the il test, a regression analysis revealed that experience writing research papers that required library resources and an individual’s use of library books throughout their academic career demonstrated significant, positive relationships with whether a student passed the information literacy test. additionally, using the longitudinal data on gpa, retention, graduation, and employment, the researchers found that students’ il scores were significantly correlated with their gpas, and that students who passed the il test were more likely to be retained or graduate within six years.   conclusion the ability to demonstrate il skills appears to contribute to retention and graduation and, therefore, may be an integral part of one’s academic success. further, experience writing research papers and other meaningful assignments contributes to student success.     introduction   this article reports the results of a study conducted as part of the first cohort of the association of college and research libraries (acrl) assessment in action (aia) initiative, in which librarians from 75 institutions in north america sought to develop and implement research projects that investigated the value of academic libraries. aia is a three-year program funded by an institute of museum and library services grant. the program offered an instructional program that took place both online and in person. the purpose of the instruction was to foster communities of inquiry between other librarians in the cohort and to develop, implement, and report upon a research plan that would investigate the value of academic libraries. oakleaf (association of college & research libraries [acrl], 2010) set forth a multifaceted agenda for exploring and demonstrating the value of academic libraries. many of the research projects conducted by aia participants were based on oakleaf’s recommendations. examples of projects included investigations of the impact of information literacy (il) programs, library use in a very general sense, and space use, just to name a few (ackermann, 2016). outcomes of these projects ranged from determining the impact of various library-related variables on student achievement such as gpa, course grades, and il test scores, to more broad goals such as impact on retention and graduation. the current study sought to determine information literacy skill levels of students in various disciplines across different student ranks (freshmen to doctoral). analyses were conducted to ascertain factors that contributed to a general level of competency in information literacy. further, three years after the il tests were administered, retention, graduation, and gpa data were collected from institutional research for the students who took the il assessment.   hofstra university regularly assesses student learning, engagement, and satisfaction. further, the university’s commitment to developing information literate students is clearly articulated in its mission statement (hofstra university, 2015). accordingly, the hofstra aia project sought to determine “which factors contribute to information literacy competencies (e.g., library instruction) as well as other outcomes such as retention and graduation.” the team was made up of the first author, who was the librarian team leader, the provost for assessment, the dean of the school of liberal arts, and several teaching faculty members in different disciplines, including the second author. this composition allowed us to get input from stakeholders at multiple levels, which facilitated us in constructing and meeting goals important to the university as a whole.   while information literacy assessment is not new, this research adds to the growing body of evidence that describes which factors affect the degree to which both undergraduate and graduate students develop information literacy competencies. we looked at the impact of various types of library instruction (one-shot and credit-bearing information literacy courses), as well as other factors such as the number of books a student borrowed, majors, and student experiences with writing research papers.                 while information literacy has become the language of librarianship, it is not solely the domain or responsibility of librarians to impart (grafstein, 2002). therefore, the research team took the approach that all faculty are responsible for imparting information literacy, so we did not primarily seek to evaluate the impact of library instruction on il skill acquisition. rather, we investigated the impact of library instruction in concert with other variables that would affect the acquisition of il skills. further, in order to meet the aia program’s larger goal of determining the “value of academic libraries” we used il and book use as two factors (out of many more potential data points) that might contribute to the larger model that affects whether or not a student is retained. toward that end, the literature review is comprised of studies that discuss what aspects of the post-secondary experience impact student retention, with a focus on the recent studies that have been conducted that connect library use to retention, graduation, and student achievement.   literature review   a recent survey of academic library deans reported that over 40% of respondents reported that their libraries support projects specifically addressing student retention. many of those respondents indicated an interest in further understanding the relationship between their own library and the retention rate at their university (hubbard & loos, 2013). while university administrators may look to librarians to support the enrollment and retention of students, and typically institutional priorities and librarians are showing an interest in doing so, there has been little guidance on how librarians can support these initiatives. therefore, the effective practices librarians employ may often be overlooked by both parties (lynch et al., 2007; murray, ireland, & hackathorn, 2016). accordingly, there is a lack of research to help guide librarians in this effort.                 one of the first articles to look at the relationship between retention and college libraries was conducted in 1968. in this article, kramer and kramer (1968) looked at the basic relationship between students who checked out books (an effective measure at that time) and the student dropout rate. this study began the conversation about the importance of research on the relationship between libraries and retention when it discovered that 43% of library non-users left the university after their first year, compared with 26% of library users who left after their first year.    in more recent years the research has evolved from looking at the number of books that have been checked out of the library to investigating other measures of library use and impact including gpa, retention, and graduation. while some research on library use and retention has examined various populations including distance education students, graduate students, and students from different ethnic backgrounds, this literature review focuses on undergraduate student retention within on-campus education which, for the purposes of this study, is defined as the majority of the courses within the degree program being taught in an on-campus setting. retention is defined as ongoing enrollment in the institution (mezick, 2007). the literature regarding the connection of library use to student retention and graduation rates touches upon a multitude of factors and draws from varied contexts. the range is from detailed and nuanced initiatives to large scale university efforts to retain students and to use the library as a resource to do so.   it has been argued that having librarian-student relationships embedded within a student’s education can positively impact retention (pagowsky & hammond, 2012). this suggestion has been supported by research that implies that when librarians proactively offer support to students, there is a positive correlation with retention rates (hagel, horn, owen, & currie, 2012). this includes offering first year orientation programs and creating strategic partnerships with student support services and faculty (blackburn, 2010; grallo, chalmers, & baker, 2012; hagel et al., 2012; soria, fransen, & nackerud, 2013). further supporting this notion is research suggesting that library expenditures, including the number of library staff, library salaries and staffing, book acquisitions, and library instruction also play a positive role in retention; the higher the expenditure, the higher the retention rate (emmons & wilkinson, 2011; mezick, 2007; teske, dicarlo, & cahoy, 2013).   research looking at retention and library services from a different angle, library usage, rather than programs designed to be proactive, has also been conducted. the number of items a student borrows and the number of log-ins to workstations, catalogs, databases, and electronic resources have been shown to have a positive correlation with retention. this suggests that students who utilize library resources are more likely to be retained and potentially graduate (haddow, 2013; haddow & joseph, 2010; soria et al., 2013; stone & ramsden, 2013). along the same lines, research suggests that library usage within the first few weeks of the semester results in a greater chance of retention (haddow & joseph, 2010). that being said, the correlation between library expenses per full time equivalent student and both graduation rates and retention is actually stronger in comparison to the correlation between library usage and retention rates (crawford, 2015). these results indicate that the relationship between expenditures and student success should be explored further.   interestingly, and expectedly, having a connection to the university and student academic success are suggested to be related to retention rates. the academic and social supports offered by libraries as well as library work-study have been examined and linked with students’ connection to the university (mezick, 2007; rushing & poole, 2002; wilder, 1990). library use has been positively correlated with academic success in the first semester (soria et al., 2013). this link was first discovered in 1986 when hiscock (1986) looked at student use of catalog and reference material and found a link between use of those resources and academic success. furthermore, library instruction and information literacy skills have been related with student academic success (bowles-terry, 2012; breivik, 1977; mark & boruff-jones, 2003; mezick, 2007). instruction on how to use the library for first-time full-time students has become a predictor of academic success in the first two semesters (gammell, allen, & banach, 2012; mezick, 2007). these findings relate back to the previous statement that much of what the library does that connects to student retention is overlooked as it is only one factor among the myriad factors related to retention, including academic achievement and feelings of connectedness to the university.   although research on libraries and retention has grown vastly in recent years, few of these studies actually attempt to connect il with retention. further, those who do examine library instruction or il have not assessed il skills with a large sample using a standardized measure as described in the current study. this research is further strengthened by using longitudinal outcome data, three years after the il test was administered, in order to obtain a fuller picture of relationships between student success variables, il skills, and il instruction.   methods   during the planning stage of this study, the aia team investigated various il tests, both free and fee-based. these tests included project sails (standardized assessment of information literacy skills), a validated and commercially available il test developed at kent state university (2015), as well as iskills, a test developed by the educational testing service (ets) that assesses students on how they evaluate, create, define, synthesize, and use different types of information via scenario-based problems and tasks (ets iskills, n.d.). the costs of these two assessments precluded their use. therefore, the researchers adapted the beile test of information literacy for educators (b-tiled) to be non-subject specific, with beile o’neil’s permission (beile o’neil, 2005). we adapted the test by removing questions that referred to resources that only education students would use, and replacing those references with more general sources. for example, one question referred to the eric database. we changed this question to refer to a multidisciplinary database. questions on the test that referred to education topics (e.g., special needs, higher education, vygotsky) were left unchanged as their presence did not alter the ability of a student in any major to answer the question. the b-tiled is a 22 item test with 13 additional demographic and self-perception items. the test covers four of the five il standards as articulated by acrl’s information literacy competency standards for higher education (association of college & research libraries [acrl], 2000), and was originally a part of the project sails initiative. the only standard not assessed is standard 4, which asks the student to demonstrate that they are able to use information effectively. in beile o’neil’s own psychometric analysis of the test given to 172 education students, the instrument demonstrated reasonable validity and reliability. reliability and validity results on the current sample are reported in the results below.   we utilized several sampling methods in order to receive higher returns on the test. first, all members of the aia team, who teach in distinctly different disciplines (geology, psychology, education, physical education and sports sciences, philosophy, and health sciences) administered the test to all the students in all of their classes (two classes per faculty member). students were offered extra credit for participation, which resulted in a 95% participation rate. while these cluster samples were not random, they did reduce the problematic bias that comes when tests are completed only by students who are inclined to answer surveys, and who are often good students. the second sampling strategy involved asking the administrators of each school at the university if they would send the survey to their faculty in order to be administered to entire classes (and therefore use cluster sampling strategy). the school of health sciences agreed with several faculty participating, resulting in a participation rate of about 90%. these students also received extra credit. the school of education agreed to send the survey to all education students directly instead of through faculty/classes. out of 1,250 education students (graduate and undergraduate) 123 completed the survey (a 10% response rate). these students were offered a chance to win a $100 gift card. data collection took place over the spring and fall semesters of 2013. these methods are also reported in ackermann (2016).   in addition to assessing il skills, the questionnaire also asked about experiences with research papers, types of library instruction received, and whether the researchers could look up other outcome data and the number of books taken out at a later date. three hundred and three participants provided us with their student identification numbers and consent. at the end of the fall 2015 semester the university’s institutional research office provided the researchers with one year retention data, four, five, and six year graduation data, as well as the gpa for each participant. data were collected using qualtrix survey software. because the test is somewhat lengthy, there were concerns about whether students were rushing through the test and answering questions randomly just to get extra credit. since qualtrix times how long it takes each respondent to complete a survey, we used this information to remove students from the sample who took less than five minutes to complete the test.   data analysis   after descriptive statistics were computed, treatment and control groups were created to determine if there were significant differences between students who received some type of library instruction and those who did not. an ancova was run on the data using spss version 21. we also examined the impact of the additional variables listed above using post hoc analyses. regression analyses were also run to determine which factors contributed to the il scores, and which factors contributed to retention, graduation, and student gpa.   results   participants   the il test was completed by 456 students, though 32 of these tests were unusable as they were incomplete or had been completed too quickly to be valid. participation was evenly distributed among different class ranks except for freshman (n = 44). the il test was taken by 85 sophomores, 100 juniors, 103 seniors, and 124 graduate students, sixteen of whom were doctoral students. english language learners comprised 11% of respondents, and responded “yes” when asked if they communicate better in another language. most of the participants were female (n =326), while 29 % were male.   reliability of the adapted b-tiled   the test demonstrated adequate reliability of a cronbach’s alpha of 0.647. beile o’neil’s analysis (2005) revealed a similar value using kuder-richardson 0.675. since reliability is a measure of how consistently participants will perform on a test, the reliability coefficient demonstrates how consistently an instrument measures a construct. reliability, however, is not to be confused with validity, as a test can be valid but not reliable, and vice versa. generally, the value of a coefficient can be interpreted as follows: .5 or below is considered unacceptable, .65-.8 is the minimum acceptable value, and .9 or above is considered excellent, although some methodologist may employ other parameters to gauge reliability.   instruction   of all of the respondents, 65 had taken library 001, a single credit information literacy course, while 260 students had at least one experience with “one-shot” library instruction. a total of 274 (60%) students had some kind of library instruction, either the credit bearing course or the “one-shot” or both, during their post-secondary education. the cut score (passing score), according to beile o’neil (2005) is 55%, which allows for a lot of error. using this cut score, the results revealed that nearly half of the participants (n = 229) passed the test. the mean score was 53.91, the standard deviation was 16.12, and the range of scores was 13.04 to 95.66.   a one-way analysis of variance test did not reveal any significant differences between students who had received any type of instruction and those who had none, f (1, 452) = .124, p = .725. students who had no instruction had a mean score of 54.21, while students who had instruction had a mean score of 53.67. however, students who had instruction had higher mean scores if they had been assigned research papers over eight pages long at some point during their post-secondary education. mean scores for students assigned research papers were 62 for students who had instruction, and 59 for those who had never had library instruction. additionally, il scores go up steadily as the page count increases in papers written (e.g., 1-5, 6-8, and 8-15 pages). see figure 1.     figure 1 mean score on il test and the number of research papers assigned.     to further analyze the factors that contributed to passing the il test, a logistic regression was employed in which two groups were created: students who passed the il test and those who did not. the variables input into the model were number of books borrowed, experiences with research papers, number of papers written, type of library instruction, major, and class rank. only two of the variables significantly contributed to passing the il test: books borrowed and experiences with research papers (chi square = 15.245, p <.05, with df = 4). these two variables explained between 3.6% and 4.8% of the variance in the model.   majors   participants from 13 majors took the tests, although most were in education (n =149) or the health sciences (n = 153). see figure 2. there were differences in il scores between majors, however, the sample sizes for each major were too diverse to run an anova. when looking at il scores as delineated by major, students who had instruction got higher mean scores on the test (except business, history, philosophy, and science, which had smaller samples of students). psychology students demonstrated a three-point difference between those who had instruction and those who did not, while sports science majors demonstrated a six-point difference. the average score for education majors was 55.9, and for health science majors the average score was 53.9. although there were not enough participants in each major to make a fair or reliable comparison between disciplines, the highest average scores of 68.8 were achieved by philosophy majors who had no instruction (n =6). it is difficult to draw conclusions about the relationship between majors and il scores based on these results as participants were students of a variety of professors in each of the disciplines.   graduate scores vs undergraduate scores   graduate students scored significantly higher than undergraduate students on the il test p<.0001 [f(1,292) = 21.44], with a mean score of 62 for graduate students as compared to a mean score of 52 for undergraduates. it is important to note, however, that the group sizes were not equivalent as there were 232 undergraduates and only 62 graduates. graduate students and undergraduate students wrote the same proportion of papers consisting of six pages in length, while 50% of papers written by both graduate students and undergraduate students were longer than six pages. when breaking down the number of students in each class standing who wrote papers longer than six pages, unexpected results emerge: 68% of freshmen report writing papers longer than 6 pages, followed by 55% of sophomores, 50% of graduate students, and 43% of both juniors and seniors. although il scores increased as the number of pages assigned increased, graduate students (who were assigned longer papers less frequently than freshmen) scored significantly higher on the il test than undergraduates.   retention, graduation, and gpa   out of the 455 students who took the il test, 328 gave the researchers permission to look up their outcome data (retention, graduation, and gpa) in the future from the institutional research office. in this study we excluded graduate students from the examination of retention and graduation. because graduate students have, by definition, already graduated with a degree, the inclusion of this population muddies the data. further, because almost all retention studies focus on undergraduate students exclusively, we sought to also focus on this group of students. for those students retained, but not graduated, it was assumed that they were not yet eligible for graduation, and a category of “retained” was created. if a student was not retained, they too, were not eligible for graduation. this resulted in the examination of 294 cases for this part of the analysis. in order to avoid attributing dual statuses to students (not retained and not graduated) and confounding the data, we created one category named “retained or graduated.” from this category we were able to define student success broadly. with the exception of the il scores and gpa, all variables were categorical.   gpa   an anova was employed to determine whether there were differences in gpa between those who had library instruction and those who did not. no significant differences were found. however, an initial correlational analysis revealed that il score and gpa were moderately significantly correlated (r = 0.392, p<0.01).   retention and graduation   since the categories of being “retained or graduated” and “retained” (yes/no) were categorized as binomial variables, a logistic regression was run to determine whether the number of books borrowed, instruction (yes/no), il score, passing/failing the il test, whether students wrote research papers that required library resources, or wrote research papers of varying lengths (under 5, 5-8, and 9-15 pages) predicted either of these outcome variables. a separate logistic regression was run for each of the dependent variables. it is important to note that the term “predicted” is associated with the use of regression tests. the use of this term does not indicate that there is a causal relationship between two variables.   for the outcome variable graduated or retained, 294 cases were analyzed. passing the il test, and writing research papers predicted whether or not a student was graduated or retained (chi-square = 4.324, p < .05 df = 1). these two variables contributed to 1.5%-2.1% of the variance, which indicates that they contribute positively, but on a small scale, to the outcome variable graduation and retention.   for the test that analyzed undergraduates who had been retained, but not yet graduated, all variables were again input into the model. one hundred and thirty-eight cases were analyzed. writing a research paper under five pages long or between five and eight pages significantly predicted retention (chi-square = 18.613, p < .05 df = 7), explaining 13% to 17% of the variance. these results reveal that writing papers moderately contributes to student retention. it is possible that other variables not included in these tests interact with the act of writing papers and how it influences retention.   discussion   the il test assessed the information literacy skills of 424 students. in addition to information literacy scores, the researchers also collected data on book use, experiences writing research papers, gpa, retention, and graduation (for a portion of the sample). the data analysis revealed that while library instruction did not significantly impact competency at information literacy skills, experiences with research papers requiring library resources and use of the library’s book collection contributed significantly to a student’s il test score. further, information literacy and writing research papers contributes significantly to retention, graduation, and gpa. the results of our project indicate that library instruction (whether one-shot or credit bearing) should be coupled with meaningful assignments requiring sustained engagement with library resources. additionally, the value of writing research papers with respect to student success is one that should be examined further given the results of this research. in this study information literacy scores went up as the number of pages a student was assigned to write increased. while some disciplines lend themselves to research papers requiring library resources more so than others, written communication skills are essential to career readiness and should be an integral part of a student’s college education.   limitations   other studies noted in the literature review investigated library use beyond book circulation, to include database use, card swipe data to indicate use of the physical space, and other indicators. although these data are not available at the authors’ institution, they would have added another layer of information to explain our results. we hypothesized that perhaps it is more use of the library itself rather than book borrowing that predicts il skills, although book checkouts do provide a small window into library user behaviour. additionally, as information literacy is often integrated in some way into the curriculum of most disciplines, it is difficult to ascertain whether library instruction really makes a difference in student acquisition of these skills. when designing this study, the aia team discussed how some disciplines approach information literacy skills more explicitly than others, and therefore the results of this study might be skewed in favour of the social sciences. although  the assessment we used was a standardized instrument, the limited nature of a multiple choice test for measuring il competency prohibits us from drawing strong inferences about the relationship between a student’s il score and their actual il competency or the other outcome variables, particularly because the il test does not assess the ability to use information effectively. although gpa and il scores were positively correlated, it is reasonable to conclude that students who do well academically would also do well on an il test regardless of library or writing experiences. it is very possible that there are other confounding factors that we did not include in these analyses that contribute to student success.   implications for future research   as noted earlier, in planning this study, the researchers investigated the merits of several il tests. a small grant was obtained by the primary author to purchase 40 licenses of the iskills test, which is useful for assessing career readiness and work place/real life information literacy skills, however the iskills assessment will be discontinued as of december 31st 2016. other means of evaluating information literacy in different disciplines should also be considered. with the incorporation of the acrl framework into the way librarians teach and assess information literacy, it is necessary to conduct future studies with the language and concepts provided in the framework. additionally, while post-secondary institutions are increasingly developing models to ascertain the characteristics of both the student and the school environment that contribute to student retention, library researchers must continue to assess the value of library services, space, and instruction with respect to the impact on retention and graduation. in this way, academic libraries can fully articulate one of the many ways that they serve the larger institution. further, our literature review indicates that institution and library expenditure is related to academic success. this phenomenon should be examined more broadly in order to explain the mechanisms at work as well as the related factors that contribute to student success.   conclusion   on post-secondary campuses, assessment may often be met with wariness or timidity, however, inquiries that will provide investigators with data with which to make improvements to instruction and policy can only benefit stakeholders, and students in particular. additionally, these results have implications for the value of assigning research projects and for instructional design when it comes to library instruction. the results make the case for project-based instruction, particularly in “one-shot” library classes, so that students may experience sustained engagement with research resources and have opportunities to integrate these sources into research projects. this research adds to the growing body of literature that highlights academic libraries as contributors to student retention.   acknowledgments   this project is part of the program “assessment in action: academic libraries and student success” which is undertaken by the association of college and research libraries (acrl) in partnership with the association for institutional research and the association of public and land-grant universities. the program, a cornerstone of acrl's value of academic libraries initiative, is made possible by the institute of museum and library services.   the authors would also like to thank the other aia team members for their participation in the planning and implementation of this project:   references   ackermann, e. (ed.). (2016). putting assessment into action: selected projects from the first cohort of the assessment in action grant. chicago, il: association of college & research libraries.   association of college & research libraries. (2000). information literacy competency standards for higher education. retrieved 27 april 2015 from http://www.ala.org/acrl/standards/informationliteracycompetency   association of college & research libraries. (2010). value of academic libraries: a comprehensive research review and report. prepared by megan oakleaf. chicago, il: association of college and research libraries. retrieved from http://www.ala.org/acrl/sites/ala.org.acrl/files/content/issues/value/val_report.pdf   beile o’neil, p. m. (2005). development and validation of the beile test of information literacy for education (b-tiled) (unpublished doctoral dissertation). university of central florida orlando, florida. retrieved from http://etd.fcla.edu/cf/cfe0000749/beileoneil_penny_m_200512_phd.pdf   blackburn, h. (2010). shhh! no talking about retention in the library!. education libraries, 33(1), 24–30. retrieved from http://eric.ed.gov/?id=ej887231   bowles-terry, m. (2012). library instruction and academic success: a mixed-methods assessment of a library instruction program. evidence based library and information practice, 7(1), 82-95.   breivik, p.s. (1977). resources: the fourth r. community college frontiers, 5(2), 46–50.   crawford, g.a. (2015). the academic library and student retention and graduation: an exploratory study. libraries and the academy, 15(1), 41-57. http://dx.doi.org/10.1353/pla.2015.0003   emmons, m., & wilkinson, f.c. (2011). the academic library impact on student persistence. college & research libraries, 72(2), 128-149. doi: 10.5860/crl-74r1   ets iskills. (n.d.). retrieved march 1, 2016, from https://www.ets.org/iskills/about#   gammell, w.j., allen, g.j., & banach, p.s. (2012). leveraging existing data: indicators of engagement as early predictors of student retention. in j.a. larusson (ed.), nerla 2012: proceedings of the first north east regional learning analytics symposium. (pp. 9–17). waltham, ma: nerla.   grallo, j.d., chalmers, m., & baker, p. g. (2012). how do i get a campus id? the other role of the academic library in student retention and success. the reference librarian, 53(2), 182–193. http://dx.doi.org/10.1080/02763877.2011.618787   grafstein, a. (2002). a discipline-based approach to information literacy. the journal of academic librarianship, 28(4), 197–204. http://dx.doi.org/10.1016/s0099-1333(02)00283-5   haddow, g. (2013). academic library use and student retention: a quantitative analysis. library & information science research, 35, 127-136. http://dx.doi.org/10.1016/j.lisr.2012.12.002   haddow, g., & joseph, j. (2010). loans, logins, and lasting the course: academic library use and student retention. australian academic and research libraries, 41(4), 233-244. http://dx.doi.org/10.1080/00048623.2010.10721478   hagel, p., horn, a., owen, s., & currie, m. (2012). how can we help? the contribution of university libraries to student retention. australian academic and research libraries, 43(3), 214-230. http://dx.doi.org/10.1080/00048623.2012.10722278   hiscock, j. e. (1986). does library usage affect academic performance? australian academic and research libraries, 17(4), 207–214.   hofstra university (n.d.). mission statement. retrieved march 1, 2016 from http://www.hofstra.edu/about/about_mission.html   hubbard, m. a., & loos, a. t. (2013). academic library participation in recruitment and retention initiatives. reference services review, 41(2), 157181. http://dx.doi.org/10.1108/00907321311326183   kent state university. (2000-2015). project sails (standardized assessment of information literacy skills). retrieved from https://www.projectsails.or g   kramer, l. a., & kramer, m. b. (1968). the college library and the drop-out. college & research libraries, 29(4), 310312.   lynch, b. p., murrary-rust, c., parker, s. e., turner, d., walker, d. p., wilkinson, f. c., & zimmerman, j. (2007). attitudes of presidents and provosts on the university library. college & research libraries, 68(3), 213227. http://dx.doi.org/10.5860/crl.68.3.213   mark, a. e., & boruff-jones, p. d. (2003). information literacy and student engagement: what the national survey of student engagement reveals about your campus. college & research libraries, 64(6), 480–493. http://dx.doi.org/10.5860/crl.64.6.480   mezick, e. m. (2007). return on investment: libraries and student retention. journal of academic librarianship, 33(5), 561–566. http://dx.doi.org/10.1016/j.acalib.2007.05.002   murray, a., ireland, a., & hackathorn, j. (2016). the value of academic libraries: library services as a predictor of student retention. college & research libraries, 77(5), 631-642. http://dx.doi.org/10.5860/crl.77.5.631   pagowski, n. & hammond, j. (2012). a programmatic approach: systematically tying the library to student retention efforts on campus. college & research libraries news 73(10), 582-585, 594. retrieved from http://crln.acrl.org/content/73/10/582.full   rushing, d., & poole, d. (2002). the role of the library in student retention. in m.c. kelly, & a. kross (eds.), making the grade: academic libraries and student success (pp. 91–101). chicago: association of college & research libraries.   soria, k. m., fransen, j., & nackerud, s. (2013). library use and undergraduate student outcomes: new evidence for students’ retention and academic success. portal: libraries and the academy, 13(2) 147164. http://dx.doi.org/10.1353/pla.2013.0010   stone, g., & ramsden, b. (2013). library impact data project: looking for the link between library usage and student attainment. college & research libraries, 74(6), 546-559. http://dx.doi.org/10.5860/crl12-406   teske, b., dicarlo, m., & cahoy, d. (2013). libraries and student persistence at southern colleges and universities. reference services review 41(2), 266-279. http://dx.doi.org/10.1108/00907321311326174   wilder, s. (1990). library jobs and student retention. college & research libraries news, 51(11), 1035–1038.   editorial evidence based library and information practice 2009, 4:3     evidence based library and information practice         editorial    stockholm shines    denise koufogiannakis  editor‐in‐chief  collections & acquisitions coordinator, university of alberta libraries  edmonton, alberta, canada  email: denise.koufogiannakis@ualberta.ca    © 2009 koufogiannakis. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      i didn’t sleep much in stockholm. the eight  hour time difference and the jet lag  completely messed me up, not to mention  the 19 hours of daylight! but during the day  when i was so tired, the content being  presented at the conference and the people i  met kept me buzzing and wide awake! it is  always nice to find a group of people who  are interested in the same things as you are,  and to do so in a beautiful setting with great  food and a friendly environment is all the  better.  in my opinion, the 5th eblip  conference was the best yet! i returned to  canada inspired and intellectually  stimulated.     the eblip conference is one of the few  times that those of us who work on the  eblip journal have a chance to get together  and talk in person. our journal editors are  situated in the u.k., canada, and the united  states, so our work is done primarily via e‐ mail and the occasional conference call.   three of us made it to stockholm (alison,  lorie and myself), so it was great to touch  base in person and have a drink or two  together! we also got to see several  members of the editorial advisory board, as  well as those who write evidence summaries  for us, or are involved in some other way.  meeting some of these people in person for  the first time after a lengthy e‐mail  relationship is quite a treat!      the editors took the opportunity at the  conference to hold a user group meeting,  inviting anyone who was interested to  attend and provide us with feedback about  the journal. to start we provided an update,  which included the fact that the journal has  over 2000 registered readers. did you know  that in just one month (may 25 – june 24,  2009) we had 4649 site visits; 3,678 of them  unique? most visits to the journal are from  the united states, followed by canada,  australia, and the u.k. but we have people  accessing the journal from all over world,  and now that we are tracking site traffic via  google analytics, it is easy to see on a map  where our readers reside.    there was a good turnout for our user  group, and a lively discussion. people told  us what they valued about the journal, with  1 mailto:denise.koufogiannakis@ualberta.ca evidence based library and information practice 2009, 4:2  much discussion focused on the evidence  summaries. the evidence summaries are  valued as a unique way to learn more about  original research, and a few people noted  that they will show evidence summaries to  senior managers when the summary focuses  on an aspect of practice that is being  considered at their institution.  one thing  that came up time and time again was how  to get more international content into the  journal.  as a result, we are going to try to  incorporate reviews of non‐english  language research. we also actively  encourage people from anywhere in the  world to submit articles or volunteer to  serve on our editorial advisory board or in  some other capacity.  the user group session  was over before we knew it, but as always, i  encourage all readers to provide us with  feedback about how we can make eblip  better.     in this issue, there are a couple of things  from the conference that i want to highlight.  andrew booth proposes an international  association of eblip in a commentary; this  was the result of an early morning open  meeting held during the conference. as  well, several conference attendees have  provided reflections on their experiences,  which you will find in the news section. for  those who could not attend in person, this  will give you a good sense of what  happened at the conference from different  perspectives. and finally, the first of what  we hope are many papers resulting from  presentations given at eblip5, an article by  michael raynor is available in this issue. in  a future issue we will have a feature  focused on all the keynote presentations that  were given in stockholm.    as the days grow short here in my part of  the world, i look back fondly on the long  days in stockholm this past summer, and  am inspired to keep contributing to this  international eblip community that  continues to question our own models of  practice in an effort to serve our local  communities in the best way we possibly  can.        three editors at the eblip5 conference: denise koufogiannakis, lorie kloda, alison brettle   photo courtesy eblip5 local organizing committee    2 evidence summary   lis practitioner-focused research trends toward open access journals, academic-focused research toward traditional journals   a review of: chang, y-w. (2017). comparative study of characteristics of authors between open access and non-open access journals in library and information science. library & information science research, 39(1), 8-15. http://dx.doi.org/10.1016/j.lisr.2017.01.002   reviewed by: richard hayman associate professor & digital initiatives librarian mount royal university calgary, alberta, canada email: rhayman@mtroyal.ca   received: 28 nov. 2018 accepted: 21 feb. 2018     c 2018 hayman. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29377       abstract    objective – to examine the occupational characteristics and publication habits of library and information science (lis) authors regarding traditional journals and open access journals.   design – content analysis.   setting – english language research articles published in open access (oa) journals and non-open access (non-oa) journals from 2008 to 2013 that are indexed in lis databases.   subjects – the authorship characteristics for 3,472 peer-reviewed articles.   methods – this researcher identified 33 total journals meeting the inclusion criteria by using the lis categories within 2012 journal citation reports (jcr) to find 13 appropriate non-oa journals, and within the directory of open access journals (doaj) to identify 20 appropriate oa journals. they found 1,665 articles by 3,186 authors published in the non-oa journals, and another 1,807 articles by 3,446 authors within the oa journals.   the researcher used author affiliation to determine article authors’ occupations using information included in the articles themselves or by looking for information on the internet, and excluded articles when occupational information could not be located. authors were categorized into four occupational categories: librarians (practitioners), academics (faculty and researchers), students (graduate or undergraduate), and others. using these categories, the author identified 10 different types of collaborations for co-authored articles.   main results – this research involves three primary research questions. the first examined the occupational differences between authors publishing in oa journals versus non-oa journals. academics (faculty and researchers) more commonly published in non-oa journals (58.1%) compared to oa journals (35.6%). the inverse was true for librarian practitioners, who were more likely to publish in oa journals (53.9%) compared to non-oa journals (25.5%). student authors, a combined category that included both graduate and undergraduate students, published more in non-oa journals (10.1%) versus in oa journals (5.0%). the final category of “other” saw only a slight difference between non-oa (6.3%) and oa (5.5%) publication venues.   this second research question explored the difference in the proportion of lis authors who published in oa and non-oa journals. overall, authors were more likely to publish in oa journals (72.4%) vs. non-oa (64.3%). librarians tended to be primary authors in oa journals, while lis academics tend to be primary authors for articles in non-oa publications. academics from outside the lis discipline but contributing to the disciplinary literature were more likely to publish in non-oa journals. regarding trends over time, this research showed a decrease in the percentage of librarian practitioners and “other” authors publishing in oa journals, while academics and students increased their oa contributions rates during the same period.    finally, the research explored whether authors formed different types of collaborations when publishing in oa journals as compared to non-oa journals. when examining co-authorship of articles, just over half of all articles published in oa journals (54.4%) and non-oa journals (53.2%) were co-authored. overall the researcher identified 10 types of collaborative relationships and examined the rates for publishing in oa versus non-oa journals for these relationships. oa journals saw three main relationships, with high levels of collaborations between practitioner librarians (38.6% of collaborations), between librarians and academics (20.5%), and between academics only (18.0%). non-oa journals saw four main relationships, with collaborations between academics appearing most often (34.1%), along with academic-student collaborations (21.5%), practitioner librarian collaborations (15.5%), and librarian-academic collaborations (13.2%).   conclusion – lis practitioner-focused research tends to appear more often in open access journals, while academic-focused researcher tends to appear more often in non-oa journals. these trends also appear in research collaborations, with co-authored works involving librarians appearing more often in oa journals, and collaborations that include academics more likely to appear in non-oa journals.   commentary   this study furthers our understanding of the trends in oa and non-oa publishing, particularly through the examination of occupational collaborations in combination with publication venue. the value of this research is recognizing that authors continue to explore options about where they choose to publish. by identifying collaborations and exploring how the occupational status of authors and collaborators may affect the selection of publication venue, this study goes beyond the typical analysis of comparing publication venue choice based on academic rank, or by marking the distinction between researchers versus practitioners.   this research concludes that traditional journal publishing is not threatened by open access publishing and that “developments in oa publishing have had little effect on most academic authors’ loyalty to traditional journals” (p. 14). since this study’s methods did not measure perceptions or preferences regarding oa and non-oa publications, it does not include reliable evidence to draw any conclusions about authors’ loyalty toward a particular publication model. however, this study does establish evidence that practitioner-based contributions appear more often in oa journals, while researcher-based contributions are more likely to appear in non-oa journals. one implication we can derive from this is that library practitioners should consult both oa and non-oa journals to inform their practice, but do so knowing that oa journals may be more useful venues for discovering practitioner-focused research. these research results may also be informative for researchers and collaborators choosing an appropriate oa or non-oa journal to publish their own research.   the author notes limitations to the study, including the fact that many articles were excluded because author information was unavailable. this brings overall generalizability into question, since there is no way to determine whether excluding these articles resulted in skewed analysis. also missing from this commentary is information about the strengths and weaknesses of using the jcr and doaj to identify the journal sources overall, aside from information about identifying gold oa journals in the doaj. with these limitations acknowledged, this study is otherwise valid using glynn’s (2006) appraisal tool. the selection criteria, data collection, and other methods appear sound and appropriate for this study. the author provides the title lists for both oa and non-oa journals included, aiding replicability. finally, they also point to further areas for research, such as the need to monitor ongoing trends related to authorship, collaborations, and choice of publishing venue. given the time period being studied (2008-2013), an update to this research to include more recent publication and collaboration trends would be meaningful to practitioners and researchers alike.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154   evidence based library and information practice commentary   time to move eblip forward with an organizational lens   alisa howlett coordinator, evidence based practice university of southern queensland library springfield, queensland, australia email: alisa.howlett@usq.edu.au   received: 16 aug. 2018                                                                accepted: 22 aug. 2018      2018 howlett. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     introduction   for more than two decades, the library and information science (lis) profession has considered the ways that “evidence based practice,” a decision-making framework that has its origins in medicine, applies to our own practice. empirical research and conceptual discussion has evolved our understanding, and yet so far, this understanding has only captured the individual practitioner experience of evidence based library and information practice (eblip). it is time to move our understanding forward from being evidence-based practitioners to becoming evidence-based libraries.   if library and information services are to truly embrace a culture of evidence based practice as a “[way] of being,” we need to apply what we know about eblip to a whole-of-organization level (partridge, edwards, & thorpe, 2010, p. 282). this commentary will highlight the current gap in our understanding about eblip but also an opportunity, to enable evidence based practice to serve at a strategic level. this review will be followed by a ‘lens’ or model that is used at the university of southern queensland (usq) library to promote a shared understanding of how evidence-based practice is applied throughout the organization.   understanding evidence based practice in the lis professional context   empirical research has helped us refine our understanding about how evidence based practice applies to professional practice and service delivery. studies in a variety of contexts such as academic libraries, teacher librarianship, public and special libraries have explored conceptualizations and lived experiences of evidence based practice by lis professionals. lis professionals identify and use a variety of evidence types, beyond the research literature (gillespie, miller, partridge, bruce, & howlett, 2017; koufogiannakis, 2012). koufogiannakis grouped sources of evidence into either “hard” or “soft,” where “hard” included sources that could be easily shared, were “written, concrete information” that was generally accepted as evidence, and “soft” included sources that focused on experience or knowledge or others that allowed a “real-life connection” to context (2012, p. 11). this broad interpretation of “evidence” was consistent with findings reported by gillespie et al. (2017), which included observations, feedback and statistics recognised as evidence by practitioners.   experiences of evidence based practice have been found to not always be a straight-forward or linear process, but “holistic” in being evidence based lis professionals (gillespie, 2014; koufogiannakis, 2013; koufogiannakis, 2015; luo, 2018; miller, partridge, bruce, yates, & howlett, 2017). one of the first empirical studies of evidence based practice in lis used a phenomenographic approach to identify five different ways that practitioners experience evidence based practice (partridge et al., 2010). findings of this study captured “dimensions of variation” that influenced how evidence based practice was conceptualised, including the practitioners’ working environment, approaches to evidence based practice and decision making (partridge et al., 2010, p. 283). more recently, researchers have categorized academic librarians’ experiences of evidence based practice into six categories: “[e]mpowering; [i]ntuiting; [a]ffirming;[c]onnecting; [n]oticing; and, [i]mpacting” (miller et al., 2017, p. 126). the model developed by miller et al. (2017) raises awareness of how these different experiences of evidence based practice contribute to lis professional practice.   experiences of evidence-based practice are highly contextual, and are influenced by workplace dynamics such as accessibility, availability, time, and organizational culture (howlett & howard, 2015; koufogiannakis, 2015; luo, 2018). exploring evidence use in practice, koufogiannakis (2013, p. 1) found “convincing” was the main way in which academic librarians use evidence. how evidence is used, whether to confirm existing knowledge or a way forward, or to influence a decision is dependent on the working environment, including how much decision-making power individual academic librarians held to make improvements to practice (koufogiannakis, 2013).   the biggest challenges and influences faced by academic librarians in using evidence in practice are in the ‘organisational dynamics’ (2015, p. 104). collaboration, support from colleagues and a culture that values evidence were enablers found to help overcome obstacles in the working environment (koufogiannakis, 2015, p. 109). findings from koufogiannakis’ study begin to describe a complex picture of the day-to-day realities of applying evidence-based practice. a better understanding of workplace dynamics would assist libraries to effectively nurture a culture of evidence based practice.    decisions in libraries are often made as a group or team, and some decisions may require approval from outside the library or information service (booth, 2009, p. 342; koufogiannakis, 2013, p. 9; koufogiannakis & brettle, 2016, p. 10). the revised eblip process model, known as the 5 a’s – articulate, assemble, assess, agree and adapt, goes some way to address practical realities of decision making in library team contexts (koufogiannakis & brettle, 2016, p. 14). however, this model continues to promote eblip as an approach to specific situations or problems that arise from professional practice, and not explicitly as a habitual or cultural way of being. the model also sustains that eblip be implemented or pursued by individual practitioners or small teams, and by doing so, leaves being evidence based to a handful of lis professionals in an organization who are inclined to practice in this way. library leaders and managers have a responsibility to cultivate evidence based approaches that make use of a variety of evidence sources when making decisions around service improvements (koufogiannakis & brettle, 2016, p. 17). being evidence based as practitioners, organizations, and as a profession should not be up to the few individuals with the capacity to do so.   as koufogiannakis has previously stated, thus far our understanding of evidence based practice in the lis professional context captures and describes the experience of individual practitioners, but not the organization (2013, p. 9). our existing understanding of eblip comes from a need to improve professional practice and service delivery, driven by curiosity, reflection, evaluation, professional responsibility, and a desire to keep up to date. but what about the need for an organization to be evidence based to achieve strategic success? we haven’t really addressed this question. little is known about how a collective group of lis practitioners becomes evidence based at a whole-of-organization level. it is time to take what we know and pave a way forward for library and information service organizations to truly embrace a culture of evidence based practice at an organizational level.   evidence based practice in the library and information service organization   there has been little research into experiences of evidence based practice in an organizational setting, within libraries or other industries and domains. an australian ethnographical study into library and information professional experiences of evidence based practice in a public library found four “interconnected and interdependent cultural orientations” which together represent a culture of evidence based practice:   1.       a culture of valuing: evidence based practice involves being valued and valuing each other. 2.       a culture of being: evidence based practice involves being and becoming a professional. 3.       a culture of learning: evidence based practice involves being a learner and supporting the learning of others. 4.       a culture of leading: evidence-based practice involves leadership at many levels and leading towards one shared vision (gillespie et al., 2016, para. 19).   findings of this study suggest that these four cultural elements need to be present to enable and support evidence based practice at an organizational level. further research is needed to better understand how a culture of evidence based practice manifests, in order to develop approaches to build organizational capability so that library services as a whole, rather than a handful of individuals, can readily respond to an ever-changing environment and demonstrate value to stakeholders.   at the university of southern queensland (usq) library, we have made evidence-based practice a primary focus by creating a role dedicated to fostering and supporting a whole of library culture of evidence-based practice. the role of coordinator, evidence based practice was created to engage with library staff to understand their business, and the evidence needed to support business improvement for the library; and to advise and support staff to engage in evidence based practice (howlett & thorpe, 2018, p. 1-2). the coordinator has a broad mandate to go beyond data collection, performance measurement, and reporting requirements in order to build capacity for evidence based practice across usq library. the position exists to ensure that not only is the right evidence identified and gathered, that evidence is critically appraised, interpreted, applied and tells the story of how the library contributes to student and institutional success.   a conceptualisation of evidence based practice at an organizational level   given the little guidance that exists to apply evidence based practice to a whole-of-organization level, understanding and generating impact with the newly created coordinator role has involved applying what is known about eblip with a “ripple effect” – promoting, mentoring, coaching, consulting with individuals and teams across usq library, to enable and influence an evidence based practice culture. with this approach, the coordinator role enables and supports the library, both individual staff and collectively as an organization, to operate consistently in an evidence based way.   an academic library needs to deliver services and programs that are aligned with and accountable to the broader institution. particularly, academic libraries, library directors, and library stakeholders are seeking and using evidence from a variety of sources to address complex decisions and communicate the value the library contributes to their university (harland, stewart & bruce, 2017; lembinen, 2018; newton miller, 2018). like many library and information service organizations, an academic library has the added layer of needing to be strategically aligned and accountable to its institution’s purpose and direction. the academic library’s strategic alignment with the parent institution needs to be reviewed and adapted where necessary, as a continuous cycle that involves a series of coordinated and interrelated activities.   at usq library, we have created a visualization or “lens” which applies existing eblip models to demonstrate how the library can implement evidence based practice as a “way of being.” this lens (figure 1) assists conversations about evidence based practice throughout usq library, promoting its relevance to the various roles and functions.   existing evidence based practice literature and observations made in professional practice have informed the design. the lens is particularly applicable to library and information services where they either have a parent organization or are accountable to a broader purpose. the lens outlines a process that reflects this accountability and recognises the realities of daily professional practice and experience, such as that the process is not always linear. the lens does not replace any existing eblip processes or models, but demonstrates an organizational perspective to applying evidence in practice.   each step in the lens describes how evidence is used and what results. each step informs the connected segments. the lens also depicts the relationship that the coordinator has between the library leadership and teams, as a facilitator and conduit, relating the library’s strategy through the teams in order to effectively communicate the library’s contribution and value to the university. the cycle is multi-directional and iterative, reflecting the interplay between the various stages and messiness that sometimes accompanies evidence based practice.   there are four elements, or steps to the model – interpret, apply, measure and communicate. firstly, the library needs to interpret its landscape and context. as an academic library, this means examining what is happening in the higher education sector, in other libraries and information services, the lis profession, learning and teaching trends, digital learning and research environments and other related areas. the strategic direction of the university must also be considered, as well as the needs and wants of the library’s clients students, staff and our community. these sources of evidence enable the library to develop its strategy, priorities and goals. figure 1 a lens for evidence based library in an organizational context.     applying the library’s strategy involves delivering services and programs through the completion of procedures, workflows and service models. professional knowledge and experience, either individual or collective, is applied alongside other evidence needed for day-to-day decision-making and service improvements. measures of outputs, such as usage statistics, are gathered, evaluated, analysed, and made sense of to identify and understand the library’s outcomes, value and impacts on the clients’ experiences.   stories of the library’s contribution to the institution’s strategic goals and objectives need then be communicated and shared, using both statistics and narratives that are relevant to stakeholders. story telling is important to “close the loop” and communicating with stakeholders the results of the library’s efforts and resources. sharing what has been learned adds to the evidence base, both within the institution and the broader lis profession (howard & davis, 2011, p. 19). the communication step, if all others are done effectively, generates influence and advocates for what the library is, and what it achieves for the university. a check back to the interpretation of the context completes the cycle and ensures that what the library offers, delivers and measures remains relevant and strategically aligned. through these steps, evidence based practice can become a conscious and deliberate process of how the library operates.   future research   an illustration of how a library can be evidence based as a “way of being” is only one step towards applying existing eblip understanding to a collective, or whole-of-organization context. further research can help clarify the steps, strategies, or initiatives needed to build evidence based organizational capability. what does an evidence based library look like? how can library leaders determine the extent to which their organization is practicing and delivering services in an evidence based way?   a research project is currently underway to help answer these questions by exploring evidence based practice maturity at different levels. the aim of this project is to develop a maturity model for australian academic libraries that describes characteristics of evidence based practice at different levels of maturity, and identifies strategies, initiatives, or activities that library leaders can implement to progress maturity on a whole-of-organization level. this research will rely upon well established discourse in the literature about the development and use of maturity models in higher education and libraries. maturity models from related areas will also inform this research. though initially limited in scope, it is envisioned that the proposed maturity model may be applied to, or help inform, evidence based practice maturity in other lis contexts.   conclusion   so far, our understanding of eblip has captured and interpreted the individual practitioner experience. this understanding reveals that workplace context heavily influences the extent to which a library implements evidence based practice. however, a library and information service should not allow the responsibility of implementing evidence based practice to fall on the few lis professionals at the institution who possess the knowledge and skills. organizations need approaches that enable and build the capacity and culture of evidence based practice through all staff, if they are readily respond to changes and opportunities, financial or otherwise.   usq library is working to harness the potential of the local evidence base to support the university’s strategic goals and objectives. creating the coordinator role signals an explicit and deliberate commitment to evidence based practice and acknowledges the need to apply it at a whole-of-organizational level. the lens presented in this commentary is a step taken by usq library to increase our collective, shared understanding and application of eblip as we seek to become an evidence based library.   acknowledgements   the author would like to thank clare thorpe for her wisdom and guidance in bringing the coordinator (ebp) role to life at usq library and providing input to this article.   references   booth, a. (2009). eblip five-point-zero: towards a collaborative model of evidence-based practice. health information and libraries journal, 26(4), 341-344. https://doi.org/10.1111/j.1471-1842.2009.00867.x   gillespie, a. (2014). untangling the evidence: introducing an empirical model for evidence-based library and information practice. information research, 19(3). retrieved from http://www.informationr.net/ir/19-3/paper632   gillespie, a., partridge, h., bruce, c., & howlett, a. (2016). the experience of evidence-based practice in an australian public library: an ethnography. information research, 21(4). retrieved from http://www.informationr.net/ir/21-4/paper730.html    gillespie, a., miller, f., partridge, h., bruce, c., & howlett, a. (2017). what do australian library and information professionals experience as evidence? evidence based library and information practice, 12(1), 97-108. https://doi.org/10.18438/b8r645   harland, f., stewart, g., & bruce, c. (2017) ensuring the academic library's relevance to stakeholders: the role of the library director. the journal of academic librarianship, 43(5), 397-408. https://doi.org/10.1016/j.acalib.2017.06.009   howard, z., and davis, k. (2011). from solving puzzles to designing solutions: integrating design thinking into evidence based practice. evidence based library and information practice, 6(4), 15-21. https://doi.org/10.18438/b8tc81   howlett, a., & howard, z. (2015). exploring the use of evidence in practice by australian special librarians. information research, 20(1). retrieved from http://www.informationr.net/ir/20-1/paper657.html   howlett, a., & thorpe, c. (2018, august). ‘it’s what we do here’: embedding evidence-based practice at usq library. paper presented at the asia-pacific library and information conference, gold coast, australia. retrieved from https://eprints.usq.edu.au/34729/   koufogiannakis, d. (2012). academic librarians’ conception and use of evidence sources in practice. evidence based library and information practice, 7(4), 5-24. https://doi.org/10.18438/b8jc8j   koufogiannakis, d. (2013). eblip7 keynote: what we talk about when we talk about evidence. evidence based library and information practice, 8(4), 6-17. https://doi.org/10.18438/b8659r     koufogiannakis, d. (2015). determinants of evidence use in academic librarian decision making. college & research libraries, 76(1), 100-114. https://doi.org/10.5860/crl.76.1.100   koufogiannakis, d., & brettle, a. (eds.) (2016). being evidence based in library and information practice. facet publishing: london.   lembinen, l. (2018). academic library directors’ strategic decision-making process. liber quarterly, 27(2), 1-21. https://doi.org/10.18352/lq.10224   luo, l. (2018). experiencing evidence-based library and information practice (eblip): academic librarians’ perspective. college & research libraries, 79(4), 554-567. https://doi.org/10.5860/crl.79.4.554   miller, f., partridge, h., bruce, c., yates, c., & howlett, a. (2017). how academic librarians experience evidence-based practice: a grounded theory model. library & information science research, 39(2), 124-130. https://doi.org/10.1016/j.lisr.2017.04.003   newton miller, l. (2018). university community engagement and the strategic planning process. evidence based library and information practice, 13(1), 4-17.  https://doi.org/10.18438/eblip29351   partridge, h., edwards, s., & thorpe, c. (2010) evidence-based practice: information professionals' experience of information literacy in the workplace. in a. lloyd & s. talja (eds.) practising information literacy: bringing theories of learning, practice and information literacy together (pp. 273-298). wagga wagga, n.s.w.: centre for information studies, charles sturt university. editorial   eblip gets an upgrade   lorie kloda editor-in-chief associate university librarian, planning and community relations concordia university montreal, quebec, canada email: lorie.kloda@concordia.ca      2018 kloda. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29427     welcome to volume 13 of evidence based library and information practice (eblip). after 12 years of publication, supported by university of alberta learning services, the journal is getting a new look thanks to an upgrade to the open journal systems (ojs) 3. ojs is an open access platform developed by the public knowledge project, and which has been used to host eblip since its inception. the ojs platform consists of both the public-facing journal website with all past and current issue contents of the journal as well as the journal management system for the editorial team to manage the editorial workflow from manuscript submission to publication. with this upgrade, our editorial team has a new interface with enhanced features, and more importantly, you, the reader, have an improved interface for engaging with the contents of eblip. authors also have a slightly new workflow for the submission, review, and copyediting process.   there is no change to the contents of the journal or to our policies and procedures. the upgrade to ojs 3 allows for additional features and flexibility in the way we process and publish the journal. each issue, which is published quarterly, will still consist of a combination of research articles, review articles, evidence summaries, classics, commentaries and other publication types. these publications are available in both html and pdf formats.  the journal website should also be easier to navigate and to read on a desktop, laptop, or mobile device. i hope you enjoy the upgrade, and welcome any feedback you have about it.   in this issue the editorial board welcomes a new associate editor (research articles), ann medaille. ann is one of two associate editors responsible for research article submissions. rebekah (becky) willson has stepped down at the end of her term to focus on her position as lecturer at the university of strathclyde and she continues to be associated with the journal as a peer reviewer. becky is also co-chair of the local organising committee of the 10th international evidence based library and information conference, which will take place in glasgow in the summer of 2019. i would like to thank becky for all her work over the past three years with eblip, and her continued involvement.   evidence summary   the types of publications read by finnish scholars vary with their purposes for reading   a review of: late, e., tenopir, c., talja, s., & christian, l. (2019). reading practices in scholarly work: from articles and books to blogs. journal of documentation, 75(3), 478-499. https://doi.org/10.1108/jd-11-2018-0178   reviewed by: barbara m. wildemuth professor emeritus, school of information & library science university of north carolina at chapel hill chapel hill, north carolina, united states of america email: wildemuth@unc.edu   received: 25 oct. 2019                                                                   accepted:  22 jan. 2020      2020 wildemuth. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29667     abstract   objective – to closely examine the role of reading in scholarly work, with particular attention to the relationships between reading practices and characteristics of the scholars, the types of publications they read, and the context of reading.   design – survey.   setting – universities in finland.   subjects – 528 academics (research directors/managers, professors, post doctoral researchers, doctoral students, lecturers, and researchers).   methods – an online survey was distributed in finland, october-december 2016. the first part of the survey asked about scholars’ general reading practices; the second part asked about their most recent reading of two particular publications, one a journal article and the other a different publication type. in relation to these two readings, the scholars provided information about the documents read, the reading process and context, how the document was identified and obtained, and the effect of the reading on their work.   main results – on average, the scholars read 59 publications per month: 20 journal articles, 3 books, 5 conference proceedings or research reports, 17 newspaper articles, 9 magazine articles, 4 blogs, and 2 non-fiction/fiction books. there was no statistically significant difference in the number of journal articles read across disciplines, but the number of books read was highest in the humanities and social sciences and lowest in the sciences and medical sciences. frequency of reading of particular publication types also varied by work focus (research vs. teaching/administrative) and by the nature of the scholar’s research (basic vs. applied).   the scholars were also asked about the importance of reading different publication types. overall, scholarly journals and article compilations were rated as most important for scholarly reading. differences in these ratings were found across disciplines, work focus, nature of the research, and scholar rank/status.   part 2 of the survey focused on the most recent items read by the scholars. their reading of journal articles, scholarly books, and conference proceedings/research reports was mainly for the purpose of research and writing. their reading of newspaper articles, magazine articles, and blogs was mainly for current awareness and continuing education. their reading of non-fiction/fiction books was mainly for their personal interest or pleasure. none of these publication types was specifically focused on supporting the scholars’ teaching.   over 70 percent of the recent readings were new, rather than re-readings. across all publication types, the scholars read at least parts of the item “with great care”. almost half of the journal articles recently read have been or will be cited in the future; this proportion was also high for scholarly books and conference proceedings/research reports, but not for the other publication types.   the most recently read journal articles were brought to the scholar’s attention primarily through searching; they became aware of scholarly books and conference proceedings/research reports through both searching and because another person told them about the item. scholars mainly obtained journal articles and scholarly books from their libraries, but they also obtained articles on the internet and scholarly books from another person.   forty percent of the scholars read journal articles by printing a downloaded copy, but over half read them on a computer, mobile phone, or e-reader. over half of the scholarly books were read from published/printed copy, but 18% read the book in an electronic version. most reading occured in the scholar’s office or lab.   over half the journal articles and conference proceedings/research reports read were published within the last year; just under half the scholarly books read were published within the last year. while these scholars worked in finland, 91% of the journal articles and 73% of the scholarly books they read were published in english.   conclusion – the results from this study confirmed and extended findings from previous studies (e.g., tenopir et al., 2010, 2015). they demonstrated that scholars read a variety of types of publications for a variety of purposes. however, journal articles still dominated the reading and the perceptions of importance among the various publication types, particularly for the purposes of research and writing. this paper provides a first look at scholars’ uses of the other publication types and the influence of work tasks on reading practices; further research is needed to understand these relationships more fully. in general, the disciplinary differences in reading practices found in this study mirror the different publishing practices of the disciplines and so may be affected by future evolution toward open access and social media use for scholarly communication.   commentary   first, it should be noted that this paper presents many more findings on the reading practices of finnish scholars than have been included in this summary. in the original paper, the results are presented in 21 tables. for those who are particularly interested in the information behaviors of scholars, a close reading of the original paper is certainly warranted.   this survey was methodologically strong (boynton & greenhalgh, 2004) and conducted by an experienced team of researchers; however, there are a few limitations that should be mentioned here. the most obvious is that the sample was drawn from a population of scholars in finland—a single country that is ranked 35th in the world in the number of scientific articles published (national science board, 2018, table 5-22). the authors thoughtfully address this limitation in their discussion, noting that scholars in finland read and publish in both finnish and other languages, especially english. in addition, the results from part 1 of the survey are consistent with findings from past studies. finally, a direct comparison of finland with australia and the u.s. found only small country-based differences (tenopir et al., 2010). based on this evidence, it is reasonable to consider the results of this study in terms of their implications for scholars in other countries and the libraries that serve them.   one key aspect of the study methods should be taken into account when interpreting the findings: in part 2 of the survey, each respondent/scholar was asked to provide detailed information about their most recent reading of a scholarly journal article and their most recent reading of some other type of publication. thus, while there is a significant amount of data supporting the findings about journal article reading, there is much less data about the reading of the other types of publications. for example, the ratings of the importance of reading scholarly books was based on only 93 responses, while the ratings of article importance were based on 454 responses. thus, the part 2 findings related to the reading of scholarly books, conference proceedings and research reports, newspapers, magazine articles, blogs, and non-fiction/fiction books should be interpreted with caution. third, the results were tested for their statistical significance, and these findings are reported. however, the authors do not report effect sizes for their findings, making it difficult to evaluate which of the differences in reading practices are most important to consider in terms of libraries’ support of those practices.   a final methodological question concerns the total number of readings undertaken by a typical scholar (i.e., the mean number of readings during the past month). the total reported by the authors is 59 readings per month, which is the sum of the means for the various types of publications. however, it’s not clear that any given scholar reads all type of publications; in other words, it’s not clear that the unit of analysis is the scholar, rather than the particular publication type. for instance, 37% of the respondents read no scholarly books and 24% read no conference proceedings or research reports. thus, it is possible that the total of 59 readings per month is an overestimate of scholarly reading.   in spite of these methodological concerns, the authors have provided a strong basis for librarians’ consideration of their collections and the way those collections support scholars’ reading practices, as well as service development and outreach programs. there are three findings that are particularly pertinent to practitioners.   first, these results can be used to understand the broad range of materials that scholars find useful in their work. while the reading of journal articles was dominant, scholars are also relying on both scholarly books and conference proceedings/research reports for their research and writing, and newspaper and magazine articles and blogs for current awareness and continuing education. they also use magazine articles and other fiction/non-fiction pieces to augment their use of journal articles for teaching purposes. as academic libraries consider both their collections and their services they provide, this wide variety of publication types and their variation with the scholars’ purposes should be taken into account.   second, these results show that scholars are strongly dependent on their libraries to support their scholarly reading needs. scholars become aware of relevant articles through searching both web search engines (47%) and databases provided by the library, such as academic search premier or web of science (37%). almost half of the scholars reported obtaining the journal articles from the library collection; an additional 15% obtained the articles from an institutional or subject repository. almost 20% of the scholars obtained the articles from the internet, but it is likely that many of these were obtained through online subscriptions maintained by the library. only about 2% of the scholars obtained their articles from their own personal subscriptions. thus, academic librarians should be vigilant in maintaining collections that will support the research work of the scholars at their institutions. without the library collections, research would be severely hobbled.   other considerations should come into play when developing e-book collections. this study took a close look at the format in which the publications were read. while journal articles were most often downloaded and printed on paper prior to reading (40%), over half of the readings were on a computer, mobile phone, or e-reader. the reading of scholarly books followed a different pattern. some were downloaded and printed prior to reading (18%) and some were read in electronic form (25%), but 56% of the readings were from a printed/published copy. most of these printed copies were obtained from the library’s collection (44%) or borrowed from another person (26%). from these findings, we can conclude that librarians should continue to develop their printed book collections in the core research areas of the scholars they support.   in summary, the results from this two-part survey provide us with a detailed look at scholars’ reading practices and have important implications for academic library collection development. journal articles and scholarly books continue to play important roles in scholars’ work, and scholars rely on their libraries for support in discovering and obtaining these publications.   references   boynton, p. m., & greenhalgh, t. (2004). hands-on guide to questionnaire research: selecting, designing, and developing your questionnaire. bmj, 328(7451), 1312-1315. https://doi.org/10.1136/bmj.328.7451.1312   national science board. (2018). science and engineering indicators 2018. nsb-2018-1. alexandria, va: national science foundation. retrieved from https://www.nsf.gov/statistics/indicators/   tenopir, c., king, d.w., christian, l., & volentine, r. (2015). scholarly article seeking, reading, and use: a continuing evolution from print to electronic in the sciences and social sciences. learned publishing, 28(2), 93-105. https://doi.org/10.1087/20150203   tenopir, c., wilson, c.s., vakkari, p., talja, s., & king, d.w. (2010). cross country comparison of scholarly e-reading patterns in australia, finland and the united states. australian academic & research libraries, 41(1), 26-41. https://doi.org/10.1080/00048623.2010.10721432 evidence summary   german-language websites containing information about rare diseases lack quality indicators   a review of: pauer, f., litzkendorf, s., göbel, j., storf, h., zeidler, j., & graf von der schulenburg, j.-m. (2017). rare diseases on the internet: an assessment of the quality of online information. journal of medical internet research, 19(1), e23. https://doi.org/10.2196/jmir.7056   reviewed by: jessica a. koos senior assistant librarian/health sciences librarian stony brook university stony brook, new york, united states of america email: jessica.koos@stonybrook.edu     received: 27 aug. 2020                                                             accepted:  31 oct. 2020      2020 koos. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29830     abstract   objective – to evaluate the quality of the information contained in websites about rare diseases and to determine if quality varies based on the supplier category of the website.   design – questionnaire and content analysis.   setting – germany   subjects – 693 german-language websites    methods – websites were identified through a google search: all 8,000 rare diseases (as listed on orphanet) and their synonyms were entered into google; the first 20 results for each disease were scanned for sites written in german. a questionnaire designed to measure the quality of information found on the websites was mailed to each identified website provider. for those who did not respond, the survey was completed by the authors using information from the site. a t test was used to examine differences in the quality of information among the types of information providers.   main results – a total of 693 information suppliers were identified. the suppliers completed 17.7% of the surveys; the other 82.3% were completed by the authors. the majority of information providers were patient organizations/support groups (38.8%) followed by medical institutions (26.8%). information provided by individuals had the lowest quality rating. there were no statistically significant differences between the quality of information supplied by patient support groups and medical institutions. the highest quality rating was provided by associations/sponsoring bodies.   conclusion – there is not much information available on the internet regarding rare diseases. patient support groups and organizations are the largest provider of information. the overall quality rating of information on rare disease websites was found to be low, particularly in areas of accessibility. website providers should be made aware of how to produce websites of higher quality with greater accessibility.   commentary   based on a brief literature review on the subject of rare disease websites, there has not been a great deal of research conducted in this area. this study helps to fill that gap. it is important to note that the website quality evaluation tool used in this article had been developed previously by several of the authors of this article (pauer et al., 2016). the questionnaire consists of a series of questions across several quality domains.   the quality of this study was appraised using “the cat: a generic critical appraisal tool” created by perryman and rathbun-grubb (n.d.). overall, the article was found to be of relatively high quality based on this assessment. the authors have academic credentials related to the subject area and have published on this topic previously. the research questions and methods were explained. the previous relevant literature was included throughout the article.   there were several limitations to this research, as stated by the authors. the websites that were examined were specific to germany, austria, and switzerland (german-speaking countries) and might not be applicable to other geographic regions. in addition, the accuracy of the medical information contained in the websites was not evaluated; however, this was most likely beyond the scope of the project.   there were other limitations to the methodology of this study. the fact that website providers were asked to rate their own sites potentially introduces significant bias. in addition, the information obtained from the respondents may have differed from that obtained by the authors, and the reliability between these sources was not addressed. there was also no breakdown of the types of providers who responded to the survey and those who did not. also, a thorough examination of the websites of those providers who did not respond could not be completed as certain data could not be extracted by the authors; therefore, this information was not included in the study.   there are several statistical concerns as well. there is no description of how questionnaire responses were scored, or how means were obtained from the results. there are also not enough data provided on the t-test results. due to these factors, the reader is unable to determine the strength of evidence.   overall, this article provides information on the quality of rare disease websites, which has not been studied extensively. as stated by the authors, patients with rare diseases often turn to the internet for more information about a disease and to connect with others who have the same condition. the conclusions of this research indicate that rare disease website providers should be more vigilant in providing higher quality websites and information. librarians should exercise caution when referring patrons to websites about rare diseases and evaluate them carefully to ensure that they are of sufficient quality. librarians might also consider providing curated research guides (or incorporating into existing guides) reliable information sources on rare diseases.   references   pauer, f., göbel, j., storf, h., litzkendorf, s., babac, a., frank, m., lührs, v., schauer, f., schmidtke, j., biehl, l., wagner, t. o., ückert, f., graf von der schulenburg, j.-m., & hartz, t. (2016). adopting quality criteria for websites providing medical information about rare diseases. interactive journal of medical research, 5(3), e24. https://doi.org/10.2196/ijmr.5822   perryman, c. & rathbun-grubb, s. (n.d.). the cat: a generic critical appraisal tool.  retrieved september 9, 2020 from http://www.jotform.us/cp1757/thecat       research article   delivering information literacy via facebook: here comes the spinach!   a. f. tyson subject librarian, learning teaching & research university of canterbury christchurch, canterbury, new zealand email: fiona.tyson@canterbury.ac.nz   anton angelo research data coordinator, learning teaching & research university of canterbury christchurch, canterbury, new zealand email: anton.angelo@canterbury.ac.nz   brian mcelwaine resource discovery manager, access and collections university of canterbury christchurch, canterbury, new zealand email: brian.mcelwaine@canterbury.ac.nz   kiera tauro subject librarian, learning teaching & research university of canterbury christchurch, canterbury, new zealand email: kiera.tauro@canterbury.ac.nz   received: 20 nov. 2018                                                                  accepted: 18 apr. 2019      2019 tyson, angelo, mcelwaine, and tauro. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29532       abstract   objective – information literacy (il) skills are critical to undergraduate student success and yet not all students receive equal amounts of curriculum-integrated il instruction. this study investigated whether facebook could be employed by libraries as an additional method of delivering il content to students. to test whether students would engage with il content provided via a library facebook page, this study compared the engagement (measured by facebook’s reach and engagement metrics) with il content to the library’s normal marketing content.    methods – we ran a two-part intervention using the university of canterbury library’s facebook page. we created content to help students find, interpret, and reference resources, and measured their reception using facebook’s metrics. our first intervention focused on specific courses and mentioned courses by name through hashtagging, while our second intervention targeted peak assessment times during the semester. statistics on each post’s reach and engagement were collected from facebook’s analytics.   results – students chose to engage with posts on the library facebook page that contain il content more than the normal library marketing-related content. including course-specific identifiers (hashtags) and tagging student clubs and societies in the post further increased engagement. reach was increased when student clubs and societies shared our content with their followers.    conclusion – this intervention found that students engaged more with il content than with general library posts on facebook. course-targeted interventions were more successful in engaging students than generic il content, with timeliness, specificity, and community being important factors in building student engagement. this demonstrates that academic libraries can use facebook for more than just promotional purposes and offers a potential new channel for delivering il content.     introduction   students are increasingly using social media to communicate, disseminate information, and learn about the world around them. they tailor their online experience so that the information they are interested in comes to them with no searching required. at the university of canterbury in new zealand, the library wanted to see if it was possible to leverage a social media platform to deliver information literacy (il) content in a way that was convenient and useful to undergraduate students.   undergraduate student success is positively associated with library use and the development of il skills (catalano & phillips, 2016; soria, fransen, & nackerud, 2014, 2017). this project sought to determine to what extent students would engage with il content from a library facebook page. rather than making students come to us, we would go to them and give them tools they could use to successfully complete assignments. prior to the intervention, the library had a facebook page with 1921 followers, and this, along with the knowledge that facebook worked well for the format of the content we intended to post, gave us a place to start.   we separated our approach into two distinct parts to identify the factors that influenced engagement. in semester one of 2017 we targeted specific large undergraduate courses, and using the course syllabi, posted relevant il content timed to correspond to important assessments in each course. we also engaged with relevant student societies and other university facebook groups by tagging them in our posts. in semester two, we were less specific with our approach and posted general il content without targeting specific cohorts.   literature review   much has been written about the potential uses of facebook by academic libraries, but a scan of the literature suggests that libraries are still conservative in their use of social media. the literature is still largely instructional, providing information on how to use facebook and suggesting best practice guidelines for developing an institutional page (burkhardt, 2010; mack, behler, roberts, & rimland, 2007; solomon, 2013; wright joe, 2015). perhaps not surprisingly, recent content analyses of facebook in academic libraries have found a focus on collection promotion and building connections between students and the library (al-daihani & abrahams, 2018; harrison, burress, velasquez, & schreiner, 2017; palmer, 2014; phillips, 2011; zhu, 2016). a 2017 analysis of six american academic library social media pages found three main themes: “making community connections”, “creating an inviting environment”, and “providing content” (harrison et al., 2017, p. 254), with “content” defined as posts relating to archives, collections, and exhibits. while two early studies indicate potential for offering reference services via facebook (click & petit, 2010; mack et al., 2007), academic libraries continue to make “quite limited use” of facebook with the platform used as a marketing tool or loudspeaker rather than for interaction or service provision (aharony, 2012, p. 369).   other disciplines have begun to explore more innovative uses of facebook, including facebook-based instruction. after finding that students already use social media networks such as facebook to converse with their classmates regarding courses and assessment (donlan, 2014; towner & lego muñoz, 2011), researchers investigated the use of facebook for formal learning, including language learning (annamalai, 2016; leier, 2017; omar, embi, & yunus, 2012), tourism education (chen, 2018), and sport coaching (donlan, 2014). they found that formally assessed facebook activities were not well-received by students, who struggled to maintain the formal strictures of educational assessment, such as appropriate language and deadlines, on an informal platform (gonzález-ramírez, gascó, & taverner, 2015; leier, 2017). in addition, students may not wish to engage with their lecturers on facebook (gonzález-ramírez et al., 2015), or only passively interacted with the facebook page/group in question (chugh & ruhi, 2018).  however, students reported appreciating being able to access academic material via facebook (gonzález-ramírez et al., 2015), being able to access the resources and support posted on facebook groups (chen, 2018), and being able to use facebook to access and post links (donlan, 2014). the dominant finding in the literature is that facebook is a useful platform for “enhancing learning, increasing participation and engagement, content dissemination, improving pedagogy and information sharing” (chugh & ruhi, 2018, p. 613).   the finding that using facebook for informal learning, rather than formal learning, is better received by students raises the question of how to measure impact. while the use of formal assessment allows for student feedback or grade analysis from an identifiable student cohort, informal learning requires different metrics. some studies of formal learning have used facebook metrics to analyse student behaviour on the facebook page, in addition to qualitative measures (donlan, 2014; leier, 2017). core facebook metrics for each post on a facebook page include the number of likes (“comparable to a non-committal smile or a nod at something you like or that you were interested in”, mauda & kalman, 2016, p. 3527), shares (“where the user is suggesting to his or her own circle of friends to engage with a certain message”, p. 3527), and comments (“comparable to expressing an opinion about a brand or an organization, or about their message”, p. 3527). other less visible engagement measures are the number of times visitors clicked on links within posts or viewed images/videos (mauda & kalman, 2016). donlan (2014), in particular, noted that facebook metrics are the best proxy for student use when participation is optional and cannot be systematically tested in any other way.   thus far, library use of facebook as a platform for developing students’ il skills has not been reported. yet there is a strong body of literature on using facebook as a pedagogical tool within traditional il instruction settings. for example, the use of keywords and controlled vocabularies in databases has been compared to social practices such as tagging on facebook; in addition, information-seeking behaviours fostered on the platform can be analogized to database searching (bobish, 2011; click & petit, 2010; godwin, 2009; witek & grettano, 2012). teaching il with reference to facebook conventions indicates an awareness on the part of academic librarians that most students use facebook (akcaoglu & bowman, 2016; chugh & ruhi, 2018). the popularity of facebook in new zealand is no different, with facebook the dominant social media platform and 75% of new zealanders visiting facebook at least monthly (nielsen, 2016, pp. 30-31).   while an early study found that students felt hesitant about communicating with library staff on facebook (chu & meulemans, 2008), a more recent study found that students ranked facebook as their preferred social media platform for communications from the library (winn, groenendyk, & rivosecchi, 2015). this is indicative of how facebook has become a default platform for everyday communication. furthermore, students increasingly use social media for information seeking. a review of seven studies of secondary and tertiary students’ use of social media found that social media “assist[ed] users in their request for information in combination with powerful search engines”, with convenience being a major contributor to the use of social media (hyldegård, 2014, p. 113). the research indicates that the role of facebook in students’ lives has been evolving, but academic libraries’ use of facebook has not, raising the question of how academic libraries can use facebook to better engage with students and share educational content.   aims   the aim of this project was to determine to what extent students will choose to engage with il content from a library facebook page. we sought to address the practical problem of how to reach students outside of traditional library services and identified facebook as a potential platform for doing so. in particular, we wanted to provide undergraduate students with il tools that would help them succeed at tertiary level study, while also building their awareness of the role the library plays in supporting their study.   methods   preparation   facebook was chosen as the social media platform for delivering targeted content to students for the following reasons:   1.           our library already had a modest established audience of 1921 facebook followers. based on facebook demographics, the majority of our audience was aged 18-34, and a large percentage had a location of christchurch, leading to the assumption that a sizeable proportion of our followers were current university of canterbury students. 2.           some platforms were discounted as impractical. for example, snapchat was not suitable because of the ephemeral nature of posts on that platform, and twitter was considered too brief and too removed from our target undergraduate audience. 3.           facebook facilitated posting content in an appropriate format more easily than other platforms. 4.           the literature indicated that more students would have a facebook account than any other social media account (nielsen, 2016). 5.           some groups that we hoped to work with, such as university of canterbury’s student engineering society, maintained a presence on facebook.   having determined that we wanted to deliver il content via facebook, we developed the following working definition of the term:   content that helps students search for, locate, evaluate and correctly reference information for their assignments.   adapted from the australian and new zealand information literacy framework: principles, standards and practice (bundy, 2004), this definition captures the specific information literacy skills we could reasonably deliver on this particular platform.   located in christchurch, new zealand, the university of canterbury is a research and teaching university, with 11 subjects ranked in the top 200 qs world university rankings (university of canterbury, n.d.).  undergraduate students are the dominant student group at the university of canterbury, with 8810 undergraduate students (55% of the student population) in 2017 (education counts, 2017, tab enr.30). it was hypothesized that by targeting large undergraduate classes in semester one, we could reach the maximum number of students with each message (and avoid alienating other users with posts targeted at small cohorts). these large classes typically were in students’ first year of study, as new students were the most likely to benefit from the bite-sized il content we could provide via facebook.   prior to the start of the semester, we reached out to subject librarians and asked if they knew of any large-scale first year classes in their areas that were suited to having il instruction delivered via facebook. after choosing five classes that seemed suitable, the subject librarians contacted lecturers to ensure that they would be happy to direct their students to the library facebook page. we were then given the syllabus for the course, and we planned our posts based on the individual course schedules so they included information that was known to be specifically useful for a forthcoming assessment. there was no further contact with lecturers past the initial green light for posting the content on facebook, but many were supportive of our pilot program and promoted our facebook page to their students, either in class or by sharing posts on facebook.   once suitable cohorts were identified, we used the university of canterbury students’ association website to identify appropriate student clubs to tag in our posts. with these pieces in place, a posting schedule was produced, detailing what would be written, by whom and when, and who we would tag in our posts to maximize the number of students reached. posts were also hashtagged with the appropriate course code so students could readily identify them.1   semester two posts were targeted at the general student population. they still contained il content but were not aimed at a particular cohort or discipline. it was decided that we would create content based on the general assessment schedule for the university. while this was not a written schedule, we know based on experience that many courses have assignments, midterms, and exams at similar points throughout the semester, and that students all have similar needs at these times. for example, we know that at the beginning of the semester, students need to know how to find their textbooks in the library, whereas just before the midterm break lots of essays are due and apa referencing resources are useful. we wanted to determine whether our posts generated engagement because the content was generally useful, or because the content was well targeted. since we were not targeting specific cohorts, we did not tag student clubs and other university pages in the posts or contact course coordinators. a posting schedule was also created for semester two, but it only detailed what would be written, by whom and when. we also tagged all our posts with the hashtag #deadlinesarecoming, a riff on the popular “winter is coming” tagline from the game of thrones tv and book series. this hashtag could be clicked on, or searched by students so that all our il posts for that semester would be viewable.   posting   we focused on delivering bite-sized chunks of information that could be easily conveyed through short posts or infographics. content included advertising just-in-time drop-in sessions focused on particular assignments, demonstrating specific il skills (such as searching a web-scale discovery tool; see figure 1 for an example), and promoting specific resources. due to the relatively ephemeral nature of facebook posts, we did not spend a lot of time on the creation of these posts, choosing instead to prioritize content over style.     figure 1 example of an intervention post regarding how to use the library’s web-scale discovery tool.     intervention posts and non-intervention posts   all posts that were designed as part of this project are referred to as “intervention posts.” during the current study, university of canterbury library staff continued to create posts about opening hours, events, and other promotional material. these posts were distinct from those that contained il content, and the statistics from these posts formed the “control group” against which the success of our il posts would be judged. we divided our total population of facebook posts over the study’s time period into intervention posts and non-intervention posts. ·         intervention posts: these posts to facebook represent those developed as part of the study. they contained il content developed by the research team. in semester one, there were 38 intervention posts, 11 of which tagged student groups. in semester two, there were 30 intervention posts, all without tagging. a post was “tagged” when the intervention included an internal facebook link to that group. it alerted the group’s members that content relevant to them had been posted, and provided an opportunity for that group to share the post to their own followers. ·         non-intervention posts: all library posts that were not developed by the research team for the purpose of delivering il material are considered to be non-intervention posts. in semester one, there were 404 non-intervention posts, and in semester two there were 280.   all four authors crafted posts, so it was important to develop a consistent style and use of images and video. we wanted to have a generally informal style sprinkled with pop culture references and memes, in part to be consistent with students’ expectations of the platform, and partly because we believed that students would respond better to this than to a more formal approach. our most social media-savvy team members vetted and edited our posts, which served to ensure consistency and to mitigate the risks of using inappropriate memes or images. this was typically just a quick glance over the posts to ensure that tone and content were appropriate for both the platform and the audience. there is a genuine risk attached to using cultural references and memes without fully understanding them, and it is important to be aware of the nuances of how specific memes are used before posting to help manage risk.   metrics   the final work in the preparation stage of the project was to consider the tools we would use to assess the impact of the il posts. this requires a few definitions surrounding the metrics available on facebook, which are not always transparent.   ·         reach: the number of people who had any content from a page or post, or about the page or post, enter their newsfeed (“what’s the difference between page views, reach and impressions?” n.d.). it can be seen as a measure of how widely a message has been disseminated. ·         engagement: the number of actions whereby users “engaged” with the content of a post or page (“post engagement,” n.d.). watching a video, clicking a link, liking, commenting, sharing, etc., all constitute engagement. it can be seen as a measure of how interesting or useful the content is to the user, since most users are not motivated to “engage” with content that does not have value to them. ·         paid vs. organic: for most of facebook’s metrics, a distinction is made between paid and organic numbers. paid numbers have arisen from financial investment in advertising with facebook. organic numbers are those that have arisen without being paid for (“what’s the difference between organic, paid and post reach?” 2018). as there was no budget for this project, all numbers presented in this study are organic.   raw data was extracted from facebook at the end of each semester: june 12, 2017 and november 9, 2017, via facebook’s “insights” function. it has been reported that facebook posts attract the majority of their engagement within 24 hours of posting, a phenomenon we also observed in the course of this study (ayres, n.d.). therefore, the timing of data collection will not have impacted the measures of engagement.   analysis of results   we used reach and engagement as a proxy measure for success of the intervention and as an indication of the value students were placing on the content, as we could not directly measure the effectiveness of instruction. we hypothesized that students would engage more with content that was useful to them and that reach may also correlate with topics of potential interest. we decided to compare intervention posts with the general posts produced by the library, believing that our best chance of demonstrating value lay in comparing reach and engagement between non-intervention and intervention posts.   results   targeting individual courses and tagging relevant student clubs and university groups in posts resulted in greater median engagement and reach than other posts on our page.   table 1 shows the reach that the university of canterbury library facebook page posts had during semester one and semester two in 2017. semester one intervention posts reached a median of 1012 and a mean of 1106 people, compared to a median of 464 and mean of 613 people reached with the non-intervention posts during the same period. posts that were part of the intervention in semester two were less successful and reached a median of 530 and a mean of 531 people, compared to a median reach of 521 and a mean of 668 for non-intervention posts. overall, when looking at the median numbers, intervention posts in semester one reached 118% more people than non-intervention posts, which is significantly higher than in semester two, when intervention posts reached only 2% more people.     table 1 comparison of reach between intervention and non-intervention posts in semesters one and two semester one semester two intervention posts (n = 38) non-intervention posts (n = 404) percent change intervention posts (n = 30) nonintervention posts (n = 280) percent change median 1012 464 +118% 530 521 +2% mean 1106 613 +66% 531 668 -21%     figure 2 shows a comparison between semester one intervention posts and semester one non-intervention posts. there was greater variance among non-intervention posts, and semester one intervention posts reached a considerably greater readership than non-intervention posts.     figure 2 semester one reach: intervention posts vs. non-intervention posts. outliers have been removed to improve the readability of the figure.     figure 3 shows reach figures for all posts, intervention and non-intervention, for both semesters. in semester two, where specific groups were not targeted, there was little difference between the reach of intervention and non-intervention posts. viewing the data this way shows that the semester one targeted posts performed above all other groups.     figure 3 semester one and semester two reach comparison. outliers have been removed to improve the readability of the figure.     table 2 shows the median and mean engagements per post. intervention posts in semester one had a greater mean and median rate of engagement than non-intervention posts from both semesters, as well as intervention posts in semester two. in semester one, intervention posts had a 136% higher median engagement rate than non-intervention posts, while in semester two there was only a 67% increase in engagement between intervention and non-intervention posts.     table 2 comparison of engagement between intervention and non-intervention posts in semesters one and two semester one semester two intervention posts (n = 38) non-intervention posts (n = 404) percent change intervention posts (n = 30) nonintervention posts (n = 280) percent change median 26 11 +136% 15 9 +67% mean 42 20 +110% 19 24 −21%     figure 4 shows a comparison of intervention posts and non-intervention posts across both semesters. semester one intervention posts received greater engagement than other post types, with semester two intervention posts performing similarly to non-intervention posts in terms of engagement.       figure 4 semester one and semester two engagement: intervention posts vs. non-intervention posts. outliers have been removed to improve the readability of the figure.     table 3 indicates posts that tagged student clubs showed higher reach and engagement.  intervention posts where a club was tagged reached a median of 1262 people, compared to a median of 1012 for all intervention posts. the mean reach for the tagged intervention posts was 1803, a substantial increase from the 1106 people reached for all intervention posts. engagement on posts where we tagged student groups was also greater; the median engagement for intervention posts with clubs tagged was 46, and only 26 for all intervention posts.   there was one il post that had such large reach and engagement that it needed to be looked at more closely. figure 5 shows the median level of engagement a typical project post received, compared to that of this outlier post. during semester one, stage one engineering students are required to complete an assignment that requires use of library resources. the engineering subject librarians created a libguide that contained information and materials that could help students complete their assignment. the week before the due date, this guide was promoted on the university of canterbury library facebook page. relevant student clubs and departments were tagged in  the post, and it was shared by the engineering student society and the college of engineering facebook pages, eventually reaching 5159 people. there were 29 comments on the post, many of which were students tagging their friends to ensure they would see it. there were also instances of non-engineering students tagging friends or family members who were in the engineering course. even if the post was not directly relevant to the individual, they were tagging someone for whom it would be useful to ensure that they would not miss it. one hundred thirty-one people clicked on the link to the subject guide, and facebook recorded 456 “other clicks” (which are clicks not on the post content, but on the institutional page or a “see more” link). this post had 410% increased reach and 1538% increased engagement, compared to the median for all intervention posts.     table 3 comparison of reach and engagement between intervention posts and intervention posts that tagged clubs reach intervention posts clubs tagged (n =11) intervention posts (n = 38) median 1262 1012 mean 1803 1106 engagement intervention posts clubs tagged (n =11) intervention posts (n =38) median 46 26 mean 88 42     figure 5 engagement for outlier post compared with non-intervention, intervention, and tagged intervention posts in semester one.     discussion   our facebook followers clicked on, liked, and shared posts we describe as “spinach”—content that is informative rather than exciting—more than the content we normally post to develop community, market our services, and entertain. we propose that two factors influence the extent to which students pay attention to posts with il content: timeliness and specificity, and community.     figure 6 a student club sharing an intervention post on their facebook page.     figure 7 a complimentary letter regarding the library’s facebook page published in the student magazine.     timeliness and specificity   in semester one, we targeted particular student courses with timely content; in semester two, we targeted our content at peak assessment times for undergraduate students as a whole. the reach of the semester two posts was comparable to non-intervention posts, while semester one posts had a 118% higher median reach than non-intervention posts. the fact that semester one posts had greater reach than semester two posts indicates that while timeliness is important, it cannot easily be disentangled from specificity. providing il content “just-in-time” while targeting cohorts’ specific il needs was the most effective way to improve reach to students. this is a more labour-intensive means of providing il content via facebook, but it provides increased reach.   community positive community interaction with the intervention posts on facebook took two forms. first, student clubs and societies shared our posts after they had been tagged. second, broader communities of students’ friends and colleagues shared posts on their personal facebook timelines or tagged friends into our posts in comments. when clubs were tagged in posts, our median engagement increased by 77%.   we propose that tagging posts increased our reach beyond the newsfeed of followers of our institutional page. in addition, the peer recommendation implicit in a share adds authority to the content as the social capital of the tagger is added to the message. while it is difficult to quantify the effect of students tagging each other, it is not to be underestimated. for example, figure 6 shows a student society sharing a post with their community with positive feedback.   an unanticipated benefit was building positive relationships between the library and student clubs and societies. while we had some trepidation about how they would react to being tagged in posts, not only did they frequently re-share tagged posts, but we received feedback from clubs who appreciated our attention and the exposure the library provided. positive feedback also came from unexpected places, such as a letter published in the university of canterbury student magazine (figure 7). while only representative of one student’s opinion, it indicates that facebook can be an effective platform for providing il content.   workload   in order for the project to have as little impact as possible on existing staff workload, the materials we developed were deliberately informal, making use of screenshots annotated with handwriting and post-it notes. other options, such as creating professional infographics or high production standard videos were rejected on the grounds of cost and speed. we believe that students responded well to the informal content and that investing more time or resources would not guarantee a higher level of reach and engagement.   limitations and further study   our colleagues’ support of the library’s facebook page may have artificially raised our reach and engagements when they liked and shared posts. we think the effect of this is minor in terms of evaluating the success of our intervention because their engagement was equally spread over intervention and non-intervention posts. nonetheless, we would recommend that anyone replicating this research may want to set a policy for their own colleagues to not like or share their posts while they are testing the effect of their intervention. while we believe we have shown that students have an appetite for il content on facebook, we have not assessed whether students have improved il skills as a result of this intervention. future studies could test a sample of students before and after an intervention, or survey those that marked assignments to see if the students demonstrated improved il skills.   potential confounding factors for this study include ways students may have been drawn to our facebook posts other than tagging. for example, a suggestion from a tutor or lecturer to check them out, or coverage in student media could have increased our reach.   using facebook to deliver a service could be seen as inequitable since not all students use it. however, none of the il content delivered via facebook was unique to this platform. the library offers numerous other portals where this content can be accessed by any student; providing il via facebook does not disadvantage any other users and in fact offers an opportunity to reach students who may not have been reached through traditional library channels.   finally, consideration needs to be given to the facebook algorithm which determines the content of an individual’s feed; we have no control over it, and cannot say whether it benefited or hindered us. furthermore, facebook is constantly adjusting its algorithm with different goals than those of the library, so directly replicating this study would be difficult. while we believe facebook is a valuable platform for providing il content given the pre-eminence of social media in students’ lives, it is important to acknowledge that libraries relinquish an element of control in using this platform.   conclusion   students responded positively to il material delivered on facebook, provided that it was timely for assessment and targeted at specific groups. under circumstances where substantial, timely help is offered to a large cohort, this positive response was further amplified. we interpret these increases, particularly in engagement, to indicate that students perceived value in what was being provided and that including the wider university social community amplified our message and helped to build relationships.   the effect was not apparent when specific student groups were not targeted. under these circumstances, there was little deviation from the performance of non-intervention posts. our results suggest that, given timely, relevant, and specific il content, students will engage with il content on facebook.   references   aharony, n. (2012). facebook use in libraries: an exploratory analysis. aslib proceedings, 64(4), 358-372. https://doi.org/10.1108/00012531211244725 akcaoglu, m., & bowman, n. d. (2016). using instructor-led facebook groups to enhance students’ perceptions of course content. computers in human behavior, 65, 582-590. https://doi.org/10.1016/j.chb.2016.05.029   al-daihani, s. m., & abrahams, a. (2018). analysis of academic libraries’ facebook posts: text and data analytics. the journal of academic librarianship 4(2), 216-225. https://doi.org/10.1016/j.acalib.2018.02.004   annamalai, n. (2016). exploring the writing approaches in the facebook environment. teaching english with technology, 16(1), 71-87. retrieved from http://www.tewtjournal.org   ayres, s. (n.d.). shocking new data about the lifespan of your facebook posts. retrieved february 9, 2019, from https://www.postplanner.com/lifespan-of-facebook-posts/   bobish, g. (2011). participation and pedagogy: connecting the social web to acrl learning outcomes. the journal of academic librarianship, 37(1), 54-63. https://doi.org/10.1016/j.acalib.2010.10.007   bundy, a. (2004). australian and new zealand information literacy framework: principles, standards and practice. retrieved from http://archive.caul.edu.au/info-literacy/infoliteracyframework.pdf   burkhardt, a. (2010). social media: a guide for college and university libraries. college & research library news, 71, 10-12, 24. retrieved from https://crln.acrl.org/index.php/crlnews/issue/archive   catalano, a., & phillips, s. (2016). information literacy and retention: a case study of the value of the library. evidence based library and information practice, 11(4), 2-13. https://doi.org/10.18438/b82k7w   chen, m. m. (2018). students’ perceptions of the educational usage of a facebook group. journal of teaching in travel & tourism, 18(4), 332-348. https://doi.org/10.1080/15313220.2018.1434448   chu, m., & meulemans, y. n. (2008). the problems and potential of myspace and facebook usage in academic libraries. internet reference services quarterly, 13(1), 69-85. https://doi.org/10.1300/j136v13n01_04   chugh, r., & ruhi, u. (2018). social media in higher education: a literature review of facebook. education and information technologies, 23(2), 605-616. https://doi.org/10.1007/s10639-017-9621-2   click, a., & petit, j. (2010). social networking and web 2.0 in information literacy. international information and library review, 42(2), 137-142. https://doi.org/10.1016/j.iilr.2010.04.007   donlan, l. (2014). exploring the views of students on the use of facebook in university teaching and learning. journal of further and higher education, 38(4), 572-588. https://doi.org/10.1080/0309877x.2012.726973   education counts. (2017). provider-based enrolments [data set]. retrieved from https://www.educationcounts.govt.nz/statistics/tertiary-education/participation   godwin, p. (2009). information literacy and web 2.0: is it just hype? program: electronic library and information systems, 43(3), 264-274. https://doi.org/10.1108/00330330910978563   gonzález-ramírez, r., gascó, j. l., & taverner, j. l. (2015). facebook in teaching: strengths and weaknesses. the international journal of information and learning technology, 32(1), 65-78. https://doi.org/10.1108/ijilt-09-2014-0021   harrison, a., burress, r., velasquez, s., & schreiner, l. (2017). social media use in academic libraries: a phenomenological study. the journal of academic librarianship, 43(3), 248-256. https://doi.org/10.1016/j.acalib.2017.02.014   hyldegård, j. (2014). young learners’ use of social media for information seeking: exploring core lis journals from 2010-2014. libres: library and information science research electronic journal, 24(2), 108-117. retrieved from https://www.libres-ejournal.info/1404/   leier, v. m. (2017). learning language through facebook. international journal of computer-assisted language learning and teaching (ijcallt), 7(3), 40-57. https://doi.org/10.4018/ijcallt.2017070103   mack, d., behler, a., roberts, b., & rimland, e. (2007). reaching students with facebook: data and best practices. e-jasl: the electronic journal of academic and special librarianship, 8(2). retrieved from http://southernlibrarianship.icaap.org/indexv8.html   mauda, l., & kalman, y. m. (2016). characterizing quantitative measures of user engagement on organizational facebook pages. paper presented at the 49th hawaii international conference on system sciences, koloa, hi https://doi.org/10.1109/hicss.2016.442   omar, h., embi, m. a., & yunus, m. m. (2012). esl learners’ interaction in an online discussion via facebook. asian social science, 8(11), 67-74. https://doi.org/10.5539/ass.v8n11p67   nielsen. (2016). media trends 2016: how new zealanders consume newspapers, magazines, tv, radio & digital content retrieved february 9, 2019, from http://www.nielsen.com/nz/en/insights/reports/2016/new-zealand-media-trends-2016.html   palmer, s. (2014). characterizing university library use of social media: a case study of twitter and facebook from australia. journal of academic librarianship, 40(6), 611-619. https://doi.org/10.1016/j.acalib.2014.08.007   phillips, n. k. (2011). academic library use of facebook: building relationships with students. journal of academic librarianship, 37(6), 512-522. https://doi.org/10.1016/j.acalib.2011.07.008   “post engagement” (n.d.). retrieved february 9, 2019, from https://www.facebook.com/business/help/735720159834389   solomon, l. (2013). the librarian's nitty-gritty guide to social media (4th ed.). chicago, il.: ala editions.   soria, k. m., fransen, j., & nackerud, s. (2014). stacks, serials, search engines, and student success: first-year undergraduate students library use, academic achievement, and retention. journal of academic librarianship, 40(1), 84-91. https://doi.org/10.1016/j.acalib.2013.12.002   soria, k. m., fransen, j., & nackerud, s. (2017). beyond books: the extended academic benefits of library use for first-year college students. college & research libraries, 78(1), 8-22. https://doi.org/10.5860/crl.78.1.8   towner, t. l., & lego muñoz, c. (2011). facebook and education: a classroom connection? in c. wankel (ed.), educating educators with social media (pp. 33-57). https://doi.org/doi:10.1108/s2044-9968(2011)0000001005   university of canterbury. (n.d.). university ranking. retrieved from https://www.canterbury.ac.nz/get-started/why-uc/prestigious/top-uni/university-ranking/   “what is tagging and how does it work?” (n.d.). retrieved february 9, 2019, from https://www.facebook.com/help/124970597582337/   “what’s the difference between organic, paid and post reach?” (2018). retrieved february 9, 2019, from https://www.facebook.com/help/285625061456389?helpref=uf_permalink   “what’s the difference between page views, reach and impressions?” (n.d.). retrieved february 9, 2019, from https://www.facebook.com/help/274400362581037?helpref=uf_permalink   winn, d., groenendyk, m., & rivosecchi, m. (2015). like, comment, retweet: identifying students’ social media preferences for receiving library communications. partnership: the canadian journal of library and information practice and research, 10(2), 1-14. https://doi.org/10.21083/partnership.v10i2.3449   witek, d., & grettano, t. (2012). information literacy on facebook: an analysis. reference services review, 40(2), 242-257. https://doi.org/10.1108/00907321211228309   wright joe, j. (2015). assessment of social media in the library: guidelines for administrators. journal of library administration, 55(8), 667-680. https://doi.org/10.1080/01930826.2015.1085251   zhu, q. (2016). the application of social media in outreach of academic libraries’ resources and services. library hi tech, 34(4), 615-624. https://doi.org/10.1108/lht-05-2016-0055     evidence based library and information practice evidence based library and information practice 2009, 4.4     evidence based library and information practice         using evidence in practice    services in a changing academic library:  patron feedback and library response    bruce stoffel  reference coordinator and associate professor  milner library, illinois state university  normal, illinois, united states of america  email: brstoff@ilstu.edu    received: 12 june 2009          accepted: 07 october 2009      © 2009 stoffel. this is an open access article distributed under the terms of the creative commons attribution  license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and  reproduction in any medium, provided the original work is properly cited.      setting  illinois state university is a comprehensive  research institution enrolling 20,856 (fall 2009)  including 18,344 undergraduates and 2,512  graduate students. graduate degree offerings  include both master’s and doctoral studies.  the university began in 1857 as a teacher  training school and became a comprehensive  university in 1964. the university employs  3,200 including approximately 1,000 faculty  members. milner library is a six‐story facility  that opened in 1976 and serves as the single  library facility on campus accessible to  students, faculty, and staff. library holdings  total approximately 1.5 million volumes. floor  1 (basement) is configured with staff offices  and closed storage. floors 2‐6 are public areas  with a mix of shelving, study areas, and group  study rooms.    problem    from the opening of milner library in 1976  through summer 2007, the library was zoned  by floor according to broad subject areas. each  floor functioned much like a separate library,  with its own monographs, serials, reference  holdings, reference services, offices, and staff.  in 2006‐2007, library administration developed  a plan to reorganize collections and service  points. the plan was a response to  deteriorating physical conditions on floor 1  (the education, psychology, and teaching  materials library), patron demand for  additional study space, and decreasing  patronage of reference service points. in  advance of the fall 2007 term, reference  materials and services were consolidated on  the main floor (floor 2). lesser‐used library  materials were relocated from upper floors to  a new closed storage area in the basement  (floor 1). emptied shelving areas on upper  floors were then removed to create additional  seating and study spaces.      library staff members were concerned about  the impact of these changes on library patrons.  specifically, staff members were concerned  whether patrons were satisfied with reference  service consolidation and changes to the  library physical environment.       84 mailto:brstoff@ilstu.edu http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2009, 4.4  evidence    a reference working group was organized in  early spring 2008 to survey the campus  community about these issues. research  approval was obtained from the campus  institutional review board.  authorization was  received from campus administration to  contact students, faculty, and staff with an e‐ mail message that included a link to an online  survey (on surveymonkey). the survey was  available during the last two weeks of april  2008. two e‐mail notices were sent to  prospective respondents, an initial invitation  and a reminder. no other methods were used  to stimulate response. participation in the  survey was voluntary.      our goal was a 100% survey of the students,  faculty, and staff. however, our campus  policy allows members of the campus  community to block research solicitations sent  by e‐mail. consequently, we sent e‐mail  invitations to approximately 70% of the  campus community, or about 17,000 persons.  to save time we did not use stratified  sampling techniques nor did we use  techniques to limit respondents to one  response. as a result, the profile of both e‐mail  recipients and respondents did not precisely  match the profile of the campus community at  large. although a few respondents may have  completed the survey more than once, our  close scrutiny of responses to open‐ended  survey questions did not identify duplicate  entries.     we asked both closed‐ and open‐ended  questions. regarding reference, we asked  which reference services respondents were  aware of and used, the reasons why  respondents asked for help from library staff,  and how satisfied they were with assistance  they received. to probe opinions about  physical changes, we asked respondents to  rate cleanliness, lighting, study space,  computers, furnishings, and security. because  opportunities to conduct campus‐wide  surveys are limited, we took this opportunity  to also ask questions about research behaviour  and library services other than reference.    of the persons solicited, 2,152 started the  survey and 1,815 completed it, for a return rate  of approximately 10%. respondents were  almost entirely library users. ninety‐six  percent indicated that they come to the library  at least once a semester, and 97% indicated  that they use the library web site at least once  a semester. we were pleased with these  numbers, because our primary interest was  receiving feedback from library users rather  than non‐users.    respondents expressed high levels of  familiarity with desk reference services (95%)  but significantly lower levels with telephone  (41%), e‐mail (49%), and im (40%).   differences between desk and other reference  services in terms of usage were more  pronounced.  eighty‐nine % of respondents  indicated having asked questions at the  reference desk, but only 18 % indicated having  asked by telephone, 23% by e‐mail, and 12%  by im. we were particularly concerned with  the lower figures for remote reference services  but of their potential use in serving patrons  working on floors no longer served by  reference desks.      ninety‐five percent of respondents indicated  that they usually or always find help in the  building when they need it, and 90%  expressed satisfaction with availability of staff  members at the time of their need. we did not  directly ask about closure of service points but  anticipated receiving comments about this in  response to two open‐ended questions: one  asking patrons to describe a negative library  experience, and another asking patrons what  they would do to make the library more  helpful to them. of the 948 negative  experiences described by survey respondents,  40 (or approximately 4%) related to  availability of reference services. of the nearly  900 suggested changes, 36 related to restoring  service points on all floors (also approximately  4%). we concluded from these numbers that  while improvement was possible in providing  reference services remotely to persons on our  upper floors, dissatisfaction with reference  consolidation was not great enough to warrant  reversing the consolidation.  85 evidence based library and information practice 2009, 4.4  a majority of respondents indicated being  satisfied with all aspects of the library physical  environment we asked them to rate. however,  amount of group study space received the  lowest satisfaction rating (62%). levels of  dissatisfaction were highest for furnishings  (20%) and the amount of quiet study space  (19%). twenty‐four percent of survey  respondents choosing to identify a negative  library experience cited excessive building  noise.  twelve percent of respondents who  suggested a way to make the library more  helpful to them mentioned better separation or  enforcement of quiet study space. an  additional 12% of the suggested changes  related to increasing study space or study  rooms, but it was unclear from the responses  whether respondents were concerned with  spaces for quiet or group study. concern  about building noise was likely heightened  because we conducted the survey right before  our finals week. however, we concluded from  these responses a need for actions to reduce  noise levels in the building, particularly in  designated quiet study areas and particularly  during peak study weeks.     implementation    a primary goal resulting from the survey was  to increase familiarity with and use of remote  reference services available to patrons seeking  assistance while on unstaffed floor,  particularly im. at the time we were  advertising our im screen name on our  website. in late spring 2008 we organized a  working group to review our im reference  service. the result of this review was  introduction of an im widget accessible from  all library web pages beginning in august  2008.  the reason for this change was to allow  patrons using our web site to communicate  with reference desk staff without having to log  into their im accounts or use cell phone  minutes.  about this same time we also  modified the presence of our reference  services on the library web site from a small  “ask a librarian” link to a set of “contact us”  buttons on the upper left side of all library  web pages. separate buttons were added for  im, phone, e‐mail, and in person. in fall 2009  we added a texting option to our reference  service. we began distributing business cards  with telephone and texting instructions to  persons leaving the reference desk for upper  floors, to aid them in remotely contacting the  desk for follow‐up service.    we also took action in response to concerns  expressed by survey respondents about noise.   throughout the fall 2008 semester we  promoted availability of designated quiet  study areas by adding signage, advertising the  spaces in library promotional materials, and  urging desk staff to inform patrons of the  areas if asked about places to study in the  building. while these changes seemed well  received by patrons, we continued to receive  noise complaints during final exams week. for  spring 2009 final exams we took a more  aggressive approach with the goal of further  reducing noise complaints. we temporarily  zoned the library into quiet and group study  areas by floor. we designated floors 2 and 3 as  group study floors, largely because they are  open to one another in their corners, allowing  noise bleed between floors. we designated  floors 4 and 5 quiet study areas for the same  reason. to alert patrons to this temporary  scheme we placed multiple signs and table  tents throughout the building and posted  announcements on the library web site,  campus web portal, campus television  network, and electronic message boards in the  building. on the library homepage we  featured a one‐minute video of our student  government president urging cooperation  with the policy. library staff members  periodically roamed quiet floors to ask  violators for their cooperation.      outcome    during the first ten weeks of the fall 2008  semester, our im reference transactions  totalled 454 compared with 93 during the first  ten weeks of fall 2007 (an increase of 388%).  we attributed this increase at least in part to  adoption of the im widget and enhanced  visibility of the service on library web pages.  while we were unable to track the location of  persons asking questions through the service,  86 evidence based library and information practice 2009, 4.4  87 desk staff reported numerous instances of  persons contacting the service from locations  within the library. in addition, between the  first ten weeks of fall 2007 and fall 2008, our  telephone reference transactions increased  35%. we attributed at least part of this increase  to increased visibility of telephone reference  on library web pages. data for fall 2009,  including figures on usage of our new text  reference service, are not yet available.     with regard to our actions to reduce library  noise, we have received numerous positive  comments and messages from patrons and  from campus administrators. no negative  comments have been received. however, we  feel that a formal evaluation is warranted.      reflection    the patron survey provided us valuable  feedback regarding changes to our services  and environment. the relatively large number  of persons surveyed and responding helped  validate our subsequent actions, likely more so  than if we had instead conducted small focus  groups.    it was encouraging and validating to receive  positive responses. feedback regarding our  remote reference services provided the  impetus for change that will hopefully further  enhance patron satisfaction levels. the level of  concern about building noise surprised many  library staff members. we would not likely  have known of the extent of the problem  otherwise. implementing the zoned study area  trial was likely easier and more effective  having patron feedback than if we had not  conducted the survey. we used survey data as  a primary justification for our actions. we  prominently mentioned survey responses  when promoting the trial, which we hope  furthered patron cooperation. we stressed  survey results when asking cooperation from  our study body president.      because our patrons and services continue to  change, at a seemingly faster pace, we now  plan to conduct patron surveys on a more  routine basis. but because of the effort  necessary to plan and implement a survey of  this size and especially the time involved with  analyzing responses, conducting such a  survey annually would be too taxing for us.  we plan instead to conduct a comprehensive  patron survey every two years beginning in  spring 2010. we also intend to use more  advanced survey and analysis techniques,  such as stratified sampling, to improve  reliability and validity of survey results. we  intend to utilize the services of a research  consultant to help improve to survey design,  implementation, and analysis.      acknowledgement    the author acknowledges the work of illinois  state university colleagues christine kubiak  diane mather, julie murphy, sharon naylor,  and jenna self on design and implementation  of the survey described in this article.       evidence summary   refugee youth leverage social, physical, and digital information to enact information literacy   a review of: lloyd, a., & wilkinson, j. (2017). tapping into the information landscape: refugee youth enactment of information literacy in everyday spaces. journal of librarianship and information science. advance online publication. http://dx.doi.org/10.1177/0961000617709058   reviewed by: rachel elizabeth scott integrated library systems librarian university libraries university of memphis memphis, tennessee, united states of america email: rescott3@memphis.edu   received: 23 aug. 2017 accepted: 5 dec. 2017      2017 scott. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to describe the ways in which refugee youth use everyday information to support their learning.   design – photo voice technique, a process by which the population under consideration is provided with cameras and asked to visually document an aspect of their experience.   setting – social agency in new south wales, australia   subjects – fifteen 16-25 year old refugees resettled from south sudan or afghanistan   methods – three workshops were conducted. in the first, participants learned how to use the cameras and the protocols for participation. between the first and second workshops, participants took several photographs of places, sources and types of information that were personally meaningful. in the second workshop, participants were first split into small groups to share and discuss the five images that they selected as their most important information sources and later reconvened as a large group in which participants again shared and discussed their images. in the third and final workshop, the authors shared their findings and analysis with the participants and invited discussion. the authors analyzed both photos and group transcripts from the workshops using charmaz’s constant comparative method.   main results – refugee youth use digital, vernacular, meditational, and visual literacies in everyday settings in to order to understand and create their new information landscapes. information literacy enactment is agile and responsive to context.   conclusion – engaging with digital, vernacular, and visual information in a variety of contexts is central to how young refugees (re)form their information landscapes.   commentary   this grant-funded paper continues the authors’ previous work on refugee youth learning in everyday spaces (lloyd & wilkinson, 2016). lloyd and wilkinson are academics affiliated with library and information science and educational leadership programs, respectively. their collaborative research is situated within community informatics and not librarianship. the approach is interdisciplinary and much of the literature cited is from wide-ranging disciplines, including refugee studies and education. the authors noted a gap in the literature on how refugee youth interact with information and identified the work of fisher, yefimova, and yafi (2016) is an exception; those authors studied how young refugees use information to help their own refugee community.   lindsay glynn’s eblip critical appraisal checklist (2006) provides questions that will be used to evaluate the methodology and reporting of this article. the study’s small population of fifteen young refugees were recruited via local social agencies. participants were excluded based on their facility with english language and comprehension; participants were asked to discuss the study details after having the study described to them. the authors indicated that the participants are from “culturally and linguistically diverse backgrounds (cald)” but do not provide further discussion of cald. all participants reported their country of origin as south sudan or afghanistan. the study was approved by the university ethics committee, but there is no description of the consent obtained.   data collection methods were clearly described and the photo voice technique was especially appropriate considering that english was not the native language of study participants. the description of the second and third workshops lacked details, which would make this study difficult to replicate. it is unclear, for example, what role the authors played in both the large and small group sessions in workshop two. there was also no description of how the discussions in the third workshop informed the results or ensured accuracy.   the primary theme identified through analysis was “tapping in,” or the ways to “recognize, locate and draw information from the host community or country” (lloyd & wilkinson, 2016). the findings section effectively quoted study participants to explain how they “tapped in” to various information sources. the article focuses on information behavior; the use of the phrase “information literacy” in the title might be misleading for librarians anticipating an article more closely related to the american library association definition of information literacy.   many practitioners will not have the opportunity to work with or offer services to refugee youth. this article’s positive contribution to the literature is the successful application of photo voice technique within information science. this methodology could be used to study and improve user experience design, among other library and information science applications.   references   fisher k. e., yefimova, k., &yafi, e. (2016). ‘future’s butterflies’: co-designing ict wayfaring technology with refugee syrian youth. in: idc ’16: proceedings of the 15th international acm chi conference on interaction design and children. (pp.25–36). new york: acm.   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154   lloyd, a. & wilkinson, j. (2016). knowing and learning in everyday spaces (kalieds): mapping the information landscape of refugee youth learning in everyday spaces. journal of information science, 42(3): 300–312. https://doi.org/10.1177/0165551515621845     evidence summary   aau library directors prefer collaborative decision making with senior administrative team members   a review of: meier, j. j. (2016). the future of academic libraries: conversations with today’s leaders about tomorrow. portal: libraries and the academy, 16(2), 263-288. retrieved from http://muse.jhu.edu/article/613842   reviewed by: carol perryman associate professor texas woman’s university denton, texas, united states of america email: cp1757@gmail.com   received: 28 feb. 2017  accepted: 7 apr. 2017      2017 perryman. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to understand academic library leaders’ decision making methods, priorities, and support of succession planning, as well as to understand the nature, extent, and drivers of organizational change.   design – survey and interview.   setting – academic libraries with membership in the association of american universities (aau) in the united states of america and canada.   subjects – 62 top administrators of aau academic libraries.   methods – content analysis performed to identify most frequent responses. an initial survey written to align with the association of research libraries (arl) 2014-2015 salary survey was distributed prior to or during structured in-person interviews to gather information about gender, race/ethnicity, age, time since terminal degree, time in position, temporary or permanent status, and current job title. 7-question interview guides asked about decision processes, strategic goals, perceived impacts of strategic plan and vision, planned changes within the next 3-5 years, use of mentors for organizational change, and succession planning activities. transcripts were analyzed to identify themes, beginning with a preliminary set of codes that were expanded during analysis to provide clarification.   main results – 44 top academic library administrators of the 62 contacted (71% response rate) responded to the survey and interview. compared to the 2010 arl survey, respondents were slightly more likely to be female (55%; arl: 58%) and non-white (5%; arl: 11%). approximately 66% of both were aged 60 and older, while slightly fewer were 50-59 (27% compared to 31% for arl), and almost none were aged 40-49 compared to 7% for the arl survey. years of experience averaged 33, slightly less than the reported arl average of 35. requested on the survey, but not reported, were time since terminal degree and in position, temporary or permanent status, and current job title.   hypothesis 1, that most library leaders base decisions on budget concerns rather than upon library and external administration strategic planning, was refuted. hypothesis 2, that changes to the academic structure are incremental rather than global (e.g., alterations to job titles and responsibilities), was supported by responses. major organizational changes in the next three to five years were predicted, led by role changes, addition of new positions, and unit consolidation. most participants agreed that while there are sufficient personnel to replace top level library administrators, there will be a crisis for mid-level positions as retirements occur. a priority focus emerging from interview responses was preparing for next-generation administrators. there was disagreement among respondents about whether a crisis exists in the availability of new leaders to replace those who are retiring.   conclusion – decisions are primarily made in collaboration with senior leadership teams, and based on strategic planning and goals as well as university strategic plans in order to effect incremental change as opposed to wholesale structural change.   commentary   with growing involvement in research support and other trends, little is known about the drivers of decision-making of academic library leaders, the focus of this inquiry. two critical evaluation checklists were used in the preparation of this evidence summary (glynn, 2006; perryman & rathbun-grubb, 2014).   a good background for the study is provided, although slight confusion arises due to terminology, with references to library directors, senior administrators, and top administrators without a differentiating definition.   demographic results were comparable to prior arl data, although slightly higher age and experience averages for directors were shown. seven of nine interview questions were included in the publication, enhancing replicability, while the other questions emerged during interviews. question 1 conflates two issues (“how do you make decisions about your organization’s future, both regarding library direction as a whole and your organizational structure?”), which is problematic, because respondents may not have responded to each the same way. of additional concern, neither “major” nor “incremental” change was defined, and the reader cannot discern whether definitions were provided to interviewees. content analysis specifics were not provided in response to open-ended questions.   thematic content analysis was performed by the author alone, and did not include an independent coder. while the author mentions this as a limitation, the rationale that this decision supports participant anonymity is not sufficiently convincing: transcripts could have been anonymized prior to independent coding. in the absence of this step, readers would benefit from category definitions and examples. responses to each question are shown in detailed tables, but were frequently counted in more than one category. it is reasonable to expect that decisions use more than one form of input, but in some instances, knowing (in question 1, for example), that a library director employed collaborative decision making as well as strategic plans and goals would enhance understanding. sole reliance upon frequencies does not provide information about priorities or unique contexts for decision-making. as well, the author found it surprising that only three respondents reported using data to support decision making. however, further questioning on this point might have revealed that “strategic planning and goals” (or other categories) were themselves supported by data. findings from the study could be expanded by looking at each participant as an individual case, in order to consider elements affecting decisions for library leaders.   the snapshot of university library directors’ decision-making processes and priorities provided by interview transcripts updates prior studies, and identifies changes from budgetto mission-driven strategic management planning. additionally, new information on the use of the arl leadership fellows program, mentoring, and other professional development opportunities used to prepare future academic librarians and future deans is provided.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi:10.1108/07378830610692154   perryman, c. & rathbun-grubb, s. (2014). the cat: a generic critical appraisal tool. in jotform – form builder. retrieved 26 february 2017 from http://www.jotform.us/cp1757/thecat     research article   what are they doing anyway?: library as place and student use of a university library   angelica ferria curator university libraries university of rhode island kingston, rhode island, united states of america email: aferria@uri.edu   brian t. gallagher associate professor university libraries university of rhode island kingston, rhode island, united states of america email: bgallagher@uri.edu   amanda izenstark associate professor university libraries university of rhode island kingston, rhode island, united states of america email: amanda@uri.edu   peter larsen associate professor university libraries university of rhode island kingston, rhode island, united states of america email: plarsen@uri.edu   kelly lemeur learning commons librarian university library roger williams university bristol, rhode island, united states of america email: klemeur@rwu.edu   cheryl a. mccarthy professor emerita graduate school of library and information studies university of rhode island kingston, rhode island, united states of america email: chermc@uri.edu   deborah mongeau professor university libraries university of rhode island kingston, rhode island, united states of america email: dmongeau@uri.edu   received: 8 nov. 2016    accepted: 10 feb. 2017        2017 ferria, gallagher, izenstark, larsen, lemeur, mccarthy, and mongeau. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective to determine student use of library spaces, the authors recorded student location and behaviors within the library, to inform future space design.   methods the case study method was used with both quantitative and qualitative measures. the authors had two objectives to guide this assessment of library spaces:  1) to determine what library spaces are being used by students and whether students are working individually, communally, or collaboratively and 2) to determine whether students use these spaces for learning activities and/or social engagement.   results after data collection and analysis, the authors determined students are using individual or communal spaces almost equally as compared with collaborative group spaces. data also revealed peak area usage and times.   conclusion observed student individual and social work habits indicate further need for spaces with ample electrical outlets and moveable tables. further study is recommended to see whether additional seating and renovated spaces continue to enhance informal learning communities at uri and whether the library is becoming a “third place” on campus.     introduction    in 2008, bennett defined information commons as spaces in libraries with technology that support individual learning and learning commons as spaces in libraries that impact or enhance the learning experience by enacting the institutional mission through collaborative partnerships with “academic units that establish learning goals for the institution” (bennett, 2008, p. 183).  in 2011, the university of rhode island (uri) redefined its library, rebranding the university library with the name robert l. carothers library and learning commons (the library). the university of rhode island is a public land, sea, and urban grant institution, offering bachelors, masters, and doctoral degrees, with three campuses across the state. the library is located on the main campus in kingston, ri. of uri’s nearly 17,000 undergraduate and graduate students, approximately 6,700 live on campus (uri communications and marketing, undated).   while the library’s mission to acquire, organize, preserve, and provide access to resources in all formats and provide instruction in their use has remained constant, its role on the kingston, ri, campus requires new and evolving ways of thinking about its physical spaces. the library’s spaces have evolved into places of individual intellectual inquiry as well as collaborative engagement where students connect with others to build shared learning communities.   academic library planners have begun to embrace the notion of creating welcoming shared learning community spaces where users connect informally and the library can become the third place on campus. ray oldenburg, in his book the great good place (1991), defined the third place in a community as a place that provides the diversity of human contact where people come together to connect and build a shared community when not at home (first place) or work (second place). arguably, academic libraries can become that third place on campus, with spaces that welcome a diversity of human contact that nurtures growth when outside the classroom (first place) or campus housing (second place). the library as the third place can enrich campus life, create a sense of belongingness, and support the institutional mission of lifelong-learning. thus, the library spaces at uri, were assessed for their impact on how students are using library spaces by identifying what spaces are used and whether students work individually, communally, or collaboratively.   literature review   the evaluation of the academic library as place, and specifically its impact on learning, has challenged the library profession, administrators in higher education, and accreditation agencies. joan lippincott of the coalition of networked information (cni) stated in an interview: “i’d like to challenge the notion that brand-new, beautiful learning spaces in and of themselves can change learning. i believe that it has to be a combination of the space and the pedagogy and the technology” (lippincott, van den blink, lewis, stuart & oswald, 2009, p. 10). lippincott (2006) advocated making managerial decisions in libraries based on assessment data that measures the effectiveness, efficiency and extensiveness of learning spaces in libraries. there is growing concern for universities to evaluate their library facilities, services, technology, and information resources to determine the impact on student learning and how libraries support the research and public service mission of the institution.   according to fox and doshi (2013), group spaces are growing. additionally, diller (2015) identified that study areas are the second highest used library spaces. khoo, rozaklis, hall, and kusunoki (2016) commented on redesigned library spaces to encourage group interaction where talking, moving around, and moving furniture is acceptable.   the advent of digital tools and resources as well as pedagogical shifts that emphasize collaboration, creation, and student centered learning have changed the library landscape. libraries have responded to calls for user-centered learning with good reason; student-centered learning is social—active and interactive (foster & gibbons, 2007). in that tradition, montgomery (2014) explained: “the importance of library space is shifting from the content on our shelves to how students use and learn in our space” (p. 71). trying to remain relevant, libraries allocate and reallocate space in recognition of the pedagogical shift toward interaction among learners (jackson & shenton, 2010) by becoming physical and virtual platforms for knowledge creation.   at the same time, there are those who want the academic library to honor its historical mandate as a place for quiet study and contemplation. gayton (2008), in particular, supports this role for the library by pointing out that, in spite of its diminished importance as a storehouse and access point, gate counts have remained steady. similarly, demas (2005) emphasized the library’s cultural roles. gayton and demas urge decision makers not to throw out the baby with the bathwater. gayton (2008) clarifies,   there is a profound difference between a space in which library users are engaged in social activity and a space in which they are engaged in communal activity. social activity in a library involves conversation and discussion among people, about either the work at hand or more trivial matters. communal activity in a library involves seeing and being seen quietly engaged in study (p. 61).   there is value to learning that takes place independently or communally in a shared space; it is a privilege students do not want to risk losing.   yoo-lee, tae, and velez (2013) found that students responded to two survey questions with contradictory preferences for library spaces: “37 percent of the participants chose quiet study spaces and 28 percent, social spaces. however, 35 percent of them responded that they used both quiet spaces and social spaces almost equally” (p. 503).   looking at the quantitative results of space studies introduces notions of capacity and occupancy that warrant consideration. applegate (2009) noted, “previous observations had shown that unaffiliated people (people not arriving together or working in a group) almost never preferred to sit right next to each other, so an area might reach ‘full’ comfortable use at 50% of maximum capacity” (p. 343). in their discussion about a place and space survey khoo et al. (2016) elaborated on this point: “thus, while seating availability is initially evidenced by an empty table, this availability is reduced incrementally and ambiguously,  . . . in agreement with gibbons and foster, this study suggests that tables may be perceived to be ‘full’ when only approximately 50 percent of the seats at each table are occupied” (p. 7).   khoo et al. (2016) advocated the use of mixed methods when studying library spaces. montgomery (2014) and holder and lange (2014) both used mixed-methods successfully. as holder and lange argued, “using survey and observation methods together provided a more complete picture of user satisfaction with the spaces, as well as user preference for particular areas and furniture types” (p. 8).   hall and kapa (2015) found in their study at concordia university that some students prefer to work in isolation, as illustrated by one of their survey responses: “more single study spaces. not beside desks or other people” (p. 14). this is consistent with applegate’s (2009) study where 30-40% of group study room users were individuals, despite signage encouraging group use.  as planning for spaces goes forward, it is worth considering the value of offering rooms for individuals versus space intended for groups, or using “territorial dividers” to subdivide groups as recommended by i̇mamoğlu and gürel (2016, p. 65).   aims   embracing the concept of the third place along with bennett’s 2008 definition of the library as learning commons, the library administration at uri assembled a team of librarians and staff during the 2014-2015 academic year to examine the evolution of library spaces to assess how the new spaces are being used and whether the library is becoming the third place on campus. the assessment team hoped to identify student preferences for type of seating and level of engagement through the behavior and activities observed. students were not asked their preferences, however we could identify the most heavily used spaces and times as well as how students were using them for individual, communal, or group activities on each level (i.e., lower level, first floor, second floor, or third floor).   the librarians used the following research questions as guides:   1.       what library spaces are being used by students and are students working individually, communally, or collaboratively? 2.       how do students use these spaces for learning activities and/or social engagement?   methods   the case study methodology used both qualitative and quantitative measurements to assess the overarching research questions. the assessment team recorded sweep counts and unobtrusive observations on maps and coding sheets and examined aggregated usage statistics including gate counts to get a complete picture of library use.   the assessment team performed sweep counts of students using the library spaces for one week at the end of two semesters, fall semester (december 1-7, 2014) and spring semester (april 25-may 1, 2015), three times a day (10 a.m.-12 p.m., 2-4 p.m., and 8-10 p.m.). the sweep counts identified the number of students using the library as well as the activities of those students for each day and time. activity codes included reading, writing, using devices, studying in groups, and using movable white boards. the assessment team also observed behavior: individual, communal, or group study. team members submitted the coded information sheets and key personnel created excel spreadsheets to compile the numbers and highlight comparisons of times, days, and semesters to determine peak use times. no identifying information about participants was recorded and thus, user privacy was protected.   in assessing the use of space, the uri assessment team devised a strategy consistent with mccarthy and nitecki (2011), given and leckie (2004), and applegate (2009). the uri researchers identified the use of library space with sweep counts and structured observations of activities and behaviors. the uri researchers recorded information directly on maps and coding sheets with predetermined categories similar to coders in other studies (may, 2011; mccarthy & nitecki, 2011).   quantitative assessment measures   1. what library spaces are being used by students and are they working individually, communally, or collaboratively?   the team identified space use by counting and recording the number of people occupying seats in the various areas (e.g., tables, group study rooms, informal spaces such as soft seating, and the 24 hour room) on all four levels of the library for each day and time slot during the two sweep count weeks. library personnel created excel spreadsheets from the coded data sheets to show occupancy rates, and the assessment team analyzed the combined data to determine the most heavily used seating areas, peak times of use, and how spaces were being used.   qualitative assessment measures   2.     how do students use these spaces for learning activities and/or social engagement?   the assessment team observed and recorded activities on coding sheets for each time period and date to identify students’ activities and behaviors, to record how the spaces appeared to enhance informal learning communities. these coding sheets were compiled into spreadsheets to compare observations of activities and behaviors such as reading, writing, and using devices and to identify commonalities using content analysis. observers determined whether students were engaged individually, communally (working alongside), or collaboratively (working together in groups) as well as their activities and behaviors. the assessment team analyzed these findings individually and collectively for relations between the two semesters, times of day, days of the week, levels of the building, and so on to determine the effectiveness of the library’s environment in building a shared learning community.     table 1 the library floor level identification floor location atmosphere/behavior noise level furnishings lower level mostly individual study, some flexible use quiet, soft voices carrels, some small tables first floor/ main floor meet and greet, constant motion, café in the 24 hour study room, learning commons spaces, group study rooms, presentation room, and collaborative spaces with whiteboards and flat screens for projection, as well as moveable furniture and roving white boards conversation, collaboration, mall or busy lobby grouped soft seating, high top bar seating, café tables, booths, moveable tables and chairs with wheels, second floor group work or communal study at tables alongside others, flexible use with roving whiteboards, group study rooms and graduate carrels (small rooms) conversation, café style seating moveable tables and chairs on wheels, bar seating, some carrels and some soft seating, group study rooms third floor library designated quiet zone silent carrels and tables     results and discussion   student use of spaces by floor   tracking student occupancy by floor is only one aspect of measuring use of space. another method is to measure use of space by specific location, time of day, and number of seats available. in this study, discerning students’ choices of seating may be influenced by segregation of library atmosphere and noise level by physical floor level as well as by flexible furnishings. the exception is the third floor, which the library has designated as a quiet zone. enforcement is primarily self-policing by other users. table 1 offers a brief snapshot of each floor, its atmosphere, and behaviors identified.   as the total number of seats varies greatly by floor, preferred use was measured by number of seats filled as compared to number of seats available on each floor. counts provided a clear picture of preferred seating across various floors by both day of week and time of day. although the percentage of seats actually taken may be one-third or one-half full, the actual number of tables occupied appears to be a full house. there may only be one or two students at a table with four to six seats. students arriving unaccompanied seemed reluctant to approach an already-occupied but not fully-used table, unless they knew the occupants. this is consistent with what applegate (2009) and khoo et al. (2016) observed in their studies.   the relatively high occupancy of first floor seating can be explained by the newly renovated learning commons area with the highly popular booths (with 1-4 students), flexible and moveable tables and seats, curtained areas, café-style tables, laptop-bar high seating, and a 24 hour room with a café where students frequently meet and greet and wait for their next class, or utilize their own electronic devices as well as library materials and white boards. thus, the first floor areas including the learning commons and the 24 hour room, appear fully occupied throughout the day and evening.  table by table, however, occupancy was approximately 30% of the seats occupied with an increase in seat occupancy between 2-4 p.m.     table 2 behavioral use of library spaces, by floor   date is/communal gs/social lower level december 2014 60.9% 39.1% april 2015 54% 46% first floor december 2014 48.2% 51.8% april 2015 51.2% 48.8% second floor december 2014 40.1% 59.9% april 2015 41.6% 58.4% third floor december 2014 69.8% 30.2% april 2015 71.1% 28.9% average for all floors december 2014 52% 48% april 2015 47.8% 52.2%     the lower level and third floors had the least amount of students occupying seats and they also do not have as much seating nor have moveable tables or seats. both levels are used primarily for quiet study or individual work in carrels and thus, may explain the significant difference in variation of seating by floor. observers noted that, where carrels were placed side-by-side, students showed a reluctance to take a seat next to an occupied carrel.   the first floor sometimes had double or triple the occupancy of the next highest used floors, with a peak usage from 2-4 p.m. on monday through friday. the second and third floors were the next highest in use. occupancy of these floors typically varied by less than twenty users (second floor being slightly higher) with patterns of occupancy that tended to move in tandem. like the first floor, peak time was 2-4 p.m. daily monday through friday. the lower level was by far the least used floor, with only half the use of the second and third floors. unlike the rest of the building, use of the lower level remained moderately steady, with variations seldom rising or falling more than 15 students between scheduled counts. saturday occupancy grew steadily across all floors for time periods measured while sunday’s use spiked at 4-6 p.m. in may but in december the numbers grew steadily throughout the day.     figure 1 carothers library occupancy by floor, day, and time for fall 2014.     figure 2 carothers library occupancy by floor, day, and time for spring 2015.     in summary, first through third floor use was consistent comparing both semesters, with heaviest use from 2-4 p.m. monday-friday. lower level floor use was steady throughout all the observation periods although the number were the least. saturday use was steady across all floors with a small spike from 4-6 p.m. sunday use in december showed a steady increase during the day and night, but in may, use spiked from 2-4 p.m. the december count (possessing greater variations) clearly aligns with the fact that classes were still in session, while the april count had less drastic variations with may 1 as a reading day prior to the start of exams.   while analyzing occupancy numbers by day of the week tends to support the observations drawn from table 2 (e.g., usage tends to be highest in the 2-4 p.m. time slot, the first floor is used noticeably more than the other floors), the data does not reveal further meaningful patterns. more than two weeks of observation are needed to uncover significant patterns at the week by week scale. note that the low values for sunday, april 26, 8-10 p.m., are the result of lack of data rather than absence of students.   behavioral use of spaces   the framework devised to show how students use library spaces originally identified three criteria to be observed as a set of behaviors defined as independent study (is), alongside study (as), and group study (gs). the charts created to record data for the sweep counts also used the codes is, as, and gs to record behaviors observed. discussion by the assessment team after the first count identified that observers may interpret these categories differently, and to label all behavior as study may be inaccurate. thus, the original category of studying alongside (as) was merged into the existing heading of individual study (is) because group work (gs) should indicate active collaboration with interaction at the time of observation. these categories correlate to a similar examination of students using library space by holder and lange (2014) who also found it necessary to clarify proximity: “interaction (students working alone/students working collaboratively/other)” (p. 9).   some observers noted that it was a subjective call whether to label student use is or as when they were working independently but at the same table or space although they were not directly interacting. so alongside (as) became identified as communal and was combined with is for the count. group work implied interaction among participants and may incorporate social activities as well.   space use   table 2 provides an overview of how students were using each floor during each of the study periods. the lower level has more carrels and fewer tables than other floors and provides more individual/communal activity rather than group work/study. accordingly, the results showed significantly more individual work: the lower level had 20% more individual than communal study in december and approximately 10% more in april.   the first floor, which includes a learning commons with booths, cluster soft seating, high top and moveable tables, a café in the 24 hour room with moveable seating, as well as service points (circulation and reference), shows almost equal use of space between individual/communal (is/communal) versus group/social activities (gs/social). data for this floor closely parallels findings for the library as a whole and is fairly consistent between semesters with almost equal behavioral use with 48% individual/communal versus 52% group work in december with 51% individual versus 49% group work in april.   the second floor shows significantly more group/social activity compared with all floors and is consistent over two semesters with approximately 40% individual versus 60% social. one reason for the high usage is the preference shown by many greek society students who use these spaces for communal study.   the third floor, designated as the silent floor, has vastly more individual/communal than group/social use and is consistent between semesters with the highest number of individual use of all floors with approximately 70% individual and only 30% group or social activity.   when all floors are averaged for behavioral use of space, it is almost equally distributed between is/communal and gs/social. in the observation of behavior, the counts indicated that the lower level 60% vs. 40% preference for individual versus group activity and third floor (quiet area) approximately 70% vs. 30% preference for individual over group activity; whereas, the first floor showed nearly equal preference for individual vs. group activity but only the second floor was higher in group work/activity with approximately 40%-60% individual vs. group engagement. the average totals for all floors for both semesters indicate approximately 52% and 48% individual vs. group activity for december but the opposite, 48% 52% individual vs. group activity, for april.   the data collected about behavioral use of library spaces revealed the total average percent for all floors in the library is almost equal for individual/communal work vs. group work or social activity/learning. the results indicate that students at uri gather in the library to work both communally and collaboratively in almost equal amounts throughout the day and evening with peak times in the late afternoon. thus, it appears that more tables and seats are needed to accommodate students’ desire to work communally or collaboratively.   the data is notably consistent. observation at the library demonstrates that close to 50% of the library is used for independent study or communal alongside and approximately 50% of the library space is used for group collaborating or social engagement. some observed activities by groups include collaborative learning projects using white boards with equations, scientific data, charts, diagrams, engineering formulas, preparing presentations, and practicing performances, as well as using roving white boards or shared electronic devices and flat screens in the group study rooms. this sort of collaborative work supports the learning commons concept as advocated by bennett (2003). at the same time, regardless of intention or design, library space is being used communally, individually, for group work with socializing, as well as for interacting with both print and electronic information resources.   group study rooms are very popular spaces. the library has 21 group study rooms of various configurations on 3 of the 4 levels. fifteen of these rooms can accommodate up to six students, and six rooms are intended for one or two students. students frequently indicate preferred spaces when they request a study room, however, they were identified as full even if only one or two students occupied the room.   some group study rooms have a small counter permanently mounted at desk height with seating for one or two students. others have freestanding tables with wall-mounted whiteboards, and some have large monitors in the rooms in the learning commons where students can plug in their laptops for greater screen visibility during group work. rooms on the second and third floor of the library are sometimes less appealing than rooms on the first floor due to their older furnishings, but they remain quite popular and all are frequently full on all floors. group study rooms are available on a first-come-first-served basis only, with no option to reserve rooms. students can check out a key to a room for up to three hours at a time, and can renew the room if no other students or groups are waiting to use the next available room.   while the group study rooms were often in use by groups during both survey periods, on a number of occasions only one student occupied a small group study room. in most cases, however, when large group study rooms were in use, groups of more than two students were using them. the few exceptions to this trend—for example, only one student occupied a room intended for use by three or more students—occurred during the early hours on weekends. this is a time when library use as a whole is lower than average, and there is consequently lower demand for group study spaces.   occupancy rate by floor and hour   although the building rarely has more than 20-35% total seat occupancy during the observation weeks, it was noted that frequently only 1-2 students occupied tables that seat 4-6, further confirmation of applegate’s observations (2009). students seem reluctant to sit next to unfamiliar students which likely accounts for similar low occupancy of the carrels on the lower level and third floor, as noted above. the 2-4 p.m. time period monday-friday accounts for the highest occupancy rates with the 8-10 p.m. time slot generally close behind. the evening count was almost always higher than the morning count in december but the opposite was true in the spring semester. another curiosity is that the first floor use drops off more than other floors between the afternoon and evening especially during the spring semester count. there is no accurate way to determine why usage declines between late afternoon and evening without more intrusive interactions with the students. it is obvious from the data summary charts that the lower level and third floor (designated quiet zone) are underutilized (see table 3).     table 3 occupancy rate (occupied seats vs. available seats) by floor and hour   december 2014 april 2015 lower level     totals 525/2289 (22.9%) 557/2289 (24.3%) 10-noon 153/763 (20.0%) 165/763 (21.6%) 2-4pm 182/763 (23.9%) 266/763 (34.9%) 8-10pm 190/763 (24.9%) 126/763 (16.5%) first floor     totals 2720/14700 (18.5%) 3548/14700 (24%) 10-noon 727/4900 (13.9%) 1255/4900 (25.6%) 2-4pm 1130/4900 (19%) 1490/4900 (30.4%) 8-10pm 893/4900 (16.8%) 783/4900 (16%) second floor     totals 1575/5796 (27.2%) 1283/5796 (22.1%) 10-noon 427/1932 (24.9%) 365/1932 (18.9%) 2-4pm 605/1932 (31.3%) 661/1932 (34.2%) 8-10pm 543/1932 (28.1%) 257/1932 (13%) third floor     totals 1504/7833 (19.2%) 1005/7833 (12.8%) 10-noon 326/2611 (12.5%) 240/2611 (9.2%) 2-4pm 599 /2611 (22.9%) 541/2611 (20.7%) 8-10pm 579/2611 (22.2%) 224/2611 (8.6%)     limitations   discussion of initial data exposed a discrepancy: unobtrusive observation could not definitively state whether people sitting in close proximity to one another were working collaboratively or if those students were working communally by sharing space. consequently, the team adjusted data categories to reflect the reality of what could be observed. this reclassification of terms reflects a standard downside to research that is limited to observation as also observed by may (2011). without direct intervention by either interviewing or surveying students, researchers could not define some behaviors and activities precisely, such as using a computer for study versus social media. likewise, the findings could have been enhanced by surveys similar to those from yoo-lee et al.’s (2013) investigation of how students perceive space. because we did not ask students directly what spaces and modes of study they preferred, we cannot speculate on their preferences with any great certainty. since this study used multiple observers, the assessment team pre-tested the coding sheets and clarified codes to minimize discrepancies and inconsistencies, however subjectivity among coders must be acknowledged.   conclusions and further research questions   this study broadly supports the conclusions of other researchers. for example, montgomery (2014) found that “…the renovation provided users with a better space to work alone in addition to it being used for social learning. we did not anticipate users seeking individual studying space in a social learning environment, but welcomed the flexibility of the space to meet this learning behavior” (p. 73). additionally, holder and lange (2014) suggested that students’ use of space is need specific: as a consequence of either opportunity or necessity students repurpose space to meet their individual, time sensitive needs. their data demonstrated that an area intended for collaborative study on the third floor of mcgill university’s mclennan building was used for quiet, singular study 50% of the time (holder & lange, 2014). the shared use of space observed at uri also supports theories and findings for the need of both types of spaces as posited by freeman (2005), demas (2005), and lin, chen, and chang (2010).   the uri case study reveals that the library is a popular venue for student use with almost equal individual or communal study as compared to group work or social engagement during these two weeks of observation. the library provides both a refuge for quiet study as well as a venue for social activity or collaborative engagement, thereby creating social learning communities where students want and need both types of spaces. differences are minimal between communal/social use as compared to individual/quiet use of spaces on each floor when the total building use is considered. it also speaks to how students use any space available, although the renovated first floor, including the learning commons area, 24 hour room and café, are the most aesthetically appealing spaces and the most used spaces in the library. given these observations, it is reasonable to say, at least provisionally, that the carothers library is serving as the third place on the uri kingston campus. without surveying or interviewing users, however, researchers cannot know why students have chosen to use a particular library space.   determining the need for both kinds of places (quiet individual study versus collaborative engagement) in the wider campus environment would help determine whether the library has become the sole third place on campus or whether there are other spaces serving these needs. further research on campus-wide availability of places for communal and social spaces could inform an understanding of what students desire and prefer and give a better view of the library’s central role in providing those needs. that kind of study might include interviews or survey questions about the appropriate applicability of other spaces to connect and build shared learning communities, such as in dormitories, social houses, classroom buildings, the student union, or other available spaces on campus for study or social and communal use by students.   if those responsible for designing library spaces document how students actually use spaces with an understanding of student-centered learning, then it may be possible to coordinate the intended function and actual use of the library’s communal space for both intellectual conversations and social engagement.   answers to the questions of purpose and student preferences by incorporating a survey or interviewing students could supplement the library observations and sweep counts and thus provide more valuable data for the allocation of both space and money. the activity recorded during this study speaks to student use of spaces and types of behavior observed but not students’ specific preferences.   as academic libraries evolve, library spaces should be continuously assessed, identified, and renovated to further identify how they are meeting the teaching, learning, research, and social learning needs of the university community. this first assessment study of the library as place at uri helped to identify what spaces are being used and how students are using them. since this study, the library has already added significant student seating and additional service points. future iterations of this study should address these physical changes, as well as develop tools to explore student choices and opinions rather than relying solely on observation.   questions for further research on use of library spaces   to determine whether the academic library is becoming the third place on campus, a comprehensive campus snapshot should investigate the availability and quality of spaces for use across campus and incorporate student preferences. questions for future investigations of the impact of the library spaces on the learning community may include:   1.       is the library becoming the sole third place on campus where students go to connect and to study individually, communally, or collaboratively by building informal learning communities outside the classroom? 2.       how do library spaces and services support the institutional mission for student success and what spaces are needed for future learning and engagement?   acknowledgements   the authors also wish to acknowledge the contributions by the following colleagues: lauren mandel for use of her data and mapping library space use, mary c. macdonald and mona niedbala for the use of their data, and celeste dicesare for her work designing maps and coding sheets.   references   applegate, r. (2009). the library is for studying: student preferences for study space. the journal of academic librarianship, 35(4), 341-346. doi:10.1016/j.acalib.2009.04.004   bennett, s. (2003). libraries designed for learning. retrieved from council on library and information resources website: http://www.clir.org/pubs/reports/pub122/pub122web.pdf    bennett, s. (2008). the information or the learning commons: which will we have? the journal of academic librarianship, 34(3), 183-185. doi:10.1016/j.acalib.2008.03.001   demas, s. (2005). from the ashes of alexandria: what’s happening in the college library? in library as place: rethinking roles, rethinking space (pp. 25-40). retrieved from council on library and information resources website: https://www.clir.org/pubs/reports/pub129/pub129.pdf   diller, k. r. (2015, march). reflective practices: library study spaces in support of learning. paper presented at the association of college & research libraries conference, portland, or. retrieved from http://www.ala.org/acrl/sites/ala.org.acrl/files/content/conferences/confsandpreconfs/2015/diller.pdf   foster, n. f., & gibbons, s. l. (2007). studying students: the undergraduate research project at the university of rochester. chicago, il: association of college and research libraries.   fox, r., & doshi, a. (2013). longitudinal assessment of “user-driven” library commons spaces. evidence based library and information practice, 8(2), 85-95. doi:10.18438/b8761c   freeman, g. t. (2005). the library as place: changes in learning patterns, collections, technology, and use. in library as place: rethinking roles, rethinking space (pp. 1-9). retrieved from council on library and information resources website: https://www.clir.org/pubs/reports/pub129/pub129.pdf   gayton, j. t. (2008). academic libraries: “social” or “communal”? the nature and future of academic libraries. the journal of academic librarianship, 34(1), 60-66. doi:10.1016/j.acalib.2007.11.011   given, l. m., & leckie, g. j. (2004). “sweeping” the library: mapping the social activity space of the public library. library & information science research, 25(4), 365-385. doi:10.1016/s0740-8188(03)00049-5   hall, k., & kapa, d. (2015). silent and independent: student use of academic library study space. partnership: the canadian journal of library and information practice and research, 10(1), 1-38. doi:10.21083/partnership.v10i1.3338   holder, s., & lange, j. (2014). looking and listening: a mixed-methods study of space use and user satisfaction. evidence based library and information practice, 9(3), 4-27. doi:10.18438/b8303t   i̇mamoğlu, ç, & gürel, m. ö. (2016). “good fences make good neighbors”: territorial dividers increase user satisfaction and efficiency in library study spaces. the journal of academic librarianship, 42(1), 65-73. doi:10.1016/j.acalib.2015.10.009   jackson, m., & shenton, a. k. (2010). independent learning areas and student learning. journal of librarianship and information science, 42(4), 215-223. doi:10.1177/0961000610380821   khoo, m. j., rozaklis, l., hall, c., & kusunoki, d. (2016). “a really nice spot”: evaluating place, space, and technology in academic libraries. college & research libraries, 77(1), 51-70. doi:10.5860/crl.77.1.51   lin, p., chen, k., & chang, s. (2010). before there was a place called library – library space as an invisible factor affecting students' learning. libri, 60(4), 339. doi:10.1515/libr.2010.029   lippincott, j. k. (2006). linking the information commons to learning. in d. oblinger (ed.), learning spaces (pp. 7-18). washington, d.c.: educause.   lippincott, j. k., van den blink, c. c., lewis, m., stuart, c., & oswald, l. b. (2009). a long-term view for learning spaces. educause review, 44(2), 10-11. retrieved from http://er.educause.edu/   may, f. (2011). methods for studying the use of public spaces in libraries. canadian journal of information and library science, 35(4), 354-366. doi:10.1353/ils.2011.0027   mccarthy, c. a., & nitecki, d. a. (2010, october). an assessment of the bass library as a learning commons environment. presented at the library assessment conference, baltimore, md.   montgomery, s. e. (2014). library space assessment: user learning behaviors in the library. the journal of academic librarianship, 40(1), 70-75. doi:10.1016/j.acalib.2013.11.003   oldenburg, r. (1991). the great good place: cafés, coffee shops, community centers, beauty parlors, general stores, bars, hangouts, and how they get you through the day. new york: paragon house.   uri communications and marketing. (n.d.) uri at a glance 2015-16. retrieved from http://web.uri.edu/about/files/2016/02/uri_at_a-glance2015-16.pdf   yoo‐lee, e., tae, h. l., & velez, l. (2013). planning library spaces and services for millennials: an evidence‐based approach. library management, 34(6), 498-511. doi:10.1108/lm-08-2012-0049     microsoft word es_brown.doc evidence based library and information practice 2007, 2:4  77 evidence based library and information practice       evidence summary    clinical librarian programs may lead to information behaviour change      a review of:  urquhart, christine, janet turner, jane durbin, and jean ryan. “changes in information  behavior in clinical teams after introduction of a clinical librarian service.” journal  of the medical library association 95 (2007): 14‐22.       reviewed by:   marcy l. brown  principal consultant, envision research  delmont, pennsylvania, united states of america  e‐mail: marcy@envisionresearch.info        received: 03 september 2007     accepted: 14 october 2007      © 2007 brown. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – assess whether the north  wales clinical librarian service changed the  information behaviour of team clinicians.  specific objectives were to:    • assess which services were used.  • estimate the effects of training on  clinician search patterns and search times.  • examine the benefits of services  regarding clinical governance.  • examine the effects of training on  clinician confidence.  • explore factors affecting librarian‐ clinician collaboration.    design – observational, longitudinal  evaluation through:    1. librarian reflective diaries  2. baseline and final questionnaire surveys  3. in‐person and telephone interviews  between surveys    setting – three national health service  (nhs) hospital trusts in north wales, uk.  these included north west wales, conwy  & denbighshire, and north east wales.    subjects – physicians, nurses, and allied  clinical staff working with clinical librarians  in one of the above three nhs trusts.      methods – the evaluation period ran from  november 2003 through january 2005. data  collected varied between trusts, since  mailto:marcy@envisionresearch.info http://creativecommons.org/licenses/by/2.0 http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:4  78 program activities varied across locations.  questionnaire data was analyzed with excel;  interview data was analyzed with qsr n6.    north east wales/conwy & denbighshire: in  these two trusts, baseline questionnaires  were distributed in april 2004; interviews  were conducted between july and october  2004, and final questionnaires were  distributed in december 2004. sixty‐nine  baseline questionnaires were returned from  the april distribution, and 57 final  questionnaires were returned in december.  additionally, 33 face‐to‐face and telephone  interviews were conducted between july  and october.    north west wales: immediate post‐training  feedback was collected from 90 participants;  questionnaires sent one month after training  had only a 32% response rate (24 of 75  questionnaires). twelve interviews were  conducted.    results – interviews at all sites  demonstrated a conflict between wanting  the librarian to perform searches for the  clinical teams, and the clinicians needing or  wanting more independence through  greater search skills. the librarian reflective  diaries showed a change in practice over the  evaluation period. administrative duties  lessened and more time was spent on  searching or teaching.     north east wales/conwy & denbighshire:  although a greater percentage of medical  staff reported using nhs and library web  sites by the end of the evaluation period  (70.7% versus 59.4%), a larger percentage  felt overwhelmed by the amount of  information retrieved (68.3% versus 60.9%).   at baseline, more than 50% wished to spend  less than 10 minutes on a search of “general  importance but not of personal interest” (18),  but follow up indicated that the number of  physicians expecting to spend that small  amount of time had dropped to 36%.  among nurses and allied professionals,  changes in information behaviour were  fewer. the interviews confirmed that  information provided by a clinical librarian  affected patient management and/or therapy;  patient diagnosis was not affected. one  interview indicated that cost savings had  resulted from information provided by a  clinical librarian. forty‐five percent of  interviewees felt that their search skills had  improved.    north west wales: ninety‐nine percent of  session attendees rated training programs  highly. although feedback indicated a  desire for more training, 88.9% did feel more  confident about their search abilities  immediately following a program. the one‐ month‐post‐training questionnaire had a  very low response rate (32%), but over 54%  of those responding did believe that their  skills had improved. but one month later, 12  interviews reported that “initial enthusiasm  had usually tapered off” (20), and those  interviewed weren’t sure if they really had  better skills.     conclusions – while findings weren’t  conclusive, they suggest that having a  librarian participate on a clinical team does  lead to changes in information behaviour.  staff members were more confident  searchers, more willing to search for  information, and more willing to delegate  that task to a librarian. the study also  suggests that library services may be  effectively targeted to specific groups of  clinicians, and may have an impact on  patient care.    commentary     the authors point out that clinical librarian  programs may possibly improve patient  outcomes, decrease costs, and save clinician  time (14). previous studies of clinical  librarianship have hinted at these positive  outcomes, but various methodology issues  evidence based library and information practice 2007, 2:4  79 make it difficult to associate such outcomes  with clinical library services. in their  introduction, the authors refer to several  published systematic reviews of clinical  library programs. because of the “lack of  rigorous comparative research methods”  (14), and the “need for higher quality  evaluation designs,” (14) these earlier  systematic reviews were able to  demonstrate only limited impact. this  introduction leads the reader of the current  study to believe that the investigators have  attempted a higher quality study  methodology.    yet the authors of this study chose an  observational design of questionable quality.  rather than develop a true comparative  evaluation with more objective endpoints,  they used surveys and interviews that asked  participants to rate their own information  skills, time spent searching, and information  behaviour change. the subjective nature of  these outcomes adds only marginally to the  literature on the importance and impact of  clinical library programs.    for instance, one of the study objectives was  to “examine whether information skills  training affected staff skills and confidence”  (15). an easy and more objective method of  measuring the effect on staff skills would  have been to offer a pre‐intervention quiz or  skills test, and then proctor the same quiz or  test at some point after the training  intervention. but it does not appear as if  investigators measured participant search  skills. instead, they asked study participants  to rate their own skills.    the authors also did not take advantage of  the opportunity for comparison that was  built in by offering services at three different  locations. while assigning participants to  study groups based on location does not  constitute random allocation, it does present  a means of comparing an intervention group  or groups with a control group. authors  evaluated programs at three locations, but  rather than assign a control group, variable  programs were observed at each of the three  hospital trusts.      while it is difficult to draw generalized  conclusions from this study, the authors do  recognize and acknowledge that difficulty  (21). perhaps the most important  contribution that this article makes to the  body of knowledge about clinical  librarianship is that it “indicates that health  library services can be targeted effectively at  particular groups” (21). the fact that the  services differed by location, and that some  services were more used and appreciated by  certain groups than others, may be an  important finding. this demonstrates the  difficulty in defining “clinical library  services” in such a broad manner; perhaps  future studies could more easily  demonstrate impact on outcomes if the  program or service under investigation was  a smaller, more targeted intervention.  evidence summary   installing noise activated warning signs in library quiet spaces does not appear to reduce actual or perceived noise levels   a review of: lange, j., miller-nesbitt, a., & severson, s. (2016). reducing noise in the academic library: the effectiveness of installing noise meters. library hi tech, 34(1), 45-63. https://doi.org/10.1108/lht-04-2015-0034     reviewed by: michelle dubroy discipline librarian (library researcher services) griffith university library southport, queensland, australia email: m.dubroy@griffith.edu.au   received: 19 aug. 2019                                                                 accepted:  18 oct. 2019      2019 dubroy. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29625     abstract   objective – to explore if installing noise activated warning signs (noisesigns) in library quiet spaces decreases perceived and actual noise levels.   design – noise monitoring and user surveys (print and online).   setting – a large university in canada.   subjects – users of library quiet spaces where noisesigns have, and have not, been installed.   methods – noisesigns provide a visual cue informing those present when noise levels exceed a pre-determined level. in this study, researchers installed two noisesigns in quiet study spaces previously identified as having the “biggest noise issues” (p. 51), and set the devices to illuminate when noise levels exceeded 65 db. user surveys investigated respondents’ perceived and desired noise levels via likert scales before and after noisesigns were installed. actual noise level measurements (via an ipad app) and headcounts were taken manually twice daily for 60 seconds during the same study phases. additionally, the noisesigns recorded noise levels after they were installed. in order to account for variation in library usage over time, control data was also collected in other spaces, where noisesigns had not been installed.   main results – a total of 96 surveys were completed and analyzed across all study locations and time periods. one-way anova tests showed there to be no significant difference in perceived noise levels after installing noisesigns in any of the intervention areas, in neither the shortor long-term. respondents’ comments suggested much of the undesired noise originated from social areas adjacent to the quiet study zones or was of a type which would not set off the noisesigns (e.g., “people chew[ing] too loud[ly]” (p. 54)). one-way anova tests also found there to be no significant difference in actual noise levels in any of the intervention areas after device installation. data logging from the noisesigns themselves showed the “majority” (p. 56) of noise measurements were in the vicinity of 45-50 db and “very rarely” (p. 56) did noise levels exceed the 65 db threshold. despite this, survey respondents appeared to be unhappy with noise, with mean desired noise levels being lower than those perceived.   conclusion – as a result of the study, the library now strives to have greater delineation between quiet and social spaces. they also seek to ensure doors between these areas are kept closed where possible. additionally, the authors suggest libraries install noise activated warning signs in social spaces adjacent to quiet study zones in order to keep these spaces from becoming noisy enough to affect nearby quiet zones. future research could look at the effect of different monitoring options (e.g., security guards, student self-monitoring) and various furniture arrangements on noise levels in the library.   commentary   concern over noise in academic libraries is not new (e.g., luyben et al., 1981). yet, new types of collaborative, technology-enhanced learning spaces can often make libraries seem noisier (mccaffrey & breen, 2016; yelinek & bressler, 2013). varied solutions have been attempted (mccaffrey & breen, 2016; yelinek & bressler, 2013), but this study appears to be the first published investigation into using devices like noisesign to combat the issue. the article was reviewed using a critical appraisal tool (glynn, 2006) and both strengths and weaknesses were found.   the researchers outline the study methodology clearly and with enough detail to allow replication. the survey, appended by the authors, is simple and outlines the ways in which the information obtained may be used. the researchers acquired ethics approval.   readers, however, do not know how representative survey respondents are of the entire user (and non-user) population. respondents were self-selected with the resulting data subject to bias (lavrakas, 2008). demographic information (e.g., age, student type) was not collected. further, readers do not know if the same people responded to the survey multiple times.   the use of control spaces was prudent. nevertheless, the suitability of the chosen control spaces is unclear. notably, for intervention, the researchers selected spaces which were previously subject to high levels of noise complaints. the authors do not disclose if control spaces were similarly affected.   limited resources meant actual noise levels were only measured via the ipad app twice daily, monday friday. it would be beneficial for authors of future studies to use automated noise measurement devices which are able to take frequent measurements any day of the week. moreover, automated devices would remove any effect the staff member's presence may have on the results.   the authors present their results logically and provide insightful commentary around these. however, the researchers could have described the data logged from the noisesigns in greater depth. results of each anova pairwise comparison could have been more clearly conveyed in a table. further, it is unclear how, or if, headcounts were used as a confounding variable in the analysis.   the study provides an important first look at the usefulness of noise activated warning signs as a tool to reduce noise in libraries, and would have a wide audience. further, the authors demonstrate the value of publishing seemingly unsuccessful results through their insightful discussion. greater benefit, however, could be achieved through a more refined methodology. the study also highlights the subjective nature of noise. noise can be an issue for people, even in environments objectively determined to be ‘quiet.’ thus, libraries should consider defining and communicating noise expectations.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154   lavrakas, p. j. (2008). self-selection bias. in p. j. lavrakas (ed.), encyclopedia of survey research methods. thousand oaks, ca: sage publications, inc. https://doi.org/10.4135/9781412963947   luyben, p. d. (1981). reducing noise in a college library. college and research libraries, 42(5), 470-481. https://doi.org/10.5860/crl_42_05_470   mccaffrey, c., & breen, m. (2016). quiet in the library: an evidence-based approach to improving the student experience. portal: libraries and the academy, 16(4), 775-791. https://doi.org/10.1353/pla.2016.0052   yelinek, k., & bressler, d. (2013). the perfect storm: a review of the literature on increased noise levels in academic libraries. college & undergraduate libraries, 20(1), 40-51. https://doi.org/10.1080/10691316.2013.761095     evidence based library and information practice using evidence in practice   24/7 library operations – will they actually come?   susan breakenridge director of technical services, facilities and business administration university libraries rowan university glassboro, new jersey, united states of america email: fink@rowan.edu   received: 14 july 2017    accepted: 16 oct. 2017      2017 breakenridge. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     setting   rowan university, a public comprehensive research institution with approximately 17,300 students, is located in southern new jersey. the university is comprised of seven colleges and five schools on three campuses. rowan is one of two public universities in the country to offer m.d. and d.o. medical degree programs. campbell library is the undergraduate library of rowan university libraries system located on the main campus in glassboro, new jersey. even though glassboro is approximately 20 miles from philadelphia, the campus is considered to be a rural college campus where over 60% of the undergraduate students are classified as commuters.   campbell library, a 5 story – 130,000 square foot building, is located near the centre of the main campus next to the student centre and recreational centre. it is open 99.5 hours during a regular semester week and has seating for over 800. the building also houses the writing center, two college departments (sociology/anthropology and law/justice studies), and five university classrooms that can have classes scheduled between 8:00a.m. to 10:45p.m.   problem   student organizations approach many academic libraries about providing 24/7 library operations especially at finals time. but just because students think they might use it, it does not mean they will. in the fall of 2013, rowan university libraries’ new administration was approached mid-semester about providing 24/7 hours during finals. the administration did not have sufficient information to determine if use would justify the cost. what they did know was (1) the undergraduates were primarily commuter students which can impact usage, (2) the cost of full-time library staff to cover the additional hours, as well as a staff member to conduct the head counts, would be expensive, and (3) the staff were concerned about safety so a public safety officer would have to be hired. the compromise was to initiate a pilot project to extend the hours during finals and conduct head counts to evaluate building use during those extended hours and to determine the continuation or expansion to overnight hours during future finals.   evidence   the library hours during the fall 2013 finals were adjusted for the days the building was already open until midnight and extended to 2:00a.m. the staff member conducting the head counts walked the building, recording the number of students in each of the different library spaces, as well as recording whether or not the students were using technology (computers or similar devices). the head counts as seen in figure 1 were about what the administration expected to see with a decline in the counts starting by 11:00p.m. that fall semester also had some weather consequences for the library hours. the library closed early on the first sunday due to a snow storm and the entire university was closed for another storm on the tuesday of finals.   the staff that conducted the head counts were debriefed after finals. their observations included:   the group study spaces were not adequate in quality or quantity. groups were putting individual tables together to make group space. furniture and chairs in the group study rooms were in poor condition and were mixed and matched. space and furniture across the library needed improvements. many spaces seemed old, dim, and dingy. stained and broken chairs had to be removed on a regular basis. some spaces were underutilized such as the spacious photocopy rooms which creative students would take over as study space. students needed more public computers. during prime times, students would line up waiting to use computers and printers.   upon review of the usage, the administration considered the pilot project a success and agreed to extend building hours the following semester.         figure 1 2013 head counts – fall semester pilot project.     figure 2 before and after photos of 4th floor open study room.     figure 3 2014 head counts – spring semester.       the pilot project observations also helped inform renovation plans that were conducted the following semester and implemented in summer of 2015. the 4th floor open study room had been a collection of mixed and matched tables and chairs that were at least 20-30 years old. see the before and after photos in figure 2.   implementation   the results of the pilot project were sufficient enough for the administration to proceed with expanding hours during finals. since one snap shot of usage is not enough to make permanent changes, the administrators committed to collecting data to determine how to optimize use of the building and the costs to keep the building open. the administration has since approved the head counts to continue for seven semesters.   spring 2014 implementation   the spring semester of 2014, the building hours were expanded to be open four consecutive days (sunday through wednesday) the week before finals and four consecutive days during finals. the thursday through saturday hours stayed the same. the staff conducted the head counts during the extra hours (midnight to 7:30a.m.) but due to staffing, only six of the eight overnight extended hours were counted as seen in figure 3. as expected by the administration, the usage declined after midnight with a significant drop between 2:00a.m. and 3:00a.m. unlike the pilot project counts, the library was not closed during the spring finals due to the weather.   fall 2014 implementation   the fall semester of 2014, the building hours were expanded to be open three consecutive days during finals. the administration was concerned about the potential of weather closures like the previous year and limited the days. like the previous semesters, the head counts were conducted only during the expanded hours and the same decline in usage appeared as seen in figure 4.   spring 2015 implementation   during the spring semester of 2015, the building hours were expanded to be open seven consecutive days (wednesday through thursday) the week before finals. the staff conducted the head counts during the extra hours during the weekdays from midnight to 7:30a.m., and most of the hours on saturday and sunday. as seen in figure 5, the trend line continued to be the same downward line from midnight to 7:00a.m., but the number of students in the building at midnight and 1:00a.m. were higher than the previous semesters.   as seen in figure 6, the hourly counts during the weekend mornings were low, however that would be expected from a commuter campus. days and hours of service are usually controversial. as lawrence and weber (2012) reiterate, certain late hours such as friday and saturdays are not used (p. 543). the graphic also shows the early evening dip on sunday before increasing toward midnight.        figure 4 2014 head counts – fall semester.     figure 5 2015 head counts – spring semester.     figure 6 2015 weekend head counts – spring semester.     fall 2015 implementation   during the fall semester of 2015, the building hours were expanded to be open two consecutive days the week before finals and 3 consecutive days during finals. the administration also agreed that all hours should be counted during the extended hours (day, evening, and overnight). as seen in figure 7, the hourly counts tell more of a story. one could have hypothesized that the midday counts would be busier than the early morning hours, but the double bumps were not obvious via observation and only detected with the head counts. one thought was the students leave to eat dinner and then return. another thought was the second bump was a different group of users than the early afternoon group.       figure 7 2015 mid-semester head counts – fall semester.       spring 2016 – spring 2017 implementation   the building hours were expanded during the semester finals for 2016 and 2017 including overnight hours. the head counts were taken during mid-semester and at finals. the numbers and the trend lines continued to be consistent with the previous semesters.   outcome   the implementation of the head counting project has had positive impact. in less than four years, the number of students using the library during the extended hours has increased. the head counts from the end of the semester and the mid-semester were used by the administration to adjust the building hours during finals and regular semester hours. additionally, the collected data and observations led the administration to update the study and collaboration spaces to better serve the students. the library administration is now positioned to explore different assessment approaches to demonstrate the library’s impact on students.   reflection   conducting head counts is not as easy as it sounds when people are busy moving about the building. the staff conducting the head counts had to agree on their approach. for example, they did not count university staff or library student employee staff. when they were walking the building, they counted only the people in front of them. if they had already passed an area, they would not turn around and try to recount.   comparing the results between days of the week or between semester counts became difficult because the finals did not always start on the same day of the week and different days have different dynamics that would impact the counts. seasonal weather changes and general differences between the fall and spring semesters made it difficult to try to compare counts.   since the data collected were just counts and did not include any interaction with the students, the administration cannot determine what might have caused the increase of use through the years. one major point to acknowledge is over the last 3.5 years, the library has completed 3 room renovations (all included new furniture); added 7 group study rooms (for a total of 18), and added 16 computer work stations (for a total of 36). renovated spaces and new furniture and equipment yield interest and use. even the 4th floor open study room that was renovated three years ago remains one of the most popular places in the building (see figure 8).     figure 8 the 4th floor open study room – spring 2017.     conclusion   assessment is a very important tool for decision making. this basic head count activity has helped improve services for students, provided information for scheduling building hours, and informed the administration on space planning decisions. will students come for 24/7 library operations during finals? yes. the question is whether it is economical for the library to be open just in case a student might come to the library. the next step is to start inquiring with students about their use of the library.   references   lawrence, p., & weber, l. (2012). midnight-2:00 a.m.: what goes on at the library? new library world, 113(11/12), 528-548. http://dx.doi.org/10.1108/0374801211282911     evidence based library and information practice 2009, 4:2 185 evidence based library and information practice news 2009 special libraries association research grants focus on evidence based practice © 2009. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. the special libraries association (sla) has posted its 2009 sla research grant guidelines and application materials to http://www.sla.org/content/slaresearchgrant . sla supports two types of proposals: research projects focused on the needs and concerns of information professionals in special libraries and related venues, with its current focus being on evidence-based practice, as in the special libraries association (sla) research statement, “putting our knowledge to work” (www.sla.org/researchstatement). awards may be granted up to $25,000. projects which directly benefit the operations of sla and its units, by furthering the scientific, literary and educational purpose for which the association is organized and operated. awards may be granted up to $10,000. this focus on evidence-based practice includes but is not limited to the following topics and approaches: impact of information professionals; roles of information professionals within organizations current/user issues; core competencies; measures of productivity and value; client/user satisfaction measures; grant applications are evaluated by sla's research & development committee based on the purpose and objectives of the proposed project within the context of evidence-based practice, the significance of the topic to the profession, the project's methodology, qualifications of staff, and the appropriateness of the project's budget and timetable. research should focus on the demand for increased professional competence and accountability through consciously and consistently making professional-level decisions that are based on the strongest evidence of what works best for our clients. areas in which evidence-based practice may be applied are listed in sla's revised competencies for information professionals of the 21st century (www.sla.org/competencies) and include selection and acquisition of information resources, methods of information access, selection and use of information technologies, and management of library and information services. research techniques that support these competencies include: benchmarking, program evaluation, quality management, performance measurement, identifying best practices, and operations research. the focus on association operation projects should be on projects which directly benefit the http://www.sla.org/content/learn/scholarship/researchgrant/index.cfm http://www.sla.org/researchstatement http://www.sla.org/competencies evidence based library and information practice 2009, 4:2 186 operations of sla and its members. they should focus on projects which can be adapted for wider use within sla’s community, and may include publications, study grants, continuing education programs, public awareness activities, special studies and reports, and information dissemination efforts. application information the grant is announced internationally. a new solicitation for applicants is made each year. application materials are available on sla’s website at http://www.sla.org/slaresearchgrant or by contacting: director, information center special libraries association 331 s. patrick street alexandria, va 22314-3501 usa e-mail: resources@sla.org. timetable march 2009 application materials are made available. august 3, 2009 deadline for submission of the proposal and application form to sla december 2009 notification of award decision. headquartered in alexandria, v.a., the sla is a nonprofit global organization for innovative information professionals and their strategic partners. sla serves more than 11,000 members in 75 countries in the information profession, including corporate, academic, and government information specialists. sla promotes and strengthens its members through learning, advocacy, and networking initiatives. for more information, visit us on the web at www.sla.org. examples of previous research grant recipients 2008 "experimenting outside the information center: non-traditional roles for information professionals in biomedical research." emily glenn and betsy rolland. 2007 '"impulse for growth!" laying foundations for sla membership acquisition and growth in germany for 2008.' michael fanning and "models of embedded librarianship: a research proposal." david shumaker and mary talley. 2003 "a survey to support 'evidence-based practice' in special libraries serving fire service personnel and researchers in public safety and homeland security areas." dr. linda c. smith and ms. lian ruan, university of illinois at urbana-champaign. 2002 “the new information professional: vision and practice.” dr. deborah barreau, school of information & library science, university of north carolina at chapel hill, chapel hill, north carolina usa. 1997 “using information in the virtual office: how special libraries are serving telecommuters,” claire r. mcinerney, school of information science, university at albany/suny. examples of previous endowment fund awards development of position profiles for the information professionals in healthcare industries; applying the sla competencies and marketing the profession. http://www.sla.org/content/learn/scholarship/researchgrant/index.cfm mailto:resources@sla.org https://www.sla.org/admin/www.sla.org evidence based library and information practice 2009, 4:2 187 grant to explore systems thinking perspectives: behaviours that support innovation in knowledge and information delivery. development of a knowledge management and sharing chapter web site structured and guided by sla competencies. support for the development of a country-wide database network for nepal. support for a state-focused marketing and advertising campaign featuring the profession. evidence summary   homeless patrons utilize the library for more than shelter but public library services are not designed with them in mind   a review of: dowdell, l., & liew, c. l. (2019). more than a shelter: public libraries and the information needs of people experiencing homelessness. library & information science research, 41(4), 100984. https://doi.org/10.1016/j.lisr.2019.100984   reviewed by: samantha j. kaplan research & education librarian, liaison to the school of medicine duke university medical center library & archives durham, north carolina, united states of america email: samantha.kaplan@duke.edu   received: 20 july 2020                                                              accepted:  30 oct. 2020      2020 kaplan. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29805     abstract   objective – the study sought to examine the information seeking behavior of homeless patrons and how public libraries meet the needs of homeless patrons.   design – qualitative phenomenological study.   setting – public libraries in new zealand.   subjects – four homeless patrons who were current library patrons and seven public library workers (senior managers and two front line workers).   methods – purposive convenience sample of homeless patrons and library workers to participate in face-to-face, semi-structured interviews. the study utilized creswell's four-step data analysis spiral to produce a synthesis.   main results – homeless patrons utilize public libraries for far more than daytime shelter, patronizing the collections, and accessing services. the participating libraries did not have existing policies, practices, services, or staff designed for the needs and wants of homeless people, however, current offerings largely met the needs of homeless patrons.   conclusion – homeless people use public libraries much like non-homeless patrons and public libraries could develop specialized offerings for them, though they must take care to do so in a way that does not further marginalize this group. additional research is needed to understand why some homeless people do not utilize the libraries.    commentary   this study adds to the limited body of research that captures the intersection of libraries, information behavior, and homeless people.   while qualitative and exploratory in nature, this research was rigorous when appraised with the critical review form for qualitative studies (letts et al. 2007). data were not collected to the point of saturation; however, the study’s authors indicated the difficulty in identifying and recruiting homeless participants. this accounts for the homogeneity of homeless participants (all were of māori descent) but makes it challenging to transfer the findings to homeless individuals of less marginalized backgrounds.   the authors could have provided more information about how their own assumptions, biases, and experiences may have influenced the interviews. the relationship of māori ancestry to homelessness was a recurring theme in the study and one of the authors shares this characteristic with the participants. understanding how this author's background guided the research and was possibly used to build rapport with participants would improve the credibility of this study and assist future researchers working with marginalized communities.   however, the credibility of this research is already very high. the researchers produce the full interview guide within the study, quote individual, anonymized participants, and describe in great detail the efforts made to build rapport with participants. this study provides a detailed framework for conducting research with a sensitive population respectfully without perpetuating their experience of 'othering.'   the authors' findings discredit the notion that homeless patrons use the library primarily as a daytime shelter. the authors identify and explicate specific barriers to access these patrons face and outline opportunities for public libraries and library workers.   references   letts, l., wilkins, s., law, m., stewart, d., bosch, j., & westmorland, m. (2007) critical review form – qualitative studies (version 2.0). retrieved from http://www.peelregion.ca/health/library/eidmtools/qualreview_version2_0.pdf       evidence summary   there can be no single approach for supporting students with autism spectrum disorder in academic libraries, but sensory-friendly spaces and clear policies may help   a review of: anderson, a. (2018). autism and the academic library: a study of online communication. college & research libraries, 79(5), 645-658. https://doi.org/10.5860/crl.79.5.645     reviewed by: michelle dubroy discipline librarian (library researcher services) griffith university library southport, queensland, australia e-mail: m.dubroy@griffith.edu.au   received: 7 feb. 2019                                                                     accepted:  16 may 2019      2019 dubroy. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29552     abstract   objective – to investigate how people with autism spectrum disorder (asd) discuss their experiences in academic libraries in an online community of their peers.   design – qualitative content analysis.   setting – online discussion forum.   subjects – an unknown number of registered members of wrong planet (wrongplanet.net), who self-identify as having asd and have posted about academic libraries on the public discussion board since 2004.   methods – potentially relevant wrong planet public discussion board threads posted between 2004 and an undisclosed collection date were retrieved using an advanced google search with the search strategy “library; librarian; lib; and college; university; uni; campus” (p. 648). each thread (total 170) was read in its entirety to determine its relevance to the study, and a total of 98 discussion threads were ultimately included in the analysis. data were coded inductively and deductively, guided by the research questions and a conceptual framework which views asd as being (at least partially) socially constructed. coding was checked for consistency by another researcher.   main results – wrong planet members expressed a variety of views regarding the academic library’s physical environment, its resources, and the benefits and challenges of interacting socially within it. many members discussed using the library as a place to escape noise, distraction, and social interaction, while other members expressed the opposite, finding the library, its resources, and its patrons to be noisy, distracting, and even chaotic. social interaction in the library was seen both positively and negatively, with members appearing to need clearly defined rules regarding collaboration, noise, and behaviour in the library.   conclusion – while there is no one-size-fits-all approach to supporting students with asd in academic libraries, the findings suggest it may be beneficial to provide sensory-friendly environments, designate defined spaces for quiet study and for collaboration, clearly state rules regarding noise and behaviour, and provide informal opportunities to socialize. the author also suggests libraries raise awareness of the needs of asd students among the entire academic community by hosting events and seminars. the author plans to build on these findings by surveying and interviewing relevant stakeholders.   commentary   this study adds to the small body of literature exploring how to support tertiary students with asd. the author has positioned the research within a paradigm that views asd as “a neurological difference that has been socially constructed as a disorder” (molloy & vasil, 2002, p. 659), while still accepting asd as a legitimate diagnosis. through this lens, the author has focused on how we can make environments more inclusive rather than on how the behaviour of those with asd can be modified.   the article was reviewed using a critical appraisal tool (glynn, 2006), a qualitative content analysis trustworthiness checklist (elo et al., 2014), and advice found in schreier (2014). several strengths and weaknesses were found. the method used was appropriate for answering the research questions. the use of existing data from a third-party website was pragmatic and no doubt cost-effective, but this has led to some limitations. specifically, readers do not know how many individual users contributed to the data, nor do we know their demographics or diagnostic status.   the author reported her findings logically and made judicious use of quotations. the suggestion, however, to educate neurotypical staff and students about asd by hosting events in the library does not appear to come from the data. indeed, the author has not provided any quotation in support of this recommendation, and it appears to come more from a position of advocacy than a position which “seeks to understand experiences as described by the individuals themselves” (p. 648).   further, the researcher did not outline details regarding the amount of data collected and analysed (e.g., number of responses per discussion thread). readers, therefore, cannot know how much data the researcher used in drawing her conclusions.   the author states her coding was double checked by a volunteer, but she has not disclosed the degree of consistency between herself and the checker. further, the author does not describe what specific actions she took to improve her coding in response to this checking. additionally, the researcher has not provided detailed definitions of the coding themes used. particularly, the theme ”library as place” could be further defined and differentiated. for these reasons, it is difficult for readers to judge the validity and overall quality of the coding frame (see schreier, 2014).   a goal of libraries globally is to provide equitable services for everyone in their communities (international federation of library associations and institutions [ifla], 2016), and this study contributes to this goal by bringing attention to a potentially under-serviced user group. additionally, the contradictory nature of the results highlights the importance of recognizing the differences between individuals within a group. the study also demonstrates a practical, useful and unobtrusive way to use extant data to gain insights into our user communities.   references   elo, s., kääriäinen, m., kanste, o., pölkki, t., utriainen, k., & kyngäs, h. (2014). qualitative content analysis: a focus on trustworthiness. sage open. https://doi.org/10.1177/2158244014522633   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154   international federation of library associations and institutions (ifla). (2016). ifla code of ethics for librarians and other information workers. retrieved from https://www.ifla.org/publications/node/11092   molloy, h., & vasil, l. (2002). the social construction of asperger syndrome: the pathologising of difference? disability & society, 17(6), 659-669. https://doi.org/10.1080/0968759022000010434   schreier, m. (2014). qualitative content analysis. in u. flick (ed.), sage handbook of qualitative data analysis (pp. 170-183). london: sage publications ltd. https://doi.org/10.4135/9781446282243.n12     evidence summary   lgbtqia+ students exhibit differing information practices based on social media presence and self-identified personality type   a review of: de la cruz, j., winfrey, a., & solomon, s. (2022). navigating the network: an exploratory study of lgbtqia+ information practices at two single-sex hbcus. college & research libraries, 83(2), 278–295. https://doi.org/10.5860/crl.83.2.278   reviewed by: hilary jasmin research and learning services librarian health sciences library the university of tennessee health science center memphis, tennessee, united states of america email: hjasmin@uthsc.edu   received: 28 sept. 2022                                                             accepted:  2 feb. 2023      2023 jasmin. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30256     abstract   objective –to explore the information practices of lgbtqia+ students and the potential for academic libraries to impact or influence these practices.   design – focus groups and individual interviews.   setting – two single-sex historically black colleges and universities (hbcus) in the usa.   subjects – twenty-three (23) lgbtqia+ students who were recruited through convenience and snowball sampling.   methods – students from the two colleges were hired and trained as recruiters and interviewers. twenty-three (23) total interviews on information practices were conducted—nine (9) via focus group, fourteen (14) via individual interviews. no two students participated in both a focus group and an individual interview. question banks were drafted by the authors for the interviewers to use.   main results – four primary themes arose in qualitative, applied thematic analysis: acceptance, support, personality, and social media. acceptance was further discussed by an interviewee stating one of the colleges in the study was started by lesbians, but this is silent in the campus history. thus, it is difficult to feel accepted on a campus with so much erasure of lgbtqia+ history. in conjunction with acceptance is the lack of support from both campuses, namely in events, activities, and other affirming programming for lgbtqia+ students. students felt the need for more explicit, unequivocal support for lgbtqia+ students from campus administration. findings also suggested that personality, namely participants’ self-identified introversion, may contribute to information deprivation due to fewer social connections and therefore less information sharing. social media, the final theme, was noted as the most powerful forum for information sharing for students, as well as a space to normalize lgbtqia+ movement and visibility. finally, students viewed the library in a traditional light, such as a study space. the reasoning for some lgbtqia+ students rejecting the library as a safe space was unclear, though the authors hypothesize this may be due to safety while returning back to their dorms at night.   conclusion – with a paucity of research in the information practices of lgbtqia+ students, specifically those enrolled at hbcus, the authors concluded that continued research is needed to understand how libraries can create safety and visibility. one primary mode for this might be to make more visible that libraries are not neutral, and that supporting lgbtqia+ students should be a priority.   commentary   the lis literature landscape presents no current studies at the intersection of lgbtqia+ students and single-sex historically black colleges and universities (hbcus). the authors were awarded an american library association diversity research grant, which funded the project, and the varied professions among the authorship (one data services specialist for a health sciences library, one non-profit knowledge manager, and one learning experience designer) allowed for research questions that went outside the traditional library scope and further into information practices from areas underrepresented in lis research.   this study performs well under the cristal checklist for appraising a user study (n.d.), holding strong to its user group by only interviewing students who identified as lgbtqia+. the strength in study methodology was also its limitation, as it used students as interviewers which led to less robust interviews with incorrect question banks. the students enable the researchers to gain insights they may not have gleaned if the authors were the interviewers, so students were by no means a strict limitation. however, trained qualitative researchers may have been more efficient if the goal was a consistent question bank across all interviews. the sample was small, but this was a credible expectation of a study within small campuses and marginalized user groups. the study could be fully or partially replicated, whether at a co-ed hbcu or in other lgbtqia+ groups at predominantly white institutions (pwis).   the authors noted two poignant areas where more research is needed: black lgbtqia+ students who self-identify as introverted and their potential for information deprivation, and identity formation for lgbtqia+ students of color. in the acceptance theme, one student noted their hope that being an hbcu alum “would ‘soften the blow of queerness’ for their parents” (de la cruz et al., 2022, p. 285), indicating competing identities within this one student and the challenges of navigating safety, acceptance, and affirming support in multiple marginalized facets of their identity. in the personality theme, introverts reported fewer social interactions overall, therefore fewer opportunities to give and receive information. information practices of those who, by default, socialize less is another area the authors would like to see further explored.   the study offered thoughtful insight into how the lgbtqia+ population uses and shares information at single-sex hbcus. concerning the research question investigating how libraries are involved, there is room for growth in the literature; discussion during interviews and focus groups around using the library for information-seeking was largely absent in this study. suggestions from the authors such as safe spaces in the library for these types of conversations/interviews, safer transportation from the library back to respective campuses, and more visibility for queer resources and information in the library would go a long way to showing the lgbtqia+ user group that the library is not neutral and is in their corner. this study will be of great value to outreach librarians in nearly any library type, particularly for those whose mission and vision seek to empower and amplify marginalized user groups.   references   cristal checklist for appraising a user study. (n.d.). in nettingtheevidence.pbwiki.com. retrieved july 21, 2022, from http://nettingtheevidence.pbwiki.com/f/use.doc   de la cruz, j., winfrey, a., & solomon, s. (2022). navigating the network: an exploratory study of lgbtqia+ information practices at two single-sex hbcus. college & research libraries, 83(2), 278–295. https://doi.org/10.5860/crl.83.2.278 editorial evidence based library and information practice 2009, 4:2 56 evidence based library and information practice article an emerging theory for evidence based information literacy instruction in school libraries, part 1: building a foundation carol a. gordon associate professor school of communication, information and library studies rutgers, the state university of new jersey new brunswick, new jersey, united states of america e-mail: cgordon@scils.rutgers.edu received: 27 february 2009 accepted: 06 may 2009 © 2009 gordon. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. abstract objective – part i of this paper aims to create a framework for an emerging theory of evidence based information literacy instruction. in order to ground this framework in existing theory, a holistic perspective views inquiry as a learning process that synthesizes information searching and knowledge building. an interdisciplinary approach is taken to relate user-centric information behavior theory and constructivist learning theory that supports this synthesis. the substantive theories that emerge serve as a springboard for emerging theory. a second objective of this paper is to define evidence based information literacy instruction by assessing the suitability of performance based assessment and action research as tools of evidence based practice. methods – an historical review of research grounded in user-centered information behavior theory and constructivist learning theory establishes a body of existing substantive theory that supports emerging theory for evidence based information literacy instruction within an information-to-knowledge approach. a focused review of the literature presents supporting research for an evidence based pedagogy that is performance assessment based, i.e., information users are immersed in real-world tasks that include formative assessments. an analysis of the meaning of action research in terms of its purpose and methodology establishes its suitability for structuring an evidence based pedagogy. supporting mailto:cgordon@scils.rutgers.edu evidence based library and information practice 2009, 4:2 57 research tests a training model for school librarians and educators which integrates performance based assessment, as well as action research. results – findings of an historical analysis of information behavior theory and constructivist teaching practices, and a literature review that explores teaching models for evidence based information literacy instruction, point to two elements of evidence based information literacy instruction: the micro level of information searching behavior and the macro level of the learning task. on the micro level users are confronting information, and searching is seen as the entire process of the interaction of users with a series of information tasks, as described in kuhlthau’s information search process. the micro level is the level of deep understanding as critical thinking skills craft the connection between information and knowledge. on the macro level the learning task, designed by an instructional team, shapes the inquiry. it is the context for information tasks. the learning task is relevant, engaging, and rigorous to sustain the interest and interaction of the user with information and emerging knowledge. the macro level is the level of instruction whereby the learning task fosters self-reflection, self-correction, and self-regulation. the role of evidence, which is generated by performance assessment and action research, is critical to both levels. on each of these levels the learning task informs the information search. conclusion – evidence is the link between information and knowledge in the learning process called inquiry. sources of this evidence are information tasks embedded in the learning task, which are nested in a teaching and learning culture of inquiry. in order to generate evidence continuously throughout the inquiry unit, the task is characterized by research-based tools such as performance-based assessment and action research. the synergy of these elements in part 1 of this article establishes the foundation for building a theory that supports further research of evidence based information literacy instruction. introduction school libraries are complex environments where information literacy is taught in the context of curriculum content. this integrated approach of academics and information literacy sets the expectation that young information users will apply targeted information skills to their learning tasks. the content of inquiry derives from school curricula and state standards that are conducive to this kind of problem based learning. the collaboration between school librarian and classroom teacher provides support to users in the information and technology rich environment of the school library. the intent is that the school library will function as a laboratory for users to construct knowledge as they use information skills as information management tools within the learning tasks that structure the inquiries. in everyday practice, however, school librarians are struggling with the recent shift to information literacy instruction that stresses information use. it is not uncommon to observe young people successfully searching, retrieving, and locating more than enough information but feeling overwhelmed when they are expected to use the information to resolve conflicting viewpoints, or synthesize facts to create new meaning, or integrate the information with what they already know about the topic. evidence based library and information practice 2009, 4:2 58 these are examples of the information to knowledge connection whereby users construct meaning from information. however, they may resort to cutting, pasting, and plagiarizing because they lack the skills to process the information and successfully create learning outcomes that are representative of their new knowledge. this paper posits that performance based assessment and action research are tools of evidence based practice when school librarians gather evidence to inform their practice and then engage in reflection to continuously improve that practice. performance based assessment is a method of teaching that generates evidence of student progress. action research provides a structure for this kind of teaching in the complex environment of the school library in order to: identify problematic areas in the design, implementation, and evaluation of the instruction; develop teaching strategies that generate data about student performance; collect evidence using qualitative methods in naturalistic settings; analyze evidence at the point of need and apply it to the revision of instruction; and apply evidence to determine how to do improve instruction in the future. the purpose of this paper is to set a foundation for building a theory that supports the study of evidence based information literacy instruction. part i describes significant shifts in school library instruction over four decades from the perspectives of theory and practice. it traces the evolution of information skills instruction, as well as information science research, which has been informing best practice since the 1970s. it examines performance based assessment as a pedagogy that develops the concept of information use. it explores the place of action research in this pedagogy. part ii of this paper will develop a theory for evidence based information literacy instruction based on this foundation. an historical review of the practice of information literacy instruction the most significant shift in information literacy instruction has been from teaching information skills in isolation to an integrated approach that facilitates the information to knowledge connection. the american association of school librarians (aasl) introduced the concept of information literacy with the publication of revised information standards (‚guidelines‛). aasl defines information literacy as the recognition of an information need and the ability to search, find, evaluate, and use information (‚partnerships‛). these standards set benchmarks for information literacy but did not develop the concept of information use. they were revised in 1998 to include two important additions: 1) school librarians and teachers work as instructional partners to teach these skills in the context of academic school curricula and state and national standards; and 2) performance based assessments (or authentic assessments) (e.g., rubrics, journals, and portfolios) provide ongoing feedback, or evidence, to information users through selfand peer-evaluation, as well as teacher-student interactions. this teaching approach is a significant shift that places the teacher in the role of facilitator and provides opportunities for school librarians to create meaningful, inquiry-based learning tasks and assessments. in practice, however, school library instruction continues to focus on the basics of information searching and finding. for several years teaching and learning in school libraries has been shifting from evidence based library and information practice 2009, 4:2 59 behavioral, tool-based instruction to a constructivist userand learner-centric approach. during the first forty years that libraries operated in schools in the united states tool-specific instruction focused on reference sources such as the readers’ guide to periodical literature and the world almanac and book of facts. scope and sequence library curriculum documents mimicked core content curriculum, dictating when information skills were taught by grade level, with little relevance to what was happening in classrooms. in the united states two recent developments signify a change in classroom and school library instruction. partnership for 21st century skills (p21) is a consortium of educators, corporate technology companies, stakeholders, and policymakers who developed a framework for 21st century teaching (figure 1). the outer rim of the rainbow in figure 1 includes life and career skills, learning and innovation skills, and information, media and technology skills. these skills center around core subjects (i.e., science, language arts, mathematics, social studies, and history) and 21st century themes (e.g., globalization and the environment). the pool at the foot of the rainbow illustrates the support systems for teaching this curriculum: standards and assessments, curriculum and instruction, professional development, and learning environments. the school library is an ideal venue for this teaching agenda that accommodates critical thinking skills and relevant, authentic, performance based assessments rather than worksheets and exercises evaluated by paper and pencil tests. the p21 curriculum framework influenced fig. 1. framework for 21st century learning source: <http://www.21stcentury skills.org>. used with permission of the author. evidence based library and information practice 2009, 4:2 60 the revision of national information literacy standards set by the american association of school librarians. aasl’s standards for the 21st century learner identified four goals: learners use skills, resources, and tools to: 1. inquire, think critically and gain knowledge. 2. draw conclusions, make informed decisions, apply knowledge to new situations, and create new knowledge. 3. share knowledge and practice ethically and productively as members of our democratic society. 4. pursue personal and aesthetic growth. (‚standards‛) these developments indicate a major shift in how information literacy is defined. it goes beyond information searching and retrieval to incorporate inquiry and critical thinking. the destination is no longer information, but knowledge, with a strong focus on information ethics, productivity, and personal growth. the p21 framework and aasl standards clearly represent a holistic approach to information literacy instruction that considers the needs of young people who are being educated for a world characterized by fast-paced change and global competition in a high tech, collaborative environment, where innovation and problem-solving are essential skills. an historical review of the theory of information behavior research the major shift in information literacy instruction described in the previous section is predicated on a theoretical shift from system to user-centric information science research. information science research has traditionally set the parameters for instructional practice in school libraries. tool-based instruction echoed information retrieval studies. most of this research consisted of online user studies focusing on adult searching behavior based on the behavioral-based best-match principle. initial studies took place in higher education and were, for the most part, experimental and related to the system-oriented school of thought. a survey of the research found that most studies were bound by the system's definition of needs, with the menu of responses coming from the system, and not the user (dervin and nilan). significant attempts to solve problems of the bestmatch principle included association relevance, or searching based solely on document collection characteristics (croft) and relevance feedback (robertson and sparck-jones), which is the use of information gathered from relevance judgments in modifying request formulations. both remained within the best-match principle paradigm since they assumed that the eventual format of the request was equivalent to the ideal document (belkin, oddy and brooks). the shift from a systemto a user-centric focus began with attempts to define relevance, and eventually multi-dimensional definitions captured the complexity of the concept (schamber and eisenberg), incorporating both topicality and user need. a description of relevance as the usefulness of an answer or an indication of its significance to an important purpose (saracevic), reflects a user-oriented approach. models of information seeking evolved toward a problem-oriented school of thought grounded in cognitive science. the search process, it was theorized, progressed through phases, beginning with a vague notion of a lack of information. belkin identified an anomalous state of knowledge (ask) in which there were inadequacies, such as misconceptions. the evidence based library and information practice 2009, 4:2 61 information need was defined as the gap between the user's knowledge about the problem and what the user needed to know in order to solve the problem. people acknowledged an information need, but because of lack of knowledge, found it difficult to specify precisely what information was missing. the expression of an information need, then, was a statement of what the user did not know (belkin, oddy, and brooks) and could not be assumed to resemble the document to be retrieved, which was a representation of a coherent state of knowledge. despite this shift to cognition, "…information science is traditionally based on the classic model that describes communication as the transfer of a message from a source to a destination in information retrieval terms, from a system to a user" (schamber and eisenberg 5). user-centric research laid the foundation for a process approach whereby users learned information skills in the context of inquiry. learning tasks that grew out of content area curricula, such as social studies, language arts, and science, served the dual purpose of teaching information skills and course content. there has been recognition of the need for a generally accepted theoretical base to judge the value of existing programs and to provide direction for developing new programs of library skills instruction (kuhlthau information skills). the concept of student-as-researcher (gordon ‚students‛) emerged to replace the transmission approach to teaching implied in behavioral learning theory (richelle), where the teacher’s instructional role is ‚sage on the stage,‛ rather than ‚guide at the side.‛ the former denotes teacher as lecturer; the latter, teacher as facilitator. the belief that behavior could be shaped by reinforcing, or rewarding, desired responses to environment led educators to devise steps to help learners achieve desired behaviors, or learning outcomes. however, behavioral approaches with step-by-step directions do not accommodate the complex thinking processes required for doing research (richelle). cognitive learning theory states that knowledge acquisition occurs when learners relate new knowledge to prior knowledge, or the relevant concepts or propositions they already possess (ausubel). constructivism supports a more flexible model for student inquiry that allows for reflexivity, active engagement, and collaborative learning. constructivist learning theory posits that learners make their own meaning by building mental models, or constructs. this derives from the work of piaget who described schemata— mental structures by which individuals organize their perceptions into categories to classify specific information. these schemata adapt during the learning process through assimilation, by which the learner integrates new information into existing schemata, or by accommodation, whereby existing schemata are modified to create new mental structures. piaget’s assumption that the individual is a critical thinker from birth laid the foundation for later research in cognition (kulleseid) and the movement toward teaching critical thinking skills. constructivist type of learning is transferable to situations in the real world. information users learn to think through issues that do not have prescribed responses or preset solutions. information users learn to identify what is important to them, to construct new meanings, and to explain their new understanding to others in some way that is authentic to the topic. (kuhlthau ‚learning‛ 711) another initiative that promotes active learning, the problem-solving approach, dates to the work of bloom and broder. identifying four categories of problemevidence based library and information practice 2009, 4:2 62 solving behavior was useful in discriminating between the problem-solving behavior of successful and unsuccessful information users. successful problemsolvers question their knowledge and use that information to break the problem down into more manageable components. the emergence of constructivism in education coincided with the emergence of cognitive psychology that has been applied to research on information searching behavior. the theory of personal constructs stated, ‚a person’s processes are psychologically channelized by the ways in which he anticipates events.‛ (kelly 46) the function of a construct is to enable learners to anticipate events and predict outcomes; behavior is based on the predictions they make. change in behavior is a response to a change in personal constructs. if the prediction proves accurate, the construct is validated; if the predication proves faulty, the construct is reconstructed. kuhlthau’s (‚facilitating‛) application of kelly’s theory to information searching is based on the premise that the search process is a process of assimilation and construction involving feelings as well as thoughts. throughout the information search users refine and re-state the research topic. for kuhlthau, users’ constructs of their information topics change as they become informed by the information they find (‚longitudinal‛). this informs kuhlthau’s information search process (isp) (‚facilitating”) in figure 2. information seekers move from feelings of uncertainty to satisfaction or dissatisfaction according to the way they have handled the search prior to and subsequent to focus formulation, which clarifies the user’s thoughts. actions are considered as searchers advance from seeking relevant information to seeking pertinent information. when applied to the high school seniors the stages indicated information seeking is a complex learning process involving finding meaning (kuhlthau ‚information search process‛). initiation selection exploration formulation collection presentation assessment feelings uncertainty optimism confusion frustration doubt clarity sense of direction; confidence satisfaction or disappointment sense of accomplishment thoughts vague focused increased interest increased selfawareness actions seeking relevant information exploring seeking pertinent information documenting fig. 2. kuhlthau’s model of the information search process used with permission of the author. evidence based library and information practice 2009, 4:2 63 another study verified the model in a wider sample of users in academic, public, and school libraries (kuhlthau, turock, george and belvin). two longitudinal studies showed that the model held up over time by tracking the perceptions of the search process after users completed four years of college (kuhlthau ‚perceptions‛ and ‚longitudinal‛). since the isp model is generalizable and predictive, it serves as a diagnostic tool for school librarians to apply interventions to help information users at specific isp stages. the theoretical framework for this concept is vygotskii's zone of proximal development (zpd) that represents the learner’s inability to move forward without assistance (mind). the isp helps educators identify zones of intervention based on the symptoms exhibited in stages of the isp. for example, some users may not have the prior knowledge needed for initiating the learning task when they are expected to engage in background reading to build their knowledge. interventions that provide images, descriptions, and explanations may be needed to help them fill gaps in their knowledge. intervention is conceived as guided inquiry when a collaborative team of school librarian and classroom teacher(s) provides help that is critical for young people to move along the continuum from novices to experts, or independent learners (kulthau, maniotes and caspari). the isp and guided inquiry call for an evidence based approach to teaching where instructors gather evidence in the guise from the work of information users who may be trapped in the zpd. evidence also helps instructors to assess the effectiveness of their interventions, or help, so they can make informed decisions about revising their instruction when needed. this historical analysis of the practice and underlying theories for information literacy instruction point to two distinct elements of evidence based information literacy instruction: the micro level of information behaviors and the macro level of the learning task. on the micro level users are confronting information and searching is seen as the entire process of the interaction of users with a series of information tasks, as described in kuhlthau’s information search process. information behavior is not confined to searching, locating, and evaluating information. it incorporates the concept of information use whereby information is the raw material for building knowledge. the micro level is the level of critical thinking that engages users in deep understanding as they craft connections between information and knowledge. on the macro level the learning task, designed by an instructional team, shapes the inquiry. it is the context for information tasks. as such, it informs information searching behavior. the learning task is relevant, engaging, and rigorous in order to sustain the interest and interaction of the user with information and emerging knowledge. the macro level is the level of instruction whereby the learning environment, or task, fosters self-reflection, self-correction, and self-regulation. the role of evidence, and action research as a tool of evidence based practice, is critical to both the micro and macro levels of information literacy instruction. a theory that supports a pedagogy for evidence based information literacy instruction rests on the substantive theory presented in this paper, i.e., a user-centric view of information seeking, a constructivist view of learning, and a process approach to library and information services that grows from the isp and guided inquiry. the learning task is the engine that drives this manner of instruction that is predicated on evidence based practice. the substantive research in constructivist learning and information-seeking theory, as well as evidence based library and information practice 2009, 4:2 64 developments that define best practice, point to the need for the practitioner to create learning tasks that relate to the real world and offer opportunities for critical thinking, problem solving, and revision of work. our understanding of the affective, as well as cognitive components of information seeking and learning indicate that learning tasks engage information users and invite them to construct their own meaning. concepts of relevance, engagement, and motivation, as they emerge from constructivist-based research, suggest that learning tasks must offer diverse opportunities for learning and interpreting information and data. it is clear that a monolithic research assignment, restricted to reading and note-taking as the only methods of discovery and presentation, is not adequate to accommodate the highly personalized model of information searching and use suggested by the convergence of substantive theories identified in this paper. literature review: research based models of evidence based information instruction authentic learning tasks (wiggins ‚creating‛) are ideal for an evidence based practice approach whereby information users engage in ‚…the kind of work real people do…‛ (wiggins educative 21). the learning task is set in a real-world problemsolving situation in which the information user plays a role. this pedagogy is performance-based: information users demonstrate their knowledge and skill through authentic assessments that are anchored in the learning task. in the world of professional work and civic or social life, selfassessment and self-adjustment are vital skills based on sizing up unfolding situations and bringing work to successful fruition by comparing where we are and where we need to be. an educative assessment system would therefore deliberately build in and make central challenges that require the student to attend to feedback and make adjustments accordingly in authentically complex situations. (wiggins educative 35) such situations might include: design work and new product development; legislation and public policy; a business or personal financial plan; raising children; winning games as a player or coach (wiggins, educative). these contexts are well-suited to role-playing as information users conduct their searches in the context of a problem or challenge, as demonstrated by the following scenario: you are a rescue worker who is sent to a natural disaster site in a developing country to identify the most pressing problems caused by a tsunami and to make recommendations to your government for the types of aid that will help victims. in this case a summative assessment might be a written report to the government outlining a rescue plan, which is assessed by traditional grades. on the other hand, formative assessment focuses on evidence generated at various times throughout the unit, rather than on the final product. evidence consists of the student’s progress and is viewed as feedback. in the case of the rescue worker scenario, it might be a series of images collected by the information users, along with their written commentary chronicling life-threatening challenges of natural disasters at the collection isp stage. the rationale for this kind of pedagogy is ongoing and continuous assessment that gives users the chance to revise. it also promotes academic rigor. evidence based library and information practice 2009, 4:2 65 do we want to evaluate student problem-solving in the visual arts? experimental research in science? speaking, listening and facilitating a discussion? thoroughly reviewing a piece of imaginative writing until it works for the reader? then let our assessment be built out of such exemplary intellectual challenges. (wiggins ‚case‛) authentic learning and assessment offer a diagnostic approach that generates indicators of where help, or intervention, is needed, so that remediation, or alternative teaching strategies, can be employed. in this case, an example of feedback might be maps generated by information users, or outlines of what they know at mid-point in the inquiry process. their work might reveal that some users have not read enough about the health hazards resulting from natural disasters. other examples of formative assessments embedded in instruction are clearly defined criteria for good work embodied in rubrics, users’ thoughts, feelings, and actions recorded in journals, blogs, checklists, peer reviews, selfevaluation charts, and collaborative processes posted on wikis. the learning tasks become the assessments, as teachers and school librarians analyze the evidence generated as part of a process, rather than restricting assessment to final products. models of evidence based information literacy instruction in school libraries that use performance based assessment evolved from the author’s work reported in three studies. the first focused on determining how ninth grade information users perceive inquiry when their teacher requires them to go to the school library to learn something about how mathematics is used in the real world (gordon ‚fish‛). this study took place in a large international school in europe. the curriculum emphasized academics and college preparation. data were collected through participant observation, focus groups, and journaling. three student collaborators were interviewed at different stages of the project; two participated in focus groups three times, and one student participated twice. information users revealed that their perception of doing research was writing a grammatically correct report that was wellpresented and provided other peoples’ answers to someone else’s question. the research process was not internalized in the school library; it was perceived as an extension of classroom practice. users talked about the research assignment as if it were a test; creativity and inquiry were not perceived as part of the process, and grades were perceived as the most important measure of success. while their teacher had a clearer conception of research as a way of learning, she thought it took too much time from the curriculum, and her students agreed. they valued what was taught in the classroom more than what they discovered in the library. they regarded their information searching in the library as inefficient, indicating that they could have been learning more in the classroom. their attitude toward discovery through information searching reflected a top-down approach to learning that valued what was transmitted through the teacher rather than what was discovered by the student. this study set the agenda for finding and testing performance based approaches to inquiry. the second study looked at performancebased learning and assessment in the context of every day information searching behavior grounded by personal interests. the learning task structured by performance based assessment was based on the information user’s personal interests and everyday information behaviors, resulting in a project that could be anything except a written paper. the study took place in a high school of four hundred students in an independent, international school located in evidence based library and information practice 2009, 4:2 66 europe. it focused on the effects of a performance based independent project on the meta-cognitive behavior and motivation of one hundred ninth grade students participating in the inquiry unit. the leaning goal of the project was to support information users’ management of their own projects. the theoretical framework for the study was grounded in research in metacognition and self-regulated learning (vygotskii). the design of the project was based on research findings that the most effective learners are self-regulatory and that key to effective self-regulation is accurate self-assessment. the learning task required users to create a project of their choice which could be any format other than written papers. the learning task was designed as an authentic learning task with performance based assessment practices, i.e., rubrics, journals, self-evaluation, and peer review. users received continuous feedback about their self-management skills. parents and other family members, advisors, and friends played important, but not dominant roles in providing help. the absence of ‚academic‛ content shifted the focus to awareness of personal interests and management skills such as meeting deadlines, organizing the project, and working with an adviser. these learning objectives were documented in student journals and self-assessed by student and adviser using a rubric. information searching during this inquiry unit was justenough-just-in-time. users relied on resources from home and community, rather than from the school library. since each information user was required to have an adviser, they relied on these adults to provide information or refer them to resources, usually community members. there was a summative assessment administered through an exhibition of student work, which was rated qualitatively by parents, teachers, and community members (gordon ‚putting‛). rather than receiving traditional grades, each grade nine student received a narrative attached to their records and a certificate of completion. the role of the school librarian centered on self-assessment and the affective aspects of information searching and learning. the absence of a set curriculum and the traditional research paper shifted the focus to information users’ awareness of their own interests and their roles as project managers. student engagement was high, as evidenced by a strong trends toward ethnic-related projects that involved families and favored the arts and non-existent curriculum subjects (e.g., design technology, crafts), or ‚minor subjects‛ (e.g., art, music, dance, theatre). non-native english speakers preferred to work in groups were homogeneous with regard to gender and culture. users who chose projects with concrete outcomes were more focused on product; those who created projects that were activist in nature described outcomes in terms of process and learning goals, and demonstrated meta-cognition more frequently. there is strong evidence from this study that users are responsive to performance assessment based learning, sustaining self-discipline, organizational skills, and responsibility to themselves and their members for the duration of the project. the third study took a different view of an intellectual challenge. rather than asking information users to create products that expressed what they learned about a personal interest, the learning task challenged them to use traditional research methods appropriate to the academic discipline in which their investigation was grounded, and to write a 2000-word research paper using primary research methods appropriate to the academic discipline. this approach, called ‘authentic research (gordon ‚students‛) goes beyond note taking to include data collection and analysis. english teachers and the school evidence based library and information practice 2009, 4:2 67 librarian collaborated to gather data in the manner of action research to assess the effectiveness of the assignment. the assignment was based on a structuralist approach whereby information users learn the structure of the academic discipline and its ways of knowing, or particular research methods, in order to engage in rigorous inquiry (bruner). this pedagogy was combined with performance based assessments, including rubrics, journals, and peer editing. information users posed research questions, theses, or hypotheses, as appropriate to their topics, developed proposals, and learned techniques of display and analysis using tables, figures, and citation. their information seeking was intensive throughout the project as they prepared to conduct interviews, create surveys, or engage in participant observation and journaling. the use of qualitative data collection motivated them to seek information from traditional and nontraditional resources. users who chose quantitative-friendly topics analyzed data using descriptive statistics. a content analysis of the papers indicated that users engaged in critical thinking: application, analysis, synthesis, and evaluation. their information searching relied heavily on library resources. they interacted with the librarian on a regular basis and received targeted instruction in the use of electronic resources. the majority of users chose topics that related to an event or situation in their personal lives, such as a learning-disabled sibling, an anorexic friend, or divorced parents. consequently, most users adopted qualitative research methods suited to social science research. concurrently, their teachers engaged in action research, gathering data from observation, journals, and a survey administered at the end of the unit to determine the strengths and weaknesses of the assignment. information users responded positively when asked, ‚how was this research assignment different from what you have done in the past?‛ ‚longer, more depth, more detailed, more demanding.‛ student-generated comments mentioned precise instructions, format, and regulation as unpleasant aspects of the assignment, but the same number of comments revealed that they felt more independent. ‚in the past i was given full instructions on the essay. now i had to do it by myself.‛ when asked about the best aspect of the project, one student wrote, ‚that we stood on our own two feet!‛ comments also reflected an appreciation for the distinction between reporting and research: ‚i never did proper research before. it was the first real serious research i have done. it was much longer and more difficult than previous papers. it was also much more interesting and more fun as well‛ (gordon ‚students‛). these studies indicate that authentic, performance-based instruction generates evidence that informs student and teacher performance in the context of a freely chosen personal task (gordon ‚putting‛) and in the context of a curriculum based academic task (gordon ‚students‛). in both cases information searching was an integral part of the learning task. however, differences between the learning tasks were paralleled by differences in information searching behavior in terms of resources used and the kind of help users sought. the findings of these studies support the design of learning tasks that generate evidence for the improvement of student and teacher performance. the methodology of authentic research promotes critical thinking skills and knowledge construction in the context of a research, rather than report mode (gordon ‚fish‛). the effective use of this approach, however presumes that school librarians know how to do research and how to teach information users to do it. justenough-just in time training is needed since the study of research methods are not part evidence based library and information practice 2009, 4:2 68 of undergraduate, or even graduate schooling for school librarians and educators. evidence based practice offers a solution that seamlessly introduces reflection into instructional practice that is informed by evidence. the evidence based practice movement in school library instruction a fundamental shift in school library instruction occurred when evidence based practice (ebp) became a best practice for school librarianship. in order for school libraries to play a key role in the information age school, i believe there needs to be a fundamental shift from thinking about the movement and management of information resources through structures and networks, and from information skills and information literacy, to a key focus on knowledge construction and human understanding, implemented through a constructivist, inquirybased framework. (todd ‚transitions‛) todd challenged practitioners to make datadriven decisions about their practice that were informed by relevant research: evidence based school librarianship uses research-derived evidence to shape and direct what we do. ebp combines professional wisdom, reflective experience, and understanding of students’ needs with the judicious use of researchderived evidence to make decisions about how the school library can best meet the instructional goals of the school. (todd ‚evidence based‛) to accomplish this, todd advocated that school librarians collect evidence to document how their practices affect student achievement, i.e., the development of deep knowledge and understanding in the competencies and skills for thinking, living, and working (‚evidence based‛). he identified three dimensions of evidence based practice: evidence for practice; evidence in practice; and evidence of practice (figure 3). fig. 3. three dimensions of evidence based practice in school librarianship (todd) evidence based library and information practice 2009, 4:2 69 1) evidence for practice examines and uses empirical research to inform practice. this dimension consists of knowledge of theory that drives practice, as well as studies that test or apply theory. it includes a clear vision of best practice that bridges theory and practice. this kind of evidence is comparable to what formal research calls the literature review. 2) evidence in practice integrates the research evidence with professional experience, as well as local evidence that identifies learning needs and achievement gaps. in this dimension, evidence in practice is the conceptual framework for action research investigation; it activates professional expertise to direct how the evidence is collected, how it is analyzed, and how it is applied to the identified problem. 3) evidence of practice is derived from systematically measured, student-based data that focuses on learning outcomes. this dimension of evidence based practice is the evidence, or data, extracted from student work or from evidence collected using qualitative or quantitative data collection methods. these three functions of evidence based practice provide the underlying beliefs and values of inquiry based learning in school libraries, thereby serving as a paradigm of best practice. the paradigm is used in two ways. it serves a social and cultural purpose in molding a culture of inquiry for information literacy instruction. evidence based practice embraces reflective practice that seeks continuous improvement through the collection and application of evidence for informed decision-making. it challenges conventions of library and classroom environments characterized by top-down lines of authority, rule-bound communication, and homogeneous, depersonalized interaction dictating a passive role for the young user-learner. implicit in this concept is the need for a systematic approach to the collection of evidence for improvement of teaching and learning, thereby enhancing the role of school librarians in fulfilling the mission of schooling. evidence based practice in school librarianship is the overriding concept for continuous improvement of practice. a need exists, however, to systematically and unobtrusively integrate evidence collection with instruction. action research as a tool of evidence based practice the confluence of constructivist learning theory, authentic learning practice, the isp, guided inquiry, and performance based assessments set the stage for using action research as a tool of evidence based practice. while there is some disagreement about action research as a methodology, its function is to collect evidence from practice to improve practice. action research is problem focused, context specific, and future oriented, and it aims at improvement and involvement (hart and bond). boomer defined it as a ‚deliberate, group, or personally owned and conducted, solutions oriented investigation‛ (8), which indicates its function as reflective practice. a more formal interpretation defines action research as ‚insider research done by practitioners using their own site as the focus of their study…it is oriented to some action or cycle of actions that practitioners wish to take to address a particular situation‛ (anderson, herr, and nihlen 2). in this instance, action evidence based library and information practice 2009, 4:2 70 research takes on a methodology resembling qualitative research methods. patterson and shannon view it as a recursive process involving reflection, inquiry, and action (figure 4). the origins of action research are attributed to lewin (‚action‛), the social psychologist whose research approach, set in social and organizational settings, is focused on social action. his work shares concepts with john dewey’s experiential learning and ‚the inductive scientific method of problem solving as a logic for the solution of problems is such fields as aesthetics, philosophy, psychology and education" (mckernan 8). lewin defined the spiraling nature of action research: identifying a general or initial idea; fact finding; planning; taking the first action step; evaluating; amending the plan; and taking the second action step (field 240). the application of action research in educational settings was established as a strong tradition by the work of corey, who defined action research as the process through which educators study their own practice to solve personal practical problems. research on educational change indicates that change is more likely to occur when participants feel ownership of a problem and feel connected to the solution (anderson, herr, and nihlen). action research can engage educators in examining the effectiveness of their methods when they have identified an area of concern and use the research process to gather evidence for their theses. reflection helps practitioners understand what they are currently doing, why they are doing it, whether it is what they want to do, and what they should do in the future (patterson). reflection can identify weaknesses and strengths, and it validates decision-making inherent in the teaching process through the use of fig. 4. action research is reflection, inquiry, and action (patterson and shannon) evidence based library and information practice 2009, 4:2 71 evidence. reporting results of action research can provide the vicarious experiences related by narrative accounts from schools and classrooms which educators find more helpful than formal educational research (anderson, herr, and nihlen). action research can be described as the illegitimate child of formal research: although it bears a strong family resemblance to its qualitative and quantitative cousins, its validity and reliability is regarded with skepticism by its positivist parents. it sits outside of the positivist tradition that sets the gold standard for empirical research, i.e., the randomized, controlled experiment. ‚unlike scientific inquiry, which is based on the premise that the environment can be controlled and that variables can be strictly introduced and accounted for, social science research is more disorderly.‛ (rose 23) the precarious status of action research raises an important question: is empirical research restricted to the positivist tradition, or can other kinds of systematic investigation be scientific, or at least credible? qualitative research struggled with this question for decades and has established its credibility through theory generation and methodologies that accommodate the contextual nature of research questions such as, ‚how do we learn?‛ situating action research in the tradition of qualitative research distinguishes it from the limitations of quantitative research in investigations of complex phenomena that defy representation through numbers. however, as a form of qualitative research, action research shares its perceived weaknesses: lack of internal validity lack of reliability contingency of the research findings; action research is seen as inappropriate to generalizable findings. low control of the environment; action research is deemed inappropriate to test and produce strong theories, or to build up research models based on solid evidence. personal over-involvement; action research introduces personal biases in its conclusions. (orlikowski and baroudi) action research establishes internal validity through replication and theory verification, and reliability through triangulation of data collection that uses multiple methods. it meets challenges of external validity, or generalizability, with the concept of transferability of results to similar populations. action research is meant to be local, contextual, and specific, and it is not generalizable from sample to population. the population is the subject of inquiry; the concept of a sample is irrelevant, because effect size is not an issue. action research is participatory ethnographic research; it is culture-bound and contextually-specific to the population studied. it collects verbal data through interviews, surveys, observation, journaling, as it uses numerical data such as descriptive statistics. in its early days lewin used quantitative measures and mathematical formulas to treat data collected in problem-specific fields of society for the purpose of social reform. action research in education, however, is largely qualitative. action research is not meant to test theory, but to examine how theory translates into practice to improve practice. it can produce models of best practice, as well as theory-based models that inform practice (gordon ‛study‛). action research is personal in that the practitioner researcher is at once the object of study and the object of improvement of practice, but this is not the same as bias, which can be controlled through researchers’ sustained awareness of themselves as objective research instruments. evidence based library and information practice 2009, 4:2 72 action research enables the study of human behavior in naturalistic settings where the phenomenon is complex. it is advantageous to the study of evidence based information literacy instruction in school libraries, which cannot be replicated in laboratories. however, models of this approach can be developed in the fieldwork of teaching and learning is school libraries. a model for evidence based information literacy instruction derives from the concurrent use of performance-based assessment and action research (gordon ‚study‛), both of which utilize the three dimensions of evidence based practice. a critical element for implementation of the model is the mentoring of educators by a trained researcher who can help them to recognize student work as evidence, create learning tasks that generate that evidence, and integrate qualitative research methods with their teaching. underpinning this model are the isp and guided inquiry, which provide diagnosis of problems and design of interventions to help information users through information tasks, as well as the learning task. in the first dimension of this model authentic learning and performance based assessment engage users in a role that situates them in a real world problem, as described in this paper. concurrently, the instructional team of school librarian and classroom teacher(s) work in the second dimension of the model: they identify a problem in their practice and collect evidence to help them make informed educational decisions about the problem. educators may want to know how a variety of note taking methods worked for users. did users prefer note cards, graphic organizers, mapping, or color-coding printouts? was there a connection between student performance and the note taking methods? there are two ways that this evidence can be generated. the first is through formative assessments flowing from authentic learning tasks, as previously described. the second way to collect evidence is through the application of traditional qualitative or quantitative data collection methods. for example, school librarians might administer a survey at the end of the unit to find out why users did not use subscription databases in their research, or to learn how users evaluate the unit in terms of what worked and what didn’t work. the first and second dimensions of this pedagogical model are concurrent as users engage in performance based learning tasks and educators engage in action research. results of the study (gordon ‚study‛) indicate that when students and educators analyze the evidence their work generates, they achieve a heightened awareness of how to revise: information users have opportunities to learn from mistakes; school librarians and teachers have opportunities to differentiate their teaching to meet individual needs of information users. everyone performs research to continuously improve performance. everyone becomes his or her own best critic through self-reflection based on evidence. action research not only functions as a tool for professional development, it fosters a culture of inquiry that features the school librarian as a leader of data-driven instruction. it brings structure to collaboration between library and classroom. it creates a community of practice that improves transactions between information users and teachers, between school librarians and teachers, and among information users. implications for further study although there is a strong tradition of constructivism that supports authentic learning, the isp, and guided inquiry, theory that supports evidence based information literacy instruction, and the place of evidence based practice as a paradigm for action research, has not been established. a theory is needed to support further research of information literacy evidence based library and information practice 2009, 4:2 73 instruction in school libraries for two major reasons. the first is the need for theory to develop and test models for evidence based information literacy instruction that works in diverse information environments. a theoretical framework that grows from such theory is indicated to support empirical studies that address the nature of this kind of action research application and the way it is conducted. the findings of the historical analysis of information theory and instructional practice, as well as the analysis of relevant research that tests an emerging information to knowledge pedagogy, point to a distinction between the macro level of information behavior, which consists of the learning task embedded in structured inquiry, and the micro level that consists of specific information searching behaviors. on the macro level, inquiry informs information searching behavior and has important implications for designing the learning task. on the micro level, information searching is seen as the entire process of the interaction of users with a series of information tasks, as described by kuhlthau’s information search process. information behavior is no longer confined to searching, locating, and evaluating information. it incorporates the concept of information use whereby information is the raw material for building knowledge. critical thinking skills are the instruments that craft the connection between information and knowledge. the second need for evidence based information literacy instruction is the development and testing of training models that prepare school librarians and educators to design learning tasks that are evidence based, i.e., that generate the evidence needed to help information users remediate and progress through the learning task. inquiry informs information searching behavior through the learning task, which emerges as the critical element that is central to theory building for information literacy instruction. research is needed to develop the concept of performance based assessment and action research as tools of professional development. how would such a theoretical framework be built, and what would it look like when applied to the learning environment of the school library? part ii of this paper will examine an emerging theory built on the foundation of substantive theory described in this paper, using evidence based practice as the paradigm that sets the beliefs and values for this construct. substantive theories discussed in this paper emerge from research from three disciplines: information science, cognitive science, and education to inform evidence based information literacy instruction. do they have a place in the theoretical framework for evidence based information literacy instruction? a metatheoretical approach determines the nature of action research from the perspective of dewey’s work in defining action and reflections as two facets of thinking. lewin’s work in defining action research for purposes of social reform contributes substantive theory from the field of social psychology. the synthesis of lewin’s substantive theory and dewey’s metatheory, along with substantive theory of constructivist learning, generates an emerging theory for studying evidence based information literacy instruction. the emerging theory indicates that a model for a culture of inquiry in the learning environment of the school library is interdisciplinary, bringing together usercentered information searching and learnercentered educational theory. it points to a research agenda and informs the methods appropriate for the study of evidence based information literacy instruction in school libraries. evidence based library and information practice 2009, 4:2 74 works cited american association of school librarians and association for educational communications and technology. information power: guidelines for school library media programs. chicago: american library association, 1988. american association of school librarians and association for educational communications and technology. information power: partnerships for learning. chicago: american library association, 1998. american association of school librarians. standards for the 21st century learner. chicago: american library association, 2007. anderson, gary k., kathryn g. herr and ann sigrid nihlen. studying your own school: an educator’s guide to qualitative practitioner research. thousand oaks, ca: corwin press, 1994. ausubel, david p. the psychology of meaningful verbal learning. new york: grune and stratton, 1963. belkin, nicholas j. ‚anomalous states of knowledge as a basis for information retrieval.‛ canadian journal of information science 5 (may 1981): 133-43. belkin, nicholas. j., robert n. oddy and helen m. brooks. ‚ask for information retrieval. part ibackground and theory.‛ the journal of documentation 38.2 (june 1982): 61-71. bloom, benjamin samuel and lois j. broder. problem-solving processes of college students: an exploratory investigation. chicago: university of chicago press, 1950. boomer, garth. ‚addressing the problem of elsewhereness: a case for action research in schools.‛ reclaiming the classroom: teacher research as an agency of change. eds. dixie goswami and peter r. stillman. portsmouth, nh: boynton/cook publishers, 1987, 4-13. bruner, jerome. the process of education. cambridge: harvard university press, 1960. corey, steven. action research to improve school practice. new york: teachers college, columbia university, 1953. croft, w. bruce. ‚organizing and searching large files of document descriptors.‛ diss., university of cambridge, 1979. dervin, brenda and michael s. nilan. ‚information needs and uses.‛ annual review of information science and technology (arist) 21 (1986): 3-34. dewey, john. how we think. boston: d.c. heath & co., 1910. gordon, carol a. ‚is fish a vegetable? a qualitative study of a ninth-grade research project.‛ school library media quarterly 25.1 (fall 1996): 2733. gordon carol a. ‚students as authentic researchers: a new prescription for the high school research assignment.‛ school library media evidence based library and information practice 2009, 4:2 75 research 2 .1 (1999): 1-21. 11 may 2009. <http://www.ala.org/>. gordon, carol a. ‚putting the learner in charge. are information literacy skills enough?‛ scan 19.1 (2000): 32-8. gordon, carol a. ‚a study of a three dimensional action research training program for school library media programs.‛ school library media research 9.1 (2006): 1-21. 11 may 2009 <http://www.ala.org/> hart, elizabeth and meg bond. action research for health and social care. buckingham: open university press, 1995. kelly, george. a. a theory of personality: the psychology of personal constructs. new york: norton, 1963. kuhlthau, carol c. facilitating information seeking through cognitive modeling of the search process: a library studies research project. ann arbor, mi.: u.s. department of education, 1986. eric document reproduction service ed328268. kuhlthau, carol c. information skills for an information society: a review of research. syracuse, ny: syracuse university, 1987. eric document reproduction service ed297740. kuhlthau, carol c. "longitudinal case studies of the information search process of users in libraries." library and information science research 10.3 (1988): 257-304. kuhlthau, carol c. ‚information search process: a summary of research and implications for school library media programs.‛ school library media quarterly 18.1 (fall 1989): 1925. kuhlthau, carol c. seeking meaning: a process approach to library and information services. 2nd ed. westport, ct: libraries unlimited, 1996. kuhlthau, carol c. ‚learning in digital libraries: an information search process approach-children and the digital library. library trends, 45.4 (1997): 708-724. kuhlthau, carol c., leslie k. maniotes, and ann k. caspari. guided inquiry: learning in the 21st century. westport, ct: libraries unlimited, 2007. kuhlthau, carol c., betty j. turock, mary w. george, and robert j. belvin. "validating a model of the search process: a comparison of academic, public and school library users.‛ library and information science research, 12 (1990): 5-32. kulleseid, ellen r. ‚extending the research base: scheme theory, cognitive styles, and types of intelligence.‛ school library media quarterly 15.1 (fall 1986): 41-8. lewin, kurt. field theory in social science: selected theoretical papers. ed: d. cartwright. new york: harper & row, 1951. lewin, kurt. ‚action research and minority problems.‛ journal of social issues 2 (1948): 3446. http://www.ala.org/ http://www.ala.org evidence based library and information practice 2009, 4:2 76 mckernan, james. curriculum action research: a handbook of methods and resources for the reflective practitioner. 2nd ed. london: routledge, 1996. orlikowski, wanda. j., and baroudi, jack j. ‚studying information technology in organizations: research approaches and assumptions.‛ information systems research 2.1 (1991): 1-28. partnership for 21st century skills. 2004. 1 may 2009 <http://www.21stcenturyskills.org>. patterson, leslie. ‚reliving the learning: learning from classroom talk and texts.‛ research in the classroom: talk, texts, and inquiry. eds. zoe donoahue, mary ann vantassell and leslie patterson. newark, de: international reading association, 1996, 3-9. patterson, leslie, and patrick shannon. ‚reflection, inquiry, and action.‛ teachers are researchers: reflection and action. eds. leslie patterson, carol minnick santa, kathy g. short, and karen smith. newark, de: international reading association, 1993, 7-11. piaget, jean. judgment and reasoning in the child. new york: harcourt press, 1928. richelle, marc n. b.f. skinner: a reappraisal. mahwah, nj: lawrence erlbaum, 1993. robertson, stephen e., and karen sparckjones. ‚relevance weighting of search terms.‛ journal of the american society for information science 27.3, (1976): 129-46. rose, amy d. ‚differing modes of inquiry in the field of adult education.‛ adult learning 11.3 (2001): 23-4. saracevic, tefko. introduction to information science. new york: bowker, 1970. schamber, linda, and michael eisenberg. ‚relevance: the search for a definition.‛ american society for information science, atlanta, ga, 1988. todd, ross j. ‚transitions for preferred futures of school libraries: knowledge space, not information place-connections, not collectionsactions, not positions-evidence, not advocacy.‛ international association of school librarianship conference, auckland, new zealand, july 10, 2001. todd, ross j. ‚evidence-based practice manifesto for school librarians.‛ school library journal 54.4 (1 apr. 2008): 38-43; 11 may 2009 <http://www.schoollibraryjournal.co m/article/ca6545434.html>. vygotskii, lev s. mind in society: the development of higher psychological processes. cambridge, ma: harvard university press, 1978. wiggins, grant. ‚the case for authentic assessment,‛ practical assessment, research & evaluation 1990; 2.2. 1 may 2009 <http://pareonline.net/getvn.asp?v =2&n=2>. wiggins, grant. ‚creating tests worth taking,‛ educational leadership 49.8 (may 1992): 26–33. http://www.21stcenturyskills.org http://www.schoollibraryjournal.com/article/ca6545434.html http://www.schoollibraryjournal.com/article/ca6545434.html http://www.schoollibraryjournal.com/article/ca6545434.html http://pareonline.net/getvn.asp?v=2&n=2 http://pareonline.net/getvn.asp?v=2&n=2 http://pareonline.net/getvn.asp?v=2&n=2 evidence based library and information practice 2009, 4:2 77 wiggins, grant. educative assessment: designing assessments to inform and improve student performance. san francisco: jossey-bass publishers, 1998. research article   reuse of wikimedia commons cultural heritage images on the wider web   elizabeth kelly, c.a., d.a.s. digital programs coordinator loyola university new orleans, louisiana, united states of america email: ejkelly@loyno.edu    received: 23 apr. 2019                                                                  accepted: 25 may 2019      2019 kelly. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29575     abstract   objective – cultural heritage institutions with digital images on wikimedia commons want to know if and how those images are being reused. this study attempts to gauge the impact of digital cultural heritage images from wikimedia commons by using reverse image lookup (ril) to determine the quantity and content of different types of reuse, barriers to using ril to assess reuse, and whether reused digital cultural heritage images from wikimedia commons include licensing information.   methods – 171 digital cultural heritage wikimedia commons images from 51 cultural heritage institutions were searched using the google images “search by image” tool to find instances of reuse. content analysis of the digital cultural heritage images and the context in which they were reused was conducted to apply broad content categories. reuse within wikimedia foundation projects was also recorded.   results – a total of 1,533 reuse instances found via google images and wikimedia commons’ file usage reports were analyzed. over half of reuse occurred within wikimedia projects or wiki aggregator and mirror sites. notable people, people, historic events, and buildings and locations were the most widely reused topics of digital cultural heritage both within wikimedia projects and beyond, while social, media gallery, news, and education websites were the most likely places to find reuse outside of wiki projects. however, the content of reused images varied slightly depending on the website type on which they were found. very few instances of reuse included licensing information, and those that did often were incorrect. reuse of cultural heritage images from wikimedia commons was either done without added context or content, as in the case of media galleries, or was done in ways that did not distort or mischaracterize the images being reused.   conclusion – cultural heritage institutions can use this research to focus digitization and digital content marketing efforts in order to optimize reuse by the types of websites and users that best meet their institution’s mission. institutions that fear reuse without attribution have reason for concern as the practice of reusing both creative commons and public domain media without rights statements is widespread. more research needs to be conducted to determine if notability of institution or collection affects likelihood of reuse, as preliminary results show a weak correlation between number of images searched and number of images reused per institution. ril technology is a reliable method of finding image reuse but is a labour-intensive process that may best be conducted for selected images and specific assessment campaigns. finally, the reused content and context categories developed here may contribute to a standardized set of codes for assessing digital cultural heritage reuse.     introduction   cultural heritage institutions with digital images online want to know if and how those images are being reused. whether the image was uploaded to a digital library by the institution or added to a website by an individual user, knowledge and understanding of digital image reuse helps cultural institutions determine the impact of their collections as well as whether they are meeting the needs of their users. one method of measuring reuse of digital images online is reverse image lookup (ril), in which the ril service searches the internet for other versions of an image. recent scholarship includes several ril studies of digital cultural heritage media from specific collections or institutions. however, research by the wikimedia foundation has found that cultural heritage institutions with digital media in wikimedia commons, the media repository for wikimedia foundation projects, want better understanding of the impact of their uploaded media, in particular as it relates to institutional goals (research:supporting commons contribution, 2018). as increasing numbers of cultural heritage institutions upload their digital media to wikimedia commons, and as users add digital cultural media found during their own research, the opportunity and necessity of assessing the impact of these objects becomes more relevant.   this study attempts to gauge the impact that digital cultural heritage images from wikimedia commons have both in and beyond wiki projects by using ril to determine quantity and quality of different types of reuse while also identifying barriers to assessing reuse in this way. rooted in empirical evidence, this study will provide concrete examples of how digital cultural heritage from wikimedia commons is used outside of the wikimedia landscape along with documented steps for finding and analyzing image reuse in order to facilitate greater reuse research among digital cultural heritage stakeholders, leading to improvements in efforts to make digital collections more widely available and reusable.   media reuse studies   media reuse research is still a relatively new field without standard or widely accepted definitions of use and reuse. the digital library federation assessment interest group (dlf-aig) content reuse working group completed a 1-year institute of museum and library services (imls) grant in 2018 to evaluate the needs and functions of a digital library reuse toolkit, and in doing so also researched digital library stakeholder interpretations of use and reuse. while refined definitions of use and reuse by the group are forthcoming, at this time and for the purposes of this paper reuse will be defined as “how often and in what ways digital library materials are utilized and repurposed” and in what contexts (o’gara et al., 2018). collection curators, digital librarians, and archivists find value in assessing the reuse of their digital collections in order to show the collection’s reach and to determine who uses collections. this data can then be used to make decisions about collection development and digitization priorities as well as to negotiate increases in staffing and funding.   while digital library stakeholders find a great deal of value in assessing the reuse of their collections, they also find it very difficult to do. a survey administered by the dlf-aig content reuse imls project team found that only 40% of respondents were gathering reuse data, usually from social media metrics or citation analysis (o’gara et al., 2018).   there is also tension between cultural heritage organizations’ missions to provide access and a desire to maintain control over collections. sometimes there are valid and commendable reasons for wishing to restrict access or mediate use and reuse of digital collections. digital content misuse and cultural appropriation are concerns for digital library stakeholders (o’gara et al., 2018). ethnographic archives, especially those that document the history and cultures of marginalized populations, prove challenging to determine meaningful impact beyond simple quantitative metrics such as clicks, likes, and downloads (punzalan et al., 2018). other times, however, archives unnecessarily attempt to control reuse of their online holdings via restrictive or unclear rights statements (dryden, 2014).   while published literature about media reuse is still somewhat limited, the existing scholarship primarily focuses on use and reuse of specific archival and digital collections, reuse of generalized collections by scholars within specific areas of study, and reuse of specific types of media. these studies are often undertaken with the purpose of improving the services and technological infrastructure that make library and archival collections reusable by researchers. studies involving focus groups, observational research, and citation analyses have evaluated the reuse of archival images by historians, archaeologists, architects, and artists (beaudoin, 2014; harris & hepburn, 2013). additional researchers, after creating or using digital media collections in their own work, have advocated for the creation of open-licensed digital collections of geology and film in order to enhance the research process for students and scholars alike (o’sullivan, 2017; rygel, 2013).   the reuse of digital cultural heritage media on social media platforms has received increasing attention in the scholarly literature over the course of the last decade. as noted in one study, “our data indicate that everyday users are repurposing digital content in ways that are meaningful to them, and they are acknowledging and fulfilling personal interests. these users are also sharing this content through a variety of environments on the web, including popular social media platforms, blogs, and personal web sites” (reilly & thompson, 2017). social media platforms like pinterest, which allow users to curate personal collections of images, blog posts, and other media from the web, have an “archival shape” due to their infrastructure that captures the provenance, or original source, of the item, making such platforms rich for analysis by media reuse researchers (summers, 2019). examples of cultural heritage media reuse could include images downloaded from digital library collections and uploaded onto a pinterest pinboard, as well as those reproduced in commercial projects like artwork or included in official government reports (thompson & reilly, 2017). reuse of digital cultural heritage media on wikimedia commons, wikipedia, and other wikimedia foundation projects has also received scholarly attention in the last year (kelly, 2018; morley, 2018).   one of the most widely documented methods for evaluating digital image reuse involves ril services such as google images or tineye, in which an image is either uploaded or an originating url is input to the search platform and then duplicates and similar images are found online. ril studies have been performed on images from nasa, academic digital libraries, the library of congress, and the british national gallery (kelly, 2015; kirton & terras, 2013; kousha et al., 2010; reilly & thompson, 2014; reilly & thompson, 2017). in all of these studies, after duplicate images were found online, the context and purpose in which the images were reused was analyzed in order to determine who uses digital cultural heritage images and for what objective.   cultural heritage, wikimedia, and impact   a ready-made platform for sharing digital cultural heritage media and encouraging reuse can be found in wikimedia commons (commons.wikimedia.org), the wikimedia foundation’s repository for photographs, artwork, video, sound, diagrams, and more. many cultural heritage institutions have developed programs to upload their digital media to wikimedia commons and enhance wikipedia articles with links to their collections and finding aids in order to increase traffic to their websites and repositories, typically with impressive results (kelly, 2018). digital cultural heritage media is added to wikimedia commons in a few ways:   ●        cultural heritage institutions upload media from their own existing digital collections; ●        cultural heritage institutions upload media directly to the commons, especially in the case of smaller institutions without existing digital repositories; ●        cultural heritage institutions and users upload media to other repositories or websites, such as flickr and the internet archive, that are then crawled by bots and added to the commons; ●        users upload media from cultural heritage institution digital collections; ●        users make their own digital reproductions of cultural heritage collections (for example, photographing a painting in a museum, or a document in an archive) and then upload them to the commons.   wikimedia commons provides user guidelines on how to reuse media from the commons on wikimedia platforms as well as outside of the wikimedia landscape (commons:first steps/reuse, 2019; commons:reusing content, 2018; commons:simple media reuse guide, 2018). but just as digital library stakeholders struggle to assess reuse of the media in their own repositories, wikipedia editors and authors, or wikipedians, struggle to assess reuse of projects, articles, and media from wikimedia foundation programs. denny vrandečić points out that readily available use metrics do not always show what is valuable or important, and instead “we should focus on measuring how much knowledge we allow every human to share in, instead of number of articles or active editors” (2014). another wikipedian argues that "the sum of human knowledge" is not the same concept as "the sum of what everyone is googling today" and that reach, importance, diversity and content gaps, uniqueness, and quality are all necessary primary measures of impact for the wikimedia movement (user:the land, 2018). the wikimedia foundation “supporting commons contribution by glam institutions” research project (glam standing for galleries, libraries, archives, and museums) noted that for cultural heritage organizations, “donating media to commons is a means to an end. glam organizations and the volunteers who work with them want to know the media they upload is being used, and to be able to evaluate the impact of their donations against institutional goals” (research:supporting commons contribution, 2018).   aims and methods   research questions   this study attempts to answer the following questions with the hopes of providing concrete strategies for assessing collection reuse to cultural heritage institutions:   what is the content of cultural heritage images found in wikimedia commons? what content gets reused most often, and where? do reused cultural heritage images from wikimedia commons carry license or attribution information with them     ?   research methods   a list of cultural heritage repositories, including museums, historical associations, and academic archives, among others, was generated from the archival discovery tool archivegrid, and a random number generator was used to pull a sample of 66 institutions from the list for inclusion in this study. searches were conducted over a two-week period for images      from these institutions’ collections, determined primarily by examining the “source” field in the wikimedia commons object metadata. while images documenting an institution’s buildings or grounds were not included in the study, user-generated photos or videos of collections, such as pictures taken of an artwork or exhibit, were included. the number of results for each institution varied greatly, with some institutions not having any related images in wikimedia commons and others having hundreds of results. a list of all institutions and counts of their reuse results is available in appendix a: 51 of the 66 institutions had digital images in wikimedia commons. as the purpose of this study was not to determine how many cultural heritage institutions have images in wikimedia commons, or how many images institutions have on average, not all results were analyzed; instead, at most 20 results from each institution were documented.[1]                 a total of 308 images from cultural heritage institutions were initially analyzed. a separate research project is underway to assess the validity of rights statements provided in wikimedia commons for all of these results. for the purposes of this study, a smaller subset was extracted for ril analyses. all results from the initial 308 images with creative commons or other open licenses were selected for inclusion, as one research question pertinent to this study is how often evidence of open licensing is available when images are reused. these accounted for 44 images to be searched using ril; an additional 126 public domain images, and two instances of images published with copyright permission from the wikimedia commons cultural heritage sample set, were selected for inclusion as well.                 wikimedia commons includes wiki reuse information on the record page for uploaded media; the number of instances of reuse, both on wikimedia commons and on other wikis, was noted for each object (see figure 1).   figure 1 screenshot of wikimedia commons file usage for “hume spring (c.1900) owned by frank hume (pictured far right).jpg” (https://commons.wikimedia.org/wiki/file:hume_spring_(c.1900)_owned_by_frank_hume_(pictured_far_right).jpg).   then each image was searched using the google chrome browser “search by image” function. when available, the option to search google for “all sizes” of the image, as opposed to just those matching the original image, was selected to receive the greatest amount of results (see figure 2).   figure 2 screenshot of google images result with multiple sizes.   for each image, a number of elements were recorded. these included:   ●        repository name ●        search term ●        wikimedia commons result url ●        original medium of reused media ●        content of reused media ●        wikimedia licensing ●        reuse url ●        reuse context (narrow) ●        reuse context (broad)    ●        reuse license and attribution ●        reuse license (categorized) ●        license compatibility     ●        notes   most of the elements only required simple analysis of frequency counts. for elements with a greater level of subjectivity, such as “content of reused media” and “reuse context,” the content analysis method was used to examine each object, label it, and then categorize the labels into broader themes. content analysis is a quantitative research method used to “examine large amounts of data in a systematic fashion, which helps to identify and clarify topics of interest” (drisko & maschi, 2015, pp. 25). here, codes or categories were developed inductively, or without a prior scheme, rather than deductively, as reuse research is still in its infancy and existing codes and theory are diverse and not yet synthesized. however, it should be noted that content analysis of some type was conducted in all of the ril studies previously mentioned, so the potential for integrating codes and developing a standard set for assessing cultural heritage via ril may be a possibility in the future. in this study, the websites featuring wikimedia commons digital cultural heritage images were analyzed as to the site’s purpose. many results were in languages other than english; for these, google translate was used to infer the content of the site. following the analysis and application of codes, tables and graphs were generated to assist in conveying the results of the study.   results   from 171 digital cultural heritage wikimedia commons images searched in google images, 34 did not have any results. of the remaining 137 images, one had been deleted in wikimedia commons since initial data collection began and couldn’t be searched in google, and two did not have any wiki results and only had results in google images that were false positives. over 25% of google images results were also discarded as being unusable. these included dead links; false results in which the image was not found on the site; spam, porn websites, and sites blocked by the computer’s antivirus program; one instance of a website that was behind a paywall; and a site that google translate could not decipher. to ensure that remaining analysis was based on true reuse, any result found by google images that matched the “source” field in wikimedia commons (for example, if the source of a painting was given as a museum, and google images located the painting on the museum’s website) was removed from analysis. finally, 21 results were for videos of a zoetrope at a museum. while instances of wiki reuse could be analyzed for these images, they were not suitable for google images, so they were removed as well. after fully cleaning any unusable, false, non-reuse, or missing results, a total of 1,533 google images and wiki search results from 51 cultural heritage institutions remained for analysis.   approximately 5% of reuse cases from the total uncleaned data set, and 51% of the cleaned data set were associated with wikimedia’s projects. this includes reuse on other wikimedia commons pages like galleries or featured images; reuse on other wikimedia projects, like wikipedia articles and wikidata; reuse by wiki mirror sites, or exact replicas of wiki projects hosted at different urls; and reuse by wiki aggregators, or sites that pull content straight from wikimedia and repurpose it for readability, content curation, usability, or other reasons (such as wikiwand and wikivividly). while wiki aggregator and mirror results were found through google images, they weren’t considered to be true examples of reuse as they simply copied entire wikipedia articles or wikimedia commons galleries without providing any additional context or value to the original wikimedia commons object.   table 1 reuse results for wikimedia commons digital cultural heritage images on wikis and related sites wiki results result type count wiki 611 wiki aggregator 158 wiki mirror site 9   the subject matter of the digital images analyzed from wikimedia commons was coded, and then google images results were analyzed to determine themes in what reusers of digital cultural heritage images are most likely to reuse. note that these subjects are not one-to-one coordinates for each image; a single image could have multiple subjects. instead, these numbers represent general areas that reusers of digital cultural heritage tend to focus on when reusing images online. a full description of the codes used to label image content can be found in appendix b. notable people or people were included in more than half of the reuse results, while images documenting historical events and buildings and locations were also widely reused. several categories identified in the initial image analysis were not reused at all outside of wiki products; these were book cover, book plate, data, diaries and personal letters, and library card.   table 2 content of reused wikimedia commons digital cultural heritage images found by google imagesa reuse content (not including wiki reuse) count percent notable people 338 31% people 251 23% historic event 157 15% buildings and locations 103 10% historic object 34 3% technology 33 3% map 32 3% animals 32 3% landscape 25 2% sports 24 2% other 56 5% a “other” includes fibre art, flowers and plants, outdoor photography, religious iconography, abstract art, diploma, currency, literature, and yearbook photos.   similar results can be found in analyzing just the reuse of these images on other wikis. the primary difference is that more of the image categories were reused in wiki products, with yearbook photos the only image content that was not reused at all. also, while the content of reused images varies slightly depending on whether the image is reused on a wiki project or elsewhere, there is generally a strong correlation (r=0.66) between wiki reuse and non-wiki reuse.   table 3 content of reused wikimedia commons digital cultural heritage images found on other wiki platforms (wikipedia, other wikimedia commons page, wiki aggregators, and wiki mirror sites) reuse content (wiki only) count percent notable people 386 36% people 159 15% buildings and locations 110 11% historic event 96 9% sports 52 5% technology 42 4% animals 33 3% book cover 28 3% fiber art 26 2% currency 24 2% landscape 23 2% otherb 83 8% b “other” includes historic object, outdoor photography, map, advertisement, diaries and personal letters, literature, flowers and plants, religious iconography, abstract art, bookplate, diploma, data, and library card.   finally, for comparison’s sake, the following table shows the percentage of instances for each reuse content category found within the initial cleaned data set. this shows a strong correlation between the number of images labeled with a content category and the number of times reused (r=0.84). however, people accounted for 38% of the data set but were only reused in 19% of reuse occurrences, while notable people accounted for 24% of the data set but were reused in 34% of instances. historic events (3% original, 12% reuse) also had a higher level of reuse.   table 4 comparison of wikimedia commons image content categories and overall reuse of those categoriesc reuse content category occurrences in data set (before reuse analysis) reuse occurrences  (wiki and google images) people 38% 19% notable people 24% 34% buildings and locations 7% 10% technology 4% 4% sports 4% 4% animals 4% 3% historic event 3% 12% landscape 2% 2% otherd 13% 13% c table’s percentages do not sum to 100% due to rounding up small percentages, d “other” includes outdoor photography, advertisement, book cover, historic object, map, diaries and personal letters, literature, religious iconography, data, fibre art, currency, flowers and plants, abstract art, bookplate, diploma, library card, and yearbook photos.   the original medium of the reused object was also documented and analyzed. photographs accounted for nearly three quarters of all reuse.   table 5 original medium of reused images   original medium of reused media count percent photograph 1104 72% two-dimensional artwork 139 9% illustration 56 4% three-dimensional artwork 54 4% map 44 3% ephemera 42 3% exhibit 34 2% monograph 26 2% othere 34 1% e “other” includes document, slide, drawing, newspaper, and three-dimensional object.   when looking at reuse outside of wiki products, there are again clear trends in how and where digital cultural heritage images are being reused. social websites, defined here to include social media, blogs, discussion boards, online journals, and other websites whose primary purpose is user-generated content and interaction, account for just under half of reuse instances outside of wiki platforms. media galleries, or user-curated collections of media (usually images), and news websites are also popular scenes for digital cultural heritage reuse. only 11% of google images results for wikimedia commons digital cultural images      were on educational sites like research guides, encyclopedias, and historical timelines. full definitions of the codes used to categorize reuse context are in appendix c.   table 6 context of reuse of wikimedia commons digital cultural heritage images found by google imagesf google images reuse context count percent social 371 49% media galleries 137 18% news 133 18% education 80 11% profiles of people and places 14 2% commerce 9 1% events 5 1% web design and development 5 1% tourism 1 0% ftable’s percentages do not sum to 100% due to rounding up small percentages.   slight variances in what subject matter is most viable for reuse on what type of websites can be found as well. while images representing notable people are the most popular reuse type across all websites, maps are almost exclusively found on social sites, whereas images representing historical objects are primarily reused by news sites. delving further into what subjects are reused most by different types of websites may help cultural heritage institutions pinpoint where their digitization and marketing efforts should lie in order to meet institutional priorities.   figure 3 reuse context of wikimedia commons digital cultural heritage content found by google images (excerpt).   wikimedia provides ample guidelines on how wiki media should be shared from wiki platforms, including providing appropriate attribution if required by the media’s license. of the sample set analyzed for this study, a mere 40 results out of a possible 755 non-wiki reuse instances had any type of license or copyright statement available. and in comparing the licenses provided in reuse instances, there were significant discrepancies between these and the licenses on wikimedia commons. “compatible” refers to instances where the wikimedia commons object and the reused object had the exact same license. the “semi-compatible” designation was used when slight differences occurred, for example, the wikimedia commons license listed cc by-sa 3.0, whereas the reused instance noted an updated cc by-sa 4.0 license. the remaining “incompatible” results referred to wholly different licenses being applied, such as wikimedia commons marking an image as being in the public domain where another website included a creative commons or copyright statement alongside the object. the two images that were copyrighted but published to wikimedia commons with permission were reused four times outside of wiki products, but none of the reuse instances included a license or attribution.   table 7 compatibility of reuse licenses found by google images with original wikimedia commons license wikimedia and non-wiki reuse license compatibility compatibility evaluation count compatible 24 incompatible 8 semi-compatible 8   finally, a few other unexpected discoveries emerged in this analysis. while only 40 reuse instances provided some sort of license, 147 results, or 19% of non-wiki reuse results at least included some sort of credit, such as the name of the work and the cultural heritage institution that held it. of these, 50 credited wikimedia commons or wikipedia in some way, or linked back to the original image on wikimedia commons.   also, in analyzing the reuse context of the digital cultural heritage images outside of wikimedia, only three results appeared to be entirely “misused.” these involved the following misidentifications or questionable reuse situations:   ●        a news article that uses an unlabeled photo of the 1966 ut austin tower shooter charles whitman’s gun to illustrate new laws for gun amnesty in canada; ●        a blog post that mislabels an image of gerald ford as richard nixon; ●        an image of railway workers laying the last rail of the union pacific railroad in 1869, used to illustrate minimum wage.   overall, reuse of cultural heritage images from wikimedia commons was either done without added context or content, as in the case of media galleries, or was done so in ways that did not distort or mischaracterize the image being reused.   conclusions   by identifying themes in what type of digital cultural heritage is reused online and where, we can begin to pinpoint possible strategies for cultural heritage institutions to maximize the impact of their digital images depending on institutional priorities. for example, institutions hoping to increase use of their collections by news organizations should focus wikimedia commons donation efforts on images related to notable people, historic photos of unidentified people, and historical events, but should also observe that photographs of historic objects ranked highly in reuse by news organizations. however, this study does not delve into great detail as to the content and context of images reused. in this sample set, all of the images labeled as “historic object” were photographs of university of texas shooter charles whitman’s guns. does this mean that images of weapons in general might be reused more by news organizations than other      topics, or would images of other historic objects be reused as frequently? this question could be tested by conducting reuse analysis on wikimedia commons images of both historic weapons and generic images of weapons, or of historic weapons and other historic objects. additional media reuse research should continue to narrow down what exactly makes one media object more reusable than another. factors such as notability or fame, uniqueness, presentation, artistic merit, and others may be analyzed to further understand reuse priorities.                 this study also does not attempt to measure the notability of specific cultural heritage institutions or collections. previous scholarship documenting cultural heritage institutions voluntarily donating digital images to wikimedia commons focuses almost exclusively on large research universities, many of whom have internationally-recognized collections. it is unknown whether smaller institutions with lesser-known or niche collections would see similar increases in website traffic or similar reuse of their digital images. while this study includes a variety of institution sizes and types, it does not attempt to qualify the notability of these institutions, nor of their collections or individual images. we can, however, see that there is a weak correlation (r=0.27) between how many images were searched from each repository and how many instances of reuse were found, so content and quality of the reused object may be larger factors in determining reuse than quantity of object per institution.   the research reported here shows that cultural heritage institutions have cause for concern about reuse of their collections without attribution. only 9% of creative commons-licensed images      that were reused outside of wiki projects were labeled as creative commons in their new context, only 19% of non-wiki reused images had any sort of credit at all, and most that did, did not include a reuse license or public domain statement. still, at least for images that are in the public domain and don’t legally require a license or attribution, perhaps cultural heritage institutions should be less concerned with attribution and more concerned with increasing reuse. unfortunately, a lack of proper attribution can make tracking reuse difficult, thus impeding the institution’s ability to measure the impact of their collections. strategies such as using ril to locate instances of reuse without text attribution included may be beneficial for image collections, but as of yet the ril process is very labour-intensive and probably unfeasible for institutions to perform on all of their digital images on a regular basis. instead, performing ril reuse analysis on selected images may be undertaken for specific assessment campaigns, such as to assess reuse of a new collection after a year’s time, to show impact for annual reports and reviews, or to highlight the success of marketing campaigns the institution has undertaken related to a collection or object. the dlf-aig imls grant project found that embedded metadata is one of the most-needed pieces of infrastructure for tracking reuse; the wikimedia foundation’s “supporting commons contribution by glam institutions” project similarly identified “demonstrating and preserving media provenance” as a priority (o’gara et al., 2018; research:supporting commons contribution, 2018). improved infrastructure for embedded or sticky” metadata may allow reuse assessment without the need for formal attribution.   what cultural heritage institutions can begin to do with this research is to determine where their digitization efforts may have the most impact and alignment with institutional goals. the dlf-aig imls grant project found that digital library practitioners had different priorities for where they hoped their digital resources would be reused; for example, some institutions might find more value in reuse by nationally-recognized news organizations, others by students and scholars, still others by community groups (o’gara et al., 2018). these goals will vary depending on the type, size, and mission of the institution the practitioner represents. by beginning to understand what types of wikimedia commons digital cultural heritage content are reused most often on what types of websites, practitioners can strategize which of their collections and objects they should focus on donating to wikimedia commons to reach the user communities they are most interested in connecting with.   figure 4 relationship between number of images searched and number of reuse results per institution.   while great care was taken in developing and analyzing the codes used for identifying content and context of reused images, it should be noted that content analysis as a method is highly subjective but often made less so by involving multiple researchers who “norm” their codes to come to agreement about classification. as this study was undertaken by a sole researcher, elements determined by content analysis may bear a higher level of subjectivity than is desired.   this paper contributes to media reuse literature, and to ril research in particular by furthering understanding of what content categories are most likely to be reused and where, both within wikimedia foundation projects and on the wider web. digital library practitioners should use the results of this study to develop digitization strategies that prioritize content attractive to the types of websites where reuse would most align with their institutional missions. this research also emphasizes the need for better education and infrastructure related to licensing and rights for digital content reuse, as reused digital cultural heritage images from wikimedia commons rarely includes attribution or licensing information. the content categories developed here may be combined with content categories found in other ril studies to begin synthesizing a common code of subjects for assessing image reuse. by continuing to deepen understanding of digital cultural heritage reuse, we can better assess the impact of our collections online and strive to meet the needs of current and potential users in line with institutional priorities and missions.   references   beaudoin, j. e. (2014). a framework of image use among archaeologists, architects, art historians and artists. journal of documentation, 70(1), 119–147. https://doi.org/10.1108/jd-12-2012-0157 commons:first steps/reuse. (2019). in wikimedia commons. retrieved 11 apr. 2019 from https://commons.wikimedia.org/wiki/commons:first_steps/reuse commons:reusing content outside wikimedia/technical. (2018). in wikimedia commons. retrieved 11 apr. 2019 from https://commons.wikimedia.org/wiki/commons:reusing_content_outside_wikimedia/technical   commons:simple media reuse guide. (2018). in wikimedia commons. retrieved 11 apr. 2019 from https://commons.wikimedia.org/wiki/commons:simple_media_reuse_guide   drisko, j., & maschi, t. (2015). content analysis. new york, ny: oxford university press.    dryden, j. (2014). just let it go? controlling reuse of online holdings. archivaria, (77), 43–71. retrieved from https://archivaria.ca/index.php/archivaria/article/view/13486   harris, v., & hepburn, p. (2013). trends in image use by historians and the implications for librarians and archivists. college & research libraries, 74(3). https://doi.org/10.5860/crl-345   kelly, e. j. (2015). reverse image lookup of a small academic library digital collection. codex: the journal of the louisiana chapter of the acrl, 3(2), 80–92. retrieved from http://journal.acrlla.org/index.php/codex/article/view/101   kelly, e. j. (2018). use of louisiana’s digital cultural heritage by wikipedians. journal of web librarianship, 12(2), 85–106. https://doi.org/10.1080/19322909.2017.1391733   kelly, e. j. (2019). 2019 wikimedia commons digital cultural media analysis (version 1) [figshare].     kirton, i., & terras, m. (2013). where do images of art go once they go online? a reverse image lookup study to assess the dissemination of digitized cultural heritage. mw2013: museums and the web 2013, portland, or. retrieved 22 apr. 2019 from https://mw2013.museumsandtheweb.com/paper/where-do-images-of-art-go-once-they-go-online-a-reverse-image-lookup-study-to-assess-the-dissemination-of-digitized-cultural-heritage/ kousha, k., thelwall, m., & rezaie, s. (2010). can the impact of scholarly images be assessed online? an exploratory study using image identification technology. journal of the american society for information science & technology, 61(9), 1734–1744.     morley, james. (2018). use and impact of cultural heritage images on wikimedia commons and wikipedia. in catching the rain. retrieved 18 mar. 2019 from http://www.catchingtherain.com/portfolio/use-and-impact-of-cultural-heritage-images-on-wikimedia-commons-and-wikipedia/ o’gara, g. m., woolcott, l., joan kelly, e., muglia, c., stein, a., & thompson, s. (2018). barriers and solutions to assessing digital library reuse: preliminary findings. performance measurement & metrics, 19(3), 130–141. https://doi.org/10.1108/pmm-03-2018-0012 o’sullivan, s. (2017). archives for education: the creative reuse of moving images in the united kingdom. the moving image, 17(2), xvi–19. https://doi.org/10.5749/movingimage.17.2.0001   punzalan, r. l., marsh, d. e., & cools, k. (2018). beyond clicks, likes, and downloads: identifying meaningful impacts for digitized ethnographic archives. archivaria, 84. retrieved from https://archivaria.ca/index.php/archivaria/article/view/13614   reilly, m., & thompson, s. (2014). understanding ultimate use data and its implication for digital library management: a case study. journal of web librarianship, 8(2), 196–213. https://doi.org/10.1080/19322909.2014.901211 reilly, m., & thompson, s. (2017). reverse image lookup: assessing digital library users and reuses. journal of web librarianship, 11(1), 56–68.  https://doi.org/10.1080/19322909.2016.1223573 research:supporting commons contribution by glam institutions. (2018). in meta-wiki.  retrieved 15 apr. 2019 from https://meta.wikimedia.org/wiki/research:supporting_commons_contribution_by_glam_institutions rygel, m. c. (2013). share and share alike: using wikimedia commons to disseminate geophotography. abstracts with programs geological society of america, 45(7), 381–381.   summers, e. (2019). archival shapes. in inkdroid. retrieved 11 apr. 2019 from https://inkdroid.org/2019/01/03/archival-shapes/ thompson, s., & reilly, m. (2017). “a picture is worth a thousand words”: reverse image lookup and digital library assessment. journal of the association for information science & technology, 68(9), 2264–2266. https://doi.org/10.1002/asi.23847   user:the land/thinking about the impact of the wikimedia movement. (2018). in meta-wiki. retrieved 15 apr. 2019 from https://meta.wikimedia.org/wiki/user:the_land/thinking_about_the_impact_of_the_wikimedia_movement vrandečić, d. (2014). a new metric for wikimedia. in wikipedia signpost. retrieved 22 apr. 22 2019 from https://en.wikipedia.org/w/index.php?title=wikipedia:wikipedia_signpost/2014-08-20/op-ed&oldid=671617000     appendix a list of cultural heritage institutions with reuse results   cultural heritage institution total reuse wikimedia commons google image alexandria library 8 7 1 amon carter museum 49 28 21 arizona state museum 41 26 15 austin public library austin history center 205 56 149 bard college 12 9 3 barnes foundation 16 9 7 central michigan university clarke historic library 6 5 1 centre college grace doherty library 28 27 1 chula vista public library 3 3 0 cincinnati art museum 14 6 8 cleveland public library 22 20 2 college of charleston 15 11 4 college of physicians of philadelphia 1 1 0 college of william and mary 15 12 3 computer history museum 67 41 26 district of columbia public library 22 10 12 folger shakespeare library 16 11 5 forest history society 9 6 3 fresno city and county historic society archives 6 5 1 georgetown university 76 36 40 gerald r. ford library 137 40 97 hagley museum and library 17 8 9 idaho state university 9 9 0 indiana university 14 5 9 lamar university 5 4 1 missouri state university 35 7 28 national gallery of art 8 1 7 oakland museum 31 12 19 princeton university firestone library 13 6 7 richmond public library 3 3 0 saint mary's college 7 4 3 santa clara university 6 4 2 seton hall university 4 4 0 smithsonian institution archives 51 44 7 stanford university archive 16 14 2 tennessee state university 35 11 24 the henry ford benson ford research center 7 0 7 trinity college 4 4 0 university of denver 5 2 3 university of idaho 22 15 7 university of louisiana at lafayette 18 3 15 university of michigan bentley historic library 40 29 11 university of missouri, kansas city 4 0 4 university of north florida 34 5 29 university of pittsburgh 13 11 2 university of puget sound 25 22 3 university of texas at austin 102 52 50 winthrop university 1 0 1 wisconsin historic society 43 27 16 yale beinecke rare book and manuscript library 179 93 86 yale university manuscripts and archives 14 10 4     appendix b image content codes   abstract art: fine art lacking recognizable visual references   advertisement: images used for the purpose of promoting a product or service, usually for monetary gain   animals: non-human biological organisms from the kingdom animalia   book cover: the front of a published monograph   bookplate: identification labels used by monograph owners   buildings and locations: architectural structures, cityscapes, towns, and non-landscape locales   currency: representations of paper or coin money   data: tables and figures used for illustrative purposes to convey information   diaries and personal letters: manuscript materials such as personal writings and correspondence   diploma: paper documenting graduation from some level of education   fibre art: fine art composed of natural or synthetic components like yarn, thread, and string; examples include tapestries, rugs, and embroidery   flowers and plants: multicellular organisms from the kingdom plantae   historic event: documentation of occurrences with remarkable significance   historic object: documentation of objects with remarkable significance   landscape: natural scenery   library card: identification used to access items at a library   literature: written works, usually published monographs   map: visual depiction of geographic spaces   notable people: individuals identified by name due to their cultural or historical recognizability on wikimedia commons   outdoor photography: camera images of the outdoors   people: primarily unidentified individuals primarily or, in a few cases, identified because their images came from yearbook scans but were otherwise not to be found identified elsewhere online   religious iconography: fine art created for the specific purpose of use in or by religious organizations and individuals   sports: athletic events, spaces, or people associated with specific athletic activities   technology: machines and systems used for carrying out technical processes   yearbook photos: images captured for school publications documenting an academic year     appendix c reuse context codes   broad code narrow codes definition commerce art store dvd reproduction for purchase trade catalogue websites whose primary purpose is the sale of commercial products education academic website dictionary digital exhibit digital library ebook encyclopedia google arts & culture page infographic institution website on this day presentation quiz quote website report research guide slide deck timeline tutorial video reference resources such as dictionaries, encyclopedias, research guides, digital libraries and exhibits, timelines, presentation slides, infographics, “on this day” websites, and academic websites events event post movie listing news or other websites with calendar or public relations-related announcements about specific events like workshops, classes, performances, and exhibits media galleries clip art gallery flickr media gallery stock image gallery websites made up of manually or automatically-generated collections of images news article magazine news article newsletter press release online publishing by television, online, radio, and print news organizations, as well as magazines and other websites for current events profiles of people and places city or company profile person profile generalized biographies or profiles of cities and towns found on specialty topic, non-educational websites social blog discussion board facebook google plus journal message board pin board pinterest reddit song lyrics annotation site tweet twitter aggregator social networks (facebook, pinterest, twitter), blogs, discussion boards, online journals, and other web 2.0 websites whose primary purpose is user-generated content and interaction tourism travel site travel websites web design and development keyword trends tools for website development such as identifying keyword trends for search engine optimization   [1] the raw, cleaned dataset used for this research paper is available in the author’s figshare repository (kelly, 2019). research article   may i borrow a stapler? is this all students ask at the service desk in a university library?    liv inger lamøy research librarian norwegian university of science and technology university library trondheim, norway e-mail: liv.i.lamoy@ntnu.no   astrid kilvik senior research librarian norwegian university of science and technology university library trondheim, norway e-mail: astrid.kilvik@ntnu.no   received: 20 nov. 2020                                                             accepted: 7 jun. 2021          2021 lamøy and kilvik. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   doi: 10.18438/eblip29883     abstract   objective the objective of the study is to increase the knowledge about what questions students ask at the library desk and what the purpose is of their use of the desk. our focus has been on the physical meetings with the students. the aim is to contribute to the discussion on the future development of the library service desk.     methods we recorded questions asked at the desks to explore how students use the library service desks. the recording, where library staff sorted questions into predefined categories, took place over four weeks between the years 2017–2018.   results our recording showed that 63% of the questions asked at the library service desks were about loan services, document delivery, and access to physical and electronic collections. practical things such as opening hours, lost and found items, and the location of the group study rooms, accounted for 16% of questions. questions about information technology (it) made up 8% of questions. finally, the results showed that 8% of the questions from the four weeks of counting were counselling and guidance questions, and 2% were about literature lists, reference management, and reference management tools. we found more questions about counselling and guidance in the spring weeks and more practical questions in the fall. we did not find any clear connection between the number of questions and the size of the branch libraries.   conclusion – by conducting this study, we have learned more about why students use the library desk. our study shows that students come to the library desk to ask about a lot more than just borrowing staples. the results from the study will inform the development of the library desk service going forward.       introduction   norwegian university of science and technology (ntnu) is the largest university in norway today. ntnu specializes in science and technology, and offers a variety of programs of professional study, along with broad range of academic disciplines. ntnu university library is a public scientific library, with branch libraries located on several campuses. the main objective of the library is to support research and education, and students and staff are its primary users. the branch libraries differ in terms of collection size, whether they have special collections or not, and how many faculties and departments they provide services to. some of the branch libraries have only one employee while the largest ones have up to 15. there are service desks at all branch libraries. most of the libraries have only one desk where patrons can ask a variety of questions about circulation, access to electronic and print collections, literature searching, reference management and tools, it, and more. questions are answered at the desks or forwarded to specialists in the library. one of the larger branch libraries has two desks, one for circulation questions and one for counselling and guidance questions.   for some years, the library desk has been the subject of discussions, valuations, and musings about what to do with it, how to staff it, and how to organize it. the fight for attention is hard, and other areas, such as digital content, new user demands, web-based services, and new self-service solutions, have for quite a long period been the centre of attention in libraries in norwegian institutions of higher education. at the same time, the anecdotes about the library desk are very much alive and may sound like: "i just get questions about printers and group study rooms", "the students don’t make use of the desks during the examination period", "the only thing they ask for at the desk is to borrow the stapler". we began to wonder if the anecdotes reflect today's reality. we have extensive experience working at the service desks and these descriptions do not accurately describe what we have seen.   there are several international studies on topics related to the library desk service, recording, and categorizing of inquiries. (gerlich & berard, 2007; henry & neville, 2008; katz, 2002; lemire et al., 2016, lenkart & yu, 2017; radford & connaway, 2013; ryan, 2008; warner, 2001). but so far, there is a lack of norwegian data in the field. to increase our knowledge about what really goes on at the library desk in a large norwegian multi-branch university library, we launched a project called treff (the "desk-project")[1] in 2017. the national library of norway provided financial support for the project. the start of this project was to map the questions received at the desks in all the university's branch libraries. our focus has been on physical meetings with students, and we describe and discuss the process and results in the following article.     literature review    to increase knowledge about what students ask at our university library´s service desks, we analyzed their questions. to facilitate the analysis, we sought out literature about similar recording projects in other libraries. we had two objectives for our literature search. the first was to find a suitable recording form. the second objective was to explore the findings in previous studies, to place our study in a wider scholarly context. because our project included all branch libraries, we needed the recording method to be as simple as possible. in our search, we did not come across any norwegian studies, but we discovered several international ones.                  according to the literature, questions asked at library desks are assigned different classifications as a means of recording the activity. katz (2002) used the traditional categories for classification of reference questions: direction, ready reference, specific-search questions, and research. in 2001, east carolina university was in the process of changing their reference desk organization to a "single point of service", and they saw a need to re-examine how they record reference transaction statistics (warner, 2001). warner created a classification system based on the resources required to answer the question: non-resource-based (level i), skill-based (level ii), strategy-based (level iii), and consultation (level iv). one of the benefits of using this system was that it identified, as early as possible in the encounter, which library staff member should answer the question. the findings from the collection of statistics using the warner classification showed that 80% of the questions from the new service desk fell into levels i and ii and could be answered by students or technicians, while 20% fell into levels iii and iv and would usually be referred to a reference librarian. henry and neville (2008) collected questions received at the reference desk and tested both katz’s resourceand time-based categories and warner’s resource-based categories. the results showed that the directional or non-resource-based questions accounted for 50% or more of the total, the skill-based or ready reference questions 30-40 %, and strategy-based or specific search questions less than 10%. they concluded that warner’s system appeared more applicable, but also that value can be added if time-of-day and time-of-semester activity is included. in a holistic approach to evaluating in-person, email, and chat reference transactions, lemire et al. (2016) considered warner’s and katz’s scales and the read scale. the latter is a six-point scale developed to record the skills, knowledge, techniques, and tools used by the librarian during a reference transaction. on this scale, level 1 questions require the least amount of effort and knowledge, while level 6 require the most effort and are time-expended inquiries. time dedicated to the transaction is also recognized in the read scale (gerlich & berard, 2007).   lemire et al. (2016) chose not to use any of the existing scales because they tended to pre-assign higher value to in-depth, subject-oriented reference questions. the authors believed that, "…even 'simple' question types can give patrons valuable help and can turn into complex information searches" (p. 231). in addition, the existing scales do not consider questions in new areas, like open-access publishing, maker spaces, and so on. the codebook developed by lemire et al. (2016) consists of nine broad categories: library information and policy, circulation/borrowing/reserves, research and reference, locate materials, sfx/ezproxy/off-campus access, technology, print/scan/copy/duplication, feedback and other. the results of their evaluation of reference desk, email, and chat transactions in a shared service point showed that 22% of the questions were in the category library information and policy, 18% were in circulation/borrowing/reserves, 16% in research and reference, 15% in locate materials, and 17% were in the print, technology, feedback and other categories. more detailed classifications have also been developed. a study of the cost-effectiveness of staffing a traditional reference desk in a university library divided 6,959 reference desk transactions into four major categories: directional, look-up, reference, and technology (ryan, 2008). the questions came in person, by phone, or by email. the reference category was subdivided further into eight categories: catalogue search, citation help, database help, guide to correct database(s), personal knowledge or referral, quick internet, research, and serials solutions. the results from ryan's study (2008) showed that 36.3% of the questions were non-informational (did not refer to the collection) or were ‘machine’ transactions (printer and copy issues). directional questions about the collections accounted for 15.4% and quick lookups for 9%. of these questions, 12.4% were about technology (excel, logins, passwords, network), 26.8% were in the reference category, of which 11% were research questions.   radford and connaway (2013) analyzed live chat and instant messaging (im) questions and used the categories of subject search, ready reference procedural, no question, holdings, research, inappropriate directional, and readers’ advisory. the results showed that subject search question frequency had sunk and that there was a shift towards more procedural questions. in their study, lenkart and yu (2017) examined 66,638 in-person, email, and phone transactions from 5 specialized and 2 general reference service points at the university of illinois library. the researchers found that 30.9% of the total number of transactions were directional, 18.1% were about library policies and services, 22.8% were questions about library materials and 6.3% were inquiries for research assistance. in addition, 2.5% were ready reference and 11.6% were related to things like printers, scanners, software and so on.                aims    in this study, we aimed to increase our knowledge about what questions students ask at the library desk in a large norwegian university. what is the purpose of the students’ use of the desk? our focus has been on physical meetings with the students. the research question guiding this study has been: what questions do the students ask at the library service desks in a large, multi-branch library at a norwegian university? by mapping the desk activity and investigating students' inquiries at all branch libraries, we hope to contribute to the discussion on the future development of the library service desk.   methods     to understand how students use the library service desks, we recorded the questions they asked at the desks. we started out searching for a suitable recording form, but the forms we found in the literature were too complex for our project. for instance, we could not use forms that included considerations about the complexity of each question or time spent to answer (gerlich & berard, 2007; katz, 2002; warner, 2001; ryan, 2008). because our recording process would include different staff members in several branch libraries, we needed the form to be as simple as possible. the more variables the greater the chance of inaccurate recordings, which could lead to errors in the data. for this reason, we decided not to use any of the forms presented in the literature and instead developed a new recording form for this study (appendix).    from the literature review, we found inspiration for subject categories (lemire et al., 2016; lenkart & yu, 2017; radford & connaway, 2013; ryan, 2008), but we chose to use terminology that would fit into the context of a large norwegian university library and that our colleagues at the service desks would be familiar with. we decided to use the following categories: collection and access, counselling/guidance[2], citing and referencing, loan, practical questions[3], it questions, and other questions[4]. although the form had to be simple, like henry & neville (2008), we chose to include time-of-day as a parameter, as this would add valuable information without complicating the recording process. the time periods we used were: opening hours until 12:00p.m., from 12:00p.m. until 3:30p.m. and from 3:30p.m. until closing time.   the project members reviewed and tested the first draft of the recording form and the subject categories. we also carried out a pilot at a couple of branch libraries in advance of the first recording week. the form worked well, but more clarifying examples were needed under some of the categories. we pointed out that questions about the location of study rooms, auditoriums, cafés, and so on were to be recorded in the practical category. we clarified that the other category should be used for questions about exams, grades, and other similar topics.   we observed five hours’ worth of interactions between students and desk staff in four of the branch libraries during the first recording week. the observations served as a validation of the recording forms and confirmed that the forms worked well. therefore, no further adjustments were needed. eventually, we found a coincidence of types and distribution of questions in the observations and in the recording weeks.   student activities vary throughout the academic year. in some periods, they are busy with exam reading, while in others, they are writing assignments. to get a good distribution of the recording weeks throughout the academic year, the first recording took place in november 2017, the rest were conducted in 2018 (february, april, and september) so that these different phases could be compared.   the recording of questions was conducted by the desk staff in all 14 branches of our university library. we notified colleagues ahead of the recording weeks and gave clear instructions on how to record. we stated that only questions from students were to be recorded (including students from other universities). since our study dealt with physical meetings with students, questions by email or phone should not be counted. the libraries used one form per day and recorded all individual questions in the correct category. there could be several questions per inquiry and all questions were recorded. the students were notified of the recording activities by placards placed on the desks. questions were recorded in the predefined categories with one tally mark for each question. we used paper forms and collected the forms after each recording week. the total results were transferred into excel for further processing.         results   in this study, in-person questions from students at the library service desks were collected through four different weeks spread throughout the academic year from november 2017 to september 2018. during this period, a total of 9,683 questions were recorded in the different 14 branch libraries. table 1 shows questions per category recorded in all 14 libraries in percentage (number). the results show that the largest number of questions was related to the physical collections and revolved around what is found in the library room.     table 1 percentage (number) of questions per category     collection and access counselling/ guidance citing and referencing loan practical it other total november 2017  18% 5% 1% 41% 23% 9% 3% 100% (479) (125) (33) (1094) (600) (226) (80) (2637) february 2018  17% 8% 2% 50% 13% 6% 4% 100% (370) (180) (35) (1074) (274) (134) (85) (2152) april 2018  19% 11% 4% 45% 13% 6% 3% 100% (445) (269) (89) (1055) (296) (138) (69) (2361) september 2018  21% 7% 1% 40% 17% 10% 5% 100% (532) (189) (25) (1003) (421) (241) (122) (2533) total  19% 8% 2% 44% 16% 8% 4% 100% (1826) (763) (182) (4226) (1591) (739) (356) (9683)   divided into 14 libraries, a total of 9,683 questions corresponds to 35 questions per library per day.     the distribution of questions in the different categories was quite stable throughout the four weeks. loan was the category with the most questions, with 44% on average for the four weeks. this was followed by collection and access with 19%. figure 1 shows the distribution of questions in the categories for all four weeks in total.     the two categories with the greatest variation were counselling/guidance, with 5% in the recording week in november 2017 and 11% in april 2018. there were also more questions in the practical category in the two recording weeks in the fall (23% and 17%) than in the other two weeks (13% both).    we found no large variation in the type of questions on the different days of the week. figure 2 shows the distribution of questions according to days for all four weeks in total. the average shows slightly fewer questions on fridays (17%) and slightly more questions on wednesdays (22%) than on the other days.     in this study, we recorded questions in three different time periods each day and looked at how the questions were spread throughout the day. we did not find any large variation in the different time periods. the pattern showed a comparatively even number of questions between opening hours and 3:30p.m. and less from 3:30p.m. until closing. figure 3 shows the distribution of questions in the different time periods.    to maintain anonymity, we were prevented from breaking down results by library. however, we did not find a clear connection between the number of questions and the size of the branch libraries. some of the smalland medium-sized branch libraries had a relatively larger number of questions than the biggest libraries. measured in questions per library employee, it was also not the largest libraries that received the most questions. to give one example, one of the smallest branch libraries that cover health and social sciences had 439 questions per employee, while one of the largest libraries had 89. we will discuss possible explanations for this in the discussion section.      figure 1 questions per category in total for all four weeks.       figure 2 questions per day of the week.      figure 3 the spread of questions throughout the day.      discussion     our study shows that students ask about a lot more than just borrowing staples at the library desk. in fact, 44% of the questions were about loan services and document delivery; 19% were about physical and electronic collections and how to access to them; 16% were about practical things, such as opening hours, lost and found items, and the location of the group study rooms; 7% of the questions were about it. finally, the results show that 8% of the questions from the four weeks of counting were counselling and guidance questions and 2% were about literature lists, reference management, and reference management tools.     the recording of students´ questions was conducted by the desk staff in all 14 branch libraries at the university. in a previous study (kesselmann & watstein, 1987), it turned out that as many as 45% of questions were categorized incorrectly. we have no reason to believe that the percentage is that high in our study, but there are of course several possible sources of error: some may have misunderstood the content of the categories, forgotten to record questions, recorded inaccurate or double-recorded questions and so on. the fact that so many different staff members were involved in the recording is another possible source of error. we must, therefore, assume that there may be some errors in our data. since both the recording form itself and the recording process were thoroughly quality checked before we started the recording, nothing indicates that major systematic errors were made.     it is interesting to compare our study with other similar ones (le mire et al., 2016; lenkart & yu, 2017; ryan, 2008). admittedly, we cannot compare the studies directly, both because the categorization of types of questions varies somewhat, plus some of the studies, unlike ours, included email and chat transactions. nevertheless, we can still see some similarities. our findings show that 19% of the questions were in the collection and access category, while the corresponding percentage in le mire et al.’s study (2016) was 15% (category named locate materials). in lenkart and yu´s study (2017), 6.3% of the questions were inquiries for research assistance. the corresponding percentage in our study was 8%.   these findings are about similarities, but we also find differences. lenkart and yu (2017) found that 22.8% were questions about library materials, while our results showed 44%. ryan (2008) found that 12.4% of the questions were about technology (excel, logins, passwords, and networks) and in our study the corresponding number was 8%.   the comparison with international studies is interesting, but it is also worth comparing our findings with other norwegian studies due to a common cultural and organizational context. several university libraries in norway are currently running projects about the service desk, including a mapping of questions. unfortunately, little has been published so far. in an unpublished study from a project conducted in 2016, researchers at oslo metropolitan university found that 53% of the questions fell into the categories of procedure and collection. this corresponds roughly to the two categories in our study, collection and access and loans, for which the percentage is 63%. there is also a similarity between these two studies when it comes to the category counselling/guidance questions in our study and the similar category subject search in the unpublished study: 8% and 12.4%, respectively. so how can one interpret this similarity in results and the fact that the percentage of counselling/guidance questions seems to be relatively low? we do not have previous data on the number of counselling/guidance questions in norwegian academic libraries. however, from norwegian official government statistics, we do know that in recent years the volume of courses and individual guidance sessions by appointment has increased (statistisk sentralbyrå, 2019a). could it be that a great deal of the counselling/guidance questions are channelled through these services instead of the library desks? the question is whether this practice is optimal or whether one should, to a greater extent, use the desk as an educational tool in such a way that students are encouraged to also use the desk for counselling and guidance questions. maybe that would be a more resource-efficient way to utilize the library desk service. however, it is important to conduct more studies on these issues.      the two categories with the greatest variation between the recording weeks were counselling/guidance and practical. the counselling/guidance category makes sense because the students submit their bachelor’s and master’s theses in the spring and therefore will have more questions on this topic. there were also some more practical questions in the two recording weeks in the fall (september and november) than in the other two weeks. the reason for this is somewhat unclear but could be connected to the fact that new students usually have more practical questions in connection with the start of the school year.      we did not find any obvious connection between the size of the branch library and the number of questions asked at the desk. it was not the case that the larger the library, the more questions were asked. in fact, at one of the smallest branch libraries, 439 questions per employee were asked, while at one of the largest, there were 89 questions. in the study, we have not investigated the reasons for this, but we have some suggestions. we ask ourselves if a higher share of printed material generates more questions at the service desks? although the proportion of electronic literature is increasing, printed literature is still widely used in norwegian university libraries (statistisk sentralbyrå, 2019b). moreover, our own experience of working for several years in libraries within different subject areas, have shown us that there are differences in library use between disciplines. we have not found any norwegian research to back this, but we believe that the findings in our study reflect that assertion. another reason may be that some libraries have a stronger position among staff and students than others do and therefore are used more. this could be due to a greater proximity in smaller professional environments, making it easier to contact the library desk.    our recording form, with few parameters and the recording process itself, worked well for this study, particularly considering that so many employees were involved in the data gathering. the subject categories were general enough to make the question recording easy and at the same time they were well suited for our analysis. the recording method can be recommended for other libraries, especially big multi-branch libraries.   even if the recording form was simple, time-of-semester and time-of-day activity was admittedly taken care of. however, unlike other studies, we did not record subcategories (ryan, 2008) or time use (gerlich & berard, 2007). our findings showed a low percentage of counselling/guidance questions, 8%, but these questions are extra time-consuming. a possible follow-up study could use a recording system that includes the time aspect, for instance the read scale (gerlich & berard, 2007). other variables that can provide richer data material and can be considered in further research are questions in new areas, like open-access publishing, maker spaces, and so on (lemire et al., 2016). further research may also include user groups other than students and other communication channels besides the personal meeting at the desk.    another interesting point, which we did not address in our study, is that simple, practical questions can lead to other, more complex ones. once the dialogue between staff and student has been established in the personal meeting at the desk, it may be easier for the student to ask even more questions and more complex and time-consuming ones. one short question may reveal a deeper need for information. in our experience, this happens at our university library, but we do not know how often and how the mechanism works. on the other hand, we also did not investigate what kind of follow-up questions the library staff asked students who approached the desk. the lack of these perspectives is a limitation in our study but are well suited for subsequent studies. even though our study aimed to broaden the knowledge about one library, our own, this is a limitation and later studies should include other libraries as well.    what significance does our study have for further practice? the results are still discussed at the university library, but so far there have been no changes in the desk staffing. recording of questions asked at the desks will continue after the project period and will be carried out twice a year in the future. most important is that awareness of the various aspects of desk service seem to have increased. this awareness had led to more discussions about desk service issues in both formal and informal meetings. also, a forum has been established for presentation and discussion about desk related issues, like dissemination of the library’s electronic collections, access to special collections, and so on.   the purpose is to develop competence to guide users. our study will, in combination with other research results and a longer time of recording, provide the library with a better basis for further development of the desk service.     conclusion     in this study, we examined students’ use of the service desks in all branch libraries at a large university. during four different weeks throughout the academic year, we mapped desk activity. we sorted the questions asked in the personal meetings at the desk into predefined subject categories. most of the questions were about loan services, document delivery, and use of and access to physical and electronic collections (63%). in total, there was little variation between the categories from week to week, but we did find some differences. for example, there were more counselling/guidance questions in the assignment and exam period in the spring and more questions of a practical nature in the two autumn weeks.    results indicated that there was no obvious relationship between the branch library size and the number of questions asked. it seems that some study programs have a greater need for library services than others. this is one of the factors to consider when dimensioning and organizing the desk service in the future.     through this study, we have gained more knowledge about the purpose of students’ use of the service desk. these findings from a large norwegian multi-branch library is a unique contribution to the body of data that already exists internationally about the topic. in our study, we have developed a recording methodology, which we think is well suited for other large, multi-branch libraries. if more researchers use the same recording methodology, it will make it easier to compare findings between libraries and between countries in the future.   our newly gained knowledge has been and will continue to be used for further development of the desk service at our university. interesting themes for subsequent studies could be to investigate the consequences of different types of follow-up approaches for the fulfilment of student information needs. other ways to broaden our knowledge could be to include other user groups as well. finally, other communication channels other than the personal meeting at the desk would be of interest for future researchers.    references   gerlich, b. k., & berard, g. l. (2007). introducing the read scale: qualitative statistics for academic reference services. georgia library quarterly, 43(4), 7-13. https://digitalcommons.kennesaw.edu/cgi/viewcontent.cgi?article=1003&context=glq   henry, d. b., & neville, t. m. (2008). testing classification systems for reference questions. reference & user services quarterly, 47(4), 364-373. https://doi.org/10.5860/rusq.47n4.364   katz, b. (2002). introduction to reference work, vol. 2: reference services and reference processes (8th ed.). mcgraw-hill.    kesselmann, m. & watstein, s. (1987). the measurement of reference and information services. journal of academic librarianship, 13(1), 24–30.   lemire, s., rutledge, l., & brunvand, a. (2016). taking a fresh look: reviewing and classifying reference statistics for data-driven decision making. reference & user services quarterly, 55(3), 230–238. http://dx.doi.org/10.5860/rusq.55n3.230     lenkart, j., & yu, j. (2017). specialized reference services at illinois: reference transactional analysis and its implications for service providers and administrators. reference & user services quarterly, 56(4), 268–276. https://doi.org/10.5860/rusq.56.4.268   radford, m. l., & connaway, l. s. (2013). not dead yet! a longitudinal study of query type and ready reference accuracy in live chat and im reference. library & information science research, 35(1), 2–13. https://doi.org/10.1016/j.lisr.2012.08.001     ryan, s. m. (2008). reference transactions analysis: the cost-effectiveness of staffing a traditional academic reference desk. the journal of academic librarianship, 34(5), 389–399. https://doi.org/10.1016/j.acalib.2008.06.002   statistisk sentralbyrå. (2019a). 10872: opplæring i bibliotekbruk og primærmålgruppe i fag-og forskingsbibliotek, etter bibliotektype 2013 – 2019 [statistikk]. https://www.ssb.no/statbank/table/10872/   statistisk sentralbyrå. (2019b). 10870: 10870: bestand og tilvekst ved i fagog forskningsbibliotek, etter statistikkvariabel, bibliotektype og år [statistikk]. https://www.ssb.no/statbank/table/10870/   warner, d. g. (2001). a new classification for reference statistics. reference & user services quarterly, 41(1), 51–55. https://www.jstor.org/stable/41240900   appendix   the treff-project, registration of questions at the library desk, week xx (monday [date] – friday [date] [year])   library:                                                                                                                                        date: use one registration form per day. register each question in the right category (there may be several questions per inquiry). one tallymark per question. registrate questions only from students. do not register questions asked via email or telephone.   categories opening hours 12.00 12.00 15.30 15.30 – closing time collection and access (digital and printed collections). e.g.: do you have <title>, where do i find…, how do i get access…       counselling/guidance (reference questions) e.g.: do you have material about…, is this a scientific journal article, how to search, where to search…       citing and referencing e.g.: how to cite, make bibliographies, use reference tools (not technical questions about installing programs = it-questions)       loan (loan, library card, request) e.g.: loan/return, help with self-check, questions about due date, resource sharing, collecting requested material, claims, return receipt, logging into databases       practical questions (rooms/buildings/equipment) e.g.: where to find study rooms, cafes, auditoriums …, complaint about bad air, what are the opening hours, where to deliver lost property, borrow a stapler etc., where is the printer       it-questions e.g.: laptop printing, printing problems, internet access, questions about installation and use of software       other questions (register a tallymark and write down the question). e.g.: administrative matters         [1] treff means “meeting” in norwegian [2] questions about writing assignments, searching literature, etc. in-depth questions that usually requires more time to answer. [3] questions about group study rooms, opening hours, etc. [4] questions that did not fit into any of the other categories, e.g., questions about exams or administrative matters.   research article   first-year students and the framework: using topic modeling to analyze student understanding of the framework for information literacy for higher education   melissa harden first year experience librarian hesburgh libraries university of notre dame notre dame, indiana, united states of america email: mharden@nd.edu   received: 26 oct. 2018                                                                   accepted: 29 apr. 2019      2019 harden. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29514     abstract   objective – the framework for information literacy for higher education has generated a significant amount of discussion among academic librarians; however, few have discussed the potential impact on learning when students interact directly with the framework itself. at the university of notre dame, over 1,900 first-year students completed an information literacy assignment in their required first-year experience course. students read a condensed version of the framework, then wrote a response discussing how a frame of their choosing was reflected in an assigned reading. the goal of this exploratory study was to determine if the students demonstrated an understanding of the themes and concepts in the framework based on this assignment.   methods – topic modeling, a method for discovering topics contained in a corpus of text, was used to explore the themes that emerged in the students’ responses to this assignment and assess the degree to which they connect to frames in the framework. the model receives no information about the framework prior to the analysis; it only uses the students’ words to form topics.   results – the responses formed several topics that are recognizable as related to the frames from the framework, suggesting that students were able to engage effectively and meaningfully with the language of the framework. because the topic model does not know anything about the framework, the fact that the responses formed topics that are recognizable as frames suggests that students internalized the concepts in the framework well enough to express them in their own writing.   conclusion – this research provides insight regarding the impact that the framework may have on student understanding of information literacy concepts.     introduction   the framework for information literacy for higher education has generated a significant amount of discussion and interest since it was first introduced. librarians have shared ideas for lesson plans and collaboration with other campus partners to help students understand the concepts described in the framework. however, little is known about the direct role of the framework on student learning. in other words, what happens when students are presented with language from the framework and asked to respond to it? this article describes a study that explores the themes that emerge in student writing from an assignment based on the framework.   beginning with the 2015-2016 academic year, all first-year students at the university of notre dame, almost 2,000 in total, enroll in a two-semester first-year experience course. throughout the course, students engage with a number of important topics, including developing successful study habits, financial literacy, mental and physical health and wellbeing, and information literacy. during the fall 2016 semester, the students completed an information literacy assignment that was created by four librarians at the university. in this assignment, the students read an article about one researcher’s description of the troublesome and often challenging aspects of the research process (tompkins, 1986) and watched a ted talk about online filter bubbles and their effect on our understanding of truth and knowledge (pariser, 2011). additionally, the students read an adapted and condensed version of the framework for information literacy for higher education and wrote a short response of approximately 200 words to a prompt in which they were asked to make connections between the article they read and the framework (see appendix a for the full text of the prompt). the librarians who created the information literacy assignment for the course provided a modified shorter version of the framework for the students to read because they thought the original document contained too much jargon. while some of the language in the adapted version was simplified or reduced, the underlying concepts described in the framework mostly remained the same.   the responses to the assignment provide a wealth of information available for analysis to explore students’ understanding of the framework. however, because nearly 2,000 students completed the assignment, manually coding the themes in each written response would be an unwieldy, time-consuming task with many opportunities for human error. therefore, topic modeling was used to analyze the responses. topic modeling is a useful method in this case because it can provide information about the patterns in the text with minimal structure or guidance, and potential bias, from the researcher.   the goal of this exploratory study was to determine if the students demonstrated an understanding of the themes and concepts in the framework based on this assignment. the responses formed several topics that are recognizable as related to the frames from the framework. further, the frames that most commonly appeared in the topics generated by the model relate well to the content of the article they read, indicating meaningful interpretation of the assignment. because the topic model did not receive information about the framework itself, but rather formed topics based solely on students’ writing, this suggests that students internalized the concepts in the framework well enough to express them in their own writing. the remainder of this paper reviews past literature on the framework and topic modeling, describes the procedures used for data collection and cleaning, provides details on modeling the texts of student responses, and presents an analysis of the results.   literature review   the framework   the introduction of the framework marked a significant shift in the practice and discussion of library instruction. although there has been support for the framework since its inception, many expressed concerns about its potential value and structure. for example, demspey, dalal, dokus, charles, and scharf (2015) considered whether the framework would “receive the same kind of widespread recognition and endorsement as the standards” by national organizations and agencies because the document does not outline standardized learning outcomes (p. 165). wilkinson (2014) expressed concern about the use of threshold concepts in the framework, and beilin (2015) noted that threshold concepts “may end up functioning as the means to merely reinforce disciplinary boundaries and institutional hierarchies.” however, many librarians used the introduction of the framework to reflect on and reevaluate their teaching (burgess, 2015) and to explore how theory impacts their practice (holliday, 2017). specifically, the introduction of the framework provided an opportunity to focus on fostering critical thinking, drawing deeper connections (pagowsky, 2015), and creating opportunities for students to grapple with the complexity of information and its context (seeber, 2015). additionally, the new document provided grounds for talking with campus partners and establishing, or reestablishing, librarians’ place on campus as educators and collaborators (gibson & jacobson, 2014; lancaster, callender, & heinz, 2016; mcclure, 2016). a beneficial feature of the framework is its adaptability to meet individual campus needs. for example, witek (2016) describes making connections between her institution’s core curriculum and information literacy outcomes, and oakleaf (2014) offers strategies for developing assessments appropriate for the local context. in a specific example, swanson (2017) co-led a six-week course where disciplinary faculty engaged deeply with the framework and discussed how the frames could influence their teaching.   even though survey results indicate that there is variation in the degree to which librarians say the framework influences their instruction (julien, gross, & lathan, 2018), there are many examples of lessons and suggestions for incorporating concepts described in the framework in instruction, whether it be in one-shot sessions or full-semester courses (for example, bauder & rod, 2016; bravender, mcclure, & schaub, 2015; jacobson & gibson, 2015; mays, 2016). kuglitsch (2015) discusses how the threshold concept theory underpinning the framework can be used to situate information literacy in the disciplines.   there are several examples of the framework influencing engagement with first-year students. librarians at depaul university used the framework as a lens to analyze student responses to a library orientation activity (dempsey & jagman, 2016), while librarians at texas tech university collaborated with a composition instructor to redesign library activities for a first-year composition course, drawing heavily upon the framework (lancaster et al., 2016). as a part of the redesign, students were presented with the language of relevant frames from the framework within specific activities. even though students are not the intended audience for the document, they presented students with the frames in order to provide context for “the why behind the instruction and, more importantly, the why behind a productive research process” (lancaster et al., 2016, p. 90). however, the authors noted that because the framework is “highly theoretical” and the language is “too dense for most freshmen students,” they suggested providing a paraphrased version of the framework that is easier to understand to students in the future (p. 91).   in a study documented in two articles, rachel e. scott (2017a, 2017b) explored whether student engagement with framework language improved their understanding of information literacy concepts. to assess engagement, students enrolled in a credit-bearing course focused on research methods took a pre-test at the beginning of the semester (scott, 2017a). each of the frames in the framework was presented to the students, followed by two open-ended questions relating to each frame. student responses to the questions were analyzed to determine how first-year students “respond to the language and concepts of the framework” and whether or not the “concepts used in the frames fit in with undergraduates’ existing understanding of research practices” (scott, 2017a, p. 2). in the second article describing this project, scott (2017b) outlines the results of the post-test the students took at the end of the semester, after having engaged with the frames throughout the course. the same survey instrument was used in the post-test as was used in the pre-test. overall, the students in the course showed improvement in their responses from the pre-test to the post-test. the author noted that, “[s]everal responses indicate that students have internalized the frames and are not merely parroting the language provided in the questions” (scott, 2017b, p. 25). as a result, scott concludes that student interaction with the language of the framework had a positive effect on their understanding of information literacy concepts.   this paper builds on previous work by exploring the themes that emerge in actual student writing about the framework. topic modeling is used to identify and analyze the themes (i.e., topics) running through the text of students’ responses. this method has the advantage of allowing a wide range of potential topics to emerge based on how students reacted to the framework. it provides a useful complement to the researcher-directed forms of inquiry seen in past work, such as surveys.   topic modeling   probabilistic topic modeling is useful for analyzing large amounts of textual data. probabilistic topic models are “statistical methods that analyze the words of the original texts to discover the themes that run through them, how those themes are connected to each other, and how they change over time” (blei, 2012, pp. 77-78; see also blei, ng, & jordan, 2003). topic modeling allows for analyzing themes that emerge in a group of texts, collectively referred to as a corpus. topics are clusters of words that appear together frequently throughout the corpus, allowing researchers to identify the hidden structure of the texts. the topics that emerge in the analysis may not be apparent to a researcher making a manual observation of the corpus. this study uses latent dirichlet allocation (lda), a specific type of topic model, which “treats each document as a mixture of topics, and each topic as a mixture of words” (silge & robinson, 2017, p. 90). in the case of the present study, the topics that emerge from the corpus of student responses provide information about how students wrote about the framework and the way they saw its concepts reflected in an assigned reading. a key benefit of this method is that the model detects the topics and brings forth the underlying structure of the text archive, allowing for an exploratory analysis of the text while minimizing any effect a researcher’s preconceived ideas might have on the analysis.   there are several examples of the use of topic modeling to assess written student responses. for example, chen, yu, zhang, and yu (2016) analyzed the reflective writing in journal responses of pre-service teachers. this analysis allowed them to not only see how much of the course content the students incorporated into their reflections but also other themes that mattered to the students. southavilay, yacef, reimann, and calvo (2013) used topic modeling to create topic evolution charts to better understand how the topics in collaborative student writing changed over time when students used google docs to write together. kakkonen, myller, and sutinen (2006) and rahimi and litman (2016) describe employing lda to assist with automatic scoring of essays. because of the high enrollment of massive open online courses (moocs), topic modeling is useful for quickly analyzing course forums, a task that would be nearly impossible to do manually. several articles describe the application of topic modeling in these environments to assist with assessment of content in course forums (ramesh, goldwasser, huang, daumé iii, & getoor, 2014; reich, tingley, leder-luis, roberts, & stewart, 2014; vytasek, wise, & woloshen, 2017).   topic modeling is widely used to analyze texts in a variety of other fields, including political science (grimmer, 2010), robotics (girdhar, giguère, & dudek, 2014), and biology (zheng, mclean jr., & lu, 2006), but has been used narrowly in the study of libraries or library services. for example, topic modeling has been used to enhance findability and access to digital library collections (cain, 2016; hagedorn, kargela, noh, & newman, 2011; newman, hagedorn, & chemudugunta, 2007; mimno & mccallum, 2007).   aims   this paper describes an exploratory study designed to determine if students demonstrate an understanding of the themes and concepts outlined in the framework based on the assignment described above. additionally, analyzing the assignment presents an opportunity to identify and address weaknesses in the assignment in future iterations of the course, or in the design of similar assignments at other universities.   a key contribution of this paper is a demonstration of how topic modeling can be used by librarians to analyze student work as it relates to information literacy. past research describes how librarians are using the framework to facilitate student understanding of information literacy concepts. however, there is little scholarship discussing student engagement with the framework itself or the direct impact it has on student learning. this study provides an opportunity to explore on a large scale how students engage with concepts from the framework.   while this study is similar in many ways to the work done by scott (2017a, 2017b), mentioned earlier, there are a few key differences. first, the present study assesses just over 1,900 student responses (the whole first-year cohort) instead of a single class. second, the students in this study engaged with the concepts of the framework for one assignment in their first-year experience course rather than over the course of a whole semester. in other words, the focus is on a larger group of students at one point in time rather than a small group over the course of a semester. these two distinct approaches complement one another to improve our understanding of whether engaging with the framework improves students’ understanding of information literacy concepts.   methods   as a part of the assignment, the students read the article “‘indians’: textualism, morality, and the problem of history” by jane tompkins. in this article, the author describes her experience attempting to reconcile the descriptions of events related to interactions between native americans and early colonists in books and articles written by historians. after failing to identify a definitive truth in these conflicting secondary sources, the author turned to primary sources, hoping to find unbiased truth in these sources. after reading several primary source documents representing different perspectives of individuals, the author realized the challenge of identifying a singular, unbiased truth about these events. after reading this article and the abridged version of the framework, the students were asked to write a response selecting one frame and discussing how its themes are reflected in the tompkins article.   students entered their responses to the information literacy assignment into the campus learning management system. after receiving institutional review board (irb) approval for access, the responses for each section of the course were retrieved. all identifying information was removed from the files, and a unique, anonymous identification number was assigned to each response. while going through the responses to de-identify them, duplicate responses and responses to different assignments that had been erroneously uploaded to that week’s assignment were removed. additionally, several students did not complete the information literacy assignment, resulting in blank rows in the file. these lines were also removed. after all blank, duplicate, or erroneous responses were removed, 1,914 responses remained.   the assignment contained two questions to which the students responded (see the assignment prompt in appendix a). when students entered their responses into the learning management system, these answers were not separated into different fields, meaning each student’s response contained the answers to both questions. however, the responses to the first question, which asked about the framework, were the focus of this study. student answers to the second question were separated from answers to the first question, isolating the text to be analyzed. the resulting files were ingested and merged in the r statistical environment (r core team, 2018).    once loaded into r, further steps were taken to prepare the data for analysis, including removing english language stop words, punctuation, separators, and special characters, and rendering all words in lowercase. additionally, a lemmatization operator was run on the corpus. this process groups words based on a common root word, regardless of different inflectional endings. for example, the word “constructed” becomes “construct” in the corpus after lemmatization. similarly, “is,” “are,” and other forms of this verb become “be.” lemmatization removes “semantic duplicates,” allowing for better interpretation of the topic model results (risueño, 2017).   next, the sparsity threshold for the analysis was set. this number determines the threshold for removing uncommon terms from the corpus (benoit, 2018). words with values that are larger than the threshold are removed from the corpus. this retains words that allow for meaningful interpretation of students’ responses while removing words that are too specific to add substantive value. please see appendix c for more details on setting the sparsity threshold.   another necessary choice the researcher must make as an initial step is to select the number of topics to return in the model. because researchers can take different approaches in determining the optimal number of topics to request from the model, the r package called ldatuning was used as a guide in this decision (murzintcev, 2016). after analyzing the output of the ldatuning package, it was determined that 10 topics is the optimal number for this corpus. thus, the model identifies how words across the corpus cluster into 10 topics. please see appendix c for detailed information on the process taken to determine the optimal number of topics.   results   after these preliminary steps, the corpus was analyzed using lda. the top five words, or terms, associated with each of the 10 topics are found in table 1.     table 1 terms in each topic topic label terms topic 1 tompkins, indian, account, reflect, academic topic 2 idea, people, work, think, important topic 3 scholarly, process, knowledge, research, purpose topic 4 research, understand, topic, learn, good topic 5 first, authority, bias, construct, author topic 6 source, find, look, tompkins, subject topic 7 information, value, give, important, take topic 8 different, perspective, view, point, fact topic 9 american, tompkins, native, history, write topic 10 research, question, new, process, academic see appendix b for graphs of the probability of each term appearing in each topic.     discussion   analyzing the topics   each of the topics contains a set of terms that are frequently clustered near each other throughout the student responses. the combinations of terms in each topic provide insight into which frames students wrote about and how students described various frames. this is the basis of the analysis of student understanding of the framework in this study.   the ordering of the topics listed in the table is random (e.g., topic 1 is not necessarily more important than topic 10). the terms listed within each topic appear in the order of greatest to least probability of each term being generated from that topic. in the analysis that follows, the topics that appear to be most closely related to the framework are discussed first. analysis includes a discussion of the terms in the topic, along with an example student response showing some of the terms in context.   the terms in topic 5 (“first,” “authority,” “bias,” “construct,” and “author”) are terms that are related to the description of the frame “authority is constructed and contextual.” the presence of this frame as one of the topics means that many students wrote about it in their responses. the article the students read touches on several themes described in this frame, including the importance of considering the authority of various sources and seeking different perspectives in the research process. the presence of this frame as a topic suggests that the students are able to successfully recognize and identify its key concepts reflected in a document other than the framework. for example, one student wrote in their response, “one aspect that stands out in tompkins’ article is her exploration of multiple sources and, especially, the bias found in each. this connects to the first frame [sic] which describes that authority is both constructed and contextual.” in this example response, the student interprets the frame “authority is constructed and contextual” to be about examining multiple sources and their potential bias. because the term “bias” appears near the terms “authority” and “construct” in the corpus (and are thus found together in topic 5), it is apparent that many students referred to bias when discussing this frame. student understanding of the social construction of authority, and thus the need to assess a source’s credibility, can be inferred from their use of the term “bias” in this context.   the terms in topic 8 are also related to the frame “authority is constructed and contextual.” the description of this frame in the student version of the framework contains the sentence “keep an open mind to varied and even conflicting perspectives.” topic 8 contains the terms “different,” “perspective,” “view,” and “point,” indicating that many students wrote about seeking different perspectives or points of view during the research process. given the content of tompkins’ article, it comes as no surprise that many students wrote about this practice. the following student response provides an example:   the first frame in “the framework for information literacy for higher education” [authority is constructed and contextual] parallels with tompkins’ account of her research project when tompkins initially mentions the conflicting arguments of multiple sources. she discovers that there is controversy because the points of views of the observer differ…for example, tompkins notes that miller describes america as “vacant”. there were indians on the land when they arrived, and miller would be mistaken if he truly believed there was no one there; however, tompkins explains the reasoning behind miller’s word choice. miller simply did not deem the indians as important enough to be noted. . .. thus, when constructing research, multiple perspectives must be accounted for, not just the observer’s.    the following excerpt from a student response also reflects how topic 5 and topic 8 are related:   i think the first frame that states that “authority is constructed and contextual” relates well to tompkins essay. she writes about how the accounts of relations between indians and early immigrants in new england are influenced by the person writing them and the period of time and social construct the author comes from. also, many of these perspectives conflict with each other and some are even irreconcilable.   in both of these example responses, the students are able to identify how an individual’s account of an event is influenced by their place and time in society. additionally, they both reference the importance of seeking out multiple perspectives rather than relying on only one interpretation of an event. these responses exemplify how students connected the concepts in the frame “authority is constructed and contextual” with the importance of recognizing bias and seeking out multiple perspectives in order to create as complete of an understanding around a topic as possible.   interestingly, other students discussed the importance of seeking multiple perspectives on a topic or issue in their responses when discussing other frames. for example, one student wrote about the importance of reviewing multiple sources while discussing how the frame “research as inquiry” is reflected in the tompkins article.   jane tompkins’ account of her academic research process reflects the fourth frame, “research is inquiry.” her first exposure to works of historians and secondary sources caused her to question the contradicting accounts with points of view so opposing that she was unable to reconcile them in any way. she then mentions her turning towards primary sources where she found the same problem of conflicting viewpoints. these experiences caused her to view research as an iterative and collaborative exploration rather than a means to find a satisfying and definitive end result…her understanding that “research is inquiry” helped her to not only accept unanswered questions, but also to formulate new questions in response to information she acquires.   as in the previous example response, this student picked up on the importance of weighing different perspectives in the research process. while these student responses discuss a similar theme (the importance of seeking out multiple perspectives on a topic) using the same terms, they are doing so while referring to different frames. this reflects the interrelatedness of the frames. as the framework states, it is a “cluster of interconnected core concepts” (association of college & research libraries, 2015, “introduction”). these responses highlight how students are discussing similar themes using some of the same keywords but referring to different frames.   in the modified version of the framework provided to the students, the frame “information creation as a process” was reworded as “knowledge is both a process and a product.” two of the terms in topic 3 (“knowledge” and “process”) relate to this frame. some language in the descriptive text of this frame was also modified. therefore a few of the other terms that appear in topic 3, such as “research,” “scholarly,” and “learn” are words that appear in the description of the student version of the frame but are not part of the wording of this frame in the original framework document. while these keywords do not necessarily reflect main points of the original wording of the frame “information creation as a process,” the students were able to successfully respond to the version of the frame that they read. for example, one student wrote,   one of the frames from the framework was the “knowledge as both a process and a product”. this frame reflects the idea that in pursuit of research, the process of trying to find answers will create more questions. this is very evident in tompkins’ account of her own academic research process because tompkins had dwelled into a world that she had thought she had known a lot about, but instead opened a whole new world as she gained so much new information in her research.   the theme of this example student response better matches the general theme of the frame “research as inquiry” in the original framework. in analyzing student responses, it became apparent that there were many similarities between the descriptions of the frames “knowledge is both a process and a product” and “research as inquiry” in the adapted student version of the framework. based on this analysis, in future iterations of this assignment, it is recommended that the adapted version of the frame “information creation as a process” be reviewed and updated to ensure fidelity between the original wording and the adapted version.   the terms in topic 7 (“information,” “value,” “give,” “important,” “take”) suggest many students wrote about the frame “information has value” as it relates to tompkins’ article. the following student response is an example of the application of this frame to the reading.   one frame discussed within the reading, “frameworks [sic] for information for literacy for higher education,” focuses on the idea that information has value. in other words, information can be used in many different avenues to raise voices and induce change. this skill can be done by not only respecting the original merit of other's information, but also by giving yourself credit for being a contributor to the outflow of information, not just a “bystander.” within tompkins’ account of her academic research progress, she discusses how a sentence in one of perry miller’s writings stopped her dead. this sentence discussed the vacancy of the american wilderness prior to european civilization…i think tompkins’ shock from reading this sentence is a good example of the framework on the value of information because it shows how even though we don’t agree with someone, we should still acknowledge the information of others and learn from its value.   this example response refers to two different aspects of the value of information described in the framework. first, the student acknowledges that information is valuable in that it can be used to bring about change and to “raise voices.” additionally, the student refers to the value of citation and giving credit to the original work used in research.   topic 1, topic 4, and topic 9 all contain terms related to the content of the tompkins article but not necessarily to the framework. because the students were asked to describe connections between one frame and the content of the article, it is no surprise that topics would surface containing terms related to the description of tompkins’ research experience outlined in the assigned article. many of the terms in topic 10 are terms that also appear in the original assignment prompt. therefore, this topic likely arises from students restating the question in their responses, thus those terms would be clustered together frequently. finally, topic 2 and topic 6 do not collectively contain any keywords that are thematically important to the corpus. while the goal is to minimize the number of topics with little meaning, it is not uncommon for a few to remain.     figure 1 proportion of responses discussing each topic     topic popularity   the analysis presented to this point has been focused on identifying and understanding the topics themselves. however, the lda model also assigns each document, or each student response in the case of the present study, with topic probabilities, which are estimates of the “proportion of words from that document that are generated from that topic” (silge & robinson, 2017, p. 95). the higher a given topic probability, the more likely a document discusses that topic. these probabilities were used to assess the relative popularity of each topic among students’ responses. for each response, the topic(s) associated with the highest topic probability were identified. then, for each of the 10 topics, the proportion of responses in the corpus for which that topic produced the highest probability was computed. figure 1 presents these proportions on the y-axis for each of the 10 topics on the x-axis.   this graph shows that, on average, topic 5 and topic 3 were the most popular among student responses. topic 5 and topic 3 both contain terms related to frames in the framework. further, these topics contain terms that are connected to the frames “authority is constructed and contextual” and “knowledge is both a process and a product” (“information creation as a process” in the original framework), whose themes are present in the assigned reading by tompkins. the intent of the assignment was for students to engage with the framework and describe how it applies to the article by tompkins. this analysis suggests that, on average, the students accomplished this objective.   limitations   this exploratory analysis has a few limitations. the assignment analyzed in the present study was administered at one university. therefore, the results may not be generalizable. additionally, this analysis looked at one assignment involving direct engagement with the framework. it does not assess the transfer of knowledge of the skills and concepts described in the framework to other situations. this analysis also captures only one point in time. as a result, it does not provide information about any potential growth in student understanding or skills related to information literacy. in the adapted version of the framework that the students read, the frame “information creation as a process” was reworded as “knowledge is both a process and a product.” when this frame was adapted for the assignment, the way in which the language in the descriptive text was modified made it similar to the descriptive text of another frame, “research as inquiry.” the similarity between the two frames in the student version may have had an effect on student understanding of the frames and the subsequent analysis. as noted previously, there were some concerns about the framework at the time it was introduced, and students are not the intended audience of the document. thus, this method is not the only way to assess student understanding of information literacy concepts.   conclusion   in this exploratory study, topic modeling was used to analyze the text of first-year undergraduates’ responses to an assignment on the framework for information literacy for higher education. topic modeling, as it was employed here, is an unsupervised, unstructured method which imposes few assumptions about what the researcher is going to find ahead of time. three frames are easily recognizable in the topics. based on the topics generated from the topic model and examining sample student responses, the analysis suggests that these first-year students made meaningful connections between the concepts described in the framework and the assigned reading. much like the results found by scott (2017b), students were not simply interacting with the language of the framework in a perfunctory manner. rather, as seen in example responses, students made meaningful connections between the assigned reading and the frame about which they chose to write. thus, there appears to be benefit in asking students to apply the concepts in the framework to an additional reading. overall, this research provides valuable information for evaluating the assignment’s effectiveness locally, as well as general insight regarding student understanding of the framework.   references   alsumait, l., barbará, d., gentle, j., & domeniconi, c. (2009). topic significance ranking of lda generative models. in w. buntine, m. grobelnik, d. mladenić, & j. shawe-taylor (eds.). machine learning and knowledge discovery in databases. ecml pkdd 2009. lecture notes in computer science, vol. 5781. berlin: springer. https://doi.org/10.1007/978-3-642-04180-8_22   arun, r., suresh, v., veni madhavan, c. e., & narasimha murthy, m. n. (2010). on finding the natural number of topics with latent dirichlet allocation: some observations. in m. j. zaki, j. x. yu, b. ravindran, & v. pudi (eds.) advances in knowledge discovery and data mining. pakdd 2010. lecture notes in computer science, vol. 6118. berlin: springer. https://doi.org/10.1007/978-3-642-13657-3_43   association of college & research libraries. (2015). framework for information literacy for higher education. retrieved from http://www.ala.org/acrl/standards/ilframework   bauder, j., & rod, c. (2016). crossing thresholds: critical information literacy pedagogy and the acrl framework. college & undergraduate libraries, 23(3), 252-264. https://doi.org/10.1080/10691316.2015.1025323   beilin, i. (2015, feb. 25). beyond the threshold: conformity, resistance, and the acrl information literacy framework for higher education. in the library with the lead pipe. retrieved from http://www.inthelibrarywiththeleadpipe.org/2015/beyond-the-threshold-conformity-resistance-and-the-aclr-information-literacy-framework-for-higher-education/   benoit, k. (2018). quanteda: quantitative analysis of textual data. r package version 0.99.22. retrieved from http://quanteda.io   blei, d. m., ng, a. y., & jordan, m. i. (2003). latent dirichlet allocation. journal of machine learning research, 3(4/5), 993-1022.   blei, d. m. (2012). probabilistic topic models. communications of the acm, 55(4), 77-84. https://doi.org/10.1145/2133806.2133826   bravender, p., mcclure, h. a., & schaub, g. (2015). teaching information literacy threshold concepts: lesson plans for librarians. chicago, il: association of college and research libraries.   burgess, c. (2015). teaching students, not standards: threshold crossings for students and instructors alike. partnership: the canadian journal of library and information practice and research, 10(1). https://doi.org/10.21083/partnership.v10i1.3440     cain, j. o. (2016). using topic modeling to enhance access to library digital collections. journal of web librarianship, 10(3), 210-225. https://doi.org/10.1080/19322909.2016.1193455   cao, j., xia, t., li, j., zhang, y., & tang, s. (2009). a density-based method for adaptive lda model selection. neurocomputing, 72(7-9), 1775–1781. https://doi.org/10.1016/j.neucom.2008.06.011   chen, y., yu, b., zhang, x., & yu, y. (2016). topic modeling for evaluating students' reflective writing: a case study of pre-service teachers' journals. in proceedings of the sixth international conference on learning analytics & knowledge (pp. 1-5). new york, ny: acm. https://doi.org/10.1145/2883851.2883951   dempsey, m. e., dalal, h., dokus, l. r., charles, l. h., & scharf, d. (2015). continuing the conversation: questions about the framework. communications in information literacy, 9(2), 164-175. https://doi.org/10.15760/comminfolit.2015.9.2.193   dempsey, p. r., & jagman, h. (2016). “i felt like such a freshman”: first-year students crossing the library threshold. portal: libraries and the academy, 16(1), 89-107. https://doi.org/10.1353/pla.2016.0011   deveaud, r., sanjuan, e., & bellot, p. (2014). accurate and effective latent concept modeling for ad hoc information retrieval. document numérique, 17(1), 61–84. https://doi.org/10.3166/dn.17.1.61-84   gibson, c., & jacobson, t. e. (2014). informing and extending the draft acrl information literacy framework for higher education: an overview and avenues for research. college & research libraries, 75(3), 250-254. https://doi.org/10.5860/0750250     girdhar, y., giguère, p., & dudek, g. (2014). autonomous adaptive exploration using realtime online spatiotemporal topic modeling. the international journal of robotics research, 33(4), 645-657. https://doi.org/10.1177/0278364913507325   griffiths, t. l., & steyvers, m. (2004). finding scientific topics. proceedings of the national academy of sciences, 101(suppl. 1), 5228-5235. https://doi.org/10.1073/pnas.0307752101   grimmer, j. (2010). a bayesian hierarchical topic model for political texts: measuring expressed agendas in senate press releases. political analysis, 18(1), 1-35. https://doi.org/10.1093/pan/mpp034   hagedorn, k., kargela, m., noh, y., & newman, d. (2011). a new way to find: testing the use of clustering topics in digital libraries. d-lib magazine, 17(9/10). https://doi.org/10.1045/september2011-hagedorn   holliday, w. (2017). frame works: using metaphor in theory and practice in information literacy. communications in information literacy, 11(1), 4-20. https://doi.org/10.15760/comminfolit.2017.11.1.44   jacobson, t. e., & gibson, c. (2015). first thoughts on implementing the framework for information literacy. communications in information literacy, 9(2), 102-110. https://doi.org/10.15760/comminfolit.2015.9.2.187   julien, h., gross, m., & latham, d. (2018). survey of information literacy instructional practices in u.s. academic libraries. college & research libraries, 79(2), 179. https://doi.org/10.5860/crl.79.2.179    kakkonen, t., myller, n., & sutinen, e. (2006). applying latent dirichlet allocation to automatic essay grading. in t. salakoski, f. ginter, s. pyysalo, & t. pahikkala (eds.) advances in natural language processing. lecture notes in computer science, vol. 4139. berlin: springer. https://doi.org/10.1007/11816508_13   kuglitsch, r. z. (2015). teaching for transfer: reconciling the framework with disciplinary information literacy. portal: libraries and the academy, 15(3), 457-470. https://doi.org/10.1353/pla.2015.0040   lancaster, a., callender, d., & heinz, l. (2016). bridging the gap: new thresholds and opportunities for collaboration. in r. mcclure (ed.), rewired: research-writing partnerships within the frameworks (pp. 85-101). chicago: association of college and research libraries.   mays, d. a. (2016). using acrl’s framework to support the evolving needs of today’s college students. college & undergraduate libraries, 23(4), 353-362. https://doi.org/10.1080/10691316.2015.1068720   mcclure, r. (ed.). (2016). rewired: research-writing partnerships within the frameworks. chicago, il: association of college and research libraries.   mimno, d., & mccallum, a. (2007, june). organizing the oca: learning faceted subjects from a library of digital books. in proceedings of the 7th acm/ieee-cs joint conference on digital libraries (pp. 376-385). new york, ny: acm. https://doi.org/10.1145/1255175.1255249   murzintcev, n. (2016). ldatuning: tuning of the latent dirichlet allocation models parameters. r package version 0.2.0. https://cran.r-project.org/package=ldatuning newman, d., hagedorn, k., chemudugunta, c., & smyth, p. (2007, june). subject metadata enrichment using statistical topic models. in proceedings of the 7th acm/ieee-cs joint conference on digital libraries (pp. 366-375). new york, ny: acm. https://doi.org/10.1145/1255175.1255248   oakleaf, m. (2014). a roadmap for assessing student learning using the new framework for information literacy for higher education. the journal of academic librarianship, 40(5), 510-514. https://doi.org/10.1016/j.acalib.2014.08.001   pagowsky, n. (2015). a pedagogy of inquiry. communications in information literacy, 9(2), 136-144. https://doi.org/10.15760/comminfolit.2015.9.2.190   rahimi, z., & litman, d. (2016). automatically extracting topical components for a response-to-text writing assessment. in proceedings of the 11th workshop on innovative use of nlp for building educational applications (pp. 277–282). stroudsburg, pa: association for computational linguistics. https://doi.org/10.18653/v1/w16-0532    ramesh, a., goldwasser, d., huang, b., daumé iii, h., & getoor, l. (2014). understanding mooc discussion forums using seeded lda. in proceedings of the ninth workshop on innovative use of nlp for building educational applications (pp 28-33). stroudsburg, pa: association for computational linguistics. https://doi.org/10.3115/v1/w14-1804   reich, j., tingley, d. h., leder-luis, j., roberts, m., & stewart, b. (2014). computer-assisted reading and discovery for student generated text in massive open online courses. harvardx working paper series number 6. https://doi.org/10.2139/ssrn.2499725    r core team (2018). r: a language and environment for statistical computing. vienna, austria: the r project for statistical computing. https://www.r-project.org/.   risueño, t. (2017, june 15). lemmatization to enhance topic modeling results. in bitext. retrieved from https://blog.bitext.com/lemmatization-to-enhance-topic-modeling-results   scott, r. e. (2017). part 1. if we frame it, they will respond: undergraduate student responses to the framework for information literacy for higher education. the reference librarian, 58(1), 1-18. https://doi.org/10.1080/02763877.2016.1196470   scott, r. e. (2017). part 2. if we frame it, they will respond: student responses to the framework for information literacy for higher education. the reference librarian, 58(1), 19-32. https://doi.org/10.1080/02763877.2016.1196471   seeber, k. p. (2015). this is really happening: criticality and discussions of context in acrl’s framework for information literacy. communications in information literacy, 9(2), 157-163. https://doi.org/10.15760/comminfolit.2015.9.2.192     silge, j., & robinson, d. (2017). text mining with r: a tidy approach. sebastopol, ca: o’reilly.   southavilay, v., yacef, k., reimann, p., & calvo, r. a. (2013). analysis of collaborative writing processes using revision maps and probabilistic topic models. in proceedings of the third international conference on learning analytics and knowledge (pp. 38-47). new york, ny: acm. https://doi.org/10.1145/2460296.2460307   swanson, t. (2017). sharing the acrl framework with faculty: opening campus conversations. college & research libraries news, 78(1), 12-48. https://doi.org/10.5860/crln.78.1.9600   ted talk: pariser, e. (2011, march). eli pariser: beware online “filter bubbles” [video file]. retrieved from https://www.ted.com/talks/eli_pariser_beware_online_filter_bubbles   tompkins, j. (1986). “indians”: textualism, morality, and the problem of history. critical inquiry, 13(1), 101-119. https://doi.org/10.1086/448376   vytasek, j. m., wise, a. f., & woloshen, s. (2017). topic models to support instructors in mooc forums. in proceedings of the seventh international learning analytics & knowledge conference (pp. 610-611). new york, ny: acm. https://doi.org/10.1145/3027385.3029486   wilkinson, l. (2014, june 19). the problem with threshold concepts [blog post]. retrieved from https://senseandreference.wordpress.com/2014/06/19/the-problem-with-threshold-concepts/   witek, d. (2016). becoming gardeners: seeding local curricula with the acrl framework for information literacy. college & research libraries news, 77(10), 504-508. https://doi.org/10.5860/crln.77.10.9572   zheng, b., mclean jr., d. c., & lu, x. (2006). identifying biological concepts from a protein-related corpus with a probabilistic topic model. bmc bioinformatics, 7(58). https://doi.org/10.1186/1471-2105-7-58     appendix a information literacy assignment   watch: beware online “filter” bubbles, ted talk, by eli pariser read: “indians”: textualism, morality, and the problem of history, article, by jane tompkins read: the framework for information literacy for higher education, document, literacy framework (association for college and research libraries)   in your reading for this week, we ask you to reflect carefully on the purpose, value, and process of scholarly research. first, choose at least one of the six frames from “the framework for information literacy for higher education” and discuss how that frame is reflected in tompkins’ account of her own academic research process. you might, for example, describe how tompkins’ back-and-forth journey between primary, secondary, and tertiary sources reflects the truth of the second frame (“knowledge is both a process and a product”) in which knowledge must be understood as “a process of discovery rather than mere reaffirmation of prior held beliefs.” second, consider your own nascent student career here at notre dame. what controversial issue might you choose to explore? what might you expect to discover? how might you negotiate any contradictions you find?   author note: when the student version of the framework was adapted from the original, several modifications were made to the frame “information creation as a process.” first, the title of the frame was changed to “knowledge is both a process and a product.” additionally, the modified description of this frame contains many similarities to the description of the frame “research as inquiry.” the titles of the other frames remained the same as in the original. similarly, while the descriptions of the other frames were shortened, the language was not changed significantly.     appendix b probability of each term belonging to topics       appendix c methodological details   sparsity threshold   the sparsity of a particular term is defined as 1 where f is the number of times the word appears in the corpus and c is the number of documents in the corpus (benoit, 2018). for instance, the sparsity of a term that appears 15 times in a corpus of 100 documents is 1 = 0.85.   selecting the number of topics   the r package called ldatuning was used as a guide in deciding how many topics to have the model return (murzintcev, 2016). this package applies four different methods for determining an appropriate number of topics for lda in a given corpus and presents the results in one graph for comparison (see figure c1). the four methods are described in griffiths and steyvers (2004), cao, xia, li, zhang, and tang (2009), arun, suresh, veni madhavan, and narasimha murthy (2010), and deveaud, sanjuan, and bellot (2014). in the models developed by cao et al. and arun et al., the number of topics closest to zero on the y-axis are considered optimal for a given corpus, whereas in the models developed by griffiths and steyvers and deveaud et al., the number of topics that are closest to 1.00 are considered optimal.   figure c1 topic selection measures.   the point on the graph in figure c1 where the four measures show the most convergence indicates 12 is the optimal number of topics. however, this choice produced several substantive topics and a few topics that contained common words across documents and seemed to be meaningless. it is not unusual to have a few topics that are thematically unimportant; however, the goal is to find as many meaningful topics as possible (alsumait, barbará, gentle, & domeniconi, 2009). running the model with 10 topics produced more meaningful topics and fewer that were meaningless, indicating that 10 is the optimal number of topics for this corpus. evidence summary   web search engines not yet a reliable replacement for bibliographic databases   a review of: bates, j., best, p., mcquilkin, j., & taylor, b. (2017) will web search engines replace bibliographic databases in the systematic identification of research? the journal of academic librarianship, 43(1), 8-17. https://doi.org/10.1016/j.acalib.2016.11.003   reviewed by: emma hughes freelance information professional norwich, united kingdom email: emma.e.hughes@outlook.com   received: 29 nov. 2017 accepted: 29 apr. 2018      2018 hughes. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29378     abstract   objective to explore whether web search engines could replace bibliographic databases in retrieving research.   design systematic review.   setting english language articles in health and social care; comparing bibliographic databases and web search engines for retrieving research published between january 2005 and august 2015, in peer-reviewed journals and available in full-text.   subjects eight bibliographic databases: assia (applied social sciences index and abstracts), cinahl plus (cumulative index to nursing and allied health literature), lisa (library and information science abstracts), medline, psycinfo, scopus, ssa (social services abstracts), and ssci (social sciences citation index) and five web search engines: ask, bing, google, google scholar, yahoo.   methods a literature search via the above bibliographic databases and web search engines. the retrieved results were independently appraised by two researchers, using a combination of tools and checklists, including the press checklist (mcgowan et al., 2016) and took guidance on developing search strategies from the centre for reviews and dissemination (2009).   main results sixteen papers met the appraisal requirements. each paper compared at least one bibliographic database against one web-search engine. the authors also discuss findings from their own search process. precision and sensitivity scores from each paper were compared. the results highlighted that web search engines do not necessarily use boolean logic and in general have limited functionality compared to bibliographic databases. there were variances in the way precision scores were calculated between papers, but when based on the first 100 results, web search engines were similar to some databases. however, their sensitivity scores were much weaker.   conclusion whilst precision scores were strong for web search engines, sensitivity was lacking; therefore web search engines cannot be seen as a replacement for bibliographic databases at this time. the authors recommend improving the quality of reporting in studies regarding literature searching in academia in order for reliable comparisons to be made.   commentary   due to the deluge of research information on the internet, web search engines could be seen as a viable, free alternative to searching bibliographic databases. this paper was reviewed using the amstar 2; a critical appraisal tool for systematic reviews (shea et al., 2017).   the systematic review’s methods and search strategy were clearly explained and provided, giving the research strong validity. unlike the studies included in the review, the authors performed their search via more than one web search engine and provided clear reasons for the search engine choices. in the methodology the authors state that the review followed the prisma guidelines (moher et al., 2009), however the authors have not included a figure or list of the excluded articles, as the primsa flow diagram would suggest including.   all findings are provided, including those of the authors’ search in both bibliographic databases and web search engines. limitations are discussed, including the age of some papers retrieved and a discrepancy in the ways different papers define “precision”. one paper used a scoring system (tober, 2011) based on their definition of recall, precision and importance, whilst others calculated precision from a selected number of hits (first 10, 50, or 100 hits) rather than the total number retrieved. this meant the authors struggled to analyse precision scores. the findings of this paper are consistent with previous studies in suggesting that web search engines, in particular google scholar, could be used in conjunction with bibliographic databases when searching for information. interestingly, two papers suggested that google scholar could offer better precision scores than some bibliographic databases (mcfadden et al., 2012; walters, 2009). however, web search engines should not at this stage be used as a reliable replacement.   the authors did not explore if there were relevant results retrieved from web search engines that were not found in the bibliographic databases. this may help to determine the value of web search engines for contributing unique evidence, that otherwise might not be identified in traditional systematic review searches.   grey literature (for example, unpublished reports or conference abstracts) is a form of evidence often required in social care research (ford & korjonen, 2012), but often not indexed in bibliographic databases. therefore, it would be interesting to see this study replicated to explore searches for different forms of evidence.   this paper highlights the need for a consistent definition of precision to assist academics comparing studies in future research. overall this paper adds to the growing body of research exploring the potential of web search engines for retrieving empirical research, so it is useful for librarians deciding whether to incorporate web search engines into their teaching.   references   centre for reviews and dissemination. (2009). systematic reviews: crd's guidance for undertaking reviews in health care. retrieved from: https://www.york.ac.uk/media/crd/systematic_reviews.pdf   ford, j., & korjonen, h. (2012). information needs of public health practitioners: a review of the literature. health information and libraries journal, 29(4), 260-273. https://doi.org/10.1111/hir.12001   mcfadden, p., taylor, b., campbell, a., & mcquilkin, j. (2012). systematically identifying relevant research: case study on child protection social workers’ resilience. research on social work practice, 22(6), 626-636. https://doi.org/10.1177/1049731512453209   mcgowan, j., sampson, m., salzwedel, d., cogo, e., foerster, v., & lefebvre, c. (2016).      press peer review of electronic search strategies: 2015 guideline statement. journal of clinical epidemiology, 75(july), 40-46. https://doi.org/10.1016/j.jclinepi.2016.01.021   moher, d., liberati, a., tetzlaff, j., altman, d. g., & the prisma group (2009). preferred reporting items for systematic reviews and meta-analyses: the prisma statement. plos medicine, 6(7). https://doi.org/10.1371/journal.pmed.1000097   shea, b. j., reeves, b. c., wells, g., thuku, m., hamel, c., moran, j., moher, d., tugwell, p., welch, v., kristjansson, e., & henry, d. a. (2017). amstar 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. british medical journal, 358(j4008),1-9. https://doi.org/10.1136/bmj.j4008   tober, m. (2011). pubmed, sciencedirect, scopus or google scholar: which is the best search engine for an effective literature research in laser medicine? medical laser application, 26(3), 139144. https://doi.org/10.1016/j.mla.2011.05.006   walters, w.h. (2009). google scholar search performance: comparative recall and precision. libraries and the academy, 9(1), 5-24. https://doi.org/10.1353/pla.0.0034 evidence based library and information practice evidence based library and information practice 2009, 4:3  evidence based library and information practice       evidence summary    reference services in australian academic libraries are becoming more multifaceted     a review of:  burke, l. “models of reference services in australian academic libraries.” journal of librarianship  and information science 40.4 (2008): 269‐86.    reviewed by:  lotta haglund  head of information and public relations, karolinska institutet, university library  stockholm, sweden  email: lotta.haglund@kib.ki.se     david herron  scholarly developer, karolinska institutet, university library  stockholm, sweden  email: david.herron@ki.se     received: 02 march 2009        accepted: 05 june 2009       © 2009 haglund and herron.  this is an open access article distributed under the terms of the creative  commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.     abstract    objective – to investigate the current  organizational models for reference work in  australian academic libraries, and how these  reference services are staffed.    design – mixed methods.    setting – academic universities in australia.     subjects – forty council of australian  university librarians (caul) member  libraries.    methods – a literature study was undertaken  to (1) find a definition of reference services  and (2) explore the development of reference  service models over time. statistics from the  caul member libraries were studied for  trends in student population and number of  academic and library staff. a web‐based  survey, with questions based on the findings  in the literature study, was then distributed to  the 40 australian university libraries in 2006.     respondents were asked when the library  commenced different reference services in five  areas: formats in which the library received  and responded to reference queries,  information literacy, subject specialization,  liaison activities, and collection development.  respondents also answered questions about  the organization of the reference department,  56 mailto:lotta.haglund@kib.ki.se mailto:david.herron@ki.se http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2009, 4:3  including: whether they had a separate or  integrated model; the size of the reference  collections; if they had a librarian dedicated to  supporting students studying in remote or  distant mode; if the interlibrary loans  department was part of the suite of reference  services; and if they had a mission or  statement of purpose for their reference  services department.    main results – based on the literature study,  the working definition of reference services (1)  for the project was “all activities which assist  in providing relevant and appropriate  information services to patrons” (270),  including:  • all interactions with patrons to assist  them in their searches for information  in all media types.  • all training by librarians of patrons to  be able to access information for  themselves.  • activities to help the library stay  informed of relevant developments,  such as establishing and maintaining  relationships with patrons.    the literature study also revealed (2) a shift  from the traditional reference service model,  focused on the reference desk and the services  delivered from that location, to new models  involving “consolidation of reference service  points, establishment of tiered reference,  reference by appointment, reorganization of  reference departments, and limiting services to  primary users” (271).   the core aspects of reference services have  changed little over time, including face‐to‐face  reference work, print collection development,  bibliographic instruction, and attending  meetings. in some aspects, however, there has  been a shift in emphasis, e.g., in bibliographic  instruction from the teaching of tools to the  teaching of information literacy. in addition,  reference work has come to include “going out  to users,” or academic liaison work, as well as  research consultation as a general way to  assist undergraduate student in getting started  on assignments and projects.     the web‐based survey (n=40, response rate  87.5%) showed that 32.4% of libraries have an  integrated inquiry point which incorporates  information queries and other queries that are  not necessarily related to traditional library  reference services (272, table 1). this survey  result supports the findings of the literature  study in showing a trend of library services  moving away from the traditional reference  desk.    a majority of the responding libraries still  retained a separate reference department, but  a significant number of libraries have  developed departments incorporating  reference services with other library services.  those that retained the separate department  varied in how they described services to  patrons, the most common name being  information services, a more user‐friendly and  descriptive name.     in staffing the reference service, the  respondents were asked to indicate the  classification level of their staff using the  higher education worker (hew) scale (an  australian salary scale, based on  competencies, minimum 1 / maximum 10).  staff spans a variety of levels (4‐10), the most  common level being hew6, a level where all  libraries had staff. this indicates that a large  part of reference staff in australian academic  libraries are highly qualified.  the shift in higher education, resulting in  greater numbers of students and fewer staff  (including librarians), has in many libraries  resulted in a more flexible organization of  reference services, and the utilization of staff  from other sections of the library for manning  the reference service point. there is also  evidence of how the changing student  population leads to changing demands for  library facilities and services, e.g. a decrease in  the traditional complex reference questions, as  well as in over the counter loan transactions,  and an increase in more general queries.     conclusion – reference services in australian  academic libraries are becoming more flexible  and integrated (although the definition for  integrated is still unclear), in part as the result  57 evidence based library and information practice 2009, 4:3  of client demand, and in part due to decreased  funding. the author sees an emerging role for  reference librarians in helping patrons to  navigate the increasingly complex information  environment, and to assist in developing the  skills to critically evaluate the information  they access for authoritativeness and  appropriateness.       commentary     the article provides an interesting historical  overview of the development of the definition  of reference services, as well as a summary of  reference service models and the different  aspects of reference services. the author  mentions samuel green as the father of  reference services, being the first librarian to  advocate personal assistance to readers. green  encouraged librarians to “mingle with  readers” (green 80), a suggestion that has  inspired librarians to start working as roving  (or roaming) reference librarians, an  alternative or complement to the traditional  reference desk, that has been discussed and  used during the last 15 to 20 years (gourtois  and liriano; kramer). one wonders why this  aspect was not revealed in the overview of  reference service models, nor did it appear in  the results of the web‐based survey. for the  international audience it would also have been  useful had the article placed the australian  findings in a broader context by comparing  results from this study with similar overseas  studies in the discussion.     as mentioned above, the article is not entirely  clear about the meaning of ‘integrated’ when,  for example, stating in the conclusion, “… to a  more flexible, integrated model …” (275). it  would have increased the understanding of  the article had the author defined the word  integrated in this context.    the article lacks some information concerning  the web‐based survey, most importantly the  survey questions, without which the validity  and reliability of the study is harder to  determine. the reviewer also notes, somewhat  surprisingly, that the respondents have not  been de‐identified. if the individual libraries  did agree to have their comments identified, it  would be preferable for this to be stated in the  article.    the appendices section comprises a large part  of the article, and, with small exceptions,  present the results in a clear and easily  understood way. the exceptions include  appendix 1 (library statistics) and appendix 2  (student statistics) which both take a fair  amount of time to draw conclusions from  without the summary in the text. it is hard to  see how interesting this is for the reader  outside australia. the usefulness of the tables  of numbers in the appendices would have  been greater had the author chosen to use  proportions instead of absolute numbers,  which tells us very little.     for the non‐australian reader the higher  education worker levels (hew) would  benefit from some additional explanation  since corresponding systems do not exist in all  countries.     the results presented in appendix 3, table a4  – a9 are not presented in the text, but only  referred to as “a summary of survey responses  is included as appendix 3” (272). in the text,  the author presents the findings concerning  service points and staffing, but leaves to the  reader to draw conclusions about the more  specific trends in reference services from the  tables. since the tables in this appendix  provide information relevant to the objectives  of the article, it would have been interesting if  the author had chosen to expand on these  findings in the text.    the article provides an interesting  contribution to the understanding of the  development of reference services in  australian academic libraries, which should  be of interest to reference service managers  everywhere.            58 evidence based library and information practice 2009, 4:3  59 works cited    courtois, m., and maira liriano. “tips for  roving reference.” college &  research libraries news 61.4 (2000):  289‐91.    green, s. “personal relations between  librarians and readers.” american  library journal 1.1 (1876): 74‐81.    kramer, e.h. “why roving reference: a case  study in a small academic library.”  reference services review 24.3 (1996):  67‐ 81.  evidence based library and information practice    evidence based library and information practice       editorial responsibilities      2017. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     editor-in-chief: lorie kloda   associate editor (evidence summaries): heather pretty   associate editor (research articles): rebekah (becky) willson, lisl zach   associate editor (classics & reviews): i. diane cooper   associate editor (commentaries, using evidence in practice, research in practice): lorie kloda   associate editor (feature): lisl zach   communications officer (news): tatiana bryant   production editor: michelle dunaway   editorial intern: rachel hinrichs   editorial advisors: lindsay alcock, alison brettle, jonathan eldredge, denise koufogiannakis   copyeditors: richard hayman (lead copyeditor), georgianne bordner, julie evener, heather healy, julie james, jane morgan-daniel, stacey penney, elizabeth stregger,  nikki tummon, alison yeoman   indexing support: kate shore   404 not found microsoft word abstracts.doc evidence based library and information practice 2007, 2:2  3 evidence based library and information practice     feature    abstracts of papers and poster sessions of the 4th international evidence based  library and information practice conference: transforming the profession    durham, north carolina, united states  6‐11 may 2007    reprinted with the permission of the conference organizing committee.      best paper   using rubrics to collect evidence for  decision‐making: what do librarians need  to learn? megan oakleaf, phd, syracuse  university, united states.  papers highly commended  1. the neglected voice: is there a role for  qualitative systematic reviews in  eblip? andrew booth, msc mclip,  school of health and related research  (scharr), university of sheffield,  united kingdom & anne brice, ba  (hons) diplib mclip, university of  oxford, united kingdom.    2. information skills training in health  libraries: are we any nearer the  evidence? alison brettle, msc, ba  (hons), university of salford, united  kingdom.     3. implementing evidence‐based practice  to enhance library web site usability: a  case study at northwestern university  library. h. frank cervone, msed,  northwestern university, united states.     4. best demonstration of the ebp model:  comparison of training interventions  for pubmed search skills amongst 3rd  and 4th year medical students. glenda  myers, d litt et phil, witwatersrand  health sciences library, university of  the witwatersrand, south africa.    5. finding our foundation: analysis of the  lisa database for research  retrievability. carol perryman, ms lis,  research fellow, school of information  & library science (sils) university of  north carolina, united states.    best poster   assessing the need to update  prevention guidelines: a comparison of  two methods. gerald gartlehner, md, mph,  suzanne l. west, md, mph, kathleen n.  lohr, phd, mphil, ma, leila kahwati, md,  mph, jana g. johnson, md, russell p.  harris, md, mph, lynn whitener, msls,  drph, christiane e. voisin, mlis, & sonya  sutton, bsph, unc cecil g. sheps center  for health services research, united states.    evidence based library and information practice 2007, 2:2  4 posters highly commended  1. testing a new approach to the manual  literature search process in systematic  reviews. laura c. morgan, ba, gerald  gartlehner, md, mph, dan jonas, md,  curtis bryant, & claire de la varre,  msis, unc cecil g. sheps center for  health services research, united states.    2. supporting evidence‐based medicine  in an academic health system: a  unique partnership between the center  for evidence‐based practice and the  biomedical library. gretchen kuntz,  msw, mslis, clinical liaison librarian,  biomedical library, anne seymour,  interim director, health sciences  libraries, craig a. umscheid md, co‐ director, center for evidenced‐based  practice, & kendal williams md,  director, center for evidence‐based  practice, university of pennsylvania  health system, united states.    3. a place for us: the relevancy of the  academic library “as a place.” jason  martin, mls, ed.d in progress,  university of central florida libraries,  united states.     4. is “igaku toshokan” providing the  evidence to help medical librarians  with decision‐making? yukiko sakai,  msis, keio university & yasuhiko  kiyama, juntendo university library,  tokyo, japan.      presentation types    contributed papers include both original  research and innovative applications of  eblip in library and information  management. papers that deal with library  support of evidence‐based practice in other  fields such as health, social work and public  policy were also welcome. accepted  research papers are published on the  conference website at  http://www.eblip4.unc.edu.    posters will allow for the presentation of  new developments in evidence‐based  practice and work in progress. posters offer  an excellent opportunity for new conference  presenters. accepted poster abstracts are  available on the conference website at  http://www.eblip4.unc.edu.      legend:       research paper      keynote     award‐winning paper or poster      speaker abstracts     libqual+ ™ and the evolution of  “library as place” at radford university.  eric ackermann, msis, mcconnell library,  radford university, united states.    objective: to determine if the libqual+ ™  survey captured any changes in user  satisfaction with the “library as place” due  to alterations made to mcconnell library  between 2002 and 2005.  design: effect size meta‐analysis of the  adequacy gap scores data from four aspects  of the library as place dimension from the  2002, 2005, and 2006 libqual+ ™ surveys.  setting: mcconnell library, the academic  library serving radford university, a  medium sized (c. 9300 students), four‐year,  primarily residential public university  located in southwestern virginia.  participants: using a sample of the whole,  undergraduates (n = 570), graduate students  (n = 172), and faculty (n = 192) who  completed the 2002 libqual+ ™ survey;  the undergraduate (n = 1237) and graduate  http://www.eblip4.unc.edu http://www.eblip4.unc.edu evidence based library and information practice 2007, 2:2  5 students (n = 232) who completed the 2005  libqual+ ™ survey, and the faculty (n =  162) who completed the 2006 libqual+ ™  survey.  intervention: increased the hours of  operation, created a designated quiet study  area, upgraded computer hardware and  software, provided more comfortable  furniture, and added more public access  computers.  main outcomes measures: for each  participant group, the mean adequacy gap  (i.e., difference or “gap” between the  perceived and minimal acceptable levels of  user satisfaction) for the library as a quiet  study place (quiet study), a haven/getaway  (haven), a comfortable place (comfortable),  and a contemplative environment  (contemplative). the metrics used were  cohen’s d (standardized mean difference),  95% confidence intervals (95% ci), binomial  effect size display (besd, or the d statistic  represented as a percent success rate  differential), and the resulting change (if any)  in the adequacy gap (ag). a positive result  indicates an improvement in user  satisfaction, a negative result, the opposite.  main results: overall the response to the  changes in the library as place between  2002 and 2005/2006 were positive (d = 0.10,  95% ci [‐0.10, 0.30], besd = 4.9%, ag = 0.19),  more so for the graduate students (d = 0.13,  95% ci [‐0.01, 0.26], besd = 6.4%, ag = 0.26)  than the undergraduates (d = 0.10, 95% ci [‐ 0.11, 0.31], besd = 5.1%, ag = 0.20), or the  faculty (d = 0.05, 95% ci [‐0.09, 0.18], besd =  2.4%, ag = 0.09).  the changes affected the overall perception  of the library as haven the most (d = 0.24,  95% ci [0.05, 0.44], besd = 12.1%, ag = 0.51),  followed by quiet study (d = 0.06, 95% ci  [0.03, 0.09], besd = 3.0%, ag = 0.12),  comfortable (d = 0.06, 95% ci [‐0.01, 0.14],  besd = 3.1%, ag = 0.12), and the  contemplative (d = 0.03, 95% ci [‐0.10, 0.30],  besd = 1.6%, ag = 0.06) aspects.  conclusion: the libqual+ ™ adequacy  gap data captured the changes in users  satisfaction related to changes in “library as  place,” and will be used in future  assessments. future research needs to  determine if the changes in user satisfaction  reflect a change in users’ perceptions or a  change in users’ minimum expectations,  both components of the adequacy gap, or  both.       eblip: clear, simple – and wrong? a  friendly debate. andrew booth, school of  health & related research (scharr),  university of sheffield, united kingdom &  t. scott plutchak, director, lister hill  library of the health sciences, university of  alabama, united states.    the theory of evidence‐based library and  information practice (eblip) has made great  strides over the past decade. during this  time, much attention has been paid to levels  of evidence and the importance of getting  the question right – that is, phrasing the  question in a way that it is amenable to  evaluation by reliable evidence. indeed  asking questions and locating answers are  key competencies for the library and  information profession. this session will  explore the degree to which the “big”  questions that face our profession can be  structured in such a way as to meet the tests  of eblip. in other words, can the eblip  approach be used for all of the questions our  profession faces, or only for some narrowly  defined subset?       the neglected voice?: is there a  role for qualitative systematic reviews in  eblip? andrew booth, msc mclip, school  of health and related research (scharr),  university of sheffield & anne brice, ba  (hons) diplib mclip, university of oxford,  united kingdom.    evidence based library and information practice 2007, 2:2  6 question: do systematic reviews of  qualitative research offer a useful way  forward for synthesis of library and  information science (lis) literature?  background: over the last decade  systematic reviews of effectiveness have  become an established methodology in  pursuit of evidence based practice. recent  years have seen the adoption of such  methodologies within evidence based  library and information practice (eblip).  nevertheless systematic reviews of  quantitative evidence within the lis  literature carry acknowledged difficulties.  these include a shortage of rigorous  comparative studies, poor quality of  reporting and the inconclusiveness of results  (lerdal). eblip “attempts to integrate user‐ reported, practitioner observed and  research‐derived evidence as an explicit  basis for decision‐making” (booth).  individual surveys of library staff or users,  or other investigative methods such as  interviews and focus groups, may provide  useful local operational data but offer little  in the way of reproducible insights.  systematic analysis of the reports of users or  the observations of practitioners, as  recorded in the research literature, offers the  potential of holistic insights into the delivery  of information services. in the absence of  compelling evidence on effectiveness,  research on acceptability of services and  preferences of service users may enable  practitioners to tailor their services more  appropriately.  additionally insights from qualitative  research conducted with information staff  and then systematically synthesized and  analysed may assist in identifying  environmental or psychological barriers to  delivery of specific services. systematic  reviews of qualitative research (qualitative  metasyntheses) are increasingly being  recognised for their potential contribution to  an understanding of public service delivery  and user views (oliver). they also have  great explanatory potential where  interventions do not prove as effective as  expected or where implementation proves  particularly problematic. do such reviews  offer similar potential within delivery of  library and information services?  methods: as co‐convenor of the cochrane  collaboration’s qualitative research  methods group the principal author has  been systematically collecting and reviewing  methodological literature on the conduct of  qualitative metasyntheses. this has also  involved identification and collection of  examples of such reviews within healthcare.  furthermore the principal author has been  developing a prototypic taxonomy of the  types of question that such reviews address.  in addition both authors have been working  together on a qualitative metasynthesis in a  library and information topic to explore  potential applications for such methods.  main results: potential applications for  qualitative metasynthesis within lis include  barriers and facilitators to delivery and  uptake of services, exploration of user views,  investigation of new roles and prioritisation  of services where evidence on effectiveness  is equivocal. these and other applications  will be illustrated with examples from  within lis literature.  conclusion: this, the first presentation to  systematically examine the potential of  qualitative metasynthesis within lis, will  provide the audience with a methodological  toolbox with which to explore the views and  attitudes of library users and information  staff. it is hoped that encouraging  exploration of questions that lie beyond  effectiveness will stimulate further  development of the lis evidence base to  answer the more  enduring questions faced by library  practitioners.       information skills training in  health libraries: are we any nearer the  evidence based library and information practice 2007, 2:2  7 evidence? alison brettle, msc, ba (hons),  university of salford, united kingdom.    introduction: for some time health  librarians have been wondering whether the  training they provide is effective (eldredge,  2001) and two systematic reviews have  attempted to find answers (brettle, 2003;  garg and turtle, 2003). both reviews  concluded that there was only limited  evidence (based on weak research studies)  that training improves search skills, there  was no evidence about different types of  training and nor was there any evidence of  the impact of training on patient care. a  further conclusion (brettle, 2003) was that  until objective and validated measures of  evaluating training were used, it would be  impossible to determine whether training  was effective. more recently australian  librarians were still asking questions  regarding the effectiveness of training  clinicians to perform literature searches  (lewis and cotter, 2006). this presentation  updates a systematic review presented at  the first ebl conference in 2001 and  questions whether, at eblip4, we are any  nearer to finding evidence to aid health  librarians in their practice?  question: to examine whether training  improves searching skills; to identify  effective methods of training; to examine  whether training effects patient care; to  determine the availability/use of objective  and validated outcome measures.  data sources: searches of eric, lisa,  medline, cinahl, personal collection of  material, handsearch of journal of medical  library association, health information and  libraries journal and evidence based  library and information practice   study selection: studies had to meet all the  following criteria to be included in the  updated review: studies published post  2001 (cut off date for previous systematic  review); health related (i.e. study on trainee  or qualified clinicians in either academic or  clinical setting); study examining  information skills training (defined as:  training that included searching electronic  databases such as medline, searching the  internet, finding information for patient care  or searching for information in relation to  evidence based practice); any research  design using an objective outcome measure  (i.e. one that measures change in  knowledge/skills)  data extraction: data was extracted by the  author using a tool adapted for the purposes  of the study. this was based on tools  developed by the university of salford  (hcprdu, 2003) for critically appraising  different types of research evidence and the  reliant data extraction tool  (koufogiannakis et al, 2006).  conclusion: the conclusions of the review  will provide more recent evidence on the  effectiveness of information skills training in  a health context. the author will reflect on  any changing patterns and quality of  research in this field and suggest how health  librarians can improve their evidence base.       transforming the evidence base for  high quality, effective academic health  sciences library services and resources: the  readiness of the library director  community in the us and canada.  gary d.  byrd, phd, university at buffalo (suny) &  steven j. squires, mls, university of north  carolina at chapel hill, united states.    question: should academic health sciences  libraries transform the evidence base for the  quality and effectiveness of their services  and resources from traditional measures of  inputs and outputs to measures of outcomes,  and are they prepared for such a transition?  design: this study will report the results of  a web‐based survey of us and canadian  academic health sciences library directors.  the survey questions will be pre‐tested for  clarity and content validity by senior staff at  the authors’ libraries. the number of  evidence based library and information practice 2007, 2:2  8 questions will be limited to those that can be  completed within 30 minutes. invitations to  participate will be sent to library directors in  the us and canada by email, with email and  telephone reminders as necessary.   setting: the study will be conducted in  february 2007 by the authors (the chair and  one of the members of the association of  academic health sciences libraries  (aahsl) assessment and statistics  committee) using a web‐based survey tool.   participants: the survey will include all  directors of academic health sciences  libraries in the us and canada (about 160  individuals). demographic data about the  directors including age, years of experience,  and sex as well as data about the size and  resources of the library will also be collected  from respondents.   intervention: the survey questions will ask  these library directors to give their current,  considered opinions about the need,  appropriateness and urgency to move away  from, or to supplement, traditional  measures of the size and scope of library  collections, expenditures, personnel, and use  of library services as collected in the annual  surveys of associations like the association  of academic health sciences libraries  (aahsl) and the association of research  libraries (arl). are they and their host  institution ready and willing to collect the  kind of data needed to measure outcomes  such as the cost effectiveness and quality of  library resources and services in support of  the goals of the academic and research  community they serve? the survey will also  ask about the availability of “measures of  excellence” from their host institution (such  as those collected since 2003 at the  university of north carolina at chapel hill)  for analysis and correlation with library  outcomes measures, and the extent to which  each library is already attempting to collect  its own outcome measures.   main outcome measures: the paper will  present a descriptive analysis of the survey  results showing the range and distribution  of responses to each question, for all  respondents and demographic subgroups.  as appropriate, simple statistical  correlations will show the relationship  between the responses to groups of  questions.   main results: we hypothesize that most  library directors will agree that additional  outcome measures are needed to  demonstrate the cost effectiveness and  quality of library resources and services.  however, we also hypothesize that few  directors will demonstrate a readiness and  willingness to begin collecting the data  needed for these measures. conclusion: the  transformative potential of outcome  measures of excellence has been argued by  health sciences library leaders since at least  1986 when virginia holtz gave her janet  doe lecture on this topic at the medical  library association annual meeting [jmla  74(4):305‐14]. we hope the results of this  study will help to stimulate concrete action.       implementing evidence‐based  practice to enhance library web site  usability: a case study at northwestern  university library.  h. frank cervone,  msed, northwestern university, united  states.    this paper examines the implementation of  evidence‐based practice within the web  advisory group at the northwestern  university library. through this  examination, some implications are  developed for the role of evidence‐based  practice in academic library web site design.   objective: in many libraries, web site  design is informed more by opinion rather  than evidence. a critical question for  libraries then is how to move from an  opinion‐based design process to one that is  based on sound principles of evidence‐ based practice. methods: following an  examination of the background of evidence‐ evidence based library and information practice 2007, 2:2  9 based practice and its applicability to the  web site design and implementation  processes, the paper examines the  application of evidence‐based practice in  one particular setting: a large arl library  serving a diverse population of  approximately 20,000 users.   results: while implementation of evidence‐ based practice within the web advisory  group has been successful, the  implementation pathway has not been  without problems and challenges. a  remaining question is how to most  effectively move evidence‐based practice  out of this small group and into other areas  of the library.   conclusions: using evidence‐based practice  in web site design can make a significant  difference in the effectiveness of the web site  design process. by evaluating the outcomes  of the implementation within the web  design group, some conclusions on how to  engage staff throughout the library in  evidence‐based practice have been  developed.        evidence and e‐government. rowena  cullen, associate professor, school of  information management, victoria  university of wellington, new zealand.    the use of online technology in government  has occurred very rapidly, driven by the  ongoing development of it, by citizens’  expectations, and by the desire of politicians  to be ‘leaders’ in e‐government. in this rapid  transition, the concepts of measurement and  evaluation have lagged behind. some of the  important questions that need to be  addressed include: how can the  effectiveness of e‐government systems and  services be measured? on what basis should  decisions about future it developments, and  redevelopments be made? what is the role  of evidence in e‐government decision‐ making? what data  and methodologies would provide this  evidence? what can our experience with  evidence‐based library and information  practice offer to improve e‐government  decision making and information  management?       preparing for a slimmer bottom line:  analyzing and reducing a serials collection  using a mixed methods approach.  lori  delaney, mlis, carolina population center  library, university of north carolina at  chapel hill, united states.    objective: the purpose of this study is to  conduct a comprehensive evaluation of a  serials collection to make evidence based  decisions during an annual subscription  renewal process. using both quantitative  and qualitative data collection and analysis  methods, this study features an evaluation  of the collection title‐by‐title and results in  subscription costs that are 45% lower than  the previous year, while also aiming to  provide a relevant specialized serials  collection.   context: the carolina population center  (cpc) library is a special library at the  university of north carolina at chapel hill  (unc), which primarily serves cpc  researchers, and also serves the university  community and the public. the cpc library  houses a collection of materials which  support a broad range of population  research. because the cpc library is solely  funded by the center, and is a not a  departmental library of the unc system,  collection development administration is the  responsibility of cpc library staff, and costs  for the serials collection are paid by the cpc  library. a thorough evaluation of the cpc  library serials collection is necessitated by a  drastically reduced budget allocation. a  review of the serials print collection is also  long overdue due to cpc’s evolving  research portfolio as well as the increase in  evidence based library and information practice 2007, 2:2  10 electronically available content via the unc  libraries databases. the cpc library uses a  subscription services agency to manage  print subscriptions to 90 titles. some titles  are unique holdings on the unc campus,  while other titles are also held by other  campus libraries. duplication of titles is  suitable when the content is highly relevant  to population research. methods and  analysis: this study utilizes a mixed  methods approach, in a sequential  triangulation process. first, a review of the  existing literature is conducted to determine  common criteria used to inform decision‐ making about serials renewals and  cancellations. quantitative data is derived  from multiple sources: cost per title  (previous year, current year, and rate of  increase between the two years), usage  statistics, availability in print from other  unc libraries, and electronic availability  via campus library databases or a  publisher’s website. in addition, data is  compiled on the number of articles written  by cpc researchers in each serial during the  previous three years. these data are  assembled in an excel spreadsheet, and then  analyzed. a collaborative review is  conducted by the two cpc librarians and a  proposal is developed listing which titles to  renew and which to cancel, with a detailed  breakdown of the financial implications.  qualitative data is obtained from the  center’s director, deputy director, and the  center’s advisory council, resulting in a  revised proposal. the revised proposal is  then distributed to cpc researchers and  staff with a request for further input.   results and conclusions: data analysis  results in a final decision of titles to renew  and to cancel, with a cost savings of 45%.  the instrument developed for the study and  the data collected can be utilized in future  years’ analysis. staff of specialized libraries  can utilize the data collection and analysis  methods for use in evaluating their own  serials collections.     assessment, evaluation, and value:  current trends. rosalind f. dudden, mls,  dm/ahip, fmla, tucker medical library,  national jewish medical and research  center, united states.   the field of library evaluation studies and  evidence‐based practice are linked together  with a common goal, managing an effective  library. the resources of a library to do  studies vary greatly by size and type of  library. this paper presents an overview of  resources and ideas uncovered during two  years of study of the field of library  evaluation with the goal of making it  understandable and useful to the librarian in  a small setting. measuring all parts of the  management continuum: needs, inputs,  quality processes, outputs, quality service,  outcomes, and impacts, is encouraged to  manage the smaller library by adapting the  philosophy of a culture of assessment now  being developed by arl. resources for  learning about outcomes measurement are  available on the web and can by adapted to  any setting. the purpose of these  assessments is multidimensional with the  goal of improving economy, efficiency and  effectiveness. if done in a systematic way,  the results of local assessments, combined  with published studies, can show the value  of library services. to communicate this  value, librarians are encouraged to   • understand the goals and values of  the organization and the library  • understand the user by being  customer focused and assessing  needs  • establish a culture of assessment  • use a variety of measures, both old  and new  • use outcomes to tell your story        evidence based library and information practice 2007, 2:2  11  cognitive biases as obstacles to  effective decision making in eblip.   jonathan eldredge, mls, phd, university of  new mexico health sciences library and  informatics center, united states.   objective:  eblip advocates have explored  in published reports the first three steps of  the eblip process in considerable detail. in  contrast, not much attention has been paid  to exploring the fourth decision making  step, particularly on the kinds of obstacles  that can thwart the ebl process in  producing an appropriate decision.  cognitive biases potentially can interfere in  this fourth step for decision makers by  either causing misperceptions about their  environments or by hindering their  performing effective analyses. this  exploratory research paper describes a  survey on those cognitive biases reported as  commonly occurring in library and  information practice decision making.  respondents were self‐selecting members of  the (us) medical library association.   methods:  narrative review. the author used  a list of cognitive biases in the wikipedia on  january 8, 2007 to determine initially what  forms of cognitive biases might be most  relevant to decision making in our field. the  author then conducted a literature review  on the relevant social psychology research to  identify approximately twenty forms of  cognitive bias that might affect decision  making processes. this social psychology  literature on cognitive biases consists  primarily of experimental study designs.  while reviewing this area of the social  psychology literature the author also noted  any discussions that offered strategies for  overcoming or at least neutralizing the  effects of specific forms of cognitive bias.  the author shared this list of definitions  with colleagues for their input on its clarity  and comprehensiveness. one colleague  suggested the addition of a new form,  worst‐case scenario, while others offered  editing suggestions for the definitions.   descriptive survey. the author obtained  human subjects research approval from his  institutional review board on april 2, 2007.  beginning on april 12th, the author  publicized the survey on listservs of three  mla sections to which he belonged:  research, public health/health  administration, and collection  development. he repeated this publicity on  april 19th and 26th. he also publicized the  survey in the april 19, 2007 issue of the  online newsletter mla‐focus. interested  mla members could click on a link in the  publicity email or online newsletter text to a  survey on a server run with opinior  software. following a screen that provided  their consent, survey participants reviewed  a list of nineteen forms of cognitive bias  with accompanying brief definitions of each  form. the next screen had an identical list  with definitions, except that respondents on  the actual survey were asked to indicate the  top three (3) forms of cognitive bias that  they had observed by librarian colleagues  during group decision making processes.  respondents were then asked to categorize  themselves by broad job descriptions,  library types, and their geographic mla  chapters. when field tested, the survey took  about five minutes to complete. the  cognitive biases list in the survey included  the following forms, with definitions:   anchoring  attribution  authority  confirmation  déformation professionnelle  expectancy effect  groupthink  halo or horns effect  naïve realism  outcome bias  perseverance of belief  positive outcome  primacy effects  question framing  recency effects  selective perception  evidence based library and information practice 2007, 2:2  12 status quo  stereotype  storytelling  wishful thinking  worst‐case scenario   results: 135 mla members completed the  survey during the survey period of april 12  through may 2, 2007. the seven most  frequently mentioned cognitive biases in  decision making appear here in descending  order:   déformation professionnelle  status quo  authority  anchoring  groupthink  halo or horns effect  confirmation     79 of the 135 respondents provided text  examples from their experiences or made  comments. many of these 79 respondents  made normative comments, almost all of  them negative, about instances when  colleagues succumbed to cognitive biases.  the author was troubled by these negative  comments that mainly cast blame on others’  personal failings because he had  emphasized in both the survey publicity and  in the instrument itself that cognitive biases  are common human characteristics. these  comments underscore the need to educate  members of our profession about cognitive  biases.   conclusions: the author plans to publish  the results of this exploratory research study  to raise awareness within our profession  about cognitive biases in decision making  processes. in addition, the author intends to  design an mla continuing education course  that offers a sympathetic view of cognitive  biases as common human foibles and that  emphasizes the use of certain strategies to  overcome or neutralize the effects of certain  forms of cognitive biases.      evidence based management as a tool  for special libraries.  bill fisher, school of  library & information science, san jose  state university, &  w. davenport (dav)  robertson, mslis, national institute of  environmental health sciences, united  states.     question: is evidence‐based management  an effective tool for use in the special library  environment?   data sources: the general management  literature of the past six years, as well as the  established evidence‐based management  literature will be reviewed. the  library/information science literature of the  past six years will then be reviewed. search  methods will include major  business/management databases, major  library/ information science databases,  relevant web sites, blogs and wikis, citation  searching on key articles and follow‐up on  references cited in the studies found.   study selection: included management  studies will discuss the major issues where  evidence‐based management has been  applied. included library and information  science studies will discuss management‐ related issues in special libraries.   data extraction: relevant data and key  issues will be identified from the literature  independently by the two researchers using  a pre‐determined extraction form.   main results*: the major issues in special  library management are compared to the  major issues in evidence‐based management.  also, the relevance to special libraries of  other evidence‐based management issues is  discussed.   conclusion*: the results of this study show  the potential for applying evidence‐based  management to special libraries. questions  to be used in further research on evidence‐ based management in special libraries are  recommended. *these sections currently  provide the anticipated results/ conclusion  since the study is in progress. when the  evidence based library and information practice 2007, 2:2  13 study has been completed, specific issues  will be identified in the main results and the  specific questions will be identified in the  conclusion.        results of participant observation in  the fifth judicial district: customizing the  county law library to meet the needs of  small firm attorneys in rural kansas.   susan g. fowler, mls, clim, lyon county  law library, united states.    a participant observation study was  conducted in the fall of 2005 to identify how  attorneys in the fifth judicial district of  kansas access and use legal information.  the goal was to obtain information that  would be used to customize services at the  lyon county law library, which had been  re‐established in 2003 following three years  of dormancy due to new courthouse  construction. while the basic legal reference  materials had been obtained, a better  understanding of how attorneys in the fifth  judicial district access and use information  was needed for continual library  improvement. since we were looking at how  attorneys access and use information, the  qualitative methodology of participant  observation was chosen to gather data.  fortunately, the information profession has  largely validated qualitative methodologies  (e.g., action research, field methods,  participant observation) to answer questions  regarding information seeking and using  habits. we know that circulation statistics  alone cannot tell the story of what users are  looking for, whether they find it and  whether it matches their information need.  we also know that to ask people how they  look for information (whether by interview  or survey) is to depend on an “in the  moment” answer, which may be unduly  influenced by that moment’s context or  suffer from memory gaps. six attorneys  (13% of the law library’s membership) were  observed for two days each. each day’s  notes were coded and analyzed for patterns.   in addition, commonalities were noted  across the types of practice and  demographics (i.e., rural vs. frontier) in  accordance with naturalistic inquiry  protocol. the subjects’ environment is one of  many factors that influence the practice of  law. factors such as population density,  office décor and dress, support staffing and  client relations were noted. participating  attorneys answered questions and  confirmed or corrected the researcher’s  impressions during follow‐up conversations.  in 2006, findings from this study were  applied to customize the resources and  services of the lyon county law library.  this research is being applied through the  acquisition of materials pertinent to the  scope of legal work in this area, providing  “ready reference” through instant  messaging (im) and email, increasing access  to research materials by providing online  database training and expanding the online  database collection, and providing a state‐ approved cle on electronic data  management as it pertains to changes to the  federal rules of civil procedure. the lyon  county law library has received very  favorable feedback, which has us committed  to continuing research and subsequent  development of services customized to the  legal community of the fifth judicial district.       evidence‐based practice in other fields:  social work. mark fraser, msw, phd,  school of social work, university of north  carolina at chapel hill, united states.    the purpose of this presentation is to  describe the impact of evidence‐based  practice in social work. emphasis is placed  on a process of using evidence to develop  more effective social work programs. in  addition, challenges in translating and  evidence based library and information practice 2007, 2:2  14 infusing research evidence into practice will  be discussed.       progress towards a ‘one‐stop‐shop’  for critically appraised methodological  search filters: the intertasc information  specialists’ sub‐group search filters  resource.  julie glanville, msc, centre for  reviews and dissemination, carol lefebvre,  msc, uk cochrane centre, & uk intertasc  information specialists’ subgroup project,  united kingdom.    objective: to develop and make publicly  accessible a web site of critically‐appraised  methodological search filters.   participants: members of the uk intertasc  information specialists’ sub‐group (issg),  who provide information specialist input for  health technology assessments (htas)  undertaken on behalf of the uk national  institute for health and clinical excellence  (nice).   project details: a web site of methodological  search filters has recently been launched  (http://www.york.ac.uk/inst/crd/intertasc/).  the first phase of the project has been to  identify and list references to published and  details of unpublished methodological  filters. this process is ongoing and the  website is regularly updated. the site  currently includes references to filters for  studies with a specific focus such as  diagnosis, economic evaluation, prognosis  and therapy. it also includes filters for  specific study designs, such as the recently  published crd / cochrane highly sensitive  search strategy for identifying randomized  controlled trials in medline, designed by  the centre for reviews and dissemination  and uk cochrane centre search filters  design group. the second phase of the  project is to critically appraise these filters  and post the appraisals on the web site.  guidelines for the critical appraisal have  been drafted and piloted by the issg and a  final draft is in preparation. the final stage  of the project is to test out the filters in a  range of independent settings. this will be  achieved initially by asking collaborators to  use the included studies of completed htas  and systematic reviews as gold standards  against which to test the performance of  individual search filters. the site will  publish these data to allow comparisons  with respect to the sensitivity and precision  data published by the original authors.   conclusion: the first phase of the project has  resulted in a bibliography of known filters  and thus provided a ’one‐stop shop’ for  information specialists, reviewers and other  researchers looking for methodological  filters in the context of htas, systematic  reviews and other research. the second  phase of the project will enable users of the  site to assess the rigour of the individual  filters on the basis of the filter design  method through the linked critical  appraisals and thus select and use filters  which are evidence‐based in their design  rather than those that are of a pragmatic or‚  `expert‐opinion’ design, where alternatives  exists. the critical appraisals will also allow  identification of gaps in search filter design  where no ’suitable’ search filter has yet been  identified. once gaps have been identified,  the site also offers filter designers examples  of research‐based methods for filter  development. the final phase of the project  will enable verification or otherwise of the  original sensitivity and precision data in the  light of whether the strategies as originally  published perform similarly in different  subject areas, and / or over different time  periods.       making a commitment to eblip: the  role of library leadership.   suzanne grefsheim, med, msls, national  institutes of health & jocelyn rankin, phd,  center for disease control, & susan  http://www.york.ac.uk/inst/crd/intertasc evidence based library and information practice 2007, 2:2  15 whitmore, ms, msls, national institutes of  health, united states.     program objective: as part of its long‐ standing commitment to create a learning  organization, the library leadership team of  a major research institution planned and  implemented a systematic approach to  fostering eblip practice and enabling  library staff to routinely apply evidence in  decision‐making.   setting: the program was implemented in a  biomedical research library located in  maryland that serves a major u.s.  government agency conducting translational,  bench‐to‐bedside research.   participants: library staff participating in  this initiative included 32 professional  librarians, 4 library technicians, a library  student intern and the head of the library’s  it office, a computer specialist. all have  library and information science questions  which can be answered with the right  evidence.   program: this research library initiated a  program to create an environment  supportive of eblip that included support  and release time for eblip projects, formal  training in eblip skills, and team mentoring  as research projects evolved, all of which  culminated in a library research festival  day. during the trainings, participants  worked in small groups to develop different  research questions and outline appropriate  methods. after training, four teams worked  individually with the instructors to further  refine their research questions and methods.  two demonstration studies using  qualitative methods were implemented  under the leadership of the qualitative  methods instructor. as all the research  studies progressed, the teams consulted  with the instructors by phone, utilized the  onsite expert for qualitative studies, and  conferred with other experienced  researchers on staff who volunteered to  serve as mentors.   main results: by providing a supportive  framework for eblip practice, library  leadership can successfully engage staff in  eblip thinking and small research studies.  after five months in which to design and  conduct a research project, the major library  services teams and others who worked more  independently presented their results at a  library research festival. highlights of  results from their studies on collection  analysis, federated searching, information  desk services and document delivery, and  informationist services will be briefly  described.   conclusions: librarians with some training  in the research process and ongoing  mentoring can design, conduct and learn  valuable information that will help them  introduce new services; improve collection  development; and better understand the  information‐seeking behavior and customer  services needs of users in general as well as  specific target groups so the right  information reaches the right person and the  right time.        substantiating the value of library and  information services: collecting and using  the evidence to tell the real story. josé  marie griffiths, dean and professor, school  of information and library science,  university of north carolina at chapel hill,  united states.    in today’s climate of strained budgets and  demands for increased accountability and  transparency, public, academic and special  libraries are under greater pressure than  ever before to “prove the worth” of  information services and libraries to their  constituencies and communities. this  presentation will provide an in‐depth  description of the methodology and results  of a series of large scale value studies  conducted over the years by griffiths and  king, the most recent involving public  evidence based library and information practice 2007, 2:2  16 libraries in the states of pennsylvania and  florida. professor griffiths will review the  evidence and suggest ways in which the  study results may be used to inform  stakeholders about the value of library and  information services.       wrestling with the appraisal and  production of evidence – and the winner  is ..!   lotta haglund, ba, mlis, & david herron,  karolinska institutet, university library,  sweden.     objective: using eblip to stimulate  professional (scholarly) development   setting: a large medical research library in  sweden.   method: two approaches have been taken  involving: 1) the development of the critical  appraisal skills of the library staff, and 2) the  production and publication of evidence. for  the development of the critical appraisal of  evidence, we have been holding a journal  club since october 2004. the club started as  a less structured meeting using a home‐ made template for the contextual analysis of  the articles, and has more recently  developed into a more stringent and  objective critical appraisal exercise including  the use of the cristal checklist on  appraising a user study. on the side of the  production and publication of evidence, we  have recently run two projects. firstly, a  qualitative research project using the  method of “participant observation” to  investigate the information searching  behaviour and it support needs of younger  researchers (under 40); based on the insight  that more local knowledge is needed about  their actual everyday information behaviour  when using the electronic library or other  resources. secondly, we have attempted to  use collaborative writing in the production  of an article on library educational  development with a view to make the  writing process easier and more effective  and spread knowledge about the project  amongst the staff.   main results: the outcomes from these two  approaches are promising. awareness about  lis research, about qualitative research  methodology (specifically) and the scholarly  awareness (generally) have increased  markedly amongst the library staff by using  the eblip approach. we are also seeing  indications of the effect of these efforts on  various library activities, i.e. team  discussions and user education. library staff  also show more interest in reading the  published literature and publishing  themselves.   conclusion: to be able to develop the  services of the library to better suit the  academic setting of the future, one strategy  is to develop knowledge about research  methodology and design, and the scholarly  work of researchers. using the approaches  described above is helping to open windows  into the research world.        professionalism and evidence‐based  practice: reflections of a university  librarian. margaret haines, university  librarian, carleton university, ottawa,  canada.    this paper will provide a personal  perspective on the impact of evidence‐based  practice on the concept of professionalism in  the library community. the author will  draw on her career in the uk and canada  and demonstrate how the concept of  evidence‐based practice was adopted by  librarians in the uk and has significantly  influenced the development of new  professional qualifications from cilip – the  chartered institute of library and  information professionals. she will also  show how leadership in evidence‐based  information practice has led to expanding  roles for library and information  evidence based library and information practice 2007, 2:2  17 professionals in many sectors as well as new  partnerships between librarians and other  evidence‐based professional groups. finally,  haines will draw on personal experiences in  introducing evidence‐based practice to  university libraries in the uk and canada.     when the evidence isn’t enough:  organizational factors that influence  effective and successful library assessment.   steve hiller, mls, ma, university of  washington libraries, martha kyrillidou,  mls, med, association of research  libraries, & jim self, mls, ma, university  of virginia library, united states.     the association of research libraries is  sponsoring an ongoing service on ‘effective,  sustainable and practical library  assessment’ led by steve hiller, jim self,  and martha kyrillidou. this ongoing  program was established after a two‐year  effort “to assess the state of assessment  efforts in individual research libraries,  identify barriers and facilitators of  assessment, and devise pragmatic  approaches to assessment that can flourish  in different local environments.” the initial  two year project involved site visits to 24  academic research libraries during 2005‐06  with each library receiving a report that  evaluated their assessment efforts and  recommended ways to move assessment  forward at that library. the following  factors were identified as important for  good library assessment:   library leadership   organizational culture   identifying responsibility for assessment   library priorities   sufficiency of resources   data infrastructure   assessment skills and expertise   sustainability   analyzing and presenting results   using results to improve libraries   this paper will describe the organizational  factors that facilitate and impede using data  effectively and the implications for  assessment in research libraries and is based  on the findings of the two year effort led by  the authors. while local conditions and  organizational cultures play important roles  in the different approaches each library has  taken to assessment, several of the factors  listed above have emerged as especially  critical for successful assessment and appear  to be the keys to developing effective,  sustainable and practical assessment in  research libraries. having good evidence is  not enough if the library cannot effectively  use that evidence to improve library services  and address customer needs. evidence‐ based librarianship is especially critical now  as libraries are increasingly asked to  demonstrate the value they add to the  academic enterprise and the difference they  make to the work of faculty, students,  clinicians and other researchers. libraries  don’t operate in a vacuum and must also  work with their academic institutions in  addressing the key issues of (a)  accountability, (b) accessibility, (c) quality,  and (d) affordability in higher education  emphasized.       the theory and practice of evidence‐ based information work – one world?   r. laval hunsucker, phd, mlis, university  library, universiteit van amsterdam,   the netherlands.    purpose: to investigate the validity of the  assumption that evidence‐based library and  information practice (eblip) – as conceived,  defined, and implemented up to now – is  indeed applicable to the full spectrum of  library and information activities.   methodology/approach: the author’s study  and conclusions are based on: 1) a survey  and critical appraisal of quantitative as well  as qualitative research as reported in our  evidence based library and information practice 2007, 2:2  18 own professional literature – particularly  with respect to intermediated information  retrieval, reference services, and collection  management; 2) a wide‐ranging  examination and appraisal of major  scholarly publications in epistemology, in  the philosophy and sociology of science and  scholarship, and in cognitive psychology; 3)  long personal experience as researcher and  university teacher, in scientific publishing,  and (since 1989) as academic librarian.   findings: this reflective investigation  suggests that it is much too early to assert  with confidence (as is often nonetheless  done) that eblip enjoys either universal or  holistic applicability in the library and  information services arena. available  evidence and insights point, rather, to at  least these four alternative conclusions: 1)  the eblip model as presently developed  and delimited is by its nature less  productive for supporting truly user  oriented library/information services for the  social sciences and humanities, than for the  medical and natural sciences communities. 2)  it is probably advisable, whatever the field,  to base practice no less on (sound) theory  and theory‐building than on “empirical”  findings. we should look for a better  balance between inductive and deductive  processes. 3) eblip is not uniformly valid  across all aspects of our services and  procedures, but is apparently less  appropriate for some of them than it is for  others (e.g. those which this author has  previously called our more “circumspect”  functions). 4) the disciplines to which one  should normally look, beyond library and  information studies, for useful evidence and  theories, ought to be expanded to include at  least philosophy, sociology, and the  cognitive sciences – and in the post‐ positivist spirit perhaps even to linguistic  and literary studies (semantics, speech acts,  genre theory, rhetoric, etc.).   research implications: the basic  assumptions behind eblip are valid, and in  fact essential to further assuring the  effectiveness and relevance of our work. we  must however, both as researchers and as  appliers of research, develop a less  monolithic and more nuanced vision of the  eblip approach. the research agenda  should incorporate explicit provisions for  accomplishing the necessary differentiations  in respect both of disciplines served and of  functions performed.   practical implications: practitioners must be  prepared to follow where the best evidence  (in the broader and more varied sense  suggested above) leads. that implies an  even more reflective (but also reflexive!)  mindset than eblip already recommends. it  means the readiness to give up – or to adapt  – even the vested interests and articles of  faith themselves on which we are  accustomed to found our professional  legitimation and identity. if, that is (but only  if), the evidence testifies that such is in the  best interests of the clientele we are there to  serve.   value/originality: encourages a more  realistic standpoint regarding, and therefore  a more workable approach to, eblip.       establishing a model for evidence  based collection management.  denise  koufogiannakis, ma, mlis, university of  alberta libraries, canada.     question: how can collection managers and  selectors structure their practice so that  collection decisions are more evidence based?  can a model be established to provide a  framework for decision making in a large  academic institution? what questions need  to be answered and what sources of  information are most appropriate? where  does one begin to find useful information  and how can it make a difference in day‐to‐ day work? setting: the model for collection  management decision making was  developed for the university of alberta  libraries (ual), a large academic library  evidence based library and information practice 2007, 2:2  19 system in edmonton, canada. the general  model can be applied to other libraries  regardless of type. examples from ual  serve to illustrate practical use of the model  and how it can be implemented in practice.   method: a complete review of collection  management decision making was  undertaken. an examination determined  what types of collections questions selectors  ask on a regular basis, what things librarians  should consider when making acquisition  decisions, and the possible sources of  evidence to aid selectors. feedback from  staff was incorporated, and the model allied  with institutional directions, focusing on a  user‐centered approach.   main results: in the past, collection  management was done purely by librarian  expertise and feedback from faculty, with  the occasional use of data as it became  available, but without a concerted effort to  systematically review data and base  collections decisions on what that data was  telling us. there are overarching questions  to which a traditional ebl model may be  used to provide insight and answers, while  at the same time there are day‐to‐day  decisions which rely on local data and  placing that data in the hands of selectors in  a timely way and in a useable format. there  are also innovative collection projects for  which assessment can become a built‐in  mechanism for future evaluation. all these  areas will be reviewed and connected to  appropriate sources of data for decision  making. conclusion: this presentation will  map out a concrete model that can be used  by any library to complement their  collections based decision making. it is  grounded in the examples of progress  within one academic library. these are  preliminary steps in a new way of thinking  about collections decision making and how  to easily implement data into our day‐today  work. this session will prove valuable for  any collection manager wanting to  implement concrete change in order to make  more effective and user‐based decisions.   the making of a journal:  disseminating research to inform practice.   denise koufogiannakis, ma, mlis,  university of alberta libraries, lindsay  glynn, mlis, health sciences library,  alison brettle, ba(hons), msc, institute for  health and social care research, & pam  ryan, ba, mlis, science and technology  library, memorial university of  newfoundland, canada.     objective: the journal evidence based  library and information practice (eblip)  published its first issue in march 2006. the  purpose of the journal is to provide a forum  for librarians and other information  professionals to discover research that may  contribute to decision making in  professional practice. eblip publishes  original research and commentary on the  topic of evidence based library and  information practice, as well as reviews of  previously published research (evidence  summaries) on a wide number of topics.   setting: eblip is an international, peer  reviewed, open access journal published  quarterly by the university of alberta  learning services, using the open journal  systems (ojs) software.   participants: the journal is comprised of an  editorial team of four editors, two  copyeditors, more than 40 editorial advisory  board members, and more than 20 evidence  summary team writers.   program: this paper will detail the process  of creating a new journal to support  evidence based librarianship. the process of  establishing the journal and the evolution  that occurred over the first year will be  discussed. the authors will discuss reasons  why the journal was necessary and the  various roles of sections and the content  within. the importance of open access to the  dissemination of this literature will also be  discussed. main results: by may 2007,  eblip will have published 5 issues. we will  examine the number of submissions to the  evidence based library and information practice 2007, 2:2  20 journal, topics included, and average  turnaround time from submission to  publication.   conclusion: with a full year’s publication  complete and moving forward into the  second year, the editors will reflect on their  experience and discuss future directions for  eblip.       eblip as a tool for knowledge  transfer in a time of demographic change.   joanne gard marshall, phd, university of  north carolina at chapel hill.     question: can eblip be used as tool for  facilitating knowledge transfer between  experienced practitioners and new entrants  to the field?   context: the first of the baby boomers, a  group born between 1946 and 1964, turned  60 in 2007 and the next two decades will be  characterized by increasing numbers of  retirements. library and information science  (lis) will likely be affected even more by  this general demographic shift than other  professions due to the high proportion of  second career entrants to the field and the  reduced hiring in libraries during the 1970s  and 1980s.   design: eblip seeks to identify the best  available evidence from the literature and to  apply it to library decision‐making. due to  their recent experience in preparing  academic assignments, new entrants are  especially well suited to identifying the  relevant literature. practitioners, on the  other hand, have a great deal of tacit  knowledge and on‐the‐ground experience in  managing library and information services.  we propose to train lis master’s students  and their field experience supervisors in  eblip methods. as part of the training, the  students and their supervisors will identify  situations in which additional evidence is  needed to inform library decision making.  observations will be made about factors  that facilitate or act as barriers to the use of  evidence in lis practice and the ability of  eblip to facilitate communication between  the new entrants and experienced  practitioners.   setting: the study will be proposed at unc  chapel hill where there is a strong history  of collaboration exists between the unc  libraries and the school of information and  library science (sils).   main outcome measures: outcome  measures are still under development;  however, there are opportunities in this  project to study information seeking and use  by lis practitioners as well as the ability of  eblip to act as a tool for sharing and  transferring knowledge between new  entrants and experienced practitioners.   results: we anticipate that we will gain an  increased understanding of how  information is currently used in libraries to  inform decision making. we will also learn  about the practicalities of introducing eblip  in academic libraries and extent to which  this tool can become a facilitator of  knowledge transfer between generations.       choices in chaos: designing research  to investigate librarians’ information  services improvised during a variety of  communitywide disasters and to produce  evidence‐based training materials for  librarians.  michelynn mcknight, phd,  ahip& lisl zach, phd, louisiana state  university school of library and  information science, united states.     objective: how can we discover patterns of  how librarians develop new information  services needed when disaster strikes the  community? the authors needed to design a  project using valid research methods to  gather consistently and to analyze  rigorously narrative data from a wide  variety of libraries that have provided  evidence based library and information practice 2007, 2:2  21 improvised services during a wide variety  of disasters.   method: the authors surveyed a variety of  accepted research methods for gathering  and analyzing qualitative narrative data  describing similar phenomena. they tested  some methods in a pilot study of services  provided by librarians in southern  louisiana after two hurricanes in 2005. they  quickly realized that surveys of hundreds of  libraries and interviews of a few librarians  did not produce the kind or amount of data  to answer the “what” and “how” questions  for a variety of libraries in a variety of  disasters. they discussed that study and its  results with several senior researchers  experienced with qualitative methods.  based on what they had learned during the  pilot study and in subsequent discussions,  the researchers designed a much larger  study to gather evidence of common  practice patterns in diverse disasters.  needing to be open to discovery of what  happens in different situations they devised  a research method based on in‐depth  interviews, multiple case studies, and  narrative data analysis to build grounded  theory. the study will conclude with the  development of best practices presented as  case studies and evidence based training  modules for lis students and practicing  librarians. (they submitted the research  proposal to the institute for museum and  library services national leadership grant  program.)   results: the researchers found evidence of  the efficacy of multiple case study and  grounded theory research methods for this  kind of for this kind of research question.  they developed a protocol to gather data  from academic, public, school and special  libraries that provided extraordinary  services during days and weeks of  community disasters caused by earthquakes,  massive blackouts, tornadoes, wild fires,  hurricanes, land slides, floods, chemical  spills and other natural or accidental events.  the imls agreed with their findings on how  to study the question and funded the grant  proposal. the researchers have begun the  two year project and will report briefly on  its progress in this paper.   conclusion: librarians need evidence‐based  case studies and educational material to  learn how to identify needed information  services during any kind of community‐ wide disaster and to respond to these needs  creatively. since this preparation is not  currently included in lis education,  standards and guidelines or research  literature, there is a need for reliable studies  of these phenomena in a variety of libraries  and a variety of disasters. the researchers  studied and tested various quantitative,  qualitative and mixed data gathering  methods, tested them, and designed a  method for gathering and analyzing the  data necessary to support such guidelines  and education. based on their resulting  research proposal to study about twenty  such phenomena, the institute for museum  and library services has awarded them a  two‐year national leadership grant to  perform the study.       paradigms as guides for library  science research.   karin e. medin, phd, mlis, university of  arkansas at little rock, united states.     librarianship is not consistently perceived  as a scientific profession, neither from  within nor from outside of the profession.  when vying for a continued role in a  research intensive university setting, it is  vital that librarians adapt and convey results  using scientific methods in order to fully  participate in the paradigm shifts that are  occurring in the information management  field. in order to powerfully impact the  constituents we serve, we need to engage  with our vendors, potential  competitors/partners in the information  industry, and our campus/complex  evidence based library and information practice 2007, 2:2  22 information technology arms as we devise  and perform our research. this endeavor  has been aptly endorsed in the classic  harvard business review article “on  reaching out in management” by givens  (1952). the paradigms the various  workgroups espouse begin to fruitfully  organize research efforts once awareness  and connections are clear and powerful.  upper‐ and middle‐ administration bear  responsibility for the future of our scientific  reputation, and we neglect rigorous  methods and our status as equal players in  normal science only to our demise. thomas  s. kuhn’s structure of scientific revolutions  (1970) provides the springboard for a  discussion of how librarianship really is a  science, and will thrive only when viewed  as such. a literature review of the peer‐ reviewed material in library science reveals  that there is a dearth of acknowledgement  that paradigms guiding our research do  exist. there is no doubt, however, that our  situation of being in the midst of a  monumental information explosion rivets us  as scientists to our very cores. as applied,  librarianship supports the notion that such  paradigms do exist and allow us to wisely  transform our every act of service and  innovation. fostering awareness of the  scientific paradigms we espouse, both  within library science curricula and as we  publish, allows our status as scientists to  become evident and valued.       comparison of training  interventions for pubmed search skills  amongst 3rd and 4th year medical students.   glenda myers, d litt et phil, witwatersrand  health sciences library, university of the  witwatersrand, south africa.     objectives: to determine if the use of a  mesh tutorial had a positive effect on  online search skills in pubmed training  programs for medical students.   setting: the study took place in a large  medical school in a developing country. the  students are enrolled in a hybrid problem‐ based learning (pbl) curriculum. internet  bandwidth is problematic, so few training  interventions can be conducted “live”. much  use is therefore made of power point screen  shots for demonstration, and students are  encouraged to conduct voluntary individual  hands‐on sessions online.   population: two hundred and fourteen  third year medical students (year 1 of the  graduate program) were introduced to  pubmed searching through four formal  lectures at the beginning of 2006. two  hundred and eight fourth year medical  students (year 2 of the graduate program)  were introduced to pubmed searching  through small group learning sessions at the  beginning of 2005.   intervention: both groups of students  received a power point refresher course for  self‐study before completing an osce  (objective structured clinical examination) in  june/july 2006. this training course  demonstrated the use of mesh, but students  could opt for text‐word searches as well.  the pubmed search station in the case of  both sets of students was one of several  osce stations for clinical skills learnt  during that block. in the second half of the  year the pubmed training intervention  changed to a facilitated clinical skills  seminar, with voluntary attendance, where  the use of mesh was stressed, and the  “links” option to import the mesh term into  the pubmed search screen was  demonstrated. a copy of this training  program was loaded onto the students’  web‐based curriculum page for further self  study. once again both groups of students  were examined by means of an osce  station in november/december 2006.   results: preliminary results show improved  osce test scores for the second intervention  (use of the mesh “links” option) for both  groups of students. final examination  results will be available at the beginning of  evidence based library and information practice 2007, 2:2  23 the academic year in january 2007. once the  final test scores are received, it will also be  possible to compare the performance of  students who received formal lectures with  students who received small group training.  conclusions: the use of the mesh “links”  option as a training intervention led to an  increased osce score over the use of the  mesh demonstration and ability to perform  a free text word search. test scores may  have been influenced by the presence of a  librarian in a facilitated clinical skills  seminar, as opposed to a voluntary self‐ study power point tutorial. final year osce  scores may also have been influenced by the  increased familiarity of students with  pubmed searching by the end of the  academic year. analysis of the final scores  should indicate whether these variables had  a significant effect on the mesh “links”  intervention.        students’ patterns of library use and  their learning outcomes.  haruki nagata,  graduate school of library, information  and media studies, university of tsukuba,  akira toda, bunkyo university shonan  library, japan, & päivi kytömäki, oulu  university library, finland.     objectives: to examine the relationship  between students’ patterns of library use  and their learning outcomes.   design: analysis of focus group interviews  and exit gate surveys of students’ library use  and attainments   setting: this study was conducted in four  academic libraries in japan and finland  between 2004 and 2006. all the libraries  belong to universities, one of which is a  private institution and three are nationally  funded. the majority of participants in this  research are undergraduate students.   methods: in initial stage two focus group  interviews were conducted in mie  university library (japan) in 2004. the  survey for this study was designed using  data derived from the findings of our  previous research (toda & nagata:  students’ library use and learning  outcomes, 2007 (in process)), which  demonstrated a positive correlativity  between students’ library use and their  learning outcomes. the survey consisted of  10 questions, 9 fixed response and 1 free text.  6 of 9 fixed questions use the likert scale,  asking about students’ motivation, library  use and attainment. the first survey was  conducted at keio university shonan‐  fujisawa campus library (japan) in 2005. in  addition we conducted two focus group  interviews of the survey respondents. the  results were impressive as they revealed  unexpected data, thus further research was  necessary. in 2005 we conducted two more  focus group interviews and then in 2006 the  survey was conducted in oulu university  library (finland). in addition, another  survey was conducted in nagoya university  library (japan) in 2006. the survey  questionnaire was almost identical for these  three surveys. main results ‐ cluster  analysis revealed 3 common patterns of  student usage: learners, socialisers and  students merely wanting somewhere to sit.  in addition there were students who simply  wanted to borrow books or use the pc’s.  students were found to gain many and  varied learning outcomes from their use of  the library. these included: academic  knowledge, general learning, new  perspectives, emotional fulfillment,  competence in finding information, critical  thinking, the habit of learning by oneself,  and the fun of studying. these differences  among students groups are a significant key  to understanding how students reach their  learning outcomes. it is not easy to show in  a direct manner how students achieve their  learning outcomes by using libraries. this  study has tried to examine it by examining  the students’ use of library and finding out  how it relates to their own learning  outcomes, which could open new avenues  evidence based library and information practice 2007, 2:2  24 to measure the impact of libraries in  universities.       using rubrics to collect evidence  for decision‐making: what do librarians  need to learn?  megan oakleaf, phd,  syracuse university, united states.     academic librarians who conduct  information literacy instruction often  encounter evidence of student learning in a  variety of forms. to improve instructional  programs, academic librarians need to  transform evidence of student learning into  data for decision‐making. educators use  rubrics to transform complex evidence into  data that is easy to understand, but  academic librarians are slow to adopt  rubrics as a form of information literacy  instruction assessment. rubrics not only  provide evidence for decision‐making but  also offer significant instructional benefits  including: allowing students to understand  their instructors’ expectations, making  scores more meaningful, providing detailed  feedback, and facilitating self‐evaluation.   librarians benefit from rubrics because they  offer the opportunity to discuss and agree  upon learning values, provide tools for  consistent scoring, and deliver data full of  rich description. for these reasons, research  is merited in this area. this study explores  the question: “to what extent can librarians  use rubrics to assess student learning?”  additionally, it documents college students’  ability to evaluate websites and librarians’  use of assessment evidence to make  improvements to online instruction. using  survey design methodology, this study  compares academic reference librarians’  ability to use rubrics with college instructors  and students. librarian participants in this  study included 5 reference librarians, 5  college english instructors, and 5 college  sophomores at one southeastern research  extensive university. these participants  received training in rubric assessment of  student work.   the study also included 10 reference  librarians at research libraries nationwide  who did not participate in rubric training.  each of the 25 participants scored 75  artifacts of student learning using a rubric.  these artifacts of student learning were  randomly selected from open‐ended  responses to questions about evaluating  websites included in an online information  literacy tutorial. while some librarians were  able to use rubrics to provide reliable and  valid assessments of students’ information  literacy skills, many could not. in fact, the  study reveals six barriers that must be  addressed before librarians can successfully  wield rubrics as tools for evidence‐based  decision‐making: 1) difficulty  understanding outcomes‐based assessment,  2) tension between holistic and analytic  rubrics, 3) failure to comprehend rubric  content, 4) disagreement with rubric  assumptions, 5) difficulties with evidence of  student learning, and 6) difficulties  understanding campus context and culture.   fortunately, the study appears to indicate  that librarians can use rubrics developed  within an individual campus context and  culture to translate complex evidence of  student learning into reliable and valid data  for improving instructional programs.  however, if librarians hope to achieve the  benefits of rubric assessment, they require  additional training to overcome the barriers  identified in this study. this paper  presentation will describe what librarians  need to learn to use rubrics effectively for  evidence‐based decision‐making.        bringing evidence to practice: a team  approach to teaching skills required for an  informationist role in evidence‐based  clinical and public health practice.   evidence based library and information practice 2007, 2:2  25 kathleen b. oliver, msls, mph & prudence  dalrymple, phd, johns hopkins university,  united states.     program objective: 1) to develop an  academic graduate level course designed for  information professionals seeking to bring  evidence to clinical medicine and public  health practice with its implied “real world”  time constraints, 2) to further specify and  realize identified elements of the  informationist concept.  setting: johns hopkins university, school  of medicine, division of health sciences  informatics.  participants: a multi‐disciplinary faculty  selected for their expertise in the course core  competencies, and students enrolled in the  course. the students included two post‐ graduate nlm informationist fellows, and  one nlm second year associate.  intervention(s): a combination of lectures  and laboratory exercises taught the skills  involved in finding, analyzing, and  delivering evidence in clinical and public  health decision making with implied time  constraints. students presented evidence to  support decision making in case scenarios  drawn from clinical or public health practice.  skills covered in the course included the  identification of a question embedded in a  case presentation, development of effective  search strategies for relevant evidence to  address the question, evaluation and  synthesis of the identified evidence, and  effective presentation of evidence. the  learning objectives were: 1) to understand  and demonstrate the evidence‐based  answering cycle which includes the ability  to define a typology of questions; 2) to  describe the answer process and identify a  question domain; 3) to learn to search,  screen and evaluate evidence to support a  query; 4) to demonstrate appropriate skills  for presenting evidence‐based data, and 5)  to demonstrate team membership and  participation in a framework defined by  “real world” time constraints evaluations  were based on participation in class, exercise  completion, and final oral and written  presentations of assigned cases.   main results: 1) students successfully  completed all laboratory exercises and final  oral and written presentations of evidence  for assigned, real world cases. 2) a course  syllabus and exercises were completed. 3)  student exercises and presentations were  evaluated by the clinical and public health  practitioners who developed case scenarios  addressed by students in their exercises and  final presentations.   conclusion: a graduate level, pilot course  taught by a multidisciplinary faculty team  further developed the informationist  concept and taught the skills required for an  informationist role in bringing evidence to  clinical and public health practice. the  course design demonstrated a novel  approach to teaching the skills identified in  the literature as important to the  informationist concept. student follow‐up is  needed to establish the success of the course,  and to confirm whether the course  objectives were effective and at the right  level. if validated, the approach is one that  could be replicated in other academic health  centers with similar pools of expertise.  further research is needed to establish an  appropriate or “good enough” standard of  evidence short of a systematic review that  meets “real world” time constraints.  ongoing “journal clubs” that reiterate the  process with new questions derived from  clinical and public health practice and  incorporate peer review and faculty  mentoring would reinforce skills acquired in  the seminar.        the practitioner’s experience and  conception of evidence based library and  information practice: an exploratory  analysis.  helen partridge, queensland  university of technology, clare glanville,  ba graddiplis gcerthed aalia,  evidence based library and information practice 2007, 2:2  26 university of southern queensland, sylvia  edwards, dip.lib., (rmit), gced(he),  mit(res.), phd, aalia, macs, school of  information systems, queensland  university of technology, & gillian hallam,  associate professor, school of information  systems, faculty of information technology,  australia.     question: this paper will consider the  question: how do practitioners’ experience  and conceive evidence based library and  information practice (eblip)?   setting: all participants will take part in a  30–60 minute audio recorded interview. the  goal of each interview will be to understand  the variation in each participant’s  experience of eblip. the questions will be  designed to orient participants towards their  experiences of eblip and will be necessarily  broad. this is to ensure the participant’s  experiences are allowed to emerge without  being confined or otherwise influenced by  the researcher’s views. data collection is  currently taking place; the full list of  questions used in the interview process will  be outlined in the paper on completion of  the study.   participants: the research is being  conducted as a pilot study. to this end only  10 to 20 participants will take part in the  study. participants will be library and  information practitioners from brisbane.  full participant details will be available after  data collection.   methods: phenomenography provides the  theoretical basis for the study’s research  method. it is an interpretive research  approach that looks at the different ways  people experience or conceive a range of  phenomenon. the intent of  phenomenographic research is to  understand variation in the collective  experience of the participants in regards a  particular phenomenon (in this instance  eblip). the strength of phenomenography  as a research tool rests in its capacity to  uncover variation in conception, awareness,  understanding or experience. that is,  phenomenography will help tease out the  full spectrum of experiences or conceptions  of eblip by practitioners; and not just the  “average” or “typical” experience or  conception. the phenomenographic  approach makes it possible to focus, not on  the participants per se, nor on the eblip,  but rather on the interrelation between these  two; that is, the practitioners varying  experiences of, and conceptions of, eblip.   main findings: the intent of the analysis is  to identify the variation in the experience of  eblip by library and information  practitioners. during data analysis the  interviews will be pooled so that interviews  are not tied to individual participants. the  analysis can thus proceed with a focus on  the phenomenon under study and not the  individual. the content of the interviews are  examined with specific attention given to  the similarities and differences between  what was said about eblip. it is then  possible to (i) seek meaning – that is,  determine the discernable variations in  ways of experiencing eblip; and (ii) seek  structure – that is determine the critical  elements and dimensions of the variations  associated with each meaning. the meaning  and structure of various experiences are  then divided into separate categories of  description. together the categories of  description represent each way of  experiencing eblip. the aim in analysis is  to describe the phenomenon of research  interest through a limited number of  descriptive categories, usually between 4  and 7. data analysis will take place in late  february/early march. conclusions: whilst  a full discussion on the conclusions is not  possible until the study is completed (early  march) it must be noted that the study is  significant because it will provide the first  model of eblip as understood by the library  and information practitioner. this model  will assist library educators, associations  and others involved in supporting,  preparing and educating current and future  evidence based library and information practice 2007, 2:2  27 evidence based professionals, so that an  evidence based culture can be firmly  established within the profession.       finding our foundation: analysis  of the lisa database for research  retrievability.  carol perryman, ms lis,  doctoral fellow, school of information &  library science, university of north  carolina at chapel hill, united states.     objective: the primary objective of this  study is to examine the library and  information science abstracts (lisa)  database to determine if research literature  can consistently be retrieved by using  keywords identifying the research  methodologies used.   question: for the journals named, are  articles identified as ‘research’ able to be  consistently retrieved by using keywords  related to research methodology?   methods: citations from the top 10 library  and information science journals for 2001 as  identified by koufogiannakis, slater &  crumley (2004) were obtained, then a filter  developed by catherine beverley (2004) was  used independently to identify research  articles. the resulting sets of citations were  compared, then the two datasets were  analyzed in order to consider retrievability  and fit for assigned keywords from the lisa  database.   results: although it would need to be  tested against a random set of citations  rather than the purposive sample tested  here, our analysis suggests that retrieval  using the descriptor terms alone may  succeed in only 31.5% ± 5.2% of attempts,  with a 95% confidence interval.   conclusions: the lisa thesaurus is not  consistent or sufficiently comprehensive to  serve the needs of researchers.  recommendations for the improved  retrievability of lis research literature from  the database are made.   transforming the library: applying  multiple assessment methodologies to  library instruction and planning.  sue f.  phelps, mls, & karen r. diller, mils,  washington state university vancouver,  united states.     question: the library at washington state  university vancouver will be remodeled  and expanded within the next five years  because of the changes in student  population and a rapidly growing campus.  the library faculty and staff ask how  instruction, services, and space must change  to improve student learning and grow with  the expanding campus. setting: the wsu  vancouver campus has gone from an upper  division and graduate only campus to a  four‐year campus and is expecting a thirty  percent increase in students from 2005‐2007.   participants: the participants in the study  were faculty, staff and students from the  wsuv campus.   methods: a literature search was conducted  prior to the project to see what other  colleges and universities were doing to  assess library services and space on their  campuses. much of what we found was that  libqual is the current standard, but libqual  did not address all of the issues we were  facing. we designed our study to address  these issues using new and innovative, as  well as more standard, methodologies.  student learning was assessed through the  new general education program built for  our first freshman class and incoming  transfer students. several months were  spent doing research on how assessment  was being done at other institutions and in  looking for software programs that would  include assessment. a portfolio program  was found that would allow assessment of  the campus learning outcomes, including  information literacy, to be built into the  program. students show progress through  evidence and reflection on that evidence,  which is assessed by a group of faculty  evidence based library and information practice 2007, 2:2  28 using a standard rubric. three  methodologies were used to assess services  and space. first, information was gathered  from all employees of the information  services department (library, information  technology, and videoconferencing) in a  nominal group process session at their fall  retreat about the future needs of the campus.  the result was an abundance of expert  qualitative data. the participants then  analyzed the data and prioritized the  outcomes to set library goals. the second  and most formal methodology provided a  more traditional approach to assessment.  focus groups of each user population were  used for exploratory data assessment. the  issues raised by the focus groups were used  to develop a survey that was distributed  electronically to all staff, faculty and  students. results were statistically analyzed  to help evaluate current services and space  and to inform future programs and space  planning. the final methodology to assess  services and space involved taking  photographs of students working alone or in  groups across campus in order to assess the  way students use space, equipment and  furniture. those photographs were  evaluated in an informal observational  study to assist in the library’s renovation  and expansion plans. main findings: this  study is currently in progress. analysis and  conclusions are expected to be completed by  march 2007.   conclusions: the areas in which we will  draw conclusions include student learning  in information literacy, the need for specific  types of space within the library and patron  satisfaction with current services, both in  person and at a distance.       why librarians leave: a proposal to  study the factors that influence librarians  to exit the profession.  susan rathbun‐ grubb, doctoral student, school of  information & library science, university of  north carolina at chapel hill, united states.     question: what are the internal and  external factors that influence librarians to  leave the profession, and who is most likely  to leave?   context: given the amount of pre‐ employment preparation for a career in  librarianship and the resources that libraries  spend to develop early‐career professionals,  the failure to retain librarians in the  profession is a great waste to the librarians  who leave and the institutions that lose  them. a better understanding of the reasons  why librarians leave could guide library  administrators in creating and improving  retention programs and may be able to help  library and information science (lis)  educators design programmatic or  curricular strategies to address these issues  before the students enter the workforce.  setting: this qualitative research project  will be a pilot study to determine the design  and feasibility of a larger main study to be  conducted afterward. participants will be  recruited using a snowball sampling  technique until a sample of 10‐15 persons is  formed. for the purposes of the study, a  former librarian is someone who holds a  masters degree in library science, who has  worked as a professional librarian in a  public, academic, school, or special library,  and who no longer works in a library setting.  design: the researcher will conduct  individual semi‐structured interviews with  each participant. questions will be open‐ ended, and designed to provoke lengthy,  reflective answers. interviews will be sound  recorded, transcribed, coded, and analyzed.  after the pilot study is completed, the entire  study design will be examined for problems  such as scope, imprecision of definitions,  underlying weaknesses in assumptions or  theoretical underpinnings, problematic  questions or questioning techniques, and  quality of coding structure. the research  questions will be refined, and the semi‐ evidence based library and information practice 2007, 2:2  29 structured interview questions will be  revised and finalized. design of the  interview questions and analysis of the  answers will be done after an examination  of the research and professional literature on  voluntary turnover and retention, and by  using the life course perspective. this  sociological framework serves as a method  for understanding the interaction of  historical, generational, geographical, and  societal forces that impact a person’s life  choices and “pathways” throughout a  lifetime. a bird’s‐eye view of a person’s  entire career path, situated in its historical,  geographical, and cultural context creates a  richer field of data for a researcher to mine.   results: it is hoped that completed research  in this area will enhance the knowledge base  of lis educators, professional librarians, and  library administrators and through them,  eventually improve the “quality of life” in  the profession. the words of librarians who  have left may inform lis educators who can  create realistic expectations of the workplace  for future practitioners.       electronic document delivery (edd)  turnaround time and its connection to  patient care in the hospital.  karen l. roth,  mba, mls, ahip, morton plant mease  health care, & thomas w. hill, medical  library, self regional healthcare, united  states.     purpose: to describe a small research study  that connects edd turnaround time to the  hospital’s mission of excellent clinical  patient care based on the abels et al. (2002,  journal of the medical library association,  90:276‐284 and 2004, journal of the medical  library association, 92: 46‐55) preliminary  taxonomy on the value of health sciences  library services.   setting/subjects: hospital libraries in the  ese/a (e‐delivery southeastern atlantic)  region docline® group. docline® is  an electronic document delivery program  for use by medical libraries in the united  states and canada administered by the  national library of medicine. within  docline®, there are reciprocal lending  groups for document delivery.   methodology: the research study involved  3 phases. phase 1 entailed extracting  docline® data from 2 hospital libraries to  determine a baseline turnaround time for  edd. phase 2 replicated and validated  phase 1 results by extrapolating the same  data from 25 libraries in the ese/a group.  phase 3 correlated the data from phase 2  into the value of hospital library services.  utilizing an online email‐linked survey  (surveymonkey), library customers were  asked if rapid turnaround time was of value  to them.   results/outcomes: survey results from  phase 3 showed that hospital library  customers valued the rapid delivery time of  edd and used the materials for both patient  care and education/teaching. using the  taxonomy provided by abels et al., the  correlation between edd and patient care  indicates that hospital library services are  consistent with the hospitals’ mission for  clinical care.   discussion/conclusion: outcomes research  is valid and necessary for hospital librarians  to demonstrate their value to their  organizations. using available data and  survey tools, the hospital librarian can show  library services contribute to the hospital’s  mission of excellent clinical care.       evidence‐based practice in us fire  library management.  lian ruan, mls,  illinois fire service institute & linda c.  smith, phd, graduate school of library and  information science, university of illinois at  urbana‐ champaign, united states.     question: this study was conducted to  determine how special libraries serving fire  evidence based library and information practice 2007, 2:2  30 professionals are used, valued, if they are  efficiently organized to give maximum  access to their resource collections, and the  impact such libraries have on information  use by fire professionals in their decision‐ making.   design: the study used the chicago,  rochester, and sla study instruments as  the basis for questionnaires to measure the  impact of library‐supplied information on  practical decision making and applied  research in the fire service sector. data were  collected at six sites over a seven‐month  period, february‐ august 2004.   setting: six fire libraries that represent  significant contributors of information  services to fire professionals (including fire  service personnel as first responders and  researchers) in the united states  participated in the study.   participants: 343 fire professionals  evaluated the impact of the information  received in response to a request for some  information from their special library  related to a recent decision‐making situation.  respondents held a variety of positions,  including firefighters (27%), training officers  (19%), fire chiefs (13%), researchers (17%),  administrative staff (13%), and other (11%).   main results: major findings from  completed surveys include that 97% of the  fire service personnel and researchers  (n=343) said that the information received  was relevant to their work. 97% reported  that the information was of practical value  and 81% said that the information was of  research value. the special libraries were  effective in supplying information in  decision‐making situations because 94% of  respondents reported the information they  received led to better‐informed decisions,  and in increasing their level of confidence in  those decisions (89%). areas of impact  included training, research, emergency  response, changed procedures and policies,  budget decisions, and personnel  management.   conclusion: the study demonstrates the  vital and integrated roles fire service  libraries can play in their organizations with  information services having a direct impact  on users’ training, research, and emergency  response activities. the overall study results  will assist librarians in many different  settings who can benefit from greater  awareness of information needs related to  public safety and homeland security and  strategies for addressing them. based on the  results of the study, special librarians can  develop services that target particular types  of impacts, as well as improving the level of  impact in various areas. the study also  clearly indicates that fire libraries need to  consider different approaches to evaluating  their services, and the library profession  needs to pay more attention to professional  development in this area. future research  can include non‐users, virtual users  accessing services via a library’s web site,  and others involved in public safety, such as  emergency medical responders. other  studies can examine and identify skills,  attributes and subject knowledge for  librarians and information professionals  working in fire emergency services and  homeland security.        distance education: a good model for  professional continuing education. connie  schardt, mls, medical center library, duke  university & julie garrison, mls, central  michigan university, united states.     while a number of studies have compared  online and face‐to‐face learning, there is still  little evidence about how effective the  online environment is for continuing  education and professional development  types of activities. drawing upon earlier  research, this paper explores the  effectiveness of a web‐based professional  continuing education course, compared  with an equivalent face‐to‐face version  evidence based library and information practice 2007, 2:2  31 designed to teach practicing medical  librarians how to participate in and  advocate for evidence based medicine at  their individual institutions. two cohorts of  practicing librarians were self‐selected to  participate in either the distance education  8‐week course or the 8‐hour peer‐to‐peer  class. the authors compared student pre‐ class, post‐class, and 6‐month post‐class  assessments of knowledge retention to  evaluate student learning and the  effectiveness of the course delivery methods.        designing an evidence‐based model  to create a core title collection. ada seltzer,  msls, ahip, fmla, university of  mississippi medical center, united states.     purpose: the objective of this paper is to  describe the evidence‐based methodology  used to develop a new annual selection  guide for the health sciences.   setting: the methodology combines the  subjective evaluation of subject authorities  with quantitative measures that generate  data and knowledge used for the decision  making process. a series of web‐based tools  developed by doody enterprises link,  tabulate and distill selections, assessments,  scores and “essential purchase”  recommendations. participants: a volunteer  17 member library board of advisors and  doody’s staff designed the evidence‐based  model and continue to evaluate and  improve the methodology.   method: the model combines different  levels of expert assessments from content  specialists and library selectors to establish a  small subset of titles that represent the  essential knowledge for a specific health  sciences specialty. this subset is evaluated  by library selectors on five key collection  development criteria using a numeric rating  system. scores by multiple selectors are  averaged for each title and used to identify  titles for the guide. a separate evaluation is  performed to determine the “essential  purchase title” (ept) status, defined as titles  recommended for purchase by a library  with an annual bookbuying budget of less  than $7,500.   main results: the methodology identifies  and determines the titles that comprise the  selection guide (doody’s core titles)  representing the best of the literature in 121  health sciences specialties recommended for  a library collection. three editions have been  published since 2004 and each edition  represents a different and separate  application of the model.   evaluation/conclusions: the model has  been successful in producing a core list that  is authoritative, comprehensive, timely and  reliable. ratings bring a separate set of  values to the evaluation and decision  making process and may help reduce the  bias inherent in the subjective aspects of the  evaluation. the methodology applies the  traditional collection development criteria in  new ways for evidence‐based selection  decisions. more study is needed to  determine the discriminating power of the  3‐point response scale in making  meaningful comparisons of titles within  specialties, the impact on collection  development in health sciences libraries,  and the success of the measures in  identifying the highest quality core titles.       understanding patron needs and  resource usage in the networked academy.   win shih, mslis, msba, denison memorial  library, university of colorado at denver  and health sciences center, united states.     objectives: the purpose of this study was  to better understand various aspects of  patrons’ access to library’s electronic  resources and services via its web site, their  motivation, and the resources they used.   brief description: as library resources and  services continue to evolve rapidly and  evidence based library and information practice 2007, 2:2  32 ever‐more‐sophisticatedly along  technological vectors, the imperative  persists for librarians to develop an  innovative, systemic methodology for  fathoming the effectiveness in meeting the  demands and necessities of our less‐visible,  yet omnipresent and “omni‐diverse”  patrons, as well as for promoting efficient  resource management and capacity  planning. methods: traditional, web‐ transactional log analyses possess a raft of  merits, including unobtrusive and automatic  logging, minimal overhead costs, and  relatively simple data analysis procedures.  however, such transaction logs cannot  provide important user demographic data  (e.g., metrics such as “academic status –  faculty/student/staff,” “departmental  affiliation,” etc.), let alone the patron’s  purpose (psychographic data) for utilizing a  library’s resources in the first place (i.e.,  “teaching,” “research,” “course work,”  “patient care,” and so forth); nor why a  specific resource is being selected. these  types of much‐more‐specific information  have the potential to assist libraries in  infinitely more useful ways for better  understanding their patrons, a patron’s  information seeking behavior, and the  multifaceted interrelationship between  patrons’ academic disciplines and the  library resources and facilities available to  assist them. in 2006, denison memorial  library of the university of colorado at  denver and health sciences center (http://  denison.uchsc.edu) conducted a year‐long,  library‐usage study to better understand  patron information needs and their  information‐seeking behaviors. based on the  association of research libraries’ “mines  for libraries” methodology, we asked  patrons visiting our web portal to complete  a brief, single‐page online survey before  they connected to their destination url. in  addition to resources consulted or chosen by  patrons, data was collected on  demographics, current locale of connectivity,  along with “purpose of library portal  utilization.” combined with unobtrusive  web‐access logging and vendor supplied  usage statistics, we are able to employ this  valid statistical and factual evidence for  collection development, resource allocation,  budget and funding requests, system  performance, and it capacity planning.   results/outcome: our presentation  therefore will provide findings of our study,  its implications, combined with  comparisons and contrasts to the results  from web transaction log analyses and  statistics garnered by serials solutions, our  e‐journal portal vendor. furthermore, we  will discuss how such exercise offers the  opportunity to assimilate evidence‐based  information practice and knowledge  management skills into our daily operation.       improving online access to electronic  journals.  steven j. squires, msls, margaret  e. moore, amls, mph, & susan h. keesee,  msls, unc health sciences library, united  states.     research question and outcome: what  percentage of articles valuable to unc  faculty, students, and staff are readily  available and accessible online? for those  found to be unavailable or inaccessible,  what are the reasons? knowing the relative  extent of access problems will allow hsl to  make informed decisions regarding best use  of limited library resources to improve  online access to electronic journals.  significance: in 2004, the hsl participated  in a service quality survey, libqual+,  designed and implemented nationally by  the association of research libraries. the  results indicated that online access to  electronic journals does not meet the  expectations of unc faculty, students, and  staff. this problem is one of the top two  problems identified for hsl and other  libraries across the country. accessing  needed knowledge resources is vital to the  http://denison.uchsc.edu http://denison.uchsc.edu evidence based library and information practice 2007, 2:2  33 success of unc faculty, staff, and students  in fulfilling the universityʹs mission.  findings will be useful to other libraries that  need to address the  same issues. the study extends classic  availability studies focusing on print  materials into the online environment.  setting/subjects: citations valuable to a)  students in health affairs courses, b)  researchers, and c) clinicians. citation sets  consisted of journal articles authored or  cited by unc‐ch authors, or chosen by  unc educators for course‐work, and  representing core constituency groups.  methodology: observational study and  analysis. availability of selected citations is  determined by repeated searches over time,  reasons for failure are compiled and  categorized, and means of ameliorating  failures assessed.  results: of 400 journal articles, 22% were  found to be unavailable. most were not  available because the library did not  subscribe to the journal or the journal article  dated ealier than available online from the  publisher. only six articles that should have  been  available were not available as a result of  linking failures. sixty‐seven percent of  unavailable articles were published before  2000.  our hypothesis that a small number of  reasons would account for most journal  availability failures was confirmed.  a  significant number of failures are amenable  to corrective action through more  acquisitions. having found the extent of our  availability problem and its causes, the  library can better move on to study question  of accessibility, or the user experience  navigating to available resources. these may  include problems with authentication when  using off‐campus computers, with browser  configurations for ez proxy, with confusing  aspects of website presentations, with  cataloging notations and directions, and  with our sometimes multiple alternate paths   to full‐text.       a comparative study of book and  journal use in the social sciences. allison m.  sutton, mlis, & joann jacoby, mslis, ma  anthropology, education & social science  library, university of illinois, united states.     academic librarians are challenged with  development and maintenance of collections  of research materials in various formats.  though it is largely assumed librarians are  surveying their constituency as a part of the  collection management decision‐making  process, the literature reflects only a small  number of studies that focus on direct  solicitation of user opinions. this project  included a 22 question survey of faculty and  graduate students in a cross‐section of social  science disciplines (anthropology,  psychology, social work, and sociology)  which was completed in 2005.   the objective is to gather data about book  and journal use that can be used to inform  collection development priorities and  discern shifting patterns of use in an  increasingly hybrid (print and electronic)  information environment. the analysis  phase of the project is currently in process.  we anticipate that our analysis will shed  light on a number of broader issues and  questions, including: what are the  prevailing collection access issues? how  might they help us to manage the continued  shift from print to electronic format? can  the findings from this survey aid in the  development of a more accurate assessment  of the collection informed by current  research interests and priorities? is book &  journal use aligned with any shifts in the  disciplines that are tied to certain subfields?  likewise, is use aligned with structural  changes within departments, divisions or to  the curriculum? how do faculty and  graduate students differ in their use of the     evidence based library and information practice 2007, 2:2  34 collection? what similarities and differences   in book & journal use are apparent across  disciplines? what specific books & journal  considered ‘most important’ by social  science faculty & graduate students fall  within or outside of the scope of the  collection? what importance might our  constituency attach to having books and  journals featuring materials by and about  classic or contemporary theorists in their  areas of interest?   we will also discuss how the input received  has helped shape our ongoing efforts to  build a collection of books, journals and  other items which are most useful to our  faculty, students and researchers. the  survey and its results targeting assessment  of user needs will serve as a model for best  practices in collection development for  academic librarians.       library leaders discuss the pros and  cons of evidence‐based practice. patricia l.  thibodeau, duke university medical center  library, jean wilkins, illinois state library  (retired), susan k. nutter, north carolina  state university, lian ruan, china  programs and the illinois fire service  institute (ifsi), university of illinois at  urbana‐champaign, & sandra hughes‐ hassell, school of information and library  science, university of north carolina at  chapel hill, united states.    we put the question to five library leaders:  does ebp work in your setting? each has  been directly concerned with practice based   decision making through the course of their  career, and offers their unique perspective  and experience in a lively discussion..       information access survey at st.  martin’s college, england, april 2005: an  investigation into the non‐use of library  services by current students.  lisa toner,  ba (hons), dip lib, harold bridges library,  united kingdom.     question: why do some students at st  martin’s college make little or no use of the  library services provided by the institution  and what are the reasons why these  students do not engage with the printed  material available to them?   setting: st martin’s college is a church of  england higher education institution  located in the north west of england with  11,274 students (6839 fte). 5918 are on part  time courses and of those 181 are on courses  taught at a distance. the college is multi‐ site having three main campuses in  lancaster, ambleside and carlisle. courses  are also provided from barrow, whitehaven,  newcastle, tower hamlets and woolwich.  the portfolio of courses taught at st  martin’s centres on professional  programmes within the areas of health and  education.   methodology: the research project was  conducted over a three month period  between march and may 2005. 2189  questionnaires were posted to students who  qualified as low or non‐ users according to  the library database, talis. low use was  defined as students who had borrowed  three items or less in a year. this amounted  to 21% of total student numbers. as  nonusers were deemed to be a difficult  target audience to contact, a further 800  questionnaires were distributed to  classrooms. the results were collated using  the spss statistical software package. the  project returned a 28% response rate. over  half of those were based at the satellite sites  and were, in fact users of those libraries.  this data was not initially known as these  sites do not use talis. therefore the actual  numbers of non users was reduced to just  13.94%.   summary of the findings: the carlisle  campus has the highest incidence of non‐use  the faculty of health and social care has  evidence based library and information practice 2007, 2:2  35 both the highest incidence of non‐use  overall and the highest incidence of non‐use  at the lancaster site but of those over half  do use other university or nhs libraries  through reciprocal access schemes such as  uk libraries plus. 78% of the non‐users in  the college were female 95% were mature  students 75% were on part time or distance  learning courses ‘distance from the library’  was the single biggest factor cited by  respondents (68%) for non‐use many  students were not aware of some of the  services on offer such as postal loans and  web services only a third of non‐users made  use of electronic resources fewer than 10%  accessed the e‐books collection 88% claimed  to use the internet as a source for their  assignments 70% of non‐users said that they  bought their own copies of books and a  further 44% borrowed from their friends  first year students represent the largest  group of nonusers with second year and  diploma students the second biggest groups   conclusion: the main groups of students  who don’t use traditional library services  are those who may feel on the fringes of the  college due to their part time or distance  mode of learning. even then, a third of those  are making use of library services nearer to  their home or place of work. this research  has analyzed some inconsistencies in the  promotion of library services to groups of  users and we will be working to redress the  balance to provide a more equitable service  to all our customers.       utilizing citation analysis to assess  the coverage of the abstracting & indexing  databases available via the digital desktop  library for usda (digitop).  matthew  treskon, national agricultural library,  united states.     question: how closely do four abstracting  & indexing (a&i) databases (scopus, biosis,  cab, agricola) cover the perceived need of  researchers at the agricultural research  service (ars), an agency of the united  states department of agriculture (usda)?  setting: the national agricultural library  (nal) is one of four national libraries. it  houses one of the world’s largest and most  accessible agricultural information  collections and serves as the nexus for a  national network of state land‐grant and  usda field libraries. the motto of the  library: advancing access to global  information for agriculture. ars is the  usda’s chief scientific research agency.  digitop, digital desktop library for usda,  delivers quality information resources and is  managed by nal with financial support  from multiple usda agencies, including  ars.   participants: according to the sampling  method detailed below, all usda ars  research published in 2005 is represented in  this study. it is assumed that this represents  the intellectual effort and output of the  agency as a whole.   methods: we used the author‐based metric  of citation analysis to identify ars  researcher need, extracting a representative  sample of works cited from ars  publications in 2005. this sample was then  checked against the four a&i databases  mentioned above (as well as digitop’s  collection of 2,000+ fulltext journals) for  availability, utilizing a strict manual  checking procedure designed to uniformly  and fairly search the four databases. main  findings: we decided that the following  means of analyzing the collected data would  provide useful insight into the ways the  databases covered the needs of usda ars  researcher need: overall coverage of works  cited, overall similarity to the sampled  works set, coverage by format, coverage by  year, uniqueness of content, and  complementary coverage.   conclusions: according to this study,  usda‐ars researchers can find either the  citations/abstracts or the full text to the  material needed 90% of the time. regarding  evidence based library and information practice 2007, 2:2  36 the specific a&i services, this study found  that biosis and scopus had significantly  higher rates of overall coverage than  agricola or cab, even though the agricola  and cab databases were found to be  considerably more similar to the  demonstrated patron need than biosis or  scopus. even though material published  before 2000 was still highly cited, analysis  by year reveals that a high percentage of  works cited were published 2000 to current,  with a peak at 2002‐ 2003. although  agricola’s overall coverage rates were not as  exhaustive as biosis or scopus, analysis by  format revealed that agricola’s coverage  rates were the highest for proceedings,  chapters, books, and second highest for  other. this study finds that agricola and  scopus contain more unique content than  cab or biosis. likewise, the a&i pair of  agricola and biosis had the highest level of  complementary coverage, followed by  agricola and scopus. this suggests that  agricola’s content complements these  commercial services, and that the  development of the database should  continue to focus on unique content.        knowledge and skills required to  provide virtual reference services (vrs):  evidence from three studies.  feili tu, phd,  & nancy p. zimmerman, phd, school of  library and information science, university  of south carolina, united states.     question: what knowledge and skills are  required to provide vrs to the public (both  adults and youth) and to health‐related  professionals, and what challenges do  library and information science (lis)  educators face in producing graduates  qualified to deliver these types of vrs?   design: three survey studies were  conducted in 2002, 2004, and 2005. both  qualitative and quantitative methods were  used to analyze the data.   settings: 2002 study: 1) academic, public,  and secondary school libraries providing  vrs; 2) vrs systems (e.g., qanda café).  2004 study: 1) public and school libraries  with vrs available for youth; 2) vrs  systems with youth services (e.g., qanda  café). 2005 study: academic  biomedical/health sciences libraries, hospital  libraries, special bio‐health‐related libraries,  and vrs systems with health‐related  services (e.g., asknow). participants:  librarians experienced in providing both e‐ mail reference services and chat‐based vrs.  purposive sampling was used to recruit  participants. 2002 study: 18 reference service  providers and administrators of reference  departments in academic, public, and high  school libraries, as well as in vrs systems.  2004 study: 16 reference service providers  and administrators of reference departments  in public and school libraries, and in vrs  systems with youth services, plus several  academic librarians providing vrs to  youths. 2005 study: 21 information  professionals providing vrs in health‐ related information settings or associated  with vrs systems with health‐related  services. intervention(s): reference curricula  in the schools of library and information  sciences (slis) at san josé state university  and the university of south carolina (usc)  have been revamped. vrs instruction has  been integrated into reference courses. a  continuing education (ce) course on vrs is  in place at the slis/usc. based on the  results of the 2005 study, a course on  electronic health information retrieval has  been created for delivery in spring 2007.   main outcome measure(s): four major  indicators are: 1) knowledge required, 2)  resources used in answering questions, 3)  essential skills, and 4) lis education and  training for the provision of vrs.   main results: in the three studies, 98% of  the subjects rated knowledge of subject  analysis, formulation of search strategies,  and problem‐solving and of theoretical  framework of reference interviews and  evidence based library and information practice 2007, 2:2  37 transactions most important. significant  skills were information retrieval, online  searching, interpersonal communication in  online environments, conducting a reference  interview, and selecting and accessing  electronic information systems. of these,  98% of the subjects reported that  problemsolving and information retrieval  skills are most essential. overall, training for  vrs practice is not adequate. 67% of the  subjects in 2005 study reported  dissatisfaction with their lis education.   conclusion: knowledge of resources in  various formats (especially electronic  databases) in general and specific subject  areas is critical in providing vrs. lis  curricula must be revamped through  integration of new technology into reference  and management curricula. further  investigation of the following is needed:  should ce courses on vrs be created in  various formats? how effective are vrs, e.g.,  live homework help, for youth? how does  live homework help influence library  services for youth? how should school  media specialists and youth service  librarians provide vrs appropriately?  should library instruction programs and/or  user educations programs be created and  provided for vrs users?        using a prompt sheet to improve the  reference interview in a health telephone  helpline service.  christine urquhart, phd,  university of wales aberystwyth & toni  price, national institute for health and  clinical excellence, united kingdom.     question: does a prompt sheet improve the  reference interview process for health  information advisers working at nhs direct,  a 24 hour telephone helpline in the uk?  design: a randomised control trial.   setting: eight nhs direct sites across  england. nhs direct provides a  confidential health care advice service to the  public, handling 6‐7 million calls per year.  around 20% of the calls are routed to health  information advisers (80% to nurse advisers).   participants: newly recruited health  information advisers (n=30), full and part‐ time, were randomly allocated to a control  group (n=15) or intervention group (n=15),  and 26 completed.   intervention: attitudes of the national staff  group of health information advisers to four  possible improvements to the reference  interview were assessed through a  questionnaire survey (response rate 88/268,  33%). as a prompt sheet was the second  (and inexpensive) choice, a prompt sheet  was designed and piloted with six  experienced health information advisers.  the prompt sheet included prompts for  demographic information, reason for call,  condition/treatment plan, existing  knowledge of caller, special needs of the  caller, conclusion, handling a call  empathetically. the final version was trialed  in three waves over several months, with  random allocation of new recruits. testing  of reference interview expertise was done at  the end of basic training, and two months  later, using the same ten test questions that  were based on common questions received  by nhs direct. a framework of possible  responses was drawn up for each question  (highly relevant, potentially relevant and  minimally relevant), and three points were  awarded for each highly relevant response,  one for each potentially relevant response  and a minus one point for each response of  minimal relevance. if the response was not  already listed it was considered on its merits  and categorised accordingly. the total  possible score for each of the 10 questions  varied.   main outcome measures: average group  score for each question, baseline and after  two months, individual scores for each  question, baseline and after two months,  and the change (improvement).   main results: the average score of prompt  and non‐prompt (control) participants  evidence based library and information practice 2007, 2:2  38 increased on the second test, for each of the  10 questions. the prompt group improved  more than the control group for 8 of the 10  questions, and this increase was due to a  larger number of highly relevant items  noted. of the 16 health information advisers  who improved their net scores, 10 had the  prompt sheet available. 15 health  information advisers (9 prompt, 6 control)  improved on five or more questions. tests  were conducted to assess whether previous  experience as a call handler or site might  affect the degree of improvement, or  whether full time advisers were more likely  to improve than part‐time advisers.   conclusion: the trial appeared to indicate  the benefit of a prompt sheet for some,  though not all, newly recruited health  information advisers to improve their  reference interview technique. it was,  however, impossible to assess objectively  the degree of compliance with the prompt  sheet by the intervention group, or possible  contamination of control and intervention  participants.       evidence based librarianship in the  social sciences: a new project of the  social science libraries section of the  international federation of library  associations (ifla).  steve witt, ms,  university of illinois, center for global  studies, united states.     the literature on evidence based practice  within libraries documents extensively the  tradition of this form of inquiry to support  professional development and improved  services within health science libraries.  much of this work is focused upon the use  of scientific and quantitative methodologies.  often absent in the discussion is the use of  evidence based practice in social science  libraries and the role of qualitative social  science research methodologies as a means  of evidence based inquiry. to address this,  the ifla social science libraries section  (ssls) is researching both the extent of  evidence‐based practice’s use within social  science libraries and the ways in which  qualitative social science methodologies  such as action research and case studies are  being used for evidence based inquiry  within libraries. through this two‐year  project, the ssls will publish of a white  paper that will contain (1) a literature  review on evidence‐based librarianship as it  is defined and practiced within social  science libraries; (2) results of a survey of  social science librarians within academic  and special libraries on the extent of usage  of evidence‐based practice and  methodologies employed; and (3) a series of  case studies that demonstrate best‐practices.   the project will formally begin at the ifla  conference in durban, south africa, where  the ssls program will focus on case studies  of evidence based inquiry within social  science libraries. in 2008, the ssls will  present research data and further case  studies at the ifla annual conference in  quebec, canada. at this time, the ssls will  also solicit further papers for publication in  an ifla monograph. this paper will present  the ifla project in detail, analyze the  current view of evidence based research  within qualitative research community,  provide a typology of qualitative based  methodologies used within the social  sciences, and examples of qualitative  applications of evidence based inquiry  within libraries.       posters    library support of evidence‐based  practice in health policy.  min‐lin fang,  mlis, the library and center for  knowledge management, & janet coffman,  ma, mpp, phd, institute for health policy  studies, university of california, san  francisco, united states.   evidence based library and information practice 2007, 2:2  39 objective: this presentation describes the  significant contributions of medical  librarians supporting evidence‐based  practice for the california state legislature.  it also discusses the development of  guidelines and procedures to work  efficiently in a team from multiple  university of california (uc) campuses.  additionally, the process and challenges of  literature review are detailed.    setting:  the library and center for  knowledge management at university  of california, san francisco (ucsf) is a large  academic health sciences library, supporting  a campus dedicated only to graduate and  professional study in the health sciences.   methods: the california health benefits  review program (chbrp) was established  in 2002 to respond to requests from the  california state legislature to provide  independent analyses of the medical,  financial and public health impacts of  proposed health insurance benefit  mandates. the chbrp formed teams with  an interdisciplinary approach. each team  consists of chbrp staff from the uc office  of president, clinical faculty, researchers  and medical librarians from multiple uc  campuses and consultants from an actuarial  consulting firm.  they have a wide range of  expertise, including clinical medicine, health  policy, public health, health economics  and actuarial analysis. medical librarians at  the uc san francisco, uc san diego, uc  davis and uc irvine libraries conducted  thorough evidence‐based literature searches  on specific health benefits mandated by the  proposed bills. the team members write  reports on medical effectiveness and cost  and public health impacts of proposed  bills based on evidence‐based medical  literature. each report summarizes  sound scientific evidence relevant to the  proposed bill, deferring policy decision‐ making to the legislature. the evidence‐ based report is independent and invaluable  to the legislature in its policy decision  making.   results: the chbrp program has been very  successful despite the challenges of time  constraints and the unpredictability of when  requests from the legislature will be  received. it has provided an excellent venue  for librarians to work with faculty and  researchers in multi disciplines  from multiple campuses. not only has it  presented a new opportunity for librarians  to contribute their expertise to shape the  health care policy, but it can serve as a new  model for bringing evidence‐based practice  into health policy decisions for legislatures  in other states.     descriptive research literature review of  instruction for business students in  academic libraries: best practices or new  directions? ann manning fiegen, mls,  california state university, san marcos,  united states.    question: what does the professional  literature reveal about information literacy  instruction for business students?  a  descriptive literature review of 25 years  should reveal evidence toward preferred  instructional practices that lead to increased  information competencies of business  students.  this study is a descriptive  synthesis of the research literature in  business instruction in academic libraries.  data sources: library and business  education bibliographic databases were  searched for english language publications  from 1980 through 2006.   databases  searched were ebscohost premier, emerald,  eric, library literature and information  science, lisa, proquest inform global, the  web of science and publication reference  lists.    study selection: each index was searched  for the terms: library and business and  (instruct* or literac* or assess*) and  (academic or higher education or college or  university). one hundred and eighty articles  evidence based library and information practice 2007, 2:2  40 form the study population. criteria for  practical and methodological screens are  review, research objectives, research design,  carnegie classification, sample population,  results and recommendations for further  research.  publications about professional  development or training librarians are  excluded.  the poster will also exhibit the  social science research methods used for the  study and describe how bibliographic  citation software was used to extract criteria  for the research review.   data extraction method: each qualified  publication undergoes a qualitative review,  is coded, and entered into a customized  endnote bibliographic record designed to  extract criteria that will address this study’s  objectives.  a subset is identified that  contain quantitative data for future meta‐ analysis.   main results: qualified articles were  categorized and analyzed according to  research objectives, methods, results and  recommendations.  preliminary results of  qualified articles indicate research objectives  include integration of new technologies,  educational pedagogy or professional  standards applied to business instruction,  and descriptive best practices.  the majority  of research methods employed is the case  study, and results are reported through a  wide variety of outcome measures that will  be described.  recommendations in the  articles generally describe limitations of the  study or suggest replication in other settings.   only 24 of the publications contain  quantitative data with potential for meta‐ analysis.  twenty‐eight have been discarded  by the practical screen.  conclusion: does the literature reflect a  trend toward preferred instructional  practice and increased information literacy?   the poster exhibit will synthesize results,  inform current practice and stimulate  research among business librarians about  future emphasis and research for  information literacy instruction.       assessing the need to update  prevention guidelines: a comparison of  two methods. gerald gartlehner, md, mph,  suzanne l. west, md, mph, kathleen n.  lohr, phd, mphil, ma, leila kahwati, md,  mph, jana g. johnson, md, russell p.  harris, md, mph, lynn whitener, msls,  drph, christiane e. voisin, mlis, sonya  sutton, bsph, unc cecil g. sheps center  for health services research, united states.    purpose: the high quality of the literature  search and the methodological rigor brought  to the analysis of that literature are essential  to sound evidence reports and any practice  guidelines that might emanate from them.  the scientific standards for this part of the  development of evidence tables and reports  continue to rise. key elements now include  clear ʺclinical pathwaysʺ to point to  significant patient care questions for the  evidence report, well‐crafted  inclusion/exclusion criteria for the literature  search with due attention to gray literature  and unpublished materials, and  standardized methods for grading the  quality of the evidence. because of the high  costs associated with updating preventive  guidelines, we wanted to compare two  literature review approaches to determine  which of these most reliably assessed the  extent of updating required while  maximizing cost‐effectiveness.   setting/subjects: the rti‐unc evidence  based practice center (epc) produces  systematic reviews and analyses of the  scientific evidence on a variety of health care  and health policy topics. epc personnel  prepare ahrq reports and technology  assessments as well as reports for public and  private health care organizations, design  and conduct customized literature searches  and quantitative analyses, and research the  best practices and methods for conducting  reviews of the scientific literature.  methodology: the main objective of this  study was to compare the shekelle et al.  evidence based library and information practice 2007, 2:2  41 assessment model (review approach) with a  conventional approach using a typical  systematic review method (traditional  approach) considered to be the gold  standard, with respect to  comprehensiveness of results and effort  expended.  two teams, consisting of two  clinicians, one health services researcher and  one librarian, each used one of these  methods to independently assess the need to  update six topics from the 1996 guide to  clinical preventive services from the u.s.  preventive services task force. the review  approach was iteratively refined over three  phases, using an experiential technique and  2 topics per phase, to achieve greater  efficiency. we compared the articles  identified by each approach to evaluate their  relevance with regard to the critical key  questions and determined whether the need  to update based on the review approach  matched the need to update from the  traditional approach. outcomes included  completeness of study identification,  importance of missed studies and the effort  expended by each team.  results: although the modified review  approach identified fewer eligible studies  than the traditional approach, none of the  studies missed was rated as important by  task force members acting as liaisons to the  project with respect to whether the topic  required an update. on average, the  modified review approach produced  substantially fewer citations to review than  the traditional approach. the effort involved  and potential time saving depended largely  on the scope of the topic but we would  conclude that the review approach provides  a valid, robust alternative to the traditional  approach.  discussion/conclusion: the modified  review approach provides an efficient and  acceptable method for judging whether a  guideline requires updating.      staying ahead of the curve: a continuous  and systematic approach to evaluating  electronic resources.  karen stanley grigg,  msls, & charlie spencer lackey, mls,  mba, ahip, duke university medical  center library.    objectives: how can the duke university  medical center library (dumcl) create a  systematic approach to regular, ongoing  evaluation of current e‐resources that  includes user feedback and expertise across  library departmental staff to improve  purchasing and implementation of e‐ resources?   methods: our population of interest  consists of a collaborative, inter‐ departmental team of  dumcl librarians,  faculty, staff and students. in response to a  ever‐increasing renewal prices of crucial  electronic databases, the library has  developed a systematic approach to  reviewing uptodate and comparable  products, and plan to apply this approach to  all of our e‐resources on a regular  basis.  this poster session will outline steps  involved and the outcome of our  review.  the steps include evaluating use  and feedback of current products, scanning  the marketplace for competing products,  assembling a team of reviewers, securing  the trials, obtaining price quotes, creating  review criteria, dividing teams into sub‐ teams that evaluate each database using the  same criteria, reassembling and comparing  databases as a larger group, promoting trials  and soliciting feedback from users, and  creating final report to guide our decisions.    conclusion: librarians involved with the  review process concluded that this process  is useful to justify subscription decisions,  even if the decision is made to continue on  the current course.  librarians also find that  an ongoing, systematic review of databases  engenders better knowledge of the  marketplace. dumcl’s approach to  database evaluation, along with results of  our uptodate evaluation will be included.   evidence based library and information practice 2007, 2:2  42 plans to measure effectiveness of evaluation  of e‐resources using a follow up survey  and/or focus groups of library users will be  discussed.      network analysis of campus libraries used  by faculty and graduate students. joann  jacoby, mlis, ma in anthropology,  university of illinois at urbana‐champaign,  united states.    purpose: the purpose of this study was to  gain insight into how the current  departmental library structure is used by  faculty in various disciplines, evaluate how  this structure might be changed to better  serve their needs, and to map the  connections between departmental libraries  from the users’ perspective.  setting/participants/methodology: using  data mined from a large‐scale survey  distributed to all faculty, academic  professionals and staff at a large publicly‐ funded research university in spring 2006,  this study uses social network and cluster  analysis to examine the inter‐relatedness of  departmental libraries from the users’  perspective.    results/outcomes: all departmental  libraries had strong connections to central,  interdisciplinary service points including  the main stacks and central reference.  the  science libraries form a semi‐independent,  strongly interconnected cluster, while the  social science libraries also form a node with  strong connections amongst themselves as  well as to science and humanities libraries.   the humanities libraries formed a structure  similar to that of the social sciences, but area  studies were more isolated (though the  latter may be and effect of low response rate  from this constituency).  these findings  have been used to inform planning for a  proposed scholarly commons and for  ongoing decisions regarding the closing and  merger of smaller departmental libraries.  discussion/conclusion: the network maps  and cluster trees reveal connections among  disciplines, both expected and unexpected,  and provide a data visualization tool that  has helped inform library decision‐making.      stat searches: what do they reveal about  the use of information by healthcare  professionals and library services? michele  klein‐fedyshin, msls, health sciences  library system, university of pittsburgh,  upmc shadyside, united states.    objectives: to study the occurrence of  “stat” searches in a hospital‐based library  to understand the motivation and  completion of urgent mediated search  requests. in an era of end‐user searching,  what do mediated stat requests reveal?  when our clients need us emergently, why  is it urgent, how do they request and  retrieve it, and what resources do we use to  respond? how does a hospital library  support the use of information and evidence  by healthcare professionals?  method: “stat” searches were defined as  those needed within the day. if necessary,  staff worked late to complete them. we  reviewed the last two+ years’ searches (july  2004 through november 2006) submitted by  healthcare professionals. each form has date  submitted and date needed lines. these  were used to cull searches requested  immediately in a retrospective review.  the  words “stat”, “now”, “today” or just the  same date in both boxes defined the period.  routine requests, even if for direct patient  care, were excluded. over 200 searches  qualified. data examined included patron  type, purpose, requesting method, delivery  method, causation, format of retrieval and  databases searched.  preliminary results: almost 10% of search  requests of healthcare professionals required  immediate turnaround. two‐thirds of these  answered patient care problems and one‐ third were considered educational or  evidence based library and information practice 2007, 2:2  43 informational. multiple databases were  accessed for information with up to four  sources being consulted for individual  topics with an average of two databases per  question. ovid medline was used most  frequently, followed by pubmed, with up to  date, embase, mdconsult and web of  science almost tied for third.   the urgency resulted from a variety of  reasons with unusual patient course being  the most frequent, surgery/procedural  queries causing the second largest set,  complications necessitating the third  group,  drug questions making up the fourth and  other therapy questions prompting the 5th  set.    clinicians phoned in most requests and  picked up the majority in person. about  60% of the answers included articles plus  abstracts and another 31% included articles  and text from another source.  conclusions: although stat searches  formed a small portion of the professional  reference component, they filled an  immediate need. the urgency of the need  coupled with time constraints may have  precipitated these requests.  several distinct  causes precipitated the urgent need, and  may define when searching is indicated.  answering these queries with articles was  possible through the extensive electronic  journal, and databases resources of the  affiliated health sciences library system.  since document delivery takes an average of  4 days in a hospital library, the importance  of the online collection surfaces. librarians  typically accessed several databases to  answer the question, which reflected the  complexity of these patients, and the time‐ intensive search required.   the library staff responds annually to  hundreds of mediated search requests and  thousands of quick reference queries. yet  none may have been more important than  these stat searches.        supporting evidence‐based medicine  in an academic health system: a unique  partnership between the center for  evidenced‐based practice and the  biomedical library. gretchen kuntz, msw,  mslis, clinical liaison librarian,  biomedical library, anne seymour, interim  director, health sciences libraries, craig a.  umscheid md, co‐director, center for  evidence‐based practice, & kendal williams  md, director, center for evidence‐based  practice, university of pennsylvania health  system, united states.    purpose: to describe the development of a  partnership between the university library  and the university health systemʹs center  for evidence‐based practice (cep). this  partnership supports enhancing health care  quality and patient safety within the  university health system through the  practice of evidence‐based medicine, which  includes medical students, residents, fellows,  librarians, faculty and staff.  setting/participants: the biomedical  library is an academic library in an urban  setting. the center for evidence‐based  practice (cep) is part of the university  affiliated multi‐hospital health system.  brief description: the center for evidence‐ based practice (cep) was created in an  academic health system to support patient  care quality and safety through the use of  evidence‐based medicine.  when a clinical  issue arises that requires an evaluation of a  drug, device or process of care, cep staff  will perform a systematic review of the issue  alongside the interested physicians to  produce a guideline or recommendation for  action for the health system.  from the  inception of the center, the cep directors  invited the biomedical library to partner  with them to achieve this goal.    the clinical liaison librarian is an active  collaborator with the cep directors and staff,  engaging in weekly team meetings and  project planning, conducting systematic  evidence based library and information practice 2007, 2:2  44 literature searches, and developing a  comprehensive training program for  literature searching which encompasses  standard ebp skills which is required for all  center staff as well as students, residents  and fellows who choose to participate in the  centerʹs evidence‐based medicine elective  program. additionally, the librarian set‐up  group accounts for the center in various  databases and bibliographic software so that  all participants have access to and can  contribute to saved search strategies/results  and selected citations for specific projects.   results/outcome: the partnership between  cep and the biomedical library  commenced in august, 2006 but even in that  short span has resulted in collaboration on  five on‐going projects involving 37 searches  in multiple databases and the training of 2  residents who opted for the cep elective  evaluation: our partnership is a work in  progress and currently evaluation is based  upon feedback from all of the participants.   one plan is to incorporate into the cep  electiveʹs electronic course evaluation  software, a section that will include the  library‐training portion of the elective. the  cep directors are working on developing  broader evaluation methodologies, which  will encompass not only the operational  functions but also the clinical outcomes  resulting from the guidelines.       libraries using evidence – eblip.net.au : a  top view from down under. suzanne  lewis, phd  ba(hons) grad dip arts (lib  inf sci) aalia manager, central coast  sector libraries, northern sydney central  coast health, australia., lisa cotter, school  librarian, nursing and midwifery, the  university of newcastle, united kingdom.    introduction: eblip4 coincides with the  ten‐year anniversary of the term “evidence‐ based librarianship”. in 1997 this concept  began its ascent from a discussion among  likeminded library professionals into the  published literature. it has continued to rise  through the profession to occupy a place in  the mainstream and afford librarians an  enhanced view of the landscape in which  they make decisions.  similarly, down under in sydney, australia,  people have, for 75 years, crossed the  sydney harbour bridge the traditional way  – along the eight‐lane carriageway. around  the same time that jonathan eldredge’s  hypothesis article introduced “evidence‐ based librarianship” , another pioneer was  establishing bridgeclimb – an exciting new  challenge with a fantastic view from the top,  climbing the sydney harbour bridge via the  steel arch. today, eblip practitioners are  like the bridgeclimbers – ready to seek new  perspectives, take a different route and  enjoy the journey as well as achieving the  destination.  adventurous bridgeclimbers ascend/cross  the steel arch with confidence, supported by  a well engineered structure and strong  foundations. libraries using evidence –  eblip.net.au offers a similarly solid  foundation of authoritative resources to  support eblip practitioners as they seek  different perspectives on their professional  practice.  objective: to develop a practical toolkit to  contribute to the evidence‐based library and  information practice (eblip) body of  knowledge and support application of  eblip by practitioners worldwide. the  toolkit will be developed with an eblip  approach, and will incorporate web  2.0/library 2.0 techniques where  appropriate.   method: a review of the literature was  conducted to identify barriers to the  application of evidence‐based practice, and  find any existing toolkits already addressing  such obstacles. lack of time and difficulties  locating and identifying eblip resources  were identified as major barriers. the toolkit  site was designed with the aim of gathering  together in one place high‐quality resources  to support eblip, thus saving librarians  evidence based library and information practice 2007, 2:2  45 time otherwise spent searching for resources.  an advisory group with international  representation was formed to guide  development of the toolkit. once the site  was built it was promoted and feedback was  invited. the site was further developed and  resources added in response to feedback  received. content contributions such as  podcasts, reports of current projects and  links to related sites were encouraged from  librarians worldwide.   results: libraries using evidence –  eblip.net.au was launched in october 2006  and reported in evidence based library and  information practice (1:3, pp.98‐100). an  international advisory group was  established and continues to oversee  development of the site. the site includes a  toolkit of core eblip resources such as  appraisal checklists, sources of evidence and  guidelines for evaluation, for the first time  providing eblip practitioners worldwide  with a single access point to these essential  tools.  libraries using evidence – eblip.net.au is  working with evidence based library and  information practice to help make evidence  summaries more accessible. the toolkit  indexes all of the journal’s evidence  summaries by domain of librarianship, and  provides an rss feed for each domain  enabling readers to monitor new summaries  in a particular area. the journal’s homepage  provides a link directly to this section of the  toolkit.  rss feeds are also maintained announcing  recent articles of interest, and libraries  using evidence – eblip.net.au site updates.  a number of high‐profile eblip advocates  contributed podcasts introducing sections of  the eblip toolkit.  libraries using evidence – eblip.net.au  continues to support the vision of eblip as  a global movement by facilitating the  exchange of ideas within the eblip  community. the site encourages  participation via links to a blog, facebook  group, and discussion lists. the toolkit is  listed as a useful resource by the recent  folio course, eblip‐gloss (http://eblip‐ gloss.pbwiki.com).  conclusion: the site will continue to be  developed and promoted, with feedback  encouraged. the 4th international evidence  based library and information practice  conference will provide a venue for  promotion and feedback. further  opportunities to include international  library community participation using web  2.0/library 2.0 techniques and technologies  will be investigated.    where are they now? a study of careers of  library and information science graduates.  joanne gard marshall, school of  information and library science, & unc  institute on aging; jennifer craft morgan,  unc institute on aging; heidi madden,  duke university library; cheryl a.  thompson, unc institute on aging; susan  rathbun‐grubb, school of information and  library science, & unc institute on aging,  united states.   purpose: workforce issues in library and  information science (wilis) is a three‐year  research project designed to study the career  patterns of graduates of lis programs. in  general, lis programs have lacked the time  and resources to track their graduates. as a  result, stakeholders do not have an adequate  understanding of what happens to lis  graduates over the long term. lis educators,  in particular, do not have consistent data on  the extent to which their programs meet  students’ expectations, prepare them for the  workplace or meet continuing learning  needs.  such an understanding would assist  in educating and managing the lis  workforce more effectively.  the purpose of  the initial phase of the wilis project is to  build an in‐depth understanding of  educational, workplace, career and retention  http://eblip%e2%80%90gloss.pbwiki.com%00 http://eblip%e2%80%90gloss.pbwiki.com%00 http://eblip%e2%80%90gloss.pbwiki.com%00 evidence based library and information practice 2007, 2:2  46 issues faced by lis graduates using the life  course perspective from the social sciences.    setting/subjects: 8000 graduates (1964‐ 2005) of lis programs in north carolina  were selected. one of the programs is in a  historically black institution, allowing a  focus on minority career and retention  issues.   methodology:  in march 2007, a pilot  sample of 750 lis graduates from five lis  programs in the state of north carolina  were asked to complete a web‐based survey.  the entire sample will receive the survey in  august 2007.   results: this poster presents data providing  the rationale for such a study, as well as the  multiple methodologies that will be used to  determine the career trajectories of lis  graduates.    discussion/conclusion: this study will  compare groups of interest such as  older/younger workers, gender, library/non‐ library employment and stayers/leavers in  the lis field. we anticipate that these  findings will inform educators, employers,  policy makers and other stakeholders who  are concerned with lis workforce issues.       a place for us: the relevancy of the  academic library “as a place.”  jason martin, mls, edd in progress,  university of central florida, united states.    are academic libraries “as a place” still a  relevant part of campus life? academic  libraries have faced a plethora of changes  over the past 15 years. online information  has become more available, reliable, and  sophisticated. college majors have shifted  from the book‐reliant areas of the  humanities and social sciences to the  subjects more dependent on journal and  online resources such as the science and  business fields. due to these changes, less  and less students and faculty actually need  to enter the physical space of the library to  find information and perform research.  today’s college students do however want a  place where they can study in groups, snack,  and have access to the latest computer  technology, which few libraries allow them  to do. added together these factors have  created a first for libraries across college  campuses; they are seriously facing the  threat of becoming irrelevant as a place in  the world of academia. in the summer of  2006 i surveyed library directors from small  and medium sized academic institutions to  gather their opinions on the current and  future state of the academic library as a  place. i then sent the same survey to library  directors of large academic institutions.  while their responses indicated that the  academic library as a place is still relevant  and needed, the results were not as strong  as expected. most of these directors realize  that libraries now compete with coffee shops,  bookstores, and computer labs on campus  for student’s attention. students no longer  simply come to the library because it is the  library. they will patronize places that meet  their demands. directors are starting to  change their libraries to meet the needs of  their students. simple things like adding  comfortable seating, relaxing food and drink  policies, and adding artwork made the  library more appealing to students. more  involved library fixes include adding coffee  shops, adding more computers, remodeling  the library space and even removing books  and changing the way libraries perform  collection development and their mission as  a whole. academic libraries as a place are  still relevant, but changes need to be made if  academic libraries want to take a central  place in higher education. this poster  session will highlight some of these ideas  and present results from the surveys that  highlight the thinking of current library  directors on this topic.      using ebl practices in strategic planning:  a case study of methods employed in a  evidence based library and information practice 2007, 2:2  47 library planning process. erik mitchell,  mlis, wake forest university, united states.    this poster will discuss how to use ebl  practices in relation to academic library  strategic planning processes.  a portion of  the poster will be devoted to presenting the  methods and resources used during a recent  environmental scan at the z. smith  reynolds library at wake forest university.   the remainder of the poster will review  applications of ebl and specific resources  available for use in strategic planning  situations for libraries.  the poster will focus  on using methods for defining relevant  questions, finding evidence through  literature review, local research, and  external/national research, and deriving  themes and trends from the gathered data.   the poster will underscore the need for the  use of a grounded approach when  beginning a planning process.       testing a new approach to the manual  literature search process  in systematic reviews.  laura c. morgan,  ba, gerald gartlehner, md, mph, dan  jonas, md, curtis bryant, & claire de la  varre, msis, unc cecil g. sheps center for  health services research, united states.    purpose: the objective of this project is to  determine the accuracy, reliability, and  efficiency of utilizing the scopus database  (“scopus approach”) as an alternative to  traditional manual literature searches  (“traditional approach”) for  systematic  reviews.  the main goal is to determine  whether scopus allows more efficient use of  resources without compromising the  process in terms of bias (i.e., missing studies  that should be considered for the systematic  review) than the traditional approach.  to  achieve this goal, we will compare both  approaches during a systematic review on  the comparative efficacy and safety of  constipation drugs, commissioned by the  drug effectiveness review project.  setting: the rti‐unc evidence based  practice center (epc) produces systematic  reviews and analyses of the scientific  evidence on a variety of health care and  health policy topics. epc personnel prepare  ahrq reports and technology assessments  as well as reports for public and private  health care organizations, design and  conduct customized literature searches and  quantitative analyses, and research the best  practices and methods for conducting  reviews of the scientific literature.     methodology: we selected 20 recent and  relevant studies, review articles, and  editorials as publications appropriate for  manual literature searches.  we will conduct  both methods independently to determine  two key outcomes: (1) the time resources  required for the traditional method vs. the  scopus method; and (2) the accuracy of the  traditional method vs. the scopus method.  during the “traditional approach,” two  researches will examine the reference list of  each article, identifying any citations  relevant to the topic of the systematic review.  the database manager will then manually  check each identified citation to determine  whether it has been included in the project’s  database through electronic literature  searches. during the “scopus approach,” the  database manager will take the relevant  background articles and download the  reference list of each article from the scopus  database. these downloaded files will then  be uploaded into a test database for the  project to determine immediately which  citations have already been included in the  project database. only new citations will be  reviewed dually by researchers.    results and conclusions: this methods  study is currently in progress.  if accepted,  we will present final results and conclusions  in may.      evidence based library and information practice 2007, 2:2  48 reference transaction records as evidence  of reference librarianʹs competencies.  mitsuhiro oda, department of education,  japan, aoyama gakuin university, tokyo,  & norihisa yoda, the national diet library,  kyoto, japan.    the collaborative reference database  project started in 2002. this project has been  hosted by the national diet library of japan,  and managed collaboratively by growing  number of various libraries of all over the  country; 286 public libraries, 110 academic  libraries, and 35 special libraries. its  database consists of 4 sub databases;   ‐ ʺdatabase on reference questions and  answersʺ(drqa), which contains the  records of reference activities of the member  libraries  ‐ ʺdatabase on reference guidesʺ, which  contains the information on how to search  information sources on specific themes and  topics  ‐ ʺdatabase on guides for special  collectionsʺ, which gives information about  collections on specific themes or on specific  kinds of library materials of member  libraries  ‐ ʺdatabase on profiles of the member  librariesʺ, which contains the information  on member libraries of the collaborative  reference database project  the authors have analyzed the drqa  which has accumulated over 20,000 data  through out the project.. the drqa  contains transaction records of reference  services which the participatory libraries  were inquired and answered. the drqa  has two distinctive features. at first, as the  transaction records are “edited and  compiled” by reference librarians after  finishing their practice, so they are the  evidence of reference librarian’s recognition  toward the transaction. secondly, this  database is the very network system of both  libraries and librarians on the evidence of  reference services.   each record is consisted of 23 elements, and  among these, “the answer” and “the process  of answer” are much important for the  analysis, because these two items contains  information about the process of the service.  the authors can recognize the professional  knowledge and skills through the  description on the search process for answer,  the selected resources, the query words for  information search, and so on. especially,  some well‐described records imply the  decision making on each step of the  librarians, and we extract the adapted  competencies of the librarians as  professionals.  there have been abundant arguments about  competencies of reference librarians;  however, most of them are argued only  based on experiences. this presentation is  intended to objectively define the  competencies of reference librarians by  using the transaction records of reference  services as their evidence. the authors also  make an assertion that the research on this  assignment has a great potential to be  utilized for assessment and evaluation of  reference services, design and development  of training programs for reference librarians,  and so on.       assessing the future: developing  information technology skills for new staff  roles in academic medical libraries. carol  perryman, mslis, trln doctoral fellow,  university of chapel hill school of  information & library science, robert james,  associate director of access services,  charlie lackey, assistant director of  cataloging and bibliographic services, &  rick peterson, deputy director, duke  university medical center library, durham,  nc, united states.    purpose: the primary objective of this study  is to assess the information technology (it)  competencies of staff at the medical center  evidence based library and information practice 2007, 2:2  49 library in order to plan for expanded roles in  the future.  setting/subjects: the medical center  library serves as the primary information  resource for the university’s medical center  and health system.   with 40.25 fte staff  and a rapidly shifting demand for  collaborative and educational spaces and  services, the medical center library is  poised for transition to a knowledge  management center.     methodology: first, a literature review was  performed in order to determine whether  other libraries had performed a similar  inventory.  technology activities derived  from a review of all job descriptions were  conceptually grouped to build a core set of  competencies and an expanded,  comprehensive listing for use as a  survey.  after pilot testing with one  department, the survey was then refined  and administered to all staff.  finally, the  results were assessed and used to support  training programs intended to address  deficits between present and future needs.  results: the it assessment and survey  instruments work well to quantify present‐ day it skills and skill deficits.  results of the  survey aid in planning for future training,  and enable more accurate position  descriptions.    discussion/conclusion: an awareness of  staff it knowledge and practice is crucial to  planning for the future of this and other  medical libraries.  in addition to addressing  position description discrepancies, focused  it training supports the career  development needs of all staff. ultimately,  we feel that having a more accurate idea of  staff it skills will serve to strengthen the  ability of library staff to collaborate with  medical center stakeholders in building and  providing support for the new knowledge  management center.      evidence‐based practice: a revolution in  library project management. carol  perryman, mslis, trln doctoral fellow,  university of chapel hill school of  information & library science, united states,  & patricia l. thibodeau, associate dean for  library services and archives, medical  center library, duke university, durham,  nc durham, nc, united states.    objective: health sciences libraries face  increasingly complex issues and projects,  but planning templates derived from the  business world have not always been  successfully integrated in library settings.  evidence‐based library practices enable  libraries to take a fresh look at the planning  process. our objective was to devise a  scaleable project planning template which  encourages the use of evidence‐based  methods.  methods: existing project management  tools from two health sciences libraries and  archives were reviewed, followed by a  survey of the library and general business  project management literature. this process  helped to identify a set of ʹcommon  denominatorʹ processes used in project  planning, which became the baseline coding  set for documentary analysis of two prior  and one current library projects. information  needs expressed in task force minutes,  emails, and other documentation created  during the planning phase were identified,  along with the sources consulted in finding  answers. findings were verified by means of  retrospective, open‐ended interviews with  planning task force members, and by  participant observation in the current  project. a particular focus was the use and  evaluation of decision support resources  and methods by task force participants. this  research forms the basis for the formation of  a new template for project management.  results and conclusions: results of library  project analysis revealed that while staff are  actively engaged in creative and purposeful  research, the current project planning  structure led to the following problems: use  of external information including published  evidence based library and information practice 2007, 2:2  50 literature was equally limited; project  documentation did not encourage follow‐up  evaluation or dissemination; and  reconstruction of projects was difficult due  to insufficient documentation and human  memory instability. a staff planning retreat  underscored the need for a formal planning  structure including teamwork guidelines  and a project template.  a new project template has been  implemented which includes key elements  of evidence‐based practice along with core  components of traditional planning.  elements include searching for evidence to  answer important questions; evaluation of  existing documentation for relevance and  usefulness using an included checklist; and  dissemination and benchmarking planning  built into the process. the template will be  further tested with work groups in practice.       is “igaku toshokan” providing the  evidence to help medical librarians with  decision‐making? yukiko sakai, msis, keio  university, & yasuhiko kiyama, juntendo  university library, japan.    objective: although evidence‐based library  and information practice is still new in  japan, since the late 1990s medical librarians  have been expected to take a leading role in  evidence‐based medicine. they are  encouraged to become research‐oriented as  a way to support scientific research for  health professionals, as well as do research  for their own practice.  “igaku toshokan”, an official journal of the  japan medical library association, is one of  the most popular lis journals among  japanese medical librarians. contributors to  the journal are mainly medical librarians.  we conducted a study to answer the  question, “is igaku toshokan now  providing the evidence to help medical  librarians with decision‐making?” by  reviewing the research‐based articles, the  study types and the domains of the articles  in the journal.  method: we collected articles from the  journal that were published between 2000  and 2006, and excluded peripheral items  such as editorials, book reviews and news  items. 598 articles were examined as to  whether or not it was research‐based  following peritz’s definition. the study type  and domains were further identified using  content analysis methodology. the study  type was borrowed from williams’s study  on the top five lis journals in 2003. the  domains were decided based on the  categories suggested in “platform for  change”, the educational statement  published by the u.s. medical library  association in 1992.  main findings: 132 articles, for an average  of 18 articles per year (22%), were identified  as research‐based articles. the proportion  was similar to an earlier study on a  canadian professional journal, while it was  lower than previous studies on purposefully  selected lis journals. however, since 2004  the number of research articles has  remained high ‐‐ over 23 per year ‐‐ and the  overall proportion marked a record high at  30% in 2006.   case studies were the most frequently used  method (56%), followed by reviews (22%),  bibliometrics (11%), surveys (5%) and others.  the tendency toward descriptive studies ‐‐  including case studies ‐‐ was also showed,  while survey was more heavily used in the  other studies on selected lis journals.   the major domains covered in the research  articles are: information services (43%),  resources (16%), instructional support  services (14%) and resource management  (14%). articles that dealt with management  issues were very few (1%). case studies  were heavily applied in most domains  except for domains related to resources in  which bibliometrics (48%) and review  (29%) were frequently applied.   conclusion: “igaku toshokan” has  published a number of research articles in  evidence based library and information practice 2007, 2:2  51 recent years, yet has been criticized by  librarians for failing to provide enough  evidence for medical librarians’ decision  making through a research methodology  stronger than case studies. the majority of  case studies among service domains may  reflect the need for practical lessons from  colleagues; reviews from foreign language  materials are always welcomed in japan.  however, in order to implement stronger  evidence‐based library and information  practices, our conclusion is that the journal  should further evolve by inviting more  medical librarians as authors, and encourage  them to use stronger, established scientific  research methods over more social‐science  based methods.      computing assistants pilot program  evaluation. laura sheble, mlis, doctoral  student, university of north carolina,  chapel hill school of information and  library science, & thomas b. wall, phd,  associate university librarian, director of  public services, duke university libraries,  united states.    context: in the information commons  service model, academic libraries are  providing augmented services to more fully  support a broader spectrum of the learning  and intellectual and creative work life cycles.   a vital component of this augmented  service model is the provision of computing  support services.    objective: to measure use of computing  support services provided by student office  of information technology staff during  evening operating hours over an eleven  week pilot program period to guide the  development of enduring computing  support service programs.  methods: case study: service transaction  logs, reflective diary entries, and a pilot‐ program end focus group were used to  collect data on multiple levels.  auxiliary  data collected through a student library user  focus group and extant library data  collection programs provide further  contextual information.  data collected  through the logs and diaries were reviewed  on a weekly basis and, along with focus  group data, were subject to summative  quantitative and qualitative analysis.  pilot  program services were available to all onsite  users of an arl library within defined  program days and hours.   results: computing assistants recorded a  total of 399 service transaction log entries  over the 11 week period.  of the recorded  questions, 79% were resolved directly by the  cas, including 86% of 214 computing‐ related questions.  in descending order,  printing, software applications, general  hardware, networking, file transfer, and  scanning were the most common question  topics.  in addition to computing questions,  those related to directional help, reference  and general library services were common.   computing assistants documented needs  for (1) toolkits that include diagnostic tools  and utilities and portable storage devices  and (2) advanced access to information  about the evolving computing environment  in the libraries and across the university,  and (3) improvements needed in the library  computing environment, all in the context of  specific service events.  discussion: weekly review of service  transaction logs and reflective diary entries  provided insight into the ongoing  development of computing support services  within the libraries and has provided data to  support the further development of  computing support services.  data collected  through the logs and diaries both supported  pre‐program assumptions and brought  unanticipated needs to our attention.  for  example, while we expected patrons would  request help with printing, software  applications and networking, we did not  anticipate that there would be a great need  for help with file transfer.  the combination  of methods used for data collection resulted  in a dataset rich in detail, contextual  evidence based library and information practice 2007, 2:2  52 information, and quantified measures we  used to modify the pilot program on an  ongoing basis as well evaluate the overall  program through component analysis.   findings of the pilot program have  informed the continuing development of  computing support services provided  collaboratively by the office of information  technology and the library system.      research tasks required of freshmen: an  evidence‐based approach to curriculum‐ integrated instruction. amy vanscoy, mlis,  school of information & library science,  university of north carolina chapel hill &  megan oakleaf, phd, school of information  studies, syracuse university, united states.    question: to identify the research tasks  first‐semester students are required to  perform in order to determine the efficacy of  existing curriculum‐integrated instruction  (cii) programs.  [curriculum‐integrated  instruction (cii) is a programmatic  approach that advocates integration of  library instruction into the entire curriculum  rather than individual courses.  it is also  characterized by a tiered approach,  beginning with basic skills and progressing  to more advanced skills later in a student’s  career.]  setting: a large research university in the  southeast.  method: a random sample of 350 first‐ semester students was obtained for the  study.  a syllabus study was conducted for  each course in which a student from the  sample was enrolled, and research  assignments were analyzed to determine  what types of research tasks students were  required to perform.  analysis was  conducted to determine the number of  students in the sample required to do library  research and what types of research  students were required to do.  the sample  was also analyzed by college in which  students were enrolled.  main results: contrary to findings from the  literature, all students in the sample were  required to do some library research.  most  students were required to find websites,  articles and books, tasks that published cii  programs do not completely address.  when  students were divided into groups  according to the college in which they were  enrolled, there were some differences in the  tasks they were required to perform.  conclusion:  curriculum‐integrated  instruction programs need to cover finding  websites, articles and books to meet the  research needs of the first‐semester students  studied.  for students in some colleges,  additional research tasks should be included  in departmental cii plans.  published cii  programs would not meet the needs of  students at this university.  evidence about  the research needs of students should be  collected as a basis for developing cii  programs in the future.      the open access revolution: patterns in  faculty free full‐text publications. k.t.l.  vaughan, msls, librarian for  bioinformatics and pharmacy, carol g.  jenkins, mls, library director, unc chapel  hill, virginia r.m. carden, msls,  administrative research librarian, medical  center library, patricia l. thibodeau,  mls,mba, associate dean for library  services and archives, medical center  library, duke university medical center, &  stefanie e. warlick, msls, liaison and  outreach services librarian, health sciences  and human services library, university of  maryland baltimore, united states.    objectives: two academic health sciences  libraries are studying the publishing  patterns of their institutions’ authors in  relation to free full‐text journals, both true  open access and those with embargo periods.  the study addresses how many open  access (oa) articles were published, in  which journals and disciplines, and  evidence based library and information practice 2007, 2:2  53 examines other similarities and differences  between the two institutions, one public and  one private. it updates a preliminary study  done in 2005, by extending the timeframe  and coverage to include articles whose  embargo period has been lifted.   methods: duke university medical center  library and the health sciences library,  unc‐chapel hill, collaborated to identify  publications by their institutions’ authors.  using standardized search strategies in  pubmed, articles published in oa journals  were identified. due to changing embargo  periods and increased awareness, further  evaluation of the data was done to identify  additional patterns in oa publishing among  biomedical faculty.   main findings: preliminary analyses  indicated that while publication rates  between the two institutions were similar,  only five of the top ten journals were  common to the two schools. the majority of  the publications were in medicine, but with  more basic science departments choosing  oa journals. this follow‐up study covers an  additional six months of publishing activity  at the institutions. analysis will compare  which journals and disciplines are  represented, and numbers of articles “born  free” versus “set free”, to determine if the  extended time period reveals different  trends.        information economics and scholarly  communication as instructional strategies  for information literacy. scott warren, ma  lis, assistant head, textiles library and  engineering services, north carolina state  university libraries, & kim duckett,  principal librarian for digital technologies  and learning, north carolina state  university libraries, united states.    over the past four years, librarians at ncsu  and ucla have experimented with  incorporating topics such as the deep web,  the nature of scholarly communication, and  the inflated costs of academic journal  subscriptions into information literacy  instruction. our goal is to provide students  with a broader context for understanding  how scholarly information functions in  society, how information is structured on  the web, and what information can or  cannot be accessed via google versus  subscription‐based resources provided by  the library. this contextualization of  academic resources is presented in tandem  with hands‐on instruction on how to access  and search such resources effectively. as  opposed to much information literacy  instruction, however, less time is spent on  the “mechanics” of searching and more time  is spent on the social and economic contexts  surrounding information and access to it.  this shift in attention away from simply  searching towards contextualization of how  information works in society falls squarely  into a paradigm shift for librarians towards  teaching the critical thinking skills that  contemporary students need. these critical  thinking skills affect not only their  mechanical ability to navigate the vast realm  of free and restricted internet information  resources, but also to understand how  economics shapes availability of and access  to information– issues that affect all people  as educated participants in an information  society. in addition to demonstrating  techniques for incorporating the economics  of information into information literacy  instruction, we will describe assessment  data explores (1) the interest and usefulness  of this contextual information for students  and (2) the appropriate combination of  contextual content and time spent teaching  the mechanics of searching during  information literacy instruction sessions. at  both ncsu and ucla this assessment data  has included end‐of‐workshop assessments  completed by students. this assessment has  focused on what they learned that was  “new” and “useful” as a way to assess how  much interest and impact the economic  information held for students compared to  evidence based library and information practice 2007, 2:2  54 simply teaching them how to search  databases and e‐journals. additionally, at  ncsu librarians are currently collecting  new assessment data on how such  instruction might affect students’ mental  models of how scholarly information is  made available and accessed on the web.  collectively, this assessment data has  enabled us to make adjustments to our  individual instructional effects focused on  the academic level of our audiences and has  provides insight into how students think  about how information works on the web.       patient 101: taking charge of your  healthcare by using evidence based  information.  susan e. werner, mls, health  sciences library, stony brook university, &  colleen kenefick, mls, health sciences  library, stony brook university, united  states.    program objective: this poster examines  the integration of internet consumer health  information literacy instruction into a  medical school’s mini‐medical school  program by using evidence based  information.  setting: the health sciences library for the  academic medical center on long island,  new york is the setting for this innovative  outreach program. informatics librarians  present “patient 101: taking charge of your  healthcare by using evidence based  information” as a two‐hour computer  training session. the informatics librarians  provide all curriculum materials for  instruction to foster best practices for  searching and evaluating internet health  information.   participants: informatics librarians provide  instruction and library staff serve as  facilitators and technical support. since  these are hands‐on sessions, almost all  public service and information systems staff  are involved in this outreach program. mini‐ medical school registration is open to the  general community and includes all  computer skill levels and ages from  adolescents to seniors.       program: for the past seven years,  community residents have registered to  participate in an eight week program  sponsored by the school of medicine. mini‐ medical school mirrors in miniature the pre‐ clinical curriculum of “real” medical school.  lectures are provided by the same faculty  who teach matriculated medical students.  mini‐medical school covers a wide range of  medical and health topics exploring diverse  healthcare and related issues. recent  sessions include the health literacy crisis,  how to get the most out of your doctor’s  visit, and dispelling myths about your body.  since the program’s inception in 2000,  librarians have taught over 1,000  community members specific methods to  improve their consumer health information  literacy skills by using evidence based  information. librarians utilize a variety of  active learning techniques to teach  participants the best evidence based sources  to answer their questions about diagnosis,  etiology, prognosis and therapy.    main results: targeted pre‐session skills  surveys divide participants into beginner,  intermediate, and advanced computer skill  level sections. a post‐session test is used  along with a program evaluation form.  evaluation forms conclude that 94% of the  attendees agreed that presented information  was useful to them and that the librarian  instructor was extremely effective.  conclusion: this program offers the  opportunity to engage public interest in  medical science while promoting health  awareness and enhancing health  information literacy. it enables the librarians  to showcase the resources of their library,  forge closer ties with the community and  raise public awareness about healthcare  literacy by using evidence based  information and practices.   evidence summary   a survey of music faculty in the united states reveals mixed perspectives on youtube and library resources   a review of: dougan, k. (2016). music, youtube, and academic libraries. notes, 72(3), 491-508. https://doi.org/10.1353/not.2016.0009   reviewed by: brittany richardson web services librarian, assistant professor university of tennessee at chattanooga library chattanooga, tennessee, united states email: brittany-richardson01@utc.edu   received: 2 sept. 2020                                                               accepted:  30 oct. 2020      2020 richardson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29835     abstract   objective – to evaluate how music faculty members perceive and use video sharing sites like youtube in teaching and research.   design – survey questionnaire.   setting – 197 music departments, colleges, schools, and conservatories in the united states.   subjects – 9,744 music faculty members.   methods – schools were primarily selected based on national association of schools of music (nasm) membership and the employment of a music librarian with a music library association (mla) membership. out of faculty members contacted, 2,156 (22.5%) responded to the email survey. participants were asked their rank and subspecialties. closed-ended questions, ranked on scales of 1 to 5, evaluated perceptions of video sharing website use in classroom instruction and as assigned listening; permissibility as a cited source; quality, copyright, and metadata; use when items are commercially unavailable; use over library collections; comparative ease of use; and convenience. an open-ended question asked for additional thoughts or concerns on video sharing sites and music scholarship. the author partnered with the university of illinois’ applied technology for learning in the arts and sciences (atlas) survey office on the construction, distribution, and analysis of the survey data through spss. the open-ended question was coded for themes.   main results – key findings from closed-ended questions indicated faculty: used youtube in the classroom (2.30 mean) more often than as assigned listening (2.08 mean); sometimes allowed youtube as a cited source (2.35 mean); were concerned with the quality of youtube recordings (3.58 mean) and accuracy of metadata (3.29 mean); and were more likely to use youtube than library resources (2.62 mean), finding it easier to use (2.38 mean) and more convenient (1.83 mean). the author conducted further analysis of results for the nine most reported subdisciplines. ethnomusicology and jazz faculty indicated a greater likelihood of using youtube, while musicology and theory/composition faculty were more likely to use library resources than others. there was little significant difference among faculty responses based on performance subspecialities (e.g. voice, strings, etc.). overall, open-ended faculty comments on streaming video sites were negative (19.3%), positive (19.3%), or a mixture of both (34.1%). themes included: less use in faculty scholarship; a need to teach students how to effectively use youtube for both finding and creating content; the value of youtube as an audio vs. video source; concerns about quality, copyright, data, and reliability; and benefits like easy access and large amounts of content.   conclusion – some faculty expressed concern that students did not use more library music resources or know how to locate quality resources. the study suggested librarians and faculty could collaborate on solutions to educate students. librarians might offer instructional content on effective searching and evaluation of youtube. open-ended responses showed further exploration is needed to determine faculty expectations of library “discovery and delivery” (p. 505) and role as the purchaser of recordings. conversations between librarians and faculty members may help clarify expectations and uncover ways to improve library resources and services to better meet evolving needs. finally, the author recommended additional exploration is needed to evaluate youtube’s impact on library collection development.   commentary   studies related to library music collections and streaming media sites are primarily institutional in scope. hooper’s (2017) literature review highlighted studies focusing on library music collection usage, collection trends, and student user preferences. clark, sauceda, and stormes (2019) used surveys and interviews to evaluate the use of resources by performing arts faculty at three institutions. this study focused on various types of library resources, including audio/visual materials, with some questions related to non-library resources. the author’s study contributed significantly to the literature by providing a national look at music faculty perceptions of streaming media. the author also previously used the survey data collected by this study to compare music librarian and faculty perspectives on youtube (dougan, 2014).   when evaluated using glynn’s (2006) critical appraisal tool, the study demonstrates several strengths. a sizable number of relevant participants were recruited using effective, bias-free criteria. the design was appropriate for the stated research outcomes. the study methodology was clearly reported, although more detail on data analysis methods would have been valuable for practitioners interested in conducting similar studies. results were well-presented. specifically, a thorough job was done connecting closed-ended and open-ended responses to draw overarching themes in the data. the author used findings to provide valuable insights and recommendations for practitioners.   although it seems likely all survey content was evaluated in the article, inclusion of the instrument would have provided the opportunity to evaluate survey construction and clarity. it is also unclear whether the instrument was validated prior to use. information on potential study limitations would have also been valuable for interpreting results.   the results of this study are of interest to academic music librarians and other practitioners involved in library support for music faculty and students. librarians may find this study useful for its glimpse into the perceptions and practices of faculty stakeholders in relationship to streaming video sites. as recommended by the author, practitioners should also engage in conversations with music faculty to help guide the evolution of services and resources for music disciplines. areas for further study may include instructional methods for library music resources and streaming media sites, the marketing of library music resources to faculty, and the evolution of music library collections to support changing needs.   references   clark, j. c., sauceda, j., & stormes, s. (2019). faculty format preferences in the performing arts: a multi-institutional study. college & research libraries, 80(4), 450-469. https://doi.org/10.5860/crl.80.4.450   dougan, k. (2014). “youtube has changed everything”? music faculty, librarians, and their use and perceptions of youtube. college & research libraries 75(4), 575-589. https://doi.org/10.5860/crl.75.4.575   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399.   hooper, l. (2018). becoming a warehouse of things: the audio world is changing, and collection development methods must change, too. music reference services quarterly, 21(3). https://doi.org/10.1080/10588167.2018.1493889     evidence summary   library staff are more motivated to engage in professional development when encouraged by library leadership   a review of: johannessen, h. t. d. (2018). the need to grow, learn and develop – how does management affect motivation for professional development? liber quarterly, 28(1), 1–16. https://doi.org/10.18352/lq.10238     reviewed by: hilary bussell the ohio state university libraries columbus, ohio, united states of america email: bussell.21@osu.edu   received: 28 nov. 2018                                                                 accepted: 16 jan. 2019      2019 bussell. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29534     abstract   objective – this study explores whether there is a correlation between academic library leaders’ support for professional development and their employees’ professional self-esteem and motivation to participate in professional development.   design – survey questionnaire.   setting – academic libraries in norway.   subjects – 1,637 full time equivalent (fte) staff.   methods – the theoretical framework for this study is knowledge management. the author defines this as “the creation and subsequent management of an environment which encourages knowledge to be created, shared, learnt, enhanced, and organized for the benefit of the organization and its customers” (sarrafzadeh, martin, & hazeri, 2006, p. 624, quoted on p. 3). an anonymous quantitative survey was made available to staff working in norwegian academic libraries. the survey included questions about to what extent their leader encourages them to attend conferences, to what extent their leader understands their skills and competencies, personal belief in their own skills and competencies to perform their work tasks, and number of professional development activities they attended in 2015 (including conferences, continuing education seminars, and interdepartmental seminars).   main results – 626 survey responses were collected, for a 38% response rate. the responses were analyzed using descriptive statistics. over 50% of all survey respondents reported high satisfaction with their professional skills and competencies. there is a difference when broken down by gender, with 77% of men reporting confidence in their professional skills versus 63% of women. education level, on the other hand, does not make a difference. the study found a correlation between perception by library staff that their library leader has a “good overview” of their professional skills and staff members’ confidence in their ability to perform their job well. library staff with leaders who encouraged professional development were more likely to participate in external professional development activities. participation in internal professional development activities was not affected by library leaders’ encouragement.   conclusion – when library staff are encouraged by library leadership to participate in professional development, they are more likely to do so. library staff who perceive that their library’s leaders recognize and value their professional skills and competencies have a higher sense of professional self-esteem. library leaders can use knowledge management to come to a better understanding of the knowledge and skills their staff members already possess, and to encourage communities of practice and the sharing of knowledge in the organization. this recognition can result in employees who are happier and more motivated to learn.   commentary   this study adds to the extensive literature on the role that library leadership plays in encouraging a culture of professional development in their organizations. in 2000, shanhong argued that professional training and education among library staff would become an important knowledge management objective in 21st century libraries. more recently, some have argued that academic libraries should improve their use of knowledge management in order to increase knowledge sharing among staff (koloniari & fassoulis, 2017; muchaonyerwa & mutula, 2017). this study reinforces the value of knowledge management for contributing to employees’ self-confidence and motivation to engage in professional development.   the study was evaluated using the “ebl critical appraisal checklist” (glynn, 2006). the methodology utilized is appropriate to studying the entire population of academic library staff in norway. a few areas of the methodology could benefit from clarification. although the paper states that a survey was conducted using the online program surveyxact, it does not discuss how participants were recruited, and it is unclear whether the survey population includes all academic library staff or only staff with “librarian” in their job title. since some academic libraries have different levels of professional development support and expectations for librarians versus non-librarian staff, this could have implications for the findings. additionally, the article does not include the survey instrument itself, precluding the possibility of other researchers replicating the study.   a potential confounding variable that was not discussed in the article is financial support for professional development. if library leaders who encourage their staff to attend conferences are more likely to provide financial support for these activities, this could be a reason why these staff members are more likely to attend. this could also explain why staff participation in in-house professional development was not correlated with a library leader’s encouragement. overall, the article would benefit from a more thorough explanation of the study’s findings.   this study will be valuable to anyone in library administration who is interested in cultivating professional skills and self-confidence among their employees. using a knowledge management frame, the paper makes a compelling case for why it is important that library managers not only familiarize themselves with the competencies and skills their staff possess, but make sure they are communicating to their staff that they are recognized and valued. the study indicates that social recognition goes a long way to improving employee self-confidence and motivation to learn, which benefits both individual employees and the organization as a whole.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387–399. https://doi.org/10.1108/07378830610692154   koloniari, m., & fassoulis, k. (2017). knowledge management perceptions in academic libraries. the journal of academic librarianship, 43(2), 135–142. https://doi.org/10.1016/j.acalib.2016.11.006   muchaonyerwa, n., & mutula, s. (2017). knowledge-sharing strategies among library staff at selected university libraries. mousaion, 35(3), 1–24. https://doi.org/10.25159/0027-2639/2003   sarrafzadeh, m., martin, b., & hazeri, a. (2006). lis professionals and knowledge management: some recent perspectives. library management, 27(9), 621–635. https://doi.org/10.1108/01435120610715527   shanhong, t. (2000, aug.). knowledge management in libraries in the 21st century. proceedings of the 66th ifla council and general conference. jerusalem, israel. retrieved from https://archive.ifla.org/iv/ifla66/papers/057-110e.htm research in practice   research grants: does my research need funding?   virginia wilson director, centre for evidence based library and information practice (c-eblip) university library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   received: 8 nov. 2016   accepted: 14 nov. 2016      2016 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     more and more librarians are conducting research to inform practice and to further the profession of librarianship. as this unfolds, more complex research is taking place and as a result, additional resources are necessary in order to complete projects. at the university of saskatchewan (u of s) library, the number of librarians who are successfully applying for research funding is growing. from the very first librarian president’s sshrc grant of $7,000cad (an internal u of s grant designed to be seed funding to get a project off the ground) in 2011 to a social sciences and humanities research council (sshrc) (canada’s national funding agency in social sciences and the humanities) insight development grant (idg) in the amount of $54,000 in 2015, librarians here are seeing the value of seeking funding to assist with research.   it wasn’t always that way. the attitude used to be, what do we need money for? our research consists of small projects that require a computer, a printer, and various other bits of office supplies…maybe some specialized software. but we have professional development funds for that, right? then our library hired a research facilitator, someone dedicated to assisting us with our research endeavours. we quickly discovered what all we could do with research funding: pay for transcription, travel for conferences, hire a research assistant, purchase equipment, get assistance from the social sciences research lab (a u of s initiative) – per our research facilitator, the list is practically endless!   however, librarians in other institutions are no strangers to applying for grants. at penn state, where research is a requirement for librarians, fennewald (2008) found that while librarians have some internal funds for conferences and research costs, “additional support comes from competitive research grants” (p. 112). in a look at the literature to support their study entitled examining success: identifying factors that contribute to research productivity across librarianship and other disciplines hoffmann, berg, and koufogiannakis (2014) found that one “measure of research productivity that was most commonly used was grants or funding received” (p. 19). another indication of librarians applying for research grant funding is the number of research grants available for librarians. for example, hlwiki international has a substantial reference list of awards available for academic librarians and others. (http://hlwiki.slais.ubc.ca/index.php/awards_for_academic_librarians )   grant writers can face barriers that are much the same as the barriers that can be involved in conducting research: lack of time, lack of experience, little support from management, low motivation, and so on. there are, however, so many benefits that can be derived from applying for grants that go beyond the money (although the funding is great; don’t get me wrong!).   some of the benefits of applying for funding include:   taking the opportunity to hone your research project: grant writing makes you think of the research objective, the methodology, the literature, what resources are essential to completing the project, timelines – everything you will need to move your project forward. acquiring experience in writing grants. practice makes better, and every time you write a funding application you will learn something. being able to take portions of your written grant proposal and incorporate them into your research ethics application. gaining the confidence to apply for subsequent grants. apparently like potato chips and tattoos, obtaining funding can be a bit addictive. bet you just can’t apply for one!   to get going at your institution, someone needs to start. back in 2011 when i was about to go on sabbatical, i came across the president’s sshrc grant, internal funds to help a researcher get a project going. i figured i had nothing to lose except a bit of time so i filled out the application and sent it in. i was pleased to be awarded $7,000 towards my sabbatical research project. i was the first u of s librarian to get the president’s sshrc, but that’s only because no one had tried before. the next year, we had four librarians apply for the grant and the majority were successful. and every year since then, a librarian’s grant application is adjudicated by the president’s sshrc committee and our success rate is very high. sometimes all it takes is someone to blaze a trail, opening the doors of possibility for others to do the same.   if there is already some funding activity in your library, share the experiences with one another. share grant proposals, form a support group for grant writing, check with professional associations for available grants, and look internally to your institution for funding opportunities that may be open to librarians. reach out to colleagues for grant mentorship. and then when you are successful, become a mentor in turn. as we continue to conduct research and to take advantage of the resources that are out there to assist us, we as librarians are contributing to the culture of research in our profession.   references   fennewald, j. (2008). research productivity among librarians: factors leading to publications at penn state. college and research libraries, 69(2), 104-116. http://doi.org/10.5860/crl.69.2.104   hoffman, k., berg, s., & koufogiannakis, d. (2014). examining success: identifying factors that contribute to research productivity across librarianship and other disciplines. library and information research, 38(119), 13-28. retrieved from http://www.lirgjournal.org.uk/lir/ojs/index.php/lir/article/view/639   uneven adherence to professional guidelines and potential ethnic bias in service provision evidenced in virtual reference service interactions evidence summary   uneven adherence to professional guidelines and potential ethnic bias in service provision evidenced in virtual reference service interactions   a review of: hamer, s. (2021). colour blind: investigating the racial bias of virtual reference services in english academic libraries. the journal of academic librarianship, 47(5), 102416. https://doi.org/10.1016/j.acalib.2021.102416   reviewed by: scott goldstein coordinator, web services & library technology mcgill university library montréal, québec, canada email: scott.goldstein@mcgill.ca   received: 1 dec. 2021                                                                    accepted:  19 jan. 2022      2022 goldstein. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30085     abstract   objective – to investigate whether there is evidence for implicit ethnic bias in virtual reference service interactions.   design – email-based structured observation study.   setting – academic libraries in england.   subjects – 158 email-based virtual reference service interactions from one of 24 academic libraries in england.   methods – the study used a sample of 24 academic libraries across eight of the nine regions of england (excluding london). the body of the email message sent to each library consisted of one of five questions and was identical except for personalization to the institution. the first three questions were designed to be more likely to be answered in response to an unaffiliated user, and the last two questions were designed to be less likely to be answered in response to such a user. each library received an email with each question from a different sender during each of five weeks, plus a repeat of question one in week six with slightly altered wording to serve as a control question. emails were sent on randomized work days at different times of day. the messages were signed with one of six names representing the largest distinct ethnic population groups in england and wales: hazel oakland (white british), natasza sakowicz (white other), zhao jinghua (north asian), priya chakrabarti (south asian), ebunoluwa nweke (black african), and aaliyah hajjar (arab). all names were feminine and represented unaffiliated users. email replies were coded according to a set of 27 characteristics based on the two most well-known professional guidelines for providing best practice reference services, namely, ifla and rusa.   main results – 133 out of 144 sent queries received a reply, of which 66 partially or fully answered the question. 158 total emails were received (since an email might receive multiple responses), and 67 of these partially or fully answered the question. differences in how the librarian’s reply addressed the user were evident. hazel was the only one never referred to by her full name, whereas jinghua was the least likely to be referred to by her given name and most likely to be referred to by her full name or no name at all. greeting phrases were used in most responses. about 20% of responses included a reiteration of the original request. elements of the response which could be seen as promoting information literacy skills were provided in only 11% of responses. natasza was the most likely to be referred to another source to answer her query, whereas jinghua was least likely. ebunoluwa was the least likely to receive a response to her query and least likely to have her question answered overall.   conclusion – the findings point to some evidence of unequal service provision based on unconscious bias. in the aggregate, ebunoluwa received the lowest quality of service, while jinghua received the highest. there were several instances of inappropriately addressing the user, or what the author refers to as name-based microaggressions, and this was most common for jinghua. the likeliest explanation is that many librarians are unfamiliar with the ordering of names traditionally found in east asian cultures. the most noticeable result of the study is an overall lack of consistent adherence to professional guidelines. for instance, most queries received a reply within a reasonable timeframe, and greeting and closing phrases were included almost universally. however, other elements of the author’s rubric, such as those corresponding to clarity and information literacy, were not consistently applied. the results point to a greater need for librarians to follow best practice in virtual reference services. furthermore, the author believes that best-practice guidelines must actively engage with anti-racist ideas to address the issues that were found in the study.   commentary   described by the author as an email-based structured observation study, the study might also be called a correspondence audit, which has been a staple in many of the social sciences since the 1960s but is relatively novel in library and information science. the study discussed here is an adaptation to a european context of shachaf and horowitz’s (2006) audit study of 23 us academic libraries, investigating whether racial or religious bias could be detected in email-based virtual reference services. they had found that, by manipulating the sender name of the unaffiliated user asking a reference question, quality of service was affected, with african-american and arab users more likely to be ignored, given a longer wait, or answered with peremptory responses compared to white, christian, asian, jewish, and hispanic users. a follow-up study two years later with a larger sample size failed to replicate the previous results; however, the methods were not identical (shachaf et al., 2008). hamer largely follows the earlier study and adapts it for the ethnic and racial distribution of england. given the wide adoption of virtual reference services in libraries, it is important to investigate whether discriminatory behaviour, even if unintentional, is observed in these interactions.   this study was evaluated using glynn’s (2006) critical appraisal tool. the study sample was representative of the population of interest, namely academic libraries in england. informed consent was not obtained, but for reasons mentioned in the article, this could be justifiable in a study of this kind to minimize participants ascertaining the true aim of the study and altering their behaviour. the methods were clearly outlined and draw on earlier scholarship. the biggest problem for the study is that the author gives no evidence to believe that the unequal service provision described—namely, similar but non-identical counts across a couple dozen characteristics, with no discernible pattern of one group outperforming others on some composite measure—represents a meaningful sign of ethnic bias. a sample size of 158 split across six groups is simply not large enough to make conclusions about group differences. the author acknowledges this as a limitation but sees it as a problem for generalizability rather than power, that is, the ability to detect an actual effect if one exists. it is inevitable that librarian responses will differ slightly across professional guidelines (as well as other dimensions) in a naturalistic setting. the question is, are they systematic differences, above and beyond natural variation, that show a clear pattern of discrimination across plausibly related measures? the author does not address that question, so claims of ethnic bias are premature.   notwithstanding its shortcomings, this paper offers many interesting takeaways for librarians, especially in the midst of the covid-19 pandemic with many libraries continuing to emphasize their virtual reference services. more attention needs to be paid to improving adherence to professional standards and best practices. this study did not examine whether the mixed service quality provided to users was the result of their being unaffiliated with the institution or perhaps general burnout from higher-than-usual queries. if it is the latter, perhaps certain “shortcuts,” such as the use of (partial) canned message replies or gifs to illustrate information literacy concepts, would make responding to emails less burdensome. a further takeaway deals with correctly addressing users by their name. to what extent are librarians familiar with name ordering practices around the world? are there best practices for when to use honorifics or how to avoid misgendering? this seems like an underexplored topic that merits consistent and easy-to-apply standards.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387–399. https://doi.org/10.1108/07378830610692154   shachaf, p., & horowitz, s. (2006). are virtual reference services color blind? library & information science research, 28(4), 501–520. https://doi.org/10.1016/j.lisr.2006.08.009   shachaf, p., oltmann, s. m., & horowitz, s. m. (2008). service equality in virtual reference. journal of the american society for information science and technology, 59(4), 535–550. https://doi.org/10.1002/asi.20757     evidence summary   “just-in-time” unmediated document delivery service provides fast delivery, helps identify collection gaps, but incurs extra costs   a review of: chan, e. k., mune, c., wang, y., & kendall, s. l. (2016). three years of unmediated document delivery: an analysis and consideration of collection development priorities. medical reference services quarterly, 35(1), 42-51. retrieved from: http://dx.doi.org/10.1080/02763869.2016.1117288   reviewed by: heather macdonald health and biosciences librarian macodrum library carleton university ottawa, ontario, canada email: heather.macdonald@carleton.ca   received: 28 feb. 2017   accepted: 4 apr. 2017      2017 macdonald. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – examine the collection development opportunities and challenges of an unmediated document delivery service.   design – case study.   setting – large comprehensive public university in the united states of america.   subjects – 11,981 document delivery requests.   methods – this library implemented copyright clearance center’s get it now (ccc-gin) service in november 2011 to supplement existing holdings, provide access to embargoed content and help support two new programs. the ccc-gin service was offered in addition to regular ill service. statistical analysis was done using usage data collected for the academic years 2012-2013, 2013-2014, and 2014-2015 (excluding june and july). usage data included: order date and time, fulfillment date and time, publication name, publication date, article name, article author, publisher name, cost, delivery e-mail address. taylor and francis publications were added to the ccc-gin service in november of 2014.   main results – the average yearly cost of titles with the largest number of ccc-gin requests was compared to the annual subscription cost of the same titles. if the annual subscription cost was less than the average yearly cost of ccc-gin requests, the library purchased a subscription. patrons ordered older journal content through ccc-gin requests. this suggested that backfile subscriptions could be cost effective means of providing content. the authors are in the process of analyzing what historical journal content should be purchased.   the addition of taylor and francis publications resulted in an increase in the average cost per article. taylor and francis publications were popular with patrons, helping boost the total number of requests. the date of the taylor and francis materials ordered through ccc-gin tended to be more recent compared to other publishers. the authors suggest ccc-gin is a possible solution for acquiring embargoed material. average fulfillment time increased during the three year time period from 1:34 (hr:min) to 3:52. the percentage of requests outside of ill working hours was consistent across all three years (62% each academic year). the authors note ccc-gin service provided the most expedient way for patrons to receive requested material.   a number of the most requested ccc-gin publications were also available in print format. the quality of print serials data was uncertain hence the decision was made to not upload this data to the ccc-gin service. this resulted in some overlap in requests with the library’s print holdings. older content was requested through ccc-gin rather than through traditional ill. this resulted in increased costs from copyright fees that would have been avoided using traditional ill services.   conclusion – the authors reference the impact of e-commerce on library patron expectations about ease of access and just-in-time delivery. they found that the ccc-gin service meets these expectations as patrons were able to access a broad selection of materials in a timely and easy to use manner. from the analysis come suggestions to help reduce costs associated with the service. they include adjusting system settings to cap spending limits, limiting who can use the service, selecting only titles that cover a gap in the collection, and including quality print serials holdings data to prevent purchase of already owned material. the authors also discuss using a mediated rather than unmediated service to help lower costs but they note this would slow down turnaround time. the authors close by saying each library will have to consider its own needs and those of its patrons with respect to ease of use, delivery time, and cost.   commentary   at the 2011 acrl conference, copyright clearance center announced the debut of the get it now service (brynko, 2011). ccc-gin provides patrons with full-text articles from non-subscribed journals within hours from the time of request, 24 hours a day, 7 days a week. a number of studies have investigated the unmediated (no human handling of requests) version of this document delivery service (hendler & gudenas, 2016; jarvis & gregory, 2016; suhr, 2013). this study discusses the collection development opportunities and challenges resulting from their analysis of three years of data. the cat critical appraisal tool (perryman & rathbun-grubb, 2014) is used here to help assess the study.   the literature review discusses the impact of e-commerce principles (fast, “frictionless ordering”) on library user expectations. it also discusses the benefits and limitations of unmediated document delivery services and copyright compliance issues. all of these issues are relevant to the subject and help lay the ground work for the study.   the project was initiated to supplement existing holdings, provide access to embargoed content and help support two new programs. the implicit question is whether the service succeeded in achieving what it set out to do. the analysis shows that the ccc-gin service does indeed provide access to content that the library does not subscribe to. however, the analysis does not provide clear evidence of patrons accessing embargoed content. the data presented show a drop in average age of requested articles that coincides with the addition of the taylor and francis publications. however, the data analysis presented is not granular enough to show how many of the requested taylor and francis publications were from the embargo period(s). the third rationale for implementing the service was to support two new programs. the analysis identifies new content and older backfiles that are relevant to the programs but the authors do not state that the acquisition of the titles was to support the new programs. the authors also identified additional costs that came out of the analysis. despite that, the library is continuing with the ccc-gin service because they are committed to quick service and patron preference for online material.   the authors mention that the fulfillment time increased over the three-year period. they did not offer a suggestion as to why. some speculation on this would have been interesting. next steps are also not included in this study. it would be interesting to see where they would like to go from here.   this study provides a clear picture of how the library and its patrons used the ccc-gin service. the paper’s unique contribution is the explanation of how and why collection development decisions were made based on the analysis. the authors also offer practical suggestions for others interested in implementing a similar service. this paper is very helpful for those interested in implementing an unmediated document delivery service or for those simply interested in finding out more about it.   references           brynko, b. (2011). acrl 2011: a few new product rollouts. information today, 28(5), 34–35.   hendler, g. y., & gudenas, j. (2016). developing collections with get it now: a pilot project for a hybrid collection. medical reference services quarterly, 35(4), 363–371. http://dx.doi.org/10.1080/02763869.2016.1220751   jarvis, c., & gregory, j. m. (2016). get it? got it. good!: utilizing get it now article delivery service at a health sciences library. journal of electronic resources librarianship, 28(2), 93–100. http://dx.doi.org/10.1080/1941126x.2016.1164556   perryman, c. & rathbun-grubb, s. (2014). the cat: a generic critical appraisal tool. in jotform – form builder. retrieved 21 aug. 2014 from http://www.jotform.us/cp1757/thecat   suhr, k. f. (2013). get it now: one library’s experience with implementing and using the unmediated version of the copyright clearance center’s document delivery service. journal of electronic resources librarianship, 25(4), 321–325. http://dx.doi.org/10.1080/1941126x.2013.847694     evidence based library and information practice editorial responsibilities      2020. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     editor-in-chief: lorie kloda   associate editor (evidence summaries): stephanie krueger, christina wissinger   associate editor (research articles): ann medaille, lisl zach   associate editor (classics & reviews): jane schmidt   associate editor (commentaries, using evidence in practice): lorie kloda   communications officer (news): kimberly mackenzie   production editor: rachel hinrichs   editorial intern: melissa cober   editorial advisors: lindsay alcock, alison brettle, michelle dunaway, denise koufogiannakis, heather pretty   copyeditors: heather healy (lead copyeditor), katelyn baroody, kirstin duffin, julie evener, emily kingsland, alison moore, jane morgan-daniel, stacey l. penney, elizabeth stregger, nikki tummon, elaina vitale, ross wilson   indexing support: kate shore   editorial assistant: samantha sheplawy   evidence summary   academic librarians in canada concerned about online and patron privacy but lack knowledge about institutional procedures and policies   a review of: tummon, n., & mckinnon, d. (2018). attitudes and practices of canadian academic librarians regarding library and online privacy: a national study. library and information science research, 40(2), 86-97. https://doi.org/10.1016/j.lisr.2018.05.002     reviewed by: stephanie krueger senior consultant national library of technology in prague prague, czech republic email: stephanie.krueger@techlib.cz     received: 13 feb. 2019                                                                  accepted: 2 apr. 2019      2019 krueger. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29555     abstract   objective – to assess attitudes of canadian academic librarians regarding online privacy issues and to gauge their knowledge of related procedures and policies at their institutions.   design – attitudinal online survey in english.   setting – english-language academic libraries in 10 canadian provinces.   subjects – english-speaking academic librarians across canada.   methods – survey, based on zimmer’s 2014 study of librarians in the united states of america, announced via email to 1,317 potential participants, managed using limesurvey, and available from april 7 to may 5, 2017. in 28 optional multiple choice or likert scale questions, the survey prompted participants to express their attitudes regarding online privacy scenarios and privacy-related library practices, including patron data collection. results were analyzed in microsoft excel and spss.   main results – the survey response rate was 13.9% (183 respondents). job position, age, or geographic location did not appear to influence attitudes towards privacy, with almost all respondents strongly agreeing or agreeing that individuals should control who sees their personal information (96.2%) and that companies collect too much such information (97.8%). respondents voiced slightly less concern about government information collection, but nearly all respondents agreed that governments should not share personal information with third parties without authorization and that companies should only use information for the purposes they specify. when asked if privacy issues are more important today than five years ago, 69.9% of respondents said they were more concerned and 78.1% noted they knew more than five years before about privacy-related risks.   regarding online behaviour, 53.3% of respondents felt web behaviour tracking is both beneficial and harmful, with 29.1% considering it harmful, and 13.7% finding it neither beneficial nor harmful. online shopping and identify theft, social media behaviour tracking, search engine policy display, and personal information sharing were also areas of concern for respondents, with the majority noting they were somewhat or very concerned about these issues.    in terms of library practices, most respondents strongly agreed that libraries should not share personal information, circulation records, or internet use records with third parties unless authorized, though 33% of respondents noted they could neither agree nor disagree that libraries are doing all they can to prevent unauthorized access to such information. the majority of respondents strongly agreed or agreed that libraries should play a role in educating patrons about privacy issues. many respondents (68.9%) did not know if their libraries had practices or procedures for dealing with patron information requests from law enforcement or governmental representatives. the majority of respondents did not know if patrons at their libraries had inquired about privacy issues, 42.3% did not know if their libraries communicate privacy policies to patrons, and 45.4% noted their libraries did not inform patrons about library e-resource privacy policies. many respondents (55.2%) had attended educational sessions about online privacy and surveillance in the past five years, while 52.2% noted their libraries had not hosted or organized such sessions over the same period.   conclusion – survey participants showed concern about online and patron privacy, though their lack of knowledge about local procedures and policies highlights a potential need for enhanced privacy education.   commentary   this study adds a canadian perspective to the corpus of attitudinal studies of academic librarians and online privacy issues, thus enriching international perspectives in this area of investigation. the area of privacy and libraries is complex and referenced in many professional library organization statements at the international level (o’brien, young, arlitsch, & benedict, 2018, p. 737). but how much academic librarians understand about developments—particularly in areas such as the tracking of reading behaviours (lynch, 2017) or against the backdrop of legal obligations such as the general data protection regulation (gdpr) in the european union (bailey, 2018) – may not necessarily be deep enough to ensure libraries are following their professional privacy statements and institutional policies. this study, by identifying a lack of knowledge about privacy-related procedures and policies, can inspire future researchers to investigate perceived educational gaps in more detail.   the study, which used zimmer (2014) as a guide, provides an illustration of how surveys about privacy can be replicated and compared, with this survey notably including a comparison of attitudes about the sharing of patron information with both zimmer (2014) and a 2008 american library association survey. while not a standardized questionnaire since the original zimmer survey was tailored to the canadian context, the comparability of results across these three studies indicates the study is valid according to the guidelines laid out in boynton & greenhalgh (2004) for questionnaire research (p. 1313). the full survey instrument is an appendix to the article.   the survey included librarians whose emails were checked on institutional websites and the response rate to the study was low (13.9%), so attitudes expressed in this study are perhaps not generalizable or representative of all canadian academic librarians. additional surveys and more inclusion of the second official canadian language, french, might enable deeper examination into national attitudes and provide insight into the question of generalizability.   researchers in other countries could easily use this study as a template for conducting their own research, translating the survey into other languages, and tailoring it to local contexts. the questions raised here about a potential education gap are important and deserve not only the attention of future researchers, but also should inspire library leaders, professional organizations, and individual institutions to analyze their local situations and implement training on privacy-related trends where there are gaps.   references   bailey, j. (2018). data protection in uk library and information services: are we ready for gdpr? legal information management, 18(1), 28-34. https://doi.org/10.1017/s1472669618000063   boynton, p. m., & greenhalgh, t. (2004). hands-on guide to questionnaire research: selecting, designing, and developing your questionnaire. bmj: british medical journal, 328(7451), 1312-1315. https://doi.org/10.1136/bmj.328.7451.1312 . lynch, c. (2017). the rise of reading analytics and the emerging calculus of reader privacy in the digital world. first monday, 22(4). https://doi.org/10.5210/fm.v22i4.7414   o’brien, p., young, s. w. h., arlitsch, k., & benedict, k. (2018). protecting privacy on the web: a study of https and google analytics implementation in academic library websites. online information review, (42)6, 734-751. https://doi.org/10.1108/oir-02-2018-0056   zimmer, m. (2014). librarians’ attitudes regarding information and internet privacy. the library quarterly, 84(2), 123-151. https://doi.org/10.1086/675329   evidence summary   the potential of a cost-per-use analysis to assess the value of library open-access funds   a review of: hampson, c., & stregger, e. (2017). measuring cost per use of library-funded open access article processing charges: examination and implications of one method. journal of librarianship & scholarly communication, 5(1), ep2182. https://doi.org/10.7710/2162-3309.2182     reviewed by: jessica a. koos senior assistant librarian/health sciences librarian stony brook university stony brook, new york, united states of america email: jessica.koos@stonybrook.edu   received: 25 may 2018                                                                  accepted: 9 sept. 2018      2018 koos. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29453     abstract   objective – to determine the feasibility and potential effects of a cost-per-use analysis of library funds dedicated to open access.   design – cost-per-use analysis, case study.   setting – plos and biomed central.   subjects – 591 articles published in plos one, 165 articles published in plos biology, and 17 articles published in biomed central.   methods – three specific examples are provided of how academic libraries can employ a cost-per-use analysis in order to determine the impact of library-based open access (oa) funds. this method is modeled after the traditional cost-per-use method of analyzing a library collection, and facilitates comparison to other non-oa items. the first example consisted of using a formula dividing the total library-funded article processing charges (apcs) by the total global use of the specific plos journal articles that were funded. the second and third examples demonstrated what a library-funded oa membership to biomed central would cost alone, and then with apcs that cost could be divided by the total usage of the funded articles to determine cost-per-use.   main results – the authors found both of the examples described in the article to be potential ways of determining cost-per-use of oa articles, with some limitations. for instance, counting article usage through the publisher’s website may not capture the true usage of an article, as it does not take altmetrics into consideration. in addition, article-level data is not always readily available. in addition, the cost-per-use of oa articles was found to be very low, ranging from $0.01 to $1.51 after the first three years of publication based on the cost of library-funded apcs. the second and third methods revealed a cost-per-use of $0.10 using membership-only payments, while using the cost of membership plus apcs resulted in a cost-per-use of $0.41.   conclusion – libraries may wish to consider using these methods for demonstrating the value of oa funds in terms of return on investment, as these techniques allow for direct comparison to the usage of traditional journals. however, several barriers need to be overcome in how article-level usage is obtained in order for these methods to be more accurate and efficient. in addition, while the authors report that "the specific examples in this study suggest that oa apcs may compare favorably to traditional publishing when considering value for money based on cost per use," they also caution that the study was not designed to answer the question if the roi is greater for oa publications than for traditional articles, stating that "...the data in this study should not be interpreted as a verification of such an argument, as this study was not designed to answer that question, nor can it do so given the limitations on the data. this paper was designed to present and illustrate a method. further study would be necessary to verify or refute this possibility" (p. 15).   commentary   open access publishing models differ from those of traditional publishing models, primarily in that the author is typically responsible for paying article processing charges in order for their work to be published, rather than the cost of publication being covered by a subscription fee. libraries have recently been making efforts to provide financial support to their faculty to publish in oa journals through designated faculty publication funds, which help to cover the expense of apcs (monson, highby, & rath, 2014). the authors suggest three different ways to demonstrate the value of oa funds through cost-per-use at the article level.   while the very low cost-per-use findings for oa journals is enticing, it is important to note that these results are very preliminary. further research across other institutions using these methods is necessary in order to confirm or refute these results.   the quality of this study was appraised using “the cat: a generic critical appraisal tool” created by perryman and rathbun-grubb (2014). based on this analysis, the quality of the study was found to be high. the authors are both university librarians at a large public research institution, which suggests that they most likely have familiarity with the topic at hand. a significant amount of background information was provided, the research question was clearly defined, and the methods used in the research were carefully explained. there was also detailed discussion of the limitations and implications of the study.   while the quality of this study was high, the strength of the evidence is fair due to the limitations of the research, as described by the authors. for example, measuring the number of article views and downloads provided by the publisher does not capture all the usage of oa articles, as they can be accessed from a variety of platforms. in order to make this method more accurate, additional types of measurements could be taken into consideration. for example, as mentioned in the article, altmetrics might be used as a means of evaluation as well, since this method takes into account how many times an article is mentioned on other online venues such as social media sites and blogs. considering the number of views from the publisher in addition to the altmetrics of an article can provide a more accurate picture of its true value.   in addition, not all oa publishers follow the counter definitions when reporting usage statistics, making it difficult to compare usage across various publishers. it is possible that oa publishers will increasingly comply with counter due to pressure from the library community if the method described by the authors gains traction.   in times of financial uncertainty, particularly in terms of limited funding and increasing journal subscription costs, it is important for libraries to be able to justify their expenditures. while the methods explored in this study are not perfect, they provide an excellent starting point for libraries to consider when justifying the existence of faculty publication funds. with increased use of these types of methods, it is possible that oa publishers may facilitate the way usage data is provided in order to increase the validity and practicality of this approach.   references   monson, j., highby, w., & rathe, b. (2014). library involvement in faculty publication funds. college & undergraduate libraries, 21(3/4) 308-329. https://doi.org/10.1080/10691316.2014.933088   perryman, c., & rathbun-grubb, s. (2014). the cat: a generic critical appraisal tool. retrieved jan. 5, 2018 from http://www.jotform.us/cp1757/thecat research article   factors affecting publication impact and citation trends over time   sandra l. de groote professor and head of assessment and scholarly communications university of illinois chicago, university library chicago, illinois, united states of america email: sgroote@uic.edu   jung mi scoulas assistant professor and assessment coordinator university of illinois chicago, university library chicago, illinois, united states of america email: jscoul2@uic.edu   paula r. dempsey associate professor and head of research services and resources university of illinois chicago, university library chicago, illinois, united states of america email: dempseyp@uic.edu   felicia barrett associate professor and regional head librarian, library of health science in rockford university of illinois chicago, university library chicago, illinois, united states of america email: fbarrett@uic.edu   received: 17 july 2022                                                               accepted: 30 jan. 2023      2023 de groote, scoulas, dempsey, and barrett. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30206     abstract   objective – the researchers investigated whether faculty use of the references in articles had a relationship with the later impact of the publication (measured by citation counts). the paper also reported on additional factors that may influence the later impact of publications.   methods – this researchers analyzed data for articles published by faculty at a large public university from 1995 to 2015. data were obtained from the scopus abstract and citation database and analyzed using spss27 to conduct pearson’s correlations and regression analysis.   results – the number of references included in publications and the number of citations articles received each year following publication have increased over time. publications received a greater number of citations annually in their 6th to 10th years, compared to the first 5. the number of references included in an article had a weak correlation with the number of citations an article received. grant funded articles included more references and later received more citations than non-grant funded articles. several variables, including number of references used in an article, the number of co-authors, and whether the article was grant funded, were shown to correlate with the later impact of a publication.   conclusion – based on the results, researchers should seek out grant funding and generously incorporate literature into their co-authored publications to increase their publications' potential for future impact. these factors may influence article quality, resulting in more citations over time. further research is needed to better understand their influence and the influence of other factors.     introduction   as noted in the association of research libraries (arl) research library impact framework, it is important to explore how the library may influence the lifecycle of research and scholarship by fostering and promoting relevant and unique research, increasing productivity, and enabling research collaborations (association of research libraries, 2019). did growing collections of e-journals over 20 years at the university of illinois chicago (uic) make it easier for researchers to obtain references for their studies, leading to greater impact on their publications? and does this relationship vary across disciplines? this study explored references used in faculty publications, which are earlier works referenced in the footnotes or bibliography of publications, alongside citations, which are references to these faculty publications in later works.   a concurrent publication with a separate focus on the study data (de groote et al., in press) showed an increase over time in the number of journals available to uic faculty through the library’s collection, the number of publications written by faculty per year, the number of references included in the publications, and the number of publications per author. author productivity (i.e., number of publications) was also explored in relation to the number of references used in publications. while highly prolific authors used fewer references overall in their publications, compared to productive authors, unproductive authors used the least number of references in their publications. grant funded research included more references than unfunded research. while the number of references included in non-grant funded publications was not as high for publications that were grant funded, the number of references included in the publications in both groups increased over time. more productive faculty members had a greater number of co-authors on their publications, and grant funded articles had more co-authors than non-grant funded articles.   for this article, researchers further analyzed data from the previous study to learn how faculty use of the literature may influence the later impact of their publications. more specifically, the researchers examined whether faculty use of references in publications has a relationship with the later impact of the publication, measured by citation counts. we also reported on additional factors that may influence the later impact of publications.   literature review   faculty access to scholarly publications has changed over the past 20 years as journals moved from print to online, influencing use patterns of journals (de groote & dorsch, 2001; de groote & barrett, 2010). the number of indexing and abstracting tools increased and moved online (de groote et al., 2007), and the number of journals available at academic institutions increased as the result of so-called big deals (bergstrom & bergstrom, 2004). faculty reported that the online journals and databases allowed for easier access to more literature (brennan et al., 2002). the density of references, measured by references per page, rose between 2001 and 2016 in articles in all disciplines except the arts and humanities, where reference density was already extremely high in 2001 and remained static (sánchez-gil et al., 2018).   a review of studies on factors influencing later citations focused on three categories: (1) paper-related factors (e.g., open access status of paper, number of authors, length of paper, number of references), journal-related factors (e.g., journal impact factor, language of journal, scope of journal, form of publication), and author-related factors (e.g., number of authors, funding) (tahamtan et al., 2016). studies on the impact of publications, as measured by their citation rates, included assessing the impact of the national institutes of health (nih) public access policy on the citation rates of journal articles (national institutes of health, 2008). in general, nih funded articles were cited more than non-nih funded articles in the same journals (de groote et al., 2015).   some studies have concluded that the more authors were credited on an article, the more the article was cited, although there were also studies that contradicted this finding (tahamtan et al., 2016). bornmann et al. (2014) found that the number of authors, the number of references, and the number of pages tended to improve citation rates in a short window of time following publication. however, if multi-authored papers took longer to complete and publish, this advantage was lost. an early study reported that longer papers tended to be cited more than shorter ones, but “the larger citation rate for many-authored papers is not due to the higher citation rate for longer papers because they tend to be shorter than few-authored papers” (abt, 1984, p. 746). given the changing landscape for identifying and accessing the literature, it is important to also explore a potential change in citation patterns, as well as changes related to co-authorship and publication lengths.   past studies have explored factors that potentially influenced the later impact of faculty journal articles (measured by number of citations), including the number of references included, journal prestige, co-authorship, and self-citation. investigating the number of references included in sociology journal publications, lovaglia (1991) found the total number of references influenced the likelihood of the articles being cited later. researchers also observed that articles in more prestigious journals received more citations, but they were unable to conclude whether the increased citations were influenced by the prestige of the journal directly or indirectly, or whether the prestigious journals published better articles. in contrast, a study of malaysian review papers and 500 highly cited papers showed a positive relationship between the number of references included in the paper and the number of citations, but it was not significant (ale ebrahim et al., 2015). lin and huang (2012) examined the relationship between co-authorship and author self-citations and discovered that authors were more likely to cite their own co-authored articles compared to their sole-authored articles. another study found that the more an author cited their own work, the more they were also cited by other scholars (fowler & aksnes, 2007).   most previous studies of the number of references and later impact of the paper in terms of citations have examined articles in specific disciplines or sets of journals. however, it is critical to understand how expanding access to information, such as the increase in online journal collections through big deals, open access journals, and increased access to databases that facilitated identification to journals articles may have influenced the use of existing literature (references) in publications. given that few studies have investigated whether the increased accessibility to online journals and databases has had an impact on the number of references included in publications and their later impact, in this study, we aimed to fill the gap by focusing on faculty from one institution over a 20-year period (1995-2015).   the primary goal of the current research project was to examine how the use of references and other variables influenced or correlated with the later impact of publications. to address these core questions, various usage statistics were collected: literature use (measured by the number of references in the publications), productivity (number of publications by faculty), publication impact (measured by the number of citations), number of co-authors, grant funding, page counts, and faculty demographic information (status and years at the institution). our research questions were:   ·        in what ways do the use of references correlate with later impact of publications? ·        what other variables (e.g., faculty’s demographics, co-authorship, grant funding and page counts) influence later impact of publications? methodology   the impact of the increased access to the literature, through factors such as library big deals, open access journals, and online database on research was explored by examining the publications of uic faculty, during the time they were at uic, using publication data obtained from the scopus abstract and citation database.   types of datasets and procedures of data collection   identification of faculty   to explore if the publication patterns of faculty at uic changed over time, a list of tenure system faculty members who had been at uic for at least 5 years was requested from the office of institutional research (oir). the 5-year period is a reasonable timeframe for faculty at the university to publish and accumulate impact from their publications. data received from the oir included faculty rank, college, department, and number of years at uic. faculty located at uic’s regional locations were omitted from the study because prior to the availability of online journals, they would have had access to much smaller print collections, and thus their reference patterns prior to the online journals could have varied from those at the main university location. in addition, faculty from the arts and humanities were omitted because they typically publish in journals far less frequently than faculty in other disciplines. also excluded were authors in fields where publications typically involve a large number of co-authors (i.e., physics). disciplines in the liberal arts and sciences were grouped in broader categories: chemistry, biological sciences, earth and environmental sciences, and math, statistics and computer science were recoded to natural sciences; sociology, economics, political science, psychology, african american (now black) studies, communication, latin american and latino studies, anthropology, criminology, and law and justice, were recoded to social sciences. other disciplines examined remained grouped by the main college discipline which included applied health science, business administration, medicine, dentistry, education, engineering, library, nursing, pharmacy, public health, social work, and urban planning and public affairs. we grouped faculty based on how long they had been at uic, and we omitted older publications written by authors before they became faculty members of uic (see table 1).     table 1 publications explored based on years at uic years at uic cut-off for publications explored 5 years no older than 2015 10 years no older than 2010 15 years no older than 2005 20 years no older than 2000 25 years no older than 1995     we searched within scopus for each author’s publications. for each publication, we recorded the author’s name, number of references used in the publication, number of citations received by the article within a 5-year and 10-year timeline of publication, how many authors were involved in the publication, the length of the publication, and whether the publication was grant funded.   procedures of recording publications   the list of faculty authors was divided up among the investigators. detailed instructions were provided to each investigator and the investigators met after an initial collection of data to ensure uniformity with the process and the data. to retrieve the data from scopus, investigators selected the author tab and entered the last name and first name of the faculty member. investigators selected the result(s) for the author if their name was a match along with the affiliation. if the investigators retrieved more than one result for an author by the same name and institution, then all were selected to obtain the full list of faculty publications. on the left-hand side of the screen, the year facet was used to exclude publications outside of the date range predetermined for the faculty member. the document type facet was used to limit results to “articles.” the main goal was to limit results to research articles. limiting to “articles” did not guarantee that only research articles were included, but it did eliminate most review articles and other article types such as editorials or conference papers. review articles were excluded because they tended to include a disproportionate number of references compared to research articles.   next, the investigators selected and exported all publications remaining in the list. funding details were included in addition to the bibliographic data selected by default in scopus (authors, title, journal name, volume, issue, pages, doi). then, investigators copied and pasted the contents of the file into a master file. an additional column in the spreadsheet contained an assigned uic author id for author, so publications by that author could be counted. finally, investigators manually searched for each publication in scopus and retrieved the number of references included.   procedures of recording later impact   to determine whether the number of references included in a publication had a later influence on the number of citations the article received, investigators also recorded the number of citations that an article received. this information was recorded in additional columns in the spreadsheet of publications. to obtain this information for each publication, the investigator scrolled to the top of the page for the publication and clicked on “view all metrics.” on the metrics page, self-citations were excluded to avoid the impact of authors citing their own work. to avoid issues with older articles receiving more citations because they had been around longer, we standardized the age of the citations that were counted by adjusting the date range relative to the age of the article to obtain the citations that an article had received in the last 5 years and the last 10 years (see table 2).     table 2 date range for citations obtained based on the year of publication year 5 years 10 years 2015-2019 n/a n/a 2014 2014 – 2018 n/a 2013 2013-2017 n/a 2012 2012-2016 n/a 2011 2011-2015 n/a 2010 2010-2014 n/a 2009 2009-2013 2009-2018 2008 2008-2012 2008-2017 2007 2007-2011 2007-2016 2006 2006-2010 2006-2015 2005 2005-2009 2005-2014 2004 2004-2008 2004-2013 2003 2003-2007 2003-2012 2002 2002-2006 2002-2011 2001 2001-2005 2001-2010 2000 2000-2004 2000-2009 1999 1999-2003 1999-2008 1998 1998-2002 1998-2007 1997 1997-2001 1997-2006 1996 1996-2000 1996-2005 1995 1995-1999 1995-2004     data preparation   once the data were collected, data from each investigator were merged into two spreadsheets. the author summary spreadsheet summarized the publications of each faculty member, and the publication details spreadsheet listed the publications of each author.   the publications details spreadsheet, in addition to including author(s), title, journal name, and date, also included the author id (assigned by uic), the number of references included in the publications, the number of times each publication was cited in a 5and 10-year period (for articles at least 5 years old), and the discipline of the author. the number of commas used to separate author names was tabulated using a formula in excel plus one (because there is one fewer comma than authors in the list of authors) to indicate the number of authors in article, so that these data became available as a variable. in addition, the length of an article was determined through a formula in excel, to subtract page start from page end, and these data became available as a variable in the spreadsheet. if some type of grant funding was acknowledged in the funding details, then in a separate column the investigators coded the article as grant funded.   the author summary spreadsheet summarized the publication information for each author. the total number of publications, total number of references, and average number of references for each author was recorded for the following time periods as appropriate: 1995-1999, 2000-2004, 2005-2009, 2010-2014, and 2015-2019. for each author, it was noted how long they were at uic, how many publications they produced within 5-year increments, and the average number of references included in these publications. once the spreadsheets were merged, they were further cleaned.   for the author summary spreadsheet:   ·        authors who did not have a consistent publication record (i.e., there were no publications in the last 5 years of the study) were removed from the study (n=43). ·        faculty who did not have any publications were removed from the study (n=52).   for the publication spreadsheet:   ·        the publications of the authors removed from the author summary spreadsheet were also removed from the study. ·        in some cases, scopus did not include the number of references that an article received. these publications were dropped from the study and the author’s publication number updated in the author summary spreadsheet. this occurred primarily with publications published between 1995 and 1999. a total of 390 articles were removed for this reason. some articles had multiple authors being faculty members of uic, which created 4361 duplicate records when we retrieved articles for each of them (total publications included = 24702). removal of duplicate publications did not impact the author summary spreadsheet.   data analysis   the investigators used spss 27 to run several statistical tests, including pearson correlations and regression analysis. pearson correlations were used to examine:   1.      whether the number of references in an article correlated to the number of citations the article received (later impact). 2.      the relationships between the number of references and the number of citations at the discipline level. 3.      correlations between the number of references, the number of citations, and the number of authors for articles that were grant funded and articles that were not funded.   in this article, guided by cohen’s (1988) criteria, the strength of correlation of r less than .30 was considered as small, r between .30 and .49 as medium, and r between .50 and 1.0 as large. multiple regression analysis determined which factors predicted the citations that articles later received. additionally, scatterplots visualized the relationships between the number of references and the number of citations for the first 10 years after publication.   results   we examined the publications of 802 faculty from the following disciplines: applied health sciences (28), business administration (28), dentistry (35), educations (23), engineering (80), library (12), medicine (322), natural science (92), nursing (20), pharmacy (36), public health (34), social science (64), social work (8), and urban planning and public affairs (20). the average references, citations, and co-authors per article over the years are displayed in table 3.     table 3 publication demographics 1995-1999 2000-2004 2005-2009 2010-2014 2015-2019 average/ all years total publications 1178 1972 3967 6827 10758 no. authors studied 118 224 376 582 802 aver. pub/ author 10 8.8 10.55 11.7 13.4 10.89 average co-authors 4.5 5 5.47 6.66 12.76 6.88 average references 29.52 35.24 37.5 40.87 44.32 37.49 average citations first 5 years 10.64 12.65 13.76 14.26 n/a 12.83     between 1995 and 1999, the average number of citations articles received after 5 years was 10.64 compared to 14.26 between 2010 and 2014. over the years, articles have received increasing numbers of citations (figure 1), when the years of the articles published were held constant, compared to averages from the past. articles received a greater proportion of citations between 6-10 years of age, compared to their first 5 years, which demonstrated that the impact of articles were not immediate, and the greater impact will typically be observed several years after an article was published. for articles written between 1996 and 2009, the number of references showed a weak correlation with the number of citations an article received (r (6160) = .180, p < .001), suggesting that the number of references included in an article may contribute to the later impact of the article.     figure 1 average citations received by year of publication for the first 5 years, 6-10 years, and all 10 years.     table 4 correlations between number of references and number of citations – publications since 2000 at the discipline levela discipline r p number of publications applied health sciences .057 .424 196 business administration .553 .000** 67 college of medicine (chicago only) .131 .000** 1979 dentistry .470 .000** 183 education .441 .000** 59 engineering .200 .000** 881 library .251 .386 14 natural sciences .162 .000** 743 nursing .272 .028* 65 pharmacy .043 .457 304 school of public health .232 .000** 284 social sciences .200 .000** 362 social work .354 .025* 40 urban planning & public affairs .402 .000** 75 all colleges .175 .000** 5252 a *p < .05, ** p < .01.     the relationship between the number of references included in an article and the subsequent citations an article received was also explored at the discipline level. there was a weak positive correlation between the number of references included in publications and the later impact of the article in all disciplines except for applied health sciences, the university library, and pharmacy (table 4). this suggested that in most of the disciplines, more references in a publication may have some influence on the number of citations later received by the publication.   the number of citations an article received had a weak correlation with the number of authors (r (5252) = .136, p < .001). this relationship between the number of authors and citations was observed despite self-citations being excluded from the data. to further explore the relationship between the number of co-authors and references included, cited references, and page count, publication data were separated between publications with one to eight authors and nine or more authors. the average references, average citations, and average page counts were obtained for the publications over time (see table 5). articles with one to eight authors had higher page counts compared to articles with nine or more authors. articles with nine or more authors had more references and were cited more than articles with one to eight authors. overall, the number of references included in publications in both groups increased over time. while the number of citations per article with one to eight authors generally increased over time, the average number of citations per article for articles with higher co-authorship (nine or more) decreased over time (1995-1999 to 2010-2014), although these articles were still cited more than articles with less authors.   the investigators also compared the number of references included in grant funded and non-grant funded publications, as well as their later impact. only publications published in 2010 and later were explored because of concerns with underrepresentation of grant funded articles as reflected through indexing in the scopus database (liu, 2020). a more in-depth explanation of grant funded articles reported through scopus is provided in the concurrent paper (de groote et al., in press). as a result, scopus funding data were used to explore but not confirm publication pattern differences between grant funded and unfunded publications. grant funded articles included more references than non-grant funded articles, and the number of citations received by grant funded articles was higher, compared to non-grant funded articles (table 6). grant funded articles also had more authors than unfunded articles.     table 5 changes in number of references, citations (first 5 years), page counts over time by co‑authorship size 1995-1999 2000-2004 2005-2009 2010-2014 average 1 to 8 authors total publications 1007 1599 2798 4175 2395  average references 29.36 39.4 37.87 40.75 36.85  average citations 8.36 10.55 12.31 12.32 10.89  page count * 8.64 9.50 9.67 9.86 9.42 9 or more authors total publications 92 157 404 820 368  average references 31.86 36.5 40.69 45.56 38.65  average citations 35.6 29.19 22.54 26.26 28.4  page count a 6.57 7.36 8.27 8.96 7.79 a publications without page counts and duplicates were excluded from the analysis.     table 6 average number of references, average number of citations, and average number of authors per publication by funding since 2010 funding number of references number of citations in first 5 years (2010-2014) number of authors unfunded mean 41.2 12.99 7.36 na 7352 3536 7352 funded mean 46.3 17.04 14.6 n 6638 2039 6638 a number of articles; duplicates were excluded from the analysis.     articles published between 2010 and 2019 showed a significant but weak negative correlation between the number of authors on a paper and its page count (r (13880) = -.022, p < .01). a positive correlation existed between the number of references included and the page count (r (13880) =.330, p < .01), and also a weak positive correlation between the page count and subsequent impact of the paper as measured through citations (r (13880) = .059, p < .001). articles published between 1995 and 2009 showed a weak but positive correlation between average citations and page count (r (6305) = .038, p < .01, and articles published between 1995 and 2014 had a weak but positive correlation between average citations and page count (r (6305) = .023, p < .05).   the investigators conducted a regression analysis to determine what factors predicted the citations that articles later received. number of references, number of authors, and whether an article was grant funded were explored as predictors of later research impact, based on the number of citations an article gets after five years. the overall model with the three predictors significantly predicts the impact of an article (r2=.16, r2adj = .16, f (3, 6822) = 435.67, p < .001). among the three predictors, all three were significant, although grant funding was right on the cusp of being insignificant (number of references: t = 23.32, p < .001, number of authors: t = 25.79, p < .001, funding received: t =1.96, p = .05). the number of references, number of authors, and grant funding all contribute to the later impact of an article.   discussion   the results of this study reflected a change in publication patterns over time. the total number of references included in articles has increased over time from 1995 to 2019. this finding was similar to that of sánchez-gil et al. (2018), who observed an increase in references in publications from 2001 to 2016. it is likely that several variables related to the increase in references. authors had direct access to more articles in more journals through big deals. in addition, the increase in open access journals also increased the number of articles directly available to authors. references may also have increased due to the increase in and availability of online databases facilitating the identification of relevant literature.    through this study, the researchers also observed that the average number of authors on an article, the total number of page numbers in an article, and the total number of citations articles received have been increasing over time. this finding prompted the question: does a greater number of references included in a publication result in greater impact later? this study demonstrated a weak correlation between the number of references included in a publication and the later number of citations an article received, even when self-citations were excluded, suggesting that the number of references included in a publication may contribute to the later impact of a paper. like the findings by lovaglia (1991), the total number of references included in a publication appeared to have a relationship with later citations. it is speculated that the greater number of references used in a paper contributed to its overall quality, and thus influenced its later impact, rather than the direct influence being the number of references themselves.   through this study, the researchers also demonstrated that research articles were cited more on an annual basis between 6 to 10 years than in their first 5 years, which indicated that articles need time before their true impact is known. often, the recognition of impactful scholarship has occurred a short period of time after the publication. for example, at uic, tenure decisions are typically made during a faculty member’s sixth year, but this study indicated a researcher’s greatest impact would be observed after this evaluation period. relying on citation metrics for retention decisions for relatively new publications would not have captured the full potential of impact for the publication.   the overall number of co-authors per publication increased over time. while publications with greater numbers of authors (nine co-authors or more) received greater numbers of citations, the average number of citations received by high co-authorship articles (nine or more co-authors) decreased over time. a weak correlation also existed between the number of authors and the number of citations an article received. also observed in this study, articles with more co-authors later receive more citations. self-citations were excluded, so it was not the situation that there were more authors to cite their own work. however, colleagues might have been more likely to cite the work of their colleagues. further investigation could explore if coauthored publications from singular colleges or universities are cited less than multi-institutional co-authored papers. the suggested implication was that coauthored papers from multi-institution authored articles will have a greater network of non-author colleagues from multiple institutions who could cite the work, whereas single institution authored articles would be more limited to the non-author colleagues at one institution. the increase in the average number of authors on publications over time may have been related to an increased emphasis on team science both at uic itself and in the broader research ecosystem (cooke & hilton, 2015), although a general increase in the overall number of faculty at uic may also have played a role in the increased co-authorship. from 2006 to 2019, the total number of teaching faculty at uic increased from 1163 to 2817 faculty (association of research libraries, 2021). further investigation is needed to determine to what extent the emphasis on collaboration and the increased faculty size had on increased co-authorship.   funded publications included more references than non-funded publications, and like the findings of de groote et al. (2015), grant funded publications received greater citations than non-grant funded publications. it is plausible that the oversight, requirements, and accountability that accompany funding may have resulted in higher quality research. this could also mean that the literature was more thoroughly explored, which led to both more references and more high-quality publications, which in turn influenced an article’s later impact. further investigation is needed to understand this observation.   the number of references, the number of authors, and grant funding were all variables contributing to the later impact of publications. similarly, bornmann et al. (2014) noted that the average number of references, authors, and page numbers all tended to have a positive relationship with citation rates. a regression analysis demonstrated that the number of references, the number of authors, and whether a publication was grant-funded all played a role in predicting the later impact of an article. this implied that writing grant funded publications with multiple co-authors and references altogether could increase the later impact of the publications. it is possible that the use of more references in an article, a greater number of co-authors, and having grant funding influenced the quality of an article, thus increasing the chances that an article was cited.   limitations   because scopus results were limited to the somewhat imprecise document type articles, the data may not have included only research articles, and some relevant research articles may have been omitted. scopus is also not fully comprehensive, as only citations from journals indexed in scopus were captured. because self-citations were excluded from the data collection, comparisons of citation counts with and without self-citations were not possible. in this study, the researchers only explored citations 10 years out from the publication date, and as a result, it is not known when annual citations to articles would begin to decline. are the 6th to 10th years when an article will have its greatest impact, as supported by the results here, or does the timeline extend beyond this? additionally, in some disciplines, such as computer science, scholarship is often published through conference proceedings rather than as journal articles. this could imply that scholarly publications were underrepresented in this study for some disciplines where their primary publications were in the form of conference proceedings. lastly, the authors would also acknowledge that collecting data was time intensive, and so future comparative research should depend on technological solutions for compiling data when possible. this would also likely reduce the potential for human error in the data collection process.   conclusion   in this study, the researchers examined the number of references included in articles and how that may have related to the later citation impact of the publication, through faculty publication data gathered from scopus. variables such as page counts, number of coauthors, and grant funding were also explored as contributors to citation impact. articles were cited more when they were 6 to 10 years old, compared to the earlier period following publication. over time, authors have included more references in their publications, and articles were also being cited more than they were in the past. co-authorship also increased over time. there were several variables that correlated with the later impact of a publication, as measured by citations, including number of references used, number of co-authors, and whether an article was grant funded.   author contributions   sandra l. de groote: conceptualization, methodology (lead), data curation, formal analysis, writing – review & editing jung mi scoulas: methodology, formal analysis (lead), writing – original draft, review & editing paula r. dempsey: writing – review & editing felicia barrett: writing – review & editing   acknowledgement   the authors acknowledge the support of the association of research libraries (arl) research library impact framework initiative, which was established in 2019 with a grant from the institute of museum and library services (imls). special thanks to deborah blecic, who contributed to data collection.   references   abt, h. a. (1984). citations to single and multiauthored papers. publications of the astronomical society of the pacific, 96(583), 746–749. https://doi.org/10.1086/131415   ale ebrahim, n., ebrahimian, h., mousavi, m., & tahriri, f. (2015). does a long reference list guarantee more citations? analysis of malaysian highly cited and review papers. the international journal of management science and business, 1(3), 6–15. https://ssrn.com/abstract=2572789   association of research libraries. (2019). research library impact framework initiative and pilots. https://www.arl.org/category/our-priorities/data-analytics/research-library-impact-framework/   association of research libraries. (2021). arl statistics. https://www.arlstatistics.org/   bergstrom, c. t., & bergstrom, t. c. (2004). the costs and benefits of library site licenses to academic journals. proceedings of the national academy of sciences, 101(3), 897-902.   bornmann, l., leydesdorff, l., & wang, j. (2014). how to improve the prediction based on citation impact percentiles for years shortly after the publication date? journal of informetrics, 8(1), 175–180. https://doi.org/10.1016/j.joi.2013.11.005   brennan, m. j., hurd, j. m., blecic, d. d., & weller, a. c. (2002). a snapshot of early adopters of e-journals: challenges to the library. college & research libraries, 63(6), 515–526. https://doi.org/10.5860/crl.63.6.515   cohen, j. (1988). statistical power analysis for the behavioral sciences (2nd ed.). l. erlbaum associates.   cooke, n. j., & hilton, m. l. (eds). (2015). enhancing the effectiveness of team science. the national academies press. https://www.ncbi.nlm.nih.gov/books/nbk310394/   de groote, s. l., & barrett, f. a. (2010). impact of online journals on citation patterns of dentistry, nursing, and pharmacy faculty. journal of the medical library association: jmla, 98(4), 305–308. https://doi.org/10.3163/1536-5050.98.4.008   de groote, s. l., & dorsch, j. l. (2001). online journals: impact on print journal usage. bulletin of the medical library association, 89(4), 372-378.   de groote, s. l., scoulas, j.m., dempsey, p.r., & barrett, f. (in press). faculty publication patterns over 25 years at a large public university. college & research libraries.   de groote, s. l., hitchcock, k., & mcgowan, r. (2007). trends in reference usage statistics in an academic health sciences library. journal of the medical library association, 95(1), 23-30.   de groote, s. l., shultz, m., & smalheiser, n. r. (2015). examining the impact of the national institutes of health public access policy on the citation rates of journal articles. plos one, 10(10), e0139951. https://doi.org/10.1371/journal.pone.0139951   fowler, j., & aksnes, d. (2007). does self-citation pay? scientometrics, 72(3), 427–437. https://doi.org/10.1007/s11192-007-1777-2   lovaglia, m. j. (1991). predicting citations to journal articles: the ideal number of references. the american sociologist, 22(1), 49–64. https://doi.org/10.1007/bf02691867   lin, w. y. c., & huang, m. h. (2012). the relationship between co-authorship, currency of references and author self-citations. scientometrics, 90(2), 343–360. https://doi.org/10.1007/s11192-011-0523-6   liu, w. (2020). accuracy of funding information in scopus: a comparative case study. scientometrics, 124(1), 803–811. https://doi.org/10.1007/s11192-011-0523-6   national institutes of health. (2008). public access policy. https://publicaccess.nih.gov/   sánchez-gil, s., gorraiz, j., & melero-fuentes, d. (2018). reference density trends in the major disciplines. journal of informetrics, 12(1), 42–58. https://doi.org/10.1016/j.joi.2017.11.003   tahamtan, i., safipour afshar, a., & ahamdzadeh, k. (2016). factors affecting number of citations: a comprehensive review of the literature. scientometrics, 107(3), 1195–1225. https://doi.org/10.1007/s11192-016-1889-2     evidence summary   librarian authors appear to favour open access journals, while academic authors appear to favour non-open access journals   a review of: chang, y.-w. (2017). comparative study of characteristics of authors between open access and non-open access journals in library and information science. library & information science research, 39(1), 8-15. https://doi.org/10.1016/j.lisr.2017.01.002   reviewed by: michelle dubroy discipline librarian (researcher services) griffith university library southport, australia email: m.dubroy@griffith.edu.au   received: 31 july 2020                                                              accepted:  30 oct. 2020                       2020 dubroy. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29812     editors' note: for an additional perspective on the original article, please see https://doi.org/10.18438/eblip29377   abstract   objective – to compare the characteristics of authors publishing in open access and non-open access library and information science (lis) journals.   design – comparative analysis of published journal articles.   setting – academic journals.   subjects – articles published in selected lis journals between 2008-2013.   methods –  journals included in the library science and information science category in the 2012 edition of journal citation reports and those listed in the library and information science category of the directory of open access journals as of may 2013 were included in the analysis. articles were examined and coded for author occupation, academic rank, and type of collaboration.   main results – the author analyzed 1,807 articles from 20 open access journals and 1,665 articles from 13 non-open access journals. an unknown number of articles were excluded because they lacked required author information. over half (53.9%) of the authors who published in the open access journals were practitioners. over half (58.1%) of the authors who published in the non-open access journals were academics. librarian-librarian collaboration was the most common type (38.6%) of collaboration found in the open access journals. academic-academic collaboration was the most common type (34.1%) of collaboration found in the non-open access journals. collaboration between librarians and academics was seen in 20.5% of open access articles and 13.2% of non-open access articles.   conclusion – in general, librarian-authored research was found more often in open access journals, while the “latest research topics and ideas” (p. 14) were found most often in non-open access journals.   commentary   a research-practice divide has been said to exist in library and information science (lis) for decades (booth, 2003). this study appears to confirm the divide.   the study was evaluated using two critical appraisal tools (perryman, 2009; perryman & rathbun-grubb, 2014). despite some weaknesses, the study and its findings are worth considering.   the literature review was useful and supported the research objectives and methodology. the methods used were a logical fit for the research questions.   the author outlined her process with enough detail to allow others to replicate it. further, she reported her findings clearly and made good use of tables and figures. additionally, she discussed at least some of the study’s limitations.   yet, a few points remain obscure. the author stated that journals had to meet six criteria, including being “indexed by at least two of four lis databases” (p. 10). she then stated that she selected journals from journal citation reports and directory of open access journals. the overlap between these requirements is unclear. additionally, the author did not disclose how many articles did not meet inclusion criteria.   findings of this study appear to be at variance with those of dalton (2013). through an online questionnaire, dalton found no significant difference in the open access publishing preferences of librarians and lis academics. the author of the present study did not discuss this apparent discrepancy. in fact, she appeared to assume publication outcomes were solely based on author preferences and “loyalty” (p.14). manuscripts, however, are not necessarily published in the first journal to which they are submitted. thus, submission behaviour and journal rejection rates should also be considered.   regrettably, the most recent articles analyzed in the study are from 2013. thus, considering the rapid rate of change in the scholarly publishing landscape, it is doubtful these findings are relevant to present-day publishing practices. transferability to disciplines outside of lis is uncertain.   this study will be of interest to any librarian who has a high degree of interest in open access publishing. an update, however, is needed. understanding how recent initiatives, such as the san francisco declaration on research assessment (american society for cell biology, 2013), have affected lis publishing preferences and practices could prove illuminating. it would also be worthwhile to expand the analysis to include a wider selection of journals. further, it may be useful to consider submission behaviour and journal rejection rates in any future analysis.   references   american society for cell biology. (2013). the san francisco declaration on research assessment (dora). retrieved from https://sfdora.org/   booth, a. (2003). bridging the research-practice gap? the role of evidence based librarianship. new review of information and library research, 9(1), 3-23. https://doi.org/10.1080/13614550410001687909   dalton, m. (2013). a dissemination divide? the factors that influence the journal selection decision of library and information studies (lis) researchers and practitioners. library and information research, 37(115), 33-57. https://doi.org/10.29173/lirg553   perryman, c. (2009). evaluation tool for bibliometric studies. retrieved from: https://www.dropbox.com/l/scl/aaal7luzple90fxfnbv5hcnoz0ctlh6rqrs   perryman, c. & rathbun-grubb, s. (2014). the cat: a generic critical appraisal tool. in jotform – formbuilder. retrieved from http://www.jotform.us/cp1757/thecat   research article   improving learner-driven teaching practices through reflective assessment   matthew t. regan instructional services program leader montana state university bozeman, montana, united states of america email: mtregan@montana.edu   scott w. h. young user experience & assessment librarian montana state university bozeman, montana, united states of america email: swyoung@montana.edu   sara mannheimer data librarian montana state university bozeman, montana, united states of america email: saramannehimer@montana.edu   received: 7 feb. 2020                                                                 accepted: 1 july 2020      2020 regan, young, and mannheimer. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29729     abstract   objective – reflective assessment is an effective method of teacher evaluation, serving as an approach for assessing teaching practices, generating insights, and connecting with colleagues, ultimately supporting meaningful transformation of teaching practice. in this paper, three librarians model a reflective assessment approach in evaluating and improving their experiences implementing learner-driven teaching practices in credit-bearing courses in topics related to library and information studies.   methods – following a model of reflective assessment, we asked ourselves how our practice can better support learner-driven teaching practices, thus assessing and improving our own teaching and improving students’ learning experiences. our process involved five steps: cohere around shared viewpoints, identify teaching practices for reflection, conduct reflection, discuss and analyze reflections to produce insights, and apply insights to improve teaching.   results – we reflect on five different learner-driven teaching practices: co-creative syllabus design, learner-defined personal learning goals, soliciting and responding to learner feedback, interdisciplinary discussions and exercises, and self-evaluation. we discuss improvements and refinements that we implemented in response to our reflective assessment, including more frequent checking in with students; more clarity regarding self-evaluation and grading; one-on-one meetings with all students; allowing students to negotiate, discuss, and determine assignment deadlines and dates; more flexibility with students’ work products; and increased pedagogical transparency. as a further result, our reflective process models an approachable framework for engaging in reflective assessment.   conclusion – this paper presents a model for reflective assessment of teaching in an academic library. we present a discussion of learner-driven teaching practices, and we offer a practical pathway for other teachers and practitioners to assess their teaching. we find that reflective assessment is an effective and insightful approach for understanding and improving learner-driven teaching practices.     introduction   as academic librarians continue to assume the role of teacher—in both traditional reference and bibliographic instruction contexts, and in delivering credit-bearing courses—the role of teacher assessment will continue to become more important. at montana state university (msu), library faculty members regularly teach credit-bearing courses on topics related to information science, including library research skills, data curation, information ethics, and privacy. to improve our teaching practice, we have followed an approach of reflective assessment.   this article demonstrates a reflective practice to assess teaching practice. our individual and shared experiences led us to articulate and cohere around a shared viewpoint: to develop and improve learner-driven teaching practices. we evaluate via reflective assessment six learner-driven teaching practices that we employ in our teaching. we find that reflection is an effective method of teacher evaluation, serving as an approach for assessing pedagogy, generating insights, and connecting with colleagues, ultimately supporting meaningful transformation of teaching practice. in reference to credit-bearing courses taught by librarians, burke (2012) noted that many librarians “find developing assessment tools daunting” (p. 169) due to insufficient teacher training. in sharing our own approach to reflective assessment of teaching, we consider this paper a response to burke’s call: “the author hopes that librarians, who have developed successful assessment strategies, will share their experiences with the larger academic library community in the not too distant future” (p. 169).   literature review   librarians as teachers   vassilakaki and moniarou-papaconstantinou (2015) identified six emerging roles for information professionals, one of which is “librarian as teacher”—referring to librarians’ active engagement in teaching and learning processes. in academic libraries, forms of teaching can vary widely, from providing workshops and guest presentations to teaching semester-long, credit-bearing courses. as vassilakaki and moniarou-papaconstantinou wrote, “it appears that librarians as teachers assume a range of responsibilities associated with teaching and learning and, thus, their educational role continues to develop in a way that incorporates them entirely into the academic community” (p. 41). cohen et al. (2016) identified credit-bearing courses as the least common form of teaching performed by librarians, amounting to only 19% of library teaching efforts, but also suggest that there is “growing recognition of the teaching mission of the library on campus” (cohen et al., 2016, p. 576). similarly, loesch (2017) described the foundations of library research assistance as being essentially oriented toward teaching learners, solidifying the role of teacher as an appropriate and even fundamental function for librarians. the association for college and research libraries (acrl) (2017) also articulated the role of librarian-as-teacher, emphasizing “activity in the classroom or other instructional environments where the librarian interacts directly with learners.” in this role, librarians enter the instructional environment prepared to deliver learner-centered instruction, establish goals and learning outcomes, employ “innovative instructional techniques and tools,” and perform assessment. of note, librarians demonstrate “enthusiasm for teaching and learning and a commitment to professional development, lifelong-learning, and reflective practice” (“teacher” section, bullet 6). the landscape of librarian-as-teacher is promising for further reflective investigation.   assessment of teaching in academic libraries   analyses of teacher evaluation methods are well-represented in the literature. ackerman, gross, and vigneron’s (2009) comprehensive review of the literature succinctly highlighted serious concerns surrounding common assessment methods such as student evaluations of teaching and peer observation reports, especially as related to validity, bias, subjectivity, and abuse. subsequent research has demonstrated student evaluations of teaching  to be inaccurate and problematic, especially as they are likely to reflect biases such as racial and gender stereotypes (boring, 2017; hornstein, 2017; mengel, sauermann, & zölitz, 2019; mowatt, 2019). assessing one’s own teaching can also potentially be “time consuming, daunting, and disheartening” (mccartin & dineen, 2018, p. 47). assessment of teaching is further complicated by the unique role of the librarian-teacher. in a 2012 study, respondents to the authors’ survey reveal a somewhat cynical perspective: “feedback is often collected randomly . . . in the event that there would be a need to ‘prove my success as a teacher someday’” (cunningham & donovan, 2012, p. 198) rather than to demonstrate an impact on student learning or to improve teaching. the question of who owns the assessment process is also a paramount consideration: those engaging in teaching assessment must assume personal ownership of the process in order to support personal responsibility and investment in the outcomes (mcgivney, 2017). in light of these concerns related to traditional methods of teacher assessment, alternate modes of assessment—such as reflective assessment—can serve as a model for an intentional assessment practice that can improve teaching practice.   reflective assessment in libraries   reflection is a process of thinking about ourselves and the world (reale, 2017). it can also be a legitimate method of assessment (moon, 2004). black and plowright (2010) offered a working definition of reflective assessment: “reflection is the process of engaging with learning and/or professional practice that provides an opportunity to critically analyse and evaluate” (p. 246). macaluso (2014) furthermore told us that “the greater purpose of self-reflection is to develop a sense of where one’s strengths and areas for improvement lie and what steps one might take to improve or grow as an educator” (p. 124). within this frame, the growth-oriented act of reflection becomes a legitimate tool of assessment (graf & harris, 2016).   reflective practice, however, has not always been present within librarianship. in calling for a practice of reflection, doherty (2005) noted that librarians at that time were “not very reflective practitioners” (p. 12). six years later, booth (2011) identified reflective practice as an element of effective instructional literacy and called for librarians to reflect upon prior experience as an act of assessment that can improve future teaching and learning. and more recently, corrall (2017) demonstrated that reflective practice is an emerging standard for teacher-librarians, as represented in textbooks, journal literature, and case study analysis that focuses on the adoption of reflection to plan, evaluate, and improve instruction; identify professional development needs; and foster a reflective teaching culture.   at this point, the act of reflection can be found across librarianship (association of college and research libraries, 2016), and particularly within the context of information literacy and instruction (downey, 2016; tewell, 2018). for example, mccartin and dineen (2018) describe an assessment practice that empowers students as participants in their own learning and assessment, along with reflective teacher self-assessments. they write that “continual assessment of teaching is essential to professional growth” (p. 47), noting that dialoguing with peers is a fruitful approach for reflective assessment. for sen and mckinney (2014), reflective writing proved effective for assessing information literacy among undergraduate students, who reflected on their own actions past and present while looking forward to future learning and professional practice. and for deitering, rempel, and jensen (2018), reflection was a successful element of assessment within a cohort of graduate student instructors.   background and institutional context   in this section, we describe our institution, the classes that we reflected upon, and a statement about our teaching philosophy, which centers around learner-driven pedagogy.   msu is a mid-sized, land-grant university. msu has a carnegie classification of “very high research activity” and a “very high undergraduate” enrollment profile (carnegie classification of institutions of higher education, 2020). in 2019, the university enrolled 16,766 students and employed 600 tenure-track faculty, 14 of whom are in the library. the msu library offers courses under the lsci rubric. one course, lsci 121: library research skills is permanently numbered and is offered every semester. other courses offered are usually “special topics” courses that are only offered once or twice.   courses   the courses assessed in this paper are:   ●       lsci 121: library research skills ○       library research skills is a course focusing on both the concepts and skills needed to conduct library research with an emphasis on electronic information sources. this course is offered every semester, has been taught in multiple modalities (face-to-face, online, and blended), and is required for some majors. undergraduate students—most often first and second year—from a variety of majors typically enroll in this course. ○       this course was taught by young in spring 2015, mannheimer in spring 2017, and regan in spring 2018, fall 2018, and spring 2019. ●       lsci 491: contemporary approaches to community building using social media ○       this course looks at social media from the perspectives of information analysis and community building. utilizing a service learning approach, students engaged in the theories and methods related to social media community building through both in-class discussion and community outreach. this special topics course was not required; 24 undergraduate students from a variety of majors enrolled in the course. ○       this course was taught by young and co-teacher doralyn rossmann in fall 2016. ●       honr 494: information ethics and privacy in the age of big data ○       this course focused on ethical theory and practice from an interdisciplinary perspective by examining the ethical guidelines of various fields, with an additional focus on practical skills related to privacy on the web. this course was an upper-division undergraduate seminar taught for the honors college. eleven students from 10 different majors enrolled in the course. ○       this course was taught by young and mannheimer in spring 2018.   we then applied our reflective assessment insights to the following courses:   ·        lsci 121: library research skills o   this course was taught again by regan in fall 2019. ·        lsci 391: data curation for a data-driven world o   this course uses hands-on activities to teach strategies for active curation and management of data, covering a broad range of practical and theoretical issues in the emerging field of data curation. this special topics course was taught for the first time in spring 2020; six students enrolled in the course—all undergraduate juniors and seniors majoring in computer science. o   this course was taught by mannheimer and co-teacher jason clark in spring 2020. ·        lsci 291: technology, ethics, and society o   this course examined ethical issues in the development, testing, and implementation of emerging technologies. students learned about a variety of different technologies and their potential applications, including crispr, synthetic biology, reproductive technologies, food technologies, geoengineering, artificial intelligence and autonomous systems, robotics, the internet of things (e.g. smart cities), social media, and other information technologies. this special-topics course was not required; 13 undergraduate students from a variety of majors enrolled. o   this course was taught by young and mannheimer, and co-teachers kristen intemann, mary ann cummings, and brock lemeres in spring 2020.   teaching philosophy: learner-driven teaching   as librarians and teachers, we draw together a wide range of interests and sources that inform our pedagogical viewpoint and teaching practices, focusing on social responsibility, participation, and power sharing. these sources include lis professional values, feminist ethics, constructivist theory, transformative learning, and participatory design.   the american library association (ala) stated core values of democracy and social responsibility (american library association, 2019) guide us toward a commitment to sharing power with students and supporting justice in the classroom. the acrl framework for information literacy for higher education (association of college and research libraries, 2016) further inspires us to consider that in the current information ecosystem, learners “have a greater role and responsibility in creating new knowledge, in understanding the contours and the changing dynamics of the world of information, and in using information, data, and scholarship ethically” (p. 7). individual practitioners within the lis profession also influence our thinking, particularly conversations around the ethics of care (nowviskie, 2015), described as an “intentionally feminist ethical framework centering relationships and emotion” (dohe, 2019, “whither the ethics of care” section, para. 1). we look beyond our discipline for additional guidance and inspiration. the professional practice of participatory design is important, as it centers on a “commitment to designing futures that challenge power relationships and transform patterns of exclusion and social injustice” (robertson & wagner, 2013, p. 68).   we also look to established learner-driven and power-sharing strategies to guide our teaching. for example, constructivist theory suggests that new learning builds on prior knowledge (good & brophy, 2008). by understanding students’ past experiences, we can build a trajectory of learning that connects prior knowledge to the current learning objectives, and then extends to a lifelong pattern of curiosity, connection, and knowledge-building (mannheimer & banta, 2018). in our approaches to learner evaluation, we are influenced by the idea of “grade contracts” that provide learners with some guidance for expectations and paths of improvements (danielewicz & elbow, 2009; posner, 2015). we are also influenced by trede and smith (2014), who marked out the complexities and power differentials of grading, recommending “transparent and self-questioning assessment dialogues amongst assessors as well as between assessors, the assessed and those who design assessments” (p. 165). the concepts of minimal grading (berns, 2020; elbow, 1997) and “ungrading” (ferguson, 2013) are also influential to us, providing alternatives to traditional grading structures. in practice, these ideas are built on a foundation of trust. in developing a pedagogy of participation, stommel (2017) said to “start by trusting students” (para. 3).   in evaluating our own teaching, we look to mezirow’s (1990) theory of transformative learning as inspiration for applying reflective assessment as a way to improve teaching practice. as reale (2017) described mezirow: “reflection paves the way for deep understanding in the form of transformative learning. as we teach, we must also think of ourselves as learners, in and among our students and colleagues” (p. 84). the practice of reflection is therefore critical not only for improvement and growth, but also for creating space for participation and trust in ourselves, each other, and our students—all as both teachers and learners together.   as a statement of teaching philosophy, we approach our teaching through the lens of participation and trust for everyone involved in the learning process. our foundational positionality and experiences inform our reflective practice, in particular, all three authors are committed to sharing power with students in order to support a trusting, inclusive, and participatory classroom. this power-sharing is foundational to our teaching practice and is the main subject of the reflective assessment described in detail below. by sharing power with students, we support students as co-creators in their own learning and assessment, and we also empower ourselves to self-assess our own teaching practice in order to build and strengthen trust and participation.   assessment aim   in applying the connected strands of teacher assessment, librarians as teachers, and reflective assessment, we sought to use reflective assessment as a method for evaluating our learner-driven teaching practices. our main research question can be formulated as follows: as teachers, how can we apply reflective assessment to improve learner-driven teaching practices that empower students as trusted participants in their own learning and assessment?   methods   reflective assessment with colleagues   within the broader methodology of reflective assessment, we specifically followed the method outlined by reale (2017). after reviewing the literature on reflective assessment, we selected reale to guide our practice because she is an lis researcher, teacher, and practitioner—as are the three of us. furthermore, not only is reale’s reflective practice rooted in the lis profession, but she also offers specific approaches for a reflective practice with colleagues. in our case, we wanted to co-develop our learner-driven teaching practices as colleagues. reale (2017) stated that “when we reflect together, we have esprit de corps, and we understand that reflection on our practice, both together and alone, is an added value to our teaching” (p. 82). reale’s approach is conversational and emergent, and she acknowledges that some may dismiss this approach as just a regular meeting. but a reflective assessment meeting is not a regular meeting—reflection happens “with both the explicit and implicit intention that we will improve our practice” (2017, p. 83). thus it is the conscious intentionality towards improvement that defines the rigor of reflective assessment for reale, and for us as well. the data that we produced during our reflective dialogues form the basis of our evidence based approach for assessing and improving our learner-driven teaching practices. as an example of reflection, reale (2017, p. 99) described how tangible evidence is produced through a deliberate attention to process that involves both questioning and answering oneself through guided writing. in this model, the reflective dialogue (regan, young, & mannheimer, 2020) is the evidence that we interpret to assess our teaching. we have attempted to model reale’s approach to reflective assessment in a way that demonstrates an approachable method for assessing teaching with colleagues.   as a scope for conducting our own reflective assessment, we focused our reflections on our individual and shared experiences in implementing learner-driven practices while teaching undergraduate credit-bearing courses in a variety of topics in library and information science, including library research skills, data curation, information ethics, privacy, and social media. our reflective outcomes were co-developed through a series of dialogues and discussions among our author group, described in more detail in the procedures section below.   procedures   in the subsections below, we provide a description of our step-by-step procedures for conducting reflective assessment, summarized in table 1. our author group conducted our procedures in summer 2019, after the close of the spring semester, with a view toward preparing for future teaching semesters.   1. articulate shared viewpoints and goals.   our author group came together in coherence around a shared viewpoint related to a learner-driven pedagogy. in our initial meeting, we articulated our foundational positionalities in order to identify a shared foundation for reflection. for us, our aim was to support a learner-driven classroom by more intentionally and effectively sharing power with students. we aimed to apply an assessment method that could help us understand our own and each other’s teaching practices, and to make improvements based on reflective insights. our goals for conducting a reflective assessment of our teaching are drawn from moon (2004), which included building shared understanding, a process of review, and action in the form of continuing development of teaching practices.     table 1 overview of reflective assessment procedures procedure description cohere around shared viewpoints our author group came together in coherence around a shared viewpoint related to power sharing and learner-driven pedagogy. identify teaching practices for reflection through conversation, we identified five teaching practices that we wanted to explore further using our reflective process. conduct reflection after responding individually to a set of reflective prompts, we compiled our responses into a single document for review and discussion. discuss and analyze reflections to produce insights we met over several sessions to dialogue through our reflective narratives, analyze our ideas, and develop shared insights. apply insights to improve teaching to close the assessment loop, we applied our reflective insights to improve learner-driven teaching practices.     2. identify teaching practices for reflection   to identify practices for reflection, we asked ourselves, “what are the practices that we employ to share power with students and support a learner-driven classroom?” through conversation, we identified five teaching practices that we wanted to explore further using our reflective process:   1.      co-creative syllabus design 2.      learner-defined personal learning goals 3.      soliciting and responding to learner feedback 4.      discipline-based discussions and exercises 5.      self-evaluation   each author has separately applied these practices in the classroom. this allowed us each to bring our independent experiences together around shared practices in our reflective dialogue.   3. conduct reflection   we structured our reflective practice around a subset of prompts offered by reale (2017), who notes that “these are the kinds of questions that can initiate and support reflective thinking and learning” (p. 88):   1.      what previous approaches have worked for you? what approach would be its opposite? 2.      what limits would you like to break through? 3.      what data are you seeing, and what story are you telling yourself?   the courses assessed in this paper are lsci 121: library research skills, lsci 491: contemporary approaches to community building using social media, and honr 494: information ethics and privacy in the age of big data. among reale’s full set of eight reflective prompts, we felt that the three identified above were the most stimulating and generative for our particular situation. a practical motivation was also present: we chose three so that we could feasibly produce and analyze the data. with the goal of our reflection in mind—to reveal insights for enhancing learner-driven pedagogy—each author independently created written reflections for each of our five learner-driven teaching practices. for each teaching practice, we responded to the three reflective prompts identified above. this procedure created a total of 15 short narratives. we then compiled all of our responses into a single google doc (selected for its collaborative functionality) for shared review.   4. discuss reflections to produce insights   the three authors spent four hours over the course of two sessions to dialogue through our reflective narratives, analyze our ideas, and develop shared insights. these discussions functioned as collegial inquiry (henderson, 2001), described by reale (2017) as “deliberate and intentional engagement with colleagues with the specific intention of improving upon practice and problem solving—together” (p. 85). the three authors read through all of the narratives and made notes about key concepts and overlapping ideas. then we discussed our reflections as a group. we identified teaching practices that have worked to help promote and support power sharing and learner-driven pedagogy. in addition, we identified practices that have not worked as well. as reale (2017) wrote, “reflective practice with colleagues is a catalyst for development. it can be a humbling experience, to be sure, and one in which we are challenged to interrogate our own truths, our own way of being” (p. 86). as a group, we interrogated our own truths and generated insights through dialogue to improve our teaching. note that our process did not take the form of a traditional content analysis. instead, we used informal discussions as guided by reale to arrive at meaningful results.   5. apply reflective insights to improve teaching   our final step was to apply our reflective insights to further develop and improve our learner-driven teaching practices for courses taught in spring 2020. these courses were lsci 121: library research skills, taught by regan in fall 2019; lsci 391: data curation for a data-driven world, taught by mannheimer in spring 2020; and lsci 291: technology, ethics, and society, taught by young and mannheimer in spring 2020. in this step, the results of our reflective assessments were used to inform revised teaching strategies, thus “closing the loop” and improving our learner-driven teaching practice.   as a note to the reader, the process of reflective assessment can be nonlinear and emergent, and does not always provide the clearest through line of analysis. our reflective texts and dialogues were wide-ranging, and our results section below captures highlights from our reflections and conversations as they relate to our central thesis of developing and implementing a learner-driven pedagogy that intentionally and effectively empowers students as trusted participants in their own learning and assessment.   results   in following reale’s (2017) approaches outlined above, we model a practice of reflective assessment with colleagues that can help us improve our teaching by sharing power with students and thus enhance the learner-driven classroom. each section below is scoped around one pedagogical practice. we begin each subsection with a quote drawn from our reflections, followed by a narrative summary of our shared experiences (for a full transcript of our reflections, please see regan, young, and mannheimer, 2020).   co-creative syllabus design   “i understand something that students don’t always immediately see: that i’m trying to tell a story when i teach a course. although these courses are often structured as a series of exercises, i want to create a through line of knowledge that builds over the semester—and i want learners to be involved in the creation of this story.” —matthew regan   our reflections showed that we all aim to incorporate student participation into syllabus design, pushing back against the idea of the professor as the expert and the authority figure. we wanted to put power into the hands of students, encouraging them to build content for their own syllabus, informed by the topics that most interest them (for example, choosing readings for the course or helping to define which skills they most wish to focus on). however, our reflections suggested that the success of this strategy has been dependent on several factors:   ·        students’ class standing—i.e. freshman, sophomore, etc.; ·        whether the course was required or an elective; ·        whether the course was predominantly skills-based or theoretical; and ·        how much prior knowledge the learners had about the topic.   for library research skills, which we have all taught, we found that most students were lower-division undergraduates, most students enrolled in the course because it was required for their major, and most students had limited knowledge of the course topics. our reflections revealed that in the context of library research skills, the idea of a flexible, evolving syllabus caused some students confusion and anxiety, rather than empowering them.   however, young and mannheimer found that when we taught the upper-level course honr 494: information ethics and privacy in the age of big data, the learners were more prepared with a foundational understanding not only of the topics of the course, but also the general structure of college courses. this allowed the students to more easily and fluently engage with selecting readings and talking about the syllabus.   based on our reflections on our experiences, we concluded that co-created syllabi may best be reserved for upper-division undergraduates or graduate students, electives that students enroll in based on interest, predominantly theoretical courses, and courses for which students have some prior knowledge of the subject matter.   learner-defined personal learning goals   “i want students to be able to guide their own education. i want them to think of a class as their own, active opportunity to pursue knowledge—not a passive opportunity to be ‘taught.’” —sara mannheimer   all of us had similar experiences when encouraging students to identify personal learning goals in each of our courses. we often found that the students had trouble articulating goals that were meaningful for their growth as learners, instead falling back on very broad goals that did not feel specific to their personal learning path. by explicitly helping students see the connection between their existing knowledge and the learning outcomes of the course, we can help them develop personal learning goals that are meaningful to their lifelong learning trajectory.   one of our responsibilities as educators is to help students understand the learning objectives of the course, and then connect those objectives both to their existing knowledge and their goals for future knowledge. our reflections revealed two key insights to support learners in developing personal learning goals.   first, all of us have conducted the learning objective review and personal learning goal identification process early in the semester, but through this assessment, we determined that if we wait until later in the semester, students may be better equipped to identify meaningful learning goals. if more time has elapsed in the semester, students have a better understanding of the topics of the course and have begun to meld this new knowledge with their existing knowledge. our reflections suggest that students may have more success building future learning goals if they start with a firmer foundation of knowledge about the topics of the course.   second, the assessment data showed that personal learning goals really are personal. it follows that we as educators should provide a safe space for students to sincerely reflect on their goals. one strategy we used to create this safe space is to have one-on-one meetings with each student in which we discuss learning objectives and learning goals, but we acknowledged that this strategy does not immediately scale up to large classes. our data revealed that our students needed more time and space to reflect on their own learning goals and on how those goals related to the learning objectives of the course.   soliciting and responding to learner feedback   “students have opinions, fears, frustrations, and joys that they may hold on to all semester long and then may or may not report out in a course evaluation. these thoughts must find a space to be aired sooner and with greater follow-through.” —matthew regan   we all recognized the importance of soliciting learner feedback and cultivating an environment where dialogue and regular, consistent exchanges of information between student and teacher take place; where learners are invested in course content and co-own the learning process; and in which feedback is not punitive.   our practices were similar in that we all created time and space for this information exchange: via start-of-semester face-to-face meetings, through early in-class listening sessions, or in midterm self-evaluations. these were met with varying degrees of success. a major obstacle arose in the form of student anxiety and uncertainty. meeting with a faculty member one-on-one can be intimidating; in-class listening sessions may cause students to bottle up, perhaps because they feel put on the spot in front of their peers. similarly, we found that midterm evaluations, if not anonymized, may stifle honest sharing. we grappled with understanding the best way to engage learners and inspire them to participate and share in the process of learning.   our reflections showed that our systems of feedback and response tended to flow in one direction, depending on the method in question. course evaluations, for example, flow anonymously from students upward to teachers and administrators. our learning management systems empower faculty members to share detailed feedback with students along with a grade for a given assignment. we found that this unidirectional feedback flow seemed to inhibit rather than foster our values of power sharing, mutual learning, and co-creation, and also did nothing to advance fostering intentional dialogue between students.   although time-intensive, our reflections suggested that face-to-face meetings are worth pursuing because they build rapport with students and help demystify the course. students who have less investment in the course (perhaps because it is required or was an elective that fit their schedules) can be honest about this and we as faculty members can commit to working with them to make the course a valuable addition to their academic experience. barriers to the success of this method include faculty and student schedule conflicts and, as mentioned above, student discomfort with face-to-face meetings. in our reflections, we identified small-group discussions as a potential solution: using class time for small groups to discuss what is working well and what could be different, then reporting on their conversations.   interdisciplinary discussions and exercises   “since we teach through a library curriculum at an institution that does not offer a library degree, the students enroll in our library classes from all over campus. this presents challenges and opportunities for bringing different disciplinary perspectives together around similar topics.” —scott young   our assessment data suggests that a significant challenge for us as librarian-teachers at msu is that our course offerings do not align with a curriculum that students are completing as they work toward a degree in our field. our classes are either electives that students opt into or skills-based courses required for their majors. however, lsci courses are not often seen as integral to students’ overall academic experience at msu. as such, students who enroll in lsci courses often come from different disciplinary backgrounds and therefore bring varying knowledges to the course topics in the content. this provides an opportunity to engage students and encourage them to bring forward their developing expertise. in practice, we apply small-group exercises and regularly reconfigure the group compositions so that students from different disciplines can come together around a common topic with unique perspectives. discussions in these interdisciplinary groups can often lead to unexpected and creative insights related to the course topics.   we determined that engagement is most evident when students feel a personal connection to the learning material. for some students, this might be working on a project from the context of their major (which for many will be closely connected to their future careers). for other students, this may be an opportunity to experiment with a personal interest or something they are intellectually curious about.   in our reflections we observed that discipline-based discussion and exercises also provide students with a chance to lean on and learn from each other—whether they share a major or not. small group work may be intentionally developed so that students from different majors work with each other to learn something about a discipline with which they are less familiar. conversely, students may work in peer groups based on their major, academic, or personal interests to better understand the power of the scholarly conversation and how different perspectives even within a discipline can contribute to problem solving and knowledge building (e.g. several students in a class may be focused on the same final project topic but be approaching it from very different perspectives—these differences can and should be highlighted to foster creative thinking). regardless of the form it takes, we found that students engaged more enthusiastically with content that related to their discipline. we also found that when students engaged with their peers in other disciplines, all students benefited from the variety of viewpoints. as lifelong learners, we librarian-teachers agree that learning experiences feel most engaging when we can bring our own interests to the table—and connect these interests to the diverse interests of others.   self-valuation   “the data i see here comes through the self-evaluations themselves, where students have shown themselves generally to be insightful, honest, and sincere in their self-reflections. this data tells a story of trust and self-realization.” —scott young   in an effort to foster the aforementioned dialogues and counteract the traditional power dynamic of the teacher-learner relationship, we have each implemented self-evaluation approaches for grading in each of the courses we have taught. in addition to the qualitative feedback that we provide throughout the semester, we offer structured prompts for students to evaluate themselves, with rubrics for consultation and guidance. we found students to be insightful, honest, and sincere in their self-reflections. in all of our experiences, we observed that students tended to offer grades that were slightly lower or exactly the same as we would have assigned. we see this as a demonstration of trust and self-realization, with self-grading serving as a powerful tool to help students recognize that a grade is earned through a complex process rather than simply given by an authority figure. we found that in taking time to discuss the self-assessment process itself, we helped students understand the motivation and value behind active student self-assessment. as a result, we reflected that our classes felt more relaxed, open, and creative, without the concern of the “gradebook.”   at the same time, we acknowledged challenges with this approach. students sometimes struggled to let go of traditional grading approaches. students found it difficult to gauge their level of effort or to measure progress when the typical markers of grades and points are absent. moreover, students still looked to us as the authority in evaluating performance. indeed, within the broader institutional context, we are the authority; we are required to assign students a grade at the end of the semester. we wished we could more effectively extend the values and practices a self-evaluative approach from our own classrooms to the wider college.   discussion   our reflective assessment—conducted in summer 2019—gave rise to a number of ideas for new or revised practices that support learner-driven teaching practices. we then implemented new and revised practices in subsequent courses. in this section, we provide descriptions of how we applied these techniques in the classroom in fall 2019 and spring 2020. in bringing assessment insights into actual practice, we “close the loop” of the assessment lifecycle by demonstrating how our teaching practice changed as a result of our assessment process.   more frequent checking in   in response to our reflections regarding soliciting and responding to learner feedback, mannheimer implemented three anonymous surveys during her data curation for a data-driven world course in spring 2020. the survey asked students to anonymously identify “roses,” i.e. helpful or interesting things about the course so far, and the elements of the course that have worked, and “thorns,” i.e., the least clear or most frustrating parts of the course so far, and anything that could be improved in the structure or content of the course. she then shared the results from the “roses and thorns” survey with the class and communicated what would be changed in response to the feedback. for instance, the survey showed that learners wanted more clarity about assignment instructions, due dates, and times; in response, mannheimer created a document that provided an at-a-glance overview of all assignments for the semester, when each would be assigned, and when each would be due.   regan also solicited feedback early in the semester for his library research skills course in fall 2019. students were provided with an anonymous check-in survey in the third week of the semester and then again in the sixth. this enabled regan to keep his finger on the pulse of student affect and to anticipate potential changes in course direction, teaching style, and assignment clarity.   providing clarity about self-evaluation and grading   mannheimer reviewed the policies for self-evaluation and grading during the first class session of data curation for a data-driven world in spring 2020. however, students were still unsure about self-evaluation and grading. therefore, she spent part of a class period during the fourth week of the semester revisiting the grading rubric and facilitating small-group conversations about self-evaluation. in the future, she will also assign reading that can provide more information and therefore help guide the discussion about self-evaluation as a grading strategy.   one-on-one meetings with all students   as librarians, we are keenly aware of the value in working with students one-on-one at the point of need, as happens in typical research consultations and reference transactions. our reflections showed that one-on-one meetings are also a valuable method for supporting the learner-driven classroom. in order to strengthen relationships with students, regan piloted an approach whereby he met with all students across two sections of his fall 2019 library research skills course. each meeting took place outside of the classroom during class time while the rest of the class was engaged in a hands-on group activity. each student was allotted approximately five minutes for this check-in; meeting with all students from two sections took approximately four weeks. the scope was made clear: “this meeting is an opportunity for us to get to know each other on a human level and for you to tell me anything that might help me understand you, your learning style, your academic goals, and your ability to successfully complete this course.”   during meetings, students revealed personality traits, discussed competing demands for time such as jobs, families, and classes they had to complete for their majors. some disclosed their status as student athletes with rigorous training and game schedules while others talked about their status as nontraditional students for whom school was “never my thing.” overall, these meetings afforded regan the opportunity to sit back and practice deep listening.   allowing students to negotiate, discuss, and determine assignment deadlines and dates   when introducing the three major assignments in his fall 2019 library research skills course, regan undertook the practice of asking students whether the assignment parameters—including the due dates and deadlines—were fair. this required flexibility on regan’s part. learning about student work outside of the class he taught was helpful to see the whole picture of a student’s academic life: some students were in writing-intensive courses for their major, others had work obligations or busy times that might have interfered with their ability to submit the best possible version of the assignment. each assignment was discussed thoroughly, with ample class time allotted for clarifying questions and negotiation of criteria and submission deadlines. students were mostly amenable to the parameters they received, but the asking was crucial: it actively demonstrated to them that their voices mattered and that they had a say in how and when their learning was to take place. this was clearly a novelty for at least one student, who remarked anonymously via course evaluation: “i loved how you would ask us how we felt about a certain assignment being due at a certain time. i've never had a professor ask a class about when they would like a project due. you are very considerate towards other projects or essays that students have due around the same time as your assignment.”   more flexibility with students’ work products   in order to create new flexibilities for student work and participation, mannheimer and young offered multiple formats for student engagement in the online discussion forums of our spring 2020 course technology, ethics, and society. where previously we asked students to respond to readings and discussion prompts in writing, we now offer three options for students to choose from:   ·        a written response, around 250 words. ·        an audio or video response, about 2 minutes. ·        a sketch that expresses your thoughts. this could be a drawn picture or a graphic using whichever medium you prefer, such as an infographic software, microsoft paint, or pen and paper. once you have created your sketch, attach a caption (2 or 3 sentences) that explains or interprets the sketch.   in practice, students responded with a variety of creative expressions that enlivened discussions. in one course, for example, an assignment that asked for student sketches produced the most discussion posts that semester, with students opening their responses with comments like “i really like your drawing here, and i think it speaks volumes,” and “i do like your graphic and thoughts here.”   pedagogical transparency   to help students respond effectively to our learner-driven methods, we realized that openness and communication is crucial for accomplishing our goal of inclusive and participatory learning. for this reason, we have each begun sharing readings and facilitating meta-conversations with students related to our pedagogy itself. this helps students see their own power in the classroom, and it also shows students that we as teachers are acting intentionally. being open with our approach helps students understand the structure and motivation of learner-driven practices, which in turn helps students perform better in a more open-ended and co-creative learning environment.   conclusion   in this article, we model an approach to reflective assessment that aims to assess learning-driven teaching practices among a small group of librarian colleagues. we find that reflective assessment is an effective and insightful approach for understanding and improving learner-driven teaching practices. our reflective dialogues produced insights into our past teaching practices that we then applied to improve existing learner-driven teaching practices and to generate ideas for new practices.   first, we articulated a set of learner-driven teaching practices that we had each implemented in previous courses. these practices formed the basis of our reflective assessment: co-creative syllabus design, learner-defined personal learning goals, soliciting and responding to learner feedback, interdisciplinary discussions and exercises, and self-evaluation. then, following a reflective activity that produced insights related to our teaching experiences, we produced refinements to these practices and we generated ideas for new practices, including more frequent checking in with students; more clarity regarding self-evaluation and grading; one-on-one meetings with all students; allowing students to negotiate, discuss, and determine assignment deadlines and dates; more flexibility with students’ work products; and increased pedagogical transparency.   this process of assessment is iterative, and so we intend to re-initiate the cycle of assessment in the future. we view the process of reflection itself as a practice of professional bonding, mutual learning, and continual improvement. as librarian-teachers, we may teach individually, but we are rooted in a community of other librarians both locally and profession-wide. by sharing our experiences via reflective assessment, we can improve our teaching practices and also connect with each other and our wider community. ultimately, this paper offers a set of replicable learner-driven teaching practices, as well as an approachable framework for conducting reflective assessment. we encourage others to follow similar learner-driven practices and reflective assessments in their own teaching.   acknowledgements   the authors wish to thank the anonymous peer reviewers and the editors of evidence based library and information practice—their guidance and feedback strengthened our thinking and clarified our perspective. we also thank our students for their inspiring participation in our shared teaching and learning practices.   references   ackerman, d., gross, b. l., & vigneron, f. (2009). peer observation reports and student evaluations of teaching: who are the experts? alberta journal of educational research, 55(1). retrieved from https://web.archive.org/web/20190526053732/https://journalhosting.ucalgary.ca/index.php/ajer/article/view/55272   american library association. (2019). core values of librarianship. american library association. https://web.archive.org/web/20190824190825/http://www.ala.org/advocacy/intfreedom/corevalues   association of college and research libraries. (2016). framework for information literacy for higher education. retrieved from https://web.archive.org/web/20200419220036/http://www.ala.org/acrl/sites/ala.org.acrl/files/content/issues/infolit/framework.pdf   association of college and research libraries. (2017). roles and strengths of teaching librarians. retrieved from https://web.archive.org/web/20191016035336/http://www.ala.org/acrl/standards/teachinglibrarians   berns, a. (2020). scored out of 10: experiences with binary grading across the curriculum. proceedings of the 51st acm technical symposium on computer science education, 1152–1157. https://doi.org/10.1145/3328778.3366956   black, p. e., & plowright, d. (2010). a multi‐dimensional model of reflective learning for professional development. reflective practice, 11(2), 245–258. https://doi.org/10.1080/14623941003665810   booth, c. (2011). reflective teaching, effective learning: instructional literacy for library educators. chicago, il: american library association.   boring, a. (2017). gender biases in student evaluations of teaching. journal of public economics, 145, 27–41. https://doi.org/10.1016/j.jpubeco.2016.11.006   burke, m. (2012). academic libraries and the credit-bearing class: a practical approach. communications in information literacy, 5(2), 156–173. https://doi.org/10.15760/comminfolit.2012.5.2.110   carnegie classification of institutions of higher education (2020). montana state university bozeman, montana. retrieved from https://web.archive.org/web/20200523005454/https://carnegieclassifications.iu.edu/lookup/view_institution.php?unit_id=180461   cohen, n., holdsworth, l., prechtel, j. m., newby, j., mery, y., pfander, j., & eagleson, l. (2016). a survey of information literacy credit courses in us academic libraries: prevalence and characteristics. reference services review, 44(4), 564–582. https://doi.org/10.1108/rsr-03-2016-0021   corrall, s. (2017). crossing the threshold: reflective practice in information literacy development. journal of information literacy, 11(1), 23–53. https://doi.org/10.11645/11.1.2241   cunningham, a. d., & donovan, c. (2012). settling uncharted territory: documenting & rewarding librarians’ teaching role in the academy. in c. w. wilkinson & c. bruch (eds.), transforming information literacy programs: intersecting frontiers of self, library culture, and campus community (pp. 181–220). chicago, il: association of college and research libraries.   deitering, a.-m., rempel, h. g., & jensen, t. (2018). reflective information literacy: empowering graduate student teachers. in c. renfro & c. stiles (eds.), transforming libraries to serve graduate students (pp. 103–111). chicago, il: association of college and research libraries.   dohe, k. (2019). care, code, and digital libraries: embracing critical practice in digital library communities. in the library with the lead pipe. retrieved from https://web.archive.org/web/20200411011451/http://www.inthelibrarywiththeleadpipe.org/2019/digital-libraries-critical-practice-in-communities   doherty, j. j. (2005). towards self-reflection in librarianship: what is praxis? progressive librarian, (26), 11–17. retrieved from https://web.archive.org/web/20200420011642/http://www.progressivelibrariansguild.org/pl/pl26/011.pdf   downey, a. (2016). critical information literacy: foundations, inspiration, and ideas. sacramento, ca: library juice press.   elbow, p. (1997). grading student writing: making it simpler, fairer, clearer. new directions for teaching and learning, 1997(69), 127–140. https://doi.org/10.1002/tl.6911   ferguson, h. j. (2013). journey into ungrading. counterpoints, 451, 194–209. www.jstor.org/stable/42982092   good, t. l., & brophy, j. e. (2008). looking in classrooms (10th ed). boston, ma: pearson/allyn and bacon.   graf, a. j., & harris, b. r. (2016). reflective assessment: opportunities and challenges. reference services review, 44(1), 38–47. https://doi.org/10.1108/rsr-06-2015-0027   henderson, j. g. (ed.). (2001). reflective teaching: professional artistry through inquiry (3rd ed). upper saddle river, nj: merrill/prentice hall.   hornstein, h. a. (2017). student evaluations of teaching are an inadequate assessment tool for evaluating faculty performance. cogent education, 4(1), 1304016. https://doi.org/10.1080/2331186x.2017.1304016   loesch, m. f. (2017). librarian as professor: a dynamic new role model. education libraries, 33(1), 31–37. https://doi.org/10.26443/el.v33i1.287   macaluso, s. j. (2014). incorporating self-assessment and peer assessment into library instructional practice. in a. blevins & m. inman (eds.), curriculum-based library instruction: from cultivating faculty relationships to assessment (pp. 123–138). lanham, md: rowman & littlefield.   mannheimer, s., & banta, r. (2018). personal digital archiving as a bridge to research data management: theoretical and practical approaches to teaching research data management skills for undergraduates. in b. h. marshall (ed.), the complete guide to personal digital archiving (pp. 169–184). retrieved from https://web.archive.org/web/20200419221124/https://scholarworks.montana.edu/xmlui/handle/1/12678   mccartin, l. f., & dineen, r. (2018). toward a critical-inclusive assessment practice for library instruction. sacramento, ca: library juice press.   mcgivney, c. (2017). assessment through peer assessment: a method of peer evaluation for the liaison model. in c. shi, l. xiong, & m. b. huang (eds.), proceedings of the international conference on embedded education and instruction of new models of library service: the 4th international conference of beijing academic network library (pp. 279–286). retrieved from https://web.archive.org/web/20200419221439/https://commons.library.stonybrook.edu/library_articles/12/   mengel, f., sauermann, j., & zölitz, u. (2019). gender bias in teaching evaluations. journal of the european economic association, 17(2), 535–566. https://doi.org/10.1093/jeea/jvx057   mezirow, j. (1990). how critical reflection triggers transformative learning . in jack mezirow & associates (ed.), fostering critical reflection in adulthood: a guide to transformative and emancipatory learning (pp. 1–20). san francisco, ca: jossey-bass.   moon, j. a. (2004). a handbook of reflective and experiential learning: theory and practice (1st ed.). https://doi.org/10.4324/9780203416150   mowatt, r. a. (2019). twelve years a servant: race and the student evaluation of teaching. schole: a journal of leisure studies and recreation education, 34(2), 109–119. https://doi.org/10.1080/1937156x.2019.1622949   nowviskie, b. (2015, october 4). on capacity and care. bethany nowviskie. https://web.archive.org/web/20190822165344/http://nowviskie.org/2015/on-capacity-and-care/   reale, m. (2017). becoming a reflective librarian and teacher: strategies for mindful academic practice. chicago: ala editions.   regan, m. t., young, s. w. h., & mannheimer, s. (2020). data from: improving learner-driven teaching practices through reflective assessment [dataset]. retrieved from https://doi.org/10.5061/dryad.qjq2bvqcx   robertson, t., & wagner, i. (2013). ethics: engagement, representation and politics-in-action. in j. simonsen & t. robertson (eds.), routledge international handbook of participatory design. new york: routledge.   sen, b. a., & mckinney, p. (2014). the sea-change model in information literacy: assessing information literacy development with reflective writing. nordic journal of information literacy in higher education, 6(1), 23–38. https://doi.org/10.15845/noril.v6i1.172   stommel, j. (2017, october 26). why i don’t grade. jesse stommel. https://web.archive.org/web/20200328123301/https://www.jessestommel.com/why-i-dont-grade/   tewell, e. (2018). the practice and promise of critical information literacy: academic librarians’ involvement in critical library instruction. college & research libraries, 79(1), 10. https://doi.org/10.5860/crl.79.1.10   trede, f., & smith, m. (2014). workplace educators’ interpretations of their assessment practices: a view through a critical practice lens. assessment & evaluation in higher education, 39(2), 154–167. https://doi.org/10.1080/02602938.2013.803026   vassilakaki, e., & moniarou-papaconstantinou, v. (2015). a systematic literature review informing library and information professionals’ emerging roles. new library world, 116(1/2), 37–66. https://doi.org/10.1108/nlw-05-2014-0060     evidence based library and information practice editorial responsibilities      2020. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     editor-in-chief: lorie kloda   associate editor (evidence summaries): stephanie krueger, christina wissinger   associate editor (research articles): ann medaille, lisl zach   associate editor (classics & reviews): jane schmidt   associate editor (commentaries, using evidence in practice): lorie kloda   communications officer (news): tatiana bryant   production editor: rachel hinrichs   editorial intern: kimberly mackenzie   editorial advisors: lindsay alcock, alison brettle, michelle dunaway, denise koufogiannakis, heather pretty   copyeditors: heather healy (lead copyeditor), julie evener, emily kingsland, alison moore, jane morgan-daniel, stacey l. penney, elizabeth stregger, nikki tummon, elaina vitale   indexing support: kate shore   editorial assistant: samantha sheplawy   evidence summary   undergraduate students seek librarian assistance only after they have searched independently without success   a review of: vinyard, m., mullally, c., & colvin, j.b. (2017). why do students seek help in an age of diy? using a qualitative approach to look beyond statistics. reference & user services quarterly, 56(4), 257-267. http://dx.doi.org/10.5860/rusq.56.4.257   reviewed by: elaine sullo coordinator, information and instructional services himmelfarb health sciences library the george washington university washington, district of columbia, united states of america email: elainej@gwu.edu   received: 30 nov. 2017 accepted: 21 feb. 2018      2018 sullo. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29379       abstract   objective – to explore how undergraduate students look for information and the reasons these students seek assistance from a librarian.   design – qualitative research.   setting – a university in southern california.   subjects – 10 students were interviewed: 1 freshman, 1 sophomore, 5 juniors, and 3 seniors.   methods – students who met with a librarian for longer than 20 minutes were invited to participate in the study, and interviews were conducted within six weeks of this interaction. semi-structured interviews were scheduled for one hour blocks and were audio-recorded and transcribed afterward. interview data was analyzed using applied thematic analysis. the researchers used nvivo to assist with the process of coding data.   main results – once all transcripts were coded, the researchers identified the following six themes related to how students look for information and the reasons they asked for assistance: how students research, personal perceptions of research skills, assumptions (students’ misperceptions about library services), motivation for asking for help, path to the librarian (how students contacted librarians and their reason for selecting a particular librarian), and experience working with a librarian.   conclusion – overall, the research results demonstrate that students prefer to conduct research independently but will consult a librarian if they are not able to find what they need, if they find the research question especially challenging, or if they have spent an unreasonable amount of time conducting research. in-class library instruction, along with professor referrals are the most effective methods for encouraging students to seek out library assistance.   commentary    as noted by the authors, much of the published literature that examines students’ help-seeking behavior focuses on the reasons that students do not seek out assistance from librarians. among the few published articles that explore the reasons students confer with librarians when they have research questions, the authors cite one qualitative study in particular and build upon this research. the current study uses a different qualitative method as well as an expanded inquiry into students’ independent research approaches and motivation for seeking out a librarian.   the study was evaluated using the critical review form for qualitative studies checklist, version 2.0 (lets et al., 2007). based upon the literature review and lack of evidence available related to the authors’ specific interests, the researchers provided a clear justification for this study. the study design, qualitative research with semi-structured interviews, was appropriate for the researchers’ questions, as they were interested in the “lived experience” of the students’ research habits and were able to follow-up with additional questions when there were topics that they wanted to explore deeper or for which they wanted to gain a better understanding. the interview questions were included in the article’s appendix, which makes for a study that could be easily replicated.   the students who participated in this research had previously sought out assistance from a reference librarian, which may be indicative of a lack of anxiety or hesitancy in seeking help or approaching librarians. for a broader perspective, the authors could replicate this study by including students who have not previously interacted with a librarian. regarding the qualitative research data analysis process, the researchers did not mention that they did member checks to ensure completeness and credibility of the data. member checks, which involve sharing the research findings with the study participants, are a way to improve the accuracy and validity of the data and make sure that the researcher has properly interpreted the participants’ statements.   although the authors used a different qualitative approach than that described in other published literature on the topic, many of the study’s conclusions seem to confirm what other researchers have concluded. while the evidence presented may not be new, it does validate what some academic librarians already know about the research skills and attitudes of undergraduates.   the study findings are relevant to all academic librarians, especially those working with undergraduate students. the evidence suggests that librarians should strategically market reference services to students; they might consider partnering with other campus entities to build trust with students as well as get the word out about library services. marketing could also serve to dispel students’ assumptions and misconceptions about the library, especially among those with the do-it-yourself mindset. furthermore, the data provides support for continued relationship building with faculty and library classroom instruction. librarians can use this opportunity to provide information to students about the ease and accessibility of meeting with a librarian, perhaps highlighting specific reasons why the meeting would be beneficial, such as saving time, developing research skills, getting to know the librarians so that students can feel comfortable asking for assistance in the future.   references   letts, l., wilkins, s., law, m., stewart, d., bosch, j., & westmorland, m. (2007). critical review form – qualitative studies (version 2.0). retrieved from http://www.peelregion.ca/health/library/eidmtools/qualreview_version2_0.pdf evidence based library and information practice 2009, 4:2 172 evidence based library and information practice evidence summary study fails to link ill usage patterns to liaison activities a review of: leykam, andrew. “exploring interlibrary loan usage patterns and liaison activities: the experience at a u.s. university.” interlending & document supply 36.4 (2008): 218-24. reviewed by: scott marsalis social sciences librarian university of minnesota libraries minneapolis, minnesota, united states of america e-mail: marsa001@umn.edu received: 02 march 2009 accepted: 18 april 2009 © 2009 marsalis. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. objective to investigate interlibrary loan (ill) usage patterns, and connect them to liaison activities beyond collection development. design – pattern analysis of ill requests. setting – library of the college of staten island, a mid-size, public university with predominantly undergraduate enrolment. subjects – 4,875 identifiable requests over a three-year period. methods – a data set of requests for ills of monographs over a period of three years was acquired from oclc resource sharing statistics. this data was manually reviewed to remove duplicate records of the same request, but not multiple requests for the same item. the data included requestor status, department, publication date and subject classification of requested items. main results – differences in use across user statuses and departments were identified. conclusion – usage patterns can accurately illustrate trends in the borrowing behaviour of patrons, and be used to inform liaison librarians about user needs. commentary serious flaws both in the design of this study, and in the interpretation of results, limit its usefulness. mailto:denise.koufogiannakis@ualberta.ca http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2009, 4:2 173 a primary problem with the study is its inclusion of ill requests of monographs only, and the failure to consider that different disciplines rely upon monographic literature and serial literature to different extents. the author’s conclusion that “high usage of ill service seems to indicate not only research needs but also a greater awareness of the library and its services, and low usage seems to suggest limited awareness of ill services” (222) not only fails to consider the disciplinary differences in relying on monographs, but also fails to examine the impact of the adequacy of the institution’s own monographic collection in meeting the needs of the users. since the study does not include information on awareness of ill services by users/non-users, the conclusion is purely speculative. the institution studied allowed the placing of requests by intermediaries, making it impossible to truly identify whether a request by a staff member, including library staff, was for their own research, or for a faculty member or student. because 21% of requests were placed by librarians, this could have a significant impact on the veracity of the data in reflecting actual usage patterns. examination of the age of materials requested is not instructive because the publication data was examined solely by year, rather than age. since the request data set spans a three-year period, from 2005-2007, an article published in 2005 could be anywhere from less than a year to three years old at time of request. also questionable is leykam’s assertion that “due to the heavy use of ill by faculty, it is believed that ill requests reflect individual research needs rather than general institutional needs,” (221) which fails to acknowledge the ongoing nature of faculty and graduate research, and the impact of faculty research interests on the research interests of their students and advisees. the originality/value statement in the article abstract states, “the current study links the assessment of actual ill usage patterns with liaison activities beyond collection development”(218). this is an intriguing area of study, and ill usage data shows potential for informing or improving liaison librarians’ work. as leykam’s literature review covers, the potential for ill usage to improve collection development is well documented (219). unfortunately, nothing in the study collects information on liaison activities, nor does it extend beyond collection development. further problems in the design, and questionable assumptions and deductions, keep the article from being useful for evidence based practice. microsoft word class_genius.doc evidence based library and information practice 2007, 2:4  80 evidence based library and information practice       classic  kuhlthau’s classic research on the information search process (isp) provides evidence  for information seeking as a constructivist process     a review of:  kuhlthau, carol c. “inside the search process: information seeking from the userʹs perspective.”  journal of the american society for information science 42.5 (1991): 361‐71.      reviewed by:   shelagh k. genuis  interdisciplinary phd student, school of library and information studies/faculty of nursing,  university of alberta  edmonton, alberta, canada  e‐mail: genuis@ualberta.ca        received: 24 august 2007      accepted: 15 october 2007    © 2007 genuis. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract  objective – to extend understanding of  purposeful information seeking and to  present a model of the information search  process (isp) from the perspective of the  user.  design – review of theoretical foundation,  summing up of qualitative and quantitative  data from a series of five foundational  studies, and presentation of isp model.  setting – summarised research was  conducted primarily in high school and  college environments where subjects were  investigating an assigned topic. a small  proportion of public libraries were used in  the fifth study within the reviewed series.   subjects – the isp model as presented in  this ‘classic’ article is based on studies  involving a total of 558 participants. the  first study involved 26 academically  advanced high school seniors, and the 2  subsequent studies involved respectively 20  and 4 of the original participants following  their completion of 4 years of college. the  final 2 studies involved respectively 147  high, middle and low achieving high school  seniors, and 385 academic, public and school  library users.   methods – this paper presents the  foundation for the isp model by reviewing  the relationship between kelly’s personal  construct theory, belkin, brooks, and  oddy’s investigation of cognitive aspects of  the constructive information seeking  mailto:genuis@ualberta.ca http://creativecommons.org/licenses/by/2.0 http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:4  81 process, and taylor’s work on levels of  information need (“question‐negotiation”)  and value‐added information (“value‐ added”). this is followed by a review of  kuhlthau’s five foundational studies, which  investigated the common information  seeking experiences of users who were  seeking to expand knowledge related to a  particular topic or problem.   the first of these studies was a small‐scale  exploration in which participants were  given two assignments. questionnaires,  journaling, search logs, and reflective  writing were used to collect data throughout  the process of assignment completion. data  collection was augmented by case studies  involving in‐depth interviews and  construction of timelines and flowcharts  with six  study participants. the six‐stage  isp model was developed from qualitative  content analysis of participants’ perceptions  and experiences (kuhlthau, “library  research process”).   in the second study, the same questionnaire  was used to determine how students’  perceptions of the isp had changed over  time. post‐college responses were compared  to responses given in high school and  statistical significance was determined  through t tests (kuhlthau, perceptions).  four of the original 6 case study participants  were interviewed in the third study, in  which interview data and search process  timelines were compared with high school  case studies (kuhlthau, longitudinal).   in the fourth and fifth studies, large‐scale  field studies were conducted to verify the  isp model. process surveys elicited  participants’ thoughts and feelings at  initiation, midpoint, and closure of a search  task. data were analysed using descriptive  and inferential statistics including measures  of significance and analysis of variance  (kuhlthau, information search; kuhlthau et  al.). following a summation of these 5  research studies, this article details and  discusses the isp model.  main results – based on the data from the  five studies, the isp presents a constructivist  approach to information seeking and  incorporates affective, cognitive, and  physical dimensions at each of six  information searching stages: initiation,  topic selection, pre‐focus exploration, focus  formulation, information collection, and  presentation. individuals become aware of  an information need at initiation. feelings of  uncertainty and apprehension are common  as wide‐ranging task exploration begins. at  topic selection a general topic is selected and  users frequently experience initial optimism,  which is commonly followed by confusion  and doubt as pre‐focus exploration  commences and users struggle to extend  personal knowledge through initial  investigation of the general topic. a turning  point occurs during focus formulation as  constructs become clearer and uncertainty  decreases. during information collection the  user is able to articulate focused need and is  able to interact effectively with  intermediaries and systems. relief is  commonly experienced at presentation stage  when findings are presented or used.  although stages are laid out sequentially,  kuhlthau notes that the isp is an iterative  process in which stages merge and overlap.  central to this model is the premise that  uncertainty is not due merely to a lack of  familiarity with sources and technologies,  but is an integral and critical part of a  process of learning that culminates in  finding meaning through personal synthesis  of topic or problem.   conclusion – kuhlthau provides evidence  for a view of information seeking as an  evolving, iterative process and presents a  model for purposeful information searching  which, if understood by users,  evidence based library and information practice 2007, 2:4  82 intermediaries and information system  designers, provides a basis for productive  interaction. while users will benefit from  understanding the evolving nature of focus  formulation and the affective dimensions of  information seeking, intermediaries and  systems are challenged to improve  information provision in the early formative  stages of a search. although kuhlthau  identifies this research on the isp as  exploratory in nature, this article affords  methodological insight into the use of mixed  methods for exploring complex user‐ oriented issues, presents a model that  effectively communicates an approximation  of the common information‐seeking process  of users, and provides ongoing impetus for  exploring the user’s perspective on  information seeking.    commentary    this article effectively sums up the  empirical basis for the isp model by  reviewing methods and data from the initial  small‐scale exploratory study that resulted  in the development of the isp, the two  longitudinal studies that tested the model,  and the two large‐scale field studies that  verified the model. this series of  foundational studies provides evidence for a  recursive, iterative process of information  seeking and provides a strong basis from  which the author presents six isp stages and  proposes potential application to reference  and user services. this article has uniquely  contributed to lis literature and to user  studies in particular by drawing attention  beyond cognition and to the investigation of  affective elements influencing information  seeking. evidence is provided for the view  that uncertainty plays an integral role in the  process of information searching and sense‐ making.     although kuhlthau’s utilisation of actual  library users, the large sample size, and her  strategic research program lends strength to  the isp model, it is important to note that  this model is rooted in the academic context.  the vast majority of participants were  responding to an imposed information need  in the form of a school or college assignment.  a small number of participants were public  library users and, while findings revealed a  similar approach to information seeking, the  searching task of these participants was not  described. subsequent research utilising the  isp model reveals an ongoing emphasis on  academic settings; however, this classic  article has also been widely cited by  researchers exploring information seeking  behaviours within a range of populations,  particularly professional populations  seeking information within the context of  work‐related needs. furthermore, as  demonstrated by vakkari, kuhlthau’s work  has influenced information access and  retrieval research by drawing attention to  the complexity of information searching  behaviour and to the need for more user  responsive information retrieval systems.      in addition to its significant impact on  research, this classic article has made three  critical contributions to lis reference work.  (1) information searching has been clearly  situated within the context of a dynamic  constructivist process involving a series of  information encounters rather than a single  reference incident. for reference librarians,  ‘help with process’ has gained equal footing  with ‘help with sources.’ (2) the isp model  formalised a holistic view of users that  incorporated the contribution of thoughts,  actions, and feelings to the searching  process, and a view of the user as one who  processes information within the context of  a given information need at a specific  moment of time. and (3), by advancing the  view that different levels of service best  serve the diverse and changing needs of  users, this classic article provides a  foundation for expanding the traditional  emphasis of reference services from locator  for physical access to advisor for intellectual  evidence based library and information practice 2007, 2:4  83 access within the context of evolving  information needs.         works cited    belkin, n. j., h. m. brooks, and r. n. oddy.  “ask for information retrieval. part i:  background and theory.” journal of  documentation 38.2 (1982): 61‐71.      kelly, g. a. a theory of personality: the  psychology of personal constructs.  new york: norton, 1963.    kuhlthau, c. c. “the information search  process of high‐middle‐low  achieving high school     seniors.” school library media  quarterly 17.4 (1989): 224‐8.    ‐‐‐. “the library research process: case  studies and interventions with high  school seniors in advanced placement  english classes using kellyʹs theory of  constructs.” diss. rutgers, the state  university of new jersey, 1983.     ‐‐‐. “longitudinal case studies of the  information search process of users in  libraries.” library and information  science research 10.3 (1988): 257‐304.  ‐‐‐. “perceptions of the information search  process in libraries: a study of  changes from high school through  college.” information processing and  management 24.4 (1988): 419‐27.    kuhlthau, c. c., m.w. george, b. j. turock,  and r. j. belvin. “validating a model of  the search process: a comparison of  academic, public, and school library  users.” library and information  science research 12.1 (1990): 5‐31.    taylor, robert s. “question‐negotiation and  information seeking in libraries.”  college and research libraries 29.3  (1968): 178‐94.    ‐‐‐. value‐added processes in information  systems. norwood, nj: ablex, 1986.    vakkari, pertti.  “a theory of the task‐ based information retrieval process: a  summary and generalisation of a  longitudinal study.” journal of  documentation 57.1 (2001): 44‐60.                      evidence summary   survey confirms strong support for intellectual freedom in public collection development librarians   a review of: oltmann, s. m. (2019). important factors in midwestern public librarians’ views on intellectual freedom and collection development: part 1. the library quarterly, 89(1), 2-15. https://doi.org/10.1086/700659     reviewed by: laura costello virtual reference librarian rutgers university libraries new brunswick, new jersey, united states of america email: laura.costello@rutgers.edu   received: 24 apr. 2019                                                                  accepted:  19 july 2019      2019 costello. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29577     abstract   objective – the article sought to explore whether librarian attitudes regarding intellectual freedom conform to the stance of the american library association (ala).   design – electronic survey.   setting – public libraries in the midwestern united states.   subjects – subjects were 645 collection development library professionals employed in public libraries.   methods – an electronic survey was distributed to public library directors in nine midwestern states and was completed by the library professional primarily responsible for collection development. the survey focused on community information and probed the participants for their stances on several intellectual freedom topics.    main results – the survey was sent to 3,018 participants via each state’s librarian and had a response rate of 21.37%. the first section of the survey focused on broad strokes statements representing the ala’s stance on intellectual freedom for public libraries. the results revealed widespread agreement on these issues. more than 88% of participants agreed with statements like “public libraries should provide their clients with access to information from a variety of sources.” despite strong agreement among participants, particular demographic characteristics were more likely to lead to disagreement with all statements including working in rural communities and not holding a master’s degree in library science.   the next section of the survey focused on how strongly participants’ personal beliefs conformed to the intellectual freedom statements in the ala’s library code of ethics. again, there was widespread agreement, with 94.9% of participants indicating that they agreed with the statement “we uphold the principles of intellectual freedom and resist all efforts to censor library materials.” only one participant disagreed with the statement “it is the right of every individual to both seek and receive information from all points of view without restriction.” when asked whether the ala’s stance on intellectual freedom ever conflicted with their personal beliefs, 39.8% of participants indicated that it did, 22% were unsure, and 40% had never experienced conflict. participants holding a master's degree in library science and librarians in large cities were less likely to experience conflict between their personal beliefs and the ala’s stance on intellectual freedom. in the free text comments, several participants indicated that they experienced conflict when the ala’s stance did not reflect their personal beliefs or community values.   conclusion – while the overwhelming majority of respondents indicated that they agreed with the ala’s stance on intellectual freedom, a minority of participants experienced some conflict. respondents indicated that personal belief could create conflict when librarians committed to intellectual freedom were required to make choices in their professional work that conflicted with their own views. conflict could also arise when collection choices made to support intellectual freedom were not supported by patrons in the community.   commentary   this article is the first in a two-part series focusing on the impact of intellectual freedom on collection development. it is loosely based on a 1972 article (busha, 1972) which found that librarians supported ideas of intellectual freedom but were less likely to apply those ideas in the context of censorship pressures. oltmann’s findings in part one reflect the first of busha’s findings and the second part of oltmann’s article will explore the application of intellectual freedom ideas on collection development practice. issues of intellectual freedom are particularly relevant in the current u.s. political climate and this article suggests that the attitudes around intellectual freedom have not meaningfully wavered in midwestern librarians.   when examined through the glynn critical appraisal tool (2006), this article represents a large sample of midwestern librarians, but it may not equally represent librarian perspectives across the midwest. the participant pool was drawn from library director contact information supplied by state librarians rather than through professional networks, so the participant pool is likely free of investigator bias, but the response rate varied by state from 5.2%-22.5%. the response rate varied enough by state that state-by-state analysis was not conducted, though there was enough representation in the other demographic categories for analysis. one of the more interesting demographic characteristics examined was whether participants held a master’s degree in library science. slightly fewer than half of participants (48.3%) did not hold a degree and that characteristic was significantly associated with a greater tendency to disagree with intellectual freedom statements. participants from rural areas were also more likely to disagree with the statements, but this may be due to the community conflicts explored in the free text analysis section.   statements for the survey were drawn from official ala documentation on intellectual freedom and the survey tool was also used in oltmann's 2016 study, limited to public librarians in ohio. this study found similar results reflecting librarian approval of intellectual freedom practices in collection development. through this larger-scale project, oltmann has made a convincing case that librarians still strongly support ideals of intellectual freedom as defined by the ala. the second part of this article will explore whether librarians apply these views in collection development.   references   busha, c. (1972). intellectual freedom and censorship: the climate of opinion in midwestern public libraries. library quarterly, 42(3), 283–301. retrieved from http://www.jstor.org/stable/4306179   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154   oltmann, s. m. (2016). public librarians’ views on collection development and censorship. collection management, 41(1), 23-44. https://doi.org/10.1080/01462679.2015.1117998       news/announcements   search extended: evidence based library and information practice (eblip) is seeking an editorial intern      2016. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     we are looking for a new editorial intern to assist our editorial team. the ideal candidate will be in place by february 2017 and will commit to serving a two year term.    the role of the intern is to:    provide a final editorial check of proofed copy before publication, using the pre-developed checklist  check and edit the item metadata on the eblip website to ensure title, author, and abstract correspond to the submitted manuscript  assist the editor-in-chief and communications officer with calls for papers or calls for volunteers  assist with investigation and implementation of new projects related to the journal  participate in monthly editorial team meetings    the ideal candidate will be an mlis (or equivalent) student or a recent graduate (past 2 years) interested in evidence based practice and research methods, and who possesses strong attention to detail. the position requires dedicated time on a regular basis, and it is therefore essential that interested persons ensure available time to devote to this position prior to applying. it is estimated that the workload is approximately 10 hours per month.    interested persons should send a statement of interest, indicating the areas of strength they would bring to the role, as well as a brief resume to lorie kloda at lorie.kloda@concordia.ca by january 9, 2017.   specific queries about the role should be addressed to michelle dunaway at mdunawa@gmail.com.    research article   a re-examination of online journal quality and investigation of the possible impact of poor electronic surrogate quality on researchers   ken ladd collection services librarian university of saskatchewan library saskatoon, saskatchewan, canada email: ken.ladd@usask.ca   received: 17 may 2018  accepted: 6 july 2018      2018 ladd. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29449     abstract   objective – this study re-examines the findings of a paper (ladd, 2010) that investigated whether evidence indicated print equivalent journal collections needed to be preserved, based on the quality of their electronic surrogates. the current study investigates whether: 1) electronic surrogate articles that failed (i.e., the print equivalent article needed to be consulted to view all the content/information) in the first study had improved in quality; and 2) there was evidence that poor-quality electronic surrogates could impact on research if the print equivalent articles did not exist.   methods – each of the 198 pdf documents identified in the 2010 study as failing were re-examined to assess whether any change in quality had occurred. to assess the possible impact for researchers if they needed to rely solely on poor-quality electronic journal surrogates, citation data were collected for each of the failed scholarly pdfs using web of science and scopus, and usage count data were collected from web of science.   results – across the electronic journal backfiles/archives examined, there were 13.6% fewer failures of electronic surrogates for all pdf documents than in the original study, while for scholarly pdf documents (e.g., research papers) there were 13.8% fewer failures. one electronic journal archive accounted for 91.7% of the improvement for scholarly pdf documents. a second archive accounted for all the observed improvement for non-scholarly pdf documents. the study found that for the failed scholarly pdf documents from the original study, 58.7% had been cited or had web of science usage counts from 2010 onward.   conclusion – the study demonstrates a continued need for retaining print equivalent journal titles for the foreseeable future, while poor-quality electronic surrogates are being replaced and digitally preserved. there are still poor-quality images, poor-quality scans of text-only articles, missing pages, and even content of pdf documents that could not be explained (e.g., incorrect text for images when compared to the print). while it is known that not all researchers will consult each of the papers that they cite, although it is best practice to do so, the extent of citations of the failed scholarly pdf documents indicate that having to rely solely on electronic surrogates could pose a problem for researchers.     introduction   there continues to be increased demand for user space within academic libraries. in recognition of these needs and with the availability of electronic journal backfiles of content held in print by libraries, there is opportunity to repurpose prime library space once occupied by print journal collections. at the same time, preservation is still recognized as a fundamental role and responsibility of research libraries (arl, 2007). with the goal of preserving information for future generations coupled with the desire to remove print collections from prime library space, this is often accomplished by the relocation of print materials into storage facilities, disposal of titles through participation in collaborative print archive initiatives, or the disposal of print journals where an electronic surrogate exists.   the strategy of removing print equivalent journals where an electronic surrogate exists is complicated by known quality issues with electronic surrogates (bracke & martin, 2005; chen, 2005; erdman, 2006; hawkins & shadle, 2004; henebry, safely, & george, 2002; joseph, 2006, 2012, 2014; kalyan, 2002; keller, 2005; ladd, 2010; martellini, 2000; mccann & ravas, 2010; robinson, 2010; sprague & chambers, 2000; thohira, chambers, & sprague, 2010; weessies, 2012), where there can be missing content (volume issues or pages), poor-quality images, and illegible text from poor-quality scans. ladd (2010) concluded that the re-digitization of failed pdf content using high-resolution technology along with good quality control practices would eliminate many of the observed failures. given the number of studies reporting quality issues with electronic surrogates, which can be corrected by re-digitization, would publishers attempt to address this significant issue? this is important as it affects users of e-journal backfiles and libraries considering the removal of print equivalent materials from their collections.   because it was known that there were quality issues associated with electronic journal backfiles, the author believed that over a seven-year period there had been sufficient time for publishers to address some of these issues. it was felt that revisiting the original study now could assist in the development or revision of recommendations for the preservation period of print equivalent titles.   in 2006, elsevier began to replace poor-quality images on a case-by-case basis, which developed into an extensive initiative that resulted in hundreds of thousands of pages being rescanned (van gijlswijk & clark, 2010). this raised two key questions:   ·         what impact has elsevier’s initiative had on the overall quality of their electronic journal backfiles? ·         have other publishers attempted to address the quality of their electronic journal backfiles and to what degree?   these questions are important, as the extent to which the quality of electronic surrogates have been improved could affect the need to preserve print equivalent titles.   joseph (2012) followed up an earlier study of elsevier’s earth and planetary sciences archive to investigate the impact of elsevier’s rescanning project. the study was, however, of one disciplinary journal archive of one publisher. the current study was designed to investigate the journal archives of multiple publishers/vendors by re-examining the results of ladd’s 2010 study. in that study, ladd chose seven electronic journal backfiles acquired by the university of saskatchewan that covered a breadth of subjects. journal titles were randomly selected from each backfile and from these titles, volumes and then issues were randomly selected. complete issues were then examined. a total of 2,633 pdf documents were examined and then compared with their print equivalents.   as noted above, the quality of electronic journal backfiles can potentially affect researchers and scholars when they attempt to access pdf documents with poor-quality images, illegible text, or missing pages. the author wanted to investigate the level of potential impact if researchers could only rely on electronic journal archives. as a proxy measure of the potential impact, the current study uses citations to scholarly articles that ladd identified in 2010 as being of poor quality and were found to still be of poor quality in 2017.   literature review   numerous researchers have investigated the differences between electronic surrogates and their print equivalents (bracke & martin, 2005; campbell, 2003; chen, 2005; chrzastowski, 2003; erdman, 2006; hawkins & shadle, 2004; henebry et al, safely, & george, 2002; joseph, 2006, 2012, 2014; kalyan, 2002; keller, 2005; ladd, 2010; martellini, 2000; mccann & ravas, 2010; robinson, 2010; sprague & chambers, 2000; thohira et al, 2010; weessies, 2012). these studies were most often conducted to determine if the electronic surrogates allowed libraries to cancel or withdraw print equivalent titles from their libraries. the studies often focused on a specific factor such as a discipline, missing content, vendor, or electronic journal backfiles or aggregators.   researchers have often found one or more of the following quality issues associated with the scanned electronic surrogates:   ·         images and figures (bracke & martin, 2005; chen, 2005; erdman, 2006; henerby et al., 2002; joseph 2006, 2012, 2014; keller, 2005; ladd, 2010; mccann & ravas, 2010; robinson, 2010; sprague & chambers, 2000; thohira et al., 2010), ·         illegible text and formulas (keller, 2005; ladd, 2010; sprague & chambers, 2000; thohira et al., 2010), ·         missing content—figures, tables, missing pages, articles or issues (bracke & martin, 2005; chen, 2005; henebry et al., 2002; joseph, 2006; keller, 2005; ladd, 2010; sprague & chambers, 2000; thohira et al., 2010).   campbell (2003) found no substantial content missing for the titles reviewed. chrzastowski (2003) noted that while quality was still a concern, over a two-year period there had been only one problem for the chemistry and chemistry-related e-journals at university of illinois at urbana-champaign, and that the vendor had quickly addressed the problem.   ladd (2010) noted that many of the quality related issues observed in the study could be resolved if the existing electronic surrogates were replaced with scans using higher-resolution scanning technology and better quality control. as noted previously, in 2006 elsevier began replacing poor-quality images on a case-by-case basis. this ultimately led to a large-scale initiative that saw hundreds of thousands of pages with poor-quality images being rescanned (van gijlswijk & clark, 2010).   there has been one study that re-examined the observed problems with the quality of electronic surrogate journals. joseph (2006) conducted a study of 35 titles in elsevier’s earth and planetary sciences archive and found that 73.6% of the volume issues had at least one figure that was of poor quality. in a follow-up study to investigate the impact of elsevier’s rescanning project, the number of issues with poor-quality images was extrapolated to have been reduced to 21.9% (joseph, 2012). the study was, however, of one electronic journal backfile from one publisher, in a disciplinary area whose papers often contain images. by contrast, the current study is multi-disciplinary, re-examining seven different electronic journal archives, with a number of different publishers to determine whether there has been an improvement in the quality of the electronic surrogates. in addition, by examining the potential impact on researchers if they needed to rely solely on poor-quality electronic surrogates, this study fills an important need since there have been no other studies of this nature.   aim   this study investigated whether there continues to be evidence print equivalent serials need to be preserved for the short to medium term because of poor-quality electronic surrogates, as concluded in a previous study (ladd, 2010). the central questions were:   ·         have the pdf documents that failed in the 2010 study subsequently improved in quality? ·         were there differences in the improvement of quality between electronic surrogate archives?   a second objective of this study was to examine whether there was evidence that having to rely solely on electronic surrogates could potentially impact researchers. to examine this issue, the study asked, for pdf documents observed to have failed in the 2010 study and found to still fail in 2017:   ·         what citations have occurred from 2010 onward? ·         is there evidence of their usage?   methods   the original 2010 study examined pdf documents from seven electronic journal backfiles (appendix) from a number of vendors with a breadth of subject coverage (humanities, social sciences, science, technology, and medicine). in that study, a pdf document from an electronic surrogate was assessed as failing any time the print equivalent needed to be consulted in order to gain access to all of the item’s information. in the current study, each of the pdf documents from the original study that were classified as failing served as the study sample.   in the fall of 2017, each of the 198 pdf documents that failed in the original study was downloaded from the publisher’s backfile and re-examined to determine if it still was classified as failing, using the original definition for a failure. data were collected for each collection archive and journal title examining:   ·         the number and percentage of the 174 pdf previously failed documents with scholarly content, which had failed again. scholarly content included research papers, case studies, review articles, short communications, technical notes, and errata. ·         the number and percentage of the 24 pdf previously failed documents with other content, which had failed again. other content included book reviews, announcements, letters to the editor, meeting programs, front and back matter, and obituaries.   these data were compared to the 2010 data to determine whether there had been an improvement in the quality of the electronic surrogates and for which electronic journal collection backfiles.   the second part of the study examined researchers’ consultation of the 150 scholarly pdf documents that were identified as still failing in the current study. these papers were published between 1938 and 1999. two proxies for consultation of these articles were used: 1) citation of the failed pdf documents from 2010 onward using citation data from web of science and scopus, and 2) the usage count feature of the web of science, which records the number of times that the full-text of a record has been accessed or where a record has been saved by any web of science user in the last 180 days or since february 1, 2013.   results   ladd (2010) found that there were 198 pdf documents that were assessed as failing—174 were scholarly and 24 consisted of other content such as book reviews and announcements. when each of these pdf documents was examined for the current study, some improvement in the quality of the electronic surrogates was observed. table 1 provides data on the frequency of failures for pdf documents (all pdfs, scholarly pdfs, and other pdfs) for the original study and the current study by electronic journal archive collection.   table 1 failed electronic surrogates (all, scholarly, and other pdf documents), 2017 compared to 2010 for scholarly pdf documents, 13.8% (24) were no longer found to have failed. the results indicate, however, that all but two of the 24 documents that now passed were from a single archive, elsevier sciencedirect backfile medicine and dentistry (a 35.5% improvement in quality). the elsevier sciencedirect backfile social science collection and springer link archives (mathematics) each had a single document that no longer failed.   for the other pdf documents, 12.5% (3) were no longer found to have failed, all from the jstor arts and science i archive. this represented a 75% increase in quality for this archive.   the original study noted that scholarly pdfs failed for a variety of reasons: quality of graphs, maps or drawings; illegible text/numbers in a table or article; missing or incorrect images or content; and quality of the image. figure 1 illustrates the frequency of scholarly pdf documents failing in the 2010 study and the current study for the elsevier sciencedirect backfile medicine and dentistry archival collection, by type of failure: quality control (pages missing or incorrect images), other (illegible text, tables, drawings, or graphs), or image (e.g., x-rays, scintigraphs, photographs, and others).   the study found that each of the pdfs that were now observed to pass had failed originally because of poor-quality images. this represents a 52.4% decrease in the number of failures because of image quality. for two of the pdf documents that still failed, there had been multiple images in each that were of poor quality in the original study, but for the current study all but one of the images in each pdf were now of good quality.   figure 1 comparison of failures by type for scholarly pdfs between the two studies for the elsevier sciencedirect backfile medicine and dentistry collection.     table 2 failed scholarly pdfs cited and total number of citations in web of science and scopus from 2010 onward     wos scopus collection failures cited % citations citations / cited article cited % citations citations / cited article elsevier science direct medicine and dentistry 40 15 37.5% 45 3.0 17 42.5% 54 3.2 elsevier science direct social science 19 16 84.2% 156 9.8 18 94.7% 186 10.3 jstor arts and science i 1 1 100.0% 1 1.0 0 0.0% 0 0.0 oxford university press digital archive 12 5 41.7% 13 2.6 5 41.7% 13 2.6 springer link archives mathematics 1 1 100.0% 3 3.0 1 100.0% 3 3.0 wiley blackwell backfiles humanities and social sciences 32 9 28.1% 669 74.3 13 40.6% 845 65.0 wiley blackwell backfiles science, technology and medicine 45 23 51.1% 162 7.0 24 53.3% 176 7.3 total 150 70 46.7% 1049 15.0 78 52.0% 1277 16.4     table 3 failed scholarly pdfs cited and total number of citations unique between web of science and scopus from 2010 onward     the current study examined the potential impact for researchers if they could consult only the poor-quality electronic surrogates. one proxy for possible impact is the citations from 2010 onward to the scholarly pdf documents that were observed to still have failed in the current study.  for web of science, scopus, and unique (between the two databases), tables 2 and 3 present the number of failed pdfs that had been cited from 2010 onward, the total number of citation counts for all pdfs, and percentage of failed articles cited for each electronic journal archive.   a total of 81 (54.0%) of the failed pdfs had been cited from 2010 onward, the year the first study was published. there were 1,449 unique citations for these 81 papers, however one paper accounted for 654 of the citations. the remaining 80 papers had 795 citations or an average of 9.9 citations each. for the five archival collections with more than 10 failed scholarly pdfs, the percent cited ranged from 40.6% to 94.7%. regardless of the disciplinary area, a significant number of the failed pdfs were cited.   the study examined the web of science usage count feature as a second proxy for the possible impact of researchers having to consult only poor-quality electronic surrogates. the web of science database (web of science core collection help, 2018) defines usage as any web of science user either “…clicking links to the full-length article at the publisher’s website (via direct link or open-url) or by saving the article for use in a bibliographic management tool (via direct export or in a format to be imported later).” table 4 presents the web of science usage count data for the scholarly pdfs that were found to still have failed in this study: number not cited in web of science or scopus, total number with usage data, and percentage of the total failures.   the study found that 36 (24.0%) of the 150 failed scholarly pdfs had web of science usage data associated with them. of these 36, seven had no citations in web of science or scopus. using the two proxies for possible impact of consulting only poor-quality electronic surrogates, there were 88 (58.7%) failed scholarly pdfs that had either citations or web of science usage data from 2010 onward.     table 4 failed scholarly pdfs with web of science usage count by collection     discussion   the current study found that only one electronic journal archive collection, elsevier sciencedirect backfile medicine and dentistry, had improved significantly in quality since the original 2010 study. in that collection, more than one-third (35.5%) of the failed scholarly pdfs were now observed to not fail. of the remaining electronic archival collections, only two had any improved scholarly pdfs: elsevier sciencedirect backfile social science collection and springer link archives (mathematics) each having a single scholarly pdf that no longer failed. figure 1 shows that all the scholarly pdfs that were observed to no longer fail for elsevier sciencedirect backfile medicine and dentistry failed originally because of poor-quality images.   the elsevier rescanning project focused on pre-1995 journals, using an algorithm to identify automatically poor-quality scanned images (van gijlswijk & clark, 2010). the initiative analyzed 19 million pages and resulted in the rescanning of 600,000 pages of poor-quality images. all of the elsevier sciencedirect backfile medicine and dentistry papers in this study were pre-1995 and it would appear that this archive’s 52.4% reduction in failures because of poor-quality images is linked to the elsevier rescanning project. compared to the current study, joseph (2012) found greater improved quality resulting from the elsevier rescanning initiative, likely the result of different study methodology, examining a different elsevier electronic journal backfile, and the timing of the original and re-examination studies. joseph’s studies were done prior to and after the elsevier initiative. ladd’s (2010) original study was done while the elsevier initiative was moving toward completion. the results of both studies, however, demonstrate that good scanning technology coupled with good quality control practices would help to eliminate the majority of observed poor-quality scans.   although the strategy employed by elsevier was successful in addressing many of the poor-quality images, there are still poor-quality images and line drawings, along with other issues found by joseph (2012) and the current study. an excellent example of problems that still exist was found in a single paper from elsevier science direct backfile – medicine and dentistry. when compared to the print equivalent paper, this scholarly pdf was found to be missing six of 12 plates of images (radiographs, micrographs or photograph), each with two figures per plate. for the six plates that were included in the e-surrogate, four plates or eight figures had the incorrect image associated with the description below the figure. for example, plate xviii had the descriptions for figure 8 and 9, but had the images for figure 12 and 13 of the print paper. two of the plates had images for the figures that were upside down, and for one of these plates, the incorrect figure appeared above the description. to verify that the print copy in hand was not the aberration, several interlibrary loan copies were acquired from other academic institutions, which were determined to be identical in content to the print copy in hand.   there are a number of approaches that can be taken to address the problem of poor-quality scans, but there are significant challenges and costs associated with each. rescanning whole issues of journals is a very time-consuming and costly approach, as is trying to find and replace poor-quality scanned pages, which are often scattered and in a minority amongst the acceptable quality scans (joseph, 2012). elsevier’s algorithmic strategy to help address the cost associated with identifying digitized articles with poor-quality images required running the algorithm on two dedicated servers, 24 hours a day, 7 days a week for almost two years (van gijlswijk & clark, 2010).   a more cost-effective approach would be to crowd-source the identification of poor-quality scans that should be replaced. researchers, readers, librarians, and others during the course of their activities could identify and report poor-quality scans to publishers as they are found, who can then replace the poor scans. this would greatly reduce the cost of identifying poor-quality scans of all types. the cost to rescan these pages would remain, however. joseph (2012) cautioned that even after massive efforts, such as elsevier’s project to address the issue, problems with poor-quality images continue, which should be taken into consideration when making decisions to store or discard print equivalent titles. the implication is that archiving of print journal runs will be needed for the foreseeable future.   since there continues to be a need for the preservation of print for the foreseeable future, a collaborative approach would logically be the most cost effective, by sharing the cost of archiving amongst many institutions. for this reason, collaborative print journal storage initiatives have existed for many years around the world, allowing participating institutions to remove these titles from prime library space. however, depending on the collaborative strategy being used, there are still potential issues, even while there are undeniable benefits. the collaborative approach is excellent for sharing costs, but unless a page-by-page review is conducted of the items being archived, along with the archiving of best copy, there is a risk of archiving a damaged copy. this could prevent the rescanning of specific journal articles, should it be needed, depending on where the damage exists.   as part of the current study, the benefits of a collaborative approach were demonstrated while consulting the print equivalent volumes held at the university of saskatchewan to compare electronic backfile and print equivalent content. it was discovered that since the 2010 study four titles had been removed from the university of saskatchewan collection. each of these titles were part of the council of prairie and pacific university libraries shared print archive network initiative. while the titles were no longer at the university of saskatchewan, they were held at partner institutions. the volume issues were able to be examined at the archive partner institutions. in one case, however, the title was not found at the initial archive partner consulted, but was available at the second archive holder. this may have been because the title was in the process of being transferred to the institution’s storage facility, but this example demonstrates the importance of having multiple archived copies.   while this study and others have shown that there are issues with the quality of electronic surrogates of print journal articles, there is a question of the extent of the impact to researchers if they had to rely solely on poor-quality electronic surrogates. in the current study, the author used two proxies to estimate the possible impact of poor-quality electronic surrogates. the first examined the citations to electronic surrogates of articles that were found in this study to fail. with 54% of the electronic surrogates having citations since the 2010 original study, it is apparent that many of the papers are still being actively consulted and referenced. on average, there were 9.9 citations per paper when the one paper with over 600 citations is not included in calculating the average.   the second proxy for impact was the web of science usage count feature. there were 36 or 24% of the failed pdfs with web of science usage. of these, seven also did not have citations from 2010 onward, bringing the total to 88 papers or 58.7% of the failed pdfs with citations or web of science usage data. the author found, however, that the web of science usage data had some issues with reliability. the original data were collected in early 2016 and in preparation for writing this paper, were refreshed in early 2017. the author was surprised to note some decreases in the web of science usage data gathered since 2013. it was logical that usage would only increase over time. yet for 26 papers, this figure actually decreased. clarivate was contacted and asked why this might be the case. clarivate responded that in april 2016, they had identified a new type of bot activity and they had adjusted their algorithms to account for the elevated usage counts (personal communication, april 4, 2017). the result was a usage count reduction to zero for 17 of the 26 affected papers.   the proxy measures for impact, particularly citations, demonstrate that researchers use the failed papers actively. the degree of impact if authors had to rely solely on poor-quality electronic surrogates will be dependent on whether the researcher needs to consult the image, text, or content in the paper that is of poor quality or missing. regardless, with 58.7% of the failed papers being cited or having web of science usage data from 2010 onward, the current study indicates that relying solely on electronic surrogates has a potentially significant impact on researchers when the electronic surrogate is of sufficiently poor quality to require consulting the original print version.   conclusion   this study was undertaken to determine whether evidence of electronic surrogate quality continued to support the need to preserve print equivalent journals collections. evidence was sought by re-examining pdf documents that had been classified as failing in a previous study (ladd, 2010) to determine if their quality had improved. the study also examined whether there was evidence of potential impact on researchers if they relied only on poor-quality electronic surrogates. an indication of the extent of the potential impact was first examined by tallying the citations to scholarly pdf documents that were observed to continue to fail in the current study, and second by recording their web of science usage counts.   the data demonstrate clearly that there continues to be an issue with the quality of pdfs held in electronic journal backfiles. almost all of the scholarly pdfs that no longer failed came from a single electronic journal archive (elsevier science direct backfile – medicine and dentistry), following a massive project conducted by the publisher to identify and replace poor-quality images. despite elsevier’s initiative being successful in addressing many of the poor-quality images, this study still observed numerous poor-quality images and other problems in their backfiles.   an alternate approach to the one used by elsevier, and likely more cost effective, may be a collaborative approach among vendors, libraries, and users to identify poor-quality scholarly pdfs and replace them with high-quality, high-resolution pdfs. joseph (2012) suggested that elsevier should at a minimum provide a form on their website to allow readers and librarians to report quality issues and incorporate addressing the reported problems into their workflows. a crowd-sourcing approach would help address the costs associated with reviewing and identifying scanned pdfs with poor-quality images, graphs, line drawings, and text. in addition, this approach would identify where poor quality control has resulted in content missing or being incorrect. while not a comprehensive strategy to address all of the quality issues with scanned journal pdfs, it would identify problems as the publications are being used, an indicator of potential future use.   because of the cost, time, and money to address this significant problem of poor-quality scanned journal pdfs, it can be concluded that it will persist for the foreseeable future and thereby require the preservation of print serials. thus, it would be desirable to have a comprehensive strategy that ensures that there are complete preserved copies available. one way to ensure this objective would be to use page-by-page verification for each preserved journal volume and issue. due to the costs in time and money, this strategy is not likely to be used extensively, but if implemented would be best achieved through a collaborative approach to share the resource implications. as a less expensive alternative, redundancy for any given title among different preservation initiatives would logically compensate for less rigorous content verification. this strategy, however, does carry its own costs since it would require a greater number of copies to be preserved.   collaborative print journal storage initiatives have existed for numerous years. this study and others indicate that there will be an ongoing need for print equivalent storage for the foreseeable future. while there have been papers written about individual initiatives and about initiatives in general, it would be of value to study at least a cohort of these initiatives to have data, for example, on their extent, retention period commitments, and validation method employed. this will shed light on whether the initiatives collectively are achieving a level of print preservation for these resources that will help to ensure that quality print journals are available, to allow for consultation or rescanning should the need arise.   references   association of research libraries. (2007). research libraries’ enduring responsibility for preservation. retrieved april 16, 2018 from http://www.arl.org/bm%7edoc/preservation_responsibility_24july07.pdf   bracke, m. s., & martin, j. (2005). developing criteria for the withdrawal of print content available online. collection building, 24(2), 61–64. http://doi.org/10.1108/01604950510592670   campbell, s. (2003). print to electronic journal conversion: criteria for maintaining duplicate print journals. feliciter, 49(6), 295–297.   chen, x. (2005). figures and tables omitted from online periodical articles: a comparison of vendors and information missing from full-text databases. internet reference services quarterly, 10(2), 75–88. http://doi.org/10.1300/j136v10n02_07   chrzastowski, t. e. (2003). making the transition from print to electronic serial collections: a new model for academic chemistry libraries? journal of the american society for information science and technology, 54(12), 1141–1148. http://doi.org/10.1002/asi.10318   erdman, j. m. (2006). image quality in electronic journals: a case study of elsevier geology titles. library collections, acquisitions, and technical services, 30(3–4), 169–178. http://doi.org/10.1016/j.lcats.2006.08.002   hawkins, l., & shadle, s. (2004). electronic journal forum: reflections on wrapping paper: random thoughts on aacr2 and electronic serials. serials review, 30(1), 51–55.                 http://doi.org/10.1080/00987913.2004.10764877   henebry, c., safley, e., & george, s. e. (2002). before you cancel the paper, beware: all electronic journals in 2001 are not created equal. the serials librarian, 42(3 & 4), 267–273. http://doi.org/10.1300/j123v42n03_17   joseph, l. (2006). image quality in electronic journals: a case study of elsevier geology titles. library collections, acquisitions, and technical services, 30(304), 169-178. http://doi.org/10.1016/j.lcats.2006.12.002   joseph, l. (2012). improving the quality of online journals: follow-up study of elsevier’s backfiles image rescanning project. library collections, acquisitions, and technical services, 36(1), 18-23. http://doi.org/10.1016/j.lcats.2011.08.001   joseph, l. e. (2014). image quality in university of illinois digital geology dissertations from proquest. issues in science and technology librarianship, (77). http://doi.org/10.5062/f4z31wm1   kalyan, s. (2002). non-renewal of print journal subscriptions that duplicate titles in selected electronic databases: a case study. library collections, acquisitions, and technical services, 26(4), 409–421. http://doi.org/10.1016/s1464-9055(02)00287-7   keller, a. (2005). the race to digitize: are we forfeiting quality? serials, 18(3), 211–217. http://doi.org/10.1629/18211   ladd, k. f. (2010). an examination of the rate and content equivalency of electronic surrogates and the implications for print equivalent preservation. evidence based library and information practice, 5(4), 7-20. http://doi.org/10.18438/b83p6v   martellini, e. (2000). physics journals and their electronic version: a comparison. high energy physics libraries webzine, (2). retrieved from http://webzine.web.cern.ch/webzine/index.html     mccann, s., & ravas, t. (2010). impact of image quality in online art history journals: a user study. art documentation: journal of the art libraries society of north america, 29(1), 41–48. http://doi.org/10.1086/adx.29.1.27949538   robinson, a. (2010). university of kansas print and electronic journal comparison study. art documentation: journal of the art libraries society of north america, 29(1), 37–40. http://doi.org/10.1086/adx.29.1.27949537   sprague, n., & chambers, m. b. (2000). full text databases and the journal cancellation process: a case study. serials review, 26(3), 19–31. http://doi.org/10.1080/00987913.2000.10764597   thohira, m., chambers, m. b., & sprague, n. (2010). full-text databases: a case study revisited a decade later. serials review, 36(3), 152–160. http://dx.doi.org/10.1080/00987913.2010.10765304   van gijlswijk, e., & clark, b. (2010). sciencedirect upgrades 600,000 backfiles pages. elsevier library connect, 8(1), 4. retrieved from http://libraryconnect.elsevier.com/sites/default/files/lcn0801.pdf   web of science core collection help. (2018). retrieved march 23, 2018, from http://images.webofknowledge.com//wokrs529ar7/help/wos/hp_usage_score.html   weessies, k. w. (2012). local history maps in full text resources. journal of map and geography libraries, 8(3), 230–241. http://doi.org/10.1080/15420353.2012.700300     appendix titles compared in each collection   elsevier science direct backfile medicine and dentistry ü  american journal of orthodontics ü  biochemical medicine and metabolic biology ü  british journal of tuberculosis and diseases of the chest ü  international journal of nuclear medicine and biology ü  prostaglandins, leukotrienes, and medicine   elsevier science direct backfile social sciences ü  government publications review ü  journal of behavioral economics ü  social science & medicine. part b, medical anthropology ü  studies in comparative communism ü  transportation research. part a, general   jstor arts and sciences 1 ü  american journal of mathematics ü  journal of health and human behavior ü  journal of the history of ideas ü  reviews in american history ü  speculum   oxford university press journals digital archive ü  occupational medicine ü  parliamentary affairs ü  past & present ü  rheumatology ü  the year's work in clinical and cultural theory   springer link archive (mathematics archive) ü  computational optimization and applications ü  constraints ü  journal of cryptology ü  journal of nonlinear science ü  k-theory   wiley blackwell backfiles humanities and social sciences (acquired as wiley interscience (synergy blackwell) – humanities and social sciences backfile) ü  papers in regional science ü  social policy and administration ü  journal of philosophy of education ü  review of policy research   wiley blackwell backfiles science, technology and medicine (acquired as wiley interscience (synergy blackwell) – science, technology and medicine backfile) ü  european journal of clinical investigation ü  international journal of experimental pathology ü  journal of human nutrition and dietetics ü  journal of oral pathology and medicine ü  sedimentology       evidence summary   motivational design and problem-based learning may increase student engagement in information literacy instruction sessions   a review of: roberts, l. (2017). research in the real world: improving adult learners web search and evaluation skills through motivational design and problem-based learning. college & research libraries, 78(4), 527-551. https://doi.org/10.5860/crl.78.4.527   reviewed by: ann glusker research & data coordinator national network of libraries of medicine, pacific northwest region university of washington health sciences library seattle, washington, united states of america email: glusker@uw.edu   received: 1 sept. 2017    accepted: 5 dec. 2017      2017 glusker. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to determine whether the use of the arcs (attention, relevance, confidence, and satisfaction) model of motivational design, combined with the problem-based learning approach, improves the skills, confidence, and perception of workshop relevance among non-traditional students in information literacy sessions.   design – experimental study, one group pre-test and post-test.   setting – community college in denver, colorado, united states.   subjects – 41 community college students.   methods – a convenience sample of three community college student groups each attended an information literacy session. the session was constructed using principles and strategies outlined in the arcs model of motivational design and the problem-based learning approach. pre-test and post-test instruments were developed by the author after a literature review. the students were given the information literacy-related pretest before the session. after receiving instruction, the comparable posttest (with different literacy challenges) was administered.   main results – a comparison of the pre-test and post-test results showed that there were increases in the students’ search skills; their confidence in their own search skills; and their perceptions of workshop relevance in relation to their needs and to real-world situations.   conclusion – this study focuses on the use of motivational design for information literacy instruction. it addresses a gap in the research literature, as it explicitly examines issues of concern regarding the instruction of non-traditional students. the conjunction of the arcs model and problem-based learning is considered to be an effective strategy for improving learning and perceptual outcomes for non-traditional students in information literacy contexts. this is important because: 1) information literacy skills are a central aspect of successfully transitioning from the educational setting to the modern workplace; 2) increased confidence can enhance students’ sense of self-empowerment and self-efficacy, as well as decreasing “library anxiety”; and 3) establishing a sense of the personal relevance of information literacy engages students with tools that they can and will actually use in work and life situations.   in addition, the author connects these findings to two other areas. one is the new acrl (association of college and research libraries) framework for information literacy for higher education; the author notes that “threshold concepts”, defined by roberts as “big picture ideas that are foundational to the field”, relate best to teaching techniques such as problem-based learning. the other is the concept of metacognition, which is an aspect of metaliteracy; the author states that the study’s information literacy session addressed three of four metaliteracy goals being considered. future avenues of research and collaboration will include librarians working with learning scientists around the framework content; finding new and engaging methods for teaching literacy concepts and assessing learning; incorporating metacognitive awareness into teaching and assessment; and specifically focusing on transferable skills and knowledge, in the service of preparing non-traditional students for the world of work.   commentary   as the modern workplace becomes more complex and information-driven, it is increasingly important that we engage students with improving their information literacy; this is especially the case for non-traditional students. this study makes important contributions: it addresses the gap in the literature regarding non-traditional students, and it tests the unusual combination of the arcs motivational design model with the problem-based learning approach (the author claimed to be the first to do this, but there is an earlier study that also uses these two methods in combination (chang and hsu, 2016)).   for this evidence summary, the paper’s methodologies were systematically assessed using glynn’s (2006) critical appraisal checklist. a number of issues arise in comparison with the checklist, most of which are included in the paper’s limitations section. these include: the use of a convenience sample rather than random selection; the resulting demographic profile of subjects not matching the student population from which they were drawn; the use of only one group for pre-testing and post-testing rather than using a control group; the subjectivity inherent in using self-reporting; the lack of a validated instrument; and the absence of testing the longer-term retention of the benefits reported at the end of the class. in addition, the instrument was only tested in one location, meaning that multiple types of educational settings and types of non-traditional students were not considered. for these reasons, although the author’s findings were positive on all outcomes, the title of this summary uses the word “may”. additionally, as gross and latham (2013) note, an increase in skills does not necessarily raise students to proficiency; attainment of information literacy is complex, so further work may be necessary to ready students for the workforce.   the author also focuses on two topics that connect with the research; metacognition and threshold concepts (as outlined in the acrl framework for information literacy for higher education). the focus on metacognition arose from the literature review, but it was not considered in depth in the study, as the study was already testing two other learning models in a detailed way. therefore, this focus could have been saved for another paper.  however, there is still a connection with the acrl framework’s “threshold concepts”: better self-cognition leads to better transfer of skills between settings and “transferability” can be considered one of the concepts. cited author kuglitsch (2015) explicitly suggests teaching toward transferability of knowledge and skills, so that students can connect the big picture of information literacy to their disciplinary environments. likewise, cited author lloyd (2013) exhorts instructors to incorporate the transition to the workforce in student information literacy sessions. thus, transferability is arguably the “big picture” for roberts’ study. future work could fruitfully focus on this, perhaps by repeating the study in workplaces, or some similar intervention.   references   association of college & research libraries (2015). "framework for information literacy for higher education", american library association. http://www.ala.org/acrl/standards/ilframework (accessed december 13, 2017). document id: b910a6c4-6c8a-0d44-7dbc-a5dcbd509e3f   chang, n.c., & hsu, h.y. (2016). a study on integrating problem-based learning into the innovative teaching in information literacy and ethics. journal of educational media & library sciences, 53(2), 171-209. https://doi.org/10.6120/joemls.2016.532/0010.rs.cm   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154   gross, m. & lathan, d. (2013). addressing below proficient information literacy skills: evaluating the efficacy of an evidence-based educational intervention. library & information science research, 35(3), 181-190. https://doi.org/10.1016/j.lisr.2013.03.001   kuglitsch, r.z. (2015). teaching for transfer: reconciling the framework with disciplinary information literacy. portal: libraries and the academy, 15(3), 457-470. https://doi.org/10.1353/pla.2015.0040   lloyd a. (2013). building information resilient workers: the critical ground of workplace information literacy. what have we learnt? in: in s. kurbanoğlu, e. grassian, d. mizrachi, r. catts, s. špiranec (eds.),  worldwide commonalities and challenges in information literacy research and practice. ecil 2013. communications in computer and information science, 397. springer, cham. https://doi.org/10.1007/978-3-319-03919-0_28   research article   exploring library activities, learning spaces, and challenges encountered towards the establishment of a learning commons   maryjul t. beneyat-dulagan librarian cordillera state institute of technical education (baguio city school of arts and trades) baguio city, philippines email: djul351@gmail.com   david a. cabonero faculty, school of graduate studies saint mary’s university bayombong, nueva vizcaya, philippines email: bluegemini7777@yahoo.com     received: 6 may 2022                                                                accepted: 10 oct. 2022      2023 beneyat-dulagan and cabonero. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30164     abstract   objectives – this study was conducted to determine the library activities, preferred learning spaces, and challenges encountered by the students of mountain province state polytechnic college (mpspc) library, philippines. specifically, it sought to answer the following problems: 1) what are the library activities of mpspc students?; 2) what are the preferred learning spaces in terms of a) physical environment and b) virtual environment?; and 3) what are the challenges associated with library learning activities encountered by the mpspc students? the study then will be used to explore the feasibility of proposing a learning commons.   methods – this study used a descriptive research method to determine the library activities, learning spaces, and challenges encountered by mpspc students in the philippines. it made use of a researcher-made survey questionnaire. problem statement number 1 dealt with the library activities of mpspc students. problem statement number 2 dealt with the preferred learning spaces. data were gathered from 500 graduate and undergraduate students from a total of 3,015 enrolled during the first semester of the sy 2019-2020 using a purposive random sampling technique. descriptive statistics such as frequency, percentage, and rank were used.   results – the most frequent library learning activities performed by the mpspc students were doing assignments, using reference books, searching/browsing printed materials, reviewing notes, and writing. students’ least frequent library activities were surfing the web, using the computer, using e-resources, eating while reading/writing, and sleeping. the most preferred physical learning spaces were a makerspace, group study spaces, quiet study rooms, and individual study spaces (individual study carrels), while the most preferred virtual learning spaces were computer workstations, interactive learning spaces, video viewing stations, and internet cafés. the overall challenges encountered by mpspc students were insufficient learning spaces, poor internet connection, inability to find documents or books needed, lack of reading area, lack of printing or photocopying service, lack of professional books, and lack of e-resources. the least challenges encountered by mpspc students included very high library fees, poor ventilation, poor lighting facility in the designated area, uncomfortable furniture, and lack of staff’s kindness.   conclusion – the mpspc students perform various educationally purposeful library activities, which are generally engaging and support the library's mission. students vary in their needs of physical and virtual learning environments. both of these learning spaces are in demand among students, which are the key components of the learning commons. also, they specified the need for adequate learning spaces to support their various library learning activities. the findings serve as the basis for crafting a project proposal to establish a learning commons tailored to mpspc students’ library activities and preferred learning spaces, with consideration for the challenges encountered by students, to support their learning and academic success.     introduction   a library is a place for nurturing the mind. it supports learning, information, and research needs; thus, it is vital to students' educational growth. in support of an institution's educational objectives to meet its diverse learners' needs, libraries should provide a quiet and social space for students’ various learning activities (choy & goh, 2016), a healthy and safe environment for learning (barton, 2018), and offer education and relaxation (waxman et al., 2007). moreover, the library is a learning environment characterized by abundant and rich information sources and well-designed learning spaces. within a library space, students identify physical and virtual environments that help them achieve their learning goals.   twenty-first century learners are connected to digital technologies as their primary learning tools, but as global changes in information occur, students’ learning activities are affected (mcleod, 2015). the nature of tertiary education drastically changed as the 21st century evolved and has impacted the nature of academic libraries (turner et al., 2013). twenty-first century learning is often connected to an inquiry approach in which students actively engage in their learning, accessing material and scaffolding their knowledge to create rather than solely acquire information (stripling, 2008). the preferences of library users in library spaces can change quickly and unpredictably (gstalder, 2017), which affects library support of the teaching and learning process (roberts, 2007). in relation to this, turner et al. (2013) observed that new teaching and learning pedagogies in higher education were influenced by social constructivist learning theories and self-discovery practices. these theories supported that the most significant learning takes place when individuals participate in social learning activities (matthews et al., 2011). as such, library users have high expectations of quality academic facilities, such as the provision of library spaces, library commons, and the like (flaspohler, 2012).   how can a library position itself in these academic environments? how can a library be responsive to the changing nature of information access and the changing nature of users? there is a need for rethinking the information and physical needs of students. moreover, there is a need for academic library innovation to better support the diverse learning needs of students and accommodate students’ learning styles. lankes (2016) suggested that redesigning and conceptualizing the library is essential to the 21st century. he further stated that a move to a learning commons approach is one tactic to meet users’ expectations. roberts (2007) believed that establishing a learning commons will support the teaching mission of an institution. it would complement new teaching and learning pedagogies in higher education which have shifted away from a teaching culture and toward a culture of learning (bennett, 2003).   furthermore, libraries reinvent themselves as they face new roles, such as making resources more accessible, connecting learners, and constructing knowledge. also, students do not just need information, they need a place that encourages active involvement and motivates them. learning commons allow various learning activities (holland, 2015), and help both libraries and students remain current with modern demands and lifestyles.   with these paradigm shifts in libraries and education, changes gradually occur with the library’s environment and ambiance. in the mountain province state polytechnic college (mpspc) library, not enough areas serve the different learning needs of library users. a lack of technology and facilities to help library users explore, create, and share knowledge has been observed. moreover, poor services have reduced both the number of users and use of the library collection. students’ needs and expectations affect all aspects of their learning, specifically in the library. diverse reading habits and preferences of the students have been observed by the researchers as well.   the mpspc library has been accessible to all users because it is almost centrally located on campus, but the reading area was not enough to accommodate the students. instead, students use the corridor as a learning area since the reading room was insufficient. students studying in the corridor and even inside the room complained to the library staff because they were distracted by students passing by. students found it hard to study and concentrate because of the noise. there were not enough individual study spaces or group discussion rooms because the room was just a common space for every user engaged in any activity. based on observation by the researchers, the problem was the slow internet connection, wherein students found it hard to conduct research online, which caused some students to leave the library. faculty and school administrators noticed complaints about the insufficient reading area and the misbehaviour of users. hence, the plight of this academic library encouraged the researchers to conduct this study.   problems of the study   this study was conducted to determine the library activities, preferred learning spaces, and challenges encountered by the students of mpspc library. specifically, the study sought to answer the following problems: 1) what are the library activities of mpspc students?; 2) what are the preferred learning spaces in terms of a) physical environment and b) virtual environment?; and 3) what are the challenges associated with library learning activities encountered by the mpspc students?   scope and limitations of the study   the study was limited to one state college in the philippines primarily to determine the learning activities and spaces in the library and the challenges encountered by the students, which served as the basis for establishing a learning commons. this was conducted during the first semester of sy 2019-2020 and focused on 500 participants who were both undergraduate and graduate students.   literature review   in the philippines, some libraries are still traditional in giving services to their users, resulting in a lack of social opportunities within the library. this limits the opportunities for students to interact with each other in the library spaces (mccunn & gifford, 2015). this can be observed through the image projected by the librarian, such as shushing students for speaking too loudly, ringing a bell to remind them of their unruly behaviour, and the like. with the new breed of library users, their diverse learning activities, habits, styles, and needs are changing and should be addressed. this could be answered by adopting a new library model such as the learning commons, which allows students to enhance their social skills while researching, reading, and learning. in establishing a model, there are imperative things to consider, such as: 1) to identify the key priorities, such as which learning activities occur in a successful learning commons (king, 2016); 2) to know the learning activities of the users better to realize their needs (spencer, 2007); 3) to analyze the various activities, including which are most prevalent among library users (choy & goh, 2016); 4) to understand the various learning needs of students, such as learning activities, preferred learning spaces, and challenges faced by students (qayyum ch. et al., 2017); and 5) to relate the various activities of the students in the library to academic achievement (paretta & catalano, 2013).   implementing a learning commons would primarily encourage students to use the library and benefit from its services. however, this idea must be supported by asking the right questions to students regarding their library activities, how they learn, and their use of library services (suarez, 2007). these learning activities and study space preferences of library users relate to establishing a functional learning commons. thus, surveying students' library activities and preferred learning spaces provides the evidence necessary to make effective decisions about what facilities and equipment should meet their various needs (mccrary, 2017).   the learning spaces model furthers the mission of the learning commons by providing various formal and informal flexible learning spaces that facilitate better learning (turner et al., 2013), and these physical and virtual learning spaces can impact learning (oblinger, 2006). it can bring people together to encourage exploration, collaboration, and discussions. these spaces should be flexible and networked, bringing together formal and informal activities in an environment that acknowledges that learning can occur anywhere, at any time, in either physical or virtual spaces. the physical and virtual environments provide students with a comfortable place to relax, learn, and create (cicchetti, 2015). moreover, spatial designs influence students' learning activities, and the relevance of spatial designs that encourage and support dynamic, engaged, and inspired learning is a fundamental feature of the learning spaces (roberts, 2007). the impact of spaces becomes more prominent as higher education pedagogical practices move from the traditional to a more flexible, student-centred approach. evolving learning spaces convey a new image of the library, marking a new direction in library and educational philosophies (somerville & harlan, 2008).   the development of learning spaces supports innovative pedagogical approaches and environments that promote student engagement in the learning process (elkington & bligh, 2019). how and why users have different preferences in learning spaces depends on their individual needs and styles. moreover, there are advantages to student learning in providing a range of spaces. various collaborative and independent spaces promote self-directed learning (keating & gabb, 2005). non-quiet spaces in the library, such as group study and flexible learning spaces, are ideal for many library users (freeman, 2005).   a learning commons consists of physical and virtual environments designed for learning. the centre for student learning fosters creativity, encourages patron use of space, offers new technologies, and uses space creatively to encourage inquiry-based thinking (mihailidis & diggs, 2010). it is a space designed for collaboration and access to information and other tools, such as electronic resources. here, students will be empowered as they take part in the learning commons, which will lead to more learning and better preparation for their careers. students’ involvement in the learning commons produces a better student success rate (khan, 2020), and students learn best when they are allowed to learn in an environment that is both welcoming and supportive (holeton, 2020).   a clear understanding of how the learning commons benefits students is also the foundation for a successful transition (cicchetti, 2015). libraries need to remain relevant and support learning in new ways. libraries recognize that, because of the internet and web 2.0 applications, students have new powers and abilities that facilitate independent access to information (watstein & mitchell, 2006). blummer and kenton (2017) mentioned that learning commons has no standard definition. yet, learning commons represent academic library spaces that provide computer and library resources and a range of academic services that support learners and learning. turner et al. (2013) argued that designers of learning commons readily understand that learners are not merely information consumers. instead, they actively participate with information to create meaningful knowledge and wisdom.   as society continues to experience a pedagogical shift in learning, students should be given more opportunities to make connections, collaborate, communicate, think critically, and be creative. learning in a learning commons environment is purposeful, authentic, active, and student-centred (mccunn & gifford, 2015). there have been numerous studies on learning commons, one of which performed surveys on their own users’ needs (yebowaah & plockey, 2017). students’ various learning activities have to be considered in order to provide appropriate learning spaces (brown-sica et al., 2010). rawal (2014) asserted that:   like bandura’s (1977) idea of “reciprocal determinism,” where the interactions among environmental, cognitive, and behavioral influences create the synergy to affect how one behaves in a specific context, so does the reciprocity among the physical, virtual, and socio-cultural aspects of a learning commons affect how students learn within a commons. a truly holistic learning commons is a nexus for negotiating ideas and producing new knowledge. it is that bustling bazaar where knowledge, discoveries, and innovations are born, nurtured, and set forth to impact the rest of the world. (p. 67)   our review of the literature revealed that our study is unique as it dwells on library activities and preferred learning spaces among students in the philippines. hence, this study will be used to explore a learning commons as one of the new features of our library. barton (2018) mentioned that the learning commons model is geared to understand and identify learning needs in accordance with the learning activities, preferences, and challenges of library users.   methods   this study utilized a descriptive method of research to determine the library activities, learning spaces, and challenges encountered by mpspc students in the philippines. it made use of a researcher-made survey questionnaire. problem number 1 dealt with the library activities of mpspc students and was based on the study of cabfilan (2012). problem number 2 dealt with the preferred learning spaces and was adopted from the study of peterson (2013). however, it has been modified to suit the research design by contextualizing the items in the mpspc library. the survey questionnaire is composed of three parts, namely: 1) the different library activities of mpspc students, 2) the preferred learning spaces, and 3) the challenges encountered by the respondents relative to learning activities within the library. this questionnaire underwent face and content validity by three library and information science professors and one research professor at saint mary’s university (philippines).   data were gathered from 500 graduate and undergraduate students from 3,015 enrolled during the first semester of the sy 2019-2020, from august to december 2019 at mpspc, bontoc campus (table 1), using a purposive random sampling technique. in gathering the needed data, the following procedures were undertaken: 1) a permission letter was sent to the mpspc president to seek approval for the conduct of the study for the students enrolled in the various programs; 2) the letter was addressed to the president through the deans of undergraduate and graduate studies; 3) upon seeking approval, a letter was submitted to the director of mpspc-registrar for the number of enrollees in the various programs to identify the number of students in each program; 4) the questionnaire was administered to the students who were visiting the library voluntarily. one of the researcher’s colleagues helped administer the questionnaire; 5) an informed consent letter was attached to the questionnaire. the respondents did not receive any payment for their participation nor any reimbursements. participants had the right to refuse to continue, with any information already provided not used in the study. it was emphasized to them the assurance of the confidentiality of their answers; 6) the questionnaires were immediately retrieved and checked if all items were answered; and 7) questionnaires were submitted to the statistician. descriptive statistics such as frequency, percentage, and rank were used.   table 1 respondents of the study course/department no. of enrollees no. of respondents (n) bachelor of science and criminology 1,244 206 bachelor of science in nursing 213 35 bachelor of science and information technology 123 20 bachelor of science office administration 75 12 bachelor of arts in political science 41 7 bachelor of science in business administration 170 28 bachelor of science in accountancy 158 26 bachelor of secondary education 400 66 bachelor of elementary education 233 39 bachelor for early childhood education 9 2 bachelor of special needs education 13 3 graduate school 174 29 bachelor of science in tourism bachelor of science in tourism management bachelor of science in hotel and tourism management bachelor of science in hospitality management associate of arts in hotel and restaurant management 162 27 total 3,015 500   results and discussion   the library activities of students   this study refers to the various activities performed by diverse students in the library. these learning activities are purposeful and aim to improve behaviour, information, knowledge, understanding, attitude, values, or skills (table 2). this includes different types of learning, such as self-learning and others, and learning could be formal or informal (eurostat, 2016).   the preferred activities done in the library were doing assignments, using reference books, searching or browsing printed materials, reviewing notes, writing research works, reading (periodical/ fiction books/ non-fiction books), studying in a group, and studying alone on my books or materials. these activities were all academic-related, supporting the fact that the library is the first place to get information as it houses universal knowledge (bailin, 2011). this indicates that libraries significantly impact students’ academic achievements (khan, 2020; sriram & rajev, 2014). it could be attributed to the availability of resources when doing their assignments. also, students go to the library to search or browse printed materials and eventually use reference books, which suggests that materials in the library are useful and relevant.   table 2 the library activities of students activities n % rank doing assignments 455 91.0 1 using reference books 396 79.2 2 searching/ browsing printed materials 385 77.0 3 reviewing notes 384 76.8 4 writing (research works) 369 73.8 5 reading (periodical/ fiction books/ non-fiction books) 362 72.4 6 studying in a group 345 69.0 7 studying alone on my own books/ materials 324 64.8 8 sitting comfortably while reflecting 316 63.2 9 interacting with librarians/ getting help from staff members 314 62.8 10 listening to music while studying/ reading/writing 290 58.0 11 surfing the web 270 54.0 12 using computer 265 53.0 13 using e-resources 255 51.0 14 eating while reading/ writing 168 33.6 15 others: sleeping 35 7.0 16   this coincides with iroaganachi and ilogho (2012), who found that students use reference materials frequently, which can be attributed to the orientation program designed for students. on the other hand, listening to music while studying, reading, or writing; using e-resources; using computers; surfing the web; and eating while reading or writing were the least common activities done in the library. also, eating while reading or writing was ranked 15th, which means that some students do not favor the library policy that food and drink are prohibited inside. however, some mpspc students prefer a place to study while having a snack, and this could be observed in some libraries allowing them to bring food and drinks. this finding corroborates the idea in 21st-century learning wherein libraries are innovating to meet the demands of these learners, in which food and drink are welcomed in the libraries (roberts, 2007).   also, it is worthwhile to mention that 35 respondents wrote sleeping as one of their library activities. this connotes that the library is not just a place to study but a place that provides relaxation to students (waxman et al., 2007).   however, it is very surprising to note that learning activities relating to computer technology, such as surfing the web, using computers, and using e-resources were ranked 12th, 13th, and 14th, respectively. seemingly, students do not prefer using information technology to satisfy their library information needs, thus resulting in minimal utilization of e-resources (yebowaah & plockey, 2017). this contradicts the findings of martin (2008), that students use technology frequently thus changing the learning environment of higher education. this suggests that a slow internet connection would make students dissatisfied with using computers and resources and make it challenging to research online.   the preferred learning spaces   according to head (2016), there are appropriate library designs for learning spaces, and they should be different in every library since it is in accordance with the learning activities and preferences of every library user. it was further pointed out by bieraugel and neill (2017) that designing library spaces is imperative for the different intended needs, activities, preferences, and styles of library users. also, choy and goh (2016) reiterated that the design of spaces in support of learning is far more complex as a variety of users’ activities and styles need to be considered.   figure 1 preferred learning spaces in physical environment.   in this study, the most preferred learning spaces in terms of physical environment were makerspace, group study spaces, quiet study rooms, and individual study spaces, respectively, as shown in figure 1. makerspace was the most prevalent, which infers that learning is best acquired through hands-on activities. group study space was second, which assumes that students may feel they can learn better in groups. this indicates that noise should be welcomed and considered in the group study area within the library (mohanty, 2002). meanwhile, study lounge was ranked 5th as the preferred learning space, implying that there are students who prefer working while socializing as well (waxman et al., 2007). undeniably, some students expect the library to offer space not only for scholarly pursuits but also for socializing (paretta & catalano, 2013). however, some students still prefer individual study spaces/ individual study carrels and quiet study rooms. seemingly, they prefer to learn best in silence and do not like being disturbed when they are studying (arenson, 2013).   figure 2 preferred learning spaces in virtual environment.   the most preferred learning spaces in the virtual environment were computer workstations, interactive learning spaces, video viewing stations, and internet cafés, respectively (figure 2). the computer workstation is the most preferred learning space in terms of the virtual environment. this implies that activities which demand computer are prevalent among the students. as mentioned by singh and wadhwa (2006), computers are an excellent learning tool. this signifies that 63% of library users prefer to work individually in a computer workstation, while others prefer working in an interactive learning space.   it is interesting to note that interactive learning space was ranked 2nd, which implies that students want spaces that encourage them to study independently through technology. this finding supports the idea that learning is engaging, and engagement is expected to increase students’ learning outcomes (vercellotti, 2018). this preference for interactive learning space implies that students have varied learning styles, and, in this case, it requires the use of technology for them to learn better.   also, the video viewing station was ranked 3rd, which implies that there are students who are both visual and auditory learners who prefer watching and listening in some areas of the library. as mentioned by alawani et al. (2016), students still prefer video technologies that boost their learning experience. however, internet café ranked last, implying that few students prefer learning while having coffee or snacks. seemingly, this idea is not yet practiced by the students and the library. perhaps their traditional beliefs of eating inside the library are not accepted as the standard norm. as mentioned, 21st-century libraries should meet the needs of these learners, thus allowing them to eat while learning in the library (holland, 2015).   as shown in figures 1 and 2, the physical and virtual environments were rated as learning space preferences among students, which are the key components of a learning commons (pressley, 2017). the findings show that students demand such spaces in accordance with their learning activities in the library.   challenges encountered by the mpspc students   mcmullen (2008) described a learning commons as a “dynamic place that encourages learning through inquiry, collaboration, discussion, and consultation.” (p. 1). she further asserted that it is necessary to understand the activities engaged by students. the learning commons is not just a concept but a place for learning in the library (roberts, 2007). these learning commons have been created to support the teaching missions of the respective parent institutions. academic institutions support this model because faculty and administration recognize that students learn in dynamic and various ways. mccrary (2017) supported the need to develop a learning commons since the library is not just a place to store books and study but rather a place where meaning and learning emerge from access to knowledge. however, its implementation can also be hampered by challenges, which are listed in table 3.   table 3 challenges encountered by the mpspc students relative to their library learning activities areas challenges n % rank physical facilities insufficient learning spaces for various activities 319 63.8 1 services poor internet connection 262 52.4 2 services inability to find documents/ books needed 231 46.2 3 physical facilities lack of reading area/ reading area is not enough 209 41.8 4 services lack of printing or photocopying services 208 41.6 5 library collection lack of professional books 207 41.4 6 library collection lack of e-resources 206 41.2 7 physical facilities lack of toilet facilities 166 33.2 8 physical facilities lack of installed security equipment 149 29.8 9 financial resources lack of support /budget is not enough to sustain library projects or programs 142 28.4 10 human resources lack of support staff 135 27 11 human resources limited number of professional librarians 124 24.8 12 financial resources very high library fee 118 23.6 13 physical facilities poor ventilation 113 22.6 14 physical facilities poor lighting facility in the designated reading areas 112 22.4 15 physical facilities uncomfortable furniture 110 22 16 services lack of staff’s kindness 109 21.8 17   among the physical facilities, insufficient learning spaces for various activities (ranked 1st, with 63.8% in agreement) and lack of reading area or reading area is not enough (ranked 4th, with 41.8% in agreement) were challenges encountered by the mpspc students. students have various activities, but not all spaces can accommodate these activities. libraries should be well designed to accommodate students' learning requirements and enhance their learning outcomes and satisfaction (li et al., 2018). furthermore, the result corroborates with the study of bailin (2011) that students demand ample space for reading, especially when they flock to the library. indeed, ranganathan’s 5th law states that the library is a growing organism (barner, 2011). as collections continuously increase, the physical spaces also widen to accommodate more library users and eventually maximize the use of the collections, thus making the library a growing institution of learning.   the least challenges encountered on physical facilities were lack of toilet facilities, lack of installed security equipment, poor ventilation, poor lighting facility in the designated reading area, and uncomfortable furniture. in relation to the findings on preferred learning spaces, these challenges reported by the respondents might impact group study spaces, study lounges, individual study spaces/ individual study carrels, and quiet study rooms. poor ventilation has great impact on students’ learning, and this was supported by haverinen-shaughnessy and shaughnessy (2015), who found that students did not perform well in a poorly ventilated environment. also, inadequate lighting in the library is not suitable for students and would affect students’ performance. the findings also imply that there are students seeking comfort while learning (mcdonald, 2011). hence, librarians and administrators should make libraries more comfortable for students (mohanty, 2002). the lack of installed security equipment (ranked 9th) can also be attributed to non-return of items by borrowers and theft of library materials (maidabino & zainab, 2011). thus, it is necessary to provide security equipment in the library to ensure longevity, availability, and effective provision of services to users.   table 3 revealed that among the top five challenges encountered by the mpspc students, three were reported that emerge from challenges encountered relative to library services, namely: 1) poor internet connection (52.4%, ranked 2nd), 2) inability to find documents or books needed (46.2%, ranked 3rd), and 3) lack of printing/ photocopying services (41.6%, ranked 5th). poor internet connection is quite noticeable because students frankly complain about the internet connection in the library. this shows that there are mpspc students who are internet users, and they surf the net since information is easily available (shrestha, 2008). thus, students prefer using the internet, as compared with printed materials, because it provides information readily at all times. the internet also gives faster access to information as well as offers a large amount of information (kumah, 2015). as mentioned by yebowaah (2018), the use of internet among students has a positive influence on their academic performance. also, the mpspc students ranked 3rd the inability to find documents or books needed (46.2%). this shows that students are not aware of how materials are organized in the library. this can be attributed either to students’ unfamiliarity with the services or how the materials are organized (hughes, 2010). lack of printing/photocopying services was also in the top five challenges encountered (41.6%, ranked 5th). this suggests the need for photocopying services to save time in taking down notes from books in the library. materials in the library often copied by students are of more rare materials that tend not to be available in book shops for sale. sriram and rajev (2014) mentioned that libraries must provide various services such as photocopying to enable users to utilize the library collections at greater potentials. on the other hand, the lack of staff’s kindness (21.8%, ranked 17th) was ranked last among the challenges encountered by the students, which shows that librarians are approachable and accommodating.   under human resources, lack of support staff (27%, ranked 11th) as one of the challenges encountered by students suggests the need for support staff. students do not just deal with librarians every time they visit the library, but also paraprofessionals serving them (guion, 2012). this implies that support staff have to undergo seminars on how to manage library patrons. the limited number of professional librarians (24.8%, ranked 12th) can be either attributed to a lack of professional librarian positions (with appropriate title, salary, and benefits) or a lack of licensed librarians. tanhueco-tumapon (2017) reiterated that librarians should be given an academic status (that is, like any teaching or research faculty member), wherein there is a corresponding increase in salary and therefore is due an academic rank provided they have a master’s degree. having an academic status in higher education leads them to be motivated in doing their functions as dignified librarians, since librarians and paraprofessionals may have different service standards.   for the library collection, lack of professional books and lack of e-resources were the challenges encountered by students. this implies that the collections of both books and e-resources were perceived to be insufficient. to address this, the library should build partnerships among other academic libraries to strengthen its collection (munro & philps, 2008) and increase its budget to purchase more collections.   in terms of financial resources, lack of support/budget is not enough to sustain library projects or programs (28.4%, ranked 10th), and very high library fees (23.6%, ranked 13th) were the perceived challenges encountered by the students. state colleges and universities in the philippines collect fewer library fees than in private schools. this may be why it is ranked almost at the bottom. although these challenges were at the bottom, the budget is essential in realizing library programs and projects, such as establishing or improving a library space. it could mean that increasing library fees would make students expect that the library can satisfy their needs and demands.    recommendations   the library should support the various learning activities of students, which include doing assignments, using reference books, searching/browsing printed materials, reviewing notes, writing, and others. it should design functional and flexible learning spaces tailored to the students’ ideal needs, such as their learning activities. thus, the study suggests strong recommendations to provide various learning spaces such as a makerspace, group study spaces, quiet study rooms, individual study spaces, computer workstations, interactive learning spaces, video viewing stations, and an internet café within the library premises to cater to the diverse students with various learning preferences and learning activities.   to continue building literature and knowledge in this area, it is recommended to conduct further research to include: 1) other areas such as policies, budgeting, and linkages; 2) categories of users such as faculty, alumni, and visitors; and 3) statistical tools such as using correlations, factor analysis, and others.   conclusion   a learning commons is a place to culture the mind wherein student learning encourages creativity, promotes social learning, enhances new information technology skills, and stimulates inquiry-based thinking. it is a space to nurture students’ minds for collaboration, learning, and interaction through a welcoming and supportive environment.   the mpspc students perform various educationally purposeful library activities. the activities among the students are generally engaging and support the library's mission. students vary in their needs of physical and virtual learning environments. both of these types of learning spaces are in demand among students, which are the key components of the learning commons. also, students specified the need for adequate learning spaces to support their various library learning activities. thus, the findings serve as the basis for crafting a project proposal to establish a learning commons tailored to mpspc students’ library activities and preferred learning spaces, with consideration for the challenges encountered by students, to support their learning and academic success.   author contributions   mrs. maryjul t. beneyat-dulagan: conceptualization (equal), data curation, formal analysis (lead), investigation (equal), writing – original draft (lead), writing – review & editing (equal) mr. david a. cabonero: conceptualization (equal), formal analysis (supporting), investigation (equal), visualization, writing – original draft (supporting), writing – review & editing (equal)   references   alawani, a. a., senteni, a., & singh, a. d. (2016). an investigation about the usage and impact of digital video for learning. in j. novotná & a. jančařík (eds.), proceedings of the 15th european conference on e-learning: ecel 2016 (pp. 1–9). academic conferences and publishing international limited.   arenson, m. (2013). the impact of a student-designed learning commons on student perceptions and use of the high school library. retrieved from https://www.semanticscholar.org/paper/the-impact-of-a-student-designed-learning-commons-arenson/83e55c93e23d913fca7565cdb210e40411c5019e   bailin, k. (2011). changes in academic library space: a case study at the university of new south wales. australian academic & research libraries, 42(4), 342–359. https://doi.org/10.1080/00048623.2011.10722245   barner, k. (2011). the library is a growing organism: ranganathan's fifth law of library science and the academic library in the digital era" (2011). library philosophy and practice (e-journal), 548. https://digitalcommons.unl.edu/libphilprac/548   barton, c. (2018). transforming an academic library to a learning commons model: strategies for success [doctoral dissertation, concordia university irvine]. cui digital repository. http://hdl.handle.net/11414/3385   bennett, s. (2003). libraries designed for learning. council on library and information resources. https://www.clir.org/wp-content/uploads/sites/6/pub122web.pdf   bieraugel, m., & neill, s. (2017). ascending bloom's pyramid: fostering student creativity and innovation in academic library spaces. college & research libraries, 78(1), 35–52. https://doi.org/10.5860/crl.78.1.35   blummer, b., & kenton, j. m. (2017). learning commons in academic libraries: discussing themes in the literature from 2001 to the present. new review of academic librarianship, 23(4), 329–352. https://doi.org/10.1080/13614533.2017.1366925   brown-sica, m., sobel, k., & rogers, e. (2010). participatory action research in learning commons design planning. new library world, 111(7/8), 302–319. https://doi.org/10.1108/03074801011059939 cabfilan, n. (2012). customers’ satisfaction on the circulation, reference, online and instruction services at benguet state university main library [master’s thesis, saint mary’s university (philippines)].   choy, f. c., & goh, s. n. (2016). a framework for planning academic library spaces. library management, 37(1/2), 13–28. https://doi.org/10.1108/lm-01-2016-0001   cicchetti, r. (2015). transitioning a high school library to a learning commons: avoiding the tragedy of the commons [doctoral dissertation, northeastern university]. northeastern university library digital repository service. https://doi.org/10.17760/d20193587   elkington, s., & bligh, b. (2019). future learning spaces: space, technology and pedagogy. advance he. https://telearn.archives-ouvertes.fr/hal-02266834   eurostat. (2016). classification of learning activities (cla): manual. https://doi.org/10.2785/874604   flaspohler, m. (2012). engaging first-year students in meaningful library research: a practical guide for teaching faculty. chandos.   freeman, g. t. (2005). the library as place: changes in learning patterns, collections, technology, and use. council on library and information resources. http://www.clir.org/pubs/reports/publ129/freeman.html   gstalder, s. h. (2017). understanding library space planning [doctoral dissertation, university of pennsylvania]. university of pennsylvania libraries scholarlycommons. https://repository.upenn.edu/dissertations/aai10289537/   guion, d. (2012, march 14). library staff: the paraprofessional. reading, writing, research. https://www.allpurposeguru.com/2012/03/library-staff-the-paraprofessional/   haverinen-shaughnessy, u., & shaughnessy, r. j. (2015). effects of classroom ventilation rate and temperature on students’ test scores. plos one 10(8), e0136165. https://doi.org/10.1371/journal.pone.0136165   head, a. j. (2016). planning and designing academic library learning spaces: expert perspectives of architects, librarians, and library consultants. project information literacy research institute. https://projectinfolit.org/publications/library-space-study/   holeton, r. (2020). toward inclusive learning spaces: physiological, cognitive, and cultural inclusion and the learning space rating system. https://er.educause.edu/articles/2020/2/toward-inclusive-learning-spaces   holland, b. (2015, january 14). 21st-century libraries: the learning commons. edutopia. https://www.edutopia.org/blog/21st-century-libraries-learning-commons-beth-holland   hughes, h. (2010). international students’ experiences of university libraries and librarians. australian academic & research libraries, 41. https://doi:10.1080/00048623.2010.10721446   iroaganachi, m. a., & ilogho, j. e. (2012). utilization of reference books by students: a case study of covenant university, nigeria. chinese librarianship: an international electronic journal, 34, 48–56. http://www.white-clouds.com/iclc/cliej/cl34ii.pdf   keating, s., & gabb, r. (2005). putting learning into the learning commons: a literature review. post-compulsory education centre, victoria university. https://vuir.vu.edu.au/id/eprint/94   khan, s. (2020). impact of learning spaces on student success. retrieved from https://www.edtechreview.in/trends-insights/insights/impact-of-learning-spaces-on-student-success/   king, j. g. (2016). extended and experimenting: library learning commons service strategy and sustainability. library management, 37(4/5), 265–274. https://doi.org/10.1108/lm-04-2016-0028   kumah, c. h. (2015). a comparative study of use of the library and the internet as sources of information by graduate students in the university of ghana. library philosophy and practice (e-journal). https://digitalcommons.unl.edu/libphilprac/1298/   lankes, r. d. (2016). expect more: demanding better libraries for today's complex world (2nd ed.). http://dx.doi.org/10.26153/tsw/13962   li, l. h., wu, f., & su, b. (2018). impacts of library space on learning satisfaction – an empirical study of university library design in guangzhou, china. the journal of academic librarianship, 44(6), 724–737. https://doi.org/10.1016/j.acalib.2018.10.003   maidabino, a. a., & zainab, a. n. (2011). collection security management at university libraries: assessment of its implementation status. malaysian journal of library & information science, 16(1), 15–33. https://arxiv.org/ftp/arxiv/papers/1301/1301.5385.pdf   martin, a. (2008). digital literacy and the “digital society.” in c. lankshear, & m. knobel (eds.), digital literacies: concepts, policies, and practices (pp. 151–176). peter lang.   matthews, k. e., andrews, v., & adams, p. (2011). social learning spaces and student engagement. higher education research and development, 30(2), 105–120. https://doi.org/10.1080/07294360.2010.512629   mccrary, q. d. (2017). small library research: using qualitative and user-oriented research to transform a traditional library into an information commons. evidence based library and information practice, 12(1), 34–49. https://doi.org/10.18438/b8863f   mccunn, l. j., & gifford, r. (2015). teachers’ reactions to learning commons in secondary schools. journal of library administration, 55(6), 435–458. https://doi.org/10.1080/01930826.2015.1054760   mcdonald, c. a. (2011). the library transformed into learning commons: a look at the library of the future [master’s thesis, university of central missouri]. james c. kirkpatrick library digital repository. https://ucmo.alma.exlibrisgroup.com/view/delivery/01ucmo_inst/1284617640005571   mcleod, s. (2015). “it’s not just about signing out books!”: from library to library learning commons: a catalyst for change [master’s thesis, university of victoria]. uvicspace. http://hdl.handle.net/1828/6315   mcmullen, s. (2008). us academic libraries: today's learning commons model (peb exchange 2008/04). organisation for economic co-operation and development. https://www.oecd.org/unitedstates/40051347.pdf   mihailidis, p., & diggs, v. (2010). from information reserve to media literacy learning commons: revisiting the 21st century library as the home for media literacy education. public library quarterly, 29(4), 279–292. https://doi.org/10.1080/01616846.2010.525389   mohanty, s. (2002). physical comfort in library study environments: observations in three undergraduate settings [master’s thesis, university of north carolina at chapel hill]. carolina digital repository. https://doi.org/10.17615/mne6-v039   munro, b., & philps, p. (2008). a collection of importance: the role of selection in academic libraries. australian academic & research libraries, 39(3), 149–170. http://doi.org/10.1080/00048623.2008.10721347   oblinger, d. g. (ed.). (2006). learning spaces. educause. https://www.educause.edu/research-and-publications/books/learning-spaces   paretta, l. t., & catalano, a. (2013). what students really do in the library: an observational study. the reference librarian, 54(2), 157–167. https://doi.org/10.1080/02763877.2013.755033   peterson, n. k. (2013). the developing role of the university library as a student learning center: implications to the interior spaces within [master’s thesis, iowa state university]. iowa state university digital repository. https://doi.org/10.31274/etd-180810-3678   pressley, l. (2017). charting a clear course: a state of the state of the learning commons. in d. m. mueller (ed.), at the helm: leading transformation: the proceedings of the acrl 2017 conference, march 22–25, 2017, baltimore, maryland (pp. 112–119). association of college and research libraries. https://www.ala.org/acrl/sites/ala.org.acrl/files/content/conferences/confsandpreconfs/2017/chartingaclearcourse.pdf   qayyum ch., a., hina, q. a., & abid, u. (2017). an empirical investigation of problems and issues being faced by the students while using the libraries in university of the punjab, lahore. bulletin of education and research, 39(2), 225–238. http://pu.edu.pk/images/journal/ier/pdf-files/17_39_2_17.pdf   rawal, j. (2014). libraries of the future: learning commons: a case study of a state university in california [master’s thesis, humboldt state university]. the california state university scholarworks. http://hdl.handle.net/10211.3/134872   roberts, r. l. (2007). the evolving landscape of the learning commons. library review, 56(9), 803–810. https://doi.org/10.1108/00242530710831257   shrestha, n. (2008). a study on student’s use of library resources and self-efficacy [master’s thesis, tribhuvan university]. e-lis. http://eprints.rclis.org/22623/   singh, s., & wadhwa, j. (2006). impact of computer workstation design on health of the users. journal of human ecology, 20(3), 165–170. https://doi.org/10.1080/09709274.2006.11905922   somerville, m. m., & harlan, s. (2008). from information commons to learning commons and learning spaces: an evolutionary context. in b. schader (ed.), learning commons: evolution and collaborative essentials (pp. 1–36). chandos. https://doi.org/10.1016/b978-1-84334-312-7.50001-1   spencer, m. e. (2007). the state-of-the-art: ncsu libraries learning commons. reference services review, 35(2), 310–321. http://dx.doi.org/10.1108/00907320710749218   sriram, b., & rajev, m. k. g. (2014). impact of academic library services on user satisfaction: case study of sur university college, sultanate of oman. desidoc journal of library & information technology, 34(2), 140–146. https://publications.drdo.gov.in/ojs/index.php/djlit/article/view/4499   stripling, b. (2008). inquiry: inquiring minds want to know. school library media activities monthly, 25(1), 50–52. https://www.teachingbooks.net/content/inquiringmindswanttoknow-stripling.pdf   suarez, d. (2007). what students do when they study in the library: using ethnographic methods to observe student behavior. electronic journal of academic and special librarianship, 8(3). https://digitalcommons.unl.edu/ejasljournal/83/   tanhueco-tumapon, t. (2017, august 18). 21st-century academic libraries. the manila times. https://www.manilatimes.net/2017/08/18/opinion/analysis/21st-century-academic-libraries/345157/   turner, a., welch, b., & reynolds, s. (2013). learning spaces in academic libraries – a review of the evolving trends. australian academic and research libraries, 44(4), 226–234. https://doi.org/10.1080/00048623.2013.857383   vercellotti, m. l. (2018). do interactive learning spaces increase student achievement? a comparison of classroom context. active learning in higher education, 19(3), 197–210. https://doi.org/10.1177/1469787417735606   watstein, s. b., & mitchell, e. (2006). do libraries matter? reference services review, 34(2), 181–184. https://doi.org/10.1108/00907320610669416   waxman, l., clemons, s., banning, j., & mckelfresh, d. (2007). the library as place: providing students with opportunities for socialization, relaxation, and restoration. new library world, 108(9/10), 424–434. https://doi.org/10.1108/03074800710823953   yebowaah, f. a., & plockey, f. d. d. (2017). awareness and use of electronic resources in university libraries: a case study of university for development studies library. library philosophy and practice (e-journal). https://digitalcommons.unl.edu/libphilprac/1562/   yebowaah, f. a. (2018).  internet use and its effect on senior high school students in wa municipality of ghana. library philosophy and practice (e-journal). https://digitalcommons.unl.edu/libphilprac/1817   appendix research instrument   dear respondents,   a pleasant day!   the undersigned is presently engaged in gathering data for her research entitled “exploring library activities, learning spaces, and challenges encountered towards the establishment of a learning commons” as a requirement for the degree master in library and information science.   in line with this, the researcher earnestly requests you to be one of the respondents of the research study. the researcher assures that your answers will be dealt with utmost confidentiality.   thank you and god bless!   sincerely yours,   researchers     name (optional): ____________________ course/year: _______________________     1.      the following are the library activities performed by students in the library. put a check mark (√) to all that applies to you. doing assignments   eating while reading/writing   interacting with librarians/ getting help from staff members   listening to music while studying/reading/writing   reading (periodical/fiction books/non-fiction books)   searching/ browsing printed materials   sitting comfortably while reflecting   studying alone on my own books/materials   studying in a group   surfing the web   using computer   using electronic resources   using reference books   writing (research works)   reviewing notes   others (pls. specify)         2.      which of the following is your favorite place to study or learn at the library? [you may check (√) one or more].     2.1. physical environment group study space a space where you can talk with friends while studying   individual study space (individual study carrels) a cubicle, stall, enclosed area for individual to read and study   makerspace -a space where you can create hands-on projects in groups or individually   quiet study room a private, very quiet workspace   research study room -a room assigned for individual for research and other scholarly activities that requires extensive use of library materials   study lounge an area open for students for gathering, studying and relaxing                                   2.2 virtual environment computer workstation an area consist of computer that is connected to a network for individual use   internet café an area where there is convenient access to coffee that offers internet access on its own computers or desktops   interactive learning space -a space provided for individual or group user/s for school work that needs computer technology   video viewing station -an area that is highly equipped with computer for watching specifically for educational purposes       3.      put a check mark (√) on the challenges you encountered in the library. [you may check one or more]. human resources limited number of professional librarian   lack of support staff   others (pls. specify)     physical facilities/ resources poor lighting facility in the designated reading areas   poor ventilation   lack of toilet facilities   lack of installed security equipment   insufficient learning spaces for various activities   uncomfortable furniture   lack of reading area/ reading area is not enough   others (pls. specify)     financial resources very high library fee   lack of support /budget is not enough to sustain library projects or programs   others (pls. specify)     library collection lack of professional books   lack of e-resources   others (pls. specify)     services inability to find documents/ books needed   lack of staff’s kindness   lack of printing or photocopying services   poor internet connection   others (pls. specify)       evidence summary   interviews with library directors suggest that political capital is linked to reputation   a review of: o'bryan, c. r. (2018). the influence of political capital on academic library leadership. library leadership & management, 34(4). retrieved from https://journals.tdl.org/llm/index.php/llm/article/view/7292     reviewed by: laura costello virtual reference librarian rutgers university libraries rutgers, the state university of new jersey new brunswick, new jersey, united states of america email: laura.costello@rutgers.edu   received: 30 nov. 2018                                                                 accepted: 16 jan. 2019      2019 costello. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29535     abstract   objective – to understand how library directors use political capital to overcome challenges and reach goals in their libraries. the author defines political capital as social power that is amassed through reputation and alliance building. this social power can be used to influence decisions and change at an organizational level.   design – narrative interview.   setting – a large state university system in the northeastern united states of america. the system includes a network of 64 independent campuses serving different communities with a total population of 460,000 undergraduate and 420,000 graduate students.   subjects – 12 library directors from within a single state university system.   methods – the author conducted in-depth narrative interviews with participants focusing on critical incidents throughout their careers and recent events. the author used restorying, reorganizing the data into chronological order before coding, and thematic analysis, using a software program to code the data and then revisit all the data with finalized codes to make any adjustments.   main results – several themes emerged in the interview data including interactions with administration, methods for building political capital, applying and using political capital, and building reputation. within the interactions with administration theme, the author observed a strong connection in the hierarchy of the institution. directors expected a high level of engagement and support from their direct reports and felt that providing this type of work to university administration would provide a return on investment for the library in terms of budget and support for new efforts or hr challenges. the theme of administrative turnover emerged as a possible barrier to establishing this relationship. in terms of building political capital, most participants did not set out to do this purposefully but instead sought to develop a reputation as a "team player" willing to participate in campus-wide initiatives and who would return positive outcomes. participants expressed that it was difficult to know how much political capital they had acquired until they attempted to use it towards a goal. eight of the participants provided narratives that included applying and using political capital, with two reporting that their political capital diminished after they had applied it towards a goal. other participants suggested that applying political capital increased their store when it was spent toward accomplishing higher-profile goals. the importance of communication was clear in the building reputation theme, several participants indicated that their communication skills helped establish a reputation for competence and credibility in interactions both up and down the chain of command. communication was a key factor in developing relationships across the institution, particularly with high-level administrators, and developing relationships was another area of importance for participants.   two of the participants indicated that they had and used political capital in specific areas and for smaller, day-to-day changes. eight participants used their political capital for bigger initiatives, such as budget, human resources, and library space.   conclusion – while a few of the directors explicitly linked their activities to political capital and felt that applying their political capital increased their standing with stakeholders, most participants did not generally link the development of political capital to individual events. instead, they suggested that generally establishing reputation and trust through excellent communication and relationship building would help them achieve success toward their goals.   commentary   positioning the academic library in the political structure of the university is an area of interest in recent research. while this article focuses on the individual political capital of library directors, adam murray and ashley ireland (2018) surveyed university provosts with a similar focus on effective communication as a strategy for accomplishing goals in the library. a preprint in the new review of academic librarianship by john cox (2018) also explores communicating value within the university, with a focus on framing library work through university goals. while these articles take different perspectives from the topic article, all three describe securing power through relationship building and effective value communication between university and library administration.   when examining the article through the glynn critical appraisal tool (2006), population is an area of interest. the author used purposive sampling within a limited community of library directors in one state university system. that system encompasses a wide spectrum of academic libraries, from community colleges to large research universities, and information about which type of library the participant directors represented was withheld to protect confidentiality. there is almost certainly a difference in administrative structures and strategy between large university centers represented in the system and much smaller, specialized colleges. the author described a split between participants who spoke explicitly about political capital and were intentional about acquiring and using it while others indicated they did not view this process as a political action. this split particularly could have benefitted from some exploration of the administrative structures that these directors were experiencing. the author suggests that random sampling should be done in future studies of this topic, but comparing the way library directors perceive political capital between different academic libraries of different types, sizes, and funding structures also represents a gap in the literature. participant information was controlled to protect the confidentiality of the sample, but it would have been illuminating to show some connection to participant demographics, since compelling arguments have been made by researchers such as barbara arneil (2006) that our understanding of political and social capital frequently leaves out diverse perspectives.   this research is relevant to academic libraries redefining their role within the university, but it is unclear whether this line of research can be generalized. the political structures of academic libraries are defined by individual factors like budget, hierarchical structure, and institution size. this is an important area of investigation, and it is clear from this research that it is an area of interest for some library directors, but further studies are needed before we fully understand the role of political capital in the landscape of academic libraries.   references   arneil, b. (2006). diverse communities: the problem with social capital. cambridge university press.   cox, j. (2018). positioning the academic library within the institution: a literature review. new review of academic librarianship. advance online publication. https://doi.org/10.1080/13614533.2018.1466342   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154   murray, a., & ireland, a. (2018). provosts’ perceptions of academic library value and preferences for communication: a national study. college & research libraries, 79(3), 336-365. https://doi.org/10.5860/crl.79.3.336     microsoft word es_hall.doc evidence based library and information practice 2007, 2:3  94 evidence based library and information practice       evidence summary    econometric analysis suggests possible crowding out of public libraries by book  superstores among middle income families in the 1990s    a review of:  hemmeter, jeffrey a. ʺhousehold use of public libraries and large bookstores.” library &  information science research 28.4 (sept. 2006): 595–616.    reviewed by:   stephanie hall  chief librarian, powell river public library  powell river, british columbia, canada  e‐mail: shall@powellriverlibrary.ca        received: 31 may 2007     accepted: 14 august 2007      © 2007 hall. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to determine the effect of large  bookstores (defined as those having 20 or  more employees) on household library use.    design – econometric analysis using cross‐ sectional data sets.    setting – the united states of america.    subjects – people in over 55,000  households across the u.s.a.    methods – data from three 1996 studies  were examined using logit and multinomial  logit estimation procedures: the national  center for education statistics’ national  household education survey (nhes) and  public library survey (pls),  and the u.s.  census bureau’s county business patterns  (cbp). the county level results of the nhes  telephone survey were merged with the  county level data from the pls and the cbp.   additionally, data on internet use at the  state level from the statistical abstract of the  united states were incorporated into the data  set. a logit regression model was used to  estimate probability of library use based on  several independent variables, evaluated at  the mean.     main results – in general, hemmeter found  that ʺwith regard to the impact of large  bookstores on household library use, large  bookstores do not appear to have an effect  on overall library use among the general  population” (613).    while no significant changes in general  library use were found among high and low  income households where more large  mailto:shall@powellriverlibrary.ca http://creativecommons.org/licenses/by/2.0 http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:3  95 bookstores were present, nor in the  population taken as a whole, middle income  households (between $25,000 and $50,000 in  annual income) showed notable declines in  library use in these situations. these effects  were strongest in the areas of borrowing  (200% less likely) and recreational purposes  (161%), but were also present in work‐ related use and job searching. hemmeter  also writes that “poorer households use the  library more often for job search purposes.  the probability of library use for recreation,  work, and consumer information increases  as income increases. this effect diminishes  as households get richer” (611). finally,  home ownership was also correlated with  higher library use.    households with children were more than  20% more likely to use the library (610).  their use of the library for school‐related  purposes, general borrowing, program  activities, and so on was not affected by the  presence of book superstores. white families  with children were somewhat less likely to  use the library, while families with higher  earning and education levels were more  likely to use the library. library use also  increased with the number of children in the  family.    shorter distances to the nearest branch and a  higher proportion of av materials were also  predictive of higher library use. educational  level was another important factor, with  those having less than high school  completion being significantly less likely to  use the library than those with higher levels  of educational attainment.    conclusion – the notable decline in public  library use among middle income  households where more large bookstores are  present is seen as an important threat to  libraries, as it may result in a decline in  general support and support for funding  among an important voting block. more  current data are needed in this area. in  addition to the type of information  examined in this study, the author  recommends the inclusion of information on  funding, support for library referenda, and  library quality as they relate to the presence  of large bookstores.     commentary    although this article may present heavy  going for the non‐statistician, the author’s  detailed examination of such a massive data  set yields some interesting observations.   unfortunately, since, as the author  acknowledges, the data were a decade old at  the time of this paper’s publication, its  relevance is much diminished. undoubtedly,  the importance of the topic justifies an  updated version of the research. in addition  to the inclusion of new types of information  proposed by the author, some consideration  should also be given to other recent research  and commentary on the topic (for example:  cartwright 2003; mckechnie et al. 2004,  among many others).     the article is also of potential interest as a  window into the mind of an economist  looking at public libraries. hemmeter starts  from assumptions such as the following:  “public services are often provided because  of a failure of the private market” (596), and  is surprised that households with higher  levels of education show higher levels of  library use, since they would experience a  “lower return” on such use (613). although  sometimes uncomfortable, seeing libraries  through an outside perspective can yield  new insights, and as the author struggled to  account for why better educated and  wealthier households would use libraries, it  became apparent that he might in fact be  unaware of the possibility that libraries may  have played a role in these attainments.  clearly credible evidence in this arena must  be compiled and communicated to policy  informants such as hemmeter, in addition  to the general public.  evidence based library and information practice 2007, 2:3  96 works cited    cartwright, helen. “change in store?”  library & information update 2.7 (jul.  2003): 52‐5.    mckechnie, lynne, p.k. french, george r.  goodall, margaret kipp, darian lajoie  paquette, and jennifer l. pecoskie.  “covered beverages now allowed:  public libraries and book  superstores.” canadian journal of  information & library sciences 28.3  (sept. 2004): 39‐51.    librarian-lead faculty learning communities offer opportunities for collaboration evidence summary   librarian-lead faculty learning communities offer opportunities for collaboration   a review of: burress, t., mann, e., & neville, t. (2020). exploring data literacy via a librarian-faculty learning community: a case study. journal of academic librarianship, 46(1).  https://doi.org/10.1016/j.acalib.2019.102076   reviewed by: jennifer kaari librarian east orange public library east orange, new jersey, united states of america email: jkaari@eopl.org     received: 2 mar. 2022                                                                    accepted:  20 apr. 2022      2022 kaari. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30115     abstract   objective – to describe a librarian-lead faculty learning community (flc) focused on data literacy.   design – case study.   setting – a public university in florida.   subjects – 10 participants in the flc.   methods – two librarians proposed the data literacy across the curriculum flc as part of the university of south florida st. petersburg center for innovation in teaching and learning. participants were recruited from all full-time instructional faculty. the group met for monthly 90-minute meetings throughout the fall and spring semesters. meetings were focused on group goal-setting, lightning talks, open discussion, data tool demonstrations, and the planning and development of work projects. in addition, the group designed an informal survey on the use of data tools across the institution.   main results – at the conclusion of the year-long flc, the group developed a frame for data literacy competencies that can be utilized across the curriculum. the flc participants created a data literacy faculty toolkit that presented that theoretical framework, as well as providing sample activities and other resources to help faculty to practically implement that framework into their instruction. the student success librarian also integrated data literacy into the first-year student information literacy curriculum.   conclusion – participation and facilitation of the flc by librarians served to further librarian-faculty collaboration, as well as demonstrating library value. the work of the data literacy across the curriculum flc raised awareness about information and data literacy on campus, and provided support to faculty members looking to further integrate data literacy into their instruction.   commentary   increasing collaboration with faculty members and improving librarian input into the curriculum are common challenges of academic librarians, as evidenced by the extensive literature on the subject (atkinson, 2018). participation in faculty learning communities (flcs) represents a promising opportunity to address some of these challenges. however, as the authors of this study note, there is a large gap in the literature regarding librarian participation in flcs. this case study offers a welcome addition to the literature, particularly as a case study of not only librarian participation in an flc, but a librarian-led flc.   this study was robust when appraised using the critical appraisal of a case study checklist (center for evidence-based management, 2014). a case study is the ideal format for presenting research like this study, given that the sample size and scope is limited by nature. the methodology was well-described and the authors thoroughly highlighted both the challenges and successes of their flc. in particular, they noted the challenge of balancing the need to communicate and work together effectively within a limited amount of time with the benefits of informal conversation and team-building to the productivity of the group.   one of the most notable aspects of the authors’ project was that they were not only participants in the flc, but they also proposed and facilitated the flc themselves. this was a good illustration of how taking initiative and being a proactive member of the institutional community is an essential part of increasing collaboration. in addition, they also chose as their subject a topic where they lacked subject-matter expertise. this allowed for the faculty and librarians to learn together, providing an opportunity for collaborative learning as well as collaborative work.   this study would be of particular interest to academic librarians who are interested in participating in or leading an flc, as well as any librarian interested in facilitating increased collaboration with colleagues outside of the library. academic librarians interested in data literacy at their own institutions would find the study to be a good overview of the subject as well as a potential model for building support for data literacy at their own institutions. it also opened up the possibility for further research and deeper analysis on academic librarian participation in flcs and collaborative learning across the institution. in particular, follow-up research into how this participation and the resulting work products influenced faculty and student success would be valuable.   references   atkinson, j. (2018). collaboration and academic libraries: an overview and literature review. in collaboration and the academic library (pp. 11-33). elsevier. https://doi.org/10.1016/b978-0-08-102084-5.00002-x   center for evidence-based management. (2014). critical appraisal checklist for a case study. https://cebma.org/wp-content/uploads/critical-appraisal-questions-for-a-case-study-july-2014-1.pdf evidence based library and information practice evidence based library and information practice 2009, 4.4  evidence based library and information practice       evidence summary    more research needed on librarian teaching anxiety    a review of: davis, kaetrena d. “the academic librarian as instructor: a study of teacher  anxiety.” college & undergraduate libraries 14.2 (2007):77‐101.    reviewed by:  stephanie j. schulte  education and reference services coordinator, assistant professor  prior health sciences library, the ohio state university  columbus, oh, united states of america  email: schulte.109@osu.edu     received: 18 september 2009        accepted: 14 october 2009      © 2009 schulte. this is an open access article distributed under the terms of the creative commons attribution   license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and   reproduction in any medium, provided the original work is properly cited.    objective – to identify the types of librarian  teaching anxiety and the coping mechanisms  that often accompany it and to compare those  findings with those described by showalter in  “teaching literature”; also, to examine  whether perceptions of librarians from both  inside and outside the profession influence  teaching anxiety.    design – a 35‐item online questionnaire  created using zoomerang; a link to the  questionnaire was distributed through the  information literacy instruction listserv     (ili‐l).     subjects – subscribers to ili‐l. there were  approximately 3,700 subscribers to ili‐l at the  time of the study. this electronic mailing list is  sponsored by the instruction section of the  association of college and research libraries  and is moderated.     methods – as previously mentioned, a link to  the questionnaire was distributed via the ili‐ l. requests for participation were sent to the  list three times during the six weeks the  survey was open for responses. the  questionnaire consisted primarily of multiple  choice questions, several with the option to  enter a free text “other” response, as well as  four likert‐type questions. after the survey  closed, the collected data was analyzed using  spss. the article did not indicate when the  survey was completed.     main results – 687 responses were collected.  of those, 657 were completed. surveys were  assessed for accuracy, during which 305  responses were eliminated, resulting in 382  “viable” responses (84). accuracy assessments  consisted of throwing out surveys in which  respondents answered questions  inappropriately, however, an explanation of  what constituted an inappropriate response is  not included.   74 mailto:schulte.109@osu.edu evidence based library and information practice 2009, 4.4  nearly three quarters of respondents (74%)  indicated they enjoyed teaching. this trend  did not appear to be related to the number of  years of experience as a librarian. the majority  of respondents (58%) had never taught full  semester or quarter courses, whereas  “virtually all” (86)  had taught one‐shot  instructional sessions. sixty‐three percent of  respondents noted being nervous prior to  teaching. although 40% of respondents noted  having no physical symptoms of anxiety, of  those who did, the main symptoms included  sweating and upset stomach. sixty‐five  percent of respondents noted experiencing  mental or emotional symptoms, mainly  identified as worries about being sufficiently  prepared and answering tough questions  (40%) and fear of public speaking (27%). these  mental and emotional symptoms were noted  to occur often in the case of 29% of  respondents, and at least some of the time in  41% of respondents. nearly three quarters of  the respondents reported using personal  strategies for dealing with teaching anxiety,  including over‐preparation, joining groups  where they were able to practice public  speaking, and prayer. most (84%) did not have  routines or rituals that they followed prior to  teaching.     some additional findings were presented  regarding librarians’ perceptions of  themselves as well as perceptions of librarians  by other faculty. eighty‐four percent of  respondents agreed or somewhat agreed that  there are many differences in the roles and  duties of librarians and paraprofessionals,  while 78% agreed or somewhat agreed that  faculty do not understand the librarian’s  teaching role. thirty‐five percent noted  defending teaching roles to other librarians.     conclusion – the role of librarians in  academic institutions continues to evolve and  include more teaching. as an increasing  number of librarians regularly teach and move  to teaching semester‐long credit courses, the  subject of teaching anxiety will continue to  grow in importance. this small study draws  attention to the need for more research in this  area.  commentary    little research exists that specifically addresses  teaching anxiety in librarians. given the  shifting role of academic librarians toward  more teaching, whether course‐integrated or  in stand‐alone courses, it is a topic worthy of  attention. while davis’ study attempted to  provide some insight into librarian teaching  anxiety, it fell short in many regards. that  being said, the discourse in the introductory  sections of the article about defining a  librarian and how that definition and  evolution of instruction relates to teaching is  thought provoking by itself, although a bit  dated.     the problems with this study began in the  construction of the questionnaire itself, which  was included as an appendix to the article.  the online survey failed to automatically skip  certain questions based on the participant’s  response. the manuscript indicates that many  respondents did not do this on their own, and  as a result, nearly half of the responses  obtained (305) were not included in the final  analysis. this elimination resulted in a low  response rate (approximately 10%) based on  the number of subscribers to the electronic  mailing list and brings into question whether  any of the results are meaningful.  additionally, there is no mention of validating  the survey or vetting it amongst colleagues for  clarity.     question design and wording are keys to  obtaining the best possible results in this type  of research; davis’ study had many problems  in this regard. in the career choice and  perception section of the survey, respondents  were asked about their level of agreement  with certain statements related to  librarianship. three of these used negative  language (“faculty don’t understand my role,”  “leave librarianship,” and “librarian  stereotypes bother me”). these questions were  immediately followed by one asking if the  respondent had ever defended their teaching  role. one must wonder if more neutral  language would have produced different  results. there are several typographical errors  76 evidence based library and information practice 2009, 4.4  77 in this section’s table as well, including the  misspelling of “agree” and using “somewhat  agree” twice, once where it should have said  “somewhat disagreed.”     another problem with the study is speculation  on the part of the researcher about how  librarians perceive their duties. the author  asserts that librarians know what they do  professionally and that their duties are clearly  outlined from a question that asked  respondents to rate their level of agreement  with the statement, “there are a lot of  differences in the roles/duties of the library  professional and paraprofessional.”  additionally, all of the data in this study are  reported in percentages. the author attempts  to draw parallels between perceptions, years  in the profession, and teaching anxiety, yet  there is no data that indicates whether any of  the relationships were statistically significant.  the use of percentages is somewhat confusing  and without statistical analysis, difficult to  evaluate. the author also failed to explain how  she analyzed questions with free text  responses in the “other” category. all  together, these issues result in some mistrust  of the results on the whole.      though the limitations and problems noted  above prevent the information from being  generalized, this study does highlight the fact  that more research is needed to examine the  causes and types of teaching anxiety in  librarians, research that could lead to  structured help in coping with this anxiety.  the study also draws attention to the need for  librarians to be cautious and seek mentorship  when designing questionnaires of any kind, so  that the data gathered from them will be more  meaningful.      works cited    showalter, elaine. teaching literature.  malden, ma: blackwell, 2003.  charting the future of the ginans: needs and expectations of the ismaili youth in the western diaspora research article   charting the future of the ginans: needs and expectations of the ismaili youth in the western diaspora   karim tharani information technology librarian university of saskatchewan library saskatoon, saskatchewan, canada email: karim.tharani@usask.ca   received: 8 oct. 2021                                                                 accepted: 23 feb. 2022      2022 tharani. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30055     abstract   objective – the heritage of ginans of the nizari ismaili community comprises hymn-like poems in various indic dialects that were transmitted orally. despite originating in the indian subcontinent, the ginans continue to be cherished by the community in the western diaspora. as part of a study at the university of saskatchewan, an online survey of the ismaili community was conducted in 2020 to gather sentiments toward the ginans in the western diaspora. this article presents the results of the survey to explore the future of the ginans from the perspective of the english-speaking ismaili community members.   methods – an online survey was developed to solicit the needs of the global ismaili community using convenience sampling. the survey attracted 515 participants from over 20 countries around the world. the english-speaking members of the ismaili community between 18 to 44 years of age living in western countries were designated as the target group for this study. the survey responses of the target group (n = 71) were then benchmarked against all other respondents categorized as the general group (n = 444).     results – overall, 85% of the respondents of the survey were from the diaspora and 15% were from the countries of south asia including india, pakistan, and bangladesh. the survey found that 97% of the target group respondents preferred english materials for learning and understanding the ginans compared to 91% in the general group. having access to online ginan materials was expressed as a dire need by respondents in the two groups. the survey also revealed that over 90% of the respondents preferred to access private and external ginan websites rather than the official community institutional websites. in addition, the survey validated the unified expectations of the community to see ginans become an educational and scholarly priority of its institutions.   conclusion – based on the survey results, it can be concluded that the respondents in the target group are educated citizens of english-speaking countries and regard the heritage of ginans to be an important part of their lives. they value the emotive and performative aspects of the tradition that help them express their devotion and solidarity to the ismaili faith and community. they remain highly concerned about the future of the ginans and fear that the teachings of the ginans may be lost due to lack of attention and action by the community institutions. the development and dissemination of curriculum-based educational programs and resources for the ginans emerged as the most urgent and unmet expectation among the survey respondents. the article also identifies actions that the community institutions can take to ensure continued transmission and preservation of the ginans in the western diaspora.     introduction   the word “ginan” is a derivative of the sanskrit term jnan, which means knowledge or gnosis. in the context of the nizari ismaili community, the term is used for the community’s collection of gnostic hymn-like poems. the religious corpus of the ginans comprises some 1,000 individual works composed primarily using indo-aryan dialects with loanwords from perso-arabic languages. while the ginans originated in the indian subcontinent, the ismaili community is now a global and culturally diverse community living in over 25 countries around the world (the ismaili, 2022).   the emotive tunes of the ginans continue to be cherished by the community members in the western diaspora, particularly those who come from the indian subcontinent lineage commonly known as khojas. due to the language barrier, however, the teachings of the ginans remain inaccessible to the english-speaking community members born and raised in the western diaspora. this issue is further compounded by the inaction of the community institutions to develop and disseminate educational resources for the motivated community youth interested in learning about the teachings of the ginans. failing to attend to the needs and expectations of the youth – who are ultimately responsible for carrying these traditions forward – may result in the loss of the tradition altogether.   to address this lacuna and explore the future of the ginans in the digital age, an online survey was conducted in 2020 to gather and analyze the needs and expectations of the ismaili community members interested in the ginans from across the globe. the survey was administered independently at the university of saskatchewan, which currently hosts the searchable online ginan portal called ginan central. following a brief review of the history of the community and the corpus of ginans, this article presents the results of the survey. the insights from the survey results inform the discussion on the future of the ginans in the western diaspora.   literature review   today’s digital age presents unprecedented opportunities for ethnocultural communities to teach and transmit their knowledge in ways that were not possible in the print era. with information technology becoming an essential enabler for learners in western countries, this study is based on the premise that the successive generations of the ismaili community expect to engage with the ginans online and on-demand. thus, the use of information technology in conjunction with traditional teaching can enhance motivation and engagement of the community youth, which in turn can ensure the long-term viability of the ginans in the western diaspora.   this research utilizes the e-learning theoretical framework which identifies three theoretical dimensions of an effective e-learning system (aparicio et al., 2016). these dimensions include people, technology, and services. the people dimension defines various roles that stakeholders may have in an e-learning system, such as learners, content providers, educators, etc. the technology dimension of an e-learning system serves as an interface to communicate and connect users with the content curated for learning activities. the services dimension of an e-learning system encapsulates the pedagogical models and instructional strategies that guide the design and development of the e-learning system.   in the context of this study of the ismaili community and its tradition of the ginans, the e-learning system framework is applied by identifying community learners (representing the people dimension) whose needs and expectations to engage with the ginans online (i.e., the technology dimension) must be gathered and analyzed to develop effective curriculum and instruction (manifesting the services dimension). this study assumes that understanding the needs and expectations of the community and its youth remains crucial for ensuring that the tradition and its teachings continue to be passed on from generation to generation in the west. thus, the initial focus of the study was to gather and analyze the needs of the ismaili community as depicted in the figure below (figure 1). figure 1 e-learning system framework and the ginans.   the ismaili community of the indian subcontinent   as mentioned earlier, the present-day nizari ismaili community members were historically referred to as the khojas in the pre-colonial indian subcontinent. the religious path of the khojas was known as satpanth (true path) which subscribed to the single spiritual reality of humans irrespective of specific religion, race, or practice. the ginans were venerated as spiritual teachings by the followers of satpanth, the authorship of which is attributed to several preacher-saints who are known as pirs and sayyids in the community. the ginans were composed using a mixed language that borrowed vocabulary primarily from indo-aryan languages, including gujarati, hindi, sanskrit, among others (shackle & moir, 2000). the use of a mixed language enabled the composers to draw from the “bewildering thicket of indian religions, mythologies and intellectual traditions… the ginans thus became and remained, until the contemporary project to reconceptualize and reformulate the ismaili tariqah (tradition), the de facto supreme scripture for satpanth ismailis.” (alibhai, 2020, n.p.). the historical practice of composing ginans came to an end in the mid-nineteenth century and no new ginans have been composed since then (asani, 2011).   the subsequent colonization of india forced the satpanth followers to choose between the two dominant religious persuasions – hinduism and islam. as a result, the community’s identity evolved from satpanthi khojas to ismaili muslims. the end of the colonial rule in 1947 resulted in the partition of the indian subcontinent into two independent countries, india and pakistan. while the tradition of ginans is the heritage of the satpanthi ismailis, it continues to be cherished around the world, albeit as one of many of the community’s diverse devotional religio-cultural traditions, including qasida, munqabat, munajat, and geet. in the context of this study, the term community is used to refer to those members of the nizari ismaili community in the diaspora who continue to recognize and revere ginans as part of their religio-cultural heritage.   the origin and evolution of the corpus of ginans   the initial efforts to formalize and preserve the ginan corpus can be traced back to various ismaili individuals and entrepreneurs in the late 19th century. for instance, lalji devraj (1842-1930) of india is credited to have published the initial canon of authorized texts of ginans in khojki (asani, 2011). a few specimens of these historic publications from india have been catalogued and preserved at harvard library (asani, 1992). in the first quarter of the twentieth century, the responsibility to publish ginans was taken over by community institutions – starting in 1922 with the recreation club institute in india and followed by the ismaili tariqah and religious education board (itreb) – a network of community-led national and regional committees across the globe. the itreb remained responsible for publishing religious materials available to the community members, including the ginans.     in 1977, the institute of ismaili studies (iis) was established for the community by the aga khan to promote historical and contemporary study of muslim cultures and their relationship with other societies and faiths. over time, the mandate of developing curriculum and instruction for religious education was gradually assumed by the iis. as karim (2022) points out, an unfortunate consequence of this transition has been the lack of ginan publications:   [the iis] has produced over a hundred books, including five volumes on the primary materials in its collection relating to the arab and persian aspects of the movement. the institute has received hundreds of satpanth-related manuscripts from communal and family collections since the late 1970s; however, these sources have suffered from neglect and their cataloguing was still awaiting completion in 2021. harvard university published its catalogue in 1992. even though the endowment of the iis has been funded mainly by khojas, it has produced only three monographs on their tradition.   asani (2021) also observes that while the satpanth ismailis continue to revere ginans, the “hinduistic” elements of the ginans remain problematic for the institutions. this divergence of perspectives on the ginans between satpanth ismailis and certain ismaili institutional circles remains a barrier in making any significant headway in preserving the ginans through formalized teaching and transmission (asani, 2021):   anxieties about perceptions that other muslims may have of the ginans, in particular their vernacular indic character, have been the primary concerns to ismaili institutions. these concerns have led to a marked de-emphasis of the semantic dimension of the gināns in the contemporary articulation of official ismaili doctrine in favor of a quranic one. instead, there is an increased focus on the performative aspects of the gināns, and their ritualization as a form of ismaili “devotional literature,” thus reframing them within the context of ismaili literary traditions in arabic and persian (p. 50).   this methodical reformulation of the ginans from “satpanthi scripture” to “devotional literature” by the community institutions led to grave concern and anxiety about the future of ginans, particularly among the khoja ismailis. consequently, local community preachers and missionaries (referred to as al-waez in the community) took it upon themselves to preserve and propagate the scriptural status of the ginans. the efforts of kamaluddin ali muhammad and his wife zarina kamaluddin, notes virani (2015), have made ginans more accessible and comprehensible to those not familiar with gujarati and khojki scripts:   al-waʿiẓ kamaluddin ali muhammad and al-waʿiẓa zarina kamaluddin have made herculean efforts to study and translate this literature. their work has added tremendously to our knowledge of not only the ginans, but of medieval south asian verse in general, for the ginanic symbols and vocabulary draw on the rich universe of mystical meaning that had become the common inheritance of sufis, yogis, sadhus, bhaktas and sants. all students of this field and lovers of the ginans are indebted to their endeavours (p. viii).   in 2020, the iis established the south asian studies (sas) unit with the mandate to “contribute to current academic debates as they relate to islam and muslims in south asia and to further scholarly understanding of satpanth history, literature, heritage and identity to promote critical thinking in the field of south asian ismaili studies” (the institute of ismaili studies, 2018).  while this broad mandate falls short of mentioning the ginans, it is hoped that a primary responsibility of this unit will be the development and dissemination of ginan materials to meet the needs of community members, in particular the english-speaking community members now living in the western diaspora.   aims   the purpose of the online survey was to gather and analyze the needs and expectations of the community members who wish to learn and understand the ginans, and to pass on the tradition and its teachings from generation to generation. this survey was guided by the following research question: what are the needs and expectations of motivated english-speaking ismaili community learners to engage with the ginans in the western diaspora? the use of information technology in learning and engaging with the ginans was assumed to be an important consideration for the english-speaking learners born and raised in the west.   methods   an online survey was developed as part of this study to solicit the needs of the global ismaili community. the survey questionnaire contained a total of 20 questions, none of which were mandatory and allowed participants to skip over any of the questions. the survey questionnaire is available as an appendix (appendix a). the online survey was administered independently of any ismaili community institutions. this independence also necessitated recruiting community participants for the survey directly using a variety of communications channels popular in the community, including ginan central at the university of saskatchewan, ismaili.net heritage, jollygul, and ginanguru.   results   the online survey was administered between july 9, 2020, and september 10, 2020. it attracted 515 participants from over 20 countries around the world. given that the global population of the ismaili community is estimated to be approximately 12 to 15 million, the results of this survey are not generalizable to the entire global ismaili community (the ismaili, 2022). as noted in the table below (table 1), most of the respondents living in the west were from canada (46%), the united states of america (23%), and the united kingdom (4%). respondents from pakistan (8%) and india (5%) were the leading participants from south asia. overall, around 85% of the survey participants were from the diaspora and 15% were from the south asian countries.   table 1 survey respondents by country country no. of respondents percentage canada 237 46% united states of america 116 23% pakistan 43 8% india 26 5% united kingdom 21 4% other  72 14 % total 515 100%   demographic statistics   the age distribution of the respondents was grouped into seven intervals between 18 years and those over 75 years of age. with a 98% of response rate for the question on age, the highest number of respondents of the survey (24%) were in the age group of 55 to 64 years old (see figure 2).   figure 2 survey respondents by age groups.   the question on gender was answered by 505 respondents, of whom 290 identified as female (57%), 213 as male (42%), and two as neither male nor female (0.4%). the lead by female respondents was consistent across all age groups except in the group aged 75 years and over, which was predominantly male at 70% (see table 2).   table 2 survey respondents by gender age groups female male other total 18 to 24 years old 52% 45% 2% 100% 25 to 34 years old 52% 48% 0% 100% 35 to 44 years old 58% 42% 0% 100% 45 to 54 years old 64% 35% 1% 100% 55 to 64 years old 65% 35% 0% 100% 65 to 74 years old 56% 44% 0% 100% over 75 years old 30% 70% 0% 100%                  in terms of educational attainment, over 75% of the respondents claimed to have at least one degree and only 3% of the respondents had not completed a high school degree (see figure 3). further analysis of this data revealed that 89% of the respondents from south asia had at least one degree as opposed to 75% of respondents in the diaspora with at least one degree.   figure 3 survey respondents by highest level of education.   around 37% of the survey respondents identified themselves as employed professionals. in addition, 20% of the respondents identified as either business owners or self-employed. over one-quarter of the respondents (27%) identified as either retired or homemakers. a visual summary of the primary occupation of the survey respondents is presented in the chart below (see figure 4).   figure 4 survey respondents by primary occupation.   target population profile   as evident from the demography analysis, there was a wide range of diversity in terms of geography, age, gender, education, and occupation among the respondents. owing to this geographic and demographic spread of the survey participants, the survey respondents were divided into two groups. the survey target group consisted of respondents who belonged to the target population defined for this study – english-speaking ismaili community members aged between 18 and 44 years who currently reside in euro-american countries. in contrast, the general group comprised the respondents who fell outside the target population. the target group respondents were identified by combining the responses of four specific questions in the survey (q1 – country of residence, q2 – age group, q6 – community membership, and q10 – primary language).   as depicted in the chart below (figure 5), of the 515 total survey respondents, 496 respondents identified themselves as members of the ismaili community. a total of 376 of these respondents lived in euro-american countries, and 238 of them designated english as their primary language. finally, the pool of respondents in the target group was reduced to 71 when the age requirement was considered. thus, the final size of the survey target was determined to be 71 or around 14% of the total number of respondents (n = 515).   figure 5 identifying respondents in the survey target group.   gathering needs and expectations   it was helpful to benchmark the target group in relation to the other respondents in the general group to analyze the needs assessment survey data. doing so helped provide a consistent baseline in identifying and comparing, as well as contextualizing, unique needs and attitudes between the two groups. for instance, when analyzing the data for the question on the importance of ginans in their lives (q7), the expected difference in the attitudes between the two groups could now be visualized. a lower percentage of the target group (89%) attached moderate to high importance to ginans than the general group (97%), as depicted in the figure below (figure 6). the evidence also validates the fundamental premise of this study that ginans must be made relevant to the younger generations based on their current socio-economic contexts.   a strong agreement between the needs of the two groups was observed with regards to having access to ginan resources. an overwhelming majority (97%) in both groups attached moderate to high importance to having online access to ginan resources. on the question of preferred devices to access online ginan resources as well, there was notable synergy in the needs of the two groups. the use of mobile phones remained the most preferred device in the target group for 97% of the respondents in comparison to 91% for the general group. the two groups diverge in their preferences, however, when it comes to accessing ginan resources in the form of books, cds, cassettes, etc. more than one-quarter (28%) of the respondents in the general group attach moderate to high importance to analog resources in comparison to less than one-tenth of the respondents in the target group.   figure 6 importance of ginan in target and general respondents groups.   given the demography of the target group, it was expected to find an overwhelming demand (97%) for the latin script and english as the language of instruction for the ginans. it was surprising, however, to find the english language to be preferred by 91% of the respondents in the general group. while none of the other scripts come close to the strong support shown for english, khojki and gujarati remain alive and important in the community even today (figure 7). the findings of the survey reflect the historical decline of khojki and gujarati scripts in favour of english as a substantial number of the community members have moved away from the indian subcontinent to the west.     figure 7 language preference for learning and understanding ginans.   most of the respondents in both groups (89%) considered online access to ginan materials to be important. due to the lack of useful ginan content on the official community institutional websites, community members use various external websites to access ginan materials (figure 8). thus, there is a deep desire in the community to use online ginan websites and content that are either produced or endorsed by community institutions. as one community participant stated, the need for developing “a unified website which is accredited by iis that is made available globally and it should have authentic text and raags of ginans” (respondent #506).   figure 8 most preferred websites for accessing ginan materials.   to ascertain the types of online materials desired by ginan learners, a variety of options were presented to the respondents to rank as part of the needs assessment survey. these options were ranked independent of each other by the respondents. having access to ginan texts and translations in english was ranked as the most desirable resource for learning and understanding of the ginans (figure 9). this outcome is not surprising as it is a common practice to use ginan text and translation side-by-side during in-person instructional sessions.     figure 9 useful resources for learning and understanding ginans.   the second most desired content type was audio recitations of ginans. in an in-person instructional setting, the instructor is responsible for reciting ginans to the learners. in an online and self-learning setting, where there are no instructors, the availability of digital audio is crucial. the need for information on ginan categories was also ranked as desirable by 84% of the respondents. the ginan categories are used to group ginans based on various ceremonial and topical themes. in recent years, the knowledge of ginan categories has been confined to the community elders and experts who typically impart this knowledge to learners during their in-person instructional sessions.                another sought-after resource for learning and understanding ginans that was ranked considerably higher in the needs assessment survey was the summary of individual ginans, commonly referred to as ginan “gist” in the community. over the past decade, it has become a common practice for reciters to read out the gist of the ginan in english that they are called upon to recite during congregational services held for special occasions. the gist texts explain the message and sentiments of ginans in broad strokes for the english-speaking members of the congregation who often struggle to understand what is being recited. despite this being a common practice, the community institutions have yet to produce any publication with ginan gists that can be readily accessed by the community.   ginan dictionary and multimedia ginan videos were ranked equally high in the survey. when analyzed based on specific groups, only 67% of the respondents in the target group attach moderate to high importance to a ginan dictionary as opposed to 80% of those outside the target group. the need for multimedia videos is also relatively less pronounced in the target group at 77% as opposed to 81% in the general group.   figure 10 thematic summary of the needs and expectations of the survey respondents.   the qualitative analysis of the only open-ended question in the survey (question # 20) revealed three distinct themes (figure 10). overall, the survey found that the availability of ginan resources in english remains a crucial need for the target group members who have little or no knowledge of the language of the ginans. the development and dissemination of curriculum-based educational programs and supporting materials for the ginans emerged as the most urgent and unmet expectations. the sentiment for a more pronounced acknowledgement of the heritage of ginans and its significance in the communal and scholarly undertakings of its institutions and leaders was found to be equally prevalent amongst the target and general group respondents.   discussion   based on the analysis of the survey data, an aggregate profile of the needs and expectations of the target group – english-speaking ismaili community members aged between 18 and 44 years who currently reside in the west – can be depicted as follows. a typical target group member is an educated citizen of an english-speaking country such as canada. they consider ginans to be an important part of their life even though the language of ginans remains mostly foreign to them. they value the emotive and performative aspects of the tradition that help them express their devotion and solidarity to the ismaili faith and community. they remain very concerned that the ginans, and more so the teachings that the tradition encapsulates, will be lost if nothing is done about it by the community and its institutions.   the survey results provide tangible evidence of the need to utilize information technology for making ginans accessible. the survey revealed that close to 90% of the survey respondents preferred having access to ginan materials online, which challenges the taboo of incorporating contemporary information technology to complement traditional ways of accessing and teaching the ginans. from a learner-centric perspective, the use of information technology is imperative for engaging learners to embrace traditional languages and traditions.   there is little doubt that the present-day ginan corpus has survived over the past centuries primarily due to the foresight and adaptability of the ismaili community to embrace print technology to preserve the ginan texts. now it is time for the community to once again summon its spirit of adaptability and courage to embrace information technology to ensure the continuity of the ginans in the digital age. the future survival of the tradition of the ginans in the west remains highly dependent on the continued engagement of the community youth through information technology.   with its deliberate focus on the people dimension of the e-learning framework to gather community needs, this research opens pathways to expand on the technology and process dimensions of the e-learning system framework in charting the future of the ginans by the ismaili community and its institutions. the community institutions are well-positioned to address some of the fundamental issues to ensure continued transmission and preservation of the ginans in the western diaspora.   standardized romanization   the phonetic demands of the oral and mixed nature of the ginan language posed challenges for the limited phonetic strength of the khojki script initially used to transcribe the ginans (virani, 2017). unfortunately, these shortcomings were never addressed systematically and were passed on as the ginan corpus was canonized from khojki to a more established gujarati script. the canonized ginan corpus in khojki and gujarati were then used as the basis for the romanization of ginans into the latin or english writing system. as a result, variant ginan romanization conventions started to emerge from different countries where the community resided. for instance, the palatal or hard “d” sound as in the word doctor, is found to be romanized with variants such as the use of successive d (dd), capitalization (d), italicization (d), and with a dot below (ḍ). community institutions such as the national itrebs and the iis are well-placed to standardize romanization conventions for the transliteration of the ginans.   having a standardized romanization convention will not only make the ginan corpus more reader-friendly but will allow ginan texts to be computationally analyzed using natural language processing for various purposes, including creation of lexical resources. as bowker (2018) explains, availability of credible and representative corpus remains at the heart of enhancing human understanding through computational analysis in language learning:   consider that a corpus is a text file. it could be made up of tens, hundreds, or thousands of documents and may run to hundreds of thousands or even millions of words. trying to count the number of words, or the number of times each word occurs, would be a time-consuming, labor-intensive and error-prone process if it were done manually. however, this type of work is easily accomplished by a computer, and corpus analysis software can be used to calculate several different measures of frequency, including raw frequency counts (e.g., word lists), measures of disproportionate frequency (e.g., keyness), and measures of relative frequency (e.g., collocations) (p. 361).   ginan dictionary   another potential measure that the community institutions can take is to commission an english dictionary of the ginans. while several ginan resources feature back-of-the-book ginan glossaries with english meanings, an english dictionary of the ginans is yet to be developed and published. if ginans are to be understood by english-speaking youth of the community, the availability of a ginan dictionary must become a priority for community elders and leaders. given the high preference in the community for online access, it may be worthwhile to make such a dictionary available online.   when it comes to forgotten and endangered languages, a dictionary becomes a tool for language preservation (gippert et al., 2006). with the language of the ginans being oral, mixed, and endangered simultaneously, it must be preserved not just for the community but also for scholarly research. thus, a comprehensive ginan dictionary will make it possible to study rare and complex forms of linguistic expressions manifested in the vocabulary of the ginans.   ginan curriculum   there appears to be a deep desire in the community for its institutions to embrace a programmatic approach for the development and administration of curriculum-based in-person and online ginan classes. the expectation here is that the curriculum for these classes will not only teach the meanings and tunes but will ensure that the history and teachings of ginans are also made relevant to the community’s contemporary context as a diaspora community. at a deeper level, this need of the community youth is indicative of their desire to find comparable and compatible expressions of their faith and devotion in english-speaking societies. from this perspective, “translation” is no longer an exercise in finding linguistic equivalence but, as stewart (2006) notes, a quest to seek equivalence of one’s faith in the local culture (p. 286-87):   [t]he search for equivalence in the encounter of religions—when understood through the translation models we have characterized as literal, refractive, dynamic, and metaphoric—is an attempt to be understood, to make oneself understood in a language not always one’s own; it does not necessarily reflect religious capitulation or theological ignorance or serve as the sign of a weak religious identity…. the texts that reveal actors attempting to locate commensurate analogues within the language tradition capture a unique ‘moment’ in the process of cultural and religious encounter, as each tradition explores the other and tries to make itself understood (p. 286-7).   the ginans, like many other ethnocultural traditions and knowledge, remain under-studied as an academic area of research in western academia. from a broader perspective, the results of this study have wider relevance to other diasporic ethnocultural communities that may be facing similar challenges in imparting and safeguarding their traditions and knowledge.   conclusion   this article presented a brief historical overview of how the ismaili community members have managed to safeguard the ginans despite the geopolitical upheavals that the community has been subjected to as a political, ethnic, and religious minority community. it also unveiled the present divergent perspectives on the ginans between the community members of khoja descent and the community institutions which continue to exacerbate the anxiety about the future of ginans. finally, the responses of the global online survey were analyzed to identify the needs and expectations of the community to chart the future of the ginans in the western diaspora. the survey found that the availability of online ginan materials and resources are highly desirable by the english-speaking community members who want to learn and understand the teachings of the ginans. in addition, the survey also uncovered the community’s strong expectation to see the ginans become a priority in educational and scholarly programming and publishing initiatives of the community institutions.   acknowledgement   this article is an abridged version of the author’s unpublished doctoral dissertation, titled tradition and technology: a design-based prototype of an online ginan semantization tool, which was supervised by professor jay wilson at the university of saskatchewan.   references   alibhai, m. (2020, march 5). tajbibi abualy aziz (1926-2019) part one: a satpanthi sita. https://theolduvaireview.com/tajbibi-abualy-aziz/   aparicio, m., bacao, f., & oliveira, t. (2016). an e-learning theoretical framework. journal of educational technology & society, 19(1), 292-307. https://www.jstor.org/stable/jeductechsoci.19.1.292   asani, a. s. (1992). the harvard collection of ismaili literature in indic languages: a descriptive catalog and finding aid. boston: g.k. hall. https://iiif.lib.harvard.edu/manifests/view/drs:13785163$1i   asani, a. s. (2002). ecstasy and enlightenment: the ismaili devotional literature of south asia. london: i.b. tauris. https://www.iis.ac.uk/publication/ecstasy-and-enlightenment-ismaili-devotional-literature-south-asia   asani, a. s. (2011). from satpanthi to ismaili muslim: the articulation of ismaili khoja identity in south asia. in f. daftary (ed.), a modern history of the ismailis: continuity and change in a muslim community (pp. 95-128). london: i. b. tauris publishers. https://words.usask.ca/ginans/files/2020/05/2011-asani-khoja-identiity-in-modern-history-of-ismailis.pdf   asani, a. s. (2020, december 20).  towards a religiohistory of ginans [webinar]. the association for the study of ginans. https://www.youtube.com/watch?v=23khqgied24   bowker, l. (2018). corpus linguistics is not just for linguists: considering the potential of computer-based corpus methods for library and information science research. library hi tech, 36(2), 358-371. https://doi.org/10.1108/lht-12-2017-0271   gippert, j., himmelmann, n., & mosel, u. (eds.). (2006). essentials of language documentation. berlín: mouton de gruyter. https://genderi.org/pars_docs/refs/58/57633/57633.pdf   ivanow, w. a. (1948). collectanea (volume 1). leiden, holland: e. j. brill. http://www.ismaili.net/source/0723.html   karim, k.  h. (2022). khoja isma’ilis in canada and the united states. oxford research encyclopedia of religion. https://doi.org/10.1093/acrefore/9780199340378.013.841   kassam, t. r. (1995). songs of wisdom and circles of dance: hymns of the satpanth isma'ili muslim saint, pir shams. suny press. https://sunypress.edu/content/download/449846/5466323/version/1/file/9780791425916_imported2_excerpt.pdf   shackle, c., & moir, z. (2000). ismaili hymns from south asia: an introduction to the ginans. richmond, surrey: curzon press. https://books.google.ca/books?id=y8lo79jtyjuc   stewart, t. k. (2001). in search of equivalence: conceiving muslim-hindu encounter through translation theory. history of religions, 40(3), 260-287. https://www.jstor.org/stable/3176699   tennant, r. (2004). a bibliographic metadata infrastructure for the twenty‐first century. library hi tech, vol. 22 no. 2, pp. 175-181. https://doi.org/10.1108/07378830410524602   the institute of ismaili studies. (2018). south asian studies. https://www.iis.ac.uk/content/south-asian-studies   the ismaili. (2022). the ismaili community. https://the.ismaili/global/about-us/the-ismaili-community   virani, s. (2015). introduction. in ginans with english translation and glossary – volume 9 (pp. viii-xvi). pakistan. https://www.academia.edu/37220475/introduction_ginans_with_english_translation_and_glossary_volume_9   virani, s. (2017, october). garlands of sounds (varṇamālā): establishing a phonology of the khojki script. paper presented at before the printed word: texts, scribes and transmission a symposium on manuscripts collections housed at the institute of ismaili studies, london, uk.                https://www.iis.ac.uk/events/printed-word-texts-scribes-and-transmission   appendix a online survey questionnaire             research article   informing evidence based decisions: usage statistics for online journal databases   alexei botchkarev senior information management advisor, ministry of health and long-term care adjunct professor, ryerson university toronto, ontario, canada email: alexei.botchkarev@ontario.ca   received: 9 july 2016      accepted: 20 feb. 2017        2017 botchkarev. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – the primary objective was to examine online journal database usage statistics for a provincial ministry of health in the context of evidence based decision-making. in addition, the study highlights implementation of the journal access centre (jac) that is housed and powered by the ontario ministry of health and long-term care (mohltc) to inform health systems policy-making.   methods – this was a prospective case study using descriptive analysis of the jac usage statistics of journal articles from january 2009 to september 2013.   results – jac enables ministry employees to access approximately 12,000 journals with full-text articles. jac usage statistics for the 2011-2012 calendar years demonstrate a steady level of activity in terms of searches, with monthly averages of 5,129. in 2009-2013, a total of 4,759 journal titles were accessed including 1,675 journals with full-text. usage statistics demonstrate that the actual consumption was over 12,790 full-text downloaded articles or approximately 2,700 articles annually.   conclusion – jac’s steady level of activities, revealed by the study, reflects continuous demand for jac services and products. it testifies that access to online journal databases has become part of routine government knowledge management processes. mohltc’s broad area of responsibilities with dynamically changing priorities translates into the diverse information needs of its employees and a large set of required journals. usage statistics indicate that mohltc information needs cannot be mapped to a reasonably compact set of “core” journals with a subsequent subscription to those.   introduction   use of research results in medical evidence based decision-making and healthcare evidence-informed policy advice has been recognized as essential to improve health outcomes (field, gauld & lawrence, 2012; lomas & brown, 2009; sosnowy, weiss, maylahn, pirani & katagiri, 2013). despite agreement on the importance of the issue and general consensus on the approaches, implementation of evidence based decision-making processes leaves much room for improvement. several barriers have been identified by researchers, including: lack of access and limited awareness of research results (wallace, nwosu and clarke, 2012); lack of practical use of systematic reviews (wallace et al., 2012); lack of organizational culture or supports (e.g. behaviour of supervisors, front-line staff and other professionals in the organization) (rapp et al., 2010); lack of time (solomons & spross, 2011); ambiguous and conflicting research (madhavji, araujo, kim & buschang, 2011; ubbink et al., 2011) or research having methodological inadequacies (o'connor & pettigrew, 2009); lack of skills, training or tools to acquire, assess, synthesize, disseminate and apply research evidence to inform policy related to health systems (ubbink, guyatt & vermeulen, 2013); lack of applicability/relevance of research (humphries, stafinski, mumtaz & menon, 2014); lack of standard knowledge translation strategies and processes effective in multiple contexts (humphries et al., 2014); lack of timely research outputs (oliver, innvar, lorenc, woodman & thomas, 2014; van der arend, 2014); and lack of interaction and collaboration between researchers and policymakers (oliver et al., 2014; wooding, hanney, pollitt, grant & buxton, 2014).   a significant challenge for health system practitioners (both in a clinical setting and in public service) in implementing research evidence is inadequate access to information, which results in doctors or analysts being unaware of the research (brownson et al., 2014; oliver et al., 2014; ubbink et al., 2013; ubbink et al., 2011; wallace et al., 2012). various types of information are required for producing high-quality evidence based policy advice, including journals, books, research reports, professional/trade magazines, etc. academic journals and professional magazines are the largest component of the potentially applicable information. arguably, almost all new research is published in journals. that makes access to journals a key pre-requisite for evidence based policy advice.    in canada, there are thirteen provinces and territories responsible for implementing evidence based health policies and services for the benefit of their populations. the ministry of health and long-term care (mohltc) of ontario, is one of the provincial ministries.   mohltc “is working to establish a patient-focused, results-driven, integrated and sustainable publicly funded health system” (mohltc, 2017a). mohltc’s mandate includes (but is not limited to): “establishing overall strategic direction and provincial priorities for the health system; developing legislation, regulations, standards, policies, and directives to support those strategic directions; monitoring and reporting on the performance of the health system and the health of ontarians; planning for and establishing funding models and levels of funding for the health care system;” etc. (mohltc, 2017a). from an organizational point of view, mohltc is a large and complex public corporation (mohltc, 2017b). the ministry has several locations in toronto, ontario and local units in major provincial cities. among the types of documentation produced by the ministry are: draft legislations and regulations, briefing notes, reports, program reviews and evaluations, recommendations, guidelines, etc. (mohltc, 2015).   the ministry, within the framework of the government of ontario, makes decisions or provides advice on a wide spectrum of issues by setting agendas, implementing and evaluating solutions and promoting improvements in the health system. the ministry oversees a health budget of more than c$50 billion per year. the following may serve as examples of types of evidence based decisions made by the ministry on an on-going basis: budget-related planning and control; allocation of public resources to the local health integrated networks; performance management of the health system and its elements; economic evaluation and health technology assessment of new medical interventions, drugs and systems; health system capacity planning, etc. specific examples of the ministry’s evidence based decision-making can be found in lomas and brown (2009) and khan et al. (2014). in 2015, the government of ontario established a centre of excellence for evidence-based decision making to enhance the capability of all ministries in making informed decisions.   mohltc takes specific measures to encourage evidence based policy-making to improve the healthcare system. these measures include, in particular, development and implementation of the policies and procedures of using research evidence, providing financial support to universities in generating new evidence and conducting knowledge transfer, and building and operating information systems to facilitate access to online journals, e.g. the journal access centre (jac).   jac is made available to all mohltc employees – several thousand people. obviously, actual use varies across the departments. for example, research, analytics, policy development, program evaluation and economic assessment units are known to be regular users of jac. the departments with primarily operational functions, such as financial management, corporate supply chain, and information technology, use jac with less regularity. informal surveys of staff conducted in 2011-2012 show that several hundred employees use jac weekly or daily and consider their jac skills as advanced.      prior to jac, access to journals was organized through subscriptions by individual ministry departments. subscriptions were not coordinated. overall, employees had access to several dozen printed journals.    objectives   the primary objective of the study was to examine online journal databases usage statistics for a provincial ministry of health in the context of evidence based decision-making. in addition, the study highlights implementation of the journal access centre that is housed and powered by the ontario ministry of health and long-term care to inform health systems’ policy making.   the research questions guiding this study were:   what are the usage statistics of searches that reflect user demand for online journals? what are the usage statistics of downloaded full-text articles that characterize the desired output of the journal access solution and reflect consumption of information? what are the usage statistics of the journal titles accessed by jac users? can information needs of mohltc users be satisfied by subscription to a core set of “most important” journals?   intent   the study has been undertaken with intent to use the results and recommendations of the research to:   inform ministry program evaluation and performance management processes; provide input into economic cost-effectiveness models to optimize jac journal acquisition; inform ministry senior management decisions on jac funding.   management decisions that were taken based on the recommendations of this study are not discussed.   out of scope   this study is the first step in assessing jac’s outputs. to assess value to the users and organization, the outcomes of higher-level processes of analyzing retrieved information and making evidence based decisions need to be examined. research questions at these phases would include: what are users doing with the articles they got from jac?; are retrieved articles pertinent and of high quality?; how do users analyze information?; how do they integrate evidence?; what is the process of making evidence based decisions in the organization? etc. all these research questions, although related to the topic of this study, are out of scope of this research.   the study examines usage statistics at the institutional level. analyses at lower levels, e.g. departmental or individual, are out of scope. similarly, usage statistics are examined at the journal title level; article-level considerations are out of scope.   methods   this study had a prospective case study design with descriptive analysis of jac usage statistics for two datasets. dataset 1 contains jac user statistics on the numbers of searches and downloaded full-text articles for the period from january 1st 2011 to december 31st 2012 (i.e. two full years of historical data). the following considerations were taken into account while selecting the period of historical data. firstly, trends in current research practice. similar or shorter periods of data collection to analyze usage patterns are quite common in academic studies, e.g.: a recent ciber research report found that two years of journal usage data is sufficient to provide insight into a journal’s usage patterns (ciber research limited, 2011), two-year periods were used in the studies of wical and vandenbark (2015), and chew, stemper, lilyard and schoenborn (2013). secondly, availability of data. a reasonable effort has been made to collect all pertinent available counter-compliant data (counter, n.d.). thirdly, alignment of the data collection period with the objective of the study and research questions. this was the first study to examine online journal access for a canadian ministry of health. the focus was primarily on capturing and interpreting the current state of usage – not on analyzing changes in trends (as there was no prior knowledge), which would have required multiple years of observation periods. analysis of the usage data allowed the objective of the study to be achieved.   it has been acknowledged that information needs and hence usage patterns of the online resources at a ministry of health are different from those in public health organizations (e.g. barr-walker, 2017; ford & korjonen, 2012), health care providers (e.g., younger, 2010) or of academic researchers (e.g., haglund & olsson, 2008; niu et al. 2010). information needs of the policy makers are commonly characterized by: firstly, a broad spectrum of the subject field which spreads beyond health care and includes economics, education, housing, etc. (e.g., brownson, royer, ewing & mcbride, 2006); secondly, a wide range of information sources which are not limited to academic journals and include grey literature and professional magazines with inputs on politics, values and opinions (e.g., ritter 2009); thirdly, a preference towards simple, uncomplicated information, for example evidence summaries (e.g., petkovic et al., 2016; ritter 2009; tricco et al., 2016); and finally, a dynamically changing environment and necessity to make rapid decisions which require availability of tools with timely access to information (e.g., oliver et al., 2014), making systems like jac indispensable.      a descriptive analysis of the usage of journal articles was conducted through the jac access tool from january 2009 to september 2013 (dataset 2). journal usage statistics for mohltc users were downloaded from the ebscohost administrative reporting site (ebscohost, n.d.). journal usage is characterized by the following indicators: i) number of searches, ii) number of full-text articles accessed in pdf or html format, iii) number of abstracts accessed, and iv) number of turnaways (access denied). these indicators were selected based on the recommendations of the internationally recognized standard: counter-2008, counting online usage of networked electronic resources (counter, n.d.). definitions of the indicators and related terms are available from counter (n.d.). numbers of searches characterize overall intensity of use of the jac and demand for this service. number of full-text articles characterizes the desired output of the solution and can be linked to the consumption of information provided by the service.   it should be noted that at the time of data collection, ebsco adhered to counter release 3. since then, the counter project has published an updated version of its code of practice – release 4. certain terminology and types of usage reports have been changed. there is no direct correspondence between all types of reports in release 3 and release 4. with some approximation, we can indicate that the data for performance measures used in the paper can be found in the following current counter reports: the number of searches – in the database report 1 (db1); number of abstracts – in the record views of db1; number of full text downloads – in the journal report 1 (jr1); and number of turnaways (access denied) – in database report 2 (dr2).   journal access centre implementation   to support evidence based decisions, mohltc of ontario, canada, built the journal access centre (jac). jac an online access tool supported by journal content selection, acquisition and consultation services – has been in operation with the mohltc since 2008, making the ministry one of the canadian healthcare pioneers of online access to academic journals. it was conceived and developed to facilitate online access to journals and serve as an enabling factor for enhanced evidence-informed policymaking. jac’s logic model is presented in figure 1.     figure 1 jac logic model.     a systems view of jac is presented in figure 2. mohltc acquire access to journals based on the annual subscriptions procured from various vendors including journal aggregators and individual publishers. the content is discussed and suggested for acquisition by the jac content selection advisory network – a permanent working group with representation from each of the ministry’s divisions.   mostly, access to content is acquired by journal databases, which represent collections of journal titles focused on a certain subject area. examples could be such well-known databases as medline, cinahl from ebsco (ebsco information services, n.d.) or academic onefile, academic onefile from gale cengage learning (gale cengage learning, n.d.). commonly, each database contains from a few hundred to several thousand journals. some journals and databases are acquired individually, e.g., the cochrane library, longwoods, etc. the total ministry subscription covers over 17,000 journal titles with over 9.0 million articles. these numbers include archives of prior years. journals cover such topics as health, medicine, social science, business, policy, economics, finance, management, risk management, etc.   usually, a database contains journals with different levels of access. some journals are provided with complete full-text article coverage, others are available only at an abstract or indexing level. some latest full-text journals have embargoes – delays in access from six months to three years. most databases contain a mix of academic journals and professional non-peer-reviewed magazines. the types of content of several databases are illustrated in table 1. the prime purpose of jac is to provide access to the full-text articles because abstracts and bibliographic data for most journals are available on the internet free of charge. hence, the most valuable segment of a database constitutes full-text, current, non-embargoed journals. despite the large number of journals and articles in the jac repositories, occasionally a need arises for an article that is not available in full-text. in these cases, the article on demand service manned by jac’s support staff orders materials and sends them to the jac users.    the technological backbone of the solution is a cloud-based application, search engine, which allows mohltc users to access journal databases offered by ebsco through the ministry’s intranet. the service is provided through the ebsco integrated search function. end users need only a web browser to access online journals. the ebsco search engine provides integrated coverage of the databases both owned by ebsco and bought from different providers – so end users can conduct a one-click search through all subscribed content. in addition, the ebsco search is integrated with google scholar search, i.e. when a ministry user is conducting a search in google scholar he/she gets reminders if an article, presented in google scholar search results, is available in the ministry’s ebsco subscriptions, and can click on the link to be immediately transferred to a full-text article in ebsco repositories. the service also provides automatic e-mail notifications of new content at an article level, which may be very specific to meet individual information needs. the service is available 24x7 with short periods of maintenance scheduled during weekends. the service has proved to be highly reliable: just one three-hour incident of service disruption has been observed in more than four years.     figure 2 jac structure.     table 1 types of database contenta database medline complete cinahl complete business source complete health business elite health policy reference center omnifile full text ageline total total number of journals in the database 2,184 5,453 5,023 714 472 3,125 209 17,180 abstracts only 0 3,825 1,191 64 37 0 209 5,326 full-text 2,184 1,628 3,832 650 435 3,125 0 11,854 stopped full-text 357 537 1,139 414 149 982 0 3,578 current full-text 1,827 1,091 2,693 236 286 2,143 0 8,276 embargoed current full-text 1,183 277 638 67 81 274 0 2,520 non-embargoed current full-text 644 814 2,055 169 205 1,869 0 5,756 non-embargoed current full-text peer-reviewed 608 587 943 140 143 1,184 0 3,605 non-embargoed current full-text magazines, trade publications 36 227 1,112 29 62 685 0 2,151 asome databases have overlapping content (journal titles). the number of unique journal titles in the system is less than shown in the table.      figure 3 number of searches (monthly average for 2011-2012).     figure 4 number of downloaded full-text articles (quarterly in 2011and 2012).     usage statistics results   this section presents the results for the number of searches from a report covering two full calendar years 2011-2012. figures 3 and 4 show monthly average number of searches, and quarterly number of downloaded full-text articles respectively. reported number of turnaways (access denied) is zero.   a report on the number of full-text articles and abstracts accessed by journal title for approximately 5000 journals is presented in additional file (appendix). this report covers the period from january 1, 2009 through september 30, 2013. a list of the most frequently used journals (top-50) is presented in table 2.     table 2 list of the most frequently used journalsa issn title full text abstract category 00178012 harvard business review 2208 734 npr 00900036 american journal of public health 303 129 pr 00284793 new england journal of medicine 270 189 pr, e3 08203946 canadian medical association journal 183 177 pr 00084263 canadian journal of public health 129 113 pr 15445208 health affairs 123 139 pr 00028614 journal of the american geriatrics society 118 74 pr, e12 03092402 journal of advanced nursing 118 40 pr, e12 01628968 inc. 114 5 npr 13558196 journal of health services research and policy 112 64 pr, e12 1357633x journal of telemedicine and telecare 94 9 pr 07067437 canadian journal of psychiatry 89 59 pr 01607480 modern healthcare 86 47 npr 09269630 studies in health technology and informatics 69 49 pr, e12 10688838 handhn: hospitals and health networks 69 25 npr 00034819 annals of internal medicine 68 44 pr 11707690 pharmacoeconomics 67 31 pr, e6 01406736 lancet 66 154 pr 0887378x milbank quarterly 64 40 pr, e12 17561833 bmj: british medical journal 63 115 pr, e3 10966218 journal of palliative medicine 62 29 pr, e12 14726963 bmc health services research 58 50 pr 13652702 journal of clinical nursing 55 23 pr, e12 13869620 health care management science 54 33 pr, e12 09660410 health & social care in the community 53 21 pr, e12 13561294 journal of evaluation in clinical practice 53 19 pr, e12     01958631 health care financing review 53 15 pr 07350732 healthcare financial management 51 40 pr 09652140 addiction 51 13 pr, e12 00197939 industrial and labor relations review 50 21 pr 14712458 bmc public health 48 29 pr 10792082 american journal of health-system pharmacy 48 18 pr 1477030x palliative medicine 48 14 pr 07461739 nursing economics 46 20 pr 00333107 psychology today 46 8 npr 03190781 toronto star (toronto, ontario) 43 32 npr 00130613 economist 42 24 npr 1095158x psychiatric rehabilitation journal 42 24 pr, e12 87569728 project management journal 41 39 pr, e12 03616878 journal of health politics, policy and law 40 40 pr, e12 14756773 health services research 40 34 pr 00413674 trustee 40 31 npr 00296570 nursing standard 40 25 pr 10903127 prehospital emergency care 38 16 pr, e18 10786767 journal of health care finance 37 19 pr 08982759 physician executive 37 14 pr 00048674 australian and new zealand journal of psychiatry 36 16 pr 08835381 healthcare executive 35 24 npr 15414469 international journal of health services 34 44 pr, e6 08949867 journal of traumatic stress (wiley) 34 23 pr, e12 aabbreviations in table 2: pr – peer-reviewed, npr – non-peer-reviewed, e – embargoed with indication of the delay in months.     figure 5 distribution of the number of journals falling into journal groups with similar numbers of full-text downloads.     based on the usage statistics, a histogram was developed. figure 5 shows distribution of the number of journals falling into journal groups with similar numbers of full-text downloads. the horizontal axis is non-linear for better visual presentation.   discussion   figures 3 and 4 are interrelated and are built in a counter-compliant way (searches, and downloads).   figure 3 shows the number of searches per month. during 2011-2012, there were over 123,000 searches with monthly averages of 5,129. smaller increments of time have been used to reveal usage patterns on a relatively short data period of dataset 1. it should be noted that this was the first effort to collect data and analyze jac usage. there was no prior information about the usage: whether the system had been used only from time to time (e.g. to support decision-making for an important health care policy or regulation) or whether it had been used at all.   it should be acknowledged that the charts are not exposing unexpected trends. figures 3 and 4 can be interpreted along the following lines. the data provides evidence that december is a slowdown period in jac use (minimum number of searches). this is quite understandable due to a number of statutory holidays and traditional vacation time. in january and february users increase jac activity (usage numbers show a steep rise) to make up for a slow year-end and to retrieve information that they need in planning for the next fiscal period (april 1 is the starting date for the government of ontario fiscal year). usage during all other months of the year does not display large variations. jac usage demonstrates a steady level of activity most of the year, i.e. it has become part of routine government knowledge-management processes.   the number of full-text articles, either downloaded in pdf format or viewed on the computer screen in html, characterizes the desired output of the system. figure 4 demonstrates the actual consumption of information in jac databases: in 2011-2012, over 4,600 full-text articles were accessed. during the same period of time more than 5,800 abstracts were accessed. this indicator is secondary, keeping in mind that the main purpose of jac is to provide access to full-text articles. however, the fact that the user accessed the abstract could be seen as an evidence of interest that the user had in the article but full-text may not have been available. if a journal, not available in full-text, has experienced multiple abstract viewing, it testifies that this journal should be considered for subscription in full-text version.   it was noted that some journals were accessed at abstract level extensively, but had zero full-text downloads. that may demonstrate that these journals publish pertinent articles but are not available with full-text. the ten journals that were most frequently accessed in this way (from 117 to 30 times) were: jama – journal of the american medical association, healthcare quarterly, medical care, healthcare papers, health policy, vaccine, journal of palliative care, diabetes educator. mohltc may consider exploring subscription to these journals with full-text taking into account cost-efficiency (botchkarev, 2013).   usage statistics as numbers of full-text articles and abstracts of individual journals for the period from 2009 to 2013 are presented in additional file (appendix). usage statistics for the top-50 most frequently used journals are shown in table 2. this data shows that mohltc users accessed 12,790 full-text articles and 14,517 abstracts. a total of 4,759 journal titles were accessed including 1,675 journals with full-text. the harvard business review is by far the most frequently used journal – it was used seven times more than the second-ranked journal: the american journal of public health.   table 2 demonstrates that the most frequently used journals include both peer-reviewed (82%) and non-peer-reviewed (18%) periodicals. approximately 50% of the most frequently used peer-reviewed journals have embargoes from 3 to 18 months, predominantly 12 months.   the histogram presented in figure 5 indicates spread of usage across multiple journal titles. usage does not demonstrate a “core” set of journals there are only 10 journals with over 100 downloads each. the total number of downloads from these journals is 3,678. similarly, there are only 30 journals that were accessed in full-text format 50 times or more. these journals contributed only 38% (4,953) of the accessed articles. of the accessed journals, 54% (908 titles) were accessed with full-text only once or twice. this group contributed 1,192 articles (9%). this pattern of usage can be attributed to the following factors. first, there is an increasing amount of research being conducted that triggers a persistently growing number of publication outlets. second, mohltc has a very broad area of responsibilities with dynamically changing priorities, which translates into the diverse information needs of its employees. a practical conclusion from the usage analysis is that mohltc information needs cannot be mapped to a reasonably compact set of “core” journals with a subsequent subscription to those. in this case, subscription economics necessitate the use of journal aggregators, e.g. ebsco, gale, etc., as the main source of journal access acquisition (botchkarev, 2013).   certain jac user feedback is notable. this information has been collected in non-structured conversations with clients and is not supported by quantitative assessments. despite availability and actual use of thousands of journals, there is a need to (i) expand access to more peer-reviewed journals, (ii) expand access to more journals with full-text articles, (iii) explore opportunities to reduce embargoes. some users expect immediate online access to the full-text articles of interest, requesting no abstract-only, no delays/embargoes content. if these expectations are not met, user satisfaction might decline rapidly.   the focus of this empirical study is on usage statistics in terms of number of downloads, etc., which represent the outputs of the jac solution. there is no implication or judgement about the value that jac collections provide to users. this study is the first step in assessing value to the users and organization. it needs to be emphasized that there is no rationale to judge the value of jac collections through the rate of downloads per user (high or low – good or bad) because there is nothing to compare to either in mohltc or in other organizations (e.g. commonly accepted benchmarks do not exist). to assess value, we would need to consider the outcomes of higher-level processes.   the findings reported in this paper can be used in various government organizations to implement journal subscriptions strategies that will better meet ministry employees’ information needs and contribute to cost-efficiency of operations.    study assumptions and limitations   to the best of our knowledge, this is the first study to address usage statistics for online journal databases in a canadian ministry of health. however, this study has certain limitations that should be made explicit.   use of jac implies that its collections are of interest for the mohltc employees. however, numbers of article downloads may not be equal to actual use or satisfaction – users may download an article and find it worthless for their task or they may be unhappy because they did not find specifics they needed.   jac statistics used in this study have been downloaded from the ebsco reporting site. this data is based on the automatic logs and is believed to be very accurate. however, the following should be noted.   firstly, jac is not an exclusive channel of information for mohltc users. some users have access to online journals at local universities through their alumni connections. others have access to journal repositories based on their memberships in professional associations. certain departments used to have subscriptions to publications in their specific narrow fields. as a result, documented in the study number of downloads represents a lower border of actual downloads.   secondly, jac is using ebsco integrated search services that allow access to the databases that are owned by ebsco, and those of the third parties external to ebsco. search results presented to the user include both internal and external documents. when a person clicks on the link to an external database, he/she is transferred to the document in the external database. as soon as a person moves to an external database, ebsco usually does not have information about what is happening there, and so cannot include activity in the report. that pertains especially to full-text documents. as a result, jac statistics may be missing data on the use of full-text documents in external databases. an example of this situation could be statistics on the use of the cochrane database shown in the additional file (appendix). jac has a direct subscription to the cochrane database with full-text documents, which makes it external to ebsco. the usage statistics indicate zero downloads of cochrane full-text documents. however, this is misleading as it has been verified through conversations with jac users that the cochrane database actually has been used.   thirdly, in some cases, ebsco usage reports are not perfect. there were a certain number of duplications of journal titles in the initial version of additional file (appendix). for example, (i) some titles were duplicated because of using different online and print issns (e.g., academic emergency medicine: official journal of the society for academic emergency medicine: usage was attributed to one title. another was deleted); (ii) some titles were duplicated because in one case the title had an issn and in another the issn was blank (e.g., the academy of management executive: usage was attributed to one title. another was deleted); (iii) some titles were duplicated because of spelling mistakes (e.g., american journal of publichealth vs american journal of public health); (iv) different title abbreviations were used – most likely in different databases – (e.g., bmc health services research vs bmc health serv res); (v) use of & instead of and; (vi) using titles with or without the definite article; and (vi) just typos.   fourthly, it was shown in annual customer surveys (not reported in the current paper), that jac users were not satisfied when search results contained a large number of articles with abstracts only. it took additional time to look through several pages of search results to find articles with full text. in march 2012, according to the recommendation of the jac content selection advisory network, the jac default search was configured to present full-text articles only. if a researcher was willing to analyze additional abstracts-only articles, he/she could adjust the search configuration. that may have decreased the number of retrieved abstracts from the second half of 2012.   conclusions   this is the first paper, to our knowledge, to examine usage statistics for online journal databases in a ministry of health.   the ministry of health and long-term care built and has maintained the journal access centre – an online access tool supported by journal content selection, acquisition and consultation services. as a key prerequisite for evidence based policy-making, jac enables access to approximately 12,000 journals with full-text articles.   jac usage statistics for the 2011 – 2012 calendar years demonstrate a steady level of activities in terms of searches with a monthly average of 5,129. jac’s steady level of activities, revealed by the study, reflects a continuous demand for the jac services and products. it testifies that access to online journal databases has become part of routine government knowledge-management processes.   the number of downloaded full-text articles characterizes the desired output of the solution. jac usage statistics demonstrate that the actual consumption in 2009-2013 was over 12,790 full-text articles or approximately 2,700 articles annually.   usage statistics helped to identify a list of journals that were accessed at abstract level extensively (117 to 30 times), but had zero full-text downloads. that led to a practical recommendation to consider subscription to these journals with full-text, taking into account cost-efficiency.   jac usage statistics for the period 2009-2013 provide evidence that a total of 4,759 journal titles were accessed including 1,675 journals with full-text. mohltc’s broad area of responsibilities with dynamically changing priorities translates into the diverse information needs of its employees and a large set of required journals. usage statistics indicate that mohltc information needs cannot be mapped to a reasonably compact set of “core” journals with a subsequent subscription to those. in this case, subscription economics necessitate the use of journal aggregators (e.g. ebsco, gale, etc.) as the main source of journal access acquisition.   future efforts could be focused on studying: (i) usage statistics complemented with data beyond ebsco reports and covering all sources of online academic journals available in jac; (ii) usage statistics by individual ebsco databases, e.g. medline, cinahl, etc., and category of journals, e.g. peer-reviewed, non-peer-reviewed, embargoed, etc.; iii) in-depth usage of information sources and patterns of behaviour at the level of individual article as compared to the journal title level in this paper; (iv) jac users’ information needs and preferences; (v) jac’s usability through the customer satisfaction survey; and (vi) development and examination of a model of a value creation chain which would integrate jac outputs with outcomes of higher-level processes of analyzing retrieved information and making evidence based decisions.       acknowledgements   the views, opinions and conclusions expressed in this document are those of the author alone and do not necessarily represent the views of the ontario ministry of health and long-term care or any of its individual departments.   the author is grateful to dr. grégory léon, université laval, dr. mathieu ouimet, université laval and dr. john n lavis, mcmaster university, for helpful comments on earlier drafts of the paper that had a distinct scope and objectives. however, mentioned scholars are not responsible for the content and conclusions of the published paper.      references   barr-walker, j. (2017). evidence-based information needs of public health workers: a systematized review. journal of the medical library association: jmla, 105(1), 69-79. http://dx.doi.org/10.5195/jmla.2017.109   botchkarev, a. (2013). mining evidence in the online academic journal databases: access and performance in public sector. e-prints in library and information science (e-lis). retrieved from http://eprints.rclis.org/19286    brownson, r. c., reis, r. s., allen, p., duggan, k., fields, r., stamatakis, k. a., & erwin, p. c. (2014). understanding administrative evidence-based practices. american journal of preventive medicine, 46(1), 49-57. http://dx.doi.org/10.1016/j.amepre.2013.08.013   brownson, r. c., royer, c., ewing, r., & mcbride, t. d. (2006). researchers and policymakers: travelers in parallel universes. american journal of preventive medicine, 30(2), 164-172. http://dx.doi.org/10.1016/j.amepre.2005.10.004   chew, k., stemper, j., lilyard, c. and schoenborn, m. (2013). user-defined valued metrics for electronic journals. in s. hiller, m. kyrillidou, a. pappalardo, j. self & a. yeager (eds.). proceedings of the 2012 library assessment conference, charlottesville, va, oct 29-31. retrieved from http://conservancy.umn.edu/bitstream/handle/11299/144461/1/userdefinedvaluebasedmetricsforelectronicjournalspdf.pdf   ciber research limited. (2011). the journal usage factor: exploratory data analysis: stage 2 final report, 27 may 2011. retrieved from http://ciber-research.eu/download/20110527-the_journal_usage_factor.pdf     counter (n.d.). counting online usage of networked electronic resources. retrieved from http://www.projectcounter.org    ebsco information services. (n.d.). retrieved from https://www.ebsco.com    ebscohost. (n.d.). retrieved from http://eadmin.ebscohost.com/eadmin/reports/selectreportsform.aspx    field, p., gauld, r., & lawrence, m. (2012). evidence-informed health policy the crucial role of advocacy. international journal of clinical practice, 66(4), 337-341. http://dx.doi.org/10.1111/j.1742-1241.2012.02883.x   ford, j., & korjonen, h. (2012). information needs of public health practitioners: a review of the literature. health information & libraries journal, 29(4), 260-273. http://dx.doi.org/ 10.1111/hir.12001   gale cengage learning. (n.d.). retrieved from http://www.gale.cengage.com     haglund, l., & olsson, p. (2008). the impact on university libraries of changes in information behavior among academic researchers: a multiple case study. the journal of academic librarianship, 34(1), 52-59. http://dx.doi.org/10.1016/j.acalib.2007.11.010   humphries, s., stafinski, t., mumtaz, z., & menon, d. (2014). barriers and facilitators to evidence-use in program management: a systematic review of the literature. bmc health services research, 14, 171. http://dx.doi.org/10.1186/1472-6963-14-171   khan, s., moore, j. e., gomes, t., camacho, x., tran, j., mcauley, g., juurlink, d.n., paterson, m., laupacis, a., & mamdani, m. m. (2014). the ontario drug policy research network: bridging the gap between research and drug policy. health policy, 117(3), 392-398. http://dx.doi.org/10.1016/j.healthpol.2014.06.007   lomas, j., & brown, a. d. (2009). research and advice giving: a functional view of evidence-informed policy advice in a canadian ministry of health. milbank quarterly, 87(4), 903-926. http://dx.doi.org/10.1111/j.1468-0009.2009.00583.x   madhavji, a., araujo, e. a., kim, k. b., & buschang, p. h. (2011). attitudes, awareness, and barriers toward evidence-based practice in orthodontics. american journal of orthodontics and dentofacial orthopedics, 140(3), 309-316. http://dx.doi.org/:10.1016/j.ajodo.2010.05.023      mohltc. (2015). publications. retrieved from http://www.health.gov.on.ca/en/public/publications/default.aspx     mohltc. (2017a). about the ministry. retrieved from http://www.health.gov.on.ca/en/common/ministry/default.aspx     mohltc. (2017b). organization chart. retrieved from http://www.health.gov.on.ca/en/common/ministry/orgchart.pdf    niu, x., hemminger, b. m., lown, c., adams, s., brown, c., level, a., mclure, m., powers, a., tennant, m.r., & cataldo, t. (2010). national study of information seeking behavior of academic researchers in the united states. journal of the american society for information science and technology, 61(5), 869-890. http://dx.doi.org/ 10.1002/asi.21307   o'connor, s., & pettigrew, c.m. (2009). the barriers perceived to prevent the successful implementation of evidence-based practice by speech and language therapists. international journal of language & communication disorders, 44(6), 1018-1035. http://dx.doi.org/10.1080/13682820802585967   oliver, k., innvar, s., lorenc, t., woodman, j., & thomas, j. (2014). a systematic review of barriers to and facilitators of the use of evidence by policymakers. bmc health services research, 14, 2. http://dx.doi.org/10.1186/1472-6963-14-2   petkovic, j., welch, v., jacob, m. h., yoganathan, m., ayala, a. p., cunningham, h., & tugwell, p. (2016). the effectiveness of evidence summaries on health policymakers and health system managers use of evidence from systematic reviews: a systematic review. implementation science, 11, 162. http://dx.doi.org/10.1186/s13012-016-0530-3   rapp, c. a., etzel-wise, d., marty, d., coffman, m., carlson, l., asher, d., callaghan j., & holter, m. (2010). barriers to evidence-based practice implementation: results of a qualitative study. community mental health journal, 46(2), 112-118. http://dx.doi.org/10.1007/s10597-009-9238-z   ritter, a. (2009). how do drug policy makers access research evidence? international journal of drug policy, 20(1), 70-75. http://dx.doi.org/10.1016/j.drugpo.2007.11.017   solomons, n. m., & spross, j. a. (2011). evidence-based practice barriers and facilitators from a continuous quality improvement perspective: an integrative review. journal of nursing management, 19(1), 109-120. http://dx.doi.org/10.1111/j.1365-2834.2010.01144.x   sosnowy, c. d., weiss, l. j., maylahn, c. m., pirani, s. j., & katagiri, n. j. (2013). factors affecting evidence-based decision making in local health departments. american journal of preventive medicine, 45(6), 763-768. http://dx.doi.org/10.1016/j.amepre.2013.08.004   tricco, a. c., cardoso, r., thomas, s. m., motiwala, s., sullivan, s., kealey, m. r., hemmelgarn, b., ouimet, m., hillmer, m.p., perrier, l., shepperd, s., & straus, s.e. (2016). barriers and facilitators to uptake of systematic reviews by policy makers and health care managers: a scoping review. implementation science, 11, 4. http://dx.doi.org/10.1186/s13012-016-0370-1   ubbink, d. t., guyatt, g. h., & vermeulen, h. (2013). framework of policy recommendations for implementation of evidence-based practice: a systematic scoping review. bmj open, 3(1). http://dx.doi.org/10.1136/bmjopen-2012-001881   ubbink, d. t., vermeulen, h., knops, a. m., legemate, d. a., oude rengerink, k., heineman, m. j., roos y.b., fijnvandraat c.j., heymans h.s., simons r., & levi, m. (2011). implementation of evidence-based practice: outside the box, throughout the hospital. netherlands journal of medicine, 69(2), 87-94.   van der arend, j. (2014). bridging the research/policy gap: policy officials' perspectives on the barriers and facilitators to effective links between academic and policy worlds. policy studies, 35(6), 611-630. http://dx.doi.org/doi:10.1080/01442872.2014.971731   wallace, j., nwosu, b., & clarke, m. (2012). barriers to the uptake of evidence from systematic reviews and meta-analyses: a systematic review of decision makers’ perceptions. bmj open, 2(5). http://dx.doi.org/10.1136/bmjopen-2012-001220   wical, s.h. and vandenbark, r.t. (2015). notes on operations: combining citation studies and usage statistics to build a stronger collection. library resources & technical services, 59(1), pp.33-42. retrieved from https://journals.ala.org/lrts/article/view/2755/2733   wooding, s., hanney, s. r., pollitt, a., grant, j., & buxton, m. j. (2014). understanding factors associated with the translation of cardiovascular research: a multinational case study approach. implementation science, 9, 47. http://dx.doi.org/10.1186/1748-5908-9-47   younger, p. (2010). internet‐based information‐seeking behaviour amongst doctors and nurses: a short review of the literature. health information & libraries journal, 27(1), 2-10. http://dx.doi.org/10.1111/j.1471-1842.2010.00883.x     appendix   additional file: jac usage statistics by journal title 2009-2013     evidence based library and information practice evidence based library and information practice 2009, 4:3  evidence based library and information practice       news    the 5th international evidence based library and information practice conference  (eblip5): conference report and reflections      © haglund, wakimoto, hunsucker, mohan, shea, usova, plevier, eldredge, de meulemeester, and wilson,  2009. this is an open access article distributed under the terms of the creative commons attribution  license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and  reproduction in any medium, provided the original work is properly cited.      conference report   lotta haglund  head of information and public relations   karolinska institutet universitetsbiblioteket  stockholm, sweden  email: lotta.haglund@ki.se    the 5th international evidence based library  and information practice conference  (eblip5) was held in stockholm, sweden  from june 30 to july 3, 2009. the conference  was attended by a total of 163 participants  from 18 countries, of which approximately  70% came from university/academic  libraries. the organizers were pleased with  the unexpectedly high turnout from sweden  (68 participants), where eblip, until  recently, was more or less unknown to most  librarians. on the top five list of the highest  number of participants after sweden came  norway (17), canada (16), the u.s. and the  u.k. (15 each). the remaining participants  came from australia, belgium, denmark,  finland, germany, hong kong, hungary,  iceland, india, iran, the netherlands, qatar,  and turkey.     the first and last conference days were  reserved for five workshops, and 61  conference delegates took advantage of the  opportunity to increase their knowledge in  the subjects being offered. during the three  main conference days there were six plenary  speakers (five invited key note speakers and  one conference showcase), as well as 35  presentations in parallel sessions, and 14  posters.    no conference can be successful without  social events, and library visits are also  important to include in the program. eblip5  offered plenty of opportunities for delegates  to meet their colleagues in a more informal  setting, and the invitation from the  stockholm city and county council to a  reception at city hall was much  appreciated. for the conference dinner the  hosts had done their best to arrange a  meeting with the swedish royal family as  well as dignitaries from the european  union, but they were a bit busy with the  inauguration of the swedish presidency of  the e.u. taking place next to the dinner  venue at skansen. the dinner was held in  the restaurant solliden, with one of  stockholm’s best views of the city, where  the guests were served swedish specialties  and a band kept the dancing going well  after midnight. both the reception and the  78 mailto:lotta.haglund@ki.se evidence based library and information practice 2009, 4:3  conference dinner were attended by roughly  130 people.     the library visits offered before the main  conference days were also well‐attended: 45  conference participants visited either the  karolinska institutet university library, the  national library of sweden, or the subway  (public) library at the tube stop  östermalmstorg.     at the closing of the conference, christine  urquhart from aberystwyth university  received an award for the best oral  presentation, and ann‐christin persson,  maria lång & jessica nilsson from lund  university for the best poster.       heading to city hall  photo courtesy virginia wilson      reflections    a number of conference delegates  submitted brief commentaries to eblip  about their conference experience, and what  they learned, for the benefit of journal  readers who were unable to attend.            diana wakimoto  online literacy librarian   california state university east bay  san francisco, california, united states of  america  email: diana.wakimoto@csueastbay.edu    one of the most exciting aspects of the  recent eblip5 conference was the  prevalence of sessions dedicated to  information literacy practice and research.  there were three parallel sessions, one  round table discussion and countless  informal opportunities to talk with other  conference participants about our  experiences teaching and assessing  information literacy learning and  instruction. the conference was a wonderful  opportunity to gain international  perspectives on many issues and to feel  connected to a community of researchers  and practitioners.                    this was my first eblip conference and it  was the best conference i have had the  pleasure of attending. the entire conference  was thoughtfully organized and provided a  depth and range of presentations that were  thought‐provoking and inspiring. i felt  energized to continue championing  evidence based practice at my home  institution.                     the parallel sessions on information literacy  were engaging and offered many different  ways of assessing information literacy  instruction, some of which i had never  heard of or considered before. it was a relief  to hear that some of the issues i deal with at  my institution (such as trying to engage  students, finding meaningful ways of  evaluating student learning, and proving  the worth of information literacy instruction  to faculty) were fairly universal issues. now,  thanks to this conference, i have more  colleagues to exchange ideas with about  solving these issues. my hope is that  together we can all contribute to the  79 mailto:diana.wakimoto@csueastbay.edu evidence based library and information practice 2009, 4:3  evidence base in order to assist more  librarians in their teaching and assessment  work.                    the inclusion of the round table discussions  in the conference program was a great,  semi‐structured way for groups to get  together and discuss issues more in‐depth  than was possible in the q&a time after the  parallel session presentations. the  information literacy round table was very  well attended and sparked a lively  discussion about how best to implement  information literacy instruction and  assessment. margy macmillan of mount  royal college did a fantastic job facilitating  this discussion so that everyone’s voice  could be heard.                    for me, some of the most helpful conference  events were the morning and afternoon  coffee breaks, which allowed time for  networking and small group discussions  about issues or ideas that had occurred to us  in prior sessions. during the coffee breaks, i  talked to colleagues about information  literacy, library staffing structures and ideas  about embedding instruction into the  curriculum. also, it was just a fun time to  chat over a cup of coffee or tea and a treat.                    without a doubt, this conference was a  worthwhile event to attend and participate  in. i would unconditionally recommend the  conference to anyone interested in eblip  and i cannot wait for the conference in 2011.       laval hunsucker  specialist voor de oudheid (antiquities  librarian)  universiteitsbibliotheek, universiteit van  amsterdam  amsterdam, the netherlands  email: r.l.hunsucker@uva.nl    what is the present status, some 12 years  after its appearance on the scene, of the  “evidence based” ideology in the world of  information and library services? what has  changed? what has not? how far have we  come, and how far do we have to go?     these questions were not openly discussed  (or even expressly posed) at the 5th eblip  conference in stockholm. yet i gathered,  from the many sessions i attended, that  answers to them were being given, even if  merely in passing, unwittingly, or implicitly.    if asked to characterize the general drift of  the answers i discerned, my impression was  that many eblip advocates seem to have  abandoned the positivistic mindset and the  science‐based approach that informed the  movement’s early development. more than  a decade of additional reflection and  experience appear to have eroded the  confidence that unequivocal evidentiary  (re)sources will be necessarily or readily  available to us; the optimism that, if they  are, their appropriateness and the  application of them to actual situations can  be considered a more or less straightforward  affair; and the too facile supposition that  empirical methodologies familiar from the  medical and some social sciences can or  should be, also for us, the great guiding  light.    this is, then, the undercurrent which i saw  flowing through many plenary and other  sessions in stockholm – in fact, already from  the conference’s beginning, and gaining  momentum down to its very end. but what  does this observation entail, more  specifically? a brief summary of some  speakers’ pertinent thoughts:    we must construct questions, but also  recognize the importance of metaquestions  (eldredge, opening keynote). good eblip  approaches to practical questions can indeed  be theory‐driven (e.g., dalrymple on   80 mailto:r.l.hunsucker@uva.nl evidence based library and information practice 2009, 4:3  wednesday morning; lam on tuesday). in  order to define what the problem to be  addressed is, one must first investigate what  the world‐views are of the persons involved  (mirijamdotter, tuesday afternoon).  information professionals still base what  they do far too insufficiently on a  knowledge of the specific discourse‐ communities that exist among their user  populations (pilerot, wednesday afternoon),  and should furthermore be more  community‐oriented even within their own  ranks (eldredge, urquhart). our orientation  to eblip should be more participative  (mirijamdotter), but also less cognitive  (pilerot, mirijamdotter). dalrymple  emphasized the value of “insight”, gained  through exploratory research, and pilerot that  of “craft knowledge” – adding emphatically  that it is also essential for us to  “problematise rationality”. jandér showed  that applying the principle of “constructive  alignment” can be useful. finally, there was  a new recognition of the evidentiary value  of narrative. dalrymple and lam were quite  clear on this, and herron, in his closing  remarks to the conference on thursday,  spoke of the “exciting possibilities” which  narrative‐derived evidence may have to  offer.      jon eldredge relaxing  photo courtesy virginia wilson      in that same closing session, björklund even  implied that he could now detect two  different kinds of eblip: the “classic” one,  as it were, on the one hand; and the  revisionary, more complex and world‐wise  version on the other. i would even go so far  as to suggest that precisely such a  heightening of awareness should be  considered the most significant  accomplishment of the 2009 eblip  conference in stockholm. and i would  predict that, if founded upon that second,  revisionary and more sophisticated, but also  more hermeneutic, conception, eblip can  have a bright future. it depends on our own  determination. making it work is going to  be more complicated and subtle than many  had perhaps previously thought, but also  (therefore) more interesting and possibly –  who knows? – more fun.       v. vishwa mohan  professor and chairman, bos   department of library and information  science  osmania university, hyderabad, india  email: drvvm321@yahoo.com    after attending the eblip conference i  developed a better understanding of the  western approach toward knowledge  generation and theory building. at the  plenary session on “applying evidence to  practice: gaps, barriers and lessons  learned” moderated by prudence  dalrymple, i learned that there is a trend in  developing new sciences such as  ‘implementation science’ and ‘team  science’. the session shed light on areas of  concern including professionalism and the  potential gaps and barriers faced when  applying evidence to practice. of particular  note was the paper presented by gordon  watson entitled, “metadata harvesting  seamlessly adds high quality national  evidence to the local library catalogue  alongside user knowledge and library  resources”. in his presentation, watson  explained how his institution had integrated  high quality national public health evidence  81 mailto:drvvm321@yahoo.com evidence based library and information practice 2009, 4:3  with locally‐held physical resources and  user‐generated knowledge in a web‐based  library management system. his  explanation of how they assess high quality  evidence was so convincing that it led me to  believe that such practices should be  adopted by library management systems  everywhere, even in developing countries.     these are some of the observations i had  and perspectives i gained; by and large it  was a significant learning experience.      alison shea  reference librarian   fordham university school of law  new york, new york, united states of  america    email: aashea@law.fordham.edu    as the only law librarian in attendance at  the 5th international evidence based library  and information practice conference this  past june, i was a bit worried about whether  i would be able to get as much out of the  conference as other professionals,  specifically medical librarians who seem to  make up the core contingent of the eblip  movement.    however, i can safely say that not only did i  come away with an incredible amount of  great ideas stemming from eblip, i was also  able to gain insight into how librarians in  other sectors conduct their jobs. for  example, while i will likely never train rural  clinicians on the use of pdas, i found rick  wallace’s presentation on this topic to be  very interesting—not only because it  allowed me to see how eblip is carried out  in the “real” world but also because it was  fascinating to learn that many medical  professionals do not consult published  information when treating a patient which  was a shock to me!    eblip5 banquet  photo courtesy virginia wilson      further, i picked up many great ideas on a  variety of topics, including: diana  wakimoto’s use of the web‐based  presentation tool prezi in her session on  information literacy instruction; a set of  criteria to evaluate virtual reference  programs that were found in the course of  lorna rourke and pascal lupien’s  systematic review of virtual reference  literature; and a number of fantastic “light  bulb” moments during the course of sue  mcknight’s plenary on how to rectify  customer expectations and service  provisions in the library. one of mcknight’s  most memorable points was something to  the effect of, “why should we be comparing  ourselves to other libraries when our  customers only ever use our library?” these  ideas and a host of others gave me much to  think about in relation to how things are  done in my own library, despite the fact that  none of the examples given occurred in a  law library setting    the conference was exceptionally well  organized, and i cannot praise the  organizers enough for distributing our  conference materials in a designer tote bag.  you can be sure my colleagues back in new  york were green with envy recalling their  disused collections of plastic, vendor‐ sponsored bags accumulated at various  inferior conferences!     82 mailto:aashea@law.fordham.edu evidence based library and information practice 2009, 4:3  in short, not only would i enthusiastically  encourage any interested librarian to pursue  eblip in his or her own work in preparation  for attendance at the 6th eblip conference,  but i would also encourage librarians to  look “outside the box” when considering  which professional organizations to belong  to and conferences to attend. although cost  is always a limiting factor, there is an  incredible amount of value in exploring new  areas and learning how other sectors  undertake tasks that transcend librarianship  as a whole.        tatiana usova  directrice   bibliothèque saint‐jean, university of  alberta libraries   edmonton, alberta, canada  email: tatiana.usova@ualberta.ca    i was thrilled to be part of the 5th  international evidence based library and  information practice conference (eblip5).  as a new step in my professional  development, the conference gave me an  excellent opportunity to attend inspiring  and thought provoking presentations, meet  like‐minded peers, and learn from them.     i am relatively new to the concept of  evidence based library and information  practice. prior to attending the conference,  my observation was that research and  library practice are like rails that exist  parallel to each other without crossing.  eblip5 provided me with a new perspective  on the profession. as a practitioner, i am  always looking for ways to increase the  effectiveness of services. searching for  evidence when making decisions is one  approach to take in order to be successful in  one’s professional practice. this resonates  well with the topic of jonathan eldredge’s  keynote speech entitled, “bridging the gaps:  linking our eblip questions to our  decisions”. i really liked his engaging way  of communicating with the audience, which  was a combination of group work and  think‐pair‐share exercises. in her talk  “bridging the gap between service  provision and customer expectations,” sue  mcknight gave an interesting insight into  research undertaken in australia and the  u.k. to identify customer values and  “irritants” to service delivery. this research  that challenged library staff assumptions  and helped to define an action plan to  bridge the service gaps.      during parallel sessions i had a chance to  listen to my colleagues, passionate adopters  of the eblip concept. i especially  appreciated the presentations by margy  macmillan and diana wakimoto (theme:  information literacy) as well as denise pan  and mary m. somerville (theme:  management). the materials referenced in  these sessions can be accessed through the  following web site:  http://blogs.kib.ki.se/eblip5/parallel‐ sessions.html    i would like to thank the local organizing  committee for all the work they have done.  the conference organization was beyond  praise, from the city center location, clear  signs and helpful staff to the comprehensive  program. i was pleasantly surprised by the  exquisite food and superior venues of the  eblip5 receptions, unexpectedly shared  with launch events for swedenʹs presidency  in the e.u.       josepha w.m. plevier  information specialist, walaeus library,  leiden university medical centre   leiden, the netherlands  email: plevier@lumc.nl    this was my first time attending an eblip  conference, and in fact, my first  83 mailto:tatiana.usova@ualberta.ca http://blogs.kib.ki.se/eblip5/parallel-sessions.html http://blogs.kib.ki.se/eblip5/parallel-sessions.html mailto:plevier@lumc.nl evidence based library and information practice 2009, 4:3  international congress which made it all the  more exciting.    my experience began with the pre‐ conference workshop led by jon eldredge  on monday morning and a visit to  karolinska institutet solna library in the  afternoon. these were perfect occasions to  get acquainted with a small group of  colleagues before jumping into the crowds  of the conference opening on tuesday.  throughout all of the lectures, parallel  sessions, workshops and poster  presentations i was amazed by how many  roles librarians play in all kinds of libraries  everywhere to enhance their professional  practice. my interest was mainly focused on  information literacy and reference desk  work, which meant that i couldn’t possibly  attend all of the sessions i was interested in.  this is, in fact, a luxury and by no means a  complaint.    the overall atmosphere in stockholm was  warm and welcoming. while the  temperatures outside were tropical, inside  the folkets hus we had good  accommodations with fantastic lunches and  coffee and tea buffets. participants walked  around proudly with their linen bags of  colourful swedish design. it was also  convenient to find computers at every  corner, allowing me the opportunity to  answer enquiries from my patrons at home  in leiden, although the swedish keyboards  are still a bit of a mystery to me. the local  organizing committee was always helpful in  directing and accompanying anyone with  questions. and last but not least, the  reception and buffet in city hall and the  dinner at skansen restaurant were perfect  for viewing stockholm. i enjoyed the  excellent food and drink, music and dance,  together with nice company and interesting  conversations with colleagues from all over  the world. fortunately, i had some spare  time to visit some of the swedish shops  before taking my train back to holland.   i would like to thank the organizing  committee for this impressive event. i will  surely tell my colleagues about evidence  based library and information practice and  the eblip journal and try to apply the  learned principles and practices whenever  possible.        andrew booth at his preconference  photo courtesy virginia wilson        jonathan eldredge  associate professor, the university of new  mexico  albuquerque, new mexico, united states of  america  email: jeldredge@salud.unm.edu    as our work lives become busier and busier,  we find ourselves setting priorities that we  previously thought unimaginable. at one  time, i tried to attend most professional  conferences related to my job duties. now i  find myself setting rigorous standards for  determining whether i will attend a  conference, and afterward how i will  evaluate its utility. my shorthand  categorizations for these time‐conserving  goals are: transactional, content, and  catalytic.    84 mailto:jeldredge@salud.unm.edu evidence based library and information practice 2009, 4:3  eblip5 succeeded in meeting all three  criteria. the first category, transactional,  involves the kinds of interactions i have  with colleagues at conferences. at eblip5 i  made a point of joining groups of colleagues  i did not know who were engaged in  conversations during breaks or meals. over  the years i have found these discussions at  various conferences to be rewarding for  learning other experiences and perspectives.  every conversation that i joined during  eblip5 yielded wonderful insights. i also  made contacts that i hope to pursue for the  virtual peer mentoring (vpm) project i  described during my opening keynote  address.     this transactional goal also involved my  spending time with colleagues whom i have  known for quite a few years. as mentioned  in my keynote, many colleagues have served  as vpms for my own practice and research  projects. these vpm contacts are my most  treasured professional relationships. the  vpms in my own circle who i was able to  see at eblip5 included andrew booth,  alison brettle, anne brice, pru dalrymple,  maria grant, denise koufogiannakis, and  joanne marshall. two of the biggest  highlights of eblip5 were having long‐time  colleague anne brice introduce me as  keynote speaker and having the opportunity  to spend large blocks of time engaged in in‐ depth discussions with andrew booth.    the eblip5 plenary sessions, presentations,  and posters yield significant content. while  overall, the plenary sessions tended to focus  on offering new perspectives, sue  mcknight’s plenary on customer value  discovery exposed me to new methods of  evaluating libraries. recent research  findings delivered in presentations or  posters formed the evidential core of  eblip5. the award‐winning contributions  generally represent the most recognized  evidence reported at eblip5. yet, i found  some promising directions for further  research among some of the less recognized  or polished communications. almost all  eblip5 paper and poster sessions were  unique opportunities to inquire about  challenges faced and eventually resolved by  presenters as they pursued their research.  the sessions also allowed for the presenters  or eblip5 participants to engage in possible  interpretations, or perhaps speculations,  about the meanings of results.    the third category, catalytic, could apply to  either the transactional or content  categories, except that i wish to elevate its  worth as a reason for attending conferences.  eblip5 exceeded my already high  expectations of providing new insights for  both my applied research and for my  practice of eblip. most of the trip from  stockholm to madrid via prague was spent  recording my thoughts prompted by eblip5  on how i could improve my practice or how  i might design new research projects.      conference bags  photo courtesy eblip5 local organizing  committee      ann de meulemeester  biomedical library of the university of  ghent  ghent, belgium  email: ann2.demeulemeester@ugent.be    this was my first time participating in an   85 mailto:ann2.demeulemeester@ugent.be evidence based library and information practice 2009, 4:3  eblip conference, and i found that the  content and quality of the presentations  were a good reason to plan to attend again. i  also found stockholm to be a warm and  welcoming place to stay.     “information literacy” (il) was the track i  chose to follow throughout the meeting, as  this is a major research topic for the field of  library and information sciences to which i  hope to contribute actively in the future. the  different presentations on this topic focused  on various methods of teaching and  evaluating information literacy skills and  how to integrate the teaching of these skills  into a curriculum. the lectures were very  inspiring and i am sure they will help me in  my future research. i learned, for example,  that the assessment of information skills  depends on different factors and there is no  consensus yet about the best way to  evaluate the real literacy skills of our  stakeholders. developing methods for  evaluating information literacy, therefore, is  our first research challenge. however, il is  not just relevant in libraries; it is the  keystone of our learning capacities.  therefore, i would suggest that researchers  from the field of pedagogy could be  involved in future conferences or invited to  be keynote speakers.     throughout the conference, several librarian  colleagues mentioned how difficult it is to  get il in the student curriculum. it is worth  investigating the reasons behind this. some  potential research questions include: how  do il assessment tools from different  suppliers compare to one another? what are  the experiences with these tools? have they  been appropriately standardized? are they  comparable across languages? so many  unanswered questions underscore the  necessity for further, ideally  interdisciplinary, research on il.     at the end of the conference, the discussion  forum was rather disappointing. many  participants in the il track were eager to  contribute to the discussion, but we were  thrown into the session without any  preparation, reference texts, questions to be  solved in advance, etc. the fact that the  chairperson was only appointed at the  beginning of the session was another proof  of lack of preparation.      in spite of this minor disappointment, all of  the participants openly shared information  and experiences, as could be expected from  persons who develop their careers within  information sciences. the general spirit of  the conference was excellent, and the overall  feeling was great satisfaction.      virginia wilson  shirp coordinator   university of saskatchewan  saskatoon, saskatchewan, canada  email: virginia.wilson@usask.ca    i took a lot of photos in stockholm. as i  reflect on my third eblip conference  experience, i find images flipping through  my head like a photo slide show:    • jonathan eldredge sitting on a  beanbag chair watching madagascar  while waiting to meet someone  prior to the opening reception;  • lyn currie and denise  koufogiannakis casting long  shadows as we walk by the water  on our way to city hall;  • my husband darryl dancing with  my friend and former colleague, pat  moore, at the conference dinner at  beautiful skansen;  • the colourful and unconventional  conference bags designed and made  by 10 swedish designers that  caused such a buzz;  • pretty much everyone trying to  avoid me and my camera!  86 mailto:virginia.wilson@usask.ca evidence based library and information practice 2009, 4:3  my photos represent the way i experienced  the conference and the eblip community  that had gathered in stockholm.        poster session  photo courtesy eblip5 local organizing  committee      after the conference was over and everyone  went their separate ways, i thought about  the case of the warm fuzzies that developed  for me during the conference. what in  particular made me feel so happy, so  exhilarated, so enthused? the sessions, the  posters, and the opportunities for  networking and connecting were definitely  inspiring and thought‐provoking. the  chance to meet like‐minded colleagues from  around the world was an opportunity not to  be missed. andrew booth’s workshop on  implementing eblip was interesting and  invigorating. the food was fantastic. but it  was more than all that. my positive  experience seemed greater than the sum of  the conference parts.     when i got home from my travels, i thought  about the notion of belonging and the  feeling of community. poking around on  google (gasp!), i stumbled across mcmillan  and chavis’ theory of the sense of  community. perusing wikipedia (gasp,  again!), i discovered that the four elements  that make up a sense of community were  present for me in my eblip conference  experience.     the four elements1:  1. membership  2. influence  3. integration and fulfillment of  needs  4. shared emotional connection    my (informal) membership in the eblip  community gives me a sense of belonging  and identification. i am personally invested  in it, and the common symbol system (the  steps of practicing eblip) provides a  common language with which to  communicate with other members.  influence works both ways: i as a member  feel i have some influence in the  community, while i also feel influenced by  the community. the notion of influence is  necessary for group cohesion. a sense of  belonging to a community also provides a  feeling of integration. needs are fulfilled by  group involvement and acceptance. i feel a  shared emotional connection with the eblip  community. evidence based practice can be  hard work. having others who know the  challenges and the rewards boosts the sense  of community.    these four elements are dynamic and there  is movement between them. it is in this  movement that i can place my eblip5  conference experience. people from around  the world attended the conference because  of a shared interest in eblip and the desire  to better inform their individual practices  (element 3). the group members are bound  by the eblip structure and their desire to  learn more about it or to better incorporate it  1 “sense of community.” wikipedia, the  free encyclopedia. [please insert date of  access]  <http://en.wikipedia.org/w/index.php?title= sense_of_community&oldid=291089681>.  87 http://en.wikipedia.org/w/index.php?title=sense_of_community&oldid=291089681 http://en.wikipedia.org/w/index.php?title=sense_of_community&oldid=291089681 evidence based library and information practice 2009, 4:3  88 into practice (element 1). we, as a group,  present sessions and posters, attend  sessions, network, and connect based on our  shared interest in eblip (element 4). and  we work together to shape future  conferences and the group as a whole based  on our needs (element 2).   that’s the long way of saying the eblip5  conference has given me a sense of  community and a sense of place. i value that  in my professional life. i say a hearty “well  done” to the conference organizers, and a  big “thank you” to everyone who  contributed their knowledge, experiences,  and ideas. i can’t wait until eblip6!  news   evidence based library and information practice (eblip) is seeking an editorial intern      2018. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29454     we are looking for a new editorial intern to assist our editorial team. the ideal candidate will be in place by october 2018 and will commit to serving a two year term.    the role of the intern is to:    provide a final editorial check of proofed copy before publication, using the pre-developed checklist  check and edit the item metadata on the eblip website to ensure title, author, and abstract correspond to the submitted manuscript  assist the editor-in-chief and communications officer with calls for papers or calls for volunteers  assist with investigation and implementation of new projects related to the journal  participate in monthly editorial team meetings    the ideal candidate will be an mlis (or equivalent) student or a recent graduate (past 2 years) interested in evidence based practice and research methods, and who possesses strong attention to detail. the position requires dedicated time on a regular basis, and it is therefore essential that interested persons ensure available time to devote to this position prior to applying. it is estimated that the workload is approximately 10 hours per month.   interested persons should send a cover letter, indicating areas of strength they would bring to the role, and resume/cv as a single pdf file to lorie kloda, editor-in-chief, at lorie.kloda@concordia.ca by july 20, 2018. specific queries about the role should be addressed to the current editorial intern, rachel hinrichs at rhinrich@iupui.edu.     **please note that evidence based library and information practice is a non-profit, open access journal and all positions are voluntary and unpaid.   about the journal:   published quarterly, and hosted by the university of alberta, this peer-reviewed, open access journal is targeted at all library and information professionals interested in an evidence based model of practice. by facilitating access to librarianship research via original research articles and evidence summaries of relevant research from the library literature, evidence based library and information practice will enable librarians to practice their profession in an evidence based manner.   please visit the evidence based library and information practice web site (http://journals.library.ualberta.ca/index.php/eblip) for further information about the journal.   evidence based library and information practice evidence based library and information practice 2010, 5.2  evidence based library and information practice       article    the life of the space: evidence from nova scotia public libraries    francine may   assistant professor  mount royal university  calgary, alberta, canada  fmay@mtroyal.ca    fiona black  associate professor / director, school of information management  dalhousie university  halifax, nova scotia, canada  fiona.black@dal.ca    received: 5 aug. 2009          accepted: 12 april 2010       2010 may and black. this is an open access article distributed under the terms of the creative commons‐ attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐ sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original  work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is  redistributed under the same or similar license to this one.    abstract     objectives – to describe aspects of the 21st century role of the public library as a  physical space by observing the actual use of a selection of public libraries. this study  seeks to reveal how patrons are using and experiencing these institutions as spaces  and how patrons and staff characterize the role of public libraries in communities.     methods – a multiple case study design was used to examine three urban and three  small town public libraries within nova scotia, canada. a triangulated set of  methods including patron interviews and questionnaires, staff interviews, and  seating sweeps was used to develop answers to the research questions.    results – these public libraries are functioning as successful public places in that  they are community spaces used in a multitude of ways and where patrons feel  welcome. these libraries play important roles in the lives of respondents and, while  respondents were willing to give critical feedback, they generally described the  spaces positively. patron use and experience of these library spaces can be broken  into three themes that describe the roles of public libraries in communities. these  5 evidence based library and information practice 2010, 5.2  6 include the role of provider of books and information, provider of access to  technology and provider of a social space where members of the public are welcome.    conclusions – patron experiences in nova scotia public libraries show that libraries  are vibrant places that are highly valued by their communities. a number of common  themes about the use and perception of these spaces emerged, yet when examined  individually each library was also revealed to be a unique place, reflecting the  particular qualities of the community and the physical space of the library building  itself. it is clear that public libraries are complex institutions which play a variety of  valuable roles in the community.    introduction    libraries’ roles evolve in line with their  societal contexts, responding to community  needs as they develop. this paper explores  roles of the public library in the twenty‐first  century. ten years ago literature relating to  the digital age spoke of libraries without walls,  and discussion around the role of public  libraries in the digital age continues to the  present (for examples see chowdhury, poulter  & mcmenemy, 2006; waller, 2008; or wooden,  2006. throughout their history, public libraries  have always been in transition, but the  introduction of the web has caused significant  changes, including a drop in the intensive  ready reference activity at many public  libraries. some fear that the internet poses a  threat to libraries, while others see net access  as an opportunity to expand the roles of public  libraries in communities (waller, 2008, p. 378).  during this same period of change, many  library leaders have advocated increasing the  civic society role for public libraries. under  this rubric, new designs and renovations often  include meeting spaces and flexible layouts in  order to accommodate local community  interests in using the library as a public  commons. thus, it is timely to study what  happens within libraries to contribute to the  growing body of literature which  demonstrates the value of public libraries.     according to nova scotia’s standards for nova  scotia regional public libraries (2001), the “core  business functions” developed for regional  public libraries include several uses that are  often dependent on library as place (p. 5).  these range from the obvious, “provide  community meeting places and opportunities  for citizen participation,” to the more subtle,  “assist in the integration of new residents into  canadian society” (p. 5). public libraries can  and do support national and local  governments in achieving their social and  economic goals. public libraries around the  world are helping to provide affordable access  to technology (bertot, jaeger, langa &  mcclure, 2006; julien & helliwell, 2001; werle  & fox, 2007). it may seem somewhat ironic  then that, due in part to the digital revolution,  some government decision‐makers are not  convinced of the need to fund public libraries  beyond a bare minimum. public libraries also  support governments in achieving societal  goals such as assisting new immigrants to  settle into communities (picco, 2008, p. 45),  supporting early childhood literacy  development, strengthening the knowledge  based economy, promoting community equity  and inclusion, and empowering citizens to  participate in a civic society (newman, 2004,  p. 4‐6). recent studies provide strong evidence  of the economic contributions of public  libraries using a variety of methods (for  examples see manjarrez, cigna & bajaj, 2007;  steffen, lietzau, lance, rybiun, & molliconi,  2009). in addition, active public libraries have  been correlated with successful retail spaces  (blankinship, 2005, p. 44; singh, 1985, p. 154).  it is alarming, therefore, to note that in nova  scotia’s strategic document opportunities for  sustainable prosperity (2006), libraries are not  mentioned. this seems especially disturbing  given that education is mentioned frequently,  and public libraries in this canadian province  are administered within the department of  education.   evidence based library and information practice 2010, 5.2  lamonde, mcnally, & rotundo (2005) note  that public libraries are public places and form  a part of the culture in canada (p. 250). there  is an increasing awareness that public places  are under pressure as aspects of society  become more culturally consumerist due to  private interests  encroaching on some aspects  of public life. this results in an erosion of  public place (banerjee, 2001, p. 12; mitchell,  1995, p. 121; williamson, 2000, p. 182). public  libraries too are being buffeted between forces  that, on the one hand, draw them closer to the  marketing principles of the private sector, and  alternatively, which value the library as a  place of information and community record  that supports democracy (alstad & curry,  2003; buschman, 2004, p. 41). research by  leckie & hopkins (2002) illustrates that the  main threats to the role of public libraries as  neutral public spaces are an increasing need  for economic justification and the  encroachment of private interests (p. 360). in  such an environment it is critical for public  libraries to preserve and strengthen their roles  as public places. a first step in achieving such  a goal is to improve our understanding of how  users make use of library spaces. in turn, this  will help determine whether libraries are  functioning successfully as public spaces. the  information gathered through this research is  also important for assessing the success of the  design of library buildings and to inform  future design decisions, renovations and  service provision. these are all important  factors for libraries, especially considering that  one of the most highly rated core  competencies for librarians is to “create a  welcoming, useful, responsive library  environment to encourage use and strengthen  support of the library by community”  (helmick & swigger, 2006, p. 63).    public libraries in canada fall under the  jurisdiction of provincial governments and are  usually administered under the department of  culture or education. most funding for public  libraries comes primarily from a combination  of provincial and municipal sources (wilson  560‐561). the public library system in canada  is well established, with most canadians  having access to services and approximately  two thirds of canadians holding library cards  (newman, 2004). despite the important roles  that public libraries play in the lives of  canadians, there have been few studies  evaluating whether they are functioning  successfully as physical public spaces. this is  an important issue, considering that the role of  library as space is a key factor contributing the  perceived value that library patrons place on  library services (ladhari & morales, 2008).  within this context, this study seeks to explore  the public’s actual use of public library spaces  in nova scotia, canada, and thereby to  describe the community roles played by these  spaces. on a practical level this study  generates information useful for informing  design, renovation, and service provision  projects in public libraries. it also contributes  to the assessment of the roles public libraries  play in the development of sustainable  communities.     literature review    public libraries as physical places can play a  number of roles in communities. a public  library supports the intellectual, social,  cultural, and community needs of the local  citizenry. it is a physical space where  community activities (political, social,  educational, and recreational) take place and  where all members of the public are  welcomed. some predict that the community  roles of public libraries will become even more  important in the future (manjarrez, cigna, &  bajaj, 2007, p. 24). in addition, the purposes of  public libraries remain unchanged with the  digital age: to promote democracy, equality,  and social justice; to increase access to  information, disseminate culture and  knowledge; to contribute to meaningful  leisure time; and to act as a communal  institution and social meeting place, (aabø,  2005, p. 210). but how do public libraries fulfill  these purposes in an era of technological  change? digital technologies are changing the  configuration of communities, and public  libraries need to consider how they can serve  in that new configuration.     one important way that public libraries serve   7 evidence based library and information practice 2010, 5.2  communities is by providing public places  where individuals can meet and interact with  each other while participating in their greater  community. public libraries are used as  community meeting places in a variety of  ways, both formal and informal (aabø,  audunson, & varheim, 2010, 25). this role is  increasingly significant as factors that tend to  isolate sections of community, such as  multiculturalism and digitalization, take on  added importance. worldwide mobility has  resulted in increasingly multicultural  populations in europe and north america.  cross‐cultural communication is, therefore,  becoming ever more important to maintaining  tolerant and democratic multicultural  communities; without it, communities risk  becoming fragmented and isolated along  cultural lines (audunson, 2005, p. 432‐433).  there is a concurrent societal development of  the ‘information society’ by which the digital  and the virtual are opening up new ways of  communicating  while also providing the  means for groups to live increasingly  segregated from their geographic  communities. this development could have  important repercussions as democracy and the  political realm are anchored geographically  and key to successful democracies in those  geographic communities (audunson, 2005, p.  438). public libraries are meeting places that  facilitate interactions across these lines of  separation, which can increase communication  and tolerance. aligning the public library’s  goals to current social needs, such as resisting  trends of community fragmentation stemming  from multiculturalism and digitalization, will  revitalize the mission of the public library and  strengthen the role of the physical place of  public libraries (audunson, 2005, p. 430).    public libraries have been found to be  functioning successfully as public places in  canada. leckie & hopkins (2002) examined  the use of two central public libraries in 1999  to investigate the roles they played as public  spaces in toronto and vancouver. in part,  leckie & hopkins (2002) concluded that the  libraries were functioning as well used public  places, that they performed educational,  informational, and social functions in their  cities, and that they provided user space that  would be hard to duplicate in other settings  (p. 359).     public libraries function more successfully as  public spaces than privately owned publicly  accessible facilities. when comparing public  libraries and book superstores, mckechnie et  al., (2004) concluded that public libraries were  functioning more successfully as public spaces  due to their higher levels of inclusiveness and  more opportunities for user control and  participation (mckechnie et al., 2004, p. 50). in  that study, socializing occurred at all the  libraries, and strangers were observed talking  to each other more often than at bookstores.  while a diversity of ages and users of both  genders were observed at both types of  locations, libraries were found to have a  higher diversity of visible minorities. it was  found that, while people undertake many of  the same activities at public libraries as at  book superstores, patrons tended to treat  libraries more as their own space through  activities such as bending the rules, moving  furniture to suit their needs, spreading their  belongings out, staying for longer periods of  time, and engaging strangers in conversation  (mckechnie et al., 2004, p. 50). given that  many of the functions of public libraries stem  from the public library as a physical space, this  study explores the use of public library spaces.     aims    this study examined the use of six public  libraries in nova scotia, canada in the  contexts of their local communities. the goals  of this study were to assess how library users  actually used and experienced the physical  space of selected public libraries, and to  determine what that use revealed about the  roles these spaces played as public spaces.  more specific objectives related to these goals  included:  • investigating who used these  libraries;  • elucidating what activities  these users engaged in while  in the library; and  8 evidence based library and information practice 2010, 5.2  9 • exploring how library users  and staff described the roles  these libraries play in their  respective communities.   this study also investigated whether the  answers to these questions differed depending  on the unique context of each of the libraries  targeted in the study. the study thus aims to  contribute new knowledge on the critical  subject of public spaces and communities.    methods    this exploratory study examined three urban  and three small town public libraries within  nova scotia. the design of this study was  based on that used by leckie & hopkins  (2002) to study central public libraries in  toronto and vancouver. this study expands  the focus to include small town and branch  libraries to maximize the variability. although  commonalities did emerge when these  multiple cases were examined side by side,  this study did not aim to discover general  patterns that may hold true for all public  libraries. instead, it explored the roles and  uses of a small group of public libraries. the  case‐oriented approach focused on making  sense of a small group of cases and was an  appropriate method of analysis for this  exploratory study (ragin, 2001, p. 1519).     as with the leckie & hopkins (2002) study,  answers to the research questions were  developed through a triangulated set of  methods, including library user interviews  and questionnaires, staff interviews, and  seating sweeps. descriptions of library roles  and use obtained from these different methods  were examined for patterns and  contradictions. all methods, including  interview and questionnaire questions, were  based on those developed by leckie &  hopkins (copies of which are published with  that 2004 report). seating sweeps are a method  used to unobtrusively observe and record use  of a physical space (see given & leckie, 2003,  for a description of this method in libraries).  data collection took place during the spring  and summer of 2006. as this research was  undertaken for partial fulfillment of a masters  degree, timing was dictated in part by the  academic schedule. results were analyzed and  common themes describing the roles of these  spaces in communities were outlined and  compared to those obtained by leckie &  hopkins (2002).     selection of study libraries    the libraries were selected based on their  diversity in terms of the communities in which  they were located, the physical facilities, the  services provided, the population served, and  their geographical proximity (figure 1). the  study included four libraries from the halifax  public libraries system, three of which are  within the urban municipality: halifax north  memorial, keshen goodman, and spring  garden memorial. the fourth branch,  tantallon, is located in a small community just  outside of halifax. the other two libraries  were new glasgow public library, located  approximately 64 kilometers north of halifax,  and lunenburg public library, located  approximately 128 kilometers south of  halifax. when examined together, these six  libraries give a broad overview of the roles of  public libraries as spaces in nova scotia  communities.     library user questionnaires    the library user questionnaire (see appendix  1) comprised 35 questions and was left for a  period of one week on reference and  information desks in all six participating  libraries for library users to complete  voluntarily. completion of questionnaires was  limited to individuals over the age of majority.  the location of the four halifax libraries  relative to the home of the researcher allowed  for more numerous visits to these libraries. in  these cases, in addition to being left at the  reference and information desks,  questionnaires were also distributed to all  library users in these branches over a three  hour period in the early afternoon on one day.  in total, 271 questionnaires from all six  libraries were returned. these methods of  questionnaire distribution have several  limitations which, to some extent, weaken the  evidence based library and information practice 2010, 5.2                                                  fig. 1. map of library locations  strength of the results obtained. respondents  were self‐selected; in‐person distribution of  questionnaires took place only at a selection of  participating libraries and for a limited period  of time (thus potentially missing patrons who  used the library at different periods of the day  or week); and respondents were not  monitored for duplicate responses (i.e.,  respondents filling out more than one  questionnaire).     library user interviews    in total, seventeen library users (two from  lunenburg and three each from the remaining  five libraries) were interviewed by the  researcher during the afternoon of one day at  each library. selected interview candidates  alternated between male and female library  users, and were selected from different areas  of the library. interviews consisted of eleven  questions and were recorded as field notes  (see appendix 2). interview candidates who  had previously completed a library user  questionnaire were excluded from interview  participation. the respective ages of  interviewees were not recorded and only  adults were interviewed. the relatively small  number of interviews at each library and the  limited timing of the interviews results in a  very limited cross‐section of patrons  interviewed.     library staff interviews    a total of 20 staff members were interviewed  by the researcher, ranging from 2–5 staff from  each library depending on the number of total  staff members at that library (see appendix 3).  invitations to participate were extended to  selected staff members based on  recommendations from library management.  at all libraries but spring garden,  management suggested specific employees to  be contacted for interviews. in contrast, spring  garden management suggested a general  email be sent to invite volunteers to be  interviewed. from that volunteer list a  selection of volunteers were selected to  represent a range of service areas. staff  members were informed that the interview  was entirely voluntary.     10 evidence based library and information practice 2010, 5.2  seating sweeps    seating sweeps were used to make  unobtrusive observations, where possible, of  all individuals in the libraries. in crowded  areas of the library, unobtrusive observation  was not possible. this was an issue  particularly in meeting or program rooms and  in some children’s play areas. in these cases,  individuals were counted but no other data  were recorded. data were collected by  walking through the library recording  observations of library users, including  approximate age, sex, possessions (e.g.,  backpack, laptop, baby carriage, etc.),  activities (e.g., reading, talking, eating) and  where the user was located in the library at the  time of the observation. sweeps were  conducted by the researcher three times,  morning, afternoon and evening, on one  weekday and two times, morning and  afternoon, on a weekend day (all libraries in  this study were closed on weekend evenings)  at each study library.     results    about the libraries    each of the six libraries is unique in the  population it serves and in the qualities and  features of the physical space. the three urban  public libraries located in halifax belong to the  halifax public libraries system. halifax north  memorial public library (hn) is a recently  renovated urban public library located in a  lower income, inner‐city neighborhood with a  high level of unemployment (see appendix 4  for additional population characteristics for  each of the study libraries). the attractive  keshen goodman public library (kg), one of  the more recent additions to the halifax public  libraries system, is built in an open concept  design with high ceilings, and includes a café  and many computers. this library serves one  of the largest immigrant populations and the  smallest proportion of residents who report  english as their mother tongue (though  english speakers still make up the vast  majority) of the library user groups in this  study. the final urban public library, spring  garden memorial public library (sg), is a  central library. the population in the  immediate catchment area has the highest  average education level and the highest  percentage of young adults of all of the  populations served by the libraries in this  study. in many ways this library has  outgrown its building, and its replacement is  the number one planning and facilities priority  for the halifax public libraries system  (halifax public libraries, 2004). an  examination of the planning project for the  new library gives an idea of the need for a new  space. the proposed new library will be  approximately 2.5 times the size of the current  building, will offer almost twelve times the  number of computers, and almost six times the  seating (halifax public libraries, 2008).    the three small‐town libraries examined are  located in different areas of the province and  vary greatly in terms of populations served.  located in a newly expanded and renovated  space, lunenburg public library (lb) serves  the smallest population of the six libraries  studied. lunenburg is a picturesque town  designated as a unesco world heritage site,  and  is a popular retirement community with a  population that is, on average, older than that  of the communities served by the other  libraries in this study. new glasgow public  library (ng) is a small town library serving a  working class community. though the  population served by this library has the  lowest unemployment rate of the libraries in  this study, it also has one of the lowest income  levels and highest rates of lone parent families.  finally, tantallon public library (tt), another  one of the newer branches within the halifax  public libraries system, serves a small town  just outside of the city of halifax. the highly  educated population served by this library has  the highest average income, the lowest  number of unemployed, and the lowest  number of lone parent families of the six  libraries.    about the library patrons     patrons at these libraries represent both the  diversity and similarities found in their  11 evidence based library and information practice 2010, 5.2  12 respective communities. for example,  questionnaire respondents varied greatly in  their ages and in their levels of educational  achievement, but very little in terms of  languages spoken. matching the demographic  data fairly closely, library patrons were  generally well educated, with 56% holding an  undergraduate degree or college diploma or  graduate degree (table 1). another 21%,  mostly representing students, reported having  some university or college qualifications. the  most common primary occupation reported  was that of student (22% of questionnaire  respondents), followed by retired persons  (16%), those working in social services,  education, government service or religion  (14%) or homemakers/full‐time parents (13%)  (table 2). questionnaire respondents  represented a variety of age groups, and when  all responses are taken together, each of the six  adult age categories provided on the  questionnaire were fairly equally represented.  although an equal number of men and  women were observed in the libraries, more  women than men filled out questionnaires:  61% of questionnaire respondents were  women, 36% were male, and 3% did not  respond to the question.     in keeping with the demographic information  for these libraries, there was not a high rate of  diversity amongst questionnaire respondents  in terms of languages spoken. three quarters  of questionnaire respondents reported being  unilingual english speakers, sixteen percent  also spoke french and, two percent of  respondents indicated that they spoke a  language other than french or english.  besides french and english, seventeen  different languages were reported as being  spoken. patrons at kg had by far the largest  diversity, with twelve different languages  reported by questionnaire respondents there,  the next highest numbers were at hn and sg  each of which had six. staff recognized the  relative diversity of patrons at kg, with one  staff member describing the library as “a  cultural centre”.      patron activities and locations in the library    the seating sweeps, questionnaires, and  interviews all indicate that patrons were using  the libraries for a variety of activities and  purposes. when asked in the questionnaires  about their primary reason for visiting the   library, the most common response by patrons  at all branches was to borrow or return  materials for themselves. patrons also  frequently reported coming to the library to  browse, use the internet, look for information,  or read. when interviewed, library staff  indicated that patrons in the library engage in  a variety of activities, including accessing the  collection, looking for information, socializing,  and using the computers. in all of the libraries  studied, large and well‐used toy collections  gave evidence of children using the library as  a place to play.      table 1  highest level of educational achievement   educational achievement  questionnaire respondents (%)  (n = 271)  less than a secondary diploma  8  secondary diploma  11  some university or college  21  university or college diploma  34  some post‐graduate  7  university post‐graduate degree  15  no response  4        evidence based library and information practice 2010, 5.2  table 2  primary occupation  primary occupation*†  questionnaire respondents (%)  (n = 271)  student   22  retired  16  social services, education,  government service, religion  14  homemaker or full‐time parent  13  art, culture, recreation or sport  11  business, finance, administration  8  health  8  unemployed  5  sales and service  4  trades, transport, equipment  operator  3  management  2  processing, manufacturing, utilities  1  natural and applied science  ‐  primary industry  ‐  other  11  no response  3  * a number of respondents selected more than one primary occupation.  † occupation categories are based on the national occupational  classification list.      when observations of both individuals and  groups are taken together, they show the use  of computer workstations (which provided  internet, library catalogue, word‐processing,  and database access) to be by far the most  popular patron activity at all libraries, with an  average of 32% of patrons engaging in this  activity during seating sweeps (figure 2).  following closely in second and third places,  patrons were either physically searching or  browsing the library collection (16% of patrons  observed), or in conversation (15% of patrons  observed). the remaining groups of patrons  observed were reading (12%), interacting with  staff (10%), or watching/sitting (5%).     when these numbers are broken down by  branch some interesting differences emerge.  although the most commonly observed  patron activity at all libraries was computer  use, the percentage of patrons engaged in that  activity at hn was much higher than the  overall average (61% of patrons observed  compared to the overall average of 32%).  patrons at this library were also more likely to  be observed talking than at other libraries  (33% of patrons observed), but less likely to be  observed in the stacks or browsing (6% of  patrons observed). lb patrons were more  likely than patrons at other libraries to be  observed interacting with staff (an average of  24% of patrons observed), but were tied with  hn for the lowest percentage of patrons  observed reading (6% of patrons observed).  an examination of the observed locations of  patrons in the libraries shows both variations  and similarities. as could be predicted based  on the observations of patrons activities, the  most commonly observed patron location at  all libraries was at computer workstations  (figure 3). the children’s area was clearly the  second most observed patron location at kg.  at the other libraries the trend is not so  obvious, with most libraries having a variety  of second most observed locations. other  popular locations at these libraries include the  stacks, staff desks, and worktables.  13 evidence based library and information practice 2010, 5.2                                                    fig. 2. seating sweep – most popular patron activities by library                                                  fig. 3. seating sweeps – most popular patron locations by library 14 evidence based library and information practice 2010, 5.2  the number of companions per library user  per visit appears to vary by library (figure 4).  while nearly three quarters of questionnaire  respondents at sg reported usually coming to  the library alone, respondents from kg  goodman are far more likely to be  accompanied, with just over 60% having one  or more companions during a usual library  visit. questionnaire respondents at hn are  split with just under half usually coming to the  library with a companion and just over half  usually coming alone. in terms of length of  stay, questionnaire respondents can be broken  into three groups, those who quickly come  and go from the library, those who stay  between 30 and 60 minutes and those who   stay longer than an hour (figure 5).     patron experiences in library spaces    patrons made many comments about the  physical spaces and the general library  environment. positive comments were made  about the seating, the general environment,  the arrangement, the décor, and other aspects  of the libraries. criticisms were leveled at the  furniture, design, lighting, size, and the  arrangement or organization of the libraries.  patrons also criticized the noise or the lack of  cleanliness, but praised the spaciousness,  natural lighting or calm atmosphere.                                                           fig. 4. patron questionnaires – usual number of companions during library visits              15 evidence based library and information practice 2010, 5.2                                                      fig. 5. patron questionnaires – usual length of stay during library visits      when patrons were asked to describe the best  feature of their library, the most common  responses related to staff, convenience, the  physical space/environment, the collection, or  access to technology. favourite features  included “the totally comfortable  atmosphere,” the “friendly, helpful staff,” the  “self‐serve checkout,” and the “computers for  children.” comments on the worst feature of  the library fell into some of the same  categories, with complaints about the  collection, the physical space/environment, the  services, and available technology. others  complained about inadequate parking, the  limited hours, and noise levels. patrons  complained that “the collection could be better  – more current books,” that there “was no  place to relax and read before borrowing,” and  that there was “not enough staff support to do  all the programming they’d like to do.” as  might be guessed from their willingness to  identify deficient features of the library,  patrons had a clear idea of improvements they  would like to see introduced, such as  improving collections, improving facilities,  introducing new services and programs, and  expanding technology. despite the criticisms  and suggestions for improvement,  questionnaire respondents generally agreed  on the importance of the library:  approximately half of respondents at all  libraries indicated that it would have a  considerable or major impact on their lives if  they no longer had access to the library.    some of the communities use their libraries in  unique ways. patrons at lb have a high degree  of social interaction with library staff. one  questionnaire respondent reported talking to  staff about “books, the world, indigestion,”  and another about “anything, girl stuff – hair,  clothes, relationships.” staff at lb confirmed  that the library is a place of social interaction.  one staff member reported that some people  16 evidence based library and information practice 2010, 5.2  come to the library “as part of their daily  routine” and that the library provides “a social  contact with someone who is friendly and  approachable.” at hn, staff describe how  many of the library’s regular patrons come  and go from the library multiple times per  day.     patrons and staff describe the role of public  libraries in community    patrons value the library primarily as a place  to access books and information. when asked  to choose the most important library service  from a list, the top two services selected were  the provision of fiction (39% of questionnaire  respondents) and non‐fiction materials (28% of  respondents) (table 3). this was true for all  libraries except lb, where these services were  among the top three. access to technology was  the third most frequently mentioned service  (mentioned by 19% of questionnaire  respondents). respondents at lb (13%) and tt  (25%) were more likely to value the library as  a place to socialize when compared to  respondents from other libraries. in  comparing sg and kg, the largest libraries in  the study and those with the greatest number  of questionnaire respondents, some key  differences emerge. respondents at kg place  more importance on the library as a place to  borrow fiction and as a place to study or  socialize, while respondents from sg were  more likely to place a higher value on the  library as a place to access technology or to  read. it is interesting to note that, although  seating sweep results indicate computer use to  be by far the most common patron activity in  the library, this service appears to be  significantly less important to questionnaire  respondents than the provision of reading  material.          table 3  most important library services (most common responses only)  percentage of respondents*  urban libraries  small town libraries     hn  (n = 39)  kg  (n = 90)  sg  (n = 88)  lb  (n = 16)  ng  (n = 26)  tt  (n = 12)  total  a place to borrow fiction  / literatures (i.e.  recreational reading)  38  47  31  56  38  33  39  a place to borrow non‐ fiction / informational  material  23  24  32  19  27  50  28  access to technology  13  14  26  31  15  8  19  a place to read  8  10  19  19  12  8  13  a place to use reference  materials and  information services  10  11  11  13  12  25  12  a place for life‐long  learning (educational  support)  8  13  14  6  4  17  11  a place to study  (homework, research,  etc.)  8  10  5  6  12  8  8  a place to socialize  0  7  1  13  8  25  5  * some respondents selected more than one response  17 evidence based library and information practice 2010, 5.2  questionnaire respondents were also asked to  choose the least important library service that  the library provides for them. these results are  interesting when compared against library  services selected as the most important (figure  6). in seeing the library as a place to socialize,  questionnaire respondents appear to be in  general agreement with very few indicating  this service as most important but nearly half  indicating it as the least important. this is  generally in line with seating sweep data  which shows conversation to be the third most  common patron activity observed (a number  that is likely low due to the inability to record  activities for patrons who were in larger  groups). other areas where questionnaire  respondents seem to generally agree in terms  of the importance of the library a place: to  access fiction; to access non‐fiction; and to  access reference material and information  sources and services. several services  provided by the library appear to be viewed  neutrally, with very few individuals indicating  them as either the most or the least important  library service, including the services the  library provides as place for community  events and a place to retrieve community  information. technology appears to be a point  on which questionnaire respondents are  divided, with just under 20% indicating this as  the most important service and just fewer than  15% as the least important service.                                                                                      fig. 6. patron questionnaires – most important and least important service the library provides for  you                      18 evidence based library and information practice 2010, 5.2  although questionnaire respondents appear to  place a low rating on the role of the public  library as a place to socialize, staff see the  library as fulfilling a variety of social functions  in the community, including facilitating  learning, strengthening the community’s  social fabric, and providing a safe, shared  environment. staff members mentioned  several different groups for whom the public  library was especially important as a social  space, including seniors, new comers to the  community, families, teens and individuals  from lower socio‐economic categories. staff  described the library as “a meeting place  where parents meet up with other parents,”  where “kids learn skills of playing with other  kids,” and where “seniors meet for coffee and  [chat] about books.” some community  members do not have many opportunities for  social contact, and library staff also identified  this as a role for the library. one staff member  commented that the library was a place where  newcomers to the community could “develop  trust and see how they [could] fit into the  community.” staff described libraries as  providing a supportive environment for  children who are just beginning to explore  their place in community, and one staff  member pointed out that children in the  library are often having their first significant  interaction with an adult other than their  parents. the importance of the library as a safe  space where people can meet and where  people who might not have anywhere else to  go can feel welcome was also highlighted by  staff.     when asked about the main purpose of the  library, questionnaire respondents were most  likely to describe the main purpose of the  library as revolving around the provision of  information or research or supporting  education and learning. it is therefore  surprising that more questionnaire  respondents did not indicate these library  services as the most important service for  them. a number of respondents also  mentioned the role of the library as a place to  support reading and literacy, to provide access  to technology and as resource for leisure  activities or entertainment may (2009) is a  good resource for further analysis of  questionnaire responses from the three urban  public libraries. it was interesting to note that  patrons at halifax north were more likely to  mention the role the public library plays in  supporting the community and facilitating  community building than patrons at other  libraries. patrons commented that “halifax  north is a community library, it is a gathering  place, a place to study, a place to rest,” and  that the purpose of the library is not only “to  provide resources for the community but to  act as a social and community space for  activities.”    similar to questionnaire respondents, staff saw  the library as having a variety of purposes.  however, they were more likely to describe a  broader purpose for public libraries and to see  that purpose as something that was not static,  but one that evolves with the needs of  community. although library staff frequently  mentioned similar utilities to those mentioned  by questionnaire respondents, they also saw  the purpose of the library more generally as a  place that served the community and  supported local culture. some staff members  commented that, as places, public libraries  were evolving and changing. pointing to the  ways that libraries evolve as they adapt to  community needs, one staff member indicated  that “libraries are not the quiet sanctuaries  they used to be,” and another explained that  the goal is to make libraries “more welcoming  and less intimidating place[s].”       discussion    there are certain challenges involved with  measuring library use, resulting in several  limitations to this study. though patrons were  observed using the library computers, no  information was recorded concerning the  purpose for which they were being used. the  seating sweep method has several limitations:  though it works best in areas that are not  overly crowded, it only records library use  that occurs physically in the branch, and  excludes the library’s considerable online  presence. finally, seating sweeps only capture  19 evidence based library and information practice 2010, 5.2  use during a select period of time, and is  therefore unable to account for the fact that  some days of the week in the library may be  busier than others, as well as fluctuations from  one month to the next. while this study took  place mainly during the summer months,  other seasons of the years may or may not  have the same patterns of use. the  supplementary data gathered through  questionnaires and interviews helped to  mitigate some of these limitations.    library patrons have different, sometimes  contradictory ideas about the role of a public  library. an analysis of the topic of library  seating illustrates some of the very different  roles that library users assign to library spaces.  some patrons stated a preference for seating in  central areas of the library, others for soft  seating, for seating tucked away in the corner  of the library, for seating in or near the stacks,  or for seating at tables or study carrels. from  these very different preferences it may be  concluded that, depending on whom you ask,  the library can be a place to observe  community and feel a part of it, a place to  relax in comfort, a place to find solitude, a  place for prolonged browsing, a place to study  or a place to work, or a combination of these.  although libraries are generally successful in  accommodating this diversity of activities,  tensions do arise. the most common  complaint made by patrons was in reference to  noise levels. a staff member from halifax  north commented that sometimes there were  “tension[s] between younger people and  adults, each with ideas about what a library  is.” nevertheless, all of the libraries in this  study were successful in accommodating a  wide range of patron activities.     library patrons feel strongly about their  libraries and exhibit attachment to these  spaces by their willingness to share both  flattering and critical opinions about library  services and features. similar to patrons at the  vancouver and toronto central public libraries  (leckie & hopkins, 2002, p. 350), nova scotia  patrons, tend to describe the library spaces  positively, even while offering suggestions for  improvements.   although the results of this study indicate a  great deal of diversity in the use of library  spaces, a number of common themes  describing the role of public libraries in nova  scotia did emerge. libraries are still strongly  associated with their traditional roles of  providing books and information, but results  also indicate that the library’s role as a  provider of access to technology is  increasingly important. there is also strong  evidence that these public libraries are playing  an important role as community social spaces,  a role that necessarily incorporates many types  of activities. although there are differences in  the data collected for each library included in  this study, there were no significant  differences between the small‐town and urban  libraries in terms of how patrons were using  and experiencing the various libraries as  physical spaces.     a place to borrow books and find information  libraries in north american communities are  playing an increasing number of roles, moving  beyond their traditional roles as information  providers and education supporters. newman  (2004) notes that newer functions include  providing access to technology, empowering  citizens to participate in civic society, and  promoting equity and inclusion in community  (p. 4‐5). some library staff members described  the physical place and overall environment in  public libraries as evolving, but this study  shows that questionnaire respondents and  staff alike continue to value public libraries  primarily for their traditional roles of  providing reading material and access to  information. this is reflected in how  respondents and staff describe the primary  purpose of the library and in respondents’  reported reasons for visiting the library. a  comparison of the results of this study with a  study conducted by leckie & hopkins (2002)  reveals that patrons from both studies have  similar beliefs regarding: the main purpose of  the library — to provide information and  support education; the most important library  service — to provide access to information and  literature; and the reason for their visit to the  library that day — top reasons include to  20 evidence based library and information practice 2010, 5.2  borrow or return materials and to look for  information (p. 351). indeed, research has  found that, in canada and internationally,  libraries continue to be associated primarily  with books (online computer library centre,  2005, p. 3‐31) and that in many cases the  association of libraries with books and  information is one that is positive (p. 3‐21).    with this in mind, it is somewhat surprising  that observations of the locations and activities  of patrons in this study revealed a relatively  low proportion of patrons located in the book  stacks or engaged in the process of physically  browsing or retrieving information (figures 2  and 3). these numbers may be partially  explained by the fact that patrons are able to  browse electronically, an activity that does not  register on the seating sweeps. there are no  comparable numbers for canada, but research  by griffiths & king (2008) in the united states  indicates that though they have not altogether  replaced in‐person visits, online library visits  are a very common way of accessing the  library (p. 17).    in an age of increasing pressure on public  libraries, it may be valuable for library  management to remain aware of the fact that,  though users engage in a variety of activities  while in the library, many continue to value  libraries for their traditional role as providers  of books and information. public libraries can  build on the positive association between  libraries, books, and information by  continuing to improve and develop their  services along traditional lines while also  supporting a range of other activities.     a place to access technology    public library computers are extremely  popular with patrons. curry’s research data  (2002), albeit somewhat dated, reveals that in  2002, when canadians accessed the internet at  the public library they did so for a wide  variety of purposes, with email being the most  common (p. 54). bertot, mcclure, & jaeger  (2008) show that free computer access in  public libraries brings many benefits to  communities, which is perhaps why computer  use appears to have increased considerably in  the years since leckie & hopkins (2002)  collected and published their data (p. 346). in  that study, reading was the most common  patron activity, whereas slightly fewer than  fifteen percent of patrons were located at  computer workstations. during the course of  the study under discussion an average of  thirty seven percent of patrons were observed  using computer workstations (figure 3). it  seems contradictory then, that though patrons  make heavy use of library computers,  computer access is not the library service that  they most highly value, nor is it the main  service or purpose that draws them to the  library. there may be a number of reasons for  this apparent dichotomy of opinion and  observed use. library patrons may have  access to computers from other sources, and so  this function of the library is not seen as  critical. it may be that patrons use the  computers for purposes that they do not rate  as important, such as entertainment, and  consequently they do not place a high value  on this library service. finally, it may simply  be that, no matter the activities patrons are  actually engaging in when in the library, they  still conceive of the library in its traditional  role as a provider of books and information.    a social, community space    people feel comfortable in public libraries and  use them as social spaces. patrons’ frequent,  often prolonged visits, the socializing  described by patrons and staff, and the  descriptions of the library as a safe space, all  point towards this conclusion. by allowing  their children to play in the library, patrons  may also be indicating that they feel  comfortable in these spaces and are treating  them as their own (mckechnie et al., 2004, p.  46). there is some evidence that newcomers to  the community, including immigrants, feel  comfortable at public libraries, as reflected by  the comments from the staff at kg, one of  whom commented that the library was “a  place where people develop a lot of trust and  see how they can fit into the community.” one  staff member reflected that when patrons meet  at the library it is different than meeting at  21 evidence based library and information practice 2010, 5.2  other public places such as the grocery store,  because patrons feel comfortable to meet for  longer and sit down.     although all libraries under study are social  spaces, that social life takes on unique  characteristics depending on the individual  library branch. lb, for example, is an  important place for social interaction,  particularly between patrons and staff.  patrons at this library have an unusually high  degree of social interaction with library staff,  and reported talking to library staff about an  astonishing array of topics ranging from  library services to personal and world issues.  the staff of lb library appear to play a unique  role as points of social contact for the  community. with its older population,  lunenburg may have a higher number of  retired persons seeking social connection, and  this library is able to meet this community  need due in part to its small size. the hn  library too, showed unique social and usage  characteristics. patrons at hn place greater  value on the library as a community space,  visit longer, and are more likely to engage in  conversation or use the computers, but are less  likely to read or browse. based on the data  available, it is difficult to completely explain  why hn is being used in this unique way. in  part, these patrons appear to be using the  library as an extension of their living room,  similar to findings reported by leckie &  hopkins (2002, p. 353). some of these use  patterns may also be related to the high rate of  unemployment in this community, where  going to the library could be replacing going  to work for some people.     how can public libraries use this information?    advocate for the public library and  publicize the ways they contribute to  communities.     advertising and highlighting the multiple  ways that they contribute to community is an  important step public libraries can take to  ensure that they are valued and understood by  their communities as more than providers of  books and information. although some are  becoming more aware of the multiple roles  played by libraries in community,  hillenbrand’s review of the literature (2005)  concludes that most community, government  and academic groups are unaware of the social  roles played by public libraries (p. 10).  hillenbrand summarizes her arguments by  claiming that it is not enough that public  libraries are playing these roles in their  communities, but that libraries should also  make sure everyone knows they are playing  them. it is vital that public libraries adopt  advocacy as an effective strategy for  publicizing the multiple roles they play in  community. linda cook, director of  edmonton public libraries and former  president of the canadian library association,  advocates strongly for public libraries in her  community, with the result that the edmonton  public libraries has since seen a major  increase in community support and  recognition (canadian library association,  2004).    in an era of uncertain funding, public library  management will find it valuable to  demonstrate their continuing community  relevance and importance by illustrating how  their library services meet community needs  and support the objectives of partner groups  and funding agencies. the canadian  association of public libraries’ library  advocacy now! training document (2009)  advises public libraries, in their advocacy  campaigns, to market issues as an effective  way of influencing a target group (p. 4). public  library advocates can strive to demonstrate to  partners and funders how the public library  can assist them in achieving their agendas.  public libraries provide communities with  affordable internet access and this study  demonstrates that this service remains popular  with library patrons. providing canadians  with equitable access to the internet is an  ongoing concern of canadian governments  according to the now defunct connecting  canadians program (as cited in julien &  helliwell, 2001, p. 115) to the current  government of canada (n.d.) broadband  program, part of the governments overall  economic action plan. the results of this study  22 evidence based library and information practice 2010, 5.2  also indicate that public libraries may offer  assistance to governments in their efforts to  provide support to immigrants who are  attempting to settle into canadian  communities.     preserve traditional services but  accommodate a wide range of activities.     results reported here indicate that public  libraries would benefit from continuing to  preserve their traditional roles and services,  including maintaining quality collections.  even though patrons were observed to be  engaged in a wide variety of activities in the  library, questionnaire respondents continue to  value the library primarily as a place to access  books and information. traditional library  services appear to serve a community need  and do attract patrons to the library. yet it is  clear that libraries are being used for much  more than just books and information, and  also that public libraries would benefit from  maintaining spaces that accommodate a wide  variety of activities. by creating attractive  spaces that accommodate multiple uses, public  libraries can continue and expand their roles  as valued public places in communities.     use this information to inform the design  and organization of library spaces.     the physical spaces of these public libraries  were important to library patrons.  questionnaire respondents who described  their likes and dislikes regarding the physical  spaces of their libraries  often noted favourite  areas in the library and commented on the  furniture, layout, design, and overall space.  although patrons were unanimous in the  importance they placed on physical space,  they were not united in what they wanted  from that space. library seating is a case in  point, although more seating was a common  request made by those who provided  responses, respondents had very different  ideas on the kinds of seating which would  meet their needs. library managers should  keep this diversity of preference in mind when  designing and organizing the physical space  of libraries.  conclusion    public libraries play a variety of roles in their  respective communities. although, still  strongly associated with the traditional role of  providing access to books and information,  libraries have also adopted the role of  provider of access to technology. beyond  access to services and resources, the public  library is also important as a public space in  itself. public libraries are central places where  members of the community feel welcome, and  this role may be particularly important in  disadvantaged communities. public libraries  are accommodating spaces that  simultaneously meet a variety of patron needs  and that adapt to changes in society. in a  society where access to information  technology is of critical importance and true  public spaces are hard to find, nova scotia  public libraries are responding to important  social needs. although this study was situated  in nova scotia, the conclusions are likely also  applicable to public libraries at an  international level as these are issues that face  all public libraries.    while exploring some aspects of the roles of  public libraries in communities, this study also  points the way to new research questions.  what roles do libraries serving  underprivileged communities play as  community spaces? are the use and value of  public library spaces valued differently by  communities depending upon community  income and employment levels? what roles do  small public libraries play in supporting the  social needs of the elderly?    to summarize, what are public libraries as  places? in the words of patrons, are they just  “a place to pick up holds,” a place for “people  to meet each other,” a place that “foster[s]  literacy and citizenship,” or does their  traditional role, “the obvious – booklending,”  take priority? the library is all of these things,  depending on whom you ask. the challenge  for libraries lies in the task of creating and  maintaining a flexible space, a place that is  many things to many people, one which will  evolve with the needs and demands of  23 evidence based library and information practice 2010, 5.2  community and society while continuing to  serve its core function as a provider of books  and information.      acknowledgements     it is our pleasure to thank the many people  who made this study possible including dr.  elizabeth kelly, mr. michael colborne, dr.  ann curry, robin illsley, and eric  mostrovitch. we would also like to thank the  staff of the libraries where this research took  place and the dalhousie university faculty of  graduate studies for providing funding in  support of this research.  references  aabø, s. (2005). the role and value of public  libraries in the age of digital  technologies. journal of librarianship  and information science, 37(4), 205‐211.   aabø, s., audunson, r., & vårheim, a. (2010).  how do public libraries function as  meeting places? library & information  science research, 32(1), 16‐26.   alstad, c., & curry, a. (2003). public space,  public discourse, and public libraries.  libres, 13(1). retrieved 12 feb. 2010  from  http://libres.curtin.edu.au/libres13n1/p ub_space.htm  audunson, r. (2005). the public library as a  meeting‐place in a multicultural and  digital context: the necessity of low‐ intensive meeting‐places. journal of  documentation, 61(3), 429‐41.   banerjee, t. (2001). the future of public space:  beyond invented streets and  reinvented places. journal of the  american planning association, 67(1). 9‐ 24.  bertot, j.c., mcclure, c.r., & jaeger, p.t.  (2008). the impacts of free public  internet access on public library  patrons and communities. library  quarterly, 78(3), 285‐301.   bertot, j.c., jaeger, p.t, langa, l.a., &  mcclure, c.r. (2006). public access  computing and internet access in  public libraries: the role of public  libraries in e‐government and  emergency situations.ʺ first monday,  11(4). retrieved 12 feb. 2010 from  http://firstmonday.org/htbin/cgiwrap/ bin/ojs/index.php/fm/article/viewartic le/1392/131  blankinship, d.g. (2005). letʹs go to the mall.  library journal, 130(2), 44‐46.  buschman, j. (2004). staying public: the real  crisis in librarianship. american  libraries, 35(7),  40‐2.  government of canada. (n.d.). broadband  program. canadaʹs economic action  plan. retrieved 12 feb 2010 from  http://actionplan.gc.ca/initiatives/eng/i ndex.asp?mode=3&initiativeid=96.  canadian association of public libraries.  (2009) library advocacy now! a training  program for public library staff and  trustees. canadian library association.  retrieved 12 feb. 2009 from  http://www.cla.ca/divisions/capl/libra ryadvocacynow.pdf  canadian library association. (2004). linda  cook named recipient of  capl/brodart outstanding public  library service award. canadian  library association. retrieved 12 feb.  2010 from  https://www.cla.ca/am/template.cfm ?section=home&template=/cm/co ntentdisplay.cfm&contentid=4438    chowdhury, g, poulter, a., & mcmenemy, d.  (2006). public library 2.0: towards a  new mission for public libraries as a  ʹnetwork of community knowledge.ʹ  24 evidence based library and information practice 2010, 5.2  online information review, 30(4), 454‐ 460.  curry, a. (2002) what are public library users  viewing on the internet? an analysis of  the transaction logs of burnaby,  brantford, calgary, winnipeg, and  halifax public libraries. industry  canada, 27 dec. 2002. retrieved 29 jul.  2009 from http://epe.lac‐bac.gc.ca   given, l.m. & leckie, g.j. (2003). ʹsweepingʹ  the library: mapping the social activity  space of the public library. library and  information science research, 25(4), 365‐ 85.   griffiths, j. & king, d.w. (2008). the imls  national study on the use of libraries,  museums and the internet. institute of  museum and library services.  retrieved 10 jan. 2008 from  http://www.interconnectionsreport.or g/  halifax public libraries. (2008). proposed  central library. 2008. retrieved 15 jan.  2009 from  http://www.halifaxpubliclibraries.ca/c ontent/view/174/366/  halifax public libraries. (2004). needs  assessment and master facilities plan:  final report. retrieved 15 jan. 2009  from  http://halifaxpubliclibraries.ca/mediar oom/terrain_report.pdf  helmick, c., & swigger, k. (2006). core  competencies of library practitioners.  public libraries, 45(2), 54‐69.   hillenbrand, c. (2005). public libraries as  developers of social capital. aplis,  18(1), 4‐12.   julien, h, & helliwell, m. (2001). libraries as  instruments of information policy: the  role of canadian public libraries in  ʹconnecting canadians. in proceedings  of the canadian association for  information science annual conference,  quebec city, quebec, 27‐28 may 2001.  retrieved 12 feb. 2010 from  http://www.cais‐ acsi.ca/proceedings/2001/julien_2001.p df  ladhari, r, & morales, m. (2008). perceived  services quality, perceived value and  recommendation: a study among  canadian public library users. library  management, 29(4/5), 352‐366.  lamonde,  y.,  mcnally,  p.f.,  &  rotundo,  a.  (2005).  public  libraries  and  the  emergence  of  a  public  culture.  in  y.  lamonde, p.f. fleming, & f.a. black  (eds.),  history  of  the  book  in  canada  volume 2: 1840‐1918 250‐271. toronto,  on: university of toronto press.    leckie, g.j., & hopkins, j. (2002). the public  place of central libraries: findings  from toronto and vancouver.ʺ library  quarterly, 72(3), 326‐372.  manjarrez, c.a., cigna, j, & bajaj, b. (2007).  making cities stronger: public library  contributions to local economic  development. urban libraries council.  retrieved 12 feb. 2010 from  http://www.urban.org/uploadedpdf/ 1001075_stronger_cities.pdf  may, f. (2009). in the words of the users: the  role of the urban public library as  place. in libraries as space and place:  proceedings of the pre‐congress satellite  for the ifla world library and  information congress, turin, italy, 19‐21  aug. 2000.  http://www.ifla2009.it/online/wp‐ content/uploads/2009/06/final.may.pd f  mckechnie, l.e.f., french, p.k., goodall, g.r.,  kipp, m., paquette, d.l., & pecoskie,  j.l. (2004). covered beverages now  allowed: public libraries and book  superstores. canadian journal of  information and library science/la revue  25 evidence based library and information practice 2010, 5.2  26 canadienne des sciences ddelʹinformation  et de bibliothéconomie, 28(3), 39‐51.   mitchell, d. (1995). the end of public space?  people’s park, definitions of the  public, and democracy. annals of the  association of american geographers,  85(1), 108‐33  newman, w. (2004). public libraries in the  priorities of canada: acting on the assets  and opportunities. provincial and  territorial library directors council.  retrieved 12 feb. 2010 from  http://www.collectionscanada.gc.ca/ob j/s7/f2/03‐e.pdf  nova scotia. (2001). standards for nova scotia  regional public libraries. 2nd ed.  department of education, nova scotia  provincial library, halifax, ns.  retrieved 12 feb. 2010 from  https://www.library.ns.ca/files/standar ds.pdf  nova scotia. (2006). opportunities for  sustainable prosperity 2006: an updated  economic growth strategy for nova scotia.  office of economic development.  retrieved 12 february 2010 from  http://www.gov.ns.ca/econ/ofsp/docs/ opportunities_for_sustainable_prosp erity_2006.pdf  online computer library centre. (2005).  perceptions of libraries and information  resources. online computer library  centre, dublin, oh. retrieved 12 feb.  2010 from  http://www.oclc.org/reports/2005perce ptions.htm  picco, m.a.p.. (2008). multicultural librariesʹ  services and social integration: the  case of public libraries in montreal  canada. public library quarterly, 27(1),  41‐56.   ragin, c.c. (2001). case‐oriented research. in  n.j. smelser & p.b. baltes, (eds.),  international encyclopedia of the social  and behavioral sciences 1519‐1535.  oxford: pergamon.  singh, s.r. (1985). happy mall fellows.  american libraries, 16(3), 154.   steffen, n., lietzau, z., lance, k.c., rybin, a.,  & molliconi, c. (2009). public libraries,  a wise investment: a return on  investment study of colorado libraries.  library research service. retrieved 12  feb. 2010 from  http://www.lrs.org/documents/closer_ look/roi.pdf  waller, v. (2008). legitimacy for large public    libraries in the digital age. library   review, 57(5), 372‐385.  wavell, c., baxter, g., johnson, i., & williams,  d. (2002). impact evaluation of museums,  archives and libraries: available evidence  project. aberdeen: the robert gordon  university. retrieved 12 feb. 2010  from  http://citeseerx.ist.psu.edu/viewdoc/d ownload?doi=10.1.1.128.2899&rep=rep 1&type=pdf   werle, j., & fox, l. (2007). internet 2 and  libraries: serving your communities at  the speed of light. computers and  libraries, 27(10). retrieved 12 feb. 2010  from  http://www.infotoday.com/cilmag/no v07/werle_fox.shtml  wilson, v. (2008). public libraries in canada:  an overview. library management,  29(6/7), 556‐570.  williamson, m. (2000). social exclusion and  the public library: a habermasian  insight. journal of librarianship and  information science, 32(4), 178‐86.  wooden, r.a. (2006). the future of public    libraries in an internet age. national    civic review, 95(4), 3‐7.      http://www.oclc.org/reports/2005perceptions.htm http://www.oclc.org/reports/2005perceptions.htm evidence based library and information practice 2010, 5.2  appendix 1 : library user questionnaire    library user profile    1)     sex:  male   female    2)   age category:  less than 18  years  18‐24 years  25‐34  years  35‐44  years  45‐54  years  55‐64  years  over 65 years    3)   highest level of educational attainment: (please check one)    elementary school    some secondary school    secondary school diploma    some university or college    university or college degree / diploma    some post‐graduate university    university post‐graduate degree    4)   languages spoken: (please check all that apply)   english   french   other(s): please list    5)   field of primary occupation: (please check one)    student:  elementary high school college  university  language  school      homemaker or full‐time parent    art, culture, recreation or sport (e.g. artist, writer, lifeguard)    sales and service (e.g. restaurant / bar server, cashier)    business, finance, administration    social services, education, government service, religion    trades, transport, equipment operator    management    health    natural and applied science    processing, manufacturing, utilities    primary industry (e.g. fishing, farming, forestry, mining)    retired    unemployed     other; please specify      about your library visits    in what part of nova scotia do you live? (e.g. halifax north, bedford, purcell’s cove)    6) how long does it take you to travel from your home to this library by your usual means of  transportation?  under 15  minutes  15‐30 minutes  31‐60 minutes  1‐2 hours  over 2 hours  27 evidence based library and information practice 2010, 5.2    7)   how long do you usually stay at this library?   under 30 minutes  30‐60 minutes  1‐2 hours  2‐4 hours  4‐6 hours  over 6 hours    8)   on average, how often have you visited this library during the past twelve months?  once a week or  more  once a  month  two to three times  a month  once every other  month  a few times a year  once (today)    9)   what day or days of the week do you usually visit this library?  (please circle all  that apply)    monday  tuesday  wednesday  thursday  friday  saturday  sunday    10) what time of day do you usually enter the library? (please circle one)  morning (before  noon)  afternoon (noon ‐ 5:00)  evening (after 5:00)  don’t know    11) what hours do you prefer the library to be open? (please circle one)  morning (before  noon)  afternoon (noon ‐ 5:00)  evening (after 5:00)  don’t know  12) how many people usually come with you to this library? (please circle one)  none   one other person  two or more other people        if one or more people usually come with you to the library, are they primarily: (please  circle one)  friends  relatives  co‐workers  others    13) will you be visiting any stores or services near the library today?   yes  no  don’t  know        about your library use    14) without this library, how would your life be affected? (please circle one)  not at all  very little  somewhat  considerably  major impact    15) why do you use this library rather than other libraries?    16) do you visit other libraries or library branches?  if yes please list them:    17) do you ever talk to the library staff when you come here?    if yes, what do you talk to them about?    18) why did you come to the library today?   (please indicate the most important reason  with the number 1 and check any others that apply with a check mark.)    use public meeting room    use children’s services    browse  28 evidence based library and information practice 2010, 5.2    borrow / return materials for myself    borrow / return materials for others (e.g. children, family, friends)    consume food / drinks    use / borrow non‐english language materials    look for information on a subject    meet a friend    view art work, displays, notice boards    obtain help from library staff    read    study in the library with own materials    use photocopiers    use microfiche / film    use cd roms    use the on‐line catalogue    use the internet    use the wireless internet    use electronic databases    use e‐mail    use the library’s outdoor benches or lawn    other: please specify:    19) when you visit this library, how often do you use electronic resources (e.g., the internet,  on‐line     catalogues, cd roms, electronic databases)? (please circle one)  always  usually  sometimes  rarely  never    20) please rank the importance to you of the electronic resources in this library (e.g., the  internet, on‐line   catalogues, cd roms, electronic databases)? (please circle one)  vital  very important  important  unimportant  very unimportant    21) do you ever bring a laptop computer with you to use in this library?  yes  no    22) do you ever bring in any other electronic equipment with you to this library? (e.g., cell  phone, digital   scanner)  yes  no  if yes, please specify:    23) how “user friendly” is this library? (please circle one)  very friendly  friendly  somewhat friendly  unfriendly    24) what other services or resources would you like to see introduced here?    25) please select the single most important service this library provides for you: (please select      only one)    access to technology    a place to read    a place to socialize    a place for community events    a source for community information    a place for life‐long learning (educational support)    a place to study (homework, research, etc.)  29 evidence based library and information practice 2010, 5.2    a place to borrow non‐fiction / informational material    a place to borrow fiction / literatures (i.e., recreational reading)    a place to use reference materials and information services    other: please specify    26) please select the single least important service this library provides for you: (please select      only one)    access to technology    a place to read    a place to socialize    a place for community events    a source for community information    a place for life‐long learning (educational support)    a place to study (homework, research, etc.)    a place to borrow non‐fiction / informational material    a place to borrow fiction / literatures (i.e., recreational reading)    a place to use reference materials and information services    other: please specify    27) do you ever use this library as a place to meet tutors?   yes   no      if yes, what kind of tutoring?   english as a second languag  secondary / high  school   adult  learner   other: please  list    28) has this library been renovated within the last three years?  yes  no  don’t know      if yes, do you use the library differently now?   yes  no  don’t know      please explain your response:    29) what is the one best feature of this library?  30) what is the one worst feature of this library?  31) where is your favourite location or place in this library?   32) what words you think best describe the physical space of this library?  (select all that         apply)    attractive    loud    badly designed    modern    bright    needing renovation    clean    quiet    comfortable    unattractive    crowded    uncomfortable    dark    unfriendly    dirty    user friendly    friendly    well designed  30 evidence based library and information practice 2010, 5.2    lots of space    other: please list:  33) in your opinion, what is the main purpose of this library?    34) please add any additional comments you wish to make.    35) please tell us your thoughts about the following features of this library.  use the following scale to rate the features by circling the most appropriate number.  1 = very unsatisfactory 2 = unsatisfactory 3 = neutral 4 = satisfactory 5 = very satisfactory  0 =  donʹt know  features  please circle your rating  reasons for your answer?   outside:                parking  1  2  3  4  5  0    walkways and other pedestrian space  1  2  3  4  5  0    building entrance  1  2  3  4  5  0    signs outside  1  2  3  4  5  0    book return  1  2  3  4  5  0    lawn / benches (if applicable)  1  2  3  4  5  0    inside:                lights  1  2  3  4  5  0    windows  1  2  3  4  5  0    signs inside  1  2  3  4  5  0    colours  1  2  3  4  5  0    tables / study carels  1  2  3  4  5  0    seating  1  2  3  4  5  0    childrenʹs area  1  2  3  4  5  0    arrangement / organization of the library  1  2  3  4  5  0                                          31 evidence based library and information practice 2010, 5.2  appendix 2: library user interview      hello, i’m from dalhousie university’s library and information studies program and we are doing a  study about this library’s use and design. would you be able to share 5‐7 minutes of your time to  answer a few questions? participation is voluntary.    1) what do you think of this library building?  prompts: well designed, hard to find things in it, bright, dark    2) what do you think of the space around this library?  prompts: spacious, dirty, unsafe due to traffic, nice place to sit    3) what is the best feature of this library building?    4) what is the worst feature of this library building?    5) where is your favourite location or place in this library?     6) is there any other library that you prefer to use?   yes/no. if yes, why do you prefer the other library?    7) why did you come to the library today?  prompts: to borrow/return materials, to attend a program, to find information    8) what have you used the library for in the past month?  prompts: to borrow/return materials, to attend a program, to find information    9) what do you think about how people use this library?  prompts: to socialise, meet friends, study, bring their children    10) how do you think people ought to use this library?  prompt: to socialise, attend programs, study    11) have you ever seen the library put to uses (either good or not so good) that have surprised you?  32 evidence based library and information practice 2010, 5.2  33 appendix 3: library staff interview questions    1) could you please provide a word or two that best describes this library:    2) what is the single best feature of this library?    3) what is the single worst feature of this library?    4) where is your favourite location or place in this library?    5) in your opinion, what is the primary purpose of this library?    6) aside from providing the public with access to information, what do you think are important  social functions of this library?    7) how do you see public libraries being used socially?    8) please briefly describe your primary job responsibilities.    9) describe the nature of your interaction with the public.    10) what are most users doing in this library?    11) what are the primary problems you’ve experienced when dealing with the public in the library?    12) how does the physical environment of this library help or hinder your job?    13) how does the physical environment of this library help or hinder your interaction with the  public?    14) in your experience, how do patrons typically behave in the library?    15)  have you seen library patrons use the library in ways that have surprised you? please explain.    16) how has computer technology affected your job?    17) how has computer technology affected your interaction with the public?    18) what proportion of your interactions with the public involves helping them use computer  technologies?    19) is there any final comments you would care to make about the design, technology, use, or  services of this library?  evidence based library and information practice 2010, 5.2  appendix 4: population characteristics of the six public library communities        population characteristic   hn *  kg *  sg*  lb†  ng†  tt†  population served  8,932  48,141  51,684  5,035  12,010  8,062  age   (two largest age categories)  1. 20‐34 (32%).  2. 35‐49 (20%)  1. 20‐34 (27%).  2. 35‐49 (23%).  1. 20‐34 (33%)  2. 35‐49 (21%)  1. 35‐54 (25%).  2. 55‐64 (17%).  1. 35‐54 (30%)  2. 5‐19 (17%).  1. 35‐54 (38%).  2. 5‐19 (22%).  marital status   (largest population group by marital  status)  people never  married ‐ 57%  married people ‐  47%.  people never  married ‐ 49%  married people ‐  55%  married people ‐  47%  married people ‐  66%  lone parent families  39% of families  are lone‐parent.  16% of families  are lone‐parent.  15% of families  are lone‐parent  10% of families  are lone‐parent.  20% of families  are lone‐parent.  7% of families  are lone parent.  mother tongue   (top three)  1. english ‐ 90%  2. french ‐ 3%  3. arabic ‐ 0.6%  1. english ‐ 87%  2. french ‐ 3%  3. arabic ‐ 3%  1. english ‐ 89%  2. french ‐ 3%  3. arabic ‐ 2%  1. english ‐ 97%  2. french – 1%  3. other ‐ 1%  1. english ‐ 98%  2. french ‐ 1%  3. other ‐ 1%  1. english ‐ 96%  2. french ‐ 2%  3. other ‐ 2%  aboriginal population  2%.  0.2%.  0.30%  0.4%   0.4%  1%  immigrant population  8%  11%  10%  8%  3%  5%  education   (residents with a university or non‐ university diploma, degree or  certificate)  48%  60%  65%  48%  46%  65%  employment status   (% of persons over 25 who are  unemployed)  10%  6%  5%  5%  4%  3%  average household income  $30,117  $58,543  $60,685  50,773  $47,459  $75,854  * 2001 census statistics as cited in: halifax public libraries planning and development department. halifax public libraries branch profiles 2004‐2005. [report],  2006.  † 2006 census statistics as cited in: province of nova scotia. community counts. 2006. 11 dec. 2008 <http://www.gov.ns.ca/finance/communitycounts/default.asp>  34 http://www.gov.ns.ca/finance/communitycounts/default.asp methods results about the libraries about the library patrons patron activities and locations in the library patrons and staff describe the role of public libraries in community a place to borrow books and find information a place to access technology a social, community space how can public libraries use this information? conclusion evidence summary   gender is a variable of interest for information literacy instruction   a review of: pinto, m., sales, d., & fernández-pascual, r. (2019). gender perspective on information literacy: an interdisciplinary and multidimensional analysis within higher education settings. library & information science research, 41(4), 100979. https://doi.org/10.1016/j.lisr.2019.100979   reviewed by: hilary bussell librarian for political science, economics, & communication the ohio state university libraries columbus, ohio, united states of america email: bussell.21@osu.edu   received: 1 dec. 2020                                                                accepted:  18 jan. 2021      2021 bussell. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29886     abstract   objective – to identify gender differences that present in 26 information literacy (il) learning competencies using a multidimensional subjective–objective approach.   design – two quantitative survey questionnaires, administered online.   setting – five spanish public universities in 2014.   subjects – thirdand fourth-year students in eight social science degree programs including information science, audiovisual communication, journalism, psychology, primary education, pedagogy, social work, and tourism.   methods – subjects were recruited using a stratified sampling approach. two survey instruments were distributed online. the il-humass instrument uses likert scales to measure students’ “belief-in-importance” (bi) of various il competencies relating to searching, evaluation, processing, and communication–dissemination, as well as their self-efficacy (se) regarding these competencies. the evalci-kn instrument measures students’ actual knowledge (kn) of the same il competencies using closed answer options. the data were analyzed using descriptive and bivariate statistics and confirmatory factor analyses.   main results – the total number of valid surveys collected was 1,575 (sampling ratio of 10.39% of eligible students). no significant differences were found between female and male students’ bi, se, or kn in the categories of searching and evaluation. statistically significant differences between genders were found relating to se and knowledge of information processing (with men having higher scores), and to knowledge of communication–dissemination (with women having a higher score). overall, students’ kn scores were higher than their se scores. statistically significant differences were found among male students in all categories and dimensions except in se of evaluation and bi of communication–dissemination and among female students except in bi of processing. information science and pedagogy were the highest scoring degree programs in different dimensions and categories; tourism and social work were the lowest. male students’ awareness of the importance of using print sources and assessing the quality of information could be improved; female students’ awareness of the importance of knowing information source typologies, academic codes of ethics, and intellectual property laws could be improved. the authors also state that male students’ kn should be increased in the areas of schematizing and abstracting information, handling statistical programs, and knowing the laws on information use and intellectual property, and they point to the need for instructional support to improve all students’ se across all il categories.   conclusion – gender differences were found in various il competencies as measured by the three scales (bi, se, kn). male students were found to believe assessment skills to be most important and to believe themselves more prepared in search skills; however, their actual knowledge was highest in the communication category. in comparison, female students prioritized communication skills and believed themselves more prepared in search skills, with their actual knowledge highest in the search and communication categories. among both genders, weaknesses were found relating to bi in four competencies (use informal electronic sources, know information search strategies, schematize–abstract information, recognize text structure), to se in six competencies (use printed sources, know information search strategies, assess quality of information, schematize–abstract information, recognize text structure, write a document), and to kn in five competencies (use printed sources, use electronic sources, use informal electronic sources, determine whether information is updated, and know the code of ethics in the academic field). the students’ mean score was higher for kn than for se in searching, evaluation, and communication–dissemination. the authors recommend instruction or awareness-raising sessions to help students acquire il competencies as well as to improve their self-esteem in these areas, with the design of these sessions incorporating the findings on gender differences. they also recommend a review of existing syllabi to help “incorporate the gender perspective into the classroom” (p. 8).   commentary   this paper builds on the authors’ previous work on university students’ motivation and se around il competencies (pinto & fernández-pascual, 2017), as well as the body of library and information science research on gender differences in il. in a 2017 study, taylor and dalal analyzed gender differences in college students’ approaches to searching and evaluating internet sources, and argued that these differences should inform “librarians’ efforts to meet the needs of all learners” (p. 106). pinto, sales, and fernández-pascual contribute to this effort by investigating gender differences in the context of il as a multidimensional learning process. their work indicates that being information literate is not simply a matter of possessing certain cognitive skills relating to information, but also being confident in one’s abilities and appreciating why the skills are important. this “subjective–objective” perspective gives us a more nuanced lens for exploring gender differences in il. the critical appraisal checklist developed by glynn (2006) was used to evaluate this article. the authors note that the two survey instruments employed in this study were previously validated by other studies. cronbach’s alpha was used to confirm the reliability of the instruments. the il-humass instrument, which measures students’ perceptions of il competencies, is included in an appendix to the article. the evalci-kn instrument, which measures actual levels of knowledge, is linked to in the body of the article, but the link was not working as of the writing of this evidence summary. the response rate for the study was 85.78%, which exceeds the recommended minimum response rate for surveys with a sampling frame of this size (fosnacht et al., 2017).   the article would have benefitted from an expanded discussion of gender in a few respects. the authors do not specify how the gender of participants was determined; this is a potentially significant limitation, given the focus of the study. further, the study uses the male–female gender binary and does not account for nonbinary or genderfluid students. this may impact the applicability of its findings for library instruction for gender diverse populations and for future research on gender in il. finally, it would have been pertinent for the researchers to have acknowledged in their analysis or recommendations that female students are often treated differently than their male counterparts, both in school and in the wider society, and that this may have measurable effects on se. librarians attempting to bolster students’ se need to grapple with the reality of institutional sexism, and consider the steps they can take to counteract it. this is a tall order, given the limitations of the traditional one-shot model (bowles-terry & donovan, 2016); to do truly meaningful work in this area, librarians will need to develop programmatic approaches to integrating gender-conscious il instruction into curricula.   this research will help library practitioners and scholars better understand il as a multidimensional phenomenon. the study explores il from subjective and objective perspectives and shows that students’ objective knowledge of il competencies does not always align with their se. these findings should encourage librarians to think about how they can incorporate activities into their instruction that increase students’ se and motivation to learn il skills. additionally, by making the case for gender as a variable to consider when researching il, the study adds to the evolving understanding of il as a socially-constructed concept shaped by our notions of gender, race, socioeconomic status, and other social categories.    references   bowles-terry, m., & donovan, c. (2016). serving notice on the one-shot: changing roles for instruction librarians. international information & library review 48(2), 137–142. https://doi.org/10.1080/10572317.2016.1176457   fosnacht, k., sarraf, s., howe, e., & peck, l. k. (2017). how important are high response rates for college surveys? the review of higher education, 40(2), 245–265. https://doi.org/10.1353/rhe.2017.0003   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387–399. https://doi.org/10.1108/07378830610692154   pinto, m., & fernández-pascual, r. (2017). a diagnosis of the levels of information literacy competency among social sciences undergraduates. portal: libraries and the academy, 17(3), 569–593. https://doi.org/10.1353/pla.2017.0035   taylor, a., & dalal, h.a. (2017). gender and information literacy: evaluation of gender differences in a student survey of information sources. college & research libraries, 78(1), 90–113. https://doi.org/10.5860/crl.78.1.90   evidence summary   the causes and consequences of low morale amongst public librarians   a review of: kendrick, k. d. (2020). the public librarian low-morale experience: a qualitative study. partnership: the canadian journal of library and information practice and research, 15(2), 1-32. http://doi.org/10.21083/partnership.v15i2.5932   reviewed by: andrea miller-nesbitt associate librarian schulich library of physical sciences, life sciences, and engineering mcgill university library montreal, qc, canada email: andrea.miller-nesbitt@mcgill.ca     received: 10 aug. 2022                                                             accepted: 18 oct. 2022      2022 miller-nesbitt. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30219     abstract   objective – to understand if, how, and within what parameters, librarians working in public libraries experience low morale.   design – semi-structured interview, phenomenology   setting – public libraries in canada and the united states   subjects – participants (n = 20) were credentialed librarians who worked or had worked in a public library, and who experienced low morale due to their work.    methods – invitations were distributed to 10 electronic mailing lists. purposive sampling was used to select the participants -they represented librarians with a range of experience, working within a variety of specialties. the researcher received informed consent and the participants completed a short survey in order to collect demographic data before taking part in semi-structured interviews. the interviews were transcribed and coded, after which data were analyzed and thematic clusters identified.    main results – various types of abuse, either performed by library users (ex. physical and verbal abuse), or by colleagues/managers/administrators (ex. emotional abuse, system abuse, and negligence) were revealed to cause low morale in public librarians. data show that the participants' responses to the abuse influenced their affective, cognitive, and physiological well-being, as well as professional expectations and trajectories. this study identified three low morale impact factors and seven enabling systems that were unique to public librarians when compared to academic librarians. the unique impact factors are: personal safety, resilience narratives, and social contexts. the enabling systems are: organizational structure, library workplace culture, on-demand relocation, policies, training, equity, diversity, and inclusion (edi), and politics.    conclusion – this study builds on the literature and provides additional evidence on the prevalence of low morale in lis workplaces. the data show that there are similarities in the causes and consequences of low morale in the workplace amongst public librarians and their academic counterparts. understaffing, mission creep, and working with underserved and marginalized communities all play a part in the morale of public librarians. low morale negatively affects public librarians’ mental and physical health, as well as their professional outlook and trajectory. the author makes a case for comprehensive leadership training for public library management, as well as the presence of people with different expertise (such as social workers and first responders) in public libraries. additionally, the author suggests the need for further research on topics that came up in this study.   commentary   there are a handful of articles published on workplace morale in libraries dating back decades (e.g. wilkins jordan, 2014), but the real proliferation of research on the topic began after the publication of the author’s seminal paper in 2017 (kendrick). recent articles use a variety of methods (e.g. kennedy & garewal, 2020), and cover a variety of settings and types of employment – public and academic librarians, as well as library staff (e.g. glusker et al., 2022).   the study was appraised using the critical review form for qualitative studies created by letts et al. (2007). the purpose of the research was clearly stated, and the context and definitions of the concepts under investigation are outlined in the literature review. the methodology chosen was appropriate given the objectives of the study. the demographics of the participants are described, and although attempts were made to ensure participants were representative, it is unclear how many are from canada. the author does not state whether the interview questions were validated, or whether the sampling was done until redundancy was reached. data coding and analysis is rigorous and well described.    the results section is well structured and very detailed. this research builds on the author’s previous work. as such, there are several references to themes elucidated in those earlier studies. for example, the unique impact factors and enabling systems identified in this study affect public librarians in addition to those identified as affecting academic librarians as previously described (kendrick, 2017; kendrick & damasco, 2019). excerpts from the interviews help the reader better understand the experiences of the public librarians from their own perspective. in addition to outlining the causes and consequences of low morale in the participant group, the author includes descriptions of coping strategies, attempts at mitigating the experience, and the recovery process for participants who moved to new positions.   the results of this study verify the experiences of public librarians’ low morale in the workplace. the author offers several recommendations on topics for further research. these include how leaders (within both the library and the political sphere) and communities’ value people working in public libraries, and the implementation and impact of security measures in public libraries. the research results confirm that mission creep, compassion fatigue, funding cuts, and workplace incivility are harming lis practitioners both personally and professionally. the author suggests countermeasures to decrease workplace abuse, neglect, and dysfunction. these measures include empathetic, authentic management and leadership training, having social workers and first responders working collaboratively with public librarians, and the need for strong library employee advocacy.   references   glusker, a., emmelhainz, c., estrada, n., & dyess, b. (2022). “viewed as equals”: the impacts of library organizational cultures and management on library staff morale. journal of library administration, 62(2), 153-189. https://doi.org/10.1080/01930826.2022.2026119    kendrick, k. d. (2017). the low morale experience of academic librarians: a phenomenological study. journal of library administration, 57(8), 846-878. https://doi.org/10.1080/01930826.2017.1368325    kendrick, k. d., & damasco, i. t. (2019). low morale in ethnic and racial minority academic librarians: an experiential study. library trends, 68(2), 174-212. https://doi.org/10.1353/lib.2019.0036    kendrick, k. d., (2020). the public librarian low-morale experience: a qualitative study. partnership, 15(2), 1-32. https://doi.org/10.21083/partnership.v15i2.5932   kennedy, s. p., & garewal, k. r. (2020). quantitative analysis of workplace morale in academic librarians and the impact of direct supervisors on workplace morale. journal of academic librarianship, 46(5), 102191. https://doi.org/10.1016/j.acalib.2020.102191    wilkins jordan, m. (2014). all stressed out, but does anyone notice? stressors affecting public libraries. journal of library administration, 54(4), 291-307. https://doi.org/10.1080/01930826.2014.924318   microsoft word news_work.doc evidence based library and information practice 2007, 2:4  112 evidence based library and information practice     news    workshop: introduction to systematic reviewing       lindsay glynn  editor‐in‐chief    public services librarian and instruction coordinator, health sciences library  memorial university of newfoundland  st. john’s, newfoundland, canada  e‐mail: lglynn@mun.ca      © 2007 glynn. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    aims     how can we translate evidence for policy  and practice? this workshop will introduce  the principles, methods and critiques of  systematic reviews.     on completion participants will:     • have a basic understanding of the  purposes of systematic research  synthesis and its relevance to evidence‐ informed policy and practice;     • have discussed a variety of systematic  review models;     • have received a general overview of the  standard stages of a systematic review.         target audience     this workshop will be of interest to policy  makers, practitioners and researchers with  an interest in conducting or using systematic  reviews. it is designed for people with no, or  almost no, prior knowledge of synthesis  research.     date and venue     9.30 – 5.30pm wednesday, march 12, 2008   esrc regional training centre, queenʹs  university, 69/71 university street, belfast,  bt7 1hl     booking     to book a place on the workshop download  the booking form  http://eppi.ioe.ac.uk/cms/portals/0/forms/bo oking%20form%20belfast%20120308.pdf   mailto:lglynn@mun.ca http://creativecommons.org/licenses/by/2.0 http://eppi.ioe.ac.uk/cms/portals/0/forms/bo evidence based library and information practice 2007, 2:4  113 and fax the completed form back to (020  7612 6400). alternatively, e‐mail  p.rose@ioe.ac.uk to have a booking form  sent out to you. please let us know if you  would like the booking form in another  format.     accessibility   if you have a disability, sensory impairment,  medical condition, specific learning  difficulty, or mental health problem  requiring extra support during the  workshop, it is essential that you contact us  as soon as possible so we will be able to  arrange this support in good time.           further information can be obtained from:     philip james rose   social science research unit  institute of education, university of london   18 woburn square  london  wc1h 0nr   e‐mail: p.rose@ioe.ac.uk   tel: 0207 612 6391     or     eppi‐centre, social science research unit,  institute of education, university of london   http://eppi.ioe.ac.uk/cms/default.aspx?tabid =830                          mailto:rose@ioe.ac.uk mailto:rose@ioe.ac.uk http://eppi.ioe.ac.uk/cms/default.aspx?tabid    evidence based library and information practice       evidence summary   many indian phd students lack motivation and skills to use academic journal articles, their libraries lack resources and standards   a review of: saxena, s. (2018). factors impacting the usage of academic journal articles by phd students in india. information discovery and delivery, 46(4), 204-213. https://doi.org/10.1108/idd-09-2017-0069   reviewed by: michelle dubroy discipline librarian (researcher services) library griffith university gold coast, queensland, australia email: m.dubroy@griffith.edu.au   received: 9 feb. 2020                                                                accepted:  30 mar. 2020      2020 dubroy. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29731     abstract   objective – to investigate the factors influencing the use of academic journals by phd students in india.   design – grounded analysis.   setting – five universities in india.   subjects – 147 phd students.   methods – subjects were selected using a mix of convenience and purposeful sampling. email was then used to send the questions, receive the responses, and seek clarification as required. this process was conducted between september 2016 and january 2017.   main results – completed responses were received from 134 students, resulting in a response rate of approximately 91%. the researcher identified five factors influencing academic journal usage: institutional, task complexity, relevance and application, information quality, and technical. there was “marked” dissatisfaction with library facilities and access to academic resources, with one respondent stating that their library “does not subscribe to a single electronic journal” (p. 209). other identified issues include students’ insufficient awareness of what is available, limited motivation to “undertake serious research work” (p. 210) and inadequate skill levels to use available resources effectively.   conclusion – universities should provide the required resources (both human and infrastructure) to ensure their academic libraries meet quality standards. to do so requires appropriate funding. additionally, researchers should be encouraged to use their library’s resources in the context of improving their scholarly contribution.   commentary   studies about the information seeking habits of doctoral students have been well synthesized (catalano, 2013; spezi, 2016). however, most of the included studies originate in the west. this study, therefore, provides a needed view of the topic from a developing country.   the study was reviewed for rigour (chiovitti & piran, 2003) and credibility (oktay, 2012). both strengths and weaknesses were found.   the researcher’s useful table of related literature identifies a sound rationale for the study. the researcher also articulates the context and demographic details of her participants.   regrettably, the paper’s methods section is inadequate for demonstrating adherence to grounded theory methods. the credibility of the research is, therefore, unclear (oktay, 2012). remarkably, the researcher has not disclosed how, or if, she coded the data. further, readers do not know how the researcher arrived at her conclusions nor if she had considered other hypotheses.   participants could "contribute any miscellaneous information” (p. 205) they wished, and clarification was sought from them as required. however, it does not appear that participants guided the research process in any way. moreover, there is no evidence the researcher used participant feedback to confirm or revise the emerging theory. the use of focus groups or live interviews, rather than an open-ended survey, could have provided a richer body of data to work with.   the researcher appears to draw conclusions from the data and connects her findings to her participants’ own words. however, she does not state how well these quotations typify the data nor if this information “earned its way into the theory... [through] repeated presence” (chiovitti & piran, 2003, p. 429).   despite these limitations, the proposed practitioner implications are sensible. academic librarians in india will no doubt welcome calls for greater funding and the pursuit of quality standards. the transferability of the findings, however, could have been further demonstrated by linking the conclusions to relevant literature.   the study will be of interest to academic librarians who work with phd students educated in developing nations. the apparent lack of library resources available to students studying in these countries is notable. ways to support research students transferring from these countries to better-resourced institutions should be considered.   references   catalano, a. (2013). patterns of graduate students' information seeking behavior: a meta‐synthesis of the literature. journal of documentation, 69(2), 243-274. https://doi.org/10.1108/00220411311300066   chiovitti, r. f., & piran, n. (2003). rigour and grounded theory research. journal of advanced nursing, 44(4), 427-435. https://doi.org/10.1046/j.0309-2402.2003.02822.x   oktay, j. s. (2012). evaluating quality. in j. s. oktay (ed.), grounded theory. (pp. 103-124). oxford: oxford university press. https://doi.org/10.1093/acprof:oso/9780199753697.003.0005   spezi, v. (2016). is information-seeking behavior of doctoral students changing?: a review of the literature (2010-2015). new review of academic librarianship, 22(1), 78-106. https://doi.org/10.1080/13614533.2015.1127831     federal library utilization of libguides to disseminate covid-19 information research article   federal library utilization of libguides to disseminate covid-19 information   sarah c. clarke medical reference librarian darnall medical library walter reed national military medical center bethesda, maryland, united states of america email: sarah.c.clarke.civ@mail.mil   emily e. shohfi clinical medical librarian darnall medical library walter reed national military medical center bethesda, maryland, united states of america email: emily.e.shohfi.civ@mail.mil   sharon han engagement specialist all of us national program network of the national library of medicine university of iowa iowa city, iowa, united states of america email: sharon-han@uiowa.edu   received: 29 july 2021                                                               accepted: 14 dec. 2021      2022 clarke, shohfi, and han. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30017     abstract   objective – in winter 2019-2020, the world saw the emergence of coronavirus disease (covid-19), caused by severe acute respiratory syndrome coronavirus 2 (sars-cov-2). more than a year later, the pandemic continues with the u.s. death toll surpassing 550,000. over the last decade, librarians have increased their roles in infectious disease outbreak response. however, no existing literature exists on use of the widely-used library content management platform, libguides, to respond to infectious disease outbreaks. this research explores how federal libraries use libguides to distribute covid-19 information throughout the ongoing covid-19 pandemic.   methods – survey questions were created and peer-reviewed by colleagues. survey questions first screened for participant eligibility and collected broad demographic information to assist in identifying duplicate responses from individual libraries, then examined the creation, curation, and maintenance of covid-19 libguides. the survey was hosted in max.gov, a federal government data collection and analysis tool. invitations to participate in the survey were sent via email to colleagues and listservs and posted to personal social media accounts. the survey was made publicly available for three weeks. collected data were exported into excel to clean, quantify, and visualize results. long form answers were manually reviewed and tagged thematically.   results – of the 78 eligible respondents, 42% (n = 33) reported that their library uses libguides to disseminate covid-19 information; 45% of these respondents said they spent 10+ hours creating their covid-19 libguide, and 60% of respondents spent <1 hour a week on maintenance and updates. most libguides were created in early spring 2020 as the u.s. first saw an uptick in covid-19 cases. for marketing purposes, respondents reported using web/internal announcements (75%) and email (50%) most frequently. all respondents reported inclusion of u.s. government resources in their covid-19 libguides, and a majority also included guidelines, international websites, and databases to inform their user communities.   conclusion – some federal libraries use libguides as a tool to share critical information, including as a tool for emergency response. results show libraries tend to start from scratch and share the same resources, duplicating efforts. to improve efficiency in libguide curation and use of library staff time, one solution to consider is the creation of a libguides template that any federal library can use to quickly set up and adapt an emergency response libguide specifically for their users. additionally, findings show that libraries are uncertain of archiving and preservation plans for their guides post-pandemic, suggesting a need for recommended best practices.     introduction   in the winter of 2019-2020, severe acute respiratory syndrome coronavirus 2 (sars-cov-2) emerged, creating a new strain of coronavirus disease (covid-19) (who, 2020). people infected with covid-19 present varying symptoms and degrees of severity; common characteristics include coughing, difficulty breathing, fever, and loss of taste or smell (cdc, 2020a). covid-19’s rapid global spread and severity in impacted populations led the world health organization (who) to declare the first pandemic caused by a coronavirus on march 11, 2020 (adhanom ghebreyesus, 2020). early official reports of covid-19 cases in the united states emerged in january 2020, and cases have exceeded 32.5 million as of may 12, 2021 (cdc, 2020b; stein, 2020). following examples set from previous management of respiratory disease outbreaks, such as middle east respiratory syndrome (mers, 2012) and severe acute respiratory syndrome (sars, 2003), public health officials recommended frequent handwashing and avoiding close contact with infected individuals to reduce transmission rates (lai et al., 2020). in the u.s., the centers for disease control and prevention (cdc) provided further guidance to slow virus transmission by limiting mass gatherings, closing schools and non-essential businesses, issuing stay-at-home orders, and wearing cloth face coverings in public areas (schuchat, 2020). as covid-19 cases continue in the u.s., mitigation and monitoring strategies remain priorities for many government and public health professionals.   literature review   over the last decade, libraries have increased their roles in infectious disease outbreak response. during the 2014-2016 ebola outbreak, librarians were involved with providing information and website support to affected african countries (jackson, 2014; landgraf, 2014). at the u.s. federal level, the disaster information management research center (dimrc) at the national library of medicine (nlm) created a special health topic page, “ebola outbreak 2014: information resources” (love et al., 2015). dimrc also curated pages for pandemic influenza, zika, and the 2018 ebola outbreak. through the medical library association (mla), librarians can earn a disaster information specialization, which provides knowledge of disaster preparedness and response structure at varying levels of government and prepares them to assist in the response of various disasters, including infectious disease outbreaks (mla). there is no existing literature focused on the use of the widely-used library content management and curation platform, libguides, to respond to information needs during infectious disease outbreaks. however, during the current covid-19 pandemic, springshare, the creator of libguides, highlighted several examples of covid-19 specialty pages on their blog (creech, 2020; talia, 2020). a recently published paper examines the roles librarians fill in response to covid-19 and determined there are three dimensions of librarian support: to promote consumer level information for preventive measures, support researchers and/or faculty in their varying needs, and maintain core-needs of patrons (ali & gatiti, 2020). libguides is an acceptable platform to meet these objectives.   libguides is an annually licensed product designed specifically for libraries. the purchased system serves as the primary web presence basis for many libraries, helping library staff curate knowledge, share information, and organize subject specific resources (springshare, 2020b). as of may 2021, libguides were employed at 6,100 institutions across 82 countries, with nearly 800,000 guides created by more than 130,000 library staff (springshare, 2021). springshare is unable to provide a complete list or data regarding federal library customers (ware, 2021). the platform is known for its ease of use and navigation, as well as its reusability for resources. it is mobile-optimized and available without institutional log-in. it provides automatic link checking and easily captures usage statistics for entire libguides or individual pages (leibiger & aldrich, 2013). because of its ease of set-up and external access, libraries that use libguides can quickly add or pivot content to meet user demands without relying on intranet administrators, therefore removing lengthy wait times. however, some barriers may exist to those acquiring libguides, such as budget constraints, staffing, or it concerns surrounding security.   the libguides community allows for libraries to choose to share all or part of their guides for other libraries to reuse. customizing a reused guide will not affect the original guide. when a library chooses to use this function, the original guide owner will be notified. best practices call for obtaining permission before copying a guide (springshare 2020a). the system allows for private and hidden pages, which many use for content pages under development, or those with more sensitive information not intended for public audiences. customization is important as it allows libraries to highlight licensed resources accessible to their patron base.      aims   multiple u.s. federal libraries utilize libguides, many of which are publicly available. with this in mind, we set out to understand how federal libraries use libguides to distribute covid-19 information throughout the course of the pandemic. the decision to restrict the scope of research to federal libraries was inspired by the literature gap in federal library response to the covid-19 pandemic. currently published literature regarding library response to the pandemic was heavily focused on public and academic library response. similarly, a gap in the literature exists with regards to federal library utilization of libguides. due to security, some federal libraries must create and maintain their libguides privately to protect their organizational mission. surveying this niche population begins a conversation surrounding which types of federal libraries utilize this tool. learning more about the creation, curation, and maintenance of covid-19 libguides will shed light on response effort capabilities within federal libraries and help determine future best practices for streamlining the urgent information-sharing process, should there be future pandemics or other emergencies.   methods   survey design   in this qualitative study, we selected a written survey method for assessing the federal library sample population on their practices in utilizing libguides for distribution of covid-19 information. surveys are common in library research, and for our purposes, were utilized for conveniently and safely obtaining information from a sizable, wide-spread sample of federal libraries amid a global pandemic. they are especially useful in eliciting information about attitudes that “are otherwise difficult to measure using observational techniques” (glasow, 2005). survey questions were designed based on research objectives and demographic information of interest. the final survey included 20 questions. two questions screened for participant eligibility for the study, and three questions collected broad demographic information used to identify duplicate responses from individual libraries. the remaining questions focused on the creation, curation and design, and engagement and preservation of covid-related libguides in federal libraries. these topics were selected to obtain a comprehensive overview of the continuum of libguide activities related to the pandemic. questions were externally reviewed by colleagues and key stakeholders and revised based on feedback. the survey was submitted for creation to max.gov, a federal government data collection and analysis tool, for hosting. max.gov administrators created the survey and made revisions before survey dissemination. max survey allows for conditional logic and used a generic survey link. survey questions can be viewed in appendix a.   survey distribution   the written survey was distributed electronically. email, forums, and social media were used to solicit responses from federal librarians. emails or forum posts were sent to federal librarian groups, such as fedlib, a listserv moderated by the library of congress, and other federal librarianship interest groups within national library associations, including the medical library association, special libraries association, and the american library association. these groups are commonly used to connect with federal librarians and are heavily utilized to recruit voluntary survey participants. the authors’ personal social media accounts on facebook and twitter were also used to invite participants. posts used the following hashtags to promote the study: #medlibs, #librarytwitter, #covidlibrary, #librarians, #libguides, and #federallibraries.   the initial timeframe for responses was two weeks, beginning september 23, 2020, and was extended an additional week until october 16, 2020. three rounds of reminders were engaged to elicit voluntary survey participants.   at the beginning of the survey, respondents were asked a screener question gauging if they were a federal librarian. if they responded no, the survey would automatically end.   data collection, cleaning, and analysis   survey responses were collected online and data was stored securely through max.gov’s max survey. survey results were exported in xlsx format view using max analytics.   results data were de-identified, cleaned, quantified, and visualized using excel. long form answers were manually reviewed and tagged based on consistent themes that appeared in responses; we reviewed these themes as a group. themes are further explained in the results and discussion. the data that support the findings of this study are openly available in open source framework (osf) at https://doi.org/10.17605/osf.io/swf34 (clarke, han, & shohfi, 2021).   results   use and creation   the survey had 96 library respondents. eighteen respondents (19%) were immediately screened out for eligibility, as they selected that they did not work in a federal library. based on the broad demographic responses provided, we did not identify a single library as having multiple staff respondents. of the remaining 78 respondents, 33 (42%) reported that their library uses libguides. one-third (n=11) of respondents reporting libguide use had publicly viewable libguide pages, 7 (21%) had libguides set to private access, and 2 (6%) had a mix of both private and public pages.   when surveyed if their library utilized libguides to disseminate covid-19 information, 20 (61%) respondents utilizing libguides stated yes, 12 (36%) stated no, and one was unsure. ten respondents explained with free text why their institutions did not share covid-19 information in their libguides. of these, 80% of the respondents elucidated that this information was out of scope for their library’s services, or the duty belonged to another department within their agency. the remaining respondents could not share covid-19 information due to logistical reasons including staffing constraints.   thirteen respondents had primary responsibility over their library’s covid-19 libguide page(s), while five shared responsibility. 2 respondents did not have responsibility over their library’s covid-19 libguide page(s) but were able to provide details about what their guides contained.   figure 1 months which respondents reported creation of their covid-19 libguides in federal libraries.   figure 2 reported time spent on initial content curation and ongoing weekly maintenance.   according to responses, the earliest federal covid-19 libguide was created in january 2020 (figure 1). 7 covid-19 libguides were created in march 2020, making it the most popular month for creation. covid-19 libguides creation continued during data collection for this study, which began in september 2020.   the estimated initial time spent on the creation/curation of covid-19 libguide information varied amongst respondents (figure 2). while one respondent spent under an hour and two spent 1-5 hours on initial libguide creation, the majority of respondents spent either 6-10 hours (40%) or more than ten hours (45%). after the initial set-up and curation of resources, respondents reported less time investment engaging in weekly updates and maintenance. twelve respondents estimated they spent under an hour each week updating covid-19 libguide information, and six respondents spent an estimate of 1-5 hours. one respondent spent between 6-10 hours, and one respondent spent over ten hours.   respondents using libguides had varied audiences for covid-19 information. the number of respondents who selected specific audiences (count and %) are represented in table 1. respondents were able to select more than one audience type for their covid-19 libguide while responding to this question on the survey. responses indicated many had overlapping or multiple audiences.   table 1 intended audiences for covid-19 libguides as selected by respondents (n = 20) intended audience number of respondents included (n = 20) % of respondents included health professionals 9 18.8% military 7 14.6% government 7 14.6% library staff 6 12.5% researchers 5 10.4% students 5 10.4% administration 4 8.3% other 3 6.3% public 1 2.1% patients 1 2.1%   figure 3 types of resources used in covid-19 libguides as reported by respondents.   curation, organization, and marketing   as seen in figure 3, respondents shared a wide variety of resources on their libguides. government websites were included in all federal libguides, and databases, guidelines, international websites, journal websites, and specific articles were included in more than half of respondents’ covid-19 libguides. less than a third of respondents indicated they included social media or internal documents on their libguides.   regarding reuse, only 3 respondents reused all or part of another library’s covid-19 libguide. thirteen respondents did not reuse all or part of another library’s covid-19 libguide, instead creating and curating their content from scratch. 1 respondent granted reuse permission to multiple libraries to reuse their covid-19 libguide, 4 did not grant reuse permission to other libraries from requests received, and 10 were not asked for their libguide to be reused.   there were 12 free-text responses on how libraries determined sources most applicable and appropriate for their covid-19 libguides. to determine which sources were most appropriate for a libguide, respondents reported consideration of the following: quality (7 responses), user feedback/focus (5), what others were doing (4), convenience (2), access/availability (2), and current topics (1).    one respondent reported great freedom and latitude in decision making, focusing content curation and resources based on conversations and feedback with their patron base and communication from leadership. they also were able to respond to needs based on common questions or ongoing topics of conversation within their organization. another respondent sought authoritative information based on recommendations from healthcare professionals.   according to free-text responses, some respondents (greater than four) looked primarily at user feedback, or focused on their user base, while others looked more toward what other libraries were doing to guide decision-making in source curation. both groups sought to have current topics and up-to-date information. in several instances, respondents looked to other individual information professionals’ pages, such as other federal health libraries, or to institutions that took a lead in information curation during the early part of the pandemic (e.g., johns hopkins university) to guide their content curation. some respondents sought broad information, rather than in-house content, while others specifically tried to tailor their pages to local resources and internal guidance.   at least two respondents mentioned convenience, access, and availability of resources being a deciding factor. some of these were resources sent to them by their agency or easily copied or downloaded from another federal agency. others used resources that were open access, or their content curation was driven by what vendors were providing through their license agreements.   across the board, respondents agreed quality of information was critical in their source curation. peer-reviewed and authoritative sources, such as federal government agencies (e.g., cdc and nlm) were cited often. there was also a focus on finding non-biased coverage of information and looking to leading institutions in the pandemic for clear direction.   twelve respondents reported four different ways of organizing their content within their libguides, which included: by content (7), resource type (6), audience (4), and chronologically (1). content included references to specific information provided by resources. while most mentions of content were specific to the spread and response to covid-19 (“disease tracking” and “local information”), at least one respondent also included information on teleworking (“teleworking during covid tips”). resource type included mentions of different publication formats, although only two respondents provided specifics (“special reports, peer-reviewed literature, preprints”; “databases, and then other research tools”). several respondents mentioned creating resource pages for specific users, particularly for medical staff (“nurses, physicians”) and patients. only one respondent mentioned ordering information chronologically so the most current information would precede older information.   sixteen respondents provided free text answers regarding marketing their new covid-19 libguide pages. to market these guides, respondents implemented the following: web/internal announcements (12), email (9), social media (2), targeted audience messaging (3), and word of mouth (1). all but one free-text response relied on electronic means of marketing. most respondents noted using a combination of two marketing practices (9), and one respondent utilized a combination of three. included in the web/internal announcements category were agency intranets, web notices, newsletters, webpage announcements, blogs, or online message boards. social media platforms mentioned in the responses included facebook, twitter, instagram, and a more general “social media channels.” targeted audience messaging referred to marketing towards leadership, relevant department heads, or specific students. while the question referred to the marketing of new libguide pages, one respondent provided that they had not “been diligent about updates.”   engagement, quality, and future plans   eleven respondents provided free text responses discussing how they measure engagement, including feedback, of their covid-19 libguide page(s). eight respondents (73%) utilized springshare’s integrated libguide web analytics, five respondents (45%) engaged patrons for feedback directly, and one library reported not currently measuring engagement. respondents using libguide’s web analytics mentioned usage statistics or page view statistics, or more broadly referenced springshare statistics. one respondent noted that “for a few months [the covid-19 libguide] was our most visited libguide page.” personal feedback included responses such as e-mails, phone calls, narratives, direct contact, and patron engagement. one respondent shared that their library tries “to engage our patrons and encourage them to submit relevant content/resources. if they provide any input, it is discussed and responded to.”   to measure the quality of covid-19 libguides, respondents relied on patron feedback (15%), analytics (7.7%), and peer/self-review (38%). three (23%) respondents were unsure or did not implement any quality measures. several respondents specifically mention taking audience needs and expectations into consideration, and one respondent mentioned directly responding to input if provided. peer/self-review described by free-text responses explicitly mentioned personal expectations or group curation efforts to produce a quality libguide (“personal high standard”; “curated by a team”). only one respondent mentioned continuous review of information in order to remove “incorrect or outdated links.” one respondent also used analytics, such as site visits, as a measure for quality.   a final question asked respondents to describe their plans for the covid-19 libguide once the pandemic ends. post-pandemic, six respondents planned to maintain and regularly update their covid-19 libguides, three planned to keep the guides viewable but no longer update, one planned to archive the page, and seven were unsure or had no plans at the time of the survey.   discussion   our findings show that, of the federal library respondents who used libguides, over half (61%) were using libguides to share covid-19-related information. this value indicates that libguides were actively being used to disseminate critical public health information in a timely manner. it is important to note that of the respondents who used libguides, those who did not use them to disseminate covid-19 information were often constrained by job responsibilities or another agency department having authority over covid-19 information dissemination.   at the beginning of the pandemic, the wellcome trust initiated a statement calling on funders, researchers, and publishers to ensure that relevant data is openly accessible to ensure a prompt health response globally (trust, 2020). because of this collaborative statement, many major publishers or journals created open access covid-19 resources and marketed them to their library customers. such resources were often used (64.7%) in federal library covid-19 libguides. such a statement ensures broad access to crucial information regardless of existing licenses. more libraries, regardless of their purchased content or operating budget, were able to share timely articles on covid-19.   there was heterogeneity with respect to the degree to which federal libraries had control over their content and where the responsibility lay for disseminating pandemic information. respondents varied in accessibility to page viewing, time spent curating and updating information, best practices for measuring quality of the information provided, and plans for archiving the information post-pandemic.    to market covid-19 libguides, respondents used a variety of techniques throughout the pandemic. most utilized internal or web announcements or email to reach their target audiences, whereas some sent targeted messages to their audiences or had to rely on word of mouth. of note, at the beginning of the pandemic, many libraries transitioned to virtual services, at least temporarily (american library association, 2020), and some lost access to their normal avenues for marketing in this transition. while some libraries were able to continue with marketing on agency intranet sites, others may have needed to rely more heavily on email communication to the agency at large or to smaller specific audiences to spread information. additionally, libraries may have relied more heavily on word of mouth or non-traditional platforms such as social media sites to inform their customer base, given the widespread access limitations the federal workforce experienced during the work-from-home transition for non-essential personnel in the early months of the pandemic.    the number of covid-19 libguides created over time appears to reflect the early spread of the disease and widespread uncertainty beginning in spring 2020. for example, the highest number of covid-19 libguides were created in march (7) and april (6), correlating with mass telework options made available to government agencies via the office of personnel management (office of personnel management, 2021). as the pandemic continued, respondents were still creating new covid-19 libguides in fall 2020. it is unclear as to why respondents specifically created libguides six months or more into the pandemic, as the survey did not ask for respondents to provide rationale for creation.   researching, reviewing, compiling, building, organizing, and formatting a new libguide on an emergent topic from scratch can be daunting and time-consuming; 40% of respondents spent 6-10 hours and 45% spent more than ten hours on libguide creation. survey results, paired with free text responses from respondents about their curation process, suggest that the bulk of the curation and creation time was spent reviewing and selecting resources. to reduce the time burden and labor of this task, three respondents reused all or part of another library’s covid-19 libguides. and while audience types varied widely, types of resources included in a covid-19 libguide had evident overlap. for example, government websites were present in all respondent libguides, acting as a baseline resource across varying audience types. in future pandemics or public health crises, libraries with similar audience bases can save time and effort perhaps by collaborating to create a libguide with standard information as a starting point to share with one another.   creating such a collaborative effort would involve proactively locating similarly scoped libraries, contacting them for interest, and creating a collaborative plan for domain of responsibility. depending on how many libraries are participating, tasks like researching, reviewing, and compiling resources at the broadest level appropriate could be assigned. furthermore, libraries or librarians with the greatest collective expertise could build, organize, and format the content in libguides. all libraries participating in this collaborative effort could have permission to reuse and edit the libguide as they saw fit for their individual library. benefits of taking part in such a collaboration include saved time and effort of library staff, both of which can impact personnel budgets. the libguide could also be peer-reviewed, as multiple library staff with different levels and areas of expertise could work collaboratively while engaging in constructive feedback.  an additional benefit could be giving back to the library science field and allowing these libguides to be available for reuse by libraries outside of the federal government who may not have the staff or means to create their own.   best practices in libguide design for specific audiences cannot be determined from this survey, as most respondents serve a wide variety of military, civilian, internal, and/or external audiences. however, general guidelines for where to start finding relevant public health information for the broadest audience could be informed by the most common resources used by respondents, to include government websites, databases, guidelines, international organizations, and journal websites. knowing where to start reviewing resources to curate could then potentially reduce the initial time and resource burden to create these guides.   limitations   due to the narrow scope of our research question and audience, addressing the use of libguides for covid-19 information in only federal libraries, we may have missed other findings from the general or specialized library populations (medical, academic, public, research, law, etc.) that also contribute to best practices for disseminating information. additionally, we only explored the use of libguides, and we recognize that while this is a broadly-used platform, many federal libraries do not use it and may be providing curated content and library services related to the pandemic in other ways. while this focus was chosen purposefully given the dramatically different relationships that exist between federal libraries and the populations they serve, and the broader library community and their patrons, it may still have excluded important findings. with these limitations in mind, this study can serve as a springboard for future inquiries into the literature and studies in the library community for pandemic planning, preparedness, and response.    conclusion   the ability to quickly disseminate information is imperative during a public health crisis, let alone a global pandemic. the emergence of covid-19 put the u.s. health response to the test as organizations at every level scrambled to provide answers to an ever-growing list of questions. federal libraries found themselves in a unique position of providing services remotely while also attempting to curate and provide quality covid-19 resources for their diverse patrons. this research surveyed how federal libraries used libguides to distribute covid-19 information throughout the course of the pandemic. federal libraries began publishing their covid-19 libguides as early as january 2020, when the u.s. announced the first case, with a spike in their creation in march 2020 as states began issuing guidance on lockdowns. creating a libguide is a time consuming process, and creating one on an ever-changing and rapidly growing topic requires dedicated time for consistent maintenance as information evolves. tracking engagement, eliciting and considering feedback, and determining quality of resources all helped shape covid-19 libguide content. results highlight the potential for future collaborative opportunities to streamline federal library public health response. this study provides valuable insight into the information-sharing process, which will help reduce the burden and save time for future libraries should there be another public health emergency.   disclaimer   the research protocol and online survey used in this study were approved by the walter reed national military medical center (wrnmmc) institutional review board (irb): wrnmmc-edo-2020-0535, 927350; and the defense health agency: department of defense (dod) survey license exemption (#9)-exempt #0053.   this research was supported in part by an appointment to the national library of medicine (nlm) research participation program. this program is administered by the oak ridge institute for science and education through an interagency agreement between the u.s. department of energy (doe) and the national library of medicine (nlm). orise is managed by orau under doe contract number de-sc0014664. all opinions expressed in this paper are the author's and do not necessarily reflect the policies and views of nlm, doe, or orau/orise.   the views expressed are those of the authors and do not reflect the official policy of the department of the army/navy/air force, department of defense, or u.s. government.   author contributions   sarah clarke: conceptualization (equal), formal analysis (supporting), investigation (equal), methodology (equal), project administration (lead), visualization (supporting), writing – original draft (equal), writing – review & editing (equal) emily shohfi: conceptualization (equal), formal analysis (supporting), investigation (equal), methodology (equal), project administration (supporting), visualization (lead), writing – original draft (equal), writing – review & editing (equal) sharon han: conceptualization (equal), formal analysis (lead), investigation (equal), methodology (equal), project administration (supporting), visualization (supporting), writing – original draft (equal), writing – review & editing (equal)   references   adhanom ghebreyesus, t. (2020). who director-general’s opening remarks at the media briefing on covid-19—11 march 2020 [interview]. world health organization. https://www.who.int/dg/speeches/detail/who-director-general-s-opening-remarks-at-the-media-briefing-on-covid-19---11-march-2020   ali, m. y., & gatiti, p. (2020). the covid-19 (coronavirus) pandemic: reflections on the roles of librarians and information professionals. health information & libraries journal, 37(2), 158-162. https://doi.org/10.1111/hir.12307    american library association. (2020). libraries respond: covid-19 survey. http://www.ala.org/tools/covid/libraries-respond-covid-19-survey   centers for disease control and prevention. (2020a). coronavirus disease 2019 (covid-19) – symptoms. centers for disease control and prevention. retrieved may 8, 2020, from https://www.cdc.gov/coronavirus/2019-ncov/symptoms-testing/symptoms.html   centers for disease control and prevention. (2020b). coronavirus disease 2019 (covid-19) in the u.s. centers for disease control and prevention. centers for disease control and prevention. retrieved may 7, 2020, from https://www.cdc.gov/coronavirus/2019-ncov/cases-updates/cases-in-us.html   clarke, s. c., han, s., & shohfi, e. e. (2021). federal library utilization of libguides to disseminate covid-19 information survey data [data set]. https://doi.org/10.17605/osf.io/swf34   creech, l. (2020, april 17). supporting patrons during the pandemic using springshare tools.  the springy share. https://blog.springshare.com/2020/04/17/supporting-patrons-during-the-pandemic-using-springshare-tools/   glasow, p. a. (2005). fundamentals of survey research methodology. the mitre corporation. https://www.mitre.org/sites/default/files/pdf/05_0638.pdf      jackson, c. (2014, october 22). chapel hill librarians helping ebola fight in west africa. north carolina public radio. https://www.wunc.org/post/chapel-hill-librarians-helping-ebola-fight-west-africa   lai, c. c., shih, t. p., ko, w. c., tang, h. j., & hsueh, p. r. (2020). severe acute respiratory syndrome coronavirus 2 (sars-cov-2) and coronavirus disease-2019 (covid-19): the epidemic and the challenges. international journal of antimicrobial agents, 55(3), 105924. https://doi.org/10.1016/j.ijantimicag.2020.105924    landgraf, g. (2014, october 3). tracking ebola in liberia. american libraries. https://americanlibrariesmagazine.org/2014/10/03/tracking-ebola-in-liberia/    leibiger, c. a., & aldrich, a. w. (2013). “the mother of all libguides”: applying principles of communication and network theory in libguide design. imagine, innovate, inspire: proceedings of the acrl 2013 conference, indianapolis, in, united states. https://www.ala.org/acrl/sites/ala.org.acrl/files/content/conferences/confsandpreconfs/2013/papers/leibigeraldrich_mother.pdf   love, c. b., arnesen, s. j., & phillips, s. j. (2015). ebola outbreak response: the role of information resources and the national library of medicine. disaster medicine and public health preparedness, 9(1), 82-85. https://doi.org/10.1017/dmp.2014.108    united states office of personnel management. (2021, february 19). policy, data, oversight: coronavirus disease 2019 (covid-19). united states office of personnel management. retrieved june 24, 2021, from https://www.opm.gov/policy-data-oversight/covid-19/   medical library association (2021). disaster information specialization. medical library association. retrieved january 1, 2021, from https://www.mlanet.org/page/disaster-information-specialization   schuchat, a. (2020). public health response to the initiation and spread of pandemic covid-19 in the united states, february 24-april 21, 2020. morbidity and mortality weekly report, 69(18), 551-556. https://doi.org/10.15585/mmwr.mm6918e2     springshare. (2020a). guides: create or copy a guide. springshare. retrieved january 1, 2021, from https://ask.springshare.com/libguides/faq/770#section2   springshare. (2020b). libguides: curate resources, share knowledge, publish content. springshare. retrieved january 27, 2021, from https://www.springshare.com/libguides/   springshare. (2021). libguides community. springshare. retrieved january 27, 2021, from https://community.libguides.com/   stein, r. (2020, january 24). 2nd u.s. case of wuhan coronavirus confirmed. national public radio. https://www.npr.org/sections/health-shots/2020/01/24/799208865/a-second-u-s-case-of-wuhan-coronavirus-is-confirmed   talia. (2020, march 16). tips & tricks for the remote-first world. the springy share. https://blog.springshare.com/2020/03/16/tips-tricks-for-the-remote-first-world/   ware, d. (2021). federal libraries using libguides from springshare! in e. shohfi (ed.). personal email communication.   wellcome trust. (2020, january 31). sharing research data and findings relevant to the novel coronavirus (covid-19) outbreak [press release]. https://wellcome.org/press-release/sharing-research-data-and-findings-relevant-novel-coronavirus-ncov-outbreak   world health organization. (2020). naming the coronavirus disease (covid-19) and the virus that causes it. world health organization. retrieved may 8, 2020, from https://www.who.int/emergencies/diseases/novel-coronavirus-2019/technical-guidance/naming-the-coronavirus-disease-(covid-2019)-and-the-virus-that-causes-it   appendix a survey questions   section 1. libguide creation 1.      does your library use libguides? o   yes o   no 2.      are your libguides publicly viewable? o   yes o   no o   some pages 3.      has your library used libguides to disseminate covid-19 information? o   yes o   no o   unsure 4.      if no to question 3, why not? o   [free text] 5.      do you have primary responsibility over covid-19 content on your libguides? o   yes o   no o   shared 6.      what month did your library begin curating covid-19 information on your libguide? o   december 2019 o   january 2020 o   february 2020 o   march 2020 o   april 2020 o   may 2020 o   june 2020 o   unsure 7.      who is the intended audience(s) for your covid-19 libguide. please select all that apply. o   general public o   health professionals o   military o   researchers o   administrators o   government o   patients o   students o   library staff o   other: [free text] 8.      estimate how much time (in hours) was spent on the initial creation/curation of covid-19 libguide information? o   <1 hour o   1-5 hours o   6-10 hours o   10+ hours 9.      estimate how much time (in hours) is spent each week updating covid-19 libguide information? o   <1 hour o   1-5 hours o   6-10 hours o   10+ hours   section 2. libguide curation and design 10.   which of the following information resource types are linked to or included in your libguides? please select all that apply. o   databases o   journal websites o   specific articles o   government websites o   international websites o   non-profit websites o   university websites o   video/audio o   internal documents o   guidelines o   academic/professional societies o   news media o   social media o   other: [free text] 11.   how did you determine which sources were most appropriate for your libguide? o   [free text] 12.   describe how you organized the content within your covid-19 libguide. o   [free text] 13.   did your library reuse any part of another library’s existing covid-19 related libguide? o   yes reused all or part of another library’s covid-19 libguide o   no did not reuse another library’s covid-19 libguide o   unsure 14.   did your library grant permission to another library to reuse your covid-19 related libguide? o   yes – we granted reuse permission o   no – we did not grant reuse permission o   n/a – no library asked permission for reuse o   unsure   section 3. libguide engagement and preservation 15.   how does your library market new libguide pages (related to covid-19) to patrons? o   [free text] 16.   how do you measure engagement, such as feedback, of your covid-19 libguide? o   [free text] 17.   how do you measure quality of your covid-19 libguide? o   [free text] 18.   currently, what are your plans for this page post-pandemic? o   maintain/update regularly o   viewable, but no longer updated o   archive it (offline or as a hidden/unpublished page) o   not sure/no plans o   other: [free text]   section 4. demographic questions 19.   which federal government agency/department/division does your library serve? o   [free text] 20.   what is the name of your library? (this information will not be shared – it’s to ensure we don’t record duplicate information) o   [free text] 21.   what is your library’s zip code? (this information will not be shared –  it’s to ensure we don’t record duplicate information) o   [free text]       evidence based library and information practice evidence based library and information practice 2009, 4.4     evidence based library and information practice         news/announcements    call for papers: becoming evidence based: a research in practice mini‐conference      © 2009. this is an open access article distributed under the terms of the creative commons attribution   license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and   reproduction in any medium, provided the original work is properly cited.      the evidence based librarianship interest  group (eblig) is hosting a preconference  prior to the canadian library association  annual conference on june 2nd, 2010, in  edmonton, alberta, canada. this half‐day  mini‐conference focuses on evidence based  practice and its incorporation into professional  decision making. becoming evidence based,  which features keynote speaker lisa m. given,  will introduce attendees to current dialogue  and research in evidence based library and  information practice.   the evidence based librarianship interest  group (eblig) invites submissions for  contributed papers and posters including both  original research and innovative applications  of evidence based practice (ebp) in library and  information practice. we welcome examples of  ebp from all library sectors.  subject areas may include, but are not limited  to:  o library and information service  administration  o technical improvements  o service delivery  o marketing and communications  o collection development  o ebp in other disciplines with  application in ebl  submissions should include:  1. title of presentation or poster  2. abstract of no more than 250 words  3. primary presenter’s name, affiliation,  email address, phone number and  brief biography.  4. if there is more than one presenter, the  names, affiliations, email addresses,  phone numbers and brief biographies  of all presenters.  5. av equipment needs or other special  needs for your presentation.  presentations at the mini‐conference will be  limited to approximately 20‐30 minutes. all  proposals will be evaluated by a refereeing  committee representing eblig.  submissions must be submitted by february  15, 2010 and can be emailed to:  renee de gannes‐marshall  eblig co‐convenor  renee.d.marshall@gmail.com  successful candidates will be notified by  march 15, 2010 via email.  please note: expenses associated with  attending the eblig mini‐conference and the  cla annual conference will be the  responsibility of the author(s)/presenter(s) of  accepted papers. the registration fee for this  half day session will be waived for all  speakers.   99 evidence summary   publons peer evaluation metrics are not reliable measures of quality or impact   a review of: ortega, j. l. (2019). exploratory analysis of publons metrics and their relationship with bibliometric and altmetric impact. aslib journal of information management, 71(1), 124–136. https://doi.org/10.1108/ajim-06-2018-0153     reviewed by: scott goldstein web librarian appalachian state university libraries boone, north carolina, united states of america email: goldsteinsl@appstate.edu   received: 1 may 2019                                                                    accepted:  14 july 2019      2019 goldstein. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29579     abstract   objective – to analyze the relationship between scholars’ qualitative opinion of publications using publons metrics and bibliometric and altmetric impact measures.   design – comparative, quantitative data set analysis.   setting – maximally exhaustive set of research articles retrievable from publons.   subjects – 45,819 articles retrieved from publons in january 2018.   methods – author extracted article data from publons and joined them (using the doi) with data from three altmetric providers: altmetric.com, plumx, and crossref event data. when providers gave discrepant results for the same metric, the maximum value was used. publons data are described, and correlations are calculated between publons metrics and altmetric and bibliometric indicators.   main results – in terms of coverage, publons is biased in favour of life sciences and subject areas associated with health and medical sciences. open access publishers are also over-represented. articles reviewed in publons overwhelmingly have one or two pre-publication reviews and only one post-publication review. furthermore, the metrics of significance and quality (rated on a 1 to 10 scale) are almost identically distributed, suggesting that users may not distinguish between them. pearson correlations between publons metrics and bibliometric and altmetric indicators are very weak and not significant.   conclusion – the biases in publons coverage with respect to discipline and publisher support earlier research and suggest that the willingness to publish one’s reviews differs according to research area. publons metrics are problematic as research quality indicators. most publications have only a single post-publication review, and the absence of any significant disparity between the scores of significance and quality suggest the constructs are being conflated when in fact they should be measuring different things. the correlation analysis indicates that peer evaluation in publons is not a measure of a work’s quality and impact.   commentary   the study is the first in-depth examination of articles in publons, a web platform that allows users to make public their peer-review reports for journals as well as rate articles on quality and relevance. previous studies looking at publons have focused on who uses the service rather than what articles are reviewed. this study sheds more light on two questions that have been asked in the literature. first, what is the relationship between traditional bibliometric indicators and peer subjective valuations (interpreted more broadly than refereed peer review as it encompasses other preand post-publication reviews)? in a comparable study focusing on faculty of 1,000 (f1000) recommendations, waltman and costas (2014) found only a weak correlation between peer evaluations and citations, suggesting that the measures may capture different types of impact. second, in what ways do the data providers used in scientometric analysis provide discrepant data? zahedi and costas (2018), for instance, conducted the most exhaustive comparison of altmetric providers and found that coverage of publications between them varies widely.   this commentary relies on perryman’s (2009) critical appraisal tool for bibliometric studies. the study fares very well when evaluated against it. for instance, the author clearly states the research questions after a literature review that motivates the need for a more in-depth look at publons. the data are analyzed using appropriate statistical methods given the research objectives, and the results are graphically displayed in an appropriate manner given the types of analyses performed. the correlations between all the metrics are presented in a correlation matrix, which, although missing a colour legend, is an excellent way to visualize this type of data. limitations of the data sources are also considered. sufficient, though perhaps minimal, detail was provided on how the author scraped the data from publons; it would be difficult for a novice researcher in this area to replicate the study. it is not exactly clear whether scraping the website violates publons’ terms of service, which explicitly state that “the correct way to access these data is via our api.”   this study highlights a couple of implications for information professionals. peer evaluations of scholarship outside of traditional (double-) blind peer review (preand post-publication review and open peer review as adopted by some journals during the refereeing process) is an emerging practice. librarians should familiarize themselves with these new methods of review and be in a position to inform scholars of their opportunities and challenges (see ford, 2016). information professionals may also take away from this study a more measured view of the role peer review plays in the assessment of scholarly literature. although this study suggests publons metrics are not suitable quality indicators for reasons that are specific to the publons platform, it would be wise to remember that peer review is not an infallible measure of quality, let alone impact. an evidence-based critical appraisal of a scholarly work does not simply consist of checking the peer review status.   references   ford, e. (2016). opening review in lis journals: a status report. journal of librarianship and scholarly communication, 4(general issue), ep2148. https://doi.org/10.7710/2162-3309.2148   perryman, c. (2009). evaluation tool for bibliometric studies. retrieved from https://www.dropbox.com/l/scl/aaal7luzple90fxfnbv5hcnoz0ctlh6rqrs   waltman, l., & costas, r. (2014). f1000 recommendations as a potential new data source for research evaluation: a comparison with citations. journal of the association for information science and technology, 65(3), 433–445. https://doi.org/10.1002/asi.23040   zahedi, z., & costas, r. (2018). general discussion of data quality challenges in social media metrics: extensive comparison of four major altmetric data aggregators. plos one, 13(5), e0197326. https://doi.org/10.1371/journal.pone.0197326     evidence based library and information practice evidence based library and information practice 2010, 5.1     evidence based library and information practice         evidence summary    information professionals’ attitudes influence the diffusion of information and  communication technologies    a review of:  rabina, d. l., & walczyk, d. j. (2007). information professionals’ attitude toward the adoption of  innovations in everyday life. information research, 12(4), 1‐15.     reviewed by:   kristen lee young  liaison and outreach librarian   university of maryland at baltimore  maryland, united states of america  email: kyoung@hshsl.umaryland.edu     received: 16 nov. 2009          accepted: 24 jan. 2010       2010 young. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which  permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly  attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the  same or similar license to this one.      abstract     objective – this study examined the general  characteristics and patterns of librarians in  connection with their willingness to adopt  information and communication technologies.    design – online questionnaire.      setting – general distribution to information  professionals through online inquiry. more  than 70% of responders worked in public or  academic libraries.      subjects – librarians and library staff at mostly  public and academic libraries.    methods – the study was conducted during a  two week period in april 2006 through an  online questionnaire that was sent to library  and librarian‐related electronic mail lists. the  questionnaire was divided into two parts and  contained a total of 39 questions. part one  contained eight questions that asked for  demographic data and the respondent’s daily  attitude toward the adoption of information  and communication technologies. questions  regarding age, number of years worked in a  library, career, type of library environment  worked in, and primary responsibilities within  that environment were asked. for one question  the respondents were asked to identify which  of the categories they fall under when adopting  a new technology. the results from part one  were used to consider the innovativeness of  librarians. the results from part two were used  for a study of opinions on innovations and their  relative advantage.      147 mailto:kyoung@hshsl.umaryland.edu evidence based library and information practice 2010, 5.1  main results – a total of 1,417 responses were  received. of those, 1,128 were fully completed  and considered valid and used for inquiry. the  majority of respondents worked in public or  academic libraries. nine hundred and twenty‐ six respondents, or 88%, were from the u.s. and  represented more than 300 distinct zip codes.  two hundred and two respondents, or 12%,  were international respondents.     this study notes that the sociologist, everett  rogers, identified and defined five adopter  categories in 1958. those categories are:  innovators, early adapters, early majority, late  majority, and laggards. the findings of this  study indicate that regardless of the  demographic variables considered, more than  60% of respondents, the majority of librarians  surveyed, fall into two contrasting adapter  categories: early adopters and early majority.  the study suggests that the efficient and  effective diffusion of new technologies in  library settings may be difficult.  three  problematic areas among librarians for the  dissemination of innovation were identified:  conflicting opinions among multiple opinion  leaders, deceleration in the rate of adoption,  and improper re‐invention.  the findings of the  study also suggest that “contrary to common  beliefs, librarians in academic or special  libraries are no more innovative than public or  school librarians” (conclusion, ¶3).     conclusion – the study concludes that  librarians’ attitudes are unevenly distributed  with most either accepting new innovations or  being late adopters. the variables of age, role,  tenure, and library type had little impact on the  approach of the professional toward  innovation. the identification of the three  problem areas: opinion leadership, deceleration  of adoption, and improper re‐invention,  represents where more time and effort may  need to be spent to make the implementation of  new technology a smoother process.        commentary     this study will be of interest to all persons that  are called upon to implement new technologies  because it gives an inside look at how and why  individuals adapt to new technologies. it drew  on the opinions of a number of diverse  professionals, but asked for their own opinions  concerning their innovativeness in relation to  their peers. as with all studies employing self‐ report, personal biases may have influenced the  results. the study suggests that further research  using case studies of individual libraries should  be completed.     the authors suggest that future surveys could  be conducted within a set of libraries within a  geographic area in order to assess local  attitudes, and then plan and implement  successful technology training accordingly. the  article provides an excellent summation and in  depth review of the existing literature, and  thereby demonstrates the contextual  framework for this survey. it would have been  helpful if the authors had provided access to  the electronic survey that could be used to  assist in the replication or refinement of the  study at a different institution at a later date.      when the questionnaire was sent out to library‐ and librarian related electronic mail lists, it was  intended to target a u.s. audience only.  however, 12% of the responses were from  international respondents, which demonstrates  the wide “nets” and webs that are frequently  created with the use of electronic mail lists. the  inclusion of these international responses  introduces many variables that are difficult to  evaluate or consider in the absence of more  information concerning the relative needs and  resources of the responders.      as the study was unrestrictive, the authors  suggest that further research would benefit  from a more controlled sample. with a more  controlled sample local attitudes could be  assessed and technology training could be  designed and implemented.      this research alerts us to the difficulty that we  are likely to face when implementing new  technologies in our institutions. as the author  notes, this study provides reason to conduct  further research using case studies of  individual libraries.    148 review article   librarian co-authored systematic reviews are associated with lower risk of bias compared to systematic reviews with acknowledgement of librarians or no participation by librarians   mikaela aamodt senior librarian medical library, university of oslo oslo, norway email: mikaela.aamodt@ub.uio.com    hugo huurdeman postdoctoral researcher university of amsterdam amsterdam, the netherlands email: h.c.huurdeman@uva.nl     hilde strømme adviser medical library, university of oslo oslo, norway email: hilde.stromme@ub.uio.no    received: 2 july 2019                                                                      accepted: 22 july 2019      2019 aamodt, huurdeman, and strømme. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29601     abstract   objective to explore the prevalence of systematic reviews (srs) and librarians’ involvement in them, and to investigate whether librarian co-authorship of srs was associated with lower risk of bias.   methods srs by researchers at university of oslo or oslo university hospital were counted and categorized by extent of librarian involvement and assessed for risk of bias using the tool risk of bias in systematic reviews (robis).   results of 2,737 identified reviews, 324 (11.84%) were srs as defined by the review authors. of the 324 srs, 4 (1.23%) had librarian co-authors, in 85 (26.23%) librarians were acknowledged or mentioned in the methods section. in the remaining 235 srs (72.53%), there was no clear evidence that a librarian had been involved. librarian co-authored srs were associated with lower risk of bias compared to srs with acknowledgement or no participation by librarians.   conclusion srs constitute a small portion of published reviews. librarians rarely co-author srs and are only acknowledged or mentioned in a quarter of our sample. the quality and documentation of literature searches in srs remains a challenge. to minimise the risk of bias in srs, librarians should advocate for co-authorship.     background   over the past decades, systematic reviews (srs) have increased as a research methodology in the health sciences, and they have a great influence on healthcare. in fact, evidence suggest that srs are the most commonly cited study type in clinical research (patsopoulos, analatos, & ioannidis, 2005). the foundation of every high-quality systematic review is a carefully designed and well-conducted literature search to avoid missing key studies and minimize bias. in addition to a comprehensive literature search, researchers should aim for full transparency when reporting how studies were identified and what was done to minimize bias, so that readers are able to determine the validity of the results (moher, liberati, tetzlaff, altman, & the prisma group, 2009) and to prevent suboptimal srs (ioannidis, 2016). librarians and information specialists (hereafter referred to as librarians) have a significant role to play in expert searching and documentation of search strategies, in the systematic review process. the importance of including a librarian when conducting the literature search has been widely recognized by prominent organisations such as the cochrane collaboration, the campbell collaboration, and the joanna briggs institute (aromataris & munn, 2017; higgins, churchill, chandler, & cumpston, 2017; the campbell collaboration, 2019). previous studies have shown that systematic review search quality and quality of reporting is a challenge (golder, loke, & mcintosh, 2008; koffel & rethlefsen, 2016; toews, 2017), even when librarians are consulted (meert, torabi, & costella, 2016; rethlefsen, farrell, osterhaus trzasko, & brigham, 2015). meert et al. (2016) and rethlefsen et al. (2015) found that having a librarian as a review team member or co-author improved the quality of searches and reporting. in preparation for developing a formal systematic review service, ross-white (2016) explored librarians’ involvement in systematic reviews at queen’s university, canada. they found that in 231 systematic reviews, librarian co-authorship was granted in 31 and librarians were acknowledged in 36, however they did not investigate whether or not librarian involvement had an impact on the quality of searches and reporting.   rethlefsen et al. (2015) examined the librarians’ impact on quality of reporting in srs from high-impact general internal medicine journals and meert et al. (2016) looked at srs in the twenty highest impact factor paediatric journals. our aim is to investigate whether the same tendency is present in srs published by researchers at the two institutions our library serves: the university of oslo and oslo university hospital. in addition, we explore the prevalence of systematic reviews compared with other types of reviews published by researchers affiliated with these institutions. the medical library at the university of oslo is about to establish a service for researchers embarking on a systematic review. using published research from our own researchers will give us a solid knowledge base when discussing and deciding how our library’s contribution to srs is best invested.   in this study, we explore the prevalence of systematic reviews and librarians’ involvement in them, and further, investigate whether librarian co-authored systematic reviews are associated with lower risk of bias compared to srs with acknowledgement of librarians or no participation by librarians.   methods   publication of systematic reviews have increased rapidly since the beginning of the 1990s (ioannidis, 2016). this increase was also observed in our institutions. before 2006, our library did not receive many requests from researchers for help with literature searches for srs. this analysis therefore includes systematic reviews published between 2007 and mid-2018. to collect systematic reviews, we conducted a search in scopus on 28 june 2018 (appendix a). we searched for oslo in the tab affiliations, where we found 15 university of oslo or oslo university hospital affiliations to include in our analysis. further, the search was limited to document type review and source type journal. with this search strategy, we could also get an understanding of the types of reviews our researchers were undertaking. a research assistant manually divided the scopus search results into review categories (systematic and non-systematic) based on how the review authors named the review. no assessment of the method was done to evaluate if the systematic review was conducted in accordance with a specific handbook. we included systematic reviews that had at least one author affiliated with the university of oslo or oslo university hospital. the following publication types were excluded: method articles (systematic review as topic), comments, letters, editorials, guidelines, book reviews, and errata.   to analyse the librarians’ involvement, two investigators reviewed the articles in full-text. we read the methods section, the acknowledgements section, and the author affiliation section in all the systematic reviews. additionally, we searched for librar or specialist or thank in the pdfs in case a librarian or information specialist was mentioned elsewhere in the document. we also examined search strategies when available.   the publications were divided into three categories: 1) librarian co-author; 2) librarian mentioned in the methods section or acknowledged; 3) librarian not mentioned. we then assessed the quality of the literature searches in a selection of 12 publications, four from each category using the risk of bias in systematic reviews (robis) tool (whiting et al., 2016). the rationale behind choosing four publications from each category was that only four publications had a librarian co-author, and we wanted an equal sample from each of the categories. based on the publication years of these four reports, we used a random number generator (haahr, 2019) to randomly select four publications with the same publication year from each of the other two categories: librarian mentioned or acknowledged, and librarian not mentioned[1]. domain 2 of robis “identification and selection of studies”, questions 2.1-2.4 aims to assess whether any primary studies that would have met the inclusion criteria were missed (whiting et al., 2016, appendix a). the robis (whiting et al., 2016, appendix a) questions concerning the search strategies are:   did the search include an appropriate range of databases/electronic sources for published and unpublished reports? were methods additional to database searching used to identify relevant reports? were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible? were restrictions based on date, publication format, or language appropriate?   there is also a question about efforts to minimise error in selection of studies, but we did not apply this question in our assessment. the robis assessment was based on the information available in the article or as supplementary material.   results   we identified a total of 2,737 reviews published by an author affiliated with the university of oslo or oslo university hospital. the reviews were categorized manually by a research assistant. of the 2,737 reviews, 324 (11.84%) were systematic reviews as defined by the review authors. figure 1 reports the number of published srs per year from 2007 to mid-2018. the figure shows that srs have increased every year, except in 2010, with a sharp increase from 2014, and the plateau has not yet been reached.   figure 1 systematic reviews per year with author affiliated to university of oslo or oslo university hospital. note that 2018 only contains data up to june 28.   table 1 number and percent of librarians’ involvement librarians’ involvement co-authors acknowledged or mentioned in methods section no evidence of librarian involvement total number of srs number & percent 4 (1.23 %) 85 (26.23%) 235 (72.53%) 324 (100%)     the 324 srs were analysed to uncover to what extent researchers collaborated with librarians. the results can be seen in table 1. the involvement is distributed in the 324 srs as follows; four srs (1.23%) had librarian co-authors, in 85 srs (26.23%) librarians were acknowledged or mentioned in the methods section. in the remaining 235 srs (72.53%), there was no clear evidence that a librarian had been involved.   in our third analysis, we wanted to explore whether librarian authorship was associated with a lower risk of bias compared to systematic reviews with no librarian involved or systematic reviews where a librarian was mentioned or acknowledged but not accredited with co-authorship. we used the robis tool (whiting et al., 2016) to assess risk of bias in four publications in each of the three categories.   first, we present an overall assessment of risk of bias. as can be seen in table 2, five of the twelve publications were judged to have an unclear risk of bias. the main reason was that the complete search strategy was not published. we contacted authors of the publications which lacked search strategies (aas et al., 2014; krølner et al., 2011; manja, saugstad, & lakshminrusimha, 2017; skarphedinsson et al., 2015; sugrue, englund, solbrekke, & fossland, 2018). manja et al. (2017) sent us a description of the search but not the complete search strategy. skarphedinsson et al. (2015) sent us the complete search strategy. the other three (aas et al., 2014; krølner et al., 2011; sugrue et al., 2018), did not respond. since we were not able to obtain all search strategies we decided to use only the published material in our robis assessment. in the discussion we elaborate on the findings.   table 2 summary of risk of bias assessments librarian co-author aune, røislien, mathisen, thelle, and otterstad (2011) “the ‘smoker's paradox’ in patients with acute coronary syndrome: a systematic review” ruddox et al. (2013) “is 3d echocardiography superior to 2d echocardiography in general practice?: a systematic review of studies published between 2007 and 2012” skarphedinsson et al. (2015) “standard individual cognitive behaviour therapy for paediatric obsessive–compulsive disorder: a systematic review of effect estimates across comparisons” borgeraas, johnson, skattebu, hertel, and hjelmesæth (2018) “effects of probiotics on body weight, body mass index, fat mass and fat percentage in subjects with overweight or obesity: a systematic review and meta‐analysis of randomized controlled trials” low risk of bias low risk of bias unclear risk of bias low risk of bias librarian mentioned in methods section or acknowledged burger, kornør, klemp, lauvrak, and kristiansen (2011) “hpv mrna tests for the detection of cervical intraepithelial neoplasia: a systematic review” roe, soberg, bautz-holter, and ostensjo (2013) “a systematic review of measures of shoulder pain and functioning using the international classification of functioning, disability and health (icf)” aamotsmo and bugge (2014) “balance artistry: the healthy parent’s role in the family when the other parent is in the palliative phase of cancer — challenges and coping in parenting young children” sugrue et al. (2018) “trends in the practices of academic developers: trajectories of higher education?” high risk of bias high risk of bias high risk of bias unclear risk of bias librarian not mentioned krølner et al. (2011). “determinants of fruit and vegetable consumption among children and adolescents: a review of the literature. part ii: qualitative studies” robsahm et al. (2013) “body mass index, physical activity, and colorectal cancer by anatomical subsites: a systematic review and meta-analysis of cohort studies” aas et al. (2014) “a systematic review of cognitive function in first-episode psychosis, including a discussion on childhood trauma, stress, and inflammation” manja et al. (2017) “oxygen saturation targets in preterm infants and outcomes at 18-24 months: a systematic review” unclear risk of bias high risk of bias unclear risk of bias unclear risk of bias   here we present a more thorough presentation of the reasoning behind the rating. the full assessment of each publication is presented in appendix b.   robis question 2.1. did the search include an appropriate range of databases/electronic sources for published and unpublished reports?   according to the robis guidance on how to rate each question, it is anticipated that a minimum of searches in medline and embase should be conducted. in addition, robis requires electronic searches for unpublished studies. searches of material published as conference reports should be considered along with a search of research registries (e.g. clinicaltrials.gov). conference reports can have preliminary results and may therefore be considered as a source for unpublished reports, but we considered this to be insufficient as a source for unpublished studies. we answered this question with “yes” when electronic searches of at least medline and embase, or other databases that include conference reports and unpublished material, such as trials registries, were searched. searches that included medline and other databases with conference reports, but no other unpublished sources, were judged as “probably yes”. a summary of the ratings for this question is presented in table 3.   table 3 summary of ratings for robis question 2.1. did the search include an appropriate range of databases/electronic sources for published and unpublished reports? librarian co-author  aune et al. (2011) ruddox et al. (2013) skarphedinsson et al. (2015) borgeraas et al. (2018) probably yes probably yes yes yes librarian mentioned in methods section or acknowledged burger et al. (2011) roe  et al. (2013) aamotsmo and bugge (2014) sugrue et al. (2018) probably yes probably yes probably yes probably yes librarian not mentioned krølner et al. (2011) robsahm et al. (2013) aas et al. (2014) manja et al. (2017) probably yes no probably no no information   librarian co-author   all authors searched medline and embase, which satisfies the minimum requirement set by robis. we consider embase to be a satisfactory source for conference reports. however, only skarphedinsson et al. (2015) and borgeraas et al. (2018) searched for unpublished reports. see table 4  for a full list of databases searched.   table 4 databases searched in publications with a librarian co-author aune et al. (2011) ruddox et al. (2013) skarphedinsson et al. (2015) borgeraas et al. (2018) ·    medline ·    embase ·    central ·    medline ·    embase ·    pubmed (ahead of print) ·    medline ·    embase ·    psycinfo ·    amed ·    central ·    pubmed ·    lilacs ·    clinicaltrials.gov ·    who international clinical trials ·    isrctn register ·    opengrey ·    medline ·    embase ·    central ·    clinicaltrials.gov   librarian mentioned in methods section or acknowledged   burger et al. (2011) and roe et al. (2013) searched the minimum requirement of medline and embase. aamotsmo and bugge (2014) did not search embase, but our judgement is that since this is a sr of qualitative studies, the databases chosen are more relevant than embase. sugrue et al. (2018) did not search medline or embase, however, considering that the topic of the systematic review was higher education, we judge the databases searched to be more relevant. all authors searched at least one database with conference abstracts (e.g. embase, cinahl, psycinfo). burger et al. (2011), roe et al. (2013), and aamotsmo and bugge (2014) did not search for unpublished reports. we cannot determine if the ebsco search performed by sugrue et al. (2018) included databases of unpublished reports. see table 5 for a full list of databases searched.   table 5 databases searched in publications where librarian was mentioned in methods section or acknowledged burger et al. (2011) roe et al. (2013) aamotsmo and bugge (2014) sugrue et al. (2018) ·    medline ·    embase ·    cochrane library ·    medline ·    embase ·    pedro ·    cinahl ·    central ·    medline ·    psycinfo ·    cinahl ·    australian education index ·    eric ·    psycinfo ·    ebsco (database(s) not specified)   librarian not mentioned   krølner et al. (2011) searched a wide range of databases including the minimum requirement of medline and embase. several of the databases they searched include conference abstracts. however, they did not do any electronic searches for unpublished studies. aas et al. (2014) searched pubmed and psycinfo. these are relevant to the subject, but based on the robis criteria they should also have searched embase. they did no electronic searches for unpublished studies. robsahm et al. (2013) only searched pubmed. the article by manja et al. (2017) does not have a description of the literature search or search strategy, therefore cannot be assessed. see table 6 for a full list of databases searched.   table 6 databases searched in publications where librarian was not mentioned krølner et al. (2011) robsahm et al. (2013) aas et al. (2014) manja et al. (2017) ·    medline ·    embase ·    anthropology plus ·    cinahl ·    csa illumine (including eric) ·    econlit ·    sociological abstracts ·    social services abstracts ·    worldwide political science abstracts ·    international bibliography of the social sciences ·    psycinfo ·    web of science ·    pubmed ·    pubmed ·    psycinfo ·    no information of databases searched   robis question 2.2. were methods additional to database searching used to identify relevant reports?   according to robis, the authors should use methods additional to electronic searches to identify relevant reports such as reference checking or contacting experts. a summary of the ratings for this question is presented in table 7.   table 7 summary of ratings for robis question 2.2. were methods additional to database searching used to identify relevant reports? librarian co-author aune et al. (2011) ruddox et al. (2013) skarphedinsson et al. (2015) borgeraas et al. (2018) yes yes yes yes librarian mentioned in methods section or acknowledged burger et al. (2011) roe et al. (2013) aamotsmo and bugge (2014) sugrue (2018) no information no information yes no information librarian not mentioned krølner et al. (2011) robsahm et al. (2013) aas et al. (2014) manja at al. (2017) no information yes no information no information   librarian co-author   in all of the four publications co-authored by librarians, aune et al. (2011), ruddox et al. (2013), skarphedinsson et al. (2015) and borgeraas et al. (2018), reference lists were checked for additional studies.   librarian mentioned in methods section or acknowledged   aamotsmo and bugge (2014) checked reference lists. the other three, burger et al. (2011), roe et al. (2013) and sugrue et al. (2018), mentioned no other methods to identify relevant reports.   librarian not mentioned   robsahm et al. (2013) checked reference lists, while krølner et al. (2011), aas et al. (2014) and manja et al. (2017) mentioned no other methods to identify relevant reports.   robis question 2.3. were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible?   according to the robis guidance, a full search strategy that can be replicated is needed to fully assess this question. a summary of the ratings for this question is presented in table 8.   table 8 summary of ratings for robis question 2.3. were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible? librarian co-author aune et al. (2011) ruddox et al. (2013) skarphedinsson et al. (2015) borgeraas et al. (2018) yes yes no information yes librarian mentioned in methods section or acknowledged burger et al. (2011) roe et al. (2013) aamotsmo and bugge (2014) sugrue et al. (2018) yes yes no information no information librarian not mentioned  krølner et al. (2011) robsahm et al. (2013) aas et al. (2014) manja et al. (2017) no information probably no probably no no information   librarian co-author   three of the publications, aune et al. (2011), ruddox et al. (2013) and borgeraas et al. (2018), used a combination of controlled terms and text words appropriately. skarphedinsson et al. (2015) states that they searched for “an extensive list of synonyms for ocd, cbt, children and adolescents”, but since the full strategy is not attached to the article, we were not able to judge the comprehensiveness of the search. we received the search strategies from one of the librarian co-authors, and if the search strategy in skarphedinsson et al. (2015) had been published, the answer to this question would have been “yes”, and the overall judgement would have been low risk of bias.   librarian mentioned in methods section or acknowledged   two of the publications, burger et al. (2011) and roe et al. (2013), used a combination of controlled terms and text words appropriately. aamotsmo and bugge (2014) and sugrue et al. (2018) list relevant search terms but lack full search strategy and therefore cannot be assessed properly.   librarian not mentioned   since manja et al. (2017) does not describe or publish their search strategy, it cannot be assessed. aas et al. (2014) presents relevant search terms but not the full search strategy to be assessed. krølner et al. (2011) describes text words used in all databases. our judgement is that these terms are likely to retrieve many relevant studies, however since the full search strategy is not available we cannot assess it fully. robsahm et al. (2013) presents the search strategy from pubmed. it contains only text words and it is difficult to know whether or not these were mapped to mesh. in addition to terms for the topic, they have searched for terms to identify study designs, but in our opinion they should have searched for variations of these. they also restricted the search with the pubmed humans filter, which is based on mesh and thus excluded all articles not indexed with mesh.   robis question 2.4. were restrictions based on date, publication format, or language appropriate?   robis guidance states that no restrictions should be made to the search strategy in order to answer this question with “yes”, and that information on all three components is necessary to fully judge this question. none of the publications made any restrictions to publication format, but some were made based on language and date. we judged language restrictions to “probably no” instead of “no” if they included more languages than english (e.g. norwegian, danish, swedish, and german). we judged unjustified date restrictions as “no”. a summary of the ratings for this question is presented in table 9.   table 9 summary of ratings for robis question 2.4. were restrictions based on date, publication format, or language appropriate? librarian co-author  aune et al. (2011) ruddox et al. (2013) skarphedinsson et al. (2015) borgeraas et al. (2018) probably no probably no yes yes librarian mentioned in methods section or acknowledged burger et al. (2011) roe et al. (2013) aamotsmo and bugge (2014) sugrue et al. (2018) no no no no information librarian not mentioned krølner et al. (2011) robsahm et al. (2013) aas et al. (2014) manja et al. (2017) yes no no information no information   librarian co-author   borgeraas et al. (2018) did not make any limitations to the search. two of the publications, aune et al. (2011) and ruddox et al. (2013), restricted the search to english, german, norwegian, danish, and swedish language. ruddox et al. (2013) also restricted their search to identify studies between 2007-2012, however, this was justified in the introduction. skarphedinsson et al. (2015) state that they have not done any language restrictions.   librarian mentioned in methods section or acknowledged   sugrue et al. (2018) lacks search strategy and does not describe any restrictions in the methods section, hence there is not enough information to assess this question. in three of the publications, aamotsmo and bugge (2014), burger et al. (2011), and roe et al. (2013), restrictions were made to publication date, none of which were justified in the article. roe et al. (2013) also restricted the search to english, norwegian, danish, and swedish language. burger et al. (2011) had no actual restrictions on language in the search strategy even though they state in the methods chapter that publications was restricted to english and scandinavian languages.   librarian not mentioned   robsahm et al. (2013) made restrictions to english language. one of the publications, krølner et al. (2011), state that they made no restrictions to either publication year or to language. aas et al. (2014) and manja et al. (2017) do not mention any restrictions.    discussion   the popularity of srs has grown immensely in the past decades (ioannidis, 2016). many research funders now request a systematic review of previous research as a part of grant applications, and phd candidates at the university of oslo are now allowed – and encouraged to – include one sr in their thesis which contributes to the increase. our first analysis shows that although the majority of reviews by researchers at the university of oslo and oslo university hospital are non-systematic, the number of srs are growing and continue to do so.   our second analysis shows that librarians’ involvement in srs at the two institutions seems to be rather low. there could be cases where the review authors did get help from a librarian, but failed to mention this. in other cases, they have not involved a librarian at all. there could be several reasons for this, one being that they are not aware of the services the library offers. some researchers are experienced searchers, and do the searching and documentation themselves. however, as our third analysis shows, the risk of bias is higher when a librarian is not a co-author. compared to the results from ross-white (2016), librarian involvement was much lower in our sample. of the 31 librarian co-authored srs in the ross-white sample, as many as 19 were co-authored by a nursing liaison librarian, showing that they have succeeded in advocating for co-authorship in the school of nursing. based on this, we see that there is room for a substantial improvement in our collaboration with systematic review authors.   due to our small sample size, we are not able to draw strong conclusions, but we can observe that overall there are some positive differences when a librarian co-authors the article. in the analysed publications, search strategies are more comprehensive and in particular the documentation is better when a librarian is co-author.    in the publications where a librarian was only mentioned or acknowledged, varying comprehensiveness of search strategies and some lack of documentation resulted in risk of bias from unclear to high risk of bias. the high risk of bias judgement of three publications was mainly due to unjustified date restrictions which may have resulted in relevant studies being missed.   when a librarian was not mentioned in the review, the reporting of search strategies was far less complete. three of the publications did not report the actual search strategy, and the one that did, only searched pubmed. manja et al. (2017) did send us a description of the search method, but were unable to provide the actual search strategy.   an explanation as to the lack of published search strategies may be that authors or publishers are not aware of the importance of the search strategy for assessment of the validity of the report and therefore fail to include it in the final article or appendices.   the results from this study are in accordance with the results from rethlefsen et al. (2015) and meert et al. (2016) which showed that librarian and information specialist co-authored srs have better reported search strategies and search documentation than srs with acknowledgement of librarians or no participation by librarians or information specialists. in order to assess the internal validity of the sr, the search strategies must be documented and reported so that they can be reproduced (aromataris & munn, 2017; higgins et al., 2017; moher et al., 2009; the campbell collaboration, 2019; whiting et al., 2016). in studies where librarians only perform the search and advise the authors, but are not co-authors, the reproducibility of searches is not ensured. these results are important when planning sr services with regards to database selection, avoidance of restrictions, and documentation and publication of search strategies.   a recent study by cooper, booth, varley-campbell, britten, and garside (2018) shows that information specialists, librarians, or trial search coordinators are pointed out as appropriate researchers in no less than six guidance documents. in the joanna briggs systematic reviews manual, it is stated that “dependent upon the type of review being conducted  review teams should ideally consist of members with […] an information scientist or research librarian with specialised skills to develop and implement a comprehensive search strategy” (as cited in aromataris & munn, 2017). considering the results from our study and the studies by rethlefsen et al. (2015) and meert et al. (2016), we support this recommendation. medical research has been criticized for being wasteful (chalmers & glasziou, 2009), and librarians providing research support should ensure that robust research is conducted in their institutions, with respect to searching and documentation of searches.   we acknowledge some limitations in this study. the main limitation being the small sample size of the second and third analyses. our search for srs to include was limited to publications between 2007 and mid-2018 representing a decade of reviews. however, there is a possibility that researchers from our institutions may have published srs with librarians as co-authors prior to 2007, which could have given us a larger sample size. to be able to generalize on this matter, in the future, the same analysis with a larger sample size, not restricted to researchers from two institutions, could be performed. in this study we aimed to examine to what extent researchers have collaborated with a librarian in the sr process. that is, on the basis of what is stated in the publication. some researchers might have had help from a librarian but for various reasons this has not been mentioned.   conclusion   the number of srs are increasing, but narrative and other types of reviews remain the core of reviews by researchers from university of oslo and oslo university hospital.   librarians are rarely involved as co-authors of srs at the two institutions in our sample. in a quarter of the analysed srs, a librarian was acknowledged or mentioned in the methods section. in the majority of the srs, there were no clear evidence that a librarian had been in involved.   due to the small sample size, we cannot draw strong conclusions about the risk of bias in srs with or without librarian involvement. however, we observed that co-authored srs have more comprehensive search strategies and better documentation, and have a lower risk of bias compared to srs with acknowledgement or no participation by librarians.   implications for practice   as can be observed from this and previous studies, librarian co-authorship improves the quality of searches and reporting in srs. this implies that librarians should advocate for co-authorship.   the strengths of the librarian co-author lie in translating the objectives into a searchable question, selecting relevant databases, identifying subject headings and text words, planning search strategies and translating them to various databases, executing and documenting the searches and ensuring that search strategies are published with the article. furthermore, the librarian should describe the search in the methods section of the systematic review.   librarians should also aim to spread knowledge about our competence, and the positive implications of librarian co-authorship in srs in medical and health journals and at conferences attended by sr authors (e.g. cochrane and campbell colloquia, htai annual meetings).   library associations such as the medical library association and the european association for health information and libraries can encourage librarians to aim for co-authorship. they can also liaise with journal editors in order to ensure that search strategies are peer reviewed and published with the article.   acknowledgements   the authors thank and acknowledge marte ødegaard for her valuable contribution with the search strategy, ole martin hagtvedt holte for valuable assistance with categorising the articles into different types of reviews, marte j. søyland for valuable help with data management, and therese skagen for valuable contributions in the writing process. thanks to veena manja and sølvi biedilæ for additional information about unpublished search strategies in their publications, and to vanessa fleming for proof reading.   funding   this article is produced by the project “fagbibliotek og systematiske oversikter” supported by the national library of norway.   references   aamotsmo, t., & bugge, k. e. (2014). balance artistry: the healthy parent's role in the family when the other parent is in the palliative phase of cancer — challenges and coping in parenting young children. palliative & supportive care, 12(4), 317-329. https://doi.org/10.1017/s1478951513000953   aas, m., dazzan, p., mondelli, v., melle, i., murray, r. m., & pariante, c. m. (2014). a systematic review of cognitive function in first-episode psychosis, including a discussion on childhood trauma, stress, and inflammation. frontiers in psychiatry, 4, 1-13. https://doi.org/10.3389/fpsyt.2013.00182   aromataris, e., & munn, z. (2017). jbi systematic reviews. retrieved from https://reviewersmanual.joannabriggs.org/   aune, e., røislien, j., mathisen, m., thelle, d. s., & otterstad, j. e. (2011). the "smoker's paradox" in patients with acute coronary syndrome: a systematic review. bmc medicine, 9, 1-11. https://doi.org/10.1186/1741-7015-9-97   borgeraas, h., johnson, l. k., skattebu, j., hertel, j. k., & hjelmesæth, j. (2018). effects of probiotics on body weight, body mass index, fat mass and fat percentage in subjects with overweight or obesity: a systematic review and meta-analysis of randomized controlled trials. obesity reviews, 19(2), 219-232. https://doi.org/10.1111/obr.12626   burger, e. a., kornør, h., klemp, m., lauvrak, v., & kristiansen, i. s. (2011). hpv mrna tests for the detection of cervical intraepithelial neoplasia: a systematic review. gynecologic oncology, 120(3), 430-438. https://doi.org/10.1016/j.ygyno.2010.11.013   chalmers, i., & glasziou, p. (2009). avoidable waste in the production and reporting of research evidence. lancet, 374(9683), 86-89. https://doi.org/10.1016/s0140-6736(09)60329-9   cooper, c., booth, a., varley-campbell, j., britten, n., & garside, r. (2018). defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies. bmc medical research methodology, 18, 1-14. https://doi.org/10.1186/s12874-018-0545-3   golder, s., loke, y., & mcintosh, h. m. (2008). poor reporting and inadequate searches were apparent in systematic reviews of adverse effects. journal of clinical epidemiology, 61(5), 440-448. https://doi.org/10.1016/j.jclinepi.2007.06.005   haahr, m. (2019). random.org: true random number service. retrieved from https://www.random.org   higgins, jpt, churchill r, chandler j, cumpston ms (editors), cochrane handbook for systematic reviews of interventions version 5.2.0 (updated june 2017). cochrane, 2017. available from training.cochrane.org/handbook   ioannidis, j. p. a. (2016). the mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. the milbank quarterly, 94(3), 485-514. https://doi.org/10.1111/1468-0009.12210   koffel, j. b., & rethlefsen, m. l. (2016). reproducibility of search strategies is poor in systematic reviews published in high-impact pediatrics, cardiology and surgery journals: a cross-sectional study. plos one, 11(9), 1-16. https://doi.org/10.1371/journal.pone.0163309   krølner, r., rasmussen, m., brug, j., klepp, k. i., wind, m., & due, p. (2011). determinants of fruit and vegetable consumption among children and adolescents: a review of the literature. part ii: qualitative studies. international journal of behavioral nutrition and physical activity, 8, 1-38. https://doi.org/10.1186/1479-5868-8-112   manja, v., saugstad, o. d., & lakshminrusimha, s. (2017). oxygen saturation targets in preterm infants and outcomes at 18-24 months: a systematic review. pediatrics, 139(1). https://doi.org/10.1542/peds.2016-1609   meert, d., torabi, n., & costella, j. (2016). impact of librarians on reporting of the literature searching component of pediatric systematic reviews. journal of the medical library association, 104(4), 267-277. https://doi.org/10.3163/1536-5050.104.4.004   moher, d., liberati, a., tetzlaff, j., altman, d. g., altman, d., & the prisma group. (2009). preferred reporting items for systematic reviews and meta-analyses: the prisma statement. annals of internal medicine, 151(4), 264-269. https://doi.org/10.7326/0003-4819-151-4-200908180-00135   patsopoulos, n. a., analatos, a. a., & ioannidis, j. p. (2005). relative citation impact of various study designs in the health sciences. jama, 293(19), 2362-2366. https://doi.org/10.1001/jama.293.19.2362   rethlefsen, m. l., farrell, a. m., osterhaus trzasko, l. c., & brigham, t. j. (2015). librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. journal of clinical epidemiology, 68(6), 617-626. https://doi.org/10.1016/j.jclinepi.2014.11.025   robsahm, t. e., aagnes, b., hjartåker, a., langseth, h., bray, f. i., & larsen, i. k. (2013). body mass index, physical activity, and colorectal cancer by anatomical subsites: a systematic review and meta-analysis of cohort studies. european journal of cancer prevention, 22(6), 492-505. https://doi.org/10.1097/cej.0b013e328360f434   roe, y., soberg, h. l., bautz-holter, e., & ostensjo, s. (2013). a systematic review of measures of shoulder pain and functioning using the international classification of functioning, disability and health (icf). bmc musculoskeletal disorders, 14, 1-12. https://doi.org/10.1186/1471-2474-14-73   ross-white, a. (2016). librarian involvement in systematic reviews at queen's university: an environmental scan. journal of the canadian health libraries association / journal de l’association des bibliothèques de la santé du canada, 37(2), 39-43. https://doi.org/10.5596/c16-016   ruddox, v., mathisen, m., bækkevar, m., aune, e., edvardsen, t., & otterstad, j. e. (2013). is 3d echocardiography superior to 2d echocardiography in general practice? a systematic review of studies published between 2007 and 2012. international journal of cardiology, 168(2), 1306-1315. https://doi.org/10.1016/j.ijcard.2012.12.002   skarphedinsson, g., hanssen-bauer, k., kornør, h., heiervang, e. r., landrø, n. i., axelsdottir, b., . . . ivarsson, t. (2015). standard individual cognitive behaviour therapy for paediatric obsessive-compulsive disorder: a systematic review of effect estimates across comparisons. nordic journal of psychiatry, 69(2), 81-92. https://doi.org/10.3109/08039488.2014.941395   sugrue, c., englund, t., solbrekke, t. d., & fossland, t. (2018). trends in the practices of academic developers: trajectories of higher education? studies in higher education, 43(12), 2336-2353. https://doi.org/10.1080/03075079.2017.1326026   the campbell collaboration. (2019). campbell systematic reviews: policies and guidelines. https://doi.org/10.4073/cpg.2016.1   toews, l. c. (2017). compliance of systematic reviews in veterinary journals with preferred reporting items for systematic reviews and meta-analysis (prisma) literature search reporting guidelines. journal of the medical library association, 105(3), 233-239. https://doi.org/10.5195/jmla.2017.246   whiting, p., savović, j., higgins, j. p. t., caldwell, d. m., reeves, b. c., shea, b., . . . robis group (2016). robis: a new tool to assess risk of bias in systematic reviews was developed. journal of clinical epidemiology, 69, 225-234. https://doi.org/10.1016/j.jclinepi.2015.06.005     appendix a search strategy scopus   date searched: 28.06.2018 number of retrieved references: 2,733 limits: source type: journal articles, year: 2007-2018 search strategy: af-id("universitetet i oslo" 60010348) or af-id("rikshospitalet-radiumhospitalet hf" 60026108) or af-id("oslo university hospital" 60105037) or af-id("ulleval university hospital" 60068729) or af-id("aker university hospital" 60091868) or af-id("university of oslo faculty of medicine" 60003938) or af-id("university of oslo institute for surgical research" 60069106) or af-id("mineralogisk-geologisk museum" 60071308) or af-id("research institute of internal medicine" 60069109) or af-id("ragnar frisch centre for economic research" 60009748) or af-id("paleontologisk museum" 60071309) or af-id("olafiaklinikken" 60080266) or af-id("university of oslo faculty of humanities" 60080171) or af-id("oslo heart center" 60069110) or af-id("universitet i oslo johan throne holst's institutt for ernaeringsforsking" 60102103) and ((title(review) or abs(review))) and ( limit-to ( pubyear,2018 ) or limit-to ( pubyear,2017 ) or limit-to ( pubyear,2016 ) or limit-to ( pubyear,2015 ) or limit-to ( pubyear,2014 ) or limit-to ( pubyear,2013 ) or limit-to ( pubyear,2012 ) or limit-to ( pubyear,2011 ) or limit-to ( pubyear,2010 ) or limit-to ( pubyear,2009 ) or limit-to ( pubyear,2008 ) or limit-to ( pubyear,2007 ) ) and ( limit-to ( srctype,"j" ) )     appendix b publication assessments     srs with librarian as co-author reference robis domain 2: identification and selection of studies comments aune et al. 2011 the "smoker's paradox" in patients with acute coronary syndrome: a systematic review. 2.1 did the search include an appropriate range of databases/electronic sources for published and unpublished reports? probably yes medline, embase, central. did not search clinicaltrials.gov or any other trials registries. 2.2 were methods additional to database searching used to identify relevant reports? yes reference lists. 2.3 were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible? yes combination of controlled terms and text words. 2.4 were restrictions based on date, publication format, or language appropriate? probably no language restrictions, but not only english (danish or english or german or norwegian or swedish). concerns regarding methods used to identify and/or select studies: low only one potential risk of bias, language restriction, was identified. our judgment is that this does not exclude too many relevant articles, and since all the other questions were answered yes or probably yes we concluded with low risk of bias.     reference robis domain 2: identification and selection of studies comments ruddox et al. 2013 is 3d echocardiography superior to 2d echocardiography in general practice?: a systematic review of studies published between 2007 and 2012 2.1 did the search include an appropriate range of databases/electronic sources for published and unpublished reports? probably yes medline, embase, pubmed (ahead of print). no searches for unpublished reports. 2.2 were methods additional to database searching used to identify relevant reports? yes reference lists. 2.3 were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible? yes combination of controlled terms and text words. 2.4 were restrictions based on date, publication format, or language appropriate? probably no language restrictions, but not only english (danish or english or german or norwegian or swedish).   date restriction 2007-2011 was justified. concerns regarding methods used to identify and/or select studies: low only one potential risk of bias, language restriction, was identified. our judgment is that this does not exclude too many relevant articles, and since all the other questions were answered yes or probably yes we concluded with low risk of bias.       reference robis domain 2: identification and selection of studies comments skarphedinsson et al. 2014 standard individual cognitive behaviour therapy for paediatric obsessive–compulsive disorder: a systematic review of effect estimates across comparisons 2.1 did the search include an appropriate range of databases/electronic sources for published and unpublished reports? yes medline, embase, psycinfo, amed, central, pubmed, lilacs, clinicaltrials.gov, who international clinical trials, isrctn register, opengrey. 2.2 were methods additional to database searching used to identify relevant reports? yes reference lists. 2.3 were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible? no information search strategy not attached to article. was later retrieved from one of the authors. 2.4 were restrictions based on date, publication format, or language appropriate? yes   none made. concerns regarding methods used to identify and/or select studies: unclear there is insufficient information reported to make a judgement on risk of bias.     reference robis domain 2: identification and selection of studies comments borgeraas et al. 2017 effects of probiotics on body weight, body mass index, fat mass and fat percentage in subjects with overweight or obesity: a systematic review and meta‐analysis of randomized controlled trials 2.1 did the search include an appropriate range of databases/electronic sources for published and unpublished reports? yes medline, embase, central, clinicaltrials.gov 2.2 were methods additional to database searching used to identify relevant reports? yes reference lists 2.3 were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible? yes combination of controlled terms and text words. appropriate use of filters. 2.4 were restrictions based on date, publication format, or language appropriate? yes none made. concerns regarding methods used to identify and/or select studies: low no potential areas of bias identified     srs with librarian lentioned in method section or acknowledged reference robis domain 2: identification and selection of studies comments burger et al 2011 hpv mrna tests for the detection of cervical intraepithelial neoplasia: a systematic review 2.1 did the search include an appropriate range of databases/electronic sources for published and unpublished reports? probably yes   medline, embase, cochrane library. did not search for unpublished reports. 2.2 were methods additional to database searching used to identify relevant reports? no information     2.3 were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible? yes combination of controlled terms and text words. appropriate use of filters. 2.4 were restrictions based on date, publication format, or language appropriate? no no actual restrictions on language in the search strategy even though the methods chapter says that it was restricted to english and scandinavian. date restriction from 1996 not explained. concerns regarding methods used to identify and/or select studies: high this was in between low and high risk of bias. we still judged it as high because of the unjustified date restriction.     reference robis domain 2: identification and selection of studies   roe et al 2013 a systematic review of measures of shoulder pain and functioning using the international classification of functioning, disability and health (icf) 2.1 did the search include an appropriate range of databases/electronic sources for published and unpublished reports? probably yes medline, embase, pedro, cinahl, central. no electronic searches for unpublished reports. 2.2 were methods additional to database searching used to identify relevant reports? no information     2.3 were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible? yes combination of controlled terms and text words. 2.4 were restrictions based on date, publication format, or language appropriate? no limitation to only five years 2005-2010, is not justified. restricted to english and scandinavian languages. concerns regarding methods used to identify and/or select studies: high judged to be of high risk of bias because of the strict date restriction. the tool in question was published in 2001.     reference robis domain 2: identification and selection of studies   aamotsmo et al 2014 balance artistry: the healthy parent’s role in the family when the other parent is in the palliative phase of cancer — challenges and coping in parenting young children 2.1 did the search include an appropriate range of databases/electronic sources for published and unpublished reports? probably yes   medline, psycinfo, cinahl. no sources for unpublished studies mentioned. 2.2 were methods additional to database searching used to identify relevant reports? yes reference lists. 2.3 were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible? no information   request for complete search strategy sent to aamotsmo 1/2/19   no complete search strategy available. terms listed but no information about subject headings/text words. only 182 hits. 2.4 were restrictions based on date, publication format, or language appropriate? no date restriction 1989-2009, is not justified. we have reason to believe that the topic was also relevant earlier than 1989. concerns regarding methods used to identify and/or select studies: high because of the date restriction and low number of returned hits, we consider methods used to identify studies to be of high risk of bias.     reference robis domain 2: identification and selection of studies comments sugrue et al 2017 trends in the practices of academic developers: trajectories of higher education? 2.1 did the search include an appropriate range of databases/electronic sources for published and unpublished reports? probably yes   australian education index, eric, psycinfo. also says that ebsco was searched separately without specifying which ebsco database. 2.2 were methods additional to database searching used to identify relevant reports? no information     2.3 were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible? no information   request for complete search strategy sent to sugrue 1/2/19. 2.4 were restrictions based on date, publication format, or language appropriate? no information   concerns regarding methods used to identify and/or select studies: unclear there is insufficient information reported to make a judgement on risk of bias.      srs with librarian not mentioned reference robis domain 2: identification and selection of studies comments krølner et al 2011 determinants of fruit and vegetable consumption among children and adolescents: a review of the literature. part ii: qualitative studies 2.1 did the search include an appropriate range of databases/electronic sources for published and unpublished reports? probably yes   anthropology plus, cinahl, csa illumine (including eric (1966 onwards), econlit (1969 onwards), sociological abstracts (1952 onwards), social services abstracts (1979 onwards), worldwide political science abstracts (1975 and onwards), embase, international bibliography of the social sciences, medline, psycinfo, web of science. no sources for unpublished studies searched. 2.2 were methods additional to database searching used to identify relevant reports? no information     2.3 were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible? no information   full search strategy not attached, but well reported in the article. only text words, no subject terms. restriction to study design was justified. 2.4 were restrictions based on date, publication format, or language appropriate? yes no restrictions to the search mentioned. concerns regarding methods used to identify and/or select studies: unclear there is insufficient information reported to make a judgement on risk of bias.      reference robis domain 2: identification and selection of studies comments robsahm et al 2013 body mass index, physical activity, and colorectal cancer by anatomical subsites: a systematic review and meta-analysis of cohort studies 2.1 did the search include an appropriate range of databases/electronic sources for published and unpublished reports? no only pubmed. 2.2 were methods additional to database searching used to identify relevant reports? yes reference lists. 2.3 were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible? probably no text words only (probably mapped to mesh terms). filter for study design is too restricted, and mesh filter for humansexcluding not indexed publications. 2.4 were restrictions based on date, publication format, or language appropriate? no english only. concerns regarding methods used to identify and/or select studies: high because only one database was searched, too restricted filters, and language restriction we consider methods used to identify studies to be of high risk of bias.     reference robis domain 2: identification and selection of studies comments aas et al 2014 a systematic review of cognitive function in first-episode psychosis, including a discussion on childhood trauma, stress, and inflammation 2.1 did the search include an appropriate range of databases/electronic sources for published and unpublished reports? probably no pubmed, psycinfo. no searches for unpublished reports. 2.2 were methods additional to database searching used to identify relevant reports? no information     2.3 were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible? probably no   request for complete search strategy sent to aas 1/2/19. 2.4 were restrictions based on date, publication format, or language appropriate? no information mention no restrictions. concerns regarding methods used to identify and/or select studies: unclear there is insufficient information reported to make a judgement on risk of bias.       reference robis domain 2: identification and selection of studies comments manja et al 2017 oxygen saturation targets in preterm infants and outcomes at 18-24 months: a systematic review 2.1 did the search include an appropriate range of databases/electronic sources for published and unpublished reports? no information     2.2 were methods additional to database searching used to identify relevant reports? no information     2.3 were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible? no information   request for complete search strategy sent to saugstad 1/2/19 (initial request sent to corresponding author was not deliverable). 2.4 were restrictions based on date, publication format, or language appropriate? no information   concerns regarding methods used to identify and/or select studies: unclear there is insufficient information reported to make a judgement on risk of bias.     [1] we used the online publication year, which in three cases (borgeraas, johnson, skattebu, hertel, & hjelmesæth, 2018; skarphedinsson et al., 2015; sugrue, englund, solbrekke, & fossland, 2018) differ from the downloaded citation. evidence based library and information practice evidence based library and information practice 2009, 4:4  evidence based library and information practice       article    testing an academic library website for usability with faculty and graduate students    judith z. emde  e‐resources librarian   university of kansas  kansas, usa  email: jemde@ku.edu    sara e. morris  american history librarian  university of kansas  kansas, usa  email: semorris@ku.edu    monica claassen‐wilson  information specialist  university of kansas  kansas, usa  email: monicacw@ku.edu    received: 17 july 2009          accepted: 10 november 2009      © 2009 emde, morris, and claassen‐wilson. this is an open access article distributed under the terms of the  creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted  use, distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objectives – this usability study was developed to observe faculty and graduate  students’ reactions to a recent redesign of the university of kansas (ku) libraries’  website. the redesign included new navigational features, introduction of a federated  search tool, a quick search box on the front page, and research subject pages. the  study also provided the opportunity to observe the practices of faculty and graduate  students in locating and retrieving information on the libraries’ website.    methods – ten participants (five faculty and five graduate students) representing  diverse disciplines were solicited for the study. participants were required to access  the libraries’ website to answer a series of questions regarding new and updated  features of the website. observational analysis using morae™ software was  24 mailto:jemde@ku.edu mailto:semorris@ku.edu mailto:monicacw@ku.edu http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2009, 4:4  25 conducted and interviews with each participant provided details of their opinions on  how these new features would influence their research and teaching activities.     results – most of the participants either did not notice or ignored the major website  changes. links to and locations of commonly used resources (e.g.  catalogue;  databases; e‐journals) had been changed minimally, and the faculty and graduate  student participants gravitated to those familiar features to complete tasks. prior to  the study, participants had not accessed the new discovery tools; however, once  previewed, responses to the tools’ utility were generally favourable. after using the  federated search tool on a familiar topic, several participants noted that, when  directed to databases they had not previously considered, they were able to locate  citations they had missed in the past. observers noted pitfalls in navigating the site  such as inconsistent underscoring of links, ambiguous terminology, and unclear icons  meant to expand subject heading lists. unexpected searching behaviours were  observed, including inconsistent and lack of conceptual understanding in searching  for e‐journal content.    conclusions –  this study provides evidence regarding the usability of a library  website with a population already familiar with library resources. it demonstrated  that faculty and graduate students are not interested in experimenting with new  discovery tools but are amenable to their potential value to undergraduate students.  the recent trend toward minimizing content and links on websites satisfies this  population, one which is already comfortable with the basic attributes of a library’s  website.        introduction    academic research libraries bear the challenge  of meeting information needs of patrons  varying from entering freshmen to eminent  scholars. historically, face‐to‐face transactions  permitted staff to adjust to the needs,  characteristics, and skills of the individuals  before them. using information gained from  the reference interview, body language, and  other visual clues, librarians easily guided  diverse patrons to the sources that best met  their informational needs. today, patrons visit  the library not only through the front door,  but also electronically via library websites.  library websites bear the burden of serving  the same range of patrons, but without the  same capabilities for adaptation. as libraries  work to better serve one group of patrons,  does this hinder services to others?     during the summer of 2008, the university of  kansas libraries revised their home page  <http://www.lib.ku.edu/> to incorporate a  variety of resource discovery tools (figures 1  and 2). the main objectives were to  incorporate tools for easier searching, such as  a federated search, and to improve access to  subject‐focused resources and services. to  achieve these objectives, a quick‐access tabbed  box for federated, catalogue, and e‐journals  searching was added to the home page along  with a “research by subject” access point.  before the new website went live, the libraries  undertook a usability study with  undergraduate students. although the  redesign was aimed to improve  undergraduates’ use of library resources, the  design team desired to also understand how  graduate students and faculty might use it as  researchers and as instructors. due to time  constraints, a usability study with graduate  students and faculty did not occur until two  months after the redesigned website went live.     literature review    leo robert klein declared in 2003 that, “the  expert user is dead.” klein urged librarians to  recognize that they could not create websites  http://www.lib.ku.edu/ evidence based library and information practice 2009, 4:4  or support databases with complicated  searching interfaces, techniques, and countless  options. although librarians aspired to change  users into experts, klein reinforced that  attempting to do so was unrealistic and  impractical.  instead, he suggested librarians  redefine their expectations of patrons and to  work at giving users what they wanted (36).  commentators and critics like klein advocated  librarians to adapt technology to meet users’  expectations.      as the web matured and matriculating college  students came of age with the internet,  libraries faced a stark reality. although  librarians suspected that college students  turned to the web for information before they  visited the bricks‐and‐mortar library, a 2005  oclc report confirmed this. the authors of  college students’ perceptions of libraries and  information resources concluded that 89% of  undergraduates begin their research with a  search engine (de rosa et al. 7). even prior to  oclc’s quantification of undergraduate  information‐seeking behaviours, librarians  looked for solutions to make using library  resources more appealing. a number of  commentators have argued that federated  searching could match google in ease of use  and return better results than a web search  (bell 107‐108; luther 36; tennant 28). a simple  search box integrated multiple databases,  assumed the boolean and operator, and  required little library knowledge from the user  (kitalong, hoeppner, and scharf 178). this  seemed like the perfect solution to meet the  needs of undergraduates because they “expect  to take their google searching skills and apply  them to find library resources” (ponsford and  van duinkerken 162).     federated searching has not been without  criticism. critics direct their frustration toward  the duplicated results from multiple sources,  slowly executed searches, and results  presented in no particular order (belliston,  howland, and edwards 474; tang, hsieh‐yee,  and zhang 216). yet librarians have generally  accepted its use as a discovery tool for users  lacking basic searching skills. for these  reasons it is perceived as a tool to initiate  searching for undergraduates and not for  extensive use by graduate students or faculty.     librarians have adopted usability testing as an  essential step in the web design process. jakob  nielsen, an authority on usability, advocates  that websites undergo frequent examination.  nielson asserts that testing more than five  participants is a waste of resources because it  is the point of diminishing returns. according  to nielsen, the only time testing needs to  extend beyond five is when a site has many  distinct user groups (nielsen).    library literature contains many usability  studies involving federated searching, with  the majority focusing on undergraduates. this  is not surprising since undergraduates make  up the largest portion of users in academic  libraries. their lack of awareness of library  resources and tools is reflected in their  perception of library websites (kitalong,  hoeppner, and scharf 178). ed tallent found  in an early study that students do not care to  learn about the various avenues available to  obtain information. instead, an individual user  desires to have everything “combined for  searching and sees little need to separate the  access” by resource (71). but federated  searching is not without its problems. as  tang, hsieh‐yee, and zhang found, patrons  do not understand what they are searching or  how to interpret results (216). belliston,  howland, and roberts emphasize that even  with these problems, federated searching  remained the choice of undergraduates. in  their examination at three brigham young  university campuses, the authors determined  that undergraduates favoured the speed of the  federated search over traditional searching  (479‐480). there are no published studies  focusing solely on graduate students and  faculty, and those studies that do include these  more advanced users among the participants  make few references to the reactions of this  subset of patrons. belliston and his fellow  authors suggested the need for studies  specifically exploring graduate student and  faculty perceptions of federated searching  (480).     26 evidence based library and information practice 2009, 4:4  usability studies focusing on library websites  have consistently documented the typical  behaviours of users. overall, patrons have  clearly established expectations of websites  based on their prior web experiences (george  14; ponsford and van duinkerken 162).   patrons are most likely to read and use links  on the left‐hand side of the screen. (fichter 8;  whang and ring 80). users do not want  anything elaborate; they expect the basics.  links, for example, need to be clearly  identifiable as links and library jargon should  be avoided. in a 2005 test, patrons sought help  on the “library instruction” page. unaware of  the meaning, they assumed the page provided  instruction on using the library (cobus, dent,  and ondrusek 237‐238). the simplicity and  success of google reinforces that sites can be  uncomplicated and still be effective. jung et al.  determined, “by attempting to make all  resources visible, the library has perhaps  made none very accessible” (384). in creating a  more user friendly website and not one just for  information professionals, the need for  multiple access points leading to the same  place has become apparent (fichter 7).     the majority of usability studies explored only  the reactions of undergraduates or a  combination of undergraduates along with an  insignificant number of graduate student and  faculty participants. overall, reported findings  inform the profession only about the habits of  undergraduates and not those of advanced  researchers. a gap in the literature exists  regarding graduate student and faculty  opinions about and usage of these new  discovery tools. this study set out to address  these gaps.    methods    ten participants consisting of five faculty and  five graduate students were solicited for the  study. the participants were recommended to  the observers by library faculty subject  liaisons, and were selected to represent the  four main disciplines from across the  university: humanities, social sciences,  science/technology, and international  programs. the graduate students represented  english, geology, journalism, musicology,  and russian, east european & eurasian   studies (rees). the faculty included  individuals from business, english,  linguistics, math, and spanish/portuguese.     prior to each observation session, the panel of  five observers selected one spokesperson to  recite the set of institutional review board‐ approved questions (see appendix), which  directed each participant to access several  features of the libraries’ website, including a  new federated search tool. the questions were  intended to address new and updated features  of the website and to ask participants’  opinions of how these features would  influence their research and teaching activities.  a specific goal was to understand how and  whether the federated search tool, structured  as a google‐like search and located at the top‐ center position of the home page, might be  utilized. each participant was interviewed  separately during a one‐hour session, and  participant mouse clicks and audio were  captured using morae™ software, a well‐ known product for recording usability testing  sessions. morae™ allowed the researchers to  later review participants’ actions as they  navigated the site and to match written  observations with actual screen captures.  the libraries’ home page (figure 1) was  redesigned in august 2008 (figure 2) to serve  as a database and subject portal, facilitating  the linking of researchers to the information  that would meet their needs. the federated  search tool, 360 search™ from serials  solutions, was added in the fall of 2008 to  provide a means for a quick search of selected  databases directly from the libraries’ home  page. the databases selected for inclusion in  the basic search were academic search  premier, business source premier, project  muse, and wilson omnifile. tabbed  searching also included quick access to the  library catalogue and e‐journal search boxes  (figure 3).          27 evidence based library and information practice 2009, 4:4                                                                                                                                                                  fig. 1. ku libraries home page prior to the redesign.                                              fig. 2. ku libraries home page as presented for the study.  28 evidence based library and information practice 2009, 4:4                          results    overall response    ninety percent of the participants did not  know about, ignored, or found ways to work  around the major website changes. for  example, the spanish/portuguese professor  indicated he had noticed the changes prior to  participating in the study but had chosen to  avoid them entirely by utilizing a link to the  library catalogue from the university’s  libraries page. all of the participants were  interested in knowing more about the purpose  and function of the new features. the main  navigation scheme of the site had been  purposely maintained with minimal changes  to facilitate a smoother transition for users  who were already familiar with the site.  observers noted that the participants  gravitated to familiar features to complete  many of the tasks requested of them. once the  tools had been previewed, participants’  responses were favourable regarding their  utility for some purposes and users.     federated search    participants were asked to give their  impressions of the quick search and to explain  the purpose of each tab in the search box (see  appendix). the “articles & databases quick  search” tab puzzled the rees graduate  student. she stated, “i know there’s a way to  select specific databases to search at once,” but  was not sure how to determine which  databases were included in the search.  participants were then asked to conduct a  basic search using the federated search tool                           fig. 3. quick search tabs on libraries’ home page.  and to comment on the results. some of them  found results that related to their topic, but the  majority commented that the results were   unrelated or too broad. the graduate student  in musicology tried a search for “peter, paul  and mary” and got results as varied as  authors’ names in a medical journal and peter,  paul, and mary magdalene. when she ran the  search a second time with her search terms in  quotations, she was pleased to find a citation  that she had not previously discovered.  another participant stated that, while some of  the results might be useful, it would take too  much time to filter through all of the citations.  after using the federated search tool on a  familiar topic, several participants noted that  by being pointed to databases they had not  previously thought of using, they were able to  locate citations they had missed in the past.     when asked if they would recommend the  federated search tool to their peers or their  students, all participants stated that they  would recommend the tool to undergraduate  students as a good place to begin their  research. according to one person, “students  don’t often know where to start. a more  restricted database set in a federated search  can be helpful and less intimidating.” while  most participants had a favourable view of the  multidisciplinary nature of the results  presented in the federated search, a few  remarked that they would not be very likely to  use the tool themselves, as they tended to rely  on one to three subject‐specific databases for  the bulk of their research activities.     in some subject areas, a subject‐specific  federated search tool was made available to   29 evidence based library and information practice 2009, 4:4                                                allow users to narrow their search to databases  selected by a subject librarian (figure 4).     responses to this tool were similar to reactions  towards the quick search on the home page.  some participants were surprised to be  presented with a search tool tailored to their  discipline. two commented that there were  databases included in the search tool that they  had not been aware of previously and that  they would be more likely to use the subject‐ specific tool as a result.    journal article search    the investigators asked participants to take a  citation for an article from social science  quarterly and demonstrate how they would  identify whether the article was available in  the libraries (see appendix). the results of  this exercise were varied; half of the  participants utilized the e‐journals link in the  left‐hand navigation section of the home page  and searched for the journal title. other  methods included searching for the article title  using the “articles & databases quick  search”, using the online catalogue to search  for the journal title, and navigating to a                                                   fig. 4. example of subject page for english language and literature.      specific social science database to search for  the article title. the least successful methods  employed by participants involved searching  for the journal title in a search box that was set   up to search for database titles, and searching  for the article’s author in the online catalogue.  the investigators were surprised that half of  the participants began the exercise by  searching for the article title or author rather  than the journal title. of the five who used this  approach, two were faculty. this may point to   a changing perception among scholars that  journal articles are unique entities apart from  the journals in which they are published. as a  result, search structures or tools that force  scholars to navigate to an article through a  database‐to‐journal‐to‐article path may  present a challenge.    navigation    observers noted pitfalls in navigating the site,  such as inconsistent underscoring of links and  unclear library verbiage. when participants  were asked to identify the purposes of the  three tabs on the search box on the home page,  their difficulty pointed to shortcomings in the  labeling and help links on the tabs.  30 evidence based library and information practice 2009, 4:4  additionally, 70% of the participants,  including four graduate students and three  faculty members, did not know how to click  on the plus symbol (+) to expand the main  subject headings to view the second tier of  database subject headings on the “research by  subject” page (figure 5). this came as a  surprise to the observers, given that the tree  structure of the subject list was assumed to be  a ubiquitous navigation feature.                                           research help    an unexpected result of the study was that,  when asked the question “if you need detailed  information about a subject in your area,  where would you go for information?” 80% of  the participants responded that they would  contact their subject librarian first. this  response was evenly divided between faculty  and graduate students. other responses  included clicking the “research by subject”  link or seeking help at a reference desk. this  contradicts the current literature that suggests  that declining statistics of face‐to‐face  reference interviews are indicators of a  changing perception of librarians as a primary  source for research assistance (applegate 184).    discussion    with studies indicating less reliance on   libraries for information, it is crucial for  institutions to learn how resources are being  accessed not just by undergraduates but also  by more advanced researchers (housewright  and schonfeld 5). for many constituents, a  library’s website may be their only interaction  with its services and collections. how can   websites be augmented and improved to  enhance the patrons’ experiences and,                                                            fig. 5. research by subject page with subject tree structure.    hopefully, encourage them to return again?  the presentation and ease of use are crucial.  clear navigational features such as font size,  unambiguous links, and comprehensible  terminology are minor attributes when  compared to content, but unless these simple  features are obvious, the content will not be  read. usability studies such as this one inform  and often supplant misinformation and  assumptions of library staff about how patrons  access library websites.     one of the major objectives of the study was to  learn how the participants reacted to the quick  search box introduced on the front page,  which included a federated search across four  pre‐selected databases (figure 2). several  years previously, a similar attempt to  introduce a federated searching tool, from a  vendor now out of business, was  technologically unmanageable and perceived  31 evidence based library and information practice 2009, 4:4  as not user‐friendly by the library staff. the  decision to implement a federated search tool  a second time supported the commitment to  improve resource discovery and make the  search easier and more familiar to patrons.  even with an improved federated search  product, some staff stated that the research  process was being “dumbed down” and that  more effort should be placed on instructing  patrons to effectively apply proven database  search techniques. the prominent quick search  box, resembling a google‐like search, was  intended to grab the attention of those  undergraduate students who are unaware of  where to begin a search for information from  the libraries’ website. the federated search  would hopefully retrieve a few good articles to  start them on a research project. the observers  wondered if the faculty, in particular, would  have similarly negative reactions to  simplifying the initial search procedure.  several participants identified the potential  value of the tool. two of the faculty  recognized that students often do not know  where to start conducting research or which  database to select. the english faculty member  noted that the clustered results identified  common keywords that students might not  think of using as search terms. even though  the faculty might not use the federated search  tool in their research, they acknowledged the  benefit of the tool for undergraduate students.  this study indicates that federated searching  as a resource discovery tool would not be  extensively used by faculty and graduate  students for their own research projects.     apart from the quick search box, the visual  and contextual changes to the website were  not obtrusive, as these modifications were  mostly unnoticed by this study’s participants.  familiar links and labels had been carried over  from the previous version to the updated  version of the libraries’ home page. “articles  & databases”, “catalog”, and “request  materials” links were recognizable to the  participants and their presence overshadowed  the new features (figure 2). these links were  retained on the left‐hand navigation panel on  the redesigned website and were typically  selected as access options.   a new feature intended to inform patrons of  available resources in a subject area was  entitled “research by subject” (figure  2). this  link leads users to organized lists of databases,  websites, contacts, and research guides by  subject. no participants in the study had  previously reviewed the content, and none  seemed overly enthusiastic about the  organization of these sources by subject, nor  expressed an interest in using this feature as a  teaching tool. the organization attempts to  compartmentalize “databases” and scholarly  “websites” which appeared to be a foreign  concept. the english faculty member had  previously attempted to locate a noteworthy  middle english website was not aware that the  libraries’ divided scholarly websites and  licensed databases into two distinct silos  (figure 4). the participants believed they were  knowledgeable of the major resources in their  area and consequently did not express a need  to consult a “resources by subject” site. the  reactions to the subject pages indicate that this  grouping of resources would not be of any  particular value for research or teaching for  these participants. subject guides are most  pertinent when designed for specific courses  where context helps to reinforce resources  introduced during classroom instruction (reeb  and gibbons 126). ku libraries will continue  to address subject organization with the  implementation and development of  libguides™ by springshare and emphasize  the application specifically for courses.    locating journal articles from known citations  is a difficult concept even for some advanced  researchers. the traditional paradigm of a  journal with issues and articles identified  within those issues is shifting even though  many electronic journals continue with those  traditional enumerations. articles can be  identified through the searching of databases  and google, and are therefore not necessarily  tied to the context of an issue and volume as  with the print version. patrons often  successfully find journal articles by searching  for the article title in a federated search or  database, rather than by the traditional means  of searching for the journal title and locating  the volume (george 13). an awareness of how  32 evidence based library and information practice 2009, 4:4  patrons are locating articles will help library  staff to guide patrons to other access options.   each participant had his/her own unique  approach to searching the libraries’ website  and locating sources. several had previously  identified their favourite databases and did  not deviate from using them. mathscinet was  the exclusive database for the mathematics  professor. the musicology graduate student  relied on proquest dissertations and theses  for recent data not available elsewhere. the  journalism graduate student only considered  resources with full‐text articles available in the  database or through the link resolver. the  linguistics faculty member found library  hours by searching google. to locate  databases or websites, the english faculty  member used the “find” application on the  browser particularly helpful if the exact title  was not known. the participants’ attention  was channeled to those links and sites that had  worked well for them in the past, and they  tended to avoid taking the time to explore the  potential value of uninvestigated links and  resources.     conclusion    faculty and graduate students participated in  a usability study of the university of kansas  libraries’ redesigned website two months  after its launch. the redesign included new  navigational features, introduction of a  federated search tool, a quick search box on  the front page, and subject‐specific research   pages. most of the participants reported that  they had not paid much attention to the                               changes. the left‐hand navigation links on the  page had been minimally changed during the  revision process, and most participants were  content with using the familiar links to  databases, e‐journals, library catalogue, and  interlibrary loan. very little interest was  exhibited in experimenting with the new  features implemented during the redesign.  once prompted to try the federated search  tool, participants noted the potential value for  novice researchers and interdisciplinary  research, along with discovery of relevant  citations in databases not previously searched.  this evidence proved that the website design  could satisfy the information needs of the  advanced researchers while introducing  resource discovery features intended for  undergraduates. this study revealed how  faculty and graduate students struggled to  locate the full text of a specific article from a  citation. to improve navigation of the website,  recommendations for highlighting links,  improving labels, and particularly changing  the tree structure to expand database subject  lists were submitted to an internal web  services council. no suggestions for changes to  the federated search tool were included in the  final list of recommendations. studying  faculty and graduate student researchers as a  separate population from undergraduate  students generated distinct data, and  reinforced the need for adequate  representation from the varied populations  that the university of kansas libraries serves  when conducting usability testing.                                   33 evidence based library and information practice 2009, 4:4  works cited    applegate, r. “whose decline? which   academic libraries are ʺdesertedʺ in  terms of reference transactions?”  reference & user services quarterly  48.2 (2008): 176‐89.    bell, colleen. “‘don’t make me think’: a plea   for simplicity and transparency.”  public services quarterly 2.4 (2007):  103‐9.    belliston, c. jeffery, jared l. howland, and   brian c. edwards. “undergraduate  use of federated searching: a survey  of preference and perceptions of  value‐added functionality.” college  & research libraries 68.6 (2007): 472‐ 86.    cobus, laura, valeda frances dent, and anita  ondrusek. “how twenty‐eight users  helped redesign an academic  library web site.” reference & users  services quarterly 44.3 (2005): 232‐46.    de rosa, cathy, joanne cantrell, janet hawk,  and alane wilson. college students’  perceptions of libraries and  information resources. dublin, oh:  oclc online computer library  center, inc.: 2006. 13 march 2009  <http://www.oclc.org/reports/pdfs/stu dentperceptions.pdf>.    fichter, darlene. “designing a better subject   page to make users’ searches more  successful.” computers in libraries  25.9 (october 2005): 6‐8, 54, 56.    george, carole a. “lessons learned: usability   testing a federated search product.”  the electronic library 26.1 (2008): 5‐ 20.    housewright, ross, and roger schonfeld.   “ithaka’s 2006 studies of key  stakeholders in the digital  transformation in higher education.”  18. august 2007. web. 28 april 2009.   <http://www.ithaka.org/ithaka‐s‐ r/research/ithakas%202006%20studies %20of%20key%20stakeholders%20in %20the%20digital%20transformation %20in%20higher%20education.pdf >.      jung, seikyung, jonathan l. herlocker, janet  webster, margaret mellinger, and  jeremy frumkin. “libraryfind:  system design and usability testing  of academic metaserach system.”  journal of the american society for  information science and technology  59.3 (2008): 375‐89.    kitalong, karla saari, athena hoeppner, and  meg scharf. “making sense of an  academic library web site: toward a  more usable interface for university  researchers.” journal of web  librarianship 2.2/3 (2008): 177‐204.    klein, leo robert. “the expert user is dead.”     library journal 128.17 (fall 2003 net   connect): 36.    luther, judy. “trumping google?   metasearching’s promise.” library  journal 128.16 (2003):   36‐9.    nielsen, jakob. “why you only need to test  with 5 users.” jakob nielsenʹs alertbox  19 march 2000. web. 16 february 2009  <http://www.useit.com/alertbox/20000 319.html>.    ponsford, bennet claire and wyoma van  duinkerken. “user expectations in the  time of google: usability testing of  federated searching.” internet  reference services quarterly 12.1/2  (2007): 159‐78.    reeb, brenda and susan gibbons. “students,   librarians, and subject guides:  improving a poor rate of return.”  portal: libraries and the academy 4.1  (2004): 123‐30.    34 http://www.oclc.org/reports/pdfs/studentperceptions.pdf http://www.oclc.org/reports/pdfs/studentperceptions.pdf http://www.ithaka.org/ithaka-s-r/research/ithakas%202006%20studies%20of%20key%20stakeholders%20in%20the%20digital%20transformation%20in%20higher%20education.pdf http://www.ithaka.org/ithaka-s-r/research/ithakas%202006%20studies%20of%20key%20stakeholders%20in%20the%20digital%20transformation%20in%20higher%20education.pdf http://www.ithaka.org/ithaka-s-r/research/ithakas%202006%20studies%20of%20key%20stakeholders%20in%20the%20digital%20transformation%20in%20higher%20education.pdf http://www.ithaka.org/ithaka-s-r/research/ithakas%202006%20studies%20of%20key%20stakeholders%20in%20the%20digital%20transformation%20in%20higher%20education.pdf http://www.ithaka.org/ithaka-s-r/research/ithakas%202006%20studies%20of%20key%20stakeholders%20in%20the%20digital%20transformation%20in%20higher%20education.pdf http://www.useit.com/alertbox/20000319.html http://www.useit.com/alertbox/20000319.html evidence based library and information practice 2009, 4:4  tallent, ed. “metasearching in boston college  libraries—a case study of user  reactions.” new library world 105.1  (2004): 69‐75.    tang, rong, ingride hsieh‐yee, and shanyun   zhang. “user perceptions of metalib  combined search: an investigation of  how users make sense of federated  searching.” internet reference  services quarterly 12.1/2 (2007): 211‐ 36.    tennant, roy. “the right solution: federated   search tools.” library journal 128.11  (2003): 28‐9.    whang, michael and donna m. ring.   “student‐focused usability study of  the western michigan university  libraries home page.” journal of web  librarianship 1.3 (2007): 67‐88.    35 evidence based library and information practice 2009, 4:4  36 appendix    questions:  a. please explain for what purpose(s) you use the libraries’ web site.  b. think of a previous project where you had to consult the libraries’ web site, preferably  before the redesign.  please show us how you would go about completing the project on this  site.  c. what is one of your favorite databases? how would you access the database?  d. with the redesign, we’ve inserted a search box on the front page. please explain to us what  you think each tab in the search box is for. would you use any of these 3 tabs? (ask the  participant to run a search on a favorite topic under quick search.) would you recommend to  your students to use the quick search box?  e. if you need detailed information about a subject, e.g. the specifics on searching a research  database, where would you go for information?  f. research by subject page questions.   a. how does this page help you to identify databases in a subject area?    b. would any of the other links on this page be useful to you? (web sites,  contacts/guides)  c. (pull up a subject page with a 360 search option and run a search on a favorite topic  or ask the participant to select databases to conduct a federated search.) review the  results. what do you think of capability of searching across pre‐selected databases?  useful to their research? useful to their students?  g. what link do you access to ask for materials not available in the libraries?  h. how would you identify if the following journal article is available in the libraries:  a. rocca, m. personal attributes and latino voting behavior in congress. social science  quarterly, 89(2):392+. 2008.  i. what is your overall reaction to the libraries’ web site? suggestions?            evidence based library and information practice 2009, 4:2 178 evidence based library and information practice commentary evidence-based practice: evolution or revolution? david v. loertscher professor, san jose state university san jose, california, united states of america e-mail: reader.david@gmail.com received: 09 april 2009 accepted: 20 april 2009 © 2009 loertscher. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. for some years, school library media specialists, like other educators, have been urged to raise their sights from the work-a-day world to the think-a-day world of reflective practice. the concepts of evidence-based practice and action research along with the general educational ideas of data mining, data-based decision making, diagnostic assessment, and a host of other terms encourage everyone to concentrate on results or the impact of actions and programs on teaching and learning. indeed, the american association of school librarians’ standards for the 21st century, and its companion document, empowering learners, challenge teacher librarians to educate learners in dispositions, responsibilities, and selfevaluation, not just the steps of inquiry. one assumes that building reflective students means that the adults are reflective as well. a teacher librarian may feel that the typical day is so overcrowded with organizational tasks, classes streaming in and out of the library, teaching lessons about research, disciplining, and trying to personalize the reference process that, by the end of the day, a sense of overwhelming fatigue makes any reflection about impact the remotest concern. simple survival might loom larger. however, the push for accountability all across education requires everyone involved with learners to come to terms with success, failure, and put in place a system of sustainable school improvement. the higher the pressure to prepare learners to compete globally, along with the alarming statistics on drop out rates and low math and reading scores compared with the rest of the world, only increases the pressure for introspection. at first, teacher librarians may feel that the pressure for accountability rests solely on the shoulders of the classroom teacher, since testing data is taken and reported teacher by teacher. the supporting specialists, such as teacher librarians, reading coaches, instructional specialists, councilors, and even music, art, or physical education teachers may feel as though http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2009, 4:2 179 they are merely bystanders and observers in the accountability game. however, in times of financial exigency, questions about direct impact on learning causes administrators to cut any programs not seemingly pulling their weight. do specialists of any kind really make any difference? any measurable difference? any significant difference? with such pressure to raise scores during the no child left behind years in the united states, many have thought that with a new administration, the testing whip would become less severe. however, it does not appear that the emphasis on accountability is going away any time soon. it is here to stay, and it is being required of all who participate in education, not just the classroom teacher. teacher librarians fighting to retain their jobs are being asked, “what direct impact do you have on teaching and learning?” thus it seems that the need for teacher librarians to embrace the ideas of evidence-based practice (ebp) and self-reflection are no longer optional. that said, however, selfreflection as a part of evolutionary practice may not be the best thing to implement. evolutionary practice vs. revolutionary practice we think of using ebp as a method of continual improvement of systems and the routines we do each day. this approach to data collection and analysis is designed to help us as professionals work smarter, not harder. this idea pervades all organizations and businesses: doing more with less; concentrating on efficiency; eliminating waste and time-consuming/unnecessary practices—all come under scrutiny to make constant and wiser decisions on a daily basis. such introspection is a sound approach as long as the antecedents or foundational inputs remain essentially the same. in other words, as long as the learner characteristics, the abilities of the teachers, and the goals of the organization remain fairly constant, emphasis on evolutionary school improvement works. we all keep refining and adjusting what we do and hopefully get better and better over time. however, thinkers in many fields have recently challenged such a stance. there is no longer time to evolve at a slow and measured pace. sudden and drastic change in many sectors of environment, business, government, and world finance requires quite a different mentality. general motors cannot continue to refine the gas-guzzling suv and expect to sell the millions of vehicles year after year as they once did. at some point, the constant rise in housing and oil prices affect business, no matter what tinkering takes place. allowing the status quo to continue with slow evolutionary progress is a recipe for disaster. it does not matter how adept we are in continual improvement through evidence-based practice, we could be without a role and a job. as much as we all might prefer evolutionary change since it is comfortable and predictable, this stance seems less and less an option in our society—our global society. think of the google generation/digital natives that surround us. think of the rise in charter schools that don’t have libraries. think of homeschool and online distance education programs that have no access to a teacher librarian or school library and often brag that a library is no longer needed. how does ebp fit into a revolutionary rather than evolutionary pattern of change? in the medical field, out of the box research often disrupts current practice. for example, the australian study that identified a bacteria that caused stomach ulcers was dismissed at first partly because it not only was counter to a previous train of research but also, because if true, it would be very unpopular with drug companies who were making a great deal of money treating stomach acidity. as evidence-based practice continued to mount, the disruptive change was inevitable. one could forecast similar innovative change to traditional treatments if a cure to alzheimer’s, cancer, or heart disease was developed. the point is that such major change has already evidence based library and information practice 2009, 4:2 180 appeared in the information culture, but few have stepped up to the plate to reinvent or do the 180-degree thinking to latch on to innovation. perhaps a school library example might clarify this point. all school libraries once used pockets and cards to circulate materials. this system was replaced by automated circulation systems. these systems keep making improvements and now are part of a larger opac system. if the teacher librarian understands the system, there are many reports that can be printed out to analyze the collection, circulation, acquisitions, etc. this is an example of an evolutionary system and much can be done to improve not only the distribution of materials but also to analyze collections and patterns of use. however, should a device like the kindle2 appear where every user has not only access to textbooks, the internet, the digital library collection, but also has a library credit card to build a private collection, the tables turn. disruption of the function of the library collection and the erasure of the need to warehouse and circulate, track, and harasses naughty patrons, changes everything. ebp strategies change overnight for some practices but not necessarily for the central mission of the library to assist every patron with information access and the building of basic literacies and lifetime reading habits. like the drug companies in our previous example, would librarians be actively rejecting user controlled digital devices because of a fear that a machine would not be snuggable in bed or the aphrodisiac smell of ink on paper is missing? could ebp practices actually impede innovation? and, when innovation does occur, how can ebp either confirm the new ideas or discard them? consider the following radical ideas and what kind of ebp would be necessary: the opac and library website are turned into a giant conversation rather than a one-way stream of information. if “they” helped build it, would they use it rather than bypassing it as they do currently? children and teens construct and build their own information spaces, learn to manage those spaces, and learn to manage themselves in those spaces. as it is now, filters theoretically “protect” access to dangerous sites and information. certainly, if they were given freedom to explore, a generation of youth would be corrupted. what experiments and ebp studies would bring some sense to this controversial arena? suppose the library were turned into a learning commons with a physical and virtual space that were the centers of all professional development and school improvement initiatives. would this marriage propel the old school library directly into the center of teaching and learning? what ebp would be needed to test this impact on administrators, teachers, and teacher librarians? suppose the digital generation took the leadership initiative to use their social networking skills over into their academic skills. would they take such a leadership role? what ebp would be needed to document any changes in information behaviors?1 perhaps we should think and rethink the central tenets of the role and function of a library/information service that is required to elevate each learner. what does a holistic learner look like? are there one or many types of learners? when information and technology systems are designed and under the control of learners, what are the implications not just for the organizational structure required to provide such an information system but the human 1 these proposals are just a few made in: loertscher, david v., carol koechlin, and sandi zwaan. the new school learning commons where learners win!. salt lake city, ut: hi willow research and publishing, 2008. evidence based library and information practice 2009, 4:2 181 intervention, personal trainer, integrated information diagnostic system do individuals require in order to excel in whatever world they are required to compete in? it is certain that evolutionary or revolutionary change never happens without reflective practice. in her article, published in this issue of evidence based library and information practice, carol gordon proposes that change will not occur unless building level professionals have mentors to partner with and those mentors ask tough questions. in this context, ebp happens. the question of “what difference has it made?” is asked over and over and over. the question for teacher librarians and many other specialists in education is whether many existing programs will be improved in time or face disintegration only to be reborn as the phoenix in new relevant clothing. perhaps we are really suggesting that the true ebp strategies must operate simultaneously in both an evolutionary manner and a revolutionary manner simultaneously. is that possible? is that the secret to a think-a-day professional vs. just a work-a-day professional? what do you think? works cited american association of school librarians (aasl). empowering learners: guidelines for school library media programs. chicago: american library association, 2009. american association of school librarians (aasl). standards for the 21st century learner. chicago: american library association, 2007. 2 may 2009 <http://www.ala.org/ala/mgrps/divs/ aasl/guidelinesandstandards/learningstan dards/standards.cfm>. gordon, carol. “emerging theory for action research as a tool of evidence-based practice in school library instruction, part i: the evolution of evidence-based teaching.” evidence based library and information practice 4.2 (2009). loertscher, david v., carol koechlin, and sandi zwaan. the new school learning commons where learners win!. salt lake city, ut: hi willow research and publishing, 2008. microsoft word ed.doc evidence based library and information practice     editorial    defending evidence based practice      lindsay glynn  editor‐in‐chief    public services librarian and instruction coordinator, health sciences library  memorial university of newfoundland  st. john’s, newfoundland, canada  e‐mail: lglynn@mun.ca      © 2007 glynn. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      my head is still buzzing from the eblip4  conference. meeting people face‐to‐face with  whom i have worked exclusively online was  delightful, as was getting to know many  people whose work i have been reading for  the past few years. it was both gratifying  and humbling to see that this journal is so  well received and useful in informing  practice.  while the journal’s scope,  standards and content were, for the most  part, positively viewed, the editorial board  (and the eblip movement in general) was  challenged on a few occasions. to  summarize, some inquired how it could be  said that eblip is any different from good  research, and questioned how it is that  previous publications are being pushed  aside in favour of the eblip journal.  my  initial reaction was to be defensive. upon  reflection, however, i am grateful for those  who are stirring the pot, so to speak, and  encouraging us to evaluate why and how  we may be making a difference.      is eblip different from good, sound  research?  this is difficult to answer since  both concepts are not mutually exclusive.   good research is an integral component of  eblip; without it, eblip could not exist  (neither would this journal exist and many  of us would not be trying to figure out how  to get to eblip5 in sweden in 2009!).  eblip  goes a few steps further, however, in that it  formalizes a process to use existing research  and evaluates the implementation of  changes made as a result of the appraisal  and application of the research.  the idea  that research published in other library and  information studies journals is being pushed  aside is at odds with the eblip movement:  it is the existing published research that is  being appraised and utilized to make  informed decisions or referred to in the  mailto:lglynn@mun.ca http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:2  2 literature review of newer published  research.  increasing the knowledge base is a  familiar theme, but it cannot be said that the  pool from which to draw research is  considered all but empty. this journal and  many others are contributing to the existing  literature by publishing original research.  one of the strengths of the eblip journal is  the evidence summaries section.  not only  are these critical appraisals of previously  published research, but the critical  appraisals themselves are peer reviewed.   the knowledge base, therefore, is being  populated with original research as well as  rigorously evaluated research.     am i still feeling defensive?  perhaps a little.  i am sure that the time and effort that i have  invested in eblip cannot help but create  some bias. on the other hand, i have seen  that evidence based practice is making an  impact on how numerous individuals and  institutions approach decision making.  i am  reading, seeing and hearing about research  being designed, performed and evaluated  within a new framework.  ebp is standard  practice in the health professions and is  swiftly gaining solid ground in the social  sciences, education and in government.  our  profession prides itself on being on the  cutting edge of information access and  management: if information professionals  choose not to adopt the model, which has  been proven time and time again to be  effective, they risk being placed outside  cutting edge, innovative practice.  is that  where we want to be?                                   academic librarians develop their teaching identities differently depending on their years of instructional experience evidence summary   academic librarians develop their teaching identities differently depending on their years of instructional experience   a review of: nichols hess, a. (2020). instructional experience and teaching identities: how academic librarians' years of experience in instruction impact their perceptions of themselves as educators. communications in information literacy, 14(2), 153–180. https://doi.org/10.15760/comminfolit.2020.14.2.1 reviewed by: michelle dubroy discipline librarian (researcher services) griffith university southport, queensland, australia email: m.dubroy@griffith.edu.au   received: 7 mar. 2022                                                                    accepted:  13 apr. 2022      2022 dubroy. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30118     abstract   objective – to examine how an academic librarian’s years of instructional experience impacts how they think of themselves as instructors.   design – survey questionnaire.   setting – american academic library profession.   subjects – 353 participants selected from 501 respondents.   methods – a qualtrics survey was sent via email to members of several american library association discussion lists. the author selected a subset of respondents for further analysis based on how they answered key questions on the survey. selected participants were those who believed they had experienced perspective transformation around their teaching identities. the author used principal component analysis and confirmatory factor analysis to identify twelve transformative constructs across three sub-themes: relational, experiential, and professional inputs. the author then labelled each construct based on its respective component parts. one-way analysis of variance (anova) tests were then conducted using spss.   main results – statistically significant differences were found between experienced and inexperienced instructional librarians. participants with more instructional experience tend to believe their teaching identities are influenced to a greater extent by these factors:   interpersonal relationships feedback from colleagues outside of librarianship self-directed learning opportunities   participants with less instructional experience tend to believe their teaching identities are influenced to a greater extent by these factors:   feedback from those within librarianship library-centric inputs such as their formal library studies   conclusion – different types of professional development opportunities will appeal to different librarians based on their level of instructional experience. less experienced librarian instructors may find mentoring and informal collegial relationships within the library to be beneficial. more experienced librarian instructors may prefer to seek out relationships with colleagues outside the library to further develop their teaching identities.   commentary   the paper is part of a series of five publications by the author, reporting the findings from a single survey questionnaire, such as (e.g., nichols hess, 2020 and nichols hess, 2019). this work explores how a range of factors influence academic librarians’ teaching identities, with various sub-findings reported separately. it is grounded in mezirow’s (1978) concept of transformative learning.   the study was evaluated using glynn’s (2006) critical appraisal tool.   the literature review provides a useful summary of the theoretical constructs underpinning the research. however, it is not readily apparent how this paper connects to the other papers in the series (nichols hess, 2020, 2019). further, the author does not clearly articulate why one would read this latest paper as opposed to the others in the series. in fact, these other related papers are barely mentioned in the literature review.   to get a complete understanding of the methodology and practical implications of this work, readers must consult all the other papers in the series. the author notes “the findings should be considered with the other statistically significant relationships from this research” (p. 167–168). this makes it difficult for readers to understand the overarching significance of the study. it can also be a burden on readers’ time (elsevier, 2019).   the author has made good use of a previously validated survey (king, 2009). the author provides the survey in full in the appendix, allowing for replication.   respondents were self-selected, so they may not accurately represent the wider population. further, this analysis considers only those respondents (353/501, approx. 70.5%) who experienced instructional identity transformation. comparing the practices of those who have not experienced transformation to those who have may have been useful.   the author provides useful discussion around the study’s limitations and gives reasonable suggestions for future research.   the study will be of interest to academic library administrators looking for ways to support staff development. readers, however, would benefit from having a single, overarching publication detailing the key findings and most important recommendations resulting from the study.   references   elsevier. (2019). factsheet: salami slicing. elsevier. https://www.elsevier.com/__data/assets/pdf_file/0011/653888/salami-slicing-factsheet-march-2019.pdf   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387–399. https://doi.org/10.1108/07378830610692154   king, k. p. (2009). the handbook of the evolving research of transformative learning: based on the learning activities survey (10th ed). information age publishing.   mezirow, j. (1978). perspective transformation. adult education, 28(2), 100–110. https://doi.org/10.1177%2f074171367802800202   nichols hess, a. (2019). academic librarians’ educational factors and perceptions of teaching transformation: an exploratory examination. evidence based library and information practice, 14(3), 52–76. https://doi.org/10.18438/eblip29526   nichols hess, a. (2020). academic librarians’ teaching identities and work experiences: exploring relationships to support perspective transformation in information literacy instruction. journal of library administration, 60(4), 331–353. https://doi.org/10.1080/01930826.2020.1721939     news   eblip article wins 2018 jesse h. shera award      2018. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29455       the american library association library research round table selected, for the 2018 jesse h. shera award for distinguished published research, the publication “a systematic review of information literacy programs in higher education: effects of face-to-face, online, and blended formats on student skills and views” by alison l. weightman, damian j. j. farnell, delyth morris, heather strange, and gillian hallam. this paper was published in evidence based library and information practice (eblip), volume 12, issue 3 in 2017.   more information about the award and previous winners of the award are available from http://www.ala.org/aboutala/offices/ors/orsawards/sherapublished/sherajesseh     evidence summary   naming specific adverse effects improves relative recall for search filters identifying literature on surgical interventions in medline and embase   a review of: golder, s., wright, k., & loke, y.k. (2018). the development of search filters for adverse effects of surgical interventions in medline and embase. health information and libraries journal, 35(2), 121-129. https://doi.org/10.1111/hir.12213     reviewed by: ann glusker research & data coordinator national network of libraries of medicine, pacific northwest region university of washington health sciences library seattle, washington, united states of america email: glusker@uw.edu   received: 1 dec. 2018                                                                    accepted: 16 jan. 2019      2019 glusker. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29537     abstract   objective – “to develop and validate search filters for medline and embase for the adverse effects of surgical interventions” (p.121).   design – from a universe of systematic reviews, the authors created “an unselected cohort…where relevant articles are not chosen because of the presence of adverse effects terms” (p.123). the studies referenced in the cohort reviews were extracted to create an overall citation set. from this, three equal-sized sets of studies were created by random selection, and used for: development of a filter (identifying search terms); evaluation of the filter (testing how well it worked); and validation of the filter (assessing how well it retrieved relevant studies).   setting – systematic reviews of adverse effects from the database of abstracts of reviews of effects (dare), published in 2014.   subjects – 358 studies derived from the references of 19 systematic reviews (352 available in medline, 348 available in embase).   methods – word and phrase frequency analysis was performed on the development set of articles to identify a list of terms, starting with the term creating the highest recall from titles and abstracts of articles, and continuing until adding new search terms produced no more new records recalled. the search strategy thus developed was then tested on the evaluation set of articles. in this case, using the strategy recalled all of the articles which could be obtained using generic search terms; however, adding specific search terms (such as the mesh term “surgical site infection”) improved recall. finally, the strategy incorporating both generic and specific search terms for adverse effects was used on the validation set of articles. search strategies used are included in the article, as is a list in the discussion section of mesh and embase indexing terms specific to or suggesting adverse effects.   main results – “in each case the addition of specific adverse effects terms could have improved the recall of the searches” (p. 127). this was true for all six cases (development, evaluation and validation study sets, for each of medline and embase) in which specific terms were added to searches using generic terms, and recall percentages compared.   conclusion – while no filter can deliver 100% of items in a given standard set of studies on adverse effects (since title and abstract fields may not contain any indication of relevance to the topic), adding specific adverse effects terms to generic ones while developing filters is shown to improve recall for surgery-related adverse effects (similarly to drug-related adverse effects). the use of filters requires user engagement and critical analysis; at the same time, deploying well-constructed filters can have many benefits, including: helping users, especially clinicians, get a search started; managing a large and unwieldy set of citations retrieved; and to suggest new search strategies.   commentary   this paper adds to the substantial literature on the creation and limitations of search filters for biomedical citation searching in order to perform systematic reviews. the authors have been prolific contributors to this literature; they appear in ten of the fifteen articles referenced in this article. this paper builds on their earlier work, looking at non-drug interventions (golder et al., 2017). while they couldn’t characterize these in general, they found that they could characterize terms for surgical interventions, and this study is the outcome of their exploration. the resulting findings, building on their own past studies in a methodical and informed manner, create a valuable resource for both librarians and clinicians, and suggest further exploration on the part of the authors, as they note in their conclusions section.   the authors mention two limitations of their work: the sample size of articles examined is small; and they lack a true measurement for precision. in addition, for this evidence summary, methodologies were systematically assessed using glynn’s critical appraisal checklist (2006), raising questions about both sample size and replicability.   as for the sample size, there may be existing resources which would be appropriate for further research; one possibility is the extensive mcmaster plus citation database developed by hiru (the health information research unit at mcmaster university) (wilczynski, 2011; available at https://hiru.mcmaster.ca/hiru/hiru_mcmaster_plus_projects.aspx). wilczynski’s description of mcmaster’s approach to search filter development highlights the specialized nature of this work, and expands on some terms and concepts that golder et al. (2018) outlined. another useful article for clear definitions and process descriptions was that on medline indexing and adverse effects of oral contraceptives (wieland and dickersin, 2005). this is not a criticism, but more an acknowledgement that it may take additional reading beyond the golder et al. (2018) article to master its content. also important would have been a brief word about why medline and embase were the chosen databases for searching; a recent study by lam et al. (2018) offers interesting insights into the nature and uses of these two resources and illuminates context in the paper reviewed here.    this leaves (besides the question of a true measurement of precision, which is beyond our scope) the question of replicability. the explanations of the process and decisions in the article are meticulous and complete, but complex. this means that potentially there are decision points that might be handled differently by a replicating researcher, such as, which articles actually had adverse effects as a primary outcome (especially given that discrepancies between these researchers were resolved by discussion alone without a third party). however, this is a very minor point.   in conjunction with some of the other supporting pieces mentioned, this paper is overall an excellent and rigorously conducted and presented study with which to introduce oneself or one’s students to the area of search filter development. it also makes important contributions to the armamentarium of librarians and clinicians as they search for studies to guide their work. for those performing and supporting systematic reviews, it is extremely useful to have such a validated set of search strategies, both for reasons of efficiency and consistency.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154   golder, s., wright, k., & loke, y.k. (2017). the feasibility of a search filter for the adverse effects of nondrug interventions in medline and embase. research synthesis methods, 8(4), 506-513. https://doi.org/10.1002/jrsm.1267   lam, m.t., de longhi, c., turnbull, j., lam, h.r., & besa, r. (2018). has embase replaced medline since coverage expansion? journal of the medical library association, 106(2), 227-234. https://doi.org/10.5195/jmla.2018.281   wieland, s. & dickersin, k. (2005). selective exposure reporting and medline indexing limited the search sensitivity for observational studies of the adverse effects of oral contraceptives. journal of clinical epidemiology, 58(6), 560-567. https://doi.org/10.1016/j.jclinepi.2004.11.018   wilczynski, n. (2011). mcmaster university – hiru’s approach to search filter development. retrieved from https://hiru.mcmaster.ca/hiru/hiru_approach.pdf   evidence based library and information practice evidence based library and information practice 2009, 4:3  evidence based library and information practice       announcement    evidence‐based scholarly communication conference    jon eldredge, philip kroth, holly phillips, sally bowler‐hill  conference planning committee  university of new mexico health sciences library and informatics center  albuquerque, new mexico, united states of america  email: jeldredge@salud.unm.edu       © 2009 eldredge, kroth, phillips, and bowler‐hill. this is an open access article distributed under the terms of  the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits  unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.      many eblip readers benefit from accessing  this peer reviewed journal at no cost. as an  open access journal, eblip brings the latest  research and commentary free to librarians  and other information professionals  throughout the world. on march 11‐12, 2010,  in albuquerque, new mexico, the evidence‐ based scholarly communication conference  will highlight strategies for promoting open  access publishing as a method for the quick  dissemination of key research findings into  practice, otherwise known as “translational”  research. full details can be found at the  conference website at <  http://hsc.unm.edu/library/ebscconference/in dex.shtml>.                          eblip readers will be most interested in the  focus on evidence‐based methods for  promoting open access journals such as eblip  that bring the latest research findings to those  who need this information. the conference  will feature noteworthy evidence‐based  research reports in either contributed paper or  poster format. the best research will be  featured in a future issue of eblip. details  revolving around the october 1, 2009 deadline  for research submissions can be found at the  conference website. finally, conference  participants can enjoy a 73% probability of  sunshine with daily temperature highs of 61of  (16oc) with world‐class skiing and unique  shopping opportunities available nearby  either before or after the conference.  89 mailto:jeldredge@salud.unm.edu http://hsc.unm.edu/library/ebscconference/index.shtml http://hsc.unm.edu/library/ebscconference/index.shtml research article   library anxiety among marginalized university students in northeast india   tripti gogoi research scholar department of library and information science school of media and communication pondicherry university, puducherry, india email: tripti10aug@gmail.com   mangkhollen singson assistant professor department of library and information science school of media and communication pondicherry university, puducherry, india email: manglien@gmail.com    s. thiyagarajan assistant professor department of international business school of management pondicherry university, puducherry, india email: sthiyags@yahoo.com    received: 19 aug. 2019                                                             accepted: 28 oct. 2020      2021 gogoi, singson, and thiyagarajan. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29626     abstract   objective – library anxiety experienced by students has been discussed extensively for many decades. while the phenomenon is widely recognized, little attention has been paid to seeing its specific effect on marginalized sections of the society. the study attempts to understand the library anxiety experienced by students at three different universities in assam. assam is the only state in northeast india to have private, state, and central universities. these universities draw their student populations from several different hill states in northeast india, all of which face significant socio-political-economic challenges.   methods – a stratified random sample technique was used for the study. a total of 150 questionnaires were distributed equally among the three universities in assam and found 119 questionnaires were fit for analysis. the study adopted the modified and validated version of the bostick library anxiety scale developed by anwar, al-kandari, and al-qallaff (aqak) in 2004, with 32 item statements and 4 categories. the questionnaire is divided into two parts:  demographic variables and the library anxiety scale. the categories used for the study were: category 1 (staff approachability) – 11 statements; category 2 (feelings of inadequacy) – 6 statements); category 3 (library confidence) – 8 statements; and category 4 (library constraints) – 7 statements.   results – the study hypothesized that factors such as gender, the language of instruction, type of university, and caste or community do not influence library anxiety among northeast india students. however, the study's findings suggest that type of university influences library anxiety among students and its three constructs. tezpur university students experience a higher level of library anxiety. although no overall significant difference in the level of library anxiety was observed among students across gender (p=0.278, p> 0.05), the language of instruction (p=0.023, p> 0.05), castes and communities (p=0.223, p> 0.05), there was a significant difference in one construct of library anxiety among students based on gender (feelings of inadequacy), the language of education instruction (staff approachability), caste and community (feelings of inadequacy).   conclusions – results from the present study provided compelling evidence to suggest that many students, irrespective of their gender, the language of instruction, type of university, discipline, and caste or community experience library anxiety. the difference levels of library anxiety among independent variables indicate a critical lack of information literacy skills. overall, library anxiety scores among the students were moderate; some categories such as staff approachability, the feeling of inadequacy, and library constraint are the attributes of the students' anxiety. however, the findings of the study also suggest that students are confident in using the library. they are optimistic, enthusiastic, and keen to use library resources.      introduction   in recent years, there has been a strong aspiration for the indian political class to upgrade the standing of indian universities among other world-class academic institutions. economist amartya sen endorsed the idea that formal education empowers individual employability, social confidence, and other ‘capabilities’ (sen, 1999). however, despite many initiatives, there has been continued criticism of publicly funded higher education in india, for underperforming in various parameters compared with global university standards (tierney, sabharwal, & malish, 2019; deshpande, 2006; chanana, 1993). reductions in higher education budgets, library budgets, and education policy have contributed significantly to the problem. more importantly, however, the government of india’s reservation policies to the historically disadvantaged section of the society in higher education is often criticized for undermining merit.   the historic hindu caste system divided the majority of hindu society into rigid hierarchical groups based on their karma (work) and dharma (religion). they are divided into brahmins (priests, teachers), kshatriyas (warriors), vaishyas (farmers, traders, and merchants), shudra (labourers), and dalits (outcastes). discrimination based on caste lines is a common practice in india (kumar & hashmi, 2020). dalits have historically experienced caste discrimination from the majority caste hindu. these marginalized (i.e., educationally or socially disadvantaged) sections of the society include the dalits (scheduled caste or sc), adivasi (tribal or indigenous, schedule tribe or st), and later extended to other backward classes (obc).    central universities are established under an act of the parliament. they are autonomous and funded by the union ministry of higher education (previously known as the ministry of human resource development). the ministry of education adopted an inclusivity policy that enabled students from the marginalized sc, st, and obc communities in india to pursue higher education and established mandatory reservation policies regarding obc students (27%), sc students (15%), and st students (7.5%). subsequently, the reservation percentage was increased to 59.5% by including an additional 10% reservation for economically weaker sections (ews) provided that they do not identify themselves with any of the previously reserved categories of obc, sc, or st students. according to tierney, sabharwal, and malish, students from marginalized communities such as sc, st, and obc communities encounter various challenges to complete graduation and perform in universities (tierney, sabharwal, & malish, 2019). research conducted at the prestigious indian institute of technology provided evidence of the problems experienced by sc and st students (singh, 2017). some of these problems may be related to library anxiety.   academic libraries routinely conduct library orientation programs to promote the library and educate students to become self-reliant in using the library and becoming a lifelong learner. the students then apply these library skills to locate the required information and resources in the library. however, due to the overwhelming amount of information resources available, students are often confused and disoriented. the introduction of computers and information technology in academic libraries has further increased students’ experience of disorientation (jiao & onwuegbuzie, 2004; ahmed & aziz, 2017). today, technologically driven electronic resources have replaced traditional print resources. thus, students who lack the necessary skills to cope with the rapid technological transition have often developed fear and anxiety when contemplating a library visit (kohrman, 2003). students may also experience library anxiety for other reasons, such as lack of self-confidence in the conduct of study, lack of information retrieval skills, lack of previous exposure to academic libraries, and lack of knowledge of library equipment and technology (jiao & onwuegbuzie, 2004).   students in northeast india may be particularly prone to experiencing these conditions because of the socio-political-economic isolation of the region from the rest of india. the northeast region of india constitutes eight states: assam, meghalaya, sikkim, tripura, mizoram, manipur, arunachal pradesh, and nagaland. the region has been marginalized in social, political, and economic terms compared with the rest of india due to its poor connectivity and communication resources, rugged terrain, and recurring armed conflicts. however, unlike the rest of india, st and sc communities in northeast india are geographically isolated, and as such they may have never experienced historical caste discrimination. however, people from the northeast india experience racial discrimination and are marginalized as they venture out from the northeast to seek employment and education (wouters & subba, 2013).   as a result of the government’s inclusivity policy, students from different parts of india and northeast india seek higher education in assam as the state provides one of the region's best educational infrastructures. this level of access for all students ensures that assam universities' demographic profiles are very multicultural, representing tribal, dalit, caste hindu, muslims, and christians, thus providing an ideal environment for research investigating factors related to library anxiety.   literature review   measuring library anxiety   various researchers have adopted the bostick library anxiety scale (las) to understand library anxiety and its associated reasons. the studies by jiao and onwuegbuzie suggests that library anxiety could be correlated with personal characteristics such as age and nationality (jiao, onwuegbuzie, & lichtenstein, 1996; jiao, onwuegbuzie, & bostick, 2006); freshmen library experience (jiao & onwuegbuzie, 2004); low levels of academic self-competence (jiao &onwuegbuzie, 1999); cooperative students (jiao & onwuegbuzie, 2002); and physiological, cognitive, affective, and behavioral ramification (jiao, onwuebuzie & lichtenstein, 1996). other researchers have also linked library anxiety with critical thinking and language (kwon, onwuegbuzie & alexander, 2007; sinnasamy & karim, 2017). in their respective countries and cultures, several researchers have validated and altered the scale (shoham and mizrachi, 2001; świgoń, 2011; anwar, al-qallaf, al-kandari & al-ansari et al., 2012; song, zhang, & clarke, 2014). however, ahmed and aziz who conducted a study at bangladesh university challenged bostick's las psychometric soundness in a developing country perspective (ahmed & aziz, 2017). they argued that today's library infrastructure has undergone a significant change ever since bostick proposed las in 1992 (ahmed & aziz, 2017).   measuring library anxiety with special populations   in 2004, anwar, al-kandari, and al-qallaff developed a new scale (aqak) based on the original bostick las to examine the status of library anxiety among undergraduate students of kuwait whose mother tongue was arabic but used english as a second language. the study found differences in categories such as feelings of inadequacy and library constraints. kuwaiti students who were weak in english were more likely to face frustration sooner and displayed a higher level of library anxiety (anwar, al-kandari, & alqallaf, 2004). similarly, jiao, onwuebuzie, and bostick (2004) conducted the first study that focused on race as a factor in the level of library anxiety. they studied the library anxiety among caucasian-american and african-american graduate students and reported that african-american students have lower library anxiety levels correlated with three dimensions of library anxiety than their caucasian-american peers (jiao, onwuebuzie, & bostick, 2004; jiao & onwuebuzie, 2004).   using the aqak scale, another survey conducted at kuwait university by anwar, al-qallaf, al-kandari, and al-ansari stated that there is no significant relationship between gender, school, or college history in the library anxiety score (al-qallaf, al-kandari & al-ansari, 2012). correspondingly, research conducted among international students indicates no substantial difference in the degree of library anxiety scores between gender and discipline (lu & adkins, 2012). however, modifying the existing bostick las, doris, provata, and vraimaki developed the greek library anxiety scale (g-las) and found that male students experienced higher anxiety scores than female students (provata & vraimaki, 2015). this phenomenon according to researchers is due to the fact that female students are more driven and prefer to go to the root of a specific problem with persistence and depth compared to their male counterparts (kennedy, wellman & klement, 2003; halder & chakrabarty, 2010). shehata and elgllab’s recent comparative study of library anxiety among saudi arabian and egyptian students found that egyptian students were more anxious about using libraries than saudi arabian students (shehata & elgllab, 2019).   library anxiety studies in developing countries   numerous studies from developing countries have also been conducted in recent years to explain library anxiety among students. noor harun and ansari implemented a modified version of bostick's las that examined the effects of gender, nationality, and bibliographic instruction on library anxiety among malaysian students. the findings of their analysis suggest that there is a statistically significant mean difference between malaysian and non-malaysian students in the affective barriers portion of library anxiety. malaysian students experience higher library anxiety related with affective barriers than their non-malaysian counterparts (noor harun & ansari, 2011). similarly, sinnasamy and karim conducted a study among malaysian students and reported the prevalence of at least one or more library anxiety dimensions among non-native english malaysian students. research shows that higher library anxiety scores have been associated with the language barrier (sinnasamy & karim, 2017).   adopting the aqak modified las, jan, anwar, and warraich conducted a study among pakistani students to examine the relationship between library anxiety with library use, gender, academic performance (grade point average), and academic discipline. the results of the study showed that students experience moderate anxiety in the library, and there is a statistically significant difference in gender and discipline anxiety in the library. however, jan, anwar, and warraich (2016) found that library anxiety and academic performance were inversely proportional. similarly, adopting the qualitative method, abusin & zainab (2017) conducted a study in africa using the diary method among sudanese university students. they found that, when using the academic library to write their first research paper, 88.2% of the students displayed fear and anxiety. the students also reported on the language barrier, particularly english, where the mastery of the english language among students was found to be poor. in bangladesh, ahmed & aziz (2017) conducted a study using the original bostick scale to indicate that university students may experience anxiety related to library services based on their technological proficiency.   aims   this research study attempts to understand the nature of marginalized students' library anxiety experience in northeast india.   the research questions that motivated this study were:   does gender, educational background, or language influence students' library anxiety score in three different university settings? does discipline or type of university in which they study influence students' library anxiety score in three other university settings? does caste and community influence students' library anxiety score in three different university settings?   methods   participants in the study   the present study selected three universities in assam. the state of assam is the only northeast state of india to have private, state, and central universities. central universities are union government-funded universities established under parliament's act, and they follow the central reservation policy. the state government funds state universities and follows the state reservation policy, which is different from the central reservation policy. private universities are purely self-funded and do not follow any reservation policy. the three universities selected for the study were: kaziranga university (private), dibrugarh university (state), and tezpur university (central). the libraries in these universities are well equipped with modern and integrated library systems. they cater to students from all of the six neighboring hill states of assam and elsewhere in the country. the total population of the students was 2,560 (kaziranga university), 4408 (dibrugarh), and 3,059 (tezpur university).   the sampling technique used for the study was a stratified random sampling technique. lists of first-year students enrolled in the year 2017 for a master's degree in arts, commerce, management, and science were considered for the study. the sample did not include medicine, engineering, or research-based degrees like jd or ph.d. the study identified 3 universities, and 50 students from each university were randomly selected using microsoft excel's random number generation function. the total number of first-year postgraduate students enrolled in the academic year 2017-18, in all three universities in total, was 1,578. a sample pool of 150 students represents approximately 10% of the population. for the data collection, the researchers personally visited each university to distribute the questionnaire and then collected them with the help of volunteers from april 23rd to june 19th, 2018. since the sampling technique was stratified random, the respondents were identified before and informed over e-mail that they would need to assemble at a venue, based on their convenience, for data collection. the number of responses received from the three universities was kaziranga university (38 responses), dibrugarh university (39 responses), and tezpur university (42 responses), with a total response rate of 79.3% in aggregate (119 out of 150). data were analyzed using excel and spss.   instrument and procedures   the questionnaire (see appendix a) consists of demographic variables and the library anxiety scale (las). the las consists of 4 categories and 32 statements as modified and validated by anwar, al-kandari & al-qallaf (2004) from the original bostick library anxiety scale. the categories used for the study are category 1 (staff approachability) – 11 statements, category 2 (feelings of inadequacy) – 6 statements, category 3 (library confidence) – 8 statements, and category 4 (library constraints) – 7 statements. these statements are rated based on students’ library experiences using a five-point likert scale (1, strongly disagree; 2, disagree; 3, undecided; 4, agree; and 5, strongly agree). the study adopted independent t-test and anova tools for statistical analysis. the independent t-test was used to analyze the difference between two categories or groups such as gender, group, or college. in contrast, the one-way anova was used to analyze the differences in more than two groups/categories such as communities, language, university, and educational background.   results   demographic data   table 1 shows the demographic profile of 119 respondents. all but 9 of the students (92.44%) hail from northeastern states, and just 7.5% of respondents hail from outside northeastern states. based on the respondent's community profile, the majority of the respondents belonged to obc (35.3%), followed by general (33.6%), st (19.3%), and sc (11.7%).     table 1 frequency and percentage of the respondents from assam universities in india sl. no. variables categorization no. of respondents % 1. gender male 60 50.42% female 59 49.58% 2. nativity (geographical origin) north-east students 110 92.44% outside north east 9 7.56% 3. caste/community general 40 33.60 obc 42 35.30 sc 14 11.76 st 23 19.34 4. college education background private 52 43.69 public 67 56.31 5. schooling education background private 67 56.30 public 52 43.70 6. language of instruction in school english 56 47.1 vernacular 25 21.0 both 38 31.9 7. school educational institution location urban 41 34.4 rural 36 30.3 semi urban 42 35.3 8. library in college yes 119 100.0 no 0 0.0 9. library in school yes 87 73.10 no 32 26.90     the study results also indicated that the majority (56.31%) of the respondents graduated from a public-funded institution, while 43.69% of respondents graduated from a private institution. by comparison, most of the respondents (56.30%) completed their school education at a private institution compared to 43.70% of respondents from publicly funded government schools. similarly, most of the respondents reported english (47.1%) as the language of instruction in their schools, while only 21% used a vernacular language; 31.9% of respondents stated that both english and vernacular were the languages of instruction. for the students who belong to northeastern states, the primary languages of instructions were english, bhutia, bodo, khasi, meitei, and mizo (lushei).   the majority (35.3%) of the respondents graduated from the semi-urban schools, followed by 34.4% urban and rural (30.3%). interestingly, while 100% of the respondents stated there was a library in their college, only 73.1% of the respondents acknowledged having a library in the school. in comparison, 26.9% of respondents did not have a library in their school. library hours in schools are mandatory in the timetable to foster good reading habits, but are not mandatory in college.   reliability and normality of library anxiety scale   cronbach's alpha is a statistical analysis or tool that demonstrates the reliability of scales. for the present study, anwar's (2004) scale was adopted to measure the level of library anxiety among postgraduate students. the las was administered and pre-tested for culturally different populations of university students in assam. the pilot study and pre-testing result indicated that the cronbach's alpha (table 2): (i) staff approachability 0.85, (ii) feeling of inadequacy 0.79, (iii) library confidence 0.73 and (iv) library constraints 0.74 and the overall scale of 0.87. from appendix b table 2 the null hypothesis of data normally distribution cannot be rejected at a 1% level for all the variables. therefore, it can be concluded that the data are normally distributed (staff approachability, feelings of inadequacy, library confidence, and library constraints).     table 2 cronbach’s alpha score sl. no. category cronbach’s alpha score 1. staff approachability 0.85 2. feelings of inadequacy 0.79 3. library confidence 0.73 4. library constraints 0.74 5. overall scale 0.87     hypothesis testing   we hypothesized that library anxiety is not prevalent among students from marginalized castes or communities. factors such as gender, the language of instruction, discipline, and type of university a student attends do not contribute to anxiety.   hypotheses for the study   h01: gender of the student does not have any influence on library anxiety.   h02: students’ language of school instruction does not influence library anxiety levels.   h03: type of university does not influence the level of library anxiety.   h04: caste and community do not influence library anxiety among students.     influence of gender on library anxiety   h01: gender of the student does not have any influence on library anxiety.   table 3 shows that there is no significant influence of gender on library anxiety scores between male and female students (t (117) = 1.090, p=0.278). however, the result of the scores suggests that there is a significant difference in library anxiety scores between males and females in feelings of inadequacy (category 2, p=0.035). the male respondents (mean=20.70) have higher library anxiety when compared with female respondents. the effect size for the test feelings of inadequacy is 0.390, which is termed as small (cohen, 1992).     table 3 t-test of library anxiety score library score of students with respect to gender and education background (school and college) variable group n category 1   staff approachability category 2   feelings of inadequacy category 3   library confidence category 4   library constraints total   mean mean mean mean mean gender male 60 44.85 (5.87) 20.70 (4.70) 16.77 (3.46) 24.97 (3.77) 107.28 (11.53) female 59 44.80 (5.09) 18.85 (4.55) 16.88 (3.42) 24.68 (3.01) 105.20 (9.12) t 0.053 2.129** 0.182 0.461 1.090 p -value 0.958 0.035 0.856 0.646 0.278 cohen’s d 0.009 0.390 -0.033 0.084 0.200 school private 67 44.36 (5.74) 19.42 (4.67) 17.34 (3.72) 24.60 (3.40) 105.72 (10.62) public 52 45.42 (5.10) 20.25 (5.01) 16.15 (2.91) 25.12 (3.42) 106.94 (10.19) t -1.053 -0.934 1.898 -0.823 -0.635 p -value 0.294 0.352 0.060 0.412 0.526 cohen’s d -0.196 -0.172 0.375 -0.152 0.117 college private 52 44.15 (6.45) 19.42 (4.80) 17.33 (3.71) 24.63 (3.60) 105.54 (11.70) public 67 45.34 (4.56) 20.06 (4.85) 16.43 (3.17) 24.97 (3.27) 106.81 (9.34) t -1.178 -0.714 1.417 -0.532 -0.657 p -value 0.241 0.477 0.159 0.596 0.512 cohen’s d -0.213 -0.131 0.259 -0.097 -0.119 library in school yes 87 44.33 (5.55) 20.08 (4.69) 16.96 (3.52) 24.74 (3.36) 106.16 (10.74) no 32 46.15 (5.17) 18.87 (5.14) 16.53 (3.21) 24.90 (3.67) 106.46 (9.96) t -1.663 1.159 0.635 -0.222 -0.144 p -value 0.102 0.252 0.528 0.825 0.886 cohen’s d -0.338 0.245 0.127 -0.045 -0.029 **significance at 5%, value in parenthesis is standard deviation (sd)     influence of medium (language) of instruction in schools on library anxiety   h02: students’ language of school instruction does not influence library anxiety levels.   table 4 shows that the significance value of the language variable (language of education instruction) is more than 0.05. hence, the null hypothesis (h02) cannot be rejected. the medium of education instruction in schools does not influence the overall library anxiety of the students. but there is an influence of medium of instruction on staff approachability (second construct of library anxiety) as the significance value is less than 0.05(p = 0.015) with the effect size f statistics for the test staff approachability being 4.33. however, there was no significant difference in the level of feelings of inadequacy, library confidence, and library constraints among students coming from different mediums of instruction in schools. the effect size (partial eta square) for category 1 (staff approachability) is 0.069 which is termed as medium effect size (cohen, 1992).     table 4 one-way anova result of library anxiety score with respect to university, discipline, community, and language variable n category 1   staff approachability category 2 feelings of inadequacy category 3 library confidence category 4 library constraints total library anxiety mean mean mean mean mean university dibrugarh university 39 43.67 (3.99) 17.87 (4.63) 17.08 (3.30) 24.18 (2.88) 102.79 (8.03) kaziranga university 38 42.82 (5.66) 19.21 (3.72) 17.53 (3.68) 23.79 (3.35) 103.34 (10.42) tezpur university 42 47.71 (5.38) 22.07 (5.02) 15.95 (3.19) 26.36 (3.42) 112.10 (10.01) f 10.84*** 9.20*** 2.31 7.45*** 12.19*** p value 0.000 0.000 0.104 0.001 0.000 effect size (partial eta square) 0.157 0.137 0.038 0.114 0.174 discipline humanities 4 45.50 (2.38) 18.75 (3.77) 16.25 (2.63) 24.75 (2.50) 105.25 (4.99) science 80 44.29 (5.59) 19.60 (4.87) 16.73 (3.37) 24.78 (3.46) 105.40 (10.71) social science 35 45.97 (5.37) 20.31 (4.87) 17.11 (3.69) 24.91 (3.43) 108.31 (10.08) f 1.19 0.36 0.21 0.02 0.97 p-value 0.308 0.699 0.809 0.983 0.381 effect size (partial eta square) 0.020 0.006 0.0040 0.000 0.016 caste/community general 40 44.98 (5.15) 19.83 (5.03) 16.60 (3.34) 25.65 (3.62) 107.05 (9.28) obc 42 44.60 (5.29) 20.90 (4.28) 17.05 (4.00) 24.50 (3.13) 107.05 (10.40) sc 14 43.93 (7.61) 16.64 (5.77) 16.00 (2.86) 24.21 (4.14) 100.79 (14.41) st 23 45.52 (5.11) 19.57 (4.14) 17.30 (2.79) 24.35 (2.93) 106.74 (9.10) f 0.279 2.89** 0.530 1.218 1.481 p-value 0.840 0.038 0.663 0.306 .223 effect size (partial eta square) 0.007 0.070 0.014 0.031 0.037 language english 56 43.34 (5.43) 19.54 (4.39) 17.45 (3.52) 24.32 (3.36) 104.64 (10.97) vernacular 25 45.60 (5.70) 20.36 (5.96) 16.68 (3.69) 25.84 (4.23) 108.48 (11.57) both 38 46.50 (5.00) 19.76 (4.69) 16.00 (3.00) 24.89 (2.72) 107.16 (8.50) f   4.33** 0.25 2.08 1.75 1.39 p-value   0.015 0.779 0.130 0.178 0.252 effect size (partial eta square)     0.069 0.004 0.035 0.029 0.023 *** significant at 1%, ** significant at 5%, value in parenthesis is standard deviation (sd)     influence of type of university on library anxiety   ho3: type of university does not influence the level of library anxiety.   table 4 shows that the significance value for university (types) is less than 0.05. hence, the null hypothesis h02 is rejected. this implies that there is an influence on the types of universities on the level of library anxiety, except category 3 (library confidence), where the p-value of category 3 is more than 0.05. the results show there is an effect of university (types) on three categories; staff approachability (p=0.000, p < 0.05, effect size =0.157), feelings of inadequacy (p= .000, p < 0.05, effect size= 0.137) and library constraints (p = .001, p < 0.05, effect size= 0.114) as their respective p values are less than 0.05. however, there was no significant difference in the level of library confidence among students studying in different universities. the results also suggest that among the universities, tezpur university students experience higher levels of anxiety than students at the other two universities. the effect size (partial eta square) value for library anxiety is 0.174, which is termed as a medium effect size (cohen, 1992).   similarly, student disciplines such as social science, science, and humanities do not influence library anxiety in all the constructs (p=0.381, p>0.05).   influence of caste/community on library anxiety   h04: caste and community do not influence library anxiety among students.   as shown in table 4, there is no significant difference in library anxiety levels among different student communities as a significant value for the variable community is more than 0.05(p=0.223). hence, the null hypothesis h03 cannot be rejected. therefore, there is no influence of the caste/community on library anxiety levels. however, the p-value in the category (feelings of inadequacy) is less than 0.05, indicating that there is an influence of community on feelings of inadequacy (p=0.038). the interesting observation from the testing communities indicated that obc students experience more anxiety during their visit to the library.   the f statistics for feelings of inadequacy are 2.89 with a p-value of less than 0.05 (p =0.038). hence, community background influences library anxiety scores among students for feelings of inadequacy. the effect size (partial eta square) for the same (feelings of inadequacy) is 0.070, which is termed as medium effect size (cohen, 1992).   subsequently, other factors considered for the study included whether the students’ educational background, such as type of school, college, or presence of the library in their school, determine their library anxiety level.   library anxiety among students with different school and college educational backgrounds   table 3 shows the library anxiety score of students based on their college and school educational backgrounds. overall result of t-test suggests that there is no significant difference among students from different educational backgrounds, for both college (t (117) = 0.657, p= 0.512) and school (t (117) = -0.635, p = 0.526). therefore, before joining the university, whether a student studied in a private or public school or college is immaterial.   discussion   the university library commonly causes library anxiety in northeast india. using library jargon to explain a complicated interface following an obscure classification scheme may lead to higher las values. the study's findings confirm the occurrence of library anxiety among the students who responded to our questionnaire. irrespective of their gender, the language of school instruction, type of university, caste, and community, library anxiety as a phenomenon is common among students. the questionnaire responses indicate that the anxiety score level was mild (overall mean score = 3.31) among the students studying at the three sample universities in assam. however, the study found interesting trends.    the present findings are in line with previous studies (anwar, al-qallaf, al-kandari & al-ansari, 2012; lu & adkins, 2012; lawless, 2011). the current study results suggest no significant difference in library anxiety scores between male and female students. however, male students experience higher anxiety levels in the categories of inadequacy (categories 2, p=0.035) than female students. the male students indicated a sign of confusion, unsure of using the library. perhaps the indian culture of very sociable female students and their motivation to compete with the opposite sex greatly reduces their anxiety while visiting the library.   although the language of instruction in their education did not significantly influence library anxiety, students coming from vernacular and both (vernacular and english) as the medium of instruction experience severe library anxiety in the category of staff approachability (average is 4.15 and 4.23).  hence, students either experience communication problems, lack the confidence to communicate with library staff, or perceive library staff as unfriendly. simultaneously, students’ low scores (average of 2.18, 2.09, and 2.00) in the category library confidence (negative construct) suggest the lack of confidence and comfort in making use of library services.   this finding suggests that teaching information literacy may lead to greater familiarity with the library resources and less anxiety, as whitmire (2007) found. furthermore, it is possible that the language barrier between the students and the library staff further increases the level of library anxiety.   the student's discipline does not influence library anxiety, but the type of university does. students at the central university (tezpur university) exhibited the highest library anxiety level (average was 3.50) among the three universities. tezpur university students experience severe anxiety in the categories staff approachability (mean average 4.34), feelings of inadequacy (mean average 3.68), and library constraints (mean average 3.77). tezpur university is a central university, and the infrastructures, including library resources, are funded by the central (union) government. it was ranked 29th by the national institutional ranking framework (nirf) in 2019, and students enrolled from all over the country. unlike state and private universities, tezpur university conducts an annual national entrance examination for university admission students. there is, therefore, an enormous competition among students to get admission to one of india's leading universities. hence, students experience anxiety in communication, assistance, restrictive library rules, regulations, and library resources.   based on previous research from the majority caste hindu perspective, the researchers hypothesized that there would be a significant difference in the experience of library anxiety among the marginalized students of northeast india based on caste/community. however, this study's results suggest that there is no significant difference in the level of anxiety among students from different castes/communities (p=0.223, p> 0.05). interestingly, students with an obc or general community profile experience higher library anxiety (average means 3.35) while schedule caste students' level of anxiety is the lowest among the group of students (average mean 3.15). this study’s findings are in line with jiao, onwuebuzie, and bostick's findings that african americans demonstrated lower library anxiety levels than their caucasian-american counterparts. however, unlike jiao, onwuebuzie, and bostick, the students who participated in the study were from three universities and did not differ in the types of the institution they attended. one possible explanation of this phenomenon is that participants were among the highest-achieving st and sc graduate students in the northeastern region and the rest of india. in addition, unlike their counterparts elsewhere in india, indigenous tribal (st) and scheduled caste (sc) groups in northeast india may have never encountered historical caste discrimination.   limitations and suggestions for future research   the limitation of the study was that the sample was limited to only three universities in assam. hence, the sample size for each category was small and limited the possibility of generalizing the results.   the las scale adopted for the survey was the aqak (a modified version of the bostick scale developed for use among kuwaiti undergraduate students whose mother tongue was arabic). although the present scale's reliability was good, developing and validating a new version of the original bostick scale with 43 statements complying with the indian multicultural context would significantly contribute to the knowledge base. the questionnaire did not include the desired number of representations from students outside assam that may help provide a better understanding of the language barrier. although the questionnaire helped determine the students' overall anxiety level, including an open-ended questionnaire for respondents' comments could provide more insight into the reasons for the anxiety.   conclusion   students enrolled in universities world-wide are required to submit assignments every semester that require a visit to the library for resources. previous research literature suggests that library anxiety may result in library avoidance, procrastination, poor quality deliverables, and even increased dropouts among the students (onwuegbuzie & jiao, 2000). therefore, it is important to look for ways to reduce library anxiety in student populations. the present study and those from the previously published research provide compelling evidence to suggest that many students experience library anxiety. they lack the confidence and communication skills to seek help from a librarian successfully. schools and colleges are intended to impart necessary prerequisite library skills, information resources, and research skills, bridge the gap, and prepare students for university; however, students often remain unprepared. although the overall library anxiety score among the students was found to be moderate (overall average score= 3.32), the findings indicate that library classification schemes, confusing formats, unfamiliar jargon, rfid, imposing online catalogue systems, and unfriendly library staff made a significant contribution to the degree of library anxiety. similarly, categories such as staff approachability, feelings of inadequacy, and library constraint indicated higher anxiety scores among the students.   many of the students in assam come from the neighbouring states of northeast and other states in india. the student's cultural background and unfamiliarity with the existing library system contributed significantly to the level of library anxiety. while some of them may be familiar with library systems in their previous schools and colleges, their present university's existing resources and technology may differ. hence, library orientation and library instruction to the newly enrolled student could reduce the anxiety levels. while the students' previous school instruction's language does not significantly influence the overall level of library anxiety, there is a significant difference in staff approachability.   although the results of this study indicated no significant difference in the level of library anxiety based on caste/community or gender, there was a substantial difference in the feeling of inadequacy across these variables. furthermore, there was a significant difference in library anxiety among students based on the different languages of instruction and the university that the student attended. students enrolling in the university whose medium of instruction while in school was vernacular or both (vernacular and english), experience higher anxiety than students from the english medium of instruction when approaching the staff of the library. furthermore, since english is now the universally accepted standard for scholarly communication in india and elsewhere, the potential for students’ scholastic achievements and academic discourse could depend on their mastery of the english language. finally, the results from the present study provide strong evidence to suggest that while many students, irrespective of their caste/community and/or gender experience library anxiety, many students are confident in using the library. these students are optimistic, enthusiastic, and keen to use library resources. overall, library anxiety scores among the student population were found to be moderate, and while some categories such as staff approachability, the feeling of inadequacy, and library constraint exist, these are areas in which increased library orientation and library instruction, to the newly enrolled student, could reduce the level of the anxiety the student experiences.   acknowledgement   the authors would like to thank mr. bobby, mr. binoy bhushan konwar, mr. nabajit bora, miss. sewali taid and mr. tanushyam mili for their help and assistance in collecting the data from the three universities and ph.d. fellowship from pondicherry university.   references   abusin, k. a., & zainab, a. n. (2017). exploring library anxiety among sudanese university students. malaysian journal of library & information science, 15(1), 55-81.   ahmed, s. z., & aziz, t. b. (2017). use of bostick’s library anxiety scale (las) in a developing country perspective. library review, 66(4/5), 282-296. https://doi.org/10.1108/lr-06-2016-0052   anwar, m. a., al-kandari, n.m. & al-qallaf, c.l. (2004). use of bostick's library anxiety scale on undergraduate biological sciences students of kuwait university. library & information science research, 26(2), 266-283. https://doi.org/10.1016/j.lisr.2004.01.007   anwar, m. a., al-qallaf, c. l., al-kandari, n. m., & al-ansari, h. a. (2012). aqak: a library anxiety scale for undergraduate students. journal of librarianship and information science, 44(1), 36-46. https://doi.org/10.1177/0961000611425568   bostick, s. l. (1992). the development and validation of the library anxiety scale. (unpublished doctoral dissertation). wayne state university, detroit. http://digitalcommons.wayne.edu/dissertations/aai9310624. (aai9310624)   chanana, k. (1993). accessing higher education: the dilemma of schooling women, minorities, scheduled castes and scheduled tribes in contemporary india. higher education, 26(1), 69-92. https://doi.org/10.1007/bf01575107   cohen, j. (1992). statistical power analysis. current directions in psychological science, 1(3),98-101. https://doi.org/10.1111/1467-8721.ep10768783.   deshpande, s. (2006). exclusive inequalities: merit, caste, and discrimination in indian higher education today. economic and political weekly, 2438-2444. https://www.jstor.org/stable/4418346   doris, k. a., provata, p. a., & vraimaki, e. (2015) library anxiety among undergraduate students in greece. international journal on integrated information management, 2 (2005), 25-37. http://ejournals.uniwa.gr/index.php/jiim/article/view/3065   halder, s., ray, a., & chakrabarty, p. k. (2010). gender differences in information seeking behavior in three universities in west bengal, india. the international information & library review, 42(4), 242-251. https://doi.org/10.1080/10572317.2010.10762869   jan, s. u., anwar, m. a., & warraich, n. f. (2016). library anxiety, library use and academic performance of undergraduate students in pakistan. library review, 65(8/9), 564-577. https://doi.org/10.1108/lr-03-2016-0024   jiao, q. g., & onwuegbuzie, a. j. (1999). self‐perception and library anxiety: an empirical study. library review, 48(3), 140-147. https://doi.org/10.1108/00242539910270312   jiao, q. g., & onwuegbuzie, a. j. (2002). dimensions of library anxiety and social interdependence: implications for library services. library review, 51(2), 71-78. https://doi.org/10.1108/00242530210418837   jiao, q. g., & onwuegbuzie, a. j. (2004). the impact of information technology on library anxiety: the role of computer attitudes. information technology and libraries, 23(4), 138-144. https://doi.org/10.6017/ital.v23i4.9655   jiao, q. g., onwuegbuzie, a. j., & bostick, s. l. (2004). racial differences in library anxiety among graduate students. library review, 53(4), 228-235. https://doi.org/10.1108/00242530410531857   jiao, q. g., onwuegbuzie a. j. & lichtenstein a. a. (1996). library anxiety: characteristics of ‘at-risk’ college students. library & information science research, 18(2), 151-163. https://doi.org/10.1016/s0740-8188(96)90017-1    jiao, q. g., onwuegbuzie, a. j., & bostick, s. l. (2006). the relationship between race and library anxiety among graduate students: a replication study. information processing & management, 42(3), 843-851. https://doi.org/10.1016/j.ipm.2005.03.018   kennedy, t., wellman, b., & klement, k. (2003).gendering the digital divide. it and society, 1(5), 72e96, retrieved 15th january from https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.177.1749&rep=rep1&type=pdf     kohrman, r. (2003, april). computer anxiety in the 21st century: when you are not in kansas any more. in eleventh association of college & research libraries conference, charlotte, united states.   kumar, a., & hashmi, n. i. (2020). labour market discrimination in india. the indian journal of labour economics, 63, 177–188.   kwon, n., onwuegbuzie, a. j., & alexander, l. (2007). critical thinking disposition and library anxiety: affective domains on the space of information seeking and use in academic libraries. college & research libraries, 68(3), 268-278. https://doi.org/10.5860/crl.68.3.268   lawless, l. j. (2011).an examination of library anxiety at cape breton university. evidence based library and information practice, 6(3), 16-27. https://doi.org/10.18438/b8jc9x   lu, y., & adkins, d. (2012). library anxiety among international graduate students. proceedings of the american society for information science and technology, 49(1), 1-4. https://doi.org/10.1002/meet.14504901319   noor harun, a. k., & ansari, n. a. (2011). examining the effects of gender, nationality, and bibliographic instruction on library anxiety. malaysian journal of library & information science, 16(2), 141–54.   onwuegbuzie, a. j., & jiao, q. g. (2000). i’ll go to the library later: the relationship between academic procrastination and library anxiety. college & research libraries, 61(1), 45-54. research libraries, 47(2), 160-165. https://doi.org/10.5860/crl.61.1.45   sen, a., (1999). development as freedom. oxford university press.   shehata, a. m. k., & elgllab, m. f. m. (2019). library anxiety among undergraduate students: a comparative study on egypt and saudi arabia. the journal of academic librarianship, 45(4), 386-393. https://doi.org/10.1016/j.acalib.2019.05.006   shoham, s. &mizrachi, d. (2001). library anxiety among undergraduates: a study of israeli b. ed students. the journal of academic librarianship, 27(4), 305-311. https://doi.org/10.1016/s0099-1333(01)00216-6   singh, c. b. (2017). multilevel exclusion of dalit students in professional elite colleges of india. social change, 47(3), 359-372. https://doi.org/10.1177/0049085717712814   sinnasamy, j., & karim, n. h. a. (2017). academic related anxieties: a case study investigating the relationships among library, communication, and language anxieties among non-native speakers of english. malaysian journal of library & information science, 20(2), 1-12. https://doi.org/10.1016/j.acalib.2014.07.010   song, z., zhang, s., & clarke, c. p. (2014). library anxiety among chinese students: modification and application of las in the context of chinese academic libraries. the journal of academic librarianship, 40(1), 55-61. https://doi.org/10.1016/j.acalib.2013.10.014   świgoń, m. (2011). library anxiety among polish students: development and validation of the polish library anxiety scale. library & information science research, 33(2), 144-150. https://doi.org/10.1016/j.lisr.2010.09.003   tierney, w. g., sabharwal, n. s., & malish, c. m. (2019). inequitable structures: class and caste in indian higher education. qualitative inquiry, 25(5), 471-481. https://doi.org/10.1177/1077800418817836   whitmire, e. (2007). a longitudinal study of undergraduates' academic library experiences. journal of academic librarianship, 27(5), 379-385. https://doi.org/10.1016/s0099-1333(01)00223-3.   wouters, j. j., & subba, t. b. (2013). the “indian face,” india's northeast, and “the idea of india”. asian anthropology, 12(2), 126-140. https://doi.org/10.1080/1683478x.2013.849484.   appendix a   library anxiety among post-graduate students in india: a survey 1. name: mr. /ms. (optional)……………………………………………………………………. 2. gender: male [ ] female [ ] 3. state/country: ……………………...………...…………………………... 4. existing educational qualification: ………………………………………… 5. pg course undertaken: …………………………………………………….. 6. department: …………………………………………………………………………………… 7. name of institute:……………………………………….……………………. 8. community: (a) general [ ] (b) obc [ ] (c) sc [ ] (d) st [ ] (e) others [ ] 9. college educational background: (a) private university [ ] (b) public [ ] (c) ngo funded [ ] 10. schooling background: (a) private institution [ ] (b) public [ ] (c) ngo funded institution [ ] 11. medium of instruction in schools: (a) english [ ] (b) vernacular [ ] (c) both [ ] 12. my college had a library: (a) yes [ ] (b) no [ ] 13. my school had a library: (a) yes [ ] (b) no [ ]   category 1: staff approachability s2 the librarians are unapproachable. s3 the reference librarians are unhelpful. s4 the librarians don't have time to help me because they're always on the telephone. s5 i can't get help in the library at the times i need it. s6 library clerks don't have time to help me. s7 the reference librarians don't have time to help me because they're always busy doing  something else. s11 the reference librarians are not approachable. s18 the reference librarians are unfriendly. s24 the library staff doesn't care about students. s29 librarians don't have time to help me. s33 the library staff doesn't listen to students.   category 2: feelings of inadequacy s1 i'm embarrassed that i don't know how to use the library. s8 i am unsure about how to begin my research. s9 i get confused trying to find my way around the library. s10 i don't know what to do next when the book i need is not on the shelf. s27 i don't understand the library's overdue fines. s32 i don't know what resources are available in the library.   category 3: library confidence s12 i enjoy learning new things about the library. s13 if i can't find a book on the shelf, the library staff will help me. s15 i feel comfortable using the library. s17 i feel comfortable in the library. s19 i can always ask a librarian if i don't know how to work a piece of equipment in the library. s20 the library is a comfortable place to study. s25 the library is an important part of my college. s26 i want to learn to do my own research.   category 4: library constraints s14 there is often no one available in the library to help me. s16 i feel like i'm bothering the reference librarian if i ask a question. s21 the library never has the materials that i need. s22 i can never find things in the library. s30 the library's rules are too restrictive. s31 the directions for using the computers are not clear. s34 the library won't let me check out as many items as i need.     appendix b   las scores   table 1 anwar et al (2004) proposed categories for las scores no anxiety 1.00 – 2.21* low anxiety 2.22 – 2.65 mild anxiety 2.66 – 3.54 moderate anxiety 3.55 – 3.98 severe anxiety 3.99 – 5.00   *the “no anxiety” level was listed in the article as “0.00 – 2.21” but the lowest achievable score on the las is actually 1.00 (lawless, 2011).     table 2 tests of normalityg   kolmogorov-smirnov shapiro-wilk statistic df sig. statistic df sig. library anxiety 0.074 119 0.159 0.986 119 0.238 staff approachability 0.075 119 0.095 0.976 119 0.037 feelings of inadequacy 0.073 119 0.180 0.982 119 0.124 library confidence 0.080 119 0.057 0.977 119 0.048 library constraints 0.083 119 0.041 0.979 119 0.059     table 3 mean score of library anxiety variable category 1   staff approachability category 2   feelings of inadequacy category3   library confidence category 4   library constraints library anxiety n mean mean mean mean mean university dibrugarh university 39 3.97 2.98 2.14 3.45 3.21 kaziranga university 38 3.89 3.20 2.19 3.40 3.23 tezpur university 42 4.34 3.68 1.99 3.77 3.50 over all mean   4.07 3.29 2.11 3.54 3.31 gender male 60 4.08 3.45 2.10 3.57 3.35 female 59 4.07 3.14 2.11 3.53 3.29 over all mean   4.08 3.30 2.11 3.55 3.32 community general 40 4.09 3.31 2.08 3.66 3.35 obc 42 4.05 3.48 2.13 3.50 3.35 sc 14 3.99 2.77 2.00 3.46 3.15 st 23 4.14 3.26 2.16 3.48 3.34 over all mean   4.07 3.21 2.09 3.53 3.30 language english 56 3.94 3.26 2.18 3.47 3.27 vernacular 25 4.15 3.39 2.09 3.69 3.39 both 38 4.23 3.29 2.00 3.56 3.35 overall mean   4.11 3.31 2.09 3.57 3.34                  research article   the information searching behaviour of music directors   martin chandler geospatial/gis services librarian brock university library st. catharines, ontario, canada email: mchandler@brocku.ca   received: 29 oct. 2018                                                                   accepted: 22 apr. 2019      2019 chandler. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29515     abstract   objective – this research project sought to elucidate some of the information searching behaviours of directors/conductors of performing music ensembles when selecting repertoire for performance. of particular focus was the kind of information needed to select repertoire and where that information was sought and acquired.   methods – semi-structured, guided interviews were undertaken with three conductors from varying musical ensemble forms (choral, orchestral, and wind). this included a graphical elicitation exercise following sonnenwald’s concept of information horizon maps. a narrative analysis was done, and recurring themes were sought in the various responses to questions and created drawings.   results – the results indicated that directors make significant use of historical and print resources in creating personal lists of repertoire for current or future use. professional connections for discussion of new or less well-known repertoire were also very important. one particularly interesting outcome was the non-temporally bound nature of conductors’ information searching behaviour, as the current models of information behaviour primarily relate to temporally bound searches. the internet was noted by the three conductors not as an information source in and of itself but rather as an extension of other information sources.   conclusions – this research highlighted the atemporal nature of information searching behaviour in music directors and suggested a similar aspect in the broader information search process. it indicated a need for libraries that cater to performers to maintain historical lists of varying types (e.g., concert programs, similar lists created by other prominent members of the community, and other types of repertoire lists). additionally, maintaining community connections and knowledge of new or newly available repertoire is important.     introduction   despite the recent growth of research in information behaviour, an under-researched area is the information behaviour of musicians. though music collections themselves require specialized knowledge, their use remains a significant yet uncertain consideration for many academic librarians, and the information needs of musicians bear study. with this in mind, conductors of ensembles have information behaviours specific to their field, notably in the search for new and relevant repertoire as well as the means of presenting the music and its context. this paper seeks to lay the groundwork for future research into the information behaviour of music directors by exploring the search process for repertoire selection.   the target population for this study is that of conductors (also known as directors or music directors) of mediumto large-sized musical ensembles. conductors of mediumto large-sized ensembles, as a group, can be further divided into orchestral, wind ensemble, and choral conductors. this population, as distinct from musicians more broadly and given their status as leaders, offers an interesting aspect of information behaviour study. leaders of small ensembles were not included, as often these ensembles tend to be more collaborative in repertoire selection.   while they are musicians, with the associated training therein, conductors are not, in fact, in the act of creating music themselves. instead, they select the creations of others (composers) and lead a group (the musicians or players) in the performance of that music. while this might lead one to view conductors as something of middlemen or middle-women, they are at the same time leaders and crafters of the interpretation of the music. a composer hands the work to the conductor, who directs the performers in how to interpret the textual information into acoustic performance.   there are, then, multiple aspects to the information world of conductors to consider. what information goes in to selection, and what is the process of information seeking and gathering? what resources are consulted in the interpretation of the music, especially for composers no longer living?   literature review   wilson (2000) noted the various iterations that information behaviour or information seeking behaviour can take, including research into information system use and user needs. these different iterations, wilson argues, have coalesced into various models, which in turn are being examined for cross-relation. each of these models seeks to understand a generalized user; however, as kim’s 2017 report on trends in information behaviour research found, information behaviour researchers have more recently worked to understand the specialized needs and behaviours of specific groups, as distinguished from the general. this paper examines the needs of musicians and in particular the information needs in repertoire selection for music directors of large ensembles.   both bates (1996) and brown (2002) noted the paucity of information behaviour research related to the arts and to music specifically. in the years since these publications, there have been attempts to fill this void, often by focusing on specific groups within the academic music realm. dougan (2012) offered a study on how and what tools music students use to search and access scores and recordings, noting differences between fields in music (e.g., performers use different information sources and have different information needs than musicologists) and observing early undergraduates seeking more readily available information (e.g., course reserves) than upper-year undergraduates and graduate students who seek more original work. dougan (2015) later studied the information source use of music students when searching for scores and recordings, specifically whether students were more likely to use library sources or rely on google and youtube, finding that while both were useful both also caused notable frustrations in findability of music-specific materials. more robust metadata and cataloguing practices were a clear need for these sources.   hunter (2007) wrote on the information needs, sources, and barriers for the information seeking of electroacoustic composers, with special note of the needs of the local community of composers and the library access that can be provided to them. liew and ng (2006) focused on academic ethnomusicologists’ search behaviours, exploring where and what kind of information academic ethnomusicologists search and how well academic libraries are meeting this need. popular magazines within the industry often offer sources for new repertoire (for an example, see tom moore’s 2007 article “locating music scores online”). while these articles offer useful sources to which the information professional can direct musicians, they are not studies of information behaviour. indeed, aside from studies of a few specific groups, including a study of amateur musicians in a concert band (kostagiolas, lavranos, korfiatis, papadatos, & papavlasopoulos, 2015), the majority of the available literature has a broader scope, relating to music-seeking in the broader population. lavranos, kostagiolas, martzoukou, and papadatos (2015) noted in their study on information seeking in musical creativity, “the impact of information and information seeking preferences on musicians’ everyday practices and more specifically on creativity is rather understudied” (p. 1071).   hunter’s study (2007) notably found that the information need among composers was “related to aesthetic issues such as needing inspiration” (p. 7), and when faced with such a need, composers would turn to professional connections. if local colleagues proved unhelpful, hunter noted that participants would turn to the internet—specifically, listservs. interestingly, “message boards, wikis, and blogs were not mentioned by any of the subjects as being helpful” (p. 8). professional conferences were similarly important to these composers, primarily for keeping abreast of trends in the field. hunter further noted gaps and problems that frustrated electroacoustic composers, particularly regarding questions surrounding the aesthetic qualities of a piece of music (p. 8) —something directly relevant to conductors. this was further supported by lavranos et al. (2015) who wrote on the implications of information on musical creativity. the authors focused primarily on modeling the intersection of creative development and information searching (p. 1071), finding that information plays a catalytic role in creativity (p. 1088).   liew and ng (2006) noted the part played by public and academic libraries in the search habits of ethnomusicologists and the difficulties created by classification systems. in particular, ethnomusicologists “found it easier to find relevant items in public libraries where cds, for instance, were catalogued according to categories such as world music, traditional music, dance, and pop, compared with cds in academic libraries that were classified according to the year of accession” (p. 62). further, the authors pointed to several problems encountered by ethnomusicologists, including a lack of relevant materials in academic libraries (something not directly noted by hunter [2007], though implicit in the lack of its mention as a major source), classification systems commonly used for library material being inappropriate for music, linguistic barriers, copyright issues, and cost and availability of materials (p. 66). dougan (2012) also discussed the difficulties of library classification systems and their relation to finding music, particularly the difficulties present in foreign language searching as well as limited holdings of available recordings (p. 565). because of these difficulties, tools such as google and youtube were preferred as a reference tool (dougan, 2015).   kirstin dougan's (2012) paper titled the “information seeking behaviours of music students” notes that for recordings students tend to look for a specific performer, ensemble, or conductor. here one can draw a parallel to conductors who may seek specific interpretations to inform their own. students also look for recording length and year of recording. often, students seek recordings in the library or through youtube or itunes (p. 561). students do not search for a specific edition of a score (p. 562). this observation is furthered by dougan’s 2015 study on score and recording seeking of music students, which found that students did not want a complete inventory of library holdings but rather sought the best example to meet their needs at the moment (p. 67).   given the previously noted paucity of research on information behaviour of musicians and given the lack of research into information needs related to repertoire selection, this paper seeks to fill the gap by studying those individuals with a clear need for information when selecting repertoire: directors of large ensembles.   aims   the primary research question of this study is: what are the information behaviours of conductors or music directors in regard to repertoire selection? this question evokes sub-questions that include the following: what kind of information do conductors need when planning repertoire? where and how do they search for this information? theoretical framework   the research was undertaken in consideration of hektor’s (2001) information behaviour model. each information need is understood as a project, hektor argues, and that project involves the seeking of information, where the individual searches for material to resolve their need, using eight distinct activities to seek, gather, communicate, and give information:   searching and retrieving: using direct effort to gather information by reading, listening, or otherwise ingesting information browsing: encountering information by moving within a specific environment, such as wandering the bookshelves or scrolling through television channels or social media monitoring: checking in with regular information sources, such as the news, email, etc. unfolding: letting others tell something and the overall communication involved, such as reading a text, watching a movie, etc. in an intentional, concerted manner to find the material relevant to the information need exchanging: the bi-directional communication of ideas between two or more participants, including face-to-face, telephone, email conversations, or even players in a game dressing: the externalized communication of information by an individual instructing: the dissemination of the dressing to others publishing: the posting of information, be it a paper in a journal, an advertisement on television, or a comment on a blog (pp. 81-88)   hektor presents the above activities as distinct from each other, though interconnected and often overlapping in the overall information behaviour. for this study, the author assumed that music directors’ search for information regarding new repertoire was similar to that for other information seekers (i.e., a time-restricted event) with informants engaging in the seeking, gathering, communicating, and giving behaviours as well as the accompanying eight activities of searching and retrieving, browsing, monitoring, unfolding, exchanging, dressing, instructing, and publishing. this assumption proved faulty, and though the informants did engage in many of hektor's behaviours, they were split between time-restricted and atemporal activities.   methods   the data-gathering method was used and approved by the university of toronto’s research ethics board as part of a class on information behaviour. it was based on sonnenwald, wildemuth, and harmon’s (2001) information horizon interview technique, which includes an interview with particular questions, followed by a drawing exercise involving the information behaviours discussed in the interview. sonnenwald’s information horizon drawing was chosen for its potential appeal to arts-based informants, and the author assumed that, due to the nature of their work, musicians would be more comfortable with the creative freedom offered by a blank page. in addition, sonnenwald and wildemuth (2001) note that graphical elicitation can lead to richer datasets and informant synthesis of their own information seeking behavior (pp. 13-15). this included an interview with the following set of questions:   1)       why did you become a conductor? 2)       are you usually the primary decision maker for repertoire for your ensemble(s)? 3)       how do you select repertoire for performance? 4)       what sources do you consult for selecting repertoire? 5)       do you search other information sources? 6)       has there been a time where you have had any particular difficulty in finding a piece? could you tell me about that? 7)       is there anything i have not asked you about that you would like to add?   between questions 6 and 7, informants were asked to take part in the information horizon map drawing. they were not given specific, explicit instructions; instead, they were simply asked to draw their information sources, or the information horizon, as they saw it. this request was adapted to the informant as seemed fitting. other questions were added as needed, especially with regard to specific individuals or sources mentioned in order to draw more information from the respondents. the interviews lasted approximately 30-45 minutes, including time for drawing, and were conducted in toronto, ontario, canada. two of the interviews (dave, erin) took place in a public café, and one interview (janet) took place in the informant’s office.   a mix of convenience and snowball sampling was used to determine informants. this author, having previously been a member of the target group, has a number of potential contacts within the community and relied on those contacts to provide other informants. music communities within geographical regions tend to be modest in size, and the community of leaders of ensembles, as a sub-group of this, is further constrained. as such, the number of available informants was small. the informants ranged in age from early 30s to late 50s, and they are all from white, european-derived backgrounds. two of the three informants were previously known to the interviewer: one on a personal level and the other through professional interactions. informants were selected to encompass a breadth of ensemble form, musical styles, and career development.   due to a lack of funding, local informants were sought. interviews were recorded and transcribed, and the recordings were destroyed within seven days to preserve anonymity. pseudonyms have been used in this paper to protect that anonymity. notes were also taken during the interviews to help guide the interview process. the interviews then underwent a thematic narrative analysis as described by parcell and baker (2017). this method was chosen due to the preliminary nature of the research on this group. as noted by labov and waletzky (1965), the “fundamental structures are to be found in oral versions of personal experiences” (p. 12). key motifs were extracted from the interviews, and notable differences sought (see table 1). as the number of informants was limited, only those motifs that the informants emphasized were identified as significant factors.     table 1 sources referenced in interviews and common motifs dave erin janet common motifs concert programs, anthologies, history books programs of other ensembles, old church bulletins lists, “standard repertoire” (“canon”) lists or historical programs fellow colleagues colleagues (“experts”) colleagues (conferences, telephone) colleagues internet (youtube & imslp) internet (cpdl) internet (composer or publisher websites, conference performances) internet maintains “wish list” “whatever fit that i thought was most beautiful” stock of pieces in mind “wish list” library (historical or primary sources, biographies) university libraries   library concert based on theme educational level         results   the three interviews are presented below as case studies to highlight the similarities in significant factors among this group of conductors of diverse age, career development, and ensemble style.   informant 1: dave   dave is a conductor primarily of “historically informed performance,” meaning the performance of music guided by writings related to a particular era and region and on instruments designed to mimic those instruments of the same. historically informed performance, then, requires a great deal of musicological research and preparation. this research also has an impact on the repertoire selection and the means by which dave undertakes his information search.   dave described his entrance into conducting as a long movement, beginning in his youth by organizing his classmates. he watched performances both live and televised and described being enraptured by “what that man up front was doing.” he describes conducting as an “urge” or a “need” and as something that one does not choose but that is “almost a requirement.” for several years, he both played in an orchestra and conducted a choir, until his duties in the latter became too demanding and he moved to conducting full time.   dave discussed repertoire selection as being very dependent on the ensemble. as his current primary ensemble is part of a larger organization, he is therefore beholden in part to the music director of the organization, and all decisions are made in conjunction with the director. he maintains a “wish list” of pieces he would like to perform that he can then fit with the rest of the organization’s programming, if possible. in some cases the ensemble will approach him with a program pre-built, with some small room for tweaks; in other cases, he is given freedom to develop a concert, either based on a theme or without condition. in either case, his conducting is requested for his specialized knowledge of historically informed performance and the repertoire from the periods the ensemble wishes to perform. dave related one instance of an orchestra looking to perform a concert celebrating the anniversary of a french outpost in canada. he then researched and prepared the entire program, including both classical pieces as well as pieces that would invoke the folk style more likely to be heard at the outpost from the designated time period.   due to the specialized nature of dave’s work, he must engage with more immediate information searches than the other two informants. primary among these are fellow colleagues; source materials such as original concert programs, anthologies, and music history books; and the internet, specifically mentioning youtube, the international music score library project (imslp, a library of scores online), and google. dave mentioned libraries but was reticent to relate to them as primary sources of his information. though he stated “i will be spending a lot more time in a library than the average conductor” and the “m2 section or m3 section in the library,” he later noted “i don’t use the resource of the orchestral librarian.” he also discussed contacting numerous libraries in england, france, and germany to acquire copies of material with no modern edition. indeed, though dave downplayed the use of librarians and seemed to try to avoid stating that libraries were sources (“library” does not appear on his drawing; see figure 1), he referred to them regularly as points with which much of his information seeking was engaged, either in searching for new repertoire or seeking unpublished pieces from which to create new editions.   dave’s information horizon drawing was primarily text based and took the form of a workflow in his information search. beginning with a catalyzing event (labeled “concept”, referring to a concert or musical event), the various sources consulted are then written out with independent arrows pointing to each and subsequent divisions of each information source (e.g., “source material” to “modern editions?” or “facsimile editions”). it is interesting to note that dave did not include any arrows connecting the sources to each other.   particularly interesting was his discussion of historic concert programs. dave offered the specific example of the concert spirituel, an 18th century organization that presented public concerts. the organization maintained a record book with every concert’s program, from the inception to the close of the organization.     figure 1 dave’s information horizon drawing.     informant 2: erin   erin is currently studying conducting and is heavily involved in the field. she discussed her choirs—a madrigal choir that she started and church choirs that she has been invited to direct—as being of varying capability.   erin was quite frank about her reasons for becoming a conductor: “i had too many opinions whenever someone else was in charge.” like the other two informants, she seemed to have come to it gradually, rather than training for it early on. erin qualified herself as a very quiet person, and her decision to take up conducting was to be in a position of power where she would not have to battle to be heard; instead, she would be expected to give her directions for others to follow.   erin’s repertoire selection has not been limited by constraints of style or era. she does, however, have greater constraints on the musical ability in her ensembles as well as budgeting and availability of scores. especially in regard to the church choir, the educational component was a major factor in her selection of repertoire. the pieces needed to be “attainable but challenging,” while also offering variety for the congregation and the singers. the third necessary consideration was “whatever fit that i thought was most beautiful.” her orchestra conducting was further constrained by the aforementioned score availability. due to the smaller budgets of the community orchestra-type ensembles she conducted in her home province, score selection was limited to those scores that were already owned or that could be borrowed from other orchestras at no cost.   erin noted the internet as a major source of information. she pointed to the choral public domain library (cpdl) as her first point of research. she also noted university libraries as being excellent sources of music for searching, particularly for orchestral pieces. for both choral and orchestral works, erin stated that she often searched online for other community ensembles and noted pieces that would work well together. from there, she returned to her discussion of choral work alone, noting that in her work as a church choir conductor, she would look through old church bulletins to see what had been programmed previously. finally, she said she relied on colleagues (referring to them as “experts”). her information horizon drawing (see figure 2) displayed these four sources in her information search; however, she labeled their order somewhat differently than described. while beginning with the internet, she then noted colleagues as the second source, followed by old programs, and finally, she indicated the library as the fourth search area.   because she is at the beginning of her career, erin did not have a wish list of pieces, though she recognized the need and desire to develop one. she noted that she would like to devote more time to exploring repertoire, but she has not yet been able to do so due to other constraints. it is also worth noting that her phrasing of “whatever fit that i thought was most beautiful” includes an implicit sense of repertoire knowledge to draw on when developing a program. she notes that she is in a fairly unique position of conducting regularly without any formal training but simultaneously feels “like a fraud when it comes to repertoire knowledge.”     figure 2 erin’s information horizon drawing.     informant 3: janet   janet is a well-respected wind conductor and conducting pedagogue at a canadian university. i had the opportunity to play in an ensemble under her direction many years ago, and we discussed this prior to the formal interview. janet was energetic and humble about her work, while greatly interested in this research.   when asked to describe how she became a conductor, janet related how she had been on track to be a music teacher and knew that she would need conducting classes. soon after beginning the classes, she stated that she “got the bug.” she noted that she knew she wanted to work with wind music, and when asked why winds in particular, she could not definitively answer. she stated that she knew the wind band world and so likely decided to stay where she was comfortable.   janet stated that she has primary say over her repertoire selection, and she said that this was a very important factor for most conductors, especially of wind ensembles. she, like erin, was keen on the educational component of her conducting; as a university conductor, the repertoire is the curriculum for the course and thus how the students learn. though she related a few stories when she did not have full and final say in repertoire selection, she was clear that it was one of the most important and enjoyable aspects of her work.   when discussing the selection process for repertoire, janet talked about standard repertoire, though later mentioned that the wind ensemble does not have a canon like the orchestral world does (with composers such as mozart, beethoven, and others). she also discussed choosing pieces for their style (e.g., march, jazz, lyrical, and others) in order to expose students to that style of playing as well as creating a varied concert program. to plan for her main ensemble, she starts with a sheet of paper and divides it into four sections for the four concerts they perform through the year. she puts various pieces onto each section that she knows already or that she has heard in concerts or heard about from colleagues. colleagues, she noted here, were a big source of information on new repertoire, both good and bad. she then balances the pieces she has already put on the paper with others to create “a program that has a shape to it, that has its own kind of dramatic arc.” she then noted that, working in the university, she is protected from concerns of selling a large number of tickets, unlike a performing orchestra.   janet shared her sources in the search for new repertoire, and she seemed to have a large stock of pieces in her mind already. she noted composers’ websites and music publishers’ websites as excellent sources, the latter often having links to the former. because of the educational nature of her work, janet pointed to the leveling system of music publishers as an important source of information, where levels of difficulty are noted by a letter and number system (e.g., “b300” is a middle-level piece suitable for early high school). conferences were then discussed, particularly the college band directors national association conference. she suggested that there are other conferences as well and that while she may not always attend, she does gather information through colleagues and the internet. often, janet told me, conferences will also post their concerts on youtube, and she will watch those if she did not attend.   she then pointed to colleagues as a major source of information, both for their research and for her sharing of her own. she noted a particular mutual acquaintance that she has a yearly phone call with to discuss repertoire. she said that these conversations also happen at the conferences, but she was specific about this yearly phone call. in her discussions with colleagues regarding repertoire, she noted delight in being able to present a new and exciting piece, while hearing about the same from her colleagues. a final source of her information was the lists created by fellow respected wind conductors and frank battisti in particular. she discussed his work as a conductor and the books he had written on the topic, which included lengthy repertoire lists.   janet took great delight in the consideration and execution of the information horizon drawing. she described her drawing as a series of tunnels (see figure 3), with each of them looping around and going in every direction. she later settled on the concept of a cloud, though her drawing does not look like a typical cloud. i then asked her where the various sources she had mentioned would fit in to the drawing because she had not labelled anything, and the only clear marker was herself as a stick figure. she added a few more bubbles, still referring to it as a cloud. she also discussed it as “non-linear.”     figure 3 janet’s information horizon drawing.     discussion   three strong themes emerged from the interviews and information horizon drawings. the first is the importance of personal and professional networks. each informant noted that they regularly interact with fellow conductors and musicians to expand their repertoire resources and knowledge. this is consistent with hunter’s (2007) findings of professional networks being vital for electroacoustic composers. further, the line between the personal and professional contact appeared to be blurred. the informants each related specific individuals that they interacted with in various contexts, and part of that interaction related to repertoire. a network of fellow musicians, then, is important when discovering new or unknown works. this is interesting given the geographically dispersed nature of music directors. because few cities are large enough to maintain multiple professional ensembles of similar make-up, networks of ensemble directors are therefore dispersed, with conferences being the primary place for in-person contact. telephone and email appeared to be the preferred methods of regular contact within networks among the informants.   the second theme elucidated by the research was tangentially related to the first: all of the informants mentioned lists, anthologies, and concert programs as sources for drawing potential repertoire suggestions. while not taking the form of a direct engagement with fellow musicians, each of the conductors did rely on the work of other musicians and mainly other conductors. they each referred to historical programs, church bulletins, and lists of repertoire developed by other conductors when engaged in their own concert programming. indeed, it was very interesting to note that none of the conductors mentioned perusing scores to consider adding to their repertoire, instead relying on the lists created by others. within this context, the internet was cited as a major source of information. though each of the informants referred to the internet as a distinct source, it was always discussed in the context of searching for lists and programs and reviewing publisher, composer, and library sites. it would behoove the music librarian, then, to research and provide access to such lists, providing divisions based on the types of ensembles, periods in which they worked, and geographic regions. it is also worth noting here that—as with dougan’s (2012) study describing students seeking particular performers, ensembles, and conductors—the conductors in this study had certain publishers that they liked or disliked. janet mentioned british publishing companies as having a propensity for more interesting music; dave, meanwhile, cited a publisher based in florida as one to avoid.   the third and most interesting aspect from an information behaviour standpoint is the bifurcated nature of the search process, with both immediate and atemporal needs. each of the conductors noted times when they needed pieces for a particular performance; however, the vast majority of their searching was done outside of specific deadlines. this is in contrast to hektor’s assumptions that information seeking is in response to “projects” (p. 70), where projects are specific wants or needs to be fulfilled either in an immediate or deferred time frame (pp. 70-76). while the search for new repertoire is done with an end goal of a specific concert in mind, new repertoire is also often added to a personal list of pieces to perform at some indeterminate point in the future. a conversation with a colleague may reveal piece x, but piece x may not make its way into performance until years later, if at all. these atemporal lists may be recorded or merely maintained in the conductor’s mind and thus prone to the difficulties of memory.   while this research is focused on the population of conductors, the atemporal nature of their information search may be relevant for other populations (e.g., hobbyists, readers). while much information searching is done for immediate needs, the internet especially has allowed an ease of information searching that may have further de-temporalized this behaviour among other populations, or it may have previously been present but unstudied. atemporal information behaviour is a potential new avenue of discussion.   one notable aspect not explicitly stated but that came out of janet’s discussion was the intersection of processes. while hektor's (2001) information behaviour model is very segmented, and hektor notes the somewhat blurred nature of the lines, for the wind conductor there did not appear to be any immediate distinction between the behaviours. this may suggest that for conductors as a whole, the barriers between information activities such as search and retrieve, browse, dress, etc., are more fluid, if present at all.   it is interesting to note that each of the informants referred to the internet not as an information source qua information source but rather as an extension of other information sources. where once an individual would contact other orchestras and repositories, some of this information has now been shared online. the internet has not replaced other information sources; colleagues and libraries are still cited and used. it has merely acted as an extension of the available resources.   limitations and methodological reflections   as this research was carried out as part of coursework at the university of toronto’s master of information program, the use of human participants was cleared by the research ethics board of that institution and a limit of three informants was allowed under this clearance. thus, the limited number of participants can be seen as a limitation. however, because informants in this study represented a breadth of music ensembles (i.e., choral, wind ensemble, and orchestral conducting), career stages (two are respected midto late-career directors and one is an early-career director), and movement between ensemble forms (two of the informants mentioned engaging in orchestral and choral conducting at different times), and because the interviews yielded rich data and common themes, this paper presents a valuable preliminary study of this group, with the potential for future delineation between the possible differing needs of the music directors of specific ensemble forms. as with hunter (2007), whose research was derived from a qualitative study of five composers, this exploratory study lays the groundwork for future research while drawing worthwhile themes in its own right. further study to confirm the findings of this research using a larger, more geographically diverse cohort would be beneficial.   one notable limitation of this study was a lack of diversity in the cultural background of the informants. all three informants come from white, european-derived backgrounds and received their primary training in north america. while the field of conducting itself is mostly limited to western musical styles, this is gradually changing. wind ensembles, especially, have shown significant growth in japan, and there are prominent conductors with backgrounds from locales around the world, including conductors such as dinuk wijeratne, seiji ozawa, and gustavo dudamel.   it is important to note that two of the three informants were known to the researcher prior to undertaking the research. given the limited and interconnected nature of the music industry, this would be difficult to avoid for the music-centred information professional. engaging in joint research with other individuals would offer a means around this problem for future research.                 the information horizon map offered an interesting and thoughtful addition to the research. it allowed the informants to consider their responses and adjust or rephrase the chronology of their search as well as the importance of the information source. each individual approached the request for a drawing somewhat differently but presented many similar sources that had previously been drawn out by discussion.   the information horizon maps were, however, somewhat limited by being used near the end of the discussion. the informants had, by this point, already cemented ideas of their information horizon in their minds, and no new sources were revealed in the mapping process. it might therefore be beneficial to engage in the mapping at the beginning of the interview, with an ongoing mapping as the interview proceeds, though this may present its own problems of discussion and distraction in the overall process. conclusion   conductors and music directors, when selecting repertoire, inhabit a niche of information seekers bound not simply by specific, temporal demands but also maintaining long-term, atemporal needs. as such, the information professional catering to this group is well served by extending the standard conceptions of service delivery to include these atemporal considerations and by incorporating new fields of information from historic documents. a music librarian, then, is well served by the interactions between libraries and archives, drawing as they must from the archival programs and lists developed by past conductors of repute. similarly, the atemporal considerations of research are not often considered in information searching behaviour studies and may offer a new line of questioning. informal communication among professionals is equally valued, and directors regularly engage in conversations on current repertoire at conferences, by phone, and via the internet. it is also worth noting that the current assumptions about the internet as an information source should be reconsidered for music directors. for them, it operates as a tool supporting other information sources.   references   parcell, e. s., & baker, b. m. a. (2017). narrative analysis. in m. allen (ed.), the sage encyclopedia of communication research methods vols. 1-4 (pp. 1069-1072). thousand oaks, ca: sage publications, inc.   bates, m.j. (1996). learning about the information seeking of interdisciplinary scholars and students. library trends, 45(2), 155-164.   brown, c. d. (2002). straddling the humanities and social sciences: the research process of music scholars. library and information science research, 24(1), 73-94. https://doi.org/10.1016/s0740-8188(01)00105-0   dougan, k. (2012). information seeking behaviors of music students. reference services review, 40(4), 558-573. https://doi.org/10.1108/00907321211277369   dougan, k. (2015). finding the right notes: an observational study of score and recording reeking behaviors of music students. the journal of academic librarianship, 41, 61-67. https://doi.org/10.1016/j.acalib.2014.09.013   hektor, a. (2001). what’s the use? internet and information behaviour in everyday life. linkoping, sweden: linkoping university.   hunter, b. (2007). a new breed of musicians: the information-seeking needs and behaviors of composers of electroacoustic music. music reference services quarterly, 10(1), 1-15. https://doi.org/10.1300/j116v10n01_01   huvila, i. (2009). analytical information horizon maps. library & information science research, 31(1), 18-28. https://doi.org/10.1016/j.lisr.2008.06.005   kim, e. (2017). the trends in information behavior research, 2000-2016: the emergence of new topical areas. journal of the korean biblia society for library and information science, 28(2), 119-135.   kostagiolas, p. a., lavranos, c., korfiatis, n., papadatos, j., & papavlasopoulos, s. (2015). music, musicians and information seeking behaviour: a case study on a community concert band. journal of documentation, 71(1), 24-3. https://doi.org/10.1108/jd-07-2013-0083   labov, w., & waletzky, j. (1967). narrative analysis: oral versions of personal experience. in j. helm (ed.), essays on the verbal and visual arts (pp. 12-44). seattle, wa: american ethnological society/university of washington press.   lavranos, c., kostagiolas, p. a., martzoukou, k., & papadatos, j. (2015). music information seeking behaviour as motivator for musical creativity: conceptual analysis and literature review. journal of documentation, 71(5), 1070-1093. https://doi.org/10.1108/jd-10-2014-0139   liew, c. l., & ng, s. n. (2006). beyond the notes: a qualitative study of the information-seeking behavior of ethnomusicologists. the journal of academic librarianship, 32(1), 60-68. https://doi.org/10.1016/j.acalib.2005.10.003   moore, t. (2007). locating music scores online. flute talk, 26(8), 22-31. retrieved from https://mholtttwoodwinds.weebly.com/flute.html    savolainen, r., & kari, j. (2004). placing the internet in information source horizons. a study of information seeking by internet users in the context of self-development. library & information science research, 26(4), 415-433. https://doi.org/10.1016/j.lisr.2004.04.004   ssonnenwald, d.h., & wildemuth, b.m. (2001). investigating information seeking behavior using the concept of information horizons. retrieved from https://jasper.ils.unc.edu/sites/default/files/general/research/tr-2001-01.pdf   sonnenwald, d. h., wildemuth, b. m., & harmon, g. l. (2001). a research method using the concept of information horizons: an example from a study of lower socio-economic students’ information seeking behaviour. the new review of information behaviour research, 2, 65-86.   wilson, t. d. (2000). human information behavior. informing science: the international journal of an emerging transdiscipline, 3(2), 49-55. https://doi.org/10.28945/576         evidence based library and information practice evidence based library and information practice 2010, 5.1  evidence based library and information practice       editorial    taking stock(holm): time to up our game?    andrew booth  reader in evidence based information practice  school of health and related research (scharr), university of sheffield   sheffield, united kingdom  email: a.booth@sheffield.ac.uk     received: 13 dec. 2009          accepted: 21 dec. 2009       2010 booth. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which  permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly  attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the  same or similar license to this one.    the international evidence based library  and information practice conference safari  continues     first the elephant, sheffield (2001), a  lumbering ungainly creature which, with  characteristic short sightedness, sought only to  target the u.k. health library audience. next  on the scene, the fast‐moving leopard,  edmonton (2003), renowned for its ability to  inhabit diverse habitats, exemplified by  participants from several academic sectors.  third, the brisbane lion (2005), letting out a  mighty roar as, enlisting the support of the  australian library and information  association (alia), it issued a rallying call to  the profession at large. simultaneously, the  eblip movement gained its own powerful  voice with launch of the eblip journal. fourth,  the relentless charge of the mighty rhinoceros  in north carolina (2007), with the muscle of  major hide‐bound u.s. professional  associations lending not inconsiderable  impetus to uptake of eblip. finally,  stockholm (2009), the buffalo, with its keen  eyesight, farsighted in planning and wide‐ ranging in identifying issues of concern to the  profession.    a decade ago, in a famous seasonal offering  from the british medical journal (bmj), a  correspondent likened the big five medical  journals (bmj, lancet, annals of internal  medicine, new england journal of medicine and  journal of the american medical association  (jama)) to the “big five” african game  animals (loefler, 1998). with the eblip  international conference series accumulating  its own “big five”, completed by the welcome  addition of stockholm in july 2009, it is fitting  to review the safari so far travelled. hopefully  our extended metaphor is not as contrived as  other big fives suggested by bmj  correspondents – you were spared a spice  girls comparison (abbasi, 1998) – it does  however provide a backdrop against which to  take stock of the eblip movement and its  challenges. as one of four participants to have  attended all five conferences (for the record  the others are alison brettle, anne brice and  jonathan eldredge), i am privileged to outline  4 mailto:a.booth@sheffield.ac.uk evidence based library and information practice 2010, 5.1  some challenges posed by the most recent  contribution to the conference series.  in his interactive plenary session, “bridging  the gaps: linking our eblip questions to our  decisions”, a first for the eblip conference  series, dr. jonathan eldredge encouraged us to  identify questions from our practice.  acknowledging that formulating the question  is one thing, while sustaining momentum to  answer it is another, jonathan shared the  concept of virtual peer mentoring and  encouraged us to pursue such peer support  beyond the spatial and temporal constraints of  the conference timeframe. such interactivity is  difficult to capture via the medium of prose  but jonathan himself attempts to distil some of  his thinking though his commentary in this  issue of eblip.   appropriately, this spirit of collegiality and  community was strengthened by “bridging  the gap between users and systems – the  potential contribution of social informatics to  evidence based library and information  practice”, delivered by dr. anita  mirijamdotter. it was stimulating to reflect on  the complexity of library and information  problems – after all, the real world is much  more messy than the rational logic of evidence  based practice might imply! i have resolved to  revisit the soft systems methodology  approaches i first encountered when studying  for my master’s to see how these might relate  to the initial stage of formulating a question. i  therefore commend anita’s commentary, also  in this issue, for your attention.  a special award of “chocolates on the pillow”  goes to professor sue mcknight for an  insightful presentation, “bridging the gap  between service provision and customer  expectations”. sue’s work has figured  prominently in previous issues of eblip and  she certainly met, and indeed exceeded, the  expectations of this particular customer. we  thought that it might be interesting to offer a  joint commentary making explicit links  between sue’s cutting edge work and current  thinking on evidence based library and  information practice.   a well respected figure from scandinavia  academia, birgitta olander, asked us to  consider the implications of the need for skills  in handling, interpreting and implementing  research in her plenary session, “bridging the  skills gap – shaping the information  professional of the future”. it was invaluable  to be able to locate the small but expanding  ripples of the eblip movement within the  wider, and sometimes engulfing, currents  within which planning for the future  workforce necessarily takes place. do make a  point of visiting birgitta’s commentary  elsewhere in this issue.  armed with a store of pebbles with which to  rattle a game animal’s cage came ola pilerot  with his provocative offering, “conditions for  research use in library and information  practice – a matter of learning”. ola made  good use of a research use versus evidence  based practice dialectic around which to  construct his argument. it was a presentation  that demanded a response from the eblip  community and i have been happy to attempt  to respond to his challenge with my own  commentary later in this issue.  finally, in the conference showcase feature, a  slot that acknowledges the highest scoring  submitted abstract, prudence dalrymple, an  eblip veteran, drew parallels with evidence  based healthcare in her plenary, “applying  evidence to practice: gaps, barriers and  lessons learned from healthcare”. even those  with a significant pedigree within this  particular sphere of evidence based practice  had much to learn and appreciate from her  fascinating chronicle, briefly summarised  within her own commentary.    of course, in addition to the plenary sessions  there was a plethora of stimulating, innovative  and creative presentations in the parallel  sessions. i have no doubt that you will  welcome the inclusion of these in future issues  of eblip, assuming that their authors respond  to jonathan eldredge’s challenge by  identifying a virtual peer mentor to help them  see their projects through to publication!    5 evidence based library and information practice 2010, 5.1  6 so much for the “bones” of the eblip5  stockholm conference. what of its spirit? as  briefly attested elsewhere, several  serendipitous themes have emerged to shape  the possible future configuration of eblip –  eblip five‐point‐zero, if you like. these  include the reconceptualization of evidence  based practice as a team‐based endeavour  requiring a different set of 5as from the  established orthodoxy (i.e., articulate,  accumulate, assess, agree and adapt)  (booth, 2009). furthermore, there is a strong  sense of the iterative nature of a process that  has been previously presented as a sequential  and logical succession of separate steps,  reflecting our social science stimulated  reflection on the complexity of the real world.  finally there seems increasing awareness of  the need to situate the evidence based practice  movement within the wider issues facing our  profession, both for reasons of sustainability  and to stimulate exciting possibilities of  synergy and cross‐fertilisation. at the very  least, eblip5 has been a stepping stone, if not  an actual bridge, to a new and widening  perspective on the paradigm.    trying to characterise a conference by its  academic programme alone is akin to  mounting trophy animal heads upon a wall. a  more appropriate analogy is the photo‐safari,  and certainly the flickr conference  photostream provides ample evidence of the  conviviality and energy of the event as a  whole  <http://www.flickr.com/photos/eblip5/>.  however, even this vivid pictorial record does  scant testimony to the prodigious efforts of  lotta haglund and her local team in  delivering the entire conference, workshops  and social events included, with consummate  professionalism. as co‐chair of the  international programme committee i would  once again formally like to thank lotta and  her apparently tireless colleagues.    roll on 2011 and… the manchester meerkat!  while addition of meerkats to an inventory of  big game animals serves to stretch impossibly  the bounds of credulity, it does perhaps reflect  that a different paradigm for conference  delivery beckons. meerkats are extremely  social animals with avid curiosities. to  survive, meerkats must live in groups for  protection, as the desert presents many  challenges. each meerkat has an important,  role to perform. i sincerely hope, and indeed  believe, that a new era of eblip as a collective,  social phenomenon awaits on the horizon!      references    booth, a. (2009). eblip five‐point‐zero:  towards a collaborative model of  evidence‐based practice. health  information and libraries journal, 29(4),  341‐344.    loefler, i. (1998). correspondence. shall i  compare thee to a… big game animal?  british medical journal, 337. retrieved  20  feb. 2010 from  http://www.ncbi.nlm.nih.gov/pmc/articl es/pmc1114520/pdf/1742.pdf    abbasi, k. (1998). correspondence. shall i  compare thee to a… spice?” british  medical journal 337. retrieved 20 feb.  2010 from  http://www.ncbi.nlm.nih.gov/pmc/articl es/pmc1114520/pdf/1742.pdf  http://www.flickr.com/photos/eblip5/ http://www.ncbi.nlm.nih.gov/pmc/articles/pmc1114520/pdf/1742.pdf http://www.ncbi.nlm.nih.gov/pmc/articles/pmc1114520/pdf/1742.pdf http://www.ncbi.nlm.nih.gov/pmc/articles/pmc1114520/pdf/1742.pdf http://www.ncbi.nlm.nih.gov/pmc/articles/pmc1114520/pdf/1742.pdf eblip: editorial responsiblities editorial responsibilities      2021. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     editor-in-chief: ann medaille   associate editor (evidence summaries): stephanie krueger, heather macdonald, christina wissinger   associate editor (research articles): erin owens, nikki tummon, lisl zach   associate editor (classics & reviews): jane schmidt   associate editor (commentaries, using evidence in practice): ann medaille   communications officer (news): kim mackenzie   production editor: rachel hinrichs   editorial intern: melissa cober   indexing coordinator: melissa cober   editorial advisors: alison brettle, michelle dunaway, lorie kloda, denise lafitte, heather pretty   copyeditors: heather healy (lead copyeditor), ève bourbeau-allard, kirstin duffin, julie evener, emily kingsland, christine moffatt, jane monson, jane morgan-daniel, elizabeth stregger, maria tan, elaina vitale, ross wilson   editorial assistant: samantha sheplawy   evidence summary   citation analysis of engineering graduate student theses indicates students are using more electronic resources   a review of: becker, d. a., & chiware, e. r. t. (2015). citation analysis of masters' theses and doctoral dissertations: balancing library collections with students' research information needs. journal of academic librarianship, 41(5), 613-620. http://dx.doi.org/10.1016/j.acalib.2015.06.022   reviewed by: heather macdonald health and biosciences librarian macodrum library carleton university ottawa, ontario, canada email: heather.macdonald@carleton.ca   received: 13 july 2016   accepted: 14 oct. 2016      2016 macdonald. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine the citation pattern of graduate students’ theses and dissertations.   design – citation analysis.   setting – an institutional repository at a south african university of technology.   subjects – 201 engineering master’s theses and doctoral dissertations.   methods – a random sample of master’s theses and doctoral dissertations from the faculty of engineering were analyzed. the theses and dissertations were drawn from the institutional repository covering the period 2005-2014. references were checked for format of the cited items including journal, book, conference proceeding, online item (resource with a url other than a journal, book or proceeding), and other (anything not in the first four categories). the date of all journal articles was recorded. journal titles were analyzed in terms of country of origin, language, availability in the library, and online access. data were categorized by department to determine if there were any differences in the use of materials by department. data were also analyzed by degree level.   main results – 101 theses and dissertations were analyzed out of a total of 201 available in the institutional repository. journals were the most used resource (42%), followed by books (30%), other (12%), online (10%), and proceedings (6%). doctoral students used a higher percentage of journals than master’s students. departmental usage differed. mechanical (54%) and chemical (48%) engineering students mainly used journals. civil engineering students mostly used resources from the “other” category (31%). students in industrial (41%) and construction (40%) engineering mostly cited books. analysis of the “other” category showed a wide variety of resources used (emails, personal interviews, course notes, conference papers, government publications, national and international standards, manuals and guides, technical reports, and technical notes).   the technology university provides access to 79% of the journal titles used by engineering students in their theses and dissertations. 84% of titles are available online. students mainly used current articles (i.e., from 2000-present). students heavily favoured journals from the united states of america and europe, although south african journals were the fifth most cited by country. english language titles dominated, however portuguese and french titles were the next most commonly cited. seventy-four titles were referenced more than 10 times.   conclusion – the authors state that more electronic resources are being used by graduate students, including “online” information. journals are the most cited information resource held by the library and the majority of journal titles that were cited can be found in the library. the authors conclude that librarians should work with graduate students to encourage the continued use of library resources. they also state that this information can be useful for identifying journals that could be canceled in times of budgetary cutbacks. the authors note that this study provides the university libraries with insight into the use of library holdings, but being limited to engineering, a more comprehensive study of subjects would provide a broader picture of the collection’s use and provide valuable information for collection development.   commentary   this article follows in the footsteps of several others investigating the referencing behaviour of engineering researchers including graduate students. perryman’s critical appraisal tool for bibliometric studies (2009) was used as a guide for this appraisal. the literature review covers the current relevant literature. the literature does not suggest a gap in the research record, however according to the authors, in the south african context engineering is an important growth area. consequently the university has an interest in understanding how the library supports graduate students in this field.   the data collection method was clearly explained and included inclusion criteria. the authors noted the main limitation of the collection method: not all theses and dissertations from the faculty of engineering for the specified time period (285) were in the institutional repository, as they had not all been digitized and hence were not accessible to be used in the sample. the authors do include a table of all of the theses and dissertations broken down by department and degree level for the time period as well as a table of the sample population analyzed. a comparison of these tables shows that the sample contains a higher percentage of older theses and dissertations (2005-2009), with the exception of the year 2014. also, the sample contains a higher percentage of the total number of master’s theses compared to the total number of doctoral dissertations. this has implications for the representativeness of the sample despite the large sample size. the authors did not discuss the rationale, nor did they include a statistical calculation of their sample size.   the authors did include appropriate and sufficient examples of the data analysis to answer all of the stated objectives. the analysis is well organized and clearly described. there was one discrepancy in the stated sample size. the text stated that 101 theses and dissertations were analyzed, however table 2 showed a total of only 98. this discrepancy is not addressed in the article. in the subsequent table 4, the total number is again referred to as 101.   this study achieves its stated objectives and builds on the prior research in this area. the real value of this analysis is for the library itself. this research helps provide a much clearer picture of the use of the collection, the collection’s strengths and weaknesses, and can help inform decision making about the collection. this paper demonstrates the utility of citation analysis as a method for collection assessment and development and provides a clear example of how to conduct citation analysis. the findings will help others doing similar investigations of their own collections.   reference   perryman, c. (2009). critical appraisal tool for bibliometric studies. retrieved from https://www.dropbox.com/l/scl/aaal7luzple90fxfnbv5hcnoz0ctlh6rqrs     microsoft word news_fahey.doc evidence based library and information practice 2007, 2:3  130 evidence based library and information practice     news    new conveners for eblig      sue fahey  public services librarian  health sciences library  memorial university of newfoundland  newfoundland, canada  e‐mail: sfahey@mun.ca    renée de gannes‐marshall  information specialist/ spécialiste de lʹinformation   canadian dental association/ lʹassociation dentaire canadienne  ottawa, canada   e‐mail: rdmarshall@cda‐adc.ca         © 2007 fahey and de gannes‐marshall. this is an open access article distributed under the terms of the  creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits  unrestricted use, distribution, and reproduction in any medium, provided the original work is properly  cited.    as new co‐conveners for cla’s interest  group, eblig, we would like to take the  opportunity to introduce ourselves. sue  fahey is a public services librarian at the  health sciences library, memorial  university of newfoundland, st. johnʹs.  renée de gannes‐marshall is the  information specialist at the canadian  dental association in ottawa.      ebligʹs goals are:    • to represent the interests of  librarians involved in evidence  based librarianship and library  related research;   • to organize continuing education  opportunities for librarians in this  area;   • to provide a means of  communication between librarians  involved in evidence based  librarianship;   • to assist librarians with the  dissemination of library research  utilizing an evidence‐based model;   • to support librariansʹ evidence  based research with grants.     mailto:sfahey@mun.ca mailto:rdmarshall@cda%e2%80%90adc.ca http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:3  131 we are pleased to announce the  introduction of a new workshop grant  which has been established to support and  encourage the development of continuing  education activities in evidence based  librarianship in conjunction with  local/regional/provincial associations or  institutions. eblig co‐conveners and  volunteers will evaluate applications and  determine awards based on the established  criteria, including relevance to the  profession and to the advancement of  evidence based librarianship, value to  expected participants and  regional/geographic location. we would like  to award the first grant this year and have  allotted a budget of $250 and set a deadline  of october 15th, 2007. please apply asap  for funding for your fall course idea! for  more information, go to  http://eblibrarianship.pbwiki.com/eblig+w orkshop+incentive+grants.     an hour long education institute audio  conference, evidence‐based librarianship:  a toolkit for public libraries, featuring  presenters virginia wilson and stephanie  hall, will be held on tuesday, november  6th, at 12 pm. virginia and stephanie will  present a toolkit tailored to public librarians  and public library workers with tips on how  to integrate ebl into the workflow. the  toolkit will include key ebl articles, sources  of evidence, and steps to take to put ebl  into practice.  registration is $54 for  members and $74 for non‐members. for  more information, go to  http://www.thepartnership.ca/partnership/b ins/calendar_page.asp.      if you managed to make it to eblip4 or you  missed out, but were wondering how you  could get to the next conference, (especially  after seeing conference abstracts posted on  the website http://www.eblip4.unc.edu/), we  have been in touch with the main contact for  eblip5 to be held in stockholm, sweden.  details on the conference, including event  dates and venue, are forthcoming.     as of july 31st, 2007, our interest group is 48  members strong from across canada and the  united states! for those who participate on  the listserv and who live outside of canada,  did you know that you can join eblig  without having to become a full member of  cla for a fee of only $30 cad annually?  for more details, go to  http://www.cla.ca/about/igroups/evidence_b ased.htm.  also of note is the fact that non‐ members can participate on the listserv.     we look forward to your comments and  questions and invite you to participate in  the activities set forth during 2007‐2008.  participate on the listserv, contribute to the  wiki, keep reading this journal, and get  involved with ebl. also, if you are a fellow  ʺfacebookerʺ, feel free to join the evidence  based librarians group (118 members  strong at the time of writing!).    we look forward to working with you.       http://eblibrarianship.pbwiki.com/eblig+w http://www.thepartnership.ca/partnership/b http://www.eblip4.unc.edu http://www.cla.ca/about/igroups/evidence_b microsoft word es_kloda.doc evidence based library and information practice 2006, 1:3      63 evidence based library and information practice       evidence summary    despite barriers, education providers, health professionals, and students perceive e‐ learning to be an effective method of education   a review of:  childs, sue, elizabeth blenkinsopp, amanda hall, and graham walton. “effective e‐learning for  health professionals and students—barriers and their solutions. a systematic review of  the literature—findings from the hexl project.” health information & libraries journal  22.s2 (2005): 20‐32.    reviewed by:  lorie a. kloda  instructional technology librarian, life sciences library, mcgill university  montreal, quebec, canada  e‐mail: lorie.kloda@mcgill.ca    received: 01 june 2006            accepted: 19 july 2006      © 2006 kloda. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to determine barriers or  problems and possible solutions related to e‐ learning, and to determine the effectiveness  of e‐learning among health professionals  and students.    design – systematic review of qualitative  literature, in addition to interviews and  questionnaires, to allow for triangulation of  the data.    setting – “the hexl project: surmounting  the barriers to nhs e‐learning in the  north‐east.” the national health service  (nhs) in the north‐east of england, from  may 2003 to march 2004.  subjects – a systematic review of 57  qualitative studies on health and e‐learning,  phone interviews with 13 managers and  trainers, and 149 questionnaires completed  by users and non‐users of e‐learning. all  participants of the interviews and  questionnaires were staff and students of  the nhs in the north‐east of england.    methods – the study used three methods to  collect data to meet the objectives of the  study.  for the systematic review, the  databases amed (allied and alternative  medicine), assia (applied social sciences),  cinahl (nursing and allied health), eric  (education), hmic (health management),  lisa (library and information sciences),  pubmed (medline), and web of science  were searched using the terms “e‐learning”  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:3      64 or “computer assisted instruction”, and  “health”, and “barriers.” any type of  research or comprehensive literature review  was selected from the results to be included  in analysis. based on the findings from the  systematic review, a semi‐structured  interview schedule was developed for use in  phone interviews to be conducted with  managers or e‐learning trainers.  also based  on the systematic review, questionnaires  were developed and distributed to users  and non‐users of e‐learning. the three  methods permitted triangulation of the data.    main results – the search produced 161  results of which 57 met the methodological  criteria. the 57 studies categorized e‐ learning barriers and solutions into eight  different issues: organizational, economics,  hardware, software, support, pedagogical,  psychological, and skills. results from the  interviews and questionnaires mirrored  those of the systematic review.  barriers to e‐ learning included managing change, lack of  skills, costs, absence of face‐to‐face learning,  and time commitment. solutions to the  barriers of e‐learning included blended  learning, better design, skills training,  removal of costs, and improved access to  technology. there were, however, some  discrepancies between the results from the  systematic review and the interviews and  questionnaires: barriers due to “lack of  access to technology” (29) were not  perceived as serious, suggested solutions  did not include better communication and  scheduling, and the solutions to provide  trainer incentives and employment  admission criteria were rejected. users and  potential users of e‐learning mentioned one  solution not found in the review: protected  time during work to partake in e‐learning.   results from the interviews and  questionnaires demonstrated that managers,  trainers, and learners thought e‐learning to  be effective.    conclusion – the researchers answered the  study’s questions to determine the  perceived barriers and solutions to e‐ learning for the nhs in the north‐east of  england. despite the barriers identified, it  was also determined from the interviews  conducted and questionnaires returned that  managers, trainers, and learners perceive e‐ learning as an effective method of education  for health professionals and students.  further research is needed to determine  whether this perception is correct. the  systematic review of the literature identified  important “factors which need to be in  place” for e‐learning to effectively take place  (29). the barriers and potential solutions  identified are useful for those designing e‐ learning programs in any professional  context. the results point to several  requirements for e‐learning success: national  standards and strategies; curriculum  integration; change management; flexible  programming; skills training; and support  and access to technology for managers,  learners, and trainers. the authors of the  article believe that librarians play an  important role in e‐learning and identify  several areas in which librarians can  contribute.    commentary    this study employs a systematic review of  the literature as one of its methodologies—a  method that is considered rigorous when  conducted properly.  this type of design  includes systematic searching of the  literature, appraisal of the research, and  synthesis of the results. the researchers in  this study searched a broad number of  health and education databases to collect  relevant literature on the topic, however the  search strategy (“e‐learning” or “computer  assisted instruction,” and “health,” and  “barriers”) is rather simplistic and excludes  many potential near‐synonyms.  it is unclear  to what extent these terms were elaborated  upon for the various databases. it is  evidence based library and information practice 2006, 1:3      65 therefore possible that studies and  significant articles may not have been  retrieved as a result. the researchers do not  make mention of any type of critical  appraisal of any of the 57 studies included  in the review. booth argues that “true”  critical appraisal normally required for  systematic reviews of quantitative research  is not required in synthesizing qualitative  literature (2001). the researchers of the  current study, however, do not address the  quality of the studies included in their  review, which would have allowed the  reader to weigh the importance of the  results and identify potential biases. the  resulting synthesis of results, while  inclusive, may subsequently be distorted or  exaggerated.      systematic reviews are traditionally  conducted using experimental studies or at  the very least quantitative studies, yet  qualitative evidence can and should be  included to answer certain types of  questions. qualitative research design  provides rich data to support and inform  current practice. currently, there are no  validated tools for assisting in the appraisal  of qualitative research, although efforts are  underway, such as those of the campbell  collaboration and dixon‐woods. the  questions in this study were best answered  by qualitative data, and the researchers  made an admirable attempt to conduct a  systematic review of the qualitative  literature.  the addition of other qualitative  data gathering techniques to allow for  triangulation of the data supports the  conclusions drawn from the review and  proposes areas for further investigation.    the secondary objective of the study—to  determine the effectiveness of e‐learning— was not addressed by the systematic review,  but by using the two other data collection  techniques. the instruments used in the  semi‐structured phone interviews to e‐ learning managers and trainers, and in the  questionnaires distributed to users and non‐ users of e‐learning, were well described and  were logically based on results from the  systematic review.  it is not known how  many individuals received questionnaires;  so it is therefore impossible to know if  enough questionnaires were returned to  ensure representative results. the fact that  the researchers fail to report the number of  distributed questionnaires, the elaborated  search strategy, and the critical appraisal  criteria could jeopardize the validity of the  results. based on the data provided, it is  impossible to know if the results are  representative of the population of nhs  members in the north‐east of england, or of  other similar environments where health  professionals are engaged in large‐scale, on‐ line training and continuing education  initiatives.     as the authors report, the results of this  study, while not specific to librarianship, are  important for librarians and information  professionals working in different contexts:  those supporting e‐learning (or distance  education), those involved in the planning  and design of e‐learning, and those  participating in e‐learning as students.  most  importantly, as the authors point out,  librarians can assist with the skills training  required by trainers and learners to use the  hardware and software required. librarians  may also wish to incorporate information  literacy learning objects within e‐learning  tools.  academic and special librarians may  want to ensure links to information  resources are present within e‐learning  modules.  the authors correctly assert that it  is important for librarians to become  involved in e‐learning initiatives and that  they make their roles in such endeavours  evident.            evidence based library and information practice 2006, 1:3      66 works cited    booth, andrew. “cochrane or cock‐eyed?  how should we conduct systematic  reviews of the qualitative research?”  qualitative evidence‐based practice  conference: taking a critical stance. 2001.  university of leeds. 3 august 2006.  <http://brs.leeds.ac.uk/~beiwww/beia/ qebp2001.htm>.    the campbell collaboration. the campbell  collaboration(c2).  3 august 2006.  <http://www.campbellcollaboration.or g/>.    dixon‐woods, mary, et al. “how can  systematic reviews incorporate  qualitative research? a critical  perspective.” qualitative research 6.1  (2006): 27‐44.    http://brs.leeds.ac.uk/~beiwww/beia/ http://www.campbellcollaboration.or evidence summary   agile project management facilitates efficient and collaborative collection development work   a review of: stoddard, m. m., gillis, b., & cohn, p. (2019). agile project management in libraries: creating collaborative, resilient, responsive organizations. journal of library administration, 59(5), 492–511.  https://doi.org/10.1080/01930826.2019.1616971   reviewed by: abbey lewis stem engagement librarian university of colorado boulder boulder, colorado, united states of america email: abbey.b.lewis@colorado.edu   received: 11 aug. 2022                                                             accepted:  14 oct. 2022      2022 lewis. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30221     abstract   objective – to examine the advantages and obstacles of using agile (an approach to project management) principles to guide collection development work in ways that allow libraries to better address user needs while increasing transparency and collaboration in their processes.   design – descriptive case study.   setting – libraries at a private, r1 university (doctoral university – very high research activity).   subjects – five cross-disciplinary teams of three to six people, with each team focusing on a separate strategic aspect of library collections work (communications and data visualization, e-resource contract negotiation, serials workflow analysis, demand driven acquisitions, and serials budget projection & assessment).   methods – the authors facilitated group reflection sessions for the teams to surface outcomes of employing agile practices and also as a means through which they could learn from their experiences with agile. the teams engaged in reflection throughout the year-long process where they were asked to share their work, respond to the work of the other teams, and contemplate their own learning and development as a member of a team.     main results – using agile principles to structure and direct collection development work allowed the libraries to meet their stated goals of spending all available funds on relevant materials within the time frame allotted. this style of collaborative work benefitted from recognition of interrelated information needs, willingness to prioritize experimentation over seeking formal training, centering user needs in planning stages, and practicing reflection as a powerful learning tool. additionally, the authors noted a strengthening of core skills held in high value throughout libraries, such as leadership and project management. task-oriented skills that included capabilities like data visualization and operational analysis also progressed through learning by working on cross-functional teams. the authors offered guidance for applying these lessons to situations in other libraries that can be generalized to fit other projects.   conclusion – based on their experiences with adopting agile practices, the authors offered scalable approaches for implementing agile that speak to employee buy-in and the overall impact of projects undertaken in this manner. training that reflects a library’s authentic level of investment in agile, whether minimal or extensive, is crucial to realizing positive outcomes. the authors also recognized that resistance to change and discomfort with working under transparent conditions will present challenges for many libraries in aligning workflows with agile methodology. however, agile did allow for positive shifts toward more investment in shared work on team and individual levels. while failure in agile projects is more visible and therefore more intimidating, librarians can find themselves able to learn from and correct mistakes more efficiently.   commentary   in the years since agile was first considered as a potential “better mousetrap for libraries” the framework has been used to create organizations that are product-driven and highly adaptable to change (heath, 1998). in libraries, agile principles have been applied to organizational roles (wu et al., 2022), library management system development (babatope et al., 2019), remote work (singh & bossaller, 2022), and in the current study, collection development   this summary used the generic cat as an assessment tool, finding the study to have strong qualities in leveraging expertise and adding to previous research (perryman & rathbun-grubb, 2014). the authors, all of whom are employed in director roles in george washington university libraries, framed their case study as investigating the advantages and complications of using agile methods to conduct interrelated collection development work. in a substantial review of previous literature, the authors provided an overview of agile principles and highlighted notable instances of agile usage in library projects and operations.   concise descriptions of the charge for each team, training, and roles within the team delivered a general sense of how the work was carried out. the final year-end reflection was the only one described, with the exact questions used included in the text, but details on how reflection was built into previous points in the process and the methods under which it was carried out are somewhat unclear. no details were given regarding methods used for analyzing the feedback provided in the reflection sessions.   despite the lack of clarity on the structure and analysis used for the reflection sessions, the authors drew out practical implications for how agile principles can be utilized in library settings through efforts like training, appropriate framing of principles versus tools, and highlighting user stories to situate team efforts. in particular, the authors’ conclusions regarding adequately preparing people to work in this manner will assist library management with setting up teams for success in many types of projects. additional research on future team and individual work after the completion of larger-scale agile projects like this may be useful for recognizing long-term benefits for library management.   references   babatope, i. s., idowu, o. f., agwu, p. u., & dumbiri, r. (2022). developing an integrated library management system with agile methods, the university of ibadan experience. 87th ifla world library and information congress (wlic). https://repository.ifla.org/handle/123456789/1953   heath, f. (1998). the agile organization: a “better mousetrap” for libraries? in l. j. haricombe & t. j. lusher (eds.), creating the agile library: a management guide for librarians (pp. 1–16). greenwood publishing group, inc.   perryman, c., & rathbun-grubb, s. (2014). the cat: a generic critical appraisal tool. https://form.jotform.us/42065968239162   singh, k., & bossaller, j. s. (2022). it’s just not the same: virtual teamwork in public libraries. journal of library administration, 62(4), 512–534. https://doi.org/10.1080/01930826.2022.2057130   stoddard, m. m., gillis, b., & cohn, p. (2019). agile project management in libraries: creating collaborative, resilient, responsive organizations. journal of library administration, 59(5), 492–511.  https://doi.org/10.1080/01930826.2019.1616971   wu, y., guimaraes, a., & wang, z. (2020). product owners at hesburgh libraries: increasing stakeholder engagement and accountability through continuous organizational enhancement. journal of library administration, 60(7), 695–713. https://doi.org/10.1080/01930826.2020.1797329 evidence based library and information practice evidence based library and information practice 2009, 4.4     evidence based library and information practice         acknowledgement of editorial advisors      © 2009. this is an open access article distributed under the terms of the creative commons attribution   license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and   reproduction in any medium, provided the original work is properly cited.      in 2009, the following people volunteered their time as editorial advisors, providing blind peer  review of articles and evidence summaries that were submitted to evidence based library and  information practice.  the editorial team thanks the editorial advisors for their time, knowledge, and  dedication which have contributed to the success of this publication.    erin alcock, memorial university of newfoundland, canada  greg bak, library and archives canada, canada  marcus banks, university of california san francisco, united states of america  anthony bernier, san jose state university, united states of america  alissa black‐dorward, fordham university school of law, united states of america  andrew booth, university of sheffield, united kingdom  cara bradley, university of regina, canada  marcy brown, silverchair science & communication, united states of america  jeanette buckingham, university of alberta, canada  deborah charbonneau, wayne state university, united states of america  su cleyle, memorial university of newfoundland, canada  barbara combes, edith cowan university, australia  james davies, loughborough university, united kingdom  sandy degroote, university of illinois at chicago, united states of america  orvie dingwall, canadian patient safety institute, canada  lani draper, stephen f. austin state university, united states of america  jonathan eldredge, university of new mexico, united states of america  juliet eve, university of brighton, united kingdom  alison farrell, memorial university of newfoundland, canada  bill fisher, san jose state university, united states of america  lyle ford, university of manitoba, canada  daniel german, library and archives canada, canada  lindsay glynn, memorial university of newfoundland, canada  susan goodwin, texas a&m university, united states of america  gillian hallam, queensland university of technology, australia  k. alix hayden, university of calgary, canada  tony horava, university of ottawa, canada  joanne jordan, keele university, united kingdom  anthi katsirikou, university of piraeus, greece  catherine king, national center for immunisation research and vaccine preventable diseases,  australia  vincent lariviere, lʹuniversité du québec à montréal, canada  4 evidence based library and information practice 2009, 4.4  5 deborah lee, mississippi state university, united states of america  michael lines, university of victoria, canada  lauren maggio, stanford university, united states of america  yazdan mansourian, tarbiat moallem university, iran  marcia mardis, wayne state university, united states of america  christine marton, university of toronto, canada  paula mcmillen, university of nevada las vegas, united states of america  mac nason, algonquin college, canada  cleo pappas, university of illinois at chicago, united states of america  t. scott plutchak, university of alabama at birmingham, united   bob pymm, charles sturt university, australia  asim qayyum, charles sturt university, australia  ann roselle, phoenix college, united states of america  robert russell, northern state university, united states of america  julie rustad, college of st. scholastica, united states of america  pam ryan, university of alberta, canada  christine sammon, alberta college of art & design, canada  becky skidmore, society of obstetricians and gynaecologists of canada, canada  mark spasser, palmetto health, united states of america  donna timm, louisiana state university, united states of america  lauri vaughan, the harker school, united states of america  li zhang, university of saskatchewan, canada    evidence summary   sustainable digital preservation initiatives benefit from multi-pronged approach   a review of: masenya, t. m., & ngulube, p. (2020). factors that influence digital preservation sustainability in academic libraries in south africa. south african journal of libraries and information science, 86(1), 52–63. https://doi.org/10.7553/86-1-1860   reviewed by: jordan patterson cataloging and metadata librarian queen elizabeth ii library memorial university of newfoundland st. john’s, newfoundland and labrador, canada email: jpatterson13@mun.ca   received: 1 dec. 2020                                                                accepted:  19 jan. 2021      2021 patterson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29887     abstract   objective – to define principles for the sustainable management and preservation of digital resources.   design – survey and literature review.   setting – academic libraries in south africa.   subjects – twenty-two academic institutions in south africa.   methods – the researchers evaluated four conceptual models of digital preservation and conducted a literature review for the same subject. informed by these reviews, the researchers developed a questionnaire for south african academic institutions, distributed the questionnaire, and studied the results using statistical analysis software.   main results – twenty-two of twenty-seven (81.5%) surveys were returned. results indicated a broad consensus about which factors were important in sustainable digital preservation; all factors listed received anywhere from 86.3% to 100% agreement among respondents.   conclusion – a proposed conceptual integrated digital preservation model recommends a three-pronged approach to address management-related, resource-related, and technological-related factors in sustainable digital preservation.   commentary   the central issue of this study is a central issue of libraries in the information age: how can libraries ensure consistent, reliable access to materials heavily dependent on technologies and services that seem to change from year to year? sustainable digital preservation is an ideal many would recognize in name but would have difficulty putting into practice. masenya and ngulube seek to solve this problem by developing a conceptual model that highlights the necessary constituent parts of successful digital preservation initiatives.   masenya and ngulube follow in the footsteps of the esteemed forebears they evaluate, from carnegie mellon university’s digital preservation capability maturity model (1990); through davies’ policy, strategy, and resources troika model (2000) and the open archival information system model (2002); to corrado and moulaison’s digital preservation triad (2014). this study, which meets accepted standards of validity when assessed with glynn’s critical appraisal tool, surveys south african academic institutions and proposes a conceptual model for a new decade (glynn 2006).   while on the surface, the research design of this study does not present any obvious flaws, the homogenous responses to the questionnaire suggest an alternate approach may have yielded more insightful results. in their environmental scan, the researchers collected factors that influenced the success of digital preservation initiatives. they then used those same factors in a questionnaire that asked librarians at academic institutions to use a likert scale to state whether they agreed if those factors were influential. the consequence is a tautological study: the results of the questionnaire provide the same information they used to create the questionnaire. put another way, the survey determined its own results.   the likert scale yields objective, quantitative results that may be easier to analyze, but allowing institutions to write in their own answers (i.e. not providing answers for them) would provide richer, if subjective and harder to analyze, results. after all, very few responsible librarians, attentive to the breadth of issues associated with sustainable digital preservation, could imaginably disagree with the importance of any one of these factors. allowing the respondents freedom in their answers would more accurately reflect their own local institutional concerns regarding digital preservation. greater freedom could also derive a greater variety of responses, attuning the researchers to previously unstated issues in digital preservation. the study in its present state, however, only allows the researchers to confirm what they already knew.   nevertheless, compounded with the researchers’ initial literature review, the results of the survey clearly indicate that long-standing issues influencing digital preservation sustainability continue to be concerns for libraries—concerns often shared among institutional peers in a region. libraries interested in pursuing digital preservation initiatives, especially those in emerging areas, should heed the study’s recommendation to build regional partnerships to develop expertise, pool resources, and benchmark their progress.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387–399. https://doi.org/10.1108/07378830610692154 research article   student use of the information commons: an exploration through mixed methods   susan gardner archambault head of reference & instruction william h. hannon library loyola marymount university los angeles, california, united states of america email: susan.archambault@lmu.edu   alexander justice reference & instruction librarian william h. hannon library loyola marymount university los angeles, california, united states of america email: ajustice@lmu.edu   received: 14 july 2017    accepted: 23 oct. 2017         2017 archambault and justice. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – in this case study, librarians at the william h. hannon library at loyola marymount university explored user behaviour in the information commons, user preferences for furniture style and configuration, and how users engaged with a mix of technology, resources, and activities inside the space.   methods – the researchers used a mixed-methods case study consisting of 2,443 “direct observations,” 646 environmental scans, 248 patron surveys, and 46 whiteboard poll questions. they created visualizations of results in tableau, with filters for zone and variable. they then carried out a follow-up furniture preferences survey with 190 respondents.   results – independent study dominated the space usage. users valued spaciousness, quiet, privacy, and a clean environment. users frequently multi-tasked with additional devices as they simultaneously used the library computers, including cell phones, headphones, and laptops. the majority of students self-reported using a library computer for email and to access the campus online learning platform. they also reported reading/studying and printing as frequent activities, although these were less frequently observed. unattended belongings were observed along with broken electrical outlets. temperature and noise levels were highly variable.   conclusions – this methodology allowed for the exploration of space use and satisfaction and uncovered implications for the redesign of the library space. the library has already taken steps toward making improvements based on this assessment project including: removing some reference stacks in favor of additional seating space, an inventory of all electrical outlets, and the exploration of new furniture and noise control strategies.     introduction   in 2009, loyola marymount university opened its doheny information commons as part of the new william h. hannon library. the hannon library is open 24/5 with free wireless and a liberal food and drink policy. the first floor of the library features a café, the circulation desk, lockers with electrical outlets, four walk-up computers, library classrooms, reference stacks, and an information commons with 80 computers, 3 enclosed presentation practice rooms, and an information desk for reference and basic technology and printing help. throughout the second and third floors are over 30 enclosed group study rooms that can be reserved online. the second floor features a branded service desk for it support, device repair, and equipment checkout, a popular reading collection, print periodicals stacks, carrels for both media and individual study, and book stacks. the third floor is the designated quiet floor and features exhibit space, archives and special collections, individual study carrels and group tables, a faculty commons and faculty technology sandbox, a large multi-purpose room, and book stacks.   the doheny information commons space met bailey & tierney’s (2008) traditional definition of an information commons by featuring 80 computers as “single workstation(s) with access to traditional library services and productivity resources in a high-technology-rich environment” (p. 2). furthermore, the library intended it to cater to the millennial student, who needs space conducive to social learning, collaboration, and group activities (lippincott, 2012). the same year the new space opened, ross housewright (2009) warned that academic libraries were “at risk of losing their functional relevance and fading to primarily symbolic importance” if they failed to adapt to the changing information environment (p. 254). additionally, housewright asserted that it is important to align with the changing needs of students and faculty by “redeploy(ing) resources flexibly” (p. 259). the mission of the new commons was to support academic life at lmu, but how long might it be until it was no longer offering optimal support?   in 2013, the library strategic plan called for post-occupancy planning of emerging space utilization needs in the public areas of the library. this reflects lippincott’s (2012) advice that “it is important to collect information on the actual needs of students and not just on needs perceived by librarians” (p. 540). the question for any institution that has set up a new space as an information or learning commons is aptly summed up by librarians from harrisburg university of science and technology: “we need to explore ways of assessing the extent to which learning commons services and resources help students succeed as self-directed learners” (adams & young, 2010, p. 159). we designed the study described in this article to address the question of whether an information commons, after six years, was still optimally supporting students’ academic life in a rapidly changing environment.   literature review   published research on the design and use of library spaces, especially for the purposes of assessment and planning (or re-planning), has appeared regularly in the second decade of the 21st century. this research has been framed often in the context of paradigm changes that appear with each generation of new libraries, as set out by the frequently cited university librarian emeritus at yale, scott bennett (2008, 2009). the most recent shift is identified as “learning-centered,” with spaces in new and renovated libraries dedicated as either learning or information commons, and bennett (2009) calls on his profession “to launch a design practice centered on learning” (p. 194). an entire library might become a university’s information commons (hisle, 2005), or through new additions or renovation libraries could now contain “a flexible, reconfigurable space that is sized to a reasonable subpopulation of students and equipped with group learning spaces as appropriate” (beagle, tierney, & bailey, 2006, p. 9). neither the name nor the space came to have a consistent definition or execution, but on the whole this situation has not impeded the beneficial results for libraries or librarianship (bonnand & donahue, 2010).   even as beagle and others were contributing the vision and summing up the new library spaces, their colleagues in the field had begun the research into library spaces and users that would soon create a substantial body of research with a broad spectrum of methodologies. these ranged from quantitative, intensive seat counts (dotson & garris, 2008) to the purely creative setting of the design charrette (oliveira, 2016; washburn & bibb, 2011). some early studies used a quantitative approach with direct observation sweeps (applegate, 2009; dotson & garris, 2008; given & leckie, 2003) and simple paper questionnaires (gardner & eng, 2005). these two modes of quantitative data collection look at what users do, on the one hand, and try to get at what users want and expect, on the other. this mixture of direct observation combined with questionnaires created a key set of studies (holder & lange, 2014; i̇mamoğlu & gürel, 2016; may & swabey, 2015; mccrary, 2017).   however, the need to acquire data about both of these important aspects of library spaces has, on the whole, led to research that uses two or more modes of collection and often more than one methodological approach. ethnographic techniques have been popular, either as the sole approach (bedwell & banks, 2013), or as part of a mixed method toolkit, such as a combination of video observation, the nsse, and user surveys (webb, schaller, & hunley, 2008). even larger mixes of methods have brought together the quantitative survey, focus groups, filmed interviews, and student-made films (cowan, 2012), or time-lapse photography, unobtrusive observation, and random-sampled surveys (asher, 2017). however, “ethnography is a complicated and time-consuming research method” (khoo, rozaklis, & hall, 2012, p. 82).   the majority of research in this literature review looked at the use of space or the behaviour of users in library spaces generally, with a minority dedicated to pre-occupancy assessment for design (23%), or post-occupancy assessment (18%). further, not all of the latter had as their subject a designated commons or similar space. some research focused solely on measures or ideal attributes of library or commons space. cha & kim (2015) used surveys of academic library users in the netherlands, while the teals standard was the product of another (abbasi, elkadi, horn, & owen, 2012). only two longitudinal studies exist (fox & doshi, 2017; montgomery, 2014). we believe that our research makes a strong contribution to this important post-occupancy category of library space assessments and data.   another gap that we identified is that of studies that provide detailed data on use of the computers and devices that are the key components of the information commons model. even though the quantitative approach predominates in the research under review (about 42%, with an additional 37% if mixed-method research is added), and direct observation is frequently employed, only two studies (5%) included observation of the screen itself: that is, what the users were actually doing on their computers and devices while they were in the library. to answer the question of what students “really do” in the library, paretta & catelano (2013) used direct observation at two sites. these researchers did not look at space use or other factors, but do provide highly detailed data on print vs. computer, and academic activity vs. leisure activity. the one other study also utilized direct observation, and set out to analyze the use of a particular library space (not a commons), but didn’t provide any detail about the frequency of academic vs. leisure use of technology (faletar tanackovic, lacović, & gašo, 2014).   is there a developing evidence base of common findings among space studies? the preponderance of users working on their own has been frequently found (bryant, matthews, & walton, 2009; crook & mitchell, 2012; ferria et al., 2017; holder & lange, 2014; thomas, van horne, jacobson, & anson, 2015). one ethnographic study highlighted single users “appropriating” group tables by spreading out personal items (bedwell & banks, 2013, p. 12). on the other hand, if group space is not provided, it will be improvised (hursh & avenarius, 2013; webb, shaller, & hunley, 2008).   the research is not clear about other elements of expectations for library spaces, however. students still value and prefer a quiet library, despite predictions about upcoming generations, but what constitutes excess or distracting noise appears to be contextual (cowan, 2012; crook & mitchell, 2012; faletar tanackovic, lacović, & gašo, 2014; khoo, rozaklis, hall, & kusunoki, 2017; mccaffrey & breen, 2016; newcomer, lindahl, & harriman, 2016; regalado & smale, 2015; suarez, 2007). even very recent studies suggest that “students consider the quiet communal spaces integral to their experience of the library” (yoo-lee, lee, & velez, 2013, p. 509).   furniture is another component of the information commons that some writers thought would need to depart from traditional library practices (hisle, 2005). this may not entirely be the case. more than a few studies indicate that the ideal workspace may be “a big desk,” that is, rectangular, no-frills tables, or the old-fashioned carrel (hall & kapa 2017). arguably, “space in which to spread themselves and their belongings out” (washburn & bibb, 2011) is the feature in question, rather than the furniture itself. not all the data agrees. ferria et al. (2017) found booths popular, and webb, schaller, & hunley (2008) found 60% of respondents split between wanting tables and a preference for soft seating.   aims   in 2009, this new library had opened with a “perfect” information commons space designed to support the millennial student. after six years, our aim was to see if students were utilizing the information commons in unexpected ways and deviating from the anticipated “millennial” behavior for which we had built it. we designed our post-occupancy study to answer the following research questions:   ·         what areas of the information commons do students really use or not use?  ·         what furniture configurations are preferred? ·         what mix of technology, resources, and activities are students engaged with inside of the space?     figure 1 observation zones of the information commons.       methods   given our research question – to assess how students and others use the hannon library’s information commons – we adopted quite naturally the case study. we wanted to know what the users were doing, and where, and with whom, but also to find out their attitudes towards key environmental factors such as comfort, temperature, and noise. simons (2009) defined case study research as “an in-depth exploration from multiple perspectives of the complexity and uniqueness of a particular project, policy, institution, programme or system in a ‘real life’ context. it is research-based, inclusive of different methods and is evidence-led” (p. 21). many case studies employ mixed methods, defined as “the combined use of both quantitative and qualitative methodologies within the same study in order to address a single research question” (hewson, 2006, p. 179).  we had little difficulty in choosing to employ mixed methods, because such “research works particularly well for case study research as it allows the researcher to take the rich empirical data yielded from case studies and apply either quantitative or qualitative methods to the data” (kitchenham, 2010, p. 2). we carried out the study during the spring semester of 2015 between april 6th and 19th. all instruments and procedures for this study were reviewed and approved by the lmu institutional review board.   direct observations   direct observation of library spaces can be as simple as the sweep or headcount (given & leckie, 2003) or more “systematic” observation that collects specific data about users such as “gender, being or not being in group” and activities or technology (applegate, 2009, p. 342). if a research question will benefit, the researchers can use direct observation to collect both quantitative and qualitative data (ferria et al., 2017). the first data collection method consisted of 2,443 direct observations made about the location and behavior of subjects who were physically present in the information commons, or about the space itself such as out of order equipment and unattended belongings. daily observations were made in the morning (between 9:00 a.m. and 12:00 p.m.), afternoon (between 1:00 p.m. and 4:00 p.m.), evening (between 5:00 p.m. and 8:00 p.m.), and late night (after 9:00 p.m.) by librarians and library staff. to better facilitate the observations, the researchers adopted the zone concept, “spaces… coherent in terms of use, environment, furniture, and so on” (khoo, rozaklis, hall, & kusunoki, 2017, p. 57). we divided the information commons into five observation zones (see figure 1) with designated observation points. observers recorded each person’s activity and use of library resources and technology using a qualtrics form on ipads (see appendix a).   environmental scans   given & archibald (2015) recommended “to use a range of methods” in assessment that would include factors such as lighting and noise, to make up for a lack of direction in this regard in library design guidelines (p. 102). therefore, the researchers collected 626 environmental scans in each zone at designated areas in a separate qualtrics form (see appendix b). they measured temperature with a digital thermometer in degrees fahrenheit with a humidity range, they measured noise level in decibels with a digital sound meter, and noted any out of order equipment (computers, printers, scanners, lights, and photocopier), as well as any messy or dirty sites.   patron surveys   while direct observation can address who and where questions, it cannot provide insight into the users’ choices and expectations. we wished to collect and incorporate data about “spatial choice” and “the importance of space attributes in different contexts” (cha & kim, 2015, p. 274). a convenience sample of patrons filled out 248 questionnaires in the information commons during the same two-week period (see appendix c). the survey asked users to self-report on activities and preferences, and it measured many of the same things we directly observed to allow for comparison and triangulation. we used a $50 amazon gift card raffle and chocolates as incentives.   whiteboard polls   in order to provide an additional “user feedback channel” (halling & carrigan, 2012, p. 70), the final data collection method consisted of polling by means of small mobile whiteboards stationed in the information commons. we posted questions on each whiteboard:   1.       i need ___ when i come to the information commons. 2.       what adjective best describes the information commons? 3.       put a smiley face on the map (of the information commons) next to the things you like. 4.       how can we improve the first floor? 5.       name one change you would make to the first floor space.   furniture survey   researchers deployed a follow-up furniture survey some two years after the primary study, and 190 respondents participated during a one-week period during the spring semester of 2017. we built the survey online using the qzzr software and provided pictures of a variety of furniture options. during the day, library staff circulated in the information commons in staggered shifts, and asked users to complete the survey to gather input on how to configure existing and future furniture (see appendix d). staff used chocolates as an incentive.       table 1 furniture survey results furniture purpose top preference reasons library computer workstations private group workstation (88%) privacy (70%), space for my stuff (67%) table & chair configuration (without desktop computer) 4 chairs facing each other (54%) studying/homework (59%), collaboration (59%), space for my stuff (52%) table shape rectangular (78%) space for my stuff (68%), studying/homework (55%), comfort (51%) working alone s-divider (52%); carrel (39%) s-divider for privacy/closed off (70%), comfort (60%), aesthetics (59%). carrel for privacy/closed off (57%), good writing surface (58%). collaborating enclosed booth (47%); high-back booth (38%) enclosed booth for privacy/closed off (81%), seating configuration (57%). high-back booth for seating configuration (63%), comfort (53%). lounge furniture high-back couch with coffee table (46%); armchair and coffee table (37%) high-back couch for comfort (89%) and aesthetics (89%). armchair and coffee table for comfort (89%). take a phone call soundproof phone booth (45%) privacy (92%), quiet (72%)       data analysis   a combination of quantitative and qualitative approaches was employed to analyze the collected data. quantitative reports were run in qualtrics for the patron survey, direct observations, and environmental scans to extract frequency distributions and percentages to ascertain usage and preferences. a similar report was run in qzzr for the furniture survey. furthermore, usage counts were filtered in qualtrics by day of the week, time of day, and zone to look for significant differences. this allowed for the construction of a heat map to visualize usage (see also asher, 2017 for a similar approach to heat mapping), and a visualization dashboard in tableau to visualize activity (see appendix f). the mean temperature and noise level was calculated as a representative measure of central tendency for each zone. also, the mean was calculated to represent the average amount of available space taken up by each user and the average number of users inside of each group study room.   answers to the qualitative questions in the patron survey and whiteboard polls were coded thematically and categorized for analysis using excel to identify patterns in users’ affective attitudes towards the information commons. the 10 major categories that emerged are in appendix g. routine usage statistics of the computers in the information commons was consulted during the same time period. the researchers compared the results from each of the methods employed to look for verification across multiple data points.   results   user demographics   there were 2,107 (86%) direct observations that recorded the presence of a person using a space, using the laser printers, or otherwise engaged in some use of the information commons. of these, 1,096 (52%) were females, while 1011 (48%) were males. undergraduates accounted for 213 (86%) of the patron survey respondents, while 22 (9%) were graduates and 13 (5%) were guests. the gender of 151 (61%) of the patron survey respondents was female, while 97 (39%) was male. the whiteboard polls and furniture survey did not ask for demographics.   seating and furniture   the most popular seating configuration during the study was the workstation with a desktop library computer designed for group work and collaboration (see appendix e and figure 2). we established this by our direct observations as well as by computer use statistics extracted by it during the same period of time. the follow-up furniture survey two years later also supported the initial findings (see table 1) where 167 respondents (88%) preferred a private group workstation over computers right next to each other because they wanted privacy and space. even though the group workstations were designed for collaboration, 207 students (84%) reported working alone in the patron survey, and we observed 1833 students (87%) working alone through direct observations. there were on average 1.1 people using the enclosed presentation practice rooms. in another averaged measurement, users took up 81% of all available space at each workstation in the observations even though there was usually only one person at a station designed for collaboration. often, the user treated the space like their home or office by spreading out study material, devices, food, and clothing. in the patron survey, respondents were asked to rank the top five reasons they chose their spot. the top 5 reasons were “access to a library computer” (164 total votes and 81 #1 rankings), “spacious” (166 total votes and 52 #1 rankings), “quiet” (135 total votes and 42 #1 rankings), “clean” (103 total votes), and “privacy” (102 total votes).     figure 2 this particular multi-person workstation is the most popular site in the information commons.   user activities   in the patron survey, 228 respondents (92%) self-reported using a library computer in the information commons, and this was also the most frequent activity directly observed (see figure 3). of the respondents, 25 students said the library computers could be improved by having more of them, disallowing those not using them to sit there, being cleaner, having more specialized software, and/or having mice that worked better. other frequent activities users self-reported: 188 (76%) said they were reading/writing/studying, which observers only recorded for 590 students (28%); and 159 (64%) self-reported printing, which observers only recorded for 253 students (12%). seven students complained that the printers could function better and be spread out to other parts of the commons. we observed a printer out of order 69 times (11%) during the 2-week period.     figure 3 surveys alone may be somewhat inaccurate.     figure 4 users’ purpose for utilizing library computers.       figure 5 use of technology.       we found that 67 respondents (27%) in the survey said they were eating or drinking, which we directly observed 695 times (33%). the users’ concerns with eating and drinking while at work appeared in survey and whiteboard poll responses, where 24 comments suggested improvements such as snack vending machines, a water bottle refilling station, or a designated food zone. there were eight users who said that there should be less trash and food.   respondents self-reported utilizing a library computer 169 times (68%) for email and 149 times (60%) for mylmuconnect, while only 119 people (48%) reported using a library computer for productivity software and 102 people (41%) for library resources (see figure 4). the only significant usage captured through direct observation was productivity software, observed 885 times (42%). users reported frequently multi-tasking with additional devices while they used the library computers, including smart phones 126 times (51%), headphones 99 times (40%), and laptops 94 times (38%) (see figure 5). direct observation noted these devices, although less frequently. both methods found tablets very little used.     figure 6 average temperature and average noise level.     environment   the average temperature was between 72 and 75 degrees fahrenheit (see figure 6) across all 5 zones, but there was variation. in zone 5, it got as low as 67 degrees, and in zone 2 as low as 67.5 degrees. there were nine respondents who commented either in the patron survey or the whiteboard polls that the temperature was too cold (see appendix g). the average decibel level was between 50 and 55, but there was also wide variation. in zone 3, observers recorded up to 76.8 dba. zones two and three were the noisiest because of their proximity to the printers and the lobby and information desk. there were 32 students who commented that it was too loud and that there should be a designated “no talking” and “no cell phones” area. remarkably, there were 161 reports of unattended belongings also observed, most frequently after 9 p.m. we also observed several broken power outlets, prompting users to plug into more distant outlets to compensate, creating a trip hazard.  of the observations in zone 5, broken lights were observed 16 (13%) times, and 4 students commented it needed to be brighter.   discussion and outcomes   although the planners and architects designed the information commons as a collaborative, social learning space, the results of the study indicate that students want quiet, privacy, and space to spread out, along with cleanliness. since we only observed 1412 (67%) students actually using a library computer (often they were using their own devices while sitting at a computer workstation), we determined that they could benefit from study space not tied to a desktop computer. we opened up more study space in the information commons in the summer of 2016 by emptying underutilized reference stacks and replacing them with over 30 additional seats (large rectangular tables and chairs). in the summer of 2017, we installed floor-box power outlets to go along with this new seating.   we also made a handful of small improvements based on the study, including the installation of self-serve dispensers with wipes for cleaning keyboards and tabletops. we worked with it and facilities to set up an inventory for outlets, ethernet ports, and light fixtures that could be checked regularly. responding to comments in our surveys, we made sure to supply new imac mice, installed a new water bottle filling station, and relocated one printer to the far side of the information commons. another upcoming change will be to install bag hooks underneath the computer workstations that are close together to offer more space.   even though our study established that a majority of our users preferred library computer workstations with privacy and space to spread out, we still had a gap in knowledge about other furniture preferences. as bieraugel & neill (2017) point out in an article applying bloom’s taxonomy to library spaces and creativity, it is important to design learning spaces for different intended behaviors (p. 37). they determined, for example, that quiet study space and computer labs support reflection, but on the other hand communal tables support networking (p.48).   there were 28 respondents who suggested napping pods, newer furniture, more desk space, and more comfort during our study. we therefore undertook a furniture survey done in spring 2017 to gather more information. this follow-up research revealed that for tables and chairs without a library computer, 103 respondents (54%) preferred 4 chairs facing each other because this was conducive to studying or homework, collaboration, and offered space for their belongings (see table 1). a great majority, 148 (78%), preferred a rectangular table shape. therefore, we rearranged the new seating into rectangular tables with four chairs facing each other.   the furniture survey asked about preferences for doing additional specific activities. for working alone, 99 students (52%) preferred the s-divider, while 74 (39%) preferred a carrel. beyond the obvious factor of privacy, those choosing the s-divider mentioned comfort and aesthetics as important factors, while those choosing a carrel mentioned a good writing surface. when asked about their preference for collaborating, 89 (47%) chose an enclosed booth for privacy and its seating configuration, while 72 (38%) chose a high-back booth for its seating configuration and comfort. this seating configuration is consistent with the preference for the table and chair configuration as four chairs facing each other. for lounge furniture, 87 respondents (46%) preferred a high-back couch with coffee table, while 70 (37%) preferred an armchair and coffee table. comfort was an important factor behind both choices, while those opting for the high-back couch also mentioned aesthetics. we were able to put this recommendation to use in the spring of 2017 when we replaced the low-back couches in the first floor lobby area with armchairs and coffee tables. we plan to make future strategic budget requests for furniture based on these preferences.   to address the general noise problem in the information commons, we consulted with newson brown acoustics, llc. they suggested an electrical sound masking system to make background noise or white noise. other suggestions included sound baffles, adding physical barriers between workstations, or designating certain areas for quiet vs. social and putting up signage. also, our follow-up furniture survey found that 86 respondents (45%) preferred to take a phone call in a soundproof phone booth because it offered privacy and quiet. these are similar to possible solutions found in the literature: mccaffrey and breen (2016) found evidence that “interventions such as the development of a noise policy, zoning, rearranging of furniture, removal of service points from reader spaces, and structural improvements to reduce noise travel are worthwhile interventions for libraries to consider when faced with noise problems” (p. 788).   limitations   the direct observation methodology had limitations. if users had multiple browsers open during the direct observations, we only recorded what was in the open window on their screen at that moment in time. it was also difficult to be discreet when observing users; we recorded “can’t tell” for what purpose they were using a library computer on 527 (25%) of the direct observations. some students failed to take the whiteboard polling methodology seriously and left facetious answers. another limitation was the lack of real measurement of users’ true level of collaboration during a seating sweep since it provides only a snapshot in time rather than an ethnography. a limitation to the follow-up furniture survey came with it being online only, and therefore we recruited only those using a computer or device. the final limitation was that all observation and survey instruments were limited to the information commons space. we therefore have data only from those who we found in that space, but no data from those who, from choice or other circumstance, were not in that space. we don’t know who chose not to be there, or who wanted to be there but couldn’t, and the insights those groups of users might have provided.   conclusions   this mixed methods case study explored the usage, satisfaction, and preferences of users in the hannon library information commons. independent study dominated the space usage. students valued spaciousness, quiet, privacy, and a clean environment. students frequently multi-tasked with additional devices as they simultaneously used a library computer, including cell phones, headphones, and laptops. also, unattended belongings were frequently observed along with broken electrical outlets. the study paved the way for improvements and the partial redesigning of the space. even though our study confirmed some findings from other studies, including a preference for working alone and the desire for quiet, it is important for each library to conduct its own assessment because “one size does not fit all academic libraries” and “designs will, and should be, different on every campus” (head, 2016, p. 26).   acknowledgements   the authors wish to acknowledge contributions by the following colleagues: lmu library reference & instruction department (elisa slater acosta, jennifer masunaga, desirae zingarelli-sweet, lindsey mclean, nataly blas, kathryn ryan, aisha conner-gaten), burney wong, denise blanchet, katherine donaldson, rachel deras, angie so, javier garibay, 2015 lmu library assessment committee, lmu library information commons student workers, lmu lac group overnight staff, and laura massa.     references   abbasi, n., elkadi, h., horn, a., & owen, s. (2012, october). teals (tool for evaluation of academic library spaces) project : evaluating physical library spaces. presented at the library assessment conference: building effective, sustainable, practical assessment, charlottesville, va. retrieved from http://dro.deakin.edu.au/view/du:30048261      adams, n. e., & young, j. b. (2010). users learning from users: building a learning commons from the ground up at a new university. college & undergraduate libraries, 17(2–3), 149–159. https://doi.org/10.1080/10691316.2010.481607    applegate, r. (2009). the library is for studying: student preferences for study space. journal of academic librarianship, 35(4), 341–346. https://doi.org/10.1016/j.acalib.2009.04.004   asher, a. d. (2017). space use in the commons: evaluating a flexible library environment. evidence based library & information practice, 12(2), 68–89. https://doi.org/10.18438/b8m659   bailey, d. r., & tierney, b. g. (2008). transforming library service through information commons: case studies for the digital age. chicago: american library association.   beagle, d. r., tierney, b. g., & bailey, d. r. (2006). the information commons handbook. new york: neal-schuman publishers.   bedwell, l., & banks, c. (2013). seeing through the eyes of students: participant observation in an academic library. partnership: the canadian journal of library & information practice & research, 8(1), 1-17. https://doi.org/10.21083/partnership.v8i1.2502   bennett, s. (2008). the information or the learning commons: which will we have? journal of academic librarianship, 34(3), 183–185. https://doi.org/10.1016/j.acalib.2008.03.001   bennett, s. (2009). libraries and learning: a history of paradigm change. portal: libraries and the academy, 9(2), 181–197. https://doi.org/10.1353/pla.0.0049   bieraugel, m., & neil, s. (2017). ascending bloom’s pyramid: fostering student creativity and innovation in academic library spaces. college & research libraries, 78(1), 35-52. https://doi.org/10.5860/crl.78.1.35   bonnand, s., & donahue, t. (2010). what’s in a name? the evolving library commons concept. college & undergraduate libraries, 17(2–3), 225–233. https://doi.org/10.1080/10691316.2010.487443   bryant, j., matthews, g., & walton, g. (2009). academic libraries and social and learning space: a case study of loughborough university library, uk. journal of librarianship and information science, 41(1), 7–18. https://doi.org/10.1177/0961000608099895   cha, s. h., & kim, t. w. (2015). what matters for students’ use of physical library space? journal of academic librarianship, 41(3), 274–279. https://doi.org/10.1016/j.acalib.2015.03.014   cowan, s. m. (2012). assessment 360: mapping undergraduates and the library at the university of connecticut. storrs, ct: university of connecticut libraries. retrieved from https://www.clir.org/pubs/resources/assessment360.pdf   crook, c., & mitchell, g. (2012). ambience in social learning: student engagement with new designs for learning spaces. cambridge journal of education, 42(2), 121-139. https://doi.org/10.1080/0305764x.2012.676627   dotson, d. s., & garris, j. b. (2008). counting more than the gate: developing building use statistics to create better facilities for today’s academic library users. library philosophy & practice, sep2008, 1–13.   faletar tanackovic, s., lacović, d., & gašo, g. (2014). student use of library physical spaces: unobtrusive observation of study spaces in an academic library. libraries in the digital age (lida) proceedings, 13. retrieved from http://ozk.unizd.hr/proceedings/index.php/lida/article/view/114/116   ferria, a., gallagher, b. t., izenstark, a., larsen, p., lemeur, k., mccarthy, c. a., & mongeau, d. (2017). what are they doing anyway?: library as place and student use of a university library. evidence based library and information practice, 12(1), 18–33. http://dx.doi.org/10.18438/b83d0t   fox, r., & doshi, a. (2017). longitudinal assessment of “user-driven” library commons spaces. evidence based library and information practice, 8(2), 85–95. http://dx.doi.org/10.18438/b8761c   gardner, s., & eng, s. (2005). what students want: generation y and the changing function of the academic library. portal: libraries and the academy, 5(3), 405–420. https://doi.org/10.1353/pla.2005.0034   given, l. m., & archibald, h. (2015). visual traffic sweeps (vts): a research method for mapping user activities in the library space. library and information science research, 37(2), 100-110. http://doi.org/10.1016/j.lisr.2015.02.005    given, l. m., & leckie, g. j. (2003). “sweeping” the library: mapping the social activity space of the public library. library and information science research, 25(4), 365–385. https://doi.org/10.1016/s0740-8188(03)00049-5   hall, k., & kapa, d. (2017). silent and independent: student use of academic library study space. partnership: the canadian journal of library and information practice and research, 10(1), 1-38. https://doi.org/10.21083/partnership.v10i1.3338   halling, t. d., & carrigan, e. (2012). navigating user feedback channels to chart an evidence based course for library redesign. evidence based library and information practice, 7(1), 70–81. https://doi.org/10.18438/b81w35   head, a. j. (2016). planning and designing academic library learning spaces: expert perspectives of architects, librarians, and library consultants. seattle, washington: the university of washington information school.   hewson, c. (2006). mixed methods research. in v. jupp (ed.), the sage dictionary of social research methods london: sage. http://doi.org/10.4135/9780857020116    hisle, w. l. (2005). the changing role of the library in the academic enterprise. presented at the 12th acrl national conference. minneapolis, mn. retrieved from http://www.ala.org/acrl/sites/ala.org.acrl/files/content/conferences/pdf/hisle05.pdf     holder, s., & lange, j. (2014). looking and listening: a mixed-methods study of space use and user satisfaction. evidence based library & information practice, 9(3), 4-27. https://doi.org/10.18438/b8303t        housewright, r. (2009). themes of change in corporate libraries: considerations for academic libraries. portal: libraries and the academy, 9(2), 253–271. https://doi.org/10.1353/pla.0.0048   hursh, d. w., & avenarius, c. b. (2013). what do patrons really do in music libraries? an ethnographic approach to improving library services. music reference services quarterly, 16(2), 84-108. https://doi.org/10.1080/10588167.2013.787522   i̇mamoğlu, ç., & gürel, m. ö. (2016). “good fences make good neighbors”: territorial dividers increase user satisfaction and efficiency in library study spaces. the journal of academic librarianship, 42(1), 65–73. https://doi.org/10.1016/j.acalib.2015.10.009   khoo, m., rozaklis, l., & hall, c. (2012). a survey of the use of ethnographic methods in the study of libraries and library users. library & information science research, 34(2), 82–91. https://doi.org/10.1016/j.lisr.2011.07.010   khoo, m., rozaklis, l., hall, c., & kusunoki, d. (2017). “a really nice spot”: evaluating place, space, and technology in academic libraries. college & research libraries, 77(1), 51-70. https://doi.org/10.5860/crl.77.1.51   kitchenham, a. d. (2010). mixed methods in case study research. in a. j. mills, g. durepos, &   e. wiebe (eds.), encyclopedia of case study research. los angeles, calif.: sage.   lippincott, j. k. (2012). information commons: meeting millennials’ needs. journal of library administration, 50(1), 27–37. http://dx.doi.org/10.1080/01930826.2012.707950   may, f., & swabey, a. (2015). using and experiencing the academic library: a multisite observational study of space and place. college & research libraries, 76(6), 771-795. https://doi.org/10.5860/crl.76.6.771   mccaffrey, c., & breen, m. (2016). quiet in the library: an evidence-based approach to improving the student experience. portal: libraries and the academy, 16(4), 775–791. https://doi.org/10.1353/pla.2016.0052   mccrary, q. d. (2017). small library research: using qualitative and user-oriented research to transform a traditional library into an information commons. evidence based library and information practice, 12(1), 34–49. http://dx.doi.org/10.18438/b8863f   montgomery, s. e. (2014). library space assessment: user learning behaviors in the library. journal of academic librarianship, 40(1), 70–75. https://doi.org/10.1016/j.acalib.2013.11.003   newcomer, n. l., lindahl, d., & harriman, s. a. (2016). picture the music: performing arts library planning with photo elicitation. music reference services quarterly, 19(1), 18. https://doi.org/10.1080/10588167.2015.1130575   oliveira, s. m. (2016). space preference at james white library: what students really want. journal of academic librarianship 42(4), 355–367. https://doi.org/10.1016/j.acalib.2016.05.009   paretta, l. t., & catalano, a. (2013). what students really do in the library: an observational study. reference librarian, 54(2), 157-167. https://doi.org/10.1080/02763877.2013.755033   regalado, m., & smale, m. a. (2015). “i am more productive in the library because it’s quiet”: commuter students in the college library. college & research libraries, 76(7), 899. https://doi.org/10.5860/crl.76.7.899   simons, h. (2009). case study research in practice. los angeles: sage.   suarez, d. (2007). what students do when they study in the library: using ethnographic methods to observe student behavior. electronic journal of academic & special librarianship, 8(3).   thomas, b., van horne, s., jacobson, w., & anson, m. (2015). the design and assessment of the learning commons at the university of iowa. the journal of academic librarianship, 41(6), 804–813. https://doi.org/10.1016/j.acalib.2015.09.005   washburn, a., & bibb, s. c. (2011). students studying students: an assessment of using undergraduate student researchers in an ethnographic study of library use. library & information research, 35(109), 55–66.   webb, k. m., schaller, m. a., & hunley, s. a. (2008). measuring library space use and preferences: charting a path toward increased engagement. portal: libraries and the academy, 8(4), 407-422. http://doi.org/10.1353/pla.0.0014   yoo-lee, e., lee, t. h., & velez, l. (2013). planning library spaces and services for millennials: an evidence-based approach. library management, 34(6), 498-511. https://doi.org/10.1108/lm-08-2012-0049     appendix a direct observation form   individual patron activity drop down menus   what day of the week is your shift? o    monday o    tuesday o    wednesday o    thursday o    friday o    saturday o    sunday   what time of the day is your shift? o    morning (9a-noon) o    afternoon (1-4p) o    evening (5-8pm) o    late night (after 9pm)   you are going to mark/record the activities of each person in your zone. fill out one form for each person. first, select your zone. then select the approximate area in the zone for the location of the person you wish to record.   are there unattended belongings in the space (belongings, but no person present)? o    yes o    no   what is the gender of the person? o    male o    female o    unknown o    n/agroup presentation room o    unattended belongings present   for observations in the group presentation rooms, complete one observation survey per room, not per person. which group presentation room technology is being used in the room? check all that apply. o    lcd o    wall talker o    camera o    laptop attached to camera   group presentation room record the total number of people in the room: ______ # females ______ # males   what is the person doing at the copier? o    scanning o    copying o    printing o    faxing o    none of the above or can't tell   is office equipment (hole punch, stapler, or paper cutter) being used? o    yes o    no   is dss software such as jaws, kurtzweil, or zoomtext being used? o    yes o    no   is the person using the scanner? o    yes o    no   is the person using the dss magnifier? o    yes o    no   what is the person doing at printer(s)? o    releasing a print job o    waiting for a print job o    having trouble with a print job o    none of the above   how long did the person use the walk-up computer? o    less than 1 minute o    1 -5 minutes o    5 -10 minutes o    > 10 minutes   record the person's level of collaboration with other people. o    no collaborationworking alone o    social interaction (appears not related to schoolwork) o    paired (working with another person on schoolwork) o    group (working with 2 or more others on schoolwork)   what percentage of the available space is the person or group using? ______ % of space   what is the seating arrangement of the pair? o    beside each other o    across from each other o    diagonal o    other ____________________   note the subject('s) technology use, if any. which of these devices are in use? check all that apply o    library computer o    laptop o    smart phone o    headphones o    e-reader o    tablet or notebook o    none o    other-specify ____________________   what was the purpose of using this library owned computer? if possible, please specify. o    library resource (catalog, database, e-book) ____________________ o    recreational (games, non-school related, social media) ____________________ o    email o    productivity software (microsoft office) ____________________ o    specialized software (such as matlab, spss, photoshop) ____________________ o    mylmuconnect ____________________ o    other/can't tell ____________________   record all the activities that you observe for this subject(s). o    browsing for/using lib book o    eating or drinking o    interacting w/ staff member o    printing o    reading, writing, or studying (non computer) o    sitting on floor / squatting o    sitting on library furniture o    sleeping o    standing o    talking o    using earplugs o    using electrical outlet o    using ethernet cable o    waiting in line o    walking (in transit) o    other ____________________   note any additional observations about the person's activities/belongings/seating or the space itself (e.g. use of extension cable, wearing a coat, frustration, confusion)   appendix b environment survey   what day of the week is your shift? o    monday o    tuesday o    wednesday o    thursday o    friday o    saturday o    sunday   what time of the day is your shift? o    morning (9a-noon) o    afternoon (1-4pm) o    evening (5-8pm) o    late night (after 9pm)   you will begin your shift by recording details about the general environment of your zone. first, select your zone. o    1 o    2 o    3 o    4 o    5   select the approximate area in zone 1 for the location of where you are standing. o    doorway of copy machine room o    gp 106 o    in between computers #89/ & #7   select the approximate area in zone 2 for the location of where you are standing. o    in front of leonardo printer o    between computer g & table   select the approximate area in zone 3 for the location of where you are standing. o    in front of walk-up computer #1 o    information commons desk   select the approximate area in zone 4 for the location of where you are standing. o    between table & computer 45 o    between couches & computer a   select the approximate area in zone 5 for the location of where you are standing. o    between computer 77 & computer 80 o    between computer 82 & reference stacks pn 1997-pq 6010 o    in front of computer 67   record the temperature level in your area (in degrees f). ______ degrees f   record the humidity range in your area (%). ______ %   record the noise level in your zone. ______ dba   is there any "out of order" equipment in your zone? o    none o    computers o    printers o    scanners o    lights o    photocopier   is there any equipment or furniture in your zone that is messy or unclean (e.g. spills, excessive trash)? o    yes o    no   please describe what was messy or unclean   appendix c patron survey please take a few minutes to fill out this brief survey so we can improve the information commons space! by completing this survey, you will have the chance to enter a raffle drawing to win a $50 amazon gift card (provided you give us your email address).  below is a consent form allowing us permission to use your anonymous feedback. no information that identifies you will be collected or released.    what activities are you engaging in today on the library's first floor? select all that apply. o    print o    study o    read o    eat/drink o    write o    get help from a staff member o    photocopy o    group work o    take a nap o    talk/socialize o    scan (scanner) o    use a reference book o    otherplease specify ____________________   which devices are you using within the library's first floor space today? select all that apply. include both devices supplied by the library and devices you bring with you. o    library computer o    laptop o    smart phone o    headphones o    e-reader o    tablet or notebook o    none o    otherplease specify ____________________   for what purpose(s) are you using the library computers today? select all that apply. o    library resources (library catalog, research databases, e-books) o    recreational (games, social media) o    email o    productivity software (microsoft office) o    specialized software (used in my schoo/college, such as matlab) ____________________ o    mylmuconnect o    otherplease specify ____________________   when you picked your spot on the first floor of the library today, which factors were most important to you? please choose the top five factors from the list on the left and drag & drop them into the “top five factors” box on the right in order of importance (#1 = most important).   top five factors ______ spacious (can spread out) ______ quiet ______ ambient or background noise ______ nice view ______ privacy ______ proximity to other students ______ bright lighting ______ comfortable seating ______ access to power outlets ______ comfortable temperature ______ access to a library computer ______ aesthetics (decor) ______ talking is tolerated ______ cleanliness   describe your level of collaboration with other people today on the first floor of the library. o    no collaborationi am working alone o    paired (working with another person) o    working with a group of 2 or more o    otherplease specify ____________________   what could be done to make the “information commons” (first floor of the library) a better space for you?   what is your gender? o    male o    female   what is your status at lmu? o    undergraduate student o    graduate student o    faculty/staff o    guest or other   is there anything you would like to add?   appendix d furniture survey (images available at https://lmu.box.com/v/furniturestudy)   these questions only pertain to the "information commons" space located on the first floor of the hannon library. this is the area you are currently sitting in. the space includes over 80 imac computers, printers, and the information desk. we need your feedback so we can improve the space. thanks for your input!   pick your preferred spot to use a library computer o    private o    open   which option(s) best explains why you chose this spot to use a library computer? o    privacy o    space for my stuff o    noise level o    open space / not closed off o    location (e.g. close to printer) o    collaboration   pick your favorite table and chair configuration o    4 chairs facing each other o    1 chair against the wall o    1 row facing the same direction o    2 chairs facing each other o    2 chairs against the wall   which option(s) best explains why you chose this table and chair configuration? conducive to studying / homework o    space for my stuff o    collaboration o    open space / not closed off o    privacy o    quiet   do you prefer a rectangular or round table? o    rectangular table o    round table   which option(s) best explains why you chose this table? o    space for my stuff o    comfort (e.g. easier on my legs) o    conducive to studying / homework o    collaboration   pick your ideal private space for working solo o    s-divider o    carrel o    space pod   which option(s) best explains why you chose this spot as your ideal private space for working alone? o    privacy / more closed off o    comfort o    writing surface (e.g. table) o    aesthetics / appearance o    compact (efficient use of space) o    space for my stuff o    noise level o    open space / less closed off   pick your ideal private space for collaborating enclosed booth o    high-back booth o    highback couch   which option(s) best explains why you chose this spot as your ideal space for collaborating? seating configuration (e.g. beside, across) o    comfort o    privacy / more closed off o    writing surface (e.g. table) o    space for my stuff o    noise level o    open space / less closed off   choose your favorite lounge furniture o    high back couch with coffee table o    armchair and coffee table o    reverse-c chair o    low back couch with coffee table o    barstool and counter   which option(s) best explains why you chose this furniture for a lounge area? comfort o    aesthetics / appearance o    conducive to work / study o    collaboration or socializing o    seating configuration (e.g. beside, across) o    space for my stuff o    writing surface (e.g. table) o    privacy   where would you prefer to take a phone call in the information commons? soundproof phone booth o    move to another area in the library o    group study room o    computer carrel o    open study area   which option(s) best explains why you chose this spot to take a phone call? o    privacy o    quiet o    open space / not closed off o    space for my stuff     appendix e heat map of people observed in each area       appendix f tableau visualization of activity in each zone     appendix g top comments from patron survey and whiteboard polls category # times details noise/too loud 32 designate a “no talking” and “no cell phones” area furniture 28 napping pods (9), newer furniture, more desk space, more comfort library desktop computer station 25 more computers, kick out people not using the computer, cleaner, more specialized software (7), mice that work better (6) food & drink 24 snacks/vending machines, water bottle refilling station, designated food zone miscellaneous space suggestions 15 charging station, plants, 3d printer, etc. temperature/too cold 9 too cold general cleanliness 8 less food and trash printers 7 spread out to more locations, improve functionality, pay without onecard hours 5 open 24/7 lighting 4 brighter       evidence based library and information practice editorial responsibilities      2018. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     editor-in-chief: lorie kloda   associate editor (evidence summaries): carol perryman   associate editor (research articles): ann medaille, lisl zach   associate editor (classics & reviews): christina wissinger   associate editor (commentaries, using evidence in practice): lorie kloda   communications officer (news): tatiana bryant   production editor: michelle dunaway   editorial intern: rachel hinrichs   editorial advisors: lindsay alcock, alison brettle, jonathan eldredge, denise koufogiannakis   copyeditors: richard hayman (lead copyeditor), georgianne bordner, julie evener, heather healy, julie james, jane morgan-daniel, stacey penney, elizabeth stregger, nikki tummon, elaina vitale, alison yeoman   indexing support: kate shore   art_panigrahi_29976 research article   re-purposing the physical space of an academic library in the digital era: a case study of jawaharlal nehru university and delhi university, new delhi (india)   sushanta kumar panigrahi assistant librarian dr. b r ambedkar central library jawaharlal nehru university new delhi, delhi, india email: sushant@mail.jnu.ac.in   prabhat kumar sangal assistant professor indira gandhi national open university new delhi, delhi, india email: prabhat.sangal@ignou.ac.in   received: 27 may 2021                                                              accepted: 6 jan. 2023      2023 panigrahi and sangal. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29976     abstract   objectives – the issue/return of books and other reading materials available on shelves in academic libraries has declined. round-the-clock availability of information resources, high-speed internet, cheaper cost of data download, laptops, and smartphones are some of the reasons behind decreased usage of print reading materials. users are spending more time studying, accessing e-resources, socializing with friends, and discussing with peer groups in the library building. libraries in developed countries have already recognized the need of the hour and redesigned their spaces to create a variety of reading spaces, creative spaces, quiet spaces, and so forth in the existing building. but the libraries of the higher education institutions in india have not considered the users' needs concerning the library space. no such study has been conducted at large universities in india. this study was conducted with library users of the central library of two large universities located in new delhi to find users' opinions and views to re-purpose the library space to meet users' needs.   methods – the survey method has been used to understand the use of the existing format of knowledge resources and the need of library users. the central point of the research questionnaire is to optimize the library space of the central libraries of the universities under study. printed questionnaires were distributed to the users present in the reading halls of the libraries at jawaharlal nehru university (jnu), new delhi, and delhi university (du), new delhi, and online questionnaires were distributed to faculty members. the collected data were analyzed with microsoft excel, and various hypotheses were tested using non-parametric tests such as the sign test, mann-whitney u test, and kruskal-wallis test.   results – students and research scholars visit daily and spend more than four hours in the library building. they use their time in study, accessing e-resources in the reading halls of the library more than any other place. the users opined the reading halls should have wi-fi facilities, a power source to charge laptops and mobile phones, washrooms near every reading hall, a quiet reading space, and a clean ambient environment for long hours of study in the library building.   conclusion – library users in the 21st century want more physical space to study, concentrate, socialize, and learn in the informal learning environment. they need library space with the latest infrastructure to connect to the digital world to retrieve study materials and print copies of study materials. educational institutions should create more varied reading spaces for serious reading, pleasure reading, in-depth reading, and interactive learning spaces in new library buildings.     introduction   every profession operates according to certain principles. libraries operate on the principle of the five laws of library science as defined by dr. s.r. ranganathan (1931):   first law: books are for use. second law: every book has its reader. third law: every reader has his/her books. fourth law: save the time of the reader. fifth law: the library is a growing organism.   zabel (2005) explained that all five laws are still relevant in the digital age. zabel advocated that the fifth law of library science, "the library is a growing organism," implies that libraries should always change their space (2005, p.24). many libraries are changing their spaces to information commons. information commons means a space in the library building where library staff provide information services, and where library users can visit and receive help from library staff with regard to basic computer knowledge, electronic resources, and other tools.   over a period of time, the collection reaches the physical capacity of the academic library. to manage the collection within limited physical space, libraries adopt the policy of weeding out the least used materials from the collection. the objectives of the library are to collect, organize, store, and disseminate knowledge resources. but, in the last two decades, libraries are witnessing a revolutionary change in the availability and accessibility of knowledge resources. information is ubiquitous. nowadays library users can access information through their mobile phones, ipads, laptops, and other handheld electronic equipment anytime and anywhere around the clock. sources such as project gutenberg, internet archive, open library, google books, hathitrust digital library, ndltd (networked digital library of theses and dissertations), google scholar, and others are the platforms from which library users are accessing many electronic books and scholarly electronic journals for their academic use. besides this, the libraries of higher education institutions in india, like universities and technical institutions, are subscribing to many online databases of journals and e-books for their users through library consortia and from their funds.   users can access academic resources around the clock across campus, barring the restriction of access to knowledge resources from 9 a.m. to 6 p.m. only as in the case of a traditional library. there is no need to visit the library building to access electronic resources. the reading contents of the library are now replaced with e-resources, leading to fewer visits from library users in the physical library and lower circulation statistics.   the deserted libraries of the nineties are now the happening places on campuses.  libraries are converting themselves into learning resource centres. the role of the librarian has also changed from custodian of books to information facilitator. this study is an attempt to explore the usage of the library space including the reading halls of two major universities of india located in new delhi. the output of the study may be implemented in other academic libraries as the type of collection and information-seeking behaviour of users in academic libraries across the world are similar.   literature review   this literature review presents the developments, challenges, and issues related to renovation of the physical space in libraries across the world, as discussed in literature published after 2008. the investigators primarily focused on literature addressing how the usage of e-resources has impacted the use of library space, how libraries redesigned their reading space to meet users’ needs, and whether physical visits by library users increased after the renovation of physical libraries throughout the world.   musoke (2008) and kathleen et al. (2019) studied users' information-seeking behavior at the universities of makerere and dayton. according to them, the library is the largest non-classroom and non-residential building on the campus, where students concentrate in a non-distracting and pleasant environment. the library provides comfort and a quiet reading space for study. many users prefer to study alone rather than in a group. academic library buildings are now repurposing their reading space as learning spaces, collaborative spaces, group study rooms, and instructional spaces to attract more students. freeman (2005) observed, “the library is the only centralized location where new and emerging information technologies can be combined with traditional knowledge resources in a user-focused, service-rich environment that supports today’s social and educational patterns of learning, teaching, and research” (2005, p. 4). applegate (2009) wrote, “the library is a campus space, one uniquely suited to meet important student needs for space as well as services and resources. where will students do what they need to do? where will they read? plug in their laptops? eat and drink? talk with their classmates and friends—and not have to listen to other people talking? have a place to concentrate in quiet?” (2009, p. 345). gayton (2008) argues that there is a decline in the circulation of print books, reduced number of reference materials, and falling gate counts in the academic library. there is a demand for café space, group study rooms, and information commons in the library. applegate (2009), gibson and kaplan (2017), hall and kapa (2015), hillman (2017) and norton et al. (2013) reported that students have diverse desires for library spaces, such as quiet reading spaces, group study rooms, research scholars’ carrels, and coffee shops. many visit the library to use computers and printers. after the renovation of the physical space, the gate counts increased by 15% (hilman, 2017). hall and kapa (2015) found that 84% of users visit the library to use the quiet space.   cheong et al. (2016), hegde et al. (2018), ozburn et al. (2020), and stemmer et al. (2019) concluded that library users need various types of space during different times of day and periods of the year. based on their need, spaces can be categorized into four types, namely collaborative space, sanctuary space, interaction space, and community space. collaborative spaces are being developed for different types of intellectual work, presentation practice, discussion, social interactions, and peer learning; these spaces can be noisy. the sanctuary space is for quiet reading. andrews et al. (2016) observed users prefer large-screen lcd, headphones, screen-sharing software, couches, bean chairs, chairs with mobility in reading halls, and 3d printers. spencer & watstein, (2017) found users need noise acoustic-designed carpeted floors in reading areas. hin et al. (2018) explained that libraries are now changing into proactive learning centres and cultural centres of universities. chaddha and kanjilal (2022) concluded that library users are not satisfied with library space and time of operation, but with a few modifications of the existing library building and incorporation of new furniture, small academic libraries can create information commons that will satisfy users.   it is evident from this literature review that libraries in the western part of the world have renovated their physical space to welcome the library users to spend more time and engage themselves in study and discussion. they have refurbished library spaces to meet the different reading needs of users like group study rooms, individual study rooms, quiet study rooms, discussion rooms with whiteboards, big screens for presentations, and more. the literature also showed increases in the post-renovation occupancy of library reading space and gate count. authors all report that users need quiet space and different types of study space for different purposes. the role of the physical library has changed from physical collections to informal learning centres where users can concentrate, study, socialize with friends, discuss academic matters, access e-resources, and others. users have varied learning behavior, and space assessment and redesign are done according to their needs.    literature shows that library users prefer a variety of reading spaces in the library to use at different times of the day and year, and usage of reading space increased when interior renovations took place. a 2022 study conducted in india reveals that library users need quiet space and collaborative space where information experts can help them in using information technology to enhance their learning (chadda & kanjilal, 2022). however, no study has been conducted in india to assess the academic library space and its usage after the introduction of e-resources in large academic libraries. this study aims to find out users' needs concerning reading space and optimization of existing reading halls of libraries of large universities in new delhi, india.   aims   indian universities are opening more schools of study to meet the needs and demands of the society and market. as a result, more faculty members and students are joining universities. due to lack of funds, academic libraries are not expanding their buildings for storage. print collections grow with time, with archive issues of research journals, theses, and dissertations occupying significant shelf space. there is no standard weeding policy to shift the least used materials from library buildings, and there is no annex to store obsolete materials.   meanwhile the increased number of physical users requires more seating space in reading halls to study and access the e-resources through their devices, while libraries have not added any more space in the existing building since establishment.   objectives   the objective of this study is to bridge the expectation of users concerning the collections and infrastructure of libraries. this study will assist librarians in developing and south asian countries in redesigning their libraries to optimize the physical space and other resources to meet users' satisfaction and the mandate of the university.   research questions   the research will attempt to answer the following questions:   rq1. when e-resources are available anywhere and anytime, why do users visit the library building?   rq2. how do we optimize the existing physical space of the library building by weeding print journals and theses or shifting them to a remote location to accommodate more library users?   rq3. which locations do library users visit during their stay in the library building, and how much time in a day do they spend in the library?   rq4. what infrastructure is required in the library reading space for various activities such as serious reading, leisure reading, quiet study, group study, and socializing with friends?   hypotheses   hypothesis testing is the use of statistics to test whether a given hypothesis is true or not. in this study, we formulate the following sets of hypotheses:   null hypothesis (h01): the visitors are not interested in the physical space being optimized with the latest tools and infrastructure.   alternative hypothesis (h11): the visitors are interested in the physical space being optimized with the latest tools and infrastructure.   null hypothesis (h02): there are no significant differences among user groups in their opinions regarding whether to optimize the library space.   alternative hypothesis (h12): there are significant differences among user groups in their opinions regarding whether to optimize the library space.   research methodology   study participants   the present study selected two central universities, jawaharlal nehru university (jnu) and delhi university (du), both located in new delhi, india. participants of the study are the undergraduate/postgraduate (ug/pg) students, research scholars, and faculty members of these universities. jnu is a single campus university, whereas du has two campuses: north campus and south campus. du has 91 constituent colleges. jnu has granted recognition and accreditation to over 18 defence and research and development institutions.   jawaharlal nehru university (jnu) comprises 13 schools, 55 centres, and 7 special centres. the university imparts education at ug, pg, m. phil., ph.d., and post-doctoral levels. recently the university has added management and engineering courses. there are around 4000 registered library users. average library traffic is around 500-800 users per day. the footfall increases during new admission and term-end examination (jnu, annual report, 2018).   delhi university (du), one of the largest universities in india, has 87 departments, 16 centres, and 90 colleges. each college or centre has its own library. the present study surveyed the users present in the central library during the researchers’ visit. the researchers visited the library for 3 days consecutively during the mid-term. the central library of du has around 4200 registered users and the daily footfall varies from 400-800 users per day. (du, annual report, 2018).   the sample population of the study includes ug/pg students, research scholars, and faculty members of the university who are registered as library users in the central library of the north campus of du and the central library of jnu. although faculty members rarely visit the library, their responses as stakeholders of the library are also important in the study. here faculty members are those engaged in teaching and research activities in the discipline, whereas the research scholars are the students who are enrolled as m. phil. (master of philosophy) or ph.d. (doctorate of philosophy) students.   sampling technique   it is practically impossible to collect the data from all registered library users therefore the investigators collected the data from the group of users who were easy to contact or easy to reach (convenience sampling). the central point of the study is related to optimization of the physical space of the library in the present era, and therefore the views and opinions of the library users who visit the library make more sense than casual users or students who hardly visit the library. the investigators visited the central library of jnu and the central library of the north campus of du consecutively for 3 days from 10 a.m. to 7 p.m. and distributed printed questionnaires to the library users present in the library premises. an online questionnaire, designed in google forms, was sent to faculty members whose e-mail addresses were available on the university faculty directory website.   instrument and procedures   a questionnaire was used as the instrument for data collection, and microsoft excel was used to analyze the data. the questionnaire (appendix a) was designed by the investigators based on the literature review and the local condition of the libraries; it consists of demographic variables, frequency of visit, preference of format of reading materials, preference of physical space, weeding of least used reading materials, and preference of infrastructure in the physical space. statements are rated based on users' preferences like "yes" and "no." for opinions about the optimization of physical space in the library, 4-point scales were used with ratings ranging from 1 ("can't say") to 4 ("must"). rank 4 is highest, meaning the visitor significantly wants to optimize the physical space and the library should redesign and develop the physical space into a learner-centric library.   at the pilot level, the questionnaire was distributed among 30 respondents, both library users and library staff, to collect their suggestions and views about any ambiguity and clarity in the questionnaire. then the questionnaire was modified according to feedback received during the pilot study. the questionnaire has both open and closed-ended questions. with each question, space was provided for respondents to give suggestions.   data collection method   the investigators visited the central library of jnu and the central library of the north campus of du consecutively for 3 days from 10 a.m. to 7 p.m. they distributed printed questionnaires to the library users present in the library premises. an online questionnaire, designed in google forms, was sent to the faculty members whose e-mail addresses were available on the university faculty directory website.    data analysis tool used   microsoft excel was used to code the ordinal data and numerical calculations. most data collected was in the form of an ordinal scale, so median and quartile deviation (qd) was used to summarize the data, and non-parametric tests such as the sign test, mann-whitney u test, and kruskal-wallis test were used to test the different types of hypotheses.   results   the printed questionnaires were distributed to ug/pg students and research scholars and emailed to the faculty members; the numbers of responses are summarized in table 1.     table 1 responses to the distributed questionnaires name of the university no. of the questionnaires distributed responses of the ug/pg students (% of response) responses of the research scholars (% of response) responses of the faculty members (% of response) total jnu 1600 220/500 (44%) 220/500 (44%) 211/600 (35%) 651/1600 (40.68%) du 1600 100/500 (20%) 205/500 (41%) 167/600 (27%) 472/1600 (29.5%) total 3200 320/1000 (32%) 425/1000 (42.5%) 378/1200 (31.5%) 1123/3200 (35.09%)     from table 1, we observe the response rate of ug/pg students is 32%, research scholars 42.5%, and faculty members 31.5%.     frequency of visits   ug/pg students, research scholars, and faculty members visit their library when they need to collect/refer to books, journals, and other reference materials from the library. the frequency of visits varies among different types of users. based on the responses received, the survey shows that more than 72% of 320 ug/pg students and 71% of 425 research scholars visit the library daily, versus only 4% of 378 faculty members. twenty percent and 19% of the faculty members visit once a month and once in a quarter. twenty-three percent of the ug/pg students and 13% of research scholars visit 2-3 times a week. eighteen percent of faculty members visit 2-3 times a week. twenty-eight percent of the faculty members did not respond. figure 1 shows the frequency of visits to the libraries in jnu and du.     figure 1 frequency of library visits.     time of stay in the library building   more than 60% of the ug/pg students and research scholars spend more than four hours in the library building whenever they visit the library. only 2% of the faculty members spend more than four hours. figure 2 presents the time in hours stayed by the visitors in the library building.     figure 2 time of stay in the library building.     library premises visited   eighty percent of ug/pg students and 82% of research scholars visit the reading hall of the library most of the time, where they read, write, access e-resources, prepare for exams, prepare notes, and other academic purposes. they visit other areas like book stacks, reference collections, periodical sections, bound volumes, and ph.d. theses “sometimes.” thirty-four percent of faculty members also visit the reading hall of the library most of the time. figure 3 indicates the areas of the library visited by the visitors most of the time.     areading hall with tables, chairs, and charging points for laptops, bstack areas in which books and textbooks are stocked, creference collection section, dperiodical section, ebound volume/ theses/ dissertation section   figure 3 premises visited by library visitors in the building.     purpose of visit   figure 4 shows the purpose of the visitors visiting the library. fifty-six percent of ug/pg students and 50% of research scholars visit to study in the reading hall more than any other purposes. twenty-six percent of faculty members visit the library to use the reading hall, and 30% of them visit to browse print journals and magazines. lower percentages of all user groups visit to reference back volumes of print journals and theses.     aissue/return of books; bstudy in reading hall; cto access e-resources using desktops; dto access e-resources using a personal laptop; ebrowsing print journals & magazines; fto refer back vol journals and ph.d. theses.   figure 4 purpose of visit to the library by visitors (most of the time).     re-purposing the reading space   data was analysed to compare visitors’ opinions of the libraries concerning the optimization of physical space. since the data is collected on the ordinal scale, the investigators have calculated the descriptive statistics median and quartile deviation in the following table.     table 2 redesign library interior for a variety of reading spaces (overall views) s. no. attributes can’t say not required desirable must total median qd 1 2 3 4 a large table study room with a whiteboard 42 224 420 437 1123 3 1 4% 20% 37% 39% b group study room with large table, whiteboard, and led screen for presentation practice, seminar talk with the peer group, etc. 41 182 474 426 1123 3 1 4% 16% 42% 38% c makers space with facilities like 3d printers for creativity, video and audio capture, large format printing, art-related workshops, music recording spaces, and so forth 100 212 468 343 1123 3 2 9% 19% 42% 31% d smart classroom with movable chairs, cozy bean chair, interactive screen, whiteboard, interactive digital wall, and projector 73 237 424 389 1123 3 2 7% 21% 38% 35% e standing workstation 150 151 479 343 1123 3 2 13% 13% 43% 31% f carrels for research scholars 115 55 424 529 1123 3 1 10% 5% 38% 47% g faculty study room with wi-fi facility 69 48 322 684 1123 4 1 6% 4% 29% 61% h quiet study room 7 29 193 894 1123 4 0 1% 3% 17% 80% i non-silent zone for attending mobile calls and socializing with friends 71 320 338 394 1123 3 2 6% 28% 30% 35% j open space: a variety of study spaces accommodating individual, small group, and large group study 36 115 554 418 1123 3 1 3% 10% 49% 37% k seminar room with 10-20 people seating capacity with whiteboard, chalkboard, and presentation display 69 137 517 400 1123 3 1 6% 12% 46% 36% l web conference room with polycom conference phone and technology 110 191 446 376 1123 3 2 10% 17% 40% 33% m presentation room with the microphone 100 210 435 378 1123 3 2 9% 19% 39% 34% n living room seating with chairs and small tables with large monitors for laptop hook up 92 200 469 362 1123 3 2 8% 18% 42% 32% o coffee dispensing machine near every reading hall 71 238 358 456 1123 3 2 6% 21% 32% 41% p small table with a chair near a glass window for natural light reading 53 100 473 497 1123 3 1 5% 9% 42% 44%   overall 7% 15% 38% 41%   3.125 1.43     table 2 reveals that 7% of all visitors (including ug/pg students, research scholars, and faculty members), opined “cannot say” about the redesign of the library interior for a variety of reading spaces, 15% of them said that these were “not required,” while 79% of them suggested optimizing the physical space of the library with latest tools and infrastructure. the overall average score of all visitors about the optimization of the physical space is 3.125 which represents the rating between "desirable" and "must." hence, this study indicates that visitors wish to optimize the physical space of the library to be learner-centric.   to validate the results, hypothesis testing was used. a hypothesis was formulated that visitors are interested in the physical space being optimized with the latest tools and infrastructure. scores of 3 and 4 (desirable and must) would represent that visitors are interested in optimizing the physical space; therefore, we test whether the overall average is 3 or more. since the data is on an ordinal scale, the non-parametric sign test is used instead of the one-sample t-test.   the results of the test presented in table 3 indicate the p-value (0.022) is less than the level of significance (0.05); therefore, the null hypothesis is rejected, and the researchers conclude that visitors are interested in the physical space being optimized with the latest tools and infrastructure.      table 3 results of one-sample sign test groups sample size test statistics conclusion all visitors 1123 s 123 p< 0.05 z –1.144 p 0.022 significant                 table 4 results of the kruskal wallis test to redesign library interior for a variety of reading spaces attributes groups sample size sum of rank test statistics conclusion to optimize the physical space, the library should redesign and develop physical space with the latest tools and infrastructure to meet the expectations of users to convert into a learner centric library research scholars 425 227118.00 h 11.16 p< 0.05 ug/pg 320 174582.00 df 2.00 faculty 378 229426.00 tabulated value 5.99   total 1123 p 0.003 significance     the second null hypothesis states that there are no significant differences among the views of visitor groups. since there are three groups and the data is in the form of an ordinal scale, the non-parametric kruskal wallis test was used instead of anova. the results of the test are given in table 4.   since the p-value (0.003) is less than the level of significance (0.05), the samples provide sufficient evidence against the null hypothesis, and the null hypothesis is rejected. hence, the study reveals significant differences among visitors’ opinions of optimizing the library’s physical space. a pair-wise comparison study is done between the ug/pg students and research scholars, research scholars and faculty members, and ug/pg students and faculty members.   to check which group is significantly different from the others, a diagram was prepared between the opinions of the ug/pg students, research scholars, and faculty members, shown in figure 5.     athe suggestion has been incorporated earlier; bgroup study room with large table, whiteboard, and led screen for presentation practice, seminar talk with the peer group, etc.; cmakers space with facilities like 3d printers for creativity, video and audio capture, large format printing, art-related workshops, music recording spaces, and so forth; dsmart classroom with movable chairs, cozy bean chair, interactive screen, whiteboard, interactive digital wall, and projector; estanding work station; ffaculty study room with wi-fi facility; gquiet study room; hnon-silent zone for attending mobile calls, and socializing with friends; inon-silent zone for attending mobile calls, and socializing with friends; jopen space: a variety of study spaces accommodating individual, small group ,and large group study; kseminar room with 10-20 people seating capacity with whiteboard, chalkboard, and presentation display; lweb conference room with polycom conference phone and technology; mpresentation room with the microphone; nliving room seating with chairs and small tables with large monitors for laptop hook up; ocoffee dispensing machine near every reading hall; psmall table with a chair near a glass window for natural light reading   figure 5 views of visitors about redesigning library interior for a variety of reading spaces.     table 5 results of mann-whitney u test to redesign library interior for a variety of reading spaces groups sample size sum of rank test statistics conclusion ug/pg students 320 162073 u 64452 p> 0.05 research scholars 425 115812 z -1.2202 total 745 p 0.1112 insignificant ug/pg students 320 163593.5 u 73068.5 p< 0.05 faculty members 378 159212.5 z -2.212 total 698 p 0.0135 significant research scholars 425 1029405 u 51580.5 p< 0.05 faculty members 378 141010.5 z -3.352 total 803 p 0.0004 significant     from figure 5, it is observed that the opinion of ug/pg students (73%), research scholars (75%), and faculty members (83%) is almost the same about the quiet study room whereas for other items, the opinions of ug/pg students and research scholars are similar but differ from faculty opinions. to validate the results, the mann-whitney u test was used. table 5 presents the results of the test.   based on the above table, no significant difference exists between the opinions of the ug/pg students and research scholars, whereas there is a significant difference between the opinions of ug/pg students and faculty members, as well as between research scholars and faculty members.   ug/pg students and research scholars' opinions are similar, as might be expected, because they spend a considerable part of their daytime in the library building for study, learning, and informal discussions with their peer group. they use the library space to access subscription e-resources, read important materials, use reference sources, and concentrate alone. they desire a comfortable setting with modern amenities so they may study for a longer period. they want various types of reading space, such as quiet study, group study, presentation room, discussion room, and research scholar’s carrels, which are either not available or few in number.   the views of faculty members vary from that of ug/pg students and research scholars. they rarely visit the library, and only for a short time to access print journals or issue/return books, so their perception of reading space is different from that of frequent users.     table 6 duplicity of availability and less readership of print resources (overall views) attributes strongly disagree disagree neutral agree strongly agree total median qd 1 2 3 4 5 these resources are available online in jstor, portico, and sodhganga (reservoir of ph.d. theses) at inflibnet 12 98 299 384 330 1123 4 2 1% 9% 27% 34% 29% there is less readership of these print resources and storing the hardcopies is a wastes important library space 85 204 333 324 177 1123 4 2 8% 18% 29% 30% 16%     the calculated value of the median for the ug/pg students, research scholars, and faculty members is 4 for both attributes, “duplicity of availability” and “less readership of print resources.” the results show that the respondents “agree” with the reason that these resources are available in jstor and sodhganga and there is less readership of these resources.   discussion   it is evident that the ug/pg students and research scholars frequently visit the library, whereas faculty members hardly go to the library. the reason for fewer visits by faculty members may be less dependency on information resources and being busy with other academic as well as administrative work of the university. they may also either use the faculty reading room for accessing e-resources or reading books/magazines.   more than 60% of the ug/pg students and research scholars spend more than 4 hours per visit in the library building, which indicates that, after classroom activities, students and researchers spend time in the library building for reading, research work, preparation of exams, and more. however, only 2% of faculty members spend more than four hours.   during their stay in the library building, most of the ug/pg students and research scholars use the reading hall. they sometimes visit the book stacks, bound periodicals sections, reference collection, and the theses/dissertations section. a small number of faculty members spend most of their time in the library reading room. it can be deduced that the reading hall is the most preferred space among all areas of the library building.   the study also shows that the perception of faculty members about the usage of reading space is different from the ug/pg students and research scholars. this may be because the faculty members visit the library only rarely or infrequently during a year. they may use the faculty reading room for accessing e-resources or reading books/magazines, while students and research scholars visit the library regularly and frequently. the study shows that most visitors are interested in optimizing the physical space. hence the library should redesign and develop the physical space with the latest tools and infrastructure to meet the expectations of users.    the study also shows that all user groups do not agree that archives of research journals and ph.d. theses are available online in jstor, portico, and sodhganga (repository of the digital copy of ph.d. theses) at inflibnet respectively. again, all users’ groups do not agree that “there is less readership of these print resources and storing the hardcopies is wastage of valuable library space.” some reasons for this may be: users may worry they will not be able to get articles in the future if online subscriptions are cancelled; or users do not know whether all ph.d. theses are uploaded to the sodhganga platform.   the users also suggested the following in response to our open-ended questions:   ·        “very quiet reading hall is a must” ·        “library should indeed be modernized but it must be maintained” ·        “space for reading hall is very less. there are many sections which are not used by many students. if these sections are converted to reading halls it will be useful” ·        “research carrels for all ph.d. research students is a must” ·        “the library foremost needs online access to all major journals” ·        “i can see that you have brilliant ideas in making library more advanced and suitable for the next generation need. despite the fact, we are moving towards the digital resources encourage soft study material” ·        “small & smart furniture for reception /issuing counter and also for book stacks” ·        “use e-resources more to eliminate hard redundant material” ·        “to weed out very old, worn-out books but at the same time at least two copies of classics should be retained” ·        “large empty spaces such as entrance area etc. can be lined with books” ·        “our library space and collection seem outdated, the reason i visit less frequently. needs urgent revamp”   it is expected nowadays that, when information resources are available at all times through laptops, desktops, and mobile phones, there may be less user demand for reading space in the library. but the study reveals there is more demand for reading space with a variety of spaces for collaborative study, group study, discussion rooms, and presentation rooms with web cameras, conference phones, and microphones. users want carrels for quiet study and concentration. for collaborative study, project-based learning users want 3d printers and other electronic gadgets which they cannot afford to buy. users need more power sources for charging laptops and mobile phones for battery power backup for long hours of study.   students like to concentrate on their project work, research writings, preparation for exams, and discussions with peers in the library, but learning does not only take place alone. it also requires discussions and sharing of ideas with peers and senior students. this study reveals that the ug/pg students and research scholars often spend more than four hours in the reading halls for informal learning from reading materials, discussions, academic discussions, and calm study. the libraries should renovate their reading areas with facilities like centralized air conditioning, power sources for recharging laptop batteries and cell phone batteries, and ergonomically built couches for extended study and concentration sessions.     library staff of the universities should regularly assess the print collection and weed out the damaged books regularly. library staff should replace damaged copies with new ones, as the damage may be indicative of high demand and overuse. these books are used by many visitors who have marked, commented, and highlighted important notes, and text in the books. in such a case, a new copy of the book should be ordered. if a title is not available in the market, it can be digitized for online access if permitted by law.   limitations   the study has collected data on the usage of space, reading materials, and time spent in the library by users through the questionnaire. to verify the validity of the responses, an observational study could have been performed, but was not possible due to time constraints and other factors.   conclusions   libraries built in the 20th century had the architectural design to accommodate ever-growing print collections. there was no concept of e-books, e-journals, or online databases.  seating arrangements were planned in reading halls, and the main emphasis was on bookshelves to stock more print material. in the last two decades, the format of reference sources and journals has changed from print to online, accessible around the clock. the number of library users has also increased, but library space remains unchanged, and funding is not available to expand spaces.   although users can access online resources anywhere, they still want to study, concentrate, socialize, and learn in the informal learning environment. users require quiet space, group study rooms, presentation rooms, wi-fi, power sources, windows for natural light, and research carrels.   new educational institutions should keep in mind the creation of varied reading spaces for serious reading, pleasure reading, long hour reading, and interactive learning spaces in new library buildings. while developing new or renovating existing reading space, librarians and administrations should consult library users for their preferences. over time, the availability and format of knowledge resources have changed. to keep library buildings relevant, institutions need to update the internal spaces as well. the library building is a key non-residential and non-formal learning space on a university campus. the library building should have the latest information technology and other infrastructure related to reading halls to create an ecosystem for learning and study. hence, there is a need to re-purpose the existing library reading space not only to access online resources but also for creativity and learning.   further research may be conducted such as a focus group study or ethnographic study to understand what users do in the library during their stay, what they do in the reading hall, for what purpose they use the reading halls, how they use the reading halls, and occupancy level of reading halls during a different time of the day and different time of the year. these studies will give an idea of human resource and infrastructure management for the library to better serve users.   although the study was conducted in two large libraries in india, the output of the research may be applied to many academic libraries in developing countries that are built in the last century. there is a need to refurbish reading halls to meet users' expectations, as now the libraries are treated as “the informal learning space where the users not only learn from the reading materials available on the shelves but also learn from their peers in the reading halls” (choy & goh, 2016).   academic libraries all over the world should update their reading areas to attract more users and increase traffic to the structure. to satisfy library users, academic libraries should reconsider turning their collection-centric buildings into learner-centric spaces and should offer a range of reading areas.   references   andrews, c., downs, a., morris-knower, j., pacion, k., & wright, s. e. (2016). from "library as place” to “library as platform”: redesigning the 21st century academic library. 145–167. https://doi.org/10.1108/s0732-067120160000036006   applegate, r. (2009). the library is for studying: student preferences for study space. the journal of academic librarianship, 35(4), 341–346. https://doi.org/10.1016/j.acalib.2009.04.004   chaddha, k & kanjilal, u. (2022). transforming academic libraries into information commons: a proposed model. evidence based library and information practice. 17(1), 6-33. https://doi.org/10.18438/eblip30004   choy, f. c., & goh, s. n. (2016). a framework for planning academic library spaces. library management, 37(1/2), 13–28. https://doi.org/10.1108/lm-01-2016-0001   freeman, g. t. (2017, january 4). the library as place: changes in learning patterns, collections, technology, and use. clir. retrieved september 5, 2022, from https://www.clir.org/pubs/reports/pub129/freeman/   gayton, j. t. (2008). academic libraries: “social” or “communal?” the nature and future of academic libraries. the journal of academic librarianship, 34(1), 60–66. https://doi.org/10.1016/j.acalib.2007.11.011   gibson, a. n., & kaplan, s. (2017). place, community and information behavior: spatially oriented information seeking zones and information source preferences. library & information science research, 39(2), 131–139. https://doi.org/10.1016/j.lisr.2017.03.001   hall, k., & kapa, d. (2015). silent and independent: student use of academic library study space. partnership: the canadian journal of library and information practice and research, 10(1). https://doi.org/10.21083/partnership.v10i1.3338   hegde, a., boucher, p., & lavelle, a. (2018). how do you work? understanding user needs for responsive study space design. college & research libraries, 79(7), 895–915. https://doi.org/10.5860/crl.79.7.895   webb, k.m., schaller, m.a., & hunley, s.a. (2008). measuring library space use and preferences: charting a path toward increased engagement. portal: libraries and the academy 8(4), 407-422. http://doi.org/10.1353/pla.0.0014   li, l. h., wu, f., & su, b. (2018). impacts of library space on learning satisfaction – an empirical study of university library design in guangzhou, china. the journal of academic librarianship, 44(6), 724–737. https://doi.org/10.1016/j.acalib.2018.10.003   musoke, m. g. n. (2008). strategies for addressing the university library users’ changing needs and practices in sub-saharan africa. the journal of academic librarianship, 34(6), 532–538. https://doi.org/10.1016/j.acalib.2008.10.002   norton, h. f., butson, l. c., tennant, m. r., & botero, c. e. (2013). space planning: a renovation saga involving library users. medical reference services quarterly, 32(2), 133–150. https://doi.org/10.1080/02763869.2013.776879   ozburn, l., kirk, j., & eastman, t. (2020). creating collaborative library spaces through partnerships with campus organizations. journal of library administration, 60(6), 600–612. https://doi.org/10.1080/01930826.2020.1748432   ranganathan, s. r. (1931). the five laws of library science. madras library association (madras, india) and edward goldston (london, uk). http://hdl.handle.net/10150/105454   spencer, m. e. & watstein, s. b. (2017). academic library spaces: advancing student success and helping students thrive. portal: libraries and the academy, 17(2), 389–402. https://doi.org/10.1353/pla.2017.0024   stemmer, j., & strawser, m. g. (2019). silence in a noisy world: using student feedback to enhance library silent study space. evidence based library and information practice, 14(3), 128–134. https://doi.org/10.18438/eblip29581   stewart, christopher. (2009). the academic library building in the digital age: a study of new library construction and planning, design, and use of new library space. dissertation available from proquest.   zabel, d., & rimland, e. (2007). ranganathan’s relevant rules. reference & user services quarterly, 46(4), 24–26. https://doi.org/10.5860/rusq.46n4.24     appendix questionnaire   personal details name of the respondent: course: school/center/dept                                                    name of the university age:               sex:   1.      do you visit your university library a. yes                                 b. no           if your answer is no the skip the questions 2-7.   if your response is "yes" then…   2.      how frequently do you visit your university library? a. daily                                             b. 2-3 times a week                         c. weekly once      d. once in month                           e. once in 2-3 months                   f. once in six months          g. once in a year   3.      how long you stay in the library building in a visit? a. less than one hour                    b. between one to two hours       c. between two to three hours    d. between three to four hour                    e. more than four hours   4.      for what purpose do you visit the library?     s.n. purpose most of the time sometimes rarely never 1 issue/return of books         2 reading of books and journals of your subject interest in the reading hall          3 to access e-resources subscribed by library using desktop available in the library         4 to access e-resources subscribed by library, email, using personal laptop in the reading hall         5 browsing and reading of print magazines, journals and newspapers         6 to consult reference materials that are not issued to use out of library building like (ph.d. theses, mphil dissertations, and bound volumes of journals)         7 to study in the research scholar’s carrel         mention if any: [comments]   5.      while in the library, which resources do you use? (rank them) s.n. resources most of the time sometimes never 1 printed resources like books and latest issue of print research journals of your area of research       2 access online resources subscribed by library.        3 refer hard copy of ph.d. theses and dissertations, reference materials and bound volume of journals           6.      during the stay in the library, what share of time do you spend in various sections of the library? s.n. areas most of the time sometimes rarely never 1 general reading hall         2 research scholar’s carrel         3 periodical section         4 stacks area of books         5 reference section         6 bound volume section         7 rare collection             7.      please show your preference of format of information sources. s.n. information resources print online 1.     general books preferred desirable neutral preferred desirable neutral             2. textbooks             3. research journals             4. reference sources (yearbook, dictionary, almanacs, handbook, encyclopedia, statistical data book, etc.)             5. back volume journals & ph.d. theses                 8.      show your agreement/disagreement to the following.   to optimize the physical space in the library, the shelf space management is required.   proposed view  strongly agree agree neutral disagree strongly disagree damaged/mutilated books should be weeded out to accommodate new titles           textbooks belong to out of syllabus, outdated, or courses closed in the university should be weeded out           equal number of old and never used/issued books since 2 decades [books not used in 20 years] should be replaced with new titles.             9.      from which location do you prefer to access e-resources (rank them 1 for first preference and 2 for second preference)   s.n. locations preference 1 in the library   2 outside the library from hostel/resident /campus using remote access facility     10.   why do you visit library building to access the subscribed online resources although remote access facility is available to access the same from outside? reasons  strongly agree agree neutral air-conditioned reading hall with wi-fi, laptop charging point for longer hour battery backup and cosy/comfortable chair/table arrangement.       reading with friends and other library users motivate you to read longer hour       mention if any [comments]   11.   show your agreement or disagreement to the following view:   “in order to optimize the physical space of the library, the bound volume of journals, print copy of ph.d. theses and dissertations, never issued/least used/old edition of textbooks, reference sources; should be shifted from main building as these materials occupy major chunk of library space,"   reasons  strongly agree agree neutral disagree strongly disagree all these resources are available online and you prefer to access online.           you require them frequently for your research and these resources are not available in online in your subject of interest.           these are the beauty of the library collection, so they should not be shifted from the main building.           duplicity of availability of knowledge resources online is wastage of physical space.             12.   show your agreement or disagreement to the following suggestion   “looking into your expectation and trends of availability of online resources, should your library; redesign its physical space to convert it into learner centric from collection centric.”?   a.     agree                        b. disagree   if your response is "disagree” skip question no 13   if your response is "agree," then   13.   to optimize the physical space; your library should redesign and develop physical space with latest infrastructure to meet the users’ expectations.   s.n. infrastructure must desirable not required can’t say 01. large table study room with whiteboard.         02. group study room with large table, whiteboard and led screen for presentation practice, seminar talk with peer group etc.           03. makers space with facilities like 3d printers for creativity, video and audio capture, large format printing, art-related workshops, music recording spaces, and so forth.         04. smart classroom with moveable chairs, cozy bean chair, interactive screen, white board, interactive digital wall and projector.         05. standing workstation.         06. carrels for research scholars study.         07. faculty study room with wi-fi facility         08. quiet study room.         09. non silent zone for attending mobile phone call and socializing with friends.         10. open space: a variety of study spaces accommodating individual, small group, and large group study.         11. seminar room with 10-20 people seating capacity with whiteboard, chalkboard, and presentation display.         12. web conference room with polycom conference phone and technology.         13. presentation room with microphone, microphone stand and polycom conference phone.         14. living room setting with chairs and small tables with large monitors for laptop hookup.         15. coffee dispensing machine near every reading hall.         16. small table with chair near glass window for natural light reading.           mention if any: [comments]   date:                                                                                                                                                                                                                                                          signature of respondent place:     evidence based library and information practice editorial responsibilities      2018. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     editor-in-chief: lorie kloda   associate editor (evidence summaries): carol perryman, heather pretty   associate editor (research articles): ann medaille, lisl zach   associate editor (classics & reviews): christina wissinger   associate editor (commentaries, feature, using evidence in practice): lorie kloda   communications officer (news): tatiana bryant   production editor: michelle dunaway, rachel hinrichs   editorial intern: rachel hinrichs, kimberly mackenzie   editorial advisors: lindsay alcock, alison brettle, jonathan eldredge, denise koufogiannakis   copyeditors: richard hayman (lead copyeditor), georgianne bordner, julie evener, heather healy, julie james, emily kingsland, alison moore, jane morgan-daniel, elizabeth stregger, elaina vitale   indexing support: kate shore   research article   an analysis of academic libraries’ participation in 21st century library trends   amy jo catalano associate professor, teaching learning and technology hofstra university hempstead, new york, united states of america email: amy.catalano@hofstra.edu   sarah glasser associate professor joan and donald axinn library hofstra university hempstead, new york, united states of america email: sarah.glasser@hofstra.edu   lori caniano adjunct professor swirbul library adelphi university garden city, new york, united states of america email: lcaniano@adelphi.edu   william caniano associate professor joan and donald axinn library hempstead, new york, united states of america email: william.caniano@hofstra.edu   lawrence paretta email: lparetta82@gmail.com   received: 18 may 2018  accepted: 17 july 2018      2018 catalano, glasser, caniano, caniano, and paretta.. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29332     abstract   objective – as academic libraries evolve to meet the changing needs of students in the digital age, the emphasis has shifted from the physical book collection to a suite of services incorporating innovations in teaching, technology, and social media, among others. based on trends identified by the association of college and research libraries (acrl) and other sources, the authors investigated the extent to which academic libraries have adopted 21st century library trends.   methods – the authors examined the websites of 100 association of research libraries (arl) member libraries, their branches, and 160 randomly selected academic libraries to determine whether they adopted selected 21st century library trends.   results – results indicated that arl member libraries were significantly more likely to adopt these trends, quite possibly due to their larger size and larger budgets.   conclusion – this research can assist librarians, library directors, and other stakeholders in making the case for the adoption or avoidance of particular 21st century library trends, especially where considerable outlay of funds is necessary.     introduction   as academic libraries evolve to meet the changing needs of students in the digital age, the emphasis has shifted from the physical book collection to a suite of services incorporating innovations in teaching, technology, and social media. these services tend to facilitate creativity, engagement, and the ability to access resources anywhere any time (andrews et al., 2016). while there is much “crystal ball gazing,” little assessment of how academic libraries have come to implement 21st century trends has taken place (garofalo, johnston, & lupold, 2015). data collection efforts such as those conducted by organizations like the integrated postsecondary education data system (ipeds) and the association of research libraries (arl) describe what libraries have done or what researchers and faculty want, but they do not operationalize what a 21st century library looks like. in this study, the authors first identified the most commonly cited trends that comprise 21st century libraries, then evaluated the websites of over 300 academic libraries to determine the extent to which they adopted these trends. the 300+ sample included a mix of arl member libraries, arl branch libraries, and randomly selected non-arl academic libraries. the authors also investigated which institutional factors, such as number of librarians on staff, budget, collection size, institution size, and institution status (private versus public), predicted the adoption of these trends.   a review of the literature indicated no single definition or description of a 21st century library. garofalo et al. (2015) advocated focusing on “engagement” and striving to connect with patrons, whether it be through personalized librarians, shared spaces, or collaboration with other services like writing centers. leong (2013) also supported community engagement and outreach as a 21st century library goal. one particularly innovative way some libraries are engaging their patrons is by creating makerspaces, which encourage exploration and innovation by allowing people to create, build, and experiment with a variety of equipment, software programs, and tools (harris & cooper, 2015; herron & kaneshiro, 2017; nichols, melo, & dewland, 2017).   in the book leading the 21st century academic library, edited by bradford lee eden (2015), contributors described emerging positions that require technological skills capable of engaging in online learning, data management, digital collections such as institutional repositories, and other technologies to provide new services. other topics include pursuing open education resources and ways to increase student engagement in library instruction.    emerging staff positions, data management, digital scholarship, and open education resources are also mentioned in the association of college and research libraries (acrl)’s report “2016 top trends in academic libraries” (2017).  the acrl report further noted activities and services in the areas of research data services (rds), information literacy and evidence of learning, collection assessment, altmetrics, and the use of social media. tenopir et al. (2015) looked specifically at research data services (rds) in relation to library demographics and found that they are more common in four-year and research universities than two-year institutions. as would be expected, tenopir et al. note that research universities are more likely to employ rds.    sewell and kingsley (2017) noted that academic librarianship is moving away from curation of material and into support for research with new staff skills needed in areas such as research data management and curation of open access resources. these shifts are confirmed by acrl’s trends toward data management services, digital scholarship/institutional repository support, and emerging staff positions with skills in scholarly and digital communications, knowledge management, data management, digital humanities, and geographic information systems (acrl, 2017). the american library association’s (ala) the state of america's libraries report (rosa, 2016) confirmed this further, noting that academic libraries are “embracing new responsibilities in such areas as scholarly communication, digital archives, data curation, digital humanities, visualization, and born-digital objects” (p. 3) as well as working in areas like altmetrics and research data management.   an important recent initiative in information literacy and evidence of learning is the assessment in action (aia) program lead by acrl. aia challenged participating institutions to plan and implement projects “that aligned with institutional priorities and contributed to campus assessment activities” (brown, 2017, p. 1). the three-year program produced several documents describing multiple ways in which libraries can positively impact student success. acrl’s report documents five areas where the library had a particularly positive impact on student learning and success, two of which concern information literacy instruction and one that concerns library partnerships with other campus units such as writing centers (brown, 2017).   innovative or non-traditional reference services are another area that could be considered a 21st century library trend. increasingly, academic librarians are developing new and innovative ways to reach their patrons, whether finding students and faculty where they are (e.g., dorms and academic buildings outside the library) or through virtual services such as chat or libanswers (via springshare). macdonald and mccabe (2011) reported on a service called iroaming in which librarians walk around the library with tablets to assist patrons at point of need. other libraries have employed tablets mounted on robots which telechat to provide reference services remotely (hartsell-gundy, johnson, & kromer, 2015). coleman, mallon, and lo (2016) investigated the impact of innovative reference services at academic libraries and found that many libraries have developed alternate ways of reaching patrons including using a cooperative reference service, methods for making appointments with librarians, creation of an faq, and creation of a blog to enable patrons to see answers to questions already asked. finally, li (2013) discussed how distance education has caused reference services to adjust, offering various virtual reference services such as chat, videoconferencing, voice-over ip, co-browsing, instant messaging, use of a toll-free telephone number, and email.   while libraries are addressing the needs of the 21st century learner in many ways, a review of the literature reveals a consensus to include some of the following: digital scholarship (including institutional repositories), data management services, makerspaces, evidence of learning with respect to information literacy instruction, innovative staffing with an emphasis on technology or digital services, engagement with open access resources (e.g., curation of open educational resources via a libguide), collaboration with other departments, innovative reference services, altmetrics, and the use of social media. in what follows, the authors describe an assessment of 314 academic libraries as to their adoption of some of these 21st century trends using the following research questions to guide our inquiry:   ·         what 21st century trends have the libraries in our sample adopted? ·         are arl libraries more likely to adopt 21st century trends than branches of arls or non-arl members? ·         what factors predict the likelihood of an academic library adopting 21st century library trends?   methods   trends selected   based on the literature review and examination of academic library websites for information commonly available on websites, the authors chose the following eight 21st century library trends to look for in this study: research data services (rds), digital scholarship (including institutional repository), makerspace, emerging staff positions, open educational resources, distance learner services, non-traditional reference services, and use of social media. additionally, the authors checked for related variables they felt important to explore, such as whether the library offered research design and analysis help (not just reference assistance), collaborated with a campus writing center, offered specific services for international students or students with disabilities, loaned out devices such as laptops and tablets, and had a mobile-friendly website.  sample   the sample included a combination of libraries that are members of the arl, libraries at branches of arl membership institutions but not arl members themselves, and non-arl libraries. arl is an organization of 123 research libraries in the united states and canada. arl libraries share similar research missions and make up a large portion of the academic and research library marketplace. they are typically at larger, comprehensive, research institutions (arl, 2017). arl libraries were used as a variable in this study because they may be more likely to adopt these trends due to their typically higher budgets and their mission to support research. the total sample included the 100 arl libraries at institutions in the united states (canada was excluded due to our sampling decision to only study the united states), 54 libraries that are branches of arl member institutions but not arl libraries themselves, and 160 randomly selected, non-arl libraries. the final sample consisted of 314 libraries from post-secondary institutions in the united states.   to obtain the random sample of non-arl libraries, a full list of all 3,148 four-year institutions was drawn from the integrated postsecondary education data system (ipeds). entries for any institutions that closed were removed. technical schools were also filtered out, as the authors were interested in academic institutions only. this brought the list to 1,653, from which the arl libraries and their branches were removed. the authors then used an online random number generator to randomly select 200 libraries from the list, 160 of which were ultimately included in the sample. the 40 that were excluded were branches where all data appeared the same for the whole school system (i.e., the system shares one library).   it is important to clarify that for university systems with several branches, only the branches designated as members of arl have arl status. being a branch of an arl library does not automatically confer arl membership to that branch. in some cases, all branches of an arl library are members (e.g., rutgers). however, in many cases the branches of arl libraries do not have arl membership (e.g., campuses of the university of michigan other than the main campus in ann arbor). in order to add this variable into the data analysis, the branches of arl libraries that were not arl members themselves were coded separately. this allowed the authors to test whether being a part of a system with an arl member may have benefits not afforded to non-arl affiliated libraries. procedure   the authors visited the website of each school in the sample in order to ascertain whether the trends described above were adopted by the libraries. although surveying librarians at the sample institutions would have allowed us to gather more complete data on trend adoption, the method we utilized allowed for 100% representation of our selected sample.   prior to data collection, the research team went through a period of training. during the first session, the team collected data for 10 libraries to develop a method for searching for each variable completely. teams of 2 researchers were then assigned 15 libraries. each team member filled out the data collection worksheet separately. those worksheets were returned to the first author who calculated interrater reliability (irr). irr was computed by calculating the number of responses that were in agreement out of the entire sample for each pair of raters using an excel spreadsheet. the first team demonstrated 92% agreement in their responses. this team then reviewed places of disagreement to determine where agreement could be improved.  irr for the second team had 73% agreement, which was lower than desirable so that team retrained and worked together on an additional set of libraries in order to norm their responses. after retraining, the irr for the second team was recalculated on a new sample of 20 schools which resulted in 90% agreement. after all teams demonstrated that their data collection was reliable, each researcher was assigned a set of libraries on which to collect data. after all data were returned to the first author, who managed and analyzed the data, each library website and data line for that website was cursorily checked to ensure that the other researchers did not miss any data. additionally, the first researcher randomly selected libraries assigned to each team member to check the data collection more thoroughly. through these procedures, the reliability of the data can be affirmed.   the next portion of the data collection included pulling institutional and library characteristics from ipeds. these data included institution size (fte student enrollment), whether private or public, carnegie classifications, highest degree granted, library budget for salaries, materials and operations, physical, electronic, and media collection size, and circulation statistics of these items. data were analyzed using spss version 23.                 results   description of the sample   a total of 314 schools were investigated. the sample included 160 randomly selected schools, the 100 arl member united states libraries, and 54 branch libraries of the arl libraries. although a portion of the sample was randomly selected, 48 of 50 states were represented.   description of trend adoption by library type   trend adoption varied by library type (arl, branch, non-arl). arl libraries more often adopted most trends, except for distance learning/learner services. the numbers presented in table 1 are raw numbers and not percentages. therefore, since there are 100 arl libraries, 84 of which have an institutional repository (ir), 84% of arl libraries have an ir, while 26% (n = 41) of non-arl libraries have an ir.  sixty-six percent (n = 66) of arl libraries openly support oer, while only 26% (n = 42) of non-arl libraries do.  rds, a relatively new service among academic libraries, has only been adopted by 10 non-arl libraries (6%), while 73 arl libraries (73%) offer such services. makerspaces too have seen limited adoption in all libraries: 39% (n = 39) of arl libraries have a makerspace while 11% of non-arl libraries (n = 18) offer these services. most libraries (n = 92 for both arl and non-arl libraries) had a social media presence; similarly, most libraries offered some form of non-traditional reference. most non-traditional reference included chat or libanswers, but also personal librarian services, delivering reference service in the residence halls, twitter, and other virtual forms of chat.   emerging staff positions as defined by acrl include scholarly communications, digital projects, data management, user experience, technical support, digital humanities, and learning commons librarians (acrl, 2016). although acrl’s top trends noted knowledge management librarian as an up-and-coming position, and one for which acrl commonly found job descriptions, no such position was found at any of the libraries in our study.     table 1 21st century trends by library type (arl, arl branch, non-arl)   ir/digital scholarship emerging staff positions social media rds maker-space oer dl services (to dl students) non-traditional reference (e.g., chat, virtual, twitter) non-arl libraries (n= 160) 41 50 92 10 18 42 42 88 arl libraries (n = 100) 84 85 92 73 39 66 41 91 arl branches (n = 44) 32 23 34 8 10 27 17 38 total 157 158 218 91 67 135 100 217     trend scores that ranged from 0-8 were calculated for each library. one point was given for each of the eight trends adopted. the mean trend score for the entire sample was 3.6 with a standard deviation of 2.2. mean trend scores for each group are as follows: non-arl, 2.4; arl, 5.7; branch, 3.6. although no benefit of arl membership is assigned to libraries that are branches of, or in the same university system as, an arl, branch scores were moderately higher than non-arl affiliated libraries. an analysis of variance (anova) revealed significant differences between these three groups on the trend scores (f (2,308) = 112.811, p < 0.001).   description of library characteristics by library type   the data from ipeds is reported in tables 2-4 as means by library type (arl, arl branch, non-arl). several outliers from non-arl libraries were removed as they greatly impacted the means superficially. for example, 2 institutions reported over 100,000 electronic databases where the means for electronic databases for arl libraries without these 2 outliers were 890 and 156 for non-arl libraries, indicating either an error in reporting or an uncommon method of characterizing an electronic database. as expected, arl libraries and their branches reported higher means for economic factors, particularly with respect to staff salaries. interestingly, the amount of money spent on salaries was similar to the amount of money spent on materials and services for all three library types. for library collection sizes (reported in table 3), arl libraries and their branches possessed greater numbers of each material type, almost by a factor of five, over non-arl libraries. physical book collections were exponentially larger for arl libraries than for non-arl libraries, with the former housing a mean of 4,399,197 while non-arl libraries housed an average of 326,572.     table 2 library expenditures on staff, materials, and operations library type n avg. total salaries avg. total materials/services expenditures avg. total operations and maintenance avg. total expenditures non-arl 151 980,016 898,717 157,130 2,207,241 arl 99 12,308,587 12,814,542 4,007,414 32,185,023 branch 46 1,413,746 1,363,755 276,372 3,313,217     table 3 size of library collections   n avg. number of physical books avg. number of electronic books avg. number of electronic databases avg. number of physical media avg. number of electronic media non-arl 157 326,572 197,348 156 46,492 46,036 arl 99 4,399,197 1,001,269 890 1,618,960 179,977 branch 45 413,957 345,059 262 130,438 45,540     table 4 circulation of materials   n avg. total physical library circulation avg. total digital/electronic circulation (media and books) avg. total library circulation non-arl 151 29,088 186,473 215,562 arl 99 220,414 2,107,993 2,328,408 branch 46 28,056 377,357 404,762     factors that impact adoption of 21st century trends   a multiple regression was conducted to determine which of the following institutional variables impacted an institution’s likelihood of adopting 21st century trends: designation as arl/non-arl/branch, private or public status, student fte, and carnegie classification. library characteristics that were examined included physical collection size, database collection size, e-book collection size, salary for staff, and total operations budget. an initial analysis revealed arl status, carnegie classification, operations budget, and student fte as the significant variables among those entered. an additional analysis was conducted with only these four variables entered. this analysis of the 292 schools for which we had complete data (some schools were removed because branches shared reported resources) confirmed that those 4 variables contributed to trend adoption. a significant model was fit (f (4, 290) = 63.538, p < .0001), with an r2 of .467. trend scores are equal to 1.110, + 537 (arl status) + -.037 (carnegie classification) + .853 (student fte) is equal to 1.110, carnegie classification is equal to -.037 (meaning that classification had an inverse relationship), and student fte is equal to .853. this analysis did not show that total operations expenditures were a significant predictor indicating that this variable was associated with other variables such as student fte. in sum, each variable contributed some portion of the trend score. for example, trend scores increased .537 points for each school with arl member status.   because the goals of an arl member are notably different than non-arl libraries, a second analysis looked at the factors that impacted adoption of trends among non-arl libraries only. this analysis included the following predictor variables: total operations expenditures, expenditures on materials, expenditures on salaries, number of physical books, number of electronic books, number of electronic databases, student fte, and carnegie classification. in this analysis, salary expenditures were a significant predictor of trend scores at p =0.05. the model fit (f [8, 141]) = 8.568, p < .0001, with an r-squared value of .327. in other words, library staffing allowed for support of trend adoption.    other trends observed   in addition to the eight trends included in the trend score calculation, the authors collected data on other, related variables that the authors thought were important to explore. these are reported below in tables 5 and 6 and include trends related to staffing positions and services.   table 5 other common staff positions found at academic libraries but not identified as a 21st century trend   outreach librarian first year librarian gis librarian distance ed. librarian embedded librarian instructional coordinator non-arl libraries  (n= 160) 27 4 3 11 10 39 arl libraries (n = 100) 38 19 40 11 6 35 arl branches (n = 44) 13 3 3 7 7 14 total (n = 304) 78 26 48 29 23 88     table 6 other services   research design and analysis help writing center collaboration services for international students disabilities services device loan mobile website non-arl libraries (n= 160) 5 31 5 31 43 123 arl libraries (n = 100) 44 43 20 66 68 89 arl branches (n = 44) 7 22 8 24 31 44 total (n = 304) 56 96 33 121 142 256     discussion   a total of 314 websites of academic libraries were analyzed to determine the extent to which these libraries adopted 21st century trends. for arl libraries the most common trends were research data services (rds), ir/digital scholarship, and innovative reference. for non-arl libraries, emerging staff, ir/digital scholarship, and innovative reference were the most commonly adopted trends. these findings are also supported by the regression analysis in which it was demonstrated that the money spent on library staffing at an institution was the only significant predictor of trend score.   arl libraries were more likely to adopt 21st century library trends than non-arl libraries, due to the size and budget of the institution. most importantly, however, the goals of arl libraries to support research likely drive the adoption of these trends. it is important to note that comparisons among arl and non-arl libraries may not be suitable because non-arl libraries include a wide variety of sizes of institutions (from very small private to very large public), while arl libraries are typically larger schools with a larger average fte.   additionally, because arl libraries are larger and better funded, they are often early adopters.  simply put, they can afford to be wrong about a trend. many libraries that do not have budgets flexible enough to tolerate chasing a trend that may turn out to be a fad, will wait until the trend is supported by the literature or becomes so commonplace in other libraries that students expect it. in this way, arl libraries can influence the trajectory of certain trends. this dynamic can be seen in the research conducted for this article. for every new service or job title, the arl libraries have adopted them in greater number.   one of the more commonly adopted trends by arl over non-arl libraries is ir/digital scholarship. irs can be prohibitively expensive and/or technologically challenging to operate.  smaller and less well funded libraries may find these significant barriers to adoption. interestingly, oers, which are about leveraging free resources and therefore appealing to the cost conscious, were more readily adopted by arl than non-arl libraries. one reason for this might be a lack of sufficient staff. while oers themselves are free resources, it takes time and expertise to vet them, clarify any copyright provisions, and keep up with the constantly changing and expanding oer landscape.   data on services other than the eight 21st century trends are represented in tables 5 and 6. as with social media, the adoption of mobile websites is ubiquitous enough to be considered a new standard in website design. out of this seemingly eclectic group of services, the most commonly adopted service is device loans, with 68 out of 100 (68%) arl libraries, 31 out of 44 (70%) arl branches, and 43 out of 160 (27%) non-arl libraries offering this service. to more thoroughly examine this trend, research into the types of devices being loaned and what the circulation statistics are, is needed.  the second most popular service trend for non-arl libraries are writing center collaborations and disability services. for arl libraries, disability services are also the second most popular. these trends, together and in conjunction with the growing number of outreach librarian positions (38% of arl libraries, 30% arl branches, and 17% of non-arl libraries), show the growing importance of outreach and external collaboration.  interestingly, while the above-mentioned collaboration services are on the rise, adoption of collaboration with international students appears to be lagging; it is the least likely service trend for both arl and non-arl libraries. this may be a weakness in solely examining library websites, as library subject specialists may very well be conducting outreach to this group without it being represented in their job titles. further research could compare the number of international students enrolled in the schools sampled. for arl libraries, the third most adopted service trend was research design and analysis assistance. as previously mentioned with respect to rds, this trend reflects the importance arl institutions place on conducting research.   another interesting observation is the number of geographic information systems (gis) librarian positions at arl libraries (40%). this is a relatively new area that seems to have become quite popular as more research in done using geospatial data. it can require significant technological knowledge, which may explain why it is not a high-ranking trend for non-arl libraries or arl branches (2% and 7% respectively). while not identified as a 21st century trend for this paper, it is a trend worth watching.   if we were to distill the findings of this study, a strong case could be made that the most important and popular trends are supporting and curating irs/digital scholarship and outreach/external collaboration. if we look to arl libraries as trend setters in the library profession, it is clear that ir/digital scholarship is the most pressing trend.    although the 2018 acrl analysis of emerging trends was published prior to the data collection in this study, it showed that some focus has shifted to fake news and information literacy, and legacy print collections, while trends that remain the same are project management, open access resources, data management and data collection, and patron driven collections.   limitations   the method of this study, while allowing for 100% participation of the selected sample, has some weaknesses. two trends that the authors could not assess through this method was collection development assessment and evidence of learning. although some websites included information about information literacy assessment, the authors could not be certain that this particular trend would be found consistently on a website. a survey would have allowed a better assessment of these trends. it is recommended that future research validate the results of this study through more interactive means. lastly, the eight trends selected for the trend score, while supported by the library literature, are somewhat subjective in that there may be multiple ways to assess one variable. for example, although we could not determine the use of altmetrics, a social media presence was a clue that libraries were using common and modern methods of connecting with patrons. it is important to note that due to random selection, a portion of the non-arl libraries are in small institutions and are therefore poorly compared to the larger arl libraries.   implications and recommendations for future research   the implications for library practice and policy as a result of this research primarily point to the need for professional development of library staff in order to be able to support these new trends, particularly with an emphasis on technology and research support. future research should continue to analyze adoption of these trends and others to both validate the results written here and to establish a model of prediction. such models can assist libraries in avoiding costly mistakes and adopting trends that are appropriate for their library type and budget.   conclusion   to define and assess what 21st century academic libraries look like, the authors identified the most commonly cited 21st century library trends and evaluated the websites of over 300 academic libraries to determine the extent to which they adopted these trends. the authors visited the websites of all arl member libraries in the united states, their branches, and 160 randomly selected academic libraries that are not arl libraries. the primary goal of this study was to identify the extent to which the libraries in the sample reflected the status of a 21st century library, as defined by adoption of the trends discussed above. additionally, the study sought to determine whether arl libraries were more likely to adopt these trends over non-arl libraries. lastly, regardless of arl status, the authors investigated which institutional factors, such as number of librarians on staff, budget, collection size, student fte, and institution status (private versus public), predicted the adoption of 21st century trends. results indicated that arl member libraries were significantly more likely to adopt these trends, quite possibly due to their larger size and larger budgets. this research can assist librarians, library directors and other stakeholders in making the case for the adoption or avoidance of some trends, especially where considerable outlay of funds is necessary.   references   acrl research planning and review committee. (2017). 2016 top trends in academic libraries: a review of the trends and issues affecting academic libraries in higher education. college & research libraries news, 77(6). retrieved from https://crln.acrl.org/index.php/crlnews/index   acrl research planning and review committee. (2018). 2018 top trends in academic libraries: a review of the trends and issues affecting academic libraries in higher education. college & research libraries news, 79(6). retrieved from https://crln.acrl.org/index.php/crlnews/index   association of research libraries, about. accessed september 13, 2017, http://www.arl.org/about.   andrews, c., downs, a., morris-knower, j., pacion, k., & wright, s. e. (2016). from “library as place” to “library as platform”: redesigning the 21st century academic library. emerald group publishing limited. https://doi.org/10.1108/s0732-067120160000036006   brown, k. (2017). academic library impact on student learning and success: finding from assessment in action team projects. retrieved from association of college & research libraries website: http://www.ala.org/acrl/sites/ala.org.acrl/files/content/issues/value/findings_y3.pdf   coleman, j., mallon, m. n., & lo, l. (2016). recent changes to reference services in academic libraries and their relationship to perceived quality: results of a national survey. journal of library administration, 56, 673-696. https://doi.org/10.1080/01930826.2015.1109879   coleman, j., mallon, m. n., & lo, l. (2016). recent changes to reference services in academic libraries and their relationship to perceived quality: results of a national survey. journal of library administration, 56, 673-696. https://doi.org/10.1080/01930826.2015.1109879   eden, b. l. (ed.). (2015). enhancing teaching and learning in the 21st-century academic library: successful innovations that make a difference. lanham, md: rowman & littlefield.   garofalo, d. a., johnston, l., & lupold, a. (2015). optimizing library services -crystal     ball gazing: academic library services in the 21st century. against the grain, 27, 55–56. retrieved from https://www.against-the-grain.com/   harris, j., & cooper, c. (2015, march). make room for a makerspace. computers in libraries, 35, 5-9. retrieved from https://www.questia.com/library/p4949/computers-in-libraries   hartsell-gundy, j, johnson, e. o., & kromer, j. (2015). testing telepresence: remote reference service via robotics. reference and user services, 55, 118-122. https://doi.org/10.5860/rusq.55n2.118   herron j., & kaneshiro, k. (2017). a university-wide collaborative effort to designing a makerspace at an academic health sciences library. medical reference services quarterly, 36, 1-8. https://doi.org/10.1080/02763869.2017.1259878   leong, j. h. t. (2013). community engagement – building bridges between university and community by academic libraries in the 21st century. libri, 63(3), 220-231. https://doi.org/10.1515/libri-2013-0017   li, p. (2013). effect of distance education on reference and instructional services in academic libraries. internet reference services quarterly, 18, 77-96. https://doi.org/10.1080/10875301.2013.804018   macdonald, j., & mccabe, k. (2011). iroam: leveraging mobile technology to provide innovative point of need reference services. code4lib journal, (13), 1-7. retrieved from https://journal.code4lib.org/    nichols, j., melo, m., & dewland, j. (2017). unifying space and service for makers, entrepreneurs, and digital scholars. portal: libraries and the academy, 17, 363-374. retrieved from https://preprint.press.jhu.edu/portal/    rosa, k. (ed.). (2016). the state of america’s libraries: a report from the american library association. retrieved from american library association website: http://www.ala.org/news/state-americas-libraries-report-2016   sewell, c. & kingsley, d. (2017). developing the 21st century academic librarian: the research support ambassador programme. new review of academic librarianship, 23, 148-158. https://doi.org/10.1080/13614533.2017.1323766   tenopir, c., hughes, d. allard, s., frame, m., birch, b. baird, l., & lundeen, a. (2015). research data services in academic libraries: data intensive roles for the future? journal of escience librarianship, 4, e1085. https://doi.org/10.7191/jeslib.2015.1085 iranian public libraries can improve self-efficacy in information literacy, especially when school library instruction is not preparing students for lifelong learning readiness evidence summary   iranian public libraries can improve self-efficacy in information literacy, especially when school library instruction is not preparing students for lifelong learning readiness   a review of: leili, s., maryam, h., & mohsen, a. (2020). the effect of information literacy instruction on lifelong learning readiness. ifla journal, 46(3), 259-270. https://doi.org/10.1177/0340035220931879   reviewed by: samantha j. kaplan research & education librarian, liaison to the school of medicine duke university medical center library & archives durham, north carolina, united states of america email: samantha.kaplan@duke.edu   received: 7 mar. 2022                                                                    accepted:  19 apr. 2022      2022 kaplan. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30119     abstract   objective – to examine the efficacy of information literacy skills instruction on the lifelong learning readiness skills of iranian public library users.   design – preand post-test experiment.   setting – two public libraries in iran.   subjects – thirty (30) high school students who were active users of two iranian public libraries.   methods – thirty (30) participants were randomized into two groups, one of which received information literacy training for seven weeks, while the other group acted as a control. participants were assessed via three instruments in information literacy and readiness for lifelong learning prior to and at the completion of the training program. the workshops included basic library skills, recognizing needed information skills, information source skills, internet skills, internet searching skills, resource instruction, database skills, and general searching skills. results of preand post-test assessments were analyzed with analysis of covariance (ancova).   main results – the group that received information literacy instruction showed increased readiness for self-directed learning, readiness to overcome deterrents to participation, and improved information literacy. the control group did not show an increase in readiness to respond to triggers for learning or an overall increase in lifelong learning readiness.   conclusion – information literacy instruction can improve elements of lifelong learning readiness in regular library users. public libraries in iran should begin long-term planning to implement this training.   commentary   this research was appraised with the british medical journal education group’s guidelines for evaluating papers on education interventions (1999). there are areas where this paper meets or exceeds the guidelines, particularly in providing a readable, well-structured manuscript with meaningful results and a study that answers the questions it poses. however, there are unfortunate omissions in this paper’s methodological reporting, particularly around description of recruiting and randomization. the authors do state the experimental and control groups were homogenous in age and education level, but do not provide a summary table comparing the groups or report on other elements that could have influenced composition (for example: sex, socioeconomic background, academic standing at school), although the pre-test comparisons do not suggest any imbalance.   participants “were active library users, based on iran’s public library rules, who had a membership card for the public library and used the library at least twice per week” and were high school students (leili, maryam, & mohsi, 2020, p. 262). how these participants were identified from the library users, sampled, and invited to participate is not described, nor is any incentive they may have been offered to complete the study. additionally, how the randomization was conducted, if any effort was made to conceal the randomization from the participants, or to blind participants or instructors, is not addressed. the authors state they “observed the performance of the participants while conducting the instruction and conducted informal interviews with them to measure their levels,” (leili, maryam, & mohsi, 2020, p. 262) suggesting they were aware of which students received the exposure and potentially could have biased the results, undermining the credibility of these findings.   why high school students were selected, instead of members of the general public, is acknowledged as a limitation but the rationale is not discussed. this is particularly important, as the findings are strong and do make a case for library instruction, but the authors note that librarian positions have largely been eliminated from iranian schools (leili, maryam, & mohsi, 2020, p. 267). thus, the findings demonstrating that high school students benefit from library instruction make a case for school librarians, or that public libraries can fill this role in the meantime, rather than demonstrating the role public libraries can have in lifelong learning readiness among the general public. further, the assessment aspect of this experiment is based on learner perception of their skills in both the preand post-test instruments available in the supplementary materials. it is useful to know that learner self-perception and reported behaviour improved in those that received the training sessions. however, without an objective measure of skill growth (such as evaluating task completion), these findings really demonstrate self-efficacy.    despite these issues, this research accomplishes several significant items of note. there are few studies testing the role of information literacy instruction in libraries in iran and this study does that while also outlining the tangible structural challenges facing iranian public libraries and patrons. the findings show how libraries can help and the authors make a strong case for investing in iranian library infrastructure, outlining what is needed. these findings are also valuable for those interested in the role of public libraries in international or lower-resourced settings.   references   education group for guidelines on evaluation. (1999). guidelines for evaluating papers on educational interventions. bmj: british medical journal, 318(7193), 1265-1267. https://doi.org/10.1136/bmj.318.7193.1265      evidence summary   bibliometric analysis provides a detailed map of information literacy literature in the social sciences and humanities   a review of: bhardwaj, r.k. (2017). information literacy in the social sciences and humanities: a bibliometric study. information and learning science, 188(1/2), 67–89. https://doi.org/10.1108/ils-09-2016-0068   reviewed by: jessica a. koos senior assistant librarian/health sciences librarian stony brook university libraries stony brook, new york, united states of america email: jessica.koos@stonybrook.edu   received: 22 aug. 2019                                                                 accepted:  28 oct. 2019      2019 koos. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29628     abstract   objective – to determine the scope and distribution of information literacy research documents in the humanities and social sciences published from 2001 to 2012.   design – bibliometric analysis.   setting – n/a   subjects – 1,990 document records retrieved from a scopus database search.    methods – using the database scopus, the author created and conducted a search for documents related to the concept of information literacy. articles, review papers, conference articles, notes, short surveys, and letters were included in the results. only documents published from january 1, 2001 to december 31, 2012 were included in the study. the author then performed various bibliometric analyses of the results.   main results – the author found that the number of publications and citations have increased over time, although the average citations per publication (acpp) decreased significantly during the time period being studied. the majority of the literature published on this topic is in english and produced within the united states. the transformative activity index was calculated to determine changes in publishing patterns across countries from 2001 to 2012. the amount of research collaboration across countries was calculated as well, with the u.s. being the most collaborative. the top journals publishing on this topic were identified by calculating the h-index. an individual from universidad de granada in spain published the greatest number of articles from a single author, and this university was found to have produced the greatest amount of research. documents produced by the united kingdom have the highest citation rates. a total of 1,385 documents were cited at least once, and each item on average was cited five times.   conclusion – most of the articles on information literacy in the social sciences and humanities comes from developed countries. the results of this study may help to inform those interested in researching this field further.   commentary   the quality of this study was appraised using “the cat: a generic critical appraisal tool” created by perryman and rathbun-grubb (2014). based on this analysis, the quality of the study was found to be moderate. the author is a librarian at a college within a large public university system in india and has a phd as well as multiple master’s degrees. a literature review was included in the article, and the research questions were clearly defined.   most of the methods were thoroughly explained; however, there was no explanation of how the author determined which documents fell within the subject area of social sciences and humanities. additionally, for some metrics, other subject areas such as health sciences were included in the analysis without any explanation as to why. a description of screening procedures was also not included. these factors significantly hinder transparency and reproducibility.   other limitations were mentioned in the article, including the fact that scopus does not have complete coverage of all relevant journals, as well as the fact that it does not include book chapters, dissertations, and theses. additionally, scopus also has limited geographic coverage, which may have altered the results. these limitations could have been mitigated by searching additional databases.   the author used bibliometric measures that are commonly used in these types of analyses, such as the h-index and the transformative activity index (tai). although it would have also been useful to include an examination of the altmetrics of these documents to more fully assess their reach, this step was most likely beyond the scope of this study.   the author describes some of the benefits of this study pertaining directly to information literacy instruction; however, these cannot be surmised from the data obtained. for example, the author states that “the study will assist in designing a new information literacy course structure through an understanding of the progress in the area …” yet this is an assumption not justified by the findings. additional research would need to be conducted in order to support these claims.   as mentioned by the author, this research is unique in that there are no previous studies detailing the scholarship on information literacy in these specific subject areas. therefore, this study provides unique information that may be used by others interested in conducting research within this field. for example, researchers may want to know which authors and previous publications have received the most citations so that they can be sure to consider these specific works when developing their own studies.   references   perryman, c., & rathbun-grubb, s. (2014). the cat: a generic critical appraisal tool. in jotform – formbuilder. retrieved from http://www.jotform.us/cp1757/thecat       review article   shifting horizons: a literature review of research data management train-the-trainer models for library and campus-wide research support staff in canadian institutions   dr. felicity tayler research data management librarian university of ottawa ottawa, ontario, canada email: ftayler@uottawa.ca maziar jafary phd candidate and part-time professor school of sociological and anthropological studies university of ottawa ottawa, ontario, canada email: mjafary@uottawa.ca   received: 4 aug. 2020                                                               accepted: 16 nov. 2020      2021 taylor and jafary. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29814     abstract   objective – in consideration of emerging national research data management (rdm) policy and infrastructure, this literature review seeks answers to the following questions: 1)      what is the most effective way for a canadian research university to build capacity among library and campus-wide research support staff, with a view towards providing coordinated rdm support services for our researcher community? 2)      what international training models and course offerings are available and appropriate for a local context? 3)      what national guidelines and best practices for pedagogical design and delivery can be adapted for a local context? methods – this literature review synthesizes a total of 13 sources: 9 articles, 2 book chapters, and 2 whitepapers. the whitepapers were selected for a narrative literature review because of their focus on case studies detailing train-the-trainer models. within the 13 sources we found 14 key case studies. this review serves as a supplement to the 2017 carl portage training expert group white paper, “research data management training landscape in canada,” the focus of which was to identify rdm training gaps in order to recommend a coordinated approach to rdm training in a national environment.   results – the narrative review of case studies revealed three thematic areas. firstly, pedagogical challenges were identified, including the need to target training to rdm support staff such as librarians and researchers, as they comprise distinct groups of trainees with divergent disciplinary vocabularies and incentives for training. secondly, the case studies cover a broad range of pedagogical models including single or multiple sessions, self-directed or instructor-led, in-person or online instruction, and a hybrid of the two. finally, rdm training also emerged as a key factor in community building within library staff units, among service units on campus, and with campus research communities. conclusion – rdm training programs at local institutions should be guided by a set of principles aligned with the training methods, modes of assessment, and infrastructure development timeline outlined in a national training strategy. when adapting principles and training strategies to a local context, the following trends in the literature should be considered: librarians and researchers must have meaningful incentives to undertake training in rdm or to join a community of practice; disciplinary-specific instruction is preferable to general instruction; a librarian’s own training opportunities will influence their ability to provide discipline-specific rdm instruction to researchers; in-person training opportunities improve learning retention and produce beneficial secondary effects, whereas online instruction is most effective when paired with an in-person component; generalized third-party rdm training should be adapted to local context to be meaningful. future directions for rdm training will integrate into open access and digital scholarship training, and into cross-disciplinary, open science communities of practice.     introduction   this literature review was undertaken to help the research services division of the university of ottawa library determine effective training methods for library and campus-wide research support staff, with a view towards providing coordinated rdm support services for the researcher community. for the last four years, university of ottawa has held an annual, in-person, campus-wide rdm training event, attended by researchers and a wider general audience. the event was also attended by rdm-curious librarians and researchers from other universities. by 2019 this event had gained national attention as the shifting horizons training series. the 2020 edition presented a national training program, developed through canada’s carl portage network. one of the goals of the program is to provide basic rdm skills training for librarians and for support staff in the research office, labs, faculty departments, and central it. despite the event’s success, the library’s research services division needed to evaluate whether a single annual event was the most effective way to achieve the vision of campus-wide rdm awareness and a coordinated service model. as was observed in the follow up documentation of the event, the rdm readiness report, the stakeholders of a coordinated rdm service model at university of ottawa continue to face the same challenges as those identified by 241 librarians in a study by tang and hu (2019). some of these challenges include issues with staffing and upskilling, promotion of the service, service quality, and shared understanding among campus departments.    aims   the articles reviewed here supplement the carl portage training expert group white paper, “research data management training landscape in canada” (fry et al., 2017). the purpose of this white paper was to identify “significant issues and gaps in rdm training in canada” and to recommend a national, coordinated approach to rdm training (fry et al., 2017, p. 2). this recommendation is driven by the understanding that expertise in data stewardship is unevenly distributed across higher education institutions and is often isolated within disciplinary areas. in contrast to rdm infrastructures elsewhere, which cohere around disciplinary or national service centres, a critical mass of rdm expertise in canada is organized within the academic library community. to date, this report’s holistic multi-platform vision of a coordinated national training curriculum, to level the “playing field” has been enacted in a modest capacity through best practices, data primers and ad-hoc webinar training, supplemented by single-day, in-person sessions reflecting the individual expertise of members of the portage training expert group (fry et al., 2017, p. 7). the day-long training event at university of ottawa, led by james doiron, who is both an author of the training landscape white paper and the rdm services coordinator at the university of alberta libraries, is one example of the in-person sessions currently offered through carl portage. once an institution has participated in the training, the next steps are unknown. for example, there is no clear direction, recommended strategies, or coordinated curriculum resources, at the national level, to support the long-term development of highly qualified personnel (hqp) providing rdm services at libraries selected to play a leadership role in this area. methods   this literature review synthesizes a total of 13 sources, including 9 articles, 2 book chapters, and 2 whitepapers from a larger sample of 35 texts published within the last ten years (2010-2020). the authors cited seven additional supporting sources in the analysis in order to provide the contextual framing for the thematic approach of this narrative review. keyword searches such as “research data management (and) training” were undertaken in databases including lista and library and information science source. because rdm training is an emerging field, contingent upon variable jurisdictional challenges, policy, and funding environments, the aim was not to be exhaustive, nor systematic in our searches. instead, a “snowball” search for key articles, white papers, and reports shared by colleagues on rdm-themed listservs such as canlib-data, or iassist, or referenced at annual rda plenaries supplemented these keyword database searches. in addition to the snowball searching, the authors contacted various content experts to review the abstracts collected to ensure that no important sources were missed. though the number of sources reviewed is minimal, this is an indicator that rdm is an emerging area of librarianship, which is also interdisciplinary in nature. there are simply not that many articles out there yet, and this literature review aims to address this gap while recognizing that there is still work to do in this area. in the 13 sources selected for synthesis, we found 14 key cases for analysis. out of the 13 sources selected for synthesis, as outlined above, 9 of the selected sources had a single case study focus (baker et al., 2016; grootveld & verbakel, 2015; haddow, 2014; helbig, 2016; papadopoulou & miller in clare et al., 2019; papadopoulou & grabauskiene in clare et al., 2019; wittenberg et al., 2018; southall & scutt, 2017; read et al., 2019). out of the original 13, 2 of the selected sources covered multiple case studies (bryant et al., 2018; surkis & read, 2015), while 2 of the sources dealt with the same case study (tang & hu, 2019; shipman & tang, 2019). in choosing the case studies, the authors prioritized european, north american, and australian examples as their social and academic contexts are comparable to those of canada. however, this geographic limitation and focus on english-language sources introduces a bias to this review. this selection bias does not reflect a deliberate exclusion of other regional models, rather it echoes a trend to build canadian digital research infrastructure on existing models such as the european open science cloud (eosc), or to look to best practices in rdm established by the digital curation centre in the uk, and american rdm service models as outlined by oclc.   results   the review of the literature is divided into three sections, reflecting themes within the articles and case studies. the first section discusses challenges and opportunities for rdm training in universities. outreach and pedagogical issues were identified by several authors, including the development of targeted rdm training to two distinct groups of trainees: rdm support staff, including librarians, and researchers. these two groups differ in their incentives for training participation and their use of discipline-specific vocabulary. with these challenges in mind, the evaluation of training models for success and areas of improvement will be discussed. the second section explains different approaches to curriculum and pedagogical design in rdm training. the case studies cover a range of pedagogical models and whenever possible evaluations of these training methods and formats of pedagogical engagement for rdm training are highlighted. finally, the third section looks at how rdm training operates as a means of community building within library staff units, between service units on campus, and within campus research communities. this final section also covers internal and external partnerships which are necessary to develop rdm training.   discussion   challenges and opportunities for rdm training in universities   while many of the texts that were retrieved in the searches addressed developing rdm services around best practices, or outlined approaches for broader data literacy training strategies, this literature review focuses on train-the-trainer models as a unique subset of the rdm training landscape. because the literature in this area is emerging, this review presents a combination of conclusions drawn from train-the-trainer models alongside approaches to training researchers. in a train-the-trainer model, the targeted audience of trainees are librarians and other research support staff. in the researcher trainer model, the targeted audience members are typically faculty, student research assistants, and other affiliates of disciplinary research projects. however, in practice the line between these roles is blurry, as trainers often become a secondary audience of the training for researchers, and researchers can also benefit from train-the-trainer sessions as they can perform a trainer role as part of their own research team. furthermore, as this review demonstrates, there is a correlation between the pedagogical model applied to train-the-trainer sessions and the effectiveness of these trainers to then shape learning experiences for researchers. by outlining the challenges to providing rdm training to researchers in this section, the recommended best-practices can inform approaches to train-the-trainer models. we begin with the principle that rdm is not generic. instead, librarians and other research support staff need a fundamental understanding of how data flows and data management differ between disciplinary research methods, and how to recommend relevant engagement with local, national, and international infrastructure contexts.   rdm training for librarians and other research support staff will have an impact on the success of rdm services delivered. both tang and hu (2019) and surkis and read (2015) identify significant barriers and pedagogical challenges of rdm training for librarians and other research support staff, beyond the administrative concerns of budget and capacity. for example, librarian language and vocabulary does not translate well to the disciplinary environment of researchers and other stakeholders. such specialized rdm vocabulary might not be well received or even understood by researchers. another challenge could be a lack of training for librarians and research support staff on different approaches to research data management within the field of study, as defined by the researchers’ peers and funding bodies. tang and hu’s (2019) needs assessment highlighted the need for key training in strategic communication of rdm service models to library and university administration, while surkis and read (2015) instead stress that when the goal is the improvement of training offerings for researchers, instructors from the library sector (and related fields), as part of their own training, should engage in interviews with researchers in different fields. this exercise would help librarians better understand researchers’ needs and expectations from rdm services. a later study by read et al. (2019) further explored this lack of disciplinary knowledge as a high barrier to librarian engagement with rdm services in biomedical fields, due to a “lack of comfort engaging with researchers” (p. 2). read et al. (2019) also noted a double gap in the training landscape, identifying that a “lack of satisfactory curricula” (p. 2) to train both librarians and researchers in rdm further contributed to the lack of rdm service offerings in biomedical fields.    engaging researchers with data management: the cookbook (clare et al., 2019), includes several case studies of rdm engagement and collaborations among researchers. the case studies demonstrate how librarians and other research support staff with disciplinary awareness can encourage researchers to consider research data management practices and services as an extension of their disciplinary peer communities. in one of the chapters focusing particularly on rdm training, papadopoulou and miller evaluate the format of training “mini-events” for their impact on building a community of rdm supports and data management best practices at the vilnius university library in lithuania. each of these mini-events (delivered either as half-day or full-day workshops) consisted of three incremental phases: familiarity of the participants with rdm support services; learning how to use various available tools; sharing research data in practice (papadopoulou & grabauskiene, 2019). papadopoulou and grabauskiene specify that one of the challenges faced by these rdm training sessions is reaching out to, and persuading, the uninterested researchers to attend. one proposed strategy is to do peer outreach rather than through a generic unit, such as information services. secondly, based on their study of a conference at the university of edinburgh, papadopoulou and miller propose that the events should include presentations by researchers from multiple university faculties. such presentations might discuss rdm best practices and their impact on researchers’ work, thereby encouraging their disciplinary peers to participate. thus, the presentations can also be interactive sessions among the researcher peers themselves (papadopoulou & miller, 2019).   approaches to research data management training   the previous section outlined challenges of rdm training such as the gap in terminology shared by research support staff and the researcher community that they support, and the researchers’ lack of interest in rdm if it is perceived to be beyond the scope of methodologies shared by their disciplinary community. these challenges support the research data management training landscape in canada: a white paper finding that pedagogical design needs to me mapped to trainee needs and is a necessary learning objective for librarians and other research support service providers (fry et al., 2017). this review has revealed multiple approaches to rdm training, specific to the trainee contexts. although we focus here on librarians and other research support staff as “trainees”, it is with an understanding that their training opportunities have an impact on the quality of rdm training and service provision available to researchers. further, this review notes several approaches to pedagogical design for rdm training, which can be broadly categorized as: generalized instruction or discipline-specific, single or multiple sessions, self-directed or instructor-led, in-person or online instruction (and most often, a hybrid of the two).    the literature shows that there are significant advantages to delivering discipline-specific or targeted rdm training. however, a generalized approach to rdm training may be favoured due to perceived scalability. as mentioned, read et al. (2019) note that available online training for librarians is inadequate to build rdm service capacity in biomedical fields, as none have the necessary disciplinary focus. this focus on general rdm training for librarians further contributes to a gap in disciplinary-specific training curricula for researchers. after reviewing humboldt university of berlin’s rdm initiative, launched as a joint venture between computer and media service, the research service centre, the university library, and the vice president for research, helbig (2016) similarly concludes, “although general workshops on research data management are more scalable in comparison to discipline-specific workshops, the advantages of a tailored approach outweighed this concern” (p. 2). humboldt university’s rdm training initiative consisted of one-day workshops aimed at helping phd students and researchers in the geography department. groups of six to eight trainees were formed in order to facilitate the learning process. rdm specialists at the university felt that a targeted approach would be advantageous. through a priori surveys and interviews with researchers and graduate students, the workshops were designed for the specific needs of that department. by understanding the nature of rdm in geography, specialists were able to provide an interactive session encouraging the full participation of the trainees. other universities such as monash university in australia, university of edinburgh in the united kingdom, and university of illinois, in the united states, offer courses to targeted campus groups based on their needs. such needs are identified through consultation with strategic research management services at these universities, as well as in-person discussions with individual researchers around the campus. bryant et al. (2018) explain that the integrated instruction model in a semester-long course is a preferable method because it is sustainable, as they observe, “the most resource-intensive approach to supporting rdm education is through in-person, instructor-led workshops” (p. 10). however, if a workshop approach is taken over a course integration approach, bryant et al. (2018), argue that rdm educational services should strategically align their workshops with course content and with broader institutional policies of the respective university (such as conforming to the requirements of data management plans).   within the literature, the choice between disciplinary focus or generalized curriculum models, is paralleled by the choice of delivery mode through online modules, in-person sessions, or a hybrid of the two. online training modules are among the most popular among rdm professionals because they are thought to allow flexibility for accommodating work schedules (tang & hu, 2019). read et al. (2019) note that the required time commitment is a strain on working librarians and there is a significant rate of non-completion of online training. read et al. (2019) also showed that while online modules improve the “understanding of and comfort level with rdm” in-person instruction resulted in “improved rdm practices” (p. 1). the differing experiences between online and in-person learning led read et al. (2019) to develop a hybrid, or “two-tier” coordinated approach to rdm training for health sciences librarians, and for biomedical researchers that the librarians will, in turn, train and support. there were seven self-paced, multi-media, online modules produced to train librarians. the modules covered general rdm topics and applications of rdm in health science methodologies and discipline-specific data standards. an evaluation form embedded at the end of each module was included for self-assessment. once a librarian indicated comfort with the content, they received a teaching toolkit which included a lesson plan and related materials to teach rdm to biomedical researchers via a 60-90 minute in-person session. this hybrid, coordinated model improved the librarian’s ability to deliver an rdm session for researchers; as read et al. (2017) observe, “the online modules were concise and directly tied to the teaching toolkit, a curriculum specifically created for use by the librarians to teach rdm locally, thus addressing the time constraints of working professionals…” (p. 8).   the learning objectives of online training options are improved when paired with in-person instruction. bryant et al. (2018) explain that the mantra research data management training modules, promoted on the website as “a free online course for those who manage digital data as part of their research project” (p.10), is a series of eight generic self-paced modules and tutorials that are supplemented by in-person training courses by rdm professionals, at the university of edinburgh. the online modules, initially built for researchers and graduate students, have influenced pedagogical design of rdm training for librarians and research support staff, not only at the host institution, but also for researchers and staff at other institutions. in 2013, mantra launched a diy training kit for librarians to facilitate the remote training modules. built for the uk research and funding environment, the course can be adapted locally to include online and in-person instruction, covering data management planning, organizing and documenting data, data storage, data sharing and ethics, and questions around data management. in a blog post, haddow (2014) writes of the experience of adapting and delivering the mantra diy training kit for librarians at the sterling university of edinburgh. according to haddow (2014), the subject librarian members of a dedicated local rdm task force, “found it beneficial to set time aside as a team to look at this issue;” (para. 2) however, they noted challenges and significant time investment for the local facilitator to adapt the course content. as haddow (2014) explains: “the instructions were sometimes not clear but by the end i figured out that i just needed to look at the manual.” (para. 4)    the “data intelligence 4 librarians course” was released in 2011 by 3tu.datacentrum, a partnership among three universities in the netherlands (the partnership was later called 4tu.researchdata (2020)). this course provides another example of a learning platform targeted to digital preservation professionals and included two in-person sessions at the beginning and the end of the training period. during the in-person sessions, coaches would teach the trainees, while during the online sessions, trainees were expected to be prepared for each unit and complete assignments by themselves or in pairs. throughout the online portion, trainees could reach out to their respective coaches through an established online platform. later, the course was transformed into “essentials 4 data support” whose target group was a more widely-defined group of professionals identified as “data supporters,” further defined as “people who support researchers in storing, managing, archiving and sharing their research data” (p. 244). trainees from multiple institutions attended and worked mostly in pairs, learning how to write research data plans for fictional scenarios. participant surveys and networking through online forums following the training were completed (grootveld & verbakel, 2015). feedback indicated that homework assignments were the most valuable element of the course, as the pairing of trainees led to enjoyable discussions. participants also appreciated learning from researchers, including how they deal with data management issues and about differences between disciplines (grootveld & verbakel, 2015). trainees admitted that the use of audio-visual elements was helpful for their learning experience. current versions of 4tu.researchdata consist of three variants: a combination of in-person sessions and online training platforms, supervised by coaches and open to online discussion forums; a self-directed, online course, open to online discussion forums; a self-directed, online course with no access to coaches or discussion forums.   a recent example of generalized, online rdm training includes the research data management librarian academy (rdmla), for librarians from multiple institutions around the globe (shipman & tang, 2019). the curriculum was based on needs gathered from interviews and a survey conducted by tang and hu (2019), as previously discussed, and its intent was to fill gaps training for librarians in higher education, through online training. although its success cannot be confirmed at this time, the online-only format of rdmla should be assessed in terms of its ability for librarian trainees to translate their knowledge into researcher training, in consideration of completion rates and the findings of studies on hybrid or in-person models. it is important to note that the rdmla training is underwritten by the publisher elsevier, with modules promoting tools in which elsevier has a vested interest, while the other trainings reviewed are developed through public or local institutional funding streams.   despite available online solutions to local training gaps, in-person instruction remains a popular approach, as it catalyzes communities of practice around complex skillsets. wittenberg et al. (2018) discussed workshops launched by a research data management team at the university of california in berkeley, and show that in-person, ongoing, and discipline-based consultations on rdm by specialized liaison librarians are among the most successful methods of rdm support by university libraries. as they mention, “participants, on average, were more satisfied with domain-based rdm training than they were with general rdm training” (p.328). at the same time, wittenberg et al. (2018) admit that the success of discipline-based training depends on a scientific community built around rdm, which is mainly based on continuous connections between liaison librarians and researchers.   the library carpentry workshops with rdm-focused content, as discussed by baker et al. (2016), are a worthwhile comparison to the online or hybrid teaching models available to librarians, due to the strong emphasis placed on in-person skill sharing and long-term community building. the multi-session workshop took place in the fall of 2015 over four, three-hour weekly evening sessions at the city university london centre for information science. the workshops had three aims: to blend non-library specific software skills training with existing library specific programs; to collect data on software skills in university libraries; and to build the foundations of a distributed community model for embracing and sustaining software skills in the library. prior to the sessions, attendees were asked to make a name badge, also identifying their level of knowledge of rdm and related software, for presenters to better guide the attendees. participants were also encouraged to note the level of knowledge of others to better assist them during the workshop, if needed. in this way, peer-to-peer collaborations were built into the workshop design. participants shaped workshop content. session one began with an introduction to basic programming concepts and attendees were asked to reflect on words and phrases associated with programming, code, and software from which they could benefit. baker et al. (2016) note that many universities around the world use “data carpentry workshops” formats and materials adapted to their local needs, which demonstrates the success of the project. however, they still recognize the need to develop a set of resources to enable workshop attendees to share software skills in their home libraries. it is anticipated that these resources would be predicated on the idea that the best way to reinforce one’s own software skills is through teaching others.   rdm training as a means of community building   research data management training landscape in canada: a white paper (2017) outlined eight principles for developing a coordinated national training curriculum. several of these principles foreground the community of practice approach adopted by the librarian-led portage network rdm expert groups. the notion of rdm as a set of skills and practices shared by a community, whether disciplinary, institutional, professional, or otherwise, is consistent with several of the articles reviewed in this paper, as well as the “data communities” model of researcher behaviour in data sharing, described by danielle cooper and rebecca springer (2019). however, while communities of practice may be wrapped in a myth of informal organizing, in reality they require leadership and intentional cultivation, particularly as etienne and beverly wenger-trayner (2015) observe, if they are used for developing the “strategic capability” of an organization or its personnel. indeed, the strategy of nurturing national rdm infrastructure, training, and support by “building partnerships in the face of complexity” has been carefully crafted by portage since its early stages (humphrey, 2020, p. 2). from this perspective, rdm librarians and other research support staff have a key role in training, as universities develop capacity to comply with rdm requirements of national and international funding agencies. for this reason, this literature review will conclude with the seven principles of rdm training developed at tu delft (2019), as well as new approaches to librarian rdm training that build upon the intersections of research data management with the workflows, best practices, and scholarly communities of open science.   the tu delft (2019) principles provide a framework whereby rdm training becomes the mechanism for cultivating a community of practice that is both campus-wide and disciplinary-focused, while reaching beyond the campus into the information circuits of the scholarly community. significantly, these principles encourage a researcher-focused rdm vocabulary; they foster collaboration between faculty and research support staff across multiple university departments and service providers; and furthermore, there is recognition that the university must provide meaningful incentives that motivate trainees, whether they are administrators, librarians, research support staff, researchers, or students, to join the community of practice. the tu delft “open working” website (2019) outlines some principles including: “whenever possible, data and software management training should be built upon the existing faculty-specific courses”; “building and delivering such training must be a collaborative effort between faculties, the library, graduate school and other university services”; and “library and graduate schools should continuously engage in consultation processes with phd students and researchers.” (para. 2, 4, 6) at the same time, the principles recommend engagement with organizations outside universities as vital in making training resources sustainable. in order to successfully implement this vision, the tu delft principles recognize that researchers must receive the proper incentives to participate and contribute to the training. the library should also solicit feedback from researchers to iteratively improve and update the training content. finally, the principles reinforce that courses should be accompanied by clear learning objectives, a lesson plan, and a description of the methods selected for the training (tu delft, 2019).   looking forward, one can imagine integrated training for librarians and researchers that establishes rdm as the foundation for data-sharing workflows and other best practices of open science scholarly communications. the international principles of fair data, findability, accessibility, interoperability and reuse, can be a shared method between cross-disciplinary open scholarship practices due to a common engagement with digital assets. as higman et al. (2019) argue, “researchers often want to be fair, and sometimes open; they are noble aspirations... by using the language of fair and open, we can engage people in data management too” (p. 2). the bodleian libraries at the university of oxford offers a model of how the integration of rdm training with other areas of open scholarship might be achieved for librarians. library rdm services are led by one specialist who has developed an rdm training series for researchers addressing key issues, such as working with confidential data, secondary use of data, and data deposit and preservation. this training series is often team-taught with it representatives or library staff with complementary expertise, highlighting the need for researchers to first contact their subject librarians with queries. rdm platforms are also supported by multiple members of library staff, not only the rdm specialist. the collaborative approach to rdm training for researchers, and a distributed technical rdm service “serves to reinforce the message of the training aimed at library staff, namely that rdm is an area that library staff across the board can support to some extent” (southall & scutt, 2017, p. 307). rdm training for librarians and library staff mirrors the content of training for researchers. there are two workshops that cover basic principles of rdm, trends in scholarly communications, and concrete examples of data management, with an emphasis placed on an “increased understanding of digital scholarship, rdm issues and where these sit in relation to the work of the academic library and new areas of scholarly activity such as open access (southall & scutt, 2017, p. 308).       conclusion   the aim of this literature review of 13 sources, containing 14 case studies, was to survey a range of rdm training and capacity-building approaches, in order to determine the next steps for our own local context at the university of ottawa. a method of looking at international training models was used in order to identify and supplement a gap in the emerging national rdm policy, infrastructure, and training environment. for instance, a notable challenge in the canadian rdm training space is that many institutions have not yet developed the rdm institutional policies that are anticipated by the draft tri-agency research data management policy (2018). this layer of institutional strategy will enable the building of rdm into graduate-level curricula for both researchers and librarians. in the meantime, the next steps for building a training program at our local institution will begin with establishing a set of principles based on the findings of this literature review. the plan is to align these principles with the training methods, modes of assessment, and infrastructure development timeline outlined in a national training strategy anticipated for release in fall 2020 by the portage training expert group as a follow-up to the 2017 white paper, tentatively titled, building a portage network training strategy: a canadian approach to research data management.   the following trends emerged though this literature review, which have informed the national training strategy, and will be taken into consideration when building local training options for librarians, research support staff, and researchers at the university of ottawa. librarians and researchers must have enough incentive to undertake training in rdm or to join a community of practice. training requires a significant investment of time, whether online or in-person, and librarians are unlikely to take on additional training, or to complete the training once enrolled, without a perceived benefit or reinforcement through regular rdm service provision. disciplinary-specific instruction is preferable over general instruction for both librarians and researchers, however, a librarian’s own training opportunities will influence their ability to provide discipline-specific rdm instruction to researchers. there is a double gap in the training landscape, as the lack of disciplinary-specific training opportunities for librarians further contributes to a lack of training options and service offerings for distinct research areas.   the range of pedagogical designs reflected in the case studies make it difficult to draw conclusions as to whether intensive events, or a series of shorter time-commitments over a longer time period, is preferable for learning outcomes. in-person training opportunities emerged as the preferred option for learning retention and secondary effects of building a community of practice. for the same reasons, online instruction was found to be most effective when paired with an in-person component. the sources in this literature review predate the global covid-19 pandemic, which has shifted higher-education into online delivery in historically unprecedented ways. this context may present an opportunity to apply the best practices of online learning design to close the gap between the benefits of in-person training and the low retention in online learning environments. initiatives such as the university of british columbia’s rdm fall series 2020 are early responses to virtual rdm instruction during the pandemic, demonstrating the importance of the adaptation to local contexts, for example. the literature review highlights the recommendation that in order to be meaningful, generalized rdm training offered by third parties must be adapted to local contexts. discipline-specific training, in-person training, and adaptation to local contexts are all resource intensive activities but they are worth the investment. librarians and other research support staff with disciplinary awareness will be more successful as they engage with researchers and help them to adopt research data management practices as an extension of their disciplinary peer communities. finally, future directions for rdm training will be integrated into open access and digital scholarship awareness training, as well as cross-disciplinary, open science communities of practice that reach beyond local campuses.   author contributions statement   felicity tayler: conceptualization (lead), methodology (lead), writing – original draft (lead) review & editing (equal) maziar jafary: methodology (supporting), writing – original draft (supporting), writing review & editing (equal)   acknowledgements   the authors would like to thank chantal ripp, jane fry, james doiron, lindsey sikora and kim powroz for valuable feed-back on drafts of this review.   references   4tu research data. (2020). https://researchdata.4tu.nl/en/   baker, j., moore, c., priego, e., alegre, r., cope, j., price, l., stephens, o., strien, d. van, & wilson, g. (2016). library carpentry: software skills training for library professionals. liber quarterly, 26(3), 141–162. https://doi.org/10.18352/lq.10176   bryant, r., lavoie, b., & malpas, c. (2018). sourcing and scaling university rdm services (the realities of research data management, part 4). dublin, oh: oclc research. https://www.oclc.org/research/publications/2017/oclcresearch-rdm-part-four-sourcing-scaling.html   clare, c., cruz, m., papadopoulou, e., savage, j., teperek, m., wang, y., witkowska, i., & yeomans, j. (2019). engaging researchers with data management: the cookbook. open book publishers. https://openresearchlibrary.org/content/03b1aed1-e57e-4305-ae7c-27c9d88b02fd/details   cooper, d., & springer, r. (2019, may 13). data communities: a new model for supporting stem data sharing (issue brief). ithaka s+r. https://sr.ithaka.org/wp-content/uploads/2019/05/sr-issue-brief-data-communities-05132019.pdf   fry, j., doiron, j., létourneau, d., perrier, l., perry, c., & watkins, w. (2017). research data management training landscape in canada: a white paper, prepared by the portage training expert group on behalf of the canadian association of research libraries (carl). http://dx.doi.org/10.14288/1.0372048     grootveld, m. j., & verbakel, e. (2015). essentials for data support: training the front office. international journal of digital curation, 10(1), 240-248.  https://doi.org/10.2218/ijdc.v10i1.364   haddow, l. (2014, january 9). training subject librarians in rdm. edinburgh research data blog. http://datablog.is.ed.ac.uk/2014/01/09/training-subject-librarians-in-rdm/   helbig, k. (2016). research data management training for geographers: first impressions. isprs international journal of geo-information, 5(4), 1-9. https://doi.org/10.3390/ijgi5040040   higman, r., bangert, d., & jones, s. (2019). three camps, one destination: the intersections of research data management, fair and open. insights, 32(1), 1-9. https://doi.org/10.1629/uksg.468   humphrey, c. (2020). the carl portage partnership story. partnership: the canadian journal of library and information practice and research, 15(1), 1-7. https://doi.org/10.21083/partnership.v15i1.5825   papadopoulou, e. & miller, k. (2019). ‘dealing with data’ conference at university of edinburgh. in clare, c., cruz, m., papadopoulou, e., savage, j., teperek, m., wang, y., witkowska, i., & yeomans, j., engaging researchers with data management: the cookbook (pp. 70-72). cambridge, uk: open book publishers.    papadopoulou, e. & grabauskiene, r. (2019). duodi: the ‘days of data’ at vilnius university. in clare, c., cruz, m., papadopoulou, e., savage, j., teperek, m., wang, y., witkowska, i., & yeomans, j., engaging researchers with data management: the cookbook (pp. 74-76). cambridge, uk: open book publishers.    read, k. b., larson, c., gillespie, c., oh, s. y., & surkis, a. (2019). a two-tiered curriculum to improve data management practices for researchers. plos one, 14(5), 1-14. https://doi.org/10.1371/journal.pone.0215509   shipman, j. p., & tang, r. (2019). the collaborative creation of a research data management librarian academy (rdmla). information services & use, 39(3), 243–247. https://doi.org/10.3233/isu-190050   southall, j., & scutt, c. (2017). training for research data management at the bodleian libraries: national contexts and local implementation for researchers and librarians. new review of academic librarianship, 23(2–3), 303–322. https://doi.org/10.1080/13614533.2017.1318766   surkis, a., & read, k. (2015). research data management. journal of the medical library association, 103(3), 154–156. https://doi.org/10.3163/1536-5050.103.3.011   tang, r., & hu, z. (2019). providing research data management (rdm) services in libraries: preparedness, roles, challenges, and training for rdm practice. data and information management, 3(2), 84–101. https://doi.org/10.2478/dim-2019-0009   tu delft. (2019, october 25). vision for research data management training at tu delft. open working. https://openworking.wordpress.com/2019/10/25/vision-for-research-data-management-training-at-tu-delft/   wenger-trayner, e., & wenger-trayner, b. (2015). communities of practice: a brief introduction. wenger-trayner. https://wenger-trayner.com/wp-content/uploads/2015/04/07-brief-introduction-to-communities-of-practice.pdf   wittenberg, j., sackmann, a., & jaffe, r. (2018). situating expertise in practice: domain-based data management training for liaison librarians. the journal of academic librarianship, 44(3), 323–329. https://doi.org/10.1016/j.acalib.2018.04.004         classics   cummings, merrill, and borrelli’s inquiry into small screen use by academic library users: timing is everything   a review of: cummings, j., merrill, a., & borrelli, s. (2010). the use of handheld mobile devices: their impact and implications for library services. library hi tech, (28)1, 22-40. https://doi.org/10.1108/07378831011026670   reviewed by: dr. catharine bomhold associate professor school of library and information science 118 college drive, box 5146 university of southern mississippi hattiesburg, mississippi, united states email: c.bomhold@usm.edu   received: 6 sept. 2017    accepted: 30 nov. 2017      2018 bomhold. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29338     abstract   objective – the authors undertook this study to understand the relatively new phenomenon of handheld computing and the use of small-screen devices among academic library users. they sought to determine if users would be inclined to search the online library catalogue on their devices and, by extension, if there would be a growing demand for small-screen compatible library services.   design – online and paper surveys were used with both closed and open questions. respondents included students, faculty, and staff at washington state university (wsu).   setting – washington state university library, pullman, washington, united states of america.   subjects – the survey was open to any user of the washington state university (pullman) library. the 206 respondents included 126 (61.2%) undergraduates, 26 (12.6%) graduate or professional students, 32 (15.3%) wsu employees, and 15 (7.3%) faculty members.   methods – a survey was distributed both online and on paper. the online version used surveymonkey.com and participation was solicited through various social media. it was open for three months during the spring semester, 2007. the paper version was distributed to all library users on two days in june 2007. eighty-four online and 122 paper responses were received.   main results – most of the respondents (58.4%) who owned a personal digital assistant (pda) or web-enabled cell phone (wecp) indicated that they would search the library catalogue on a small-screen device. responses to the open question “how would you use the opac [online public access catalogue] if it was available on a pda or wecp?” were mixed, both positive and negative. the positive responders noted the possible time savings associated with the availability of more information on their devices. the negative responders noted the cost of data, the annoyance of public phone use, and the complex format of the current catalogue that would not transfer to a small screen.   conclusion – the authors cited the growing usage trends in handheld devices, along with the willingness of current owners to use their devices, to predict an increase in usage of small screen searching. they speculated that further research should investigate how small screens would be used and what would that experience look like, rather than if patrons would use them.   commentary   at a time when 77% of the population owns an applications-based smartphone, (pew research center, 2017) it is difficult to remember when such devices were not omnipresent. even harder to recall may be the brief period of time from 2002 to 2007 when “web-enabled” cell phones and personal digital assistants were used to access the internet through general packet radio services (gprs) or wireless application protocol (wap) technology (“the evolution of cell phone design”, 2009). yet this was the state of technology when cummings et al. conducted their research study. data collection was conducted from february to may 2007, in the few months between announcement of the forthcoming production of the iphone in january and its release in june of that year. unsurprisingly, the amount of research on smartphone use and its impact on library services has grown exponentially in the years since, and many of the researchers base their studies on the suggestions of cummings et al.   to be clear, the authors did not present their research as revolutionary, but as a part of a growing interest in the use of handheld devices driven by significant increases in ownership. cummings et al. cited previous studies on the use of pdas for information seeking (carney, koufogiannakis, & ryan, 2004; tenopir, king, clarke, na, & zhou, 2007; spires, 2008); the technical challenges for libraries to support the new technology (deneen & allert, 2003; garrison, anderson, macdonald, schardt, & thibodeau, 2003; peters, dorsch, bell, & burnette, 2003; good, 2007); and changes in physical environments (cowart, 2006; duncan, 2006; evans, 2006; whelan, 2007; lever & katz, 2007) as precursors to their study. together, these studies form a nucleus of early research on the possibilities, problems, and promises of information use through hand-held devices. cummings et al.’s paper stands out, not only because it coincided with the advent of the apple iphone and its counterparts, but also because the research questions were broad enough to be applied to a variety of subsequent studies.   while this study was not couched in theory, and the approach was a customary mixed method survey, the findings provided a baseline for subsequent research articles and justification for further investigations. cummings et al. stated that their intent was to both “measure whether or not people wanted to access the … opac with a small screen, but also … [to] gain a broader understanding of handheld mobile computing’s impact on academic libraries” (pp. 25-26). to determine how the research was used, citing articles were examined to understand the impact. in june 2017, google scholar listed 102 citations for this paper. of those, 35 were unique research articles in english. in addition to research in foreign languages, other factors for dismissal included lack of availability, descriptive book chapters, dissertations, and master’s level research. while cummings et al.’s article is a broad study that does not limit its scope to a particular audience or type of device, subsequent articles reveal a breadth of research that can be organized into three dominant streams that reflect the findings and suggestions of the original research: device ownership and barriers to use, user behavior, and service surveys.   device ownership and barriers to use   repeated topics addressed within this paper include: who owns small-screen devices, what type of devices are used, how they are used, and barriers to using them to their fullest extent. cummings et al.’s first research question investigated the participants’ desire to access academic information on a small screen. the survey also inquired about ownership, actual use, and the intent or desire to use handheld devices for accessing academic information. further research studies have expanded on these questions to include new types of devices and what barriers users encounter that prevent usage. song and lee (2012) inquired into how international students in the united states used their devices. among the participants, ownership of mobile technology, including smartphones, electronic readers (e-readers), and tablets, was high (82%), but they were mainly used for communication and social networking. there was a marked lack of interest in e-readers among this group as well (song & lee, 2012). the majority of students and faculty in kisii, kenya used mobile devices to access the opac (72.5%), electronic books (e-books) (77.5%), and the library website (74.75%); significantly fewer (52.5%) used the devices to access full-text journal articles (george, maina, & wanangeye, 2016).   the usability of small screens for research or academic work is frequently noted as a barrier.  halevi, moed, and bar-ilan (2015) and madhusudhan (2015) looked at information use on small screens by academics. while searching for information on mobile devices was common, most researchers preferred to download and print material in portable document format (.pdf) to interact with it (halevi et al., 2015). library and information science students in hong kong, japan, and taiwan restricted their smartphone use to search engines for school work, not library resources; they did not use the devices for academic reading (ko, chiu, lo, & ho, 2015). the lack of a mobile-ready website for the library was a barrier to use of resources by the students (ko et al., 2015). respondents in croatia who owned a smartphone or tablet, also expressed the need for interfaces customized to those devices (pažur, 2014).   investigations into screen size as a barrier to use have led to questions about other barriers, such as connection speed and access to quality resources. while the devices make it easier and faster to find research material (madhusudhan, 2015), art and design students considered connection speed for downloading information a primary barrier to academic use (lo et al., 2016). while the students were all active users of their smartphones, use of the devices for academic work was limited because of slow connection time (lo et al., 2016).   a lack of instruction can also be perceived as a barrier. for example, bushhousen et al. (2013) asked health sciences library patrons about potential use and perceived barriers. the students felt that their ability to use the devices was restricted by a lack of instruction on the specific apps, however they were eager to use the devices because they were required by the medical programs that they were enrolled in. the team of researchers and technology specialists found that education on app use was high on the list of required services by their patrons (bushhousen et al., 2013).   user behavior   the questions proposed by cummings et al. regarding users’ experiences and their intent to “gain a broader understanding of … [the] impact on academic libraries” (p. 26), is reflected in studies on user behavior. research into user behavior considers if respondents use mobile services, how they are used, and which applications, sites, or functions are most commonly accessed. an early study explored the strategies used for searching on smartphones; it found high use of new input tools such as voice, global positioning system (gps), barcode, and quick response (qr) codes (yarmey, 2011). android and iphone users considered themselves first adopters, ahead of their peers in information use, and considered themselves well aware of the need to evaluate the sources of information that they used (yarmey, 2011). the majority of undergraduate users reported employing the devices for academic work, but the most common types of apps used were search engines, and apps for websites that the user was already familiar with from their full-sized devices (bomhold, 2013).   dresselhaus and shrode (2012) examined smartphone use by different types of students; fifty-four percent of undergraduates and 50% of graduates reported using mobile devices for academic work. the highest use by students (63%) was in the college of business followed by 59% of engineering students (dresselhaus & shrode, 2012). art and design undergraduate students used their mobile devices no differently than their peers in other colleges, aside from image and audio-visual needs (lo et al., 2016). while they were all active users of their smartphones, use of the devices for academic work was limited, and they were only interested in library services of an administrative nature, such as hours, requests, check-outs, and renewals (lo et al., 2016). library and information science students in asia restricted their smartphone use to search engines for school work, not library resources, and did not use the devices for academic reading (ko et al., 2015). similarly, undergraduate library and information science students in greece preferred laptops and personal computers over mobile devices for school work and library use, but this preference was inverted when the students were looking for entertainment (vassilakaki, moniarou-papaconstantinou, & garoufallou, 2016).   inquiry into potential and current use by academic library patrons was also used to rationalize the implementation of mobile-based library services. students in bangladesh were “very interested” in mobile services, especially in administrative functions such as texts for overdue messages and reminders (elahi & islam, 2014). furthermore, engineering library patrons indicated that they felt mobile access would increase the use of services outside the library. interest in e-books and online journals was high (74.2% and 67.74% respectively), but interest in using the online catalogue (25.8%) and databases was low (38.7%) (kumar, 2013).   service surveys   in trying to understand the broader impact of hand-held computing on academic libraries, cummings et al. discuss the adaptations libraries were making to their online services, in order to make them more accessible to mobile users. in particular, they mention that most libraries had adopted a “tailored” approach to designing mobile services (p. 34). they also stated that “the question for libraries will become not will users access library services through mobile devices, but what type of experience will the eventual user have and what library services will be available to them” (p. 34).   the impact of this statement has been extensive. several surveys have been published that investigate what types of research services are available on mobile platforms. an early assessment by canuel and crichton of mobile library services available at canadian university libraries revealed that only a very small fraction (14%) offered some kind of mobile web presence, most often mobile-ready websites. functionality, design, and intuitive usability made native apps more appealing to users over mobile-ready websites. a dichotomy between the need for services as reported by the users and the actual availability of that service was found in a survey of the usability of mobile web interfaces of academic libraries (canuel and crichton, 2011). in the rush to provide mobile services, han and jeong (2012) concluded, libraries were neglecting the needs of those they were trying to serve by not asking the users.   to evaluate library responsiveness to researchers’ needs for mobile information and their ability to provide adequate services for research on the fly, bomhold (2015) surveyed libraries at carnegie-rated research universities (ru) and very heavy research universities (vh) universities in the u.s. while there was a three-fold increase of available mobile services in just three years, the types of services offered demonstrated a lack of consensus among the libraries as to what those services should be or should look like. the opac was the most commonly offered, followed by article databases, and assistance from a librarian (dresselhaus & shrode, 2012), and libraries abroad provided mobile services similar to those in the united states, including unique mobile sites or applications, the opac, text messaging for both notifications and reference, and qr codes (kubat, 2017).   functionality as determined by the design of the application was the best predictor of student use. du (2015) surveyed library websites and reference services, and included learning management systems (lms) such as blackboard. to determine if the app was effective for their use, students compared what was available on the app to the full version that they were familiar with (du, 2015). in a study of mobile use of library services by university faculty, staff, and students, half of those surveyed accessed the library catalogue and databases through their mobile devices, but the usability of the apps was considered a barrier to using them effectively (caniano & catalano, 2014). in a survey used to determine the feasibility of using a specific mobile app for providing different types of information to their users, miller, vogh, and jennings (2013) concluded that successful implementation depends on the simplicity of design of the app as well as careful curation of available resources.   moving forward   increasingly, researchers are shifting their focus from potential use of mobile services and user behavior to applying information theories to improve technology for mobile devices. the technology acceptance model (davis, 1989) was applied to student use of apps. in addition to perceived usefulness and perceived ease of use, the quality of the service, that is, the actual execution of the technology, plays a significant role in students’ adoption of mobile library services (adil, izhar, & khajazi, 2016). quality of service as a factor in adoption also appears when information systems theory is applied to student use of electronic bookshelves (chiu, chao, kao, pu, & huang, 2016).   without diminishing the quality of the research or the work of cummings et al., the primary significance of this article can be credited to its timeliness. the authors were prescient in their investigation of the potential use of small screen, hand-held devices for finding information, and were able to collect data shortly before the release of the first iphone, which ushered in the era of the smartphone. the number of articles that cite this research continues to grow as the prevalence of smartphones in society rises. in 2011, the first-year data was collected, 35% of americans owned smartphones; by november 2016, that number had skyrocketed to 77%, with 95% owning some kind of cellular phone (pew research center, 2017). likewise, the number of articles investigating the use of such devices continues to climb.   the broad questions asked about the potential use of any small-screen device by any patron of the academic library, have given way to a progression of more focused studies in subsequent research. a review of the literature demonstrates a pattern of increasingly specific questions. the original article asked very broad questions about “small screen” (p. 23) ownership and use with any device by any patron of an academic library. it was also predictive, looking at what the respondents “might” do (p. 29) if the technology was available. after the introduction of the iphone in 2007 (“the evolution of cell phone design”, 2009) and the android operating system in 2008 (german, 2011) smartphones are now pervasive, and most research proceeds on the assumption that this is the device that will be used to access mobile information. the ideas put forth by cummings et al. have been taken up and narrowed in focus, creating a profuse body of literature for practitioners. the question of potential use has developed into questions on current use behaviors, barriers to use, and explanations of how to make apps more efficient and effective in order to encourage adoption by patrons in situ, outside of the library, and on the go.   references   adil, t., izhar, t., & khajazi, n. n. i. m. (2016). framework based mobile library usage and success factor: review and extention concept. international journal for innovative research in multidisciplinary field, 2(8), 283-290. https://www.researchgate.net/profile/tengku_adil_tengku_izhar/publication/308117293_framework_based_mobile_library_usage_and_success_factor_review_and_extention_concept/links/57db4b8908ae5292a376a49e.pdf anbu, k. j. p., & kataria, s. (2015, jan.). access to library resources through portable devices: a pre-design prototype for creating library websites. in 4th international symposium on emerging trends and technologies in libraries and information services (ettlis). (1-5). https://doi.org/10.1109/ettlis.2015.7048162   bomhold, c. (2013). educational use of smartphone technology: a survey of mobile phone application use by undergraduate university students. program, 47(4), 424-436. https://doi.org/10.1108/prog-01-2013-0003   bomhold, c. (2015). research and discovery functions in mobile academic libraries: are university libraries serving mobile researchers? library hi tech, 33(1), 32-40. https://doi.org/10.1108/lht-09-2014-0084   bushhousen, e., norton, h. f., butson, l. c., auten, b., jesano, r., david, d., & tennant, m. r. (2013). smartphone use at a university health science center. medical reference services quarterly, 32(1), 52-72. https://doi.org/10.1080/02763869.2013.749134   caniano, w. t., & catalano, a. (2014). academic libraries and mobile devices: user and reader preferences. the reference librarian, 55(4), 298-317. https://doi.org/10.1080/02763877.2014.929910   canuel, r., & crichton, c. (2011). canadian academic libraries and the mobile web. new library world, 112(3/4), 107-120. https://doi.org/10.1108/03074801111117014   carney, s., koufogiannakis, d., & ryan, p. (2004). library services for users of personal digital assistants: a needs assessment and program evaluation. portal: libraries and the academy, 4(3), 393-406. https://doi.org/10.1353/pla.2004.0043   chiu, p. s., chao, i. c., kao, c. c., pu, y. h., & huang, y. m. (2016). implementation and evaluation of mobile e-books in a cloud bookcase using the information system success model. library hi tech, 34(2), 207-223. https://doi.org/10.1108/lht-12-2015-0113   cowart, e. (2006). take it outside. library journal, 131(14), 50. available from: academic search premier, ipswich, ma. accessed january 18, 2018.   cummings, j., merrill, a., & borrelli, s. (2010). the use of handheld mobile devices: their impact and implications for library services. library hi tech, 28(1), 22-40. https://doi.org/10.1108/07378831011026670   davis, f. d. (1989). perceived usefulness, perceived ease of use, and user acceptance of information technology. mis quarterly, 13(3), 319-340. retrieved from: https://misq.org/perceived-usefulness-perceived-ease-of-use-and-user-acceptance-of-information-technology.html   deneen, l., & allert, j. (2003). hand-held computers in the classroom and the library: teaching and learning resource issues resulting from widespread deployment at the university of minnesota duluth. library hi tech, 21(4), 419-425. https://doi.org/10.1108/07378830310509727   dresselhaus, a., & shrode, f. (2012). mobile technologies & academics: do students use mobile technologies in their academic lives and are librarians ready to meet this challenge? information technology and libraries, 31(2), 82-101. https://doi.org/10.6017/ital.v31i2.2166   du, y. (2015). information use and barriers on a mobile app in distance learning. journal of library & information services in distance learning, 9(3), 204-220. https://doi.org/10.1080/1533290x.2015.1052608   duncan, m. (2006). courtesy is key. library journal, 131(14), 50. available from: academic search premier, ipswich, ma. accessed january 18, 2018.   elahi, h., & islam, s. (2014). go fast, go with mobile: students perception on implementing mobile based library services at dhaka university library. library philosophy and practice, 1197. retrieved from: http://digitalcommons.unl.edu/libphilprac/1197   evans, w. (2006). phones are everywhere. library journal, 131(12), 47. available from: library & information science source, ipswich, ma. accessed january 18, 2018.   garrison, j., anderson, t., macdonald, m., schardt, c., & thibodeau, p. (2003). supporting pdas: the experience of a health sciences library. library hi tech, 21(4), 412-418. https://doi.org/10.1108/07378830310509718   george, b. o., maina, j., & wanangeye, l. w. (2016). analyzing the use of mobile phone technology in access and utilization of library resources in mount kenya university, kisii campus. world journal of computer application and technology, 4(1), 15-21. https://doi.org/10.13189/wjcat.2016.040103   german, k. (2011). a brief history of android phones. retrieved from: https://www.cnet.com/news/a-brief-history-of-android-phones/   good, s. (2007). pda serials: practical and policy issues for librarians. the acquisitions librarian, 19(1-2), 145-160. https://doi.org/10.1300/j101v19n37_12   halevi, g., moed, h., & bar-ilan, j. (2015). accessing, reading and interacting with scientific literature as a factor of academic role. publishing research quarterly, 31(2), 102-121. https://doi.org/10.1007/s12109-015-9404-9   han, h., & jeong, w. (2012). mobile web interface of academic libraries. proceedings of the association for information science and technology, 49(1), 1-4. https://doi.org/10.1002/meet.14504901279   ko, e. h., chiu, d. k., lo, p., & ho, k. k. (2015). comparative study on m-learning usage among lis students from hong kong, japan and taiwan. the journal of academic librarianship, 41(5), 567-577. https://doi.org/10.1016/j.acalib.2015.07.005   kubat, g. (2017). the mobile future of university libraries and an analysis of the turkish case. information and learning science, 118(3/4), 120-140. https://doi.org/10.1108/ils-09-2016-0063   kumar, k. (2013). application of mobile technology among engineering college library users: a study. community of knowledge. retrieved from: http://www.community-of-knowledge.de/fileadmin/user_upload/attachments/application_of_mobile_technology_among_engineering_college_library_users_a_study.pdf   lever, k., & katz, j. (2007). cell phones in campus libraries: an analysis of policy responses to an invasive mobile technology. information processing & management, 43(4), 1133-1139. https://doi.org/10.1016/j.ipm.2006.07.002   lo, p., cho, a., leung, m. h., chiu, d. k., ko, e. h., & ho, k. k. (2016). use of smartphones by art and design students for accessing library services and learning. library hi tech, 34(2), 224-238. https://doi.org/10.1108/lht-02-2016-0015   madhusudhan, m. (2015). use of mobile devices for improving the research work by research scholars of the university of delhi and university of hyderabad: a study. world digital libraries an international journal, 8(2), 127-144. https://doi.org/10.18329/09757597/2015/8204   miller, r. e., vogh, b. s., & jennings, e. j. (2013). library in an app: testing the usability of boopsie as a mobile library application. journal of web librarianship, 7(2), 142-153. https://doi.org/10.1080/19322909.2013.779526   pažur, i. (2014). attitude of the rudjer boškovic institute’s scientists to the small screen mobile devices library services: a user survey. library hi tech, 32(4), 628-644. https://doi.org/10.1108/lht-03-2014-0025   peters, t., dorsch, j., bell, l., & burnette, p. (2003). pdas and health sciences libraries. library hi tech, 21(4), 400-411. https://doi.org/10.1108/07378830310509709   pew research center. (2017). mobile fact sheet. retrieved from: http://www.pewinternet.org/fact-sheet/mobile/   song, y. s., & lee, j. m. (2012). mobile device ownership among international business students: a road to the ubiquitous library. reference services review, 40(4), 574-588. https://doi.org/10.1108/00907321211277378   spires, t. (2008). handheld librarians: a survey of librarian and library patron use of wireless handheld devices. internet reference services quarterly, 13(4), 287-309. https://doi.org/10.1080/10875300802326327   tenopir, c., king, d., clarke, m., na, k., & zhou, x. (2007). journal reading patterns and preferences of pediatricians. journal of the medical library association, 95(1), 56-63. retrieved from: https://www.ncbi.nlm.nih.gov/pmc/articles/pmc1773049/pdf/i0025-7338-095-01-0056.pdf   the evolution of cell phone design between 1983-2009. (2009). web designer depot. retrieved from: https://www.webdesignerdepot.com/2009/05/the-evolution-of-cell-phone-design-between-1983-2009/   vassilakaki, e., moniarou-papaconstantinou, v., & garoufallou, e. (2016). identifying the uses of mobile technology among library and information science undergraduate students. program, 50(4), 417-430. https://doi.org/10.1108/prog-10-2015-0069     whelan, d. (2007). school cell phones: lock em up. school library journal, 53(2), 117. available from: library & information science source, ipswich, ma. accessed january 18, 2018.   yarmey, k. (2011). student information literacy in the mobile environment. educause quarterly, 34(1). https://er.educause.edu/articles/2011/3/student-information-literacy-in-the-mobile-environment   evidence based library and information practice using evidence in practice   silence in a noisy world: using student feedback to enhance library silent study space   john stemmer director of the library w. l. lyons brown library bellarmine university louisville, kentucky, united states email: jstemmer@bellarmine.edu   michael g. strawser assistant professor nicholson school of communication and media university of central florida orlando, florida, united states email: michaelstrawser@ucf.edu   received: 3 may 2019                                                                     accepted: 29 july 2019      2019 stemmer and strawser. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29581   setting   bellarmine university library provides services and support to the university’s roughly 3,200 fte students (approximately half living on-campus). undergraduates are about 2,500 of the student body with graduate programs contributing about 700 students. bellarmine university is a private, catholic university located in louisville, ky. over 80% of bellarmine’s undergraduate students attend full-time and are under 25 years of age. the library building is not home to only the library; other units located in the building include the thomas merton center,   information technology, the student success center, disability services, and the president’s office. the four-floor building is heavily used with a gate count of almost 300,000 (297,572) during the 2017-18 academic year. in 2014-15, the first floor was renovated to create the lansing learning commons, which was intended to provide a student-centered collaborative learning space with technology support. at the same time, staff space on the second floor—the quiet study floor—was reduced, doubling the individual seating, and old-style, individual study carrels from the first floor were repurposed on this quiet floor.   problem   in 2017-18, the president’s office and boardroom, on the second floor, were slated to move out of the building, resulting in the availability of boardroom space to be reassigned for new functions. at the university, space is always in high demand. given that the university had conducted a major renovation of the library specifically to address student needs, it was now looking to address the space needs of a number of programs and projects, such as creating more classrooms, faculty development space, or the veterans affairs office among others. not all proposed uses would be compatible with the quiet nature of the floor, and this was a major concern of the library. the library desired to convert the space into a modern quiet study space. the library had some anecdotal indications that the students would use more quiet space, but the library needed to be able to convey that this plan was the best use of the space.   evidence   to address the problem, the library needed to demonstrate that there was an ongoing student desire for quiet space. anecdotal staff observations would not be sufficient; more compelling evidence of student demand would need to be identified or collected. the library reviewed the data from two sources that could convey student opinion on the question of quiet space. we used data collected through a library satisfaction survey and from the headcount data routinely collected by the library. the survey was concerned with three primary guiding questions:   ·         what is the perceived student satisfaction level of library services? ·         in what ways is the library most used? ·         what library services are most valuable for student stakeholders?   a total of 574 (n = 574) participants completed the survey. participants were asked various questions related to their perceptions of the institution’s library. pertinent questions are included below however, to aid in clarity, tables have been created for responses to negate narrative representations of the results and hopefully increase clarity.   students were asked to select reasons why they came to the library. students could select more than one answer. results are in table 1.   table 1 reasons you come to the library   options     count   percentage to check out books 167 29% to check out media 96 17% to locate journal/newspaper articles 125 22% to get help with research papers or other course assignments 197 34% to read newspapers or current magazines 19 3% to use items placed on reserve by your professor 111 19% to use media equipment 37 6% to study alone* 473 82% to study with a group* 367 64% to use the group study rooms* 367 64% to use the second-floor quiet study area* 279 49% to use a printer, photocopier 433 75% to use the computers for academic purposes 359 63% to use the computers for recreational/personal use 60 10% to use a mac computer 64 11% to visit the “ask us” desk for research assistance 82 14% to visit the technology support center 141 25% to visit the student success center for writing assistance 97 17% to visit the student success center for advising 139 24% to visit the student success center for tutoring 202 35% to visit disability services 46 8% to visit the merton center 55 10% to meet friends 207 36% to look for information in online databases 197 34% i have class in the library 142 25% to use the library as a “late night” study place 233 41% to browse the popular fiction collection 23 4% to browse the dvd collection 78 14% to use the smartboard 33 6%     for purposes of this study, four components (indicated by * in table 1) stand above the rest: the number of people who want to study with a group, use group study rooms, study alone, and use the quiet study area. table 2 illustrates how often participants use the library in person.     table 2 on average, how often do you use the library in person?   options     count   percentage daily 103 17.98% 2 to 4 times a week 206 35.95% once a week 94 16.40% 2 to 3 times a month 74 12.91% once a month or less 76 13.26% never 20 3.49%     students were also asked how often they complete or perform certain tasks during the school year. responses are in table 3.     table 3 library related tasks   library use   never   occasionally   often   very often   used the library as a place to study alone 14% 26% 26% 34% used the library as a place to study in a group   18% 40% 24% 18%     students were asked to evaluate various library areas in terms of their importance from 1 (least important) to 5 (most important). dk, or don’t know, was also an option. students were asked to rank the importance of several areas, and the most relevant are included in table 4.     table 4 evaluate importance of library areas   library areas     1 (least important)   2   3   4     5 (most important)     dk   rank the importance of … group study rooms 4% 2% 6% 21% 60% 7% rank the importance of … quiet study rooms   4% 3% 9% 15% 62% 7%     researchers asked students to share their opinions regarding various aspects of library services on a scale of strongly disagree to strongly agree. in addition, don’t know was also an option. the most relevant responses for this study are in table 5.     table 5 opinion of library services   services     strongly disagree   disagree   agree   strongly agree     don’t know   i can find a quiet place to study in the library 2% 7% 47% 37% 7% i can find areas to work in groups in the library   1% 8% 45% 37% 9%                       students were also asked to rate the library’s renovation and creation of the lansing learning commons and its value as a work space. in two questions, the students broke down roughly into a two-thirds and one-third split. most students clearly see the renovated collaborative space as valuable work space. however, a sizable minority of students, about one-third indicated that the space was too noisy or busy to work in effectively. on a scale from strongly disagree to strongly agree, 223 (39.26%) said agree, and 163 (28.70%) responded strongly agree. also, 145 (25.53%) responded don’t know. in addition, when asked if the lansing learning commons is too noisy or busy for me to work effectively, 114 (20.04%) said agree, and 81 (14.24%) responded strongly agree. also, 142 (24.96%) responded don’t know.   finally, students were asked to provide their general rating of the library. responses are included in table 6.     table 6 in general, how would you rate the library?   options     count   percentage excellent 187 33.45% very good 311 55.64% average 58 10.38 below average 3 .54% poor 0 0%     in addition to the numeric responses, the survey also provided students with the opportunity to provide feedback to open-ended questions. the survey asked, “if we could improve one thing about the library for you, what would it be?” we received 367 written responses. the results were categorized and tabulated to identify areas that could use attention. the top three improvement requests follow:   1.       space:                   34.6% 2.       quiet study:       9.54% 3.       resources:           7.9%   the overwhelming “one thing” students requested was more space in some way—more study rooms, more seating, more tables, or a bigger library. however, the desire for quiet study space is a demand for a significant number of our students.   a comparative analysis of responses in 2012 to 2018 showed an interesting connection. table 7 below outlines survey differences regarding suggested improvements for the library. one chronological note, the responses from 2012 were collected prior to the renovation, and the responses from 2018 were collected after the renovation was completed. also, all student responses are not listed below, only the most relevant for this study.     table 7 suggested improvements   improve one thing     2012   2018 furnishings .45% 6% hours 11% 6% quiet study 8% 10% resources 19% 8% space 26% 35% technology (computers) technology (printers) 24 hours study room (larger size) 7% 3% 2% 2% 2% n/a     post renovation and the significant increase in the number of individual study spaces on the quiet floor, student demand for quiet study space continued to increase. in addition, these survey responses were supported by the headcount data. while overall use of the library increased about 23% from pre-renovation (13-14) to post-renovation (16-17). the largest increase was for the quiet, second floor, which saw a 27% increase. students were not just saying they wanted more quiet space; when provided with more seating, they used it.   implementation   as the university considered how to allocate the recently freed up presidential space in the library, we were able to present data that demonstrated student desire for quiet space beyond the recently renovated learning commons collaborative space. over 60% percent of the students rated the quiet study spaces as very important. about 1/3 of the respondents indicated that the learning commons area was too loud or busy to work effectively. this is a significant number of students who were looking for something other than collaborative work space. in looking at the trend of student responses, we were able to demonstrate that this was in fact an ongoing and increasing desire of the students. finally, we were able to verify these user survey results with observed data in our headcounts. students were using the library’s quiet space more. as a result, the university moved veteran’s and international student recruitment into other available spaces on the campus, but designated the large open spaces to be library quiet study space.    outcome   the use of the new study spaces has been strong. the addition of the boardroom as a more relaxing quiet study space has also been well received. in the fall of 2018, the boardroom study space was 23% of the use of the quiet floor. unexpectedly, when the university was looking for a more centralized location for a meditation, prayer, or reflection space, the library’s quiet floor was selected, and additional space was provided so as not to cut into student study space.   reflection   faced with a potential space vacancy that would attract many programs, it was important to be able to go to the university administration with evidence of the need for quiet student space. the first step in this process was acknowledging that anecdotal staff observation was not going to be sufficient to maintain the quiet space in the library. university priorities had to be considered and empirical evidence was needed to make a compelling case that quiet study space would meet a significant student demand.   the student user survey is conducted regularly, so the next step was to review it for data that would support the library’s contention that more quiet study space was an outcome desired by students. it was fortunate for us that a number of questions addressed the quiet space concern, and we were even more fortunate that the answers all consistently pointed in the same direction—a significant part of the student respondents wanted quiet library space and not just collaborative library space. realizing that the historical trends also supported the library’s point of view was a surprising discovery. finally, having two complementary data sets, student responses on the survey and student actions from the headcounts collected, contributed to making a compelling case.   taken together, the results reveal necessary truths about library functionality and physical space design. libraries today are central features on campus (head, 2016). in some ways the library has become a cultural icon. it must satisfy several dimensions of campus needs. the ability and necessity of the library to become a chameleon campus service are well known (oliveira, 2018). as universities attempt to re-design the library to reach all student stakeholders uniquely, an emphasis on space to achieve individual learning mixed with collaborative design techniques is appropriate (spencer & watstein, 2017).   the results from this study should encourage library staff and university administrators to consider library design from two perspectives. while libraries continue to explore ways to develop open concept physical space design, individual study rooms and, more importantly, quiet study spaces are still a desirable feature (diller, 2015).   we would be remiss to sacrifice all features of the traditional library, including quiet study rooms or study spaces in order to develop more high-tech collaborative open-space design.  students still appreciate the opportunity and ability to study alone and to separate themselves from distraction. library services would do well to remember what goodnight and jeitner reminded us of in 2017: “they [students] come to the library searching for spaces that are quiet, where they can settle down to read and study and write papers in silence, without distractions they find everywhere else in their lives” (p. 100). collaborative space is necessary, but remember to offer students a place for silence in this noisy world.   references   diller, k. r. (2015, march 25-28). reflective practices: library study spaces in support of learning. paper presented at the association of college and research libraries, portland, or.   goodnight, c., & jeitner, e. (2016). sending out an sos: being mindful of students’ need for quiet study spaces. in s. s. hines & k. m. crowe (eds.), the future of library space (pp. 217-234). united kingdom: emerald group publishing limited.   head, a. j. (2016). planning and designing academic library learning spaces: expert perspectives of architects, librarians, and library consultants. project information literacy research report. the practitioner series. retrieved from https://ssrn.com/abstract=2885471   oliveira, s. m. (2018). trends in academic library space: from book boxes to learning commons. open information science, 2(1), 59-74. https://doi.org/10.1515/opis-2018-0005   spencer, m. e., & watstein, s. b. (2017). academic library spaces: advancing student success and helping students thrive. portal: libraries and the academy, 17(2), 389-402. https://doi.org/10.1353/pla.2017.0024       evidence based library and information practice using evidence in practice   catalogue analytics to improve delivery in a special collections library: an evidence based approach to catalogue maintenance   elizabeth hobart special collections cataloging librarian penn state university libraries university park, pennsylvania, united states email: efh7@psu.edu   received: 5 july 2019                                                                      accepted: 10 july 2019      2019 hobart. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29603   setting the eberly family special collections library is located on the university park campus of pennsylvania state university. housing over 200,000 printed volumes, the special collections library serves a range of researchers, including undergraduate and graduate students, professors, and community members.   in the past, the special collections library was three distinct units: the rare books room, historical collections and labor archives, and the penn state room (later called university archives) (penn state university libraries, n.d.). the three units were brought together administratively in the 1970s, and moved into a shared physical space in 1999. although all materials are delivered to patrons through one service point, behind the scenes, materials remain organized in these three historic units.   problem   legacy practices for assigning home locations have led to retrieval problems. the special collections library uses nearly 100 home locations. for example, within rare books, artists' books are shelved together in a "fine printing" home location, while books in the utopia collection are assigned the "utopia" home location. some of these are then further subdivided into sub-locations. "fine printing," for example, is divided by publisher, so that all books published by bird & bull are shelved in one location, books published by compagnie typographique in another, and so forth. in total, 19 publishers had established fine printing sub-locations. to add to this confusion, some sub-locations are actually located in different physical areas. the allison-shelley collection, named for a donor, is shelved partially in the special collections stacks, and partially in a named room on a different floor. both are assigned the "allison-shelley" home location. only sub-location indicates which item is where.   this arrangement allows curators, instruction librarians, and exhibition planners to quickly locate materials, but is not intuitive for the reference staff who retrieve and shelve items. as a result, item retrieval was frequently time consuming, causing patrons to wait while staff looked for their item.   prior to this project, cataloguers recorded sub-location in a public note in the item record, which presented several problems. first, while public notes display in the online catalogue, they do not print on retrieval slips or call number labels. in addition, text in this field is not searchable, making it impossible to generate accurate shelflists for sub-locations. finally, if penn state were to migrate to a new library system, there is no guarantee that these notes would transfer.   we needed to devise a new approach for recording sub-location information. we needed the new approach to allow printing on retrieval slips to make the item's location clear to staff and decrease retrieval time. staff needed to be able to search sub-location to generate accurate shelflists. finally, it needed to be protected in the event of future migration.   while the easiest approach would have been to create separate home locations, due to the large number already established, our systems librarians preferred we find another option. instead, we elected to implement a new, locally-defined marc field, marc field 799, to capture sub-location information. adding the field to the catalogue was a simple matter of defining a new policy in our ils. populating the field with sub-location information, however, was more involved. in total, we identified 5 home locations with sub-location information we needed to record in the 799 field, totaling 63 sub-locations and over 6,500 items. adding this information by hand would have been time-consuming and risked introducing human error.   evidence   to gather sub-location information, we decided to use analytics software. ben showers defines "analytics" as the "discovery and communication of meaningful patterns in data" and "analyzing data to uncover information and knowledge (discovery) and using these insights to make recommendations (communication) for specific actions or interventions" (p. xxx, emphasis in original). analytics reports would allow us to generate lists of all public and internal notes, find patterns, and spot variations.   penn state university uses bluecloud analytics from sirsidynix. we generated a report to retrieve public and internal notes from item records for the five collections with sub-locations. the report output included: title control number, title, author, barcode, call number, home location, internal notes, and public notes.     figure 1 sub-location recorded in a public note field.   figure 2 bibliographic record with marc field 799 inserted.     after running the analytics report for each home location, we exported it to a csv file. a few problems became immediately apparent. first, while we had expected to see sub-location information recorded in public notes, we learned that this information was also recorded in internal notes. the fine printing collection, for example, contained 2,743 items. of these, 379 items had a public note, where 1,018 had an internal note, showing that the internal note was actually used more frequently than the public note. second, while many of these notes recorded sub-location, some recorded other information, such as limitation statements or binding notes. finally, we found numerous variations in name form for some sub-locations. for example, "children's literature" was recorded variously as "c.l.," "child. lit.," "children's lit.," and so forth, totaling over 20 variations.   using openrefine (http://openrefine.org/), an open source tool for cleaning data, we separated this information into different columns, isolating the sub-location information. following this, we used openrefine again to normalize location names. using openrefine, we were able to edit all identical cells, so variants were quickly updated to the full name form for each sub-location.   implementation   after successfully isolating sub-location information and normalizing name forms, we needed to push this information into bibliographic records. using the item information from the analytics report, our digital access team successfully pushed marc 799 fields into the appropriate bibliographic records, successfully updating all 6,500 records across 5 home locations. moving forward, cataloguers will add this information directly to the 799 field rather than using the note fields. in addition, since we had discovered all the variations in names for sub-location, we were able to normalize and document name forms, ensuring that cataloguers will enter the correct form in the future.   outcome   implementing the marc 799 field for sub-location had some immediate impacts. first, we were able to map the marc 799 field to our aeon retrieval system. sub-location information now prints on retrieval slips, which enables faster and more accurate retrieval and re-shelving of these items.   adding sub-locations in the marc 799 also allows us to generate shelflist reports reflecting actual shelving order. now, we can simply search for records with a given sub-location name in the 799 field and sort the results in call number order. staff can perform shelf-reading more easily, which in turn improves collection maintenance and security. in addition, as sub-location data is now in the bibliographic record rather than the item record, it is more visible and protected in the event of future migration. this has become an even more pressing issue as penn state university is preparing to implement a new catalogue discovery layer, in which public notes will no longer be visible.   reflection   our chief obstacle in this process was gathering data from bluecloud analytics. bluecloud relies heavily on pre-packaged reports, and none of the reports available provided the information we needed. we worked closely with our local bluecloud analytics expert team to write and test the report, making changes as needed to ensure we captured all of the note fields, along with item information to update records later.   the rest of the process was relatively straightforward. in addition, since the report has already been written, it's now available for use to other local bluecloud analytics users, and we won't have to repeat creating this report in the future.   however, while the addition of the marc field 799 fulfilled the immediate project goals, the larger problem of having 100 home locations remains. moving forward, we hope to address this, potentially condensing home locations to a smaller number. when (and if) we do this, the 799 fields may be obviated, but it could be several years before we take this step. in the meantime, the sub-location information in the 799 field will play a valuable role in retrieval and collection maintenance. if we do later decide to condense our home locations, the shelflist reports made using the 799 fields will be invaluable for ensuring accurate interfiling of materials.   conclusion   analytics are a powerful tool for finding patterns in catalogue data and targeting records to edit. using analytics allowed us to fulfill our project goals by getting a list of every sub-location recorded in a note, normalizing sub-location name, and pushing the marc 799 into targeted records. we completed this work quickly and accurately, and in a fraction of the time that we would have required to do this work manually. running an analytics report has become standard anytime we need to update catalogue information on a large scale. subsequently, we have used analytics reports for updating call numbers, maintaining genre headings, and updating home locations following collection moves.   references   a short history of penn state special collections. (n.d.). retrieved from https://libraries.psu.edu/about/libraries/special-collections-library/short-history-penn-state-special-collections.   showers, b. (ed.). (2015). library analytics and metrics: using data to drive decisions and services. london: facet publishing.       evidence based library and information practice evidence based library and information practice 2009, 4:3  evidence based library and information practice       evidence summary    university student and faculty opinions on academic integrity are informed by social  practices or personal values    a review of:  randall, ken, denise g. bender and diane m. montgomery. “determining the opinions of health  sciences students and faculty regarding academic integrity.”  international journal for  educational integrity 3.2 (2007): 27‐40.    reviewed by:  matthew thomas  health sciences librarian, bracken health sciences library, queenʹs university  kingston, ontario, canada  e‐mail:  matthew.james.thomas@gmail.com     received: 19 april 2009         accepted: 21 july 2009      © 2009 thomas. this is an open access article distributed under the terms of the creative commons attribution  license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and  reproduction in any medium, provided the original work is properly cited.    abstract     objective – to understand the opinions of  students and faculty in physical therapy (pt)  and occupational therapy (ot) regarding  issues of academic integrity such as plagiarism  and cheating.    design – q method (a mixed method of  qualitative data collection with application of  quantitative methods to facilitate grouping  and interpretation).    setting – an urban university‐affiliated health  sciences facility in the mid‐western united  states.    subjects – thirty‐three students and five  faculty members of ages 21 to 61 years, 30  associated with the physical therapy program  and 8 with occupational therapy, including 6  males and 32 females.    methods – initially, 300 opinion statements  for, against, or neutral on the subject of  academic integrity were gathered from journal  articles, editorials and commentaries, internet  sites, and personal web logs, 36 of which were  selected to represent a full spectrum of  perspectives on the topic. participants in the  study performed a “q‐sort” in which they  ranked the 36 statements as more‐like or less‐ like their own values. a correlation matrix was  developed based on the participantsʹ rankings  to create “factors” or groups of individuals  with similar views. two such groups were  found and interpreted qualitatively to  meaningfully describe the differing views of  each group. three participants could not be  sorted into either group, being split between  the factors.  49 mailto:matthew.james.thomas@gmail.com http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2009, 4:3    main results – analysis of the two groups,  using software specific to the q method,  revealed a good deal of consensus,  particularly in being “most unlike” those  statements in support of academic dishonesty.  the two groups differed primarily in the  motivation for academic honesty. factor one,  with 21 individuals, was labeled “collective  integrity,” (ci) being represented by socially  oriented statements such as “i believe in being  honest, true, virtuous, and in doing good to all  people,” or “my goal is to help create a world  where all people are treated with fairness,  decency, and respect.” factor two, with 14  individuals, was described as “personal  integrity,” (pi), and focused on an internal  sense of values and self‐modulation,  identifying with statements like “honour  means having the courage to make difficult  choices and accepting responsibility for  actions and their consequences, even at  personal cost.” there were also some  demographic patterns in the results. twenty of  the 31 students, 20 of the 29 females, and 17 of  the 25 participants aged 30 and under were in  the ci group, while 3 of the 4 faculty were in  pi. males, occupational therapists, physical  therapists, and those over the age of 30 did not  belong clearly to one or the other group,  having close to equal numbers in both.    conclusion – given the two factors, ci and pi,  this sample of ot and pt students and faculty  can be seen to make academic decisions based  on either what they believe society deems  correct or what their own internal values tell  them. the discovery that more females,  students, and those 30 and under were  associated with ci resonates with the some  key claims in the literature, such as that  younger individuals tend to have a more  social outlook on academic integrity, or that  womenʹs ethic of care is often focused on  connections among people. most importantly,  students and faculty appear to share a notable  degree of common ground as it relates to their  opinions on academic integrity. additional  exploration and the continued use and  development of policies promoting academic  integrity is called for.    commentary    given that this study is looking at the opinions  of ot and pt students and faculty concerning  academic integrity issues, the conclusions and  discussion would be of interest to those  shaping policies in this area as well as those  concerned with educating faculty and students  on the topic. although further study is  needed, the ideas expressed here could be  used to guide and phrase discussion and  policy, suggesting the use of two versions at  the two factors/groups.    there were several aspects of this mixed  methods study that call into question some of  the conclusions that the authors came to.  firstly, the participants sample was composed  primarily of younger female students. that  there were not very many faculty members (5  out of 38, with only four being included in the  final two factors) makes it difficult to be sure  that other important viewpoints were not  missed, let alone claim that they effectively fall  into one or another factor. also, in the “30 and  under” category, all were either 21 or 26, the  specific characteristics of which may have  resulted in a substantial number of missed but  important opinion statements as well as a  skewed demographic conclusion about this  group.    the nature of the ranking of opinion  statements that participants were ask to do,  namely the fact that there were a fixed number  of “slots” for each likeness ranking, could  possibly lead to inaccurate assumptions about  the participants responses, despite the efforts  to provide a complete spectrum of opinions.  for example, there may be individuals who  would otherwise place seemingly  incompatible opinion statements at one end of  the spectrum, revealing a supposed irrational  view of the subject, which could be a valuable  insight into their academic behaviour. also,  the assumption that those statements placed in  the middle of the “less like – more like”  ranking are neutral does not follow for the  same reason.    50 evidence based library and information practice 2009, 4:3  51 this research does however present several  valuable insights into the topic of integrity  such as the binary split in perspectives, the  demographic layout of the two factors, and  possible reasons for the patterns. as  mentioned in the work, q methodology is not  able to reveal more substantial differences  people have on this topic, and that more  investigation into our understanding of the  behaviour of those in academic environments  is warranted, perhaps through comparable  additional q studies, comprehensive surveys  and/or psychological analysis and  substantiation.  microsoft word es_stephens.doc evidence based library and information practice 2007, 2:4  64 evidence based library and information practice       evidence summary    school libraries:  are they places to learn or places to socialize?      a review of:  tallaksen rafste, elisabeth. “a place to learn or a place for leisure?  students’ use of the school  library in norway.” school libraries worldwide 11.1 (2005): 1‐16.      reviewed by:   julie stephens  retired media specialist/educator  calhoun, georgia, united states  e‐mail: jecs57@bellsouth.net       received : 31 august 2007      accepted : 04 november 2007      © 2007 stephens. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objectives – to explore how students use  the school library in their daily activities,  who visits the school library, what activities  occur during these visits, and how students  value the school library.    design – comparative, multi‐case study.    setting – two norwegian senior high  schools in two different counties.    subjects – students in year one, two, and  three at two high schools; and teachers,  principals, and school librarians at each of  the two schools.    methods – data was collected from  interviews, observations, documents, and  questionnaires during the first five months  of 1998.  most data was gathered from 25  observations in the school library (each  observation was 3‐4 hours in length).   observations were made in three specific  areas of each library: work tables, the  computer site, and a reading hall quiet area.   in addition, seventeen 45‐minute  observations were made in various  classrooms.  to gain student perspectives  and to learn how and why students valued  the school library, in‐depth interviews were  conducted with 28 students, consisting of 2  boys and 2 girls from each of years 1, 2, and  3 at each school, plus 2 boys and 2 girls from  the international baccalaureate classes at  one school.  four teachers from each school,  the school librarians, and the principals  from each school were also interviewed to  explore attitudes about the school library,  mailto:jecs57@bellsouth.net http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:4  65 how they valued it and what instructional  role they believed the library played in  students’ daily lives.  sixty students  completed questionnaires that asked when  and for what reason students used the  library, what locations in the library they  used, and what the library meant to them in  both their schoolwork and free time.   documents such as class schedules and  curricula, and school policies and rules were  also considered.    main results – data analysis indicated  students had a lot of appreciation for the  school library, but mainly for its role as a  “social meeting place,” rather than as  resource center for information.  students  were aware of the function, purpose, and  importance of the school library, but rarely  used it for projects or research.  the library  was most appreciated for the fact that users  went there to meet friends and talk.  one  observed group did not borrow books or  bring work to do, clearly demonstrating that  their purpose in the library was strictly  social.  there were students who used the  library for research and information  retrieval, but these students were the  minority.  most of the students who did  instruction‐related activities in the library  did homework from textbooks they brought  to the library.  there was no indication that  teachers or the school librarians made any  efforts to alter the attitudes of students or  their use of the library as a social club.   based on observations, the researcher  offered several possible reasons for her  findings: weak rules and few sanctions,  invisibility of the school librarians, failure of  teachers to use the library or make  assignments that required information  seeking, and lack of a cafeteria in school a  (which may have also contributed to the  value of the library as a “meeting place”).   leisure‐related activities in the quiet  reading hall were highest among the girls,  and highest among the boys at the work  tables and computer sites.  daily users  (occupants) of the library at school a were  second and third year boys and girls. only  boys from first, second and third year  vocational classes were “occupants” at  school b.  the occupants at both schools  influenced the activities of new users.    conclusion – the findings of this study  reveal a “gap between the rhetoric on  instruction and school library use and actual  practice” (12).  students were rarely given  assignments that required use of the library  and there was no collaboration between the  classroom teachers and the school librarian.   the library was not perceived as a resource  center and was not viewed as an integral  part of daily instruction.  weak rules, few  sanctions, misperceptions, and inadequate  instructional leadership by the school  librarian appeared to contribute to the  observed behaviors related to library use in  the two schools.  the author suggests the  need for organization, leadership, and the  proper training of students on the use of the  library.  she mentions the need for  principals, teachers, librarians, students, and  teacher preparatory colleges to work hand‐ in‐hand to bring about a change of attitude  about – and usage of – the school library.    commentary    the topic of this research is of significant  value to teachers, principals, and school  librarians.  if retraining or special in‐service  training is needed to better develop an  understanding of the efforts needed to  ensure collaboration between teachers and  librarians, it should be pursued. a chart was  included that summarized observed  activities. an extensive review of related  literature was examined throughout the  paper.  this research was very limited in the  fact that only two schools were subjects of  the study.  the data that was collected was  almost ten years old and may or may not  reflect practices at these two schools today.   future research might include more schools  evidence based library and information practice 2007, 2:4  66 to determine if these findings are typical or  unique at these two particular schools.  it  might also be interesting to determine if  similar attitudes and behaviors are  prevalent among elementary‐age students  as well.  microsoft word es_lewis.doc evidence based library and information practice 2006, 1:3      57 evidence based library and information practice       evidence summary    open access articles have a greater research impact than articles not freely  available   a review of:  antelman, kristin. “do open‐access articles have a greater research impact?” college &  research libraries 65.5 (sep. 2004): 372‐82.    reviewed by:  suzanne p. lewis   library manager, central coast sector, northern sydney central coast health   gosford, new south wales, australia  e‐mail: slewis@nsccahs.health.nsw.gov.au    received: 30 may 2006              accepted: 23 june 2006      © 2006 lewis. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to ascertain whether open  access articles have a greater research  impact than articles not freely available, as  measured by citations in the isi web of  science database.    design – analysis of mean citation rates of a  sample population of journal articles across  four disciplines.    setting – journal literature across the  disciplines of philosophy, political science,  mathematics, and electrical and electronic  engineering.    subjects – a sample of 2,017 articles across  the four disciplines published between 2001  and 2002 (for political science, mathematics,  and electrical and electronic engineering)  and between 1999 and 2000 (for philosophy).     methods – a systematic presample of  articles for each of the disciplines was taken  to calculate the necessary sample sizes.  based on this calculation, articles were  sourced from ten leading journals in each  discipline. the leading journals in political  science, mathematics, and electrical and  electronic engineering were defined by isi’s  journal citation reports for 2002. the ten  leading philosophy journals were selected  using a combination of other methods.    once the sample population had been  identified, each article title and the number  of citations to each article (in the isi web of  science database) were recorded. then the  article title was searched in google and if  any freely available full text version was  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:3      58 found, the article was classified as open  access. the mean citation rate for open  access and non‐open access articles in each  discipline was identified, and the percentage  difference between the means was  calculated.    main results – the four disciplines  represented a range of open access uptake:  17% of articles in philosophy were open  access, 29% in political science, 37% in  electrical and electronic engineering, and  69% in mathematics. there was a significant  difference in the mean citation rates for open  access articles and non‐open access articles  in all four disciplines. the percentage  difference in means was 45% in philosophy,  51% in electrical and electronic engineering,  86% in political science, and 91% in  mathematics. mathematics had the highest  rate of open access availability of articles,  but political science had the greatest  difference in mean citation rates, suggesting  there are other, discipline‐specific factors  apart from rate of open access uptake  affecting research impact.    conclusion – the finding that, across these  four disciplines, open access articles have a  greater research impact than non‐open  access articles, is only one aspect of the  complex changes that are presently taking  place in scholarly publishing and  communication. however, it is useful  information for librarians formulating  strategies for building institutional  repositories, or exploring open access  publishing with patrons or publishers.    commentary    journal impact factors have long been of  interest to librarians building or maintaining  core collections and to academics wishing to  be published in high‐impact journals. the  latter group are also interested in how many  times their published research is cited by  other authors. this article, which examines  the research impact of individual articles,  will be of interest to both groups as they face  the challenges and opportunities of open  access publishing.     in this well‐constructed, highly readable  study, antelman demonstrates that open  access articles have a greater research  impact than non‐open access articles in the  disciplines of philosophy, mathematics,  political science, and electrical and  electronic engineering. the methodology  used in this research is sound: select a  number of high‐impact journals, identify a  sample of articles published in those  journals, and compare the number of  citations to open access versus non‐open  access articles. enough detail is given in the  methodology section to suggest that the  sample size is sufficient. articles published  in 2001 and 2002 were selected on the basis  that a two‐year lag between publication and  citation analysis is the commonly accepted  time required for confirmation of impact  (garfield 92). the time frame was adjusted  for philosophy where there is a lower level  of citation of articles (reference period 1999‐ 2000). the methodology used also takes into  account the skewed distribution of citations  in most fields, where 20% of articles may  account for 80% of citations (garfield 91), by  using nonparametric methods.    this study raises two main points of  discussion: one in relation to methodology,  which is addressed by the author; and one  in relation to the conclusions drawn from  the study, which the author has addressed  since this article was published.    the study’s methodology is based on using  ‘citedness’ as a measure of research impact.  citation has long been the standard  measurement of impact in the scholarly  community and the means of mapping  networks of scientific research. however, in  the complex environment of open access  publishing, citation on its own is an  evidence based library and information practice 2006, 1:3      59 inadequate measure of research impact. the  author acknowledges this when she states  that “citedness as measured by isi [web of  science] is a measure that is commonly  relied on as a surrogate for [research]  impact.” she admits that this study is  concerned with “the effect of open access on  one traditional and frequently used measure  of research impact” (emphasis added).  other researchers in the field agree. steven  harnad and tim brody, for example, have  used the same methodology in their  ongoing collaborative study of 14 million  open access and non‐open access articles  across ten disciplines over ten years.  however, others are developing new  bibliometric methodologies appropriate to  web environments, giving rise to new terms  such as webometrics, cybermetrics, and  influmetrics (cronin).     the more problematic issue raised by this  study lies in the conclusions drawn by the  author. this study demonstrates that open  access articles have greater research impact  than non‐open access articles, but it does not  demonstrate that open access causes greater  research impact. philip m. davis has  challenged the author on this point by  proposing two alternative explanations for  the greater research impact of open access  articles: article duplication (open access  itself being a form of duplication), and  author self‐promotion (such as posting  articles published in prestigious journals to  an author’s own website) (103‐4).    antelman has replied that “[w]hile i  intentionally phrased my conclusion as an  association, rather than a causation [. . .]  there clearly is an implied causation and i  should have been more explicit that the data  do not support that” (105). she also  responds that, since conducting the original  research reported in this article, she has  collected additional data which suggests  that author self‐promotion, or “quality bias”  is a significant issue in scholarly publishing.  this study—which, of course, is freely  available online (<http://www.la‐ press.com/include/antelman.pdf>)—will be  of great interest to academic and special  librarians negotiating the dynamic  environment of open access publishing and  document retrieval. it uses traditional  citation analysis methodology but points to  evolving bibliometrics and new citation  measures. the author achieves the stated  objective of demonstrating that freely  available papers have a greater research  impact than those not freely available online.  however, it is identifying and measuring  the multiple, interdependent factors that  cause the greater impact that poses  challenges for current and future  researchers in this area.    works cited    antelman, kristin. ʺletter to the editor.ʺ  college & research libraries 67.2  (2006): 105.    cronin, blaise. ʺbibliometrics and beyond:  some thoughts on web‐based citation  analysis.ʺ journal of information  science 27.1 (2001): 1‐7.    davis, philip m. ʺ letter to the editor.ʺ  college & research libraries 67.2  (2006): 103‐4.    garfield, eugene. ʺthe history and meaning  of the journal impact factor.ʺ jama  295.1 (2006): 90‐93.    harnad, steven, and tim brody.  ʺcomparing the impact of open access  (oa) vs. non‐oa articles in the same  journals.ʺ d‐lib magazine 10.6 (2004):  1.    http://www.la%e2%80%90press%00 evidence based library and information practice evidence based library and information practice 2009, 4.4     evidence based library and information practice         news/announcements    cla/acb library research and development grants      © 2009. this is an open access article distributed under the terms of the creative commons attribution   license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and   reproduction in any medium, provided the original work is properly cited.      the canadian library association/association  canadienne des bibliothèques (cla/acb) is  offering library research and development  grants totalling $1,000 to members. these  grants:    * support theoretical and applied research in  the field of library and information services  * encourage and support research undertaken  by practitioners in the field of library and  information services  * promote research in the field of library and  information services by and/or about  canadians    the deadline for submission of proposals is  february 28th, 2009.  proposals are limited to  four pages in length.    for more information please visit  http://www.cla.ca/awards/grants.htm    102 http://www.cla.ca/awards/grants.htm microsoft word es_blythe.doc evidence based library and information practice 2007, 2:4  70 evidence based library and information practice       evidence summary    the academic reward system is the primary influence toward faculty non‐ participation in institutional repositories      a review of:  davis, phillip m., and matthew j.l. connolly. “institutional repositories: evaluating the reasons  for non‐use of cornell university’s installation of dspace.”  d‐lib magazine 13.3/4  (2007). 16 oct. 2007 <http://www.dlib.org/dlib/march07/davis/03davis.html>.      reviewed by:   kurt blythe  serials access librarian, university of north carolina‐chapel hill  chapel hill, north carolina, united states of america  e‐mail: kcblythe@email.unc.edu        received : 31 may 2007      accepted : 22 september 2007      © 2007 blythe. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to better understand the lack of  faculty participation in cornell university’s  dspace institutional repository (ir), and to  learn if this lack of participation is peculiar  to cornell or reflective of a larger trend in  faculty non‐participation in irs.      design – comparative analysis and  interviews.     setting – cornell university’s dspace ir  and sciences, social sciences, and humanities  faculties; and dspace installations at 7 other  universities.    subjects – the dspace ir at cornell  university and at 7 other locations. eleven  sciences, social sciences, and humanities  faculty members at cornell university.      methods – the authors analyzed data over a  fifteen‐month period from cornell’s dspace  ir to determine the total deposits, the types  of objects deposited, the communities and  collections that received deposits, the  frequency of deposits, the ip addresses  which made deposits, and how often objects  in the ir were viewed.  these data were  compared to equivalent data taken from  seven other irs on all aspects except  deposits from ip addresses and how often  objects were viewed.  finally, 11 cornell  faculty members from various departments  http://www.dlib.org/dlib/march07/davis/03davis.html mailto:kcblythe@email.unc.edu http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:4  71 in the sciences, social sciences, and  humanities were interviewed over a two‐ month period to provide context to the  comparative analysis.    main results – at the time of the study, the  ir at cornell was organized into 193  communities of collections.  these  collections numbered 196, with 139 of them  holding a combined total of 2646 objects:   the other 57 collections were empty.  while  the ir as a whole showed steady growth,  77% of cornell’s collections reflected a  plateau growth pattern of primarily “one‐ time deposits,” approximately 18%  exhibited a stair‐step growth pattern of  “periodic batch additions of material,”  approximately 3% showed steady growth,  and 1.4% were “uncatagorizable.”  five‐ hundred nineteen unique ip addresses made  deposits to cornell’s ir over the course of  the fifteen‐month study, but 50% of these  deposited only one object, and only 32 ip  addresses deposited 10 or more objects.      of the other irs studied, the lowest number  of communities is zero and the highest is  390, the number of collections ranged from  10 to 282, and the number of objects ranged  from 500 to 32,676.  in most statistical  categories, cornell fell in the midrange.  the  two repositories with the fewest  communities and collections – zero  communities and 18 collections in one  instance, and 6 and 10 in the other – are the  only two with no empty collections.  the  repository with the most communities and  collections also had the most empty  collections (58%).  the repository with the  most objects was the one with zero  communities and only 18 collections; and  the repository with the fewest objects was  the one with only 6 communities and 10  collections.  the third largest ir, with 3111  objects, had far and away the highest rate of  steady growth (16.7%); while the ir with the  most objects had the highest rate of stair‐ step growth (56.3%), and was the only ir to  have a higher percentage of growth in any  category other than plateau.      interviews with faculty indicated that they  do not make deposits to irs for a number of  reasons.  faculty considered their primary  audience to be their peers, so access to their  scholarship was largely considered a “non‐ issue” as it was adequately provided  through personal web pages, subject  repositories, or journal literature.  likewise,  long‐term preservation was not an  overarching area of concern.  the chief  factors for not using an ir, however, all  revolved around restrictions brought on by  the academic reward system.  questions of  copyright and whether depositing objects  qualifies as publishing, thereby hindering  efforts to publish in journals, were  paramount, as were fears that depositing  scholarship alongside less rigorous works in  a catch‐all ir would diminish the work and  the reputation of the scholar by association.   hesitancy to make work available before it  had been certified and peer‐reviewed was  also a foremost concern.                conclusion – although objects in cornell’s  dspace are accessed both locally for items  that are tied into the curriculum, and  outside of the university for items that are of  national (and international) interest, the  repository was not supported well by the  faculty.  the majority of the collections  defined in cornell’s ir were under  populated, and what growth was evident  arose primarily from deposits made by non‐ faculty.  the reasons for this were manifold,  but centered primarily on the established  culture of the academic reward system,  which encourages publishing in recognized  journals and does little to foster thoughts for  long‐term preservation or dissemination  outside of a given scholar’s peer group.   these issues were evident in faculty  concerns that depositing materials in an ir  might prevent later publication in a journal;  the idea that depositing scholarship in a  evidence based library and information practice 2007, 2:4  72 non‐vetted repository would diminish that  work by association with less scholarly  materials; the feeling in some fields that it  would be irresponsible to provide access to  any unfinished, non‐vetted work; the  thought that irs are not sufficient to the task  of certifying scholarship; and the concern  that deposit in an ir might lead to  plagiarism or the loss of initiative on  unpublished ideas.     commentary     the authors effectively explicate the  obstacles faced by irs in terms of gaining  faculty participation.  the interviews are  enlightening, and contribute greatly to  placing the data taken from the studies of  cornell’s and the seven other institutions’  irs in context.  the study makes it clear that  the academic reward system does not  encourage faculty participation in cornell’s  ir.  moreover, the evidence provided by the  study pointing to more highly structured  examples of dspace, which receive fewer  deposits, is interesting and pertinent to ir  administrators considering how to shape  their own installations.  at the same time,  however, the impact of analyses into the  numbers of communities and collections in  irs, the percentages of empty collections,  and the total numbers of objects in  repositories is lessened somewhat by the  readers inability to determine what  distinctions exist between the various  structures presented, whether the defined  spaces are sufficient or insufficient for their  purposes, and whether contributions come  generally from the same types of sources  regardless of institution.  a larger sample  size of dspace installations might mitigate  the discrepancies in internal construction.   data on the number of faculty at each  institution studied would do well alongside  data regarding the number of contributions  made, so that potential could be assessed  against actual deposits.  however, the  authors have certainly laid out sufficient  information to begin answering the question  that motivated their research, that of  determining why cornell’s installation of  dspace exhibits less steady growth and  holds fewer deposits than is desired.      it is clear, at least at this time, that irs have  little hope of usurping the entrenched role  of journals as agents of certification in the  realm of scholarly information creation and  dissemination.  ir administrators would do  well to attend to the lessons taught by the  article and adapt plans for their repositories  respectively.  assurances from publishers  that depositing scholarship in a repository  will not violate copyright or preclude  publication must go hand‐in‐hand with  demonstrations of how an ir can certify  work by establishing firm dates of deposit.   the ability of irs to guarantee long‐term  preservation, wide access through  searchability in common search engines like  google, and ease of use in terms of  information upload must all be  demonstrated, and care must be taken to  neither overwhelm faculty with too intricate  a structure of communities and collections,  nor to disregard the specificity of their  scholarship with too few.  in other words,  only a close association with faculty, one  which fosters understanding on both sides,  will ensure the success of irs.  call for applicants for eblip journal: communications officer news   call for applicants for eblip journal: communications officer    2022. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30223     evidence based library and information practice (eblip), a peer reviewed open access journal published since 2006, seeks to fill the position of communications officer. the successful candidate will be expected to begin their role by november 2022 and serve a three-year term (with possibility of renewal).   the communications officer will be responsible for:   developing and implementing a communications plan for the journal. developing, designing, writing, and editing material to promote the journal in collaboration with the editor-in-chief. this can include email messages to mail lists, postings on social media, and announcements in the eblip journal. communicating and consulting with the editor-in-chief on a regular basis. attending editorial board meetings via zoom on a quarterly basis.   the ideal candidate for communications officer will have skills and knowledge in areas of communications, preferably in the scholarly/academic realm. excellent written communication skills in english are required. experience with social media, developing and executing communication plans, and an interest in evidence based librarianship and open access are considered assets. this position requires dedicated time on a regular basis, and it is therefore essential that interested persons ensure available time to devote to this position prior to applying. it is estimated that the workload is approximately 5-10 hours per month.   interested persons should send a cover letter, indicating areas of strength they would bring to the role, and resume/cv as a single pdf file to ann medaille, editor-in-chief, at amedaille@unr.edu by october 10, 2022. please email ann with any questions.   **please note that evidence based library and information practice is a non-profit, open access journal, and all positions are voluntary and unpaid. about the journal:   published quarterly and hosted by the university of alberta, this peer-reviewed, open access journal is targeted at all library and information professionals interested in an evidence based model of practice. by facilitating access to librarianship research via original research articles and evidence summaries of relevant research from the library literature, evidence based library and information practice enables librarians to practice their profession in an evidence based manner. please visit the evidence based library and information practice web site (https://journals.library.ualberta.ca/eblip/index.php/eblip) for further information about the journal.   structured interviews reveal that reference and liaison librarians—as engaged, proactive partners—are vital to the academic enterprise evidence summary   structured interviews reveal that reference and liaison librarians—as engaged, proactive partners—are vital to the academic enterprise   a review of: johnson, a.m. (2020). reference and liaison librarians: endangered species or “vital partners?” views of academic library administrators. journal of library administration, 60(7), 784-799. https://doi.org/10.1080/01930826.2020.1786979   reviewed by: joanne m. muellenbach library director and associate professor california health sciences university clovis, california, united states of america email: jmuellenbach@chsu.edu   received: 11 nov. 2021                                                                 accepted:  4 jan. 2022      2022 muellenbach. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30080     abstract   objectives – to investigate the current state and prospects of reference and liaison librarianship.   design – structured interviews consisted of 10 questions that lasted between 30 and 75 minutes.   setting – fourteen medium-sized, urban universities geographically spread across the united states of america.   subjects – fifteen library administrators with at least 10 years of experience.    methods – the author contacted academic library leaders from 17 benchmark institutions and head librarians from other r1 institutions whose libraries were members of the association of research libraries (arl) or whose campus size and characteristics mirrored the author’s institution in that they were medium-sized urban universities. the study examined five primary questions and included an appendix with the 16-item survey instrument. the structured interviews included 10 questions about the current state and prospects of reference and liaison librarianship, along with questions related to demographics. the author transcribed the interviews and removed all identifying information. since the interviews were structured and thus thematically similar, coding software was not used. the author compiled and analyzed the responses to the questions.    main results – the concepts of connecting, discovering, listening, and partnering were inherent in the definition of being a liaison librarian. in general, the library administrators, all of whom had been in the profession for 10 years or more, felt that liaison librarians should be active in furthering scholarly activities in such areas as grant-writing, generating scholarship, or data curation. there was an emphasis on outreach, being proactive, and engaging with faculty, which raised an important question for administrators: is this skill set too broad for any one person, and if so, how can the library profession collaborate to draw upon each other’s strengths? there was a consensus that while the work of reference and liaison librarians is vital to the academic enterprise, this work need not be situated at a central reference desk. rather, librarians would be physically embedded or electronically linked to students and faculty, helping them to formulate answerable questions, locate high-quality, evidence-based information in specialized databases, or provide support in such areas as open educational resource development, augmented reality, or scholarly communications.      conclusion – in the view of current library administrators, being a reference and liaison librarian means partnering proactively with students and faculty to ensure a deep understanding of their teaching, learning, and research needs while also maintaining a thorough knowledge of the libraries’ collections and resources. to accomplish this, the librarian must be visible to their constituencies, tell memorable, authentic stories of what they have to offer, and build lasting relationships. reference and liaison librarians require traditional knowledge of library functions and systems and teaching skills and possess qualities such as collaboration, communication, and flexibility. overall, library leaders believe that liaison librarians will continue to be vital partners and that without a central reference desk, there will be a deeper integration within the academic enterprise.   commentary   even as far back as 1978, academic reference librarians were challenged to demonstrate their value and engage in high-level scholarly and teaching activities (aluri & st. clair, 1978). to this end, library leaders are involved in reenvisioning the liaison program structure and balancing functional versus subject expertise. an arl report regarding liaison roles revealed an emerging focus on engagement, requiring an outward focus as libraries monitor and respond to changes in stakeholder demands, needs, and practices (jaguszewski & william, 2013). since 2015, the arl has offered eight oneto two-day liaison institutes to their members, and the overarching recommendations for liaisons are to foster deeper and frequent communications, seek workplace collaborations, and reach out to non-traditional campus units, which are often drivers of institutional initiatives and priorities (vine, 2018).    applying the generic cat appraisal tool (perryman & rathbun-grubb, 2014), i found the study to be clear and well organized, with sample responses to the interview questions and an appendix with the interview questions. the study's author, johnson, has in-depth knowledge about liaison librarian services. she conducted a thorough literature review that traced the historical perceptions of reference librarians, published a peer-reviewed study on this topic, and served as a library administrator at an r1 institution whose library is an arl member. the author acknowledged a limitation related to the types of libraries included in the study: library leaders were only from benchmark institutions or from medium-sized, urban universities. future research may wish to expand to include library leaders from diverse ethnic groups and larger, international academic institutions, focusing on specialized subject disciplines.   johnson provided the participants’ anecdotal responses to the questions examined. still, the study would have been enhanced if it provided an analysis for the question, “what reference model do you use?” such responses would have revealed a useful benchmark about the participants’ reference philosophy. using participant codes or pseudonyms would have added context to their comments and allowed us to link sets of responses throughout the analysis. details regarding the interview period, mode, and the process for collecting, analyzing, and storing the data would have decreased any bias concerns. there is also a lack of clarity regarding liaison, reference, and subject librarians and if they are the same or maintain unique aspects. while the study participants were individuals with at least 10 years’ library experience, it is unclear how long they served in leadership positions. further information regarding the methodology could help future researchers build upon this subject area.    the findings of this study can assist academic library administrators in strategic planning and inform programs in library and information studies that include courses on emerging library roles or a library administration component. in addition, librarians in these roles may wish to solicit user feedback by conducting focus groups or distributing surveys to inform their future services. this study is an important starting point from which further research will provide more significant insights into the role of liaison librarians as vital partners in the academic enterprise.    references   aluri, r., & st. clair, j. w. (1978). academic reference librarians: an endangered species? journal of academic librarianship, 4(2), 82–84.   jaguszewski, j. m., & williams, k. (2013). new roles for new times: transforming liaison roles in research libraries. association of research libraries. https://www.arl.org/wp-content/uploads/2015/12/nrnt-liaison-roles-revised.pdf     perryman, c., & rathbun-grubb, s. (2014). the cat: a generic critical appraisal tool. https://www.jotform.us/cp1757/thecat   vine, r. (2018). realigning liaison with university priorities: observations from arl liaison institutes 2015–18. college & research library news, 79(8), 420–423, 458. https://doi.org/10.5860/crln.79.8.420   evidence based library and information practice evidence based library and information practice 2010, 5.1  evidence based library and information practice       ebl101    an introduction to critical appraisal    virginia wilson   shirp coordinator, health sciences library   university of saskatchewan   saskatoon, saskatchewan, canada   email: virginia.wilson@usask.ca    received: 30 jan. 2010          accepted: 05 feb. 2010       2010 wilson. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which  permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly  attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the  same or similar license to this one.      last time, this column focused on librarians  conducting research as part of step 2 – find  the evidence – in evidence based library and  information practice, and what that might  mean for the individual and the profession.  it’s time to move to step 3 – appraise the  evidence. when you’ve looked to the  literature and have found some research  articles that just might inform your practice,  the next step is to look critically at these  articles to assess whether or not the research  could be used to improve practice or help  inform decision making. critical appraisal is  the process of assessing and interpreting  evidence by systematically considering its  validity, reliability, and applicability for  quantitative articles, and its credibility,  transferability, dependability, and  confirmability for qualitative articles. critical  appraisal is a developable skill that uses  explicit methods.    this process may sound daunting, but i would  argue that we do this in our everyday lives to  some extent. reading newspapers, watching  television, scanning our rss feeds…we  process what we read, see, and learn through  our own experiences, and we don’t take  everything at face value. critical thinking is a  necessity in a world where “information  overload” is not just a catch phrase. there has  to be some way to filter what is out there so  that parts of it can be useful to the individual  and not just so much white noise. this  process, with added rigor, is the next step in  the practice of ebl.    there are a few criteria that we either  implicitly or explicitly use to determine what  is worth reading. there’s the interest factor:  are we interested in the topic of the article as  outlined by the abstract or the title? then there  are extrinsic factors: have we heard of the  article? is it published in a peer reviewed  journal? is the author from a reputable  institution? then there are the intrinsic factors,  and these are what the critical appraisal  process can tease out. these factors “relate to  research design and aspects of methodology”  (booth & brice, 2004, p. 105).  155 mailto:virginia.wilson@usask.ca evidence based library and information practice 2010, 5.1  so to that end, let’s look at some term  definitions:    • validity: “the extent to which the results  of the research are likely to be free from  bias” (booth & brice, 2004, p. 105).     • reliability: “the likelihood that this study  could be reproducible, rather than the  results just being a ‘fluke’ or chance  result” (booth & brice, 2004, p. 105)  • applicability: “the extent to which the  results are likely to impact on practice” or  be able to effectively inform decision  making (booth & brice, 2004, p. 105).     a good round of critical appraisal can help to  determine the presence (or not) of these terms,  and thus the usability of the research article.  qualitative studies are no less relevant to  libraries. trisha greenhalgh and rod taylor  have an excellent article entitled “how to read  a paper: papers that go beyond numbers  (qualitative research).” in it, they outline the  questions to ask when reading a paper based  on qualitative analysis.     so luckily, the tools are out there to aid in the  critical appraisal process. in addition to the  greenhalgh & taylor article, there have been  several checklists developed to guide the  reader through research articles in a  systematic way.     • lindsay glynn created a useful checklist  especially for library and information  science. this checklist can be downloaded  here:  http://ebltoolkit.pbworks.com/f/eblcritic alappraisalchecklist.pdf .   • the cristal checklists for various types  of studies (appraising a user study:  http://www.shef.ac.uk/scharr/eblib/use.ht m and appraising an information needs  analysis:  http://www.shef.ac.uk/scharr/eblib/use.ht m).  • reliant is the readers’ guide to the  literature on interventions addressing  the need for education and training:  http://eprints.rclis.org/7163/1/reliant__f inal_.pdf   • tools were developed through the critical  appraisal skills program (casp) in the  uk:  http://www.phru.nhs.uk/pages/phd/reso urces.htm    it might seem like there are just too many  checklists and tools to wade through to get  started. however, i suggest that you choose  one checklist to start with. personally, i  recommend the one created by glynn to get  you going. then sit down with the list and the  article and work through them both just to get  a feel for the process. you might not be able to  answer some of the questions on the checklist,  which is alright. just look at the article as  objectively as you can, and a useful  assessment of the article will start to emerge.    as i mentioned before, critical appraisal is a  skill, and it can be honed through practice.  there are a couple of things that you can do to  increase your skill level. journal clubs are a  good way to practice critical appraisal in a  collegial setting.  a journal club is a group of  individuals who meet regularly to critically  evaluate articles in particular practice settings.  a journal club was recently started in the  health sciences library at the university of  saskatchewan. health sciences librarians  gather to discuss a previously chosen research  article in a critical fashion. this process  increases our understanding of research  methods, keeps us abreast of research in our  field, and provides a comfortable atmosphere  in which to expand our critical appraisal skills.    as well, the evidence summaries found in  each issue of evidence based library and  information practice are all examples of the  critical appraisal process. koufogiannakis  (2006) states that “reading critical appraisals  informs us all of the questions we should be  asking when we approach a research article  and allows us to become more familiar with a  critical approach to reading the literature of  our field” (p. 81).     undertaken as a step in the process of  evidence based library and information  practice, or as a stand alone endeavor, critical  156 http://ebltoolkit.pbworks.com/f/eblcriticalappraisalchecklist.pdf http://ebltoolkit.pbworks.com/f/eblcriticalappraisalchecklist.pdf http://www.shef.ac.uk/scharr/eblib/use.htm http://www.shef.ac.uk/scharr/eblib/use.htm http://www.shef.ac.uk/scharr/eblib/use.htm http://www.shef.ac.uk/scharr/eblib/use.htm http://eprints.rclis.org/7163/1/reliant__final_.pdf http://eprints.rclis.org/7163/1/reliant__final_.pdf http://www.phru.nhs.uk/pages/phd/resources.htm http://www.phru.nhs.uk/pages/phd/resources.htm evidence based library and information practice 2010, 5.1  157 appraisal is a worthwhile activity for  information professionals. bringing your own  knowledge and experience to the table in  combination with research in our field creates  a bridge that spans the research/practice gap.     next time, a look at applying research  evidence to practice, and in particular,  applicability: what is it? how do you find it?      references    booth, a. & brice, a. (2004). appraising the  evidence. in a. booth & a. brice (eds.)  evidence‐based practice for information  professionals: a handbook (pp. 104‐118).  london: facet.    booth, a. & brice, a. (n.d.).  appraising a user  study. in cristal online. retrieved  from  http://www.shef.ac.uk/scharr/eblib/us e.htm   booth, a. & brice, a. (n.d.).  appraising an   information needs analysis. in  cristal online. retrieved 20 feb. 2010  from  http://www.shef.ac.uk/scharr/eblib/ne eds.htm     critical appraisal skills program. (2007).  appraisal tools. in public health  resource unit. retrieved 20 feb. 2010  from  http://www.phru.nhs.uk/pages/phd/r esources.htm      glynn, l. (2006). a critical appraisal tool for  library and information research.  library hi tech, 24(3), 387‐399.    greenhalgh, t. & taylor, r. (1997). how to  read a paper: papers that go beyond  numbers (qualitative research). bmj,  315(7110), 740‐743.    koufogiannakis, d. (2006). small steps  forward through critical appraisal.  evidence based library and information  practice, 1(1), 81‐82.     koufogiannakis, d., booth, a., & brettle, a.  (2005). reliant: reader’s guide to  the literature on interventions  addressing the need for education and  training. lir, 30(94), 44‐51.  http://www.shef.ac.uk/scharr/eblib/use.htm http://www.shef.ac.uk/scharr/eblib/use.htm http://www.shef.ac.uk/scharr/eblib/needs.htm http://www.shef.ac.uk/scharr/eblib/needs.htm http://www.phru.nhs.uk/pages/phd/resources.htm http://www.phru.nhs.uk/pages/phd/resources.htm evidence based library and information practice evidence based library and information practice 2010, 5.2  evidence based library and information practice       evidence summary     library school curricula in the us should address liaison responsibilities for students  interested in academic librarianship    a review of:  attebury, r. i., & finnell, j. (2009). what do lis students in the united states know about liaison  duties? new library world, 110(7), 325‐340.     reviewed by:   nazi torabi  reference and instructional librarian, university of western ontario  london, ontario, canada  email: ntorabi@uwo.ca     received: 16 nov. 2009          accepted: 7 feb. 2010       2010 torabi. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which  permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly  attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the  same or similar license to this one.    abstract   objectives – the two main objectives of this  study were to determine the level of  prevalence of liaison work in academic library  job advertisements and to investigate whether  the current library & information science (lis)  students are aware of liaison duties.   design – the mixed methods used in this  study are job postings analysis and online  survey.     setting – the research settings were the  following:  (1) online academic job advertisements  published between november 15, 2007  and january 15, 2008 and collected  from chronicle of higher education’s  web site and lisjobs.com;  (2) fifty‐three electronic mail lists of  ala‐accredited library schools in the  us.      subjects – the subjects of the study were 313  online academic job advertisements and 516  lis students.      methods – the sample size and methodology  for the first part of this study were based on  four previously published studies. duplicated  job postings were removed and the remaining  were organized into 15 categories of  access/public services, reference, instruction,  bibliographer/subject specialist, combination  (instruction and reference), archives/special  collections, special libraries, director/dean,  department head or coordinator, interlibrary  loan (ill), systems/web development,  cataloguing, outreach, and  100 mailto:ntorabi@uwo.ca evidence based library and information practice 2010, 5.2  acquisitions/collection‐development. only  those job ads containing the term “liaison”  were included in the analysis.     for the second part of the study, the authors  conducted an online survey. they attempted  to investigate the knowledge of lis students  on liaison librarianship, to measure the level of  exposure to liaison responsibilities in their  course work, and to gauge the confidence of  the individual in their ability to become  successful liaison librarians. the survey was  distributed among 53 lis school electronic  mail lists, resulting in 516 respondents.     main results – the job ad analysis revealed  that 29% of job postings were directly related  to liaison duties. the liaison component of the  positions related to access/public services,  instruction, bibliographer/subject specialist,  special, and outreach were the highest (50% or  more). the liaison activities described in the  job ads related to reference, a combination of  reference and instruction, ill, department  head/coordinator, and system/web  development were also high (29% to 50%). the  positions categorized as librarian, archives,  director/dean, cataloguing, and collection  development/acquisitions had less liaison  responsibilities (<29%) (p. 331).     according to the survey results, lis students  are negatively affected by limited training for  liaison work. only 16.8% of students were  introduced to liaison responsibilities through a  required class and 16.5% heard about it in an  elective class. when these results were limited  to those who were interested in academic  librarianship, the numbers improved  somewhat to 20.8% and 23.9% for each group,  respectively (p. 332).     the survey compared the degree in which  those students who showed interest for  academic librarianship, with or without  exposure to liaison training, were aware of  some fundamental aspects of liaison work.  the first group provided better responses on a  different range of liaison activities,  appropriate communication methods, and  confidence level. among them only 1.3%  responded that they had never considered  liaison activity. on the other hand, more  people in the second group (enthusiasm for  academic librarianship with no exposure to  liaison activity) provided the same responses  (16.2%) (p. 332).     similar results were obtained when they asked  about communication methods that are  appropriate for liaison librarianship. self‐ confidence in the respondent’s ability to  become a successful liaison librarian was also  determined. overall, the self‐assessment  indicated that 42.5 % of lis students could see  themselves in the position of a successful  liaison librarian. the authors argued that this  technique was not the best method to assess  the level of self‐confidence without taking into  account personality characteristics and  previous library work experiences.     conclusion – the authors concluded that the  liaison component of academic library  positions is noticeable. the survey results  showed that the liaison training has a  considerable positive impact on students’  knowledge and confidence level (pp. 333‐ 334).the library schools in the us need to  undertake curriculum redesign to address  different components of liaison responsibilities  to lis students interested in academic  librarianship. the study did not present a  specific liaison training model but some broad  recommendations were provided.       commentary     this study raises some useful points about the  current state of liaison duties associated with  academic librarianship. however, greater care  should have been applied for selecting the  most appropriate and relevant studies as  models for job postings analysis. three out of  four studies cited here (chaudhry and  komathi, 2001; croneis and henderson, 2002;  white, 2000) have collected the job ads over  the period of nine or ten years (1990‐1999) (p.  329). one could argue that 313 academic job  ads are not representative of all academic job  postings in this current study. due to  101 evidence based library and information practice 2010, 5.2  102 inappropriate sampling method and size, they  have reported that only 50% of outreach  library positions require liaison  responsibilities. this is because there were  only two job postings in this category during  two months of data collection (attebury, 2009,  p. 331). the sample size should have been  based on statistical methods and the total  number of library job ads posted  annually. the authors should have conducted  a random sampling for the longer period of  time to reach a more adequate sample size.  in  addition, the authors did not provide any  rational for selecting chronicle of higher  education’s web site and lisjobs.com as the  two resources for data collection (p. 329).      they applied a very rigid coding scheme by  including only those job ads containing the  term “liaison” (p. 329). while this approach  increased the intercoder reliability, the content  validity of analysis suffered. there were only  two job ads that contained the term “liaison”  in the title. this shows that the employers  view liaison responsibilities very differently.  therefore, a different coding scheme to  support a higher content validity should have  been applied.        they have stated the limitation of their survey  instrument in the second part of the study  clearly.  the students were not asked how  many courses they had taken in their program  nor whether they had prior experience  working in the library. the geographical  location of the students was not identified  either. since the authors have relied on  convenience sampling, the 516 survey  responds might not be representative of the  entire population (p. 330).    there is an inconsistency on how many library  schools participated in the survey. in the  abstract section, the authors report that 52  library school electronic mail lists were  contacted while on page 330, they report 53  schools. also, four or five schools were not  contacted at all, but the authors did not  provide any explanation as to why this  occurred. one could assume that canadian  schools listed in the ala website were not  contacted but this is not clear in the article.         stronger research design, sampling methods,  and evidence is required to show the extent of  lis students’ knowledge about liaison  activities. while the limitation of the survey  affects our ability to draw a concrete  conclusion, it seems that librarians‐in‐training  will benefit from exposure to the concept of  liaison roles and responsibilities. we should  bear in mind that the goal of any ala‐ accredited lis program is to prepare its  graduates for their first professional position  as much as possible. given the current state of  liaison roles and responsibilities, it seems  impossible for the curricula to capture every  details of every type of job that is available. in  addition, different organizations operate  differently and the way employers view  liaison works varies. it would be a great  challenge for library schools to consider all  those differences during curricula redesign.         the main implication of this study is to  provide the hiring institutions and academic  libraries with some understanding about the  extent of lis students’ knowledge about  liaison duties. they can use this information to  establish orientation and training programs  for new hires and create a liaison manual  specific to their institution.         references     chaudhry, a. s., & komathi, n. c. (2001).  requirements for cataloguing  positions in the electronic  environment. technical services  quarterly, 19(1), 1‐23.     croneis, k. s., & henderson, p. (2002). electronic  and digital librarian positions: a  content analysis of announcements  from 1990 through 2000. journal of  academic librarianship, 28(4), 232‐237.     white, g. w. (2000). head of reference positions  in academic libraries. reference & user  services quarterly, 39(3), 265‐272.   research article   information seeking anxiety and preferred information sources of first-generation college students   stacy brinkman head of education and outreach university of california, irvine irvine, california, united states of america email: brinkmas@uci.edu   josefine smith assistant professor, instruction and assessment librarian shippensburg university shippensburg, pennsylvania, united states of america email: jmsmith@ship.edu   received: 4 sept. 2020                                                               accepted: 3 jan. 2021      2021 brinkman and smith. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29843       abstract   objective – to determine whether information seeking anxieties and preferred information sources differ between first-generation college students and their continuing-generation peers.   methods – an online survey was disseminated at two public college campuses. a total of 490 respondents were included in the results. independent variables included institution, year in college, and generational status. instead of using a binary variable, this study used three groups for the independent variable of generational status, with two first-generation groups and one continuing-generation group based on parental experience with college. dependent variables included 4 measures of information seeking anxiety and 22 measures of preferred information sources. responses were analyzed using spss. one-way independent anova tests were used to compare groups by generational status, and twoand three-way factorial anova tests were conducted to explore interaction effects of generational status with institution and year in college.   results – no significant differences in overall information seeking anxiety were found between students whose parents had differing levels of experience with college. however, when exploring the specific variable of experiencing anxiety about “navigating the system in college,” a two-way interaction involving generational status and year in school was found, with first-generation students with the least direct experience with college reporting higher levels of anxiety at different years in college than their peers. two categories of first-generation students were found to consult with their parents far less than continuing-generation peers. the study also found that institutional or generational differences may also influence whether students ask for information from their peers, librarians, tutoring centers, professors, or advisors.   conclusion – this study is one of the first to directly compare the information seeking preferences and anxieties of first-generation and continuing-generation students using a non-binary approach. while previous research suggests that first-generation students experience heightened anxiety about information seeking, this study found no significant overall differences between students based on their generational status. the study reinforced previous research about first-generation college students relying less on their parents than their continuing-generation peers. however, this study complicates previous research about first-generation students and their utilization of peers, librarians, tutoring centers, professors, or advisors as information sources, and suggests that institutional context plays an important role in shaping first-generation information seeking.     introduction   in the past three decades, the number of individuals attending higher education for a bachelor’s degree has increased: according to the current population survey, 33.4% of adults over 25 in 2016 held a bachelor’s degree, a figure that has increased from 4.6% in 1940 (u.s. census bureau, 2017). one group that has been receiving increasing attention is first-generation (fg) college students, a population that accounts for up to 56% of undergraduates, depending on the parameters used to define this group (center for first-generation student success, 2019). while a large body of literature exists on characteristics of fg students, less is known about fg students’ information seeking behavior, particularly in comparison to non-fg students.   the current paper builds on previous research that explored fg students’ information seeking strategies, as well as their self-perceptions of their information seeking abilities (brinkman et al., 2013). brinkman et al. found relationships between affective concerns of information seeking anxiety and academic information seeking behaviors in fg students, but did not compare fg students to non-fg students. our study adds to the existing literature by exploring levels of information seeking anxiety as well as information source preferences and comparing responses from categories of fg and non-fg students, and also samples students from two institutions.   literature review   historically, researchers studying fg students emphasize the “challenges” that this population faces (ilett, 2019). surveys conducted by the national center for educational statistics indicated that fg students were more likely to come from lower socioeconomic backgrounds, were ethnic minorities, had taken fewer college-preparatory classes (choy, 2001), or were more likely to have children and work full-time while enrolled (nunez & cuccaro-alamin, 1998). in early foundational studies, researchers discussed several challenges faced by fg students in higher education: lower levels of persistence and academic success, differing experiences in higher education, and their need for academic intervention (chen & carroll; 2005; engle & tinto, 2008; pascarella et al, 2004; terenzini et al, 1996). in addition, authors of several qualitative studies suggested that fg students experienced anxieties from impostor syndrome or feeling like an outsider (london, 1992; whitehead & wright, 2017). in effect, the dominant mode for discussing fg college students has been through the language of the deficit model (valencia, 1997) – framing a population’s differences from the dominant group as “deficiencies,” and exploring ways to support a non-dominant population so that they can “overcome” these deficiencies.   another problematic trend has been the lack of clarity around the term “first-generation.” first defined in the higher education act (1965) with the creation of the federal trio programs, “first-generation college student” originally meant “(a) an individual both of whose parents did not complete a baccalaureate degree; or (b) in the case of any individual who regularly resided with and received support from only one parent, an individual whose only such parent did not complete a baccalaureate degree” (p. 3-4). however, many researchers and institutions defined “first-generation” differently: peralta and klonowski (2017) found 9 different definitions for this label, from parents with no schooling past high school to parents who may have attended a 4-year institution but did not complete a bachelor's degree. among policymakers or school administrators, the term “first-generation” can be used as a catch-all or substitute phrase for various “underprivileged” identities such as race, ethnicity, or class (sharpe, 2017). other scholars have noted that because the category “first-generation” is typically constructed in studies as a binary variable (first-generation vs. continuing-generation), the way “first generation” is defined can lead researchers to drawing different conclusions about fg students as a population (toutkoushian et al., 2019).   in the past decade, a growing body of scholarship on fg students has emerged in the library and information science field. in a critical review of this literature, ilett (2019) identified four dominant themes in discussing fg students: they are presented as (1) outsiders, (2) a problem, (3) reluctant library users, and (4) capable students. a few researchers have focused on fg information seeking behaviors. some researchers suggested that fg students may prefer different formats of information sources, such as preferring to use online reference sources (soria et al., 2015) or preferring to seek information from peers and pamphlets over advisors and mentors (torres et al., 2006). logan and pickard (2012) found that fg students were most likely to seek help from instructors or teaching assistants, and unlikely to seek help from librarians or family members. tsai (2012) found that, when seeking information about coursework, fg students were not likely to consult family members, but turned to peers instead. fg students in another study expressed frustration in not only their inability to turn to parents for information, but also in their perception that for other students, “their parents are their mentors and they can tell them what to do” (brinkman et al., 2013, p. 646). significantly, however, none of the studies on fg information seeking directly compared fg students to other populations.   aims   in this study, we explored whether information seeking patterns or anxieties differ between students whose parents have different levels of college experience. we separated generational status into three variables: fg-no college (neither parent attended college), fg-attended (one or more parents may have attended college, but none graduated), and cg (continuing generation, at least one parent graduated from college). the main research questions were as follows:   q1: do students report different levels of anxiety in seeking information on college campuses based on generational status?   q2: do students of different generational statuses report different preferences for information sources about questions related to academics?   q3: do students of different generational statuses report different preferences for information sources about questions related to college life?   methods   research was conducted at two public, four-year residential universities, one in the midwestern united states and one in the eastern united states. the student population was predominantly white and traditionally aged at both institutions. at the time the study was conducted, institution a enrolled approximately 19,000 students, and institution b enrolled approximately 7,000 students.    each author disseminated an online survey at their home institution. the study was reviewed and deemed exempt by both institutional review boards. however, slightly different sampling methods and tools were used based on the tools and protocols available to each institution. at institution a, the office of institutional research prepared a randomized sample of 2000 undergraduate participants with a 200% oversampling of fg students in order to ensure that enough fg students were included in the sample. to encourage participation in the survey, students were eligible to win one of five $50 amazon gift cards. prior to data cleaning, 326 initial responses were collected, for a response rate of 16%. data were collected in qualtrics. at institution b, the office of research and institutional assessment provided a population list of all 6,305 enrolled undergraduate students. through this method, 208 initial responses were collected for a response rate of 3%. data were collected in google forms. surveys and follow-up emails were sent at the end of the fall semester and at the beginning of the spring semester at both institutions. data were imported into spss for analysis.   instrumentation   demographics   eight demographic questions were collected in this study: participants’ year in school, age, gender, whether they identified as an international student, whether they had a sibling who attended college before them, parental level of education, self-reported estimated grade point average (gpa), and major.   generational status   student responses to the demographic question on the highest level of parental education were re-coded into the following three variables in order to avoid a binary variable for generational status, while still maintaining a large enough sample size in each category to conduct valid tests:   ●       first-generation, no college (fg-nc): students who reported that neither parent attended college ●       first-generation, attended (fg-a): students whose parents may have attended college, but did not graduate ●       continuing-generation (cg): students who reported at least one parent who graduated from college   information seeking behaviors   twenty-six exploratory survey questions (see appendix) regarding information seeking were developed from data collected in a qualitative study by brinkman et al. (2013).    ●       college information seeking anxiety. four questions about student anxiety levels about information seeking on campus were based on recurring statements made by students who participated in the previous qualitative study. students were asked to rate their agreement with four statements on a likert scale from 0-10. two statements were framed positively and two statements were framed negatively. ●       college information sources. twenty-two questions concerning information sources were also included. one set of 11 questions asked students to use a 5-point likert scale to rate their likelihood of seeking help from specific information sources when seeking information about academics. the other set of 11 questions asked students to use a 5-point likert scale to rate their likelihood of seeking help from the same set of information sources if they were seeking information about college life.   results   the initial data set included 534 participants. through data-cleaning procedures, we identified 44 participants who skipped more than 10% of the survey. these cases were excluded listwise, yielding a final data set of 490 responses, with 59.4% (n = 291) from institution a and 40.6% (n = 199) from institution b. students were distributed across by year in college (19.7% first year, 24% sophomore, 23.2% junior, 32.8% senior, and 0.4% “other”). the majority of respondents (71%) identified as female and reported their age range as 18-22 years old (88.5%). a portion of students (40.6%) indicated that they had an older sibling who attended college before them. the majority of students were high achievers: 38.4% of students reported a cumulative gpa of 3.5 or higher, and an additional 34.7% reported a gpa between 3.0 and 3.49. based on the highest reported level of education by their parents, 20.5% of students (n=100) were coded as firstgeneration, no college (fg-nc), 16.4% (n=80) as first-generation, attended (fg-a), and 63.1% (n=308) as continuing-generation (cg).   table 1 college information seeking anxiety levels by generational status means based on generational status   fg-nc fg-a cg total f-test i don't know who to turn to if i have questions about college 3.36 (2.16) 3.51 (2.28) 3.67 (2.28) 3.58 (2.26) f(2, 477) = .75, p = .48 other students around me know more about college than i do 5.18 (2.54) 5.43 (2.31) 5.25 (2.39) 5.27 (2.41) f(2, 465) = .24, p = .79 people on campus are not helpful when i ask them questions 3.76 (2.27) 3.87 (2.09) 4.15 (2.08) 4.02 (2.12) f(2, 449) = 1.44, p = .24 it is difficult to navigate the system in college 4.74 (2.44) 5.06 (2.53) 4.96 (2.48) 4.93 (2.47) f(2, 476) = .41, p = .66   figure 1 agreement with q4 “it is difficult to navigate the system in college”: interaction of generational status and year in school.   college information seeking anxiety   after calculating the mean, an initial one-way anova was used to explore the relationship of generational status on college information seeking anxiety. no significant effects were found, and students reported low-to-medium levels of anxiety overall. table 1 summarizes means, standard deviations, and overall effects.   because we sampled students across four years in college and from two institutions, a series of three-way anovas were conducted to explore the main effects of generational status and the interaction effect between generational status, institution, and year in college on college information seeking anxiety variables. while no significant three-way interactions between all three variables of generational status, institution, and year in school were found for any of the questions, a two-way interaction involving generational status and year in school was found for question 4 (“it is difficult to navigate the system in college”), f(6, 453) = 2.322, p = .03. specifically, fg-nc students reported the lowest levels of difficulty navigating the system during their first year m = 3.47 (sd = 1.77) and the highest levels of difficulty during their second year m = 5.42 (sd = 2.59), decreased difficulty in their third year m = 4.56 (sd = 2.35), and increased difficulty again in their final year m = 5.03 (sd = 2.53). fg-a students displayed a similar pattern to cg students for the first three years of college, with decreasing levels of reported difficulty in navigating the system with each passing year. however, in their final year of college, fg-a students reported a sharp increase in difficulty navigating the system m = 5.32 (sd = 2.77), whereas cg students continued to report lower levels of difficulty in navigating the system m = 4.43 (sd = 2.58). figure 1 illustrates these differences.   table 2 one-way anova results across academic information source variables means based on generational status   fg-nc fg-a cg total f-test parent 2.16 (1.30) 2.74 (1.43) 3.50 (1.33) 3.10 (1.45) f(2, 485) = 40.89, p < .01* friend 3.62 (1.17) 3.90 (.89) 4.08 (.85) 3.95 (.95) f(2, 483) = 9.13, p < .01* other relative 2.60 (1.41) 2.48 (1.28) 2.47 (1.22) 2.50 (1.27) f(2, 481) = .43, p = .65 professor 4.41 (.85) 4.38 (.70) 4.37 (.70) 4.38 (.74) f(2, 483) = .10, p = .91 academic advisor 4.10 (1.10) 3.89 (1.34) 3.95 (1.23) 3.97 (1.22) f(2, 482) = .77, p = .46 resident advisor 2.20 (1.31) 2.48 (1.33) 2.37 (1.25) 2.36 (1.28) f(2, 484) = 1.12, p = .33 librarian 2.66 (1.30) 2.68 (1.34) 2.30 (1.12) 2.43 (1.21) f(2, 485) = 5.47, p < .01* tutoring center 2.78 (1.28) 2.69 (1.31) 2.37 (1.19) 2.51 (1.24) f(2, 483) = 5.22, p < .01* coworker or supervisor 2.51 (1.40) 2.58 (1.24) 2.46 (1.22) 2.49 (1.26) f(2, 480) = .25, p = .78 i would look it up on my own 3.69 (1.30) 3.68 (1.17) 3.50 (1.24) 3.57 (1.24) f(2, 482) = 1.22, p = .30 other 2.46 (1.24) 2.33 (.96) 2.14 (1.16) 2.26 (1.15) f(2, 172) = 1.31, p = .27 * significant at the 0.01 level   college information sources: academic information   students were asked to rate their likelihood of consulting with ten potential information sources when they had questions about academics. they were also given the opportunity to select other and write in a response. the most common write-in response was a synonym of “spouse/partner” (n=4), but the majority of students selecting other left the write-in section blank. we used a one-way anova to examine the effect of generational status across the ten information source variables. generational status had a significant overall effect on whether students were likely to consult the following sources for academic information: parents f(2, 485) = 40.89, p < .01, friends f(2, 483) = 9.13, p < .01, librarians f(2, 485) = 5.47, p < .01 and tutoring centers f(2, 483) = 5.22, p < .01. table 2 summarizes means, standard deviations, and anova results.   we used a series of three-way anovas to examine whether institution or year in college interacted with generational status on likely academic information sources. no significant three-way interactions were observed for any academic information source variables. however, significant two-way interactions with generational status and institution were found for the variables “friend” f(2, 460) = 5.089, p = .007 and “librarian” f(2, 462) = 3.306, p = .038. figure 2 illustrates that fg-nc students at institution a were significantly less likely to consult with friends for academic information (m = 3.24, sd = 1.36) than fg-nc students at institution b (m = 3.84, sd = .99). there were no significant differences between institutions for fg-a students (institution a m = 3.93, sd = .99; institution b m = 3.87, sd = .81) or cg students (institution a m = 4.08, sd .84; institution b m = 4.07, sd = .88).   for the variable “librarian,” figure 3 illustrates that cg students at institution a were far less likely to ask a librarian for help with academic information (m = 2.18, sd = 1.01) than cg students from institution b (m = 2.54, sd = 1.18). furthermore, both fg-nc and fg-a students at institution a (fg-nc m= 2.76, sd = 1.42, fg-a m = 2.83, sd = 1.26) were more likely to consult with a librarian as an institutional source than similar groups of students at institution b (fg-nc m = 2.60, sd = 1.24, fg-a m = 2.53, sd 1.33).   figure 2 likelihood of asking a friend for academic information: interaction of generational status and institution.   figure 3 likelihood of asking a librarian for academic information: interaction of generational status and institution.   table 3 one-way anova results across college life information source variables means based on generational status   fg-nc fg-a cg total f-test parent 2.22 (1.40) 2.54 (1.39) 3.31 (1.36) 2.96 (1.45) f(2, 484) = 28.135, p < .001** friend 3.93 (1.23) 4.35 (.87) 4.55 (.65) 4.39 (.87) f(2, 484) = 21.223, p < .001** other relative 2.53 (1.41) 2.65 (1.29) 2.79 (1.36) 2.71 (1.36) f(2, 481) = 1.557, p = .212 professor 2.78 (1.35) 2.59 (1.25) 2.40 (1.07) 2.51 (1.17) f(2, 482) = 4.144, p = .016* academic advisor 2.71 (1.36) 2.46 (1.23) 2.34 (1.16) 2.43 (1.22) f(2, 483) = 3.596, p = .028* resident advisor 2.57 (1.52) 2.90 (1.52) 2.80 (1.32) 2.77 (1.40) f(2, 481) = 1.451, p = .235 librarian 2.06 (1.14) 1.81 (.98) 1.65 (.86) 1.76 (.95) f(2, 482) = 7.792, p = .001** tutoring center 2.08 (1.14) 1.75 (.97) 1.65 (.87) 1.75 (.96) f(2, 481) = 7.792, p < .001** coworker or supervisor 2.64 (1.45) 2.85 (1.38) 2.53 (1.26) 2.60 (1.32) f(2, 482) = 1.871, p = .155 i would look it up on my own 3.52 (1.46) 3.40 (1.25) 3.40 (1.23) 3.42 (1.28) f(2, 481) = .316, p = .729 other 2.36 (1.24) 2.52 (1.15) 2.13 (1.09) 2.27 (1.15) f(2, 157) = 1.506, p = .225 * significant at the 0.05 level ** significant at the 0.01 level   college information sources: college life information   the next set of questions asked students to rate their likelihood of consulting with an information source when seeking information about college life. we used a one-way anova to examine the effect of generational status across information source variables. significant differences were found for multiple variables. students whose parents had less college experience were less likely to turn to parents f(2, 484) = 28.135, p < .001 and friends f(2, 484) = 21.223, p < .001 for information about college life, but more likely to turn to professors f(2, 482) = 4.144, p = .016, academic advisors f(2, 483) = 3.596, p = .028, librarians f(2, 482) = 7.792, p = .001, and the tutoring center f(2, 481) = 7.792, p < .001. table 3 summarizes means, standard deviations, and anova results.   interactions between generational status and institution or year in college were also explored through a series of three-way anovas. a two-way interaction between generational status and institution was significant for the variable “friend” f(2, 461) = 3.204, p = 0.42. specifically, fg-nc students at institution a were less likely to consult with friends for college life information (m = 3.68, sd = 1.31) than fg-nc students at institution b (m = 4.08, sd = 1.26). see figure 4.   a two-way interaction was also found for generational status and year in school for the likelihood of asking a professor for college life information f(6, 458) = 2.385, p = .028. figure 5 illustrates how fg-nc, fg-a, and cg students reported different patterns of behavior by year. fg-nc students were most likely to consult with professors in their junior year (m = 3.35, sd = 1.23) but least likely in their senior year (m = 2.54, sd = 1.30). fg-a students followed a different pattern, and were most likely to consult with professors in their first year (m = 2.83, sd = 1.38) declining each year to their senior year (m = 2.32, sd = 1.29). cg students, however, were the most likely to consult with professors during their senior year (m = 2.50, sd = 1.19).   finally, a two-way interaction was found between generational status and year in school for the variable “academic advisor” f(6, 460) = 2.555, p = .019. see figure 6. fg-nc students reported a significantly higher likelihood (m = 3.35, sd = 1.33) of consulting with an academic advisor for college life information in their junior year, whereas fg-a and cg students reported declining or flat likelihood of asking an academic advisor for college life information as they advanced toward their senior year.   figure 4 likelihood of asking a friend for college life information: interaction of generational status and institution.   figure 5 likelihood of asking a professor for college life information: interaction of generational status and year in school.   figure 6 likelihood of asking an academic advisor for college life information: interaction of generational status and year in school.   discussion   information seeking anxiety   the main purpose of this study was to determine if generational status had any effect on college student information seeking anxiety and sources. in contrast with previous studies that suggested that fg students may have experienced increased anxiety or feelings of impostor syndrome (brinkman et al., 2013; london, 1989; whitehead & wright, 2017) in this study we did not find that fg students reported higher anxiety overall about information seeking than their cg peers. on only one information seeking anxiety variable (the statement “it is difficult to navigate the system in college”) did the responses of fg-nc students follow a different curve than those of other students: fg-nc students did not find the “system” in college to be particularly difficult to navigate in their first year, whereas fg-a and cg students thought college was the most difficult to navigate in their first year. in their second year, however, fg-nc students reported much higher levels of anxiety about navigating the system in college while other groups reported decreasing levels of concern. finally, both fg-nc and fg-a students found navigating the system more difficult in their senior year.   the dunning-kruger effect provides one potential explanation for the variation in fg student responses over time. with this theory, kruger and dunning (1999) described how individuals who lacked “competence” or expertise in a domain tended to lack the metacognitive skills in evaluating their own performance, and consequently tended also to be overly optimistic and confident in their abilities in that domain. in the current study, fg-nc may have reported very low levels of concern about navigating the system in college because they didn’t know enough about what there was to navigate, whereas other groups of students with more familial knowledge about the college system might have less confidence. this may have been particularly true for first-year students in this particular study, since the surveys were sent at the end of the students’ fall semester, with follow-up emails sent at the beginning of the spring semester. therefore, students would have only had one semester of experience in trying to “navigate the system in college” on which to base their responses. by their sophomore year, students had more time to develop awareness of the nature of the domain (navigating the system in college), and therefore their evaluation of their abilities to navigate that domain changed and they became less confident. however, further research would be needed to establish such links.   the overall result that fg students showed no more agreement with the information seeking anxiety statements than cg students was surprising, because the statements for the current study were formed from brinkman et al.’s (2013) qualitative study on fg students, in which “many first-generation students perceived that other students could ask their parents when they had questions about the ‘big picture’ of navigating college life, whereas they could not” (p. 648). in other qualitative studies based on interviews and focus groups, fg students reported feeling like “outsiders,” or lacking information or capital when compared with non-fg students (bergerson, 2007; cushman, 2007; london, 1992). while the current study does not disprove these previous studies, we do suggest that fg students may have internalized a sense of deficit that they have then attributed to their identity as first-generation. this phenomenon is interesting and worthy of future research, as other studies have suggested that the “first-generation college student” identity is a relatively newly formed identity for fg students in comparison to other intersecting identities such as race, gender, and class (orbe, 2004). fg students are continually forming and performing this new identity while in college and, if their identity as a “college student” is still relatively weak, they may therefore experience impostor syndrome (whitehead & wright, 2017). it is possible that intervention efforts targeted to fg students that emphasize deficits in information, experience, or capital may increase fg students’ internalization of deficit thinking and impede their ability to form strong identities as fg students who “belong” in college, thus causing them to feel that other students know more about college, or fit in better, than they do. more research is needed to explore these potential connections.   information seeking sources   in this study, we confirmed previous research that parents are a low information source for fg students (logan & pickard, 2012; tsai, 2012). however, while previous research has suggested that friends or peers are very high information sources (tsai, 2012), we found that compared with continuing-generation students, fg students were less likely to ask their peers for both academic or college life information. there may be several reasons for this. for example, if an institution offers highly visible alternative support programs or information pathways specifically for fg students, this may also alter the likelihood of fg students consulting with their peers for information. at both institutions in this study, fg students reported being more likely to seek information (both academic and non-academic) from the campus tutoring center. if a campus makes tutoring services more visible to this population, then this can explain why fg students would report seeing this service as a resource. the same phenomenon can be observed with librarians: fg students at institution a also reported being more likely to consult with librarians for academic information than their cg peers. this runs counter to previous studies that suggested that fg students were reluctant library users (ilett, 2019; logan & pickard, 2012; long, 2011). intervention efforts by librarians may explain some of these differences: at institution a, librarians had been involved for several years in campus wide programs and courses aimed at fg students, including offering fg-specific orientations.   however, the visibility of resources such as tutoring services or library services does not alone explain why fg students might be less likely to ask their friends for information about college life or academics. as evidenced in figures 2 and 5, the most pronounced difference in seeking information from friends was in fg students from families with no college experience (fg-nc) at institution a. campus culture may also provide an important explanation for the reason why fg students at one institution may be less likely to ask their friends for information about college. institution a is a more selective university, has a small overall percentage of fg students in their total student body, and also has a considerable percentage of “legacy” students (meaning their parents, siblings, or other relatives attended the university). in research on fg identity, orbe (2004) suggested that for some fg students, “especially those who were attending more selective universities, coming from a family without college degrees was ‘embarrassing’” (p. 143). thus, fg students at institution a who felt themselves to be in a minority group may have felt reluctant to disclose to their peers that they lacked knowledge about college, and may have consequently sought alternative pathways to information, such as librarians, tutors, or other support services. fg students at institution b, on the other hand, may not have felt as different or marginalized in comparison to their peers, and may therefore have felt more comfortable asking their peers for information.   we also found an inverse relationship with parental experience in college and the likelihood of students turning to academic sources, such as professors, advisors, librarians, and tutors, for non-academic information about college life. this finding was similar to that of given (2002) in a study of mature undergraduate students, who tended to turn to on-campus academic sources for everyday life information seeking needs such as childcare. we also found an interesting pattern, where fg-nc students were mostly likely to report seeking information about college life from an academic source (professor or academic advisor) in their junior year. brinkman et al. (2013) suggested that some students felt a “perceived a lack of follow-up” with campus support systems after their first and second year (pp. 645-646), which, if true, may partially explain why students would turn to alternative sources of information in their junior year. an alternative explanation could be that students by their junior year were more embedded into their major field of study and had identified faculty members who had become their mentors. a third explanation could be that students would be more likely to be living in off-campus housing starting in their third year, particularly at institution a, which had a two-year residential requirement. further research would be needed to establish the motivations of students for seeking out non-academic information from academic sources at specific years in their college career, as well as to establish what kinds of non-academic information was being sought.   limitations   although this study extends existing literature on information seeking behavior in first-generation college students, there are several limitations. first, data were collected from self-report surveys that, while based on previous qualitative research, were not validated. there was no way to verify the accuracy of a participant’s response. data were collected from two predominantly white four-year public institutions from the midwest and east. it is possible that these results may not generalize to institutions that have different demographic or geographic compositions, and may also not generalize to two-year institutions. finally, the survey did not account for the growing use of social media and unofficial information networks such as reddit or online communities for information that have increased in popularity since this study was conducted. future studies should take these networks into account more rigorously.   conclusions and directions for future research   in this study, we found that there were no general differences in information seeking anxiety between students whose parents had differing levels of experience with college. however, one variable exposed that students who were the first in their family to go to college experienced levels of anxiety about “navigating the system in college” during very different times than their peers. we confirmed that first-generation students consulted with their parents far less than their continuing-generation peers. we also found that institutional or generational differences may influence whether students ask for information from their peers, librarians, tutoring centers, professors, or advisors.    the results of this study have several possible implications for library practice. most broadly, this research demonstrated that framing services and support for fg students as “at risk” can be problematic at best, and can also be counterproductive or marginalizing. this research is part of a growing body of literature calling for more critical reflection on inclusive library practice. rather than creating prescriptive programming that reinforces an “at risk” narrative for fg students, libraries and librarians have an opportunity to engage fg students more holistically. for both authors, this current research has influenced how we approach instruction to focus more on metacognitive aspects of information literacy based on the students’ learning experiences and a reflection on their understanding. in practice, this might translate to an increase in reflective activities in a library session, enabling the librarian to adapt their lesson in response to the student learning experience. shifting to a more responsive instructional practice creates a space for the student holistically and avoids transactional, “banking” models of pedagogy (freire, 2000).    the other important takeaway from this study is that fg students are not a homogenous group; rather, they are negotiating their identities and navigational strategies within a campus culture over time. it is important for librarians to understand their own institutional culture and context, whether it is in learning more about campus demographics as a whole, or in identifying groups on campus that are already providing services for fg students. in a 2019 paper, brinkman, natale, and smith discussed examples of how libraries can collaborate with student affairs units in promoting existing programs that celebrate fg identity, or can situate library services in a larger context of resources for student success. as an example, one author of this paper was invited to staff a library table at a campus-wide “first-gen day” event. rather than using the table to distribute library brochures, the table became a zine-making workshop station, covered with magazines, scissors, glue, stencils, pens, and pencils. students were invited to make a page for a collaborative zine on “what first-gen means to me,” which was then included in the university archives and distributed digitally to contributors and participants. in the course of inviting students to become authors of their own unique stories and then archiving them, fg students and library staff had the opportunity to converse about other library services. this example demonstrates the effectiveness of creating programming specifically for fg students that is aligned with campus outreach activities, while also celebrating students’ identity holistically.   this study also exposed several areas for further research. one particular line of inquiry is that of the intersections of information seeking, first-generation identity formation, and campus culture. the current research suggested that fg students do lack a major pathway (parents) that continuing-generation students use for academic and non-academic information. interventions may help forge alternative pathways for such information. at the same time, interventions – especially if framed in the language of deficit may reinforce a campus culture where fg students may feel singled out, or choose not to disclose their fg identity to their peers for risk of embarrassment, or alternatively, may cause fg students to internalize a sense of deficit (orbe, 2004). framing interventions through other approaches, such as “funds of knowledge” approaches (ilett, 2019), strengthening fg students’ identity as “college students” by presenting college as a path to “something greater” than college itself (whitehead & wright, 2017), or placing more value on the capital that fg students possess rather than the capital they lack (bergerson, 2007), may be helpful areas of future investigation.    references   bergerson, a. a. (2007). exploring the impact of social class on adjustment to college: anna’s story. international journal of qualitative studies in education, 20(1), 99–119. https://doi.org/10.1080/09518390600923610   brinkman, s., gibson, k., & presnell, j. (2013, april 10-13). when the helicopters are silent: the information seeking strategies of first-generation college students [paper presentation]. association of college & research libraries 2013 conference, indianapolis, in, united states. http://www.ala.org/acrl/sites/ala.org.acrl/files/content/conferences/confsandpreconfs/2013/papers/brinkmangibsonpresnell_when.pdf   brinkman, s., natale, j., & smith, j. (2019). meeting them where they are: campus and library support strategies for first-generation students. in n-y. tran & s. higgins (eds.), supporting today’s students in the library: strategies for retaining and graduating international, transfer, first-generation, and re-entry students (pp. 185-198). acrl.   center for first-generation student success. (2019). first-generation college students: demographic characteristics and postsecondary enrollment. https://firstgen.naspa.org/files/dmfile/factsheet-01.pdf   chen, x., & carroll, c. d. (2005). first-generation students in postsecondary education: a look at their college transcripts. postsecondary education descriptive analysis report. nces 2005-171 (ed485756). eric. https://files.eric.ed.gov/fulltext/ed485756.pdf   choy, s. (2001). students whose parents did not go to college: postsecondary access, persistence, and attainment (nces 2001–126). national center for education statistics. https://nces.ed.gov/pubs2001/2001126.pdf   cushman, k. (2007). facing the culture shock of college. educational leadership, 64(7), 44–47.   engle, j., & tinto, v. (2008). moving beyond access: college success for low-income, first-generation students (ed504448). eric. https://files.eric.ed.gov/fulltext/ed504448.pdf    freire, p. (2000). pedagogy of the oppressed, 30th anniversary edition (m. b. ramos, trans.). bloomsbury. (original work published 1968)   given, l. m. (2002). the academic and the everyday: investigating the overlap in mature undergraduates’ information–seeking behaviors. library & information science research, 24(1), 17–29. https://doi.org/10.1016/s0740-8188(01)00102-5   higher education act of 1965, 20 u.s.c. ch. 28, subch. iv, part a § 1070a–11(1965). https://www.govinfo.gov/content/pkg/statute-79/pdf/statute-79-pg1219.pdf   ilett, d. (2019). a critical review of lis literature on first-generation students. portal: libraries and the academy, 19(1), 177–196. https://doi.org/10.1353/pla.2019.0009   kruger, j., & dunning, d. (1999). unskilled and unaware of it: how difficulties in recognizing one’s own incompetence lead to inflated self-assessments. journal of personality and social psychology, 77(6), 1121–1134. https://doi.org/10.1037/0022-3514.77.6.1121   logan, f., & pickard, e. (2012). first-generation college students: a sketch of their research process. in l. m. duke & a. d. asher (eds.), college libraries and student culture: what we now know (pp. 1–208). ala editions.   london, h. b. (1989). breaking away: a study of first-generation college students and their families. american journal of education, 97(2), 144–170.   london, h. b. (1992). transformations: cultural challenges faced by first-generation students. new directions for community colleges, 1992(80), 5-11. https://doi.org/10.1002/cc.36819928003   long, d. (2011). latino students’ perceptions of the academic library. the journal of academic librarianship, 37(6), 504–511. https://doi.org/10.1016/j.acalib.2011.07.007   nunez, a-m., & cuccaro-alamin, s. (1998). first-generation students: undergraduates whose parents never enrolled in postsecondary education (nces 98-082). national center for education statistics. https://nces.ed.gov/pubs98/98082.pdf   orbe, m. p. (2004). negotiating multiple identities within multiple frames: an analysis of first‐generation college students. communication education, 53(2), 131–149. https://doi.org/10.1080/03634520410001682401   pascarella, e. t., pierson, c. t., wolniak, g. c., & terenzini, p. t. (2004). first-generation college students: additional evidence on college experiences and outcomes. the journal of higher education, 75(3), 249–284. https://doi.org/10.1080/00221546.2004.11772256   peralta, k. j., & klonowski, m. (2017). examining conceptual and operational definitions of “first-generation college student” in research on retention. journal of college student development, 58(4), 630–636. https://doi.org/10.1353/csd.2017.0048   sharpe, r. (2017, november 3). are you first gen? depends on who’s asking. the new york times. https://www.nytimes.com/2017/11/03/education/edlife/first-generation-college-admissions.html   soria, k. m., nackerud, s., & peterson, k. (2015). socioeconomic indicators associated with first-year college students’ use of academic libraries. the journal of academic librarianship, 41(5), 636–643. https://doi.org/10.1016/j.acalib.2015.06.011   terenzini, p. t., springer, l., yaeger, p. m., pascarella, e. t., & nora, a. (1996). first-generation college students: characteristics, experiences, and cognitive development. research in higher education, 37(1), 1–22.   torres, v., reiser, a., lepeau, l., davis, l., & ruder, j. (2006). a model of first-generation latino/a college students’ approach to seeking academic information. nacada journal, 26(2), 65–70. https://doi.org/10.12930/0271-9517-26.2.65   toutkoushian, r. k., may-trifiletti, j. a., & clayton, a. b. (2019). from “first in family” to “first to finish”: does college graduation vary by how first-generation college status is defined? educational policy. advance online publication. https://doi.org/10.1177/0895904818823753   tsai, t-i. (2012). coursework-related information horizons of first-generation college students. information research, 17(4). http://informationr.net/ir/17-4/paper542.html#.xviu0mo3noq   united states census bureau. (2017, march 30). highest educational levels reached by adults in the u.s. since 1940 [press release]. https://www.census.gov/newsroom/press-releases/2017/cb17-51.html   valencia, r. r. (1997). the evolution of deficit thinking: educational thought and practice. routledge. https://doi.org/10.4324/9780203046586   whitehead, p. m., & wright, r. (2017). becoming a college student: an empirical phenomenological analysis of first generation college students. community college journal of research and practice, 41(10), 639–651. https://doi.org/10.1080/10668926.2016.1216474     appendix   college information seeking anxiety how much do you agree with the following statements?  0 = do not agree 5 = neither agree nor disagree 10 = agree completely     0 1 2 3 4 5 6 7 8 9 10 i know who to turn to if i have questions about college                       other students around me know more about college than i do                       people on campus are helpful when i ask them questions                       it is difficult to navigate the system in college                          reverse-code responses to statements 1 and 3   college information sources if you had a question about academics in college, how likely are you to seek help from…     very unlikely (1) unlikely (2) undecided (3) likely (4) very likely (5) parents (1) m    m    m    m    m    friends (2) m    m    m    m    m    other relatives (3) m    m    m    m    m    professors (4) m    m    m    m    m    academic advisor (5) m    m    m    m    m    residence advisor (ra) (6) m    m    m    m    m      library (7) m    m    m    m    m      tutoring center (8) m    m    m    m    m    coworker or supervisor (9) m    m    m    m    m    no one – i would look it up on my own (10) m    m    m    m    m    other (specify) m    m    m    m    m      1 = low information source 5 = high information source     if you had a question about college life, how likely are you to seek help from…     very unlikely (1) unlikely (2) undecided (3) likely (4) very likely (5) parents (1) m    m    m    m    m    friends (2) m    m    m    m    m    other relatives (3) m    m    m    m    m    professors (4) m    m    m    m    m    academic advisor (5) m    m    m    m    m    residence advisor (ra) (6) m    m    m    m    m    library (7) m    m    m    m    m    tutoring center (8) m    m    m    m    m    coworker or supervisor (9) m    m    m    m    m    no one – i would look it up on my own (10) m    m    m    m    m    other (specify) m    m    m    m    m      1 = low information source 5 = high information source         evidence summary   users with disabilities, especially invisible disabilities, provide insight into how libraries can frame accessibility webpages   a review of: brunskill, a. (2020). “without that detail, i’m not coming”: the perspectives of students with disabilities on accessibility information provided on academic library websites. college & research libraries, 81(5), 768–788. https://doi.org/10.5860/crl.81.5.768   reviewed by: scott goldstein coordinator, web services & library technology mcgill university library montréal, québec, canada email: scott.goldstein@mcgill.ca   received: 1 dec. 2020                                                                accepted:  20 jan. 2021      2021 goldstein. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29888     abstract   objective – to understand the needs and preferences of users with disabilities for libraries’ accessibility webpages (webpages dedicated to information on disability and accessibility).   design – semi-structured interviews.   setting – a large public university in the united states of america.   subjects – 12 students who self-identify as having a disability.   methods – participants were asked about their expectations (if any) and experiences using library accessibility webpages, how they felt they should be organized, and where and how they would expect to find such webpages. two lists were printed out and provided to the participants. the first, compiled from a previous study, listed common website headings (categories) under which accessibility webpages had been found, and this aided participants in selecting where they would go to find such a webpage. the second listed common types of information found on accessibility webpages. participants were asked to use the second list to come up with their five highest priority items for accessibility webpages to cover. interviews were recorded, transcribed, and analyzed for responses to specific queries, but inductive coding was also used.   main results – in most of the five response clusters of interest to the author (experience/expectation of such a page existing, navigation and language preferences, overall tone and feel for the website, organization for the page, and content for the page), answers were mixed. no consensus emerged with respect to participants’ expectation of an accessibility webpage existing, how they would find the page (including the best website heading), and what content the page should contain. participants noted that language should be welcoming and inclusive and vetted for sensitivity. the physical layout of the library and information about ambiance and furniture was frequently noted as being an important and overlooked detail to include. some services, such as shelf pulling and online chat, were highlighted as appealing to those with “invisible” disabilities.   conclusion – the needs and preferences of users with disabilities are varied and sometimes mutually conflicting. based on the findings, fourteen recommendations are suggested, including providing detailed information about sensory aspects of the library, listing contact information (preferably to a named individual or group), providing useful headings within the page, and evaluating whether language on the website is welcoming.   commentary   this paper is part of an area of research that looks at accessibility in libraries. unlike some earlier studies that look only at users with mobility and vision issues, this one takes a broader view of the concept of disability to include “invisible disabilities,” defined as anything that interferes with day-to-day functioning without having a physical manifestation. examples include adhd, depression, dyslexia, and autism. this paper examines library webpages that are devoted to providing information to users with disabilities, referred to by the author as “accessibility webpages.” addressing a gap in the literature mentioned by cassner et al. (2011, p. 49), the author interviews users who self-identify as disabled to get their perspective on the organization, content, and look and feel of the accessibility webpage at their university.   this commentary relies on the critical review tool of letts et al. (2007). the purpose of the study was clearly stated and motivated by a gap in the literature, although there did not seem to be an explicit research question. this is not necessarily a problem, but had there been one, it could have added some focus to the findings section. the study design was adequate given the exploratory nature of the study purpose. the sample of participants was small and unrepresentative of users with disabilities within the setting, but the author acknowledges this and does not attempt to make generalized claims. the data collection process was well explained. one puzzling finding was that some participants did not view their disabilities in terms of accessibility, assuming the term pertained specifically to mobility accommodations such as ramps for wheelchair users (p. 774). given the broad range of disabilities, as well as expectations of how libraries should respond to them, perhaps a term such as accommodation or inclusivity, rather than accessibility, would have better conveyed the concept discussed in this article.   it is often said that accessible design translates into better design for all users. a key lesson from this study is that the same thing applies to targeting library services and resources beyond a preconceived audience. for example, displaying information on the noise level and lighting of a space is helpful to all users, even if the original motivation is to accommodate users with anxiety and sensitivity. the author’s recommendation to provide detailed information about sensory aspects of the library, and more generally to avoid assuming who might need what kind of information, is probably the most unheeded one on library websites. this recommendation can be incorporated into a website redesign process or even a content audit of webpages. it would also be a wise idea for libraries to survey and interview users with disabilities in their own setting since they may discover perspectives not discussed in this study.   references   cassner, m., maxey-harris, c., & anaya, t. (2011). differently able: a review of academic library websites for people with disabilities. behavioral & social sciences librarian, 30(1), 33–51. https://doi.org/10.1080/01639269.2011.548722   letts, l., wilkins, s., law, m., stewart, d., bosch, j., & westmorland, m. (2007). critical review form – qualitative studies (version 2.0). peel health library. http://www.peelregion.ca/health/library/eidmtools/qualreview_version2_0.pdf     news   call for expressions of interest to host eblip10    2017. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     the call for expressions of interest to host the next international evidence based library & information practice conference (eblip 10) will be announced by the international advisory committee around mid-september 2017.   follow @eblip9 on twitter or check http://eblip9.org for the call and its supporting documentation.   previous eblip conferences have been hosted in australia, canada, sweden, the uk, and the us. where will it take place next—might you be the host?   evidence summary   researchers may need additional data curation support   a review of: johnston, l. r., carlson, j., hudson-vitale, c., imker, h., kozlowski, w., olendorf, r., & stewart, c. (2018). how important are data curation activities to researchers? gaps and opportunities for academic libraries. journal of librarianship and scholarly communication, 6(1), 1-24. https://doi.org/10.7710/2162-3309.2198     reviewed by: robin e. miller associate professor and research & instruction librarian mcintyre library university of wisconsin-eau claire eau claire, wisconsin, united states of america  email: millerob@uwec.edu   received: 3 dec. 2018                                                                    accepted: 18 feb. 2019      2019 miller. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29539     abstract   objective – to identify the data curation activities most valued by researchers at universities.   design – focus group and survey instrument.   setting – six r1: doctoral universities in the united states of america that are part of a data curation network (dcn) project to design a shared data curation service.   subjects – 91 researchers, librarians, and support staff.   methods – the authors used focus group methodology to collect data about valued data curation activities, current practices, and satisfaction with existing services or activities. six focus groups were conducted at participants’ places of employment. participants reviewed a list of 35 possible data curation activities, including documentation, data visualization, and rights management. a card-swapping exercise enabled subjects to rank the most important issues on a scale of 1-5, with “most important” activities becoming the subject of a facilitated discussion. in a short paper-based survey, participants also noted whether a data curation practice is in place at their institution, and their satisfaction with the practice.   main results – twelve data curation activities were identified as “highly rated” services that academic institutions could focus on providing to researchers. documentation, secure storage, quality assurance, and persistent identifier were the data curation activities that the majority of participants rated as “most important.” participants identified the data curation practices in place at their institutions, including documentation (80%), secure storage (75%), chain of custody (64%), metadata (63%), file inventory or manifest (58%), data visualization (58%), versioning (56%), file format transformations (55%), and quality assurance (52%). participants reported low levels of satisfaction with their institutions’ data curation activities.   conclusion – academic libraries have an opportunity to develop or improve existing data curation services by focusing on the twelve data curation activities that researchers, staff, and librarians value but that could be implemented in a more satisfactory way. the authors conclude that their organization, the data curation network, has an opportunity to improve data curation services or to offer new or expanded services.   commentary   the strength of this research is in the methods employed to gather data from employees of the nine data curation network member institutions. while focus groups can often be conducted with a rigid set of questions, in this study each focus group’s facilitator used rating and card-swapping to direct the inquiry to the primary interests of the participants at each institution. quantitative data collected aided in the interpretation of verbal comments about the challenges and barriers to curating data. card-swapping exercises demonstrated that the participants valued 12 data curation practices in particular. a subsequent questionnaire about data curation efforts actually in practice revealed that most participants were relatively unsatisfied with institutional practices, even among data curation services they valued. for example, “documentation” was rated as “most important” and 80% of participants indicated this is in place at their institutions. however, only 46.2% of participants were “somewhat” satisfied, and 9.9% were not satisfied with documentation of data curation processes. “secure storage” received the highest satisfaction ratings among participants, with 38.3% expressing satisfaction, although 40% of participants responded “n/a.”   the authors identify self-selection as a limitation of their research. in light of the study’s specialized topic, another view is that participant knowledge and experience enhanced the focus group’s outcomes. additional granularity in reporting participant views would improve the results. for example, the authors do not indicate whether the data show differences of opinion between the researchers, librarians, and staff who participated in the study. researchers, particularly those who require data curation services in order to fulfill contractual obligations, may have different expectations for the outcome of data curation activities at their institutions than the support staff or librarians developing data curation services.   most of the study’s participants agreed on a set of highly valued data curation activities, which may form the basis of any academic library’s data curation program. while the authors do not directly suggest that the study’s results are generalizable, the title of the article implies that “academic libraries” have opportunities to invest in, develop, and market data curation services. however, the authors repeatedly use the phrase “research libraries,” implying that the results of this research are more likely directed to practitioners at large research universities, if not exclusively at dcn member institutions.   the article does not indicate that the researchers coded the qualitative data collected during the six focus groups. “case studies” of two of the six focus groups are presented, highlighting problems with the data curation process, like limited time, de-identification of sensitive data, and a desire for standardized data curation practices. the authors also point out that the literature about data curation raises themes similar to those that emerged during facilitated focus group discussions, including limited time and staffing, and pointing to a need for greater support in the form of documentation, templates, and standards. coding the qualitative data collected during focus group discussions would improve the authors’ communication about the prevalence and frequency of the issues raised by participants in this study.   while the results of the study cannot be generalized to all universities or libraries, library practitioners building a data curation service may find that this research serves as a reference point for the data curation services that researchers value or need.     call for applicants for eblip journal: evidence summaries writers news   call for applicants for eblip journal: evidence summaries writers    2022. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30120     eblip seeks to add five writers to the evidence summaries team. evidence summaries are critical appraisal syntheses, which provide analysis regarding the validity and reliability of the methodology used in an original research article. as such, they are a key component of eblip to aid readers in making informed decisions in their local practice. evidence summaries team members are required to write two evidence summaries per year, with a two-year commitment to the journal. evidence summaries cover all areas of library and information studies and we encourage applications from information professionals in areas such as school, public, and special libraries, as well as academic settings. april 30. applicants who are shortlisted will be asked to submit a sample evidence summary. *please note that evidence based library and information practice is a non-profit, open access journal and all positions are voluntary and unpaid. the positions are an excellent opportunity for continuing professional development and gaining experience in reviewing and critically appraising library-related research. **only those applicants who are selected or shortlisted will be contacted by the editors. about the journal:   published quarterly and hosted by the university of alberta, this peer-reviewed, open access journal is targeted at all library and information professionals interested in an evidence based model of practice. by facilitating access to librarianship research via original research articles and evidence summaries of relevant research from the library literature, evidence based library and information practice enables librarians to practice their profession in an evidence based manner. please visit the evidence based library and information practice web site (https://journals.library.ualberta.ca/eblip/index.php/eblip) for further information about the journal. evidence summary   a landscape study highlights the urgent need for evidence based strategies to enable electronic health records integration in the national healthcare systems of lowand middle-income countries   a review of: kumar, m., & mostafa, j. (2020). electronic health records for better health in lowerand middle-income countries: a landscape study. library hi tech, 38(4), 751–767. https://doi.org/10.1108/lht-09-2019-0179   reviewed by: joanne m. muellenbach library director and associate professor california health sciences university clovis, california, united states of america email: jmuellenbach@chsu.edu   received: 28 may 2021                                                                  accepted:  12 july 2021      2021 muellenbach. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29981     abstract   objective – to identify how lowand middle-income countries (lmics) approached the development of national and subnational electronic health records (ehrs) and to understand the challenges related to ehr research priorities and sustainability.    design – landscape study consisting of a review of the scientific literature, country-focused grey literature, and consultation with international experts.     setting – hospitals and healthcare systems within lmics.    subjects – the 402 publications retrieved through a systematic search of four scientific electronic databases along with 49 publications found through a country-focused analysis of grey literature and 14 additional publications found through consultation with two international experts.    methods – on 15 may 2019, the authors comprehensively searched four major scientific databases: global health, pubmed, scopus, and web of science. they also searched the grey literature and repositories in consultation with country-based international digital health experts. the authors subsequently used mendeley reference management software to organize and remove duplicate publications. peer-reviewed publications that focused on developing national ehrs within lmic healthcare systems were included for the title and abstract screening. data analysis was mainly qualitative, and the results were organized to highlight stakeholders, health information architecture (hia), and sustainability.   main results – the results were presented in three subsections. the first two described critical stakeholders for developing national and subnational ehrs and hia, including country ehealth foundations, ehrs, and subsystems. the third section presented and discussed pressing challenges related to ehr sustainability. the findings of the three subsections were further explored through the presentation of three lmic case studies that described stakeholders, hia, and sustainability challenges.   conclusion – the results of this landscape study highlighted the scant evidence available to develop and sustain national and subnational ehrs within lmics. the authors noted that there appears to be a gap in understanding how ehrs impact patient-level and population outcomes within the lmics. the study revealed that ehrs were primarily designed to support monitoring and evaluating health programs focused on a particular disease or group of diseases rather than common health problems. while national governments and international donors focused on the role of ehrs to improve patient care, the authors highlighted the urgent need for further research on the development of ehrs, with a focus on efficiency, evaluation, monitoring, and quality within the national healthcare enterprise.   commentary   electronic health records (ehrs) can revolutionize the healthcare industry by providing the needed health information to make informed decisions and improve patient care. access to ehrs is especially critical in lowand middle-income countries (lmics) that are faced with professional healthcare shortages. a systematic review reported that the main criteria for ehr success includes system functionality, organizational structure and support, and availability of the technical infrastructure (fritz, 2015). this landscape study confirmed the urgent need to build ehr development models to enable the sharing of meaningful data for better health within lmics. this study builds on the authors’ previous study that investigated strategies for ehr integration within lmics (kumar & mostafa, 2019).   based on the joanna briggs institute’s checklist for systematic reviews and research syntheses (2017), the study was concise and systematic. the research questions were clearly stated, the tables provided details of the search strategies, and the inclusion and exclusion criteria were specified. the lead author has in-depth knowledge about ehr systems as evidenced by his senior position with the carolina population center at the university of north carolina at chapel hill and over 15 years of global and country-level experience in public health informatics and health information systems.    the scientific database and grey literature searches were organized and systematic. in their quest for a comprehensive and exhaustive search, the authors noted that they received valuable input regarding key phrases and terms from an ehr expert and a health informatics librarian at their university. the investigators used mendeley, a reference management software tool, to organize and share their publications. however, the study did not provide specific details related to their critical appraisal process and their methods for minimizing bias in the data extraction. while inclusion and exclusion criteria were presented, the reliability of the study would have been more substantial if they had expanded on their description of the screening process.    regarding the study synthesis, the authors presented their findings in three subsections with accompanying narratives, but the process for combining the studies was not specified. in addition, the study would have been more robust if there were further details about how they assessed any possibility of publication bias. as a part of their analysis, the authors applied a draft toolkit for health information system evaluation developed by the university of washington.  they reported that the tool did not provide evidence that pertained to ehr development in lmics. still, additional information about their methods for applying the instrument would have enhanced the validity and applicability of the results.    the findings of this study may have the potential to assist programs in health informatics and library and information studies that include a data sciences, e-science, or informatics component. in addition, librarians may wish to investigate possible collaborations with system leaders to integrate evidence based knowledge resources within ehrs, including those available by subscription, such as bmj best practice and dynamed, or freely available, such as pubmed and medlineplus. libraries may wish to consider developing programs that involve librarians who participate in clinical rounds and provide search support. librarians may also want to consider partnerships with software developers to ensure authority control and consistent use of controlled vocabularies between the library and the ehr systems. this study is an important starting point from which further research will provide more significant insights into the role of ehrs for improving healthcare.   references   fritz, f., tilahun, b., & dugas, m. (2015). success criteria for electronic medical record implementations in low-resource settings: a systematic review. journal of the american medical informatics association, 22(2), 479–488. https://doi.org/10.1093/jamia/ocu038   the joanna briggs institute. (2017). checklist for systematic reviews and research syntheses. https://jbi.global/sites/default/files/2019-05/jbi_critical_appraisal-checklist_for_systematic_reviews2017_0.pdf   kumar, m., & mostafa, j. (2019). research evidence on strategies enabling integration of electronic health records in the health care systems of lowand middle-income countries: a literature review. the international journal of health planning and management, 34(2), e1016–e1025. https://doi.org/10.1002/hpm.2754   news   call for peer reviewers for evidence based library and information practice (eblip)    2020. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29815     the evidence based library and information practice journal would like to announce an opportunity to become a peer reviewer.   the role involves:   providing in-depth peer review of original research articles, review articles, evidence summaries, or classics. the total number of peer review requests will vary depending upon content submitted to the journal, but reviewers, typically, are asked to review 2 to 4 submissions per year in one or more of these categories. contributing to the overall success of the journal by providing unbiased, fair, and timely reviews of submissions that are assigned. communicating with the editorial board about areas where the journal could improve. promoting eblip as an avenue of publication to colleagues.   the ideal candidate is a professional or researcher in the area of librarianship and information practice and well-versed in evidence based practice and research methods. the peer reviewer term is for two years, and is renewable. peer reviewers for the journal follow detailed guidelines supplied by the editorial board and typically have 3-4 weeks to complete a review. the double-blind peer review process is managed through the online journal system for notification, acceptance, and reviewers’ comments.   the deadline for individuals to indicate their interest in response to this call is october 15, 2020. however, new peer reviewers may be invited to join at any time. interested persons should send a statement of interest, indicating areas of methodological and subject expertise, to dr. lorie kloda, editor-in-chief, at eblipjournal@gmail.com   about the journal: published quarterly by the university of alberta, this peer reviewed, open access journal is targeted at all library and information professionals interested in an evidence based model of practice. by facilitating access to library and information studies research via original research articles and evidence summaries of relevant research from the library literature, evidence based library and information practice enables information professionals to practice their profession in an evidence-based manner. please visit the evidence based library and information practice website (http://ejournals.library.ualberta.ca/index.php/eblip) for further information about the journal.       evidence based library and information practice evidence based library and information practice 2009, 4:3  evidence based library and information practice       editorial responsibilities      © 2009. this is an open access article distributed under the terms of the creative commons attribution  license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and  reproduction in any medium, provided the original work is properly cited.    editor‐in‐chief: denise koufogiannakis    associate editor (articles): alison brettle    associate editor (evidence summaries): lorie kloda    associate editor (classics, using evidence in practice): scott walter    associate editor (features, commentaries, ebl101, news): denise koufogiannakis    production editor: katrine mallan    copyeditors: priscilla stephenson (lead copyeditor), richard hayman, lisa shen, dale storie, mary  virginia taylor, ute wilkinson, elizabeth zeeuw    indexing support: pam morgan  3 microsoft word art_ed_2.doc evidence based library and information practice 2006, 1:2    1 evidence based library and information practice       editorial    the current state of ebl      lindsay glynn  editor‐in‐chief    public services librarian and instruction coordinator, health sciences library   memorial university of newfoundland   st. johnʹs, newfoundland, canada  e‐mail: lglynn@mun.ca      © 2006 glynn. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      evidence based librarianship, still in its  infancy, is slowly gaining interest and  momentum.  increased exposure to the  evidence based model of practice is allowing  information specialists to learn more about  what is involved and how its application  can affect practice.  events such as the  international conference on evidence based  library and information practice and  publications such as eblip help to increase  awareness of ebl as well as increase the  body of evidence necessary to fully  implement such a practice model.  having  said that, barriers remain that hinder the  application of evidence based librarianship  in its most basic form.      brice, booth and bexon established six steps  involved in the practice of ebl: specify the  problem, find evidence to answer the  problem, appraise the evidence, apply the  results, evaluate the outcome, and redefine  the problem.  without a large body of  research from which to draw evidence,  information professionals are often forced to  replace the step of finding the evidence with  designing and performing their own  research.  what this means is that rather  than being able to find, appraise, and  implement existing research, practitioners  are finding themselves in a position where  they have to design and carry out original  research in order to obtain applicable  evidence.  aside from being time consuming  and costly, many are not comfortable with  research design or data analysis, and they  may lack administrative support or the  needed resources available to support  research.  until the body of evidence is  substantially increased, it is essential that  original research be encouraged and  published.    i recently posted a message to the eblig  electronic mail list requesting ebl success  stories.  i hoped to hear how people had  used evidence to assist in their decision  making as well as whether the applied  evidence had a positive or negative result  once implemented.  the response was  minimal, save for three people who had  carried out original research and used the  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:2    2 results to modify practice or implement a  new service or policy.  nobody mentioned  using existing evidence, and nobody  mentioned evaluating implementation  results.  on the surface, this supports the  general hypothesis that the body of research  is low and that ebl still has a way to go  before it is practised regularly and  systematically.  but that’s just my opinion;  perhaps there’s a research paper out there  somewhere.    my intent is not to cause information  specialists to feel intimidated or pressured  to research and publish like there’s no  tomorrow.  rather, i am taking this  opportunity to encourage information  professionals to practice ebl and to write  about their experiences, whether the results  are negative or positive.  if existing research  helped you make a decision, write about the  process and tell how well it worked (or  didn’t work).  if there was no existing  research, design your own study and  publish the results.  if you thought outside  the box and looked beyond library literature  to find applicable research and evidence,  share the methodology and rationale with  your peers by submitting it to a journal or  presenting it at a conference.  the more we  share through publications and  presentations, the better equipped we will  be to adopt the evidence based model of  practice.  the application of ebl can not  only save time and money, but can also  position information professionals as the  highly competent researchers and prolific  writers that we are.                        works cited    brice, anne, andrew booth, and nicola  bexon. “evidence based librarianship:  a case study in the social sciences.”  proceedings of the 71st ifla general  congress and council, august 14‐18,  2005, oslo, norway. 19p. ifla. 4 june  2006,  <http://www.ifla.org/iv/ifla71/papers/11 1e‐brice_booth_bexon.pdf>.       http://www.ifla.org/iv/ifla71/papers/11 evidence summary   collaborations between libraries and writing/tutoring services are diverse and provide opportunities to support student success and information literacy outcomes   a review of: jackson, h. a. (2017). collaborating for student success: an e-mail survey of u.s. libraries and writing centers. the journal of academic librarianship, 43(4), 281-296. https://doi.org/10.1016/j.acalib.2017.04.005     reviewed by: brittany richardson web services librarian, assistant professor university of tennessee at chattanooga library chattanooga, tn, united states email: brittany-richardson01@utc.edu   received: 25 may 2018  accepted: 10 aug. 2018      2018 richardson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29452     abstract   objective – to collect information on the existence and characteristics of collaborative partnerships between libraries and writing centers/writing tutoring services.   design – email survey questionnaire.   setting – academic libraries, writing centers, and writing tutoring services at two-year, four-year, and graduate/professional institutions across the united states of america.   subjects – 1,460 librarians, writing center staff, and tutoring services staff.   methods – subjects were invited to participate based on a “. . . random sampling of 33% of each institutional “size and setting” group from the 2010 carnegie classification of institutions of higher education” and the availability of contact information for the library or writing center at the randomly sampled institutions (p. 282). respondents who identified an existing partnership between the library and writing center/tutoring services answered questions regarding collaboration methods, training, and promotion as well as open-ended questions on goals, assessment, ideal relationship qualities, strengths, and weaknesses. in the absence of a known partnership, questions focused on potential for, and ideal methods of, collaboration.   main results – the survey had a response rate of 13.5%, based on the 197 responses that met the criteria for inclusion in the results. of the respondents, 117 identified as librarians, 59 as writing center staff, and 21 as tutoring services staff. respondents were affiliated with institutions in 43 states and the district of columbia. 65% of respondents reported that a collaborative relationship between the writing center and library existed at their institution. of those without a known current partnership, 77% believed there was potential for collaboration. top existing collaborations included instruction (21%), student orientations (16%), appointments (14%), classroom presentations (14%), and writing tutors embedded in the library (14%). only 35% identified strategic goals for collaborations. respondents engaged in partnerships highlighted shared space, referrals, a unified focus on student success, and defined roles as top ideal partnership characteristics. key partnership strengths included teamwork/relationship, focus on student success, and shared goals/knowledge/resources. common weaknesses included lack of communication, planning, shared space, patron awareness, funding, staff, and collaboration.   conclusion – diverse collaborations between libraries and writing centers/writing tutoring services exist. these collaborations may provide opportunities to support student success and information literacy outcomes. based on survey results, the author suggested that improved communication between partners could mitigate identified weaknesses and assist in achieving partnership ideals. additionally, increased creation and assessment of strategic partnership goals may strengthen communication and planning. many respondents were interested in shared library and writing center space, an area which requires further research. ultimately, the author concluded that more investigation is needed to inform best practices for partnerships.   commentary   case studies, primarily at four-year institutions, comprise much of the published literature on library and writing center collaborations (elmborg & hook, 2006; montgomery & bradshaw, 2015). a few studies have undertaken broader analysis. todorinova (2010) conducted a telephone survey of reference librarians at 154 institutions of varying types, finding 26.7% had a partnership with the writing center. torodinova’s survey served as a “starting point” for the study author’s expanded email survey (p. 282). ferer (2012) authored a literature review identifying themes in collaborative efforts, many of which are reflected in the author’s survey results. ferer observed that little has been written from the writing center perspective. the author’s study fills a gap in the literature by providing an updated overview of collaborative trends across various institution types from the perspective of both writing center and library professionals.   when evaluated against glynn’s (2006) ebl critical appraisal checklist criteria, this study excels in several ways: the criteria for population selection was clearly defined; the data collection methods were clearly articulated; the full survey instrument was provided; and the results section effectively summarized potential applications and areas for further research. there were, however, several areas that could be strengthened as well. first, the survey instrument could be validated. it may also be useful to devise an alternative way of connecting with potential respondents: invitations to participate in this study were sent based on availability of contact information for the library and/or writing center, which may have led to less representation of certain groups. it would also be ideal to have a mechanism to determine if libraries and writing centers at the same institution submit potentially overlapping responses. additionally, the language of some survey questions could be revised: while most survey questions were constructed to elicit clear responses, the author acknowledged that several respondents misinterpreted the open-ended question on how they “assess the success of the relationship,” responding with a measure of quality (e.g. “good”) rather than an assessment mechanism (p. 283). finally, further details on the process used to analyze open-ended responses would be helpful for practitioners interested in conducting similar analysis.   this study may be informative for academic libraries and writing centers/writing tutoring services embarking upon or evaluating existing partnerships. the survey results offer an overview of current endeavors, while providing some insight into successful strategies and potential pitfalls. as the study author suggests, additional work is needed to establish best practices for partnerships. this research provides direction for increasingly robust evaluation of specific library and writing center partnership aspects in future studies.   references           elmborg, j. k. & hook, s. (eds.). (2006). centers for learning: writing centers and libraries in collaboration. chicago, il: association of college and research libraries.   ferer, e. (2012). working together: library and writing center collaboration. reference services review 40(4), 543-557. https://doi.org/10.1108/00907321211277350   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154   montgomery, s. e., & robertshaw, s. d. (2015). from co-location to collaboration: working together to improve student learning. behavioral and social sciences librarian, 34(2), 55–69. https://doi.org/10.1080/01639269.2015.1047728   todorinova, l. (2010, april). writing center and library collaboration: a telephone survey of academic libraries. academic services faculty and staff publications. paper 56. retrieved from http://scholarcommons.usf.edu/cgi/viewcontent.cgi?article=1055&context=tlas_pub   evidence based library and information practice evidence based library and information practice 2010, 5.2  evidence based library and information practice       evidence summary    citation analysis shows promise as an effective tool for monograph collection  development    a review of:  enger, k. b. (2009). using citation analysis to develop core book collections in academic libraries.  library & information science research, 31(2), 107‐112.    reviewed by:   scott marsalis  social sciences librarian, university of minnesota libraries  minneapolis, minnesota, united states of america  email: marsa001@umn.edu    received: 23 mar. 2010          accepted: 12 may 2010       2010 marsalis. this is an open access article distributed under the terms of the creative commons‐ attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐ sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original  work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is  redistributed under the same or similar license to this one.    abstract   objective – to test whether acquiring books  written by authors of highly cited journal  articles is an effective method for building a  collection in the social sciences.    design – comparison study.    setting – academic library at a public  university in the us.    subjects –  a total of 1,359 book titles, selected  by traditional means (n=1,267) or based on  citation analysis (n=92).    methods – the researchers identified highly‐ ranked authors, defined as the most frequently  cited authors publishing in journals with an  impact factor greater than one, with no more  than six journals in any category, using 1999  isi data. they included authors in the  categories business, anthropology,  criminology & penology, education &  education research, political science,  psychology, sociology/anthropology, and  general social sciences. the books in print  bibliographic tool was searched to identify  monographs published by these authors, and  any titles not already owned were purchased.  all books in the study were available to  patrons by fall 2005. the researchers collected  circulation data in spring 2007, and used it to  compare titles acquired by this method with  titles selected by traditional means.    main results – overall, books selected by  traditional methods circulated more than  those selected by citation analysis, with  differences significant at the .001 level.  86 mailto:denise.koufogiannakis@ualberta.ca evidence based library and information practice 2010, 5.2  however, at the subject category level, there  was no significant difference at the .05 level.  most books selected by the test method  circulated one to two times.    conclusion – citation analysis can be an  effective method for building a relevant book  collection, and may be especially effective for  identifying works relevant to a discipline  beyond local context.      commentary     although not without limits, this study was  well designed, and adds depth to the literature  in its investigation of an innovative approach  to collection development. using glynn’s  (2006) critical appraisal checklist, the study  scores well for validity. the methodology is  well thought out and is described in detail.  analysis of results is appropriate and  outcomes are clearly discussed and presented.  while the results may not be generalized  without further research, the study does hold  promise for future researchers? there are,  however, some limitations to the study that  should be considered when planning further  research or application of results.    one primary assumption made by the  investigators of the study was that authors of  journal literature in the social sciences are  likely to also be the authors of important book  literature, an assumption that remains  untested. in fact, cronin (1997) discovered that  in sociology literature, there were distinctly  higher sets of frequently cited authors in the  monograph literature versus the journal  literature, lending doubt to the assumption.      while enger examines the theoretical  underpinnings of citation analysis and reviews  the related literature, she fails to adequately  address some of the primary arguments  against citation analysis, or more specifically,  isi’s data and methodologies, especially in the  social science context. noticeably missing in  her literature review is nederhoff’s review of  citation analysis in the humanities and social  sciences. a more thorough discussion of  variables such as author age stratification,  reasons for citation, us‐centricity and other  factors described in the literature (for example,  case & higgins, 2006; leimu & koricheva,  2005; porta, 2006) might have strengthened the  article and assisted readers in using the study  to inform their own decision making.     one problematic aspect of the methodology is  that the books selected by citation analysis  only included titles that had not already been  selected by traditional means. it would have  been interesting to examine the overlap  between the two sets in order to better  understand the strengths and limitations of  each means of selection. the sample of books  selected by citation analysis, perhaps because  of this, is quite small (less than 7% of the total).  it is also unclear what the publication date  range was for each sample. if the books  selected by citation analysis alone were older  (which is likely, considering the citation cycle),  publication date might have been a  confounding variable, since a recent study  found that among books acquired in a specific  year, those with an older publication date  circulated more frequently (adams, 2008).     there is also some confusion regarding the isi  journal citation reports (jcr) categories used,  as the lists differ between the abstract, article  text, table 1 in the article, and my own  consultation of the online isi jcr for 1999.  general social sciences, listed only in the  article text, is not reflected in the current  online product. the article text also lists both  “anthropology” and  “sociology/anthropology”, although the latter  category doesn’t exist in jcr, nor does the  author use it in other places.    citation analysis, while well established, is in  a dynamic period. efforts by administrators to  be more “data driven” strengthen the appeal  of measures including isi’s impact factor, and  isi is increasingly extending the reach of its  scope to quantify the impact of authors and  institutions. scopus and google scholar are  providing alternative sources of data and new  measures, such as the h‐index and eigen  factor, are being developed to address  87 evidence based library and information practice 2010, 5.2  88 criticisms to isi’s products. at the same time,  traditional monograph collection development  procedures can be too time consuming as  blended and liaison librarians struggle to  balance increasing and diversifying  workloads. enger’s study provides an  interesting approach that bears further  exploration.    references   adams, b. (2008). circulation statistics in the  evaluation of collection development.  collection building, 27(2), 71. doi:  10.1108/01604950810870227.  case, d. o., & higgins, g. m. (2000). how can  we investigate citation behavior? a study  of reasons for citing literature in  communication. journal of the american  society for information science, 51(7), 635‐ 645. doi: 10.1002/(sici)1097‐ 4571(2000)51:7<635::aid‐asi6>3.0.co;2‐ h.   cronin, b., snyder, h., & atkins, h. (1997).  comparative citation rankings of authors  in monographic and journal literature: a  study of sociology. journal of  documentation, 53(3), 263‐273. doi:  10.1108/eum0000000007200.  glynn, l. (2006). a critical appraisal tool for  library and information research. library  hi tech, 24(3), 387‐399. doi:  10.1108/07378830610692154.  leimu, r., & koricheva, j. (2005). what  determines the citation frequency of  ecological papers? trends in ecology  evolution, 20(1), 28. doi:  10.1016/j.tree.2004.10.010.   nisonger, t. e. (2004). the benefits and  drawbacks of impact factor for journal  collection management in libraries. the  serials librarian, 47(1‐2), 57. doi:  10.1300/j123v47n01_04.  oromaner, m. (1981). the quality of scientific  scholarship and the ʺgrayingʺ of the  academic profession: a skeptical view.  research in higher education, 15(3), 231.  doi: 10.1007/bf00976418.  porta, m. (2006). commentary: the  ʹbibliographic impact factorʹ and the still  uncharted sociology of epidemiology.  international journal of epidemiology, 35(5),  1130. doi: 10.1093/ije/dyl196.  evidence based library and information practice evidence based library and information practice 2010, 5.1  evidence based library and information practice       commentary     provocative and stimulating – but eblip (and information literacy) are moving targets! 1    andrew booth  reader in evidence based information practice  school of health and related research (scharr)   university of sheffield   sheffield, united kingdom  email: a.booth@sheffield.ac.uk     received: 13 dec. 2009          accepted: 21 dec. 2009        2010 booth. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which  permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly  attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the  same or similar license to this one.      recent years have seen several commentators,  including myself, complain that the eblip  movement is in danger of becoming too  “comfortable” (booth & brice, 2007). indeed it  can seem like a family where you try to be  polite to each other today in the knowledge  that you will still have to get on with each  other on the next day, and the next, ad  infinitum. the international programme  committee for the 5th international eblip  conference in stockholm faithfully discharged  their commission to make the plenary content  more provocative and controversial. focus for  such a session was ola pilerot, a spokesman  for the information literacy movement (2006a,  2006b), asked to bring a welcome outsider’s  view to the conference. certainly, from         comments made in the conference evaluation,  ola seems to have fitted the bill.     starting with a description of the process of  information literacy from the association of  college and research libraries, information  literacy competency standards (2000), no doubt  like many in the audience, i was struck with  similarities with the process of evidence based  practice (table 1).  of course, it is differences between the two  processes that are most illuminating. evidence  based practice includes the closing of the cycle  with “assess”. information literacy includes a  stage of incorporating new (incident)  knowledge with previous (prevalent)   1 response to the keynote address by ola pilerot at the 5th international conference for evidence based library  and information practice, stockholm, sweden, july 2009. all interpretations are the author’s own. ola pilerot  declined the opportunity to write his own commentary but positively welcomed someone else’s commentary on  his presentation.  37 mailto:a.booth@sheffield.ac.uk evidence based library and information practice 2010, 5.1  table 1   similarities and differences between information literacy and evidence based practice (ebp)  information literacy competency  corresponding steps in ebp process  determine the extent of information needed  ask  access the needed information effectively  and efficiently  acquire  evaluate information and its sources  critically  appraise  incorporate selected information into one’s  knowledge base   [assimilate?]  use information effectively to accomplish a  specific purpose  act    assess  understand the economic, legal, and social  issues surrounding the use of information,  and access and use information ethically and  legally    knowledge (mills & gray, 2007) – in  attributing a new “a” to this stage of the  evidence based process we might settle on  “assimilate”. however, the most telling  addition from information literacy is the final  item, “understand the economic, legal, and  social issues surrounding the use of  information, and access and use information  ethically and legally”. this made me realise  that the “instrumentality” of information (i.e.  information as a tool) when used in evidence  based practice, perhaps neglects sufficient  acknowledgement of the economic, legal, and  social context. it is true that koufogiannakis  and crumley (2004) include such  considerations in their examination of  “apply” as embodied in the libraries using  evidence instrument (applicability checklist,  n.d.), but evidence based practice carries the  implicit, almost naïve, assumption that  information, providing that it is good enough,  can simply be re‐used. ola pilerot thus  provokes me to plead for “eblip in context”.    indeed this theme of context is picked up with  pilerot’s (2009) subsequent definition of  information literacy:     to learn information literacy means to  develop a discursive understanding of  the practice in which one is active.  information literacy thus comprises an  understanding of and a familiarity  with how information is sought and  used in a [certain social context]. (p.  65)    furthermore, the problematising of  information behaviour as a rational and  individual behaviour, stimulated by  consideration of context and advocated by  pilerot, is particularly welcome. we have  frequent cause to question the “information  deficit model” (marteau, sowden, &  armstrong, 2002) with its arrogant  assumption that “if you build a digital library  [the users] will come.” indeed much evidence  based practice seems to be based more on the  concept of how little information literacy the  busy practitioner can manage to get away  with, rather than realistic achievement of some  black belt standard of “information mastery”  (ebell & shaughnessy, 2003, p. s.53). why,  even the original proponents of evidence  based practice lowered their aspiration, at  least to “brown belt”, when faced with the  practical realities of following the entire  evidence based practice process.  my own views of evidence based practice  have been shaped, even transformed, by a  recurring theme from pilerot and other  plenary speakers at the conference who see  information behaviour as a collective and  collaborative activity. if other eblip  conferences have focused on the role of the  individual library practitioner then eblip5  38 evidence based library and information practice 2010, 5.1  (stockholm) may well be remembered as the  conference of the eblip team. this  underpinning theme was nicely embodied in  pilerot’s quotation from hakkarainen,  palonen, paavola, and lehtinen (2004):    the idea of producing knowledge in  one place and simply installing it for  use in another may not be valid. it  requires at least a partially or  completely shared frame of reference,  and has to become an object of  collective inquiry. knowledge, as  opposed to information, cannot be  disseminated as such without shared  practices or frames... (p. 73)    pilerot does well to remember, however, that  enhanced appreciation of the complexity of a  particular approach does not per se invalidate  that approach. indeed such complexity can be  viewed as a necessary prerequisite of the  evolution of an approach, paradigm or  movement.    there is a certain irony that, typically, when a  speaker prefaces his remarks with “i am now  going to be controversial,” what follows  usually appears less controversial than was  anticipated. indeed much of what ola pilerot  had to say was both welcome and congruent  with current thinking within eblip. the  corollary, equally ironic, is that unintentional  controversy can be more controversial than  even the presenter intended.  nevertheless,  such controversy is still to be welcomed – no  lesser a proponent of evidence based practice  than muir gray attests to the value of  “irritants” as the grains of sand that may  result in a pearl.    so what were my particular irritants from this  presentation? certainly neither the causes of  information literacy and evidence based  practice were advanced by his espousal of the  narrative review of a decade ago:    …when one is attempting to link  together many studies on different  topics, either for purposes of  reinterpretation or interconnection. as  such, narrative literature reviewing is  a valuable theory building technique,  and it may also serve hypothesis‐ generating functions (baumeister &  leary, 1997, p. 312).    narrative review is a flawed product that  enshrines subjective author opinion, opaque  findings and non‐auditable conclusions as  mulrow (1987), among others, ably  demonstrates. if narrative literature review  was apparent state of the art in 1997, we now  have a toolkit of more sophisticated  methodologies (barnett‐page & thomas, 2009),  such as meta‐ethnography and meta‐theory,  that are equally valuable, and  methodologically more mature, for theory‐ building and hypothesis‐generating. the  difference is that these methodologies   systematically and explicitly capture insights  from multiple studies employing the reviewer  as a filter rather than a curtain.    throwing out the baby…    perhaps my biggest irritant was reserved for  pilerot’s conclusion. his final slide (table 2)  sacrificed conceptual clarity for rhetorical  contrast. in dichotomously ranging eblip and  “research use” as antagonists he favoured  both caricature and anachronism.   checking one of pilerot’s original sources, i  found that mcintyre (2005) states, within a  similar context of debate :    i do want to emphasise, however, that  this is a continuum. in practice, a good  deal of knowledge and thinking will  fall between the points identified on  the continuum. and many other  distinctions might fruitfully be made  for other purposes, distinctions that  would involve highlighting additional  points on the continuum. (p. 3)     39 evidence based library and information practice 2010, 5.1  table 2   a manufactured dichotomy between eblip and research use (pilerot, 2009)  eblip    •evidence  •question –answer  –knowledge transfer  •gap‐metaphor  •epistemologically  –quantitative  –sciences  •instruction  •process/source approach  •rational information use  research use    •learning  •exploration  –shared references  •continuum‐metaphor  •epistemologically  –qualitative  –social sciences  •conversation  •communicative approach   •problematising rational information use    this was exactly the point made in a follow up  question to the plenary presentation. creating  a false dichotomy is destructive and  exaggerates tension and conflict. a more  constructive approach recognises that both  ends of the continuum or spectrum have  something valuable to contribute. certainly  the addition of context, perversely ignored by  pilerot in his use of a quotation in isolation  (incidentally, one danger of narrative  approaches to review!), is something from  which evidence based practice might learn  and benefit. similarly we welcome the  addition of evidence based practice conceived  as a collective endeavour.    the version of evidence based practice  characterised by pilerot is not one that i find  easy to recognise. while ebp 1.0 might have  been criticised for an excessive reliance on the  quantitative paradigm, the movement of  subsequent versions into the social sciences  and education, as heralded at previous eblip  conferences, signals the paradigm’s increasing  maturity (given, 2006). recognition is now  afforded to qualitative research while mixed  methods approaches are frequently  articulated, if less frequently practised. pilerot  targets the “baby” when the movement itself  approaches adolescence (booth & brice, 2007).    conclusion    leaving the plenary hall, i found myself   drawing an analogy with shooting ducks at a  fairground. instead of shooting directly at the  target, a proficient marksman allows for the  distance travelled and thus shoots ahead of the  swift moving duck. evidence based practice in  general, and certainly in its variant eblip  form, has moved on. indeed pilerot himself  has been an actor, unwitting or otherwise, in  this very evolution and development. i  suspect, and i am on less certain ground here,  that information literacy is an equally rapidly  moving field. indeed it is already starting to  embrace some mechanisms and tools of the  evidence based practice movement (partridge  & hallam, 2007; koufogiannakis & wiebe,  2006; brettle, 2007, 2003).    an advert for a uk variant of wood varnish  espouses the virtue “it does exactly what it  says on the tin.” while this presentation on  information literacy aspired less to varnish  and more to paint stripper we can confidently  affirm that in being provocative, controversial,  and (in the precise sense used above)  “irritating” pilerot toiled industriously to  justify his label.    implications for practice  • evidence based practitioners may find  it helpful to make explicit connections  between the process of evidence based  practice and competencies required  for information literacy.  40 evidence based library and information practice 2010, 5.1  • consideration of context has the  potential to enhance interpretation  and application of evidence within a  library setting.  • research use, and by implication  evidence based practice, is a collective  endeavour. in reality these labels  represent designated points on a  wider continuum or spectrum.     implications for research  • information literacy may benefit from  use of evidence based approaches and  tools in addressing longstanding  research questions.  • exploration of information literacy  usefully employs a mixture of  quantitative and qualitative  approaches.   • information literacy and evidence  based practice are rapidly moving  fields and necessitate clear recognition  of the journey already travelled and  the challenges that lie ahead.      references  association of college and research libraries.  (2000). information literacy competency  standards for higher education. chicago:  american library association.  retrieved 20 feb. 2010 from  http://www.ala.org/ala/mgrps/divs/ac rl/standards/standards.pdf  barnett‐page, e., & thomas, j. (2009). methods  for the synthesis of qualitative  research: a critical review. bmc  medical research methodology, 9, 59.   baumeister, r.f., & leary, m.r. (1997). writing  narrative literature reviews. review of  general psychology, 1, 311‐320.  booth, a., & brice, a. (2007). prediction is  difficult, especially the future: a  progress report. evidence based library  and information practice, 2(1), 89‐106.  retrieved 20 feb. 2010 from  http://ejournals.library.ualberta.ca/ind ex.php/eblip/article/view/99/242   brettle, a. (2003). information skills training: a  systematic review of the literature.  health information & libraries journal,  20(suppl. 1), 3‐9.  brettle, a. (2007). evaluating information skills  training in health libraries: a  systematic review. health information  & libraries journal, 24(suppl 1), 18‐37.  ebell, m.h., & shaughnessy, a. (2003).  information mastery: integrating  continuing medical education with the  information needs of clinicians. journal  of continuing education in the health  professions, 23(suppl. 1), s53‐62.   given, l. (2006). qualitative research in  evidence‐based practice: a valuable  partnership. library hi tech., 24(3),  376‐386.   guyatt, g.h., meade, m.o., jaeschke, r.z.,  cook, d.j., & haynes, r.b. (2000).  practitioners of evidence based care.  not all clinicians need to appraise  evidence from scratch but all need  some skills. bmj, 320(7240), 954‐955.   hakkarainen, k., palonen, t., paavola, s., &  lehtinen, e. (2004). communities of  networked expertise: professional and  educational perspectives. amsterdam:  elsevier.    koufogiannakis, d., & crumley, e. (2004).  applying evidence to your everyday  practice. in a. booth & a. brice (eds.),  evidence‐based practice for information  professionals: a handbook (pp. 119‐126).  london: facet publishing.  koufogiannakis, d., & wiebe, n. (2006).  effective methods for teaching  information literacy skills to  undergraduate students: a systematic  review and meta‐analysis. evidence  based library and information practice,  41 http://www.ala.org/ala/mgrps/divs/acrl/standards/standards.pdf http://www.ala.org/ala/mgrps/divs/acrl/standards/standards.pdf http://ejournals.library.ualberta.ca/index.php/eblip/article/view/99/242 http://ejournals.library.ualberta.ca/index.php/eblip/article/view/99/242 evidence based library and information practice 2010, 5.1  42 1(3). retrieved 20 feb. 2010 from  http://ejournals.library.ualberta.ca/ind ex.php/eblip/issue/view/9   libraries using evidence. (n.d.).  applicability  checklist retrieved 20 feb. 2010 from  http://www.newcastle.edu.au/service/l ibrary/gosford/ebl/toolkit/docs/applic ability%20checklist.pdf  marteau, t.m., sowden, a.j., & armstrong, d.  (2002). implementing research  findings into practice: beyond the  information deficit model. in a.  haines, &  a. donald (eds.). getting  research findings into practice (pp. 68‐ 76). london: bmj books.  mcintyre, d. (2005). bridging the gap between  research and practice. cambridge  journal of education, 35(3), 357‐382.  mills, i., & gray, j.a.m. (2007). better allocation  for better healthcare: the annual review of  evidence, process, outcome and  configuration (epoc) handbook:  how to  use the resources offered in national  knowledge weeks to improve value, safety  and quality. united kingdom: nhs  national knowledge service,  commissioning directorate.  department of health.   mulrow, c. d. (1987). the medical review  article: state of the science. annals of  internal medicine, 106(3), 485‐488.   partridge, h. l., & hallam, g. c. (2007).  evidence‐based practice and  information literacy. in s. lipu, k.  williamson, & a. lloyd (eds.).  exploring methods in information literacy  research: topics in australasian library  and information studies (p. 28). wagga  wagga, new south wales: centre for  information studies, charles sturt  university.  pilerot, o. (2006a). how do students develop  information literacy: through formal  education or social participation?  librariansʹ information literacy  annual conference (lilac), leeds,  united kingdom.   pilerot, o. (2006b). information literacy: an  overview. in a. martin, & d. madigan  (eds.). digital literacies for learning (pp.  80‐88). london: facet publishing.    pilerot, o. (1 july 2009). conditions for research  use in library and information practice:  a  matter of learning. fifth international  evidence based library and  information practice conference,  stockholm, sweden. retrieved 20 feb.  2010 from  http://blogs.kib.ki.se/eblip5/presentati ons/opilerotpresentation.pdf   http://ejournals.library.ualberta.ca/index.php/eblip/issue/view/9 http://ejournals.library.ualberta.ca/index.php/eblip/issue/view/9 http://www.newcastle.edu.au/service/library/gosford/ebl/toolkit/docs/applicability%20checklist.pdf http://www.newcastle.edu.au/service/library/gosford/ebl/toolkit/docs/applicability%20checklist.pdf http://www.newcastle.edu.au/service/library/gosford/ebl/toolkit/docs/applicability%20checklist.pdf http://blogs.kib.ki.se/eblip5/presentations/opilerotpresentation.pdf http://blogs.kib.ki.se/eblip5/presentations/opilerotpresentation.pdf evidence based library and information practice evidence based library and information practice 2010, 5.2  evidence based library and information practice       news/announcements    dates and venue announced for 6th international evidence based library and  information conference       2010. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which  permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly  attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the  same or similar license to this one.    the 6th international evidence based library  and information practice conference (eblip6)  will take place june 27‐30, 2011, at the  university of salford, uk.     o are you a library or information  practitioner?  o interested in discovering research for  professional decision making?  o do you want to know more about  using evidence based approaches in  your own practice?  o are you being encouraged to evaluate  or demonstrate the value and impact  of your service?  o have you got some research or good  practice to share?     then the 6th international evidence based  library and information practice (eblip6)  conference is for you!     through workshops, presentations and  discussion, eblip6 will provide a practical  and accessible forum for librarians and  information practitioners from all sectors to  discover and disseminate evidence that may  contribute to decision making in professional  practice.     previous eblip conferences have been held in  sheffield, canada, australia, usa and  sweden.  returning to the uk, the university  of salford is proud to host the sixth evidence  based library and information practice  conference.      the university is based is based one mile  outside the city of manchester and has a rich  history stretching back to 1896.  as one of the  worldʹs first industrial cities salford has many  historic achievements including the worldʹs  first free public library.     add the dates to your diary and book mark  the web site to find out more.     key dates:  o call for abstracts: september 2010  o deadline for abstracts: december  2010  o notification of acceptance: january  2011  o registration opens: march 2011        for further information visit the web site at  http://www.eblip6.salford.ac.uk/  email: eblip6@salford.ac.uk  114 http://www.eblip6.salford.ac.uk/ mailto:eblip6@salford.ac.uk call for applicants for eblip journal: editorial intern news   call for applicants for eblip journal: editorial intern    2022. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30224     evidence based library and information practice (eblip), a peer reviewed open access journal published since 2006, seeks to fill the position of editorial intern. the ideal candidate will be in place by november 2022 and will commit to serving a two-year term.    the role of the intern is to:    provide a final editorial check of proofed copy before publication, using a predeveloped checklist. check and edit the item metadata on the eblip website to ensure title, author, abstract, and references correspond to the submitted manuscript. assist the editor-in-chief and communications officer with calls for papers or calls for volunteers.  assist with investigation and implementation of new projects related to the journal. participate in quarterly editorial board meetings over zoom.    the ideal candidate will be an mlis (or equivalent) student or a recent graduate (past 2 years) interested in evidence based practice and research methods, and who possesses strong attention to detail. the position requires dedicated time on a regular basis, and it is therefore essential that interested persons ensure available time to devote to this position prior to applying. it is estimated that the workload is approximately 5-7 hours per month.    interested persons should send a cover letter, indicating areas of strength they would bring to the role, and resume/cv as a single pdf file to ann medaille, editor-in-chief, at amedaille@unr.edu by october 10, 2022. specific queries about the role should be addressed to rachel hinrichs, production editor, at rhinrich@iupui.edu.   **please note that evidence based library and information practice is a non-profit, open access journal and all positions are voluntary and unpaid. about the journal:            published quarterly and hosted by the university of alberta, this peer-reviewed, open access journal is targeted at all library and information professionals interested in an evidence based model of practice. by facilitating access to librarianship research via original research articles and evidence summaries of relevant research from the library literature, evidence based library and information practice enables librarians to practice their profession in an evidence based manner. please visit the evidence based library and information practice web site (https://journals.library.ualberta.ca/eblip/index.php/eblip)   for further information about the journal.   assessing the impact of an information literacy course on students' academic achievement: a mixed-methods study research article   assessing the impact of an information literacy course on students' academic achievement: a mixed-methods study   wilma l. jones professor and reference/instruction librarian college of staten island city university of new york staten island, new york, united states of america email: wilma.jones@csi.cuny.edu   tara mastrorilli director of institutional research college of staten island city university of new york staten island, new york, united states of america email: tara.mastrorilli@csi.cuny.edu   received: 14 dec. 2021                                                              accepted: 14 mar. 2022      2022 jones and mastrorilli. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30090     abstract   objective – the aim of this study is to demonstrate the impact of a stand-alone, credit-bearing information literacy course on retention and gpa for students at an open access urban college.   methods – researchers conducted a mixed-methods study with a two-part focus. the first examined the impact of a credit-bearing course using propensity score matching (psm) techniques to compare academic outcomes for students who participated in the course versus outcomes for similar students who did not enroll in the course. multiple regression analyses were conducted to measure impact on gpa and performance in 100-level introductory english general education courses. logistic regression analysis was used to determine persistence one year after enrolling in the course. the second part utilized a questionnaire to survey students of this targeted group to determine impact of the course on their information-seeking behaviour in subsequent academic courses and for non-academic purposes.   results – the quantitative analyses showed: (a) a higher gpa, though slight, for students who have taken the course over the matched comparison group; (b) an increase in persistence for students who have taken the course over the matched comparison group after one year of taking the course; but (c) lower performance in 100-level introductory english courses by students who have taken the course in contrast to the matched comparison group. qualitative data provided through the questionnaire revealed positive and substantive reflective statements that support learning outcomes of the course.   conclusion – the findings in this study underscored the importance of a stand-alone, credit-bearing information literacy course for undergraduate students, particularly for first-generation students attending an open access urban institution. the findings also demonstrate the academic library’s contribution to institutional retention efforts in support of students’ academic success.     introduction   for the past two decades, academic librarians have had to ensure that students, most of whom are digital natives, acquire information literacy (il) skills to be able to find and critically sift through the appropriate materials for their research. moreover, the tenets of il enable college students to distinguish between scholarly, non-scholarly, or misinformation resources (e.g., fake news, alternate facts). academic librarians in the u.s. and around the world have continued to revamp and develop innovative methods for delivering il sessions using current and emerging technologies (burkhardt, 2016; diep & nahl, 2011; inuwa & abrizah, 2018; mackey & jacobson, 2010; mcdevitt, 2011; polger & sheidlower, 2017; swanson & jagman, 2015). instruction librarians at the college of staten island (csi), an open access urban public institution, have also experienced similar challenges in teaching students how to access and retrieve the most appropriate information for their assignments and research papers, and assessing how students have applied the knowledge gained from library instruction sessions. the first-year students entering our open access institution are primarily first-generation college students, many of whom need to learn basic library skills to find research resources and how to critically evaluate sources to use in their term papers or assignments. we, librarians, continue to contemplate, grapple with, and develop effective methods of instruction to engage our unique population of students in providing them with the means to conduct academic-level research. these endeavors by academic libraries are rarely counted as contributions towards the collective institutional retention efforts unless quantitative evidence is provided.   among the new modes of instruction that csi librarians have explored and implemented are: one-shot sessions tailored to faculty requests, 50-minute workshops for co-curricular credits specifically designed for first-year students, online tutorials with interactive videos and mini quizzes attached, and course-embedded library instruction. however, none of these new modes of instruction have been more satisfying to students than the one-credit il course, lib102: beyond google: research for college success, which was developed in 2014. it became apparent to us that long-term instruction, rather than short-term instruction, provided our students with multiple opportunities to become familiar, in depth, with library resources and il concepts. these multiple interactions enabled students to see how we, as librarians, could really be more of a resource to them. assessment of this course, after one year of instruction, was done using a 5-item, post-evaluation questionnaire in every section taught, and it revealed only positive comments from students. moreover, feedback from students who self-identified as having 60+ credits had two common comments in the open-ended section: (i) they wished they had taken this course when they were first-year students, and (ii) they suggested that this course be made mandatory for all first-year students. the above feedback and many other insightful comments were found to be a rewarding source that validated the library’s worth in terms of contributing positively to students’ experiences at csi.   these preliminary observations were the impetus for this study, and the authors of this study set out to obtain quantitative and qualitative evidence that objectively supported the anecdotal feedback. the authors wanted to find out whether there was a quantifiable association in persistence of students who had enrolled in the stand-alone il course. a secondary focus sought to examine the information-seeking behaviour of these students to see whether knowledge gained from lib102: beyond google was ever utilized in other classes or for non-academic purposes. while the library offers many opportunities for student success, attempts to measure its impact on student learning, one that ultimately contributes to institutional goals, are lacking in the library literature as noted by oakleaf (2015), oliveira (2018), and pierard and graves (2007). as such, the result of this study adds to the small but growing literature of the academic library’s impact on student academic success and institutional retention efforts.   background about the lib102 course   the creation of a one-credit course, lib102: beyond google: research for college success (which from here on will be referred to as lib102), emerged as an opportunity to develop a course to address critical thinking and il skills for first-year students. the course aimed to introduce students to the concept of strategic exploration of a research topic, in particular learning how to match information needs to appropriate sources. course goals were to teach students search techniques to efficiently retrieve reliable sources. the course discussed research-based vs. fake news, citation styles, and how to avoid plagiarism. in its design, lib102 provided valuable research techniques to enable students to move beyond the incessant use of google to complete their assignments and to become efficient in utilizing a variety of library research resources as required by their instructors (e.g., peer-reviewed journals, reputable newspapers, as well as some verifiable digital sources). course topics included distinguishing types of publications within the online discovery tool, basic and advanced searching skills (i.e., boolean operators, truncation, wildcards), as well as effective internet search strategies. embedded in the course design were online interactive tutorials, group work activities, discussion forums, a preand post-test, and weekly self-assessment quizzes to evaluate learning throughout the course.    learning outcomes of the course enabled students to have hands-on experiences in a computer lab while learning about identifying, retrieving, and evaluating the appropriate research materials, as prescribed at the time by the information literacy competency standards for higher education (association of college and research libraries [acrl], 2000). these were fundamental skills that could be applied to any research assignment in other classes. however, the framework for information literacy for higher education (acrl, 2016) replaced the standards, which was timely for us as we were revising the final design of the course for adoption into the college catalog. the learning outcomes for lib102 were slightly modified to embrace the threshold concepts of the new standards, the framework, and included weekly assessment of the continuous process of research. in addition to identifying, retrieving, and evaluating materials, learning outcomes enabled students to develop and refine an academic research topic or question, understand the concept of authority, and be able to think critically about information and communication media. lib102 was designed as a seven-and-a-half week course, and fittingly, the end of the first seven-and-a-half week course coincided with mid-term assessments, and the second session coincided with final examinations.   this design, we envisioned, would complement the gateway writing courses of eng111 and eng151 to enable first-year students who were enrolled in them to prepare for first outlines/drafts of research papers due at midterms or to complete their research papers due at finals. eng111 and eng151 are required courses in the general education curriculum and are named introduction to college writing and college writing, respectively. in the undergraduate catalog, eng151 includes in its description “attention to reading, library skills, and research methods.” lib102 strives to reinforce all three items, including swapping out “attention to reading” for “critical thinking.”    institutional setting   the csi library is situated in an urban comprehensive college, one of 25 colleges that make up the open access city university of new york. at present, csi has an enrollment of close to 13,000 students primarily consisting of first-time, first-generation college students. located on staten island, the college carries the distinctive honor of being the only public institution among four other institutions of higher learning in the borough. seventy percent of current students at csi are residents of staten island (csi, office of institutional research, 2020), which has a population of 474,893 (u.s. census bureau, 2019). since the establishment of a residential hall in 2014, the diversity of the student body has broadened. in fall 2020, data from the office of institutional research presented the profile of the students as: 0.1% american indian or alaskan native, 11% asian, 13.5% black or african american, 25% hispanic, 45.3% white, and 5% other. in 2020, first-time, first-generation students were 47% of the student body. also, in 2019, the college became a hispanic-serving institution (hsi) when self-reported data from hispanic students reached a minimum of 25%, full-time equivalent.      consistent with the mission of the college, the csi library strives to play its role in assisting students, who are primarily first-generation students, in becoming information competent, critical thinkers, and life-long learners through traditional and virtual il sessions.   literature review   over the past 20 years, there has been a pointed focus on the academic library’s impact on students’ academic success and retention as a result of accrediting bodies of higher education institutions seeking such evidence, i.e., evidence demonstrating the academic library’s contribution towards institutional priorities of learning outcomes (britto & kinsley, 2018; oakleaf, 2010). il instruction that is embedded in other courses have garnered much success, yet only few studies in the library literature have captured empirical evidence of its contribution towards student’s academic performance and success (black & murphy, 2017; hauck, 2017; inuwa & abrizah, 2018; luetkenhaus et al., 2015; tumbleson et al., 2019). there are several publications describing the positive relationship between students’ use of library resources and services and their impact on higher gpa and retention (bowles-terry, 2012; cook, 2014; haddow 2013; pierard & graves, 2007; rysavy et al., 2017; soria et al., 2013; soria et al., 2014); however, the following are the very few that discuss the impact of a credit-bearing course on students’ academic achievement and retention.    since our study proposed the utilization of propensity score matching (psm) techniques to compare outcomes of students who had taken an il course versus those who had not, we searched for studies that had used psm. we found two studies, one by soria et al. (2017) and another by chiteng kot and jones (2015), that had utilized psm to examine outcomes of library resources and services as it related to student achievement and retention. soria et al. (2017) explored “the impact of library resources on college students’ longitudinal outcomes as related to their graduation or continued enrollment after four years of study” (p. 813). they used quasi-experimental procedures (psm techniques) to construct control (non-library users) and treatment (library users) groups, they found that “using the library at least one time in the first year of enrollment significantly increased the odds that students would graduate in four years or remain enrolled after four years as opposed to withdrawing from the university'' (p. 819). chiteng kot and jones (2015) also used psm to conduct a study on three cohorts of first-time, full-time undergraduate students at a large, metropolitan, public research university that provided evidence that the library had an impact on academic performance. their study revealed that first-term gpa, with regard to students’ use of one of three library resources (workstations, study rooms, and research clinics), was associated with a higher first-term gpa, though small.   only a handful of studies have reported on the contributions of a stand-alone il course and its impact on students’ academic performance that we found relevant to this study. the study conducted by daugherty and russo (2011) found that “students who took a library course understood the usefulness of what they had learned as evidenced by their continued use of both the skills and resources taught in the course” (p. 321). heady et al. (2018) conducted a 20-year longitudinal study examining records of students who had taken an il course to determine the impact of the course over time. results of their study were mixed as first-time, full-time students “who did not take the library course had higher graduation rates” (p. 646), versus slightly higher gpa—“roughly 0.1 point”—for those who took the course (p. 647). the study by mayer and bowles-terry (2013) explored how students performed in an upper-level il course and found them to be more engaged when there was a class theme. the study by lwoga (2014), though focused on the impact of an il course, was centered around undergraduate students in a health sciences program. lwoga reported that not only did students continue “to use online searching techniques for their academic assignments eleven months after training” (p. 10), they utilized these search skills in their personal lives, and “the il module plays an important role in supporting lifelong learning” (p. 11).   our study is unique in the following two ways: 1) it compares retention and gpa outcomes of students who have taken a library course with those who have not, utilizing propensity score matching techniques; and 2) it explores how students have applied the skills learned in the information literacy course to other research activities, academic and non-academic, via a qualitative survey.   aim   this study sought to obtain quantitative and qualitative evidence to demonstrate the impact of the lib102 course on undergraduate student performance throughout their time in college. in particular, the authors examined how this course contributes to student academic success or retention, in support of institutional goals. the following questions guided this study:   ·        how does the cumulative grade point average (gpa) of students who took lib102 compare to those in a matched comparison group? ·        how do students who took lib102 perform in mandatory 100-level english general education courses (i.e., eng111 and eng151) compared to those in a matched comparison group? ·        how do one-year persistence rates of students who took lib102 compare to students in a matched comparison group? ·        how have students who took lib102 applied the knowledge gained in subsequent academic courses or outside the classroom, and how does this compare to a matched comparison group?   methods   to determine the impact of the lib102 course, it was important to compare outcomes for students who participated in the course to outcomes for students who did not enroll in the course. therefore, a match comparison group was devised using propensity score matching (psm). initially introduced by rosenbaum and rubin (1983), psm is a statistical method used to match treated and control subjects using a set of baseline characteristics. for our study, our treated subjects were students who enrolled in lib102. the control subjects were those who did not and will be referred to as the “matched comparison group” from here on. baseline characteristics used for matching included: gender, race/ethnicity, tuition assistance program (tap) or pell grant status, degree level, class level, age, and college admissions average (caa), i.e. cuny’s equivalence for high school average.   the psm routine was run separately for 10 cohorts of students, from fall 2014 to spring 2019. descriptive statistics were calculated, including frequencies, means, and standard deviations on the demographic, enrollment, and survey data. multiple regression analyses were conducted to measure the impact of the lib102 course on student outcomes, specifically cumulative grade point average (gpa) and performance in introductory 100-level english gateway courses, eng111 and eng151. logistic regression measured the impact of the course on one-year persistence. to reduce threats to the study’s internal validity, potential confounding factors or covariates that have a relationship with the program or outcome are included in each regression model. covariates included gender, race/ethnicity, tap or pell grant status, age, degree sought, class status, caa, and cumulative gpa. continuous covariates were grade mean centered.   a questionnaire assessed learning outcomes from lib102 as well as knowledge applied from lib102 to subsequent academic courses or outside the classroom. the survey instrument was adapted from two similar studies on information seeking behaviours of undergraduate students (daugherty & russo, 2011; lwoga, 2014). it consisted of 23 questions, including four open-ended questions to allow for qualitative reflective statements. the first section asked participants to provide basic demographic information such as class level, major, and other background information (nine questions). the second section consisted of six questions that examined how students applied il skills in classes other than lib102. the third section included five questions that sought to find out whether participants used il skills outside of the academic environment, with questions that spoke to applying lifelong learning skills. the last section included three general questions: two open-ended questions about the course, and the third, which was optional, asked participants to provide contact information to enter a drawing for one of eight $50 amazon.com gift cards. the questionnaire is included in appendix a.   data collection and instrumentation   in compliance with csi’s institutional review board (irb) protocol, and in collaboration with the office of institutional research, the researchers accessed transcripts of students who had participated in lib102 and an equivalent number who had not participated. transcript information included student demographic and academic performance data as well as student email addresses. information on students’ prior achievement, as measured by their caa was also retrieved. these data were used as a resource for the psm criteria as well as the outcome data around performance. the academic performance data used for outcome analyses included students’ cumulative gpa at the end of the semester in which the student took lib102 and 100-level english grades (if taken during or after lib102). retention and graduation data were retrieved one year after participating in the lib102 course. the retention and graduation data were combined to calculate a persistence rate. the survey was administered using surveymonkey, an online tool, to survey students enrolled in fall 2019. the survey was emailed to those who had participated in lib102 and those targeted in the matched comparison group. interested participants were then directed to a web link where they anonymously filled out the survey. it remained open for six weeks and reminders were sent out four times after the initial deployment in an effort to receive a 15% confidence level of response.   demographics of students who have taken lib102: beyond google   across 10 semesters, fall 2014 through spring 2019, lib102 had enrolled a total of 675 students. more than half (59%) of the students were female, 56% were students of color, and the average age of participating students was 21. although the course was designed for first-year students, 31% of those enrolled were juniors or seniors. with a cap of 20 students per section, on average, 14.4 students persisted in each section. at the time of enrollment into the study, 57% of the participating students were pursuing a bachelor’s degree and the remaining students were pursuing an associate’s degree (43%). the average caa score for this group of 675 students was 79.0.   multiple regression analyses were conducted using spss to measure the impact of the lib102 course on student outcomes, specifically cumulative gpa and performance in eng111 and eng151. table 1 presents the outcomes of students who took lib102. of the 675 students who took the course, 147 of them who were enrolled in eng111 (introduction to college writing) at the same time or after taking lib102 were found to have a mean gpa of 2.63 (1.30 sd).    table 1 lib102: beyond google course student outcomes   n value mean eng111 gpa (sd) 147 2.63 (1.30) mean eng151 gpa (sd) 162 2.84 (1.22) overall mean gpa (sd) 675 2.63 (0.83) one-year retention (%) 675 53.9   a slightly higher mean gpa of 2.84 (1.22 sd) was found in 162 students who were taking eng151 (college writing) while being enrolled or after enrolling in lib102. the overall mean gpa of all 675 students who took lib102 was 2.63 (0.83 sd). logistic regression was used to measure the impact of lib102 on the retention of students one year after enrolling in the course. slightly more than half (54%) of students who took lib102 were retained or graduated after one year of taking the course. the most recent csi first-time, full-time freshmen retention rate is 59.5%.   the matched comparison sample   students in the lib102 course from fall 2014 semester to spring 2019 were matched to the students who did not participate in the course using psm. separate psms were conducted for each lib102 cohort. table 2 follows with the demographic profile of the analysis sample of 458 students in each sample group used for this study. the difference between lib102 enrollment and the lib102 analysis sample is the result of missing data in the matching criteria used as well as the attempt to produce highly accurate matches in the psm model. to determine if the students in the match comparison groups are similar to students in the lib102 group, statistical significance testing was conducted by cohort of the analysis sample.   table 2 demographic profile of overall analysis sample for the study by group   lib102 analysis sample (n=458) matched comparison group (n=458) female (%) 62.0 62.7 american indian or alaska native (%) 0.0 0.0 asian or pacific islander (%) 13.8 13.8 black (%) 23.4 17.2 hispanic (%) 19.4 19.4 white (%) 43.4 49.1 received tap or pell (%) 58.3 59.0 associate (%) 44.1 37.8 bachelor (%) 55.9 58.5 freshmen (%) 48.7 30.8 sophomore (%) 24.2 31.0 junior (%) 8.3 22.5 senior (%) 18.8 15.7 mean age 20.8 24.9 mean college admissions    average 78.7 77.7   table 3 presents the outcomes of the overall analysis sample by group. of the 458 students in the analysis sample, only 105 of them were enrolled in eng111 during or after taking lib102 and they were found to have a mean gpa of 2.62. similarly, 67 students in the matched comparison group who had enrolled in eng111 were found to have a mean gpa of 2.57.   table 3 student outcomes of overall analysis sample by group   lib102 analysis sample matched comparison group   n value n value mean eng111 gpa (sd) 105 2.62 (1.26) 67 2.57 (1.38) mean eng151 gpa (sd) 124 2.85 (1.23) 89 2.99 (0.96) mean gpa (sd) 458 2.59 (0.83) 458 2.63 (0.90) one-year persistence (%) 458 59.2 458 53.1   of the 458 students in the analysis sample, there were only 124 students who had taken eng151 during or after enrolling in lib102 and they had a mean gpa of 2.85. likewise, 89 students in the matched comparison group were found to have a mean gpa score of 2.99. overall, the lib102 analysis sample had a mean gpa score of 2.59 versus 2.63 for the matched comparison group. the one-year persistence rate for the lib102 analysis sample was 59.2% versus the 53.1% for the matched comparison group.   although the psm resulted in highly similar matched comparison groups, there was no way to identify whether students in the matched comparison groups received instruction similar to that provided in the lib102 course. as a result, there may be students in the matched comparison groups that received similar instruction (e.g., one-shot library instruction sessions in a 100-level psychology course, or similar introductory instruction when students attended another college). we are cautious that this could have potentially influenced the outcome results.   demographics of survey participants   the survey was sent to 212 students who were in the lib102 analysis sample and 143 students in the match comparison group. table 4 displays the results of students who participated in the survey. of the lib102 students who received the survey, 20.2% responded, and 12.6% of the match comparison group students responded. given these low response rates, we were cautious when analyzing and interpreting the results.   the majority of respondents in the lib102 group were juniors and seniors, 60.5% of them having earned 61–124+ credits, and the remainder was almost an even split with 20.9% having 60 or less credits, and 18.6% were graduate students. likewise, the majority of respondents in the matched comparison group were seniors and graduate students, 61.1% of them having earned 91–124+ credits, while 27.8% had 61–90 credits, and 11.2% had 60 credits or less. of the respondents in the lib102 group, 97.6% were non-transfer students vs. 44.4% in the matched comparison group. of the lib102 group, one-quarter of the respondents were male (23.8%), unlike the matched comparison group where one-third were male (33.3%). with regard to ethnicity, the two groups were very different; only 14.3% of the lib102 group were white, whereas more than half (55%) of the respondents in the matched comparison group were white. of note, 50% of the respondents in the lib102 group checked the box for being the “first in their family to attend college” versus 33.3% of the matched comparison group. in both groups, social sciences majors held the highest percentage—65.2% in the lib102 group and 39.0% in the matched comparison group. lastly, 38.9% of the lib102 students were also part of a learning community program (i.e., accelerated studies in associate programs (asap), black male initiative (bmi), and the perry ellis sutton seek program (seek)), whereas only 5.6% of the matched comparison group reported that they had been part of a learning community.   table 4 profile of respondents who participated in the survey   lib102 students n=42   matched comparison group n=18 female 32 (76.2%) 12 (66.7%)       american indian or alaska native 0 0 asian or asian american 9 (21.4%) 1 (5.6%) black or african american 9 (21.4%) 3 (16.7%) hispanic or latino 13 (31.0%) 1 (5.6%) white or caucasian 6 (14.3%) 10 (55.6%) two or more races 3 (7.1%) 0 prefer not to disclose 2 (4.8%) 3 (16.7%)       0–30 credits 1 (2.3%) 1 (5.6%) 31–60 credits 8 (18.6%) 1 (5.6%) 61–90 credits 14 (32.6%) 5 (27.8%) 91–124+ credits 12 (27.9%) 9 (50.0%) graduate student 8 (18.6%) 2 (11.1%)       non-transfer student 41 (97.6%) 8 (44.4%)       major:         humanities 10 (21.8%) 6 (33.3%)     science & technology 4 (8.7%) 5 (27.8%)     social sciences 30 (65.2%) 7 (38.9%)     undeclared 2 (4.3%) 0       first generation to attend college 21 (50.0%) 6 (33.3%)       students in a learning community 16 (38.9%) 1 (5.6%)   results   in this section, researchers first report on the findings from the multiple regression and logistic regression analyses to show evidence of the impact of lib102 on students’ academic achievements. following this, researchers report on the survey results from the same targeted group of students who were currently enrolled at the time of this study.   impact findings using psm   multiple regression analyses were conducted using spss to measure the impact of the lib102 course on students’ cumulative grade point average (gpa) and performance in 100-level english courses, after controlling for student demographics (i.e., gender, race/ethnicity, tap or pell status, age, degree sought, and class level) and previous achievement (i.e., college admissions average). this was done in pursuit of guiding question #1: how does the cumulative gpa of students who took lib102 compare to those in a matched comparison group? table 5 displays the results of the multiple regression analysis on 916 students’ cumulative gpa. lib102 students’ cumulative gpa represents their gpa at the end of the semester in which they took the lib102 course.   table 5 cumulative grade point average multiple regression results (n=916)   b b se constant 2.67** 0.07 female 0.25** 0.05 blacka -0.19* 0.07 hispanica -0.21* 0.07 received tap or pell 0.09 0.05 associateb -0.38** 0.07 freshmanc -0.15 0.08 sophomorec -0.28 0.07 age 0.02** 0.01 college admissions average 0.02** 0.00 group 0.08 0.06 r2 0.21   f 23.6**   a the comparison group included white or asian. b the comparison group included bachelor or other degree-seeking students. c the comparison group included juniors or seniors. * p < .05, ** p < .001.                the gpa for the matched comparison group represents their gpa in the same semester of their matched lib102 student. the results indicate that, after controlling for differences in demographics and previous achievement, the students who took the lib102 course had a 0.08 higher cumulative gpa than the matched comparison group. this difference, however, was not statistically significant. factors that did have a significant relationship with cumulative gpa include: gender, race/ethnicity, degree level, age, and caa.   table 6 displays the results of the multiple regression analysis on 172 students’ performance in their eng111 course and 213 students’ performance in their eng151 course. we conducted this multiple regression analysis in pursuit of guiding question #2: how do students who took lib102 perform in mandatory 100-level english general education courses (i.e., eng111 and eng151) as compared to those in a matched comparison group? after controlling for differences in demographics and previous achievement, the results indicate that the students who took the lib102 course had a 0.22 lower grade in eng111 than the matched comparison group and a 0.21 lower grade in eng151 than the matched comparison group. these differences, however, were not statistically significant. factors that did have a significant relationship with eng111 performance include: gender, race/ethnicity, and degree level. the caa is the only factor that had a significant relationship with eng151 performance.   table 6 multiple regression results for eng111 and eng151   eng111 (n=172) eng151 (n=213)   b b se b b se constant 3.29* 1.30 2.65** 0.40 female 0.45* 0.20 0.25 0.16 blacka -0.06 0.23 -0.20 0.19 hispanica -.071* 0.25 -0.29 0.20 received tap or pell 0.07 0.20 -0.11 0.16 associateb -0.71* 0.29 -0.16 0.19 freshmanc -0.08 1.31 0.51 0.43 sophomorec 0.05 1.38 0.77 0.45 age -0.01 0.02 0.00 0.02 college admissions average 0.01 0.01 0.02* 0.00 group -0.22 0.26 -0.21 0.21 r2 0.13   0.09   f 2.46*   1.87   a the comparison group included white or asian. b the comparison group included bachelor or other degree-seeking students. c the comparison group included juniors or seniors. * p < .05, ** p < .001.   table 7 displays the results of the logistic regression analysis on 916 students’ performance in their one-year persistence after enrollment in the lib102 course semester, i.e., 458 from the lib102 group and 458 from the matched comparison group. we conducted this logistic regression analysis in pursuit of guiding question #3: how do one-year persistence rates of students who took lib102 compare to students in a matched comparison group? the results indicate, after controlling for differences in demographics and previous achievement, that the students who took the lib102 course were retained at higher rates (0.23) than the matched comparison students.   when comparing students who took lib102 to the matched comparison group, lib102 students were 1.26 times more likely to return to csi or graduate the next year. the difference, however, was not statistically significant. factors that did have a significant relationship on retention include: gender, tap or pell status, class level, and caa.   table 7 one-year persistence logistic regression results (n=916)   b b se odds ratio constant 0.11 0.18 1.12 female 0.29* 0.14 1.34 blacka -0.19 0.18 0.82 hispanica -0.17 0.18 0.84 received tap or pell 0.44* 0.14 1.55 associateb -0.23 0.18 0.79 freshmanc -0.42* 0.22 0.66 sophomorec -0.13 0.19 0.88 age -0.01 0.02 0.99 college admissions average 0.02* 0.01 1.03 group 0.23 0.15 1.26 χ2 50.4**     a the comparison group included white or asian. b the comparison group included bachelor or other degree-seeking students. c the comparison group included juniors or seniors. * p < .05, ** p < .001.   survey findings   survey results provided rich data that support knowledge gained in lib102 and the application of that knowledge in other academic courses or outside the classroom.   application of il skills and techniques in courses other than lib102   to explore how students have applied lib102 skills in other academic courses, the survey included questions that prompted the following results (table 8). six databases rose to the top as the most commonly used for academic purposes: onesearch (21.2%), academic search complete (14.4%), the new york times (13.6%), jstor (10.6%), the wall street journal (10.6%), and gale virtual reference library (9.1%). of the matched comparison group, the most commonly used databases were the same, except in different order and with a lower percentage of utilization: onesearch (14.6%), the new york times (14.6%), the wall street journal (12.5%), gale virtual reference library (10.4%), jstor (10.4%), and academic search complete (8.3%). it was rewarding to see that two of three databases regularly taught in the course, onesearch and academic search complete, were heavily used by the lib102 group to do research in other academic classes. given that over 78% of respondents in the control and treatment group were juniors, seniors, and graduate students, we presumed that the high use of the new york times and the wall street journal was due to the fact that they would have come across these resources in other one-shot instructional sessions or encountered them by virtue of residing in the state of new york, where both newspapers are published and widely available. both newspapers are also freely available via an app or online to students, staff, and faculty, a welcome arrangement secured by our parent university.   table 8 use of databases for academic purposes   lib102 group matched comparison group     most used   onesearch – 21.2% academic search complete – 14.4% the new york times – 13.6% jstor – 10.6% the wall street journal – 10.6% gale virtual reference library – 9.1% onesearch – 14.6% the new york times – 14.6% the wall street journal – 12.5% gale virtual reference library – 10.4% jstor – 10.4% academic search complete – 8.3%     least used   psycinfo – 6.8% ebook central – 3.8% medline – 2.3% kanopy – 1.5% nexis uni – 1.5% cinahl – 0.8% opposing viewpoints – 0.8% ebook central – 6.3% kanopy – 6.3% cinahl – 4.2% medline – 4.2% psycinfo – 4.2% nexis uni – 2.1% opposing viewpoints – 2.1%   of the 32 responses given by the lib102 group as to whether they encountered any difficulty using databases, 75% indicated no problems or difficulty. difficulties that were expressed were user–interface related, e.g., “some websites not user-friendly,” and “problems specifying or narrowing search results.” however, of the 12 responses given by the matched comparison group, seven (58%) were trouble-free interactions and the remaining five responses were mainly about technical difficulties including “unable to log in,” “problems logging on from off campus,” and “pdf not available.”   newspaper articles, peer-reviewed articles from journals, and research reports topped the list for both groups at 20%, respectively, when asked about types of materials they were seeking. magazine articles and statistical information ranked higher for the matched comparison group at 14% and 10%, respectively, whereas the lib102 group sought these information sources at 10% and 9%, respectively. of note, there was one recorded response for a podcast search by a respondent in the lib102 group.   with regard to skills used when searching library databases, keyword searching was the most popular among both groups (21.2% in the lib102 group vs. 26.5% in the matched comparison group), followed by subject searching (19.2% vs. 22.4%; see appendix b). regarding advanced searching, 16.4% of the lib102 group indicated that they used it, as did 10.2% of the matched comparison group respondents. though keyword searching remained the number one skill used by both groups, it was heartening to see the high use of the advanced searching option by the lib102 group, which would indicate a good understanding of using it to yield the best results. similar proportions (between 8% and 12%) indicated having used google search operators, phrase searching, and use of synonyms in both groups. nine respondents in the lib102 group selected boolean operators and browsing the index (corresponding to 6% of responses), contrary to our expectations of more use of these skills. not surprisingly, only 2 respondents (4%) in the matched comparison group indicated that they had used boolean operators and browsing the index. remarkably, in the lib102 group, there were two responses for truncation or wildcard, whereas there were no selections for these by the matched comparison group.   to assess whether students had any difficulty using search terms when utilizing databases for their assignments, respondents were given the opportunity to indicate this in an open-ended question. the majority of responses in both groups signaled that they had no difficulty (73.3% of the lib102 group and 63.6% of the matched group). sample statements of difficulty expressed by the lib102 group include: “getting the right word can be hard to find the perfect source. it’s time consuming,” “i wasn’t sure when to use the key term,” and “using the right keywords to get the most accurate results.” similarly, sample statements about difficulty using databases from the matched comparison group were mainly expressed as follows: “could not access kanopy – a requirement for my french class,” “google gives more exact matches better,” “not finding what i’m looking for,” and “not knowing about other techniques.”   application of il skills and techniques for non-academic purposes   this section of the survey sought to determine use of il skills outside the academic environment, for personal use. with regard to the lib102 group, the most common databases used were: google (30%), onesearch (13.3%), the new york times (11.1%), and wikipedia (10%; see table 9). the top four databases selected for non-academic purposes by the matched comparison group were google (41.7%), wikipedia (20.8%), the new york times (8.3%), and medline (8.3%). even though the use of google still reigned supreme for non-academic purposes, it was rewarding to see that the respondents in the lib102 group had also used library subscription resources for non-academic purposes. this signaled that the lib102 course, which was designed to enable students to move beyond the incessant use of google as a research tool for their assignments was playing a large role in their information seeking behaviour for non-academic purposes.   table 9 use of databases for non-academic purposes   lib102 group matched comparison group   most used   google – 30% onesearch – 13.3% the new york times – 11.1% wikipedia – 10% google – 41.7% wikipedia – 20.8% the new york times – 8.3% medline – 8.3%     least used   academic search complete – 6.7% the wall street journal – 6.7% jstor – 4.4% ebook central – 2.2% gale virtual reference library – 2.2% psycinfo – 2.2% opposing viewpoints – 1.1% cinahl – 4.2% jstor – 4.2% kanopy – 4.2% onesearch – 4.2%     when asked in an open-ended question about difficulty using any of the databases for non-academic purposes, 86.2% of respondents in the lib102 group reported no difficulty, and likewise 90.9% of respondents in the matched comparison group reported the same. appendix c shows results for use of search skills when seeking information for non-academic purposes, where keyword searching was the most popular among both groups (21.6% in the lib102 group vs. 27.5% in the matched comparison group). participants were also asked to identify types of information they would search for non-academic purposes. both groups identified social/ entertainment as the top subject information sought, with 21.7% for the lib102 group and 18.9% for the matched comparison group (see appendix d). thereafter, the groups differed slightly in subsequent subjects most searched. for the lib102 group, medical information was second (13.2%), political information was third (12.4%), and nutrition/diet information was fourth (10.9%). for the matched comparison group, it was technological (15.1%), environmental (13.2%), and political and nutrition/diet information were a tie for fourth (11.3%). when asked if they were ultimately successful in finding the non-academic information needed, a yes/no question, 93% of the respondents in both groups indicated success.   benefits of taking the course and additional feedback   the last two questions were open-ended, specifically targeted to the lib102 group to provide evaluative statements about the course. in the first question, respondents were asked to share any benefits of having taken the course. out of 43 respondents, 36 provided complimentary responses of how the course had helped them. some of the comments were as brief as: “it helped me write better papers”; “being able to use the databases efficiently”; “i learned how to find research material for my other classes quickly and easily”; “yes, it has helped me be able to identify credible online resources to use for my research papers”; “it has helped [me] develop skills to do research for classes and how to find valid sources and spot bias”; and “yes it showed me different ways to research and how to identify legitimate sources of information.”   other, lengthier, comments were:   i learned how to access research materials, as well as the skills necessary for writing citations, annotations, etc. i honestly feel this class should be required core curriculum along with eng111 & 151.   i took the class during my freshman year. i found it helpful because i was new to the college where expectations of writings were on a standard, and i did not know that. taking that class helped me shape my foundation for the expectations of college writing.   it made me ahead of all other students in terms of knowing how to do research papers. many students don’t know that they have access to the library from home.   yes i have taken lib102, it helped me with developing new skills when searching for topics from articles on the web by using techniques like typing in keywords and synonyms to narrow down my search.   yeah! i took lib102 with [name redacted] which was great. she was wonderful and while a lot of the stuff was intuitive, a lot of my friends are actually very confused by it. it was helpful being walked through it.   the last open-ended question asked respondents in the lib102 group to share any additional comments or feedback. out of 30 responses, 26 noted “none” or “nothing,” “n/a,” or “no.” the remaining four were enlightening enough to document here:   g[r]e[a]t class i think it should be a gen ed requirement for all incoming freshmen.   i think this class should be mandatory for all incoming freshman [sic].   csi library should send out emails about any updates or specific changes to the website. i also think that the library should send out brief emails with information or instructions on how to do things on the library’s website just so that taking a class or making a trip to see the librarian isn’t the only way of learning how to use the csi library website.   first, i want to thank you for conducting this study and consider the interests of the csi students. second, i feel library resources and search skills learned are not heavily promoted among students because my friends and i sometime[s] would have questions or face problems regarding looking up an article and did not know where to go. again, thank you for your time.   discussion   results of the multiple regression and logistical analyses on outcomes of students who took lib102 show evidence that the course has had a slight impact on their persistence and gpa. data from the regression analysis suggest that students who have enrolled in lib102 gain a slightly higher cumulative gpa than those who have never taken the course. though the magnitude of effects was small, this study supports results of other studies that have shown that multiple sessions of library instruction increased confidence of first year students in conducting research and, in turn, have slightly impacted their overall gpa (bowles-terry, 2012; daugherty & russo, 2011, heady et. al, 2018; soria et al., 2017). the results of the logistic regression analysis on students’ performance in their persistence one year after enrollment in the lib102 course semester reveal that students were 1.26 times more likely to return to or graduate than the matched comparison students. while the authors are cautioned by the fact that the difference was not considered to be statistically significant, this study does corroborate studies by catalano and phillips (2016), cook (2014), haddow (2013), and murray et al. (2016), who also found that an il credit-bearing course does have an impact on students’ academic performance, and in particular pertaining to persistence.   like lwoga (2014) and stonebraker and fundator (2016), we discovered from the survey that students who have enrolled in lib102 continue to utilize online searching skills gained in the course both in other academic classes and for non-academic purposes. the lib102 group differed from the matched comparison group in higher use of academic databases frequently taught in the course, having less difficulty using databases than the matched group, and having less difficulty using search operators, including the use of wildcards and truncations. these findings validate that taking the il course may assist students in being better at navigating familiar or new information resources. moreover, in the answers to the open-ended questions, the lib102 group confirm findings in studies by daugherty and russo (2011), mery et al. (2012) and soria et al. (2017) that showed that an il course has a lasting impact on students as they progress through college. in comment after comment, each student indicated how lib102 had served them well in becoming better at conducting research and writing papers. their brief but concise testimonies spoke volumes about the goals of the course. several comments seemed to be directed at librarians requesting that the course be made mandatory for freshmen or first-year students.   similar to results in the studies conducted by heady et al. (2018) and squibb and mikkelsen (2016), a surprising outcome in this study was the results of the psm analysis, which revealed a lower performance of the lib102 group in the mandatory 100-level english courses than the matched comparison group. the lower performance of students in these courses, even after taking lib102, may be due to several external factors beyond our control. among them could be the fact that the lib102 course was taken at the same time as eng111 and eng151, and that students were not able to make the association while taking the course. a second possible factor could be that the intention of lib102, which is to teach students how to conduct research, was not in parity with the designs of eng111 (introduction to college writing) or eng151 (college writing), which are designed to teach writing as their titles indicate. the lower performance by the lib102 group was surprising given their cumulative gpa was 0.08 points higher than the matched comparison group. the rationale in the study by heady et al. (2018) was that most students “pass the course with a b or above, which would modestly contribute to their gpa” (p. 647). the authors of this current study agree that this could be a possible explanation, as the average grade in lib102 over 10 semesters has been a solid b. either way, this is a revelation that should be further examined and discussed by those teaching in the lib102 program.   a disappointing outcome was in the low utilization of boolean operators reported by the lib102 group. the use of boolean operators is a skill that is always taught in the course, in a dedicated class session. librarians emphasize that these same search skills can also be applied when searching in google to retrieve the most relevant sources. however, two recent students by lowe et al. (2018) and lowe et al. (2020) confirm that teaching boolean searching to first-year students is a waste of class time, and that it is time well spent for upper-level students who find it useful for their discipline-specific databases. nevertheless, this raises a second question regarding assessment of the ways search skills and techniques are being taught, and whether some of these skills and techniques are still relevant.   limitations   there are a couple of limitations to note in this study. first, as with any study involving regression analysis, the findings should be interpreted with caution. in our study, while the results show that the matched comparison group performed better in eng111 and eng151 than students who had taken the lib102 course, the analysis reflects their course performance at the same time as or after taking lib102. it is unknown whether the negative coefficient is greatest when taking eng111 or eng151 concurrently with lib102 or after taking lib102. also, although the analyses controlled for student demographics and previous achievement, threats to internal validity still exist. students in the matched comparison group may have received instruction in il in other courses that are not controlled for in this study.      second, while the authors utilized several interventions prescribed by dillman et al. (2014) to obtain a high response rate and quality data when using web-based surveys, the 30% response rate that is widely considered a robust response in the social sciences was not achieved for this survey. however, the thoughtful and lengthy responses to the open-ended questions were rich contributions to the study.   implications for future study   a strength of this study, at this time of writing, is that this research is among a few studies to utilize the psm method to discover the impact of a stand-alone il credit-bearing course with regard to student academic achievements. as with any study, the use of these results should inform an examination of the subject or inform changes to be implemented. if this study were to be repeated, it is advisable to conduct the multiple regression analyses separately for those students who took 100-level english courses at the same time as lib102 from those who took the english course after lib102. this might ensure an unambiguous result. the negative coefficient results from the multiple regression analyses suggest eng111 and eng151 may not have been the appropriate courses to which lib102 was linked. it may be worth an in-depth examination of whether the intentions of lib102 are in parity with the introductory 100-level english courses. perhaps lib102 could be better paired with other courses in which students conduct research.   furthermore, given the results of the logistic regression analysis regarding the one-year persistence of students who have taken lib102, it would be worth conducting the analysis specifically on first-year students as compared with non-first-year students. the course was actually designed for first-year students, and we are remiss not to have included this analysis at the outset.   conclusion   the findings in this study underscore the importance of a stand-alone, credit-bearing il course for undergraduate students who are primarily first generation in an open access urban institution. while results of the multiple regression and logistical analyses on outcomes of students who took lib102 show evidence that the course has had a slight impact on their persistence and gpa, our survey findings yield valuable insights and validate why taking this course may well assist students in becoming confident at searching and retrieving information resources for their academic assignments. this is illustrated by the lib102 group attaining a slightly higher gpa than the matched comparison group, as well as a persistence rate that showed that students were 1.26 times more likely to return one year after taking lib102. our findings also show that the knowledge gained in lib102 is applied in students’ personal lives for non-academic purposes, thus recognizing information literacy as a lifelong learning skill. moreover, the survey revealed numerous positive testimonies of how helpful this course has been to students for academic and non-academic purposes, further validating the value of the course. in their own words, several students emphatically prescribed that the course should be made mandatory for freshmen or included in the general education curriculum. this, the researchers ascertained, comes from students feeling empowered by the skills they learned in lib102.   the researchers are confident that the quantitative and qualitative evidence presented here not only support the anecdotal feedback that served as an impetus for this study, but also demonstrate the library’s commitment to supporting institutional outcomes that pertain to students’ development of learning. indeed, this is a testament to one of the goals in the csi department of the library’s (n.d.) mission statement, which states that we “cultivate scholarly inquiry and discovery through teaching information literacy and critical thinking skills.” moreover, this study makes a substantial contribution to the gap in the library literature. in oakleaf’s (2010) comprehensive report for acrl, she urged academic librarians to be more systematic in collecting data that connect and demonstrate the library’s value in institutional terms. this study answers that call, and the authors of this paper aim to conduct follow-up studies.   author contributions   wilma l. jones: conceptualization (lead), methodology, formal analysis, writing – original draft (lead), writing – review & editing (equal) tara mastrorilli: conceptualization (supporting), methodology (lead), formal analysis (lead), writing – review & editing (equal)   references   association of college and research libraries. (2000). information literacy competency standards for higher education. http://hdl.handle.net/11213/7668   association of college and research libraries. (2016). framework for information literacy for higher education. https://www.ala.org/acrl/standards/ilframework   black, e. l., & murphy, s. a. (2017). the out loud assignment: articulating library contributions to first-year student success. the journal of academic librarianship, 43(5), 409–416. https://doi.org/10.1016/j.acalib.2017.06.008   bowles-terry, m. (2012). library instruction and academic success: a mixed-methods assessment of a library instruction program. evidence based library & information practice, 7(1), 82–95. https://doi.org/10.18438/b8ps4d   britto, m., & kinsley, k. (2018). academic libraries and the academy: strategies and approaches to demonstrate your value, impact, and return on investment. association of college and research libraries.   burkhardt, j. m. (2016). teaching information literacy reframed: 50+ framework-based exercises for creating information-literate learners. ala neal-schuman.   catalano, a., & phillips, s. r. (2016). information literacy and retention: a case study of the value of the library. evidence based library & information practice, 11(4), 2–13. https://doi.org/10.18438/b82k7w   chiteng kot, f., & jones, j. l. (2015). the impact of library resource utilization on undergraduate students' academic performance: a propensity score matching design. college & research libraries, 76(5), 566–586. https://doi.org/10.5860/crl.76.5.566   college of staten island, office of institutional research. (2020). semester enrollment: student demographic profile, fall 2019. the city university of new york. https://applications.csi.cuny.edu/institutional_profile/semesterenroll_profile.html   cook, j. m. (2014). a library credit course and student success rates: a longitudinal study. college & research libraries, 75(3), 272–283. https://doi.org/10.5860/crl12-424   daugherty, a. l., & russo, m. f. (2011). an assessment of the lasting effects of a stand-alone information literacy course: the students' perspective. the journal of academic librarianship, 37(4), 319–326. https://doi.org/10.1016/j.acalib.2011.04.006   department of the library. (n.d.). the city university of new york, college of staten island. https://www.csi.cuny.edu/academics-and-research/department-library   diep, k. c., & nahl, d. (2011). information literacy instruction in four vietnamese university libraries. international information & library review, 43(4), 198–206. https://doi.org/10.1016/j.iilr.2011.10.002   dillman, d. a., smyth, j. d., & christian, l. m. (2014). internet, phone, mail, and mixed-mode surveys: the tailored design method. john wiley & sons.   haddow, g. (2013). academic library use and student retention: a quantitative analysis. library & information science research, 35(2), 127–136. https://doi.org/10.1016/j.lisr.2012.12.002   hauck, j. (2017). from service to synergy: embedding librarians in a digital humanities project. college & undergraduate libraries, 24(2–4), 434–451. https://doi.org/10.1080/10691316.2017.1341357   heady, c., morrison, m. m., & vossler, j. (2018). ecological study of graduation rates and gpa in a library credit course. the journal of academic librarianship, 44(5), 642–649. https://doi.org/10.1016/j.acalib.2018.07.010   inuwa, s., & abrizah, a. (2018). embedded librarianship in research in nigerian universities: practices and sources of practice knowledge. the journal of academic librarianship, 44(6), 738–746. https://doi.org/10.1016/j.acalib.2018.10.002   luetkenhaus, h., borrelli, s., & johnson, c. (2015). first year course programmatic assessment: final essay information literacy analysis. reference & user services quarterly, 55(1), 49–60. https://doi.org/10.5860/rusq.55n1.49    lowe, m. s., maxson, b. k., stone, s. m., miller, w., snajdr, e., & hanna, k. (2018). the boolean is dead, long live the boolean! natural language versus boolean searching in introductory undergraduate instruction. college & research libraries, 79(4), 517–534. https://doi.org/10.5860/crl.79.4.517   lowe, m. s., stone, s. m., maxson, b. k., snajdr, e., & miller, w. (2020). boolean redux: performance of advanced versus simple boolean searches and implications for upper-level instruction. the journal of academic librarianship, 46(6), 102234. https://doi.org/10.1016/j.acalib.2020.102234   lwoga, e. t. (2014). mapping information literacy outcomes and learning experiences of health sciences undergraduate students. partnership: the canadian journal of library and information practice and research, 9(1). https://doi.org/10.21083/partnership.v9i1.2695   mackey, t. p., & jacobson, t. e. (2010). collaborative information literacy assessments: strategies for evaluating teaching and learning. neal-schuman.   mayer, j., & bowles-terry, m. (2013). engagement and assessment in a credit-bearing information literacy course. reference services review, 41(1), 62–79. https://doi.org/10.1108/00907321311300884   mcdevitt, t. r. (2011). let the games begin! engaging students with field-tested interactive information literacy instruction. neal-schuman.   mery, y., newby, j., & peng, k. (2012). why one-shot information literacy sessions are not the future of instruction: a case for online credit courses. college & research libraries, 73(4), 366–377. https://doi.org/10.5860/crl-271   murray, a., ireland, a., & hackathorn, j. (2016). the value of academic libraries: library services as a predictor of student retention. college & research libraries, 77(5), 631–642. https://doi.org/10.5860/crl.77.5.631   oakleaf, m. (2010). the value of academic libraries: a comprehensive research review and report. association of college and research libraries. http://www.ala.org/acrl/sites/ala.org.acrl/files/content/issues/value/val_report.pdf   oakleaf, m. (2015). the library’s contribution to student learning: inspirations and aspirations. college & research libraries, 76(3), 353–358. https://doi.org/10.5860/crl.76.3.353   oliveira, s. m. (2018). retention matters: academic libraries leading the way. new review of academic librarianship, 24(1), 35–47. https://doi.org/10.1080/13614533.2017.1365003   pierard, c., & graves, k. (2007). research on student retention and implications for library involvement. in l. hardesty (ed.), the role of the library in the first college year (pp. 155–168). university of south carolina, national resource center for the first-year experience and students in transition.   polger, m. a., & sheidlower, s. (2017). engaging diverse learners: teaching strategies for academic librarians. libraries unlimited.   rosenbaum, p. r., & rubin, d. b. (1983). the central role of the propensity score in observational studies for causal effects. biometrika, 70(1), 41–55. https://doi.org/10.2307/2335942   rysavy, m. d. t., michalak, r., & wessel, a. (2017). 8 years of institutional assessment feedback: students’ satisfaction with library services. reference services review, 45(4), 544–561. https://doi.org/10.1108/rsr-03-2017-0005   soria, k. m., fransen, j., & nackerud, s. (2013). library use and undergraduate student outcomes: new evidence for students' retention and academic success. portal: libraries and the academy, 13(2), 147–164. https://doi.org/10.1353/pla.2013.0010   soria, k. m., fransen, j., & nackerud, s. (2014). stacks, serials, search engines, and students' success: first-year undergraduate students' library use, academic achievement, and retention. the journal of academic librarianship, 40(1), 84–91. https://doi.org/10.1016/j.acalib.2013.12.002   soria, k. m., fransen, j., & nackerud, s. (2017). the impact of academic library resources on undergraduates' degree completion. college & research libraries, 78(6), 812–823. https://doi.org/10.5860/crl.78.6.812   squibb, s. d., & mikkelsen, s. (2016). assessing the value of course-embedded information literacy on student learning and achievement. college & research libraries, 77(2), 164–183. https://doi.org/10.5860/crl.77.2.164   stonebraker, i. r., & fundator, r. (2016). use it or lose it? a longitudinal performance assessment of undergraduate business students' information literacy. the journal of academic librarianship, 42(4), 438–444. https://doi.org/10.1016/j.acalib.2016.04.004   swanson, t. a., & jagman, h. (2015). not just where to click: teaching students how to think about information. association of college and research libraries.   tumbleson, b., burke, j., & long, j. (2019). assessment, analytics, and analysis: demonstrating the impact of lms embedded librarians on student learning. journal of library & information services in distance learning, 13(1–2), 196–214. https://doi.org/10.1080/1533290x.2018.1499252   u.s. census bureau. (2019). staten island borough, richmond county, new york. https://data.census.gov/cedsci/profile?g=0600000us3608570915   appendix a questionnaire   impact of an information literacy course   i. demographics 1.      please identify your current academic status. how many credits will you have earned by the end of this semester? a.            ___ 0 – 30 credits b.            ___ 31 – 60 credits c.            ___ 61 – 90 credits d.           ___ 91 – 124+ credits e.            ___ graduate student f.             ___ other     2.      gender: a.            ___ female b.            ___ male c.            ___ prefer not to disclose d.           ___ other     3.      ethnicity (check all that apply): a.            ___ american indian or alaska native b.            ___ asian or asian american c.            ___ black or african american d.           ___ latino or hispanic e.            ___ native hawaiian or other pacific islander f.             ___ white or caucasian g.            ___ prefer not to disclose h.            ___ other __________   4.      what is your current major/intended major? a.            (drop down menu that includes all majors, including “undeclared” major)   5.      are you a transfer student to csi?                               ___ yes ___ no   6.      are you (or your siblings) the first in your family to attend college?                               ___ yes ___ no   7.      are you part of any one of the learning community programs below: a.            ___ asap – accelerated studies in associated program b.            ___ bmi program – black male initiative program c.            ___ c-step – college science & technology entry program d.           ___ macaulay honors college e.            ___ seek program – search for education, elevation, & knowledge program f.             ___ teacher education honors academy g.            ___ verrazano school h.            ___ other _____________________________ i.             ___ no, i was never enrolled in a learning community   8.      have you taken the course “beyond google: research for college success” labeled as lib102 or lib501?                               ___ yes ___ no    (if the answer is no, software skips to #10)   9.      what type of lib102/lib 501 did you take? a.            ___ hybrid (where you had 3 or 4 classes held online) b.            ___ in classroom (where you had face-to-face interactions for all 7 class meetings)   ii. applications of il skills and information resources in classes other than beyond google: lib102 / lib 501   10.   please identify the type(s) of class project(s) you have taken at csi (select all that apply): a.            3-5 page research paper/project b.            6-10 page research paper/project c.            10-20 page research paper/project d.           technical report for science lab course e.            undergraduate seminar project f.             master’s thesis g.            capstone research project h.            other __________________   11.   please indicate which of any of the library databases you have used for your class projects (select all that apply): a.            none b.            onesearch c.            academic search complete d.           cinahl complete e.            ebook central f.             gale virtual reference library g.            jstor h.            kanopy i.             lexis-nexis (now nexis uni) j.             medline k.            the new york times l.             opposing viewpoints m.          psycinfo n.            the wall street journal o.            other(s) ______________   12.   please explain any difficulty you encountered when using any of the identified databases above. ____________________________________________________________ ____________________________________________________________ ____________________________________________________________   13.   what types of information were you searching for while using the above identified databases? (select all that apply:) a.            newspaper articles b.            magazine articles c.            peer-reviewed articles d.           research reports e.            government documents f.             statistical information g.            blogs or websites h.            podcasts i.             films or video-clips j.             other _____________   14.   please indicate which searching skills or techniques you used (select all that apply): a.            keyword searching b.            subject searching c.            boolean operators (and, not, or) d.           advanced searching e.            use of synonyms f.             google search operators (site, link, source, filetype, etc.) g.            browsing the index h.            phrase searching i.             truncation or wildcard (?, *) j.             physical navigation of the stacks k.            other _____________ l.             none of the above   15.   please explain any difficulty you encountered when using those skills or techniques: ____________________________________________________________ ____________________________________________________________ ____________________________________________________________   iii. applications of il skills and information resources for non-academic purposes   16.   please indicate which information resources you have used for non-academic purposes. examples of non-academic purposes: finding information on future employer/corporation, researching a product you wish to purchase (i.e., car, washer/dryer, etc.), or locating medical information on treatment of a disease for a family member. (select all that apply:) a.            none b.            onesearch c.            academic search complete d.           cinahl complete e.            ebook central f.             gale virtual reference library g.            google h.            jstor i.             kanopy j.             lexis-nexis (now nexis uni) k.            medline l.             the new york times m.        opposing viewpoints n.            psycinfo o.            the wall street journal p.            wikipedia q.            other ___________   17.   please indicate which searching skills or techniques you have used (check all that apply): a.            keyword searching b.            subject searching c.            boolean operators (and, not, or) d.           advanced searching e.            use of synonyms f.             google search operators (site, link, source, filetype, etc.) g.            browsing the index h.            phrase searching i.             truncation or wildcard (?, *) j.             physical navigation of the stacks k.            other __________ l.             none of the above   18.   please explain any difficulty you encountered when using those skills or information resources? ____________________________________________________________ ____________________________________________________________   19.   were you ultimately successful in finding the non-academic information you were looking for? ___ yes                ___ no   20.   what type(s) of information were you trying to find? (check all that apply:) a.            social/entertainment b.            political c.            legal d.           medical e.            nutrition/diet f.             financial/economic g.            environmental h.            technological i.             exercise j.             other___________   21.   if you have taken lib102/lib 501 “beyond google,” has it helped you in any way? any benefits? ____________________________________________________________ ____________________________________________________________   22.   do you have any additional comments or feedback that you would like to share? ____________________________________________________________ ____________________________________________________________   23.   if you would like to participate in the drawing to win one of eight (8) $50 amazon.com gift cards, please provide your name and email address (it does not have to be your csi address) in the slot below.   thank you for your time and contribution in completing this survey.   appendix b search skills used for academic purposes   lib102 group matched comparison group   search skills keyword searching – 21.2% subject searching – 19.2% advanced searching – 16.4% google search operators – 9.6% phrase searching – 8.2% use of synonyms – 8.2% boolean operators – 6.2% browsing the index – 6.2% physical navigation of the stacks – 2.1% truncation or wildcard (?, *) – 1.4% keyword searching – 26.5% subject searching – 22.4% advanced searching – 10.2% phrase searching – 10.2% use of synonyms – 8.2% boolean operators – 4.1% browsing the index – 4.1% physical navigation of the stacks – 2.0%   appendix c search skills used for non-academic purposes   lib102 group matched comparison group   search skills used for non-academic purposes keyword searching – 21.6% google search operators – 18.9% subject searching – 14.4% advanced searching – 12.6% use of synonyms – 10.8% phrase searching – 7.2% boolean operators (and, not, or) – 5.4% browsing the index – 4.5%   keyword searching – 27.5% subject searching – 22.5% google search operators – 12.5% advanced searching – 10% use of synonyms – 10% phrase searching – 7.5% browsing the index – 5% boolean operators (and, not, or) – 2.5%     appendix d types of information sought for non-academic purposes   lib102 group matched comparison group   types of information sought for non-academic purposes social/entertainment – 21.7% medical – 13.2% political – 12.4% nutrition/diet – 10.9% environmental – 8.5% exercise – 8.5% technological – 8.5% financial/economic – 8.5% legal – 6.2% other – 1.6 %   social/entertainment – 18.9% technological – 15.1% environmental – 13.2% political – 11.3% nutrition/diet – 11.3% medical – 9.4% financial/economic – 7.5% exercise – 5.7% legal – 3.8% other – 1.9 % personal – 1.9%       user-focused values of empathy, empowerment, and communication are unheralded in previous conceptualizations of reference and information services evidence summary   user-focused values of empathy, empowerment, and communication are unheralded in previous conceptualizations of reference and information services   a review of: vanscoy, amy. (2021). using q methodology to understand conflicting conceptualizations of reference and information service. library and information science research, 43(1), 101107. https://doi.org/10.1016/j.lisr.2021.101107   reviewed by: jordan patterson associate librarian st. peter’s seminary london, ontario, canada email: jpatte46@uwo.ca   received: 30 nov. 2021                                                             accepted:  20 jan. 2022      2022 patterson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30081     abstract   objective – to understand how experienced librarians conceptualize reference and information service (ris), and to determine if and to what extent these conceptualizations match existing ris models.   design – q methodology card sort followed by short interview.   setting – academic, public, school, and special libraries in slovenia, south africa, and the united states.   subjects – sixty-six (66) librarians from slovenia, south africa, and the united states.   methods – the researcher asked participants to sort 35 statements about ris from “least like how i think” to “most like how i think.” the participants had the opportunity to comment on their card sort. from these card sorts, the researcher used statistical methods to generate factors describing underlying conceptualizations of ris. these factors were compared to existing literature on ris.   main results – departing from the prevailing “information provision/instruction” conceptualizations of ris, the researcher found that most respondents conceptualized ris according to three previously unacknowledged paradigms: 1) transformation and empathy; 2) communication and information provision; and 3) empowering and learning. fifty-three (53) of the 66 participants loaded on to one of these three factors, i.e. sorted their cards in a similar way to other participants in that factor. factors 2 and 3 supported existing ideas of ris in the literature, whereas factor 1 presented a novel understanding of ris. common to all three factors, however, is a strong focus on the user.   conclusion – traditional models conceptualize ris as emphasizing either information provision or instruction. the practical judgments of experienced, working librarians, however, gesture toward different, more nuanced theoretical conclusions. beyond the traditional poles of ris, librarians consider empathy, empowerment, transformation, and communication as other important aspects of the ris function.   commentary   reference interviews are notoriously nebulous. a hypothetical patron may approach the reference desk with only a half-formed query and the knowledge that they need assistance. what is the ris librarian to make of this situation? existing conceptions of ris typically describe a dichotomy of competing approaches, contrasting the timely but transactional paradigm of information provision with the “teach-a-man-to-fish” paradigm of instruction. before we ask which conceptualization provides a better model of ris, however, we should question whether either accurately describes the work of ris librarians in the first place. this study’s author has steadily been working toward a theoretical resolution in this area, with relevant publications in 2010, 2012, 2013, 2016, 2017, and now, 2021. this well-designed study seeks to address the gap between theory and practice by asking professional librarians how they characterize their own work at the reference desk.   under the rubric of glynn’s critical appraisal tool (2006), this study meets an established standard of validity. the researcher sampled widely, data collection was fair and objective, and the study would be reproducible from the researcher’s notes. q methodology is a well-established system for exploring subjective opinions, and is an appropriate choice for the stated research question, “how do experienced librarians conceptualize ris?” by employing a widely-recognized methodology, the study’s author could refer to numerous previous studies to satisfactorily justify her decisions. for example, the author references a systematic review of q methodology studies (kampen & tamas, 2014) to explain methodological considerations about the number of participants.   this circumspect, reflective concern is a hallmark of the study. the author makes tentative supportive links between this study’s findings and the literature, but is careful not to state anything too strongly where it is not warranted. for instance, while the findings suggest that conceptualizations of ris vary across types of libraries, the researcher cautions that these results should be considered exploratory. the author also acknowledges certain methodological limitations encountered. for example, when participants interpreted certain phrases in the card sort negatively—such as statements perceived to be too self-centered beginning with “i feel…” or “i am…”— and were therefore reluctant to rank them highly. however, since the three factors revealed patterns of opinion across users, a shared lower ranking of perceived-negative phrases could be significant in itself and in fact confirm the characteristics of each factor.   for scholars of ris, this study helpfully crosses an important threshold from the descriptive to the prescriptive. illustrating what ris is by asking how practicing reference librarians themselves conceptualize their work, this study motions toward what ris should be. for instance, how should we teach ris in library schools? toward what ideal of practice should ris librarians aim? already this study suggests that what is most important in ris is not a choice between a and b, but an understanding that a diverse set of users has a diverse set of needs to be met with more than a single-minded approach to service.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154   kampen, j.k., and tamás, p. (2014). overly ambitious: contributions and current status of qmethodology. quality and quantity: international journal of methodology, 48(6), 3109-3126. https://doi.org/10.1007/s11135-013-9944-z   evidence based library and information practice evidence based library and information practice 2010, 5.1     evidence based library and information practice         editorial    looking at things in new ways    denise koufogiannakis  editor‐in‐chief  collections & acquisitions coordinator, university of alberta libraries  edmonton, alberta, canada  email: denise.koufogiannakis@ualberta.ca       2010 koufogiannakis. this is an open access article distributed under the terms of the creative commons‐ attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐ sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original  work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is  redistributed under the same or similar license to this one.      welcome to the first issue of our 5th volume of  evidence based library and information practice.  and what an issue it is! we are very pleased to  showcase the keynote addresses from the 5th  international evidence based library and  information practice (eblip5) conference that  took place in stockholm, sweden in july 2009.  andrew booth, who was chair of the  international program committee for the  conference, kindly agreed to take on a guest  editorial role in gathering these keynote  presentations together. i believe it is the first  time the eblip community has been able to  have the keynote papers from an eblip  conference gathered in this way. reading  these commentaries will give you a true sense  of the themes and questions that were woven  throughout the conference and will give you  lots to think about. for a fuller description and  to put this feature section in context, please be  sure to read andrew’s editorial at the  beginning of the section.    as we begin our fifth year of publication,  submissions of original research articles  continue to grow. in this issue we have five  original articles on topics ranging from  information literacy instruction to cataloguing  of e‐books. two articles look at using  communities of practice in the support of  evidence based practice; clearly this is an area  of emerging research within eblip that  readers will benefit from learning more about.  one of these articles, by urquhart, brice,  cooper, spink, and thomas, won the best oral  presentation award at the eblip5 conference.     in addition to the original research articles, we  also have seven evidence summaries, covering  topics such as unionization, information  seeking behaviour, and the lis blogosphere.  the ebl101 column tackles critical appraisal, a  daunting task that will not seem quite so scary  once you’ve read the column! as well, in the  commentary section, jessie mcgowan and  colleagues have contributed a checklist for the  peer review of electronic search strategies,  developed from research they have recently  completed, which will be of certain interest to  all librarians who do systematic review or  other in‐depth searching.    readers may notice a change to our citation  style with this issue. eblip has made the  switch from mla to apa style. we hope  readers are pleased with this change. the  1 mailto:denise.koufogiannakis@ualberta.ca evidence based library and information practice 2010, 5.1  editorial team felt that apa was a more  common style in the social sciences, and  therefore one with which authors would be  more familiar, leading to fewer copyediting  problems. it also removed the need to  continually insert page numbers whenever the  author was referring to another’s work, and  we like that apa puts the date of the work  front and centre in the in‐text citation so  readers can know when the paper being  referred to was published. certainly we will  go through a transition period that won’t be  perfect, but i hope that this change is one that  is for the better.     finally, i would like to welcome jonathan  eldredge to the editorial team. jon takes over  from scott walter as associate editor for  classics. jon is very familiar with the classics  section, as he himself has written two classic  summaries in the past. we are aiming to  publish at least two classics per year, and jon  would love to hear from you with suggestions  of older research papers that have influenced  practice, stood the test of time, and still have  much to teach us. a major goal of eblip is to  bring research closer to practitioners, and to  make it useful. classics and evidence  summaries are ways in which our team  continually strives to bridge this gap, and  continues to inspire our minds to look at  things in new ways.    2 evidence summary   combining surveys with seating sweeps and observational data yields insights into physical space usage in an academic library   a review of: dominguez, g. (2016). beyond gate counts: seating studies and observations to assess library space usage. new library world, 117(5/6), 321-328. https://doi.org/10.1108/nlw-08-2015-0058   reviewed by: jennifer kaari librarian east orange public library east orange, new jersey, united states of america email: jkaari@eopl.org     received: 10 sept. 2020                                                             accepted:  30 oct. 2020      2020 kaari. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29848     abstract   objective – to propose a new method to assess library space usage and the physical library user experience utilizing multiple data collection techniques.   design – seating usage studies, surveys, and observation.   setting – large university in the southern united states.   subjects – students who physically use the library spaces.   methods – the researcher performed seating sweeps three times a day for one week at time, using a counter to get an accurate headcount of each area of the library. the number of users was recorded on paper and then transferred to excel. a survey for library patrons was created using typeform and distributed through both email and in-person. in addition, the researcher created a photo diary to document how students were using the space, particularly creative and flexible uses of the library space. these photos were collected to be shared with library administration. the researcher conducted the study twice, once at each main campus library.   main results – the initial seating sweeps at one location showed an average of 57 to 85 users engaging in active study, and 57% of users engaged in individual study vs. group study. the sweeping study at the second campus location found that floors designated as quiet floors were the most overcrowded. the researcher found that overall, the actual library use surpassed expected library use. the survey results indicated patron concerns about the lack of available seating, noise policies, uncomfortable furniture, and technology issues such as power outlets and wi-fi connectivity.   conclusion – the researcher found that utilizing surveys in addition to observational data provided a more complete picture of the user experience. photographs also provided depth and texture to the observational data. based on the findings the librarians and administration plan to upgrade furniture and technology options, as well as make changes to the noise policy.   commentary   how do patrons use the physical library space and how can that experience be improved? these are essential questions for many librarians, particularly at a time when the physical function of the library is in flux. to find answers, librarians in a variety of settings have developed methods to count and observe how and when patrons use the library space, most notably through the use of seating sweeps, headcounts, and observational methods (linn, 2013; jaskowiak et al., 2019; olsen, 2019). this study builds on that literature by proposing a combination of sweeping and observational methods with user surveys. this approach allows librarians to incorporate more direct user feedback into the decision-making process when choosing how to improve the library space and its features.   this study had a 75% validity rating based on glynn’s critical appraisal tool for library and information science (2006). the methodology is well described, particularly the emphasis on the adaptions that the researcher made along the way. the author provides a brief but effective review of the literature that helps contextualize a useful synthesis of how to prepare for and implement a library user space study. the study itself was limited by its small scale, as noted by the author. providing the full survey instrument and more detailed quantitative and qualitative data would have been helpful for researchers looking to build on this paper and would have provided a more complete picture of the data collected.   the author’s experiences provide a persuasive case study for the importance of combining multiple data collection methods to acquire a more complex and thorough picture of the user experience. in addition to the user surveys, the photo diaries focused on creative space usage and provided insights into not just how the patrons use the space, but how they adapt when the space does not meet their needs. more complete data allows librarians to make decisions about library upgrades that will truly meet the users’ needs. librarians with an interest in performing physical usage studies at their own institutions would find much practical and theoretical value from this paper.   this paper also opens up several areas for potential future research. for example, the author suggests that the combination face-to-face surveys and physical sweeps could provide an opportunity to increase librarian visibility among patrons. the author also brings up using user experience (ux) as a lens through which to look at physical library spaces, a concept that could use more robust exploration. although primarily used to discuss digital library usage, the methods and tools of ux have intriguing implications when applied to the physical patron experience.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154   jaskowiak, m., garman, k., frazier, m., & spires, t. (2019). we’re all in this together: an examination of seating and space usage in a renovated academic library. library philosophy and practice (e-journal). retrieved from https://digitalcommons.unl.edu/libphilprac/2645   linn, m. (2013). seating sweeps: an innovative research method to learn about how our patrons use the library. in acrl 2013 conference, association of college & research libraries (pp. 511-517).   olsen, h. k. (2019). what are they doing? and where? tracking the traffic as one of the instruments in an evidence-based redesign of a university library. liber quarterly, 29(1), 1-32. https://doi.org/10.18352/lq.10276       evidence based library and information practice evidence based library and information practice 2010, 5.4 7 evidence based library and information practice article an examination of the failure rate and content equivalency of electronic surrogates and the implications for print equivalent preservation ken ladd associate dean university of saskatchewan library university of saskatchewan, saskatoon, canada email: ken.ladd@usask.ca received: 30 june 2010 accepted: 23 oct 2010 2010 ladd. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – this study sought to determine whether evidence indicates a need to preserve print equivalent journal collections. in addition, this research aimed to provide data on the failure rate of print equivalent materials for possible digitization to replace existing poor quality or defective electronic surrogates. methods – the project compared the content of randomly selected journal titles, volumes, and issues from seven electronic journal archives and their print equivalents held at the university of saskatchewan library. the archives were obtained from five separate vendors representing humanities, social sciences, science, technology, and medicine. data were collected on the frequency and types of failure of electronic surrogates, supplemental content missing from electronic surrogates, and frequency and types of failure of print equivalent materials. results – across all electronic journal archives the failure rate of electronic surrogates was 7.5% for all pdf documents and 11.5% for scholarly pdf documents. for individual electronic journal archives the failure rate ranged from 0.7% to 19.5% for all pdf documents and from 0.3% to 26.5% for scholarly pdf documents. data is presented on the failure rate of individual electronic journal archives, types of failure, and missing supplemental content. an examination of print equivalent titles found 1.7% of print scholarly articles could not be used or were not optimal for digitization. mailto:ken.ladd@usask.ca� evidence based library and information practice 2010, 5.4 8 conclusions – the study demonstrates the need for preserving print equivalent journal titles for at least the short (less than 5 years) to medium term (up to 10 years), while poorly digitized materials are identified, replaced, and digitally preserved. while electronic surrogates of image-rich scholarly papers are more likely to have quality issues, the study found some text-only pdf scholarly documents were illegible, indicating the need for caution against liberally applying this as a criterion for disposal of print equivalent titles. there is significant supplemental content absent from electronic surrogates which indicates a need for further discussion of the necessity for such information or for incorporating it into the digitization process to ensure a complete record of the print equivalent journals for future use. the failure rate of print equivalent titles for possible digitization provides additional data for discussions related to the determination of optimal overlap. it also suggests that the number of copies required for a full set of preserved journals over a specified time horizon may be greater than anticipated, unless page level validation is performed. introduction within the past decade there has been a massive migration of libraries moving from print to electronic resources. at the same time, academic research libraries have continued to face space issues related to their primary stacks and storage facilities due to simple growth in their collections. other changes in the teaching and learning environment have resulted in an increased demand for social learning space within libraries. coupled with the development of programs such as jstor, portico, clockss, and publishers’ electronic backfiles of serials, libraries have explored the storage or disposal of print journals when electronic surrogates exist. preservation is recognized as a fundamental role and responsibility of research libraries (association of research libraries, 2007). the ultimate goal of these preservation activities is to ensure that the information contained within library collections is not lost for future access. however, with the shift toward acquiring electronic resources, there are questions about which titles need to be preserved or retained. do both the print materials (print equivalents) and their electronic surrogates need to be preserved? what evidence exists to support either the disposal of print equivalents or their preservation? are there certain indicators of whether both need to be preserved? in 2006 the university of saskatchewan library, in collaboration with other units on campus, initiated a project to transform two floors of the murray library (primarily a humanities and social science library) to create an enhanced learning commons, requiring the removal of approximately five kilometres of shelving and associated contents. with tight deadlines to remove materials, the library decided to discard print journals for which it had electronic surrogates. the library determined the strategy also needed to incorporate preservation as a principle. the strategy was multi-faceted, with jstor titles being flagged for possible disposal. when other print equivalent titles’ preservation was considered less reliable, those titles were flagged for an on-campus temporary storage facility. as the project was rolled out, there was considerable debate between library and other faculty regarding the storage or disposal of print equivalent titles. the spectrum of opinions ranged from both extremes – some felt that any print equivalent title could be discarded, while others felt no print titles should be discarded. literature review a variety of papers have examined either directly or peripherally the differences that exist between electronic surrogates and their print equivalents (bracke & martin, 2005; campbell, 2003; chen, 2005; chrzastowski evidence based library and information practice 2010, 5.4 9 2003; erdman, 2006; henebry, safley, & george, 2002; joseph, 2006; kalyan, 2002; keller, 2005; martellini, 2000; shadle, 2004; sprague & chambers, 2000). these studies were often performed to determine if libraries were able to cancel or withdraw print equivalents from their collections. however, these studies have tended to focus on a specific discipline, specific content issue, specific vendor, or electronic journal databases and aggregators, not the electronic journal itself. studies that have been more multidisciplinary in nature include sprague and chambers (2000), kalyan (2002), and chen (2005). these studies compared electronic surrogates from multidisciplinary full-text databases with their print equivalents. studies by keller (2005) and henerby, safely and george (2002) examined electronic journals from a variety of publishers and disciplines. many of the studies, however, have focused on the sciences:  chemistry chrzastowski (2003)  earth and planetary sciences joseph (2006)  geology erdman (2006)  physics martellini 2000  science and engineering bracke and martin (2005)  science, technology, and medicine – campbell (2003) studies have found that there are often quality issues associated with digitized images and figures (bracke & martin, 2005; chen, 2005; erdman, 2006; henebry, safely, & george, 2002; joseph, 2006; keller, 2005; sprague & chambers 2000), while others have not (campbell 2003; chrzastowski 2003). few studies have indicated a quality issue with text in electronic surrogates. sprague and chambers (2000) noted that formulas and other mathematical expressions were often unclear. keller (2005) noted that some pages of electronic surrogates were not readable. in addition to the issue of quality, some studies have examined or noted the issue of missing content:  missing figures, tables, or graphics – chen (2005); sprague and chambers (2000)  missing pages or articles – bracke and martin (2005); henebry, safely, and george (2002); keller (2005)  missing issues – bracke and martin (2005); joseph (2006); keller (2005)  missing volumes – joseph (2006); keller (2005) two studies did not find an issue with missing content (martellini, 2000; chrzastowski, 2003). keller (2005) and shadle (2004) identified inconsistencies in the journal titles as presented by publishers on their websites, which might be confusing to users. some publishers are very good at noting title changes, while for others only the most current title is displayed. there have been several studies that have focused either on full text journal databases and aggregators (chen, 2005; kalyan, 2002; sprague & chambers 2000) or on a specific vendor – elsevier (bracke & martin, 2005; erdman, 2006; joseph, 2006). with the placement of numerous back runs of journals into its temporary storage facility, and the need to store other back runs permanently, the university of saskatchewan library was interested in exploring opportunities for a collaborative approach for the preservation of print journals. preservation of print journals is complicated, because many factors need to be considered in determining the redundancy required to ensure the existence of a complete run of a journal. schonfeld and housewright (2009) discussed these factors, including the work commissioned by ithaka s+r (yano, shen, & chan, 2008), which noted the redundancy required is dependent on a number of risk factors, including defects in the print materials and loss. the present study was initiated in the spring of 2009 to systematically compare print journals and their electronic surrogates from a variety of vendors across all disciplines. the evidence based library and information practice 2010, 5.4 10 study expands on the existing literature by identifying and quantifying discrepancies between print journals and their electronic surrogates. it also quantifies damage or other irregularities in print journals that limit their use for digitization purposes. aim the primary purpose of this study was to examine whether there is evidence to support the null hypothesis that there is a need to preserve print equivalent serials, at least for the short to medium term. to examine this issue the following questions were asked: o what types and frequency of failure occurred for electronic surrogates? o were there differences in failure rates between electronic surrogate archives? o in addition to failures, what content differences existed? for example, tables of contents, indices, advertising, etc. o what other issues affected access to electronic surrogate content? a secondary purpose was to examine whether the suggestion made by yano, shen, and chan (2008) on the defect rate of print resources for digitization purposes was supported by evidence from the collection at the university of saskatchewan library. to examine this issue, data were collected on the types of failures that occurred with print equivalents and the failure frequency. methods at the time the study was initiated, the university of saskatchewan library had acquired access to 28 archive collections of electronic journal backfiles from a number of vendors. from these collections, seven were chosen to provide a breadth of subject coverage (humanities, social sciences, science, technology, and medicine) from a variety of vendors. appendix a provides a listing of the collections and which of their journals were included in this study. from each collection, five titles held at least partially in print by the university of saskatchewan library were randomly selected. from each of these titles, a specific number of volumes was randomly selected. the number of random volumes was pre-determined based on the library’s holdings and ranged from one to three volumes, but was usually three (60%). one exception occurred when six volumes for a title were selected because each volume had only one issue. within the volumes, a random number of specific issues was selected, based on the number of issues in the volume, as indicated in the electronic surrogate. where there were three or fewer issues in a volume, one issue was selected; with four to twelve issues in a volume, two issues were selected. rare exceptions occurred when combined issues for a volume were randomly selected. for each randomly selected journal issue, the electronic surrogate and the print equivalent were examined. study data for each collection and journal title included several aspects of failure: • the frequency that an electronic surrogate failed at the article (or pdf document) level was defined as any time the print equivalent needed to be consulted to access all the information in the item. while there could be multiple failures within a pdf document, together they would be counted as a single failure for the journal title and collection. • the failure rate for each journal was determined for all pdf documents as well as for all “scholarly” content. for this study, scholarly content included research papers, case studies, review articles, short communications, technical notes, and errata. in addition to the “scholarly content,” “all content” included book reviews, evidence based library and information practice 2010, 5.4 11 announcements, letters to the editor, meeting programs, and obituaries. a distinction was made between the two content types, because users seek the scholarly content of scholarly journals most often. thus, a failure in a scholarly pdf document was more likely to result in the print equivalent being required. • the type of failure was observed for electronic surrogates, and it was recorded for only the initial failure in a pdf document. eight categories were used to describe the observed failures. five of the categories related to the quality of the scan or of the digitization. the other three categories related to missing, incorrect, or additional content. • frequency and types of supplemental content missing from the electronic surrogate were noted. supplemental content was defined as anything beyond the article level. examples could be advertising, tables of contents, instructions for authors, and so on. on a single page more than one type of supplemental content could occur, and each was recorded. • the research team noted the frequency that print equivalent articles failed as a possible source for digitization. failure was defined as any occurrence of missing or damaged pages, such as markings or tears. • the study also identified several types of failure for print equivalent articles. each digital surrogate within an issue was examined for content equivalency and the legibility of text, graphs, figures, and images. the digital surrogate was first examined for quality issues and any perceived problems were compared to the print equivalent to determine if they were an artifact of the digitization process. a second examination evaluated the print equivalent for quality issues, while at the same time noting irregularities with content equivalency. results the study involved an examination of seven archival electronic journal collections covering the humanities, social sciences, science, technology, and medicine. table 1 indicates the number of titles in each collection and the number of titles sampled. table 1 electronic archival collections examined titles volumes pdf collection archive sample archive held sample compared elsevier medicine and dentistry 44 5 128 104 14 344 elsevier social science 26 5 72 54 12 455 jstor arts and science i 175 5 311 301 14 701 oxford university press digital archive 50 5 355 339 18 553 springer link historical archives mathematics 34 5 32 26 8 53 wiley humanities and social sciences 69 5 169 118 13 247 wiley science, technology and medicine 79 5 105 97 12 280 total 477 35 1,172 1,039 91 2,633 evidence based library and information practice 2010, 5.4 12 for the titles sampled, the table notes the number of volumes within each electronic archive held in the university of saskatchewan library that were sampled. finally, the table indicates the number of electronic surrogate pdf documents that were compared with their print equivalents. the initial part of the study examined the quality of the sampled electronic surrogates by determining failure rate – how frequently print equivalent materials had to be consulted in order to access all the information contained in the item. each vendor was found to follow digitization practices for print equivalent journals that impacted the failure rate for all pdf documents. for some vendors the number of pdf documents was higher because individual pdf files were created for each book review, table of contents, obituary, and announcements, in addition to the scholarly articles, case studies, review articles, errata, and short communications. for example, whether a vendor used one pdf document for ten book reviews or presented them as ten pdf documents impacted the calculation of the failure rate. the study found a wide variance of failure rates between backfile collections. figure 1 shows the percentage failure rate of scholarly pdf documents and of all pdf documents for each of the seven archival collections and across all collections. five of the seven collections had higher failure rates for scholarly pdf documents when compared to all pdf documents. there was a wide variance in failure rate between collections, with the science, technology, and medicine collections usually having the highest failure rates. for scholarly pdf documents, jstor’s failure rate was at least an order of magnitude lower than any other collection. three of the collections had higher failure rates for scholarly pdf documents than the average rate of 11.5% observed for all seven collections. while jstor’s mission differs from that of commercial vendors, the results demonstrate the quality that can be achieved with journal digitization initiatives, which logically can be ascribed to excellent quality control practices. fig. 1. failure rate for electronic surrogates (scholarly and all pdf documents) when compared to print equivalents. 19,5% 4,8% 0,7% 3,4% 5,7% 14,2% 16,8% 7,5% 26,5% 10,9% 0,3% 3,9% 3,9% 18,0% 18,4% 11,5% 0,0% 10,0% 20,0% 30,0% elsevier medicine and dentistry backfile elsevier social science backfile jstor arts & science i oxford university press digital archive springer link historical archives mathematics wiley (synergy blackwell) humanities & social science backfile wiley (synergy blackwell) science, technology & medicine backfile all collections scholarly all evidence based library and information practice 2010, 5.4 13 the study examined types of observed failures (fig. 2). fig. 2. types of failures observed first within an electronic surrogate pdf file as a percentage of all observed failures. the predominant type of failure (39.9%) was related to the quality of an image, such as an x-ray, photomicrograph, chromatograph, or scintograph. other quality-related issues were observed for text and numbers: 20.7% of these failures occurred in the body of a paper, and 7.1% occurred in a table or figure. the quality of graphs, maps, and drawings was an issue in 18.7% of failures resulting from difficulty distinguishing different shading or fill for a bar graph, symbols on a graph, or lines on a graph, drawing, or map. missing pages accounted for nearly 10% of the failures. for two cases there appeared to be missing pages, but closer inspection determined that one page appeared as a miniature image within the pdf. by clicking on the image and increasing magnification to 2400% or higher, it was possible to read the content. while only the initial failure was recorded, there were 15 pdf documents that had a second type of failure, or 7.6% of all failed pdf documents. one observation of importance not included in the calculations for failure rates or types of failures was the complete absence of electronic surrogates for two issues from one volume of a journal, or 1.1% of all volumes sampled for the study. the international journal of nuclear medicine and biology is part of the elsevier medicine and dentistry collection. the random selection of volumes and issues was based on the electronic surrogate journal issues listed by the vendor. in this case, the vendor’s site indicated four issues for volume 12. after a comparison with the print equivalent, six issues were identified as having been published in this volume. the missing issues contained one editorial, twentytwo papers, one technical note, three letters to the editors, reports of eleven new patents, one book review, one announcement, and advertising. the study compared the electronic surrogate pdf documents associated with a journal issue and their print equivalent journal issues to determine whether there was supplemental content not included as an electronic surrogate. the research team analyzed missing pages for all journal issues selected for this study, and results were graphed as a percentage of all journal issues examined (fig. 3). 1,0% 1,5% 18,7% 39,9% 1,5% 9,6% 20,7% 7,1% 0,0% 10,0% 20,0% 30,0% 40,0% 50,0% additional page advertisement quality graph/map/drawing quality image quality image incorrect/absent/duplicated missing/appears missing page text/numbers in article text/number in table or figure evidence based library and information practice 2010, 5.4 14 fig. 3. supplemental content found in print equivalent journal issues not present in electronic surrogates. the study found eight different types of missing supplemental content in over 25% of the journal issues examined, with two types being above 50%. one function of a print archive can be digitization at the article level to replace poor quality or defective electronic surrogates. data collected on the failure of print equivalent materials are shown in table 2. failure of print equivalent materials at the individual document level was quite low. for all print equivalent scholarly documents, the failure rate was 1.7% (26 of 1,552 items). for all print equivalent documents examined (scholarly and other), the failure rate was 1.3% (34 of 2,633 items). over half of the failures (n=14) for scholarly print equivalent documents were the result of articles being marked up by users with pens, pencils, or highlighters. there were 6 damaged issues, primarily due to pin binding, and two papers that had been removed intentionally. (one paper had been completely removed, resulting in a page missing from the preceding paper in the same issue.) two papers could not be read as bound, but could perhaps have been used for digitization purposes if unbound. finally, there were two papers that were illegible due to poor print resolution. table 2 occurrences of types of print failures type of failure scholarly other tight or close binding, not legible 2 0 print faded or otherwise not legible 2 0 damaged page 6 2 missing page 2 3 page markings 14 3 total 26 8 44,9% 15,2% 39,9% 51,3% 6,3% 44,9% 33,5% 3,8% 27,8% 41,8% 58,2% 10,1% 0,0% 20,0% 40,0% 60,0% 80,0% advertisements association/society information copyright information editors/editorial board index instructions to authors journal information list of contributors, referees, etc. printer/publisher information subscription information table of contents title page for section evidence based library and information practice 2010, 5.4 15 at the issue level, the highest frequency of print failure was associated with the removal of covers or pages for binding. a total of 19% (30 of 158) physical issues had this material removed. in addition to the problems associated with electronic surrogates noted above, there were a variety of other problems observed. for some vendors the only journal title listed for an electronic surrogate was the current title. if there had been one or more title changes, the previous titles were not listed. many vendors had pagination errors associated with a particular issue or with individual pdf documents. a total of 57 pagination errors were noted, with 54.4% of these errors being associated with scholarly pdf documents. discussion failure of electronic surrogates one of the challenges associated with comparing electronic surrogates with their print equivalents is the subjectivity involved. at what point does an electronic surrogate fail by requiring the user to access the print equivalent? for this study a somewhat liberal definition was used – if the text was not clearly legible but the word(s) in question could be deduced from the context provided by other legible text, it was not considered a failure. thus, the failure rate of 11.5% for all scholarly pdf documents can be considered conservative. a significant observation of the study was the absence of corresponding electronic surrogates for two issues of one elsevier journal title and volume. because the study design used the electronic surrogates as the basis for randomly selecting journal issues, the two missing journal issues were not included for possible random selection and therefore not incorporated into the calculation of failure rate of electronic surrogates. if either of these issues had been included in the study, the observed failure rate for scholarly pdf documents would have increased by approximately 0.5% to 12%. in examining the different failure rates by collection, it was evident that for collections containing image-intensive papers, there was a corresponding increase in failure rates. this interpretation was supported by the examination of types of failures, where approximately 40% of all initial failures were due to poor image quality. including failures associated with poor quality graphs, maps, and drawings increased the failure rate to almost 60% for poor quality images and figures. this result supports the use of image intensiveness of publications as a criterion for preservation, as noted in ithaka’s recent paper on what to withdraw (schonfeld & housewright, 2009). at the other end of the spectrum are text-only publications and the possible use of this criterion for materials that could be potentially withdrawn with minimal risk. while the study confirmed that there are certainly fewer problems associated with text-only publications, they are not without significant problems. of the initial problems, approximately 20% of the failures were associated with text. for one particular title, papers in regional science (current title), the failures were frequently associated with mathematical formulas where there were superor sub-scripts. however, for this title the overall quality of the electronic surrogate was poor, making text difficult to read in general. if the text associated with tables and figures had been included, the overall failure rate related to text would have increased to almost 28%. the study found that many of the failures appeared to be with earlier volumes for titles. this might imply that the digital surrogates were created with technology that produced lower resolution or quality of electronic surrogates. re-digitization of these materials would likely eliminate many of the failures, especially for text-only titles. while the quality of the electronic surrogate is something that can be addressed by the redigitization of the print equivalents, there also is an overall quality control issue. almost 10% evidence based library and information practice 2010, 5.4 16 of all initial failures were associated with pages missing from the electronic surrogate. an additional 1.5% of the failures were associated with incorrect digital content. each of these issues is related to quality control applied following the digitization of the materials. there is an obvious cost associated with applying high quality control practices, but even a minimal inspection should catch many of the observed failures, such as incorrect pages scanned or missing pages. it was apparent that jstor, at least for the titles examined, has incorporated quality control practices to ensure high-resolution electronic objects and almost no failures. thus, the issue appears to be more about quality control and the use of high-resolution digitization technology than whether a title is text-only or image-intensive. supplemental content missing from electronic surrogates when comparing print equivalents and electronic surrogates, the issue of content equivalency emerges. steve mckinzie (2005) argued that back runs of print equivalent journals should be kept, as electronic surrogates may not include advertisements, conference announcements, and other material. this study examined the scope of this issue. any content, beyond scholarly work, book reviews, letters to the editor, editorials, and so on was noted. this was not intended to be a comment on the value of this supplemental content but rather an inventory of the type of material being excluded. it should be noted, however, that in some cases vendors do provide this content, often as “front matter” or “back matter,” so in those cases others have made some value judgments. for 58.2% of the issues examined, the table of contents was not provided as an electronic surrogate. as vendors often provide a table of contents in the form of a list of electronic surrogates for a particular issue, this may be less of an issue. other types of supplemental content that may have less impact when absent are indexes and title pages for sections in an issue. there was a variety of other supplemental content not included that historians, sociologists, librarians, and others might wish to consult. this included information related to editors and editorial boards, which was missing from over 50% of the electronic journal issues examined for the study. this information would be needed to identify editorial changes that have occurred over time. advertisements were absent 44.9% of the time and would likely be of interest for sociological research. information about the journals such as aims and scope, objectives, editorial policy, indexing resources, and availability of back and special issues was absent more than one-third of the time. missing information about the association or society publishing the journal included directories of officers, membership lists, and association histories. other missing content of possible interest included instructions to authors and lists of contributors, reviewers, and referees. the impact of supplemental content being absent is dependent on the perceived value of the information and the need to access it. it could be argued that the information might be used for research or general information purposes. if we continue to move towards reliance on electronic surrogates and the disposal of print, the potential impact of its absence increases. thus, it would be beneficial to include this information in the electronic surrogate collections to ensure that options are not limited in this area. failure rate for print equivalent journals two variables of interest to discussions of optimal overlap are the defect rate and the loss rate of print equivalent journals that could be used for digitization (yano, shen, & chan, 2008). defects could include damage (e.g., pages removed intentionally, marked pages, or torn pages). the loss of content, whether through defect or loss, impacts on access and the ability to digitize. evidence based library and information practice 2010, 5.4 17 in determining the suitability of the print equivalents examined in this study for digitization purposes, it was found that overall the worst-case scenario was a failure rate of 1.7% for scholarly print equivalent documents and 1.3% for all print equivalent documents. the most frequently observed damage to content was from users marking the item for their own use, which occurred in more than 50% of the instances, or 0.9% of the print equivalent documents. in most instances the markings did not obscure the content, indicating digitization would be possible but not optimal. damage to pages was the second most frequent observation for print failure. at the university of saskatchewan this was primarily from the past practice of occasionally using pin binding. holes drilled for pin binding at times went through text, rendering it unusable for digitization purposes. at the outset of the study, there was some initial concern about the level of damage that might be observed from users intentionally removing content. surprisingly, only one journal issue was found to have materials intentionally removed, which represented only 0.1% of the print equivalent scholarly content. while the individual was targeting only one scholarly paper, its removal resulted in four print equivalent documents being affected – one page from a second article, the targeted article, and two non-scholarly documents. the observed level of intentional removal of content may be due in part to the journals that were randomly selected, as anecdotal evidence indicates that certain collections that do not circulate at the university of saskatchewan library (such as nursing) or those that contain art-related images are more susceptible to intentional damage. in the two instances where the print equivalent materials were not legible, the problem was due to poor printing processes used for the original documents. the electronic surrogate for these items was also illegible, resulting in the loss of some content. as a result, re-digitization would not be an option. for the two cases where pin binding made the materials illegible, removing the binding might have resolved the issues for digitization purposes. yano, shen, and chan (2008) noted that for 23 of 25 jstor journals being prepared for scanning, there was a defect rate of one per 10,000 to 100,000 pages. the 2 journals with a higher defect rate were a nursing journal and a medical journal. the authors speculated that the defects might have been due to higher usage. they noted that these statistics might not represent journals in general, because jstor had sought copies that were relatively clean. they observed that a significant portion of these materials were obtained from major research university libraries and suggest that “off-the-shelf” journals will generally be of very good condition. the results of this study cannot be directly compared; however, the defect rate appears to be higher here with 1.7% of scholarly print equivalent documents failing. these results suggest that the condition of “off-the-shelf” journals will generally be good, but this risk factor is higher in the current study. other observed issues in comparing print equivalents with the listing of electronic surrogates provided by vendors, there were frequent errors associated with the pagination listed for the surrogates. this could be potentially confusing to users trying to locate a specific paper. in most cases it would only be an obstacle, as an examination of article titles and authors should result in accessing the desired paper. it also reflects an overall quality control problem that may be a flag for other issues. perhaps a more confusing problem, particularly for individuals who are not familiar with the journal in question, is the practice of some vendors of not noting title changes. both jstor and elsevier were particularly good at tracing title changes, while wiley and oxford university press were less so. this issue of inaccurate journal titles evidence based library and information practice 2010, 5.4 18 has been noted before (shadle, 2004; keller, 2005), but has not been resolved in the intervening years. librarians and users must continue to work with publishers and vendors to ensure they are aware of the importance of recording title evolution to ensure users can easily locate the resources they require, and to enable libraries to confidently identify their electronic surrogate collections. conclusion this study was initiated to determine whether evidence supported the preservation of print equivalent journal collections. evidence was sought by determining how frequently electronic surrogates failed to provide access to all content within an individual pdf document. of particular interest was the failure rate associated with the scholarly content of journals. recording the types of failures provided evidence associated with the use of different criteria for the preservation or withdrawal of print equivalent journals. in addition, the study examined the frequency that print equivalent materials failed to be eligible for digitization to replace poor quality electronic surrogates. recording the frequency and types of failures provided evidence associated with calculating optimal overlap for archiving of print journals. the study clearly demonstrates there is a need for preserving print equivalent journal titles for at least the short to medium term. while the electronic surrogates of image-rich scholarly papers are more likely to have quality issues, the study found some text-only scholarly papers were illegible, indicating caution for liberally using this criterion for disposal of print equivalent titles. this is further supported by ample evidence of quality control related issues, such as duplicate pages, missing pages, missing issues, additional pages, and poor quality scans. re-digitization with high-resolution scanning technology and good quality control practices would eliminate many of the observed failures. retaining print equivalent journals for the short to medium term will place additional pressure on libraries already facing space issues related to expanding collections and the demands for user-related space. this pressure will be best met by collaborative approaches to retaining materials at the regional or national level. the absence of supplemental content in many cases indicates the need for further discussion of the necessity for such information or incorporating it into the digitization process to ensure a complete record of the print equivalent journals for future use. the failure rate of print equivalent titles suitable for digitization provides additional evidence of defect rates that applies to work by yano, shen, and chan (2008) on optimal overlap for print preservation models. this study’s results indicate the risk factor was greater than that noted by yano and her colleagues for titles being prepared for scanning by jstor. thus, the number of copies required for a full set of preserved journals over a specified time horizon may be greater than anticipated, unless page level validation is performed. while this study demonstrates a variety of deficiencies related to electronic surrogates of print equivalent journals, a future study of the impact of these deficiencies on libraries and their users would be useful. determining the impact will indicate the risks associated with not addressing these deficiencies and will assist decision-making related to digitization, preservation, and retention of print equivalent volumes. in addition, a study that quantifies the issue of web-based title inconsistencies would be helpful. although several studies, including this paper, have observed and commented on this issue, it has not been quantified. such a study should shed light on the extent of the problem and explore current practices of specific publishers regarding tracing title histories and best practices. evidence based library and information practice 2010, 5.4 19 references association of research libraries. (2007). research libraries’ enduring responsibility for preservation. retrieved 25 nov. 2010 from http://www.arl.org/bm~doc/preservation_ responsibility_24july07.pdf bracke, m. s., & martin, j. (2005). developing criteria for the withdrawal of print content available online. collection building, 24(2), 61-64. campbell, s. (2003). print to electronic journal conversion: criteria for maintaining duplicate print journals. feliciter, 49(6), 295-297. chen, x. (2005). figures and tables omitted from online periodical articles: a comparison of vendors and information missing from full-text databases. internet reference services quarterly, 10(2), 75-88. chrzastowski, t. e. (2003). making the transition from print to electronic serial collections: a new model for academic chemistry libraries? journal of the american society for information science and technology, 54(12), 1141-1148. erdman, j. m. (2006). image quality in electronic journals: a case study of elsevier geology titles. library collections, acquisitions, and technical services, 30(3-4), 169-178. henebry, c., safley, e., & george, s. e. (2002). before you cancel the paper, beware: all electronic journals in 2001 are not created equal. the serials librarian, 42(3-4), 267-273. joseph, l. e. (2006). image and figure quality: a study of elsevier's earth and planetary sciences electronic journal back file package. library collections, acquisitions, and technical services, 30(3-4), 162-168. kalyan, s. (2002). non-renewal of print journal subscriptions that duplicate titles in selected electronic databases: a case study. library collections, acquisitions, and technical services, 26(4), 409-421. keller, a. (2005). the race to digitize: are we forfeiting quality? serials, 18(3), 211-217. martellini, e. (oct. 2000). physics journals and their electronic version: a comparison. high energy physics libraries webzine, (2), retrieved 20 nov. 2010 from http://library.web.cern.ch/library/webzine /2/papers/3/ mckinzie, s. (2005). op ed: troubling choices: full-text access and the old hard copy back runs. against the grain, 17(1), 60-61. schonfeld, r. c., & housewright, r. (29 sept 2009). what to withdraw? print collections management in the wake of digitization ithaka. retrieved 25 nov. 2010 from http://www.ithaka.org/ithaka-sr/research/ what-to-withdraw shadle, s. (2004). electronic journal forum: reflections on wrapping paper: random thoughts on aacr2 and electronic serials. serials review, 30(1), 51-55. sprague, n., & chambers, m. b. (2000). full text databases and the journal cancellation process: a case study. serials review, 26(3), 19-31. yano, c. a., shen, z. j. m., & chan, s. (2008). jstor seeks efficiency and security for print backups of online journals. berkeley, ca: department of industrial engineering and operations research, university of california. evidence based library and information practice 2010, 5.4 20 appendix a: titles compared in each collection elsevier – medicine and dentistry  american journal of orthodontics  biochemical medicine and metabolic biology  british journal of tuberculosis and diseases of the chest  international journal of nuclear medicine and biology  prostaglandins, leukotrienes, and medicine elsevier – social sciences  government publications review  journal of behavioral economics  social science & medicine. part b, medical anthropology  studies in comparative communism  transportation research. part a, general jstor arts and sciences 1  american journal of mathematics  journal of health and human behavior  journal of the history of ideas  reviews in american history  speculum oxford university press  occupational medicine  parliamentary affairs  past & present  rheumatology  the year's work in clinical and cultural theory springer mathematics  computational optimization and applications  constraints  journal of cryptology  journal of nonlinear science  k-theory wiley interscience (synergy blackwell) – humanities and social sciences  papers in regional science  social policy and administration  journal of philosophy of education  psychology of women quarterly  review of policy research wiley interscience (synergy blackwell) – science, technology and medicine  european journal of clinical investigation  international journal of experimental pathology  journal of human nutrition and dietetics  journal of oral pathology and medicine  sedimentology / evidence based library and information practice/ / introduction aim the primary purpose of this study was to examine whether there is evidence to support the null hypothesis that there is a need to preserve print equivalent serials, at least for the short to medium term. to examine this issue the following questions w... methods results table 1 fig. 1. failure rate for electronic surrogates (scholarly and all pdf documents) when compared to print equivalents. the study examined types of observed failures (fig. 2). fig. 2. types of failures observed first within an electronic surrogate pdf file as a percentage of all observed failures. table 2 discussion failure of electronic surrogates supplemental content missing from electronic surrogates failure rate for print equivalent journals other observed issues conclusion references appendix a: titles compared in each collection elsevier – medicine and dentistry elsevier – social sciences jstor arts and sciences 1 oxford university press springer mathematics wiley interscience (synergy blackwell) – humanities and social sciences wiley interscience (synergy blackwell) – science, technology and medicine research article   research productivity and its relationship to library collections   sandra l. de groote professor and head of assessment and scholarly communications university of illinois at chicago, university library chicago, illinois, united states of america email: sgroote@uic.edu   beyza aksu dunya clinical assistant professor, university library, university of illinois at chicago assistant professor, bartin university college of education chicago, illinois, united states of america email: baksu2@uic.edu   jung mi scoulas clinical assistant professor and assessment coordinator university of illinois at chicago, university library chicago, illinois, united states of america email: jscoul2@uic.edu   mary m. case university librarian and dean of libraries university of illinois at chicago, university library chicago, illinois, united states of america email: marycase@uic.edu   received: 14 feb. 2020                                                               accepted: 7 july 2020      2020 de groote, akusa dunya, scoulas, and case. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29736       abstract   objective – the purpose of this study was to explore in the current academic library environment, the relationship between library collections data (collections’ size, expenditures, and usage) and faculty productivity (scholarly output). the researchers also examined the degree to which new and existing library metrics predict faculty productivity.   methods – demographic data (e.g., faculty size, student size, research and development expenditures), library budget data (e.g., collection expenditures), collection use data (e.g., full-text article requests and database searches), and publication output for 81 doctoral granting universities in the united states were collected to explore potential relationships between research productivity, collection use, library budgets, collection size, and research expenditures using partial correlations. a hierarchical multiple regression was also used to ascertain the significance of certain predictors of research productivity (publications).   results – a correlation existed between the number of publications (research productivity) and library expenditures (total library expenditures, total library material expenditures, and ongoing library resource expenditures), collection size (volumes, titles, and ebooks), use of collection (full-text article requests and total number of references in the articles), and research and development expenditures. another key finding from the hierarchical multiple regression analysis showed that full-text article requests were the best predictor of research productivity, which uniquely explained 10.2% of the variation in publication.   conclusion – the primary findings were that full-text article requests, followed by library material expenditures and research expenditures, were found to be the best predictor of research productivity as measured by articles published.     introduction   in this study, the authors examined the relationship between academic library collections and research output at research intensive doctoral granting academic institutions. with shrinking library budgets and increasing costs for the online resources licensed or purchased by libraries, it is becoming more challenging to provide access to the information resources needed by researchers at academic institutions. librarians are increasingly spending more time trying to determine how best to spend their limited budgets as they consider what new resources to purchase, what resources to maintain, and what resources to cancel. academic libraries face challenges demonstrating the need for appropriate funding to continue to meet the information needs of researchers.   academic libraries have evolved greatly since the migration to electronics resources (e.g., online journals, databases). many libraries have increased the number of journal titles available to users through the licensing of “big deal” journal packages (packages are selected for needed journal titles but also typically include journal titles that a library would not necessarily choose). journal collections have moved from a print to an online format and abstracting and indexing (a&i) tools for finding journal literature, once found only in print, have also moved online and increased in numbers. as a result, the way in which researchers seek and obtain information from the library has changed dramatically.   with resources online, new sources of usage data are also available. for example, libraries can obtain data on how many full-text article downloads occur and how many database searches are run, a direct reflection of patron activities. in databases such as scopus and web of science, data can be more readily obtained, compared to a print-based environment, on how many articles were published by an institution, how many references were included in those articles, and how many times those articles were cited. how libraries are asked to report collections and collections usage has also changed. between 2011 and 2012, the association of research libraries (arl) ceased asking for data on certain metrics (monographs purchases, total current serials, expenditures for monographs, expenditures for serials), and began asking for data on such metrics as number of electronic books (ebooks), ongoing resource purchases, collection support, number of successful full-text article requests (journals), and number of regular searches (databases) for their arl annual library statistics. one question that arises is whether any of the new metrics available can illustrate a relationship between faculty productivity and the use or availability of library resources.   the purpose of this study was to explore what new and existing library metrics demonstrate a relationship with faculty productivity. this study explored the relationship between research productivity, as defined by the number of journal articles published by an institution, and (a) the size of library collection budgets, (b) the size of the collection (e.g., number of titles held, number of volumes held), (c) use of the collection (number of successful full-text article requests, number of database searches, references included in publications), and (d) other library and institutional expenditures.   literature review   several researchers have quantitatively examined the relationship between research productivity and various metrics reflecting resources available to an institution, including library collections. rushton and meltzer (1981) studied 169 leading universities from the united states, the uk, and canada, exploring the relationship between total publications and total faculty size, total student size, revenue, number of volumes in the library, and number of current library periodical subscriptions. they found that when universities were high on one measure, they were high on others as well. specific to the american universities, there was a positive correlation between the number of volumes in the library and total publications and the number of current journal subscriptions and total publications. revenue was found to be the principal factor that could predict the result of the other variables being examined, concluding that “the quality and wealth of a university are clearly related” (rushton & meltzer, 1981, p. 301). dunbar and lewis (1998) also explored quantitative factors influencing and contributing to research performance at doctoral institutions in the united states, using data from the 1993 national research council study. they explored 30 doctoral programs in the biological sciences, physical sciences and mathematics, social and behavioral sciences, and engineering from top carnegie classification research universities. they found programs with more faculty were more productive, although when some programs got beyond a certain size, productivity declined. in addition, more full professors and more financial research support within a department also resulted in increased research productivity. with respect to the library, except for engineering and social sciences, departmental research productivity was found to have a significant positive relationship with library expenditures. in a more recent study, rawls (2015) used arl annual library statistics prior to the 2011/2012 change of variables. rawls utilized arl library expenditure variables and other data including total number of faculty publications, faculty size, research expenditures, and grant awards from a five-year period (2005–2009) to explore the relationship between faculty research productivity and library investment. rawls found that research productivity was positively correlated with library investment. more specifically, electronic library resource expenditures correlated positively with an increase in productivity.   surveys of academic researchers have also been conducted to examine researchers’ use of the library and their information seeking behavior, and to assess the value of the library’s collection as it relates to their research and other academic activities. results of a survey of randomly selected faculty at four large state universities in texas found only 2.6% reported that the library resources were not important in their research efforts and over 68% indicated library resources were of “considerable” or “very high” importance to their research (cluff & murrah, 1987). the findings also suggested that the larger the university, the more likely faculty would report dependence on the library for research purposes. faculty members from seven universities (five u.s. and two australian) were surveyed in 2004/2005 about their scholarly article reading habits (tenopir, king, spencer, & wu, 2009). faculty were asked to recall how many scholarly articles they had read in the past 30 days and the source of the articles they read. faculty members in more research-oriented positions reported reading more for research purposes (62%) compared to the amount reported by teaching-oriented faculty (49%). also, the more a faculty member published, the greater they reported their reading was for research purposes. faculty members in research-oriented positions also reported that 58% of their reading materials were provided by the library, in contrast to faculty in teaching-oriented positions, who reported that 37.8% of their reading materials were provided by the library. a more recent survey of provosts assessed their perceived value of the academic library (murray & ireland, 2018). respondents from universities falling under the carnegie classification as “research very high” institutions perceived the academic library as very involved (84.21%), somewhat involved (10.53%), or marginally involved (5.26%) in faculty research productivity.   longitudinal survey data collected over a 30-year period has also provided insight toward researcher behavior over time and their use of the library. tenopir, king, edwards, and wu (2009) concluded that faculty are reading more articles than they had in the past and that faculty were relying more on the library to provide access to articles rather than the personal subscriptions they had relied upon in the past. the availability of online articles resulted in faculty using more methods to identify the articles to read, including browsing the table of contents of online journals and searching for articles using internet search engines, full-text databases, and online a&i databases, compared to how they searched for articles in the print environment.   the value of using literature in grant proposals, grant reports, and articles has also been studied, in part to examine the impact of increased access to journals available in the electronic collections of academic libraries. a 2009 online survey, sent to faculty at seven different institutions in seven different countries, examined faculty citing behavior (tenopir et al., 2011). on average, 90% of the respondents indicated that citations were “important”, “very important” or “essential” as part of the grant writing process. approximately 69.6% of respondents reported citing 10 or more references in grant proposals, and 82.2% of respondents reported citing at least one reference in final grant reports. seventy-five percent reported accessing more than half of the articles through their library’s electronic collection, and over 50% reported accessing 75% of the articles from the library’s electronic collection. using a return on investment (roi) model, kaufman (2008) explored the connection between use and investment in the library and funded grant proposals. grant applicants at university of illinois at urbana-champaign were surveyed regarding the role of the library in their research and grant processes. ninety-five percent of respondents indicated that references were important for obtaining grants, and approximately 75% of respondents noted that 75% of the references used in grant proposals were provided through the library. the resulting roi calculation found that for every dollar invested in the library, there was a return on investment of $4.38 in grant funding (kaufman, 2008).   using various methodologies, the studies noted above illustrated that academic library collections, such as journal articles, books, and databases, are important sources of information for use in research, teaching, and grant proposals. however, while surveys assessing information seeking behavior, library collections use, and the value of the library provide valuable insight and allow for flexibility in terms of the questions asked, they rely on memory and perception to provide the data from which the findings are drawn. while researchers in older studies have shown a relationship between research productivity and collection size (rushton & meltzer, 1981) or collection expenditures (dundar & lewis, 1998; rawls, 2015), these studies were based on data gathered prior to or during the transition of the library from a print based to an online environment. academic library collections have changed, user information seeking behavior has changed, and the measures for tracking library usage have also changed and evolved. newer literature exploring quantitative data in relation to research productivity is limited. exploring older metrics in the present library environment and exploring newer library metrics, such as collections usage data, in relation to research productivity can contribute to validating the impact academic libraries have on scholarly output. this study will explore faculty productivity and its relationship with library resource usage, library budgets, and collection size. because past researchers have noted the relationship with faculty productivity and overall institutional support (rushton & meltzer, 1981), research support (dunbar & lewis, 1998), and library expenditures (dunbar and lewis, 1998; rawls, 2015), we also explore overall research expenditures with respect to faculty productivity and library expenditures.   aims   this study aimed to examine the relationship between library collections, budgets, and use and research productivity among academic research institutions in the united states that were both members of the arl and designated as doctoral universities according to the carnegie classification of institutions of higher education.   methods   to identify the list of institutions to include in the study, the list of arl institutions was downloaded from the arl website (association of research libraries, 2016) and the list of universities designated as very high research activity or high research activity doctoral institutions was downloaded from the carnegie classification of institutions of higher education website using the basic classification feature to select the doctoral universities (carnegie classifications, 2016). this data was entered into a single spreadsheet, where a total of 104 academic research institutions were identified for inclusion in the study.   data regarding library collection size, budget, and use, research productivity (journal publications), and institutional demographic data were also obtained from various resources and entered into the spreadsheet. because arl surveys its arl member libraries on an annual basis related to multiple data points including staffing collections, expenditures, services, and usage, the arl annual library statistics was an ideal resource for library related data. scopus, an online indexing and abstracting database produced by elsevier, was selected to provide the number of publications for each institution included in the study. scopus has been reported to be the most comprehensive article-level index of scholarly articles (laakso & bjork, 2012). in order to have a metric for the overall expenditures of an institution, the research and development expenditures from the higher education research development (herd) survey was selected, as this is the primary source of information on research and development expenditures at u.s. universities (national science foundation, 2017). further details on how the data were collected is provided below.   data collection   arl annual library statistics data   data related to library collections size, use, expenditures, and additional institutional demographic information reported by each institution to arl statistics (https://www.arlstatistics.org) was obtained for 2015, 2016, and 2017. because data and resources might fluctuate somewhat from year to year, instead of examining a single year of data, the last three available years of data were collected. the three years of data were then averaged for each variable to obtain the final data used in the analyses. the variables collected included:   ·        ongoing library resources expenditures – total amount spent on subscriptions and annual license fees (online and print serials, online indexes and abstract resources, et al.) ·        total materials expenditures – includes one-time purchases (non-subscription, one-time purchases such as books, software, backfiles, et al.), ongoing library resources expenditures, and other collections support ·        total library expenditures – the total expenditure of all library funds (includes total library materials, total salaries and wages, and other operating expenditures but excludes fringe benefits) ·        ebooks – total number of ebooks available in the collection ·        volumes held – total number of print only items and ebooks ·        titles held – total number of print and electronic serials, monographs, manuscripts, dissertations and theses, archives, microforms, and computer files held in the collection (excludes duplicates) ·        number of successful full-text article requests for journal articles (defined by the counter code of practice – www.projectcounter.org) ·        number of regular searches – number of database searches as defined by the counter code of practice – www.projectcounter.org) ·        total full-time students (undergraduates and graduates) and total faculty (full-time members of the instruction/research staff) ·        size of students and faculty were included as control variables that represents the institutional size, which may have an impact on research productivity and use of the resources.   research productivity data   research productivity in this study is referring to the number of journal publications produced by an institution. in august of 2018, we searched the scopus database using the affiliation field and entering the name of each institution included in the study. abbreviations from a sample of institutions were tested to ensure they would link to the full names of the institutions. if multiple variations of the same institution were displayed in the results of the institutional names, all relevant versions were selected to provide the total number of publications. the search results were limited by “document type” to articles or review, to retrieve the number of journal article publications published in 2016, 2017, and 2018 at each institution.   references data   as another measure of potential use of library collections, the total number of references used in the publications studied were also obtained. for each list of institutional publications presented in scopus, the number of total references included in these publications were obtained by clicking all the publications in a list and selecting “view references.” the scopus system limits reference lists to a maximum of 2000. when institutions had more than 2000 publications, results were grouped using some of the limiting features of the system to obtain numbers for the full set. the total number of references included in the institutional publications was also entered into the spreadsheet. because the number of references was not displayed per article but per set of articles, it is possible an article was cited multiple times, but would only be displayed once in the list of references. thus, this data may underreport the number of references included in the studied articles when references to articles were cited by multiple articles within a set of publications.   as with other data included in the study, the average number of publications and the average number of total references were obtained by averaging the three years of data. because of the time delay between writing a manuscript and it being published, it is likely that much of the literature searching and use of the library is done in the year previous to an article being published. therefore, arl data from 2015 to 2017 was collected while publication data was obtained from 2016 to 2018 to better approximate and coincide with potential library usage. for example, while it is not a perfect assumption, given the time to write and publish an article, if an article was published in 2018, there is logic in assuming that in many cases the literature review and use of library collections occurred in 2017, or potentially earlier.   since the total number of references included in all publications at an institution is impacted by the number of articles written, the average number of references per article was obtained by dividing the total number of references included in the publications (all three years) by the total number of publications (all three years).   research expenditures   research and development expenditures for each institution for 2015-2017 were obtained from the herd, where universities report research expenditure and sources of revenue (national science foundation, 2017). the data were entered into the spreadsheet and the average over the three years was calculated.   it was not always possible to match up institutional data from the four data sources. for example, some universities have multiple locations and it was not always clear if data sets covered all locations or a specific location. in other situations, it appeared that medical colleges’ libraries often had separate budgets, and data might not have included data from the full institution. in other cases, full data was not available for all the years. in situations where the limits of the data were not clear or if data were missing (except for collections use data), the institution was dropped from the study. as a result, 81 institutions remained in the study. table 1 summarizes the average numbers of all the variables used in the analysis to demonstrate the overall data patterns of a total of 81 arl member libraries.   data analysis and research questions   all data were analyzed using spss 26. multiple statistical tests were employed to examine direct and indirect impacts of the library on faculty productivity. the analyses included partial correlations and hierarchical multiple regression. partial correlation is a measure of strength and direction of the linear relationship between two variables, while controlling for the effect of one or more variables (covariates). partial correlation allows finding a unique relationship between two variables while eliminating the influence of a third variable, which may drive the relationship. hierarchical multiple regression, a form of multiple regression in which independent variables are entered into the regression “in the order specified by the researcher based on the theoretical grounds” (pallant, 2012, p.149), is used to predict the value of one dependent variable after controlling for another, in this case, for faculty size. as shown in the literature, there are several factors that influence faculty productivity. as part of the institutional factors, faculty size (e.g., dunbar & lewis, 1998) and overall wealth (e.g., rushton & melzer, 1981) were found to be linked to research productivity. among library factors, library expenditure is well known to correlate with publications (dunbar & lewis, 1998; rawls, 2015). however, library expenditures is a broad category and contains expenses beyond just materials (operating budgets, salaries). to compare our findings to past studies, we also compared library expenditures data to productivity. in addition, we explored narrower categories of library expenditures including the overall materials budgets and ongoing library resource expenditures. moreover, our current study expanded on the previous literature by examining whether factors related to library resource use (full-text article requests, database searches) contribute to faculty productivity.     table 1 descriptive statistics for all variables (institution size and expenditures, library budgets, collection size, collection use, and research productivity) n minimum maximum m sd institution demographics total full-time students 81 6,253 69,939 25,285 11,674 total faculty 81 659 4,481 1,792 821 institutional expenditures herd 81 38,244 71,840,290 1,381,179 7,933,921 library budgets total library expenditures 81 10,349,703 116,533,712 31,751,291 17,713,781 total library materials expenditures 81 4,606,644 47,791,377 14,339,988 7,292,618 ongoing resource expenditures 80 3,865,090 20,754,521 10,230,944 3,494,542 size of library collections volumes held 81 1,941,116 20,837,233 5,849,571 3,433,706 titles held 81 970,064 14,863,477 4,653,645 2,583,329 ebooks 81 134,801 3,291,347 1,205,576 570,905 use of the collection full-text article requests 75 192,686 12,752,344 4,100,529 2,779,468 regular searches 73 636,732 7,8174,661 9,303,921 12,823,470 total number of references in the publications 81 20,430 861,817 195,173 131,808  number of references per article 81 40 51 46 2 research productivity  total publications 81 458 19,171 4,306 2,976     table 2 statistical tests statistical test variables research question partial correlations     collection use and research productivity ·        successful full-text article requests, number of regular database searches, number of references included in publications, average number of references per publication (ivs) ·        total number of articles published (dv) ·        total number of full-time students and faculty, research and development expenditures (herd), total materials expenditures (covariates) holding number of full-time students, faculty, total materials expenditures, and research and development expenditures constant, what is the relationship between use of the collection and total number of articles published in an institution? library budgets and research productivity ·        total materials expenditures; total library expenditures; ongoing resource expenditures (ivs) ·        total number of articles published (dv) ·        total number of full-time students and faculty, research and development expenditures (covariates) holding number of full-time students, faculty and research and development expenditures constant, what is the relationship between library budgets and total number of articles published in an institution? collection size and research productivity ·        volumes held, titles held; ebooks (ivs) ·        total number of articles published (dv) ·        total number of full-time students and faculty, research and development expenditures (covariates) holding number of full-time students, faculty, and research and development expenditures constant, what is the relationship between library collection size and total number of articles published in an institution? research expenditures and research productivity ·        research and development expenditures (iv) ·        total number of articles published (dv) ·        total number of full-time students and faculty, total library expenditures, total materials expenditures, ongoing resource expenditure (covariates) holding number of full-time students, faculty, total library expenditures, total materials expenditures, and ongoing resource expenditures constant, what is the relationship between research and development expenditures and total number of articles published in an institution? library budgets and research expenditures   ·        total library expenditures, total materials expenditures, ongoing resource expenditure (ivs) ·        research and development expenditures (herd) (dv) ·        total number of full-time students and faculty (covariates) holding number of full-time students and faculty, what is the relationship between research and development expenditures and library expenditures? hierarchical multiple regression         ·        total number of faculty, research and development expenditures, library material expenditure and full-text article requests (ivs) ·        total number of publications from 2016 to 2018 (dv) controlling for the possible effect of total number of faculty, is the set of variables (research and development expenditures, library materials expenditures and full-text article requests) still able to predict a significant amount of the variance in total number of publications?     table 2 outlines which tests were used to address the research questions. before running the statistical analyses, tests of assumptions were run to confirm it was appropriate to run the proposed analysis including the possibility of multicollinearity using cut-off points for tolerance value of less than .10 or vif value of above 10 guided by pallant (2010). there is no violation of the multicollinearity assumption.   results   partial correlations among collection use, budgets, collections, research expenditures, and research productivity   results from partial correlations are displayed in table 3. as the number of faculty and students, amount of research and development expenditures, and library materials expenditures were likely to influence the number of publications and use of the collection, their contribution to the relationship was eliminated through partial correlation. the first partial correlation explored collection use and research productivity. it revealed a moderate positive relationship between successful full-text article requests and the number of articles published at an institution, r (69) = .504, p < .001, suggesting higher use of the library (successful full-text article requests) is associated with research productivity. not surprisingly, there was a strong positive correlation between the total number of references (average over 3 years) included in the publications and the total number of publications per institution, controlling for the number of full-time students and faculty, total library materials expenditures, and research and development expenditures, r (75) = .994, p < .001. one would expect as the number of total publications increased, so would the number of references included in those publications. however, when the average number of references per article was compared to the total number of publications, there was a weak negative correlation, r (75) = -.279, p = .014. this suggests the more references used, the fewer publications or, the more articles published, fewer references will be included. there was not a significant correlation found between the number of publications and the number of regular database searches, r (67) = -.200, p = .100.   the second set of partial correlation analyses explored library expenditures and research productivity. these analyses demonstrated a strong positive relationship between articles published at an institution and total library materials expenditures (r (76) =.661, p < .001) and total library expenditures (r (76) =.748, p < .001), but only a moderate positive correlation between articles published at an institution and ongoing resource expenditures (r (75) =.551, p < .001). the higher the expenditures allocated for a library, the higher the numbers of publications were produced at an institution. this demonstrates a significant relationship between library expenditures and research productivity.     table 3 partial correlations among collection use, library budgets, collection size, research productivity, and research expendituresa   df r p collection use and research productivity       article requests 69 .504 < .001* database searches 67 -.200 .100 references included in publication 75 .994     < .001* average number of references per publication 75 -.279 .014 * library budgets and research productivity       ongoing resources expenditures 75 .551 < .001* materials expenditures 76 .661 < .001* library expenditures 76 .748 < .001* collection size and research productivity       volumes held 76 .708 < .001* titles held 76 .646 < .001* ebooks 76 .282 .012* research expenditures and research productivity       research and development expenditures 73 .323 .005* library budgets and research expenditures       ongoing resource expenditures 76 .294 .009* material expenditures 77 .135 .236 library expenditures 77 .126 .269 a strength of correlations as indicated by dancey and reidy (2011) for absolute value of r – |.10| < r < |.30| weak, |.40| < r < |.60| moderate, |.70| < r < |.90| strong * indicates significant p value.     the partial correlations exploring collection size and research productivity suggested that articles published at an institution correlated positively with the size of library collections (volumes held, r (76) = .708, p < .001; titles held, r (76) = .646, p < .001; and ebooks, r (76) = .282, p = .012). this means the greater the size of a library’s collection (volumes, titles and ebooks), the greater the number of publications produced at the institution. however, the strength of correlations for ebooks were weak r < .30, as dancey and reidy (2011) previously found.   the partial correlation revealed a positive moderate relationship between research development expenditures and the number of articles published at an institution, controlling for the number of full time students and faculty, total library expenditures, total library materials expenditures, and ongoing resource expenditures, r (73) = .323, p = .005, suggesting the higher the research and development expenditures obtained by an institution, the higher the number of publications produced at the institution.   the last partial correlation suggested that the amount of research and development expenditures correlated positively with ongoing resource expenditures (r (76) = .294, p = .009); however, the strength of the correlation is weak r < .30. this means the greater the amount of ongoing resource expenditures, the greater the amount of research and development expenditures. there was not a significant correlation found between other library expenditures and the amount of research and development expenditures (total materials expenditures, r (77) = .135, p = .236; total library expenditures, r (77) = .126, p = .269).   predicting research productivity (publications)   next, a four-stage hierarchical multiple regression was conducted to examine the degree to which research and development expenditures and the library related collection measures (library materials expenditures and full-text article requests) affected research productivity, after controlling for the influence of the institutional size (total number of faculty). the sample size of 81 was considered adequate given four independent variables subjected to the analysis: a ratio of 15 cases for every independent variable (tabachnick & fidell, 2001). total number of faculty was entered at stage one of the regression to control for faculty size. the research and development expenditures variable was included at stage two; the library materials expenditures variable was entered at stage three, and full-text article requests was entered at stage four. institutional funding (research and development expenditures), library related variables (i.e., library materials expenditures and full-text article requests) were entered in this order because research and development expenditures represents the institutional funding size, library materials expenditures represents library collections, and library usage (full-text article requests) is followed by it. library collection size was not included in this analysis because collection size is highly correlated with library materials expenditures. the reason for choosing full-text article requests, rather than the number of regular database searches at stage four, is that this variable has a higher correlation with publications in the partial correlation described above. we did not use total library expenditures as the variable here because it contains expenditures beyond the collection, such as salaries and operational expenses. we wanted to explore the unique contribution of the collections expenditures, as reflected through the use of materials expenditures.   table 4 indicates the significance of each of the four anova models. while all four models were significant at p < .001, the f value was largest for model 4 with four predictors (total number of faculty, research and development expenditures, library materials expenditures, and full-text article requests), meaning that model 4 as a whole is the most significant (f (4, 70) = 45.932, p <.001) as a predictive model.   the hierarchical multiple regression showed that in model 1, the total number of faculty contributed significantly to the regression model, f (1,73) = 23.063, p < .001 and accounted for 24% of the variation in publications (table 4). in model 2, research and development expenditures explained an additional 9.8% of the variation in publications, after controlling for total number of faculty; this change in r² was significant, f change (1,72) = 10.676, p = .002. after introducing library materials expenditures in model 3, the total variance explained by the model as a whole (which includes faculty, research and development expenditures and library materials expenditures) was 62.2%, f (3,71) =38.911, p < .001. the library materials expenditures explained an additional 28.4% of the variance in publications, after controlling for total number of faculty and research and development expenditures. the change in r² was highly significant, f change (1,71) = 53.239, p < .001. this result clearly showed that library materials expenditures contributed significantly to the total number of publications. finally, including full-text article requests in the fourth and final model explained an additional 10.2% of the variation in publications, after controlling for total number of faculty, research and development expenditures, and library materials expenditures. this change in r² was also significant, f change (1,70) = 25.960, p = .001, indicating that full-text article requests has a significant effect on the publications. when all four independent variables were included in stage four of the regression model, the total number of faculty was not a significant predictor of publications. as shown in table 5, the best predictor of publications was full-text downloads (β =.509), which uniquely explained 10.2% of the variation in publications. in order of the next important predictors of publications they were library materials expenditures (β =.341) and research and development expenditures (β =.150). together the four independent variables accounted for 72.4% of the variance in publications.     table 4 anova results of the four model-hierarchical regression analysisa     sum of squares df mean square f model 1b regression 164658876.662 1 164658876.662 23.063*   residual 521175432.352 73 7139389.484     total 685834309.014 74     model 2c regression 231958134.313 2 115979067.157 18.398*   residual 453876174.701 72 6303835.76     total 685834309.014 74     model 3d regression 426452545.391 3 142150848.464 38.911*   residual 259381763.622 71 3653264.276     total 685834309.014 74     model 4e regression 496623237.92 4 124155809.48 45.932*   residual 189211071.094 70 2703015.301     total 685834309.014 74     a dependent variable: total number of publications b predictors: total number of faculty c predictors: total number of faculty, research and development expenditures d predictors: total number of faculty, research and development expenditures, library materials expenditures e predictors: total number of faculty, research and development expenditures, library materials expenditures, full-text article requests * indicates p is significant at < .001.     table 5 summary of hierarchical regression analysis for variables predicting publicationsa   β t r2 dr2 f df model 1     .240 .240 23.063*** 23.063*** total number of faculty .490 4.802***         model 2     .338 .098 18.398*** 10.676** total number of faculty .506 5.269***          herd .314 3.267**         model 3     .622 .284 38.911*** 53.239*** total number of faculty .173 2.377*          herd .227 3.514**         library materials expenditures .595 7.296***         model 4     .724 .102 45.932*** 25.960** total number of faculty .052 .636         herd .150 2.284*         library materials expenditures .341 3.965***          full-text article requests .509 5.095***         a dependent variable: total number of publications.  * p < .05. ** p < .01. *** p < .001.     discussion   the purpose of this study was to obtain information concerning the relationship between the use of library collections and research productivity in the electronic era. the findings illustrated the strength of this relationship and document the contributions that today’s academic library has on an institution’s research success. because previous literature exploring quantitative library metrics with research productivity is limited and older, this study also bridges a gap.   like previous work (rushton & melzer, 1981), this study found a correlation between research productivity and library expenditures, collections held, and research and development expenditures. as rushton and melzer concluded, the overall wealth of an institution likely contributes to faculty productivity because a strong infrastructure of support is likely to be in place. more research is needed to better understand and to uncover underlying factors. similar to rawls’ (2015) exploration of the arl data from 2005 to 2009, this study of 2015 to 2018 data also found productivity was positively correlated with library expenditures. we also found that total materials expenditures and ongoing library resource expenditures were also correlated, but not as strongly as total library expenditures. distinct from studies of the past, this study examined usage data and found a more direct link between use of the collection (full-text article requests) and research productivity. the greater the research productivity (journal article publications), the greater the use of the library’s collection, as demonstrated through full-text article requests.   based on the findings from the partial correlations and literature review, we further examined if a set of variables (research and development expenditures, library materials expenditures, and full-text article requests) were still able to predict a significant amount of the variance in total number of publications after controlling for the possible effect of total number of faculty. the primary findings from the hierarchical multiple regression analysis was that full-text article requests were found to be the strongest predictor of research productivity as measured by articles published, followed by the library material expenditures and research expenditures. even when controlling for the total number of the faculty, research expenditures, and library materials expenditures, full-text article requests uniquely explained 10.2% of the variation in publications. these findings provide strong evidence that funding libraries supports faculty research success. the findings demonstrated not just that an investment in library collections correlated with productivity, but that the use of the library collections positively contributed to faculty productivity. given the cyclical nature of research, faculty productivity likely leads to further faculty success, through additional research development and expenditures. libraries can use this information to communicate the library’s impact on faculty productivity with various stakeholders. libraries should also explore their faculty’s research agendas and the use of the journal collection through full-text article requests, to assist with future collection development decisions so they are in line with the needs of the faculty. examining both existing metrics (e.g., collection size or collection expenditures) and new metrics (e.g., the use of the collection) in the current study expands on the existing literature and confirms that the use of library collections has a great impact on research productivity.   one of the unexpected findings of this study is that the number of average references used per publication decreased the more productive an institution was. one speculation for this is that the more productive faculty are, the less likely they may be to search broadly for articles. it could also be that as experts in their field, they are able to be more selective in the publications that they choose to cite to address their findings. alternatively, it might be the case that as productivity increases, the articles produced are more narrowly focused or cutting edge, and fewer relevant resources are available for citing. further exploration at the author level is needed to understand this finding.   limitations   there were some limitations to the data collected. in this study, we largely explored research productivity as it related to journal articles, and library usage as it related to journal article usage (database searches, full-text article requests, and number of references in journal articles). therefore, disciplines that do not produce journal articles or are not reliant upon them for research are excluded from this study. although scopus is the most comprehensive journal literature database, it does not index all journals, nor books or book chapters. thus, the publication and reference data obtained from scopus was limited to journal publications indexed in scopus. the above-mentioned factors tend to bias the data toward those disciplines (e.g., stem, social sciences) whose research is reported primarily through the journal literature. in addition, it was not possible to limit the examination of the data collected in this study to specific disciplines. for example, arl annual library statistics are reported in aggregate for each academic institution, although a broad category of health sciences is available. with respect to disciplines and publications, only the institutional affiliation is indexed using standardized terminology within the scopus database, thus making it difficult to retrieve comprehensive publication data from a college or department. this means the findings of this study will apply broadly to institutions but will not provide insight into correlations or relationships within specific disciplines.   the “number of full-text article requests” and the “number of regular searches” were obtained from vendors that provide “counter” statistics. counter statistics were developed to provide consistent and credible data regarding the usage of databases and journals. however, not all vendors provide this data, so the numbers provided to arl from each institution were likely not complete. this study was also reliant upon the accuracy of the reported survey data collected and used in the study (arl annual library statistics, herd); while institutions attempt to report the most accurate information, there is always the potential for error or incomplete data reporting.   conclusion   as found in past studies, research productivity correlated positively with library expenditures. we also found that the use of the collection had a relationship with research productivity. even more important, full-text article downloads uniquely explained approximately 10% of variation in research productivity, over and above other factors including research and development expenditures and library expenditures. full-text article downloads were a better predictor of research productivity than research and development expenditures or library expenditures. this finding suggests that the use of collections has more impact on the articles published than the total collections dollars libraries spend. collections developed to fit with the current research agendas of faculty may impact their productivity. this finding may support library decisions surrounding expenditures and future selections of resources related to research support. this may also be important information for academic libraries at other carnegie levels that are building support for their research programs.   references   association of research libraries. (2016). arl statistics [data file].  retrieved from https://www.arlstatistics.org/analytics                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       carnegie classification (2016). custom listings – basic classification: carnegie classification of institutions of higher education [data file]. retrieved from http://carnegieclassifications.iu.edu/   cluff, e. d., & murrah, d. j. (1987). the influence of library resources on faculty recruitment and retention. journal of academic librarianship, 13(1), 19-23.   dancey, c. p., & reidy, j. (2011). statistics without maths for psychology (5th ed.). prentice hall/pearson.   dundar, h., & lewis, d. r. (1998). determinants of research productivity in higher education. research in higher education, 39(6), 607-631.   kaufman, p. t. (2008). the library as strategic investment: results of the illinois return on investment study. liber quarterly, 18(3-4), 424-436. https://doi.org/10.18352/lq.7941    laakso m., & bjork b. c. (2012). anatomy of open access publishing: a study of longitudinal development and internal structure. bmc medicine, 10(124). https://doi.org/10.1186/1741-7015-10-124   murray, a., & ireland, a. (2018). provosts' perceptions of academic library value & preferences for communication: a national study. college & research libraries, 79(3), 336-365. https://doi.org/10.5860/crl.79.3.336    national science foundation. (2016). higher education research development survey (herd) [data file]. retrieved from https://www.nsf.gov/statistics/srvyherd/   pallant, j. f. (2011). spss survival manual: a step by step guide to data analysis using the spss program (4th ed.). allen & unwin.   rawls, m. m. (2015). looking for links: how faculty research productivity correlates with library investment and why electronic library materials matter most. evidence based library and information practice, 10(2), 34-44. https://doi.org/10.18438/b89c70   rushton, j. p., & meltzer, s. (1981). research productivity, university revenue, and scholarly impact (citations) of 169 british, canadian and united states universities (1977). scientometrics, 3(4), 275-303. https://doi.org/10.1007/bf02021122   sutter, w. n. (2012). introduction to educational research: a critical thinking approach (2nd ed.). sage publications.   tabachnick, b. g., & fidell, l. s. (2001). using multivariate statistics (4th ed.). boston, ma: allyn and bacon.   tenopir, c., king, d., edwards, s., & wu, l. (2009). electronic journals and changes in scholarly article seeking and reading patterns. aslib proceedings: new information perspectives, 61(1), 5–32. https://doi.org/10.1108/00012530910932267   tenopir, c., king, d. w., spencer, j., & wu, l. (2009). variations in article seeking and reading patterns of academics: what makes a difference? library & information science research, 31(3), 139-148. https://doi.org/10.1016/j.lisr.2009.02.002   tenopir, c., mays, r., & wu, l. (2011). journal article growth and reading patterns. new review of information networking, 16(1), 4-22. https://doi.org/10.1080/13614576.2011.566796           evidence based library and information practice 2009, 4:2 182 evidence based library and information practice ebl 101 looking to the literature: domains to help determine where to look virginia wilson shirp coordinator, health sciences library university of saskatchewan saskatoon, saskatchewan, canada e-mail: virginia.wilson@usask.ca received: 26 april 2009 accepted: 26 april 2009 © 2009 wilson. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. last time in ebl 101, we looked at matching question types to study designs. the process of determining the type of question you’ve asked can point you in the direction of the research design best suited to answering your question. this time around, we’ll stay with the question in order to help determine where to look for evidence in the literature. it’s been suggested, and further confirmed through research, that many if not all questions generated in librarianship fall under one or more of the following six domains: 1. collections 2. education 3. information access and retrieval 4. management 5. professional issues 6. reference/enquiries this classification scheme was originally proposed to deal with published research in order to make it ‚easier to map sources, study types and search strategies pertaining to each domain‛ (koufogiannakis, slater, and crumley 228). deciding in which domain your question belongs is helpful in the ebl process. the domain can point you to the type of literature likely to contain articles pertaining to your topic. as an aside, there is a bit of discrepancy around the domains in some ebl literature that was published around 2004. for example, in booth and brice’s evidence-based practice for information professionals: a handbook, reference to the six domains includes marketing as a separate domain. koufogiannakis, slater, and crumley, in their content analysis of librarianship research, tested the six initially proposed domains, and found that evidence for marketing research was limited and removed it as a separate domain from the list, including it instead in the management domain. additionally, based on the results of their content analysis, they added a domain: professional issues. so 2004 saw the move from mailto:virginia.wilson@usask.ca evidence based library and information practice 2009, 4:2 183 the initially proposed classification taxonomy to the six domains listed above. it’s time to take your question and find out in which domain it belongs. each domain has its own definition: collections: building a high-quality collection of print and electronic materials that is useful, cost-effective and meets the users’ needs. education: incorporating teaching methods and strategies to educate users about library resources and how to improve research skills. lis education is a subset of the education domain, and it specifically pertains to the professional education of librarians. information access and retrieval: creating better systems and methods for information retrieval and access. management: managing people and resources within an organization. this includes marketing and promotion as well as human resources. professional issues: exploring issues that affect librarianship as a profession. reference/enquiries: providing service and information access that meets the needs of library users. (koufogiannakis, slater, and crumley, table 2) now that you have more information about your question, it’s time to turn to the literature. the challenging part of evidence-based librarianship is that not only should you look in the library and information studies (lis) literature, but also further afield in disciplines that relate to your question’s domain(s). the starting point is usually the traditional lis databases, in addition to the suggestions below. 1. collections: collection management as a domain is almost completely covered within the library literature. in this case, the library literature includes ‚the wider scope offered by museum and archival activities‛ (booth, collection, 187). 2. education: research studies about education can be found in several places: educational resources information center (eric), psycinfo, and the campbell collaboration. however, brice and carlson report that ‚much education research sits in the ‘grey literature’‛ (168). 3. information access & retrieval: in the last few decades, more research into digital libraries has been found in the information retrieval research. additionally, information science and computing literature also looks at information retrieval and access. it’s been noted that there is not a lot of cross over between public and commercial sectors and the academic sector. thus, ‚the wheel is being reinvented‛ (booth, evidence, 233). therefore, there should be more impetus to search broadly across different disciplines. 4. management: informative research on a variety of management topics can be found in the harvard business review, as well as in general management databases such as abi-inform; in sectorspecific management databases and collection, i.e. health business elite; and in resources covering specific facets of management, i.e. marketing, hr, etc. (booth, towards, 201). 5. professional issues: this is a wideranging domain that explores issues pertaining to librarianship as a profession. the traditional lis databases will most likely be the only stop to make. however, depending on the question, literature from other professions could be consulted. 6. reference/enquiries: questions that fall under the reference domain can also be explored in literature pertaining to ‚the wider subject discipline of human interaction and communication‛ which also includes ‚research studies from evidence based library and information practice 2009, 4:2 184 psychology (psycinfo), computer science (inspec), and other behavioural and social sciences‛ (booth, examining, 150). it is important to think outside the lis box when looking for evidence to inform your question. thinking about the question in terms of domains can help you do this and in turn broaden the range of useful resources. but what happens if there is no access to for-fee databases? next time: open access sources of lis evidence. works cited koufogiannakis, denise, linda slater, and ellen crumley. ‚a content analysis of librarianship research,‛ journal of information science 30.3 (2004): 227-39. brice, anne and cindy carson. the contribution of evidence-based practice to educational activities. evidence based practice: an information professionals handbook. eds. andrew booth and anne brice. london: facet, 2004.164-77. booth, andrew. an evidence-based approach to collection management. evidence based practice: an information professionals handbook. eds. andrew booth and anne brice. london: facet, 2004. 185-95. . evidence-based perspectives on information access and retrieval. evidence based practice: an information professionals handbook. eds. andrew booth and anne brice. london: facet, 2004. 231-46. . examining the evidence base for reference services and enquiry work. evidence based practice: an information professionals handbook. eds. andrew booth and anne brice. london: facet, 2004. 148-58. . towards evidence-based management. evidence based practice: an information professionals handbook. eds. andrew booth and anne brice. london: facet, 2004. 200-09. evidence based library and information practice evidence based library and information practice 2010, 5.1  evidence based library and information practice       evidence summary    the lis blogosphere contains tags that can be categorized and it disseminates  professional content    a review of:  aharony, n. (2009). librarians and information scientists in the blogosphere: an exploratory analysis.  library & information science research, 31(3), 174‐181.     reviewed by:  virginia wilson  shirp coordinator  health sciences library, university of saskatchewan    saskatoon, saskatchewan, canada    email: virginia.wilson@usask.ca    received: 29 nov. 2009          accepted: 12 jan. 2010         2010 wilson. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which  permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly  attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the  same or similar license to this one.    abstract    objective – this study analyzes library and  information studies (lis) oriented blogs to  determine the content, and looks at tags and  folksonomies of these blogs to determine  whether they form a consistent, coherent  scheme or whether they are lacking in internal  logic.    design – a qualitative content analysis of tags  assigned to 30 lis blogs.    setting – the research took place on the  internet from may to july, 2008.    subjects – thirty lis blogs were examined,  each of which was written by a librarian or an  information scientist.  methods – the researcher reviewed 100 blogs  that were found by browsing the top 25  librarian bloggers as published by the online  education database in 2007 and by searching  technorati, one of the main search engines for  blogs, using the term “library and information  science.” thirty blogs were chosen for analysis  based on two criteria: the blog had to be  written by a librarian or an information  scientist, and the blog had to be active during  the period studied (may‐july, 2008).     a content analysis was undertaken on the tags  assigned to the 30 blogs by categorizing the  tags that appeared as tag clouds (visual  representations of user‐generated tags in  which the tags used more frequently are  depicted in larger, bolder font) in technorati.  in order to validate the technorati tags, the  138 mailto:virginia.wilson@usask.ca evidence based library and information practice 2010, 5.1  researcher’s coders read and analyzed all the  blog posts over the given time period. the  categorization consists of five major  categories, each with several subcategories.  the categories were developed using a  clustering approach, with new categories  coming into being when a tag did not fit into  an already established category.    main results – the tag categorization  resulted in five broad categories, each with  several sub‐categories (a few of which are  listed here):  1. general (nouns, disciplines, place  names)  2. library‐related (web 2.0, librarians’  activities, catalogues)  3. technology‐related products,  technology – types, people)  4. information‐related (access to  information, information sources)  5. social web‐related (names of blogs,  names of social networks)    the tag analysis resulted in the following  percentages of distribution:  • 33.62% of the tags associated with lis  blogs were general in nature  • 20.21% of the tags were technology‐ related  • 19.12% of the tags were library‐related  • 14.60% of the tags were information‐ related  • 12.90% of the tags were related to the  social web  these percentages add up to 100.45%. the  author makes no mention of this oddity and it  is assumed to be an error.    the researcher attempted to determine if tags  and folksonomies form a consistent scheme. in  reporting her findings, she concluded that four  major categories of professional‐related  content were revealed, which reflect the  blogger‐librarians’ fields of interest. the  prominence of the general category revealed  that bloggers’ personal interests and  experiences were written about more often. as  well, it appears that although bloggers seem to  assign non‐related tags randomly, the analysis  shows that tags still can be categorized.   conclusion – the researcher concludes that  this study is helpful for librarians and  information scientists because it can help them  to navigate the lis blogosphere. she reports  that the categories of tags beyond the general  category, which mainly contains tags related  to bloggers’ personal interests and  experiences, shows that blogs can contribute  to professional development. although more  informal in nature, the research has shown  that lis blogs do contain professional  information, and it behooves professionals to  become familiar with the tag scheme in topic  oriented blogs, and to try to work within the  scheme to make use of the content within. the  researcher suggests further ideas for research,  including the differences in lis blogs written  by a single blogger as compared with blogs  written by multiple authors, as well as gender  differences between male and female authored  blogs. the author also suggests further  research on multimedia blogs such as  photoblogs, and audio and video blogs.      commentary    an analysis of lis blogs for content is an  interesting concept. the categorization of the  tag clouds by using grounded theory and  content analysis came up with some fruitful  results. that the “general” category was the  most prevalent suggested that blogs are  indeed the casual information dissemination  medium that they appear to be. additionally,  the four other categories, along with the sub‐ categories, which relate to professional  practice, illustrate that content that falls within  these categories is being disseminated in the  lis blogosphere. one drawback is that other  than listing the sub‐categories and examples of  tags which fall under each, the researcher does  not explicitly define the tag categories. the  article contains a very thorough literature  review broken down by various topics  pertinent to the research.    the researcher uses tables to outline the  categories and their sub‐categories. it would  perhaps have been helpful to also include this  information in the narrative portion of the  139 evidence based library and information practice 2010, 5.1  140 article in order to more directly associate the  tag categories with the results given.    the section on main results (“findings”) could  have been more robust, as there are gaps in  the details that would be helpful to know. the  author mentions coders in the data analysis  section, yet there is no mention of how many  coders were involved in the research or who  the coders were. as well, there were no details  given about the analysis of each blog post  undertaken by the coders in order to confirm  the technorati tags. the method section  (“procedures”) would have been enhanced by  greater detail on this aspect of the research.     in the data analysis section of “procedures,”  the researcher discusses how she chose the  tags to analyze, stating that not all the blogs  she chose to analyze provided tags, so she  used technorati tags, “which are given to  technorati by bloggers” (p. 176). it is unclear  what this means. does the researcher mean  that the tags are some kind of blogging  controlled vocabulary? if the reader is  unfamiliar with the way in which technorati  works, this section will prove to be confusing.  more detail about technorati and the  implications of its tagging system and how it  relates to this research would have been  helpful.   the discussion contains a lengthy comparison  of this research study with a similar study the  author did on 30 educational‐technology  oriented blogs. the author does not mention  that she will be undertaking this comparison,  and its sudden appearance could give the  impression that the researcher feels that the  current study perhaps does not have enough  substance to stand on its own.     this research is useful for those who are  interested in and want more information  about the lis blogosphere, and who might  want to contribute their own content to the  general knowledge‐ base via a blog. it is also  useful for bloggers who want to think more  carefully about their tags and folksonomies.  perhaps the categories and sub‐categories  developed during this research can provide  bloggers with an idea of how to structure their  own folksonomies. the ideas for further  research are intriguing, such as focusing on  multimedia modes of blogging, looking at  blogs written by single authors as opposed to  multiple authors, and looking at blogs written  by males and females. one gets the feeling  that more work is in the offing from this  particular researcher.     evidence based library and information practice evidence based library and information practice 2010, 5.3 75 evidence based library and information practice news/announcements call for abstracts – 6th international evidence based library and information practice conference 2010. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. eblip6: valuing knowledge and expertise university of salford, salford, uk 27th-30th june 2011 the international programme committee for the 6th international evidence based library and information practice (eblip6) conference invites you to submit oral presentations and posters for the conference 'eblip6: valuing knowledge and expertise' to be held at the university of salford, greater manchester, uk from june 27-30, 2011. submission is by a form which can be downloaded from the conference website at: http://www.eblip6.salford.ac.uk/abstracts.php and emailing it to <eblip6@salford.ac.uk> by december 3, 2010. conference themes eblip6 is aimed at all library and information professionals, whatever the discipline, sectors, role or stage in their career. the themes have been selected to reflect this and build on the success of past conferences, providing an opportunity for a wide range of practitioners and researchers to present their work. previous conferences have provided a supportive environment to those presenting or embarking on evaluation or research for the first time and eblip6 is set to continue this approach and will provide stimulating presentations from those with a more established portfolio. research papers, reviews of evidence, reports of innovative practice, hot topic discussion papers and poster presentations are invited. key themes include, but are not restricted to: evidence based practice: reflection  reflective practice and its relationship to eblip  reflection on eblip  questioning and adapting the ebp model for the specific needs of eblip evidence based practice: impact and value  methods of demonstrating impact and value within an evidence based context  the use of eblip to demonstrate value or impact  practising eblip in times of financial uncertainty  maximising the value of the individual case study in library research  outcome indicators: what must we measure, what should we measure and what could we measure evidence based library and information practice 2010, 5.3 76 evidence based practice: practicality and applicability?  the practicality of using the eblip model  is eblip applicable in your setting  practical methods for eblip  applicable methods for eblip  implementing evidence based practice into systems or culture evidence based practice: innovation, education and research  eblip in the lis curriculum  web 2.0 as a vehicle for enabling eblip  evaluating web 2.0 using the eblip model  the inter-relation of innovation, good practice and eblip  relationship between eblip and research important dates  submission of abstract deadline: 3rd december 2010  notification of acceptance deadline: 17th january 2011  confirmation of participation deadline: 18th february 2011  registration opens: 1st march 2011 register your interest demand is expected to be high so to ensure you remain up-to-date on all eblip6 news, register your interest on the eblip6 web site: http://www.eblip6.salford.ac.uk/register.php you can also join us on facebook <http://tiny.cc/3n3lb> and follow us on twitter <http://twitter.com/eblip2011>. alison brettle and maria j grant eblip6 conference chairs generation 1.5 and academic libraries: strategies for supporting english learners (els) in reference and instruction review article   generation 1.5 and academic libraries: strategies for supporting english learners (els) in reference and instruction   megan margino marchese reference and instruction librarian farmingdale state college farmingdale, new york, united states email: megan.marchese@farmingdale.edu   received: 23 aug. 2021                                                             accepted: 4 nov. 2021        2021 marchese. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   doi: 10.18438/eblip30023     abstract   background – compared to native english speakers, english learners (els) often face additional barriers to academic success. though typically competent in social english, generation 1.5 els struggle with academic english at the postsecondary level and are still considered to be in the process of learning english. as colleges become increasingly linguistically diverse, academic librarians must adapt to support the growing numbers of els in the campus community.   objective – this paper aims to provide academic librarians with information on the scope of english learners in k-12 through postsecondary education, academic challenges of generation 1.5 students at the postsecondary level, and strategies that librarians can employ to support english learners in the contexts of reference and instruction.   methods – the author searched journals in the disciplines of academic libraries, higher education, teaching english to speakers of other languages (tesol), and linguistics. additional resources searched include education data and statistics, research institute publications, and english as a new language (enl) teaching resources. these sources were explored in regard to the topics of el educational statistics, k-12 enl programs, enl pedagogy, els in postsecondary education, generation 1.5 students, els’ academic challenges and educational needs, and academic libraries and els.   results – a review of the literature on els in academic libraries, particularly generation 1.5 students, reveals that generation 1.5 is a population that is in need of support at the postsecondary level. because generation 1.5 students often hold strong social english skills, they may enter college without an el designation or specialized academic support. however, research shows that generation 1.5 students struggle with college-level academic english, specifically in grammar and vocabulary. these challenges impact students’ communicative success both in college classroom and library environments.   conclusion – academic librarians may adopt pedagogical strategies commonly employed in enl classrooms to use in reference and instruction environments. techniques include themes such as awareness of language use and reinforcement of content, and require low-stakes implementation into library practice. though librarians may be unaware of the language learning needs of their students, such strategies have shown to be useful for all students. because techniques that are helpful to els also typically benefit all students, these strategies are also applicable to native english speakers.     introduction   despite growing numbers of english learner (el) students across the united states, academic librarians are often unaware of best practices in working with students whose native language is not english (tran & aytac, 2018). across the united states, 10.2% of public school k-12 students are enrolled as english learners (els). in most areas of the country, the number of el students is increasing each year (national center for education statistics, 2020), and the national education association (2020) projects that 25% of k-12 students throughout the united states will be els by 2025. these statistics suggest that institutions of higher education will also be impacted by the continued growth of the language learner population in future years.   in the k-12 environment in the united states, els receive language assistance programs to help them attain english proficiency. in higher education settings, el support is dependent upon the characteristics of the language learner. outside of k-12, bergey et al. (2018) assert that els may be considered in three categories: international students, recent immigrants, and generation 1.5 students. the generation 1.5 designation comprises young people who are u.s.-born children of immigrants, individuals who immigrated to the united states as children, or those who were raised in environments where english is not the primary language of the home and community (huster, 2012). while college-level english as a second language (esl) programs in the united states are often designated for international students and recent immigrants, generation 1.5 els are not typically in language support programs at the college level. this is because generation 1.5 attended k-12 schools in the united states and they are often proficient in social english. however, while generation 1.5 students are competent in conversational communication, they may still struggle with academic english and can be challenged by academic tasks at the college level (asher et al., 2009; bergey et al., 2018; di gennaro, 2008; haras et al., 2008; harklau, 2003; huster, 2012; roessingh & douglas, 2012).   aims   there is currently limited research on best practices for working with generation 1.5 els in academic libraries, particularly concerning strategies for supporting generation 1.5 through reference and instruction practices. this paper aims to apply research conducted in the areas of k-12 el pedagogy and els in higher education to the context of academic libraries. specifically, this paper will explore how academic librarians in reference and instruction settings can employ strategies that are beneficial to els while simultaneously supporting all students.   methods   the author of this paper conducted a literature review in disciplines related to academic libraries, higher education, teaching english to speakers of other languages (tesol), and linguistics. databases searched include ebscohost library, information science & technology abstracts with full text, ebscohost education source, ebscohost academic search complete, proquest education database, proquest research library, and eric. the author also searched the primo discovery service via farmingdale state college, google scholar, and online resources published by enl and esl educators. insight on united states federal data pertaining to els was gathered through the u.s. census bureau, research organizations such as the american institutes for research, the migration policy institute, and united states education associations. consistent terminology was used in searching across platforms through utilizing combinations of the following search terms: english language learners, english learners, english as a new language (enl), english as a second language (esl), generation 1.5, international students, academic libraries, reference, information literacy instruction, higher education, k-12 education, pedagogy, teaching methodology, teaching strategies, teaching techniques, and academic challenges. backward citation searching was used to explore major researchers in the field of generation 1.5, the evolution of the generation 1.5 definition, trends in u.s.-based el pedagogy over time, historical legal protections for els in the united states, and changing el terminology. forward citation searching was also used to discover the most recent available research on these topics.   further, the author’s institution, farmingdale state college, a technology college in the state university of new york (suny) system, acts as an example institution to investigate the availability of postsecondary statistics maintained on non-native english speakers, offerings of language support programs, and general academic support prerequisites. this institution is located on long island, new york, and serves approximately 10,000 students primarily at the undergraduate level, including 47% of students from minority populations largely from long island and the new york city area (farmingdale state college, n.d.-a; farmingdale state college, n.d.-b).   the research process began by exploring journal articles on general strategies for supporting english language learners in academic libraries. through the review process, the term generation 1.5 revealed itself to be both relevant to the diverse composition of the author’s institution and in need of further consideration in regard to academic libraries. at the time of this writing, journal articles that focused on generation 1.5 students in academic libraries were primarily targeted toward students’ perceptions of libraries and their library space usage. a review of the literature revealed a lack of resources with a specific focus on methodology for supporting generation 1.5 els in academic library reference and instruction contexts. the author’s background holding a master of arts in tesol and serving as a reference and instruction librarian lent itself to search a range of resources including journals, reports, and educator-based online publications in the fields of academic libraries, tesol, linguistics, higher education, k-12 education, and united states education data.    resources included as references in this paper provided information in one or more of the following areas: generation 1.5 students and academic libraries; academic librarian el-based pedagogy; generation 1.5 students in higher education; the history of els in united states k-12 education; generation 1.5 students’ academic and linguistic challenges; els’ transition experiences from high school to college; el pedagogical strategies that can be applied in academic library contexts; and united states el statistics.    resources that were examined in the literature review process, but were ultimately excluded from this article include the following categories: research on international students in higher education that does not pertain to generation 1.5 els; united states k-12 teaching pedagogy that is not applicable for academic libraries; technical linguistics research that is not suitable for academic library audiences; el-based research that does not provide practical teaching applications; and new york-specific el statistics. searching the literature through the above methods resulted in the emergence of several themes presented in this paper which are organized to highlight the importance of language learner identification in united states k-12 education, generation 1.5 students’ academic challenges, and el-based pedagogy that is applicable to academic libraries.   results   a review of the literature resulted in the emergence of several major themes surrounding the identification of els in k-12 education compared to that of postsecondary education, and the academic support differences typically received at each level. through exploring els’ academic needs, it is evident that generation 1.5 students are an underserved population that is often challenged by academic language at the college level, despite their appearance of social language proficiency. these academic challenges include all forms of academic communication, such as grammar and vocabulary, which impact students’ success in the college classroom and also extends to library usage.   identifying english learners in k-12 education   in the k-12 environment, united states schools are federally mandated to provide els with equal access to the curriculum through linguistic and academic support (nunez et al., 2016). while this support is a federal requirement in the united states, individual states and school districts have the autonomy to (1) establish the processes for identifying els who need linguistic support, (2) manage the level of support provided over time, and (3) determine when students no longer need support (nunez et al., 2016). for example, in new york state, support for els and multilingual learners include programs such as (1) english as a new language (enl), where instruction is primarily in english using specific enl instructional strategies, and, (2) bilingual education (be), which provides instruction in both english and students’ native language to varying degrees (new york state education department, n.d.). in the united states, many schools use standardized test scores on english language proficiency tests to decide whether a student needs linguistic support and should be classified as an el. the el students who score at grade level on standardized academic assessments may meet state criteria to lose their el classification and change their status to reclassified as fluent english proficient (rfep) (nunez et al., 2016). this reclassification means that el students lose access to specialized language support services (sugarman, 2018). in theory, the reclassification signifies that students who were formerly considered els are no longer in need of language and academic support. however, nunez et al. (2016) argue that while these reclassified students would theoretically no longer need english support, in reality, many rfep students “are not truly ‘fluent english proficient’ nor sufficiently academically prepared to be self-sufficient in the mainstream classroom” (p. 58). achieving a predetermined academic threshold sufficient to exit a k-12 enl program is not necessarily indicative of full english proficiency; students who were formerly classified as els might not be at comparable language and academic levels compared to their non-el peers. this notion indicates that united states els, including els who have achieved grade-level test scores, are at risk for being underprepared to succeed in mainstream k-12 classes, and subsequently in college.   identifying english learners in postsecondary education   research shows that els in the united states are often underprepared in regard to college readiness and els’ postsecondary outcomes are lower than those of native english speakers (kanno & cromley, 2015; nunez et al., 2016; roessingh & douglas, 2012). while kanno and cromley (2015) found that 45% of native english speakers attend four-year colleges, only 35% of english-proficient el students and 19% of all els advance to four-year colleges following graduation. it is more common for els to attend community college, though els’ degree completion rates are lower than those of community college students in general (nunez et al., 2016). in terms of six-year college degree completion, one in eight els completed a college degree in this time frame compared with one in three non-el students (bergey et al., 2018). however, data on the transition of els from high school to postsecondary education in the united states is limited.   because students’ el status is not a permanent label, the fluidity of this group lends difficulty to tracking a cohort of language learners over time (sugarman, 2018). in the united states, there is no long-term coordination between identifying els in k-12 schools and later at the college level. longitudinal data is needed to assess long-term language skills; however, federal data sets have limited data on language (nunez et al., 2016). el status is often measured in different ways according to different jurisdictions and other data pose challenges in generalizing outside of specific geographic areas (nunez et al., 2016). this lack of coordinated longitudinal data between els in high school and college results in difficulty identifying and assessing el students who were formerly in a k-12 enl program, as their el or rfep status no longer exists at the postsecondary level (nunez et al., 2016).   while institutions of higher education usually collect demographic data such as race, ethnicity, and first-generation college student status, they typically do not systematically collect data about language background and progress toward language proficiency (bergey et al., 2018). for example, the suny system asks applicants “is english your native language?” however, because this was not a required question on the application, the data on this topic is incomplete. for example, of the fall 2020 applicants at farmingdale state college, 66% left this question blank (farmingdale state college office of institutional research, personal communication march 22, 2021).   most four-year institutions require students to demonstrate a certain level of english proficiency (bergey et al., 2018). though el students’ english skills might be at levels high enough for college acceptance, they still may benefit from specialized el support at the college level. however, there is no state or federal guidance to determine how els should be “assessed, monitored, and served in colleges and universities” (bergey et al., 2018, p. 7). there are also no legal requirements for the academic support services that els’ receive in k-12 schools to extend to postsecondary education. institutions of higher education make their own decisions about whether and how to identify students who come to their institutions with limited english skills. at the postsecondary level, el identification is not standardized and language learners might be identified by a number of standardized tests, such as the scholastic assessment test (sat) reading test, the test of english as a foreign language (toefl), college board’s accuplacer, the america college testing (act) compass placement test, or in-house placement tests (nunez et al., 2016). based on the results of test scores such as these, linguistic and academic support may be provided to postsecondary els in a number of ways.   some institutions require students to complete non-degree english as a second language (esl) courses prior to or alongside their participation in regular academic courses (bergey et al., 2018). however, not all colleges and universities offer esl courses or programs. schools may also offer other types of developmental courses which are often non-credit and not applicable to a degree, but are required in order to advance to credit-bearing courses. at farmingdale state college, students are required to pass english 097: basic writing skills if they do not achieve a certain score on the writing or reading sections of the sat or act, or if they do not pass an english department diagnostic exam on the first day of english 101 composition: college writing, a required course for all students (farmingdale state college course catalog, 2020-2021). nunez et al. (2016) note that four-year colleges may be moving towards integrating language with writing instruction by recognizing the need to “take responsibility for the regular presence of second language writers in writing classes, to understand their characteristics, and to develop instructional and administrative practices that are sensitive to their linguistic and cultural needs” (p. 67).   aside from developmental courses that postsecondary institutions might require, college students are primarily responsible for addressing their own linguistic challenges. while k-12 schools are legally required to provide els with linguistic support, there is no such requirement at the college level and this responsibility instead shifts to the students themselves (nunez et al., 2016). additionally, some els, particularly generation 1.5 els, might score high enough on standardized exams to determine that they are language proficient and they are not placed in any type of support program at the college level. however, generation 1.5 students often struggle to perform at the same academic level as native english speakers. research shows that even students who were formerly in a k-12 program and were reclassified as fluent english proficient (rfep) do not necessarily have the english skills needed for academic success at the college level (nunez et al., 2016; roessingh & douglas, 2012).   generation 1.5 students   coined by rumbaut and ima (1988), the term generation 1.5 was first used to describe children born outside the united states who hovered between the traditional definitions of first-generation and second-generation immigrants. this term has evolved over time and roberge (2002) argues for a definition of generation 1.5 that encompasses young people who were born in the united states in addition to those who immigrated to the united states at a young age. further, huster (2012) emphasizes that the generation 1.5 designation includes individuals who (1) immigrated to the united states as children, (2) are u.s.-born children of immigrants, or (3) were raised in environments where english is not the primary language of the home and community. many generation 1.5 students do not identify as esl students upon entering postsecondary education because they are native-born, or because they have spent a great deal of their lives in the united states and have received much of their k-12 education in the united states (bergey et al., 2018; haras et al., 2008).   roughly one in five children in the united states lives in a home where a language other than english is spoken (migration policy institute, n.d.). the u.s. census bureau’s american community survey reports that 72% of public school students ages 5 to 17 who report speaking english “less than very well” were born in the united states (bialik et al., 2018; u.s. census bureau, 2016). these students attended schools in the united states from a young age or throughout their entire lives. because english is not the primary language spoken at home or in the community, generation 1.5 els are likely to have been part of linguistic support programs at some point in the k-12 public school system (huster, 2012). throughout their time in english-language schools, most els develop strong conversational english skills. however, while generation 1.5 els appear fully conversant in american english and culture, they are still in the process of learning english when they enter college and face a variety of academic english challenges (asher et al., 2009; di gennaro 2008; haras et al., 2008; huster, 2012). despite their participation in k-12 u.s. schools, many generation 1.5 students still need support to meet the demands of academic english required at the postsecondary level (bergey et al., 2018).   understanding academic challenges faced by generation 1.5   academic english   els can develop communicative proficiency in social english within two years of participating in an english-speaking school setting (gonzalez, 2016). in contrast, developing academic language proficiency is a long and gradual process, sometimes taking up to ten years to master (gonzalez, 2016; roessingh & douglas, 2012). communicative language includes everyday language needed for basic conversation that may be accomplished with a vocabulary of up to 5,000 words. in comparison, a native english-speaking high school graduate might have a vocabulary of about 18,000 word families (roessingh & douglas, 2012). roessingh and douglas (2012) explain that academic language demands “abstract uses of language that require language itself to interpret and comprehend” (p. 291). college-level reading and writing assignments often include academic and technical language that can prove to be difficult even for native english speakers. if generation 1.5 students have not fully mastered academic english, they are likely to face reading and writing challenges in college. the discrepancy of appearing to be fluent while facing certain academic difficulties might result in instructors’ reaction of judgment towards students and a lack of understanding their need for extra support.   academic language stigma   generation 1.5 students may also feel embarrassed and disappointed in their english ability, particularly for the purposes of college writing and participating in the college classroom (huster, 2012). huster’s (2012) study of generation 1.5 revealed that these students often exhibit deviations from standard english. though the meaning of their communication was usually clear, the students felt that grammar and vocabulary errors limited their ability to fully participate in class and stigmatized them as “nonnative, foreign – or even worse as uneducated, or unintelligent” (huster, 2012, p. 21). roessingh and douglas (2012) found that students’ self-reported level of spoken english might not reflect the level of academic english needed for postsecondary education and these struggles might not become evident to students until reaching college. in particular, one student in huster’s (2012) study did not feel limited by her english ability in high school, but began to feel “awkward and unprepared” once attempting to participate in college classes (p. 18). this may be partially due to els’ k-12 educational experiences.   els’ educational background   els in the united states are more likely to have attended an under-resourced school compared to native english-speaking classmates and are more likely to have received insufficient instructional support (asher et al., 2009; bergey et al., 2018). further, els are more likely to have been tracked into low-level courses that do not prepare them for college (bergey et al., 2018; haras et al., 2008; nunez et al., 2016; roberge, 2002). els are underrepresented in high-level academic courses in middle school and high school and nunez et al. (2016) argue that students’ el status in k-12 education acts as a barrier to access to ap college preparatory courses. this “limited access to rigorous curricula perpetuates a cycle of low achievement” (nunez et al., 2016, p. 65). students placed in lower-ability courses in high school had little experience with academic writing and were socialized into literary practices that were different from those used in higher track courses (harklau, 2003). such high school tracking practices may heighten the challenges that generation 1.5 students experience in college, as work that was once deemed acceptable is now held to a higher standard.   specific academic struggles   generation 1.5 students often struggle with academic reading and writing and the overall demands of college coursework due to a host of factors (bergey et al., 2018). these students have limited literacy in their first language, lack fully developed oral or written systems of their first language, and may be at risk for losing their first language. unlike international students, generation 1.5 often did not learn a writing system in their first language (roessingh & douglas, 2012). roessingh and douglas (2012) compared els to native speakers by profiling excerpts of assigned first-year college readings and other coursework. while el students were found to be academically competent and showed high math scores, their reading levels were lower than native speakers. though able to effectively convey their meaning through verbal and written communication, huster’s (2012) research shows that generation 1.5 students commonly use grammatical and vocabulary errors. because postsecondary education places a high value on academic language proficiency, such as advanced reading ability, independent library research, group work, and presentation skills, els’ difficulty in these areas may negatively impact their success (roessingh & douglas, 2012).   grammar   generation 1.5 els often deviated from standard american english by using irregular forms of nouns and verbs, grammatical inconsistencies with non-count nouns, and subject-verb agreement errors, present in both their writing and speech (huster, 2012). the participants in huster’s (2012) study attended k-12 united states schools for their entire education and did not have a distinctive pronunciation or accent. however, they did show language pattern alterations related to english morphology (the way words are constructed, such as roots, prefixes, and suffixes), and syntax (the arrangement of words into phrases and sentences). sample errors are italicized below:   generation 1.5 els may show errors in grammatical structure such as past tense, plural forms, and whether words are regular or irregular (harklau, 2003), e.g.: “we seeked for a higher education” (huster, 2012, p. 12). another variation example includes noncount nouns, which are nouns that cannot be counted, e.g. “my older sister will call me… and ask me for advices” (huster, 2012, p. 13). further, generation 1.5 showed difficulty in subject-verb agreement (harklau, 2003), e.g. “i love to help those… families that doesn’t speak english. it makes me happy knowing that i can be a middle man for those that needs translating” (huster, 2012, p. 13).   vocabulary   huster (2012) also found that generation 1.5 els were frustrated with the limitations of their vocabulary and felt that this impacted their ability to perform academically. even though generation 1.5 students have been exposed to academic vocabulary throughout their time in united states schools, they have not necessarily mastered that vocabulary. huster’s (2012) study revealed that the participants sometimes had difficulty answering questions posed by the researcher because they did not understand a word in the question. this problematic gap in vocabulary also occurs in classroom contexts and can significantly affect els’ writing and their ability to effectively participate in class. one student in huster’s (2012) study revealed dissatisfaction with her range of vocabulary, noting: “i frustrate often at finding the right word” (p. 15). another student expressed the following acknowledgement:   “i don’t feel like i have a big enough vocabulary cause i guess just hearing other people talk with you, they use a lot of big words. sometimes when people speak out in classes, like other students or my peers, there’s some times i don’t know what they mean because i’ve never heard [the words] before. or like i’ve heard of it, but i don’t remember the definition or what it actually means” (huster, 2012, p. 16).   lack of sufficient support   though generation 1.5 students’ meaning was generally clear in their verbal and written communication, obvious surface errors are present and are likely to have a negative impact in postsecondary coursework. els’ writing may exhibit less fluency (fewer words), less accuracy (more errors), and less effectiveness (lower holistic scores) (di gennaro, 2008). bergey et al. (2018) argue that k-12 enl programs need to provide students with increased academic writing instruction that can be applied to real life academic and professional contexts beyond high school. ideally receiving instruction that differs from the content and methods used for international or immigrant students, generation 1.5 would benefit from targeted instruction that is personalized to their proficiency levels (bergey et al., 2018). however, while scaffolding supports are present in high school, they are absent at the college level and many generation 1.5 els are unidentified as language learners who are tackling college-level coursework. els’ easily noticeable errors noted by huster (2012) may be bothersome to college-level instructors, particularly those who do not have background knowledge of els. it is common for students to encounter teachers who lack training in how to work with els and who are unaware of their specific needs, and this may negatively impact students’ college performance (harklau, 2003).   the aforementioned reasons illustrate why els, including students who were formerly classified as el, may benefit from specialized strategies and pedagogical techniques known to help language learners. as academic libraries are entities of support on college campuses, it is beneficial for librarians to be familiar with the el community, including potential challenges they may face and strategies to support them in their transition to postsecondary education.   generation 1.5 and academic libraries   because generation 1.5 els are proficient in social english, librarians might be unaware of students’ academic english skills. further, as this student population is largely unidentified at the college level, academic librarians might also be unaware of the overall needs of generation 1.5 students. while limited research has been conducted on generation 1.5 students and academic libraries, asher et al. (2009) found that they attached greater value to the library facilities compared to non-generation 1.5 students. generation 1.5 students reported that the library’s physical space was important to them because it allows for unstructured social learning and provides the technology they need for coursework, even if they had the technology they needed at home (asher et al., 2009). however, while generation 1.5 els are strong users of library spaces, they may still face difficulties conducting library research and seeking the help they need in libraries.   library-specific challenges   while some students’ information literacy skills are primarily developed at the k-12 level (haras et al., 2008), not all students are equipped with a background of familiarity and confidence using libraries. because els in the united states often attend under-resourced k-12 schools in non-college preparatory classes, they may not have had experience using a school library or searching databases prior to entering college. haras et al. (2008) emphasize the importance of high school librarians working with classes to teach them the research process and prepare them for what is expected in college, though students do not always have these opportunities. for example, an information literacy class taught at farmingdale state college may yield student feedback showing that some students learned how to search databases in high school while others did not.   els who lack knowledge of academic libraries and the research process may also lack the confidence needed to seek assistance from librarians. compared to native english-speaking students, els tend to have higher library anxiety in communicating with librarians and using libraries (ishimura & bartlett, 2014). generation 1.5 was also reported to be less confident in using electronic resources (asher et al., 2009). while it can be difficult for any student, even native speakers, to admit they do not know something, els’ language challenges often result in a lack of confidence to advocate for themselves in regard to the help they need to succeed. this hesitation may preclude them from approaching the reference desk or asking a question in an information literacy class.   as huster’s (2012) study showed, generation 1.5 students can easily misunderstand a word in a conversation. these vocabulary gaps may contribute to students’ potential confusion during a reference interaction or during an information literacy class. adding to their lack of confidence, information literacy classes also often involve vocabulary with which els may be unfamiliar. a major aspect of database searching includes selecting keywords, a task that becomes increasingly difficult as one’s vocabulary range decreases. in addition to experiencing difficulty selecting keywords, els might be challenged by other database-related language uses such as using plural forms, synonyms, and correct spelling. such factors can limit students’ effectiveness in searching (ishimura & bartlett, 2014). librarians’ awareness of these common challenges that generation 1.5 els face enables them to better provide support to this student population.   discussion   strategies for supporting generation 1.5   academic librarians have the ability to recognize and address these challenges by understanding the needs of the el community and adopting el-specific strategies into daily interactions with students. librarians can employ a variety of strategies both at the reference desk and in information literacy classes to support el students. these methods can be used to reinforce content for students and establish rapport with students, both of which are key to helping the el student community. as it is generally difficult to recognize generation 1.5 students due to their proficient social english skills, their possible lack of an accent, and the common loss of the el label at the college level, librarians might not be aware of which students are els. the benefits of adopting el-based techniques are not limited to el students only. librarians’ implementation of methods that are known to help els, including generation 1.5, can also be used in support of all students. because els may not be identifiable by accent or pronunciation, the following recommendations are not only beneficial for els, but are also useful for all academic library users.   acknowledging anxiety   one introductory step that librarians can take to support generation 1.5 is acknowledging potential anxieties that students may hold about using the library or doing research. conteh-morgan (2002) surveyed el students about their ideal learning environment. students reported that they wanted teachers to have “enthusiasm, patience, a warm reception, and personalized acceptance and concern for each student” (p. 193). these characteristics are often exhibited in a typical k-12 enl classroom, and students would like these qualities to be replicated in other learning contexts as well. actions such as offering a welcoming smile, providing students with an introduction, and making an effort to acknowledge and address students’ concerns can make a significant difference in els’ comfort level. librarians who frequently work with el students reported using techniques such as consciously showing empathy, understanding, and interest in students’ questions and responses, in addition to using active listening techniques (ishimura & bartlett, 2014). while research shows this is an el-based strategy, fostering a welcoming environment for students is a core mission of libraries. reassuring students that help is available to them is a small step in making them feel comfortable using the library.   language use   it is also recommended to avoid using technical and library-specific jargon (ishimura & bartlett, 2014), and to limit the use of slang and idiomatic expressions in conversations with els both at the reference desk and in information literacy classes (conteh-morgan, 2002). some examples of idioms that might be used while teaching include phrases such as “to make sure we’re all on the same page,” “so far so good,” “cover a lot of ground,” and “draw a blank,” etc. refraining from these types of expressions can help ensure that els, including generation 1.5 students, understand the content that is being communicated. it can be helpful for librarians to adjust their speaking style when working with els, such as repeating or rephrasing words to help with any gaps in vocabulary students might have (ishimura & bartlett, 2014). further, it is beneficial to pause between different groups of ideas and to restate ideas to help learner comprehension.   it is also important to be aware of how language and culture can intertwine. different communication styles can result in misunderstandings. wang and frank’s (2002) research on this topic showed that some els were confused by the phrase “check out books” on library signage. focus groups revealed that the students thought that “check out” implied “examining or searching” and they were not interested in being examined or searched. another group of students thought the phrase “checking out books” was associated with paying for books. in this case, the library changed the signs to “borrow books” (wang & frank, 2002). this example illustrates how easily miscommunications can occur between non-native and native speakers.   additional language considerations include nonverbal communication. different cultures have different views of facial expressions, physical gestures, posture, eye contact, and voice pitch or volume. american students usually use eye contact in one-to-one conversations, which typically indicates interest and respect. wang and frank (2002) explain that students from other cultures might look away in conversations which could possibly be perceived as not paying attention. personal space can also be different in different cultures. one example is that middle eastern students tend to be physically closer to people to whom they are speaking, indicating a sign of interest. according to wang and frank (2002), japanese students tend to maintain some physical distance in conversations, demonstrating respect for others. additional examples include correct posture being related to respect by some chinese students and sitting with one leg crossed over the other leg possibly being viewed as offensive by some middle eastern students (wang & frank, 2002). librarians’ acknowledgment that different cultural norms exist for different library systems and services can help them understand students’ actions and foster a non-judgmental environment throughout the library (ishimura & bartlett, 2014).   implementing language objectives   enl classes often incorporate language objectives as part of the lesson. language objectives aim to give language learners equal access to the curriculum, even though these students may not be fully english proficient (himmel, 2012). often, k-12 content area classes will have both content objectives and language objectives for each lesson. in order to outline new and important vocabulary, librarians can adapt the concept of establishing language objectives in information literacy classes by outlining the academic language that will be learned and mastered in that lesson. for example, language objectives for an information literacy class might include explaining vocabulary words such as abstract, plagiarism, peer-reviewed journals, and so on. this explicit outlining of key terms can help els identify important vocabulary, particularly if they struggle in this area.   non-linguistic cues   because generation 1.5 students may have gaps in their academic vocabulary, visuals, gestures, intonation, and other non-verbal cues can make language and content more accessible to students (ferlazzo, 2016). teaching with visual representations of concepts can be very helpful for els, and providing any visual example of class content is beneficial. for example, if different types of sources are being discussed, provide images of a peer-reviewed article compared to a magazine article; if a citation format is being taught, display images of in-text citations or a reference list. even though students may have mastered social english, it can be more difficult for els to absorb content in an information literacy class, which might include new terminology and concepts. visuals are key instructional tools in k-12 enl classes and can be easily incorporated into library instruction.   graphic organizers are a type of visual scaffolding tool that are beneficial for els because they help students plan and organize their ideas. one way that graphic organizers can be used is with concepts such as citing sources. this graphic organizer would prompt students to identify different elements of a citation in a chart, such as author, title of source, date, and so on. once students identify what the different elements are, they can use the chart to construct the citation (tran & aytac, 2018). graphic organizers are widely used in k-12 enl programs and the act of breaking up a process into multiple steps can be very helpful for els.   another type of useful graphic organizer is a kwl chart, abbreviated from know, want to know, and learned (tran & aytac, 2018). the k section, what students know, helps to activate background knowledge and make connections to the class content. this section might prompt students to consider if they already know something about the content that is going to be taught that day, such as searching library databases. this step helps prepare students for what is going to be taught. the w section, what students want to know, helps to engage them in a new topic. in addition to encouraging students to think about the class content, this tool also helps the instructor learn about students’ prior knowledge. these responses could be submitted in a form at the beginning of class for the librarian to quickly review before beginning teaching. the last section of the kwl chart, what students learned, can be filled out at the end of class and used as a short term assessment tool. this section could also be completed with the class as a whole to create a master list displaying what students learned (tran & aytac, 2018).   wait time   providing students with enough wait time to form their responses when posing a question in an information literacy class is another helpful strategy for els. students might be thinking and producing in two or more languages and they need time to process the question (ferlazzo, 2016; huster, 2012; roessingh & douglas, 2012). instead of calling on someone immediately, allow wait time for students to have enough time to comprehend the question, think about their answer, and feel comfortable answering in front of a class. in addition to seeking verbal responses from students in class, using an online polling program such as poll everywhere allows students to anonymously ask and answer questions, or share quick thoughts (poll everywhere, n.d.). anonymity provides students with the freedom to contribute and get the answers they need without exposing themselves as not knowing something, or publicly getting an answer wrong. this method also aids the instructor in checking students’ understanding of class content and offers opportunities to address any uncertainties. as with many other el-targeted strategies, providing ample wait time during class benefits all students, including generation 1.5, among others.   cooperative learning   similar to the concept of providing wait time, cooperative learning, or group work, is another practice that permits students thinking time and is a demonstrated support for els (tran & aytac, 2018). known to provide els with a low-stress and friendly learning environment, cooperative learning allows students to brainstorm ideas with peers and try out their ideas in a low-stakes setting. group work also facilitates peer to peer learning, as students ask each other questions and explain tasks and concepts to each other. students are more likely to ask questions and share their opinions in a small group compared to an entire class. in addition to dividing students into groups, librarians might also implement think-pair-share as a quick collaboration method. following a question, think-pair-share first prompts students to independently think or write about a response themselves. next, students turn to someone near them and discuss their thoughts. lastly, the librarian can ask a few groups to share with the entire class. this strategy supports els by providing thinking time and a low-stakes setting to test their ideas with minimal time and effort required. additionally, story reenactment is another collaborative strategy that is useful for language learners. in this method, students act out stories as part of the learning process. for example, in a lesson on plagiarism, students can be assigned different roles, such as someone who cuts and pastes, someone who quotes information without using quotation marks, and so on (tran & aytac, 2018). this strategy both helps to reinforce content and fosters a memorable learning experience.   modeling and thinking aloud   els benefit from seeing a step-by-step process of how to complete a task (ferlazzo, 2016). when explaining a concept such as searching databases or citing sources, it is helpful to provide granular details of each step in the process. while information literacy classes often include searching demonstrations, it is important to describe the actions that are being taken and the thought process behind those actions. at the reference desk, it might sometimes be easy to forget that routine tasks such as navigating the library website and searching databases could be completely new to someone. instead of providing students with the materials they request with minimal or inconsistent explanations, using the think-aloud technique allows researchers to understand librarians’ complete thought process behind their searching. though explaining each step as it is being taken might feel redundant, sharing thinking processes is very helpful to els. additionally, providing models of finished products is particularly useful for els. rather than telling els what to do, it is better to show them what to do (ferlazzo, 2016).   creating resources for your learners   because els might have difficulty understanding every word in information literacy classes or reference conversations, written instructions can be very helpful for els (tran & aytac, 2018). depending on the class, students might be trying to follow along on their own computer or learning new library-related vocabulary, and it may be difficult to remember every detail once class is over. librarians who are experienced in working with els reported utilizing prepared handouts and written communication such as writing down key points in order to benefit their learners (ishimura & bartlett, 2014). providing written resources for els allows them to have more time to process information in class because they do not need to write down as much information (conteh-morgan, 2002). including examples of successful end-products is also significantly helpful for els, as it shows them how to complete a task. models of finished products might include example citations, a sample paper formatted in a particular citation style, or an annotated bibliography. in reference interactions, librarians might also create simple written instructions on-the-fly in order to provide students with a guide for referencing later. further, creating an electronic handout, such as a google doc, easily allows for students to access links to other resources, such as class presentation slides, research guides, databases, or video tutorials.   flipped classroom   el students benefit from having a preview of class content and a flipped classroom lesson is one way to allow students extra time to absorb content prior to class (tran & aytac, 2018). though this method requires advance coordination between librarians and teaching faculty, allowing students to have multiple exposures to content can be very helpful to els (tran & aytac, 2018). students can review information literacy resources prior to class, such as videos, handouts, or research guides. with buy-in from the instructor, these resources could be posted as an assignment through their course’s learning management system (lms) page. this allows for class time to be used to do the harder work of assimilating those resources, such as having more hands-on time to practice searching databases, identifying keywords, building a reference list, etc. depending on the language needs of the class, it could also be helpful to provide the instructor with the library-specific vocabulary to be covered prior to class.   word walls   another strategy that addresses library-specific vocabulary is word walls. a word wall is a location on the classroom wall, or possibly a whiteboard, where relevant vocabulary is listed. this vocabulary might also include an image or brief definition. many enl classes use word walls to help els with vocabulary, and this practice could also be implemented for information literacy classes (tran & aytac, 2018). while not applicable for every class, creating a word wall in the classroom space or on a handout provides opportunities for pre-teaching and vocabulary front-loading. identifying vocabulary before being used in the context of a lesson is a useful strategy when teaching language learners.   checking for understanding   throughout teaching and working with students at the reference desk, it is important to assess students’ understanding of content (ferzallo, 2012). els in particular might be hesitant to ask questions, and librarians who work with language learners noted the importance of paying attention to students’ reactions to their explanations to see if they were understanding (ishimura & bartlett, 2014). while not applicable for all els, some cultures might be sensitive about “saving face,” and would not want to “lose face” by admitting that they do not understand a concept. in these situations, students might say they understand even when they do not, and they also might not want their lack of understanding to negatively reflect on the instructor as well (wang & frank, 2002). because of students’ possible hesitancy to share that they do not understand, it is recommended to periodically check in with students throughout class. anonymous polling can be a helpful tool to gauge students’ understanding without requiring students to stand out as someone who needs assistance. because students might be hesitant to ask questions due to embarrassment or fear of judgement, anonymity provides the freedom to be honest. students are also more likely to ask questions if provided time during class to discuss individually with the instructor.   informal practices   additionally, informal practices can also supplement el-based pedagogy in supporting generation 1.5 students in libraries. students’ awareness and willingness to seek assistance when needed is a significant component of college success. some students might face difficulty in approaching the reference desk or asking for help, particularly if they are not sure how to phrase a question or if they are concerned they are going to be judged for not knowing something. having a friendly face can go a long way in building rapport with students (conteh-morgan, 2002). it is important to emphasize that librarians aim to help students and they should not hesitate to ask questions or seek assistance. it is also necessary for librarians to acknowledge that students might come from all types of backgrounds. though students’ conversational english might seemingly indicate that they are native english speakers, it is important to remember their possible difficulties with academic english. students’ comfort level in seeking out help plays a significant role in helping them succeed in college.   conclusion   the aforementioned el-targeted pedagogical strategies address challenges that generation 1.5 students may face at the postsecondary level. though observed to be resilient and tenacious (roessingh & douglas, 2012), generation 1.5 els often experience struggles that are detrimental to college students, such as insufficient academic grammar and vocabulary (huster, 2012). as academic librarians are often unaware of students’ linguistic backgrounds, it is beneficial to avoid the assumption that students are native speakers based on their proficiency in social english; generation 1.5 students still benefit from el-based teaching techniques despite their conversational english competency. librarians should be aware that students may come from a variety of educational backgrounds and els in particular may lack adequate college preparation at the high school level. because generation 1.5 students often lose their language learner label once entering college, it is often the case that these students no longer receive specialized support. however, the academic library is an entity of support itself, and librarians are equipped to employ el-based strategies in order to better meet students’ needs, even if they are unaware of a student’s language proficiency. it is beneficial for librarians to be aware that generation 1.5 students might feel stigmatized because of academic language barriers. in libraries, these barriers may present themselves through difficulties with keyword searching and understanding explanations at the reference desk and in information literacy classes. overall, it is most important for librarians to provide students with a non-judgmental environment of support, both meeting their research needs and allowing them to feel comfortable and confident advocating for the help they need. while the strategies presented in this paper are targeted toward els, these methods also help all students learn to effectively conduct research and support their transition to college.   references   asher, c., case, e., & zhong, y. (2009). serving generation 1.5: academic library use and students from non–english-speaking households. college & research libraries, 70(3), 258-272. https://doi.org/10.5860/0700258   bergey, r., movit, m., baird, a. s., & faria, a. (2018, march 5). serving english language learners in higher education: unlocking the potential. american institutes for research. https://www.air.org/sites/default/files/downloads/report/serving-english-language-learners-in-higher-education-2018.pdf   bialik, k., scheller, a., & walker, k. (2018, october 25). 6 facts about english language learners in u.s. public schools. pew research center. https://www.pewresearch.org/fact-tank/2018/10/25/6-facts-about-english-language-learners-in-u-s-public-schools/   conteh-morgan, m. (2002). connecting the dots: limited english proficiency, second language learning theories, and information literacy instruction. the journal of academic librarianship, 28(4), 191-196. https://doi.org/10.1016/s0099-1333(02)00282-3   di gennaro, k. (2008). assessment of generation 1.5 learners for placement into college writing courses. journal of basic writing, 27(1), 61-79. https://doi.org/10.37514/jbw-j.2008.27.1.04   farmingdale state college. (2020). course catalog 2020-2021. https://catalog.farmingdale.edu/2020-2021/catalog.pdf   farmingdale state college. (n.d.-a). facts and figures. https://www.farmingdale.edu/about/index.shtml   farmingdale state college. (n.d. -b). student demographics dashboard. https://www.farmingdale.edu/institutional-research/demos_dashboard.shtml   ferlazzo, l. (2016, november 3). do’s & don’ts for teaching english-language learners. edutopia. https://www.edutopia.org/blog/esl-ell-tips-ferlazzo-sypnieski   gonzalez, a. (2016, november 1). 10 assumptions to rethink about english-language learners. education week. https://www.edweek.org/teaching-learning/opinion-10-assumptions-to-rethink-about-english-language-learners/2016/11   haras, c., lopez, e. m., & ferry, k. (2008). (generation 1.5) latino students and the library: a case study. the journal of academic librarianship, 34(5), 425-433. https://doi.org/10.1016/j.acalib.2008.06.004   harklau, l. (2003, october). generation 1.5 students and college writing. eric digest. https://files.eric.ed.gov/fulltext/ed482491.pdf      himmel, j. (2012). language objectives: the key to effective content area instruction for english learners. colorín colorado. https://www.colorincolorado.org/article/language-objectives-key-effective-content-area-instruction-english-learners   huster, k. (2012). the grammar and vocabulary challenges of generation 1.5 hmong college women in academia. hmong studies journal, 13(1), 1-30.   ishimura, y., & bartlett, j. (2014). are librarians equipped to teach international students? a survey of current practices and recommendations for training. the journal of academic librarianship, 40(3-4), 313-321. https://doi.org/10.1016/j.acalib.2014.04.009   kanno, y., & cromley, j. (2015). english language learners’ pathways to four-year colleges. teachers college record, 117(12), 1-44.   migration policy institute. (n.d.). children in u.s. immigrant families. https://www.migrationpolicy.org/programs/data-hub/charts/children-immigrant-families   national center for education statistics. (2020, november). table 204.20. english language learner (ell) students enrolled in public elementary and secondary schools, by state: selected years, fall 2000 through fall 2018. digest of education statistics. https://nces.ed.gov/programs/digest/d20/tables/dt20_204.20.asp   national education association. (2020, july). english language learners. https://www.nea.org/resource-library/english-language-learners   new york state education department. (n.d.). program options for english language learners/multilingual learners. bilingual education & english as a new language. http://www.nysed.gov/bilingual-ed/program-options-english-language-learnersmultilingual-learners   nunez, a. m., rios-aguilar, c., kanno, y., & flores, s. m. (2016). english learners and their transition to postsecondary education. in m. b. paulsen (ed.), higher education: handbook of theory and research (vol. 31, pp. 41-90). springer. https://doi.org/10.1007/978-3-319-26829-3_2     poll everywhere. (n.d.). poll everywhere for universities. https://www.polleverywhere.com/university-student-response-system   roberge, m. (2002). california’s generation 1.5 immigrants: what experiences, characteristics, and needs do they bring to our english classes. the catesol journal, 14(1), 107-129. http://www.catesoljournal.org/wp-content/uploads/2014/07/cj14_roberge.pdf      roessingh, h., & douglas, s. (2012). english language learners’ transitional needs from high school to university: an exploratory study. journal of international migration and integration, 13(3), 285-301. https://doi.org/10.1007/s12134-011-0202-8   rumbaut, r. g., & ima, k. (1988). the adaptation of southeast asian refugee youth: a comparative study. final report to the office of refugee resettlement. san diego state university. https://eric.ed.gov/?id=ed299372      sugarman, j. (2018, may). a guide to finding and understanding english learner data. migration policy center institute, national center on immigrant integration policy. https://www.migrationpolicy.org/research/guide-finding-understanding-english-learner-data   tran, c. y., & aytac, s. (2018). strategies for teaching information literacy to english language learners. collaborative librarianship, 10(4), 251-266. https://digitalcommons.du.edu/collaborativelibrarianship/vol10/iss4/5/   united states census bureau. (2016). selected social characteristics in the united states. [data set]. american community survey. https://data.census.gov/cedsci/table?q=language%20proficiency&d=acs%205-year%20estimates%20data%20profiles&tid=acsdp5y2016.dp02   wang, j., & frank, d. g. (2002). cross-cultural communication: implications for effective information services in academic libraries. portal: libraries and the academy, 2(2), 207-216. https://doi.org/10.1353/pla.2002.0046   evidence based library and information practice evidence based library and information practice 2009, 4.4     evidence based library and information practice         news/announcements    call for juried proposals: integrating practice and research library research seminar v      © 2009. this is an open access article distributed under the terms of the creative commons attribution   license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and   reproduction in any medium, provided the original work is properly cited.      the fifth library research seminar (lrs‐v)  will bring together a diverse community of  scholars from academia and practitioners from  libraries and archives who are interested in  research that informs policy‐making, decision‐ making, and best practices. participants will  share research projects and explore ways to  develop future research agendas, refine  research methods, and facilitate successful  completion of research projects.    the lrs‐v program committee invites  proposals for various types of contributions  (types are described below) on topics related  to libraries and archives including but not  limited to:  • services in challenging economic  times  • marketing and advocacy  • leadership and workforce  development  • information and reference services  • international perspectives  • contributions to and preservation of  cultural heritage  • gender, ethnicity, age, and disability  status  • copyright, privacy, and other legal,  ethical, and policy issues  • technical services  • user studies  • web 2.0, social networking, and new  media  • information literacy  • digital libraries and archives.  possible types of contributions:  • papers: research studies that will be  presented at the conference and  included in proceedings  • panels: a group of experts discussing  related topics, themes or issues in  library research  • workshops: tutorial sessions that will  be educational in nature  • roundtable discussions: informal  discussion amongst participants  focused on a particular topic or theme  • posters: graphic presentations on  research studies, methods, advances,  or preliminary work  • other ʺwildcardʺ program formats –  you tell us what you would like to do!    doctoral and mastersʹ students are especially  encouraged to submit proposals.    please send submissions to <lrs‐v@umd.edu>  in either ms word or pdf format. proposals  should be no more than 1000 words in length  and must include: title; author/organizer  name, affiliation, and contact information;  names and contact information for any other  participants.  the email address <lrs‐ v@umd.edu> may also be used for inquiries  and questions.            103 mailto:lrs-v@umd.edu mailto:lrs-v@umd.edu mailto:lrs-v@umd.edu evidence based library and information practice 2009, 4.4  important dates    proposal submissions: february 15, 2010  notification: april 15, 2010  conference dates:  october 13‐16, 2010  venue: university of maryland, college park  <http://ischool.umd.edu>    lrs‐v co‐chairs:  diane l. barlow and trudi  bellardo hahn, university of maryland    sponsored by: library research roundtable of  the american library association and the  institute of museum and library services.    104 http://ischool.umd.edu/ evidence summary   libraries’ contributions to the quality of uk university research environments were not acknowledged in ref 2014, but could be made more visible in ref 2021   a review of: walker, d. (2020). libraries and the ref: how do librarians contribute to research excellence? insights, 33(1), 6. https://doi.org/10.1629/uksg.497   reviewed by: barbara m. wildemuth professor emeritus school of information & library science university of north carolina at chapel hill chapel hill, north carolina, united states of america email: wildemuth@unc.edu   received: 2 dec. 2020                                                                accepted:  22 jan. 2021      2021 wildemuth. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29889     abstract   objective – to measure the extent to which libraries’ contributions to united kingdom (uk) university research excellence were referenced in the research excellence framework (ref) 2014 unit-level research environment statements, and to make recommendations to libraries for increasing their visibility in the research setting.   design – content analysis of an existing corpus.   setting – evaluation of research environments conducted as part of the uk ref 2014 assessment.   subjects – 1,891 unit-level research environment statements submitted for ref 2014.   methods – each unit-level research environment statement was categorized in terms of how extensively it referenced library or librarian contributions: no mention, brief mention, or substantive mention. the frequency and percentage of each level of mention are reported overall and by disciplinary panel.   main results – across all panels, only 25.8% of the statements included substantive references to the library or librarians; most of these were lists of electronic and physical collections, but they also included discussions of the research support services offered by librarians. there were disciplinary differences in the extent of the references to libraries, from 7.2% containing substantive references in a panel examining science, technology, engineering, and mathematics (stem) units to 44.0% containing substantive references in the panel examining arts and humanities units.   conclusion – in ref 2014, libraries and librarians were rarely discussed in unit-level research environment statements. while this lack of representation may be due to shortcomings of the library’s relationship with the university’s research office, librarians could use a number of approaches to becoming more visible in the ref 2021 research environment statements. specifically, they could highlight their roles in: ensuring discoverability and accessibility of information resources to researchers; improving research practices through teaching informational and organizational skills, providing direct support to research students and staff, and providing research data management services; managing the research information systems that capture and make discoverable the university’s non-article research outputs; providing support in relation to the responsible use of bibliometrics and other measures of article quality and impact; further developing article impact by training researchers to use social media to their advantage; developing open research initiatives; and assisting with the ref submission process.   commentary   walker’s study is relatively straightforward and well conducted (glynn, 2006). the study sample included the complete population of research environment statements submitted for ref 2014, so no inferences about the generalizability of the results need to be made. there is some ambiguity in the research methods description: the criteria for a “mention” of a library are not provided. also, there is no report of inter-rater reliability in the categorization, so it is assumed that walker conducted the analysis on his own. neither of these methodological issues is likely to cast much doubt on the overall conclusions of the study, since the categorization scheme was relatively simple and relatively objective. the results of the study are clearly presented and the examination of disciplinary differences is appropriate. the primary way in which the study might have been improved is through a more detailed qualitative examination of the substantive references to the library’s role that consisted of more than a list of resources in the collection. such an examination may have yielded data that would more strongly support walker’s recommendations.   about half of walker’s paper reports on the analysis of the research environment statements, and the other half focuses on ways that libraries and librarians can improve their visibility within their own institutions and in the ref 2021 process. these recommendations are consistent with, but provide more specificity than, previously published recommendations related to positioning academic libraries within their institutions (boyce et al., 2019; cox, 2018). in addition, the recommendations are useful because they are discussed at a level of detail that is directly applicable to individual institutions. academic librarians in the uk will be able to identify specific ideas that most closely fit their own settings and work with their university research offices to incorporate those ideas in the ref 2021 research environment statements.   the publication of walker’s paper was timely, since the ref 2021 submissions were being completed as it was published. thus, the results of the ref 2014 analysis and the recommendations that came from it could be directly applied by academic libraries in the uk. the results and recommendations would also be directly applicable in other countries that employ some form of performance-based university research funding system, as outlined by hicks (2012), that includes the evaluation of the research environment in addition to research outputs. beyond the context of a national research assessment exercise, walker’s recommendations will be useful to academic librarians in their own strategic planning. for example, most academic libraries would benefit by developing and marketing research data management services; these services could be directly aligned with and linked to the university’s research strategy, making the library’s critical role more visible to university administrators. acting on some of walker’s recommendations may help academic libraries deliver on the maxim that the library is the heart of the university.   references   boyce, g., greenwood, a., haworth, a., hodgson, j., jones, c., marsh, g., mawson, m., & sadler, r. (2019). visions of value: leading the development of a view of the university library in the 21st century. the journal of academic librarianship, 45(5), 102046. https://doi.org/10.1016/j.acalib.2019.102046   cox, j. (2018). positioning the academic library within the institution: a literature review. new review of academic librarianship, 24(3-4), 217-241. https://doi.org/10.1080/13614533.2018.1466342   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154   hicks, d. (2012). performance-based university research funding systems. research policy, 41(2), 251-261. https://doi.org/10.1016/j.respol.2011.09.007     news   evidence based library and information practice (eblip) is seeking an editorial intern    2020. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29816     we are looking for a new editorial intern to assist our editorial team. the ideal candidate will be in place by november 2020 and will commit to serving a two-year term.    the role of the intern is to:    provide a final editorial check of proofed copy before publication, using the pre-developed checklist  check and edit the item metadata on the eblip website to ensure title, author, and abstract correspond to the submitted manuscript  assist the editor-in-chief and communications officer with calls for papers or calls for volunteers  assist with investigation and implementation of new projects related to the journal  participate in occasional editorial team meetings    the ideal candidate will be an mlis (or equivalent) student or a recent graduate (past two years) interested in evidence based practice and research methods, and who possesses strong attention to detail. the position requires dedicated time on a regular basis, and it is therefore essential that interested persons ensure available time to devote to this position prior to applying. it is estimated that the workload is approximately 10 hours per month.    interested persons should send a cover letter, indicating areas of strength they would bring to the role, and resume/cv as a single pdf file to lorie kloda, editor-in-chief, at eblipjournal@gmail.com by october 9, 2020.   specific queries about the role should be addressed to the current editorial intern, kimberly mackenzie at kimberly.mackenzie@umassmed.edu.   **please note that evidence based library and information practice is a non-profit, open access journal and all positions are voluntary and unpaid.   about the journal:   published quarterly, and hosted by the university of alberta, this peer-reviewed, open access journal is targeted at all library and information professionals interested in an evidence based model of practice. by facilitating access to librarianship research via original research articles and evidence summaries of relevant research from the library literature, evidence based library and information practice will enable librarians to practice their profession in an evidence based manner.   please visit the evidence based library and information practice web site (http://journals.library.ualberta.ca/index.php/eblip) for further information about the journal.     microsoft word edresp evidence based library and information practice 2009, 4:1 3 evidence based library and information practice editorial responsibilities © 2009. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. editor in chief: denise koufogiannakis associate editor (articles): alison brettle associate editor (evidence summaries): lorie kloda associate editor (commentaries, using evidence in practice, ebl101, news): denise koufogiannakis publication editor: katrine mallan copyeditors: lisa shen, priscilla stephenson, elizabeth zeeuw, dale storie research article   sharing success: a review of strategic planning, annual reports, and publicly available information from academic libraries   kaitlin springmier instruction & learning assessment librarian sonoma state university library rohnert park, california, united states of america email: kaitlin.springmier@sonoma.edu   elizabeth edwards assessment librarian university of chicago library chicago, illinois, united states of america email: eee@uchicago.edu   michelle b. bass population research librarian lane medical library & knowledge management center stanford university stanford, california, united states of america email: michellebbass@stanford.edu   received: 14 july 2017    accepted: 8 mar. 2018      2018 springmier, edwards, and bass. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29316     abstract   objective – this paper reports on a study which explored web-based information sharing practices in north american academic libraries. this study specifically focused on how selected academic libraries use data, reports, and other strategic planning documents to communicate success and demonstrate impact to stakeholders, administrators, and peers.   methods – an environmental scan was conducted to explore the assessment programs and communication practices of 97 north american academic libraries. the population for this study was identified on the basis of several metrics: consortial membership, association of research libraries (arl) ranking on various criteria, and institutional attendance at the 2014 and 2016 library assessment conferences (lac). researchers conducted content analyses on the websites of the 97 libraries to identify measures of institutional support for assessment and to explore the range, depth, and quality of data made available. these iterative analyses were supported by the use of a rubric developed based on emergent criteria observed during multiple phases of review.   results – of the libraries reviewed, 57% made some form of data available to the public. the most robust and effective use of data observed in this study involved the use of data to tell stories about the library and its impact. while this study found a positive correlation between libraries with clear investments in assessment and their use of data in public documents, it found that other factors such as an institution’s consortial memberships or funding model may more strongly influence a library’s decision to make data available.   conclusions – while observations gleaned from this study may serve as a benchmark for evaluating communication practices in academic libraries, further research is necessary to understand how factors within an academic library, its parent institution, or the profession at large may contribute to this decision making process.         introduction   in 2016, a group of librarians at the university of chicago library conducted an environmental scan to learn how academic libraries were sharing information about their assessment programs on their websites. while conducting the scan, project members were struck by the myriad ways libraries were (or were not) using data in public-facing communications. as a result, the focus of this project evolved to explore how libraries use data, reports, and other strategic planning documents to communicate success and demonstrate impact to stakeholders, administrators, and peers.   literature review   formative work by mcclure and samuels shows that transparent, evidence-based decision making contributes to a more productive, innovative library. beginning in 1985, mcclure and samuels found that academic librarians had an overwhelming preference for internal information sources, and tended to ignore external information (such as patron preferences) during the decision making process. the authors claimed that these internally-based decisions encourage ineffective library activity because they create a “‘closed,’ inflexible environment” (p. 496) that is unable to adapt to the changing needs of library clientele. mcclure continued this work in 1986, focusing on the need for staff to have a clear understanding of performance measure data for library planning and decision making. when interviewing public services academic librarians ranked as middle management, he found that these librarians were unlikely to use data for library decision making because they neither trusted the quality of the data nor were motivated to make effective use of it. mcclure claimed that a library must have an “organizational system in place that recognizes the interactive aspects of policy making, encourages wide access to the data, and recognizes that empirical data are used in a much broader psychological context of organizational politics, personalities, and conflicting objectives” (p. 333). while these studies are almost 30 years old, the scholarship on the use of information and data in decision making within libraries (koufogiannakis, 2014) and many of their main conclusions about library communication and decision making hold true today.   assessment may feel like the latest trend in libraries, but the use of data to inform library service decisions is well-documented in the literature (lundstrom, martin, & cochran, 2016; manzuch & maceviciute, 2014; paulus, 2014; seago, schlesinger, & hampton, 2002; van house, 1989). in fact, van house (1989) documents examples of the use of performance or output measures dating back to the early 1970s. in more recent years, paulus (2014) collected data on questions asked to roving library workers and at service desks to inform staffing levels, hours, and roving locations in the library’s new learning commons. lundstrom and colleagues (2016) partnered with academic departments to map research-related student outcomes; they made significant changes in programming after determining that current levels of library integration were failing.   it is less clear what might motivate a library to make their data available to an external audience, particularly data related to strategic decision making. in their review of association of research libraries (arl) institutions’ use of data in assessment and strategic planning, lewin & passonneau (2012) gave examples of productive ways libraries used assessment to improve service to their stakeholders and parent institutions. saunders (2016) performed content analyses on libraries’ strategic directions documents, and found that slightly over 25% of her institution’s cohort “integrated explicit performance related metrics into their strategic plans” (p. 10). given that libraries have been using a range of data for decision making for several decades, it is worth measuring the profession’s progress towards making these data and the resulting decisions available to their stakeholders.   aims   this project explored the hypothesis that an academic library’s demonstrated commitment to assessment would correlate with the web-based presence of data or other strategic documents demonstrating institutional progress or data-driven decision making. after conducting content analyses of web-based documents made available by 97 north american academic libraries, this hypothesis was only partially supported, as it seems that other characteristics of academic libraries were more closely correlated with the presence of data and other reports on their websites.   methods   research design   this study was conducted between may 2016 and june 2017 by researchers affiliated with the university of chicago library.   a total of 97 north american academic research libraries comprised the population for this study. this population was identified on the basis of several metrics: consortial membership, arl ranking, and institutional representation at the 2014 and 2016 library assessment conferences (lac).   at the time of this study, the university of chicago library was a member of two consortia: the ivy plus libraries (ivy plus) and the big ten academic alliance (btaa). consortial relationships are frequently the basis of institutional comparisons; however, the researchers felt it appropriate to expand the pool beyond geography or institutional stature with other metrics. the researchers reviewed rankings data made available by arl for fiscal years 2014 and 2015, and selected the following criteria as potentially meaningful for comparison: investment index ranking, total library expenditures, and total staff. institutions ranked in the five positions above and below the university of chicago library were added to the population.   finally, researchers looked to the attendee lists from recent library assessment conferences (lac) to identify other types of institutions with a demonstrated commitment to assessment. given the costs associated with sending staff to a national conference, the researchers felt that the expenditures associated with sending three or more staff members could be taken as an indication of institutional commitment to assessment. attendee rosters were procured from the lac websites for the 2014[1] and 2016[2] conferences.   these three sets of criteria yielded a population of 97 institutions, with several institutions included on the basis of multiple criteria.   methodology   the data for this study were collected between may 2016 and april 2017 through environmental scanning. content analysis was supported by the development of a rubric which was itself based on themes that emerged from the data.   as a method, “environmental scanning is not just a source of data on the external world...it provides that backdrop against which internal values may be clarified” (mitchell & witthus, 1991, p. 162). the first phase of the environmental scan tested the study’s hypothesis by seeking evidence of library-based assessment programs and public-facing data through a content analysis, or “a systematic and iterative review,” as described by saunders (2015, p. 287). researchers reviewed websites and other public-facing documents made available by the institutions included in the population seeking evidence of the existence of an established assessment program, with each researcher reviewing about one-third of the included institutions. evidence of assessment programs included the word ‘assessment’ or ‘evaluation’ in a job title or in the name of a department or committee; it could also take the form of an assessment webpage, libguide, or stand-alone website, as described by lewin & passonneau (2012, pg. 89-90). if evidence was not found by browsing a library’s website, researchers would use a site search utility (if available) to search for terms like ‘assessment’ or ‘evaluation’ on the website or in the library’s staff directory. any evidence identified was recorded; researchers also noted names of personnel holding assessment-type positions, where applicable.   researchers also looked for examples of public-facing data on the libraries’ websites. public-facing data included annual reports, strategic planning documents, visualizations, infographics, or fact sheets used to benchmark library progress, success, or impact to external stakeholders. if evidence was not found by browsing a library’s website, researchers would use a site search utility (if available) to search for terms like ‘report’, ‘annual report’, or ‘strategic plan’ on the website. if no evidence was found by searching the library’s website, researchers would repeat these searches in combination with the word ‘library’ or the library’s name using a site search utility (if available) on the parent institution’s website. any examples of these data or of other forms of strategic communication were recorded and linked to data collected for the library’s assessment program (if applicable).   the researchers then reviewed data found in order to better understand how the previously identified public-facing data were being used in library communications. because the population had a diverse range of communication practices, a rubric (see appendix) was created using grounded theory, a methodology that embraces “the discovery of theory from data systematically obtained from social research” (glaser & strauss, 2006, pg. 3). during an initial scan, the researchers’ comparative analysis generated five conceptual categories that appeared to be indicators for excellence in data reporting: accessibility, or ease of access[3] (e.g., reports are easily found on the library’s website), communication (e.g., information is clear and without jargon), data (e.g., reports include quantitative or qualitative data), documentation (e.g., reports are up-to-date and publicly available), and reporting (e.g., evidence of and access to historical reports). these criteria were chosen because the researchers were interested in investigating libraries’ varied use of qualitative and quantitative data in external communication, and because 47% of the population included no quantitative or qualitative data in reports, methods in which libraries were communicating success. each criterion was worth up to 3 points, allowing each institution to receive a maximum of 15 possible points     table 1 rubric scores by institutional assessment   meana medianb modec all libraries  (97) 8.7 10 10 libraries with assessment (52) 9.4 10 10 libraries without assessment (45) 7.3 9 0 a mean, the average of a group of numbers, is calculated by adding a group of numbers and then dividing by the count of those numbers. b median is the middle number of a group of number; that is, half the numbers have values that are greater than the median, and half the numbers have values that are less than the median. c mode is the most frequently occurring number in a group of numbers.       results   of the 97 institutions reviewed, the researchers identified 52 institutions with demonstrated investments in assessment as met the criteria for the first phase of content analysis: at least one position or title focused primarily on assessment (e.g., assessment coordinator, assessment data specialist), an assessment committee, or a department dedicated to assessment. the grades of the 52 institutions ranged from 0-15, the most common grade being 10 (n=10).   the 52 libraries identified as having investments in assessment consistently scored higher on the rubric than the 45 libraries without. while grades of this cohort also ranged from 0-15, these libraries received an average grade of 7.3, 2.1 points lower than libraries with assessment personnel. the most common grade assigned was zero.   researchers found that data sharing practices from different types of institutions varied widely. of the ivy plus members, 9 of 12 demonstrated a commitment to assessment in accordance with the study's criteria; these institutions received an average grade of 8.4. the same number of btaa libraries (out of 14) demonstrated a commitment to assessment; however, these libraries received an average grade of 10.3.[4]  although the same number of libraries in both consortia had dedicated resources to assessment, one consortium received a markedly higher average score as a result of the range and depth of data and documents made publicly available.     table 2 rubric scores by consortia     mean median mode all consortial libraries  (26) 8.4 10 0   btaa (14) 9.9 11 10   ivy plus (12) 6.7 7.0 0 consortial members with assessment (18) 9.4 10 10   btaa (9) 10.3 11 11   ivy plus (9) 8.4 9 9 consortial members without assessment (8) 5.6 4.5 0   btaa (5) 8.2 11 n/a[5]   ivy plus (3) 1.3 0 0     table 3 rubric scores by public, private, and arl affiliation     mean median mode all libraries  (97) 8.7 10 10   public (65) 9.2 10 10   private (32) 7.5 8 8 arl members (65) 9.1 10 10   public (45) 9.5 10 10   private (20) 8.1 8 8 non-arl members (32)         public 8.4 10.5 11   private 6.6 7.5 0       the notable discrepancy between ratings of ivy plus and btaa libraries inspired additional investigation into factors that might contribute to libraries’ propensity for data sharing. one possible explanation for this discrepancy could be that the majority of btaa institutions are publicly funded and so may be required by state law or mandate to make more financial and strategic planning data available to the general public. the majority of ivy plus institutions, on the other hand, are private, and so have no such mandate for financial transparency. another possible explanation for greater transparency in some institutions could be the requirement for annual data reporting to arl[6] that affects 67% of the institutions included in this study. might required reporting whether to taxpayers or professional organizations affect a library’s data sharing?   further analysis of library ratings by public, private, or arl affiliation determined that public institutions were much more likely to score highly on the study’s rubric than private institutions, regardless of arl membership. membership in an organization with (possible) mandated reporting may contribute to more information-sharing, as seen when the ratings of private arl members (average 8.1) are compared to private non-arl members (average 6.6). however, this type of organizational mandate does not consistently correlate with higher ratings, as can be seen by comparing public arl members (mode 10) and public non-arl members (mode 11). finally, the researchers saw that an analysis of libraries’ dedication to assessment as measured by funding for lac attendance correlated with higher grades on the study’s rubric; however, there was a notable discrepancy between lac 2014 attendees and lac 2016 attendees. libraries represented by three or more individuals at lac 2014 were rated much higher than the aggregate; they were also rated higher than institutions with three or more lac 2016 attendees. this could be because libraries that attended a conference three years ago have had time to expand or develop a culture of assessment and reporting; it could also be that the dramatic growth in lac attendance is also indicative of a wider range of experience with and commitment to assessment.   discussion   in total, 57% of libraries reviewed made some form of data available to the public. the most common form of data sharing was a facts and figures type-page on the library’s website. this type of page typically presented the library’s “tombstone statistics” data points regularly collected for external reporting, including titles or volumes held, classes taught, or gate counts.[7] it is likely that libraries share these types of pages because they are familiar library data points and are relatively easy to produce. as mcclure and samuels discovered 30 years ago, “the closer and more familiar a source is, the more it is likely to be used” (1985, p. 495). however, while libraries often provide “tombstone statistics” as a measure of library value, these data provide an incomplete picture of the library’s service to and impact on the campus, and can be incomprehensible to outside stakeholders.   an improvement on the “tombstone statistics” approach involved the use of data for (internal) benchmarking or (external) comparison.[8] some libraries use their “tombstone statistics” to demonstrate change over time; they may also use these standard data points to compare themselves to peer institutions that collect and report the same data. by establishing benchmarks, libraries are able to measure and communicate ways in which they are or are not achieving their goals. in this study, libraries that provided benchmarks or comparisons tended to receive higher grades, as the presence of benchmarks or comparisons by definition exemplified good communication.   the most robust and effective use of data observed in this study involved the use of data to tell broader stories about the library and its impact. the seven institutions receiving the highest scores this study made use of data —qualitative or quantitative —to tell such stories on their websites or in other reports, and by doing so, were able to effectively communicate the library’s impact on campus research, teaching, and learning. while the institutions tended to rely heavily on numbers to demonstrate impact, they provided context by also supplying narratives describing why the numbers mattered. these institutions often had a section of their website dedicated to assessment in the library which directed viewers to multiple years of archived documentation of library assessment initiatives. others included yearly initiatives in their long-term strategic plans and updated the status of these initiatives in subsequent annual reports. infographics complemented the text-based discussion of assessment and data-based decision making; this was particularly effective in illustrating the financial reasons behind reallocation of library funds to or away from collections budgets to meet other library service demands. [9]   limitations   this study emerged from an environmental scan conducted at the university of chicago library with the specific purpose of informing internal decision making related to the representation of the library’s assessment presence on its website. as a result, the first two criteria for determining the study’s population identified institutions that more closely align with the university of chicago library, rather than the average north american academic library.   similarly, the third criterion staff attendance at a professional conference makes assumptions about the institutions represented at this conference. first, while many institutions provide funding for their employees to attend such events, many individuals are required to pay at least some of the costs of attendance. while an institution may be willing to support this type of professional development, staffing needs may limit the number of individuals who are able to be away from the library at any a given time. additionally, many other reasons contribute to an individual’s decision or ability to attend a conference. finally, the lac is a relatively small conference, so while it was not clear from the lac website whether registration had reached capacity in 2014 or 2016, it is possible that the number of individuals in attendance was limited due to the size of the conference itself. as a result, while an institution’s representation at this conference in recent years can be taken as a measure of some institutions’ investment in assessment, it is an incomplete measure at best.   finally, the use of grounded theory for the development of this study’s rubric limits the generalizability of this study’s findings. as thomas and james note, the problem with grounded theory is that it is a scientific instrument similar to one’s everyday “practical syllogism” (2006, pg. 773). since the practice of developing theory from observation is highly subjective, this study’s rubric should not be utilized in other studies without further testing; similarly, the findings of this study should be treated as observations subject to further investigation.   areas for future research   this study used publicly-available data to make inferences about external factors affecting libraries’ information-sharing processes. while shared characteristics among institutions in this study’s population could be correlated with expanded sharing of data on library websites, none of these characteristics reflect internal institutional factors that contribute to this area of decision making. further research is needed to identify these factors and to explore their implications for information sharing in the larger academic library community. additionally, further research is needed to explore how the external factors identified in this study (e.g., consortial membership) shape the internal decision making around these processes.   conclusion   increasingly, libraries are relying on their assessment programs to collect the data needed to demonstrate the value libraries contribute to their institutions’ missions and goals. evidence of the movement can be seen through acrl’s impact of academic libraries, megan oakleaf’s recent work (oakleaf, 2016; oakleaf et al, 2017) on integrating the library in campus data collecting initiatives, and emerging papers considering the library’s role in protecting student data (jones & salo, 2017). however, this study demonstrates that a library’s investment in an assessment program does not guarantee that the data collected by such programs will be made available to external stakeholders.   this study sought to explore factors that influence the ways academic libraries choose to share data and other reports on their websites. while the researchers found a slight correlation between libraries’ investment in assessment and the presence of outward-facing reporting, the correlations were observably impacted by other factors. a library’s demonstration of data could be influenced by employees’ engagement in assessment projects, participation in a consortia that requires regular reporting, or the receipt of taxpayer funding. future studies might investigate which, if any, of these factors greatly increase or diminish the likelihood of data being made publicly available.   it is the researchers’ hope that observations gleaned from the content analysis can serve as a benchmark for measuring changes in library communication practices. slightly more than half of the libraries reviewed made data or strategic documents available on their websites. however, those institutions that made data or documents available frequently did so without providing meaningful context for external audiences, thus missing an important opportunity to articulate the value expressed in the data. there is clearly significant room for improvement.   references   glaser, b. d., & strauss, a. l. (2006). the discovery of grounded theory: strategies for qualitative research. new brunswick, nj: aldine transaction.   jones, k., & salo, d. (2017). learning analytics and the academic library: professional ethics commitments at a crossroads. college & research libraries 79(3), 304-323. https://doi.org/10.5860/crl.79.3.304   koufogiannakis, d. (2014). mcclure and samuels’ study on information sources used for decision making and the connection to organizational climate still resonates today. evidence based library and information practice, 9(4), 78-81. https://doi.org/10.18438/b8788q   lewin, h. s., & passonneau, s. m. (2012). an analysis of academic research libraries assessment data: a look at  professional models and benchmarking data. the journal of academic librarianship, 38(2), 85-93. https://doi.org/10.1016/j.acalib.2012.01.002   lundstrom, k., martin, p., & cochran, d. (2016). making strategic decisions: conducting and using research on the impact of sequenced library instruction. college & research libraries, 77(2), 212-226. https://doi.org/10.5860/crl.77.2.212   manzuch, z. & maceviciute, e. (2014). library user studies for strategic planning. information research, 19(4), 71-84.   mcclure, c. r. (1986). a view from the trenches: costing and performance measures for academic library public services. college & research libraries, 47(4), 323-336. https://doi.org/10.5860/crl_47_04_323   mcclure, c. r., & samuels, a. r. (1985). factors affecting the use of information for academic library decision making. college & research libraries, 46(6) 483-498. https://doi.org/10.5860/crl_46_06_483   mitchell, m., & witthus, r. w. (1991). planning for diversity: strategic planning for an urban academic library. journal of library administration, 13(3-4), 157-165. https://doi.org/10.1300/j111v13n03_12   oakleaf, m. (2016). getting ready & getting started: academic librarian involvement in institutional learning analytics initiatives. journal of academic librarianship, 42(4), 472-475. https://doi.org/10.1016/j.acalib.2016.05.013   oakleaf, m., whyte, a., lynema, e., & brown, m. (2017). academic libraries & institutional learning analytics: one path to integration. journal of academic librarianship, 43(5), 454-461. https://doi.org/10.1016/j.acalib.2017.08.008   paulus, a. r. (2014). using data to assess staffing and services: university of iowa main library. journal of access services, 11(3), 189-205. retrieved 19 march 2018 from eric database (ej1033784).   saunders, l. (2015). academic libraries’ strategic plans: top trends and under-recognized areas. the journal of academic librarianship, 41(3), 285-291. https://doi.org/10.1016/j.acalib.2015.03.011   saunders, l. (2016). room for improvement: priorities in academic libraries’ strategic plans. journal of library administration, 56(1), 1-16. https://doi.org/10.1080/01930826.2015.1105029     seago, b. l., schlesinger, j.b., & hampton, c.l. (2002). using a decade of data on medical student computer literacy for strategic planning. journal of the medical library association, 90(2), 202-209.   thomas, g., & james, d. (2006). reinventing grounded theory: some questions about theory, ground, and discovery. british educational research journal, 32 (6), 767-795. retrieved from http://www.jstor.org/stable/30032707 van house, n. a. (1989). output measures in libraries. library trends, 38(2), 268-279. retrieved from http://hdl.handle.net/2142/7661         appendix study rubric     excellent good satisfactory needs improvement accessibility[10] documents are easily found on the library's website (within 5 clicks) documents can be found on the library's website, but it takes some time (5+ clicks?) documents can be found by searching the library's website documents not available communication communication is clear and accessible for non-librarians (e.g. lack of jargon) communication is directed towards non-librarians, but contains some jargon library's message or assessment contains jargon and seems to be directed mainly to staff no direct message publicly available documentation most recent strategic directions and annual report publicly available, as well as archived documentation most recent strategic directions and annual report publicly available strategic directions or annual report publicly available; out of date documents not available data strategic directions or annual report uses qualitative and quantitative data to tell a story about the library's achievement or struggles draws links between qualitative/quantitative data collected by the library with strategic directions and/or annual report makes qualitative or quantitative data related to library assessment publicly available. data not available. reporting publicly available documents are up-to-date and there is evidence of historical reporting and evaluation. publicly available documents are up-to-date publicly available documents are 1 year or less out of date documents are not current     [1] http://libraryassessment.org/archive/2014-library-assessment-conference.shtml [2] http://libraryassessment.org/archive/2016-library-assessment-conference.shtml [3] ‘ease of access’ better conveys the concepts intended by ‘accessibility’, a term with a well-established meaning in the library community; however, the latter is included here as it appears in the original rubric (see appendix). [4] consortial numbers exclude the university of chicago. [5] each institution in this cohort received a different grade, making it impossible to determine a mode. [6] see http://www.arl.org/publications-resources/arlstatistics/terms/summary for more information. [7] for example, see “by the numbers” on the university of chicago library’s about the library webpage, accessed from https://www.lib.uchicago.edu/about/thelibrary/. [8] for example, see “ncsu libraries strategic plan fy14/fy16, accessed from https://www.lib.ncsu.edu/sites/default/files/files/images/ncsu_libraries_strategic_plan_fy14-fy16-062813final.pdf. [9] for example, see creighton university library’s “budget challenge” from their library assessment webpage, accessed from http://culibraries.creighton.edu/assessment/budget. [10] as noted before, ‘ease of access’ better conveys the concepts intended by ‘accessibility’; however, the latter is included here as it appears in the original rubric. evidence summary   students experience cognitive and emotional responses to academic library spaces   a review of: mei, x. y., aas, e., & eide, o. (2020). applying the servicescape model to understand student experiences of a norwegian academic library. library & information science research, 42(4), article 101051. https://doi.org/10.1016/j.lisr.2020.101051   reviewed by: hilary bussell assistant professor, research and education the ohio state university libraries columbus, ohio, united states of america email: bussell.21@osu.edu   received: 1 june 2021                                                                    accepted:  12 july 2021      2021 bussell. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29988     abstract   objective – to understand how the physical environment of an academic library influences user behaviour.   design – qualitative explorative.   setting – an academic library at a large university in norway.   subjects – twelve bachelor’s and master’s students at a business school.   methods – the researchers used a two-step approach, with the servicescape model from the marketing discipline serving as a theoretical framework. subjects completed several tasks involving drawing and elaborating on their usage of the library space, utilizing a bird’s-eye floor plan. this was followed by semi-structured interviews to explore how subjects use and experience the library facilities.   main results – students found it important to be physically comfortable and to have enough room for the items they need while studying. the library in this study was seen more as a place for studying than for social interactions, and while some subjects reported being motivated by seeing students around them studying, others said they are distracted by having other students in their sightline. the time of the semester appeared to influence user experience and satisfaction with the library space, with spaces conducive to group work desired at some points in the semester and with single seating preferred when individual exams are taking place.   conclusion – the library’s physical environment triggers cognitive and emotional responses in users. these responses influence how frequently they visit the library and how well they are able to concentrate while there. because academic library spaces have an impact on student learning, it is important to design libraries with user comfort in mind. libraries should accommodate the different ways students work throughout the semester by providing flexible study space configurations.   commentary   this study uses the servicescape model to understand how academic library spaces influence student learning. developed in marketing literature and originally focused on customers and employees, servicescape is a framework for understanding how the design of physical spaces influences user behaviours (bitner, 1992). the model draws a connection between the physical and ambient conditions of a space, the cognitive and emotional responses elicited in users, and overall perceptions of the services received within the space (hooper, coughlan, & mullen, 2013). though a great deal of scholarship exists on the role of library spaces and services in student learning, the servicescape model has seen little application in library and information studies. lin & chiu (2012) used the photo-elicitation method to understand college students’ preferences regarding different aspects of an academic library’s servicescape. in the current study, the authors extend the application of the servicescape model by exploring not only how academic library spaces influence user behaviour and satisfaction, but also how they impact student learning experiences.   the study was assessed using the casp qualitative checklist (critical appraisal skills programme, 2018). the study was particularly strong in its in-depth description of the data analysis process. the authors explained recruitment and data collection procedures clearly, as well as the rationale for employing an exploratory qualitative method. they used visualizations effectively to illustrate the qualitative coding process and to map the findings onto the servicescape model. though the bird’s eye view floor plan with relevant tasks was included in the appendix, the semi-structured interview guide was not. information on sample size determination, whether the data collection and analysis activities overlapped chronologically, and whether they reached data saturation were also missing from this article. despite these minor weaknesses, the detailed description of methods and analysis procedures provides a clear blueprint for anyone who might be interested in conducting a similar study.   this study’s insights into how physical discomfort and distractions cause negative cognitive and emotional responses in students are important considerations for anyone involved with designing library spaces. the research provides evidence for prioritizing comfort and flexibility over aesthetic considerations. by illustrating how the servicescape model can provide insight into student learning experiences, this study demonstrates its usefulness for academic libraries. the authors acknowledge where their findings diverge from previous studies (e.g., attitudes towards the use of library for studying versus socializing) and suggest that similar studies be carried out with other types of users and in academic libraries where there are fewer nearby options for socializing. in addition to these recommendations, it is also important to consider the role cultural differences play in student library usage (wertman, campbell, blummer, & kenton, 2018). future studies conducted in other geographical locations as well as other types of libraries could help to determine the wider applicability of this study’s findings.   references   bitner, m. j. (1992). servicescapes: the impact of physical surroundings on customers and employees. journal of marketing, 56(2), 57–71. https://doi.org/10.2307/1252042   critical appraisal skills programme. (2018). casp qualitative checklist. https://casp-uk.net/wp-content/uploads/2018/01/casp-qualitative-checklist-2018.pdf   hooper, d., coughlan, j., & mullen, r. m. (2013). the servicescape as an antecedent to service quality and behavioral intentions. journal of services marketing, 27(4), 271–280. https://doi.org/10.1108/08876041311330753   lin, y. c., & chiu, m. h. (2012). a study of college students' preference of servicescape in academic libraries. journal of educational media & library sciences, 49(4), 609–636. http://joemls.dils.tku.edu.tw/en/issues/detail/?articleid=49405    wertman, l, campbell, c. j., blummer, b, & kenton, j. m. (2018) optimizing library services — international students and academic libraries: identifying themes in the literature from 2001 to the present. against the grain, 30(2), 51–53. https://doi.org/10.7771/2380-176x.8059   research article   reference mode preferences of community college (two-year) and four-year college students: a comparison study   john carey head librarian health professions library hunter college, city university of new york new york, new york, united states of america email: john.carey@hunter.cuny.edu   ajatshatru pathak health and informatics librarian health professions library hunter college, city university of new york new york, new york, united states of america email: apathak@hunter.cuny.edu   received: 21 july 2016   accepted: 2 jan. 2017           2017 carey and pathak. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – the purpose of this study was to examine the reference service mode preferences of community college (two-year) and four-year college students.   methods – the researchers administered a paper-based, face-to-face questionnaire at two institutions within the city university of new york system: hunter college, a senior college, and queensborough community college, a two-year institution. during the summer of 2015, the researchers surveyed 79 participants, asking them to identify their most and least preferred mediums for accessing library reference services.   results – nearly 75% of respondents expressed a preference for face-to-face reference, while only about 18% preferred remote reference services (online chat, e-mail, text message, and telephone). close to 84% of the participants cited remote reference services as their least preferred modes and slightly more than 10% said this of face-to-face. the data reveal a widespread popularity of face-to-face reference service among all types of participants regardless of institutional affiliation, age, gender, academic level, field of study, and race or ethnicity.   conclusion – this study suggests that given the opportunity academic library users will utilize face-to-face reference service for assistance with research assignments. academic libraries at both two-year and four-year institutions might consider assessing user views on reference modes and targeting support toward services that align with patron preferences.       introduction   this study augments the existing literature on user perceptions of reference services by not only analyzing the reference medium preferences of students from different academic disciplines, levels, and demographic groups but also by comparing the preferences of students from two-year institutions against those of four-year college students. this topic bears direct relevance for librarians within the public city university of new york (cuny) system, where every year thousands of students transfer from the numerous community colleges to the senior colleges, a transition which can pose its own challenges. furthermore, students from any unit of cuny have reciprocal library access privileges throughout the system, so that librarians at all campuses find themselves working with current or former community college students. thus, the authors undertook this study in an effort to understand better the potentially differing needs and impressions of the two-year and four-year students who make up the libraries’ patron base.   the researchers conducted this study at two institutions, hunter college and queensborough community college (qcc), both of which are units of the cuny system. hunter is a four-year liberal arts college located in manhattan with a current enrollment of nearly 23,000; it offers undergraduate and graduate degrees in “more than 170 areas of study” (hunter college, 2016). queensborough community college is a two-year college in queens, ny offering associate degrees and certificates in a wide variety of disciplines, including business, health professions, and sciences, with a fall 2015 enrollment of 15,493 degree and non-degree students (queensborough community college, 2015). hunter has four branch libraries located across three campuses while qcc has one library on its campus.   the hunter and qcc libraries offer face-to-face, e-mail, telephone, and online chat reference services. hunter offers 24/7 chat service via the questionpoint reference cooperative, whereas qcc provides chat service during selected hours only and uses the libraryh3lp software. in addition, qcc offers text message reference service to its patrons, while hunter discontinued its text service after spring 2015 (p. swan, personal communication, july 18, 2016). both libraries provide information on their websites about what reference services are available and how to utilize them. for purposes of this study, the authors define face-to-face reference as service that allows a patron to obtain assistance in person from a credentialed reference librarian. e-mail reference service involves “either e-mailing the reference desk via an online e-mail form . . . or contacting a departmental liaison directly” while telephone reference involves users “calling the general reference desk or . . . departmental liaison directly” (chow & croxton 2012, p. 249). text message reference service allows patrons with mobile phones to send text messages directly to an account operated by the library.   literature review numerous researchers have investigated the reference medium preferences of academic library users. a recent study at the university of north carolina at greensboro employed online survey, focus group, and interview data collection methods with 936 participants to examine the reference-seeking preferences of undergraduate students, professors, librarians, and staff (chow & croxton, 2012). the investigators found that face-to-face was the most popular reference mode among respondents, noting that “[d]espite the growing plethora of ways for library users to seek help through technology mediums, face-to-face would appear to still be the preferred method of choice . . . in a university academic setting” (chow & croxton, 2012, p. 259). the authors also reported that undergraduate students preferred online chat and text reference modes, and acknowledged the impact of age on participants’ reference medium choices.   previous research suggests broad trends in favor of in-person reference service. a washington state university study surveying reference service use among 276 students and faculty found that patrons strongly preferred face-to-face reference, with 49% of faculty members, nearly 77% of undergraduate students, and 65% of graduate students citing it as their favourite mode (johnson, 2004). granfield and robertson (2008) addressed information-seeking preferences in a study of 348 academic library patrons conducted at two canadian institutions, using focus group and survey methods. the authors found a strong preference for face-to-face reference among college student patrons, observing that “the reference desk continues to be the most popular method of getting help in the library” (granfield & robertson, 2008, p. 51). in a study at the university of illinois at chicago analyzing the reference transaction data of a health science library, the investigators noted that patrons used the reference desk more than they did web-based services, and that “traditional reference remains the core of information services in this health science library” (de groote, hitchcock, & mcgowan, 2007, p. 23). a related study found that 82% of undergraduate students and 86% of graduate students preferred “ask[ing] their questions in-person” (de groote, 2005, p. 20).   despite the attention that these and other studies have brought to the topic of reference medium preferences among academic library users, some questions remain open. notably, the current investigators found a gap in the research literature with regard to drawing comparisons between students from two-year and four-year institutions. moreover, the studies cited above did not seek to distinguish variations in preferences according to factors such as academic major or year of study. the current study seeks to address these and other omissions.   aims   the primary goal of this study was to examine the reference service preferences of two-year college students in comparison with those of four-year students. this paper will examine the following research questions:   ·         rq 1. which reference mediums do community college (two-year) students prefer? ·         rq 2. which reference mediums do four-year college students prefer? ·         rq 3. which reference mediums do students pursuing different academic disciplines prefer? ·         rq 4. which reference mediums do undergraduate and graduate students prefer? ·         rq 5. which reference mediums do undergraduate freshman, sophomore, junior, and senior students prefer? ·         rq 6. what are reference preferences of students belonging to different age, gender, and racial/ethnic groups?   methods   research methodology, instrument, and pilot study   this study adopted a survey research method. after reviewing questionnaires from previous studies examining the reference mode preferences of academic library users (chow & croxton, 2012; johnson, 2004; granfield & robertson, 2008), the researchers developed an instrument of 15 items (see the appendix). the survey instrument contained 13 closed and two open questions, sequenced from simple to more complex. the questions were neutral, balanced, specific, easy to understand, and were written in complete sentences.   the first seven questions asked participants about their institutional affiliation, academic level, year of study, gender, age, race, and academic major. subsequent questions asked respondents whether they were familiar with the reference services of their college libraries and if they had used those services. participants then selected their most and least preferred reference mediums via multiple choice; this article focuses on the responses to these questions. the instrument also included a multiple choice question asking about preference for electronic versus print books, as well as two open questions soliciting suggestions for improvements to reference services and any other comments, but these did not yield findings relevant to the scope of this paper.   the investigators pre-tested the questionnaire with the help of two reference librarians at hunter college and conducted a pilot study with five students (three undergraduate and two graduate, also from hunter) to test the validity and reliability of the instrument. the investigators considered the pilot study participants’ suggestions and revised the questionnaire accordingly.   data collection and analysis   the researchers distributed paper-based questionnaires to hunter and qcc students during the summer of 2015. they recruited participants in cafeterias, hallways, in front of classrooms, outside library entrances, and outside campus buildings. participants who completed the survey received one dollar in appreciation of their time. the researchers orally informed subjects about the goal of this study, requested consent to participate, and provided written copies of the informed consent script if requested. the investigators analyzed the results using spss.   participant profiles   the investigators approached 100 students, of which 79 agreed to participate in the study. the completion rate of the survey was 100%. of the 79 participants, 49 (62%) were hunter students and 19 (nearly 24%) were qcc students. the remaining 11 participants (14%) were affiliated with other institutions but taking summer courses at hunter or qcc. among study participants, 81% (n=64) were undergraduate, 6.3% (n=5) graduate, 3.5% (n=3) continuing education, 2.5% (n=2) visiting students, and 6.4% (n=5) other (for example, alumni, non-degree, or ba/ma students). among undergraduate students, approximately 13% (n=10) were freshmen, 24% (n=19) sophomores, close to 17% (n=13) juniors, and 29% (n=23) seniors. nearly 18% (n=14) could not be classified in the aforementioned categories.   of the study participants 63% (n=50) identified as female and nearly 37% (n=29) as male. an overwhelming number were below 24 years of age (72%, n=57). close to one quarter (n=19) belonged to the 25-34 age group, and 1.3% (n=1) each were from the 45-54 and 55-64 age groups. the study subjects were diverse in terms of race and ethnicity: 20.3% (n=16) were white, 27.8% (n=22) black, 13.9% (n=11) hispanic, 25.3% (n=20) asian, 3.8% (n=3) multiracial, and 9% (n=7) from other racial and ethnic groups.     figure 1 participants’ fields of study by percentage.     participants represented a wide variety of academic majors: 30.4% (n=24) of respondents were pursuing social science degrees, 24.1% (n=19) natural sciences, 16.5% (n=13) health sciences, 7.6% (n=6) arts or humanities, 5.1% (n=4) mathematics or statistics, and 3.8% (n=3) education. the remaining 12.7% (n=10) indicated other majors. (see figure 1.)   a strong majority of the respondents (nearly 79%, n=62) reported that they were aware that their college library offered reference services, but 21% (n=17) said they were not.   results   major findings   overall, face-to-face reference emerged as a clear favourite among participants, while telephone reference was consistently the least favoured—nearly 75% (n=59) of the respondents reported that face-to-face was their most preferred reference mode, and approximately 34% (n=27) identified telephone reference as their least preferred. nonetheless, eight respondents (10.1%) still cited face-to-face as their least preferred medium. other noteworthy findings include:   online chat and e-mail reference both had higher negative than positive ratings. chat was most preferred by 11.1% (n=9) of respondents and least preferred by roughly 13%; for e-mail this difference was greater, with 5.1% (n=4) of participants citing it as their favourite mode and 25.3% (n=20) as their least favourite. similarly, text message reference was the least favourite of nine respondents (11.4%) and the favourite of only one (1.3%). none of the participants favoured telephone reference. six participants (7.6%) said they did not use reference services.   for an overview of most and least preferred reference modes please see figures 2 and 3.   figure 2 most preferred reference service modes of all participants.     figure 3 least preferred reference service modes of all participants.   rq 1.     which reference mediums do community college students prefer?   approximately 74% (n=14) of community college students reported that when finding scholarly resources such as books or journal articles for research papers or other academic assignments, they prefer face-to-face reference help. one community college student (5.3%) preferred e-mail reference and one chat. none favoured telephone or text message reference mediums, and close to 16% (n=3) said they did not use any type of reference service. (see figure 4.)   equal numbers of community college students cited telephone and e-mail reference as their least preferred mediums (26.3%, or n=5, for each). four community college subjects (21.1%) indicated face-to-face as their least preferred medium, one (5.3%) cited text message, and one had not used reference services. (see figure 5.)     figure 4 most preferred reference service modes of community college students.     figure 5 least preferred reference service modes of community college students.     rq 2. which reference mediums do four-year college students prefer?   nearly 70% (n=34) of four-year college students reported that they preferred face-to-face reference for assistance with finding resources such as books or journal articles for research assignments. eight four-year students (16.3%) preferred chat reference, followed by e-mail at 6.1% (n=3). an equal percentage (6.1%, n=3) of four-year students said they do not use reference services. only one four-year student preferred text message reference (2%), and none chose telephone reference (see figure 6).   with regard to least preferred reference mediums, close to 39% (n=19) of four-year students selected telephone, followed by e-mail reference at 26.5% (n=13), text message at 14.3% (n=7), and chat at 10.2% (n=5). despite its status as the most preferred reference medium among this cohort, face-to–face nevertheless emerged as the least favourite of 6.1% of respondents (n=3). two four-year college students reported that they do not use reference service. (see figure 7.)     figure 6 most preferred reference service modes of four-year college students.     figure 7 least preferred reference service modes of four-year college students.     rq 3. which reference mediums do students from different academic disciplines prefer?   the researchers also attempted to determine the reference mode preferences of students according to field of study as reported by the participant (two respondents declined to identify an academic major). overall, majorities of students across all disciplines preferred face to-face reference, while telephone, e-mail, and chat reference emerged as the mediums most likely to be cited as participants’ least favourite. table 1 lists complete data for most preferred reference modes by field of study; table 2 lists data for least preferred reference mode.     table 1 most preferred reference service modes by field of study a a number of respondents/percentage     table 2 least preferred reference service modes by field of study b b number of respondents/percentage     rq 4. which reference mediums do undergraduate and graduate students prefer?   the survey also collected data on preferences according to graduate or undergraduate status. as with other groups, face-to-face was the preferred medium for majorities of both undergraduate (71.9%, n=46) and graduate (80%, n=4) students surveyed. e-mail and chat reference services fared better with undergraduate students, at 6.3% (n=4) and 11% (n=7) respectively, than they did with graduate students (n=0 for both). again, no participant cited telephone as a favoured medium. see figures 8 and 9 for complete findings regarding most and least preferred modes according to academic level.   rq 5. which reference mediums do undergraduate freshman, sophomore, junior, and senior students prefer?   within the undergraduate population, the researchers further attempted to determine preferences according to year of study. seventy percent (n=7) of undergraduate freshman students, nearly 79% (n=15) of sophomores, 69.2% (n=9) of juniors, and close to 74% (n=17) of seniors chose face-to-face as their most preferred reference medium. telephone was the least popular medium for all groups except sophomores, who indicated that they disliked e-mail more by a difference of 15 percentage points. with regard to library usage, seniors were the most likely to report that they made use of reference services. please see tables 3 and 4 for complete data on preferences according to year of study.     figure 8 most preferred reference service modes by academic level.     figure 9 least preferred reference service modes by academic level.     table 3 most preferred reference service modes by year of study c reference medium freshman sophomore junior senior face-to-face 7(70.0) 15(78.9) 9(69.2) 17(73.9) telephone 0(0.0) 0(0.0) 0(0.0) 0(0.0) e-mail 2(20.0) 0(0.0) 2(15.4) 0(0.0) online chat 0(0.0) 1(5.3) 0(0.0) 6(26.1) text message 0(0.0) 1(5.3) 0(0.0) 0(0.0) i do not use reference services 1(10.0) 2(10.5) 2(15.4) 0(0.0) c number of respondents/percentage   table 4 least preferred reference service modes by year of study d reference medium freshman sophomore junior senior face-to-face 2(20.0) 2(10.5) 0(0.0) 2(8.7) telephone 4(40.0) 4(21.1) 8(61.5) 8(34.8) e-mail 2(20.0) 7(36.8) 3(23.1) 6(26.1) online chat 1(10.0) 2(10.5) 1(7.7) 3(13.0) text message 1(10.0) 1(5.3) 0(0.0) 4(17.4) i do not use reference services 0(0.0) 2(10.5) 1(7.7) 0(0.0) d number of respondents/percentage   rq 6. what are the reference medium preferences of students belonging to different age, gender, and racial or ethnic groups?   the survey captured data regarding the preferences of participants according to demographic characteristics such as age, gender, and identification with a particular racial or ethnic group, presented below.   gender male participants showed slightly greater preference for face-to-face reference than did female respondents (86% as compared with 68%); also, more female students identified face-to-face as their least favourite medium than did males (12% vs. 7%). no one of either sex preferred telephone reference, although e-mail drew an even stronger negative response among male students. female students were slightly more likely to state that they do not use reference services. see figures 10 and 11 for complete data on most and least preferred reference modes by gender.     figure 10 most preferred reference service modes of male and female participants.     figure 11 least preferred reference service modes of male and female participants.     age nearly 79% (n=45) of respondents below 24 years of age and 68.4% (n=3) from the 25–34 age group selected face-to-face as their most preferred reference mode, as did the sole participant from the 55–64 age group. no participant from any age group favoured telephone reference, which was cited as least preferred by 35.1% (n=20) of the respondents under age 24, 31.6% (n=6) of those in the 25–34 age group, and 100% (n=1) of the 45–54 age group. figures 12 and 13 display the complete findings for most and least preferred reference modes by age group.     figure 12 most preferred reference service modes by age group.     figure 13 least preferred reference service modes by age group.     race/ethnicity a majority of students within each racial or ethnic group selected face-to-face reference as their preferred medium. this was true of approximately 69% (n=11) of white, nearly 73% (n=16) of black, 63.6% (n=7) of hispanic, and 80% (n=16) of asian students, as well as 70% (n=7) of those identifying as other races or ethnicities. however, african-american respondents were more likely than other groups to cite face-to-face as their least preferred, with 27.3% (n=6) indicating this. please see tables 5 and 6 for complete data regarding most and least preferred reference modes per group.     table 5 most preferred reference service modes by race/ethnicity e reference medium white black hispanic asian other face-to-face 11(68.8) 16(72.7) 7(63.3) 16(80) 7(30.0) telephone 0(0) 0(0.0) 0(0.0) 0(0.0) 0(0.0) e-mail 1(6.3) 0(0.0) 0(0.0) 2(10) 1(10.0) online chat 2(12.5) 4(18.2) 2(18.2) 0(0.0) 0(0.0) text message 1(6.3) 0(0.0) 0(0.0) 0(0.0) 0(0.0) both face to face and e-mail 0(0.0) 0(0.0) 1(9.1) 0(0.0) 0(0.0) i do not use reference services 1(6.2) 2(9.1) 1(9.1) 2(10.0) 0(0.0) e number of respondents/percentage   table 6 least preferred reference service modes by race/ethnicity f reference medium white black hispanic asian other face-to-face 0(0.0) 6(27.3) 0(0.0) 1(5) 1(10.0) telephone 6(37.5) 5(22.7) 6(54.5) 7(35.0) 3(30.0) e-mail 5(31.3) 5(22.7) 4(36.4) 5(25.0) 0(0.0) online chat 0(0.0) 2(9.1) 1(9.1) 5(25.0) 2(20.0) text message 4(25) 2(9.1) 0(0.0) 0(0.0) 3(30.0) both e-mail and online chat 0(0.0) 1/(4.5) 0(0.0) 0(0.0) 0(0.0) i do not use reference services 1(6.3) 1(4.5) 0(0.0) 2(10.0) 0(0.0) did not answer 0(0.0) 0(0.0) 0(0.0) 0(0.0) 1(10.0) f number of respondents/percentage   discussion   the findings of this study reveal noteworthy patterns concerning academic library users’ reference service preferences. foremost, regardless of institutional affiliation, academic major, undergraduate or graduate status, year of study, gender, age, and race or ethnicity, respondents overwhelmingly preferred face-to-face reference over remote mediums. these findings run counter to widely held perceptions of college students as being highly drawn toward technology, and could indicate that either students do not see added value in virtual reference services or their use of technology in other spheres does not necessarily carry over to academic tasks.   only some of the results of this survey are consistent with those of earlier studies. as in johnson’s (2004) study, face-to-face emerged as the most preferred reference mode of undergraduate and graduate students, and telephone was the least preferred medium of undergraduates. the current results, however, differ from johnson’s in that graduate students did not display any considerable interest in e-mail reference. also, in johnson’s (2004) study a high percentage of undergraduate and graduate students predicted that remote reference services would be “heavily used in ten years” (p. 241), a finding which this study does not corroborate. similar to granfield and robertson (2008), data from the current study reveal the popularity of in-person reference among college library users, but do not support that study’s findings that graduate students prefer virtual reference modes. this is surprising given that many researchers have suggested that “graduate students seem more likely to conduct their research outside the library” (granfield & robertson, 2008, p. 44).   finally, it must be noted that 21% of all respondents (n=17) indicated that they were not aware that their institution’s library offered reference services at all. this serves as a sobering reminder that librarians can take nothing for granted regarding patron awareness of even basic library services. clearly, a need exists among the population sampled here for outreach and education about reference services.   limitations and future research   this study had several limitations. to begin with, the researchers conducted the survey during the summer, when only a minority of the student body is on campus. as with many colleges and universities in north america, hunter and qcc organize the academic year into a 16-week semester during the fall and another during the spring, with shorter terms of anywhere from three to twelve weeks held during the january intersession and the summer months. according to the cuny office of institutional research and assessment (2016a), total enrollment for degree-seeking students at hunter was 15,204 for the spring 2015 semester and 15,465 for the fall; at qcc, these figures were 13,272 for spring 2015 and 13,692 for the fall. during the sessions comprising summer 2015, however, enrollment dropped to 4,998 degree-seeking students at hunter and 4,805 at qcc. the demographic data available does not indicate any great difference with regard to gender or ethnic characteristics between the cohort of students who took courses during the summer of 2015 and those enrolled during spring and fall semesters. for instance, women made up 64.6% of hunter undergraduates in spring, 65.7% in summer, and 64.7% in fall 2015 (cuny office of institutional research and assessment, 2016a). percentages of students enrolled by ethnic group similarly vary within a limited range only. the only marked difference that emerges between regular semester and summer-session students is in the area of fullversus part-time status. in spring 2015, hunter had 11,234 full-time undergraduate degree-seeking students and only 3,970 part-time; at qcc, this figure was 8,706 full-time and 4,566 part-time. fall 2015 showed a similar composition of 11,942 full-time and 3,523 part-time students at hunter, with 9,252 full-time and 4,440 part-time students at qcc. during the summer 2015 sessions, however, this pattern was reversed, with 4,846 undergraduate degree-seeking part-time students at hunter and only 152 full-time, and 4,747 part-time versus 58 full-time students at qcc (cuny office of institutional research and assessment, 2016b). thus, part-time students are heavily over-represented in the summer population, which may limit the applicability of this study to the overall student body. the exact make-up of the group sampled in this study remains unknown because the survey instrument did not include a question about fullor part-time status.   apart from the timing of the study, the small sample size and convenience sampling method might limit the generalizability of these findings to other college library users, and the reliance on self-reported data weakens the validity of the findings. with regard to the academic level of certain participants, the study failed to capture some pertinent data: there were 14 undergraduate participants who selected neither freshman, sophomore, junior, nor senior status, yet the survey instrument offered only those four options with no mechanism to indicate a different status. thus, data on the academic level of almost 18% of undergraduate respondents went unrecorded. finally, due to a misprint on the instrument used, the 24-and-under age group was mislabeled as “under 24,” which might have created confusion for some respondents and introduced ambiguity into the results concerning ages of participants.   despite its limitations, this study offers opportunities for further investigation. researchers could replicate this study with larger, randomized samples or expand it to include students from both public and private institutions. future studies could employ interview and observation methods to gain a deeper understanding of students’ reference-seeking preferences and behaviors, or examine the impact of factors not considered here such as fullor part-time status, daytime or evening attendance, use of mobile devices, and english language skills. future researchers may also find it fruitful to more deeply investigate some of this study’s findings regarding demographic groups (for instance, that african-american students are less likely to prefer face-to-face interaction with a librarian, or that female students are less likely to use reference services) to determine whether these results reflect any broader trends.   conclusion   the researchers found no marked differences between students at two-year and four-year institutions with regard to reference mode preferences. rather, as far as this survey could determine, community college students largely share the attitudes of students at the senior colleges when accessing reference services. nonetheless, these findings help illuminate the experiences of both types of students at large, public, urban campuses and may help librarians better support community college students making the transition to a four-year institution. moreover, knowing the practices and preferences of such patrons can help librarians situate reference services within a context more likely to maximize their use and relevance, thereby forging stronger connections with users.   libraries today have much to gain by developing such connections. as thorpe, lukes, bever, and he note, academic libraries “face increasing competition for institutional funding and student attention” along with growing pressure to demonstrate the contribution of library services to student success (thorpe, lukes, bever & he 2016, p. 387). at the same time, many librarians struggle to counter perceptions of decreased relevance in an age of free online resources and sophisticated search algorithms. under these circumstances, academic libraries at both two-year and four-year institutions might enhance their impact by assessing user views on reference modes and targeting support toward those programs that more closely align with patron practises and preferences.   references   chow, a. s., & croxton, r. a. (2012). information-seeking behavior and reference medium preferences: differences between faculty, staff, and students. reference & user services quarterly, 51(3), 246–262.   cuny office of institutional research and assessment. (2016a). total enrollment by undergraduate and graduate level, gender and college: fall 2015. retrieved from http://www.cuny.edu/irdatabook/rpts2_ay_current/enrl_0023_uggr_gen_hist.rpt.pdf   cuny office of institutional research and assessment. (2016b). undergraduate enrollment by degree/non-degree status, full-time/part-time attendance, and college: fall 2015. retrieved from http://www.cuny.edu/irdatabook/rpts2_ay_current/enrl_0020_dstat _ftpt_hist.rpt.pdf   de groote, s. l. (2005). questions asked at the virtual and physical health sciences reference desk: how do they compare and what do they tell us? medical reference services quarterly, 24(2), 11-23. doi:10.1300/j115v24n02_02   de groote, s. l., hitchcock, k., & mcgowan, r. (2007). trends in reference usage statistics in an academic health sciences library. journal of the medical library association, 95(1), 23–30.   granfield, d., & robertson, m. (2008). preference for reference: new options and choices for academic library users. reference & user services quarterly, 48(1), 44–53.   hunter college. (2016). welcome to hunter college. retrieved from http://www.hunter.cuny.edu/abouthunter. accessed on 7 february, 2017   johnson, c. m. (2004). online chat reference: survey results from affiliates of two universities. reference & user services quarterly, 43(3), 237–247.   queensborough community college. (2015). a stellar year in achievement: annual report. retrieved from http://www.qcc.cuny.edu/advancement/qcc-annualreport/index.html   thorpe, a., lukes, r., bever, d. j., & he, y. (2016). the impact of the academic library on student success: connecting the dots. portal: libraries and the academy 16(2), 373–392.   appendix   questionnaire: library reference services   which college do you attend? please select one of the following: a.       hunter college b.       queensborough community college c.        another cuny college d.       a private college e.       other college f.        i do not attend a college   what is your academic level? please select one of the following: a. undergraduate student b. graduate/professional student c. continuing education student d. alumni e. other (please specify)   if you are an undergraduate student, please select one of the following. otherwise skip this question. a.       freshman b.       sophomore c.        junior d.       senior   what is your gender? please select one. a.       male b.       female c.        transgender   approximately, what is your age? please select one. a.       under 24 b.       25-34 c.        35-44 d.       45-54 e.       55-64 f.        65 and up   how do you identify yourself? please select one. a.       white b.       black c.        hispanic d.       american indian or alaska native e.       asian f.        arab/middle eastern g.       native hawaiian or other pacific islander h.       multiracial i.         other (please specify)   what is your major or intended major or in what subject area do you hope to obtain a degree or have a degree? please indicate:    how often do you go to your college library? a.       several times in a week b.       once a week c.        once a month d.       more than one time in a month e.       rarely visit library   reference services in libraries assist people to find information that they need. are you aware that your college library offers reference services? a.       yes b.       no   have you used the reference services of your college library? a.       yes b.       no   if you need help finding materials (e.g., books, research articles, and other items) for any academic purpose (e.g., research papers, assignments, etc.) which one of the following reference services would you prefer to use first: a.       face-to-face reference at the reference desk b.       telephone reference c.        e-mail reference d.       online chat reference e.       text message reference service f.        i do not use reference services    what is your least preferred reference medium for the purpose of finding materials (e.g., books, research articles, and other items) for any academic purposes (e.g., research papers, assignments, etc.)? please select one of the following: a.       face-to-face reference at the reference desk b.       telephone reference c.        e-mail reference d.       online chat reference e.       text message reference service f.        i do not use reference services    do you prefer: a.       electronic (e-books) b.       print books c.        do not know d.       none   any suggestions for improving reference services (e.g., skype video reference, make an appt. with librarians):   any other comments and/or suggestions:                 research article   librarians’ participation in the systematic reviews published by iranian researchers and its impact on the quality of reporting search strategy    rogheyeh eskrootchi[1] associate professor department of medical library and information science school of health management and information sciences iran university of medical science tehran, iran email: eskrootchi.r@iums.ac.ir   azita shahraki mohammadi ph.d. candidate in medical librarianship and information sciences school of health management and information sciences iran university of medical sciences tehran, iran email: shahraki.a@iums.ac.ir   sirous panahi assistant professor health management and economics research center department of medical library and information science school of health management and information sciences iran university of medical sciences tehran, iran email: panahi.s@iums.ac.ir   razieh zahedi ph.d. candidate in medical librarianship and information sciences school of health management and information sciences iran university of medical sciences tehran, iran email: zahedi@iums.ac.ir   received: 20 july 2019                                                               accepted: 5 feb. 2020      2020 eskrootchi, mohammadi, panahi, and zahedi. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29609     abstract   objective – the validity of the results from systematic review studies depends largely on the implementation and the reporting of the search strategy. using an experienced librarian can greatly enhance the quality of results. thus, the present study aimed to investigate the relationship between the librarian’s participation and the quality of reporting search strategy in systematic reviews published by iranian researchers in medical fields.   methods – three databases were searched to identify the systematic review studies conducted by iranian researchers from 2008 to 2018. a total of 310 studies were selected using systematic random sampling, and the quality of their search strategy reports was reviewed by the institute of medicine checklist. a short questionnaire about the librarians’ participation in the search strategy of these studies was sent to the corresponding authors of the selected studies. a total of 229 questionnaires was returned. the data obtained from the questionnaire about the librarians’ participation in reporting search strategy in systematic review studies and also from the evaluation checklist for reporting search strategy in systematic review studies were analyzed by descriptive and inferential statistics.   results – the mean value of the evaluation checklist for reporting search strategy in systematic review studies was low. the librarians’ participation rate for these studies was 13.6%. no meaningful relationship was found between the librarians’ participation and the mean value of the evaluation checklist for reporting search strategy of systematic review studies. however, an investigation of the relationship between each of the items in the evaluation checklist for reporting search strategy in systematic review studies and librarians’ participation as the corresponding author or a member of the research team showed a meaningful relationship in five items.   conclusion – the results showed that the quality of reporting the search strategies in systematic reviews was low and the librarians’ participation in designing and reporting the search strategy in systematic reviews was limited. the authors of the systematic review studies, as well as the journals’ editors and referees, need to pay more careful attention to reporting the search strategy exactly and comprehensively. employing librarians in this area can have a major impact on this part of systematic review studies.     introduction   a systematic review study is a valuable research tool for collecting valid evidence to develop evidence based guidelines, plan decisions, and inform future studies (patrick et al., 2004). such studies can offer some important advantages: synthesizing large bodies of data, comparing as well as evaluating the results obtained by prior research, eliminating biased inferences, and finally, drawing more compelling conclusions related to the research questions (liberati & taricco, 2010). a systematic and comprehensive search is crucial for any systematic review (liberati et al., 2009). a weak search strategy may not find all eligible studies. a weak report, in turn, makes it difficult to determine whether the search itself has been inefficient or the report has been poorly presented (koffel, 2015). researchers need to present a comprehensive report of their search strategy, as an accurate and complete report of the search strategy can be seen as a criterion for evaluating the quality, validity, and methodology of the report in systematic reviews (moher & tsertsvadze, 2006).   to carry out a meticulous and comprehensive search in systematic review studies, the researcher needs to choose relevant terms and appropriate databases as well as obtain the necessary knowledge and skills to conduct a successful search in those databases. several leading organizations have provided guidelines for conducting a successful literature search and also for reporting the results effectively (moher, liberati, tetzlaff, & altman, 2009; stroup et al., 2000). prisma, cochrane handbook, press, and amstar are examples of the most popular guidelines helping researchers to conduct and report systematic reviews and meta-analyses in a more systematic and standard way. the prisma (preferred reporting items for systematic reviews and meta-analyses) checklist provides 27 items and a four-phase flow diagram in this regard (prisma transparent reporting of systematic reviews and meta-analysis, 2015). the cochrane handbook for systematic reviews of interventions also provides methodological guidance for the preparation and maintenance of cochrane reviews (higgins et al., 2019). press (peer review of electronic search strategies) mostly focuses on improving the quality of the literature search strategy as a key step for systematic review studies (mcgowan et al., 2016). amstar (the assessment of multiple systematic reviews) also provides a checklist containing 11 items guiding authors in conducting high quality systematic reviews (pieper, buechter, jerinic, & eikermann, 2012).   among these guidelines, the institute of medicine has introduced the iom guideline, which provides some specialized guidelines for designing and implementing a quality search strategy (institute of medicine, 2011). interestingly, an experienced librarian is recommended in all of these guidelines to design and implement an appropriate search strategy (centre for reviews and dissemination, 2009; higgins et al., 2019). librarians not only save time and reduce bias by conducting a comprehensive and accurate search, but they also facilitate the collaboration between the research team members, solve potential technological problems, and help with designing a method for doing systematic review studies (dayani, 2001). therefore, employing librarians in the design and reporting of the search strategy in systematic review studies is of special importance. following the growing interest in conducting systematic review studies, iranian researchers are increasingly more inclined to research this area. in most universities of medical science in iran, experienced and trained librarians are willing to work with researchers who intend to conduct systematic review studies. hence, the present study set out to examine the relationship between librarians’ participation and the quality of reporting search strategies in systematic reviews published by iranian researchers.      literature review   given the importance attached to the search strategy in systematic review studies, the number of studies that examine and evaluate the search strategy and its reporting from different aspects is on the rise. various criteria and standards are used for evaluating the quality of reporting search strategy in systematic review studies. examples include checklists provided by cochrane reviews (franco, garrote, escobar liquitay, & vietto, 2018; koffel, 2015; opheim, andersen, jakobsen, aasen, & kvaal, 2019; page et al., 2016; yoshii, plaut, mcgraw, anderson, & wellik, 2009), prisma (opheim et al., 2019), peer review of electronic search strategies (franco et al., 2018; rethlefsen, farrell, osterhaus trzasko, & brigham, 2015), and the iom standard (koffel, 2015; meert, torabi, & costella, 2016; rethlefsen et al., 2015). in several studies, certain instruments were used for evaluating the quality of reporting search strategy in systematic review studies that had been developed based on prior research and the authors’ personal experience and knowledge (koffel & rethlefsen, 2016; salvador-oliván, marco-cuenca, & arquero-avilés, 2019).     regarding examining the quality of reporting search strategy, in most studies that evaluated the reporting of the search strategy in systematic review studies, some errors were observed and the design and reporting of the search strategy was weak (faggion, huivin, aranda, pandis, & alarcon, 2018; franco et al., 2018; koffel & rethlefsen, 2016; opheim et al., 2019; salvador-oliván et al., 2019; sampson & mcgowan, 2006). according to the criteria used for investigation, the errors made in the reporting of the search strategy included: errors related to missing terms (faggion et al., 2018; salvador-oliván et al., 2019; sampson & mcgowan, 2006), not reporting the time span and the date at which the search was performed (koffel & rethlefsen, 2016; opheim et al., 2019; yoshii et al., 2009), not reporting the strategy syntax in at least one database (koffel & rethlefsen, 2016; opheim et al., 2019), not using specific search facilities within databases (faggion et al., 2018; salvador-oliván et al., 2019), not searching in gray literature, not doing manual searching in journals and conferences (faggion et al., 2018; franco et al., 2018), and not using the prisma flowchart as a graphical representation of the study selection and searching processes during different phases of a systematic review (opheim et al., 2019). yoshii et al. (2009) examined the search strategy reports of 65 systematic review studies using seven cochrane criteria (“databases searched,” “name of host database,” “date search was run,” “years covered by search,” “complete search strategy,” “one or two sentence summary of the search strategy,” and “language restrictions”). according to their study, more than 68% of systematic review studies had used four or fewer criteria (yoshii et al., 2009).   few studies have investigated the role of librarians in the design and reporting of the search strategy in systematic review studies. in one scoping review, however, the role of librarians was examined, where roles such as searching, choosing the resources, and training the researchers had received more attention (spencer & eldredge, 2018). in their review study, townsend et al. (2017) identified six competencies for librarians involved in systematic review studies: “systematic review foundations,” “process management and communication,” “research methodology,” “comprehensive searching,” “data management,” and “reporting” (townsend et al., 2017). some studies also examined the role of librarians in the quality of reporting search strategy in systematic review studies, indicating that the librarians did not play a very important role in the design and reporting of search strategy, although their participation could have a positive impact on improving the quality of reporting the search strategy in systematic review studies (koffel, 2015; meert et al., 2016; rethlefsen et al., 2015). moreover, rethlefsen et al. (2015) found a high correlation between the level of librarians’ participation and search reproducibility of strategies reported in systematic review studies.   figure 1 search strategy for pubmed.   aims   given that few studies have examined librarians’ participation in systematic review studies even though it could improve the quality of search strategy reports, the current study aimed:   1. to evaluate the quality of reporting search strategy in systematic reviews published by iranian researchers. 2. to identify the librarians’ participation in reporting search strategy in systematic reviews published by iranian researchers. 3. to investigate the relationship between librarians’ participation and the quality of reporting search strategy in systematic reviews published by iranian researchers.   methods   the present study was conducted in two stages using surveys and evaluations. these two stages are briefly explained in this section.   stage one: evaluating the quality of reporting search strategy in systematic review studies done by iranian researchers   to retrieve systematic review studies done by iranian researchers from 2008 to 2018, three databases, web of science, scopus, and pubmed, were searched using relevant keywords. the search strategy for the pubmed database is shown in figure 1. all searches were done in may 2018. the inclusion criteria for these studies were: systematic review studies done by iranian researchers, the date of publication between 2008 and 2018, and affiliation of the corresponding author with one of the medical universities in iran. the studies done before 2008 and those considered to be irrelevant or repetitive were deleted.   after searching the three databases, a total of 4,963 studies published by iranian researchers was retrieved. as a result of a preliminary review, 1,930 studies were found to be duplicated, 1,320 studies were not systematic reviews, 52 were recorded as systematic review protocol, and those with no full-text availability were removed. eventually, 1,652 studies were finalized for further analysis. to calculate the size of the sample, the cochrane formula was used. in this formula, p and q (the probability of success and failure) equaled 0.5. the value of zα/2 in the error level of 0.05 was 1.96 and the error of d equaled 0.05. the value of n was equal to the population size, 1,652. according to this formula, the sample size was estimated to be 310. these studies were selected based on systematic random sampling. first, all 1,652 studies were fed to excel. unique but consecutive numbers were allocated to each study. of all the numbers, 310 numbers that belonged to 310 systematic review studies were systematically selected at regular intervals of 5. the 310 systematic review studies were chosen as the sample for examining the librarian’s participation in these studies and its effect on the quality of reporting search strategy in systematic review studies. the questionnaires were sent to the corresponding authors of the 310 sample articles to identify the librarians’ participation in conducting, designing, and reporting the search strategy in the systematic review studies. the flowchart in figure 2 provides the details.   figure 2 flow chart of study selection.   to evaluate the reporting of search strategy in systematic review studies, a standard checklist has been designed by the institute of medicine (iom) as a guideline for conducting high-quality systematic review studies (institute of medicine, 2011). the iom checklist includes 15 standards that provide exact and accurate guidelines for the implementation and reporting of a strong search strategy. these 15 iom standards, along with their descriptions, are presented in table 1.   to collect descriptive data, including the study title, publication year, journal name, and the organizational affiliation of the author, the researchers reviewed the full text of the studies. in cases where the full text of the article was not available, an email was sent to the corresponding authors explaining the purpose of the study and asking them to provide the full text of the study if possible. the data needed for examining the quality of reporting search strategy in systematic review studies were transferred to excel 2013. to avoid any bias and enhance the accuracy of all stages in selecting the studies and evaluating their qualities, two researchers (ash, rz) performed the analysis of the studies independently. the score for the quality of reporting the search strategy in each study was estimated by summing up the scores in the iom checklist (with a maximum score of 15). in case of any disagreement in scoring, a third researcher (sp) was consulted.   table 1 15 iom standards and their descriptions item description 3-1-1 “work with a librarian or other information specialist trained in performing systematic reviews to plan the search strategy” 3-1-2  “design the search strategy to address each key research question” 3-1-4 “search bibliographic databases” 3-1-5  “search citation indexes” 3.1.6 “search literature cited by eligible studies” 3-1-7  “update the search at intervals appropriate to the pace of generation of new information for the research question being addressed” 3-1-8 “search subject-specific databases if other databases are unlikely to provide all relevant evidence” 3-1-9 “search regional bibliographic databases if other databases are unlikely to provide all relevant evidence” 3-2-1  “search grey literature databases, clinical trial registries, and other sources of unpublished information about studies” 3-2-2  “invite researchers to clarify information about study eligibility, study characteristics, and risk of bias” 3-2-3  “invite all study sponsors and researchers to submit unpublished data, including unreported outcomes, for possible inclusion in the systematic review” 3-2-4 “hand search selected journals and conference abstracts” 3-2-5 “conduct a web search” 3-2-6  “search for studies reported in languages other than english if appropriate” 3-4-1 “key words, subject headings, terms” source: institute of medicine, 2011   stage two: examining librarians’ participation in reporting search strategy in systematic review studies done by iranian researchers   a short questionnaire was used for examining the level of librarians’ participation in designing and reporting search strategy in systematic review studies. meert et al. (2016) used this questionnaire for investigating the role of librarians in reporting search strategy in systematic review studies conducted in pediatrics. the questionnaire’s face validity was approved by several faculty members of the medical library and information sciences department. the questionnaire included questions about the type and extent of librarians’ participation in the design, implementation, and reporting of the search strategy in systematic review studies.   to examine the librarians’ role, the corresponding authors were queried, through the questionnaire’s items, about whether the study was informed by a librarian’s consultation and participation. in the case of the librarian’s participation, the author was asked to determine the type and quality of the role or participation. the role of librarians was divided into three groups: a non-participant, a counselor, or a member of the research team and an author. “non-participant” indicates that the librarian had no participation in designing and reporting search strategy in the systematic review. a “counselor” means that the research team received consultative services from the librarian in designing and reporting search strategy, and, therefore, the librarian was not among the authors of the research study. “a member of the research team” refers to a librarian who was one of the main members and authors of the research team in the systematic review study.   the questionnaires were designed online in google docs and sent to the academic emails of the 310 corresponding authors of the retrieved studies in the first stage in november 2018. in some cases, the authors’ academic emails were not valid. to solve this problem, the authors of this study searched the names of the corresponding authors on researchgate, or, in the case of having their phone number, they were contacted about sending the questionnaire. if no response was received after two weeks, a reminder was sent to the author.   from 310 submitted questionnaires for identifying the librarians’ participation in systematic review studies, 229 questionnaires were returned (response rate = 73.8%) by the corresponding authors of the included studies. the 81 studies whose corresponding authors did not respond were excluded from further analysis. the above-mentioned 229 studies were evaluated by the iom checklist.   statistical analysis   in this study, descriptive statistics such as frequency, percentage, mean, median, variance, and standard deviation were used. also, a kolmogorov-smirnov test was used to evaluate the data normalization. the non-parametric mann-whitney u test was used to analyze non-normal data. the chi-square test was also used for examining the relationship between the two qualitative variables. the data were analyzed using spss 22 software.   results   the quality of reporting search strategy in systematic reviews published by iranian researchers   the analysis of the data obtained by evaluating the quality of reporting the search strategy showed that the mean score of the search strategy report for all of the 229 systematic review articles, based on the iom checklist, was 4.23 (sd = 1.69) out of 15. in only 32% of these studies had the procedures been fully presented as specified by the standard, “design the search strategy to address each key research question” (standard 3.1.2). the highest score was for the item of “search bibliographic databases” (standard 3.1.4), 97.8%. the lowest scores were also related to the items of “invite researchers to clarify information about study eligibility, study characteristics, and risk of bias” (standard 3.2.2), 1.7%; “invite all study sponsors and researchers to submit unpublished data, including unreported outcomes, for possible inclusion in the systematic review” (standard 3.2.2), 4.4%; and “work with a librarian or other information specialist trained in performing systematic reviews to plan the search strategy” (standard 3.1.1), 5.3%. the item “search grey literature databases, clinical trial registries, and other sources of unpublished information about studies” (standard 3.2.1) was reported in only 15.3% of these studies. the results of evaluating the quality of the search strategy for systematic review studies are presented in figure 3.   figure 3 the frequency of presenting each of the items in the iom checklist for reporting search strategy in systematic review studies (n = 229).   librarians’ participation in the quality of reporting search strategy in systematic review studies published by iranian researchers   findings showed that a librarian was employed in 13.6% of the systematic review studies, either as a co-author (7.0%) or just as a search counselor (6.6%), contributing in designing and reporting the reviews’ search strategies. the role and the level of librarians’ participation were analyzed through nine questions administered through a questionnaire (table 2). the results showed that the highest participation was for “consulting for selecting resources, databases and suggested strategies” with 9.1% and the lowest participation was for “writing some parts of the study” and “article editing” with 1.7%. the details are presented in table 2.   table 2 the type of librarians’ participation based on their role in the process of conducting systematic review studies activities   team member/ co-author (n = 16) search counselor (n = 15) total (n = 229) consulting for selecting resources, databases, and suggested strategies 13 (81.2%) 8 (53.3%) 21 (9.1%) reviewing the search strategies written by the main researchers 9 (56.2%) 5 (33.3%) 14 (6.1%) designing a complete search strategy 12 (75.0%) 1 (6.6%) 13 (5.6%) modifying and reviewing the references 8 (50.0%) 2 (13.3%) 10 (4.3%) searching and collecting the required information and all resources about research 12 (75.0%) 2 (13.3%)   14 (6.1%) implementing manual search 9 (56.2%) 3 (20.0%) 12 (5.2%) searching for gray literature 6 (37.5%) 4 (26.6%) 10 (4.3%) writing some parts of the study 4 (25.0%) 0 (0.0%) 4 (1.7%) article editing 4 (25.0%) 0 (0.0%) 4 (1.7%)   examining the relationship between librarians’ participation and the quality of reporting search strategy in systematic review studies   the mann-whitney u test was used to test for significant differences between librarians’ participation and the mean score obtained from evaluating the quality of reporting search strategy in systematic review studies done by iranian researchers. the results indicated that there is no significant difference between librarians’ participation and the mean score obtained from evaluating the quality of reporting search strategy in systematic review studies. however, the mean score and the median of the quality of reporting search strategy for the group that employed a librarian were higher than those in the group without a librarian. the results are shown in table 3.   table 3 the significant difference between librarians’ participation and the mean score of the quality of reporting search strategy in systematic review studies mann-whitney u test median median rank mean score )sd) n use of librarian z: -0.824  p value: 0.4 4 113.6 4.17 (±1.69) 198 without librarian 4 123.9 4.54 (±1.68) 31 with librarian   the chi-square test was used to examine the hypothesis that there is a relationship between librarians’ participation and the quality of reporting search strategy in systematic review studies based on each of the items in the iom checklist. the results showed that there was a meaningful relationship between librarians’ participation and the rate of presenting the items in the iom checklist in reporting search strategy in systematic review studies in five items (p < 0.05). in the three items of “work with a librarian or other information specialist trained in performing systematic reviews to plan the search strategy” (standard 3.1.1), “design the search strategy to address each key research question” (standard 3.1.2), and “search subject-specific databases if other databases are unlikely to provide all relevant evidence” (standard 3.1.8), the rate of reporting these items in the search strategy for studies with librarians was higher than that of studies without a librarian. for the two items of “search for studies reported in languages other than english” (standard 3.2.6) and “search citation indexes” (standard 3.1.5), the rate of reporting these items in the search strategy for studies without a librarian was higher than that of those with a librarian. additionally, the results showed that, on average, the rate of reporting the items in the iom checklist was higher in studies with a librarian. the results are shown in table 4.   table 4 frequencies and chi-square results of librarians’ participation in studies, sorted by the iom standard iom standard without librarian (n = 198) with librarian (n = 31) total (n = 229) p value chi-square test value   3.1.1 0 (0.0%) 8 (25.8%) 8 (3.5%) 0.00* 52.94 3.1.2 60 (30.3%) 15 (48.4%) 75 (32.8%) 0.04* 3.98 3.1.4 193 (97.5%) 31 (100%) 224 (97.8%) 0.37 0.80 3.1.5 111 (56.1%) 11 (35.5%) 122 (53.3%) 0.03* 4.55 3.1.6 8 (4.0%) 3 (9.7%) 11 (4.8%) 0.17 1.86 3.1.7 10 (5.1%) 3 (9.7%) 13 (5.7%) 0.30 1.07 3.1.8 20 (10.1%) 7 (22.6%) 27 (11.8%) 0.04* 4.01 3.1.9 41 (20.7%) 8 (25.8%) 49 (21.4%) 0.52 0.41 3.2.1 32 (16.2%) 3 (9.7%) 35 (15.3%) 0.35 0.87 3.2.2 4 (2.0%) 0 (0.0%) 4 (1.7%) 0.42 0.63 3.2.3 10 (5.1%) 0 (0.0%) 10 (4.4%) 0.20 1.63 3.2.4 11 (5.6%) 4 (12.9%) 15 (6.6%) 0.12 2.36 3.2.5 71 (35.9%) 11 (35.5%) 82 (35.8%) 0.96 0.00 3.2.6 88 (44.4%) 8 (25.8%) 96 (41.9%) 0.05* 3.82 3.4.1 168 (84.8%) 29 (93.5%) 197 (86%) 0.19 1.68                                       *significant at p < .05   discussion   the first aim of the present study was to examine the quality of reporting search strategy in systematic review studies done by iranian researchers. based on the results obtained from the iom checklist, the mean score of the quality of reporting search strategy in systematic review studies was not high. only less than one-third of the systematic reviews investigated in this study disclosed the full search strategy used in at least one database. this is consistent with the results of page et al. (2016) and opheim et al. (2019), where the full search strategy in at least one database was presented by one-third and less than one-third of the systematic review studies examined. a detailed and accurate reporting of the search strategy in systematic reviews allows for reproduction, particularly in those studies in which strong evidence is not gained to draw conclusions and updating the systematic review might be needed (moher & tsertsvadze, 2006). presenting information on the latest date of searching in reporting the search strategy in systematic reviews is necessary for reproducing the search strategy and updating the review (liberati et al., 2009). despite the importance of this issue, only a few studies had provided some information on the date of searching and its updating for searching relevant studies that might have been recently conducted. searching the gray literature is regarded as an important factor in obtaining information that is often less accessible. in a few of the systematic reviews, searching the gray literature had been reported. this is consistent with the results of page et al. (2016), where features of the reports in systematic reviews in biomedical research were examined and few studies were found to have reported the searching of gray literature. given the fact that much of reporting the search strategy in systematic review studies is done based on one of the most reliable guidelines, such as prisma (asar, jalalpour, ayoubi, rahmani, & rezaeian, 2016) and cochrane (franco et al., 2018), not reporting these issues in systematic reviews examined by this study can probably be due to: a scarcity of guidelines and resources related to standard reporting of strategies (moher, tetzlaff, tricco, sampson, & altman, 2007); lack of necessary training for the researchers with regard to methods of systematic searching or standard reporting (koffel & rethlefsen, 2016); or a lack of the required software to help researchers in reporting their systematic reviews (page et al., 2016). moreover, most editors or reviewers of the journals may not be well aware of the importance of reporting the search strategy in systematic review studies, often leading to lower quality and critical mistakes in search strategy (sampson & mcgowan, 2006). seeking an expert librarian’s opinion in the review process of systematic review studies might be helpful. since most reputable journals tend to publish quality articles, using the standards such as iom or press for peer review before publishing an article can reduce the errors in this field, create a comprehensive search retrieval strategy, and increase the trust in the results of these studies and journals.     the second purpose of this study was to examine the type and the level of librarians’ participation in reporting the search strategy in systematic review studies done by iranian researchers. librarians’ participation in systematic reviews was very limited. in 13.6% of all the systematic review studies investigated in this study, the librarian was a member of the authors’ team. one probable reason for the low participation of librarians might be that there is a lack of cooperation between researchers in different fields and librarians, as well as the researchers’ failure to be aware of librarians’ knowledge and skill in systematic review studies. the results of meert et al.’s study (2016) showed that librarians’ participation in reporting search strategy in systematic review studies was low, around 44%. this is consistent with the results of the present study. much of the librarians’ participation was in “consulting for selecting resources, databases and suggested strategies” and “searching and collecting the required information and all resources about research.” the lowest participation of librarians was in searching gray literature, authoring parts of the study, and editing the study. employing a librarian as a team member in systematic review studies can have some advantages. among these advantages are: saving time by performing an exact search, reducing the number of studies in the primary screening, avoiding repetitive terms in the search strategy, and finally, increasing the number of studies under investigation (sampson et al., 2009). in some guidelines, the presence of a librarian is recommended in planning, performing, and investigating the search strategy in systematic review studies (institute of medicine, 2011; mcgowan et al., 2016; sampson et al., 2009). the aforementioned guidelines and journals’ editors and referees can be helpful in attracting the attention of researchers doing systematic review studies toward employing librarians in designing, performing, and reporting search strategy in systematic reviews.   the third purpose of the present study was to examine the relationship between librarians’ participation and the quality of reporting search strategy in systematic review studies done by iranian researchers. we found out that there was no significant difference between the mean score of the quality of reporting the search strategy in systematic reviews and the librarians’ participation, although the mean score and the median rank were higher for those groups that had used a librarian as a member of the authors’ team. results showed that, on average, the librarians’ participation in systematic review studies affected increasing the level of presenting the items of the iom checklist in reporting search strategy. meert et al. (2016) reached the same conclusion that there was a meaningful relationship between the librarians’ participation and the quality of reporting search strategy in systematic review studies. the relationship between the librarians’ participation and the items on the iom checklist was meaningful in five items. employing a librarian in systematic review studies could result in an increase in reporting search strategy in items related to designing the search strategy, searching subject-specific databases, and reporting the use of a librarian. the results in this study showed that the level of use and observance in the two items of “search for studies reported in languages” (standard 3.2.6) and “search citation index” (standard 3.1.5) in the iom checklist was higher in the group without a librarian. one of the reasons can be that researchers were more familiar with these two items due to the importance attached to these two items by prior research on systematic review studies.   limitations and future directions   the main limitation faced by this study was that the results were limited to the systematic review studies done by iranian researchers, and the level of librarians’ participation was limited, which limits the possibility of generalizing the results to other systematic review studies.   most of the previous studies set out to investigate the quality of the search strategy in systematic review studies and also the role of librarians in certain cases; therefore, some factors need to be recommended: examining the quality of designing, performing, and reporting other parts of the systematic review studies, such as selection and screening of the studies; evaluating the quality of the studies under investigation; reporting the risk of bias according to some standards like iom and prisma; and examining the role of librarians. the quality of designing, performing, and reporting search strategy in systematic review studies in top-ranked journals should be compared to less prestigious journals in different medical fields, and the librarians’ participation in this area is recommended. we also suggest that the quality of reporting the search strategy in systematic review studies done in developed countries be compared with those of developing countries, and the level of librarians’ participation should be used to analyze the results.   conclusion   the purpose of the present study was to investigate the relationship between librarians’ participation and the quality of reporting search strategy in systematic review studies conducted in iran. the results showed that the librarians’ participation in designing and reporting search strategy in systematic reviews was low. moreover, the quality of reporting the search strategy in systematic reviews based on the iom checklist was not satisfactory. in five out of 15 items in the checklist, there was a positive correlation between the librarians’ participation and the quality of reporting the search strategy in systematic reviews. in general, the level of observing the iom checklist items in reporting the search strategy in systematic reviews was higher in groups that had used a librarian.   the methods used for reporting the search strategy in systematic reviews based on the iom checklist can affect the judgments on the quality and capability of the results obtained by these studies. selecting and employing experts, especially librarians, in the research team can have a positive impact on designing, performing, and reporting the search strategy. on the other hand, training researchers, proposing guidelines for reporting the search strategy in a standardized and comprehensive manner by the stakeholders and the editors of the journals, and employing librarians in evaluating and refereeing systematic review studies can help to enhance researchers’ ability to prepare an exact, comprehensive, and clear report of the search strategy. consequently, the validity of the obtained results can be verified more rigorously than before.    acknowledgements   the authors would like to thank all corresponding authors of systematic review studies who provided the data needed for this research. the authors would also like to thank the college of business administration, california state university san marcos for their support of this project. this study was funded and supported by iran university of medical sciences; grant no. 97-01-136-32607.   references   asar, s., jalalpour, s., ayoubi, f., rahmani, m., & rezaeian, m. (2016). prisma: preferred reporting items for systematic reviews and meta-analyses. journal of rafsanjan university of medical sciences, 15(1), 68–80. retrieved from http://eprints.rums.ac.ir/id/eprint/5573   centre for reviews and dissemination. (2009). systematic reviews: crd’s guidance for undertaking reviews in healthcare. york: university of york.   dayani, f. (2006). professional medical library medicine abroad: letter to the editor. journal of health information management, 3(1). http://him.mui.ac.ir/index.php/him/article/view/39/1466   faggion, c. m., jr., huivin, r., aranda, l., pandis, n., & alarcon, m. (2018). the search and selection for primary studies in systematic reviews published in dental journals indexed in medline was not fully reproducible. journal of clinical epidemiology, 98, 53–61. https://doi.org/10.1016/j.jclinepi.2018.02.011   franco, j. v. a., garrote, v. l., escobar liquitay, c. m., & vietto, v. (2018). identification of problems in search strategies in cochrane reviews. research synthesis methods, 9(3), 408-416. https://doi.org/10.1002/jrsm.1302   higgins, j., thomas, j., chandler, j., cumpston, m., li, t., page, m., & welch, v. (2019). cochrane handbook for systematic reviews of interventions version 6.0 (updated july 2019). cochrane. retrieved from https://training.cochrane.org/handbook   institute of medicine (2011). finding what works in health care: standards for systematic reviews. washington, dc: national academies press. https://doi.org/10.17226/13059   koffel, j. b. (2015). use of recommended search strategies in systematic reviews and the impact of librarian involvement: a cross-sectional survey of recent authors. plos one, 10(5). https://doi.org/10.1371/journal.pone.0125931   koffel, j. b., & rethlefsen, m. l. (2016). reproducibility of search strategies is poor in systematic reviews published in high-impact pediatrics, cardiology and surgery journals: a cross-sectional study. plos one, 11(9).                https://doi.org/10.1371/journal.pone.0163309   liberati, a., altman, d. g., tetzlaff, j., mulrow, c., gøtzsche, p. c., ioannidis, j. p. a., clarke, m., devereaux, p. j., kleijnen, j., & moher, d. (2009). the prisma statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. plos medicine, 6(7). https://doi.org/10.1371/journal.pmed.1000100   liberati, a., & taricco, m. (2010). how to do and report systematic reviews and meta-analysis. in f. franchignoni (ed.), research issues in physical & rehabilitation medicine (pp. 137-164). pavia, italy: maugeri foundation books.   mcgowan, j., sampson, m., salzwedel, d. m., cogo, e., foerster, v., & lefebvre, c. (2016). press peer review of electronic search strategies: 2015 guideline statement. journal of clinical epidemiology, 75, 40–46. https://doi.org/10.1016/j.jclinepi.2016.01.021   meert, d., torabi, n., & costella, j. (2016). impact of librarians on reporting of the literature searching component of pediatric systematic reviews. journal of the medical library association. 104(4), 267–277. https://doi.org/10.3163/1536-5050.104.4.004   moher, d., liberati, a., tetzlaff, j., & altman, d. g. (2009). preferred reporting items for systematic reviews and meta-analyses: the prisma statement. annals of internal medicine, 151(4), 264–269. https://doi.org/10.7326/0003-4819-151-4-200908180-00135   moher, d., tetzlaff, j., tricco, a. c., sampson, m., & altman, d. g. (2007). epidemiology and reporting characteristics of systematic reviews. plos medicine, 4(3). https://doi.org/10.1371/journal.pmed.0040078   moher, d., & tsertsvadze, a. (2006). systematic reviews: when is an update an update? the lancet, 367(9514), 881–883. https://doi.org/10.1016/s0140-6736(06)68358-x   opheim, e., andersen, p. n., jakobsen, m., aasen, b., & kvaal, k. (2019). poor quality in systematic reviews on ptsd and emdr: an examination of search methodology and reporting. frontiers in psychology, 10. https://doi.org/10.3389/fpsyg.2019.01558   page, m. j., shamseer, l., altman, d. g., tetzlaff, j., sampson, m., tricco, a. c., moher, d. (2016). epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. plos medicine, 13(5). https://doi.org/10.1371/journal.pmed.1002028   patrick, t. b., demiris, g., folk, l. c., moxley, d. e., mitchell, j. a., & tao, d. (2004). evidence-based retrieval in evidence-based medicine. journal of the medical library association, 92(2), 196–199. retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/pmc385300/   pieper, d., buechter, r., jerinic, p., & eikermann, m. (2012). overviews of reviews often have limited rigor: a systematic review. journal of clinical epidemiology, 65(12), 1267–1273. https://doi.org/10.1016/j.jclinepi.2012.06.015   prisma transparent reporting of systematic reviews and meta-analysis. (2015). retrieved from http://www.prisma-statement.org/   rethlefsen, m. l., farrell, a. m., osterhaus trzasko, l. c., & brigham, t. j. (2015). librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. journal of clinical epidemiology, 68(6), 617–626. https://doi.org/10.1016/j.jclinepi.2014.11.025   salvador-oliván, j. a., marco-cuenca, g., & arquero-avilés, r. (2019). errors in search strategies used in systematic reviews and their effects on information retrieval. journal of the medical library association, 107(2), 210–221. https://doi.org/10.5195/jmla.2019.567   sampson, m., & mcgowan, j. (2006). errors in search strategies were identified by type and frequency. journal of clinical epidemiology, 59(10), 1057-1063. https://doi.org/10.1016/j.jclinepi.2006.01.007   sampson, m., mcgowan, j., cogo, e., grimshaw, j., moher, d., & lefebvre, c. (2009). an evidence-based practice guideline for the peer review of electronic search strategies. journal of clinical epidemiology, 62(9), 944–952. https://doi.org/10.1016/j.jclinepi.2008.10.012   spencer, a. j., & eldredge, j. d. (2018). roles for librarians in systematic reviews: a scoping review. journal of the medical library association, 106(1), 46–56. https://doi.org/10.5195/jmla.2018.82   stroup, d. f., berlin, j. a., morton, s. c., olkin, i., williamson, g. d., rennie, d., moher, d., becker, b. j., sipe, t. a., & thacker, s. b. (2000). meta-analysis of observational studies in epidemiology: a proposal for reporting. jama, 283(15), 2008–2012. https://doi.org/10.1001/jama.283.15.2008   townsend, w. a., anderson, p. f., ginier, e. c., maceachern, m. p., saylor, k. m., shipman, b. l., & smith, j. e. (2017). a competency framework for librarians involved in systematic reviews. journal of the medical library association, 105(3), 268–275. https://doi.org/10.5195/jmla.2017.189   yoshii, a., plaut, d. a., mcgraw, k. a., anderson, m. j., & wellik, k. e. (2009). analysis of the reporting of search strategies in cochrane systematic reviews. journal of the medical library association, 97(1), 21–29. https://doi.org/10.3163/1536-5050.97.1.004 [1]rogheyeh eskrootchi can also be contacted at psebt@csusm.edu.   review article   a rapid review of the reporting and characteristics of instruments measuring satisfaction with reference service in academic libraries   heidi senior reference/instruction librarian clark library university of portland portland, oregon, united states of america email: senior@up.edu    tori ward  mlis graduate  syracuse university  syracuse, new york, united states of america  email: tjward@syr.edu      received: 13 feb. 2019                                                                   accepted: 6 nov. 2019      2019 senior and ward. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29556     abstract   objective – the objective of this review was to examine research instrument characteristics, and to examine the validity and reliability of research instruments developed by practicing librarians, which measure the construct of patron satisfaction with academic library reference services. the authors were also interested in the extent to which instruments could be reused.   methods – authors searched three major library and information science databases: library and information science technology abstracts (lista); library science database (ld); and library literature & information science index. other databases searched were current nursing and allied health literature (cinahl); education resources information center (eric); google scholar; pubmed; and web of science. the authors identified studies of patron satisfaction with academic library reference services in which the researcher(s) developed an instrument to study the satisfaction construct. in this rapid-review study, the studies were from 2015 and 2016 only. all retrieved studies were examined for evidence of validity and reliability as primary indicators of instrument quality, and data was extracted for country of study, research design, mode of reference service, data collection method, types of questions, number of items related to satisfaction, and content of items representing the satisfaction construct. instrument reusability was also determined.   results – at the end of the screening stage of the review, a total of 29 instruments were examined. nearly all studies were quantitative or mixed quantitative/qualitative in design. twenty-six (90%) of the studies employed surveys alone to gather data. twelve publications (41%) included a discussion of any type of validity; five (17%) included discussion of any type of reliability. three articles (10%) demonstrated more than one type of validity evidence. nine articles (31%) included the instrument in full in an appendix, and eight instruments (28%) were not appended but were described adequately so as to be reusable.    conclusions – this review identified a range of quality in librarians’ research instruments for evaluating satisfaction with reference services. we encourage librarians to perform similar reviews to locate the highest-quality instrument on which to model their own, thereby increasing the rigor of library and information science (lis) research in general. this study shows that even a two-year rapid review is sufficient to locate a large quantity of research instruments to assist librarians in developing instruments.     introduction   reference services are a primary function of nearly every library. library staff make themselves available to patrons through multiple communication modes such as in-person, chat, phone, and email in order to “recommend, interpret, evaluate, and/or use information resources to help others to meet particular information needs” (reference and user services association, 2008). they might gather statistics relating to the number and type of questions patrons ask, and perhaps the difficulty of answering those questions according to the read scale (gerlich & berard, 2007), but these statistics do not express whether patrons were satisfied with the answer. to determine if their library’s patrons are satisfied with the provided service, librarians need to obtain patrons’ opinions directly through data gathering methods such as surveys or   interviews, known collectively as research tools or instruments. they might then publish the results of their study to help fellow librarians develop their own patron-satisfaction tools. one study found that reference topics represented 9.5% of all library and information sciences research (koufogiannakis, slater, & crumley, 2004).   systematic instrument review has been a common practice in health science research and has developed to the extent that standards exist for specific topic areas, such as the consensus-based standards for the selection of health measurement instruments (cosmin) initiative (2018). this type of study uses systematic review methodology to identify and analyze the psychometric characteristics of research instruments. while anthologies of research instruments produced by librarians and measuring satisfaction with reference services exist, such as those found in the reference assessment manual (american library association, evaluation of reference and adult services committee, 1995, pp. 255-345), to date it seems that no one has published a systematic instrument review that would obtain an overall image of the state of instrument development in this area. we therefore decided to conduct a review to gain an understanding of the quality of instruments produced by academic librarians studying patron satisfaction with reference service.   literature review   while at the time of our study no reviews of instruments fully using the systematic review methodology had appeared in lis literature, we found that researchers had mentioned instruments and evaluated them to varying extents in articles on faculty attitudes toward open access publication (otto, 2016); information literacy (beile, 2008; schilling & applegate, 2012); information seeking behavior (mckechnie, chabot, dalmer, julien, & mabbott, 2016); satisfaction with chat reference (lasda bergman & holden, 2010); and assessment of individual research consultations (fournier & sikora, 2015).   of these researchers, only lasda bergman & holden (2010) followed a systematic review protocol in their research criteria and search methods, retaining after their final appraisal stage 12 studies regarding user satisfaction with electronic reference. however, because they did not present details of each instrument in an evidence table, we were unable to reproduce their data extraction process. schilling and applegate (2012) identified 27 tools in their survey of academic library literature on student learning assessment from 2007 to 2012 but did not take a systematic approach and emphasized each instrument’s content rather than construction and measurement concerns. similarly, fournier and sikora’s 2015 scoping review located 20 studies using various methods to assess individual research consultations but did not review instrument characteristics beyond the type of assessment method. beile’s (2008) report covered widely-known information literacy assessment tools that would provide data “considered acceptable evidence for program reviews” (p. 1) such as standardized assessment of information literacy skills (sails) and educational testing service’s iskills, but did not describe a process for identifying the seven tests and four rubrics included in the paper. mckechnie et al.’s approach to evaluating research rigor involved using a checklist that asked whether authors attached or included their instrument – an element we included in our study – and whether the instrument had undergone pre-testing, an important component in demonstrating an instrument’s validity (2016). while conducting a literature review prior to studying effective faculty outreach messages regarding open access publication, otto (2016) realized that the studies reviewed did not accurately reflect faculty understanding due to flaws in their underlying instruments such as failing to define terms, adapting previous surveys without updating questions, and inserting inadvertent bias into survey questions and response options. although otto did not report evidence of the instruments’ validity and reliability specifically, several of the issues otto identified might have been resolved had the instruments’ developers paid closer attention to determining their validity.   shortly before completing our manuscript, we learned of the publication late in 2017 of a systematic review of 22 self-efficacy scales assessing students’ information literacy skills (mahmood) from 45 studies published between 1994 and 2015. because mahmood’s review was limited to studies in which authors reported the use of any validity and also any reliability indicators, it differs from our relatively unrestricted approach. mahmood’s study likely omits scales and does not provide a full picture of the state of instrument development in this area. we identified two instrument reviews from the field of education (gotch & french, 2014; siddiq, hatlevik, olsen, throndsen, & scherer, 2016), the second of which served as a preliminary model for the data extraction stage of our pilot study. gotch & french (2014) reviewed 36 measures published between 1991 and 2012 of classroom teachers’ assessment literacy, using “evaluation of the content of assessment literacy measures beyond literature review and solicitation of feedback,” “internal consistency reliability,” and “internal structure” to demonstrate validity, and “score stability” (p. 15) to demonstrate reliability of each instrument. we decided not to use gotch and french as a model because the authors did not rigorously follow a systematic approach in database searching or in presenting their results in evidence tables. the second systematic instrument review (siddiq, et al., 2016) covered 38 information and communication technology instruments aimed at primary and secondary school students, and was a useful framework to emulate because, like our study, it was descriptive rather than evaluative in design. furthermore, the authors carefully documented their search strategy and findings in a way that adhered closely to systematic review methodology. like our study, the authors appeared to be concerned to represent the state of the field and included instruments whose developers did not address evidence of their validity or reliability.   our review of librarians’ studies examining instruments determined that the instrument review methodology is under-used in librarianship, and that our pilot study identifies a new area of research. by drawing on similar reviews in education, we demonstrate the usefulness to librarian-researchers of breaking out of disciplinary compartmentalization for assistance with promising methodologies.   research questions   we began this study with a basic question: what is the quality of research instruments produced by librarians? we developed the following more specific questions using patron satisfaction with reference services in academic libraries as a focus. we defined reference service as librarians helping others to meet particular information needs in-person at a desk, roaming, or via consultations; through virtual methods such as chat and email; or over a telephone.   q1: how did lis researchers gather data on patron satisfaction with academic library reference services in the years 2015-2016?   q2: to what extent did the instrument developers document the validity and reliability of their instruments?   q3: to what extent are the instruments provided in an appendix or described in the publication, to assist in reuse?   method   selection of review type   the systematic review is considered the most rigorous methodology for gathering and synthesizing information based on predetermined inclusion/exclusion criteria, clear and reproducible search methods, and quality assessment, with results presented in an evidence table (xu, kang, & song, 2015; see also mckibbon, 2006; phelps & campbell, 2012). traditional systematic reviews, however, aim to be comprehensive in coverage. because the lead author wanted to accomplish as much work as possible during a sabbatical, we elected to perform a rapid review, which follows the systematic review methodology (predetermined inclusion/exclusion criteria, clear and reproducible search methods, and quality assessment, with results presented in an evidence table) but is limited in time (grant & booth, 2009, p. 100). see table 1 for distinctions between systematic and rapid reviews.      table 1 differences between systematic and rapid review types review type description   search   appraisal   synthesis   analysis   rapid review   assessment of what is already known about a policy or practice issue, by using systematic review methods to search and critically appraise existing research   completeness of searching determined by time constraints   time-limited formal quality assessment   typically, narrative and tabular   quantities of literature and overall quality/direction of effect of literature   systematic review   seeks to systematically search for, appraise and synthesis research evidence, often adhering to guidelines on the conduct of a review   aims for exhaustive, comprehensive searching   quality assessment may determine inclusion/exclusion   typically, narrative with tabular accompaniment   what is known; recommendations for practice. what remains unknown; uncertainty around findings, recommendations for future research   grant & booth, 2009, p. 95.     inclusion criteria   we assembled and agreed upon the following criteria:   quantitative, qualitative, or mixed-method research studies measuring satisfaction with reference service carried out in any type of academic library including health science libraries, addressing any type of patron. we included studies measuring satisfaction with several library services, as long as one question asked about reference service. studies published in 2015 or 2016 instruments developed by front-line librarians, including adaptations of a standardized instrument such as servqual or servperf english, french, or spanish language   search strategy   ebsco’s library and information science technology abstracts (lista) and proquest’s library science database (ld) were the primary sources of studies; we also searched library literature & information science index, cinahl, eric, google scholar, pubmed, and web of science. in addition to these databases, we searched the american library association, the association of college and research libraries, and assessment conference programs that were published online for the years of interest.   when developing our search strategies, we kept in mind the caveat raised by lis authors that database thesauri might be incomplete or that subject headings might not be applied uniformly. vanscoy and fontana noted in their 2016 study of reference and information service (ris) research that “this method relies on the ris research articles being correctly assigned the relevant descriptor in the databases” (p. 96). this warning echoes that of mckechnie, baker, greenwood, & julien in 2002 who said “both [ebsco and proquest] indexes used terms … that were too general to be useful” (p. 123) and found that indexing terms were incorrectly applied in 28-34% of the articles they examined, as well as greifeneder who warned in 2014 that one of the studies in her literature review might have a biased retrieval set because it included articles indexed under only two subject terms rather than searching more widely (background, para. 8). we therefore decided to run both subject and keyword searches.   after a careful examination of subject terms used in either lista or ld, and heeding past research on effective search strategy, we performed the following searches:   lista: (academic and librar* and (reference or "user satisfaction")) and (su (research or surveys or questionnaires) or ab (study or survey* or interview* or research*))     ld: all ((academic* and librar* and (reference or "user satisfaction" or "customer satisfaction" or "customer services"))) and su(research or surveys or questionnaires) and ab(study or survey* or interview* or research*). note that the ld search is identical to the lista search except for the inclusion of “customer satisfaction” and “customer services,” which are subject terms not used in the lista database.   given the inconsistent application of subject terms in library literature databases, we note that articles given the subject term “academic libraries” might not describe undergraduate or community college libraries. however, we found no additional articles when we re-ran searches with the subject terms “community college libraries” and “undergraduate libraries.”   we then examined abstracts and developed a free-text keyword search that we adapted for use in all of the databases, in an attempt to find all articles that might not have had correct subject-term labels: (reference or "research consultation") and (satisf* or evaluat* or assess* or improve*) and (experiment* or survey* or qualitative or servqual or instrument or investigat* or analysis or questionnaire*) and "academic librar*". we also ran a broad search for librar* and reference and satisfaction, being mindful that the “academic libraries” label might not be uniformly applied and that some articles might use “college” or “university” instead, or that various labels might be used to represent different categories of library patron, or different types of data-gathering instruments. we removed search terms related to research methodologies to have broad retrieval.   conforming to our inclusion criteria, we limited results in each database to journal articles from the years 2015 and 2016 and checked each database for conference papers as a separate source type. we did not apply language or geographic location limiters and were prepared to examine articles in french or spanish in addition to english, but our searches retrieved only english-language publications. in preparation for retrieving a large amount of results, such as within google scholar, we determined that we would review the first 300 items only. within those 300 results, we ceased reviewing when we began encountering irrelevant items.   when searching pubmed, we applied a search filter provided by cosmin in order to better identify all studies containing measurement properties.  in google scholar, we utilized the advanced search feature to narrow our results. in eric and cinahl, we utilized the database thesauri to identify subject terms. we also hand searched 10 online journals (journal of academic librarianship; college & research libraries; library & information science research; portal; journal of the medical library association; journal of librarianship and information science; reference services review; medical reference services quarterly; reference librarian; and college and undergraduate libraries), adhering to our year restriction of 2015-2016. to standardize our searches, we created a table in which both authors’ search strings were input to compare and ensure that we were staying consistent with our searches and results.   all search strategies are provided in appendix a.   reviewing process and study evaluation   we compiled citations in a refworks database and removed duplicates using the refworks tool. we examined bibliographic information from the databases, such as title and abstract, to screen for relevant articles. to add a peer reviewing element to our searches, we kept track of our subsequent searches on a separate workbook so that each author could observe and be able to discuss the quality of each search with the other. in those workbooks, we documented the search conducted, the database in which the search was conducted, the limiters set in each search, the results of each search, the citations found from each search (if applicable), and any notes.   data extraction   as stated earlier, we used siddiq et al.’s (2016) extraction sheet as a model because we aimed to be descriptive rather than evaluative in scope. following their model, we extracted the following data: country of study; stated purpose of study; mode of reference service; age/level of students (if students were part of the targeted population); size of targeted population; usable responses; sampling strategy; research design; data collection method; types of questions, other than demographic (i.e., likert scale format; presence of open-ended questions); demographics gathered; technical aspects, such as distribution, availability of translations, duration of survey period; time allotted to complete survey; validity indicators; reliability indicators (see below for definitions of validity and reliability); instrument availability in appendix; reusability of instrument, if not appended; number of items related to satisfaction; content of items representing the satisfaction construct.   definitions generally speaking, an instrument is said to provide valid results when it measures what the instrument’s developer intended it to measure within a study’s setting and population, and reliable results when the instrument provides the same score if repeatedly implemented among the same population. researchers have further specified various elements that assist in demonstrating the validity of information obtained via an instrument. we used these definitions when examining the instruments we gathered. tables 2 and 3 include the codes we assigned to each element, to make our reporting table more compact.   table 2  definitions of validity title code definition evidence face validity v1       “the instrument appears to measure what it claims to measure” (gay, mills, & airasian, 2006 as quoted in connaway & radford, 2017, p. 82) demonstrated through pre-testing, ideally with subjects similar to the target population, and with instrument development specialists content validity: item v2a “…the items of the instrument or test …represent measurement in the intended content area” (connaway & radford, 2017, p. 81). demonstrated through item analysis during pre-testing   content validity: sampling v2b “…how well the instrument samples the total content area” (connaway & radford, 2017, pp. 81-82) demonstrated through discussion of included constructs construct validity v3 “…instrument measures the construct in question and no other.” (connaway & radford, 2017, p. 83)   demonstrated through factor analysis, other tests of dimensionality, to retain convergent (contributing) items and remove divergent (non-contributing) ones intercoder reliability v4 degree to which scorers or raters agree on evaluating or observing a variable (connaway & radford, 2017, p. 316). demonstrated through percentage agreement among raters we used these definitions of reliability, and assigned these codes: table 3 definitions of reliability title code definition evidence internal consistency r1 how well items on a test relate to each other.  (connaway & radford, 2017, p. 84) demonstrated through cronbach’s alpha, kuder-richardson 20 tests (catalano, 2016, p. 8). measurement reliability r2 “the degree to which an instrument accurately and consistently measures whatever it measures” (connaway & radford, 2017, p. 83). demonstrated through test-retest correlation, meaning repeated administration to same group of the whole instrument (catalano, 2016, p. 8) or split-half method, meaning correlation of scores obtained from each half of a tested population or from each half of an instrument that measures a single construct (catalano, p. 9; connaway & radford, pp. 83-84).     we described an instrument as “resusable” only when we could answer three questions about the instrument: are the number of questions reported? is the full text of each question provided, and associated items? is the format of each question described: scale values, anchor labels such as “very satisfied” and “very unsatisfied”? if we felt that we had to guess as to whether the author fully described an instrument, we labeled it not replicable. we automatically coded appended instruments as replicable. the most frequent reason for describing an instrument as “not replicable” was that authors did not supply the number of questions and items, so that we could not be sure if they had described the entire instrument.   as explained earlier, we restricted the definitions of validity and reliability to those used by connaway and radford (2017), with occasional details borrowed from catalano (2016). we decided not to use the more expansive definitions that siddiq et al. (2016) employed in which for example an instrument’s having a basis in theory could be perceived as evidence that it produced valid results. we developed our own evidence extraction sheet to avoid obscuring the definition of each of these concepts: we wanted to focus on precise definitions of validity and reliability, and siddiq et al.’s criteria extended beyond those definitions.   after completing the process of acquiring and screening studies, we jointly read and reviewed six studies in duplicate and compared our extracted data, to ensure that we agreed. we then separately reviewed the remaining 23 studies and consulted with each other on any confusing elements. after our subsequent searches we divided responsibility similarly to review the nine additional articles. when we disagreed, we located more information on the issue to arrive at a consensus. for example, a disagreement about validity types might require refreshing our understanding of the definitions. we did not require a third party to resolve disagreements. we recorded our data in a shared spreadsheet.     figure 1 flow chart of review process   results   through our initial searches in seven databases, we found 2,189 articles that appeared to be relevant to our study. after removing duplicates, we were left with 2,108 relevant articles. we further reviewed the article titles and abstracts and found that 1,770 were not truly relevant to our study. we assessed the remaining 338 articles for eligibility and rejected 309 articles because they described instruments measuring satisfaction with only the library as a whole, or instruments measuring usage of or familiarity with reference services, or instruments measuring satisfaction with services other than reference. we extracted data from the final set of 29 studies. figure 1 shows a prisma flow chart of our process, a standard component of systematic review articles (moher, liberati, tetzlaff, & altman, 2009).   we concluded during the extraction process that certain criteria were more relevant to our focus on the instrument development process and therefore decided not to report irrelevant criteria such as “purpose of study,” “age of respondents,” “mode of service,” and “response rate.” we excluded “time needed to complete the instrument” from this paper because many authors did not report it. additional criteria included article title, journal, age/level of student, type of institution, sampling strategy, technical aspects (e.g., distribution and survey period), and demographics gathered. although we gathered data in these categories, we found that these criteria did not further our understanding of how librarian researchers report on instrument development and implementation. our evidence tables focus on the following criteria: study; country; research design; data collection method; types of questions; validity evidence; reliability indicators; whether the instrument is appended; whether the instrument is mentioned in the abstract as appended; and replicability of the instrument if not appended. the list of extraction elements is in appendix b, and the full data extraction spreadsheet is available online (https://docs.google.com/spreadsheets/d/1m7myhnckbscrak9cqgpbg0x6z-d6v5qs-iwie4i92sq/edit#gid=0).   outside of the data relating to our research questions, the data on country of study might be of interest to researchers. ten studies took place in the united states of america; the second most common country was india with six studies. the remaining studies took place in china (2), ghana (1), jamaica (1), malaysia (2), nigeria (4), philippines (1), and taiwan (2).   these are the results of our data extraction as they relate to our research questions:   q1: how did lis researchers gather data on patron satisfaction with reference services in the years 2015-2016?   of the 29 studies we gathered, 18 (62%) were solely quantitative in design and one (3%) solely qualitative. we labeled ten (34%) studies as combining both quantitative and qualitative designs, but this was usually because we defined “mixed methods” broadly to allow open-ended questions to be called qualitative; only two studies (7%) (jacoby, ward, avery, & marcyk, 2016; verma & parang, 2015) were truly mixed using the more conservative approach as defined by fidel (2008) in which qualitative and quantitative methods were used to answer the same research question. askew (2015) and yap and cajes (2016) employed quantitative methods to ask students about their satisfaction with roaming reference service and qualitative methods to ask librarians about their experience with providing the service; these studies are therefore labeled “quantitative” for the purposes of this review.   twenty-six (90%) of the studies employed surveys alone to gather data. construction of the surveys varied; the number of items related to satisfaction ranged from 1 to 16. eight (29%) of the studies asked only about overall satisfaction with reference service, while another 10 (34%) included an item about overall service satisfaction as well as other attributes contributing to satisfaction. respondents were asked to consider aspects of librarian behaviour such as approachability and responsiveness, helpfulness, respect for confidentiality, and offering referrals; and aspects of librarian performance such as ability, accuracy, knowledge, and inspiring confidence. five instruments (blake et al., 2016; butler & byrd, 2016; huang, pu, chen, & chiu, 2015; jacoby et al., 2016; luo & buer, 2015) asked students to gauge their likeliness to use, re-use, or recommend the service. masrek and gaskin reported presenting respondents with 10 items about service quality, usefulness, and satisfaction (p. 42) but unfortunately did not provide the full text of the items within their article, making it impossible to determine how they conceptualized these elements of satisfaction.   researchers commonly used 5-point likert scales to quantify respondents’ agreement or disagreement with statements; this type of scale occurred in 13 (45%) of the 29 survey instruments.  three (10%) of these scales did not have the traditional neutral midpoint. two (7%) of the scales offered three positive scores versus two negative (sivagnanam & esmail, 2015; xie & sun, 2015); the third scale was recoded by its developers to have three negative scores and two positive (yan et al., 2015). similarly, researchers employing 3-point scales did not always include a midpoint; three studies (butler & byrd, 2016; ekere, omekwu, & nwoha, 2016; yap & cajes, 2016) offered two positive options and one negative. two 4-point scales (khobragade & lihitkar, 2016; yap & cahes, 2016) were likewise unbalanced, with three positive and one negative choices. duan (2016) used 4-point scales to measure satisfaction with different modes of reference service and 6-point scales to measure satisfaction with reference librarians’ behavior. the remaining scales ranged in size from two scale points to nine.   most authors used typical labels for scale points, e.g., variations on “very satisfied,” “somewhat satisfied,” “satisfied,” and “very dissatisfied,” “somewhat dissatisfied,” and “dissatisfied,” or related labels such as “useful,” and “adequate,” but some authors labeled scale points differently from these norms. duan (2016) provided explanatory text for each scale point, e.g., “unsatisfied, because they solved few of my problems but were not willing to help me again” (p. 164). sivagnanam and esmail (2015) labeled their scale points “not satisfied,” “not much satisfied,” “particularly satisfied,” “fairly satisfied,” “absolutely satisfied.” yan, et al. (2015) were not clear, as it seemed they gave their scale two midpoint labels, “neutral” and “not familiar.” most 5-point scales had a midpoint labeled “neutral” (askew, 2015; blevins, et al., 2016; boyce, 2015; huang et al, 2015; mohindra & kumar, 2015) or “neither agree nor disagree” (jacoby et al., 2016; masrek & gaskin, 2016). three authors did not report the label used for their midpoint (chen, 2016; ganaie, 2016; swoger & hoffman, 2015).   a list of studies with their associated research design, data collection method, and likert-scale type is available in table 4.   table 4 studies included in this review study research design data collection method types of questions akor & alhassan, 2015 quant. survey 4-point scale askew, 2015 quant. survey 5-point scale blake et al., 2016 mixed survey 4-point scale, open-ended blevins, deberg, & kiscaden, 2016 mixed survey 5-point scale, open-ended boyce, 2015 mixed survey choose from list, 5-point scale, yes/no, open-ended butler & byrd, 2016 mixed survey 3-point scale, open-ended chen, 2016 quant. survey (based on servqual) 5-point scale dahan, taib, zainudin, & ismail, 2016 quant. survey (based on libqual) 9-point scale duan, 2016 quant. survey 6-point and 4-point scales ekere, omekwu, & nwoha, 2016 quant. survey 3-point scale ganaie, 2016 quant. survey 5-point scale huang, pu, chen, & chiu, 2015 quant. survey 5-point scale ikolo, 2015 quant. survey 2-point scale jacoby, ward, avery, & marcyk, 2016 mixed survey, focus groups, interviews 5-point scale; open-ended khobragade & lihitkar, 2016 quant. survey 4-point scale kloda & moore, 2016 mixed survey 3-point scale, open-ended luo & buer, 2015 mixed survey 5-point scale, open-ended masrek & gaskin, 2016 quant. survey 5-point scale mohindra & kumar, 2015 quant. survey 5-point scale nicholas et al., 2015 mixed survey choose from list, open-ended sivagnanam & esmail, 2015 quant. survey 5-point scale swoger & hoffman, 2015 mixed survey 5-point scale, open-ended tiemo & ateboh, 2016 quant. survey 4-point scale verma & laltlanmawii, 2016 quant. survey 3-point scale verma & parang, 2015 mixed surveys, interviews 3-point scales watts & mahfood, 2015 qual. focus groups open-ended xie & sun, 2015 quant. survey 5-point scale yan, hu, & hu, 2015 quant. survey 5-point scale yap & cajes, 2016 quant. survey 3-point and 4-point scales; another scale not specified     q2: to what extent are the instruments documented or included in the body of a publication?   nine articles (31%) included the instrument in full in an appendix, and of the remaining studies we found that eight instruments (28% of the total) were replicable according to our criteria as described earlier. detailed information is provided in table 5.   we noticed that two of the instruments (duan, 2016; xie & sun, 2015) were translated into chinese as well as english; both versions were available to respondents, but the author described the english-language instrument within the publication. we were unable to determine if any differences might exist between the two versions.   q3: to what extent are the instruments’ reliability and validity documented?   twelve publications (41%) included a discussion of any type of validity; five (17%) included discussion of any type of reliability. three articles (10%) demonstrated more than one type of validity evidence. see table 6 for a complete list.   validity evidence   face validity   face validity was the most common type of validity represented, as nine authors (31%) had pre-tested their instruments; however, we found that in two cases (7%) (akor & alhassan, 2015; blevins, deberg, & kiscaden, 2016) only librarian colleagues participated rather than members of the target population or instrument development specialists. the pre-testing process with potential respondents varied; blake et al. (2016) held campus interview sessions, while butler and byrd (2016) informally polled library student employees. kloda and moore (2016) and three sets of researchers (askew, 2015; huang et al., 2015; masrek & gaskin, 2016) presented instrument drafts to members of their respondent population. only three studies (10%) specifically reported pre-testing with a population contrasted with librarians and therefore presumably instrument development specialists: chen (2016) met with “academic experts” (p. 319); blake et al. worked with “experts from the university’s educational innovation institute” (p. 227), and masrek and gaskin (2016) pre-tested their instrument with “experts in the faculty” (p. 42).   table 5 reusability of instruments within studies study instrument appended reusability of instrument akor & alhassan, 2015 no no askew, 2015 no yes blake et al., 2016 yes (online) yes blevins, deberg, & kiscaden, 2016 yes yes boyce, 2015 yes yes butler & byrd, 2016 yes (online) yes chen, 2016 no yes dahan, taib, zainudin, & ismail, 2016 no yes duan, 2016 no no ekere, omekwu, & nwoha, 2016 no yes ganaie, 2016 no no huang, pu, chen, & chiu, 2015 yes yes ikolo, 2015 no yes jacoby, ward, avery, & marcyk, 2016 yes yes khobragade & lihitkar, 2016 no no kloda & moore, 2016 no yes luo & buer, 2015 no yes masrek & gaskin, 2016 no no mohindra & kumar, 2015 no no nicholas et al., 2015 no no sivagnanam & esmail, 2015 no no swoger & hoffman, 2015 yes yes tiemo & ateboh, 2016 no yes verma & laltlanmawii, 2016 no no verma & parang, 2015 no no watts & mahfood, 2015 yes yes xie & sun, 2015 yes yes yan, hu, & hu, 2015 no no yap & cajes, 2016 no no     table 6 validitya and reliabilityb study validity evidence reliability indicators akor & alhassan, 2015 v1 not stated askew, 2015 v1 not stated blake et al., 2016 v1, v2a, v3 not stated blevins, deberg, & kiscaden, 2016 v1 not stated boyce, 2015 not stated not stated butler & byrd, 2016 v1 not stated chen, 2016 v1 r1 dahan, taib, zainudin, & ismail, 2016 v3 r1 duan, 2016 not stated not stated ekere, omekwu, & nwoha, 2016 not stated not stated ganaie, 2016 not stated not stated huang, pu, chen, & chiu, 2015 v1, v3 r1 ikolo, 2015 not stated not stated jacoby, ward, avery, & marcyk, 2016 v4 not stated khobragade & lihitkar, 2016 not stated not stated kloda & moore, 2016 v1 not stated luo & buer, 2015 v2b not stated masrek & gaskin, 2016 v1, v3 r1 mohindra & kumar, 2015 not stated not stated nicholas et al., 2015 not stated not stated sivagnanam & esmail, 2015 v1 not stated swoger & hoffman, 2015 c not stated tiemo & ateboh, 2016 not stated not stated verma & laltlanmawii, 2016 not stated not stated verma & parang, 2015 not stated not stated watts & mahfood, 2015 not stated not stated xie & sun, 2015 not stated not stated yan, hu, & hu, 2015 v3 r1 yap & cajes, 2016 not stated not stated av1 = face validity; v2a = content validity (item); v2b = content validity (sampling); v3 = construct validity; v4 = intercoder reliability. see table 2 for full definitions. br1 = internal consistency. see table 3 for full definitions. cintercoder reliability coefficients not reported.     content validity: item   blake et al. (2016) were the sole authors to refer to item validity as part of their instrument development process, borrowing the definition from another paper by calling it “internal structure” (downing, 2003, as cited in blake et al., p. 227).   content validity: sampling   luo and buer (2015) were the sole researchers to address sampling validity; their instrument measured variables drawn from the five areas outlined in the reference and user services association’s (rusa) guidelines for behavioral performance of reference and information service providers, as well as from past research on evaluation of reference service.   construct validity   five publications (17%) addressed construct validity as demonstrated by factor analysis and other tests of dimensionality; three of these are described in the “multiple examples” section below because they tested construct validity along with other forms of validity. two studies (7%) addressed construct validity alone.    yan et al. (2015) determined the value of average variance extracted (ave) to demonstrate convergent validity of the constructs in their instrument, and reported that “all of the ave values range from 0.6727 to 0.8019” (p. 562), and considered these values satisfactory citing fornell and larcker’s 1981 publication in which 0.5 is the threshold value for ave. while not specifically using the term “divergent validity,” yan et al. demonstrated that they identified divergent variables, stating that “six variables ... are dropped due to their relatively low factor loadings for its construct” (p. 562).   dahan et al. (2016) used exploratory factor analysis, assessed using bartlett’s test for sphericity and the kaiser-meyer-olkin test, and determined that the analysis was significant (p. 41). they then conducted varimax testing with the kaiser normalization rotation method and found “that all varimax values are greater than 0.4 and therefore reflect the valid construct of all items” (p. 41).   intercoder reliability   of the ten mixed methods studies (34%), two authors (7%) presented validity evidence in their reports in the form of inter-rater agreement on thematic analysis. two sets of researchers (jacoby, ward, avery, & marcyk, 2016; swoger & hoffman, 2015) showed evidence of intercoder reliability, as they both discussed and reviewed their coding process; however, they did not report reliability coefficients.   multiple examples of validity evidence   blake et al. (2016) provided evidence of face validity, item validity, and construct validity within their study; huang et al. (2015) demonstrated testing for face validity and construct validity; and masrek and gaskin (2016) also showed evidence of a combination of face validity and construct validity. blake et al. changed their survey “to reflect the responses received from librarian reviews and campus interview sessions,” and consulted instrument development experts who helped them address content (item) and internal structure (construct) validity components (p. 227). huang et al. invited 15 members of the college faculty to participate in their pre-test, changing the wording of some items based on the faculty’s suggestions (p. 1181), and tested for convergent and divergent validity using composite validity and average variance extracted; they determined that convergent validity was “good” and discriminant validity was “strong” (p. 1185). masrek and gaskin showed evidence of a combination of face validity and construct validity as they pre-tested their instrument with students who were part of the target population, as well as with experts in the faculty, and as they analyzed the scales within their instrument for convergent and discriminant validity (p. 44). we did not find evidence that any of these researchers looked for convergent and discriminant validity with similar or different instruments.   reliability indicators   most (83%) of the studies did not state if they had tested their instruments for reliability. five articles (17%) (chen, 2016; dahan, et al., 2016; huang et al., 2015; masrek & gaskin, 2016; yan, et al., 2015) reported measurement of internal consistency for each component of the satisfaction construct when the component was measured by multiple scale items. all of the researchers used cronbach's ɑ (alpha) test of internal consistency, in which a value of 0.70 is commonly believed to be a basic threshold of acceptable level (nunnally, 1978). chen (2016) reported ɑ values ranging from 0.7305 to 0.8020, which represented a “satisfactory level of reliability” (p. 322). dahan et al. (2016) reported alpha values ranging from 0.813 to 0.942 (p. 41). values in huang et al.’s study (2015) ranged from .809 to .919 (p. 1184). masrek & gaskin recorded ɑ “well above 0.7” (p. 42), with values ranging from 0.707 to 0.812. yan, et al. (2015) did not report separate values for each factor, stating that “cronbach's alphas of all factors exceed 0.8” (p. 562).   discussion   this rapid review demonstrates that a less comprehensive and time-consuming type of systematic review of measurement properties can be a useful approach to gaining an overview of research by practicing librarians, as well as pointing to areas for improvement. our review confirms some aspects of research studies that other librarian researchers have attended to and identifies opportunities for further research. this discussion will place our results within a broader context, followed by recommendations for improvements in practicing librarians’ instrument design.   while solely quantitative study designs continue to be most common in studies of satisfaction with reference services, approximately one-third of the studies we located also gathered patron feedback via open-ended questions. for comparison, vanscoy and fontana determined in their study of research approaches to reference and information service that quantitative studies ranged from 56.65% in 2005 to 83.33% in 2009 (2016, p. 96). in our review, researchers used surveys alone to gather data 86% of the time, which is higher than the usage of surveys by 50.5% of practitioner researchers according to hildreth and aytac (2007) or the 62.3% of researchers writing about library instruction, as determined by crawford and feldt (2007, p. 84). however, our results are not surprising given the quantitative design and measurement goal of the studies we identified. we found an improvement over mckechnie et al.’s study in which 17.6% of articles included an appended instrument (2016), with 31% providing this service.   mahmood’s systematic review found that likert scales “or likert-type scoring methods” were used in 15 of 22 scales, and that the “points for scoring options ranged from 2 to 11” but did not report further detail about the design of each likert scale (p. 1044). it could be useful therefore to compare our results regarding likert-scale design with studies outside of lis. our partial model for this study, siddiq et al.’s systematic review of information and communication technology-literacy assessment instruments, did not include this information, but roth, ogrin, and schmitz (2016) reported in their systematic instrument review that of seven instruments containing likert scales, three employed 4-point likert scales, three contained 5-point scales, and two had 7-point scales. these findings indicate that little agreement exists as to best practices in scale formation. research on likert scale questions suggests that 4-point response scales with a “no opinion” option avoid the 5-point scale’s potential for central tendency bias (respondent desire to appear moderate rather than extreme) or social desirability bias (respondent desire to avoid controversial topics). this research implies that if a 5-point scale is offered, the midpoint should be clearly labeled, as otherwise respondents might assign various meanings to the midpoint such as “don’t know,” “neutral,” or “unsure” (nadler, weston, & voyles, 2015).   librarian researchers might not adequately define “satisfaction,” as only four researchers (14%) developed question items addressing more than one aspect of this construct. lasda bergman and holden (2010) identified four components of the satisfaction construct: willingness to return, positivity of experience, staff quality, and willingness to recommend a service to a colleague. luo and buer’s instrument (2015) included 10 components to express satisfaction but did not address another potential component: ethical issues as identified by kloda and moore’s (2016) question item, “the consult reflected a respect for my confidentiality as a library user.” on their survey measuring satisfaction with digital library service, including virtual reference, masrek and gaskin (2016) included 24 items representing 6 component factors of satisfaction, in addition to three items related to overall satisfaction (masrek & gaskin, personal communication, april 20, 2017). instrument developers might consider that responses to a single question about satisfaction are likely to be positive because “providing tailored individual help … will always be appreciated, which skews user satisfaction in survey results” (fournier & sikora, 2015, p. 255). when measuring multiple aspects of the satisfaction construct, a researcher can determine which aspect most likely detracts or adds to patron satisfaction, and initiate training, other services, and environmental improvements to address any issues.   we were surprised to find that four of the studies we examined (akor & alhassan, 2015; dahan et al., 2016; duan, 2016; xie & sun, 2015) contained “double-barreled questions” (olson, 2008, p. 210) or “multi-concept” to use glynn’s phrase (2006, p. 394), which asked respondents to agree with statements containing two themes combined with “and” such as “librarians are competent and helpful,” or to rate librarians’ “help and answers.” because the researcher doesn’t know which aspect of librarian service respondents are rating – competence or helpfulness? help or answers? – these items cannot contribute meaningfully to statistical analysis. moreover, respondents will likely take more time to consider each concept, potentially leading to survey fatigue. bassili and scott (1996) found that “questions took significantly longer to answer when they contained two themes than when either of their themes was presented alone” (p. 394). if researchers might design a survey instrument addressing the various components that make up the satisfaction construct, and thus listing several items to cover these components, it is important to make the items as simple to answer as possible, to encourage respondents to complete the survey. researchers will usually catch multi-concept questions during a careful pre-testing process.     half of the studies we located contained evidence of instrument validity, while more than three-quarters did not report data on instrument reliability, which is comparable to results from mahmood’s (2017) systematic review of instruments, and results from similar reviews in other disciplines. while explaining that the study excluded articles without validity or reliability evidence, mahmood (2017) stated that “a large number of studies reported surveys on assessing students’ self-efficacy in il skills but without mentioning any reliability and validity of scales” and that “the present study’s results are consistent with systematic reviews in other areas,” reporting that between 25% and 50% of studies in three systematic reviews outside of librarianship included information on validity and reliability of instruments (p. 1045):   for example, the reliability and validity were reported in only one-third of studies about evaluation methods of continuing medical education. . . . a study of 11 urbanicity scales found that psychometric characteristics were not reported for eight instruments. . . . a recent systematic review in the area of assessing students’ communication skills found that less than half of studies reported information on reliability and validity . . . (p. 1045).   our model instrument review researchers siddiq et al. found that 12 of 30 test developers (40%) reported validation of the test in at least one publication, and that 24 of the 30 (80%) reported reliability evidence according to the authors’ criteria (p. 75). the reporting of validity and reliability evidence can help the reader determine which instrument to use in replicating a study and could aid in future development of an instrument that might combine constructs and items identified through a similar review.   recommendations   obtain training and refer to research-evaluation checklists   compared to classroom faculty, librarians are frequently at a disadvantage in designing research projects because they lack coursework in research methods. initiatives such as loyola marymount university’s institute for research design in librarianship, the medical library association’s research training institute for health sciences librarians, and occasional professional development opportunities, assist librarians to build their research knowledge but can’t reach every librarian. for these reasons, we recommend that librarians become more familiar with existing checklists of research evaluation (e.g., those provided by glynn, 2006; and mckechnie et al, 2016) that can ensure a basic level of structure and rigor, and further recommend that researchers expand upon these lists as the need for research guidance becomes apparent. based on our study, we believe that checklists for librarians need to include more guidance in instrument design and in communicating instrument details, e.g., by making sure the target construct is adequately measured; by addressing validity and reliability; by designing questions and response items carefully; including the full instrument; and citing prior instruments.   completely measure the construct   when designing a research instrument, a researcher needs to determine which construct to measure, and which items will best represent that construct, whether it be satisfaction or any other construct. the researcher should keep in mind that more specific items avoid the problem of confounding variables which influence the respondent’s answer, or of misinterpretation in which the respondent’s definition of a construct differs from the researcher’s intended definition. in the realm of “satisfaction” with a service, many factors could influence respondents’ opinion of the service being measured, such as librarian behaviour or performance. it is therefore important to offer several items, rather than a single question about satisfaction.   address validity and reliability   after drafting questions and items, researchers will want to ensure their instrument has face validity by pre-testing it, with non-librarian subjects similar to the target respondent population and with experts in instrument design. these pilot testers should look for bias, for example avoiding questions such as “how much has this service improved your life?” which assume a positive response; for clarity and avoiding the use of jargon, defining terms as otto (2016) recommended; and for evidence that the question or item addresses what it is intended to address. if researchers try to address all variables encompassing the “satisfaction” construct and report this effort in their paper, that will show evidence of sampling validity. an instrument with many items could be refined by performing analyses to determine convergent and divergent items, thus demonstrating construct validity. if the instrument has been translated into or from a language other than english, developers should report separate validity and reliability information for each version of the instrument.   design questions and response items carefully   we repeat glynn’s (2006) recommendation that not only questions but also their “response possibilities” should be “posed clearly enough to be able to elicit precise answers” (p. 389). glynn cautions that the likert scale (i.e., strongly agree, agree, no opinion, disagree, strongly disagree) “[lends itself] to subjectivity and therefore the accuracy of the response is questionable” (p. 394).  regardless of the scale researchers select, we recommend employing a 4-, 5-, or 7-point likert scale. avoiding 2-point scales allows for variance in opinion, and avoiding 9-point scales or higher avoids dilution of opinion. we further recommend that researchers label the scale points in a uniform fashion but minimally, e.g., “strongly/somewhat/agree” and “strongly/somewhat/disagree,” rather than offer lengthy definitions of each point as seen in duan(2016).   include the full instrument   several authorities on research (connaway & radford, 2017; glynn, 2006; mckechnie et al., 2016) also agree that, to quote glynn, “the data collection method must be described in such detail that it can easily be replicated” (2006, p. 393). ideally, these authors further agree, researchers would include their instrument within the body or as an appendix of their publication, or as an online appendix. we recommend expanding existing checklists for the evaluation of research in librarianship, e.g., glynn’s critical appraisal tool for library and information research (2006) and mckechnie et al.’s research rigour tactics (2016), to remind authors that when they include an appendix containing the instrument, they should also note its inclusion in their abstract, to increase the likelihood that future researchers will locate it. if this inclusion is not possible, then a detailed description of the instrument should be reported in the body of the paper:   the number of questions and items the full text of each question and associated item question format: scale range and endpoint labels, e.g., “agree” and “disagree” where relevant, the average time needed to complete the instrument   with this information in hand, researchers can readily reproduce the instrument and use it in their own research.   cite prior instruments   we recommend also that authors cite sources if they base their instrument on previous efforts, demonstrating connections with prior research and further helping to identify useful instruments. blevins et al. (2016) wrote that “three librarians reviewed the existing literature for similar surveys and developed a set of questions to assess customer service quality” (p. 287) but did not cite the similar surveys. by citing contributing studies, librarians uphold the professional value of encouraging their colleagues’ professional development as stated in the ala code of ethics (2008).   further research   as more librarians implement the instrument review methodology, opportunities for future research will abound. reviews are needed in other research areas, for example to evaluate instruments gathering librarian attitudes toward teaching, collection development, or collaborating with faculty. while we have presented one model for this methodology, there is ample room for improvement and refinement of the method; we foresee that specific standards for instrument review could be developed for librarianship. as described above, another opportunity for future research would be to examine concerns of sampling validity, i.e., which items best demonstrate the patron satisfaction construct.   limitations   as a rapid review examining two years of librarian research, this study’s results are not necessarily representative of the body of work on student satisfaction with academic library reference services. although we ran keyword as well as subject searches, it is possible that we did not gather all possible studies presenting librarian-developed instruments due to inconsistent indexing. it is possible that we have missed relevant articles due to not manually searching all lis journals related to our research topic.     our descriptive model does not extend to evaluation of the instrument’s appropriateness in different scenarios such as in-house research versus research intended for publication. while we generally recommend designing an instrument that offers questions with several items measuring the satisfaction construct, it could be appropriate to include a single question and item addressing satisfaction when service quality assurance is the goal. for example, swoger and hoffman (2015) incorporated a single question about the usefulness of a specific   type of reference service; in their context the single question was primarily used for local service evaluation and might have been appropriate.   conclusion   the quality of a research project depends on valid and reliable data collection methods. in preparation for a study, librarians should search broadly and attempt to locate the best instrument exemplars on which to model their own data-gathering method. if researchers do not have time for a comprehensive systematic review, the present study demonstrates that a rapid review can reveal a range of research instruments and guide the development of future instruments. it further demonstrates that the characteristics of librarian-produced research instruments vary widely, and that the quality of reporting varies as well. if librarians do not aim to produce high-quality data collection methods, we need to question our collective findings. by following the recommendations presented here, future researchers can build more robust lis literature.   acknowledgements   the authors would like to thank the university of portland for granting the lead author a sabbatical, during which much of the work of this study was accomplished.   references   akor, p. u., & alhassan, j. a. (2015). evaluation of reference services in academic libraries: a comparative analysis of three universities in nigeria. journal of balkan libraries union, 3(1), 24-29.   american library association. evaluation of reference and adult services committee. (1995). the reference assessment manual. ann arbor, mi: pierian press.   askew, c. (2015). a mixed methods approach to assessing roaming reference services. evidence based library and information practice, 10(2), 21-33. https://doi.org/10.18438/b8f60v   bassili, j. n., & scott, b. s. (1996). response latency as a signal to question problems in survey research. public opinion quarterly, 60(3), 390–399. https://doi.org/10.1086/297760   beile, p. (2008, march). information literacy assessment: a review of objective and interpretive methods and their uses. paper presented at the society for information technology & teacher education international conference (site), las vegas, nv. retrieved from http://eprints.rclis.org/15861/1/info%20lit%20assessment.pdf   blake, l., ballance, d., davies, k., gaines, j. k., mears, k., shipman, p., . . . burchfield, v. (2016). patron perception and utilization of an embedded librarian program. journal of the medical library association: jmla, 104(3), 226-230. https://doi.org/10.3163/1536-5050.104.3.008   blevins, a. e., deberg, j., & kiscaden, e. (2016). assessment of service desk quality at an academic health sciences library. medical reference services quarterly, 35(3), 285-293. https://doi.org/10.1080/02763869.2016.1189782   boyce, c. m. (2015). secret shopping as user experience assessment tool. public services quarterly, 11, 237-253. https://doi.org/10.1080/15228959.2015.1084903   butler, k., & byrd, j. (2016). research consultation assessment: perceptions of students and librarians. journal of academic librarianship, 42(1), 83-86. https://doi.org/10.1016/j.acalib.2015.10.011   catalano, a. j. (2016). streamlining lis research: a compendium of tried and true tests, measurements, and other instruments. santa barbara, ca: libraries unlimited.   chen, y. (2016). applying the dematel approach to identify the focus of library service quality. the electronic library, 34(2), 315-331.  https://doi.org/10.1108/el-08-2014-0134     connaway, l. s., & radford, m. l. (2017). research methods in library and information science (6th. ed.). santa barbara, ca: libraries unlimited.   crawford, g. a., & feldt, j. (2007). an analysis of the literature on instruction in academic libraries. reference & user services quarterly, 46(3), 77-87.   dahan, s. m., taib, m. y., zainudin, n. m., & ismail, f. (2016). surveying users' perception of academic library services quality: a case study in universiti malaysia pahang (ump) library. journal of academic librarianship, 42(1), 38-43. https://doi.org/10.1016/j.acalib.2015.10.006   duan, x. (2016). how they search, how they feel, and how to serve them? information needs and seeking behaviors of chinese students using academic libraries. international information & library review, 48(3), 157-168. https://doi.org/10.1080/10572317.2016.1204179   ekere, j. n., omekwu, c. o., & nwoha, c. m. (2016). users’ perception of the facilities, resources and services of the mtn digital library at the university of nigeria, nsukka. library philosophy & practice, 3815, 1-23.   fidel, r. (2008). are we there yet? mixed methods research in library and information science. library & information science research, 30(4), 265-272. https://doi.org/10.1016/j.lisr.2008.04.001   fornell, c., & larcker, d. f. (1981). evaluating structural equation models with unobservable variables and measurement error. journal of marketing research, 18(1), 39-50.   fournier, k., & sikora, l. (2015). individualized research consultations in academic libraries: a scoping review of practice and evaluation methods. evidence based library and information practice, 10(4), 247-267. https://doi.org/10.18438/b8zc7w   ganaie, s. a. (2016). satisfaction of library and information science students with the services provided by allama iqbal library of kashmir university. international research: journal of library and information science, 6(3), 504-512.   gerlich, b. k., & berard, g. l. (2007). introducing the read scale: qualitative statistics for academic reference services. georgia library quarterly, 43(4), 7-13.   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399.   gotch, c. m., & french, b. f. (2014). a systematic review of assessment literacy measures. educational measurement: issues and practice, 33(2), 14-18.   grant, m., & booth, a. (2009). a typology of reviews: an analysis of 14 review types and associated methodologies. health information and libraries journal, 26, 91-108.  https://doi.org/10.1111/j.1471-1842.2009.00848.x   greifeneder (2014). trends in information behaviour research. paper presented at the proceedings of isic: the information behaviour conference, leeds, england. retrieved from http://www.informationr.net/ir/19-4/isic/isic13.html   hildreth, c. r., & aytac, s. (2007). recent library practitioner research: a methodological analysis and critique. journal of education for library and information science, 48(3), 236-258.   huang, y., pu, y., chen, t., & chiu, p. (2015). development and evaluation of the mobile library service system success model. the electronic library, 33(6), 1174-1192.   ikolo, v. e. (2015). users satisfaction with library services: a case study of delta state university library. international journal of information and communication technology education, 11(2), 80-89.   jacoby, j., ward, d., avery, s., & marcyk, e. (2016). the value of chat reference services: a pilot study. portal: libraries and the academy, 16(1), 109-129.   jordan, j. l. (2015). additional search strategies may not be necessary for a rapid systematic review. evidence based library & information practice, 10(2), 150-152. https://doi.org/10.18438/b8fc77   khobragade, a. d., & lihitkar, s. r. (2016). evaluation of virtual reference service provided by iit libraries: a survey. desidoc journal of library & information technology, 36(1), 23-28.   kloda, l.a., & moore, a. j. (2016). evaluating reference consults in the academic library. in s. baughman, s.hiller, k. monroe, & a. pappalardo (eds.), proceedings of the 2016 library assessment conference building effective, sustainable, practical assessment (pp. 626-633). washington, dc: association of research libraries.   koufogiannakis, d. (2012). the state of systematic reviews in library and information studies. evidence based library & information practice, 7(2), 91-95.   koufogiannakis, d., slater, l., & crumley, e. (2004). a content analysis of librarianship research. journal of information science, 30(3), 227-239.   lasda bergman, e. m., & holden, i. i. (2010). user satisfaction with electronic reference: a systematic review. reference services review, 38(3), 493-509.  https://doi.org/10.1108/00907321011084789   luo, l., & buer, v. b. (2015). reference service evaluation at an african academic library: the user perspective. library review, 64(8/9), 552-566.   mahmood, k. (2017). reliability and validity of self-efficacy scales assessing students’ information literacy skills. electronic library, 35(5), 1035-1051. https://doi.org/10.1108/el-03-2016-0056   masrek, m. n., & gaskin, j. e. (2016). assessing users satisfaction with web digital library: the case of universiti teknologi mara. the international journal of information and learning technology, 33(1), 36-56.   mckechnie, l. e., baker, l., greenwood, m., & julien, h. (2002). research method trends in human information literature. new review of information behaviour research, 3, 113-125.   mckechnie, l., chabot, r., dalmer, n., julien, h., & mabbott, c. (2016, september). writing and reading the results: the reporting of research rigour tactics in information behaviour research as evident in the published proceedings of the biennial isic conferences, 1996 – 2014. paper presented at the isic: the information behaviour conference, zadar, croatia. retrieved from http://www.informationr.net/ir/21-4/isic/isic1604.html    mckibbon, k. a. (2006). systematic reviews and librarians. library trends, 55(1), 202-215.  https://doi.org/10.1353/lib.2006.0049   moher, d., liberati, a., tetzlaff, j., & altman, d. g. (2009). preferred reporting items for systematic reviews and meta-analyses: the prisma statement. plos medicine, 6(7), e1000097.   https://doi.org/10.1371/journal.pmed.1000097   mohindra, r., & kumar, a. (2015). user satisfaction regarding quality of library services of a.c. joshi library, panjab university, chandigarh. desidoc journal of library & information technology, 35(1), 54-60.   nadler, j. t., weston, r., & voyles, e. c. (2015). stuck in the middle: the use and interpretation of mid-points in items on questionnaires. journal of general psychology, 142(2), 71-89.  https://doi.org/10.1080/00221309.2014.994590   nicholas, p., sterling, j., davis, r., lewis, j. c., mckoy-johnson, f., nelson, k., tugwell, y., & tyrell, k. (2015). “bringing the library to you!”: the halls of residence librarian program at the university of the west indies, mona library. new library world, 116(5/6), 316-335.   nunnally, j. c. (1978). psychometric theory. new york, ny: mcgraw-hill.   olson, k. (2008). double-barreled question. in p. j. lavrakas (ed.), encyclopedia of survey research methods (p. 210). thousand oaks, ca: sage publications ltd.  https://doi.org/10.4135/9781412963947.n145   otto, j. j. (2016). a resonant message: aligning scholar values and open access objectives in oa policy outreach to faculty and graduate students. journal of librarianship and scholarly communication, 4, ep2152. https://doi.org/10.7710/2162-3309.2152   phelps, s. f., & campbell, n. (2012). systematic reviews in theory and practice for library and information studies. library & information research, 36(112), 6-15.   reference and user services association. (2008). definitions of reference. retrieved from http://www.ala.org/rusa/guidelines/definitionsreference   roth, a., ogrin, s., & schmitz, b. (2016). assessing self-regulated learning in higher education: a systematic literature review of self-report instruments. educational assessment, evaluation and accountability, 28(3), 225-250.   schilling, k., & applegate, r. (2012). best methods for evaluating educational impact: a comparison of the efficacy of commonly used measures of library instruction. journal of the medical library association: jmla, 100(4), 258-269. https://doi.org/10.3163/1536-5050.100.4.007   siddiq, f., hatlevik, o. e., olsen, r. v., throndsen, i., & scherer, r. (2016). taking a future perspective by learning from the past – a systematic review of assessment instruments that aim to measure primary and secondary school students' ict literacy. educational research review, 19, 58-84. https://doi.org/10.1016/j.edurev.2016.05.002   sivagnanam, r., & esmail, s. m. (2015). user satisfaction and performance of arts and science college libraries situated in cuddalore district: a study. international journal of library and information studies, 5(5), 145-155.   swoger, b. j. m., & hoffman, k. d. (2015). taking notes at the reference desk: assessing and improving student learning. reference services review, 43(2), 199-214.  https://doi.org/10.1108/rsr-11-2014-0054   tiemo, p. a., & ateboh, b. a. (2016). users’ satisfaction with library information resources and services: a case study college of health sciences library niger delta university, amassoma, nigeria. journal of education and practice, 7(16): 54-59.   vanscoy, a., & fontana, c. (2016). how reference and information service is studied: research approaches and methods. library & information science research, 38(2), 94-100.  https://doi.org/10.1016/j.lisr.2016.04.002   verma, m. k., & laltlanmawii, r. (2016). use and user’s satisfaction on library resources and services by ug students of government hrangbana college, aizawl: a study. journal of advances in library and information science, 5(1), 18-23.   verma, m. k., & parang, b. (2015). use and user’s satisfaction on library resources and services by students of school of physical sciences, mizoram university, aizawl: a study. information studies, 21(2), 193-204.   watts, j., & mahfood, s. (2015). collaborating with faculty to assess research consultations for graduate students. behavioral & social sciences librarian, 34(2), 70-87.  https://doi.org/10.1080/01639269.2015.1042819   xie, j., & sun, l. (2015). exploring chinese students' perspective on reference services at chinese academic libraries: a case study approach. journal of academic librarianship, 41(3), 228-235.  https://doi.org/10.1016/j.acalib.2015.04.002   xu, j., kang, q., & song, z. (2015). the current state of systematic reviews in library and information studies. library & information science research, 37(4), 296-310. https://doi.org/10.1016/j.lisr.2015.11.003   yan, w., hu, y., & hu, c. (2015). using sem and tcm to improve services to libraries: a comparative study on chinese academic libraries. journal of academic librarianship, 41(5), 558-566. https://doi.org/10.1016/j.acalib.2015.07.006   yap, j. m., & cajes, m. l. (2016). where are the librarians? the roving reference experience of dlsu libraries. information world/bilgi dünyası, 17(2), 232-246. retrieved from http://bd.org.tr/index.php/bd/article/view/506/575      appendix a database searches   cinahl search id search terms s5 librar* and (reference or consultation or roaming or chat) and (satisfaction or attitudes) and (survey* or instrument* or questionnaire* or interview* or focus group*) s4 (s1 and s2 and s3) s3 (mh "research instruments+") s2 (mh "consumer satisfaction+") s1 (mh "library reference services")   eric search id search terms s9 ((surveys or questionnaires or instruments or measures or interviews)) and (s2 and s4 and s7 and s8) s8 (surveys or questionnaires or instruments or measures or interviews) s7 de "libraries" or de "research libraries" or de "medical libraries" or de "college libraries" or de "academic libraries" or de "two year colleges" s6 ((surveys or questionnaires or instruments or measures or interviews) and (de "surveys" or de "attitude measures" or de "interviews" or de "measures (individuals)" or de "questionnaires" or de "research" or de "semi structured interviews" or de "structured interviews")) and (s1 and s2 and s4 and s5) s5 (surveys or questionnaires or instruments or measures or interviews) and (de "surveys" or de "attitude measures" or de "interviews" or de "measures (individuals)" or de "questionnaires" or de "research" or de "semi structured interviews" or de "structured interviews") s4 de "attitudes" or de "satisfaction" or de "job satisfaction" or de "life satisfaction" or de "marital satisfaction" or de "participant satisfaction" or de "student satisfaction" or de "user satisfaction (information)" or de "school attitudes" or de "student attitudes" s2 de "reference services" or de "library services" s1 (de "academic libraries" or de "college libraries" or de "research libraries") and (de "reference services" or de "library services")   pubmed –with applied cosmin filter search id search terms s1 academic library and reference or consultation or roaming or chat) and (satisfaction or attitudes) and (survey* or instrument* or questionnaire* or interview* or focus group*) s2 academic librar* and reference or consultation or roaming or chat) and (satisfaction or attitudes) and (survey* or instrument* or questionnaire* or interview* or focus group*) s3 (academic librar*) and (reference or consultation or roaming or chat) and (satisfaction or attitudes) and (survey* or instrument* or questionnaire* or interview* or focus group*) s4 ((academic library) and consumer satisfaction) and (survey or instrument or questionnaire or interview or focus group) s5 (((library reference services) and consumer satisfaction) and (((((survey) or instrument) or questionnaire) or interview) or focus group))   proquest library science database (ls) –  initial search search id search terms s1 all((academic* and librar* and (reference or "customer satisfaction" or "user satisfaction" or "customer services") )) and su(research or surveys or questionnaires) and ab(study or survey* or interview* or research*)   proquest library science database (ls)subsequent search search id search terms s1 all((reference or "research consultation") and (satisf* or evaluat* or assess* or improve*) and (experiment* or survey* or qualitative or servqual or instrument or investigat* or analysis or questionnaire*) and "academic librar*")   google scholar search id search terms s1 "library reference services" and "consumer satisfaction" and research instruments" s2 academic library and (reference or consultation or roaming or chat ) and (satisfaction or attitudes ) and ( survey* or instrument* or questionnaire* or interview* or focus group*) s3 librar* and reference and satisfaction s4 librar* and (reference or consultation or roaming or chat) and (satisfaction or attitudes) and (survey* or instrument* or questionnaire* or interview* or focus group*) s5 librar* and (reference or consultation or roaming or chat) and (satisfaction or attitudes) and (survey* or instrument* or questionnaire* or interview* or focus group*) s6 library and reference and satisfaction s7 (academic librar*) and (reference or consultation or roaming or chat) and (satisfaction or attitudes) and (survey* or instrument* or questionnaire* or interview* or focus group*)   library, information science & technology abstracts (lista) – initial search search id search terms s1 (reference or "research consultation" ) and ( satisf* or evaluat* or assess* or improve*) and ( experiment* or survey* or qualitative or servqual or instrument or investigat* or analysis or questionnaire*) and "academic librar*" s2 (academic and librar* and (reference or "user satisfaction")) and ( su ( research or surveys or questionnaires ) or ab (study or survey* or interview* or research*) )   lista – subsequent search search id search terms s1 librar* and reference and satisfaction   web of science search id search terms s1 (academic librar*) and (survey* or instrument* or questionnaire* or interview* or focus group*) and (satisfaction or attitudes) and (reference or consultation or roaming or chat) s2 (library reference services) and (survey* or instrument* or questionnaire* or interview* or focus group*) and (satisfaction or attitudes) s3 (consumer satisfaction) and (library reference services) s4 (consumer satisfaction) and (library reference services) and (survey* or instrument* or questionnaire* or interview* or focus group*) s5 (librar*) and (consumer satisfaction) and (survey* or instrument* or questionnaire* or interview* or focus group*) and (reference or consultation or roaming or chat) s6 (librar*) and (consumer satisfaction) and (survey* or instrument* or questionnaire* or interview* or focus group*) and (reference or consultation or roaming or chat)     appendix b data extraction elements apa citation article title journal country stated purpose of study area of reference service: general, desk, embedded librarianship, external, research consultation, virtual reference (chat or email) age/level of students -if students were part of the targeted population targeted population usable responses sampling strategy research design data collection method types of questions, other than demographic demographics gathered technical aspects, such as distribution, availability of translations, duration of survey period time allotted to complete survey validity indicators reliability indicators instrument availability replicability of instrument comments       evidence summary   academic library patrons value personalized attention and subject matter expertise in reference consultations   a review of: rogers, e., & carrier, h. s. (2017). a qualitative investigation of patrons’ experiences with academic library research consultations. reference services review, 45(1), 18–37. https://doi.org/10.1108/rsr-04-2016-0029     reviewed by: jennifer kaari digital medical librarian allmedx new york, new york, united states of america jkaari@allmedx.com   received: 3 dec. 2018                                                                    accepted:  7 mar. 2019      2019 kaari. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29540     abstract   objective – to examine the experiences of patrons with one-on-one reference consultation services.   design – qualitative analysis of open-ended interviews.   setting – academic library at a public university in the southern united states.   subjects – students who attended a consultation with a reference librarian.   methods – all students who attended a reference consultation with a librarian were invited to participate in an interview. open-ended interviews were conducted after informed consent was collected. interviewers were provided with prompts to help participants discuss their experiences but were not intended to guide the conversation. the interviews were recorded and then transcribed line-by-line. the transcripts were then analyzed using a conventional, inductive model of content analysis. transcripts were first analyzed in an initial phase to identify basic themes, and then further examined in an advanced analysis in light of these themes.   main results – 10 students agreed to participate for a response rate of 38%.  most participants became aware of the reference consultation service by receiving library instruction as part of their course or through word-of-mouth recommendations from peers or faculty. no participants were aware of consultations through library marketing efforts or the library website. the major theme that emerged from the analysis was that patrons chose a reference consultation because it allowed them one-on-one attention from the librarian and because of the librarian’s perceived subject expertise. the primary problems participants identified with the service were that it was not adequately marketed to the students and that students were not aware of the service. participants intended to use the skills and information gathered from the consultation to continue their independent research and they also largely intended to use librarian’s services as they continue working on their projects.   conclusion – the authors found that the reference consultation is a valuable service for academic libraries and that consultation with a librarian in their office provides unique perceived benefits to the patrons compared to a traditional reference desk interaction. further research is suggested to determine the value of consultations for distance or online students, to ensure that reference consultations services are sustainable, and to further examine student’s emotive reactions to the consultation experience.   commentary   in a time of limited resources and increasing change for academic libraries, establishing the value of time and resource-intensive services is essential. the authors of this study investigate the value of face-to-face research consultation services by examining the experiences of patrons. this study builds on the considerable literature looking at individualized reference consultations  by taking a qualitative approach using extensive interviews with individual students (faix, macdonald, & taxakis, 2014; fournier & sikora, 2015; magi & mardeusz, 2013).   this study was assessed using glynn’s critical appraisal tool for library and information research and was found to have an 80% validity rating (2006). the research methodology is clearly described and the outcomes are thoroughly discussed. as the interviews were intended to be open-ended, there was no survey instrument used. however, the authors do state that there were prompts provided to the interviewers to help guide the discussion, and it would have been beneficial for these to be included with the paper. in addition, full transcripts of the interviews would have also been helpful for assessing the author’s conclusions.   the authors openly recognize that volunteer bias was a major concern and a potential weakness in their study. interviewers were directed to ask participants why they agreed to the study to determine the presence of volunteer bias. the responses did confirm that volunteer bias was a major limit to the study; a majority expressed that they wanted to participate because they wanted to help the librarians due to their positive experiences with the library. the authors acknowledge that this limits their study, although overall they feel that the results are still valid, which is supported by the 80% rating found in the critical appraisal process.   the researchers were librarians at the institution and they ensured that no participants were interviewed by the librarian with whom they had a consultation. however, the fact that the interviewers were also librarians could be another source of bias in the results and this is not addressed by the authors. the study could possibly have been stronger if the interviewers were a more neutral party, to ameliorate the possibility of participant bias.   despite the limitations, this study adds insight to patron’s experiences with reference consultations and potentially opens up new areas to examine in terms of both analyzing patron experience and marketing of consultation services. particularly notable was that the authors found that patrons highly valued librarians’ perceived subject matter expertise, despite the fact that librarians at this institution are hired as and marketed as generalists. if this tension was further examined in research, it could have considerable practical implications for the model of reference services at both this and other institutions. this study would be useful for librarians and library administrators in academic libraries looking for insights into how reference consultation is structured and perceived by patrons at other institutions as well as a basis for further research.   references   faix, a., macdonald, a., & taxakis, b. (2014). research consultation effectiveness for freshman and senior undergraduate students. reference services review, 42(1), 4–15. https://doi.org/10.1108/rsr-05-2013-0024   fournier, k., & sikora, l. (2015). individualized research consultations in academic libraries: a scoping review of practice and evaluation methods. evidence based library and information practice, 10(4), 247–267. https://doi.org/10.18438/b8zc7w   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387–399. https://doi.org/10.1108/07378830610692154   magi, t. j., & mardeusz, p. e. (2013). why some students continue to value individual, face-to-face research consultations in a technology-rich world. college & research libraries, 74(6), 605–618. https://doi.org/10.5860/crl12-363         evidence summary   library leadership faced numerous challenges during the covid-19 pandemic   a review of: shaghaei, n., knowles, c., morley, f., eveleigh, a., casaldàliga, n., nolin, e., tatai, a., cohen, m., pronk, m., & ghesquière, e. (2022). library resilience and leadership in a global crisis. liber quarterly: the journal of the association of european research libraries, 32(1), 1-21. https://doi.org/10.53377/lq.10930   reviewed by: kathy grams associate professor of pharmacy practice massachusetts college of pharmacy and health sciences boston, massachusetts, united states of america email: kathy.grams@mcphs.edu   received: 22 aug. 2022                                                             accepted:  7 oct. 2022      2022 grams. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30228     abstract   objective – to investigate the experiences, perceptions, and principles put into action by library leaders during the covid-19 crisis.   design – survey questionnaire.   setting – european organization of research libraries webinar series.   subjects – webinar attendees and viewers of recorded webinar series.   methods – in november 2020, the authors conducted two webinars titled “how are research libraries leading through covid-19?” and “new challenges and leading into the postcovid recovery for research libraries” for the fifth cohort of the liber emerging leaders’ programme. the authors drew on their own experiences, addressing leadership in a time of crisis, the challenges of remote leadership, and how to create clarity, build resilience, and catalyze positive change. the webinars were shared with previous cohorts of the liber emerging leaders’ programme. following the webinars, a link to an online survey was emailed to attendees and previous emerging leaders, as well as shared on social media. the survey was anonymous, open for a total of 21 days, and included a cover letter that stated its purpose. there were nine survey questions, eight of which were open-ended. the survey questions were grouped into four webinar themes; communication, strategy, values, and changes made during the covid-19 pandemic that library leaders would like to keep.   main results – the total number of respondents was 24; 84% were in leadership roles and 16% were employed as professional librarians. respondents were asked if their library’s strategic goals were still broadly relevant and asked to provide examples for how their existing strategies influenced their research library’s responses to the coronavirus crisis. of the respondents, 91% felt that their library’s strategic goals remained relevant during the coronavirus crisis. this was mainly due to the transformation to digitization (30%) and user-centered services (28%) that had occurred prior to the pandemic: digital resources, virtual training, the promotion of open access materials, more electronic books, digital services, and scan and deliver. respondents reported more user-centered strategies such as new reservation systems for study places, computer loans, click-and-collect, and postal loan. library values that were challenged during the pandemic were reported in the following categories: user-based (32%), collaboration (21%), social responsibility (21%), openness (16%), and collections or access (10%). within the theme of communication, 41% described it as negative which was defined as difficult, challenging, insufficient, overwhelming, chaotic, bad, or erratic. challenges of using online tools to communicate were described in categories of quality (24%), informal exchange (19%), time (21%), skills (17%), technical issues (9%), and leadership and personal issues (10%). the main challenges in communication related to not being able to interpret body language and non-verbal communication, lack of informal conversations or spontaneous interactions, increased time invested working, being permanently connected, difficulty acquiring the skills needed to use various tools, and the technological problems that exist when the network is interrupted. advantages noted with online communication tools were efficiency and accessibility. when asked for examples of techniques or methods used to communicate with staff, most reported communication as formal (70%) using tools such as slack, microsoft teams planner, jamboard, and whiteboards, while 22% of respondents reported informal communication strategies such as coffee via zoom, video lunches, informal mails, and a reading club.   conclusion – the covid-19 pandemic resulted in many challenges for research libraries that included maintaining strategic goals and values, communication, hybrid working, and flexible work schedules.   commentary   this research was appraised with boynton and greenhalgh’s (2004) guide to questionnaire research appraisal. the major limitations of the published report are the omission of the total number of attendees or the total number of people who viewed the webinars, as well as incomplete or unclear reporting of results, with few examples of the original open responses. the survey questions were derived from the discussions that took place in the webinar but could be considered narrowly focused.   liber (the association of european research libraries) (liber, n.d.) is a network of europe’s research libraries and includes approximately 440 national, university, and other libraries across 40 countries. liber has a vast membership, yet authors do not share how many members were part of the liber emerging leaders’ programme who were sent their webinars to view, nor how many members attended one or both webinars. a response of 24 is considered small, however the information gathered from these respondents confirms the many challenges of the pandemic that were previously published in the literature. since the survey was anonymous and included no demographic markers, it is also unclear if the 24 responses came from members of 24 unique institutions.   the final three questions of the authors’ survey relate to changes put in place that respondents felt they should keep post-covid-19: “what two or three covid-19 changes would you like to keep related to staff/human resources in the library?,” “what two or three covid-19 changes would you like to keep related to digital in the library?,” and “what two or three covid-19 changes would you like to keep related to physical spaces in the library?” the responses to these questions were not included or not differentiated in the results.   the overarching theme for this publication was based on building resilient libraries. authors state that the “paper discusses leadership skills and practical techniques that can be applied to help build resilient libraries and deliver positive new change in the post-covid-19 recovery period.” this was not completely achieved. the details of these “practical techniques” are not clear or not provided. much of the data collected reflect challenges of and the responses to a pandemic. it is unclear whether the decisions made by an institution during a pandemic reflect directly on any one individual member of the emerging leaders’ programme. the data does not illustrate whether these decisions were successful or efficient. additionally, it is difficult to extrapolate if the decisions reflect leadership skills because many leadership skills are soft skills and difficult to measure. the authors presented two webinars aimed at “mid-level management teams at university libraries.” subsequently, they provided a descriptive report on the challenges that libraries faced during the covid-19 pandemic through the experiences of 24 respondents, most of whom held leadership roles in their institutions. many of these challenges faced by library leaders were not unique to libraries. leaders in business, medicine, education, and more were faced with similar challenges navigating the remote work environment, adapting to new communication tools, and managing disruptions in network connections. as the world started to reopen, challenges faced by libraries were also similar to other establishments, such as the need for physical changes to maintain social distancing and maintaining hygiene in the workplace. the survey did not provide the opportunity for respondents to describe which techniques were successful or how libraries overcame the challenges faced during the pandemic. these details may be a measure of resilience and may help those readers who seek solutions. even though these were research libraries, the authors did not gather information on the change in research inquiries during the pandemic or which techniques, if any, were successful in sustaining research.   the definition of resilience is the “ability of a system or organization to respond to or recover readily from a crisis” or the “ability to be happy, successful, etc. again after something difficult or bad has happened.” (dictionary.com, 2022, cambridge university press, 2022) providing readers with a summary of the final three questions of the survey which reflected the changes respondents felt they should keep post-covid-19, may reflect the changes that were successful, keeping within the theme of resilience.   the authors successfully described that most libraries were in a unique position at the start of the pandemic because of the digitization of resources and services that occurred prior to the pandemic. a digital response to the pandemic was necessary and accelerated for those libraries who were behind in this digital age.   the authors conducted a literature search and presented a broad discussion relating to topics of strategies and values, communication, resilience, hybrid working, and flexible work schedules, providing readers with useful resources on the topics. they present challenges that were faced in a time of crisis from the literature and in their own research. although they may not have clearly described how to build resilience, the authors achieve the purpose of describing “experiences, perceptions, and principles put into action by library leaders during the covid-19 crisis”. library leaders may benefit from reviewing the background and aggregate data presented in this paper to build on the changes that were successful and to propose solutions for remaining challenges.   references   boynton, p. m., & greenhalgh, t. (2004). selecting, designing, and developing your questionnaire. bmj: british medical journal, 328(7451), 1312–1315. https://doi.org/10.1136/bmj.328.7451.1312   cambridge university press. (2022). definition of resilience. cambridge dictionary. https://dictionary.cambridge.org/us/dictionary/english/resilience   dictionary.com. (2022). definition of resilience. https://www.dictionary.com/browse/resilience   liber. (n.d.). about us. https://libereurope.eu/about-us/   shaghaei, n., knowles, c., morley, f., eveleigh, a., casaldàliga, n., nolin, e., tatai, a., cohen, m., pronk, m., & ghesquière, e. (2022). library resilience and leadership in a global crisis. liber quarterly: the journal of the association of european research libraries, 32(1), 1-21. https://doi.org/10.53377/lq.10930   evidence based library and information practice evidence based library and information practice 2010, 5.2  evidence based library and information practice       news/announcements    question, find, evaluate, apply: translating evidence based practice into library  instruction     2010. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which  permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly  attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the  same or similar license to this one.    the acrl instruction section (is) invites you  to attend “question, find, evaluate, apply:  translating evidence based practice into  library instructionʺ at the ala annual  conference in washington, dc.  when:  1:30 – 3:30 pm on sunday, june 27,  2010  where: room 146b, washington convention  center (wcc)  this year’s is annual conference program  will illustrate the direct connection between  evidence based practice (ebp) and  information literacy in an engaging and  practical panel. before tying together these  two essential concepts, panelists and evidence  based practice professionals megan  oakleaf (syracuse university) and diana  wakimoto (california state university, east  bay) will provide a strong foundation on the  background and theory of ebp. using their  personal research and expertise, they will  illustrate how ebp is not just the latest fad, but  in fact a functional way in which academic  librarians can market and teach information  literacy and inform assessment and  improvement of information literacy  programs.    what you will learn    to define evidence based practice in order to  explore the use of ebp models for information  literacy instruction and assessment.    to provide the means for evaluating the  applicability of ebp to information literacy  course or program assessment within a  particular institution.    to present ebp techniques that are applicable  in the library instruction classroom.  for more information and recommended  reading, visit the is committee’s website  at http://is.gd/cesgk. feel free to spread the  word with our flyer: http://is.gd/bu1km.  questions?     please contact amy deuink (ald120@psu.edu)  or michele ostrow  (micheleo@austin.utexas.edu).  we hope to see you there!    2010 acrl instruction section conference  program planning committee,   amy deuink and michele ostrow (co‐ chairs), jane currie, cynthia fisher, and sheila  stoeckel  115 http://is.gd/cesgk http://is.gd/bu1km mailto:ald120@psu.edu mailto:micheleo@austin.utexas.edu though virtual reference services have increased, they face challenges and opportunities in the wake of covid-19 evidence summary   though virtual reference services have increased, they face challenges and opportunities in the wake of covid-19   a review of: gerbig, m., holmes, k., lu, m., & tang, h. (2021). from bricks and mortar to bits and bytes: examining the changing state of reference services at the university of toronto libraries during covid-19. partnership, 16(1), 1–10. https://doi.org/10.21083/partnership.v16i1.6450   reviewed by: hilary bussell assistant professor, research and education the ohio state university libraries columbus, ohio, united states of america email: bussell.21@osu.edu   received: 30 nov. 2021                                                                 accepted:  17 jan. 2022      2022 bussell. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30082     abstract   objective – to compare data about the provision of reference services at the university of toronto libraries (utl) prior to and during the covid-19 pandemic, in order to identify obstacles and opportunities facing utl reference services in the future.   design – survey questionnaire.   setting – a large public research university in ontario, canada.   subjects – thirty-nine libraries across the three campuses of utl.   methods – a microsoft forms survey comprised of 37 questions was distributed in august and september 2020.   main results – twenty-four libraries responded to the survey, for a response rate of approximately 62%. utl’s chat service saw a 200% increase in september 2020 compared to september 2019 (since utl participates in chat as part of the ontario council of university libraries scholars portal, some traffic may have been from non-ut users). the option to book a reference appointment with a librarian was available at most of the libraries before the pandemic, and remained available during the pandemic. the survey results suggested that the shift to remote learning resulted in a significant expansion of virtual reference appointments; 75% of libraries reported offering virtual reference, compared to 17% before the pandemic.   consultations and in-depth reference questions rose during the pandemic, with a quarter of responding libraries reporting an increase. librarians became a larger share of the staff providing reference services during the pandemic, whereas the number of libraries using library technicians or student assistants to staff their reference services decreased. there were changes to formal reference service hours as well, with half of responding libraries reporting a reduction; however, most noted that they continued to answer reference questions over email at other times.   in response to the survey question asking for general comments about reference services, some respondents described worries about whether students taking only online classes would engage with online reference services, and whether overstressed faculty members would refer their students to librarians. several respondents noted positive outcomes in moving towards a primarily online reference model, including more options to connect with students and an uptick in reference requests.   conclusion – the authors note several challenges and opportunities for libraries in shifting to a remote reference model. challenges include confusion on the part of users about where to go for help and increased workload for librarians. opportunities include the chance to explore how virtual technologies can be used to make reference services more easily available to library users even after physical spaces have opened back up.   commentary   this article’s findings on the challenges of conducting reference and engaging faculty and students in a remote environment reflect other recent research on this topic (cohn & hyams, 2021; mehta & wang, 2020). while virtual reference services have been around for several decades, there was a sharp increase in virtual reference activity during the pandemic (ayeni et al., 2021). libraries have confronted numerous challenges in shifting most of their reference services online, including confusion on the part of library staff and users about how to use online resources and difficulties in collaborating on reference questions across library units (ayeni et al., 2021).   glynn’s (2006) critical appraisal checklist was used to evaluate this article. the study’s response rate of 62% is well above the average when surveying librarians (vander kooy, 2020). the use of descriptive statistics limits its generalizability to the wider library community; however, since its objective was focused only on understanding the local environment at utl, this is not a source of criticism.   there are a few methodological elements that would have been good to include, such as the survey instrument used and a description of how the authors analyzed responses to the additional comments or feedback. the article states that the survey was sent to each library, but it is unclear exactly to whom it was sent. this would be useful information, particularly for understanding the responses to the final question, which elicited reflections on the challenges and opportunities for reference services in the wake of the pandemic.   a notable finding is that the percentage of librarians staffing reference services increased during the pandemic, while the number of libraries with library technicians and student assistants performing reference decreased. it would be interesting to dig into why these staffing changes occurred and to compare them with staffing changes at similar institutions; such an exploration could potentially shed light on the types of library labour issues that have occurred in the wake of the pandemic.   this article joins a small but growing body of research on the impact of the covid-19 pandemic on academic library services. much of the literature on this topic to date consists of case studies (ayeni et al., 2021). this article’s use of a survey may provide a general guide for other libraries wanting to conduct a similar study at their institutions. while the article is limited in its generalizability, its recommendations for future research are valuable for librarians wanting to explore the impact of the pandemic on library services. these suggestions include studies on how well the changes to reference services met user needs as well as on the impact of the increased workload on library workers’ wellness and mental health.   references   ayeni, p. o., agbaje, b. o., & tippler, m. (2021). a systematic review of library services provision in response to covid-19 pandemic. evidence based library and information practice, 16(3), 67–104. https://doi.org/10.18438/eblip29902   cohn, s., & hyams, r. (2021). our year of remote reference: covid19’s impact on reference services and librarians. internet reference services quarterly, 25(4), 127–144. https://doi.org/10.1080/10875301.2021.1978031   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387–399. https://doi.org/10.1108/07378830610692154   mehta, d., & wang, x. (2020). covid-19 and digital library services – a case study of a university library. digital library perspectives, 36(4), 351–363. https://doi.org/10.1108/dlp-05-2020-0030   vander kooy, s. a. (2020). surveying the surveyors: an analysis of the survey response rates of librarians. proceedings of the annual conference of cais / actes du congrès annuel de l’acsi. https://doi.org/10.29173/cais1185 research article   assessment on a dime: low cost user data collection for assessment   eric dillalogue assistant director of operations kislak center for special collections, rare books and manuscripts university of pennsylvania philadelphia, pennsylvania, united states of america edillalo@upenn.edu   michael koehn director for library operations augustus c. long health sciences library columbia university irving medical center new york city, new york, united states of america mdk2126@cumc.columbia.edu   received: 6 may 2019                                                                accepted: 27 mar. 2020      2020 dillalogue and koehn. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29582     abstract   objective – this article describes the construction and use of a low cost tool for capturing user demographics in a physical library.   methods – at the health sciences library of columbia university irving medical center, we created the tap in/tap out tool to learn about the demographic details of our library visitors, such as their status, school affiliation, and department. the tap in/tap out tool was implemented twice for two weeks in 2013 and 2017, with users voluntarily tapping their campus id when entering and leaving the library. we checked campus id numbers against university databases to fill in demographic details of the library users.   results – we constructed the tap in/tap out tool using a raspberry pi and rfid card readers mounted on a foam board poster and placed near the library entrance. participation in the tap in/tap out tool ranged from 5-7% of the library gate count numbers during the survey periods. though low, this participation provided a useful indication of user demographics that helped to strengthen library discussions with university administration. the 2013 survey results, which showed that the library space was actively used by students from all the constituent medical center schools, were used to support funding justifications. the 2017 survey results, which showed continued library usage, were used to illustrate the value of the library to the medical center community.   conclusion – the tap in/tap out tool was inexpensive to implement and provided more information about library visitors than gate counts alone. findings from the tap in/tap out results were used to demonstrate library usage and justify funding. we describe how other libraries might create and implement the tool to capture greater levels of detail about the users visiting their spaces.     introduction   assessment is an important function in any library for evaluating usage, resource effectiveness, staff efficiency, and myriad other elements for internal and external purposes. though assessment is a constant point of discussion today, it is not a new initiative for libraries. in 1976, the association of research libraries laid out the need to assess all aspects of the library because “concern has increased for determining library output or effectiveness – how well the library actually meets users’ needs. this concern relates to the library’s responsibility to serve, with limited budgets, groups widely diversified … [and] by the need to further justify activities to the parent institution, to evaluate possible new services, and to predict and influence change” (p. 1).   traditionally, libraries have used gate counts as one way to assess space use – how many patrons entered or exited the space at a given time on a day. though this data provides insight into the level of library foot traffic – with characteristic spikes for testing periods and dips during breaks – the data does not include information on who used the space. as nackerud, fransen, peterson, and mastel (2013) note, there are important questions that go unanswered if all we know are numbers: “who are these students? what colleges do they represent? does the library reach a majority of students in some measurable way?” (p. 140). of particular interest to libraries are the demographic details of visitors: their university affiliation, status, school, department, and/or program. these demographic details allow libraries to demonstrate value to the wider campus through assessment efforts.   however, gathering patron demographic information can be difficult. some libraries have id-swipe gate systems that consistently record demographic information, but libraries without this gate functionality face a technological hurdle. to address this issue, we have developed a tool called “tap in/tap out” to acquire patron demographics at the entry and exit points of the library. we will discuss the technical aspects of setting up a tap in/tap out tool, the data we collected, and how we used that information.   the purpose of this article is to present a low cost user information capture tool that can be adapted for assessment in many types of settings. we discuss the tool’s use in a pre-renovation environment, with the acquired information used to demonstrate library usage across five health sciences schools and to make a case for funding, and in a post-renovation environment to explore current user demographics. though there are some limitations to the tap in/tap out tool, we believe it provides an innovative method to assess library usage at the point of entry. this paper describes how to create and implement the tap in/tap out tool so other libraries that lack the technological infrastructure or financial means to install swipe-card entry gates can capture greater levels of detail about the people using their spaces.   literature review   a common way to assess and compare library usage is to count the number of users who visit a library, most often done with gate counts at the entry point(s). whether using mechanical turnstiles or infrared beam-counting gates, gate counts “remain one of the most widely accepted methods for measuring facility use” (stewart, 2011, p. 539). typically, commercial providers supply gates that combine methods of tracking library materials (e.g., books) and counting people.   but gate counts with any device only provide information on how many users enter or exit a space and do not include information on who the users were nor how long they stayed in the library space. the lack of patron details has been partially addressed in newer gate technologies that require a user to swipe their id for access, allowing the library to count the number of visitors and, when connected to a database of patron information, access additional demographic details on those visitors (jones, 2011). for example, scarletto, burhanna, and richardson (2013) were able to determine the demographics of users visiting the library during overnight hours by capturing swipe-card entries at the library’s entrance and sending “identification numbers of users during the study period … to the university research, planning and institutional effectiveness (rpie) office for associated demographic data relating to department, major, grade point average, class standing, international status, home campus, ethnicity, and gender” (p. 373). using that information, the library had a more informed view of their overnight users.   however, swipe-card entry gates are expensive and not feasible for all libraries. to address the need for more user information to assess library usage, some libraries are using alternate demographic data collection methods to get a better understanding of their spaces and users. for example, dotson and garris (2008), employed building surveys to measure usage of their library’s physical resources. these surveys were used to “examine exactly what physical resources people are using” and to “use that information to make improvements to existing resources, services and spaces” (p. 11). nackerud, fransen, peterson, and mastel (2013) were able to capture unique patron identifiers via computer log-ins and circulation transactions, combining that data with other points of patron access, to analyze user demographics and determine which students were (or were not) using the library. for example, using this method they found “that 77 percent of undergraduate students and 85 percent of graduate students made use of the library during the fall 2011 semester” (p. 142). and for those undergraduate students who did visit, nackerud et al. found that they “consistently had higher semester gpas than students who made no use of the library” (p. 140). and lux, snyder, and boff (2016) used touch-based surveys loaded on ipads to capture patron demographics and reasons for coming to the library. the authors found that graduate students and faculty used the library to either study individually or to access library materials; whereas undergraduate students primarily visited to study alone or as part of a group (p. 112). similarly, there are studies that include observations and seating sweeps, but these do not provide details on who the users are in the space; hence, those studies are not included here.   any collection of user data also raises privacy concerns. as the american library association notes, “users expect … to have their personally identifiable information and library-use data protected and kept private and confidential …” and “libraries should collect and store only personally identifiable data required for specific purposes that are disclosed to the users” (2002). privacy has been described as a “tightrope” along which we walk and requires “a type of judgment call by a library staff member or administrator which pits the mission and goals of the library against user privacy” (coombs, 2004, p. 495). there is a wealth of thoughtful literature on privacy in libraries and we will not explore it all here, but we do address data collection issues with regards to the tap in/tap out tool in the methods section.   we find that jones’ statement from nearly a decade ago is still true, that there is an “absence of literature describing tools for analysis of and practical use for swipe-card data, particularly in informing library decisions …” (2010, p. 12). this paper adds to the literature on swipe-card data by presenting the tap in/tap out tool. this tool bridges the gap between the raw numbers from an infrared beam gate counter and the detailed user information from a swipe-card entry system, while keeping costs and staff involvement low.    background/aims   as do many libraries, the health sciences library at columbia university struggled to understand the populations who used its space. as the primary provider of study space on the columbia university irving medical center (cuimc) campus, library staff could determine that most library space users were students. beyond that, assumptions were made about the demographics of users in the space (status, school affiliation, program, etc.) based on staff observations and anecdotal evidence, but these assumptions were not backed up by quantitative evidence.   space is a highly valued commodity at cuimc. the library lost square footage at several points in its history.  for example, a renovation completed in 2009 converted more than 20,000 sq. ft. of library stacks space into classroom space. the library needed to demonstrate that students, faculty, and staff of all programs used the library’s remaining physical space. this would allow the library to advocate for enhanced maintenance of and upgrade to these spaces, which had seen only minor updates since the building was completed in 1975.   other libraries at columbia university installed entry gates activated by the university’s rfid badges, gaining useful demographic information about the users entering their spaces. rather than just a raw gate count, these libraries had a wealth of demographic information. users could be identified by their university affiliation (student, faculty, or staff),  their school or department, and their enrolled program. in addition, time of day data was available for each entry. this additional level of detail allows those libraries with swipe card entry gates to get more information than just a gate count – they can understand how users from different university departments or programs use their spaces and how the makeup of users changes over time. these data points help to strengthen library decision making and the case for funding.   administrators at the health sciences library examined the library’s setting and determined that, aside from the financial costs of implementing swipe-card access entry gates, the logistics of the library’s location created challenges as well. the library was located immediately behind entry gates for the hammer health sciences center, so there was no security benefit added through entry gates. in addition, since space users had swiped into the building just seconds before entering the library, staff thought space users would be frustrated by the need to swipe in again, especially given the challenges users were having with the existing gates.   but the basic concept of swipe-to-enter provided a starting point. the library already had rfid card readers in use for circulation transactions at the service desk. if similar readers could be coupled with an inexpensive data collection device, it might be possible to produce similar demographics to other libraries’ entry gates at a much lower cost. the raspberry pi (rpi) single board computer seemed like an ideal candidate for our data collection needs.   to avoid frustrating users of the space, data collection had to be voluntary and time limited. the resulting data would be a sample, but it would be an indicator of usage patterns in the space, and the information returned would be far more detailed than the library’s existing gate count data. staff realized that the same methodology could be used to also collect exit data, providing length-of-stay data. thus the tap in/tap out tool was created.   methods   we designed the tap in/tap out tool as a type of survey data collection device. specifically, this tool was created to take a snapshot of the number and types of patrons entering and exiting the library during a short sampling period. that tap in/tap out tool follows the concept of a “survey” defined by cohen, manion, and morrison (2007) as a method to “gather data at a particular point in time with the intention of describing the nature of existing conditions …" (p. 205). participation was voluntary. we were not seeking statistical significance in collecting the information, but rather data representative of use and demographics, providing more details than existing gate counts or patron in/out numbers. we used historical gate count information, academic calendars, and staff knowledge to choose the two weeks for surveying, selecting a time in the middle of the semester that avoided any school breaks or testing days. the study was conducted twice, once in 2013 (pre-renovation of the library) and once in 2017 (post-renovation). in addition, each iteration included two rounds of data collection, once in the spring and once in the fall semesters. the exact dates in 2013 and 2017 were different due to the academic calendars in those years.   the study was designed as a management study of the library’s patrons, rather than as a research study to develop generalizable knowledge. as such, it is not human subject research and did not require irb approval (columbia university, 2012). information on the purpose of the study, use of information collected, and contact information for additional details were posted at the point of participation. (see appendix a for posted text.) patron participation was voluntary. staff were instructed to encourage participation during the study, but patrons could, and often did, enter and leave the library without participating. this intervention was the extent of participation by front line staff, fulfilling the goal of low staff involvement.   several factors were considered when examining the risk of disclosure of collected data. id card numbers and date/time data stored on rpi could only be linked to an individual through secure university databases. access to these databases is restricted to employees with a legitimate business need. the work to link the id card numbers to user demographics was performed on secure endpoints as required by university policy. at no time was demographic data stored on the rpi. data on the rpi computers was protected by standard linux login security. theft was not considered a significant threat, given the low value of the hardware and constant surveillance of the space. given these factors, it was determined that accidental or malicious disclosure of collected data represented little to no harm to participants in the study.   experimental setup   the library used two rpi single-board computers and two rfid card readers compatible with the university’s id system. both rpis were running the latest build of raspbian linux available at the time. for the 2013 study, the library used first generation model b rpis, which lack wi-fi capability. because the rpi has no real-time clock, network connectivity (via wired ethernet) was required for accurate timekeeping. for the 2017 study, we used third generation rpis with built-in wi-fi connectivity.   one of the goals of this project was to minimize the costs associated with technology and supplies.  the total bill of materials was just over $170 (see appendix b for a breakdown). we were able to use rfid card readers from our spare stock, otherwise these would have cost approximately $160.00 each. the above hardware would have sufficed for the 2017 rounds of data collection, but we elected to update our hardware to gain wi-fi networking. for the 2017 project, we spent a similar amount, less the cost of the foam core board.   the rfid card readers are usb human interface devices, appearing to the rpis as a keyboard. when an id card is placed on the reader, it reads the card number and outputs it as a series of digits, followed by the enter key (similar to an individual typing the number and pressing enter).   one rpi was designated “in” and the other was designated “out” for collecting library entry and exit data, respectively. both rpis were configured to turn off the graphical interface and to auto-login on the first virtual terminal (/dev/tty1) to the “cardstats” user, which ran a script, written in perl, to collect the 9-digit card number along with date and time each time a card was read, and write it to local storage. if the script were to crash or exit, the account would log out, and the auto-login process would re-start it.   the library produced a poster, printed on foam core board, with graphics indicating which rfid reader was “in,” which was “out,” and mounted the rfid readers to the board with tape (figure 1). in addition, the poster contained information about the purpose of the project and how the collected card data would be used (see appendix a for the poster text). to maximize impact, the library designed a reminder poster that could be seen easily as users entered or exited the library.   figure 1 poster design for tap in/tap out on one board.   the poster was placed on a tripod in a conspicuous location near the entry of the library, near power, and, for the 2013 study, near ethernet ports.   each time the survey ran for just over two weeks, to allow 14 full days of data collection. to protect against accidental data loss, card data was copied daily to a secure location. during the 2013 study, staff used scp to retrieve the data daily. for the 2017 study, a nightly job ran to copy the data to a windows share via samba.   data analysis   data collected was saved in a tab-delimited, plain text file, with the first column containing the card number, the second column an iso 8601 date and time statement, and the third column a directional indicator (in or out). (see appendix c for an example of the raw data.) after collection, data was transferred to a secure endpoint for further analysis.   for the 2013 study, data was analyzed using a custom perl script (appendix d). during this study period, the library closed daily before midnight. therefore, any in taps without a corresponding out tap at the end of the day represented instances where the users had tapped in, but not tapped out. (see appendix c for an example of merged data.)   after collecting all card data, the list of card numbers was sent to an analyst at columbia university information technology (cuit), who matched the card numbers against the university’s id management database. cuit returned the list with added affiliation information, including role (student, faculty, or staff), school, division, department, and academic program.  (see appendix c for example demographic data.)   the 2017 study involved some adjustments to the data analysis process. the change to the library’s hours to a 24-hour space required the ability to compare data spanning two days. it was no longer possible to determine who had left without tapping their id as we did in the 2013 study. to resolve this issue, we used a sliding six hour window based on the tap in time to look for any exit taps. if no matching exit was found within the six hour window, we assumed the user left the library without tapping. we chose the six hour window based on the duration of visit information from the 2013 study, where only one user exceeded four hours in the library and none exceeded six hours.    results/outcomes   the primary use of the tap in/tap out data has been to quantitatively demonstrate to university administration the continuing value of library space to the medical center community. the data strengthened the library’s case for increased investment in library spaces. we found the two most useful reports for administration to be a tabular count by department (appendix e) and a pie chart of library visits by school (appendix f).   response rates to the tap in/tap out surveys were low – 5% of the gate count in both 2013 surveys and the fall 2017 survey, 7% in the spring 2017 survey – but provided a useful snapshot of patron demographics. the response rate was calculated for each day in the study period and remained within a 4% to 6.5% daily range. while we acknowledge that the information collected was not representative of all library users, it was still useful for management purposes to indicate usage and justify funding.   results from the tap surveys confirmed that the health sciences library was actively used by students from each of the four main cuimc schools – the college of dental medicine, joseph l. mailman school of public health (msph), school of nursing, and college of physicians & surgeons (now vagelos college of physicians and surgeons, vp&s) – as well as by the cross-campus students from the graduate school of arts and sciences and by students enrolled in programs at the university’s morningside heights campus. the results also showed taps from faculty and staff. since there had been a perception that the library was only a student space, this was useful hard data.   for those who tapped out, we were able to extrapolate the duration of their stay. for the 2013 study, approximately two-thirds of taps recorded were part of a matched pair, meaning we could determine the length of a user’s time in the library. analysis showed that 44% of respondents spent less than a half-hour in the library during their visit. the second largest group (about 30%) were those who stayed for at least one and up to three hours. this data supported anecdotal staff observations that a number of library users mainly came to access a computer or printer, and then left shortly thereafter. but it also showed that we also had users who spent more time in individual study or on collaborative projects.   the 2013 study was used to determine the level of use from each of the health sciences schools. we found that, in both the spring and fall semesters, students from each of the cuimc schools visited the library in roughly proportional numbers in comparison to each school’s campus enrollment. the largest school by enrollment, the mailman school of public health (msph), were also the users who tapped the most. the finding that msph users were the largest library user population was a bit of a surprise. for many years, the working assumption had been that the vagelos college of physicians & surgeons (vp&s) students were the largest library user population. in addition, the duration data indicated that the space had to support both quick, in-and-out uses such as printing, as well as extended study sessions.   at the time of the 2013 study, the library had started planning a major renovation of its main reading room, but it had no source of funding for the improvements. the demonstration that students from all of the cuimc schools used the library space allowed the library to tap into funding earmarked to benefit students of all of the constituent schools. recognition that msph students were our largest community, along with knowledge that the msph curriculum focused heavily on small group cooperation pointed toward the need for spaces accommodating such groups. finally, duration data led us to design spaces suitable for both quick, in-and-out type visits, and for long study sessions.   the 2017 study results were used to examine the extent of library use (post-renovation) by students from each of the schools, as well as to compare with the 2013 results to assess if our patron base had changed. comparing the same periods for both survey periods, one of the most significant findings was that the vp&s students, one of our largest groups of users, had decreased in the number of taps by 11 percentage points for the spring semester and 13 percentage points for the fall. this decrease was expected, as vp&s had completed construction of a new medical education building in the interim, but it was a welcome confirmation of the effect of this change and a reassurance that the medical students had not completely abandoned the library, as was feared.   limitations   the primary limitation of the tap in/tap out tool is that the data is a self-selected sample. the study only captures a brief period of time and participation is voluntary. staff encouraged users to participate, but it was observed that not all users swiped their ids. as shown in the comparison of taps with gate count, participation was low. to overcome this, more active staff involvement in soliciting users to tap their ids could help to increase participation rates.   a second limitation is related to the concept of a “visit” to the library. users of the health sciences library space will interrupt their study time to access facilities outside of the library, such as restrooms and the café. if users return immediately after leaving, is this one visit or two? the correct answer to this question likely depends upon the intended case use for the collected data. however, it is possible to process the data and screen for an “out” tap followed closely by an “in” tap so as to remove these brief exits and re-entrances.   a third limitation is the requirements of the raspberry pi computer. as a data collection device, the rpi performed admirably for such an inexpensive device. as noted above, the lack of built-in wi-fi capability in the first generation rpi dictated some logistics of design and placement, which was resolved when we obtained a newer, wi-fi capable, version of the rpi. the need for electricity continues to constrain placement. in addition, as it is a general purpose computer, configuration requires staff with knowledge of the linux operating system and general programming that some libraries may not possess. however, the raspberry pi online community includes a wealth of helpful tutorials to configure and operate the computer. academic institutions with a computer science program could also potentially borrow the expertise of students.     conclusion   the tap in/tap out tool met the design needs of the library our goal in creating the tap in/tap out tool was to create a low-cost method of collecting more details about the people using the library space. the tap in/tap out tool met our design needs because the raspberry pi computer system and auxiliary items were low cost (under $200) and allowed us to gather more demographic information on our users, such as their status, school affiliation, and department. the tool was also voluntary and required little interaction from library staff.   the tap in/tap out tool allowed us to gather useful demographic data to inform decision making the data we collected with the tap in/tap out tool helped to flesh out the quantitative information we were collecting about library space use during gate counts and to back-up or refute staff observations about space use. the tool was a way to compensate for a lack of card-swipe entry gates and the demographic detail those gates can provide. the tap in/tap out tool allowed us to gather similar demographic information about the users visiting the space, such as their university affiliation, status, school, department, and/or program. the 2013 and 2017 surveys were used to assess library space visits by users from each of the medical center schools, make informed justifications for library funding and renovation, and to help show that the library is a vital element to the campus community.   the tap in/tap out tool data was persuasive to university administration the primary use of the tap in/tap out data was to quantitatively demonstrate to university administration the continuing value of library space to the medical center community. the 2013 survey results helped to demonstrate that students from all of the cuimc schools used the library space. this allowed the library to tap into funding earmarked to benefit students of all of the constituent schools. the 2017 survey findings helped to demonstrate to university administration that there was continued use by members from each cuimc school.   the tap in/tap out tool is a possible option for other institutions looking to gather more detailed user information institutions that want to know more than the number of people walking through their doors, but who do not have the technological or financial means to implement swipe-card entry gates, could use the tap in/tap out tool as a low-cost solution for gathering user information. as implemented, there are some limitations due to the voluntary nature of participation, but more active involvement by library staff in encouraging participation could overcome this. setting-up the tool is possible at most institutions, requiring only minimal technical knowledge of the linux operating system and a suitable programming language.   references   american library association. (2002). privacy: an interpretation of the library bill of rights. retrieved from http://www.ala.org/advocacy/intfreedom/librarybill/interpretations/privacy   association of research libraries. (1976). user statistics and studies. spec flyer, 25, 1-2. retrieved from https://catalog.hathitrust.org/record/000689858   cintron, j., courtier, d., & delooper, j. (2017). testing three types of raspberry pi counters. code{4}lib journal, 38. retrieved from https://journal.code4lib.org/articles/12947   cohen, l., manion, l., & morrison, k. (2007). research methods in education. (6th ed.). routledge.   columbia university. (2012). institutional review board standard operating procedures. retrieved from https://research.columbia.edu/sites/default/files/content/ hrpo/irb_sopsuperseded_v4.2_nov2.12.pdf   coombs, k. a. (2004). walking a tightrope: academic libraries and privacy. the journal of academic librarianship, 30(6), 493-498. https://doi.org/10.1016/j.acalib.2004.08.003   dotson, d. s., & garris, j. b. (2008). counting more than the gate: developing building use statistics to create better facilities for today's academic library users. library philosophy & practice, 1-13. retrieved from https://digitalcommons.unl.edu/libphilprac/208   jones, j. l. (2011). using library swipe-card data to inform decision making. georgia library quarterly, 48(2), 11-13. retrieved from https://scholarworks.gsu.edu/univ_lib_facpres/21   lux, v., snyder, r. j., & boff, c. (2016). why users come to the library: a case study of library and non-library units. the journal of academic librarianship, 42(2), 109-117. https://doi.org/10.1016/j.acalib.2016.01.004   nackerud, s., fransen, j., peterson, k., & mastel, k. (2013). analyzing demographics: assessing library use across the institution. portal, 13(2), 131-145. https://doi.org/10.1353/pla.2013.0017   sanders, m., & hodges, c. (2014). an overnight success?: usage patterns and demographics of academic library patrons during the overnight period from 11 p.m.-8 a.m. journal of access services, 11(4), 309-320. https://doi.org/10.1080/15367967.2014.945121   scarletto, e. a., burhanna, k. j., & richardson, e. (2013). wide awake at 4 am: a study of late night user behavior, perceptions and performance at an academic library. the journal of academic librarianship, 39(5), 371-377. https://doi.org/10.1016/j.acalib.2013.02.006   stewart, s. (2011). building measurements: assessing success of the library’s changing physical space. the journal of academic librarianship, 37(6), 539-541. https://doi.org/10.1016/j.acalib.2011.09.002   appendix a tap in/tap out poster text   why tap?   q: why is the library collecting data? a: the library is collecting data to know more about space use patterns.    q: how is the library collecting data? a: library users should tap their id cards when they enter and exit the library.    q: what data is the library collecting? a: the library is collecting id card number, time of tap, and direction (“in” or “out”).  cuit will look up card numbers and provide demographic information such as school and department.  the library will not have access to personally identifiable information.   q: how does this information help the library? a: the library will be better able to understand who is using the library, when, and for how long.  this understanding will inform space planning and future surveys.   for more information, please contact the library at hs-library@columbia.edu     appendix b the 2013 tap in/tap out study bill of materials   product unit price ($) quantity total ($) raspberry pi model b 35.00 2 70.00 rpi case 7.35 2 14.70 rpi power supply 7.00 2 14.00 8gb sd flash drive 10.75 2 21.50 printed foam core mounting board 50.00 1 50.00 total     170.20     appendix c sample of data collected   sample of raw data card number date/time direction xxxxxx365 2013-04-22t08:30:30 in xxxxxx985 2013-04-22t08:33:10 in xxxxxx121 2013-04-22t08:36:06 in   sample of merged data card number date/time in time in date/time out time out duration of visit (minutes) duration of visit hh:mm:ss xxxxxx379 2013-04-25t08:37:16 8:37:16 2013-04-25t08:44:12 8:44:12 6.93 0:06:56 xxxxxx839 2013-04-25t09:06:00 9:06:00 2013-04-25t09:34:37 9:34:37 28.62 0:28:37 xxxxxx324 unknown unknown 2013-04-25t09:39:40 9:39:40 unknown unknown xxxxxx991 2013-04-25t21:04:48 21:04:48 unknown unknown unknown unknown   sample of demographic data security role acad level school code school name div code div name acad dept code acad dept name prog code program name cuimc ft support staff p spub the joseph l. mailman school of public health ph pub health hpm health policy management pmhpm health policy and management cuimc ft student u snur school of nursing rn nursing-ugrd nurs school of nursing rnetp combined bs/ms program in nursing cuimc ft student p cops college of physicians and surgeons pt physcl thrpy phyt physical therapy ptdpt physical therapy cuimc ft student g snur school of nursing np nursing-grad nurs school of nursing npwomh women's health     appendix d perl script   1)      combine the in and out files for each date. 2)      sort the combined files for a single date in ascending order by timestamp. 3)      for each in record, save the card number and the time. 4)      for each out record, check previous ins to see if there’s a corresponding entry. a.      if yes, calculate duration of stay and output card number, entry and exit times, and duration, write the result to file, and clear the saved in entry. b.      if no, set in time and duration to unknown and write the result to file. 5)      at the end of processing, write out any unmatched in entries, setting out data and duration to unknown.     appendix e count by department   sun mon tue wed thu fri sat total college of dental medicine     7 4 6 2 2 21 college of physicians and surgeons 17 27 32 30 32 18 17 173 human nutrition   5 1 2 2 6   16 medicine 10 14 15 14 17 11 11 92 occupational therapy 1   1 2 1     5 physical therapy 6 8 15 12 12 1 6 60 columbia college   1           1 graduate school of arts and sciences 1 1 3 3 2 4 2 16 school of continuing education     1 1   1   3 school of general studies   2           2 school of nursing 4 9 21 11 6 3 4 58 the fu foundation school of engineering & applied science         1     1 the joseph l. mailman school of public health 15 46 41 47 36 23 11 219 biostatistics dept 2   2   3     7 environmental health sciences 1 3 1 1 1   1 8 epidemiology 7 23 15 23 16 12 7 103 health policy management 2 10 6 10 5 1 2 36 population and family health 1 3 7 4   7   22 public health     3   4     7 sociomedical sciences 2 7 7 9 7 3 1 36 (blank) 2 8 12 11 15 8 2 58 grand total 39 94 117 107 98 59 38 552 gate count 628 2425 2241 2344 1913 1428 572 11551 response rate 6.2% 3.9% 5.2% 4.6% 5.1% 4.1% 6.6% 4.8%     appendix f usage charts   usage chart 1 library visitors by school -spring 2013 (based on a 2-week sample)     usage chart 2 duration of visit by school -spring 2013 (based on a 2-week sample)   evidence summary   research supports are effective in increasing confidence with research skills in early career academic librarians   a review of: ackerman, e., hunter, j. & wilkinson, z. t. (2018). the availability and effectiveness of research supports for early career academic librarians. the journal of academic librarianship, 44(5), 553-568. https://doi.org/10.1108/ils-09-2016-0068   reviewed by: jessica a. koos senior assistant librarian/health sciences librarian stony brook university libraries stony brook, new york, united states of america email: jessica.koos@stonybrook.edu   received: 18 feb. 2020                                                              accepted:  7 apr. 2020      2020 koos. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29739     abstract   objective – to identify the type and efficacy of research supports currently available to early career academic librarians.   design – survey.   setting – the united states.   subjects – 213 academic librarians who were not yet promoted or have received tenure, or those up to three years post-tenure or promotion.    methods – the researchers created a survey containing 39 closed and open-ended questions using the software qualtrics. the question types included multiple choice, likert scale, and free text. the survey was distributed through direct emails and various professional electronic mailing lists.   main results – the majority of respondents listed finding time as the most significant barrier to conducting research. respondents listed informal mentoring as the most commonly used and most widely available form of research support. statistical analyses revealed that for every type of research support a librarian engaged in, on average confidence increased by 0.10.   conclusion – engagement in formal and informal research supports may influence early career academic librarians’ confidence levels in regards to conducting research projects. academic institutions as well as professional organizations should ensure that ample opportunities are available.   commentary   the quality of this study was appraised using “the cat: a generic critical appraisal tool” created by perryman and rathbun-grubb (2014). overall, the study was found to be of relatively high quality based on this assessment. the first author, ackerman, is a social sciences librarian, and the two co-authors are librarians as well. the research questions and methods were, for the most part, clearly explained. there was significant attention given to previous relevant literature throughout the article.   however, the authors did not discuss the limitations of the study. some of the limitations included sample size and the methods by which the survey was distributed, yet these were not acknowledged in the article. there was also no mention of how many surveys were distributed, which prevents the calculation of the response rate, which is used to determine the generalizability of the findings. there was the potential for a self-selection bias, as the participants who chose to take the survey may have had a greater interest in research than those that did not take the survey. additionally, the sample consisted of librarians that “have not yet earned tenure or promotion or to have been tenured or promoted within the last three years.” this definition seems to be fairly broad and does not necessarily match with the concept of “early career.” most of the respondents reported having “six to nine years of professional experience,” which means these librarians had been in the field for some time. an explanation of why this range of experience levels was included would have been helpful. regardless of these limitations, the article provides some very important insights into this area. it is unique to the literature in that it explores which types of research supports are most effective for early career academic librarians. one of the major implications of this research is that it is important for organizations that support these librarians to provide some mechanisms to help foster their development as researchers.   this research also provides evidence that time is the largest obstacle to librarians in accomplishing their research goals. interestingly, “designated research time” or “requestable research time” were the most popular formal supports provided by institutions, although they were found in less than half. employers should take the time factor into serious consideration when evaluating the workloads of their employees and also consider the inclusion of similar mechanisms to increase time available for research, especially when it is critical to promotion or tenure.   informal mentoring seems to be the most highly used form of research support, and the respondents indicated that it was significantly more helpful than formal mentoring. while there are a few brief explanations from the respondents as to some of the reasons as to why informal mentoring is preferred, this is an area that warrants additional study; a more complete understanding can help to inform institutional practices.   additionally, participation in collaborative research projects was the most prevalent theme when respondents were asked to provide written answers as to what factors contribute to scholarly success. increasing opportunities for networking with colleagues and facilitating collaborative research are important considerations.   finally, this study yields statistical evidence that early career librarians gain confidence incrementally with each research support that they access. thus, increasing the number and type of supports available can have a positive impact on research output. librarians, institutions, professional organizations, and even library science educators should take this under consideration when determining how to improve research confidence and skills.   references   perryman, c. & rathbun-grubb, s. (2014). the cat: a generic critical appraisal tool. in jotform – formbuilder. retrieved from http://www.jotform.us/cp1757/thecat         microsoft word es_needham.doc evidence based library and information practice 2007, 2:1  147 evidence based library and information practice       evidence summary    there is a significant relationship between computer attitudes and library anxiety  among african american graduate students      a review of:  jiao, qun g., and anthony j. onwuegbuzie. “the impact of information technology on library  anxiety: the role of computer attitudes.” information technology & libraries 23.4 (dec.  2004): 138‐44.    reviewed by:   gill needham  head of strategic and service development, the open university library and learning  resources centre  walton hall, milton keynes, united kingdom  e‐mail: g.needham@open.ac.uk      received: 2 june 2006    accepted: 8 january 2007      © 2007 needham. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to investigate whether african  american students’ computer attitudes  predict levels of library anxiety.    design – a user study in which two  instruments were administered to a group  of graduate students to measure computer  attitudes and library anxiety.    setting – the college of education at an  historically black college and university in  the united states of america.    subjects – ninety‐four, predominantly  female, african american graduate students,  ranging in age from 22‐62 years old, and  enrolled in either a statistics or a  measurement course.     methods – two instruments, the computer  attitude scale (cas) and the library  anxiety scale (las) were administered to  all the study participants. the computer  anxiety scale contains forty likert‐type  items that assess individuals’ attitudes  toward computers and their use. it includes  four scales which can be administered  separately:    1. anxiety or fear of computers  2. confidence in the ability to use  computers  3. liking or enjoying working with  computers  4. computer usefulness  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:1  148 the las contains forty‐three, 5‐point,  likert‐format items that assess levels of  library anxiety experienced by college  students. it also has five subscales as follows:  1. barriers with staff  2. affective barriers  3. comfort with the library  4. knowledge of the library  5. mechanical barriers    main results – there were twenty  correlations between the library anxiety  subscale scores and the computer attitude  subscale scores. four of these correlations  were statistically significant. liking or  enjoying working with computers was  statistically significantly linked to affective  barriers, comfort with the library, and  knowledge of the library. there was also a  statistically significant association between  an attitude of computer usefulness and  knowledge of the library.    conclusion – these findings suggest that in  this group of students there is a medium to  strong relationship between computer  attitudes and library anxiety.     commentary    the study sample consisted of 94 african  american graduate students. the authors  point out that it is not possible to extrapolate  the findings to other racial groups. the  students were enrolled in either a statistics  or a measurement course. these subjects do  not lend themselves to particularly high  levels of use of library resources, which may  also affect the transferability of the results.  no details are given in the paper about how,  where, or by whom the instruments were  administered. lack of familiarity with the  two instruments used in the study makes it  difficult to judge either their  appropriateness or reliability, but the  authors quote other studies in the discussion  that have used similar scales and produced  consistent results.    it would have been helpful to have more  specific data about the gender split in the  sample. the paper merely states that the  majority of the participants were women. it  is possible that the generalizability of the  results may be affected by the very wide age  range in the sample. the authors give the  mean age, but it would have been useful to  know the median age also. the median age  would illustrate whether it was primarily  participants under 25 with a couple of  outliers which increased the mean age.  further, how does the age range and gender  split compare with the previous studies  cited by the authors?    this paper is useful in confirming the  hypothesis that students who are less  confident with computers are more likely to  experience library anxiety. this finding is  not in itself surprising, but could be  valuable evidence to support bids for more  resources for the support and training of  less confident students.            modernization of the care library by the belgian red cross: leveraging digitalization and volunteers to reduce loneliness with library services using evidence in practice   modernization of the care library by the belgian red cross: leveraging digitalization and volunteers to reduce loneliness with library services   gaelle huysentruyt manager social activities & youth belgian red cross mechelen, belgium email: gaelle.huysentruyt@rodekruis.be   mathilde krols project manager humanitarian services belgian red cross mechelen, belgium email: mathilde.krols@rodekruis.be   liesbeth vercammen coordinator volunteering belgian red cross mechelen, belgium email: liesbeth.vercammen@rodekruis.be   fritz schiltz chief of staff belgian red cross mechelen, belgium email: fritz.schiltz@rodekruis.be   philippe vandekerckhove managing director, belgian red cross mechelen, belgium department of public health and primary care, faculty of medicine, ku leuven leuven, belgium centre for evidence-based health care, stellenbosch university cape town, south africa email: philippe.vandekerckhove@rodekruis.be   received: 28 dec. 2021                                                             accepted: 25 jan. 2022    2022 huysentruyt, krols, vercammen, schiltz, and vandekerckhove. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30091     setting   in 1937, the belgian red cross launched the care library, a library service tailored to the needs of people who are unable to visit a regular library. although originally targeted towards in-hospital patients, this service now mainly concerns people who are staying in residential care centres. the care library is essentially a social activity, aiming to prevent and/or break through loneliness, hence our slogan: more than a book. this is what distinguishes the care library from both web shops and regular libraries: it brings the library to people unable to visit in person and allows volunteers to spend time with them.   the central warehouse contains more than 200,000 materials adapted to the specific target group of the care library. there are also 40,000 books with large print, books for young people, picture books, and books in other languages for shelters for asylum seekers operated by the red cross. there is also guidance material to trigger memories. for example, scent boxes are used to play the game bingo. every player gets a game board and tries to complete it by correctly guessing the scent flasks. other examples include audio bingo games, pictures of everyday objects, memory games, and puzzles with pictures of the past.   lending points become a mini library with a tailored service, thanks to the materials from the central warehouse and the book tour by more than 700 volunteers. lending points have a small adapted part of the library collection they can use to lend out materials. seventy-seven percent (77%) of lending points are residential care centres, 15% are hospitals, and 6% psychiatric centres or others (2%). volunteers act as the point of contact between the care location and the library. as lending point coordinators, they ensure that materials are requested according to the wishes of the residents, and they make time to speak to residents. in this way, bonds of trust are created for people in vulnerable and sometimes lonely situations.   the belgian red cross team manages the collection, delivers tailor-made packages to the lending points, and trains the volunteers before they go on a book tour. the volunteers use a book cart with different materials to go on their book tour. if possible, they go from room to room and speak with the residents of the care centres. the training of volunteers hence guarantees both the quality of the collection and the service provided.   problem   in contrast to traditional libraries in belgium, which experienced a 30% drop in number of operating points between 2008 and 2015 (statistics flanders, 2016), there has been a strong increase in demand for the care library. in the past five years, the number of lending points has increased by almost 40%. a possible explanation is the shift in the target group. for example, 77% of the people reached by the lending points are residents of a residential care home. social contact therefore plays an even more important role, which distinguishes the care library from traditional libraries. demographic trends in belgium point to a growth of 22% in the number of people over 65 between 2017 and 2027, indicating a continued shift in the coming years.   the ever-growing demand poses challenges for the belgian red cross. in addition to the purchase of materials and the use of the warehouse, baumol’s law characterizes a library, like other services, as follows (baumol & bowen, 1966): slower growth of labour productivity compared to general productivity results in relatively high service costs. the care library is a free service, only possible thanks to the daily efforts of volunteers and support from government funding. a decrease in the number of volunteers in belgium (20% in the past five years) was compensated by the deployment of employees, which further increased the cost. in order to meet this challenge the belgian red cross invested heavily in the modernization of its central operations.   evidence   in the case study of the care library, there are two main levels of evidence. first, our internal research team conducted two systematic reviews of the literature on the impact of in-person visits and book reading on the well-being and health of senior citizens. second, we used a data-driven approach to tailor book deliveries to residents’ demands. both systematic reviews are currently under review. the protocols for these systematic reviews are published and available online (laermans et al., 2020a, 2020b).   the literature review provides a scientific base for the activities of the care library. it includes research on (1) the effect of friendly visiting by a volunteer on loneliness of residents of an assisted living facility (e.g. arthur, donnan, & lair, 1973; reinke & holmes, 1988), and (2) on the impact of book reading on the physical and mental health of older adults (e.g. bavishi, slade, & levy, 2016; krell-roesch et al., 2017). the former concludes that friendly visits benefit the life satisfaction and self-assessed health of residents of an assisted living facility, although it should be noted that further research is needed due to a lack of recent evidence and rather low quality of available evidence. the latter concludes that reading a book enhances both the mental health (reduced risk of developing dementia, and improved social connectedness), and physical health (lower mortality rates) of older adults. however, caution should be exercised regarding this latter set of results due to the relatively low quality of available evidence.   a data-driven approach enables deliveries and new purchases that are better attuned to the wishes of residents. this is possible through real-time insights into the collection, but also the size of the lending points (number of beds), participation (% readers), and areas of interest. volunteers who are paying friendly visits can submit data through an online portal, helping the central team to continuously improve its offerings. by equipping the lending points with specific materials, personal conversations are facilitated (e.g. reminiscence books with pictures of the past are used for residents with dementia).   implementation   the project “optimization central library” was launched, with the objective of modernizing the care library, in order to increase its capacity without increasing costs. the components of this project are shown in the figure 1.   figure 1 modernization of the care library in five steps.   the entire book collection was inventoried in a data warehouse by the belgian red cross team. each item in the collection was given a unique code and was linked to specific characteristics of the material such as its genre. reporting using cognos software allows insight into the stock. to interact with this new source of information, an application was developed: zorgbib.be. this application is publicly available and allows users to consult the collection directly. the application was developed because the purchase and disposal was previously based on the perception of the staff, volunteers, and institutions instead of standardized records. in addition, our customers have changed over the last years to a specific profile: seniors in care centres. to follow the growth in lending points for this target group, we needed an optimized stock system and maximal rotation of the existing collection.   lending points can use this application to request a change directly, but they can also enter their own data (e.g. number of readers). in addition, volunteers can follow an e-learning training for free before going on a book tour. when a new lending point is launched or a change is requested, a tailor-made package with materials is put together by employees and volunteers of the belgian red cross. when the package is delivered, the used materials are returned and added to the collection again.   outcome   the modernization as described above has improved the quality of the functioning of the care library, increased its reach, and reduced costs. between 2016 and 2021, the number of conversations, as logged by volunteers in the web-based application, increased by almost 60% to 158,000 per year. at the same time, financial resources are made available that can be used elsewhere by the belgian red cross. by offering training online and automating certain processes (e.g. requesting a change via an application instead of via email), staffing could be reduced by almost 40%. targeted purchases and better stock management mean that more lending points can be reached with the same materials. at the same time, the shorter shelf life of books means that less space is needed in the warehouse. figure 2 summarizes the increased reach of the care library (+145% since 2006), while government funding has remained constant during the same period (+1% after correcting for inflation in the same period).     figure 2 evolution of lending points and funding over time.   reflection   throughout the process of modernizing the care library, two challenges emerged. first, the ability to shift to a data-driven approach crucially hinges on the quality of the data. as volunteers provide data, a complementary training needed to be developed. volunteers can now follow online classes to get to know the application, making sure that data quality is maintained. in addition, volunteers (and employees) with lower levels of digital literacy are receiving in-person training, provided by staff of the belgian red cross. modernizing our process without supporting our volunteers along the way would limit – or nullify – the impact of the renewed care library.   second, cooperation with experienced partners helps to overcome technical difficulties. our team working in the care library has deep expertise in materials for our specific target groups, and in training volunteers to prepare for their visits. however, the team has limited expertise in digital transformations, requiring an experienced partner to support this transformation process. because of the organizational structure of the belgian red cross, support services such as it, facilities, and learning and development can be leveraged to successfully complete this project. for organizations that do not have this scale or organizational structure, our key recommendation is to find a partner that has experience in digital transformations such that the library team can fully focus on its core expertise.   author contributions   gaelle huysentruyt: project administration, supervision, writing – review & editing mathilde krols: project administration, writing– review & editing liesbeth vercammen: project administration fritz schiltz: writing – original draft, writing – review & editing philippe vandekerckhove: funding acquisition, supervision, writing – review & editing   references   arthur, g. l., donnan, h. h., & lair, c. v. (1973). companionship therapy with nursing home aged. the gerontologist, 13(2), 167-170. https://doi.org/10.1093/geront/13.2.167   baumol, w. j., bowen, w. g. (1966). performing arts, the economic dilemma: a study of problems common to theater, opera, music, and dance. twentieth century fund.   bavishi, a., slade, m. d., & levy, b. r. (2016). a chapter a day: association of book reading with longevity. social science & medicine, 164, 44-48. https://doi.org/10.1016/j.socscimed.2016.07.014   krell-roesch, j., vemuri, p., pink, a., roberts, r. o., stokin, g. b., mielke, m. m., christianson, t. j., knopman, d. s., petersen, r. c., kremers, w. k., & geda, y. e. (2017). association between mentally stimulating activities in late life and the outcome of incident mild cognitive impairment, with an analysis of the apoe ε4 genotype. jama neurology, 74(3), 332-338. https://doi.org/10.1001/jamaneurol.2016.3822     laermans, j., scheers, h., vandekerckhove, p., & de buck, e. (2020a). protocol: recreational book reading for promoting cognitive functioning and emotional well-being in older adults: a systematic review. campbell systematic reviews, 16(4). https://doi.org/10.1002/cl2.1117   laermans, j., scheers, h., vandekerckhove, p., & de buck, e. (2020b). protocol: friendly visiting by a volunteer for reducing loneliness and social isolation in older adults. campbell systematic reviews, 16(2), e1084. https://doi.org/10.1002/cl2.1084     reinke, b., & holmes, d. s. (1988). the “friendly visitor” program data: a reply to denney. american journal of community psychology, 16(3), 427-433. https://doi.org/10.1007/bf00919380      statistics flanders. (2016). bios library statistics. interactief platform bibliotheken. https://statistieken.vlaanderen.be/qvajaxzfc/opendoc.htm?document=cjsm%5ccjm%20-%20bib2.qvw&lang=nl-nl&host=qvs%40cwv100154&anonymous=true   editorial responsibilites editorial responsibilites © 2010. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. editor-in-chief: denise koufogiannakis associate editor (articles): alison brettle associate editor (evidence summaries): lorie kloda associate editor (features, commentaries, ebl101, news): denise koufogiannakis guest editor (features): andrew booth production editor: katrine mallan editorial intern: andrea baer copyeditors: priscilla stephenson (lead copyeditor), richard hayman, heather pretty, lisa shen, dale storie, mary virginia taylor, elizabeth zeeuw indexing support: pam morgan relationship between academic library workers’ outlooks on life, personality, and goal-setting behavior and achievement evidence summary   relationship between academic library workers’ outlooks on life, personality, and goal-setting behavior and achievement   a review of: lo, l.s. & anderson, a.m. (2020). personal goal setting behavior and professional outlooks of academic library employees. journal of new librarianship, 5, 204-236. https://doi.org/10.33011/newlibs/9/21   reviewed by: eugenia opuda health & human services librarian assistant professor dimond library university of new hampshire durham, new hampshire, united states of america email: eugenia.opuda@unh.edu   received: 27 aug. 2021                                                             accepted:  26 oct. 2021      2021 opuda. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30024     abstract   objective – to identify a correlation between academic library employees who set new year’s resolutions and goal-setting behavior in professional contexts, and to explore practices, personal attitudes, and outlooks that influence goal-setting and goal-achievement   design – non-experimental multiple choice questionnaire   setting – online   subjects – 308 adult participants (over 21 years old) who work in academic library settings including staff, librarians, and administration   methods – the authors designed an online, non-experimental multiple choice questionnaire through qualtrics. the authors distributed study invitations to multiple professional library listservs, though it is unclear which listservs were included and what geographic location was covered. the survey was available for roughly a month from february 1-26, 2016. the survey screened participant demographics to omit those under 21 years of age and all identifying information was removed in order to protect participant privacy. all participation was voluntary and participants who were interested in contributing to a follow-up research study were asked to share their contact emails.   main results – most participants (n=182, 59%) set no new year's resolutions in 2015 and half (n=155, 50%) set no resolutions in 2016. when asked to explain, 23% noted that they hadn't considered setting resolutions in 2016, 9% did not prioritize setting goals, and 5% felt that they could not achieve their goals. additionally, over 50% articulated other reasons including not prioritizing goal-setting for new year’s, noting that setting goals around the academic year was timelier, and that some participants already had enough goals to achieve. in 2016, half of participants (n=153, 50%) set new year’s resolutions. by far the most common resolution was physical fitness and healthy eating (n=64, 42%). about 19% set occupational goals including skill building, and 15% set emotional goals including cultivating optimism and mindfulness. when asked about goal-setting practices, 36% of the 2016 resolution setters described writing or typing out their goals, 59% shared their goals with others, and nearly 90% enacted changes in their daily routines in order to achieve their goals. 26 participants used all of the goal setting practices above. this group prioritized their top goals and felt confident about reaching those goals. four participants did not practice goal-setting techniques, and also felt less confident about achieving their goals. 49% of 2016 resolution setters had somewhat optimistic outlooks, and 24% had very optimistic life outlooks. of those with pessimistic life outlooks, nearly all believed it would be difficult to accomplish goals. respondents who claimed to be very ambitious were likely to set occupational goals as their top goal. 81% of those in dean and director positions reported being very ambitious and 85% also reported being optimistic. all deans and directors felt confident about accomplishing their goals. for middle managers, 75% felt ambitious and 72% felt optimistic. professional librarians were 66% ambitious and 72% optimistic.   conclusions – this study's findings align closely with united states national averages about the percentage of americans who set new year’s resolutions and achieve their goals. data suggests some relationship between academic library workers’ outlooks on life and confidence in achieving their goals, as well as a correlation between goal setting strategies and achieving goals. the authors express optimism that 20% of participants who set new year's resolutions chose to list occupational goals as their top goals, especially considering that resolution-setting comprises an incredibly broad array of options. the authors suggest that data can be used by academic library administrators to increase worker job performance, improve worker wellness, establish mentorship programs, and train workers to set attainable goals.   commentary   this research builds off of the authors’ previous work on the correlations between setting new year’s resolutions, outlook on life, and professional goal setting among a general population of library employees (anderson & lo, 2019) and focuses on a target population of academic library workers. dai et al. (2014) underscore the “fresh start effect” which suggests that people tend to undertake new goals most commonly during “temporal landmarks” such as new year’s. in a study about new year’s resolutions, oscarsson et al. (2020) identified a correlation between level of support with behavior change and goal-achievement. furthermore, locke and latham (1990) note the importance of leadership support and worker self-management on positive performance. the “fresh start effect” may kick off a desire to undertake new goals, which may lead to success in part through internal and external support and clarity.   this publication was appraised using the “ebl critical appraisal checklist” (glynn, 2006). the methodology is appropriate for the identified outcomes, though it is unclear whether the questionnaire was validated. the method is not rigorous with little explanation of methodology, ethics or validity of survey instrument. the authors do not explicitly state whether institutional review board approval was obtained nor if participant informed consent was provided. though the authors include all of the questionnaire items in the study, more clarity around the research methodology is needed including details about how the questionnaire was distributed, and how data was collected, stored, and analyzed. aside from age and academic library setting, the authors did not identify any other inclusion or exclusion criteria. it is unclear what geographic locations the population sample was recruited from, though it might be assumed that the study was conducted in the united states since the authors work in the us and cite data from the us. exact study replication without these full details may be difficult.   in the limitations, the authors are clear that data from this research are self-reported and likely to be inflated. though the study population size is large, the authors note that results are not generalizable to a broader academic library population since participation was self-selected and not randomized. the authors’ conclusions accurately reflect their analysis and they were able to address the objectives of the study. however, the authors do not suggest future areas of research.   this study provides a unique exploration of goal setting in academic library spaces, a significantly understudied area of research. however, the authors do not discuss how setting resolutions relates to professional goal-setting and how different contexts may impact goal-setting behavior and achievement. in professional contexts, achieving goals may have time constraints and significant implications such as promotions and annual review reports. personal goals may vary significantly and have fewer time constraints and repercussions. nonetheless, the results of this research could help inform library leaders and administrators on goal-setting best practices, but also encourage organizations to investigate other variables that might impact goal-setting behaviors such as workplace morale, alignment of worker responsibilities with the greater library mission and vision, and the impact of leadership support on worker outlooks and goal achievement.   references   anderson, a., & lo, l. (2019). new year’s resolutions, career outlook, and personality: an investigation of library employees’ goal setting behaviors. 33(2), 1-20.   dai, h., milkman, k. l., & riis, j. (2014). the fresh start effect: temporal landmarks motivate aspirational behavior. management science, 60(10), 2563–2582. https://doi.org/10.1287/mnsc.2014.1901   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154   locke, e. a., & latham, g. p. (1990). work motivation and satisfaction: light at the end of the tunnel. psychological science, 1(4), 240–246. https://doi.org/10.1111/j.1467-9280.1990.tb00207.x   oscarsson, m., carlbring, p., andersson, g., & rozental, a. (2020). a large-scale experiment on new year’s resolutions: approach-oriented goals are more successful than avoidance-oriented goals. plos one, 15(12), e0234097. https://doi.org/10.1371/journal.pone.0234097     evidence summary   nigerian medical libraries face challenges with high hopes for the future   a review of: popoola, b., uzoagba, n., & rabiu, n. (2020). “what’s happening over there?”: a study of the current state of services, challenges, and prospects in nigerian medical libraries. journal of the medical library association, 108(3), 398–407. https://doi.org/10.5195/jmla.2020.607   reviewed by: kimberly mackenzie research data and scholarly communications librarian university of massachusetts medical school worcester, massachusetts, united states of america email: kimberly.mackenzie@umassmed.edu    received: 2 dec. 2020                                                                accepted:  23 jan. 2021      2021 mackenzie. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29890       abstract   objective – this study examined the field of medical librarianship as it is currently practiced in nigeria.   design – mixed methods: electronic survey and in-person interview.   setting – the survey was advertised via an email list and a whatsapp discussion group, both based in nigeria. the interviews were requested directly by the authors.   subjects – librarians working in medical libraries in nigeria for the survey; library heads for the interviews.   methods – the survey was created in google forms and shared via the nigerian library association’s email discussion list and the whatsapp group for the medial library association of nigeria. question categories included personal and library demographics, library patronage/social media use, library services for users, and librarians’ training and challenges. most questions were closed-ended. survey data was analyzed in spss for response frequencies and percentages. the interviews were conducted in person. questions covered topics such as demographics, challenges, and prospects (for medical librarianship in nigeria). interview transcriptions underwent thematic content analysis.   main results – the majority of the 58 survey respondents (73%) reported seven or more years of medical library experience. there was no consensus on classifications schemes used throughout medical libraries in nigeria, with 43% using the us national library of medicine classification and 32% using the library of congress. social media use also varied, but the majority (approximately 45%) reported using social media less than monthly to promote their libraries or programming.   monographs were the main collection material reported by roughly 35% of respondents. journals followed at approximately 24% while only 10% reported electronic resources as the main collection material. the majority of respondents (53%) noted that their library did not offer specialized services. others (31%) reported “selective dissemination of information, current awareness services, or reference services” (p. 402) as specialized services; 7% reported literature searching. the majority of respondents (70-75%) rated their skill levels in evidence based medicine and systematic reviews as beginner/intermediate. half of respondents reported that their libraries had not held any training programs or seminars for library users in the six months prior.   interviews with library heads revealed that they all had high hopes for the future of medical libraries in nigeria but also noted many challenges. these included a lack of cooperation between libraries, a lack of interlibrary loan services, budget deficiencies, and insufficient access to the internet. this mirrored survey responses, 50% of which noted access to electronic information was a “significant barrier to improved services” (p. 402) along with a lack of training (53%) and low library usage (57%).   conclusion – medical libraries in nigeria face multiple challenges. budgetary constraints, a lack of library cooperation, and internet accessibility limit the availability of electronic collections. the authors suggest that library associations in nigeria focus on education and training opportunities for current and future medical librarians.   commentary   boynton and greenhalgh (2004) was consulted while evaluating the survey portion of this study and glynn (2006) was consulted while evaluating the interview portion. while both methods were appropriate for this research, the survey dissemination and choice of interview subjects could potentially have introduced bias into the results. as can be the case with any survey sent via email discussion lists or social media, the results are skewed towards the people who choose to answer. it is unclear from the reported results how respondents found the survey (either the email list or the medical library association of nigeria’ whatsapp group), though the authors report a 52% response rate based solely on the number of users in the whatsapp group. in terms of the interviews, it is not explained why the interviewees were chosen, other than that they were the library heads at nigeria’s first-generation universities. the authors note that four of the five first-generation universities are located in the southern region of nigeria, which is “educationally more developed than the northern region” (p. 400). to include viewpoints from multiple geographical areas, including those of less developed regions, heads of medical libraries from all over nigeria should have been contacted.   when evaluating survey questions, boynton and greenhalgh (2004, p. 1314) note that closed ended questions may lead to a lack of richness in responses, as users are only allowed to choose from the researcher’s choices. in the current survey, the closed-ended questions offered a partial view of the state of medical libraries in nigeria, but adding the option of a short explanation or further thought could have granted more insight. for example, respondents were asked to “rate users’ understanding of the importance of the library and what it stands for” (appendix a, question 9) on a scale of very low to very high. this is a very subjective question that would have benefitted from users having had the option to explain why they answered as they did. in some cases, closed-ended questions may have been leading for the respondents, such as the question “which of the following do you agree is a barrier to your service delivery as a health science librarian? (check all that you agree with)” (appendix. a, question 18). the respondent is not able to give their own thoughts on barriers to service. similarly, the interview question on challenges (appendix b, section b) included possible challenges to ask about if interviewees did not offer their own ideas. this could have been leading the interviewees to respond in a way they would not have otherwise (though it is not clear whether these prompts were ever used).   an interesting area for future study based on this work involves respondent demographics. demographic questions, such as type of library worked at, length of employment, and geographical location in nigeria, were included, but comparisons were not made between these areas. it also would have been interesting to know what role the respondents held at their libraries, as different levels of experience could make a difference in how the status of the field is viewed.   while the authors note that access to electronic resources and the internet are barriers to service, it is not clear whether this is true in all areas of the country or if service barriers differ with location. in terms of electronic resource access and since another major challenge mentioned was a lack of interlibrary loan services, it would be interesting to know what use libraries were making of open access journals, open educational resources, and other freely available resources. however, the fact that respondents also noted insufficient internet access underscores the difficulty in relying on electronic resources, open or otherwise. it is not clear what role budget insufficiencies play in the lack of internet access.   survey responses did offer several ways that medical libraries can improve in nigeria, some of which could be of use to libraries everywhere. professional development and continuing education are among the most important to consider. in nigeria it is needed to increase librarians’ skill sets, particularly in areas of evidence based medicine and systematic reviews, but continued skill building in staff is essential to the continued relevance of libraries in general. cooperation between libraries, such as the formation of consortiums, can be a safety net in times of financial strain. in the case of nigerian medical libraries, sharing resources can help to increase access for users. finally, the survey responses highlight the importance of outreach to users, through programming and social media in order to promote the library and library resources. this is a lesson to remember for all libraries, regardless of where they are in the world.   references   boynton, p. m., & greenhalgh, t. (2004). hands-on guide to questionnaire research: selecting, designing, and developing your questionnaire. bmj, 328(7451), 1312–1315. https://doi.org/10.1136/bmj.328.7451.1312   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387–399. https://doi.org/10.1108/07378830610692154 evidence summary   incoming undergraduate students struggle to accurately evaluate legitimacy of online news   a review of: evanson, c., & sponsel, j. (2019). from syndication to misinformation: how undergraduate students engage with and evaluate digital news. communications in information literacy, 13(2), 228-250. https://doi.org/10.15760/comminfolit.2019.13.2.6   reviewed by: sarah bartlett schroeder research & instruction librarian university of washington bothell/cascadia college campus library bothell, washington, united states of america email: sarahkb6@uw.edu   received: 21 sept. 2020                                                             accepted:  5 jan. 2021      2021 schroeder. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29854     abstract   objective – to determine how new undergraduate students access, share, and evaluate the credibility of digital news.   design – asynchronous online survey and activity.   setting – a small private, liberal arts college in the southeastern united states of america.   subjects – participants included 511 incoming first-year college students.   methods – using the moodle learning management system, incoming first-year students completed a mandatory questionnaire that included multiple choice, likert scale, open-ended, and true/false questions related to news consumption. two questions asked students to identify which news sources and social networking sites they have used recently, and the next two questions asked students to define fake news and rate the degree to which fake news impacts them personally and the degree to which it impacts society. the end of the survey presented students with screenshots of three news stories and asked them to reflect on how they would evaluate the claim in the story, their confidence level in the claim, and whether or not they would share this news item on social media. the three items chosen represent certain situations that commonly cause confusion for news consumers: (a) a heading that does not match the text of the article, (b) a syndicated news story, and (c) an impostor url and fake news story. researchers coded the student responses using both preset and emergent codes.   main results – eighty-two percent of students reported using at least one social media site to access political news in the previous seven days. students reported believing that fake news is a worrying trend for society, with 86% labelling it either a “moderate” or “extreme” barrier to society’s ability to recognize accurate information. however, they expressed less concern about their own ability to navigate an information environment in which fake news is prevalent, with 51% agreeing that it has only somewhat of an effect on their own ability to effectively navigate digital information. of the three news items presented to them, students expressed the least confidence (an average of 1.55/4) and least interest in sharing (12%) the first news item, in which the heading does not match the text. however, only 14% of respondents noted this mismatch. in evaluations of the second item, an ap news item on the breitbart website, 35% of students noted the website on which the article was found, but fewer noted that the original source is the associated press. student responses to the third article, a fake news item from a website masquerading as an nbc website, show that 37% of students believed the source to come from a legitimate nbc source. only 7% of students recognized the unusual url, and 24% of respondents indicated that they might share this news item on social media.   conclusion – the study finds that impostor urls and syndicated news items might confuse students into misevaluating the information before them, and that librarians and other instructors should raise awareness of these tactics.   commentary   project information literacy took on the subject of fake news in 2018. in a large scale survey across 11 u.s. colleges, they found that social networking sites were a prominent source of news for college students, and that 45% of respondents lacked confidence in distinguishing fake news from real news (head et al., 2019). some smaller scale studies have sought to determine how effective undergraduate students are at distinguishing fake news from real news. for example, in one study of 63 students, researchers found that students were able to correctly identify whether a news item was real or fake approximately 62% of the time (leeder, 2019). researchers have also looked into how and where students get their news (zakharov et al., 2019).   this review analyzes the design and data analysis of this study using a checklist evaluation from the center for evidence based management. the authors present a broad research question, seeking to explore how incoming college students engage with news in an online environment. the questions they asked students are appropriate for this line of inquiry, but the topic remains too large to cover adequately with a single study. nearly the entire incoming class completed the mandatory questionnaire, which provides an excellent opportunity to study the information behaviours of their student body, though they do not share details of their student demographics. because students’ backgrounds may impact way they engage with news, this information would be useful for practitioners in other geographic areas.   the article is sparse on details about the authors’ coding processes. they provide a list of codes in the appendix and state that some codes were preset and others were emergent codes that came about during their coding process. it is not known whether the authors coded individually and later pooled their work or coded together. more information on the coding process would help others replicate such a study more easily. those seeking to conduct a similar study could also benefit from reflections on the process of choosing sample articles. the authors do not explain whether they sought potentially controversial articles or not, but students may have stronger reactions to hot-button topics than more benign ones, which could impact their evaluations.   the authors note several times that one significant weakness of their study was an inability to share actual links with students rather than screenshots of news items. this prevented students from engaging with the news item in its entirety (i.e., they could not follow links or fact-check information in other sources), meaning they were evaluating news stories in a manner that is possibly inconsistent with how they would evaluate such news items if they encountered them on their own. observations in a more natural online news environment may yield different results.   the data here provides evidence to support adding information about news syndication and imposter urls in news evaluation instruction, as these present points of confusion for some students. however, as the authors note, online news systems evolve quickly. it may be beneficial to repeat this study over multiple years to track how student abilities change over time.   references   center for evidence-based management. (n.d). critical appraisal of a survey. retrieved from https://www.cebma.org/wp-content/uploads/critical-appraisal-questions-for-a-survey.pdf   head, a., defrain, e., fister, b., & macmillan, m. (2019). across the great divide: how today's college students engage with news. first monday, 24(8). https://doi.org/10.5210/fm.v24i8.10166   leeder, c. (2019). how college students evaluate and share “fake news” stories. library & information science research, 41(3), 100967. https://doi.org/10.1016/j.lisr.2019.100967   zakharov, w., li, h., & fosmire, m. (2019). undergraduates’ news consumption and perceptions of fake news in science. portal: libraries and the academy, 19(4), 653-665. https://doi.org/10.1353/pla.2019.0040       evidence based library and information practice evidence based library and information practice 2010, 5.2  evidence based library and information practice       using evidence in practice    mining e‐mail to improve information literacy instruction    margy macmillan  instruction librarian/associate professor  mount royal university   calgary, alberta, canada  email: mmacmillan@mtroyal.ca    received: 29 mar. 2010          accepted: 06 apr. 2010       2010 macmillian. this is an open access article distributed under the terms of the creative commons‐ attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐ sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original  work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is  redistributed under the same or similar license to this one.    setting     mount royal university (mru) is an  undergraduate institution with an enrollment  of 8,000 fte, located in calgary, alberta,  canada. the library provides a range of  course and program‐integrated instruction in  information literacy (il) conducted by liaison  librarians in collaboration with faculty in the  disciplines. the faculty of communication  studies currently serves students in three  degree programs, the bachelor of  communication in journalism, public  relations, or information design, and a  diploma program in broadcasting.    the liaison librarian works with students in all  communication programs, both in the  development and provision of il classes, and  in the provision of one‐to‐one assistance  through appointments, e‐mail, and regular  reference hours at both the main library  information desk and in the centre for  communications learning space. there are il  classes in most first‐year courses, and  increasingly specialized sessions in later years  that focus on particular resources, course  content and higher‐level concepts in  information literacy. the librarian is known by  the students to be somewhat compulsive  about answering e‐mails, and often responds  during evenings and weekends.   problem     while there was no defined problem with the  services being provided, the librarian was  curious about the amount and type of e‐mail  questions students were asking. patterns  within the types of questions being asked, the  programs, levels, courses or assignments that  were generating the questions, or the  individuals asking the questions might reveal  areas where il might be improved to better  meet student needs. the librarian was also  interested in the time of day students asked  questions as an indication of when students  were working on their assignments.      103 mailto:mmacmillan@mtroyal.ca evidence based library and information practice 2010, 5.2  evidence     in consultation with members of mru’s  human research ethics board, the author  determined that unless consent was obtained  from those students who e‐mailed the  librarian, no data could be published. as  seeking such consent might unduly constrain  students from contacting the librarian, and  might therefore be detrimental to their studies,  consent was not sought. therefore, while the  process of the study can be described as an  encouragement to other librarians to mine this  rich supply of data, i cannot share the results.    beginning in august of 2008, the librarian  collected e‐mail from students and placed it in  a folder. excluded were e‐mails that were  required, such as assignments that were e‐ mailed to the librarian, and, for the first part of  the year, communications through facebook.  these last messages were routinely deleted by  the librarian, until it became evident how  many of them were reference questions. in  july 2009, the librarian extracted the file of  student e‐mails into a spreadsheet and coded  it by type, such as search strategy help,  citation questions, requests for appointments,  and help with statistics. she also recorded data  about what might have triggered the  questions, including the program, year of  study, and, where determinable, course and  assignment. this was possible due to the  librarian’s knowledge of the curriculum and of  the students. information on the time and date  that questions were sent was also part of the  data. coding a year’s worth of questions  (n=543) took approximately eight hours.  the results regarding the type and timing of  questions would not have surprised any  academic librarian. in fact, results were similar  to larger scale studies carried out to evaluate  e‐mail and chat reference services (lee, 2004;  leykam & perkins, 2008; rourke & lupien,  2007). there were interesting patterns in the  levels, courses and assignments that generated  questions. for example, there was a significant  lack of questions from students in the second  year of any communications program. this  could have been read a number of ways: the  students had learned everything they needed  to in the first year (optimistic but unlikely); the  students had no significant research  assignments (on investigation, not the case); or  the students, who did not meet with the  librarian in as many courses in the second  year, forgot about this resource (sadly, the  most probable cause). in investigating the  assignments required in the second year, the  librarian ascertained that these had not  prompted in‐person consultations either.     assessing students’ needs by examining only  one form of interaction is problematic. there  may be students who are not comfortable  using e‐mail, some preferring to phone or  meet face‐to‐face. some may be getting all the  help they need at the information desk, where  only very broad statistics are kept, or from  their peers and instructors. a study like this  can only be used to measure one aspect of  interaction with users, and should be used in  concert with other assessments to see the  whole picture of who is or is not requesting  help, and what kind of help they are  requesting.    quite apart from evidence about the kinds of  questions students were asking, the study  provided data about the librarian’s work  activities, including how significant an aspect  of teaching the e‐mail communication  represented.     implementation     in the 2009‐10 academic year, a number of  changes were made to il delivery based on  the evidence of the e‐mails. the content of  some sessions was updated and clarified in  response to common areas of difficulty.     in response to the problem of ”missing”  second‐year students, the librarian asked for  15 minutes in the main second‐year class of  the degree disciplines, information design,  journalism, and public relations, at the  beginning of the fall semester of 2009. she  used this time to review key materials  students might use for assignments, to  highlight new resources and changes to the  104 evidence based library and information practice 2010, 5.2  105 library website, and mostly to remind the  students that they could ask for assistance. in  each of these “promotional” talks, the librarian  spoke directly of the evidence from e‐mails as  the reason for the session. there was generally  a brief period for questions and answers.    information from the study also contributed to  a better understanding and a better balancing  of the librarian’s workload. in part because of  data from the study, the librarian reduced the  number of shifts at the main reference desk  and added regular weekly reference hours in  the centre for communication studies, where  many of the students do their work.  outcome     while the full impact of implementing the  changes will only be evident after reviewing  the current year’s student e‐mails, there has  been some noticeable effect. students  apparently found the promotional talks in the  second‐year courses helpful, and instructors  certainly did. as a result of discussions with  faculty about the ”missing” second‐year  students, several faculty members have  booked longer sessions on particular aspects  of information resources, or specific skills such  as using statistics, required by assignments in  these courses.    reflection     the study has been very successful as a means  of satisfying the librarian’s curiosity about an  aspect of her work. it was relatively easy and  efficient to carry out, requiring comparatively  little time to develop rich results. these results  provide evidence where there had been only  suspicion and anecdote, evidence that  provided clear directions for change. the  broad outcomes of those changes have yet to  be discovered, but the process of examining  work on the basis of communications from  students has been useful in and of itself. so  useful, in fact, that the librarian has repeated  and expanded the study for the 2009‐2010  academic year, to include files of e‐mails from  faculty, staff, and other groups, in addition to  students.   further examination of the data might reveal  demographic patterns or individual usage  patterns that might also be enlightening. it  would be interesting, for example, to find out  if students asked the same type of question  more than once, or if questions from  individual students showed development of  skill and grew more complex over time. in the  current study, the librarian is also more careful  about including reference questions from  facebook, and it may be that over time, those  form a higher proportion of the  communication. deeper discourse or content  analysis might reveal other aspects of  communication, especially if the study is  replicated over a longer period of time.    the most frustrating aspect of the study is the  difficulty in presenting results. just asking for  prior consent of everyone who e‐mailed the  librarian would likely reduce the e‐mail, and  potentially disadvantage those who would  otherwise seek help. if consent was requested,  the populations who gave or did not give it  might be different enough to further confound  the data, and certainly it would complicate the  process to the point where the usefulness of  the study would be compromised. however,  while in terms of dissemination this kind of  study has limited worth, it has been very  productive in terms of informing the work of  the librarian at a local level. i would encourage  others to make use of their e‐mail as evidence  for refining practice.  references    lee, i. j. (2004). do virtual reference librarians  dream of digital reference questions?:  a qualitative and quantitative analysis  of email and chat reference. australian  academic & research libraries, 35(2), 95‐ 110.    leykam, a., & perkins, c. (2008). is this the  right tool for our library? a look at e‐ mail virtual reference use patterns. the  reference librarian, 48(1), 1‐17.  doi:10.1300/j120v48n99_01    evidence based library and information practice 2010, 5.2  rourke, l., & lupien, p. (2007). out of the  question!...how we are using our  studentsʹ virtual reference questions  to add a personal touch to a virtual  world. evidence based library and  information practice, 2(2). retrieved 2  apr. 2010, from  http://ejournals.library.ualberta.ca/ind ex.php/eblip/article/view/236/421  106 microsoft word es1_879_final evidence based library and information practice 2008, 3:1  53 evidence based library and information practice       evidence summary    further study is needed to define and measure the use of reflective practice in library  and information science    a review of:  grant, maria j. “the role of reflection in the library and information sector: a systematic  review.” health information and libraries journal 24.3 (2007): 155‐66.      reviewed by:  carol perryman  trln doctoral fellow, school of information & library science, university of north carolina at  chapel hill  chapel hill, north carolina, united states of america  e‐mail: cp1757@gmail.com          received: 18 november 2007        accepted: 29 december 2007      © 2008 perryman. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    abstract    objectives ‐ to identify and review the  literature of reflective practice in library and  information science (lis) in order to  understand its role, particularly with regard  to health libraries.    design ‐ systematic review    setting ‐ lis english‐language articles  published between 1969 and 2006, and  indexed in the library and information science  abstracts (lisa) database.    subjects ‐ 929 citations retrieved from the  lisa database.    methods ‐ the author conducted free text  searches in the lisa database for the terms  ‘reflective,’ ‘reflection*,’ or ‘reflexion*.’ an  initial search series was conducted in 2004 in  order to retrieve items published between  1969‐2003, then in 2007 for articles published  between 2004‐2006. in all, 929 article  citations were retrieved. exclusion criteria  included those articles addressing the  facilitation of reflective practices in others,  as well as non‐english language materials,  and those predating 1969. after review, 55  evidence based library and information practice 2008, 3:1  54 articles met the author’s relevance standard.  citation tracking then added 10 articles to  the total. from this dataset, full‐text articles  were obtained where possible, if determined  on initial scrutiny to be deserving of further  examination. thirteen articles (.013%) were  ultimately selected for analysis. these  articles were categorized as analytical or  non‐analytical, with respect to perspective  (individual or organizational), and recency  of events (retrospective or recent). in  addition, a determination was made about  whether the articles’ foci were reflection  occurring on (in retrospect to) or in (during)  practice.    main results ‐ of thirteen articles, five were  found to be non‐analytical, with the other  six being analytical. three of the non‐ analytical items were the reflections of an  individual, while the remaining two offered  an organizational perspective. the non‐ analytical accounts were found to be  descriptive accounts by an individual,  mostly retrospective and offering no  consideration of implications for lis  practice. analytical reflective accounts  attempt to systematically appraise events  from the recent past, and draw conclusions  in order to improve future actions.     conclusion – a gradual increase in the use  of analytical reflective practice is  demonstrated over the period from 1969‐ 2006, although insufficient examples of the  practice were found in the published  literature. reflective practice is likely to be  beneficial to lis practitioners, especially  when time is spent in considering the  implications of lessons learned from  practice.    commentary:  this article was evaluated with the use of  greenhalgh’s checklist for appraising  systematic reviews, and one provided by  mcmaster university for the appraisal of  qualitative studies. several checklists were  used because this reviewer could find none  that were sufficiently applicable for an  evaluation of a systematic review of this  nature.    to begin with, the author has claimed this  work is a systematic review, (grant), and  references the text evidence‐based practice for  information professionals: a handbook (booth  and brice) in the process of performing  analysis. yet there has been no effort made  to perform what these same authors define  as the process of systematic review:    a systematic review tries to  answer a clear question by  finding and describing all  published and, if possible,  unpublished work on a  topic […] uses explicit  methods to perform a  thorough literature search  and critical appraisal […]  and uses appropriate  statistical techniques to  combine these valid studies.  (booth and brice 11)     the researcher in lis attempting to measure  qualitative phenomena in a systematic  manner immediately encounters a serious  barrier consisting of the use of  nonstandardized vocabulary. if the process  of reflection is not, as the author claims, well  established in lis, the use of the terms  ‘reflective,’ ‘reflection*,’ or ‘reflexion*’ used  to retrieve literature will not necessarily  provide any comprehensive set of data. in  establishing the need for an inquiry about  the use of reflective practice, grant cites the  words of booth and brice in a chapter  entitled “evaluating your performance”  (booth and brice 127‐137) which clearly  conflates the concept of reflection with that of  performance evaluation. despite grant’s  provision of the definition of reflection from  the oxford english dictionary, and her  mention of a definition provided for the  evidence based library and information practice 2008, 3:1  55 term in the literatures of nursing and  midwifery, no attempt is made to define  reflection for lis. in the absence of a  working definition, the differentiation of  reflective practice from the practice of  evaluative analysis has not taken place.  because of this, a search for literature is  problematic, and becomes an exercise in  searching for term occurrences which are  likely to be only part of the actual practice of  reflection in lis.    furthermore, even if lis publications  adhered to more standardized terminology  (as is the case in nursing, one of the  disciplines cited as preceding lis in the use  of reflective practice), the process of  systematic review demands examination of  resources beyond those indexed in only one  bibliographic database. grant explains her  lack of access to lis bibliographic databases  other than lisa, and projects the need to  enlarge an inquiry for evidence of reflective  practice, but without a broader effort,  including books, databases, hand‐searching,  and other resources, her work cannot  properly be called a systematic review.     searching articles indexed in the lisa  database alone is not sufficiently  comprehensive to ensure thorough  searching has been done when the intent is  to perform a systematic review, even if  descriptor terms are available (see eldredge  36‐48). for example, a search of the library  literature and information science database  with (reflective <in> kw or reflection* <in>  kw or reflexion* <in> kw and between  1969 and 2006) yielded 1802 items,  compared to the 929 retrieved from lisa.  this reviewer notes that articles from such  publications as publishers weekly are  included in the total retrieved from the  library literature and information science  database, and wonders what other criteria  were used to determine relevance in the  article under review, since that detail is not  provided: “[…] efforts were made to obtain  the full text of those that appeared to be  relevant” (grant 157).    the resulting literature, a total of 13 articles,  is analyzed and categorized, after which the  author draws conclusions in response to her  research question. assessment of the  “methodologic quality” (another criteria of a  well‐performed systematic review) has been  done only to the level of identifying the 13  articles as “analytic” or “non‐analytic”, as  “recent” or “distant past,” and as  “reflections in practice” or “reflections on  practice” (grant 156).      due to methodological problems, the study  cannot be replicated or generalized,  although unquestionably, a study of the use  of reflective practice in lis would be  beneficial to the profession for the reasons  mentioned by the author. the study might  have been better approached as an initial  review of the literature, as a way to assess  the status of reflection in (or on) practice.  this work would have been improved by an  attempt to discuss the problems  encountered in trying to perform the study,  and a clearer discussion of the  exclusion/inclusion criteria. the author  contributes to the sparse literature about the  use of reflective practice in lis with this  effort, providing readers with an overview  of its application in nursing and education,  and pointing the way to future research.     works cited    booth, andrew, and anne brice.  “appraising the evidence.”  evidence‐based practice for  information professionals: a  handbook. eds. andrew booth and  anne brice. london: facet  publishing, 2004. 114‐118.     eldredge, jonathon. “how good is the  evidence base?” evidence‐based  practice for information  evidence based library and information practice 2008, 3:1  56 professionals: a handbook. eds.  andrew booth and anne brice.   london: facet publishing, 2004. 36‐ 48.     grant, maria j. “the role of reflection in the  library and information sector: a  systematic review.” health  information and libraries journal  24.3 (2007): 155‐66.    greenhalgh, trisha. how to read a paper.  2nd ed. london: bmj books, 2001.  netlibrary.[oclc]. [unc  university libraries, chapel hill,  n.c.]. 14 nov. 2007  <http://www.netlibrary.com>.    mcmaster university. health sciences  library. worksheet for critical  appraisal of qualitative research.  n.d. 14 nov. 2007  <http://hsl.mcmaster.ca/documents/ education/nursing/  qualitative.doc>  evidence based library and information practice evidence based library and information practice 2010, 5.4 83 evidence based library and information practice evidence summary free access does not necessarily encourage practitioners to use online evidence based information tools a review of: buchan, h., lourey, e., d’este, c., & sanson-fisher, r. (2009). effectiveness of strategies to encourage general practitioners to accept an offer of free access to online evidence-based information: a randomised controlled trial. implementation science, 4, article 68. reviewed by: heather ganshorn librarian health information network calgary university of calgary, calgary, alberta, canada email: heather.ganshorn@ucalgary.ca received: 27 aug. 2010 accepted: 25 oct. 2010 2010 ganshorn. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objectives – to determine which strategies were most effective for encouraging general practitioners (gps) to sign up for free access to an online evidence based information resource; and to determine whether those who accepted the offer differed in their sociodemographic characteristics from those who did not. design – descriptive marketing research study. setting – australia’s public healthcare system. subjects – 14,000 general practitioners (gps) from all regions of australia. methods – subjects were randomly selected by medicare australia from its list of gps that bill it for services. medicare australia had 18,262 doctors it deemed eligible; 14,000 of these were selected for a stratified random sample. subjects were randomized to one of 7 groups of 2,000 each. each group received a different letter offering two years of free access to bmj clinical evidence, an evidence based online information tool. randomization was done electronically, and the seven groups were stratified by age group, gender, and location. the interventions given to each group differed as follows: • group 1: received a letter offering 2 years of free access, with no further demands on the recipient. mailto:heather.ganshorn@ucalgary.ca� evidence based library and information practice 2010, 5.4 84 • group 2: received a letter offering 2 years of free access, but on the condition that they complete an initial questionnaire and another one at 12 months, as well as allowing the publisher to provide de-personalized usage data to the researchers. • group 3: same as group 2, but with the additional offer of an online tutorial to assist them with using the resource. • group 4: same as group 2, but with an additional pamphlet with positive testimonials about the resource from australian medical opinion leaders. • group 5: same as group 2, but with an additional offer of professional development credits towards their required annual totals. • group 6: same as group 2, but with an additional offer to be entered to win a prize of $500 towards registration at a conference of the winner’s choice. • group 7: a combination of the above interventions. the group received the opinion leaders’ pamphlet, the online tutorial, and eligibility for professional development points. the online survey and usage data from groups 2 through 7 was to be analyzed as part of a companion study, and is not reported in this article. to protect the privacy of individual subjects, medicare australia mailed out the offers and provided the authors with anonymized data, in table format, on response status by intervention group and by the following sociodemographic variables: age, gender, geographic remoteness as determined by the accessibility/remoteness index of australia (aria), country of graduation, and years since graduation. baseline characteristics were compared between the intervention groups, and then response rates were also compared between intervention groups and between the above-mentioned variables to see whether any of these variables affected the likelihood of practitioners being interested in an online evidence based tool. all comparisons were done using a chi-square test. main results – overall, 2,105 subjects returned their acceptance forms, out of the total sample of 14,000 (15%). the true acceptance rate was 12.5%, however, when adjusted for the number of subjects in groups 2 through 7 who went on to complete the online questionnaire. there was a statistically significant difference in response rates between the seven groups, with the greatest acceptance rate (27%) coming from group 1 (who received only the letter of offer, with no experimental demands). the other groups averaged a response rate of 10% collectively, with the lowest rates (8.0% and 8.5% respectively) from group 5 (offer of professional development points) and group 7 (combination of interventions). the large sample size offered adequate power to detect differences in characteristics between responders and non-responders. the study found that responders were more likely to be younger, male, recent graduates, and practising in less remote locations. among responders, there were no statistically significant differences in most of these characteristics among the seven groups, with the exception of time since graduation, which varied somewhat. conclusion – the authors conclude that funding of access to free online resources for large groups of practitioners may not be cost-effective if calculations of cost are based on total eligible populations rather than on the number of practitioners who may be interested. they also conclude that the low response rates generated by their offer indicate a need to find ways to increase gps’ interest in using online evidence based tools and in accessing best practice evidence. further research into how to achieve behaviour change among practitioners may be needed. commentary evidence based library and information practice 2010, 5.4 85 this is a strong study that addresses the information behaviours of an important group of healthcare practitioners: gps. while the population was australian, it seems likely that the results could be generalized to the rest of the english-speaking world. the participation of medicare australia in the project ensured that the researchers were able to draw their sample from a very accurate pool of the total population of gps in australia. the large sample size of 14,000 meant that although there were seven intervention groups, each group contained 2,000 subjects, thus forming a large enough sample of its own to provide precise estimates. although response rates were quite low for some of the groups, the goal was simply to discover whether subjects would accept an offer of free access to an evidence based tool, so it is probably safe to conclude that non-respondents were not interested. the authors acknowledge some limitations to their study. there is a small possibility that since the letters of offer came from medicare australia, the organization responsible for gps’ reimbursement, doctors in the groups with experimental demands may have been reluctant to accept the offer because of the possibility that their individual usage data might be shared with this organization. response rates may also have been low because of a dislike of responding to unsolicited mail. (a letter was the only form of communication by which the offer was made.) more importantly, it is very likely that there was some contamination between groups, as doctors within the same practice may have received different letters of offer. there are a few other difficulties with this study, the most important of which is the low response rate, given that the highest response rate among groups was 27%, and the other groups were much lower. this could pose a problem for the groups that are participating in surveys or usage monitoring, in that the results would not be very representative. it is not clear whether lower response rates in groups two through seven were due to doctors’ unwillingness to complete two surveys, reluctance to have their online activity monitored, or a combination of these things, as there were no groups to which each of these conditions was given separately. given the large difference in response rates between group 1 (27%) and the other groups (10%), this is an important question. the authors do not indicate how they phrased their request to monitor doctors’ activity, so it is difficult to determine whether the subjects could be reassured that their data would not be linked directly to them at the individual level. it is possible that the incentives offered simply did not offset the costs in terms of physician time, learning curve, and effort. for example, $500 to attend a conference would not come close to covering the full cost of attendance. the authors also mention that the offer of professional development points was made during a time at which most practitioners had likely accumulated most or all of the points they needed for the year, a factor which undoubtedly affected acceptance rates among subjects in the group that received this offer. because there was no follow-up with group 1, it is unclear how many of the doctors in this group who responded to the free offer would actually end up using the resource. however, this could be remedied with the development of some sort of voluntary survey during the two-year free access period. it would have been interesting to know about subjects’ access to the internet, experience using it, and access to other online databases. the authors might even have been able to gather some useful data from those who didn’t accept the offer, had they supplied a postage-paid envelope and asked those who weren’t interested in the offer to indicate whether this was due to lack of internet connectivity, inexperience/discomfort using the internet in practice, or access to other preferred resources. it is possible that some of the recipients of this offer may already have had access to clinical evidence (for example, through status as faculty in a medical school) or to a competitor product such as uptodate. evidence based library and information practice 2010, 5.4 86 such access may have reduced the incentive to sign up for this study’s offer. the authors indicate that they used a chisquare test for comparisons, but they don’t report the chi-square test results in the article. what the study clearly shows is that merely offering a resource free of charge does not guarantee that it will be adopted, and that certain time-honoured strategies for increasing uptake, such as incentives or opinion-leader endorsements, may not be very effective. given that younger practitioners were more likely to accept the offer, one can hope that acceptance of such resources will increase in the future, but it also seems likely that supports are needed to encourage and assist older practitioners in the adoption of online evidence based tools. a very important take-away point is the issue of cost. it may be more cost-effective for healthcare organizations to purchase individual licenses for interested practitioners than to buy a resource for the entire population, given that uptake is likely to be slow, at least initially. it is possible that the enthusiastic respondents to such an offer could form an “early adopter” group that would eventually promote the resource to their colleagues, thereby increasing uptake. that, however, is a subject for a different study. / evidence based library and information practice evidence summary evidence summary   engagement with search-based advertising on search engine results pages varies based on the user’s prior knowledge and screen size   a review of: schultheiß, s., & lewandowski, d. (2021). how users’ knowledge of advertisements influences their viewing and selection behavior in search engines. journal of the association for information science and technology, 72(3), 285–301. https://doi.org/10.1002/asi.24410   reviewed by: scott goldstein coordinator, web services & library technology mcgill university library montréal, québec, canada email: scott.goldstein@mcgill.ca   received: 1 june 2021                                                                    accepted:  26 july 2021      2021 goldstein. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29989     abstract   objective – to examine how users’ understanding of ads on search engine results pages (serps) influences their viewing and selection behaviour on computers and smartphones.   design – mixed methods approach consisting of pre-study interview, eye-tracking experiment, and post-study questionnaire.   setting – usability lab at a university in germany.   subjects – 50 students enrolled at the hamburg university of applied sciences and 50 non-students recruited in hamburg.   methods – after giving informed consent and receiving payment, participants provided information on demographics as well as how they use search engines as part of a pre-study interview. for the eye-tracking experiment, each participant completed 10 tasks each on a desktop computer and smartphone. both the device condition order and task order were randomized. tasks were broken down into five informational tasks (e.g., how do i build a desktop computer?), three transactional tasks (e.g., how would i go about buying a refrigerator?), and two navigational tasks (e.g., i need to go to the apple website). the software displayed clickable screenshots of serps, and all clicks were recorded. imotions eye-tracking software recorded eye fixations on areas of the page featuring organic search results and paid ads. a post-experiment questionnaire asked participants about google’s business model and probed them about the extent to which they were able to differentiate between organic results and ads. answers to the questionnaire were weighted and normalized to form a 0–100 scale.   main results – the first set of research hypotheses examining the correlation between participants’ knowledge of ads and viewing and clicking behaviour was partially confirmed. there was no significant correlation between participants’ questionnaire score and visual fixations on ads, but there was a significant negative correlation between questionnaire score and the number of clicks on ads. users with questionnaire scores in the bottom quartile paid significantly less attention to organic results than those in the top quartile, but users in the top quartile still fixated on ads and did so comparably to users in the bottom quartile. the second set of research hypotheses examining the relationship between viewing and clicking behaviour and device (desktop versus mobile) was also partially confirmed. users on a smartphone had significantly higher fixation rates on ads than users on a desktop computer, although click rates on ads did not differ significantly between the two conditions.   conclusion – knowledge about ads on serps influences selection behaviour. users with a low level of knowledge on search advertising are more likely to click on ads than those with a high level of knowledge. users on smartphones are also more likely to pay visual attention to ads, probably because the smaller screen size narrows content “above the fold.”   commentary   this paper is part of the literature on search-based advertising, focusing on the paradigmatic example of google, which has a europe-wide market share of over 90%. advertisements generated in response to user queries and appearing on search engine results pages (serps) account for the majority of google’s revenue. however, according to one of the coauthor’s earlier articles, 40% of german internet users were unaware of this or listed other incorrect sources of revenue (lewandowski, 2017). given that previous studies have shown users trust and rely on google search, especially organic results high up on the first result page, it is important to know how ads contribute to or interfere with users’ information search behaviour. the authors’ eye-tracking experiment relies on data from 100 individuals, only half of them students, which is a departure from many other eye-tracking studies.   perryman’s critical appraisal tool was used to appraise this study (perryman & rathbun-grubb, 2014). this research was extremely well planned from start to finish. the authors conducted a thorough literature review and generated two specific research questions. their recruitment strategy is commendable for its large size and diversity even if it is not a perfectly representative probability sample. consistent with similar types of studies, the authors minimized bias with condition and question order randomization and withheld certain details of the study to reduce demand characteristics. considering that 26 of the 100 participants were from the department of information, the authors ought to have looked at whether this large subgroup scored significantly higher on the questionnaire. nonparametric statistical tests were used. one minor error is the use of “p = .000” (p. 297), which is, strictly speaking, impossible; presumably they meant to write “p < .001.” the authors included a list of limitations at the end of their study. finally, all of the study’s data and materials are openly available on zenodo.   there are a couple of implications of this research for libraries. the first is mentioned by the authors themselves, who write that with respect to information literacy, “we deem it imperative to help users understand that search engines do not necessarily act in their best interest, but search engine providers have interests of their own” (p. 299). it is not clear how much of information literacy currently is devoted to google’s revenue model, but perhaps more should be taught on this to regular users. secondly, this research has the potential to influence how libraries customize their discovery systems, especially next-generation systems that permit tweaking the display of serps. for instance, some libraries might be considering adding widgets and other query-based displays that, while not actually advertisements, might be viewed as such by users. if the benefits of these query-based displays outweigh the cognitive load costs to users, they should be clearly labelled and visually differentiated from the organic results.   references   lewandowski, d. (2017). users’ understanding of search engine advertisements. journal of information science theory and practice, 5(4), 6–25. https://doi.org/10.1633/jistap.2017.5.4.1   perryman, c., & rathbun-grubb, s. (2014). the cat: a generic critical appraisal tool. in jotform – form builder. retrieved from https://www.jotform.us/cp1757/thecat   research article   evaluation of integrated library system (ils) use in university libraries in nigeria: an empirical study of adoption, performance, achievements, and shortcomings   saturday u. omeluzor, ph.d. systems librarian library department federal university of petroleum resources effurun delta state, nigeria. email: omeluzor.saturday@fupre.edu.ng   received: 8 july 2019                                                                 accepted: 2 oct. 2020      2020 omeluzor. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29604     abstract   objective the aim of this study was to evaluate integrated library system (ils) use in university libraries in nigeria in terms of their adoption, performance, achievements, and shortcomings and to propose a rigorous model for ongoing evaluation based on use of candidate variables (cvs) derived from the approach used by hamilton and chervany (1981) and from evaluation criteria suggested by farajpahlou (1999, 2002).   methods the study adopted a descriptive survey design. nigeria is made up of six geo-political zones including: north-east (ne), north-west (nw), north-central (nc), south-south (ss), south-east (se), and south-west (sw). the population for this study comprised systems/it and e-librarians in the university libraries from all six of the geo-political zones of nigeria. because of the large number of universities in each of the zones in nigeria, a convenience sampling method was used to select six universities representing federal, state, and private institutions from each of the six geo-political zones of nigeria. a purposive sampling method was used to select the systems/it and e-librarians who were directly in charge of ils in their various libraries. therefore, the sample for this study was made up of 36 systems/it and e-librarians from the 36 selected universities in nigeria. the instrument used to elicit responses from the respondents was an online questionnaire and was distributed through the respondents’ email boxes and whatsapp. the questionnaire administration received a 100% response rate.   results findings revealed that university libraries in nigeria have made remarkable progress in the adoption and use of ils for library services. the findings also showed that much has been achieved in the use of ils in library services. evidence in the study indicated that the performance of the ils adopted in the selected university libraries in the area of data entry and currency, accuracy, reliability, completeness, flexibility, ease of use, and timeliness was encouraging.   conclusions adoption and use of ils in libraries is changing the way libraries deliver services to their patrons. traditional methods of service delivery are different from the expectations of the 21st century library patrons. the transformation seen in the university libraries in nigeria using ils was tremendous and is changing the narratives of the past. however, several shortcomings still exist in the adoption and use of ils in university libraries in nigeria. overcoming some of the limitations would require a conscious effort and decisiveness to ensure that librarians and library patrons enjoy the best services that ils can offer. ils developers should consider the dynamic needs of libraries and their patrons and incorporate specific candidate variables (cvs) in their ils designs to enhance the quality of the services being offered to the library patrons.     introduction   “library and information science occupies a vantage position in the educational sector and plays a strategic role in national growth and development” (shekarau, 2014). university libraries today are adopting information and communication technologies (icts) to deliver information to their patrons. icts are playing a pivotal role in the way and manner in which information is being handled in the library. before the use of ict tools in libraries, traditional methods were employed to deliver most library services. traditional library processes have been judged as unable to respond quickly enough in a technologically driven environment (ayiah & kumah, 2011). with a steady growth in library collections for various programs that are offered in the university, and the decentralization of library activities, it is essential to use an integrated library system that responds quickly to the needs of librarians and library patrons (omeluzor, adara, madukoma, bamidele & umahi, 2012). an ils has been defined as “a series of interconnected operations that streamline input and retrieval of information for both information professionals and researchers” (lucidea.com, n.d.).  since the concept of an ils was first introduced by harder in 1936, it has been developed and modified to suit different ideas and purposes in different sectors of the economy, for example, banking, marketing, and aviation among others. in the library, the term ils has been used interchangeably for both mechanization and automation (riaz, 1992). singh (2013) defined library automation as the computerization of library records and functions, using computer hardware and software for tasks that may require a lot of paperwork and staff time. singh (2013) noted that ils is the use of computers, and associated technology, to do exactly what has been traditionally done in libraries with the justification of reducing cost or increasing performance. ils enables adequate monitoring, controlling, service delivery, access to bibliographic records, collaboration among libraries, and enhanced access to information materials irrespective of patrons’ geographical locations (omeluzor & oyovwe-tinuoye, 2016).   university libraries in nigeria and other countries have increasingly used new tools such as ils and methods for delivering information services to their patrons since the beginning of the 21st century (sharma, 2009; oladokun & kolawole, 2018). however, in spite of the advances made in ils adoption by nigeria libraries, ani (2007), aguolu and aguolu  (2006) argued that libraries in nigeria have been slow to adopt this level of automation and that most academic and research libraries in nigeria had not computerized any of their functions. studies such as osaniyi (2010) and omeluzor, et al. (2012) have also shown that some of the ils adopted in nigeria libraries are not performing optimally, impacting negatively on the libraries’ achievements. it is against this backdrop that it becomes crucial to evaluate ils adoption, performance, achievements, and shortcomings in nigeria university libraries.   although there have been many studies conducted that identify issues related to ils in nigeria university libraries, none have used a rigorous model for the evaluation. the use of appropriate candidate variables (cvs) and evaluation criteria may provide evidence of the performance of ils that could support the decision for its adoption in library services. this study focusing on ils adoption, performance, achievements, and shortcomings adapted hamilton and chervany’s (1981) cvs approach to identify the performance features of ils used in nigeria university libraries.   background   the use of ils to automate or streamline library management, processes, and services is not a new phenomenon in developed countries. in developing countries, and especially nigeria, ils is gradually gaining momentum but not without some shortcomings. over the last decade, a considerable number of ils have been developed and deployed in libraries to facilitate easier access to information. in nigeria, efforts were made to adopt and use ils in library services. since the 1990s, when the world bank in 1990 deployed management information system in some selected federal universities to improve institutional capacities of nigeria universities, a considerable number of ils have been developed and deployed in nigeria university libraries to ease access to information. the intervention by the world bank to deploy management information system included the deployment of unified ils known as tinlib for library automation. in addition, some federal university libraries in nigeria, such as the university of ibadan and university of nigeria nsukka enugu centre, among others, had adopted cd/isis, x-lib, lib+, glass, and alice for windows to provide library services. similarly, several private university libraries have also adopted ils for library services. for example, bowen university, iwo (bui), and babcock university (bu) libraries had at different times adopted koha ils for library services.   university libraries in nigeria have adopted both proprietary and open source ils. proprietary ils products have been available for many years and are characterized by expensive customized coding; these products have remained the dominant approach used for library automation (uzomba et al, 2015). in contrast to proprietary ils products, open source software (oss) ils products provide the original source code used in creating it, as well as the right of redistribution, which provides users the freedom to modify and customize them in order to suit one’s own purposes. conversely, a closed proprietary system limits the way the library can access the underlying data (breeding, 2009). oss is freely developed for the enhancement of routine library activities. oss is available for anyone to have; not only is the software free, but it is also free for anyone to  run, copy, distribute, study, change, improve, modify, and share for any purpose, thus enabling libraries to have greater control over their working environments (kumar & jasimudeen, 2012). whether the libraries have adopted either proprietary or open source ils in nigeria university libraries, there is evidence in the literature that challenges facing the adoption of ils in nigeria university libraries abound. however, few if any studies have focused on the adoption, achievements, performance, and shortcomings of ils, which this present study tries to accomplish. the researcher believes that this study will contribute in developing a model for the evaluation of the adoption, achievements, performance, and shortcomings of ils using the already proposed model by hamilton and chervany (1981) together with farajpahlou’s (1999, 2002) evaluation criteria for the evaluation of ils in university libraries across the world. there are already numerous studies on the prospects, performance, successes and challenges of ils adoption and use in university libraries, especially in developing nations (osaniyi, 2010; omeluzor, et al, 2012; breeding, 2009; uzomba,  et al, 2015; atua-ntow, 2016). however, none of these studies have revealed the use of hamilton and chervany (1981) cvs to evaluate the capability of ils in library services.   aims   this study is aimed at designing a model that would be fundamental for evaluating the adoption, performance, achievements, and shortcomings of ils in university libraries. it is guided by the following objectives, to:   1.      evaluate the extent of ils adoption in nigeria university libraries. 2.      evaluate the achievements made so far with ils in nigeria university libraries. 3.      evaluate the performance of ils in library services in nigeria university libraries. 4.      evaluate the shortcomings of ils in nigeria university libraries.   literature review   evaluating ils in nigeria university libraries   hamilton and chervany (1981) proposed an approach for the evaluation of management information systems (mis) that involved the use of candidate variables (cvs) such as: data currency, accuracy, reliability and completeness, system flexibility, ease of use, response time, and turnaround time. farajpahlou (1999) proposed the use of specific criteria for assessing the success of ils. these criteria were in four broad categories including: management of the system, usage of the system, technicalities of the system, and boundary issues. each of the criteria is found to be useful in this present study as together they present the basis for identifying the achievements, performance, and shortcomings of ils in university libraries. farajpahlou (2002) further emphasized that a successful automated library system would require pre-conditions such as a well-prepared automation plan and implementation program. consistent evaluation of ils is important to identify areas of improvement for effective services. hill and patterson (2013) noted that assessment could present challenges but is still worthwhile to undertake if the aim is to create and add value to that which is being assessed. similarly, okpokwasili and blakes (2014) believe that assessments of ils, library services, and resources need to be carried out on a continuous basis to ensure that they remain relevant to the needs of their patrons and stakeholders.   omeluzor and oyovwe-tinuoye (2016) assessed the adoption and use of ils for library services in university libraries in edo and delta states. a section of the instrument used for the study elicited information on the use, achievement, effectiveness, and challenges of ils in academic libraries in the two states. findings in the study revealed that the automation software adopted in some of the university libraries were effective for accessing books, journals, and other library materials, as well as for bibliographic search and retrieval. although the study presented some issues about use, achievement, effectiveness, and challenges, it did not focus on hamilton and chervany’s (1981) cvs or farajpahlou’s (1999, 2002) criteria, which is a gap that this present study tries to bridge. some studies (akpokodje & akpokodje, 2015; ojedokun, olla & adigun, 2016) have shown one or two of the variables, such as adoption, achievements, performance or shortcomings. surprisingly, none of these studies has tried to integrate hamilton and chervany’s (1981) cvs which could have provided a clearer view of the performance and perhaps records of achievements and shortcomings of ils in library services. a deliberate study, with a focus on cvs, could reveal some underlying attributes of ils and the reasons for its adoption in library services.   adoption of ils in nigeria university libraries   libraries in nigeria have had their share of problems in the adoption of ils. for example, the world bank in collaboration with the national universities commission (nuc) in 1990 supported 20 federal universities in nigeria with tinlib automation software among other ict tools for 20 participating libraries. the effort did not yield expected results since sani and tiamiyu (2005) in their evaluation of automated services in nigerian universities found that the system fell short of some of the evaluation criteria and cvs proposed by hamilton and chervany (1981) and farajpahlou (1999, 2002). sani and tiamiyu (2005) observed that the state of automated library services in the universities that were visited was haphazard, with the situation in state and private universities being particularly pathetic. this scenario may not be unconnected to lack of evaluation on the achievements, performance, and shortcomings of the ils before adoption. some laudable initiatives in the adoption of ils for library services in nigeria have failed in the last two decades due to lack of evaluation (okiy, 1998; nok, 2006; osaniyi, 2010; adegbore, 2010; omeluzor et al., 2012; mbakwe & ibegbulam, 2014).  aguolu, et al. (2006) reported the non-computerization of library functions in nigeria university libraries. a study by oladokun and kolawole (2018) revealed that 35 libraries across the six geo-political zones of nigeria had adopted koha open source software. findings in that study revealed that 13 (36%) of the respondents indicated lack of support from their institutions as a major reason for non-adoption of koha in their libraries.   shortcomings of ils in nigeria university libraries   on the shortcomings of ils in nigeria university libraries, most of the studies focus on the challenges of the automation process, such as: technical problems, problems with retrospective conversion, non-availability of the software and vendors’ attitudes, inadequate funding, lack of skill, inadequate ict facilities, power supply, and others (agboola, 2000; sani & tiamiyu, 2005; osaniyi, 2010; omeluzor, et al., 2012; mbakwe & ibegbula, 2014). no single research study has been conducted showing a step by step approach on the evaluation, performance, achievement, and shortcomings of ils as portrayed in figure 1, which perhaps would have shown evidence and criteria for its adoption in libraries. the pitfalls of adopting one ils and switching over to another could perhaps be avoided if libraries adopt hamilton and chervany’s cv (1981) and even farajpahlou’s (1999, 2002) evaluation criteria before adoption of ils.   the model in figure 1 proposes the evaluation of the performance, achievement, and shortcomings of ils before adoption. the model is an expansion of hamilton and chervany cvs who in 1981 proposed evaluating only performance and achievement of mis, including: accuracy, reliability, completeness, flexibility, ease of use, and timeliness excluding shortcomings. emphasis in the model proposed by hamilton and chervany (1981) is primarily on the performance of information systems in the delivery of services. an evaluation of the shortcomings of ils as part of evaluation criteria would provide insights, helpful when making decisions about the adoption of ils in university libraries in the future.     figure 1 a model for the evaluation of ils in a university library     methods   research approach   the study adopted a descriptive survey design. the adoption of descriptive survey design provides the researcher the opportunity of using data collected for this study for ils evaluation using cvs in nigeria university libraries. according to nworgu (2006) a descriptive survey design describes a condition or phenomenon as it exists naturally without manipulation.   population   nigeria is made up of six geo-political zones including: north-east (ne), north-west (nw), north-central (nc), south-south (ss), south-east (se), and south-west (sw). the population of this study comprised systems/it and e-librarians in the university libraries from all six geo-political zones of nigeria. since the aim of this study was to evaluate ils, a purposive sampling method was used to select the systems/it and e-librarians who are directly in-charge of ils in their respective libraries. because of the large number of universities in each of the zones in nigeria, a convenience sampling method was used to select six universities, comprised of federal, state, and private universities, from each of the six geo-political zones of nigeria, offering a good representative sample to achieve the purpose of this study. therefore, the sample for this study is made up of 36 systems/it and e-librarians from the 36 selected universities in nigeria as shown in table 1.   research instrument development   based on the theoretical framework identified in the previous studies described above, the researcher developed a structured online questionnaire using a google online form with five sections (see appendix a) to provide answers to the questions raised on the evaluation of the adoption, performance, achievements, and shortcomings of ils in nigeria university libraries. sections 3 and 5 of the instrument were adopted from omeluzor et al. (2012) and omeluzor & oyovwe-tinuoye (2016). the study by omeluzor et al. (2012) reported the implementation of koha ils at babcock university library and the elements adopted are those that reveal the achievements that were made with ils such as: “provide on-the-spot access to resources,” “enable sharing of resources with other libraries,” “enable online cataloguing” and “provide access to books and external sources” (p. 218) that are relevant in this present study. on the other hand, omeluzor and oyovwe-tinuoye (2016) assessed the adoption and use of ils in academic libraries in edo state and delta state, nigeria. the elements adopted from that study are those that show the shortcomings of adopting ils such as: “inadequate training and technical knowhow for librarians,” “cost of implementation,” and “inadequate skilled personnel”. the elements presented in both studies are limited to one private university library and academic libraries in edo and delta states. using those elements in this present study provides more insight on how they affect the overall achievement and performance on the varied ils adopted in nigeria university libraries.   section 4 of the instrument was adopted from hamilton and chervany (1981) cvs. the researcher found the cvs proposed in hamilton and chervany (1981) for the evaluation of mis to be relevant in this present study, as it reveals the variables that should be considered for inclusion in the evaluation of ils in the university library. the cvs that were identified from previous research studies as being relevant to the evaluation of ils in library settings are: data entry and currency, accuracy, reliability, completeness, flexibility, ease of use, and timeliness. a 4-scale measuring instrument was used for sections 4 and 5 with 4 being the highest and 1 being the lowest.   distribution and data collection   before the administration of the questionnaire to the intended respondents, a pre-test was conducted to assess the reliability of the instrument on ten systems/it and e-librarians working in public libraries, who were not part of the study. the 10 responses were retrieved and analysed using cronbach alpha correlation co-efficient at 0.50 level of acceptance which gave a result of r =0.85. this indicates that the instrument is reliable and appropriate for data collection for this study since the test result is above the acceptance point of 0.50. furthermore, the instrument was also examined by an ils researcher to ensure content and construct validity. the questionnaire was then emailed to some of the respondents. emailing the respondents directly eliminated the possibility of receiving responses from unintended respondents. however, because the researcher could not access all the respondents via email, the use of nigeria library association (nla) online forum, nla it section and whatsapp group became unavoidable. the use of the platforms was found by the researcher as an alternative to contact those respondents that could not be reached, since all of them are registered members. the responses received through those platforms were carefully sifted to eliminate double response from those respondents that were earlier contacted via email as well as from non-systems librarians. the google response page was also very helpful in catching duplicate responses or any two respondents from the same university. the use of those platforms helped the researcher receive 100% of the responses needed to reach the goals of this study. the instrument elicited information on the ils adopted, its achievements, performance level, and shortcomings. data collected were analyzed using statistical package for social sciences (spss) version 7.0 and results are presented in frequency table, mean, standard deviation, chart, and percentage for clarity and understanding. in tables 2 and 3 the mean scores are rated as follows: mean is 0.1 to 1.9 = very low, 2.0 to 2.4 = low, 2.5 to 2.9 = high, 3.0 and above = very high.   results   demographic information of the respondents   results show that 47% of the respondents with the role of systems/it and e-librarians in university libraries in nigeria are female and 53% are male. the majority of the respondents (48%) have worked between 6-10 years. another 24% of the respondents have worked between 1-5 years, and 20% of the respondents have worked between 11-15 years. results also shows that a low percentage (8%) of the respondents have worked for 16 years or longer.     table 1a state of ils adoption in university libraries university ils currently in use ils earlier used federal koha alice for windows federal koha vitrual federal koha   federal koha alexandria federal strategic library automation (slam)   federal cds isis   federal koha   federal virtua   federal koha   federal new gen lib slam federal readable virtual, alice for windows federal koha alice for windows federal   virtual federal new gen lib millennium federal koha glass, lib+ federal koha virtual                       state slam   state     state koha   state koha x-lib, slam state     state senayan lms slam state     state koha   state slam   state alice for windows   state koha   state koha   state     private greenstone   private koha x-lib private new gen lib   private koha   private millennium   private koha   private koha       table 1b type of ils used earlier and currently in use in the selected university libraries university ils currently in use ils used earlier open source ils proprietary open source ils proprietary federal koha     alice for windows federal koha     vitrual federal koha       federal koha     alexandria federal   strategic library automation (slam)     federal   cds isis     federal koha       federal   virtual     federal koha       federal new gen lib     slam federal   readable   virtual, alice for windows federal koha     alice for windows federal       virtual federal new gen lib     millennium federal koha     glass, lib+ federal koha                                             virtual state   slam     state koha       state koha     x-lib, slam state         state senayan lms (slms)     slam state         state koha       state   slam     state   alice for windows     state koha       state koha       state         state         private greenstone       private koha     x-lib private new gen lib       private koha       private   millennium     private koha       private koha           research objective 1: extent of ils adoption in nigeria university libraries.   table 1a shows the state of ils adoption in federal, state, and private university libraries in nigeria (see full list of university libraries in this study in appendix b).   results in table 1a show the ones adopted earlier as well as the ones in use in the various university libraries represented in this study. it is evident in table 1a that the majority of the nigeria university libraries have adopted ils for the delivery of library services to their patrons. the results show that some libraries had adopted a different ils before adopting the current one in use while for some libraries, the one currently in use is their first ils. results in table 1a also show a shift from the adoption of proprietary ils among the federal university libraries to open source ils with 9 out of the 16 federal university libraries in this study using koha ils. this may be connected to the problems associated with the adoption and use of proprietary ils. results in table 1a also show that only 10 federal university libraries, 2 state university libraries and 1 private university library out of the 36 libraries represented in this study had earlier adopted ils while 23 of the libraries had none in use. however, results in table 1a show some improvement on the extent of adoption of ils as 31 of the libraries among the 36 in this study have adopted ils with only 5 of the libraries that have none in use.   the results in table 1b show the types of ils that are adopted in university libraries in nigeria. it is evident in table 1b that all the libraries that started the use of ils adopted only proprietary ils since none of the libraries in this study had open source ils. this shows that in the past, nigeria university libraries selected proprietary ils more frequently than open source ils, allowing the proprietary ils to thrive despite research that reported its shortcomings. results also show that out of the 36 libraries in this study, 8 libraries (4 federal, 3 state, and 1 private) are currently using proprietary ils.  furthermore, results in table 1b show that out of the 13 state university libraries, 4 are yet to adopt ils for library services. it is interesting to note that federal libraries have predominantly changed systems in this study, while open source ils was not initially adopted by any of the libraries. this means that the federal university libraries are at the forefront of adopting ils in nigeria than the state and private universities. the higher number of federal university libraries that adopted ils may be the result of the effort made by the world bank in collaboration with the national universities commission (nuc), which supported 20 federal universities in nigeria with tinlib automation software.   research objective 2: level of achievements with ils in nigeria university libraries   the results shown in figure 2 demonstrate some strides that have been made by the university libraries in nigeria with the use of ils. the results indicate that out of the 36 respondents from the various libraries in this study, as shown in appendix a, a majority or 30 of them specify that the ils adopted at their various libraries provided on-the-spot access to resources for their patrons. results shown in figure 2 also suggest that adoption of ils in university libraries in nigeria enable sharing of information resources with other libraries as attested by 26 of the respondents. others key results include the use of online cataloguing, accessing books and external sources, and making online instruction available for staff and students. the result shows that nigeria university libraries are making progress in library services through the use of ils. perhaps this is also due to the increasing number of ils that is now in use in the university libraries as shown in table 1a.     figure 2 achievements made with ils in university libraries in nigeria     table 2 performance of ils in library services     research objective 3: performance of ils in library services in nigeria university libraries   in table 2, the results show an impressive performance of the ils that are adopted in various university libraries throughout nigeria. results reveal that ils accuracy and flexibility are good  = 4.66 respectively while data entry and currency has = 4.55.  on the ease of use, reliability and completeness, ils results show higher mean,  = 4.38 and 4.33 respectively. result on the timeliness of ils shows a mean of  = 4.27.   the findings in table 2 indicate that the adoption of ils would improve the overall performance and increase productivity of the library because of timeliness, reliability, and accuracy in providing resources for teaching, learning, and research, as well as gathering statistics of all the activities within the ils. in addition, the flexible nature of ils makes it easier for both librarians and users to use it in performing their duties and assignments.   research objective 4: shortcomings of ils in nigeria university libraries   the results in table 3 reveal the shortcomings of ils adoption in university libraries in nigeria. the table focuses on those factors (shortcomings) that impede the performance and achievements of university libraries in their use of ils in nigeria.   the results in table 3 show that among the factors that count as shortcomings in the use of ils in libraries, robustness, inadequate training, and technical knowhow have a higher mean of 4.0 respectively. this means that these are the major shortcomings of ils in the libraries. other shortcomings of ils in nigeria university libraries in table 3 are inadequate skilled personnel  = 3.91, high cost of implementation  = 3.77, inadequate ict facilities in the library x = 3.69, tedious and difficult to manage  = 3.08, lack of vendors’ support  = 3.27, and frequent system failure  = 2.52. results in table 3 indicate that non-compatibility with the internet is less of a shortcoming for the adoption of ils in libraries than the other items listed in table 3. this implies that improving all of the shortcomings as shown in this result, would enhance the achievements and performance of ils in university libraries.   discussion   the findings in this study clearly demonstrate the importance of adopting hamilton and chervany’s cvs for the evaluation of ils in university libraries in nigeria and across the world. it further reveals the significance of using a holistic approach for the evaluation of ils, with the inclusion of shortcomings as proposed in the model in figure 1 before the adoption of ils, to eliminate the risk of failure and the tendency of switching from one system to another. it is evident from this study that most of the university libraries in nigeria had at one time or another switched from one system to another which might have affected their performance, achievement, and productivity. this scenario could be avoided when a thorough evaluation of the performance, achievements, and shortcomings is done on a new ils prior to implementation. table 3 shortcomings of ils in nigeria university libraries   taking a critical look at hamilton and chervany cvs and farajpahlou’s criteria, including the evaluation of shortcomings, before deciding on the adoption of ils, has the potential of helping the library to identify and avoid problems when implementing a new or second ils. such conscious evaluation would help to identify a feasible system. the findings in this study showed a gap in the evaluation criteria that has been bridged by the model as shown in figure 1. it would also be beneficial for university libraries in nigeria and other parts of the world that are yet to adopt any ils, to focus its evaluation criteria on the hamilton and chervany proposal, farajpahlou evaluation criteria, this current study, and possibly others. this approach could help to eliminate challenges that may arise in the future.   the findings in table 3 show some of the shortcomings of ils in nigeria university libraries. they indicates that among the factors, robustness, inadequate training, and technical knowhow have a higher mean of 4.0 respectively. this means that those variables are the major shortcomings of ils adoption in nigeria university libraries. these shortcomings needs to be critically examined because it has become a recurrent issue in some recent studies, for instance, osaniyi (2010), omeluzor et al. (2012) and ojedokun, olla and adigun (2016) have reported on some of these shortcomings without recommending the use of either cvs or farajpahlou’s evaluation criteria, which would have provided acceptable criteria to evaluate ils in libraries. the same shortcomings as shown in table 3 have remained major hindrances for the adoption of ils in some university libraries in nigeria and other parts of the world. these shortcomings are directly or indirectly affecting the functions and performance of the university libraries when it comes to the delivery of quality library services to their patrons.   limitations and opportunities for further study   this study was limited to 36 university libraries from the 6 geo-political zones in nigeria. the method of the study was limited to a structured online questionnaire without a face-to-face administration of the questionnaire and interview guide. the data collected and analyzed in this study were from the few selected university libraries in the six geo-political zones. a study of more university libraries in nigeria using qualitative methods might produce different results and provide additional information. future studies might investigate factors that would influence the choice for the adoption of ils in libraries and how to overcome some of the shortcomings that are revealed in this study.   conclusion and recommendations   the main reason for the adoption and use of ils in libraries is to enable quality management and delivery of library services, improving access and easy retrieval of information resources. in my opinion, the library patrons of today have high expectations of library services and are, for the most part, not satisfied with traditional methods. within the last decade, university libraries in nigeria have witnessed a turnaround in the adoption of ils despite the disparaging remarks of aguolu and aguolu (2006). the transformation being witnessed in the nigeria university libraries through the use of ils is tremendous as revealed by oladokun and kolawole (2018). furthermore, the performance of ils in nigeria university libraries as it relates to farajpahlou’s (1999) criteria for assessing the success of ils is encouraging. these criteria include data and currency, accuracy, reliability and completeness, flexibility, ease of use, and timeliness. that said, several shortcomings still exist in the adoption and use of ils in nigeria university libraries. improving ils adoption will require a conscious effort and decisiveness to ensure that librarians and library patrons enjoy the benefits that ils offer. ils developers should be able to consider the dynamic needs of university libraries and their patrons and therefore incorporate those specific features of hamilton and chervany’s cvs in their ils design, while keeping in mind the shortcomings presented in this study. this type of thoughtful design will enhance the quality of library services offered to patrons. due to the various challenges facing university libraries in nigeria, and some other countries in the world, and the failure of some libraries to adopt a robust ils, the following recommendations are put forward:   proper planning, adoption and implementation of ils in libraries should be the library’s first step. ils developers should strive to gather feedback from the ils user community to identify some of the shortcomings, leading to product enhancement. future developments in ils should incorporate tested theories such as hamilton and chervany’s cvs and farajpahlou’s criteria in order to meet certain expectations of librarians. university libraries should be at the forefront of providing necessary changes in the delivery of information services through the adoption and use of viable ils. tetfund support for the funding of government owned universities in nigeria should be encouraged and sustained while alternative sources of funding should be sought for ils adoption in university libraries. university librarians in nigeria should adopt a system for the training and up skilling of staff to help in acquiring relevant skills and knowledge related to the use of ils for library services.   references   adegbore, a. m. (2010). automation in two nigerian university libraries. library philosophy and practice (e-journal), paper 425. retrieved from: https://digitalcommons.unl.edu/libphilprac/425   agboola, a.t. (2000).  five decades of nigerian university libraries: a review.  libri, 50 (41), 27-34. https://doi.org/10.1515/libr.2000.280   aguolu, c. c. & aguolu, i. e. (2006). the impact of technology on library collections and services in nigeria, in the impact of technology on asian, african and middle easter library collections, ed. r. sharma (lanham, md.: scarecrow press, 2006), 145.   akpokodje, n. v. & akpokodje, t. e. (2015). assessment and evaluation of koha ils for online library registration at university of jos, nigeria. asian journal of computer and information systems, 3 (1), 20-27.   ani, o.  e. (2007). ict revolution in african librarianship: problems and prospects. gateway library journal, 10 (2), 111-117.   atua-ntow, c. (2016). staff assessment of the success of the integrated library system: the case of the university of ghana library system. http://hdl.handle.net/2263/59625 retrieved from https://repository.up.ac.za/bitstream/handle/2263/59625/ntow_staff_2017.pdf?sequence=4&isallowed=y   ayiah, e. m. & kumah, c. h. (2011, august 13 18). social networking: a tool to use for effective service delivery to clients by african libraries [paper presentation]. a paper presented at the world library and information congress: 77th ifla general conference and assembly. san juan, puerto rico. https://cf5-www.ifla.org/past-wlic/2011/183-ayiah-en.pdf   breeding, m. (2009). opening up library automation software. computers in libraries, 29(2), 25-28. retrieved from https://librarytechnology.org/document/13803    farajpahlou, a. h. (1999). defining some criteria for the success of automated library system. library review, 48(4), 169-180. https://doi.org/10.1108/00242539910276451    farajpahlou, a. h. (2002). criteria for the success of automated library systems: iranian experience (application and test of the related scale). library review, 51(7), 364-3721. https://doi.org/10.1108/00242530210438664    hamilton, s. & chervany, n. l. (1981). evaluating information system effectiveness – part 1: comparing evaluation approaches. mis quarterly, 5(3), 55-69. https://doi.org/10.2307/249291   hill, j.c. & patterson, c. (2013). assessment from a distance: a case study implementing focus groups at an online library. college & undergraduate libraries, 20(3-4), 399-413. https://doi.org/10.1080/10691316.2013.829376   kumar, v. & jasimudeen, s. (2012). adoption and user perception of koha library management system in india. annals of library and information studies, 59, 223-230. retrieved from https://core.ac.uk/reader/11890295   lucidea.com (n.d.). the integrated library system (ils) primer. retrieved from https://lucidea.com/special-libraries/the-integrated-library-system-ils-primer/   mbakwe, c. e. & ibegbulam, i. j. (2014). efforts and challenges of automation of university of nigeria, enugu campus library [paper presentation]. a paper presented at the nigeria library association, enugu state chapter 14th annual conference and general meeting, enugu state., november 25 – 29.   nok, g. (2006). the challenges of computerizing a university library in nigeria: the case of kashim ibrahim library, ahmed bello university, zaria. library philosophy and practice, 8 (2), 1-9. https://core.ac.uk/download/pdf/188041074.pdf   nworgu, b.g. (2006). educational research: basic issues and methodology. (2nded.). nsukka: university trust publishers, p. 23.   ojedokun, a. a., olla, g.o.o., & adigun, s.a. (2016). integrated library system implementation: the bowen university library experience with koha software. african journal of library & information science, 26(1), 3142. http://www.ajol.info/index.php/ajlais/article/view/135088   okiy, r. b. (1998). nigerian university libraries and the challenges of information provision in the 21st century. library bulletin, 3(1 & 2), 17-28.   okpokwasili, n. p. & blakes, e. (2014). users’ participation in acquisition and users’ satisfaction with the information resources in university libraries in the south-south zone of nigeria. journal of research in education and society, 5(3), 71.-85. retrieved from https://www.researchgate.net/publication/332321278_authors'_reputation_and_users'_satisfaction_with_the_information_resources_in_university_libraries_in_the_south-south_zone_of_nigeria   oladokun, t. & kolawole, l.f. (2018). sustainability of library automation in nigerian libraries: koha open source software. library philosophy and practice. paper 1929.  retrieved from https://digitalcommons.unl.edu/libphilprac/1929/   omeluzor, s. u. & oyovwe-tinuoye, g. o. (2016). assessing the adoption and use of integrated library system (ils) for library service provision in academic libraries in edo and delta states, nigeria. library review, 65(8/9), 578-592. https://doi.org/10.1108/lr-01-2016-0005     omeluzor, s.u., adara, o., madukoma, e., bamidele, i.a. & umahi, f.o. (2012). implementation of koha integrated library management software (ilms): the babcock university experience. canadian social science, 8(4), 211-221. https://doi.org/10.3968/j.css.1923669720120804.1860   osaniyi, l. (2010). evaluating the x-lib library automation system at babcock university, nigeria: a case study. information development, 26(1), 87-97. retrieved from https://doi.org/10.1177/0266666909358306   riaz, m. (1992). library automation. itic publishers and distributors: new delhi. 61.   sani, a. & tiamiyu, m. (2005). evaluation of automated services in nigerian universities. the electronic library, 23 (3), 274-288. https://doi.org/doi 110.1108/02640470510603679    sharma, r. n. (2009). technology and academic libraries in developing nations [paper presentation]. a paper presented at international conference on academic libraries (ical), india. p.23.   shekarau, m. i. (2014). list of certified librarians in nigeria, 2014. librarians’ registration council of nigeria (lrcn) directories, p. iii   singh, k. (2013). impact of technology in library services. international journal of management and social sciences research (ijmssr), 2(4), 74-76. retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.300.9109&rep=rep1&type=pdf     uzomba, e. c., oyebola, o. j. & izuchukwu, a. c. (2015). the use and application of open source integrated library system in academic libraries in nigeria: koha example. library philosophy and practice. paper 1250.  http://digitalcommons.unl.edu/libphilprac/1250     appendix a questionnaire on the evaluation of integrated library system in university libraries in nigeria: an empirical study of the achievements, performance and shortcomings   dear respondent,   this questionnaire may take about 10 minutes and is designed to elicit data for the evaluation of integrated library system (ils) adoption in nigeria university libraries focusing on the achievements, performance and shortcomings. information provided on this questionnaire will be used strictly for the purpose of this research. please note that the respondent is not under any obligation to respond to the questions. however, the researcher appeals for your assistance in order to achieve the purpose of this study on schedule.   thank you. researcher   section 1: demographic information of respondents a)      what is the name of your university ……………………………………………………………… b)      what is your gender   male         female c)      how long have you worked in your university 0-5 years                             6-10 years                          11-15 years                        16-20 years 21-25 years                        26-30 years                        31 years and above   section 2: extent of ils adoption in nigerian university libraries d)      which ils is in use at your library? please specify: …………………………………………….. e)      does your university use one previously? yes no maybe f)       if your answer to question 5 is yes, which one was that ………………………………………….   section 3: achievements made with the adoption of ils in library services g)      kindly indicate some of the achievements that your university has made with ils statement tick provide on-the-spot access to resources to patrons   sharing of resources with other libraries   enable online cataloguing   provide access to books and external sources   online instruction of staff and students   not in-use at the moment     section 4: performance level of ils in nigeria university libraries h)      how will you rate the performance of the ils in your library? items very good good poor very poor highly poor data entry           accuracy           reliability and completeness           flexibility           east of use           timeliness             section 5: shortcomings of ils adoption in nigeria university libraries i)       what are the shortcomings of ils adoption in university libraries in nigeria? items strongly agree agree strongly disagree disagree not sure tedious and difficult to manage           too expensive to implement           inadequate ict facilities in the library           non-compatibility of ils with the internet           frequent system failure           not robust or enough features to help achieve tasks           lack of vendors’ support           inadequate skilled personnel           inadequate training and technical knowhow for librarians               appendix b the selected federal, state and private universities in nigeria used in the study   sn university ownership 1. ahmadu bello university, zaria (abu) federal 2. bayero university kano (buk) federal 3. federal university lokoja (ful) federal 4. federal university of petroleum resources effurun (fupre) federal 5. federal university of technology, akure federal 6. michael okpara uni. of agric., umudike (mou) federal 7. nnamdi azikiwe university, awka federal 8. obafemi awolowo university, ile-ife (oau) federal 9. university of agriculture, markudi federal 10 university of benin (uniben) federal 11 university of ibadan (ui) federal 12. university of ilorin (unilorin) federal 13. university of jos (unijos) federal 14. university of lagos (unilag) federal 15. university of nigeria, nsukka (unn) federal 16. university of port harcourt (uniport) federal 17. ambrose ali university, kano (aau) state 18. bauchi state university state 19. benue state university (bsu) state 20. delta state university, abraka (delsu) state 21. ebonyi state university, abakiliki (esua) state 22. ekiti state university (esu) state 23. ignatius ajuru university of education(iauoe),rumuolumeni, port harcourt state 24. imo state university, owerri state 25. kogi state university, anyigba state 26. lagos state university, ojo (lasu) state 27. olabisi onabanjo university, ago iwoye (oou) state 28. tai solarin university of education, ijebu-ode (tasued) state 29. university of medicine, ondo (umo) state 30. american university of nigeria, yola (aun) private 31. babcock university, ilishan-remo (bu) private 32. bingham university private 33. bowen university, iwo (bui) private 34. landmark university, omu-aran. private 35, rhema university, obeama-asa, abia state private 36. samuel adegboyega university, ogwa (sau), ogwa, edo private     evidence based library and information practice evidence based library and information practice 2010, 5.2  evidence based library and information practice       evidence summary    neuroscientists’ domain knowledge does not improve search performance in pubmed     a review of:  vibert, n., ros, c., le bigot, l., ramond, m., gatefin, j., & rouet, j.‐f. (2009). effects of domain  knowledge on reference search with the pubmed database: an experimental study. journal  of the american society for information science and technology, 60(7), 1423‐1447.    reviewed by:  giovanna badia  librarian, royal victoria hospital medical library, mcgill university health centre  montreal, quebec, canada  email:  giovanna.badia@mail.mcgill.ca    received: 16 nov. 2009          accepted: 6 feb. 2010       2010 badia. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which  permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly  attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the  same or similar license to this one.    abstract   objective – to determine whether  neuroscientists and life scientists’ domain  knowledge affects their search performance in  pubmed for neuroscience topics.    design – cross‐sectional experimental study.    setting – state‐funded research laboratories in  the cities of paris, bordeaux and poitiers,  france.    subjects – there were 32 participants in the  study: 16 neuroscientists and 16 life scientists  with no experience in neuroscience. both  groups were similar in terms of age, gender,  occupation, and online database search  experience.      methods – all participants were asked to  complete the same five tasks in pubmed to  assess their search performance with this  database. each task consisted of finding and  selecting bibliographic references on a  neuroscience topic within 15 minutes. the  instructions for these tasks were hidden from  view during the search process. participants  performed the tasks on their office computers  between may 2005 and june 2006 in the  presence of one researcher who prompted  them to verbally describe what they were  doing and thinking as they searched. each  participant also filled out a questionnaire  about their personal characteristics at the  beginning of the search session and completed  a second questionnaire about their knowledge  of pubmed at the end. the entire experimental  procedure lasted between 60 and 90 minutes  and was recorded. the relevancy of the  97 mailto:giovanna.badia@mail.mcgill.ca evidence based library and information practice 2010, 5.2  bibliographic references selected was later  scored by two neuroscientists who did not  participate in the study. the data were  analyzed using multivariate analysis of  variance (manova) and qualitative analysis  of verbal protocols.    main results – the manova analysis did  not show any significant differences between  the total scores obtained by the neuroscientists  and the life scientists. both groups were able  to find relevant pubmed references for each  task within the time allotted. contrary to the  researchers’ first main hypothesis, the  neuroscientists’ domain knowledge did not  result in a superior search performance (i.e., in  less time spent searching and more relevant  results) compared to that of the life scientists.    however, domain knowledge did affect the  method of searching, confirming the  researchers’ second hypothesis. the life  scientists spent more time reading the  instructions for each task, included more  keywords in their search queries, and opened  more abstracts to select relevant references  than the neuroscientists. the life scientists also  used keywords that were almost exclusively  taken from the instructions for each task when  they searched pubmed and made significantly  more mistakes than the neuroscientists.    furthermore, the participants’ knowledge of  pubmed was poor as was expected, despite  stating they used it very frequently. half of the  participants did not attempt to use limits even  when the task called for it. the majority only  used pubmed in the most basic way, that is,  by typing keywords in the search box.      conclusion – domain knowledge affects how  end users search pubmed for topics in their  specialty, but it does not impact their  performance. both the neuroscientists and the  life scientists successfully completed the  search tasks on neuroscience topics within the  allotted time. both groups had basic  knowledge of pubmed, but were satisfied with  their performance and results.   the authors suggest that scientists would only  be interested in attending a pubmed training  session if they are convinced that they will  learn how to search more quickly.  further  experiments are needed to verify the effects of  domain knowledge on search performance  with topics that are more general. the search  tasks used in this study were very specific,  which may have positively influenced the  performance of all participants. a different  control group that shares less basic domain  knowledge with the neuroscientists, such as  mathematicians or chemists, may also be  tested.      commentary     lindsay glynn’s eblip critical appraisal  checklist (2006) was used to critically appraise  this study. to place this study in context, the  authors conducted a comprehensive literature  review to identify the factors that influence  scientists’ online information seeking  behaviour. these factors are: experience in  online searching, knowledge or experience  using the search tool, domain knowledge, and  general cognitive abilities such as spatial  visualization, vocabulary level, and processing  speed. this study examined one of these  factors, domain knowledge, and the results  indicate that domain knowledge does not  affect search performance in pubmed, which  contradicts the authors’ original hypothesis.      however, there are confounding variables in  the study that may negate its findings. for  example, the search tasks were very specific.  eighty‐four percent of the keywords used by  the neuroscientists to compose their queries  and 93% of the keywords used by the life  scientists were taken directly from the search  question. participants were also required to  find only one or two articles to complete each  task. both points raise questions about the  validity of the search tasks. rather than  measuring the participants’ search  performance, the tasks may actually have  assessed the effectiveness of pubmed to return  some relevant results to pertinent keywords,  which were supplied by the researchers in the  search question. secondly, the control group  (the life scientists) followed a similar  98 evidence based library and information practice 2010, 5.2  99 undergraduate program to the test group (the  neuroscientists), which may have resulted in  sharing some basic domain knowledge that  gave both groups the same advantage in  answering the questions. the authors  suggested further experiments that will  address these weaknesses, including using less  well‐defined questions and adding a control  group that does not share basic domain  knowledge with the neuroscientists.    in addition, the small sample size and the  authors’ omission of their selection criteria for  the sample population make it difficult to  generalize the study’s findings. stronger  evidence is needed to confirm that domain  knowledge does not improve search  performance in pubmed. it would also be  difficult for other researchers to replicate this  study, since the pre‐task and post‐task  questionnaires are not included in the  appendices.    despite its weaknesses, this study will interest  reference and instructional librarians who seek  to understand how scientists search pubmed.  the appendix includes a detailed list of errors  made by participants during their searches,  which may suggest concepts or features that  should be stressed during a pubmed training  session. the authors feel that pubmed training  should focus on showing users how to  decrease the number of their mistakes and  search more quickly, rather than emphasizing  how to increase the relevancy of their search  results. the authors remind readers that  scientists and health professionals exhibit  satisficing behaviour when it comes to  information seeking, searching only as long as  it takes them to find enough useful references  to satisfy their information need.      references    glynn, l. (2006). a critical appraisal tool for  library and information research.  library hi tech, 24(3), 387‐399.  evidence summary   green deposit rates in lis taylor & francis journals: are librarians “practicing what they preach?”   a review of: emery, j. (2017). how green is our valley: five year study of selected lis journals from taylor & francis for green open access. insights, 31(23). http://doi.org/10.1629/uksg.406     reviewed by: jessica a. koos senior assistant librarian/health sciences librarian stony brook university stony brook, new york, united states of america email: jessica.koos@stonybrook.edu   received: 26 feb. 2019                                                                  accepted: 2 apr. 2019      2019 koos. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29560     abstract   objective – to investigate the green deposit rate for articles published in five taylor & francis lis journals.   design – content analysis.   setting – the author conducted an analysis of the following journals: behavioral & social sciences librarian, collection management, college & undergraduate libraries, journal of electronic resources librarianship and journal of library administration.    subjects – 87 articles/columns in behavioral & social sciences librarian, 78 in collection management, 134 in college & undergraduate libraries, 108 in journal of electronic resources librarianship, and 264 in journal of library administration.    methods – the author chose five taylor & francis lis journals to analyze over a period of five years for the green open access article deposit rate. the author selected taylor & francis journals due to the publisher’s policy of not requiring an embargo period on lis journals. the specific journal titles were selected based on the author’s perception of their relevance to a broad array of academic libraries. the author determined if green deposit had occurred by first using the “oa button” on the article’s homepage to locate the full text. if nothing was found, the author then searched each author’s institutional repository using the doi. if the full text was still not located using this method, then a google scholar search for the full text was performed.   main results – the author found that the full text was available for 22% of the 671 total articles included in the study, which was significantly below the author’s proposed success rate of 50%.   conclusion – the results of this study indicate that a relatively low number of articles in the lis field are available via open access, even though there were no restrictions from the publisher on green deposits. some potential influencing factors for the low deposit rate include lack of encouragement from administration on utilizing repositories, imposter syndrome, and a lack of awareness of taylor & francis’s green deposit policies. the author recommends that librarians and their administrators support and encourage one another to make articles available via open access. the author also recommends that taylor & francis further publicize this policy to make more authors aware of it.   commentary   the quality of this study was appraised using “the cat: a generic critical appraisal tool” created by perryman & rathbun-grubb (2014). based on this analysis, the quality of the study was found to be moderate. the author is a librarian at a large public academic library, and holds the title, “collection development & management librarian, associate professor.” a brief literature review was included in the article, and the author clearly defined the research question.   for the most part, the author clearly explained the research methods. however, there was no in-depth discussion as to the limitations of the study. for example, there was a lack of clarity as to how exactly the journals to be analyzed were selected. a more thorough explanation of this process could possibly reduce concern about selection bias.   the author also seems to have arbitrarily chosen 50% as the success rate for green oa deposits, as there is not adequate justification as to why the author used this specific figure. a more scientifically based figure could potentially make the results of the study more reliable. additionally, there was no measurement of the rate of deposit prior to the release of the embargo, so it is impossible to determine if the embargo had any effect on oa deposits. it is important to take these factors into account when determining the strength of evidence, as they have a direct impact on the conclusions being presented by the author.   additionally, it is important to consider the fact that only taylor & francis journals were included in this study, and the relatively small sample size of 671 articles (compared to the total amount of existing lis articles) published within a five-year time period greatly limits generalizability. a study including articles from a larger sample size and various publishers of lis research could provide a more accurate picture of deposit rates.                 overall, the author’s research helps to raise awareness about green open access in the field of lis, more specifically, the policy of taylor & francis which allows green deposits without imposing an embargo period. as the author mentioned, it was interesting that the group of professionals who should be the strongest supporters of oa were not “practicing what they preach.” increased awareness and further research on the topic can help to change that. in addition, when selecting journals in which to publish their work, librarians should take into account publisher oa policies and select those that allow for the green deposit of their work. choosing to publish in oa journals helps to uphold one of the core tenets of librarianship: to ensure equal access to information for all.   references   perryman, c. & rathbun-grubb, s. (2014). the cat: a generic critical appraisal tool. in jotform – formbuilder. retrieved from http://www.jotform.us/cp1757/thecat microsoft word es_haigh.doc evidence based library and information practice 2006, 1:3      77 evidence based library and information practice       evidence summary    library catalogue users are influenced by trends in web searching search strategies     a review of:  novotny, eric. “i don’t think i click: a protocol analysis study of use of a library online  catalog in the internet age.”  college & research libraries, 65.6 (nov. 2004): 525‐37.    reviewed by:  susan haigh   senior policy officer, library and archives canada  ottawa, ontario, canada  e‐mail: susan.haigh@lac‐bac.gc.ca    received: 25 may 2006            accepted: 23 june 2006      © 2006 haigh. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to explore how web‐savvy  users think about and search an online  catalogue.     design – protocol analysis study.    setting –  academic library (pennsylvania  state university libraries).    subjects – eighteen users (17 students, 1  faculty member) of an online public access  catalog, divided into two groups of nine  first‐time and nine experienced users.       method – the study team developed five  tasks that represented a range of activities  commonly performed by library users, such  as searching for a specific item, identifying a  library location, and requesting a copy.  seventeen students and one faculty member,  divided evenly between novice and  experienced searchers, were recruited to  “think aloud” through the performance of  the tasks. data were gathered through audio  recordings, screen capture software, and  investigator notes. the time taken for each  task was recorded, and investigators rated  task completion as “successful,” “partially  successful,” “fail,” or “search aborted.”  after the searching session, participants  were interviewed to clarify their actions and  provide further commentary on the  catalogue search.      main results – participants in both test  groups were relatively unsophisticated  subject searchers. they made minimal use of  boolean operators, and tended not to repair  failed searches by rethinking the search  vocabulary and using synonyms.  participants did not have a strong  understanding of library catalogue contents  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:3      78 or structure and showed little curiosity in  developing an understanding of how to  utilize the catalogue.      novice users were impatient both in  choosing search options and in evaluating  their search results. they assumed search  results were sorted by relevance, and thus  would not typically browse past the initial  screen. they quickly followed links,  fearlessly tried different searches and  options, and rapidly abandoned false trails.      experienced users were more effective and  efficient searchers than novice users. they  used more specific keyword terms and were  more persistent to review their search  options and results. through their prior  experience, they knew how to interpret call  numbers, branch library location codes, and  library terminology such as ‘periodicals’.      participants expected the catalogue to rank  results based on relevancy like an internet  search engine. while most were observed to  understand intuitively the concept of  broadening or narrowing a search, a  ‘significant minority’ added a term to an  already too‐narrow search to improve their  search results.      when interviewed, participants suggested  several ways to improve the catalog search  query, such as adding summaries and  contents, ranking results by relevance and  degree of exact match to search terms,  including an amazon‐like “find more like  this” feature, and providing context‐ sensitive and interactive online help,  especially at the point when a search has  produced too many or too few hits.      conclusions – the study concluded that  library catalogue users are heavily  influenced by trends in web searching. no  matter what type of search a task called for,  the participants tended to expect a simple  keyword search to lead to optimal results  presented in relevancy‐ranked order.     because users do not generally know or care  about the structure of a bibliographic record,  and many have little concept of what a  library catalogue is for or what it contains,  novotny suggests that user instruction  needs to address these basics.  he also  suggests that library professionals and  library system vendors must work together  to address the clear evidence that library  catalogues are failing their users.     commentary    this was a thorough and generally  methodologically sound study.  however,  given that the study objective was to assess  how a group of “web‐savvy generation”  users use an online catalogue, it is not clear  enough that the sample, especially the  ‘experienced’ group, were all of that  generation and indeed web‐savvy.     one regrettable shortcoming in data  collection was that user satisfaction ratings  were not collected.  there are also a few  shortcomings in data reporting. for example,  no table is provided to show success rates  by task and type of user, although these  data were captured. thus, the reader cannot  confirm the second part of the assertion that  “on average, [experienced users] spent less  time per task and were more likely to  complete tasks successfully” (530).  in fact,  more supporting data in the discussion  would have been helpful in the discussion.  for example, the discussion states that users  were “far more likely to try another search  than to examine the reasons why the initial  search failed” (529). but the number of  failed searches is not reported, nor is the  percentage of times that participants tried  new searches instead of asking themselves  why a search failed.     evidence based library and information practice 2006, 1:3      79 although the study was intended to suggest  interface changes for the pennsylvania state  libraries catalogue, it is unclear whether  any changes were made as a result of the  study. by describing changes the author  personally made to his instruction sessions,  he seems to give more credence to adjusting  user education than to revising local  interface choices, though both are no doubt  merited.      ultimately, the author lands—correctly in  my opinion—on the fact that local tweaks,  whether to interface or instruction, are not  the whole answer. we cannot persist in  thinking that users will come to understand  our way of doing things if only we educate  them enough.     the evidence suggests that the long‐ standing and well‐documented failures of  the online library catalogue are now simply  more acute, given the pervasiveness of  alternate search models and interfaces  provided by internet search tools.                                               the library profession needs to  acknowledge how users actually search,  then work proactively with vendors to build  systems that accommodate these actual user  behaviors. that likely means that we need to  build toward proven, analogous search  interfaces, such as amazon.com; to  introduce more full‐text data into our  catalogues; to put controlled vocabularies to  more effective use; to support user  participation features; and to make greater  use of data mining.              microsoft word es_perryman.doc evidence based library and information practice 2006, 1:3      84 evidence based library and information practice       evidence summary    differences between library instruction conference attendees and their institutional  affiliations in the united states and canada are discernible    a review of:  willingham, patricia, linda carder, and christopher millson‐martula. “does a border make a  difference? library instruction in the united states and canada.” journal of academic  librarianship 32.1 (jan. 2006): 23‐34.    reviewed by:  carol perryman  trln doctoral fellow, school of information & library science  university of north carolina at chapel hill  chapel hill, north carolina, united states of america  e‐mail: cp1757@gmail.com    received: 29 may 2006            accepted: 03 july 2006      © 2006 perryman. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – the primary intent of this study  was to identify differences among library  instruction conference attendees and their  institutions between the united states and  canada. the overall hypothesis was that  there would be areas of measurable  distinction between the two countries. the  authors tested nine hypotheses: #1, that the  largest number of survey respondents  would be employed at large institutions; #2,  that statistically, the majority of well‐ developed instructional programs are found  at universities rather than colleges; #3, that  beginning programs are more often found at  four‐year institutions; #4, that program  development and technological issues  predominate among instructional foci in the  early twenty‐first century; #5, that more  experienced librarians are more likely to  attend library instruction conferences; #6,  that loex (originally an acronym for  library orientation exchange) is perceived  as the most valuable conference in library  instruction; #7, that the impact of conference  attendance upon library program  development is only moderate; #8, that  conference theme and reputation are the  two greatest factors contributing to  attendance; and #9, that the majority of  conference attendees are from the united  states.    design – historical research, and an e‐ mailed survey.    http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:3      85 setting – libraries and library instruction  conferences in the united states and canada.    subjects – one hundred thirty‐two  librarians who were attendees at one of  three library user instruction conferences:  loex, loex of the west, and wilu  (workshop on instruction in library use).    methods – first, a brief historical review  was conducted on the influence of social,  economic, and political events on the  development of library user instruction, the  creation of conferences focused on library  instruction in from the united states and  canada, and national surveys looking at  institutional support for instructional  development. next, a survey instrument  consisting of fifteen demographic and  attitudinal questions was sent via e‐mail to  all 508 attendees of major library instruction  conferences (loex and wilu for 2001, and  loex of the west for 2000) in the united  states and canada. responses from the 132  returned surveys were tabulated and used  to evaluate their linked hypotheses.    main results – of the nine initial hypotheses,  five were supported, and the remaining four  were either partially supported or rejected.  supported hypotheses included: #1, that  most participants in the top library  instructional conferences came from  institutions with >5,000 student populations;  #2, that the majority of fully developed  instructional programs were in universities;  #5, that librarians with greater seniority  were more likely to attend instructional  conferences; #7, that conference attendance  has only a medium impact on program  development at participants’ home  institutions; and #9, that most conference  attendees come from the united states.  partially supported hypotheses were: #4,  that factors most highly rated by  participants were program development  and technology, and #8, that conference  theme and reputation are ranked higher in  terms of influence in attendees’ decision to  participate in the conferences. rejected  hypotheses included: #3, that “beginning  programs are typically found at four‐year  institutions,” #4, that “program  development and technology rank as the  two most important instruction‐related  issues” (note that hypothesis #4 is both  rejected and partially supported), and #6,  that “loex is considered the most valuable  conference.”    conclusion – the authors confirmed their  overall hypothesis that significant  differences exist between the united states  and canada regarding library instructional  programs. although the two countries  developed at very different rates prior to the  1960s, technology and cross‐border sharing  has meant that they are now developing  along parallel paths. the authors suggest  several avenues for further study including  the need to consider attendees over a greater  time span, the differences in responses  between younger and more senior  participants, and questions about the real  differences between library instructional  programs in canada and the united states.    commentary    the inclusion of a thorough history of the  development of library instruction  programming in libraries in the u.s. and  canada is a valuable addition to the  literature, and would have been  strengthened by publication as a standalone  article. as it exists, the links between the  historical overview and the research  components of this work do not appear  directly pertinent to the research questions.    an additional value of the work is  instructional, in that the authors openly  discuss problems with the data collection  instrument. based on the critical appraisal  checklist created by lindsay glynn  (pending publication in library hitech), the  evidence based library and information practice 2006, 1:3      86 study is invalid both in terms of the sections  of appraisal (population, data collection,  study design, and results), and overall.  although the authors clearly define their  population, and the sample size is  sufficiently large for precise estimates, the  respondents are self‐selected, due to the  survey’s distribution by e‐mail. a major  concern is that with the low respondent rate  from canada (28), analysis of the survey  questions is questionable, especially when  broken down into sub‐categories of  seniority and institution type.    the authors obtained names and contact  information from program attendance  records, and fail to mention whether they  had deduplicated the records, which means  that individuals attending more than one of  the conferences (reported as 25.1% of  respondents, or 33/132) may have falsely  inflated data for analysis. we are also not  informed of the country of origin of multiple  conference attendees. the overall hypothesis  of this study, which questions whether there  are measurable distinctions between the  library instruction of the united states and  canada, is unclearly defined and ultimately  better addressed by the historical section of  the article than by the research section,  which focuses on conference attendance.    for the second area of appraisal, data  collection, critical analysis of the study  establishes that the methods are not well  described. validity in this area would have  been greatly enhanced by the inclusion of  the survey instrument, though we recognize  that space constraints may have prevented  its publication. participants attended  conferences from 2000 to 2001, but it is not  indicated when the survey was conducted.  too lengthy a time from conference  attendance to the survey may have  introduced recall bias, an issue that is not  addressed. in addition, there was no attempt  to pilot the instrument, so that errors  present in the design were not discovered  until post‐administration analysis. the e‐ mailed survey questions were not provided,  so readers cannot discern bias, especially  when the results were summarized rather  than enumerated. for example, tables 12‐15  in the study inexplicably contain only  rankings, and no numeric information.     the study design, a third area of appraisal,  is undermined by the lack of detail provided,  although the researchers’ choice of an e‐mail  survey sent to all conference participants is  appropriate for the research questions asked.  the reader would benefit from a discussion  of response coding, inter‐rater reliability  methods (if any), and term definition, such  as what determines the developmental level  of an educational program (a focus for  hypothesis #2). further enhancement could  have been gained by discussion of whether  ethics approval had been sought.    one strong component of the study is the  results section. the authors carefully and  succinctly separate the results of each of the  nine hypotheses, and are frank in their  appraisal of the study’s weaknesses. finally,  they provide thoughtful suggestions for  further research in this understudied area.  even with the design errors, the results of  this study can somewhat benefit program  organizers for future conference planning,  and form the basis for further research  among the two populations.     unfortunately, the results are not rigorously  considered in the summary and conclusions  section, where the researchers make several  unsupported statements. for example, the  conclusion is made that differences in  program maturity levels between the two  countries are significant (hypothesis #2),  even though the authors had previously  discussed instrument and study design  weaknesses that would render these same  conclusions questionable. another example  is the inference for hypothesis #5 that  librarians with more seniority were more  evidence based library and information practice 2006, 1:3      87 likely to attend conferences due to their  seniority and greater access to funds – the  first part was established, but not the cause.  ultimately, the study’s primary question of  whether the ‘border makes a difference’ is  unanswered. future research in this area  would benefit from a more narrowly  defined set of hypotheses, a pilot study, and  a larger population sample size.     evidence based library and information practice evidence based library and information practice 2011, 6.1 68 evidence based library and information practice evidence summary scholars in the humanities are reluctant to cite e-texts as primary materials a review of: sukovic, s. (2009). references to e-texts in academic publications. journal of documentation, 65(6), 9971015. reviewed by: deena yanofsky liaison librarian walter hitschfeld geographic information centre mcgill university montreal, quebec, canada email: deena.yanofsky@mcgill.ca received: 15 sept. 2010 accepted: 13 feb. 2011 2011 yanofsky. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to explore the role of electronic texts (e-texts) in the research and citation practices of scholars in the humanities. design – a mixed-methods approach, employing a combination of quantitative and qualitative methods extended by semistructured interviews. setting – six universities in australia and one university in the united states. subjects – sixteen humanities scholars participated in the study, including nine historians and seven literary scholars. methods – the study had two stages. the first stage included in-depth, semi-structured interviews, examination of participants’ manuscripts and published works, and evaluation of some e-texts mentioned during interviews. the second stage involved detailed data gathering from a group of four participants selected from the participants in stage one, followed by a final interview. data were collected throughout 2005 and 2006. in total, the study investigated thirty research projects. in stage one, participants were asked to discuss one finished and one current research project in which e-texts were used as primary sources. participants in the second stage were asked to record data about their interactions with e-texts during the current research project on forms and audiotapes. researchers mailto:deena.yanofsky@mcgill.ca� evidence based library and information practice 2011, 6.1 69 who completed forms and recorded comments were asked to discuss their view of e-texts in the research process. the interviews and tapes with comments were transcribed and coded to protect participants’ anonymity and strengthen the interpretive validity of the study. data were analyzed by adopting a hermeneutical approach. the study results do not have any statistical significance and the findings cannot be generalized beyond the study data. main results – participants in this study used a wide variety of e-texts for their research, but seldom included citations to these electronic sources in their published work. e-texts included digitized materials from libraries, archives, and private collections as well as ‘electronically born’ documents, works of art and popular culture artifacts. of the 22 works resulting from the research projects examined during the study period, half did not cite etexts as primary materials. the 11 works that made at least one reference to an e-text included 4 works in which the only reference was to e-texts created by the actual author. in total, only 7 works referred to e-texts created by outside authors. these 7 final works were written by 5 participants, representing 31 percent of the total number of study participants. analysis of the participants’ citation practices revealed that decisions to cite an electronic source or omit it from publication were based on two important factors: (1) the perceived trustworthiness of an e-text and (2) a sense of what was acceptable practice. participants established trustworthiness through a process of verification. to confirm the authenticity and reliability of an e-text, most participants compared electronic documents against a print version to verify provenance, context, and details. even when digitized materials were established as trustworthy sources, however, hard copies were often cited because they were considered more authoritative or accurate. traditions of a particular discipline also had a strong influence on a participant’s willingness to cite e-texts. participants working on traditional historical topics were more reluctant to cite electronic resources, while researchers who worked on topics that explored relatively new fields were more willing to acknowledge the use of e-texts in published works. traditional practices also influenced participants’ decisions about how to cite materials. some participants always cited original works in hard copy, regardless of electronic access because it was accepted scholarly practice. conclusions – the results of this study suggest that the small number of citations to electronic sources in publications in the humanities is directly related to researchers’ doubts about the reliability and authenticity of e-texts. electronic resources provide a large body of primary data for research in literary and historical studies; however, the uncertain trustworthiness of many primary materials in electronic form makes these texts less acceptable for academic publications. the study provides suggestions for further research into the social dynamics that underpin and determine academic research practices and contemporary processes of knowledge production. commentary this study attempts to go beyond the citation studies of previous researchers by providing a meaningful analysis of the decision-making process directly related to citation practices in the humanities. the author provides a comprehensive overview of the research literature and presents evidence to support two competing theories of behaviour that frame the results of the study: normative and social constructive views. the discussion of these theories offers important insight into the contemporary research practices of scholars working in literary and historical studies and the reasons why researchers decide to include or exclude digital resources from publications. evidence based library and information practice 2011, 6.1 70 the author’s analysis of the reasons for citing preferences and practices provide evidence to support a strong relationship between citation and the influence of social rules in academia. the study, however, struggles to provide insight into the characteristics of acceptable academic (versus non-academic) electronic sources in the context of humanities scholarship. participant responses seem to offer conflicting definitions of primary materials and what constitutes “real” scholarship. some of the responses may be determined by the researcher’s age or topic rather than by characteristics of the online resources themselves. the author makes passing mention to these contributing factors, yet does not extend the analysis to include these important attributes. the methods used to collect the data in this study are insufficiently described in this article. although the author does point to an earlier publication in which more details about the study’s methodology can be found (sukovic, 2008), a sample of the study questions and responses would have been a useful appendix to this article. as the author notes, this study was based on in-depth data gathering from a small group of participants; the study has no statistical significance and the findings cannot be generalized. nevertheless, to anyone working in the humanities the description of researchers’ use and non -use of e-texts will seem quite familiar. the results from the study seem useful, especially the insight into the social constructs of authority and acceptable academic practices in shaping the way researchers interact with and use electronic resources. references sukovic, s. (2008). convergent flows: humanities scholars and their interactions with electronic texts. library quarterly, 78(3), 263-284. / evidence based library and information practice evidence based library and information practice evidence based library and information practice 2010, 5.2  evidence based library and information practice       editorial responsibilities       2010. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which  permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly  attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the  same or similar license to this one.    editor‐in‐chief: denise koufogiannakis    associate editor (articles): alison brettle    associate editor (evidence summaries): lorie kloda    associate editor (classics): jonathan eldredge    associate editor (features, commentaries, ebl101, news): denise koufogiannakis    production editor: katrine mallan    editorial intern: andrea baer    copyeditors: priscilla stephenson (lead copyeditor), georgianne bordner, susan dewar, richard  hayman, heather pretty, lisa shen, dale storie, mary virginia taylor, elizabeth zeeuw    indexing support: pam morgan    writing assistance: molly des jardin, hope leman, tracy powell iwaskow, carol waseleski  4 evidence based library and information practice evidence based library and information practice 2010, 5.1  evidence based library and information practice       commentary     identifying customer expectations is key to evidence based service delivery    susan mcknight  director, libraries & knowledge resources  nottingham trent university   nottingham, united kingdom  email: sue.mcknight@ntu.ac.uk    andrew booth  reader in evidence based information practice  school of health and related research (scharr)   university of sheffield   sheffield, united kingdom  email: a.booth@sheffield.ac.uk     received: 17 dec. 2009           accepted: 21 dec. 2009        2010 mcknight and booth. this is an open access article distributed under the terms of the creative  commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐ nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the  original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is  redistributed under the same or similar license to this one.    as librarians and information professionals  we share a common rationale: to deliver  enhanced services for our customers. the  importance of this is self‐evident ‐ if we don’t  have customers we don’t have a job. we  therefore put our services at peril if we don’t  put the customer at the heart of what we are  trying to do.     the now‐familiar description of evidence  based library and information practice  reminds us that we need “to integrate user‐ reported, practitioner‐observed and research‐ derived evidence as an explicit basis for  decision‐making” (booth, 2006). this begs  several important questions – who are our  users? how can we best capture reports from  these users regarding their expected  outcomes? how might we as library  practitioners observe (and act upon!) what our  users require?    in attempting to answer such questions we  discover potential value in methodologies  with a business orientation; utilising tools  from the commercial sector such as customer  value discovery research (mcknight, 2007a;  mcknight & berrington, 2008). we also need  to recognise that customers don’t come as  “one size fits all”. indeed it behoves us as  library managers to understand this.     rather than attempting to deliver services to a  single “amorphous mass” we need to  recognise that there is no such thing as a  “typical library user” (booth, 2008) and thus to  26 mailto:sue.mcknight@ntu.ac.uk mailto:a.booth@sheffield.ac.uk evidence based library and information practice 2010, 5.1  identify the needs of particular customer  segments. this commentary focuses on the  complex interactions of customer expectations,  staff perceptions and appropriate methods of  inquiry. it reveals that the classically‐focused  evidence based (population‐intervention‐ comparison‐outcome‐[study type]) pico[s]  question (kloda, 2006) faces challenges from  the reality of multiple customers, conflicting  desired outcomes and a bewildering choice of  study methodologies.    delivering a customer‐focus    if we are to put customers at the front of our  services, instilling a true customer‐focused  culture amongst library staff, we must first  identify the various customer segments for  those services. to take as an example the  university sector, with which the authors are  most familiar, some of the customer segments  include those itemised in table 1. such  multiplicity of customers is also true for other  sectors such as public libraries, school libraries  and health libraries. each customer segment  may have different needs or, indeed, may  possess similar needs that they express  differently.    furthermore, we should recognise that such  needs do not remain static. expectations for  service provision constantly change as the  environment, personal experiences of  customers, and technology and other service  delivery options evolve. this poses an  ongoing management challenge to understand  customer expectations, to see what is  happening on the horizon that will impact  customer service and the capacity of the  service to respond to the changes, and, to  engage proactively in service improvements,  i.e., understanding what the customer expects.    needs and expectations of types of different  customers groups are demonstrably different.  our expectations are shaped not only by our  experiences of libraries – if i am located out in  the provinces it is largely irrelevant to me  what library services in the capital look like –  but also by our day‐to‐day experiences of  other services, be they supermarkets,  bookshops or hotels. in a broader context we  recognise that the latest talk within service  industries about “delighting your customers”  is not the complete story (kalikaskis, 2009).  many of us will have experienced the hotel  with the crisp sheets folded back and the  chocolate on the pillow (tisch & weber, 2007)  but will such “delight” stand the test of an  unemptied wastepaper bin or undesirable  detritus in the shower plug‐hole? we therefore  instinctively recognise the intrinsic truth of the  hierarchy of value (mcknight, 2006) (table 2)  – we may be pleasantly surprised by  unanticipated aspects of service provision but  “irritants” have a much more profound effect  in shaping our overall perception of service  quality.  27 table 1   some illustrative customer segments for a university library  staff  students  –teachers  –researchers  –teachers/researchers  –tutors  –administrators  –support staff  –etc  –undergraduates  –1st years  –post graduates –coursework  –post graduate –research  –further education   –part time  –mature age  –school leavers  –off‐campus  –off‐campus international  –on‐campus international  –students with a disability  –those who don’t use the service –etc    evidence based library and information practice 2010, 5.1  table 2   the hierarchy of value (based on mcknight, 2006)    uuunnnaaannntttiiiccciiipppaaattteeeddd   ☺☺☺    dddeeesssiiirrreeeddd      eeexxxpppeeecccttteeeddd         bbbaaasssiiiccc      fffrrruuussst tttrrraaatteeeddd       iiirrrrrri tiitttaaatteeeddd    aaannngggrrryyy      the customer value discovery research  methodology seeks to contextualise the library  service for the customer and seeks the  customers’ perceptions of service excellence.  for example, workshop participants are  prompted by questions to visualise excellence:  as a student you’ve just graduated; or as a  principal investigator you’ve just secured a  new research grant; or as an academic you’ve  just received your professorship… and you’re  down at “the pub” with your mates and you  are telling them that your success is due to the  fabulous library service. they are asked “what  was it about the service that helped you  achieve success?” in this way you are not  leading the customers towards a shortlist of  pre‐specified priorities. instead you are letting  your customers tell you what is important.    recognising that there is not one single  population for our classically focused picos‐ style question allows us to explore differences  between customer segments in a comparative  way. for example students may feel that the  availability and reliability of photocopying  facilities is a key determinant of service  quality. for academic staff, however, such a  consideration may be of lesser importance as  they typically have ready access to such  facilities in their department. in identifying  such differences, we face the practical  decision; upon which target segment will we  focus our energies and resources?     incorporating the staff viewpoint    so much for the complexity of the user  perspective. what though of the “practitioner  observed” component of evidence based  practice? data collected at nottingham trent  university (mcknight & berrington, 2008) has  powerfully demonstrated that library staff  members are seldom as good at anticipating  users’ expectations as they think they are! in  this context it has proved valuable to have  library staff present as observers when users  are prompted to share their expectations of the  service. in this way they engage with the  whole process of quality improvement from  the start, understand why changes are  necessary and have an incentive, and a  commitment, to work on exploring and  devising possible solutions. such participation  does require bravery if one is to listen to frank  responses to a question such as: what really  annoys (irritates) you about this service?    however, once this surprisingly cathartic  process is complete one can move onto more  constructive lines of enquiry such as those for  the “success scenario” mentioned above.  library staff assumptions are challenged by  gap analyses generated as both customers and  staff members vote, in a managed workshop  voting process, on those factors for a quality  service that they think are important to  customers. this helps library staff to “throw  out some myths”, or “make hamburgers of  sacred cows” (booth, 2006), and reveals how  we often make assumptions about what  customers want and how the data shows that  often we can get this wrong! mcknight (2002)  identified that:    involving staff in this customer  research, by seeking their analysis of  the research findings on what adds  value for the customer, by their  28 evidence based library and information practice 2010, 5.1  participation in teams established to  define what change is required within  the organisation to deliver the  customer value package, we create an  internal environment that is not only  ready for change, but which is driving  the change from the ground up, rather  than imposed from management  above.  in this way, there is a much  greater chance of staff “buy‐in” and  the change process is much more  likely to be successful and sustaining.  (p. 268)    it should be recognised, however, that such a  challenging, not to say potentially threatening,  process needs to be accompanied by  appropriate cultural and management  frameworks in which to make it work. it must  be clear to all how findings are to be used in  deciding: what is it possible to deliver or to  remedy? over what time period? with what  impact upon customers perceptions?    the importance of marketing and a customer  focused culture among library staff for  delivering service quality is well‐recognised  (singh, 2009). library staff engagement in the  customer value discovery process and its  consequent actions helps foster this vital  organisational culture. nottingham trent  university captures customer values as  objectives in annual operational plans, with  actions undertaken each year to bridge the gap  between service delivery and excellence as  defined by the customers themselves. by using  customer values in this way, bridging the gap  between “desire and performance” is  embedded within a public statement of  intention.    which methodologies?    this commentary is not intended to suggest  the innate superiority of one methodology  over other alternatives. “one size fits all” is no  more appropriate an assumption for  methodologies of inquiry than it is for  customer segments. however we do need to  be aware that methodologies frequently make  assumptions (e.g. the inclusion of “library as  place”) about what is important to our  customers. we should therefore resist the  tendency to be too high‐minded about the  academic rigour of methodologies. it is far  more important to understand why you need  the evidence and how you are going to use it.  a methodology for exploring customer needs  is only as good as the questions you ask. if you  ask the wrong questions you inevitably get the  wrong answers.    for example, standard satisfaction  methodologies used nationally by universities  may reveal that customers are very happy  with the services that a library provides. at  the same time comparison with findings from  the customer value discovery methodology  may reveal that that same library is only  delivering at 50% against value propositions  (mcknight, 2008). satisfaction is not only  about service delivery it is also about  expectations. if customer expectations increase  yet service provision remains the same,  satisfaction will invariably go down.  conversely if customer expectations remain  low, rates of satisfaction may appear  consistently high. the customers may simply  be grateful that they have a service at all!     a related point, that the delighters or “wow  services” of today easily become the basic  taken‐for‐granted components of tomorrow’s  services, is well‐illustrated within the familiar  context of electronic journals. in 1999 the  provision of digital full‐text services by an  academic library was widely regarded as  being a “wow” factor. six years later, in 2005,  digital full text journals were expected. they  no longer excited nor warranted a perception  of excellence for a library service. indeed one  could go further and say that if they were not  provided this “irritant” would seriously annoy  an academic library customer    an increased awareness of the complexity of  such interactions has helped library staff to  realise that achieving satisfaction is not the  end‐game. libraries cannot, and should not,  rely upon customer satisfaction alone!  whatever methodology they use must also  factor in the pivotal role of customer  29 evidence based library and information practice 2010, 5.1  expectations. only such a perspective can help  library staff appreciate why, despite  considerable investment in library services  and against an ever‐shifting backdrop of  increasing customer expectations, student  ratings of satisfaction remain surprisingly  constant.    conclusion    such detailed exploration of customer  expectations is closely allied to “narrative  based librarianship” as described by brophy  (2004, 2007). library staff observe, and then  actively participate in, the telling of stories  within the library service; the linking of  actions to the defined values and irritants of  customers; the positioning required for a  change of culture placing the customer at the  centre of the library service. they begin to  acknowledge that customer experiences and  perceptions are their reality (mcknight, 2007b).  customers’ narratives thus become powerful  enablers in imbuing hard data with context,  significance and meaning to inform library  decisions on the development of appropriate  services (brophy, 2004).     by embedding within the organisation an  appropriate management framework that  requires customer consultation, staff  engagement, and constant feedback, evidence  is gathered to inform decision‐making. it is  thus feasible “to integrate user‐reported,  practitioner‐observed and research‐derived  evidence as an explicit basis for decision‐ making” (booth, 2006). furthermore this  approach provides the basis for two‐way  communication between customers and  stakeholders and the service provider. success  breeds success…and we will never become  either bored or complacent!    implications for practice    • focusing on customer expectations  allows library managers to identify  “low hanging fruit” and more  challenging targets upon which to  focus their energies and resources.  • customers compare a library service  with their life experiences, not just  other library services, so managers  need to broaden their horizons when  considering how to improve customer  experiences.   • reconsidering established library  services against the backdrop of  customer expectations helps to  identify “sacred cows” and get rid of  them.  • purposeful inquiry must be  accompanied by a feasible and  realistic programme of change ‐ if you  don’t make changes you have  expended an awful lot of time and  energy for no reason.    implications for research    • investigation of customer satisfaction  is not sufficient if a mechanism for  factoring in changing customer  expectations has not been identified.   • those using standard methods for  exploring library service quality must  be cognisant, and wary, of implicit  assumptions built into most of the  existing methodologies and  instruments.    acknowledgement    this paper is based on a keynote address by  susan mcknight at the 5th international  evidence based library and information  practice conference, in stockholm, sweden,  july 2009. a full paper reporting susan  mcknight’s work will be available in  “bridging the gap between service provision  and customer expectations.” performance  measurement & metrics, (in press).      references   booth, a. (2006). counting what counts:  performance measurement and  evidence‐based practice. performance  measurement and metrics 7, 63–74.  30 evidence based library and information practice 2010, 5.1  31 booth, a. (2008). in search of the mythical  ʹtypical library userʹ. health information  & libraries journal, 25, 233‐236.  brophy, p. (2004). narrative‐based  librarianship. studies in library and  information science, 10, 188‐195.  brophy, p. (2006). measuring library  performance: principles and techniques.  london, england: facet publishing.  brophy, p. (2007). narrative based practice.  evidence based library and information  practice, 2(1), 149‐157. retrieved 20  feb. 2010 from  http://ejournals.library.ualberta.ca/ind ex.php/eblip/article/view/137/248  kalikaskis, a. (2009). delight your customers:  your personal power to make a  difference. retrieved 20 feb. 2010  from  http://www.sabinet.co.za/conference2 008/presentations/delight‐your‐ customers.pdf  kloda, l. (2008). asking the right question.  evidence based library and information  practice, 3(4), 79‐81.  retrieved 20 feb.  2010 from  http://ejournals.library.ualberta.ca/ind ex.php/eblip/article/view/4426/3725  mcknight, s. (2002). managing cultural  change: the challenge of merging  library services, curriculum  development and academic  professional development. ifla  journal, 28(5‐6), 266–272.  mcknight, s. (2006). customer value research.  in. t. kolderup flaten & k.g. saur  (eds.), management, marketing and  promotion of library services based on  statistics, analyses and evaluation (pp.  206‐216). munchen: ifla publications.  mcknight, s. (2007a). acquisition and  cataloguing processes: changes as a  result of customer value discovery.  evidence based library and information  practice, 2(4), 22‐35. retrieved 20 feb.  2010 from  http://ejournals.library.ualberta.ca/ind ex.php/eblip/article/view/521/667  mcknight, s. (2007b).  the expatriate library  director. library management, 28, 231‐ 241.    mcknight s., & berrington, m. (2008).  improving customer satisfaction:  changes as a result of customer value  discovery. evidence based library and  information practice. 3(1), 33‐52.  retrieved 20 feb. 2010 from  http://ejournals.library.ualberta.ca/ind ex.php/eblip/article/view/920/1074    mcknight, s. (2008). are there common  academic library customer values?  library management, 29, 600‐619.    mcknight, s. (2009) bridging the gap between  service provision and customer  expectations. performance measurement  & metrics, 10, 79‐93.    singh, r. (2009). mind the gap: unlocking the  relationship between market‐ orientation and service performance.  library review, 58, 28‐43.    tisch, j. m., & weber, k. (2007). chocolates on  the pillow arenʹt enough: reinventing the  customer experience. chichester: john  wiley.  evidence summary   academic library use is positively related to a variety of educational outcomes   a review of: soria, k. m., fransen, j., & nackerud, s. (2017). beyond books: the extended academic benefits of library use for first-year college students. college & research libraries, 78(1), 8-22. https://doi.org/10.5860/crl.78.1.8     reviewed by: rachel e. scott ils librarian university libraries the university of memphis memphis, tennessee, united states of america email: rescott3@memphis.edu   received: 10 may 2019                                                                  accepted:  14 july 2019      2019 scott. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29583     abstract   objectives – to consider the relationship between academic library use and four specific outcomes: academic engagement, engagement in scholarly activities, academic skills development, and grade point average.   design – hierarchical regression analysis.   setting – a large, public research university in the midwest us.   subjects – 1,068 non-transfer, first-year students who voluntarily completed the student experience in the research university (seru) survey.   methods – the seru survey results were analyzed alongside student data derived from institutional records and 10 library usage variables generated from library systems. velicer’s minimum average partial (map) method was employed to develop a factor analysis. hierarchical regression analyses measured the relationships between independent variables (demographic characteristics, collegiate experiences, and libraries use) and dependent variables (students’ academic engagement, academic skills, engagement in scholarship, and fall semester grade point average).   main results – students’ use of academic libraries was reported to have a positive relationship with all four dependent variables, above and beyond those explained by pre-college and collegiate experiences: academic engagement (r2∆= .130, p < 0.001), academic skills development (r2∆= .025, p < 0.001), fall semester grade point average (r2∆= .018, p < 0.001), and engagement in scholarship (r2∆= .070, p < 0.001). use of books and web-based library resources had the most positive relationships with academic outcomes; workshop attendance and use of reference services had limited positive relationships with academic outcomes; and use of library computer workstations had no significant effects on academic outcomes.   conclusion – undergraduate student use of the academic library is positively associated with diverse academic outcomes. although the explanatory power of library use was relatively low, ranging from 1.8 to 13.0 percent of final variance in the dependent variables, library use is nonetheless reported to contribute significantly to academic outcomes.   commentary   this study builds on existing studies that measure the impact of library use on the academic success of undergraduate students, including similar studies on the topic by these researchers, two of which focus on the relationship between library use, student grade point average, and retention (soria, fransen, & nackerud 2013 & 2014). this study however, is unique in its analysis of the relationships between a variety of library usage metrics and academic success metrics beyond grade point average and retention, specifically academic engagement, engagement in scholarly activities, and academic skills development. this study dovetails current work on library involvement in student privacy and learning analytics, such as the institute of museum and library services-funded “datadoubles” project.   the authors cite relevant literature to convey the increasing need for accountability within higher education and libraries. astin’s (1993) input-environment-output model from what matters in college is employed as a conceptual framework and is used to control for inputs, such as student demographic characteristics and college experiences, which may contribute to academic success.   the evaluation methods are appropriate to the objectives and required statistical expertise that may serve as an obstacle to librarians who do not know how to conduct or interpret a hierarchical regression analysis. it is worth noting that the study is co-written by librarians and an employee of the office of institutional research; this collaboration would likely prove essential to academic librarians seeking to recreate elements of the study. student data is frequently unavailable to academic librarians, particularly at the aggregate level. the authors note limitations with their sample, the most problematic of which is that those students who completed the survey are likely more engaged than their peers. additionally, the reliability of the blocks of self-report information are presented, but the reader does not learn how the reliability was tested.   the authors meet their objective of measuring the relationship between academic library use and academic engagement, engagement in scholarly activities, and academic skills development; this study quantifies several ways in which the library plays an active role in the academic success of undergraduate students. by identifying a variety of standard library services that may be assessed and by detailing how to collaborate with campus partners to measure this impact, this article makes a significant contribution to the literature. future studies might make explicit how the privacy of student data is considered and protected in this process.   references astin, a. (1993). what matters in college?: four critical years revisited. san francisco, ca: josey-bass.   soria, k. m., fransen, j., & nackerud, s. (2013). library use and undergraduate student outcomes: new evidence for students' retention and academic success. portal: libraries and the academy, 13(2), 147-164. https://doi.org/10.1353/pla.2013.0010   soria, k. m., fransen, j., & nackerud, s. (2014). stacks, serials, search engines, and students' success: first-year undergraduate students' library use, academic achievement, and retention. the journal of academic librarianship, 40(1), 84-91. https://doi.org/10.1016/j.acalib.2013.12.002   soria, k. m., fransen, j., & nackerud, s. (2017). beyond books: the extended academic benefits of library use for first-year college students. college & research libraries, 78(1), 8-22. https://doi.org/10.5860/crl.78.1.8   evidence summary   embedded librarianship is not well understood by librarians at chinese universities, but represents a promising service model   a review of: sun, h., liu, y., wang, z., & zuo, w. (2019). embedded librarianship in china: based on a survey of university libraries. the library quarterly, 89(1), 53–66. https://doi.org/10.1086/700663   reviewed by: judith logan assistant head, user services, john p. robarts library university of toronto libraries toronto, ontario, canada email: judith.logan@utoronto.ca   received: 24 feb. 2020                                                              accepted:  30 mar. 2020      2020 logan. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29742     abstract   objective – to determine the extent to which embedded librarianship is understood and implemented with a focus on service models, best practices, and barriers.   design – survey questionnaire with follow up interviews.   setting – provincial and ministerial university libraries in china.   subjects – subject or liaison librarians from the 84 institutions with science and technology “information searching and evaluation centres” called s&tns (p. 56).   methods – the authors identified potential participants through the eligible institutions’ library websites or by contacting the library’s managers. then they randomly selected three librarians (n = 252) from each library to be invited to participate. 56 responded from 41 unique institutions. when respondents indicated that their library had embedded library services, the authors contacted them for follow up interviews.   main results – results of the questionnaire revealed that most respondents were unclear about the concept of embedded librarianship with many mistaking traditional models of librarianship as embedded. roughly half (n = 21) of respondents reported embedded librarians at their institution.   follow up interviews revealed five models of embeddedness: (1) subject librarianship, (2) teaching information retrieval or library orientation sessions, (3) participation in research teams, (4) co-location with academic departments, and (5) assisting university administration with decision-making. only half of these libraries (n = 11) conducted some form of assessment.   conclusion – embedded librarianship is a promising, but not yet widely adopted model in chinese university libraries. more should be done to advocate for its implementation or libraries risk obsolescence.   commentary   this study describes a situation that will be familiar to many academic librarians in north america. the information landscape has changed significantly in the past three decades, necessitating a transformation of traditional library services. in many cases, this transformation has been slow and often stalled at what sun et al. (2019) call a “first generation” embedded service: liaison librarians (p. 63). in recent years, the association of research libraries (arl) has been exploring the current state of liaison librarianship through a series of two-day liaison institutes with member libraries. a summary of these institutes echoes sun et al.’s (2019) conclusions: “[participants] struggled to find value in aspects of traditional services, but had little appetite for serious reconsideration of services that may have lost all or most of their value relative to the time and energy expended to deliver them” (vine, 2018, p. 422). academic libraries may want to be more embedded, but are unsure of what that might mean and afraid to let go of current practice. sun et al.’s (2019) work demonstrates that chinese libraries face similar struggles.   authors of online questionnaires should consider using eysenbach’s (2004) checklist for reporting results of internet e-surveys (cherries) when composing their manuscripts to improve the comprehensiveness of their reporting. while the authors helpfully included their survey instrument in an appendix, several elements were missing from the cherries survey administration, response rates, preventing multiple entries from the same individual, and analysis sections which makes it difficult for readers to appraise the study critically using a tool such as glynn’s (2006). similarly, a tool such as the consolidated criteria for reporting qualitative research (coreq) could be used to describe qualitative work such as follow up interviews (tong et al., 2007).  for example, the authors of the present study did not mention how they analyzed the qualitative data collected during their follow up interviews.   despite this, the study is useful as an exploration of embedded library practices in china. as this topic has been the subject of many recent publications in the chinese library literature (sun et al., 2019, p. 62), it is clear that there is growing interest in embedded librarianship. practitioners can use it as an advocacy tool to promote the model. the authors have included several ideas for what would be needed to make this a reality including changing reward systems within libraries.   references   eysenbach, g. (2004). improving the quality of web surveys: the checklist for reporting results of internet e-surveys (cherries). journal of medical internet research, 6(3), e34. https://doi.org/10.2196/jmir.6.3.e34   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387–399. https://doi.org/10.1108/07378830610692154   tong, a., sainsbury, p., & craig, j. (2007). consolidated criteria for reporting qualitative research (coreq): a 32-item checklist for interviews and focus groups. international journal for quality in health care, 19(6), 349–357. https://doi.org/10.1093/intqhc/mzm042   vine, r. (2018). realigning liaison with university priorities: observations from arl liaison institutes 2015–18. college & research libraries news, 79(8), 420–424. https://doi.org/10.5860/crln.79.8.420     microsoft word es_kloda.doc evidence based library and information practice 2007, 2:4  67 evidence based library and information practice       evidence summary    skills gained from university library instruction sessions are perceived as useful  four to eight weeks later      a review of:  wong, gabrielle, diana chan, and sam chu. “assessing the enduring impact of library  instruction programs.” journal of academic librarianship 32.4 (july 2006): 384‐95.      reviewed by:   lorie a. kloda  phd student, school of information studies, mcgill university  montreal, quebec, canada  e‐mail: lorie.kloda@mcgill.ca    received: 05 september 2007     accepted: 20 september 2007      © 2007 kloda. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to assess the impact of a  university library instruction program.    design – survey questionnaire  administered post‐intervention.    setting – a mid‐size science and technology  university in hong kong.    subjects – student and staff participants in  either course specific or open (elective)  library instruction workshops.    methods – surveys were conducted to  measure the perceived effectiveness of the  library instruction program, including  various types of course specific (cs) and  open workshops (ow). librarians  responsible for teaching nominated the  sample of workshops for evaluation.  students in all but one cs workshop were  provided with a 14‐question paper  questionnaire in class by their course  instructor, while participants in all of the  open workshops and one cs workshop  received the same questionnaire via e‐mail.  the questionnaires were distributed  between four to eight weeks following the  workshops in order to gauge the “enduring”  impact of the instruction. most questions  were closed, forcing participants to choose  an answer from a list or select from a 4‐ or 7‐ point likert scale. comments were also  solicited. results were summarised and  analysed using spss software. the cs and  ow questionnaires were studied separately  to allow for comparisons between groups.    mailto:kloda@mcgill.ca http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:4  68 main results – out of 133 workshops taught  in the fall of 2004, 25 were included in the  sample: 15 cs and 10 ow. the overall  response rate was 68%, with 466 participants  completing questionnaires. most  participants indicated that the workshops  were useful for learning about sources and  search methods for finding information  quickly. the majority (72.2%) responded  that they felt an increase in confidence when  conducting library research and slightly  more than half (57.9%) agreed the  workshops led to an increased interest in  using the library. the responses differed  significantly for the cs and ow groups: ow  participants consistently rated the  usefulness of the workshops higher than cs  participants. in regards to retention of skills,  68.5% of participants responded in the  affirmative when asked of they had  continued using the skills taught, with rates  ranging from 56 to 83% depending on the  workshop. there was little difference in  perceived retention between the cs and ow  groups. the skills most frequently identified  as having been learned included the abilities  to “form better search strategies” and “find  better internet resources.” written feedback  included remarks on reducing class size and  length, and increasing practice time and the  number of handouts.    conclusion – a “delayed perception  survey” revealed positive feedback from  library workshop participants on questions  about confidence, usefulness, and retention  of skills learned. there was a significant  difference in confidence levels reported  between cs and ow groups, with ow  participants reporting higher levels of  confidence. the researchers surmise this  might be a result of self‐selection, as ow  participants volunteered both to attend the  library instruction workshops and to  respond to the survey questionnaire. the  short questionnaire is an efficient tool for  assessing the perceived usefulness of library  workshops for both course‐integrated  sessions and elective workshops.     commentary    this study attempts to assess the quality of a  university library’s instruction program in a  given semester. assessment on such a large  scale is a difficult undertaking, especially  considering the lack of validated  standardised assessment tools available to  librarians. the authors selected a reasonable  method, the perception survey, to conduct  their assessment. their questionnaire is also  made available in the publication, thus  enabling others to adapt it for their own use.  while this study makes a contribution to  assessment studies of library instruction  programs at institutions of higher education,  it falls short in several areas.    the convenience sample was not  randomised, making it difficult to assume  representativeness. in addition, there is no  indication that the results of the survey are  normally distributed, and that the 2 groups  exhibit homogeneity of variance –  requirements for parametric testing such as  the t‐test employed. it is therefore  questionable as to whether the results have  any statistical significance. also typical of  many assessment studies is the absence of a  control group or testing prior to the  intervention. this also makes the results less  conclusive. the article omits an explanation  as to how class size was determined, placing  into question any conclusions on the impact  of class size on retention. it is also important  to keep in mind the survey did not assess  actual learning, but the perception of such  learning. while the authors state that the  survey assessed the “enduring impact” of  the instruction program, further assessment  is required to conclude that any long‐term  learning has taken place beyond the four to  eight week period.     evidence based library and information practice 2007, 2:4  69 assessment initiatives are rampant in lis,  particularly for information literacy  programs at the university level. librarians  are asked to justify resources spent on such  programs to administrators, who in turn use  the quantitative data as a means of  comparison with other teaching and for  decision making. it is understandable that  universities wish to make comparisons  across the institution using standardised  measures, such as the 7‐point likert scale for  evaluating instructors used in this study.  the end, however, does not necessarily  justify the means. students and other  workshop participants should not be forced  to switch from 4‐point to 7‐point likert  scales in order to accommodate bureaucratic  requirements. to assess library instruction  at the programmatic level, a more complex  design is required, one that ideally involves  qualitative methods and multiple data  sources. in addition to workshop  participants, course instructors and  librarians should also be interviewed  individually or in groups to provide richer  data. the authors of this study attempted to  include some qualitative data by leaving  space for comments on the survey, and as a  result received some very useful feedback.  a follow‐up study delving further into these  comments would be very welcome in a  literature already overflowing with numbers.     editorial   reflecting on six years as editor-in-chief   lorie kloda editor-in-chief associate university librarian, planning and community relations concordia university montreal, quebec, canada email: lorie.kloda@concordia.ca      2020 kloda. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29894     this is my last editorial as editor-in-chief of evidence based library and information practice (eblip). the month of december 2020 marks the end of my second term, after taking on the role six years ago. ann medaille will be taking over the position after serving as associate editor (research articles) for the last three years, and before that as a peer reviewer and member of the evidence summaries writing team. so much has happened during the previous six years for eblip, and yet in many ways it remains the same journal as when it was first launched.   most recently, the journal welcomed new members of the editorial team. kimberly mackenzie, formerly the editorial intern, has stepped into the role of communications officer. melissa cober is the new editorial intern. and to fill the role of one of two associate editors (research articles) vacated by ann is erin owens.   i have been affiliated with the journal since its inception 16 years ago. i began as a member of the evidence summary writing team, publishing my first summary in the first issue. eventually, i was appointed to the role of associate editor for evidence summaries and then for research articles, before taking on the role of editor-in-chief. the journal has been an important part of my career over the last decade and half, not only as part of my professional identity but as a source of information and community. the evidence based library and information practice framework is embedded in my practice as a librarian, and eblip has and will continue to influence me.   during my tenure as editor-in-chief, the journal has made several significant achievements: digital object identifiers (dois) were implemented for every paper, a data sharing policy was launched, and a new practice to acknowledge all contributor roles for original research is coming in volume 16. the journal published a special issue in 2016 compiling the ebl 101 column (https://journals.library.ualberta.ca/eblip/index.php/eblip/issue/view/1571), and to honour our 10th anniversary, issue 11(1) included a series of commentaries from past and current members of the editorial team.   in 2019, the journal received its first influx of funding from the social sciences and humanities research council (sshrc) of canada aid to scholarly journals award. this funding, which lasts for three years, allowed eblip to hire a part-time editorial assistant, samantha sheplawy. the funding will also allow the journal to undertake some exciting projects in the next couple of years.   this issue marks the completion of the first 15 years of eblip – that’s 61 issues (including the special issue) – an impressive run for any journal, and in particular for one that is grassroots,  open access, and edited by volunteers. i leave the journal in the competent hands of ann medaille and the entire editorial team, the evidence summary writing team, the peer reviewers, and you, the readers. i am confident that this journal will continue for another 15 years and look forward to seeing it evolve.   microsoft word news_needham.doc evidence based library and information practice 2007, 2:3  132 evidence based library and information practice     news    first international m‐libraries conference 13th – 14th november 2007      gillian needham  head of strategic and service development  the open university library and learning resources centre  milton keynes uk  e‐mail: g.needham@open.ac.uk      © 2007 needham. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    the open university (uk) and athabasca  university (canada) are hosting the very  first international conference to focus on the  delivery of library services and resources to  users via mobile and handheld devices. this  event on november 13th and 14th at the open  university in milton keynes will bring  together researchers, managers and  practitioners from around the world who  are interested in extending the value and  flexibility of libraries in new and exciting  ways.     for more details and to register, please visit  the website at:  http://library.open.ac.uk/mlibraries/.  mailto:needham@open.ac.uk http://creativecommons.org/licenses/by/2.0 http://library.open.ac.uk/mlibraries essential academic journals tend to be of universal importance, while many journals available on for-profit platforms appear to be ancillary evidence summary   essential academic journals tend to be of universal importance, while many journals available on for-profit platforms appear to be ancillary   a review of: mongeon, p., siler, k., archambault, a., sugimoto, c. r., & larivière, v. (2021). collection development in the era of big deals. college & research libraries, 82(2), 219–236. https://doi.org/10.5860/crl.82.2.219   reviewed by: michelle dubroy discipline librarian, researcher services griffith university library southport, queensland, australia email: m.dubroy@griffith.edu.au   received: 29 aug. 2021                                                             accepted:  12 oct. 2021      2021 dubroy. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30025     abstract   objective – (1) present a method of journal appraisal that combines reference list, article download, and survey data. (2) gauge journal usage patterns across selected universities.   design – analysis of reference lists, article downloads, and survey data.   setting – 28 canadian universities.   subjects – 47,012 distinct academic journal titles.   methods – download data for the 2011-2015 period was sourced from standard journal report 1 (jr1) usage reports as supplied by the vendors. download figures were summed for journals that were available through multiple platforms. reference list data (i.e., the number of times documents published in each journal were cited by authors affiliated with a participating institution) was sourced from clarivate analytics’ web of science, limiting for the years 2011-2015. an unknown number of researchers at 23 of the 28 participating universities were invited by email to complete a survey. the survey asked respondents to list the scholarly journals they considered essential for their research and teaching (up to 10 journals for each purpose).   the three datasets (download, reference list, and survey data) were then merged. duplicates and non-academic journals were removed. journals were then grouped into broad discipline areas. a list of “core journals” (p. 228) was created for each institution. these journals produce 80% of downloads, 80% of citations, or 80% of survey mentions at each institution. a journal only had to reach the threshold in one category (i.e., in either downloads, citations, or mentions) to make it onto the core journals list. a “low” (p. 228) survey response rate meant “one mention [was] generally enough" (p. 228) for a journal to be classified as core.   main results – fewer than 500 titles (n=484, ~1%) made it to the core journals list at all 28 universities. two thirds (66%, n unknown) of journals did not make it onto the core list of any university. of the journals deemed to be core, most (60%, n unknown) were shared across all institutions. on average, platforms from not-for-profit organizations and scientific societies contain a higher proportion of core journals than for-profit platforms. notably, 63.6% of springer journals, 58.9% of taylor & francis journals, and 45.8% of elsevier’s journals do not appear on the core journal list of any university.   conclusion – libraries should consider ways to share resources and work more cooperatively in their negotiations with publishers. further, libraries may be able to cancel entire journal bundles without this having a “sizable” (p. 233) impact on resource access.   commentary   this study adds to the growing body of literature examining the value of big deals (shu et al., 2018; strieb & blixrud, 2014). big deals, or bundled journal subscriptions with major publishers, are said to be financially unsustainable due to rising costs and limited budgets (mckenzie, 2018).   the study was reviewed using a critical appraisal tool (perryman, 2009). both strengths and weaknesses were found.   a concise literature review outlines the context and rationale for the study. and, while the study is quite complex, the authors have reported their findings logically, making effective use of tables and figures. further, the triangulation of reference list, download, and survey data was innovative and valuable.   the authors have provided their survey questions in full, allowing future researchers to replicate this work. regrettably, there is a lack of clarity around how the researchers retrieved reference list data. readers do not know whether the researchers performed an address or organizations-enhanced search when retrieving the articles.   more importantly, the authors have not disclosed how many of the journals analyzed were in fact part of a large journal bundle. this is odd considering the article’s title and bold claims made around the effectiveness of big deals. notably, just because a journal is available on a publisher’s platform does not necessarily mean it was acquired through a big deal.   the authors’ suggestion that “a complete cancellation of [an entire] journal bundle would not have sizable effects on access to relevant resources” (p. 233) is questionable. libraries need to provide access to more than a core list of resources based on an arbitrary threshold. the needs of faculty and students are often diverse. the fact that so many journals (66%, n unknown) did not make it to the core list of any institution calls into question the soundness of the 80% threshold used in the analysis. it does not necessarily mean those journals are not important to clients or that they do not offer value for money.   this study would be of interest to any library comprehensively reviewing its journal subscriptions. the triangulation of download, reference list, and survey data is particularly compelling. the suggestion that libraries continue to join forces to negotiate better deals with publishers is also worthwhile.   the study’s usefulness is limited, however, because cost and value for money were not considered. in practice, libraries cannot ignore the financial implications of collection decisions. they also cannot ignore the needs of the diverse communities they serve.   references   mckenzie, l. (2018, may 8). ‘big deal’ cancellations gain momentum. inside higher ed. https://www.insidehighered.com/news/2018/05/08/more-institutions-consider-ending-their-big-deals-publishers   perryman, c. (2009). critical appraisal tool for bibliometric studies. retrieved from https://www.dropbox.com/l/scl/aaal7luzple90fxfnbv5hcnoz0ctlh6rqrs   shu, f., mongeon, p., haustein, s., siler, k., alperin, j. p., & larivière, v. (2018). is it such a big deal? on the cost of journal use in the digital era. college & research libraries, 79(6), 785–798. https://doi.org/10.5860/crl.79.6.785   strieb, k. l., & blixrud, j. c. (2014). unwrapping the bundle: an examination of research libraries and the “big deal.” portal: libraries and the academy, 14(4), 587–615. https://doi.org/10.1353/pla.2014.0027   evidence based library and information practice using evidence in practice   a comeback library: re-establishing a library presence in a diverse regional campus   isabel vargas ochoa stockton campus & web services librarian california state university, stanislaus stockton, california, united sates of america email: ivargas2@csustan.edu   received: 23 aug. 2022                                                            accepted: 7 dec. 2022      2023 vargas ochoa. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30230     setting   this article describes the use of evidence to inform the development and growth of library services on a regional campus in the diverse city of stockton, california. the stockton campus library (scl) at the stockton campus, a regional campus of the california state university (csu), stanislaus, serves nearly 12% of the school’s 8,847 fte student population, and serves 20 full-time stockton based faculty (california state university, stanislaus [csu], 2021). csu stanislaus is a hispanic-serving and minority-majority university with ethnic minorities making up 70% of total enrollment and three-fourths of our undergraduates are first-generation students. out of the stockton students enrolled in fall 2021, 48% are hispanic/latino, 17% are asian, 8% are black/african american, and 24% are first-generation students (csu, stanislaus, 2021).   the scl strives to offer the same services that are provided at the main campus located in the city of turlock. the scl is committed to providing equitable access to all stanislaus state students, faculty, and staff. the j. burton vasché library at the turlock campus is a newly renovated building featuring 3 floors, over 380,000 physical books, 19 study rooms, and more than 1,000 study seats. the scl on the stockton campus currently occupies a 1,400 sq. ft. suite and offers 300 print books and 56 study seats. the re-establishment and expansion of the scl is an ambitious goal, and reaching the current stage is an encouraging accomplishment.   the scl was established in 1999 after the university formally founded the regional campus in stockton in 1998. the first stockton library was named stockton library access center (lac) and was allotted a suite of 2,700 sq. ft., funding for a print collection, a full-time staff member, and a librarian. the stockton lac flourished for almost a decade, providing full library services to the growing stockton campus community. the direction of the stockton campus took a sharp turn when the united states economy was hit by the 2008 housing market crisis. under the pressure of a sharply reduced budget, the university decided to partially close the stockton campus. the lac became primarily a remote service as onsite staff were re-directed to the main campus. in 2012, when the university was attempting to establish self-support programs and a well-situated library space at stockton, the lac was moved to an 840 sq. ft. room. it took almost another decade for the university to revisit the potential of the stockton campus. in 2020, the stockton campus library returned to a 1,400 sq. ft. library suite and opened in person services in fall 2021.   problem   when the scl opened in fall 2021, we were under specific safety regulations due to the covid-19 pandemic which made offering in-person services challenging. however, our goal was to offer equitable access despite these challenges. in describing a post-pandemic future for library services, dave shumaker (2021) has explained that “effort to overcome unequal access to information and technology will continue to be a top priority” (p. 15). to offer equal access to virtual and in-person library services that the turlock campus offers, the scl, as well as many other university departments, needed additional funding. however, there was a decline in student enrollment throughout the university, at both turlock and stockton campuses (csu, stanislaus, 2021), which impacted available funding. for the scl, funding was an issue even before the pandemic. now as we return to in-person services, scl is fighting to stay present and to continue expanding while working with limited resources.   to start re-establishing the new space and library, i assessed the design and structure of the library space to better fit the needs of the stockton students and the services to be implemented. deciding where elements of a new library space would go prompted anticipated challenges. a complex obstacle for library managers in similar situations, is to “plan for a new or renovated library space,” and determine “how to incorporate as much flexibility as possible,” (p. 410) as mary augusta thomas (2000) explained in her article on redefining library spaces. flexibility in services and design relies heavily on the needs of the community that the library serves. the challenges for a library re-design and transformation increase when there is a shortage in personnel and funding. currently, the scl has one full-time librarian hired in spring 2021, and one full-time library staff hired in spring 2022.   priorities for the library also align heavily with traditional library services: access to books, study spaces, computers and printing, and research or reference help. a front desk for check-outs and research help was necessary. study space and computers and printing were other necessities when designing the space. because the scl re-opened with no browsable print collection, building a library collection was fundamental as well.   considering the limited space in the library suite, it was crucial to document the necessary programs and services the library wanted to offer to students. this also meant surveying the stockton campus community, preferably those primarily serviced, like students and faculty, and aggregating their responses.   connecting with the community at stockton is the foundation to building the library at stockton and implementing services. after i was hired as the stockton campus librarian, my objective was to build a print collection, since the initial collection was lost after the recession in 2008, and to design services for the opening in fall 2021. a survey was conducted, and students and faculty were invited to participate.   evidence   initial connection with the campus community: surveys   surveys were created and shared with stockton faculty in april 2021, and with stockton students in may 2021, to aid in the planning and implementation of in-person library services for the start of the fall semester in august 2021. emails were distributed to both parties with a survey deadline included, and an incentive (gift card) was offered to students who completed the survey.   both faculty and students were offered separate surveys. they were given four weeks to complete the survey, and for the students, to also be considered for the gift card drawing. in total 11 stockton based faculty and 83 stockton students participated in the survey. the following were the questions asked in the student survey:   what is your major? (text input) what is your academic status? (radio multiple choice: undergraduate student; graduate student) prioritize the times below for when you would like to visit the scl (ranking choice: 8 a.m.-12 p.m. mornings; 12 p.m.-5 p.m. afternoons; 5 p.m.-9 p.m. evenings, saturdays, sundays) in what ways do you prefer to get research help? (checkmark multiple choice: walk-in front desk, in-person appointment, email, zoom or virtual appointment, phone, text, other) prioritize time blocks to get research help, from most crucial to least crucial: (ranking choice: 8 a.m.-12 p.m. mornings; 12 p.m.-5 p.m. afternoons; 5 p.m.-9 p.m. evenings) what services are you looking forward to at the scl? (checkmark choice: checking out books; using the computers and printing; checking out course materials; study spaces and areas; checking out laptops; research help) do you have any questions or suggestions? is there anything you’d like the scl or scl librarian to know? (text input) if you’d like to enter to win one of the (3) $25 amazon gift cards, please provide your email.   the following were the questions asked in the faculty survey:   what is your email? (text input) what is your department? (text input) what types of materials do you wish to see at the scl? (checkmark multiple choice: books; course reserves; journals/magazines; dvd’s; music cd’s; other) what subject materials would like you like to see at the scl? (checkmark multiple choice: [multiple subjects were listed as options]) do you have any questions/suggestions regarding collection development for the scl? (text input) due to the pandemic, this spring semester, we are offering instruction sessions virtually only. in fall 2021, we plan to offer face-to-face instruction sessions as well. if you were to request a research instruction session, which method do you prefer for the fall? (radio multiple choice: virtual (asynchronous/synchronous); in-person/face-to-face; both; i’m not sure yet/depends on campus planning; i don’t know what research instruction sessions are) prioritize the crucial time blocks for the scl to be staffed and open, from most crucial to least crucial: (ranking choice: 8 a.m.-12 p.m. mornings; 12 p.m.-5 p.m. afternoons; 5 p.m.-9 p.m. evenings, saturdays, sundays) is there anything you’d like the scl or scl librarian to know? (text input)   student survey: results   the majority of stockton students who participated in the survey were psychology, nursing, business, liberal studies, social sciences, and criminal justice majors. 82% of the student participants were undergraduate students, and 18% were graduate students.   figure 1 illustrates students’ preference to visit the scl during the afternoon shifts. many stockton and distance learning students work during the day, thus most courses are taught in the evenings. as expected, students would also like the library to be open in the evenings. similarly, students are looking forward to receiving research help in the afternoons (figure 2). student preference was split when it came to how they would like to receive research help from the librarian, as seen in figure 3. most students preferred walk-in research help, followed respectively by email, in-person appointment, and a virtual appointment.   figure 1 student survey: time visitation preferences.   study spaces and computers and printing were unsurprisingly the most requested services for the scl to offer. students were primarily interested in having a study space or area in which they can utilize wi-fi, research, and print course assignments and papers (figure 4), although it is also important to note that students were also looking forward to checking out books as well.   when students were asked if they had any questions or suggestions, most asked for study spaces, including reservable study rooms, and for extended opening hours.   figure 2 student survey: time preferences for getting research help.   figure 3 student survey: preferred methods of getting research help.   figure 4 student survey: services students are looking forward to.   faculty survey: results   similar to students, stockton faculty chose afternoons as the preferred time for the library to be open, followed closely by evenings, then mornings (figure 5). faculty also preferred weekend hours, as opposed to students who did not rank the weekends so highly.   faculty were asked to provide the scl with feedback regarding collection development and materials they would like to see at the library. there was a general response from stockton faculty wanting to see literature in all subjects offered at the university at the scl, including a reference section and a children’s book collection.   figure 5 faculty survey: time preferences for library hours.   implementation   faculty and students were eager to have a space to study and a print collection with literature relevant to their major programs and departments. gathering from the survey’s open-ended feedback, students and faculty expressed gratitude for an on-site full-time librarian to conduct research help and offer information literacy instruction. students and faculty were also eager for study spaces, a library collection, and wi-fi-access, as well as computers and printing.   these results helped us design the library space and services in our preparation for our re-opening in fall 2021, the first time in-person services were offered at the scl since the pandemic. the survey results also serve as proof that the scl needs consistent and unwavering attention and funding. offering the same services and materials that are part of larger university libraries, like print collections, study spaces, and computers, is the necessary initial step to rebuilding and connecting with the community in a regional campus.   we planned our library hours based on the feedback received from the survey. although weekend hours were not offered due to staffing, in fall 2021 the library opened from 9 a.m.-5 p.m., monday through friday. after a library staff member was hired, evening hours were implemented in spring 2022 from 9 a.m.-7 p.m., monday through thursday, and fridays from 9 a.m.-5 p.m.. following the request from students and faculty to have a diverse collection, we purchased print materials utilizing the one-time academic collection fund of $10,000. given the need for study spaces, we designed the layout of the library suite to make use of the library spaces as much as possible which included adding a printer, copier, and computer tables with six computers that were spaced out to practice social distancing. finally, we offered a research help desk and research help hours for virtual and in-person consultations.    outcome   there were both successful and unsuccessful changes in the library. evening hours had twice as much patron count and usage as the morning hours. because most classes are held in the evenings, students and faculty appreciated the extended opening hours.   in terms of the library layout, the arrangement of the bookshelves proved to be a success. we assembled the bookshelves against a wall in a hallway that crosses the center of the library. when students, staff, or faculty walk across the library, they are met and drawn to the books displayed. it has since prompted the community to stop and browse the collection.   the initial layout arrangement of the reading room in the library, meant to be a quiet lounge area, did not attract many students at the start of the 2021-2022 academic year; however, i was unsure of whether this was because the library initially opened in the fall semester when many classes were held online and in hybrid formats, or if the open space layout was not attractive to students. nonetheless, at the start of the fall 2022 semester, the reading room was reconceptualized and rearranged to remove empty spaces and provide a much more intimate study space area. in spring 2022, after the study space was rearranged, students used the lounge area more than they did in fall 2021.   during fall 2021, laptops, webcams, and other technology equipment check out statistics were low. the library promoted library technology equipment for check out via the library website and through social media to encourage students to utilize technology equipment. however, students often asked for laptops they could check out for the semester and the library offered laptops to check out for up to four hours daily. yet in spring 2022, technology check out statistics grew. as we offer more in-person services and academic courses, we anticipate higher usage in the upcoming academic year.    reflection   it is important to continue to work with students to grasp the community’s needs and help with the extended design of the library space and services. it is not only critical to have students involved in the initial planning process but having students as partners is essential. this type of student-as-partners approach will include shared decision making and will foster inclusivity (salisbury, 2020). as we re-establish our presence, our true focus is in supporting the success and research of the students.   the survey provided the library with data utilized for identifying and confirming services necessary for student success. since the survey was conducted virtually and during the covid-19 pandemic when most students had not visited the library or used its resources, i plan to conduct an in-person study as well. an in-person survey with students, while they are physically using the library space and services, may reveal new insights and provide us with alternate and relevant feedback.   our next steps for the scl are to continue developing the collection and continue to design the library space and library programs and services. for future design and management of library services, the scl will follow a similar project management style as that of the national health service library’s redesign. in the case study for the national health service library, the redesign transformation followed a four-stage approach for project management: start, plan, implement, and close (scott, 2021). the scl is currently in the implement stage and will re-route to plan and implement a couple more times over, until the transformation of the scl is complete and ready for its “close” stage. until then, the library will connect with the community and strive for a new and enhanced library presence as the campus offers in-person services again.   references   california state university, stanislaus. (2021). enrollment. https://www.csustan.edu/iea/institutional-data/enrollment   salisbury, f., dollinger, m., & vanderlelie, j. (2020). students as partners in the academic library: co-designing for transformation. the new review of academic librarianship, 26(2-4), 304–321. https://doi.org/10.1080/13614533.2020.1780275   scott, r. j. (2021). a best-it solution: transforming an nhs library and knowledge service in readiness for a new hospital building without a traditional library space. journal of the medical library association, 109(3), 483–489. https://doi.org/10.5195/jmla.2021.1167   shumaker, d. (2021). the next normal: the post-pandemic future of library services. information today, 38(4), 14–16.   thomas, m. a. (2000). redefining library space: managing the coexistence of books, computers, and readers. the journal of academic librarianship, 26(6), 408–415.   evidence based library and information practice evidence based library and information practice 2009, 4.4     evidence based library and information practice         news/announcements    two upcoming seminars from the library & information research group      © 2009. this is an open access article distributed under the terms of the creative commons attribution   license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and   reproduction in any medium, provided the original work is properly cited.      are you interested in doing some research?  would you like to improve your success rate  for your research proposals?  do you have a great idea but do not know  how to turn this into funding?  … then the library & information research  group have two seminars to help you.    how to write a successful research  proposal: secrets of success – session 1  the first half‐day seminar consists of  presentations by successful library and  information researchers in the art of writing a  successful research proposal. providing ideas  and tips in:    • writing research proposals and  experiences of actually conducting the  research  • using research as evidence to support  your own library/service development  ‐ in benchmarking, etc.  • how to use research in your  library/service to contribute to the  wider profession, by publishing the  results in the library journals, etc.  who should attend: practitioners, students  and academics. it is particularly suitable for  those who have never submitted a research  proposal and/or those who are interested in  submitting proposals for the forthcoming  lirg funding awards.  duration: 3 hours  cost: £25 members of lirg or cdg scotland;  £50 non‐members.  locations and dates:  glasgow: 8 january 2010, 10am‐1pm.  strathclyde university, 26 richmond street.  this session will be held in conjunction with  the career development group scotland and  is eligible for the cilip seal of recognition.  birmingham: 15 january 2010, 1pm‐4pm.  birmingham central library.  london: 12 january 2010, 10am‐1pm.  university of east london, docklands  campus.          107 evidence based library and information practice 2009, 4.4  how to write a successful research  proposal: proposal workshop ‐ session 2   this second session is a half day workshop  offering participants an opportunity to:    • discuss sample proposals   • contribute a proposal of their own for  workshop discussion  • network and discuss experiences   who should attend: practitioners, students  and academics. it is particularly suitable for  those who have never submitted a research  proposal and/or those who are interested in  submitting proposals for future lirg funding  awards.    duration: 3 hours  cost: £25 members of lirg or cdg scotland;  £50 non‐members.   locations and dates:  glasgow: 29 january 2010, 10am‐1pm.  strathclyde university, 26 richmond street.  this session will be held in conjunction with  the career development group scotland and  is eligible for the cilip seal of recognition.  birmingham: 29 january 2010, 10am‐1pm.  birmingham central library.  london: 26 january 2010, 10am‐1pm.  university of east london, docklands  campus.     why two seminars? the two seminars are  linked – with the first providing an  introduction and the second a more practical  experience to develop skills.  participants may  attend both or sign up for either seminar.  for  attendance at the second seminar, participants  will be asked to bring a draft proposal for  discussion in the workshop session.  reduced rates:  for attending both seminars,  the cost will be: £40 lirg and cdg scotland  members; £75 non‐members  for further details, contact alison brettle  a.brettle@salford.ac.uk   for booking, contact alan poulter  alan.poulter@cis.strath.ac.uk   final date for booking:  6th january 2010 (for  all seminars)    108 mailto:a.brettle@salford.ac.uk mailto:alan.poulter@cis.strath.ac.uk editorial evidence based library and information practice 2009, 4:2 1 evidence based library and information practice editorial showing value denise koufogiannakis editor-in-chief collections & acquisitions coordinator, university of alberta libraries edmonton, alberta, canada e-mail: denise.koufogiannakis@ualberta.ca © 2009 koufogiannakis. this is an open access article distributed under the terms of the creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. when su cleyle and i first decided to start evidence based library and information practice, one of the things we agreed upon immediately was that the journal be open access. we knew that a major obstacle to librarians using the research literature was that they did not have access to the research literature. although su and i are both academic librarians who can access a wide variety of library and information literature from our institutions, we belong to a profession where not everyone has equal access to the research in our field. without such access to our own body of literature, how can we ever hope for practitioners to use research evidence in their decision making? it would have been contradictory to the principles of evidence based library and information practice to do otherwise. one of the specific groups we thought could use such an open access venue for discovering research literature was school librarians. school librarians are often isolated and lacking access to the research literature that may help them prove to stakeholders the importance of their libraries and their role within schools. certainly, school libraries have been in decline and the use of evidence to show value is needed. as ken haycock noted in his 2003 report, the crisis in canada’s school libraries: the case for reform and reinvestment, “across the country, teacherlibrarians are losing their jobs or being reassigned. collections are becoming depleted owing to budget cuts. some principals believe that in the age of the internet and the classroom workstation, the school library is an artifact” (9). within this context, school librarians are looking to our research literature for evidence of the impact that school library programs have on learning outcomes and student success. they are integrating that evidence into their practice, and reflecting upon what can be improved locally. they are focusing on students and showing the impact of school libraries and librarians on student achievement. todd notes, “if we do not show value, we will not have a future. evidence-based practice is not about the mailto:denise.koufogiannakis@ualberta.ca evidence based library and information practice 2009, 4:2 2 survival of school librarians, it’s about the survival of our students” (40). in this issue we feature school libraries and their connection to evidence based practice. former editor-in-chief, lindsay glynn, began putting the wheels in motion for this feature almost a year ago. she invited carol gordon and ross todd to act as guest editors of the section, drawing upon their contacts and previous work in this field. the result is an issue with five feature articles exploring different aspects of the connection between school libraries and evidence based practice, from the theoretical to the practical. in addition, there is a thought-provoking commentary by david loertscher, asking whether we need the evolutionary model of evidence based practice, or something more revolutionary! in addition to the feature section, we have a well-rounded issue with articles on the topics of library human resources, and the development of a scholars’ portal. as well, there are a record 10 evidence summaries and our educational ebl101 column. i hope there is something for everyone in this issue of eblip – enjoy, and see you soon in stockholm! works cited haycock, ken. the crisis in canada’s school libraries: the case for reform and reinvestment. a report for the association of canadian publishers. toronto, on: june 2003. 15 may 2009 <http://www.peopleforeducation.co m/adx/aspx/adxgetmedia.aspx?doc id=999>. todd, ross. “the evidence-based manifesto.” school library journal 54.4 (april 2008): 38-43. http://www.peopleforeducation.com/adx/aspx/adxgetmedia.aspx?docid=999 http://www.peopleforeducation.com/adx/aspx/adxgetmedia.aspx?docid=999 http://www.peopleforeducation.com/adx/aspx/adxgetmedia.aspx?docid=999 http://www.peopleforeducation.com/adx/aspx/adxgetmedia.aspx?docid=999 evidence based library and information practice evidence based library and information practice 2010, 5.3 1 evidence based library and information practice editorial the appropriateness of hierarchies denise koufogiannakis editor-in-chief collections & acquisitions coordinator, university of alberta libraries edmonton, alberta, canada email: denise.koufogiannakis@ualberta.ca 2010 koufogiannakis. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. in the early days of eblip, then referred to as evidence based librarianship (ebl), there were calls to strengthen our research base with "better" forms of evidence. these proposed better quality research methods were all quantitative and i admit myself to saying that ‚librarianship tends to reflect more qualitative, social sciences/humanities in its research methods and study types which tend to be less rigorous and more prone to bias‛ (crumley and koufogiannakis 2002, p.61). although this was not meant to be a slight to qualitative research, i can see how it came across as one. now, i would not put ‚less rigorous and more prone to bias‛ in that sentence, although the first half of the statement certainly still holds true. in our 2002 article, the general point that ellen crumley and i were trying to make is that a medical style research hierarchy is not a good fit for librarianship, where qualitative methods are generally more appropriate. at that time, we proposed a ‚core-centred approach to librarianship research‛ (p.68) rather than a hierarchical one, although this did not gain much traction within the eblip literature. we noted: ‚rather than relying on an evidence hierarchy, which is an artificial concept for librarians, fig.3 suggests a core-centred approach. the types of studies that are likely to be conducted by librarians are placed near the centre, moving from a hierarchical to an encompassing model. … *this+ presents a more equitable view of a model for research in the profession‛ (p.67). today i am even more resolved that it is time to remove the concept of a hierarchy of evidence from eblip. this concept is tied very closely to the medical model of evidence based medicine (ebm) and is solely focused on quantitative research. library and information studies (lis) is a social sciences discipline and as such is concerned mostly with questions of why we do things and how people function in the world. the actions of people are complex and not easily fit into tidy boxes of absolute truth, as are commonly given preference in science, technology and medicine fields. i believe the focus on quantitative research arose within eblip for two reasons: 1) ebl formed within medical librarianship and those librarians identified more closely with the medical model of problem solving, which placed a higher level of trust in quantitative mailto:denise.koufogiannakis@ualberta.ca evidence based library and information practice 2010, 5.3 2 research; and, 2) there was a tide-swell of professional librarians who were tired of seeing our literature dominated by stories of ‚how i did it good‛, and wanted to see more research being published rather than opinion. this led to sometimes strongly worded and simplistic rhetoric noting that some forms of publication were not scientific, in order perhaps to highlight the importance of quality research. unfortunately, qualitative research was caught up in this wave and projected as not as worthy as quantitative research. this is clearly noted in the levels of evidence hierarchies that became part of the eblip doctrine. in both his original published article on the levels of evidence within ebl (2000a), and a subsequent version (2002), eldredge places qualitative forms of evidence at or near the bottom of the spectrum, although elsewhere he has recognized the role of both quantitative and qualitative research (2000b). the 2002 version of eldredge’s levels of evidence does put ‚qualitative research‛ at the middle of the continuum for exploration-type questions, but it still seems marginalized, summed up by a single line as if all qualitative research is one; whereas in the other categories, quantitative methods are given granular description. this model has been readily accepted, but there remains a lack of research on why this would work for librarianship. my concern is that the hierarchical structure may have been too easily accepted and taken as truth, marginalizing other forms of research and not referring back to the original research question. it also could have alienated a number of practitioners who think that eblip is not for them because of the focus on quantitative methods. lis does not need to be able to point and say that we have a certain number of randomized controlled trials in our field to prove that our research has worth. we don't need to pat ourselves on the back for doing a particular type of research. what we need to do is ensure the research method is the right one to better understand the question posed, and that the research is well done. if we find research that meets those two criteria, regardless of the type of method used, then we will be in a much better position to incorporate that research into our decision making, and can pat ourselves on the back for doing good quality, rigorous research that is valid or trustworthy. after more than 10 years of growth in eblip, i trust that many of us who try to practice in an evidence based manner realize that the type of evidence we choose has to be directly drawn from the research question we are asking (eve 2008; robson 2002; given 2006). for example, one would not use a randomized controlled trial to study whether children who are read to exhibit greater literacy skills at school age. and if you did find an rct on that topic, you would certainly need to question it! research, and trying to incorporate research into decision making, is far more complicated than following the rules of a simple hierarchy – in fact, doing so may harm the entire process and leave practitioners without any hope of finding ‚good‛ evidence on which to base their decisions. in my opinion, it is time to do away with hierarchies, as they do us no favours. rather than challenging us to think critically about each and every question, a hierarchy creates a false sense of being able to quickly determine research worth. booth (2010), in his health information and libraries journal column, ‚using evidence in practice‛, also bemoans the seeming authority of ‘the hierarchy of evidence’. he proposes two alternatives to evidence hierarchies, signal-to-noise ratios and evidence typologies. i personally favour the latter, in which ‚a typology of evidence stresses the need to consider which type of studies are most appropriate for answering different types of questions‛ (p.87). but whether we can find a way to create such a typology within lis that would be more than a token is questionable. where it could be helpful is in providing practitioners with ideas of the types of research they may consider for different types of questions, and the reasons why these methods are appropriate. evidence based library and information practice 2010, 5.3 3 why bog ourselves down in a context of one research method being better than another? this is simply divisive and does not contribute solutions. let's stop talking in terms of research hierarchies and inherent worth of particular methods and instead talk about appropriateness and good research design. even better, let’s get on with doing studies that are appropriate, rigorous, and relevant to practice. references booth, a. (2010). on hierarchies, malarkeys and anarchies of evidence. health information and libraries journal 27: 8488. crumley, e., and koufogiannakis, d. (2002). developing evidence-based librarianship: practical steps for implementation. health information and libraries journal 19(4): 61-70. eldredge, j. (2002). evidence-based librarianship levels of evidence. hypothesis 16(3): 10-13. eldredge, j. d. (2000a). evidence based librarianship: an overview. bulletin of the medical library association 88(4): 289-302. eldredge, j.d. (2000b). evidence-based librarianship: searching for the needed ebl evidence. medical reference services quarterly 19(3): 1-18. eve, j. (2008). writing a research proposal: planning and communicating your research ideas effectively. library and information research 32(102): 18-28. given, l. (2006). qualitative research in evidence-based practice: a valuable partnership. library hi tech 24(3): 376386. robson, c. (2002). real world research: a resource for social scientists and practitioner-researchers. oxford, uk: blackwell publishers. evidence based library and information practice evidence based library and information practice 2010, 5.4 130 evidence based library and information practice ebl 101 evaluating the results of evidence application, part two: at the practice level virginia wilson client services librarian, murray library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca received: 01 nov. 2010 accepted: 02 nov. 2010 2010 wilson. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. evaluation after implementation of evidence is a step that can be easily overlooked. however, its importance cannot be overestimated. last time i wrote about evaluation at the practitioner level, or reflection on your own performance as an evidence based practitioner. this time, it is evaluation at the practice level—you want to discover if, as booth puts it, “the service that you introduced or modified as a result of undertaking the evidence-based process actually made the anticipated difference” (p. 127). evaluation will do one of two things: confirm that the actions taken had the anticipated effect, or it may lead you to rethink your original issue. either way, evaluation can generate valuable information for your practice. evaluation or assessment is not exclusive to evidence based library and information practice. many libraries are cultivating a culture of assessment. there is a conference that focuses on assessment (library assessment conference, http://libraryassessment.org), academic libraries are using tools such as libqual+ (http://library.queensu.ca/webir/canlibqual/car l-libqual.htm) for assessment activities, and there are even assessment librarians in place in some libraries. the value of looking inward at the overall practice of the institution, in whatever library sector you may be situated, is high. while libqual+ is a massive assessment tool, evaluation can be done on a much smaller scale for individual evidence based projects. ideally, plans for evaluation are included at the beginning of an evidence based project, or at least at the start of the implementation of whatever change is being made. the exact steps to be taken to perform the evaluation will vary depending on the scope of the project and what changes are occurring or being evaluated. evaluation for a newly implemented reference model will be different than evaluation of a new instructional design approach. evaluation that is undertaken as the project is ongoing is referred to as formative evaluation, and focuses on the process. summative evaluation, undertaken at the end of a project, illuminates the ultimate mailto:virginia.wilson@usask.ca� evidence based library and information practice 2010, 5.4 131 effectiveness of the implementation. both can be helpful when looking at an evidence based project. there are several questions to ask when thinking about evaluation: • exactly what is being evaluated? • how will it be measured? • how will the data be collected and analyzed? • what indicates improvement or success? • what are the goals to be accomplished? • what is the timeline for the evaluation? • who will be involved in the evaluation process? there are many different methods of evaluation, depending on what you are attempting to evaluate and what data you need to collect. approaches such as focus groups, interviews, surveys, questionnaires, and observation are all ways to assess whether or not your changes have achieved the desired effect. while it is far beyond the scope of this column to explore the explicit details of these various kinds of evaluation, there are many resources out there to help you out. here is a brief bibliography to get you started. crawford, j. (2006). the culture of evaluation in library and information services. oxford, uk: chandos. mathison, s. (ed.). (2005). encyclopedia of evaluation. london: sage. matthews, j.r. (2007). the evaluation and measurement of library services. westport. ct.: libraries unlimited. wallace, d.p. & van fleet, c. (2001). library evaluation: a casebook and can-do guide. englewood, co: libraries unlimited. in practical terms, evaluating the implementation of evidence from the perspective of practice involves undertaking the steps of evidence based library and information practice again: formulate a question (decide what it is you are evaluating or measuring), find the evidence (in this case, you will be generating your own evidence by evaluating the changes made), appraise the evidence (this will involve making sure your evaluation methods and techniques are sound), and apply the evidence (by holding it up to your indicators of success). in terms of the evaluation step in this instance, you can be reflective and evaluate your own work as an evidence based practitioner. as with anything in the evidence based process, starting small can help overcome feeling overwhelmed. eventually, the process will become more streamlined and easier to manage. consulting with colleagues as well as the literature can help to get you going. all you need to do is take that first step. next time in ebl101, i will take a look at the process of disseminating your research. you have done an evidence based project, now what? get it out there to help others in the library world. reference booth, a. (2004). evaluating your performance. in a. booth & a. brice (eds.), evidence–based practice for information professionals: a handbook (pp. 127-137). london: facet. / evidence based library and information practice evidence based library and information practice evidence based library and information practice 2010, 5.1 evidence based library and information practice news/announcements evidence based practice for information professionals book now freely available 2010. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. a pdf version of the pre-publication manuscript of the now out-of-print book, evidence based practice for information professionals: a handbook, edited by andrew booth and anne brice (2004) is now freely available for reading and download at: http://ebliptext.pbworks.com/f/booth+%26+brice+2004+ebp+for+info+professionals+-+a+handbook.pdf it may also be accessed through the wiki site, ebliptext, at: http://ebliptext.pbworks.com/ this manuscript version includes the following that did not make it into the published version: bexon, n., & brice, a. (2004). special topic: what is the best way to train my staff. unpublished manuscript in booth, a. & brice, a. (eds.), evidence-based practice for information professionals: a handbook (pp. 170-175). london: facet publishing. beverley, c., & bath, p. (2004). case study 2: the health information needs of visually impaired people. unpublished manuscript in booth, a. & brice, a. (eds.), evidence-based practice for information professionals: a handbook (pp. 276-286). london: facet publishing. beverley, c., & winning, a. (2004). case study 1: the clinical librarian. unpublished manuscript in booth, a. & brice, a. (eds.), evidence-based practice for information professionals: a handbook (pp. 266-275). london: facet publishing. booth, a. (2004). special topic: provision of mediated information searching alongside end user services [guideline]. unpublished manuscript in booth, a. & brice, a. (eds.), evidence-based practice for information professionals: a handbook (pp. 239-243). london: facet publishing. the following previously published content is also included: part 1: the context for evidence-based information practice chapter 1) why evidence-based information practice? andrew booth & anne brice chapter 2) a brief history of evidence-based practice. anne brice & alison hill chapter 3) evidence-based information practice: a pre-history. jonathan eldredge chapter 4) how good is the evidence base? jonathan eldredge chapter 5) why don't librarians use research? paul genoni, gaby haddow & ann ritchie part 2: skills and resources for evidence-based information practice chapter 6) formulating answerable questions. andrew booth chapter 7) identifying sources of evidence. alison winning chapter 8) searching the library and information science literature. catherine beverley chapter 9) appraising the evidence. andrew booth & anne brice chapter 10) applying evidence to your everyday practice. denise koufogiannakis & ellen crumley chapter 11) evaluating your performance. andrew booth chapter 12) disseminating the lessons of evidence-based practice. ellen crumley & denise koufogiannakis part 3: using the evidence base in practice introduction: six domains of evidence-based information practice. andrew booth & anne brice chapter 13) examining the evidence base for reference services and enquiry work. andrew booth special topic: provision of a current awareness service for research staff [guideline]. robert kiley chapter 14) the contribution of evidence based practice to educational activities. anne brice & cindy carlson special topic: how can i train my users? [evidence digest]. alison brettle chapter 15) an evidence-based approach to collection management. andrew booth special topic: electronic or paper: how do i manage my journals collection? [evidence digest]. david peacock chapter 16) towards evidence based management. andrew booth special topic: how do i measure the impact of my service? [guideline]. christine urquhart special topic: should i charge and, if so, what should i charge for? [evidence briefing]. lynette cawthra chapter 17) evidence based perspectives on information access and retrieval. andrew booth special topic: what are the characteristics of a good searcher? [critically appraised topic]. catherine beverley special topic: which database, which interface [guideline]. maria j. grant chapter 18) introducing an evidence based approach to marketing and promotional activities. andrew booth special topic: determining the information needs of practising nurses post registration in the uk from 1990 to the present [evidence digest]. jennie kelson chapter 21) a future for evidence-based information practice? anne brice, andrew booth, ellen crumley, denise koufogiannakis & jonathan eldredge page 159 _1201335999.psd _1201336156.psd evidence based library and information practice evidence based library and information practice 2010, 5.1  evidence based library and information practice       evidence summary    existing analytical frameworks for information behaviour don’t fully explain  hiv/aids information exchange in rural communities in ontario, canada    a review of:  veinot, t., harris, r., bella, l., rootman, i., & krajnak, j. (2006). hiv/aids information exchange in  rural communities: preliminary findings from a three‐province study. canadian journal of  information and library science, 30(3/4), 271‐290.      reviewed by:   kate kelly  chief librarian, royal college of surgeons in ireland (rcsi)   dublin, ireland  email: katekelly@rcsi.ie    received: 01 dec. 2009          accepted: 24 jan. 2010       2010 kelly. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which  permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly  attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the  same or similar license to this one.    abstract    objective –to explore and analyze, against  three theoretical frameworks of information  behaviours, how people with hiv/aids, their  friends, and their family living in rural  communities find information on hiv/aids.     design – qualitative, individual, in‐depth,  semi‐structured interviews.    setting – two rural regions in ontario,  canada.    subjects – sixteen participants; 10 people with  hiv/aids (phas) and 6 family members or  friends.    methods – participants were recruited  through health care providers, social service  agencies and through snowball sampling.  semi‐structure interviews were conducted  focusing on participants’ experience with  hiv/aids, how they find and use information  on hiv/aids, networks for information  exchange and the effect of technology on  information exchange. interviews were taped,  transcribed, analyzed qualitatively using  nvivo software. results were compared to  three theoretical frameworks for information  behaviour: 1. purposeful information seeking  (i.e., the idea that people purposefully seek  information to bridge perceived knowledge  gaps); 2. non‐purposeful or incidental  information acquisition (i.e., the idea that  people absorb information from going about  daily activities); and 3. information gate  135 mailto:katekelly@rcsi.ie evidence based library and information practice 2010, 5.1  keeping (i.e., the concept of private  individuals who act as community links and  filters for information gathering and  dissemination).    main results – consistent with the theories:  • phas prefer to receive information  from people they have a personal  relationship with, particularly their  physician and especially other phas.  • phas’ friends and families rely on their  friends and family for information, and  are particularly reliant upon the pha  in their lives.  • fear of stigma and discrimination cause  some to avoid seeking information or  to prefer certain sources of information,  such as healthcare providers, who are  bound by codes of professional  conduct.  • emotional support is important in  information provision and its presence  supersedes the professional role of the  provider (social workers and  counsellors were identified as key  information sources over medical  professionals in this instance).  participants responded negatively to  the perceived lack of support from  providers including doubting the  information provided.  • phas monitor their worlds and keep  up to date about hiv/aids.    inconsistent with theories:  • reliance on caregivers for information  is not solely explained by fear of stigma  or exposure. rather, it is the specialized  knowledge and immersion in  hiv/aids which is valued.  • the distinction between peer or kin  sources of information and institutional  information sources is less clear and  relationships with professionals can  turn personal over time.  • inter‐personal connections include  organisations, not just individuals,  particularly aids service  organizations and hiv specialist  clinics.  • relatively few incidents of finding  useful information about hiv/aids  incidentally were described. the  concept of information just being “out  there” was not really applicable to rural  settings, likely due to the lack of  discussion within participant  communities and local media. when it  was discussed, participants reported  being more likely to gain  misinformation through their personal  networks.  • incidental information acquisition  originates mostly from professional  and organisational sources. participants  identified posters, leaflets, and, for  those who interacted with  organisations, information via mail as  contributing to current awareness.   • the gate keeping concept does not  capture all the information sharing  activities undertaken by “gate keepers”  in rural areas, and neither does it  include formal providers of  information, yet all phas interviewed  identified formal providers as key  sources.    conclusion – the findings reinforce some of  the existing analytical framework theories,  particularly the importance of affective  components (i.e. emotional supports) of  information seeking, the presence of  monitoring behaviours, and of interpersonal  sources of information. however, alternate  theories may need to be explored as the role of  institutional information sources in the lives of  phas doesn’t match the theoretical  predication and the “gate keeper” concept  doesn’t capture a significant portion of that  role in rural hiv/aids information exchange.      commentary    the results of this particular study are  preliminary findings from a larger study, the  rural hiv/aids information networks study  (2005‐2008), whose long‐term goal was to  investigate new approaches to providing  information in rural communities. the latter  136 evidence based library and information practice 2010, 5.1  137 study was carried out in ontario,  newfoundland and labrador and british  colombia, hence the reference to three  provinces of the title of the original article.       that the results of the study under review  originate from a larger set of results leads to  some contradictions between the methodology  described and the results as reported. for  example, the methodology section describes  the research as being undertaken in three  canadian provinces, and that all phases of the  research involved community representatives  who interviewed phas, their friends, family  members, health care providers and service  providers. the interviews described covered a  broad range of topics.      in contrast, the results reported are based on  interviews with 10 phas and 6 family  members in two areas in ontario only and  focus on a section of the interview that covers  the participants’ information network related  to hiv/aids. while this doesn’t necessarily  detract from the findings, and the authors are  clear that these are preliminary findings, it is  confusing. in addition, the lack of sufficient  detail about the content of the interview  questions in general, and the information  exchange in particular, even if semi‐ structured, adds to the overall lack of clarity  and begs the question of how representative  the participants were and whether there where  findings from other areas. further, the reader  is led to wonder why this particular group of  16 was chosen. no details are provided about  gender, age, or socio‐economic background of  participants, leaving the question of whether  these factors had an impact on information  seeking behaviour unanswered.      while the theoretical frameworks used are  well documented and the discussion is  thorough, the reference to social network  analysis as a potentially more robust  theoretical framework could be pursued. an  overall explanation of the choice of  frameworks would have been a useful  addition. the discussion relating findings to  theory uncovers areas of divergence from the  analytical frameworks and in so doing  identifies several areas for further research.  librarians working in rural areas or those  targeting rural populations should note the  importance of organizations and healthcare  professionals in providing information to  phas and their families, and of incidental  information acquisition via clinics, specialist  organizations, posters, leaflets and mail  inserts.  however, it is worth acknowledging  that since these are preliminary findings and  that the authors intended to conduct further  research, no firm conclusions can be drawn  from this work alone.     finally, it cannot be overlooked that this  article is dated from 2006 and that the rural  hiv/aids information networks study was  completed in 2008. those seeking more  comprehensive details might consult tiffany  veinot’s more recent publications on the  subject, listed in the reference list.       references    harris, r.m., veinot, t.c., bella, l., rootman,  i., & krajnak, j. (2008). helpers,  gatekeepers and the well‐intentioned:  the mixed blessing of hiv/aids  informediation in rural canada. in  wathen, n., wyatt, s., & harris, r.m.  (eds.), health information and  technology: mediating health information  in a changing socio‐technical landscape  (pp. 167‐181). new york: palgrave  macmillan.    veinot, t.c. (2008). social capital and  hiv/aids information/help exchange  networks in rural canada.  (unpublished doctoral dissertation,  university of western ontario, 2008.)     veinot, t.c. (2009). interactive acquisition and  sharing: understanding the dynamics  of hiv/aids information networks.  journal of the american society for  information science & technology, 60(9),  1‐20.   evidence summary   weak correlation between circulation and citation numbers suggests that both data points should be considered when deselecting print monographs   a review of: white, b. (2017). citations and circulation counts: data sources for monograph deselection in research library collections. college & research libraries, 78(1), 53 – 65. https://doi.org/10.5860/crl.78.1.53   reviewed by: melissa goertzen consultant and information manager halifax, nova scotia, canada email: goertzen.melissa@gmail.com   received: 11 june 2019                                                                  accepted:  23 oct. 2019      2019 goertzen. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29606     abstract   objective – to facilitate evidence-based deselection of print monographs, this study examines to what extent there are correlations between circulation data (past and future usage) and between the borrowing and citation of print monographs.   design – collections assessment project that used a variety of data sources and techniques, including spearman’s rank correlation coefficient, statistical analysis, and the analysis of circulation data, last-use dates, and citation data.   setting – an academic library in new zealand. subjects – two ranges of books were chosen for the study: 591 (specific topics in zoology) and 324 (the political process). from these ranges, monographs published prior to 2001 were selected as the study sample.   methods – this project relied on two data sources: circulation data from the library’s ils and citation data from scopus. all data was downloaded to an excel spreadsheet in preparation for analysis. the researcher examined call numbers, authors and editors, titles and subtitles, publication dates, circulation counts, dates of last check-in, total number of citations, number of citations from publications released in 2010 and on, and number of citations from institution-affiliated documents. renewal data was omitted, as it did not provide evidence of additional instances of use.   where multiple copies of a specific title appeared in the data set, the researcher totalled all circulations and recorded the most recent check-in date. the researcher found that some titles in the study sample were generic and it was impossible to determine if citation data from scopus linked to the monograph in the library collection. these titles were eliminated from the study.   once data collection was complete, the researcher calculated two additional data elements: the number of months since the last check-in date and the number of citations from items published before 2010. data in the excel spreadsheet was analyzed using spearman’s rank correlation coefficient to determine the relationship between past and future usage and between circulation and citation data.   main results – findings indicated that circulation and citation data are highly skewed. many monographs in the study sample had never been borrowed and had few citations, while a small number of “celebrity titles” were borrowed or cited at a much higher rate than other monographs in the same classification.   further, results indicated that historic circulation numbers are imperfect predictors of future probability that a book will be borrowed. when taking a high-level view of the collection, highly circulated books tend to be borrowed more often than average. however, when examining monographs at the title level, high circulation is more of a probability instead of a robust indicator.   an investigation of whether historic citation counts serve as an indicator of future citation followed previously established trends: monographs not heavily cited in the past are less likely to be cited in the future. findings also found a weak correlation between local-institution monograph citation counts and total citation counts.   finally, the results demonstrated a weak correlation between circulation and citation data. as a group, well-cited books are borrowed more often than others, but at the individual title level, the effect is too random for either data set to predict the other in a reliable way. as such, circulation data and citation data can not be used as a proxy for each other.   conclusion – neither circulation nor citation data can stand as full proxies of the value of a title. however, both provide information that reflects the status of a title within the scholarly community. in this environment, citation data should be considered equally with circulation figures. both data points measure different phenomena and the weak correlation between them suggests that both are required to inform decisions about deselecting print monographs. commentary   the goal of collection development activities is to build collections that meet users’ information needs. studies conducted over the past several decades indicate that usage statistics support decisions to deselect print monographs from the collection; evidence suggests that past use is the best predictor of future use (dinkins, 2003). citation data is not typically factored into these discussions, as it is seen as a metric that guides decisions about journals as opposed to monographs. however, some argue that citation data can be used to supplement measures like faculty input and frequency of use (burdick, 1989). this paper investigates the relationship between circulation and citation data, and how both metrics inform decisions to deselect print monographs.    when evaluated using the “evaluation tool for bibliometric studies,” strengths and limitations of this study emerge (perryman, 2009). the strength of the piece is its organization and detailed descriptions of the methodology, data collection activities, and study results. the author identifies all data points used to analyze the past and future use of print monographs, describes the search scripts crafted to pull citation data from scopus, and lists correlations between data points. based on this discussion, information professionals at other institutions could replicate the study and compare results against those outlined in this paper. essentially, the author provides a roadmap for other librarians wishing to examine how circulation and citation data inform deselection decisions.   limitations of this study include incomplete data sharing. the author stated that the distribution of both circulation counts and citations are highly skewed, and that collecting citation data was error-prone and incomplete. it would have been beneficial if the author provided the numbers and percentages of non-circulated and non-cited books, along with highly circulated and highly cited books. this would give a full picture of the results and allow readers to judge the validity of findings. also, while the author explained the search strategy to identify citation data for monographs in the text, it would be more helpful to provide the exact search string for other librarians wishing to replicate the study. finally, in the last paragraph of the paper, the author mentions a study by kousha and thelwall (2014) that utilized apis to harvest citation data and standardize title-level citation metrics. it would have been interesting to see a similar method used in this study, as it provides an efficient and modern means to assess collections of print monographs.   overall, this study provides value to librarians working in the area of collection development and monograph acquisitions. the paper presents a low-cost and sustainable methodology that supports decisions to deselect print monographs based on readily available data. references   burdick, a. j. (1989). science citation index data as a safety net for basic science books considered for weeding. library resources & technical services, 33(4), 367–73.   dinkins, d. (2003). circulation as assessment: collection development policies evaluated in terms of circulation at a small academic library. college & research libraries, 64(1), 46–53. https://doi.org/10.5860/crl.64.1.46   kousha, k. & thelwall, m. (2014). an automatic method for extracting citations from google books. journal of the american society for information science and technology, 66(2), 309-320. https://doi.org/10.1002/asi.23170   perryman, c. (2009). evaluation tool for bibliometric studies. retrieved 30 june 2019 from http://libjournalclub.pbworks.com/f/journal%20club%20jan%2020%202011.pdf   evidence based library and information practice evidence based library and information practice 2010, 5.3 42 evidence based library and information practice evidence summary consumer health information websites with high visual design ratings likely to be also highly rated for perceived credibility a review of: robins, d., holmes, j., & stansbury, m. (2010). consumer health information on the web: the relationship of visual design and perceptions of credibility. journal of the american society for information science and technology, 61(1), 13-19. reviewed by: kate kelly chief librarian royal college of surgeons in ireland lower mercer street, dublin 2. ireland email: katekelly@rcsi.ie received: 1 june 2010 accepted: 4 aug. 2010 2010 kelly. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. objective – to answer two research questions: 1) what is the relationship between the visual design of a consumer health information web site and perceptions of the credibility of information found on it? 2) is there a relationship between brand recognition, visual design preference, and credibility judgments? design – qualitative (correlation of rating of response to stimulus) and quantitative (credibility coding of participant comments) setting – not stated; assumed to be academic institutions in the united states. subjects – thirty-four participants over the age of 35 (34 for statistical power and age over 35 on the hypothesis that this age group is most likely to seek health information on the internet). methods – screen shots of 31 consumer health information sites chosen from the results of a google search using the term “consumer health information” were converted to slide format and shown to participants. the 31 sites included 12 of the top ranked consumer health information sites derived from three sources: the consumer and patient health information section (caphis) of the medical library association (mla), the mla itself, and consumer reports. participants were read and shown a script explaining the process prior to being asked to view and rate the 31 sites. participants were first shown a blank slide with a crosshair to focus attention. then a mailto:katekelly@rcsi.ie evidence based library and information practice 2010, 5.3 43 stimulus slide was shown for 2.8 seconds. a blank black screen was shown while they determined their rating. participants were first asked to rate the visual design and aesthetics of the 31 web sites using a rating scale of -4 to 1 for negative judgments and +1 to +4 for positive judgments. then they were asked to remember why they had made positive or negative ratings and why some web sites were preferred to others. the process was repeated with the slides re-ordered, and participants were asked to rate the credibility of the sites using the same rating scales. upon completion, participants were asked to recall their reasons for positive or negative credibility ratings. all ratings were converted to positive numbers and a scale of 1-8 was used to present results. a variety of statistical analyses were carried out on the data, including correlation, means ratings, and rankings. in addition, all solicited comments on credibility were coded using fogg’s four types of credibility (surface, earned, presumed, and reputed) in order to try to understand why participants rated the credibility of sites as they did. main results – for the first question, concerning the relationship between visual design preferences and perceived credibility, the results are complicated. a statistically significant correlation was reported between visual design preference and perceived credibility in 8 of the 31 sites (26%). in these instances where visual design is rated highly, so is credibility. when visual design ratings were ranked highest to lowest, credibility ratings followed the same pattern. similarly, when credibility ratings were ranked highest to lowest, visual, design ratings followed. a ttest confirmed that sites perceived to have higher credibility were also perceived to have better visual design. furthermore, when design and credibility ratings were compared to site traffic rankings, as measured by alexa (http://www.alexa.com), the trend was for both visual design and credibility ratings to decline as the site traffic ranking declined. this finding was also confirmed by a t-test. while there is not an exact relationship, the tendency is for sites with higher visual design ratings to also receive higher ratings for perceived credibility. on the second question, concerning the relationship between brand recognition and visual design and perceived credibility judgments, the results suggest a possible influence of brand name. this relationship is not clear, and as visual designs were always presented and rated first, there is possibly a co-founder. the analysis of participant comments found that participants performed credibility judgments in a very short time using a variety of criteria, including visual design, source of the site, reputation of the site, and prior use. there were negative reactions to the use of advertisements, drug and insurance company sponsorship, and dot com sites, as well as some suspicion that nonus consumer health information sites were less trustworthy. conclusions – visual design judgments bore a statistically significant similarity to credibility ratings. sites with recognizable brands were highly rated for both credibility and visual design, but this relationship was not statistically significant. the relationship is complicated and more research is needed on what visual design cues are important to credibility judgments. commentary as an exploratory study, this is a very interesting and highly readable piece of research. it is, however, undermined by a lack of detail about the research participants and questions about the statistics. irrespective of whether the sample size of 34 gives statistical power, the information that the participants were chosen by age alone tells us nothing about their gender (studies indicates that females are more likely to seek health information on the internet); their experience of searching for health information; comfort level with using the internet; exactly how old or how it/information literate they were; their ethnic background, or if english http://www.alexa.com/ evidence based library and information practice 2010, 5.3 44 was everyone’s first language. was it a convenience sample, or a random sample? the hypothesis that people over the age of 35 are more likely to seek health information on the internet is debatable. for instance, the pew report 2009 indicated that 72% of 18-29 year olds sought health information on the internet (fox & jones, 2009). with regard to the statistics, the authors state that the statistical power is 0.80, but provide no details of how this was calculated. while 0.80 is a desired level of power, the minimum sample size for relationship testing with a single predictor using correlation or regression analysis is 50 cases (harris, 1985; green, 1991). according to cohen and cohen (1975), with an effect size of 0.30, 124 participants are needed to maintain 80% power. the sample size in this study does not meet the minimum requirement for statistical power of 50 cases, let alone the sample size required for statistical power of 0.80. readers should also note that only 8 out of 31 (26%) websites tested showed a significant correlation between visual design and credibility. that means that for 74% of the websites tested, there was no correlation between visual design and credibility. this would appear to invalidate the authors’ conclusion that design is an important attribute of website credibility. without details, the validity and generalizability of the findings has to be open to question. while the research methodology is described in reasonable detail, it is unclear whether participants were shown slides in a group or individually; and similarly, whether comments were solicited individually or as part of a group setting. these details would be useful for replicating the study. the literature review, however, is excellent and places the research in context in terms of an overall credibility framework, credibility on the web, and credibility of health information on the web. the exploration of credibility ratings is instructive, as is previous research by robins and holmes describing how impressions of web sites are formed within 2.4-3.2 seconds. this is clearly a precursor to the research under review (robins & holmes, 2008). health sciences librarians are likely to find the list and analysis of the 31 web sites very interesting. it includes expected sites like medlineplus, but also drug companies such as novartis and authoritative non-us sites such as the royal australian college of general practitioners. the authors discuss fully the limitations of this study and identify several areas for further research. while it is too early to say with any certainty, it is clear that this line of research could yield results which may influence website design, health information literacy, and the teaching of critical appraisal and evaluation of health internet sites. for those with responsibilities in these areas, this study should be essential reading, even with the limitations noted. it may be interesting for non-us readers to consider replicating the study to test the generalizability of these results within their own national and cultural contexts. references cohen, j., & cohen, p. (1975). applied multiple regression/correlation analysis for the behavioral sciences. hillsdale, nj: erlbaum. fox, s., & jones, s. (2009). social life of health information. pew research centre. retrieved 12 may 2010 from http://www.pewinternet.org/reports/ 2009/8-the-social-life-of-healthinformation.aspx green, s. b. (1991). how many subjects does it take to do a regression analysis? multivariate behavioral research, 26(3), 499-510. http://www.pewinternet.org/reports/2009/8-the-social-life-of-health-information.aspx http://www.pewinternet.org/reports/2009/8-the-social-life-of-health-information.aspx http://www.pewinternet.org/reports/2009/8-the-social-life-of-health-information.aspx evidence based library and information practice 2010, 5.3 45 harris, r. j. (1985). a primer of multivariate statistics (2nd ed.). new york: academic press. robins, d., & holmes, j. (2008). aesthetics and credibility in web site design. information processing and management, 44(1), 386-399. retrieved 13 may, 2010 from www.sciencedirect.com doi:10.1016/j.ipm.2007.02.003 http://www.sciencedirect.com/ an examination of academic library privacy policy compliance with professional guidelines research article   an examination of academic library privacy policy compliance with professional guidelines   greta valentine data & research analyst university of kansas libraries lawrence, kansas, united states of america email: greta.valentine@ku.edu   kate barron research data curator stanford libraries stanford, california, united states of america email: katebar@stanford.edu   received: 15 mar. 2022                                                                  accepted: 7 june 2022      2022 valentine and barron. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30122     abstract   objective – the tension between upholding privacy as a professional value and the ubiquity of collecting patrons’ data to provide online services is now common in libraries. privacy policies that explain how the library collects and uses patron records are one way libraries can provide transparency around this issue. this study examines 78 policies collected from the public websites of u.s. association of research libraries’ (arl) members and examines these policies for compliance with american library association (ala) guidelines on privacy policy content. this overview can provide library policy makers with a sense of trends in the privacy policies of research-intensive academic libraries, and a sense of the gaps where current policies (and guidelines) may not adequately address current privacy concerns.   methods – content analysis was applied to analyze all privacy policies. a deductive codebook based on ala privacy policy guidelines was first used to code all policies. the authors used consensus coding to arrive at agreement about where codes were present. an inductive codebook was then developed to address themes present in the text that remained uncoded after initial deductive coding.   results – deductive coding indicated low policy compliance with ala guidelines. none of the 78 policies contained all 20 codes derived from the guidelines, and only 6% contained more than half. no individual policy contained more than 75% of the content recommended by ala. inductive coding revealed themes that expanded on the ala guidelines or addressed emerging privacy concerns such as library-initiated data collection and sharing patron data with institutional partners. no single inductive code appeared in more than 63% of policies.   conclusion – academic library privacy policies appear to be evolving to address emerging concerns such as library-initiated data collection, invisible data collection via vendor platforms, and data sharing with institutional partners. however, this study indicates that most libraries do not provide patrons with a policy that comprehensively addresses how patrons’ data are obtained, used, and shared by the library.     introduction   the tension between upholding privacy as a professional value and the ubiquity of collecting patrons’ data to provide online services is now a common one in libraries. patrons who use digital library services are constantly providing the library with their personal data whether they know it or not. as stewards of this data, librarians are obligated to be transparent about the uses of patron data to provide services or make continuous improvements to these services. privacy policies that detail the collection and use of patron data are one way to provide such transparency. while guidelines for creating comprehensive privacy policies exist, literature indicates that these guidelines are often not applied to actual policies. this study examines privacy policies from u.s. academic libraries and describes the content of these policies in relation to privacy guidelines. it also describes content contained in these policies which is not addressed by existing guidelines. ultimately, this overview provides an environmental scan of recent policies across academic libraries in the us. the trends and gaps in this exploratory scan can inform the creation of robust policies that adequately address current privacy concerns in academic libraries.   literature review   libraries and privacy   librarians have long been advocates of privacy, largely in ways that emphasize the protection of users’ information and reading behavior. the american library association’s (ala) 1939 code of ethics outlines support for the privacy of users to pursue topics of interest without surveillance or punishment. since that time, ala’s office for intellectual freedom and the intellectual freedom committee, along with the associated privacy committee, have produced a variety of documents expanding on circumstances in which user privacy should be protected. these include the policy on confidentiality of library records (1986), privacy: an interpretation of the library bill of rights (2002, amended 2014 and 2019), resolution on the retention of library usage records (2006), and the policy concerning confidentiality of personally identifiable information about library users (2004) (vaughan, 2020).   in addition to its ethical documents, the ala first produced the privacy tool kit in 2005, providing librarians with a set of practical resources focusing on privacy (ala, 2014). the tool kit includes a list of privacy guidelines and checklists for navigating the exchange of data when using networked devices, assistive technology, public access computers, and other contexts that commonly require the collection or storage of patron data. much of the content of the tool kit is included in the issues and advocacy section of ala’s website and is periodically updated (ala, 2017a). the most recent version contains resources that support librarians in advocating for "the right to read, consider, and develop ideas and beliefs free from observation or unwanted surveillance by the government or others" (ala, 2021a, para. 1), including a privacy field guide that addresses privacy policies specifically (ala, 2021c).   when this study was initiated, the privacy field guide was not yet available. the primary documents referenced in this study are the privacy and confidentiality policy checklist (ala, 2017b) and the page titled developing or revising a privacy policy (ala, 2017c), which was available in the privacy audits section of the website. these resources were chosen since ala’s guidelines, ethical documents, and the privacy tool kit are widely cited in library literature that addresses the creation or evaluation of library privacy policies (see magi, 2010; nichols hess et al. 2014; and vaughan, 2020 for a sampling).   threats to privacy in libraries   libraries today face a variety of external and internal threats to privacy. the primary tensions discussed in library literature tend to focus on digital technologies that proliferated over the past decade and have come to shape the way patrons interact with library resources. while libraries have always required some level of information about patrons to provide core services such as circulation and interlibrary loan (coombs, 2005), early analog library accounts were established with information provided explicitly by the patron and used during discrete transactions, often face to face. today, simply accessing a library’s website can provide the site host with data such as a user’s ip address, geolocation, cookies, and other potentially identifying information without notifying the user or requesting consent. common cloud-based online platforms such as discovery layers, databases, and online public access catalogs (opacs) are “largely based on the tracking, collection, and aggregation of user data” (kritikos & zimmer, 2017, p. 24). libraries require this type of data collection when patrons discover and access proxied academic articles on digital platforms (o’brien et al., 2018; pekala, 2017), as well as any time they download materials, search the web, swipe an id card, or log into a virtual environment (jones & salo, 2018).   third-party platforms have introduced another layer of tension into libraries’ attempts to provide quality service while safeguarding patron data. some of these platforms implement their own data collection for analytics or provide options for customization and personalization of their services. when patrons provide their information to create a personalized browsing experience, libraries no longer have oversight of this data (magi, 2010). in this complex data landscape, some literature posits that a library “can no longer be considered the sole gatekeeper of its patrons’ private information, emphasizing the present reality that data privacy can be confusing, ambiguous, and opaque” (vaughan, 2020). opacity contributes to what affonso and sant’ana (2018) define as information asymmetry, a power imbalance between information holders and users in which those who hold the data have more power. to access core library services such as checking out books, using interlibrary loan, or searching online databases, patrons must now participate in this invisible data collection – a threat to privacy insofar as privacy allows users a measure of control over releasing their data to others (crawford & schultz, 2014).   in addition to data collection for the provision of library services, it has become increasingly common for libraries themselves to mine or reuse patron data for purposes that go beyond providing a service. academic libraries especially face pressure to collect and analyze student data to demonstrate the value of library services (prindle & loos, 2017). however, with their accompanying privacy concerns, projects that fall into the category of learning analytics have proven a contentious example of a library-initiated assessment (rubel & jones, 2016). some literature advocates for such studies, contending that the benefits outweigh the risks (beile et al. 2017; jones, 2010; oakleaf, 2010; oakleaf 2018) or that these studies can be implemented in an ethical manner (drachsler & greller, 2016). there is widespread adoption of learning analytics in academic libraries. of the 53 association of research libraries (arl) libraries that responded to a 2018 survey focused on learning analytics adoption, 83% indicated they were participating in learning analytics projects. however, there has also been a backlash against these types of library-initiated studies, pointing out ways they threaten patrons’ privacy (farkas, 2018; jones & salo, 2018; perry et al., 2018; prindle & loos, 2017). this single but prominent example of library-initiated projects that collect and retain patron data outside the scope of providing a service exemplifies a relatively recent shift in attitude toward privacy within the library profession (asher, 2017; prindle & loos, 2017).   the role of privacy policies   part of the concern regarding privacy violations in libraries deals with a lack of transparency and consent that would allow patrons the agency to control information about themselves or make informed decisions about how to interact with the library. one way that libraries deal with opacity surrounding data collection is to provide a privacy policy that details the purposes for which patrons’ data are collected and used. this is only one purpose of a policy, there are many that focus on library values and guidance for staff. a comprehensive privacy policy allows the library to set privacy protection standards (yoose, 2018), as well as foster appropriate conduct, consistency, and uniformity in how privacy is implemented while reducing confusion and empowering library workers (magi, 2007). these policies also protect the organization and provide guidance should legal action arise (magi, 2007).   however, much library literature argues that patrons are the primary audience for a library’s privacy policy. privacy policies can signal a commitment of integrity to a library’s patrons (vaughan, 2020), can represent “an enforceable guarantee to users” (briney et al., 2018, p.15), and can act as “a first step in removing barriers to information and....closing the information divide” (voeller, 2007, p. 18). nichols hess et al. (2014) document how 11 areas of library service encounter personally identifiable information, and how a privacy policy should address each of these scenarios to maximize appropriate handling of patrons’ data. the ala (2017c) states that a well-defined privacy policy should tell patrons how their data are being used and in what circumstances it might be disclosed. a policy that outlines how data are obtained, stored, and used is a baseline step towards allowing patrons to participate in informed consent, and has been shown to increase patrons’ trust in the library (nichols hess et al., 2014; sutlieff & chelin, 2010).   the ala (2017a, para. 4) stresses the importance of posting these policies where patrons can find them, indicating that patrons "have the right to be informed what policies and procedures govern the amount and retention of personally identifiable information, why that information is necessary for the library, and what the user can do to maintain his or her privacy.” further, literature indicates that patrons use these policies to guide their browsing and transaction decisions (vaughan, 2020) and to increase awareness of how the library collects data about them (affonso & sant’ana, 2018). the absence of a public policy deprives patrons of the ability to communicate discomfort with the use of their data or to withdraw consent (asher et al., 2018). while simply posting a privacy policy is not an adequate method of obtaining consent from a research perspective, it is the method many libraries employ, as there is often no means for patrons to indicate whether they want to supply or withhold data from the library beyond what is outlined in the privacy policy (asher, 2017).   issues with library privacy policies   despite the importance of privacy policies, several common threads in the literature indicate issues with current library policies. one of the most prevalent is the issue of locating policies to analyze. this issue is cited in studies across library types in both recent and more dated library privacy policy analyses (affonso & sant’ana, 2018; magi, 2007; vaughan, 2020; voeller, 2007). this does not necessarily mean that the libraries studied did not have a privacy policy in place, but the fact that researchers were unable to locate policies in multiple instances is a concern from the viewpoint of transparency to patrons. in 2018, 10% of 50 arl libraries responding to a survey about learning analytics indicated they did not have a privacy policy in place (perry et al., 2018).   a related issue is that some libraries may refer to the privacy policy of their parent institutions rather than a library-specific policy. sturges et al. (2003) found that few of the 336 higher education and special libraries surveyed had a privacy policy separate from their parent organization. the same arl survey mentioned above indicated that in 2018, 45 responding arl member libraries (90%) had a privacy policy in place, but only 31 (62%) had a policy separate from that of their parent institution (perry et al., 2018). the ala guidelines (2017c) indicate that library privacy policies should comply with the policies of their parent institutions. however, in addition to this compliance there are cases where libraries may want to implement privacy practices more pertinent to the library itself, which may have goals and values distinct from those of the parent institution (nichols hess, et al. 2014).   studies also highlight that many current privacy policies are not comprehensive enough to adequately inform patrons about library privacy practices. according to asher et al. (2018), “[f]ew to no institutions have established comprehensive policies and procedures around collection, retention, use, and reuse of student and employee data, in research or for any other reason” (p. 5). some policies are simply not detailed enough. voeller (2007) indicated that most of the 30 policies she examined contained less than 10 sentences, suggesting that they were unlikely to cover information in adequate depth.   another concern is that many policies have not been updated to address current and emerging privacy concerns (nichols hess et al., 2014). a 2018 (perry et al.) study surveyed arl libraries about whether they had updated their privacy policies to reflect participation in learning analytics activities. of the 53 libraries that responded, 43 indicated that they participated in learning analytics activities. however, only 7 (13%) had updated their privacy policies to account for this activity. complexities in protecting patron privacy exist where such practices occur in libraries, yet policies across the profession do not provide adequate guidance in navigating these situations. this is one example of a myriad of cases where data collection has become prevalent in ways not conceived of by older privacy policies (nichols hess et al., 2014).   aims   the library profession continues to tout privacy as a professional value despite the current challenges in implementing it. while a comprehensive, easily accessible privacy policy is certainly not the only way to uphold this value, it is a good first step.   this study aims to provide library policymakers with an overview of policy content across a subset of academic libraries in the us, and to compare these library privacy policies against professional guidelines. this study also examines policy content not addressed in professional guidelines, as the guidelines themselves may need to be updated or clarified. by analyzing policy content not addressed in the guidelines, the study results can describe content that librarians have deemed important enough to include in a policy beyond what professional guidelines have suggested.   this overview attempts to answer the following questions:   ·         are the privacy policies of arl libraries in the us readily available to patrons? ·         do these policies contain each of the elements specified in the ala’s privacy and confidentiality policy checklist? ·         do these policies contain additional elements not specified in the ala checklist? what is the nature of these additional elements?   methods   target population and sampling   while libraries of all kinds face the challenge of dealing with patron data in an ethical manner, academic libraries face unique pressures to demonstrate and quantify their value, resulting in more retention of patron data (tenopir, 2010). for this reason, the authors chose arl institutions in the us as the study population and gathered privacy policies from the websites of these 99 libraries in the summer of 2019. the number of arl libraries was manageable to examine within the scope of this study meaning that there was no sampling; the authors attempted to obtain policies from all 99 libraries.   data collection and analysis   this study employed direct observation to locate privacy policies that were publicly available on the websites of arl libraries. this method of collecting policies was important because it mimics the steps patrons must take to locate information about how the library uses their data. the authors located 78 privacy policies from the 99 libraries in the study population, for a success rate of 79%.   the authors employed document analysis, a subset of content analysis, to examine the text of these policies. document analysis allows for the examination of material that has not been created or modified for the purpose of the study by either the researcher or the subject (atkinson & coffey, 1997; bowen, 2009). this approach aims to limit the bias that a method like interviews or surveys may introduce, in which library staff may be inclined to respond to questions about the potentially sensitive topic of patron data stewardship in ways that reflect more favorably on their organization.   document analysis is also ideal in that its systematic, iterative nature allows for detailed investigation of complex topics (erlingsson & brysiewicz, 2017). privacy can be abstract, complex, and context-dependent, so a method that allows for a nuanced approach is ideal (bengtsson, 2016). document analysis can be implemented using both a deductive and an inductive approach. this study employed both, first using a deductive codebook based on professional guidelines to code all policies, followed by a second round of inductive coding to illuminate themes in text that remained uncoded after the initial round. because the study’s research aims focused on whether content was present or absent in each policy, the full policy was used as the unit of analysis rather than a line, paragraph, or other segment of text. deductive codebook development   the authors created an initial version of the deductive codebook based on the policy checklist document included in ala’s (2017b) online privacy tool kit. several documents in the tool kit provided guidance for shaping a privacy policy including the much more in-depth “sections to include in a privacy policy” (ala, 2014). however, the “sections to include” document focused on privacy best practices and behavior in addition to policy content. in comparison, the policy checklist provided a concrete basis for an initial list of codes that captured the spirit of the “sections to include” document while focusing on policies specifically. the authors provided the draft list of codes derived from the checklist to four colleagues who were asked to apply them to the original text of the checklist. based on their responses, the authors refined the code definitions. when colleagues who had no familiarity with the privacy tool kit documents could apply the codes with a high level of accuracy, the draft codebook was considered complete.   the authors then piloted the draft codebook on five policies outside of the study. they met to review this initial coding, resolve discrepancies, and adjust definitions in the codebook accordingly.   deductive codebook application   once the initial codebook was created, the authors each coded individual copies of half of the study policies. they did so by highlighting pdf documents of each policy and labeling highlighted text with the appropriate code. a spreadsheet was used to indicate whether each policy contained a given code, and the authors’ results were compared to determine agreement at the policy level. the authors then met to resolve any discrepancies and arrive at a final consensus on which policies contained which codes. initial agreement at the policy level was 88% after coding half of the policies. for individual codes where agreement was below 80%, the authors adjusted code definitions for clarity. they then proceeded to code the second half of the study policies with the updated codebook. this time, initial agreement before resolving discrepancies was 92%. they once again adjusted definitions and resolved discrepancies. using this consensus-based method, the authors ensured that the codebook accurately described the policies to which it had been applied. the final version of the codebook is available in appendix a.   a potential limitation of this study that bears mentioning here is the lack of a definition for which types of privacy policies to analyze in this study. any policy labeled as a privacy policy or statement was included to accurately reflect where a given library did address privacy concerns in some form. however, policies pertaining only to the website (15% of the policies in this study), will naturally include fewer checklist items. because the ala guidelines were written to address policies that cover all library operations, codes that fall outside of these narrower policies’ scopes will appear less frequently across all policies, perhaps resulting in a more negative assessment of the body of policies as a whole. if these are the only publicly available policies on a library’s website, this lack of accessible privacy information remains a valid concern. however, future studies should define whether these service-specific or website-specific policies should be included, or whether only policies that address all library services should be evaluated.   inductive codebook development   when deductive coding was complete, the authors examined the uncoded text that remained in the policies. with the intent of identifying themes that fell outside ala’s checklist, they followed elo and kyngas’ (2008) and guest et al.’s (2012) methodology to create an inductive codebook. while guest et al. (2012) focus on applied thematic analysis, the codebook creation process for document analysis is largely the same.   the authors began by reading through the policies several times and noting any concepts that might be good candidates for codes. they then compared and combined individual lists of concepts (126, initially) and grouped similar ideas together. the initial codebook identified 20 unique codes that could be expanded or split as coding progressed, depending on their prevalence in the data.   inductive codebook application   because creating the inductive codebook was a more complex process than creating the deductive codebook (which was based on pre-existing guidelines), the authors met more frequently during coding to determine whether code definitions should be updated. policies were coded in four sessions, with the authors meeting each time to resolve discrepancies and update the codebook. initial agreement after each round of coding ranged from 89% to 93%. each time, the authors focused on clarifying definitions for codes where agreement was less than 80%. while no codes were added or removed throughout inductive coding, the definitions of several codes were updated. the final inductive codebook is available in appendix b.   results   are privacy policies in united states arl libraries readily available to patrons?   of the 99 libraries in the study population, only 9 (9%) did not have an immediately discoverable privacy policy of any kind on their website. another 11 libraries linked to their parent institutions’ privacy policies rather than library-specific policies. one library posted a privacy policy that was restricted from public view. the remaining 78 libraries provided publicly discoverable library-specific policies which the authors analyzed in this study.   most policies were available within three clicks of the libraries’ main webpages. thirty-five policies (45%) were in an “about” or similar section linked from the libraries’ main webpages. the policy itself was most frequently located either on the “about” page or in a “policies” sub-section of the “about” page. another 30 policies (38%) were linked directly from the websites’ footers, or a “policies” link included in the footers. most other policies were available via a “policies” link located somewhere on the main webpages outside of the sites’ footers. eight policies (10%) were discovered only by doing a search for “privacy” or “privacy policy” in the websites’ search function.   do policies contain each of the elements specified in the ala’s privacy and confidentiality policy checklist?   the following table shows the results of deductive coding. the full codebook is available in appendix a.     table 1 deductive coding resultsᵃ code policy count (n = 78) percent laws 71 91 limit-2 59 76 list 56 72 principles 37 47 purpose 30 38 contact 27 35 retention 23 29 security 21 27 need_to_know 18 23 vendors 17 22 purge 16 21 limit-1 16 21 access 15 19 unnecessary_records 7 9 pii_in_public 5 6 mission 4 5 review 2 3 local_server 1 1 breach 0 0 notify 0 0 ᵃfull code definitions available in appendix a.     protecting patron records   the most prevalent codes dealt with limiting the degree to which patrons’ data will be disclosed and distributed. most policies referenced state or federal laws that dictate cases in which the library must disclose records (laws, 91%). three quarters of policies also included a statement that the library would limit the degree to which patron records would be disclosed (limit-2, 76%). though not a distinct code, it is notable that 17 policies (22%) specifically referenced the united states patriot act, which complicates libraries’ protection of circulation records, and consequently the conditions for patrons’ academic freedom (asher et al., 2018; magi, 2007).   less than a quarter of policies contained statements that the library would limit collecting or monitoring patron data in the first place (limit-1, 21%). a similar number of policies address data security (security, 27%) and internal data governance (retention, 29%; purge, 21%). it was less common for policies to state that the library would avoid creating unnecessary records in the first place (unnecessary_records, 9%) or placing personally identifiable information (pii) on public view (pii_in_public, 6%).   only one policy explicitly stated that patron records will not be stored on a third-party server (local_server, 1%), a testament to the prevalence of third-party infrastructure used by academic libraries. twenty-three percent of policies indicated that only library staff with a need to access data for the purpose of providing a service would be able to access patron data (need_to_know). transparency about data collected   many policies described individual data points collected from patrons for use by the library (list, 72%). this code was applied in any case where specific data points were explicitly described. however, most policies did not contain a data dictionary or exhaustive list of the data their library obtain from patrons, and over a quarter of policies did not list any specific data points collected from patrons. it is likewise concerning that less than a quarter of policies indicated that the library ensures contracts with third-party vendors will reflect library policies and legal obligations concerning privacy (vendors, 22%).   the language in the ala (2017b, para. 7) checklist suggests that libraries should “[n]otify users whenever the library collects their personally identifiable information...”, implying a notification in real time. no policy examined for this study included a statement that patrons would be notified at the time of collection. this is an obvious challenge when library platforms or websites often collect data from or about patrons without their knowledge or consent.   policy purpose and guiding values   slightly more than one third of policies stated the policy's purpose or scope (purpose, 38%). purpose statements tended to scope the policy by audience (campus, library branch, patron type) or by resource type (all library records, electronic records, website only). it was apparent from policy content that approximately 15% of the study policies applied only to the libraries’ websites. most purpose statements indicated that the policy addressed data collected for use in providing library services. however, some addressed patrons directly, informing them about choices they could make regarding sharing their data with the library.   the principles code identified mentions of library or archives-specific documents that describe professional values. the document most frequently mentioned was the ala code of ethics. while 47% of policies referenced at least one such document, very few went on to describe how professional values relate to the mission of that specific library (mission, 5%).   do policies contain additional elements not specified in the ala checklist?   the following table shows the results of inductive coding. for the full inductive codebook, see appendix b.     table 2 inductive coding resultsᵇ code policy count (n = 78) percent library business 49 63 values 48 62 institutional policy 45 58 assessment 40 51 liability 35 45 cookies 31 40 third-party analytics 19 24 definitions 15 19 institutional data 14 18 advising 11 14 enforcement 11 14 customization/personalization 10 13 extra-institutional policies 8 10 workstation 7 9 security cameras 6 8 notify patrons of changes 5 6 children’s privacy 4 5 do not notify patrons of changes 3 4 video/image capture 2 3 social media 1 1 ᵇfull code definitions available in appendix b.     what is the nature of these additional elements?   collecting patron data to provide services   only four of the codes identified in the inductive portion of this project appeared in more than half of the privacy policies. the most common code was library business (63%), which described how collecting patron data is necessary to provide certain library services. some statements were general: “we use this information to maintain your library account and to provide services to you” (duke university libraries, 2013, para. 5), while other statements described the data points necessary to supply a specific service. the library business code often coincided with the deductive list code and provided additional context about how various data were used to provide a service.   guiding values   a high percentage of policies asserted the libraries’ belief in or support for concepts such as confidentiality, intellectual freedom, academic freedom, or privacy itself (values, 62%). these statements often coincided with the policy’s scope and the ethical and legal documents guiding its creation. most policies, however, did not provide definitions for these terms (definitions, 19%).   compliance with institutional policy   just over half of the policies in this study referenced either another library policy, or a policy from the library’s parent institution (institutional policy, 58%). compliance with the parent institution’s privacy policy was the most common occurrence. other mentions included institutional policies that addressed acceptable use of electronic resources, information technology issues, or handling student records. only 8 policies referenced a policy outside their own institution (extra-institutional policy, 10%). several of these addressed the handling of medical or academic records, while others referenced a privacy policy from another institution that the library had used as a model.   limitations on protecting patron data   just under half of policies referenced contexts to which the policy did not extend (liability, 45%). this most often coincided with information on third-party platforms that the library provided access to but did not maintain. seventeen libraries stated that they negotiated with vendors to ensure privacy protection on third-party platforms. somewhat surprisingly, 21 policies (27%) stated either that the library did not negotiate with vendors for these protections or stated that these external platforms were beyond the library’s control. twelve of the policies that included the liability code (15% of all study policies) did not mention vendors specifically but stated that the library policy did not apply to linked external sites or platforms. they urged patrons to read these platforms’ privacy policies for themselves.   assessment and invisible data collection   assessment, cookies, and third-party analytics are discussed in 51%, 40%, and 24% of policies, respectively. all three codes were mentioned in the context of what was broadly termed “continuous improvement” and explained how data collected in the process of using the library may be used to improve library services. contexts included troubleshooting technical issues, making purchasing decisions, and ensuring compliance with policies on the acceptable use of library resources. statements addressing third-party analytics most often referenced google analytics and indicated that this data were used to understand trends in library usage or to make improvements to the website. ten policies stated that patron-provided data could be used to create a personalized web browsing experience (customization/personalization, 13%).   in a small number of cases, the assessment code referenced data collected explicitly through surveys or focus groups. four policies stated that they obtained permission from their institutional review board (irb) or similar oversight office to use patron data in assessment projects. though references to assessment frequently indicated that data would be stored securely or de-identified, only two policies indicated that patron consent would be obtained during the data collection process. additionally, despite 76% of policies indicating that the library would limit sharing patrons’ data beyond the library, 14 policies stated that the library could provide patron data to or receive patron data from the library’s parent institution (institutional data, 18%).   educating patrons about privacy   eleven policies recommended some form of action patrons could take to protect their privacy such as logging off public workstation computers, guarding personal information on shared software platforms, and creating sound passwords (advising, 14%). in some cases, the policy acknowledged choices patrons could make to limit sharing their data: “if you do not want your email address released in response to a public records request, do not send electronic mail to the university” (university of florida george a. smathers libraries, 2016, para. 11); “if you are concerned about someone else seeing a list of what you are reading or searching for, the safest step is to not choose this option” (university of arizona libraries, 2016, para. 8).   library accountability and policy enforcement   of the 78 policies analyzed, only 11 included a statement about how the policy would be enforced (enforcement, 14%). while five policies indicated that patrons would be made aware of changes to the policy (notify patrons of changes, 6%), another three explicitly stated patrons would not be notified of changes (do not notify patrons of changes, 4%).   discussion and limitations   in response to this study's aims, the authors found that a majority of arl institutions (79%) do provide some type of privacy policy to their patrons. however, these policies demonstrate low compliance with ala guidelines. while all but two deductive codes were present in at least one policy, none of the 78 policies in this study contained all 20 checklist items, and only 6% contained more than half. no policy contained more than 75% of the content suggested by ala. it should not be assumed that because a policy contains more checklist items, that institution better implements or enforces privacy. however, if one purpose of a policy is to inform patrons about how their data are collected and used, it is concerning that more policies do not follow the guidelines in a comprehensive way.   this lack of guideline coverage may be due in part to policy length. the median number of sentences in the policies analyzed was 16, with 36% containing 10 sentences or fewer. short policies may contain valuable details pertinent to patrons, but a policy 10 sentences in length is unlikely to address data collection and handling in depth. as mentioned in the methodology section, this may be due in part to the fact that some of these policies do not cover all library operations. however, there is little difference between providing patrons with a privacy policy that only covers certain services or the website and providing them with a library-wide policy that does not adequately cover uses of their information in depth.   this study’s final aim was to uncover whether policies contain themes beyond what ala guidelines recommend. the significant amount of uncoded text that remained when deductive coding was complete indicated that there was more to uncover in these policies, and inductive coding revealed that there were indeed common threads across policies which were not explicitly addressed in the guidelines. while some themes simply provided additional detail on items included in the policy checklist, several codes touched on entirely new topics. for example, the assessment and institutional data codes addressed library-initiated data collection that occurred in many cases without the patrons’ knowledge; and security camera and video/image capture addressed explicit collection of patron data in physical library spaces.   since this study’s methodology was created in 2019, ala has replaced the 2017 policy checklist (and several of its other privacy-related documents) with an updated privacy policy field guide (ala, 2021c). the 2021 field guide does address some of the areas mentioned above, including cookies, data encryption, network security, and facial recognition software, as well as how to spot red flags in vendor policies. it urges policy makers to “[r]emind users that their information is confidential, but also tell them who has access to it at your library” (ala, 2021c, p. 18).   while the privacy policy field guide includes promising updates, based on the policies analyzed in this study, there is still room to improve the guidelines by addressing assessment projects and institutional data sharing in more detail. this is particularly true of library-initiated analyses that use patron data for purposes other than the one data was originally collected for. this lack of explicit transparency contributes to what affonso and sant’ana (2018) have referred to as information asymmetry, a power imbalance in which the patron has no ability to consent to or control uses of data about themselves, since they have no knowledge that data collection or data sharing is occurring.   a limitation that should be addressed in future research on this topic relates to content analysis as a methodology. as with many qualitative approaches, document analysis is typically used alongside additional methods to triangulate the validity of findings (bowen, 2009). ideally, another research method such as interviews could provide additional context for the creation and implementation of privacy policies that a static document does not allow. future research could include interviews or focus groups that would address the decisions that go into policy creation, as well as discussing whether alternate means may be appropriate for informing patrons about the uses of their data.   conclusion   both academic library privacy policies and professional guidelines appear to be evolving to address emerging concerns such as surveillance technology, data collection via vendor platforms, and data sharing with institutional partners. while the breadth of detail included across all policies in this study was encouraging, very few individual policies addressed most of the content suggested by professional guidelines in depth. even with an abundance of privacy-related guidelines available from sources like ala, many policies still leave patrons without the detail required to understand how the library collects and uses their data. additionally, the inclusion of assessment projects and sharing data with institutional partners in the policies analyzed indicates that current guidelines may benefit from expanding to address library-initiated projects in more detail.   author contributions   greta valentine: conceptualization, methodology, formal analysis, writing – original draft kate barron: formal analysis, writing – review & editing   acknowledgements   the authors wish to thank the institute for research design in librarianship (irdl) for valuable input that shaped the research design of this study.   references   affonso, e. p., & sant’ana, r. c. g. (2018). privacy awareness issues in user data collection by digital libraries. ifla journal, 44(3), 170–182. https://doi.org/10.1177/0340035218777275   american library association. (2014). privacy tool kit. ala intellectual freedom committee. https://alair.ala.org/handle/11213/16714   american library association. (2017a). privacy. https://web.archive.org/web/20171101171848/https://www.ala.org/advocacy/privacy   american library association. (2017b). privacy and confidentiality policy. https://web.archive.org/web/20170322042558/http://www.ala.org/tools/challengesupport/privacypolicy   american library association. (2017c, april 28). developing or revising a library privacy policy. https://web.archive.org/web/20210428022310/http://www.ala.org/advocacy/privacy/toolkit/policy#sectionstoinclude   american library association. (2021a). privacy. https://www.ala.org/advocacy/privacy   american library association. (n.d.). privacy field guides for libraries. privacy policies. https://libraryprivacyguides.org/privacy-policies/   asher, a. d. (2017). risk, benefits, and user privacy: evaluating the ethics of library data. in b. newman & b. tijerina (eds.), protecting patron privacy: a lita guide. (pp. 43–56). rowman & littlefield. https://hdl.handle.net/2022/22035   asher, a., briney, k., gardner, g., hinchliffe, l., levernier, j., nowviskie, b., salo, d., & shorish, y. (2018). ethics in research use of library patron data: glossary and explainer. https://osf.io/xfkz6/   atkinson, p. a., & coffey, a. (1997). analysing documentary realities. in d. silverman (ed.), qualitative research: theory, method and practice. (pp. 45-62). sage publications.   beile, p., choudhury, k., & wang, m. c. (2017). hidden treasure on the road to xanadu: what connecting library service usage data to unique student ids can reveal. journal of library administration, 57(2), 151–173. https://doi.org/10.1080/01930826.2016.1235899   bowen, g. a. (2009). document analysis as a qualitative research method. qualitative research journal, 9(2), 27–40. http://dx.doi.org/10.3316/qrj0902027   briney, k., yoose, b., ockerbloom, j. m., swauger, s., harper, c., levernier, j., & shorish, y. (2018). a practical guide to performing a library user data risk assessment in library-built systems. https://doi.org/10.17605/osf.io/v2c3m   coombs, k. a. (2005). protecting user privacy in the age of digital libraries. computers in libraries, 25(6), 16–20.   crawford, k., & schultz, j. (2014). big data and due process: toward a framework to redress predictive privacy harms. boston college law review, 55(1), 93–128. https://lawdigitalcommons.bc.edu/cgi/viewcontent.cgi?article=3351&context=bclr   drachsler, h., & greller, w. (2016, april 25). privacy and analytics – it’s a delicate issue. a checklist for trusted learning analytics. lak '16: proceedings of the sixth international conference on learning analytics & knowledge, 89-98. https://doi.org/10.1145/2883851.2883893   duke university libraries. (2013, september 6). duke university libraries privacy statement. https://library.duke.edu/about/privacy   elo, s., & kyngäs, h. (2008). the qualitative content analysis process. journal of advanced nursing, 62(1), 107–115. https://doi.org/10.1111/j.1365-2648.2007.04569.x   erlingsson, c., & brysiewicz, p. (2017). a hands-on guide to doing content analysis. african journal of emergency medicine, 7(3), 93-99. https://doi.org/10.1016/j.afjem.2017.08.001   farkas, m. (2018). we can, but should we? when trends challenge our professional values. american libraries magazine, 49(3/4), 46-47. https://americanlibrariesmagazine.org/2018/03/01/learning-analytics-we-can-but-should-we/   guest, g., macqueen, k. m., & namey, e. e. (2012). applied thematic analysis. sage publications.   jones, j.  l. (2010). using library swipe-card data to inform decision making. university library faculty presentations, 21, 1-9. https://scholarworks.gsu.edu/univ_lib_facpres/21   jones, k. m. l., & salo, d. (2018). learning analytics and the academic library: professional ethics commitments at a crossroads. college & research libraries, 79(3), 304-323. https://doi.org/10.5860/crl.79.3.304   kritikos, k. c., & zimmer, m. (2017). privacy policies and practices with cloud-based services in public libraries: an exploratory case of bibliocommons. journal of intellectual freedom & privacy, 2(1), 23-37. http://dx.doi.org/10.5860/jifp.v2i1.6252   magi, t. j. (2007). the gap between theory and practice: a study of the prevalence and strength of patron confidentiality policies in public and academic libraries. library & information science research, 29(4), 455-470. https://doi.org/10.1016/j.lisr.2007.07.001   magi, t. j. (2010). a content analysis of library vendor privacy policies: do they meet our standards? college & research libraries, 71(3), 254-272. https://doi.org/10.5860/0710254   nichols hess, a., laporte-fiori, r., & engwall, k. (2015). preserving patron privacy in the 21st century academic library. the journal of academic librarianship, 41(1), 105-114. https://doi.org/10.1016/j.acalib.2014.10.010   oakleaf, m. (2010). the value of academic libraries: a comprehensive research review and report. association of college & research libraries. http://www.ala.org/acrl/sites/ala.org.acrl/files/content/issues/value/val_report.pdf   o’brien, p., young, s. w. h., arlitsch, k., & benedict, k. (2018). protecting privacy on the web: a study of https and google analytics implementation in academic library websites. online information review, 42(6), 734-751. http://dx.doi.org/10.1108/oir-02-2018-0056   pekala, s. (2017). privacy and user experience in 21st century library discovery. information technology and libraries, 36(2), 48-58. https://doi.org/10.6017/ital.v36i2.9817   perry, m. r., briney, k. a., goben, a., asher, a., jones, k. m. l., robertshaw, m. b., & salo, d. (2018). spec kit 360: learning analytics. https://publications.arl.org/learning-analytics-spec-kit-360/   prindle, s., & loos, a. (2017). information ethics and academic libraries: data privacy in the era of big data. journal of information ethics, 26(2), 22-33. https://www.proquest.com/scholarly-journals/information-ethics-academic-libraries-data/docview/2027533656/se-2?accountid=10920   rubel, a., & jones, k. m. l. (2016). student privacy in learning analytics: an information ethics perspective. the information society, 32(2), 143-159. https://doi.org/10.1080/01972243.2016.1130502   sturges, p., davies, e., dearnley, j., iliffe, u., oppenheim, c., & hardy, r. (2003). user privacy in the digital library environment: an investigation of policies and preparedness. library management, 24(1/2), 44-50. http://dx.doi.org/10.1108/01435120310454502   sutlieff, l., & chelin, j. (2010). ‘an absolute prerequisite’: the importance of user privacy and trust in maintaining academic freedom at the library. journal of librarianship and information science, 42(3), 163-177. https://doi.org/10.1177/0961000610368916   tenopir, c. (2010). measuring the value of the academic library: return on investment and other value measures. the serials librarian, 58(1-4), 39-48. https://doi.org/10.1080/03615261003623005   university of arizona libraries. (2020, august 24). privacy statement. university libraries. https://web.archive.org/web/20201021081113/https://new.library.arizona.edu/policies/privacy   university of florida george a. smathers libraries. (2020). privacy policy. https://web.archive.org/web/20200909053228/https://uflib.ufl.edu/about/user-policies/privacy-policy/   vaughan, j. (2020). library privacy policies. library technology reports, 56(6), 5–53. https://doi.org/10.5860/ltr.56n6   voeller, s. (2007). privacy policy assessment for the livingston lord library at minnesota state university moorhead. library philosophy and practice, 151. https://digitalcommons.unl.edu/libphilprac/151/   yoose, b. (2018, february 15). data analytics and patron privacy in libraries: a balancing act. https://osf.io/xb4mf/   appendix a deductive codebook label definition qualifications/exclusions access states that patrons have the right to access, see, or update personally identifiable information (pii) that the library collects about them deals with patrons accessing data about themselves. does not apply to the library releasing information to third parties. breach states that the library will notify patrons in the event of a data breach deals with the inadvertent release of patron pii or records. contact provides a means for the patron to contact the library regarding the policy must include actual contact information (phone number, email address, chat link) rather than just a name or title for patrons to contact. we did not count contact information in the footers of websites. laws policy includes references to any federal, state, or local laws that impact the policy can include generic reference to being in compliance with “state and federal laws”, or similar limit-1 states that the library will limit the degree to which patrons’ pii will be monitored/collected deals with how data are obtained from the patron limit-2 states that the library will limit the degree to which patrons' pii will be disclosed/distributed deals with how data are disclosed by the library. can include instances of aggregating data (see nc state) list lists the personally identifiable information (pii) the library will be collecting from patrons when they use library services can apply to individual instances of pii collection – does not have to reference a comprehensive list. must refer to pii. can list specific pii or library records that include pii. applies to sections that list electronic information such as ip address, browsing history etc., even if the policy does not refer to this information as pii. local_server states that patron records will remain on a local server rather than being exported to the cloud or a third-party server   mission explains how protecting user privacy and confidentiality relates to the mission of the library relates only to the specific mission of that particular library, not ala/professional ethics need_to_know states that only authorized library staff will access patron records can refer to library employees or university employees as long as the case for accessing the records is clear. use only for blanket statements, not particular instances. notify states that the library will notify patrons when data are being collected from them refers to or states that the library will notify patrons in real time when information is being collected from them. should not be used for policy content that only addresses opting in or out of using cookies. pii_in_public states that the library will avoid placing patron records in public view   principles refers to principles on which the library's commitment to protecting privacy is based relates to the principles of librarianship (i.e., references to the ala code of ethics, intellectual freedom, etc.) can include generic reference to library principles or professional documents. purge states that the library will regularly purge identifiable patron records can be a general statement about purging records after a set period of time, or an example of removing specific records. must indicate a removal or deletion of records, as opposed to retention, which indicates cases where records are kept. purpose states the purpose of the policy states the purpose of the policy. may include scope: "this policy applies to x campus, x patrons, x resources etc." retention states that the library will not retain patron records that are not needed for efficient operation of the library must indicate keeping records and the purpose for keeping those records, as opposed to purge, which indicates a removal or deletion of records review states how often the policy will be reviewed must state actual cycle or dates when policy is updated. does not apply to date policy was last updated, or statements that the policy is updated periodically. security states that patron pii will be stored securely must refer to patron records unnecessary_records states that the library will avoid creating unnecessary records about the patron   vendors states that the library will ensure contracts and licenses with vendors will reflect library policies and legal obligations concerning privacy must refer to licenses negotiated with library vendors     appendix b inductive codebook label definition advising policy advises users regarding privacy best practices assessment policy refers to library-initiated analysis for continuous improvement children's privacy policy describes special privacy protections for children cookies policy describes or defines cookies or their implications for privacy and personalization customization/personalization policy describes options for submitting personal information in order to customize the use of a digital library service definitions policy includes definitions of terms relating to patron privacy, confidentiality and patron rights. also includes statements informing the patron how these concepts relate to one another. do not notify patrons of changes policy states that privacy policy may change without notice to patrons enforcement policy describes how the libraries will enforce the policy, including how they will deal with violations or prevention measures such as privacy audits extra-institutional policies policy refers to non-university policies or other library policies. codes of ethics should be coded principles. institutional data policy describes situations in which the library receives or shares patron data with other campus units institutional policies policy refers to library's other policies or parent institution’s policies liability policy describes contexts to which the policy does not extend. includes statements that library operations or systems may not be 100% secure. library business policy describes data collected or used to provide services or facilitate smooth operations notify patrons of changes policy states library will notify patrons of changes to privacy policy. includes cases where the library notifies individual patrons, or where they post changes in a public place before the changes take effect. security cameras policy describes library use of security cameras social media policy describes use of patron information or images on their social media accounts third-party analytics policy mentions use of google analytics or other third-party web analytics on the library website. policy must identify the tool as a web analytics tool. values policy asserts belief in or support for professional library values such as privacy, confidentiality, intellectual freedom, or academic freedom. policy states clear position or attitude towards upholding these values. video/image capture policy describes restrictions on capturing image or video of patrons in the library workstation policy describes use of on-site computers or workstations     microsoft word news_editor.doc evidence based library and information practice 2007, 2:3  133 evidence based library and information practice     news    evidence based library and information practice seeks associate editor (evidence  summaries)       © 2007 glynn. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    evidence based library and information  practice (eblip) is seeking an associate  editor (evidence summaries) to join its  editorial team.      evidence summaries (es) provide critical  appraisal syntheses for specific research  articles.  these research synopses provide  readers with information regarding the  original research article’s validity and  reliability, thus providing information on  the presence or absence of evidence with  which to make informed decisions.     the evidence summaries are a key  component of this journal.  es are written by  a team of experienced authors who follow a  strict format to ensure consistency.  all es  undergo peer review to ensure quality. up  to ten es are published in every issue of  eblip.    the associate editor (evidence summaries)  is responsible for:    • monitoring a set of top research  journals in librarianship for new  research articles  • assigning articles to es writers  • seeing es through all stages of the  publication process including  assigning peer reviewers &  copyeditors  • working closely with other editorial  team members to ensure a  consistent, high‐quality journal  • maintaining a reliable, experienced  es writing team with a variety of  areas of expertise  • maintaining es writing guidelines  and acting as a support for the es  writing team.    the ideal candidate will be well‐versed in  evidence based practice and critical  appraisal.  this position requires dedicated  time on a regular basis and is labour  intensive.  it is therefore essential that  interested persons ensure available time to  devote to this position prior to applying.   this is an unpaid position.    interested persons should send their  resumes by october 1, 2007, to:    lindsay glynn  editor‐in‐chief  lglynn@mun.ca  (709) 777‐6026  http://creativecommons.org/licenses/by/2.0 mailto:lglynn@mun.ca evidence summary   multidisciplinary databases outperform specialized and comprehensive databases for agricultural literature coverage   a review of: ritchie, s. m., young, l. m., & sigman, j. (2018). a comparison of selected bibliographic database subject overlap for agricultural information. issues in science and technology librarianship, 89. http://doi.org/10.5062/f49z9340     reviewed by: melissa goertzen consultant and information manager halifax, nova scotia, canada email: goertzen.melissa@gmail.com   received: 1 mar. 2019                                                                    accepted: 23 apr. 2019      2019 goertzen. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29561     abstract objective – to determine the most comprehensive database(s) for agricultural literature searching.   design – data collection and analysis was conducted using a modified version of the bibliography method, overlap analysis, chi square tests, and data visualization methods.   setting – an academic library in the u.s.   subjects – eight commonly used bibliographic databases, including comprehensive agricultural indexes (agricola, agris, and cab abstracts), specialized databases (biosis previews and fsta), and multidisciplinary databases (google scholar, scopus, and web of science).   methods – the researchers selected three review articles that represented sub-topics within the field of agriculture. sources listed in the bibliographies of the three review articles were used to build a bibliographic citation set for analysis.   using a modified version of the bibliography method, 90 citations were randomly selected from the above-mentioned citation set. researchers then turned to the 8 selected databases and searched for all 90 citations in each platform. search queries were crafted in two ways: unique title strings in quotation marks and combinations of terms entered into the “title”, “keyword”, “journal source”, and “author” fields. citations were considered to be covered in a database if the full bibliographic record was located using the above-mentioned search strategy.   next, chi square tests were used to evaluate if the expected number of citations from the sample group were found in each database or if the frequency differed between the eight databases. the overlap analysis method provided numerical representation of the degree of similarity and difference across the eight databases. finally, data visualizations created in excel and gephi enhanced comparisons between the eight databases and highlighted differences that were not obvious based solely on the analysis of numerical data.   main results – researchers found that comprehensive databases (agricola, agris, and cab abstracts) were not in fact comprehensive in their coverage of agricultural literature. however, the results suggested that cab abstracts was more comprehensive than agricola or agris, particularly in regard to its coverage of the sub-topics “agronomy” and “meat sciences”. however, coverage of the sub-topic “sustainable diets” lagged behind multidisciplinary databases, which may be explained by the fact that the topic is interdisciplinary in nature. the superior coverage of cab abstracts over other comprehensive databases is consistent with findings reported by kawasaki (2004).   the analysis of specialized databases (biosis previews and fsta) suggested that citations within the scope of the database were covered very well, while those out of scope were not. for instance, the sub-topics “sustainable diets” and “meat science” are out of scope of the biological sciences and thus, were not well covered in biosis.   the multidisciplinary databases (google scholar, scopus and web of science) provided the most comprehensive coverage agricultural literature. all three databases covered most citations included in the data set. however, researchers noted that all three databases provided weak coverage of trade published items, books, or older journals.   conclusion – the study found that multidisciplinary databases provide close to full coverage of agricultural literature. in addition, they provide the best access to content that is interdisciplinary in nature. specialized and comprehensive databases are recommended when research topics are within the scope of the database. also, they best support in-depth projects such as bibliographies or comprehensive review articles.   commentary   in the current information landscape, academic librarians are called upon to find innovative ways to do more with less. it is essential that librarians understand the value of information resources and how they support research and learning activities.   the paper under review discusses a sustainable methodology that academic librarians can utilize to analyze database content. specifically, it provides an effective strategy to identify areas of information coverage and overlap, determine the strengths and limitations of database content, and promote discovery of literature in a specific field of study. at the same time, the methodology allows for the standardized comparison of databases, which feeds into the notion of evidence-based collection development. the results of the comparison can easily provide a baseline for future evaluations that inform decisions regarding the renewal or cancellation of information products. the results also build on a previous study by kawasaki (2004) and provide insight into how the information landscape in the field of agriculture has evolved over the past 15 years.   the reviewer evaluated the paper using the “evaluation tool for bibliometric studies” (perryman, 2009). the strength of the piece is in its clarity of language, organization, and detailed descriptions of the methodology and data analysis activities. the discussions of how citations were collected, the strengths and limitations of statistical data, and the ways that data visualization tools fill knowledge gaps present in numerical data provide a roadmap for other information professionals wishing to examine content coverage. the researchers also demonstrate how study findings enhance knowledge of the collection. for instance, it was determined that because of the broad content coverage in multidisciplinary databases, many researchers do not require specialized or comprehensive databases to complete projects; these sources are better suited to in-depth topics that are well within the scope of the databases’ subject focus.   one limitation of the paper is its focus on numerical data. it would have been interesting to learn if researchers primarily use multidisciplinary databases because of the extensive content coverage or whether they prefer to use specialized or comprehensive databases due to their focus on specific subject areas. essentially, a discussion of this nature would have provided insight into whether library users value databases because of their content coverage or if there are other factors involved, such as the functionality of a database. however, the researchers do acknowledge this limitation and state that there is value in conducting a future study that examines how library users search for content in databases.   overall, the researchers present a strong study that provides value to academic librarians working in the area of collection development. the paper presents a low-cost and sustainable methodology that promotes standardized database evaluations at institutions. it would be interesting to read future studies about how this methodology supports evidence-based collection development decisions at other institutions.   references   kawasaki, j. l. (2004). agriculture journal literature indexed in life sciences databases. issues in science & technology librarianship 40. https://doi.org/10.5062/f4m61h61   perryman, c. (2009). evaluation tool for bibliometric studies. retrieved 20 march 2019 from http://libjournalclub.pbworks.com/f/journal%20club%20jan%2020%202011.pdf   ritchie, s. m., young, l. m., & sigman, j. (2018). a comparison of selected bibliographic database subject overlap for agricultural information. issues in science and technology librarianship, 89. https://doi.org/10.5062/f49z9340     research article   connecting users to articles: an analysis of the impact of article level linking on journal use statistics   michelle swab public services librarian health sciences library memorial university st. john’s, newfoundland, canada email: mswab@mun.ca   received: 29 july 2019                                                                   accepted: 15 oct. 2019      2019 swab. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29613     abstract   objective – electronic resource management challenges and “big deal” cancellations at one canadian university library contributed to a situation where a number of electronic journal subscriptions at the university’s health sciences library lacked article level linking. the aim of this study was to compare the usage of journals with article level linking enabled to journals where only journal level linking was available or enabled.   methods – a list of electronic journal title subscriptions was generated from vendor and subscription agent invoices. journal titles were eligible for inclusion if the subscription was available throughout 2018 on the publisher’s platform, if the subscription costs were fully funded by the health sciences library, and if management of the subscription required title-by-title intervention by library staff. of the 356 journal titles considered, 302 were included in the study. negative binomial regression was performed to determine the effect of journal vs. article level linking on total counter journal report 1 (jr1) successful full-text article requests for 2018, controlling for journal publisher, subject area, journal ranking, and alternate aggregator access.                                                                                                                          results – the negative binomial regression model demonstrated that article level linking had a significant, positive effect on total 2018 jr1 (coef: 0.645; p < 0.001). article level linking increased the expected total jr1 by 90.7% when compared to journals where article level linking was not available or enabled. differences in predicted usage between journals with article level linking and those without article level linking remained significant at various journal ranking levels. this suggests that usage of both smaller, more specialized journals (e.g., journal of vascular research) and larger, general journals (e.g., new england journal of medicine) increases when article level linking is enabled.   conclusions – this study provides statistical evidence that enabling article level linking has a positive impact on journal usage at one academic health sciences library. although further study is needed, academic libraries should consider enabling article level linking wherever possible in order to facilitate user access, maximize the value of journal subscriptions, and improve convenience for users.     introduction   library link resolver systems are designed to facilitate seamless connections between full-text journal content and article databases and library discovery layers. when working optimally, link resolvers connect users directly to a full-text html or pdf version of a particular journal article. linkage failures remain common in libraries, however, despite attempts to make improvements (stuart, varnum, & ahronheim, 2015).   one form of suboptimal linking occurs when link resolvers connect at the journal level rather than the article level. in journal level linking, a link to a particular article resolves to the table of contents or homepage of a journal, rather than the article itself. the user must then browse to the volume and issue of interest, or search the journal platform for the article. article level linking functionality depends on the availability of accurate linking parser information in an institution’s link resolver software, as well as support for link resolvers from journal vendors.   at memorial university’s health sciences library, many journal titles lacked article level linking throughout 2018. a number of factors contributed to this situation, including the cancellation of several “big deal” publisher packages during the preceding years. big deal journal packages are becoming financially unsustainable for many institutions, but provide greater efficiencies with regard to electronic resource management processes (cleary, 2009). in some cases, big deal publisher package linking can be activated with a few mouse clicks. in contrast, creating and maintaining link resolver information for individual journal titles can be labour intensive. library personnel must select individual titles to activate, edit journal holdings information to match institutional entitlements, and ensure that linking information is accurate.   big deal cancellations increased the number of individual journal subscriptions at memorial university libraries, thus creating additional burdens on acquisitions personnel (ambi, morgan, alcock, & tiller-hacket, 2006). moreover, memorial university libraries had transitioned to a new library services platform during this time period. the transition meant that many electronic collections required cleaning and updates, and it took some time to determine what electronic resource management workflows would work best for the institution.   while journal level linking is not ideal for users, the situation at the health sciences library created a unique opportunity to study the effects of article level linking on journal usage. article level linking was unavailable for many smaller publishers, and also for a variety of larger publishers including cambridge, oxford, wiley, springer, and elsevier. this resulted in a more wide-ranging sample than would typically be possible in an observational study of article level linking.   literature review   initially developed in the late 1990s by herbert van de sompel, link resolvers were quickly recognized as a “silver bullet solution” to the problem of context sensitive linking (mcdonald & van de velde, 2004, p. 32). link resolver tools provided seamless connections between bibliographic databases, publisher websites, and library catalogue holdings. in the following years, link resolver technologies were described as “indispensable” (singer, 2006, p. 15) and “essential” (chisare, fagan, gaines, & trocchia, 2017, p. 93) for academic libraries.   early research into openurl tools and link resolvers suggested that their implementation increased electronic journal usage. kraemer (2006) observed that electronic journals with advanced linking features were more highly used at the medical college of wisconsin libraries, while mcdonald (2007) reported that openurl resolver availability was correlated with a large and significant increase in publisher-reported electronic journal usage in a number of subject areas. early research also indicated that patrons exhibited positive attitudes towards link resolver services (eason, macintyre, & apps, 2005).    link resolvers are not the only way for library users to access journal content, however. ip-authenticated users may click on publisher-direct article links in databases such as pubmed to access content directly. ip-authenticated users may also access articles through publisher-direct article links on search engines such as google. in contrast to the usage gains and favourable patron attitudes reported during earlier research, more recent studies suggest that many library users bypass library link resolvers for publisher-direct linking. one study of health science journal usage found that publisher website usage statistics were much higher than usage reported by link resolver click-through statistics (de groote, blecic, & martin, 2013). this trend is even more apparent in a study from 2017, which reports:   on average (for the 18 months in the sample) the publishers’ full text request report was 45,512 per month. the average full text requests registered from the library discovery tool (through its url link resolver service) in the same period was 14,612 per month (i.e. publishers report 3.1 times more downloads than requested from the library discovery tool). (greenberg & bar-ilan, 2017, p. 460)   creating and maintaining link resolver tools is a complex and time-consuming process (samples & healey, 2014). journals that require title-by-title intervention demand even greater resources in terms of library staff time, so it is useful to understand the relationship between article level linking and journal usage by library patrons. if the effect size is small or insignificant, it might indicate that libraries should allocate their resources elsewhere. alternatively, if the effect size is larger, it might signal that libraries should prioritize link resolver maintenance and cleanup. to date, there appear to be no published studies that examine the impact of article level linking versus journal level linking on publisher reported usage statistics.   aims   this observational, cross-sectional study attempts to understand the effects of article level linking at one academic health sciences library. the study uses statistical modeling to compare the usage of journals with article level linking enabled to journals where only journal level linking was available. the study attempts to control for other factors which may affect journal usage.   this study’s research question is: what is the effect of article level linking vs. journal level linking on publisher reported successful full-text article requests at one academic health sciences library, controlling for journal subject area, journal ranking, publisher, and alternate aggregator access?   methods   journals were eligible for inclusion in the study if they met the following criteria: the journal subscription cost was fully funded by the health sciences library, the journal was available via a publisher-direct platform, access to the journal was available throughout 2018, and the journal was part of a selective e-collection. selective e-collections require at least some amount of title-by-title intervention during electronic resource management processes. journals available through packages that did not require title-by-title intervention were excluded from the study (e.g., lww nursing and health professions premier collection). journals partially funded by the health sciences library were excluded because the size and composition of the user group for partially funded titles (e.g., nature) varies significantly from the size and composition of the user group for fully funded titles.   a list of selective journal subscriptions requiring title-by-title intervention was developed from publisher and serial agent invoices. print and electronic issns from this list were input into the ex libris alma overlap and collection analysis module in order to generate a spreadsheet outlining electronic resource portfolio availability. duplicate journal titles, which resulted from the use of both electronic and print issns, were removed. journals missing one or more variables in the statistical model were excluded from the study (e.g., journal report 1 [jr1] was unavailable). journals that changed publishers or platforms mid-year were also excluded from the analysis, as these changes can cause significant variation in usage (bucknell, 2012). a total of 356 journal titles were considered for inclusion in this study; jr1 reporting was unavailable for 21 journal titles, journal impact factor (jif) was unavailable for 41 journal titles, and 3 journal titles changed publishers or platforms over the course of the year. the final dataset included 302 journal titles.   dependent variable: total jr1 for 2018   this study used vendor-supplied counter reports in order to measure ip-authenticated, publisher-direct journal usage by library users at memorial university. librarians frequently use counter reports to evaluate electronic resources and make subscription decisions (baker & read, 2008). project counter (counting online usage of networked electronic resources) is a non-profit member-based organization of libraries, publishers, and vendors that have developed standards and definitions for electronic resource usage data. the counter code of practice improves comparability of electronic resource usage data between vendors, although several studies suggest that vendor platform design decisions may lead to inflated usage statistics for some publishers (davis & price, 2006; kohn, 2018; wood-doughty, bergstrom, & steigerwald, 2019).   counter release 4 usage reports for january 1 through december 31, 2018 were obtained for all included journal titles. usage reports were examined for usage spikes and other indications of potential misuse (bucknell, 2012). no indications of misuse were observed.   although a number of different types of counter reports are available, jr1 was selected for this study due to its ready availability and frequent use by librarians in journal evaluation. journal report 1 indicates the number of successful monthly full-text article requests in both html and pdf format on the publisher’s website. all successful requests are included, regardless of how the request originated (e.g., library link resolver, direct to publisher links in databases or search engines, journal browsing). journal report 1 includes usage of backfile content, as well as gold open access articles. journal report 1 excludes usage of journal content through other platforms such as aggregators (journal usage statistics portal, 2013). journal report 5 (jr5), which reports monthly requests by year of publication, was also considered as a potential outcome measure. the availability of jr5 reports was more limited, however, as fewer publishers were able to provide jr5 reports. using jr1 allowed for the inclusion of more small publishers in the study.   total jr1 is a count variable, and the shape of its frequency distribution is long-tailed. dependent variables that are not normally distributed require specific considerations in statistical modelling, as described in further detail below.   independent variable: article level linking   in may 2018, journal title electronic portfolios were examined via ex libris alma to determine whether article level linking was enabled for the journal. article level link testing was also conducted in the user-facing ex libris primo interface. no substantive changes were made to portfolio linking level until mid-december 2018, when article level linking was enabled wherever possible. it was expected that the december update would not result in substantial changes in usage due to academic holidays.   control variables   a number of other factors may impact the usage of journal titles. this study attempts to control for these factors by including them in the statistical model.   journal publisher is included as a categorical variable because there is evidence that publisher platform design decisions affect usage statistics (davis & price, 2006; kohn, 2018; wood-doughty et al., 2019). publishers are anonymized within the study due to license restrictions around the sharing of usage data. it would take considerable effort to obtain permission to share usage data from each publisher included in the study, and the effort did not seem sufficiently beneficial given that publisher platform effects were not the variable of primary interest. publishers comprising less than 5% of the total number of journals included in the dataset were grouped together in an “other” category.   there is also evidence to suggest that usage statistics are impacted by academic discipline or subject area (gorraiz, gumpenberger, & schlögl, 2014; mongeon, archambault, & larivière, 2018). included journals were categorized by top-level scopus subject area. in cases where a journal was included in more than one top-level subject area, the subject area where the journal had the highest scimago ranking was selected. subject area categories comprising less than 5% of the total number of journals included in the dataset were grouped together in an “other” category.   journal usage may also be influenced by the relative size and importance of a journal. here, journal size is defined by the number of articles that the journal publishes in a given year. a 2004 study demonstrates that indicators of journal quality are correlated with journal usage in one medical library (wulff & nixon, 2004). both scimago’s scientific journal rank (sjr) and clarivate’s journal impact factor (jif) were explored as proxy measures of journal size and importance. these measures are strongly correlated, suggesting that either measure may be appropriate (elkins, maher, herbert, moseley, & sherrington, 2010). ultimately, jif was selected because it provided a slightly better model fit. finally, alternate access to journals via an aggregator database may decrease usage on publisher websites (bucknell, 2012). due to the importance of recency in medical libraries, journals were categorized as having alternate aggregator access if the embargo period for the journal title was six months or less.   because there is some evidence of home country bias in journal readership (thelwall & maflahi, 2015), a dummy variable representing canadian journal titles was initially considered for inclusion in the model. the number of canadian journal titles within the sample was small (n = 7), however, so ultimately this variable was not included.   statistical modelling   like many outcomes of interest in library and information science, the dependent variable in this study is not normally distributed (figure 1). in such cases, it is not generally appropriate to use multiple linear regression, and generalized linear models should be considered.   figure 1 distribution of jr1 2018 (dependent variable). created using 538 schemes (bischof, 2017).     a study analyzing statistical modelling of infometric data suggests that the negative binomial regression model (nbrm) may be most appropriate for infometric studies with count response variables (ajiferuke & famoye, 2015). the study dataset was modelled in stata (release 15) with both the poisson regression model and the nbrm using methods outlined by long and freese (2006). ultimately, the nbrm with robust standard errors was selected because there was significant evidence of overdispersion (g2 = 1.5e+05, p < 0.001).   results   tables 1 and 2 present summary statistics relating to variables that were included in the nbrm. for the 302 journals included in the study, the average total jr1 usage in 2018 was 554 per journal. the median jr1 usage was 234.5, with a range from 0 uses to 9202 uses. article level linking was enabled for 64% of included journals (n = 193), and journal level linking was offered for 36% journal titles (n = 109).   table 1 descriptive statistics for categorical variables variables article level linking enabled no article level linking total n % n % n % publisher publisher a 76 76.8 23 23.2 99 33 publisher b 32 100.0 0 0.0 32 11 publisher c 18 94.7 1 5.3 19 6 publisher d 21 100.0 0 0.0 21 7 publisher e 27 96.4 1 3.6 28 9 other publisher (reference category) 19 18.4 84 81.6 103 34 aggregator access available to most recent 6 months yes 40 65.6 21 34.4 61 20 no 153 63.5 88 36.5 241 80 scimago subject category biochemistry, genetics, and molecular biology 19 67.9 9 32.1 28 9 nursing 14 63.6 8 36.4 22 7 other subject 31 64.6 17 35.4 48 16 medicine (reference category) 129 63.2 75 36.8 204 68 total 193 63.9 109 36.1 302 100   table 2 descriptive statistics for continuous variables variables mean median std. dev. min. max. jr1 2018 553.97 234.50 1039.94 0 9202 subgroup article level links enabled 676.91 273 1228.04 16 9202 subgroup no article level linking 336.28 187 508.19 0 3271 jif 2017 6.57 3.75 8.92 0.42 79.26 subgroup article level links enabled 7.57 3.87 10.53 0.66 79.26 subgroup no article level linking 4.8 3.63 4.43 0.42 23.43   table 3 presents the results of the nbrm with robust standard errors. regression coefficients, z, and p values are presented. the table also includes the percent change in the expected count for each unit increase in order to assist with interpretation of the coefficients. for categorical variables that do not have defined units, the percent change in the expected count for each unit increase indicates the change that would occur when switching from one category (e.g., no article level linking enabled) to another (e.g., article level linking enabled), controlling for other variables.   table 3 results of negative binomial regression model with robust standard errors variable coefficient z p value % change in expected count for unit increase in x independent variable         article level linking enabled 0.645*** 3.819 .000 90.7 control variables         publisher publisher a 0.063 0.353 .724 6.5 publisher b -1.353*** -5.312 .000 -74.2 publisher c 0.019 0.058 .954 1.9 publisher d -0.336 -1.109 .267 -28.5 publisher e -1.055*** -4.009 .000 -65.2 other publisher (reference category) aggregator access available 0.106 0.531 .595 11.2 jif 2017 0.039*** 5.031 .000 4.0 scimago subject category biochemistry, genetics and molecular biology 0.005 0.021 .983 0.5 nursing 1.052*** 3.995 .000 186.3 other subject (non-medicine) 0.355* 2.106 .035 42.6 medicine (reference category) *p < .05; **p < .01; ***p < .001.   article level linking is shown to have a positive effect on total jr1, and the relationship is significant (p < .001). article level linking increases expected total jr1 by 90.7%, when compared to titles with journal level linking and controlling for other variables.   several control variables were statistically significant. as expected, jif had a significant, positive effect on total jr1. for every one-point increase in jif, a 4% increase in total jr1 would be expected. interestingly, only two publisher dummy variables reached significance when compared to the reference category (all other publishers). in both cases, the relationship was negative. the nursing and “other” subject category dummy variables also reached statistical significance when compared to the reference category (medicine).   in order to clarify the effects of journal level vs. article level linking at various journal ranking levels, the marginsplot stata command was used to graph predicted total jr1 over the 95% range of jif 2017, holding other variables constant. figure 2 presents the predicted results for jif scores ranging from 0–20, and includes the 95% confidence interval. plotting the predicted margins results demonstrates that the difference in usage between journal level and article level linking remains significant at both lower and higher levels of journal impact. this suggests that usage of both smaller, more specialized journals and larger journals increases when article level linking is enabled.   figure 2 adjusted jr1 predictions by linking level, including 95% ci. created using 538 schemes (bischof, 2017).   discussion   amid emerging priorities in academic libraries such as research data services, digital humanities support, and research impact assessment (lewis & proffitt, 2019), it is important to examine the value and impact of current library practices (booth, 2006). what practices should be prioritized, and what should libraries stop doing?   the circumstances surrounding this study provided a unique opportunity to assess the impact of article level linking versus journal level linking on journal usage. although maintaining article level linking can be a relatively simple process for packages and consortial purchases, configuring and maintaining link resolver information for a multitude of individual journal titles can be time consuming and labour intensive. past research provides evidence that a high percentage of journal usage originates outside of link resolver pathways (greenberg & bar-ilan, 2017), so it is important to examine the impact of article level linking to determine whether efforts to maintain link resolver information for individual titles are worthwhile.   the results of the study demonstrate that article level linking has a large, statistically significant effect on journal usage at one academic health sciences library. enabling article level linking increases journal usage by 90.7%. contextualizing the size of the effect is challenging due to the paucity of quantitative research on factors contributing to journal usage in academic libraries. nevertheless, it is difficult to think of other interventions that the library might be able to implement that would increase journal usage so substantially.   enabling article level linking wherever possible also improves the library user experience. convenience is an extremely important factor in information seeking behaviours (connaway, dickey, & radford, 2011); enabling article level linking wherever possible provides more convenient pathways for library users. for example, a report by a library link resolver implementation team at the university of michigan noted that journal level linking “requires a substantial increase in user attention and effort” (varnum et al., 2016, p. 21). with article level linking, average time elapsed to first user interaction with the article was 35 seconds; with journal level linking, this time increased to 2 minutes, 45 seconds. another study indicated that having to perform additional steps to locate articles on publisher websites can result in students becoming confused and overwhelmed (mann & sutton, 2015). finally, an early study of the sfx link resolver observed that journal level linking is a source of user frustration (wakimoto, walker, & dabbour, 2006).   overall, this study provides evidence to support the importance of enabling article level linking at memorial university’s health sciences library. although enabling article level linking for individual journal titles is labour intensive, it increases journal usage and thus maximizes the value of library subscriptions. furthermore, enabling article level linking increases convenience for users and lowers the click burden. providing convenient access to articles should be prioritized by libraries, particularly given the rise of highly convenient, alternative access avenues such as sci-hub (nicholas et al., 2019).   while not of primary interest, several other control variables included in the model were statistically significant. in comparison with the reference category that included all other publishers, publisher b and publisher e demonstrated a strong, negative impact on expected usage counts when controlling for all other variables. unlike other recent work (wood-doughty et al., 2019), this study did not observe evidence of usage inflation by publishers. however, it should be noted that this study was smaller, and that publisher inflation was not of primary concern. the other control variable of note was the nursing subject category. in comparison with the reference category (medicine), the nursing subject category increased expected usage by 186%. while it is difficult to understand this result without further study, it may be related to the number of subscribed journals per user in the medicine and nursing programs. the study included far fewer journals in the nursing subject category than in the medicine subject category, which is likely related to the greater availability of, and greater user requirements for, specialized medical journals.   this study was undertaken at one academic health sciences library, and results may or may not be replicable at other libraries or institutions. while it seems likely that article level linking would increase journal usage at other institutions as well, further research investigating the size of the effect is warranted. a further limitation of the study is that it is observational in nature. like other observational studies, there may be confounding variables that have not been accounted for in the statistical model. while attempts have been made to control for various factors that could affect usage, the groups of journals with and without article level linking enabled may have been different in other ways that are not considered. for example, the number of gold open access articles in each journal may have had an impact on usage. future studies may be better positioned to control for the presence of gold open access articles due to the enactment of the counter 5 code of practice in early 2019. counter 5 journal request reports now exclude gold open access usage.   conclusions   this study analyzed the impact of article level linking on journal usage statistics at one academic health sciences library. negative binomial regression was used to examine the impact of article level linking on jr1, while controlling for journal subject area, journal ranking, publisher, and alternate aggregator access. article level linking increased total jr1 by 90.7% (p < 0.001), when controlling for all other variables. the differences between journal level linking and article level linking remained statistically significant at various journal ranking levels. this study provides evidence that article level linking should be prioritized at memorial university’s health sciences library, since it increases usage and provides greater convenience for users. although further study is needed, academic libraries should consider enabling article level linking wherever possible in order to facilitate user access, maximize the value of journal subscriptions, and improve convenience for users.   acknowledgments   thanks to sue fahey, christine doody, dion fowlow, and kim whitfield for generously sharing their knowledge and expertise in electronic resources management with me. thanks also to kristen romme for her thoughtful comments on the initial draft of this paper.   references   ajiferuke, i., & famoye, f. (2015). modelling count response variables in informetric studies: comparison among count, linear, and lognormal regression models. journal of informetrics, 9(3), 499-513. https://doi.org/10.1016/j.joi.2015.05.001 ambi, a., morgan, p., alcock, e., & tiller-hackett, a. (2016). using data to break apart journal packages. evidence based library and information practice, 11(4). https://doi.org/10.18438/b81k9b baker, g., & read, e. j. (2008). vendor-supplied usage data for electronic resources: a survey of academic libraries. learned publishing, 21(1), 48-57. https://doi.org/10.1087/095315108x247276 bischof, d. (2017). update: a final stata gift: 538 schemes. retrieved from https://danbischof.com/2017/09/05/a-final-stata-gift-538-schemes/ booth, a. (2006). clear and present questions: formulating questions for evidence based practice. library hi tech, 24(3), 355-368. https://doi.org/10.1108/07378830610692127 bucknell, t. (2012). garbage in, gospel out: twelve reasons why librarians should not accept cost-per-download figures at face value. the serials librarian, 63(2), 192-212. https://doi.org/10.1080/0361526x.2012.680687 chisare, c., fagan, j. c., gaines, d., & trocchia, m. (2017). selecting link resolver and knowledge base software: implications of interoperability. journal of electronic resources librarianship, 29(2), 93-106. https://doi.org/10.1080/1941126x.2017.1304765 cleary, c. (2009). why the “big deal” continues to persist. the serials librarian, 57(4), 364-379. https://doi.org/10.1080/03615260903206853 connaway, l. s., dickey, t. j., & radford, m. l. (2011). “if it is too inconvenient i’m not going after it:” convenience as a critical factor in information-seeking behaviors. library & information science research, 33(3), 179-190. https://doi.org/10.1016/j.lisr.2010.12.002 davis, p. m., & price, j. s. (2006). ejournal interface can influence usage statistics: implications for libraries, publishers, and project counter. journal of the american society for information science and technology, 57(9), 1243-1248. https://doi.org/10.1002/asi.20405 de groote, s. l., blecic, d. d., & martin, k. (2013). measures of health sciences journal use: a comparison of vendor, link-resolver, and local citation statistics. journal of the medical library association, 101(2), 110-119. https://doi.org/10.3163/1536-5050.101.2.006 eason, k., macintyre, r., & apps, a. (2005). a ‘joined-up’ electronic journal service: user attitudes and behaviour. in p. brophy, j. craven & m. markland (eds.), libraries without walls 6: evaluating the distributed delivery of library services. (pp. 63-70). london: facet. elkins, m. r., maher, c. g., herbert, r. d., moseley, a. m., & sherrington, c. (2010). correlation between the journal impact factor and three other journal citation indices. scientometrics, 85(1), 81-93. https://doi.org/10.1007/s11192-010-0262-0 gorraiz, j., gumpenberger, c., & schlögl, c. (2014). usage versus citation behaviours in four subject areas. scientometrics, 101(2), 1077-1095. https://doi.org/10.1007/s11192-014-1271-1 greenberg, r., & bar-ilan, j. (2017). library metrics – studying academic users’ information retrieval behavior: a case study of an israeli university library. journal of librarianship and information science, 49(4), 454-467. https://doi.org/10.1177/0961000616640031 journal usage statistics portal. (2013). jr1, jr1a and jr1 goa reports. retrieved from https://jusp.jisc.ac.uk/guides/journal-reports/jr1.html kohn, k. (2018). effects of publisher interface and google scholar on html and pdf clicks: investigating paths that inflate usage. the journal of academic librarianship, 44(6), 816-823. https://doi.org/10.1016/j.acalib.2018.09.014 kraemer, a. (2006). ensuring consistent usage statistics, part 2. the serials librarian, 50(1-2), 163-172. https://doi.org/10.1300/j123v50n01_15 lewis, v., & proffitt, m. (2019). trends and priorities in canadian research libraries: results of a carl-oclc survey. report presented at the cni spring 2019 membership meeting. retrieved from www.carl-abrc.ca/wp-content/uploads/2019/03/carl-oclc-innovation-survey_en.pdf long, j. s., & freese, j. (2006). regression models for categorical dependent variables using stata: stata press. mann, s., & sutton, s. (2015). why can’t students get the sources they need? results from a real electronic resources availability study. the serials librarian, 68(1-4), 180-190. https://doi.org/10.1080/0361526x.2015.1017419 mcdonald, j. (2007). understanding journal usage: a statistical analysis of citation and use. journal of the american society for information science and technology, 58(1), 39-50. https://doi.org/10.1002/asi.20420   mcdonald, j., & van de velde, e. f. (2004). the lure of linking. library journal, 129(6), 32-32. retrieved from https://www.libraryjournal.com/?detailstory=the-lure-of-linking mongeon, p., archambault, a., & larivière, v. (2018). the concentration of journal use in canadian universities. paper presented at the proceedings of the annual conference of cais/actes du congrès annuel de l'acsi. retrieved from https://journals.library.ualberta.ca/ojs.cais-acsi.ca/index.php/cais-asci/article/viewfile/984/877 nicholas, d., boukacem‐zeghmouri, c., xu, j., herman, e., clark, d., abrizah, a., . . . świgoń, m. (2019). sci‐hub: the new and ultimate disruptor? view from the front. learned publishing, 32(2), 147-153. https://doi.org/10.1002/leap.1206 samples, j., & healy, c. (2014). making it look easy: maintaining the magic of access. serials review, 40(2), 105-117. https://doi.org/10.1080/00987913.2014.929483 singer, r. (2006). helping you buy: link resolver tools. computers in libraries, 26(2), 15-23. stuart, k., varnum, k., & ahronheim, j. (2015). measuring journal linking success from a discovery service. information technology and libraries (march), 52-76. https://doi.org/10.6017/ital.v34i1.5607 thelwall, m., & maflahi, n. (2015). are scholarly articles disproportionately read in their own country? an analysis of mendeley readers. journal of the association for information science and technology, 66(6), 1124-1135. https://doi.org/10.1002/asi.23252 varnum, k. j., desai, s., folger, k., stuart, k., saylor, k., tuckett, h., . . . ahronheim, j. (2016). report of the link resolver implementation team. university of michigan. retrieved from http://hdl.handle.net/2027.42/135723 wakimoto, j. c., walker, d. s., & dabbour, k. s. (2006). the myths and realities of sfx in academic libraries. the journal of academic librarianship, 32(2), 127-136. https://doi.org/10.1016/j.acalib.2005.12.008 wood-doughty, a., bergstrom, t., & steigerwald, d. g. (2019). do download reports reliably measure journal usage? trusting the fox to count your hens? college & research libraries, 80(5), 694. https://doi.org/10.5860/crl.80.5.694 wulff, j., & nixon, n. (2004). quality markers and use of electronic journals in an academic health sciences library. journal of the medical library association, 92(3), 315-322.     microsoft word news_ebm.doc evidence based library and information practice 2007, 2:4  123 evidence based library and information practice     news    ebm and the medical librarian: registration open        © 2007 glynn. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    registration is now open for the spring 2008  online course ebm and the medical  librarian. the course will run from january  14 – march 9, 2008.    this course is an introduction to evidence‐ based medicine (ebm) for medical librarians  and the role of the librarian to support its  practice. participants will learn to identify  the parts of a well built clinical question, to  recognize the validity criteria for research  studies, and to identify roles that librarians  can undertake in providing ebm training  and support. the course utilizes course  material, independent readings, reviews,  and practice exercises.  it is offered as an  eight week distance education mla ce  course through the university of north  carolinaʹs school of information and  library science. the distance education  version of this course will be held january  14 to march 7, 2008. the course uses the  blackboard course management system. it is  approved for 10 mla ce units.  the cost of the course is $400 (usd) and  payment must be received with the  registration form.  there are only 4 seats left  in the spring class.  those unable to secure a  seat can be added to the waitlist for the next  session scheduled for fall 2008.  if you want  to reserve your seat while processing  payment, please email stephanie peterson at  peterssb@email.unc.edu with your  intentions.    additional course information and the  registration form are available at:  http://sils.unc.edu/programs/continuing_ed/ ebm.html.    if you have any questions about the course,  please contact one of the course instructors:   connie schardt (schar005@mc.duke.edu) or  angela myatt (myattae@ucmail.uc.edu).         http://creativecommons.org/licenses/by/2.0 mailto:peterssb@email.unc.edu http://sils.unc.edu/programs/continuing_ed mailto:schar005@mc.duke.edu mailto:myattae@ucmail.uc.edu evidence based library and information practice evidence based library and information practice 2011, 6.1 41 evidence based library and information practice article the usefulness of related functions in web of science and scopus dorothy c.p. char public works and government services canada gatineau, quebec, canada email: dorothy.char@tpsgc-pwgsc.gc.ca isola ajiferuke faculty of information and media studies, university of western ontario london, ontario, canada email: iajiferu@uwo.ca received: 18 nov. 2010 accepted: 19 feb. 2011 2011 char and ajiferuke. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – this study evaluates the effectiveness of the related search functions in web of science and scopus. web of science has one related search function (searching by references) whereas scopus has three related search functions (searching by references, authors, or keywords). methods – thirty queries were searched in both web of science and scopus. for each query, the most relevant document was retrieved and its first thirty related documents were assessed for relevancy to the original query. results for both databases were compared using the median values of precision. for scopus the three different methods of relevance were compared using median precision values. results – the median precision value for the related documents retrieved from web of science was 0.63, while the median for those retrieved from scopus using the related by references function was 0.62. a wilcoxon test showed no significant difference in the two medians. in the comparison of the three related functions in scopus, the median precision values were 0.62, 0.42, and 0.43 for the related search functions by references, authors, and keywords respectively. a friedman’s test showed that the median precision value for relatedness by references was significantly higher than the median vales for the other two related functions. in scopus, the mailto:dorothy.char@tpsgc-pwgsc.gc.ca� mailto:iajiferu@uwo.ca� evidence based library and information practice 2011, 6.1 42 effectiveness of the related search function using all keywords was not as effective when compared to the effectiveness using select keywords. the median precision value with select keywords was 0.17. conclusions – the related search functions by references for both web of science and scopus were moderately effective in retrieving additional relevant documents on a given topic, and there was no significant difference in their performance. when comparing the three methods available in scopus, the related search function by references was found to be more effective than the system’s related functions by authors and keywords. introduction in online searching, the size of the initial search output may be too few, just adequate, or too much. to address the quantity of output problems, searchers often adopt tactics to narrow or broaden search results. however, even when the quantity of output is acceptable, the examination of the retrieved documents may indicate deficiency in terms of quality, i.e., the retrieved set may contain too many irrelevant documents. to address this, various retrieval refinement methods have been suggested (salton & mcgill, 1983). these include relevance feedback, clustering, and citation searching. over the years, online retrieval systems have provided practical ways of conducting these refinements. in the case of relevance feedback, the systems allow users to view the terms (whether keywords, identifiers, or descriptors) that have been used to index a document, and by clicking on a particular term for a relevant document, other documents that have been indexed by that term can be retrieved. clustering of search output is common with clustering search engines, such as yippy, though traditional retrieval systems such as scopus and web of science now provide categorization of search output by subject areas. in the case of citation searching, many of the retrieval systems allow searchers to view either the list of references or citations referencing a selected document. however, due to the limitations of citing practices, such as honorific or perfunctory citations, citing methodological procedures or sources of data, or negative citations (libmann, 2007), some of these references or citations may not be relevant to the original query. others have suggested that the use of bibliographic coupling or co-citation might produce better results (bichteler & eaton, 1980; badran, 1984). bibliographic coupling is the act of relating two documents together based on having a certain number of common references, while co-citation is based on having a certain number of common citations (wissmann, 1993). while no retrieval system has provided a means of retrieving documents which have a certain number of common citations within a selected document, web of science and scopus allow users to view documents related to a selected document by common references. in the case of scopus, documents can also be related by authors or by common keywords (similar to functions provided by google or pubmed). yet, the question remains, how relevant to the original query are additional documents produced through bibliographic coupling? the objective of this study is to answer this question by determining (and comparing) the effectiveness of the related functions of web of science and scopus. given that scopus provides three different functions for relating documents (common references, common authors, and keywords), these methods will be compared against each other. from a practical standpoint, this information will provide searchers with knowledge of the most useful means of obtaining additional relevant documents on a given topic. evidence based library and information practice 2011, 6.1 43 literature review studies comparing web of science and scopus can be categorized into two groups, namely those comparing their searching features and those comparing their capabilities for conducting informetric studies. in the first category, laguardia (2005) asserted that scopus is more suited for subject searching than web of science. in terms of subject coverage, scopus covered the sciences better than web of science, while web of science covered arts and humanities and social sciences better than scopus. in his review of the two systems jacso (2005) showed that 77% of the web of science records were in the science citation database, 14% in the social sciences citation database, and 9% in the arts and humanities citation database. in scopus, 60% of the records were in the health and life science subject areas, 25% in the chemistry, physics, math and engineering subject areas, 13 % in the biological, agricultural, earth and environmental science subject areas, and 2% in social science, psychology and economics subject areas. so while scopus covers science better than web of science, web of science’s coverage of science is better than its coverage of arts and humanities. in their study of the coverage overlap between the two systems, gavel and iselid (2008) found that web of science included 8,901 journal titles in its databases, while scopus had 13,690 journal titles. there were 7,434 titles indexed by both databases. in their citation analysis of medical documents, a number of studies have found scopus retrieved more citations than web of science. for example, falagas, pitsourni, malietzis, and pappas (2008) found that scopus listed 20% more citations than web of science for some biomedical articles. torressalinas, lopez-cozar, and jimenez-contreras (2009) found that articles in the area of the health sciences authored by researchers at the university of navarra (spain) received 14.7% more citations in scopus than in the web of science. kulkarni, aziz, shams, and busse (2009) found that for articles published in general medical journals, scopus retrieved more citations per article (median=149) than did web of science (median =122). in their comparison of the suitability of scopus and web of science for bibliometric analyses, gorraiz and schloegl (2008) found that of the top 100 pharmacy journals listed by isi’s journal citation reports -science, the impact factor was higher for 82 titles, and the immediacy index was greater for 78 journals in scopus in 2005. these results are not totally surprising given scopus’ wider coverage of the health and life science subject areas. similar results have been obtained in other subject areas. in a study of the impact of 22 top human-computer interaction (hci) researchers from equator, a large british interdisciplinary research collaboration project, meho and rogers (2008) found that the study sample had been cited in 6,919 papers indexed in scopus and 4,011 papers indexed in web of science. in a similar study of the scholarly impact of 80 randomly selected full-time, information studies faculty members from north america, meho and sugimoto (2009) found that the study sample had been cited in 7,103 papers in scopus and only 5,269 papers in web of science. in a citation analysis of the 25 most accessed articles in 163 social sciences journals, levineclark and gil (2009) found that the average number of citations received by these articles in web of science was 7.95, while the average citation count for the same articles in scopus was 9.27. the higher number of citations found in scopus may not necessarily be due to better coverage of a particular subject area, but it may be because scopus covers more source titles, as discussed by gavel and iselid (2008). in summary, this review shows that web of science and scopus have been examined and compared regarding their subject coverage, reflecting their capabilities as tools for informetric studies. however, no studies have compared them in terms of the effectiveness of their related search functions. this study will evidence based library and information practice 2011, 6.1 44 fill that gap in the research literature with the intention of providing useful guidance for users of these databases. methods a total of 30 queries (appendix a) were searched in both web of science and scopus. the subjects reflected the broad scope of the databases’ content. the queries were selected in such a way that 10 of them could be searched in each of the three indexes of web of science (i.e., arts and humanities, social sciences, and science), and five in each of the six subject areas of scopus (i.e., life sciences, physical sciences, health sciences, social sciences, and arts and humanities). the queries were collected from assignments given in an information retrieval course as well as from the text retrieval conference (trec) web site (http://trec.nist.gov/). they were reformatted into queries suitable for the selected databases and comprehensible to the researchers. the first author of this paper identified the key concepts of each query and formulated the appropriate search strings for each database. unless otherwise stated in the query, the search output was not limited by any restrictions (e.g., language, year of publication, or document type). the results of each query searched in web of science were sorted according to relevance. for each query the highest ranked document had the top 30 related articles of each query’s top-related references assessed for relevancy to the original query. the relevance ranking of web of science (as well as that of scopus) is based on the number of search terms found in each document record. it is not unusual for a user to find that the topmost ranked document is not the most relevant, or on occasion, not even relevant to his or her query. in this study and for each query, the goal of this research was to find whether the topmost ranked document was relevant to the query. the number of related documents was limited to 30, as previous studies have shown that most users usually view no more than the top 30 documents retrieved in response to a web query (spink & wolfram, 2001). researchers recorded both the total number of references, as well as the number of shared references between the related and original document, were recorded. the percentage of common references for each document in each of the 30 queries was computed according to the formula: (the number of common references) divided by [(the number of references in the original document plus the number of references in the related document) minus the number of common references]. the first author of this paper assessed the relevancy of each reference using as either “relevant” or “not relevant.” for the queries collected from assignments in an information retrieval course, the relevance judgment was based on familiarity with the queries (from previous searching experience). guidelines for judging relevance were used for those taken from trec (text retrieval conference, 2000). for example, for query #20 (appendix a), the instruction for judging relevance of a retrieved document stated “a relevant document will contain an argument for or against requiring students to wear uniforms in schools. advertisements for uniforms and simple statements that particular school districts require uniforms are not relevant” (text retrieval conference, 2001, number 533) the two most commonly used measures of effectiveness of an information retrieval system are recall and precision. recall is defined as the proportion of relevant items retrieved (i.e., the number of relevant items retrieved divided by the number of relevant items in the database). precision is defined as the proportion of retrieved items that are relevant (i.e., number of relevant items retrieved divided by the number of retrieved items). however, due to the inherent difficulty in determining the number of relevant items in a large database, precision was the only practicable measure for this study. hence, the precision value for each of the 30 queries was evidence based library and information practice 2011, 6.1 45 computed as the number of relevant related documents divided by the total number of related documents examined. this calculation attempted to determine the precision of the databases in identifying related documents. table 1 precision values of the related documents by common references retrieved from web of science and scopus query # precision for web of science precision for scopus 1 .7667 not available 2 .7333 not available 3 .8667 .8000 4 .7333 .5000 5 .4667 .3000 6 .9667 1.000 7 .6667 not available 8 .7667 .6000 9 not available not available 10 .7000 .7667 11 .9333 not available 12 not available .1667 13 .8000 .7000 14 .6000 .6667 15 .3000 not available 16 .7333 .9000 17 .8333 .7333 18 .3667 .4333 19 .2000 .5000 20 .4333 not available 21 .1000 .6333 22 .6333 .9000 23 not available .5667 24 .2667 .5000 25 .3667 not available 26 .8000 .3333 27 .6000 not available 28 .5000 .6667 29 .3667 .1667 30 .0667 not available the results of each query searched in scopus were also sorted by relevance, and the first 30 related articles of the topmost relevant document (either all references, all authors, or all keywords) were assessed for relevancy to the original query. the total number of references, authors, and keywords, as well as the number of such shared elements, were recorded, in the same manner as the web of science data was evaluated, the percentage of common references, common authors, and keywords were also recorded. the relevancy assessment of the related documents was performed using the same method, and the precision values were obtained using the formula referenced above. the option to search for related documents to the topmost ranked document by using some, instead of all, of its keywords was explored; for each topmost ranked document, three to five of its keywords were used to search for related documents. descriptive statistical methods described the effectiveness of each related function, and inferential statistical tests compared the effectiveness of the functions. results comparison of relatedness by common references in web of science and scopus the study determined the effectiveness of the related search function for 27 of the queries in web of science and 20 queries in scopus. for the 27 queries in web of science that produced related documents, the mean and median precision values for the related documents retrieved were 0.58 and 0.63 respectively. for scopus, the mean and median precision values were 0.59 and 0.62 respectively (table 1). the distributions of precision values shown in table 1 are skewed, which implies that the median is the more appropriate measure of central tendency to be used in comparing the two sets of values. hence, the wilcoxon test, a nonparametric statistical test used to compare the median values of two groups, was used to determine if the median precision values were the same. the test yielded a z-score, of -0.218 with the corresponding p-value of 0.828. the p-value indicated no statistical significant difference between the effectiveness of the related search function by references for web of science when compared to that of scopus. evidence based library and information practice 2011, 6.1 46 table 2 details of the related documents retrieved for the topmost document (with 14 references) for query 1 in web of science related document # number of references number of common references with the topmost document percentage of references in common with the topmost document relevancy of the related document 1 132 8 5.80 yes 2 23 7 23.33 yes 3 110 7 5.98 yes 4 21 4 12.90 yes 5 32 3 6.98 yes 6 997 3 0.30 yes 7 221 3 1.29 yes 8 109 3 2.50 yes 9 16 3 11.11 yes 10 43 3 5.56 yes 11 35 3 6.52 yes 12 70 2 2.44 no 13 53 2 3.08 no 14 48 2 3.33 no 15 29 2 4.88 yes 16 36 2 4.17 no 17 23 2 5.71 no 18 41 2 3.77 yes 19 81 2 2.15 yes 20 36 2 4.17 yes 21 44 2 3.57 yes 22 35 2 4.26 no 23 21 2 6.06 yes 24 33 2 4.44 no 25 51 2 3.17 yes 26 6 2 11.11 yes 27 13 2 8.00 yes 28 15 2 7.41 yes 29 135 2 1.36 yes 30 20 2 6.25 yes the study attempted to determine the minimum number of references (or percentage of references) that a related document needed to have (relative to the topmost document) for it to be judged relevant. however, as evidenced by the sixth related document in table 2, for one of the queries searched in web of science, there was no discernible pattern. hence, it was not possible to determine such a number (or percentage) for all the queries in either database. comparison of relatedness by common references, keywords and authors in scopus given that scopus can obtain related documents through three different search functions, the study compared the effectiveness of these three related functions against each other. the precision values for relatedness by all references were the same as described above. the mean and median values for relatedness by all authors were .41 and .42 respectively, while for relatedness by evidence based library and information practice 2011, 6.1 47 keywords, the mean and median values were .44 and .43 respectively (table 3). in order to statistically compare the three related search functions, the friedman test, a nonparametric statistical test used to compare the medians of more than two groups, was used to determine if the median precision values for all three functions were the same. the test statistic for this test is the chi-square, and we obtained a value of 6.125 with 2 degrees of freedom and a p-value of 0.047. hence, we rejected the hypothesis that the three medians were the same. a follow-up multiple comparison test showed that the related search function by all references was more effective than the other two related functions. comparison of relatedness by all keywords and select keywords in scopus each of the three related search functions in scopus has two options of using all (e.g., all references) or a select few. it might be difficult for a user to determine which references or authors to select, but relatively easy to choose a few keywords related to his or her original query. as a result, the study compared the effectiveness of using all keywords and select keywords. the performance values for “all keywords” were as described above (i.e., mean and median precision values of .41 and .42 respectively), while the mean and median precision values for relatedness by “select keywords” were .25 and .17 respectively (table 4). table 3 precision values of related documents by references, authors and all keywords retrieved from scopus query # precision for all references precision for all authors precision for all keywords 1 not available not available .0667 2 not available .2333 .3667 3 .8000 .3333 .7667 4 .5000 .4000 .1667 5 .3000 not available .1667 6 1.000 .4333 .8667 7 not available .3000 .2000 8 .6000 .5667 .3333 9 not available not available not available 10 .7667 .4333 .9000 11 not available .5333 .1667 12 .1667 .0909 .4783 13 .7000 .1667 .6000 14 .6667 .5667 .4333 15 not available .2000 .3000 16 .9000 .3000 .7333 17 .7333 .5333 .5000 18 .4333 .0800 .6000 19 .5000 not available not available 20 not available not available not available 21 .6333 .6667 .3333 22 .9000 .5667 .4000 23 .5667 1.000 .6000 24 .5000 not available .4333 25 not available not available not available 26 .3333 not available .6333 27 not available not available not available 28 .6667 not available .1000 29 .1667 not available not available 30 not available not available not available evidence based library and information practice 2011, 6.1 48 the distributions in table 4 were skewed. in order to accurately assess whether the median precision values for the two keywords options were the same, the wilcoxon test for paired samples provided the needed statistical analysis. the z-score of the test was -3.761 with a p-value of 0.000, which implied a significant statistical difference between the effectiveness of the two related search functions. hence, the research concluded that the effectiveness of the “all keywords” related function was better than that of “select keywords” in scopus. discussion the results demonstrate the usefulness of the related search functions in both web of science and scopus. researchers often need to examine a moderate amount of relevant materials on a topic, especially if they are compiling a bibliography or writing a literature review. this may be equally true for professional librarians, who may be engaged in their own research projects, compiling course bibliographies, or conducting searches on behalf of their patrons. table 4 precision values of related documents by all keywords and select keywords retrieved from scopus. query # precision for “all keywords” precision for “select keywords” 1 .0667 .0000 2 .3667 .5000 3 .7667 .4000 4 .1667 .1000 5 .1667 .0000 6 .8667 .7667 7 .2000 .1333 8 .3333 .1000 9 not available not available 10 .9000 .6667 11 .1667 .0000 12 .4783 .0000 13 .6000 .3667 14 .4333 .2333 15 .3000 .4000 16 .7333 .5000 17 .5000 .0000 18 .6000 .1667 19 not available not available 20 not available not available 21 .3333 .1333 22 .4000 .3667 23 .6000 .2333 24 .4333 .1667 25 not available not available 26 .6333 .4333 27 not available not available 28 .1000 .0667 29 not available not available 30 not available not available evidence based library and information practice 2011, 6.1 49 the following example demonstrates how the functions work in practice: search for relevant materials on the applications of tagging in online public catalogs in web of science, using “tag* and (opac* or online public catalog*)” in the topic field. this search yielded twelve documents, of which three were relevant. the most relevant of those was, “enhancing library services with web 2.0 functionalities” by gavrilis, kakali, and papatheodorou (2008) a review of the related records for this article retrieved 371 documents with at least one reference in common. further analysis identified four relevant documents with three references in common with the target article. of the 47 documents with two references in common with the target article, 17 were identified as relevant to the original search question. a review of the records of related references to the article by gavrilis, et al. (2008) retrieved 21 additional relevant documents. the search of the same terms in scopus yielded five documents, of which only three were included in the list of those found in web of science. of the five documents from scopus, three were found to be relevant, with the most relevant again being the article by gavrilis et al. (2008). the similarity in the effectiveness of the two related search functions by references may be due to the considerable overlap in the subject coverage of the two databases or the similarity in the operationalization of the methods used in selecting the related documents. even though scopus does not explicitly state the criterion used in its ranking of related documents, they are likely to be similar to web of science’s ranking methods, that is to place, in decreasing order, the number of common references the related documents have with the document already identified. in scopus, there are three types of related functions, while only one is available in web of science. for scopus there are related search functions by references, authors, and keywords. under each related function type, the user can select the option of using all attributes from the article (e.g., all references) or the user can choose to use only a selected number of attributes (e.g., a selected number of references). this study investigated the effectiveness of using all attributes for all three types, but used only a selected number of attributes for keywords. the reason for this was that while there is often not a clear criterion to use in selecting from the list of reference or authors, it would be possible to select keywords related to the original topic of inquiry. the study determined that the effectiveness of using select keywords was much worse than when using all keywords. the recommendation would be to discourage users from making use of the select keywords for the related search function. of the three related search functions, the most effective proved to be the search by all references. this may be due to the fact that while two documents may have the same author, the subject areas or topics that they cover may be different. also, two documents may be indexed by the same keyword but with varying depth of treatment of the keyword in the two documents. for scopus the result of this research indicates that the user should try the related search function by all references first, and should only explore the other two types afterwards if there is still a need for more relevant documents. documents obtained using the related search function based on the gavrilis et al., (2008) article and using all three search functions (i.e., all references, all authors, and all keywords) demonstrated this. results yielded 710 documents that shared at least one common reference, 42 documents that shared at least one author, and 4,452,046 documents that shared at least one keyword with the target article (gavrilis et al., 2008). the 710 documents retrieved by all references were sorted by relevance (and not by the number of common references as with web of science). only 20 of the first 60 documents were determined to be relevant. of the 42 documents retrieved by all authors, only one was found relevant. finally, when the 4,452,046 documents retrieved by all keywords were ranked by relevance, only 10 of the first 60 documents were determined to be relevant. evidence based library and information practice 2011, 6.1 50 there were several limitations of this study. first, the queries were not obtained from users with real information needs, and users were not involved in the assessment of the relevance of the related documents retrieved. second, some top-ranked documents did not produce any related documents, because they either had either too few references, or too few authors, or too few keywords. third, a few of the top-ranked documents did not have detailed records in scopus, so users were redirected to another database, csa illumina. for such documents, it was not possible to obtain their related documents from scopus. due to the last two limitations, it was not possible to determine the minimum number of references that a related document needed to share with an already identified relevant document for it also to be judged relevant. conclusions the main objective of this study was to determine the usefulness of the related search functions in web of science and scopus with regard to finding additional relevant documents on a given topic. the related search functions by references for both web of science and scopus proved to be moderately effective in retrieving additional relevant documents, and there was no significant difference in their effectiveness. users who want to obtain additional documents relevant to a particular query, should do so by retrieving documents related by common references to an already identified relevant document in either web of science or scopus. when comparing the related search functions within scopus, the related by references search function was found to be more effective than either the related function by author or the related function by keyword. if using scopus alone, the relevant by references function was more useful than relevant by either author or keyword. future research will use a higher number of queries to further explore the possibility of determining the minimum number of references that a related document needs to share with an already identified relevant document to a query for it also to be judged relevant. references badrn, o. m. (1984). alternative search strategy to improve information retrieval. proceedings of the asis annual meeting, 21, 137-140. bichteler, j., & eaton iii, e. a. (july 1980). the combined use of bibliographic coupling and cocitation for document retrieval. journal of the american society for information science, 31(4), 278-282. falagas, m. e., pitsouni e. i., malietzis, g. a., & pappas, g. (feb. 2008). comparison of pubmed, scopus, web of science, and google scholar: strengths and weaknesses. faseb journal, 22(2), 338342. doi: 10.1096/fj.07-9492lsf gavel, y., & iselid, l. (feb. 2008). web of science and scopus: a journal title overlap study. online information review, 32(1), 8-21. doi: 10.1108/14684520810865958 gavrilis, d., kakali, c., & papatheodorou, c. (2008). enhancing library services with web 2.0 functionalities. lecture notes in computer science, 5173/2008: 148-159. research and advanced technology for digital libraries. ecdl '08 proceedings of the 12th european conference on research and advanced technology for digital libraries, aarhus, denmark. sept. 14-19 2008. doi: 10.1007/978-3-54087599-4_16 gorraiz, j., & schloegl, c. (oct. 2008). a bibliometric analysis of pharmacology and pharmacy journals: scopus versus web of science. journal of information science, 34(5), 715-725. doi: 10.1177/016555150708669991 jacso, p. (nov./dec. 2004). isi web of science, scopus, and sportdiscus. online, 28(6), 51-54. jacso, p. (10 nov. 2005). as we may search – comparison of major features of the evidence based library and information practice 2011, 6.1 51 web of science, scopus, and google scholar citation-based and citationenhanced databases. current science, 89(9), 1537-1547. kulkarni, a. v., aziz, b., shams, i., & busse, j. w. (9 sept. 2009). jama, 302(10), 10921096. doi: 10.1001/jama.2009.1307 kumar, m. j. (2009). evaluating scientists: citations, impact factor, h-index, one page hits and what else? iete technical review, 26(3), 165-168. laguardia, c. (jan. 2005). e-views and reviews: scopus vs. web of science. library journal, 130(1), 40-42. levine-clark, m., & gil, e. (2009). a comparative analysis of social science citation tools. online information review, 33(5), 986-996. doi: 10.1108/14684520911001954 libmann, f. (may/june 2007). web of science, scopus, and classical online: philosophies of searching. online, 31(3), 36-40. lópez-illescas, c., de moya anegón, f., & moed, h. f. (april 2009). comparing bibliometric country-by-country rankings derived from the web of science and scopus: the effect of poorly cited journals in oncology. journal of information science, 35(2), 244-256. doi: 10.1177.165551508098603 meho, l. i., & rogers, y. (sept. 2008). citation counting, citation ranking, and h-index of human-computer interaction researchers: a comparison of scopus and web of science. journal of the american society for information science and technology, 59(11), 1711-1726. meho, l. i., & sugimoto, c. r. (dec. 2009). assessing the scholarly impact of information studies: a tale of two citation databases – scopus and web of science. journal of the american society for information science and technology, 60(12), 2499-2508. doi: 10.1002/asi.21165 meho, l. i. & yang, k. (nov. 2007). impact of data sources on citation counts and rankings of lis faculty: web of science versus scopus and google scholar. journal of the american society for information science and technology, 58(13), 2105-2125. doi: 10.1002/1si.20677 morris, s. a. (2005). manifestation of emerging specialties in journal literature: a growth model of papers, references, exemplars, bibliographic coupling, cocitation, and clustering coefficient distribution. journal of the american society for information science and technology, 56(12), 1250-1273. salton, g., & mcgill, m. j. (1983). introduction to modern information retrieval. new york: mcgraw-hill, inc. spink, a., & wolfram, d. (feb. 2001). searching the web: the public and their queries. journal of the american society for information science and technology, 52(3), 226-234. text retrieval conference. (2000). text retrieval conference (trec). retrieved from http://trec.nist.gov/ text retrieval conference. (2001). trec 2001 web ad hoc topics. in data – english test questions (topics) files list. retrieved from http://trec.nist.gov/data/topics_eng/topi cs.501-550.txt torres-salinas, d., lopez-cozar, e. d., & jimenez-contreras, e. (2009). ranking of departments and researchers within a university using two different databases: web of science versus scopus. scientometrics, 80(3), 761-774. doi: 10.1007/s11192.008.2113.9 http://trec.nist.gov/� http://trec.nist.gov/data/topics_eng/topics.501-550.txt� http://trec.nist.gov/data/topics_eng/topics.501-550.txt� evidence based library and information practice 2011, 6.1 52 wissmann, c. (1993). techniques of data retrieval for scientometric research in the isi citation indexes. journal of information science, 19(5), 363-376. doi:10.1177/016555159301900504 zhao d., & strotmann a. (2008). evolution of research activities and intellectual influences in information science 1996-2005: introducing author bibliographic-coupling analysis. journal of the american society for information science and technology, 59(13), 2070-2086. evidence based library and information practice 2011, 6.1 53 appendix a: list of queries query no. query 1 what is the eating behaviour of ladybugs in north america? please find documents in english. 2 i am looking for english documents that discuss the way in which radiation treats burkitt lymphoma. 3 documents since 2005 that discuss photosynthesis for evergreen trees. 4 i am looking for english documents that discuss the evolutionary history of sharks. 5 i am looking for english documents that discuss the differences between darwin's theory of evolution with theories of creationism. 6 what is the impact of farm raising salmon on ecosystem health? 7 what is the drainage basin of the danube, and thorugh which countries does it flow? 8 i am interested in the vesuvius: what were the causes of its past erruptions, and how was it formed? 9 i am interested in documents that discuss the predictability of astrology on human relationships. 10 what do recent documents, since 2000, state regarding the composition of the earth's crust? 11 what are sub-atomic particles, and how do they impact energy? 12 find documents in english that discuss rutherford's contribution to physics and his work with niels bohr. 13 what are the pros and cons of stretching versus not stretching before exercise. i am looking for documents since 2000 in english. 14 what is the impact of smoking on the prevalence of heart attacks in canadian women? 15 i am interested in documents that discuss how to prevent osteoporosis. 16 i would like to know the effects of decaffeinated and regular coffee on diabetes. 17 what is the relationship between pain, psychological distress, and demographic variables at the end of life? 18 i am looking for documents that discuss the overall dental health of female adolescents, since 2005. 19 i am looking for documents that discuss the advantages or disadvantages of placing both male and female youths in the same residential units in correctional centres. 20 i am interested in documents that discuss the pros and cons of students wearing a school uniform or adhering to a dress code. the documents should be in english. evidence based library and information practice 2011, 6.1 54 21 find english documents that discuss the barriers to women’s participation in sporting activities in muslim countries. 22 i am interested in english documents that compare the reasons why men have eating disorders with the reasons why women have such disorders. 23 what factors impede the integration of immigrants in canada? documents from last 5 years. 24 find english documents that discuss government social programs for immigrants in only canada. the documents must have been published since 2000. 25 i am looking for information on lacan and his writings on literary theory, specifically, those that mention freud. 26 i am interested in the origins of hinduism and what current research states about the religion. i would like documents in english. 27 find documents that discuss the history of the english language. 28 i am looking for documents in gender theory that discuss performativity in english literature. 29 i am interested in english documents that discuss wordsworth's use of pastoral imagery in his romantic poems. 30 find documents in english that discuss women in both muslim and christian faith. / evidence based library and information practice/ / research article   a content analysis of systematic review online library guides   jennifer lee associate librarian libraries and cultural resources university of calgary calgary, alberta, canada email: jennifer.lee@ucalgary.ca   k. alix hayden librarian libraries and cultural resources university of calgary calgary, alberta, canada email: ahayden@ucalgary.ca   heather ganshorn associate librarian libraries and cultural resources university of calgary calgary, alberta, canada email: heather.ganshorn@ucalgary.ca   helen pethrick graduate student werklund school of education university of calgary calgary, alberta, canada email: helen.pethrick@ucalgary.ca   received: 10 aug. 2020                                                             accepted: 15 jan. 2021      2021 lee, hayden, ganshorn, and pethrick. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29819     abstract   objective – online library guides can serve as resources for students and researchers conducting systematic literature reviews. there is a need to develop learner-centered library guides to build capacity for systematic review skills. the objective of this study was to explore the content of existing systematic review library guides at research universities.   methods – we conducted a content analysis of systematic review library guides from english-speaking universities. we identified 18 institutions for inclusion using a scopus search to find the institutions with the highest number of systematic review publications. we conducted a content analysis of those institutions’ library guides, coding for the types of resources included, and the stage of the systematic review process to which they referred. a chi-square test was used to determine whether the differences in distribution of the resource types within each systematic review stage were statistically significant.   results – the most common type of resource was informational in content. only 24% of the content analysed was educational. the most common stage of the systematic review process was conducting searches. the chi-square test revealed significant differences for seven of the nine systematic review stages.   conclusion – we found that many library guides were heavily informational and lacking in instructional and skills focused content. there is a significant opportunity for librarians to turn their systematic review guides into practical learning tools through the development and assessment of online instructional tools to support student and researcher learning.     introduction   in systematic, scoping, and other related knowledge synthesis reviews, researchers use transparent procedures to find, critically appraise, analyze, and synthesize the results of relevant research. systematic reviews became established in the health sciences literature in the late 1990s, with an exponential increase since 2011 (page et al., 2018). systematic reviews are also becoming common in disciplines outside of the health sciences, such as business, ecology, education, the social sciences, and humanities. in 2018, visintini et al. conducted a scoping review investigating research support in health sciences libraries. they determined that “support for systematic reviews was another highly represented service” (p. 63) and that providing this support was described in numerous articles (25 out of 75). instruction, “training, developing search strategies, running searches, managing search results, obtaining full-text reports, and providing methods write-ups” (visintini et al., 2018, p. 63) were some of the specific supports provided.   over the last several years, academic librarians at our institution have seen an increase in requests to teach graduate students and research assistants how to conduct a systematic review. these students often required considerable support, and requested multiple individual consultations with a librarian. we determined that we needed to develop a more efficient way of providing this support to respond to the increasing demands and build students' knowledge and skills. in consulting the literature to inform our own practice, we hoped to find guidance on how to incorporate asynchronous instructional content into our online library guides, rather than simply offering them as repositories of information.   mckeown and ross-white (2019) described the development of a service designed “to build capacity for increased librarian support and to maximize librarians’ time and expertise in providing this support” (p. 2). we similarly recognized that libraries need to build capacity for systematic reviews, but it appears that the capacity-building is most often focused on the librarian. we wanted to build capacity, expertise, and knowledge amongst our students and researchers, to foster more independent learning and make consultations with librarians more effective, and perhaps less frequent. to that end, we received a teaching and learning grant from the university of calgary to develop asynchronous online instructional tools relevant to systematic review methods.   we began by surveying the field to determine what instructional resources to which we could link and which we would have to develop ourselves. this content analysis of library resources supporting systematic reviews at other academic libraries is part of the process. we wanted to focus on those common skills or tasks (e.g., translating a search, saving .ris files, de-duplication in endnote) that are best demonstrated with video or step-by-step instructions and alleviate the need to teach these skills during one-on-one consultations.   literature review   systematic review support and instruction   as demand for systematic review support increases, supporting this area becomes a challenge. in one study, researchers estimated that the median amount of time spent by librarians on all systematic review-related tasks (interview, search strategy, search translation, documentation/writing, and team instruction) is 18.5 hours per review, but can vary greatly (bullers et al., 2018). the same researchers also found librarians with more experience in systematic reviews are more likely to spend time providing instruction on this topic. spencer and eldredge (2018) identified 18 roles or functions that a librarian may fulfill in systematic review support. these included activities such as search filters and hedges, searching, source selection, question formulation, planning, reporting and documentation, deduplication, and technological and analytical roles. they did not include roles such as screening, nor data management beyond traditional citation management; however, we have been asked to provide instruction and support in these areas, and communication with librarians at other institutions suggested that this is a wider trend. this was echoed in a commentary by roth (2018), who established a systematic review service that was initially very searching-focused, but who “quickly learned that researchers were seeking more training about other aspects of the systematic review process” (p. 514). accordingly, roth developed a learning outcomes model that incorporated training librarians in other parts of the review process, so that librarians can teach these skills.   as knowledge synthesis matures as a field, the involvement of librarians in this area grows more complex, and demand for librarian assistance increases. haddaway et al. (2015) noted that systematic review methods can be used in traditional literature reviews to help mitigate bias, increase transparency, provide consistency and objectivity, and critically appraise the evidence (i.e., the literature). students who are inexperienced in these methods are often directed to librarians for advice and assistance.   developing online training is a potential solution to some of the increasing demands for librarian instruction and support. parker et al. (2018) conducted “an environmental scan and assessment of online systematic review training resources in order to describe available resources and to evaluate whether they follow current best practices for online instruction” (p. 2). these researchers assessed the quality of 20 training resources and determined an average grade of only 61% based on content, design, interactivity, and usability. this scan also found that the highest-scoring resources were courses that required a time commitment of more than five hours. of note, parker et al. only assessed those online training resources that included at “least three of six systematic review steps” (p. 2). the researchers did not investigate online training or tutorials that focused on singular tasks required for systematic reviews, such as how to deduplicate in endnote. rather, their study focused on more comprehensive and holistic online training for systematic reviews. there appears to be a gap in the literature investigating online training and instructional resources focused on individual skills or tasks required to conduct systematic reviews. given the recent suspension of in-person assistance and instruction in many academic libraries due to the covid-19 pandemic, online library resources are now even more critical to student success.   online library guides   online library guides, such as libguides hosted on the springshare platform, have become ubiquitous in academic libraries. library guides are often used as subject pathfinders, course-integrated and class assignment resources, and instructional supports. baker (2014) noted that libguides have a tendency toward providing “too much information: what might be termed the ‘kitchen sink’ approach” (p. 110). essentially, librarians include links and annotations to all possible resources and services. as baker noted, libguides can result in cognitive overload for students.   bergstrom-lynch (2019) noted that the majority of research on libguides has focused “almost exclusively on issues of usability, resulting in best practices that are user-centered but not necessarily learner-centered (i.e. designed to support the special needs of learners)” (p. 205). german (2017), an instructional design librarian, also believes that we need to shift to a learner-centered perspective. german suggested that once we change our focus, we will view “libguides as an e-learning tool” (p. 163). the focus should be not on what resources to include in a guide, but how the guide can help students be successful learners (german & graves, 2016). bremner (2019) defined learner-centred education as a “teaching approach in which learners cease to be passive receivers of knowledge and become more active participants in their own learning process; learning is contextualised, meaningful, and based, wherever possible, around learners’ prior knowledge, needs and interests” (p. 54). online library guides that are learner-centred provide opportunities for active learning, are meaningful, encourage learner/student engagement, and, most importantly, meet the needs of the learner.   stone et al. (2018) contended that library guides should be designed pedagogically, “where the guide walks the student through the research process” (p. 280). they noted that most research on library guides can be categorized in one of three ways: best practices and design, student use of guides, and guides used for instruction. the researchers conducted a pilot study where two libguides were developed for two first-year courses, and student learning was assessed. one guide was a traditional pathfinder which focused on resources, and the other was pedagogical, organized as a research process. content was the same for both guides. stone et al. discovered that “a pedagogical guide design, organizing resources around the information literacy research process and explaining the ‘why’ and ‘how’ of the process, leads to better student learning than the pathfinder design” (p. 290). lee and lowe (2019) recently conducted a study also comparing guide design (pathfinder or pedagogical) to determine which design would best support “the student information learning experience outside of a classroom setting” (p. 205). the pedagogical guide utilized a “visually attractive infographic” (p. 211) of the research process (question, background, find materials, evaluate, refine topic, organize, and so on) rather than a format approach (books, reference materials, articles, current awareness, for example). further, the guide included sequential numbering of each component of the research process, which, the researchers suggested, reduced students’ uncertainty and encouraged them to review the entire guide. although no statistical difference was found between guide types, the researchers concluded that student engagement with a pedagogically designed guide is enhanced. specifically, students reported “a more positive experience” (lee & lowe, 2019, p. 221) when using the pedagogical guide, spending more time interacting with the content and consulting more resources.   aims   there is a need for evidence about the types of content included in systematic review online library guides to help librarians move toward learner-centered guides. we were interested in locating more skill or task-focused, point-of-need resources that could be delivered as short, 5-to-10-minute videos, or other interactive modules, particularly for mechanical tasks such as deduplication of results. however, we were unable to locate any existing literature that described or assessed this type of instructional resource for systematic reviews; therefore, we decided to conduct a content analysis of systematic review online library guides at research universities.   our research sought to examine the content available in existing systematic review library guides and to determine the degree to which the guide content was learner-centered and provided instruction on specific systematic review skills.   methods   we conducted a content analysis of systematic review online library guides. content analysis “is a highly flexible research method that has been widely used in library and information (lis) studies with varying research goals and objectives” (white & marsh, 2006, p. 22). kim and kuljis (2010) established that content analysis methods were appropriate for examining web-based content. these researchers suggested that content analysis is a fairly straight-forward research process to perform, can be done at the convenience of the researchers, and ethics approval is not required as web-based content is usually publicly posted.   our methods were informed by yoon and schultz’s (2017) content analysis that investigated libraries’ research data management websites. their study examined library research data management websites focusing on four main areas: service, information, education, and network. these areas were developed a priori. their content analysis “categorized content displayed on the webpages into different types based on the purpose of the content” (yoon & schultz, 2017, p. 923). we approached our content analysis in a similar fashion. prior to collecting data, we identified types of resources, based on yoon and schultz’s categories. we adapted their definitions for service, information, and education content and added a new category of tool (see table 2).   sample   assuming that universities publishing the most systematic reviews would have the most demand for support from their libraries and librarians, and that the support would be in the form of an online library guide, in january 2018, we searched the scopus database for the most prolific universities in two phases. as this is not a systematic review with a comprehensive search across multiple databases, no other databases were searched. institutions were identified using the keywords “systematic review” or “scoping review” in the title field. since these are the most common knowledge synthesis review types, the results would reflect the most prolific universities. every author affiliation is counted in scopus and is summed up as a list of institutional affiliations. from this list, the top 15 english-speaking universities were identified according to the number of results (figure 1). in order to focus on university sites, search results were excluded if no affiliation was mentioned; the publication was affiliated with a non-university institution, or the affiliated institution was a non-english speaking institution.   figure 1 number of systematic and scoping reviews published by university: initial scopus search. note: ucl is university of college london; scopus outputs the abbreviated name.   surprisingly, this search resulted in only one institution from the united states (u.s.). to reflect the prominence of the u.s. in research, we conducted a second search of scopus, where the search results were limited to u.s. universities. the top five u.s. universities were included from this search, excluding harvard medical school which was identified in the initial search (figure 2).   figure 2 number of systematic and scoping reviews published by university: scopus search limited to u.s. universities.   after we identified the top 20 universities in terms of number of systematic reviews published, we then searched for their published online, library-produced systematic review guides. two universities—university of birmingham and johns hopkins university—were excluded because they did not have these library-produced guides. the total number of university systematic review online library guides included in our content analysis was eighteen (n = 18), which was feasible for us. of the universities included in our analysis, four were from australia, five were from canada, four were from the united kingdom, and five were from the u.s. (table 1). table 1 included universities (n = 18) australia canada united kingdom united states university of sydney university of toronto ucl (university college london) harvard university university of melbourne mcmaster university king’s college london university of california san francisco monash university university of alberta university of oxford university of washington, seattle university of queensland university of ottawa imperial college london university of north carolina at chapel hill   university of british columbia university of pennsylvania   data collection and analysis   the contents of systematic review online library guides were coded in the winter of 2018. the research team developed a deductive, directed coding procedure for content analysis of the included guides (hsieh & shannon, 2005). first, we established a set of code definitions (table 2). the research team discussed, and came to consensus on, any emergent codes in an iterative process as data collection and analysis occurred. one of the researchers (hp) coded initial samples. two members of the research team (kah, jl) then met to review the coding, clarifying as necessary. hp coded all subsequent samples. based on yoon and shultz’s (2017) definitions, we developed the following codes for categorizing the type of resource included in each systematic review guide: information, education (internal), education (external), service, tool (educational), and tool (informational). we considered resources coded under the education categories to be “learner-centered.” we also identified codes for the stage of the systematic review process: introductory, guidelines, planning phase, conducting searches, reference management, screening, data extraction, critical appraisal, and reporting.   we imported content from each library guide into nvivo 11, a data analysis tool that allows researchers to assign codes to text and to portions of web pages. each page of a guide was downloaded as a pdf and then imported into nvivo. the contents of all guides were coded for both type of resource and stage of the systematic review. for example, if a guide suggested covidence for title and abstract screening, and linked out to the tool, that portion of the page was given the resource type code of tool (information) and the systematic review stage code of screening. however, if the guide provided instructions on how to screen using covidence, that portion of the page would be coded with the resource type of tool (educational) and the stage code of screening. these would be counted as one occurrence each for tool (educational) and screening. if the same guide provided instructions on how to screen using covidence in more than one place, each occurrence was counted. data from nvivo was exported as a comma-separated value (csv) file using nvivo’s matrix coding query feature, which cross-tabulated the coding between the type of research and stage of the systematic review. the resulting file was imported into excel for descriptive statistical analysis. we counted the guides that had content pertaining to a stage and to a resource type. we calculated the proportion of resource types per guide. chi-square tests were used to compare the differences in distribution of the resource types within seven of the nine sr stages (introductory, planning phase, guidelines, conducting searches, reference management, data extraction, and reporting) to an expected hypothetical even distribution. a chi-square test “is formulated to determine whether the difference observed was due to a chance occurrence” (gordon, 2018, p. 269). if, for example, a stage was comprised of many more occurrences of one resource type than another, the results of the chi-square test would be significant. expected values for the screening and critical appraisal stages did not meet the conditions for a chi-square test, so the test was not run for those categories. minitab was used to run the chi-square tests. a value of p < .05 was considered significant.   table 2 descriptions of codes used for content analysis code description type of resource education (external) “the library’s educational efforts: that is, whether the libraries offer any educational services to the faculty, staff, and students at their institution” (yoon & shultz, 2017, p. 923). only includes online resources. includes detailed instruction, tutorials, quizzes, case studies, annotated screen captures, video tutorials. external: an educational resource developed by any institution that is not the institution that developed the library guide (e.g., a case study in the university of toronto guide developed by the university of pennsylvania). education (internal) “…the library’s educational efforts: that is, whether the libraries offer any educational services to the faculty, staff, and students at their institution” (yoon & shultz, 2017, p. 923). only includes online resources. includes detailed instruction, tutorials, quizzes, case studies, annotated screen captures, video tutorials. internal: an educational resource developed by the institution that developed the library guide (e.g., a video tutorial in the university of toronto guide developed by the university of toronto) information “…when libraries only provided descriptions … offering information about what it is and how researchers can do it, this study considered these passive services and coded them under the information category” (yoon & shultz, 2017, p. 923). this includes links to non-educational resources (e.g., a database, the prisma or cochrane webpage), definitions (e.g., defining “systematic review”), and descriptions (e.g., describing pico, describing a search strategy, but without instructions) service “…active library engagement with intended users (researchers) to help them and provide necessary information” (yoon & shultz, 2017, p. 923). services will include in-person services offered by the library to faculty and students conducting systematic reviews. includes consultations, co-author, facilitated searches. tool (educational) a resource that can be used to ease the systematic review process. for example, software to help with reference management, screening, critical appraisal, or data management. tools will be coded as educational if they provide instructions about how to use the tool. tool (information) a resource that can be used to ease the systematic review process. for example, software to help with reference management, screening, critical appraisal, or data management. tools will be coded as informational if they provide descriptions or links without instruction. stage of systematic review (sr) introductory definitions of srs or other related review types (e.g., scoping review), overviews of process or stages, rationale for conducting a systematic review, timelines, or team members appropriate for a systematic review planning phase question development (e.g., pico) but not search terms, consulting with a librarian during this phase, protocols (e.g., references to prospero), or developing protocols guidelines standards for systematic reviews, though not standards for reporting (e.g., crd, cochrane, campbell, jbi, iom). can include mentions of books about the entire process. conducting searches lists of databases to consider or search, database-specific or general filters/hedges that can be applied for searching, government documents, conferences, clinical registries, definitions of grey literature, search mechanics, boolean operators, saving searches on databases, creating appropriate search terms, search alerts reference management exporting searches to reference management software, de-duplicating searches, exporting references to excel, or interlibrary loan screening software or tools for screening abstracts or full texts, inclusion/exclusion criteria data extraction extracting of qualitative or quantitative data from studies for analysis critical appraisal quality assessment reporting prisma, other standards for reporting, writing of results, search documentation   results systematic review stages   no online library guide provided content on every stage of the systematic review. the most common stages included were introductory, conducting searches, and reporting. these stages were included in 17 of the 18 guides. the stage covered by the fewest libguides was critical appraisal which was addressed by nine guides (figure 3).   figure 3 number of guides by stage of systematic review (n = 18).   resource types: proportions by guides   most of the guides included text, infographics, embedded videos, links to external resources, and screenshots. however, university college london and mcmaster university only offered a single webpage of text and links.   for 17 of the 18 guides, information comprised over half of the resource types coded. in the case of one university (mcmaster), it was the only type of resource. only four guides included more than 30% of their guide dedicated to internally developed education resources. none of the guides had any content coded as tool (educational) (figure 4). every guide had content coded as information (figure 5). figure 4 proportion of resource types by guide (n = 18). values are included for the resource type: information and education (internal); percentages of all resource types within each column add up to 100.   figure 5 number of guides with each resource type (n = 18).   content   a total of 689 occurrences were coded across the 18 guides. the conducting searches stage had the most occurrences, 286 (42%), while critical appraisal had the fewest, 17 (2%) (figure 6).   of the 689 occurrences coded, most (458, 66%) were coded as information resources. interestingly, 20% (136) of the occurrences focused on internally developed education. this indicates that, to a small extent, locally created instructional resources are incorporated into systematic review online library guides. education (internal) and education (external) resources comprised 24% of the occurrences coded. apart from tool (educational), for which there were no occurrences coded, service had the fewest number of occurrences (19, 3%) (see figure 7).   within the stages of the systematic review, information comprised between 33% (screening) and 98% (guidelines) of the resource types. in many of the stages, information comprised the largest proportion of the resource types. the exception was the screening stage, where tool (informational) comprised 46% (11) of the occurrences. tool (informational) comprised between 1% (conducting searches) and 46% (screening) of the resource types within a stage. the education (internal) resource type comprised between 0% (guidelines) and 31% (reference management) of the resource types within a stage (figure 8).   table 3 displays the p-values for the valid chi-square tests. significant differences in distribution were found within the following stages: introductory, guidelines, planning phase, conducting searches, reference management, data extraction, and reporting. figure 6 number of occurrences coded: stage of systematic review (n = 689).   figure 7 number of occurrences coded: resource type (n = 689). figure 8 proportion of resource type by stage of systematic review (n = 689). percentages for only the relevant stages tool (informational), information, and education (internal) are shown for readability; percentages of all resource types within each stage add up to 100. note that tool (educational) is included in the legend; however, no stage included that resource type. table 3 significant differences in distribution of resource types within systematic review stagesa stage χ2 p introductory 286.28 (5, n = 95) < .001* guidelines 273.21 (5, n = 57) < .001* planning phase 119.72 (5, n = 67) < .001* conducting searches 545.02 (5, n = 286) < .001* reference management 35.45 (5, n = 35) < .001* screening see note   critical appraisal see note   data extraction 32.65 (5, n = 34) < .001* reporting 171.70 (5, n = 74) < .001* a the calculated expected values for the screening and critical appraisal stages did not meet the conditions for a chi-square test so the test was not run for those categories. *p < .05, statistically significant.   discussion   the stages most addressed by the systematic review online library guides aligned with the roles identified for librarians by spencer and eldredge (2018). the most common stages included in the guides were introductory, conducting searches, and reporting. the conducting searches stage described in our study included spencer and eldredge’s roles of search filters and hedges, searching (including subcategories of databases and other resources, grey literature, and search strategies), and source selection. spencer and eldredge’s planning role and general subcategory of searching is included in our introductory stage, and their reporting and documentation role is included in our reporting stage. their analysis of librarians’ roles in systematic reviews based on the literature is borne out, in part, by our analysis of library content on library guides. we found that the stage included by the fewest number of guides was critical appraisal (found in nine of 18 guides). interestingly, critical appraisal was also not included in spencer and eldredge’s 18 roles. this may be because critical appraisal, as a part of the systematic review process, should be covered by a guide, but is not a librarian role because it requires content expertise.   our chi-square tests showed that there was a significantly uneven distribution of resource types within seven of the nine stages. therefore, the resource types are distributed significantly differently from a hypothetical even distribution, and the difference is not due to chance. this can be seen especially within the introductory and guidelines stages, where the majority of the content is coded into one resource type. these results align with our initial observations that there is a preponderance of content in one resource type (information) over the others.   while all online library guides provided detailed information about systematic reviews, as well as some instructional resources on how to conduct systematic reviews, no guide provided instruction on how to use the tools related to the process. the resource type information comprised the majority of the resource types for almost all of the guides. in one case, it was the only resource type on the guide. it was also the only resource type that was found in all guides. therefore, the guides we found are lacking in instructional and teaching resources.   our content analysis showed that other institutions' systematic review online library guides are similar to our own: focused on information and links instead of on the instructional content to develop systematic review skills (tutorials, videos, step-by-step guides, and others) suggested by stone et al. (2018). this was most evident in regards to instruction on how to use various tools and software programs to carry out systematic reviews. this suggests that library guides on systematic reviews currently serve as information repositories rather than teaching tools.   despite the lack of learner-centered guides, two exemplars showcased a relatively high proportion of skills-focused resources in their systematic review guides. the university of british columbia (2021) linked educational worksheets in their library guide for researchers and students to develop their research questions, identify search terms, and create a prisma flow diagram, among other skills. monash university (2021) included an interactive case study tutorial about a student working through the stages of a systematic review. these two guides provided different approaches (worksheets and case studies) for educational resources.   we believe that libraries need to evolve their systematic review guides to better support and incorporate instruction grounded in pedagogical approaches. it appears that many librarians are designing guides as an informational supplement to in-person instructional sessions. while this serves a purpose, we believe that librarians should develop online instructional components for a few reasons:   to align with the larger move to asynchronous online learning (including flipped classrooms) that is occurring more broadly in post-secondary education (brown et al., 2020); to make such learning more widely accessible to those whose schedules make attending in-person workshops challenging; and to relieve the workload on librarians who are repeatedly required to teach clients how to use tools to carry out processes, rather than focusing on the higher-level skills required for a review.   the covid-19 pandemic, which resulted in the mass shutdown of the physical spaces of universities and their libraries worldwide, and the resulting rapid transition to online learning, illustrated the urgent need for libraries to evolve their online guides further in the direction of educational rather than primarily informational resources. while not something the authors considered when embarking on this project, we have been grappling recently with how to move our instruction online, either live via platforms such as zoom, or in the form of video tutorials and step-by-step guides that can be accessed at the point of need by remote learners. such tutorials can either stand on their own as a resource for clients who need to learn a particular task or tool, or supplement and reinforce synchronous instruction, face-to-face or online.   limitations   our assumption was that prolific universities would have online library guides that supported systematic reviews, including providing some form of online instruction. however, none of the 18 guides we looked at included any educational tools. the 18 guides were by no means representative of all systematic review library guides. the small sample size, 18, was the most feasible for us; however, it is a limitation. this study focused on university library guides; however, future studies may involve seeking out other, specific guides that contain educational tools, with the intent to analyze the stages that those tools support.    another limitation is that our content analysis did not evaluate the quality of the systematic review library guides. we did not make judgements as to whether the appropriate or correct information was included in any guide. further, we did not assess usability of the guides. a future study could investigate students’ and researchers’ perspectives and expectations when using a systematic review library guide.   conclusion   we undertook a content analysis of systematic review library guides in order to inform our own development of skills focused on instructional tools for those undertaking systematic reviews. we sought to determine what stages of the systematic review process libraries were supporting, and what educational resources already existed. we found that many guides reflected the state of our own: heavily informational and lacking in instructional and skills content.   we had hoped to avoid reinventing the wheel as we developed our own instructional tools; however, what we found was that the wheel appears not to have been invented. we suggest, as a future direction for systematic review instructional research and practice, that there is significant opportunity for librarians to turn their systematic review guides into learning tools through the development of online instructional tools to support student and researcher learning in this area.   acknowledgements   thanks to tak fung, phd, mathematical/statistical consultant, university of calgary for statistical advice; and margy macmillan and others, for feedback and comments.   author contributions   heather ganshorn: funding acquisition, writing – original draft, writing – review & editing k. alix hayden: conceptualization, funding acquisition, methodology, project administration, supervision, writing – original draft, writing – review & editing jennifer lee: formal analysis, funding acquisition, methodology, validation, visualization, writing – original draft, writing – review & editing helen pethrick: formal analysis, investigation, visualization, writing – original draft, writing – review & editing   references   baker, r. l. (2014). designing libguides as instructional tools for critical thinking and effective online learning. journal of library & information services in distance learning, 8(3-4), 107-117. https://doi.org/10.1080/1533290x.2014.944423   bergstrom-lynch, y. (2019). libguides by design: using instructional design principles and user-centered studies to develop best practices. public services quarterly, 15(3), 205-223. https://doi.org/10.1080/15228959.2019.1632245   bremner, n. (2019). from learner-centred to learning-centred: becoming a ‘hybrid’ practitioner. international journal of educational research, 97, 53-64. https://doi.org/10.1016/j.ijer.2019.06.012   brown, m., mccormack, m., reeves, j. brooks, d. c., & grajek, s. (2020). 2020 educause horizon report: teaching and learning edition. educause. https://library.educause.edu/resources/2020/3/2020-educause-horizon-report-teaching-and-learning-edition   bullers, k., howard, a. m., hanson, a., kearns, w. d., orriola, j. j., polo, r. l., & sakmar, k. a. (2018). it takes longer than you think: librarian time spent on systematic review tasks. journal of the medical library association, 106(2), 198-207. https://doi.org/10.5195/jmla.2018.323   german, e. (2017). information literacy and instruction: libguides for instruction: a service design point of view from an academic library. reference & user services quarterly, 56(3), 162-167. https://doi.org/10.5860/rusq.56n3.162   german, e., & graves, s. (2016). infusing pedagogy into libguides. in a. w. dobbs & r. l. sittler (eds.), integrating libguides into library websites (pp. 177-188). rowman & littlefield.   gordon, b. s. (2018). chi-square test. in b. b. frey (ed.), the sage encyclopedia of educational research, measurement, and evaluation (pp. 269-271). https://doi.org/10.4135/9781506326139.n109   haddaway, n. r., woodcock, p., macura, b., & collins, a. (2015). making literature reviews more reliable through application of lessons from systematic reviews. conservation biology, 29(6), 1596-1605. https://doi.org/10.1111/cobi.12541   hsieh, h. f., & shannon, s. e. (2005). three approaches to qualitative content analysis. qualitative health research, 15(9), 1277-1288. https://doi.org/10.1177/1049732305276687   kim, i., & kuljis, j. (2010). applying content analysis to web-based content. journal of computing and information technology, 18(4), 369-375. https://doi.org/10.2498/cit.1001924   lee, y. y., & lowe, m. s. (2018). building positive learning experiences through pedagogical research guide design. journal of web librarianship, 12(4), 205-231. https://doi.org/10.1080/19322909.2018.1499453   mckeown, s., & ross-white, a. (2019). building capacity for librarian support and addressing collaboration challenges by formalizing library systematic review services. journal of the medical library association, 107(3), 411-419. https://doi.org/10.5195/jmla.2019.443   monash university library. (2021, january 28). systematic review: case study starting developing a question. https://guides.lib.monash.edu/c.php?g=915307&p=6596596   page, m. j., shamseer, l., & tricco, a. c. (2018). registration of systematic reviews in prospero: 30,000 records and counting. systematic reviews, 7(1), 1-9. https://doi.org/10.1186/s13643-018-0699-4   parker, r. m. n., boulos, l., visintini, s., ritchie, k., & hayden, j. (2018). environmental scan and evaluation of best practices for online systematic review resources. journal of the medical library association, 106(2), 208-218. https://doi.org/10.5195/jmla.2018.241   roth, s. c. (2018). transforming the systematic review service: a team-based model to support the educational needs of researchers. journal of the medical library association, 106(4), 514-520. https://doi.org/10.5195/jmla.2018.430   spencer, a. j., & eldredge, j. d. (2018). roles for librarians in systematic reviews: a scoping review. journal of the medical library association, 106(1), 46-56. https://doi.org/10.5195/jmla.2018.82   stone, s. m., sara lowe, m., & maxson, b. k. (2018). does course guide design impact student learning? college & undergraduate libraries, 25(3), 280-296. https://doi.org/10.1080/10691316.2018.1482808   university of british columbia library. (2021, february 4). systematic and scoping reviews search methodology. https://guides.library.ubc.ca/systematicreviews/intro   visintini, s., boutet, m., manley, a., & helwig, m. (2018). research support in health sciences libraries: a scoping review. journal of the canadian health libraries association, 39(2), 56-78. https://doi.org/10.29173/jchla29366   white, m. d., & marsh, e. e. (2006). content analysis: a flexible methodology. library trends, 55(1), 22-45. https://doi.org/10.1353/lib.2006.0053     yoon, a., & schultz, t. (2017). research data management services in academic libraries in the us: a content analysis of libraries’ websites. college & research libraries, 78(7), 920-933. https://doi.org/10.5860/crl.78.7.920   syntheses synthesized: a look back at grant and booth's review typology classic   syntheses synthesized: a look back at grant and booth's review typology   a review of: grant, m. j., & booth, a. (2009). a typology of reviews: an analysis of 14 review types and associated methodologies. health information and libraries journal, 26(2), 91–108. https://doi.org/10.1111/j.1471-1842.2009.00848.x reviewed by: carrie price health professions librarian albert s. cook library towson university towson, maryland, united states of america email: carrieprice@towson.edu   received: 11 jan. 2022                                                                   accepted:  22 mar. 2022      2022 price. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30093     abstract   objective – the article, published in 2009, aims to provide a descriptive analysis of common review types to dispel confusion and misapplication of terminology.   design – an examination of terminology and methods applied in published literature reviews.   methods – grant and booth preliminarily performed a scoping search and drew on their own experiences in health and health information theory and practice. using literature reviews from the health information and libraries journal review feature and reviews identified in a previously published evaluation of methods in systematic reviews and meta-analyses (ankem, 2008), grant and booth examined characteristics of literature reviews. they subsequently identified variations in literature review methodologies and correlating vocabulary. after arriving at the conclusion that probing the review titles and descriptions—or alternatively, examining review workflow and timeframe processes—were not accurate for classifying review types, the authors chose to apply an analytical framework called search, appraisal, synthesis, and analysis (salsa). by examining the scope of the search, the method of appraisal, and the nature and characteristics of the synthesis and analysis, salsa helped the authors describe and characterize the "review processes as embodied in the description of the methodology" (grant & booth, 2009, p. 104). by employing an objective technique to categorize literature review types, the authors generated a descriptive typology.   main results – the authors provided a descriptive typology for 14 different literature reviews: critical review, literature review, mapping review/systematic map, meta-analysis, mixed studies review/mixed methods review, overview review, qualitative systematic review/qualitative evidence synthesis, rapid review, scoping review, state-of-the-art review, systematic review, systematic search and review, systematized review, and umbrella review. with the application of the salsa framework, the literature review types were defined and narratively described and summarized, along with perceived strengths, weaknesses, and a previously published example provided for comparison. two tables supplied a quick reference for comparing literature review types and examining selected reviews. a breakdown of review types was followed by a discussion of using and developing reviews in the library and health information science domain.   conclusion – inconsistency in nomenclature and methods across literature reviews perpetuates significant confusion among those involved in authoring or deciphering literature reviews. grant and booth noted the lack of an internationally agreed-upon set of review types, the formulation of which would set a precedent for a better understanding of what is expected and required of such publications. in supplying a historical context of the literature review (detailing both its importance as a synthesis of primary research and its value to users), grant and booth provided a useful narrative and typology to "inform how lis workers might approach the appraisal or development of a health information review" (p. 106).   commentary   background of the work   the publication of literature reviews, systematic and otherwise, has been on the rise for decades, at the very least since epidemiologist archie cochrane famously criticized the lack of a formal summary of research evidence in medicine (cochrane, 1979). evidence-inclined educators, researchers, practitioners, and students in various domains place value in reviews of the literature and consequently seek them out to inform research, policy, and practice. readers of reviews have found that methodological quality cannot be assumed and that reviews may follow disparate methods, introduce bias, or lack quality assessment or appropriate synthesis (shea et al., 2002; shea et al., 2007) despite stated methods.   prior to 2009, publications had evaluated and described singular literature review methodologies, such as systematic and scoping reviews. one famous example is the cochrane handbook for systematic reviews of interventions, the first version of which was developed and published in 1994 (cochrane, 2021) with successive versions to come. yet another example of an in-depth description of methodology is the still-relevant scoping studies' methodological framework articulated by arksey & o'malley in 2005.   likewise, other publications focused on the art and conduct of general literature reviews. perhaps most notable is a publication that appeared in 1988. oxman and guyatt in "guidelines for reading literature reviews" created a guide for readers of literature reviews who were grappling with increasing, and increasingly overwhelming, amounts of published medical literature. in their parting sentiments, oxman and guyatt (1988) state that "just as flawed methods in a study of diagnosis or therapy may invalidate the results, an unscientific literature review may come to incorrect conclusions" (p. 697). the 1988 publication facilitated future work, such as a literature review checklist (oxman, 1994), an article series (oxman et al., 1993) on reading the medical literature, and the user's guide to the medical literature, now in its third edition (guyatt et al., 2001). in 1997, greenhalgh included a chapter on "papers that summarize other papers" in her extensive how to read a paper: the basics of evidence-based medicine, focusing primarily on systematic reviews of the literature.   a few publications began to compare evidence synthesis methods but with limited attention or momentum. in 1988, cooper devised a taxonomy of literature reviews in education and psychology. there was difficulty in separating "what one hears from what one hopes to hear, and what is from what ought to be" (p. 123), leading again to the conclusion that "efforts at systematic evaluation will be fruitless unless a descriptive scheme . . . exists to structure the discussion" (p. 125).   finally, it was just before grant and booth's typology publication that literature review assessment tools started to appear such as the assessment of multiple systematic reviews (amstar) checklist and the subsequent amstar-2. tools such as these aimed to help readers assess methodological quality in systematic reviews (shea et al., 2007; shea et al., 2017).   prior to grant and booth (2009), no other authors had so clearly categorized and described 14 common literature review types. grant and booth, both fully aware of evolving trends in evidence-based medicine, describe the use of the systematic literature review to synthesize research evidence in health care: "gathering research, getting rid of rubbish and summarizing the best of what remains captures the essence of the science of systematic review" (p. 92).   the application of explicit methods to literature reviews is, booth anticipated as early as 2001, a practice that should be adopted in the library and information science sector for the purpose of "establishing a solid evidence base" within a field that had little (grant & booth, 2009, p. 92). "given the importance evidence-based practice places upon the retrieval of appropriate information," ambiguous and inconsistent terminology only perpetuates further uncertainty (grant & booth, 2009, p. 93).   in a 2017 letter in the health information and libraries journal, grant reflects on one of the typology's catalysts. as a new review editor at the journal in 2007, grant realized that "there were no consistent guidelines on the features a review should incorporate perpetuating a sense of confusion about what . . . was an indistinct and misapplied term" (p. 1). grant goes on to note that it was the shortage of consistent guidelines that led her to coauthor with booth.   expanding on ankem's (2008) work, which evaluated systematic review and meta-analysis methods in lis journals, grant and booth provided an "explicit basis . . . to gain a clear understanding of what [method] is being requested and the resources . . . required to meet the specification" for various types of literature reviews (grant & booth, 2009, p. 104).   impact of the work   grant and booth's work ignited a spark for those, both inside and outside of health information science, who hoped to see an implementation of more appropriate, more transparent, and more consistent methods for knowledge and evidence synthesis publications. while grant and booth originally set out to describe literature reviews in health information topics, the resulting publication proved useful for its generalizability to other fields and disciplines. it is likely for this reason that the article has been so popular in higher education, becoming highly cited not only in library science and health methods research but in other domains such as social and environmental science, psychology, business, engineering, computer science, and even the arts and humanities (scopus, n.d.). the article has had over 3,100 citations as of march 2022, with over 900 citations just in 2021 alone (scopus, n.d.). google scholar reports almost twice as many cumulative citations as scopus (google scholar, n.d.). the article's altmetric attention score, which is representative of online attention and other nontraditional metrics, was 394 as of march 2022 (altmetric, n.d.; elmore, 2008). a score of 20 or more, according to altmetric, demonstrates a higher-than-average level of attention (altmetric support, 2020). the article's significant altmetric attention score indicates that through media attention, including tweets, blog posts, and more, it has been shared widely on various digital platforms.   another indicator of this article's impact is the frequency with which it appears on library websites. a web search locates this article on many academic library literature review sites and guides, with over 100,000 google results in a search for this article's title limited to the .edu domain. these types of guides are frequently disseminated to educators or researchers by library staff when approached seeking guidance about literature reviews. in consulting the typology, potential authors can simplify the task of selecting the appropriate literature review methodology for their research and choose the one that aligns with their goals and objectives.  readers of the typology will also note the early discussion of the utility and purpose of "systematized" reviews, instructed to be "typically conducted as a postgraduate student assignment, in recognition that they are not able to draw upon the resources required for a full systematic review" (grant & booth, 2009, p.102).   since 2009, both authors have contributed further to the topics of methods, searching, writing, and reporting with grant's focus on public health writing and booth's attention on qualitative evidence synthesis and methods. in 2019, booth collaborated once again, this time with different coauthors, on a publication that identified 48 review types and categorized them into seven broad review families (sutton et al., 2019). despite the thoroughness and expansion on methodological approaches of this more recent article and others that followed, the simplicity of the 2009 typology can be more easily understood and therefore more easily applied by those starting out in evidence synthesis.   implications of the work   the grant and booth typology helps authors contribute to the evidence base and transform the knowledge landscape in their own disparate fields in consistent ways (grant & booth, 2009). the work encourages transparency in applied methods and sets out best practices, such as defining the scope of the search, describing the need and nature of quality assessment, and identifying the appropriate types of synthesis and analysis. all of these topics are receiving significant attention in academic communities today.   the typology has made its way into instructions for authors: any potential authors wishing to submit a review article to health information and libraries journal will find a link to the grant and booth article on the "author guidelines" page (health information and libraries journal, 2022), setting a worthy example for other journals and placing a responsibility on editors to adhere to published standards and consistency.   furthermore, review authors are beginning to understand the value of collaborating with a librarian or information specialist. various research demonstrates that librarians can contribute to more thorough search methods reporting (meert et al., 2016), higher quality searches (rethlefsen et al., 2015), and other unique skills such as records management, methods education (dudden & protzko, 2010), and publishing. professional expert review guidance like the institute of medicine's finding what works in healthcare: standards for systematic reviews (institute of medicine, 2011), the cochrane handbook for systematic reviews of interventions (higgins et al., 2022), and the jbi manual for evidence synthesis (aromataris et al., 2020) all encourage authors to collaborate with a librarian or information specialist for search strategy development in reviews of the evidence. along with this evolving trend comes the need for established best practices and guidance. over the past decade, a corpus of literature has started to emerge for those who would take on roles in literature review teams and participate in educating potential authors about appropriate review methodologies. many of these publications reference or build upon grant and booth's classic article. for example, the jbi manual for evidence synthesis (aromataris et al., 2020) explores methodological approaches for systematic reviews that are both interventional and non-interventional in nature while also advancing the methodology of scoping reviews that was first thoroughly examined by arksey and o'malley (2005) and later listed among grant and booth's (2009) 14 review types. another notable development in the literature since the publication of grant and booth's typology article is the standardization of reporting for systematic and scoping reviews through the prisma statement and its extensions for protocols, searching, scoping reviews, and more (page et al., 2020).   grant and booth succeeded in establishing a well-regarded typology that has stood the test of time and is still being frequently consulted and cited over a decade later. the 14 identified review types, which align with prospective objectives and research designs, is referenced by subsequent detailed guidance. developing terminological and methodological consistency will allow the "distinguishing features of each review type [to] be built up within the systematic review community through both direct comparison and emerging precedent" (grant & booth, 2009, p. 104). certainly, grant and booth's contributions benefit the global evidence synthesis community. the article's simplicity and straightforward approach make it an ideal starting point for many. through their typology, grant and booth have empowered those involved in evidence and knowledge synthesis to make appropriate decisions for the conduct and methodological applications of literature reviews not only in health information science but also in numerous other fields.   references   altmetric. (n.d.). altmetric bookmarklet [application]. https://www.altmetric.com/products/free-tools/   altmetric support. (2020). putting the attention score in context [support page]. https://help.altmetric.com/support/solutions/articles/6000233313-putting-the-altmetric-attention-score-in-context   ankem, k. (2008). evaluation of method in systematic reviews and meta-analyses published in lis. library and information research, 32(101), 91–104. https://doi.org/10.29173/lirg58   arksey, h., & o'malley, l. (2005). scoping studies: towards a methodological framework. international journal of social research methodology, 8(1), 19–32.   aromataris, e., munn, z. (eds.). (2020). jbi manual for evidence synthesis. https://synthesismanual.jbi.global   booth, a., noyes, j., flemming, k., gerhardus, a., wahlster, p., van der wilt, g. j., mozygemba, k., refolo, p., sacchini, d., tummers, m., & rehfuess, e. (2018). structured methodology review identified seven (retreat) criteria for selecting qualitative evidence synthesis approaches. journal of clinical epidemiology, 99, 41–52. https://doi.org/10.1016/j.jclinepi.2018.03.003   chandler j., cumpston m., thomas j., higgins j. p. t., deeks j. j., & clarke, m. j. (2021). chapter i: introduction. in j.p.t. higgins, j. thomas, j. chandler, m. cumpston, t. li, m. j. page, & v. a. welch. (eds.). cochrane handbook for systematic reviews of interventions version 6.2. cochrane. www.training.cochrane.org/handbook.   dudden, r. f., & protzko, s. l. (2011). the systematic review team: contributions of the health sciences librarian. medical reference services quarterly, 30(3), 301–315. https://doi.org/10.1080/02763869.2011.590425   cochrane, a. l. (1979). 1931 – 1971: a critical review with particular reference to the medical profession. in g. teeling smith, & n. wells (eds.), medicines for the year 2000, pp. 2–12. office of health economics. https://www.ohe.org/publications/medicines-year-2000#   cooper, h. m. (1988). organizing knowledge syntheses: a taxonomy of literature reviews. knowledge in society, 1(1), 104–126.   greenhalgh, t. (1997). how to read a paper: the basics of evidence-based medicine. bmj books.   elmore, s. a. (2018). the altmetric attention score: what does it mean and why should i care? toxicologic pathology, 46(3), 252–255. https://doi.org/10.1177/0192623318758294   elsevier b.v. (n.d.). scopus [database]. https://www.scopus.com/   google. (n.d.). google scholar [search engine]. https://scholar.google.com/   grant m. j. (2017). ten years of reviews. health information and libraries journal, 34(1), 1–4. https://doi.org/10.1111/hir.12175   grant, m. j., & booth, a. (2009). a typology of reviews: an analysis of 14 review types and associated methodologies. health information and libraries journal, 26(2), 91–108. https://doi.org/10.1111/j.1471-1842.2009.00848.x   guyatt, g., rennie, d., & american medical association. (2001). users' guides to the medical literature: a manual for evidence-based clinical practice. chicago, ill: american medical association.   health information and libraries journal. (2022, jan 28). author guidelines. https://onlinelibrary.wiley.com/page/journal/14711842/homepage/forauthors.html   higgins, j. p. t., thomas, j., chandler, j., cumpston, m., li, t., page, m. j., & welch, v. a. (eds.) (updated 2022 february). cochrane handbook for systematic reviews of interventions version 6.3. cochrane. https://training.cochrane.org/handbook   institute of medicine (us) committee on standards for systematic reviews of comparative effectiveness research, eden, j., levit, l., berg, a., & morton, s. (eds.). (2011). finding what works in health care: standards for systematic reviews. national academies press.   meert, d., torabi, n., & costella, j. (2016). impact of librarians on reporting of the literature searching component of pediatric systematic reviews. journal of the medical library association, 104(4), 267–277. https://doi.org/10.3163/1536-5050.104.4.004   oxman a. d. (1994). checklists for review articles. bmj (clinical research ed.), 309(6955), 648–651. https://doi.org/10.1136/bmj.309.6955.648   oxman, a. d., & guyatt, g. h. (1988). guidelines for reading literature reviews. cmaj: canadian medical association journal, 138(8), 697–703.   oxman, a. d., sackett, d. l., guyatt, g. h., browman, g., cook, d., gerstein, h., ... & wilson, m. (1993). users' guides to the medical literature: i. how to get started. jama, 270(17), 2093-2095.   page, m. j., mckenzie, j. e., bossuyt, p. m., boutron, i., hoffmann, t. c., mulrow, c. d., shamseer, l., tetzlaff, j. m., akl, e. a., brennan, s. e., chou, r., glanville, j., grimshaw, j. m., hróbjartsson, a., lalu, m. m., li, t., loder, e. w., mayo-wilson, e., mcdonald, s., mcguinness, l. a., . . . moher, d. (2021). the prisma 2020 statement: an updated guideline for reporting systematic reviews. bmj, 372(71), n71. https://doi.org/10.1136/bmj.n71   rethlefsen, m. l., farrell, a. m., osterhaus trzasko, l. c., & brigham, t. j. (2015). librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. journal of clinical epidemiology, 68(6), 617–626. https://doi.org/10.1016/j.jclinepi.2014.11.025   shea, b., moher, d., graham, i., pham, b., & tugwell, p. (2002). a comparison of the quality of cochrane reviews and systematic reviews published in paper-based journals. evaluation & the health professions, 25(1), 116–129. https://doi.org/10.1177/0163278702025001008   shea, b. j., grimshaw, j. m., wells, g. a., boers, m., andersson, n., hamel, c., porter, a. c., tugwell, p., moher, d., & bouter, l. m. (2007). development of amstar: a measurement tool to assess the methodological quality of systematic reviews. bmc medical research methodology, 7, 10. https://doi.org/10.1186/1471-2288-7-10   sutton, a., clowes, m., preston, l., & booth, a. (2019). meeting the review family: exploring review types and associated information retrieval requirements. health information and libraries journal, 36(3), 202–222. https://doi.org/10.1111/hir.12276     research article   first-year students’ research challenges: does watching videos on common struggles affect students’ research self-efficacy?   savannah l. kelly research & instruction librarian j. d. williams library university of mississippi oxford, mississippi, united states of america email: slkelly@olemiss.edu      received: 28 july 2017    accepted: 2 nov. 2017         2017 kelly. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – the purpose of this quantitative study was to measure the impact of providing research struggle videos on first-year students’ research self-efficacy. the three-part video series explicated and briefly addressed common first-year roadblocks related to searching, evaluating, and caring about sources. the null hypothesis tested was that students would have similar research self-efficacy scores, regardless of exposure to the video series.   methods – the study was a quasi-experimental, nonequivalent control group design. the population included all 22 sections (n = 359) of first-year writing affiliated with the fastrack learning community at the university of mississippi. of 22 sections, 12 (n = 212) served as the intervention group exposed to the videos, while the other 10 (n = 147) served as the control group. a research self-efficacy pretest – posttest measure was administered to all students. in addition, all 22 sections, regardless of control or intervention status, received a face-to-face one-shot library instruction session.   results – as a whole, this study failed to reject the null hypothesis. students exposed to the research struggle videos reported similar research self-efficacy scores as students who were not exposed to the videos. a significant difference, however, did exist between all students’ pretest and posttest scores, suggesting that something else, possibly the in-person library session, did have an impact on students’ research self-efficacy.    conclusion – although students’ research self-efficacy may have increased due to the presence of an in-person library session, this current research was most interested in evaluating the effect of providing supplemental instruction via struggle videos for first-year students. as this was not substantiated, it is recommended that researchers review the findings and limitations of this current study in order to identify more effective approaches in providing instructional support for first-year students’ research struggles.     introduction   academic faculty who teach a three-unit lecture course spend 45 hours per semester with students in the classroom. this in-person delivery is in addition to the 90 hours of accompanying homework per semester expected of students for a three-unit lecture course. in contrast, an academic librarian working alongside that faculty member is allotted approximately one hour of in-person class time with students during the semester, with no expectation that students will complete homework in preparation for that period. librarians are asked to use the designated hour to introduce students to the breadth and depth of academic research, often within the context of a particular assignment and with the expectation that students will engage in active learning (e.g., hands-on searching and evaluating activities). this type of library instruction is referred to in the literature as the one-shot. although academic librarians have objected to this arrangement, it continues to be the de facto assumption between classroom faculty and librarians at many institutions, including ours.   at the university of mississippi, each librarian in the research & instruction department teaches approximately 100 one-shot library instruction sessions annually. as requests for in-person course instruction continue to increase, especially within first-year curriculum, librarians struggle to balance faculty demand while also adequately supporting students’ needs. the perspective among our librarians is   that the current one-shot model mocks our best efforts in providing valuable and impactful pedagogy for undergraduate students. as tenure-track professionals we understand first-hand the difficulties of confronting irrelevant search results, evaluating whether an article aligns with one’s research question, and generating enough energy to follow-through on a difficult topic. yet as librarians in the classroom we often set aside the complexities of the research process due to the inherent limitations of the one-shot. the vast majority of instructional time, especially with first-year students, is spent establishing foundational concepts (e.g., what are keywords? what sources exist?) with limited opportunity to address where students struggle most (e.g., why is this search not working? does this source agree with my research argument? why should i care about sources?). consequently, when students encounter research struggles—failed searches, roadblocks, or dead ends—they tend to do so on their own.   our department wanted to fully support students who encounter such difficulties, but it was not feasible to double the workload by asking librarians to address research struggles via a second in-person session for every first-year course. therefore, in order to offer additional instructional support to students, while also maintaining current levels of in-person instruction, we decided that video tutorials might serve as a viable option for providing supplemental instruction aimed at addressing students’ research challenges.   the majority of one-shot library sessions correspond to first-year and second-year writing courses, and our department had a positive relationship established with the teaching faculty in the department of writing & rhetoric. first-semester, first-year students often have the most difficulty acclimating to academic research expectations and therefore our department decided to pilot the research struggle videos in the first-year writing (writ 101) course. in order to ensure that all students enrolled in writ 101 were first-semester, first-time students in college, the sample was limited to sections affiliated with the fastrack learning community. otherwise, the sample might have been an amalgamation of first-year students along with juniors or seniors who had delayed taking writ 101.   the purpose of this applied research was to measure the impact of providing supplemental video tutorials for first-year students in addition to the one-shot library session. the video series addressed common research roadblocks related to searching, evaluating, and caring about sources. to measure the effectiveness of the videos, a pretest-posttest quasi-experimental research study with control and intervention groups was designed. this research article outlines the development, execution, and effectiveness of this approach on first-year students’ research self-efficacy.   literature review   as early as the mid-90s, academic librarians were creating controlled studies to compare computer-assisted instruction (cai) with traditional face-to-face delivery. in 1998, college & research libraries published a study conducted by ucla librarians kaplowitz and contini using pretest-posttest data comparing cai and in-person instruction with biology students during the 1994-1995 academic year. the authors concluded that cai, although time-consuming and expensive, was a worthwhile endeavor and likely alternative to conducting face-to-face library instruction. this research was supported a few years later by germain, jacobson, and kaczor (2000) who found that their web-based instructional model improved students’ library skills as effectively as in-person library instruction. while academic librarians continued to embrace online methods of instruction, australian librarians at deakin university published an article comparing face-to-face with standalone and meditated tutorials (churkovich & oughtred, 2002). their findings, in contrast to previous studies, supported that students’ library skills increased more with in-person instruction.   in 2008, zhang, watson, and banfield conducted a systematic review of all library literature from 1990 to 2005 that compared cai with face-to-face instruction. they limited their study designs to rigorous randomized controlled trials, controlled trials, cohort studies, and case studies that used both pretest and posttest measures. of 728 potential studies, only 10 were included in the final analysis. even so, zhang et al. asserted that those ten studies lacked methodological rigor, notably internal and external validity. despite that admonition, research studies comparing face-to-face and online instruction continued to advance the literature without addressing some of the methodological concerns pointed out by zhang et al. anderson and may (2010) compared three forms of instruction—in-person, blended, and online—and concluded from pretest-posttest data that the method did not affect students’ retention of information literacy skills. shortly thereafter, archambault (2011) analyzed student artifacts created from different methods of instruction and found that students performed better with cai alone than with combined in-person and cai instruction. continuing the trend, walton and hepworth (2012) published a study analyzing u.k. students’ source evaluation comments resulting from three different interventions. their research shared similarities between archambault’s artifact analysis and anderson and may’s research design. however, walton and hepworth’s quantitative study diverged from anderson and may’s (n = 103) in that their sample size was significantly smaller (n = 35). the following year, hess (2014) compared in-person, online, and combined instruction with upper-division sociology students with a sample size (n = 36) as small as in the walton and helpworth study. controlled studies published in the library literature during the past few years have had significantly fewer participants than some of the studies evaluated in zhang et al.’s systematic review, including kaplowitz and contini (1998) with 423 students and germain et al. (2000) with 303 students. these smaller sample sizes are of concern, especially when comparing multiple interventions, as their size often affects the study’s power to detect statistical significance as well as external validity. most recently in the literature, bordignon et al. (2016) conducted a controlled study comparing students’ information literacy skills in response to online il learning objects and face-to-face workshops. their sample (n = 110) was comprised of 75 students during spring semester and 35 students the subsequent fall term. results indicate that statistically significant differences existed between participants’ pre-and postresponses in relation to finding articles.       the aforementioned studies compared cai or online methods with face-to-face instruction. in each of these cases, the dependent variables were skill-based outcomes (anderson & may, 2010; bordignon et al., 2016; germain et al., 2000; hess, 2014) or a combination of skill-based and affective measures (churkovich & oughtred, 2002; kaplowitz & contini, 1998). other studies have more thoroughly investigated students’ affective approaches, though not within the context of comparing online and in-person interventions. kracker’s (2002) pretest-posttest mixed methods study measured the “awareness of the affective aspects of the research process” (p. 284), as well as anxiety and satisfaction between students who were presented with information on kuhlthau’s information search process (isp), and those that were not. kracker and wang (2002) then used qualitative data from the same study to categorize students’ research experiences into three affective dimensions: emotional states, perceptions of the process, and affinity to research. such empirical studies helped set the stage for addressing students’ affective approaches to research.   in recent years, librarians have argued that standards of information literacy are incomplete without particular attention to students’ affective dimensions (fourie & julien, 2013; schroeder & cahoy, 2010). these concerns were publically addressed when the association and college and research libraries (acrl) created, revised, and officially adopted the framework for information literacy for higher education (2015). the framework addressed both cognitive (knowledge practices) and affective (dispositions) engagement within the context of information literacy.   it is of interest that librarians continue to explore the interactions between information literacy and students’ affective dimensions. this current empirical study measured the impact of research struggle videos on students’ research self-efficacy. bandura’s (1977; 1982; 1984; 1986; 1997) foundational and prolific work on self-efficacy undergirds this study, as well as almost all studies (kurbanoglu, 2003; mi & riley-doucet, 2016) investigating people’s “beliefs in [their] capabilities to organize and execute the courses of action required to produce given attainments” (bandura, 1997, p. 3). the construct of self-efficacy is closely related to, although not the same as, measures of self-confidence. both self-confidence and self-efficacy scales measure confidence levels, but only a confidence scale assumes that an action has taken place (stankov, 2013). perceived self-efficacy, on the other hand, is “not a measure of the skills one has but a belief about what one can do under different sets of conditions with whatever skills one possesses” (bandura, 1997, p. 37). such self-efficacy beliefs “affect thought processes, the level and persistency of motivation, and affective states...people who have strong beliefs in their capabilities approach difficult tasks as   challenges to be mastered rather than as threats to be avoided” (bandura, 1997, p. 39).   there is no single instrument for measuring self-efficacy (bandura, 2006); this construct is notably contextual and framed in relation to particular domains of functioning (byrne, flood, & griffin, 2014; kurbanoglu, 2003). in the current study, research self-efficacy was operationalized as first-year students’ academic research skills in relation to searching, evaluating, and caring about sources. according to bandura (1997), levels of self-efficacy are affected by four inputs: mastery experiences, vicarious experiences, verbal persuasions, and psychological and affective states. the most important, mastery experiences, reflect an individual’s prior successes or failures in relation to a particular task. most first-semester, first-year students lack successful mastery experiences in relation to academic research skills. vicarious experiences, on the other hand, are those successes or failures as modeled by someone else. in this current study, video tutorials were employed as a form of modeling successful approaches in overcoming common research struggles in relation to searching, evaluating, and caring about sources.    as a whole, quasi-experimental studies are largely underrepresented within the library and information science field. this study complements, but does not replicate, the extant literature comparing online and face-to-face interventions. rather than examining two different formats, this study sought to measure the impact of providing supplemental video tutorials in addition to, but not in lieu of, in-person library instruction. this study also diverges from the literature in that these particular videos directly addressed common research struggles with the expectation that modeling these experiences via video tutorials would have a positive impact on students’ confidence and reported research self-efficacy.   aims   the purpose of this study was to examine the impact of providing research struggle videos to first-year students enrolled in first-year writing (writ 101) courses. specifically, does watching research struggle videos prior to in-person library instruction affect students’ research self-efficacy? the null hypothesis tested was that first-year students enrolled in fastrack writ 101 courses who watched research struggle videos prior to in-person library instruction would report the same levels of research self-efficacy as students who did not watch the videos prior to in-person library instruction.   methods   this research study was a nonequivalent control group design (design 10: campbell & stanley, 1966). the quasi-experimental approach diverges from a traditional experimental pretest-posttest control group design due to the inability to randomize individual participants. although random assignment was used to determine which class sections would serve as control and intervention groups, random sampling was not possible given that individuals signed up for sections based on course schedule preferences. twenty-two fastrack first-year writing sections were offered during fall 2016 (n = 359); of those, 10 sections served as the control group (n = 147), while the other 12 sections served as the intervention group (n = 212). eleven sections were originally scheduled for each group, however one of the control sections was unintentionally designated as intervention and treated accordingly. the control and intervention groups were randomly assigned among the seven fastrack writing instructors after ensuring that each faculty member had at least one intervention group during the semester. an open channel of communication was established between the author and writing instructors regarding the development of this project to ensure buy-in and full collaboration prior to submitting the proposal to the institutional review board (irb) in august 2016.   the struggle video tutorials were created to explicate and briefly address common first-year students’ research struggles as it related to searching, evaluating, and caring about sources. the content was developed based on the author’s 10 years of academic library experience working with first-year students. the first video established a research claim and demonstrated ways to search iteratively when encountering poor results; the second video evaluated an academic source that both agreed and disagreed with the hypothetical research claim; and the final video discussed the value of expending time and energy in caring about one’s sources. the purpose of these video tutorials was to provide a vicarious experience (i.e., modeling) through which students would learn how to overcome common research struggles. this modeling via video tutorials was intended to increase students’ confidence in their academic research skills (i.e., their research self-efficacy).   the length, pacing, and approach of each video followed recommendations set forth by van der meij and van der meij (2013) for instructional content. after writing scripts, the videos were recorded via screenflow, and a personalized introductory video was added using a green screen in the library’s recording studio. the final video series would take students approximately seven minutes to watch. videos were played consecutively in the following order:   1.       introduction to videos, 0:35 2.       the struggle with searching, 2:36 3.       the struggle with evaluating, 2:15 4.       the struggle to care, 1:11   in this study, research self-efficacy was operationalized as first-year students’ academic research skills—searching, evaluating, and caring about sources. after the development of the video series, the self-efficacy scale was created to measure students’ confidence with respect to the domains of functioning that were addressed in the video series. this alignment between the independent and dependent variables sought to capture the impact of the struggle videos on students’ research self-efficacy.   the construction of the research self-efficacy scale followed examples bandura set forth in his chapter entitled “guide for constructing self-efficacy scales” (2006). examples used three anchors (cannot do at all, moderately can do, highly certain can do) with a confidence range of 0-100, coupled with imperative statements (e.g., stop yourself from worrying about things; get students to work well together). in order to replicate bandura’s recommended structure, the same anchors and confidence range were employed, while the imperative statements were revised to reflect academic research skills for first-year students. several items on the scale were designed to present “types of dissuading conditions” (bandura, 2006, p. 311) that first-year students would encounter (i.e., research struggles) when locating and evaluating sources. these “gradations of challenge” (bandura, 2006, p. 311) are intrinsic to measurements of self-efficacy and notably contingent on context.   in the current study, it was not feasible to establish construct validity through rigorous factor analysis prior to administration. however content validity was addressed by ensuring that the scale represented all three components of research self-efficacy as operationalized in this study as the ability to search, evaluate, and care about sources. in addition, face validity was established among undergraduates after piloting the survey with eight lower-division students who participated in cognitive interviews while responding to the scale. the current scale went through two revisions based on student feedback prior to administration. both the pretest (a = .827) and posttest (a = .869) scores in this current study indicated strong internal consistency reliabilities of cronbach’s alpha. the full scale as provided to the students is included in appendix a.      table 1 means and standard deviations for pretest and posttest responses by group variable control group intervention group pretest posttest pretest posttest (n = 147) (n = 126) (n = 212) (n = 187) m   sd      m   sd        m    sd m    sd use google.com 88.53 15.64 90.44 14.48 86.44 18.99 86.45 18.95 use um library website 61.39 29.03 76.55 22.92 60.30 27.40 77.41 21.38 adjust search terms 82.16 17.81 84.95 15.89 76.04 22.84 79.96 19.72 evaluate agree 83.34 17.12 84.28 17.25 81.33 18.31 85.29 15.81 evaluate disagree 82.84 18.35 85.24 15.80 79.58 20.01 84.59 16.92 continue looking 76.76 21.04 80.87 19.28 72.49 21.49 78.04 20.22 care about quality 80.81 18.85 83.21 17.18 77.21 21.09 80.67 17.04 keep from being frustrated 55.48 27.21 65.92 24.44 55.26 26.19 63.94 26.40 not care about topic 68.50 24.74 74.70 21.80 63.81 24.82 71.14 22.83 care due tomorrow 76.17 25.70 81.19 22.01 75.81 24.80 80.66 20.26       once the struggle videos and research self-efficacy scale were created, it was essential to coordinate the exact timing of these variables across all 22 writ 101 sections. to ensure as much internal validity as possible and to offset potential intervening variables, the researcher provided a regimented timeline to all seven writing instructors. one-shots were scheduled for all 22 classes during a two-week period in october. faculty administered the research self-efficacy scale (pretest) on paper to their sections during class the day prior to each section’s scheduled one-shot. the 12 sections designated as the intervention group then watched the research struggle videos collectively during class immediately after taking the pretest measure. the following class period, regardless of control or intervention status, students participated in an active, one-shot session. the faculty then administered the research self-efficacy scale again (posttest) to all sections the same day students turned in their research assignment, approximately two weeks after the one-shot. students who were absent during the pretest measure were asked by their writing instructors to refrain from taking the posttest measure. the difference between the control and intervention groups was the presence of the video series. everything else, including the teaching librarian and the content of the one-shot, was kept the same for all 22 class sections.   results   measures of central tendency and variability are provided in table 1 for both the control and intervention groups’ pretest and posttest scores. all students’ initial pretest scores were higher than the researcher anticipated, with several means in the mid-80s. although it is possible that these reported levels were a reflection of the anchors adopted during the creation of the scale (i.e., would responses have differed if the anchor near 90-100 range stated “absolutely certain can do” rather than “highly certain can do”?), the responses are more likely a manifestation of the dunning-kruger effect as noted throughout the literature on self-assessments (guillory & blankson, 2017; kruger & dunning, 1999; miller & geraci, 2011). however high the pretest means, it was still possible for upward movement in the posttest measure.   in the pretest for both the control (m = 88.53) and intervention (m = 86.44) groups, students were most confident in their ability to use  google to locate sources, and solidly confident that they could evaluate whether a source agrees (m = 83.34, m = 81.33) or disagrees (m = 82.84, m = 79.58) with a research argument. both groups reported the least confidence that they could keep from being frustrated when unable to locate relevant sources on a topic (m = 55.48, m = 55.26), and using the um library’s website to locate relevant sources for writ 101 class assignments (m = 61.39, m = 60.30). the two aforementioned items also represented the largest variability of responses in the pretest (i.e., standard deviations were much higher; frustrated: sd = 27.21, sd = 26.19; library: sd = 29.03, sd = 27.40). in other words, although the means were lower, students overall reported a wider range of confidence for these two measures. a final observation from the pretest data was that students in the control group had slightly higher means than students in the intervention group on several items, most likely due to differences in sampling. an independent samples t-test between the control and intervention groups on the pretest responses indicated no significant differences existed between the groups, except for one variable: adjust your search terms if the results from a search are not relevant or useful. here, the control and intervention groups were significantly different at the outset, t(352.084) = 2.848, p = .005, equal variances not assumed.     table 2 independent samples t-test between control and intervention posttest responses variable t df sig. (2-tailed) cohen’s d use google.com 2.11 307.93^ .035* .24 (small) use um library website -.30 312 .764 .03 adjust search terms 2.49 303.32^ .013* .28 (small) evaluate agree -.47 312 .636 .05 evaluate disagree .40 312 .686 .05 continue looking 1.27 312 .205 .15 care about quality 1.28 312 .201 .15 keep from being frustrated .71 312 .477 .08 not care about topic 1.40 312 .163 .16 care due tomorrow .22 311 .826 .03 total score (å 1-10) 1.24 311 .217 .14       it is clear from table 1 that students’ responses in both groups increased from the pretest to posttest measure, but it is not clear whether the control and intervention groups’ posttest responses were markedly different from one another. in order to test the null hypothesis of no difference between the control and intervention groups, an independent t-test (a = .05, two-tailed) was computed using the posttest scores. a paired t-test was not possible due to students’ anonymity taking the pretest and posttest measure. the independent samples t-test between the control and intervention posttest responses is provided in table 2. as a whole, the study failed to reject the null hypothesis (p > .05). the total score between students who watched the videos and students who did not watch the videos was not significant, t(311) = 1.24, p = .217, d = .14. however, a statistically significant difference did exist between the groups on two independent scale items: using google, t(307.93) = 2.11, p = .035, d = .24, and adjusting search terms, t(303.32) = 2.49, p = .013, d = .28). the difference between the control (m = 90.44) and intervention (m = 86.45) groups regarding google was unexpected given that the struggle videos did not discourage viewers from using google. as evident in table 1, while students in the control group reported increased confidence in their ability to use google between the pretest and the posttest (m = 88.53, m = 90.44), those in the intervention group remained stable across both measures (m = 86.44, m = 86.45). the second item—adjusting search terms—in which the control and intervention groups were significantly different on the posttest, is difficult to interpret without acknowledging that the control and intervention groups were at the outset significantly different on this item in the pretest.   as evinced in table 1, students’ research self-efficacy levels increased from the pretest to the posttest in all measures, regardless of control or intervention groups (total scores: pretest, n = 359, m = 739.64, sd = 140.67; posttest, n = 313, m = 795.88, sd = 134.57). thus a final independent t-test was computed to determine if a statistically significant difference existed between all groups’ pretest and posttest responses. as provided in table 3, nine of the ten items, as well as the total score, were statistically significant (p < .05). this significance, however, should be considered alongside the corresponding effect sizes (cohen’s d), which ranged from negligible to small to moderate. when working with large sample sizes, such as in this current study (n=359), it is often the effect sizes rather than the presence of statistical significance that relay the true magnitude of the difference between groups.     table 3 independent samples t-test between all groups’ pretest and posttest responses variable t df sig. (2-tailed) cohen’s d use google.com -.57 671 .572 .04 use um library website -8.47 663.02^ .001* .65 (moderate) adjust search terms -2.26 670.99^ .024* .17 evaluate agree -2.10 671 .037* .16 evaluate disagree -2.85 671 .005* .22 (small) continue looking -3.13 668.63^ .002* .24 (small) care about quality -2.09 670.15^ .037* .16 keep from being frustrated -4.67 671 .001* .36 (small) not care about topic -3.76 670.35^ .001* .29 (small) care due tomorrow -2.76 668.68^ .006* .21 (small) total score (å 1-10) -5.28 670 .001* .41 (small)       discussion   this current research tested the null hypothesis that first-year students enrolled in fastrack writ 101 courses who watched research struggle videos prior to in-person library instruction would report the same levels of research self-efficacy as students who did not watch the videos prior to in-person library instruction. as indicated in the results, the researcher failed to reject the null hypothesis. the probability was greater than .05 that the observed difference between means in the total score would have occurred by chance if the null hypothesis were true. although it is not uncommon for controlled studies to yield insignificant results after comparing instructional approaches (germain et al., 2000; hess, 2014; kaplowitz & contini, 1998;yong, levy, & lape, 2015), such outcomes should always be considered alongside effect sizes (e.g., walton & hepworth, 2012), as well as within the larger research framework. in this current study, the results can be evaluated within the specific context of a quasi-experimental design: namely, did the approach itself limit the impact of the video series on students’ research self-efficacy?   in order to preserve the integrity of the intervention, it was critical that students in the control group were not exposed to the videos. therefore the videos were not posted on youtube, the learning management system (lms), or emailed directly to the intervention group. the only way to maintain complete control over which students were exposed to the videos was to have the faculty member play the videos during class time. this, however, was not ideal, and did not allow students the opportunity to engage individually with the videos. students could not adjust the speed of the videos, or view again outside of that class period. although the videos followed best practices in terms of pacing, content, and “look and feel” (bowles-terry, hensley, & hinchliffe, 2010, p. 26), there was limited authentic engagement between students and the video tutorials in the classroom. in addition, it is recommended that future research studies employ undergraduate students, rather than librarians, to serve as narrators in the video tutorials. this important distinction is based on the theoretical consideration that vicarious experiences are most effective when the model is similar to, rather than different from, the viewer (bandura, 1997).   the timing of the video series in this current study is worth reconsideration. the intent behind providing video tutorials prior to the one-shot was that students who viewed the videos would be aware of potential roadblocks before they attended the one-shot. however, this approach most likely provided the video tutorials too early during the semester, during a time in which students had little to no context for understanding academic roadblocks. the videos may have also affected the level of engagement during the one-shot (i.e., did the videos prime or dissuade students from paying attention?). a more effective approach might have been to expose the intervention group to the videos after students gained hands-on academic experience during the one-shot library session.   a third limitation was the creation and administration of the research self-efficacy scale. although it is relatively common within library and information science literature (mahmood, 2017) to develop in-house self-assessment instruments, attempts to establish psychometric properties should be made prior to administration. although reliability and content and face validity were established for this current scale, it is highly recommended that future studies establish the instrument’s construct validity prior to administration with additional populations. a related consideration was the timing of the research self-efficacy scale. as noted previously, students reported surprisingly high levels of confidence during the pretest. this phenomenon was most likely a manifestation of the dunning-kruger effect, which is essentially that “the skills that engender competence in a particular domain are often the very same skills necessary to evaluate competence in that domain” (kruger & dunning, 1999, p. 1121). thus, first-year students’ lack of experience with academic research skills also made them unable to accurately assess their own competence in that domain. this overconfidence effect is most prominent among low-performers (guillory & blankson, 2017; kruger & dunning, 1999; miller & geraci, 2011), although low-performing students who overestimate their abilities also have less confidence in reported self-assessments than high-performing students (guillory & blankson, 2017; miller & geraci, 2011). on the other hand, it is also important to recognize that students’ posttest scores were higher than their initial pretest scores. this is an interesting observation given that self-assessments, including academic self-efficacy beliefs, tend to be more accurate when administered at the end of the semester rather than the beginning (gore, 2006; guillory & blankson, 2017). in this particular study, it is possible that the posttest measure was a more reliable instrument of students’ research self-efficacy beliefs since it was administered later in the semester than the pretest, and after students had the opportunity to engage in academic research.   although valuable to recognize the potential limitations of the current research design, it is equally sensible to acknowledge the outcome of this study: the struggle videos did not have a significant effect on students’ research self-efficacy. notwithstanding that, something did influence students’ research self-efficacy as evidenced in the upward trend between the pretest and posttest results (tables 1 and 3). given the effect sizes and that a statistically significant difference was observed across all students, regardless of control or intervention group, it is likely that all participants were exposed to the same experience. a plausible explanation, although not necessarily the only one (i.e., maturation), is the impact of the one-shot instruction session that all students received. it is likely that the in-person instruction, not the presence or absence of the struggle videos, affected students’ research self-efficacy during the semester.   conclusion   the purpose of this research was to measure the impact of providing supplemental instructional content via struggle videos for first-year students. the outcome was that students who were exposed to the video series reported similar research self-efficacy as students who were not exposed to the video series. although students’ research self-efficacy scores increased overall from the pretest to the posttest, this current study was not investigating the impact of the in-person library instruction.   the quasi-experimental approach, although rigorous, presented particular challenges, especially in providing an authentic environment for students to engage with the video tutorials. it is recommended that subsequent research examine the impact of providing self-paced, or interactive, struggle videos outside of the classroom environment. it is also important to recognize that the self-efficacy scale used in this study was created in-house. future researchers are encouraged to evaluate the scale’s construct validity prior to additional administrations. the limitations of this current study have been clearly delineated in the discussion for the benefit of researchers who, like our instruction librarians, are interested in more fully supporting students’ research struggles.   references   anderson, k. & may, f. a. (2010). does the method of instruction matter? an experimental examination of information literacy instruction in the online, blended, and face-to-face classrooms. journal of academic librarianship, 36(6), 495-500. https://doi.org/10.1016/j.acalib.2010.08.005   archambault, s g. (2011). library instruction for freshman english: a multi-year assessment of student learning. evidence based library and information practice, 6(4), 88-106. https://doi.org/10.18438/b8q04s   association of college and research libraries. (2015). framework for information literacy for higher education. retrieved from http://www.ala.org/acrl/standards/ilframework   bandura, a. (1977). self-efficacy: toward a unifying theory of behavioral change. psychological review, 84(2), 191-215. https://doi.org/10.1037/0033-295x.84.2.191   bandura, a. (1982). self-efficacy mechanism in human agency. american psychologist, 37(2), 122-147. https://doi.org/10.1037/0003-066x.37.2.122   bandura, a. (1984). recycling misconceptions of perceived self-efficacy. cognitive therapy and research, 8(3), 231-255. https://doi.org/10.1007/bf01172995   bandura, a. (1986). the explanatory and predictive scope of self-efficacy theory. journal of social and clinical psychology, 4(3), 359-373. https://doi.org/10.1521/jscp.1986.4.3.359   bandura, a. (1997). self-efficacy: the exercise of control. new york: freeman.   bandura, a. (2006). guide for constructing self-efficacy scales. in f. pajares & t. c. urdan (eds.), self-efficacy beliefs of adolescents (307-337). greenwich, ct: information age.   bordignon, m., strachan, g., peters, j., muller, j., otis, a., georgievski, a., & tamim, r. (2016). assessment of online information literacy learning objects for first year community college english composition. evidence based library and information practice, 11(3), 50-67. https://doi.org/10.18438/b8t922   bowles-terry, m., hensley, m. k., & hinchliffe, l. j. (2010). best practices for online video tutorials in academic libraries. communications in information literacy, 4(1), 17-28. retrieved from http://www.comminfolit.org/index.php?journal=cil&page=article&op=view&path%5b%5d=vol4-2010ar1   byrne, m., flood, b., & griffin, j. (2014). measuring the academic self-efficacy of first-year accounting students. accounting education, 23(5), 407-423. https://doi.org/10.1080/09639284.2014.931240   campbell, d. t. & stanley, j. c. (1966). experimental and quasi-experimental designs for research. boston: houghton mifflin.   churkovich, m. & oughtred, c. (2002). can an online tutorial pass the test for library instruction? an evaluation and comparison of library skills instruction methods for first year students at deakin university. australian academic & research libraries, 33(1), 25-38. retrieved from http://dro.deakin.edu.au/view/du:30001851   germain, c. a., jacobson, t. e., & kaczor, s. a. (2000). a comparison of the effectiveness of presentation formats for instruction: teaching first-year students. college & research libraries, 61(1), 65-72. https://doi.org/https://doi.org/10.5860/crl.61.1.65   gore, p. a. (2006). academic self-efficacy as a predictor of college outcomes: two incremental validity studies. journal of career assessment, 14(1), 92-115. https://doi.org/10.1177/1069072705281367   guillory, j. j., & blankson, a. n. (2017). using recently acquired knowledge to self-assess understanding in the classroom. scholarship of teaching and learning in psychology, 3(2), 77-89. https://doi.org/10.1037/stl0000079   hess, a. n. (2014). online and face-to-face library instruction: assessing the impact on upper-level sociology undergraduates. behavioral & social sciences librarian, 33(3), 132-147. https://doi.org/10.1080/01639269.2014.934122   kaplowitz, j. & contini, j. (1998). computer-assisted instruction: is it an option for bibliographic instruction in large undergraduate survey classes? college & research libraries, 59(1), 19-27. https://doi.org/10.5860/crl.59.1.19    kracker, j. (2002). research anxiety and students’ perceptions of research: an experiment. part i. effect of teaching kuhlthau’s isp model. journal of the american society for information science & technology, 53(4), 282-294.   kracker, j. & wang, p. (2002). research anxiety and students’ perceptions of research: an experiment. part ii. content analysis of their writings on two experiences. journal of the american society for information science and technology, 53(4), 295-307.   kruger, j. & dunning, d. (1999). unskilled and unaware of it: how difficulties in recognizing one’s own incompetence lead to inflated self-assessments. journal of personality and social psychology, 77(6), 1121-1134. https://doi.org/10.1037/0022-3514.77.6.1121   kurbanoglu, s. (2003). self‐efficacy: a concept closely linked to information literacy and lifelong learning. journal of documentation, 59(6), 635-646. https://doi.org/10.1108/00220410310506295   mahmood, k. (2017). reliability and validity of self-efficacy scales assessing students’ information literacy skills: a systematic review. the electronic library, 35(5), 1035-1051. https://doi.org/10.1108/el-03-2016-0056   mi, m., & riley-doucet, c. (2016). health professions students’ lifelong learning orientation: associations with information skills and self efficacy. evidence based library and information practice, 11(2), 121-135. retrieved from https://journals.library.ualberta.ca/eblip/index.php/eblip/article/view/26088/20416   miller, t. m. & geraci, l. (2011). unskilled but aware: reinterpreting overconfidence in low-performing students. journal of experimental psychology: learning, memory, and cognition, 37(2), 502-506. https://doi.org/10.1037/a0021802   schroeder, r. & cahoy, e. s. (2010). valuing information literacy: affective learning and the acrl standards. portal: libraries and the academy, 10(2), 127-146. https://doi.org/10.1353/pla.0.0096   van der meij, h. & van der meij, j. (2013). eight guidelines for the design of instructional videos for software training. technical communications, 60(4), 205-228.   walton, g. & hepworth, m. (2013). using assignment data to analyse a blended information literacy intervention: a quantitative approach. journal of librarianship and information science, 45(1), 53-63. https://doi.org/10.1177/0961000611434999     yong, d., levy, r., & lape, n. (2015). why no difference? a controlled flipped classroom study for an introductory differential equations course. primus, 25(9-10), 907-921.   zhang, l., watson, e. m., & banfield, l. (2007). the efficacy of computer-assisted instruction versus face-to-face instruction in academic libraries: a systematic review. the journal of academic librarianship, 33(4), 478-484. https://doi.org/10.1016/j.acalib.2007.03.006     appendix a pretest – posttest research self-efficacy scale   this scale is designed to help us get a better understanding of the kinds of things that are difficult for students. please rate how certain you are that you can do each of the things described below. consider only what you think you can do at this time (not at some point in the future). your answers are anonymous and confidential.   by continuing with this scale, you agree that you are 18 years of age or older.   rate your degree of confidence by recording a number from 0 to 100 using the scale given below:   0                   10            20           30          40            50           60          70            80              90           100 cannot                                                              moderately                                                         highly certain do at all                                                                can do                                                                  can do     confidence                                                                                                                                                                         (0-100)   use google.com to locate relevant sources for writ 101 class assignments                                                ______   use the um library’s website to locate relevant sources for writ 101 class assignments                       ______   adjust your search terms if the results from a search are not relevant or useful                                          ______   evaluate whether a source agrees with your research argument                                                                      ______   evaluate whether a source disagrees with your research argument                                                                ______   get yourself to continue looking for relevant sources when you can’t seem to find                    ______ what you need   get yourself to care about the quality of sources you use                                                                                     ______                 keep from being frustrated when you can’t find any sources related to your topic                                     ______   get yourself to care about locating sources when you do not care about your                                              ______ assignment topic   get yourself to care about source quality when your assignment is due tomorrow                                    ______             today’s date _______ & time  _________           evidence summary   public youth librarians use technology in ways that align with connected learning principles but face challenges with implementation   a review of: subramaniam, m., scaff, l., kawas, s., hoffman, k. m., & davis, k. (2018). using technology to support equity and inclusion in youth library programming: current practices and future opportunities. the library quarterly, 88(4), 315–331. https://doi.org/10.1086/699267     reviewed by: hilary bussell the ohio state university libraries columbus, ohio, united states of america email: bussell.21@osu.edu     received: 22 may 2019                                                                  accepted: 26 july 2019      2019 bussell. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29586     abstract   objective – to understand how public youth librarians use technology in their programming and what challenges and opportunities they face incorporating connected learning into their programming.   design – qualitative study   setting – phone calls and three library conferences (the young adult library services association symposium, the american library association midwinter meeting, and the maryland/delaware library association conference) in the united states. phone calls; in-person interviews; focus groups at the young adult library services association (yalsa) symposium, the american library association (ala) midwinter meeting, and the maryland/delaware library association conference.   subjects – a total of 92 youth-serving librarians and library staff in rural, urban, and suburban public libraries across the united states.   methods – subjects were recruited via social media, partner librarians, the project website, an association e-newsletter, and printed materials. the researchers conducted 66 semi-structured interviews between december 2015 and may 2016 and 3 focus groups between november 2015 and may 2016. the transcripts of the interviews and focus groups were coded using a thematic analysis approach informed by a connected learning framework.   main results – a total of 98% (65) of interview participants said they use technology in their youth programming; 69% (18) of focus group participants mentioned using technology in their youth programming. many youth-serving librarians use technology in ways that align with connected learning. youth-serving library workers are successful in finding community partners to help plan technology-enabled programming, they strive to develop connected learning programming based on the interests of their youth patrons, and they often take on the role of “media mentor” by exploring technology collaboratively with their patrons. youth-serving library workers face several challenges in implementing connected learning. these include difficulties with openly networked infrastructures, struggling to create learning environments that align with the hanging out, messing around, and geeking out (homago) stages of connected learning, and lack of confidence and experience in mentoring youth patrons on how to use technology.   conclusion – the authors recommend that library administrators improve access to openly networked technology both within and outside the library, and loosen overly-restrictive social media policies to give youth-serving library workers more flexibility and control. they also recommend that library administrators implement more training for library staff in skills relating to connected learning. the authors are creating a professional development toolkit to help public youth library workers to incorporate digital media and connected learning into their work with young patrons.   commentary   this study builds on the work of mizuko ito and others on connected learning. connected learning is an educational movement that “advocates for broadened access to learning that is socially embedded, interest-driven, and oriented toward educational, economic, or political opportunity” (ito et al., 2013). central to connected learning is the idea that new media can be harnessed to create learning outcomes that promote equity and inclusion for young people from diverse and underserved backgrounds (jenkins, 2009). public libraries are particularly suited to be sites of connected learning because of their historic role and mission of providing technology, spaces, and programming to foster learning and exploration. connected learning proponents argue that libraries can help bridge the digital knowledge divide by providing young people with opportunities to pursue hands-on, self-directed learning with digital and social media (peyton, 2018). however, in order to support these efforts, library administrators must be willing to be flexible on policies and practices, such as the use of social media in the library (ito & martin, 2013).   the article was evaluated using the casp qualitative checklist (critical appraisal skills programme, 2018). the aim of the research is clear, and the qualitative methods used are appropriate to investigate how youth-serving library workers use technology in their programming and what they perceive to be challenges and opportunities. both the data collection and data analysis processes are clearly articulated, and the findings are discussed in detail. the authors acknowledge in the limitations section that there may be a self-selection bias in the sample, given that participation in the study was voluntary; however, both the size and geographic diversity of the sample is robust.   this study provides evidence that the majority of youth-serving library workers are using technology in their programming, but that they need the support of their administrators to fully implement connected learning at their libraries. the authors give several specific recommendations that will be valuable both to library administrators who want to make sure their staff are getting the support they need, and to library workers who need specifics in making the case for this support. further, the researchers plan to use their findings to develop a professional development toolkit for youth-serving public library workers, thus increasing the usefulness of the research for library practitioners.   references   critical appraisal skills programme. (2018). casp qualitative checklist. retrieved from https://casp-uk.net/wp-content/uploads/2018/01/casp-qualitative-checklist-2018.pdf   ito, m., gutiérrez, k., livingstone, s., penuel, b., rhodes, j., salen, k., … & watkins, s. c. (2013). connected learning: an agenda for research and design. irvine, ca: digital media and learning research hub. retrieved from https://dmlhub.net/wp-content/uploads/files/connected_learning_report.pdf   ito, m., & martin, c. (2013). connected learning and the future of libraries. young adult library services, 12(1), 29-32. retrieved from http://leonline.com/yals/12n1_fall2013.pdf   jenkins, h. (2009). confronting the challenges of participatory culture: media education for the 21st century. cambridge, ma: mit press.   peyton, a. (2018). serving teen cocultures: what teens need from public libraries (an overview through the lens of the connected learning model). library philosophy and practice (e-journal). retrieved from https://digitalcommons.unl.edu/libphilprac/1881 evidence based library and information practice evidence based library and information practice 2010, 5.1  evidence based library and information practice       commentary    applying evidence in practice: what we can learn from healthcare    prudence w. dalrymple  institute for healthcare informatics  the ischool at drexel university  philadelphia, pennsylvania, united states of america  email: pdalrymple@drexel.edu    received: 29 dec. 2009          accepted: 04 jan. 2010       2010 dalrymple. this is an open access article distributed under the terms of the creative commons‐ attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐ sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original  work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is  redistributed under the same or similar license to this one.    applying research findings to practice is the  foundation of evidence based practice. in  healthcare, evidence based practice depends  upon the development, promulgation and  application of clinical guidelines. while  evidence based medicine (ebm) has been  enthusiastically embraced by many, gaps  persist, and transmission from research to  practice remains slow and uneven. the  perception that ebm threatens professional  autonomy accounts for some resistance, but  even among practitioners who support ebm  in concept, the uptake of guidelines has  encountered numerous barriers. a recent  study of guideline implementation by  residents in a tertiary care medical center  provides insight into the barriers to guideline  adoption, and draws parallels between the  uptake of ebm in the healthcare sector and the  uptake of evidence based library and  information practice (eblip) in the library and  information field. through increased  understanding of the diffusion of evidence  based practice in one field, lis practitioners  can position themselves to avoid similar  impediments.     background    in 2001, the institute of medicine (iom)  published its landmark report crossing the  quality chasm, which noted that it takes an  estimated 17 years for scientific evidence to  move from research finding to implemented  practice. the iom made four  recommendations to move evidence into  practice more swiftly:    • develop guidelines based on evidence  • disseminate guidelines through  application of information technology  • develop financial incentives for  adoption  • prepare the workforce and set goals  for improvement    clearly, the iom was calling for greater use of  evidence—most often through applying  43 mailto:pdalrymple@drexel.edu evidence based library and information practice 2010, 5.1  evidence based guidelines—and for effective  ways to implement them.    the library and information community has  been instrumental in efforts to increase the use  of evidence in practice. the growth of the ebm  movement has often included librarians  whose activities include quality filtering the  literature to find the best evidence, attending  clinical rounds to answer questions arising  during the review of individual patients,  conducting systematic reviews to support  guideline development, and working with the  medical community to propose a new kind of  health professional—the informationist—who  would provide the best evidence to the clinical  team during the course of clinical care.  librarians have also worked to introduce  evidence based practice into the field of library  and information practice and have adopted  many practices that are similar to those  provided as part of evidence based healthcare.     professional fields other than clinical medicine  have also been affected by the evidence based  movement. library and information science  has spawned the eblip movement which has  grown in stature and visibility over the past  decade or so. though clinical medicine has  arguably the longest and most robust  experiential basis for evidence based practice,  it is by no means universally adopted nor is it  free from criticism. despite criticism, calls for  evidence based practice have become  ubiquitous in many fields.     implementing ebm, however, has proven to  be more challenging than initially expected.  the literature of evidence based practice is  replete with studies on implementing  guidelines, and various theories have been  employed to explain, predict, and accelerate  adoption. one of the most frequently used  theoretical frameworks for understanding  implementation of evidence based guidelines  is diffusion of innovation. originated by  everett rogers (1995), diffusion of innovation  is defined as the process by which an  innovation is communicated through channels  over time among members of a social system.  (it has been popularized by malcolm gladwell  (2000) in his book the tipping point.) the key  elements in diffusion theory are: the  innovation, the communication channel the  rate of adoption, and the social system—the  context and organizational environment. the  innovation itself has also been found to have  an effect on the decision to adopt or reject an  innovation. these characteristics as they relate  to an evidence based guideline are: 1) relative  advantage over current practice (this could be  in terms of cost, convenience, efficacy, ease); 2)  compatibility with workflow; 3) complexity  (difficulty of use negatively affects adoption);  and, 4) observability (innovations that are  visible to others are more likely to be adopted  and spread to others). the process of adoption  itself is posited to have five stages: knowledge  or awareness, persuasion, decision,  implementation, and confirmation (or  maintenance).      diffusion of innovation provided the  theoretical framework for a case study of  medical residents’ use of a guideline for  managing a high risk, anticoagulant drug in a  top‐rated, academic medical center  (dalrymple, lehmann, roderer, & streiff, in  press). in‐depth interviews and participant  observation revealed that residents initially  consulted the evidence based guideline  provided to them, but when it didn’t produce  results as expected, or when its  implementation created problems in  workflow, pragmatism trumped evidence. the  residents disregarded the recommendations in  favor of their own experience. when a newer  guideline became available, however, the  residents adopted its recommendation because  it was more compatible with their workflow  while producing comparable results. in this  instance, the residents chose the evidence  based guideline that had the greatest relative  advantage and compatibility, as predicted by  diffusion theory. this qualitative case study,  while not generalizable, illustrates how  diffusion of innovation can help explain why  individuals may not adopt evidence based  guidelines despite their apparent commitment  to delivering excellent, safe patient care.     44 evidence based library and information practice 2010, 5.1  implications for evidence based  librarianship    what can be learned about evidence based  library and information practice from this  experience? insights from evidence based  medicine, and in particular, health care  informatics, can yield valuable lessons for  health information professionals who are  interested in encouraging the diffusion and  adoption of lis evidence into practice. they  can help information professionals understand  and appreciate the challenges inherent in  evidence based practice. the following section  will address some of the issues inherent in  diffusing lis evidence into practice.     the evidence base    creating an evidence base requires that  individuals must undertake research studies.  once studies are conducted, published, and  made accessible for systematic review process  and/or meta‐analysis, an evidence statement is  made available, along with an assessment of  its strength. typically these have been made  available in healthcare through the cochrane  collaboration. a final step is the issuance of a  guideline, generally carrying the imprimatur  of a recognized group or organization.      although the evidence base in lis is smaller  by far than the evidence base in medicine, it  does exist, and efforts on the part of groups  such as eblip and professional associations  such as the medical library association and  the special libraries association have done  much to encourage its growth. however,  barriers to conducting research, such as the  lack of time, money, knowledge and skill and  the absence of a culture of inquiry and  incentives from leadership all affect research  productivity.    availability of evidence    in medicine, the evidence is presented in the  form of guidelines, few of which are either  presented or consulted at the point of care.  both cognitive science and information  technology are being harnessed to bring  guidelines to the point of care through  computerized decision support systems  (cdss) as well as to a variety of mobile  devices. although they are very promising,  decision support systems are still being  refined, even after years and millions of  dollars have been spent in developing and  testing them.    though it is unlikely and probably not  necessary to design decision support systems  for information professionals, it may be both  feasible and desirable to deliver evidence to  the information professional’s desk top or  handheld device. it surely should be made  available at the time organizational decisions  are being made, yet many libraries and  librarians lack adequate access to the lis  research evidence base. the evidence  summaries published in this journal are a  good first step in making accessible the  evidence resulting from the lis research base.  that it is open access is also a very important  step. however, accessing the evidence  summaries requires the initiative to formulate  a question, undertake a search and access the  journal.     application of evidence     a premise of the ebm movement is that the  application of best evidence positively affects  patient care, and that the presentation of  evidence to health practitioners will  automatically result in their use of it. the  reality appears to be more complex. simply  providing evidence does not automatically  ensure adoption, because adoption usually  implies behavior change on the part of the  clinician.      this has implications for librarians and  information professionals both in their roles as  providers of evidence that supports  professional practice, and as professionals  themselves making decisions in the course of  daily work. in the first instance, information  professionals must recognize that even an  informationist service cannot guarantee that  the evidence presented will be adopted.  because it is difficult to know whether, when  45 evidence based library and information practice 2010, 5.1  and to what extent the presented evidence is  adopted, it is challenging to argue that  evidence has made a difference in patient  outcomes. as almost any health researcher  will attest, demonstrating effect in medicine is  very, very difficult; there are simply too many  intervening variables.      in the second instance, providing evidence  will not necessarily change librarians’  behaviors either. that is, diffusion and  adoption of an innovation—in this case, an  evidence based guideline—must be  understood not so much as an information  problem but as a communications and  behavioral change problem. it is essentially  change management, a point that andrew  booth (2009) suggests in his recent piece in  health information and libraries journal. to  implement change management, we need to  look beyond evidence summaries to examine  potential users, their problems, and their  organizations. as booth states, we need to  “look more widely at evidence that examines  the process by which we achieve quality  improvement and change strategies more  generally” (p. 83). indeed, the research issues  surrounding implementation of changes in  practice, especially within an organizational  context, have commanded greater attention in  the informatics literature of late.    data, information, knowledge     in healthcare informatics, one of the  challenges is to create and manage data  according to agreed‐upon standards so that it  can be aggregated across patients, institutions,  and regions. once data is aggregated, it can be  transformed into information that provides  the basis for decision‐making. while not  regarded as research in the sense that the rct  stands as the highest level of evidence, the  hierarchy of data, information and knowledge  can produce usable evidence that can be  applied in organizations to solve problems.  most clinicians don’t conduct research, but  they are often very keen to understand how  their patient outcomes compare to local,  regional and national norms. similarly, most  information professionals may not conduct  research, but they can collect data  systematically so that they can compare their  performance to other similar organizations.     initiatives such as libqual+ and libqual  lite, developed by the association of research  libraries, enable library and information  professionals “to assess and improve library  services, change organizational culture and  market the library.” (association of research  libraries, 2009). used in over 1000 libraries  worldwide, libqual provides data that can  be used to generated comparative studies  about how a library is performing. such  activities can lead to creation of best practices  and to comparative effectiveness research,  goals that are very similar to those of evidence  based practice in healthcare. increasingly,  health professionals acknowledge that failure  to apply evidence may be as much an  organizational issue linked to change  management as it is a decision on the part of  an individual practitioner. gathering data to  feed back to individuals and organizations  about their performance may be another route  to achieving the goals of evidence based  library and information practice, one more  suited to a world in which the individual  professional must work within the constraints  of an organizational culture.     conclusion    several insights from the iom’s qualitative  study of guideline adoption are applicable to  library and information professionals.  through increased understanding of the  diffusion of evidence based practice in one  field, eblip practitioners can position  themselves to overcome or avoid similar  obstacles and facilitate the adoption of  evidence based practice.     implications for practice   • challenges in applying research  findings to practice are informed by  studying guideline implementation.  • rogers’s theory on the diffusion of  innovation provides a useful frame of  reference for adoption of evidence.  46 evidence based library and information practice 2010, 5.1  47 • in a work‐based setting pragmatism  frequently triumphs over evidence.  • particular challenges for eblip  include the creation of an evidence  base, poor availability of evidence,  devising strategies for applying  evidence and the need to develop  routine data systems.    implications for research  • more research is needed around  evidence to support library decisions  and the utility of decision support  systems.  • we need to look beyond mere  provision of evidence summaries to  investigate potential users, their  problems, and their organizations.      references    association of research libraries. (2009).  libqual+. retrieved 20 feb. 2010 from  http://www.libqual.org/    booth, a. (2009). using evidence in practice:  eleven steps to eblip service. health  information and libraries journal, 26(1),  81‐84.    dalrymple, p.w., lehmann, h.p., roderer,  n.k., & streiff, m.b. (in press).  applying evidence in practice: a  qualitative case study of the factors  affecting residents’ decisions. health  informatics journal.     gladwell, m. (2000). the tipping point: how  little things can make a big difference.  new york, ny: little brown.    institute of medicine, committee on quality of  health care in america. (2001).  crossing the quality chasm: a new health  system for the 21st century. washington,  dc: national academy press.     lorenzi, n.m., novak l.l., weiss j.b., gadd  c.s., unertl k.m. (2008). crossing the   implementation chasm: a proposal for  bold action. journal of the american  medical informatics association, 15(3),  290‐296.    rogers, e.m. (1995). diffusion of innovations  (4th ed.). new york, ny: free press.  http://www.libqual.org/  develop guidelines based on evidence  disseminate guidelines through application of information technology  develop financial incentives for adoption  prepare the workforce and set goals for improvement implications for practice implications for research evidence based library and information practice evidence based library and information practice 2011, 6.2 45 evidence based library and information practice evidence summary indigenous traditional medical practitioners’ lack of formal medical education impacts their choices of information resources for the treatment of sickle cell anemia a review of: olatokun, w. m., & ajagbe, e. (2010). analyzing traditional medical practitioners’ informationseeking behavior using taylor’s information-use environment model. journal of librarianship and information science, 42, 122-135. reviewed by: maria c. melssen head of learning and information services medical library, herbert wertheim college of medicine florida international university miami, fl, united states of america e-mail: mmelssen@fiu.edu received: 8 feb. 2011 accepted: 29 mar. 2011 2011 melssen. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to determine the information seeking behaviours of traditional medical practitioners who treat sickle cell anemia patients. design – qualitative, intervieweradministered, structured questionnaire. setting – city and surrounding rural area of ibadan, nigeria. subjects – the researchers selected for this study 160 indigenous traditional medical practitioners who specialize in the treatment of sickle cell anemia. the majority of the subjects were male, with 96 male and 64 female. the practitioners were selected from four traditional medical practitioner associations in ibadan, nigeria. the researchers met with the leaders of the four organizations and identified which of the 420 members specialize in the treatment of sickle cell anemia. methods – the subjects were asked survey questions orally during face-to-face interviews. the decision to conduct interviews and ask the survey questions orally (rather than having the subjects complete the survey questions on their own) was based on the perceived low literacy level of the traditional medical practitioners. survey questions were mailto:mmelssen@fiu.edu� evidence based library and information practice 2011, 6.2 46 written using the analytical framework of taylor’s information use environment model. according to the authors, the premise of taylor’s information use environment model is that individuals can be grouped according to their “professional and/or social characteristics” (p. 124). the group is then characterized by the members’ approach to problem solving: the type of problems they encounter, the setting they find themselves in during the problem, and how the group as a whole determines what course of action needs to be taken in order to solve the problem. the problem solving strategy of the group impacts its need for information and how that information is located and used. the questions asked by the researchers fell into one of five research areas: • the environment of the group • the diagnosis and treatment methods of traditional medical practitioners and how they obtain information that shapes their diagnosis and treatment choices • sources of information for the treatment of sickle cell anemia and the factors that encourage or discourage the use of those sources • how information about sickle cell anemia is communicated amongst the traditional medical practitioners • the extent to which orthodox and traditional approaches to the treatment of sickle cell anemia are integrated. all 160 subjects completed the interview and all of the surveys were determined to be usable. main results – the main sources of professional knowledge and training of the traditional medical practitioners are their fathers (55%) and master healers (42.5%). this knowledge is orally preserved: none of the respondents completed a formal training program at a university. the information used to select the best treatment options for patients with sickle cell anemia is the patient’s diet or eating habits (62%) and new traditional remedies (55%). new traditional remedies are defined by the authors as “the location and potency of herbs, roots, bark and parts of animals used to compound drugs or make ritual sacrifices” (p. 128). the information found least useful by the traditional practitioners is the authenticity of new remedies (20%). the traditional practitioners would wait for their patients to report back regarding the success or failure of the treatment they were provided. the researchers also discovered that traditional practitioners rarely, if ever, share their diagnosis and treatment methodologies with other practitioners. the diagnostic tests for sickle cell anemia used most often by traditional practitioners are visual observation (32.5%) and history taking (48%). only a fraction of the practitioners (10%) utilize “orthodox methods” which include hb electrophoresis. the treatment option of choice by the majority of practitioners is concoctions (62.5%). the traditional practitioners favour informal sources of information over formal sources. the informal sources most commonly used are local associations (55%), colleagues (55%), and master healers (52.5%). such formal resources as medical journals, seminars or workshops, the internet, and libraries are rarely if ever used. the factors influencing the practitioners’ resource choice include relevance (87.8%), suitability (70%), and availability (67.5%). many practitioners also refer their patients to other traditional medical practitioners; however, very few (27.5%) refer patients to orthodox physicians. the traditional practitioners felt that they can treat their patients on their own and do not need the orthodox physician’s help. the traditional practitioners also feel that there is little or no information sharing between the traditional practitioners and the orthodox physicians: the only time information is exchanged between the two groups is when the orthodox evidence based library and information practice 2011, 6.2 47 physicians want to conduct research on traditional medical practices. conclusion – the traditional practitioners rely heavily on information from local experts to guide their treatment plans for sickle cell anemia patients. the success or failure of a given treatment plan is also based on what did or did not work in the past. these practitioners do not have a formal education and have a low literacy level. this group is not recognized by western medical culture as a result of their lack of professional, western medical training. another issue is that there is not a solid documentation system of the treatment and management of sickle cell anemia by this group. this is due to their fears of having their methods “stolen” by fellow practitioners. recommendations by the authors include having the association leaders document and track the treatment and disease management methods used by their members and implementing a training program for the indigenous traditional medicine practitioners. further research needed includes exploring the various ways to integrate western medical practices with traditional practices as well as investigating ways to encourage collaboration and sharing of information between indigenous medical practitioners. commentary though the data provided would be of interest to librarians whose patron population includes indigenous traditional medical practitioners and the patients this group serves, the subject selection process, study design, and methodology employed by the researchers puts the credibility of the data into question. the credibility concerns with this study began with the selection of research participants. there was no randomization of subjects selected: the leaders of the four traditional medical associations selected the research subjects and those subjects were the only individuals interviewed. this lack of randomization presents possible bias. also, there is no indication that ethics approval or consent from the research participants was obtained prior to conducting the interviews. the study design and methodology are also lacking in sufficient detail. the article does not indicate who conducted the 160 interviews or where the interviews were held. the only identifiable evidence of these two logistical matters is the citations under the tables on pages 128-130. it appears that this survey was conducted in the “field” between june 2007 and july 2007. this information is not provided in any other section of the article. the research instrument is not provided and there is no information regarding the number of questions asked or the content of those questions. the researchers did provide details regarding the five research questions they were seeking to answer; however, the lack of details regarding the specific questions asked negatively impacts the credibility of the study. a specific area of concern regarding insufficient details is the term “orthodox methods.” the authors, unfortunately, do not provide a specific definition for “orthodox methods.” this leaves the reader to infer that “orthodox methods” are current, modern medical methods as opposed to traditional methods. also of interest is that the use of spss was never mentioned in the article: spss is only mentioned in the abstract. further information regarding who handled the data or what, if anything, was done to minimize bias was not provided in the article. the researchers also state that all 160 surveys were usable. regretfully, there was no discussion on what criteria was used to determine usability of the surveys nor were any confounding variables addressed. the findings of this study underscore the need for further research on the information seeking behaviours of indigenous medical practitioners; however, the lack of an adequate description of the study gives the reader pause as to the credibility of the information provided. / evidence based library and information practice evidence based library and information practice evidence based library and information practice 2010, 5.3 53 evidence based library and information practice evidence summary lack of congruence between analyses and conclusions limits usefulness of study of socio-cultural influences on student choice of lis field in greece a review of: moniarou-papaconstantinou, v., tsatsaroni, a., katsis, a., & koulaidis, v. (2010). lis as a field of study: socio-cultural influences on students’ decision making. aslib proceedings: new information perspectives, 62(3), 321-344. reviewed by: diana k. wakimoto online literacy librarian, california state university, east bay doctoral student, san jose-qut gateway program hayward, california, united states of america email: diana.wakimoto@csueastbay.edu received: 8 july 2010 accepted: 27 aug. 2010 2010 wakimoto. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. objective — to determine how social and cultural factors influence students’ decision to study library and information science (lis) as undergraduates. design — semi-structured interviews and quantitative analysis of questionnaire data. setting — three schools in greece with lis programs at the undergraduate level. subjects — one hundred eighty-seven firstyear students enrolled in greece’s lis schools’ undergraduate programs in the autumn semester of the 2005-2006 academic year. methods — the authors piloted the questionnaire with 52 students at the lis school in athens and had three faculty members review the questionnaire. after modification, the two-part questionnaire was administered during the first week of classes to all first-year undergraduate students enrolled in greece’s three lis schools. the first section of the questionnaire collected data on student gender, age, area of residence, school from which they graduated, and parental occupation and level of education. the second part of the questionnaire covered students’ reasons for choosing lis as a field of study, the degree to which students agreed with dominant public views (i.e., stereotypes) of librarianship, and practical issues that influenced students’ decision-making processes. the authors conducted two rounds of semi-structured interviews with students mailto:diana.wakimoto@csueastbay.edu evidence based library and information practice 2010, 5.3 54 from the same 2005-2006 cohort. they interviewed 41 self-selected students and then interviewed a purposive sample of 15 students from the same cohort in the fifth semester of the students’ studies. main results — the questionnaire was completed by 187 lis students, with 177 responses considered relevant and used in the analyses. demographic information showed that 78% of the respondents were female, 85.8% were from urban areas, and 98.9% graduated from public schools. the authors constructed two indices to assist with further analyses: the educational career index, which quantified students’ educational experience prior to study at the university, and the divergence index, which was created by comparing students’ university entrance exam scores and students’ ranking of lis as a preferred field of study. the authors determined that 65% of the variance in the data was explained by two factors: students’ responses to library stereotypes and students’ self-reported reasons for choosing to study lis. the self-reported reasons for studying lis were combined into four variables (extrinsic reasons, intrinsic professional reasons, intrinsic academic reasons, and intrinsic social reasons) to be used in the multivariate analysis of variance tests (manovas). three distinct clusters of students were found using the indices and parental education level in cluster analysis: cluster 1 (low parental education, low educational career, and low divergence indices scores), cluster 2 (intermediate parental education, high educational career, and low divergence scores), and cluster 3 (high parental education, high educational career, and low divergence scores). for three of the factors for choosing the lis field (intrinsic professional reasons, intrinsic academic reasons, and intrinsic social reasons), cluster 1 showed statistically significant differences (p<.05) from cluster 2. cluster 1 showed statistically significant differences (p<.05) from cluster 3 for two aspects (intrinsic academic reasons and intrinsic social reasons). cluster 2 and cluster 3 showed no statistically significant differences. conclusion — the authors concluded that students with different socio-cultural characteristics have different reasons for choosing lis as a field of study and differ in their abilities to make competent decisions about their education. students with high socio-cultural resources choose lis for its intrinsic values and are able to make competent decisions. students with low sociocultural resources cannot make informed decisions regarding their chosen career paths and choose lis purely for the prospect of future employment. commentary the authors use theories and methodology influenced by the work of pierre bourdieu to analyze and explain why students choose to study lis at greek universities with respect to their socio-cultural backgrounds. it will be of interest to those desiring an international perspective on the demographics of lis students; it is less useful for understanding how demographics impact decision-making processes. awkward language and lack of definitions for key sociological terms are hurdles that detract from the clarity of this article. some awkwardness may be explained by translation issues if the authors’ first language is not english. the more serious problem is the reliance on undefined sociological jargon. for example, for those unfamiliar with the specific usage in sociology of the terms “social capital,” “agents,” or “social actors” (p. 327), the jargon would pose a significant obstacle to understanding the theoretical framework. the methodology and statistical analyses seem to be used appropriately, but inclusion of the questionnaire would have strengthened the article and clarified issues of data collection. the authors were clear on their use of statistics and there was only a minor discrepancy in the number of responses used in creating the evidence based library and information practice 2010, 5.3 55 divergence index (136 noted in table iv, but 128 according to calculations in the text). all variables were accounted for except age, and the authors did discuss some of the study’s limitations. the article’s major flaw is that the conclusions appear to be based on the interviews rather than the analyses of the questionnaires. while the analyses did show statistically significant differences between the clusters, it is a stretch to conclude that students with lower sociocultural resources have "a limited capacity to deal competently with complex decisions on educational matters" (p. 340) without also believing many of the assumptions upon which the authors built their analyses and conclusions. the authors cite their interview results to support this and other conclusions which could be valid, but lack of reported results from the interviews in this article makes it impossible to judge their validity. the authors should be commended for integrating sociological theories in lis research, but the lack of congruence between the questionnaire data analyzed and the interview results used to support the conclusions limits the usefulness of the article. this line of research warrants further study of the connections between students’ backgrounds and their choice to study lis, the results of which could be applied to the recruitment of future lis students. evidence based library and information practice evidence based library and information practice 2009, 4:4     evidence based library and information practice         article    it’s all fun and games until someone learns something: assessing the learning  outcomes of two educational games    jennifer mccabe  assistant director  james madison university, east campus library  harrison, va 22807, usa  email: mccabeja@jmu.edu    steven wise  vice president of research and development  northwest evaluation association  lake oswego, or 97035, usa  email: steve.wise@nwea.org    received: 28 august 2009        accepted: 2 november 2009      © 2009 mccabe and wise. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution,  and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to determine whether educational games can be designed that are both  fun and effective in improving information seeking skills.    methods – two skills that are known to be particularly difficult for students taking a  required information literacy test were identified.  these skills are the ability to  identify citations and the ability to search databases with keywords.  educational  games were designed to address these two skills.  the first game, citation tic tac toe,  placed commonly used bibliographic citations into a tick tac toe style grid.  students  were required to play the tic tac toe game and subsequently given citation  identification exercises.  the second game arranged key concepts related to search  phrases in a magnetic keyword interface.  students were observed searching  databases before and after playing the magnetic keyword game and their pre‐ and  post‐play searches were analyzed.     results – students who played the tic tac toe game improved more from pretest to  posttest than students who only took an online tutorial.  in addition, students who  played the magnetic keyword game demonstrated quicker database searching for  their topics and expressed increased satisfaction with their results.   6 mailto:mccabeja@jmu.edu mailto:steve.wise@nwea.org http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2009, 4:4  7   conclusions – games can be created which have measurable educational outcomes  and are fun.  it is important, however, to establish the educational objective prior to  beginning game design.       introduction    scholars in many disciplines are beginning to  see the value of games and investigate their  design and use as educational tools.  in 2005,  the american library association hosted  gaming in libraries, a symposium on the  relationship between games and library  services.  since then, countless other venues  have sprung up for the discussion of the value  of gaming for educational purposes.  those  interested include all types of educators, game  designers, computer programmers, marketers,  and even librarians.  educators are using  games to teach everything from financial  literacy (wpi media relations) to social  responsibility  <http://www.gamesforchange.org/> to surgical  procedures (sourin 6‐9).  games hold vast  appeal to educators and players in all age  groups.     fortunately for those interested in exploring  the use of games, profiles of people who play  online games, and the kinds of games that  appeal to different people are plentiful.  the  pew internet & american life project reported  that fifty‐three percent of american adults  over eighteen years old play games, with  twenty one percent of adults playing every  day (lenhart and jones 1‐9).   an earlier pew  report, let the games begin: gaming technology  and college students, found that in 2003 seventy  percent of college students reported playing  online games, and one hundred percent of  students surveyed had played a video game at  some point in their lives (jones 1‐14). evidence  suggests that players are likely to be educated  and dedicated to game‐play.      given the plethora of data available about  adult gamers, it seems unwise not to  investigate ways of exploiting the medium for  use in higher education.  the challenge of  designing an educational game, that retains  the essential intrinsic motivation of fun, lies in  beginning with the learning objective in mind  and ensuring that the fun objective is  included.     this project originated when two university  administrators at james madison university  (jmu), virginia, usa asked the same person  to address two different challenges.  in 2004  the dean of libraries and educational  technologies at jmu began to notice many  students engaged in what appeared to be  gaming activity on the public computers in the  library.  concurrently, an associate dean in  the college that includes the health science,  nursing, and social work programs identified  a need to help pre‐professional students  understand the complex issues related to  health literacy.  after considerable thought  and discussion the decision was made by the  dean of the library to apply for an institute  for museum and library services (imls)  national leadership grant to determine  whether a game‐like environment could meet  the unique challenges of teaching health  literacy and information literacy.     this paper describes the design and testing of  two games specifically created to impart  information literacy skills in measurable ways.   the first game, citation tic tac toe, was  designed to help undergraduate students  correctly identify a variety of bibliographic  citations.  the second game, magnetic keyword,  was designed to assist the same student  population translate phrases into keywords in  order to search bibliographic databases more  effectively.      literature review    in recent years, the literature on games in  education has grown substantially.  literature  about the relationship between games and  library services is extensive, ranging from  evidence based library and information practice 2009, 4:4  discussions on the characteristics of gamers  (martin and ewing 209‐225), the process of  designing games (markey et al. 663‐681), what  distinguishes a game from other active  learning techniques (both electronic and  otherwise), the unique challenges of collecting  games in libraries, to student satisfaction with  games (smith 1‐12), or how games can be  integrated into library programming at all  levels.  when the game creation work  described in this paper began, there was a  notable absence in the literature of models for  assessing games and evidence demonstrating  the effectiveness of games relative to  traditional methods in teaching specific library  related skills.  since that time, there has been  continued research into the efficacy of games  as educational tools.     the previously mentioned lack of evidence  does not mean that games don’t teach.   indeed, programs similar to that described by  smith, in which the primary goal was to  counteract the boredom that students felt  while in mandatory classes, often get high  student satisfaction marks.  surely students  are more likely to learn if they are engaged  and have fun in class?  but in today’s  educational atmosphere the need for evidence  and accountability is stronger than for fun and  engagement.    a recent addition to the nascent concept of  demonstrating that games are effective is the  study conducted by hickey, ingram‐goble,  and jameson on the use of a commercial  immersive online game to teach socio‐ scientific inquiry in ecological sciences.  they  describe studies wherein sixth‐grade students  used the game quest atlantis to learn specific  federally mandated scientific concepts.   moreover, they describe the assessment of the  use of quest atlantis and offer principles for  the design of assessments in virtual  environments.  the experience described is  valuable and significant since, as they state,  “failure to produce convincing evidence of  increased achievements (and models for  obtaining such evidence) is likely to constrain  research funding and broad adoption of  educational games and simulation, and  redirect educational technology resources  toward narrow test‐preparation and tutoring  applications.” (hickey, ingram‐goble, and  jameson 187‐208).    constance steinkuehler and sean duncan  conducted a more ambitious study to  determine whether the wildly popular  multiplayer game, world of warcraft, could be  shown to spontaneously engender “scientific  habits of mind”.   they analyzed the  transcripts of nearly two thousand online  discussions carried out by players of world of  warcraft to determine whether they contained  evidence of eighteen different markers of  scientific thought.  they found that the  discourse that playing the game elicited  consisted of “social knowledge construction”  eighty‐six percent of the time ( 530‐543).  they  further concluded that the “predominant  epistemological disposition exhibited in the  forum posts was ‘evaluative’ and therefore  appropriate to science” (530‐543).      these examples represent a small selection of  the ways that games are being integrated into  education and studied for impact.  it must be  noted that online games themselves are so  varied that no single model of assessing their  impact will work for all activities described as  games.  online games represent novel ways of  interacting, of being in real and virtual worlds,  and of learning.   quantitative methods of  generating scientific evidence can work in  some cases; qualitative methods are more  appropriate for others.  this point was  eloquently argued by steinkuehler, black &  clinton, who stated that “the phenomena of  study are ever widening while the range of  methods considered legitimate for their  investigation is increasingly narrowed.” (95‐ 100).   in keeping with this sentiment, this  paper describes the study of two games by  two different means.  the conclusions of each  experiment, while generally in agreement with  each other, illuminate the unique challenges of  using games to address specific learning  objectives.  most interesting, perhaps, are the  questions that the study raises regarding how  to teach the constellation of skills that  contribute to research and information  literacy.     8 evidence based library and information practice 2009, 4:4  context and aims    james madison university (jmu) is a  comprehensive university with approximately  eighteen thousand students located in the  central shenandoah valley of virginia, usa.   the university’s mission is to prepare  students to be educated and enlightened  citizens who lead productive and meaningful  lives.  as an association of college and  research libraries (acrl) best practice  institution for its information literacy  program, all undergraduate students must  complete an online information literacy  tutorial and pass an information seeking skills  test (isst) before they can register as second‐ year students.  this requirement means that  valuable data is gathered each year about  students’ skills related to the acrl  information literacy competency standards  (association of college and research  libraries). students who do not pass the isst  may choose to repeat sections of the tutorial or  meet with a librarian for extra assistance with  information seeking.  because of this  requirement, librarians have access to  extensive data as well as anecdotal  information about which specific skills hinder  students’ ability to pass the test.  librarians at  jmu have learned that two skills that students  struggle with are the ability to identify  bibliographic citations and the elements that  comprise them (acrl information literacy  competency standard 1.2), and the ability to  break a topic into keywords (acrl  information literacy competency standard  2.2.b) (association of college and research  libraries).      in 2005, a grant was received from the  institute for museum and library services to  create games that teach health literacy and  information literacy.  the grant application  stated, “the goals of this project are threefold;  to produce a series of games that help develop  health literacy and information literacy skills,  to evaluate whether traditional information  literacy skills can be acquired through game‐ like experiences, and to determine if this  approach is more appealing and successful  with our students than our current text‐based  approaches.”   staff from the jmu libraries  and educational technologies division  worked to create several games.  the first  game was a complex role‐playing game  designed to teach pre‐professional health  students about the challenges of compromised  health literacy.  the subsequent games, which  are the subject of this paper, were designed to  address students’ ability to correctly identify  bibliographic citations and to identify  keywords related to search phrases.     because there is currently no “gold standard”  for assessing the impact of educational games  on explicit learning objectives, the authors  chose to evaluate the two games in different  ways.  the first study was a quantitative study  in which players took an achievement test (the  pretest), played the game (citation tic tac toe),  and repeated the achievement test (the  posttest) to determine whether progress was  made in their ability to identify citations.   players were compared to a control group  who did not play the game.  this study design  was chosen since there are absolute correct  and incorrect answers to citation  identification.  the goal of this study was to  determine whether students who played the  citation tic tac toe game were better at  identifying bibliographic citations by type  than students who took the online tutorial.      the team designed the second game, magnetic  keyword, to address a competency that does  not lend itself to definitive correct answers.   the acrl information literacy competency  standards address the ability to construct a  search query in standard two, specifically  stating in 2.2, “the information literate  student constructs and implements effectively  designed search strategies.”  outcome 2.2b  states that this competency is shown by  “[identifying] keywords, synonyms and  related terms for the information needed.”   from a practical perspective, this skill is often  addressed in library instruction sessions by  having students break their search phrase into  its component keywords and brainstorming  lists of broader and narrower terms and  synonyms.  measuring a student’s ability to do  this, and to refine their search based on  9 evidence based library and information practice 2009, 4:4  results, is a subjective matter that is influenced  by the search topic, the student’s familiarity  with the database content and interface, and  other variables.  because this competency is  more of an art than a science, a qualitative  method was chosen to evaluate the  effectiveness of the magnetic keyword game.   students were videotaped while playing the  game, and recordings were later analyzed to  determine whether search speed and  satisfaction improved after playing.                                             citation tic tac toe was developed and studied  first.  it is a standard tic tac toe game with  bibliographic citations in each box.   players  select a square on the grid and see a  bibliographic citation with four choices as to  the type of citation.  choices include book,  journal article, a variety of websites,  government document, book chapter,  dissertation, and newspaper articles.  if the  player selects the correct type, an x is awarded  for that box.  if the player chooses incorrectly,  the game gives the box an o.  when an  incorrect choice is made, the game provides  feedback in the form of a clue as to how to  identify that kind of citation.  for example, if  the player sees a citation for a book but makes  the incorrect choice that it is a journal article,  the game generates feedback that says “this is  a book.  you can tell this quickly by locating  the city and name of the publisher.”  this  feature was designed to help the player learn  the rules of citation identification as they play  the game.  once the player correctly identifies  three citations in a row, they win the game.  if  they fail to make the correct choices in three  adjacent boxes, the game wins.  upon  completion of each game, the player can  choose to play again.  figure 1 contains a  screen shot of the feedback screen for the tic  tac toe game.                                                            fig.1 screen shot for tic tac toe game  the second game was designed to look like a  refrigerator with magnetic words affixed to it,  similar to the popular magnetic poetry® toy  sold in bookstores and toy shops. one section  of the screen has a simple search phrase  displayed, and the other section contains a  collection of random words interspersed with  words that correspond to the search phrase.      for example, the search phrase “do students  who participate in sports get higher grades?”  has the words “students”, “sports” and  “grades” mixed in with other random words  like “heart rate” “traffic” and “practice.”  in  most cases there are three correct words mixed  in among 20 distracters.  the player drags the  identified keywords below the line on the  refrigerator and clicks on submit when it is  believed the correct keywords have been  10 evidence based library and information practice 2009, 4:4  selected.  the game then generates feedback  based on whether all of the correct words,  some of the correct words, or none of the  correct words have been chosen.  two points  are received for all correct words, one for  some words, and zero points for no correct  words chosen.  to integrate this game with a  course assignment, the player can be assigned  to play until a certain score is achieved or  correctly chosen keywords are associated with  a certain number of search phrases.  figure 2  contains a screen shot for the magnetic keyword  game.      both games were designed using flash with a  microsoft access database running in the  background to manage the game data and  player data.                                           methods     the impact of the tic tac toe game was  evaluated by comparing the relative  achievement levels of students who had  played the games with a group that had not  played.  the central research question  concerned the degree to which students’  proficiency in identifying citations improved  as the result of playing the game.    seven sections of gcom 103 (a mandatory  communications class for first‐year students)  participated in the study during the spring,  2008 semester.  these students were chosen  based on their enrollment in the course, in  which the online information literacy tutorial  is required, and their instructors’ willingness  to participate in the study.  it was decided that  a true experimental design, in which each  student would be randomly assigned to either  the experimental group (played the game) or  the control group (did not play the game)  would not be appropriate for this study.  this  was because it would be instructionally  awkward for some of the students in a section  to have access to the game, while others did  not.  instead, class sections were randomly  assigned, which resulted in three sections  receiving the game, and four sections not  receiving the game.                                                       fig. 2. screenshot for magnetic keyword game  a thirty‐five item multiple‐choice test was  developed to assess student proficiency in  identifying citations.  a copy of this test is  included as appendix 1.  each item used four  response options.  all students in the study  were asked by their instructors to take the  citation test twice—both at the beginning and  the end of the semester.  thus, although the  same test was used for both the pretest and  posttest, the testing periods were separated by  nearly three months.  for comparison  purposes, students were asked to include their  student id number.  11 evidence based library and information practice 2009, 4:4  the magnetic keyword game was tested using  morae™ usability testing software along with  bailey’s usability testing environment (ute).   morae™ usability testing software is a  program that creates video and audio  recordings of users interacting with computer  applications, including the face of the user and  the software performance.  the videos can  then be viewed and tagged for later analysis.   bailey’s ute is a piece of software that can run  on top of morae™.  it can be pre‐programmed  with specific tasks for the user to complete, as  well as allotting time for the tasks and  providing a venue for immediate user input.   bailey’s ute obviates the need for a human‐ mediated usability test, eliminating the  possibility of interference from the mediator.   both programs were chosen based on their  current use in the library for website usability  testing and because of the researchers’  familiarity and satisfaction with them.    student testers were recruited for this study  via a notice on the library’s website and  compensated with a gift card to the university  bookstore.  participants were assigned the task  of searching for journal articles on a given  topic in the database academic search  premier.  after each search, the user was  asked via bailey’s ute if they were satisfied  with their results.  after completing four  unique searches, students were asked to play  the magnetic keyword game for five minutes.   they were then asked to search for articles on  four new topics in the same database, and  again asked whether they were satisfied.  by  analyzing the time that lapsed between the  first round of searching and the second round  (post game play), the researcher could  determine whether their search speed  improved after playing the magnetic keyword  game.                          results    evaluating the impact of the tic tac toe game on  student achievement    the data was analyzed to evaluate students’  pretest and posttest scores in order to  determine whether the game had a positive  impact on student achievement.  however, the  data analysis was complicated by two factors.   first, some students took only the pretest or  the posttest, but not both.  second, a number  of students did not provide their student id  numbers when they took a test, which  precluded merging the pretest and posttest  data.  our solution for addressing this  problem is described below.    table 1 shows the basic descriptive statistics  for the overall pretest and posttest data.  at  both time points, the reliability estimates  (using coefficient alpha) indicated satisfactory  reliability.  in addition, as would be expected,  test performance was found to generally  improve from pretest to posttest, as students  learned more about citation identification.    to accurately compare the experimental and  control groups, only the data from students  who took both the pretest and posttest (and  provided ids on each occasion) were analyzed  further.  the results for these students, shown  in table 2, showed several interesting  findings.  first, the 27 students who played the  game showed a strong, statistically significant  pretest‐posttest gain.  the effect size index (d)  indicated that the mean gain was nearly two‐ thirds of a standard deviation.  however, the  47 students who did not play the game also  showed significant improvement in citation                       table 1  descriptive statistics for the citations test  test  n  minimum  maximum  mean  standard deviation  reliability  pretest  149  8  31  21.31  4.97  .78  posttest  134  13  32  22.72  4.64  .77  12 evidence based library and information practice 2009, 4:4  identification, albeit to a lesser extent.  for  these students, test performance improved  significantly, with an effect size of just under  one third of a standard deviation.  hence, the  overall findings were that (a) both groups  improved in citation identification, (b) the  group playing the game improved, on  average, by a larger amount, and (c) the group  playing the game performed much higher at  posttest.                            one problematic aspect of the results shown in  table 2 is that the experimental and control  groups also differed markedly in their pretest  means.  the group that played the game had a  much higher pretest score than the group that  did not play the game.  this complicates  interpretations of the results, because the large  differences in posttest performance are to  some extent confounded by the pretest  differences.  to address this issue, an analysis  of covariance (ancova) was performed on  the posttest scores, with pretest scores used as  a covariate.  the assumptions of the ancova  model (homogeneity of variance and  homogeneity of regression) were found to be  satisfied in the analysis.  the results showed  that the adjusted posttest scores were higher  for the group that played the game (mean =  25.27) than the non‐game group (mean =  23.14).  this difference was statistically  significant (f(1,71) =  8.69, p = .004) and  corresponded to an effect size of .74 standard  deviations.  thus, even when there was  control for the differences in pretest scores, the  group that played the game substantially  outperformed the group that did not.    evaluating the impact of the magnetic keyword  game on student search speed and satisfaction    evaluation of the impact of the magnetic  keyword game was based on whether students’  speed and self‐reported satisfaction improved   after playing the game.  the test was  administered using bailey’s usability testing                                 table 2  test statistics for students who took both the pretest and posttest    pretest    posttest        played game?  mean  sd    mean  sd  t  t prob.  effect  size (d)  yes (n = 27)  23.78  4.34    26.37  3.73  4.71  < .001  0.65  no (n = 47)  21.13  4.60    22.51  4.34  2.75  .008  0.31  environment (ute) to display the questions  and record the answers, and morae™ was  used to record the search sessions.  a  convenience sample of five undergraduate  students completed the test protocol.  the  protocol for the test was as follows:   1. ask the user to search in academic  search complete for articles on the  topic “does eating cruciferous  vegetables prevent cancer in women?”   ask whether the user is satisfied with  the results.  2. ask the user to search in academic  search complete for articles on the  topic “what kind of pollution occurs  in water sources near paper mills?”   ask whether the user is satisfied with  the results.  3. ask the user to search in academic  search complete for articles on the  topic “what is the average wage for a  construction worker in the united  states?”  ask whether the user is  satisfied with the results.  4. ask the user to search in academic  search complete for articles on the  13 evidence based library and information practice 2009, 4:4  topic “is child abuse more likely to  occur in families living below the  poverty line?”  ask whether the user  is satisfied with the results.  5. ask the user to play the magnetic  keyword game to identify the  keywords in at least 4 search phrases.  6. ask the user to search academic  search complete for the topic “does  recreational reading improve grades?”   ask whether the user is satisfied with  the results.    7. ask the user to search academic  search complete for the topic “is there  racism in american colleges?”  ask  whether the user is satisfied with the  results.  8. ask the user to search academic  search complete for the topic “how  prevalent is post‐traumatic stress  among iraq war veterans?”  ask  whether the user is satisfied with the  results.  9. ask the user to search academic  search complete for the topic “what  diseases are affected by volcanic  eruptions?”  ask whether the user is  satisfied with the results.  10. ask whether the user would describe  the game as fun (on a scale of 1‐10  with 1 being the least fun and 10 being  the most fun.                  11. ask whether the user would  recommend the game to a friend (yes  or no).  12. ask what the user would do to  improve the game (fill in the blank).  the data was analyzed by viewing the video  recordings of the students working through  the test protocol and recording the amount of  time between initiating the searches and  indicating satisfaction with results.  the time  each student spent searching in the database  decreased in each player after they played the  magnetic keyword game.  for the first search,  the average time it took for students to register  satisfaction with their results was 184 seconds.   for the final search, the average time between  searching and satisfaction was 55 seconds.   figure 3 below represents the average search  time for the eight search tasks described  above.    discussion    each experiment described above included  several survey questions to determine whether  the students who participated found the  activity to be fun.  because fun is an intrinsic  motivator, a game that can simultaneously  educate and entertain is a powerful tool.   students who played the citation tic tac toe  game described it as “a nifty program” and “a  fun way to practice the information.”  several  student evaluators offered suggestions on how  to make the game more useful, including  increasing the number and types of citations                               14                         fig. 3. average search time per task  evidence based library and information practice 2009, 4:4  and creating a pre‐play tutorial.  comments  indicated that students enjoyed the game as it  helped them achieve the goal of correctly  identifying citations, a skill they realize they  must master.  because of the simplicity of the  interface and gameplay, it is unlikely that any  student would play it solely for entertainment.      comments on the magnetic keyword game  indicated that students found it to be  moderately fun, with three players answering  the question related to recommending the  game to a friend affirmatively and two  negatively.  in keeping with the sentiments  expressed for the tic tac toe game, one player  wrote“…i think the game has been made as  fun as possible.”  several players indicated  that they would prefer more challenging  search phrases.  at the conclusion of one  recording, a student is overheard saying, “is  that all? i want to search more which was  interpreted as an endorsement of the “replay  value” of the game.  as a usability test, the  results indicated that the game was highly  usable, with no student experiencing  navigation or game playing problems.  this  game could be further studied by integrating  it into an advanced research and writing  assignment and determining whether it aids  students in their database searching skills in a  “real‐world” situation.  the structure of the  game is such that additional search phrase and  distracter data can be added to increase the  replay value or to create a subject specific  game.    it should be noted that the magnetic keyword  game is limited because most of the distracter  words were randomly chosen and bear no  relation to the correct keywords.  this made it  easier than necessary for players to make the  correct choices.  one way to make the game  more challenging while reiterating the  keyword concept would be to include more  word variations instead of random unrelated  words as distracters.  for example, in the  search where the phrase is “do students who  participate in sports get higher grades?” the  distracters could include words like grading,  graduate, graded, study, etc.  carefully chosen  distracters could demonstrate the value of  truncating keywords.  an additional limitation  of the research design is the fact that students  likely improved their familiarity and skill with  the academic search premier database during  the course of the test.  a more thorough test  could use several different databases with  different interfaces for comparison.    the experience of creating and studying two  different kinds of games demonstrates that  games can prove effective for learning certain  kinds of skills.  reflection on the processes  involved reveals that the most interesting  learning occurs when the game in use is an  analogy for a skill, as opposed to simply using  the game to deliver new content.   future  investigation into the use and study of games  as analogies for specific skills, such as that  conducted by steinkuehler and duncan (530‐ 543), would reveal the pedagogical value of  commercial games, obviating the need for the  intense design‐assessment cycle in which  educational game designers must currently  engage.  because commercial games are a for‐ profit business, the resources and talent they  attract is unmatched in academia.  the future  study of games in higher education should  concentrate on exploring how commercial  games teach and how best to integrate them  into our curricula, rather than on designing  new games.         conclusions    this research found that it is possible to  design games that are both fun and effective in  imparting educational skills.  a game‐like  approach is one that integrates graduated  levels of expertise, new powers, and rewards  with subject‐specific content.  the games  described here integrated game‐like  approaches with identification of  bibliographic citations and isolation of  keywords from search phrases.  games that  attempt to teach new skills are easier to design  and measure than games that attempt to  cultivate new behaviors. reflection on the time  spent developing and analyzing the games  compared to studies of the novel uses of  existing games leads the authors to conclude  15 evidence based library and information practice 2009, 4:4  that the latter approach is more efficient and  richer with research possibilities.      in considering how to integrate games into  education, it is wise to heed the adage that  “everything looks like a nail when all you  have is a hammer.”  some situations lend  themselves well to a game approach, while  others may not.  sara defrietas states that  “…games and simulations can effectively  support learners with skills‐based needs (such  as literacy and numeracy)” ( 249‐264) an  assertion that is confirmed in both the creation  and the study of the tic tac toe game.   citation identification is a discrete skill, one  which can be learned effectively by using the  game.      as noted earlier, constructing effective search  strategies is more an art than a science.   to be  a good searcher one must be familiar with a  variety of databases and their unique thesauri,  the lexicon of the search topic, the conventions  of research of the field, and even the places  where the relevant scholarship is being  conducted.  scholarly literature published in  languages other than english presents unique  challenges to english‐only speakers.   moreover, there is a certain element of  creativity involved in searching, as a dose of  curiosity and bravery are required to  experiment with word combinations.  all of  these are characteristics that develop with  time, experience, failure, and maturity.  to  assume that the ability to be an artful searcher  can be deconstructed in the same way as the  identification of citations can is naïve and  potentially offensive to the scholars who  produce the work being searched.      in her introduction to the world of warcraft  study, constance steinkuehler astutely  referred to henri poincare’s quote, “science is  built up of facts, as a house is built of stones;  but an accumulation of facts is no more  science than a heap of stones is a house.”  (steinkuehler and duncan 530‐543).  the main  challenge librarians face as they attempt to  impart the skills that students need to be  competent researchers lies in guiding them to  make meaning of the information they find.   in approaching the myriad of challenges of  teaching research in an ever more complex  environment, wise librarians will differentiate  between what students must know absolutely  and what skills must be developed  incrementally, and choose approaches  accordingly.  successful games can be  designed that include the intrinsic motivator  of fun, but care should be taken to avoid the  implication that all learning must be fun.           works cited   association of college and research libraries.  information literacy competency  standards for higher education.  chicago: american library association,  2000.   de freitas, sara, and martin oliver. ʺhow can  exploratory learning with games and  simulations within the curriculum be  most effectively evaluated?ʺ computers  & education 46.3 (2006): 249‐64.   hickey, daniel t., adam ingram‐goble, and  ellen m. jameson. ʺdesigning  assessments and assessing designs in  virtual educational environments.ʺ  journal of science education and  technology 18.2 (2009): 187‐208.   jones, steven.  ʺlet the games begin: gaming  technology and entertainment among  college students.ʺ  pew internet and  american life project. (2003):1‐14. 1 dec.  2009  <http://www.pewinternet.org/reports/20 03/let‐the‐games‐begin‐gaming‐ technology‐and‐college‐students.aspx>  lenhart, amanda, sydney jones, and  alexandra rankin macgill. ʺpew internet  project data memo re: adults and video  games.ʺ pew internet and american life  project. (2008):1‐9.1 dec. 2009   <http://www.pewinternet.org/reports/20 08/adults‐and‐video‐games.aspx>  16 evidence based library and information practice 2009, 4:4  17 markey, karen, et al. ʺdesigning and testing a  web‐based board game for teaching  information literacy skills and  concepts.ʺ library hi tech 26.4 (2008):  663‐81.   martin, justine, and robin ewing. ʺpower up!  using digital gaming techniques to  enhance library instruction.ʺ internet  reference services quarterly 13.2 (2008):  209‐25.   smith, felicia a. ʺgames for teaching  information literacy skills.ʺ library  philosophy & practice 9.2 (2007): 1‐12.   sourin, a. ʺvirtual orthopedic surgery  training.ʺ ieee computer graphics and  animations.may/june 20.3 (2000): 6‐9.   steinkuehler, constance a, rebecca w. black,  and katherine a. clinton. ʺresearching  literacy as tool, place, and way of  being.ʺ reading research quarterly 40.1  (2005): 95‐100.   steinkuehler, constance, and sean duncan.  ʺscientific habits of mind in virtual  worlds.ʺ journal of science education &  technology 17.6 (2008): 530‐43.   wpi media relations. ʺwpi students create  online game to simulate massachusetts  state budget process.ʺ news release may  24, 2003. 1 dec.2009  <http://www.wpi.edu/news/20023/massb alance.html>                            evidence based library and information practice 2009, 4:4  appendix  citation tic tac toe pre/post‐test  citation test    identify the type of publication for each citation below.  mark the correct answers on a form  provided.    do not write your name on this sheet or the card, please use your peoplesoft number  only on the form provided.    1. fraser, k. (ed.). (2005). education development and leadership in higher education : developing an  effective institutional strategy. london ; new york: routledgefalmer.    a‐‐book    b‐‐ journal article    c‐‐ magazine article  d‐‐chapter or article in a book    2. king, jacqueline.  “academic success and financial decisions: helping students make crucial  choices.”  engaging the first‐year student. ed. robert feldman, mahwah, nj: lawrence  erlbaum, 2005.  pp. 3‐25.  a‐‐newspaper article  b‐‐book  c‐‐government publication  d‐‐chapter or article in a book    3. flea beetles. (2006, april). horticulture, 103, 16.    a‐‐government publication    b‐‐book    c‐‐magazine article  d‐‐chapter or article in a book    4. united states. presidentʹs committee on employment of the handicapped. (1980). the college  student with a disability: a faculty handbook. washington, d.c.: u.s. g.p.o.    a‐‐government publication    b‐‐magazine article    c‐‐journal article  d‐‐chapter or article in a book    5. ghaderi, a. (2006). attrition and outcome in self‐help treatment for bulimia nervosa and  binge eating disorder: a constructive replication. eating behaviors, 7(4), 300‐308.      a‐‐magazine article    b‐‐journal article    c‐‐book  d‐‐chapter or article in a book    6. hartley, j.t., harker, j.o., & walsh, d.a. (1980). contemporary issues and new directions in  adult development of learning and memory. in l.w. poon (ed.), aging in the 1980’s:  psychological issues (pp. 239‐252).washington, dc : american psychological association.  a‐‐book  b‐‐journal article  c‐‐newspaper article  18 evidence based library and information practice 2009, 4:4  d‐‐chapter or article in a book    7. bremner, m., aduddell, k., bennett, d., & vangeest, j. (2006). the use of human patient  simulators: best practices with novice nursing students. nurse educator, 31(4), 170‐174.    a‐‐journal article    b‐‐book    c‐‐magazine article  d‐‐chapter or article in a book    8. united states. congress. house. committee on ways and means. (1992). adequacy of the  administrationʹs fiscal year 1993 budget proposals for the internal revenue service.  washington: u.s. g.p.o;     a‐‐magazine article    b‐‐government publication    c‐‐journal article  d‐‐chapter or article in a book    9. girden, e. r. (1996). evaluating research articles from start to finish. thousand oaks, calif.: sage  publications.    a‐‐journal article    b‐‐magazine article    c‐‐book  d‐‐chapter or article in a book    10. turman, l.u., self, p. c., calacaro, p.v. (2004). teaching a web‐based course in health  informatics. reference services review, 32(1), 21‐25.    a‐‐government publication    b‐‐book    c‐‐journal article  d‐‐chapter or article in a book    11. mayouf, j.s. (1998, december 5). crisis in sub‐prime mortgage market worries investors.  philadelphia enquirer, pp. a31, a14.    a‐‐journal article    b‐‐newspaper article    c‐‐magazine article  d‐‐chapter or article in a book    12. condon, john c. good neighbors. 2nd ed. yarmouth, me.: intercultural press, 1997.    a‐‐journal article     b‐‐magazine article    c‐‐book  d‐‐chapter or article in a book    13. zhang, g., & lu, j. (2003). an integrated group decision‐making method dealing with fuzzy  preferences for alternatives and individual judgments for selection criteria. group decision &  negotiation, 12(6), 501‐515.    a‐‐journal article    b‐‐book    c‐‐newspaper article  d‐‐chapter or article in a book  19 evidence based library and information practice 2009, 4:4    14. block, robert. ʺkenya, tanzania offer fertile soil for intrigue,ʺ wall street journal, august  11, 1998, a12.  a‐‐magazine or journal article  b‐‐book  c‐‐government publication  d‐‐newspaper article    15. goulding, anne. ʺjoking, being aggressive and shutting people up: the use of focus groups  in lis reseach.ʺ education for information 15.4 (1997): 331. .    a‐‐book    b‐‐journal article    c‐‐government publication  d‐‐chapter or article in a book    16. cameron, theresa. 2002. foster care odyssey : a black girlʹs story. jackson: university press of  mississippi.    a‐‐book    b‐‐magazine article    c‐‐journal article  d‐‐chapter or article in a book    17. shapiro, s.c.(1979). the sneps semantic network processing system. in findler, n.v., editor,  associative networks: representation and use of knowledge by computers, pages 179‐203.  academic press, new york  a‐‐magazine or journal article  b‐‐newspaper article  c‐‐book  d‐‐chapter or article in a book    18.  anderson, deborah, martha finne, and jeanette swofford. 1986. jasonʹs story : going to a foster  home. minneapolis, minn: dillon press.    a‐‐journal article    b‐‐magazine article    c‐‐book  d‐‐chapter or article in a book    19. adamha news on alcohol, drug abuse, and mental health. rockville, md; washington,  d.c.: u.s. dept. of health and human services, national institute on alcohol abuse and  alcoholism; supt. of docs., u.s. g.p.o., distributor.    a‐‐magazine article    b‐‐government publication    c‐‐book  d‐‐chapter or article in a book    20. kant, immanuel. “what is enlightenment?” trials of modernity: europe and the modern  world. eds. stacy burton and dennis dworkin. 3rd ed. boston: pearson, 2001. 139‐43.  a‐‐newspaper article  b‐‐book  c‐‐government publication  d‐‐chapter or article in a book  20 evidence based library and information practice 2009, 4:4    21. walker, j. t., lofton, s. p., haynie, l., & martin, t. (2006). the home health nurseʹs role in  geriatric assessment of three dimensions: depression, delirium, and dementia. home  healthcare nurse, 24(9), 572‐580.    a‐‐journal article    b‐‐book    c‐‐magazine article  d‐‐chapter or article in a book    22. “french find.” choir & organ, may/june 2006, 3.    a‐‐magazine article    b‐‐government publication    c‐‐book  d‐‐chapter or article in a book    23. allen, j. f., hendler, j., and tate, a., editors (1990a). readings in planning. morgan kaufmann,  san mateo, california.  a‐‐magazine or journal article  b‐‐book  c‐‐government publication  d. chapter or article in a book    24. suellentrop, c. ʺ playing with our heads:  why video games are making our kids smarter  and more obedient.” utne reader 139 january‐february (2007): 78‐83.    a‐‐book    b‐‐magazine article    c‐‐newspaper article  d‐‐chapter or article in a book    25. lubin, j.s. (1980, december 5). on idle: the unemployed shun much mundane work, at least  for a while. san francisco chronicle, pp. a1, a25.    a‐‐journal article    b‐‐newspaper article    c‐‐book  d‐‐chapter or article in a book    26. united states, general accounting office, money laundering: regulatory oversight of offshore  private banking activities: report to the chairman, washington, d. c.: the office, 1998.  a‐‐newspaper article  b‐‐book  c‐‐government publication  d‐‐chapter or article in a book    27. yan, xuetong. ʺterrorism remains a significant threat to mankind,ʺ china daily, [beijing],  18, no. 5351, september 23, 1998, 4.  b‐‐essay or book chapter  c‐‐book  d‐‐government publication  e‐‐newspaper article      21 evidence based library and information practice 2009, 4:4  web site evaluation questions    28. embassy of brazil. (2005). brazilian embassy in washington. retrieved march 19, 2006, from  http://www.brasilemb.org  a‐‐commercial web site  b‐‐organizational web site  c‐‐academic web site  d‐‐government web site    29. spearing, melissa k. “schizophrenia.” national institute of mental health. 1999. national  institute of mental health. 13 april 2005 <http://www.nimh.nih.gov/publicat/schizoph.cfm>.  a‐‐commercial web site  b‐‐organizational web site  c‐‐academic web site  d‐‐government web site    30. the andy warhol foundation. the warhol foundation: 1997. online:  http://www.warholfoundation.org/  a‐‐government web site  b‐‐commercial web site  c‐‐educational web site  d‐‐organizational web site    31. “generalized anxiety disorder.” medicinenet.com.  2005.  medicinenet.  21 september 2007   http://www.medicinenet.com/anxiety/article.htm.  a‐‐government web site  b‐‐commercial web site  c‐‐educational web site  d‐‐organizational web site    32. hanson, d. j.  (2007).  binge drinking.  alcohol problems and solutions  retrieved september 19,  2007, from http://www2.potsdam.edu/hansondj/bingedrinking.html  a‐‐government web site  b‐‐commercial web site  c‐‐educational web site  d‐‐organizational web site   22 http://www.brasilemb.org/ http://www.medicinenet.com/anxiety/article.htm evidence based library and information practice 2009, 4:4  23 acknowledgements    this work was supported in part by a national leadership grant from the institute for museum and  library services (grant # lg06‐05‐0150‐05).  in the digital services department at james madison  university greg brown (browngh@jmu.edu) contributed substantial effort to the design and  programming of citation tic tac toe and harper holsinger (holsinhd@jmu.edu) contributed  significantly to the design and programming of the magnetic keyword game.  both games described  in this paper can be viewed and played at http://www.lib.jmu.edu/games/ .     mailto:browngh@jmu.edu mailto:holsinhd@jmu.edu http://www.lib.jmu.edu/games/ evidence based library and information practice evidence based library and information practice 2010, 5.1  evidence based library and information practice       news/announcements    researcher‐librarian partnership program seeks applicants       2010. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which  permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly  attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the  same or similar license to this one.    would you like to learn how to gather the  evidence to make your services and resources  even more effective in meeting your clients’  needs? have you always wanted to learn the  steps involved in conducting a research  project?     the researcher‐librarian partnership  provides the opportunity for new library and  information science professionals to develop  and refine their skill, knowledge and  confidence in conducting research.  the partnership is sponsored by the library  theory and research section of the  international federation of library  associations and institutions (ifla).     research skill and knowledge are the essential  tool kit for ensuring that libraries and  librarians continue to effectively and  efficiently meet the evolving needs of the  clients and communities they serve.  the partnership is open to practising  librarians who have no more than seven years  professional experience in the library and  information sector at the time of application.  the partnership is not open to individuals  occupying research or teaching positions.     the purpose of the partnership is to provide  the opportunity for recipients to receive expert  coaching and guidance as they undertake a  research project on a selected library or  information issue.     up to six recipients will be selected to take  part in the partnership in 2010. the successful  recipients will be matched with a mentor who  has experience in conducting research.     nominations will be considered by a selection  committee and a recommendation made to  the library theory and research standing  committee.     nominations close on 3 may 2010.     for further information (including the  application form), visit  http://www.ifla.org/en/library‐theory‐and‐ research/projects or contact helen partridge,  chair of the selection committee, at  h.partridge@qut.edu.au.    160 http://www.ifla.org/en/library-theory-and-research/projects http://www.ifla.org/en/library-theory-and-research/projects mailto:h.partridge@qut.edu.au evidence based library and information practice evidence based library and information practice 2010, 5.1  evidence based library and information practice       evidence summary    quality of online chat reference answers differ between local and consortium library  staff: providing consortium staff with more local information can mitigate these  differences    a review of:  meert, d.l., & given, l.m. (2009). measuring quality in chat reference consortia: a comparative  analysis of responses to users’ queries.” college & research libraries, 70(1), 71‐84.    reviewed by:    laura newton miller  science & engineering librarian,   carleton university   ottawa, ontario, canada  email: laura_newtonmiller@carleton.ca     received: 17 nov. 2009          accepted: 24 jan. 2010       2010 newton miller. this is an open access article distributed under the terms of the creative commons‐ attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐ sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original  work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is  redistributed under the same or similar license to this one.    abstract    objective – to evaluate the quality of answers  from a 24/7 online chat reference service by  comparing the responses given by local and  consortia library staff using in‐house reference  standards, and by assessing whether or not the  questions were answered in real time.     design – comparative analysis of online chat  reference transcripts.    setting – large academic library in alberta,  canada.    subjects –  a total of online chat reference  transcripts from the first year of consortium  service were analyzed for this study. of these,  252 were answered by local library staff and  226 from consortia (non‐local) library staff.    methods – a stratified random sample of  1,402 transcripts were collected from the first  year of consortium service (beginning of  october to end of april). this method was  then applied monthly, resulting in a sample  size of 478 transcripts. in the first part of the  study, responses were coded within the  transcripts with a “yes” or “no” label to  determine if they met the standards set by the  local university library’s reference  management. reference transaction standards  included questions regarding whether or not  correct information or instructions were given  and if not, whether the user was referred to an  authoritative source for the correct  132 mailto:laura_newtonmiller@carleton.ca evidence based library and information practice 2010, 5.1  information. the second part of the study  coded transcripts with a “yes” or “no”  designation as to whether the user received an  answer from the staff member in “real time”  and if not, was further analyzed to determine  why the user did not receive a real‐time  response. each transcript was coded as  reflecting one of four “question categories”  that included library user information, request  for instruction, request for academic  information, and miscellaneous/non‐library  questions.      main results – when all question types were  integrated, analysis revealed that local library  staff met reference transaction standards 94%  of the time. consortia staff met these same  standards 82% of the time. the groups showed  the most significant differences when  separated into the question categories. local  library staff met the standards for “library  user information” questions 97% of the time,  while consortia staff met the standards only  76% of the time. “request for instruction”  questions were answered with 97% success by  local library staff and with 84% success by  consortia. local library staff met the “request  for academic information” standards 90% of  the time while consortia staff met these  standards 87% of the time.  for “miscellaneous  non‐library information” questions, 93% of  local and 83% of consortia staff met the  reference transaction standards. for the  second part of the study, 89% of local library  staff answered the questions in real time, as  opposed to only 69% of non‐local staff. the  three most common reasons for not answering  in real time (known as deferment categories)  included not knowing the answer (48% local;  40% consortia), technical difficulty (26% local;  16% consortia), and information not being  available (15% local; 31% consortia).    conclusion – the results of this research  reveal that there are differences in the quality  of answers between local and non‐local staff  when taking part in an online chat reference  consortium, although these discrepancies vary  depending on the type of question. providing  non‐local librarians with the information they  need to answer questions accurately and in  real time can mitigate these differences.    commentary     online reference chat has become an  increasingly popular way to meet patrons’  reference needs. as patron expectations of  service grow, libraries are exploring ways to  save time and money, using existing resources  to benefit the most people. the decision to join  a consortium chat reference service can be a  difficult one, because many librarians are  concerned about the ability of outside staff to  answer their users’ questions successfully. but  what is a “successful” transaction? the  researchers discuss the challenges regarding  “success” and “quality” in terms of library  versus user perspectives, and in virtual versus  traditional reference desk transactions.      meert and given have developed a new  measure for assessing the quality of a chat  reference transaction by determining whether  or not correct information was provided in  real time and if not, whether patrons were  provided with an authoritative source for the  correct information.  what is “correct” may be  up to interpretation, as sometimes there are  several ways to answer a reference question  (especially regarding category three,  “request for academic information”). it is  easier to measure “correctness” in the other,  more factual categories, although  interestingly, this was the category where both  local and non‐local library staff appear to be  almost equally competent in answering  questions.      the researchers discuss some of the issues and  concerns in the literature regarding the in‐ person reference desk transaction and the  online chat interaction. in this study, librarians  in both groups did not answer some of the  questions in real time because they did not  know the answer. this could be characteristic  of certain questions in general. with this in  mind, meert and given point out that it would  be interesting to examine whether or not there  are differences in answering questions in real  133 evidence based library and information practice 2010, 5.1  134 time on the physical reference desk compared  to in an online chat service.    the researchers suggest that more local user  information provided to non‐local staff may  allow them to answer these types of questions  more easily and in real time. since the time  that the research was conducted, an  information page was created to offer non‐ local library staff information about facts,  policies and procedures that were not being  answered correctly or in real time. this need  to be more familiar with, or have access to,  information about library procedures, policies  and records raises some interesting concerns  for a local library. for example, some libraries  may have only internal access to certain  policies and procedures (i.e., on a staff  intranet). it may not be an issue for some  kinds of information to be shared, but how  much personal information should an external  librarian know about a local user? privacy and  legal concerns would require that during  times when consortium librarians are staffing  the chat reference service, they would only be  able to guide users to personal information,  not have access to it themselves. for example,  a consortium librarian could provide steps to  reset a personal identification number (pin)  but not be able to give out the user’s actual  pin.      great care was taken in the selection of  transcripts and preparation of data to easily  repeat the study in another context. it would  be useful to repeat this study with the same  library and consortium to ensure that the  measures put in place since the study  (providing more local information) actually  help the non‐local staff meet the reference  standards. it would also be valuable to repeat  this study with another local library and  consortium arrangement to validate the  findings. academic libraries can incorporate  this research into their own decision‐making  processes when analyzing the pros and cons of  joining an online consortium, and any library  that uses online chat to provide reference will  find the standards set up by this particular  library’s management beneficial in  maintaining the quality of chat transactions.   microsoft word es_hannigan.doc evidence based library and information practice 2006, 1:4      54 evidence based library and information practice       evidence summary    identifying controlled clinical trials for systematic reviews requires searching  multiple resources – and, even then, comprehensiveness is questionable    a review of:  crumley, ellen t., natasha wiebe, kristie cramer, terry p. klassen, and lisa hartling. “which  resources should be used to identify rct/ccts for systematic reviews: a systematic  review.” bmc medical research methodology 5.24 (2005). 4 nov. 2006  <http://www.biomedcentral.com/1471‐2288/5/24>.      reviewed by:  gale g. hannigan  professor, texas a&m medical sciences library  college station, texas, united states of america  e‐mail: g‐hannigan@tamu.edu      received: 27 august 2006            accepted: 23 october 2006      © 2006 hannigan. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to determine the value of  searching different databases to identify  relevant controlled clinical trial (cct) and  randomized controlled trial (rct) reports  for systematic reviews.    design – systematic review.    methods – seven electronic databases  medline, embase, cinahl, eric,  psycinfo, web of science, and cochrane  library were searched to april 2004. four  journals, health information & libraries  journal – (formerly health libraries review),  hypothesis, journal of the medical library  association– (formerly bulletin of the medical  library association), and medical reference  services quarterly were handsearched from  1990 to 2004. all abstracts of the cochrane  colloquia (1993‐2003) were handsearched,  and key authors found from any portion of  the searching process were contacted and  relevant article references screened. two  reviewers independently screened results  for studies that compared two or more  resources to find rcts or ccts using  defined inclusion and exclusion criteria.  two reviewers assessed studies for quality  using four criteria: adequate descriptions of  what the search was attempting to identify,  the methods used to search, the reference  standard, and evidence that bias was  http://www.biomedcentral.com/1471%e2%80%902288/5/24%00%00 http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:4      55 avoided in selection of relevant studies.  screening and assessment differences  between reviewers were resolved through  discussion. using a standard form, one  investigator extracted data for each study,  including study design and results (e.g.,  recall, precision), and a second investigator  checked these data. authors were contacted  to provide missing data. results were  grouped by the compared resources and  these comparisons were summarized using  medians and ranges. using a classification  modified from hopewell et al., search  strategies were categorized as complex  (using a combination of types of search  terms), cochrane (the cochrane highly  sensitive search strategy or hsss), simple  (using five or fewer search terms which may  include a combination of mesh, publication  type, and keywords), and index (using one  or two terms to check/verify if the study is  in the database) (2004).    main results – sixty‐four studies met  criteria for inclusion in the analysis. four  major comparisons were: medline vs.  handsearch (n=22), medline vs. medline  + handsearch (n=12), medline vs. other  reference standard (n=18), and embase vs.  reference standard (n=13). thirteen other  comparisons had only one or two studies  each. the most common comparison was  medline vs. handsearching. data  analyzed from 23 studies and 22 unique  topic comparisons showed a 58% median for  search recall (range 7‐97%). data for search  precision based on 12 studies and 11 unique  topic comparisons indicated a median of  31% (range 0.03‐78%). data based on more  than four comparisons, shows no median  recall more than 75% (range 18‐90%) and no  median precision more than 40% (range 13‐ 83%). recall was higher for trial registries  vs. reference standard (89%, range 84‐95%)  but these numbers were based on two  studies and four comparisons; one study  with two comparisons measured precision  at a range of 96‐97% for trial registries vs.  reference standard. subgroup analyses  indicate that complex and cochrane  searches each achieve better recall and  precision compared to simple searches.  forty‐two studies reported reasons why  searches miss relevant studies. the reason  cited most often for electronic databases was  inadequate or inappropriate indexing.     conclusion – the results of this systematic  review indicate that no single resource  results in particularly high recall or  precision when searchers look for rcts and  ccts.     commentary     well‐performed, up‐to‐date systematic  reviews of methodologically sound rcts  and ccts currently represent the highest  level of evidence. therefore, it is critical that  they are based on all appropriate and  relevant rcts and ccts. simply put, the  question asked by the researchers is “which  resources and types of searches are most  productive for locating clinical trials?” if  their research answered this question, it  might suggest an optimal, cost‐effective  approach in which systematic reviewers  could be confident. to address the question,  the authors analyze studies of the recall and  precision of searches of the most common  resources used to identify rcts and ccts,  following the standard methodology of  systematic reviews (e.g., they define their  own search methods, inclusion/exclusion  criteria, employ more than one person to  evaluate studies, and describe their  methodology in detail, to the point of  providing a link for more information about  the studies). as an aside, it is always helpful  to include a figure that graphically describes  the study process. unfortunately, despite  the rigor of this systematic review, the  quality of the studies included varies and  the numbers of truly comparable studies are  quite small. these are inherent limitations to  any conclusions the authors make. that said,  evidence based library and information practice 2006, 1:4      56 if true, the reported results make one  question the comprehensiveness of any  systematic review literature search. overall,  medians for recall and precision are low and  ranges are wide. in their conclusion, the  authors recommend consultation with a  librarian even though only one study  provides direct comparison between  librarian and non‐librarian authored  searches. further, subgroup analyses do not  consistently show that searches by librarians  are better. this is the only hint of bias  creeping into an otherwise well‐executed  study, and a reminder to us all not to take  an article’s abstract at face value. this  research underscores the need to find a  better way of identifying rcts and ccts   and it suggests that registries and/or better  indexing hold promise of making that  possible. for the time being, though,  systematic reviewers cannot rely on one  resource in their effort to be complete and,  until there are comprehensive clinical trial  registries, even multiple‐resource searching  is most likely not all‐inclusive.                                                works cited    hopewell s, m. clarke, c. lefebvre, r.  scherer. “handsearching versus  electronic searching to identify reports  of randomized trials.” cochrane  database of methodology reviews  2002.4 (2002). 5 nov. 2006   <http://www.mrw.interscience.wiley.co m/cochrane/clmethrev/articles/mr0000 01/frame.html>.  http://www.mrw.interscience.wiley.co cultivating our practice: a reflection on library synthesis review services in the context of patient-oriented research commentary   cultivating our practice: a reflection on library synthesis review services in the context of patient-oriented research    catherine boden associate librarian leslie and irene dube library university of saskatchewan saskatoon, saskatchewan, canada email: catherine.boden@usask.ca   angie gerrard associate librarian murray library university of saskatchewan saskatoon, saskatchewan, canada email: angie.gerrard@usask.ca   received: 12 july 2021                                                              accepted: 26 oct. 2021      2021 boden and gerrard. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29997     academic libraries offering synthesis review (sr) services must address researchers’ needs amidst an ever-changing environment. srs refer to various methodological review types which adhere to rigorous methods (e.g., systematic reviews, scoping reviews, realist reviews, and rapid reviews). since the early 2000s, researchers have noted increasing numbers of systematic reviews being published (bastian et al., 2010; ioannidis, 2016; page et al., 2016), and a proliferation of sr types (e.g., kastner et al., 2016; sutton et al., 2019; tricco et al., 2016). for each review type, the methods for finding literature share similarities but also nuanced differences (sutton et al., 2019). furthermore, health sciences research trends impact review team composition, such as patient engagement in the research endeavour.   patient oriented research (por) places individuals with lived experience at the centre of the research pursuit with the goal of improving the relevance and applicability of health care research (canadian institutes of health research, 2019a). the canadian institutes of health research (2019b) defines “patient” in this context as an overarching term that includes individuals with personal experience of a health issue, and includes patients and informal caregivers, such as family and friends. patient partners, and indeed other stakeholders (e.g., slattery et al., 2020), engage in srs, as evident in collaborations like the cochrane collaboration (2021).    patient partners can be involved at any or all stages of a review. examples of patient partner contributions include: research agenda setting (e.g., identifying a need for reviews in particular research areas), defining the research question, identifying patient-oriented outcomes, planning the methods, writing protocols, and so forth. patient partners can also be full collaborators on the review (pollock et al., 2018). as sr practices evolve, there may be downstream consequences to library services. in this commentary, we explore implications for librarians who may now be engaged in supporting patient-oriented reviews and potential consequences to library sr service models based on our experience with a patient-oriented rapid review.   to understand how libraries can better support por through their sr services, we took a two-pronged approach: (1) we initially conducted a rapid review of the perspectives of patient partners engaged in srs, and (2) we then reflected on the experience of engaging patient partners in the rapid review process. this meant that we took an evidence-based approach to enhancing our sr supports in a meta manner. in other words, the patient partners were in a sense researching (meaningful engagement patent partners in a review) what they were experiencing (collaborating on a review as patient partners). after completing the review, we undertook a reflective process guided by rolfe’s (2010) framework for reflexive practice to examine our experience collaborating on this patient-oriented rapid review. rolfe and jasper (2010) suggest “that critical reflection can be regarded not only as a form of research, as a way for practitioners to conduct a critical inquiry into their own practice, but also as a reflexive integration of research and practice into a single act” (pp. 7-8). thus, we employed reflection to enhance both our understanding of the research methods and the implications to practice.   rolfe’s (2010) framework was developed for practitioners to improve professional practice. it provided a straightforward framework for reflecting and reporting on our own experience. this framework, an expansion of borton’s earlier work (1970), asks three main questions: what?, so what?, and now what?. the first stage in this framework, the what, is the description level of reflection and includes reflection cues such as “what is the problem or difficulty,” “what was i trying to achieve,” and “what was good/bad about the experience” (p. 45). the next stage is the theoryand knowledge-building level of reflection. in this stage, practitioners are encouraged to reflect on so what prompts such as “so what does this tell me or teach me,” “so what is my new understanding of the situation,” and “so what broader issues arise from the situation” (p. 45). the third stage is the action-oriented (reflexive) level of reflection focusing on the now what and includes cues such as “now what do i need to do in order to improve things,” “now what broader issues need to be considered if this action is to be successful,” and “now what might be the consequences of this action” (p. 45). while this model is sequential, it is also cyclical where in this final stage one returns to the initial level to form a reflexive cycle (pp. 45-46).   background: patient oriented research rapid review project   a collaboration between one author (cb) of this commentary from the university of saskatchewan library and colleagues at the saskatchewan centre for patient-oriented research (scpor) has focused on research capacity development activities for srs in the province of saskatchewan. products of this collaboration include webinar series and in-person workshops. a self-directed learning sr toolkit was also developed (university of saskatchewan, 2021). the addition of guidance on patient-oriented reviews is the second phase of the toolkit development, which was an impetus for the project upon which we are reflecting in this paper. we decided to employ an evidence-based approach to enhancing the sr toolkit by conducting the rapid review. this approach simultaneously enabled us to gain an understanding of the literature on patient engagement in reviews, have hands-on experience doing a patient-oriented review, and answer a specific research question on patient-oriented review methods.     given the topic, it was natural that the review team should include patient partners. the review team was recruited in the summer of 2019 and comprised two librarian faculty, two patient partners, two research faculty, and two scpor staff. one librarian (cb) was the principal investigator, and all team members contributed to all aspects of the review. we collectively determined that the rapid review would examine patient partner perceptions of meaningful engagement in srs. the review took place from august 2019 to september 2020, with timelines extended due to the covid-19 pandemic. patient partners bring their lived experience, rather than research expertise, to a review and therefore are not expected to be well-versed in the research methods. for this reason, we were very deliberate about adding training at all stages of the review. most meetings consisted of training to varying degrees, discussion, and task allocation. team members completed tasks between meetings and met in small groups where necessary. prior to march 2020, regular team meetings were held in person, and after that date the team shifted to online meetings due to public health guidelines related to the covid-19 pandemic.     in most instances, librarians join a review team to fulfill specific needs or roles on the team, such as expert searcher (dudden & protzko, 2011; gore & jones, 2015; spencer & eldredge, 2018). both authors of this paper have engaged in srs in various capacities as researchers and in our professional practice providing sr services. in this review, we initiated and led the review which included facilitating the formation of the research question, leading the training, and not only participating in, but also managing the conduct of the review. researchers and stakeholders we had previously provided sr services to now became part of our review team, leading to a transition in our roles from service-provider to research partner. both authors enhanced their knowledge and skills in all aspects of an sr by engaging fully in all steps of the research project, as well as learning the nuances of por, recognizing resource requirements, managing workload, providing ongoing methodology training, and ensuring clear communication.    on a very practical level the immediate and concrete outcomes of our engagement in this research will be the por additions to the toolkit. we have identified key articles (e.g., black et al., 2018; hamilton et al., 2018) and methods for meaningfully engaging patient partners (boden et al., 2021), and points in the sr process at which this information will be useful to researchers. in this paper, however, we wish to focus at a deeper and more generalizable level, extending this learning to the potential impacts of patient-oriented or stakeholder-engaged reviews to librarians and library sr services. below we discuss the implications first for librarians and then for library sr services.     implications for librarians providing synthesis review services   our experience, existing literature (black et al., 2018; hamilton et al., 2018) and the results of our rapid review (boden et al., 2021) emphasize the importance of strong communications skills, understanding team dynamics, relationship building, and the ability to provide training, such as on literature search methods. emotional and interpersonal skills are infrequently talked about as competencies for providing sr services (townsend et al., 2017). however, research team dynamics are affected by relationships, power dynamics, trust, and demands on time (thurow et al., 1999; whitehair & berdanier, 2017). key approaches for engaging patient partners and other stakeholders in reviews include relationship building, clear communication, and clarity on roles and time commitments (black et al., 2018; boden et al., 2021; hamilton et al., 2018; pollock et al., 2021). applying these approaches requires inter-related knowledge and skills to be successful in working with review teams. these include: navigating research team dynamics; communicating to diverse audiences within a review team; and sensitivity to the ‘place’ (motivations, expertise, goals, perspectives, and others) from which team members are contributing. below we illustrate the role of these competencies through our experience with this review.   the dynamics within each review team are unique and librarians providing sr services often interact with teams without the benefit of knowing those dynamics. for instance, the inclusion of patient partners in our review added different dimensions to the team dynamics as they brought unique perspectives, competencies, and experience. this kind of dynamic may heighten the need for librarians to understand their audience and tailor communications accordingly, especially when patients and other stakeholders experience the review process through different lenses and knowledge bases. this has also been noted for other kinds of multi-disciplinary teams (thurow et al., 1999).     interpersonal and communication skills are infrequently discussed and under-appreciated in regard to librarian competencies for supporting srs. an exception is nicholson et al. (2017) who identified interpersonal interactions as a challenge for librarians supporting srs. they recommended clear communication as a strategy to address this challenge. our experience highlighted the need for all team members, but particularly the individual leading a discussion, to possess skills at facilitating complex discussions and communicating clearly to a diverse audience. there were many times throughout the process where we had to recalibrate through some challenging conversations, particularly defining the research question. as the expert searchers on a review team, librarians may be in situations where there is a need to lead complex conversations through multiple points of view, for instance emphasizing the centrality of a well-formed research question and literature search to the overall sr quality (schvaneveldt & stellrecht, 2021). for these kinds of multi-disciplinary teams, it is all the more important to be able to communicate clearly to diverse audiences.   in addition, librarians must be cognizant of and sensitive to the personal needs and experiences of all team members, including the patient partners involved in the review. patient partners often engage in research with specific motivations and share their lived experience. librarians need to be aware of these factors as these may direct, influence, or impact the research process, such as formulating the research question, and the team dynamics (e.g., when delegating tasks). the choice of language and terms should be chosen carefully as language that may be appropriate for some circumstances may be perceived as too blunt or insensitive for individuals who have a personal stake or experience with the research topic. to illustrate, vale et al. (2012) describe the surprise of a patient partner to the blunt and scientific communication about the disease used by a clinician in a research setting as compared to the bedside (p. 5). this means not only being aware of patient partners’ or stakeholders’ motivations for being on the research team, but also being cognizant of their emotional engagement, personal goals, and reactions to the experience of being involved in the review.    in light of the reflections above, we considered the competencies and associated upskilling of librarians supporting patient-oriented srs. librarians who are new to por or stakeholder engagement would benefit from an introduction to the nuances of engaging in this form of research. we believe there is a need for librarians supporting patient-oriented or stakeholder-engaged reviews to have additional training in various areas related to interpersonal skills and communication. these areas include: understanding why and how patients (or indeed other stakeholders) may engage in the review; the ability to adapt to the unique needs of a particular team, particularly that of the patient partners who may come to the project with a diverse and unique skill set; knowledge and skills at facilitating discussions; and understanding of research team dynamics. libraries providing sr services can support librarians by providing formal support, such as mentoring or training in these skills beyond the technical or methodological kinds. recognizing that these skills are important and there is increasing interest in patient-oriented and stakeholder-engaged srs, these skills should be explicitly included in librarian competencies for supporting srs.    implications for synthesis review service models   in addition to reflecting on the relevance of our experience to individual librarians conducting por srs, we also reflected on the provision of library sr services. a common factor with most srs is that the process takes time (bullers et al., 2018). patient and stakeholder approaches to reviews may increase the unpredictability of review timelines, increase the time demands, or lengthen the timelines. in our review, timelines were affected by a variety of factors, including the need to provide training to all team members as an integrated element of the review process.   while it was not our own experience, the literature (e.g., oliver et al., 2019) suggests potential challenges for researchers involved in stakeholder-engaged research, some of which may also impact librarians supporting por. co-producing research can introduce tensions for researchers and stakeholders between the individual consequences and the potential positive impacts on health research (oliver et al., 2019). a recent qualitative study suggests some researchers and patient partners report a degree of ambivalence in their experiences and perceptions of engaging in por; they see value and reward, but also the need for additional support and infrastructure to manage some of the complexities (boylan et al., 2019). while a patient-oriented approach offers the promise of enhanced health research, it also comes with complexities that researchers and patient partners must consider. in describing realist reviews, abrams and colleagues noted a need for researchers and patient partners to clarify how and why they would be involved (abrams et al., 2020). they provide reflection prompts to engage researchers and patient partners in examining their own expectations, expertise, scope of commitment, communication needs, and so forth prior to committing to the project. although their focus was on realist reviews, we believe this can be applied to other kinds of srs. in our view, these questions shed light on some of the complexities of engaging in patient-oriented reviews and may also be appropriate for librarians providing sr support or entering into these types of collaborations.   to address some of the additional complexity of engaging patient partners in srs, libraries should consider developing guidelines to manage boundaries and expectations of the sr services (e.g., time, ethical, legal), particularly if their librarians are supporting patient-oriented or stakeholder-engaged srs. especially in light of time demands, this could improve the sustainability of the sr services. these guidelines could include agreements on the average allowable time per project (e.g., based in hours or months, depending on the level of involvement) and articulate the extent and nature of the role of the librarian on the research team. these guidelines or policies at the library level, formalize, or at least recognize, ways of partnering with stakeholders or stakeholder organizations to manage expectations, outcomes, ethical issues, and legal issues.     conclusion    by engaging in an evidence-based approach, we not only contributed to the research on patient-oriented reviews (boden et al., 2021), but also enhanced our understanding as it applied in our practice (e.g., will provide guidance on patient-oriented reviews to our synthesis toolkit). employing a reflective approach (rolfe, 2010) allowed us to consider the broader implication of these kinds of reviews to library sr services and models. we believe competencies in navigating research team dynamics, clear communication to diverse audiences within a review team, and sensitivity to the “place” from which team members are contributing (motivations, expertise, goals, perspectives, and others) should be recognized. awareness of the unique attributes of patientor stakeholder-oriented reviews can prepare libraries and librarians to offer services that accommodate the requirements of this methodology. we argue in this commentary that while technical competencies for librarians providing sr services are well described in the literature, additional knowledge and skills deserve consideration, and that sr services should include policies, guidelines, and training to support librarians who engage in this ever-evolving area of researcher support.   references   abrams, r., wong, g., hamer-hunt, j., gudgin, b., tierney, s., dawson, s., boylan, a.-m., & park, s. (2020, january 13). the role (or not) of patients and the public in realist reviews. the national institute for health research (nihr) school for primary care research. https://www.spcr.nihr.ac.uk/news/blog/the-role-or-not-of-patients-and-the-public-in-realist-reviews    bastian, h., glasziou, p., & chalmers, i. (2010). seventy-five trials and eleven systematic reviews a day: how will we ever keep up? plos medicine, 7(9). https://doi.org/10.1371/journal.pmed.1000326   black, a., strain, k., wallsworth, c., charlton, s.-g., chang, w., mcnamee, k., & hamilton, c. (2018). what constitutes meaningful engagement for patients and families as partners on research teams? journal of health services research & policy, 23(3), 158-167. https://doi.org/10.1177/1355819618762960   boden, c., edmonds, a. m., porter, t., bath, b., dunn, k., gerrard, a., goodridge, d., & stobart, c. (2021). patient partners’ perspectives of meaningful engagement in synthesis reviews: a patient-oriented rapid review. health expectations, 24(4), 1056-1071. https://doi.org/10.1111/hex.13279   borton, t. (1970). reach, touch, and teach: student concerns and process education. mcgraw-hill.   boylan, a.-m., locock, l., thomson, r., & staniszewska, s. (2019). “about sixty per cent i want to do it”: health researchers’ attitudes to, and experiences of, patient and public involvement (ppi)-a qualitative interview study. health expectations: an international journal of public participation in health care and health policy, 22(4), 721-730. https://doi.org/10.1111/hex.12883   bullers, k., howard, a. m., hanson, a., kearns, w. d., orriola, j. j., polo, r. l., & sakmar, k. a. (2018). it takes longer than you think: librarian time spent on systematic review tasks. journal of the medical library association, 106(2), 198-207. https://doi.org/10.5195/jmla.2018.323   canadian institutes of health research. (2019a, may 8). foundations of spor. https://cihr-irsc.gc.ca/e/51039.html   canadian institutes of health research. (2019b, may 27). patient engagement. https://cihr-irsc.gc.ca/e/45851.html   the cochrane collaboration. (2021). involving people: a learning resource for systematic review authors. cochrane training. https://training.cochrane.org/involving-people   dudden, r. f., & protzko, s. l. (2011). the systematic review team: contributions of the health sciences librarian. medical reference services quarterly, 30(3), 301-315. https://doi.org/10.1080/02763869.2011.590425   gore, g. c., & jones, j. (2015). systematic reviews and librarians: a primer for managers. partnership: the canadian journal of library and information practice and research, 10(1). https://doi.org/10.21083/partnership.v10i1.3343   hamilton, c. b., hoens, a. m., backman, c. l., mckinnon, a. m., mcquitty, s., english, k., & li, l. c. (2018). an empirically based conceptual framework for fostering meaningful patient engagement in research. health expectations, 21(1), 396-406. https://doi.org/10.1111/hex.12635   ioannidis, j. p. a. (2016). the mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. the milbank quarterly, 94(3), 485-514. https://doi.org/10.1111/1468-0009.12210   kastner, m., antony, j., soobiah, c., straus, s. e., & tricco, a. c. (2016). conceptual recommendations for selecting the most appropriate knowledge synthesis method to answer research questions related to complex evidence. journal of clinical epidemiology, 73, 43-49. https://doi.org/10.1016/j.jclinepi.2015.11.022   nicholson, j., mccrillis, a., & williams, j. d. (2017). collaboration challenges in systematic reviews: a survey of health sciences librarians. journal of the medical library association, 105(4), 385-393. https://doi.org/10.5195/jmla.2017.176   oliver, k., kothari, a., & mays, n. (2019). the dark side of coproduction: do the costs outweigh the benefits for health research? health research policy and systems, 17(33). https://doi.org/10.1186/s12961-019-0432-3   page, m. j., shamseer, l., altman, d. g., tetzlaff, j., sampson, m., tricco, a. c., catalá-lópez, f., li, l., reid, e. k., sarkis-onofre, r., & moher, d. (2016). epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. plos medicine, 13(5), e1002028. https://doi.org/10.1371/journal.pmed.1002028   pollock, a., campbell, p., struthers, c., synnot, a., nunn, j., hill, s., goodare, h., morris, j., watts, c., & morley, r. (2018). stakeholder involvement in systematic reviews: a scoping review. systematic reviews, 7(208). https://doi.org/10.1186/s13643-018-0852-0   pollock, a., campbell, p., synnot, a., & smith, m. (2021). patient and public involvement in systematic reviews. guidelines international network (g-i-n) public toolkit: patient and public involvement in guidelines. https://g-i-n.net/wp-content/uploads/2021/04/ppi-in-systematic-reviews-final-for-pdf-publication-1.pdf   rolfe, g. (2010). models and frameworks for critical reflection. in d. freshwater, g. rolfe, & m. jasper (eds.), critical reflection in practice: generating knowledge for care (2nd ed., pp. 31-51). red globe press.   rolfe, g., & jasper, m. (2010). critical reflection and the emergence of professional knowledge. in d. freshwater, g. rolfe, & m. jasper (eds.), critical reflection in practice: generating knowledge for care (2nd ed., pp. 1-8). red globe press.   schvaneveldt, n., & stellrecht, e. m. (2021). assessing the roles and challenges of librarians in dental systematic and scoping reviews. journal of the medical library association, 109(1), 52-61. https://doi.org/10.5195/jmla.2021.1031   slattery, p., saeri, a. k., & bragge, p. (2020). research co-design in health: a rapid overview of reviews. health research policy and systems, 18(17). https://doi.org/10.1186/s12961-020-0528-9   spencer, a. j., & eldredge, j. d. (2018). roles for librarians in systematic reviews: a scoping review. journal of the medical library association, 106(1). https://doi.org/10.5195/jmla.2018.82   sutton, a., clowes, m., preston, l., & booth, a. (2019). meeting the review family: exploring review types and associated information retrieval requirements. health information and libraries journal, 36(3), 202-222. https://doi.org/10.1111/hir.12276   thurow, a. p., abdalla, c. w., younglove-webb, j., & gray, b. (1999). the dynamics of multidisciplinary research teams in academia. the review of higher education, 22(4), 425-440. https://doi.org/10.1353/rhe.1999.0019   townsend, w. a., anderson, p. f., ginier, e. c., maceachern, m. p., saylor, k. m., shipman, b. l., & smith, j. e. (2017). a competency framework for librarians involved in systematic reviews. journal of the medical library association, 105(3), 268-275. https://doi.org/10.5195/jmla.2017.189   tricco, a. c., soobiah, c., antony, j., cogo, e., macdonald, h., lillie, e., tran, j., d’souza, j., hui, w., perrier, l., welch, v., horsley, t., straus, s. e., & kastner, m. (2016). a scoping review identifies multiple emerging knowledge synthesis methods, but few studies operationalize the method. journal of clinical epidemiology, 73, 19-28. https://doi.org/10.1016/j.jclinepi.2015.08.030   university of saskatchewan. (2021, august 31). synthesis review toolkit introduction: synthesis review toolkit. https://libguides.usask.ca/synth-review   vale, c. l., tierney, j. f., spera, n., whelan, a., nightingale, a., & hanley, b. (2012). evaluation of patient involvement in a systematic review and meta-analysis of individual patient data in cervical cancer treatment. systematic reviews, 1(23). https://doi.org/10.1186/2046-4053-1-23   whitehair, c., & berdanier, c. g. p. (2017, june). the role of trust in collaborative research settings: opportunities for future research in graduate engineering education [paper presentation]. american society for engineering education (asee) annual conference and exposition, columbus, oh, united states. https://peer.asee.org/the-role-of-trust-in-collaborative-research-settings-opportunities-for-future-research-in-graduate-engineering-education     evidence based library and information practice evidence based library and information practice 2009, 4.4     evidence based library and information practice         news/announcements    salford, uk to host 6th international evidence based library and information practice  conference in 2011      © 2009. this is an open access article distributed under the terms of the creative commons attribution   license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and   reproduction in any medium, provided the original work is properly cited.      eblip6, the sixth in a series of successful  international conferences to promote  evidence based approaches to library and  information practice will take place in salford,  greater manchester, united kingdom in  summer 2011.  a team based in manchester  and its environs successfully competed against  three other high quality expressions of interest  from around the world to secure a  unanimously favourable verdict.  the international advisory committee (iac)  for the international evidence based library  and information practice conference series  awarded the eblip6 conference to the salford  bid, coordinated by dr. alison brettle and  maria grant of the university of salford.  competing against high quality expressions of  interest from canada, the middle east and  elsewhere in the united kingdom, salford  performed strongly against all the rigorous  evaluation criteria.   this is the first time that an open competition  has been used to select the successful hosting  site. criteria for selection included the  location, facilities, regional and national  support, the strength of the organising team  and unique selling points. salford achieved a  first place ranking from each of the five voting  members on the iac. in particular salford was  commended for the “strength of its local         organising committee and the overall extent of  national and regional support”.    andrew booth, reader in evidence based  information practice at the university of  sheffield, convenor for the iac, stated that  “the international advisory committee is  mindful of a continuing need to branch out to  a wider international community. at the same  time the responsibility to support an already  active concentration of those involved in  eblip remains fundamental to the ongoing  success of the movement. these  considerations, coupled with the undoubted  quality of the four bids made our decision  very challenging. ultimately, however, the  case for the salford bid was an irresistibly  compelling one”.    welcoming the committee’s decision dr.  alison brettle, chair of the local organising  committee and one of only four participants  to have attended all five conferences to date,  said “it is great to see such positive  comments”. she described how maria grant,  herself and the salford team would keep these  comments in mind as they tackled “the hard  work that lies ahead”.  ʺthis is a wonderful and exciting prospect”  enthused biddy fisher, current vice president  and president designate for 2010 of cilip, the  uk chartered institute for library and  100 evidence based library and information practice 2009, 4.4  information professionals. she continued: “i  will do all i can to assist the bid….in my  official capacity, as ebp is an underlying  theme in my presidential manifesto”.   for ongoing information during 2010 on the  eblip6 conference check the open access  journal, evidence based library and information  practice, the principal organ of the eblip  community and the evidence‐based‐libraries  jiscmail  (http://www.jiscmail.ac.uk/lists/evidence‐ based‐libraries.html) discussion list. you  can also contact andrew booth at the e‐mail  address given below. about the eblip  conference series  the eblip conference is a biennial conference  designed to promote the use of best available  evidence to improve library and information  practice in all types of libraries and to thus  address issues around inequitable use of  resources, unacceptable provision of services                             and ineffective comprehension and use of  information that continue to plague human  society. it aims to bring together practitioners,  researchers and students involved in critical  and reflective information practice to think  through new ways to address contemporary  issues of evidence based practice in less  isolated, more collaborative, ways. the first  ebl(ip) conference took place in sheffield,  uk in 2001. the success of this conference led  to four further conferences in edmonton  (canada, 2003), brisbane (australia, 2005),  chapel hill (usa, 2007) and stockholm  (sweden, 2009).   contact  andrew booth,   convenor of the international advisory  committee,  c/o scharr, university of sheffield  +44(0)114 222 0705  a.booth@sheffield.ac.uk      101 http://www.jiscmail.ac.uk/lists/evidence-based-libraries.html http://www.jiscmail.ac.uk/lists/evidence-based-libraries.html mailto:a.booth@sheffield.ac.uk microsoft word news_cilip.doc evidence based library and information practice 2007, 2:4  118 evidence based library and information practice     news    uk library and information research group (cilip) research award 2008           © 2007 glynn. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    the uk library & information research  group (lirg) research award is designed to  promote research in library and information  science.  the aim of the award is to  encourage and facilitate research by  practitioners in the field.     lirg welcomes proposals from all sectors of  the profession, and particularly invites  practitioners to apply either as sole  applicants, or in collaboration with  academic or independent  researchers.  applications from academics in  library and information science departments  are welcome and proposals that are  submitted in partnership with practitioners  are particularly welcome.  the award is  open to applicants from the uk only.     the award is worth £1000, and designed to  support small‐scale research projects for  which it may otherwise be difficult to find  funding. the award may be used to defray  research expenses (e.g. travel, postage costs),  to fund attendance at high level meetings or  to fund a study tour. however, a clear  overall aim for the research must be stated.     guidelines for submission and evaluation  criteria are on the lirg web pages  at:     http://www.cilip.org.uk/specialinterestgrou ps/bysubject/research/activities/awards/rese archaward.htm     the closing date for submissions is february  29th, 2008     applications and enquiries should be sent,  preferably by e‐mail, to:    dr jean yeoh  lirg awards and prizes coordinator  information services & systems  kingʹs college london  4th floor waterloo bridge wing  franklin wilkins building  150 stamford street  london se1 9nn  jean.yeoh@kcl.ac.uk  020 7848 4460  http://creativecommons.org/licenses/by/2.0 http://www.cilip.org.uk/specialinterestgrou mailto:yeoh@kcl.ac.uk evidence based library and information practice evidence based library and information practice 2010, 5.1  evidence based library and information practice       article    course‐integrated learning outcomes for library database searching: three assessment  points on the path of evidence    dianne cmor  head, information services  hong kong baptist university library  kowloong tong, kowloon  email: dcmor@hkbu.edu.hk    alison chan  information services  hong kong baptist university library  kowloong tong, kowloon  email: hychan@hkbu.edu.hk    teresa kong  shek mun campus library  hong kong baptist university library  kowloong tong, kowloon  email: teresak@hkbu.edu.hk     received: 5 aug. 2009          accepted: 22 jan. 2010       2010 cmor, chan, and kong. this is an open access article distributed under the terms of the creative  commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐ nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the  original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is  redistributed under the same or similar license to this one.    abstract    objective ‐ this study aims to assess student learning with respect to basic database  searching at three different points within a required first year course.     methods ‐ three methods were employed at three different points to identify  evidence of successful learning:    1. analysis of in‐class exercises from the initial library workshop, e.g. how many  students showed evidence of satisfactorily achieving the stated learning  outcomes.  64 mailto:dcmor@hkbu.edu.hk mailto:hychan@hkbu.edu.hk mailto:teresak@hkbu.edu.hk evidence based library and information practice 2010, 5.1  65 2. participant observation of student presentations, noting themes, strengths and  weaknesses of student research strategy; written observation reports from  librarians were coded and quantified to identify major themes.    3. interviews with course instructors responsible for grading the final submitted  projects, focusing on both student achievement and instructor perceptions of  the impact of library involvement.    results ‐ though performance on in‐class exercises showed evidence of successful  learning in over 70% of students, observational data indicated that very few students  showed evidence of applying new knowledge and new search skills to their own  topics two weeks later. instructor interviews revealed a perception of similar  difficulties in final project submissions, and instructors suggested that students did  not appreciate the need for library resources.     conclusion ‐ in this study, students showed evidence of learning in a simulated  environment, but were unable or unwilling to demonstrate this learning in authentic  situations. multiple assessment methods reveal a lack of student ability to apply  search skills.    introduction    this study aims to assess student learning  with respect to basic database searching at  three different points within a required first  year information, communication and  technology (ict) course. for the purpose of  this study, basic database searching is defined  as accessing two recommended databases,  constructing an effective search statement, and  accessing the full text of relevant articles.  student exercises, oral presentations and final  projects were examined to determine whether  or not students were able to effectively find  newspaper, magazine and journal articles to  support a project designed to introduce  students to the information management  cycle.     the authors also self‐assessed the study  against an evidence based practice framework,  with the aim of improving research  performance at the hong kong baptist  university library. for many practitioners, the  idea of research or evaluation often comes as  an afterthought to the daily work that needs to  be done – work that often arises quickly with  little time for thorough planning. still, the  methods, data, and results of this are often  both rich and plentiful. if used and improved  upon, both in terms of programme  development and assessment methodology,  practicing librarians can take further steps  towards using and further developing the  evidence‐base of our profession.    while looking for opportunities to comment  on and improve upon the methodologies  employed in this study (which will be  discussed separately in the discussion section  of this paper), the main focus remains on the  data that was collected in the hopes of  improving student learning in future  implementations. specific questions for this  study are:       1. were students able to successfully  complete assigned exercises submitted  at the end of a library resources  problem‐solving session?    2. two weeks later, during their  presentations on data collection, did  students demonstrate that they could  successfully apply what they had  learned about database searching to  their own project topics?      3. did final project submissions show  evidence of student ability to  evidence based library and information practice 2010, 5.1  effectively utilize library databases in  their information management cycle  projects as perceived by course  instructors?    context    hong kong baptist university (hkbu) is a  mid‐sized government‐funded university  offering a wide range of undergraduate and  postgraduate programs. in the spring of 2008,  the library identified a first year, required  course in information technology as a course  that might benefit from an information literacy  component, and approached the computer  science department who had been running  the course for several years. the course was  being redesigned and renamed “information  management technology”, and would include  lectures, labs and tutorials. tutorials were  named “problem‐solving sessions” (pss), and  would focus on the “information management  cycle”. it was decided that the library would  facilitate a pss session on using library  resources for the data collection stage of the  overall cycle.         the library had recently adopted a clear  outcomes‐based template for designing and  delivering library instruction – one that  identifies outcomes, assessment methods,  curriculum and pedagogy. the learning  outcomes for the library pss were that  students would be able to:    1. access two recommended library  databases (academic search premier  and wisenews)  2. construct effective search strategies for  use in these databases  3. access the fulltext of articles where  available, using our link resolver, when  necessary.    as 50 sections of the course had to be taught in  one week (approximately 20 students per  section), nine librarians were involved in the  teaching. session content included what a  database is and how to access them, how to  construct a simple search statement, and how  to access full text.  pedagogies employed were  large‐group discussion, mini‐lectures and  hands‐on exercises. two weeks after the  library pss, a phase one submission was  required of students focusing on needs  identification and data collection activities for  their independent projects. though all  students had to submit a powerpoint  presentation at this point in the semester  outlining their work thus far, one third of  students were also required to orally present  their work. librarians were invited to attend  these 50 presentation classes, and, in  collaboration with the course instructors, to  provide on the spot feedback to students.     literature review    the literature on assessment of library  instruction is voluminous, and has grown  considerably in recent years. rader (2002)  noted in her thirty‐year review of information  literacy literature, that assessment of learning  outcomes had been minimal but had begun to  increase (p. 244). in his review of the literature  on information literacy assessment, matthews  (2007) revealed that studies reported  contradictory results as to whether or not  library instruction improves student  performance (p. 72). koufogiannakis and  wiebe (2006) systematically reviewed the  literature on effective teaching methods for  information literacy skills instruction to  undergraduate students focusing on the state  of the research base and comparative teaching  methods (p. 3‐43). for the purpose of this  study, a focus on current methods used in  assessing learning outcomes will be  emphasized over comparative assessments of  instructional methods.     lorenzen (1999) articulated a clear connection  between outcomes‐based learning and library  instruction. after defining outcome‐based  education (obe) as “a method of teaching  that focuses on what students can actually do  after they are taught” (p. 142), he posited that  library instruction has always had an  outcomes‐based perspective due to the goal of  having students be able to do something  following instruction, such as use a catalog or  find an article. brettle (2007) reviewed  66 evidence based library and information practice 2010, 5.1  assessment methods in health libraries noting  that most study designs were quantitative in  nature (p. 22). beyond health libraries,  quantitative designs are an increasingly  prevalent assessment approach in many  academic libraries as adopted in studies  conducted by burkhardt (2007, p. 32), emmett  and emde (2007, p. 216), byerly, downey, and  ramin (2006, p. 592), and portmann and  roush (2004, p. 462). multiple‐choice  questionnaires, surveys and student  perception instruments are also frequently  used, for example, in furno and flanagan  (2008, p. 265), houlson (2007, p. 101), and  wong, chan and chu (2006, p. 386). walsh  (2009, p. 21) reviewed the common types of  information literacy assessments categorizing  them under the headings of questionnaires,  tests, self‐assessments, analysis of  bibliographies or essays, portfolios,  observations of search behaviors, and  simulations. most assessment studies focus on  the cognitive and affective domains ‐ on what  students know at a given time, and their  perceptions of what and how well they had  learned from library instruction interactions.  the challenge of assessing concrete ability to  demonstrate and apply information literacy  skills in authentic situations remains.     fewer articles discussed using formative  assessment such as classroom assessment  techniques (cats), and authentic assessment  such as performance assessment methods, to  measure student learning outcomes following  instruction. choinski and emanuel (2006) used  the one‐minute paper together with an  assessment rubric to assess student learning in  one‐hour library sessions (p. 150). knight  (2006) assessed bibliographies compiled by  undergraduate students in their first‐year  research and writing course according to a  scoring rubric based on course learning  objectives (p. 46). diller and phelps (2008)  discussed learning outcomes and authentic  assessment through the use of electronic  portfolios (p. 79). the portfolios contained self‐ selected samples of student work and self‐ reflections on perceived learning in relation to  the learning outcomes. judd et al. (2004) used  a worksheet to assess students’ ability to  search the catalogue/database in a business  course library session, and then further  determined how well students understood the  resources taught by observing students work  in out‐of‐class group assignments that  followed the library instruction (p. 276).  emmett and emde (2007) adopted a pre‐ and  post‐test model with the assessment questions  developed as “information‐needed scenarios”  with multiple‐choice and short‐answer tests  (p. 216). warner (2003) used multiple methods  including student journals, librarian  reflections, observation of student speeches,  and a survey tool that asked students about  resource use (p. 171). overall, a general trend  in these studies can be observed wherein  students tend to perform lower in  demonstrating or applying higher order skills  in formative and authentic assessment  situations.    though our study did not set out to measure  student attitude about information sources or  searching, it became an increasingly important  theme, and as such a brief review of recent  literature provides some necessary context.  several studies (ciber, 2008, p. 31; d’esposito  & gardner, 1999, p. 458; griffiths & brophy,  2005, p. 545; head & eisenberg, 2009, p. 11;  kim & sin, 2007, p. 658; oclc, 2005, p. 1‐26;)  reported on students’ preference and use of  internet search engines for assignments,  especially as their first source. these studies  noted student satisfaction with their search  results and the information found, or student  frustration with the difficulties involved in  using library resources.  zoellner, samson and  hines (2008) investigated the importance that  students ascribed to different information  resources, and found that library resources  increased in importance following instruction  (p. 378). ursin, lindsay and johnson (2004)  reported that despite anecdotal evidence  supporting student appreciation and learning,  students did not actually use the resources  recommended by librarians (p. 291). kim and  sin (2007) studied perception and selection of  sources, and suggested that accessibility and  ease of use (real or perceived) played a crucial  role in source selection (p. 663).    67 evidence based library and information practice 2010, 5.1  interestingly, the literature on assessment of  information literacy instruction suggests what  it embodies – that is, many authors suggest  that multiple assessment methods are  necessary in any study in order to assess the  richness of student learning, and, in parallel, a  wide‐range of assessment approaches and  methods exist in the literature itself. this  study begins from the perspective of using  multiple assessment methods to assess very  specific learning outcomes, but will eventually  lead to an even broader view of the many  variables that affect deep learning and  authentic assessment.      methods    a multiple assessment approach was  undertaken which assessed student learning  through submitted database‐searching  exercises, librarian observation of student  presentations two weeks later, and  perceptions of imt instructors on student  performance in their final project submissions.  with over 1000 students enrolled in the  course, 966 students submitted the three in‐ class exercises, and approximately one third of  students made oral presentations.    method for assessment of in‐class exercises    as the course was constantly evolving, even  during the semester, it was only at the last  minute that the computer science instructors  requested that librarians collect student  worksheets in the library pss, and provide  feedback on them before the next pss the  following week. all librarians were asked to  review the submitted exercises from their  sections (see appendix) and provide general  comments on the strengths, weaknesses, and  patterns to the coordinating librarian. the  coordinating librarian, taking all comments  into account, wrote a general summary of  student work, along with providing  suggestions for improvement. this feedback  was posted on the course website so that all  students could access it. students did not  receive individual feedback on their submitted  work.    further, given that the exercises would be  submitted to and reviewed by librarians, a  new opportunity was seized upon – to very  quickly ascertain the number of students who  were successfully achieving the learning  outcomes that had been set. two librarians,  who had been involved in the library’s  outcomes assessment exercises for other  courses, assessed each of the student  worksheets to determine whether the learning  outcomes were being achieved to a  satisfactory degree. due to time constraints, a  very simple “yes” or “no” scheme was used  for each student worksheet. the database  access outcome was considered achieved if at  least one of the two exercises requiring  database use (one exercise for wisenews, and  one exercise for academic search premier)  was completed without major errors. the  search strategy outcome was considered  achieved if the strategy was free of major  errors that would make the strategy  untenable, e.g. wrong boolean operator, and  included at least two of the following  components correctly: synonyms, the boolean  and, the boolean or, truncation, and  parentheses as needed.    method for assessment of presentations    two weeks following the library pss, selected  students, who were randomly selected by the  course instructors, orally presented their  phase one submissions of their final projects.  there were 50 pss sections in which students  presented their chosen topics, focus areas,  needs identification, and the resources they  used or planned to use for data collection.     for each section, the assigned instructor and  librarian provided on the spot feedback to  students – sometimes after each presentation,  but usually as a summary at the end of the  class period. after the first two sections, the  coordinating librarian decided to seize on  another assessment opportunity and asked all  participating librarians to take brief notes and  write a short one or two paragraph summary  of their observations of each class, i.e., each set  of presentations, with each set consisting of 8‐ 10 presentations. as six sections did not  68 evidence based library and information practice 2010, 5.1  69 require student presentations, and two  librarians combined two sections in one  report, 40 observational reports of 350‐400  student presentations were available for  analysis.  the observational reports were compiled into  one document, and a simple content analysis  was undertaken to determine major themes,  issues, strengths and weaknesses. a first  reading of the reports was undertaken by the  project team of three librarians to get a general  “feel” for the nature of the data. a second  reading was undertaken by two of the  librarians (librarian a & b) in order to  identify themes that were embedded in the  textual content. the themes that naturally  came from the observational data were then  mapped to and categorized under the three  learning outcomes, in order to focus our  analysis, i.e. access databases to find articles,  construct an effective search strategy, and  access full text. the themes were further  subdivided in order to tally whether the  observational data reported evidence of  successful learning, evidence of unsuccessful  learning, or no evidence as shown in table 1.   the third reading of the reports was  undertaken separately by two of the librarians  (librarian a & c) and resulted in the analysis  of each report against the categorized themes  and the evidence of learning provided.  statements not related to the learning  outcomes were classified into separate clusters  by themes, and tallied to simply indicate  whether or not a report mentioned this theme.  the third librarian (librarian b) then reviewed  the two files, identified discrepancies, made  the final judgments, and prepared the final  version. this process resulted in the  identification of the most common threads  observed by librarians, and noted, where  applicable, evidence of successful or  unsuccessful learning.    method for instructor perceptions/assessment of  learning    interviews with the imt instructors were  conducted four weeks after the student  presentations and by the time of the interview,  the instructors had received final project  submissions from students. six out of a total of  nine instructors accepted our interview  invitation with a participation rate of 66.7%.  the interviews were conducted by two  members of the project team, with an attempt  to match interviewers and instructors who had  not worked together during the course. this  was the case in five of the six interviews. open‐ended questions were used to allow  interviewees to express thoughts and opinions  freely without being influenced by pre‐defined  answers suggested by the interviewer.  five  questions were formulated for the interviews.  three questions focused on the general  perception and evidence of student learning  based on instructors’ observations throughout  the course and on the final project  submissions. two further questions focused  on instructor perceptions of library  involvement and impact, and on soliciting  suggestions for improvement in the future.     interview content was transcribed by the two  interviewers (librarian a and b). content was  then analyzed and coded under categories in  relation to the interview questions by one  librarian (librarian a). these categories were  derived from the interview content, and  further signified as being either positive or  negative in nature. the interview content and  coding were then second read and reviewed  by the other two librarians (librarian b and  c). positive comments would speak to areas of  successful student learning or benefit from  library involvement in the course, and  negative comments would speak to areas of  weak student learning or lack of benefit from  library involvement.      results    a total of 966 students attended the library  pss and submitted the three required  exercises.  a total of 954 students attended the  phase one presentation sessions, with  approximately one third of those students   evidence based library and information practice 2010, 5.1          table 1   content analysis worksheet in an excel format  learning  outcomes  access an appropriate database to  find news/journal articles  construct an effective search strategy  access ft  using    web  bridge  themes  derived  from data  g oo gl e/   w ik i/  w eb   l ib  d b s  l ib   ca ta lo gu e  t hi nk   ar ou nd /  d if fe re nt   an gl es   fo cu s  th e  to pi c  id en ti fy   m ai n  co nc ep ts /  k w s  id en ti fy   sy no ny m s/   r t   a vo id  s of t  w or d s  t ru nc at io n  b oo le an   op er at or s  a n d / o r   se ar ch   lim it s  pr ov id ed   by  d b  a cc es s  ft   evidence  suggests  achieve‐ ment  y  n  n  m  y  n  n  m  y  n  n  m  y  n  n  m  y  n  n  m  y  n  n  m  y  n  n  m  y  n  n  m  y  n  n  m  y  n  n  m  y  n  n  m  y  n  n  m  session 1                                                                          session 2                                                                          …                                                                          total                                                                          key:  y=yes   n=no   nm=not mentioned 70 evidence based library and information practice 2010, 5.1  giving a five minute oral presentation of their  project work to date. as feedback on  presentations was provided from librarians  and instructors, it was hoped that such  feedback could reinforce learning for all 954  students in attendance.    in‐class exercise results    written comments provided by all librarians  following each of the library pss sessions  showed similar threads. most students were  able to access databases from the library  homepage, apply limits provided by databases  platforms such as date and publication type,  and determine if full text was available for a  given article. students struggled with working  through the steps of building a search  strategy, and needed quite a lot of feedback  and support. this was viewed positively as  the “problem‐solving” approach gave  students the opportunity to work through  their difficulties and get feedback from the  librarian and the course instructor who were  moving around the room, helping students  work through their exercises. librarians noted  that more time would have been useful, so  they could attend to all students or provide  more examples.         in assessing the student worksheets, using a  simple pass or fail methodology (wherein a  pass meant that the two learning outcomes  that were feasible to assess were demonstrated  to an acceptable level on the submitted  exercises), 71% of students achieved the  learning outcomes, that is, they were able to  access the recommended databases and  construct an effective search strategy.     presentation observation results    the following observations were most  commonly made in the reports prepared by  the attending librarians relating to the learning  outcome which addressed the ability to access  appropriate databases to find articles. use of  library databases was mentioned in 39 of the  40 reports, with 29 reporting that evidence  suggested achievement of the learning  outcome, and 10 reporting that the outcome  was not achieved. use of internet search  engines was noted in 30 of the 40 reports, with  all mentioned indicating evidence that  students could successfully use a search  engine to find information for their projects.    it was regularly commented upon that the  presentations seemed to indicate that internet  searching had actually been undertaken by  students, while library searching, when  mentioned, was often as part of a future plan.  it seemed that many students had not yet  searched for articles on their topics in library  databases ‐ of the 39 reports commenting on  use of library databases, only 9 provided  evidence of actual use (by stating what they  had found or what limits they had used).    constructing an effective search strategy was  the second learning outcome. unfortunately,  as a group, students were unable to  demonstrate their knowledge or skills to a  great extent. there were some indications of  learning ‐ 24 reports mentioned identifying  main keywords with 22 indicating success,  and 2 indicating problems. 25 reports  mentioned student ability to identify  appropriate synonyms/related terms, with 20  indicating success and 5 indicating lack of  success. however, use of inappropriate terms  in searches, such as “in” “of” and “effect”  were noted in 11 reports. use of boolean  operators was noted in 28 reports, with 14  suggesting that generally students successfully  used these operators, and the other 14  suggesting that students demonstrated  considerable difficulty.  truncation was  mentioned in 23 reports, with only 10  suggesting appropriate use and 13 reporting  problems. several reports noted that students  were using google‐type search operators such  as “+” in databases that would not support  such operators. the fact that 30‐40% of reports  did not mention boolean or truncation  suggested that students were not sharing their  search strategies in their presentations at all,  indicating a lack of evidence of learning.    overall, the student achievement as  demonstrated in their presentations was far  less than satisfactory. it was clear that students  71 evidence based library and information practice 2010, 5.1  had spent most of their information‐searching  energy using internet search tools such as  google and yahoo. many presentations only  included “plans” to use library databases.  even for those students who had used library  databases, their ability to construct effective  searches was quite weak overall. most  students could not put together all the pieces  they had learned two weeks earlier and apply  what they had learned to a new topic. finally,  many students simply did not demonstrate  what they had tried so it was difficult to assess  their progress thus far.      interview results    instructors were asked how well students  performed in demonstrating achievement of  course learning outcomes in their final project  submissions. responses on 9 out of 17  statements (53%) focused on students’  weaknesses in the areas of creative thinking,   formulating search strategies, and awareness  of the importance of using academic resources                                                         in their research. responses that spoke to  students’ strengths (47%) highlighted strong  software application skills, and the ability to  identify synonyms and related terms for  searching purposes.     when specifically asked whether students  demonstrated concrete evidence of learning to  effectively access or search the library  databases, responses from the instructors  varied. half of the responses were positive,  especially in identifying keywords, coming up  with synonyms and related terms, and listing  at least one database taught in the imt classes.  the other half of the responses were negative,  with comments on an inability to focus and  narrow a topic or search, and the lack of  justification for the search tools selected.  another common problem identified was  confusion in using appropriate search  operators in google and library databases (see  figure 1).                                                      ─ a (4) ─ b (4) ─ c (0) ─ d (3) ─ e (0) ─ f (5) ─ g (1) ─ h (0) ─ a (2) ─ b (2) ─ c (1) ─ d (4) ─ e (1) ─ f (5) ─ g (1) ─ h (1) 0 5 10 15 20 number of positive statements number of negative statements [a] identify kws [e] write down search strategy for lib [b] come up with synonyms/ rt [f] list lib dbs used [c] avoid soft words/ unnecessary words [g] focus/ narrow down the search [d] use proper search operators for the tools, e.g. and/or/* for lib dbs while using +/-/"…" for google? [h] justify the search tools they selected to use     fig. 1. themes found in comments on whether students demonstrated concrete  evidence of learning to effectively access/search library databases.  72 evidence based library and information practice 2010, 5.1  the majority of instructor statements (13 out  of 16, 81%) indicated that there was  insufficient change in student information  searching abilities resulting from librarian  involvement in the course. students’  perception on the need for scholarly journal  articles, along with the difficulty of searching  library databases were mentioned as  hindering any changes in students’ abilities.    despite the lack of results in student learning,  instructors’ overall impression of librarians   participating in the imt classes was positive  (15 out of 20 statements, or 75%). comments  about the positive contribution of the library  involvement included: librarians gave  enthusiastic motivation for students to go  further in their research, constructive  comments were given to students, the  databases introduced were very appropriate to  year‐1 level study, and a general  understanding of the differences between  magazines and journals was beginning to take  shape. the negative or unsuccessful aspects of  library involvement included: student anxiety  over the presence of a librarian in the  classroom, insufficient time spent on database  searching, and, as some of the project topics  were very general, scholarly articles were not  necessary.  suggestions for future improvement are  divided into two categories: those addressed  to the library (10 out of 24 statements, or 42%)  and to the imt course designers (14 out of 24  statements, or 58%). many suggestions were  made in relation to time allocated, content  covered, project topics, and instructions and  guiding questions in the project handbook.  suggestions addressed specifically to the  library included providing more specific  comments to individual presentations (not  only general comments to all which was often  the case due to time constraints), spending  more time on search strategies and less time  on navigating the library’s database pages,  and comparing how search results would  differ between the internet and library  databases. suggestions addressed to the  course designers included clarifying the  questions (expectations) and terminology used  in the project handbook, announcing the  possible project topics earlier to students,  connecting the library session worksheet  directly with the students’ project topics, and   increasing the time allocated for the library  session. z   discussion    discussion of results    learning is about the interplay of knowledge,  skills and attitudes, and perhaps more  importantly, it is about learning to apply new  knowledge, skills and attitudes. the three  questions posed by this study were aimed at  determining whether 1) students were able to  successfully complete in‐class exercises, 2)  students were able to apply skills to their own  topics as demonstrated in presentations, and  3) students showed evidence of an ability to  effectively utilize library databases in their  information management cycle projects as  perceived by course instructors. though 71%  of students successfully completed the  exercises, very few demonstrated that they  could utilize new tools and apply new search  skills to a new topic, or that they could   incorporate new knowledge of resources and  searching techniques into their final projects.  given that the “information management  cycle” conceptual approach and the “problem  solving session” pedagogical approach was  being used for the first time, it was not  surprising that outcomes would only partially  be met. previous experience with the  assessment‐improvement cycle in our classes  suggests that in the next incarnation of the  library session, with modifications to content  focus and wording of exercises, it is likely that  approximately 80% of students will be able to  demonstrate successful learning via the  exercises. however, the low success rate in  students being able to later apply this learning  to their own information needs is problematic  indeed. even if a 100% success rate with  immediate, in‐class exercises was achieved,  how useful is such success if only a small  percentage of students can transfer and apply  their learning to their own projects?     73 evidence based library and information practice 2010, 5.1  in‐class exercises showed a basic level of  comprehension, but a deeper level of affective  and cognitive understanding appeared to be  lacking. most students were unable to  demonstrate in their presentations that they  could apply searching skills to new situations,  and they showed little evidence of believing  that this was worrisome. performance‐based  exercises at the end of a library session,  though perhaps more useful than self‐ assessment or multiple‐choice tests, is clearly  not enough to gauge true, internalized  learning.     interview data revealed that instructors of the  imt course believed that one of the reasons  students did not use library resources to a  satisfactory extent was that they did not  understand or appreciate the need for  including professional or academic sources in  their projects. were students unable or simply  unwilling to find resources from library  databases? as the 50 minute library pss  session focused on how to search and access  articles, where in the course was the question  of why one would search and access articles  covered?      the results of this study suggest that authentic  application and attitude should be closely  examined in relation to student learning. less  time might be spent on skill development and  more time on attitudinal perspectives about  the use of library/scholarly resources. what  use is it for students to learn to access  appropriate databases and construct an  effective search strategy, if they are not  inclined to use a database in their information  gathering? if more attention is paid to student  knowledge and attitudes about scholarly  resources, then motivation might make up for  lack of skill. even if their database searching  skills are somewhat basic, they might apply  these skills more readily in authentic  information‐seeking situations, and they  might seek assistance and respond to feedback  if they believe that library resources are  important to the quality of their work. in this  way, their skills can be improved upon over  time, and actually put to use.    discussion of methodology    the newness of the course meant that course  delivery was constantly in flux, leaving  librarians to quickly respond to requests, and  in some instances identifying assessment  opportunities only after the fact. still, certain  comments can be made about the methods  employed to assess student learning, and how  they might be improved upon in the future.    as it was not known that there would be the  opportunity to assess in‐class exercises ahead  of time, the method used was not as rigorous  as it could have been. with no time for  creating assessment rubrics and only one week  to return close to 1,000 worksheets, the “quick  and dirty” assessment was considered the best  that could be done at the time. experience  with assessing similar types of worksheets for  other courses provided an acceptable degree  of confidence in the findings.    observation “data” from the presentations  was not initially identified as such. curiosity  of the library course coordinator prompted the  request for librarians to send emails  summarizing their observations of the  presentations. the realization that these emails  held an immense amount of insightful  qualitative data only came after the fact. as  such, observation data was difficult to  normalize and analyze, but the major themes  and threads were apparent. the data indicated  that student application of the knowledge and  skills they had learned two weeks earlier was  weak.    interviewing imt instructors was better  planned and more standardized than the other  two methodologies employed in this study.  however, our interest in library‐related  learning outcomes did not necessarily match  with how instructors were assessing the final  projects.  instructors were certainly able to  give their general perceptions of student  ability and attitude, which was certainly a  useful indicator of student learning, but it was  not possible to determine how many students  showed evidence of achieving the library‐ related outcomes in their final projects.     74 evidence based library and information practice 2010, 5.1  conclusion     this study attempted to assess student  learning of database searching skills, both  within a library session and as applied over an  entire course. three different assessment  methods were used at three different points in  the course, which shed light both on student  learning, behaviors and attitudes. though  students showed evidence of learning in a  simulated environment, that is, successfully  completing exercises as part of classroom  learning, they were unable or unwilling to  demonstrate this learning in authentic  situations as reflected in their presentations  and from instructor interviews. when working  on their own projects, students generally did  not use library databases, or if they did, they  did not apply the searching skills that they  had previously demonstrated.     simply put, the information seeking behaviour  of students did not change as a result of  librarian involvement in this course. course  instructors suggested that students did not  appear to value the use of library resources  over internet resources, and did not  understand that university level projects  would require the use of professional and/or  scholarly resources. though students can learn  the skills related to library information seeking  (as shown in other studies as well), this study  suggests that much more attention needs to be  given to the attitudes that drive student  information seeking behaviour. given that  google is so much easier than our  comparatively complicated databases,  students need to learn why it is important to  search library databases, not just how to do so.  without the former, the latter is without  meaning or use.    recommendations    recommendations for the course      discussions with course instructors have  already identified some changes, which will be  implemented into the course. students will be  required to submit the results of preliminary  searching before the library session, in the  hope that this will better introduce the library  session as a way to expand upon  “background” internet searching, and move  into a more academic or professional arena. it  was agreed that an attitudinal focus on why  and in what circumstances one might go  beyond google was badly needed.    assessment of student learning can be  improved in a number of ways, and thus  continue to inform a practice of continuous  assessment and improvement. assessment of  in‐class exercises should be done using a  standardized assessment rubric allowing for  consistency and the identification of varying  degrees of success. gathering qualitative data  from observing student presentations would  also benefit from a more structured approach,  possibly using observational checklists or from  following a stronger methodological  framework for content analysis of  observations. further, inter‐rater reliability  methods should be adopted for example,  oakleaf (2009) investigated a rigorous rubric  approach to information literacy assessment  by analyzing the inter‐rater reliability of rubric  scores (p. 970).     finally, library‐related learning outcomes   should be adopted by imt instructors so that   they can clearly ascertain whether or not  students are achieving these outcomes as  evidenced in their final projects. such  outcomes, with an assessment rubric, might be  developed by librarians, accepted by  instructors, and shared with students  (alongside other project learning outcomes) in  order to clearly guide students in their  learning and their ability to demonstrate that  learning.    recommendations for practice    this study confirms what most practitioners  already know from anecdotal experience ‐  that assessing skills is not enough when it  comes to assessing authentic student learning  in the realm of information literacy.  librarians cannot limit assessment efforts to  determining whether or not students know  how to search (skills), but must also pay  75 evidence based library and information practice 2010, 5.1  attention to whether or not students know  why to search certain resources for certain  types of information needs (knowledge &  attitude).    undergraduate students need to learn how to  think about research, not just how to carry out  certain skill‐based steps. if librarians are to  move beyond skills, and become more deeply  involved in the richness of authentic student  learning, close collaboration and teamwork  with faculty is crucial. wider exploration and  discussion of the processes involved when  undertaking research should be included in  instruction, and reflective research exercises  should be built into instruction sessions and  into course research assignments.    results of the systematic review by  koufogiannakis and wiebe (2006) indicated  that inadequate research has been conducted  to assess higher level learning outcomes (p.  12). as large‐scale information literacy  assessment tools are becoming more widely  adopted at the institutional level, it is  important that librarians rigorously assess  these tools in themselves to ensure that the  affective and cognitive realms are adequately  addressed. if such tools only tell us that  students are aware of a variety of resources  and that they possess the requisite search  skills to use them, they do little in telling us  whether students have truly learned about the  importance of information in all of its many  forms, for all of its many purposes, and can  put that learning into appropriate action.      recommendations for further study    as a result of this study, several areas for  further study have also presented themselves,  leading us further down the path of  integrating research into practice. possible  studies may include pre‐ and post‐testing with  a focus on cognitive awareness and attitudinal  change, investigating whether a short,  additional online exercise immediately  following the in‐class exercises will result in  improved retention and application of skills,  investigating whether library‐constructed  rubrics for course learning outcomes have an  effect on student learning, and comparing  various ways in which such rubrics might be  used. studying permutations of course  delivery comparing librarians and instructors  delivering different parts of the content, or  comparing face‐to‐face and online instruction  are also potential studies. two systematic  reviews highlighted that the majority of  studies reviewed were in the u.s. context  (brettle, 2007, p. 21; koufogiannakis & wiebe,  2006, p. 20). further studies that can add to an  international evidence base are certainly  needed, as students in hong kong, brazil,  nigeria or europe might learn quite  differently than the average american student.    research should inform the decisions and  activities of practitioners, and the activities of  practitioners should form part of the research  base. “evidence‐based library and information  practice (eblip) seeks to formalize the links  between research and professional activities  …” (given, 2007, p. 16). eldredge (2000)  proposed a conceptual framework for ebl  that recognizes and allows for pragmatism,  diversity, continuous improvement in research  rigor, and the application of best‐available  evidence (p. 291). this study begins from a  place of relative immaturity, where  environmental constraints, the speed at which  a new course changes, and the culture of  librarian practice did not encourage the use of  research in daily practice. in light of pragmatic  considerations and best‐available evidence  within a local setting, first attempts at  undertaking research will need to be assessed  and improved upon in later incarnations.  librarians are very familiar with the  continuous‐improvement cycle used in library  instruction and other areas of service  performance assessment. it would be of great  enefit to apply such a cycle to our research  ractices, as well.  b p     references     brettle, a. (2007). evaluating information skills  training in health libraries: a systematic  review. health information & libraries  journal, 24(suppl. 1), 18‐37.   76 evidence based library and information practice 2010, 5.1  burkhardt, j. m. (2007). assessing library   skills: a first step to information literacy.  portal: libraries & the academy, 7(1), 25‐49.    byerly, g., downey, a., & ramin, l. (2006).  footholds and foundations: setting  freshmen on the path to lifelong learning.  reference services review, 34(4), 589‐598.     centre for information behaviour and the  evaluation of research. (2008).  information behaviour of the researcher of the  future ‐ a british library/ jisc study.  london: university college london.  retrieved from  http://www.ucl.ac.uk/infostudies/researc h/ciber/downloads/     choinski, e., & emanuel, m. (2006). the one‐ minute paper and the one‐hour class.  reference services review, 34(1), 148‐155.     dʹesposito, j. e., & gardner, r. m. (1999).  university studentʹs perception of the  internet: an exploratory study. journal of  academic librarianship, 25(6), 456‐461.     diller, k. r., & phelps, s. f. (2008). learning  outcomes, portfolios, and rubrics, oh my!  authentic assessment of an information  literacy program. portal: libraries & the  academy, 8(1), 75‐89.     eldredge, j. d. (2000). evidence‐based  librarianship: an overview. bulletin of the  medical library association, 88(4), 289‐302.     emmett, a., & emde, j. (2007). assessing  information literacy skills using the  acrl standards as a guide. reference  services review, 35(2), 210‐229.     furno, c., & flanagan, d. (2008). information  literacy: getting the most from your 60  minutes. reference services review, 36(3),  264‐271.     given, l. m. (2007). evidence‐based practice  and qualitative research: a primer for  library and information professionals.  evidence based library and information  practice, 2(1), 15‐22. retrieved from  http://ejournals.library.ualberta.ca/index. php/eblip/article/view/155/237     griffiths, j. r., & brophy, p. (2005). student  searching behavior and the web: use of  academic resources and google. library  trends, 53(4), 539‐554.     head, a. j., & eisenberg, m. b. (2009). finding  context: what todayʹs college students say  about conducting research in the digital age  (project information literacy progress  report). university of washingtonʹs  information school. retrieved from  http://www.projectinfolit.org/pdfs/pil_pr ogressreport_2_2009.pdf     houlson, v. (2007). getting results from one‐ shot instruction: a workshop for first‐ year students. college & undergraduate  libraries, 14(1), 89‐108.     judd, v., tims, b., farrow, l., & periatt, j.  (2004). evaluation and assessment of a  library instruction component of an  introduction to business course: a  continuous process. reference services  review, 32(3), 274‐283.     kim, k., & sin, s. j. (2007). perception and  selection of information sources by  undergraduate students: effects of  avoidant style, confidence, and personal  control in problem‐solving. journal of  academic librarianship, 33(6), 655‐665.     knight, l. a. (2006). using rubrics to assess  information literacy. reference services  review, 34(1), 43‐55.     koufogiannakis, d., & wiebe, n. (2006).  effective methods for teaching  information literacy skills to  undergraduate students: a systematic  review and meta‐analysis. evidence based  library and information practice, 1(3), 3‐43.  retrieved from  http://ejournals.library.ualberta.ca/index. php/eblip/article/view/76/153     77 http://www.ucl.ac.uk/infostudies/research/ciber/downloads/ http://www.ucl.ac.uk/infostudies/research/ciber/downloads/ http://ejournals.library.ualberta.ca/index.php/eblip/article/view/155/237 http://ejournals.library.ualberta.ca/index.php/eblip/article/view/155/237 http://www.projectinfolit.org/pdfs/pil_progressreport_2_2009.pdf http://www.projectinfolit.org/pdfs/pil_progressreport_2_2009.pdf http://ejournals.library.ualberta.ca/index.php/eblip/article/view/76/153 http://ejournals.library.ualberta.ca/index.php/eblip/article/view/76/153 evidence based library and information practice 2010, 5.1  78 lorenzen, m. (1999). using outcome‐based  education in the planning and teaching of  new information technologies. journal of  library administration, 26(3), 141‐152.     matthews, j. r. (2007). library assessment in  higher education. westport, conn.:  libraries unlimited.     oakleaf, m. (2009). using rubrics to assess  information literacy: an examination of  methodology and interrater reliability.  journal of the american society for  information science and technology, 60(5),  969‐983.     online computer library center. (2005).  perceptions of libraries and information  resources. dublin, ohio: oclc online  computer library center. retrieved from  http://www.oclc.org/reports/2005percepti ons.htm     portmann, c. a., & roush, a. j. (2004).  assessing the effects of library  instruction. journal of academic  librarianship, 30(6), 461‐465.           rader, h. b. (2002). information literacy 1973‐  2002: a selected literature review. library  trends, 51(2), 242‐259.      ursin, l., lindsay, e. b., & johnson, c. m.   (2004). assessing library instruction in the  freshman seminar: a citation analysis  study. reference services review, 32(3),  284‐292.     walsh, a. (2009). information literacy  assessment: where do we start? journal of  librarianship & information science, 41(1),  19‐28.     warner, d. a. (2003). programmatic  assessment: turning process into practice  by teaching for learning. journal of  academic librarianship, 29(3), 169‐176.     wong, g., chan, d., & chu, s. (2006).  assessing the enduring impact of library  instruction programs. journal of academic  librarianship, 32(4), 384‐395.     zoellner, k., samson, s., & hines, s. (2008).     continuing assessment of library  instruction to undergraduates: a general  education course survey research project.  college & research libraries, 69(4), 370‐383.    http://www.oclc.org/reports/2005perceptions.htm http://www.oclc.org/reports/2005perceptions.htm evidence based library and information practice 2010, 5.1  appendix 1: database exercise example part ii: search strategies for library databases what is the most important part of searching a database? taking a minute to think about: • the most important aspects of your topic • what keywords could be used • how to connect your terms and use * to form the best search 1. step one: define your question/information need. do children in hong kong eat a healthy diet? 2. step two: identify the most important ideas (tip: ignore ‘soft’ words) 3. step three: brainstorm for synonyms or related terms 79 evidence based library and information practice 2010, 5.1  80 4. step four: decide which terms to try first, and combine terms to build your search. common connectors search example search result “ ” “health care” exact phrase search and influenza and children articles containing both/all terms or influenza or flu articles containing either term * child* all words starting with child e.g. child, children, childhood ( ) child* and (influenza or flu) orders search properly so results are relevant write out two possible searches for our topic: now, using your search strategies above, try searching academic search premier (from ebscohost) for a good article on our topic. article title: ________________________________________________ journal title: ________________________________________________ volume/issue: _____________ year: _________ pg: ___________ is it available in fulltext? yes___ no ___ evidence based library and information practice 2010, 5.1  appendix 2. textual analysis of the observations in the problem‐solving sessions  learning outcomes access an appropriate database to find news/journal articles construct an effective search strategy themes derived from data google / wiki / web sites (to find basic info) lib dbs, (or print / e-journals) lib catalogue (to find books) think around / different angles (creative thinking) focus the topic identify main concepts/ kws identify synonyms / rt avoid soft words / unnecessary words truncation boolean operators and / or search limits provided by database e.g. date, headline, academic evidence suggests achievement y n nm y n nm y n nm y n nm y n nm y n n m y n n m y n n m y n nm y n nm y n nm total 30 0 10 29 10 1 8 2 30 2 0 38 6 5 29 22 2 16 20 5 15 6 11 23 10 13 17 14 14 12 4 0 36 learning outcomes access ft using webbridge other observations not related to learning outcomes themes derived from data access ft on the nature of the topic on the nature of the info. sources and evaluation criteria on the experience of using internet search on the experience of using lib. resources on confusion relating to use of lib db page or homepage on confusion relating to the assignment e.g. what to write for "search strategy" evidence suggests achievement y m nm m nm m nm m nm m nm n nm m nm total 0 4 36 2 38 10 30 9 31 12 28 3 37 5 35 y= yes n= no nm= not mentioned or evidence not provided m= mentioned 81 evidence summary   first-year student essays shed light on their experience of acrl framework threshold concepts   a review of: dempsey, p. r., & jagman, h. (2016). i felt like such a freshman: first-year students crossing the library threshold. portal: libraries and the academy, 16(1), 89-107. https://doi.org/10.1353/pla.2016.0011   reviewed by: heather macdonald health and biosciences librarian macodrum library carleton university ottawa, ontario, canada email: heather.macdonald@carleton.ca   received: 31 july 2019                                                                   accepted:  30 oct. 2019      2019 macdonald. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29614     abstract   objective – to synthesize student narratives on searching for an item in the library and to identify information literacy threshold concepts students encountered during their searching.   design – constant comparative analysis.   setting – academic library at an urban american university.   subjects – a sample of 97 1-to-2 page ungraded first year student essays.   methods – a library assignment was developed for first year students in a required academic skills course. students wrote the essay for peer mentors. after completing the essay, students were asked if they wanted to participate in the study. for the assignment, students were asked to find a library item of interest and write a reflective essay on the process. essays were analyzed using nvivo software. the researchers developed codes independently, then came together to review, discuss and recode the essays. using the constant comparison method, themes were identified from the coding. narrative analysis was used to understand the coding in the context of the students’ experiences.    main results – the authors outlined various search paths that the students described in their essays. the main emotional responses in the essays were surprise, confusion, and excitement. three acrl framework il concepts were identified in the analysis: scholarship as conversation, searching as strategic exploration, and research as inquiry. scholarship as a conversation was exemplified through students’ selection of a library item. students chose topics that were of academic interest or associated with personal identity. in the essays, students explained their connection to the item they found, making the connection to the ongoing scholarly conversation. searching as strategic exploration was expressed through student descriptions of connecting the call number to the subject classifications. some students sailed through, whereas others encountered challenges. some found that previous library mental models failed, found the catalogue overwhelming, or thought the organization of material was at fault rather than their own skills. some students described how they overcame their challenges. students also discussed balancing self-reliance and seeking help when searching for an item. this related to the acrl frames of research as inquiry and searching as strategic exploration. attitudes on seeking help ranged from complete reliance to anxiety.   conclusion – this library assignment offered students the opportunity to pursue their own interests and goals. it also encouraged exploration, problem-solving, and reflection. the assignment design allowed students to grapple with information literacy threshold concepts in a safe and independent environment, demonstrating learning and engagement with academia.   commentary   this commentary uses the cat critical appraisal tool (perryman & rathbun-grubb, 2014) to guide the appraisal. this paper is the second part of a study; the first part was presented as a poster (jagman, davidson, dietz, falk, & fitzpatrick, 2013). the authors do not clearly articulate a research question for this paper. their stated goal is to provide a synthesis of student narratives showing how students learn to navigate the library, the challenges they face, and their emotional responses to these challenges. the authors present a succinct and clear introduction to the topic, discussing student engagement, reflective learning, and the acrl framework threshold concepts.    the authors selected a qualitative approach to analyze student essays. given their focus on exploring students’ feelings and perceptions of their experiences navigating the library, qualitative analysis is an appropriate choice. the authors provided a detailed explanation of their methods, although an appendix of the codes and how they were translated into the narrative would have been helpful. the authors explained that the sample they used was not meant to be a representative sample; rather, their goal was to study the richness of this sample and explore the various threshold concepts encountered and experienced. there is no discussion of ethics approval in this article, which seems odd considering they are using student essays as their data source.   the findings presented a detailed account of the students’ experiences. they are presented in a variety of formats – flow charts describing navigation pathways, emotions expressed in a word cloud, and narrative with quotations – allowing the reader to understand the results easily. in addition, the assignment and representative samples are included in the appendix. the narrative analysis follows the stages of the assignment, identifying the acrl threshold concepts encountered along the way. this logical progression is easy to follow. the authors provide extensive quotations supporting their interpretation of student experience with the threshold concepts.   whereas the findings focused on the navigation pathways and the threshold concepts experienced, the discussion and conclusions emphasized the instructional design aspects of the exercise. this comes across as disconnected from the findings. if there had been a specific research question, this section could have referred back to it. in its absence, linking the findings back to the literature on the acrl framework would have strengthened this section. the discussion about instructional design was interesting to read but not what was analysed.   the lack of a research question weakened this paper. the analysis was well executed, showing that students do encounter the threshold concepts, but was not discussed further in the discussion and conclusions section, making the paper disjointed. the findings will benefit those interested in learning about first year student experiences of threshold concepts and the discussion on instructional design will help those interested in designing assignments to capture student library experiences.    references                 perryman, c. & rathbun-grubb, s. (2018). the cat: a generic critical appraisal tool. in jotform – form builder. retrieved from http://www.jotform.us/cp1757/thecat   jagman, h., davidson, l., dietz, l., falk, j., & fitzpatrick, a. (2013). i felt like such a freshman: creating library insiders. retrieved from http://libguides.depaul.edu/ala2014aia       evidence based library and information practice evidence based library and information practice 2010, 5.3 39 evidence based library and information practice evidence summary google scholar out-performs many subscription databases when keyword searching a review of: walters, w. h. (2009). google scholar search performance: comparative recall and precision. portal: libraries and the academy, 9(1), 5-24. reviewed by: giovanna badia librarian, royal victoria hospital medical library, mcgill university health centre montreal, quebec, canada email: giovanna.badia@mail.mcgill.ca received: 2 june 2010 accepted: 19 july 2010 2010 badia. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to compare the search performance (i.e., recall and precision) of google scholar with that of 11 other bibliographic databases when using a keyword search to find references on later-life migration. design – comparative database evaluation. setting – not stated in the article. it appears from the author’s affiliation that this research took place in an academic institution of higher learning. subjects – twelve databases were compared: google scholar, academic search elite, ageline, articlefirst, econlit, geobase, medline, pais international, popline, social sciences abstracts, social sciences citation index, and socindex. methods – the relevant literature on later-life migration was pre-identified as a set of 155 journal articles published from 1990 to 2000. the author selected these articles from database searches, citation tracking, journal scans, and consultations with social sciences colleagues. each database was evaluated with regards to its performance in finding references to these 155 papers. elderly and migration were the keywords used to conduct the searches in each of the 12 databases, since these were the words that were the most frequently used in the titles of the 155 relevant articles. the search was performed in the most basic search interface of each database that allowed limiting results by the needed publication dates (1990-2000). mailto:giovanna.badia@mail.mcgill.ca evidence based library and information practice 2010, 5.3 40 search results were sorted by relevance when possible (for 9 out of the 12 databases), and by date when the relevance sorting option was not available. recall and precision statistics were then calculated from the search results. recall is the number of relevant results obtained in the database for a search topic, divided by all the potential results which can be obtained on that topic (in this case, 155 references). precision is the number of relevant results obtained in the database for a search topic, divided by the total number of results that were obtained in the database on that topic. main results – google scholar and ageline obtained the largest number of results (20,400 and 311 hits respectively) for the keyword search, elderly and migration. database performance was evaluated with regards to the recall and precision of its search results. google scholar and ageline also obtained the largest total number of relevant search results out of all the potential results that could be obtained on later-life migration (41/155 and 35/155 respectively). no individual database produced the highest recall for every set of search results listed, i.e., for the first 10 hits, the first 20 hits, etc. however, google scholar was always in the top four databases regardless of the number of search results displayed. its recall rate was consistently higher than all the other databases when over 56 search results were examined, while medline out-performed the others within the first set of 50 results. to exclude the effects of database coverage, the author calculated the number of relevant references obtained as a percentage of all the relevant references included in each database, rather than as a percentage of all 155 relevant references from 1990-2000 that exist on the topic. google scholar ranked fourth place, with 44% of the relevant references found. ageline and medline tied for first place with 74%. for precision, google scholar ranked eighth among the 12 databases when the complete set of search results was examined, but ranked third within the first 20 search results listed. within the first 20, 55% of the search results were relevant. this precision rate put google scholar in third place, after medline (80%) and academic search elite (70%). google scholar’s precision and recall statistics may have been positively affected by its search for a keyword in the full-text content of indexed articles, rather than just searching in the bibliographic records as is the case for the other 11 databases. the author re-calculated the recall and precision rates for a title search in google scholar using the same keywords, elderly and migration. compared to the standard search on the same topic, there was almost no difference in recall or precision when a title search was performed and the first 50 results were viewed. conclusion – database search performance differs significantly from one field to another so that a comparative study using a different search topic might produce different search results from those summarized above. nevertheless, google scholar out-performs many subscription databases – in terms of recall and precision – when using keyword searches for some topics, as was the case for the multidisciplinary topic of later-life migration. google scholar’s recall and precision rates were high within the first 10 to 100 search results examined. according to the author, “these findings suggest that a searcher who is unwilling to search multiple databases or to adopt a sophisticated search strategy is likely to achieve better than average recall and precision by using google scholar” (p. 16). the author concludes the paper by discussing the relevancy of search results obtained by undergraduate students. all of the 155 relevant journal articles on the topic of laterlife migration were pre-selected based on an expert critique of the complete articles, rather than by looking at only the titles or abstracts of references as most searchers do. instructors and librarians may wish to support the use of databases that increase students’ contact with high-quality research documents (i.e.., documents that are authoritative, well written, contain a strong analysis, or demonstrate evidence based library and information practice 2010, 5.3 41 quality in other ways). the study’s findings indicate that google scholar is an example of one such database, since it obtained a large number of references to the relevant papers on the topic searched. commentary this study evaluated keyword searching in google scholar by calculating the recall and precision rates of the search results with regards to finding references to a preestablished set of 155 relevant papers on the topic. these relevant papers were selected by looking at several factors, such as the subject and importance of conclusions, in the content of the complete articles. according to the author, evaluating google scholar’s search results with a definition of relevancy that is based on the content of the published literature that exists on the search topic, rather than looking at just the titles and abstracts of references, is what makes this particular study unique. the study’s findings suggest that google scholar will obtain above average recall and precision when using a keyword search to find references on a multidisciplinary topic. there are confounding variables in the study that may contradict these findings. the search terms used in each database were based on the two words that appeared most frequently in the titles of the 155 relevant articles. rather than measuring the search performance of each database, the study may have actually assessed the author’s search strategy. the search strategy was very precise and strongly favoured the relevant papers that were preselected on the topic. furthermore, this does not match the author’s intention, which was to reflect the actual behavior of inexperienced searchers. instead, the search strategy does the opposite, since it is unnatural; searchers will not know the exact words that are used in the titles of the majority of documents on a specific topic. a major concern for this reviewer is that this study does not assess google scholar’s search performance in obtaining references to recently published documents, which is an extremely important factor for examining the relevancy of search results on many topics, especially in the health sciences. google scholar’s recall and precision rates were calculated based on finding references published between 1990 and 2000. despite its weaknesses, this study improves our understanding of recall and precision for keyword searching in each of the 12 databases examined. this will help reference librarians to recommend the best database to inexperienced searchers who wish to find a few relevant papers on a specific topic. evidence summary   updated survey information about librarian-researchers prompts authors to consider revising the curriculum for their institute for research design in librarianship course    a review of: kennedy, m.r., & brancolini, k.r. (2018). academic librarian research: an update to a survey of attitudes, involvement, and perceived capabilities. college and research libraries, 79(6), 822-851. https://doi.org/10.5860/crl.79.6.822   reviewed by: elaine sullo coordinator, information and instructional services himmelfarb health sciences library the george washington university washington, district of columbia, united states of america email: elainej@gwu.edu   received: 28 feb. 2019                                                                  accepted: 15 apr. 2019      2019 sullo. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29562     abstract   objective – to examine academic librarians’ current attitudes and preparedness to conduct research in order to update the knowledge gained from the authors’ 2010 survey, and to determine if changes were needed in their institute for research design in librarianship continuing education curriculum.   design – web based survey.   setting – institutions that employ academic and/or research librarians.   subjects – 793 academic and research librarians.   methods – the researchers posted a call for participation in their 2015 librarian research survey on listservs where academic and research librarians are members. the survey expanded upon the authors’ 2010 survey by adding questions to more fully explore three areas: research self-efficacy; master’s thesis and statistics courses, and; research mentoring and institutional support. 793 librarians responded to the survey, and 669 of these respondents completed it. all data from incomplete surveys was included in the analysis. survey results were compared with the results from the 2010 survey as well as with the responses from a survey conducted in 2000 by powel, baker, and mika, which addressed many of the same topics under investigation.   main results – the authors analyzed the survey results based on four areas: the current research practice of responding academic librarians; a self-evaluation of their confidence in performing the steps in the research process; methods training courses in which they have participated, and; demographics and institutional data related to support of library research.      regarding current research practices, 84% of respondents said it is assumed that they will read research-based literature as part of their job as academic librarians; 80% are allowed time at work for this purpose; 6% did not know if it was assumed that they will read research-based literature as part of their job; and 9% were unsure if they were allowed to use work time to read the literature. 78% scan tables of contents for research-based journals, while 58% regularly read the full content of these articles (this is a significant drop from the 78% who reported that they regularly read full text articles in the 2010 survey).   time was the primary reason cited for not regularly reading research-based literature. 77% of respondents have conducted research since completing their library science degree (although 2% did not have a master’s degree).    respondents rated their confidence on a scale of one to five, with one being “not at all confident” and five being “very confident.” overall, there were 38 components related to the steps in the research process, which were grouped into 8 questions on the survey.  for these questions, an average rating of 3.41 was calculated. from statistical analysis, the authors determined that there is a significant correlation between conducting research and librarian confidence in the process.   the survey contained seven questions related to methods training. the authors were specifically interested in the correlation between librarians having conducted research since completing their degree and librarians’ belief that their degree adequately prepared them to do so. statistical analysis revealed that the relationship between these factors was not significant; this result was consistent with the results from the authors’ 2010 survey as well as from the findings of powell, baker, and mika. the authors were also curious as to whether librarians who had written a thesis as part of their library science degree were more likely to have conducted research since earning their degree. this relationship between these variables was not significant, however the relationship between writing a thesis for another graduate degree and conducting research was significant.   conclusion – the number of survey responses indicates that research is still a current, important issue for academic and research librarians. the authors will use the information from the surveys to revise their educational curriculum, specifically in the areas of current research practice, librarian confidence, and methods training.   commentary    as noted by the authors, academic librarians are actively conducting research and contributing their publications to library and information science journals. while both intrinsic and extrinsic factors contribute to this scholarship, the “publish or perish” pressure that accompanies tenure and promotion decisions most likely has a large impact on librarians’ decision to conduct research. the obstacles as well as the factors that contribute to research success for librarians have been noted in the literature, and were examined in the authors’ 2010 survey of academic librarians. from the original survey, the authors developed the institute for research design in librarianship; they now want to dig deeper into research barriers and enablers, and as such, have revised and expanded their survey with the intention of updating their class curriculum for librarian-researchers.   the critical appraisal tool developed by glynn (2006) was used to evaluate this study. while 793 librarians responded to the survey, this number only represents 3% of the total u.s. academic librarian population (american library association, 2018). the sample was not systematically generated, and the results were not intended to be generalized to the larger population of academic librarians. this purposeful sampling may introduce selection bias, as only those librarians who were subscribed to the chosen listservs were invited to participate in the study. regarding data collection, the methods are clearly described, the survey was field tested, and the instrument was included in the publication. the authors provide a detailed rationale for including additional questions in the 2015 survey and list the study results in a comprehensive, organized fashion, including tables that compare results from the 2010 survey, the 2015 survey, and the 2000 powell, baker, and mika survey.   the study findings, while meant to provide insight into the research capabilities of a particular group of librarians, may be broadly applicable to academic librarians; many academic librarians, whether on tenure track or not, engage in research. the evidence suggests that one of the primary barriers for librarians is time, both for reading research articles and for conducting research, as well as confidence in certain steps in the research process. institutions that employ librarian-researchers should consider this data when making decisions about librarian duties, responsibilities, and continuing education opportunities, especially for those librarians on the tenure track or those who hope to contribute to the broader field of library science. because librarian-generated research may benefit the author’s institution and the profession, the findings of this study may have a wider impact beyond the individual librarian.   references   american library association. (2018). number employed in libraries: ala library fact sheet 2. retrieved from http://www.ala.org/tools/libfactsheets/alalibraryfactsheet02   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154   powell, r.p., baker, l.m., & mika, j.j. (2002). library and information science practitioners and research. library & information science research, 24(1), 49-72. https://doi.org/10.1016/s0740-8188(01)00104-9     research article   information services in evidence based medical education: a review of implementation trends   sedigheh khani ph.d. candidate in medical librarianship and information sciences school of health management and information sciences iran university of medical sciences tehran, iran email: khani.se@iums.ac.ir; khani.sedigheh@gmail.com   sirous panahi associate professor department of medical library and information science school of health management and information sciences iran university of medical sciences tehran, iran email: panahi.s@iums.ac.ir   ali pirsalehi assistant professor of internal medicine clinical development research center of taleghani hospital medical school shahid beheshti medical university tehran, iran email: pirsalehi@sbmu.ac.ir   ata pourabbasi assistant professor endocrinology and metabolism clinical sciences institute tehran university of medical sciences tehran, iran email: atapoura@tums.ac.ir   received: 7 oct. 2021                                                                     accepted: 26 apr. 2021       2021 khani, panahi, pirsalehi, and pourabbasi. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   doi: 10.18438/eblip29860     abstract   objective – evidence based medical education requires supportive information services to facilitate access to the needed educational evidence. information services designed specifically for evidence based medical education are limited or locally developed for educational units. for librarians to have an opportunity to cooperate efficiently with medical educators in evidence based medical education, they require an empirical prototype for transmission of clinical evidence at the right place and the right time. therefore, there is a need to recognize types of information services which support evidence based medical education. the purpose of this review is to identify implementation trends of evidence based educational information services.   methods – we found related studies by implementing search strategies in pubmed, embase, web of science, scopus, lista, and google scholar with keywords like: evidence based medical education, information services, and library services. we used reference-checking and citation-checking of related articles for completing the process of locating relevant articles. after employing inclusion and exclusion criteria, we selected 11 articles for inclusion in the review and analyzed them using a narrative review technique.   results – after analyzing the results of the included studies, we identified two elements categorized as program development and five elements categorized as implementation trend. prerequisites of program and the process of designing were essential parts of program development of information services. schedule and type of access, how to receive educational-clinical questions, information services types, responding time, and providing evidence based outputs were the elements of the implementation process of educational supported information services.   conclusion – designing an evidence based educational information service strongly depends on the information needs of learners at each educational level. schedule and type of access to information service, time of responding to the received query, and preparation of evidence based output are essential factors in designing practical educational-developed information services.     introduction   in the 1990s, david sackett introduced the concept of evidence based medicine (ebm). ebm was defined as the use of up-to-date, best evidence in clinical decision making for a better understanding of causation and prognosis of disease, and selecting more appropriate diagnostic tests and treatment strategies based on patient preferences and the clinical condition of the patient (sackett et al., 1996). in the practice of ebm, clinicians complement their clinical expertise with the best available evidence (sackett et al., 1996), which evidence is available from systematic clinical research like systematic reviews, cohort studies, and randomized control trials (burns et al., 2011).   ebm and its applications in different functions of medicine have empowered the medical community (djulbegovic & guyatt, 2017; sur & dahm, 2011). medical practice, healthcare management, clinical research, and of course, medical education has been affected by ebm principles (djulbegovic & guyatt, 2017; shortell et al., 2007). evidence based education/best evidence in medical education (ebe/beme) makes an effort to utilize evidence in education (davies, 1999) and reshape the practices and approaches of learner training based on the best available evidence (harden et al., 2000; hart & harden, 2000). the goal of ebm in clinical practice is to enhance patient treatment, but in medical education, educators train learners in the practice of ebm to empower them to use evidence in clinical practice (guyatt et al., 1992).   medical educators try to have an updated and evidence based approach to their teaching practice in processes such as curriculum revision or implementing new instructional techniques (poirier & behnen, 2014). in the evidence based paradigm of teaching, educators combine up-to-date, quality evidence with previous experience and current educational approaches (chessare, 1996). typical tasks required for evidence based practice in medical education include phrasing a question, designing a search strategy, appraising the evidence, and making the required intervention in the teaching approaches (davies, 1999; harden et al., 2000; hart & harden, 2000). a primary challenge of the above procedure is searching the published literature (poirier & behnen, 2014).   finding the best evidence is one of the main challenges of ebe/beme for medical educators; often they need assistance to effectively find required evidence (chessare, 1996; harden et al., 2000; reed et al., 2005). difficulty in accessing the empirical educational knowledge has a multidimensional nature. medical instructors have expressed some barriers to implementing evidentiary information in education. lack of time for finding evidence based knowledge, the volume of research evidence, lack of educational evidence, lack of access to evidence based educational databases, and difficulty in finding educational evidence were found to be obstacles for accessing relevant evidence (emami et al., 2019; onyura et al., 2015; sandars & patel, 2015; suttle et al., 2015; thomas et al., 2019). searching for evidence consists of two core challenges: how to search for evidence and where to search for evidence (haig & dozier, 2003a, 2003b).   information services fulfill the need for access to evidence in medical practice. the main purpose of information services in a health system is to enhance the decision-making of clinicians in the treatment of patients. the actors of an information service are skilled librarians, and the core activity of information services is transforming requests for evidence into relevant, evidence based information which then impacts clinical decision making (fennessy, 2001). in the process of evidence based decision making, information services with different implementation trends were developed to supply qualified and up-to-date evidence for healthcare practice.   jordan and porritt (2004) established an information service to provide evidence based information for clinicians and patients. the information service supported both access to evidence and education for how to utilize what they could access. mcmaster+ was another type of evidence based information service, which organized information based on evidence hierarchy and facilitated finding required evidence to address related clinical questions (holland & haynes, 2005). mcgowan et al. (2010) developed an information service to provide evidence for primary care practitioners and enhance clinical decision making. these information services to support clinical practice had commonalities in their implementation processes. for example, the process of developing reference services for clinicians consisted of two main components: first, selecting and adapting appropriate technology, and second, training the librarian to deliver the information service. most of the information services supporting clinical decision making were developed on the web with a well-defined, user-friendly interface that enhanced physician access to the best evidence (holland & haynes, 2005; jordan & porritt, 2004; mcgowan et al., 2010). the process of delivering needed evidence began from searching, appraising, and summarizing evidence to transferring  it into practice (davies et al., 2017; holland & haynes, 2005; jordan & porritt, 2004; mcgowan et al., 2010), and reviewing and updating collected evidence periodically (jordan & porritt, 2004).   aims   all of the above evidence based information services were established for clinical practice, but providing evidence for medical education needs its own educational-developed information services (emami et al., 2019; onyura et al., 2015). onyura et al. (2015) stated that the delivery approaches for evidence based knowledge currently available were insufficient and there was a need for new approaches for delivering synthesized evidence that have a concise presentation and are accessible at the point-of-need. in this respect, identifying the implementation trends of information services designed for evidence based education can be prototypical for designing evidence based information services for medical education.   based on the hierarchy of information services in the library, information science & technology abstracts (lista) database thesaurus (ebsco, n.d.-a), information services are developed to fulfill information needs in various fields such as business, agriculture, community, education, and more. in the lista thesaurus, “information services in education” was defined as the “use of data storage, organization, search, retrieval, and transmission services in education” (ebsco, n.d.-b). in the current study, we identified search, retrieval, and transmission aspects of information services in education. therefore, the aim of this review was to identify the types of information services that were provided for ebe/beme and compare the trends of supplying evidence for supporting student teaching and learners training under the concept of evidence based educational information services (ebeis).   methods   article selection   we accessed studies on information services that supported ebe/beme by searching databases and performing forward and backward citation tracking of related articles. we searched pubmed, embase, web of science, scopus, lista, and google scholar using keywords such as “evidence based education,” “evidence based medical education,” “information service,” and “library service.” table 1 depicts our search strategy for the pubmed database.   table 1 search strategy of pubmed results search strategy no. 1,522,557 (“education, medical”[exp.] or teaching[noexp.] or education[noexp.] or “education, professional”[noexp.] or “education, graduate” [noexp.] or “education, continuing” [noexp.] or “education, medical, continuing” [mesh]) or (teaching or training or education* or instruction*) [ti, other term] 1 163,230 (“evidence-based medicine” [exp.] or “evidence-based practice” [noexp.] or evidence-based emergency medicine--education[mesh] or evidence-based medicine--education[mesh: noexp] or evidence-based practice--education[mesh:noexp]) or “evidence-based”[ti, ab, other term] 2 161,765 (“information services” [noexp.] or “information storage and retrieval” [noexp.] or librarians [mesh] or “libraries, medical” [noexp.] or “libraries, hospital” [mesh] or “library services” [noexp.]or“information dissemination” [mesh]) or (librar* or information*)[ti, other term] 3 1,421 1 and 2 and 3 4 1,344 limit to: english language 5 721 limit to: 2010/1/1 and 2020/1/2 6   inclusion and exclusion criteria   after implementing search strategies in each database, we excluded non-english articles, as well as articles focused on evidence based dentistry, nursing, and pharmacy studies. because fields like dentistry have unique educational needs versus medicine, we omitted them from the review. we also excluded study types such as letters, chapters, book reviews, editorials, conference abstracts, and viewpoints. we included studies that described an empirical experiment on the structure and trend of implementing an ebeis. in this review, it was important that the information services were not applied in non-educational clinical settings, nor were proposed templates or opinions. we ended up with 11 articles published between 2010 and 2020 included in the review. we have described the process of selecting related studies in figure 1.   figure 1 prisma flowchart of study selection (liberati et al., 2009).   we compared the bibliographic and introductory data of included studies in table 2.   table 2 introductory data of included studies trend of implementation type of education setting country year implemented year published first author searching evidence based information and delivery of documents rea morning report/ rounds/ team conference canada 2009 2011 (aitken et al., 2011) curriculum architecture-based libguides ugmeb case-oriented problem solving curriculum canada 2008-09 2011 (neves & dooley, 2011) learning package service re morning report usa 2002-10 2011 (weaver, 2011) consulting, searching and delivering of information resources cpec morning rounds philippines 2013 2014 (santos & mariano,  2014) searching and providing evidence for clinical questions re patient rounds usa 2012 2014 (yaeger & kelly, 2014) consult searching service re ebm conference usa 2013-15 2015 (zeblisky et al., 2015) consulting and delivery searching service ced patient-family centered rounds usa 2014-15 2017 (herrmann et al., 2017) information service supporting patient-based scenarios designing/ consulting searching service ugme simulated patient scenarios usa 2017 2018 (blake et al., 2018) consult and delivery information service ce inpatient rounds usa 2016 2018 (brian et al., 2018) consulting and assisting searching service ugme personal librarian program usa 2013-18 2018 (gillum et al., 2018) real-time clinical searching service ce clinical rounds usa 2014-19 2019 (gibbons & werner, 2019) a residency education, b under-graduated medical education, c continuing professional education, d clinical education   data analysis   in this study, we used a narrative review technique for bringing together findings of the different studies and accomplishing the review. narrative analysis with tabular accompaniment is a typical analysis technique for reviews (grant & booth, 2009). a narrative review synthesizes the available evidence from different studies to provide a conclusion from collected literature (green et al., 2006). for the analysis of included studies, first we read the articles carefully. second, we compared the implementation trends of applied information services in the educational clinical setting and identified the similarities and differences between structures of implementation trends. third, we extracted the related themes for each similar part of the identified structure through note-taking. also, we considered the related themes for any differences between applied information services. finally, we organized the related themes of similar parts of implementation trends within the comparison tables.   results   implementation of a program was defined as developing performing procedures for planned tasks and achieving determined objectives (national minority aids council, 2015). in this regard, we tried to highlight typical characteristics of implementation trends in ebeis which were common amongst included studies. after the analysis and comparison of studies, we recognized five oft-mentioned elements of information services implementation trends. in addition, for a better understanding of the implementation process of information services, we summarized the program development process and practical effects of information services.   program development of information services   program development has a multi-step process. the main elements of the program development process are required resources for program implementation, program designing, and predefined measures for determining outputs of the program (national minority aids council, 2015). we determined two elements of program development by comparing the findings of the included studies.   prerequisites of programs   one of the prerequisites of using ebeis is understanding ebm principles. it is essential to ask an evidence based question to receive a relevant response from the information service (aitken et al., 2011; brian et al., 2018). it is important to have a librarian present at the point of teaching when the cases are presented. it helps the librarian more quickly and effectively respond to the learners’ queries (aitken et al., 2011; blake et al., 2018; gibbons & werner, 2019; herrmann et al., 2017; yaeger & kelly, 2014). other prerequisites for an effective information service are speed of internet connection and access to evidence based databases. providing appropriate evidence based information on an educational-clinical question strongly depends on the accessibility of information sources like databases (santos & mariano, 2014). in this regard, the availability of infrastructures like a reference-tracker or data repository which deposits data like educational-clinical questions/answers, frequency of responded/non-responded questions, and common clinical patient problems is essential. deposits of interacted data can be used for subsequent referencing and establishing a database of evidence based educational information for high prevalence clinical disorders (gillum et al., 2018).   process of designing   if an information service is intended to support the evidence based needs of a curriculum, the librarian should consider the structure and needed resources of the curriculum in the design process (neves & dooley, 2011). in this regard, surveying the information needs of intended users helped to design the most appropriate services (zeblisky et al., 2015). the diversity of access channels to information services is an essential factor in the design process. access via multiple communication channels like email, web, social networks, or face-to-face communication facilitates the use of information services for busy clinicians (brian et al., 2018).   implementation trends of ebeis   schedule and type of access to information services   the schedule of implementing information services strongly depended on the volume of assigned tasks that the librarian had to do alongside the duties of information services. in addition, information services which used telecommunications channels like phone or email (brian et al., 2018; gillum et al., 2018; herrmann et al., 2017; santos & mariano, 2014; weaver, 2011) could provide services during a wider span of time (table 3).   in the included studies, information services were implemented in different levels of medical education from undergraduate to postgraduate degree programs. in undergraduate medical education, medical students receive the knowledge and skills needed to be a junior doctor. then, the junior doctor receives more training, especially via clinical education, to gain experience, develop skills for patient care, and prepare for entrance into residency education. this period is considered the internship. residency education is a period of training to educate competent clinicians in a specific medical specialty such as internal medicine. internship and residency programs are the two stages of postgraduate medical education. clinical education provides an opportunity for the trainees to acquire practical skills by rotating between clinical departments of a hospital. clinical education is an essential part of postgraduate training (weggemans et al., 2017; wijnen-meijer et al., 2013). in addition, the final stage of medical education is continuing professional education (cpe), which promotes lifelong learning for clinicians within their clinical settings. cpe supports clinical skill development of medical doctors and enhances the outcomes of patient treatment (bennett et al., 2000). cpe programs are delivered via different methods such as rounds, workshops, seminars, conferences, online learning, telemedicine, and other methods.   medical trainees at the undergraduate and graduate levels receive clinical education in the teaching setting of morning reports and rounds. morning report is a case-based meeting where medical students and their educators discuss a clinical case related to a patient recently admitted to the teaching hospital (amin et al., 2000). rounds or ward rounds are held beside the patient’s bed and consist of medical educators and students that listen to the patient and discuss the case of disease presented (o'hare, 2008).   table 3 schedule/access to information services setting type of education schedule of implementation type of access morning report/ rounds/ team conference (aitken et al., 2011) rea 10-12 hours per week face-to-face patient rounds (yaeger & kelly, 2014) re once per week face-to-face morning rounds (santos & mariano, 2014) cpeb every working days face-to-face/email/ phone morning rounds (santos & mariano, 2014) cpe 24 hours/ all days of week phone/email morning report (weaver, 2011) re 5 days a week face-to-face / email ebm conference (zeblisky et al., 2015) re once per month face-to-face patient-family centered rounds (herrmann et al., 2017) cec not mentioned face-to-face/ email clinical rounds (gibbons & werner, 2019) ce once a week face-to-face inpatient rounds (brian et al., 2018) ce between 3 to 5 days a week face-to-face/ email personal librarian program (gillum et al., 2018) ugmed when the users needed face-to-face/ email a residency education, b continuing professional education, c clinical education, d under-graduated medical education   methods of receiving educational-clinical questions   in most of the implementation trends for ebeis, there is a preference for the presence of a librarian in educational-clinical meetings such as rounds, morning reports, and ebm conferences (aitken et al., 2011; brian et al., 2018; gibbons & werner, 2019; herrmann et al., 2017; santos & mariano, 2014; weaver, 2011; yaeger & kelly, 2014; zeblisky et al., 2015). however, some of the information services were provided only virtual, with online chatting as a predefined connection channel between librarians and users. also, users were able to submit their feedback on the quality of information services via a text box on the web (neves & dooley, 2011). another channel that was provided for receiving educational-clinical queries was an online submission form. receiving queries online made access to information services easier (brian et al., 2018).   types of delivery of information services   the most prevalent type of ebeis was mediated searching and document delivery based on educational-clinical queries (aitken et al., 2011; brian et al., 2018; gibbons & werner, 2019; herrmann et al., 2017; santos & mariano, 2014; weaver, 2011; yaeger & kelly, 2014), consulting services for how to formulate a question, and assistance searching the evidence (blake et al., 2018; gillum et al., 2018; herrmann et al., 2017; santos & mariano, 2014; zeblisky et al., 2015). with mediated searching, the librarian received queries, searched appropriate databases, and delivered relevant evidence to the student.   table 4 evidence based outputs of information services reference number case presentation controlled vocabulary key words applied search strategy search results full-text of search results abstract of search results type of education (yaeger & kelly, 2014) a + b + + + residency education (santos & mariano, 2014) + + continuing professional education (weaver, 2011) + + residency education (herrmann et al., 2017) + clinical education (brian et al., 2018) + clinical education (blake et al., 2018) + under-graduate medical education a not-provided, b provided   time of responding to queries   the time that it took a librarian to respond to the educational-clinical questions influenced the intended learning of trainees. some of the information services were designed to provide the needed evidence based information at the educational session itself or on the same day (aitken et al., 2011; brian et al., 2018; gibbons & werner, 2019; yaeger & kelly, 2014). in other studies, the authors did not mention time expectations for receiving answers (herrmann et al., 2017; santos & mariano, 2014; weaver, 2011). with information services that provided online access to questions and answers, librarians responded to queries in one to three days (brian et al., 2018).   providing evidence based outputs   the preparation of evidence based outputs for evidence requests is an essential part of an educational information service. evidence based output is a document of what a librarian has done to fulfill an evidence request. the evidence based output consists of three distinct parts: a) the clinical case presentation of the patient, b) a record of what keywords and search strategies were used for retrieving evidence, and c) the retrieved search results, which may include the abstracts and full text. each information service examined presented at least one aspect of the outputs, but a service with all these outputs better supports the educational needs. the purpose of preparing outputs is to provide a documented record for what librarians do, thereby helping trainees and educators learn to better perform their own search for retrieving needed evidence. preparing an evidence based output for each request of clinical evidence is time-consuming for the librarian, but educators and learners then have the information documented for further learning and later referrals, as well as evidence based data to deposit in local evidence based databases for future educational purposes (see table 4).   effects on trainee learning   in the field of medical education, ebeis enhanced learners’ understanding of evidence based practice in medicine (blake et al., 2018; brian et al., 2018; yaeger & kelly, 2014). after learning about ebm resources (blake et al., 2018; brian et al., 2018; gibbons & werner, 2019), the evidence retrieval behaviour of medical students shifted to more reliable databases for finding answers to clinical questions (aitken et al., 2011). the evidence based searching skills of learners were strengthened and learners were able to formulate more meaningful evidence based searches (brian et al., 2018; herrmann et al., 2017; zeblisky et al., 2015). in addition, providing such information services meant learners were supplied up-to-date, high-quality information more quickly (brian et al., 2018; gibbons & werner, 2019), and enhanced the learning process (gibbons & werner, 2019). another practical effect of ebeis was saving time for learners in finding needed evidence (herrmann et al., 2017).   discussion   according to an analysis of the included studies, ebeis have been implemented in different types of teaching-related units (e.g., teaching hospitals), and in varied target settings (e.g., clinical rounds). in all educational settings, there is a need for learners to access evidence. ebeis were flexible in servicing different needs within their predetermined teaching programs. in this regard, information services can be implemented in different educational settings with diverse types of access and schedules of service delivery. consequently, changes in curricula and teaching programs that produce new information needs can be met with reciprocal revisions in the implementation plan of the information services.   it is noteworthy that some of the studied information services had unique procedures in their implementation, which were not executed in the other information services, and therefore were not categorized into identified characteristics as a part of this study. yaeger and kelly (2014) stated in their study that a pre-prepared summary of the patient’s clinical situation and current clinical management was provided for the librarian ahead of clinical meetings. this procedure helped the librarian to present in the meetings with more confidence, especially for librarians who are new to delivering ebeis.   in some circumstances, the librarian taught the trainees ebm principles and skills, including understanding and creating pico questions and designing a search strategy according to the pico structure, to help accomplish one of the prerequisites of using ebeis (aitken et al., 2011; zeblisky et al., 2015). in this regard, librarians in some of the information services collaborated with teaching teams to prepare educational materials for trainees. in such situations, librarians working in the clinical environment could provide more applicable materials than those excluded from clinical situations (blake et al., 2018). in this regard, safdari et al. (2018) found the types of educational roles and activities of health care librarians in teaching information literacy skills and evidence based practice principles to medical students, educators, and clinicians, especially in the location of clinics or via online training. such educational activities included developing interactive online tutorials, developing video instructions, and co-teaching in medical faculties. safadari et al. identified librarian participation methods in educational programs that can be considered in the development of ebeis.   another unique procedure which supported student learning was assigning a group of trainees with a set number to each librarian. the librarian monitored the students’ skill learning according to pre-determined learning objectives, and reciprocally, each student knew which librarian to contact when they encountered a learning problem (gillum et al., 2018).   figure 2 presents a schematic diagram of identified characteristics of information services within the sequence of program implementation. also, we included uncategorized characteristics of information services in the diagram, described in the previous paragraphs.   the main limitation of this study was differentiating between evidence based information services which were designed for clinical practice, medical education, or both simultaneously. in this respect, we tried to include studies which explained the implementation of an information service for supporting evidence for any type of educational procedure.   figure 2 schematic diagram of implementation characteristics of information services.   conclusion   we conducted this study to identify the structure of the implementation process of information services which supported evidence based medical education. after conducting search strategies in target databases and employing inclusion/exclusion criteria, we selected and analyzed 11 articles. information services which were studied in this review supported empirical knowledge for evidence based medical education at different levels of training and facilitated evidence based change in educational approaches. the summarized trend of implementing ebeis consisted of:   (1) schedule and type of access; (2) methods for receiving questions; (3) information service types; (4) response time; and (5) preparation of evidence based outputs.   on the basis of the implementation trends of information services being studied, an applicable ebeis based on the needs of each educational level can be designed.   according to the findings of the current review, we propose the following practical recommendations. first, a needs assessment of predefined users is a necessary prerequisite before designing a practical ebeis. based on the characteristics of stakeholders of information services, librarians can benefit from various needs assessment techniques. the characteristics and needs of stakeholders should determine the appropriate assessment technique that results in the most useful data. second, each educational level needs to have a specifically-designed information service separately. third, mediated searching can be used for undergraduate levels and consulting information services can be used for graduate or professional levels. fourth, types of data needed in the evidence based outputs depend on the needs of intended users. finally, more detailed evidence based outputs will fulfill more educational needs in the future.   it is also plausible to suggest future studies to compare the structure of evidence based information services which support clinical practice with information services that were developed for medical education, in order to identify additional characteristics of implementation trends of evidence based information services.   acknowledgements   this study is part of a ph.d. thesis supported by the iran university of medical sciences under grant no. ir.iums.fmd.rec_1396.9381623003.   author contributions   sedigheh khani: conceptualization, data curation, methodology, visualization, writing – original draft sirous panahi: conceptualization, data curation, methodology, writing – review & editing ali pirsalehi: conceptualization, writing – review & editing ata pourabbasi: conceptualization, writing – review & editing   references   aitken, e. m., powelson, s. e., reaume, r. d., & ghali, w. a. (2011). involving clinical librarians at the point of care: results of a controlled intervention. academic medicine, 86(12), 1508-1512. https://doi.org/10.1097/acm.0b013e31823595cd   amin, z., guajardo, j., wisniewski, w., bordage, g., tekian, a., & niederman, l. g. (2000). morning report: focus and methods over the past three decades. academic medicine, 75(10), s1-5. https://doi.org/10.1097/00001888-200010001-00002   bennett, n. l., davis, d. a., easterling, w. e., friedmann, p., green, j. s., koeppen, b. m., mazmanian, p. e., & waxman, h. s. (2000). continuing medical education: a new vision of the professional development of physicians. academic medicine, 75(12), 1167–1172. https://doi.org/10.1097/00001888-200012000-00007   blake, l., yang, f. m., brandon, h., wilson, b., & page, r. (2018). a clinical librarian embedded in medical education: patient-centered encounters for preclinical medical students. medical reference services quarterly, 37(1), 19-30. https://doi.org/10.1080/02763869.2018.1404384   brian, r., orlov, n., werner, d., martin, s. k., arora, v. m., & alkureishi, m. (2018). evaluating the impact of clinical librarians on clinical questions during inpatient rounds. journal of the medical library association, 106(2), 175-183. https://doi.org/10.5195/jmla.2018.254   burns, p. b., rohrich, r. j., & chung, k. c. (2011). the levels of evidence and their role in evidence-based medicine. plastic and reconstructive surgery, 128(1), 305-310. https://doi.org/10.1097/prs.0b013e318219c171     chessare, j. b. (1996). evidence-based medical education: the missing variable in the quality improvement equation. the joint commission journal on quality improvement, 22(4), 289-291. https://doi.org/10.1016/s1070-3241(16)30232-2   davies, p. (1999). what is evidence-based education? british journal of educational studies, 47(2), 108-121. https://doi.org/10.1111/1467-8527.00106   davies, s., herbert, p., wales, a., ritchie, k., wilson, s., dobie, l., & thain, a. (2017). knowledge into action supporting the implementation of evidence into practice in scotland. health information and libraries journal, 34(1), 74-85. https://doi.org/10.1111/hir.12159   djulbegovic, b., & guyatt, g. h. (2017). progress in evidence-based medicine: a quarter century on. lancet, 390(10092), 415-423. https://doi.org/10.1016/s0140-6736(16)31592-6   ebsco. (n.d.-a). information services. in library, information science & technology abstracts thesaurus. retrieved march 15, 2021, from http://www.libraryresearch.com/    ebsco. (n.d.-b). information services in education. in library, information science & technology abstracts thesaurus. retrieved march 15 2021, from http://www.libraryresearch.com/    emami, s. a. h., khankeh, h., karbasi motlagh, m., zarghi, n., & shirazi, m. (2019). exploring experience of iranian medical sciences educators about best evidence medical education: a content analysis. journal of education and health promotion, 8, 247-252. https://doi.org/10.4103/jehp.jehp_428_19   fennessy, g. (2001). knowledge management in evidence-based healthcare: issues raised when specialist information services search for the evidence. health informatics journal, 7(1), 4-7. https://doi.org/10.1177/146045820100700102   gibbons, p., & werner, d. a. (2019). embedded clinical librarianship: bringing medical reference services bedside. public services quarterly, 15(2), 169-175. https://doi.org/10.1080/15228959.2019.1583153   gillum, s., williams, n., herring, p., walton, d., & dexter, n. (2018). encouraging engagement with students and integrating librarians into the curriculum through a personal librarian program. medical reference services quarterly, 37(3), 266-275. https://doi.org/10.1080/02763869.2018.1477710   grant, m. j., & booth a. (2009). a typology of reviews: an analysis of 14 review types and associated methodologies. health information and libraries journal, 26, 91-108.  https://doi.org/10.1111/j.1471-1842.2009.00848.x   green, b. n., johnson, c. d., & adams a. (2006). writing narrative literature reviews for peer-reviewed journals: secrets of the trade. journal of chiropractic medicine, 5(3), 101-117. https://doi.org/10.1016/s0899-3467(07)60142-6   guyatt, g., cairns, j., churchill, d., cook, d., haynes, b., hirsh, j., irvine, j., levine, m., levine, m., nishikawa, j., sackett, d., brill-edwards, p., gerstein, h., gibson, j., jaeschke, r., kerigan, a., neville, a., panju, a., detsky, a., … enkin, m. (1992). evidence-based medicine: a new approach to teaching the practice of medicine. journal of the american medical association, 268(17), 2420-2425. https://doi.org/10.1001/jama.1992.03490170092032   haig, a., & dozier, m. (2003a). beme guide no 3: systematic searching for evidence in medical education—pt. 1: sources of information. medical teacher, 25(4), 352-363. https://doi.org/10.1080/0142159031000136815   haig, a., & dozier, m. (2003b). beme guide no. 3: systematic searching for evidence in medical education—pt. 2: constructing searches. medical teacher, 25(5), 463-484. https://doi.org/10.1080/01421590310001608667   harden, r. m., grant, j., buckley, g., & hart, i. r. (2000). best evidence medical education. advances in health sciences education: theory and practice, 5(1), 71-90. https://doi.org/10.1023/a:1009896431203   hart, i. r., & harden, r. m. (2000). best evidence medical education (beme): a plan for action. medical teacher, 22(2), 131-135. https://doi.org/10.1080/01421590078535   herrmann, l. e., winer, j. c., kern, j., keller, s., & pavuluri, p. (2017). integrating a clinical librarian to increase trainee application of evidence-based medicine on patient family-centered rounds. academic pediatrics, 17(3), 339-341. https://doi.org/10.1016/j.acap.2016.11.005   holland, j., & haynes, r. b. (2005). mcmaster premium literature service (plus): an evidence-based medicine information service delivered on the web. amia ... annual symposium proceedings / amia symposium, 2005, 340-344. https://www.ncbi.nlm.nih.gov/pmc/articles/pmc1560593   jordan, z., & porritt, k. (2004). evidence-based health information provision: development of an online consumer 'request for information' service. worldviews on evidence-based nursing, 1(4), 237-240. https://doi.org/10.1111/j.1524-475x.2004.04065.x   liberati, a., altman, d. g., tetzlaff, j., mulrow, c., gøtzsche, p. c., aioannidis, j. p., clarke, m., devereaux, p. j., kleijnen, j., & moher, d. (2009). the prisma statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. british medical journal, 339, b2700. https://doi.org/10.1136/bmj.b2700   mcgowan, j., hogg, w., rader, t., salzwedel, d., worster, d., cogo, e., & rowan, m. (2010). a rapid evidence-based service by librarians provided information to answer primary care clinical questions. health information and libraries journal, 27(1), 11-21. https://doi.org/10.1111/j.1471-1842.2009.00861.x   national minority aids council. (2015). program development. http://www.nmac.org/wp-content/uploads/2015/04/program-development.pdf   neves, k., & dooley, s. j. (2011). using libguides to offer library service to undergraduate medical students based on the case-oriented problem solving curriculum model. journal of the medical library association, 99(1), 94-97. https://doi.org/10.3163/1536-5050.99.1.017   o'hare, j. a. (2008). anatomy of the ward round. european journal of internal medicine, 19(5), 309-313. https://doi.org/10.1016/j.ejim.2007.09.016   onyura, b., légaré, f., baker, l., reeves, s., rosenfield, j., kitto, s., hodges b., silver i., curran v., armson h., & leslie, k. (2015). affordances of knowledge translation in medical education: a qualitative exploration of empirical knowledge use among medical educators. academic medicine, 90(4), 518-524. https://doi.org/10.1097/acm.0000000000000590   poirier, t., & behnen, e. (2014). where and how to search for evidence in the education literature: the wheel. american journal of pharmaceutical education, 78(4), 70-77. https://doi.org/10.5688/ajpe78470   reed, d., price, e. g., windish, d. m., wright, s. m., gozu, a., hsu, e. b., beach m. c., kern d., & bass, e. b. (2005). challenges in systematic reviews of educational intervention studies. annals of internal medicine, 142(12 pt. 2), 1080-1089. https://doi.org/10.7326/0003-4819-142-12_part_2-200506211-00008   sackett, d. l., rosenberg, w. m., gray, j. a., haynes, r. b. & richardson, w. s. (1996). evidence based medicine: what it is and what it isn't. british medical journal, 312(7023), 71-72. https://doi.org/10.1136/bmj.312.7023.71    safdari, r., ehtesham, h., & bahadori, l. (2018). highlighting a valuable dimension in health care librarianship: a systematic review. medical journal of the islamic republic of iran, 32, 1-7. https://doi.org/10.14196/mjiri.32.42   sandars, j., & patel, r. (2015). it's ok for you but maybe not for me: the challenge of putting medical education research findings and evidence into practice. education for primary care, 26(5), 289-292. https://doi.org/10.1080/14739879.2015.1079008   santos, m. a. a., & mariano, g. s. l. (2014). information professional at the point of care: clinical librarian service in neurocritical care. journal of philippine librarianship, 34, 61-69. https://journals.upd.edu.ph/index.php/jpl/article/view/4585   shortell, s. m., rundall, t. g., & hsu, j. (2007). improving patient care by linking evidence-based medicine and evidence-based management. jama, 298(6), 673-676. https://doi.org/10.1001/jama.298.6.673   sur, r. l., & dahm, p. (2011). history of evidence-based medicine. indian journal of urology: iju: journal of the urological society of india, 27(4), 487-489. https://doi.org/10.4103/0970-1591.91438   suttle, c. m., challinor, k. l., thompson, r. e., pesudovs, k., togher, l., chiavaroli, n., lee a., junghans b., stapleton f., watt k., & jalbert, i. (2015). attitudes and barriers to evidence-based practice in optometry educators. optometry and vision science, 92(4), 514-523. https://doi.org/10.1097/opx.0000000000000550   thomas, a., gruppen, l. d., vleuten, c. v. d., chilingaryan, g., amari, f., & steinert, y. (2019). use of evidence in health professions education: attitudes, practices, barriers and supports. medical teacher, 41(9), 1012-1022. https://doi.org/10.1080/0142159x.2019.1605161   weaver, d. (2011). enhancing resident morning report with “daily learning packages”. medical reference services quarterly, 30(4), 402-410. https://doi.org/10.1080/02763869.2011.609077   weggemans, m. m., dijk, b, v., dooijeweert, b. v., veenendaal, a. g., & cate, o. t. (2017). the postgraduate medical education pathway: an international comparison. gms journal for medical education, 34(5), 1-16. https://doi.org/10.3205/zma001140   wijnen-meijer, m., burdick, w., alofs, l., burgers, c., & cate, o. t. (2013). stages and transitions in medical education around the world: clarifying structures and terminology. medical teacher, 35(4), 301-307. https://doi.org/10.3109/0142159x.2012.746449   yaeger, l. h., & kelly, b. (2014). evidence-based medicine: medical librarians providing evidence at the point of care. missouri medicine, 111(5), 413-415. https://www.ncbi.nlm.nih.gov/pmc/articles/pmc6172079   zeblisky, k., birr, r. a., & guerrero, a. m. s. (2015). effecting change in an evidence-based medicine curriculum: librarians' role in a pediatric residency program. medical reference services quarterly, 34(3), 370-381. https://doi.org/10.1080/02763869.2015.1052702     microsoft word editorial.doc evidence based library and information practice 2007, 2:4  1 evidence based library and information practice     editorial    when evidence doesn’t work      lindsay glynn  editor‐in‐chief    public services librarian and instruction coordinator, health sciences library  memorial university of newfoundland  st. john’s, newfoundland, canada  e‐mail: lglynn@mun.ca      © 2007 glynn. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.        i was listening intently to a discussion on  the radio recently between newfoundland  and labrador’s minister of education and a  professor from memorial university’s math  department.  they were debating the  efficacy of the math curriculum in the  province’s school system.  as a parent of a  grade 3 student, i have my own thoughts on  how the curriculum is affecting kids’ math  skills (and their anxiety levels, but let’s not  go there).  the professor echoed the concern  that parents, teachers and students have  been expressing: quite simply, it’s not  working.  far too many children are failing  math and are struggling with the both the  content and pace of the required modules.   why am i telling you this? one particular  comment made by the minister of education  struck me.  she said that there was evidence  to suggest that this curriculum should work.   while i’m always delighted to see the  evidence based practice model being used,  particularly for the betterment of my kids’  education, it is dismaying to see that it is not  always applied well.  in this particular case,  evidence was collected from somewhere and  a decision was made to implement a new  math curriculum based on the gathered  evidence.  assuming that this truly was  good evidence upon which to base such a  decision, then i would have to concede that  the appropriate steps were taken up until  that point.  unfortunately, it appears that  the entire process stopped there.  as we  know, one of the most important  components of a thorough ebp‐based  implementation is an internal evaluation.   what might work somewhere else is not  guaranteed to work in another environment,  and it is essential to determine why an  implementation or intervention worked or  didn’t work.  it would seem, in this case,  that formal evaluations of the effectiveness  of the new math curriculum have not been  mailto:lglynn@mun.ca http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:4  2 performed and therefore, the powers that be  rely solely on the fact that it worked  somewhere else.  this is not evidence based  practice at its finest.    so, what happens when evidence doesn’t  work?  we try to figure out why it didn’t  work.  did we miss something in the critical  appraisals?  what is inherently different in  the population or system at hand? are there  other confounders in your environment that  you had not considered (i.e. time of year,  available resources, courses being offered,  etc.)? as pointed out in this issue’s  commentary, a good idea is to plan your  project with research and assessment in  mind.  not only will you be able to track the  various stages of implementation and  reactions to it, it will save you the time that  you may have taken weeks or months later  to retrospectively evaluate.  and, never to  let an opportunity be wasted, i would  welcome an article submission outlining an  evidence based implementation that didn’t  work.  if it doesn’t work, it doesn’t mean  that you have failed.  it means that there  was something you had not anticipated that  had a negative effect on your intervention.   we can all benefit from such information.    speaking of benefiting, i will take this  opportunity to bid a fond farewell to two of  eblip’s original editorial board members:  denise koufogiannakis and pam ryan.   although denise and pam have made  numerous contributions to evidence based  librarianship, their work on this journal has  arguably made the biggest footprint.  denise  co‐founded this journal and has worked  tirelessly to create an avenue for high  quality publishing in this subject area.  she  has passionately maintained the evidence  summaries for each issue – a task that has  required a great deal of both time and  expertise.  she created an excellent team of  writers with whom she works closely and  she consistently provides feedback to ensure  that first‐rate summaries are published in  every issue.  pam courageously agreed to  take on the task of production editor with  the first issue.  no one on the editorial  board had experience with the journal  publishing software, ojs, and pam was able  to calmly work out the bugs, respond to our  calls of frustration and panic, and she is  solely responsible for the final look of the  publication every 3 months.  she has kept  denise, alison and me on schedule and has  caught more last minute typos and  formatting issues than i could possibly  count.  both denise and pam have led the  journal to where it is today as a result of  their commitment, expertise, enthusiasm,  and sincere belief in what the board is trying  to accomplish through this journal.  on  behalf of the editorial board, the evidence  summary writers, the peer reviewers, the  copyeditors, and the readers, i thank them  both and wish them the best success in their  future endeavors.  thankfully, their future  endeavors include continued involvement  with this journal.  denise will continue her  work with classics evidence summaries,  which will be a semi‐regular feature, and  both denise and pam have joined the  editorial advisory team.    on that note, i would like to welcome two  new editorial board members. lorie kloda  has joined the board as the associate editor,  evidence summaries.  lorie has been  contributing to the journal as an evidence  summaries author.  she hails from mcgill  university where she is currently pursuing  her phd.  katrine mallan is assuming the  role of production editor.  katrine currently  works at the university of calgary as an  instruction librarian.  please join me in  welcoming both lorie and katrine.  they  may have big shoes to fill, but they have the  skills, enthusiasm and expertise to do so  seamlessly.      this is our last issue for the year, and it’s a  big one.  aside from 7 evidence summaries,  we are featuring 5 summaries of classic  evidence based library and information practice 2007, 2:4  3 articles.  many of you will, no doubt, be  familiar with these seminal papers and will  be interested to read how they fare today  and the impact that they have had on our  profession.  also in this issue are 3 original  research articles and one article outlining  how to create effective questions for surveys.   there is much information here to discuss  on your coffee break.      well, that’s year two under our belts.  have  a lovely holiday season and a very happy  and safe new year.  we’ll see you in 2008!        evidence based library and information practice evidence based library and information practice 2010, 5.1  evidence based library and information practice       article    using evidence based practice in lis education: results of a test of a communities of  practice model    joyce yukawa  assistant professor  st. catherine university  2004 randolph ave.  st. paul, minnesota 55105, united states of america  email: jyukawa@stkate.edu    received: 27 sept. 2009          accepted: 6 feb. 2010       2010 yukawa. this is an open access article distributed under the terms of the creative commons‐ attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐ sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original  work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is  redistributed under the same or similar license to this one.    abstract    objective ‐ this study investigated the use of a communities of practice (cop) model  for blended learning in library and information science (lis) graduate courses. the  purposes were to: (1) test the model’s efficacy in supporting student growth related to  core lis concepts, practices, professional identity, and leadership skills, and (2)  develop methods for formative and summative assessment using the model.    methods ‐ using design‐based research principles to guide the formative and  summative assessments, pre‐, mid‐, and post‐course questionnaires were constructed  to test the model and administered to students in three lis courses taught by the  author. participation was voluntary and anonymous. a total of 34 students completed  the three courses; response rate for the questionnaires ranged from 47% to 95%. the  pre‐course questionnaire addressed attitudes toward technology and the use of  technology for learning. the mid‐course questionnaire addressed strengths and  weaknesses of the course and suggestions for improvement. the post‐course  questionnaire addressed what students valued about their learning and any changes  in attitude toward technology for learning. data were analyzed on three levels. micro‐ level analysis addressed technological factors related to usability and participant skills  and attitudes. meso‐level analysis addressed social and pedagogical factors  influencing community learning. macro‐level analysis addressed cop learning  outcomes, namely, knowledge of core concepts and practices, and the development of  professional identity and leadership skills.   104 evidence based library and information practice 2010, 5.1  105   results ‐ the students can be characterized as adult learners who were neither early  nor late adopters of technology. at the micro‐level, responses indicate that the online  tools met high standards of usability and effectively supported online communication  and learning. moreover, the increase in positive attitudes toward the use of  technology for learning at the end of the course may indicate that an effective balance  between face‐to‐face and online media was achieved. at the meso‐level, students  valued learning in community for developing mutual respect, confidence building,  risk taking, deeper and more varied learning, learning with and from their peers, and  greater enjoyment in the classes. students found that the online environments were  useful for organizing the class objectives and subject matter, “staying connected”  between classes, sharing ideas, keeping track of their work, and preparing them for  future work in blended environments. at the macro‐level, the findings of the effects  on student growth related to core lis concepts, practices, professional identity and  leadership skills were inconclusive. however, students’ expressed a high regard for  the value of collaboration, and there were indications that the model supported  differentiated learning of professional knowledge and skills.    conclusion ‐ the findings strongly suggest that the use of the cop model had positive  effects on the learning process. students’ high regard for the value of collaboration  appears to be a clear effect of using the cop model. the assessment methods were  sufficient for testing the efficacy of most aspects of the model under the limited  conditions of this study. student responses led to refinements in both the model and  methods. this study contributes to social constructivist learning approaches and lis  curricular development by presenting an innovative model for supporting  professional growth among adult learners, as well as a conceptual framework to guide  evidence based practice. further testing and refinement of the model in other contexts  and by other educators are needed to ensure that the model is robust and broadly  applicable.     introduction    contemporary learning approaches emphasize  the importance of learner agency  (constructivism) and social learning (social  constructivism), building on dewey’s (1933)  influential work on learning through  communication and inquiry in a social context.  dewey’s ideas have continued to inform work  on experiential learning (kolb, 1984) and  learning through dialogue (edwards, 1991;  mercer, 1995; wells, 1999). vygotsky’s theory  connecting learning to the sociocultural  context (1978) and bruner’s (1990) work on  cultural psychology and education also form  part of the foundation of principles for active,  collaborative learning (larreamendy‐joerns &  leinhardt, 2006; salomon & perkins, 1998).       moreover, many library and information   science (lis) students are adult learners,  returning to school to pursue second careers.  adult learning is problem‐based, dialogic, and  focused on critical reflection for more effective  action (mezirow, 2000). new knowledge is  created from a base of prior knowledge and  experiences (gagne, 1985) and enacted  through an instructional design that requires  the integration of various types of knowledge,  intellectual skills, and cognitive strategies to  achieve the goals of a purposeful enterprise  (gagne & merrill, 1990). dialogue and social  interaction are “essential aspects of knowing a  domain” (larreamendy‐joerns & leinhardt,  2006), and professional discourse and patterns  of practice emerge through mutual  participation (wenger, 1998). social learning  approaches such as communities of practice  stress that learning is not only knowledge  acquisition, but also a process of identity  evidence based library and information practice 2010, 5.1  formation and empowerment (lave &  wenger, 1991).        with the increasing demand for online  learning, professional education is challenged  to provide virtual learning experiences  appropriate for adult learners. while online  environments provide tools for intellectual  and social discourse that support cognitive  development, social networking, and  community building (palloff & pratt, 1999;  preece, 2000), the primary means of online  communication today is typed text. this mode  of communication lacks the expressive  features of nonverbal communication and  voice tone, as well as a shared awareness of  context. thus, some studies stress the  importance of early face‐to‐face encounters  among students to promote bonding as a basis  for continued interaction and participation  online (haythornthwaite et al., 2000).  although text‐based online communication  may be superior to face‐to‐face for supporting  reflection (e.g., yukawa, 2006; garrison,  anderson & archer, 2000; larreamendy‐ joerns & leinhardt, 2006), writing cannot be  viewed in isolation from other factors in the  learning environment (mimirinis &  bhattacharya, 2007; tolmie & boyle, 2000;  ziegler, paulus, & woodside, 2006), nor is the  ability to review online communication  sufficient encouragement in and of itself to  ensure that critical reflection takes place (seale  & cann, 2000).    a potentially valuable approach for effective  professional education is blended learning in a  classroom community of practice (cop). this  paper describes an investigation of the use of a  cop model for blended learning in three lis  courses. the purposes were to: (1) test the  model’s efficacy in supporting student growth  related to core lis concepts, practices,  professional identity, and leadership skills;  and (2) develop methods for formative and  summative assessment using the model. the  paper begins with a description of the design  and implementation of the cop blended  learning model, then describes assessment  methods, and concludes with a discussion of  the findings and their implications.   communities of practice model for blended  learning    the origins of many contemporary studies of  communities of practice are lave and  wenger’s examinations of apprenticeship  learning (1991). to support blended learning  based on social learning principles, i  developed a model based on wenger’s (1998)  elaboration of the cop framework. while the  process of model building is beyond the scope  of this article and has been described  elsewhere (yukawa, 2010), this section  summarizes the essential characteristics of  wenger’s framework as i view them through  the lens of social constructivist and adult  learning theories.    communities of practice have three  dimensions – a community, a domain, and a  practice. community refers to the relationships  and interactions among members of a  professional group, domain refers to the  community’s focal issues, and practice consists  of the discourse, tools, methods, and skills  used and transformed by the community over  time (snyder, wenger, & de sousa briggs,  2004). grounding learning in these dimensions  serves to ensure that it is social, has a clear  disciplinary focus, and is focused on actual  practice.     cop learning incorporates three “modes of  belonging”: engagement, imagination, and  alignment (wenger, 1998, p. 237‐239). i  consider these as stages of learning, although  actual learning is more complex and iterative:  (1) engage: connect to one’s own prior  knowledge and experiences, and share  experiences, ideas, and competence over time;  (2) imagine: extend knowledge and experience  through critical reflection and creative  exploration; and (3) align: converge around  vision, goals, practices, and accountability to  each other. each of these stages is complex  and multidimensional. engagement addresses  the adult learner’s need to connect learning to  prior experience and established frames of  reference, as well as interact socially and  emotionally with others. imagination  encompasses the critical, creative, and  106 evidence based library and information practice 2010, 5.1  107 reflective thinking processes that lead to  questioning assumptions, imagining  alternatives, and transforming one’s  knowledge in response to new information  and ideas from others. alignment focuses the  learning process on a common vision,  solutions to shared problems, clear decision‐ making, and socially responsible action.     wenger’s framework also incorporates the  concept of dualities, perhaps the most  influential aspect of his work. he defines a  duality as a pair of elements that are reciprocal  and inseparable rather than oppositional  (wenger, 1998, p. 66‐69). one of his major  contributions was to reexamine ideas of  dichotomy through the use of four dualities  that address “the fundamental issues of  meaning, time, space, and power” in  community learning (wenger, 1998, p. 231): (1)  participation and reification, (2) the designed  and the emergent, (3) the local and the global,  and (4) identification and negotiability. my  analysis of the essence of these dualities can be  summarized as four creative tensions that  shape the learning experience: (1) negotiating  meaning: learning key concepts by connecting  to prior knowledge and experience, by  questioning assumptions and expanding one’s  knowledge, and by developing shared  meanings with others; (2) negotiating practice:  learning new practices by connecting them to  one’s own, expanding these practices through  exposure to the practices of others, and  aligning to best practices; (3) negotiating  expertise: sharing and building on one  another’s expertise and learning from real  world expertise; and (4) negotiating identity and  leadership: developing an identity in  relationship to the community, influencing  others, and taking leadership.     barab, makinster, and scheckler (2004)  propose a fifth duality or creative tension  important for online and blended learning:  face‐to‐face and online communication. herbert  clark’s (clark & brennan, 1991) theory of  common ground provides a basis for  understanding this creative tension through  the inhibiting and enabling factors  characteristic of different types of media. eight  “constraints,” or enabling factors, affect  communication efficiency (table 1). face‐to‐ face communication allows the greatest  efficiency because it provides the richest array  of enablers. wikis are subject to more  constraints than face‐to‐face but provide the  advantages of reviewability and revisability,  i.e., messages are recorded permanently for  review and reflection, but can also be revised.    effectively negotiating this creative tension is  fundamental to learning, as usability issues,  lack of technical skills, or negative attitudes  toward technology can inhibit communication  and potentially derail the blended learning  process at the outset.    table 1   clarkʹs communication constraints (enablers)  (yukawa, 2007; based on clark & brennan, 1991)  enabler  description  copresence  communicators share the same physical environment.  visibility  communicators are visible to each other.  audibility  communicators can hear each other.  cotemporality  communicators receive messages at roughly the same time they are produced.  simultaneity  communicators can send and receive at once and simultaneously.  sequentiality  communicatorsʹ conversation turn taking cannot get out of sequence.  reviewability  communicators can review each otherʹs messages (i.e., messages are  permanently recorded).  revisability  communicators can revise messages for each other (e.g., letters, email).  evidence based library and information practice 2010, 5.1  i developed a model for blended learning  based on wenger’s cop design framework  (wenger, 1998, p. 240) to address the  limitations to meaningful learning imposed by  online communication. the model (figure 1)  focuses on the four creative tensions that  shape community learning. these creative  tensions are articulated through the three  stages of learning (engagement, imagination,  alignment) and enacted through face‐to‐face  and online communication (the fifth creative  tension), requiring a negotiation of the benefits  and costs of each medium to meet learning  needs. the desired result of negotiating all five  creative tensions is the personal realization of  professional identity, principles, practices, and  leadership skills that help students become  respected and contributing members of the  profession.     fig. 1. cop learning processes in the blended classroom.        108 evidence based library and information practice 2010, 5.1  implementation of the model    guided by the model, i redesigned three  blended lis courses taught for the master of  library and information science (mlis)  program at st. catherine university during  the 15‐week winter 2009 semester: “reference  & online services,” a core course that met for  3 hours weekly, and the elective courses,  “library user instruction” and “information  seeking behavior,” which met on alternate  weekends for 5 hour sessions. the model was  iteratively tested and refined during  implementation. in this section, i first address  the design of the online environment, followed  by brief descriptions of the face‐to‐face and  online strategies used to enact the creative  tensions for learning. (for a fuller discussion  of the design and implementation process, see  yukawa, 2010.)    design of the online learning environment    my goals in designing the learning  environment were to ensure that the online  tools: (1) met usability guidelines – ease of  navigation, clear information design, and  dialogue support (preece, 2000), (2) were  integral to meeting learning objectives, and (3)  provided support for the cop learning  processes (figure 1). the online tools used in  the three courses included a wiki as the main  course website (pbworks, formerly pbwiki,  http://pbworks.com), a lightweight chat  application for synchronous communication  (yaplet, http://www.yaplet.com), blogs for  individual reflections on learning, and a  course management system, moodle, for  grading (http://moodle.org). online dialogue  was supported through the use of the  comment feature on wiki pages, the chat  application, and the blogs.     the wiki provided a combination of structure,  functionality, and flexibility needed to create  course websites that support cop blended  learning. i designed the wiki to communicate  the core concepts, models of practice, and  course expectations in an easily navigable  format. the folders of wiki pages visually  presented the main course activities, e.g.,  about the course (syllabus and course  calendar), assignments and assessment, class  sessions (with lecture notes provided on wiki  pages in lieu of slide presentations or  handouts), class workshops, course  readings, feedback, and personal pages.    the two pages with the most essential  information at the beginning of the course  were the syllabus and the course calendar.  creating a wiki page syllabus easily supported  embedding images (such as a course concept  map) and hyperlinks to other wiki pages and  external web pages. the ability to create a  customized course calendar also enhanced  usability. through a table that linked to pages  with weekly information about topics to be  covered, reading lists, and instructions for  assignments due, the calendar provided on a  single page a week‐by‐week visual index to  the most essential instructor‐created pages  (figure 2).  the ability to hyperlink allowed me to create  multiple pathways for finding pages  depending on context, increasing ease of  access. for example, readings could be  organized by topic in the course readings  folder, but were also accessible by date from  the calendar page and by session from the  class session pages. within this basic  organization, content was added to wiki pages  by the students and myself as one of the  means for negotiating learning.    negotiating identity and leadership    of the four substantive creative tensions,  negotiating identity and leadership is the most  encompassing and challenging. i believed that  engaging students to find personal meaning in  the focus of the course and to commit to  collaborative learning were the most critical  early tasks. my strategies included building an  atmosphere of trust and a climate of  collaboration in order to encourage confidence  building, risk taking, and relationship  building. i encouraged students to create  personal wiki pages and post many of their  assignments (such a learning reflections) on  the wiki. i also required them to conduct  109 http://pbworks.com/ http://www.yaplet.com/ http://moodle.org/ evidence based library and information practice 2010, 5.1    fig. 2. example of a course calendar.    group work and aspects of team projects on  the wiki (not graded). these uses of the wiki  were intended in part to foster identification  with the class community and provide  opportunities to share expertise, collaborate,  inspire each other, and take leadership.    negotiating the meaning of core concepts    the core concepts of a subject are the basic  building blocks of knowledge. students  initially engaged with core concepts through  the wiki content – syllabus, readings and  resources, class session notes, assignment  instructions, and assessment rubrics.  throughout the semester, negotiations of  meaning were supported face‐to‐face through  lectures, dialogue, collaborative questioning  and synthesis, group work, team projects, and  student presentations with class discussion.  during the process of negotiating core  concepts, the wiki evolved as the students and  i added new content. for example, in  reference & online services, in two large  groups, students debated the advantages and  disadvantages of encyclopaedia britannica  versus wikipedia, considering uses, knowledge  creation processes, and criteria for accuracy  and authority. the groups used wiki pages to  brainstorm ideas, develop convincing  arguments, and present their cases.    negotiating models of practice    negotiating models of practice involves an  iterative process of identifying one’s own  models from existing knowledge and practical  experience, exposure to expert models,  reflection on practice, and model building and  refinement. one example from reference &  online services is how the views that students  held of information seeking behavior in their  own lives changed with exposure to  kuhlthau’s model of information seeking  (2004). while a number of models of  information seeking behavior exist (wilson,  1999), i chose to introduce kuhlthau’s model  to students new to the lis discipline, as a  framework for reflecting on their own  information seeking processes. other  110 evidence based library and information practice 2010, 5.1  examples are simulations and problem‐ and  project‐based learning, such as the teaching  demonstrations developed and conducted by  students in library user instruction. these  strategies were used to help students build  and refine their own models, coordinate group  activity, self‐assess, and align to standards of  good practice as articulated by experts and  practicing professionals.    negotiating expertise    strategies for negotiating expertise included  sharing stories from my professional life,  inviting stories from students, and exposure to  professionals in action (e.g., guest speakers  and site observations). theoretical discussions  were often linked to real world problems and  events. contributions to the class wikis  supported sharing of expertise and added to  the community knowledge base. through  collaboration on group projects, students  developed shared standards, practices, and  values around centers of expertise within the  group.     formative and summative assessment  methods    the formative and summative assessment  methods used in this study were derived from  design‐based research (dbr). dbr involves  the integration of design, theory, research, and  practice in naturalistic settings (bell, 2004). the  design and the hypotheses upon which it is  based are continually refined during  implementation until a more coherent theory  emerges, embodied in the design. sandoval  (2004) calls these hypotheses “embodied  conjectures” (p. 213). in this study, the  embodied conjectures are the cop learning  processes (figure 1).     data collected for dbr studies typically  include a comprehensive record of the design  process, how learning was organized and  supported, and data about the learning  processes and outcomes (borko, liston, &  whitcomb, 2007). while previous work has  examined the design and implementation of  the cop blended learning model in greater  depth (yukawa, 2010), this paper focuses on  cop learning processes and outcomes. the  primary data sources for this study were  anonymous student responses to closed‐ended  and open‐ended questions on pre‐, mid‐, and  post‐course questionnaires. the pre‐course  questionnaire1 addressed attitudes toward  technology and the use of technology for  learning and daily life. the mid‐course  questionnaire addressed what had helped  students most and least in learning about the  subject of the course and what could be  improved. the post‐course questionnaire  addressed what students valued about their  learning related to the model, their attitudes  toward the technology used in the courses,  and any changes in attitude toward  technology for learning. informed consent was  received from all participants. content  analysis was used to analyze the responses to  the open‐ended questions. an additional data  source was an instructor’s journal with  documentation of the design process and  teaching reflections.     the number of participants and the response  rates on the questionnaires varied  considerably, influenced by an unusually high  number of course withdrawals that resulted  from setbacks in the mlis program’s progress  toward initial accreditation by the american  library association. of the 42 students  enrolled in the courses, 34 completed the  classes. thirty‐three students completed the  pre‐course questionnaire (79% response rate),  40 students completed the mid‐course  questionnaire (95% response rate), and 16  students completed the post‐course  questionnaire (47% response rate). while the  small number of participants, the change in  student population over the semester, and the  variation in response rates limit the ability to  draw firm conclusions from the data, data  across the three courses are consistent.     data were analyzed on three levels: micro,  meso, and macro. micro‐level analysis  examines the use of online tools to support the  fifth creative tension – negotiating face‐to‐face  and online media. the questions asked at this  level are: (1) do students have basic fluency  111 evidence based library and information practice 2010, 5.1  112 with the online tools? (2) what are students’  attitudes toward the online tools? (3) how  well do the online tools meet usability  guidelines? (4) how well do the online tools  support interaction? (5) how well do the  online tools support learning?     meso‐level analysis addresses social and  pedagogical factors influencing social  constructivist learning that are primarily  associated with the stages of learning –  engagement, imagination, and alignment. the  questions considered at this level included:  what evidence exists that shows that students  shared experiences, ideas, and competence  over time (engagement); extended experience  through reflection and exploration  (imagination); and converged around vision,  goals, practices, and accountability to each  other (alignment)?     macro‐level analysis attempts to ascertain  whether or not students achieved the goals of  the learning experience through examining  student perceptions of their learning in two  interrelated ways: (1) as related to course  learning objectives and (2) as cop learning  outcomes, namely, knowledge of core  concepts, acquisition of new practices, and the  development of professional identity and  leadership skills.     the strengths of this study are that: (1) it  presents a realistic view of learning as the  result of a complex interaction between  educational intervention, learner attitudes and  needs, personal histories, and situational  contexts; (2) it recognizes that the educational  intervention, the cop blended learning model,  is itself an outcome; and (3) it provides reports  from the insider’s intimate knowledge of the  design process, the students, and teaching and  learning in context. dbr may in fact be more  rigorous than experimental research at  aligning theory, treatments, and measurement  in complex classroom situations (hoadley,  2004). the study’s major weaknesses are: (1)  the threat to rigor of a single individual as  designer, researcher, and implementer; (2) the  difficulty of ascertaining whether outcomes  result from the design rather than other factors  in the learning environment; (3) limitations of  the study’s population – the small number of  classes and participants, and the change in  student population over the semester; and (4)  the variation in response rates to the  questionnaires.      0 1 2 3 4 5 6 7 8 9 20-24 25-29 30-34 35-39 40-44 45-49 over 50 age groups   fig. 3. age of students.          evidence based library and information practice 2010, 5.1  table 2  student attitudes toward new technologies: pre‐course questionnaire  n=33  question 15. which of the following best describes you?1  responses  percent  i love new technologies and am among the first to experiment with and  use them.  0  0  i like new technologies and use them before most people i know.  4  12%  i usually use new technologies when most people i know do.  25  76%  i am usually one of the last people i know to use new technologies.  4  12%  i am skeptical of new technologies and use them only when i have to.  0  0    findings    this section begins with a description of the  study participants. it then presents findings on  the effectiveness of the cop model for  fostering student learning, using the three  levels of analysis.   profile of the students    while the overwhelming majority of the  students were female, in order to preclude  identification of individual students, a gender  question was not included on the  questionnaires. while the overall age range  was from the early 20s to over 50 years of age,  the majority of students were in the middle  age ranges (figure 3).    question 7 of the pre‐course questionnaire  asked students which technologies they used,  from a list of 31 common as well as emerging  technologies. more than half used  technologies such as presentation software,  spreadsheets, multimedia downloading, text  messaging, and social networking websites,  while less than 1% had used video creation  software, webcasts, or online virtual worlds.    regarding their attitudes toward new  technologies, the responses on the pre‐course  questionnaire indicate that the students were  neither strongly positive nor negative about  using new technologies, with the vast majority  following the trends of their peers (table 2).     in sum, the students in the three courses can   be characterized as adult learners who were  neither early nor late adopters of technology  (rogers, 2003).         micro‐level analysis    micro‐level analysis examines the use of  online tools to support the fifth creative  tension – negotiating face‐to‐face and online  media. questions addressed were: do  students have basic fluency with the online  tools? what are students’ attitudes toward the  online tools? how well do the online tools  meet usability guidelines – ease of navigation,  clear information design, and dialogue  support? how well do the online tools support  interaction? how well do the online tools  support learning? micro‐level analysis is  fundamental, as usability issues, lack of  technical skills, or negative attitudes toward  technology can inhibit communication and  potentially derail the blended learning process  at the outset.    at the start of the semester, nearly half of the  students had never contributed content to  either wikis or blogs (table 3).     students new to wikis expressed initial  trepidation but also confidence and a sense of  accomplishment after use. at the end of the  semester, on a scale from strongly disagree (1)  to strongly agree (5), the mean response of all  16 respondents for a rating of pbwiki as  helpful to learning was 4.75.         113 evidence based library and information practice 2010, 5.1  table 3   student use of social networking tools: pre‐course questionnaire  n=32  use of social networking  tools1  never  once  per  year  several  times a  year  monthly  weekly  several  times  per  week  daily  question 9a. how often do  you contribute content to  wikis (e.g., wikipedia) (for  school, work, or  recreation)?  15  (47%)  0  (0%)  5  (16%)  2  (6%)  10  (31%)  0  (0%)  0  (0%)  question 9b. how often do  you contribute content to  blogs (for school, work, or  recreation)?  15  (47%)  3  (9%)  8  (25%)  3  (9%)  3  (9%)  0  (0%)  0  (0%)    thirteen of the 16 respondents to the final  questionnaire commented on the usefulness of  the online tools (table 4). these comments  indicate that the tools were easy to navigate,  provided a clear information design, and  allowed students to stay connected between  classes. the responses also indicate the varied  ways the students used the tools for learning,  as well as comments for improvement.    on the final questionnaire, in answer to the  question “in what ways did using both face‐ to‐face and online environments not help you  learn? what improvements would you  suggest?”, student responses appeared to  confirm the overall usefulness of the online  tools. four respondents answered that they  had no suggestions for improvement and  eight respondents did not answer. four  respondents provided suggestions for  improvement, which are included in table 4.    students also provided valuable information  to aid in negotiating face‐to‐face and online  media. in answering the question, “in what  ways did using both face‐to‐face and online  environments help you learn?”, students  identified preferences for each type of media.  in general, they stated that face‐to‐face was  essential for creating a collaborative  environment, getting to know each other,  learning from presentations and  demonstrations, and deepening their  understanding of ideas initially discussed  online. online environments were useful for  “staying connected” in biweekly classes,  submitting assignments, keeping track of one’s  own work, balancing the workload, and  submitting peer assessments.     several questions related to the use of  technology for learning were repeated on the  pre‐ and post‐course questionnaires to  ascertain any changes in attitude. the use of  the technology tools seems to have positively  affected attitudes toward technology for  learning (table 5). the percentage of  respondents who strongly agreed or agreed  with the statement, “i get more actively  involved in courses that use information  technology,” increased from 48% in the pre‐ course questionnaire to 82% in the post‐course  questionnaire, with the mean score rising from  3.48 to 4.00. similarly, the percentage of  respondents who strongly agreed or agreed  with the statement, “the use of information  technology in my courses improves my  learning,” increased from 69% to 88%, with the  mean score rising from 3.81 to 4.13.        114 evidence based library and information practice 2010, 5.1  table 4  usefulness of online tools for learning: post‐course questionnaire  n=16  course  student responses  (r=anonymous respondent number)  it is a great approximation of the “real” world. since we will very likely be serving  patrons in all of these environments, it is good to practice in a multitude of settings.  (r9)  one aspect i really liked was having access to the lecture notes before class via pb  wiki. i like to read ahead. i found this useful when addressing topics i might have  otherwise struggled with. (r12)  i think the scope of coverage was very helpful and interesting, and [i hope we have]  the possibility to revisit these pages for future reference. i know i will if it is possible ...  moodle was great for feedback ‐ wiki works fine otherwise. (r14)  this class made use of technology as a way to organize the class objectives and subject  matter more than any previous course. i found the combination of moodle and pbwiki  to be very effective. i liked to use the calendar on pbwiki to get an excellent overview  of each session’s lecture notes, readings and assignments all in one place. (r15)  reference &  online  services  i am also grateful for all of the wonderful examples and resources [the instructor]  provided for the class (including the use of this wiki) … i don’t think i have ever had a  class that provided such wonderful and extensive resources. what a gift! because of  this, i probably learned more in this course than in any other … i don’t have any  suggestions for improvement, except options for when technology isnʹt cooperating. i  appreciated the experience. (r16)  it made it easy to balance workload. i found using the focus questions online really  helped to keep everything in one location. it was the best “journal” i’ve ever kept and i  don’t do journals. this is one that will help me (i’m printing it off to keep) …  sometimes, i didn’t feel i had the technology down. i’m not tech savvy, so it was  difficult finding methods that i could use effectively and not look old fashioned. i  would have liked to have a little more exposure to some tech. (r2)  the wiki was structured in a very thoughtful way. it was easy to find the information  … i think some of the information on the wiki just never gets read. as students we  posted information but were not encouraged to view what others posted. (i  occasionally did as time allows, but then was wondering if i was “snooping”.) (r3)  i think these [face‐to‐face and online environments] compliment each other and help  keep us connected. i would have enjoyed more online sharing & communication. (r7)  i liked the pbwiki for following the calendar and syllabus … i didn’t like the pbwiki for  posting assignments and reviewing other people’s posts. (r8)  the face‐to‐face was an important part of this class, but because the class met every  other weekend the wiki was also a very important way to stay connected. (r10)  library  user  instruction  the wiki was very well done! it could be used as an example for other classes. it took a  while to get used to it, but once i had, it was very thorough. (r13)  the online environment made turning in assignments, peer evaluation, etc. much  easier and kept us connected to the class between sessions. (r1) information  seeking  behavior  i really liked the wiki page to keep track of my research!!!!! without it i would have  completely lost track of what i was doing and when … i canʹt think of anything [for  improvement]. (r4)        115 evidence based library and information practice 2010, 5.1  116 table 5  student attitudes toward information technology for learning: pre‐ and post‐comparison  key:     5 = strongly agree     4 = agree     3 = neutral     2 = disagree     1 = strongly disagree  pre‐course questionnaire: n = 33; post‐course questionnaire: n = 16   use of  information  technology  pre  5  post  5  pre  4  post  4  pre  3  post  3  pre  2  post  2  pre  1  post  1  pre  mean  post  mean  i get more  actively involved  in courses that  use information  technology.  12% 4  19 %  3  36 %  12  63 %  10  39 %  13  19 %  3  12 %  4  0%  0  0%  0  0%  0  3.48  4.00  the use of  information  technology in my  courses improves  my learning.  21% 7  25 %  4  48 %  16  63 %  10  24 %  8  13 %  2  6%  2  0%  0  0%  0  0%  0  3.81  4.13  information  technology makes  doing my course  activities more  convenient.  39% 13  75 %  12  55 %  18  25 %  4  9%  3  0%  0  0%  0  0%  0  0%  0  0%  0  4.28  4.75  i have less  incentive to  attend class when  materials from  course lectures  are available  online.  0%  0  0%  0  3%  1  0%  0  9%  3  0%  0  30 %  10  75 %  12  58 %  19  25 %  4  1.27  1.25    at the micro‐level of analysis, on the whole,  student responses indicate that the online tools  met high standards of usability and effectively  supported online communication and  learning. more positive attitudes toward the  use of technology for learning at the end of the  course may indicate that an effective balance  between face‐to‐face and online media was  achieved.     meso‐level analysis    meso‐level analysis addresses social and  pedagogical factors influencing social  constructivist learning that are primarily  associated with the stages of learning. the  questions considered were: what evidence  exists that shows that students shared  experiences, ideas, and competence over time  (engagement); extended experience through  reflection and exploration (imagination); and  converged around vision, goals, practices, and  accountability to each other (alignment)? this  level of analysis draws attention to individual  factors such as learning styles and individual  barriers to communication or collaborative  learning, as well as factors in the learning  environment such as norms of communication  and facilitation strategies.     in response to open‐ended questions on the  post‐course questionnaires, all 16 respondents  provided evidence that engagement,  imagination, and alignment were part of their  collective learning experiences. table 6  summarizes aspects identified by students  with the frequencies that the items   evidence based library and information practice 2010, 5.1  table 6  student views on community learning: post‐course questionnaire  n=16  aspect     response  frequency   sample responses (r=anonymous respondent number)  engagement: sharing  collegial  atmosphere   2  classroom environment was comfortable and conducive to learning.  (r14)  diverse  perspectives   2  it was really valuable to hear from several perspectives. (r1)  self confidence  and comfort  with peers   2  i learned to be less shy about asking for input and feedback from the  professor and to not be nervous to ask others for help in getting the  data collection done. (r6)  enjoyable  learning   2  the community style of the class really made five hours go by very  quickly. (r1)  encourages  collaboration   1  this is a career change for me. my current career is competitive not  collaborative. working with my peers really helped me to overcome  that obstacle. (r12)  encourages  respect   1  developed great respect for fellow students, both our differences and  common goals. (r7)  learning from  each other   4  i enjoy, and get more from, classes where there is a lot of discussion  on topics. loved working in teams or partners in assignments. i  “think” better and have better ideas when bouncing them off of  others. (r5)  imagination: extending  better problem  solving   1  [working with peers] was huge for me ‐ as many times the way a  classmate would approach solving a problem was in and of itself an  eye opener. i learned that there are as many ways to think about  solving a problem as there are problems to be solved. (r9)   broaden one’s  perspective on  issues   3  because so many of my classmates have actual library experience, it  was great to hear real life examples of concepts we were learning.  (r15)  new ideas for  completing  course projects   2  it was helpful to see what other people were doing and how they  were approaching their instructional sessions ‐ it gave me a good idea  of how i wanted to approach mine. (r10)  feedback on  course projects   1  i really liked all the additional help i received on my project and all  the insight people had with my area. it made me understand what i  didn’t explain well enough for others to understand the subject. (r4)  alignment: converging  convergence  around best  practices   1  there was a lot more discussion and idea sharing in this class. it was  more than a traditional class with a huge project, small groups, etc.  the whole class worked toward an end result (best practice) and each  assignment was incorporated into that final event. this caused me to  focus specifically on parts and whole at the same time in such a depth  that i was able to keep up and really take hold of the information  more completely. (r2)  discussion  focus led by  students   1  this class felt very “organic.” there was a structure and a plan, but it  also felt flexible, so that class discussions and topics could naturally  evolve. i appreciated that we, as students, could focus on the areas  that were important to us. (r11)      117 evidence based library and information practice 2010, 5.1  were mentioned in student responses,  accompanied by a sample response to indicate  how the information was coded, labeled by  anonymous respondent number. while the  opinions of the other 53% of students who did  not complete the final questionnaire could  paint a different picture, the views of the 16  respondents are consistent across the three  courses. these responses also indicate that  students valued community learning for  developing mutual respect, confidence  building, deeper and more varied learning,  collaborative learning, and greater enjoyment  in the class.     one comment in particular clearly indicates  the value of community learning for this  student: “i really enjoyed working with my peers  and getting to know their searching styles and  communication styles – i think i learned as much  from them as the instructor.”     on the final questionnaire, in answer to the  question “in what ways did working with  your peers not help you learn? what  improvements would you suggest?”, student  responses appeared to confirm the value of  community learning. five respondents stated  that there were no disadvantages, eight  respondents did not answer, and one noted, “i  can only say that i wish i had gotten to work with a  larger variety of the students. there are a few that i  never worked with at all.”     one student’s response on the final  questionnaire indicates a recognition that the  sense of community that developed was at  least in part due to the cop design as an  educational intervention:     my favorite part of this course … was the  wonderful mix of personalities we had in  the room. i really appreciated that  everyone was encouraged to participate  and that any input was respected and  valued. there was a real sense of  community that is not found in most  classes. i think it was deliberately crafted  and that it was simply brilliant.”    as noted previously, engaging students to   commit to collaborative learning necessitates  building an atmosphere of trust and a climate  of collaboration in order to encourage  confidence building, risk taking, and  relationship building. as i implemented the  model, i realized the critical importance of  facilitation strategies for learner‐centered and  social constructivist teaching approaches (cf.  savin‐baden, 2007). selected student responses  from the three classes to the question, “in  what ways did your instructor’s teaching style  and methods help you learn?”, indicate that  these cop‐based facilitation strategies  achieved some success:     • you were very collaborative/inclusive,  focused, positive, and genuinely interested  in topic and students. (reference &  online services, respondent 14)  • allowed me to focus on specific areas that  were of interest to me, fostered thought‐ provoking class discussions, encouraged  class members to talk to each other and  learn from each other, promoted self‐ discovery. (library user instruction,  respondent 11)  • she really facilitated the learning process  in a very collegial manner. (information  seeking behavior, respondent 1)    at the same time, teaching to a cop design  requires changes and innovations that  challenge instructors to develop new skills  and students to be open to nontraditional  formats. several responses on the mid‐course  questionnaire indicated the need for more  lectures: (1) “we still lack the traditional lecture  about the subject” (library user instruction)  and (2) “perhaps more lecture about the different  parts of the overall research proposal: literature  review, research methods, research analysis, etc.”  (information seeking behavior). these may be  indications that my teaching methods were  ineffective or that the cop approach did not  meet with student expectations of a more  traditional format.    at the meso‐level of analysis, the findings  indicate that engagement, imagination, and  alignment were part of the students’ collective  learning experiences. students also valued  118 evidence based library and information practice 2010, 5.1  119 community learning for developing mutual  respect, confidence building, deeper and more  varied learning, collaborative learning, and  greater enjoyment in the classes.    macro‐level analysis    the micro‐level and meso‐level analyses have  established with reasonable certainty that the  students were able to effectively use the tools  for learning and that students experienced  engagement, imagination, and alignment in  their learning. we now turn to the macro‐level  analysis of whether or not the cop learning  outcomes were achieved.     this analysis examines student perceptions of  their learning in two interrelated ways: (1) as  related to course learning objectives and (2) as  cop learning outcomes, namely, knowledge of  core concepts, acquisition of new practices,  and the development of professional identity  and leadership skills. my course learning  objectives in the three courses were aligned to  the mlis program’s student learning  objectives and focused on disciplinary  knowledge and practices. consideration of  these is thus integral to an examination of the  outcomes of negotiating core concepts and  practices – the first and second creative  tensions of the cop model. the third and  fourth creative tensions (negotiating expertise  and identity/leadership) go beyond the course  learning objectives and were more challenging  to analyze. the reasons for this will be  discussed in the final sections of this article.    knowledge of core concepts    based on the assumption that highly valued  learning is a valid indicator of learning  achievement, the final questionnaire included  the open‐ended question, “what concepts did  you learn/develop in this course that were  most valuable to you?”    of the 14 students who completed reference  & online services, 6 completed the final  questionnaire (response rate of 43%).  responses to this question were coded by core  concepts included in the course learning  objectives that were articulated in the syllabus,  as well as one concept not included in the  syllabus, the structure of the learning. in their  answers, respondents identified from one to  five highly valued concepts. the numbers of  occurrences of these concepts are summarized  in table 7, with sample responses to indicate  how the data were coded.     of the core concepts articulated in the  syllabus, three were not mentioned by the  respondents – kuhlthau’s (2004) model of  information seeking, information literacy  instruction, and issues and trends in reference  services. this may indicate that these concepts  were not highly valued or were not learned.     of the 13 students who completed library  user instruction, 7 completed the final  questionnaire (response rate of 54%).  responses to this question were coded by core  concepts included in the course learning  objectives that were articulated in the syllabus.  in their answers, respondents identified from  one to three highly valued concepts. the  numbers of occurrences of these concepts are  summarized in table 8, with sample  responses.    the highly valued concepts mentioned by  respondents included all course learning  objectives in the syllabus. a number of  responses articulated the core concepts as a  process of instructional design, such as this  response: “the language of teaching: learning  styles, active learning & its importance,  goals/objectives, assessments & rubrics. how  to put this all together to design a teaching  program/lesson.”    of the 7 students who completed information  seeking behavior, 3 completed the final  questionnaire (response rate of 43%). the core  concepts articulated in the course learning  objectives included information behavior  concepts, models and theories of information  seeking behavior, and research methods. the  three respondents mentioned all of the core  concepts in their answers:  evidence based library and information practice 2010, 5.1  • qualitative research, the research process,  and professional presentations during this  course. (respondent 1)  • i am amazed at all the models associated  with qualitative techniques. (respondent  4)  • reviewed some research methodologies i  had previously learned and was  introduced to new ones. (respondent 6)    the fact that the highly valued concepts were  correlated with most of the core concepts  included in the course learning objectives is  not necessarily a comment about the efficacy  of the cop model in comparison to other  methods. at the least, it can be said that the  model probably did not interfere with student  achievement of the learning objectives.  however, what is revealed by this analysis is  that students varied in the concepts they most  valued from their learning. because adult  learners create new knowledge from a base of  previous experiences and worldviews, it is not  surprising that no two students were alike in  this regard. while this may happen in any  class, it may be an indication that negotiating  the creative tensions in a learning community  grounds the learning outcomes in a process of  individual identity formation and  empowerment. because each individual brings  a different set of assumptions, experiences,  expectations, and needs to the learning  experience, this process is unique to each  individual. the result appears to be  differentiated learning of professional  knowledge. this will be further examined in  the “discussion” section.    table 7  most valuable concepts learned in reference & online services: post‐course questionnaire  n=6  core concept   response  frequency  sample responses  (r=anonymous respondent number)  organization  of information  for access   3  how information is organized in resources. types of resources  available. (r5)  search and  retrieval   4  the concept of carefully observing what works in research and  retrieval and what does not was very valuable to me. with the  number of database searches involved, it was critical to learn  various steps and options to finish the exercises and bibliography  plan. (r16)  criteria for  evaluating  references  sources   2  the concept of authority in reference resources. that is okay to use  a resource with questionable authority (like wikipedia) as a  starting point. (r15)  reference  interview   2  reference interview. it was not something i had thought about  before ... the need to work with someone to figure out exactly  what they were trying to say. (r12)  reference  ethics and  values   2  i think that one of the most important concepts of ref librarianship  is that it is a service oriented profession that we are in … it is  professionally incumbent upon us to thoroughly research and  learn the best ways to “service” users. (r14)  structure of  learning   2  i appreciated most that the learning was structured so that we  looked at not just the technological aspects but the human  centered part of reference service. (r9)       120 evidence based library and information practice 2010, 5.1  table 8   most valuable concepts learned in library user instruction: post‐course questionnaire  n=7  core concept   response  frequency  sample responses  (r=anonymous respondent number)  information  literacy  concepts   2  before this class, i knew virtually nothing about information  literacy instruction. learning about this concept and how to  practice it has been incredibly valuable. (r11)  learning  theories   4  learning styles and the effect that has on successful teaching.  especially active learning, which has not been my preference, but  now i understand how effective it is and how it can also be used  for assessment. (r7)  instructional  methods   5  i’ve never taught anything‐‐so i learned a lot about teaching and  instruction in this course, including: the idea that people have  different learning styles‐‐and to incorporate as many as possible in  lessons. (r13)  instructional  planning and  assessment   5  building assessment into teaching [is] essential for evaluation. (r7)    development of professional practices    in order to ascertain practices learned or  developed in the courses, the final  questionnaire included the open‐ended  question, “what professional practices did you  learn/develop in this course that were most  valuable to you?”    in reference & online services, the six  respondents identified from one to four highly  valued practices in their answers. responses  to this question were coded by skills of  practice included in the course learning  objectives that were articulated in the syllabus,  as well as an additional category,  collaboration. the numbers of occurrences of  these practices are summarized in table 9,  with sample responses. of those articulated in  the syllabus, two areas were not mentioned by  the respondents – synthesis and presentation  of information for users, and basic information  literacy instruction.     in library user instruction, the practices  articulated in the course learning objectives  included reviewing the literature for evidence  of best practices in user education and  designing and implementing user education  sessions. four of the 7 respondents identified  specific aspects of designing and  implementing user education sessions as the  most valuable. no respondent mentioned the  literature review. in addition, 2 mentioned the  importance of collaboration for fostering  information literacy instruction, not included  in the course objectives.    in information seeking behavior, the practices  articulated in the course learning objectives  included conducting research into information  seeking behavior and presenting the results.  the three respondents mentioned these skills,  as well as the importance of research  collaboration:    • professional practices learned/developed  during this course include thinking about  approaching an information seeking  behavior research projects, how to design a  study that will effectively measure/answer  research objectives/questions.  additionally, i further developed  professional speaking skills, working in  groups, and critically thinking about the  literature. (respondent 1)  • how to conduct/setup an interview.  (respondent 4)  • i learned to incorporate research practices  into the library profession and how to  121 evidence based library and information practice 2010, 5.1  collaborate with others in a library setting  for research purposes. (respondent 6)    again, the fact that the highly valued practices  were correlated with most of the practices  included in the course learning objectives is  not necessarily an indication that they were  the result of the use of the cop model.  however, this analysis revealed that  collaboration, not included in the course  learning objectives, was highly valued by  some students in all three classes. this appears  to be an effect of using the cop model.     table 9  most valuable practices learned in reference & online services: post‐course questionnaire  n=6  professional  practice   response  frequency  sample responses  (r=anonymous respondent number)  basic search  skills   3  learning more in depth about resources and how best to search  databases. (r12)  evaluation of  reference  sources   1  evaluate, question information sources. (r5)  reference  interview  skills   5  reference interview skills. in addition to class discussions, the  hands‐on online reference chat exercise and reference observations  were very helpful. (r15)  collaboration   2  knowing our own strengths and developing rapport and  relationships with librarians who have other strengths so that, as a  team, we can best serve the people who come to us. (r9)      table 10  most valuable practices learned in library user instruction: post‐course questionnaire  n=7  professional  practice   response  frequency  sample responses  (r=anonymous respondent number)  developing  lesson plans   2  i increased my ability to plan lessons and to create learning  objectives. (r8)  developing  rubrics   1  i also learned how to develop and use rubrics well. (r2)  developing  assessment  measures   3  the necessity of assessment ‐ on a small scale and on a much  larger level. (r10)  use of  technology for  instruction   1  use of technology in developing instruction. (r3)  collaboration   2  collaboration ‐‐ how important it can be for information literacy  instruction, and ideas for fostering collaboration. (r11)          122 evidence based library and information practice 2010, 5.1  development of professional identity and  leadership skills    the fourth creative tension involves  developing an identity in relationship to the  community, influencing others, and taking  leadership. based on the assumption that the  acquisition of professional values is an  important dimension of professional identity,  the final questionnaire included the open‐ ended question, “what professional values  did you learn/develop in this course that were  most valuable to you?” student responses  across all courses indicate a range of  professional values addressed in the classes  (table 11). the emphasis on the value of  collaboration, as well as respect for diverse  perspectives, may indicate that the use of the  cop model encouraged a regard for  characteristics that foster community building.    table 11  most valuable professional values developed ‐ all courses: post‐course questionnaire  n=16  professional  value   response  frequency  sample responses  (r=anonymous respondent number)  collaboration   7  the most valuable professional value that was reinforced in this  class concerned learning from others. this course really  demonstrated the value of hearing different perspectives and how  these can benefit the work that we do. (r1)  respect for  diverse  perspectives   5  respecting and providing for different learning styles and levels;  encouraging and fostering lifelong learning and information  literacy in people of all ages; making instruction fun & stress‐free.  (r11)  service   5  we are at the service of users, so it is our obligation to be neutral,  non‐judgmental providers of information that they request. (r14)  integrity   2  reaffirmed the integrity of our profession. (r7)    table 12  most valuable identity and leadership skills developed ‐ all courses: post‐course questionnaire  n=16  leadership  skills   response  frequency  sample responses  (r=anonymous respondent number)  self  confidence   5  i learned that i could actually instruct a class! i had no experience  in instruction or teaching, and have been surprised to find that not  only can i actually do it, but that i enjoy it, too. the mini  demonstration and the teaching demonstration have been great  confidence builders. (r11)  collaboration   2  learned when to say something wasn’t quite right or to take the  initiative to guide a project to keep it on track. also, when to step  back and ask for assessment from others to make sure it was still  an effective lesson. (r2)  risk taking   1  interestingly, the strongest leadership skills i feel that i personally  have developed came from having to do the reference  observations. making contacts, meeting librarians, going outside  of my comfort zone. (r9)  flexibility   1  not sure what these would be except to ask and then pursue a  different path with my project than what was originally expected.  (r3)  123 evidence based library and information practice 2010, 5.1  in order to ascertain leadership skills  developed in the courses, the final  questionnaire included the open‐ended  question, “what leadership skills did you  learn/develop in this course that were most  valuable to you?” the respondents across  courses identified self confidence, risk taking,  flexibility, and collaboration skills as  important leadership skills (table 12).     five of the 16 respondents either did not  answer this question or stated that they did  not know how to answer it. students who had  difficulty identifying leadership skills  developed in the classes may have had a view  of leadership as “taking charge.” effective  leaders are more than decision makers. they  also demonstrate interpersonal effectiveness  and valued personal attributes, many of which  were evident in the students’ post‐course  questionnaire responses. appreciation of  diverse perspectives and the willingness to  learn from others are consistent themes in the  student responses, as well as desirable traits of  lis professionals and leaders (ammons‐ stephens et al., 2009).    in sum, the macro‐level analysis has shown  that the concepts and practices most valued by  the students at the end of the courses  corresponded to most of the core concepts and  practices stated in the syllabi, with one  exception: collaboration. in addition,  collaboration was mentioned most frequently  as the professional value most valued at the  end of the course. collaboration was also  mentioned twice as an important leadership  skill that had been developed. this emphasis  on collaboration, which had not been included  in the course learning objectives of any of the  courses, appears to be an effect of using the  cop model.    discussion    the purposes of this study were to: (1) test the  efficacy of a cop model for blended learning  in supporting student growth related to core  lis concepts, practices, professional identity,  and leadership skills; and (2) develop methods  for formative and summative assessment  using the model. this section discusses how  well these goals were achieved, insights  gained from the study, and directions for  future research using the cop model.    the aspects of the cop model tested in this  study were the three stages of learning and the  five creative tensions. the micro‐level analysis  has established with reasonable certainty that  the online tools met high standards of  usability and effectively supported online  communication and learning. this indicates  that the design of the course and the wiki  websites enabled a satisfactory negotiation of  face‐to‐face and online media.    the meso‐level analysis has indicated that  engagement, imagination, and alignment were  present in the students’ learning experiences  and resulted in mutual respect, confidence  building, deeper and more varied learning,  collaborative learning, and greater enjoyment  in the classes. this indicates that the course  design and facilitation strategies were  adequate to implement the model with regard  to the stages of learning.    the macro‐level analysis has shown that most  of the course learning objectives were  correlated with highly valued concepts and  practices learned or developed by the students  in the three courses. while it may be that the  negotiation of core concepts and models of  practice effectively supported students in  achieving the course learning objectives, firm  conclusions cannot be drawn from the data.  however, one important effect of the use of  the cop model appears to be an enhanced  regard for the value of collaboration. the data  gathered through the questionnaires were  insufficient to support definitive conclusions  about negotiating expertise or  identity/leadership. one reason may be that  the survey questions were designed to elicit  perceptions of individual learning and were  inappropriate for determining the outcomes of  reciprocal processes, such as shared expertise  or influence and leadership. such outcomes  may be better identified through observation  or group reports. as discussed previously,  another reason may be that students who had  124 evidence based library and information practice 2010, 5.1  difficulty identifying leadership skills  developed in the classes may have had a view  of leadership as “taking charge.”    a tentative finding that deserves further  research attention is that the cop model may  effectively support differentiated professional  learning through grounding the learning  outcomes in a process of identity formation  and empowerment. adult learners need to  create new knowledge from a base of previous  experiences and worldviews. because each  student brings different personal history,  assumptions, and needs to the learning  experience, the learning process is individual.  i suggest that meaningful, differentiated  learning about the profession was achieved by  students in this study, through dialogue,  exploration, critical reflection, negotiating  meanings, sharing expertise, collaborative  problem solving, and teamwork.     the demographics of the student population  in these three classes are not atypical of  students in lis programs in general – a wide  range of ages and experience with libraries  and technology, career changes from varied  professions, and differing expectations of their  education, among others. teachers at k‐12  levels have used differentiated learning to  serve academically diverse learners and  promote educational equity. tomlinson et al.  (2003) define differentiation as:    an approach to teaching in which teachers  proactively modify curricula, teaching  methods, resources, learning activities,  and student products to address the  diverse needs of individual students and  small groups of students to maximize the  learning opportunity for each student in a  classroom (p. 121).     learners’ readiness levels, interests, and  modes of learning are critical factors. the  implications for lis education are to recognize  the need for differentiation and to adapt  teaching and learning approaches for  differentiated learning. the cop model may  provide one alternative.    working with the model revealed new  dimensions of the first four creative tensions,  with implications for the design of future  courses. while i have viewed myself as a  learning facilitator and a co‐learner with the  students, my view of our roles has become  more distinct. to support negotiating the  meaning of core concepts and models of  practice, the instructor plays a large role in  designing for effective learning through  determining learning goals, course structure,  resource materials, activities, assignments, and  assessment, as well as careful website design.  students respond to the initial design,  bringing their own experience, expertise, and  values into the negotiation to achieve better  concepts and practices for effective action.  students play a larger role than the instructor  in negotiating expertise, identity, and  leadership. while this can be aided by design  (e.g., future courses will include leadership as  a core concept), the role of the instructor here  is based less on design than on fostering the  conditions for these negotiations to take place.     modeling may be one of the most effective  strategies for a cop approach, rooted as it is in  apprenticeship learning. this was affirmed by  one student response on the mid‐course  questionnaire: “[the instructor] is modeling a  fine example of information literacy  instruction; we are experiencing it first hand.  expectations are defined, readings are  appropriate and meaningful. many modes of  instruction included, feedback encouraged ...  it is easy to understand the concept when it is  modeled so clearly for us.” the implication is  that solely online learning environments may  be insufficient for developing classroom cops  and by extension the skills needed for a full  range of face‐to‐face and online professional  settings.    the formative and summative assessment  methods and instruments were sufficient for  testing the efficacy of most aspects of the cop  model. responses to the self‐perception  questionnaires provided data adequate for  conducting the micro‐ and meso‐levels of  analysis but did not provide data of sufficient  depth and breadth to examine cop learning  125 evidence based library and information practice 2010, 5.1  outcomes at the macro‐level of analysis,  particularly regarding the third and fourth  creative tensions. further iterations of design‐ based research are planned to develop a better  articulation of the model and an integrated set  of methods and tools for the analysis of data,  including additional sources such as site use  statistics, student‐generated wiki content, and  student products with self‐ and peer‐ assessments. these will be important for a  fuller examination at the meso and macro  levels of analysis.     an important difference for future analyses is  that the identity of students will be known,  allowing for a better understanding of how  differentiated professional learning occurs.  more probing questions will be asked about  the stages of learning. group reports will be  analyzed for evidence of the development of  shared expertise and leadership skills. other  research questions relate to possible  differences in results among the courses.    conclusion    the findings strongly indicate that the use of  the cop model had positive effects on the  learning process among students in three  graduate‐level lis courses. students valued  learning in community for developing mutual  respect, confidence building, risk taking,  deeper and more varied learning, learning  with and from their peers, and greater  enjoyment in the classes. while the findings of  the model’s effects on student growth related  to core lis concepts, practices, professional  identity, and leadership skills were suggestive  but not conclusive, students’ high regard for  the value of collaboration appears to be a clear  effect of using the cop model. moreover, there  were indications that the model supported  differentiated learning of professional  knowledge and skills.    the formative and summative assessment  methods were sufficient for testing the efficacy  of major aspects of the model under the  limited conditions of the study. the planning,  implementation, and assessment process has  led to a deeper understanding of how the  creative tensions operate in practice, as well as  refinements to the model and the strategies for  its successful implementation. this study  contributes to social constructivist learning  approaches and lis curricular development  by presenting an innovative model for  supporting professional growth among adult  learners, as well as a conceptual framework to  guide evidence based practice. further testing,  refinement, and use of the model in other  contexts and by other educators are needed to  ensure that the model is robust and broadly  applicable.      references      ammons‐stephens, s., cole, h. j., jenkins‐ gibbs, k., riehle, c. f., & weare, w.  h., jr. (2009). developing core  leadership competencies for the  library profession. library leadership  and management, 23(2), 63‐74.      barab, s. a., makinster, j. g., & scheckler, r.  (2004). designing system dualities:  characterizing an online professional  development community. in s. a.  barab, r. kling, & j.a. gray (eds.),  designing virtual communities in the  service of learning (pp. 53‐90).  cambridge: cambridge university  press.    bell, p. (2004). on the theoretical breadth of  design‐based research in education.  educational psychologist, 39(4), 243–253.    borko, h., liston, d., & whitcomb, j. a. (2007).  genres of empirical research in  teacher education. journal of teacher  education, 58(1), 3‐11.    bruner, j. (1990). acts of meaning. cambridge,    ma: harvard university press.  clark, h. & brennan, s.e. (1991). grounding in  communication. in l.e. resnick & j.m.  levine (eds.), perspectives on socially  shared cognition. washington, dc:  american psychological association.  126 evidence based library and information practice 2010, 5.1    dewey, j. (1938). experience and education. new  york: simon and schuster.    edwards, d. (1991). discourse and the  development of understanding in the  classroom. in o. boyd‐barrett, & e.  scanlon (eds.), computers and learning  (pp. 186‐204). wokingham, england:  addison‐wesley.    gagne, r. m. (1985). the conditions of learning  and theory of instruction. new york:  holt, rinehart and winston.    gagne, r. m. & merrill, m. d. (1990).  integrative goals for instructional  design. educational technology research  and development, 38(1), 23‐30.     garrison, d. r., anderson, t., & archer, w.  (2000). critical inquiry in a text‐based  environment: computer conferencing  in higher education. internet and  higher education, 2(2‐3), 87‐105.    haythornthwaite, c., kazmer, m. m., robins,  j., & shoemaker, s. (2000). community  development among distance learners:  temporal and technological  dimensions. journal of computer‐ mediated communication, 6(1).  retrieved from  http://jcmc.indiana.edu/vol6/issue1/ha ythornthwaite.html.     hoadley, c. m. (2004). methodological  alignment in design‐based research.  educational psychologist 39(4), 203–212.    kolb, d. a. (1984). experiential learning.  englewood cliffs, nj: prentice‐hall,  1984.    kuhlthau, c. (2004). seeking meaning: a process  approach to library and information  services. 2nd ed. westport, ct: libraries  unlimited/greenwood press.    larreamendy‐joerns, j., & leinhardt, g. (2006).  going the distance with online  education. review of educational  research, 76(4), 567‐605.    lave, j., & wenger, e. (1991). situated learning:  legitimate peripheral participation.  cambridge, ma: cambridge  university press.       mercer, n. (1995). the guided construction of  knowledge. tonawanda, ny:  multilingual matters.    mezirow, j. (2000). learning to think like an  adult: core concepts of transformation  theory. in j. mezirow (ed.), learning as  transformation (pp. 3‐34). san  francisco, ca: jossey‐bass.    mimirinis. m., & bhattacharya, m. (2007).  design of virtual learning  environments for deep learning.  journal of interactive learning research,  18(1), 55‐64.    palloff, r. m., & pratt, k. (1999). building  learning communities in cyberspace:  effective strategies for the online  classroom. san francisco, ca: jossey‐ bass.    palloff, r. m., & pratt, k. (2003). the virtual  student: a profile and guide to working  with online learners. san francisco, ca:  jossey‐bass.    preece, j. (2000). online communities: designing  usability and supporting sociability. new  york: john wiley.    rogers, e. m. (2003). diffusion of innovations. 5th  ed. new york: free press.     salomon, g., & perkins, d. (1998). individual  and social aspects of learning. review  of research in education, 23(1), 1‐24.     sandoval, w. a. (2004). developing learning  theory by refining conjectures  embodied in educational designs.  educational psychologist, 39(4), 213–223.      127 http://jcmc.indiana.edu/vol6/issue1/haythornthwaite.html http://jcmc.indiana.edu/vol6/issue1/haythornthwaite.html evidence based library and information practice 2010, 5.1  128 savin‐baden, m. (2007). a practical guide to   problem‐based learning online. london:  routledge.    seale, j. k., & cann, a. j. (2000). reflection on‐ line or off‐line: the role of learning  technologies in encouraging students  to reflect. computers and education,  34(3‐4), 309‐320    snyder, w.m., wenger, e., & de sousa briggs,  x. (2004). communities of practice in  government: leveraging knowledge  for performance. the public manager,  32(4), 17‐21.    tolmie, a., & boyle, j. (2000). factors  influencing the success of computer‐ mediated communication (cmc)  environments in university teaching:  a review and case study. computers  and education, 34, 119‐140.     tomlinson, c. a., brighton, c., hertberg, h.,  callahan, c. m., moon, t. r.,  brimijoin, k., conover, l. a., &  reynolds, r. (2003). differentiating  instruction in response to student  readiness, interest, and learning  profile in academically diverse  classrooms: a review of the  literature.” journal for the education of  the gifted, 27(2/3), 119‐145.    vygotsky, l. s. (1978). mind and society: the  development of higher mental processes.  cambridge, ma: harvard university  press.    wells, g. (1999). dialogic inquiry: toward a  sociocultural practice and theory of  education. cambridge: cambridge  university press.     wenger, e. (1998). communities of practice:  learning, meaning, and identity.  cambridge, ma: cambridge  university press.    wenger, etienne, richard mcdermott, and  william m. snyder. (2002). cultivating  communities of practice: a guide to  managing knowledge. cambridge, ma:  harvard business school press.    wilson, t.d. (1999). models in information  behaviour research. journal of  documentation, 55(3), 249‐270.     yukawa, j. (2006). co‐reflection in online  learning: collaborative critical  thinking as narrative. international  journal of computer‐supported  collaborative learning, 1(2), 203‐228.    yukawa, j. (2007). factors influencing online  communication style in   lis problem‐based learning. journal of  education for library and   information science, 48(1), 52‐63.    yukawa, j. (2010). communities of practice for  blended learning:   toward an integrated model for lis  education. journal of education for   library and information science, 51(1),  54‐75      ziegler, m., paulus, t., & woodside, m. (2006).  creating a climate of engagement in a  blended learning environment. journal  of interactive learning research, 17(3),  295‐318.    endnote    1 the survey questions were adapted from  students and information technology in  higher education: 2008 survey questionnaire,  created by and publicly available from the  ecar‐educause center for applied  research  (http://net.educause.edu/ir/library/pdf/si/esi0 8a.pdf).    http://net.educause.edu/ir/library/pdf/si/esi08a.pdf http://net.educause.edu/ir/library/pdf/si/esi08a.pdf microsoft word es_mckenna.doc evidence based library and information practice 2006, 1:3      73 evidence based library and information practice       evidence summary    the quality of academic library building improvements has a positive impact on  library usage   a review of:  shill, harold b. and shawn tonner. “does the building still matter? usage patterns in new,  expanded, and renovated libraries, 1995‐2002.” college & research libraries 65.2 (mar.  2004): 123‐150.     reviewed by:  julie mckenna  services assessment librarian, university of regina  regina, saskatchewan, canada  e‐mail: julie.mckenna@uregina.ca    received: 02 june 2006            accepted: 13 july 2006      © 2006 mckenna. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to measure the impact of  academic library facility improvements on  physical library usage.    design – the facility improvement data  used for this study were previously  collected through a 68‐item web survey for  the companion article “creating a better  place: physical improvements in academic  libraries, 1995‐2002” (shill and tonner). the  measurement of library usage was by exit  gate counts before and after library  improvements.    setting – american academic libraries in  which: facility improvement projects were  completed between 1995 and 2002, the  project space was not smaller than 20,000  square feet, the project space did not include  off‐site storage or non‐public space, and  gate‐count statistics from before and after  facility changes were available.    subjects – ninety of 384 identified academic  libraries were able to provide usable data on:  exit gate count, total circulation, in‐house  collection use, and reference transaction  data.    methods – the data collection was  undertaken in 2003 for the companion study  (shill and tonner). a population of 384  libraries potentially able to meet criteria for  the study was gathered and each library was  invited by e‐mail to complete a web‐based  survey. through this initial contact, 357  libraries were confirmed as meeting the  study criteria, and responses were received  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:3      74 from 182 of those providing a 51% overall  response rate.    respondents were asked about institutional  characteristics (public or private, carnegie  classification, etc.); project specific features  (year of completion, nature of project, etc.);  nature and extent of changes (seating,  wiring, hvac, etc.); presence of non‐library  services in the facility; collection  arrangements; before and after quality  changes in lighting, seating and a range of  services (as assessed by the survey  respondent); and before and after project  completion gate count usage statistics.  respondents were asked a set of eleven  questions each with a five‐point scale about  facility quality and librarian satisfaction  with the former and the changed facility.    a further criteria requirement of the  availability of pre‐ and post‐project gate  count was implemented, reducing the  number of libraries to be studied to 90.  facility usage changes were calculated by  subtracting the gate count total for the last  complete year pre‐project from the most  recent year gate count post project.     main results ‐ eighty percent of the 90  libraries reported increased gate count post‐ project, and 20 percent reported a decline in  usage. the median increase across the  libraries was 37.4 percent with 25.6 percent  of libraries experiencing a post‐project  increase of 100 percent or more. renovated  facilities were more likely to see usage  decline, but there was no statistically  significant difference in usage change  between renovated and new facilities.  libraries more recently upgraded saw  greater usage growth than those renovations  completed earlier in the study period,  although 75 percent of the facilities  continued to experience higher post‐project  usage levels. nearly all of the private  institutions (93.1%) experienced usage  increases and almost half experienced  growth of 100 percent or more.    no statistically significant relationship was  found between changes in post project  usage and:  • the proportion of facility space  allocated for library functions  • the physical location of the library  on campus  • the size of the library facility  • the level of degrees offered at the  institution  • the availability of wireless access  • the number of computers in the  instruction lab  • the number of public access  workstations  • a larger number of seats  • the number of group study rooms  • the shelving capacity, the use of  compact shelving or off‐site storage  • the presence of coffee or snack bars  • the presence of any non‐library  facilities    there was a statistically significant  correlation (pearson’s r) between increased  post project usage and:  • the institution type (public or  private) (p=.000)  • the number of data ports in the  facility (p=.005)  • the percent of wired seats (p=.034)    ten elements relating to improved quality  emerged as statistically significant in  relation to increased usage, although the  correlation for quality of artificial lighting  was not statistically significant (p=.162 n.s.).  the statistically significant correlations  (pearson’s r) between quality and increased  usage in order of strength of correlation  were: the quality of the instruction lab  (p=.000); layout (p=.001); public access  workstations (p=.006); natural lighting  (p=.007); user workspace (p=.008);  telecommunications infrastructure (p=.014);  evidence based library and information practice 2006, 1:3      75 overall ambience (p=.020); collection storage  (p=.026); heating, ventilating, and air‐ conditioning system (p=.026); and service  point locations (p=.038).    conclusion – this study confirmed that 80  percent of libraries experience usage  increase after a library improvement project.  the study revealed those investments that  cause increased use, and also found that a  number of variables previously predicted to  cause usage growth were not significant.  the study also found that quality of the  improvements, additions, and the building  are a significant driver of increased use. the  median 37.4 percent increase demonstrates  that, contrary to reports in the literature  (shill and tonner 460), overall library usage  is increasing in these institutions.    commentary    the future of ‘the library as place’ has been  the subject of discussion in the academic  library literature for more than 20 years  (shill and tonner 431), and the debate over  the future of the academic library as a  physical entity continues as millions of  dollars are spent each year on new design  and renovation. in order to undertake  effective planning of facilities, libraries need  evidence that will inform best decisions.  there has been a lack of validated empirical  evidence in the literature about the impact  of building design on the use of library  facilities. the information gathered by this  study and its companion are a significant  contribution to our understanding of the  academic library as a facility. this study  does not address the debate, but does  provide some empirical findings that could  be used for further discussion. it should be  noted that this study does not attempt to  measure cost‐benefit of investment in the  redesign of library facilities.    the study was undertaken in a thorough  and credible approach, yet the use of gate  counts as the dependent variable gave  concern for a variety of reasons. the report  of gate count is not a mandatory measure  for most consortia and associations. it is a  measure fraught with problems in  consistency of collection and comparability  among institutions, including such concerns  as measuring use in pass‐through facilities,  new systems in place following renovations,  differences in data collection, and sampling  techniques among others. shill and tonner  also failed to account for the increase in  student population at many of the  institutions studied during the period of  measurement, which could also account for  increased traffic. gate count may reveal how  many individuals have passed the entrance,  but it does not reveal how individuals are  using the library or whether student usage  of services and resources has changed. the  authors of the study recognize the need for  further research on a number of questions  such as the specific uses that students make  of academic libraries today and student  attitudes toward the library. this study thus  lacks the concerns and views of the  community about the libraries’ designs and  facility planning efforts. a community‐ centered library quality study approach  would have greatly enhanced this study.    the contribution of shill and tonner  through this study and its companion study  is significant to our understanding of the  academic library as place. the  comprehensiveness of their approach and  their analysis of results are apparent  throughout the 64 pages of the combined  articles.               evidence based library and information practice 2006, 1:3      76 work cited    shill, harold b. and shawn tonner.  “creating a better place: physical  improvements in academic libraries,  1995‐2002.” college & research  libraries 64.6 (nov. 2003): 431‐66.     404 not found evidence based library and information practice evidence based library and information practice 2011, 6.2 72 evidence based library and information practice evidence summary higher education librarians are comfortable and confident with their teaching responsibilities and pedagogical knowledge a review of: bewick, l., & corrall, s. (2010). developing librarians as teachers: a study of their pedagogical knowledge. journal of librarianship and information science, 42(2), 97-110. reviewed by: mê-linh lê health sciences librarian health sciences library, university of saskatchewan saskatoon, saskatchewan, canada email: me-linh.le@usask.ca received: 25 feb. 2011 accepted: 21 apr. 2011 2011 lê. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – the primary objective of this study was to gather quantitative information on the need, development, acquisition, and application of pedagogical knowledge by academic librarians with teaching responsibilities. design – online survey questionnaire. setting – higher-education (he) institutions (i.e., post-secondary institutions such as colleges and universities) in the united kingdom. subjects – subject librarians from 82 he libraries (one from each). methods – of the 191 he institutions in the united kingdom (determined via a nowunavailable directory), 137 supplied an online staff directory with contact information. one subject librarian from each he institution was contacted; librarians were selected from the online directory by taking a name systematically from a different point in each listing (i.e., first, second, third, etc). each librarian was sent an email that contained an introductory message as well as a link to the questionnaire. the online questionnaire was created using survey monkey and piloted before and after input. it employed mostly multiple-choice tick boxes as well as openended questions and comment boxes. the 35question survey questionnaire was developed in part through email interviews with two leading researchers in the field (identified via mailto:me-linh.le@usask.ca� evidence based library and information practice 2011, 6.2 73 the literature). responses were received from 82 librarians (60%). the answers were analyzed and cross-tabulated using spss. komogorov-smirnov tests were done to determine the significance of some results. open-ended questions and comment boxes were placed into categories using microsoft excel to identify patters and themes. main results – the 82 librarians who responded to the survey came from a wide variety of backgrounds: the majority were subject librarians from arts & humanities (31%), had spent more than ten years in their position (38%), worked full-time (71%), were members of pre-1992 he institutions (59%), and went by the job title of subject librarian (30%) (or a slight variation thereof). respondents indicated a significant amount of variation in terms of the number of hours spent preparing and teaching each week, ranging from 0 to 25 hours per week (ft staff) and 0 to 12 hours (pt staff). eleven librarians spent 40% or more of their time teaching. due to the high standard deviation (5.71), however, and the fact that many librarians indicated difficulty providing precise figures, these percentages should only be used as general estimates. the study found that librarians were involved in a variety of teaching settings, including onthe-spot support, writing materials, teaching small groups, conducting one-on-one instruction, and teaching large groups. evaluation was not indicated as a regular practice. with regard to pedagogical development, most training was of an informal nature learned on the job (75%) or through trial and error (61%). other training methods included short courses or training programs, conferences, peer-support groups, and committee work. the most valuable lessons librarians took from their training were awareness of different learning styles (37%), the use of new techniques (29%), and contribution of planning and preparing (27%). a kolmogorov-smirnov test was performed on the correlation between involvement in formal training and knowledge of designing learning activities or teaching and learning theories (97.4% and 100% likelihood of a significant impact, relatively). while the authors state in the article that a chi-square test was done, the original thesis upon which the article is based (available via http://dagda.shef.ac.uk/dissertations/200607/external/cox_laura_malib.pdf), notes that the test was not completed as more than 20% of the expected counts were less than five (a frequent limiting constraint of the test). the vast majority felt their work was either important or very important; that they were sufficiently qualified and knowledgeable; and that they were confident providing the sessions. responses were mixed when it came to determining whether additional training would improve their confidence or aid them in their teaching role. the majority of respondents also felt they had reasonable or extensive knowledge of topics related to their teaching roles (e.g., delivering sessions and information literacy). indeed, when questioned as to what could improve the effectiveness of their teaching only 36 (44%) responses were given: subject knowledge and e-learning/new technologies were indicated as the most helpful. librarians indicated that their preferred delivery methods when learning new skills were (in order of popularity): formal sessions, on-the-job development; short courses; and other options (e.g., increased library school training). the majority of respondents noted there were enough opportunities available for them to develop their teaching skills. when asked at the end for additional comments on teaching knowledge and development in general, responses were varied, although many offered suggestions for ways of improving the development and provision of teaching skills within the library profession as a whole. conclusion – the authors posit that information literacy instruction has become a major component of the job requirement of a subject librarian. survey results indicate that evidence based library and information practice 2011, 6.2 74 subject librarians recognize the value of pedagogical knowledge but question the importance of the theoretical aspects; spend a significant amount of time on various types of instruction; learn on-the-job; feel they are sufficiently knowledgeable in their role and feel confident providing instruction; and favour short-courses or formalized programs for continued learning. future studies would benefit from interviews, additional tools to track instruction details, and perhaps a larger scale. commentary this helpful study serves as further evidence of the growing role of librarians as teachers. its brief but thorough review of the literature acts as a good starting point for those unfamiliar with the development of information literacy, focusing particularly on the last decade. this research will be of use to those in an instruction coordinator position or involved in professional development at a he library. insights from the article will assist them in planning the content and delivery format for training sessions, workshops, and webinars. the article provides an overview of how subject librarians wish to view their instructional role, and what routes and tools might be used to expand on librarians’ knowledge of pedagogy, particularly with respect to how librarians wish to receive future training however, with some minor modifications to the study design, the resulting data might have been more generalizable and representative of the true state of pedagogical knowledge of librarians. one shortcoming is the relatively small sample size. the format and nature of an online survey with mostly quantitative answers (six questions required a written response that were not coded) is well suited for obtaining information from large groups of individuals. when one considers the heterogeneity of instruction in he institutions, a larger sample size would be beneficial to more accurately capture the current climate surrounding teaching. the authors note that they were limited by time and resource constraints, which may partially explain the small sample size. closely connected with the sample size is the randomization tool used to contact librarians. the article notes that names were taken from different places (first, second, third, etc.) on staff directories. no further information is given on the selection process, the likely disparities in how staff directories are laid out, or how the variable staff sizes of libraries were handled (i.e., how ‘high’ did the first, second, third, method go?). it is suggested that future studies of this nature use some type of online randomization tool to alleviate such issues. the article would have benefited by including its survey instrument. it is referred to frequently, is noted as a limitation of the study, and contained errors that led to difficulties answering certain questions. finally, while the validity tests used in this study are sufficient for a project this size, future studies that expand on its scale and scope should consider including additional measures to test the reliability and validity of the survey instrument. although these shortcomings limit the study slightly, the end results are still informative. it builds on the work of others in some areas but adds new knowledge to the acquisition method of pedagogical knowledge. while not emphasized, the survey provides hard numbers on the relative dearth of librarians involved in assessment. if instruction is to continue as a major aspect of the work of a librarian, assessment must be more broadly addressed. / evidence based library and information practice evidence based library and information practice using evidence in practice   reading ghosts: monitoring in-library usage of ‘unpopular’ resources   stacey astill senior library assistant keyll darree library learning education and development (lead) cabinet office, isle of man government braddan, isle of man email: stacey.astill@gov.im   jessica webb library assistant keyll darree library learning education and development (lead) cabinet office, isle of man government braddan, isle of man email: jessica.webb@gov.im   received: 15 july 2017    accepted: 31 oct. 2017      2017 astill and webb. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     setting   keyll darree library is situated opposite noble’s hospital in braddan, on the isle of man. it is the only health and social care library on the island. keyll darree library is responsible for supporting the entire department of health and social care, nursing and medical education departments, health and social care related charities, private care facilities, and any other groups with a need for these services.   not all library users are library members (with some using the facilities for reference purposes only, or mainly accessing the computers), and they vary widely in age and discipline. many of the most regular users are students actively engaged in a degree or other qualification, although this is often seasonal with peak usage around january, april, and november – tying in with exams, and essay deadlines.   problem   in 2010 library staff started to realise that they were removing items from the collection which library users would then claim they had regularly engaged with. this made no sense, as the record in heritage (our library management software) was always checked for loan statistics prior to the removal of any resources. it then came to light that some library members, especially students, had been using the books in the library to allow them to share more effectively, and thus there were no loan statistics.   when this was combined with the fact that not all library users were actually members, so were unable to physically borrow books (even if they were reading them in the library), and the issue of swiftly reducing budgets it was decided that we needed to capture these statistics.  at this time, our heritage library management software did not include a function for recording this data, thus, the team devised a method of creating a ‘dummy account’ for our ghosting procedure in order to work around this system limitation. this has now been rectified in the latest software update and the system now has a specific function for recording in-library usage. once we knew what was being used, we would be able to make more effective choices, and not have to replace books we had removed from the collection. these objectives have been met over the six years since implementation.   evidence   effectively, the aim of our procedure was to keep track of all resources being used, not just those on loan. this would mean that all relevant well used resources would be kept, ultimately ensuring our user needs were engaged, and our collection was relevant for them. we have termed this procedure ‘ghosting’. all library users (members and non-members) are asked to leave items they have used but are not borrowing on the tables. a member of staff collects these items twice daily and issues them to our dummy “writer ghost” account in heritage and returns them to the library. this ensures that we gather statistics for items used within the library as well as those borrowed by users.   the statistic gathering ghosting process was implemented in a variety of ways, as this was a big change for a lot of people. initially, staff put out signs asking library users to leave their books on the tables once they had finished using them, and highlighted the new policy during orientations. luckily natural instinct also played to our favour as many of our users were pleased at not having to tidy up.   staff also had to consider stealth ghosting. some users who felt untidy leaving books out (but weren’t dedicated enough to re-shelve) would leave piles of books on trolleys, shelves, and under cubbies in an effort to be tidier. we still find piles like this to this day, and now ghost these too.   initially we compiled the data collected from our ghosting procedure into yearly amounts; we then compared this to loan values for the comparative years. this general overview of total resource (from both print and audio visual collections) usage and the breakdown can be seen in figure 1. the general trend across total usage is quite interesting in itself, with the average usage remaining relatively constant between current values and those from the start of the statistical recording in 2010. similarly, figure 1 also demonstrates how significant the ghosted resources are in the total library resource usage, making up 28% of the total resources used in the most current years data, 2016/2017, a significant amount of our yearly loans, a similar trend to a study by rose-wiles & irwin (2016) which also found that nearly 30% of their circulation transactions were used ‘in house’, a significant amount of usage which potentially might have been overlooked if not, for the implementation of ‘ghosting’.   another trend we have been able to use ghosting to identify is the shift away from the traditional build up to april. historically, there has been a dip in usage from the middle to the end of the year, as shown in figure 2 with january to march showing a marked increase in usage before a high peak in april.     figure 1 a breakdown of resource use at keyll darree library 2010 – 2017.     figure 2 2010/11 and 2011/12 monthly ghosting data.     staff have traditionally assumed that as april is a dissertation deadline it will be the busiest for ghosting, and the early figures seemed to fit with this. however, by considering the ghosting statistics in figure 3 we have been able to see that this trend actually changed in 2013 – yet this has still not filtered into staff consciousness. by reviewing the statistics from figure 3 we observe that from 2012 until 2016 this altering trend which has seen a second peak in october continues through the later years (which has sometimes become the heaviest period of usage).   implementation   due to these observed trends, we are able to plan the library’s summer tasks more effectively. in previous years, we had budgeted time from may to november for large scale projects such as stock taking, and collection weeding these are obviously processes which benefit from having a quiet library as they are disruptive to users. since 2013/14 we have seen a second yearly peak taking place in october, and therefore we were able to schedule our project between the end of may and mid-september. this transpired to be a beneficial course of action as the october ghosting for 2016/17 transpired to be significantly higher than the peak in “dissertation season” (february to april).       figure 3 2013/14 onwards monthly ghosting data       outcome   overall, the process of ghosting is suited well to our service. this process was introduced to allow monitoring of in-library resource usage, and does so.   alongside the variety supportive measures used to ensure that we are tracking resource usage within the library, the library has a final fall back for the library users in the form of a withdrawn book for sale shelf – if a book is somehow withdrawn despite regular usage then it is possible for a library user to purchase it.   as a small library with a strong core of regular users we are highly able to engage with them regarding their reading habits, ask questions about the resources, and understand what they want from our service. because of the benefits we have seen, such as a reduction in the removal of well used items; better tracking of busy periods for study desk use (allowing us to plan staff projects); and a fuller picture of resources usage as a whole, ghosting is a process which we will continue. reflection   it is important to note that ghosting is most effective because it is used in tandem with other methods. the process itself is not without limitations and therefore other safeguards must be in place. it is possible that users are leaving them because the items are not useful and there are more relevant resources which they then borrow from the library. purely because an item has been taken off the shelf, we cannot actually guarantee that it is being used on every occasion. however, to combat this issue there is a suggestions box in the library where users can mention limitations or benefits of certain resources. staff are often approached by users who want to provide feedback about the resources they have been using. we also have a system of online reviews to support user feedback, although this is underused at present. staff are working to continually promote it, and encourage users to provide feedback via a text review, or a star rating system (1-5). when verbal reviews are given, staff (after gaining permission) will write these up and add them to the catalogue.     references   rose-wiles, l. m., & irwin, j. p. (2016). an old horse revived?: in-house use of print books at seton hall university. journal of academic librarianship, 42(3), 207-214. http://doi.org/10.1016/j.acalib.2016.02.012 evidence based library and information practice evidence based library and information practice 2010, 5.4 135 evidence based library and information practice news/announcements call for submissions: information: interactions and impact (i³) this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. the 3rd i³ conference – 20-23 june 2011 at robert gordon university in aberdeen, scotland, united kingdom the i3 conference is concerned with the quality and effectiveness of the interaction between people and information and how this interaction can bring about change in individuals, organizations, communities and society. the conference will look beyond the issues of use and accessibility of technology to questions about the way people interact with the information and knowledge content of today's systems and services, and how this might ultimately affect the impact of that information on individuals, organizations and communities. keynote speakers in 2011 will include: • dr. ross j. todd (school of communication and information, rutgers university); • dr. jette hyldegård (royal school of library and information science, copenhagen); • dr. eric t. meyer (oxford internet institute, university of oxford). we invite the submission of high quality papers that report original research or critically discuss underlying methodological issues associated with the main themes of the conference. papers may reflect ongoing or completed research studies and should not have been previously published or be currently under consideration for publication elsewhere. we would particularly welcome papers which address two or more of the following conference themes: • the quality and effectiveness of user/information interactions (e.g. information literacy); • patterns of information behaviour in different contexts; • impact of information or information services on people, organizations, communities and society (e.g. social, learning, cultural and economic outcomes of engagement with information); and • more effective use of information in decision making. submissions are invited for: • full papers; (40 minutes duration: 35 minutes, 5 minutes for questions); • short papers (15 minutes duration: questions at end of session); • posters; and • round table discussions (60-80 minutes duration). authors who are accepted to give full papers at i3 are also invited to submit a full paper for publication in a special issue for the journal libri. libri: international journal of libraries and evidence based library and information practice 2010, 5.4 136 information services is a peer-reviewed journal, established more than 60 years ago, whose contents are monitored by the isi citation indexes. it is published by k.g. saur verlag, an imprint of de gruyter, and is available online through reference global. contributors should submit abstracts of 1000 words for full papers, and 300-500 words for short papers, posters and round table discussions. these should be submitted electronically by downloading the online submissions form, located in the call for papers section of the conference website at: http://www.i3conference2011.org.uk. this should then be e-mailed (as an attachment) to i3submissions@rgu.ac.uk. the conference language will be english. submissions will be anonymously reviewed by two members of an international programme committee specialising in one or more of the conference research streams. notification of acceptance will be e-mailed to authors and will also include the comments of the reviewers. full details of the requirements can be found on the conference website (http://www.i3conference2011.org.uk). the deadline for receipt of abstracts is friday 14 jan. 2011. for any further information please see http://www.i3conference2011.org.uk, or contact the conference team at i3information@rgu.ac.uk. / evidence based library and information practice evidence based library and information practice evidence based library and information practice 2009, 4.4     evidence based library and information practice         news/announcements    isic 2010 call for papers      © 2009. this is an open access article distributed under the terms of the creative commons attribution   license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and   reproduction in any medium, provided the original work is properly cited.      the field of human information behaviour is  multi‐disciplinary in scope; researchers from  information science, information management,  psychology, social psychology, sociology,  information systems, computer science, and  other disciplines all contribute to this field of  investigation.     isic: the information behaviour conference  <http://www.um.es/isic2010/> intends to  reflect this interdisciplinary character through  attracting papers from researchers in all of  these areas. the unifying characteristic, which  we see as essential in developing a  programme, is the relationship between the  needs or requirements of the information user,  the means for the satisfaction of those needs,  and the uses to which those means are put in  practice organizations or disciplines. thus,  papers that deal solely with technological  aspects of system design, for example, will not  be appropriate for the conference.    themes of the conference include the  following:    1. theories and models of information  seeking and searching: particular  theoretical frameworks that are  currently of interest include (but are  not restricted to) social network  theory, actor network theory, cultural‐ historical activity theory, genre theory,  etc.  2. research approaches and  methodologies, both interpretative  and positivist, employing either  qualitative or quantitative methods.  3. information seeking, searching, and  use in specific contexts, e.g. health  care, education, business, industry, the  public services and government, the  emergency services, etc.  4. organizational structures and  processes and information seeking,  searching, and use.  5. information seeking and searching in  virtual social networks, including  gaming and virtual worlds as arenas  for information exchange.  6. information behaviour in everyday  life: in communities both real and  virtual, including its role in  indigenous communities.  7. integrating studies on information  seeking and interactive retrieval.  8. information use: the nature of  information and how information is  used to help solve problems, aid  decision making or satisfy an initial  need.  9. the mediation of information  behaviour: how human or software  agents can respond to information  needs.  10. the design of information delivery  systems to meet information needs  generally, or in organizational or  disciplinary contexts, including web  105 http://www.um.es/isic2010/ evidence based library and information practice 2009, 4.4  2.0 developments such as blogs, wikis,  e‐learning platforms, and open access  information resources.  11. information seeking and information  requirements: integrating information  science and information systems.  12. the communication of information to  users: relationship between  communication theory and  information behaviour, e.g. the  relationship of information  architectures to information seeking  behaviour and the design of  information products on sound  communication principles; including  audio and visual communication  media.    papers that deal with the information  behaviour of practitioner groups, such as  scientists, engineers, local government works,  politicians, and other less‐studied (in this  context) groups, will be particularly welcome.  also, analytical, rather than descriptive  investigations will be sought, with strong  connections to previous work and to  theoretical or conceptual frameworks.    for the 2010 conference, we shall be  particularly interested in papers in any of  these areas that address the connection  between information research and information  practice.    paper preparation and submission deadline is  february 1, 2010. please submit a prepared  paper in microsoft word format (.doc or .rtf  files) to: isic2010@um.es    we also invite doctoral students to submit an  application for participation in the doctoral  workshop held in conjunction with the  conference on the 28th of september, 2010.  we especially welcome submissions from  researchers and doctoral students based in  spain, portugal, and latin america.    106 mailto:isic2010@um.es microsoft word es_corkett.doc evidence based library and information practice 2006, 1:4      74 evidence based library and information practice       evidence summary    the quality of canadian and u.s. government health documents remains  unchallenged until better research can be undertaken    a review of:  lambert, frank. “assessing the authoritativeness of canadian and american health documents:  a comparative analysis using informetric methodologies.” government information  quarterly 22.2 (2005): 277‐96.      reviewed by:  michael corkett  senior information specialist  national collaborating centre for women’s and children’s health  london, england, united kingdom  e‐mail: mcorkett@ncc‐wch.org.uk      received: 25 july 2006            accepted: 25 october 2006      © 2006 corkett. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective ‐ to assess by means of citation  analysis whether the public trust afforded  health documents published by the  canadian and u.s. governments is  appropriate, and to ascertain whether  differences in the respective health care  systems influence how publications are  produced.    design – comparative study.    setting – the canadian depository service  program (dsp) and the u.s. department of  health and human services (dhhs) web  sites.  subjects – one hundred sixty‐six electronic  documents sourced from the dsp website,  and 284 electronic documents sourced from  the dhhs website.     methods – subjects were randomly selected  from repositories offering the most  comprehensive collections. documents with  evidence of references to other works used  in preparation were separated from those  without such characteristics. data variables  were collected from documents with  evidence of references. statistical analysis of  the data was undertaken.     main results – of the respective samples, 89  (53%) from the dsp and 109 (38.4%) from  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:4      75 the dhhs contained references. personal  authors were identified in 46 (51.7%) and 63  (58%) of the respective subsets. handbooks  and guidebooks accounted for the largest  portion of the dsp subset (29; 32.6%) and  government periodicals were the largest  constituent of the dhhs subset (41; 37.6%).  scholarly journals were the most common  reference type for both the dsp (44%) and  the dhhs (58.5%) subsets. the number of  references per document was widely  dispersed for both subsets; the dsp mean  was approximately 64 (sd=114.68) and the  dhhs was 73.71 (sd=168.85). kruskal‐ wallis subset analysis of median number of  references by document type found  differences generalizable to the entire dsp  (p<0.01) and dhhs (p<0.01) populations.  health canada reports, handbooks, and  guidebooks contained significantly more  references than periodical articles or fact  sheets. certain dhss documents, classified  as “other,” contained more references than  periodical articles. canadian documents  were more likely to contain references than  u.s. documents (p<0.01). comparison of  documents to determine whether one  country employs more rigorous citation  practices did not produce statistically  significant results. u.s. federal government  documents are more likely to be referenced  in other u.s. government health documents,  compared to canadian publications (p<0.01).  the presence of references in documents  from either country significantly affected  likelihood of being cited by web authors  (p<0.01).      conclusion – significant differences in  reference use frequencies between dsp and  dhhs documents challenges foskett’s  stance that documents of value contain  references (foskett). use of peer‐reviewed  scholarly journals for both dsp and dhhs  publications was reassuring, suggesting a  fairly rigorous publication standard.  reliance of dhhs publications upon federal  government documents remains unclear.  referencing of dsp documents, irrespective  of reference usage suggests a level of trust  towards canadian government health  publications. web authors appear more  comfortable citing referenced dhhs  documents. further study could involve the  examination of reference frequency by  journal compared against journal impact  factors.    commentary    this is an ambitious study, which attempts  to go beyond the descriptive studies of  previous researchers by providing a  meaningful comparative analysis. however,  the study struggles to provide an insight to  levels of public trust in government health  documents. the choice of a comparative  study design is appropriate. however,  methodological concerns exist for all areas.    despite the study objectives, the existence of  a concise and pragmatically considered  research question is uncertain. as previous  research demonstrates, citation analysis as  an indicator of individual or institutional  quality is often criticised and certainly  flawed (seglen).    the literature review describes the paucity  of relevant evidence and lack of inferential  statistics from research to support previous  findings, yet the author’s search for  evidence could have been extended beyond  the databases library literature and web of  science. methodological insight and  evidence from citation analysis of  biomedical journal quality may have proven  useful.         population samples are described as  random, however detail is lacking as to how  randomisation was achieved. indication of  sample stratification by health subject is not  provided, however narin states that citation  dynamics can be so different across research  fields to make evaluative comparison on the  evidence based library and information practice 2006, 1:4      76 basis of citations difficult or impossible.  sample comparability at baseline therefore  remains unclear.     the data collection methods are poorly  described, and details of the collection  instrument are not provided. the main area  of concern is the lack of primary data  presented. several analyses investigating  differences in reference type across  publication format were carried out,  however the reference type data cited in  each type of publication are not presented.  whilst such an omission probably does not  affect the results and interpretation, it is not  good practice to present inferential statistics  without first presenting the actual data from  which the inferences arise.     results are described in detail and  informetric techniques employed, however  analysis fails to satisfy the research  objectives. the dsp documents are more  likely to contain references than the dhhs  documents, yet there is no presentation of  evidence to indicate any significant step  towards answering the research questions.  of genuine interest is the level of dhhs  referencing of u.s. government publications,  the overwhelming use of specific  publications, and the extent of self‐citation.  data relating to document referencing of  peer‐reviewed journals implies that health  publications from both the dsp and the  dhhs are subjected to a fairly rigorous  publication standard, yet little is presented  to support or discount levels of trust, and no  explicit connections are made between the  different types of health care system and  implications for referencing habits.     opportunity exists for future research in the  area of public trust in government  publications, including health issues.  however, any study would demand a more  rigorous methodological standard than  demonstrated in this study. it is worthwhile  to appreciate that when attempting to  measure the value of any publication,  citation analysis should not be considered as  the one and only evaluation criterion  (shoonbaert).     works cited    foskett, antony charles. the subject  approach to information. london:  clive bingley, 1977.    narin, f. and kimberly s. hamilton.  “bibliometric performance measures.”  scientometrics 36.3 (1996): 293‐310.    schoonbaert, d. and g. roelants. “citation  analysis for measuring the value of  scientific publications: quality  assessment tool of comedy of  errors?” tropical medicine &  international health 1.6 (1996): 739‐52.    seglen, per o. “why the impact factor of  journals should not be used for  evaluating research.” british medical  journal 314.7079 (1997): 498‐502.    evidence based library and information practice evidence based library and information practice 2011, 6.1 4 evidence based library and information practice article evidence or evidence based practice? an analysis of iasl research forum papers, 19982009 marcia a. mardis, ed.d. school of library and information studies college of communication and information the florida state university, florida, united states of america email: mmardis@fsu.edu received: 28 aug. 2010 accepted: 31 jan. 2011 2011 mardis. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective conferences are essential opportunities for professional development and for learning about research. this study analyses papers presented in the research forum track of the international association of school librarians (iasl) conferences to determine whether the amount of school library research reporting increased or decreased over time; who (i.e., what author roles and affiliations) has written about research; which countries were represented in the research articles; what topics were discussed in research articles; and what research methodologies were used. the aim was to determine the extent to which the research forum provides research evidence that relates to practice. methods this study continues the longitudinal analysis of published school library research begun by clyde (1996) by analyzing research forum papers published in iasl conference proceedings from 1998-2009 and using the same approaches and metrics as previous studies by clyde (e.g., 1996; 2002; 2004), clyde and oberg (2004), and oberg (2006). results conference paper topics, author origins, quantities, and research approaches remained static through the 11 years analyzed. the analysis reveals that the papers’ authors, methods, and topics reflected those found in previous studies of school library research. as well as replicating previous studies, the role of academic research at a practitioner-based conference was investigated. mailto:mmardis@fsu.edu� evidence based library and information practice 2011, 6.1 5 conclusions based on long-established imperatives from leaders in the profession, the iasl conferences provide both evidence and evidence based practice for school librarians from all over the world. however, when scholarly research is shared at practitioner venues, it is possible that school librarians may assume that research results constitute evidence based practice (ebp), not evidence upon which practice should be based. this distinction is important if considering that the purpose of academic research is to objectively inform, not to advocate a particular position or practice. the research forum can be a valuable venue for the presentation of empirical research findings and conclusions and objective program evaluations and provide a valuable complement to the evidence based practice descriptions shared in the professional papers portion of the conference program. it is argued that the research forum must be clear in its purpose: to present the results of research; to present effective practice determined by rigorous evaluation; or to present research-supported arguments for the support of school libraries. through a reconceptualization of ebp, the paper demonstrates how ebp is both a method and a methodology for the presentation of school library research and practice in a conference atmosphere. introduction conferences are essential sources of professional development for school librarians (alaimo, 2004; genco, 2003). often working in isolation, school librarians have few learning and professional growth opportunities that link directly to their work (miller, 1999). with limited access to published research conducted in their field, school librarians may be left to make decisions without the benefit of objective evidence in which to ground to their practice (clyde, 2003a). in an effort to promote evidence based practice, in 1998, the international association of school librarians (iasl) conference began providing participants with a variety of presentations that encompass trends and innovations in school librarianship through the sharing of best practices and original research (clyde & oberg, 2004). each iasl conference identifies a unique overarching theme and subthemes; participants tailor their paper presentations to the specific themes of the conferences through professional papers presented in the professional papers track and research papers presented in a conference track called the research forum. purpose of this research this study analyzes, forum papers published in iasl conference proceedings from 19982009 based on the approaches used in previous studies by clyde (e.g., 1996; 2002; 2004), clyde and oberg (2004), and oberg (2006). guided by a desire to discover the extent to which research in school librarianship constituted evidence based practice (oberg, 2006), this analysis focuses on the extent to which lis journals facilitated school librarians’ use of “‘good evidence’ including evidence from research and evidence from practice” (clyde, 2003, p.26). to this end, data were analysed to determine whether the amount of school library research reporting increased or decreased over time; who (i.e., what author roles and affiliations) has conducted the research; which countries were represented in the research articles; what topics were discussed in research articles; and what research methodologies were used. the purposes of this study were to determine the extent to which an analysis of the research forum papers confirms the findings of clyde and oberg’s prior analyses of journal and conference papers between 1995 and 2006 and if the research forum papers affirm iasl’s mission “to provide an international forum for those people interested in promoting effective evidence based library and information practice 2011, 6.1 6 school library media programs as viable instruments in the educational process” (iasl, 2007, ¶2) for, in the words of iasl founder jean lowrie (2011), “the beginning librarian as well as the person who does research…and to push the balance between research and practice.” theoretical perspective the needs of new practitioners provide an additional motivation for characterizing research in school librarianship: to determine the extent to which it can contribute to evidence based practice (ebp) (oberg, 2006). ebp is an approach to practice from medicine, social work, public health, and related fields in which practitioners develop strategies to aid clients and make changes based on the findings of objective, scientifically based research. (mckibbon, 1998). ebp has formed the basis for evidence based librarianship (ebl), in which librarians make decisions about their practice based on objective research findings and evaluate the outcomes of those decisions in light of research, using evidence in a cycle of improvement (booth, 2003). through its research forum, iasl hopes to promote “the message that research studies are important in informing and advancing professional practice in school librarianship” (lighthall & haycock, 1997, p.xiii). literature review in an effort to characterize research in library and information studies (lis), researchers have examined the contents of journals to identify trends in research topics and approaches; these researchers found that surveys and descriptive studies relating to information retrieval and library user education tended to dominate the research articles (järvelin & vakkari, 1993; julien, 1996; clyde 2002). the examined studies tended to describe the method (i.e., the “how to”) of research versus the methodology (i.e., the philosophical underpinnings) of research (hjørland, 2000), leaving the reader to appreciate the results of the reported study but not the motivations behind it. prior characterizations of school library research the role of research in school librarianship has long been debated. as early as 1950, school library leader frances henne lamented the lack of scientific research in the field and pointed out that: a lag exists between the theoretical opinion in the field and research. most functions and objectives of school library practice have been formulated pragmatically on the basis of opinion and experience rather than on evidence produced in research. too, most of these judgement values, even after long adoption and operation, have not been tested or evaluated through the medium of objective research (henne, 1950, p.701). for decades, researchers (e.g., lowrie, 1968; clyde 1996) have been examining published school library research. many of these studies (e.g., clyde 2003b; clyde & oberg, 2004; oberg, 2006) reported that researchers from the united states, canada, and australia authored the majority of published school library research (i.e., peer-reviewed journal articles and refereed conference papers) in all languages available in school librarianship. university faculty and doctoral students overwhelmingly provided authorship while practitioners, government officials, and other policymakers and stakeholders represented a scant number of authors (lowrie, 1968; clyde, 1996; clyde & oberg, 2004). research approaches in school librarianship were typically focused on survey and qualitative methods like focus groups, observation, interview, and analysis of student work (lowrie, 1968; clyde, 1996; clyde, 2003b). investigations conducted in school libraries have predominantly centered on information literacy, reading, and instruction (lowrie, 1968; clyde, 1996; clyde, 2003b; oberg 2006). utility of characterizing research reasons for characterizing published research are manifold. knowledge of the subject distribution of research in library and evidence based library and information practice 2011, 6.1 7 information science can identify gaps for further research exploration (blessinger & hrycaj, 2010). likewise, an understanding of the research approaches commonly used can identify overreliance on particular approaches or the need to equip researchers with methods more appropriate for the topics under investigation (clyde, 2003b). the checks of the peer review process often act as an endorsement of a particular piece of published research (blessinger, & hrycaj, 2010). for university faculty and doctoral students, conducting and presenting objective research is part of the academic culture and research quality is primarily based on the execution of the method and analysis of the data; for practitioners, research quality is determined by the ease with which study’s conclusions can be enacted in their own environments and the benefits those actions yield. consideration of the tension between the research objectivity and advocacy may be particularly important to the study of school library research (clyde & oberg, 2004). school library research and evidence based librarianship the need to promote ebp is inherent in school library research. historically, school library research has not been considered extensive, rigorous, or of high quality (haycock, 1996; mcclure & bishop, 1989). this legacy is particularly troubling in an era when the “gold standard” of educational research has been promoted as the use of randomized controlled trials (rcts) (what works clearinghouse, 2010), a method frequently used improperly in educational research (brass, nunez-neto, & williams, 2006), infrequently seen in lis research (järvelin & vakkari, 1993) and rarely used in school library research (gordon, 2006). because rcts involve the use of randomly assigned participants in control groups and treatment groups, to deprive children of a possibly beneficial educational intervention by including them in a control group raises ethical concerns as well (morrison, 2001). therefore, researchers have sought other methods to assess the soundness of school library research (e.g., clyde, 2004) and its ability to inform practice that benefits children’s learning (todd, 2009) to remain relevant in the educational policy context (booth, 2003). to date, these alternate methods of establishing the soundness of research have been based on an investigation of the peer review process (clyde 2004, 2006) and the evaluation of the outcomes of practice based on research (todd, 2009). as the use of ebp has become more common in lis, it has evolved into a type of cycle in which the outcomes of the evidence based decision have become the object of study and, in essence, project and performance evaluations have become data sources. program and project evaluation, often in the form of action research, has been positioned as an inferential method of determining if a particular practice achieves a desired outcome, especially if that outcome is improved student learning. however, the use of ebl in school librarianship has been labeled as blurring the evidence based paradigm because research pursuits are often a response to a perceived threat to the profession or a need to advocate for an aspect of practice. this implicit motivation for undertaking research as ebp shapes the context in which results are examined and the ways in which conclusions are framed: [s]chool librarianship has forsaken two central tenets of ebp [evidencebased practice]...first, it has neglected the requirement for impartiality in the collection and interpretation of data. second, it has diverted the focus of the evidence-based analysis away from client needs. the result is a topsyturvy practice of collecting evidence for the express purpose of promoting school libraries as an effective educational intervention…advocacy has no place in ebp. in fact, avoiding bias and prejudiced agendas is the main reason these practices developed in the first place (lyons, 2009, p13). evidence based library and information practice 2011, 6.1 8 despite valid as concerns about the use of a strict definition of ebl in school librarianship, critics (e.g., lyons, 2009) seem to have overlooked the incompatibility between scientific methods and the unique political and policy context of research in schools as well as the different uses for evidence in an applied science like lis versus an original context like medical research (booth, 2003). proponents of school library ebl have explored and justified their adapted definitions (clyde, 2006; todd, 2009) based on this incompatibility. aims the july 2006 issue of school libraries worldwide (slw), iasl’s peer-reviewed research journal, contained an article that presented distributions of papers published between 1995-2006 (oberg, 2006) in a replication of clyde’s earlier studies (1994, 2002, 2003b, 2006) and clyde and oberg’s work together (2004). oberg’s 2006 study was a quantitative analysis that focusing on the frequency of the occurrence of the phenomena specified in the research questions and an assessment of the degree to which slw’s contents provided support for school library professionals engaging in evidence based practice. the study described here, repeats and extends oberg’s questions and analyses. research questions the following research questions were investigated. these questions were also used by clyde (2003a), clyde and oberg (2004), and oberg (2006): 1. how have the numbers of research papers changed over time? 2. what were the roles, affiliations, and geographic origins of the research paper authors? 3. what were the research paper topics and methods? how have the topics and methods changed? 4. to what extent do the research forum papers constitute support for evidence based practice? the fourth research question required an interpretation of the relationship between ebp and research as well as of the research forum. the unique nature of this question necessitated that it be addressed in the discussion and conclusion section rather than addressed with the quantitative results. methods previous studies examined papers between 1995 and 2006. for this study, papers in conference proceedings from 1998 through 2009 were analysed, an overlapping 11 year time frame. though research forum began in 1997, the papers were not assigned a research forum designation in the 1997 proceedings. only papers presented as part of the research forum track of the conferences were included in this analysis. ultimately, 199 papers were included in the study sample. description of the analysis in the first phase of the research project, a working data file that contained metadata for each paper was created: year; title; author; role; affiliation; author origin; and conference location. then, additional metadata for each paper including was generated: topic; research approach; research method. the papers’ abstracts and full text were analyzed to populate the topic field with controlled vocabulary based on the categorization of oberg (2006). the research papers were designated as using a quantitative, a qualitative, or a mixed methods approach. the method used within each approach was described based on a controlled vocabulary of the research designs informed by creswell (2006). the next phase of the project included an examination of the results in light of the research questions and the preparation of a research report. the statistical package of the social sciences (spss) was used to generate evidence based library and information practice 2011, 6.1 9 frequency and descriptive statistics, charts, and tables in response to the research questions. in the final phase of the project, the coding of topic, research approach, and research method designations were validated through two additional checks by two doctoral students. results overall, the results of the analyses of the first three research questions were similar to the previous findings of clyde (2003a), clyde and oberg (2004), and oberg (2006) as detailed explorations of each research question illustrate. numbers of research papers over time a total of 199 papers were published in the research forum proceedings between 1998 and 2009, with a mean of 18 papers published in each proceedings volume. table 1 illustrates the number of research forum papers presented at each conference location and that conference’s percentage of the overall total. as figure 1 illustrates, the number of research forum papers varied from conference to conference, but overall, the number of papers tended to increase with a peak (n=30) in the 2008 proceedings. the results suggested that although the number of research papers presented at the iasl conference varied significantly from year to year, the number of papers presented tended to increase over time. roles, affiliations, and geographic origins of the research paper authors usa (n=60), canada (n=24), and australia (n=21) represented the largest number of first author origins, as table 2 depicts. because all but one jointly authored paper represented the same country collaborations and the sole jointly authored paper represented authors from usa and australia, no statistical diversity was gained by counting additional authors. table 1 number of research forum papers from 1998-2009 (n=199) year (location) frequency percent 1998 (israel) 9 4.5 1999 (usa) 20 10.1 2000 (sweden) 11 5.5 2001 (new zealand) 16 8.0 2002 (malaysia) 12 6.0 2003 (south africa) 11 5.5 2004 (ireland) 9 4.5 2005 (hong kong) 20 10.1 2006 (portugal) 21 10.6 2007 (taiwan) 13 6.5 2008 (usa) 30 15.1 2009 (italy) 27 13.6 total 199 100.0 evidence based library and information practice 2011, 6.1 10 figure 1. numerical trend of research forum papers presented from 1998-2009 (n=199). another way of looking at the geographic origins of the research papers is to categorize the authors by their iasl membership zones. the terms of iasl membership are based on country of residence and published gross national product indexes for that country, with international schools being in zone a. zone a countries are the most developed and zone c countries are the least developed. the appendix includes a list of countries in each zone. zone a authors represented 93% (n=185) of the research forum papers in proceedings with zone b representing 6% (n=12) and zone c representing 1% (n=2) of the papers. the total number of authors for the papers was 308. university faculty or students wrote approximately 77% (n=238) of the research papers. school librarians authored about 6% (n=19) of the papers. the remaining author roles included librarians in and directors of other types of libraries (n=30 or 10%); principals, teachers, technology directors, and guidance counselors (n=9 or 3%); consultants (n=6 or 2%); and education officials (n=6 or 2%). research paper topics and methods for this question, the classification scheme for library and information science (lis) topics developed by järvelin and vakkari (1990) as amended by oberg (2006) was applied to the papers. table 4 illustrates that the “information skills and literacy” category was applied to the most papers (n=56 or 28.1). these papers reported research relating to the teaching and learning of information literacy skills and processes as opposed to papers coded as “information seeking” (n=10 or 5%) which reported research pertaining to query formation and search strategy. “information technology” was the topic of 10.6% of papers (n=21) while “reading and reading promotion” papers (n=19 or 9%) reported efforts to increase interest and frequency in the reading of text presented in paper media. papers that reported research relating to preparation of school librarians, “education in lis” comprised 8% (n=16) of papers while papers about “the profession” (n=14 or 7%) addressed topics such as professional standards, roles, and responsibilities as well as perceptions of in-service school librarians. evidence based library and information practice 2011, 6.1 11 papers relating to research on school libraries in the context of lis, “analysis of lis” (n=12) information policy and librarianship, “other aspects of lis “(n=11), and the library program in the school environment, “library and information science activities,” (n=10) comprised 17.5% of papers. studies that pertained to the philosophy of and techniques for conducting research in school libraries were coded as “methodology” and comprised 4.5% (n=9) of papers. the remaining 10% of papers addressed publishing children’s and professional literature, “publishing” (n=7 or 3.5%), surveys of national conditions of school libraries “national survey,” (n=6 or 3%), “censorship” of library resources (n=4 or 2%), “library history,” (n=2 or 1%) and “principal support” (n=2 or 1%). three of järvelin and vakkari’s classifications, “other studies,” “scientific and professional communication” and “information storage and retrieval,” did not apply to any papers. in an effort to gain insight into the potential professional applications of the research reports, paper topics were also matched to school librarian roles as espoused by information power (aasl & aect, 1998) and affirmed by empowering learners (aasl, 2008), guidelines for school librarians supported by the american association of school librarians (aasl). these roles provided classifications for the possible practice application of the research papers’ content. as shown in figure 2, research that was classified as reflecting the “teacher” role (n=80) described studies in which the school librarian led students and fellow educators through the development, execution, and assessment of information literacy curriculum, instruction, and professional development. the second most frequent application of the research was to the “program administrator” (n=50) role. the role encompasses tasks relating to developing and evaluating school library programs, policies and procedures. the role also includes leadership, management, and advocacy for the school library program. table 2 first author country of origin (n=199) author origin frequency percent usa 60 30.2 canada 24 12.1 australia 21 10.6 hong kong 11 5.5 taiwan 11 5.5 israel 10 5.0 new zealand 9 4.5 south africa 9 4.5 iceland 7 3.5 portugal 4 2.0 botswana 3 1.5 france 3 1.5 japan 3 1.5 malaysia 3 1.5 uk/england 3 1.5 uk/scotland 3 1.5 brazil 2 1.0 croatia 2 1.0 jamaica 2 1.0 denmark 1 .5 finland 1 .5 indonesia 1 .5 italy 1 .5 netherlands 1 .5 new guinea 1 .5 spain 1 .5 sweden 1 .5 uk/wales 1 .5 total 199 100.0 evidence based library and information practice 2011, 6.1 12 table 3 authors by year, conference location, and origin zone (n=199) year (location) origin zone total zone a zone b zone c 2009 (italy) 24 3 0 27 2008 (usa) 29 1 0 30 2007 (taiwan) 12 1 0 13 2006 (portugal) 18 2 1 21 2005 (hong kong) 17 2 1 20 2004 (ireland) 9 0 0 9 2003 (south africa) 11 0 0 11 2002 (malaysia) 10 2 0 12 2001 (new zealand) 16 0 0 16 2000 (sweden) 10 1 0 11 1999 (usa) 20 0 0 20 1998 (israel) 9 0 0 9 total 185 12 2 199 table 4 research forum paper topics, 1998-2009 (n=199) paper topic frequency percent information skills and literacy 56 28.1 information technology 21 10.6 reading and reading promotion 19 9.0 education in lis 16 8.0 the profession 14 7.0 analysis of lis 12 6.0 other aspects of lis 11 5.5 information seeking 10 5.0 library and information science activities 10 5.0 methodology 9 4.5 publishing 7 3.5 national survey 6 3.0 censorship 4 2.0 library history 2 1.0 principal support 2 1.0 scientific and professional communication 0 0.0 information storage and retrieval 0 0.0 other studies 0 0.0 total 199 100.0 evidence based library and information practice 2011, 6.1 13 in the “information specialist” role (n=46), the school librarian provides leadership and expertise in acquiring and evaluating information resources; in bringing an awareness of information issues to teachers, administrators, students, and others; and in modelling strategies for locating, accessing, and evaluating information. as an “instructional partner” (n=23), the school librarian collaborates with individual teachers to design and implement learning tasks and assessments as well as to integrate technology and information literacy into classroom content. as table 5 depicts, the papers were distributed fairly evenly across topics from year to year. most topics and numbers of papers relating to those topics were consistent across the data set. exceptions were “principal support” which reflected papers only in 1998 and 1999; “methodology” which was reflected only in the 2005, 2006, and 2007; and “information seeking” was which reflected in 2008 and 2009. likewise, “national survey” and “library history” were both only represented in 2006 and 2008. “censorship” is the only topic that experienced a peak, decline, and resurgence with papers in 1998, 2000, 2008, and 2009. “reading and reading promotion,” present during all conference years, peaked in 2009 (n=7). figure 2. professional roles (aasl & aect, 1998) in research forum papers (n=199). evidence based library and information practice 2011, 6.1 14 table 5 frequencies of papers by topic for each conference year and location (n=199) paper topic∗ year (location) 1998 (israel) 1999 (usa) 2000 (sweden) 2001 (new zealand) 2002 (malaysia) 2003 (south africa) 2004 (ireland) 2005 (hong kong) 2006 (portugal) 2007 (taiwan) 2008 (usa) 2009 (italy) analysis of lis 0 0 0 2 3 0 0 2 0 0 2 3 censorship 1 0 1 0 0 0 0 1 1 0 0 0 education in lis 0 2 2 2 0 3 0 2 2 1 1 1 information seeking 0 0 0 0 0 0 0 0 0 4 6 0 information skills & literacy 1 6 5 5 4 4 4 5 8 2 4 8 information technology 3 5 1 3 1 1 1 1 1 3 0 1 lis activities 0 3 0 0 0 1 2 0 2 0 2 0 library history 0 0 0 0 0 0 0 0 1 0 1 0 methodology 0 0 0 0 0 0 0 4 4 1 0 0 national survey 0 0 0 0 0 0 0 0 1 0 3 2 other aspects of lis 2 0 2 0 0 0 0 1 0 0 2 4 principal support 1 1 0 0 0 0 0 0 0 0 0 0 publishing 0 0 0 3 1 1 1 1 0 0 0 0 reading & reading promotion 1 2 0 0 1 1 1 1 1 1 7 3 the profession 0 1 0 1 2 0 0 2 0 1 2 5 ∗ the list of paper topics was initially defined by järvelin and vakkari (1990) and amended by oberg (2006). evidence based library and information practice 2011, 6.1 15 as figure 3 shows, the majority of the papers (n=87 or 43.7%) used qualitative methods. mixed methods (a combination of quantitative and qualitative approaches) was the next most popular, used in 21.1% (n=42) of the papers. finally, quantitative methods were used in the remaining 19.6% (n=39) of papers. other methods (e.g., policy analysis, literature review) constituted the remaining 15.6% (n=31). as table 6 below illustrates, researchers used one method (method i) or multiple methods (method ii). some authors disclosed their research design and data analysis procedures; many did not. in these instances, research designs were deduced from an examination of the abstract and full text of the paper. table 6 depicts frequencies of data collection approaches. in the instances a data collection method is followed by the word “analysis,” i.e., content analysis, log analysis, stat(istical) analysis, and meta-analysis, these approaches involved the analysis of existing data not collected by the researcher. quantitative methods were used in 39 papers, as table 6 shows. while these methods included experimental (n=2) and statistical analysis of existing data sets (n=3), most of the quantitative studies used surveys (n=34) to collect data. no researcher used more than one method in a quantitative study. as figure 3 above showed, qualitative methods comprised the majority of the research approaches (n=87). the most popular qualitative data collection methods illustrated in table 6 were interview (n=19) and content analysis (n=14). interviews included one study participant, as opposed to focus groups (n=1) that included groups of participants. questionnaires, which require participants to provide text answers to questions, unlike surveys that include numerical or scale responses, were used in 6 studies. many qualitative studies used more than one qualitative method to collect data (n=33) with observation-interview (n=8) and questionnaire-interview (n=7) being two popular choices. figure 3. distribution of research approaches 1998-2009 (n=199). evidence based library and information practice 2011, 6.1 16 table 6 frequencies of research approaches and methods in research forum papers, 1998-2009 (n=199) research approach method ii total method i case study content analysis focus group interview observation questionnaire other method i policy analysis 3 3 non-research 8 8 meta-analysis 3 3 lit review 17 17 total 31 31 mixed methods method i survey 2 7 5 20 1 4 39 stat analysis 0 1 0 0 0 0 1 log analysis 0 0 0 0 0 2 2 total 2 8 5 20 1 6 42 qualitative method i questionnaire 6 -1 -7 1 1 16 observation 9 -1 -8 0 0 18 interview 24 -2 -0 4 3 33 focus group 1 -0 -1 0 1 3 content analysis 14 - 0 - 2 0 1 17 total 54 -4 -18 5 6 87 quantitative method i survey 34 34 stat analysis 3 3 experimental 2 2 total 39 39 evidence based library and information practice 2011, 6.1 17 the second most popular research approach was mixed methods (n=42), an approach that uses both quantitative and qualitative methods to collect data. the researcher integrates the data in the analysis phase (creswell, 2006). most mixed methods were survey-interview (n=20), survey-content analysis (n=7) and survey-focus group (n=5). other research approaches were literature review (n=17), meta-analysis (n=3), and policy analysis (n=3). in a literature review, the researcher synthesizes results of previously conducted research to establish a position whereas meta-analyses use statistical methods to count and characterize research (glass, 1978). eight papers were not research, but were descriptions of programs and policy initiatives. discussion in terms of the first two research questions, “how have the numbers of research papers changed over the last 11 years” and “what were the roles, affiliations, and geographic origins of the research paper authors,” the results of this study were not remarkably different than those of clyde’s studies (e.g., 1996; 2002; 2003a), clyde and oberg (2004), and oberg’s 2006 analyses of 11 years of research papers between 1995 and 2006. it should be noted that this study included the same number of years (1998-2009) as the previous studies but the number of research papers presented from year to year fluctuated, perhaps due to the proximity of the conference location to researchers but possibly due to organizers’ classification of papers into conference tracks. that is, because iasl does not have a set policy for the types of papers presented in the research forum (oberg, 2010), some conference organizers may have chosen to place papers that described a program or practice into the research forum and some organizers may have chosen to place certain kinds of research in the professional papers track. still, the overall number of research papers had a steady increase of almost 300% from 1998 to 2008. as in earlier studies, the results of this study suggest that authors are still largely american, canadian, or australian and university faculty, doctoral students, or master’s students. with the predominance of scholars from wealthy zone a countries among the authors, the concentration of geographic origins and professional roles calls into question the assumptions and applicability of research findings to practitioner situations in less developed nations. there were some differences pertaining to the third research question, “what were the research paper topics? how have the topics changed over the last 11 years?” for example, reading and reading promotion and national survey were two frequent topics in published school library research (oberg, 2006) that were not seen frequently in the research forum papers. however, both sources included few articles about principal support and focused mainly on research relating to the day-to-day activities of the school librarian as a teacher, an instructional partner, and as an information specialist. the extent to which research forum papers constitute support for evidence based practice this final research question was challenging to address. clyde and oberg did not define a set of steps they used to answer this question. rather, they inferred whether the papers supported ebp through a consideration of the number of paper and the variety in their authorship and methods. while the direct answers to the stated research questions of this study are not particularly ground breaking and almost entirely repeat past characterizations of school library research (e.g., clyde, 1996; clyde 2002; clyde 2003a; clyde 2006; clyde & oberg 2004; oberg 2006), more interesting are the possible conclusions one can draw in light of the extent to which the research forum can be used as a source of evidence based practice, the fourth research question. while this question was taken from clyde (2006) and oberg (2006), this researcher found a slightly different one in need of response: should the research forum be a source of evidence based practice? evidence based library and information practice 2011, 6.1 18 by definition, ebp is not research; it is practice guided by the belief that resolution of the research problem is valuable and that the solution should be based on evidence (booth 2003; todd, 2009). in figure 4, stokes’ (1997) model, “pasteur’s quadrant,” adapted by mardis (2009), provides a useful illustration for the relationship between research and practice. whereas ebp would occupy the bottom right quadrant labelled researchbased practice, empirical research occupies the top two quadrants of the model. research is pure basic research that is conducted without any intention for practical application of results. in contrast, practice-based research is research for which practice is the source of the research problem and/or means of data collection and research results are meant to be applied to practice. practice that is not informed by research resides in the bottom left quadrant. given the tension between objective, scientific research (the core of evidence-based practice) and data collection prompted by advocacy, todd (2009) proposed a holistic model for evidence and practice in school libraries that encompasses many of the relationships explored by stokes (1997) and mardis (2009): evidence for practice focuses primarily on examining and using best available empirical research to form practices and inform current actions, and to identify best practices that have been tested and validated through empirical research. this is posited as the informational dimension of school library practice. evidence informs practice. evidence in practice focuses on reflective practitioners integrating available research evidence with deep knowledge and understanding derived from professional experience, as well as implementing measures to engage with local evidence to identify learning dilemmas, learning needs, and achievement gaps to make decisions about the continuous improvement of the school library practices to bring on optimal outcomes and actively contribute to school mission and goals. this is posited as the transformational figure 4. the relationship between research and practice in school librarianship based on stokes (1997) and mardis (2009). evidence based library and information practice 2011, 6.1 19 dimension of school library practice. evidence of practice as the measured outcomes and impacts of practice, is derived from systematically measured, primarily user-based data. it focuses on the real results of what school librarians do, rather than on what school librarians do. it focuses on impacts, going beyond process and activities as outputs. it established what has changed for learners as a result of inputs, interventions, activities, processes, and charting the nature and extent and quality of effect (p.89). todd’s holistic model values the research/practice cycle and honors the many ways in which the effects of practice on student outcomes in school libraries can be measured without the use of a rct. critics of a holistic approach (e.g., lyons, 2009) perhaps have oversimplified the need for school libraries to use experimental designs without considering ethical implications; taking into account the issues inherent in the use of rcts in schools, todd has broadened the definition of ebp to both method and methodology by acknowledging the essential advocacy philosophy underpinning school library research. by giving school library research a core methodology, todd has provided something prior school library research has lacked (hjørland, 2000). this view of ebp encompasses the many professional needs met at the iasl conference. when todd’s levels of ebp are mapped to the mardis (2009) model in figure 5, they clearly mesh with aspects of research and practice in lis. the findings of this study suggest that the research forum, then, meets the informational aspects of evidence for practice while the professional papers meet the formational outcomes of evidence of practice; for practitioners, each aspect of ebp illustrated in figure 5 has function and value. in response to the fourth research question, the research forum is a source for evidence, which is an aspect of ebp. and, in response to the rejoinder as to whether the research forum should be a source of ebp, the research forum figure 5. the relationship between research and practice based on the holistic model of ebp for school libraries (todd, 2009, p.89). evidence based library and information practice 2011, 6.1 20 should certainly be a source of evidence, but the iasl conference as a whole is the source of ebp. conclusion as a source of professional development and support, iasl’s conferences have many missions to fulfill. based on the longestablished imperatives from leaders in the profession to provide support for both research and practice, the conference provides evidence for practice, evidence in practice, and evidence of practice. the research forum can be a valuable venue for the presentation of empirical research findings and conclusions and objective program evaluations and provide a valuable complement to the evidence based practice descriptions shared in the professional papers portion of the conference program. the research forum must be clear in its purpose: to present the results of research; to present effective practice determined by rigorous evaluation; or to present research-supported arguments for the support of school libraries. until the purpose of the research forum is clarified and stated, iasl participants may mistake practice for research and the quest to improve the quality of school library research by building on a core methodology will be frustrated. still, the relatively static representations of researcher origins and roles as well as research approaches and topics suggest that the frontier of school librarianship remains relatively unexplored. as ritchie pointed out, “in order for research to be read by practitioners, it needs to be accessible…to practitioners” (2009, p.31), and the research forum provides such access. however, the concentration of researchers from the united states, canada, and australia calls into question iasl’s ability to fulfill its mission provide an “international forum” (iasl, 2007) and whether research findings can apply to a range of national contexts. armed with the tailored philosophical underpinning of the methodology of ebp as a core function of improving student learning in school libraries as well as the methods described in the research forum papers, perhaps more researchers will be willing to share their work and research forum will expand its representation. references alaimo, r. (2004). top six reasons to attend a conference. knowledge quest, 33(1), 1112. american association of school librarians [aasl], & association for educational communications and technology [aect]. (1998). information power: building partnerships for learning. chicago, il: american library association. blessinger, k., & hrycaj, p. (2010). highly cited articles in library and information science: an analysis of content and authorship trends. information processing & management, 32(2), 156-162. doi: 10.1016/j.lisr.2009.12.007 booth, a. (2003). bridging the research practice gap? the role of evidence based librarianship. the new review of library and information science research 2003, 3-24. doi: 10.1080/13614550410001687909 brass, c.t., nunez-neto, b., & williams, e.d. (2006). congress and program evaluation: an overview of randomized controlled trials (rcts) and related issues. washington, d.c.: congressional research service, the library of congress. retrieved from: http://www.opencrs.com/document/r l33301/2006-03-07/download/1005/ clyde, l. a. (1996). introduction to the collection. in l. a. clyde (ed.), sustaining the vision: a collection of articles and papers on research in school librarianship (pp. 3-11). castle rock, co: hi willow. evidence based library and information practice 2011, 6.1 21 clyde, l. a. (2002). developing the knowledge base of the profession: research in school librarianship. in d. singh, a. abdullah, s. fonseka & b. de rozarion (eds.), school libraries for a knowledge society: proceedings of the 31st annual conference of the international association of school librarianship and the sixth international forum on research in school librarianship, petaling, jaya, malaysia, 5-9 august (pp. 59-75). seattle: international association of school librarianship. clyde, l. a. (2003a). evidence-based practice in school librarianship: evaluating the research evidence. access, 17(4), 26-29. clyde, l. a. (2003b). research in school librarianship 1991-2000: australia in an international setting. the australian library journal, 53(2). retrieved from http://alia.org.au/publishing/alj/53.2/f ull.text/clyde.html clyde, l. a. (2004). evaluating the quality of research publications: a pilot study of school librarianship. journal of the american society for information science and technology, 55(13), 1119. clyde, l. a. (2006). the basis for evidence based practice: evaluating the research evidence. new library world, 107(5/6), 180-192. doi: 10.1108/03074800610665194 clyde, l. a., & oberg, d. (2004). lis journals as a source of evidence for evidence-based practice: the case of school libraries worldwide. paper presented at the world library and information congress: 70th ifla general conference and council, 22-27 august, buenos aires, argentina. http://archive.ifla.org/iv/ifla70/papers/ 051e-clyde_oberg.pdf creswell, j. w. (2006). research design: qualitative, quantitative, and mixed methods approaches. thousand oaks, ca: sage publications. genco, p. (2003). the international association of school librarianship: school librarianship without borders. knowledge quest, 31(5), 17-18. gordon, c. (2006). a study of a three dimensional action research model for school library programs. school library media research, 5. retrieved from http://www.ala.org/ala/aasl/aaslpubsa ndjournals/slmrb/slmrcontents/volum e9/actionresearch.cfm haycock, k. (1996). research in teacher ibrarianship and the institutionalization of change. in l. a. clyde (ed.), sustaining the vision: a collection of articles and papers on research in school librarianship (pp. 1325). castle rock, co: hi willow. henne, f. (1950). libraries, school. in w. monroe (ed.), encyclopedia of educational research, rev. ed. (pp. 701712). new york: macmillan. hjørland, b. (2000). library and information science: practice, theory, and philosophical basis information processing & management, 36(1), 501531. doi: doi:10.1016/s03064573(99)00038-2 international association of school librarianship [iasl]. (2007, march 4). information about iasl: mission of iasl retrieved january 22, 2010, from http://www.iaslonline.org/about/ järvelin, k., & vakkari, p. (1990). content analysis of research articles in library and information science. library and information science research, 12(4), 395421. järvelin, k., & vakkari, p. (1993). the evolution of library and information science 1965–1985: a content analysis http://alia.org.au/publishing/alj/53.2/full.text/clyde.html� http://alia.org.au/publishing/alj/53.2/full.text/clyde.html� http://archive.ifla.org/iv/ifla70/papers/051e-clyde_oberg.pdf� http://archive.ifla.org/iv/ifla70/papers/051e-clyde_oberg.pdf� http://www.ala.org/ala/aasl/aaslpubsandjournals/slmrb/slmrcontents/volume9/actionresearch.cfm� http://www.ala.org/ala/aasl/aaslpubsandjournals/slmrb/slmrcontents/volume9/actionresearch.cfm� http://www.ala.org/ala/aasl/aaslpubsandjournals/slmrb/slmrcontents/volume9/actionresearch.cfm� http://www.iasl-online.org/about/� http://www.iasl-online.org/about/� evidence based library and information practice 2011, 6.1 22 of journal articles. information processing & management, 29(1), 129144. doi: 10.1016/0306-4573(93)90028-c julien, h. (1996). a content analysis of the recent information needs and uses literature. library & information science research, 18(1), 53-65. doi: 10.1016/s0740-8188(96)90030-4 lighthall, l. & haycock, k. (1997). introduction. in lighthall, l. & haycock, k. (eds.) information rich but knowledge poor? research and professional papers p. resented at the 26th annual conference of the international association of school librarianship held in conjunction with the association for teacher-librarianship in canada at the university of british columbia vancouver, british columbia, canada july 06-11, 1997. seattle, wa: international association of school librarianship. lowrie, j. (1968). a review of research in school librarianship. in h. goldhor (ed.), research methods in librarianship: measurement and evaluation, papers presented at a conference conducted by the university of illinois, graduate school of library science, september 10-13, 1967 (pp. 51-69). urbana, il: university of illinois graduate school of library and information science. lowrie, j. (2011, january 19). [what is the purpose of the research forum?] personal communication. lyons, r. (2009). the call for evidence based practice: speaking louder than words. evidence based library and information practice (eblip), 4(3). retrieved from http://ejournals.library.ualberta.ca/ind ex.php/eblip/article/view/6530/5537 mardis, m. a. (2009). a gentle manifesto on the relevance and obscurity of school libraries in lis research. library trends, 58(1), 1-8. mcclure, c., & bishop, a. (1989). the status of research in library/information science: guarded optimism. college and research libraries, 50, 127-143. mckibbon, k. a. (1998). evidence-based practice. bulletin of the medical library association, 86(3), 396-401. miller, s. j. (1999). professional development for the library media specialist. book report, 17(5), 20-21. morrison, k. (2001). randomized controlled trials for evidence-based education: some problems judging ‘what works.’ evaluation and research in education, 15(2), 69-83. oberg, d. (2006). school libraries worldwide as a source for evidence for evidencebased practice, 1995-2006. school libraries worldwide, 12(2), i-xii. oberg, d. (2010, july 10). [what is the research forum?]. ritchie, c. s. (2009). filling the gap: would evidence-based school librarianship work in the uk? library and information science research, 33(104), 26-35. stokes, d. e. (1997). pasteur's quadrant: basic science and technological innovation. washington, d.c.: brookings institution press. todd, r. j. (2009). school librarianship and evidence-based practice: progress, perspectives, and challenges. evidence based practice in library and information practice, 4(2), 78-82. what works clearinghouse [wwc]. (2010). wwc frequently asked questions retrieved july 1, 2010, from http://ies.ed.gov/ncee/wwc/references/ idocviewer/doc.aspx?docid=15&tocid =5 http://ejournals.library.ualberta.ca/index.php/eblip/article/view/6530/5537� http://ejournals.library.ualberta.ca/index.php/eblip/article/view/6530/5537� http://ies.ed.gov/ncee/wwc/references/idocviewer/doc.aspx?docid=15&tocid=5� http://ies.ed.gov/ncee/wwc/references/idocviewer/doc.aspx?docid=15&tocid=5� http://ies.ed.gov/ncee/wwc/references/idocviewer/doc.aspx?docid=15&tocid=5� evidence based library and information practice 2011, 6.1 23 appendix. iasl countries and membership zones countries and zones are featured at http://www.iasl-online.org/about/joiniasl.html zone a countries: international schools (except local employees) are included in zone a. australia, austria, bahamas, bahrain, belgium, brunei, canada, china/hong kong, china/macao, cyprus, denmark, finland, france, germany, greece, iceland, ireland, israel, italy, japan, kuwait, liechtenstein, luxembourg, monaco, netherlands, new zealand, norway, portugal, qatar, san marino, saudia arabia, singapore, slovenia, south korea, spain, sweden, switzerland, taiwan, united kingdom, uk/bermuda, uk/british virgin islands, uk/cayman, uk/england, uk/gibraltar, uk/falkland, uk/north ireland, uk/scotland, uk/wales, united arab emirates, usa, usa/virgin islands. zone b countries: algeria, angola, antigua & barbuda, argentina, azberbaijan, barbados, belize, botswana, brazil, bulgaria, chile, cook islands (new zealand), costa rica, croatia, czech republic, dominica, dominican republic, equatorial guinea, estonia, france/french guinea, france/french polynesia, france/guadeloupe, france/martinique, france/new caledonia, france/reunion, france/st. pierre & miquelon, gabon, grenada, hungary, jamaica, kazakhstan, latvia, lebanon, libya, lithuania, malaysia, malta, mauritius, mexico, micronesia (usa), nauru, neth. antilles, oman, panama, peru, poland, romania, russia, seychelles, slovakia, south africa, st. kitts & nevis, st. lucia, st. vincent & the grenadines, thailand, trinidad & tobago, turkey, turkmenistan, uk/anguilla, uk/monserrat, uk/saint helena, uk/turks and caicos, uruguay, usa/american samoa, usa/guam, usa/northern marianas, usa/palau, usa/puerto rico, venezuela. zone c countries: afghanistan, albania, andorra, armenia, bangladesh, belarus, benin, bhutan, bolivia, bosnia and herzigovina., burkina faso, burundi, cambodia, cameroon, cape verde, central african rep, chad, china, cocos (keeling) islands (australia), colombia, comoros islands, congo, cote d'ivoire, cuba, dem rep of congo, djibouti, east timor, ecuador, egypt, el salvador, eritrea, ethiopia, fiji, france/wallis et futuna, gambia, georgia, ghana, guatemala, guinea, guinea-bissau, guyana, haiti, honduras, india, indonesia, iran, iraq, jordan, kenya, kiribath, kyrgyzstan, laos, lesotho, liberia, macedonia, madagascar, malawi, maldives, mali, myanmar, marshall islands (usa), mauritania, moldova, mongolia, morocco, mozambique, namibia, nepal, niue, nicaragua, niger, nigeria, north korea, pakistan, papua new guinea, paraguay, philippines, rwanda, samoa, sao tome/principe, senegal, serbia/montenegro, sierra leone, solomon islands, somalia, sri lanka, sudan, surinam, swaziland, syria, tajikistan, tanzania, togo, tonga, tunisia, tuvalu, uganda, ukraine, uzbekistan, vanuatu, vietnam, west bank/gaza strip, yemen, zambia, zimbabwe. http://www.iasl-online.org/about/joiniasl.html� / evidence based library and information practice prior characterizations of school library research the role of research in school librarianship has long been debated. as early as 1950, school library leader frances henne lamented the lack of scientific research in the field and pointed out that: a lag exists between the theoretical opinion in the field and research. most functions and objectives of school library practice have been formulated pragmatically on the basis of opinion and experience rather than on evidence produced in research. to... for decades, researchers (e.g., lowrie, 1968; clyde 1996) have been examining published school library research. many of these studies (e.g., clyde 2003b; clyde & oberg, 2004; oberg, 2006) reported that researchers from the united states, canada, and ... utility of characterizing research reasons for characterizing published research are manifold. knowledge of the subject distribution of research in library and information science can identify gaps for further research exploration (blessinger & hrycaj, 2010). likewise, an understanding... therefore, researchers have sought other methods to assess the soundness of school library research (e.g., clyde, 2004) and its ability to inform practice that benefits children’s learning (todd, 2009) to remain relevant in the educational policy cont... however, the use of ebl in school librarianship has been labeled as blurring the evidence based paradigm because research pursuits are often a response to a perceived threat to the profession or a need to advocate for an aspect of practice. this impli... research questions the following research questions were investigated. these questions were also used by clyde (2003a), clyde and oberg (2004), and oberg (2006): how have the numbers of research papers changed over time? what were the roles, affiliations, and geographic origins of the research paper authors? what were the research paper topics and methods? how have the topics and methods changed? to what extent do the research forum papers constitute support for evidence based practice? description of the analysis in the first phase of the research project, a working data file that contained metadata for each paper was created: year; title; author; role; affiliation; author origin; and conference location. then, additional metadata for each paper including was ... the papers’ abstracts and full text were analyzed to populate the topic field with controlled vocabulary based on the categorization of oberg (2006). the research papers were designated as using a quantitative, a qualitative, or a mixed methods approa... frequency and descriptive statistics, charts, and tables in response to the research questions. in the final phase of the project, the coding of topic, research approach, and research method designations were validated through two additional checks by two doctoral students. numbers of research papers over time a total of 199 papers were published in the research forum proceedings between 1998 and 2009, with a mean of 18 papers published in each proceedings volume. table 1 illustrates the number of research forum papers presented at each conference location ... roles, affiliations, and geographic origins of the research paper authors usa (n=60), canada (n=24), and australia (n=21) represented the largest number of first author origins, as table 2 depicts. because all but one jointly authored paper represented the same country collaborations and the sole jointly authored paper repr... for this question, the classification scheme for library and information science (lis) topics developed by järvelin and vakkari (1990) as amended by oberg (2006) was applied to the papers. table 4 illustrates that the “information skills and literacy”... discussion appendix. iasl countries and membership zones evidence based library and information practice evidence based library and information practice 2009, 4.4  evidence based library and information practice       evidence summary    web usability policies/standards/guidelines do not influence practices at arl  academic libraries    a review of:  chen, yu‐hui, carol anne germain and huahai yang. “an exploration into the practices of     library web usability in arl academic libraries.” journal of the american society for    information science and technology 60.5 (2009): 953‐68.    reviewed by:  shandra protzko  information specialist,   tucker medical library, national jewish health  denver, colorado, united states of america  email: protzkos@njhealth.org     received: 27 july 2009          accepted: 29 july 2009      © 2009 protzko.  this is an open access article distributed under the terms of the creative commons attribution  license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and  reproduction in any medium, provided the original work is properly cited.     objective – to survey the current status of  web usability policies/standards/guidelines  (psgs) found in academic libraries of the  association of research libraries (arl).  researchers sought to investigate whether  psgs are in place, the levels of difficulty  surrounding implementation, the impact of  psgs on design, testing, and resource  allocation, and the relationship between arl  ranking and usability practice or psgs.     design – survey.    setting – north america.    subjects – academic libraries of the arl.    methods – an 18‐question survey consisting  of multiple choice, likert scale, and open‐ ended questions was sent to all 113 arl  libraries in november 2007. survey recipients  were selected as the person in charge of web  site usability by visiting library web sites and  phone inquiry. the survey was concluded in  january 2008 with a response rate of 74% (84  institutions). the researchers used t‐test to  detect any difference in arl library ranking  between libraries with and without psgs.  pair‐wise t‐tests were conducted to identify  gaps in difficulty implementing psgs. in  addition, they used pearson’s correlation to  investigate any significant correlations  between variables such as arl rank and  resource allocation.     main results – of the 84 respondents, 34  (40%) have general library web psgs and 25  (30%) have specific usability psgs; 41 (49%)  have at least one type of in‐library psg. of the  43 (51%) libraries that do not have psgs, 30  67 mailto:protzkos@njhealth.org http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2009, 4.4  (36%) are at universities with institutional  web usability psgs; 26 (87%) follow those  guidelines. there was no statistically  significant relationship between arl ranking  and psg status (see table 1).                               the authors asked about difficulty in  implementing psgs. of the 32 libraries  responding to a question about general library  web psgs, most had slight or moderate  difficulty. twenty‐three libraries with specific  usability psgs identified difficulty levels;  some had no difficulty, but a majority had  moderate difficulty. for the 26 libraries using  institutional web usability psgs, most had no  or slight difficulty. pair‐wise t‐tests showed  that library web usability psgs were  significantly more difficult to implement than  university web usability psgs.     enforcement/agreement issues were reported  as the primary difficulty in implementing in‐ library psgs. technical issues and ambiguity  were obstacles at the institutional level. more  than half of the 84 libraries have web advisory  committees and about one third have usability  committees or web usability subcommittees.  several libraries answered that they have none  of these committees, but indicated that they  have some sort of ad hoc committee or user  study group to address usability issues.    of the 84 respondents, 71 (85%) have  conducted usability testing. sixty‐two libraries  (73.8%) rated usability testing as important,  very important, or extremely important: the  rate given for the importance of usability  testing did not correlate with arl ranking.  cited most often in open ended questioning   were the importance of iterative testing,  library wide buy‐in, and staff and resource  availability. main web pages were tested most                               table 1   libraries/universities with/without psgs    library web psgs  library web usability  psgs  university web  usability psgs    no. of libraries (%)  no. of libraries (%)  no. of libraries (%)  with  34 (40%)  25 (30%)  30 (36%)  without  50 (60%)  58 (69%)  31 (37%)  not sure      22 (26%)  no answer    1 (1%)  1 (1%)  total  84 (100%)  84 (100%)  84 (100%)          frequently. fifty‐three libraries (74.6%) tested  their lower level pages at least once. opacs  were tested the least often. the amount of  testing was impacted neither by the existence  of library web psgs nor usability psgs. the  top two testing methods were in‐person  observation and think aloud protocol.     of the 84 libraries, 24 (28%) reported having  staff dedicated to web usability issues; twenty  full‐time staff and four part‐time staff. there  was a weak association between arl ranking  and hours worked by dedicated staff; no  association existed for regular staff who take  on web responsibilities. fifty‐one (60%) of  libraries had regular staff whose duties  included web usability; forty‐six full‐time and  five part‐time. training did not correlate to  amount of testing methods used. there was a  weak link between alr ranking and  availability of resources and, the authors  showed, more testing was done as resources  increased. in response to a query about future  web usability plans, the focus was on usability  testing and site redesign, with only three  libraries planning to refine or establish  usability psgs.    conclusion – the authors hypothesized that  “web usability psgs would influence usability  practice within libraries and other  institutions” (953). the data show that psgs  68 evidence based library and information practice 2009, 4.4  69 do not influence practices. the authors  conclude that there is no significant  relationship between psgs and testing  practices or psgs and the availability of  resources. likewise, arl ranking had no  effect on the establishment of usability psgs.  most libraries are conducting usability testing,  and there was a weak link between arl  ranking and availability of testing resources.  highlighted in the open‐ended questions is  the lack of usability expertise among  stakeholders. workload, inadequate human  resources, and lack of organizational cohesion  are also cited as barriers to the adoption of  web usability psgs. the authors speculate  that web professionals likely use their own  working knowledge and internalized  guidelines without having formal  documentation. the authors further speculate  that the difficulty related to creating mental  models that adequately represent library tasks  may hinder the use of formal usability psgs.  additionally, libraries may not regard the lack  of usability psgs as a liability, especially in  light of the lack of government mandates or  standards. the authors recommend  educational efforts for key players on the  value of web usability, support for hiring  dedicated staff, and formal documentation to  guide design practice. the authors plan to  compare the collected psgs in an upcoming  project. future research could focus on non‐ arl libraries, the relationship between psgs  and user experience, and content  management system (cms) usability  characteristics.      commentary     web usability psgs help developers and  administrators create sites with the end user’s  needs in mind. a focus on usability can help  limit user frustration and enhance site  functionality and effectiveness. while there  are ample design guidelines in the literature,  the authors highlight the lack of studies of the  association between psgs and actual practice.  efficiency and functionality are important  access issues and the authors ask a reasonable  question about the impact of usability psgs,  expanding on the current literature.  unfortunately, as presented in the survey  questionnaire, the distinction between a web  site policy that addresses usability issues and a  specific usability policy is ambiguous. without  a clear working definition of usability,  participant responses may not accurately  reflect actual practice.     the authors clearly describe the study  population, setting and data collection  methods, and they account for limitations of  the study, recognizing that the study is not  generalizable.  since the survey was self‐ report, the validity might be limited; and  while the survey was long, addition  information would have been of benefit. for  example, the authors suggest questioning why  some libraries have no psgs. the  methodology was described to allow for  replication and the results were clearly  outlined and presented in tables. this study  offers an indication of current practice in arl  libraries, the most prestigious libraries in  north america.    there is sound evidence in the literature to  support the use of established design  principles and usability testing practices. the  authors highlight research‐based web design &  usability guidelines, published by the u.s.  department of health and human services, as  one authoritative resource. however, it is not  obvious that just because a library has no  formal documentation supporting usability  efforts that those efforts are not being made.  that arl study participants think usability  testing is important, conduct some form of  testing, and think implementing psgs is at  least somewhat difficult, suggests a need to  convince people why formal psgs are  important, if indeed, they are. it seems  intuitive that written usability psgs would  contribute to the development of quality web  sites, and future studies might indicate as  much. lack of knowledge and resources seem  to be the biggest barriers. it is worth asking,  “what policies or practices actually improve  user experience?” while exploratory, this  study serves as a reminder to all library web  evidence based library and information practice 2009, 4.4  administrators to at least continue to ask  questions about usability practice and  resource allocation.       works cited    united states department of health and   human services. research‐based web  design & usability guidelines.  washington, dc: u.s. government  printing office, 2006.    70 evidence based library and information practice evidence based library and information practice 2010, 5.4 125 evidence based library and information practice classic accurate answers to reference queries may be provided less frequently than expected a review of: hernon, p., & mcclure, c. (1986). unobtrusive reference testing: the 55 percent rule. library journal, 111(7), 37-41. reviewed by: eamon c. tewell senior library assistant moore college of art & design philadelphia, pennsylvania, united states of america email: etewell@moore.edu received: 21 june 2010 accepted: 30 sept. 2010 2010 tewell. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to determine the number of government documents reference questions that are answered correctly by professional library staff. design – the authors utilized unobtrusive reference testing: reference queries posed to library personnel who were unaware they were being evaluated. as opposed to other designs that require the researcher’s presence in the setting, unobtrusive testing utilizes proxies to administer test questions to the subjects, reducing the possibility of reporter bias. setting – twenty-six public and academic libraries participating in the u.s. government printing office depository program located in the western, southern, and midwestern united states. the federal depository program consisted of 1400 libraries at the time of the study. one public and one academic library were chosen for each city. subjects – reference and government documents librarians. these two staff types were selected in order to compare the accuracy of each group’s responses to the queries. methods – a set of 15 predetermined factual and bibliographic questions were developed by the authors and administered to library staff respondents by proxies. government documents were selected as the foundation for the test questions. in selecting federal depository libraries for their sample the authors could ensure all queries may hypothetically be answered using u.s. government printing office documents, as all mailto:etewell@moore.edu� evidence based library and information practice 2010, 5.4 126 of the libraries would hold the resources in question. graduate students enrolled in the university of arizona and university of oklahoma library science programs were trained by the authors to serve as proxies. the proxies posed as library users and administered the set of queries at each selected library. reference librarians and government documents librarians were tested separately, receiving seven and eight questions respectively at each library visited. over a four-month period a total of 390 questions were posed and their answers recorded. main results – the respondents correctly answered 241 of 390 queries (62 percent). government documents librarians accurately answered 65 percent of questions, while reference librarians successfully responded to 59 percent. hernon and mcclure derived the “55 percent rule” for reference accuracy from these results and previous unobtrusive studies conducted by both the authors and other researchers. this body of research estimates the rate of accurate answers of factual and bibliographic questions to be between 50 and 62 percent. data regarding the “interview and search process” (i&s), defined as the activities between the time a query was posed and when a resolution was provided, also yielded intriguing findings. regardless of the question asked, the average i&s duration was three to five minutes. two-thirds of the accurate answers were supplied within three minutes, and 89 percent within five minutes of the initiation of i&s. the duration of i&s did not vary significantly by library type or librarian type. reasons for the provision of inaccurate answers included providing the wrong data (64 percent of instances), responding with “don’t know” and ending the interaction (20 percent), or claiming the library did not own a source that would answer the query (15 percent). other findings included the fact that respondents infrequently offered referrals, which took place in 17 percent of all interactions, and that the three geographic regions studied had an even distribution of correct answers. conclusions based on their research results as well as those of similar unobtrusive studies, the authors propose two rules regarding reference assistance that apply to public and academic libraries. first, reference librarians correctly answer approximately 55 percent of factual and bibliographic queries received (the 55 percent reference rule). second, librarians spend no more than five minutes on most factual and bibliographic questions (the fiveminute answer rule). ultimately, hernon and mcclure recommend reevaluating the centrality of reference services to library operations. if trained professionals are unable to answer more than 60 percent of factual questions correctly, should reference services continue to receive such considerable staffing and funding? or, alternatively, should libraries increase efforts to improve the accuracy of answers? the authors close by challenging the profession to address the need for strategic assessment of reference effectiveness. commentary the accuracy and delivery of reference services remains a focal point for public and academic libraries, and as an area involving substantial resources and staff time it demands informed decision-making. the authors’ goal was to emphasize the user’s perspective as a means to examine widely held assumptions about reference services. hernon and mcclure’s ultimate aim, however, appears to have been to encourage practicing librarians to apply empirical evidence to critical issues in the profession (1987, p. 283). in many ways, their work is an improvement on the authors’ past studies utilizing unobtrusive testing methodology. their approach is commendable, yet the study evidence based library and information practice 2010, 5.4 127 contains several issues of note and potential limitations: • the graduate student proxies were instructed to pose questions to the individual they presumed to be the reference librarian, resulting in the possibility of incorrect respondents (such as support staff) and erroneous data. a more rigorous design would include a second observer to corroborate proxy observations and minimize observer biases. • accuracy is presented as the sole indicator of quality reference service. in practice, a number of other factors not addressed by accuracy contribute to excellent service from the user’s perspective. • not all varieties of reference questions are represented by the factual and bibliographic queries posed. queries requiring ambiguous answers, for example, are not easily measured and might be left unaddressed. • a potential bias was introduced because the government documents sought were not identified as such, yet government document librarians’ expertise could result in identifying these requests. • government documents inquiries, the foundation for the authors’ test questions, are more likely to require additional expertise to successfully navigate compared to standard reference questions. however, it should be noted that all of the resources requested were indeed accessible through the respective library’s catalogue, and could potentially have been located by the librarians. • brief, non-routine factual questions such as those developed by the authors do not constitute the majority of questions received at reference desks, possibly resulting in a lower accuracy rate compared to that which might be found while observing everyday reference operations. • schumacher correctly notes that the inclusion of routine factual and directional questions would cause the accuracy rate to increase dramatically (1987). while their research contains these aforementioned flaws, hernon and mcclure succeeded in creating sustained discussion on the topic of improving the evaluation of reference services. their landmark study contributed significantly to the popularity of unobtrusive testing to assess services. applications of the methodology have been limited to investigating the quality of reference, yet unobtrusive measures could in fact be applied to multiple areas, particularly instruction and outreach. unobtrusive testing, also frequently referred to as unobtrusive observation, provides unique advantages not afforded by other methodologies. since research subjects are unaware their responses are being recorded, observation does not change their behaviour. as whitlach states, subjects “cannot introduce bias into the study because they are unaware of it” (1989). additional strengths beyond the elimination of self-reporting bias are the significant scope of data that can be collected, such as non-verbal behaviours, and the lack of reliance on the respondent’s recollection of events. weaknesses of unobtrusive testing include generalizability and ethical concerns. due to the method’s basis in observation, a sample must be examined numerous times to produce generalizable findings. generalizability poses a considerable obstacle for unobtrusive studies examining specific populations. a thornier issue is that of ethics. the lack of informed consent, participant unawareness of the study, and potential invasion of privacy can raise significant concerns depending on the research being conducted. on a practical level, the ethical questions raised by unobtrusive observation can make it more difficult for researchers to receive institutional or ethics review board approvals. evidence based library and information practice 2010, 5.4 128 as might be expected of a work that is highly critical of library performance and contains major implications for practitioners, “the library community responded to these unobtrusive studies first with shock, then with denial” (hults, 1992, p. 143). bailey, a vocal detractor of hernon and mcclure’s methodology, stated that the 55 percent rule “has caused a stir among librarians in this country and has reverberated internationally” (1987, p. 280). in 2005, hubbertz sharply criticized the article’s methods and findings, arguing that unobtrusive observation is better suited to comparing and ranking libraries than assessing the quality of a service (p. 6), and that the validity of the results was seriously compromised due to test subjects being asked different questions. numerous articles have responded directly to the findings, testing the validity of the 55 percent rule or altering the methods, with varying results (durrance, 1989; whitlatch, 1989; whittaker, 1990; jardine, 1995; richardson jr., 2002; hubbertz, 2007). the influence of hernon and mcclure’s study is evidenced in part by future research employing the same methods and applying them to other indicators of patron satisfaction. durrance, for example, sought to address the larger reference context and measured the “willingness of the inquirer to return to the same staff member at a later time” (1989, p. 32), while dilevko and dolan applied the design of hernon and mcclure’s study to a different geographic location (2000). unobtrusive testing has also surfaced in “mystery shopping” evaluation programs. originating in the business world and designed to assess employee behaviour, this practice has been adopted by a number of libraries and utilizes the unobtrusive method (thomas, 2000). without hernon and mcclure’s controversial findings based on a user-centered research design, it is questionable whether researchers would have worked to develop other measurements of reference effectiveness. other authors have utilized the framework laid by hernon and mcclure’s study in innovative ways. curry, attempting to gauge the quality of reference services to a particular marginalized youth population, asked teenagers to pose gayand lesbian-related questions at twenty public libraries in british columbia (2005). shachaf and horowitz’s 2006 article examined whether the inferred ethnicity of a fictitious patron’s e-mail request affects the response of reference librarians in twenty-three association of research libraries member libraries. while the quality of reference services is still the subject of evaluation, these recent studies demonstrate the potential for additional work in the unobtrusive testing domain. hernon and mcclure’s study does not hold to current evidence based library and information practice standards. as published, the study contains little information allowing for replication. additional factors, such as potential observer bias and different questions asked of test subjects, diminish the study’s reliability. however, in a 1987 response to the intense debate their article had provoked, the authors’ laudable objective can be seen in the statement: “research may serve as an objective counterweight to ‘informed opinion’—a widespread commodity in librarianship.” indeed, the greatest contribution of hernon and mcclure’s work is the reminder that assumptions regarding library service merit rigorous study and debate. references bailey, w. g. (1987). the “55 percent rule” revisited. journal of academic librarianship, 13(5), 280–82. childers, t. (1987). the quality of reference: still moot after 20 years. journal of academic librarianship, 13(2), 73-74. crowley, t., & childers, t. (1971). information service in public libraries: two studies. metuchen, nj: scarecrow press. curry, a. (2005). if i ask, will they answer? reference & user services quarterly, 45(1), 65-75. evidence based library and information practice 2010, 5.4 129 dilevko, j., & dolan, e. (2000). government documents reference service in canada: a nationwide unobtrusive study of public and academic depository libraries. library & information science research, 22(2), 185-222. durrance, j. c. (1989). reference success: does the 55 percent rule tell the whole story? library journal, 114(7), 31-36. hernon, p., & mcclure, c. r. (1987). where do we go from here? a final response. journal of academic librarianship, 13(5), 282-284. hubbertz, a. (2007). the fallacy in the 55 percent rule. dttp: a quarterly journal of government information practice & perspective, 35(3), 15-17. hubbertz, a. (2005). the design and interpretation of unobtrusive evaluations. reference & user services quarterly, 44(4), 327-335. hults, p. (1992). reference evaluation: an overview. the reference librarian, 17(38), 141-150. jardine, c. w. (1995). maybe the 55 percent rule doesn’t tell the whole story: a usersatisfaction survey. college & research libraries, 56(6), 477-85. richardson, j. v., jr. (2002). reference is better than we thought. library journal, 127(7), 41-42. schumacher, m. (1987). the continuing debate on library reference service: a minisymposium. journal of academic librarianship, 13(5), 278-79. shachaf, p., & horowitz, s. (2006). are virtual reference services color blind? library & information science research, 28(4), 501520. thomas, j. (2000). mystery shoppers at the library: a planning report. california state university long beach library. retrieved 20 may 2010 from: http://www.csulb.edu/divisions/aa/grad _undergrad/senate/committees/assessm ent/dev/awards/documents/thomas_99. pdf whitlatch, j. b. (1989). unobtrusive studies and the quality of academic library reference services. college and research libraries, 50(2), 181-194. whittaker, k. (1990). unobtrusive testing of reference enquiry work. library review, 39(6), 50-54. / evidence based library and information practice microsoft word class_eldredge.doc evidence based library and information practice 2007, 2:4  84 evidence based library and information practice       classic    do clinical librarians matter? the first randomized controlled trial in librarianship      a review of:  marshall, joanne gard, and victor r. neufeld. “a randomized controlled trial of librarian  educational participation in clinical settings.” journal of medical education 56.5 (1981):  409‐16.      reviewed by:   jonathan eldredge  associate professor, school of medicine and health sciences library & informatics center  university of new mexico  albuquerque, new mexico, united states of america  e‐mail: jeldredge@salud.unm.edu        received :  24 september 2007    accepted : 06 november 2007      © 2007 eldredge. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objectives – to determine whether clinical  librarian services cause healthcare providers  to change their information seeking  behaviors. to evaluate librarians’  educational roles for clinicians, patients, and  patients’ families.    design – randomized controlled trial.    setting – an academic, health‐sciences‐ center, teaching hospital in canada.    subjects – a total of eight teams, each  consisting of at least eight members who  represented at least three different types of  health professionals. four teams  (rheumatology, obstetrics, neurology, and  pediatrics) were randomized into the  intervention group to receive clinical  librarian services for a six‐month period,  and four teams (hematology, diabetic day  care, pain clinic, and community psychiatry)  were randomized into the control group that  did not receive clinical librarian services.    methods – two half‐time clinical librarians  attended the intervention groups’ rounds,  clinics, and conferences identified as having  educational components or where questions  would likely arise related to patient care.  the two clinical librarians handled 600  perceived or actual information requests,  delivered 1,200 documents, and provided  over 3,000 references during the twelve‐ mailto:jeldredge@salud.unm.edu http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:4  85 month study period of september 1978 to  august 1979. the typical service consisted  of the clinical librarian securing one or two  articles relevant to the question raised along  with pertinent references placed in a “hot  topics” ring binder located in the clinical  wards. healthcare providers were alerted to  or reminded about the clinical librarian  service through a brochure and an exhibit.  the brochure also advertised the clinical  librarian service to patients or their families.  approximately 24% of all information  requests fielded by the clinical librarians  originated from patients or their families.  the remaining information requests  originated from physicians (40%), allied  health professionals (21%), and nurses (15%)  belonging to these interdisciplinary  intervention group teams.    main results – trained impartial  interviewers conducted in‐depth interviews  with members of both the intervention  group teams and the control group teams  immediately following the first six‐month  study period and then again three months  after the end of the study period. following  the initial six months of the study period,  67% of the members of the intervention  group compared to 37% of the members of  the study group used the library’s reference  services. three months after the study  period had ended, 76% of the members of  the intervention group compared to 49% of  the members of the study group had used  the reference services. the authors reported  in a one‐sentence page note that these  findings were statistically significant beyond  the .05 level as measured by chi‐square and  analysis of variance tests. three months  after the study period had ended, 60% of the  members of the intervention group  compared to 38% of the members of the  study group reported rating highly the use  of library resources. in addition, three  months after the study period had ended,  36% of the members of the intervention  group compared to 27% of the members of  the study group reported rating highly the  use of reference librarians. although  patients or their families were generally  positive in rating the clinical librarians’  services, they proved to be a difficult  population upon which to conduct a  comprehensive follow‐up evaluation study  once patients had left the hospital.    conclusion – the authors conclude that the  clinical librarian services to the four  intervention groups had changed the group  members’ information seeking behavior.     commentary    clinical librarian programs emerged during  the early 1970s, surrounded with great  enthusiasm among medical librarians.  several retrospective cohort studies or case  studies attempted to determine the actual  usefulness of these clinical librarian  interventions. one such study even  combined a cohort study with a cost‐benefit  analysis. yet, the utility of clinical librarian  programs could not be proven (schnall and  wilson). staudt, halbrook, and brodman in  1976 published a retrospective cohort study  that evaluated a clinical librarian program.  they used the phrase “an attempt at  evaluation” in their title and they concluded  with some apparent frustration that, despite  the methodological rigor of their cohort  study effort, “the worth of clinical  librarians’ programs has not yet been  proved quantitatively or unequivocally”  (238).    this important applied research question  seemed unanswerable until marshall and  neufeld utilized the more appropriate  randomized controlled trial (rct) method  for answering this intervention question  more successfully. current eblip levels of  evidence indicate that an rct represents the  highest level of evidence for a single study  to answer an intervention question  (eldredge “levels of evidence”).  during  evidence based library and information practice 2007, 2:4  86 1978, however, the authors were pioneering  an rct when no other rct had ever been  applied to a library practice research  question (eldredge “trial design”).     to pioneer an rct to answer an important  question alone would constitute an  important landmark event for our  profession. equally important, marshall and  neufeld adapted the rct design  innovatively to evaluate the effectiveness of  the clinical librarian program by using  clinical teams as the unit for randomization,  while measuring individual responses to  questionnaires as the unit of statistical  analysis. rcts date back to 1948, but in 1981  were still relatively rare within clinical  medicine as compared to 2007. a pubmed  search using rcts as a publication type  reveals that there were almost 11,000 rcts  published during the decade leading up to  the marshall and neufeld rct, whereas the  past decade has witnessed over 126,000  published rcts.  the predominant model  for an rct during 1978, then just as now,  principally involved administering drugs to  individual patients and then comparing any  effects of the drugs on the intervention  group members compared to control group  members. marshall and neufeld  demonstrated a profound understanding of  research methodology in order to apply the  rct so appropriately in a different context.  even when viewed today, with the rct  method now far more developed and better  understood, their adaptation represents a  significant achievement.    two systematic reviews published in 2003  and 2004 evaluated clinical librarian  programs by synthesizing the available  research literature. a 2003 systematic review  (sr) by winning and beverly covered the  years 1983‐2002 due to the expressed desire  of the authors to update cimpl’s narrative  review published in 1985. by their own  reckoning, however, they noted that cimpl’s  work had been based on a selective  collection of studies so it could not even  approximate the comprehensiveness of an  sr. winning and beverly’s inclusion criteria  therefore excluded the marshall and  neufeld 1981 study by two years. this  starting date for winning and beverly’s sr  seems unnecessarily arbitrary considering  the more inclusive approach of a second sr  published in 2004.     this second sr by kay cimpl wagner and  gary byrd was far more comprehensive  (1974‐2002) than winning and beverly in its  search of the literature. wagner and byrd  noted importantly that the marshall and  neufeld rct was one of only four studies  that displayed methodological rigor by  “using historically controlled before‐and‐ after methods or comparison control  groups” (30). a third sr published during  2005 by alison weightman and jane  williamson reviewed the impact of library  services on patients and on the specific  dimension of time saved for clinicians.  weightman and williamson’s sr included  the aforementioned 1976 study by schnall  and wilson, but consciously excluded the  1981 marshall and neufeld rct since it did  not report any patient outcomes (21). this  explicit exclusion in weightman and  williamson’s sr can be defended  methodologically (mulrow and cook).    the marshall and neufeld study stands out  as the only rct to evaluate a clinical  librarian program, thereby representing the  highest form of evidence in answering an  intervention question.  one could find ways  to improve upon this rct by using more  advanced methodological approaches  available to us now such as minimizing  contamination between groups and  instituting more detailed measures. yet, it  seems most noteworthy that our profession  has failed even to utilize a single rct since  then to answer what our profession  considers to be the important question on  the efficacy of clinical librarian programs!  evidence based library and information practice 2007, 2:4  87 works cited    cimpl, kay. “clinical medical librarianship:  a review of the literature.” bulletin of  the medical library association 73.1  (1985): 21‐8.    eldredge, jon. “evidence‐based  librarianship: levels of evidence.”  hypothesis 16.3 (2002): 10‐3. 21 nov.  2007  <http://research.mlanet.org/ebllevels. pdf>.    eldredge, jonathan d. “the randomised  controlled trial design: unrecognized  opportunities for health sciences  librarianship.” health information  and libraries journal 20 supp. 1 (2003):  34‐44.    mulrow, cynthia, and deborah cook, eds.  systematic reviews: synthesis of best  evidence for health care decisions.  philadelphia, pa: american college of  physicians, 1998.      schnall, janet g., and joan w. wilson.  “evaluation of a clinical medical  librarianship program at a university  health sciences library.” bulletin of  the medical library association 64.3  (1976): 278‐83.  staudt, cecilia, barbara halbrook, and  estelle brodman. “a clinical  librarians’ program – an attempt at  evaluation.” bulletin of the medical  library association 64.2 (1976): 236‐8.    wagner, kay cimpl, and gary d. byrd.  “evaluating the effectiveness of  clinical medical librarian programs: a  systematic review of the literature.”  journal of the medical library  association 92.1 (2004): 14‐33.    weightman, alison l., and jane williamson.  “the value and impact of information  provided through library services for  patient care: a systematic review.”  health information and libraries  journal 22.1 (2005): 4‐25.     winning, m.a., and c.a. beverly. “clinical  librarianship: a systematic review of  the literature.” health information  and libraries journal 20 supp. 1 (2003):  10‐21.      http://research.mlanet.org/ebllevels.pdf http://research.mlanet.org/ebllevels.pdf evidence based library and information practice evidence based library and information practice 2011, 6.1 56 evidence based library and information practice evidence summary face-to-face training is the preferred modality of professional continuing education for librarians of all ages, but more evidence is needed a review of: lynn, v. a., bose, a., & boehmer, s. j. (2010). librarian instruction-delivery modality preferences for professional continuing education. journal of the medical library association, 98(1), 57-64. reviewed by: kathryn oxborrow collection management librarian, national library of new zealand te puna mātauranga o aotearoa department of internal affairs te tari taiwhenua wellington, new zealand email: kathryn.oxborrow@dia.govt.nz received: 23 nov. 2010 accepted: 13 feb. 2011 2011 oxborrow. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to establish the preferred modality for professional continuing education (ce) among members of three library associations. the primary hypothesis was that face-to-face training is the preferred modality, and the secondary hypothesis was that younger librarians are more likely to favour online or blended training modalities. in addition, the authors sought to investigate which factors influence participants’ decisions to take up training. design – online questionnaire. setting – three library associations based in the united states of america. these were the american library association (ala), the special libraries association (sla), and the medical library association (mla). subjects – a random sample of 328 members of the ala (86 participants), sla (63 participants), and mla (291 participants). some participants were members of more than one association. methods – participants were recruited to complete an online survey via direct e-mail contact (mla), messages on email discussion lists (sla) and social networks (ala). the survey asked about participants’ experience of, and preference for, five different training modalities for ce. these were: face-to-face (classroom instruction), web-based synchronous (with real-time participantinstructor interaction), web-based mailto:kathryn.oxborrow@dia.govt.nz� evidence based library and information practice 2011, 6.1 57 asynchronous (with instructor involvement, but not in real time), blended (a combination of different modalities), and webcasts (live online presentations with limited participantinstructor interaction). participants were then asked to rank factors which would influence their decision to undertake ce courses. the factors were cost, opportunity to socialize/network, time away from work, learning at their own pace, and having immediate access to either the class instructor or other participants. participants were also given space to comment on both ce modalities and influencing factors. main results – there was a statistically significant preference for face-to-face instruction in this sample, being preferred by at least 73.1% of participants in all age ranges. younger librarians did not display a preference for online or blended training modalities. there was a significant difference in second preference between ala and mla members, who both preferred web based asynchronous training, and sla members, who preferred the web-based synchronous format. participants’ preferences for all modalities apart from face to face were significantly different depending on whether or not they had experienced the particular modality. cost was ranked as the most influential factor in the decision to undertake ce by members of all three library associations (significant at p<0.001). the second most important factor was immediate access to the class instructor. this was also significantly higher than the other factors, which did not differ significantly between one another. participants raised other issues such as the importance of the location of face to face training or hosted webcasts, and the likelihood of self paced training being put aside in favour of everyday work. conclusion – the results confirm the hypothesis that face to face is the preferred training modality for this sample of members of the ala, mla and sla. however, the secondary hypothesis, that younger librarians are more likely to prefer online or blended training methods was disproved in this sample. since this is the case, and there is a strong influence of cost on the uptake of ce courses, the authors suggest that providers of ce should consider these results when planning training to suit the needs of their members. commentary this article begins by giving a clear explanation of the different training modalities investigated in this study, and the contexts in which training is offered to members of the three library associations surveyed. a little more explanation could have been given to the system of professional development operated by these associations, with an unqualified mention of “credits” which may be out of context for readers who are not involved with these library associations. hypotheses regarding participant preferences for different types of training for ce were based on a small number of studies, most of which were about health professionals rather than librarians. no hypotheses were made regarding factors influencing participants’ decisions to take up training, and there was no reference to the literature on this subject. references to such studies would help readers to understand how the authors decided on the factors to include in this part of the questionnaire. the questionnaire itself has a clear and comprehensible format, and is included as an online-only appendix to the article. it is also well described in the text. the questionnaire was piloted on a group of librarians before use. the authors appear to have chosen not to collect data about the ethnic origin and gender of participants. face-to-face was found to be the preferred modality of ce for the sample. this finding is statistically significant, but it is important to emphasize that the sample does not represent an even distribution of members of the three library associations, with the majority coming from the mla. this is most likely due to the fact that the authors used random and convenience sampling and had to employ different methods for contacting members of evidence based library and information practice 2011, 6.1 58 the different associations. in addition to this, two thirds of the sample were aged over 44 years or had been working as a librarian for over 10 years, which impacts on the sample as a whole. this may have been a demographic reflection of the librarian population in the united states of america, but this was not mentioned explicitly in the text, so can only be inferred. the participants were distributed thus: 18-34 years (n=37), 35-44 years (n=46), 45-54 years (n=106), and 55 years+ (n=135). the disproval of the secondary hypothesis, that younger participants would prefer web-based or blended training modalities, should be treated with caution on the grounds of the relatively small participant numbers in the 18-34 age group. this study looks at an interesting topic, and uses a strong questionnaire, but further research is needed to confirm its findings. a detailed discussion of the findings in relation to similar studies was not possible, due to the lack of such studies. the authors highlight the limitations of their research, such as the high proportion of mla members in the sample, and the influence of experience of a training modality on their preference for it. on their own, the results of this study should not be used to inform the development of future training. there is plenty of scope, however, for further research in this area, and the authors suggest several topics for this including the relationship between the content, structure or objectives of courses and their optimal instructional modality; the inclusion of web 2.0 and mobile technologies in ce for librarians; and studying a more evenly proportioned sample across the three library associations surveyed. other interesting topics could be a more detailed qualitative study on participants’ motivations for deciding whether to undertake training, or a study of preferences from the perspective of trainers rather than participants. / evidence based library and information practice evidence based library and information practice 2010, 5.1 82 evidence based library and information practice article information literacy instruction assessment and improvement through evidence based practice: a mixed method study diana k. wakimoto senior assistant librarian california state university, east bay hayward, california, united states of america email: diana.wakimoto@csueastbay.edu received: 21 july 2009 accepted: 01 feb. 2010 2010 wakimoto. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective— this study explored first-year students’ learning and satisfaction in a required information literacy course. the study asked how students understand connections between themselves and information literacy in terms of power, society, and personal relevance to assess if students’ understanding of information literacy increased after taking the course. student satisfaction with the course also was measured. methods—the study used preand post tests and focus group session transcripts which were coded and analyzed to determine student learning and satisfaction during the regular 2008-2009 academic year at california state university, east bay. results— many students entered the course without any concept of information literacy; however, after taking the course they found information literacy to be personally relevant and were able to articulate connections among information, power, and society. the majority of students were satisfied with the course. the results from analyzing the preand post-tests were supported by the findings from the focus group sessions. conclusion— the results of this study are supported by other studies that show the importance of personal relevancy to student learning. in order to fully assess information literacy instruction and student learning, librarians should consider mailto:diana.wakimoto@csueastbay.edu evidence based library and information practice 2010, 5.1 83 incorporating ways of assessing student learning beyond testing content knowledge and levels of competency. introduction being information literate can be considered one of the most important characteristics for an individual to possess in today’s information driven world. as such, information literacy and the assessment of information literacy instruction are topics of great interest and discussion within many university systems, including the california state university system (dunn, 2002). in order to support the goal of graduating students who are information literate, the librarians at california state university, east bay (cal state east bay) teach a two-credit information literacy course that is required for all first-year university students. the learning objectives for this course are the ability to: formulate a research question, develop and use appropriate search strategies, evaluate strategies and results, describe research processes and communicate results, and understand and apply principles of information ethics. all of these learning objectives are in support of the information literacy learning outcomes for lower division general education at cal state east bay (california state university, east bay, n.d.). all instructors must cover these course objectives and traditional assessment methods, such as exams, ensure evaluation of students’ content knowledge and research skill competencies. however, little assessment has been conducted on student understanding of information ethics. this is an important facet of information literacy and supports the university’s mission and values statements of helping students to become “socially responsible contributors” and critical thinkers who are able to make ethical decisions (california state university, east bay, 2008). this study assessed information literacy instruction by evaluating student understanding of information literacy in terms of: personal relevancy, the connections between information and power, and the connections between information and society, all of which are tied to the learning objective of understanding and applying principles of information ethics. the ultimate goal is to use the results to improve the course through increased student learning. literature review standards, learning outcomes, and assessment tools for information literacy are not new phenomena. the american library association created its presidential committee on information literacy: final report in 1989. other associations have published more recent standards: the association of college & research libraries published information literacy competency standards for higher education in 2000, and the australian and new zealand institute for information literacy created the second edition of australian and new zealand information literacy framework in 2004. these definitions and standards have been considered as authoritative definitions of information literacy in various contexts (bruce, 2000). and these standards, among others, have been adapted and used to assess students’ mastery of information literacy skills and competencies at the university level (dunn, 2002; maughan, 2001). these standards for information literacy, much like cal state east bay’s information literacy learning outcomes, mainly focus on the skills and competencies that individuals must master to be considered information literate. while these skills are by no means unimportant, information literacy can be seen as more than a set of tools or skills to be used when researching (webber & johnston, 2000). as the field of information literacy research has expanded and matured, new frameworks for viewing information literacy instruction have been created. in these frameworks, information literacy is presented as a way of learning or interacting with information. bruce, edwards, and lupton’s “six frames for evidence based library and information practice 2010, 5.1 84 information literacy education: a conceptual framework for interpreting the relationships between theory and practice,” which extends bruce’s seven faces of information literacy (1997), defined six ways, or frames, of learning, teaching, and understanding information literacy. the six frames are: content, competency, learning to learn, personal relevance, social impact, and relational (bruce, edwards, & lupton, 2006). the “six frames for information literacy education” is only one of the newer studies that extend the concept of information literacy beyond content and competency skills. these studies extend information literacy into sociopolitical and socioeconomic spheres (andersen, 2006) and extend the study of variations of experiencing information literacy to focus on student perceptions (maybee, 2006). these expanded views are important to the integration of information literacy into students’ lives outside of the university. the shift in focus from skills and competencies to integrating and internalizing information literacy into all facets of life has the possibility of transforming information literacy into something that is personally relevant to students. this study used the “six frames” as a guide for examining student understanding of learning in multifaceted ways that include content and competency. it also incorporates other facets such as seeing information literacy as relevant to life outside of school and impacting social interactions. these areas were not fully assessed before this study at cal state east bay. this was an important, overlooked gap in the evaluation of student achievement in terms of reaching the learning objectives of the course and of fulfilling the university mandates. if instruction is seen as transformative (bruce, 2008), then the students’ progression to more nuanced ways of experiencing and understanding information literacy becomes important in the assessment of information literacy instruction. aims this study explored students’ learning and satisfaction in a credit-bearing information literacy course that all first-year students are required to take in order to be eligible to graduate from the university. the study asked how students defined information literacy, understood connections between information and power, and understood the relevancy of information literacy in their lives. student satisfaction with the course was also measured. the ultimate objective is to improve the course by using the results of this study to increase student learning in relation to the course objectives. methodology this study used preand post-tests and focus group sessions for collecting data on student understanding of information literacy and satisfaction with the course. during each of the three regular academic quarters in the 2008-2009 academic year (fall, winter, and spring), students in one section of the introduction to information literacy course completed preand post tests. there were a total of 90 students in these three sections. the introduction to information literacy course covers basic information literacy concepts, skills, and applications. the content of the course includes: library skills, database searching, the research process, types of information formats, evaluation, web 2.0, communication, information ethics issues (e.g. privacy, plagiarism, copyright, and censorship), and applications of information literacy outside of the classroom. mastery of these skills and concepts, as noted in the learning objectives (see introduction), are evaluated via weekly homework assignments, a reflective essay, a final project consisting of a research diary and annotated bibliography, and an experiential final exam. the preand post-tests used open-ended questions (appendix 1) and were administered anonymously in the first and last weeks of the quarters. the responses were open coded, analyzed, and categorized. open coding was used since the categories of the responses could not be pre-determined before evidence based library and information practice 2010, 5.1 85 the assessments as all of the questions required free-text, written responses, unlike a multiple choice questionnaire. the responses were transcribed and after careful readings of the answers, the author derived categories for each question (auerbach & silverstein, 2003). student responses that conveyed the same concepts or themes were assigned to the same category. the coding of responses was checked to ensure reliable coding of responses for the three quarters. results from the analysis of the pre-tests were compared to the post-test results for similarities and differences in categories of responses. the results were compared only in the aggregate. student volunteers from the course sections that completed the preand post-tests participated in focus groups. students received lunch for participating. there were focus group sessions at the end of the fall and winter quarters. there was no focus group in the spring quarter due to lack of student volunteers. the fall quarter focus group consisted of five students and the winter quarter focus group consisted of three students. the author facilitated both focus groups. these focus groups allowed for more in-depth discussion of information literacy understanding and student satisfaction with the course. questions covered in the focus group sessions included: relevance of the course to the students, using what had been learned in the course, information and society, course improvements, and any other issues the students wanted to discuss. the author transcribed the audio recordings of each focus group session for open content analysis. these results were compared to those from the pre and post test results as a reliability check. results results from coding responses from the pre and post-tests and the transcripts of the focus group sessions are reported below. table 1 shows the number of students who responded to each question on the tests. table 2 summarizes the main categories or themes that were found after coding the student responses to the test questions. the percentage of the responses coded to each category are reported in parentheses. table 1 number of student responses question pre-test post-test 1 86 65 2 86 66 3 85 66 4 81 62 5 68 59 6 n/a 58 preand post-tests: defining information literacy results from coding student responses on the pre-tests showed three main categories, which together totaled 85% of the responses. these categories were: guesses about the definition of information literacy, information literacy meant finding information, and information literacy meant understanding information. the majority of the student responses fell under guesses. as one student in the spring 2009 section wrote, “it does not mean anything to me because i never heard of it.” or, as another student wrote in the fall 2008 quarter, “it means literacy which is…informative.” on the post-tests, the top categories of defining information literacy were: guesses, information literacy meant finding and using information, and information literacy meant using information. the number of responses coded as guessing dropped for the post-tests and no longer formed the majority of the responses. on the post-test a student from fall 2008 wrote that information literacy was “being aware of whats [sic] going on around you, getting info quickly, evaluating sources for credibility, using databases, citing sources correctly, and a lot more.” there were ways of defining information literacy in the post-tests that were not present from analysis of the pretests. these categories included: using information in a positive manner, the understanding that information is everywhere and everything, using information in daily life, and creating knowledge out of information. evidence based library and information practice 2010, 5.1 86 table 2 summary table of major categories of coded student responses on preand post-tests question/topic pre-test categories post-test categories 1. what does information literacy mean to you? guessing (60%) finding information (15%) understanding information (10%) guessing (22%) finding and using information (12%) using information (12%) 2. personal relevance of course learn to use library/library resources (34%) learn research skills (20%) learn the purpose of the course (10%) learn to use databases (28%) learn research skills (18%) learn to cite sources (18%) 3. connection between information and power empowerment (28%) information/knowledge is power (25%) get the “upper-hand” (10%) empowerment (20%) applying information gives power (16%) need information for power (15%) 4. connection between information literacy and society empowerment (55%) place self above others (19%) help others (9%) empowerment (52%) place self above others (16%) help others (14%) 5. modes of accessing information internet (41%) books/library (19%) media (13%) internet (42%) media (14%) someone (13%) pre and post tests: personal relevance in response to the question on the pre-tests of what students wanted to obtain from the class, the main categories included: wanting to learn to use the library and library resources, learning research skills, and learning what the course was about. these three categories consisted of 64% of the coded responses. recurring categories also included: wanting to get a good grade, learning to cite sources, and learning better writing skills. as one student from winter quarter stated: “honestly, i want to get a good grade and get it over with.” on the post-tests, when students responded to what they believed was the most relevant thing they learned in the course, the main categories were: learning to use research databases, learning how to research, learning how to cite sources, and learning to evaluate sources. almost half of the responses noted that learning to use databases was most relevant. preand post-test results: information, power, and society all but one of the responses on the pre-tests to the question of there being a connection between information and power were affirmative. the main themes from coding the students’ explanations of the connection between information and power were: information allowed the student to empower him or herself, information or knowledge was power, and that information allowed the students to have power over others because they could gain the “upper-hand” over those who did not possess the same information. responses from the post-tests fell into three main categories: empowerment of the student, applying information gave the student power, and that the student needed to have information in order to have power. a student from winter quarter wrote on the post-test, “yes what you do with the information you know is what gives you power.” all students affirmed that there is a connection between information and power. evidence based library and information practice 2010, 5.1 87 in response to the question of how being information literate impacted their place in society, the main categories from the pre-tests were: empowerment of the student, enabled the student to place him or herself above others in a social hierarchy, and enabled the student to help others. on the post-tests these same three categories were the top categories (empowerment of the student, placing the student above others, and helping others). a student from winter quarter wrote on the post-test, “to be able to find the information you need is important because you can use that information to make changes in society that could impact more than just yourself.” pre -and post test results: accessing information students on the pre-tests indicated that they accessed information mainly through the internet, books and the library, and media sources (such as television, radio, and newspapers). on the post-tests, students reported that they access information through the internet, media sources, and by asking another person for help or information. post-test results: satisfaction the majority of the students, approximately 90%, were satisfied with the course based on responses on the post-tests. many of the students noted that what they learned in this course would help them in other courses at the university. this sentiment is demonstrated by a student who wrote on the fall quarter posttest, “i am very satisfied with this course. i learned about a lot of research and so far it has helped me greatly in school.” focus groups the student volunteers in the focus groups spoke about what they appreciated learning, their changes in perception about the course, and what they felt could be improved in the course. as a participant from the fall quarter focus group stated when asked what was most helpful, “like for me, the databases. like, i love using them now.” students also remarked that they helped family and friends find information (”…she [laboratory partner] hasn’t taken this class yet so i gave her tips…so you can do it all yourself”.). students also stated that their views on the course changed from their initial perceptions of the course as boring and unnecessary to their subsequent perceptions that the course was helpful and that they learned more than they thought they would from the course. as a participant in the fall quarter focus group said, “at first i thought it was like this class, it sounds kind of boring. like, i don’t know if it’s going to help me, but it definitely really helped.” as to improving the class, students wanted “hands-on more” as noted by one participant in the winter quarter focus group. discussion based on the results of the preand post-tests and the focus group sessions, students increased their understanding of information literacy and found information literacy personally relevant. overall, student understanding of information literacy had improved by the end of the course. while the majority of the students, when asked directly to define information literacy, still saw information literacy as finding or using information, the other questions on the posttests showed that students came to broaden their understanding. there were no noteworthy differences among the three quarters in regard to student responses to the questions on the preand post-tests or between focus group sessions. differences in the number of responses received to questions on the pre-tests versus the post-tests (table 1) are due to the number of students in class on the days the preand post-tests were administered. student learning there was definite growth in students’ understanding of the term “information literacy”; this is vital for students to be able to apply what they learn and feel that they have a personal stake in becoming information literate. their perceptions of information, the use of information and the impact of being evidence based library and information practice 2010, 5.1 88 information literate, both on themselves and on society, grew and became more nuanced by the end of this course. the students still focused on competencies when asked directly what information literacy meant “being able to access information of all types in an efficient manner”, but they also understood information literacy as personally important. along with learning more about information literacy, there were changes in students’ perceptions of the course. in contrast to the more nuanced understanding of information literacy that students demonstrated on the post-tests and in the focus group sessions, responses on the pretests showed that some students did not feel that information literacy instruction was important, necessary, or relevant for their academic careers. as one student on the spring quarter pre-test wrote, “there isn’t anything that i want to get out of this class.” this dismissal of information literacy instruction as unimportant or simply another requirement for graduation was prevalent in the pre-test answers. other students noted on the pre-tests that they wanted “to know what this class is for,” get “a good grade,” or earn “ge [general education] credit” by passing the course. these findings support earlier research that students underestimate their need for instruction (freeman, 2004; maughan, 2001). by the end of the course, the majority of the students had learned more than just “…what this class is for.” the majority of students found that information literacy was personally relevant and could positively affect their lives and communities based on their responses to the post-test questions. however, a few students noted the negative uses of information in connection to power. on a pre-test in spring quarter, a student wrote in response to whether there is a connection between information and power, “yes because the more informed you are with everything the more power you have over others.” these negative connotations of the connection between information and power were far outweighed by the positive examples written by students. as a student in fall quarter wrote on the posttest, “the smarter you are, the more things you can do. the more things you can do, the more of an impact you can make on the community.” this is representative of student responses that focused on how becoming information literate empowered them and enabled them to help others. it also shows that students found ways of incorporating information literacy into their lives and found relevancy in becoming information literate. it is interesting to note that while the majority of students guessed when defining information literacy on the pre-tests, they also wrote that being information literate would help them in society. this could be explained by students writing in what they thought was the “correct” answer to the question of the connection between information literacy and society on the pre-tests. this view is supported by a student’s response on the spring quarter pre-test, “i don’t quite know what ‘information literate’ means but it probably can do a lot in one’s life. it can probably help us perform tasks that we find difficult, much easier.” while the main categories did not change from the pre-test responses to the post-test responses for the question about information literacy and society (table 2), the students’ understanding and ability to explain information literacy did change. this could mean that students did in fact grow in their understanding of the impact of being information literate in society. however, more research is needed before drawing definite conclusions in this area. students’ growth throughout the course showed development of a more complex and multi-faceted understanding of information literacy, which supports previous research (johnston & webber, 2003; maybee, 2006). the course was a growth experience for many students. for example, one student in the fall 2008 focus group said, i did a lot of research papers in high school, but my teachers they never, like, they were more worried about oh, your thesis and all that, they weren’t worried like where are you getting the article or evidence based library and information practice 2010, 5.1 89 have to evaluate the author…but this class has really helped me, how to be more accurate and look at information more in depth. this student’s way of looking at and relating to information had changed during the quarter. this view is echoed by what another student wrote on the winter quarter post-test, “i learned things about obtaining information that i never knew. my idea of libraries and information has really changed.” students themselves realized that not just their perceptions but also their understanding changed throughout the quarter. the students expanded their conception of information, sources of information, and uses of information (table 2). while the majority still relied most heavily on the internet for accessing information, on the post-tests they acknowledged that information could come from other, non-textual sources. this too could be seen as an expansion of understanding and an integration of what they were learning in the course to their lives outside of the classroom; many of the students noted that they accessed information by talking with other people about current events, family history, and other topics and used this information to help their families, become more informed voters, or solve a problem. student satisfaction contrary to anecdotal evidence from other faculty members that students dislike or are not satisfied with the information literacy course, this study showed that the majority of the students were satisfied with the course. a common response from students was that they were surprised by the helpfulness and usefulness of the class. a student noted on the winter quarter post-test “i am well satisfied with the course, it ended up being more than i expected.” a student in the winter quarter’s focus group said, “i’m not bored. i really, really enjoy this class…i feel that i’ve learned so much more than pretty much most of my classes.” as their understanding of information literacy increased, so did their appreciation of the relevancy of the course, and their satisfaction with what they learned. these results support previous research findings that relevancy is very important to student motivation in learning (baker, 2006). while some students were convinced that they did not need the course at the beginning of the quarter, the majority were satisfied at the completion of the course and believed the course was, and would continue to be, helpful to them. course improvement while most students were satisfied with the course, they had suggestions for improvement. the most consistent comments were that students liked hands-on activities and small group work in class. they suggested increasing the number of in-class activities and time in the computer lab. as one student from the fall quarter focus group said, “more hands on experience right there and then” would improve the course. as a result of these suggestions, the course will be modified to include more group activities and work in the computer lab when possible. this should be especially helpful for those students who may be kinesthetic learners or more comfortable working in smaller groups. as students responded well to what they believed to be personally relevant, more emphasis will be given to demonstrating how information literacy intersects with their lives. by looking at issues that directly affect the students (control of their personal information on social networks and security of data on bluetooth-enabled devices, to name two examples), they can see how being information literate is important for everyday issues and not just in the classroom. more reflective activities (edwards & bruce, 2004) will also be incorporated to try to increase student engagement with processes and issues discussed. while still covering the course objectives, the course will be modified to more fully incorporate the personal relevance and social impact frames (bruce, edwards, & lupton, 2006) to increase student engagement and understanding and to extend the concepts of assessment in the course to more fully evidence based library and information practice 2010, 5.1 90 capture a picture of student learning. these changes to the course will be implemented in the coming academic years. this study will be continued to determine if the changes improve the course by increasing student learning. limitations and future research as this study did not use a random sample and was conducted at one university, the results cannot be generalized. future research could include expanding this study’s approach to assessing information literacy instruction to other information literacy courses at cal state east bay and to other universities. this would increase the ability to generalize and the validity of the results this study. conducting longitudinal studies would provide more information about the impact of information literacy instruction on students throughout their academic careers. conclusion standards for information literacy instruction and learning often place an emphasis on skills and competencies; however, additional assessments of information literacy instruction in terms of relevancy and impact for the student can be undertaken to evaluate student learning and facilitate course improvement using the results. this study showed that pre and post-tests and focus group sessions can be used to assess information literacy instruction by gauging student learning, as well as student satisfaction. students’ reflections and evaluations are useful indicators of learning. while some students may start an information literacy course believing that they do not need it (gross & latham, 2007), many will acknowledge learning something useful by the end. information literacy instruction is not inherently boring to students and many become engaged as they see the relevance of what they are learning to present tasks and future work. students remember what is personally relevant to them, a deceptively simple statement that should not be forgotten when planning and implementing information literacy courses. content knowledge and skill competencies are important for students to learn, but the ability to see relevancy and the impact of information literacy in life is equally important (bruce, 2008). acknowledgements funding for this research was provided through a 2008-2009 california state university, east bay new faculty grant. this article was first presented as a paper at the 5th international evidence based library and information practice (eblip5) conference, stockholm, sweden, june 2009. references american library association. (1989). presidential committee on information literacy: final report. washington, d.c.: american library association. retrieved from http://www.lita.org/ala/mgrps/divs/ac rl/publications/whitepapers/presidenti al.cfm. andersen, j. (2006). the public sphere and discursive activities: information literacy as sociopolitical skills. journal of documentation, 62, 213-228. association of college & research libraries. (2000). information literacy competency standards for higher education. chicago, il: american library association. retrieved from http://www.ala.org/ala/mgrps/divs/ac rl/standards/informationliteracycomp etency.cfm. auerbach, c. f., & silverstein, l. b. (2003). qualitative data: an introduction to coding and analysis. new york: new york university press. baker, l. (2006). library instruction in the rearview mirror: a reflective look at the evolution of a first-year library program using evidence-based http://www.lita.org/ala/mgrps/divs/acrl/publications/whitepapers/presidential.cfm http://www.lita.org/ala/mgrps/divs/acrl/publications/whitepapers/presidential.cfm http://www.lita.org/ala/mgrps/divs/acrl/publications/whitepapers/presidential.cfm http://www.ala.org/ala/mgrps/divs/acrl/standards/informationliteracycompetency.cfm http://www.ala.org/ala/mgrps/divs/acrl/standards/informationliteracycompetency.cfm http://www.ala.org/ala/mgrps/divs/acrl/standards/informationliteracycompetency.cfm evidence based library and information practice 2010, 5.1 91 practice. college & undergraduate libraries, 13, 1-20. bruce, c. (1997). the seven faces of information literacy. adelaide: auslib press. bruce, c. (2000). information literacy programs and research: an international review. the australian library journal, 49, 209-218. bruce, c. (2008). informed learning. chicago, il: association of college and research libraries. bruce, c, edwards, s., & lupton, m. (2006). six frames for information literacy education: a conceptual framework for interpreting the relationships between theory and practice. italics, 5,1-18. retrieved from http://www.ics.heacademy.ac.uk/italic s/vol51/pdf/sixframes_final%20_1_.pdf. bundy, a. (ed). (2004). australian and new zealand information literacy framework: principles, standards and practice (2nd ed.). adelaide, australia: australian and new zealand institute for information literacy. retrieved from http://www.anziil.org/resources/info% 20lit%202nd%20edition.pdf. california state university, east bay. (n.d.). information literacy learning outcomes for lower division general education. retrieved from http://www20.csueastbay.edu/about/a ccreditation/files/pdf/slos%20inform ation%20literacy%20lower%20divisi on%20approved.pdf. california state university, east bay. (2008). mission, values, and vision (2006). retrieved from http://www20.csueastbay.edu/about/st rategic-planning/mission-valuesvision.html. dunn, k. (2002). assessing information literacy skills in the california state university: a progress report. the journal of academic librarianship, 28, 26-35. edwards, s. l., & bruce, c. (2004). the assignment that triggered change: assessment and the relational learning model for generic capabilities. assessment & evaluation in higher education, 29, 141-157. freeman, c. a. (2004). the relationship of undergraduate students’ selfassessment of library skills to their opinion of library instruction: a selfreporting survey. the southeastern librarian, 52,39-46. retrieved from http://selaonline.org/southeasternlibr arian/fall2004.pdf. gross, m., & latham, d. (2007) attaining information literacy: an investigation of the relationship between skill level, self-estimates of skill, and library anxiety. library & information science research, 29, 332-353. johnston, b., & webber, s. (2003). information literacy in higher education: a review and case study. studies in higher education, 28, 335-352. maughan, p. d. (2001). assessing information literacy among undergraduates: a discussion of the literature and the university of california-berkeley assessment experience. college & research libraries, 62, 71-85. maybee, c. (2006). undergraduate perceptions of information use: the basis for creating user-centered student information literacy instruction. the journal of academic librarianship 32,7985. webber, s., & johnston, b. (2000). conceptions of information literacy: new perspectives and implications. journal of information science, 26, 381-397. http://www.ics.heacademy.ac.uk/italics/vol5-1/pdf/sixframes_final%20_1_.pdf http://www.ics.heacademy.ac.uk/italics/vol5-1/pdf/sixframes_final%20_1_.pdf http://www.ics.heacademy.ac.uk/italics/vol5-1/pdf/sixframes_final%20_1_.pdf http://www.anziil.org/resources/info%20lit%202nd%20edition.pdf http://www.anziil.org/resources/info%20lit%202nd%20edition.pdf http://www20.csueastbay.edu/about/accreditation/files/pdf/slos%20information%20literacy%20lower%20division%20approved.pdf http://www20.csueastbay.edu/about/accreditation/files/pdf/slos%20information%20literacy%20lower%20division%20approved.pdf http://www20.csueastbay.edu/about/accreditation/files/pdf/slos%20information%20literacy%20lower%20division%20approved.pdf http://www20.csueastbay.edu/about/accreditation/files/pdf/slos%20information%20literacy%20lower%20division%20approved.pdf http://www20.csueastbay.edu/about/strategic-planning/mission-values-vision.html http://www20.csueastbay.edu/about/strategic-planning/mission-values-vision.html http://www20.csueastbay.edu/about/strategic-planning/mission-values-vision.html http://selaonline.org/southeasternlibrarian/fall2004.pdf http://selaonline.org/southeasternlibrarian/fall2004.pdf evidence based library and information practice 2010, 5.1 92 appendix 1: preand post-test instructions and questions please answer the following questions to the best of your ability. your answers will not be graded. please do not put your name on this test. your answers will help me make this class better. 1. what does the phrase “information literacy” mean to you? 2. what do you want to get out of this class? (for pre-test) what did you learn from the class that was most relevant for you? (for post-test) 3. do you think there is a connection between information and power? please explain. 4. how do you think being information literate impacts your place in society? please explain. 5. how do you access and use information in your life? please explain or give an example. 6. how satisfied with the course are you? what could we do to improve the course? (for post-test) evidence based library and information practice evidence based library and information practice 2009, 4:3  evidence based library and information practice       article    an emerging theory for evidence based information literacy instruction in school  libraries, part 2:  building a culture of inquiry    carol a. gordon  associate professor  school of communication, information and library studies  rutgers, the state university of new jersey  new brunswick, new jersey, united states of america  email: cgordon@scils.rutgers.edu     received: 17 july 2009          accepted: 09 august 2009      © 2009 gordon. this is an open access article distributed under the terms of the creative commons attribution  license, <http://creativecommons.org/licenses/by/2.0>, which permits unrestricted use, distribution, and  reproduction in any medium, provided the original work is properly cited.    abstract     objective – the purpose of this paper is to articulate a theory for the use of action  research as a tool of evidence based practice for information literacy instruction in  school libraries. the emerging theory is intended to capture the complex  phenomenon of information skills teaching as it is embedded in school curricula.  such a theory is needed to support research on the integrated approach to teaching  information skills and knowledge construction within the framework of inquiry  learning. part 1 of this paper, in the previous issue, built a foundation for emerging  theory, which established user‐centric information behavior and constructivist  learning theory as the substantive theory behind evidence based library instruction  in schools. part 2 continues to build on the information search process and guided  inquiry as foundational to studying the information‐to‐knowledge connection and  the concepts of help and intervention characteristic of 21st century school library  instruction.     methods – this paper examines the purpose and methodology of action research as a  tool of evidence based instruction. this is accomplished through the explication of  three components of theory‐building: paradigm, substantive research, and  metatheory. evidence based practice is identified as the paradigm that contributes  values and assumptions about school library instruction. it establishes the role of  evidence in teaching and learning, linking theory and practice.  action research, as a  tool of evidence based practice is defined as the synthesis of authentic learning, or  performance‐based assessment practices that continuously generate evidence  throughout the inquiry unit of instruction and traditional data collection methods  19 mailto:cgordon@scils.rutgers.edu evidence based library and information practice 2009, 4:3  20 typically used in formal research. this paper adds social psychology theory from  lewin’s work, which contributes methodology from gestalt psychology, field theory,  group dynamics, and change theory. for lewin the purpose of action research was  social reform, while action research in education targeted self through the  improvement of practice.  the dichotomy between purposes of self and society is  resolved by the lewin‐dewey connection, where the reiterative cycle of action and  reflection is the basis for a common intent for both types of action research. dewey’s  approach comprises the metatheory for emerging theory: a philosophy of purpose  and methodology that determines how the research is done.    results – the emerging theory developed in this paper postulates that evidence  based information literacy instruction uses action research for two purposes. self‐ oriented action research (ar(s1)) targets self‐improvement on the local level of  teaching and learning in school libraries;  social‐oriented action research (ar(s2))  targets social reform on the global level of educational improvement. corollaries of  the theory indicate a research agenda and methodologies for the research.     conclusion –  implicit  in  the  content  of  the  research  is  methodology  that  evolves  from  the  distinction  between  the  purposes  of  self‐  and  social‐oriented  action  research. clearly, evidence is generated in the field of teaching and learning that is  situated  in  theory‐based  practices,  such  as  user‐centered  information  processing,  constructivist  learning, and a culture of  inquiry  that grows  from social processes.  librarianship is well suited to developing practitioner‐researchers who are proficient  in making the information‐to‐knowledge connection that informs their professional  performance.     introduction: building a foundation    the overall purpose of this paper, written in  two parts, is to articulate a theory for evidence  based information literacy instruction in  school libraries. this introduction serves to  summarize the foundation for theory building  in part 1 of the article, published in the  previous issue of eblip (gordon “emerging”).  a comprehensive theory is needed to guide  formal research for this complex phenomenon  characterized by:   • inquiry learning,   • information, resources, technology,  and their attendant skills, and   • collaboration of school librarians and  classroom teachers who bring their  respective expertise in information  skills and content knowledge to  teaching through inquiry.     their common intent is to design, implement,  and evaluate learning tasks that facilitate an  information‐to‐knowledge connection.  understanding action research in this context  is critical to theory building, which is  described in part 1 as evidence based inquiry  learning. teaching decisions are made from  evidence continuously generated from student  work. this kind of pedagogy is labeled  “authentic learning” to denote its derivation  from the authentic assessment movement in  education, currently known as “authentic  education” (wiggins).  for the purpose of this  paper, it is a method of action research that  aims to improve instruction and the  performance of user‐learners through  transactions between learners and teachers  characterized by continuous assessment and  feedback.      a focused review of the literature in part 1  presented research for building theory to  support empirical study of this kind of  instruction‐specific action research (gordon  “fish”; “students”; “putting”). user‐learners  in these studies engage in everyday  information seeking (“putting”) and highly  evidence based library and information practice 2009, 4:3  structured, academic information tasks nested  in real‐world learning tasks (“students”). the  learning task is, in fact, a series of formative,  activity‐based assessments that continuously  generate evidence of learning, as well as gaps  and misconceptions. these studies reveal a  distinction between micro and macro learning  tasks that are useful for studying the learning  task as a component of the inquiry process  (gordon “emerging”). on the micro level  user‐learners confront information and engage  in a series of information processing tasks, as  described in kuhlthau’s information search  process (kuhlthau “facilitating”).  user‐ learners apply basic information searching  skills, as well as critical thinking, to craft  connections between new information and  existing knowledge. micro tasks occur  throughout the information search process as  the knowledge built from found information  generates questions that require more  information. on the macro level the learning  task, designed by an instructional team of  school librarians and classroom teachers,  shapes the inquiry. user‐learners relate  emerging knowledge to a problem or issue in  the real world. the learning task is relevant,  engaging, and rigorous to sustain the user‐ learner’s interaction with information and  emerging knowledge. the macro level is  characterized by continuous assessment  through self‐reflection, self‐correction, and  self‐regulation as new knowledge is  continuously built from found information.  the role of evidence in determining the user‐ learner’s progress is essential to both kinds of  tasks, yet formal research studies do not  differentiate between micro and macro tasks  as part of inquiry learning. this distinction  between tasks of information handling and  knowledge building is critical to empirical  research that studies the information‐to‐ knowledge connection in order to determine  how information searching informs  knowledge construction, and how the learning  task informs information seeking. the studies  reviewed in part 1 pointed to the need for  more research on interdependent and group  work, as well as individual user‐learner  information and learning behaviors.     one of the action research studies described in  part 1 identified action research as a tool of  evidence based practice (gordon “study”).  that project tested a three‐dimensional action  research training model, implemented by  school librarians and collaborating teachers,  which involved integration of authentic  teaching with traditional qualitative research  methods. the study investigated how the  model worked and the impact of the action  research experience on the practice of school  librarians. the formal researcher operates in  the third dimension, simultaneously collecting  data and mentoring school librarians. in the  first dimension, the librarians function as  designers and implementers of authentic  learning tasks and assessments. in the second  dimension they are practitioner‐researchers  who identify a problem in their practice, and  then develop and implement their own action  research projects. the mentor‐researcher  guides school librarians in formulating  research questions, outlining proposals,  constructing theoretical frameworks,  collecting and analyzing data, and presenting  findings.     the research objective was to explore the  elements of the three‐dimensional model. the  researcher collected data from email  transactions, telephone conversations,  interviews, meetings during on‐site visits, a  questionnaire, and a content analysis of the  documentation submitted by the school  librarians. findings indicated that the three‐ dimensional training model elevated the  quality of action research to that of formal  academic research.     practitioner‐researchers engaged in research  that modeled methodology for students doing  inquiry learning and for teachers who  collaborated with school librarians to conduct  action research. action research emerged as a  professional development tool that uses the  school library as a laboratory where educators,  as well as students, can learn to be reflective  and self‐critical through systematic collection  and analysis of evidence. while interactions  between librarians and teachers revealed  underlying tensions of collaboration, the  21 evidence based library and information practice 2009, 4:3  22 action research project had a positive effect on  the librarians’ practice, their confidence in  managing the collaboration process, and their  perceptions of themselves as leaders and  researchers. the current study raises questions  about the role of the school librarian in  collaborative action research and the role of  action research in school reform. further  research is needed to develop other training  models, as well as models for implementing  and studying action research as a tool of  evidence based information literacy  instruction. in subsequent years replication of  this study has had a transformative effect on  the school district.  decision‐making in  schools, especially in the context of  instruction, is driven by a culture of evidence  based practice, inquiry, collaboration, and  action research.      theory building supports research that defines  a culture of inquiry unique to school libraries  in which evidence based practice and action  research shape teaching and learning. the goal  of building theory to support the study of  action research based teaching and learning is  twofold: to determine the purpose and the  methodology for studying action research.  this paper (part 2) studies the purpose and  methodology of action research to gain  insights into why and how the study of action  research is conducted. the framework used  for theory building bridges practice and  theory, creating a theory of methodology for  researching evidence based information  literacy instruction.       methodology for theory building      the framework for theory building used in  this paper is dervin’s sense‐making  methodology and its three components:  paradigm, substantive theory, and metatheory  (dervin “what” 26), as illustrated in fig.1.   paradigm: beliefs, assumptions, values evidence based practice substantive theory (type 1): information behavior cognitive science education social psychology (lewin) metatheory (type 2): philosophy of purpose and methodology action and reflection (dewey)   fig. 1. three components of theory building (adapted from dervin “what” 26)  evidence based library and information practice 2009, 4:3  paradigm and the culture of inquiry    the paradigm for building theory for the  study of action research‐based information  literacy instruction is evidence based practice.  a paradigm consists of the beliefs,  assumptions, values and techniques accepted  by a community of practitioners (fig. 1). for  example, everyone is considered a researcher  in the evidence based practice paradigm.  evidence is valued as the currency for  improvement by students and their teachers.  educators are empowered to direct their  professional growth. learners are empowered  to become their own best critics. these beliefs  establish expectations for teaching and  learning, and they predicate roles. they  influence dispositions to learning and  decisions about what is worth knowing and  learning to do. evidence based practice is  implicit in conceptualizing action research as a  way for practitioners to improve their  performance. todd described three sources of  evidence:   • evidence for practice, or the research  that supports best practice.   • evidence in practice, or the expertise  of professionals as they define their  practice.   • evidence of practice, or the evidence  generated by practice.   when applied to instruction, evidence based  practice is a function of best practice: evidence  based practice elicits documentation that  demonstrates how school libraries, which can  be considered agents of educational reform  not yet integrated with mainstream education,  make a difference in teaching and learning.     paradigms are integral to change and reform.  a paradigm shift from professional  commitments to shared assumptions takes  place when an anomaly “subverts the existing  tradition of scientific practiceʺ (kuhn 6). the  anomaly that subverts traditional  bibliographic teaching is instruction that is  learner‐ rather than resource‐centered. such an  approach re‐conceptualizes librarianship to  value the question, “did patrons learn  anything?” rather than, “did patrons find  what they needed?”  evidence based practice  suggests shared values of self‐reflection and  the continuous improvement of practice. this  paradigm shift adds the dimension of  documentation for the purpose of determining  the difference school library instruction makes  in the teaching and learning mission of the  school. while supporting this mission, school  libraries go against the grain of traditional,  classroom‐centered schooling. this poses a  conundrum for school librarians who support  teaching and learning through school library  based inquiry which challenges the way these  commodities are delivered through  traditional, classroom‐centered schooling.  these kinds of shifts are what kuhn describes  as “scientific revolutions”, ʺthe tradition‐ shattering complements to the tradition‐bound  activity of normal science” (6).  action  research‐based information literacy instruction  constitutes a shift in the culture of teaching  and learning in schools that has the potential  to reform education. streatfield and markless  define school culture as pedagogy, resources,  organizational structures, and the key players  in teaching and learning.  they report that  learning through the school library is  influenced by the school culture in which the  library operates. research is needed to  understand how inquiry and action research‐ based information literacy instruction in  school libraries can influence school culture.  there is no comprehensive theory to support  research that studies the counter‐culture of the  school library learning environment in the  mainstream culture of schooling. such a  theory emerges from theory‐building based on  the paradigm of evidence based practice.      a paradigm is essential to scientific inquiry, as  well as to defining best practice. ʺno natural  history can be interpreted in the absence of at  least some implicit body of intertwined  theoretical and methodological belief that  permits selection, evaluation, and criticismʺ  (kuhn 16‐7). a kuhnian approach defines  paradigm in two ways:    on the one hand, it stands for the  entire constellation of beliefs, values,  techniques, and so on shared by the  members of a given community. on  23 evidence based library and information practice 2009, 4:3  the other, it denotes one sort of  element in that constellation, the  concrete puzzle‐solutions which,  employed as models or examples, can  replace explicit rules as a basis for the  solution of the remaining puzzles of  normal science (kuhn 175).    the evidence based practice paradigm, in the  first sense of kuhn’s definition, suggests the  lay person has a role in generating new  knowledge. this timely idea drives social  networking, for example, empowering  individuals to participate in virtual  communities to share knowledge, remix  content, and create unique representations of  content and knowledge. it is a mindset that  thrives in a participatory and collaborative  environment where each participant is a  scientist who ponders problems and forms  hypotheses (kelly).  evidence based practice  values the relationship of research to practice,  and the critical role theory plays in defining  best practice. action research, as a tool of  evidence based practice, not only generates  evidence; it prescribes techniques and  methods that empower practitioners to  continuously improve their practice. ʺone of  the things a scientific community acquires  with a paradigm is a criterion for choosing  problems that, while the paradigm is taken for  granted, can be assumed to have solutionsʺ  (kuhn 37). paradigms set the research agenda  by indicating research questions that are  relevant to the shifts in values and  assumptions. ʺin the absence of a paradigm, or  some candidate for paradigm, all the facts that  could possibly pertain to the development of a  given science are likely to seem equally  relevantʺ (kuhn 15).     in the second sense of kuhn’s definition of  paradigm, evidence based instruction is  inherent in action research as the “concrete  puzzle‐solution.” it is an exemplar that can  replace existing rules, i.e., traditional teaching  practices exhibited by teacher‐centered talk  rather than learner‐centered activity.  the new  rules are exhibited by active learning and self‐ reflection. the evidence based practice  paradigm raises questions about the nature of  learning tasks, the role of evidence in  improving learning and teaching, and purpose  and methodology of action research when it  functions as a tool of evidence based practice.   in a broader sense, it frames the overarching  question of educational reform for 21st century  learning, presenting evidence based  information literacy instruction as a solution  to outmoded paradigms of teaching and  learning.    in summary, paradigm is a foundational  concept for theory building. it bridges the gap  between research and practice, and between  problem and solution in the following ways: 1)  paradigm defines a culture of practice in a  community that sets the stage and functions as  the context for the research; 2) paradigm sets  the research agenda, pointing to relevant  questions and problems that originate in the  culture and context to be studied; 3) paradigm  points to reform, setting the purpose for the  research and indicating solutions; 4) paradigm  suggests techniques, or methods, to achieve  the research purpose.     substantive theory and the culture of inquiry    substantive theory, or type 1 theory (fig. 1),  provides theory that is empirically tested to  support further research.  part 1 of this paper  takes an interdisciplinary approach,  presenting a triumvirate of interlocking  substantive theories: 1) user‐centered  information theory (dervin and nihlan;  belkin; kuhlthau; 2) learner‐centered  cognitive theory (kelly’s cognitive, affective,  and behavioral perspectives of learning; 3)  interventionist teaching (vygotskii’s zone of  proximal development 4) constructivist  learning theory (dewey’s experiential  learning; piaget’s constructivism. these  theories support the distinction between the  macro learning task and micro information  tasks that offer a way of studying how user‐ learners make the information‐to‐knowledge  connection. information behavior research and  theory point to researchable questions about  micro information tasks, i.e., the purpose of  the research and methods appropriate to the  purpose. cognitive psychology and education  24 evidence based library and information practice 2009, 4:3  provide the same functions for theory based  research of macro learning tasks.     substantive theory results from observation  (dervin “what”) and provides theoretical  frameworks for empirical studies of evidence  based information literacy instruction. lewin’s  addition of  social psychology to information  and learning theory supports research needed  to understand information behavior and  learning through group processes, as well as  through individual information processing.  highly collaborative instruction in school  libraries and virtual environments present a  need for this kind of research, which is  supported by constructivism; construction of  meaning is seen as a social process (vygotskii).  it is understood as an interaction in which  user‐learners’ thoughts and communicative  actions take shape and where user‐learners, in  various ways, coordinate their actions. this  interpretation is about intersubjectivity, which  is connected to user‐learners’ action, both their  verbal communication and the operations that  they perform when they search for  information through the library (limberg and  alexandersson 1‐2). while constructivism  recognizes that learning is social (vygotskii), it  is not often applied to interdependent, or  collaborative learning to the extent that it is  applied to independent learning. limberg and  alexandersson reported students’  interpretations of meanings of school libraries  that they experienced reflected individual,  rather than collective, action and  understanding. they concluded that “…to  inspire alternative meanings of the school  library, teachers and librarians are the key  actors and need to interact differently with  students” (11). the addition of social  psychology to the triumvirate of substantial  theory adds a socio‐cultural dimension to the  way teaching and learning is viewed in the  school library. lewin’s social psychology  theory includes gestalt psychology, field  theory, group dynamics, and change theory,  all of which contribute a cohesive set of beliefs  and research techniques that inform action  research purpose and methodology (“action”;  field). social psychology broadens the scope of  the empirical study of inquiry learning in  school libraries from individual learning to  learning through collaboration and group  processes.     in summary, substantive research supports  emerging theory in the following ways: 1)  substantive theory provides a family of  relevant research, making it possible to move  theory through generations to maintain its  relevancy to evolving practice; 2) substantive  theory functions as a foundation for creating  distinctions in phenomena, such as macro and  micro inquiry tasks, that are contexts for  research; 3) substantive theory, like paradigm,  originates in practice and suggests  researchable questions for the research; 4)  substantive theory provides models of  methods relevant to the emerging theory and  subsequent research. this article adds social  psychology theory to the body of substantive  theory to expand the purpose of and  methodology for studying action research‐ based information literacy instruction to  include interdependent learning, or group  processes. lewin’s methodologies offer  interesting options for the application of field  theory and group dynamics, for example, to  study social learning.    metatheory and the culture of inquiry    the third component to theory‐building is  metatheory (type 2 theory) which provides  the philosophy behind the purpose and  methodology of emerging theory (fig. 1). the  lewin‐dewey connection forms the  metatheory for the emerging theory of action  research‐based information literacy  instruction. through the concept of  experiential learning and the importance of  relevancy in education, dewey’s metatheory  sets a progressive agenda for educating youth  that is participatory rather than passive. in  addition, metatheory based on dewey’s  philosophy of education makes the connection  between inferential thinking and the process  of action and reflection, not unlike lewin’s  description of the action research cycle. the  connection between lewin’s substantive  theory and dewey’s metatheory results in a  theory of methodology.  25 evidence based library and information practice 2009, 4:3  26 relationships among paradigm, substantive  theory and metatheory    this paper proposes that sense‐making theory  building (dervin “what”) clarifies the purpose  of action research and suggests methods that  ensure its legitimacy so that it can be a viable  subject for study through empirical research.  paradigm, substantive theory, and metatheory  are not discrete components of theory  building. rather, they inform each other, in no  particular order, to define purpose and  methodology inherent in emerging theory.  this seemingly chaotic model “…catapults us  into an world where we must accept the idea  that different observers whose observing are  driven by different theories are necessarily  operating in different worlds and creating  different (type 1) theories even while looking  in the same direction from the same vantage  point.” (dervin “what” 25‐6) the significance  of the relationship between paradigm, type 1,  and type 2 theory is not unlike the  relationship between practice and theory. for  example, evidence based practice establishes a  culture of inquiry that has implications for  research. it contributes beliefs, values and  assumptions of that culture. paradigm is  critical to establishing what best practice looks  like. because it is practice, rather than theory,  it provides rich opportunities for identifying  problems and issues that spark empirical  research, and implies solutions to be tested by  empirical research, which in turn, become  substantive theory for further theory building.    the rationale for using this framework is  borrowed from dervin’s sense‐making  methodology, which is “…based on the  premise that it is philosophically anchored  methodological consideration which builds  the bridge between these two kinds of theory”  (dervin “what” 26). dervin considers this  kind of methodological explication that  connects substantive theory with metatheory  to be a third kind of theory ‐ theory for  methodology (fig. 2). dervin’s theory for  methodology provides a rationale and  framework for studying what methodology  does to theory, particularly with regard to the  connections between substantive theory and  metatheory. the relationships among these  three types of theory generate a  conceptualization of action research that  clarifies its purpose, or how we think about it,  and its methodology, or how we do it.      action research how do we think about it? how do we do it? substantive theory theory of methodology metatheory sense-making methodology     fig. 2. how we think about action research and how we do it (adapted from dervin  “what” 26)  evidence based library and information practice 2009, 4:3  the substantive theory and metatheory that  contribute to a theory of methodology for  studying action research based teaching are  embedded in social psychology through the  work of lewin, and in the philosophy of  education through the work of dewey. in this  study of theory building, substantive theory,  as shown in fig. 2, is lewin’s social  psychology theory describing the cycle of  action research methodology (type 1 theory).  it dovetails with metatheory (type 2 theory),  as shown in fig. 2. since “…it is widely agreed  there is no observing which is not guided by  theory”, metatheory consists “…of  philosophically grounded assumptions about  the phenomena and about how to study it”  (dervin “what” 25).  as noted earlier,  dewey’s philosophically grounded  assumptions are known as learning by doing,  or experiential learning. dewey’s descriptions  of action and reflection that lead to inferential  thinking processes add cognitive structure to  the transparent processes of reflective  thinking. this structure indicates how to study  learning and thinking processes. the lewin‐ dewey connection constitutes the theory of  methodology for action research‐based  information literacy instruction.     lewin’s substantive theory and its role in  action research purpose and methodology     the lewin‐dewey connection forms a theory  of methodology for studying action research‐ based teaching. the discussion of this  connection between substantive theory and  metatheory is prefaced by an exploration of  how lewin’s work in social psychology  contributes to defining action research  purpose and methodology.  although  consensus is lacking concerning the origin of  the term, “action research”, lewin (“action”)  is generally accepted as the first to coin the  term (argyris, putnam, and smith; kemmis  and mctaggart). lewin distinguishes action  research from formal research.     the research needed for social practice  can best be characterized as research  for social management or social  engineering. it is a type of action‐ research, a comparative research on  the conditions and effects of various  forms of social action, and research  leading to social action. research that  produces nothing but books will not  suffice (lewin field  202‐3).    the dichotomy of self and society in action  research purpose     lewin’s work clearly established the social  purpose of action research by applying group  dynamics to problems of drug addiction,  crime, and the rehabilitation of world war ii  veterans. in this tradition action research is  ʺthe systematic collection of information that is  designed to bring about social change”  (bogdan and biklen 223). although lewin  used scientific experimentation, action  research for social purposes resembles  ethnographic research in its cultural and  contextual sensitivity. action research has  spread beyond the boundaries of social  psychology and social purpose to health and  education, where it is has assumed the goal of  improvement of practice. initially, however,  educators viewed action research as an agent  of social change when applied to school  reform. schwab developed a holistic  perception of curriculum that identified four  commonplaces of schooling: teachers,  students, subject matter, and milieu. as the  definition of curriculum narrowed to what  was explicitly taught, attention shifted to  instruction. stenhouse noted, ʺ...we should  move from product and process models of  curriculum development toward a research  modelʺ (125).  he believed that curriculum  development presupposed that teachers  engage in systematic study of their own work,  because the world of school can be changed  only by teachers who understand it. this  influenced mckernan, who developed a  handbook of curriculum in which he stated,  ʺcurriculum will only be improved by  researching our teachingʺ (262). carr and  kemmis described moreno’s definition of this  brand of action research as:    27 evidence based library and information practice 2009, 4:3  ... simply a form of self‐reflective  enquiry undertaken by participants in  social situations in order to improve  the rationality and justice of their own  practices, their understanding of these  practices, and the situations in which  the practices are carried out (162).    lewin used collaborative action research with  teachers at the teachers training college at  columbia university, leading to its application  in educational settings. in this setting corey  (70) defined action research as the process  through which practitioners study their own  practices to solve personal practical problems  by their involvement in the research as well as  the application of solutions.     we are convinced that the disposition  to study…the consequences of our  own teaching is more likely to change  and improve our practices than is  reading about what someone else has  discovered of his teaching.     when action research takes on self‐ rather than  social‐orientation, it is defined as a “deliberate,  group, or personally owned and conducted,  solution oriented investigation” (boomer 8).  the teacher‐as‐researcher movement of the  1970s exemplified the application of action  research for the purpose of improving  practice.     action research in education    action research, as it applies to education, and  particularly to information behavior and  inquiry in school libraries, inherits both facets  of purpose: improvement of practice on the  local level and global educational reform. in  either case, action research is an instrument of  change., but in every case it involves changing  the behavior of individuals. the purpose of  action research, then, is defined by its outcome  rather than by its origins. concurrent social‐  and self‐oriented intentions of action research  are demonstrated by its evolution in  education, where a duality of intent is not only  possible, but perhaps inevitable.      the following notation is useful to distinguish  the two intentions of action research identified  in this paper:    ar(s1) = action research that targets  self‐improvement of practice   ar(s2) = action research that targets  social reform      lewin’s social psychology theory  accommodates an integrative, rather than  dichotomous interpretation of action research  intent.  he wrote, “there is nothing more  practical than a good theory” (field 169).  he  viewed the role of theory as an instrument that  could produce insights into problematic  situations. inherent in theory are possible  solutions, or pathways to solutions. “the  creation of an empirically verifiable theory,  lewin knew, was the essence of science;  research therefore had to be guided by the  need to develop an integrated concept of the  processes of group life” (morrow 183).  however, one will not find a theoretical  framework for action research explicitly stated  in lewin’s work. it resides, instead, in the  corpus of his work, which provides a source of  substantive theories that address action  research processes. these theories include  gestalt theory, field theory, and group  dynamics, as shown in fig. 3.   field theory clearly serves a social purpose,  since it is situated in real world problems.  however, field theory is rooted in theory, a  highly individualistic construct. “a gestalt is a  coherent whole. it has its own laws, and is a  construct of the individual mind rather than  ‘reality’” (lewin field 240). this kind of  subjectivism is friendly to the insider nature of  action research as the individual practitioner  interprets the workplace where action research  takes place in a social context, regardless of its  orientation. the relevance of gestalt theory to  action research aimed at self‐improvement is  enhanced when the connection among gestalt,  field theory, and group dynamics is made.  lewin (field 240) defined a “field” as “the         28 evidence based library and information practice 2009, 4:3                                    totality of coexisting facts which are conceived  of as mutually interdependent.”  lewin’s  theory proposed that the whole psychological  field within which individuals behaved, is  called “life space,” (lewin principles 16) which  has to be viewed to understand behavior.   individuals operate in multiple life spaces,  such as family, work, and school. lewin’s  conceptualization of “life space” as the whole  psychological field of human behavior,  including family, community, and school can  be applied to the personalized learning of  young people in a formalized learning  environment of the school library.  personalized learning through information  behavior is clearly exhibited in digital  environments when young people participate  in online communities, or simply search for  information on the world wide web. a public  type of information behavior occurs in school,  for example, when young people engage in  structured information searching that is  connected to specific learning outcomes.  personal and public information behaviors are  concurrent in the school library, for example,  when the “life space” includes formal  information behavior such as searching the  library catalog or subscription database, as  well as the everyday social networking  behaviors of the millennium generation. the  school library presents the potential for formal  and informal “life spaces” to co‐exist.    the theory of life space suggests not only a                                      fig. 3. substantive theory that informs the purpose of action research     purpose, but a method for research that  involves qualitative instruments such as  observation, interview, and focus groups. an  example of how lewin’s conceptualization of  life space could inform an emerging theory for  inquiry learning in school libraries is  suggested in the work of maniotes, who  theorized about connecting the student’s  world to schooling. maniotes (guided 32;  transformative) described a third space  comprised of the interaction of personal  experience and curriculum content. she  considered the student’s everyday life as the  first space; while school life, including content  of the curriculum, was the second space. the  third space is created when the first and  second space come together in equal amounts.  the third space is the context for studying the  design and implementation of school library  instruction; it clarifies dimensions of learning  that may be studied through empirical  research.     group dynamics and field theory have  significant implications for defining the  purpose of action research and its  transformative effect on the culture of teaching  and learning. when the life space is the  workplace, such as a school library, there are  social implications even when the purpose of  the research is to study self‐improvement of  practice. when it appears that the school  librarian or teacher acts gather and use  evidence to inform and reform their practice,  29 evidence based library and information practice 2009, 4:3  they are functioning as leaders of their  respective professions who have the capacity  to initiate social reform through education.  the distinction between self and social intent  becomes blurred as the practitioner interacts  with the work environment and becomes part  of it.     on the theoretical level, a more integrative,  rather than dichotomous, view of self and  society is supported by lewin’s equation for  human behavior, described more fully in  principles of topological psychology (12). the  equation states that behavior is a function of  personal characteristics and environment.       b = f (p, e)      when applied to the school library,  information seeking and knowledge building  behavior is a function of the personal,  everyday information behaviors of user‐ learners and educators, as well as an  environment unique to the school library.  unlike classroom environments, school  libraries have the option to conduct  instruction in many modes, including those  that accommodate the personal characteristics  and needs of the user‐learner. in addition,  collaboration between school librarian and  teachers contradict the educational tradition of  the teacher as sole authority. these  collaborations take on mutual intentions and  dependencies that did not exist prior to the  collaboration. a culture of inquiry emerges as  teachers become learners, learners are self‐  and peer‐taught, and everyone becomes a  researcher. this is a rich context for the  researcher who applies methods of group  dynamics and field theory to the study of  evidence based information literacy  instruction. another aspect of the culture of  inquiry in the school library informed by  lewin is interdependency. brown (28‐32)  argued that lewin’s concepts of  interdependence of fate and task  interdependence contradicted psychoanalytic  theory that attributed behavior to impulse. he  based his argument on tesser’s view of  behavior:    …our behavior is purposeful; we live  in a psychological reality or life space  that includes not only those parts of  our physical and social environment  that are important to us but also  imagined states that do not currently  exist (tesser 340).    brown (28‐32) posited that field theory  contributes the concepts of interdependence of  fate and task interdependence.  interdependency of fate explains that groups  form “…when people in it realize their fate  depends on the fate of the group as a whole”  (brown 28). an example of interdependency in  the school library is the problematic  phenomenon of collaboration between school  librarian and classroom teacher. collaboration  is critical to designing the learning task that is  the context for inquiry learning, yet it  contradicts the culture of teaching that is  essentially isolationist and individualistic  (lortie 73‐4).  collaboration between the  school librarian and teacher is problematic  because teachers do not see that their success  depends on groups. implicit in the role of the  school librarian are multiple layers of  collaboration throughout the inquiry process.  they include collaborations between learner  and teacher, between learner and librarian,  and between and among learners. teaching  and learning are synergistic and co‐dependent  processes. this has significant implications for  the study of teaching or learning: studying  only one or the other risks ignoring the  synergistic relationship between the two.     another appropriate subject for research is  interdependency in social learning, when user‐ learners participate in networked discussions  in web 2.0 environments such as blogs or  wikis. interdependency also occurs in the real  environment of the school library when user‐ learners work in groups. fig. 4 illustrates the  relationship between self‐ and social‐ oriented  action research with respect to independent  and interdependent learning.           30 evidence based library and information practice 2009, 4:3                                      both the self‐ (ar(s1)) and social‐ (ar(s2))  oriented modes of action research‐based  teaching  employ independent and  interdependent modes, regardless of whether  the purpose of the research is self  improvement or social reform. theoretical  support for interdependent learning that  situates the learner in formal or informal  groups is supported by lewin’s field theory.  the concept of life space, as well as gestalt  theory and group dynamics inform the  learning environment that constitutes a  culture of inquiry known as evidence based  instruction. when applied to school library  instruction, the social outcome is educational  reform; the self‐oriented outcome is  improvement of the practice of an individual  or small collaborative group. when this  duality of purpose is viewed as dichotomous,  it fragments definitions of action research’s  purpose and methodology.      the lewin‐dewey connection: a  methodology for studying action research  for self and social outcomes      while lewin’s substantive theories from social  psychology, noted in fig. 3, clarify the  purpose of action research in terms of self and  society, lewin’s change theory gives insights  into the way action research is undertaken.        ar(s1) self-oriented action research                                   change theory supports a model of action  research as an iterative inquiry process using  problem solving in a collaborative context.  data‐driven collaborative analysis is applied  to understand causes and enable predictions  about change (reason and bradbury). it  describes change in three stages (table 1). the  first stage, “unfreezing,” occurs when the  force field is altered. a force field supports the  stability of human behavior, based on ʺquasi‐ stationary equilibria” (schein). when  restraining forces, such as personal  psychological defenses or group norms are  removed, equilibrium can be more easily  shifted. these restraining forces are embedded  in the culture of a community (schein). the  change occurs in the second stage, which  precipitates a period of confusion and  transition. the third stage, called “freezing,”  occurs when a new mindset becomes accepted  and comfort levels return to their original  state.     table 1 contains two scenarios that illustrate  how change theory informs the methodology  of two hypothetical action research studies.  column 1 describes a study that adopts the  self‐improvement purpose of action research  targeting improvement of practice; column 2  describes a study that targets educational  reform.      independent learning ar(s2) socially oriented action research inter dependent learning 31 evidence based library and information practice 2009, 4:3  32   table 1  methodology informed by lewin’s change theory (adapted from schein; reason and brandbury)      column 1  improvement of practice  column 2  educational reform  problem solving  action research questions  that drive formal research  methods  how can the school librarian  improve the unit of inquiry and  do it better next time?  how can school librarians,  working with administrators,  change the classroom paradigm  of instruction using 21st century  learning methods?        collaborative context  form partnerships with teachers  to collect user‐learner surveys for  unit revision.  school librarian works with the  principal to adopt inquiry  learning in the classroom.        data driven  collaborative analysis  school librarian meets with  teachers to analyze and discuss  findings.  school administrators analyze  school data to identify areas of  testing that do not meet annual  yearly progress on state tests.        stage 1: unfreezing  remove restraining forces  that resist change  remove personal psychological  defenses of teachers who resist  working outside the classroom;  principal mandates  collaboration.  remove group norms of little  red schoolhouse, classroom  centered teaching; plan virtual  and blended resource‐based  courses that use technology.        stage 2: confusion  integrate changes in the unit  based on discussion with  teachers  choose courses for online  development; collaborate with  teachers to develop courses.        stage 3: acceptance  implement changes in the unit  for the next year.  provide training and support;  implement virtual and blended  courses.      both studies investigate phenomenon in the  field and are conducted by practitioners.  however, the purposes and methods of these  studies can be the same as those used in  formal research, meeting rigorous standards  for reliability, validity, and transferability,  using triangulation to ensure attainment of  these standards.  in this way, action research  purpose and methodology inform formal  research that researches evidence based  information literacy instruction using action‐ research based teaching.    the psychological descriptions of change  process noted in table 1 (i.e., unfreezing,       confusion, acceptance) inform lewin’s  description of action research as a spiral of  steps (fig. 5).  “each…is composed of a circle  of planning, action and fact‐finding about the  result of the action” (lewin resolving 206).       in the first step an idea is generated and  examined. “unfreezing” occurs as the idea is  accepted. planning consists of fact finding, an  overall plan to reach the objective, and a  decision about the first step of action. planning  usually results in modifying the original idea  (lewin “action” 205).  this is when confusion  and transition occur: the process is recursive.  it is “composed of a circle of planning,   evidence based library and information practice 2009, 4:3                              executing, and reconnaissance or fact finding  for the purpose of evaluating the results of the  second step, and preparing the rational basis  for planning the third step, and for perhaps  modifying again the overall plan” (lewin  “action” 206).  in this process change is  accepted, and equilibrium is restored.      change theory also describes the emotional  and behavioral components that accompany  these steps. the model strongly suggests a  problem‐solving approach that externalizes  internal processes. “the action of action  research, whether in a small or large scale,  implies change in people’s lives, and therefore  in the system in which they live” (mcniff 3).   action research is political, consistently  effecting change.      the relationship between lewin’s model and  other theoretical perspectives    lewin’s reflexive model of action research  spirals from reflection to action to more  reflection to new action. some detect a kinship  between this spiral of action and reflection and  grounded theory (glaser and strauss; strauss  and corbin). like grounded theory, action  research is reiterative. theory is not imposed                                               fig. 5. conceptualization of lewin’s action research methodology (adapted from lewin  resolving 206).    on the action research at the onset; it evolves  from the practice itself. the theory reflects the  practitioners’ perspective (mcniff). action  research, as noted in part 1 of this paper  (gordon “emerging”), is a legitimate form of  qualitative research when data collection is  carefully monitored. as such, action research  is not only a tool of evidence based practice,  but also a tool for formal research. like  qualitative research, action research takes  place in a naturalistic setting using  participant‐observation methods of  ethnographic research. it is collaborative and  includes characteristics of case study  methodology (belanger). anderson, herr, and  nihlen defined action research as “insider  research done by practitioners using their own  site as the focus of their study…it is oriented  to some action or cycle of actions that  practitioners wish to take to address a  particular situation” (2). garner saw it as a  cyclical process, but action research is often  defined as linear. takala identified steps in the  process of action research: formulating the  question, creating a solution, implementing  the solution, evaluating; and modifying ideas  and actions. gummesson labeled its  components as data collection, analysis,  action, decision making, implementation, and  33 evidence based library and information practice 2009, 4:3  change. consensus resides in the concepts of  reflection, inquiry and action, and the  repetitive nature of the cycle (patterson and  shannon “reflection”).  kemmis and  mctaggart viewed action research as a spiral  in which each cycle increased the participantʹs  knowledge of the question, leading to a new  solution and a new question.      like a variety of forms of action research that  have evolved, fig. 6 illustrates a methodical,  iterative approach that includes problem  identification, action planning,  implementation, evaluation, and reflection  (carr and kemmis). the model describes the  movement from plan to action through  observation and reflection. reflection,  complementary to the iterative nature of  action research, can be used to generate  models from a body of prior knowledge that  reframes a problem. this is followed by  experiments that bring about outcomes that  are subjected to further analysis. this model is  called reflection‐in‐action (schon). it  recognizes that there is little separation  between research and practice, or between  knowing and doing. when the plan is revised,  the cycle begins again. the insights gained  from the initial cycle inform the planning of                                               the second cycle. the action is modified and  the process is repeated.     mctaggart cautioned that it is a mistake to  think that following the action research spiral  constitutes “doing action research…. action  research is not a method or a procedure for  research but a series of commitments to  observe and problematize through practice a  series of principles for conducting social  enquiry.”  as a tool of evidence based practice,  action research is concerned with collecting  evidence, or data, in a systematic way that  ensures that the analysis and findings have  integrity. this evidence then defines a new  practice, or modifies existing practice. in order  to study the phenomenon of evidence based  instruction, it is useful to use this function of  action research and study it in a systematic  way. the instruments, procedures, and  findings of action research, as a tool of  evidence based information literacy practice,  inform the methods used to study action  research (gordon “study”). this is not to say  that commitment to observe and problematize  is not a prerequisite, or that a series of  principles for conducting action research are  not needed. however, building a theory of  methodology for the study of action research                                               fig. 6. the action research cycle (kemmis and mctaggart)   34 evidence based library and information practice 2009, 4:3  rests on the premise that action can be  systematic and predictable in its execution.  this paper posits that authentic learning  methods, as well as traditional qualitative  research methods, comprise an instruction‐ specific form of action research that informs  the formal research that studies it as a  phenomenon of teaching and learning.      the strong connection between purpose and  methodology for action research assumes that   practitioners need to know why they do what  they do in order to be able to make informed  teaching decisions. this kind of practice‐based  theory is the only valid form of action research  theory (mcniff). this paper posits that since  the boundary between research and practice is  fuzzy, practice must be theory‐based. this is  demonstrated in a study (gordon “study”)  where the action research training model  contained a theoretical component that  required practitioner‐researchers to identify a  learning theory that informed their research  question, or supported the way they collected  evidence, or reinforced their findings. it is  possible for action research to bring theory,                                                   formal research, and practice into alignment.  the role of theory in action research is  practical in the sense that it drives  methodology. logical argumentation, rather  than a proposed plan, develops the action  research as it evolves. this is not to say that  methodology in action research does not exist  or that it always exists, but that it can exist.  action research is a creative way to effect  change through the continuous improvement  defined through theory that drives best  practice. it requires acceptance of the  problematic and unstable nature of  professional practice (mcniff).     dewey’s metatheory    dewey’s descriptions of action and reflection  offer the metatheory that complements  lewin’s action research spiral. in fact, there  are several points of intersection between the  work of lewin and dewey. in terms of  understanding learning behavior, lewin’s  change theory accounted for behavioral and  affective components. dewey’s metatheory  describes cognitive processes predicated on                                                     35 evidence based library and information practice 2009, 4:3  principles for talking about thinking. this has  important implications for action research  methodology (fig. 5) and the cycle of action  research (fig. 6). dewey analyzed reflective  thinking as a process that goes from a pre‐ to a  post‐reflective state. the process begins with  observing the surrounding conditions (fig. 7),  which dewey called the ʺfacts of the case.ʺ  coupled with recollection, or observations  rooted in the past, reflection is a way of  exploring a situation to discover the facts.     facing the conditions, or situations, to  discover facts requires effort until the habit of  thinking is formed. suggestions compete with  each other as the person seeks the best  solution. facts and suggestions guide new  observations, which is an important aspect of  the reflective thought process. the continuous  interaction of facts, disclosed by observation,  and suggestions, or suggested proposals of  solutions, continues until some suggested  solution meets all the conditions (dewey how  we think).    the interaction of perceived facts and  suggestions generates ideas (fig. 8). the  observed facts are called the data. data are  material to be interpreted, explained,  managed, and utilized, resulting in suggested  solutions that form ideas.                                         data and suggestions form indispensable and  correlative factors of all reflective activity. this  process is accomplished through inference,  going beyond what is actually observed or  projecting what is possible. it proceeds by  anticipation, supposition, conjecture and  imagination. what is inferred demands a  double test: 1) the process of forming the idea  or supposed solution in constant cross‐ reference to conditions observed (facts); 2) the  idea after it is formed is tested by acting upon  it. the consequences of the action confirm,  modify, or refute the idea (dewey how we  think).    dewey’s (1910) model of reflective practice  comprised of action and reflection is a kind of  metatheory: it is a set of beliefs that reasonably  explains the complex phenomenon of thinking  around which educators can build a set of  values and practices about what teaching and  learning looks like when it best serves the  learner. when these models of thinking  connect with action research conducted in  school libraries, pedagogy of self‐reflection  evolves. instruction is never perfect; sooner or  later something goes wrong. even when  instruction goes well, the question of  improvement arises. in either case, there is a  cycle of action and reflection that is  continuous. for example, typical problems                                           36 evidence based library and information practice 2009, 4:3  that school librarians identify in their practices  include plagiarism, student indifference to  subscription databases (gordon “study”), and  “reporting” rather than researching (gordon  “fish”). dewey’s model applied to these  “cases” includes observation and recollection  of past experience that generate the facts of the  case, or the data. without a deliberate and  conscious process, perceptions and  misperceptions may masquerade as fact as the  school librarian struggles to define the  problem. past experience may even hinder a  clear understanding of the problem. unaware  of the fallacies, the school librarian begins to  generate ideas that are flawed because they  are not based on the actual facts of the case. a  process is needed to help the librarian test  ideas or supposed solutions in constant cross‐ reference to conditions observed. the process  structures the testing of actions that result  from acting on the idea. action and reflection  are synergistic components of action research  process. this is an example of how dewey’s  thinking model adds cognitive structure to the  reiterative model of action research, offering a  method for thinking about the action and  reflection (fig. 6). it is also applicable to the  thinking processes of students who engage in  self‐reflection through authentic learning, as  described in part 1 of this article (gordon  “emerging”) and explained in fig. 7 and 8.   the connection between dewey’s metatheory  and lewin’s substantive social psychology  theory generates a theory of methodology  (fig. 2) for studying action research‐based  teaching informed by the methods prescribed  in lewin’s work and the cognitive processes  described by dewey.    implications of action research as a tool of  evidence based information literacy  instruction theory     two purposes for action research are defined  in this paper. they are viewed as comple‐ mentary, rather than dichotomous. this  section of the paper explores the relationship  of self (ar(s1) and social (ar(s2)) oriented  action research, followed by an explication of  the theory and its corollaries.    the  relationship  between  self‐oriented  and  social‐oriented action research    the emerging theory developed here  postulates that evidence based information  literacy instruction uses action research as a  tool for two purposes: improvement of self  and society.  self‐oriented action research  (ar(s1)) targets self‐improvement at the local  level through teaching and learning in school  libraries. authentic learning, as well as  traditional data collection methods, generates  evidence for user‐learners to critique and  revise their work and for educators to revise  their teaching, based on evidence of student  work and progress. dewey’s experiential  learning supports this teaching methodology  that encourages creativity and innovation.  often called “active learning’, it incorporates  the critical element of reflection which  internalizes learning, making it specific to  individual learners.  lewin addressed the  behavior of group members, advocating  participant observation whereby they were  emotionally involved, but could observe  themselves with objectivity, similar to  dewey’s self‐reflection. lewin’s methods  resemble dewey’s active learning: he  advocated the use of film and other cognitive  aids to engage group members.    the social orientation of lewin’s action  research and dewey’s action and reflection  models also complement each other on social  and political levels,  democratic values, not  unfamiliar to libraries, are central to their  work. lewin (“patterns”) explored the effects  of various types of leaders on the structure of  groups and the behavior of group members.  three leadership models, democratic,  autocratic, and laissez‐aire, were compared in  a terms of the effects they elicited from group  members.  reid (115) reports that democratic  leadership was the most effective in eliciting  creativity, positive group feelings, and  friendliness while the other leadership models  elicited more hostility, aggression and  dissatisfaction. dewey was also interested in  democracy in terms of learning behavior.  allport points out the kinship between lewin  and dewey.   37 evidence based library and information practice 2009, 4:3  both agree that democracy must be  learned anew in each generation, and  that it is a far more difficult form of  social structure to maintain than is  autocracy. both see the intimate   dependence of democracy upon social  science. without knowledge of, and  obedience to, the laws of human  nature in group settings, democracy  cannot succeed. and without   freedom for research and theory as  provided only in a democratic  environment, social science will surely  fail. dewey, we might say, is the  outstanding philosophical   component of democracy; lewin is its  outstanding psychological component.  more clearly than anyone else has he  shown us in concrete , operational  terms what it means to be a  democratic leader, and to create  democratic group structure (xi).    social‐oriented action research (ar(s2))  targets social reform on the global level of  educational improvement. this occurs when  administrators, policy‐makers, and  government agencies incorporate action  research into their decision‐making processes.  in this case, the purpose of the action is reform  imposed from the top. for example, a national  department of education may use evidence  based practice to study the effectiveness of  uniform learning standards. as a result of the  evidence collected, a top‐down decision may  mandate differentiated levels of standards, not  merely minimal standards, for testing and  evaluating student achievement. this  dimension of educational reform is outside of  the province of school librarians and  classroom teachers. however, this is not to say  that local action research‐based reform  (ar(s1)) cannot effect global consequences  when the school librarian assumes the role of  professional developer, for example. action  research situated at this level (ar(s2)) effects a  grassroots type of reform, where practitioners  reform teaching and learning from within  their practices, and their findings and results  are disseminated to create a new paradigm or  norm. for example, locally generated  educational reform could be considered a type  of social reform that is a by‐product of  evidence based practice and action research.    further study is needed to assess  social  oriented action research for the purpose of  global educational reform, particularly in  terms of 21st century learning in virtual  environments (gordon “emerging”) that  challenge traditional assumptions about  where and how education takes place.     corollaries to the theory of evidence based  information literacy instruction    emerging theory posits that evidence based  information literacy instruction uses action  research as a tool of evidence based practice.  an investigation reveals two purposes of  action research that originate in self‐ improvement and social change. the former  leads to local improvement of practice; the  later to global educational reform.  exploration  of methodology concludes that authentic  learning is a tool for action research that aims  at improvement of practice. sense‐making  methodology generates a theory of  methodology through the synergy of  substantive theory and metatheory. lewin’s  social psychology theory and dewey’s  metatheory of inferential thinking provide  behavioral, affective, and cognitive  dimensions for studying evidence based  information literacy instruction through action  research. corollaries to evidence based  information literacy instruction theory address  the major tenets of self and social action  research as defined in this paper.      self‐oriented action research ‐ ar(s1)    action research for the purpose of self‐ improvement or improvement of one’s  practice has several corollaries that inform the  theory of evidence based information literacy  instruction. inherent in the corollaries that  address these two purposes is a theory of  methodology that derives from the lewin‐ dewey connection.     corollary 1: evidence, action, and reflection  are central to self‐oriented action research  38 evidence based library and information practice 2009, 4:3  (ar(s1)). user‐learners as well as educators  improve their performance through evidence  that informs the revision of their work.  teaching and learning become synergistic in  this model. to study either in isolation loses  the synergy of evidence based transactions  between learner and teacher. clarification of  the role of evidence in teaching and learning  and development of ways to manage  diagnoses, interventions, and feedback are  part of these transactions.       corollary 2: self‐oriented action research  (ar(s1))  is generated by authentic learning  that generates evidence through performance  based formative assessments. this teaching  methodology can be viewed as a tool of action  research. the evidence serves two functions: 1)  to help user‐learners perform self‐ and peer‐ evaluations and to revise their work; 2) to  inform the teaching decisions and revision of  teaching strategies of school librarians and  teachers.    corollary 3: self‐oriented action research  (ar(s1)) takes place in the context of inquiry  learning defined by two types of tasks: 1)  micro level information tasks that include  basic information literacy skills (searching and  retrieving information) and advanced  information literacy skills (evaluating and  applying, or using information; 2) macro level  learning task that is the context for  information tasks. the learning task is related  to real world contexts, problem‐solving, and  decision making. this distinction between  micro and macro levels of tasking provides the  researcher with a way of tracking the  information‐to‐knowledge connection while  preserving the integrity of information  searching and knowledge building.    corollary 4: self‐oriented action research  (ar(s1)), or action research that purposefully  focuses on self‐exploration and self‐ improvement of user‐learners and educators,   addresses inner processes described by  constructivist learning theory as they occur in  interdependent, or collaborative learning, as  well as independent, or individual learning.  ar(s1) explores the importance of  interdependent learning and how information  behavior differs in individual and social  contexts. the identification of new modes for  evidence based strategies and models includes  finding methods for collecting, organizing,  and storing evidence generated by  interdependent learning, and applying  creative ways of analyzing evidence that are  efficient and accurate.    corollary 5: self‐oriented action research  (ar(s1)) supports the study of the  information‐to‐knowledge connection in  which instructional teams of school librarians  and classroom teachers contribute their  respective expertise of information processing  and use, as well as curriculum content specific  to academic disciplines, to design, implement,  and evaluate micro information tasks and  macro learning tasks. collaboration is seen as  critical to this connection.     these corollaries indicate research directions  for self‐oriented action research aimed at  improving school library instruction. implicit  in these directions are methods specific to   constructivism and social psychology,  including dewey’s description of inference.    social‐oriented action research – ar(s2)    social reform can be viewed on a continuum  that includes local and global change. action  research for the purpose of social reform is  local when it effects change in a school or  school district. in this case, the reform is  embedded in action research for self‐ improvement when the school librarian  assumes a leadership role in instruction, a  professional development role that informs  classroom teaching, and a management role  that informs the direction of the profession.    corollary 6: the school librarian’s role as  leader in instruction originates in methods  that empower learners and teachers to self‐ evaluate and revise their work. the methods  of authentic learning, for example, are  instruments for reform‐oriented modes of  teaching. because they are situated in macro  tasks that raise the user‐learner’s awareness of  39 evidence based library and information practice 2009, 4:3  relevant social problems and issues, such as  the environment, globalization, fiscal  awareness, and personal health. these issues  are identified as 21st century themes integrated  with core subject areas in order to educate  youth to live and work in a fast paced society  driven by technology.    corollary 7:  the school librarian’s role as  professional developer raises the level of  action research from self‐improvement to  social reform. in this role the school librarian  transforms the teaching profession by shifting  the paradigm from classroom‐centered  learning to inquiry learning integrating  resources and technology. this form of social‐ oriented action research (ar(s2)) infiltrates  learning through the macro learning task, and  incorporates real‐world problem‐solving.  evidence based instruction inspires  pedagogical products and processes that  reflect deep understandings, rather than  shallow fact‐finding. school librarians lead the  instructional team, helping classroom teachers  to craft learning tasks that develop  competencies specific to the information‐ knowledge connection and technology‐driven  life and work contexts. challenges for 21st  century educators call for new models of  teaching that are multi‐modal, multi‐media,  and multi‐disciplinary and that enable user‐ learners to be creative problem‐solvers.      corollary 8: socially oriented social reform can  be achieved through the school librarian’s role  as manager, conceiving the school library  program as an educational reform universally  applied to all students, rather than as a  specialty or supplement to the school’s  instructional program. a 21st century learning  environment challenges the restrictions of time  and place that characterize traditional  schooling and opens opportunities for school  library practices to transcend the walls of the  library, contributing in a broader context of  school reform. social‐oriented action research  requires methods of effecting political change  that have long range benefits. the school  librarian, as manager, engages the school  community in practices that are evidence  based for the purpose of making education  relevant to an increasingly disengaged  generation of learners. virtual learning  environments and alternative ways of  schooling present rich opportunities for  educational reform through evidence based  information literacy instruction. in this role,  the school librarian may venture into the  world of politics and legislation to apply  evidence based advocacy. evidence is  generated from various dimensions of the  school library program, in addition to  instruction. for example, the school librarian  uses public media, strategies for parental  involvement, publication of evidence based  instruction using action research, and other  modes of promotion to inform and model the  role of the school library in 21st century  learning.    self‐ and social oriented action research –  ar(s1) and ar(s2)    when considered together, self‐ and social‐ oriented intents of action research suggest  corollaries that point to research that studies  the reform of the school library and teaching  professions.    corollary 9: self‐ and social‐oriented action  research (ar(s1) and ar(s2)) call for  development of additional action research  training models for school librarians and  classroom teachers that empower these  practitioners to transform schooling from the  perspectives of self and society. these models  are needed for pre‐service, as well as  practicing teachers, to include the role of  information, technology, and inquiry in 21st  century teaching and learning. a discrete  approach to professional preparation that  isolates self‐ and social‐oriented reform  through teaching loses the dynamic of a  holistic approach to reform education in  classrooms and libraries that blurs the  distinction between the two.     corollary 10: self‐ and social‐oriented action  research (ar(s1) and ar(s2)) are the building  blocks for a culture of inquiry in school  libraries and in the school community that is  40 evidence based library and information practice 2009, 4:3  evidence based. evidence based instruction  requires a socio‐cultural perspective in which:    …students’ contact with artifacts and  people ‐ in and through the school  library ‐ as participation in a  socialization exercise where the school  library can be understood as a  “cultural tool” with a communicative  function. the activities that take place  in or through the school library are  social and communicative. they are  part of a cultural context and can vary  from school to school, but they can  also have a great deal in common  (limberg and alexandersson 3).    conclusion: implications for research  driven by the emerging theory    emerging theory of evidence based  information literacy instruction and its  corollaries indicates a research agenda for  studying evidence based instruction in school  libraries, as well as virtual environments that  is rich in information, resources, and  technology. at the heart of the theory are  evidence and action research conceived as  tools for thinking and building knowledge,  commonly called “learning”.    the greeks acutely raised the  question: how can we learn? for  either we know already what we are  after, or else we do not know. in either  case, learning is impossible… (dewey  democracy 142)    john dewey tackled this dilemma and  concluded:    the dilemma makes no provision for  coming to know, for learning; it  assumes either complete knowledge  or complete ignorance. nevertheless  the twilight zone of inquiry, of  thinking, exists. the possibility of  hypothetical conclusions, of  tentative results, is the fact which the  greek dilemma overlooked.…     systematic advance in invention and  discovery began when men  recognized that they could utilize  doubt for purposes of inquiry by  forming conjectures to guide action in  tentative explorations, where  development would confirm, refute,  or modify the guiding conjectures.  while the greek made knowledge  more than learning, modern science  makes conserved knowledge only a  means to learning, to discovery  (dewey democracy 142‐3).    the theory of evidence based information  literacy instruction views evidence based  information literacy instruction and action  research as instruments for teaching youth  how to think and for raising the bar for  teaching to accommodate the “twilight zone of  inquiry”. it goes beyond teaching information  use in the guise of information literacy, visual  literacy or technological literacy, to teaching  knowledge use.    while all thinking results from  knowledge, ultimately the value of  knowledge is subordinate to its use in  thinking. for we live not in a settled  and finished world, but in one which  is going on, and where our main task  is prospective, and where retrospect ‐  and all knowledge as distinct from  thought is retrospect ‐ is of value in   the solidity, security, and fertility it  affords our dealing with the future  (dewey democracy 146).    a theory that supports the pedagogy of school  libraries as an alternative to instruction  oriented to the “right answer,” where testing  is the measure of knowledge, and mistakes are  equated with failure rather than opportunity  to discover, is highly relevant to much needed  research for alternatives. a theory that  accommodates research embedded in digital  environments, where information seeking and  use and knowledge construction are all self‐ directed, points to a pedagogy for the future  and is, in itself, a social reform.  librarianship  seems the logical profession to lead the  41 evidence based library and information practice 2009, 4:3  systematic and methodical development of  models for evidence based practices that  inform the educator’s decision‐making and  problem solving. like the medical model for  bridging theory and practice, the instructional  model of action research for librarianship  depends on the training of practitioners to  apply evidence to practice. librarianship is  well‐suited to developing practitioner‐ researchers who are self‐reflective in their  practice and proactive in helping patrons to  make the information‐to‐knowledge  connection. the implications for self‐ improvement in teaching and learning that  leads to educational reform on a social scale  that is lead by the library profession reinforce  the paradigm shift that redefines librarianship  as it moves from an information age  profession to that one enables a knowledge  society.      works cited    allport, gordon w. “introduction” resolving  social conflicts; selected papers on  group dynamics. gertrude w. lewin  (ed.). new york: harper & row, 1948.    anderson, gary k., kathryn g. herr, and ann   sigrid nihlen. studying your own  school: an educator’s guide to  qualitative practitioner research.  thousand oaks, ca: corwin press,  inc., 1994.    argyris, chris, robert putnam, and diana m.   smith. action science. san francisco,  ca: jossey bass, 1985.    belanger, joe. teacher as researcher: roles  and expectations. an annotated   bibliography. feb.1992. eric  document reproduction service.  ed342751.       belkin, nicholas j. “anomalous states of     knowledge as a basis for information   retrieval.” canadian journal of  information science 5 (may 1981): 133‐ 43.  bodgan, robert, and sarah k. biklen.   qualitative research for education.   boston: allyn and bacon, 1992.    boomer, garth. “addressing the problem of   elsewhereness: a case for action  research in our schools.” reclaiming  the classroom: teacher research as an  agency of change, eds. dixie  goswami and peter stillman.  portsmouth, nh: boynton/cook                publishers, 1987, 4‐13.     brown, rupert. group processes. dynamics   within and between groups. oxford:     blackwell, 1988.    carr, wilfred, and stephen kemmis. becoming  critical: knowing through action     research. london: falmer press, 1986.    corey, steven. action research to improve   school practice. new york: teachers  college, columbia university, 1953.    dervin, brenda. “what methodology does to   theory: sense‐making methodology  as exemplar.” theories of information  behavior. eds. karen e. fisher, sandra   erdelez, and lynne (e.f.) mckechnie.  medford, nj: asist monograph  series. 2005, 25‐9.    dervin, brenda, and michael s. nilan.  “information needs and uses.”  annual review of   information science and technology  (arist) 21 (1986): 3‐34.    dewey, john. how we think. boston: d.c.     heath & co., 1910.    dewey, john. democracy and education: an   introduction to the philosophy of   education. new york: macmillan,  1916.    garner, betty. “using action research:  challenges and opportunities for the  beginner.” greater st. louis sixth  42 evidence based library and information practice 2009, 4:3  action research collaborative  conference, st. louis,     mo, 1996.     glaser, barney g., and anselm s. strauss. the   discovery of grounded theory:  strategiesfor qualitative research.  chicago: aldine, 1967.    gordon, carol a. “an emerging theory for     evidence based information literacy   instruction school libraries, part 1:  building a foundation.” evidence   based library and information  practice 4.2 (june 2009): 56‐77. 20 aug.  2009   <http://ejournals.library.  ualberta.ca/index.php/eblip/article/vi ew/4637/5318>.    gordon, carol a. “is fish a vegetable? a   qualitative study of a ninth‐grade  research project.”  school library  media quarterly 25.1 (fall 1996): 27‐ 33.    gordon, carol a. “putting the learner in   charge. are information literacy  skills enough?” scan 19.1 (2000): 32‐ 8. 20 aug. 2009  <http://www.curriculumsupport.  education.nsw.gov.au/schoollibraries/ scan/content/scanarch2000‐2001.htm>.  gordon, carol a. “students as authentic  researchers: a new prescription for  the high school research  assignment.” school library media  research 2 .1 (1999): 1‐21. 20 aug. 2009  <http://www.ala.org/ala/mgrps/divs/a asl/aaslpubsandjournals/slmrb/slmrco ntents/ volume21999/  volume21999.cfm>.  gordon, carol a. “a study of a three‐  dimensional action research training  program for school library media  programs.” school library media  research 9.1 (2006): 1‐21. 20 aug. 2009   <http://www.ala.org/ala/mgrps/divs/a asl/aaslpubsandjournals/  slmrb/slmrcontents/volume09/volume _9_2006.cfm>.     gummesson, evert. qualitative methods in   management research. newbury  park: sage, 2000.    kelly, george. a.  a theory of personality:   the psychology of personal  constructs. new york: norton, 1963.    kemmis, stephen, and robert mctaggart.   “action research in retrospect and  prospect.” action research reader. 3rd  ed. eds. stephen kemmis and robert  mctaggart.  geelong, victoria: deakin  university press, 1988, 4‐13.    kuhlthau, carol c.   facilitating information   seeking through cognitive modeling  of the search process: a library  studies research project, 1986. eric  document reproduction service.  ed328268.      kuhlthau, carol c., leslie k. maniotes, and   ann k. caspari. guided inquiry:  learning in the 21st century.  westport, ct: libraries unlimited,  2007.    kuhn, thomas s. the structure of scientific  revolutions, 3rd ed. chicago:  university of  chicago press, 1970.    lewin, kurt., r. lippitt and r. white.  “patterns of aggressive behavior in  experimentally created ‘social  climates’.”journal of social  psychology 10 (1939): 271‐99.    lewin, kurt. “action research and minority     problems.” journal of social issues 2     (1948): 34‐46.     lewin, kurt. field theory in social science:     selected theoretical papers. ed. d.   cartwright.  new york: harper &  row, 1951.      43 evidence based library and information practice 2009, 4:3  lewin, kurt. principles of topological   psychology. new york: mcgraw hill,  1936.    lewin, kurt. resolving social conflicts;   selected papers on group dynamics.  gertrud w. lewin (ed.). new york:  harper & row, 1948.    limberg, louise, and mikael alexandersson.     “the school library as a space for   learning.” school libraries  worldwide 9.1 (jan. 2003): 1‐15. 20  aug. 2009 <http://www.iasl‐ online.org/pubs/slw/jan03‐ limberg.htm>.    lortie, dan c. schoolteacher: a sociological   study. chicago: university of chicago                 press, 2002, 1975.    maniotes, leslie k. the transformative power   of literary third space. university of  colorado  at boulder, co: proquest  dissertation and theses, 2005. aat  3168285.    mckernan, james. curriculum action  research: a handbook of methods  and resources for the reflective  practitioner, 2nd ed. london:  routledge, 1996.    mcniff, jean. action research: principles and   practice. 2nd ed. london: routledge,     2002.    mctaggart, robin. “issues for participatory   action researchers.” new directions  in action research.  ed: ortrun zuber‐ skerritt. london: falmer press, 1996.    morrow, alfred j.  the practical theorist: the   life and work of kurt lewin. new  york: basic books, inc., 1969.    patterson, leslie, and patrick shannon.   “reflection, inquiry, and action.”  teachers are researchers: reflection  and action. eds. leslie patterson,  carol minnick santa, kathy g. short  and karen smith. newark, de:  international reading association,            1993, 7‐11.    piaget, jean. judgment and reasoning in the  child. new york: harcourt press,  1928.    reason, peter, and hilary bradbury, eds.  handbook of action research.  london: sage, 2001.     reid, kenneth e. from character building to  social treatment: the history of the  use of groups in social work.  westport, connecticut: greenwood  press, 1981.    schein, edgar h. kurt lewin’s change theory  in the field and in the classroom:  notes toward a model of managed  learning. 2006. 20 aug. 2009   <http://www.a2zpsychology.com/artic les/kurt_lewin%27s_change_theory.ht m>.    schon, donald a. the reflective practitioner:  how professionals think in action.  new york: basic books, 1983.    schwab, joseph  j.  college curriculum and   student protest. chicago: university  of chicago press, 1969.    smith, mark k.  kurt lewin: groups,   experiential learning and action  research. 18 june 2009. 20 aug. 2009.   <http://www.infed.org/thinkers/et‐ lewin.htm>.    stenhouse, laurence. an introduction to   curriculum research and  development. london: heinemann,  1975.    strauss, anselm c., and juliet corbin.  basics   of qualitative research: grounded  theory procedures and techniques.  newbury park: sage, 1990.      44 evidence based library and information practice 2009, 4:3  45 streatfield, david, and shirley markless.   invisible learning? the contribution  of school libraries to teaching and  learning. london: british library,  1994.    takala, sauli.  action research in the   classroom: content instruction  through a foreign language. a  report on the 1992‐93 tce  programme. 1994. eric document     reproduction service. ed383197.    tesser, abraham. advanced social   psychology. new york: mcgraw‐hill,  1995.              todd, ross j. “the evidence‐based practice  manifesto for school librarians: if  school librarians can’t prove they  make a difference, they may cease  to exist.” school library journal 54.4  (1 apr. 2008): 38‐43. 20 aug. 2009  <http://www.  schoollibraryjournal.com/article/ca65 45434.html>.    wiggins, grant. authentic education. 2007. 20   aug. 2009  <http://www.grantwiggins.org>.    evidence based library and information practice commentary   a case for the use of nonparametric statistical methods in library research   megan hodge assistant head for teaching & learning vcu libraries virginia commonwealth university richmond, virginia, united states of america email: mlhodge@gmail.com   received: 28 feb. 2019                                                                 accepted: 1 may 2019      2019 hodge. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29563     if called upon to name statistical methods, the average librarian would likely reply with examples such as correlation and t-test. those with more experience conducting research might name more comparatively exotic tests such as mancova or factor analysis. these are all examples of what are known as parametric statistical tests: tests designed to identify a limited number of things about a data set where most characteristics of the data are already known or assumed. as parametric tests depend upon these assumptions, librarians who intend to use parametric tests must take care to collect data with these assumptions in mind.   while these assumptions vary somewhat from test to test, some are common to most parametric tests. one is the assumption that the data at least approximately resembles a normal distribution (also known as a “bell-shaped curve”), with most scores falling around a central score, fewer scores falling further from that central score, and few or no extreme scores. another is the assumption that data is measured on a continuous scale, with the dependent (observed) variable measured on a real numerical scale, such as gre scores or number of program attendees. a third common assumption is that there is a minimum number of participants in each group.   if even one of these assumptions is not true, then it is likely that parametric tests should not be used. library research often violates these assumptions, with data that may be heavily skewed, with a tendency towards larger or smaller values; a variable of interest that is often categorical, or non-numeric (e.g., emotions elicited by the library: anxiety, gratitude, wonder, frustration), or that may have an order but not a numerical value naturally associated with that order (e.g., degree of comfort using a database: not comfortable, neutral, comfortable); finally, sample sizes in library research are often small.   there are strategies that researchers can use to reduce the likelihood of these issues: rewrite questions to use a continuous rather than non-continuous scale; develop a plan to recruit a larger number of participants; remove outliers from the data. sometimes these strategies are not feasible, however: the data may already have been collected and the sample size cannot be increased; the variable of interest cannot be measured on a continuous scale; the outliers are valid, if inconvenient, scores.   fortunately, there is an alternative to parametric statistical tests: nonparametric statistical tests. nonparametric tests tend not to rely upon the same assumptions required by parametric tests. instead, nonparametric tests rely upon the median as a measure of a data set’s central tendency, rather than the mean, which is the measure used by parametric methods. the mean is influenced by outliers in the data set; the median is not. nonparametric alternatives exist for most common parametric methods, including anovas, pearson product-moment correlations, and t-tests.   using a parametric statistical test when one or more of that test’s core assumptions have been violated compromises the validity of the inferences that can be drawn from the test results and, by extension, the rigor of the research. in this case, the term ‘inferences’ refers to the conclusions that may be drawn from a test’s results. it is usually not possible to survey or test every member in the population of interest (for example, academic librarians who have advanced into middle management positions within the last five years), and as such, inferential statistical tests may be used on a much smaller sample of that population to make inferences (generalizations) about that larger population. parametric tests can have greater power to detect statistically significant differences and effects than their nonparametric equivalents; in other words, they can be more sensitive to effects and differences that are smaller in scale. however, using a parametric test when one or more of its assumptions has been violated may result in an inaccurate representation of the data — for example, when the mean is skewed far from the centre of the data by a few extreme values — which in turn means the inferences made about the larger population from which the sample was drawn may be flawed or inaccurate. therefore, nonparametric tests, when called for, increase the rigor of a study’s conclusions and the extent to which such conclusions are justified for use in evidence based practice.   a number of research scenarios common to library scholarship warrant the use of nonparametric statistical methods. their use may be called for in order to increase a study’s internal validity (the extent to which the study is able to investigate the topic of interest), to increase the study’s statistical rigor, or both. several of these research scenarios are described below.   surveys are a popular research method for librarians, as is evident from the number of requests for participation that come through email discussion lists. many of the sorts of questions that are asked in librarian-designed surveys would best be analyzed with nonparametric statistical methods, as our research interests often tend to elicit categorical or ordinal data. for example, librarians often employ likert scale questions. these sorts of questions ask participants to respond on a five-point scale whether they strongly disagree, disagree, neither disagree nor agree, agree, or strongly agree with the question stem. likert-type questions, which use similar scales but which may have more points or ask about frequency rather than agreement, are also common. ideally, in addition to identifying the construct(s) or variable(s) of interest, survey designers will have also identified all of the subconstructs making up the construct(s). for example, the construct of library anxiety might have attitudinal, cognitive, and behavioral subconstructs. a rigorous survey will have at least three questions that speak to each subconstruct of library anxiety; the survey designer will have determined the survey’s construct validity (the extent to which the subconstructs do or do not represent all aspects of the construct itself); and evaluated whether the questions themselves adequately speak to each subconstruct. to analyze data collected from likert scales, the response options are converted to artificial scores, with, for example, a ‘strongly agree’ converted to a one, an ‘agree’ converted to a two, and so on. responses to likert or likert-type questions designed in this way allow for responses for each subconstruct to be combined, resulting in data on an interval scale that may be analyzed with parametric statistical methods.   if, however, there are only one or two questions that speak to each subconstruct or construct, the data created will be ordinal in scale: the difference in strength of feeling between one respondent’s “strongly agree” and “agree” may not be the same as the difference between that respondent’s “agree” and “neither agree or disagree.” further, the differences in strength of feeling are likely to differ between respondents. for example, a respondent who only slightly agrees with the question stem, and another who wholeheartedly agrees but does not consider their agreement ‘strong,’ may both choose a response of “agree.” and, if there are few survey respondents, there may not be enough responses to meet the minimum number required for the anticipated parametric statistical test (for example, 15 per group for a t-test), or the data may not have a normal (bell-shaped) distribution: responses may be heavily skewed. all of these scenarios warrant the use of nonparametric tests.   another common type of question on librarian-designed surveys are ranking questions. for example: “please rank the following methods of receiving information from acrl in the order in which you are most likely to use them.” “please rank the usefulness of each of the topics you learned about in today’s webinar.” “please rank the following mediums of professional development in order of their desirability.” before analyzing the statistical significance of the data distribution, it is important to first assess whether respondents agree in their rankings: a given item may appear to be the most popular, but upon reviewing the data it may be revealed that the item was ranked last by a good number of respondents. this requires a nonparametric test that, essentially, tests inter-rater reliability on a large scale.   quasi-experimental studies that evaluate the effectiveness of a program or instructional strategy are also common in the library literature. in all but the rarest of occasions, however, library studies do not have sufficient participants to meet the minimum threshold for the parametric tests librarians commonly use for these research designs, such as a t-test or ancova (at least 15 per group, or 30 in a single group). parametric statistical methods are influenced by outliers and therefore require a minimum number of participants to counteract the effect of any outliers. additionally, most if not all parametric methods assume independence of observations: that each participant has received the treatment independent of all other participants. independence of observations is important both because it ensures participants do not influence each other’s scores, but also because it mitigates the risk of systematic bias in the scores. systemic bias could be introduced in many ways: a fire alarm, resulting in all students in a class missing the same piece of content; a discussion that takes place in one class section but not another; or seemingly minor differences in delivery between classes offered by different librarians. in short, a librarian who wishes to evaluate the effectiveness of a lesson taught to one class of 20 students has an n of 1, not 20. unless the librarian is teaching for a course that has many sections, such as a first-year writing course, or is willing to collect data over multiple years (which introduces validity threats of its own), it is likely that the librarian will have a very small n. data collected from these small or radically non-normal samples should be analyzed using nonparametric methods such as the mann-whitney u test or the sign test (alternatives for the independent samples t-test and paired samples t­-test, respectively), which do not rely upon the assumption of a normal distribution.   further, nonparametric tests may also increase librarians’ understanding of the practical significance of their research. statistical significance, or the likelihood of the findings not being due to chance, can be manipulated by increasing sample size; with a sufficiently large sample, most measured relationships/differences will be found to be statistically significant. a nonparametric test such as kendall’s w evaluates agreement among a large number of raters, is not affected by sample size, and will, for example, allow the researcher to determine the extent to which survey-takers agree on the order of ten ranked items.   one explanation for nonparametric tests’ relative obscurity may be their lack of power (ability to detect small differences/associations between groups) when compared with their parametric counterparts. however, data that do not meet the assumptions undergirding parametric tests can in some cases be more powerfully analyzed with nonparametric tests. more statistical power results not just in a greater ability to detect differences or associations between groups, but in the statistical significance of the difference/association, or all-important p-value, to be much stronger.   these are just a few of the reasons that nonparametric statistical methods are more appropriate than parametric tests for many of the research designs favored by librarians. when used appropriately, nonparametric statistical methods can result in research findings of greater statistical validity and explanatory power. the subscription-based (but inexpensive) website laerd statistics is recommended as a resource for librarians wishing to identify nonparametric alternatives to specific parametric tests or learn more about nonparametric methods.   microsoft word class_pikas.doc evidence based library and information practice 2007, 2:4  88 evidence based library and information practice       classic    communication is the key skill for reference librarians      a review of:  taylor, robert s. ʺquestion‐negotiation and information seeking in libraries.ʺ college &  research libraries 29.3 (1968): 178‐94.      reviewed by:   christina k. pikas  librarian, r.e. gibson library and information center, the johns hopkins university applied  physics laboratory   laurel, maryland, united states of america  e‐mail:  christina.pikas@jhuapl.edu        received: 31 august 2007    accepted: 21 october 2007      © 2007 pikas. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to better understand the  question negotiation process in libraries  both in intermediated and in self‐help  situations.  to achieve a richer  understanding of the relationship between  library users and library systems in order to  establish a research agenda and inform  librarian education.    design – the first part consisted of  qualitative research involving interviews.   the second part consisted of a diary study.    setting – special engineering libraries in the  united states and a university campus  (lehigh in bethlehem, pennsylvania).    subjects – the participants in the interviews  were special librarians.  special librarians  were selected because they have more  specialized knowledge and respond to more  substantive questions in greater depth than  do public and academic librarians who  emphasize instruction and who encounter  staffing restrictions that prevent them from  spending too much time on each inquiry.   detailed information on the selection of the  individual participants is not provided.    the participants in the diary study were  twenty undergraduate students who were  enrolled in an information science course.    methods – the interviews were open‐ended  and unstructured.  the interviews lasted  sixty to ninety minutes and were taped.  no  mailto:pikas@jhuapl.edu http://creativecommons.org/licenses/by/2.0 http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:4  89 information is provided on transcription or  analysis methods or paradigms.    in the second part, the students were given a  reading assignment on information seeking.   they then had to select a search topic and  document the steps they took, decisions  they made, and resources they used to  answer the question.  the participants were  asked to analyze their original question, the  type of answer required, and decisions they  made in the process.  no details are  provided on the analysis of the diaries.    main results – taylor found five filters  required for search definition:  1. determination of subject;  2. objective and motivation;  3. personal characteristics of the  inquirer;  4. relationship of inquiry description  to file organization;  5. anticipated or acceptable answers  (183)    these five filters provide general  information necessary for the for the search  definition.  these types are not mutually  exclusive and may occur simultaneously.    in the diary portion he found:  1. all participants consulted other  people including librarians and  fellow students;  2. none considered the library as a  whole;  3. all inquiries required multiple  sources; all answers were  synthesized from multiple sources;  4. participants were familiar with  library research:  they used the  classification schedule to search,  used subject headings, and used  indexes or tables of contents.  5. question or research problems  changed as a result of information  found    conclusion – question negotiation is a  dynamic process which requires feedback  and iteration to come to a conclusion.  the  librarian’s job is to work with the inquirer to  understand the information need and then  to translate the negotiated need into  appropriate search strategies.    the author suggests that library school  reference courses be updated to include  instruction related to communication and  negotiation in addition to the instruction on  resources.  he suggests more emphasis on  questions instead of commands; that is, a  cooperative process to determine what  information is needed and how to best fulfill  the need instead of assuming the inquirer  “knows exactly what he wants, can describe  its form (book, paper, etc.) and its label  (author and title)” (191).    to aid self‐help situations, the author  recommends better subject description of  resources and inquiry‐oriented instead of  object‐oriented systems.  he suggests  building better query negotiation into self‐ help systems.  at minimum the system  should request the user state his objective, if  for no other reason than to force the user to  reflect on or analyze his question.  help  should be available at the time of need, and  this can be offered through technology  instead of through staffing.    commentary    this article provides inadequate information  to judge the appropriateness of the research  methods or validity of the study.  no  information is provided on how the  participants were chosen or how those  choices may have influenced the results.  no  details are provided for how the data were  analyzed.  only a few quotes are used to  support the author’s conclusions.  this is,  however, a summary report.  the full  technical report has been scanned and is  available online through the defense   evidence based library and information practice 2007, 2:4  90   0 5 10 15 20 25 19 68 19 70 19 72 19 74 19 76 19 78 19 80 19 82 19 84 19 86 19 88 19 90 19 92 19 94 19 96 19 98 20 00 20 02 20 04 20 06 year c ita tio ns     figure 1: citations of taylor’s article, 1968‐2007      technical information center (taylor,  question‐negotiation 88).    some of the results of this study seem  obvious now, after decades of qualitative  studies of information behaviour; yet, this  article has had tremendous influence,  showing a steep upward trend in the  number of citations received per year (see  graph below).  evidence of this article’s  influence can be found in articles by belkin,  oddy, and brooks on the anamolous states  of knowledge (ask) model and many other  models of human information behaviour.   more recent articles citing this article discuss  interactive information retrieval, virtual  reference, query analysis, and relevance,  among other topics. in part, this influence is  due to parts of the paper not based on the  actual research.  for example, this paper is  frequently cited for its discussion of the  “four levels of question formation,” i.e.,     1. the actual, but unexpressed need for  information (the visceral need);  2. the conscious, within‐brain description  of the need (the conscious need);  3. the formal statement of the need (the  formalized need);  4. the question as presented to the  information system (the compromised  need) (182)    this listing appears in taylor’s earlier work  on question analysis where it is credited in  part to another author.  the levels are  presented in the 1968 article to describe the  necessity of negotiating back to the  conscious need instead of taking the  compromised need as given, especially if the  need is expressed in writing.    despite all of this, the paper’s continuing  relevance and importance cannot be  understated.  beyond the objectives of the  research mentioned above in the summary,  evidence based library and information practice 2007, 2:4  91 taylor clearly had a larger goal:  to enable  transformation in order to ensure  continuing relevance of the library in times  of dramatic changes.  he foresaw the  communications changes and the drastic  impacts these would have on how libraries  and librarians serve their patrons.  he  suggested that libraries could respond to  these challenges by withering away or by  undergoing a gradual transformation.  he  aimed his work at enabling this second  option and he was successful.  this paper  has been tremendously influential both in  generating new research streams, in training  practitioners, and in designing systems.    unfortunately, there are some valuable  lessons in this article that have not been  learned:  how to improve library signage,  how to provide help where and when  needed using feedback and iteration, the  importance of stressing the reference  interview and question negotiation  processes in library school, and so forth.   this article should be mandatory reading in  library school and professional librarians  should re‐read it annually.    works cited  belkin, nicholas j. ʺanomalous states of  knowledge as a basis for information  retrieval.ʺ canadian journal of  information science 5 (1980): 133‐43.  taylor, robert s. ʺprocess of asking  questions.ʺ american documentation  13.4 (1962): 391‐6.  ‐‐‐. question‐negotiation and information‐ seeking in libraries. bethlehem, pa:  center for the information sciences,  lehigh university, 1967. 5 nov. 2007  <http://handle.dtic.mil/100.2/ad659468 >.    http://handle.dtic.mil/100.2/ad659468 http://handle.dtic.mil/100.2/ad659468 http://handle.dtic.mil/100.2/ad659468 http://handle.dtic.mil/100.2/ad659468 evidence based library and information practice evidence based library and information practice 2010, 5.1  evidence based library and information practice       evidence summary    benefits of unionization still unclear for u.s. academic libraries and librarians     a review of:   applegate, r. (2009). who benefits? unionization and academic libraries and librarians. the library  quarterly, 79(4), 443‐463.    reviewed by:   diana k. wakimoto  online literacy librarian  california state university  east bay, hayward, california, united states of america  email: diana.wakimoto@csueastbay.edu    received: 26 oct. 2009          accepted: 04 jan. 2010       2010 wakimoto. this is an open access article distributed under the terms of the creative commons‐ attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐ sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original  work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is  redistributed under the same or similar license to this one.    abstract    objective – to investigate the quantitative  benefits of unionization for libraries,  librarians, and students at academic libraries  in the united states.     design – quantitative analysis of existing  datasets.    setting – academic libraries in the united  states.     subjects – one thousand nine hundred four  accredited colleges and universities in the  united states.     methods – institutions that provided data for  both the national center for education  statistics (nces) academic libraries survey  (als) and the nces integrated postsecondary  education data service (ipeds) survey series  in 2004 were considered for inclusion in this  study. of these institutions, only those with  student populations over 500 and employing  more than one librarian were included. the  study did not include specialized libraries at  institutions where “most of their degrees were  awarded in a single area” (p. 449). the  institutions were categorized by type derived  from data by carnegie and the association of  research libraries. the final categories were:  arl, doctoral non‐arl, masters,  baccalaureate, and associates. governance  was determined by using information from  ipeds that classified the institutions as public,  private not‐for‐profit, and private for‐profit.  unionization status was derived from the  directory of faculty contracts and bargaining  agents in institutions of higher education. after  144 evidence based library and information practice 2010, 5.1  private not‐for‐profit and private for‐profit  classifications were collapsed into one  category, governance and unionization  information were combined to create the final  governance categories of: private, public  nonunionized, and public unionized. the  study analyzed the following characteristics in  terms of institution type, governance, and  institution type and governance interaction:  ratio of students to librarians, ratio of library  expenditures to institutional budget  expenditures, average librarian salary,  percentage of staff who were librarians,  librarian salaries as a percentage of staff  salaries, and percentage of the library budget  spent on staff salaries.     main results – analysis revealed statistically  significant differences (p< .05) between  governance and student‐librarian ratio and  between governance and percentage of library  budget spent on staff salaries. no consistently  beneficial relationship between governance  and student‐librarian ratio was determined. a  consistently positive relationship was found  between governance and percent of the library  budget spent on librarian salaries; all public  unionized institution types had higher  percentages of the library budget devoted to  librarian salaries than private and public  nonunionized institutions. all five dependent  variables showed statistically significant  differences (p< .05) when analyzed by  institution type. analysis by institution type  and governance interaction found statistically  significant differences (p< .05) for student‐ librarian ratios, librarian salary, and  percentage of library budget devoted to staff  salaries. strong r2 values were determined for  the dependent variables of: staff salaries as a  proportion of library budget (.51) and student‐ librarian ratio (.34).      conclusion – based on the results, the author  stated that unionization appears to have  positive or neutral benefits for the library,  librarians, and students, regardless of  institutional type. further quantitative and  qualitative research is needed to analyze the  effects of unionization on library quality.    commentary    this study looks at whether quantitative  “benefits” are associated with unionization in  u.s. academic libraries. it will be of interest to  academic librarians, especially those who are  interested in comparing the benefits of  working at a different institution or under a  different governance structure. the author  clearly defined the context of the study with  an overview of the previous research on  unionization and libraries. while this study  does add to the understanding of unionization  of academic libraries, the stated results do not  support the conclusions of this article.     the author clearly explained the methodology,  described the variables analyzed as well as the  confounding variables, and used appropriate  descriptive and inferential statistics for the  variables analyzed. however, one of the  variables, librarian salaries as percentage of  staff salaries, was never analyzed. data from  either als or ipeds was lacking for some of  the institutions for three of the variables  analyzed. it is impossible to determine the  effect on the analyses as there were no further  explanations on which categories the lack of  data affected.     while the results do support the conclusion  that “being private is ‘best’ on most  quantitative measures for the library,” they do  not support the conclusion that “unionization  is best for librarians” (p. 461). private  institutions showed lower student‐librarian  ratios than either unionized or nonunionized  public institutions. librarians at public  unionized institutions have higher student‐ librarian ratios than at public nonunionized  institutions, except at the arl and  baccalaureate levels. librarians at unionized  public institutions also have lower salaries  than at public nonunionized institutions,  except at the masters and baccalaureate levels.  most of the statistically significant findings  reported in this study came from the analyses  by institutions type and not by governance.  also, it is unclear how the study is  summarized results in table 9 relate to the  145 evidence based library and information practice 2010, 5.1  146 previously discussed statistically significant  findings.     applegate frankly states “this study provides  some, but very limited, answers in the  unionization area” (p. 461) and the results  cannot be generalized beyond u.s.  “traditional, mainstream academia” (p. 449).  the analyses do reveal that some quantitative  differences can be correlated with governance,  but add more to the understanding of  differences correlated with institution type.  therefore the results are most helpful for  librarians who are considering where it would  be “best” to work based on institution type  and not on governance type. further  qualitative and quantitative research, as noted  by the author, is warranted in analyzing the  effects of unionization for academic libraries  and librarians.  it’s what’s on the inside that counts: analyzing student use of sources in composition research papers research article   it’s what’s on the inside that counts: analyzing student use of sources in composition research papers   james w. rosenzweig education librarian jfk library eastern washington university cheney, washington, united states of america email: jrosenzweig@ewu.edu   frank lambert assistant professor & mls program coordinator college of education middle tennessee state university murfreesboro, tennessee, united states of america email: frank.lambert@mtsu.edu    mary c. thill humanities librarian ronald williams library northeastern illinois university chicago, illinois, united states of america email: m-thill@neiu.edu   received: 30 aug. 2021                                                                  accepted: 12 oct. 2021      2021 rosenzweig, lambert, and thill. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30026     abstract   objective – this study is designed to discover what kinds of sources are cited by composition students in the text of their papers and to determine what types of sources are used most frequently. it also examines the relationship of bibliographies to in-text citations to determine whether students “pad” their bibliographies with traditional academic sources not used in the text of their papers.   methods – the study employs a novel method grounded in multidisciplinary research, which the authors used to tally 1,652 in-text citations from a sample of 71 student papers gathered from english composition ii courses at three universities in the united states. these data were then compared against the papers’ bibliographic references, which had previously been categorized using the why method.    results – the results indicate that students rely primarily on traditional academic and journalistic sources in their writing, but also incorporate a significant and diverse array of other kinds of source material. the findings identify a strong institutional effect on student source use, as well as the average number and type of in-text citations, which demographic characteristics do not explain. additionally, the study demonstrates that student bibliographies are highly predictive of in-text source selection, and that students do not exhibit a pattern of “padding” bibliographies with academic sources.   conclusion – the data warrant the conclusions that an understanding of one’s own institution is vitally important for effective work with students regarding their source selection, and that close analysis of student bibliographies gives an unexpectedly reliable picture of the types and proportions of sources cited in student writing.     introduction   many librarians may recognize the scenario in which an undergraduate student consults the reference desk shortly before their assignment is due, with a draft paper and bibliography already in hand, looking for a few more articles to reach the quota of peer-reviewed scholarship required by their course instructor. but how common is this scenario, really? and are these articles appearing only as bibliographic window dressing, or are they fully incorporated into the papers’ final text?    it is difficult to measure student reliance on a source. when is a student “citing” a source in the text of their paper? as novice researchers are not yet proficient in citation standards, authentic student writing is often messy and imprecise. incomplete citations, run-on sentences, and other errors in usage and formatting are abundant. moreover, student voice is often ambiguous, leaving the reader uncertain about what basis the student is using to make their claims. given this context, a consistent and accurate method is necessary for tallying in-text citations that is sufficiently flexible to handle the vagaries of student writing.   this study presents a new and rigorous method for counting citations in student writing. this method was applied to actual papers written by composition students, and those data were compared to the picture of student source selection as shown through analysis of those papers’ bibliographies using the why method, a research-validated taxonomy designed for the classification of individual sources. this comparison illuminates whether attention to in-text citations supplies insights that are unavailable from bibliographic analysis, and whether students’ bibliographies are reliable indicators of the sources used in the text of their papers.   background   this study is part of a multi-year, multi-institutional research project investigating student source use in academic writing, which has yielded multiple, previously published journal articles (lambert et al., 2021; rosenzweig et al., 2019). the research team is comprised of three librarians, whose collaboration began in 2013. three public universities from across the united states, designated with the pseudonyms pacific coast university (pcu), midwest state university (msu), and southeast university (seu), provided the most recent collection of student data in 2019. pcu and msu are master’s level institutions with m1 carnegie classifications, while seu is a doctoral-level institution with a d/pu carnegie classification. the universities were selected through convenience sampling, as we needed to leverage existing relationships with english teaching faculty to gain access to students for recruitment.    we collected research papers from english composition students at all three universities in order to subject them to an initial analysis of their bibliographic references. once student research papers had been collected and de-identified, the research team analyzed a representative sample of 71 student bibliographies using the why method. the why method, which stands for who, how, and why, consists of three facets that are the building blocks of source authority: 1) the credentials of the author or authoring organization as they pertain to the topic of the source; 2) the editorial process that the source underwent; and 3) the source’s publication purpose. each of the three facets is then divided into seven subfacets (see appendix a for a complete list and description of all subfacets). in combination, each source’s three subfacets provide a more nuanced description of the kind of authority that source claims.   for example, a piece in the economist by a professor of political science discussing the history of the filibuster would receive three subfacets. first, as a person holding a postgraduate degree in a field relevant to the subject at hand, the author identity is wf (academic professional). second, as the economist is edited by professional journalists, the editorial process is he (editor & editorial staff). lastly, since the economist is a for-profit publication, receiving both subscription and advertising revenue, the publication purpose is yb (commercial). the source’s full classification of wfheyb can be used to group it with similar sources, and provides insight into what kinds of authorities students trust. these classifications are value-neutral, and do not depend on document format: they represent characteristics that occur in both traditional and non-traditional sources, and therefore are flexible in describing the landscape of source material currently used by university students. materials for implementing the why method are available freely online (thill et al., 2021).   the process of coding the sources in student bibliographies took place in 2019 and 2020. we concluded that student source selection in research bibliographies is affected most powerfully by the variables of which institution a student attends, student age, and whether the student is a first-generation university student. moreover, the two categories most closely associated with library resources, wfhfyf (academic professional; peer-reviewed; higher education) and weheyb (applied professional; editor & editorial staff; commercial), account for 55% of all references in student bibliographies across the three universities, while the remaining 45% came from a wide range of sources.    subsequent to the 2019–2020 study, which was published as a journal article in july 2021 (lambert et al., 2021), we shifted our attention from student bibliographies to the texts of their final research papers, to see whether bibliographies offer an accurate portrait of student source use, or whether students rely more heavily on some source types over others within the text of their papers.   literature review   in seeking to better understand source use in undergraduate research writing, we had already conducted and published previous analyses of bibliographies from final papers submitted in first-year composition courses (lambert et al., 2021; rosenzweig et al., 2019) using a three-facet classification method modified from a taxonomy developed by leeder, markey, and yakel (2012). we hoped to uncover an analogous citation tallying method that had been previously published to use as a basis for this present study.   it was immediately evident that the library literature, which is rife with studies examining the contents of student research bibliographies, has relatively few articles explicitly counting and classifying the use of these bibliographic sources in the text of the written assignments themselves. scharf et al. (2007) addressed the integration of bibliographic sources into the body of the paper, but used a holistic scoring rubric as opposed to a more granular approach. knight-davis and sung (2008) established that in-text citations must be present in a paper to make it eligible for their sample, but they did not analyze these citations directly. a study by clark and chinburg in 2010 was the first article published by librarians to both count in-text citations and group them by source type. their work was an important precursor to our present study, but as their article did not include a rubric describing how they defined an in-text citation for counting purposes, it was not possible to model this study on their approach. furthermore, clark and chinburg grouped their bibliographic sources by broad categories, rather than classifying them by a more detailed method such as that used in this present study, which allows for more potential insight into the types of sources being used.   since clark and chinburg’s study, little further has been done to advance this kind of close analysis of in-text citations in the library literature. cappello and miller-young’s (2020) recent article did engage in a serious classification of in-text citations. however, their sample was composed of journal articles produced by highly trained scholars, which had a substantial impact on the methods they used to categorize and analyze the use of sources in that material. for our present study to be grounded on good research practice, it was necessary to look beyond the library literature to other disciplines in order to devise a plan for analyzing in-text citations in papers written by novice researchers in first-year composition courses.   literacy and language educators have made serious efforts to examine student citation behaviour. in 2010, ling shi worked with undergraduates who analyzed their own citation behaviour in their written work. she found that students make decisions about what to cite and when to cite it based on many factors. shi’s (2010) analysis did not, however, attempt to count citations or classify source types, focusing instead on the students’ stated rationales for their citation behaviour (p. 21). in their 2017 study, list, alexander, and stephens adopted a more quantitative approach by offering undergraduates a curated collection of six sources connected with an assigned question and counting how frequently students cited each source in writing a response to that question. their work is crucially relevant to this present study for several reasons: in it, they established a simple protocol for tallying both direct and parenthetical citations, and they examined student engagement with both traditional and non-traditional texts (list et al., 2017, pp. 89–91). although the writing task their study participants completed was artificial (and their list of sources necessarily limited by the nature of that task), the implication is clear that it is possible to gain insight into student behaviour by counting citations and comparing the use of different types of sources.   one important strand in this literature is the collection of studies published by the researchers working on the citation project. the citation project is a long-term, multi-institutional project that collects and analyzes cited material in the papers submitted by first-year undergraduates. as sandra jamieson (2017a) describes it, “the citation project is concerned with the ways students use [sic] material from the sources they cite” (p. 48). to facilitate this analysis, citation project researchers developed coding procedures that included specific instructions about what constituted an in-text citation for their purposes (jamieson and howard, 2011). their criterion for marking a citation was the presence of a “signal phrase identifying the source in some way (author, title, etc.),” a parenthetical citation, or both (jamieson and howard, 2011, p. 1). a number of studies followed their coding system, but did not examine either how closely the citations in the text matched the references in the bibliography or what types of sources were most commonly cited in the text (gocsik et al., 2017; scheidt et al., 2017).   jamieson’s (2017b) study, however, classified sources from research papers in first-year writing courses into 14 categories that reflected different combinations of format and type of content and examined how often sources from each category appeared in the bibliography and how often sources from each category were cited in the text. jamieson (2017b), in analyzing those data, concluded that first-year writing students rely largely on traditionally acceptable sources, and that students cite most of their sources only once in their paper’s text (pp. 127–128). while jamieson’s work bears many similarities to this present study, the why method’s system of source classification is both more objective and more granular, which may provide added insight into patterns of student source use. moreover, while jamieson’s (2017b) study commendably drew on a multi-institutional sample, that study’s data were gathered from so many different institutions—16 in total—that no statistical comparisons were possible between individual institutions (pp. 117–118), leaving open the question of what insights might be possible from such a comparison. the overall impact of the citation project’s work is undeniably significant and usefully guides our study to more meaningful levels of analysis.   the other important element from outside of the library literature is linguistic research into the various approaches authors take in referring to sources in their writing. howard williams (2010), in addressing the ways authors imply and readers infer source attribution, breaks down attribution into four categories: direct citation, textual phoric devices, free and quasi-free indirect speech, and implicit attribution (pp. 620–622). the first of these categories roughly corresponds to the methods of attribution counted in jamieson’s (2017b) study, and the final two categories are dependent enough on subjective impressions and subtle rhetorical indications that it is difficult for readers to agree regarding whether a given statement was being attributed to an external source (williams, 2010, pp. 621–624). the second category, however, was of a different kind: the use of phoric devices like pronouns to refer to a source is so consistent that williams (2010) describes the reader’s understanding of attribution as “practically guarantee[d]” (p. 621). although the example williams provides in his article is the use of a pronoun, other kinds of noun phrases also ordinarily serve as phoric devices in student writing—phrases such as ‘this article’ or ‘these scholars’—that act as a kind of in-text citation with relative unambiguity. as a result, we concluded that the citation counting procedure described by jamieson and howard (2011) could be improved by the inclusion of a thoughtfully constructed standard for tallying phoric devices, in addition to more formal citations. while williams (2010) was interested in both anaphoric (referring back to something previously identified) and cataphoric (referring to something not yet identified) devices, to avoid ambiguity we chose to focus solely on anaphoric devices.   aims   for the present study, our aim was to bridge from student bibliographies into the text of the associated research papers, to see if certain types of sources were more or less prevalent in the body of the paper than their proportion of each bibliography would predict. we developed an approach to facilitate this analysis, grounded in the available literature on accurately counting student in-text citations. if successful, this project would answer the following research questions: what kinds of sources do students in first-year english composition classes view as authoritative, based on how often sources from their bibliographies are cited within their research papers? to what extent do demographic characteristics and institutional differences influence student citation behaviour?   methods   the study participants are students enrolled in english composition ii, a standard course at institutions across the united states that prepares students to write college-level research papers. english department faculty offer composition courses to students from a variety of intended and declared disciplinary majors. students of english composition are often in their freshman or sophomore year of university, meeting the acrl framework’s definition of a “novice learner,” in that they are “developing their information literate abilities” (association of college & research libraries, 2016).   after obtaining irb approval from their respective universities, as well as permissions from supportive teaching faculty at the three universities, members of the research team visited in-person sections of english composition ii in spring of 2019 to enlist student participants. participants agreed to share the following items with members of the research team: a clean, ungraded copy of their final research paper and bibliography; selected personal information held in the office of institutional research at their university, including their age, their class standing, and their cumulative gpa; and a completed survey with additional demographic questions about gender, race/ethnicity, and first-generation status. we collected 239 english composition papers from 32 sections from all universities: 167 papers from 17 sections at pcu, 53 papers from 10 sections at msu, and 19 papers from 5 sections at seu.    in fall of 2020, the research team began our examination of student in-text citations. we planned to count each time a student cited a source from the bibliography in the course of their research paper in order to determine which sources students might view as the most authoritative. in conducting this new study, we used the same systematic sample of references obtained from 71 student papers whose bibliographies had been previously analysed using the why method so that we could track what types of sources were cited most frequently in the text. by institution, pcu contributed 35 papers and 954 of the citations in our sample, msu contributed 19 papers and 518 citations, and seu contributed 17 papers and 179 citations. developing a reliable method for counting student references, given the irregularities in the writing and citation practices of novice learners, was a first objective of this analysis. we considered employing the method used by jamieson and howard (2011) of tallying parenthetical citations and direct mentions of sources in the text but were concerned that some source use would go unrecorded, given the variations in how students approach written argument. the use of anaphoric phrasing to refer to sources, as described by williams (2010), was frequent enough in the sample that a more comprehensive model for citation counting was necessary.   ultimately, we created a flowchart and guidelines that combined the jamieson/howard and williams approaches (appendix b). the tallying method captures three different kinds of in-text citations: direct, parenthetical, and anaphoric. in direct citations, students reference a source from their bibliography by providing some piece of identifying information (such as the source’s author, authoring organization, title, or publisher) within the text of a sentence. in parenthetical citations, students provide identifying information within parentheses at the end of a sentence. in anaphoric citations, students use personal pronouns or noun phrases to refer to a source already cited (directly or parenthetically) in that paragraph. when a student used both direct and parenthetical conventions to reference the same source within a single sentence, we tallied that sentence as a single, direct citation. when a student used either a direct or a parenthetical convention in addition to an anaphoric device in one sentence, we tallied that sentence as either direct or parenthetical and did not tally it as anaphoric.   following the established flowchart, two members of the research team jointly counted the number of times each source from a student’s bibliography was cited within the student’s research paper. the third member of the research team tallied in-text citations from a systematic sample of 10% of the papers in our overall sample to validate the counting method. the third member matched the counts of the two research team members for 96.14% of all citations, 97.6% of direct citations, 97.08% of parenthetical citations, and 73.9% of anaphoric citations. these high rates of agreement demonstrate the rigorous method we developed to classify these citations, while indicating the greater challenge of determining what constitutes an anaphoric citation. we also tracked sources from student bibliographies that never appear within the text of the papers. this study refers to this phenomenon as “ghost sources”—sources whose presence in the paper is ephemeral and potentially misleading.   the citation count data analysed here is novel, but the demographic characteristics of the student authors are necessarily the same as those reported in the previous bibliographic analysis study (lambert et al., 2021). for that reason, detailed demographic information is already available in our previously published research, but key characteristics are repeated here for the convenience of the reader. the mean average age of our sample population was 20.33 years old. the majority of student participants described themselves as female (70%) with a freshman class standing (73.1%). sophomores comprised 19.1% of the study’s sample, with juniors and seniors making up the small remainder. forty percent of participants self-identified as first-generation students. the self-reported racial/ethnic origins for students in this sample were 59.4% white, 18.8% hispanic, and 10.1% asian (with biracial, black, and pacific islanders comprising the remaining 11.6% of the sample).   findings   this study presents descriptive and inferential statistical findings of the citing characteristics and behaviours of the sample population.   overall, there was a mean of 2.69 citations (sd = 2.515) per bibliographic source in our sample. a plurality of these citations were parenthetical citations (mean = 1.18, sd = 1.361), followed by direct citations (mean = 1.17, sd = 1.641), and then anaphoric citations (mean = 0.34, sd = 3.328). in all cases, these data had a positive skew in excess of 2.479, with phoric citations having the greatest skew (3.328) due to two or three outliers. as with most citation data, our data’s distributions match the distribution of a power curve (brzezinski, 2015). the team’s hypothesis that these data are non-normally distributed is confirmed by the use of a single-sample komogorov-smirnov (ks) test on all citations and on each citation type. each of these four variables (total citations; direct citations; parenthetical citations; and anaphoric citations) have distributions that match a poisson distribution (z = 4.136; 2.332; 3.718; and 1.722, respectively, p < 0.001). based on these large numbers of variability and skew, using non-parametric inferential statistical tests that focus on the median as the measure of central tendency will result in more accurate calculations of the test statistic.   table 1 number of citations per paper, by type, and by institution   mean, all citations/source mean, direct citations/source mean, parenthetical citations/source mean, phoric citations/source pcu 3.49 1.67 1.27 0.57 msu 2.09 1.45 0.37 0.27 seu 2.03 0.55 1.36 0.12   a mann–whitney independent samples test, which compares the mean rank of two separate ordinal or non-normally distributed distributions, reveals that the student’s gender predicts certain citation behaviours. overall, females were significantly more likely than males to cite their sources (z = -1.964, p = 0.05) and to use parenthetical citations (z = -2.4, p < 0.01).   the types of citations vary considerably between universities, as may be seen in table 1. a kruskal–wallis test, which compares three or more independent variables all at once, thus increasing the power of the test result, reveals significant differences between institutions for all citations (pcu papers having significantly more on average; h = 46.476, p < 0.001) and direct citations (pcu having considerably more direct citations on average per paper; h = 24.084, p < 0.001).   the writer’s first-generation status has a significant impact on the number of anaphoric citations found in each paper, with non-first-generation students using anaphoric citations far more than first-generation students (z = -2.586, p < 0.05). student class (freshman, sophomore, junior, and senior) is predictive of the type of citation used in their papers. a kruskal-wallis test reveals there were significant differences in all citations (h = 8.410, p < 0.05), direct citations (h = 6.777, p < 0.05), and parenthetical citations (h = 15.836, p < 0.01) based on the respective student’s class. in all citations, freshmen and sophomores had considerably more citations in their papers compared to juniors and seniors (with the caveat that upperclassmen comprised only 7.1% of our sample). juniors used significantly more direct citations than either freshmen or sophomores. sophomores, juniors, and seniors used significantly more parenthetical citations than did freshmen.    the complete citation count data (as seen in appendix c) show that approximately 75% of all in-text citations are represented by just eight source types, consistent with the bibliographic reference data from our previous study (lambert et al., 2021). a substantial plurality of citations came from wfhfyf (academic professional; peer-reviewed; higher education) sources. when we analyzed source use separately by university, however, each institution’s data diverge significantly from the overall pattern. for example, at pcu, 61.2% of cited resources were classified as wfhfyf (academic professional; peer-reviewed; higher education). at the other two universities, students use these types of resources far less. seu students cited wfhfyf sources 22.3% of the time, whereas msu students cited this same resource type 25.8% of the time. each individual university’s top eight most cited source types are presented in tables 2–4.   table 2 pacific coast university’s (pcu’s) eight most frequently occurring citation types citation type frequency percent cumulative percent wfhfyf 584 61.2 61.2 weheyb 51 5.3 66.6 wehfyf 40 4.2 70.8 weheyf 29 3.0 73.8 wfheyf 27 2.8 76.6 wfheyc 20 2.1 78.7 wbheyb 17 1.8 80.5 wfhdyf 17 1.8 82.3   table 3 southeast university’s (seu’s) eight most frequently occurring citation types citation type frequency percent cumulative percent wfhfyf 40 22.3 22.3 weheyb 24 13.4 35.8 wfheyb 19 10.6 46.4 wbheyb 14 7.8 54.2 wbheyc 14 7.8 62.0 wchayc 10 5.6 67.6 wfheyf 8 4.5 72.1 wfheyd 6 3.4 75.4   table 4 midwest state university’s (msu’s) eight most frequently occurring citation types citation type frequency percent cumulative percent wfhfyf 134 25.8 25.8 weheyb 117 22.5 48.4 wbheyb 44 8.5 56.8 wchaye 28 5.4 62.2 weheyc 22 4.2 66.5 wfhdyc 14 2.7 69.2 wchayc 13 2.5 71.7 wfheyb 13 2.5 74.2   table 5 documents the sources listed in the bibliographies that were never cited in the text, grouped by source type. the research team calls these sources “ghost sources” due to their evanescent presence in the students’ papers. the research team identified exactly 100 ghost sources, which comprised 14% of the 712 bibliographic references in the sample, with the largest proportion being wfhfyf (academic professional; peer-reviewed; higher education) sources. lastly, the four bar charts (figures 1–4) display the eight source types appearing most frequently in the student bibliographies, along with a category for all other types. the light bar represents the percentage of all in-text citations for that source type; the dark bar represents the percentage of all bibliographic references for that source type. as is evident in each chart, at both the aggregate and individual university level, there was considerable consistency between reference and citation data.   table 5 facet combination in bibliographies, not cited in papers, most frequent to least frequent facet combination facet translation number of occurrences wfhfyf academic professional; peer-reviewed; higher education 47 weheyb applied professional; editor & editorial staff; commercial 16 wfheyb academic professional; editor & editorial staff; commercial 5 wchayc corporate author; self-published; nonprofit 4 wfheyf academic professional; editor & editorial staff; higher education 3 wzhayf source unknown; self-published; higher education 3 wfhdyc academic professional; moderated submission; nonprofit 3 weheyc applied professional; editor & editorial staff; nonprofit 3 wbheyb layperson; editor & editorial staff; commercial 2 wzhzyz all sources unknown 1 wfheyc academic professional; editor & editorial staff; nonprofit 1 wfhdye academic professional; moderated submission; government 1 wehfyf applied professional; peer reviewed; higher education 1 weheyf applied professional; editor & editorial staff; higher education 1 wdheyc professional amateur; editor & editorial staff; nonprofit 1 wchfyf corporate authorship; peer reviewed; higher education 1 wcheyc corporate authorship; editor & editorial staff; nonprofit 1 wcheyb corporate authorship; editor & editorial staff; commercial 1 wbhfyf layperson; peer reviewed; higher education 1 wbhdyf layperson; moderated submission; higher education 1 wbhayf layperson; self-published; higher education 1 wahcyc unknown authorship; collaborative editing; non-profit 1 wahayb unknown authorship; self-published; commercial 1 total   100   figure 1 all universities, percent of all citations and percent of all references compared.   figure 2 pacific coast university (pcu), percent of all citations and percent of all references compared.   figure 3 southeast university (seu), percent of all citations and percent of all references compared.   figure 4 midwest state university (msu), percent of all citations and percent of all references compared.   discussion   given that the study’s citation counting method was novel (though grounded in literature from other disciplines), it was encouraging to see the very high rates of interrater agreement for total citations, direct citations, and parenthetical citations. these data bolster our confidence in our method and suggest that it has potential for use in subsequent research in librarianship. while the interrater agreement regarding anaphoric citations lagged behind the other citation types, it nevertheless reached an acceptable level for analysis. anaphoric citations are, by their very nature, more subject to interpretation than the other kinds of citation measured. we suspect that there will always be a higher level of variation between raters in tallying anaphoric citations, but the successful rate of agreement realized in this initial use of the citation counting method is a good indication that these anaphoric citations can be analyzed meaningfully.   while demographic characteristics such as gender, class level, and first-generation status show modest impacts on certain elements of student citation behaviour, the institution the student attends was the factor with the broadest range of impact, affecting every student citation behaviour we recorded. each institution’s students favoured different source types in their in-text citations. for instance, sources by non-professional authors in professionally edited commercial publications (wbheyb) made up 8.5% of msu’s citations and 7.8% of seu’s citations, while only 1.8% of pcu’s citations fell into this category. many of these institutional dissimilarities were already evident in the papers’ bibliographies (lambert et al., 2021), but they are even more pronounced in the findings presented here. with no clear demographic explanation for these divergences, a reasonable explanation lies in elements of institutional culture and pedagogy, such as required textbooks or carnegie classification, which are beyond the scope of this study. to be effective in supporting students, librarians must arrive at a deeper understanding of how sources are actually used in academic writing at their own institution. this may involve analysis of student work, as in this study, and it may also require engagement with classroom instructors or program administrators.   there were other institutional patterns in the citation data, as well: pcu students cited each source notably more often, on average, than either msu or seu students did. furthermore, while msu and seu students had a nearly identical mean average number of citations per source, the underlying citation behaviour showed a wide divergence. seu students largely relied on parenthetical citations, which msu students used only sparingly, and an inverse pattern is evident with direct citations. we had not anticipated these inter-institutional variances, and believe that it is possible the different approaches to in-text citations may be indicative of differing relationships between students and source texts at each institution. our feeling is that it is likely no accident that pcu, the university whose students cited by far the most traditional academic sources, is also the university whose students cite their work most extensively in the text, and that the same underlying factors causing one behaviour may cause the other. but we remain curious about the less explicable differences between the behaviours exhibited at msu and seu, and wonder if institutional culture and pedagogy alone could be responsible for establishing highly dissimilar expectations for citation practices.   in spite of the profound effect of institution on source selection, this study reveals that students, regardless of institution, do broadly share some general attitudes about source authority. most notably, they rely heavily on those sources traditionally recommended by librarians and composition instructors, namely peer-reviewed journal articles by academic professionals (wfhfyf) and professionally edited work for commercial publications by journalists or other skilled professionals (weheyb), which comprise 57.5% of all in-text citations (as seen in appendix c). in an era when many express fears that young people are susceptible to “fake news,” these data indicate that college students remain reliant on sources that have long been trusted within the academy. while this finding may reassure those who value these traditional expectations, it is not clear from the data that this reliance on dense, scholarly material in particular is well-advised for the level of argument employed by composition students, particularly given available reports on literacy levels for american high school graduates (goodman et al., 2013; kutner et al., 2006). it is also unclear whether students are learning to select wfhfyf sources in compliance with an obligation demanded by instructors, or whether they are savvy consumers of information who recognize when these highly-credentialed sources are most appropriate.   these citation data reveal a larger information literacy challenge in the less traditional materials appearing in student papers across all three institutions. the 42.5% of in-text citations that do not refer to wfhfyf or weheyb sources encompass a total of 54 other source types, including works by untrained or anonymous authors, articles edited by people who lack journalistic experience, and even personal blogs. we want to emphasize that authoritative knowledge is disseminated in more settings than peer-reviewed journals; podcasts, twitter feeds, and video essays (to name a few examples) are all appropriate media through which to share knowledge in the 21st century. composition papers encompass such a range of subjects and questions that none of this non-traditional material can be automatically ruled out as inappropriate. but have students been equipped to assess the credibility of this array of sources? what expectations do librarians and instructors have—or should they have—about non-traditional material? the framework for information literacy charges librarians not only with promoting the high-quality traditional sources available through our subscriptions, but also with preparing students to navigate an increasingly complex information environment online (association of college & research libraries, 2016). the why method illuminates the salient characteristics of the diverse array of material being used by students in their papers. librarians and instructors are therefore strongly encouraged to leverage the why method and its clear, consistent terminology in their teaching to make students more critically aware of the building blocks of source authority (thill et al., 2021).   the most surprising conclusions in the data came from the relationships we discovered between student bibliographies and student citation behaviour. based on our anecdotal experiences working with students, we had expected to see that students relied on some types of sources disproportionately in the text of their papers, citing those source types far more often than their presence in the bibliography had suggested. instead, we found that the proportions for each source type in the student bibliographies were excellent predictors of how often each source type would be cited in the text. similarly, we had expected that composition students would “pad” their bibliographies with traditional materials such as peer-reviewed journal articles and journalistic pieces in order to please professors or meet minimum assignment requirements but would fail to cite these sources in the text. instead, when we reviewed the ghost sources appearing in the bibliography but not the text, we found no evident pattern: the ghosts comprised a wide array of bibliographic sources that proportionally resembled the source types in the bibliographies as a whole. these discoveries lead us to a single overarching conclusion: that students do not make the kind of secondary judgments about source use that we had anticipated. the decision to include a source in their bibliography and the decision to cite a source in the text are a single choice. therefore, although our analysis of student in-text citations did yield useful insights, it also suggests that future studies can rely principally on bibliographic analysis, since the contents of those bibliographies will be a good indicator of what students employ in the text.   limitations and future research   while this study produced clear answers to its research questions, there remain several potentially interesting avenues for future research inquiry. the richest area for investigation is the evident institutional effect on student source selection and use. the data collected by this study do not indicate any demographic trends that would explain these differences. the explanation may lie in student demographic differences unexplored by this study, or it may more likely lie in institutional decisions made by instructors or administrators, whether explicit or implicit. future research will either need to collect additional student demographic data, perhaps using qualitative approaches like interviews or focus groups, or else collect documentation of instructor and administrator expectations, which may reside in syllabi, assignment descriptions, or departmental or programmatic guidelines. these same data might also yield insights into the unusual divergences between institutions in the average number of in-text citations per source and the types of citation that are most prevalent at each university.   there are in fact two types of “ghost sources”—this study addressed just one type, what the authors call “downstairs ghosts,” which are sources appearing in the bibliography but uncited in the text. it leaves as yet unaddressed the other type, the “upstairs ghosts,” which are sources cited in the text yet absent from the bibliography: exploring and analyzing this phenomenon requires that these source materials be sought out, to the extent that they can be found using the student’s in-text citation information, and therefore that research would take time.    because the institutions providing data for this study are broadly similar as public universities described by the carnegie classifications of d/pu (doctoral/professional universities) or m1 (master’s colleges & universities larger programs), it might be revealing to repeat this study using data from a major research university classified as r1 (doctoral universities very high research activity) or from a smaller institution, such as a baccalaureate college or associate’s college. these data might show even starker differences between institutions or uncover cases in which student citation and bibliographic reference data are not as closely linked as they are in this study.   conclusion   we embarked on this study after a multi-year focus on bibliographic analysis, curious to see how observing and understanding in-text citation behaviour would add to our understanding of student ideas about source authority. what we found helped to broaden our understanding of student relationships to sources, as well as to reaffirm the insights gained from bibliographic analysis. the citation counting method was effective and showed high rates of agreement. the data indicate that the institution variable has a more pervasive relationship to citation behaviour, both in what source types are cited in the text and how they are cited, than any other demographic factors we collected. furthermore, across all three universities, students cited sources in direct proportion to each source type’s presence in their bibliographies. this conclusion dispels the notion that procrastinating students might select scholarly sources strictly to impress their writing instructor and with no intention of incorporating the source into their final paper. likewise, it indicates that, if poor quality resources appear in a student’s bibliography, these resources are as likely to be cited as more credible materials. the fundamental takeaways are that an understanding of one’s own institution is vitally important for effective work with students regarding their source selection, and that attention to student bibliographies gives a reliable picture of the landscape of sources used in student writing. given these realities, librarians and composition instructors are advised to conduct bibliographic analysis of student work using the why method as an expeditious measure of the types and proportions of sources trusted by student writers, and to do so, whenever possible, by gathering data from multiple institutions for comparison.   author contributions   james rosenzweig: conceptualization, investigation, methodology, project administration, writing – original draft, writing – review & editing. frank lambert: conceptualization, formal analysis, methodology, validation, visualization, writing – original draft, writing – review & editing. mary thill: conceptualization, investigation, methodology, writing – original draft, writing – review & editing.   references   association of college & research libraries. (2016). framework for information literacy for higher education. https://www.ala.org/acrl/standards/ilframework   brzezinski, m. (2015). power laws in citation distributions: evidence from scopus. scientometrics, 103, 213–228. https://doi.org/10.1007/s11192-014-1524-z    cappello, a., & miller-young, j. (2020). who are we citing and how? a sotl citation analysis. teaching & learning inquiry, 8(2), 3–16. https://doi.org/10.20343/teachlearninqu.8.2.2   clark, s., & chinburg, s. (2010). research performance in undergraduates receiving face to face versus online library instruction: a citation analysis. journal of library administration, 50(5–6), 530–542. https://doi.org/10.1080/01930826.2010.488599   gocsik, k., braunstein, l. r., & tobery, c. e. (2017). approximating the university: the information literacy practices of novice researchers. in b. j. d’angelo, s. jamieson, b. maid, & j. r. walker (eds.), information literacy: research and collaboration across disciplines (pp. 163–184). university press of colorado. https://doi.org/10.37514/per-b.2016.0834.2.08   goodman, m., finnegan, r., mohadjer, l., krenzke, t., & hogan, j. (2013). literacy, numeracy, and problem solving in technology-rich environments among u.s. adults: results from the program for the international assessment of adult competencies 2012. first look (nces 2014-008). national center for education statistics. https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2014008   jamieson, s. (2017a). the evolution of the citation project: developing a pilot study from local to translocal. in t. serviss & s. jamieson (eds.), points of departure: rethinking student source use and writing studies research methods (pp. 33–61). utah state university press. https://doi.org/10.7330/9781607326250.c001   jamieson, s. (2017b). what the citation project tells us about information literacy in college composition. in b. j. d’angelo, s. jamieson, b. maid, & j. r. walker (eds.), information literacy: research and collaboration across disciplines (pp. 115–138). university press of colorado. https://doi.org/10.37514/per-b.2016.0834.2.06   jamieson, s., & howard, r. m. (2011). procedures for source coders. the citation project. http://www.citationproject.net/wp-content/uploads/2018/04/source-code-procedures-2011.pdf   knight-davis, s., & sung, j. s. (2008). analysis of citations in undergraduate papers. college & research libraries, 69(5), 447–458. https://doi.org/10.5860/crl.69.5.447   kutner, m., greenberg, e., & baer, j. (2006). a first look at the literacy of america’s adults in the 21st century (nces 2006-470). national center for education statistics. https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2006470   lambert, f., thill, m., & rosenzweig, j. w. (2021). making sense of student source selection: using the why method to analyze authority in student research bibliographies. college & research libraries, 82(5), 642–661. https://doi.org/10.5860/crl.82.5.642   leeder, c., markey, k., & yakel, e. (2012). a faceted taxonomy for rating student bibliographies in an online information literacy game. college & research libraries, 73(2), 115–133. https://doi.org/10.5860/crl-223   list, a., alexander, p. a., & stephens, l. a. (2017). trust but verify: examining the association between students' sourcing behaviors and ratings of text trustworthiness. discourse processes, 54(2), 83–104. https://doi.org/10.1080/0163853x.2016.1174654   rosenzweig, j. w., thill, m., & lambert, f. (2019). student constructions of authority in the framework era: a bibliometric pilot study using a faceted taxonomy. college & research libraries, 80(3), 401–420. https://doi.org/10.5860/crl.80.3.401   scharf, d., elliot, n., huey, h. a., briller, v., & joshi, k. (2007). direct assessment of information literacy using writing portfolios. the journal of academic librarianship, 33(4), 462–477. https://doi.org/10.1016/j.acalib.2007.03.005   scheidt, d., carpenter, w., fitzgerald, r., kozma, c., middleton, h., & shields, k. (2017). writing information literacy in first-year composition: a collaboration among faculty and librarians. in b. j. d’angelo, s. jamieson, b. maid, & j. r. walker (eds.), information literacy: research and collaboration across disciplines (pp. 211–233). university press of colorado. https://doi.org/10.37514/per-b.2016.0834.2.08   shi, l. (2010). textual appropriation and citing behaviors of university undergraduates. applied linguistics, 31(1), 1–24. https://doi.org/10.1093/applin/amn045   thill, m., lambert, f., & rosenzweig, j. (2021, july 23). the why method: an updated approach to source evaluation. eastern washington university libraries. https://research.ewu.edu/thewhymethod   williams, h. (2010). implicit attribution. journal of pragmatics, 42(3), 617–636. https://doi.org/10.1016/j.pragma.2009.07.013   appendix a why attribute codes for coding paper references   resources related to this study, including the full coding taxonomy that includes scope notes, may be found at the following online libguide: https://research.ewu.edu/thewhymethod   author (who) identity attribute author identity category brief description wa: unknown authorship no identification is possible. wb: layman a person without demonstrated expertise in the area being written about wc: corporate authorship no single author identified on a work issued by an organization wd: professional amateur a person with a degree in another field, but demonstrating interest, dedication, and experience in the area being written about we: applied professional a person with relevant experience, training or credentials relevant to the area being written about (i.e., journalist with journalism degree or substantive professional experience) wf: academic professional a person with a master’s or doctoral degree in the area being written about, which they held at the time the content was published. wz: source unknown no information on the category could be found   editorial (how) process attribute editorial process category brief description ha: self-published material made public directly by the author hb: vanity press material the author paid to publish, generally as self-promotion hc: collaborative editing material that is reviewed or edited by multiple possibly anonymous collaborators hd: moderated submissions contributed content that has been accepted or approved by someone other than the author he: editor and editorial staff professionally reviewed and approved by editor/editorial staff hf: peer reviewed evaluated by members of the scholarly community before acceptance and publication hz: source unknown no information on the category could be found   publication (why) purpose attribute publication purpose category brief description ya: personal  material is published without commercial aims yb: commercial  material is published for commercial gain yc: non-profit  material is published by a non-profit organization yd: k-12 education  material is published for educational purposes ye: government  material is published by the government yf: higher education  material is published for an academic audience yz: source unknown no information on the category could be found   appendix b citation counting method flowchart   appendix c frequency of all citations from all universities’ english composition classes   citation type frequency percent cumulative percent   wfhfyf 758 45.9 45.9 weheyb 192 11.6 57.5 wbheyb 75 4.5 62.0 wfheyf 48 2.9 65.0 wehfyf 43 2.6 67.6 wfheyb 43 2.6 70.2 wchaye 40 2.4 72.6 weheyc 34 2.1 74.6 weheyf 34 2.1 76.7 wfheyc 32 1.9 78.6 wfhdyc 30 1.8 80.4 wbheyc 26 1.6 82.0 wchayc 25 1.5 83.5 wfhdyf 25 1.5 85.0 wchayb 21 1.3 86.3 wbhfyf 18 1.1 87.4 wbhdyb 16 1.0 88.4 wahfyf 14 .8 89.2 wbhdyf 14 .8 90.1 wchayf 12 .7 90.8 wdheyb 11 .7 91.5 wbhayf 10 .6 92.1 wcheyb 10 .6 92.7 wcheyc 9 .5 93.2 wfhdye 9 .5 93.8 wbhaya 8 .5 94.2 wdhfyf 8 .5 94.7 wbhayb 6 .4 95.1 wfheyd 6 .4 95.5 wbheyf 5 .3 95.8 wcheyf 5 .3 96.1 wehdyc 5 .3 96.4         weheyd 5 .3 96.7 wchdyb 4 .2 96.9 wcheyd 4 .2 97.2 wdheyd 4 .2 97.4 wfhdyb 4 .2 97.6 wzheyb 4 .2 97.9 wzhzyz 4 .2 98.1 wbheya 3 .2 98.3 wdhdyc 3 .2 98.5 wehayb 3 .2 98.7 wfhayb 3 .2 98.8 wahdyb 2 .1 99.0 wahdyf 2 .1 99.1 wchfyf 2 .1 99.2 wehaya 2 .1 99.3 wehfyc 2 .1 99.5 wfheye 2 .1 99.6 wbhdyc 1 .1 99.6 wbheyd 1 .1 99.7 wcheye 1 .1 99.8 wdhdyb 1 .1 99.8 wehdye 1 .1 99.9 wehdyf 1 .1 99.9 weheye 1 .1 100.0 total 1652 100.0             microsoft word art_bennett.doc evidence based library and information practice 2006, 1:1  37 evidence based library and information practice     article    name authority challenges for indexing and abstracting databases      denise beaubien bennett  engineering librarian and online coordinator, marston science library  university of florida, george a. smathers libraries  gainesville, florida, united states of america  e‐mail: dbennett@ufl.edu    priscilla williams  head, authorities and metadata quality unit cataloging and metadata department  university of florida, george a. smathers libraries  gainesville, florida, united states of america   e‐mail: priwill@uflib.ufl.edu      received: 01 december 2005    accepted: 22 february 2006      © 2006 bennett and williams. this is an open access article distributed under the terms of the creative  commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    abstract    objective ‐ this analysis explores alternative methods for managing author name changes  in indexing and abstracting (i&a) databases. a searcher may retrieve incomplete or  inaccurate results when the database provides no or faulty assistance in linking author  name variations.    methods ‐the article includes an analysis of current name authority practices in i&a  databases and of selected research into name disambiguation models applied to authorship  of articles.    results ‐ several potential solutions are in production or in development. mathscinet has  developed an authority file. the method is largely machine‐based but it involves time‐ consuming manual intervention that might not scale up to larger or multidisciplinary  databases. the use of standard numbers for authors has been proposed. solutions in  practice include author‐managed registration records and linking among several authority  files. information science and computer science researchers are developing models to  automate processes for name disambiguation, shifting the focus from authority control to  access control.  successful models use metadata beyond the author name alone, such as co‐ http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:1  38  authors, author affiliation, journal name, or keywords.  social networks may provide  additional data to support disambiguation models.     conclusion ‐ the traditional objective of name authority files is to determine precisely  when name variations belong to the same individual.  manually‐maintained authority files  have served library catalogues reasonably well, but the burden of upkeep has made them  ill‐suited to managing the volume of items and authors in all but the smallest i&a databases.  to meet the access needs of the 21st century, both catalogues and i&a databases may need  to implement options that present a high degree of probability that items have been  authored by the same individual, rather than options that provide high precision with the  expense of manual maintenance.  striving for name disambiguation rather than name  authority control may become an attractive option for catalogues, i&a databases, and  digital library collections.      introduction    indexing and abstracting (i&a) databases  generally have not implemented name  authority control as is used in many library  catalogues. most i&a databases burden the  searcher with identifying and selecting  name variations. the use of widely variant  forms of authors’ names without reference  or linkage to alternatives causes hardship  for searchers.  end‐users’ search results may  be inaccurate or incomplete, resulting in a  decrease in the scientific integrity of the  research. this article will explore various  approaches to solving these challenging  name variation issues.    for many years, across research  communities, librarians and researchers  have had to deal with the problem of  increasing numbers of variant forms of an  author’s name. some variants are created  and occur over the life of a publishing career;  some may be attributed to author  preferences while others are created to  conform to requirements of publishing  guidelines. variant forms due to  misspellings, spacing, cultural norms, and  use of initials supply one set of concerns.  name changes, outgrowths of an author’s  life over time as a result of outside  influences involving such personal matters  as marriage and legal name changes,  provide a special challenge for database  maintainers as well as searchers.     individual library online catalogues have  been capable of applying authority control  methods since the implementation of  aacr2 (taylor 224). personal name  authorities bring together works by an  author, regardless of the variations in name  as identified in the work itself (tillett  “authority control” 24). name authorities  and related issues tend not to be discussed  in the database indexing world to the extent  they are discussed in cataloguing and back‐ of‐the‐book indexing (taylor 225; spink and  leatherbury 143‐44).     name authorities present many challenges  for i&a databases beyond those facing  maintainers of library catalogues. in  addition to variations in language  translations and cultural naming customs,  publication editors frequently dictate  whether authors may use their full names or  are restricted to their initials (see appendix  a). thus, i&a databases receive items that  may already contain name variations. i&a  databases may choose to exert some sort of  name authority control over the variations  to ensure that a search on one form of the  author’s name will retrieve all works by that  author. i&a databases tend to develop their  own procedures for handling name  evidence based library and information practice 2006, 1:1  39 authority issues, such as stripping all author  names down to initials. most i&a databases  cluster works by the form of author name,  but don’t provide redirects to other forms of  the authors’ names. for example, the  searcher must note and select all relevant  entries such as “last, f.,” “last, first,” “last,  f. m.,” “last, first m.,” “last, first middle,”  “last, f. middle,” middle last, f.” where all  of these variations are included in the  author index. some i&a databases, such as  the web of science  <http://scientific.thomson.com/products/wos />, both strip author names down to initials  and deliberately choose not to exert any  authority control, cautioning searchers to try  all likely name variations (“author names”;  web of science 7.0 workshop 41).    one particular challenge lies in managing  author name changes.  indexing practices  recommend appropriate treatment, such as:  “but if a person was well‐known also under  a previous name, cross‐references from and  to the changed name should be made…. the  same treatment applies to married women  who become well‐known under their  maiden names and continued to create  literary or artistic works or became  otherwise known also under their married  names.” (wellisch 360‐61).  few databases  have chosen to link the variations or name  changes to facilitate searching and retrieval  of an author’s works (see appendix b). i&a  databases may also move all of an author’s  works from the former name to the current  name (see appendix c), thus altering some  records so the author name no longer  matches that displayed on the original  article.    regardless of whether i&a databases  choose to link author variations, searchers  expect the form of name on the retrieved  bibliographic records to match the form of  the name on the published article. when the  names are significantly mismatched  between the i&a database and the article  itself, the searcher is likely to be confused.  future researchers may cite an article by  copying the form used in the i&a database,  thus carrying over the disconnect from the  name used on the article. further chaos  ensues when citations are gathered by  citation indexes and linking databases, such  as the web of science. any citation that uses a  form of the author’s name other than that on  the article will not match the correctly  identified items already in the web of science  database.    the challenges of coping with name  variations multiply when end‐users search  across multiple databases while formulating  their literature searches.  automated or  manual de‐duplication of identical items  becomes more problematic, whether end‐ users create their own bibliographies or  employ bibliography management software  to manage their citations, with name  changes than with simpler name variations.     linking services such as crossref  <http://www.crossref.org/> rely on digital  object identifiers and other numerically  hashed methods of identifying identical  citations to link through openurl both to  full text options and to shared citations.  where the matching and passing algorithms  rely only on numbers (such as issn, year,  volume, issue, starting page), problems with  name variations and changes may be  reduced from chaotic to merely puzzling.  where the algorithms include author names,  variations may reduce the probability of  matches and linkages. as long as  researchers rely on author names to identify  works, i&a databases can assist by clearly  identifying the name on the article as well as  its variations.    examples of problems with name changes    one author has published works under two  forms of her name: denise m. beaubien until  mid‐1992, and denise beaubien bennett  http://scientific.thomson.com/products/wos http://www.crossref.org/ evidence based library and information practice 2006, 1:1  40 after mid‐1992. a search for her works in  wilsonweb’s library literature & information  science full text database   <http://www.hwwilson.com/databases/liblit .htm>  for <denise beaubien> in [all ‐ smart  search] yields disturbing results (figure 1).     the author name on the articles of the five  oldest items, published 1988‐1992, is denise  m. beaubien. however, only one of the  citations [beaubien, d.m. “the changing  roles of online coordinators.” online  (weston, conn.) v. 15 (september 1991) p.  48‐50+] displays this form of the author’s  name. the other four older citations display  a form of name that (1) does not appear on  the articles and (2) has never been used by  the author but which appears to be an  amalgamation of the two forms of her name  created by the database indexers:  denise m.  beaubien bennett. all but one of the  citations from 1993 to the present also  display the amalgamated form of name, but  the initial “m” does not appear on the   articles and has not been used by the author  in any context, legally or professionally,  since mid‐1992. the most recent citation  [bennett, d.b., et. al., “a class assignment  requiring chat‐based reference.” reference  & user services quarterly v. 44 no. 2 (winter  2004) p. 149‐63] uses the form of the name  on the article, without the “m.”           figure 1: s earch for author’s older form of name in wilsonweb’s all – smart search. copyright © 2006 by the  h. w. wilson company. material reproduced with permission of the publisher. permission granted 2/13/2006. http://www.hwwilson.com/databases/liblit evidence based library and information practice 2006, 1:1  41   figure 2: citation in wilsonweb for older item displays author’s newer name. copyright © 2006 by the h. w.  wilson company. material reproduced with permission of the publisher. permission granted 2/13/2006.      other authors who have changed their  names suffer a similar fate.  a search for  <kathleen heim> in <author> yields 88  items, published from 1984‐2004.  however, a record in library literature &  information science for a publication from  1990 displays as is shown in figure 2., while  the author’s name on the article is:    kathleen m. heim    louisiana state university  school of library & information  science    the amalgamated name does serve to draw  the author’s works together. however,  searchers may be lulled into assuming the  amalgamated name is used throughout the  database. but redirecting a search on the  amalgamated name does not retrieve the  citations that use the form of name as listed  on all of the articles. in the case above, two  citations are not retrieved from the  redirected search.  examples from both of  these authors indicate that authority control  is applied incompletely in the database,  negating many of its benefits.  unsuspecting  searchers will not know they may have  missed some hits.     the concept of the amalgamated name may  aid in retrieving database records, but the  practice can lead to errors or variations in  citation functions. many researchers create  bibliographies by copying and pasting  citations from databases. this practice is  encouraged by database producers, who  develop excellent tools for marking, sending,  and saving records; and by librarians, who  encourage patrons to use these database  features as well as bibliography  management software to reduce citation  errors. when database citations do not  indicate the form of the name as used on the  article, errors in proper citing may follow.    the web of science, the original citation tool,  uses the form of author name (and the rest  of the citation) exactly as it appears in the  citing article, stripping all but the surname  down to initials. isi’s long‐standing policy is  not to over‐correct “variations” because its  indexers cannot check them all (cited  reference searching 3) and will not second‐ guess an author’s intentions. to search for  cited references in the web of science to all of  the first author’s works above, a searcher  should only have to enter two strings:  <beaubien dm or bennett db>.  however, if  authors copy the citation from a wilsonweb  database, a searcher must add <or bennett  dmb> to the web of science search string to  retrieve all the matches. the problem is  magnified when searching for citations to a  particular work.  when the searcher limits to  only the name on the article but an i&a  database has reformatted an author’s name  and a citer chooses the name from the i&a  database rather than the name on the article,  some citations will not be retrieved.   searching the web of science is challenging  enough when accommodating for authors’  typos. accommodating deliberate variations  and name changes introduced by i&a  databases adds to the complexity and  reduces the recall of items retrieved. in  addition to retrieval challenges, incorrect  evidence based library and information practice 2006, 1:1  42   figure 3: cited references in web of science displaying how the error in the index results in an error in the  cited reference matching and counting. thomson scientific, inc. is the publisher and copyright owner of web of  science®.  the screen shots are used with the expressed permission of thomson scientific.  permission granted  2/9/2006.      use of an author’s name by an i&a database  results in the creation of an additional  unlinked record in the web of science plus a  failure to increment the “times cited”  counter on the valid record for the item.  having one’s cited references grouped for  easy and complete counting is increasingly  important among authors (monastersky, sec.  2). the first entry in figure 3 is the valid  record. the second entry was created  because a citer (cardina and wicks 142)  copied the author’s name from a wilsonweb  database rather than from the article. not  only is the author’s work not officially or  correctly counted in the web of science, but  subsequent searchers cannot view the full  record of the original article within the web  of science because the view record link fails to  form. most i&a databases force the searcher  to generate all variations on the author  name to assure high recall of results. library  literature & information science and all the  wilsonweb databases are rare among i&a  databases in deliberately changing an  author’s name on a citation to correspond to  its latest known form (see appendix b).    potential solutions: overview    solutions to the problem of identifying and  linking author name changes within i&a  databases can take many approaches.   solutions both in production and in the  research modeling stage are clustered into  categories and described below:      1) authority control through the use or  linking of name authority files  a)  uses a file: mathscinet, wilsonweb  b)  proposed file: international  standard authority name/data  number  c)  linking across files: hopec, anac  levy project; leaf  2) name disambiguation through  automated methods  a)  in practice: author‐ity  b)  models in development by research  teams, including use of social  networks       maintaining name authority files requires a  high amount of labor but benefits the end‐ user with results of both high recall and  high precision (lancaster 131‐4) in  identifying documents by or about the same  individual.  automated methods of name  disambiguation may require less manual  labor but likely cannot achieve the level of  high recall and high precision of well‐ maintained authority files unless they also  employ substantial manual checking.    potential solutions: authority file in the  mathscinet database    the mathscinet  <http://www.ams.org/mathscinet/> database  creates and maintains a name authority file  to control variations. much of the  identification process is automated;  however, approximately twenty percent of  http://www.ams.org/mathscinet/ evidence based library and information practice 2006, 1:1  43 the items require manual checking.  (tepaske‐king and richert par. 10; uniquely  identifying mathematical authors).   “authors are distinct entities in the mr  database, independent of name variations  used in particular publications.”  (mathscinet author database help). in the  author database, search results are  displayed as a headline list of authors. the  primary listing is the preferred or fuller  form of the name. listed below the headline  are the variations on the name as used on  articles cited in the database. the searcher  immediately sees the name variations and  accepts that the variations point to the same  author. the headline name serves to group  the variations, but the form of name  displayed in each citation matches the name  on the article. searchers who mark and save  records to import into their bibliographies  will pass along the name variant as used on  the article, enabling future researchers to  match the citation and article without  confusion over the authorship.    a quick search in mathscinet for a  truncated author search retrieves records  that match only that form of name, and  many include more than one author, as is  typical in most i&a databases. a search in  the author database rather than quick  search for <kister, j*> displays two matches  on the truncated form (figure 4). each  match displays the entry from the authority  file and all name variations.  author <kister,  jane> has published using three variant  names.  the display indicates that all three  variations belong to one author, and  confirms the preferred form.      figure 4: author database entry in mathscinet. reprinted with permission by the american mathematical  society. permission granted 2/10/2006.  evidence based library and information practice 2006, 1:1  44 figure 5:  records retrieved from selecting radio button for authority name in mathscinet author database.  reprinted with permission by the american mathematical society. permission granted 2/10/2006.  selecting the radio button next to the chosen  author from the author database and  clicking the view all items button displays  a list of all items written by the author,  regardless of name variations on the articles  (figure 5). in contrast, searching the full  database by author’s earlier name <bridge,  jane*> retrieves only those records with that  form of name on the article (figure 6).  figure 6: direct search of mathscinet by non‐authority variation retrieves only those matches. reprinted  with permission by the american mathematical society. permission granted 2/10/2006. evidence based library and information practice 2006, 1:1  45 mathscinet’s solution is elegant and  workable in the relatively small database  where its authors come from a size‐limited  community and where it is possible for  human indexers to check and correct  problematic entries manually. although this  solution might not scale up to large  databases such as psycinfo, biosis,  chemical abstracts, or the web of science, it  should be possible to implement in  databases covering narrow disciplines such  as library literature & information science.    potential solutions: more examples  creating, using, or linking authority files    indexing and abstracting databases may  follow library of congress (lc) practice but  might find an additional benefit in perusing  the library of congress name authority  file (lc/naf) to assist in collocating the  variant names in author databases.    the library of congress name authority  file contains over 5 million name authority  records.  over 2 million of these records are  contributed by naco, the name authority  cooperative of the program for cooperative  cataloging run by the library of congress  (<http://www.loc.gov/catdir/pcc/naco/nacop ara.html> or  <http://authorities.loc.gov/help/contents.htm >). institutions become members of the  naco community and participate in the  shared environment of authority control by  contributing records to the lc/naf  following lc practice.  new and changed  name authority records are contributed to  the file. as the number of contributions  increase, the number of available names that  can be used increases. aside from providing  controlled author name access, the records  in the lc/naf are rich in a cross‐reference  structure that links name changes and  provides additional information that can be  used effectively in compiling author  databases. the lc/naf, designated the  ʺnationalʺ resource authority file, is not  strictly national and hasnʹt been since 1975.   an agreement with the national library of  canada (nlc) to use nlc headings when  creating new name authority records for  canadian personal name authors afforded  lc the opportunity to pursue its goal of an  international authority file.  also, lc is very  likely to use personal name author headings  already established by the nlc.  in addition  to nlc headings, the lc/naf contain  british and australian personal name  authors (kuhagen 132‐133).      although the lc/naf is created with data  from published books rather than from  published articles, i&a databases may  benefit from the effort that goes into  compiling the lc/naf. wilsonweb  databases check the lc/naf (see appendix  b), but err in changing authors’ names  rather than pointing to the variations as  given on the article. the lc/naf supports  high precision in linking name variations to  an individual, but the identification and  linking work is largely done by slow and  manual, albeit distributed, methods.  several  projects build on lc/naf and other  authority files; selected descriptions follow.    the ifla working group on functional  requirements and numbering of authority  records (franar) is working to develop a  “conceptual model to assist in an assessment  of the potential for international sharing and  use of authority data both within the library  sector and beyond” (g. patton 41).  one  charge to franar is “to study the  feasibility of an international standard  authority data number (isadn)” (g.  patton 40) which, if created, might serve as a  model for i&a databases as well as for  library catalogues, digital libraries, archives,  museums, and rights management  organizations. at present, the franar  draft report titled functional requirements for  authority records: a conceptual model (ifla  ucbim) does not yet address the isadn  issue. in a related effort, snyman and jansen  http://www.loc.gov/catdir/pcc/naco/nacop http://authorities.loc.gov/help/contents.htm evidence based library and information practice 2006, 1:1  46 van rensburg argue for the use of an  international standard author number  (isan) to reduce dependence on identifying  author name variations (“naco vs. isan”;  “reeingineering name authority control”).   opponents of standard number approaches  express concerns regarding organizational  maintenance costs (tillett “authority  control” 30; delsey 74).    the hopec system (cruz et al. 1‐8) controls  author records within the repec economics  library <http://authors.repec.org/>.  hopec  implements an author registration  component that enables authors to create  and maintain their own authority records.   hopec thus shifts the maintenance burden  away from a centralized group. authors  wishing for their papers to be clustered  must identify and manage their own name  variations.  reliance on authors leads to  uneven participation and data quality, but  the model offers a distributed solution with  low organizational maintenance costs.    librarians recognized long ago that linking  methods could substitute for authorized  forms of names (tillett “authority control”  25).  in the automated environment, a  system does not have to select one “correct”  form as long as all the variations link to each  other. the getty union list of artist names  online  <http://www.getty.edu/research/conducting _research/vocabularies/ulan/> links records  that have been created within several  separate authority files.    members of the large‐scale levy project to  digitize a sheet music collection have  created an automated name authority  control system (anac) based on the lc  name authority file to facilitate  interoperability (dilauro et al. sec. 3;  warner and brown 21‐2). the metadata  include the statement of responsibility, such  as “composer” or “words by.”  probability is  based on a model that permits updating  after new data are added. anac was  successful in establishing matches 58% of  the time: 77% when a name existed in  lc/naf and 12% when a record did not  exist in lc/naf. anac took about eight  seconds per name to perform the  classification and is viewed as a complement  to human effort (m. patton et al. sec. 6).    the leaf project for linking and exploring  authority files creates a “shared name  authority file” (weber 233) that can be used  by all participating database producers.  leaf automatically links all authority  records that pertain to the same person,  based on the automatic linking rules of the  project and including birth/death dates (232).  leaf utilizes the z39.50 protocol for  searching across authority files. leaf does  not merge the records into a new entity; it  preserves each local file’s practices.  multidisciplinary databases might follow  the leaf lead in linking authority files that  may exist within smaller or narrower  disciplines.    barbara tillett outlines the progress toward  building a virtual international authority file  in a series of papers (“virtual international  authority file”; “aacr2 and metadata”;  “authority control”). these cooperative  efforts are based on linking parallel  authority records that will continue to be  maintained locally rather than attempting to  merge metadata into super records. tillett  favors testing of unique, persistent record  control numbers within existing services  (“authority control” 30) or any method that  does not require establishing an  international organization to maintain  standard numbers. ki‐tat lam proposes  converting authority files to an xml format  and enabling the files as soap nodes (93‐95)  to achieve global name access. linked  authority records may assist efforts at  identifying more name variations that point  to a single individual. however, name  variations occur more frequently in the  http://authors.repec.org/ http://www.getty.edu/research/conducting evidence based library and information practice 2006, 1:1  47 journal literature than in library catalogues  due to editing and indexing practices.  linked authority records are still limited to  the metadata variations included in those  records.      potential solutions: alternative  approaches using name disambiguation    digital libraries are examining the issues  involved in name authority control as well  as topical authority control. “such name  ambiguity affects the performance of  document retrieval, web search, [and]  database integration, and may cause  improper attribution to authors.” (han et al.  “a hierarchical naïve bayes mixture  model” 1065). rather than devising name  authority files, researchers are aiming for an  outcome of name disambiguation, or an  automated method of examining more than  the author name to determine the likelihood  that any two papers with similar author  names i.e., last name and first initial, have  been written by the same person.  the  challenges are summed up by malin, airoldi  and carley who state, “in the real world, it  is not clear if any observed name ever has  complete certainty. this suggests  probabilistic models of certainty may be  useful for disambiguating names when  many names are potentially ambiguous.”  (136).     eugene garfield, founder of the science  citation index (now in database form as the  web of science) long ago acknowledged the  need to examine more data than name and  initials alone to disambiguate authors. “on  the other hand, when using the source index  of the sci to locate articles written by a  particular author it is not possible to  differentiate between two different men  with the same name and initials, unless one  knows something about their fields of  work.” (2)      the term authority control is generally  restricted to the library world, and is  increasingly limited to catalogues. other  disciplines solve similar records‐ management problems.  digital libraries  strive instead to create access control, where  variations are linked without establishing an  official or preferred version (cruz et al.).  statisticians discuss record linkage to match,  for example, family members in health care  or census files (bhattacharya and getoor 12;  fellegi and sunter 1183‐4). database  maintainers use deduplication or citation  matching or identity uncertainty (pasula et al.  sec. 1; on et al. 346), which librarians  generally consider as a method for  identifying entire records that match rather  than matching just the author fields in  records. all of these fields offer models that  assist with fuzzy matching, but many are  not geared specifically toward  accommodating name changes that  incorporate different words.      authority name issues can be grouped into  three categories: (1) multiple name  variations that signify the same author; (2)  similar or homonymic names that belong to  more than one author, and (3) linear changes  when an author alters his/her name,  generally due to changes in marital status or  other religious or legal reasons.  terminology is not standardized, even  within research teams, and varies whether  researchers are discussing the state of pre‐ processed records or the process applied to  organize the records. the following terms  may be used outside of the library science  discipline to indicate research into authority  control issues. lee et al. (69) define mixed  citations as authors with similar/homophonic  names grouped or mixed together and split  citations when one author generates name  variations; while hong, on, and lee (137)  define split as the process of separating  multiple authors with similar names and  merge as the process of merging one author’s  name variations into one cluster. malin,  evidence based library and information practice 2006, 1:1  48 airoldi, and carley (120) use variation to  indicate one author with many names and  ambiguity to indicate similar names/many  authors. niu, li, and srihari (sec. 1) define  alias association as the process of managing  one author with many names and  disambiguation as the process of tackling  similar names that indicate many authors.  linear name changes generate less attention,  probably because the other categories seem  more readily solvable without human effort.    the torvik team is developing “several  planned steps toward our long‐term goal of  completely partitioning medline into  unique authors.” (157). their model  examines mesh headings, title words,  journal names, and coauthors to estimate the  probability that a pair of author names  refers to the same individual. from this  model, the team developed a name  disambiguation tool for the medline  database. author‐ity  <http://arrowsmith.psych.uic.edu> provides  “a list of articles ranked by decreasing  probability that the author name [searched]  given on the article [selected] refers to the  same individual.”    the teams led by han are testing various  models of machine learning against the  dblp computer science bibliography data  (han et al. “mining and disambiguating”;  han et al. “information access”; han et al.  “name disambiguating”). the models use  data from co‐author names, keywords in  paper titles, and source titles in addition to  the solo author name. the various models  all point to similar ways to add data to  enhance disambiguation. the number of  features included and the weight assigned  to these features can improve  disambiguation performance (han et al.  “name disambiguating” 338). authors with  both similar names and similar research  interests pose greater challenges for  successful disambiguation. since the  keywords present in article and source titles  may be sparse, using word clustering  techniques to group research areas (such as  reference or cataloguing) may enhance  disambiguation. the team could also  consider including the author‐supplied  keywords where present.    malin, airoldi, and carley (136) and mann  and yarowsky (2) argue for the use of social  networks to assist in disambiguation.  social  networks provide context surrounding a  name, similar to the manner in which  coauthors and keywords provide a context  for distinguishing among authors.  researchers acknowledge the depth of the  problem when a manual examination of the  data is insufficient for determining whether  a name belongs to one or two individuals  (bekkerman and mccallum 469; fleischman  and hovy conclusion).  these projects do  not focus on the narrower problem of  disambiguating names when all are known  to be authors and where the metadata reside  in tagged author fields, but techniques  resulting from these projects may apply to  structured bibliographic databases.    the pasula team admit  “[w]e do not  currently model the fact that erroneous  citations are often copied from reference list  to reference list …” (8) indicating a rare  acknowledgement of the copying problem  and perhaps a promise to include the  chaining of error‐filled citations in future  models.    the selected disambiguation projects  described above, share similar attributes. all  use metadata beyond the author name alone.  most have proven that adding more data  elements to their models can serve to  disambiguate names in a faster manner and  with a higher probability of success than in  relying on single author names alone. all  models are tested on databases of limited  subject scopes (music, medicine, computer  science, economics) and thus each group of  researchers is uncovering similar successes  http://arrowsmith.psych.uic.edu evidence based library and information practice 2006, 1:1  49 and challenges. none have yet tested their  models on data from multidisciplinary or  extremely large databases.  merging the  techniques of adding data elements and  relying on disciplines to maintain their own  linked name files may result in long term  success for large, multidisciplinary  databases such as i&a databases.     conclusion    most i&a databases place the burden on the  searcher to identify and select author name  variations. the wilsonweb databases impose  authority control by altering author names,  but this practice causes the index entries to  fail to match the name on the article.  maintaining an authority file to manage  name variations, such as the mathscinet  approach, is an effective service for the  searcher but is not likely to scale well for  larger databases. alternative solutions must  be implemented to assure access, retrieval,  and proper crediting of authors’ works.  without control or linkage to name  variations, searchers may retrieve  incomplete or inaccurate results.    the traditional objective of name authority  files is to determine precisely when name  variations belong to the same individual.   manually‐maintained authority files have  served library catalogues reasonably well,  but the burden of upkeep has made them ill‐ suited to managing the volume of items and  authors in all but the smallest i&a databases.  to meet the access needs of the 21st century,  both catalogues and i&a databases may  need to implement options that present a  high degree of probability that items have  been authored by the same individual,  rather than options that provide high  precision with the expense of manual  maintenance.  striving for name  disambiguation rather than name authority  control may become an attractive option for  catalogues, i&a databases, and digital  library collections.  i&a databases may soon have many  automated options for facilitating name  disambiguation. we encourage i&a  database producers to examine and  implement options researched by the digital  library community. developing automated  methods can reduce the searcher’s burden of  determining author name variations while  ensuring that the author index entries match  the names on the article and that the end‐ user can successfully retrieve all of an  author’s works from that database.    works cited    “about library of congress authorities.”  library of congress authorities help  pages. washington, dc: library of  congress. 1 december 2005.  <http://authorities.loc.gov/help/contents. htm>.    about name authority control in h.w.  wilson’s indexing services.  new york:  h.w. wilson, 2005. 1 december 2005.  <http://www.hwwilson.com/databases/ names_authority_control.htm>.    asist digital library.  new york: wiley  interscience. 1 december 2005.  <http://www3.interscience.wiley.com/cg i‐bin/jtoc/27981/>.     “author names.” web of science help.  philadelphia: thomson corporation,  2005. 1 december 2005.  <http://wos17.isiknowledge.com/searcha id/help/h_author_names.htm>.    author‐ity. arrowsmith project home page.  university of illinois at chicago. 1  december 2005.  <http://arrowsmith.psych.uic.edu>.    bekkerman, ron, and andrew mccallum.  “disambiguating web appearances of  people in a social network.”  proceedings of the 14th international  http://authorities.loc.gov/help/contents http://www.hwwilson.com/databases/ http://www3.interscience.wiley.com/cg http://wos17.isiknowledge.com/searcha http://arrowsmith.psych.uic.edu evidence based library and information practice 2006, 1:1  50 conference on world wide web. new  york: acm press, 2004. 463‐470. 10  february 2006. <http://portal.acm.org/>.    bhattacharya, indrajit, and lise getoor.  “iterative record linkage for cleaning  and integration.”  proceedings of the 9th  acm sigmod workshop on research  issues in data mining and knowledge  discovery. ed. gautam das, bing liu,  and philip s. yu. new york: acm press,  2004. 11‐18. 1 december 2005.  <http://portal.acm.org/>.    cardina, christen, and donald wicks. “the  changing roles of academic reference  librarians over a ten‐year period.”  reference & user services quarterly 44  (winter 2004): 133‐142.    cited reference searching: an introduction.  a tutorial using web of science.  philadelphia: thomson corporation,  2004. 1 december 2005.  <http://scientific.thomson.com/tutorials/ citedreference/crs3.htm>.    crossref. lynnfield, ma: publishers  international linking association inc.  (pila), 2003. 1 december 2005.  <http://www.crossref.org/>    cruz, josé manuel barrueco, markus j.r.  klink, and thomas krichel. “personal  data in a large digital library.”   research and advanced technology for  digital libraries: 4th european  conference, ecdl 2000, lisbon,  portugal, september 2000. lecture notes  in computer science. 1923. ed. jose  borbinha and thomas baker. berlin:  springer, 2000. 127‐134. 1 december  2005.  <http://openlib.org/home/krichel/phoeni x.letter.pdf>.    dblp computer science bibliography.   trier, germany: universitat trier. 1  december 2005.  <http://www.informatik.uni‐ trier.de/~ley/db/> and mirrored at  <http://www.sigmod.org/dblp/db/index. html>.    delsey, tom. “authority records in a  networked environment.” international  cataloguing and bibliographic control  33.4 (october/december 2004): 71‐74.    dilauro, tim, g. sayeed choudhury, mark  patton, james w. warner, and elizabeth  w. brown. “automated name  authority control and enhanced  searching in the levy collection.” d‐lib  magazine 7.4 (2001).  1 december 2005.  <http://www.dlib.org/dlib/april01/dilaur o/04dilauro.html>.    fellegi, ivan p., and alan b. sunter. “a  theory for record linkage.” journal of  the american statistical association  64.328 (december 1969):1183‐1210.    fleischman, michael ben, and eduard hovy.  “multi‐document person name  resolution.” 10 february 2006.  <http://www.mit.edu/~mbf/acl_04.pdf >    garfield, eugene. “a suggestion for  improving the information content of  authors’ names.”  current contents 6  (feb 11, 1970). 1 december 2005.  <http://www.garfield.library.upenn.edu /essays/v1p077y1962‐73.pdf>.    getty research institute. getty union list of  artists names.  los angeles: j. paul  getty trust. 1 december 2005.  <http://www.getty.edu/research/conduc ting_research/vocabularies/ulan/>.    han, hui, lee giles, hongyuan zha, cheng  li, and kostas tsioutsioulikis.   “supervised learning approaches for  name disambiguation in author  http://portal.acm.org/ http://portal.acm.org/ http://scientific.thomson.com/tutorials/ http://www.crossref.org/ http://openlib.org/home/krichel/phoeni http://www.informatik.uni%e2%80%90trier%00 http://www.sigmod.org/dblp/db/index http://www.dlib.org/dlib/april01/dilaur http://www.mit.edu/~mbf/acl_04.pdf http://www.garfield.library.upenn.edu http://www.getty.edu/research/conduc evidence based library and information practice 2006, 1:1  51 citations.” jcdl 2004 :  proceedings of  the fourth acm/ieee joint conference  on digital libraries : global reach and  diverse impact : tucson, arizona, june  7‐11, 2004.  ed. hsinchun chen, michael  christel, and ee‐peng lim. new york:  acm press, 2004. 296‐305. 1 december  2005. <http://portal.acm.org/>.    han, hui, wei xu, hongyuan zha, and c.  lee giles. “a hierarchical naive bayes  mixture model for name  disambiguation in author citations.”  proceedings of the 2005 acm  symposium on applied computing. ed.  lorie m. liebrock. new york: acm  press, 2005. 1065‐1069. 1 december 2005.  <http://portal.acm.org/>.    han, hui, hongyuan zha, c. lee giles.  “name disambiguation in author  citations using a k‐way spectral  clustering method.” proceedings of the  5th acm/ieee‐cs joint conference on  digital libraries: denver, june 7‐11,  2005. new york: acm press, 2005. 334‐ 343. 1 december 2005.  <http://portal.acm.org/>.    hong, yoojin, byung‐won on, and  dongwon lee.  “system support for  name authority control problem in  digital libraries: opendblp approach.”  research and advanced technology for  digital libraries: 8th european  conference, ecdl 2004. lecture notes  in computer science 3232. ed. rachel  heery and elizabeth lyon. berlin:  springer, 2004. 134‐144.    ifla ubcim working group on functional  requirements and numbering of  authority records (franar).  functional requirements for authority  records: a conceptual model . draft  2005‐06‐15. 1 december 2005.  <http://www.ifla.org/vii/d4/franar‐ conceptual‐m‐draft‐e.pdf>.  kuhagen, judith a. “standards for name  and series authority records.”  cataloging & classification quarterly   21.3‐4 (1996): 131‐54.    lam, ki‐tat. “xml and global name  access control.” oclc systems &  services 18.2 (2002): 88‐96.    lancaster, frederick wilfrid. vocabulary  control for information retrieval. 2nd ed.  arlington: information resources press,  1986.     lee, dongwon, byung‐won on, jaewoo  kang, and sanghyun park. “effective  and scalable solutions for mixed and  split citation problems in digital  libraries.” proceedings of the 2nd  international workshop on information  quality in information systems, iqis  2005. baltimore, june 17, 2005. new  york: acm press, 2005. 69‐76. 1  december 2005. <http://portal.acm.org/>.    library literature & information science  full text.  new york: h.w. wilson, 2005.  1 december 2005.  <http://www.hwwilson.com/databases/l iblit.htm>.      library of congress authorities help pages.   washington, dc: library of congress,  2005.  1 december 2005.  <http://authorities.loc.gov/help/contents. htm>.    malin, bradley, edoardo airoldi, and  kathleen m. carley. “a network  analysis model for disambiguation of  names in lists.” computational &  mathematical organization theory 11.2  (2005): 119‐139.    mann, gideon s., and david yarowsky.  “unsupervised personal name  disambiguation.” proceedings of the 7th  conference on natural language  http://portal.acm.org/ http://portal.acm.org/ http://portal.acm.org/ http://www.ifla.org/vii/d4/franar%e2%80%90conceptual%e2%80%90 http://portal.acm.org/ http://www.hwwilson.com/databases/l http://authorities.loc.gov/help/contents evidence based library and information practice 2006, 1:1  52 learning. edmonton, canada, may 31‐ june 1, 2003. ed. walter daelemans and  miles osborne. 10 february 2006.  <http://www.cs.jhu.edu/~gsm/publicatio ns/pd.gsm03.pdf>.    mathscinet. providence: american  mathematical society, 2005. 1 december  2005.  <http://www.ams.org/mathscinet/>     mathscinet author database help.  providence: american mathematical  society, 2005. 1 december 2005.  <http://www.ams.org/msnhtml/authid_ help.html>    monastersky, richard. “the number that’s  devouring science.” chronicle of  higher education. 52.8 (2005): a12  (october 14). 1 december 2005.  <http://chronicle.com/weekly/v52/i08/08 a01201.htm>.    naco ‐ the name authority component  of the pcc. washington, dc: library of  congress, 2005. 1 december 2005.  <http://www.loc.gov/catdir/pcc/naco/na copara.html>.    niu, cheng, wei li, and rohini k. srihari.  “weakly supervised learning for cross‐ document person name  disambiguation supported by  information extraction.” proceedings of  the 42nd annual meeting of the  association for computational  linguistics acl 2004, barcelona, spain,  july 2004. 598‐605. 1 december 2005.  <http://acl.ldc.upenn.edu/acl2004/main/ pdf/372_pdf_2‐col.pdf>.    on, byung‐won, dongwon lee, jaewoo  kang, and prasenjit mitra.  “comparative study of name  disambiguation problem using a  scalable blocking‐based framework.”  jcdl 2005 : proceedings of the fifth  acm/ieee joint conference on digital  libraries: denver, colorado, june 7‐11,  2005. new york: acm press, 2005. 344‐ 353. 1 december 2005.  <http://portal.acm.org/>.    pasula, hanna, bhaskara marthi, brian  milch, stuart russell, and ilya shpister.  “identity uncertainty and citation  matching.” advances in neural  information processing systems 15. san  mateo, ca : m. kaufmann publishers,  2003: 1 december 2005.   <http://books.nips.cc/papers/files/nips15 /ap01.pdf>.    patton, glenn e.  “extending frbr to  authorities.”  cataloging &  classification quarterly 39.3/4 (2005):  39‐48.    patton, mark, david reynolds, g. sayeed  choudhury, and dilauro, tim.  “toward a metadata generation  framework: a case study at johns  hopkins university.” d‐lib magazine  10.11 (2004): 1 december 2005.  <http://www.dlib.org/dlib/november04/ choudhury/11choudhury.html>.    repec author service. storrs, ct:  university of connecticut, department  of economics. 1 december 2005.  <http://authors.repec.org/>.    snyman, marieta m. m., and marietjie  jansen van rensburg. “naco versus  isan: prospects for name authority  control.” the electronic library 18.1  (2000): 63‐68.    ———. “reengineering name authority  control.” the electronic library 17.5  (october 1999): 313‐322.    spink, amanda, and maurice c.  leatherbury. “name authority files  and humanities database searching.”  http://www.cs.jhu.edu/~gsm/publicatio http://www.ams.org/mathscinet/ http://www.ams.org/msnhtml/authid_ http://chronicle.com/weekly/v52/i08/08 http://www.loc.gov/catdir/pcc/naco/na http://acl.ldc.upenn.edu/acl2004/main/ http://portal.acm.org/ http://books.nips.cc/papers/files/nips15 http://www.dlib.org/dlib/november04/ http://authors.repec.org/ evidence based library and information practice 2006, 1:1  53 online & cdrom review 18 (june  1994): 143‐148.     taylor, arlene g. “variations in personal  name access points in oclc  bibliographic records.” library  resources & technical services 36  (april 1992): 224‐241.     tepaske‐king, bert, and norman richert.  “the identification of authors in the  mathematical reviews database.”  issues in science & technology  librarianship 31 (summer 2001). 1  december 2005. <http://www.istl.org/01‐ summer/databases.html>.    tillett, barbara b. “a virtual international  authority file.” 67th ifla council and  general conference, august 16‐25, 2001,  boston.  the hague: international  federation of library associations and  institutions, 2001. 1 december 2005.  <http://www.ifla.org/iv/ifla67/papers/09 4‐152ae.pdf>.    ———. “aacr2 and metadata: library  opportunities in the global semantic  web.” cataloging & classification  quarterly 36.3/4 (2003): 101‐119.    ———. “authority control: state of the art  and new perspectives.” cataloging &  classification quarterly 38.3/4 (2004):  23‐41.    torvik , vetle i., marc weeber , don r.  swanson , neil r. smalheiser. “a  probabilistic similarity metric for  medline records: a model for author  name disambiguation.”  journal of the  american society for information  science and technology  56.2 (2005):  140‐158.    uniquely identifying mathematical authors  in the mathematical reviews database.   providence: american mathematical  society, 2005. 1 december 2005.  <http://www.ams.org/mr‐database/mr‐ authors.html>.    warner, james w., and elizabeth w. brown.  “automated name authority control.”  proceedings of the 1st acm/ieee‐cs  joint conference on digital libraries   jcdl ’01, june 24‐28, roanoke, va.  new york: acm press, 2001. 21‐22. 1  december 2005. <http://portal.acm.org/>.    web of science. philadelphia: thomson  corporation, 2004. 1 december 2005.  <http://scientific.thomson.com/products/ wos/>.    web of science 7.0 workshop. philadelphia:  thomson corporation, 2004. 1  december 2005.  <http://www.thomsonscientific.com/me dia/scpdf/wos7‐workshop_en.pdf>.     weber, jutta. “leaf: linking and exploring  authority files.”  cataloging &  classification quarterly 38.3/4 (2004):  227‐236.    wellisch, hans h. indexing from a to z.   2nd ed. new york: h. w. wilson, 1995.  http://www.istl.org/01%e2%80%90 http://www.ifla.org/iv/ifla67/papers/09 http://www.ams.org/mr%e2%80%90database/mr%e2%80%90authors%00 http://portal.acm.org/ http://scientific.thomson.com/products/ http://www.thomsonscientific.com/me evidence based library and information practice 2006, 1:1  54 appendix a: sample of variations in instructions to authors for formatting names in the  “references” section of submissions    journal of academic librarianship (published by elsevier): jal follows the 15th edition of the  manual of style, published by the university of chicago press.  examples: article from a journal:  paul metz, …  guide for authors. 1 december 2005,  http://www.elsevier.com/wps/find/journaldescription.cws_home/620207/authorinstructions    information processing & management (published by elsevier): you are referred to the publication  manual of the american psychological association, fifth edition … examples: fox, e. a. &  marchionini, g. …  guide for authors. 1 december 2005,  http://authors.elsevier.com/guideforauthors.html?pubid=244&dc=gfa    reference services review (published by mcb press/emerald): references to other publications  should be complete and in harvard style.  (c) for articles: surname, initials,… e.g.fox, s….  author guidelines. 1 december 2005,  http://www.emeraldinsight.com/info/journals/rsr/notes.htm      http://www.elsevier.com/wps/find/journaldescription.cws_home/620207/authorinstructions http://authors.elsevier.com/guideforauthors.html?pubid=244&dc=gfa http://www.emeraldinsight.com/info/journals/rsr/notes.htm evidence based library and information practice 2006, 1:1  55 appendix b:  the presence/absence of name authority control in databases     chart 1 includes several large library‐subscribed and library‐managed databases.  chart 2  describes databases that are not managed in traditional library environments.    chart 1: traditional library‐ based databases  is name  authority  control used?  how is control applied, or how must searcher identify  and select name variations?  abi/inform  no  identify and select from author index  acm digital library  no  no author index  chemical abstracts  no  identify and select from author index  compendex  no  identify and select from author index  econlit  no  identify and select from author index  eric  no  no author index  georef  no  identify and select from author index  library literature &  information science  yes, but  inconsistent  changes some author names to current version  inspec  no  identify and select from author index  mathscinet  yes   mostly‐automated name authority file  paisinternational  no  identify and select from author index  psycinfo  no  identify and select from author index  pubmed  no  truncates to initials, except post‐2002 if full name on  article  web of science  no  identify and select from author index; initials only used      chart 2: non‐library‐ managed databases  is name  authority  control used?  how is control applied, or how must searcher identify  and select name variations?  anac (levy project)  yes  automated name authority file  arxiv.org  no  no author index  author‐ity  no  disambiguation based on probability  citeseer  no  no author index  dblp bibliography  no  select from author index  getty union list of artist  names  yes  links among separate authority files  google scholar  no  no author index  hopec  yes  author‐maintained registry  leaf  yes  links among separate authority files    evidence based library and information practice 2006, 1:1  56 illustrated below is a typical view of an author index that includes name variations. a searcher  might select all the “cl” variations, but no searcher would know to scroll through to “lee”  without having noticed or known that dr. giles emphasizes his middle name.    giles, c. l.  giles, c. lee  giles, c. o.  giles, c. r.  giles, c. randy  giles, c.a.  giles, c.g.  giles, c.h.  giles, c.l.  giles, c.lee  [skip 144 lines].  giles, l.j.  giles, lee                evidence based library and information practice 2006, 1:1  57 appendix c ‐ wilsonweb     http://www.hwwilson.com/databases/names_authority_control.htm  about name authority control in h.w. wilson’s indexing services    h.w. wilson controls names used as subjects. no user should have to search under multiple  forms of a name. personal names are cited consistently across all the wilson indexes and  databases.     names are established according to the latest revision of aacr2, so h.w. wilson names are  consistent with conventional library cataloging. (the names department staff—who are  responsible for maintaining the wilson names authority file—are all professional librarians.)     new names are routinely checked against the library of congressʹs lcweb names authority file,  to ensure consistency with national cataloging standards. chances are, names will be cited in  h.w. wilson files the same way as they appear in a libraryʹs own online catalog, if they are  indeed the same person.     all personal name subjects are carefully checked against the individual periodical databases,  including retrospective files, to avoid duplication and to distinguish between similar but  different instances of names.  similar but distinct names are distinguished from one another by expansion (e.g. inclusion of a  full name instead of initials) or the addition of dates.     in cases where the form of a name is uncertain, h.w. wilson names authority staff will search  for an authoritative form in appropriate dictionaries, encyclopedias, and directories. the specific  sources depend on the discipline, and on the dates and nationalities of the person in question.     h.w. wilson names authority staff routinely establish cross‐references from variant forms of a  name to the form we cite. wilsonweb users will be automatically switched from variants to  preferred forms of names.      accessed 1 december 2005.  copyright © 2006 by the h. w. wilson company. material reproduced with permission of the publisher. permission  granted 2/13/2006.  http://www.hwwilson.com/databases/names_authority_control.htm evidence based library and information practice evidence based library and information practice 2011, 6.2 69 evidence based library and information practice evidence summary field experiences provide value for library students, but more research is needed into their administration a review of: malik, a., & ameen, k. (2010). effectiveness of library practicum: perceptions of lis graduates in pakistan. library review, 59(8), 573-584. reviewed by: ann medaille reference & instruction librarian university of nevada, reno reno, nevada, united states of america e-mail: amedaille@unr.edu received: 23 feb. 2011 accepted: 21 apr. 2011 2011 medaille. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to evaluate an eight-week practicum requirement for library and information science students. design – survey questionnaire. setting – the university of the punjab and participating libraries in pakistan. subjects – a total of 118 graduates of the library and information science program at the university of the punjab who had completed an eight-week-long practicum experience in a public, academic, or special library between 2002 and 2006. methods – snowball sampling was used to select the subjects. surveys were delivered and collected through personal visits, email, and postal mail, and the authors asked follow-up questions of some subjects whose surveys were collected in person. the survey contained questions about several topics, such as the type of professional tasks (e.g., acquisition, circulation, reference service, etc.) that students did during the practicum, professional skills developed, practicum length, comfort in sharing problems with supervisors, and the evaluation process. the authors used statistical analysis software (spss, version 12) to analyze the survey results. mailto:amedaille@unr.edu� evidence based library and information practice 2011, 6.2 70 main results – the majority of subjects reported that they had received the opportunity to train in several sections and functions of the library, and they felt that the practicum was important in improving their professional skills. most reported being comfortable sharing problems with their supervisors and were satisfied with the evaluation process. the survey also included open-ended questions about problems and suggestions, but only 28% of the subjects responded with comments. conclusion – the authors concluded that the library practicum plays an important role in the professional development of future library professionals, and that the length of the practicum should be extended from its current eight-week requirement. commentary within the library profession, there has been much discussion about how best to prepare library students for work in 21st century libraries. some critics of traditional library school curricula assert that too much emphasis is placed on theory and not enough time is devoted to practice (ball, 2008; coleman, 1989). but theory and practice are not necessarily antithetical; in fact, practical experience can help to reinforce concepts taught in the classroom (coleman, 1989) while providing students with professional skills that are highly valued by library employers (orme, 2008). however, high-quality field work experiences can be difficult for library schools to administer and may take time away from an already full curriculum of classes (ball, 2008; grotzinger, 1971; coleman, 1989). because many studies of field work experiences are several decades old, new studies are needed to determine the value of these programs for the training of new librarians (ball, 2008). the present study contributes to the literature by explaining how the practicum experience is conducted at one institution and by raising questions about how these programs should best be administered to ensure that students participate in highquality field experiences. for the study design, the authors surveyed 118 out of 367 graduates and asked in-person questions of some of the subjects. unfortunately, the survey instrument is not included in the article, so the reader is unable to see how the questions were worded and understand how the results were interpreted. the authors also make no mention of providing anonymity to the subjects. this is of some concern, especially since only 28% of the subjects were willing to respond to openended questions. this failure to guarantee confidentiality may have compromised the validity of the results. the authors provide feedback from subjects whose responses indicated areas that were lacking in the practicum. for example, some subjects were not provided with opportunities to train in all areas of the library, such as circulation or computer technologies, while others did not gain experience with important professional tasks, such as work with audiovisuals, reference service, or written communication. while the feedback described above is of value, some of the authors’ conclusions do not follow logically from the data. for example, the authors asked whether the practicum experience met with expectations, and all subjects responded that it did, at least to some extent. the authors conclude from this that the majority of the subjects had a satisfactory practicum experience, but their conclusion does not consider that some subjects may have expected a negative experience. often the data analysis seems incomplete. in a question about a journal-writing activity, for example, the authors reveal that some students responded with “we copy from others.” the authors conclude that more efforts are needed to make journal writing a worthwhile exercise. since this type of selfreflection activity can be an effective means of bridging the gap between theory and practice, a more thoughtful analysis of the value of this activity could have been instructive. evidence based library and information practice 2011, 6.2 71 the main problem with this article, however, is its lack of a clear research question. the authors state that their objective is to evaluate their program; however, they would have been better served by more clearly and specifically relating library field experiences to a question that contributes to a more general understanding of the issues. the authors conclude definitively that field experiences provide value to library students, but the literature already supports this assertion. in addition, the authors wish to derive suggestions for improving the practicum experience; however, the only suggestion they make is that the current practicum requirement be lengthened from eight weeks. despite the problems with the study design, questions surrounding the role of field work in educating new library professionals remain important. thus, this article contributes to discussions about the best ways to administer field work experiences so as to ensure that library students receive practical training that adequately prepares them for careers in a rapidly changing profession. references ball, m. a. (2008). practicums and service learning in lis education. journal of education for library and information science, 49(1), 70-82. coleman, jr., j. g. (1989). the role of the practicum in library schools. journal of education for library and information science, 30(1), 19-27. grotzinger, l. (1971). the status of “practicum” in graduate library schools. journal of education for librarianship, 11(4), 332-339. orme, v. (2008). you will be . . . : a study of job advertisements to determine employers’ requirements for lis professionals in the uk in 2007. library review, 57(8), 619-633. / evidence based library and information practice evidence summary   preference for print or electronic book depends on user’s purpose for consulting   a review of: rod-welch, l.j., weeg, b.e., caswell, j.v., & kessler, t.l. (2013). relative preferences for paper and for electronic books: implications for reference services, library instruction, and collection management. internet reference services quarterly, 18(3-4), 281-303. doi: 10.1080/10875301.2013.840713   reviewed by: laura newton miller assessment librarian carleton university library ottawa, ontario, canada email: laura.newtonmiller@carleton.ca   received: 21 may 2014  accepted: 23 jul. 2014      2014 newton miller. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine patron format preference, perceived usability and frequency of e-book usage, and to study use and preference of e-reading devices.   design – survey questionnaire.   setting – large public research university in the united states of america.   subjects – 339 students, faculty, and staff members   methods – an anonymous 23-item survey was available in online and print formats. print surveys were distributed in the lobby of the library and throughout various buildings on campus. a direct link to the online version of the survey was included in e-newsletters, on the library homepage, and on the library’s facebook site. a definition of e-book was placed prominently at the beginning of the survey. questions included information on preference of format (11), experiences using e-books (3), ownership of particular devices for reading e-books (1), attitudes regarding library purchase of e-books and readers (3), demographic information (4), and additional comments (1).   main results – of the 339 completed surveys, 79 were completed online and 260 in print. when asked about preference in format for reading, 79.6% of respondents preferred print books compared to 20.4% choosing e-books. if the library was purchasing a book to support class research and projects, 53.9% preferred print and 46.1% preferred electronic, but if the library purchased a book for leisure reading, 76% preferred print and 24% preferred electronic. in response to the question about how often they used e-books from the library, 50.1% of respondents never used library e-books, 21.1% used once per year, 20.8% monthly, 7.4% weekly and 0.6% daily. of those who used e-books, 38.1% read only sections they needed, 31% searched keywords, 24.2% downloaded and printed pages to read later, 21.8% read the most relevant chapters, 17.1% skimmed the entire book and 14.2% read the entire book. if both formats were available, 25.1% felt that the library should purchase the print book, 16.7% the e-book, and 58.2% chose both formats. when asked about downloading e-books, 51.1% of respondents would use an e-book only if they could download it to a hand-held device. a majority of the respondents, 81.7%, felt that the library should provide e-readers for checkout if the library purchased e-books instead of print books. when asked which types of books they preferred to read in electronic format in an open-ended question, 22% preferred textbooks, 21% leisure reading, 18% research books, 15% other types, 6% journals, 5% reference books, and 3% anything. regarding which types of books were preferred in print format, 42% preferred leisure reading, 21% other, 14% all, 11% textbooks, 6% research books, 2% no e-books, 2%  journals and 2% reference books.   conclusion – preference for book format (electronic or print) depends on the users’ purpose for reading the text. this will likely change over time, as users gain more familiarity and experience with e-books, and better support is provided from the library.     commentary although e-journals have become popular with most university users, the adoption of e-books in the academic setting has been slower. a variety of literature discusses issues with platform, e-reader downloading, lack of uniformity in licencing, and other time-consuming access issues. this paper looks at attitudes toward e-books and how patrons use e-books in an academic setting.   the ebl critical appraisal checklist (glynn, 2006) was used to determine strengths and weaknesses of the study. data collection and research methodology were clearly described and the use of a survey was appropriate for learning more about book format preferences. it would have been preferable for a copy of the survey to be available as an appendix for others to refer to for their own research. however, the researchers referred to the questions throughout the article, giving one a good sense of the questions asked.   although demographic information was attained during the study, the article does not refer to any particular information about the users. this was a strong paper that explored some limitations to the study (such as confusion related to the definition of an e-book), however, the lack of demographic information makes this reader question the findings. were the respondents mostly faculty? undergraduates? graduates? staff? what was the age range of respondents and was there a difference in answers depending on the age of the respondent?   information regarding how respondents use an e-book (read or skim sections, read or skim entire book, or download and print some pages for further reading) was very interesting and useful for librarians who may be trying to convince others that reading in electronic format is not entirely different from reading in print. recommendations related to library instruction for both students and faculty are appreciated. reference staff have heard students say, “i need a source but my professor said it can’t be from the internet,” not realizing an e-book is an appropriate resource for a research paper. respondents slightly preferred print (53.1%) to electronic (46.1%) books that support class research projects, which corresponds to answers in the open-ended question, where a total of 45% indicated they preferred e-books for textbooks, research books, or reference books. what are the implications for collection development strategies when there is a major divide between those who prefer print versus electronic, and the fact that if both formats were available, 58.2% would prefer the purchase of both? e-book use is still in the early stages of acceptance in the university setting, but with space constraints and evolving budget issues, e-books are not going away. acceptance of e-books will continue to be a major challenge for librarians. this divide is in sharp contrast to the journal world, where the e-format has been adopted very successfully over the past decade. this paper shows different perspectives of format preference, and taken with other papers about e-book concerns, can help librarians make more informed choices about e-book purchasing for their particular library.   reference   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi: 10.1108/07378830610692154   evidence based library and information practice evidence based library and information practice 2011, 6.1 53 evidence based library and information practice evidence summary decline in reference transactions with few questions referred to librarian when the reference desk is staffed by a paraprofessional a review of: dinkins, d., & ryan, s. m. (2010). measuring referrals: the use of paraprofessionals at the reference desk. the journal of academic librarianship, 36(4), 279-286. reviewed by: diana k. wakimoto online literacy librarian, california state university, east bay doctoral student, san jose-qut gateway program hayward, california, united states of america email: diana.wakimoto@csueastbay.edu received: 5 nov. 2010 accepted: 17 dec. 2010 2011 wakimoto. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective — to determine the type and percentage of questions referred to a librarian by a paraprofessional (i.e., an individual without an mlis) staffing the reference desk, whether the percentage of referrals would decrease over time, and any consequences from having a paraprofessional rather than a librarian staffing the desk. design — quantitative analysis of reference desk transaction statistics. setting — reference desk at the main library of stetson university, a private university in the united states of america with approximately 2,500 fte (full-time equivalent) students. subjects — a total of 486 reference desk transactions recorded by a paraprofessional staffing the reference desk during the fall and spring semesters of the 2008 -2009 academic year. methods — the first year that he worked in the library at stetson university, a paraprofessional recorded all reference desk transactions during his shift from 10:00am to 12:00pm, four days a week, for the fall and spring semesters of the 2008 -2009 academic year. this paraprofessional, with computer expertise, received "relatively minimal" (p. 281) training on "reference desk policies and procedures… the use of the catalogue and the subscription databases" (p. 281). for each transaction, the paraprofessional categorized the question as "direction, " "reference, " or mailto:diana.wakimoto@csueastbay.edu� evidence based library and information practice 2011, 6.1 54 "machine. " he was instructed to contact a librarian if he could not answer a reference question. the paraprofessional also completed a questionnaire regarding his level of comfort answering questions and his thoughts on the training at the end of his first year of staffing the reference desk. main results — in the fall semester, 9.5% of all reference desk transactions were referred to a librarian. this decreased to 4.2% of the total transactions during the spring semester. the percentage of reference questions referred to a librarian in the fall semester was 21.9% and only 5.0% in the spring semester. there was a 49.5% decrease in the number of reference desk transactions during the paraprofessional's hours on the desk compared to the previous year when the desk was staffed by professional librarians. overall, reference desk transactions for all hours decreased 4.1% compared to the previous year. the results from the questionnaire on his experiences at the reference desk showed that the paraprofessional was satisfied with his training, comfortable with referring questions to the librarians, did not use the print reference collection extensively, thought the "interface for searching the library's catalogue/databases is dated at best" (p. 285), and felt that being close in age to many of the students was a disadvantage while working at the reference desk. conclusion — the authors concluded that staffing the reference desk with a paraprofessional was a success and that the "referrals to librarians had been made appropriately and when necessary" (p. 285). the results corroborated previous studies that showed only a "small percentage of reference desk transactions would need to be referred to a librarian" (p. 285). in part, because of the success of staffing the desk with a paraprofessional, the authors suggest that reference desk staffing configurations at academic libraries should be reevaluated. librarians freed from duties at the physical reference desk could use this time to "develop virtual reference services" and expand information literacy programs (p. 286). to explain the decrease in number of transactions during the paraprofessional's time on the reference desk, the authors surmised four possibilities: patrons' reluctance to ask questions of someone new on the desk, their dissatisfaction with the paraprofessional's answers, the similarity in age between the paraprofessional and the "age of the student population" (p. 284), or the librarians being more conscientious in tallying every transaction. however, the authors doubted that users perceived the paraprofessional's answers as "less satisfactory" as "patrons likely got a higher level of service on computer-related queries from the computer science-trained paraprofessional" (p. 284). computer-related queries, coded as “machine” transactions, formed the majority of queries answered by the paraprofessional. commentary this study extends current research on alternative reference desk staffing arrangements. it suggests that paraprofessionals with minimal training can handle the majority of questions asked, including reference questions. this study will be of interest to those considering implementing or expanding the staffing of the reference desk with paraprofessionals; however, caution should be used when applying this study's results as they are based on the experience of only one paraprofessional and therefore may not be generalizable to other situations. the authors provided context for their study with a thorough overview of previous studies on reference desk staffing and they explained their methodology clearly. however, a more detailed description of the training provided to the paraprofessional after he was hired and details on when the authors informed the paraprofessional about this research study would have strengthened the article. it would have allayed questions of whether the paraprofessional knew about the study before he was hired or whether he felt pressured to evidence based library and information practice 2011, 6.1 55 participate in the study as it was his first year working for the library. it is also unclear why the questionnaire was included in the article as the questions do not directly map to the research questions as stated. while the descriptive statistics, as shown in the tables and graphs, are easily understood, the inclusion of inferential statistics to clarify the level of statistical significance for the changes seen in the number of referrals and the overall decrease in number of questions would have been appreciated. there is also the more fundamental question of how the librarians know that the referrals were made "appropriately and when necessary" (p. 285). another major unanswered question that needs to be explored further is what, if any, connection is there between a paraprofessional staffing the reference desk and the precipitous decline in reference questions during his shifts as compared to previous years. future research could include qualitatively exploring the effects of staffing the desk with paraprofessionals to complement quantitative studies such as this one. the extension of the "one-year trial of using a paraprofessional at the desk" (p. 285) will serve to answer more fully the authors' question of whether the paraprofessional's gaining familiarity with the library and campus will decrease the number of referrals. with changes in the reference landscape, staffing decisions "should be realigned to best meet the needs of the library's users" (p. 286); studies such as this, when combined with other research, can assist library managers in this realignment. / evidence based library and information practice evidence summary evidence based library and information practice evidence based library and information practice 2010, 5.4 108 evidence based library and information practice commentary evidence-based scholarly communication: information professionals unlocking translational research philip j. kroth assistant professor director, biomedical informatics research, training and scholarship health sciences library and informatics center the university of new mexico albuquerque, new mexico, united states of america email: pkroth@salud.unm.edu holly e. phillips lecturer iii coordinator of resource access & delivery health sciences library and informatics center the university of new mexico albuquerque, new mexico, united states of america email: hphillips@salud.unm.edu jonathan d. eldredge associate professor health sciences library and informatics center the university of new mexico albuquerque, new mexico, united states of america email: jeldredge@salud.unm.edu received: 26 july 2010 accepted: 12 oct. 2010 2010 kroth, phillips and eldredge. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/bync-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. introduction the evidence-based scholarly communication conference (ebscc) was held march 11-12, 2010 in albuquerque, nm. the conference addressed the perceived gap in knowledge and training for scholarly communication principles in the national institutes of health (nih) clinical and translational science award (ctsa) program. the ebscc brought together librarians and information specialists to share evidence based strategies for developing effective local scholarly communication support and training and, it is hoped, to form new coalitions to address this topic at the local and national levels. this brief communication summarizes the need for the mailto:pkroth@salud.unm.edu� mailto:hphillips@salud.unm.edu� mailto:jeldredge@salud.unm.edu� evidence based library and information practice 2010, 5.4 109 conference, highlights the general sessions in order of presentation, and introduces the ebscc research papers appearing in this issue of evidence based library and information practice (eblip). it also includes a description of a unique peer-review process methodology pioneered at ebscc. background the united states’ national institutes of health (nih) clinical and translational sciences award (ctsa) program replaced, with the goal of improving upon, the thirty year-old general clinical research center program. the ctsa program seeks to increase the speed in which scientific discoveries are translated from the laboratory bench into healthcare practice in the community. the ctsa program intends to accomplish this goal by funding the establishment of multiple clinical and translational science centers that will, in turn, provide a wide range of support and resources designed specifically to support the training of and the work done by translational investigators. the ctsa in the united states resembles similar initiatives in other countries including japan (nakaya, shimizu, tanaka, & asano, 2005), australia (australian national university, 2010; glascow, 2010; national health and medical research council, 2010), the united kingdom (national health service, 2010), and costa rica (colón-ramos lindsay, monge-rojas, greaney, campos, & pearson, 2007). in canada, both the national research council canada and the canadian institutes of health research pursue translational sciences activities (wilson & neilson, 2010). translational science as a discipline embraces the motto “from bench to bedside to the community.” paradoxically, the nih ctsa program does not require established centers either to train or support translational investigators in the use of the increasing number of new venues for sharing and disseminating research results. these venues include institutional repositories, open access electronic journals, and electronic materials to supplement traditional publications. although, the nih does insert wording about mandatory compliance with its public access policy at the end of the funding opportunity announcement for the ctsa program, it does not include any mention of these new scholarly dissemination options (national center for research resources, 2010). in addition, the funding opportunity announcement does not include improved dissemination of research results in its list of suggested curricular elements such as biomedical informatics, research ethics, and biostatistics. nor is there mention of any scholarly communications skills training that might encompass copyright management, open access publication, or the use of institutional repositories. these scholarly communications skills are arguably critical if translational investigators are going to capitalize on the expanding number of novel scholarly communication venues to foster better translation of their scientific discoveries into practice and get the most value out of their scholarly work. the authors attempted to address this gap by coordinating a conference specifically designed to foster the exchange of ideas on how to best promote new methods in scholarly communication, specifically in translational research communities. we were involved in securing a ctsa grant for the university of new mexico’s clinical and translational sciences center, and were already teaching a biomedical informatics course that included open-access publishing options among other scholarly communication topics (kroth, phillips, & eldredge, 2009). from our experience in promoting local nih public access policy compliance and supporting our institution’s digital repository, we recognized the challenges inherent in engaging the community of translational investigators and the need for a national effort by information professionals to address this dissemination problem. two of the authors (pk, hp) attended the association of research libraries (arl) institute on scholarly communication in evidence based library and information practice 2010, 5.4 110 chicago in december of 2006 (association of research libraries, 2010). this outstanding institute was targeted toward a wide spectrum of academic domains including liberal arts, law, and engineering, in addition to the biomedical sciences. the two attendees felt the institute model could be adapted to the translational research investigator community, which had sufficiently unique needs to justify a special focus. another author (je) gave a presentation at the 2008 medical library association annual meeting in a section program devoted to serving institutions with ctsas. health sciences librarians attending this program responded enthusiastically to the reported efforts. this enthusiasm reinforced the authors’ belief in the need for a special conference. we applied for and received a nih national library of medicine conference grant to partially offset costs. forty-six librarians and other information professionals, representing 25 institutions around the us, attended the ebscc conference on march 11-12, 2010 in albuquerque, new mexico. the majority of the attendees were from ctsa funded institutions or institutions currently applying for a ctsa. a complete copy of the conference’s notebook that includes a detailed agenda and speaker biographies is freely available. (kroth, bowler-hill, eldredge, & phillips, 2010) proceedings the conference began with a welcome from the university of new mexico’s vice-president for translational research, dr. richard larson, who is also the principal investigator on unm’s funded ctsa. a lively keynote address from kara malenfant, scholarly communications & government relations specialist at the association of college and research libraries, followed. ms. malenfant reviewed the history of academia and reminded the group to “respect the pain of change” in long-standing institutions, especially when attempting to promote and foster the use of open-access publication venues by translational investigators. ms. malenfant also applied organizational change theories to the dynamics of fostering change on large and complex organizations such as institutions with ctsas. kevin l. smith, scholarly communications officer at duke university next led a session on “the legal landscape for moving from bench to bedside” by highlighting copyright issues germane to the translational research environment (smith, 2010). mr. smith also discussed the novel copyright issues related to new technologies, such as shared electronic lab notebooks intended to foster collaboration. after lunch, holly phillips, one of the conference’s organizers, led a session entitled “novel uses of institutional repositories” (phillips, 2010). ms. phillips described the relative success of institutional repositories since inception and highlighted new userbased initiatives that show a promising future for institutional repositories. she then led the group through an exercise on applying the lessons learned with institutional repositories and changing investigator needs, such as data curation, to the ctsa community. the afternoon proceedings concluded with the research presentations session facilitated by jonathan eldredge, of which two presentations appear in this issue of eblip. the first day of the conference concluded with a dinner at the domenici center for health sciences education on the unm health sciences center (unm hsc) campus with dinner speaker dr. holly shipp buchanan, associate vice president for knowledge management and it, unm health sciences library and informatics center. dr. buchanan described how strengthening scholarly communications “empowers the powerhouse” of the “great american university” (buchanan, 2010). dr. buchanan highlighted unm health sciences library and informatics center’s (hslic) diverse efforts to support scholarly communication at the unm hsc through the model of librarian as educators and researchers. some of these efforts included an annual scholarly communications conference, support for an institutional repository, inclusion of a required for-credit biomedical evidence based library and information practice 2010, 5.4 111 informatics course taught by hslic faculty in the masters of science in clinical research degree program, and a biomedical informatics program. dr. philip kroth, conference planning committee chair and david gillikin, chief of the bibliographic services division at the u.s. national library of medicine began the second day with a two-person panel discussion. mr. gillikin provided an overview of the national institutes of health public access policy webpage (national institutes of health, 2010) and provided an update on “resources for complying with the nih public access policy” (gillikin, 2010). dr. kroth provided an overview of how the unm group leveraged the need for a biomedical informatics course as a required part of unm’s ctsa program to effectively engage future translational investigators in training on various aspects of open access publication (kroth, 2010). in addition he shared unm’s experience with nih policy promotion and support activities and summarized research results on unm’s policy compliance. a lively question and answer period followed on how to best promote compliance with the policy at “ctsa-minded institutions,” i.e., institutions with funded ctsas or those interested in applying for one. an upcoming publication will report on the main ideas, common challenges, and success stories captured during this session. dr. jonathan eldredge from unm hslic facilitated a two-hour interactive “advocacy communication workshop: crafting a 3minute message for open access”. special guest nancy ridenour, dean of the unm college of nursing and former robert wood johnson (rwj) foundation fellow assisted with the workshop. rwj fellows spend 2-3 year residing in washington, dc, learing about health policy by working on a congressional or white house committee. dr. ridenour gave an overview of her experience in washington working on the politically powerful united states house of representatives ways and means committee and shared tips for gaining the attention of busy, goal-oriented people. drs. eldredge and ridenour led the participants through several exercises teaching participants to focus on the salient points of scholarly communication of interest to translational researchers and administrator (eldredge & ridenour, 2010). the conference concluded with a session entitled “where do we go from here?” facilitated by karen butter , assistant vice chancellor for library services and instructional technology, library and center for knowledge management at the university of california san francisco. ms. butter used the nominal group technique (cross, 2005; delbecq, van de ven, & gustafson, 1975; van teijlingen, pitchforth, bishop, & russell, 2006) to develop priorities for action following the conference. these recommendations will be summarized in a forthcoming publication. the conference organizers also conducted a pre and post-test of attendees on the efficacy of this type of venue. these results will be highlighted in an upcoming publication as well. research program the evidence based scholarly communication conference (ebscc) featured four research project presentations. the papers were selected on the basis of relevance to the conference, the inclusion of applied research methods and the potential for post-conference publication. the selected papers reflect a diversity of research methods including program evaluation, cohort design, citation analysis, and action research techniques. the conference paper sub-committee (je and pk) worked extensively with the authors of the papers after the conference and two of them are included in this issue of eblip. (donahue, 2010b; gilliland, 2010a) ebscc featured a new peer review method. moderator jonathan eldredge asked each audience member to engage in “real-time peer review” by completing anonymous feedback forms on methodology, interpretation, and presentation immediately following each presentation. audience evidence based library and information practice 2010, 5.4 112 members could also volunteer their assistance with improving the authors’ manuscript by providing their names and email addresses on the forms. the four presenters received suggestions for improvement from ebscc participants as indicated by the numbers in parentheses following their names: banks (26); bardyn (20); donahue (24); and gilliland (18). dr. eldredge reviewed, scanned, and returned the forms to the presenters within two hours of pesentation to ensure immediate feedback. each presenter resubmitted revised manuscripts that integrated the “real time” peer reviewers’ comments following ebscc. drs. eldredge and kroth performed a final review before the authors were asked to submit their manuscripts to eblip for independent, anonymous peer review. marcus banks, and anneliese taylor, from the university of california, san francisco presented their work titled “an analysis of the impact of open access articles in translational medicine” (banks & taylor, 2010). this work compared the performance of both an open-access journal and a subscription-based journal using ”…techniques of article-level metrics recently established by the public library of science (public library of science, 2010) to enable a robust comparison of the differential impact of open and closed access.” tania p. bardyn, et al. from the university of california, los angeles (ucla), presented the work titled “measuring the effectiveness of nih public access policy programming and its capacity as a model for open access” (bardyn, brennan, camp, carter, & farb, 2010). the research analyzed attitudes of translational investigators regarding in-person workshops on nih policy compliance at ucla. interestingly, the study concluded that translational investigators preferred selfdirected formats over in-person training or individualized librarian consultation. amy donahue, from the bio-medical library at the university of minnesota, presented her work entitled “google wave: have ctsiminded institutions caught it?” (donahue, 2010a). ms. donahue conducted a case study involving action research evaluations of the utility of google wave by actually using google wave itself to conduct a survey on its usefulness in ctsa-minded institutions. ms. donahue concluded that the technology is too new yet to determine whether this will be of value to translational investigators (donahue, 2010b). anne gilliland, from ohio state university, presented her work titled “open access day at ohio state university” which reviewed the lessons learned from the prior health sciences library’s attempt at open-access publication promotion (gilliland, 2010b). a small-sized survey showed very favorable feedback from participants and interest in another such event for the following year (gilliland, 2010a). conclusion the research reports and attendees identified common themes. first, health sciences librarians experience difficulty in identifying translational investigators at an institution, a prerequisite to effectively publicizing specific resources for supporting publication in open access venues. librarians could explore ways to identify translational investigators within their institutions. the process would be helpful for libraries and investigators alike since collaboration is central to translational science. librarians could become involved in supporting investigator networks like vivo (www.vivoweb.org) and/or other productivity and data management tools. information on open access could be woven into these activities. second, health sciences librarians generally find it challenging to engage with, support and train translational investigators. librarians must have a practical, proven and timely open-access toolkit to offer investigators. an often repeated theme was the desire for a central repository or meeting for evidence based practices in order to save time and costly duplication of effort. work in this area could be critical to the sharing of best practices across similar institutions and the ultimate success of open-access publication. the conference confirmed the authors’ intial http://www.vivoweb.org/� evidence based library and information practice 2010, 5.4 113 premise that ctsa-funded institutions pose unique challenges in support of open access. there is great potential in this area moving forward. acknowledgement this work was supported in part by award number r13lm010054 from the national library of medicine. the content is solely the responsibility of the authors and does not necessarily represent the official views of the national library of medicine or the national institutes of health references association of research libraries. (2010). institute on scholarly communication. retrieved july 1, 2010 from http://www.arl.org/sc/institute/index.s html australian national university. (2010). retrieved 22 july 2010 from http://jcsmr.anu.edu.au/org/trans/inde x.php banks, m. & taylor, a. (2010). an analysis of the impact of open access articles in translational medicine. retrieved 22 july 2010 from http://hdl.handle.net/1928/10988 bardyn, t.p., brennan, m. j., camp, p., carter, j.d., & farb, s.e. (2010). measuring the effectiveness of nih public access policy programming and its capacity as a model for open access. retrieved 24 july 2010 from http://hdl.handle.net/1928/11002 buchanan, h. s. (2010). empowering the powerhouse: scholarly communication at the unm hsc. retrieved 22 july 2010 from http://hdl.handle.net/1928/10990 colón-ramos, u., lindsay, a.c., monge-rojas, r., greaney, m.l., campos, h, & peterson, k.e. (2007). translating research into action: a case study on trans fatty acid research and nutrition policy in costa rica. health policy and planning., 22(6), 363-374. cross, h. (2005). consensus methods: a bridge between clinical reasoning and clinical research? international journal of leprosy and other mycobacterial diseases., 73(1), 28-32. delbecq, a.l., van de ven, a.h., & gustafson, d.h. (1975). group techniques for program planning: a guide to nominal group and delphi processes. glenview, il: scott foresman. donahue, a. (2010a). google wave: have ctsa-minded institutions caught it? retrieved 22 july 2010 from http://hdl.handle.net/1928/10993 donahue, a. (2010b). google wave: have ctsa-minded institutions caught it? evidence based library and information practice, 5(4), 70-82. eldredge, j.d. & ridenour, n.a. (2010). advocacy communication workshop. retrieved 22 july 2010 from http://hdl.handle.net/1928/10991 gillikin, d. (2010). resources for complying with the nih public access policy. retrieved 21 july 2010 from http://hdl.handle.net/1928/10987 gilliland, a. (2010a). open access day at ohio state university. evidence based library and information practice, 5(4), 121-124. gilliland, a. (2010b). open access day at ohio state university. retrieved 22 july 2010 from http://hdl.handle.net/1928/10989 glascow, n. (2010). towards a national health and translational science institute for the act and canberra region. canberra doctor, 22(4), 1. evidence based library and information practice 2010, 5.4 114 kroth, p.j. (2010). nih policy at unm: how to get translational investigators to participate. retrieved july 21, 2010 from http://hdl.handle.net/1928/10455 kroth, p.j., bowler-hill, s.a., eldredge, j.d., & phillips, h.e. (2010). evidence-based scholarly communications conference: empowering information professionals to unlock translational research for our communities. retrieved july 21, 2010 from http://hdl.handle.net/1928/10467 kroth, p.j., phillips, h.e., & eldredge, j.d. (2009). leveraging change to integrate library and informatics competencies into a new ctsc curriculum: a program evaluation. medical references services quarterly, 28(3), 221-234. doi: 10.1080/02763860903069888 nakaya, j., shimizu, t., tanaka, h., & asano, s. (2005). current translational research in australia and national translational research supporting center in japan. chem-bio informatics journal, 5(2), 27-38. national center for research resources. (2010). institutional clinical and translational science award (u54). retrieved 1 jul 2010 from http://grants.nih.gov/grants/guide/rfafiles/rfa-rm-10-001.html national health and medical research council. (2010). 2010-2012 strategic plan: working to build a healthy australia. retrieved 22 july 2010 from http://www.nhmrc.gov.au/_files_nhm rc/file/publications/synopses/nhmrc_st rategic_plan_2010-2012_web.pdf national health service. (2010). national institute for health research. retrieved 16 may 2010 from http://www.nihr.ac.uk/research/pages/ default.aspx national institutes of health. (2010). public access policy webpage. retrieved 1 jul 2010 from http://publicaccess.nih.gov/ phillips, h. e. (2010). novel uses of institutional repositories. retrieved july 21, 2010 from http://hdl.handle.net/1928/10468 public library of science. (2010). article-level metrics at plos – addition of usage data. retrieved 1 jul 2010 from http://www.plos.org/cms/node/485 smith, k. l. (2010). copyright issues: the legal landscape for moving research from bench to bedside. retrieved 21july, 2010 from http://hdl.handle.net/1928/10457 van teijlingen, e., pitchforth, e., bishop, c., & russell, e. (2006). delphi method and nominal group technique in family planning and reproductive health research. journal of family planning and reproductive health care, 32(4), 249252. doi: 10.1783/147118906778586598 wilson, v., & neilson, c. (2010). personal communication with jd eldredge 12 july 2010. / evidence based library and information practice news   announcing the library journal club network      2017. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     we are happy to announce the launch of the library journal club network (https://journalclubnetwork.wordpress.com/), a space where those interested in establishing and sustaining journal clubs can share information, ask questions, and find answers.   the site includes:   guidelines for creating and managing a library-related journal club a list of readings and resources about journal clubs a directory of journal clubs   the site is currently set up as a resource for librarians who lead and participate in journal clubs. going forward, we hope the site will facilitate information sharing through the network. to get started, we invite journal club leaders / facilitators to visit our directory page (https://journalclubnetwork.wordpress.com/directory/) and submit information about their group to be added to the site.   we also welcome feedback about the site and ideas for expanding it in the future. you may use the contact section of the site to reach us.   megan fitzgibbons, university of western australia lorie kloda, concordia university andrea miller-nesbitt, mcgill university   evidence based library and information practice evidence based library and information practice 2010, 5.3 56 evidence based library and information practice evidence summary research into the impact of facebook as a library marketing tool is inconclusive a review of: xia, d. z. (2009). marketing library services through facebook groups. library management 30(6/7), 469-477. reviewed by: lotta haglund head of information and public relations karolinska institutet university library, stockholm, sweden email: lotta.haglund@ki.se david herron scholarly developer karolinska institutet university library, stockholm, sweden email: david.herron@ki.se received: 1 mar. 2010 accepted: 2 sept. 2010 2010 haglund and herron. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/bync-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. objective – to investigate whether facebook groups are useful for library marketing. design – content analysis of membership and activity of university library-related facebook groups. setting – two global facebook groups, and the facebook groups of two academic libraries in the us (rutgers university and indiana university, both with populations in excess of 30 000 students). subjects – a total of 28 facebook groups were analyzed. methods – facebook global groups are open to all users, while groups based in a network (e.g., a university) only allow access for those in the network. therefore, to collect data, the author used personal connections to log on to members’ profiles within university networks. the 26 university groups were selected by searching facebook for groups belonging to the two university networks, using the word ‚library.‛ groups unrelated to library business mailto:lotta.haglund@ki.se mailto:david.herron@ki.se evidence based library and information practice 2010, 5.3 57 were discarded. a total of 11 groups within the rutgers network were analyzed. of these, only one was organized by a librarian; the rest were organized by students. from indiana, 15 groups were identified, three of which were organized by librarians. in table 1 (p. 474), all groups are listed: 2 global groups and 26 groups within the two university networks. the author then visited all groups, read all posts, and recorded the total number of members; status of each member, divided into faculty, staff and students; dates of first and last post; and discussion activity. the author analyzed group activity by keeping a tally of how often each member participated in discussions, as there was no way to see the number of times a member returned. the author also paid special attention to groups with a large number of staff and faculty members, to gain information about the efforts of librarians to support or start new groups. main results – there were a total of 652 members in the 26 university groups (mean number of members was 25, ranging from 2 176). the two global groups had a total of 12,665 members. students were most active at starting new groups, but these were on average very small (around 20 members), with very little discussion. most discussions focused on limited topics or were event-driven, and therefore failed to retain member participation. the most active facebook groups were the global groups. these groups had a high staff and faculty membership, and librarians played an important role in promoting and maintaining group discussions. conclusion – according to the author, a successful facebook group should be managed by active organizers, and discuss a broad range of topics. good examples of active groups were the two global groups. group activity should be diverse, include discussion topics and wall posts, as well as messages sent to group members. the messages were found to be critical for library marketing as they appear as personal messages in members’ inboxes. commentary the objective of this article was to investigate if facebook groups are conducive to library marketing. this reviewer finds that the objective was not met by the research, since the article does not provide evidence of successful university library marketing through facebook use, and the impact of marketing via facebook was not measured. the validity and reliability of the study are thought to be weak. when evaluating the usefulness of facebook groups for marketing purposes, facebook groups are to be considered as a marketing tool, comparable to more traditional marketing tools (pamphlets, articles, web pages, newsletters etc), and not as marketing itself. this means that measuring the activity of facebook groups is not an indication of success when it comes to marketing libraries. to evaluate the usefulness of facebook groups for library marketing and to analyze these activities in comparison to other marketing tools, one would have to measure the outcomes of a marketing activity using facebook groups, thus being able to determine the usefulness of facebook groups to market libraries. one would need to: 1. define the aim of the marketing activity (e.g., a certain number of ‚fans‛ of the facebook group, a number of new library card holders, an active facebook group discussion, a certain number of attendees at a function in the library, etc.); 2. identify the target group for the marketing activity (e.g., students, staff, faculty, other librarians); 3. examine one or more control groups using other marketing channels/tools to reach the target audience. evidence based library and information practice 2010, 5.3 58 a further shortcoming is the lack of statistics to validate conclusion two above. the author states that groups were checked for ‚status of each member‛ (p. 472), which can be interpreted to mean whether the members are students, faculty, staff or librarians. in the article, no results are given of this status check; however the author refers to table 1 (p. 473), while saying ‚particular attention was paid to groups with a large amount of staff and faculty members.‛ however, table 1 does not contain information about the number of members from each group in the facebook groups. furthermore, the author argues that ‚messages are especially critical for library marketing‛ (p. 475). this statement highlights a problem with research into new technology, as it changes frequently, and the applicability therefore varies. the time period when the research took place is unclear, but it probably occurred during the second half of 2008 and january 2009. at the time of writing, a library can set up a ‘page’ on facebook, similar to a profile, where one becomes a ‘fan’ instead of a friend. postings to the page will show up in the newsfeed of the fan. for libraries starting activities on facebook, the reviewer thinks that a ‘page’ would be the more obvious choice instead of a group, where new postings would be visible to all fans without the use of messages. the author compares three different kinds of groups; 1) global groups for librarians, organized by librarians; 2) university network groups, organized by librarians; and 3) university network groups, organized by students. the groups in the first category are judged as successful by the author. no difference in impact can be found between category two and three groups. the reviewer finds it biased to compare local nonprofessional groups (category three) to professional groups with a common interest (category one). one can also question the decision to analyze groups organized by students when evaluating the use of facebook groups for library marketing. in spite of these shortcomings, the research offers additional insight (connell, 2009) into students’ willingness to accept libraries as contacts in social media. the research could be interpreted as students being spontaneously engaged in library issues when setting up facebook groups, but with short-lived interest. unfortunately, the article lacks information about the number of student members in library groups, making it hard to determine the extent of student commitment in the more active librarian-run groups. the author comes across as a strong advocate of the use of facebook in libraries, urging librarians to become more passionate about joining the ‘adventure.’ in the conclusion, the author states that facebook groups can be used ‚as a platform to support the teaching and research of faculty,‛ (p. 477) a seemingly isolated statement not discussed earlier in the article, but nevertheless interesting as a suggestion for further research. even though it does not meet the stated objective, the research presented may be useful to those planning to set up a facebook group to increase library visibility among students and faculty, by keeping topics general rather than specific, and by having proactive and enthusiastic organizers. references connell, r. s. (2009). academic libraries, facebook and myspace, and student outreach: a survey of student opinion. portal: libraries and the academy 9(1), 25-36. evidence based library and information practice evidence based library and information practice 2011, 6.1 90 evidence based library and information practice evidence summary development of technology competencies for public services’ staff has limited external validity a review of: wong, g. k. w. (2010). information commons help desk transactions study. journal of academic librarianship, 36(3), 235-241. reviewed by: jason martin associate librarian university of central florida libraries orlando, florida, united states email: michael.martin@ucf.edu received: 23 nov. 2010 accepted: 6 feb. 2011 2011 martin. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective to develop an understanding of the types of technology questions asked at an information commons help desk for the purposes of staffing the desk and training. specifically, the study looked to answer the following questions: 1. what kind of assistance do users seek from the help desk? 2. how complex is it to handle the technology questions? 3. what are the key competencies desirable of the help desk staff? design qualitative analysis of transactions completed at an information commons help desk. setting a medium sized academic library located in hong kong. data 1,636 transactions completed at an information commons help desk between january 2007 and may 2009. methods from the opening in 2006, the staff of the information commons help desk recorded all transactions electronically using a modified version of the open source software libstats. the author examined the transactions for roughly the second and third weeks of each month from january 2007 to may 2009 in mailto:mjmartin@mail.ucf.edu� evidence based library and information practice 2011, 6.1 91 an effort to determine the types of questions asked and their complexity. main results in response to question one, 86.3% of questions asked at the help desk concerned technology; the majority of those questions (76.5%) were about printing, wireless connection, and various software operation. for question two, 82% of technology questions were determined to be of the lowest tier (tier 1) of complexity, one-third of the questions required only “direct answers,” and 80% of questions could be answered consistently via the creation of a “knowledge base of answers for these foreseeable questions.” for question three, a list of fourteen competencies for help desk staff were created. conclusion with the low complexity of the technology questions asked, the creation of a knowledge base of common questions and answers, and proper training of staff based on the competencies identified in the study, an information commons could be effective with one integrated desk staffed by a librarian and paraprofessional staff member. commentary more and more academic libraries house information commons, and even those libraries without a proper information commons still have a plethora of computers in their buildings. technical knowledge of computers and computer software and applications has become de rigueur for librarians and library staff who answer questions at a public desk. this study attempts to address what technology competencies are needed for staffing a public service desk in an information commons. the author qualitatively analyzed questions asked at an information commons help desk over a two-year span, and her methodology is lacking in several areas. wong never explains why she chose to only examine the questions from two weeks of each month; she neglected analyzing half her data. wong never describes how the data is analyzed (i.e., did she use software like nvivo? or did she analyze the questions by hand?), nor does she provide a copy of the coding chart. wong admits determining the complexity of a question is very subjective, and she created a framework to assist her in this task; however, she gives no explanation as to what criteria she based her framework. as a result, her findings seem to be mostly opinion based on her level of computer literacy; others examining the data might come to different conclusions about the complexity of the technology questions being asked at the help desk. at the very least, the study needed another person to examine the findings to provide some reliability. most importantly, the study has limited applicability to other libraries. the competencies wong developed grew from the questions patrons asked at the help desk. however, wong works in a science and technology university in hong kong, and the questions her library’s patrons ask may differ greatly from those asked at a small liberal arts college in the northeast united states or a large university in australia. in addition, technology competencies are also based on the types of software and applications a library installs on its public computers. the study does provide a model for other libraries to determine the competencies needed by their public services’ staff, provided the short comings above are fixed. / evidence based library and information practice evidence based library and information practice evidence based library and information practice 2010, 5.1  evidence based library and information practice       evidence summary    undergraduate student use of the physical and virtual library varies according to  academic discipline    a review of:   bridges, l.m. (2008). who is not using the library? a comparison of undergraduate academic  disciplines and library use. portal: libraries and the academy, 8(2), 187‐196.    reviewed by:  megan von isenburg  associate director of public services, duke medical center library  durham, north carolina, united states of america  email: megan.vonisenburg@duke.edu    received: 20 nov. 2009          accepted: 06 feb. 2010       2010 von isenburg. this is an open access article distributed under the terms of the creative commons‐ attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐ sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original  work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is  redistributed under the same or similar license to this one.    abstract     objective – to determine differences in  undergraduate studentsʹ use of the physical  library and virtual library by academic  disciplines.    design – online multiple‐choice survey  followed by focus groups and secondary  online survey with open‐ended questions.    setting – oregon state university (osu), a  land‐grant university with over 19,000  students located in corvallis, oregon, united  states.    subjects – a random sample of 22% (n =  3,227) of the undergraduate population (n =  14,443), drawn by the registrarʹs office.  distance education and students at branch  campuses were not included. from this pool,  949 usable survey responses (29% of the  sample) were collected. the respondent  demographics proved to be reasonably  equivalent to those of the total undergraduate  population in terms of class standing  (freshman, sophomore, junior, or senior) and  academic discipline.    methods – the study consisted of three  phases. in phase one, an email invitation with  a link to the four‐item multiple choice online  survey was sent to students in the sample  population. results were analyzed using  pearson chi‐square tests to determine  goodness of fit between the following  variables: class standing and library visits,  class standing and virtual library use,  academic college and library visits, and  academic college and virtual library use.  129 mailto:megan.vonisenburg@duke.edu evidence based library and information practice 2010, 5.1  when significant dependence was detected,  researchers examined relationships between  the specific groups (e.g., freshman and  sophomore) and library use, and also  compared each group to one another using  odds ratios and by constructing 95%  confidence intervals.     phase two was intended to gather qualitative  information from the 275 infrequent or non‐ users of the library in focus groups. however,  researchers invited the 95 students in this  group who had indicated a willingness to be  contacted for further study, and only five  students participated. the author therefore  does not report on this limited data.     in phase three, researchers invited the 95  students who had self‐reported as infrequent  or non‐users of the library and who had  indicated a willingness to be contacted for  further study to complete an online survey  consisting of 36 open‐ended questions. 38  students responded. much of the data for  phase three is reported on in a separate  research article (vondracek, 2007).    main results – results from phase one are  reported in detail: in response to the question  of how often undergraduates visit the physical  library, 24.6% visited several times a year,  29.6% visited several times a month, 34%  visited several times a week, 7.7% visited once  or more per day, and 4% reported that they  did not visit at all. response to how often  undergraduate students use the online library  resources or website from outside the library  were: 37.7% use them several times a year,  32.8% use them several times a month, 12%  used them several times a week, 1.3% used  them once or more per day, and 16.2%  reported that they did not use them at all.    no significant relationships were found  between class standing and visits to the  physical library or class standing and virtual  library use.    researchers determined a significant  relationship between academic college and  visits to the physical library (p=0.003): college  of agriculture students were significantly less  likely to visit the library than students from  the colleges of health and human sciences,  liberal arts, and sciences.    researchers also determined a significant  relationship between academic college and  virtual library use (p=0.008): students in the  college of engineering were significantly less  likely to use the virtual library resources than  students in the college of liberal arts.    the survey from phase three of this study  asked students further questions about their  library use and relevant results are discussed  in this article. five students from the college  of agriculture responded to the survey and all  five students noted that they study at home.  when asked about where they go for help  with research, three reported that they ask a  friend or peer, one noted a professor and the  fifth did not respond to the question. four  engineering students responded to this  survey; when asked about where they carry  out online research, two responded that they  use google, one responded that he/she uses  the library, and the fourth noted that he/she  uses a building on campus.    conclusion – this study determined that  college of agriculture students were less  likely to use the physical library than their  counterparts in the colleges of health and  human sciences, liberal arts, and sciences,  and that college of engineering students were  less likely to use the virtual library resources  than students in the college of liberal arts.       commentary    this study seeks to collect data about the  differences in undergraduate studentsʹ use of  the library by class standing and academic  discipline. as the author acknowledges, there  is a commonly held belief among academic  librarians that students in some academic  disciplines, such as liberal arts, use the library  more frequently than those in other  disciplines, such as science or engineering.  however, most literature on this subject either  130 evidence based library and information practice 2010, 5.1  131 focuses on graduate students or faculty, not  undergraduate students, or pre‐dates online  article databases and other electronic  resources. this study therefore represents a  solid attempt to provide some evidence in an  area where assumptions may have dominated.    the author acknowledges some potential  weaknesses in the study: students were aware  that librarians were administering the survey  and could have reported more frequent library  use since this could be perceived as the best or  proper response. response rates for the survey  (29% or 949 out of 3,227) were within the  expected range, but response for the focus  groups (5% of infrequent or non‐users who  expressed willingness to participate further  and only 1% of all infrequent or non‐users)  was poor. the author wisely noted that the  data collected in the focus groups was not  sufficient for comparison to the phase one  survey data.     the phase three open‐ended survey sought to  fill in the gaps created by the low response  rate to the focus groups, though only a few  students from the relevant colleges provided  answers. it is difficult to extrapolate from  these individual responses precisely why these  groups of students use the physical and  virtual libraries less than other students.     to determine why the student groups use the  library to varying degrees, the author cites  supporting examples from the literature and  offers some additional suggestions based on  observation. for example, the college of  agriculture (whose students were less likely  to come to the library) holds most of its classes  at a greater physical distance from the library  than most classes in the liberal arts, sciences,  and health and human services. the college of  engineering students (who were less likely to  use the virtual library) were not able to log on  to their engineering student accounts at the  library and received free printing at their own  building while the library charged $0.07 per  page for printing services.     the survey instrument is not included in the   article, and it is not clear whether examples for  virtual library resources were listed. this, too,  could have impacted the results as students  may not be aware that online journal articles  accessed through a web resource such as  google scholar are, in fact, provided by the  library. similarly, students could be accessing  what has been termed ʺhidden libraries:ʺ  library resources provided in course  management sites such as blackboard by  instructors who post articles or links to  databases and thus remove the need for  students to access the library directly (van  scoyoc, 2006). perhaps, as the author suggests,  students in some academic disciplines are  simply not assigned projects or papers that  require library resources.    these and other potential reasons for the  varying use of the physical and virtual  libraries are worthy of a follow‐up study.  interestingly, the original research design  intended to use the online survey, which  provided the bulk of the data for this article, as  a screening tool for student focus groups.  given the low response rate to the focus  groups and the low numbers of engineering  and agriculture students who responded to  the phase three survey, perhaps the next step  is a targeted study of these two groups of  students.    it would also be worthwhile for other  institutions to survey their own  undergraduate students to help determine  whether the differences found in this study are  unique to osu or more widely applicable.      references    van scoyoc, a. m., & cason, c. (2006). the   electronic academic library. portal:  libraries and the academy, 6(1), 47–58.    vondracek, r. (2007). comfort and  convenience? why students choose  alternatives to the library. portal:  libraries and the academy, 7(3), 277– 294.   editorial   evidence of impact   alison brettle editor-in-chief school of nursing, midwifery and social work university of salford, united kingdom email: a.brettle@salford.ac.uk      2014 brettle. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   welcome to the june 2014 issue of the eblip journal. while we were pulling together the content for this issue, i found myself thinking about impact and how this relates to evidence based practice. the reason for this was because i was preparing for a talk on “measuring impact in practice” for the scottish health information network (shine) annual study day. evidence based practice in library and information work is all about using evidence in our decision making, and is usually associated with effectiveness or whether something works. on the other hand, impact is something that as library practitioners we are increasingly asked to demonstrate to our stakeholders; and is about whether our services make a difference. however just as we can use or locate evidence to help us in our own decision making to find out whether an aspect of our service works, we can also use evidence to help us understand whether our service makes a difference. not only can we use this evidence of impact for ourselves in our own decision making, we can use evidence to demonstrate the impact of our practice or service to our stakeholders and help them make decisions regarding our services. thus measuring impact has an important place within evidence based practice.   the shine study day (http://www.shinelib.org.uk/news/596) provided plenty of useful information to those who want to measure or provide evidence to demonstrate impact about any aspect of their library service. wilson (2014) presented a theoretical background and detailed framework that has been implemented across the national health service scotland knowledge services. this framework can be used to measure a wide range of different impacts, and complements other guidance about impact, such as those expressed by hall (2011), tenopir (2013), or bawden et al. (2009). lemay (2014) provided an example where evidence from library services has been incorporated into a clinical decision making system and then monitored to document impact on patient care, while brettle (2014) addressed the practicalities that need to be considered when measuring impact. all speakers provided slightly different definitions of impact, but a common theme was the need to specify the outcomes you are measuring, so that you know what evidence you need to collect and the need to be aware of your stakeholders so that you can ensure you collect evidence that is important and relevant to each particular stakeholder.   as well as evidence to inform your practice, this issue has a number of news items for grants and conferences that might help give you a boost, so go ahead and make an impact!     references   bawden, d., calvert, a., robinson, l., urquhart, c., bray, c. & amosford, j. (2009). understanding our value; assessing the nature of the impact of library services. library and information research, 33(105), 62-89.   brettle, a. (2014, may). measuring the impact of health libraries in practice. presented at the shine agm & study day 2014, glasgow, scotland.   hall, h. (2011). project output versus influence in practice: impact as a dimension of research quality. evidence based library and information practice, 6(4), 12-14. retrieved from http://ejournals.library.ualberta.ca/index.php/eblip   lemay, j. (2014, may). cebis: outcomes and impact, capture and use. presented at the shine agm & study day 2014, glasgow, scotland.   tenopir, c. (2013). building evidence of the value and impact of library and information services: methods, metrics and roi. evidence based library and information practice, 8(2), 270-274. retrieved from http://ejournals.library.ualberta.ca/index.php/eblip   wilson, s. (2014, may). an evaluation framework for nhsscotland knowledge services. presented at the shine agm & study day 2014, glasgow, scotland.   microsoft word es_hall.doc evidence based library and information practice 2006, 1:3      70 evidence based library and information practice       evidence summary    communication breakdown: librarian and student approaches to virtual reference  differ    a review of:  walter, virginia a. and cindy mediavilla. ʺteens are from neptune, librarians are from pluto:  an analysis of online reference transactions.” library trends 54.2 (2005): 209‐227.    reviewed by:  stephanie hall  chief librarian, powell river public library  powell river, british columbia, canada  e‐mail: shall@powellriverlibrary.ca      received: 01 june 2006            accepted: 17 july 2006      © 2006 hall. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to evaluate the effectiveness of  an online reference and referral service for  students (primarily those in middle school  and high school) seeking homework help.    design – analysis of 114 transcripts of  reference transactions.     setting – a centralized homework reference  and tutor referral service provided on behalf  of the california state library by the  metropolitan cooperative library system.     subjects – virtual reference librarians at a  large urban library system and middle and  high school students in california.    methods – one hundred fourteen virtual  reference transactions recorded between  october 12 and november 8, 2003 were  evaluated against the reference and user  services association’s (rusa) “guidelines  for behavioral performance of reference  and information service providers.”  secondly, the transcripts were subjected to  discourse analysis.     main results   performance of virtual reference librarians vis‐à‐ vis rusa guidelines  in the majority of cases, there was some  evidence that librarians communicated  clearly (110 out of 114 transactions). in 78  cases, it appeared that a librarian was  available quickly, and in 92 of the  transactions a friendly greeting was given.  what was striking, however, was that in a  clear majority of cases, some of the classic  reference interview strategies were not  employed. in 100 or more cases each, the  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:3      71 following strategies were not observed:  repeating or paraphrasing the question;  helping to interpret the question; verifying  mutual understanding; asking if the  question has been answered; asking if the  student needs more information.  furthermore, in 75 cases librarians did not  probe for further information to clarify the  question, while in 87 cases they did not  check that information had been clearly  understood.     possibly related to these findings, the  researchers felt that the transcripts revealed  “a conviction that homework questions are  not the proper content for reference  transactions” (222). in addition, librarians  were found to be frequently too quick to  refer students to a tutor, when a query  might have been better answered as a  reference question.    findings of discourse analysis  in general, the virtual reference librarians  used impersonal, formal language to  “reinforce the professional’s superior  position vis‐à‐vis the help‐seeker” (217).  there were repeated attempts by the  students to interject a lighter or warmer tone  (using humour, emoticons, informal  language, introducing a personal note, etc.).  these attempts were rarely reciprocated,  with librarians continuing to use impersonal  language, including stock messages such as:  “we are experiencing a very busy time  right now,” (217); or,  “i am going to send you a page which  will give you some help with your  homework. after we disconnect this  session, click on this link and follow the  instructions to be connected with a tutor.  please do not click on any links on this  page until after we have disconnected”  (217).    in several cases librarians were preoccupied  with defining their roles—for example,  specifying that they could give reference  help but not advice. in all, the distancing  mechanisms used by librarians, combined  with occasional inaccurate referrals and  technical problems with the software, were  seen to create an enormous potential for  frustration on the part of the student.    conclusions – the most important  implication of the study was that librarians  and students were worlds (or planets) apart  in their approach to the reference interaction.  while “teens attempted to create meaning  by recreating the chat discourse  environment in which they were most at  home”, librarians “tried to create meaning  in a parallel discourse environment that  duplicated as much as possible the standard  impersonal protocols of a face‐to‐face  reference counter” (223). one suggested  way to alleviate the disconnect between  librarians and students was to involve  students in the planning of the services. for  now, however, the authors conclude that  “teens are from neptune, librarians are from  pluto. better services would result if they  could meet somewhere closer together”  (224).    commentary    the authors note that they did not have the  opportunity to follow up with the librarians,  nor did they have access to other virtual  reference transactions, such as queries by  adult patrons, for comparison. there was a  suggestion that reference librarians viewed  questions by students as ‘second class’  reference questions. follow‐up interviews  and a comparison of adult and student  transactions might have clarified the  librarians’ attitudes in this regard.     in addition, further information would have  been useful on several points: did the 114  transcripts (originally 115) represent a  sample of a larger number of student virtual  reference transactions, or were they, as it  appears, all that were collected within the  evidence based library and information practice 2006, 1:3      72 time period specified? how did the total  number of student queries compare to  queries from other age groups? also,  somewhat more detailed information on the  virtual reference librarians, such as the  number of staff responding and their  qualifications, would have been of interest.    as the study was limited to a single  location, it is possible that the widespread  deviation from the rusa guidelines was the  result of a localized training or planning  deficiency. also, a follow‐up survey or  interviews with students themselves might  have given a more precise sense of their  satisfaction or dissatisfaction with the  service offered.     while limited in scope, the study presents  its evidence clearly and uses established  guidelines to evaluate the reference  transactions. the study offers useful                                                      insights into the differences that can emerge  between adult librarian and teen/student  patron in the virtual reference arena. as we  look forward, it is to be hoped that the  unique opportunity afforded by the virtual  reference transaction record (the ability to  preserve the transaction in its entirety) will  prove a rich source for analysis aimed at  improving reference service.     work cited    “guideline for behavioral performance of  reference and information service  providers.” reference & user services  quarterly 43.1 (2003): 43‐7. research article   practice of infopreneurship by librarians in public university libraries in south-south nigeria   jerry eyerinmene friday cataloguing & classification librarian university library federal university otuoke bayelsa state, nigeria email: fridayje@fuotuoke.edu.ng   oyinkepreye sawyer-george polytechnic librarian bayelsa state polytechnic aleibiri, bayelsa state, nigeria email: oyinkepreyesawyer-george@byspoly.edu.ng   received: 11 sept. 2022                                                             accepted: 31 mar. 2023      2023 friday and sawyer-george. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30235     abstract   objective – the aim of this study was to examine the practice of infopreneurship by librarians in public university libraries in south-south nigeria. the study specifically intended to identify purposes of engaging in infopreneurship, methods of running infopreneurship, forms of infopreneurship practiced, benefits derived from practicing infopreneurship, and challenges encountered in practicing infopreneurship by the librarians.   methods – the population of the study comprised all 175 librarians in 13 public university libraries in south-south nigeria, which were purposively chosen for the study. the study employed convenience sampling to engage 102 librarians in the university libraries, who were involved in one form of infopreneurship or another. the librarians were identified through preliminary investigation, observation, and interaction with the librarians by the researchers. the instrument for data collection was a self-designed online questionnaire titled, “librarians’ infopreneurship practice questionnaire (lipq).” the instrument was validated by two experts in the department of library and information science in niger delta university, bayelsa state, nigeria. reliability test was not conducted on the instrument, based on the knowledge that a valid test tends to be reliable. the researchers distributed 128 digital copies of the draft of the validated questionnaire to the librarians through personal whatsapp accounts of the librarians, whatsapp groups of the various university libraries and whatsapp groups of the different state chapters of the nigerian library association to which the librarians belonged. out of the 128 questionnaires administered, 102 were properly completed by the librarians and returned, producing a response rate of 97.69%. the data collected were analyzed using weighted mean and standard deviation.   results – the results from data analysis revealed that the librarians’ purposes of engaging in infopreneurship were to earn extra income, provide for post-retirement, meet unforeseen demands, and develop themselves. the librarians’ methods of running infopreneurship were leveraging, customizing, facilitating access to, and providing instant delivery of information. they used the following forms of infopreneurship: information brokerage, reprographic services, research-aid services, book vending and internet services. finally, challenges faced by the librarians in practicing infopreneurship were lack of adequate finance, business infrastructure, technical skill, and high rate of presence of non-professional infopreneurs.   conclusion – the findings in this study demonstrate that librarians under study practice infopreneurship primarily for purposes of making money and self-sustenance. they achieve this by leveraging, customizing, facilitating access to and providing quick delivery of information. with these methods, the librarians engage in information brokerage, reprographic services, internet services, research-aid services and book vending. in return, these information professionals enjoy additional income, financial independence, accumulated knowledge, and enhanced sense of fulfillment. however, the practice of infopreneurship by the librarians is hindered by shortage of sufficient funds, technical know-how, business facilities and high rate of presence of unprofessional infopreneurs in the business.     introduction   the world economy has shifted from the agricultural age to the industrial age, and currently, to the age often referred to as the information age. the information age is an era in which success, progress and survival at individual, organizational, national, and global levels is mainly based on the generation, access, dissemination and use of information. traditionally, the agricultural and industrial ages mainly contributed to world and national economies. in recent times, the potentials of information have made it a critical factor that enhances such economies. presently, in africa, many economies are of a hybrid nature, whereby traces of the agricultural, industrial and information ages are observed. however, features of the information age seem to be more prominent in the continent. a prominent characteristic of the information age in the continent of africa is the emergence of a new economy referred to as knowledge economy. the knowledge economy is an economy in which information is treated as a good or commodity.  it is an economy which is primarily based on the production, marketing, sale and utilization of information for the satisfaction of human needs. the knowledge economy has been springing up in many countries of the world, including countries in africa.   one of the countries in africa in which the knowledge economy has been growing is nigeria, comprising six geo-political zones: north-central, north-west, north-east, south-south, south-east and south-west. the knowledge economy has been growing as an essential part of the nigerian predominantly-oil and gas-based economy with south-south nigeria providing a significant support to its economy through the production of oil (my guide nigeria, 2017). the development of the economy in the country is as a result of the realization of the economic potentials of information in contemporary times. such potentials are reflected in the ability of information to be transferred, modified, traded, stored, customized, distributed and replicated (masron et al., 2017). thus, information has become an important element of the knowledge economy (du toit, 2000). consequently, individuals, organizations and countries make huge investments in information as a commodity because of their growing recognition that information is useful for their survival (ocholla, 1999). its status as an important factor of production in the knowledge economy has provided opportunities for information-based businesses, and by extension, has led to the emergence of the concept of infopreneur (lose & khuzwayo, 2022), which was first designed as a trademark by harold f. weitzen on january 31, 1984 (lahm & stowe, 2011).   an amalgam of "information" and "entrepreneur," (taylor, 2020), an infopreneur is someone whose main business is to gather and sell information, including online information, by identifying opportunities for creating profit-oriented information-based businesses based on the recognition of knowledge gaps, and by selling customized information products and services to clients (adetayo & hamzat, 2021; el-kalash et al., 2016; akpena & duosakeh, 2020). two types of infopreneurs exist: those who sell the information they have personally collected and those who help others to sell their information (akpelu, 2019). the activities of infopreneurs have resulted in a new form of business called infopreneurship.   in the context of this study, infopreneurship is defined as the act of running a business in which information is gathered from multiple sources and sold in exchange for money. kazungu (2021) defined infopreneurship as the ability to organize resources, bear risk and undertake intelligent searches to gather information from different sources, combine it into novel ways, and serve the readers’ needs through internet tools as a value-added service for profit. nweze (2018) opines that infopreneurship involves taking advantage of business opportunities to produce information products and services and to manage information. it is a business that involves the creation, capture, acquisition, processing, sharing and use of information as products and services (aregbesola, et al., 2019). infopreneurship is a business model where an individual, or "infopreneur," shares their life experience, knowledge, and passion with others through information products and services that create value and generate income (ogbonna & dare, 2019). it could involve selling print information, online information, or both. the types of information sold may include books/e-books, audio products, online courses, coaching, virtual summits, workshops, masterminds (outstanding personal creative works), presentations, videos, in-person seminars, tele-seminars, web seminars, special reports, and workbooks (chandler, 2007; ogbonna & dare, 2019. although these information products are sold through cd-roms, audiotapes, audio cds, videos, talk shows and conferences, the advent of information and communication technology (ict) has caused a shift in the method of running infopreneurship from traditional information products and services to ict-driven electronic information services, where the key players are the internet and cybercafe (lahm & stowe, 2011).   infopreneurship covers such areas as provision of internet services, e-services and m-services, computer trouble-shooting services, provision of student’s essays, term papers, research proposals and research reports writing services; proofreading and editing, provision of access to e-books, e-publishing, business analysis consultancy, web newsletters and online solutions through recorded videos and referencing (dewah & mutula, 2016; ramugondo, 2021). infopreneurship is considered an important source of accurate, valuable and reliable information, which helps in saving time and costs related to information seeking among individuals and organizations (masron, et. al., 2017).   the availability of various forms of ict, information explosion, and the ever-increasing needs of individuals, organizations and institutions for teaching, learning, research, decision making, planning, and production create increasing infopreneurship opportunities. onaade (2012) observes that the opaque characteristic of information and the varied information needs of clients gave birth to a complex crop of infopreneurs, who are in such fields as records and archives management, library and information science, journalism, media studies, and publishing studies (david & dube, 2013). in south africa, these individuals operate from such disciplines as library and information science, ict/computer science, telecommunications, printing and publishing, records and archives management and mass communication (david & dube, 2014).   in nigeria, individuals from similar disciplines operate infopreneurship. these fields include library and information science, computer science, mass communication, and others. infopreneurship is one of the business activities springing up in the economic environment of nigeria. several nigerian government workers and unemployed individuals (graduates and non-graduates) have taken advantage of the opportunities presented by the knowledge economy through the establishment of infopreneurial businesses. a core part of these government employees are librarians in nigerian universities. many librarians in these universities have drawn from their wealth of skills and knowledge, as information professionals, to engage in infopreneurship. akpena and duosakeh (2020) observe that, with its status as a marketable commodity and high users’ demands for information, infopreneurship has become a viable business for information professionals in nigeria.   some librarians, particularly those who are widely published, take advantage of their access to closedand open access journals by collecting journal articles from foreign journals, university library databases, institutional repositories and from reserve units of universities libraries and selling them to undergraduate and postgraduate students for a fee. sometimes, these librarians gather articles from closed-access journals and sell them to fellow librarians who lack access to those journals but need them to write their research papers. some have also established computer business centers or cybercafés where they download relevant information for undergraduate and postgraduate students for the execution of their assignments, seminars, projects, and other academic write-ups, in exchange for money. some librarians are also providing reprographic services such as typing, printing, scanning, photocopying as well as internet services such as browsing, uploading of documents, online examination registration, result checking, e-mailing, etc.   as government employees, most of these infopreneurial librarians have hired people to run the business outlets while they act as business managers or directors. on some occasions, the librarians themselves provide similar services to the clients after working hours. most of the clients are undergraduate and postgraduate students, teachers, and researchers of tertiary institutions. these clients often use the information for academic purposes. the fair use copyright exception permits use of information for academic purposes, private or personal study, review, or criticism. in most cases, a good number of the information materials with which these information professionals render these services are often freely-available online information resources. the librarians seldom use university library information resources for the provision of these infopreneurial services. consequently, they are usually free from the copyright policies implemented by the libraries. with this, infopreneurship has become a means adopted by librarians for sustaining their lives and responding to the deteriorating economy in which they work. thus, elumelu (2014) sees infopreneurship as a strategy through which information practitioners control their destiny and tackle the serious risks that mass unemployment poses to the stability of societies and economies.   statement of the problem                the increasing daily users’ demands for information for such activities as teaching, learning, research, decision making, planning, production, etc. has presented librarians with opportunities for running an abundance of information-based businesses, one of the popular forms of which is infopreneurship. with these opportunities, a number of librarians across the world have drawn from their wealth of experience as information professionals to venture into infopreneurship as a means of complementing their salary and enjoying some level of financial satisfaction. a number of librarians in public university libraries in south-south nigeria are involved in operating various forms of infopreneurship such as information brokerage, establishment of computer business centers or cybercafés for provision of reprographic services such as typing, printing, scanning, photocopying as well as internet services such as browsing, uploading of documents, online examination registration, result checking, e-mailing, and more. according to akpena and duosakeh (2020), the status of information as a factor of production and increased users’ need for it has made infopreneurship a viable business for nigerian information professionals. in recent years, infopreneurship has drawn the attention of researchers in library and information science around the globe. while a growing body of research is available on infopreneurship in south africa, botswana, and other african countries (ivwurie & ocholla, 2016; ocholla, 1998; 1999, etc.), very little research has been conducted on this topic in nigeria. these nigerian studies (adetayo & hamzat, 2021; akpena & duosakeh, 2020) have only been executed in south-west and north-east nigeria respectively. hence, our study examined the practice of infopreneurship by librarians in public university libraries in south-south nigeria.   purpose of the study   this research set out to investigate the operation of infopreneurship by librarians in public university libraries in south-south nigeria. specifically, the study intended to:   1.      identify the aims of engaging in infopreneurship by librarians in public university libraries in south-south nigeria. 2.      discover the ways of carrying out infopreneurship by librarians in public university libraries in south-south nigeria. 3.      describe the forms of infopreneurship practiced by librarians in public university libraries in south-south nigeria. 4.      identify the perceived benefits gained from running infopreneurship by librarians in public university libraries in south-south nigeria. 5.      identify the obstacles in practicing infopreneurship by librarians in public university libraries in south-south nigeria.   research questions   the following research questions were raised to guide the study:   what are the purposes of engaging in infopreneurship by librarians in public university libraries in south-south nigeria? what are the methods of carrying out infopreneurship by librarians in public university libraries in south-south nigeria? what are the forms of infopreneurship practiced by librarians in public university libraries in south-south nigeria? what are the perceived benefits derived from practicing infopreneurship by librarians in public university libraries in south-south nigeria? what are the challenges in practicing infopreneurship by librarians in public university libraries in south-south nigeria?   literature review   purposes of engaging in infopreneurship by librarians in university libraries   individuals, including librarians in university libraries, have several purposes or reasons for infopreneurship. these purposes have been hinted at by expert observations and revealed by previous studies. for instance, research conducted in botswana by ocholla (1998; 1999) reveal several reasons for engaging in infopreneurship, including:   ·        decline of resources to sustain wage employment in the public and private sector, ·        increase in unemployment calling for self-employment, ·        dead-end jobs retarding professional and career growth, ·        the inability of existing information provision centers to provide information services needed and increased demand for specialized information services, ·        the willingness of information consumers to pay for consultancy services and recognition that information is a commodity that can create wealth, ·        acceptance that information is power necessary for individuals and firms to stay in the market and to keep the competition off-balance, ·        interest in self-employment, ·        inadequacy of incomes which force income earners to sell skills and knowledge for additional earnings, ·        social change witnessed in the creation of small businesses, ·        encouragement by entrepreneurs who buy expertise, hire consultants and avoid obligations for hiring staff on a long-term basis, ·        increased need for proper information management, and ·        size and complexity of the information industry.   ivwurie and ocholla (2016) found that most graduates from library and information science and other related fields in selected cities in lagos state and oyo state in nigeria and kwazulu natal in south africa, who venture into information-based businesses do so for financial reasons and survival. in a subsequent study in nigeria, akpena and duosakeh (2020) found that librarians in bauchi state, nigeria, engage in infopreneurship with a view to earning extra income, providing for post-retirement, and responding to identified demands. a fairly recent research study by adetayo and hamzat (2021) shows that gaining leadership experience, self-development, library education and training practice, socialization, and money-making are the objectives of venturing into infopreneurship among library professionals in tertiary institutions in ede, osun state, nigeria.   methods of carrying out infopreneurship by librarians in university libraries   individuals, including librarians in university libraries, run infopreneurship in various ways. expert opinion and observation provide insight into these methods. taylor (2020) observes that an infopreneur gleans information from either a variety of sources or from their life experiences and uses it to create a saleable content. this content has value for the intended user. amin et al. (2011) noted that an infopreneur creates value by gradually processing data into information and knowledge relevant to the user. the value could be a result of the potential of information, such as its ability to fill a gap in a person’s knowledge, aid someone’s understanding, planning, decision making, or gaining of awareness. thus, david and dube (2014) assert that an infopreneur is one who spots opportunities for creating information-based businesses by identifying knowledge gaps and selling target-based information products and services, mainly through the internet. david and dube further note that infopreneurs take advantage of such attributes as curiosity, enthusiasm, courage, and spirit to conduct intelligent searches on a broad spectrum of information items and evaluate, repackage, and determine the significance, relevance and value of information and information services which they then sell by means of the internet. coulson-thomas (2000) observes that the rise in the demand for specialized and repackaged information has driven infopreneurs to devise strategies to enable information users manage the abundance of information sources and resources by evaluating, sifting, screening, and classifying an abundance of information to meet users’ needs. weitzen (1998) provides a comprehensive view of the methods adopted in operating infopreneurship. he identifies six ways in which infopreneurs gainfully operate their business. these six ways, which shapiro and varian (1999) claim to be widely accepted by different researchers as the methods of running infopreneurship, include leveraging information, customizing information, facilitating access to information, speeding up the flow of information, repackaging information and providing around-the-clock delivery of information to clients.   forms of infopreneurship practiced by librarians in university libraries   there are a number of businesses under the umbrella of infopreneurship which could be described as forms or areas of infopreneurship. ocholla (1999) sees areas of infopreneurship as projects that individuals who have opted for infopreneurship are likely to embark upon. scholars and experts’ opinions suggest two forms of infopreneurship: those operated in the traditional (print) information environment and those run in the digital information environment. a critical review of literature indicates that the digital forms of infopreneurship far outnumber their traditional counterparts. in a way that leans more on digital forms of infopreneurship, ocholla (1999) identifies the following areas of infopreneurship: research (exploratory and evaluative) in information and related fields such as user studies and market analysis, compilation of bibliographic lists, provision of current business information, compilation of directories, publishing, translation services, information repackaging, writing, editing and proof reading, collection management, records management and cataloguing. regarding digital forms of infopreneurship, jennings (1998) identifies webmasters, cybrarians, web managers, private and public cloud services. in a more comprehensive way, chandler (2007) identifies sale of electronic books, web newsletters, engagement in electronic publishing, provision of business analysis consultancy, subscription-based access, and online solutions on methods of referencing, research proposal writing, or career options in the field of library and information science through recorded videos or chartrooms/ online conferences through social media such as facebook and skype. david and dube (2014) highlight other digital forms of infopreneurship to include web designing, software development, provision of print-on-demand services, online publishing, software and hardware installation, library automation, marketing of information products, information repackaging, records management, proofreading and editing, internet service provision, electronic abstracting and indexing and online broadcasting. in a subsequent observation, ramugondo (2021) identifies others to include internet providers, e-services and m-services, computer troubleshooting services, student’s essay, term paper, research proposal and research project writing, proofreading, and editing. moreso, adetayo and hamzat (2021) highlight records management, library automation, data management services such as database creation and management, cloud services, data analysis, content analysis, online teaching services, webinars, workshops, information-based consulting services, knowledge management services and online marketing services.   similar forms of infopreneurship have been revealed by research findings. from a study conducted in nigeria and south-africa, ivwurie and ocholla (2016) found that infopreneurship is a growing practice among registered small, medium information-based business owners and library and information science graduates in the two countries. their study reveals that most of its products and services are driven by information and communication technology, and these include software and hardware installation, tracking services, troubleshooting, web designing, programming, cctv installation and online televised broadcasting and so on. in a subsequent study executed only in nigeria by akpena and duosakeh (2020), it was shown that book vending, information brokerage, indexing services and digital publishing are the major forms of infopreneurial services provided by librarians in bauchi state of the country.   librarians’ perceived benefits derived from practicing infopreneurship   individuals, including librarians in university libraries, who engage in the business of infopreneurship derive a number of benefits from it. chandler (2007) states that the benefits gained from practicing infopreneurship are that it attracts a passive income, requires little capital, offers an entrepreneur the chances of acting as a specialist in the business, is easy to market and can be developed effortlessly. in the same vein, skrob (2009) opines that infopreneurship facilitates manual business by accelerating the provision of services, gives an easy revenue, has a low entry cost, offers specialized services, presents new items which bring new clients, provides advertising opportunities, develops cross-promotion opportunities, has potential for corporate deals and probable circulation via verbal exchange and next-to-zero interaction with customers or clients. in addition to these core benefits, it has also been hinted that infopreneurship has the potential to provide self-employment for unemployed individuals. for example, kehinde (2021) asserts that infopreneurship serves as an alternative source of employment to graduates, of which a growing number of nigerian university graduates, according to olorunfemi (2021), have been incapable of obtaining full-time employment in their disciplines since the nigerian economic recession began in 1984. thus, infopreneurship has been viewed as capable of solving problems of mass youth unemployment (chux-nyehe & nwinyokugi, 2020), a core part of which are the unemployed nigerian university graduates, comprising 52million (national bureau of statistics, 2016). furthermore, infopreneurship has been conceived to be beneficial to business managers, such as empowering them to be employers as well as developing their intrinsic and extrinsic qualities. in this regard, amakiri et al (2019) observe that infopreneurship enables managers to become employers of labor and helps them to build such qualities or skills as focus, innovativeness, creative thinking, and strategic planning. a nigerian study conducted by adetayo and hamzat (2021) shows that development of self-confidence, acquisition of knowledge, life enhancement, boosting of sense of fulfillment, financial independence and recognition are the benefits of infopreneurship gained by library professionals in tertiary institutions in ede, osun state, nigeria.   librarians’ challenges in practicing infopreneurship   infopreneurship, just like any other business undertaking, is not free of obstacles. expert observations and findings in previous studies on infopreneurship attest to this. olien (2013) maintains that not all infopreneurs can create a successful enterprise. this indicates that some infopreneurs are confronted by certain challenges as they attempt the business of infopreneurship. challenges could be classified into three major groups: personal, regulatory, and societal.   personal challenges   ocholla (1999) observes that the field of infopreneurship suffers from individuals who are not passionate about becoming infopreneurs, adding that if an individual starts an information business and it fails, that individual will just abandon his or her dream of becoming an infopreneur, and decide to seek a formal job. lack of finance can also be seen as another personal challenge. samitowska (2011) and smith and beasley (2011) observe that lack of finance or access to finance is acknowledged as the greatest challenge to growing infopreneurship. this probably explains why about 72% of nigerian entrepreneurs experience financial constraint in their attempt to advance and operate their businesses (mambula, 2002). hopes of developing a new business in the country are often dashed as a nigerian bank hardly provides a loan to an upcoming skilled entrepreneur without requesting for collateral or charging a high interest rate (onwuka et al., 2014). a dearth of business infrastructure is another personal obstacle. the small enterprise development agency ([seda], 2016) observes that a challenge encountered by infopreneurs in developing countries is the lack of business infrastructure. absence or shortage of skill is a further personal challenge. david and dube (2014) report that information science graduates lack the business and technical skills required to translate infopreneurial ideas into businesses. javier et al. (2012) note that shortage of information and communication technology skills among african infopreneurs prevents majority of them from succeeding in their businesses. furthermore, lack of managerial skill among infopreneurs, has been identified as another obstacle that has a negative effect on the practice of infopreneurship (agbenyegah, 2013). the lack of identity has been conceived as another barrier. fisher and kunaratnam (2007) observe that the absence of a collective name for infopreneurs has adverse effects on their identity such as the reduction of their legitimacy, true value, validation, and sense of belonging and depreciation of their chances of development. in the same vein, ivwurie and ocholla (2016) opine that a lack of a universally acceptable name may cause problems in the identification and verification of information-related businesses.   regulatory challenges   auriol (2013) maintains that excessive or inappropriate government regulations significantly hinder infopreneurship. as a result, stringent governmental regulations stagnate the development of infopreneurship (legas, 2015). these regulations often require registration of a business or company with a constituted authority in a country which is charged with the responsibility for regulating businesses, companies, and other economic activities. in south africa, the companies and intellectual property registration office (cipro) is responsible for registering new profit and non-profit businesses. the registration process involves registering the business to give it legal backing and automatically registering it as a taxpayer. in addition, new firms are required to obtain an income tax reference number. a business owner is obliged to complete all forms of registration within 60 days at the beginning of his or her business (akinyemi & adejumo, 2018). these processes tend to frustrate the development of emerging businesses in south africa. studies indicate that these regulatory procedures constitute one of the obstacles to start-ups and growth of medium and small-scale enterprises in south africa (mbonyane & ladzani, 2011; olawale & garwe, 2010).   a similar scenario is reflected in nigeria where high bureaucracy has been acknowledged as a factor that largely contributes to main challenges hampering entrepreneurship development. this is attributed to the complexity of the nature of laws regulating private enterprise, especially laws dealing with business registration and taxation in the country (onwuka et al., 2014). a business owner in nigeria must abide by a multiplicity of laws, including laws governing social, economic, and environmental issues, occupational health and safety, employment, criminality, as well as laws that are peculiar to one’s form of business. a prospective nigerian entrepreneur is required to get his or her company or business incorporated or registered by the corporate affairs commission (cac), which is the agency charged with incorporation of companies and business registration. it is an independent body responsible for implementing the provisions of the companies and allied matters act (cama), which is the primary law governing companies and businesses in nigeria (ndukwe & allison, 2021). other obstacles arise from procuring construction or business premises’ permit, such as high cost and extended period required to obtain letter of consent/certificate of occupancy, tax clearance costs, poor communication between the planning office and developers, costly application fee, inability of the design plan to satisfy established guidelines or standards, cost of paying professionals, lack of transparency in the approval process and so on (ajibola, 2019).   within the purview of regulations, amin et al. (2011) affirm that copyright, intellectual property rights, and privacy constitute obstacles to every information worker. copyright which empowers the copyright holder to exclusively enjoy moral (citation) and exploitation (financial) rights from an intellectual property tends to militate against the smooth operation of infopreneurship as such right restrains the manipulation of the intellectual property for commercial purposes except with the authorization of the copyright holder. an intending infopreneur may have to enter into an agreement with the copyright holder to be empowered to manipulate the content of an intellectual property for commercial purposes with a view to avoiding infringement of copyright. this may be a content or deposit agreement or license. the processes involved in reaching such an agreement with an author or a publisher, in cases where the right has been ceded to the publisher by an author, can be tedious and frustrating to an aspiring or practicing infopreneur.   societal challenges   legas (2015) and ngorora and mago (2013) opine that corruption prevents organizations from lending total support to intending infopreneurs, noting that, sometimes, these infopreneurs are expected to bribe officials to process funding applications. onwuka and ile-chika (2006) identified high corruption in government as one of the problems of entrepreneurship development in nigeria. aregbesola et al. (2019) observe that the practice of infopreneurship in nigeria is challenged by lack of structure in several public and private organizations, absence of coordination from several operators and the activities of fraudsters who take advantage of the internet to deceive people. similarly, lose and khuzwayo (2022) identified other challenges to include the existence of cyber risks, misinformation, legalization, reliability, and credibility issues. on the whole, findings from a study by ivwurie and ocholla (2016) indicate that lack of business experience and required mentorship, difficult legal registration process, high cost of office space, unpleasant behavior and poor attitudes of information consumers regarding payment for services, high rate of non-professionals involved in infopreneurship, irregularities in the quality of information products and services and absence of standardized professional ethics and pricing for information services as some of the barriers to operating infopreneurship in a developing economy.   methods                for our study, we adopted a descriptive survey research design. the study population comprised 175 librarians in 13 public university libraries in south-south nigeria. this population was sourced from the university libraries’ websites, library annual duty postings and from the information provided by the librarians who participated in the study. we used a purposive sampling technique to select the 13 public university libraries. we employed convenience sampling to choose 102 librarians engaged in infopreneurship; we identified these librarians through our preliminary investigation, observation, and interaction with the respondents. we designed a structured online questionnaire titled “librarians’ infopreneurship practice questionnaire (lipq)” (see appendix a) as our data collection instrument, to make data analysis easy and obtain responses in line with the study's specific objectives. the instrument was validated by two experts in the department of library and information science in niger delta university, bayelsa state, nigeria. reliability testing was not conducted on the instrument based on the strength of the expert notion that a valid test tends to be reliable (nworgu, 2015).   the questionnaire consisted of two parts: part a for demographic information, and part b with five sections to collect information dealing with purposes of engaging in infopreneurship (section a), methods of carrying out infopreneurship (section b), forms of infopreneurship practiced (section c), perceived benefits of practicing infopreneurship (section d), and challenges in practicing infopreneurship (section e). a two-point response category of “adopted (1)” and “not adopted (0)” was used for section b, whereas a two-point response option of “practiced (1)” and “not practiced (0)” was employed for section c. a four-point likert-type response category of “strongly agree (4)”, “agree (3)”, “disagree (2)” and “strongly disagree (1)” was adopted for sections a, d and e.   the researchers distributed 128 digital copies of the draft of the questionnaire to librarians in infopreneurship, through librarians' personal whatsapp accounts, whatsapp groups of the various university libraries, and whatsapp groups of the different state chapters of the nigerian library association to which the librarians belonged. whatsapp is a popular social media platform commonly used for sharing ideas, opinions and information among friends, professional colleagues, business partners, etc. while a librarian could have a personal whatsapp account for purposes of communicating ideas, opinions and information with other colleagues, various whatsapp groups are created among librarians for sharing ideas, opinions and information related to librarianship. the whatsapp groups could be institution-based such as a university library whatsapp group or job-based such as whatsapp group of cataloguing and classification librarians. all librarians who participated in the study were instructed to click on links to the online questionnaires sent to them and fill in. reminders and calls were issued and made to the respondents to fast-track the data collection process. a three-month period was used for data collection. out of the 128 questionnaires administered, 102 questionnaires were properly completed by the librarians and retrieved, producing a response rate of 97.69%. the distribution of respondents among institutions is shown in appendix b.   the data collected were analyzed using weighted mean and standard deviation. a mean of 0.50 was arrived at for sections that adopted a two-response category while a mean of 2.50 was obtained for sections that employed a four-point response category. thus, these means were adopted as criteria points for acceptance or rejection. the decision rule applied for interpretation of results from data analysis for sections that adopted a two-response category was that any questionnaire item with a weighted mean of 0.50 and above was regarded as adopted/practiced while an item with a weighted mean below 0.50 was considered as not adopted/not practiced. for sections that employed a four-point response category, the decision rule was that an item with a weighted mean equal to or above 2.50 was regarded as “agreed,” while any item with a weighted mean less than 2.50 was regarded as “disagreed.”   results   this section presents the results of data analysis in tables in line with the research questions earlier formulated to guide the study.   research question 1: what are the purposes of engaging in infopreneurship by librarians in public university libraries in south-south nigeria?   table 1 shows that the purposes of engaging in infopreneurship by librarians in public university libraries in south-south nigeria are to earn extra income, provide for post-retirement, meet unforeseen demands, and develop themselves. however, they do not engage in infopreneurship with the aim of socializing and gaining business experience.   table 1 mean ratings of responses on purposes of engaging in infopreneurship by librarians in public university libraries in south-south nigeria purposes of engaging in infopreneurship mean (x) standard deviation (sd) remarks to earn extra income 3.50 1.81 agreed to provide for post-retirement 3.22 1.20 agreed self-development 2.50 0.82 agreed socialization 2.42 0.60 disagreed to meet unforeseen demands 2.53 0.70 agreed to gain business experience 2.35 0.78 disagreed grand mean and standard deviation 2.75 0.99 agreed   research question 2: what are the methods of carrying out infopreneurship by librarians in public university libraries in south-south nigeria?   table 2 shows that the methods of carrying out infopreneurship by librarians in public university libraries in south-south nigeria are: leveraging, customizing, facilitating access to, and rendering round-the-clock delivery of information. the librarians do not adopt information repackaging as a method.   table 2 mean ratings of responses on methods of carrying out infopreneurship by librarians in public university libraries in south-south nigeria methods of carrying out infopreneurship mean (x) standard deviation (sd) remarks leveraging information 0.50 0.04 adopted customizing information 0.70 0.01 adopted facilitating access to information 0.55 0.21 adopted speeding up the flow of information 0.59 0.10 adopted repackaging information 0.48 0.05 not adopted round-the-clock delivery of information 0.51 0.01 adopted grand mean and standard deviation 0.56 0.07 adopted   research question 3: what are the forms of infopreneurship practiced by librarians in public university libraries in south-south nigeria?   table 3 reveals that the forms of infopreneurship practiced by librarians in public university libraries in south-south nigeria are: information brokerage, reprographic services, research-aid services, book vending, and internet services. information consultancy, abstracting/indexing, publishing, translation services, information repackaging, website creation/design, bibliography/directory compilation, publishing, records management, cataloguing, computer troubleshooting services and library automation are not forms of infopreneurship practiced by the librarians.   table 3 mean ratings of responses on forms of infopreneurship practiced by librarians in public university libraries in south-south nigeria forms of infopreneurship mean (x) standard deviation (sd) remarks information brokerage 0.51 0.03 practiced reprographic services 0.55 0.11 practiced information consultancy 0.49 0.02 not practiced abstracting and indexing services 0.48 0.05 not practiced publishing 0.42 0.05 not practiced translation services 0.13 0.07 not practiced information repackaging 0.05 0.02 not practiced research-aid services 0.53 0.11 practiced website creation/design 0.49 0.10 not practiced book vending 0.57 0.09 practiced bibliographic/directory compilation 0.34 0.08 not practiced records management 0.31 0.09 not practiced cataloguing 0.43 0.07 not practiced internet services 0.57 0.20 practiced computer troubleshooting services 0.27 0.06 not practiced library automation 0.40 0.05 not practiced grand mean and standard deviation 0.41 0.06 not practiced   research question 4: what are the perceived benefits derived from practicing infopreneurship by librarians in public university libraries in south-south nigeria?   table 4 indicates that the perceived benefits derived from practicing infopreneurship by librarians in public university libraries in south-south nigeria are enjoyment of extra income and financial independence, development of one’s knowledge, and enhancement of one’s sense of fulfillment. however, the practice of infopreneurship does not boost their self-confidence.   table 4 mean ratings of responses on perceived benefits derived from practicing infopreneurship by librarians in public university libraries in south-south nigeria benefits of practicing infopreneurship mean (x) standard deviation (sd) remarks it gives an extra revenue 3.17 1.10 agreed it boosts self-confidence 2.49 0.89 disagreed it builds one’s knowledge 2.51 0.77 agreed boosts one’s sense of fulfillment 2.50 0.57 agreed it brings financial independence 3.00 1.00 agreed grand mean and standard deviation 2.73 0.87 agreed   research question 5: what are the challenges in practicing infopreneurship by librarians in public university libraries in south-south nigeria?   table 5 reveals that the challenges in practicing infopreneurship by librarians in public university libraries in south-south nigeria are lack of adequate finance, business infrastructure, technical skill, and high rate of non-professional infopreneurs. however, they are not faced by such challenges as lack of passion/will, business experience, legal bottlenecks, activities of fraudsters and reluctance of clients to pay for services.   table 5 mean ratings of responses on challenges in practicing infopreneurship by librarians in public university libraries in south-south nigeria challenges mean (x) standard deviation (sd) remarks lack of passion/will 2.24 0.11 disagreed lack of business experience 2.49 0.45 disagreed lack of technical skill 2.56 0.98 agreed legal bottlenecks 2.41 0.97 disagreed activity of fraudsters 2.45 0.56 disagreed lack of adequate finance 3.49 1.33 agreed lack of business infrastructure 2.56 1.24 agreed high rate of non-professional infopreneurs 2.78 0.79 agreed reluctance of clients to pay for services 2.37 1.07 disagreed grand mean and standard deviation 2.59 0.83 agreed   discussion   this study shows that librarians in public university libraries in south-south nigeria engage in infopreneurship in order to make additional revenue, provide for life after retirement, meet unexpected needs, and to achieve self-development. this is in line with findings by akpena and duosakeh (2020), which indicate librarians engage in infopreneurship to make extra income, secure life after retirement, and meet identified demands. it also agrees with that of adetayo and hamzat (2021) which reveals that the reason library professionals involve themselves in infopreneurship is to make money. however, our findings contradict that of adetayo and hamzat (2021) which identifies socialization as one of the purposes. that socialization is not one of the reasons librarians engage in infopreneurship in our study could be because financial wellbeing is at the center of running the business. most of these librarians face financial pressure, consequently, they appear driven by the basic need to improve their finances and not necessarily to socialize with other individuals.   the research indicates that librarians in infopreneurship leverage, customize, accelerate information flow, ease information access, and deliver information without delay, but they do not repackage information. this result is partially consistent with weitzen (1998) who posits that individuals run infopreneurship by leveraging, customizing, and repackaging information, facilitating information access, and delivering immediate information. however, the finding that librarians in our study do not repackage information disagrees with weitzen who considers repackaging information as one of the operational methods of infopreneurs. the outcome that the librarians in our study do not repackage information may be because their information users consume mostly academic-based information, which is usually written in english and often does not need to be put in another form such as an annotated, translated, or abridged version. south-south nigeria is an anglophone region, so most information providers, including libraries and information centers, supply information materials mainly in english and this seems to be the reason these infopreneurial librarians do not engage in information repackaging. the result is also consistent with the observation of coulson-thomas (2000) that the rise in users’ need for expert and repackaged information causes infopreneurs to evaluate, sift, screen, and classify information to satisfy those demands. this outcome of the study could have turned out this way because many of these infopreneurs operate on university campus where students need information mainly for academic activities such as writing assignments, seminars, projects, theses, and dissertations, etc. thus, the librarians seem particularly interested in gathering or leveraging information and tailoring it to meet the specific needs of students through customization and supplying it through such means as social media and email to ensure quick delivery.   the research also reveals that infopreneurship enables the librarians to generate supplementary income, be financially independent, amass knowledge and be fulfilled. the result is in line with chandler (2007) and skrob (2009) who observe that infopreneurship offers its operator an easy or passive income. it also agrees with the outcome of the study by adetayo and hamzat (2021) which indicates that infopreneurship helps library professionals, who engage in it, to acquire knowledge, enjoy financial independence and increase their sense of fulfillment. the outcome is also in consonance with the finding of the study by ivwurie and ocholla (2016) which suggests that most library and information science graduates in lagos, nigeria and kwazulu natal, south africa, partake in information-based businesses for money making and survival.   the investigation also finds that librarians operate infopreneurship by brokering information, providing reprographic services, internet services, research-aid services, and vending books. the finding tallies with that of akpena and duosakeh (2020) which indicates that librarians in bauchi state of nigeria run infopreneurship mainly by selling books, brokering information, indexing information resources and publishing online. the result, however, runs contrary to this finding which nullifies the practice of publishing by the librarians under study. probably due to lack of finance required to acquire publishing facilities and ict incompetence, many infopreneurial librarians resort to reprographic services such as typing, photocopying, and printing and have not expanded their businesses to include publishing, either manual or digital. furthermore, the fact that students on university campuses particularly need academic-based information could have contributed to this finding.   finally, absence of sufficient funds, skill, business facilities and high presence of unprofessional infopreneurs in the business make it difficult for the librarians to engage in infopreneurship. the finding is consistent with samitowska (2011) and smith and beasley (2011) who observe that a dearth of funds constitutes the greatest obstacle to developing infopreneurship. it also agrees with the assertion of javier et al. (2012) that shortage of ict skills among african infopreneurs makes them unsuccessful as infopreneurs. it is also in agreement with small enterprise development agency ([seda], 2016) which notes that infopreneurs in developing countries are confronted by lack of infrastructure as they attempt infopreneurship. it is also consistent with javier et al. (2012) who observe that weak ict skills among african infopreneurs contribute to their failure in infopreneurship.   suggestions for further research   based on the knowledge gained from the findings of the study and from extensive literature review, the authors identified some gaps in literature that need to be filled. we hope that these can be addressed by conducting a study on the practice of infopreneurship by librarians in polytechnic libraries in south-south nigeria. in addition, research should be executed on the infopreneurial attitudes and skills among librarians in public university libraries in south-south nigeria. furthermore, an investigation should be carried out on the strategies adopted by librarians in overcoming challenges in operating infopreneurship in south-south nigeria.   conclusion   this study investigated the practice of infopreneurship by librarians in public university libraries in south-south nigeria. it has shown that these librarians engage in infopreneurship with a view to earning extra income, providing for post-retirement, meeting unforeseen demands, and developing themselves. they carry out the business by leveraging, customizing, facilitating access to and providing quick delivery of information. they operate this business through such ways as information brokerage, reprographic services, research-aid services, and book vending. while running this business, the librarians gain such benefits as extra income, financial independence, accumulation of knowledge and promotion of sense of fulfillment. despite these benefits, a number of obstacles frustrate the practice of infopreneurship by the librarians, including lack of adequate finance, technical skill, business infrastructure and high rate of presence of non-professional infopreneurs. in the light of the findings of the study, it is recommended that librarians should access small and medium enterprise loans from government to boost the capital of their infopreneurial businesses. practicing and would-be infopreneurs should undergo digital training to equip themselves with the technical competence required to successfully operate infopreneurship in the current digital information environment.   author contributions   jerry eyerinmene friday: conceptualization (lead), methodology, data analysis, writing – original draft (lead), writing – review & editing (equal) oyinkepreye sawyer-george: conceptualization (supporting), methodology (supporting), writing – review & editing (equal)   references   adetayo, a. j., & hamzat, s. a. (2021). infopreneurship and financial satisfaction among library professionals in tertiary institutions in ede, osun, nigeria. library philosophy and practice. (e-journal), 4749. https://digitalcommons.unl.edu/libphilprac/4749/   agbenyegah, a.t. (2013). challenges facing rural entrepreneurship in selected areas in south-africa. (unpublished ph. d dissertation). north-west university. http://hdl.handle.net/10394/9493   ajibola, a. e. (2019). democracy, good governance and development in nigeria. journal of public administration and policy research, 8(7), 80-88. https://doi.org/10.5897/jpapr2016.0378   akinyemi, f. o. &adejumo, o. o. (2018). government policies and entrepreneurship phases in emerging economies: nigeria and south africa. journal of global entrepreneurship research, 8(35). 7. https://doi.org/10.1186/s40497-018-0131-5   akpelu, f.e. (2019). job enhancement strategies for teachers’ performance in public senior secondary schools in rivers state. (unpublished phd thesis). university of port harcourt.   akpena, j. e., & duosake, h. a. (2020). exploring the practice of infopreneurship among librarians in bauchi state, nigeria. international journal of applied technologies in library and information management, 6(2), 32-33. http://www.jatlim.org/volumes/volume%206%20no%202/johnson.pdf   amakiri, d. s., & tatfeng, m. m. o. (2019). galvanizing the infopreneurship career development for office and information management professionals. iiard international journal of economics and business management, 5(4), 10-14. https://www.iiardjournals.org/get/ijebm/vol.%205%20no.%204%202019/galvanizing%20the%20infopreneurship%20career.pdf   amin, p., heikkilä, h., hyvämäki, t., korolyuk, a., parkkinen, j., zhang, z., & nangini, c. (2011). infopreneur: from nomads to knowmads. bitbang: entrepreneurship and services, 3, 6-20. http://lib.tkk.fi/reports/20 11/isbn9789526035734.pdf   aregbesola, a., van der walt, t., owolabi, s., idiegbeyan-ose, j., okocha, f., & eyiolorunshe, t. (2019). infopreneurship in a developing country: opportunities and challenges. iop conference series: materials science and engineering, 640(1), 012123. https://doi.org/10.1088/1757-899x/640/1/012123   auriol, e. (2013). barriers to formal entrepreneurship in developing countries. world development report. https://openknowledge.worldbank.org/handle/10986/16364   bureau for economic research. (2016). the small, medium and micro enterprise sector of south africa. ber research note, 1. http://www.seda.org.za/publications/publications/the%20small,%20medium%20and%20micro%20enterprise%20sector%20of%20south%20africa%20commissioned%20by%20seda.pdf   chandler, s. (2007). from entrepreneur to infopreneur: make money with books, ebooks and information products. john wiley & sons. chux-nyehe, g. c., & nwinyokugi, p. n. (2020). infopreneurship: values and implications on employment sustainability of smes in rivers state nigeria. international journal of research and innovation in social science, 4(3), 265-267. https://www.researchgate.net/publication/341527031_infopreneurship_values_and_implications_on_employment_sustainability_of_smes_in_rivers_state_nigeria   coulson-thomas, c. (2000). developing and supporting information entrepreneurs. industrial and commercial training, 32(6),231–235. https://doi.org/10.1108/00197850010354214   david, r., & dube, a. (2013). infopreneurial behaviour among university graduates in the information science faculty of a university in zimbabwe. engineering education and research and international conference on information technology, cape town, south africa: cape peninsula university of technology.   david, r. & dube, a. (2014). infopreneurial behaviour among university graduates in the information science faculty of a university in zimbabwe. infopreneurship journal, 1(2), 18–30. https://doi.org/10.13140/rg.2.1.1190.9601   dewah, p., & mutula, s. (2016). students’ perceptions of the infopreneurship education in the department of records and archives management at the national university of science and technology. south african journal of information management, 18(1), 717. https://doi.org/10.4102/sajim.v18i1.717   du toit, a. (2000). teaching infopreneurship: students’ perspectives. aslib proceedings, 52(2), 83-90. https://doi.org/10.1108/eum0000000007003   el-kalash, k. i., mohammed, s. b., & aniki, y. (2016). exploring infopreneurship for economic growth and sustainability in a developing economy: the nigeria perspective. journal of educational policy and entrepreneurial research, 3(7), 241. https://www.academia.edu/27429447/exploring_infopreneurship_for_economic_growth_and_sustainability_in_a_developing_economy_ the_nigeria_perspective     elumelu, t. o. (2014). african entrepreneurs will create the jobs africa needs. http://www.africanglobe.net/business/african-entrepreneurs-create-jobs-africa/   fisher, k., & kunaratnam, y. eds. (2007). between ourselves: the new generation of information and knowledge intermediaries. signposters, summarisers and synthesizers. report of a workshop organized by the institute of development studies at the university of sussex, brighton: united kingdom, may 2007. http://www.ids.ac.uk/files/between_ourselves_report.pdf   ivwurie, o. m., & ocholla, d. n. (2016). the status and development of infopreneurship in nigeria and south africa. education for information, 32, 323-339. https://doi.org/10.3233/efi-160980   javier, e., chevrolier, n., leenderste, m., quigless, m., & verghese, t. (2012). ict competitiveness in africa. world bank. http://documents.worldbank.org/curated/en/431511468193130115/pdf/882260 wp0box380petitiveness0summary.pdf   jennings, s. (1998). librarians, cybrarians, infopreneurs and webmasters. vine, 28(4), 13-17. https://doi.org/10.1108/eb040699   kazungu, i. (2021). infopreneurship as a career option among african university students. southern africa journal of communication and information science, 1, 66. https://journals.nust.ac.zw/index.php/sajcis/article/download/18/14   kehinde, b. t. (2021). influence of demographic factors on interest of lis undergraduates in infopreneurship: a comparative analysis. library philosophy and practice, 1-23. https://www.proquest.com/openview/5ba35dd7721bd55b4d9e870cfb3c16af/1?pq-origsite=gscholar&cbl=54903   lahm, r. j., & stowe, r. b. c. (2011). infopreneurship: roots, evolution and revolution. entrepreneurial executive, 16(1), 107–119. https://www.abacademies.org/journals/month-june-year-2011-vol-16-issue-1-journal-ee-past-issue.html legas, h. (2015). challenges to entrepreneurial success in sub-saharan africa: a comparative perspective. european journal of business and management, 7(11), 25-31. https://iiste.org/journals/index.php/ejbm/article/view/21804   lose, t., & khuzwayo, s. (2022). attitudes towards infopreneurship among information science students at a higher education institution. problems and perspectives in management, 20(1), 50-51. https://doi.org/10.21511/ppm.20(1).2022.05   mambula, c. (2002). perception of sme growth constraints in nigeria. journal of small business management, 40(1), 58-65. https://doi.org/10.1111/1540-627x.00039 masron, m. z., yatin, m. s. f., hussin, n., taib, k. m., & wah, y. b. (2017). infopreneurship: new career for university graduates. journal of informatics and mathematical sciences, 9(4), 1219–1226. https://doi.org/10.26713/jims.v9i4.996   mbonyane, b., & ladzani, w. (2011). factors that hinder the growth of small businesses in south african townships. european business review, 23(6), 550–560. https://www.emerald.com/insight/content/doi/10.1108/09555341111175390/full/html                 my guide nigeria. (2017). south-south region: your local guide to the south-south region of nigeria. https://www.myguidenigeria.com/regionalinfo/south-south-region   national bureau of statistics. (2016). labour force statistics. federal republic of nigeria.   ndukwe, o. j., & allison, p. (2021). good governance and ease of doing business in nigeria: problems and prospects. international journal of academic management science research (ijamsr), 5(1), 92. http://ijeais.org/wp-content/uploads/2021/1/ijamsr210113.pdf   ngorora, g. p. k. & mago, s. (2013). challenges of rural entrepreneurship in south africa: insights from nkonkobe municipal area in the eastern cape province. international journal of information technology and business management, 16(1), 3. https://www.semanticscholar.org/paper/challenges-of-rural-entrepreneurship-in-south-from-ngorora-mago/8c8d848242629c4b60c81a6af5bb0db59c99a733   nweze, c. a. (2018). utilization of ict facilities for quality teaching and learning in the 21st century: an overview of public secondary schools in rivers state. african journal of education, 12(2), 5-6. http://ajerduniport.com/wp-content/uploads/2019/05/j19.pdf   nworgu, b. g. (2015). educational research: basic issues and methodology. (3rd ed.). university trust publishers.   ocholla, d. n. (1998). information consultancy and brokerage in botswana. journal of information science, 24(2), 83–95. https://dx.doi.org/10.1177/016555159802400203 ocholla, d. n. (1999). information intermediaries in the next millennium: an agenda for action for the development of information consultancy and brokerage in africa. library management, 20(2), 105–114. https://dx.doi.org/10.1108/01435129910251584   ogbonna, a. u., & dare, l. a. (2019). infopreneurship potentials to financial downturn of the unemployed graduates in nigeria. (unpublished ph.d. seminar paper). university of nigeria.   olawale, f., & garwe, d. (2010). obstacles to the growth of new smes in south africa: a principal component analysis approach. african journal of business management, 4(5), 729–738. https://doi.org/10.5897/ajbm.9000434   olien, j. (2013). entrepreneurial competency. https://www.scribd.com/document/351418487/entrepreneurial-competency.pdf   olorunfemi, g. c. (2021). addressing the state of youth unemployment in nigeria. international journal of innovative psychology and social development, 9(4), 106. https://seahipaj.org/journals-ci/dec-2021/ijipsd/full/ijipsd-d-10-2021.pdf   onaade, o. j. (2012). information brokerage: an entrepreneurial approach to information services in nigeria. global journal of management and business research, 12(20), 12-20. https://globaljournals.org/gjmbr_volume12/2-information-brokerage-an-entrepreneurial.pdf   onwuka, e. m., & ile-chika, m. (2006). entrepreneurship development. conceptual, theoretical and practical framework, anambra, nigeria: jossy printing press.   onwuka, e. m., ugwu, k. e., & kalu, a. u. (2014). the effect of policy measures on entrepreneurship development analysis of nigeria’s experience. european journal of business and management, 6(23), 271. https://iiste.org/journals/index.php/ejbm/article/view/14850   ramugondo, l. s. (2021). infopreneurship as an employment option for library and information science students and graduates in south africa: a literature review. library philosophy and practice (e-journal), 5948, 5-7. https://digitalcommons.unl.edu/libphilprac/5948   samitowska, w. (2011). barriers to the development of entrepreneurship demonstrated by micro, small and medium enterprises in poland. economics and sociology, 4(2), 45. https://dx.doi.org/10.14254/2071-789x.2011/4-2/5   shapiro, c., & varian, h. r., (1999). information rules: a strategic guide to the network economy. harvard business school press.   smith, k., & beasley, m. (2011). graduate entrepreneurs: intentions, barriers and solutions. education + training, 53(8/9), 737. https://dx.doi.org/10.1108/00400911111185044   skrob, r. (2009). benefit of infopreneurship. www.info-marketing.org   taylor, d. p. (2020, nov. 5). 7 infopreneur business ideas to explore. https://www.fool.com/the-ascent/small-business/articles/infopreneur/   weitzen, h.s. (1998). infopreneurs: turning data into dollars. john wiley and sons.   appendix a questionnaire: librarians’ infopreneurship practice questionnaire (lipq)   please complete the following information about your background and current employment as a librarian and tick √ as appropriate.   part a: demographic information of librarians   name of university: status of university:                      federal university           state university   part b   section a: purposes of engaging in infopreneurship by librarians in public university libraries in south-south nigeria   items 1-6 show probable purposes of engaging in infopreneurship. indicate your purpose using the following options: sd (strongly agree=4), a (agree=3), d (disagree=2) and sd (strongly disagree=1).   s/n item sa (4) a (3) d (2) sd (1)   to earn extra income           to provide for post-retirement           self-development           socialization                         to meet unforeseen demands                         to gain business experience                         section b: methods of carrying out infopreneurship by librarians in public university libraries in south-south nigeria   items 1-6 show methods of carrying out infopreneurship. please indicate the method you adopt using the two options: adopted (1) and not adopted (0).   s/n item adopted  (1) not adopted (0)   leveraging information       customizing information       facilitating access to information       speeding up the flow of information       repackaging information       round-the-clock delivery of information       section c: forms of infopreneurship practiced by librarians in public university libraries in south-south nigeria   items 1-16 show various forms of practicing infopreneurship. please indicate the form you practice using the two options: practiced (1) and not practiced (0).   s/n item practiced (1) not practiced (0)   information brokerage       reprographic services       information consultancy       abstracting and indexing services       publishing       translation services       information repackaging       research-aid services       website creation/design       book vending       bibliographic/directory compilation       records management       cataloguing       internet services                    computer troubleshooting services       library automation              section d: perceived benefits from practicing infopreneurship by librarians in public university libraries in south-south nigeria   items 1-5 show likely benefits derivable from running infopreneurship. please indicate the benefit (s) you gain from operating infopreneurship using the following options: sd (strongly agree=4), a (agree=3), d (disagree=2) and sd (strongly disagree=1).   s/n item sa (4) a (3) d (2) sd (1)   it gives an extra revenue           it boosts self-confidence           it builds one’s knowledge           boosts one’s sense of fulfillment           it brings financial independence                          section e: challenges in practicing infopreneurship by librarians in public university libraries in south-south nigeria   items 1-9 reflect possible challenges individuals encounter in operating infopreneurship. kindly indicate the challenge (s) you are faced with by using the following options: sd (strongly agree=4), a (agree=3), d (disagree=2) and sd (strongly disagree=1).   s/n item sd (4) a (3) d (2) sd (1)   lack of passion/will                         lack of business experience           lack of technical skill           legal bottlenecks           activity of fraudsters           lack of adequate finance           lack of business infrastructure           high rate of non-professional infopreneurs           reluctance of clients to pay for services           many thanks for your time and input in making this survey a successful one.     appendix b distribution of respondents of the study   names of institutions of librarians respondents federal university otuoke 8 niger delta university 9 university of portharcourt 9 rivers state university 7 ignatius ajuru university of education 7 federal university of petroleum resources 8 delta state university 6 university of benin 9 ambrose ali university 8 university of calabar 10 cross rivers state university of technology 6 university of uyo 10 akwa-ibom state university 5 total 102       research article   occupational stress and job performance among university library professionals of north-east india   pallabi devi research scholar dept. of library & information science gauhati university guwahati, assam, india email: devipallabi.pd@gmail.com   prof. narendra lahkar former professor & head dept. of library and information science gauhati university guwahati, assam, india email: nlahkar@gmail.com   received: 17 aug. 2020                                                             accepted: 9 mar. 2021      2021 devi and lahkar. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29821     abstract   objective – the present study intends to investigate the occupational stress and job performance of university library professionals in north-east india. the main objective of the study is to assess the perceived level of occupational stress among library professionals and to identify any relationship between occupational stress and library professionals’ job performance. the study also aims to study gender differences regarding perceived occupational stress and job performance among library professionals as well as examine the influence of occupational stress on perceived job performance.   methods – descriptive survey method was used for the study. the sample population consisted of 123 library professionals from different parts of north-east india selected through convenience sampling technique. the survey consisted of a structured questionnaire divided into three sections: demographic information, self-perceived occupational stress, and self-rated job performance. descriptive and inferential statistical techniques including frequency, mean, standard deviation, t test, correlation co-efficient, and simple linear regression analysis were used to analyze data and interpret results with the help of the statistical package spss version 20.   results – the findings of the study established that a majority of library professionals working in university libraries of north-east india perceived a moderate level of occupational stress. it was also determined that male and female library professionals do not differ in their perception of occupational stress (p > 0.05), while a significant mean difference was found between male and female library professionals’ perceptions towards their job performance (p < 0.05). males scored themselves higher than females in terms of eight indicators of job performance: quality of work performance, ability to handle multiple jobs, communication skills, decision making, problem solving, technical skills, ability to perform competently under pressure, and contribution to the overall development of the library. regarding the relationship between occupational stress and job performance, the data indicated a significant negative relationship between occupational stress and job performance (r = -0.296, p < 0.01). in addition to this, intrinsic impoverishment, under participation, low status, and poor peer relationships were some of the factors negatively affecting the job performance of library professionals.   conclusion – the present study provides an insight about how occupational stress affects job performance of library professionals working in academic libraries. the findings revealed that there exists a modest but statistically significant negative relationship between occupational stress and job performance, which implies that an increment in the level of perceived occupational stress tends to influence library professionals’ self-perception of job performance negatively.     introduction   stress is a “perceived phenomenon associated with tension and anxiety. one is considered as being under stress when a situation is perceived as presenting an extra demand on the individual’s capabilities and resources” (nawe, 1995, p. 30). most often, stress can be defined:   as a way a human body reacts to stimuli from the environment; it can influence one’s psychological and physical condition. experiencing lower levels of stress can be stimulating, but being exposed to higher levels of stress for long periods of time may affect one’s health and cause negative emotions, feelings of pressure, anxiety, irritability, loss of appetite, and others, and finally bad performance at work. (petek, 2018, p. 129)   research has shown that stress in general exists in different forms; it may be psychological, emotional, social, occupational, or job related. over the past few decades, occupational stress, or job stress, has been emerging as a growing concern because we spend a lot of time at the workplace. blix et al. (1994) stated that “occupational stress is considered to be one of the ten leading work-related health problems” (p. 157). according to kaur and kathuria (2018):   occupational stress is a mental or physical tension or both, created and related to occupation and its environment which comprise of persons and objects from within and outside the work place resulting into absenteeism, lack of motivation and initiative, low productivity and service efficiency, job dissatisfaction and disruption of the smooth functioning of the organization. (p. 13)   employees’ efficiency in the organization is evidenced in terms of their performance at the workplace. job performance is an important criterion for organizational outcomes and success. ojo (2009) defined job performance as an extent to which the day-to-day work is being carried out. job performance can be defined as individual productivity in terms of both quantitative and qualitative aspects of the job. it indicates how well a person is performing their job and to what extent, the employee is able to meet their job duties as well as policies and standards of the organization.   several studies have pointed out that there are emerging issues in the library and information science profession that poses a threat or stress factor to library professionals, especially the academic librarians. these include “new expectations and the constantly changing role of librarians due to the dynamic nature of information and its delivery in the university system, triggered by the emergence of ict in the library and information practice” (ajala, 2011, para. 2). reena (2009) further supported this by averring that one of the realities of 21st century is that the library professionals are faced with constant challenges in their working environments. this is not only because of the role they have to play inside the libraries but also due to the increasing demands and expectations of the users within the libraries. moreover, as said by saqib saddiq, librarians were mostly unhappy with their workplace, often finding their job repetitive and unchallenging. they complained about their physical environment, saying that they were sick of being stuck between bookshelves all day as well as claiming that their skills were not used and that they felt they had very little control over their career (“librarians ‘suffer most stress’,” 2006). according to topper (2007), after years of doing the same tasks can be stressful and many librarians may feel that they are not being challenged in their work.   statement of the problem   the university library constitutes a vital element in any academic institution, and hence library professionals play a significant role in promoting teaching, learning, and research by providing information sources and services to students, researchers, and faculty. library staff should be concerned about the needs of the library users so that optimum utilization of the resources available can be achieved. the library staffs are the facilitators for the contact between users and resources. the work of the library professionals in service delivery is a key element that contributes to overall effectiveness of the organization. in extending their services as much as possible, stress should not be a hurdle in enacting efforts to serve the user community. research has already established that a high level of occupational stress may lead to a high level of dissatisfaction among the employees, a lack of job mobility, burnout, poor work performance, and less effective interpersonal relations at work (manshor et al.,  2003).   gender is another variable that can potentially affect the attitudes and perceptions of employees at the workplace. a few studies have already asserted that though the library profession is open for all genders, it is mostly female (carmichael, 1992; wiebe, 2004). “although librarianship is a female-dominated profession, both males and females within the profession suffer from work-related pressures based on the practices of gender bias” (greer et al., 2001, p. 127). several studies have shown that some differences exist in the level of dissatisfaction between male and female library staff. graddick and farr (1983), for example, pointed out that females often view themselves as being treated worse than males in the workplace. kirkland (1997) argued that most of the women in libraries suffer from a deprivation of inside information, challenging assignments, and recognition in their organizations. thus, several studies have discussed the gender-related issues of different aspects of work, but very few empirical studies directly examine gender-based differences among library employees’ perceptions regarding occupational stress and job performance at the university level.   it is in the light of these problems that the present study seeks to gain an insight on how occupational stress affects job performance of library professionals working in university libraries of north-east india. the study also attempts to explore gender differences among library professionals in their perceptions of occupational stress and their own job performance.   scope and limitation of the study   the study limits itself to measuring the perceived level of occupational stress and examining the relationship between perceived occupational stress and self-rated job performance of university library professionals in north-east india. the libraries attached to eight central universities, four state universities, and one institution of national importance located in various states of north-east india were picked up for the study. it should be noted that, north-east india is made up of total eight states: assam, arunachal pradesh, meghalaya, sikkim, manipur, mizoram, nagaland, and tripura. the geographical coverage of the study includes only assam, arunachal pradesh, meghalaya, sikkim, manipur, and tripura. the newly established universities including private universities are excluded from the study because these institutions are still in their infancy.   literature review   numerous bodies of literature have explored stress from different perspectives in different organizational settings and highlighted various stressors related to those situations. stress can be caused by many problems, such as problems at the workplace, financial problems, family problems, and problems in employees’ surroundings.   the health and safety executive (hse, 2001) defined stress as an adverse reaction to excessive or extreme pressures or demands that may be placed upon individuals. the pressure and demands that causes stress are known as stressors. according to hinkle (1974), the term stress denoted “force, pressure, strain or strong effort” exerted upon a material object or a person or upon a person’s “organs or mental powers” (p. 337). in this definition, individuals were acted upon by external forces.   occupational stress or work-related stress arises when work demands of various types and combinations exceed the person’s capacity and capability to cope with it. somvir and kaushik (2013) investigated occupational stress among library professionals in haryana state and reported that most of the librarians were frustrated because they were compared with clerical staff and had to work under the in-charge of a non-professional, who did not know about the duties and responsibilities of being a librarian. low salary, less freedom to make decisions related to budget, responsibility for loss of books, technological changes, and a lack of interaction among library professionals were some of the factors discouraging librarians to provide better library services. ratha et al. (2012) highlighted that workload, technology, shift work, user satisfaction, job insecurity, lack of administrative support, low status, inadequate salary, changing library environment, and reduced staff strength were some of the leading causes of occupational stress among library professionals in private engineering colleges in indore city. mahanta (2015) carried out a study to determine the sources of stress and magnitude of stress among the library professionals of central library, tezpur university, assam. the researcher found that the library employees in the study experienced organizational role stress to a moderate extent. the study identified that role ambiguity, inter-role distance, role stagnation, and role erosion were the powerful sources of stress among the library staff. in fact, role conflict, role ambiguity, and role overload have also been studied as antecedents of occupational stress (brief & aldag, 1976; ivanceyich et al., 1982).   gender seems to play a significant role in employees’ perception of work-related stress and job performance. jick and mitz (1985) stated that workplace stress is a major problem and suggested that gender may be considered an important demographic characteristic in the experience of stress. mosadeghrad (2014) revealed in his study that there was a strong correlation between the occupational stress of hospital employees and their gender. female employees reported higher occupational stress than their male colleagues. dina (2016) found that women suffered from stress more frequently than men owing to their dual responsibilities including work in the library and taking care of children or parents at home. oloruntoba and ajayi (2006) found that most male academic librarians have higher job performance than their female colleagues. oyeniran and akphorhonor (2019) stated that male librarians working in the university libraries in nigeria contributed more than their female counterpart in terms of performance. the gender difference had a positive influence on the job performance of librarians in the university libraries in nigeria.   much of the earlier literature on occupational stress emphasizes its effects on job performance. ali et al. (2011) found that there exists a highly significant positive relationship between job stress and job performance among banking employees (i.e., job performance was found to be better under stressful situations at workplace). in addition, all the three indicators of job performance—skills, efforts, and working conditions—had a positive direct relationship with job stress. conversely, ahmed and ramzan (2013) reported the existence of a significant negative relationship between job stress and job performance in the banking sector, which implied that both variables were inversely proportional to each other. when job stress was low, job performance increased, and when job stress was high, job performance decreased.   dina (2016) carried out a study to investigate the impact of stress on professional librarian’s job performance in nigerian university libraries. the findings showed that high amounts of stress can affect a professional librarians’ quality in terms of job performance in relation to their job demands and expectations. those professional librarians engaged in other activities besides their primary assignments for which they are employed were found more likely to be stressed than the others thereby affecting their job performance negatively. occupational stress was identified as one of the major problems impacting professional librarians’ wellbeing, commitment, and job performance.   kaur and kathuria (2018) conducted a study among 301 library professionals working in central libraries of 24 universities in punjab and chandigarh. the study revealed that occupational stress and job performance shared a negative but significant co-efficient of correlation with each other, which implies that as the level of occupational stress increased, the level of job performance decreased. ilo et al. (2019) investigated the relationship between job stress and job performance in university libraries in nigeria. the study identified low productivity, increased absenteeism, hypertension, job dissatisfaction, frustration, depression, and negative job attitude as negative effects of stress on the job performance of librarians. amusa et al. (2013) revealed in their study that a significant correlation exists between the work environment and job performance of librarians. moreover, the study highlighted that the librarians’ job performance was considered fair with regard to variables such as professional practice, contribution to the overall development of the library, ability to attend promptly to client’s request, and meeting minimum requirements for job promotion.   in summary, after reviewing all the relevant studies, occupational stress clearly exists in academic library environments and some of the common stressors affecting maximum the number of library employees include role overload, role conflict, role ambiguity, low status, lack of administrative support, and changing library environment. both occupational stress and job performance were found to be interrelated with each other, which imply that higher levels of occupational stress are related to lower levels of job performance and vice-versa. gender proved to be one of the significant factors influencing both occupational stress and job performance.   aims   the aims of the present study are presented here:   to assess the perceived level of occupational stress among library professionals working in university libraries of north-east india. to study the gender differences regarding perceived occupational stress and job performance among the library professionals. to identify the relationship between occupational stress and job performance. to examine the impact of occupational stress on job performance.   hypotheses of the study   based on the aims of the study, the following null hypotheses were formulated:   h0 – there is no significant difference between male and female library professionals regarding perceived level of occupational stress. h01 – male and female library professionals do not differ in their perception of job performance. h02 –no significant relationship exists between occupational stress and job performance. h03 – there is no significant impact of occupational stress on job performance.   methods   the population   a descriptive survey method was employed to collect primary data from library professionals who work full time and who have a minimum qualification of a diploma in library & information science in different universities of north-east india that were recognized by the university grants commission (ugc) of india. convenience sampling technique was used to gather data from a sample population of 123 library professionals who were easily available as well as willing to participate in the study from various states of north-east india. the breakdown of the sample population is given in table 1.   research instrument   a structured questionnaire was constructed in print and distributed personally to the participants, making quantitative data relatively easy to collect. the questionnaire was divided into three sections: demographic information, self-perceived occupational stress, and self-rated job performance. to measure the level of perceived occupational stress, we designed an occupational stress scale that was adapted from the occupational stress index (osi) of srivastava and singh (1984) and the organisational role stress (ors) scale of pareek (1983). the osi scale, a widely used scale in india was adopted by ratha et al. (2012) and chandraiah et al. (2003) in their studies. similarly, the ors scale, which is more specifically used in indian socio-cultural settings, was used by mahanta (2015) and jena and pradhan (2011) in their research studies. reena (2009) used both the osi and ors scales in order to construct an instrument especially useful for the library and information science profession. the scale used in the study consists of a total of 23 items on 11 dimensions of occupational stress: role overload, role conflict, role ambiguity, under participation, low status, poor peer relationship, personal inadequacy, strenuous working conditions, career stagnation, intrinsic impoverishment, and unreasonable groups & political pressures. brief descriptions of the dimensions of occupational stress used in the context of present study are stated here:   role overload arises when employees feel pressured because of added duties and responsibilities and lack the resources to perform them. role conflict refers to situations with conflict of role expectations. role ambiguity refers to a situation caused by lack of clarity or understanding about job expectations and responsibilities in the performance of a particular role. under participation is when there is a lack of one’s influence on the decision-making process of the organization. low status refers to a state of insignificance in the organizational as well as in the social system. poor peer relationship occurs when there is lack of mutual co-operation between coworkers in solving organizational problems. personal inadequacy refers to employees lacking the required skills to perform tasks expected to function within their roles. strenuous working conditions refers to a lack of comfort and safety in the work environment. career stagnation occurs when a employees feel a lack of engagement with their work or career. intrinsic impoverishment refers to monotonous nature of assignments, lack of ample opportunity to utilize one’s abilities and develop one’s aptitude, etc. unreasonable group and political pressure evolve from a situation where one is required to take a lot of decisions against his will or against formal rules and procedures under pressure.     table 1 breakdown of the sample population type of university no. of universities surveyed no. of respondents state universities 4 33 central universities 8 79 institution of national importance 1 11 total 13 123     table 2 descriptive statistics of occupational stress and job performance variable n minimum score maximum score mean standard deviation occupational stress 123 40 85 60.91 9.069 job performance 123 40 70 59.02 6.968     responses on all items were gathered through a five-point likert scale (strongly agree = 5, agree = 4, disagree = 3, strongly disagree = 2, and undecided = 1). the cronbach’s alpha coefficient of reliability was computed to verify the internal consistency of items used to measure a variable which was found to be .744. nunnally (1978) recommended at least .70 alpha coefficients for social sciences as acceptable.   similarly, to measure job performance, a self-assessment “job performance scale” was constructed that consists of a total of 14 job performance indicators (including completion of tasks on a given time, quality of work performance, ability to handle multiple jobs, communication skills, decision making, problem solving, technical skills, managerial skills, ability to perform competently under pressure, punctuality and regularity at work, meeting minimum requirements for promotion, interpersonal relationship with co-workers, contribution to the overall development of the library, and overall capacity to work)  rated on a five-point likert scale (very good = 5, good = 4, average = 3, poor = 2, and very poor = 1). the purpose of designing the scale was to gather input from the library professionals about their self-perception of how well they are performing their job. the reliability coefficient of the scale was found to be .905 using cronbach’s alpha method. statistical techniques like frequency, mean, standard deviation, t test, correlation co-efficient, and simple linear regression analysis were used to analyze the data and interpret the results with the help of the statistical package spss version 20. the descriptive statistics of the two variables selected for the study, i.e., occupational stress and job performance, are presented in table 2.   table 2 reflects that the mean and standard deviation of the total scores of perceived occupational stress is 60.91 and 9.069 respectively, whereas the mean and standard deviation of the total scores of self-rated job performance is calculated to be 59.02 and 6.968 respectively. the overall score ranges from a minimum of 40 to a maximum of 85 in the case of perceived occupational stress while the job performance scores ranges from a minimum of 40 to a maximum of 70. the table shows that the mean of both variables (i.e., occupational stress and job performance) seems to be identical; however, the range of scores was found to be greater in the case of occupational stress.   results   the results and their analysis are presented here and keeping in mind the aims of the study.   demographic information   the demographic data collected are presented in table 3 and describe the demographic characteristics of the sample population.   the demographic profile of the respondents in the present study demonstrated that with respect to responses on gender, 77 (62.60%) respondents were males while 46 (37.39%) were females. in response to age distribution, the highest number of respondents (35.77%) belongs to the age group of 31 to 40 years, which indicates a youthful working class. table 3 also shows that a majority of the respondents (55.28%) hold master’s degrees as their highest professional qualification and a plurality (45.52%) had work experience of above 15 years.   perceived levels of occupational stress   in order to assess the perceived levels of occupational stress, the mean (x) and standard deviation (sd) of the total scores of occupational stress obtained from the sum of the responses of all respondents were considered. therefore, the total scores of occupational stress were divided into three categories on the basis of their x and sd. following the principles of normal distribution, the scores falling above or equal to  x + sd, between x + sd and x – sd, and below or equal to x – sd were categorized as high level, moderate level, and low level, respectively.   level of occupational stress ·        high level = above or equal to 70 ·        moderate level = between 52 and 70 ·        low level = below or equal to 52     table 3 demographic profile of the respondents demographic variables frequency (n = 123) percentage (%) gender male 77 62.60 female 46 37.39 age group (in years) 21–30 12 9.75 31–40 44 35.77 41–50 34 27.64 51–60 33 26.82 highest professional qualification ph.d. 23 18.69 m.phil. 8 6.50 master’s degree 68 55.28 bachelor’s degree 16 13.00 certificate/diploma 8 6.50 years of work experience 0–5 19 15.44 6–10 36 29.26 11–15 12 9.75 above 15 56 45.52     both table 4 and figure 1 depict that a majority of library professionals surveyed perceived a moderate level of occupational stress (i.e., 63.41%), which consists of 47 males and 31 females. of the remaining library professionals, 18.69% perceived a low level of stress and 17.88% experienced a high level of occupational stress. gender differences with regard to perceived occupational stress   the results in table 5 clearly depict that t value for mean difference in occupational stress between male and female library professionals is -0.741, which is not significant (p > 0.05). the overall mean and standard deviation of male and female library professionals are found to be 60.44 (sd = 9.372) and 61.70 (sd = 8.581) respectively regarding their perceived level of occupational stress. this implies that the male and female library professionals working in university libraries do not differ in their perception of occupational stress. thus, the null hypothesis (h0) is accepted. the dimension-wise comparative analysis between male and female library professionals in terms of perceived occupational stress is presented in table 6.     table 4 perceived levels of occupational stress among library professionals levels levels n (%) gender n (%) occupational stress high level 22 17.88 male 14 63.63 female 8 30.43 moderate level 78 63.41 male 47 60.25 female 31 39.74 low level 23 18.69 male 16 69.56 female 7 30.43     figure 1 perceived levels of occupational stress.     table 5 significance of mean difference in perceived occupational stress of library professionals between male and female variable gender n mean standard deviation t value p value occupational stress male 77 60.44 9.372 -0.741 0.460 female 46 61.70 8.581             table 6 comparative analysis between male and female library professionals in terms of occupational stress dimensions dimensions of occupational stress gender n mean standard  deviation t value p value role overload male 77 9.60 1.982   0.743   0.459 female 46 9.33 1.921 role conflict male 77 2.99 0.939   -0.184   0.854 female 46 3.02 1.125 role ambiguity male 77 4.75 1.425   -0.455   0.650 female 46 4.87 1.276 low status male 77 4.47 1.586   -0.377   0.707 female 46 4.59 1.881 under participation male 77 5.10 2.043   -0.754   0.452 female 46 5.39 2.049 poor peer relationship male 77 7.71 1.891   0.053   0.958 female 46 7.70 1.860 personal inadequacy male 77 5.86 1.457   0.120   0.905 female 46 5.83 1.270 career stagnation male 77 3.01 1.082   -0.257   0.798 female 46 3.07 1.104 strenuous working conditions male 77 10.13 2.232   -0.882   0.380 female 46 10.50 2.288 intrinsic impoverishment male 77 4.45 1.667   -1.717   0.088 female 46 5.02 1.938 unreasonable groups & political pressures male 77 2.36 0.872   -0.180   0.858 female 46 2.39 0.745                                                         gender differences with regard to perceived job performance   table 7 reveals that the t value for the mean difference in terms of job performance between male and female library professionals is 3.163 (p < 0.05). there exists a significant mean difference in library professionals’ perception of job performance based on their gender. the overall mean and standard deviation of male and female library professionals are found to be 60.51 (sd = 6.522) and 56.54 (sd = 7.051) respectively. since the mean score of male library professionals is greater than their female counterpart, we can derive that the male library professionals perceived their level of job performance as better compared to the female library professionals. thus, the null hypothesis (h01) is rejected. table 8 shows the comparative analysis between male and female library professionals in terms of their self-perception towards job performance indicators.   from table 8, we can observe a statistically significant difference between the mean scores of male and female library professionals with regard to eight indicators of job performance: quality of work performance, ability to handle multiple jobs, communication skills, decision making, problem solving, technical skills, ability to perform competently under pressure, and contribution to the overall development of the library. the mean score of male library professionals is greater than their female counterpart in terms of these eight indicators of job performance. hence, it indicates that the male library professionals had a better self-perception than the female library professionals in the case of quality of work performance, ability to handle multiple jobs, communication skills, decision making, problem solving, technical skills, ability to perform competently under pressure, and contribution to the overall development of the library.     table 7 significance of mean difference in perceived job performance of library professionals between male and female variable gender n mean standard deviation t value p value job performance male 77 60.51 6.522 3.163 0.002* female 46 56.54 7.051             *significant at 0.05 level (2-tailed)     table 8 comparative analysis between male and female library professionals in terms of job performance indicators indicators of job performance gender n mean standard deviation t value p value completion of tasks on a given time  male 77 4.49 0.620 1.035 0.303 female 46 4.37 0.679 quality of work performance male 77 4.47 0.575 2.306 0.023** female 46 4.22 0.593 ability to handle multiple jobs male 77 4.40 0.712 2.326 0.022** female 46 4.09 0.755 communication skills male 77 4.26 0.715 2.333 0.021** female 46 3.93 0.800 decision making male 77 4.09 0.747 2.501 0.014** female 46 3.72 0.886 problem solving male 77 4.26 0.637 3.348 0.001* female 46 3.85 0.698 technical skills male 77 4.27 0.719 3.816 0.000* female 46 3.74 0.801 managerial skills male 77 3.92 0.900 1.100 0.274 female 46 3.74 0.880 ability to perform competently under pressure male 77 4.10 0.836 3.307 0.001* female 46 3.57 0.935 punctuality and regularity at work male 77 4.64 0.605 1.615 0.109 female 46 4.46 0.585 meeting minimum requirements for promotion male 77 4.06 0.978 1.053 0.294 female 46 3.87 1.024 interpersonal relationship with coworkers male 77 4.52 0.641 0.749 0.455 female 46 4.43 0.544 contribution to the overall development of the library male 77 4.56 0.573 2.243 0.027** female 46 4.30 0.662 overall capacity to work male 77 4.45 0.527 1.765 0.080 female 46 4.26 0.681                                                                     *significant at 0.01 (2-tailed) level; **significant at 0.05 (2-tailed) level     table 9 correlation between occupational stress and job performance variables occupational stress job performance occupational stress pearson correlation sig. (two-tailed)     n   1       123 -0.296**   0.001   123 job performance pearson correlation sig. (two-tailed)       n   -0.296**   0.001   123 1       123 **correlation is significant at the 0.01 level (two-tailed).     relationship between occupational stress and job performance   karl pearson’s coefficient of correlation was used to investigate the relationship between occupational stress and job performance in totality as well as through eleven dimensions of occupational stress. the level of significance of coefficient of correlation was calculated through two-tailed significant value.   a highly significant relationship was found from the above analysis between occupational stress and job performance through karl pearson’s coefficient of correlation, which means that the correlation is significant at 0.01 level. the results from table 9 reveals that there is a negative relationship that proves to be significant (p < 0.01) between occupational stress and job performance of library professionals (r = -0.296). hence the null hypothesis (h03) is rejected.   the objective of identifying the relationship between occupational stress and job performance of library professionals was further studied by focusing on the relationship of each dimension of occupational stress with job performance. table 10 demonstrates dimension-wise values of coefficient of correlation. it is evident from the table that intrinsic impoverishment has the strongest value of coefficient of correlation (r = -0.352) followed by under participation (r = -0.331), low status (r = -0.242), and poor peer relationship (r = -0.188). the remaining seven dimensions of occupational stress (role overload, role conflict, role ambiguity, personal inadequacy, career stagnation, strenuous working conditions, and unreasonable groups & political pressures) have shown no correlation with job performance. this means that the library professionals working in university libraries moderately experiencing these seven dimensions of occupational stress are not likely to bear a definite effect of it on their job performance.     table 10 relationship between dimensions of occupational stress and job performance dimensions of occupational stress coefficient of correlation (r) role overload -0.005 role conflict -0.051 role ambiguity -0.133 low status -0.242** under participation -0.331** poor peer relationship -0.188* personal inadequacy -0.096 career stagnation -0.087 strenuous working conditions -0.054 intrinsic impoverishment -0.352** unreasonable groups & political pressures -0.036   *correlation is significant at the 0.05 level (two-tailed) **correlation is significant at the 0.01 level (two-tailed)   impact of occupational stress on job performance   simple linear regression analysis was chosen to determine whether there is significant impact of occupational stress on job performance of library professionals working in university libraries. the present study was conducted to find out any association between the two variables selected (i.e., occupational stress and job performance). in this case, occupational stress was used to predict the dependent variable job performance. no doubt, there may be other parameters or factors affecting job performance that are not presented in the study because of its limitations. the value of r2 is found to be 0.088, which means that 8.8% of the variance in job performance can be explained by occupational stress. furthermore, the value of f = 11.629 (1,121) with significance level of p = 0.001 determined the linear regression model as statistically significant.   the criterion to assess the contribution of the predictor variable given by cohen (1988) was used in this study. according to this source, for linear regression models in behavioural sciences, the proportion of variance explained by the predictor variable an r2 value between 2% and 12.99% suggests a small effect size, a value between 13% and 25.99% indicates a medium effect size, and a value of 26% and greater suggests a large effect size. since the correlation coefficient in the present study is -0.296 and the r2 value is equal to 8.8% variance, the independent variable—occupational stress—is having a small but significant impact on the dependent variable job performance in a negative manner. from table 11, we can observe that occupational stress is able to explain the variance in job performance by the b value of -0.227. since the sign of regression coefficient value is negative, it indicates that as occupational stress increases by one unit, job performance decreases by 0.227 units. therefore, the null hypothesis (h04) is rejected.     table 11 simple linear regression analysis between occupational stress and job performance dependent variable independent variable unstandardized coefficients standardized coefficients t sig. b std. error beta     job performance (constant)   occupational stress 72.882   -0.227 4.108   0.067     -0.296 17.741   -3.410 0.000   0.001 r = -0.296     r2 = 0.088     adjusted r2 = 0.080   f = 11.629     sig. = 0.001     discussion   the main purpose of the present study was to investigate the occupational stress and job performance of university library professionals in north-east india. the results obtained from the current study revealed that a majority of the university library professionals perceived occupational stress to a moderate extent. this finding obtained from table 4 is in agreement with the result obtained from the study carried out by mahanta (2015) and wijetunge (2012), where the existence of a moderate level of work-related stress was reported among university library professionals. however, a few studies carried out by ogunlana et al. (2013), saddiq (2015), and agyei et al. (2019) were not in agreement with the prior result and reported a higher level of work-related stress. the variation in stress levels recorded in the previous studies may be a result of different organizational factors like conditions of service, size of the user community served by the library, status of library staff, financial availability, job security, career growth, and other reasons that might have brought about different perceptions about work-related stress among library professionals.   the result obtained from both tables 5 and 6 reveal that male and female library professionals do not differ in terms of perceived occupational stress, which is in line with the studies carried out by kaur and kathuria (2018) and somvir and kaushik (2013), wherein there was no significant difference found between male and female library professionals in terms of occupational stress. in spite of dual responsibilities at both home and workplace, women library professionals did not differ from their male counterpart in terms of their perception of occupational stress. this is contrary to the findings of ogunlana et al. (2013), who exposed that male librarians were more susceptible to job stress than female librarians despite the fact that both were working in the same environment. the data acquired from table 7 indicates that significant mean difference exists between male and female library professionals’ perception of their job performance. males scored themselves better than females in the areas of quality of work performance, ability to handle multiple jobs, communication skills, decision making, problem solving, technical skills, ability to perform competently under pressure, and contribution to the overall development of the library (table 8).   the result presented in table 9 show a significant negative relationship between perceived levels of occupational stress and job performance of library professionals. this finding gives further support to the literature that demonstrates a significant negative relationship between occupational stress and job performance, including those studies conducted by smith (2000), kaur and kathuria (2018), and nwadiani (2006). this is also in agreement with the claims of palmer et al. (2004), which stated that stress beyond an optimal point can lead to low productivity. similarly, hansen (2008) also claimed that stress is critical to maximizing one’s job performance. furthermore, mcgrath (1976) emphasized that job stress is considered a factor that may affect organizational effectiveness through lowering employee’s performance. the stressor intrinsic impoverishment has proved to be the most negative predictor influencing job performance (table 10). it implies that the monotonous nature of library jobs and the lack of ample opportunities to utilize the abilities and experience of library professionals independently can also yield negative outcomes on their self-perception of job performance. other stressors like under participation, low status, and poor peer relationship were some of the factors found to negatively affect the job performance of library professionals in university libraries of north-east india. furthermore, based on the findings of table 11, it was established that occupational stress has a statistically significant impact on job performance.   conclusion   based on the findings from the investigation, it can be concluded that occupational stress exists among university library professionals in north-east india, and majority of the professionals experienced stress up to moderate extent. though the level of occupational stress is moderate among library professionals, the study reveals significant negative relationship between perceived occupational stress and job performance. it implies that an incremental increase in the level of perceived occupational stress tends to influence library professionals self-perception of job performance negatively. stressors like intrinsic impoverishment, under participation, low status, and poor peer relationship were some of the factors negatively influencing their perception of job performance. male and female library professionals did not differ with regard to their perceived occupational stress. on the other hand, males scored themselves better than females on of eight indicators of job performance: quality of work performance, ability to handle multiple jobs, communication skills, decision making, problem solving, technical skills, ability to perform competently under pressure, and contribution to the overall development of the library. the results reveal a negative relationship between the two variables of occupational stress and job performance, but the current study cannot be generalized due to a limited sample size. further studies can be conducted with a larger sample size in order to realize the other organizational or socio-cultural factors that may have an effect on job performance.   author contributions   pallabi devi: conceptualization, data curation, formal analysis, investigation, methodology, resources, validation (lead), visualization, writing – original draft prof. narendra lahkar: conceptualization (lead), methodology (lead), project administration, supervision, validation, writing – review & editing   references   agyei, d. d., aryeetey, f., obuezie, a. c., & nkonyeni, s. (2015). the experience of occupational psychosocial stress among librarians in three african countries. library management, 40(6/7), 368–378. https://doi.org/10.1108/lm-11-2017-0122   ahmed, a., & ramzan, m. (2013). effects of job stress on employees job performance: a study on banking sector of pakistan. iosr journal of business and management (iosr-jbm), 11(6), 61–68. https://doi.org/10.9790/487x-1166168   ajala, e. b. (2011). work-related stress among librarians and information professionals in a nigerian university. library philosophy and practice (e-journal), 450. https://digitalcommons.unl.edu/libphilprac/450   ali, f., farooqui, a., amin, f., yahya, k., idrees, n., amjad, m., ikhlaq, m., noreen, s., & irfan, a. (2011). effects of stress on job performance. international journal of business & management tomorrow, 1(2), 1–7.   amusa, o. i., iyoro, a. o., & olabisi, a. f. (2013). work environments and job performance of librarians in the public universities in south–west nigeria. international journal of library and information science, 5(11), 457–461. https://academicjournals.org/journal/ijlis/article-abstract/07a6ff242586   blix, a. g., cruise, r. j., mitchell, b. m., & blix, g. g. (1994). occupational stress among university teachers. educational research, 36(2), 157–169. https://doi.org/10.1080/0013188940360205   brief, a. p., & aldag, r. j. (1976). correlates of role indices. journal of applied psychology, 61, 468–472.   carmichael, j.v. (1992). the male librarian and the feminine image: a survey of stereotype, status, and gender perceptions. library and information science research, 14(4), 411–446.   chandraiah, k., agrawal, s. c., marimuthu, p., & manoharan, n. (2003). occupational stress and job satisfaction among managers. https://www.ijoem.com/temp/indianjoccupenvironmed726-8600177_235321.pdf   coetzer, w., & rothmann, s. (2006). occupational stress of employees in an insurance company. south african journal of business management, 37(3), 29–39.   dina, t. (2016). the effect of stress on professional librarians’ job performance in nigerian university libraries. library philosophy and practice (e-journal). https://digitalcommons.unl.edu/libphilprac/1431   graddick, m. m., & farr, j. l. (1983). professionals in scientific disciplines: sex-related differences in working life commitments. journal of applied psychology, 68(4), 641–645.   greer, b., stephens, d., & coleman, v. (2001). cultural diversity and gender role spillover: a working perspective. journal of library administration, 33(1/2), 125–140. https://dx.doi.org/10.1300/j111v33n01_09   hansen, r.s. (2008). managing job stress: ten strategies for coping and thriving. https://www.quint-careers.com   hinkle, l. e. (1974). the concept of “stress” in the biological and social sciences. the international journal of psychiatry in medicine, 5(4), 335–357. https://doi.org/10.2190/91dk-nkad-1xp0-y4rg    hse. (2001). tackling work related stress: a managers’ guide to improving and maintaining employee health and well-being. health & safety executive.   ilo, p. i., amusa, o. i., chinwendu, n. a., & esse, u. c. (2019). job-related stress and job performance among librarians in university libraries in nigeria. library philosophy and practice (e-journal). https://digitalcommons.unl.edu/libphilprac/3650   ivancevich, j. m., matteson, m. t, & preston, c. (1982). occupational stress, type a behavior, and physical well-being. academy of management journal, 25, 373–391.   jena, p. and pradhan, s. (2011). impact of organisational role stress among library professionals of odisha: a study. pearl a journal of library and information science, 5(3), 1–8.   jick t. d. & mitz l. f. (1985). sex differences in work stress. academy of management review, 10(3), 408–420. https://doi.org/10.5465/amr.1985.4278947   kaur, h., & kathuria, k. (2018). occupational stress among library professionals working in universities of punjab and chandigarh. international journal of information dissemination and technology, 8(1), 22–24. https://www.ijidt.com/index.php/ijidt/article/view/8.1.5/399   kaur, h., & kathuria, k. (2018). occupational stress and job performance among university library professionals. educational quest: an international journal of education and applied social science, 9(1), 13–17. https://doi.org/10.30954/2230-7311.2018.04.02   kirkland, j. j. (1997). the missing women library directors: deprivation versus mentoring. college and research libraries, 58, 376–84. https://crl.acrl.org/index.php/crl/article/viewfile/15144/16590   bbc news. (2006, january 12). librarians ‘suffer most stress’. https://news.bbc.co.uk/2/hi/uk/4605476.stm   mahanta, k. (2015). assessment of stress in university library: a case study. in m. saikia, m. eqbal & d. pratap (eds.), library management: new trends and challenges (1st ed., pp. 33–41). academic publication.   manshor, a. t., rodrigue, f., & chong, s. c. (2003). occupational stress among managers: malaysian survey. journal of managerial psychology, 18(6): 622–628   mcgrath, j. e. (1976). stress and behavior in organizations. in m. d. dunnette (ed.), handbook of industrial and organizational psychology (pp. 1351-1395). rand mcnally.   mosadeghrad, a. m. (2014). occupational stress and its consequences. leadership in health services, 27(3), 224–239. http://tums.ac.ir/1395/02/22/occupational%20stress%20and%20its%20consequences.pdf-mosadeghrad-2016-05-11-10-24.pdf   nawe, j. (1995). work-related stress among the library and information workforce. library review, 44(6), 30–37. https://doi.org/10.1108/00242539510093674   nunnally, j. c. (1978). psychometric theory. mcgraw-hill.   ofoegbu, f., & nwadiani, m. (2006). level of perceived stress among lectures in nigerian universities. sa journal of industrial psychology, 33(1), 66–74.  https://www.researchgate.net/publication/292796793_level_of_perceived_stress_among_lecturers_in_nigerian_universities/citations     ogunlana, e. k., okunlaya, r. o. a., ajani, f. o., okunoye, t., & oshinaike, a. o. (2013). indices of job stress and job satisfaction among academic librarians in selected federal universities in south west nigeria. annals of library and information studies (alis), 60(3), 212–218. http://op.niscair.res.in/index.php/alis/article/view/2204   ojo, o. (2009). impact assessment of corporate culture on employee job performance. business intelligence journal, 2(2), 389–412. https://www.researchgate.net/publication/26844798_impact_assessment_of_corporate_culture_on_employee_job_performance   oyeniran, k., & akphorhonor, b.a. (2019). assessment of the influence of demographic factors on job performance of librarians in university libraries in south west, nigeria. research journal of library and information science, 3(2), 13–19. https://www.sryahwapublications.com/research-journal-of-library-and-information-science/volume-3-issue-2/3.php   palmer, s., cooper, c., & thomas, k. (2004). a model of work stress to underpin the health & safety executive advice for tackling work-related stress and stress risk assessments. counselling at work, winter, 2-5.  https://www.academia.edu/3814856/palmer_s_cooper_c_and_thomas_k_2004_a_model_of_work_stress_to_underpin_the_health_and_safety_executive_advice_for_tackling_work_related_stress_and_stress_risk_assessments_counselling_at_work_winter_2_5)   pareek, u. (1983). role stress scales manual. navin publications.   petek, m. (2018). stress among reference library staff in academic and public libraries. reference services review, 46(1), 128–145. https://doi.org/10.1108/rsr-01-2017-0002   ratha, b., hardia, m., & naidu, g.h.s. (2012). occupational stress among library professionals: a study at indore city. pearl a journal of library and information science, 6(1), 1–7. https://www.indianjournals.com/ijor.aspx?target=ijor:pjolis&volume=6&issue=1&article=001   reena, k.k. (2009). quality of work life and occupational stress among the library professionals in kerala [doctoral dissertation. university of calicut]. shodhganga. http://shodhganga.inflibnet.ac.in/handle/10603/1415   saddiq, s., & iqbal, z. (2019). environmental stressors and their impact at work: the role of job stress upon general mental health and key organisational outcomes across five occupational groups. psychology & behavioral science international journal, 11(2). https://doi.org/10.19080/pbsij.2019.11.555809   smith, a. (2000). the scale of perceived occupational stress. journal of occupational medicine, 50(5), 294–98. https://doi.org/10.1093/occmed/50.5.294   somvir, s. k. (2013). occupational stress among library professionals in haryana. international journal of knowledge management & practices, 1(1), 19–24. http://www.publishingindia.com/ijkmp/57/occupational-stress-among-library-professionals-in-haryana/210/1588/    srivastava, a. k., & singh, a. p. (1984). manual of occupational stress index. manovaigyanik parikchhan sansthan.   wiebe, t.j. (2004). issues faced by male librarians: stereotypes, perceptions, and career ramifications. colorado libraries, 31(1), 11–13.   wijetunge, p. (2012). work-related stress among the university librarians of sri lanka. journal of the university librarians association, sri lanka, 16(2), 125–137. https://doi.org/10.4038/jula.v16i2.5204   evidence based library and information practice evidence based library and information practice 2011, 6.1 85 evidence based library and information practice evidence summary a graduate degree in library or information science is required, but not sufficient, to enter the profession a review of: reeves, r., & hahn, t. (2010). job advertisements for recent graduates: advising, curriculum, and jobseeking implications. journal of education for library and information science, 51(2), 103-119. reviewed by: nazi torabi reference and instructional librarian university of western ontario london, ontario, canada email: ntorabi@uwo.ca received: 31 aug. 2010 accepted: 7 dec. 2010 2011 torabi. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to analyze the current state of the job market for recent mls or mlis graduates. design – content analysis of job postings. setting – online library or archival job advertisements published between 15 april, 2006 and 10 may, 2009 and collected from two national library publications (american libraries and library journal), two electronic lists (maryland's ischool discussion list and the archives and archivists list sponsored by the society of american archivists), two internet job banks (usajobs.gov and lisjobs.com), and several local chapters of library and archival organizations in the south atlantic region of the united states. subjects – 1,042 online library or archival job advertisements. salary data were obtained from 401 available online job advertisements. methods – the methodology for collection and content analysis of job ads was adapted from earlier studies, with slight modification wherever appropriate. the following criteria for selecting the ads were applied: • ad says "entry-level" • no mention of professional experience • no experience or duties impossible for entry-level librarians to gain • only ads that required an mls or mlis degree from an ala-accredited institution • part-time or temporary positions of less than nine months were excluded mailto:ntorabi@uwo.ca� evidence based library and information practice 2011, 6.1 86 the authors removed duplicated job postings and identified three major areas of content analysis. table 1 lists a brief summary of these areas and further sub-categories for each area. the content analysis was performed using a custom microsoft access database for data organization and storage and microsoft excel spreadsheet for data manipulation. spss was used for statistical analysis. main results – the two largest represented institution types for library positions were academic (63.6%) and public (17.5%). for archival positions, the academic (62.7%) and "other" (25.4%) institutions rank first and second. when the job ads were broken down into the position types, which were covering a wide range of responsibilities, the greatest numbers of entry-level library positions are found to be public service (52.2%) and technical services (23.9%) positions. the two largest represented position types in archives are technical services (50.7 %) and generalist (40.5%). table 1. the three major areas of job ads content analysis basic information personal attributes knowledge and experience • date • source of advertisement • position title • location (by state, census region, and division) • salary • type of institution • type of position • years of required nonprofessional experience • emotional intelligence (ei) competency (p. 115) o e1self awareness o e2self regulation o e3motivation o e4empathy o e5social skills • general o 2nd master degree o budgeting o foreign language o marketing o preservation o children’s programming o supervisory experience • technology o general it skills o automated library systems o next-gen library systems o computer/network hardware & software o digitization o programming/markup language o web design o web maintenance • library/archive specific o administrative o instruction o public/access services o reference o systems o technical services evidence based library and information practice 2011, 6.1 87 while average salaries increased slightly over the four years of study, there is a more significant increase in the salaries of positions posted in 2009. the highest average salaries were found to be $43k for archivists working for government and $60k for library positions in the "other" category. in addition, the number of entry-level positions has increased from year to year over this period. social competencies such as communication, collaboration and team work, and service orientation were the most emphasized traits for novice librarians and archivists. general information technology skills and knowledge of technical services were the most common skills required for both library and archive positions. overall, the entry-level job postings did not require non-professional experience. however, 13.6% of the library and 18.7% of the archival positions required supervisory experience. experience with preservation of physical objects and the knowledge of programming and mark-up languages were also common requirements for archives positions. instructional and reference experience ranks the second and third essential skills for librarians. conclusion – based on the research results, a graduate degree in library or information science is required, but not sufficient, to enter the profession. practical experience, either through internships, co-op programs, or parttime or full-time employment, is essential for new graduates seeking employment, but the majority of postings do not require a subject expertise, second master’s degree, or knowledge of a foreign language. since the job content analysis in this study only evaluated broad components of library services and archival operation, it might not provide sufficient data on new trends in the job market for the mls curriculum review. commentary this study offers useful information about the current state of the job market for new mls graduates. the authors took advantage of previous research in this area by adapting the methodology and research design. more specifically, they aimed to update the research conducted by sproles and ratledge in 2004 by focusing primarily on positions geared toward recent mls graduates. however, several significant flaws in the methodology and the data presentation affected the validity of the results. the authors retained almost the same research design, sources of data collection, and a standardized list of terms for content analysis from previous studies. while the previous studies, conducted by promis on emotional intelligence competencies (2008) and grimes and grimes on the academic library labour market (2008), proved to be useful to establish a solid framework for content analysis (pp. 105-106), the research suffers from the same shortcoming as other research in this area. a new study conducted by applegate (2010), has questioned the sources of academic job ads and how representative the data is for job content analysis studies. according to applegate, collecting job ad samples directly from institutions is time-consuming but proves to be more representative of all available job postings (p. 163). in addition, she has shown that the most appropriate academic library job aggregators are academic employment network and ala joblist (p. 167). similar analysis must be conducted for other areas in the field of libraries and archives before the most representative data source could be selected. in this current work, the source of advertisements was chosen without any good reasoning. the authors noted that almost half of the positions collected for this study were located in the south atlantic region of the u.s. (p. 109). therefore, one can argue that the job distributions by institution and position types may not be representative of all the job postings in the u.s. data analysis is not clearly described and needs more explanation. for example, as one of the selection criteria, the authors explicitly removed ads stating the need for professional experience. however, they found that evidence based library and information practice 2011, 6.1 88 institutions increasingly required entry level professionals to have some experience. also, job ads for less than nine months have been removed. this might raise some concerns about the validity of the data, since new graduates might strongly consider temporary positions. in addition, during economic downturn (the last period of study), there might have been more temporary or part-time positions available for new graduates. on several occasions, the way results are presented can be misleading for the readers. it also prevents drawing solid conclusions. the authors of the article do not provide any rationale for combining the results obtained for archival positions with those for library positions. they noted the wide variance between archive and library positions in terms of position and institution types (e.g., 0.7% for public archives vs. 17.5% public libraries) (p. 109). however, they still chose to aggregate the results and present the total number in table 2. in subsequent sections, the great difference between the two professions (library and archive) is again clear. for example, on page 110, the greatest number of entry-level positions for libraries and archives was found to be 52.2% for public service and 50.7 % for technical services, respectively. the results are aggregated in table 3 and again presented in percentage in figure 1, to which there is no reference in the text. this can be confusing for readers. on the same page, figure 2 is also redundant and provides the same information as table 2. the competencies and required knowledge and experience for the two professions (library and archive) also differ. yet, the results for both professions have been combined and presented as a total. for example, in table 10, knowledge of technical services was found to be a requirement for archival positions for 69.7% of cases while this number is 28.8% for library positions. the total of 39.9% is under-representative of what is required for archival positions and it is overrepresentative of the requirements for library jobs. there are many similar examples presented in tables 8, 10 and 11. the authors mentioned that only 38.5% of ads reported salaries (p. 111). it is important to determine whether this is a sufficient sample size to investigate the difference between salaries among different positions in various geographical locations. also, the authors conducted the anova test for salary means across position and institution types. similar analysis for salaries by u.s. census region and division would be ideal. one main conclusion by the authors is that job ads have shifted from print to electronic sources. although this is based only on anecdotal evidence and previous studies, it is notable for the profession as well as potential researchers in this area as electronic job ads are not always archived as effectively as print sources, making it difficult for other researchers to conduct historical research or refer to the raw data. the main implication of this study is to emphasize the importance of technical skills and practical experience to recent graduates of mlis programs. as information technology continues to advance, it should be assumed that experience with technology will appear even more in job ads. mlis programs should provide students with multiple practical opportunities, especially for those students who have entered the program directly from undergraduate school and with limited professional experiences in any area. also, mlis programs should integrate practical experience as part of the core curriculum to facilitate the students’ transition from school to the profession (sproles & ratledge, 2004). references applegate, r. (2010). job ads, jobs, and researchers: searching for valid sources. library & information science research, 32(2), 163-170. doi: 10.1016/j.lisr.2009.12.005 grimes, m., & grimes, p. (2008). the academic librarian labor market and the role of the master of library science degree: evidence based library and information practice 2011, 6.1 89 1975 through 2005. journal of academic librarianship, 34, 332-339. promis, p. (2008). are employers asking for the right competencies? a case for emotional intelligence. library administration & management, 22(1), 24-30. sproles, c., & ratledge, d. (2004). an analysis of entry-level librarian ads published in american libraries, 1982-2002. electronic journal of academic and special librarianship, 5(2/3). retrieved 24 jan. 2011 from http://southernlibrarianship.icaap.org/ content/v05n02/sproles_c01.htm http://southernlibrarianship.icaap.org/content/v05n02/sproles_c01.htm� http://southernlibrarianship.icaap.org/content/v05n02/sproles_c01.htm� / evidence based library and information practice evidence based library and information practice evidence based library and information practice 2011, 6.1 24 evidence based library and information practice article image-seeking preferences among undergraduate novice researchers laurie m. bridges teaching and engagement librarian oregon state university library oregon state university, corvallis, oregon email: laurie.bridges@oregonstate.edu tiah edmunson-morton, instruction and reference coordinator oregon state university archives oregon state university, corvallis, oregon email: tiah.edmunson-morton@oregonstate.edu received: 14 dec. 2010 accepted: 05 feb. 2011 2011 bridges and edmunson-morton. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – this study investigated the image-seeking preferences of university freshmen to gain a better understanding of how they search for pictures for assignments. methods – a survey was emailed to a random sample of 1,000 freshmen enrolled at oregon state university in the fall of 2009. a total of 63 surveys were returned. results – the majority of students indicated they would use google to find a picture. nineteen respondents said they would use a library, librarians, and/or archives. conclusions – the results indicate the majority of students in our study would use google to find an image for coursework purposes; yet the students who suggested they would use google did not mention evaluating the images they might find or have concerns about copyright issues. undergraduate students would benefit from having visual literacy integrated into standard information literacy instruction to help them locate, evaluate, and legally use the images they find online. in addition, libraries, mailto:laurie.bridges@oregonstate.edu� mailto:tiah.edmunson-morton@oregonstate.edu� evidence based library and information practice 2011, 6.1 25 librarians, archivists, and library computer programmers should work to raise the rankings of library digital photo collections in online search engines like google. introduction today’s undergraduate students have grown up in a social environment dominated by online interactions and have a seemingly infinite stream of information available ondemand. new undergraduate students enter the university with pre-conceived ideas about how to effectively and successfully conduct research for course assignments, yet university librarians and archivists who provide information literacy instruction often challenge these ideas. the american library association defines information literacy as the ability to “recognize when information is needed and […] the ability to locate, evaluate, and use effectively the needed information” (american library association, 1989). one often overlooked form of information students might use in their assignments and papers is images. in general, while most librarians do not routinely address visual literacy skills in their instruction sessions, archivists working with users on an individual basis to find pictures give the archivist a unique understanding of students’ visual information needs. in addition, many archivists have made their collections available on the web using software such as contentdm or the popular social networking site flickr. although students across all disciplines are using images more frequently, little published research exists about how students search for images or if they are aware of online archival collections. research into the image-seeking preferences of undergraduates is an important first step toward integrating visual literacy into information literacy instruction at the university level. literature review in the introduction of her 2006 study on information seeking behaviour in digital image collections, matusiak summarized there had been “very little research on the use of digital image collections, particularly on user behaviour or the process of seeking images.” (p. 479) her finding still holds true in 2011, but the importance of researching users and their image-searching preferences and behaviours grows more important as the ability to insert images into assignments, projects, and presentations becomes increasingly easier. in our review of the library and archives literature we found no studies that investigated undergraduate behaviour or preferences related to image searching. undergraduates and information searching in 2009, head and eisenberg published two important reports examining how college students seek information in the digital age. their first report, finding context: what today’s college students say about conducting research in the digital age, shows that although students know “finding context” is critical for comprehensive research, they characterize it as laborious and frustrating. in their second report, lessons learned: how college students seek information in the digital age, the authors build on these initial findings, showing that when students conduct “everyday life research, nearly all the respondents used google, wikipedia, and friends for finding context” (p. 32). additionally, all of the students surveyed used a combination of “course readings, library resources, and public internet sites such as google and wikipedia, when conducting course-related research—no matter where they were enrolled, no matter what resources they had at their disposal” (p. 32). head and eisenberg (2009b) also found students want to find information quickly, regardless of its relevance, at times selecting only the pieces of evidence that support their claims. in finding context head and eisenberg (2009a) discovered that when most students began researching, they used strategies that brought them luck in the past; consequently, while students might consult library evidence based library and information practice 2011, 6.1 26 resources, if they did not immediately find something that met their needs after entering a term into a library database search box, they turned to an online search engine. correspondingly, in their second report head and eisenberg (2009b) found that students “have developed sophisticated problemsolving strategies that help them to meet their school and everyday needs as they arise,” but that “many students’ research methods appear to be far from experimental, new, developmental, or innovative” (p. 34). head and eisenberg (2009a, 2009b) do not specifically address image-searching behaviour or preferences in their two studies, but we believe their findings can be generalized to various formats of information, including visual information. this is important to bear in mind as recent studies on image-searching behaviour and preferences reveal the use of images for personal and academic use is increasing (davison, 2009; green, 2006; hartley, 2007; jansen, 2008; krause, 2008; malkmus, 2008; mccay-peet & toms, 2009; shonfeld, 2006; tally & goldenberg, 2005). students: libraries and archival digital collections regardless of their age or academic level, users want a clear path for accessing digital content; this is a critical concern for many university libraries that are directing dwindling budget resources toward increasing access to online image collections and historic materials. studies by krause (2008) and davison (2009) looked at the variety of ways archives and special collections are working to digitize previously uncatalogued or inaccessible historical materials, putting them online in digital collections and, consequently, dramatically increasing access points for onsite and offsite researchers. continuing the analysis of online retrievability and accessibility, kathleen fear (2010) investigated the usefulness of metadata from the user perspective. she examined the information needs of non-expert users and how assigned metadata might compliment those needs, concluding that librarians and archivists ought to look at the larger issues concerning image use. specifically, she recommends looking at whether “the information is presented in a way that supports exploration of individual images and collections as a whole” (p. 52). she also suggests sharpening our understanding of what users need, saying it is “a necessary foundation for determining what balance of process is needed to allow users to find, access, and use the products made available to them” (p. 52). clearly, while online collections have successfully matched people with items that may meet their needs, librarians and archivists need to invest time researching the experiences of specific members of their user communities, looking at both successes and failures to ensure digital resources are being used to their fullest research potential. silipigni, connaway, and dickey (2010) compiled a comprehensive report synthesizing twelve studies commissioned by online computer library center (oclc), research information network (rin), and joint information systems committee (jisc), which provides an analysis of the informationseeking behaviours and needs of library users. they conclude that libraries need to provide users with coherent and consistent access to information, consider a greater variety of digital formats and content, and function more like search engines such as google. aim the primary purpose of this study was to investigate undergraduate novice researchers’ image-searching preferences when looking for pictures to use in assignments. our study originated in a desire to understand the habits and needs of novice researchers in order to determine how librarians and archivists might better meet those needs. additionally, our professional concerns about library system usability and item retrievability coincided with a flood of new digital content from libraries and archival collections. evidence based library and information practice 2011, 6.1 27 the secondary purpose of the study was to discover the specific web sites or search engines students had used in the past and which web sites or search engines they were likely to use in the future to find images. methods we developed a pilot survey in spring 2009 and administered it in-person to two lowerdivision and two upper-division liberal arts classes. the survey questions were revised based on student feedback. after creating a draft survey, we worked closely with the senior faculty research assistant for questionnaire design at the oregon state university research center (osurc) in the department of statistics to further refine our questions and develop a web-based survey (appendix a). the osu registrar’s office generated a random sample list of 1,000 students out of the 4,376 enrolled freshmen at oregon state university. the confidential survey was administered by the osurc in november 2009 and was open for two weeks. all responses were anonymous. the survey was estimated to take 10 minutes to complete. the survey contained both open-ended and close-ended questions about image-searching practices, along with two questions about demographics (age and academic discipline). the osurc sent an email to the selected freshmen announcing the survey two days before the survey was distributed; approximately one week after the survey was distributed the osurc sent a reminder email to the non-respondents. our objective was to study the preferences and behaviours of novice researchers, therefore only freshmen were selected to participate in the study. we assumed the majority of freshmen were new to universitylevel research and had likely received little to no formal instruction from university librarians, archivists, or professors on image searching. additionally, these freshmen were in the early stages of oregon state university’s baccalaureate core curriculum (bacc core), which represents what the osu faculty believes is the foundation for students’ further understanding of the modern world. informed by natural and social sciences, arts, and humanities, the bacc core requires students to think critically and creatively, and to synthesize ideas and information when evaluating major societal issues. importantly, the bacc core promotes understanding of interrelationships among disciplines in order to increase students’ capacities as ethical citizens of an ever-changing world. (oregon state university, 2010) with the bacc core’s focus on interdisciplinary studies, it was likely that freshmen would use a variety of sources and material types for their assignments in the future, regardless of their selected major. open-ended questions in our survey students were asked to respond to three open-ended questions. we intentionally used language in the first question of the survey to link it to research students might do for an actual course assignment, asking them to discuss a specific assignment-driven task for finding a historic image. the questions included: • for this question we’d like you to imagine you are doing a project or writing a paper for a class. your instructor has asked you to include a picture of people logging in oregon forests in the 1930s or 1940s. please explain your search process for finding a picture. you might start your explanation with the phrase, ‘i would begin my search for the picture by’ • please list any other sites you have used to search for images and indicate whether you used the site to search for personal images or to search for images for an assignment. • is there something librarians can provide that would help you search for online images? evidence based library and information practice 2011, 6.1 28 the respondents’ answers to the open-ended uestions were analyzed using conceptual ordering (strauss & corbin, 1998). we began by using an open-coding procedure to identify the information resources students named in their preferences for finding an image (for example, “google” and “books”). we then used axial coding to map the identified sources into categories that related to the point in the search process the resource would be used (for example, “first step in the research process,” “only step in the search process,” and “last step in the research process”). closed-ended questions the closed-ended questions in the survey began with this statement: “now thinking about searching for any image online, please indicate in what ways you have used each of the following websites to search for images.” the survey then presented the name of each site, along with the url for google, google images, yahoo, wikipedia, facebook, osu libraries’ digital collections, and osu libraries’ archives. as an example, the first closed-ended question appears in table 1; the survey in its entirety is found in appendix a. the results were analyzed via a frequency analysis (appendix b), which was conducted by the oregon state university research center. the results provide further insight into the responses of the open ended questions and will be referenced in the results/discussion. results and discussion overview sixty-three freshmen responded to the survey as a whole, a return-rate of 6%, and 61 responded to the first open-ended question. although this is a limitation of the study, we feel the answers given may be of interest to librarians as a springboard for more research in this area. because of the low response rate, the closed-ended question results in particular should be viewed with some caution, as this small sample cannot be generalized to all freshmen at oregon state university however, the frequency analysis reveals further insight and the results are discussed throughout this section. thirty-three of the respondents, comprising 52% of the total, indicated they were females, while the remaining 30 respondents, 48% of the total, indicated they were males. students also indicated their academic discipline: six in agricultural science, seven in business, seventeen in engineering, one in forestry, one in health and human sciences, one in liberal arts, one in science, two in honors, and three were “undecided.” table 1 first closed-ended question of the survey now thinking about searching for any image online, please indicate in what ways you have used each of the following websites to search for images. (check all that apply) a. flickr (flickr.com) � heard of this site but have not used to search for images � have used for an assignment to search for images � have used for personal/pleasure to search for images � likely to use in future for assignments if searching for images � likely to use in future for personal/pleasure if searching for images � not at all familiar with this site � don’t know evidence based library and information practice 2011, 6.1 29 google when the freshmen participants were asked to describe how they would find an image of oregon loggers from the 1930s or 1940s they overwhelmingly responded “google.” of the 61 students who responded to the question, 41 specifically mentioned google or google images as part of their search strategy. twenty-one of the respondents listed google and/or google images as their only search strategy, as conveyed in these quotes: • “i would begin my search for the picture by going to google [sic] image search and searching oregon [sic] forest 1930 1940” • “i would first go to google [sic] images. then i would type in ‘logging industry 1930’. i would search through those pictures and if i didn’t find any good ones i would simply change the words i used. i only ever use google.” • “i would begin my search for the picture by going to google [sic] and type in loggin [sic] in oregon forests in the 1930s to 1940s. just in case some one [sic] has already written a paper on this topic.” • “i would look for a picture first on the internet. using google images, most likely. see what i can find. and i would probably find something there. so my search would end.” furthermore, when students described using google as one part of their overall search strategy, they almost always started their search with google. for example: • “i would begin my search for the picture by typing into google, ‘1930 logging oregon forests pictures.’ i would find a website that matches this and has pictures. i would read the content to confirm it is what i am looking for, then view the images. if i could not find a picture that way, i would click on ‘images’ to try to find an accurate depiction. if the internet was not helpful, i would search for nonfiction books that are related to logging in oregon, then look specifically for the dates.” • “i would begin my search for the picture by going to google [sic] images and searching for ‘loggers in oregon + 1930s +1940s.’ if that didn’t yield what i needed i would go look for pictures in some books if possible from the library.” • “first, i would think of some keywords that would help me find the type of picture that i'm looking for. in this case, i would pick out ‘oregon’ and ‘logging’ and enter them into google images to see if i can find any images i want. if i look through a couple pages and don’t find anything that i want, then i would try to change my keywords to be more specific if possible and repeat. i would then try another image website in order to try and find the correct image.” finally, google was the only search engine directly named by students, other than one mention of ask.com; however, 12 students mentioned searching the “internet” or a “website” without naming a particular search engine. in line with the findings of head and eisenberg (2009a), participants in our study preferred to conduct their image research for an assignment using a search strategy that had worked for them in the past; in this case, google. when asked where they would begin their hypothetical search for a historic logging image, 41 of the participants indicated they would use google or google images as part of their research strategy. moreover, 21 of those 41 students, approximately one-third of the total study participants, stated they would only use google or google images. this correlates to the frequency analysis from the closed-ended questions of our survey, which showed 54 respondents had used google and 55 had used google images to search for images for assignments in the past. given the preference for using google and google images as a first step in the research evidence based library and information practice 2011, 6.1 30 process, are libraries prepared to support students in their use of google? librarians and archivists frequently voice their professional concerns at library conferences about undergraduate students’ preference for “nonlibrary” resources and reliance on popular search engines for research projects. it is our opinion that dismissing google, when it has proven successful for patrons in the past, only serves to undermine the librarian’s role in the research process. further, because our findings showed students turn to search engines first and then look to librarians for assistance afterward, we believe it is imperative that librarians, archivists, and library computer programmers understand how search engines, like google, index digital image collections and that these professionals work toward improving the collections rank in these search engines. students will likely focus on the content of sites appearing in the first two or three pages of google search results (nichols, rowlands, clark, & williams, 2011). also, by taking advantage of popular photo sharing sites like flickr, libraries can markedly improve the ranking of their images in google. books sixteen of the 61 open-ended question respondents indicated they would look in books when searching for an image of loggers from the 1930s and 1940s. the majority of these respondents indicated finding books was the last step of a multi-step search strategy, preceded by searching the internet and/or google. • “i would begin my search for the picture by first looking up logging in oregon + 1930s in google [sic] images. i would look through the pictures and if they weren’t what i expected, i would try another search engine, after trying the internet i would turn to books or another source.” • “googling [sic] it on the internet, and if that didn’t work look up environmental science books or logging books that could possibly have that in it.” • “i would begin my search for the picture by looking on the internet. if i didn’t find it there i might try and find a picture from a book.” approximately one-quarter of the respondents said they would look in books for the images. it is possible students were referring to textbooks, although only one of the students actually used the word “textbook” in his/her response. library / librarians / archives nineteen of the 61 open-ended question respondents indicated they would use a librarian and/or the library and archives in their search for images of loggers from the 1930s and 1940s. similar to the previous discussion of search strategies that included books, the majority of respondents included the library or librarians as part of a multi-step search strategy, usually preceded by searching the internet or google. • “i would begin my search for the picture by searching google [sic] for it. if that did not turn up any acceptable results, i would go to the library and browse through potential books and then photo copy/scan the desired picture out of the book.” • “i would definitely use the internet to find a picture, if that didn't work obviously i would have to find something at the library. where which [sic] i would ask a librarian to direct me to where i could find a document with pictures to use for the project.” • “i would begin my search for the picture by looking for images online. failing this i would look in books pertinent to the subject. if that didn’t work i would ask friends for help or ask library staff.” the words students use to describe their search process reveals a thinly veiled certainty that, in all likelihood, the image actually would be found online and not in a library. for example, a statement that began, “i would evidence based library and information practice 2011, 6.1 31 definitely use the internet” concluded, “if that didn’t work, obviously i would have to find something at the library.” although most students included the library as a final step, five students indicated they would start their search with the library or archives. • “i would begin my search for the picture by asking the librarian because i have no idea where to find that.” • “i would begin my search for the picture by asking someone that works at the library for help. i could also search online for where the picture may be...in the oasis archives.” • “i would begin my search for the picture by searching the osu database and archives for photos of people logging in oregon forests in the 1930s or 1940s.” • “i would begin my search for the picture by using the internet to find where books on that subject and time period are located in the library.” • “i would begin the search for my picture by checking the library catalog for books relating specifically to logging, then within them 1930-1940.” because only 19 of the 61 respondents mentioned a library, librarians, or archives, and because the majority of those students described using them as a last resort after searching the internet, we conclude that libraries and archives have two main options for increasing novice-researcher traffic to their online images and repositories. first, they can promote and advertise their services heavily to increase awareness of collections. secondly, they can work to raise the rankings of their images in search engines such as google and google images. from a financial standpoint, perhaps the most cost-effective method is a subscription for a flickr professional account, which currently has a $24.95 annual fee. despite staff time for scanning and image uploading, the popularity of the flickr site will help library collections rise to the top of google searches without major expenditures. wikipedia interestingly, in the frequency analysis of the closed-ended questions, 32 students, or just over one-half of all respondents to this question, indicated they had used wikipedia in the past to find images for assignments, yet none mentioned wikipedia in their search strategy for finding logging images in the open-ended question. head and eisenberg (2009a) have suggested that novice researchers are likely to use search strategies that have worked in the past, so it is worth highlighting that none of the participants mentioned wikipedia in their search strategy. this point is an interesting question for future research. we hypothesize that students did not mention wikipedia because they were told by high school teachers and librarians they should not use wikipedia for academic research. although students might admit having used it in the past, perhaps they are reluctant to say they will use it in the future, especially when addressing an assignment proposed by a librarian or professor. or, perhaps they simply realize google images will provide hundreds of images for the task, while wikipedia may serve up only one or two. copyright although we did not ask questions in our survey about copyright, we believe it noteworthy to mention that none of the students mentioned concern for copyright when searching for or finding an image for use in an academic paper. future research into students’ understanding of copyright should be conducted and librarians may want to consider adding information about copyright as it applies to images to their standard information literacy instruction. conclusion we began our research by exploring questions pertaining to image-seeking and novice historical researchers. however, in the process of reviewing the literature and having anecdotal conversations with colleagues and discipline-specific faculty members, we began evidence based library and information practice 2011, 6.1 32 to realize our burgeoning research about image-seeking had broader implications for instruction librarians across all disciplines. our study of the image-searching preferences and behaviours of freshmen raised more questions than it answered, yet the questions offer excellent opportunities for future research by librarians. for example, “what do faculty members tell students about finding and citing images?” and “how often are students using images in their assignments?” although our study raised a lot of questions, it did answer the question “where do students go to search for images for assignments?” resoundingly, the answer was “google.” however, since only 6% of the invited participants responded, the results of this study need to be treated with caution until more research is done in the area of searching for images. in the meantime, we recommend that staff responsible for the development of library and archival online image collections work toward raising the rankings of their images in google. in addition, because the respondents in our survey did not mention evaluating the images they might find or concern for copyright, we suggest librarians begin to consider integrating visual literacy instruction into standard information literacy instruction for undergraduates. references american library association, presidential committee on information literacy. (1989). presidential committee on information literacy: final report. retrieved from http://www.ala.org/ala/mgrps/divs/ac rl/publications/whitepapers/presidenti al.cfm davison, s. (2009). if we build it, will they come? strategies for teaching and research with digital special collections. rbm, 10(1), 37-49. fear, k. (2010). user understanding of metadata in digital image collections: or, what exactly do you mean by "coverage"? american archivist, 73(1), 26-60. flickr. (2010). more about the commons. retrieved december 14, 2010, from http://www.flickr.com/commons/#faq green, d. (2006). using digital images in teaching and learning: perspectives from liberal arts institutions (academic commons). georgetown, tx: national institute for technology and liberal education and wesleyan university. retrieved july 1, 2010, from http://www.academiccommons.org/fil es/image-report.pdf hartley, d. (2007). use and users of digital resources: a focus on undergraduate education in the humanities and social sciences. educause quarterly, 4, 12-24. head, a. j., & eisenberg, m. b. (2009a). finding context: what today's college students say about conducting research in the digital age (project information literacy). seattle, wa: university of washington. retrieved july 1, 2010, from http://projectinfolit.org/pdfs/pil_prog ressreport_2_2009.pdf head, a. j., & eisenberg, m. b. (2009b). lessons learned: how college students seek information in the digital age (project information literacy). seattle, wa: university of washington. retrieved july 1, 2010, from http://projectinfolit.org/pdfs/pil_fall2 009_year1report_12_2009.pdf holloway, i. (1997). basic concepts for qualitative research. london: blackwell science. jansen, b. j. (2008). searching for digital images on the web. journal of documentation, 64, 81-101. krause, m. g. (2008). learning in the archives: a report on institutional practices. http://www.ala.org/ala/mgrps/divs/acrl/publications/whitepapers/presidential.cfm� http://www.ala.org/ala/mgrps/divs/acrl/publications/whitepapers/presidential.cfm� http://www.ala.org/ala/mgrps/divs/acrl/publications/whitepapers/presidential.cfm� http://www.flickr.com/commons/#faq� http://www.academiccommons.org/files/image-report.pdf� http://www.academiccommons.org/files/image-report.pdf� http://projectinfolit.org/pdfs/pil_progressreport_2_2009.pdf� http://projectinfolit.org/pdfs/pil_progressreport_2_2009.pdf� http://projectinfolit.org/pdfs/pil_fall2009_year1report_12_2009.pdf� http://projectinfolit.org/pdfs/pil_fall2009_year1report_12_2009.pdf� evidence based library and information practice 2011, 6.1 33 journal of archival organization, 6(4), 233-268. malkmus, d. k. (2008, august). teaching undergraduates with primary source: highlights of survey. research paper presented at the society of american archivists 2008 research forum, san francisco, ca. retrieved august 1, 2010 from http://www.archivists.org/publication s/proceedings/researchforum/2008/pa pers/dorismalkmus-saaresearchpaper-2008.pdf. matusiak, k. k. (2006). information seeking behavior in digital image collections: a cognitive approach. journal of academic librarianship, 32(5), 479-488. mccay-peet, l., & toms, e. (2009). image use within the work task model: images as information and illustration. journal of the american society for information science and technology, 60(12), 24162429. nichols, d., rowlands, i., clark, d., & williams, p. (2011). google generation ii: web behaviour experiments with the bbc. aslib proceedings, 63(1), 28-45. doi: 10.1108/00012531111103768 oregon state university. (2010). oregon state university online catalog – earning a degree at oregon state university. retrieved december 14, 2010, from http://catalog.oregonstate.edu/chapter detail.aspx?key=6 shonfeld, r.c. (2006). the visual resources environment at liberal arts colleges. georgetown, tx: national institute for technology and liberal education. retrieved july 1, 2010, from www.nitle.org/transformations.html silipigni connaway, l., & dickey, t. j. (2010). the digital information seeker: report of findings from selected oclc, rin and jisc user behaviour projects. london, england: jisc. retrieved july 1, 2010, from http://www.jisc.ac.uk/media/documen ts/publications/reports/2010/digitalinf ormationseekerreport.pdf strauss, a. l., & corbin, j. m. (1998). basics of qualitative research: techniques and procedures for developing grounded theory. thousand oaks: sage publications. tally, b., & goldenberg, l. b. (2005). fostering historical thinking with digitized primary sources. journal of research on technology in education, 38(1), 1-21. http://www.archivists.org/publications/proceedings/researchforum/2008/papers/dorismalkmus-saa-researchpaper-2008.pdf� http://www.archivists.org/publications/proceedings/researchforum/2008/papers/dorismalkmus-saa-researchpaper-2008.pdf� http://www.archivists.org/publications/proceedings/researchforum/2008/papers/dorismalkmus-saa-researchpaper-2008.pdf� http://www.archivists.org/publications/proceedings/researchforum/2008/papers/dorismalkmus-saa-researchpaper-2008.pdf� http://catalog.oregonstate.edu/chapterdetail.aspx?key=6� http://catalog.oregonstate.edu/chapterdetail.aspx?key=6� http://www.nitle.org/transformations.html� http://www.jisc.ac.uk/media/documents/publications/reports/2010/digitalinformationseekerreport.pdf� http://www.jisc.ac.uk/media/documents/publications/reports/2010/digitalinformationseekerreport.pdf� http://www.jisc.ac.uk/media/documents/publications/reports/2010/digitalinformationseekerreport.pdf� evidence based library and information practice 2011, 6.1 34 appendix a: osu image search methods for undergraduates survey q1. for this question we’d like you to imagine you are doing a project or writing a paper for a class. your instructor has asked you to include a picture of people logging in oregon forests in the 1930s or 1940s. please explain your search process for finding a picture. you might start your explanation with the phrase, ‘i would begin my search for the picture by’ q2. now thinking about searching for any image online, please indicate in what ways you have used each of the following websites to search for images. (check all that apply) a. flickr (flickr.com) heard of this site but not used to search for images have used for an assignment to search for images have used for personal/pleasure to search for images likely to use in future for assignments if searching for images likely to use in future for personal/pleasure if searching for images not at all familiar with this site don't know b. google (google.com) heard of this site but not used to search for images have used for an assignment to search for images have used for personal/pleasure to search for images likely to use in future for assignments if searching for images likely to use in future for personal/pleasure if searching for images not at all familiar with this site don't know c. google images (images.google.com) heard of this site but not used to search for images have used for an assignment to search for images have used for personal/pleasure to search for images likely to use in future for assignments if searching for images likely to use in future for personal/pleasure if searching for images not at all familiar with this site don't know d. yahoo! (yahoo.com) heard of this site but not used to search for images evidence based library and information practice 2011, 6.1 35 have used for an assignment to search for images have used for personal/pleasure to search for images likely to use in future for assignments if searching for images likely to use in future for personal/pleasure if searching for images not at all familiar with this site don't know e. wikipedia (wikipedia.org) heard of this site but not used to search for images have used for an assignment to search for images have used for personal/pleasure to search for images likely to use in future for assignments if searching for images likely to use in future for personal/pleasure if searching for images not at all familiar with this site don't know f. facebook (facebook.com) heard of this site but not used to search for images have used for an assignment to search for images have used for personal/pleasure to search for images likely to use in future for assignments if searching for images likely to use in future for personal/pleasure if searching for images not at all familiar with this site don't know g. osu libraries' digital collections (osulibrary.oregonstate.edu/digitalcollections) heard of this site but not used to search for images have used for an assignment to search for images have used for personal/pleasure to search for images likely to use in future for assignments if searching for images likely to use in future for personal/pleasure if searching for images not at all familiar with this site don't know h. osu libraries' university archives (osulibrary.oregonstate.edu/archives) heard of this site but not used to search for images have used for an assignment to search for images evidence based library and information practice 2011, 6.1 36 have used for personal/pleasure to search for images likely to use in future for assignments if searching for images likely to use in future for personal/pleasure if searching for images not at all familiar with this site don't know q4. is there something librarians can provide that would help you search for online images? q5. which college are you primarily associated with? agricultural science business education engineering forestry health and human sciences liberal arts science honors undecided other don’t know q6. are you male or female? male female evidence based library and information practice 2011, 6.1 37 appendix b: frequency analysis question: now thinking about searching for an image online, please indicate in what ways you have used each of the following websites to search for images. (check all that apply) frequency analysis for flickr flickr (flickr.com) frequency percent cumulative frequency cumulative percent heard of this site but not used to search for images 23 28.05 23 28.05 have used for an assignment to search for images 9 10.98 32 39.02 have used for personal/pleasure to search for images 10 12.20 42 51.22 likely to use in future for assignments if searching for images 7 8.54 49 59.76 likely to use in future for personal/pleasure if searching for images 8 9.76 57 69.51 not at all familiar with this site 23 28.05 80 97.56 don`t know 2 2.44 82 100.00 heard of this site but not used to search for images 23 28.05 23 28.05 have used for an assignment to search for images 2 2.44 32 39.02 frequency analysis for google google (google.com) frequency percent cumulative frequency cumulative percent heard of this site but not used to search for images 2 1.08 2 1.08 have used for an assignment to search for images 54 29.19 56 30.27 have used for personal/pleasure to search for images 47 25.41 103 55.68 likely to use in future for assignments if searching for images 43 23.24 146 78.92 likely to use in future for personal/pleasure if searching for images 39 21.08 185 100.00 evidence based library and information practice 2011, 6.1 38 frequency analysis for google images google images (images.google.com) frequency percent cumulative frequency cumulative percent have used for an assignment to search for images 55 28.80 55 28.80 have used for personal/pleasure to search for images 51 26.70 106 55.50 likely to use in future for assignments if searching for images 44 23.04 150 78.53 likely to use in future for personal/pleasure if searching for images 41 21.47 191 100.00 frequency analysis for yahoo! yahoo! (yahoo.com) frequency percent cumulative frequency cumulative percent heard of this site but not used to search for images 45 54.88 45 54.88 have used for an assignment to search for images 10 12.20 55 67.07 have used for personal/pleasure to search for images 8 9.76 63 76.83 likely to use in future for assignments if searching for images 8 9.76 71 86.59 likely to use in future for personal/pleasure if searching for images 6 7.32 77 93.90 not at all familiar with this site 2 2.44 79 96.34 don`t know 3 3.66 82 100.00 evidence based library and information practice 2011, 6.1 39 frequency analysis for wikipedia wikipedia (wikipedia.org) frequency percent cumulative frequency cumulative percent heard of this site but not used to search for images 27 21.26 27 21.26 have used for an assignment to search for images 32 25.20 59 46.46 have used for personal/pleasure to search for images 25 19.69 84 66.14 likely to use in future for assignments if searching for images 23 18.11 107 84.25 likely to use in future for personal/pleasure if searching for images 19 14.96 126 99.21 not at all familiar with this site 1 0.79 127 100.00 frequency analysis for facebook facebook (facebook.com) frequency percent cumulative frequency cumulative percent heard of this site but not used to search for images 41 49.40 41 49.40 have used for an assignment to search for images 3 3.61 44 53.01 have used for personal/pleasure to search for images 21 25.30 65 78.31 likely to use in future for assignments if searching for images 2 2.41 67 80.72 likely to use in future for personal/pleasure if searching for images 13 15.66 80 96.39 don`t know 3 3.61 83 100.00 evidence based library and information practice 2011, 6.1 40 frequency analysis of osu libraries’ digital collection osu libraries` digital collections (osulibrary.oregonstate.edu/digitalcollections) frequency percent cumulative frequency cumulative percent heard of this site but not used to search for images 9 12.16 9 12.16 have used for an assignment to search for images 1 1.35 10 13.51 likely to use in future for assignments if searching for images 10 13.51 20 27.03 likely to use in future for personal/pleasure if searching for images 3 4.05 23 31.08 not at all familiar with this site 49 66.22 72 97.30 don`t know 2 2.70 74 100.00 frequency analysis of osu libraries’ university archives osu libraries` university archives (osulibrary.oregonstate.edu/archives) frequency percent cumulative frequency cumulative percent heard of this site but not used to search for images 17 22.37 17 22.37 have used for an assignment to search for images 2 2.63 19 25.00 likely to use in future for assignments if searching for images 11 14.47 30 39.47 likely to use in future for personal/pleasure if searching for images 5 6.58 35 46.05 not at all familiar with this site 39 51.32 74 97.37 don`t know 2 2.63 76 100.00 / evidence based library and information practice library workers experiencing or observing sexual harassment in university of california libraries is commonplace and commonly unreported evidence summary   library workers experiencing or observing sexual harassment in university of california libraries is commonplace and commonly unreported   a review of: barr-walker, j., hoffner, c., mcmunn-tetangco, e., & mody, n. (2021). sexual harassment at university of california libraries: understanding the experiences of library staff members. college & research libraries, 82(2), 237. https://doi.org/10.5860/crl.82.2.237   reviewed by: samantha j. kaplan research & education librarian, liaison to the school of medicine duke university medical center library & archives durham, north carolina, united states of america email: samantha.kaplan@duke.edu   received: 1 sept. 2021                                                               accepted:  12 oct. 2021      2021 kaplan. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30030     abstract   objective – to identify whether academic library workers at the university of california libraries (ucl) system experienced or observed sexual harassment and to measure their reporting and disclosure behavior.   design – anonymous online survey with open and closed-end questions.   setting – all ucl system campuses (berkeley, davis, irvine, los angeles, merced, riverside, santa barbara, santa cruz, san diego, and san francisco).   subjects – all 1610 non-student employees working in ucl system were invited to participate, 579 (36%) responded.   methods – the authors engaged multiple stakeholder groups to refine and promote this census of ucl non-student workers. the survey was distributed via redcap and remained open for six weeks of november to december 2018. all questions were optional. certain demographic information was not collected because respondents might have been identified via deductive disclosure. the first author conducted descriptive statistical analysis and pairs of authors conducted thematic analysis.   main results – more than half of respondents experienced or observed sexual harassment in the workplace; women were more likely to experience than observe and vice versa for men. harassment was most likely to be exhibited by a coworker. less than half of respondents felt that the ucl system administration considered the issue important. nearly three out of every four respondents who had experienced harassment at work chose not to report or disclose; this did not vary significantly between women and men.   conclusion – sexual harassment of library workers, often by other library workers, is widespread. staff training and policies should incorporate the reality of gender harassment and commenting on a person's appearance—the two most common forms of harassment exhibited and observed.    commentary   sexual harassment occurring in libraries is not a surprise, but research on this topic has only recently entered library scholarship (benjes-small et al., 2021; ford, 2017). this study is the first to attempt a census, via an anonymous online survey, to understand how widespread the problem is in a specific population of library workers—in this case, non-student employees in the 10-campus university of california libraries system.   the center for evidence-based management provides a checklist to appraise survey research (cebma, n.d.). in assessing the research by barr-walker et al. using this checklist, several aspects must be considered. first, while this was a census—all possible members of this population were contacted and invited to participate—not all chose to do so. second, aside from broad gender identity (terminology used by the survey), the researchers did not gather demographic data (e.g., race, sexual orientation, campus). this protects participants' identities but also presents challenges for generalizability of the findings. not requiring any of this information means that it is difficult to judge the success of a 36% response rate (q.7 of the cebma tool), as one cannot ascertain if there are meaningful clusters of characteristics of participants who did not respond. this highlights the tension in collecting information about sensitive topics that could potentially endanger participants, as 12% of survey respondents did not report or disclose their experiences of harassment due to fear of retaliation, embarrassment, or being seen as a troublemaker.   barr-walker et al’s research meets the cebma's criteria of addressing a clearly focused question and utilizes an appropriate design. while it is not possible to appraise the study on many of the cebma criteria because it lacks the typical statistical significance of surveys (particularly in comparison between groups), the authors do provide overwhelming evidence that sexual harassment is a widespread problem in the ucl system. perhaps most notable and most disheartening, the study demonstrates how ubiquitous and typical this experience is at ucl; among participants who had experienced harassment at work and chose not to report it, 41% indicated the problem was not egregious enough. one participant reported, "if i were to report every time that i felt unfairly treated … i'd spend more time reporting than working."    these findings are as disturbing as they are revelatory. academic library workers engage with members of the public and their patron populations in service roles where, as one participant acknowledged, "[you] feel like you cannot leave your post." this sentiment becomes far more sinister when contextualized by a workplace environment where behavior like this is typical. according to the survey, the most common source of harassment was a ucl staff member. if individuals do not feel they can avoid being treated this way by their coworkers, why would they expect better from their patrons? adding to the issue, more than half of respondents did not feel their library administration considered the issue important. in a workplace where one regularly experiences or witnesses harassment, the sentiment that those in power do not care enables the continuation of this widespread problem. if campus administration is perceived as indifferent to this issue, they become complicit.   one important limitation of this study is the population was defined as non-student library workers. many academic libraries rely heavily on student work forces. if staff are experiencing harassment, immediate attention should be directed toward understanding if this extends to students who do not have the benefits and theoretical protections of full-time employees. an additional limitation of this study is that it did not make comparisons across campuses. while the authors did this intentionally, it prevents further consideration of the role of setting (urban versus rural), size of library, or other characteristics that could contribute to or enable a workplace ambience that allows harassment. the authors may not have wanted to risk sowing division among campuses by comparing them to one another, but combining the campuses into larger categories would have allowed for statistical significance testing to provide insight if there are factors that make harassment more pervasive.   references   benjes-small, c., knievel, j., resor-whicker, j., wisecup, a., & hunter, j. (2021). #metoo in the academic library: a quantitative measurement of the prevalence of sexual harassment in academic libraries. college & research libraries, 82(5), 623. https://doi.org/10.5860/crl.82.5.623   center for evidence-based management (cebma). (n.d). critical appraisal of a survey. https://www.cebma.org/wp-content/uploads/critical-appraisal-questions-for-a-survey.pdf   ford, a. (2017). stop sexual harassment in your library. american libraries, 48(11-12), 16–19. https://americanlibrariesmagazine.org/2017/11/01/stop-sexual-harassment-your-library/   evidence based library and information practice evidence based library and information practice 2009, 4:3  evidence based library and information practice       commentary    is now the time for an international association of eblip?    andrew booth  reader in evidence based information practice  university of sheffield  sheffield, south yorkshire, united kingdom  email: a.booth@sheffield.ac.uk    received: 27 july 2009           accepted: 29 july 2009      © 2009 booth.  this is an open access article distributed under the terms of the creative commons attribution  license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and  reproduction in any medium, provided the original work is properly cited.     “if it ain’t broke, don’t fix it” is a maxim that  many will apply to the current success of the  evidence based library and information  practice (eblip) community. after all, they  may argue, doesn’t eblip number many  impressive achievements? indeed as i  comment in a forthcoming quinquennial  review:    eblip now has its own international  open access journal, contributors from  outside healthcare are starting to  attend the international biennial  conference series and to showcase the  potential of eblip in a wide range of  professional and peer‐reviewed  journals (booth “a bridge too far”).    furthermore, it is not the mere existence of the  open access journal and international  conference series that attest to the movement’s  extraordinary energy. following a  predominantly parochial first conference in  sheffield in 2001, each successive event has  generated enthusiastic, energetic, and far‐ sighted individuals willing to organise and  host the next fixture in the series. no doubt  this would also have been the case at the  recently concluded 5th international evidence  based library and information practice  conference (eblip5) in stockholm; sweden  had not the need been identified for a more  formal application process for the eblip6  conference. equally the journal has continued  to prosper and grow despite an almost non‐ existent business model.    why then did those assembled at eblip5 on  july 1, 2009, forego the undoubted pull of their  hotel beds to meet for a lively pre‐session  discussion on the need for an international  eblip association? first, one suspects, such  an assembly came about in unspoken  acknowledgement of a prevailing zeitgeist  regarding the maturity of the movement and  its spread across sectors, countries and  knowledge domains. however, more prosaic  drivers include concerns regarding the lack of  governance structures and the absence of a  sponsoring organisation. while freedom from  “red tape” has undoubtedly yielded benefits  in terms of flexibility and mobility of  conference organisation, the vulnerability of  this model can be simply illustrated in a brief  68 mailto:a.booth@sheffield.ac.uk http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2009, 4:3  sequence of “what if’s:” what if a future local  organising committee decides unilaterally to  cancel a conference? what if there is a  difference in opinion on the conference  “vision” between future local organising and  international programme committees? what  if a future local organising committee  decides to double the conference registration  fees? the fact that such informal arrangements  have worked so well in the past meant that all  those assembled could securely voice such  otherwise unthinkable concerns.     another area for attention is the financial basis  for activities such as the journal and the  conference series. in previous years conference  organisers have worked from an implicit and  non‐contractual assumption that, once costs  have been recovered, any profits from their  event would be passed to subsequent eblip  conference organisers. there is no mechanism  at present, either, for use of conference  revenue to stimulate the maintenance and  ongoing development of the eblip journal –  arguably the vehicle for the greatest common  good within the international eblip  community – or, indeed, for any similarly  worthy venture.     of course the idea of an international  collaboration within eblip is not new. back in  1999, eldredge (14) highlighted the need for  such collaboration. two years later, booth  (“will health librarians” 62) concluded a  research column with a plea for activities to  mirror the “celtic fringe” of the cochrane and  campbell collaborations ‐ even nominating  eligible namesakes for such a venture!  appropriately, in a chapter collectively  authored by collaborators from three  countries, the need for more formal  international ties was reiterated in a  cumulative wish list for the movement as a  whole (booth et al “a future for evidence  based information practice”). this theme was  revisited in the pages of this journal  concluding that:    the eblip movement requires both  strategic coordination and the  development of international consensus.  strategic coordination would yield  opportunities for a more planned and  systematic approach to conference  planning and the development and  allocation of resources and initiatives  according to genuine need. this would  serve to channel the prodigious energies  and enthusiasms of those already involved  in eblip more productively as well as  providing a rallying point for those  interested in future participation (booth &  brice “prediction is difficult” 100).    less clearly articulated, however, have been  the specifics of what such an international  association might have as its purpose and  indeed what its future activities might be.  among suggestions advanced at the meeting  were roles as a membership organisation, as a  home for the eblip journal, as a neutral  “honest broker” that invites contributions  from the widest possible constituency, as a  social network for communication between  conferences, as a strategic body to ensure the  sustainability of emerging or recurring  themes, as a conference organiser and as an  implementation enabler in promoting  research, implementation, training and the  development of tools and instruments. clearly  the time is right to seek the views of the  international community in shaping the future  configuration of such a body.    potential association models     participants at the meeting rehearsed some of  the arguments for and against a formal  association. on the one hand, for example, an  association would provide opportunities to  enable eblip to become more integrated into  everyday practice. on the other hand, it holds  the danger of becoming a self‐serving  organisation divorced from the concerns of  daily practice. similar debates ranged to and  fro. certainly if one needs any reminder of the  constraints of allying oneself to an existing  international organisation, this has come  recently with the decision of the international  federation of library associations (ifla) to  move its congress and all associated events  from australia to sweden (international  69 evidence based library and information practice 2009, 4:3  federation of library associations).  nevertheless those in attendance were keen to  explore the widest possible range of models  for a potential association. some of these are  listed in box 1.     1. independent international association e.g.  cochrane collaboration  2. international conference of medical  librarianship (icml)  3. international federation of library  associations (ifla) section  4. official satellite of ifla congress, co‐ sponsored by a section of ifla (e.g.  northumbria international conference on  performance measurement in libraries  and information services)  5. international federation of library  associations (ifla) special interest group   6. national governance, international  membership (e.g. canadian library  association special interest group ‐  http://www.cla.ca/am/template.cfm?secti on=evidence_based_librarianship)  7. a “work in progress” association (i.e. it  adapts as each successive need arises)  box 1    potential models for an international eblip  association    way forward    with such a range of models to be explored,  not to mention any that those assembled were  unable to identify, it is not surprising that the  group felt unable to resolve such an important  issue immediately. however the group did  feel it important to separate the longer term  strategy, perhaps involving creation of an  international association, from the more  medium term issue of where to hold the next  conference.     the following actions were therefore proposed  and accepted:  a working group, led by jonathan  eldredge from the university of new  mexico, should be formed to investigate  potential association models and consider  the suitability or viability of each model in  relation to the eblip movement.  membership of the working group is yet  to be confirmed.  this working group would inform a wider  consultation to establish a favoured  association model, (including the option of  no international association), using all  available communication routes (e.g.  former eblip conference attendees, the  evidence‐based‐libraries discussion list and  associated discussion lists), an online  survey and via the pages of a future issue  of the evidence based library and  information practice journal.    an international advisory committee,  comprising one representative from each  previous conference team and chaired by  andrew booth, outgoing co‐chair of the  eblip5 international programme  committee, would invite expressions of  interest to host the eblip6 conference in  2011.  completion of two‐page expression  of interest application forms would take  place in august 2009. it is hoped that the  venue of eblip6 will be announced in  september/october 2009.    just as a parent mourns the transition of their  child from the spontaneity of primary school  to the exactingly choreographed formal  timetable of secondary school so too many of  us have cause to mourn the passing of the  informal arrangements under which no fewer  than five successful eblip conferences have  prospered. nevertheless, we have every  confidence that addressing the concerns of  longer term sustainability, accountability and  governance will help contribute to the  inevitable maturity of the eblip movement. i  am reminded of the following quotation from  colin powell, himself the chair of a (much less  enlightened!) proponent of international  “collaboration”: “if it ainʹt broke, donʹt fix itʹ is  the slogan of the complacent, the arrogant or  the scared. itʹs an excuse for inaction, a call to  non‐arms.”     if you would like to contribute to our call to  arms, whether randomised or otherwise, then  please feel free either to contact the author,  andrew booth <a.booth@sheffield.ac.uk>,  70 http://www.cla.ca/am/template.cfm?section=evidence_based_librarianship http://www.cla.ca/am/template.cfm?section=evidence_based_librarianship mailto:a.booth@sheffield.ac.uk evidence based library and information practice 2009, 4:3  71 informally or to make your views heard  during the formal consultation period as soon  as it is announced. alternatively use the  existing organs of the eblip community,  namely the pages of the eblip journal and the  evidence‐based‐libraries discussion list  <http://www.jiscmail.ac.uk/lists/evidence‐ based‐libraries.html> to shape thinking  around a future configuration of an  international eblip association. your  movement needs you!     acknowledgements     the author is grateful to maria j grant,  research fellow (information) at the centre  for nursing, midwifery and collaborative  research, university of salford for the meeting  notes upon which the factual content of this  otherwise personal commentary is based.      works cited    booth, andrew. “a bridge too far?: stepping  stones for evidence based practice in  an academic context.” new review  of academic librarianship  (forthcoming).    booth, andrew. “will health librarians and  related information workers ever  work together to create an  international network?” health  information and libraries journal 18.1  (2001): 60‐3.     booth, andrew, and anne brice. “’prediction  is difficult, especially the future’: a  progress report.ʺ evidence based  library and information practice 2.1  (2007): 89‐106. 27 jul. 2009  <http://ejournals.library.ualberta.ca/in dex.php/eblip/article/view/99/242>.    booth, andrew, anne brice, jonathan  eldredge, ellen crumley and denise  koufogiannakis. “a future for  evidence based information  practice?” eds. andrew booth and  anne brice. evidence‐based practice  for information professionals: a  handbook. london, uk: facet  publishing, 2004, 279‐92.     eldredge, jonathan. “international research  reviews: evidence‐based  librarianship (ebl) needs  international collaboration.”  hypothesis 13.2 (1999): 14‐6.    international federation of library  associations. “göteborg, sweden to  host the 2010 ifla world library and  information congress.” (8 july 2009).  27 jul. 2009  <http://www.ifla.org/en/news/g‐ teborg‐sweden‐to‐host‐the‐2010‐ifla‐ world‐library‐and‐information‐ congress>    powell, colin. 27 jul. 2009  <http://thinkexist.com/quotation/if_it_ ain‐t_broke‐don‐t_fix_it‐ is_the_slogan_of/147415.html>  http://www.jiscmail.ac.uk/lists/evidence-based-libraries.html http://www.jiscmail.ac.uk/lists/evidence-based-libraries.html http://ejournals.library.ualberta.ca/index.php/eblip/article/view/99/242 http://ejournals.library.ualberta.ca/index.php/eblip/article/view/99/242 http://www.ifla.org/en/news/g-teborg-sweden-to-host-the-2010-ifla-world-library-and-information-congress http://www.ifla.org/en/news/g-teborg-sweden-to-host-the-2010-ifla-world-library-and-information-congress http://www.ifla.org/en/news/g-teborg-sweden-to-host-the-2010-ifla-world-library-and-information-congress http://www.ifla.org/en/news/g-teborg-sweden-to-host-the-2010-ifla-world-library-and-information-congress http://thinkexist.com/quotation/if_it_ain-t_broke-don-t_fix_it-is_the_slogan_of/147415.html http://thinkexist.com/quotation/if_it_ain-t_broke-don-t_fix_it-is_the_slogan_of/147415.html http://thinkexist.com/quotation/if_it_ain-t_broke-don-t_fix_it-is_the_slogan_of/147415.html potential association models way forward evidence based library and information practice 2011, 6.3 71 evidence based library and information practice evidence summary there are discipline-based differences in authors’ perceptions towards open access publishing a review of: coonin, b., & younce, l. m. (2010). publishing in open access education journals: the authors’ perspectives. behavioral & social sciences librarian, 29, 118-132. doi:10.1080/01639261003742181 reviewed by: lisa shen reference librarian newton gresham library, sam houston state university huntsville, texas, united states of america email: lshen@shsu.edu received: 23 feb. 2011 accepted: 15 july 2011 © 2011 shen. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to determine reasons authors choose to publish in open access (oa) education journals, which provides readers with unrestricted free online access to published articles, and investigate ways in which publishing practices in the discipline of education affects authors’ willingness to publish in these journals. design – web-based survey questionnaire. setting – the survey was conducted over the internet through email invitations. subjects – a total of 309 authors who published in oa journals in education participated in this survey for a response rate of 27.9%. methods – researchers surveyed authors who published in selected education journals from 2007 to 2008. the journal titles where generated from the directory of open access journals (doaj). all chosen journals were peerreviewed and published either original research or overviews of research results. in addition, all were in english and published in the united states. a total of 1,107 authors were invited to participate via email. the survey was delivered through commercial online survey tool mailto:lshen@shsu.edu� evidence based library and information practice 2011, 6.3 72 surveymonkey and consisted of multiple choice and open-ended questions. it was open from early march to april 16, 2009. main results – the survey had a response rate of 27.9%. the majority of participants were tenured faculty (42.0%), tenure-track faculty (25.9%), and non-tenure track faculty (12.1%). the rest of participants (20%) consisted of adjunct instructors, graduate students, administrators, and individuals working in nonacademic institutions such as government agencies. most authors surveyed have published between 10 and 20 articles (20.6%), or over 20 articles (30.4%) in print and electronic journals (ejournals). the majority of authors also reported that one (23.3%) or between 2 to 5 (54%) of their articles was published in oa format. when choosing a journal for publications, authors surveyed ranked peer-review to be the most important determinant. other important determinants included “good match” (ranked second most important) for authors’ manuscripts and reputation of the journal (third) and editorial board (fourth). citation impact, such as the isi impact factor (eighth), and copyright retention (tenth) were ranked as some of the least important factors. researcher also noted a “surprisingly low” (p. 124) correlation between authors’ interest in copyright retention and practices of selfarchiving. thirty-seven percent of authors surveyed reported self-archiving at least one of their publications, but just over 35% of the same group considered copyright retention a determinant when choosing journals for publication. overall, only 22% of the authors surveyed deemed e-journals to be “less desirable” than print journals. the majority of both tenured faculty (77.4%) and tenure-track faculty (72%) surveyed found e-journals “acceptable” or difference between print and electronic journal format “not an issue.” only 16.8% of authors surveyed had published in journals that required author fees. moreover, over 56% of authors indicated they would not publish in journals requiring such fees. most authors reported they were either very aware (45.1%) or somewhat aware (38.9%) of the concept of oa publishing. however, their perceptions of oa publishing varied: • 47.7% believed oa journals have faster publication times, while 33.6% disagreed and 18.5% offered no opinion. • 57.3% of authors believed oa journals have larger readerships. however, when asked whether oa articles would be cited more frequently than others, only one third of authors agreed, while one third disagreed and one third offered no opinion. • just under half of the authors (49.4%) thought oa journals are not less prestigious than subscription based journals, while 18.8% had no opinion. lastly, it should be noted that only 7.1% of authors credited their institution’s library for making them aware of the oa publishing concept. most credited their colleagues (42.1%), google searches for publishing opportunities (40.4%), and professional societies (29.3%) for raising their awareness of oa. moreover, based on voluntary general comments left at end of the survey, researchers observed that some authors viewed the terms open access and electronic “synonymously” and thought of oa publishing only as a “format change” (p.125). conclusion – the study revealed some discipline-based differences in authors’ attitudes toward scholarly publishing and the concept of oa. the majority of authors publishing in education viewed author fees, a common oa publishing practice in life and medical sciences, as undesirable. on the other hand, citation impact, a major determinant for life and medical sciences publishing, was only a minor factor for authors in education. these findings provide evidence based library and information practice 2011, 6.3 73 useful insights for future research on disciplinebased publication differences. the findings also indicated peer review is the primary determinant for authors publishing in education. moreover, while the majority of authors surveyed considered both print and ejournal format to be equally acceptable, almost one third viewed oa journals as less prestigious than subscription-based publications. some authors also seemed to confuse the concept between oa and electronic publishing. these findings could generate fresh discussion points between academic librarians and faculty members regarding oa publishing. commentary although this study raised a number of interesting issues, a close examination using the ebl critical appraisal checklist (glynn, 2006) indicates that the overall validity of this study is less than 75%. therefore, readers are encouraged to use the results for generating research ideas, but not for drawing generalized conclusions. additionally, the survey instrument was not appended to the article, limiting the transparency of this research. a fundamental flaw of this study is the disparity between its research objective and choice of population sample. the researchers intended to examine authors’ perspectives on publishing in open access education journals, but only surveyed authors with experience in publishing in limited types of oa journals. therefore, the study results cannot be generalized to represent all authors’ perspectives. for instance, the majority of authors surveyed indicated moderate to high levels of understanding of oa publishing and perceived oa journals to be no less prestigious than print, yet it would be invalid to conclude all authors in education hold such knowledge. this study did not include perspectives of education authors who have published in non-peer reviewed oa journals, nor those who have not published in any oa publications, including those who may have chose not to publish in this medium due to negative perceptions of oa. additionally, as indicated by the comments submitted with the surveys, some authors appeared to think of the terms “electronic” and “open access” as synonymous concepts. this is a significant misunderstanding: oa journals provides unrestricted electronic access to published articles, while electronic journals includes both journals that follows the oa model and those that that charge fees for access. the researchers’ own remarks indicate that definitions of “e-journal” and “open access journal” were not provided in the survey. thus, all responses regarding authors’ perceptions towards either electronic or oa publishing may be unreliable. for instance, it is possible that when asked to compare electronic and print journals, some authors’ answers were based on their perception of oa journals instead of ejournals. despite issues highlighted above, the rest of the survey questions appeared well constructed and investigate pertinent aspects of respondents’ perceptions of oa. given appropriate population sampling and provision of definition for key terms, the rest of the survey could provide a solid template for those who wish to further examine perceptions towards oa publishing in different academic disciplines. references glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. / evidence based library and information practice evidence based library and information practice evidence based library and information practice 2010, 5.4 133 evidence based library and information practice news/announcements call for presentation proposals: 2011 library research round table forums at ala annual conference 2010. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. the library research round table (lrrt) will sponsor two research forums at the 2011 american library association annual conference in new orleans (june 23-28, 2011). the lrrt forums are a set of programs at the ala annual conference featuring presentations of lis research, in progress or completed, followed by discussion. two lrrt research forums are scheduled for 2011, one on general lis research and one on a more specific topic that will emerge as we evaluate the submissions. the two forums are: 1. research to understand users: issues and approaches this session will feature three libraryrelated research papers investigating users and their use of libraries and information. an lrrt committee will select the winning papers based on quality of study design, significance of the research topic, and potential for significant contribution to librarianship. 2. four-star research this session will feature three libraryrelated research papers describing studies of libraries and librarianship. an lrrt committee will select the winning papers based on quality and creativity of study design, significance of the research topic, and potential for significant contribution to librarianship. this is an opportunity to present and discuss your research project conducted in the broad area of library and information science or in a more specialized area of the field. lrrt welcomes papers emphasizing the problems, theories, methodologies, or significance of research findings for lis. topics can include, but are not limited to, user studies and user behavior, electronic services, service effectiveness, organizational structure and personnel, library value determination, and evaluation of library and information services. both completed research and research in progress will be considered. all researchers, including practitioners from all types of libraries, library school faculty and students, and other interested individuals are encouraged to submit proposals. lrrt members and nonmembers of lrrt are invited and welcomed to submit proposals. the committee will use a blind review process to select a maximum of six projects, three for each of the two forums. the selected researchers will be required to present their papers in person at the forums and to register for the conference. criteria for selection are: evidence based library and information practice 2010, 5.4 134 1. significance of the study to library and information science research; 2. quality and creativity of the methodology; 3. potential to fill a research gap or to build on previous lis studies; 4. adherence to submission requirements (see below). please submit a two-page proposal by friday, december 17, 2010. late submissions will not be considered, and submissions must be limited to two pages in length. on the first page, please list your name(s), title(s), institutional affiliation(s), and contact information (telephone number, mailing address, and email address). the second page should not show your name or any other identifying information. instead, it must include: 1) the title of your project, and 2) a 500-word or less abstract. the abstract must include a problem statement, problem significance, project objectives, methodology, and conclusions (or tentative conclusions for work in progress), and an indication of whether the research is in-progress or completed. previously published research or research accepted for publication by december 15, 2010, will not be considered. notification of acceptance will be made by monday, february 21, 2011. please send submissions (via email or snail mail) to: john m. budd library research round table chairelect school of information science & learning technologies 303 townsend hall university of missouri columbia, mo 65203 phone: 573.882.3258 fax: 573.884.4944 email: buddj@missouri.edu mailto:buddj@missouri.edu� / evidence based library and information practice review article   twenty years of business information literacy research: a scoping review   meggan a. houlihan college liaisons coordinator for social sciences, humanities, arts, and business morgan library colorado state university fort collins, colorado, united states of america email: meggan.houlihan@colostate.edu   amanda b. click head of research & instruction nimitz library united states naval academy annapolis, maryland, united states of america email: click@usna.edu   claire walker wiley research & instruction librarian lila d. bunch library belmont university nashville, tennessee, united states of america email: claire.wiley@belmont.edu   received: 27 feb. 2019                                                               accepted: 17 aug. 2020      2020 houlihan, click, and wiley. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29745     abstract   objective – this study analyzes and synthesizes the business information literacy (bil) literature, with a focus on trends in publication type, study design, research topic, and recommendations for practice.   methods – the scoping review method was used to build a dataset of 135 journal articles and conference papers. the following databases were searched for relevant literature published between 2000 and 2019: library and information science source, science direct, proquest central, project muse, and the ticker journal site. included items were published in peer reviewed journals or conference proceedings and focused on academic libraries. items about public or school libraries were excluded, as were items published in trade publications. a cited reference search was conducted for each publication in the review dataset.    results – surveys were, by far, the most common research method in the bil literature. themes related to collaboration were prevalent, and a large number of publications had multiple authors or were about collaborative efforts to teach bil. many of the recommendations for practice from the literature were related to collaboration as well; recommendations related to teaching methods and strategies were also common. adoption of the framework for information literacy for higher education in bil appears slow, and the citations have decreased steadily since 2016. the majority of the most impactful bil articles, as measured by citation counts, presented original research.    conclusions – this study synthesizes two decades of literature and contributes to the evidence based library and information science literature. the findings of this scoping review illustrate the importance of collaboration, interest in teaching methods and strategies, appreciation for practical application literature, and hesitation about the framework.     introduction   business librarians face unique challenges in the classroom. from faculty partner expectations to the diverse research skills required, this group must think creatively in order to achieve learning outcomes and demonstrate the value of information literacy (il) on their campuses. this study, which is focused on the intersection of information literacy and the discipline of business, is important because business is the most popular undergraduate degree in the u.s. and has been for decades (national center for education statistics, 2017). business librarians can have a great impact on this large group of students with innovative and effective approaches to information literacy. this study uses the scoping review method in order to explore innovations and approaches to information literacy in business.   two foundational documents from the association of college & research libraries (acrl) have guided information literacy practice over the last 20 years: the information literacy competency standards for higher education (2000) and the framework for information literacy for higher education (2015). the standards and framework are built on the same principles, but the theory behind them and the implications for practice are quite different. the standards include information literacy competencies and performance indicators, while the framework includes knowledge practices and dispositions that can be harder to assess. the definition of information literacy has also evolved, and this change is reflected in the framework document. this shift reflects a change in thinking in library and information science, but it has been met with some resistance. survey results published in 2005 and 2018 demonstrate that business librarians have struggled with integrating them into their teaching practice for a number of reasons. in cooney’s (2005) survey of business librarians, only a third of survey respondents reported incorporating the standards into their instruction, and assessments of student learning in this area were rarely conducted. cooney also discovered that business information literacy (bil) instruction was still developing and that there was great room for improvement in collaboration between librarians and business faculty. guth and sachs (2018) recreated cooney’s survey by exploring implementation of both the standards and the newer framework and discovered several interesting points of comparison with the 2005 responses. most notably, both the average number of information literacy sessions taught annually and the number of librarians with business as part of their job title decreased. responses showed an increase in the use of online tutorials for bil efforts. guth and sachs also found that more than half (58%) of their survey respondents had incorporated or were in the process of incorporating the standards in 2015, which is a notable increase from cooney’s survey in 2005. however, 39% of the 2015 respondents had incorporated the framework into their il efforts.   these surveys provide valuable information on how business librarians are approaching information literacy, but these responses also prompt additional questions that may be answered through a scoping review of the literature. examining the evidence available in the literature can provide deeper insight into these topics and serve as complementary evidence to inform the future direction of bil.   aims   this study utilizes the scoping review method in order to explore the following research question: how can the business information literacy literature be characterized regarding publication type, study design, findings, impact, and recommendations for practice? this scoping review aims to add to the evidence based literature in library and information science (lis), report on the current state of bil, and provide business librarians with insight that can be used to improve future information literacy efforts.   methods   scoping reviews are best used when the researcher wants to examine the nature of research activity in a particular field, summarize and disseminate findings, or identify gaps in the literature (arksey & o’malley, 2005). thus far, this method is not common in the lis discipline, aside from the health and medical librarianship subfield. it has, however, been used to explore mentoring programs for academic librarians (lorenzetti & powelson, 2015), implementation of web 2.0 services (gardois, colombi, grillo, & villanacci, 2012), individualized research consultations (fournier & sikora, 2015), researchers’ use of social network sites (kjellberg, haider, & sundin, 2016), and generational differences in library leadership (heyns, eldermire, & howard, 2019).   this method aims to “map the literature on a particular topic or research area and provide an opportunity to identify key concepts; gaps in the research; and types and sources of evidence to inform practice, policymaking, and research” (daudt, van mossel, & scott, 2013, p. 8). they differ from systematic reviews in a number of ways. scoping reviews may be designed around broader research questions. research quality may not be an initial priority. these studies may or may not include data extraction, and synthesis tends to be more qualitative (brien, lorenzetti, lewis, kennedy, & ghali, 2010). arksey and o’malley (2005) identify the following stages in their scoping study framework:   identify the research question(s) identify relevant studies select the studies chart the data collate, summarize, and report the results   the following sections describe each of these scoping review steps in the context of this study as well as an additional step we took in completing the review.   identify the research question   this study was designed to analyze the bil literature in order to identify trends in authorship, method, theory, research topic, findings, impact, and recommendations for practice.   identify relevant studies   in order to identify the databases to be searched, we used a list of the top 25 lis journals (nisonger & davis, 2005) and added two business librarianship-specific titles: journal of business and finance librarianship and ticker: the academic business librarianship review. we then identified the databases in which these 27 journals are indexed and conducted systematic searches. we searched the following databases for relevant literature published between january 2000 and december 2019: library and information science source, science direct, proquest central, project muse, and the ticker journal site. we searched for articles with “information literacy” and business or economics in the following fields: title, abstract, subject terms, and author-supplied keywords. we utilized database thesauri, when possible, as well as keyword searching.   select the studies   items were included in the review if they were published in peer reviewed journals or conference proceedings and focused on academic libraries. items about public or school libraries were excluded, as were items published in trade publications.   the lis literature tends to include a great deal of articles that simply describe practice. for example, the publication might describe a teaching method, newly developed learning object, or outreach effort. this type of literature, which we have classified as “practical applications,” may inform the practice of other librarians and thus was included in the scoping review. the goal of the study was to identify publication trends not to exclude non-rigorous work.   chart the data   the publication dataset was divided into three sections, and two of the three researchers coded each third. coding disagreements were settled by the third researcher. each publication was coded for publication title and type, document type, authorship and collaboration, study population, research methods, theories and models, topics, key findings, and recommendations. the dataset was stored in a spreadsheet that included document citations and fields for every item in table 1, with the exception of key findings and recommendations. qualitative data analysis software nvivo version 12 was used to code the publications, including key finding and recommendation text. some codes were selected prior to coding, but others emerged from the data throughout the coding process. the same 30 codes were used for topic, key findings, and recommendations, a list of which can be found in appendix a.   models and theories were coded for each publication only if they informed the study design or interpretation of the findings. merely mentioning a theory or model in a literature review without specific application was not enough to warrant coding. thirty research topics were used to code every publication, and each publication was assigned up to three topic codes.   collate and summarize the results   the dataset was analyzed to identify trends in topics, research populations, methods, and more. findings and recommendations that could inform the bil instruction practice of academic librarians were of particular interest.     table 1 publication feature types and items feature type item publication category (e.g., journal article, conference paper)   date of publication   research classification (e.g., original research, literature review) study design theory or model (e.g., grounded theory, technology acceptance model)   methods (e.g., interviews, surveys)   population (e.g., undergraduate business students, librarians) content topics (e.g., assessment, information-seeking behavior, workplace information literacy)   key findings   recommendations     figure 1 prisma flow diagram for bil scoping review.     cited reference search   in order to explore the impact of the publications included in the scoping review, we conducted a cited reference search. we searched for each publication in google scholar and recorded the number of times each had been cited. note that this part was an addition to the study design and not a step in the scoping review method.   results   the original searches outlined in the methods identified more than 1,200 articles, but after removing duplicates and out-of-scope articles, the final dataset included 135 publications. these 135 publications met the criteria for inclusion and were further analyzed. figure 1 provides more detail on the publication selection process in the form of a prisma flow diagram. see appendix b for the list of all included publications.   publication categories   of these 135 included publications, 132 (98%) were published in peer reviewed journals. although, it is important to note that not all of these articles presented original research, despite their peer reviewed status. forty-two different journal titles and two conference proceedings were represented. only four journals published five or more articles that met the study criteria, including the journal of academic librarianship (5 articles), journal of information literacy (8 articles), reference services review (15 articles), and journal of business & finance librarianship (49 articles). three papers published in conference proceedings met the study criteria and were included. two papers were published in procedia social and behavioral sciences and one in qualitative & quantitative research methods in libraries. a list of all titles can be found in appendix c.   date of publication   as demonstrated in figure 2, there has been a continued but irregular growth in the number of bil publications per year between january 2000 and december 2019. the average number of publications per year is 6.75, and publications on the topic peaked in 2012 and 2016, with fifteen publications each year.   research classification   of the 135 publications included in the study, 85 were identified as research articles (63%), 37 as “practical applications” publications (27%), nine as think pieces (7%), and four literature reviews (3%). any publication with a methods section was considered to be original research, although exceptions were made for non-u.s. publications that used alternative research paper terminology or format. if a methods section was clearly present but not labeled as such, it was included in the dataset. “practical applications” publications typically described a successful lesson plan, collaboration, or learning activity implemented by a library. think pieces are publications that usually include an extensive review of the literature but also the author’s analysis of or opinion on the topic. figure 3 shows the number of each document type published by year.   study population   publications were coded for study population if appropriate, including populations like undergraduate business students and business faculty. populations were identified in three publication types: original research, practical applications, and think pieces. for example, a practical applications publication might describe a new bil initiative that focused specifically on mba students, and so it would be coded with a population even though it was not a research study. sixty-one percent of the publications in the dataset studied undergraduate business students. some specified subgroups, such as first-year business students (14 publications), undergraduate marketing students (six publications), and undergraduate management students (six publications). twenty-six articles focused on master’s level graduate business students, and 15 of these 26 studied mba students specifically. of the 85 original research articles, 68% studied undergraduate business students and 20% studied graduate business students. the most common populations are listed in table 2. all population types outside of these four (e.g., corporate librarians, phd business students) appeared fewer than five times.     figure 2 bil publications per year, 2000–2019.     figure 3 document type by year, 2000–2019.     table 2 study populations with total number and percentages of appearances study populations total number of publications percentage of publications undergraduate business students 83 61% graduate business students (master’s level) 26 19% business faculty 8 6% business librarians 5 4%     authorship   a total of 263 authors from various disciplines and positions are represented in the study. author position (e.g., business librarian, lis faculty) was not always clear. authors were only coded when positions were specified in the article or in the database record, resulting in some authors being coded as unknown. fifty-two publications were published by a single author, and 83 publications were collaboratively authored. the most common type of collaboration involved librarian co-authorships (26) followed by at least one librarian and one business faculty member (25). interestingly, seven publications were authored solely by business faculty collaborations that did not include librarians. there was a steady increase in co-authored publications between 2000 and 2019 (see figure 4).   research methods   eighty-five publications used a research method to gather information related to bil. within this dataset, eight unique research methods were applied. surveys were by far the most common method, used in 72% of the original research publications. many studies used multiple types of surveys, and in fact there were five different survey types: il self-assessment, preand posttest, il skills assessment, feedback, and other. distinctions between the categories were as follows: il self-assessment surveys gauged student perceptions of their individual il skill levels (e.g., how comfortable are you identifying peer reviewed sources?). preand posttest surveys were distributed both before and after an instruction session or il intervention. il skills surveys focused on assessing il skill level (e.g., please identify the boolean operators in the following search statement.). feedback surveys requested input on a learning object or activity such as a research guide or lesson plan. the other survey category covered any survey that did not fit into those listed above. see figure 5 for more detail about the multiple types of surveys. additional methods included content analysis, interviews, case studies, and focus groups. nineteen publications utilized more than one research method, and 66 publications relied on one method only. the most popular research methods and the frequency of each can be found in table 3; all other methods appeared fewer than five times.     figure 4 number of publications with multiple authors by year, 2000–2019.     table 3 most popular research methods with number and percentage of publications in which they appeared research method total number of publications percentage of publications survey 61 72% content analysis 17 13% interviews 12 10% case study 10 7%     figure 5 percentages of surveys by type.     applied theories and models   only 15 of the 135 (11%) publications indicated use of a theory or model in informing their study design, and seven of those publications used more than one. only three models or theories appeared more than once, bloom’s taxonomy (jefferson, 2017; nentl & zietlow, 2008), adult learning theory (an & quail, 2018; quinn & leligdon, 2014), and the seven pillars of information literacy (mckinney & sen, 2012; webber & johnson, 2000). research topics   the top six codes applied were collaboration and faculty partnerships, teaching methods and strategies, assessment, il skills, information-seeking behavior, and online tutorials. the top ten topics can be seen in table 4. all other codes appeared nine or fewer times. see appendix a for the topics codebook.   key findings and recommendations   key findings were coded for original research articles. the top five key findings were related to il skills, instruction impact, student perceptions, information-seeking behavior, and online resources. the top ten key findings topics can be seen in table 5. some publications warranted the use of multiple codes related to the same idea. for example, “instruction impact” was used in conjunction with an additional code such as “evaluation of information” in order to reflect that 1) learning was self-reported and 2) learning was related to information evaluation. in a 2012 article, finley and waymire found that students self-reported an increased comfort level with “evaluating the credibility, accuracy, and validity of sources” (p. 34) after receiving il instruction. regarding the nesting of codes, evaluation of information is an il skill and thus might be considered part of that topic. however, publications are often focused on this specific skill, more so than other il skills. evaluation of information clearly emerged from the data as its own code.    fewer than half of the publications offered specific recommendations. the recommendations that did appear were most frequently related to collaboration/faculty partnerships, teaching methods/strategies, and assessment.     table 4 most popular research topics with number and percentage of publications in which they appeared research topic number of publications percentage of publications collaboration and faculty partnerships 47 35% teaching methods and strategies 46 34% assessment 42 31% il skills 20 15% information-seeking behavior 15 11% online tutorials 15 11% one-shot sessions 14 10% instruction impact 13 10% student perceptions 12 9% workplace il 12 9%     cited il standards and frameworks   this body of literature cited a variety of il standards and frameworks, including the australia and new zealand information literacy framework (anzil), association to advance collegiate schools of business (aascb) accreditation standards, society of college, national and university libraries (sconul) seven pillars of information literacy, association of college & research libraries (acrl) information literacy competency standards for higher education, acrl framework for information literacy for higher education, and brass’s business research competencies. overall, the following standards were cited most often: acrl standards (59 references), aascb standards (24 references), and acrl framework (16 references). figure 6 illustrates the number of citations per year for each of these. twenty-five publications cited more than one standard or framework. the business research competencies developed by brass, the business reference and services section within rusa (reference & user services association), were cited only twice.   cited reference search   in order to better understand the impact of the bil literature, a cited reference search was conducted in google scholar for all 135 publications. table 6 lists the top ten most highly cited publications from the dataset. there are, of course, numerous ways to measure the impact of a publication, but for the purposes of this study citations were chosen to illustrate the impact snapshot. in addition, it is important to note that some of the publications in the dataset were published recently and thus have not yet been cited frequently.     table 5 most popular key findings topics with number and percentage of publications in which they appeared, examples from the publications, and topic definitions key finding topics and definitions number of publications percentage of publications example from publications il skills: assessment or perception of the ability to evaluate, locate, or use information ethically 28 21% “generally speaking, librarians, library administrators, and faculty believe that students are lacking the necessary information literacy skills. this stands in contrast to the perceptions of many students, who tend to see their skills as well developed or adequate for completing school assignments” (detlor, julien, willson, serenko, & lavallee, 2011, p. 583). instruction impact: participant self-reported change in learning or understanding due to il instruction or learning object 23 17% “based on the quiz performance, it seems that the instructional videos did prepare students for the library instruction session by teaching basic business research concepts” (camacho, 2018, p. 33). student perceptions: participant self-reported learning or understanding of the library, librarian, or resources 16 12% “the feedback…indicated that this group of first year [business] students were comfortable with the prospect of undertaking library research and expected to be able to meet course research expectations” (matesic & adams, 2008, p. 7). evaluation of information: assessment of or self-reported information evaluation skills and/or behaviors 13 10% “prior studies have suggested that some employees do not always evaluate information . . . but this study found that 82% of all jobs mentioned evaluation skills” (gilbert, 2017, p. 127). information-seeking behavior: behaviors related to finding needed information inand outside of the library setting 13 10% “the results also confirmed the authors’ suspicions that students largely rely on web-based search engines, like google, to conduct their research” (bryant & hooper, 2017, p. 411). online resources: feedback on or reported use of online resources such as a database, website, or research guide 12 9% “research analysis found a range of attitudes toward the use of wikipedia in higher education, with all interviewees expressing a level of caution regarding its use” (bayliss, 2013, p. 49). workplace il: needed or used il skills in the workplace setting 12 9% “the university students who performed better on a commercial assessment of information literacy produced better emails, memos, and technical reports as reflected in their grade in a business communications course” (katz, haras, & blaszczynski, 2010, p. 146). assessment: measured student learning through a preand posttest or similar method 12 9% “across all four categories of knowledge including library usage experience, post-instruction session averages are significantly higher than pre-instruction session” (gong & loomis, 2009). collaboration, and faculty partnerships: identified collaboration within the library or institution in il efforts 10 7% “we found that successfully implementing the integration of il skills into the business curriculum was contingent upon the level of continuous institutional support and faculty commitment to the process” (rodríguez, cádiz, & penkova, 2018, p. 127). teaching methods and strategies: reported use of a specific teaching method or strategy used for il efforts 9 7% “this study confirms the findings from the library science literature that a research guide is effective when targeted to a class as a course page and there is concurrent instruction on how to use the page by the librarian” (leighton & may, 2013, p. 135).     figure 6 number of publications citing the acrl standards, acrl framework, and aacsb standards by year, 2000–2019.     table 6 ten most highly cited publications in this study with citation count number of times cited in google scholar full citation 490 johnston, b., & webber, s. (2003). information literacy in higher education: a review and case study. studies in higher education, 28(3), 335–352. https://doi.org/10.1080/03075070309295 482 webber, s., & johnston, b. (2000). conceptions of information literacy: new perspectives and implications. journal of information science, 26(6), 381–397. https://doi.org/10.1177/016555150002600602 203 williams, j., & chinn, s. j. (2009). using web 2.0 to support the active learning experience. journal of information systems education, 20(2), 165–174. available at http://jise.org/volume20/n2/jisev20n2p165.html 159 o’sullivan, c. (2002). is information literacy relevant in the real world? reference services review, 30(1), 7–14. https://doi.org/10.1108/00907320210416492 100 fiegen, a. m., cherry, b., & watson, k. (2002). reflections on collaboration: learning outcomes and information literacy assessment in the business curriculum. reference services review, 30(4), 307–318. https://doi.org/10.1108/00907320210451295   91 donaldson, k. a. (2000). library research success: designing an online tutorial to teach information literacy skills to first-year students. the internet and higher education, 2(4), 237–251. https://doi.org/10.1016/s1096-7516(00)00025-7   87 lombardo, s. v., & miree, c. e. (2003). caught in the web: the impact of library instruction on business students' perceptions and use of print and online resources. college & research libraries, 64(1), 6–22. https://doi.org/10.5860/crl.64.1.6   81 detlor, b., julien, h., willson, r., serenko, a., & lavallee, m. (2011). learning outcomes of information literacy instruction at business schools. journal of the american society for information science and technology, 62(3), 572–585. https://doi.org/10.1002/asi.21474   76 cooney, m., & hiris, l. (2003). integrating information literacy and its assessment into a graduate business course: a collaborative framework. research strategies, 19(3–4), 213–232. https://doi.org/10.1016/j.resstr.2004.11.002  75 klusek, l., & bornstein, j. (2006). information literacy skills for business careers: matching skills to the workplace. journal of business & finance librarianship, 11(4), 3–21. https://doi.org/10.1300/j109v11n04_02 discussion   competing il standards and frameworks   citation of the standards in bil peaked in 2012, more than a decade after they were adopted (see figure 6). adoption of the framework seems slow, and the citations have actually decreased steadily since 2016. this is potentially due to unfamiliarity with the document, which was finalized just four years ago, coupled with the lengthy scholarly publishing process. however, there may well be a spike in usage as more business librarians become knowledgeable about and comfortable with it. acrl has made a concerted effort to educate librarians on the framework and promote its use in the information literacy instruction classroom. the acrl publication disciplinary applications of information literacy threshold concepts (godbey, wainscott, & goodman, 2017) shared 25 examples of ways that subject librarians have successfully incorporated the framework into class content, and the book includes one example from business-related disciplines. the widely popular acrl sandbox, which is an open access repository where librarians can share lesson plans and activities that incorporate the framework, had 25 out of almost 225 lesson plans focused on business or economics at the time of this writing (acrl, 2020).     the aacsb standards were cited far less often than the standards but more often than the framework. while these standards do not specifically use the phrase “information literacy,” mcinnis bowers et al. (2009) point out that “four of the six curricular standards for quality management education put forth by aacsb international were closely tied to information-literacy skills, namely, communication abilities, ethical understanding and reasoning abilities, analytical skills, and use of information technology” (p. 113). more than three-fourths of the articles that cited the aacsb standards also cited the ala standards.     figure 7 original research and practical application publications by year, 2000–2019.     study design   research or practice?   in the bil literature, original research and practical applications are the two most common publication types. both original research and practical application publications generally increased in frequency between 2000 and 2019—although original research increased more. figure 7 shows a trend in the bil literature, beginning in 2010, in which original research was published more commonly than practical application publications practical applications publications are common in the overall lis literature, and the bil subset is no exception. these types of publications have been criticized for not being generalizable or rigorous (wilson, 2013, 2016). potential explanations for this trend in lis have been explored, and a main reason for this is the lack of formalized support for librarians to conduct their own research. babb summarizes the issue in this way: “research carried out by librarians was considered important for the profession, while often simultaneously considered extraneous to the individual jobs of librarians” (babb, 2017, p. ii). wilson (2016) notes that this issue is not unique to lis, and that all disciplines have a range of quality that appears in the literature. she recommends these six strategies or areas for improvement in lis research: confidence, collaboration, mentorship, education, recognizing that practice makes better, and developing specific research needs for specific areas of librarianship. it is important to keep in mind, however, that the practical applications publications are highly valued and used by librarians because they are, in fact, practitioners.   use of the survey method   the survey method is clearly popular with lis practitioners and researchers. the prevalence of the survey method is not surprising. a 2004 content analysis of “librarianship research” (koufogiannakis, slater, & crumley, 2004) and a 2018 systematic review of lis research (ullah & ameen, 2018) both found the questionnaire/survey to be the most common method. of the studies that used the survey method, many used multiple types of surveys. for example, camacho (2018) reported on a project in which librarians and business faculty collaborated on the development of instructional videos for a flipped classroom. the first survey tested the il skills of the students who had watched the video (e.g., “why are peer-reviewed articles considered authoritative?”) (p. 30). a second follow-up survey collected feedback on the new instructional videos (e.g., “what suggestions do you have for improving the videos in the future?”) (p. 33).   it seems that the survey method is often used to demonstrate impact and effectiveness in the classroom. half of the 62 survey method publications had assessment as a topic, and many shared key findings related to instruction impact (29 publications), il skills (26 publications), and student perceptions (24 publications). atwong and heichman taylor (2008), for example, developed a survey “to measure students' self-reported knowledge before and after a training module developed and conducted by librarian and faculty” in order to demonstrate instruction impact (p. 433). detlor et al. (2011) used the standardized il testing instrument sails, in conjunction with interviews, to study undergraduate business students. findings from this paper indicated that students were skilled at evaluating sources but struggled with search skills.   researchers most often used il self-assessment surveys and preand posttests to study undergraduate business students, and il self-assessment surveys and il skills surveys to study graduate business students. note that preand posttests and il skills surveys may ask the same types of questions (e.g., which words in the following list are boolean operators?), but the il skills survey is given just one time and the preand posttest is given before and after some sort of il intervention, such as a tutorial or one-shot session. for example, a business librarian and a communications librarian collaborated to develop new il instruction for undergraduate business students taking a public speaking course. preand posttest surveys using likert-scale responses measured the effectiveness of the il sessions. participants responded to statements such as “i feel comfortable accessing business-related information through the library” (nielsen & jetton, 2014, p. 347). in this case, the survey was both a preand posttest and also an il self-assessment. cooney and hiris (2003) developed an information literacy inventory, a survey instrument that combined il skills (e.g., “information posted on the internet is available for fair use and is not covered by copyright restrictions. true or false?”) and il self-assessment questions (e.g., “how would you rate your comfort level in conducting the research for the term paper required in this course?”) (p. 226, 227). the authors surveyed graduate business students taking a course on international financial markets and used the findings to develop bil instruction for the mba program.   focus on undergraduate business students   the bil literature is generally focused on improving instruction practice. business librarians tend to spend much of their teaching time with undergraduate students. in a 2019 survey, 90% of business librarian respondents reported teaching undergraduate students, and 54% reporting teaching graduate students (houlihan, wiley & click, 2019). thus, it is not surprising that undergraduate business students made up the study population in more than half of the publications in this dataset. stonebraker & fundator (2016) conducted a longitudinal study of undergraduate management students, using a preand posttest that “measured students’ knowledge of business resources, as well as students’ ability to recognize when different types of information are needed to answer specific business questions” (p. 440). in a departure from the heavy use of surveys in the bil literature, bauer (2018) used journaling, an ethnographic method. upper-level business undergraduate students kept journals about their research processes as they completed semester-long projects. findings showed that participants often struggled in the early stages of the research process, were concerned with the credibility of information, and understood that web searching alone was not sufficient for their assignments (bauer, 2018, p. 6).    authorship and collaboration   collaboration was a very common topic in the bil literature; 41% of the practical application and 31% of the original research publications were about collaboration or faculty partnerships. the most common types of author collaboration in this dataset were between two librarians or between a librarian and a business professor. librarian collaborators were more likely to publish practical application papers. original research publications were more likely to be authored by a librarian and business faculty. these findings support wilson’s (2016) recommendation, noted previously in the “research or practice?” section, that collaboration is an important strategy in improving the quality of lis research. librarian’s collaborative efforts tended to focus on teaching methods and strategies, which may explain why practical application publications are more common with this population. for example, librarians detmering and johnson (2011) describe the revision of bil instruction for an introductory course, “highlighting the importance of thinking critically throughout the information-seeking process” (p. 105) instead of demonstrating library tools. papers authored by librarian and business professor teams were, not surprisingly, often about collaboration and faculty partnerships. many of these publications focused on assessment efforts as well. in one case, a business librarian and an accounting professor collaborated to design a research assignment for a class on government and nonprofit accounting (finley & waymire, 2012). they assessed student il skills by analyzing the bibliographies of the first draft and final version of student papers. this article is notable because it described one of the few librarian/business faculty collaborations in which the librarian participated in the grading process.   interdisciplinary collaboration on research has many benefits. scholars can experience personal growth as they learn to approach research from a different perspective. they have the opportunity to learn about different methods, models, and theories. this type of work can be especially rewarding for business liaison librarians as they forge deeper connections with the faculty they work with and learn more about the business research landscape. in a recent study, tran and chan (in press) found that librarians are motivated to seek research collaborators for a number of reasons, including accessing needed expertise, seeking a sounding board, and sharing the research workload. respondents indicated that seeking collaborators in the workplace is a preferred strategy. these findings all support the idea that business librarians can benefit from collaborating with business faculty—and vice versa.      impact   a cited reference search was conducted in google scholar to identify the most impactful publications as illustrated in table 6. seven of the top ten publications were published between 2000 and 2003, which is to be expected; the longer a publication has been out, the more opportunity it has to be cited by other scholars. interestingly, five of the top ten publications were written by authors outside of the united states, including the top two. six of the most highly cited publications present original research.   it is also interesting to note that three of these publications appear in journals outside the lis field (studies in higher education, journal of information systems education, and the internet and higher education). more than one-third of the publications in the 135 paper dataset were published in the journal of business & finance librarianship, but only one of the top 10 most highly cited articles was published here. according to google scholar’s lis journal rankings, three of the journals represented here are considered top publications in the field: journal of the american society for information science and technology (jasist), journal of information science (jis), and college & research libraries (c&rl). in the complete dataset of 135 articles, these journals appear eight times total: three articles in jasist, three in c&rl, and two in jis. all eight were published more than five years ago, with the exception of one c&rl paper published in 2018.     recommendations for practice   while all of the publications shared findings or described experiences, many did not provide specific recommendations for practice. of those that did, however, these recommendations most commonly fell under one of the following categories: teaching methods and strategies, collaboration, or assessment.   teaching methods and strategies recommendations focused on the flipped classroom, problem-based learning, and the use of business models and concepts in il. cohen (2016) calls the flipped-instruction model a “catalyst for collaboration” and recommends bringing “disciplinary faculty ‘on board’ with homework assignments, in-class activities, assessment” and supporting technologies (p. 20). fiegen (2011), who reviewed 30 years of bil literature, advises librarians to adopt “a regular practice of preassignments” (p. 287). problem-based learning was also regularly endorsed. brock & tabaei (2011) recommend “using real-life problems and scenarios to encourage the development of information literacy skills” (p. 367), while devasagayam, johns-masten, and mccollum (2012) suggest “experiential exercises that demand involvement, engagement, application, and reinforcement through repetition” (p. 6). authors also recommend that librarians use methods, frameworks, and concepts that are familiar to business students when teaching bil. o’neill (2015) uses the business canvas model, a “popular tool for helping entrepreneurs plan and iterate their business concepts,” in the bil classroom (p. 458). others recommend using the case method, which students regularly encounter in their business classes, to teach bil concepts (spackman & camacho, 2009; stonebraker & howard, 2018).      the nature of teaching in this discipline is more practical than theoretical since bil requires a unique set of knowledge and search skills. the low number of theories and models used as well as the scant evidence for implementation of the framework could indicate that some librarians teaching business prioritize teaching disciplinary knowledge over more abstract information literacy concepts.   the many recommendations related to collaboration tended to be vague in nature, positing that collaboration between librarians and business faculty is important and necessary but giving few practical ideas for how to build these relationships. the literature does, however, identify some specific ways that librarians and business faculty can work together, including identifying resources for purchase (camacho, 2015), supporting experiential learning (griffis, 2014), identifying skills gaps (macy & coates, 2016), and developing il outcomes (stagg & kimmins, 2014).   the assessment recommendations ranged from general calls for more assessment to the recommendation of specific methods. as a result of her review of the bil literature, fiegan (2011) recommends preand posttests and graded assessments. in our study, we tracked the number of publications in which librarians were part of the grading process, and six met this criterion. examples of librarians participating in the grading process included strittmater’s (2012) study about a faculty-librarian collaboration in which the author creates online exercises and participates in the grading process. additionally, librarian-business professor team cooney and hiris (2003) collaboratively graded term papers for il related skills based on a checklist. other methods are recommended as well, including reflective writing (mckinney & sen, 2012), rubrics (mezick & harris, 2016), and systematic reviews (fiegen, 2010). sokoloff and simmons (2015) write about the value of citation analysis but note that “the method would elicit more meaningful results in the presence of other, complementary evidence” (p. 170).   conclusions   this scoping review was designed to explore the last two decades of bil research, in order to support lis practitioners in their evidence based practice. findings indicated a dependence on the survey method in bil research, a focus on collaboration between business librarians and business faculty, interest in new teaching methods, and a hesitation to implement the acrl framework in bil. with the introduction of the framework in 2015, all teaching librarians have the opportunity to rethink information literacy efforts based on this new paradigm. while there is an abundance of literature about the acrl framework and threshold concepts, relatively little literature exists that specifically focuses on how business librarians have utilized this document to improve information literacy assignments, lesson plans, learning activities, and assessments. further research on this topic would help inform efforts to integrate the framework into bil.   references   an, a., & quail, s. (2018). building bryt: a case study in developing an online toolkit to promote business information literacy in higher education. journal of library & information services in distance learning, 12(3/4), 71–89. https://doi.org/10.1080/1533290x.2018.1498615   arksey, h., & o'malley, l. (2005). scoping studies: towards a methodological framework. international journal of social research methodology, 8(1), 19–32. https://doi.org/10.1080/1364557032000119616   association of college & research libraries (acrl). (2000). information literacy competency standards for higher education. http://hdl.handle.net/11213/7668   association of college & research libraries (acrl). (2015). framework for information literacy for higher education. http://www.ala.org/acrl/standards/ilframework   association of college & research libraries (acrl). (2020). acrl framework for information literacy sandbox. https://sandbox.acrl.org/   atwong, c. t., & heichman taylor, l. j. (2008). integrating information literacy into business education: a successful case of faculty-librarian collaboration. journal of business & finance librarianship, 13(4), 433–448. https://doi.org/10.1080/08963560802202227   babb, m. n. (2017). an exploration of academic librarians as researchers within a university setting [master’s thesis, university of alberta]. education and research archive. https://era.library.ualberta.ca/items/15c87b25-5656-41c6-9242-42a0fb7ed74c   bauer, m. (2018). ethnographic study of business students' information-seeking behavior: implications for improved library practices. journal of business & finance librarianship, 23(1), 1–10. https://doi.org/10.1080/08963568.2018.1449557   bayliss, g. (2013). exploring the cautionary attitude toward wikipedia in higher education: implications for higher education institutions. new review of academic librarianship, 19(1), 36–57. https://doi.org/10.1080/13614533.2012.740439   brien, s. e., lorenzetti, d. l., lewis, s., kennedy, j., & ghali, w. a. (2010). overview of a formal scoping review on health system report cards. implementation science, 5(2). https://doi.org/10.1186/1748-5908-5-2   brock, s., & tabaei, s. (2011). library and marketing class collaborate to create next generation learning landscape. reference services review, 39(3), 362–368. https://doi.org/10.1108/00907321111161377   bryant, n. p., & hooper, r. s. (2017). learning to learn: using an embedded librarian to develop web-based legal information literacy for the business student. southern law journal, 27(2), 387–416.   camacho, l. (2015). the communication skills accounting firms desire in new hires. journal of business & finance librarianship, 20(4), 318–329. https://doi.org/10.1080/08963568.2015.1072895   camacho, l. (2018). if we built it, would they come? creating instruction videos with promotion in mind. journal of business & finance librarianship, 23(1), 26–34. https://doi.org/10.1080/08963568.2018.1431867   cohen, m. e. (2016). the flipped classroom as a tool for engaging discipline faculty in collaboration: a case study in library-business collaboration. new review of academic librarianship, 22(1), 5–23. https://doi.org/10.1080/13614533.2015.1073162   cooney, m. (2005). business information literacy instruction: a survey and progress report. journal of business & finance librarianship, 11(1), 3–25. https://doi.org/10.1300/j109v11n01_02   cooney, m., & hiris, l. (2003). integrating information literacy and its assessment into a graduate business course: a collaborative framework. research strategies, 19(3–4), 213–232. https://doi.org/10.1016/j.resstr.2004.11.002   daudt, h. m., van mossel, c., & scott, s. j. (2013). enhancing the scoping study methodology: a large, inter-professional team’s experience with arksey and o’malley’s framework. bmc medical research methodology, 13(48). https://doi.org/10.1186/1471-2288-13-48   detlor, b., julien, h., willson, r., serenko, a., & lavallee, m. (2011). learning outcomes of information literacy instruction at business schools. journal of the american society for information science and technology, 62(3), 572–585. https://doi.org/10.1002/asi.21474   detmering, r., & johnson, a. m. (2011). focusing on the thinking, not the tools: incorporating critical thinking into an information literacy module for an introduction to business course. journal of business & finance librarianship, 16(2), 101–107. https://doi.org/10.1080/08963568.2011.554771   devasagayam, r., johns-masten, k., & mccollum, j. (2012). linking information literacy, experiential learning, and student characteristics: pedagogical possibilities in business education. academy of educational leadership journal, 16(4).   donaldson, k. a. (2000). library research success: designing an online tutorial to teach information literacy skills to first-year students. the internet and higher education, 2(4), 237–251. https://doi.org/10.1016/s1096-7516(00)00025-7   fiegen, a. m. (2010). systematic review of research methods: the case of business instruction. reference services review, 38(3), 385–397. https://doi.org/10.1108/00907321011070883   fiegen, a. m. (2011). business information literacy: a synthesis for best practices. journal of business & finance librarianship, 16(4), 267–288. https://doi.org/10.1080/08963568.2011.606095    fiegen, a. m., cherry, b., & watson, k. (2002). reflections on collaboration: learning outcomes and information literacy assessment in the business curriculum. reference services review, 30(4), 307–318. https://doi.org/10.1108/00907320210451295   finley, w., & waymire, t. (2012). information literacy in the accounting classroom: a collaborative effort. journal of business & finance librarianship, 17(1), 34–50. https://doi.org/10.1080/08963568.2012.629566   fournier, k., & sikora, l. (2015). individualized research consultations in academic libraries: a scoping review of practice and evaluation methods. evidence based library and information practice, 10(4), 247–267. https://doi.org/10.18438/b8zc7w   gardois, p., colombi, n., grillo, g., & villanacci, m. c. (2012). implementation of web 2.0 services in academic, medical and research libraries: a scoping review. health information & libraries journal, 29(2), 90–109. https://doi.org/10.1111/j.1471-1842.2012.00984.x   godbey, s., wainscott, s. & goodman, x. (2017). disciplinary applications of information literacy threshold concepts. chicago, il: association of college and research libraries. https://www.alastore.ala.org/content/disciplinary-applications-information-literacy-threshold-concepts   gilbert, s. (2017). information literacy skills in the workplace: examining early career advertising professionals. journal of business & finance librarianship, 22(2), 111–134. https://doi.org/10.1080/08963568.2016.1258938   gong, x., & loomis, m. (2009). an empirical study on follow-up library instruction sessions in the classroom. electronic journal of academic and special librarianship, 10(1). https://southernlibrarianship.icaap.org/content/v10n01/gong_x01.html     griffis, p. j. (2014). information literacy in business education experiential learning programs. journal of business & finance librarianship, 19(4), 333–341. https://doi.org/10.1080/08963568.2014.952987   guth, l., & sachs, d. e. (2018). national trends in adoption of acrl information literacy guidelines and impact on business instruction practices: 2003–2015. journal of business & finance librarianship, 1–23. https://doi.org/10.1080/08963568.2018.1467169   heyns, e. p., eldermire, e. r. b., & howard, h. a. (2019). unsubstantiated conclusions: a scoping review on generational differences of leadership in academic libraries. journal of academic librarianship, 45(5). https://doi.org/10.1016/j.acalib.2019.102054 houlihan, m., wiley, c., & click, a. (2019). information literacy survey of business librarians [unpublished raw data].   jefferson, c. o. (2019). good for business: applying the acrl framework threshold concepts to teach a learner-centered business research course. ticker: the academic business librarianship review, 2(1), 1–17. https://doi.org/10.3998/ticker.16481003.0002.101   johnston, b., & webber, s. (2003). information literacy in higher education: a review and case study. studies in higher education, 28(3), 335–352. https://doi.org/10.1080/03075070309295   katz, i. r., haras, c., & blaszczynski, c. (2010). does business writing require information literacy? business communication quarterly, 73(2), 135–149. https://doi.org/10.1177/1080569910365892   kjellberg, s., haider, j., & sundin, o. (2016). researchers' use of social network sites: a scoping review. library & information science research, 38(3), 224–234. https://doi.org/10.1016/j.lisr.2016.08.008   klusek, l., & bornstein, j. (2006). information literacy skills for business careers: matching skills to the workplace. journal of business & finance librarianship, 11(4), 3–21. https://doi.org/10.1300/j109v11n04_02   koufogiannakis d. (2012). the state of systematic reviews in library and information studies. evidence based library and information practice, 7(2), 91–95. https://doi.org/10.18438/b8q021    koufogiannakis, d., slater, l., & crumley, e. (2004). a content analysis of librarianship research. journal of information science, 30(3), 227–239. https://doi.org/10.1177/0165551504044668   leighton, h. v., & may, d. (2013). the library course page and instruction: perceived helpfulness and use among students. internet reference services quarterly, 18(2), 127–138. https://doi.org/10.1080/10875301.2013.804019   lombardo, s. v., & miree, c. e. (2003). caught in the web: the impact of library instruction on business students' perceptions and use of print and online resources. college & research libraries, 64(1), 6–22. https://doi.org/10.5860/crl.64.1.6   lorenzetti, d. l., & powelson, s. e. (2015). a scoping review of mentoring programs for academic librarians. the journal of academic librarianship, 41(2), 186–196. https://doi.org/10.1016/j.acalib.2014.12.001   mcinnis bowers, c. v., chew, b., bowers, m. r., ford, c. e., smith, c., & herrington, c. (2009). interdisciplinary synergy: a partnership between business and library faculty and its effects on students’ information literacy. journal of business & finance librarianship, 14(2), 110–127. https://doi.org/10.1080/08963560802362179   macy, k. v., & coates, h. l. (2016). data information literacy instruction in business and public health. ifla journal, 42(4), 313–327. https://doi.org/10.1177/0340035216673382   matesic, m. a., & adams, j. m. (2008). provocation to learn: a study in the use of personal response systems in information literacy instruction. partnership: the canadian journal of library and information practice and research, 3(1), 1–14. https://doi.org/10.21083/partnership.v3i1.327   mckinney, p., & sen, b. a. (2012). reflection for learning: understanding the value of reflective writing for information literacy development. journal of information literacy, 6(2), 110–129. https://ojs.lboro.ac.uk/jil/article/view/llc-v6-i2-2012-5    mezick, e. m., & hiris, l. (2016). using rubrics for assessing information literacy in the finance classroom: a collaboration. journal of business & finance librarianship, 21(2), 95–113. https://doi.org/10.1080/08963568.2016.1169970   national center for education statistics. (2017). bachelor's degrees conferred by postsecondary institutions, by field of study: selected years, 1970-71 through 2016-17. digest of educational statistics. https://nces.ed.gov/programs/digest/d18/tables/dt18_322.10.asp   nielsen, j., & jetton, l. l. (2014). assessing the effectiveness of collaborative subject specific library instruction. qualitative and quantitative methods in libraries, 3(1), 343–350. http://qqml-journal.net/index.php/qqml/article/view/143   nentl, n., & zietlow, r. (2008). using bloom's taxonomy to teach critical thinking skills to business students. college & undergraduate libraries, 15(1–2), 159–172. https://doi.org/10.1080/10691310802177135   nisonger, t., & davis, c. (2005). the perception of library and information science journals by lis education deans and arl library directors: a replication of the kohl–davis study. college & research libraries, 66(4), 341–377. https://doi.org/10.5860/crl.66.4.341   o’neill, t. w. (2015). the business model canvas as a platform for business information literacy instruction. reference services review, 43(3), 450–460. https://doi.org/10.1108/rsr-02-2015-0013   o’sullivan, c. (2002). is information literacy relevant in the real world? reference services review, 30(1), 7–14. https://doi.org/10.1108/00907320210416492   rodríguez, k., cádiz, l., & penkova, s. (2018). integration of information literacy skills into the core business curriculum at the university of puerto rico río piedras. journal of business & finance librarianship, 23(2), 117–130. https://doi.org/10.1080/08963568.2018.1467168   spackman, a., & camacho, l. (2009). rendering information literacy relevant: a case-based pedagogy. the journal of academic librarianship, 35(6), 548–554. https://doi.org/10.1016/j.acalib.2009.08.005    stonebraker, i., & fundator, r. (2016). use it or lose it? a longitudinal performance assessment of undergraduate business students' information literacy. the journal of academic librarianship, 42(4), 438–444. https://doi.org/10.1016/j.acalib.2016.04.004   stonebraker, i., & howard, h. a. (2018). evidence-based decision-making: awareness, process and practice in the management classroom. the journal of academic librarianship, 44(1), 113–117. https://doi.org/10.1016/j.acalib.2017.09.017   sokoloff, j., & simmons, r. (2015). evaluating citation analysis as a measurement of business librarian consultation impact. journal of business & finance librarianship, 20(3), 159–171. https://doi.org/10.1080/08963568.2015.1046783   stagg, a., & kimmins, l. (2014). first year in higher education (fyhe) and the coursework post-graduate student. the journal of academic librarianship, 40(2), 142–151. https://doi.org/10.1016/j.acalib.2014.02.005   strittmatter, c. (2012). developing and assessing a library instruction module for a core business class. journal of business & finance librarianship, 17(1), 95–105. –https://doi.org/10.1080/08963568.2012.630645    quinn, t., & leligdon, l. (2014). executive mba students’ information skills and knowledge: discovering the difference between work and academics. journal of business & finance librarianship, 19(3), 234–255. https://doi.org/10.1080/08963568.2014.916540   tran, n-y., & chan, e. (in press). seeking and finding research collaborators: an exploratory study of librarian motivations, strategies, and success rates. college & research libraries. https://works.bepress.com/yen-tran/25/   ullah, a., & ameen, k. (2018). account of methodologies and methods applied in lis research: a systematic review. library & information science research, 40(1), 53–60. https://doi.org/10.1016/j.lisr.2018.03.002   webber, s., & johnston, b. (2000). conceptions of information literacy: new perspectives and implications. journal of information science, 26(6), 381–397. https://doi.org/10.1177/016555150002600602   williams, j., & chinn, s. j. (2009). using web 2.0 to support the active learning experience. journal of information systems education, 20(2), 165–174. available at http://jise.org/volume20/n2/jisev20n2p165.html   wilson, v. (2013). formalized curiosity: reflecting on the librarian practitioner-researcher. evidence based library and information practice, 8(1), 111–117. https://doi.org/10.18438/b8zk6k   wilson, v. (2016). librarian research: making it better? evidence based library and information practice, 11(1), 111–114. https://doi.org/10.18438/b8vd0n     appendix a codebook for research topics, key findings, and recommendations   active learning assessment case study (student assignment) client-based projects/student consulting/problem-based learning collaboration/faculty partnerships credit-bearing courses critical thinking data literacy embedded librarianship evaluation of information financial literacy acrl framework information literacy skills information literacy standards information access information seeking behavior instruction impact international libraries non-traditional students one-shot sessions online resources online teaching online tutorials orientation outreach reference services scholarly communication student perceptions teaching methods & strategies technology     appendix b all included publications   akhras, c. (2013). interactive technology: enhancing business students’ content literacy. procedia social and behavioral sciences, 83, 332–336. https://doi.org/10.1016/j.sbspro.2013.06.065   an, a., & quail, s. (2018). building bryt: a case study in developing an online toolkit to promote business information literacy in higher education. journal of library & information services in distance learning, 12(3/4), 71–89. https://doi.org/10.1080/1533290x.2018.1498615   artemchik, t. (2016). using the instructional design process in tutorial development. reference services review, 44(3), 309–323. https://doi.org/10.1108/rsr-12-2015-0050   atwong, c. t., & heichman taylor, l. j. (2008). integrating information literacy into business education: a successful case of faculty-librarian collaboration. journal of business & finance librarianship, 13(4), 433–448. https://doi.org/10.1080/08963560802202227   bauer, m. (2018). ethnographic study of business students' information-seeking behavior: implications for improved library practices. journal of business & finance librarianship, 23(1), 1–10. https://doi.org/10.1080/08963568.2018.1449557   baxter, k., johnson, b., & chisholm, k. (2016). evaluating and developing an information literacy programme for mba students. new zealand library & information management journal, 56(1), 30–45. https://ndhadeliver.natlib.govt.nz/delivery/deliverymanagerservlet?dps_pid=ie26797875   bayliss, g. (2013). exploring the cautionary attitude toward wikipedia in higher education: implications for higher education institutions. new review of academic librarianship, 19(1), 36–57. https://doi.org/10.1080/13614533.2012.740439   booker, l. d., detlor, b., & serenko, a. (2012). factors affecting the adoption of online library resources by business students. journal of the american society for information science and technology, 63(12), 2503–2520. https://doi.org/10.1002/asi.22723   borg, m., & stretton, e. (2009). my students and other animals. or a vulture, an orb weaver spider, a giant panda and 900 undergraduate business students .... journal of information literacy, 3(1), 19–30. https://ojs.lboro.ac.uk/jil/article/view/219   boss, k., & drabinski, e. (2014). evidence-based instruction integration: a syllabus analysis project. reference services review, 42(2), 263–276. https://doi.org/10.1108/rsr-07-2013-0038   bravo, r., lucia, l., & martin, m. j. (2013). assessing a web library program for information literacy learning. reference services review, 41(4), 623–638. https://doi.org/10.1108/rsr-05-2013-0025   brock, s., & tabaei, s. (2011). library and marketing class collaborate to create next generation learning landscape. reference services review, 39(3), 362–368. https://doi.org/10.1108/00907321111161377   brodsky, m. (2019). the role of business librarians in teaching data literacy. ticker: the academic business librarianship review, 1(3), 1–6. https://doi.org/10.3998/ticker.16481003.0001.301    bryant, n. p., & hooper, r. s. (2017). learning to learn: using an embedded librarian to develop web-based legal information literacy for the business student. southern law journal, 27(2), 387–416.   camacho, l. (2015). the communication skills accounting firms desire in new hires. journal of business & finance librarianship, 20(4), 318–329. https://doi.org/10.1080/08963568.2015.1072895   camacho, l. (2018). if we built it, would they come? creating instruction videos with promotion in mind. journal of business & finance librarianship, 23(1), 26–34. https://doi.org/10.1080/08963568.2018.1431867   campbell, d. k. (2011). broad focus, narrow focus: a look at information literacy across a school of business and within a capstone course. journal of business & finance librarianship, 16(4), 307–325. https://doi.org/10.1080/08963568.2011.605622   cochrane, c. (2006). embedding information literacy in an undergraduate management degree: lecturers' and students' perspectives. education for information, 24(2-3), 97–123. https://doi.org/10.3233/efi-2006-242-301   cohen, m. e. (2016). the flipped classroom as a tool for engaging discipline faculty in collaboration: a case study in library-business collaboration. new review of academic librarianship, 22(1), 5–23. https://doi.org/10.1080/13614533.2015.1073162   cohen, m. e., poggiali, j., lehner-quam, a., wright, r., & west, r. k. (2016). flipping the classroom in business and education one-shot sessions: a research study. journal of information literacy, 10(2), 40–63. https://ojs.lboro.ac.uk/jil/article/view/pra-v10-i2-3   conley, t. m., & gil, e. l. (2011). information literacy for undergraduate business students: examining value, relevancy, and implications for the new century. journal of business & finance librarianship, 16(3), 213–228. https://doi.org/10.1080/08963568.2011.581562   cooney, m. (2005). business information literacy instruction: a survey and progress report. journal of business & finance librarianship, 11(1), 3–25. https://doi.org/10.1300/j109v11n01_02   cooney, m., & hiris, l. (2003). integrating information literacy and its assessment into a graduate business course: a collaborative framework. research strategies, 19(3-4), 213–232. https://doi.org/10.1016/j.resstr.2004.11.002   costa, c. (2009). use of online information resources by rmit university economics, finance, and marketing students participating in a cooperative education program. australian academic & research libraries, 40(1), 36–49. https://doi.org/10.1080/00048623.2009.10721377   cramer, s., campbell, d. k., & scanlon, m. g. (2019). increasing research quality in entrepreneurial students: best practices in faculty-librarian partnerships. ticker: the academic business librarianship review, 2(1), 18–25. https://doi.org/10.3998/ticker.16481003.0002.102   cullen, a. (2013). using the case method to introduce information skill development in the mba curriculum. journal of business & finance librarianship, 18(3), 208–232. https://doi.org/10.1080/08963568.2013.795740    cullen, j. g. (2013). applying radar with new business postgraduates. journal of information science, 40(1), 25–27. https://doi.org/10.1177/0165551513510094   cunningham, n. a., & anderson, s. c. (2005). a bridge to fars and information literacy for accounting undergraduates. journal of business & finance librarianship, 10(3), 3–16. https://doi.org/10.1300/j109v10n03_02    d’angelo, b. j. (2001). using source analysis to promote critical thinking. research strategies, 18(4), 303–309. https://doi.org/10.1016/s0734-3310(03)00006-5   decarie, c. (2012). dead or alive: information literacy and dead(?) celebrities. business communication quarterly, 75(2), 166–172. https://doi.org/10.1177/1080569911432737   detlor, b., julien, h., willson, r., serenko, a., & lavallee, m. (2011). learning outcomes of information literacy instruction at business schools. journal of the american society for information science and technology, 62(3), 572–585. https://doi.org/10.1002/asi.21474   detmering, r., & johnson, a. m. (2011). focusing on the thinking, not the tools: incorporating critical thinking into an information literacy module for an introduction to business course. journal of business & finance librarianship, 16(2), 101–107. https://doi.org/10.1080/08963568.2011.554771   devasagayam, r., johns-masten, k., & mccollum, j. (2012). linking information literacy, experiential learning, and student characteristics: pedagogical possibilities in business education. academy of educational leadership journal, 16(4).   donaldson, k. a. (2000). library research success: designing an online tutorial to teach information literacy skills to first-year students. the internet and higher education, 2(4), 237–251. https://doi.org/10.1016/s1096-7516(00)00025-7   dubicki, e. (2010). research behavior patterns of business students. reference services review, 38(3), 360–384. https://doi.org/10.1108/00907321011070874   dunaway, m. k., & orblych, m. t. (2011). formative assessment: transforming information literacy instruction. reference services review, 39(1), 24–41. https://doi.org/10.1108/00907321111108097   feast, v. (2003). integration of information literacy skills into business courses. reference services review, 31(1), 81–95. https://doi.org/10.1108/00907320310460942   fiegen, a. m. (2010). systematic review of research methods: the case of business instruction. reference services review, 38(3), 385–397. https://doi.org/10.1108/00907321011070883   fiegen, a. m. (2011). business information literacy: a synthesis for best practices. journal of business & finance librarianship, 16(4), 267–288. https://doi.org/10.1080/08963568.2011.606095   fiegen, a. m., cherry, b., & watson, k. (2002). reflections on collaboration: learning outcomes and information literacy assessment in the business curriculum. reference services review, 30(4), 307–318. https://doi.org/10.1108/00907320210451295   finley, w., & aurand, t. w. (2019). the business school undergraduate: an explorative study of exposure to writing and secondary research terminology. journal of higher education theory and practice, 19(4), 121–128. https://doi.org/10.33423/jhetp.v19i4.2207   finley, w., & waymire, t. (2012). information literacy in the accounting classroom: a collaborative effort. journal of business & finance librarianship, 17(1), 34–50. https://doi.org/10.1080/08963568.2012.629566   frand, j. l., borah, e. g., & lippincott, a. (2008). infoiq: targeting information and technology lifelong needs. public services quarterly, 3(3-4), 95–113. https://doi.org/10.1080/15228950802110486   friehs, c. g., & craig, c. l. (2008). assessing the effectiveness of online library instruction with finance students. journal of web librarianship, 2(4), 493–509. https://doi.org/10.1080/19322900802484438   gil, e. l. (2017). maximizing and assessing a one-shot information literacy session: a case study. journal of business & finance librarianship, 22(2), 97–110. https://doi.org/10.1080/08963568.2017.1285748   gil, e. l. (2019). introducing information literacy into a marketing budget class assignment: a case study. journal of business & finance librarianship, 24(1/2), 1–16. https://doi.org/10.1080/08963568.2019.1632646   gilbert, s. (2017). information literacy skills in the workplace: examining early career advertising professionals. journal of business & finance librarianship, 22(2), 111–134. https://doi.org/10.1080/08963568.2016.1258938   gong, x., & loomis, m. (2009). an empirical study on follow-up library instruction sessions in the classroom. electronic journal of academic and special librarianship, 10(1). https://southernlibrarianship.icaap.org/content/v10n01/gong_x01.html   griffis, p. j. (2014). information literacy in business education experiential learning programs. journal of business & finance librarianship, 19(4), 333–341. https://doi.org/10.1080/08963568.2014.952987   gunasekara, c. s. (2008). fostering independent learning and critical thinking in management higher education using an information literacy framework. journal of information literacy, 2(2), 74–85. https://ojs.lboro.ac.uk/jil/article/view/art-v2-i2-2008-1   gunasekara, c., & gerts, c. (2017). enabling authentic assessment: the essential role of information literacy. journal of the australian library and information association, 66(4), 393–405. https://doi.org/10.1080/24750158.2017.1399014   gunn, m., & miree, c. e. (2012). business information literacy teaching at different academic levels: an exploration of skills and implications for instructional design. journal of information literacy, 6(1), 18–34. https://ojs.lboro.ac.uk/jil/article/view/pra-v6-i1-2012-2   guth, l., & sachs, d. e. (2018). national trends in adoption of acrl information literacy guidelines and impact on business instruction practices: 2003–2015. journal of business & finance librarianship, 23(2), 131–153. https://doi.org/10.1080/08963568.2018.1467169   henninger, e. a., & hurlbert, j. m. (2007). using the seven principles for good practice in undergraduate education: a framework for teaching cultural diversity in a management course. journal of business & finance librarianship, 12(2), 3–15. https://doi.org/10.1300/j109v12n02_02   hesseldenz, p. (2012). information literacy and the evolving mba degree. journal of business & finance librarianship, 17(4), 287-299. https://doi.org/10.1080/08963568.2012.712496   holler, c. (2008). incorporating vendor-created training into information literacy instruction: a case study. communications in information literacy, 2(2), 99–108. https://doi.org/10.15760/comminfolit.2009.2.2.61   holler, c. m. (2009). client-based projects and the acrl information literacy competency standards for higher education. behavioral & social sciences librarian, 28(1-2), 7–22. https://doi.org/10.1080/01639260902862041   hossain shoeb, z. (2011). information literacy competency of freshman business students of a private university in bangladesh. library review, 60(9), 762–772. https://doi.org/10.1108/00242531111176781   howard, h. a., wood, n., & stonebraker, i. (2018). mapping information literacy using the business research competencies. reference services review, 46(4), 543–564. https://doi.org/10.1108/rsr-12-2017-0048   hristova, m., & miree, c. (2013). longitudinal update: business information literacy teaching at different academic levels. journal of information literacy, 7(2), 153–155. https://ojs.lboro.ac.uk/jil/article/view/prj-v7-i2-2013-2   jefferson, c. o. (2019). good for business: applying the acrl framework threshold concepts to teach a learner-centered business research course. ticker: the academic business librarianship review, 2(1), 1–17. https://doi.org/10.3998/ticker.16481003.0002.101   johnston, b., & webber, s. (2003). information literacy in higher education: a review and case study. studies in higher education, 28(3), 335–352. https://doi.org/10.1080/03075070309295   julien, h., detlor, b., serenko, a., willson, r., & lavallee, m. (2011). preparing tomorrow's decision makers: learning environments and outcomes of information literacy instruction in business schools. journal of business & finance librarianship, 16(4), 348–367. https://doi.org/10.1080/08963568.2011.605669   katz, i. r., haras, c., & blaszczynski, c. (2010). does business writing require information literacy? business communication quarterly, 73(2), 135–149. https://doi.org/10.1177/1080569910365892   keiser, b. e. (2016). how information literate are you? a self-assessment by students enrolled in a competitive intelligence elective. journal of business & finance librarianship, 21(3–4), 210–228. https://doi.org/10.1080/08963568.2016.1226613   kelly, a. s., williams, t. d., matthies, b., & orris, j. b. (2011). course-integrated information literacy instruction in introduction to accounting. journal of business & finance librarianship, 16(4), 326–347. https://doi.org/10.1080/08963568.2011.605651   kirkwood, h., & dejoie, r. (2016). business information literacy integration into a freshman summer bridge course for underrepresented minorities. journal of business & finance librarianship, 21(3–4), 198–209. https://doi.org/10.1080/08963568.2016.1226611   klusek, l., & bornstein, j. (2006). information literacy skills for business careers: matching skills to the workplace. journal of business & finance librarianship, 11(4), 3–21. https://doi.org/10.1300/j109v11n04_02   lahlafi, a. e., rushton, d., & stretton, e. (2012). active and reflective learning initiatives to improve web searching skills of business students. journal of information literacy, 6(1), 35–49. https://ojs.lboro.ac.uk/jil/article/view/pra-v6-i1-2012-3   leavitt, l. (2016). taking the plunge! a case study in teaching a credit-bearing information literacy course to business undergraduate students. journal of business & finance librarianship, 21(3-4), 274–287. https://doi.org/10.1080/08963568.2016.1226617   leigh, j. s. a., & gibbon, c. a. (2008). information literacy and the introductory management classroom. journal of management education, 32(4), 509–530. https://doi.org/10.1177/1052562908317023   leighton, h. v., & may, d. (2013). the library course page and instruction: perceived helpfulness and use among students. internet reference services quarterly, 18(2), 127–138. https://doi.org/10.1080/10875301.2013.804019   li, j. (2012). serving as an educator: a southern case in embedded librarianship. journal of business & finance librarianship, 17(2), 133–152. https://doi.org/10.1080/08963568.2012.661198   lipu, s., & hill, a. (2005). high level learning by design: the nuts and bolts of assessment and evaluation in a doctorate of business administration program. the australian library journal, 54(3), 200–225. https://doi.org/10.1080/00049670.2005.10721758   lombardo, s. v., & miree, c. e. (2003). caught in the web: the impact of library instruction on business students' perceptions and use of print and online resources. college & research libraries, 64(1), 6–22. https://doi.org/10.5860/crl.64.1.6   long, c. m., & shrikhande, m. m. (2005). improving information-seeking behavior among business majors. research strategies, 20(4), 357–369. https://doi.org/10.1016/j.resstr.2006.12.022   macy, k. v., & coates, h. l. (2016). data information literacy instruction in business and public health. ifla journal, 42(4), 313–327. https://doi.org/10.1177/0340035216673382   magi, t. j. (2003). what's best for students? comparing the effectiveness of a traditional print pathfinder and a web-based research tool. portal: libraries and the academy, 3(4), 671–686. https://doi.org/10.1353/pla.2003.0090   majid, s., hayat, i., patel, r. p., & vijayaraghavan, s. (2012). information needs and seeking behaviour of business students. singapore journal of library & information management, 41, 14–35.   makani-lim, b., agee, a., wu, d., & easter, m. (2014). research in action: using rubrics to assess information literacy skills in business education. journal of business and educational leadership, 5(1), 3–17. https://www.questia.com/library/journal/1p3-3554636121/research-in-action-using-rubrics-to-assess-information   matesic, m. a., & adams, j. m. (2008). provocation to learn: a study in the use of personal response systems in information literacy instruction. partnership: the canadian journal of library and information practice and research, 3(1), 1–14. https://doi.org/10.21083/partnership.v3i1.327   mcfarland, d., & chandler, s. (2002). “plug and play” in context: reflections on a distance information literacy unit. journal of business & finance librarianship, 7(2–3), 115–129. https://doi.org/10.1300/j109v07n02_10   mcinnis bowers, c. v., chew, b., bowers, m. r., ford, c. e., smith, c., & herrington, c. (2009). interdisciplinary synergy: a partnership between business and library faculty and its effects on students’ information literacy. journal of business & finance librarianship, 14(2), 110–127. https://doi.org/10.1080/08963560802362179   mckinney, p., & sen, b. a. (2012). reflection for learning: understanding the value of reflective writing for information literacy development. journal of information literacy, 6(2), 110–129. https://ojs.lboro.ac.uk/jil/article/view/llc-v6-i2-2012-5     mezick, e. m., & hiris, l. (2016). using rubrics for assessing information literacy in the finance classroom: a collaboration. journal of business & finance librarianship, 21(2), 95–113. https://doi.org/10.1080/08963568.2016.1169970   michalak, r., & rysavy, m. d. (2016). information literacy in 2015: international graduate business students' perceptions of information literacy skills compared to test-assessed skills. journal of business & finance librarianship, 21(2), 152–174. https://doi.org/10.1080/08963568.2016.1145787   natt, a. (2013). what is business information literacy and can the corporate librarian contribute anything to the discourse? journal of business & finance librarianship, 18(2), 146–174. https://doi.org/10.1080/08963568.2013.769290   nentl, n., & zietlow, r. (2008). using bloom's taxonomy to teach critical thinking skills to business students. college & undergraduate libraries, 15(1–2), 159–172. https://doi.org/10.1080/10691310802177135   nielsen, j., & jetton, l. l. (2014). assessing the effectiveness of collaborative subject-specific library instruction. qualitative and quantitative methods in libraries, 3(1), 343–350. http://qqml-journal.net/index.php/qqml/article/view/143   o’neill, t. w. (2015). the business model canvas as a platform for business information literacy instruction. reference services review, 43(3), 450–460. https://doi.org/10.1108/rsr-02-2015-0013   o’sullivan, c. (2002). is information literacy relevant in the real world? reference services review, 30(1), 7–14. https://doi.org/10.1108/00907320210416492   orme, w. a. (2004). a study of the residual impact of the texas information literacy tutorial on the information-seeking ability of first year college students. college & research libraries, 65(3), 205–215. https://doi.org/10.5860/crl.65.3.205   perkins, d., & solis, e. (2017). traveling forward: revisiting online instruction for global sites. public services quarterly, 13(4), 304–308. https://doi.org/10.1080/15228959.2017.1357517   petermanec, z., & šebjan, u. (2018). the impact of components of information literacy on student success in higher education. knjiznica, 62(1–2), 151–168.   phillips, c. m. h. (2010). students’ research experiences during consulting projects: three themes emerging from case studies. behavioral & social sciences librarian, 29(2), 91–108. https://doi.org/10.1080/01639261003742157   price, r., becker, k., clark, l., & collins, s. (2011). embedding information literacy in a first-year business undergraduate course. studies in higher education, 36(6), 705. https://doi.org/10.1080/03075071003725350   quinn, t., & leligdon, l. (2014). executive mba students’ information skills and knowledge: discovering the difference between work and academics. journal of business & finance librarianship, 19(3), 234–255. https://doi.org/10.1080/08963568.2014.916540   rae, s., & hunn, m. (2015). assessing the impact of embedding online academic and information literacy resources into a first year business course. evidence based library and information practice, 10(4), 95–112. https://doi.org/10.18438/b80c76   rae, s., hunn, m., & lobo, a. (2019). sustainable, continuous improvement in online academic and information literacy support. journal of the australian library & information association, 68(1), 68–77. https://doi.org/10.1080/24750158.2018.1561600   reilly, p. (2015). strategies to ensure deep learning occurs in mba information literacy workshops. journal of business & finance librarianship, 20(3), 172–188. https://doi.org/10.1080/08963568.2015.1042812   reilly, p. (2017). creative approaches to teaching graduate research methods workshops. nordic journal of information literacy in higher education, 9(1), 21–36. https://doi.org/10.15845/noril.v9i1.232    reiter, l., & huffman, j. p. (2016). yes virginia, it will scale: using data to personalize high-volume reference interactions. the journal of academic librarianship, 42(1), 21–26. https://doi.org/10.1016/j.acalib.2015.09.011   rodríguez, k., cádiz, l., & penkova, s. (2018). integration of information literacy skills into the core business curriculum at the university of puerto rico río piedras. journal of business & finance librarianship, 23(2), 117-130. https://doi.org/10.1080/08963568.2018.1467168   ross, m., perkins, h., & bodey, k. (2013). information literacy self-efficacy: the effect of juggling work and study. library & information science research, 35(4), 279–287. https://doi.org/10.1016/j.lisr.2013.04.008   rushton, d., & lahlafi, a. (2013). development, implementation and impact of active and reflective learning initiatives to improve web searching skills of international business students at sheffield hallam university, uk. procedia social and behavioral sciences, 93, 885–894. https://doi.org/10.1016/j.sbspro.2013.09.298   serenko, a., detlor, b., julien, h., & booker, l. d. (2012). a model of student learning outcomes of information literacy instruction in a business school. journal of the association for information science and technology, 63(4), 671–686. https://doi.org/10.1002/asi.22606   simon, c. (2009). graduate business students and business information literacy: a novel approach. journal of business & finance librarianship, 14(3), 248–267. https://doi.org/10.1080/08963560802361981   skipton, m. d., & bail, j. (2014). cognitive processes and information literacy: some initial results from a survey of business students’ learning activities. journal of business & finance librarianship, 19(3), 181–233. https://doi.org/10.1080/08963568.2014.915080   sokoloff, j. (2012). information literacy in the workplace: employer expectations. journal of business & finance librarianship, 17(1), 1–17. https://doi.org/10.1080/08963568.2011.603989   sokoloff, j., & simmons, r. (2015). evaluating citation analysis as a measurement of business librarian consultation impact. journal of business & finance librarianship, 20(3), 159–171. https://doi.org/10.1080/08963568.2015.1046783   solis, e., & perkins, d. (2017). globetrotting students and faculty: adapting library instruction to global sites. journal of library & information services in distance learning, 11(1-2), 123–139. https://doi.org/10.1080/1533290x.2016.1229419   spackman, a. (2016). client-based experiential learning and the librarian: information literacy for the real world. journal of business & finance librarianship, 21(3-4), 258–273. https://doi.org/10.1080/08963568.2016.1226616   spackman, a., & camacho, l. (2009). rendering information literacy relevant: a case-based pedagogy. the journal of academic librarianship, 35(6), 548–554. https://doi.org/10.1016/j.acalib.2009.08.005    spahr, k. (2015). influences on undergraduate business students' perceptions about the adequacy of library information resources. behavioral & social sciences librarian, 34(4), 214–29. https://doi.org/10.1080/01639269.2015.1096183   stagg, a., & kimmins, l. (2012). research skills development through collaborative virtual learning environments. reference services review, 40(1), 61–4. https://doi.org/10.1108/00907321211203630   stagg, a., & kimmins, l. (2014). first year in higher education (fyhe) and the coursework post-graduate student. the journal of academic librarianship, 40(2), 142–151. https://doi.org/10.1016/j.acalib.2014.02.005   stebbing, d., shelley, j., warnes, m., & mcmaster, c. (2019). what academics really think about information literacy. journal of information literacy, 13(1), 21–44. https://ojs.lboro.ac.uk/jil/article/view/pra-v13-i1-2   stonebraker, i. (2015). flipping the business information literacy classroom: redesign, implementation, and assessment of a case study. journal of business & finance librarianship, 20(4), 283–301. https://doi.org/10.1080/08963568.2017.1288519   stonebraker, i. (2016). toward informed leadership: teaching students to make better decisions using information. journal of business & finance librarianship, 21(3-4), 229–238. https://doi.org/10.1080/08963568.2016.1226614   stonebraker, i. (2017). library-sponsored case competitions: best practices and assessment of learning gains. journal of business & finance librarianship, 22(1), 46–60. https://doi.org/10.1080/08963568.2016.1258937   stonebraker, i., & fundator, r. (2016). use it or lose it? a longitudinal performance assessment of undergraduate business students' information literacy. the journal of academic librarianship, 42(4), 438–444. https://doi.org/10.1016/j.acalib.2016.04.004   stonebraker, i., & howard, h. a. (2018). evidence-based decision-making: awareness, process and practice in the management classroom. the journal of academic librarianship, 44(1), 113–117. https://doi.org/10.1016/j.acalib.2017.09.017   stonebraker, i., maxwell, c., garcia, k., & jerrit, j. (2017). realizing critical business information literacy: opportunities, definitions, and best practices. journal of business & finance librarianship, 22(2), 135–148. https://doi.org/10.1080/08963568.2017.1288519   strittmatter, c. (2012). developing and assessing a library instruction module for a core business class. journal of business & finance librarianship, 17(1), 95–105. https://doi.org/10.1080/08963568.2012.630645   tariq, m., mahmood, k., ur rehman, s., & mustafa, g. (2018). online information searching skills of business students. pakistan journal of information management & libraries, 20, 39–59.   taylor, l. h. (2008). information literacy in subject-specific vocabularies: a path to critical thinking. college & undergraduate libraries, 15(1-2), 141–158. https://doi.org/10.1080/10691310802177127   tingle, n. (2018). taking care of business (before class): information literacy in a flipped classroom. journal of business & finance librarianship, 23(2), 183–198. https://doi.org/10.1080/08963568.2018.1510254   tooman, c., & sibthorpe, j. (2012). a sustainable approach to teaching information literacy: reaching the masses online. journal of business & finance librarianship, 17(1), 77–94. https://doi.org/10.1080/08963568.2012.629556   usry, m. l., law, m., leinbach, w., magolis, d. e., staley, a. b., & shapeero, m. (2010). e-discovery: what future business leaders need to know. journal of business & finance librarianship, 15(2), 78–93. https://doi.org/10.1080/08963560903554674   valle santos, m., & mayoral, r. m. (2018). information literacy in managers’ education. journal of business & finance librarianship, 23(2), 167–182. https://doi.org/10.1080/08963568.2018.1510253   watstein, s. b., scanlon, m. g., & cramer, s. (2015). q/a on teaching credit classes for entrepreneurship research. reference services review, 43(3), 480–490. https://doi.org/10.1108/rsr-06-2015-0030   webber, s., & johnston, b. (2000). conceptions of information literacy: new perspectives and implications. journal of information science, 26(6), 381–397. https://doi.org/10.1177/016555150002600602   wilhelm, j., & vaaler, a. (2018). looking for the library: using an undergraduate business syllabi analysis to inform an instruction program. journal of business & finance librarianship, 23(3–4), 225–236. https://doi.org/10.1080/08963568.2019.1624451   williams, j., & chinn, s. j. (2009). using web 2.0 to support the active learning experience. journal of information systems education, 20(2), 165–174. available at http://jise.org/volume20/n2/jisev20n2p165.html   wu, y. d., & lee kendall, s. (2006). teaching faculty's perspectives on business information literacy. reference services review, 34(1), 86–96. https://doi.org/10.1108/00907320610648789     appendix c all journal titles   academy of educational leadership journal australian academic & research libraries australian library journal behavioral & social sciences librarian business communication quarterly college & research libraries college & undergraduate libraries communications in information literacy education for information electronic journal of academic & special librarianship evidence based library & information practice ifla journal internet reference services quarterly journal of academic librarianship journal of business & finance librarianship journal of business and educational leadership journal of higher education theory and practice journal of information literacy journal of information science journal of information systems education journal of library & information services in distance learning journal of management education journal of the american society for information science & technology journal of the australian library & information association journal of web librarianship knjiznica library & information science research library review new review of academic librarianship new zealand library & information management journal nordic journal of information literacy in higher education pakistan journal of information management & libraries partnership: the canadian journal of library & information practice & research portal: libraries and the academy public services quarterly reference services review research strategies singapore journal of library & information management southern law journal studies in higher education the internet and higher education the journal of academic librarianship ticker: the academic business librarianship review   evidence summary   diagnoses, drugs, and treatment are the main information needs of primary care physicians and nurses, and the internet is the information source most commonly used to meet these needs   a review of: clarke, m. a., belden, j. l., koopman, r. j., steege, l. m., moore, j. l., canfield, s. m., & kim, m. s. (2013). information needs and information-seeking behaviour analysis of primary care physicians and nurses: a literature review. health information & libraries journal, 30(3), 178-190. http://dx.doi.org/10.1111/hir.12036   reviewed by: carol perryman assistant professor texas woman’s university denton, texas, united states of america email: cp1757@gmail.com     received: 27 may 2014  accepted: 6 aug. 2014      2014 perryman. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to improve information support services to health practitioners making clinical decisions by reviewing the literature on the information needs and information seeking behaviours of primary care physicians and nurses. within this larger objective, specific questions were 1) information sources used; 2) differences between the two groups; and 3) barriers to searching for both groups.   design – literature review.   setting – scopus, cinahl, ovid medline, and pubmed databases.   subjects – results from structured searches in four bibliographic databases on the information needs of primary care physicians and nurses.   methods – medical subject heading (mesh) and keyword search strategies tailored to each of four databases were employed to retrieve items pertinent to research objectives. concepts represented in either controlled or natural language vocabularies included “information seeking behaviour, primary health care, primary care physicians and nurses” (p. 180). an initial yield of 1169 items was filtered by language (english only), pertinence to study objectives, publication dates (2000-2012), and study participant age (>18). after filtering, 47 articles were examined and summarized, and recommendations for further research were made.   main results – few topical differences in information needed were identified between primary care physicians and nurses. across studies retrieved, members of both groups sought information on drugs, diagnoses, and therapy. the internet (including bibliographic databases and web-based searching) was the source of information most frequently mentioned, followed by textbooks, journals, colleagues, drug compendiums, professional websites, and medical libraries. there is insufficient evidence to support conclusions about the differences between groups. in most research, information needs and behaviours for both groups have been discussed simultaneously, with no real distinction made, suggesting that there may not be significant differences even though a few studies have found that nurses’ emphasis is on policy and procedures. barriers to access include time, searching skills, and geographic location; for the last, improvements have been made but rural practitioners continue to be adversely affected by limited access to people and resources.   conclusion – both primary care physicians and nurses seek information on diagnosis and treatment. the internet is of increasing utility for both groups, but all resources have advantages and disadvantages in identifying evidence based information for use in practice. further research is required to support access and use of evidence based resources, and to explore how focused, evidence based information can be integrated into electronic health record systems.     commentary   authors extracted studies from a much broader literature on the information needs, behaviours, and barriers to access of practitioners across all areas of healthcare. barriers to information needed for direct patient care have long been understood to include time and access to resources, but the explosive growth of health technology supports the need for further research on information behaviour in the rapidly changing environment of care. ­   two critical appraisal instruments were used to evaluate this study (glynn, 2006; perryman, 2009).   search strategies were clearly detailed (table 1). mesh and natural language were used in searching, but there was no discussion of how natural language terms were acquired, tested, and compiled. syntax was reported for all databases except ovid medline, with no explanation for the omission. insufficient use of controlled vocabulary and synonyms and no use of truncation for key concepts severely limited comprehensive retrieval. for example, only two major headings were used in a cinahl search: “information seeking behavior” and “nurses,” while scopus searches used keyword-only searches in the title, abstract, and keyword fields, as well as extremely limited use of synonyms. use of additional synonyms (e.g., search*, seek* retriev*, doctor, nurs*, and primary practice) in all databases would have provided a far larger set of items for study. also missing were definitions of terms used to categorize topics of information needed, and any explanations of validation processes conducted during filtering.   no qualitative assessment of the articles described was performed, and based upon the inconsistency of the literature being examined, the authors were of necessity confined to a limited review. a claim is made that information behaviours and needs are similar across geographical boundaries, but there is no evidence to support this contention. a 2011 study sponsored by the european center for disease control (ecdc) concluded, in part, that “differential access and use is apparent both within countries and between countries in the european union” (higgins, sixsmith, barry, & domegan, p. 1). within the authors’ own final set of articles, several support the ecdc conclusion, particularly the norbert and lwoga study (2012, citation not provided) which found that a poor technology infrastructure and uncertain power provision frequently hampered access; another study (davies, 2011) comparing physicians in the united states, england, and canada concluded that physicians in the united states and canada “were more likely to use electronic resources [than] physicians based in the united kingdom” (table s1, online only). in the reported frequencies of resources consulted, there are five discrepancies between the narrative text and data provided online, in table 3: internet 19 (18 in table 3); textbooks 18 (17 in table 3); articles 17 (15 in table 3); professional websites 8 (6 in table 3); and drug compendiums 7 (6 in table 3). further discrepancies are noted in the authors’ exclusion of any mention of ovid medline in the methods section, though search terms are noted in figure 1 and in online data. correction of these and other errors of language use would have improved the article.   rather than providing new information, the article serves as a limited review over the past decade. suggestions are made about the use of the review to enhance electronic health records, but readers should be aware of limitations to this and other research on the same topic. in particular, the materials reviewed are older (two thirds were published between 2003 and 2007), and they represent a widely disparate body of literature. as an example, three of the studies examined the use of particular products such as dynamed, another examined information behaviours related to cancer treatment, and five discussed the use of infobuttons. additionally, inclusion of the cochrane library, web of science, and ovid embase databases in searching would have provided more comprehensive results, and may have affected findings.   barriers to the performance of more rigorous cross-study comparisons due to methodological differences and the lack of shared definitions have been recognized elsewhere (e.g., del fiol, workman & gorman, 2014; higgins et al., 2011). physicians and nurses working in settings such as hospitals and medical librarians intending to structure services or collections on this research should consult more recent and narrowly-focused literature, and expand upon the authors’ conclusions with added local data.     references   del fiol, g., workman, t. e., & gorman, p. n. (2014). clinical questions raised by clinicians at the point of care: a systematic review. jama internal medicine, 174(5), 710-18. http://dx.doi.org/10.1001/jamainternmed.2014.368 .   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech 24(3), 387-399. http://dx.doi.org/http://dx.doi.org/10.1108/07378830610692154   higgins, o., sixsmith, j., barry, m. m., & domegan, c. (2011). a literature review on health information seeking behaviour on the web: a health consumer and health professional perspective. stockholm: ecdc. http://dx.doi.org/10.2900/5788   perryman, c. (2009). evaluation tool for bibliometric studies. retrieved 13 aug. 2014 from http://cperryman.com/critical%20appraisal%20tool%20-%20bibliometric%20study.doc   research article   the information needs of canadian midwives and their evidence informed practices: a canada-wide survey   lindsay barnes research support officer faculty of medicine and health the university of sydney sydney, new south wales, australia email: lindsay.barnes@sydney.edu.au   luanne freund associate professor school of information university of british columbia vancouver, british columbia, canada email: luanne.freund@ubc.ca   dean giustini university of british columbia biomedical branch librarian vgh diamond health care centre vancouver, british columbia, canada email: dean.giustini@ubc.ca   received: 4 aug. 2019                                                               accepted: 29 jan. 2020      2020 barnes, freund, and giustini. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29616     abstract   objective – the study aim was to understand the extent to which canadian registered midwives have access to and make use of clinically relevant information for evidence based midwifery practice.   methods – a survey instrument was created consisting of 17 multiple choice, matrix table, and short answer questions and distributed to 1,690 recipients on the canadian association of midwives email list in fall 2018. in total, 193 responses were included in the analysis.   results – one third of midwives do not have library memberships. midwives reported that limited access to clinically relevant information is a key challenge in applying information in practice. midwives with library memberships reported more frequent use of high-quality information while midwives without memberships reported more frequent use of websites. midwives with advanced degrees (graduate, phds) were more likely to be high-frequency information users and rank themselves higher on evidence based competency scales than their undergraduate-holding colleagues. clinical practice guidelines were important information sources and used frequently by midwives.   conclusion – midwives reported low levels of academic or hospital library memberships and yet used information frequently. clinical practice guidelines support the work of midwives but are inaccessible to some due to paywalls. midwives lacked confidence in evidence based practice and reported critical appraisal as an area for development. solutions to these problems could be addressed at the hospital, health authority, provincial, or national association level, or within midwifery departments at canadian universities. hospital and academic libraries should prioritize the information needs of students and practicing midwives and identify ways to foster use of library resources through administrative or educational interventions.     introduction   midwives in canada are autonomous care providers who provide evidence based care for their clients throughout pregnancy, birth, postpartum, and the newborn period. midwives hold a high degree of professional responsibility and accountability and must abide by guidelines and standards appropriate to the clinical context (ontario medical association & association of ontario midwives, 2005). according to the canadian association of midwives (cam) (2015), “midwifery practice is informed by research, evidence-based guidelines, clinical experience, and the unique values and needs of those in their care” (p. 2). this approach aligns with evidence based medicine (ebm), defined by guyatt, rennie, meade, and cook (2008) as an approach to patient care that uses best evidence to guide decision making and emphasizes the importance of patient values and preferences.   canadian midwives routinely integrate discussions about risk from the literature into their conversations with clients (van wagner, 2016); further, midwives must maintain their knowledge and clinical skills to ensure clients are treated according to current best evidence (college of midwives of ontario, 2018). research on the information behaviour and evidence based practice (ebp) of canadian midwives is lacking; however, a growing number of international studies of midwives and nursing professionals point to a gap between high level commitments to ebp and the actual practice of it (spenceley, o’leary, chizawsky, ross, & estabrooks, 2008; de leo, bayes, geraghty & butt, 2019). this is a serious issue, as noted by de leo et al. (2019): “the evidence‐to‐practice gap in maternity services remains a global issue for midwives and demands prompt action from both knowledge producers and knowledge users” (p. 4,234).   a recurring theme in this literature is the importance of ease-of-information-access on practice. canadian midwives are expected to practice evidence based midwifery; however, the extent to which midwives use and access sources of evidence—such as those found through systematic reviews, original articles, or databases—is unknown. as autonomous care providers, midwives have a great need to stay up to date; however, midwifery associations do not operate libraries for their members. this contrasts with other professional groups, such as the canadian nurses association and the college of family physicians of canada, which have created digital or physical libraries to provide members with access to current, clinically relevant literature. while many summaries of the literature are freely available online, key guidelines, policy statements, and committee opinions from the society of obstetricians and gynecologists of canada (sogc) are published in the journal of obstetrics and gynecology of canada which is accessible only to members of the sogc by subscription.   to understand the extent to which midwives are able to satisfy their need for clinically relevant information, we conducted a survey focused on the types of information sources midwives prefer and their proficiency with and views about evidence based practice given the dominant medical model of childbirth in canada. the aim was to examine cross-canada access to and use of evidence based information by midwives.   literature review   midwifery practice in canada   the professionalization and regulation of midwifery in canada developed out of the women’s movement of the 1970s to offer an alternative to the medicalization of pregnancy and childbirth (parry, 2008). the canadian midwifery model of care is based on principles of professional autonomy, continuity of care, informed choice, choice of birth place, collaborative care, and evidence based practice (canadian association of midwives, 2015). in this model, a midwife is a responsible and autonomous community-based provider working in partnership with women and their families to balance patient values with community standards of practice and current best evidence (college of midwives of ontario, 2018).   midwifery education programs (meps) in canada are direct-entry, four-year undergraduate degrees leading to a bachelor of midwifery or bachelor of health sciences. there are “bridging” or “pre-registration” midwifery programs for internationally educated midwives who seek to practice in canada (college of midwives of british columbia, n.d.). during their time as students, midwives have access to library databases and electronic resources through their universities. upon completion of their programs of study, their access to university library collections ends or is restricted to alumni privileges or library walk-in access.   funding models for midwifery practice vary by province and territory, but most midwives work as independent contractors under a fee-for-service agreement with their home province (british columbia, alberta, and ontario) or under a salaried model (manitoba, saskatchewan, the northwest territories and quebec). while no studies have been conducted on the advantages of each model with respect to library access or use of clinically relevant information, under the fee-for-service model “midwives have greater flexibility and fewer bureaucratic barriers to establishing midwifery practice in diverse geographic settings” (thiessen, haworth-brockman, nurmi, demczuk, & sibley, 2018, p. 7). fee-for-service encourages the establishment of midwifery practices at a distance from hospitals and their onsite libraries and professional networks. in comparison to other jurisdictions, such as the u.k. and australia, midwives practicing in canada experience a greater degree of autonomy and a greater responsibility to proactively consult and collaborate with other health professionals (mallot et al., 2009).   information seeking and evidence based practice   two studies offer useful conceptual frameworks for this project. leckie, pettigrew, and sylvain (1996) proposed a model of the information seeking of professionals, taking into account their roles, tasks, and information needs. as primary care providers, midwives’ roles overlap with both physicians and nurses, indicating a broad range of information needs. the model identifies certain constants across professional groups, including the importance of information access, and it points to a high degree of complexity in professional work settings, which leads to variability and unpredictability in information seeking (leckie et al., 1996). the second model (spenceley et al., 2008) identifies a range of factors that shape information seeking activity and outcomes in evidence based nursing practice: the context of practice, which includes aspects of the individual practitioner (e.g., education, skills, attitudes), the work setting (e.g., training, information resources); the sources of information, which have attributes, such as availability and trustworthiness; and a number of mediating factors such as time pressures, the expectations of others, and situational barriers. notable themes include the constraints of time and access to information on the search process, the need for administrative support for ebp, and the preference for trusted interpersonal sources of information (spenceley et al., 2008). both frameworks situate information practices in context and highlight the awareness of and access to a range of sources for diverse tasks in complex work settings.   although studies of information seeking and ebp of canadian midwives are lacking, studies of midwives and nurses from the u.k. and australia indicate that ebp is consistent with the philosophy of midwifery and is valued by practitioners (bayes, juggins, whitehead & de leo, 2019; de leo et al., 2019; fairbrother, cashin, conway, symes, & graham, 2016; toohill, sidebothan, gamble, fenwick, & creedy, 2017; veeremah, 2016). notably, ebp has been recognized by midwives as a means to reduce the medicalization of pregnancy and birth, including the overuse of interventions (de leo et al., 2019; kennedy, doig, hackley, leslie & tillman, 2012; miller et al., 2016). at the same time, there is considerable evidence that midwifery care is not always reflective of ebp guidelines, raising questions as to the reasons for this gap (bayes et al., 2019; de leo et al., 2019; fairbrother et al., 2016; toohill et al., 2017).   a recent integrative review of midwives’ ebp sought to investigate this issue through close examination of six studies, several of which included both nurses and midwives (de leo et al., 2019). the authors identified a number of themes. practitioners are aware of ebp and confident with their skills; however, published information sources are underused, with practices based more on convention and information gained from patients and other professionals (bayes et al., 2019; fairbrother et al., 2016; heydari et al., 2014). for example, in a survey of 297 australian midwives to evaluate the uptake of evidence based guidelines on normal birth, toohill et al. (2017) found that almost all respondents were familiar with the guidelines, but only 71% routinely used them. three barriers to ebp implementation are widely identified: a lack of time to find and use evidence based resources (bayes et al., 2019; fairbrother et al., 2016; toohill et al., 2017; veeramah, 2016); organizational barriers, such as resistance to change, lack of support from colleagues, and structural impediments (bayes et al., 2019; heydari et al., 2014; toohill et al., 2017; veeramah, 2016); and limited access to information and computers in the workplace (fairbrother et al., 2016; toohill et al.; 2017; veeramah, 2016).   information literacy and skills training   training medical practitioners to search and appraise high-quality evidence has long been recognized as an important factor in the provision of patient care (guyatt, meade, jaeschke, cook, & haynes, 2000). informed clinicians are able to assess their own knowledge gaps and formulate effective research questions (mckibbon, wyer, jaeschke, & hunt, 2008). lack of access to libraries has been identified as an obstacle to developing these skills among physicians (coumou & meijman, 2006). in an early survey of the information needs of 1,715 u.k. nursing professionals, including midwives, bertulis and cheeseborough (2008) found that lack of training in information seeking was an obstacle to applying evidence in practice. more recent research continues to identify education and training gaps (e.g. veeramah, 2016) and the need for capacity building among nurses and midwives (fairbrother et al., 2016), although there is evidence that ebp skills are rising over time. a survey of australian nurses and midwives found higher self-reported expertise compared to earlier studies conducted in the u.k. and australia (fairbrother et al., 2016). however, less than 40% of respondents considered themselves competent or expert at finding research evidence or using the library to locate information. rates of internet competency were higher, at 63%, a finding supported by additional research on australian midwives (mckenna & mclelland, 2011).   longstanding information practices, including source preferences, also impact ebp. ebenezer (2015) reviewed the literature on the information behaviour of nurses and midwives between 1998 and 2014 and identified a strong preference for gaining information through human sources. this preference for what estabrooks et al. (2005) terms “interactive” and “experiential” sources of information over formal or “documentary” sources (p. 471) is one of the most frequent findings in studies of information seeking of nursing and midwives (bertulis & cheeseborough, 2008; de leo et al., 2019; ebenezer, 2015; fairbrother et al., 2016; ricks & ham, 2015; thompson et al., 2001b). interestingly, this preference for social information sources does not seem to extend to social media. dalton et al. (2014) conducted a mixed methods study of information communication technology use among australian midwives and found a high degree of consensus that social media is an inappropriate and high-risk means of sharing information in a healthcare context.   use of pre-appraised evidence, such as practice guidelines, systematic reviews, and computer decision support systems, continues to impact the quality of clinical decision making. guyatt, mead, jaeschke, cook, and haynes (2000) noted the critical importance of pre-appraised evidence for clinicians and observed that, while not technically practicing evidence based care, the clinical trainees whom they studied acquired a “restricted set of skills” which included the ability to track down and use secondary sources of pre-appraised evidence (p. 955). lafuente-lafuente et al. (2019) found that health practitioners used primary evidence infrequently which “suggests that many professionals probably do not (or are unable to) verify independently, by their own means, the validity of what is stated in guidelines, or otherwise what is presented to them as ‘ebm-based’” (2019, p. 5). this is supported by the work of fairbrother et al. (2016), which found that articles published in research, nursing, and medical journals were the least frequently used sources of information in practice.   summary   the gap between ebp commitments and reliance upon evidence in every day practice is widely documented among midwives and the nursing professionals more generally. this situation persists across the diverse national contexts and workplace settings in which midwives practice. relevant models of information seeking stress the complex and situated nature of the work of health professionals, in which the roles, tasks, information sources, and work environments significantly shape and constrain practices. a broad, but generally consistent, set of barriers to ebp in midwifery have been identified in studies conducted outside canada. among these, access to resources, ebp literacies and skills, and longstanding information practices are key. other factors also emerge as important, notably time, organizational culture, and receptivity to change. canadian midwives operate with more autonomy than their colleagues in the u.k. and australia, where the majority of studies have been conducted, which suggests that these findings may not generalize to the canadian context.    aims   rather than examining the full range of factors known to shape ebp, the current study focuses on access to libraries and information, source use, and ebp literacies in the canadian context, as a first step towards understanding the local situation and constraints upon ebp. a survey was designed to investigate the extent to which canadian registered midwives had access to and made use of clinically relevant information in practice. the research questions were: how frequently do canadian midwives use published information and which sources do they prefer? what challenges do they encounter in accessing and using information? how do they acquire the information literacy skills needed to find and apply clinical information and are those skills well-developed? to provide a more nuanced picture of the status quo, we compare responses across several factors, including region, work conditions and settings, and access to libraries.   methods   survey design   the survey instrument (see appendix) was developed iteratively by soliciting feedback from library and midwife professionals. the range and types of questions were developed by using the association of college & research libraries’ “information literacy competency standards for nursing” (association of college and research libraries, 2013). the frameworks of leckie et al. (1996) and spenceley et al. (2008) indicated the importance of collecting contextual data on demographics, specialization, career stage, and other factors likely to influence information behaviour. an early version of the questionnaire was piloted with two registered midwives, both educators, and a health librarian from british columbia, canada. the final version incorporated feedback from the pilot sessions.   the questionnaire begins by asking for demographic data to establish the personal and practice context (q1-6). the next set of questions (q7-10) focuses on general access and use of information, including library memberships and frequency of use of information. q11 asks about information source types and use frequency. the list of information sources was derived from the work of mckibbon et al. (2008). these categories were not explicitly defined in the questionnaire, but were presented with illustrative examples, as follows:   ·        summaries (clinical practice guidelines and systematic reviews) ·        textbook-like e-resources (uptodate, accessmedicine) ·        studies (original research articles) ·        print books (monographs, textbooks) ·        apps ·        popular websites (webmd, mayo clinic) ·        social media (twitter)   q12 asks participants to rate challenges in applying clinically relevant information, with a focus on access and literacy skills. the final set of questions (q14-16) ask about information literacy training and competencies. q14 asks participants to report on their level of expertise in ebp using a five-point scale ranging from novice to expert, across four categories of ebp competencies (q14). using this data, we calculated an aggregate metric of ebp competency using the sum of the responses across the four areas converted to a score out of 10 for ease of interpretation, such that an expert level of competency across all areas received a top score of 10. this metric allowed us to compare self-reported ebp competency levels across groups.   the study received approval from the behavioural research ethics board at the university of british columbia. the questionnaire was implemented and distributed online using the qualtrics platform.   study responses and recruiting   survey responses were collected during the fall of 2018. the study population was registered midwives in canada, of which there were 1,690 at the time of the study (canadian association of midwives, 2018). cam is the national body representing midwives and it collects and maintains a database of registered midwives using data from provincial and territorial associations and colleges. all registered midwives in the database (n = 1,690) received an invitation from cam to participate. to encourage further participation, we distributed invitations through the midwives association of british columbia’s email list and the canadian registered midwives facebook group, which served as reminders or reinforcements. no compensation was provided for completing the questionnaire, which took, on average, five to 10 minutes. in total, 218 midwives participated in the survey, representing a 12.8% response rate. of the 218 questionnaires submitted, 25 were found to be substantially incomplete and were removed, leaving 193 questionnaires for analysis.   responses were received from eight provinces and one territory, but most respondents were from ontario (40%) and british columbia (bc, 39%), followed by alberta (8%) and quebec (6%). the mean number of years of respondents’ midwifery practice was nine, with responses ranging from zero to 40 years. the educational profile of respondents included 67% (n = 130) holding a bachelor degree, 23% (n = 44) master’s degree, and 4% (n = 7) phd. a small number held other credentials. a majority of respondents (71%, n = 137) were practicing full time with 17% (n = 32), practicing part time, 7% (n = 13) non-practicing, many of whom were educators, and 6% (n = 11) reporting some other status, including those on temporary leaves. we reviewed the responses from the last two categories and determined that these responses were valid for our purposes, as these were experienced midwives, whose responses were consistent with the broader sample.   table 1 summarizes the distribution of responses by province and work setting. the majority (68%, n = 131) of respondents were practicing in an urban or suburban setting, while 26% (n = 51) were rural and 6% (n = 11) remote. these categories were provided to participants without definitions, and so were subject to interpretation. given the low number of responses from most provinces, we report comparisons by province only for bc and ontario in this report. the bc-ontario comparisons are not meant to be representative of conditions across canada and are not generalizable. rather, the data analysis is descriptive in nature and is meant to provide a starting point for examination and further research.   table 1 distribution of responses by region and work setting urban suburban rural remote total alberta 11 3 2 16 (8%) british columbia 36 10 24 3 73 (38%) manitoba 3 3 (2%) new brunswick 1 1 (1%) nw territories 1 1 (1%) nova scotia 1 2 3 (2%) ontario 36 19 20 3 78 (40%) quebec 4 3 2 3 12 (6%) saskatchewan 1 1 (1%) multiple provinces 2 1 3 (2%) other   1   1 2 (1%) grand total 95 (49%) 36 (19%) 51 (26%) 11 (6%) 193   results   library membership   we asked participants if they held membership in an academic or hospital library, based on the assumption that library privileges, such as access to bibliographic databases and electronic resources, require library membership. figure 1 summarizes responses. overall, 67% (n = 129) of respondents reported some type of academic library membership and 33% (n = 64) reported having no library membership (or unsure). of those who had membership, 53% (n = 68) had access through a university as a student, employee, or faculty member; 57% (n = 74) through a hospital; and 9% (n = 12) through a college or private library. many had access through multiple avenues. several respondents indicated in their comments that they held alumni library privileges, which were described as “limited” or due to expire one year after graduation. others said they gained access to academic collections through use of their colleagues’ or partners’ passwords or via credentials gained through other professional associations. one respondent added the comment that “hospitals refuse midwives access.”   figure 1 library membership (academic or hospital libraries).   we compared the rates of library membership across different geographic regions. a higher percentage of respondents from bc (74%) reported having membership access compared to those from ontario (65%). with respect to work setting (figure 2), rates of library membership in urban work environments were highest at 72% followed by suburban at 69%, remote at 64%, and rural at 57%. midwives with graduate degrees reported higher levels of library membership (78%) compared to those with undergraduate degrees (62%). there was no difference between those who reported working full time or part time. in a separate question, we asked if respondents used public libraries as a resource for their clinical information needs. this proved to be highly uncommon, with 96% of respondents indicating that they very rarely or never use public libraries to stay informed for practice.   figure 2 library membership by work setting (percent).   use of clinically relevant information   participants were asked how frequently they refer to clinically relevant information in their midwifery practices. the most common response was: frequently – several times a week (65%), followed by occasionally – a few times a month (18%), and very frequently – several times a day (14%). figure 3 compares the frequency of information usage by those with and without membership in an academic or hospital library. not surprisingly, a higher percentage of those with access to a library reported using information at a high frequency, while those without access were more likely to use information occasionally, rarely, or very rarely.   figure 3 comparison of information use frequency by library membership.   group comparisons were made based on high-frequency information usage (defined as the percentage of those who reported using information frequently or very frequently):   ·        85% (n = 62) of respondents from bc were high-frequency users compared to 76% (n = 59) in ontario; ·        85% (n = 81) of respondents in urban settings were high-frequency users compared with 72% (n = 26) in suburban, 73% (n = 37) in rural, and 82% (n = 9) in remote settings; ·        82% (n = 113) of respondents working full time were high-frequency users compared with 63% (n = 40) of those working part time; ·         88% (n = 45) of respondents with graduate degrees were high-frequency users compared with 77% (n = 96) of those with undergraduate degrees.   participants were asked to report on their use of nine different types of information sources in terms of frequency. the results, presented in figure 4 showing higher percentages in darker shading, indicate summaries and colleagues are used most frequently as information sources.   figure 4 heat map of information source frequency of use (percent).   we were interested in the impact of library access on the types of information sources consulted. we focused our analysis on a subset of all information source types, including those we thought would be most affected. the results, presented in table 2, showed some interesting patterns. at the highest level of frequency, those without library access reported less frequent use of summaries, colleagues, and textbook-like e-resources and more use of websites than colleagues with library access. research articles were used less frequently by those without access, although this shows up in the frequent (lower) and rare (higher) use categories. lack of membership did seem to influence resource use patterns, but it did not prevent most midwives from using a range of resource types on a regular basis.   table 2 comparison of information source frequency of use (percent) by library membership (yes/no) very frequently frequently occasionally rarely never information sources yes no yes no yes no yes no yes no summaries 48 40 44 48 8 12 0 0 0 0 colleagues 40 31 51 48 8 17 2 3 0 0 textbook-like e-resources 21 10 37 28 28 33 10 17 5 12 original research articles 10 10 36 20 43 45 11 22 1 3 popular websites 5 15 33 27 41 42 17 10 5 7   borrowed library access   in the final open-ended question in the survey, a number of responses raised concerns that midwives are forced to ask for help in accessing essential information. one midwife commented “i always meet my information needs by using online resources or by asking my senior student to use her online connection to university libraries to locate resources…” another stated, “i don't have other means of ‘official’ access but i frequently use a family member’s library card to access the university library.” other respondents mentioned using a partner’s university library membership and a friend’s login to uptodate. many midwives noted that they had alumni library privileges but commented on the limitations of such access, especially in cases where such privileges do not extend to online databases.   library skills and evidence based practice competencies   question 14 asked participants with multiple choice and an optional short answer question if they had ever received library skills training. twenty-eight percent reported that they had never received training (or could not remember). overall, 54% reported receiving some library skills training during their midwifery education and training, and the remaining 18% had exposure to this training through other means. those who made additional comments for this question noted that such training was “limited,” or superficial: “we touched on it.” several respondents noted that they received more training in library skills during the acquisition of degrees unrelated to midwifery. other comments highlighted that training was brief, years ago, and that the specifics of the instruction were difficult to recall:   ·        “this training was 10-15 years ago. a lot has changed since then.” ·        “but i can’t call on that knowledge even though it’s been 7-10 years.” ·        “20 years ago, too long ago”   participants reported interest in receiving library skills training; however, a number of comments indicated that training would not be useful without access to library resources.   results of the ebp competency question summarized in figure 5 indicate that very few respondents identified as expert in any of the ebp competency areas, and the majority of responses were in the middle points of the scale. the highest levels of expertise were reported for converting the need into an answerable question and application of evidence to practice and the lowest levels for critical appraisal of evidence.   using an aggregate metric of ebp competency, we compared competency scores across groups. self-reported competency scores increase with experience, with mean scores of 5.7 for those with less than 5 years in practice; 6.0 for those with between 5 and 19 years, and 6.5 for those with 20 or more years. similarly, higher competency levels are reported for those with graduate level education (6.4 for master’s and 7.1 for phd) as compared to those with undergraduate degrees (5.7). respondents with library access report higher levels of expertise (6.0) than those who do not report access to a library (5.5).   challenges to using clinically relevant research in practice   participants were asked to indicate their level of agreement with a set of five challenges as factors in their own practice. figure 5 summarizes the results, showing the percent of all respondents who indicated agreement or strong agreement with each statement. lack of access to information showed the highest level of agreement (53%), followed by the difficulty in judging the quality of research (41%), which reinforces the low self-reported competency in this area.   further analysis regarding the challenges associated with lack of information access suggests that some variation exists across groups:   ·        62% (n = 31) of midwives working in rural settings agreed that information access was a challenge in comparison with 47% (n = 45) of those working in urban settings; ·        66% (n = 51%) of early career midwives agreed that information access was a challenge in comparison with 29% (n = 6) of those with more than 20 years’ experience;   the high costs of paying to download articles and buy memberships to point of care tools such as uptodate were mentioned repeatedly in response to the open question. comments included the following:   “i think the biggest limitation is access to information. sogc guidelines, uptodate and several other critical sources are now fee for service and some midwifery clinics will not pay these fees, leaving the midwives at a disadvantage for relevant data.” “subscriptions to library databases is very expensive particularly for small practices or individual midwives.” “the main barrier to accessing medical libraries and online journals is cost. for a midwifery clinic or individual midwife to afford this they need to pay midwives adequately.”   figure 5 levels of reported expertise across ebp competency areas.   figure 6 levels of agreement with challenges to using clinically relevant research in midwifery practice.   discussion   this study set out to understand the extent to which canadian midwives have access to and make use of clinically relevant information in practice. one third of survey respondents reported having no library membership. urban midwives and those with graduate degrees reported higher levels of library membership. this level of access is consistent with veeremah’s (2016) study, in which one third of respondents reported limited access to information, and it affirms prior research indicating that access to information continues to be a challenge in professional contexts (leckie et al., 1996; spenceley et al., 2008), even as digital and mobile information technologies proliferate. results further show that those respondents without library access were less likely to be frequent users of clinically relevant information and were more likely to refer to websites, which undergo less quality control than published summaries, textbook-like e-resources, or research articles. considering the high degree of responsibility and technical knowledge required in midwifery, this finding is concerning, as it brings into question the quality, consistency, and equity of care across settings. while other health practitioners in canada enjoy access to digital libraries offered by their colleges and associations, midwives do not benefit from such a program, which could be considered a basic component of ebp.   more than 50% of survey respondents agreed that access to information is an obstacle to ebp, with more limited agreement that finding, using, and evaluating clinically relevant information is challenging. a surprising result was that respondents reported finding creative ways to access the information they needed for ebp by bypassing paywalls and borrowing memberships. one respondent linked this issue of information access with the broader issue of hospital integration and hospital privileging, by indicating that they were denied access to information by the hospital. if midwives are barred from using library services by virtue of not being “staff” and holding “privileges,” this bureaucratic barrier raises ethical implications for patient safety and should be addressed as a matter of urgency. in this context, “borrowing” access privileges, which breaks licensing agreements and constitutes a misuse of library systems, can be viewed as a form of activism designed to redress an imbalance of power and privilege within the healthcare field. while this study did not examine the impact of organizational factors on ebp, studies such as that carried out by bayes et al. (2016) show that midwives are vulnerable to opposition to ebp from hospitals, colleagues, and superiors, in part because ebp guidelines may run counter to dominant medicalized approaches to childbirth (toohill et al., 2017). this perspective should be explored further, as the role of power and privilege in information seeking and ebp is largely absent from prior research and existing conceptual frameworks (e.g. leckie et al., 1996; spenceley et al., 2008), and it may offer new insights, particularly for the study of midwifery in canada.   the majority of midwives reported difficulty judging the quality of the evidence. this finding is echoed in studies by fairbrother et al. (2016) and by ross (2010) who reported that nurses had difficulty understanding articles and had insufficient skills critiquing the literature. midwives indicated that the information literacy skills they received during their education was limited and, in many cases, stale. however, midwives with advanced degrees (graduate, phds) ranked themselves more highly on evidence based practice competency scales than their undergraduate-holding colleagues. guyatt et al. (2000) considered the skills of critical appraisal of primary studies to be invaluable, stating that health care providers who had these skills would be better able to identify when attempts to influence practice were made based on evidence (or to justify childbirth interventions). lafuente-lafuente et al. (2019) echo this sentiment, stating that practitioners who used primary studies infrequently were less able to independently verify the guidance provided in clinical practice guidelines. in the context of increasing childbirth interventions, midwives with quality appraisal skills of original literature may be better able to identify when clinical guidelines are out of date, biased, or have used poor methodology. this finding points to the fact that increasing the accessibility of information is only one component of a much broader set of challenges, which includes the need for training and capacity building.  results of this study reinforce previous findings that clinical practice guidelines are an essential resource for clinicians. in total, 91% of midwives in this study reported frequent use of summaries such as those published by the society of obstetricians and gynecologists of canada, the association of ontario midwives (aom), and the perinatal services of british columbia. however, clinical practice guidelines developed by the sogc are currently behind a membership firewall, despite their relevance for maternity care professionals. membership in sogc is fee-based ($160 per year) and many midwives in this study indicated that they did not have access to these guidelines. the college of midwives of british columbia recently rescinded their clinical practice guidelines (which were freely available on their website), and no longer create or maintain them due to a lack of funding and the existence of guidelines from national expert bodies such as the sogc and the perinatal services of british columbia (r. comfort, personal communication, july, 10, 2019). this decision is concerning in light of the findings of this study and because midwife-specific guidelines address the person-centered model of care that characterizes canadian midwifery.   the opinions of trusted colleagues were the second most common information source used by surveyed midwives, which is not surprising given that this is one of the most consistent themes in the information-seeking research on professionals, nurses, and midwives (de leo et al., 2019; leckie et al., 1996; spenceley et al., 2008). while this research did not examine why this is the case, it is clear from the wider literature that interpersonal sources of information are valuable in the context of ebp for a number of reasons, including that people provide information that is contextualized and experience based, and in some cases they perform a translational role, by sharing information in a form that is more easily understood or made more relevant for a particular audience (thompson et al., 2001b). given this preference for collegial information sharing, mentorship or peer-based training models for ebp in midwifery may prove to be effective (fairbrother et al., 2016). at the same time, the reliance on colleagues as information sources reinforces the role that a supportive organization plays in enabling ebp, including midwife leaders able to champion change (bayes et al., 2019; spenceley, et al., 2008).   this study has focused only on select components of the conceptual frameworks developed by leckie et al. (1996) and by spenceley et al. (2008), notably, aspects of the individual (education, work experience, skills); the work setting (region, employment status); the sources of information (accessibility, source type); and the outcomes (information behaviours). results are based on limited self-report data and statistical tests were not conducted. therefore, these results are not generalizable. a more comprehensive study would need to consider additional features, notably, the impact of organizational factors and time pressures, which are known to influence ebp. given the lack of prior research in the canadian context and the unique nature of midwifery practice in canada, further research is needed to validate and extend these findings. one contribution of this study is the identification of information-seeking strategies that sidestep existing systems and norms in order to meet needs, and which may reflect structural barriers and power imbalances that are not currently addressed in these models.   conclusion   canadian midwives, as experts in physiologic birth, enjoy an expanded scope of practice that requires frequent and ongoing consultation with information. as professionals committed to ebp, access to high-quality information would seem to be a given; however, the results of this survey indicate that a substantial number of midwives are practicing without such access. clinical practice guidelines support the work of midwives but are inaccessible to many midwives due to paywalls. respondents lacked confidence in evidence based practice and reported critical appraisal as an area for development. while no canadian universities currently offer higher degrees in midwifery, it may be that future offerings of advanced midwifery programs would have a beneficial effect on evidence based practice proficiency as midwives with advanced degrees had higher self-reported ebp expertise. coordinating access to digital biomedical collections or removing barriers to midwife access of these collections is one way that hospitals, health authorities, and their libraries, provincial, or national associations could help midwives practice ebp. hospital and academic libraries should prioritize the information needs of student and practicing midwives and identify ways to foster use of library resources through educational interventions.   acknowledgements   we would like to thank all those midwives who responded to the questionnaire and shared their experience and insights. we appreciate the support of the cam for allowing us to recruit via their listserv, and to cathy ellis, rm, alixandra bacon, rm, and brooke ballantyne scott, manager of library services, royal columbian hospital, who provided comments and a review of the survey instrument.   references   association of college and research libraries. (2013). information literacy competency standards for nursing. retrieved from http://www.ala.org/acrl/standards/nursing   bayes, s., juggins, e., whitehead, l., & de leo, a. (2019). australian midwives’ experiences of implementing practice change. midwifery, 70, 38–45. https://doi.org/10.1016/j.midw.2018.12.012   bertulis, r., & cheeseborough, j. (2008). the royal college of nursing’s information needs survey of nurses and health professionals. health information and libraries journal, 25(3), 186–197. https://doi.org/10.1111/j.1471-1842.2007.00755.x   canadian association of midwives. (2015). canadian midwifery model of care [position statement]. retrieved from https://canadianmidwives.org/wp-content/uploads/2018/10/finalmocps_o09102018.pdf   canadian association of midwives. (2018). midwifery care for all: building the profession, annual report, 2017-2018. retrieved from https://canadianmidwives.org/wp-content/uploads/2018/10/annual-report-2017-2018.pdf   college of midwives of british columbia (n.d.). recognized midwifery education programs. retrieved from https://www.cmbc.bc.ca/registration/inter-provincial-registration/recognized-midwifery-education-programs/   college of midwives of ontario. (2018). professional standards for midwives. retrieved from https://www.cmo.on.ca/wp-content/uploads/2018/06/professional-standards.pdf   coumou, h. c., & meijman, f. j. (2006). how do primary care physicians seek answers to clinical questions? a literature review. journal of the medical library association, 94(1), 55–60. retrieved from https://www.ncbi.nlm.nih.gov/pubmed/16404470   dalton, j. a., rodger, d. l., wilmore, m., skuse, a. j., humphreys, s., flabouris, m. & clifton, v.l. (2014). “who’s afraid?”: attitudes of midwives to the use of information and communication technologies (icts) for delivery of pregnancy-related health information. women and birth, 27(3), 168-173. https://doi.org/10.1016/j.wombi.2014.06.010   de leo, a., bayes, s., geraghty, s. & butt, j. (2019). midwives’ use of best available evidence in practice: an integrative review. journal of clinical nursing, 28, 4225-4235. https://doi.org/10.1111/jocn.15027   ebenezer, c. (2015). nurses’ and midwives’ information behaviour: a review of literature from 1998 to 2014. new library world, 116(3/4), 155–172. https://doi.org/10.1108/nlw-07-2014-0085   estabrooks. c. a, rutakumwa, w., o’leary, k. a., profetto-mcrath, j., milner, m., levers, m. j. & scott-findlay, s. (2005). sources of practice knowledge among nurses. qualitative health research, 15(4), 460-476. https://doi.org/10.1177/1049732304273702   fairbrother, g., cashin, a., conway, r., symes, a., & graham, i. (2016). evidence based nursing and midwifery practice in a regional australian healthcare setting: behaviours, skills and barriers. collegian, 23(1), 29–37. https://doi.org/10.1016/j.colegn.2014.09.011   guyatt, g., meade, m., jaeschke, r., cook, d., & haynes, r. (2000). practitioners of evidence based care. not all clinicians need to appraise evidence from scratch but all need some skills. bmj (clinical research ed.), 320(7240), 954–955. https://doi:10.1136/bmj.320.7240.954   guyatt, g., rennie, d., meade, m. & cook, d. (2008). users’ guides to the medical literature: a manual for evidence-based clinical practice (2nd edition). new york, ny: mcgraw-hill.   heydari, a., mazlom, s., ranjbar, h., & scurlock‐evans, l. (2014). a study of iranian nurses and midwives’ knowledge, attitudes, and implementation of evidence‐based practice: the time for change has arrived. worldviews on evidence based nursing, 11(5), 325–331. https://doi.org/10.1111/wvn.12052   kennedy, h. p., doig, e., hackley, b., leslie, m. s., & tillman, s. (2012) “the midwifery two step”: a study on evidence-based midwifery practice. journal of midwifery & women’s health, 57(5), 454-460. https://doi.org/10.1111/j.1542-2011.2012.00174.x   lafuente-lafuente, c., leitao, c., kilani, i., kacher, z., engels, c., canoui-poitrine, f., & belmin, j. (2019). knowledge and use of evidence-based medicine in daily practice by health professionals: a cross-sectional survey. bmj open, 9, 1-6. https://doi.org/10.1136/bmjopen-2018-025224   leckie, g., pettigrew, k., & sylvain, c. (1996). modeling the information seeking of professionals: a general model derived from research on engineers, health care professionals, and lawyers. the library quarterly, 66(2), 161-193.   malott, a., davis, b., mcdonald, h., & hutton, e. (2009). midwifery care in eight industrialized countries: how does canadian midwifery compare? journal of obstetrics and gynaecology canada, 31(10), 974–979. https://doi.org/10.1016/s1701-2163(16)34328-6   mckenna, l. & mclelland, g. (2011) midwives’ use of the internet: an australian study. midwifery, 27(1), 74-79. https://doi.org/10.1016/j.midw.2009.07.007   mckibbon, a., wyer, p., jaeschke, r., & hunt, d. (2008). finding the evidence. in g. guyatt, d. rennie, d. cook, m. mead & american medical association (eds.), users guides to the medical literature (pp. 29-58). new york, ny: mcgraw hill.   miller, s., abalos, e., chamillard, m., ciapponi, a., colaci, d., comandé, d., … althabe, f. (2016). beyond too little, too late and too much, too soon: a pathway towards evidence-based, respectful maternity care worldwide. lancet, 388(10056), 2176-2192. https://doi.org/10.1016/s0140-6736(16)31472-6   ontario medical association & association of midwives of ontario (2005). a joint statement of professional relations between obstetricians and midwives in ontario [position statement]. retrieved from https://www.ontariomidwives.ca.   parry, d.c. (2008). “we wanted a birth experience, not a medical experience”: exploring canadian women’s use of midwifery. health care for women international, 29(8-9), 784-806. https://doi.org/10.1080/07399330802269451   ricks, e., & ham, w. (2015). health information needs of professional nurses required at the point of care. curationis, 38(1), 1-8. https://doi.org/10.4102/curationis.v38i1.1432   ross, j. (2010). information literacy for evidence-based practice in perianesthesia nurses: readiness for evidence-based practice. journal of perianesthesia nursing, 25(2), 64-70. https://doi.org/10.1016/j.jopan.2010.01.007   sackett, d., strauss, s., richardson, w., rosenberg, w., & haynes, r. (2000). evidence-based medicine: how to practice and teach ebm (2nd ed.) edinburgh: churchill livingstone.   spenceley, s.m., o’leary, k.a., chizawsky, l.l.k., ross, a.j., & estabrooks, c.a. (2008). sources of information used by nurses to inform practice: an integrative review. international journal of nursing studies, 45(6), 954-970. https://doi.org/10.1016/j.ijnurstu.2007.06.003   thiessen, k., haworth-brockman, m., nurmi, m., demczuk, l., & sibley, k. (2018). delivering midwifery: a scoping review of employment models in canada. journal of obstetrics and gynaecology canada, 42(1), 61-71. https://doi.org/10.1016/j.jogc.2018.09.012   thompson, c., mccaughan, d., cullum, n., sheldon, t., mulhall, a., & thompson, r. (2001). research information in nurses clinical decision making: what is useful? journal of advanced nursing, 36(3), 376-388. https://doi.org/10.1046/j.1365-2648.2001.01985.x   toohill, j., sidebotham, m., gamble, j., fenwick, j., & creedy, d. (2017). factors influencing midwives’ use of an evidenced based normal birth guideline. women and birth, 30(5), 415–423. https://doi.org/10.1016/j.wombi.2017.03.008   van wagner, v. (2016). risk talk : using evidence without increasing fear. midwifery, 38, 21–28. https://doi.org/10.1016/j.midw.2016.04.009   veeramah, v. (2016). the use of evidenced-based information by nurses and midwives to inform practice. journal of clinical nursing, 25(3–4), 340–350. https://doi.org/10.1111/jocn.13054     appendix   survey instrument   how many years have you been practicing as a midwife? ·        0-40 what is your highest degree earned? ·        bachelors ·        masters ·        phd ·        other which province or territory do you practice in predominantly? what is your status as registered midwife? ·        full time ·        part time ·        non-practicing ·        other are you a midwifery educator? (preceptor, nrp instructor, faculty position etc.) ·        y/n in what setting are you currently practicing? select the best answer. ·        urban ·        suburban ·        rural ·        remote how often do you refer to clinically-relevant, external information in the practice of midwifery? (guidelines, manuals, books, websites)? 6-point very frequently – never do you hold membership in an academic library or hospital library? (check all that apply) ·        yes, through an academic library at a university where i am faculty/employed/studying ·        yes, through membership in a college or private library (e.g. cma, cfpc, cna) ·        yes, through my hospital (privileges/staff)   ·        no, i do not hold membership in an academic/hospital library ·        unsure ·        other if you answered yes to the above, how often do you access clinically-relevant information through your academic or hospital library membership? (7 point very frequently – never, n/a) how often do you visit your local public library to find clinically-relevant information for midwifery practice? (6 point very frequently – never) please rank use of the following information sources and modes of delivery in your midwifery practice: ·        manuals (nrp, alarm) ·        colleagues ·        studies (primary studies) ·        popular websites ·        print books ·        social media ·        summaries (clinical practice guidelines, systematic reviews, algorithms) ·        textbook-like e-resources (uptodate, dynamed) ·        apps please rank the challenges to using clinically relevant research in your practice ·        lack of information access ·        difficulty in judging the quality of the research evidence ·        difficulty relating research evidence to clinical practice ·        difficulty understanding statistical terms or jargon ·        lack of skills in using specialized search tools the association of college and research libraries (2016) recognizes that information sources must be evaluated “to acknowledge biases that privilege some sources of authority over others, especially in terms of others’ worldviews, gender, sexual orientation, and cultural orientations.” how relevant do you think this statement is to the practice of midwifery? (5 point very relevant – completely irrelevant) sackett et al. (2000) describe evidence-based medicine as “the integration of best research evidence with clinical expertise and patient values.” based on this statement how do you rate your competence in the following: (5-point expert – novice) ·        converting the need for information (about prevention, therapy, diagnosis) into an answerable question ·        tracking down the best evidence with which to answer that question ·        critically appraising that evidence for its validity (closeness to truth) and usefulness (clinical applicability) ·        applying evidence to the context of professional practice have you ever received library skills training (i.e., boolean operators, truncation, controlled vocabularies, databases)? check all that apply: ·        yes, during my midwifery education and training ·        yes, through my hospital or place of work ·        yes, during the acquisition of a post-midwifery credential ·        no or cannot remember ·        other (please elaborate) how interested would you be in receiving library skills instruction relevant to midwifery? (5-point very interested – uninterested) would you like to share more detail about any of the above questions?     feature   in memoriam      2013. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   the evidence based library and information practice (eblip) community is saddened by the sudden loss of dr. neils ole pors in march 2013. dr. pors was a professor at the royal school of library and information science in denmark. he previously had been dean at the school with responsibilities for both research and education. dr. pors published more than 250 papers and articles in academic and professional journals, was a member of several editorial boards of academic journals, and sat on the international program committee for eblip7. dr. pors was active in the eblip community, attending the past few conferences. he had planned to attend and present at eblip7. dr. pors will be missed.   evidence based library and information practice evidence based library and information practice 2009, 4.4     evidence based library and information practice         editorial     learning opportunities    denise koufogiannakis  editor‐in‐chief  collections & acquisitions coordinator, university of alberta libraries  edmonton, alberta, canada  email: denise.koufogiannakis@ualberta.ca      © 2009 koufogiannakis. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      we’ve reached the end of another year of  publication at eblip, my first at the helm as  editor‐in‐chief, and four full years of  publication since we began. this year was a  busy one with several changes to the editorial  team, the addition of more editorial advisors,  and new evidence summary writers joining  our team. most importantly, the journal  continues to grow and thrive, with more  people than ever participating in its success.  this year we added a new section called using  evidence in practice, and in this issue there are  two articles in that section which provide  practical examples of applying evidence in the  workplace. putting evidence into practice is  what eblip is all about, so it is my hope that  this new section makes the application of  evidence based practice more concrete for  readers.    as we began working on issue 4.4, the  editorial team decided that it would be a  good idea to seek out an editorial intern to  help with some of the tasks we never seem to  get to, given that we are all volunteers. we see  the internship as an opportunity to give a  library and information studies student a  chance to get involved with an open access  journal and learn about publishing in library  and information studies, which will hopefully  benefit them as they begin their new career.  the editorial intern will assist with marketing  and promotion of the journal, soliciting  potential manuscripts, and proofreading. they  will also participate in all editorial meetings  and general discussions. we anticipate that  they will bring a wealth of enthusiasm and  fresh ideas to our conversations so that eblip  editors can also continue to learn and grow by  having a fresh set of eyes involved with our  processes. it will certainly be a win‐win  situation and a learning opportunity for all  involved.     we are pleased to announce that andrea baer  has accepted the position of editorial intern  and joined our team in mid‐november.  andrea is currently attending the university  of tennessee, where she is a master’s student  in information science. she anticipates  completing her program in december 2010.  welcome andrea!    in this issue, we have three interesting articles  covering a variety of topics; learning outcomes  of educational games, a library website  usability study, and insights into iranian  medical librarians’ attitudes and perceptions  1 mailto:denise.koufogiannakis@ualberta.ca evidence based library and information practice 2009, 4.4  of evidence based practice. we also have  evidence summaries on topics as diverse as  professional development, information  literacy, and web usability. and don’t forget to  checkout the popular ebl101 column by  virginia wilson, which always attracts a high  number of downloads. with all this content  there should be a learning opportunity for  every reader. enjoy!  2 evidence summary theme: collections editorial   evidence summary theme: collections   fiona inglis associate editor (evidence summaries) librarian, health sciences & wellness humber college libraries toronto, ontario, canada email: fiona.inglis@humber.ca      2022 inglis. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30237     the evidence summaries in this issue all focus on the domain of collections. koufogiannakis et al. (2004) define this domain as “building a high-quality collection of print and electronic materials that is useful, cost-effective and meets the users’ needs” (p. 233).   the sudden need for more robust remote services over the past few years means that many libraries shifted their collections to prioritize access to electronic books and journals rather than print materials. in this issue, shen looks at a study on the student perspective on digital resources and patterson provides an appraisal of a study on changing print book usage at an academic library.   there is also an increased awareness of the care that libraries need to take with the cultural heritage materials that are entrusted to them. bussell explores a study about the impact of digitized indigenous knowledge collections on knowledge transmission and cultural identity.   many libraries are looking at ways to audit their collections to ensure that they reflect the communities that they serve and that a diverse range of voices are included. prince looks at an example from the field of architecture, which employs a novel approach to answering this question.   finally, as we work to build collections that are relevant to our users, it is always interesting to find out more about what they say they need. bridgeman explores a study on the role that unfulfilled requests can play in collection development.   we hope that you enjoy reading these evidence summaries and that they provide both inspiration and support for your own evidence based collections work.   references   koufogiannakis, d., slater, l., & crumley, e. (2004). a content analysis of librarianship research. journal of information science, 30(3), 227–239. https://doi.org/10.1177/0165551504044668   evidence based library and information practice evidence based library and information practice 2010, 5.2  evidence based library and information practice       article    learning from chatting:  how our virtual reference questions are giving us answers    lorna rourke  library director  st jerome’s university library, canada  email: lerourke@uwaterloo.ca    pascal lupien  research and scholarly communications librarian  university of guelph, canada  email: plupien@uoguelph.ca    received: 02 dec. 2009          accepted: 22 may 2010       2010 rourke and lupien. this is an open access article distributed under the terms of the creative commons‐ attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐ sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original  work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is  redistributed under the same or similar license to this one.    abstract    objective ‐ this research compares two types of online reference services and  attempts to determine whether the same sorts of questions are being asked; which  questions are being asked most often; and whether patron and staff behaviour is  consistent or different in the two types of online reference sessions. patron  satisfaction with the two types of online reference services is also examined.    methods ‐ the researchers reviewed over 1400 online reference transcripts,  including 744 from docutek virtual reference (vr) transactions and 683 from msn  chat reference (im) transactions. the questions were classified according to  categories of reference questions based on recurring questions discovered during  the review. each transaction was also categorized as ʺinformalʺ or ʺformalʺ based  on patron language and behaviour, and general observations were made about the  interactions between patrons and librarians. in addition, results from 223 user  surveys were examined to determine patron satisfaction with online reference  services and to determine which type of service patrons preferred.     results ‐ the analysis suggests that patrons are using vr and im services  differently. in general, vr questions tend to be more research intensive and  formal, while im questions are less focused on academic research and informal.  63 evidence based library and information practice 2010, 5.2  64 library staff and patrons appear to alter their behaviour depending upon which  online environment they are in. user surveys demonstrated that patrons are  generally satisfied with either type of online reference assistance.     conclusion ‐ both types of online reference service are meeting the needs of  patrons. they are being used for different purposes and in different ways, so it  may be worthwhile for libraries to consider offering both vr and im reference.  the relationship building that appears to take place more naturally in im  interactions demonstrates the benefits of librarians being more approachable with  patrons in order to provide a more meaningful service.    introduction     virtual reference (vr) and instant messaging  (im) reference have become increasingly  common in libraries. the library at the  university of guelph, canada has offered  online reference services since 2001. the  university of guelph is a research‐intensive  and learner‐centred institution with 18,000  full‐time students and 16,000 distance  education course enrolments. virtual reference  was introduced using library services and  systems inc. (lssi) and later using docutek  software; in 2006 an instant messaging (im)  reference service was added, using windows  live messenger (msn), so users could choose  either the docutek or msn option. the service  is currently offered as part of ʺaskonʺ online  reference service, which is provided by a  consortium of public and post‐secondary  libraries in ontario.    librarians at the university of guelph have  examined the provision of virtual reference  services to gain insights into ways to make  library users’ experiences more productive,  effective, and positive. rourke & lupien  (2007) analyzed docutek transcripts to  determine the types of questions being asked  and the language that patrons use during  virtual reference transactions. the research  presented here focuses on different methods of  providing virtual reference services to provide  evidence to help make decisions regarding  future library service provision. this includes  an analysis and comparison of transcripts  from sessions offered using docutek software  (referred to here as vr) and msn (or instant  messaging, referred to here as im) together  with a survey of user views and preferences.     review of the literature     virtual reference (vr) and instant messaging  (im) reference have become increasingly  common in libraries and there is a wealth of  literature on these services. many of the  studies have focused on analyzing the  transcripts of either vr or im sessions. these  have looked at various issues, such as: types of  questions asked; types of users and resources  used to answer questions; and quality of  responses based on standards such as those of  the association of college & research  libraries (acrl).      a number of studies have attempted to  understand what users are asking by  classifying questions into categories. sears  (2001) analyzed four months of transcripts to  identify the types of users, types of questions,  resources used by staff to answer the  questions, and the extent to which library  resources were used. marsteller & mizzy  (2003) found that of the 425 transcripts they  reviewed from carnegie melon university’s  vr service, 64% involved reference interviews.  they also classified questions according to  sears’ categories, and their results showed that  policy and directional questions  predominated. using similar categories,  rourke & lupien (2007) found that patrons  are primarily using virtual reference services  for higher‐level research assistance, rather  than using these tools to obtain quick answers  to simple questions. the two most common  types of questions required provision of  evidence based library and information practice 2010, 5.2  detailed information or instruction on a topic.  more specifically, the most frequently  occurring type of question was related to  finding journal articles on a given topic.  broughton (2003) analyzed usage data and  user surveys for one academic year at bowling  green state university (bgsu). included in  that analysis was an examination of question  type according to categories including: finding  articles on various topics; patron records;  university information; off‐campus access; and  finally, referrals. the librarians who provided  virtual reference at the library in question  created these categories.     others have conducted similar studies of im  transcripts to analyse their contents. foley  (2002) described a university of buffalo pilot  project to determine the viability of providing  a chat reference service. the service utilized  america online’s instant messenger software  and data was collected for a period of one  academic year. foley reviewed and  categorized the questions as such: information  literacy; catalogue; navigation of library  website; general library information; technical  issues; general web navigation; reserves;  university information; research; ready  reference; and, questions about virtual  reference itself. desai (2003) described the use  of im at southern illinois university by  identifying the usage, the types of questions  asked, and the impact of technology on the  reference interaction. similar to other studies,  desai categorized questions based on katz’s  categorization of reference questions with the  addition of an “other” category.      a few studies have focused on the language  that patrons use, including how users  understand library terminology and the  differences in online communication styles  between librarians and young people.  kupersmith’s (2005) research focused on  library websites with the goal of helping  libraries “decide how to label key resources  and services” so that users can “understand  them well enough to make productive  choices”. some of the important findings  include the assertion that the average user  success rate for finding journal articles or  article databases from a library website is 53%.  he attributes this relatively low success rate to  library terminology. other authors have  discussed the differences in online  communication styles between young people  and adults. maness (2007) conducted language  analysis on im transcripts to examine factors  such as word choice, grammatical structure,  and the use of emoticons. he found that the  language used during chat reference service  interactions was more formal than chat  between students, but that in terms of user  satisfaction, students seemed to appreciate  more informal language. fagan & desai (2003)  examined how effective communication  strategies could be adapted to address  common communication problems in online  environments. using transcripts to establish  best practices, they argued that the skills  needed to communicate effectively with  patrons were different in online environments,  and suggested that librarians must avoid  using library jargon and ʺrobot‐like  instructionsʺ by “speaking the patron’s  language”. they provided tips on adding a  human touch to im reference transactions,  including the use of more natural language  and emoticons. part of their research focused  specifically on language and grammar, and  the authors warned that spelling‐ and  grammar‐conscious librarians might need to  lower their standards when chatting with  patrons in an im environment. they argued  that using common im language, which may  include abbreviations, shortcuts, and  misspellings, could make the librarian seem  “more approachable and less robotic” (fagan  & desai, 2003). similarly, janes (2003) suggests  that librarians must understand instant  messaging lingo and culture to communicate  effectively with young people. radfordʹs  (2006) study of 289 virtual reference  transcripts suggested techniques that make  online reference more effective, including  ways to compensate for the lack of nonverbal  cues and strategies for building relationships.  she identified helpful library staff behaviours  to aid online interactions, including the use of  humour, emoticons, and informal language.  radford also provides examples of library  staff behaviours that diminish the  65 evidence based library and information practice 2010, 5.2  66 effectiveness of virtual reference sessions, such  as ignoring cues that the user wants more  help, providing robotic answers, and  responding in a condescending manner to  user inquiries.     while many studies have focused on  analyzing the transcripts of vr and im  sessions, there has been little research that  compares the difference with respect to how  patrons use each of these services. some have  expressed concern that, due to the less formal  and more familiar online environment, im  reference would be seen as a less “serious”  service. this brings up important questions:  do all patrons use vr and im the same way?  do they ask the same types of questions? do  they use the same language while chatting  with library staff? given that many libraries  are deciding whether to offer vr, im, or  perhaps both services, this represents a gap in  the literature that this study will address.  understanding how patrons use these services  should help libraries make decisions regarding  the type of services to offer.    methods and context     the university of guelph library offered both  vr and im reference service from 2007 to  2009. a webpage providing access to both of  these services was linked from every page on  the library website. this webpage presented  users with access to both services, with a brief  description of each, making it clear that these  were two separate services and allowing  patrons to choose the service that most  appealed to them. the same pool of staff  members, comprised of both librarians and  library associates, provided service for both  vr and im simultaneously.     types of questions asked    all of the transcripts received during a one  year period from both vr (docutek) and im  (msn) were analyzed and categorized into  five broad categories based on sears’  adaptation of katz’s categories of reference  questions (sears, 2001; katz, 1997), as outlined  in table 1 below.   in an attempt to maintain as much consistency  as possible, one person was responsible for the  question categorization. while the focus was  on the questions rather than on the answers, it  was often necessary to examine the entire  transcript to determine what the user was  asking. categorization of reference questions  is a difficult task, as it is often the case that the  question as it is asked is not necessarily the  question that needs to be answered.  furthermore, many sessions involved more  than one question. in such cases the session  was categorized based on the original question  and each session was counted in only one  category.   table 1  categories of questions based on sears’ and katz’s classification  directional questions  directing the user to a geographical location or to a place on the  library website  policy and procedure questions  inquiries about such topics as borrowing periods and fines  ready reference questions  requests for factual information; can be answered using  reference sources  specific search questions  require detailed information or instruction on a topic; users are  often referred to various sources; (e.g. questions about finding  journal articles on a topic)  research questions  higher‐level research that would normally be conducted by  faculty or graduate students.        evidence based library and information practice 2010, 5.2  the categories were broken down into  subcategories based on the topics of the  individual sessions. a total of 39 subcategories  were created across three broader categories,  directional, policy and procedural, and  specific search. the subcategories were  created based on the content of recurring  questions, i.e., if there were a sufficient  number of questions (more than five) on a  particular topic, a subcategory was created to  classify similar questions. the intention was to  be as specific as possible about the types of  subjects that patrons were asking about. this  helped to determine, among other things, the  most commonly asked questions. this process  was completed for transcripts from both the  vr and im services. once the analysis of  questions from both services was complete,  the percentages for each category and  subcategory were calculated to facilitate  comparisons between the two methods of  service provision.    language and behaviour    following categorization, the language used  by patrons and the behaviour demonstrated  for each question was examined. in comparing  the vr and im transcripts, the interactions  were categorized as either “formal” or  “informal”. the researchers appraised  language and behaviour by reviewing  spelling, greetings, emoticons, punctuation,  word and phrase choice, shortcuts, and  goodbyes. decisions on categorization were  made based on the overall tone of the  transcript, and took into account the  behaviours previously noted. once the  language coding was completed, it was  possible to compare the types of language  used in both services.    user preferences survey    for a period of one year, users were surveyed  about their satisfaction and preferences with  respect to vr or im. the surveys were pushed  to users in vr through the docutek software  and the responses went directly into a  docutek database. im users were provided  with a link to an identical online survey, and  completed surveys were stored in the library’s  own database. users of both services were  asked if they would prefer to receive service  through the library’s docutek software or  through msn, and were asked how they rated  each service (see appendix 1).    at the end of the data collection phase the  researchers reviewed 223 surveys, of which  151 were from vr and the remaining 72 were  from the im service. a simple excel  spreadsheet was used to track responses and  to calculate percentages.     results    types of questions asked: categories    the researchers reviewed a total of 1,427  reference transcripts, 744 from vr (docutek)  and the remaining 683 as im (msn)  transcripts. an analysis and comparison of  questions from both services demonstrates  that patrons use vr and im reference services  differently (see figure 1). in vr, specific search  questions make up nearly half (47%) of the  questions asked by patrons in comparison to  31% in im. these questions typically involve  actual information seeking on the part of the  patron. policy and procedural questions  comprise the next largest category in vr  (35%), reflected by a similar number received  in this category via im (32%). the vr service  received fairly few directional questions (13%)  and even fewer ready reference questions  (4%). less than 1% of vr‐based questions  could be classified as higher‐level research  questions. in im, there were far more  directional questions (37%), more ready  reference questions (10%), and no higher‐level  research questions. these findings suggest  that patrons are using these services for  different purposes. whereas vr questions  tend to be more research intensive, with a  large number of patrons seeking information  for academic purposes, patrons use im to ask  fewer information‐seeking questions and more  of the other types of questions.   67 evidence based library and information practice 2010, 5.2  fig. 1. categories of questions by service, shown as percentages   the subcategorization of questions allowed for  the development of lists of commonly asked  questions for both vr and im. for both  services, finding information was the most  common subcategory. for vr, the most  frequent type of question involved finding  journal articles for a paper (n=241). the second  most frequent type of question was general  information search (n=152). questions were  classified into this category when the focus  was on finding any information on a topic, not  specific to locating journal articles. both of  these subcategories belong to the specific  search category. for im, these subcategories  were reversed, with the largest number of  questions being requests for general  information (n=152) and the second largest  number being for assistance finding journal  articles (n=73). however, even though these  two subcategories occur the most frequently,  the numbers are much smaller for im than for  vr, and this is reflected by the overall  percentages for the broad categories discussed  in the previous section. while questions from  these two subcategories were the most  frequently asked questions via im, there were  fewer of these questions overall, and therefore  specific search questions made up a smaller  percentage of the overall total.     other than these top two subcategories, the  most frequent questions were different in vr  and im, but all belonged to the policy and  procedure or directional categories. in vr,  these top five most frequent questions made  up nearly 75% of the questions asked. again,  this is reflected in the percentages: nearly half  of questions are specific search, and most of  those are finding journal articles or general  search (the top two subcategories). in im, the  top five most frequent questions made up only  half of all questions asked.  this illustrates the  much greater variety of questions asked via  im, which is demonstrated by the fact that the  numbers for the top two subcategories are  much lower.  table 2  top 5 recurring questions by service (subcategories)   vr (docutek)  im (msn)  1. finding journal articles (n=241)  1. finding general information on a topic  (n=152)  2. finding general information on a topic  (n=152)  2. finding journal articles (n=73)  3. access problems/trouble accessing a  particular resource (n=84)  3. technical help (word, e‐mail, etc.)  (n=55)      4. circulation (n=44)  4. citing sources (n=48)  5. logging in from off campus (n=41)  5. finding something on the library  website (directional) (n=47)  68 evidence based library and information practice 2010, 5.2  online language and behaviour    researchers examined the online language  and behaviour used in vr and im reference  interactions in order to help librarians better  understand the patrons they may be helping.  it became evident that the use of spelling and  grammar in im was predictably similar to the  spelling and behaviour patrons would use in  any casual msn interaction, as with a friend.  typical im transcripts included such  ‘shortcuts’, words, misspellings, and  expressions as:    • “hey”, “yup”, “thank u”, “ya”,  “thx”, “u 2”  • “bye4now”, “☺”, “yea”, “hey  baby”, “sup”  • “sujestion”, “convience”, “can u  verify”, “emprirical”, “bye for  nowwwwwwwww i havr to go”,  “infromation”, “familair”, “capital  punichment”    in vr, 149 transactions (21%) were considered  to be informal, while 565 (79%) were formal.  in im, the percentages were reversed, as 498  (77%) of transactions were informal and 145  (23%) were formal. the pie charts in figure 2  illustrate the near‐opposite types of behaviour  demonstrated across the two types of services.  staff behaviour    the researchers noticed that library staff  behaviour also changed depending on the  online environment they were using. staff  members tended to use less formal language  in im than they did in vr, and were less  careful with spelling and sentence structure.  staff also provided longer and sometimes  more thorough answers in vr than in im,  perhaps because of the casual approach taken  by both patron and user in im sessions and the  more formal approach taken in vr. for  example, staff would search for information in  several sources during vr interactions, while  they tended to perform less detailed searches  in im sessions. the researchers observed that  staff tended to refer patrons elsewhere in im  more frequently than they did in vr,  sometimes commenting that the question was  too complex to be answered in im and that the  student should visit a reference desk in the  library. in some sessions library staff indicated  that the patron would need to be shown how  to use a database or other reference source in‐ person by library staff; this sort of assistance  was not possible in the im environment  because of the limitations of the software.  when questions could not be completely  answered online, an offer by staff to follow up  with patrons was made in about the same  number of vr and im sessions.      fig. 2. formal vs. informal transactions by service                69 evidence based library and information practice 2010, 5.2  user preference surveys     undergraduate students comprised 85% of the  respondents to the im survey, while 9% were  graduate students. for the vr survey, 60% of  respondents were undergraduates, followed  by graduate students (19%) and people not  affiliated with the university (17%).     the results showed that patrons were satisfied  with both methods of service provision, as  70% expressed satisfaction with vr and 71%  with im. while the overall number of  unsatisfied patrons was small, there were  twice as many not satisfied in im, at 16%,  versus the 8% of  vr respondents who  indicated they were not satisfied. when asked  about their preferences with respect to using  these services, the majority of im users  preferred im to vr. close to 80% of users who  answered the im online survey said they  would rather use im than the library’s vr  software. more surprisingly, over 50% of vr  respondents who used the vr software also  claimed to prefer im. however, a significant  minority of vr patrons indicated their  preference for vr service (about 40%).     when the numbers from both surveys are  combined, 67% of online reference users  preferred im. furthermore, when users were  asked to choose from all of the methods of  getting help provided by the library, im was  chosen by almost twice as many users (about  40%) as the next highest option, in‐person  reference, at just over 20%. it is essential to  remember that these responses were from  patrons who were already using vr or im.    discussion    this study was undertaken first to help inform  decisions regarding future provision of virtual  reference services, particularly whether one or  both service‐types should be retained, or if  online reference should be discontinued. the  study enabled library staff to make an  informed decision and to determine if the  same types of patrons were using the services  for the same purposes. secondly, the  researchers wanted to develop their  understanding of patrons’ online behaviours  in order to serve them better.   in terms of distinguishing between vr and  im, it is clear that these services are used  differently in terms of both the questions  asked as well as the online behaviours  exhibited by patrons. the greater formality of  the vr environment and the fact that more  information‐seeking questions were asked via  vr are of particular interest. by extension,  more instruction takes place in vr due to the  types of questions being asked. if patrons are  using vr for more “serious” academic work  than via im, librarians may need to account  for this when considering which service is  most appropriate for their libraries.    with respect to language and online  behaviour, the more informal and  approachable online environment offered by  im appears to be conducive to establishing  strong relationships with users. in im, patrons  tended to come back throughout the day and  to get to know staff on a first name basis,  elements that were not apparent in the vr  interactions. the relationship created in im  may provide some benefits for promoting the  library as a welcoming place, and library staff  as approachable people.    for many of the library staff members, their  first exposure to online chat occurred when  they began participating in online reference  service. most had not previously used msn or  any similar service. as a result, it was  necessary for those individuals to learn the  shortcuts and abbreviations used in msn in  order to provide good reference service.  expressions such as “brb” (be right back),  “lol” (laugh out loud”), and “g2g” (got to go)  were puzzling to staff at first, but as time  passed they learned their meanings and  became adept im language interpreters. one  of the im transcripts used for this study  demonstrated how participating in im service  can provide a learning experience not only for  the patrons but for the staff as well. in this  example, the staff person (who had  incidentally grown up on a dairy farm) had a  picture of a cow as his im picture. the  student’s first comment in the im interaction  70 evidence based library and information practice 2010, 5.2  was “nice dp”, to which the staff member  replied “dp??”. the student explained  “display picture”, meaning that he liked the  picture of the cow. the staff member took this  in stride and responded “ah! gotcha! that’s  useful.” this very informal interaction  provides insight into the high comfort level of  patrons in im as well as the possibility for  relationship building and light hearted  communication between librarians and  patrons.    it is important to consider the reasons for the  discrepancy in behaviour between the two  types of reference services. in vr, the online  environment is a foreign one: it does not  resemble other online communication venues  commonly used by the patrons, such as msn,  aol, twitter, or google chat. sessions do not  begin until the student has entered a  username, email address, and question, and  the library staff member must accept the  patron before the interaction can begin.  patrons are asked (but not required) to enter  their university of guelph email address and  program of study, which means that they are  usually not anonymous. the initial message  the patron receives from the librarian is pre‐ scripted and is delivered to the patron as soon  as the library staff member accepts them. it  reads: “welcome to your virtual reference  session. i am looking at your question right  now. i will be just a minute.” the fact that the  librarian is completely in control of the session  and that the interaction begins so formally  may set the tone for these interactions, and  may explain, at least in part, why almost 80%  of these exchanges are formal. librarians and  patrons tend to write in complete sentences  and almost always use correct spelling. there  is usually a formal goodbye, which includes  the scripted librarian message: “thank you for  using our virtual reference service. please do  not hesitate to ask for help again in the  future”.     by comparison, im sessions begin when the  patron initiates the chat, and there is no need  for the library staff member to accept the  patron. most interactions begin with the  patron saying “hey”, “yo”, or a similarly  informal greeting, or they launch immediately  into their question. the im environment is  familiar to them, as it is one they use with  their friends, so they are comfortable and do  not hesitate to add the library staff to their  “friend” list. there are no pre‐scripted  messages in im, so library staff tend to use  more natural language when dealing with  patrons. while patrons use their real names  and university email addresses in vr, their im  names and email addresses are of their own  choosing and are often extremely informal.  some of the im names used in the transcripts  being studied demonstrate the casual nature  of msn. they include:    • a day without sunshine is like, you know,  night   • procrastinators unite tomorrow!   • association of anti‐redundancy  association   • last night i got hit in the head with a  brick and woke up a conservative    one student’s msn name, “rip squeekers, my  little man”, led to a discussion between the  student and librarian about a beloved hamster  that had recently died. this sort of interaction  humanizes the library and can lead to positive  interactions between staff and patrons. in  reviewing the ways im sessions ended, many  did not have any formal closure or goodbye.  patrons either closed the session once their  question had been answered, said “bye” and  closed the session, or in some cases they kept  the conversation open all day as they popped  in and out with questions. this last behaviour  provided opportunities to build relationships,  as the reference interactions became more  conversational.    there is also the possibility, based on the type  of language used in im, that this service  attracts younger users. in another survey  conducted at the university of guelph,  researchers noticed a clear trend that younger  patrons are far heavier users of im and other  social software (rourke & lupien, 2007). the  more informal language and behaviour in im  may be a clue that this service is attracting a  younger clientele.   71 evidence based library and information practice 2010, 5.2  the fact that staff behave more casually in im  interactions than they do in vr can probably  be attributed to the same reasons that patrons  behave differently. the online world is a  distinct environment, and library staff take  their behavioural cues from the patrons,  although not to the same degree of  informality. there are advantages and  disadvantages to the fact that staff behave in a  more relaxed manner in im. for example,  sessions are friendlier and library staff usually  come across as approachable. however, the  reference interview tends to be more casual  and the service given may not be as thorough  as that provided in vr. researchers found that  im interactions tended to be more  conversational and less probing.    with respect to user preferences revealed in  the surveys, one can question why the patrons  using vr who claimed to prefer im had  chosen to use the former. it is possible that  they were not aware that the library offered  reference service through im, even though the  availability of both services were displayed  side‐by‐side on the library’s website. library  statistics did demonstrate that the number or  im users steadily increased while the number  of vr users was stagnant.     the transcript analysis project allowed  librarians to learn from patrons’ online  language and behaviours. understanding the  language used in im can help library staff  serve patrons better by improving  communication techniques in vr and im, and  with other types of social software as well.  users employ “chat talk” in various social  software tools and the web 2.0 environment  (facebook, myspace, second life). since  libraries use these tools to deliver reference  and other services, understanding the  language used in these environments enables  more effective communication between library  staff and patrons. one of the criticisms of  online communication is that it lacks the non‐ verbal cues of  in‐person communication, and  further, im conversations can be cold and  impersonal, or stiff and robotic. developing a  better understanding of how patrons  communicate online will allow library staff to  overcome these weaknesses. as in the real  world, librarians can make themselves  approachable online by understanding the  online communication style and using it to  their advantage. librarians and other library  staff must ask themselves how they are going  to approach future im and vr interactions.     this article and others have examined the kind  of language and grammar used by students  and young people in online communication.  since the advent of vr service several years  ago there has been some debate about whether  librarians should actually be using chat talk to  respond to patrons (gorman, 2001; coffman,  2004). the debate continued with the advent  of im service, since people are even more  likely to use chat talk, including non‐standard  spelling and grammar, in that particular  environment. those who study reference  services point out that librarians should  always make an effort to speak the patrons’  language (katz, 2007; fagan & desai, 2003).  the idea behind this principle is to make the  patron feel comfortable, and therefore  librarians should “speak the patrons’  language” in the online environment. to what  extent librarians should do that is a question  still open for debate.    future areas of study    while this research has provided insight into  some aspects of online reference, there are  other areas that could benefit from further  study. for example, future research could  focus on the differences and quality of  answers provided in vr and im  environments. there could also be an  investigation into the reasons for the increase  in im interactions while the number of patrons  using vr is stagnant.     conclusion    this study suggests that software‐based vr  and im reference services are assisting  different types of patrons and users who are  looking for help with different types of  questions. in general, vr interactions are  associated with more “academic” types of  72 evidence based library and information practice 2010, 5.2  73 activities, such as finding journal research, and  vr language and behaviour tends to be more  formal. in contrast, im was used more  frequently for other, less “academic” types of  queries, im appears to be more popular with  patrons, and interactions between patrons and  library staff take a friendlier tone in the im  environment. there are therefore advantages  and disadvantages for each service, and  different libraries will have different issues to  consider and may not all reach the same  conclusions. libraries looking to adopt or  expand their reference services will have to  make decisions according to their respective  situations and their patron needs. overall,  however, this research suggests that the two  types of service are complementary, providing  evidence to suggest that both services should  be continued.      acknowledgements    the authors acknowledge the generous  support given to their research by st. jerome’s  university and the university of guelph  library.      references    broughton, k. m. (2003). usage and user  analysis of a real‐time digital reference  service. the reference librarian,  38(79/80), 183‐200.  coffman, s., & arrent, l. (2004). to chat or not  to chat: taking yet another look at virtual  reference. searcher, 12(8), 49‐56  desai, c. m. (2003). instant messaging  reference: how does it compare?  electronic library, 21(1), 21‐30.   fagan, j. c., & desai, c. m. (2003).  communication strategies for instant  messaging and chat reference services.  the reference librarian, 38(79/80), 121‐55.   foley, m. (2003). instant messaging reference  in an academic library: a case study.  college & research libraries, 63(1), 36‐45.    gorman, michael. (2001). values for human‐ to‐human reference. library trends,  50(fall), 165‐182.  janes, j. (2003). digital reference for teens.  voice of youth advocates, 25(6), 451.  katz, w.a. (1997). introduction to reference work,  volume 1: basic information service. new  york: mcgraw‐hill.    kupersmith, j. (2005) library terms that users  understand. retrieved 29 nov. 2009 from  http://www.jkup.net/terms.html    maness, j. (2008). a linguistic analysis of chat  reference conversations with 18‐24 year  old college students. journal of academic  librarianship, 34(1), 31‐38.    marsteller, m. r. & mizzy, d. (2003).  exploring the synchronous digital  reference interaction for query types,  question negotiation, and patron  response. internet reference services  quarterly, 8(1&2), 149‐165.    radford, m. l. (2006). encountering virtual  users: a qualitative investigation of  interpersonal communication in chat  reference. journal of the american society  for information science and technology,  57(8), 1046‐1059.    rourke, l. e. & lupien, p. (2007). out of the  question: how we are using our students’  virtual reference questions to add a  personal touch to a virtual world.  evidence based library and information  practice, 2(2), 67‐80.  sears, j. (2001). chat reference service: an  analysis of one semesterʹs data. issues in  science & technology librarianship, 32.   retrieved 3 may 2010 from  http://www.library.ucsb.edu/istl/01‐ fall/article2.html    evidence based library and information practice 2010, 5.2  appendix 1  virtual reference user survey, vr (docutek) service   please help us improve our service to you by filling out this survey. thank you.      1. please select the group that best describes you     u of g undergraduate student       u of g graduate student        u of g faculty        other        2. which of the following ask us options would you prefer to use?       msn messenger instant messaging/chat        current software (docutek)      (in the im survey,   the “current software” was   msn and the docutek was given as a second  option)    3. what would be your preferred method of getting research help? (select all that apply)     instant messaging/chat (msn, yahoo messenger, googletalk, etc.)        email        current software (docutek)        telephone        in person in‐person        4. how would you rate your overall satisfaction with the answer(s) provided to you in this  service?       not satisfied        somewhat satisfied        very satisfied      74 evidence summary   open access works are as reliable as other publishing models at retracting flawed articles from the biomedical literature   a review of: peterson, g.m. (2013). characteristics of retracted open access biomedical literature: a bibliographic analysis. journal of the american society for information science and technology. 64(12), 2428-2436. doi: 10.1002/asi.22944   reviewed by: elizabeth stovold information specialist, cochrane airways group st george’s, university of london tooting, london, united kingdom email: estovold@sgul.ac.uk   received: 16 jun. 2014   accepted: 23 jul. 2014      2014 stovold. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to investigate whether the rate of retracted articles and citation rates post-retraction in the biomedical literature are comparable across open access, free-to-access, or pay-to-access journals. design – citation analysis. setting – biomedical literature. subjects – 160 retracted papers published between 1st january 2001 and 31st december 2010. methods – for the retracted papers, 100 records were retrieved from the pubmed database and 100 records from the pubmed central (pmc) open access subset. records were selected at random, based on the pubmed identifier. each article was assigned a number based on its accessibility using the specific criteria. articles published in the pmc open access subset were assigned a 2; articles retrieved from pubmed that were freely accessible, but did not meet the criteria for open access were assigned a 1; and articles retrieved through pubmed which were pay-to-access were assigned a 0. this allowed articles to be grouped and compared by accessibility. citation information was collected primarily from the science citation index. articles for which no citation information was available, and those with a lifetime citation of 0 (or 1 where the citation came from the retraction statement) were excluded, leaving 160 articles for analysis. information on the impact factor of the journals was retrieved and the analysis was performed twice; first with the entire set, and second after excluding articles published in journals with an impact factor of 10 or above (14% of the total). the average number of citations per month was used to compare citation rates, and the percentage change in citation rate preand post-retraction was calculated. information was also collected on the time between the date the original article was published and the date of retraction, and the availability of information on the reason for the retraction.  main results – the overall rate of retracted articles in the pmc open access subset compared with the wider pubmed dataset was similar (0.049% and 0.028% respectively). in the group with an accessibility rating of 0, the change in citation rate preand post-retraction was -41%. for the group with an accessibility rating of 1, the change was -47% and in those with a rating of 2, the change in citation rate was -59%. removing articles published in high impact factor journals did not change the results significantly. retractions were issued more slowly for free access papers compared with open or fee-based articles.  the bibliographic records for open access articles disclosed details of the reason for the retraction more frequently than free, non-open papers (91% compared to 53%). conclusion – open access literature is similar in its rate of retraction and the reduction in post-retraction citations to the rest of the biomedical literature, and is actually more reliable at reporting the reason for the retraction.   commentary   open access is a growing movement, and therefore an examination of the quality of biomedical literature published through an open access arrangement is of great importance. retracted articles in the biomedical literature are relatively rare, but can have a big impact if the error is of such significance that it could negatively affect research or practice based on the evidence presented in the retracted article. to illustrate this the author cites the example of a discredited paper on mmr vaccines which resulted in a reduction of the uptake of mmr vaccines in the uk.   this study was evaluated using the critical appraisal tool developed by perryman (2009). the inclusion and exclusion criteria of the articles were appropriate, and reasons for excluding articles from the analysis were explicitly stated. the sources used to retrieve retracted articles and citation information were clearly reported and the choice justified. the author used the number of average citations per month to compare citation rates which controlled for differences in the length of the preand post-retraction periods and allowed articles published at different times during the decade under study to be compared. the author also examined other factors which may affect the citation rates following retraction, such as the impact factor of the journal. the author discussed limitations of the study; it was noted that there was no control group for the study, the sample of retracted articles were retrieved from a single source (pubmed), and it may not be possible to extrapolate these findings to subject areas outside of biomedicine, or to other types of post-publication changes in the literature.                 potential limitations not discussed in the paper were that a single author conducted the study, therefore judgements on the accessibility value of each paper were not validated, and the sample of retracted articles in the analysis was small. the study by furman, jensen & murray (2012) used a sample of 677 retracted articles and 1,340 control articles, compared to 160 in this study. however, the sample size in this study is restricted by the availability of retracted articles published under an open access arrangement.   overall this study presents a thorough evaluation of the characteristics of retracted articles compared by their accessibility and finds that open access journals are as reliable as the traditional publishing models in retracting flawed work. this is reassuring for librarians and searchers who may be recommending open access journals to researchers and practitioners in the biomedical field as sources of reliable evidence.     references   furman, j. l., jensen, k., & murray, f. (2012). governing knowledge in the scientific community: exploring the role of retractions in biomedicine. research policy, 41(2), 276–290. doi: 10.1016/j.respol.2011.11.001   perryman, c. (2009). critical appraisal tool for bibliometric studies. in carol perryman. retrieved from http://cperryman.com/critical%20appraisal%20tool%20-%20bibliometric%20study.doc microsoft word news_chla.doc evidence based library and information practice 2007, 2:4  119 evidence based library and information practice     news    chla: call for contributed papers and posters        © 2007 glynn. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    the canadian health libraries association  invites you to submit contributed papers or  posters for its 2008 annual conference, to be  held in halifax, nova scotia, may 23‐30.  papers may describe innovative  programs/practices or new research findings  and should relate to the overall conference  theme ‐ “navigating the seas of change”.    notice: the deadline for the submission of  abstracts for papers and posters has been  extended to january 15, 2008.     call for contributed papers    in the library world, change is practically  the only constant. the 2008 chla  conference  will focus on managing change  and changing practice. as information  professionals we seek out new practices and  technologies. we evaluate them and adopt  the useful ones. evidence based library and  information practice (eblip) uses evidence  derived from research, user feedback and  observation to improve practice. it provides  excellent guidance to improve library and  information service.      you are invited to share your research  evidence, combined with practical  observations about the primary ways  information professionals improve  information delivery. how has the  framework moved from librarians storing  physical information, to information  professionals facilitating knowledge  use?  how do we plan to survive and thrive  in tomorrow’s information age?      papers will be accepted from a broad range  of topics, including in particular:    changing access   • implication for information  professionals of new technologies and  portals like nurse one;  • health literacy – access for patients  where it counts;  • use of health technology assessment  from cadth & other agencies.    changing practice and collaboration   • research, teaching, reaching beyond the  walls and outside the profession, for  example collaborations between  professional organizations, libraries in  different countries, and partnerships  between hospitals and universities;  • implementing research, knowledge  transfer, evidence‐based practice &  systematic reviews.     http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:4  120 publishing/ open access  • oa research and projects like eblip  journal and open medicine.    accepted contributed papers will each be  allotted approximately 20 minutes for  presentation, plus 5‐10 minutes for  questions.    to be considered, please submit the  following information by january 15, 2008.    • title of the paper;  • a short structured abstract of 250 words  or less;  • primary authorʹs name, address and  business phone number.    structured abstracts should follow jchla  instruction to authors at   http://pubs.nrc‐cnrc.gc.ca/jchla/jchla26/c04‐ 900.pdf    submissions should be e‐mailed to the  program committee member, paul clark, at  paul.clark@rvh.nb.ca. the program  committee will contact all those submitting  abstracts for contributed papers by january  19, 2008. conference registration fees will  not be waived for presenters of contributed  papers.    call for posters    poster sessions comprise visual  presentations of information about research  or other projects. they offer conference  delegates a less formal alternative to  contributed papers and plenary sessions, as  well as an opportunity to interact directly  with those responsible for the presentations.    posters are invited along the same themes as  those of the contributed papers, although  submissions along other themes also are  welcome.    poster presenters will be allotted a display  board (4ʹ x 8ʹ) and table space.  a limited  number of electrical connections will be  available for electronic poster presentations.  if you are planning an electronic poster  presentation, you must bring your own  laptop computer as well as a standalone  version of your presentation.     you will be required to staff your print or  electronic poster during a portion of the  conference; posters will be open to viewing  by conference delegates for two days of the  conference.    to submit a poster for consideration, please  send an email by january 15, 2008 with the  following information:    • outline of the topic;  • a brief structured abstract (no more than  250 words).  structured abstracts should follow  jchla/jabsc instruction to authors.   (http://www.chla‐absc.ca/journal/index.html)    submissions should be e‐mailed to the  poster chair, ann barrett, at  ann.barrett@dal.ca.    conference registration fees will not be  waived for presenters of posters. the poster  committee will contact all those submitting  abstracts for posters by january 19, 2008.     tips on designing posters: include the title,  the author(s), affiliation(s), and a description  of the research, highlighting the major  elements that are covered in the structured  abstract. posters are visual – add pictures,  graphs, charts etc. to make the poster  interesting. keep text to a minimum and use  a large font size so that the poster can be  read from a distance. keep some white  space ‐ don’t overwhelm the reader with too  much text or graphics. use color creatively.  if you laminate the poster, consider using a  low‐glare or matte finish. consider bringing  copies of your poster for handout. ensure  http://pubs.nrc%e2%80%90cnrc.gc.ca/jchla/jchla26/c04%e2%80%90900.pdf http://pubs.nrc%e2%80%90cnrc.gc.ca/jchla/jchla26/c04%e2%80%90900.pdf mailto:clark@rvh.nb.ca http://www.chla%e2%80%90absc.ca/journal/index.html mailto:barrett@dal.ca evidence based library and information practice 2007, 2:4  121 that your poster is set up and taken down at  the specified times.    for more information, please check these  websites:  http://www.soe.uoguelph.ca/webfiles/agalv ez/poster/  http://educ.queensu.ca/~ar/poster.htm  http://www.esf.edu/its/html/posterref.htm     for more information about themes and  sample poster topics, contact the poster  chair. posters from previous conferences  may be found at:  http://www.chla‐ absc.ca/assoc/conference.html                  http://www.soe.uoguelph.ca/webfiles/agalv http://educ.queensu.ca/~ar/poster.htm http://www.esf.edu/its/html/posterref.htm http://www.chla%e2%80%90absc.ca/assoc/conference.html http://www.chla%e2%80%90absc.ca/assoc/conference.html http://www.chla%e2%80%90absc.ca/assoc/conference.html evidence based library and information practice evidence based library and information practice 2010, 5.4 evidence based library and information practice evidence summary identifying the most popular entry routes into a public library using gis can be a tool to increase ease of navigation and identify placement of marketing materials a review of: mandel, l. h. (2010). toward an understanding of library patron wayfinding: observing patrons’ entry routes in a public library. library & information science research, 32, 116-130. reviewed by: genevieve c. gore library coordinator, ruis mcgill life sciences library, mcgill university montreal, quebec, canada email: genevieve.gore@mcgill.ca received: 1 sept. 2010 accepted: 16 oct. 2010 2010 gore. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to evaluate and measure how patrons physically navigate entry routes within a public library and determine whether gis is a useful instrument for this purpose. design – unobtrusive, covert observational study. setting – medium-sized public library in the united states. subjects – 1,415 patrons were observed as they entered the library. methods – routes used by patron cases were selected as the unit of analysis. patron cases were either individuals entering the building alone or groups entering the building together. patrons were observed from a stationary and unobtrusive location. arcmap (gis software) was used to develop the floor plan instrument on which entry routes were recorded and then later analyzed. the paths analyzed were limited to what was considered the “entry area.” data were collected during three separate one-hour periods for six consecutive days in the fall of 2008. the researcher chose three purposive one-hour time samples with the intention of distributing them across the library’s opening hours. main results – the 1,415 patron cases used 195 unique routes that were recorded from the two entrances of the facility, with the east (right) entrance accounting for 83.3% of the cases (n=1178). two entry routes were consistently the most popular overall and across each of the sample days. the next-mostmailto:genevieve.gore@mcgill.ca� evidence based library and information practice 2010, 5.4 popular entry routes did not remain constant across the total observed cases and each day’s observed cases or across the sample days. over 75% of all observed patrons used 22 of the 195 entry routes: 7 routes were used by 30 or more cases each (n=836, 59.1% of all cases), 4 by 20 to 29 cases each (n=95, 6.7% of all cases), and 11 by 10 to 19 cases each (n=159, 11.2% of all cases). the route to the circulation desk was the most popular entry route for patrons. the other most popular route passed toward the rear of the library, but the observer could not record the final destination(s) of that route due to the restricted viewable area. conclusion – the study helped the researcher to establish what areas would be ideal locations for the placement of marketing materials and a book display. knowledge of popular entry routes can also be useful in identifying routes that could be enlarged to ease patron navigation. gis was shown to be a useful mapping instrument for recording and analyzing routes taken. commentary this study looks at public library use in terms of how patrons navigate entry routes. according to the author, little research is available on how patrons of public libraries use existing facilities, as user-oriented studies have tended to focus on patron satisfaction with public library services rather than patron use of facilities. based on the author’s literature review, this study presents a new approach to research on public library use. the decision to use gis software was well justified and innovative: the author contributed a novel approach to mapping wayfinding behaviour in public libraries, which could be adapted by any library interested in this worthwhile area of research. the observational design provided a nondisruptive method of recording patrons’ use of entry routes. as pointed out by the author, the design was not ethically ideal given patrons’ lack of knowledge of being observed. the study was not, however, designed to elicit additional information from patrons and did maintain anonymity. as a result, it was not possible to know, for example, how many of their routes led to successful or efficient outcomes (i.e., patrons easily reaching their intended destination). the design did allow the researcher to establish empirical knowledge of frequently used routes, thereby providing useful insight into how the layout could be improved to increase the navigability of the most frequently used routes as well as insight into well-frequented locations for the placement of marketing materials. based on the nature of the study design, however, the author chose not to use statistical tests of significance. therefore, it was not possible to establish statistically significant differences in route popularity. the results with regard to the popularity rankings of the routes themselves were somewhat limited by the fact that the observer was only viewing wayfinding in the entry area, and the units of analysis were not, due to the nature of the study, necessarily indicative of single routes: one route may have actually been several combined, due to the possible variations beyond the field of visibility. under “directions for future research,” the author makes some interesting suggestions for future wayfinding studies, such as involving patrons in surveys, interviews, and focus groups, designing experiments around possible interventions, and using “public participatory gis” to increase patron involvement and input into our understanding of wayfinding behaviour. in conclusion, this study provided a highly transferable methodological model for the use of gis in mapping wayfinding behaviour, while also addressing an important and useful area for further study and consideration. / evidence based library and information practice evidence summary research article   coping with impostor feelings: evidence based recommendations from a mixed methods study   jill barr-walker clinical librarian  zuckerberg san francisco general hospital library university of california, san francisco san francisco, california, united states of america  email: jill.barr-walker@ucsf.edu   debra a. werner director of library research in medical education john crerar library university of chicago chicago, illinois, united states of america email: dwerner@uchicago.edu   liz kellermeyer biomedical research librarian library & knowledge services national jewish health denver, colorado, united states of america email: kellermeyerl@njhealth.org   michelle b. bass manager, research and instruction countway library of medicine harvard medical school boston, massachusetts, united states of america email: michelle_bass@hms.harvard.edu   received: 8 jan. 2020                                                                  accepted: 11 apr. 2020    2020 barr-walker, werner, kellermeyer, and bass. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     data availability: barr-walker, j., bass, m. b., werner, d. a., & kellermeyer, l. (2019). coping with impostor feelings: evidence-based recommendations from a mixed methods study (v2) [survey instrument, data, codebooks]. uc san francisco. https://doi.org/10.7272/q65t3hp6. the survey instrument is available in appendix 1. the de-identified data and codebook as well as qualitative codebooks are available in data appendices 2 and 3.   doi: 10.18438/eblip29706     abstract   objective – the negative effects of impostor phenomenon, also called impostor syndrome, include burnout and decreased job satisfaction and have led to an increased interest in addressing this issue in libraries in recent years. while previous research has shown that many librarians experience impostor phenomenon, the experience of coping with these feelings has not been widely studied. the aim of our study was to understand how health sciences librarians cope with impostor phenomenon in the workplace.    methods – we conducted a census of 2125 medical library association members between october and december 2017. an online survey featuring the harvey impostor phenomenon scale and open-ended questions about coping strategies to address impostor phenomenon at work was administered to all eligible participants. we used thematic analysis to explore strategies for addressing impostor phenomenon and one-way analysis of variance (anova) to examine relationships between impostor phenomenon scores and coping strategies.   results – among 703 survey respondents, 460 participants completed the qualitative portion of the survey (65%). we found that external coping strategies that drew on the help of another person or resource, such as education, support from colleagues, and mentorship, were associated with lower impostor scores and more often rated by participants as effective, while internal strategies like reflection, mindfulness, and recording praise were associated with less effectiveness and a greater likelihood of impostor feelings. most respondents reported their strategies to be effective, and the use of any strategy appeared to be more effective than not using one at all.   conclusions – this study provides evidence based recommendations for librarians, library leaders, and professional organizations to raise awareness about impostor phenomenon and support our colleagues experiencing these feelings. we attempt to situate our recommendations within the context of potential barriers, such as white supremacy culture, the resilience narrative, and the lack of open communication in library organizations.      introduction   impostor phenomenon, also known as impostor syndrome or impostor experience, is defined as an internal feeling of not deserving personal success that has been rightfully achieved (clark et al., 2014). individuals experiencing impostor   phenomenon may believe that they have fooled others into overestimating their own abilities; attribute personal success to factors other than their own ability or intelligence, such as luck, extra work, charisma, or misjudgment; and fear exposure as an impostor (clark et al., 2014). existing research has shown that up to 70% of the population has experienced impostor phenomenon (harvey & katz, 1985), and many people suffer from its associated adverse effects such as anxiety, depression, lack of confidence, decreased job satisfaction and performance, and inability to achieve in the face of self-imposed unattainable goals, which can lead to burnout (parkman, 2016).     the current study is part of a larger research project that hypothesized higher rates of impostor phenomenon among health sciences librarians compared to college and research librarians, primarily because of the lack of educational background health sciences librarians hold in their subject areas. this effect was not found, suggesting that the current study's findings around coping strategies are broadly applicable across the academic librarian community.   while anecdotal evidence and a growing body of research have shown that impostor phenomenon in librarianship exists, no studies have examined how librarians cope with this phenomenon. our study attempts to provide an evidence base for recommendations to address impostor phenomenon among librarians.    literature review   impostor phenomenon has been studied extensively in academia, with two comprehensive literature reviews focusing on the existing research in this field (parkman, 2016; parkman & beard, 2008). many studies have documented the prevalence of impostor phenomenon among faculty, students, and staff, with some noting that the academic environment of “scholarly isolation, aggressive competitiveness, disciplinary nationalism, a lack of mentoring and the valuation of product over process” (mcdevitt, 2006, p. 1) may cultivate impostor feelings. race and gender, previously thought to be unrelated to the experience of impostor phenomenon, have recently shown associations with impostor scores in academic populations, including women graduate students (collett et al., 2013; oriel et al., 2004) and black undergraduate students (bernard et al., 2018; cokley et al., 2017). within academic libraries, two studies have shown that one in eight librarians have experienced impostor phenomenon, with younger and newer librarians demonstrating higher impostor scores than their older and more experienced colleagues (barr-walker et al., 2019; clark et al., 2014).   although countless studies have measured the prevalence of impostor phenomenon using two validated measurements (clance & imes, 1978; harvey & katz, 1985), few studies have examined the ways in which people with impostor phenomenon successfully cope with these feelings. two studies discovered a range of coping strategies used by faculty, including seeking support from colleagues, family, and friends; correcting cognitive distortions about the meaning of success and validating successes; and using avoidant behaviors like substance use, ignoring impostor feelings, and working harder (hutchins & rainbolt, 2017; rakestraw, 2017). mentor relationships were consistently reported as a successful strategy for addressing impostor phenomenon, with mentors encouraging participants to own their accomplishments, reassuring them about the normalcy of impostor feelings, modeling positive behaviors like avoiding unrealistic comparison with others, and providing emotional and practical support, including ideas and advice about their work (hutchins & rainbolt, 2017). importantly, while all strategies lessened impostor feelings, this lessening was reported as a temporary effect overall.    despite the lack of research on the use of coping strategies, most of the literature on impostor phenomenon recommends similar strategies at the individual level (e.g., recording accomplishments, self-evaluation, collaborating with colleagues), the managerial level (e.g., giving praise to direct reports, accepting mistakes), and the organizational level (e.g., creating mentoring programs) (de vries, 2005; parkman, 2016; rakestraw, 2017). within academia, programs for faculty, staff, and students have been developed to acknowledge and address impostor phenomenon, including awareness workshops at faculty orientations, mentoring and peer group programs, regular discussions for first-year employees, and the implementation of structured feedback systems (parkman, 2016). recommendations in the literature for supervisors include giving positive, documented feedback, facilitating a written record of accomplishments for employees, discussing clearly what success and excellence in a particular position might look like, modeling work-life balance while eschewing expectations of perfection, and being aware of the signs of impostor phenomenon in order to prevent and address it (de vries, 2005; rakestraw, 2017).   efforts to raise awareness about impostor phenomenon within academic librarianship have increased in the last five years, with american library association-sponsored webinars (conner-gaten, van ness, & tate-louie, 2018; puckett, 2018), the appearance of regular workshops at conferences like the new librarian symposium and the association of college and research libraries (acrl) conference, and a proliferation of published articles on the topic (agostino & cassidy,  2019; lacey & parlette-stewart, 2017; murphy, 2016; rakestraw, 2017; sobotka, 2014). despite this increased interest, no evidence yet exists to support the range of recommendations that are widely suggested to address impostor phenomenon.    aims   the aim of this study was to understand how health sciences librarians cope with impostor phenomenon. our research questions include the following: how are librarians experiencing impostor phenomenon at work? what types of coping strategies do librarians use to address these impostor feelings? how effective are current coping strategies at addressing impostor phenomenon? by answering these questions, we seek to provide evidence based solutions for addressing the experience of impostor phenomenon among librarians.    methods   we developed an online, anonymous survey using redcap, a secure, web-based application designed to support data capture for research studies. our survey contained the 14-statement harvey impostor phenomenon scale, seven demographic questions, and two open-ended questions: "do you use any strategies to address feelings of inadequacy at work?" and "if applicable, how effective are these strategies in addressing those feelings of inadequacy?" (see  appendix 1 referenced in data availability).    the development of our survey instrument has been described in detail elsewhere (barr-walker et al., 2019). briefly, the harvey impostor phenomenon scale, developed in 1981, is a validated tool widely used to measure impostor phenomenon (harvey & katz, 1985). the harvey scale contains 14 statements that respondents score on a scale of 1 to 7, representing "not at all true" to "very true"; some statements are reverse scored. for example, a statement like "in general, people tend to believe i am more competent than i really am" is scored as 1 for "not at all true" and 7 for "very true," while a statement like "i feel i deserve whatever honors, recognition, or praise i receive" is reverse scored. overall scores can range from 0 to 84, with higher scores corresponding to higher instance of impostor phenomenon. scores of 42 and higher "may indicate possible troubles due to impostor feelings, and scores in the upper range suggest significant anxiety" (harvey & katz, 1985).   univariate analysis and one-way anova with tukey’s post-hoc test were completed in stata, a statistical analysis software, by one author (author 1). for qualitative analysis, two authors (author 1 & author 2) used thematic analysis (silverman, 2003; wolcott, 2001) to create 14 codes from the responses to the open-ended survey questions using google sheets. participant responses were reviewed independently by both authors who then shared their ideas about emerging themes and came to consensus on codes; the two authors then independently assigned thematic codes to each response. inter-rater reliability was conducted after this step, with both authors checking each response and resolving any discrepancies between code assignments via deliberation until resolution. study data and corresponding codebooks are available in appendices 2 and 3 noted in data availability.   all 2,125 eligible members of the medical library association (excluding students, unemployed members, and members located outside the united states) were emailed an invitation to complete the online survey in october 2017, with two reminders sent over the next two months; the survey closed for responses on december 31, 2017. this study was approved as exempt research by the university of california, san francisco irb (study #17-22873).    results   respondents   of those surveyed, 703 respondents completed the survey (33% response rate), and 460 (65% of those who completed the survey) provided information for the two open-ended questions about strategies used to address feelings of inadequacy at work and the effectiveness of those strategies. quantitative analysis of the study results has been reported elsewhere (barr-walker et al., 2019); for the purposes of this paper, we will report only on the 460 respondents who completed the open-ended questions.    most of the 460 respondents identified as women (85%) and white (84%), worked in an academic library (58%), had an mls degree (98%), did not have educational training in a health sciences field (62%), and had 11 or more years of experience working in libraries (55%). age varied among respondents: the majority were between 36-50 (37%) or over 51 (44%), with a smaller percentage under 36 (19%). while total impostor phenomenon scores ranged from 5-70 (out of a possible 84), the average impostor score was 28.69, with 15% of respondents scoring 42 or above, indicating an experience of impostor phenomenon (harvey & katz, 1985). these results reflected the overall trends among the 703 participants of the survey (barr-walker et al., 2019), indicating that the 460 respondents examined in the current study were representative of this larger population.    strategy types   we identified 22 types of coping strategy themes, listed in table 1 (see end of the article). the most frequently reported strategy was education (n = 173), followed by support from colleagues (n = 133), reflection (n = 87), perseverance (n = 54), and mindfulness (n = 47). about half of respondents (55%) reported a single strategy, while the rest described multiple strategies: there were no significant differences in impostor scores at the p < 0.05 level between those who selected one or multiple strategies [f(1,458) = 1.92, p = 0.17]. the strategies that corresponded to the highest mean impostor phenomenon scores, indicating greater likelihood of impostor feelings), were avoidance (38.59), other external (37.13), recording praise (34.5), and fake it ’til i make it (34.1), with the lowest mean impostor phenomenon scores for support from colleagues (28.95), acknowledgement (27.19), and education (25.95) (figure 1).   we categorized each strategy as “internal” or “external” based on whether or not the respondent relied on another person or resource as part of the strategy. for example, internal strategies included reflection, mindfulness, and perseverance, all of which can be done by one individual without the assistance of another, while external strategies include education, mentorship, and support from colleagues. respondents were split between reporting strategies categorized as internal (n = 176), external (n = 172), or both (n = 111). many respondents included multiple internal or external strategies in their responses, but only those who included at least one strategy of each type were counted as having used both. using one-way anova with tukey’s post hoc test, significant differences at the p < 0.0001 level were observed in impostor scores between two groups, with those reporting only internal strategies having higher mean scores than those who reported only external strategies [f(2, 457) = 9.24, p = 0.0001]. although mean scores were lower for those that used both internal and external strategies than those using internal strategies only, this was not a significant effect (figure 2). there were no significant differences in the use of external versus internal strategies between demographic groups such as age, years of experience, race, gender, or type of library, with usage remaining consistent between groups.    figure 1 summary of the strategies with the highest and lowest impostor score means.      figure 2 comparison of impostor scores by type of strategy.   strategy effectiveness   most participants rated their strategies favorably, with 74% of participants reporting that their strategies were effective (n = 320, m = 26.81, sd = 10.20). using one-way anova and tukey’s post hoc test, we found that the mean impostor scores for participants who reported that their strategies were somewhat effective (n = 84, m = 38.25, sd = 9.59) or not effective (n = 21, m = 41.33, sd = 12.20) were significantly higher than those who reported effective strategies [f(3, 428) = 38.35, p = 0.0001], showing that self-reported effectiveness corresponded to lower impostor scores. the most frequently reported ineffective strategies were avoidance, perseverance, and reflection, all internal strategies. the most frequently reported effective strategies represented a mix of external and internal strategies: education, support from colleagues, reflection, mindfulness, and perseverance.   examples of strategies   reflection, an internal strategy, was an individual process focused on reviewing what one has accomplished in order to reach the current position in their career. examples focused on both general accomplishments, “i remind myself of what i’ve done in the past, and the things i’ve learned, and the fact that i can learn more,” and achievements related to their job roles:    “i have my school diplomas on my office wall -both my undergrad… and grad, which was a master's degree; the purpose isn't to impress or intimidate other people -they are there to remind *me* that i do belong legitimately in my office. sometimes i wonder if that's why other people have theirs on their own walls, too.”   education was a commonly referenced external coping strategy that involved engagement with resources beyond an individual’s own knowledge. one participant stated, “i try to participate in a lot of professional development, especially free professional development opportunities: moocs for example.” another participant gathered ideas for educational opportunities by “attending professional development sessions; reading colleagues’ resumes and linkedin accounts to learn about ways to improve my own.”   effectiveness of strategies   out of the 74% of participants that rated their strategies as effective, the most frequently reported were education and support from colleagues, external strategies that were associated with lower impostor scores. while responses around formal education like ce courses, advanced degrees, and professional conference participation were common, participants also found informal educational opportunities and support from colleagues to be effective:    “i take additional classes or read articles and books to improve areas that need work. these strategies help, but implementing them and seeing improvements helps improve feelings more than just completing a class or readings.”   “ask questions, seek help, go to experts, seek feedback. it broadens my knowledge and makes me more confident.”   receiving support from colleagues and mentors, inside and outside their own libraries, was often mentioned as an effective strategy:   “i usually find a colleague that is at the same stage or slightly further along than me to bounce ideas off of. i also try to reach out to mentors who may not necessarily be in my field, to compare my ideas with them. [this is] usually very successful, i think often i underestimate my thought process, and they often assure me that i am on the right path.”    “i have several trusted colleagues at my place of work, and several from previous employment that i discuss any uncertainties i am feeling to work through my impostor syndrome. [this is] highly effective. sometimes bouncing ideas off of another person is all i need, and occasionally reassurance that i'm not inept or that i am the right person for the job is necessary. mostly it just helps boost my confidence and strengthen my ideas.”   “i try to talk to colleagues in other medical libraries who can understand my feelings. [this is] very effective! my fellow librarians are so helpful and empathetic they make me feel that i am not alone."   although internal strategies alone were associated with higher impostor scores, some individuals reported their use of these strategies to be effective. many responses combined internal and external strategies, such as the following example which includes reflection, recording praise, and support from colleagues:   “i list all of the projects that i am currently working on, and all of the projects that i have completed, whether i did a great job or a not-so-good job. i sometimes also think about how i could be doing a worse job and imagine what that would look like. i then think about what i could be doing better and list small steps for improvement. talking to peers that you are close with also helps. you recognize that you are not alone and that you may be doing better than you thought. the list helped me to recognize the hard work that i've put in and does help me feel like i'm doing enough, or more than enough, in my position. imagining what doing a worse job would look like helps a great deal in addressing feelings of inadequacy. talking to peers helps significantly.”    discussion   individual coping strategies    the two measurements used in our study, impostor phenomenon scores and self-reported effectiveness, provide evidence of the association between external strategies, lower impostor scores, and greater self-reported effectiveness. however, variations exist among individual strategies: perseverance, for example, was one of the most frequently reported ineffective strategies but was associated with a lower impostor score than mentorship, a strategy often self-reported as effective. how can we explain this? looking closer at this particular strategy reveals that 72% of respondents that reported perseverance also used at least one additional strategy; because almost half of respondents used more than one strategy, it is difficult to separate the effects of any individual strategy from another. what seems clear from our aggregate data is the fact that using any strategy to deal with feelings of impostor phenomenon seems to be helpful, both in terms of self-reported effectiveness and impostor phenomenon score. within the choice of strategy type, our evidence points to the use of external strategies.    education was the most commonly reported external coping strategy, was self-reported as effective, and was associated with the lowest mean imposter score of all reported coping strategies. we did not distinguish between formal and informal education in our analysis; therefore, educational strategies could include anything from taking for-credit courses to reading articles. recommending educational strategies to combat impostor phenomenon, then, seems straightforward, and for those who are able to participate in educational activities, it is our highest recommendation to counter impostor feelings. however, we must also acknowledge the potential barriers to utilizing this strategy: lack of resources to pay for courses, webinars, or paywalled articles; uncertainty, particularly for newer librarians, about whether engaging in educational activities while at work is acceptable; and lack of time to engage in these activities. suggestions for organizations and leaders to address these barriers are discussed in the next section.   another self-reported effective external strategy was support from colleagues. this strategy may work well when one has already established a network of trusted colleagues, but in some workplace environments, this is not a feasible option. solo librarians, for example, must look for support outside of their own libraries where they may lose the shared experience that support from an institutional colleague often provides. librarians new to an institution might not know others well enough or be unsure when to ask for support. additionally, while many respondents described trusted colleagues, others described environments where they lacked support or encountered toxic colleagues. for librarians who are able to develop networks of trusted colleagues, the ability to share feelings around impostor phenomenon can help confirm that these feelings are shared by successful people; many respondents in our survey described not feeling so alone after discussions with colleagues about these issues. support from colleagues, therefore, can serve as an individual coping strategy and a way to raise awareness about impostor phenomenon within our field.    one self-reported effective internal coping strategy was acknowledgement: stating and accepting one’s lack of knowledge on a given topic. this strategy, while not reported nearly as frequently as education or support from colleagues, reflected the second lowest mean impostor score, following education. overall, internal strategies like mindfulness, fake it ’til i make it, and avoidance were associated with higher mean imposter scores than external strategies, but acknowledging a gap in knowledge is a necessary step before taking action, like seeking additional education or support from colleagues; in this context, its effectiveness makes sense.  some differences observed in internal and external scores and self-reported effectiveness may be explained by the fact that several internal strategies match impostor phenomenon indicators. for example, overpreparing, fake it ’til i make it, perseverance, and avoidance are coping strategies that also describe the characteristics of those with impostor feelings. it is not surprising that some of these internal strategies, including avoidance and perseverance, were self-reported as ineffective.    less obvious is why the strategy of reflection was also described as not effective and associated with higher impostor scores. one possible explanation is that self-reflection, if using the warped mirror of impostor phenomenon, can reinforce negative thoughts and perceptions. impostorism has been described as “an inability to accurately self-assess with regard to performance” (parkman, 2016). when reflecting on performance, those who experience impostor feelings will likely undervalue their strengths and achievements and overemphasize their mistakes and failures. reflection and recording praise (i.e., looking back at the things you have accomplished and praise you have received) are commonly recommended techniques to combat impostor phenomenon (de vries, 2005; hutchins & rainbolt, 2017; lacey & parlette-stewart, 2017). however, these internal strategies were associated with higher mean impostor scores in our study, indicating that if one is to use them, they should be combined with external strategies to increase their likelihood of effectiveness.    addressing impostor phenomenon through organizational culture change    beyond individual coping strategies, another method of addressing impostor phenomenon may come at the leadership level. several studies suggest a potential association between impostor feelings and job roles with a lack of clarity in their scope (lacey & parlette-stewart, 2017; parkman, 2016). in librarian positions where individuals are often responsible for a broad variety of tasks, performance targets can be vague and may lead to uncertainty about what success in one’s job looks like. as our findings have shown, support from colleagues and mentorship are associated with lower impostor scores: improving communication with librarians, including feedback on job performance, is a first step toward using these coping techniques. it is important to clarify, however, that not all feedback leads to decreased uncertainty. a recent study confirmed that supervisors do not always have an understanding of librarians' work; thus, feedback received in these cases can be frustrating (thomas et al., 2017). alternative models of feedback such as appreciative inquiry (rosener et al., 2019) and two-way feedback systems may help to provide a shared understanding of librarians' work and allow library leaders to change their expectations based on librarian feedback. leaders that prioritize clear, specific feedback as part of their regular conversations with employees can begin to create a culture of open communication in which impostor feelings can be acknowledged and addressed. leaders in our field have previously advocated for transparency in communication from leadership (robertson, 2017) in order to “create a safe environment for library workers … to talk with one another about their concerns and needs without fear of reprisal or rejection” (lew, 2017).    another opportunity for library leaders who want to create supportive environments for their staff to address impostor phenomenon is to reject and disrupt aspects of white supremacy culture in their organizations. white supremacy culture is the series of characteristics that institutionalize whiteness and westernness as normal and superior to other ethnic, racial, and regional identities and customs (gray, 2019). naming whiteness as a culture helps us question its neutrality and normativity, including how white culture shapes the norms, beliefs, and ideas of everyone in it (e.g., creating standards of professionalism for dress code and speech that privilege whiteness) (gray, 2019; hathcock, 2015). impostor phenomenon can thrive in this culture because its norms are often not named as such, and librarians whose work (and, often, personal selves) do not fit these norms may question their own success and ways of doing things.    many of the hallmarks of white supremacy culture can reflect the manifestations of impostor phenomenon in libraries, including perfectionism, a sense of urgency, individualism, either/or thinking, and quantity over quality, with several studies linking these two concepts (berg et al., 2018; dudău, 2014; henning et al., 1998; okun & jones, 2016; ross et al., 2001; thompson et al., 2000). although our institutions have historically been shaped by white supremacy culture, libraries can begin to dismantle these systems by proactively naming our norms and standards of behavior to reflect the type of culture that we want to see: one that does not facilitate impostor phenomenon. to do this, library leaders can recognize that projects often take longer than expected and create realistic work plans; create environments where it is expected that everyone will make mistakes and recognize that these mistakes sometimes lead to positive results; develop a values statement for the library which expresses the ways in which people want to do their work; evaluate people based on their ability to delegate to others; and/or make sure that everyone knows and understands their level of responsibility and authority in the organization (okun & jones, 2016). library leaders interested in continuing the anti-racist work of disrupting “the neutrality of whiteness” (gray, 2019) in their organizations can look to scholars in our profession who have written extensively on this topic (bourg, 2016; ettarh, 2014; schlesselman-tarango, 2017).    in the current study, over 10% of participants described a coping strategy related to perseverance or not giving up on a task even though you do not feel fully capable of completion. this feeling of perseverance or resilience is challenged in the library literature for obscuring structural issues and shifting responsibilities to library workers to overcome barriers for success (berg et al., 2018). using an example from our study, librarians may feel that obtaining education in order to relieve feelings of impostor phenomenon is their responsibility. framing education as a coping strategy that individual librarians must seek out ignores structural inequalities that prevent librarians in low-resource settings or with limited support from their library administrations from accessing these resources. according to this theory, resilience encourages library workers “to manage up, to ignore systemic inequalities, to return to a status quo which too often upholds silence over difficult change, and reinforces fictions of neutrality” (berg et al., 2018). library leaders can recognize the manifestations of resilience in their environments and begin to build organizational cultures that reframe resilience. resilience can be reimagined in libraries within the context of addressing the negative effects of impostor phenomenon: library leadership can create organizations whose values include letting go of unnecessary tasks, embracing discomfort during new training efforts, and helping staff accept “done” rather than perfection (berg et al., 2018).     moving forward: working together as a profession   professional organizations have a role to play in raising awareness about impostor phenomenon and supporting librarians with educational and mentorship opportunities. our study shows that education, support from colleagues, and mentorship are some of the most effective strategies that librarians can use to deal with impostor feelings. acrl, mla, and others can strengthen their existing mentorship programs, specifically targeting those who are younger or new to the profession, groups that displayed higher impostor scores in our study (barr-walker et al., 2019). professional organizations can also work together across disciplines (e.g., acrl, mla, pla, sla) to share expertise and connect members in different job roles for peer mentoring programs. local chapters may be able to play a role in creating a network of supportive colleagues and mentorship, but these chapters are often underfunded and understaffed by volunteer librarians. while our professional organizations currently offer regular educational opportunities including continuing education classes, webinars, and conferences, we must consider as a profession how fee-based education creates barriers for librarians in low-resource settings and how we can support our colleagues without financial resources to pay for existing educational opportunities. as the results of our study show, external strategies like education and mentorship are associated with lower impostor scores; these evidence based approaches should be valued when library leaders consider budgetary decisions around professional development for staff.    when advocating for mentorship within professional organizations, we must point out that formal mentorship programs often fail to address the impacts of white supremacy culture on librarianship, especially around how librarians of colour must navigate the whiteness of our profession (brown et al., 2018). when mentorship programs do not name, identify, or interrogate the whiteness of our institutions, they are unable to provide a supportive environment for participants of colour, and may facilitate feelings of impostor phenomenon, the very thing these programs are designed to disrupt (brown et al., 2018; dancy & brown, 2011). in addition to supporting and expanding existing diversity-centered mentorship programs, our organizations can create supportive environments for librarians of colour in all mentorship programs by acknowledging the harmful effects of white cultural norms and allowing participants to express their authentic selves (brown et al., 2018). great strides have been made to create informal and volunteer networks of peer mentors as a response to the lack of support for librarians of colour in formal mentoring spaces (brown et al., 2018); our professional organizations can recognize this as an opportunity to leverage these networks of experts to improve existing programs.    moving forward, future studies can build on our work by examining the differences in effectiveness between individual coping strategies and how the use of multiple strategies affects one's experience. additionally, there is a lack of research on how librarians' intersectional identities (e.g., race, gender, socioeconomic status) affect their feelings of inadequacy at work and how the coping strategies recommended in this study may be experienced differently based on these identities, rather than as a universal approach. although our study did not show differences in impostor scores by race or gender, the lack of diversity in librarianship combined with the ways in which dynamics of privilege are enacted in our field may inform interpretations of these results (barr-walker et al., 2019). future study in this area would allow us to better understand the associations between impostor phenomenon, how white supremacy culture is enacted in libraries, and how these intersectional identities are experienced.    conclusion   in our census of members of the medical library association through an online survey, 15% of librarians experienced impostor phenomenon, and most reported using one or more coping strategies to address these feelings. external strategies like education, support from colleagues, and mentorship were associated with self-reported effectiveness and lower impostor scores. although our findings showed less evidence for the use of commonly recommended strategies such as reflection, mindfulness, and recording praise, it appears that using any strategy at all is more effective than using none. we encourage librarians and library leaders to develop and utilize evidence based recommendations to address impostor phenomenon, with careful consideration given to structural barriers, such as the resilience narrative and white supremacy culture, within our field.    acknowledgements   we thank the institute of research design in librarianship (irdl), notably marie kennedy and lili luo, for their support throughout the research process, particularly in the creation and development of this project. special thanks to our colleagues who provided feedback on drafts of the manuscript: nisha mody, charlie macquarie, and eamon tewell, irdl mentor, whose meaningful support throughout this project was unwavering and essential to its completion.    table 1 qualitative themes for coping strategies identified by respondents strategy definition external or internal example number of responses mean impostor score education taking a class, attending a conference, reading resources on a given topic, working towards a degree external “take continuing education classes, attend conferences, work on certifications.” 173 25.95 support from colleagues talking with colleagues about tasks and/or seeking their advice external “i think it is important to be able to communicate with my supervisor and peers. right now i have a very approachable boss who i trust and can share my feelings with.” 133 28.95 mentorship mention of a mentor or someone who is in a formal coaching or guidance position external “i have a wonderful mentor whom i ask for advice. she is an excellent librarian, and a strong supporter of my work an[d] my morale." 17 30.06 support from friends/family talking with friends/family about tasks and/or seeking their advice external “talk to family, talk to friends. talk to myself as if i were one of my friends.”   19 31.16 other—external (includes therapy) combination of external strategy categories that had <15 responses external “i see a mental health professional for career burnout and anxiety.” 8 37.13 acknowledgement stating and accepting one’s lack of knowledge on a given topic internal “i find it ok to say ‘i don't know’ or ‘i don't know how’ but add ‘but i will find out.’” 31 27.19 personal best doing one’s best on a given task internal “i mainly do my best to work hard and do a good job no matter how small or large the task. the thought that i am doing all i can helps counteract feelings of inadequacy.” 26 28.96 perseverance not giving up on a task even though you don’t feel fully capable of completion internal “i don't have a choice, but to keep trying.” 54 29.69 other— internal (age, comparison, music, organization, personal activities, physical activities, religion, and substance use) combination of internal strategy categories that had <15 responses internal “ride my bike after work.” 47 31.04 mindfulness mentioning of meditation or mindfulness techniques internal “i start each day having a pep talk with myself. i see myself being successful and happy. and most of the time it comes true.” 47 31.45 reflection thinking about the work one has done internal “i tell myself you have accomplished these efforts before and you can do it again. i take a deep breath and go for it.” 87 31.56 over-preparing doing as much as possible before completing a task internal “prepare for meetings etc. more thoroughly than other people.” 21 32.39 fake it ’til i make it working through the skill as you learn how to do it internal “just keep going. ‘fake it until you make it.’” 19 34.10 recording praise creating/saving a physical or electronic copy of a verbal or written compliment internal “i keep a file of notes of praise or thanks that i have received from a job well done, and refer to them if i doubt myself.” 22 34.5 avoidance not doing a given task internal “i just ignore them.” 27 38.59   references   agostino, c., & cassidy, m. (2019). failure to launch: feelings of failure in early career librarians. partnership: the canadian journal of library and information practice and research, 14(1). https://doi.org/10.21083/partnership.v14i1.5224   barr-walker, j., bass, m. b., werner, d. a., & kellermeyer, l. (2019). measuring impostor phenomenon among health sciences librarians. journal of the medical library association: jmla, 107(3), 323. https://doi.org/10.5195/jmla.2019.644   berg, j., galvan, a., & tewell, e. (2018). responding to and reimagining resilience in academic libraries. journal of new librarianship, 3(1). https://doi.org/10.21173/newlibs/4/1   bernard, d. l., hoggard, l. s., & neblett, e. w., jr. (2018). racial discrimination, racial identity, and impostor phenomenon: a profile approach. cultural diversity and ethnic minority psychology, 24(1), 51. https://doi.org/10.1037/cdp0000161   bourg, c. (2016, april 18). diversity, inclusion, social justice and libraries: proposing a framework. feral librarian. https://chrisbourg.wordpress.com/2016/04/16/diversity-inclusion-social-justice-and-libraries-proposing-a-framework/     brown, j., ferretti, j. a., leung, s., & méndez-brady, m. (2018). we here: speaking our truth. library trends, 67(1), 163–181. https://doi.org/10.1353/lib.2018.0031   clance, p. r., & imes, s. a. (1978). the imposter phenomenon in high achieving women: dynamics and therapeutic intervention. psychotherapy: theory, research & practice, 15(3), 241. https://doi.org/10.1037/h0086006 clark, m., vardeman, k. k., & barba, s. e. (2014). perceived inadequacy: a study of the imposter phenomenon among college and research librarians. college & research libraries, 75(3), 255–271. https://doi.org/10.5860/crl12-423   cokley, k., smith, l., bernard, d., hurst, a., jackson, s., stone, s., awosogba, o., saucer, c., bailey, m., & roberts, d. (2017). impostor feelings as a moderator and mediator of the relationship between perceived discrimination and mental health among racial/ethnic minority college students. journal of counseling psychology, 64(2), 141–154. https://doi.org/10.1037/cou0000198     collett, j. l., avelis, j., & lizardo, o. (2013, august 11). family-friendliness, fraudulence, and gendered academic career ambitions [conference paper]. american sociological association annual meeting, new york, ny.   conner-gaten, a., van ness, s., & tate-louie, k. (2018, june 13). battling impostor syndrome in the workplace webinar [webinar]. library leadership & management association, american library association (ala).   dancy, t. e., & brown, m. c. (2011). the mentoring and induction of educators of color: addressing the impostor syndrome in academe. journal of school leadership, 21(4), 607–634.   de vries, m. f. r. k. (2005). the dangers of feeling like a fake. harvard business review, 83(9), 108.   dudău, d. p. (2014). the relation between perfectionism and impostor phenomenon. procedia-social and behavioral sciences, 127, 129–133. https://doi.org/10.1016/j.sbspro.2014.03.226 ettarh, f. (2014, july). making a new table: intersectional librarianship. in the library with the lead pipe. http://www.inthelibrarywiththeleadpipe.org/2014/making-a-new-table-intersectional-librarianship-3/   gray, a. (2019, june 4). the bias of ‘professionalism’ standards. stanford social innovation review. https://ssir.org/articles/entry/the_bias_of_professionalism_standards     harvey, j. c., & katz, c. (1985). if i'm so successful, why do i feel like a fake?: the impostor phenomenon. st. martin's press.   hathcock, a. (2015, october). white librarianship in blackface: diversity initiatives in lis. in the library with the lead pipe. http://www.inthelibrarywiththeleadpipe.org/2015/lis-diversity/   henning, k., ey, s., & shaw, d. (1998). perfectionism, the impostor phenomenon and psychological adjustment in medical, dental, nursing and pharmacy students. medical education, 32(5), 456–464. https://doi.org/10.1046/j.1365-2923.1998.00234.x   hutchins, h. m., & rainbolt, h. (2017). what triggers imposter phenomenon among academic faculty? a critical incident study exploring antecedents, coping, and development opportunities. human resource development international, 20(3), 194–214. https://doi.org/10.1080/13678868.2016.1248205   lacey, s., & parlette-stewart, m. (2017). jumping into the deep: imposter syndrome, defining success and the new librarian. partnership: the canadian journal of library and information practice and research, 12(1). https://doi.org/10.21083/partnership.v12i1.3979   lew, s. (2017). creating a path to feminist leadership. in s. lew & y. baharark (eds.), feminists among us: resistance and advocacy in library leadership (pp.29–41). library juice press.   mcdevitt, n. (2006). unmasking the imposter phenomenon: fear of failure paralyzes students and faculty. the mcgill reporter, 38(17), 1-2.   murphy, b. (2016, march 28). imposter syndrome as a student. hack library school. https://hacklibraryschool.com/2016/03/28/imposter-syndrome-as-a-student/   okun, t., & jones, k. (2016). dismantling racism: a workbook for social change groups. drworks. https://resourcegeneration.org/wp-content/uploads/2018/01/2016-drworks-workbook.pdf     oriel, k., plane, m. b., & mundt, m. (2004). family medicine residents and the impostor phenomenon. family medicine, 36(4), 248–252.   parkman, a., & beard, r. (2008). succession planning and the imposter phenomenon in higher education. cupa-hr journal, 59(2), 29–36.   parkman, a. (2016). the imposter phenomenon in higher education: incidence and impact. journal of higher education theory and practice, 16(1).   puckett, j. (2018, september 19). impostor syndrome in instruction librarians: impact and solutions webinar [webinar]. association of college & research libraries (acrl).   rakestraw, l. (2017). how to stop feeling like a phony in your library: recognizing the causes of the imposter syndrome, and how to put a stop to the cycle. law library journal, 109(3), 465–477.   robertson, t. (2017). a feminist among us: an interview with chris bourg. in s. lew & y. baharark (eds.), feminists among us: resistance and advocacy in library leadership (pp.172–188). library juice press.   rosener, a., frigo, e., ponischil, s., bélanger, a., rander, j., & salazar, e. (2019, september). leading from the center: reimagining feedback conversations at an academic library. in the library with the lead pipe. http://www.inthelibrarywiththeleadpipe.org/2019/reimagining-feedback/   ross, s. r., stewart, j., mugge, m., & fultz, b. (2001). the imposter phenomenon, achievement dispositions, and the five factor model. personality and individual differences, 31(8), 1347–1355. https://doi.org/10.1016/s0191-8869(00)00228-2   schlesselman-tarango, g. (2017). topographies of whiteness: mapping whiteness in library and information science. library juice press.   silverman, d. (2003). analyzing talk and text. in n. k. denzin & y. s. lincoln (eds.), collecting and interpreting qualitative materials (pp.340–362). sage publishers.   sobotka. c. (2014, may 8). dealing with imposter syndrome and feeling like you belong. inalj blog. http://inalj.com/?p=70926   thomas, c., trucks, e., & kouns, h. b. (2017, april). preparing early career librarians for leadership and management: a feminist critique. in the library with the lead pipe. http://www.inthelibrarywiththeleadpipe.org/2019/early-career-leadership-and-management/   thompson, t., foreman, p., & martin, f. (2000). impostor fears and perfectionistic concern over mistakes. personality and individual differences, 29(4), 629–647. https://doi.org/10.1016/s0191-8869(99)00218-4   wolcott, h. (2001). description, analysis, and interpretation in qualitative inquiry. in c.f. conrad, j. g. haworth, & l. r. lattuca (eds.), qualitative research in higher education: expanding perspectives (pp.573–598). pearson custom publishing.       evidence based library and information practice evidence based library and information practice 2009, 4.4  evidence based library and information practice       using evidence in practice    reorganizing a technical services division using collaborative evidence based  information practice at auraria library    denise pan  associate director of technical services and assistant professor  university of colorado denver  denver, colorado, united states of america  email: denise.pan@ucdenver.edu     zaana howard  project executive, knowledge exchange  cpa australia  melbourne, victoria, australia  email: zaanahoward@gmail.com    received: 05 august 2009        accepted: 12 november 2009      © 2009 pan and howard. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution,  and reproduction in any medium, provided the original work is properly cited.    setting    as a tri‐institutional academic library, auraria  library is administered by the university of  colorado denver and also serves metropolitan  state college of denver and the community  college of denver. committed to providing an  environment which stimulates and nourishes  the growth of ideas, auraria library offers  collections and services that support the  curriculum of the three institutions, ranging  from vocational training to doctoral programs.   situated in the heart of the city’s commercial,  cultural and recreational district, the campus  attracts a diverse population consisting of  nearly 50,000 students. in general, students of  auraria campus tend to be mature and  distant learners who commute to campus,  work part‐time, and rely on accessing online  resources to meet their research needs.    in response to local user needs and in  alignment with international collection  development trends, the auraria library in  denver, colorado shifted its focus from print  to electronic formats. this change forced the  technical services division to reorganize and  revise its workflow. economic conditions  prohibited expanding or replacing existing  staff members.  instead, collaborative  evidence based information practice (ebip)  was implemented to guide problem  identification, data collection, evidence  interpretation, and knowledge creation  changes (somerville et al. 141‐61). this paper  focuses on the evidence‐creating aspects of the  ‘knowledge enabling’ (von krogh et al. 4)  elements which facilitate organizational  members’ synthesis and analysis of collective  experience. based on swedish philosophy,  fortified by australia ideas, and implemented  88 mailto:denise.pan@ucdenver.edu mailto:zaanahoward@gmail.com http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2009, 4.4  in north america (somerville, mirijamdotter,  and howard 119‐26), the collaborative ebip  approach invites and enables employees to  identify problems and create solutions. the  authors offer a practical and replicable model  for other organizations to repurpose,  reorganize, and retrain their human resources  (callahan and howard 85‐90).      problem    in the increasingly complex area of electronic  resources management, there is a great deal of  confusion amongst players ‐ library  administrators, librarians, publishers, content  providers, and subscription agents to name a  few. this turmoil is evidence by the  proliferation of various national information  standards organization (niso) initiatives,  including sushi, onix, counter (pesch  483), core (aipperspach et al.), seru  (tenopir 26), kbart (mccracken and arthur  231). each standard represents a new function  or responsibility within technical services, not  previously present in the print world. as a  result, libraries globally are struggling to  manage workflow issues.  the literature is  well documented with this fact, but identified  solutions primarily involve purchasing  electronic resource management systems  (erms) (harvell 127‐31; meyer 19; sadeh and  ellingsen 208‐13; pan).     in order to respond to new electronic  resources management demands for  acquiring, accessing and discovering, libraries  must develop viable workflow efficiencies  where highly complex and dispersed  processes and procedures are clearly defined.  this represents a marked change from  traditional print‐oriented technical services  departments, where processes and personnel  could be isolated from each other. in this  environment, colleagues communicate by  manoeuvring carts of physical materials from  one department to another and thereby  initiating the next processing activity  primarily through visual cue.    in the new information world, however,  electronic resource management  responsibilities are distributed among several  individuals in a variety of roles ranging from  collection development, acquisitions,  cataloguing, and library management systems  administration. when acquisitions order a  new e‐journal, cataloguing personnel are  unaware unless that information is  communicated. given the large volumes of  transactions ‐ as of april 2009, auraria library  subscribes to over 31,000 e‐journals via 277  databases ‐ dependence on face‐to‐face  conversations or email communications is no  longer sustainable.      the literature on reorganization of technical  services reveals case studies but no formal  quantitative or qualitative research. the  library purchased an erms, as the literature  suggested, however workflow issues were not  resolved. consequently, the library turned to  established experts in workflow analysis, r2  consulting, who advised the organization to  “recognize e‐resources as the library’s  mainstream” and “expand e‐resources staff in  both number and level” (lugg and barnes 22).  given that the library was unable to budget  for new positions, the library had to  repurpose, retool, and reorganize existing staff  members in order to satisfy r2’s  recommendations.     evidence    as previously mentioned, the electronic  resources management literature did not  provide advice or solutions sufficient to  address the question of workflow issues.  evidence was gathered through the collective  experience and knowledge of the  organizational members, which was  synthesized and analyzed within the context  of local environment requirements.  however,  the technical services division was not capable  of developing solutions until their workplace  practices transitioned from isolation to  collaboration. wisdom and understanding  was enabled through fostering a culture of  conversation, learning, and contribution based  upon collaborative ebip. shared leadership,  appreciative inquiry, and knowledge creation  provide the theoretical framework for auraria  89 evidence based library and information practice 2009, 4.4  library to reconcile challenges within the  technical services division.    the shared leadership approach (deiss and  sullivan 6) moved decision making beyond  organizational hierarchies to distribute  influence and authority throughout the  workplace, in recognition that organizational  success relies upon individuals, teams,  departments, and divisions working  collaboratively, cross‐functionally and across  hierarchies. conversations about the ‘best of  what is’ currently and ‘what could be’ in the  future (sullivan 220; whitney et al. 34)  provided a foundation for organizational  revitalization.     over time and with practice, the electronic  resources team was transformed into an  intentional learning environment. these  ‘communities of practice’ (wenger 2; nonaka  et al.) are an integral part of the fabric of life,  identifiable everywhere – at home, school,  work, and play, whereby membership is self‐ defined by aligning with community mission,  actions and results. developing communities  are essential for maintaining shared  understandings for knowledge exchange and  interpretation, possessing institutional  memory, encouraging forward thinking, and  developing group identity. in a workplace,  this is ideally translated into a shared space for  creating knowledge and developing collegial  relationships where the collective may  produce a “transcendental  perspective…which integrates information  into knowledge, within a context that harbors  meaning” (somerville, mirijamdotter and  collins 4).      through implementing these concepts in a  workplace learning environment, the  organization and its employees can grow  together by developing knowledge through  meaningful encounters that activate prior  understanding within individuals and among  groups (howard and somerville 4). solving  immediate problems is only the starting point.  by defining and resolving challenges, ideas  are generated from existing information, and  can further formulate new knowledge;  employees ‘learn‐by‐doing’ with one another.  in other words, “in organizational knowledge  creation, one plus one could be more than two.  it can be also zero, if interactions among  individuals work negatively” (nonaka 3).  these various theories conceptualize an  organization as a purposeful social interaction  system in which collective capabilities develop  through workplace socialization processes  (howard and somerville 4).     due to the longevity of service that many staff  members had invested at auraria library,  personnel were largely conditioned to work in  hierarchical structure and discouraged from  innovation or initiative without prior  permission. they were victims of entrenched  hierarchies, institutional memory, and long  standing processes that had not evolved at the  same pace as industry. applying shared  leadership, appreciative inquiry and  knowledge creation theories produced a new  workplace culture for technical services  librarians and paraprofessionals. they were  empowered with a voice, recognized for their  service and wealth of experience, and  encouraged to create their own evidence.      through this process of discovery, personnel  were able to reframe their histories and  renegotiate their roles. conversations emerged  throughout the appreciative inquiry which  enabled experiences to be collected, shared,  discussed and new knowledge created for the  application of new thinking to the local  environment. this reinforced the wisdom of  the notion that “one plus one, could be more  than two” (nonaka 3). the emergent  workplace model provides evidence that  employees can cooperatively generate a  positive and productive workplace  environment, when enabled to work  autonomously and appreciate the contribution  of their colleagues. in turn, the technical  services division continues to develop their  own solutions within an ever‐changing digital  landscape.    implementation    to begin the transformation within the   90 evidence based library and information practice 2009, 4.4  auraria library technical services department,  new collaboration processes were established  by reorganizing and redefining workflows.   restructuring was necessary because of the  need to alter established processes due to new  technologies and formats, and limited budget  for materials and staff.  team members were  encouraged to work cross‐functionally,  establish backup training, and increase  communication within the department and the  library. appreciative inquiry sessions were  conducted individually with each team  member to establish a framework for shared  understanding (sullivan 225) through the  introduction of the learning resource life  cycle (see fig. 1).                            focusing on this cycle enabled the  development of a strategic dialogue about  team member technical services roles and  responsibilities, rather than concentrating on  operational matters such as reporting  structures and organizational charts. after  being appreciatively ‘heard and understood,’  staff members willingly participated in  conversations about their capabilities and  aspirations.      in redesigning the work flow processes in  technical services, four factors were  considered for redefining roles: library needs,  department needs, personal interests, and  individual capabilities. concurrently, similar           fig. 1. learning resource life cycle.                  91 evidence based library and information practice 2009, 4.4  reorganization processes were occurring  across auraria library. as a result of this  comprehensive appreciative inquiry process,  three library personnel transferred to the  technical services department and two  individuals reassigned to other departments.  library staff members were reorganized to  stimulate collaboration and thereby establish  communities of practice and create  knowledge, in order to realize the benefit of  organizational learning beyond individual  pursuits.    outcome    once staff members’ assignments were  finalized, the department focused on  implementing the library’s strategic plan.  technical services staff members explicitly  linked the departmental annual plan and  personal performance plans to strategic plan  initiatives which underscored the importance  of cross functional electronic resource  management. by doing so, they ensured goals  and objectives that would become tangible  outcomes of individual and group efforts.     recognizing and appreciating what each  person “brings to the table” celebrates  individuals’ skills, expertise, knowledge and  contribution, regardless of title or rank.  through emphasizing the value of employees’  skills, expertise and capabilities, commencing  with the appreciative inquiry process,  employee professional confidence was  developed. reinforced through a collaborative  ebip framework, this confidence enabled staff  members to engage at a higher level within the  workplace. this resulted in greater staff  member cooperation and communication,  with questions being posed and addressed at  all team levels for the growth and benefit of  the organization. this community  development and collaborative process of  knowledge creation has resulted in a culture  change within the department where  problems are actively solved and decisions  made through evidence generated collectively.    with electronic resources, it is essential that  co‐workers cooperate and communicate. as a  result of collaborative ebip, technical services  personnel have a heightened sense of  camaraderie, which encourages teamwork,  creative synergies, and knowledge creation.  information exchange and shared leadership  is proactively expressed at the peer level  rather than being reliant on knowledge and  permission from supervisors. amidst  dynamically changing circumstances, staff  members have gained the capability to ask  questions, produce workflows and assign  responsibilities. their efforts provide  compelling evidence that harnessing collective  experience to produce new knowledge can  satisfy contemporary organizational needs for  agile responsiveness and perpetual learning.     reflection    according to ebip, challenges within  librarianship can be met by identifying and  applying the best available evidence (eldredge  291). this paper is firmly grounded in ebip,  but places greater emphasis on two defining  characteristics: best available evidence and  incorporation of the user perspective (booth  and brice 54). with collaborative ebip,  “evidence based processes are driven not by  librarians but rather by stakeholders who  assume responsibility for problem definition,  methodological implementation, and data  analysis activities. reliance on student‐framed,  student‐conducted, and student‐reported  research results serves to shift the locus of  decision making control from ‘library centric’  to ‘user centric’” (somerville et al. 144). in the  auraria library case study, best evidence is  found in organizational management theories  and the user perspective present in library  staff. similarly, the decision‐making authority  shifts from ‘employer centric’ to ‘employee  centric.’     auraria library’s technical services  department created a collaborative ebip  environment by flattening workplace  hierarchies, distributing problem solving and  encouraging reflective dialogue. embracing  the collective knowledge and experiences of  technical services staff members enables them  to be valued and respected leaders and  92 evidence based library and information practice 2009, 4.4  followers. in doing so, participants are  encouraged and empowered to identify  problems and create solutions amidst a  dynamically changing electronic resources  environment.      in redefining the question for the future, there  are several imminent challenges ahead  including a significant number of retirements  and continuing issues in resource  management within a growing digital  environment. through these personnel and  resource changes, maintaining the  collaborative ebip framework will be tested to  see if the model remains sufficiently robust to  evolve despite increasingly harsh demands ‐  including the organization’s inability to  replace staff during this economic downturn.      although this case study occurred within a  north american academic institution, the  principles of flattening workplace hierarchies,  distributing problem solving and enabling  reflective dialogue can be applied to any  library, information, or knowledge  organization. the auraria library’s  experiences to date suggest that through  embracing the collective knowledge and  experiences of staff members and empowering  them to be leaders and followers, individually  valued and respected, collaborative ebip can  be initiated and maintained.       works cited    aipperspach, jeff, tina feick, and john   mcdonald. “standards in the e‐ resource world: counter, core  and i2.” electronic resources &  libraries conference. university of  california, los angeles, ca. 9‐12 feb.  2009.     booth, andrew, and anne brice. evidence   based practice for information  professionals: a handbook. london:  facet, 2004.        callahan, anne, and zaana howard. planning   for success: reprioritising,  repurposing and retooling with  results. proceedings of 5th  international lifelong learning  conference: reflecting on successes  and framing futures. ed. deborah  orr et al. rockhampton, queensland:  central queensland up, 2008. 85–90.  web. 23 nov. 2009.  <http://hdl.cqu.edu.au/10018/14315>.     deiss, kathryn j., and maureen sullivan. “the   shared leadership principle: creating  leaders throughout the  organization.” issues and trends in  diversity, leadership and career  development 2 (1998): 2‐6.    eldredge, jonathan d. “evidence based   librarianship: an overview.” bulletin  of the medical library association  88.4 (1997): 289‐302.    harvell, tony a. “electronic resources   management systems: the experience  of beta testing and implementation.”  the serials librarian 47.4 (2005): 125‐ 36.    howard, zaana. and mary m. somerville.   “building knowledge capabilities: an  organisational learning approach.”  11th annual australian conference on  knowledge management and  intelligent decision support.  university of ballarat, ballarat,  victoria, australia. 8‐10 dec. 2008.    lugg, rick. and barnes, matt. auraria library:   library workflows and  organisational structure:  observations and recommendations.  r2 consulting, llc, 2008.     mccracken, peter. and michael a. arthur.   “kbart: best practices in  knowledgebase data transfer.” the  serials librarian 56.1‐4 (2009): 230‐5.      93 evidence based library and information practice 2009, 4.4  94 meyer, stephen. “the helping you buy series:   electronic resource management  systems.” computers in libraries  25.10 (2005): 19‐24.    nonaka, ikujiro, ryoko toyama, and akiya   nagata. “a firm as a knowledge‐ creating entity: a new perspective  on the theory of the firm.” industrial  and corporate change 9.1 (2000): 1‐20.    pan, denise. “leaping before looking:   implementing an electronic resource  management system in three days.”  electronic resources & libraries  conference. university of california,  los angeles, ca. 9‐12 feb. 2009.      pesch, oliver. “library standards and   e‐resource management: a survey of  current initiatives and standards  efforts.” the serials librarian 55.3  (2008): 481‐86.     sadeh, tamar. and ellingsen, mark.   “electronic resource management  systems: the need and the  realization.” new library world  106.1212‐1213 (2005): 208‐18.    somerville, mary m. working together:   collaborative information practices  for organizational learning. chicago:  association of college and research  libraries, 2009.    somerville, mary. m., anita mirijamdotter,   and lydia collins. “systems thinking  and information literacy: elements of  a knowledge enabling workplace  environment.” 39th annual hawaii  international conference on systems  sciences (hicss‐39). los alamitos,  ca: ieee computer society, 2006.  web. 21 jan. 2008.  <http://csdl2.computer.org/comp/proc eedings/hicss/2006/2507/07/250770150. pdf>      somerville, mary m., anita mirijamdotter, and   zaana howard. “workplace  information literacy: cultivation  strategies for ‘working smarter’ in  21st century libraries.” pushing the  edge: explore, engage, extend: 14th  association of college & research  libraries national conference. ed.  dawn m. mueller. chicago:  association of college and research  libraries, 2009. 119‐26.    somerville, mary m. et al. “collaborative   evidence‐based information practice:  the cal poly digital learning  initiative.” evidence‐based  librarianship: case studies and  active learning exercises. ed.  elizabeth connor. oxford: chandos,  2007. 141‐61.    sullivan, maureen. “the promise of   appreciative inquiry in library  organisations.” library trends 53.1  (2004): 218‐29.    tenopir, carol. “seru: a licensing   advance.” library journal 133.10  (2008), 26.    von krogh, george, kazuo ichijo, and ikujiro   nonaka. enabling knowledge  creation—how to unlock the  mystery of tacit knowledge and  release the power of innovation. new  york: oxford up, 2000.    wenger, etienne. “communities of practice:   learning as a social system.” the  systems thinker 9.5 (1998). web. 5  aug. 2009. <http://www.co‐i‐ l.com/coil/knowledgegarden/cop/lss.s html>.    whitney, diana, amanda trosten‐bloom, and   david cooperrider. the power of  appreciative inquiry: a practical  guide to positive change. san  francisco: berrett‐koehler, 2003.    evidence based library and information practice evidence based library and information practice 2009, 4.4  evidence based library and information practice       evidence summary    a list of best practices that may improve the use of telephone interviews during the  recruitment process    a review of:  engel, debra, and sarah robbins. ʺtelephone interviewing practices within academic libraries.ʺ  journal of academic librarianship 35.2 (2009): 143‐51.    reviewed by:  david herron  scholarly developer  karolinska institutet university library  stockholm, sweden  email: david.herron@ki.se    lotta haglund  head of information and public relations  karolinska institutet university library  stockholm, sweden  email: lotta.haglund@ki.se    received: 08 september 2009        accepted: 06 november 2009      © 2009 herron and haglund. this is an open access article distributed under the terms of the creative  commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    objective – to investigate the use of  telephone interviews in academic libraries and  identify best practices when conducting  telephone interviews.    design – survey and open‐ended questions.    setting – academic libraries in the united  states.     subjects – academic institutional members of  the association of research libraries.    methods – a fifteen‐item survey (appendix  a, 150) concerning telephone interviewing  practice was sent to 112 institutional members  of the association of research libraries (arl).  the survey contained multiple choice‐type  questions as well as open‐ended questions.    main results – the response rate was 66% (74  of 112 research libraries; 56 public institutions  and 17 private). of the respondents, 90% used  telephone interviews to screen applicants for  professional positions (ranging from  occasionally to always) and only 10% never  used telephone interviews. the main reason  for holding telephone interviews was to  “screen candidates in order to narrow the  pool of applicants who will be invited for in‐ 81 mailto:david.herron@ki.se mailto:lotta.haglund@ki.se evidence based library and information practice 2009, 4.4  person interviews” (146). other reasons given  included minimizing expenses involved in  interviewing out‐of‐town candidates (39%  respondents), shortening the length of time to  complete the search process (27%) and  meeting library or campus hiring  requirements (3%).    on average, the majority of libraries (51%)  hired between 2‐4 professional positions each  year. for each open professional position, the  number of candidates telephone‐interviewed  varied from less than 3 to 9 depending on  library and position.    interviews typically lasted between 16 and 45  minutes (77% respondents) with all the search  committee members (staff involved in the  recruitment process) being present (75%) and  taking turns to ask questions to the candidates  (90%). questions were most often the same for  all candidates applying for a particular  position (91%) and candidates were nearly  always allowed in return to ask questions of  the committee (96%).    in answer to the open‐ended question, “in  your opinion, what best creates a collegial and  effective environment for conducting  telephone interviews?”, the dominating  responses included: all search committee  members being present and participating  actively, using a script and taking notes,  introducing all interview participants, and  giving candidates adequate notification and  documentation.    the majority of respondents “would not  change anything about their institution’s  current (telephone interview) practice” (147).  some thought that “training and/or a need for  consistency in procedure for all interviews  conducted” (147) would be relevant changes  in practice.     the results of the survey were compared to  the findings in the library, personnel  management and human resources literature.    conclusion – from the survey and responses  to open‐ended questions, a number of best  practices when conducting telephone  interviews emerged (148–150):  1. properly train the search committee  (e.g., in knowing about the position,  organization and protocols for  conducting interviews properly)  2. involve the search committee  throughout the recruitment process  3. help the interviewee be prepared  (e.g., by sending institutional  information packages)  4. maintain and use suitable technology  (e.g., by choosing suitable  conferencing facilities with the  interviewee situation in mind)  5. put the candidate at ease (e.g., by  explaining who will be present, how  long the interview will last and how  many questions there will be)  6. provide introductions (e.g., both in the  beginning of the interview and even  before individual questions)  7. listen and take notes during the  interview  8. discuss the interviews immediately  afterwards    engel and robbins suggest that further  research could include looking into job  candidate and search committee experiences  of the telephone interviewing procedure.      commentary     the paper provides a straight‐forward survey  analysis concerning the use of telephone  interviews during the recruitment process at  research libraries in the usa. from the  analysis, a list of best practices emerges which  may useful for staff involved in staff hiring  and lead to improved practice.    generally, the article is well written and  informative although the many statistical  analyses of the results make the survey  findings (146‐7) section somewhat difficult to  follow. a more general overview of the results  would have been useful.    82 evidence based library and information practice 2009, 4.4  83 the authors of this evidence summary used  lindsay glynn’s critical appraisal checklist  to estimate the face validity to be over 80%;  therefore the research seems robust. however,  it is unstated in which year the survey was  carried out so it is hard to judge the actuality  of the research.    the overall response rate of the survey was  66% (74 of 112 recipients) which seems  reasonable. however, there is a slight  mismatch in the figures later on “fifty‐ six…indicated they were from public institutes  while the remaining seventeen…were from  private institutions (i.e. a total of 73)” (146).    one possible source of bias could be that  smaller libraries (<60 professional positions)  are over‐represented in the survey (table 1,  146). for example, do smaller libraries tend to  use telephone interviews more often? if so  why? engel and robbins make no comment  on this.    about 10% of the respondents never used  telephone interviews. however, this is  unlikely to have biased the results of the  survey, since these libraries clearly would be  unable to respond to the majority of the  questions.    engel and robbins cite an extensive literature  concerning staff recruitment, including  guidelines, manuals, and contributions from  individual libraries. this research provides a  helicopter perspective on telephone  interviewing practice in us academic libraries  which is interesting. however, it remains  unclear to the authors of this evidence  summary exactly what new information has  emerged from this effort.    the article would have been even more  interesting if the recruitment process had been  placed in the wider context of strategic library  development, i.e. which kinds of competencies  do we need to recruit for the future library.  however, this lies outside the objectives of the  research study.    the generic list of best practices suggested by  engels and robbins seems widely applicable  to all library types. therefore, the content of  the article is a good example of bridging the  gap between research and practice.      works cited    glynn, lindsay. a critical appraisal tool for  library and information research.  library hi tech 24.3 (2006): 387‐ 99.  evidence based library and information practice evidence based library and information practice 2010, 5.1  evidence based library and information practice       commentary    toward collaborative evidence based information practices: organisation and  leadership essentials    anita mirijamdotter  professor of informatics  linnaeus university  kalmar/växjö, sweden  email: anita.mirijamdotter@lnu.se     received: 10 january. 2010        accepted: 12 january. 2010       2010 mirijamdotter. this is an open access article distributed under the terms of the creative commons‐ attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐ sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original  work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is  redistributed under the same or similar license to this one.    this commentary is based on my keynote  address for the eblip conference held in 2009  in stockholm, sweden. the title was: bridging  the gap between users and systems – the potential  contribution of social informatics to evidence  based library and information practice. in the  following commentary, i focus on the  application of social informatics principles to  develop a collaborative evidence based  approach grounded in shared workplace  leadership. my remarks highlight some main  contributions from the field of library and  information science and social informatics1  and conclude with implications for practice,  including further research.    introduction    booth states “eblip (evidence based library  and information practice) appeals to the  scientific rationality that underpins much of  1 the research area social informatics is in this research equivalent to information systems.   our day‐to‐day work” (2006, p.51). this  formulation is well suited as a rationale for the  library leadership approach that is the focus of  this paper. lyons provides additional  affirmation of using evidence based practice  for leadership and management (2009). he  argues that in the field of business  management, it is well known that verifiable  and valid data, including feedback processes,  are essential for company improvement and,  thus, ought to be part of any decision‐making  process. toward that end, i propose building  on theories and models which clarify best  available evidence through social learning  processes that promote ‘learning the way’  through a systemic process of finding out,  modelling, assessing and evaluating.     the leadership model presented in this paper  has been tested and refined through action  research and participatory action research  fortified mainly by theories and models from  information and library science (e.g., bruce,  1997, 2008) and systems thinking (e.g.,  17 mailto:anita.mirijamdotter@lnu.se evidence based library and information practice 2010, 5.1  checkland 1981, 2000; checkland & poulter,  2006). it has been applied in three university  library organisations in the united states; an  account of the work at the first site at  california polytechnic state university in san  luis obispo, california is given in somerville  (2009), and somerville, rogers, mirijamdotter,  and partridge (2007). the philosophical  underpinnings of the work are based on the  assumption that organisations are constructed  and therefore can be reconstructed (norum,  2001). thus, leaders are responsible for the  design of organisations, processes, and  learning environments to further individual  and organisational learning.     the model and application process have been  guided by a social informatics design  orientation, i.e., user‐centred ict system  design principles are also applied to  (re)designing organisations, as well as systems  and processes. this orientation has further  benefited from scandinavian participatory  design philosophy which acknowledges that  inclusive design processes can cultivate  organisational learning which changes  workplace assumptions, fosters ‘shared  leadership’, and enables democratic decision‐ making (somerville, howard & mirijamdotter,  2009).     in the following paper, two models are  presented which together provide a  framework for organisational leadership  activities aimed at cultivating collaborative  evidence based information practices. theories  guiding implementation are highlighted and  suggested ways ahead are given.    activity model for library leadership    figure 1, below, illustrates a process‐oriented  and systems thinking based leadership model.  the model is conceptualised as numbered  activities for which organisational leaders are  responsible. it is depicted in layers to increase  readability.     at the very centre of the figure, ‘activity 1’  represents activities that are aimed at  encouraging creativity and collectivity to  foster a robust learning environment. its  inclusion in the model underscores the belief  that knowledge and information‐based  organisations, such as libraries, cannot be  managed in the traditional sense. rather,  organisational members should be encouraged  to actively and dynamically engage in  information exchange, guided by evidence  based practice and relevant models and  theories.     next, ‘activity 2’ signifies our belief that  active learning environments are based in  systems thinking which advances  understanding of organisational parts and  their interrelations. thereby we cultivate a ‘big  picture’ understanding which bridges  organisational functions.  systems thinking  and organisational learning is linked to the  creation of a shared organisational vision and  mission. ‘activity 3’ represents such processes  fortified by systems thinking principles, tools,  and models. the final activity in this layer,  ‘activity 4’, illustrates the significance of  organising information and communication  strategies for ‘sense‐making’ opportunities as  well as for information sharing activities  related to day‐to‐day work practice.     for the sake of model completeness, ‘activity  5’ acknowledges the importance of leading  operational‐level work. its counterpart,  ‘activity 6’, refers to engaging in collegial  relationship building activities, both inside  and outside organisational boundaries.  ‘activity 7’ represents understanding how and  why the present situation has come into being.  knowing history gives a relational context for  envisioning the future (‘activity 8’) including  anticipating service and system refinements.    finally, central to systems thinking, processes  and outcomes need to be appreciated and  evaluated in light of organisational purpose  and vision (‘activity 9’). in the case of the  leadership model, assessment involves how  well these activities contribute to nourishing  an active learning environment, which  supports developing and sustaining a learning  organisation.   18 evidence based library and information practice 2010, 5.1      fig. 1. process model for library organisation leadership (somerville, 2009, p. 56).    the modelling technique of figure 1 is based  on soft systems methodology guidance for  action research (checkland & holwell, 1998)  and it has additionally been used for practical  transformative processes (mirijamdotter &  somerville, 2009).     organisational leadership priorities    reflective evaluation of this participatory,  outcome‐oriented model for library  leadership, as depicted in figure 1, has raised  awareness of the importance of ‘activity 4’,  organising communication strategies to enable  organisational learning. this essential  leadership component requires the design of  effective organisational communication  strategies that allow collegial information  access and exchange and enable reflective  settings.     effective communication strategies which  enable collegial information access and  exchange are important because our belief is  that active participation in decision‐making  and action taking – as expressed in the concept  of shared leadership – enriches organisational  life and workplace effectiveness. therefore, we  strive to create tailored workplace  environments that enable ongoing dialogue,  reflection, and learning. these evidence based  conclusions emerge from the results of a study  conducted from 2003 to 2006 (mirijamdotter &  somerville, 2004, 2005, 2008, 2009; somerville,  2009; somerville, mirijamdotter & collins,  2006; somerville et al., 2007).  19 evidence based library and information practice 2010, 5.1  to enable effective information practices, we  build on informed workplace learning theories  (e.g., bruce 1997, 2008). bruce states that for  learning to occur, information encounters  must be experienced as sufficiently  ‘contextualized’ to activate and extend prior  understanding. in addition, dialogue and  reflection should be encouraged to promote  learning and the transfer of insights to novel  conditions. for shared leadership, this means  that members must appreciatively inquire and  examine experiences of information use and  information content on organisational as well  as individual levels. heightened engagement  with and learning from these experiences can  then be purposively incorporated into  organisational learning processes. as staff  members experience the efficacy of  ‘information in context’ usage, they develop  an appreciation for its practical application in  furthering organisational purposes  (somerville et al., 2009).    to ‘contextualize’ information experiences, we  draw insights from the work of stafford beer  (1979, 1985 and 1989). he proposed an  organising model, the viable system model  (vsm) represented in figure 2 below. the  model provides a generic systems framework  that can be used as a diagnostic device to  explain, analyse, and plan for organisational  sustainability. it includes five subsystems,  numbered as s1‐s5, which depict  organisational structures, roles, and functions.    system one (s1) is illustrated to the right as a  circle in the figure. it represents the  subsystems in which operational work is  carried out. for instance, in a library, learning  materials are acquired and organised,  discovery and access systems are  implemented and refined, research services  are delivered and evaluated, and facilities  spaces are configured and maintained.     the model assumes that there are several  operational systems (i.e. several s1s) –  although in the main of the figure only one is  represented – and therefore there is a need for  oversight and coordination. this is the  function of system two (s2), illustrated as a  triangle to the right. by providing  infrastructure and processes aimed at  coordinating operational work, operational  systems (s1s) can perform more smoothly. in  the case of a traditional library, this function is  often not made explicit and is traditionally  assigned to senior administration and/or to  groups or committees. in a shared leadership  environment, the coordinating function  becomes explicit and is a vital part of co‐ managing.    the square boxes at the right top represent the  meta‐system of the operational system(s).  system three’s (s3) function is to govern the  internal stability of the organisation by, for  instance, planning operational strategies,  strategically allocating resources, and  monitoring and controlling operations. this  function is also referred to as ‘here‐and‐now’.  closely related to s3 is the auditing function,  illustrated in the figure as a triangle to the left.  this function includes sporadic audits, in  addition to formal assessments and reviews  that are regularly made through explicit s3  functions. in traditional organisations, senior  administrators hold authority; when  accountability for planning and finances are  shared, authority is decentralised.    the concerns of system four (s4) are related to  organisational developmental issues aimed at  ‘there‐and‐then’, including strategic planning  for long‐term survival. activities of this  function also include environmental scans and  futurist scenarios. once again, these  responsibilities can be ‘closely held’ or shared,  depending on leadership philosophy.    finally, system five (s5) focuses on policy  making and organisational direction. it  furthers coherence among the interacting sub‐ systems represented by s1 through s4. vision,  values, and initiatives are examples of this  function, which informs the organisation’s  purposes, policies, and decisions. within a  shared leadership environment, development  of vision, values, and goals create the ‘glue’ for  evolving workplace relationships based in  perpetual learning.  20 evidence based library and information practice 2010, 5.1  21     fig. 2. viable systems model (vsm) with two embedded levels (nyström, 2006).    the thicker and thinner lines between the  functions represent information and  communication channels that are important  for the whole system’s viability. the main  channels between the meta‐system and the  operational system(s) support resource  allocation, accountability, and legal and  corporate requirements. additional channels  represent interaction, negotiation and  dialogue. although not illustrated in this  figure, channels should also enable interaction  between equivalent functions in embedded  levels. this embedment is illustrated in  system one (s1) of the figure by showing an  identical structure for the five functions,  including interrelated information flows. in  the figure we find three systems on the  embedded level of s1.     one important concept in systems thinking is  environment and the ‘disturbances’ and  constraints that it imposes upon the system. in  figure 2, cloud shapes on the left represent the  environmental concerns. each operational  system has its own environment; though they  may be overlapping, they are not identical.  additionally, the importance of looking  ‘outside’, an s4 function, is specially  emphasized.     in fulfilling the roles and responsibilities of the  leadership model depicted in figure 1, leaders  must not only mindfully enable process  intensive learning activities that promote  organisational learning but also attentively  advance (re)design of the larger organisational  structures depicted in figure 2. this will  provide the infrastructure in which to carry  out insights generated by dialogue and  reflection activities. such a framework for  conceptualising organisational leadership  recognizes the efficacy of generating  collaborative ‘evidence’ for decision‐making  evidence based library and information practice 2010, 5.1  and action taking through the social processes  of purposeful workplace learning.     evidence based model for information and  communication practices    most managers struggle to keep their  organisations viable amidst relentlessly  changing internal and external circumstances.  one promising application informed by this  leadership model (figure 1) uses collaborative  evidence based information practices (ebip)  (pan and howard, 2009) at the auraria library  in denver, colorado, u.s. in this instance,  organisational members gather their collective  knowledge and experience for analysis and  synthesis within relevant contexts formulated  in communities of practice. the collaborative  ebip activities are not limited to  organisational boundaries; they also include  stakeholders that are affected by the  organisation’s activities, processes, and vision.     the participants in this particular example  apply the theoretical frameworks discussed in  the process model for library organisation  leadership (figure 1) and the viable system  model (figure 2) which provide the  infrastructure and approach for  contextualising information seeking and use.  together these two models permit thoughtful,  contextualised consideration of questions. the  organisation is thereby conceptualized as an  information processing system in which the  leader’s role involves clarifying sustainable  information flows, activities, and functions  needed for perpetual workplace learning. this  is no easy task, as the following discussion of  more high level, nuanced aspects of the vsm  illustrates.  the key principles of vsm are recursion,  variety, and autonomy. recursion means that  there exists an identical structure for the five  functions, including interrelated information  flows, embedded in each level s1 –  see figure  2 for an illustration of how this structure is  embedded in s1. in the auraria library  example, this kind of embedded structure is  visible in the organisational chart’s four  divisions, headed by an associate director, and  containing subunits within each division.  however, the implications are that each and  every vsm function should be exercised in  every division and in each of its subunits.  further, to enable communication and  interaction between subordinate units and  subunits, the identical information and  communication infrastructure should be in  place. some of these complex information and  interaction needs have been met by re‐ organising work in various teams and  committees. further exploration of  information use and content needs will enable  analysing and diagnosing how best to more  adequately fulfil all of the vsm functions,  channels and flows.     variety is a term used to indicate diversity and  complexity. it can be exemplified by  information created in subsystems on  embedded levels and forwarded upwards in  the structure. to use a metaphor, consider that  information in subsystems is like streams that  flow into the river, eventually increasing the  variety. however, the systems on higher  recursive levels do not have superior variety;  rather their ability to manage that kind of  generated variety is inadequate. the same  ideas are valid for information flows in the  opposite direction. therefore, it is important to  examine experiences of information use for  each function and level to avoid information  overload or information inadequacy and to  enable appropriate content, delivery method,  and channel. in the auraria library example,  this principle allows additional focus for  investigation of organisational members’  experiences of information use and  information content. such an investigation  could enable one to analyse and diagnose the  handling of ordinary requests presented by  internal and external clients and, perhaps  more importantly, the effects of unexpected  opportunities and requests – why they were  unexpected and, if they were dealt with, in  what way and with which effects? for  instance, did the organisation change its  operations and processes, did the  opportunity/request affect the objective, or  was the decision to ‘ignore’, i.e. do nothing?    22 evidence based library and information practice 2010, 5.1  the third principle – autonomy – emphasizes  freedom and the authority to manage one’s  own affairs. this indicates that people have  work tasks related to functions at embedded  levels for which they have, or should have,  autonomy. in exchange, individuals are  expected to be responsible and accountable for  their autonomous work tasks. this principle is  aligned with the philosophical underpinnings  of the leadership model discussed above.  however, to take responsibility for  autonomous work necessitates well‐ functioning vertical information flows on, for  instance, rules and policies. additionally, to  continually refine individuals’ capacity to  make appropriate interpretations when faced  with novel circumstances, they must be  afforded ongoing dialogue, reflection, and  learning opportunities, rather than relying  merely on formal reporting. thus, information  flows need to be robust and ensure that issues  and resolutions are widely transmitted  horizontally and vertically throughout the  workplace. in the auraria library example,  the principle of autonomy is explored through  appreciative inquiry and shared leadership  (pan & howard, 2009). to support shared  leadership and autonomy, strategies of  (re)organising for information dissemination  and two‐way communication flows are being  explored using both push and pull it as well  as in face‐to‐face meetings. however, more  exploration and experience is needed to clarify  ‘best practices’.      it follows that library organisation leadership  practice should cultivate a continuous  participatory research, implementation, and  evaluation culture supported by the allocation  of financial and human resources to  ‘incentivize’ collaborative innovation and  creativity. this organisational outcome  emerges from a continuing emphasis on  organising effective information and  communication flows. it implies that members  of the organisation appreciatively examine  experiences of information use and  information content on organisational as well  as individual levels. for instance, work to date  at the auraria library suggests the importance  of attending to:  • the autonomy and scope for decision‐ making in relation to function and  embedded level,   • the collective process of gathering  evidence for decision‐making,  • methods for sharing knowledge  within the organisation, and  • methods for implementing collective  workplace learning.    additionally, attention must be directed  toward enhancing individual and collective  capacity to better understand: how to package  information, what to filter out, what to  emphasize, to which function, to which  functional level, and with which frequency.     and finally, central to systems thinking,  organisational members must elaborate:  criteria and methods for evaluating  performance in different functions and  embedded levels (mirijamdotter, 2009).    implications for practice    these models guide an intrinsically  participatory and action‐oriented research  approach. key elements of implementation  involve learning with and for beneficiaries,  evaluating frame of references, and using  systems thinking for collaborative ebip. the  approach has proven to be a fruitful way  forward for professional and organisational  development in three increasingly ambitious  examples of implementation in north  america. in each instance, because the same  water never flows under the same bridge,  specifics of the workplace circumstances and  culture encouraged variations in the  expression and outcomes of these evidence‐ producing leadership models.      implications for research    library, information, and knowledge  professionals are increasingly confronted with  novel circumstances requiring nimble and  agile organisational responsiveness grounded  in sound evidence based practices and  processes. therefore, there is much to gain by  researching the efficacy of organisational  23 evidence based library and information practice 2010, 5.1  learning approaches which conceptualize the  organisation as an information processing  system. the potential of harnessing collective  wisdom among communities of practice is an  argument for continued research into the  effectiveness of the systems approach for  fostering collaborative evidence based  information practices.      references      beer, s. (1979). the heart of enterprise. newyork,  ny: john wiley and sons ltd.   beer, s. (1985). diagnosing the system for  organisations. chichester, uk: john  wiley and sons.   beer, s. (1989). the viable system model: its  provenance, development,  methodology and pathology. in raul  espejo and roger harnden (eds.), the  viable systems model: interpretations and  applications of stafford beer’s vsm (pp.  11‐37). chichester, uk: john wiley  and sons.  booth, a. (2006). the unteachable in pursuit of  the unreadable? evidence based library  and information practice, 1.2, 51‐56.  retrieved 20 feb. 2010 from  http://ejournals.library.ualberta.ca/ind ex.php/eblip/article/view/48/118.   bruce, c.s. (1997). the seven faces of information  literacy. adelaide, australia: auslib  press.   bruce, c.s. (2008). informed learning. chicago,  il: association of college and  research libraries.  checkland, p.b. (1981). systems thinking,  systems practice. chichester: john  wiley & sons.   checkland, p.b. (2000). soft systems  methodology: a thirty year  retrospective. systems research and  behavioral science, 17(s1), s11–s58.   checkland, p.b. & holwell, s. (1998). action  research: its nature and validity.  systemic practice and action research,  11(1), 9–21.  checkland, p.b. & poulter, j. (2006). learning  for action: a short definitive account of  soft systems methodology and its use for  practitioners, teachers and students.  chichester: john wiley & sons.  lyons, r. (2009). the call for evidence based  practice: speaking louder than words.  evidence based library and information  practice 4.3, 63‐67. retrieved 20 feb.  2010 from  http://ejournals.library.ualberta.ca/ind ex.php/eblip/article/view/6530/5537  mirijamdotter, a. (2009). auraria library report  on communication, decision making, and  planning systems. denver, colorado:  auraria library.  mirijamdotter, a. & somerville, m.m. (2004).  systems thinking in the workplace:  implications for organizational  leadership. proceedings from the 3rd  international conference on systems  thinking and management  (icstm04). philadelphia, pa.  retrieved 20 feb. 2010 from  http://www.acasa.upenn.edu/icstm04.  mirijamdotter, a. & somerville, m.m. (2005).  dynamic action inquiry: a systems  approach for knowledge based  organizational learning. proceedings  from the 11th international  conference on human‐computer  interaction, las vegas, nv: lawrence  erlbaum associates, inc.  mirijamdotter, a. & somerville, m.m.  (2008).  ssm inspired organizational change in  a  north  american  university  library:  lessons learned. viveca asproth et al.  (eds.).  proceedings  of  the  31st  information  systems  research  seminar  in  scandinavia  (iris31):  public  systems  in  the  future  –  possibilities,  challenges,  and  pitfalls,  åre,  sweden.  retrieved  from  http://www.iris31.se/papers/iris31‐ 046.pdf.   mirijamdotter, a. & somerville, m.m. (2009).  collaborative design: an ssm‐enabled  organizational learning approach.  international journal of information  24 http://www.iris31.se/papers/iris31-046.pdf http://www.iris31.se/papers/iris31-046.pdf evidence based library and information practice 2010, 5.1  25 technologies and systems approach 2(1),  48–69.  norum, k.e. (2001). appreciative design.  systems research and behavioural  science 18, 323–333.  nyström, c.a. (2006) design rules for  intranets according to the viable  system model. systemic practice and  action research, 19, 523–535.  pan, d. & howard, z. (2009). reorganizing a  technical services division using  collaborative evidence based  information practice at auraria  library. evidence based library and  information practice 4.4, 88‐94.  retrieved 20 feb. 2010 from   http://ejournals.library.ualberta.ca/ind ex.php/eblip/article/view/6516/5869.   somerville, m.m. (2009). working together:  collaborative  information  practices for  organizational learning. chicago, il:  association of college and research  libraries.   somerville, m.m., rogers, e., mirijamdotter,  a., & partridge, h. (2007).  collaborative evidence‐based  information practice: the cal poly  digital learning initiative. in e. connor  (ed.), evidence based librarianship: case  studies and active learning exercises (pp.  141‐161). oxford: chandos.  somerville, m.m., howard, z. &  mirijamdotter, a. (2009). workplace  information literacy: cultivation  strategies for working smarter in 21st  century libraries ed. d. m. mueller.  proceedings of the 14th association of  college & research libraries national  conference: pushing the edge: explore,  engage, extend (pp. 119‐126). chicago:  association of college and research  libraries.  somerville, m.m., mirijamdotter, a. & collins,  l. (2006). systems thinking and  information literacy: elements of a  knowledge enabling workplace  environment. proceedings of the 39th  annual hawaii international  conference on systems sciences  (hicss‐39), koloa, kauai. los  alamitos, california: ieee computer  society. retrieved 20 feb. 2010 from  http://csdl2.computer.org/comp/proce edings/hicss/2006/2507/07/250770150.p df.        http://csdl2.computer.org/comp/proceedings/hicss/2006/2507/07/250770150.pdf http://csdl2.computer.org/comp/proceedings/hicss/2006/2507/07/250770150.pdf http://csdl2.computer.org/comp/proceedings/hicss/2006/2507/07/250770150.pdf librarians are interested in finding research collaborators evidence summary   librarians are interested in finding research collaborators   a review of: tran, n. y., & chan, e. k. (2020). seeking and finding research collaborators: an exploratory study of librarian motivations, strategies, and success rates. college & research libraries,81(7), 1095. https://doi.org/10.5860/crl.81.7.1095   reviewed by: jennifer kaari librarian east orange public library east orange, new jersey, united states of america email: jkaari@eopl.org   received: 2 sept. 2021                                                               accepted:  12 oct. 2021      2021 kaari. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30031     abstract   objective – to explore research collaboration among librarians, including librarians’ motivations for collaboration, methods for finding collaborators, and how they perceive the success of these methods.   design – online survey questionnaire.   setting – n/a   subjects – a total of 412 librarians took the survey, and 277 respondents completed the entire survey.    methods – the researchers developed a survey using qualtrics, including questions focused on whether respondents had sought research collaboration, factors that motivated them to collaborate, methods they used for finding collaborators, and success rates of these methods. demographic questions were also included.    main results – the survey results indicated that librarians are very interested in research collaboration, with 91.8% of respondents answering that they had sought collaborators, were currently collaborating, or were interested in seeking collaborators in the future. the top motivating factor for seeking collaboration was to gain expertise that the respondent lacked. the most common strategy for finding collaborators was through a respondent’s current or past place of employment, and this method was rated as extremely successful by more than 50% of respondents. demographically, 70.1% of respondents worked in academic libraries.    conclusion – the results of this study indicate that research collaboration is of interest to librarians at a higher rate than previously observed. these results can help inform initiatives to support and promote collaboration in library and information science research, as well as provide a groundwork for further research in this area.   commentary   why are librarians seeking out research collaborations and how are they finding their collaborators? the authors of this paper set out to explore these questions, building on an established literature that indicates research collaboration and co-authorship are both common and increasing in practice among library and information science researchers (chang, 2018; cheng et al., 2019).   utilizing glynn’s critical appraisal tool, this study was found to have 80% validity (2006). the authors clearly describe their methods, include the survey in the published paper, and provide well-outlined results. the largest weakness of the study is the small sample size, which as the authors note, makes the results not generalizable to a large population.   this study would have been strengthened by narrowing the focus of the survey population. the survey was open to librarians from all disciplines and areas of library science; however, the environment around research and collaboration varies across different types of institutions and disciplines within librarianship. given that academic librarians were the largest subgroup of respondents and that the environment around publication is very specific in academia, focusing the study on academic librarians exclusively may have made the results more insightful, even with a small sample size.  in addition, it seems unclear if the term “research collaboration” was clearly defined for the respondents of the survey. the authors define the term as research wherein professional contributions are written by two or more authors, but it seems unclear if the term is meant to encompass published papers as well as other research outputs such as posters and presentations.   as an exploratory study, this paper does provide a good foundation for further research into the area of research collaboration. the authors note many areas for future inquiry, including opportunities for cross-sectional analysis and further study into the researchers with whom librarians are collaborating. it would be interesting to know how often librarians are collaborating with researchers from outside library science.   institutions and individuals seeking to strengthen and encourage research collaboration would find insights into where librarians are looking for collaborations, and they could potentially find opportunities to encourage use of these existing methods as well as develop pathways to explore new methods for collaboration seeking. the results of this study seem to indicate that librarians may find collaborators primarily through existing connections—it would be very intriguing to investigate how new connections for research collaboration are, or could, be formed. although this study found that very few respondents utilized social media to find collaborators, a deeper investigation into the potential of social media to facilitate collaboration is also an area for further research.   references   chang, y.-w. (2018). research collaboration by practitioners in computer science, library science, and management. portal: libraries and the academy, 18(3), 473–490. https://doi.org/10.1353/pla.2018.0029   cheng, f.-f., huang, y.-w., tsaih, d.-c., & wu, c.-s. (2019). trend analysis of co-authorship network in library hi tech. library hi tech, 37(1), 43–56. https://doi.org/10.1108/lht-11-2017-0241   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387–399. https://doi.org/10.1108/07378830610692154     review article   teaching knowledge synthesis methodologies in a higher education setting: a scoping review of face-to-face instructional programs   zahra premji research and learning librarian libraries & cultural resources university of calgary calgary, alberta, canada email: zahra.premji@ucalgary.ca   k. alix hayden senior research librarian libraries & cultural resources university of calgary calgary, alberta, canada email: ahayden@ucalgary.ca   shauna rutherford information literacy coordinator (retired) libraries & cultural resources university of calgary calgary, alberta, canada email: shauna.rutherford@ucalgary.ca   received: 11 jan. 2021                                                               accepted: 17 mar. 2021      2021 premji, hayden, and rutherford. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29895     abstract background knowledge synthesis (ks) reviews are increasingly being conducted and published. librarians are frequently taking a role in training colleagues, faculty, graduate students, and others on aspects of knowledge syntheses methods. objective in order to inform the design of a workshop series, the authors undertook a scoping review to identify what and how knowledge synthesis methods are being taught in higher education settings, and to identify particularly challenging concepts or aspects of ks methods. methods the following databases were searched: medline, embase & apa psycinfo (via ovid); lisa (via proquest); eric, education research complete, business source complete, academic search complete, cinahl, library & information science source, and socindex (via ebsco); and web of science core collection. comprehensive searches in each database were conducted on may 31, 2019 and updated on september 13, 2020. relevant conferences and journals were hand searched, and forward and backward searching of the included articles was also done. study selection was conducted by two independent reviews first by title/abstract and then using the full-text articles. data extraction was completed by one individual and verified independently by a second individual. discrepancies in study selection and data extraction were resolved by a third individual. results the authors identified 2,597 unique records, of which 48 full-text articles were evaluated for inclusion, leading to 17 included articles. 12 articles reported on credit courses and 5 articles focused on stand-alone workshops or workshop series. the courses/workshops were from a variety of disciplines, at institutions located in north america, europe, new zealand, and africa. they were most often taught by faculty, followed by librarians, and sometimes involved teaching assistants. conclusions the instructional content and methods varied across the courses and workshops, as did the level of detail reported in the articles. hands-on activities and active learning strategies were heavily encouraged by the authors. more research on the effectiveness of specific teaching strategies is needed in order to determine the optimal ways to teach ks methods.   introduction knowledge synthesis (ks), also known as evidence synthesis (es), is defined as “the contextualization and integration of research findings of individual research studies within the larger body of knowledge on the topic” (grimshaw, 2010, p. 2; cochrane, 2020). furthermore, ks uses methods that are transparent and reproducible (chandler & hopewell, 2013).   there are many types of knowledge synthesis reviews (sutton et al., 2019), and one of the most well-known is the systematic review (sr). a systematic review “seeks to collate evidence that fits pre-specified eligibility criteria in order to answer a specific research question” and attempts “to minimize bias by using explicit, systematic methods documented in advance with a protocol” (chandler et al., 2020, p. 1). srs provide an up-to-date synthesis of the state of knowledge on a topic, which can aid in decision-making for practice or policy, identify and indicate gaps in knowledge or lack of evidence, and reveal the limitations of existing studies on a topic (lasserson et al., 2020). whereas srs have been prevalent in the health sciences for some time, they are gaining popularity in a broader range of disciplines.   while systematic reviews are being increasingly published, many have incomplete reporting or were conducted poorly (bassani et al., 2019; page et al., 2016; pussegoda et al., 2017). experts recommend that both researchers and journal editors should be better educated on sr methodologies (page & moher, 2016; page et al., 2016). they specifically advocate for education focused on strategies to identify bias in a sr, as well as strategies to minimize these biases, which will help to improve the quality of published systematic reviews, and, subsequently, help to “reduce this avoidable waste in research” (page et al., 2016). cochrane, an evidence synthesis organization, recommends that first time review authors attend relevant training and work with others who have experience conducting srs (lasserson et al., 2020).   currently, education on ks methods takes many forms such as higher education courses, continuing education courses, workshops, webinars, and elearning modules. many evidence synthesis organizations including cochrane, joanna briggs institute, and the campbell collaboration offer fee-based workshops and courses that focus on ks methods (cochrane, n.d.; campbell collaboration, n.d.; joanna briggs institute, n.d.). sr instruction is also offered as credit-bearing courses to undergraduate or graduate students in post-secondary institutions (himelhoch et al., 2015; li et al., 2014). some professional development workshops on ks methods are available at conferences. additionally, academic libraries offer workshops on some steps of the systematic review methodology (campbell et al., 2016; lenton & fuller, 2019). all of these different programs vary in terms of learner audience, breadth and depth of content covered, and delivery methods, while having the shared goal of educating researchers in the steps and processes necessary to conduct ks reviews.    objectives   we undertook the study as two of the authors were beginning to design of a series of in-person workshops to teach systematic or scoping review methodology. we wanted to learn which teaching methods work well and what challenges we might encounter. we initially considered a systematic review, however, we realized that we were conducting an exploratory study where the literature had not been previously mapped in a structured way. munn et al. (2018) note that an indication for conducting a scoping review is “to identify key characteristics or factors related to a concept” (p. 2). further, we wanted to include all forms of evidence, including quantitative or qualitative studies, scholarship of teaching reflections, opinion articles, and program descriptions. given our openness to all evidence types from all disciplines, we expected that the retrieved literature could be quite heterogenous, which is one reason to choose a scoping review (peters et al., 2020). scoping reviews “are more appropriate to assess and understand the extent of the knowledge in an emerging field or to identify, map, report, or discuss the characteristics or concepts in that field” (peters et al., 2020, p. 2121). therefore, we decided that a scoping review was the best approach to inform development of both the content and the delivery of our workshop series.   the objective of our scoping review was to identify the extent of the literature and summarize articles that describe the teaching and learning of any knowledge synthesis methodology in a post-secondary setting, with at least a partial in-person (face-to-face) component, to determine:   1)      steps of the knowledge synthesis process taught 2)      teaching methods and learner activities used 3)      learner challenges encountered   a recent environmental scan focusing on online ks instructional courses already exists (parker et al., 2018). the authors evaluated 20 online training resources against best practices for online instruction using a rubric. to avoid duplication, we decided to exclude online courses and focus solely on face-to-face educational options.   methods   a protocol outlining the objectives, inclusion criteria, and methods for this scoping review was developed in may 2019 to inform our study, and is available from the first author. the protocol is based on the methodological guidelines outlined by the joanna briggs institute (jbi) for the conduct of scoping reviews (peters et al., 2017). additionally, our study is reported according to the prisma-scr (preferred reporting items for systematic reviews and meta-analysis extension for scoping reviews) guidelines (tricco et al., 2018).   study eligibility   the population (p) in our scoping review is individuals at post-secondary institutions, which includes students, staff, librarians, and faculty. the concept (c) of interest is instructional interventions for learning knowledge synthesis methodologies. the specific context (c) we are interested in is non-commercial courses that had some in-person component.   specifically, articles were eligible for inclusion if they describe a course or workshop that:   ●       taught knowledge synthesis methodology, which we operationalized as the teaching of at least two steps of the knowledge synthesis methodology (protocol development, question formulation, data collection, study selection, data extraction, critical appraisal, narrative synthesis, meta-analysis, or reporting) ●       was offered in a higher education setting (for example, credit-bearing, professional development, optional workshop) ●       included at least some in-person components (blended courses or entirely in-person course) ●       incorporated an evaluative, reflective, or assessment component (this could take the form of assessments of student learning, workshop/course evaluations, or instructor observations or reflection)   additionally, articles were considered ineligible if they:   ●       covered a course where teaching was entirely online or via asynchronous methods ●       discussed commercially offered courses such as those being offered by organizations involved in knowledge synthesis (e.g. cochrane, joanna briggs institute, and others) ●       focused on evidence based medicine/practice, where methodology of systematic reviews is not significantly covered ●       discussed only one step of the knowledge synthesis methodology ●       were published in languages other than english   search strategy and information sources   we utilized a three-step search strategy, as outlined by jbi (peters et al., 2017). first, we conducted an exploratory search in google scholar to discover relevant seed studies that met the inclusion criteria for our review. the articles’ titles and abstracts were analyzed and mined for keywords. as well, we analyzed the seed article records in the medline (ovid) database to identify relevant subject headings. from this analysis, a search was developed in medline (ovid), and was piloted against the known seed articles to ensure relevant studies were captured. this medline search was developed by a librarian (zp) and peer-reviewed by a second librarian (kah). the search was then translated for all databases identified in our search protocol. the searches incorporated subject headings when available and free-text terms were combined using appropriate boolean operators. no language, date, or study design filters were used. the complete search strategies for all databases are included in the appendix.   the choice of databases was purposefully exhaustive so that as many different disciplines as possible would be represented in our scoping review. the following ovid databases were searched:    ·        medline(r) and epub ahead of print,in-process & other non-indexed citations and daily (1946 – sept 13, 2020), ·        embase (1974 – sept 13, 2020), ·        apa psycinfo (1806 to sept 13, 2020).   ebsco databases were searched simultaneously. one of the major reasons to search databases individually is to ensure effective subject heading searching. however, with the ebsco databases, none of the existing subject heading were useful in retrieving relevant articles. the subject headings related to ks retrieved knowledge synthesis reviews rather than articles that discuss ks as a topic. as no appropriate subject terms/headings were found, we decided that there was no need to search the databases discretely. ebsco databases included were:   ·        socindex with full-text (1908 to sept 13, 2020) ·        education research complete (1880 to sept 13, 2020), ·        eric (1966 to sept 13, 2020), ·        cinahl plus with full-text (1937 to sept 13, 2020), ·        library and information science source (1901 to sept 13, 2020), ·        academic search complete (1887 to sept 13, 2020), ·        business source complete (1886 to sept 13, 2020),   additional databases searched included:   ·        lisa: library and information science abstracts (proquest, 1969 to sept 13, 2020), ·        web of science core collection. this core collection includes: ·        science citation index-expanded (1900 to sept 13, 2020), ·        social sciences citation index (1900 to sept 13, 2020), ·        arts & humanities citation index (1975 to sept 13, 2020), ·        conference proceedings citation index science (1990 to sept 13, 2020), ·        conference proceedings citation index social sciences & humanities (1990 to sept 13, 2020), ·        emerging sources citation index (2005 to sept 13, 2020).   searches were conducted on may 31, 2019 and updated on september 13, 2020. results were downloaded in ris or text format, and deduplicated in covidence software ("covidence," n.d.).   our third and final step included the hand-searching of relevant journals and conferences, as well as scanning the reference lists of included articles and the associated cited-bys. we hand-searched issues published within the last three years (2017-2019) of the following journals: journal of the canadian health libraries association, journal of the medical library association, evidence based library and information practice, and research synthesis methods. we also hand-searched the programs from the following annual conferences: european association of health information and libraries (2017-2019), medical library association (2017-2019), canadian health libraries association (2017-2019), association of european research librarian liber (2017-2019), and evidence based library and information practice (2019). additionally, we conducted forward and backward citation searching by scanning the reference lists and the cited-bys (via google scholar) of all included articles. where further details were required, authors of the included studies were contacted via email.   study selection   study selection was conducted in two phases, first by title/abstract, and then using the full-text. the process was completed in duplicate, using two independent reviewers (zp and sr). we first piloted a random set of 50 records to ensure that the eligibility criteria were clear and consistently applied by both screeners. a third independent reviewer resolved discrepancies (kah). a similar process was followed for the full-text screening, which was also done independently in duplicate (zp and sr), with a third reviewer resolving discrepancies (kah). covidence software was used to facilitate the study selection process.   data extraction   data were extracted in excel. the following categories were extracted from each included article:   author and year title participants (discipline and level) instructor (librarian, faculty, or other) location of course course format/structure course-integrated or stand-alone course objectives steps of ks methodology taught (specifically, defining the question, developing a protocol, searching the literature, citation management, screening, data extraction, narrative synthesis, meta-analysis, reporting, or critical appraisal) assessment of student learning/learner activities course evaluation/reflection outcomes of course assessment   a data extraction template was created in excel and was piloted by two individuals independently, using 3 studies. data extraction was then completed by one individual (sr) and was verified independently by a second individual (zp). verification was done by checking each data point extracted by the first individual against the original source article. when discrepancies were found, they were first discussed between the data extractor and data verifier. a third individual reviewed any discrepancies in coding that were not easily resolved through the initial consensus process (kah).   results   the data collection process identified 4,857 records for title/abstract screening, of which 2,112 were duplicates. after applying inclusion criteria to the 2,597 unique records, 48 articles were left for full-text screening. at the end of the full-text screening process, 17 articles remained that met the inclusion criteria for this scoping review. inter-rater agreement for the title/abstract screening was 98%, and for the full-text screening was 87%. the inter-rater agreement was calculated automatically by covidence and is the proportionate agreement level between the two reviewers across the entire set of records or articles. this means that the two reviewers voted the same way on 98% of the total records during title/abstract screening and 87% of the articles during the full-text screening stages. the results of the study selection process are reported in a modified prisma flow diagram (moher et al., 2009) in figure 1 below.   figure 1 prisma flow diagram.     description of included articles   the population (discipline, learner level) and intervention characteristics (course-integrated or stand-alone, instructors, location) of the 17 included articles in this review are shown in table 1 below.   the majority of articles describe interventions from north america, with six from the united states, and four from canada. three were located in the united kingdom, with an additional one each from germany, italy, new zealand, and zimbabwe. most instruction targeted graduate students as learners. the majority (12) of the articles describe instruction where ks was the focus of an entire credit course or where teaching ks was integrated into such a course, whereas the other five articles describe stand-alone workshops. ks instruction was taught to a broad range of disciplines. many of the articles describe ks instruction related to the health sciences (i.e., dentistry, nursing, biomedical sciences, exercise science, public health and speech pathology) which reflects the prevalence of ks in these disciplines. faculty were involved as instructors in all but three of articles, the remaining of which were taught by librarians. in seven of the articles, teaching was shared to varying degrees among faculty, librarians, teaching assistants and facilitators.   inclusion criteria for our review dictated that all included workshops or courses covered content related to at least two steps of the ks process, but as table 2 shows, most covered many more. the stand-alone workshops, which were of shorter duration than the credit courses, included fewer steps of the ks process. the three workshops taught exclusively by librarians (campbell et al., 2016; conte et al., 2015; lenton & fuller, 2019) taught the fewest steps. this could be due to the fact that these workshops were shortest in length, and also because the steps covered (problem definition, searching, and citation management) are those that align most closely with librarian expertise (spencer & eldredge, 2018). all 12 credit-bearing courses taught research question formulation, searching, screening and data extraction. two of the articles for course-based instruction (azarpazhooh et al., 2008; groller et al., 2020) explicitly describe the teaching of five steps, whereas all other courses covered six or more. the most commonly taught step was “searching the literature,” which all 17 articles describe. this was followed by “defining the question” (16 articles), “reporting” (15 articles) and “screening” (14 articles). the least common step to be taught was “narrative synthesis” (five articles).     table 1 population and intervention characteristics of included studies characteristic n population       discipline       mixed 3   dentistry 2   nursing 2   biomedical sciences 1   business 1   educational psychology 1   engineering 1   exercise science 1   health economics professional librarians 1 1   psychology 1   public health 1   speech pathology 1       learner level   graduate students undergraduates mixed librarians   9 4 3 1         intervention       workshop design     course integrated 12 stand-alone 5           instructors         faculty only 7     faculty + librarian(s) 4     librarians only 3     faculty + tas 2     faculty + librarians + tas 1           location         united states 6     canada 4     united kingdom 3     germany italy 1 1     new zealand 1     zimbabwe 1     table 2 steps of the knowledge synthesis process that were taught in the content of each course or workshop     our review captured a very diverse set of courses and workshops teaching knowledge synthesis review methodology. tables 3 and 4 display the summaries of instruction interventions described in the included literature. the data are presented in two tables, with course-based instruction and stand-alone workshops displayed separately because of some clear differences between the two types of offerings.   table 3 presents a summary of the stand-alone workshops. all five workshops included limited contact time with learners, ranging from three hours in total (campbell et al., 2016) to five full days (flores-mir et al., 2015; jack et al., 2020). the course objectives for these workshops are stated in terms of preparing attendees to participate in future reviews, which is appropriate given their short duration. they aim to build capacity rather than to give students extensive experience in conducting reviews. librarians were the sole instructors in three of the workshops (campbell et al., 2016; conte et al., 2015; lenton & fuller, 2019). the workshops targeted a more diverse group of learners than the credit courses, usually including a mix of levels (undergraduates, graduate students, post-docs, researchers, librarians, professional staff). also, without the graded assignments available to instructors in a credit course, there were more limited examples of student assessment. two of the workshops (campbell et al., 2016; flores-mir et al., 2015) do not mention assessment of student learning at all. two of the articles mention conducting pretests and posttests (conte et al., 2015; jack et al., 2020), and two articles describe assigning participants an assessment activity at the end of the workshop (jack et al., 2020; lenton & fuller, 2019). all of the workshops offered some form of post-course evaluation survey.   the 12 credit courses are summarized in table 4. four of the courses were offered to undergraduate students, while eight were at a graduate level. faculty members were the primary instructors for all the courses, and the sole instructors for seven. five articles (briner & walshe, 2014; gorczynski et al., 2017; groller et al., 2020; li et al., 2014; proly & murza, 2009) explicitly mention librarian involvement either within the original course or as a modification for later offerings based on feedback. the course objectives generally focus on students developing an understanding of reviews and the skills to conduct one. some unique objectives include learning to teach others about systematic reviews (li et al., 2014) and gaining project management skills and leadership experience (proly & murza, 2009). the articles present a variety of graded assignments designed to assess student learning, many of them tied to specific steps of the review process. oral presentations were assigned in five courses and students created a poster presentation for one course (bourke & loveridge, 2013). ten of the courses required students to hand in either a written summary of findings or a research manuscript based on their review. the most common form of course assessment used was a post-course questionnaire or survey, mentioned in seven articles. groller et al. (2020) also discusses an online survey specific to the information searching session offered by the librarian. other forms of course assessment include a focus group (azarpazhooh et al., 2008), student self-assessments (briner & walshe, 2014), faculty observations (briner & walshe, 2014), and an analysis of student performance (land & booth, 2020).   one of the primary goals of this study was to investigate instruction methods for teaching knowledge synthesis methodology. table 5 explores the variety of teaching and learning strategies implemented for different steps of the knowledge synthesis process. we did not include traditional lecturing as we were most interested in discovering active teaching and learning strategies. the coding is not discreet that is, multiple learning strategies may be employed in teaching a single step. for example, database searching may be coded as both hands-on and small group, as the participants worked together to develop search strategies. the majority of articles only briefly mention specific teaching and learning strategies. li et al. (2014), for instance, states “we developed this course with a philosophy of “learning by doing” (p. 255) but provides little detail on the learning activities and teaching strategies used.  similarly, jack et al. (2020) mentions that learners participated in interactive exercises in groups; however, only one example is given. pieper et al. (2019), who also noted that they used a “learning by doing” philosophy, followed a unique approach implementing a “guiding systematic review” which is a published systematic review used as a “working example throughout the course” (p. 3). a wide range of active learning and teaching strategies were employed across the courses and workshops, with hands-on or small group activities being most commonly mentioned. hands-on activities were used most for teaching the steps of question development, database searching, screening, data extraction, and critical appraisal. these steps are mirrored in the small group activities, as small group activities often included hands-on experiences.     table 3 summary of workshop characteristics author, date, country  discipline, level, workshop structure instructors  course objectives  assessment of student learning  course evaluation  campbell, 2016, canada  mixed, students/faculty/ researchers, 3hr workshop   librarians  participants will: identify systematic reviews, recognize the range of resources required to execute a systematic review search, develop a well-formulated search question and structure a search using the picos format, learn to apply appropriate search limits, document a search in a standardized form, understand the importance of peer-review of systematic review searches, and recognize the level of expert searching needed for a systematic review    evaluation questionnaire   conte, 2015, usa  mixed, librarians, 2-day workshop  librarians  students will gain knowledge of best practices in conducting systematic reviews and create a personalized action plan to establish their libraries as centers of expertise for systematic reviews   online pre and posttests,  online post-course survey, mla evaluation form, focus group  flores-mir, 2015, canada  dentistry, faculty/graduate students/staff, 5 x 8hr sessions  faculty, librarian (as guest lecturer)   students will broaden knowledge of evidence based practice principles in dentistry and gain hands-on experience in designing, conducting, writing, and critiquing health care systematic reviews.      post-workshop evaluation forms    jack, 2020, zimbabwe  mixed, phd/post-doc/librarians/ program managers, 5-day workshop  faculty  to teach trainers from three african countries to conduct systematic review workshops at their home institutions in order to broaden mental health research capacity  online pre and posttests, learner presentations at end of workshop  post-workshop survey assessing learner satisfaction and perception of confidence in conducting a sr  lenton, 2019, canada  mixed, graduate students, 3 x 2.5hr sessions  librarian  students will learn to identify differences between types of reviews, incorporate tools & resources for proper reporting & management of the review, utilize strategies for creating a searchable question with inclusion/exclusion criteria, identify relevant databases, practice using a structured method for developing advanced search strategies   student observation during activities, ticket-out-the-door evaluation forms. short post-course reflection questionnaire  short post-course reflection questionnaire        table 4 summary of course characteristics author, date, country  discipline, level, course structure  instructors  course objectives  assessment of student learning  course evaluation  azarpazhooh, 2008,  canada  dentistry, undergrad, 3 x 1hr lectures, 3 x 2-3hr discussion sessions, 1 2-3hr presentation session  faculty, facilitators  students will develop and apply skills in evidence based dental practice by finding relevant literature, evaluating and selecting the strongest evidence, summarizing findings, and communicating results   students evaluated on quality of participation in group discussions, group presentations and on summary reports of findings  online pre and posttests, online post-course survey, mla evaluation form, focus group  baldasarre, 2008, italy  electrical engineering, masters, 10 sessions  faculty, phd students  students will be introduced to empirical research methods and trained to empirically evaluate software engineering tools, techniques, methods and technologies  definition of research protocol assignment, definition of inclusion/exclusion criteria assignment, data extraction assignment     post-course questionnaire     bourke, 2013, new zealand   educational psychology, masters, not specified  faculty  not provided in article  poster presentations of initial finding of systematic reviews; students then submit full systematic review incorporating faculty & peer feedback on posters.  student self-assessments throughout course    briner, 2014, uk    business, masters, 7 x 3hr sessions  faculty librarian (as guest lecturer)  students will gain understanding of evidence based practice and conduct a rapid systematic review  5-minute presentation on the review question. research question and outline a few weeks before final deadline. rapid evidence assessment (max 4000 words), evaluated on a clear, answerable review question, sound justification for conducting the review, explicit search strategy, ways of judging the quality of the research, and conclusions that accurately reflect the findings.    faculty observations of student experience; student presentations had to answer question "what problems or pleasant surprises have you encountered so far?”  gorczynski, 2017, uk  exercise science, masters, not specified  faculty  students will learn to structure evidence based interventions and carry out valid and reliable evaluations    students identify an area of mental health and conduct a qualitative systematic review that examines the impact of physical activity on their chosen mental health topic. solve weekly case studies using new knowledge and lead discussions presenting their proposed interventions and supporting rationale.   quantitative and qualitative mid-year and year-end evaluations    groller, 2020, usa  nursing, undergrad, approx. 120 hrs  faculty, librarian  students will learn to design, conduct and disseminate results of a collaborative scoping review  individual paper reviewing about seven articles, determining suitability for answering research question, and then summarizing implications for clinical practice, policy, education and further research.   group oral presentation of research findings, open to campus community.  online survey on library session, with three open-ended questions. online post-course evaluation survey with 15 likert-scale  & 4 open-ended questions  himelhoch, 2015, usa  psychology, residents, 9 lectures  faculty  students will learn the fundamentals of systematic reviews and meta-analysis, learn to select a good research question, establish eligibility criteria, conduct a reproducible search, assess study quality, organize data and conduct meta-analysis, and present findings  eight assignments. 1) create a pico informed research question 2) define and describe eligibility criteria 3) conduct literature search and document results 4) interrater reliability assignment and prisma flow diagram 5) create risk-of-bias table and summary table for included papers 6) collect, organize, and document data to enable calculation of weighted effect size 7) present and interpret forest and funnel plots 8) write scientifically formatted manuscript ready for peer review.  anonymous course evaluation 38-questions on likert scale + 3 open-ended questions    land, 2020, uk  biomedical science, undergrad, 3 x 2hr classes, ongoing faculty consultation  faculty    students will develop the skills to conduct an independently researched systematic review and meta-analysis (srma) capstone project in their final year  a systematic review and meta-analysis, done as a proforma report  analysis of student performance across program to measure effectiveness of the systematic review exercise li, 2014, usa   public health, masters & phd, 6hr/wk x 8 weeks  faculty, librarians, teaching assistants  students will learn the steps of performing   systematic reviews and meta-analyses and improve their ability to perform, critically appraise, and teach others about systematic reviews    graded assignments include three open-book quizzes, individually submitted review protocol, and individually submitted final report on group's systematic review.  students orally present reviews to class and respond to comment.  anonymous evaluation before final paper. postcourse survey offered to students who took course 2004-2012; second survey sent to past participants on long-term effects of course. pieper, 2019, germany health economics, undergrad, 1.5 hrs x 14 or 15 weeks faculty students will learn the fundamentals of systematic review methodology and develop skills to critically appraise other systematic reviews students complete a 10-12 page systematic review based on topics selected by instructor and reported according to prisma guidelines students complete a validated post-course questionnaire to assess instructional quality   proly, 2009, usa    speech language pathology, masters & phd, not specified  faculty, librarian (as guest lecturer)  students will: develop understanding of intervention research design and clinical implications of evidence based practice, develop analytical skills to assess the quality of research evidence, gain project management skills. doctoral student will gain leadership experience.  major course assignment was development of a coding form and code-book specific to each group’s topic, research question and inclusion/exclusion criteria.   students also had an assignment requiring hand calculation of effect sizes. all students had to register their topic with the education coordinating group of the campbell collaboration.   not specified  upchurch, 2002, usa    nursing, masters, not specified  faculty  course 1: students will gain skills to examine the literature, maintain a bibliographic database, practice statistical analysis, select a problem area and type of data for a research project. course 2: students will complete the literature review or simple meta-analysis and prepare a written report.   students do a class presentation of their problem area, research question, background and significance. students design a coding sheet specific to their research question.  students write a research manuscript emphasizing their methods, findings and implications.    not specified      assessment outcomes, student/instructor feedback, and recommendations   designing an effective course or workshop series to teach knowledge synthesis methodology requires an understanding of which steps and discrete tasks students find challenging. instructors must incorporate appropriate time, content, and learning activities to address these challenges and develop assessments to evaluate whether these interventions are effective. all the articles in our study include some form of assessment, although some discuss assessment in far greater detail than others. challenges and benefits to teaching knowledge synthesis, student feedback, and instructor reflections from the 17 included articles are summarized below, categorized by the methodology step(s) to which they correspond.   question formulation and refinement   almost all of the included courses and workshops (16) teach question formulation or topic refinement, which often also includes setting inclusion criteria and limits. (see table 2)   determining and focusing the research question is an important first step in a knowledge synthesis project. a broad question may be feasible for a research team with many members working over an extended time period, but may be overwhelming for a small group of students completing a course project. some articles report that students found this step challenging, either due to the ambiguity and iterative nature of the question refinement process, or because of the difficulty in finding a question that is manageable and appropriate for a course assignment (briner & walshe, 2014; upchurch et al., 2002). in one article that describes two sequential research courses, graduate students initially pick a topic of interest, although they do not complete a knowledge synthesis project during the first course (upchurch et al., 2002). however, in the subsequent course the students take their previously-chosen topic and refine it into a question appropriate for research synthesis. upchurch et al. (2002) report that developing the final research question and clarifying inclusion criteria is an iterative process that students may find frustrating. the instructors built in extra time at the beginning of the course for students to refine their question. even when students know they are picking a topic for the purpose of conducting a small systematic review, the process of settling on an appropriate review question can still be challenging. briner and walshe (2014) emphasize this through a student’s quote, stating that they “really underestimated the difficulty of asking the right question ahead of formulating a search strategy” (p.426). they also mention that students were often frustrated by the lack of consistency in the way that concepts were defined in the literature, making it difficult to operationalize what seemed like a simple idea or concept. this further adds to the difficulty in settling on an appropriate research question.     table 5 teaching and learning strategies for knowledge synthesis steps knowledge synthesis step     teaching and learning strategies hands-on activities/ experiential learning small group work/ discussion student presentations case studies / guiding review guided / facilitated exercises individual work large group work/ discussion working in pairs/ pair activities reflection peer feedback/ evaluation role playing analyze seminal readings protocol development   15   3  15   7  9 3           defining the question 1  6  10  11 13 1  2  4  7  10 1  4  6 3  15 6 6 4     4     searching 1  2  3  4  6  7  8  9  10  11  13  14  15  17 2  3  4  7  10  13  14  15 6  13 3  16 4  11 13 3  6  13  16 7   6  13       screening inclusion / exclusion 1  2  3  4  6  7  17 1  2  3  4  7 15   3  16  17   15  16 15 6  17         citation/reference management 1 4   7 7                     data extraction 1  2  3  4  6  7  9  17 1  2  3  4  7  15   3  17 3  4 3  15  16 15 3         critical appraisal / risk of bias 2  4  7  10  17 2  4  7  10   16  17   16 9           synthesis           15             meta-analysis 1  7  10  17 1  7  10   17     7           manuscript draft/ completed review 10 2  7  10 2  5  6  7  10  15     1  7  15  16 2     1  5     reporting / data management 13  17 13   17     8           phase not specified   8   12 16 8 8 12     2 8 12   1 upchurch, 2002 5 bourke, 2013 9 flores-mir, 2015 13 lenton, 2019 17  pieper, 2020   2 azarpazhooh, 2008 6 briner, 2014 10 himelhoch, 2015 14 jack, 2020     3 baldassarre, 2008 7 li, 2014 11 campbell, 2016 15 groller, 2020     4 proly, 2009 8 conte, 2015 12 gorczynski, 2017 16 land, 2020               one strategy to simplify the process of research question development is for the course instructors to pick a list of topics that they know may be feasible for a course assignment. in one course, this was done effectively by using a set of topics or areas of focus that were important to stakeholders as a starting point from which to develop a relevant question (bourke & loveridge, 2013). in pieper et al. (2019) and land and booth (2020), students were either given a specific topic or selected from a carefully curated list of topics; topics were vetted by the instructor in order to ensure a manageable volume of results from the search. however, li et al. (2014) describe another situation where, despite best intentions and a clear set of criteria, some of the topics suggested each year “result in students’ searches that identify tens of thousands of titles and abstracts requiring screening or many more primary research articles meeting the students’ inclusion criteria” (p. 258). therefore, further intervention and guidance is required from the instructors on how to narrow a topic. however, selecting appropriate topics for students is a challenging task. pieper et al. (2019) discuss some criteria they felt would be appropriate when identifying suitable topics, such as a small number of search terms and synonyms, reasonable volume of search results, and so on.   several articles suggest highlighting the difficulty and importance of rigorous question formulation (briner & walshe, 2014; gorczynski et al., 2017; upchurch et al., 2002). furthermore, students learned the importance of the research question in determining the body of evidence (baldassarre et al., 2008) and of choosing the right question (briner & walshe, 2014).   protocol development   developing a protocol was either taught or assigned as an assessment of student learning in seven of the courses or workshops. in one class, determining inclusion/exclusion criteria, collecting search terms, and defining the data extraction criteria were assigned as homework , and in the following class students discussed their submissions (baldassarre et al., 2008). although not explicitly about protocol development, students in one course requested additional information and assistance with setting inclusion criteria which is one of the components that needs to be defined in a protocol (gorczynski et al., 2017). in li et al. (2014), creating the protocol was worth a significant portion of their final course grade, and students suggested that this be a group assignment rather than an individual assignment. protocol development as a group reflects the real-life experience of researchers when developing their review protocol as a team. in proly and murza (2009), the goal of the 15-week course was to submit a review title and protocol to the campbell collaboration.   searching for studies (data collection)   in all of the 17 included articles, searching for evidence was taught as part of the course or workshop.   searching for ks research must be comprehensive and exhaustive, and attempts must be made to gather all relevant evidence. for students conducting a ks project for the first time, this level of comprehensiveness in searching is likely new. ks course assignments may not require the level of exhaustive searching expected in a full ks review, however the level of comprehensiveness required is still likely greater than what students may be doing for other assignments. in faculty-led courses or workshops, librarians were sometimes invited to teach the search process; this was mentioned in six articles (briner & walshe, 2014; flores-mir et al., 2015; gorczynski et al., 2017; groller et al., 2020; li et al., 2014; proly & murza, 2009).   student feedback suggests that they recognized the importance, difficulty, or time-consuming nature of searching for evidence (baldassarre et al., 2008; briner & walshe, 2014; groller et al., 2020). they suggested that more time be allocated for learning how to search, and that additional guidance or handouts to aid with searching be included as part of the content (campbell et al., 2016; gorczynski et al., 2017; lenton & fuller, 2019). groller et al. (2020) report that the librarian provided additional, unplanned sessions with each group in order to meet the criteria set out in the pre-established search protocol. these consultations with librarian search experts were found to be beneficial. despite the challenges, students felt that their experiences in the courses led to improved abilities, skills, or confidence in gathering, searching, or locating evidence (azarpazhooh et al., 2008; conte et al., 2015; proly & murza, 2009). the course described by groller et al. (2020) included an evaluation of the library research session. student feedback highlighted learning about new databases, learning the different way that searches can be executed, and noting that the library skills learned would have been useful throughout their four years at university.   study selection and data extraction   study selection and data extraction were taught in 13 courses/workshops (see table 2). all 12 articles that describe credit-bearing courses covered both study selection and data extraction. study selection is required to arrive at a set of included studies from which data can be extracted and the evidence synthesized. one of the stand-alone workshops (jack et al., 2020) discussed the step of study selection, but did not address data extraction.   even though inclusion criteria are determined in the earlier stages of a ks review, further refinement to the criteria can sometimes occur during the study selection process. additionally, reading and interpreting academic literature are skills that are required within the study selection and data extraction steps of a knowledge synthesis project. reading, analyzing, and interpreting academic research were reported as challenging by students (briner & walshe, 2014; upchurch et al., 2002). students sometimes requested additional information or further assistance with the process of extracting data (gorczynski et al., 2017). both students and instructors suggested allocating more time for extracting data (gorczynski et al., 2017; li et al., 2014). briner and walshe (2014) state that the process of developing and applying criteria for inclusion/exclusion helps learners become active and critical consumers of information.   in pieper et al. (2019), students practiced data extraction by extracting data for one of the studies included in a previously published systematic review; students then checked their data extraction against the published systematic review, thus allowing students to verify the accuracy of their work. in another course, students participated in a pilot data extraction exercise in class to prepare them for the data extraction process (baldassarre et al., 2008). students had to independently extract data from one of two pre-selected papers, and then compared their results with another student who worked on the same paper. eventually, students received the instructor’s data extraction for final comparison. feedback on the guided exercise was positive, but “some students found it difficult to understand the meaning of the cells in the table” (p.422). this exercise highlights the value of piloting the data extraction process, but also demonstrates the challenges of the data extraction step. students also found it difficult to extract data from articles on unfamiliar topics (baldassarre et al., 2008). this underscores the value of having some familiarity with the topic for data extraction.   synthesis and critical appraisal   5 articles cover narrative synthesis, 9 articles explicitly mention meta-analysis, and 11 articles include the step of critical appraisal/risk of bias (see table 2).   analysis or synthesis were either noted as challenging tasks (upchurch et al., 2002) or mentioned as particularly time-consuming, with a suggestion that additional time be allocated to this step (li et al., 2014). however, students also felt that learning this step improved their ability to analyze, critically evaluate, or apply information (azarpazhooh et al., 2008; groller et al., 2020; proly & murza, 2009). they also became more critical of evidence (bourke & loveridge, 2013) or skeptical of research findings (briner & walshe, 2014). students were surprised by the limited quantity, quality, and relevance of the research they found. the synthesis and appraisal process thus allowed learners to develop an awareness of the variations in quality and relevance of existing research (briner & walshe, 2014). students improved their critical thinking skills and their ability to critique published systematic reviews (flores-mir et al., 2015). however, land and booth (2020) note that students “tend to gloss over the detail of forest plots to focus on the bottom-line result” or to focus on the basic interpretation of the funnel plots “without attempting a deeper analysis of the data” (p.283). suggestions and guidance for addressing these challenges are also provided in their article.   data management, documentation, and reporting   due to the volume of references or citations that need to be downloaded and managed, and the explicit requirement to report every aspect of the methods, data management, documentation and reporting are often taught as part of both stand-alone workshops and credit courses. all 17 articles include either data/citation management (10 articles), or documentation and reporting (15 articles), and nearly half included both (see table 2).   conte et al. (2015) suggest incorporating additional content on data management, reporting, and documentation. upchurch et al. (2002) suggest that learners should keep a procedure manual to document the research process. introducing different reference management software is also recommended (gorczynski et al., 2017). campbell et al. (2016) initially included a greater amount of time to cover reference management, but time constraints resulted in less coverage in a later iteration of the workshop. instead, instructions on reference management were provided via tutorials made available prior to the in-class workshop.   instructional design and teaching strategies   in addition to discussing challenges, feedback, suggestions, or recommendations related to course content, many articles discuss instructional design or course structure. azarpazhooh et al. (2008) mention that frequent, shorter sessions were preferred over a longer 3-hour session, however lenton and fuller (2019) state that students in their workshops preferred longer sessions in order to more fully cover the content. allowing more time for learning activities, hands-on practice, or group work is suggested in many articles (campbell et al., 2016; flores-mir et al., 2015; gorczynski et al., 2017; li et al., 2014).   there is no consensus on whether group or individual assignments are preferred, however many articles stress the value of students working collaboratively with peers or advisors throughout the review process. li et al. (2014) mention that assignments should be group rather than individual, whereas conte et al. (2015) suggest that the group project be changed to an individual assignment. pieper et al. (2019) included in-class activities completed in pairs or groups, however the course assignment was done individually. incorporating peer activities and regular feedback from instructors is also mentioned in the literature. upchurch et al. (2002) and himelhoch et al. (2015) write about the value of consulting with peers or faculty, while baldassarre et al. (2008) mention that group discussions with peers was motivation for students to complete their assigned tasks. land and booth (2020) encouraged students to share search strategies on a discussion board, which is another form of peer learning. pieper et al. (2019) incorporated frequent contact with the instructor during search development, and students were required to have their search approved by the instructor before continuing on to the next step.   there are a few other note-worthy recommendations. gorczynski et al. (2017) and upchurch et al. (2002) both suggest working with external experts such as methods or information experts. furthermore, both groller et al. (2020) and li et al. (2014) discuss the value of integrating an information literacy expert into the course. campbell et al. (2016) and gorczynski et al. (2017) mention providing or requiring readings in advance and providing more information in general. a structured stepwise approach to the content (himelhoch et al., 2015) and consistency and repetition (gorczynski et al., 2017) are discussed. the provision of examples for in-class activities or course assignments is also suggested (baldassarre et al., 2008; conte et al., 2015). pieper et al. (2019) describe their approach of using a “guiding systematic review” (p.3) which is an existing published systematic review. the authors suggest using a systematic review that is well-conducted and has high reporting quality. during class, students complete various tasks related to specific steps of a systematic review, and compare their results to those in the chosen “guiding systematic review.” the major benefit for this approach is the ability for students to reproduce some of the work and compare their work to the published results.   the use of active learning and hands-on practice is also mentioned both in general (conte et al., 2015; flores-mir et al., 2015; jack et al., 2020; lenton & fuller, 2019; pieper et al., 2019) and in regard to specific steps of the knowledge synthesis process. in terms of learner baseline knowledge, it is recommended that instructors assume learners have no working knowledge of the topic or only basic skills (campbell et al., 2016; gorczynski et al., 2017). a further suggestion is to cover students’ muddiest points from the previous session at the beginning of the next session so as to ensure that everyone is on the same page (lenton & fuller, 2019).   student engagement with the content is another theme in a number of articles. briner and walshe (2014) emphasize the importance of students choosing their own research questions for this reason. given the challenging nature of conducting a review, “it is more likely that students will stay motivated if they have chosen a topic that interests them” (briner & walshe, 2014, p. 425). gorczynski et al. (2017) suggest making “the experience fun and enjoyable by allowing students to lead seminars and bring in their own reviews” (p. 13). the capstone project in the workshop described by conte et al. (2015), included “a personalized action plan tailored to the unique needs, missions, organizational goals, and resources of the librarians’ home institutions” (p. 71). jack et al. (2020) required participants to be involved in a systematic review in order to participate in the workshop; this ensured meaningful engagement with the skills and concepts, as participants had to immediately apply them to an existing project. in addition to being a motivating factor, real-life projects also generate complexities and issues to be resolved, and these can provide additional learning that may not happen with a perfectly designed course assignment. this can be both a challenge and a benefit, as excessively complex issues may frustrate the learner in the moment, but in the right dose may lead to opportunities for deeper learning.   benefits of participation in a knowledge synthesis course/workshop   despite the complexity and challenge of teaching knowledge synthesis methodology, especially in a course setting, instructors and students alike found many benefits from the experience. students developed a greater appreciation of the importance of evidence based clinical practice (azarpazhooh et al., 2008) or increased knowledge of evidence based practice (flores-mir et al., 2015). in one course, for example, a student mentioned learning “the importance of balancing research with stakeholder opinions” (bourke & loveridge, 2013, p. 19).   students developed increased skills, confidence, or motivation (campbell et al., 2016; flores-mir et al., 2015; himelhoch et al., 2015; jack et al., 2020) or even feelings of empowerment (briner & walshe, 2014). developing communication skills (azarpazhooh et al., 2008), and “a new ability to incorporate the learned material into their classroom lectures or clinical bedside teaching” (flores-mir et al., 2015, p. 4) are also mentioned. interestingly, students also learned the importance of teamwork in the research process (groller et al., 2020).   another point commonly arising from the literature is that learners not only gained the skills to conduct knowledge synthesis reviews, they also became better and more critical consumers of research. bourke and loveridge (2013) provide a series of students quotes emphasizing this concept, including “because of this course i now not only look at the evidence supporting the research i read, but i also think about how that evidence was obtained” and “when i read about research in the media i wonder about the study’s methodological quality, how this might have influenced the results, and how the study compares to others” (p. 19). li et al. (2014) surveyed past students, discovering that one of the long-term impacts of the course was an increased ability to appraise reports of systematic reviews and other primary studies (p. 263). therefore, even those students who may not complete another knowledge synthesis review benefit from taking a ks course in terms of their development as student researchers.   discussion   the courses and workshops discussed in our scoping review are diverse in structure, learner population, academic discipline, and location, making them a challenge to synthesize. furthermore, some articles provide rich information and detailed descriptions, whereas others include very few specifics about the instructional content or teaching and learning strategies. therefore, it is not surprising that the recommendations emerging from the included articles are also diverse, regarding both content and instructional design choices. there are, however, some common themes that emerge.   several articles describe similar features regarding course or workshop design. ks methodology is complex, requiring an understanding of both the conceptual underpinnings of the process as well as the practical implementation. consequently, the articles we found frequently mention providing both prescriptive information on how specific steps of a knowledge synthesis review are conducted along with opportunities for hands-on learning and practice. recommendations to allocate additional time for specific steps of a review also arise several times in the literature. additionally, group-work is mentioned frequently. these suggestions touch on aspects that are common in the process of conducting a systematic review, meta-analysis, or other ks review; reviews should be conducted in teams, require a significant amount of time, and have many intricate steps that need to be followed (higgins et al., 2020).   most courses and workshops included in our review incorporated active learning, which led to student engagement. active learning, as defined by bonwell and eison (1991) in their seminal work, includes instructional activities which “involve students in doing things and thinking about what they are doing” (p. 2). student engagement is a “process and a product that is experienced on a continuum and results from the synergistic interaction between motivation and active learning” (barkley & major, 2020, p. 8). through the incorporation of various active learning activities, such as collaborative group activities, case studies, hands-on practical exercises, individual and group projects, and presentations, students experience “real world” knowledge synthesis research. students thereby develop a better understanding of the steps associated with the ks methodology, and, as with the students in the course described by jack et al. (2020), they may be motivated and confident to conduct their own ks project.    some longer-term benefits or outcomes of the instruction are also mentioned, including: interest in carrying out another systematic review (baldassarre et al., 2008); interest in future presentation, publication, or professional development (proly & murza, 2009); and subsequent publications, conference abstracts, or dissertation topics that resulted from the course assignments (briner & walshe, 2014; himelhoch et al., 2015; jack et al., 2020; li et al., 2014). the land and booth (2020) course was scheduled in the penultimate year of the program in order to provide students with the skills needed to conduct a systematic review/meta-analysis capstone project in the final year. case studies and student reflections have shown that students perceive their participation in systematic reviews as leading to their growth as student learners and researchers, and helps form their identities as academics (look et al., 2020; pickering et al., 2015).   implications for practice and future research   when designing a course or workshop on knowledge synthesis methods, many factors affect the choice of course structure, teaching and learning strategies, activities and assignments. in all cases, careful consideration of the baseline knowledge and skills of learners is necessary. for example, undergraduate students may find it challenging to read and understand the literature, so it may be necessary to schedule more time for extracting or analyzing data when teaching these learners than for a course for graduate students, post-docs or professionals. if there is not time in the course or workshop to teach the necessary foundational knowledge or skills (such as academic reading or basics of research study design, etc.), then instructors should provide a set of pre-course readings or tutorials and communicate explicit expectations to ensure students are able to adequately prepare prior to attending. pre-reading and pre-work may help to avoid spending unplanned course time addressing students’ lack of baseline knowledge. it may also prevent students who do have the requisite baseline knowledge from being disengaged while the instructor teaches basic concepts.   overwhelmingly, the articles in our scoping review advocate active learning and hands-on practice. skills such as searching, objectively applying inclusion/exclusion criteria, data extraction, assessing risk of bias, and others need to be practiced in order for learners to fully understand the messiness and complexity involved. the specific teaching approaches used in the included articles varied, which is similar to the approaches used to teach research methods in the social sciences. variability in teaching methods was highlighted in a review stating that “authors advocate a range of approaches: exercises, problem‐based learning, experiential learning, collaborative and group work, computer‐based learning, tutorials, workshops, simulations and projects” (wagner et al., 2011, p. 80). sufficient time should be allocated to allow students to actively participate and experience these knowledge synthesis steps. however, students also need to be taught the conceptual underpinnings of the various ks steps, and of the implications of the specific choices that they make, so that they understand the importance of resolving issues based on methodological principles.   the value of discussion, consultation, and regular feedback is also emphasized in the included literature. these elements should be built into the course to ensure students have a way to assess their own progress and to ask for help as they work on their reviews or assignments. while most stand-alone workshops or workshop series are short and may not allow time for this, explicitly stating that participants may reach out to workshop instructors in the future may serve a similar purpose. similar to feedback on a student’s assignment, these types of one-on-one consultations with a librarian search expert providing personalized and tailored guidance on a task associated with a student’s systematic review project should be encouraged.   given the increased focus on reproducibility in the scientific literature in general, and in knowledge synthesis in particular, instructors should ensure that students or participants explicitly learn the difference between expectations for a course assignment versus those for a publishable ks review. this ensures that concessions made for course assignment feasibility are not replicated when students later work on a ks review for submission to a journal. one way to help students appreciate the full extent of work required for a publishable review is to expose them to relevant methodological conduct guidelines from evidence synthesis organizations in their disciplinary areas (higgins et al., 2020; peters et al., 2017) which will demonstrate the expectations with regard to comprehensiveness, rigor, and adherence to a strict methodological approach that are required to produce a high-quality knowledge synthesis review. furthermore, if a complete review protocol or review manuscript is an assignment in the course, instructors could require students to submit a completed prisma (moher et al., 2009; moher et al., 2015; tricco et al., 2018), or other reporting checklist along with their submission to reinforce reporting expectations. such an assignment would encourage best practices, and may contribute to a reduction in the poor reporting currently being observed in published reviews (bassani et al., 2019; page et al., 2016).   additionally, there is a potential for greater librarian involvement in the teaching of knowledge synthesis. librarians have reported involvement in all steps of the systematic review process, as well as other roles such as peer review, evaluation, and teaching (spencer & eldredge, 2018). the positive impact of librarian involvement on the quality of the review has also been reported (meert et al., 2016; rethlefsen et al., 2015). furthermore, librarians trained in ks methods have both methodological expertise and information science expertise and are therefore ideal collaborators, co-instructors, or guest lecturers for faculty-led ks courses (wissinger, 2018). li et al. (2014) refer to “the experience and engagement” of informationists as being “a key contributor to the success of the course” and further state that librarians “contribute by lecturing, advising, modeling the benefit of collaborating with experts, and signing off on search strategies” (p.261).   as search experts, librarians should be teaching the searching section of a ks course (mcgowan & sampson, 2005). librarian involvement is mentioned in 5 of the 11 credit course articles, though it is possible that librarians were involved in the other courses but not explicitly mentioned. of these five articles, four were about ks courses in medical and allied health disciplines including public health, speech language pathology, nursing, and exercise science. this comes as no surprise as librarians in health-related disciplines have an established role in ks, and guidance documents recommend working with a librarian to develop and implement the search for evidence (lefebvre et al., 2020). in other disciplines, however, this may still be an emerging role and is therefore an area of potential growth for librarians working in these fields.   further research focused on teaching knowledge synthesis methods is required. despite the growth of knowledge synthesis reviews published in the academic literature, there is a very limited number of articles published on how the methodology is taught in higher education settings. we found only 17 articles that met our inclusion criteria. more program descriptions and evaluation studies are needed that show what content is covered, how the content is taught, and which instructional strategies are successful for teaching the various steps of knowledge syntheses methods.   strengths and limitations   strengths of our study include the following. we utilized a robust methodology used to identify and synthesize the literature on this topic. we searched 12 databases, including both subject-specific and multidisciplinary ones. we also hand searched related journals’ tables of contents and conference proceedings. prior to screening, we piloted our inclusion/exclusion criteria on 50 random titles/abstracts. further, we contacted authors to verify information and course descriptions when necessary.   however, despite our rigorous methodology and exhaustive search strategy, it is possible that we missed potentially relevant articles. one limitation of our review is that we only included articles that were published in english. additionally, the selection of conferences to hand-search were limited to known, major health or evidence based practice related librarian conferences in north america and europe. our choices were based on our existing knowledge of the types of conferences more likely to contain presentations about systematic review workshops. however, knowledge synthesis is not limited to a specific discipline or geographic area, and it was not feasible to search conferences across all disciplines and across all regions, so this is a further limitation to our study. another limitation is that we used adjacency for searching the two concepts near each other, which affects the sensitivity of the search strategy. however, a balanced approach was required, in order to prevent retrieving every knowledge synthesis review about teaching and learning, and to ensure the process/scope would be feasible. to overcome this, as noted above, we searched a wide variety of databases from different disciplines and supplemented the database searches with supplementary searching.   not all of the included articles provide a high level of detail about the learning objectives/outcomes, specific in-class activities and assessments, teaching strategies, or outcomes with the published article. therefore, it is possible that some courses included more than is reported in our scoping review.   conclusion   our scoping review aimed to summarize the extent of the literature on courses and workshops that teach ks methods. the 17 included articles helped further our understanding of what content is taught, which instructional methods are commonly used, and what outcomes are achieved through teaching ks. common elements that arose from the literature include a focus on active learning and group work and an increase in participants’ skills and interest in conducting systematic reviews. our review results were limited because we only included articles that contain an evaluative or reflective component rather than ones that simply describe the programs. however, the assessment component is what truly adds value to this review as it allowed us to determine what parts of the process students find challenging and what the learners actually gain through instruction. while identifying what is currently known about the teaching of ks in higher education environments, as well as highlighting the lack of articles that provide specific information on teaching and learning strategies used, our review may inform the development and improvement of courses or workshop teaching knowledge synthesis methodology.   author contributions   zahra premji: conceptualization, funding acquisition, investigation, methodology, formal analysis, project administration, supervision, visualization, writing – original draft, writing – review & editing k. alix hayden: conceptualization, funding acquisition, investigation, formal analysis, visualization, writing original draft, writing review & editing shauna rutherford: investigation, formal analysis, visualization, writing original draft, writing review & editing   acknowledgments   the authors would like to acknowledge funding from a development and innovation, teaching and learning grant funded by the provost’s office, university of calgary. we would also like to thank dr justine wheeler, assessment, social work and sociology librarian, libraries & cultural resources, university of calgary and robin parker, evidence synthesis librarian, w.k. kellogg health sciences library, dalhousie university for reviewing and providing suggestions and feedback on our manuscript.   references   azarpazhooh, a., mayhall, j. t., & leake, j. l. (2008). introducing dental students to evidence-based decisions in dental care. journal of dental education, 72(1), 87-109. https://doi.org/10.1002/j.0022-0337.2008.72.1.tb04457.x   baldassarre, m. t., boffoli, n., caivano, d., & visaggio, g. (2008). a hands-on approach for teaching systematic review. in product-focused software process improvement, proceedings (vol. 5089, pp. 415-426). springer-verlag berlin.   barkley, e. f., & major, c. h. (2020). student engagement techniques: a handbook for college faculty. john wiley & sons.   bassani, r., pereira, g. k. r., page, m. j., tricco, a. c., moher, d., & sarkis-onofre, r. (2019). systematic reviews in dentistry: current status, epidemiological and reporting characteristics. journal of dentistry, 82, 71-84. https://doi.org/10.1016/j.jdent.2019.01.014   bonwell, c. c., & eison, j. a. (1991). active learning: creating excitement in the classroom. 1991 ashe-eric higher education reports: eric.   bourke, r., & loveridge, j. (2013). a scientist-practitioner model for inclusive education: supporting graduate students to conduct systematic reviews for evidence-based practice. new zealand journal of teachers' work, 10(1), 4-23.   briner, r. b., & walshe, n. d. (2014). from passively received wisdom to actively constructed knowledge: teaching systematic review skills as a foundation of evidence-based management. academy of management learning & education, 13(3), 415-432. https://doi.org/10.5465/amle.2013.0222   campbell collaboration (n.d.). training. https://campbellcollaboration.org/research-resources/training-courses.html   campbell, s. m., kung, j. y. c., & dennett, l. (2016). a curriculum for an introductory systematic review searching workshop for researchers. the journal of the canadian health libraries association = journal de l'association des bibliothèques de la santé du canada, 37(1), 2. https://doi.org/10.5596/c16-003   chandler, j., cumpston, m., thomas, j., higgins, j. p. t., deeks, j. j., & clarke, m. j. (2020). chapter 1: introduction. in j. p. t. higgins, j. thomas, j. chandler, m. cumpston, t. li, m. j. page, & v. a. welch (eds.), cochrane handbook for systematic reviews of interventions (vol. 6.1). https://training.cochrane.org/handbook   chandler, j., & hopewell, s. (2013). cochrane methods-twenty years experience in developing systematic review methods. systematic reviews, 2(1), 76. https://doi.org/10.1186/2046-4053-2-76   cochrane. (n.d.). about cochrane training. https://training.cochrane.org/about-cochrane-training   cochrane. (2020). evidence synthesis what is it and why do we need it? https://www.cochrane.org/news/evidence-synthesis-what-it-and-why-do-we-need-it   conte, m. l., maceachern, m. p., mani, n. s., townsend, w. a., smith, j. e., masters, c., & kelley, c. (2015). flipping the classroom to teach systematic reviews: the development of a continuing education course for librarians. journal of the medical library association, 103(2), 69-73. https://dx.doi.org/10.3163%2f1536-5050.103.2.002   covidence. (n.d.). melbourne, australia. https://www.covidence.org/   flores-mir, c., pacheco-pereira, c., de luca canto, g., elyasi, m., & saltaji, h. (2015). oral health research methods summer institute: a systematic review methodology workshop. journal of the canadian dental association, 81, f17. https://jcda.ca/article/f17   gorczynski, p., burnell, k., dewey, a., & costello, j. t. (2017). teaching evidence-based synthesis: an examination of the development and delivery of two innovative methodologies used at the university of portsmouth. journal of evidence-based medicine, 10(1), 11-15. https://doi.org/10.1111/jebm.12241   grimshaw, j. (2010). a knowledge synthesis chapter. https://cihr-irsc.gc.ca/e/documents/knowledge_synthesis_chapter_e.pdf   groller, k. d., adamshick, p., & petre, k. (2020). embracing evidence-based nursing and informational literacy through an innovative undergraduate collaborative project. international journal of nursing education scholarship, 18(1), 1-9. https://doi.org/10.1515/ijnes-2019-0138   higgins, j. p., thomas, j., chandler, j., cumpston, m., li, t., page, m. j., & welch, v. a. (2020). cochrane handbook for systematic reviews of interventions [6.1]. https://training.cochrane.org/handbook   himelhoch, s., edwards, s., ehrenreich, m., & luber, m. p. (2015). teaching lifelong research skills in residency: implementation and outcome of a systematic review and meta-analysis course. journal of graduate medical education, 7(3), 445-450. https://doi.org/10.4300/jgme-d-14-00505.1   jack, h. e., merritt, c., medhin, g., musesengwa, r., mafuta, c., gibson, l. j., hanlon, c., sorsdahl, k., chibanda, d., & abas, m. (2020). developing sustainable capacity-building in mental health research: implementation outcomes of training of trainers in systematic reviewing. global health action, 13(1), 1715325. https://doi.org/10.1080/16549716.2020.1715325   joanna briggs institute. (n.d.). education: short courses. https://joannabriggs.org/education   land, s. c., & booth, d. (2020). systematic review and meta-analysis as a structured platform for teaching principles of experimentation. advances in physiology education, 44(3), 276-285. https://doi.org/10.1152/advan.00131.2019   lasserson, t. j., thomas, j., & higgins, j. p. (2020). chapter 1: starting a review. in j. p. t. higgins, j. thomas, j. chandler, m. cumpston, t. li, m. j. page, & v. a. welch (eds.), cochrane handbook for systematic reviews of interventions (pp. 1-12).  http://www.training.cochrane.org/handbook   lefebvre, c., glanville, j., briscoe, s., littlewood, a., marshall, c., metzendorf, m. i., . . . thomas, j. (2020). chapter 4: searching for and selecting studies. in j. p. t. higgins, j. thomas, j. chandler, m. cumpston, t. li, m. j. page, & v. a. welch (eds.), cochrane handbook for systematic reviews of interventions (pp. 67-107). https://training.cochrane.org/handbook   lenton, e., & fuller, k. (2019). explaining the method behind our madness: 3-part series on comprehensive searches for knowledge syntheses. journal of the canadian health libraries association / journal de l'association des bibliothèques de la santé du canada, 40(1), 18-22. https://doi.org/10.29173/jchla29391   li, t., saldanha, i. j., vedula, s. s., yu, t., rosman, l., twose, c., goodman, s. n., & dickersin, k. (2014). learning by doing-teaching systematic review methods in 8 weeks. research synthesis methods, 5(3), 254-263. https://doi.org/10.1002/jrsm.1111   look, r., shoemaker, h., hoepner jerry, k., & blake margaret, l. (2020). reciprocal benefits of engaging undergraduate researchers in conducting a systematic literature review. perspectives of the asha special interest groups, 5(3), 699-709. https://doi.org/10.1044/2019_persp-19-00072   mcgowan, j., & sampson, m. (2005). systematic reviews need systematic searchers. journal of the medical library association, 93(1), 74-80.   meert, d., torabi, n., & costella, j. (2016). impact of librarians on reporting of the literature searching component of pediatric systematic reviews. journal of the medical library association, 104(4), 267. https://doi.org/10.3163/1536-5050.104.4.004   moher, d., liberati, a., tetzlaff, j., altman, d. g., & the prisma group. (2009). preferred reporting items for systematic reviews and meta-analyses: the prisma statement. plos medicine, 6(7), e1000097. https://doi.org/10.1371/journal.pmed.1000097   moher, d., shamseer, l., clarke, m., ghersi, d., liberati, a., petticrew, m., shekelle, p. stewart, l. a. (2015). preferred reporting items for systematic review and meta-analysis protocols (prisma-p) 2015 statement. systematic reviews, 4(1), 1. https://doi.org/10.1186/2046-4053-4-1   munn, z., peters, m. d., stern, c., tufanaru, c., mcarthur, a., & aromataris, e. (2018). systematic review or scoping review? guidance for authors when choosing between a systematic or scoping review approach. bmc medical research methodology, 18(1), 143. https://doi.org/10.1186/s12874-018-0611-x   page, m. j., & moher, d. (2016). mass production of systematic reviews and meta‐analyses: an exercise in mega‐silliness? the milbank quarterly, 94(3), 515. https://doi.org/10.1111/1468-0009.12211   page, m. j., shamseer, l., altman, d. g., tetzlaff, j., sampson, m., tricco, a. c., catalá-lópez, f., li, l., reid, e. k., sarkis-onofre, r., & moher, d. (2016). epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. plos medicine, 13(5), e1002028. https://doi.org/10.1371/journal.pmed.1002028   parker, r. m., boulos, l. m., visintini, s., ritchie, k., & hayden, j. (2018). environmental scan and evaluation of best practices for online systematic review resources. journal of the medical library association, 106(2), 208. https://doi.org/10.5195/jmla.2018.241   peters, m. d., godfrey, c., mcinerney, p., baldini soares, c., khalil, h., & parker, d. (2017). chapter 11: scoping reviews. joanna briggs institute reviewer's manual.   peters, m. d., marnie, c., tricco, a. c., pollock, d., munn, z., alexander, l., mcinerney, p., godfrey, c. m., & khalil, h. (2020). updated methodological guidance for the conduct of scoping reviews. jbi evidence synthesis, 18(10), 2119-2126. https://doi.org/10.11124/jbies-20-00167   pickering, c., grignon, j., steven, r., guitart, d., & byrne, j. (2015). publishing not perishing: how research students transition from novice to knowledgeable using systematic quantitative literature reviews. studies in higher education, 40(10), 1756-1769. https://doi.org/10.1080/03075079.2014.914907   pieper, d., muller, d., & stock, s. (2019). challenges in teaching systematic reviews to non-clinicians. zeitschrift fur evidenz, fortbildung und qualitat im gesundheitswesen, 147-148(101477604), 1-6. https://dx.doi.org/10.1016/j.zefq.2019.10.004   proly, j. l., & murza, k. a. (2009). building speech-language pathologist capacity for evidence-based practice: a unique graduate course approach. evidence-based communication assessment and intervention, 3(4), 220-231. https://doi.org/10.1080/17489530903432383   pussegoda, k., turner, l., garritty, c., mayhew, a., skidmore, b., stevens, a., boutron, i., sarkis-onofre, r., bjerre, l. m., hróbjartsson, a., altman, d. g., & moher, d. (2017). systematic review adherence to methodological or reporting quality. systematic reviews, 6(1), 1-14. https://doi.org/10.1186/s13643-017-0527-2   rethlefsen, m. l., farrell, a. m., trzasko, l. c. o., & brigham, t. j. (2015). librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. journal of clinical epidemiology, 68(6), 617-626. https://doi.org/10.1016/j.jclinepi.2014.11.025   spencer, a. j., & eldredge, j. d. (2018). roles for librarians in systematic reviews: a scoping review. journal of the medical library association, 106(1), 46. https://doi.org/10.5195/jmla.2018.82   sutton, a., clowes, m., preston, l., & booth, a. (2019). meeting the review family: exploring review types and associated information retrieval requirements. health information & libraries journal, 36(3), 202-222. https://doi.org/10.1111/hir.12276   tricco, a. c., lillie, e., zarin, w., o’brien, k. k., colquhoun, h., levac, d., moher, d., peters, m. d. j., horsley, t., weeks, l., hempel, s., akl, e. a., chang, c., mcgowan, j., stewart, l., hartling, l., aldcroft, a., wilson, m. g., garritty, c., … straus, s. e. (2018). prisma extension for scoping reviews (prisma-scr): checklist and explanation. annals of internal medicine, 169(7), 467-473. https://doi.org/10.7326/m18-0850   upchurch, s., brosnan, c. a., & grimes, d. e. (2002). teaching research synthesis to advanced practice nurses. the journal of nursing education, 41(5), 222-226. https://doi.org/10.3928/0148-4834-20020501-08 wagner, c., garner, m., & kawulich, b. (2011). the state of the art of teaching research methods in the social sciences: towards a pedagogical culture. studies in higher education, 36(1), 75-88. https://doi.org/10.1080/03075070903452594   wissinger, c. l. (2018). is there a place for undergraduate and graduate students in the systematic review process? journal of the medical library association, 106(2), 248-250. https://doi.org/10.5195/jmla.2018.387     appendix electronic database search strategies   database(s): ovid medline(r) and epub ahead of print, in-process & other non-indexed citations and daily 1946 to september 11, 2020 # searches 1 meta-analysis as topic/ 2 "review literature as topic"/ 3 systematic reviews as topic/ 4 1 or 2 or 3 5 teaching/mt or exp education, continuing/mt or exp education, graduate/mt 6 4 and 5 7 (("systematic review*" or "meta-analys?s" or "knowledge synthes?s" or "evidence synthes?s" or "research synthes?s" or "metaanalys?s" or "rapid review*" or "realist review*" or "realist synthes?s" or "integrative review*" or "umbrella review*" or "review* of review*" or "scoping review*") adj5 (teach or teaching or course or courses or workshop* or instruct* or lecture* or tutorial* or curriculum or curricula or curricular)).tw,kf. 8 (("meta-synthes?s" or "metasynthes?s" or "synthes?s method*" or "critical interpretative synthes?s" or "meta-ethnograph*" or "meta-study" or "meta-summar*" or "narrative synthes?s" or "qualitative synthes?s" or "mixed method* synthes?s" or "multilevel synthes?s" or "network review*" or "health technolog* assessment*" or "network meta*" or "meta* review*") adj5 (teach or teaching or course or courses or workshop* or instruct* or lecture* or tutorial* or curriculum or curricula or curricular)).tw,kf. 9 6 or 7 or 8   database(s): embase 1974 to 2020 september 11 (ovid) # searches 1 "meta analysis (topic)"/ 2 "systematic review (topic)"/ 3 1 or 2 4 *medical education/ 5 *continuing education/ 6 *nursing education/ 7 *teaching/ 8 4 or 5 or 6 or 7 9 8 and 3 10 (("systematic review*" or "meta-analys?s" or "knowledge synthes?s" or "evidence synthes?s" or "research synthes?s" or "metaanalys?s" or "rapid review*" or "realist review*" or "realist synthes?s" or "integrative review*" or "umbrella review*" or "review* of review*" or "scoping review*") adj5 (teach or teaching or course or courses or workshop* or instruct* or lecture* or tutorial* or curriculum or curricula or curricular)).tw,kw. 11 (("meta-synthes?s" or "metasynthes?s" or "synthes?s method*" or "critical interpretative synthes?s" or "meta-ethnograph*" or "meta-study" or "meta-summar*" or "narrative synthes?s" or "qualitative synthes?s" or "mixed method* synthes?s" or "multilevel synthes?s" or "network review*" or "health technolog* assessment*" or "network meta*" or "meta* review*") adj5 (teach or teaching or course or courses or workshop* or instruct* or lecture* or tutorial* or curriculum or curricula or curricular)).tw,kw. 12 9 or 10 or 11 13 limit 12 to (editorial or letter or note or tombstone) 14 12 not 13   database(s): apa psycinfo 1806 to september week 1 2020 (ovid) # searches 1 meta analysis/ 2 "literature review"/ 3 1 or 2 4 *medical education/ 5 *continuing education/ 6 *nursing education/ 7 *graduate psychology education/ or *psychology education/ or *education/ or *health education/ or *graduate education/ 8 4 or 5 or 6 or 7 9 8 and 3 10 (("systematic review*" or "meta-analys?s" or "knowledge synthes?s" or "evidence synthes?s" or "research synthes?s" or "metaanalys?s" or "rapid review*" or "realist review*" or "realist synthes?s" or "integrative review*" or "umbrella review*" or "review* of review*" or "scoping review*") adj5 (teach or teaching or course or courses or workshop* or instruct* or lecture* or tutorial* or curriculum or curricula or curricular)).tw,hw. 11 (("meta-synthes?s" or "metasynthes?s" or "synthes?s method*" or "critical interpretative synthes?s" or "meta-ethnograph*" or "meta-study" or "meta-summar*" or "narrative synthes?s" or "qualitative synthes?s" or "mixed method* synthes?s" or "multilevel synthes?s" or "network review*" or "health technolog* assessment*" or "network meta*" or "meta* review*") adj5 (teach or teaching or course or courses or workshop* or instruct* or lecture* or tutorial* or curriculum or curricula or curricular)).tw,hw. 12 9 or 10 or 11 13 limit 12 to ("column/opinion" or "comment/reply" or editorial or letter or obituary or poetry or review-book or review-media or review-software & other) 14 12 not 13   academic search complete, business source complete, cinahl plus with full-text, eric, education research complete, library & information science source, and socindex (via ebsco) – searched together. # query last run via s1 ti ( (("systematic review*" or "meta-analys?s" or "knowledge synthes?s" or "evidence synthes?s" or "research synthes?s" or "metaanalys?s" or "rapid review*" or "realist review*" or “realist synthes?s” or "integrative review*" or "umbrella review*" or “review* of review*” or “scoping review*”) n5 (teach or teaching or course or courses or workshop* or instruct* or lecture* or tutorial* or curriculum or curricula or curricular)) ) or ab ( (("systematic review*" or "meta-analys?s" or "knowledge synthes?s" or "evidence synthes?s" or "research synthes?s" or "metaanalys?s" or "rapid review*" or "realist review*" or “realist synthes?s” or "integrative review*" or "umbrella review*" or “review* of review*” or “scoping review*”) n5 (teach or teaching or course or courses or workshop* or instruct* or lecture* or tutorial* or curriculum or curricula or curricular)) ) interface  ebscohost database  academic search complete; business source complete; cinahl plus with full text; library & information science source; socindex with full text; eric; education research complete s2 ti ( ((“meta-synthes?s” or “metasynthes?s” or “synthes?s method*” or “critical interpretative synthes?s” or “meta-ethnograph*” or “meta-study” or “meta-summar*” or “narrative synthes?s” or “qualitative synthes?s” or “mixed method* synthes?s” or “multilevel synthes?s” or “network review*” or “health technolog* assessment*” or “network meta*” or “meta* review*”) n5 (teach or teaching or course or courses or workshop* or instruct* or lecture* or tutorial* or curriculum or curricula or curricular)) ) or ab ( ((“meta-synthes?s” or “metasynthes?s” or “synthes?s method*” or “critical interpretative synthes?s” or “meta-ethnograph*” or “meta-study” or “meta-summar*” or “narrative synthes?s” or “qualitative synthes?s” or “mixed method* synthes?s” or “multilevel synthes?s” or “network review*” or “health technolog* assessment*” or “network meta*” or “meta* review*”) n5 (teach or teaching or course or courses or workshop* or instruct* or lecture* or tutorial* or curriculum or curricula or curricular)) ) interface  ebscohost database  academic search complete; business source complete; cinahl plus with full text; library & information science source; socindex with full text; eric; education research complete s3 s1 or s2     web of science core collection (which includes: sci-expanded, ssci, a&hci, cpci-s, cpci-ssh, esci.) # search 1 topic: ((((("systematic review*" or "meta-analys?s" or "knowledge synthes?s" or "evidence synthes?s" or "research synthes?s" or "metaanalys?s" or "rapid review*" or "realist review*" or “realist synthes?s” or "integrative review*" or "umbrella review*" or “review* of review*” or “scoping review*”) near/5 (teach or teaching or course or courses or workshop* or instruct* or lecture* or tutorial* or curriculum or curricula or curricular))))) 2 topic: (((((“meta-synthes?s” or “metasynthes?s” or “synthes?s method*”  or “critical interpretative synthes?s”  or “meta-ethnograph*”  or “meta-study”  or “meta-summar*”  or “narrative synthes?s”  or “qualitative synthes?s”  or “mixed method* synthes?s”  or “multilevel synthes?s”  or “network review*”  or “health technolog* assessment*”  or “network meta*” or “meta* review*”) near/5 (teach or teaching or course or courses or workshop* or instruct* or lecture* or tutorial* or curriculum or curricula or curricular))))) 3 1 or 2   library & information science abstracts (via proquest) # search s1 ti(((((“meta-synthes?s” or “metasynthes?s” or “synthes?s method*” or “critical interpretative synthes?s” or “meta-ethnograph*” or “meta-study” or “meta-summar*” or “narrative synthes?s” or “qualitative synthes?s” or “mixed method* synthes?s” or “multilevel synthes?s” or “network review*” or “health technolog* assessment*” or “network meta*” or “meta* review*”) near/5 (teach or teaching or course or courses or workshop* or instruct* or lecture* or tutorial* or curriculum or curricula or curricular))))) or ab(((((“meta-synthes?s” or “metasynthes?s” or “synthes?s method*” or “critical interpretative synthes?s” or “meta-ethnograph*” or “meta-study” or “meta-summar*” or “narrative synthes?s” or “qualitative synthes?s” or “mixed method* synthes?s” or “multilevel synthes?s” or “network review*” or “health technolog* assessment*” or “network meta*” or “meta* review*”) near/5 (teach or teaching or course or courses or workshop* or instruct* or lecture* or tutorial* or curriculum or curricula or curricular))))) s2 ti((((("systematic review*" or "meta-analys?s" or "knowledge synthes?s" or "evidence synthes?s" or "research synthes?s" or "metaanalys?s" or "rapid review*" or "realist review*" or “realist synthes?s” or "integrative review*" or "umbrella review*" or “review* of review*” or “scoping review*”) near/5 (teach or teaching or course or courses or workshop* or instruct* or lecture* or tutorial* or curriculum or curricula or curricular))) )) or ab((((("systematic review*" or "meta-analys?s" or "knowledge synthes?s" or "evidence synthes?s" or "research synthes?s" or "metaanalys?s" or "rapid review*" or "realist review*" or “realist synthes?s” or "integrative review*" or "umbrella review*" or “review* of review*” or “scoping review*”) near/5 (teach or teaching or course or courses or workshop* or instruct* or lecture* or tutorial* or curriculum or curricula or curricular))) )) s3 s1 or s2       evidence based library and information practice evidence based library and information practice 2011, 6.1 71 evidence based library and information practice evidence summary statistical measures alone cannot determine which database (bni, cinahl, medline, or embase) is the most useful for searching undergraduate nursing topics a review of: stokes, p., foster, a., & urquhart, c. (2009). beyond relevance and recall: testing new user-centred measures of database performance. health information and libraries journal, 26(3), 220-231. reviewed by: giovanna badia librarian, royal victoria hospital medical library mcgill university health centre montreal, quebec, canada email: giovanna.badia@mail.mcgill.ca received: 15 dec. 2010 accepted: 23 feb. 2011 2011 badia. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – the research project sought to determine which of four databases was the most useful for searching undergraduate nursing topics. design – comparative database evaluation. setting – nursing and midwifery students at homerton school of health studies (now part of anglia ruskin university), cambridge, united kingdom, in 2005-2006. subjects – the subjects were four databases: british nursing index (bni), cinahl, medline, and embase). methods – this was a comparative study using title searches to compare bni (british nursing index), cinahl, medline and embase. according to the authors, this is the first study to compare bni with other databases. bni is a database produced by british libraries that indexes the nursing and midwifery literature. it covers over 240 british journals, and includes references to articles from health sciences journals that are relevant to nurses and midwives (british nursing index, n.d.). the researchers performed keyword searches in the title field of the four databases for the dissertation topics of nine nursing and midwifery students enrolled in undergraduate dissertation modules. the list of titles of mailto:giovanna.badia@mail.mcgill.ca� evidence based library and information practice 2011, 6.1 72 journals articles on their topics were given to the students and they were asked to judge the relevancy of the citations. the title searches were evaluated in each of the databases using the following criteria: • precision (the number of relevant results obtained in the database for a search topic, divided by the total number of results obtained in the database search); • recall (the number of relevant results obtained in the database for a search topic, divided by the total number of relevant results obtained on that topic from all four database searches); • novelty (the number of relevant results that were unique in the database search, which was calculated as a percentage of the total number of relevant results found in the database); • originality (the number of unique relevant results obtained in the database for a search topic, which was calculated as a percentage of the total number of unique results found in all four database searches); • availability (the number of relevant full text articles obtained from the database search results, which was calculated as a percentage of the total number of relevant results found in the database); • retrievability (the number of relevant full text articles obtained from the database search results, which was calculated as a percentage of the total number of relevant full text articles found from all four database searches); • effectiveness (the probable odds that a database will obtain relevant search results); • efficiency (the probable odds that a database will obtain both unique and relevant search results); and • accessibility (the probable odds that the full text of the relevant references obtained from the database search are available electronically or in print via the user’s library). students decided whether the search results were relevant to their topic by using a “yes/no” scale. only record titles were used to make relevancy judgments. main results – friedman’s test and odds ratios were used to compare the performance of bni, cinahl, medline, and embase when searching for information about nursing topics. these two statistical measures demonstrated the following: • bni had the best average score for the precision, availability, effectiveness, and accessibility of search results; • cinahl scored the highest for the novelty, retrievability, and efficiency of results, and ranked second place for all the other criteria; • medline excelled in the areas of recall and originality, and ranked second place for novelty and retrievability; and • embase did not obtain the highest, or second highest score, for any of the criteria. conclusion – according to the authors, these results suggest that none of the databases studied can be considered the most useful for searching undergraduate nursing topics. cinahl and medline emerge as consistently good performers, but both databases are needed to find relevant material on a topic. friedman’s test clearly differentiated between the databases for the accessibility of search results. odds ratio testing may assist librarians to make decisions about database purchases. bni scored the highest for availability of results and cinahl ranked the highest for retrievability. statistical measures need to be supplemented with qualitative data about user preferences in order to determine which database is the most useful to our users. evidence based library and information practice 2011, 6.1 73 commentary this study contributed to the existing literature in that it was the first study to compare bni, cinahl, medline, and embase, and the first one to combine the novelty, originality, availability, and retrievability of search results with the traditional testing criteria of precision and recall to compare database performance. its findings confirmed what is already known, i.e., “that searching a single database is likely to miss relevant articles, and that some databases may be in general good performers” (pp. 230). the statistical measures used for comparative database evaluation, i.e., friedman’s test and odds ratios, could not determine which database was the most useful. this reviewer questions whether odds ratio was an appropriate statistical test to compare bni, cinahl, medline, and embase, since the authors state that “the odds ratio is comparing each database individually against the pool of data; it does not compare the four databases with each other” (pp. 229-230). the authors also suggest that odds ratio may assist in the selection of databases to purchase, but do not explain how. the study would have benefited from including a brief description of friedman’s test and odds ratios, as well as an explanation of how the data from both tests were combined. a table with the raw data from the searches could also have included in the article itself. unfortunately, the appendix containing the search data is no longer available on the publisher’s website. the missing appendix and the lack of sufficient explanatory details in the article make it difficult to replicate, or completely understand, the study’s research methodology. it is also difficult to generalize the study’s findings due to: its small sample size (i.e., nine students’ topics); the use of keyword searching in the title field to obtain relevant results, which may not be a user’s typical searching behaviour; and the use of database testing criteria that are dependent on an individual library’s subscriptions rather than on database search performance (i.e., the use of the availability, retrievability, and accessibility criteria). despite its weaknesses, this study reminds librarians that precision and recall are not the only criteria that should be used to measure the performance of a database. references british nursing index (n.d.). about bni. retrieved 20 feb. 2011 from http://www.bniplus.co.uk/about_bni.html / evidence based library and information practice evidence based library and information practice evidence based library and information practice 2010, 5.2  evidence based library and information practice       evidence summary     an awareness of library patrons’ social styles may play a role in librarians’  satisfaction with the reference interview    a review of:  sisselman, p. (2009). exploiting the social style of patrons to improve their satisfaction with the  reference interview. library review, 58, 123‐133.     reviewed by:  cari merkley  librarian, mount royal university  calgary, alberta, canada  email: cmerkley@mtroyal.ca     received: 1 mar. 2010           accepted: 16 apr. 2010       2010 merkley. this is an open access article distributed under the terms of the creative commons‐ attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐ sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original  work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is  redistributed under the same or similar license to this one.    abstract   objective – to determine if a patron’s social  style affects how satisfied the patron and the  librarian working with him/her are with the  reference interview, and if a librarian’s  knowledge of social styles leads to greater  satisfaction with the reference interview for all  involved.    design – direct observation, two survey  instruments, and a checklist used to identify  patrons’ social styles.    setting – a public library system in the  northeastern us.    subjects – a total of 24 library patrons who  sought assistance at the reference desk of a  public library and the five librarians who  delivered reference services to them.    methods – the researcher observed 24  reference interviews conducted by five  different librarians at a reference desk in a  public library system. it is unclear if all 24  interviews took place in the same library.  reference interviews that took place during  the times the researcher was on site and did  not relate to the use of the public computer  terminals were included in the study.     during each interaction, the researcher  compared the patron’s behaviour to a checklist  of traits relating to assertiveness and  responsiveness. for example, more assertive  behaviours included moving and talking  quickly and sustained eye contact. less  assertive behaviours included “waiting to be  89 mailto:cmerkley@mtroyal.ca evidence based library and information practice 2010, 5.2  asked” and soft speech (p. 127). more  responsive behaviours usually consist of a  greater willingness to engage on an emotional  level with the librarian and more open facial  expressions and gestures.    the balance of behaviours across the two  categories was used by the researcher to  determine which of the following social style  categories a patron belonged to: driver,  analytical, amiable, or expressive. drivers,  described in social style theory as “practical  and task orientated,” were those who  demonstrated “less responsive” and “more  assertive” behaviours at the desk (pp. 127‐ 128). those who were “less responsive” and  “less assertive” were labelled analytical (p.  128). according to social style theory,  analyticals “collect quite a bit of data before  making decisions” and are methodical (p. 127).  patrons who were “more responsive” and  “less assertive” were determined to be  amiable; and expressives were those who were  “more responsive” and “more assertive” (p.  128). amiables are described as “easy‐going”  and expressives as “outgoing” and  “spontaneous” (p. 127).    after the researcher had observed twelve  reference interviews, the participating  librarians received information on how to  recognize and adapt to individual social styles.  the information they received took the form  of an article, a diagram of the four social style  categories, and examples of famous people  and their social styles. after the librarians  received this information, the researcher then  observed twelve additional reference  interviews.    after each reference interview, the patron and  librarian completed separate surveys. the  surveys were adapted from the tools used by  the wisconsin‐ohio reference evaluation  program. each survey consisted of ten  questions, which gauged the patron’s and the  librarian’s level of satisfaction with the  reference experience. librarians responded to  questions relating to whether they were able  to answer the patron’s question, their  perceptions of the patron’s satisfaction, how  comfortable and confident they felt during the  interaction, and whether they “taught the  patron something new” (p. 132). the patron  survey asked about the quality of service  patrons feel they received from the librarian,  whether or not they found what they were  looking for, and if they learned something  new from the experience. a score was  calculated based on the following possible  responses: yes (4 points), partly (2 points) and  no (0 points).     main results – of the 24 patrons observed in  the study, the majority (10) were analyticals.  the next largest group was expressives (6),  followed by amiables (4), and drivers (3). the  remaining patron was classified as both a  driver and an expressive because he/she held  an equal number of characteristics for both  categories.      the median survey scores indicated that,  overall, librarians rated the quality of  reference interview lower than their patrons.  the median score for the librarian survey after  the first twelve interviews was 28, while the  median score for their patrons was 36. the  correlation between these scores was r=0.27.  after the librarians received information on  social styles, the median score of the librarian  survey rose to 32. however, the patrons’  median score dropped slightly to 35. the  correlation of the scores after the intervention  was r=0.57.    the responses of librarians and patrons most  closely corresponded when the patrons were  amiables. amiables and the librarians who  worked with them were in all in agreement on  whether the patron’s question was answered.  only one of the expressives and one of the  drivers disagreed with the librarians’ response  to this question. in both cases, the librarians’  responses were “partly” while the patrons  answered “yes” (p. 129). twenty percent of the  analyticals were satisfied with the information  they received from the librarian. in some  cases, they were satisfied even though the  information they were provided was not what  they were looking for when they first   approached the reference desk.    90 evidence based library and information practice 2010, 5.2  the study suggests that there may be a  disconnect between librarian and patron  responses in regard to whether the librarian  taught the patron something new in the  exchange. according to sisselman, all  expressives and 75% of amiables responded  that they had learned something new, but only  93% of the librarians who worked with the  expressives and 50% of those working with  amiables felt that they had done so. the scores  for drivers for this question were double those  of the librarians they worked with – the actual  figures were not provided by the author. the  results for analyticals on this question were  not reported.     conclusion – the findings of the study  suggest that a patron’s social style may play a  role in how they perceive the reference  interview. the author also suggests that “there  may be a correlation between librarians’  understanding of the social styles of patrons  (analytical, expressive, driver, or amiable) and  the outcomes of reference interviews” (p. 130).  possible areas of application for these findings  include improved public service and tailoring  of communication and marketing strategies to  the diverse social styles of current and  potential patrons.      commentary    sisselman’s exploration of social styles offers  another angle at which to examine the  reference interview.  however, the conclusions  drawn by the author about the influence of  social styles are undermined by the fact that  there was inadequate representation of some  social style groups amongst the research  participants. more than twice as many  analyticals participated than amiables or  expressives. sisselman acknowledges this  imbalance, suggesting that because of their  personality traits, analyticals may be more  likely than other groups to approach the  reference desk. this assumption has not been  tested, and the author identifies this as an area  for future research.   what is also not clear is if the pre‐ and post‐ intervention sample of patrons was similar in  terms of the mix of social styles represented.  did the librarians’ scores rise because they  better understood the verbal and non‐verbal  cues the patron was sending after being  provided with additional information, or did  the more expressive patrons largely fall into  the post‐intervention sample? providing the  breakdown of the social styles of both the pre‐  and post‐intervention samples would have  provided some clarity on this issue.     this was not the only place where more  information about the study framework  would have been desirable. the researcher  isolated the social styles of patrons from other  variables that may or may not have had an  impact on how they perceived the reference  interview, such as the gender and age of the  patrons. reference interviews that took place  when the researcher was present were  included in the study, but we are not told if all  the interviews took place in the same library,  in the same general timeslot, or on the same  day of the week – all of which could have  influenced who was present in the library to  ask reference questions or the type of  questions the librarian was tasked with  answering. we do not know if the participants  were equally divided among the five  librarians. the possible impact of observer  bias is also unaddressed. the lack of  information about other factors which may  have influenced the reference interview or  who was available to participate makes it  difficult to know how much weight to place  on the variable of social styles in impacting the  results.     the results of the study should also not be  overstated. the training the librarians received  only increased the median survey scores for  the librarians – the median score amongst  patrons actually decreased after the  intervention. no information was provided as  to how the intervention changed the  librarians’ behaviour, and there was no  evidence presented by the author that the  patron’s satisfaction with the reference  interview was influenced by librarians having  this knowledge. only the librarians’  perceptions of the encounter appear to have  91 evidence based library and information practice 2010, 5.2  92 shifted. in fact, sisselman states that patron  responses to survey questions relating to the  broader reference experience (for example, if  the librarian was “courteous and considerate”)  had no “significant impact” on the overall  survey results (p. 128; p. 133). in addition, the  author’s inclusion of percentages without  clearly indicating the number of responses,  particularly when those numbers are very  small, lends more weight to conclusions than  is warranted. social styles offer some  intriguing possibilities for libraries and  librarians, particularly in aiding library staff in  better understanding the behavioural cues of  their patrons, but further studies are required  to determine the extent to which they play a  role in a patron’s satisfaction with library  services.  microsoft word class_preddie.doc evidence based library and information practice 2007, 2:4  104 evidence based library and information practice       classic    hospital libraries have a positive impact on clinical decision making and patient  care     a review of:  marshall, joanne gard. “the impact of the hospital library on clinical decision making: the  rochester study.” bulletin of the medical library association 80.2 (1992): 169‐78.      reviewed by:   martha i. preddie  doctoral student, school of information and library science, university of north carolina at  chapel hill  chapel hill, north carolina, united states of america  e‐mail: mipreddie@gmail.com        received: 09 september 2007      accepted: 18 november 2007      © 2007 preddie. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to determine the impact of  hospital library services on clinical decision  making.      design – a descriptive survey.      setting – fifteen hospitals in the rochester  area of new york, united states of america.   seven hospitals were in the city of rochester,  and eight were in surrounding rural  communities.    subjects – active physicians and residents  affiliated with the rochester hospitals.      methods – this study built upon the  methodology used in an earlier study by d.  n. king of the contribution of hospital  libraries to clinical care in chicago.  lists  were compiled of all the active physicians  and residents who were affiliated with the  rochester hospitals.  in order to ensure that  there was a reasonable number of  participants from each hospital, and that  librarians in hospitals with larger numbers  of staff were not overburdened with  requests, predetermined percentages were  set for the sample: 10% of active physicians  from hospitals with more than 25 medical  staff members, 30% from hospitals with less  staff, and 30% of residents and rural  physicians.  this resulted in a desirable  sample size of 448.  a systematic sample  with a random start was then drawn from  each hospital’s list, and physicians and  residents were recruited until the sample  mailto:mipreddie@gmail.com http://creativecommons.org/licenses/by/2.0 http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:4  105 size was achieved. participants were asked  to request information related to a clinical  case from their hospital library, and to  evaluate its impact on patient care, by  responding to a two‐page questionnaire.     main results – based on usable  questionnaires, there was an overall  response rate of 46.4% (208 of 448).   eighty  percent of the respondents stated that they  probably (48%) or definitely (32.4%)  handled a clinical situation differently due  to the information received from the library.   in terms of the specific aspects of care for  which changes were made, 71.6% reported a  change in advice given to the patient, 59.6%  cited a change in treatment, 50.5% a change  in diagnostic tests, 45.2% a change in drugs,  and 38.5% a change in post‐hospital care or  treatment.  physicians credited the  information provided by the library as  contributing to their ability to avoid  additional tests and procedures (49%),  additional outpatient visits (26.4%), surgery  (21.2%), patient mortality (19.2%), hospital  admission (11.5%), and hospital‐acquired  infections (8.2%).  in response to a question  about the importance of several sources of  information, the library received the highest  rating amidst other sources including lab  tests, diagnostic imaging, and discussions  with colleagues.       conclusion – this study validates earlier  research findings that physicians view the  information provided by hospital libraries  as having a significant impact on clinical  decision making.  library supplied  information influences changes to specific  aspects of care as well as the avoidance of  adverse events for patients.  the significance  of this influence is underscored by the  finding that relative to other sources,  information obtained from the hospital  library was rated more highly.          commentary          marshall’s rochester study built on  previous research conducted by king in  chicago on the contribution of hospital  libraries to patient care.  the elimination of  the hospital library requirement by the new  york state department of health was a  driving force behind the decision to conduct  the rochester study.  the study was  conceptualized as a means of investigating  the value and impact of information in the  hospital setting in an effort to ascertain the  direct contribution of libraries to health care  institutions. the rochester study went  beyond king’s study by measuring the  importance of changes in clinical care that  could be attributed to the use of information  obtained from the library by identifying the  specific aspects of clinical care that changed  due to information provided by the library,  by discovering the situations in which  library provided information prevented the  occurrence of adverse events for patients,  and by computing the relative value of this  information in comparison with other  sources of clinical decision making data.      apart from extending the research  parameters beyond king’s, the impact of  marshall’s study was heightened by the  finding that the utilization of information  from literature searching resulted in saving  the lives of 40 persons (19.4%) in 200 cases.   this discovery was particularly significant  since previous studies (e.g. lindberg et al.)  found reduced impact on mortality figures  of around 10%. another landmark finding  was that literature searching had a greater  impact on patient care management than  other sources of clinical information, namely  diagnostic imaging, lab tests, and  consultations with colleagues.  in going  further than king’s work and klein’s  research on the effect of online literature  searching on length of stay and patient care  costs, the rochester study supplied detailed  information about the “specific nature and  evidence based library and information practice 2007, 2:4  106 extent of the impact of information provided  by the hospital library” (169) in terms of the  delivery of patient care, and hence health  care outcomes.  the study’s findings  confirmed what librarians previously felt to  be true based on their observations and  interactions with users, and provided  tangible proof about the significance of their  work.      the rochester study continues to be of  significance today.  this can in part be  attributed to the fact that the questions  raised by this study – to what extent do  literature searches save lives and how does  the information they provide relate to other  clinical information in importance – are still  to be answered.  additionally, the issue of  adverse events arising from patient  management errors, identified in the  harvard medical practice study in 1991, still  exists today (leape, brennan, laird, et al.).  equally true is marshall’s statement that  information provided by hospital libraries  can contribute positively to reducing “the  frequency and severity of adverse events in  hospitalized patients” (177), thus rendering  the focus of the rochester study of  continuing significance.      the importance of this work is not limited to  the sphere of medical librarianship. the  library and information science research  literature and the health sciences literature  bear testament to the significance of the  rochester study.  sherwill‐navarro and  wallace’s article about the impact of  research on the value of medical library  services positioned the rochester study as  one of the most read, cited, and valuable  articles published on the topic.  these  authors found that in the year of its  publication, the citation frequency for  marshall’s work was “more than ten times  the average frequency for an article  published the same year” in the bulletin of  the medical library association (36).  a recent  search of the isi web of science revealed that  this study has been cited every year since its  publication in a total of 70 english‐language  articles out of 95 citations that include  reviews, letters, and editorials.      urquhart and hepworth, burton, casado‐ uriguen et al., and ali have all conducted  research patterned after various aspects of  the rochester study.  marshall’s specific  categories of impact (avoidance of  unnecessary tests, hospital admissions,  patient mortality, reduction in length of  hospital stay, and improvement of clinical  decisions) were included in a taxonomy  developed by abels, cogdill, and zach of  library and information services  contributions to hospitals and academic  health centers.  other citing publications  focus not only on the value and impact of  information on clinical decision‐making but  also on a range of related topics, including  total quality management service  benchmarks, evidence based medicine,  evidence based nursing, evidence based  strategic planning in libraries, the impact of  clinical information retrieval technology on  physicians, patient safety, user education,  the role of librarians, library outreach, the  information needs and information seeking  behaviors of health practitioners and  administrators, decision support systems,  and continuing medical education.     marshall acknowledged the study’s  limitations: a response rate of 46.4%, the  possibility that the medical society  president’s letter of support that  accompanied the package sent to  participating physicians could have  increased the overall response, results based  on self‐reporting, and the asking of yes or  no answers where qualitative type  responses would have been more suitable.   despite these limitations, marshall’s work is  valuable not only as a piece of research  demonstrating value, but also as an example  of quality lis research.  although the level  of evidence is not high since the study was  evidence based library and information practice 2007, 2:4  107 based on self‐reporting of impact, the study  is nevertheless meticulous in design and  analysis.       undoubtedly the annals of medical  librarianship will record the rochester study  as historically significant in posterity  because it has contributed to the store of  knowledge directly demonstrating the value  of libraries to clinical care, and has formed  the basis of many further studies.  in so  doing, this seminal study has contributed  positively to the research literature of the  health sciences and library and information  professions.      works cited    abels, eileen g., keith w. cogdill, and lisl  zach. “the contributions of library  and information services to hospitals  and academic health sciences centers:  a preliminary taxonomy.” journal of  the medical library association 90.3  (2002): 276‐84.    ali, irena. “library provided information  and clinical decision making: a study  of two hospitals in canberra.”  australian academic and research  libraries 31.1 (2000): 30‐45.     burton, j. e. “the impact of medical  libraries and literature on patient  care in new zealand.” bulletin of the  medical library association 83.4 (1995):  425‐30.    casado uriguen, m., m. a. garcia martin, p.  m. la torre, m. i. montes del olmo, t.  mas vilardell, and m. ribes cot.  “importance of hospital library  facilities to clinical decision making  by medical staff.” health information  – new possibilities: proceedings of the  4th european conference of medical  and health libraries held on 28 june‐2  july 1994, oslo. eds. t. mcsean, j. van  loo, e. coutinho. dordrecht: kluwer  academic publishers, 1995. 126‐8.     king, d.n. “the contribution of hospital  library information services to patient  care: a study in eight hospitals.”  bulletin of the medical library  association 75.4 (1987): 291‐301.    klein, m. s., f. v. ross, d. l. adams, and c.  m. gilbert. “effect of online literature  searching on length of stay and  patient care costs.” academic  medicine 69.6 (1994): 489‐95.      leape, l. l., t. a. brennan, n. laird, et al.  “the nature of adverse events in  hospitalized patients. results of the  harvard medical practice study ii.”  new england journal of medicine  324.6 (1991): 377‐84.      lindberg, d. a., e. r. siegel, b. a. rapp, k.  t. wallingford, and s. r. wilson. “use  of medline by physicians for clinical  problem solving.” journal of the  american medical association 269.24  (1993): 3124‐9.    sherwill‐navarro, pamela j., and addajane  l. wallace. ʺresearch on the value of  medical library services: does it make  an impact in the health care  literature?ʺ journal of the medical  library association 92.1 (2004): 34‐45.     urquhart, c.j., and j. b. hepworth.  “comparing and using assessments of  the value of information to clinical  decision‐making.” bulletin of the  medical library association 84.4 (1996):  482‐9.   microsoft word art_lewis.doc evidence based library and information practice 2007, 2:1  107 evidence based library and information practice     article    have the most relevant and answerable research questions facing librarians  changed between 2001 and 2006?       suzanne lewis  manager, central coast health service library  northern sydney central coast health  gosford, new south wales, australia  email: suzanne@eblip.net.au    lisa cotter  school librarian, nursing and midwifery  the university of newcastle and northern sydney central coast health  gosford, new south wales, australia  email: lisa@eblip.net.au      received: 07 september 2006    accepted: 25 february 2007      © 2007 lewis and cotter. this is an open access article distributed under the terms of the creative  commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    abstract    objective ‐ to examine the similarities and differences between research questions asked  by librarians in 2001 to those posed in 2006, and to explore to what extent the published  research supports the questions being asked.      methods ‐ questions collected in 2001 by members of the evidence‐based librarianship  implementation committee (eblic) of the mla research section were compared with  questions collected in 2006 at a cross‐sectoral seminar introducing evidence based library  and information practice to australian librarians. questions from each list were categorized  using the domains of librarianship proposed by crumley and koufogiannakis in 2001, and  examined with reference to a content analysis of the library and information studies (lis)  research published in 2001 by koufogiannakis, slater, and crumley in 2004.       http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:1  108 results ‐ in 2001 and 2006 the most commonly asked questions were in the domain of  management (29%, 33%), followed by education (24%, 18.5%). in 2001 questions in the  marketing/promotion category ranked lowest (1%), however representation was much  greater in 2006 (18.5%) ranking an equal second with education. questions in the lowest  ranked domain in 2006 (collections, 6%) had been more common in 2001 where collections  ranked third, representing 19% of the questions. koufogiannakis, slater, and crumley’s  content analysis of lis research published in 2001 revealed that the most popular domain  for research was information access and retrieval (38%) followed by collections (24%). only  1% of published lis research (seven articles) was in the domain of marketing/promotion. in  contrast, 36 articles originally assigned to one of the six established domains could more  appropriately have been included in a proposed new domain of professional issues.    conclusion ‐ the disparity between questions being asked by practitioners and the  evidence being generated by researchers suggests that the research‐practice gap is still an  issue. a content analysis of more recently published lis research would be a useful  comparison to koufogiannakis, slater, and crumley’s analysis of research published in 2001.      introduction  in june 2006 librarians at northern sydney  central coast health, australia, and the  university of newcastle, australia,  organized an event entitled evidence‐based  librarianship (ebl) in practice: a seminar  with andrew booth. one of the seminar  exercises resulted in the production of a list  of focused and answerable questions  representing current workplace issues  reported by the seminar delegates. the  process of compiling the questions for  distribution to participants after the seminar  inspired the authors to subject the list to  further scrutiny. to identify trends in the  kinds of questions asked by librarians, this  paper examines similarities and differences  between research questions asked by this  group of australian librarians in 2006 with  questions asked by health librarians and  solicited via discussion lists in 2001. to  explore the extent to which published lis  research supports the questions being asked  and to identify research‐practice gaps, both  sets of questions were compared with the  results of a content analysis of lis research  articles published in 2001.     methods  a total of 87 librarians attended two  seminars conducted in sydney and gosford,  representing all types of libraries including  special, health, law, government, school,  public, technical college, and university  libraries. one of the exercises conducted  during each seminar required participants  to document a current, relevant workplace  issue they faced in spice (setting,  perspective, intervention, comparison,  evaluation) format on a post‐it note. the  notes were then stuck to the wall to share  with the group. after the seminars, the  spice breakdowns of relevant issues facing  practicing librarians were transformed into  focused and answerable questions (cotter  and lewis). the 49 questions were grouped  according to the six domains of librarianship  proposed by crumley and koufogiannakis  in 2001: reference/enquiries, education,  collections, management, information access  and retrieval, and marketing/promotion (9).  these questions are presented in appendix  a.     permission to publish the questions was not  sought, although participants were advised  the questions would be documented and  shared. participation in the exercise was  voluntary, and the questions reported did  not contain details that could be used to  identify the persons asking the question.  when the list of questions was forwarded to  evidence based library and information practice 2007, 2:1  109 participants, the report included advice that  the authors intended to further analyze the  list of questions. no feedback was received.     the questions asked by australian librarians  in 2006 were then compared to 114  questions asked by health librarians  worldwide in 2001, collected by members of  the evidence‐based librarianship  implementation committee (eblic) of the  mla research section (eldredge). these  questions were collected by canvassing  various international medical librarian  groups. questions came from the us,  estonia, sweden, and australia. details of  the person asking the question and the  eblic mentor who received the question  are given.     eldredge organized these questions within  the categories of resources, library skills  education, searching, clinical librarians,  role/impact of the medical librarian, and  management; two of which are specific to  health librarianship. for the purposes of  comparison, eldredge’s questions were  recategorized according to crumley and  koufogiannakis’ six domains which are  applicable to librarians across all sectors.  where eldredge’s categories and crumley  and koufogiannakis’ domains coincided –  for example, education, management, and  resources (eldredge)/collections (crumley) –  the process of recategorizing was  straightforward. however questions from  2001 reported under the headings of clinical  librarians and role/impact of the medical  librarian were assigned to a range of  domains.    this study’s two researchers categorized  both sets of questions into domains  independently. they then discussed their  results and resolved discrepancies through  discussion and consultation with colleagues.  this process involved some professional  judgments in relation to questions that  could have been assigned to more than one  domain.   finally, the questions asked in 2001 and  2006 were examined in relation to a content  analysis of lis literature carried out in 2004  (koufogiannakis, slater, and crumley). this  study surveyed the lis research published  in 2001 and tested the six eblip domains  proposed by crumley and koufogiannakis.    results  a comparison of the most commonly asked  questions in 2001 and 2006 arranged by  domain is provided in table 1.  in both  years, the most commonly asked questions  were in the domain of management  followed by education.  however, in 2006     domain  eldredge  2001  ranking  cotter & lewis   2006  ranking  management  33   (29%)  1  16  (33%)  1  education  27   (24%)  2  9    (18.5%)  2/3  collections  22   (19%)  3  3    (6%)  6  reference/enquiries  20   (17.5%)  4  8    (16%)  4  information access &  retrieval  11   (9.5%)  5  4    (8%)  5  marketing/promotion  1     (1%)  6  9    (18.5%)  2/3            total  114  (100%)    49  (100%)      table 1. comparison by domain of questions asked by librarians in 2001 (eldredge) and 2006 (cotter and  lewis)  evidence based library and information practice 2007, 2:1  110     domain  number of articles  ranking  information access & retrieval  314   (38%)  1  collections  193   (24%)  2  management  135   (16%)  3  education  95   (12%)  4  reference/enquiries  77   (9%)  5  professional issues*  36     6  marketing/promotion  7   (1%)  7    table 2. lis research articles published in 2001. *professional issues domain has been excluded from comparative  analysis in this paper.      marketing and promotion scored an equal  number of questions as education, whereas  in 2001 only one question fell in this domain.  the percentage of questions asked about  collections fell from 19% in 2001 to only 6%  in 2006, while the percentage of questions  about reference/enquiries (17.5% in 2001,  16% in 2006) and information access and  retrieval (9.5% in 2001, 8% in 2006) remained  steady.  in the content analysis of lis research  published in 2001, 807 articles published in  91 journals were examined by three  independent reviewers and classified by  study type and domain. the number of  articles identified for each domain is  presented in table 2.         0 5 10 15 20 25 30 35 40 % r ef er en ce /e nq ui rie s e du ca tio n c ol le ct io ns m an ag em en t in fo rm at io n a cc es s & r et rie va l m ar ke tin g/ p ro m ot io n 2001 questions 2006 questions 2001 published research     figure 1. comparison by domain of questions asked and research published      evidence based library and information practice 2007, 2:1  111 questions asked in both years were  compared to the published research of 2001  by domain and can be seen in figure 1.  this  shows a wide gap between research and  practice in the library and information  science field.  this is most apparent in the  area of information access and retrieval,  where a large amount of research was  published in comparison to few questions in  both 2001 and 2006.    discussion  koufogiannakis, slater, and crumley  suggest that the results of their content  analysis of lis research published in 2001  support replacing the domain of  marketing/promotion with one of  professional issues. however, they  acknowledge that further studies analyzing  lis research over additional years are  required to determine whether the  marketing/promotion domain should be  retained or incorporated into the  management domain and a new domain of  professional issues added. for this reason,  the present study uses the original six  domains proposed in 2001.     in the set of questions from 2001, up to 19  could have been appropriately categorized  in a professional issues domain. for  example, the question “what’s the role of  the librarian in a world awash with print  and digital information, with publishers  wanting to connect directly to non‐users,  and authors wanting to bypass publishers?”  (eldredge) has been allocated to the domain  of management, but would fit equally well  into a domain of professional issues, or even  into the collections domain. in contrast, in  2006 only one question dealt with  professional issues.    therefore, the 2001 set of questions and the  content analysis of lis research published in  2001 both indicate a need for a new domain  of professional issues, whereas the 2006 set  of questions does not. the jury is still out on  this question, but a new content analysis of  more recent lis research using the same  methodology as koufogiannakis, slater, and  crumley would provide further evidence to  arrive at a consensual set of domains.    the marked difference in the number of  questions dealing with professional issues  between 2001 and 2006 may be an artifact  attributable to the way in which the  questions were asked. in 2001 the evidence‐ based librarianship implementation  committee (eblic) called on medical  librarian groups to “identify the most  important contemporary research questions  facing our profession” (eldredge), thus  prompting librarians to think broadly about  their role as well as their practice. in contrast,  during the evidence‐based librarianship  seminar in 2006, andrew booth asked  participants to analyze a workplace issue  they faced, thereby directing them to  examine what librarians do rather than who  they are.    the fact that the 2001 questions were  solicited via a discussion list is a potential  source of bias. booth warns that such a  sample methodology “may attract those  with polarized views rather than the ‘silent  majority’” (ʺclearʺ 359). the 2006 questions  also constitute a sample of convenience and  since not all workshop participants  volunteered questions, those who did are  not necessarily representative of the group.  participants in the 2006 seminar had no  prior warning of the exercise and only about  ten minutes in which to formulate their  questions. it is safe to assume that  respondents to the call for questions in 2001  had more time in which to consider the most  important and relevant questions. the  authors observe, however, that use of spice  in the seminar exercise has resulted in a set  of more refined and answerable questions in  2006. this became evident when the  researchers were allocating the questions in  each set to the eblip domains. difficulties  evidence based library and information practice 2007, 2:1  112 were encountered categorizing the 2001  questions as the focus of some questions  was ambiguous.    neither sample is representative of the  profession as a whole. the 2001 sample  spans several countries but consists largely  of health librarians, whereas the 2006  sample is drawn only from australia but is  multi‐sectoral (health, academic, school,  public, and special libraries being  represented). the 2006 sample is less than  half the size of the 2001 sample (49  compared to 114 questions). although the  2006 questions provide an example of  relevant and better‐formulated questions  facing librarians across sectors today, it  remains difficult to tell if these are truly  representative of those questions facing all  librarians internationally.    despite the limitations of the samples and  methodology noted above, broad analysis of  the two groups of questions separated by a  period of five years reveals some interesting  trends. for example, the hard copy versus  online collections debate was clearly evident  in 2001, but in 2006 was the subject of only  one collections question. in 2006 the hard  copy/online debate had shifted to the  different but related issue of providing  current awareness services with librarians  asking whether it was more effective to  provide tables of contents online, by email,  or in hard copy. questions about the  effectiveness of library skills education were  prominent in both 2001 and 2006,  suggesting that this is one area where  library practitioners have not found the  answers to their questions.     what does this all mean? do management  questions lead the field in both years  because usually the decision to introduce  new services or discontinue old ones has to  be justified in some way, or because there is  little research in this field to date? why has  the percentage of education questions not  decreased between 2001 and 2006, given the  large amount of time and money that most  libraries devote to information skills  education? of the 49 questions asked by  australian librarians in 2006, 8 (16%) dealt  with the question of whether face to face or  online delivery of training was more  effective.     it is generally accepted that the research  base for library and information science is  still small (marshall) compared to other  disciplines such as health where evidence  based practice is well established. a survey  of uk librarians across all sectors carried out  in 2003 by the university of central england  found that around half the responding  libraries across all sectors had been involved  in some form of research in the previous  two years but that research findings were  not being widely disseminated (mcnicol, ʺis  researchʺ 125). a similar survey was  completed in 2006 by the same researcher,  this time on behalf of evidence base (see  www.ebase.uce.ac.uk). there was evidence  that sharing of research findings by the  surveyed librarians had improved since  2003 (mcnicol, ʺoverviewʺ 7). however  these results should be interpreted  cautiously, because both surveys included  statistics collection as a research activity,  whereas many library practitioners would  regard this not as research but simply as  standard library management practice. the  word “research” can be interpreted in  varying ways:    to some people, research is simply a  carefully conducted investigation of  a subject or a situation. to others it  is the discovery of previously  unknown facts. to still others, it  implies a highly specific approach  to designing and conducting  research studies in keeping with  externally determined guidelines or  methods. (special libraries  evidence based library and information practice 2007, 2:1  113 association quoted in  koufogiannakis and crumley 325)         why is there a gap between research and  practice as demonstrated by the comparison  of questions asked with research published  (figure 1)? librarians may continue to ask  questions about management and education  because relatively little research is published  within these domains in the lis literature.  also, research in these domains is spread  across databases covering the disciplines of  business and education, as well as library  and information science, and therefore is  more difficult to find. in contrast, a  relatively large amount of research in  information access and retrieval is being  published in the lis literature. however, the  findings of this study show that  practitioners are asking relatively few  questions in this domain.    the research‐practice gap has been the  subject of considerable discussion in the lis  literature. andrew booth has observed:    the research‐practice gap is usually  discussed with reference to the  failure of practitioners to implement  the findings of research. a less  commonly explored, yet  nevertheless equally important  manifestation of this same gap is the  failure of researchers to address  questions of direct relevance to  practitioners (ʺturningʺ 130).    the results of this study ‐ particularly the  fact that information access and retrieval  accounted for 38% of the published lis  research in 2001 but only 9.5% and 8% of  questions asked by librarians in 2001 and  2006 respectively ‐ illustrate just such a  discrepancy between the questions  practitioners are asking and the issues  researchers are examining.    in fact, it has been claimed that “there is not  just a gap between research and practice in  lis, but a ‘communication chasm’” (k.  turner quoted in haddow and klobas 30).   if researchers and practitioners in lis were  communicating well, research would inform  practice and practice would be based on  research evidence and, in turn, generate  more questions for future research. instead,  as this study demonstrates, researchers are  focusing their activities in the domains of  information access and retrieval and  collections while practitioners are looking  for answers to questions in the domains of  management and education, specifically  information literacy skills training.  therefore, this study is further evidence of  the “relevance gap” between practice and  research identified by haddow and klobas  in which “researchers and practitioners  value investigation of different types of  problem” (31).    kim’s study of the perceived barriers to use  of research in practice by university  librarians also reported that communication  barriers affected the utilization of research.  such barriers included “a lack of clarity with  respect to the implications of the research in  practice, an inability to demonstrate the  relevance of the research to practice, and a  lack of readable research” (444). lack of  “readable” research was highlighted as a  problem for librarians of a non‐english  speaking background.    crumley and koufogiannakis have  identified the traditional hierarchy of  evidence as a potential barrier to librarians  finding the research that will answer their  questions (“developing 2002” 65).  librarians, particularly those working in the  health field, may feel that anything less than  a randomized controlled trial is not  sufficient evidence to answer their questions  or change their practice. yet, out of the 807  articles identified as research articles  published in 91 library and information  evidence based library and information practice 2007, 2:1  114 studies journals in 2001, only 12 articles  could be described as systematic reviews,  meta‐analyses, randomized controlled trials,  or controlled trials – the recognized higher  levels of evidence (koufogiannakis, slater  and crumley). librarians must be prepared  to locate, critically appraise, and use other  kinds of research, including qualitative  research, to inform decision‐making. that is,  “finding usable research for practical  situations” (bayley and mckibbon 320).    a further reason why librarians are not  finding the research to answer their  questions may be that they are prevented  from looking by factors such as time  constraints, difficulties in accessing the  relevant databases, or unsupportive  management. certainly there is discussion,  if not conclusive answers, in the recent  literature addressing delivery of information  skills training – consistently ranked highly  in the questions of both 2001 and 2006.  although education accounted for only 12%  of articles identified in the content analysis  of literature published in 2001, it is the  subject of recent high quality research.  several papers were presented in this area at  the 3rd international evidence based  librarianship conference in 2005. a  systematic review of the literature on  information skills training was published in  2003 (brettle). a systematic review to  identify the most effective methods of  teaching information literacy skills has been  published (koufogiannakis and wiebe), and  has also yielded the reliant tool for use  by library practitioners when appraising  published reports of education and training  interventions (koufogiannakis, booth, and  brettle). in the same issue as the systematic  review, a randomized controlled trial  comparing e‐learning and a taught  workshop for teaching literature search  skills to health professionals is also reported  (pearce‐smith). it seems that research is now  being done in the eblip domain of  education to investigate at least some of the  questions of relevance to lis practitioners.  the authors support koufogiannakis and  crumley’s (329) call for a review of which  practitioner questions have been answered  in published lis research, and which remain  in need of answering.    conclusion  this study builds upon the list of relevant  and answerable research questions facing  health librarians published by eldredge in  2001 by providing a more recent set of  questions facing librarians from a wider  range of library sectors. the results show  that the most commonly asked questions by  librarians are about management or  education issues. however the disparity  between questions being asked by  practitioners and the evidence being  generated by researchers suggests the  research‐practice gap is still an issue.  researchers need to be aware that  practitioners are consistently asking  questions falling into the domains of  management and education. practitioners  must be prepared to look beyond the lis  literature for evidence to answer their  questions – particularly those relating to  management and marketing/promotion.    further content analysis of the lis literature  is required to identify any changes since  koufogiannakis et al examined the literature  published in 2001. a content analysis of  literature published in 2005 would be of  great interest to compare with the questions  asked by australian librarians in june 2006.    works cited    bayley, liz, and ann mckibbon. ʺevidence‐ based librarianship: a personal  perspective from the medical/nursing  realm.ʺ library hi tech 24.3 (2006):  317‐23.    brettle, alison. ʺinformation skills training:  a systematic review of the literature.ʺ  evidence based library and information practice 2007, 2:1  115 health information & libraries journal  20.s1 (jun. 2003): 3‐9.    booth, andrew. ʺclear and present  questions: formulating questions for  evidence based practice.ʺ library hi  tech 24.3 (2006): 355‐68.    ‐‐‐. ʺturning research priorities into  answerable questions.ʺ health  information & libraries journal 18.2  (jun. 2001): 130‐2.    lewis, suzanne, and lisa cotter. ʺseminar  spice questions.ʺ libraries using  evidence ‐ eblip.net.au. july 2006. 1  august 2006  <http://eblip.net.au/seminar/spiceques tions.pdf>.    crumley, ellen, and denise koufogiannakis.  ʺdeveloping evidence based  librarianship in canada: six aspects  for consideration.ʺ hypothesis 15.3  (fall 2001): 9‐10.    ‐‐‐. ʺdeveloping evidence‐based  librarianship: practical steps for  implementation.ʺ health information  & libraries journal 19.2 (jun. 2002): 61‐ 70.    eldredge, jon. ʺthe most relevant and  answerable research questions facing  the practice of health sciences  librarianship.ʺ hypothesis 15.1 (spr.  2001): 9‐17.    evidence‐based librarianship (ebl). a  seminar with andrew booth.”  libraries using evidence ‐ eblip.net.au.  29 june 2006. 27 february 2007  <http://www.newcastle.edu.au/service/ library/gosford/ebl/jun06papers.html>.    haddow, gaby, and jane e. klobas.  ʺcommunication of research to  practice in library and information  science: closing the gap.ʺ library &  information science research 26 (win.  2004): 29‐43.    kim, kapseon. ʺperceived barriers to  research utilization by korean  university librarians.ʺ journal of  academic librarianship 31.5 (sep.  2005): 438‐48.    koufogiannakis, denise, and ellen crumley.  ʺresearch in librarianship: issues to  consider.ʺ library hi tech 24.3 (2006):  324‐40.    koufogiannakis, denise and natasha wiebe.  “effective methods for teaching  information literacy skills to  undergraduate students: a systematic  review and meta‐analysis.” evidence  based library and information practice  1.3 (2006): 3‐43.    koufogiannakis, denise. ʺeffective methods  for teaching information literacy  skills to undergraduate students:  what does the library research  literature reveal?ʺ 3rd international  evidence based librarianship  conference. evolution of evidence :  global perspectives on linking  research with practice. october 2005.  27 february 2007  <http://conferences.alia.org.au/ebl2005/ koufogiannakis.pdf>.    koufogiannakis, denise, andrew booth, and  alison brettle. ʺreliant: readerʹs  guide to the literature on  interventions addressing the need for  education and training.ʺ library &  information research 30.94 (spr. 2005):  44‐51.    koufogiannakis, denise, linda slater, and  ellen crumley. ʺa content analysis of  librarianship research.ʺ journal of  information science 30.3 (2004): 227‐39.  http://eblip.net.au/seminar/spiceques http://www.newcastle.edu.au/service/ http://conferences.alia.org.au/ebl2005/ evidence based library and information practice 2007, 2:1  116 marshall, joanne gard. ʺinfluencing our  professional practice by putting our  knowledge to work.ʺ information  outlook 7.1 (jan. 2003): 40‐4.    mcnicol, sarah. ʺis research an untapped  resource in the library and  information profession?ʺ journal of  librarianship & information science  36.3 (sep. 2004): 119‐26.    ‐‐‐. ʺoverview report ‐ research, evaluation  and evidence collection in library  services. a survey of librarians in the  uk and republic of ireland.ʺ evidence  base: using research to support  practitioners and policy makers.  january 2007. 1 february 2007  <http://www.ebase.uce.ac.uk/docs/over view_report.pdf>.    pearce‐smith, nicola. “a randomised  controlled trial comparing the effect  of e‐learning, with a taught workshop,  on the knowledge and search skills of  health professionals.” evidence based  library and information practice 1.3  (2006): 44‐56.  http://www.ebase.uce.ac.uk/docs/over evidence based library and information practice 2007, 2:1  117 appendix a  spice questions by domain  reference/enquiries  when do our customers use the helpdesk rather than the website and why?     is an sdi service more effective when delivered electronically, or manually?     has electronic delivery to hospital staff of tocs resulted in increased requests for articles,  compared to requests generated by paper delivery?     is it more effective to provide a weekly current awareness bulletin comprising journal contents  pages in hard copy or electronic format to health service staff without physical access to the  library?     do clinical staff regard literature searches carried out by librarians as successful in locating best  practice information?     what aspects of the technical library intranet site needs to be redesigned to increase usability and  level of usage by technical library staff and students?     to what extent do literature searches supplied to researchers and health professionals meet their  information needs?     in a university library, is instant messaging, email, or chat most effective in providing a virtual  reference service to staff and students of the university in terms of level of use, customer  satisfaction, and impact on library staff rosters?     education  are compulsory core units in information literacy, voluntary sign‐up sessions, or online tutorials  more effective at positively affecting the quality (as assessed by academics) of bibliographies in  the assignments of 1st year undergraduate students?     is face‐to‐face training by a librarian or provision of a self‐paced online tutorial more effective at  improving the database searching skills of nursing staff?     will provision of training and support in database searching for night duty nurses result in them  performing more searches?     is information skills training for staff and students of the building & construction department of  a regional university more effective when delivered in the library or when delivered either online  or in the department as measured by resource usage and assignment results?     does delivery of a motivational library seminar to a group of postgraduate health students with a  range of information skills result in more effective use of the library and its resources compared  to a traditional, skill‐based educational session?     evidence based library and information practice 2007, 2:1  118 will more academic staff and students attend academic requested library skills sessions than  currently attend the open access pre‐scheduled sessions?     in a college library is face‐to‐face library skills training more effective in increasing students’ use  of the library than delivering the same training online?     are small group practical sessions or small group seminars more effective in teaching first year  nursing students how to search online databases, as measured by user knowledge of database  searching and user satisfaction?     does an interactive online tutorial increase university students’, staff, and researchers’ skill and  satisfaction in using endnote more than face‐to‐face training?     collections  does a current awareness service that involves posting a list of new material on a library website  result in more requests for the promoted material, compared to a service that emails the list to  library patrons?     in an academic library, would making readings available on the web or only maintaining hard  copy readings lead to higher usage statistics, increased satisfaction of students, and be more cost  effective for staff to manage?     how can non‐fiction resources be purchased or donations secured for the non‐fiction collection of  a private library?     management  does the effort, time taken, and cost‐benefits required by hospital library staff to manage  duplicates lists (compiling and searching) outweigh the effort/time taken to retrieve articles from  missing issues via interlibrary services?     how often should news items be updated on a hospital library intranet site to best serve the  current awareness needs of hospital staff?     does our website provide useful information for library managers related to performance  indicators?     in a hospital library with 24 hour client access, but only staffed 4 hours per week, is it cost  effective to install a theft prevention system, compared to the losses incurred without a security  system?     is it more effective to provide outreach library services and resources via branch libraries and an  intranet site, or via a mobile library unit and increased electronic resources?     what service innovations could be implemented in a public library to better meet the needs of  young adults in the community?     evidence based library and information practice 2007, 2:1  119 is desk skills training for library staff working on a specialised information desk more effective  when delivered online or using training worksheets?     does having a liaison librarian located in the faculty offices rather than in the library increase  academic staff’s satisfaction with and use of the library?     can the cost of operating a library be justified to staff of its organisation in terms of client  satisfaction and cost effectiveness when compared to a model of service that provides  unsupported access to electronic resources?     would participation by staff of a multi‐site, inter‐region, academic library in an online journal  club be more effective than participation in a face‐to‐face journal club with respect to the  quantity/quality of innovative workplace ideas or developments initiated by library staff?     do usage statistics of a hospital library’s trial period of weekend opening justify the library  permanently extending its hours to include weekend opening?     would automation of the library registration process improve customer satisfaction and reduce  the number of data input errors, as opposed to manual registration of borrowers by lending  services staff?     from the point of view of clinicians, what are the necessary conditions for introducing a clinical  librarian service?     in an academic library what services could be developed to support improvement in work  practices of non‐academic staff?     how should an academic library make optimal use of existing space (physical collection footprint,  study space available) to meet student priorities for library facilities and services?     does a 3‐day course (involving hands on demonstrations/tasks) result in better‐trained  information commons support staff than self‐paced online delivery training?     information access and retrieval  how does the provision of federated search software in a virtual library service affect clients’ use  of the online services, help desk, and online articles, and the volume of document delivery  requests?     would area health employees prefer to access online information resources via the library’s  online catalogue or via ciap and other internet sites?     does teaching students in stages 4, 5, and 6 to search the library catalogue using subject headings  rather than keywords only result in them retrieving more relevant resources for a specific  assignment?     what alternative ill systems could be implemented to improve customer satisfaction with the  current service of requesting/supplying ills?   evidence based library and information practice 2007, 2:1  120   marketing/promotion  will marketing of the library service to government departments and their agencies result in  increased revenue for the library?     is provision of a college library website a more effective communication mechanism than  traditional methods such as mail‐outs, hard copy documentation, face‐to‐face meetings, and the  helpdesk in terms of user satisfaction and usage of the site?     does distribution of library service brochures at corporate orientation sessions for new hospital  employees lead to increased knowledge or use of the library service?     does a listing of library services on the library intranet site satisfy client needs for information  about those services?     does a presentation by library staff at induction sessions have a measurable impact on library  usage by new junior medical officers?     will notifying library clients by email of release of the current library newsletter increase  readership as measured by hits on the newsletter page?     does marketing an interlibrary service to health service staff increase usage of and satisfaction  with the service?     will a marketing campaign or information literacy program be more effective at increasing the  number of new, active users of a special library?     in a law library does posting the library bulletin on a website result in increased usage of the  library and/or greater user satisfaction with the service than emailing the bulletin to staff of the  organisation?     news/announcements   lirg and jibs student prizes    2014. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   the uk library and information research group (lirg) and the jibs user group (jibs) each award a student prize each year for an outstanding research-based project.   both prizes will be for a postgraduate dissertation or a final year undergraduate project.   •    the lirg student prize will be awarded for a research based project on any lis topic. •    the jibs student prize will be restricted to the area of library information systems, bibliographic databases or other electronic technologies and how such resources or technologies are being developed or exploited; practical projects that exploit such technologies may also be submitted.   each uk school of library and information studies is invited to nominate one of their student's projects for each prize. a student project may be submitted for both prizes but no single project will be given both prizes.   the closing date for submission is 20 july 2014.   further details on the prizes and how to submit are now available: •        lirg prize:  https://sites.google.com/site/lirgweb/home/awards/lirg-student-prize   •         jibs prize: http://www.jibs.ac.uk/   research article   moral distress among consumer health information professionals: an exploratory study   robin o'hanlon  associate librarian, user services  memorial sloan kettering cancer center library new york, new york, united states of america email: ohanlonr@mskcc.org   katelyn angell associate professor/coordinator of library instruction long island university, brooklyn campus library brooklyn, new york, united states of america email: katelyn.angell@liu.edu   samantha walsh manager of information & education services levy library icahn school of medicine at mount sinai new york, new york, united states of america email: samantha.walsh@mssm.edu   received: 10 oct. 2020                                                               accepted: 5 apr. 2021      2021 o’hanlon, angell, and walsh. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   data availability: o'hanlon, r., angell, k., walsh, s. (2021). raw dataset and interview codebook for "moral distress among consumer health information professionals: an exploratory study"[dataset]. edmonton, canada: ual dataverse. retrieved from https://doi.org/10.7939/dvn/lzhzhc   doi: 10.18438/eblip29871     abstract   objectives – in recent years, moral distress has become a topic of interest among health professionals. moral distress is most commonly described in the nursing literature, and refers to a situation wherein an individual knows the correct action to take, but is constrained from doing so. while moral distress differs from the classic ethical dilemma, in recent years practitioners and theorists have advocated for a broadening of the definition of moral distress. to date, no study has examined another group of individuals who frequently interact with patients and who may be constrained by the confines of their role consumer health information professionals (chips). the objective of this study was to determine if chips experience moral distress and/or ethical dilemmas, and to determine what, if any, coping strategies these individuals have developed.   methods – this study employed a mixed methods approach. quantitative data were gathered via an online survey which was distributed to relevant consumer health information professional electronic mail lists. the survey contained demographic questions and a series of questions related to potential discomfort within the context of work as a consumer health information professional. qualitative data were also gathered through phone interviews with chips. interview questions included the participant’s definition of moral distress, professional experiences with moral distress, and any coping strategies to manage said distress.   results – the authors received 213 survey responses. to test whether any of our demographic variables help to explain survey response, we used stata to calculate pearson correlation coefficients. individuals who were more likely to experience discomfort in their occupation as chips included individuals with less experience and individuals who identified as black and latinx. interview data indicated that participants most commonly experienced ethical dilemmas related to censorship, providing prognosis information, and feeling constrained by institutional policies. few interview participants described scenarios that reflected moral distress.   conclusions – chips do not appear to experience moral distress, at least according to its most narrow definition. chips do consistently experience distinct ethical dilemmas, and the most durable patterns of this phenomenon appear to be related to experience level and racial identity. in recent years, researchers have raised calls to broaden the definition of moral distress from its narrow focus on constraint to include uncertainty, and chips do experience moral uncertainty in their work. further study is needed to determine how to best address the impacts of discomfort caused by ethical dilemmas among these groups.     introduction   originally discussed in nursing literature, the concept of moral distress is evolving and has more recently been explored in various healthcare professions. in 1984, andrew jameton described moral distress as a phenomenon that arises “when one knows the right thing to do, but institutional constraints make it nearly impossible to pursue the right course of action” (jameton, 1984, p. 6). while jameton and his contemporaries’ discourse focused on the experiences of nurses, researchers have become interested in exploring this concept outside of the nursing profession, as well as beyond situations involving an institutional constraint. fourie (2015) sought to expand the definition of moral distress beyond nurses and experiences of constraint, proposing that moral distress occurs when health professionals experience a psychological response due to a moral conflict, restraint, or uncertainty.   while this evolved definition allows us to explore the experiences of all healthcare workers, it is important to understand the difference between moral distress, moral conflict, and ethical dilemmas. moral conflict occurs when “when duties and obligations of healthcare providers or general guiding ethical principles are unclear” (jormsri, 2004, p. 217), while an ethical dilemma involves “the need to choose from among two or more morally acceptable options or between equally unacceptable courses of action, when one choice prevents selection of the other” (ong et al., 2012, p. 11). ethical dilemmas and moral conflicts are often closely related to the experience of moral distress in healthcare professionals. as these concepts are explored and refined, the authors of this study sought to understand the experience of consumer health information professionals (chips). chips are information professionals providing consumer health information, or health information to non-healthcare professionals, in a variety of settings (reference and user services association, 2015). working in public, hospital, and other specialized libraries, these professionals regularly interact with patients and families at distressing times. while there is a growing interest in moral and ethical issues within the library and information sciences profession, information professionals who provide healthcare information to the public have not been the focus of a study on moral distress. these information professionals frequently interact with patients in a variety of settings, and may be constrained by their role, resources, or institution. furthermore, one author personally experienced a feeling of constraint while assisting a patron with a consumer health information inquiry.   literature review   beginning in the 1980’s, the majority of studies exploring moral distress consider the experiences of nurses. this continued focus is because jameton’s formative definition of moral distress necessitates the existence of “institutional constraints” (1984, p. 6). many researchers describe nurses as particularly prone to situations where they must carry out and often bear the consequences of others’ decisions (marshall & epstein, 2016). similarly, the moral distress scale, developed by corley (1995), which measures nurses’ experiences of moral distress, focuses on various limitations of agency, such as “institutional constraint.” exploring nurses’ experiences using the moral distress scale as well as other measures, researchers have found that moral distress manifests in various forms of psychological distress as well as physical manifestations. in a recent review, morley, ives, and bradbury-jones (2019) report “sleeplessness, nausea, migraines, gastrointestinal upset, tearfulness and physical exhaustion” (p. 655) in nurses experiencing moral distress. this phenomenon also has a direct effect on patient care, as oh and gastmans’ (2015) report that nurses with a high level of moral distress experience depersonalization, where they emotionally distance themselves from patients. finally, moral distress is a documented threat to the healthcare workforce itself, as whitehead, herbertson, hamric, epstein, and fisher (2015) reported that “providers who had left or considered leaving a position in the past reported moral distress mean levels significantly higher than those who had never considered leaving” (p. 123). it is important to note that whitehead et al.’s survey included all healthcare professionals in the authors’ healthcare system.   in recent years, researchers have studied moral distress in non-nursing healthcare professions, such as healthcare assistants (rodger, blackshaw, & young, 2019), veterinarians (moses, malowney, & wesley boyd, 2018), medical students (camp & sadler, 2019), physician trainees and residents (dzeng et al., 2015; sajjadi, norena, wong, & dodek, 2016), and oncologists (hlubocky, spence, mcginnis, taylor, & kamal, 2020). this research is happening in tandem with the evolution of jameton’s formative definition, as evidence of moral distress is becoming apparent across healthcare fields. like many nursing researchers, sajjadi et al. (2016) also report an increased likelihood to leave the job or profession in internal medicine residents experiencing high levels of moral distress. while we are beginning to appreciate the prevalence of moral distress among a variety of patient-facing healthcare workers, studies have not focused on the experiences of chips.   at the time of this writing, library and information science (lis) literature has reported very little research on moral distress and moral conflict. most research on the broader topic of the distinction between right and wrong within matters relating to the profession has focused primarily on ethical dilemmas. despite the fact that research nearly 30 years ago explored “moral conflict” as experienced by librarians, recent scholarship has not expanded upon this topic much. this is particularly surprising within health and medical librarianship, as broader medical literature continues to assess related concepts of morality. this paper can help contribute to the development of a body of knowledge on morality within library and consumer health information literature.   in 1993 broderick describes 19th century librarians as self-defined “moral arbiters” (p. 447) of society, responsible for determining appropriate and inappropriate information for their constituencies. framed in the context of collection development, the central thesis of the piece is the obligation of public libraries to shirk the idea of a universal morality and acquire materials with myriad points of view on a subject. low (2002) also examines moral conflicts within collection development, specifically related to the tension film librarians can face when deciding between acquiring movies featuring a diverse array of perspectives and dominant preferences of the library’s parent company. he argues that true morality cannot exist in a library collection “without recognizing all voices, i.e. without a balance of perspectives” (p. 40).   however, ethical dilemmas have been repeatedly addressed in various library settings, including hospital, academic, and public. librarians experience ethical quandaries across departments and roles, including reference services (luo & trott, 2016), reader’s advisory (lawrence, 2020), the organization of information (mccourry, 2015), privacy/confidentiality (elliott, 2015), and rfid technology (thornley, ferguson, weckert, & gibb, 2011). some researchers are generating strategies for preparing people to resolve ethical dilemmas before they even complete their lis graduate programs. walther (2016) details the development of problem-based learning techniques to teach lis graduate students critical skills for handling ethical dilemmas in their future careers. this pedagogy is framed in part by the definition of an ethical l dilemma as occurring when “two or more moral obligations come into conflict” (walther, p.181).   murphy (2001) elucidates ethical dilemmas faced by hospital librarians, chiefly the pressure to choose between prioritizing the needs of their institution versus collective social welfare, or the mores of the broader library profession. the stakes are high here, as the actions of hospital librarians directly impact the physical and psychological well-being of patrons (patients and their loved ones). rigorous training in and dissemination of the professional ethics of the field can help this disconnect. professional codes, such as the medical library association’s code of ethics for health sciences librarianship (last updated in 2010), can play an important role in individually or collaboratively working through job-related ethical dilemmas.   in 2014 byrd, devine, and corcoran surveyed 500 mla members and learned that while 80% of respondents knew of the code, nearly one third were unaware when they last consulted it for guidance. while clearly an invaluable resource for information professionals, the code’s principles do not directly address morality within informed decision making. one participant of this study, when surveyed on key issues that the code does not explicitly cover, responded that “honesty, fairness and morality” (p. 269) should be added as principles that librarians are professionally obligated to follow.   aims   the impetus for this study was grounded first and foremost in a combination of shared professional and close personal connections with nurses, as well as professional experiences as information professionals. one author identified a feeling of moral distress caused by constraint in assisting a patron with a consumer health information inquiry, and began to construct a project to deeper examine these experiences. all three authors have encountered ethical dilemmas in the course of either providing consumer health information services or teaching research skills to nursing students.    two research questions can be used to frame this study. first, do chips experience moral distress or ethical dilemmas while performing their daily job duties?  secondly, if individuals experience moral distress or ethical dilemmas, what coping strategies, if any, have they developed?   methods   the study employed a mixed methods approach. in april 2020, the study was determined to meet the regulatory exemption for irb by memorial sloan kettering cancer center’s human research protection program under 45 cft.101(d)(2).   contemporary moral distress instruments (e.g., the moral distress scale-revised) are heavily focused on issues surrounding direct patient care, which may not be applicable to chips. as a result, we developed an instrument using the secure web application redcap.   the survey contained questions on basic demographic and occupational questions along with a series of questions designed to measure feelings of discomfort and distress within the context of consumer health information librarianship. to assess personal values, the survey also asked belief-oriented questions related to patient advocacy and empowerment. non-demographic questions were posed on a likert scale from 1-6 (1 = strongly disagree; 2 = disagree; 3 = mildly disagree; 4 = mildly agree; 5 = agree; 6 = strongly agree). the survey instrument has been included as appendix a. the survey questions were primarily intended to measure chips’ experiences of ethical dilemmas. however, because of the somewhat ambiguous nature of moral/ethical phenomena, some of the survey questions could address both ethical dilemmas and moral distress, depending on how the respondent interpreted the question. for example, for question #3 (“i feel licensing agreements with vendors prohibit me from sharing information with patients in the way i would like”) one could argue that a respondent who “agrees” or “strongly agrees” with this statement is experiencing moral distress because he/she/they feels that providing free and unencumbered access to information for all consumers is the morally correct course of action, and feels constrained by licensing agreements. this person may, on moral grounds, feel that all information should be free and that any barriers to openness are morally reprehensible. however, one could also argue that a respondent who “agrees” or “strongly agrees” is experiencing an ethical dilemma if they feel that both choices are morally acceptable and simply don’t know which to choose. this respondent might respect the legality of restrictions to proprietary information and feel they have these restrictions have value, but at the same time may wish their patrons could have free access.   the survey was disseminated in early may 2020 and remained open to responses until june 16, 2020. it was disseminated to 22 electronic mail lists geared towards medical, academic, and public library information professionals. no incentive was offered for completing the survey. stata was used to complete statistical analysis. the raw survey data has been openly deposited.    survey respondents had the option to include their contact information if they wished to participate in a follow-up interview. while the survey assessed if chips were experiencing distress in general, the aim of the interviews was to determine if the distress chips experience occurs within the context of moral distress or ethical dilemmas.   interview questions were open ended and focused on three components: 1) the participant’s understanding and personal definitions of moral distress, 2) the participant’s experience with moral distress in the context of being a consumer health information professional, 3) any coping strategies the participant had developed to manage moral distress. interviews continued until a saturation point in thematic information was reached, resulting in 14 total interviews. due to time constraints, only one author coded the interviews. interviews were manually coded in a google sheet, resulting in 21 codes. wherever possible, the author used rich or thick descriptions assessing the interview data, making the code descriptions as detailed as possible. appendix b includes the interview schedule. the interview codebook has been openly deposited with the raw data.   interviews were conducted by one author using zoom. prior to the interviews, participants received informed consent documentation. phone interviews were recorded and transcribed using tapeacall pro software. zoom interviews were recorded and manually transcribed. interviews completed by phone were automatically transcribed using the tapeacall pro transcription feature, but required some manual cleanup. interviews took place in may 2020 and june 2020.   survey population and demographics   consumer health information (chi) is defined as “information on health and medical topics provided in response to requests from the general public, including patients and their families. in addition to information on the symptoms, diagnosis and treatment of disease, chi encompasses information on health promotion, preventive medicine, the determinants of health and accessing the health care system” (reference and user services association, 2015). any professional working in this role and self-identifying as a chip qualified to participate in this study.   the authors received 213 survey responses; table 1 presents summary statistics of this sample. the majority of the respondents identified as female (81%); white (62%); had a mls, mis, or mlis degree (66.2%); or had an mls, mis, or mlis and another advanced degree (16.9%), bringing the total of respondents who had an mls, mis, or mlis to 83%. respondents were aged 41-60 (49%), and were not licensed as medical professionals (94%). ten respondents preferred not to provide either racial identity (3.3%) or gender identity (1.4%) and were excluded from the regression analyses. respondents were mainly employed by academic medical centers (43%), hospitals (28%), and public libraries (16%). the amount of experience among respondents was fairly evenly distributed, 8-20 years’ experience was most frequently reported (30%), and over 20 years’ experience the least frequently reported (21%).   in order to understand how our sample compared to the overall population, we examined data from a demographic survey of medical library association members (pionke, 2019). we found that compared to the respondents in the mla survey (n=918), our respondents identified as being less white (62% vs. 72%), slightly more female (81% vs. 79%), and were similar in age range. it should be noted, however, that only 1% of the respondents in the mla study identified “consumer health” as their primary job function.     table 1 sample characteristics   n % age     under 25 5 2.3% 25-30 16 7.5% 31-40 35 16.4% 41-50 50 23.5% 51-60 56 26.3% 61-70 40 18.8% 70+ 11 5.2% gender identity     female 173 81.2% male 33 15.5% gender non-binary 4 1.9% prefer not to say 3 1.4% racial identity     white/caucasian 133 62.4% african american/black 24 11.3% hispanic/latinx 24 11.3% asian american/asian 12 5.6% american indian/alaska native 1 0.5% native hawaiian/pacific islander 0 0.0% middle eastern/north african 2 0.9% prefer not to respond 7 3.3%% other/not provided 10 4.7% educational background     master of library and information science/master of library studies/master of information studies 141 66.2% mls, mlis, or mis and other advanced degree (i.e. other master’s degree or doctoral degree) 36 16.9% advanced degree (i.e. other master’s degree or doctoral degree), not mis, mlis, or mls 18 8.4% other 11 5.2% undergraduate degree only 7 3.3% medical license     does not have a medical license 201 94.4% does have a medical license 12 5.6% years of consumer health experience     less than 12 months 8 3.8% 1-4 years 39 18.3% 4-8 years 39 18.3% 8 years-15 years 53 24.9% 15 years-20 years 29 13.6% 20 years-25 years 22 10.3% 25 years-over 35 years 23 10.8% years in current position     less than 12 months 12 5.6% 1-4 years 60 28.2% 4-8 years 40 18.8% 8 years-15 years 43 20.2% 15 years-20 years 28 13.2% 20 years-25 years 15 7.0% 25 years-over 35 years 15 7.0% type of institution where employed     private hospital 5 2.4% not-for-profit hospital 45 21.1% community hospital 6 2.8% academic medical center 48 22.5% academic library 39 18.3% community health center 0 0% government agency 17 8.0% public library 30 14.1% unemployed 1 0.47% other 22 10.3%     results   table 2 presents average likert scores (1 = strongly disagree, 6 = strongly agree), standard deviations, and overall proportions of responses for questions relating to moral distress. high average likert scores (5+) indicate that the majority of respondents overwhelmingly agreed or strongly agreed with a particular statement. for example, respondents expressed strong preferences for patient rights (90.6% agreed or strongly agreed that patients should have access to as much health information as they wish; 92.9% agreed/strongly agreed that health professionals should take an active role in patient education and engagement). similarly, these variables report small standard deviations (less than 1), indicating that the distribution of likert scale responses is highly bunched.   conversely, low average likert scores (less than 3) were more common on questions which emphasized chip unpreparedness. over half of the sample disagreed with the statements “i often feel unable to provide patients with the health information they are looking for” and “i often worry that i lack the necessary skills, education or knowledge.” however, standard deviations on these “disagree” statements were larger (1.4 and 1.5, respectively), indicating more variability in response.   while clear majorities emerged in response to certain issues, respondents were also divided around numerous issues. a fairly even split in agreement and disagreement can be seen in response to the question regarding patients confusing the role of the information professional with the role of a health care provider and not having enough time to spend with patients. about a third of respondents said they did not feel pressured to provide prognosis information to patients, or provide them with positive information about their prognosis, but respondents were likely to feel that they often must inform patients that the information resources they have located on their own were not evidence based.   similarly, responses related to institutional pressure were mixed. while almost a quarter of respondents agreed with the statement that licensing agreements prohibited sharing information with patients, more than one quarter disagreed with this statement. even though one third of respondents did not feel torn between the different constituencies they worked with, nearly 44% reported feeling frustrated with the many roles they were expected to perform.     table 2 likert scores question % strongly disagree % disagree % mildly disagree % mildly agree % agree % strongly agree "i often feel unable to provide patients with the health information they are looking for." 16.9% n=36 37.6% n=80 10.8% n=23 17.8% n=38 14.6% n=31 2.3% n=5 "i often worry that i lack the necessary skills, education, or knowledge to provide patients with the information they are looking for." 16.9% n=36 38% n=81 10.8% n=23 15% n=32 13.1% n=28 6.1% n=13 "i feel like licensing agreements with vendors prevent me from sharing information with patients in the way i would like." 16.0% n=34 26.3% n=56 8.0% n=17 15.5% n=33 24.4% n=52 9.9% n=21 "i often feel like patients confuse my role with their health care provider." 9.4% n=20 27.2% n=58 10.3% n=22 15.5% n=33 29.1% n=62 8.5% n=18 "i often feel like i do not have adequate time to spend on search requests for patients." 10.8% n=23 28.2% n=60 10.8% n=23 14.1% n=30 26.3% n=56 9.9% n=21 "i often feel pressured to provide prognosis information or survival rates for patients." 16.4% n=35 30% n=64 12.2% n=26 11.3% n=24 24.4% n=52 5.6% n=12 "i often feel patients expect me to provide them with positive information about their prognosis." 8.5% n=18 31.0% n=66 12.2% n=26 17.8% n=38 22.1% n=47 8.5% n=18 "i often feel i must inform patients the resources they have found on their own are not evidence based, credible or reliable." 3.8% n=8 11.3% n=24 6.1% n=13 17.8% n=38 41.3% n=88 19.7% n=42 “i often feel torn between the different constituencies (e.g. patients, administrators, clinicians) with whom i work.” 7.5% n=16 33.3% n=71 9.9% n=21 16.9% n=36 22.5% n=48 9.9% n=21 "i feel frustrated with the many roles i am expected to perform." 7% n=15 25.4% n=54 7.5% n=16 16% n=34 29.1% n=62 15% n=32 "i often feel caught in the middle between trying to appease patients, caregivers, and their own health care providers." 11.7% n=25 17.8% n=38 12.2% n=26 16.4% n=35 33.3% n=71 8.5% n=18 "i am able to successfully cope with the challenges of my position." 2.3% n=5 11.3% n=24 5.6% n=12 16.4% n=35 54% n=115 10.3% n=22 "my library has an adequate budget." 8.9% n=19 26.8% n=57 8.9% n=19 14.1% n=30 33.3% n=71 8% n=17 "my library has adequate staff with expertise in providing consumer health information services." 8.4% n=18 28.2% n=60 10.3% n=22 20.2% n=43 28.2% n=60 4.7% n=10 "i am able to acquire the resources i need to meet the information needs of my users." 2.3% n=5 40.4% n=86 9.9% n=21 17.8% n=38 21.6% n=46 8% n=17 "i have been concerned for my physical or mental health during times of emergency (e.g., terrorist attacks, pandemics, natural disasters) at my library." 8.5% n=18 24.4% n=52 9.9% n=21 22.5% n=48 21.6% n=46 13.1% n=28 "the administration of my organization understands the value and importance of my library." 9.4% n=20 24.9% n=53 6.6% n=14 14.1% n=30 33.3% n=71 11.7% n=25 "i believe patients and caregivers should have access to as much health information as they wish." 0% n= 0 1.9% n=4 2.3% n=5 5.2% n=11 33.8% n=72 56.8% n=121 "i believe patients and caregivers should be active advocates for their own health care." 0% n= 0 0% n= 0 1.4% n=3 5.2% n=11 33.3% n=71 60.1% n=128 "i believe health professionals should take an active role in patient education and engagement." 0% n=0 0.5% n=1 0.5% n=1 6.1% n= 13 32.9% n=70 60.1% n=128     regression analysis   to test whether any of our demographic variables help to explain survey response, we used stata 14 to calculate pearson chi-squared tests of independence. response patterns about patient access and advocacy were systematically different by racial identity in a statistically significant manner (χ2=27.4, p=.007 and χ2=18.2, p=.033). similarly, men tended to disagree more with the idea that health professionals should take a role in educating patients, and that they were being asked to provide prognosis information (χ2=29.8, p=.000 and χ2=23.9, p=.008). figure 1 displays a spineplot for the results, by gender, for the question "i often feel pressured to provide prognosis information or survival rates for patients." figure 2 presents the question results by racial identity. spineplots were created in stata, utilizing a software package designed by cox (2008).   those who worried about being unprepared were statistically more likely to be the young (χ2=58.1, p=.000, for the variable “unprepared,” and χ2=84.1 , p=.000 for the variable “imposter”) and those in the field for shorter durations ( χ2=25 p=.049 for the variable “unprepared” and  χ2=45.7 p=.000 for the variable “imposter”). we also found that those who identify as black or latinx are more likely to feel frustrated with role confusion when it concerns their jobs as chips ( χ2=54 , p=.000). these racial identity groups also report that they feel required to provide positive responses to patron inquiries, to provide prognosis information, and are more likely to fear for their safety while at work in statistically significant patterns (χ2=84.1 , p=.000, χ2=86.6 , p=.000, and χ2=56.7 , p=.000).   interview data characteristics   fourteen interviews were conducted and interview participants were first asked to give their personal definition of moral distress. some interviewees gave definitions that closely matched the “classic” definition of moral distress, but more often their definitions were more closely aligned with a moral or ethical dilemma. some interviewees were unable to provide a clear definition, but instead referred to a survey question that particularly resonated with them. this resulted in 4 original definitions provided during the 14 interviews.      figure 1  respondents’ perceptions of providing prognosis information by gender.     figure 2 respondents’ perceptions of providing prognosis information by racial identity.     the authors asked interview participants to identify how they experienced moral distress in their work as chips, and several themes emerged. the first centered around role confusion, wherein consumers do not understand the purpose of the consumer health library or the role of the consumer health information professional, and do not recognize that the consumer health information professional is not a health professional. respondents noted that this role confusion usually manifests in patrons seeking specific medical advice or recommendations from them, including dosing information for medications or interpreting medical test results.   several interview participants reported feeling discomfort when having to inform patients that the information they have found on their own was not evidence based. in doing their own health information research, consumers may find health information and then desire “confirmation” from a consumer health information professional that the information they have found is in fact evidence based. respondents report this is more common when patients are seeking alternative or integrative therapies in place of, rather than in complement to, traditional medicine. another common source of internal conflict among the interview participants related to being asked to provide prognosis information including survival rates/outcomes:   "i have that mental list of diagnoses that i want people to not ask me. because i know what the situation looks like, pancreatic cancer, for example. i hate it. especially in a case like that, when survival rates aren't good, but they don't know that yet. so they are just looking at treatment situations or whatever. and i never know how far to go. like should i be offering them information on palliative care?”   several interview participants raised the issue of providing “too much” information, particularly when assisting patients who were newly diagnosed with a condition that has particularly dire outcomes. participants reported not wanting to “overwhelm” their patrons with information, but feeling that not providing them with the level of information requested would amount to censorship:   "i… worry about inadvertently being a censor, not providing them with enough information for them to make a health care decision because i know they're not at a place where they can do that effectively."   a lack of available information on a particular health topic was also often a source of discomfort for interview participants. in this instance, assisting individuals with rare diseases and conditions can be particularly challenging. other participants reported frustration with being unable to find health information resources available in languages other than english.   interview participants described several instances wherein they struggled with institutional policies to remain “neutral:”   “the library can't recommend one [health care] facility over another, even though i might know that one facility has a worse record on something. and that i struggle with, too, so i'm always saying get a second opinion, look at other places. here are the statistics."   one interview participant described a scenario wherein despite being hired as a “community health librarian,” she had been constrained by the administration of her institution from actively providing services to the local community and was instead relegated to providing services to a lower need community that was directly affiliated with her institution (a university).   while not directly related to ethical constraints or dilemmas, interview participants reported several scenarios related to their work as chips that caused them to feel a sense of generalized distress. two interview participants felt they lacked the necessary training and skills to function competently in their positions and that they experienced feeling of inadequacy and stress. finally, interview participants reported that simply being exposed to the stress of patients, caregivers, and family members can be upsetting, particularly if they have not received adequate training to cope with these types of stressors:   "they come in with a diagnosis, and i'll help them, it's not a good diagnosis, and they'll be upset. you're trying to help them, and they start crying. they're visibly upset, which makes me upset."   interview participants identified several coping strategies for managing their experiences of moral distress, as well as emotional distress in the professional setting. six interview participants reported relying on a network of professionals for additional support when the patron they were helping was in distress or if the patron asked for resources/information beyond the scope of the consumer health information professional’s role. these professionals include social workers, patient advocates, volunteer services, patient educators, and dieticians. one respondent noted the benefits of support from health care providers:   "it's good to have nurse colleagues who can help me process things, and know how to deal with weird situations, like being pulled into people's medical and legal issues. as librarians we want to help, so it's helpful because they know where to refer people for things like living wills."    participants reported strengthening their professional networks over time, as they became more familiar with institutional resources and personnel. several participants reported that they simply felt less distress as they gained familiarity with the types of encounters and requests that typically upset them, and as they became more comfortable with the demands of their role and their surroundings. using a disclaimer (either verbally or in a written form such as a sign) which described the role of the consumer health information professional and its limitations was also reported.   working to ensure patients are effective advocates for their own health was another coping strategy reported by participants. one subject described alleviating discomfort by encouraging the patients he worked with who were feeling overwhelmed by their diagnosis to write down specific questions they have for their health care provider and to practice asking them aloud. other participants reported encouraging patients to bring research studies or consumer health information they had located to their health care provider. other coping strategies were less frequently reported and included indulging in escapist entertainment, using reflective practice (e.g., journaling), using institutional staff assistance programs, and actively circumventing bureaucratic systems to aid their patrons.   discussion   the study finds that chips do experience generalized distress within their professional roles, and in some cases this distress appears to be directly related to the nature of their role. for example, one interview participant described her struggle with providing information on morgellons disease, a controversial condition which many health professionals describe as a form of delusional parasitosis, but has also been described by a smaller group of medical professionals as a legitimate dermatological condition. the controversial nature of the condition left this interview participant feeling torn between her patrons who were convinced they suffered from the condition and the lack of evidence that the condition actually exists in the physical sense. chips exist within a sort of interspace, with significantly more expertise of health information than the average consumer, but frequently without the licensure, education, and hands on knowledge of medical professionals. navigating this interspace may prove challenging, particularly for individuals who are already faced with navigating organizational power structures and systemic pathologies (ex., racism, ageism).   indeed, the distress experienced by consumer health professionals appears to be related at least in part to the level of support, or lack thereof, that they receive from their institution at large. while the majority of respondents felt they were able to successfully cope with the challenges of their positions (54%), one third reported that their library had an inadequate budget (33%), inadequate staffing to support consumer health information services (33%), and that the administration of their organization did not understand the value and importance of their library (33.3%). about a third (34.7%) of respondents also reported being concerned for their physical or mental health during times of an emergency.   while chips appear to experience distress, it is beneficial to distinguish between distress that occurs in the course of one’s occupation and distress caused by an ethical dilemma in a professional context. again, an ethical dilemma is a situation in which two moral principles conflict with one another. not all the scenarios described by the interview participants were true ethical dilemmas, but some were, including concerns about censorship, providing prognosis information, and feeling constrained by institutional policies. the latter phenomenon, a feeling of institutional constraint, is associated with moral distress, but our interview participants were more likely to experience scenarios with competing moral drawbacks, rather than one obvious morally superior option. the most durable patterns of these experiences of ethical dilemmas appear to be related to experience level and racial identity.   chips do not appear, though, to experience moral distress, at least according to its narrow definition (knowing the correct action to take, but being constrained from doing so by external forces). why don’t chips experience moral distress? one reason may be that the key component of moral distress, as traditionally defined, of “constraint” is less likely to be present. it may be that case that the constraint chips experience within the course of their profession is felt less acutely than frontline medical professionals, such as nurses, who are directly responsible for the health and safety of patients. it also appears that chips may confuse moral distress and ethical dilemmas, or conflate the two.   if moral distress is defined more broadly, as suggested by fourie (2015), one could argue that chips do indeed experience a degree of moral distress. fourie argues that we should recognize that “constraint is not a necessary condition of moral distress and that such distress can arise from morally troubling situations other than those of moral constraint” (p. 580) and that moral distress should be expanded to include experiences related to moral uncertainty.   limitations   the authors acknowledge that the study is limited by several factors. first, because there was no validated instrument to measure moral distress and ethical dilemmas among consumer health professionals, we did not use a validated tool. in future research, a validated tool may aid in further distinguishing between the nuanced and complex experiences of moral distress and ethical dilemmas. the purposive sampling technique used (leveraging electronic mail lists of interest to chips) to identify potential survey respondents may have resulted in a degree of selection bias. finally, only one interviewer coded the interviews. using two or more coders in future studies would reduce the potential for bias when identifying interview themes, as long as proper interrater reliability protocols are implemented.   further research   while this study provides burgeoning insight into the exploration of moral distress among chips, additional research is needed to validate and expand on these findings to draw broader conclusions.   first, it would be valuable for outside researchers to apply the survey instrument developed by the authors to their own samples. the instrument was used for the first time in the present study, and did not undergo validity or reliability calculations. this process would help to ensure the instrument is consistent across applications, measures what it intends to measure, and that results can be extrapolated to a broader population.   next, the results of the survey indicated that participants who are black or latinx experience greater distress in the chi profession than people of other racial identities. these statistics are concerning and need further investigation in efforts to identify and ameliorate any racism or microaggressions causing this distress. while there aren’t similar studies within lis scholarship to compare these findings dyo, kalowes, and devries (2016) found that hispanic nurses reported much higher rates of moral distress than other ethnic groups, “suggesting that culture and ethnicity may play a role in the perception and experience of moral distress” (p. 1). a pertinent solution identified within nursing literature to address this problem was to begin studying moral distress as experienced by non-western nurses (prompahakul & epstein, 2020), a project that could easily be replicated with chips or librarians.   conclusions   this study examined how chips experience moral distress, ethical dilemmas, and the use of coping strategies for managing the negative impacts of these phenomena. while the study finds that chips do not appear to experience moral distress according to its narrow definition focused on constraint, study results indicate that chips do experience ethical dilemmas in the course of their work. the most durable patterns of ethical distress experienced by chips appear to be related to experience level and racial identity, with younger, black, and latinx chips experiencing ethical dilemmas at higher rates. further study is needed to determine why there is a statistically significant relationship between these groups and their experiences with ethical distress. the interview data further elucidates how chips interpret the phenomenon of moral distress and how this term is sometimes confused with ethical dilemmas. this issue could be ameliorated by professional associations creating a module on integral ethical codes of their area of librarianship and encouraging libraries to include participation in the module in onboarding for new hires. additionally, library and information science graduate programs can build greater content on morals and professional ethics into their foundational courses. finally, while the experiences of the study participants do not fit the classic definition of moral distress, which is characterized by the presence of constraint, they do align with a more broadly defined version of moral distress. this definition, as described by fourie (2015) allows for the inclusion of the experience of uncertainty to co-exist with the experience of constraint in moral contexts.   acknowledgements   the authors would like to thank all survey respondents and interview participants. many thanks to kendra godwin for her assistance with reviewing the survey prior to distribution, and thanks to tedi brash, aman kaur, barnaby nicolas, robb mackes, and shawn stedinger for their assistance with survey distribution.   the authors would like to thank sam stabler for his assistance with statistical analysis and creating the spineplots.   author contributions   robin o’hanlon: methodology, formal analysis, data curation, visualization, writing – original draft (equal), writing – review & editing (equal) katelyn angell: methodology, writing – original draft (equal), writing – review & editing (equal) samantha walsh: methodology, writing – original draft (equal), writing – review & editing (equal)   references   broderick, d. m. (1993). moral conflict and the survival of the public library. american libraries, 24(5), 447–448. https://www.jstor.org/stable/i25632891   byrd, g. d., devine, p. j., & corcoran, k. e. (2014). health sciences librarians' awareness and assessment of the medical library association code of ethics for health sciences librarianship: the results of a membership survey. journal of the medical library association: jmla, 102(4), 257–270. https://doi.org/10.3163/1536-5050.102.4.007    camp, m., & sadler, j. (2019). moral distress in medical student reflective writing. ajobempirical bioethics, 10(1), 70–78. https://doi.org/10.1080/23294515.2019.1570385   corley, m.c. (1995). moral distress of critical care nurses. american journal of critical care, 4(4), 280–285. https://doi.org/10.4037/ajcc1995.4.4.280   cox, n. j. (2008). speaking stata: spineplots and their kin. the stata journal, 8(1), 105-121. https://doi.org/10.1177/1536867x0800800107   dyo, m., kalowes, p., & devries, j. (2016). moral distress and intention to leave: a comparison of adult and paediatric nurses by hospital setting. intensive & critical care nursing, 36, 42-48. https://doi.org/10.1016/j.iccn.2016.04.003   dzeng, e., colaianni, a., roland, m., levine, d., kelly, m., barclay, s., & smith, t. (2015). moral distress amongst american physician trainees regarding futile treatments at the end of life: a qualitative study. journal of general internal medicine, 31(1), 93-99. https://doi.org/10.1007/s11606-015-3505-1   elliott, k. (2015). planning for barriers in hospital library social media implementation. journal of hospital librarianship, 15(1), 13–22. https://doi.org/10.1080/15323269.2015.982013   fourie, c. (2015). moral distress and moral conflict in clinical ethics. bioethics, 29(2), 91–97. https://doi.org/10.1111/bioe.12064   hlubocky, f. j., spence, r., mcginnis, m., taylor, l., & kamal, a. h. (2020). burnout and moral distress in oncology: taking a deliberate ethical step forward to optimize oncologist well-being. jco oncology practice, 16(4), 185–186. https://doi.org/10.1200/op.20.00030   jameton, a. (1984). nursing practice: the ethical issues. prentice hall.   jormsri, p. (2004). moral conflict and collaborative mode as moral conflict resolution in health care. nursing & health sciences, 6(3), 217-221. https://doi.org/10.1111/j.1442-2018.2004.00191.x   lawrence, e. e. (2020). on the problem of oppressive tastes in the public library. journal of documentation, 76(5), 1091–1107. https://doi.org/10.1108/jd-01-2020-0002   low, d. (2002). moral conflict for the film librarian. journal of information ethics, 11(2), 33-45. https://ir.uwf.edu/islandora/object/uwf:23834   luo, l., & trott, b. (2016). ethical issues in reference: an in-depth view from the librarians’ perspective. reference & user services quarterly, 55(3), 189–198. https://doi.org/10.5860/rusq.55n3.188    marshall, m. f., & epstein, e. g. (2016). moral hazard and moral distress: a marriage made in purgatory. the american journal of bioethics: ajob, 16(7), 46–48. https://doi.org/10.1080/15265161.2016.1181895   mccourry, m. w. (2015). domain analytic, and domain analytic-like, studies of catalog needs: addressing the ethical dilemma of catalog codes developed with inadequate knowledge of user needs. knowledge organization, 42(5), 339–345. https://doi.org/10.5771/0943-7444-2015-5-339   medical library association (2010). code of ethics for health sciences librarianship. https://www.mlanet.org/page/code-of-ethics       morley, g., ives, j., bradbury-jones, c., & irvine, f. (2019). what is ‘moral distress’? a narrative synthesis of the literature. nursing ethics, 26(3), 646–662. https://doi.org/10.1177/0969733017724354    moses, l., malowney, m. j., & wesley boyd, j. (2018). ethical conflict and moral distress in veterinary practice: a survey of north american veterinarians. journal of veterinary internal medicine, 32(6), 2115–2122. https://doi.org/10.1111/jvim.15315   murphy, s.a. (2001). the conflict between professional ethics and the ethics of the institution. journal of hospital librarianship, 1(4), 17-30. https://doi.org/10.1300/j186v01n04_02   oh, y., & gastmans, c. (2015). moral distress experienced by nurses: a quantitative literature review. nursing ethics, 22(1), 15–31. https://doi.org/10.1177/0969733013502803   ong, w. y., yee, c. m., & lee, a. (2012). ethical dilemmas in the care of cancer patients near the end of life. singapore medical journal, 53(1), 11–16. https://pubmed.ncbi.nlm.nih.gov/22252176/    pionke, jj. (2019). medical library association diversity and inclusion task force 2019 survey report. journal of the medical library association, 108(3), 503–512. https://doi.org/10.5195/jmla.2020.948    prompahakul, c., & epstein, e. g. (2020). moral distress experienced by non-western nurses: an integrative review. nursing ethics, 27(3), 778–795. https://doi.org/10.1177/0969733019880241 reference and user services association (2015). health and medical reference guidelines. http://www.ala.org/rusa/resources/guidelines/guidelinesmedical     rodger, d., blackshaw, b., & young, a. (2019). moral distress in healthcare assistants: a discussion with recommendations. nursing ethics, 26(7–8), 2306–2313. https://doi.org/10.1177/0969733018791339   sajjadi, s., norena, m., wong, h., & dodek, p. (2017). moral distress and burnout in internal medicine residents. canadian medical education journal, 8(1), e36–e43. https://pubmed.ncbi.nlm.nih.gov/28344714/   thornley, c., ferguson, s., weckert, j., & gibb, f. (2011). do rfids (radio frequency identifier devices) provide new ethical dilemmas for librarians and information professionals? international journal of information management, 31(6), 546–555. https://doi.org/10.1016/j.ijinfomgt.2011.02.006   walther, j. h. (2016). teaching ethical dilemmas in lis coursework: an adaptation on case methodology usage for pedagogy. the bottom line, 29(3), 180-190. https://doi.org/10.1108/bl-05-2016-0020   whitehead, p. b., herbertson, r. k., hamric, a. b., epstein, e. g., & fisher, j. m. (2015). moral distress among healthcare professionals: report of an institution-wide survey. journal of nursing scholarship: an official publication of sigma theta tau international honor society of nursing, 47(2), 117–125. https://doi.org/10.1111/jnu.12115     appendix a survey instrument   how old are you? under 25 25-30 31-40 41-50 51-60 61-70 over 70   what is your gender? female male gender non-binary prefer not to say   which of these best describes your racial identity? african american/black american indian/alaska native asian american/asian native hawaiian/pacific islander hispanic/latinx middle eastern/north african white/caucasian other/not provided prefer not to respond   what is your educational background? master of information science master of library and information science master of library science advanced degree (i.e., other masters degree or doctoral degree), not mis, mlis, or mls mls, mlis, or mis and other advanced degree (i.e., other masters degree or doctoral degree) undergraduate degree only other if "other," please describe.   are you currently, or have you ever been, a licensed medical professional (e.g. registered nurse, medical doctor)? yes no   if you are currently or have been licensed medical professional in the past, please describe:   years of consumer health librarian experience/providing health information to the public: less than 12 months 1 year 2 years 2 years 4 years 4 years 6 years 6 years 8 years 8 years 10 years 10 years 12 years 12 years 15 years 15 years 20 years 20 years 25 years 25 years 35 years over 35 years   years in current position: less than 12 months 1 year 2 years 2 years 4 years 4 years 6 years 6 years 8 years 8 years 10 years 10 years 12 years 12 years 15 years 15 years 20 years 20 years 25 years 25 years 35 years over 35 years   type of institution where you are employed: private hospital not-for-profit hospital community hospital academic medical center academic library community health center government agency public library unemployed other if "other," please describe. for the remaining questions, please select one of the following values that best describes how you feel about each statement below:   strongly disagree disagree mildly disagree mildly agree agree strongly agree   1)      i often feel unable to provide patients with the health information they are looking for. 2)      i often worry that i lack the necessary skills, education, or knowledge to provide patients with the information they are looking for. 3)      i feel licensing agreements with vendors prohibit me from sharing information with patients in the way i would like. 4)      i often feel that patients confuse my role with their health care provider. 5)      i often feel i do not have adequate time to spend on search requests for patients. 6)      i often feel pressured to provide prognosis information or survival rates for patients. 7)      i often feel patients expect me to provide them with positive information about their prognosis. 8)      i often feel i must inform patients the resources they are have found on their own are not evidence based, credible, or reliable. 9)      i often feel torn between the different constituencies (e.g. patients, administrators, clinicians) with whom i work. 10)   i feel frustrated with the many roles i am expected to perform. 11)   i often feel caught in the middle between trying to appease patients, caregivers, and their health care providers. 12)   i am able to successfully cope with the challenges of my position. 13)   my library has an adequate budget. 14)   my library has adequate staff with expertise in providing consumer health information services. 15)   i am able to acquire the resources i need to meet the information needs of my users. 16)   i have been concerned for my physical or mental health during times of emergency (e.g. terrorist attacks, pandemics, natural disasters) at my library. 17)   the administration of my organization understands the value and importance of my library. 18)   i believe patients and caregivers should have access to as much health information as they wish. 19)   i believe patients and caregivers should be active advocates for their own health care. 20)   i believe health professionals should take an active role in patient education and engagement.   contact information (optional) if you are willing to participate in a phone interview about moral distress and consumer health librarianship, please include your contact information (name and email address). any information professional who provides health information to the public can participate. if you decide to participate in a phone interview, your information will by anonymous in the final publication.   please only include your contact information if you are interested in participating in a phone interview. if you are not interested in participating in a phone interview, leave this section blank.     appendix b interview schedule   can you define “moral distress”? a) do you feel you have ever experienced moral distress in your role as a consumer health librarian (or as an information professional who provides health information to the public)? b) if yes, in what ways have you experienced moral distress in your role as a consumer health librarian (or as an information professional who provides health information to the public)? c) if yes, how has your experience of moral distress affected your ability to function in your job and your attitude toward your job? a) if you have experienced moral distress in the course of your profession, what factors have contributed to your distress (e.g., number of years of experience, type of patient)? b) if you have not experienced moral distress in your profession, how do you feel you have avoided this phenomenon? a) if you have experienced moral distress in the course of your profession, have you employed to lessen your experience of moral distress?                b) if you have employed coping strategies, which strategies did you find the most effective and why?       evidence summary   researchers at arab universities hold positive views on research data management and data sharing   a review of: elsayed, a. m., & saleh, e. i. (2018). research data management and sharing among researchers in arab universities: an exploratory study. ifla journal, 44(4), 281–299. https://doi.org/10.1177/0340035218785196   reviewed by: jennifer kaari librarian east orange public library east orange, new jersey, united states of america email: jkaari@eopl.org   received: 29 feb. 2020                                                              accepted:  20 apr. 2020      2020 kaari. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29746     abstract   objective – to investigate researchers’ practices and attitudes regarding research data management and data sharing.   design – email survey.   setting – universities in egypt, jordan, and saudi arabia.   subjects – surveys were sent to 4,086 academic faculty researchers.   methods – the survey was emailed to faculty at three arab universities, targeting faculty in the life sciences and engineering. the survey was created using google docs and remained open for five months. participants were asked basic demographic questions, questions regarding their research data and metadata practices, and questions regarding their data sharing practices.   main results – the authors received 337 responses, for a response rate of 8%. the results showed that 48.4% of respondents had a data management plan and that 97% were responsible for preserving their own data. most respondents stored their research data on their personal storage devices. the authors found that 64.4% of respondents reported sharing their research data. respondents most frequently shared their data by publishing in a data research journal, sharing through academic social networks such as researchgate, and providing data upon request to peers. only 5.1% of respondents shared data through an open data repository.  of those who did not share data, data privacy and confidentiality were the most common reasons cited. of the respondents who did share their data, contributing to scientific progress and increased citation and visibility were the primary reasons for doing so. a total of 59.6% of respondents stated that they needed more training in research data management from their universities.   conclusion – the authors conclude that researchers at arab universities are still primarily responsible for their own data and that data management planning is still a new concept to most researchers. for the most part, the researchers had a positive attitude toward data sharing, although depositing data in open repositories is still not a widespread practice. the authors conclude that in order to encourage strong data management practices and open data sharing among arab university researchers, more training and institutional support is needed.   commentary   the issues surrounding open data sharing and data management are important topics of discussion in the scientific and scholarly community. studies have found that acceptance of data sharing and a willingness to share their own data has been increasing among researchers (tenopir et al., 2015). however, many researchers have reservations regarding their own skills and knowledge regarding research data management, as well as increasing concerns about the risk of openly sharing data (perrier & barnes, 2018). most studies regarding these issues have examined the north american or european contexts; this study provides valuable insights as the first study of research data management and data sharing practices in the arab world.   this study had an 80% validity rating when examined using glynn’s critical appraisal tool for library and information research (2006). the methodology and results are well-described. the full survey instrument is provided as an appendix. the survey questions are very thorough and well-designed to yield precise and comprehensive answers to the research questions presented.   this survey was sent to researchers at only three universities and the response rate was low. the authors identify this as a limitation of their study, a problem which is compounded by the number of non-functional email addresses the authors encountered. it’s also worth noting that although the full instrument is available, the full data is not openly available. given that this study is the first of its kind, providing the full data so that future researchers can build on the results of this survey would be particularly helpful.   academic librarians working in arab universities or with interest in the global state of data management will find this study to be informative, although the current implications for practice remain limited. this study has the greatest value as a baseline for future research. the authors suggest many areas that may be pursued including an expanded study population and research into other topics suggested by the results, including the relationship between research data management, and arab academic libraries. in addition, a comparison between the results from this survey with similar studies in other geographic and social contexts would also be potentially illuminating.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech 24(3), 387–399. https://doi.org/10.1108/07378830610692154   perrier, l., & barnes, l. (2018). developing research data management services and support for researchers: a mixed methods study. partnership: the canadian journal of library and information practice and research 13(1). https://doi.org/10.21083/partnership.v13i1.4115   tenopir, c., dalton, e., allard, s., frame, m., pjesivac, i., birch, b., pollock, d., & dorsett, k. (2015). changes in data sharing and data reuse practices and perceptions among scientists wworldwide. plos one, 10(8).  https://doi.org/10.1371/journal.pone.0134826       evidence summary   the form of search tool chosen by undergraduate students influences research practices and the type and quality of information selected   a review of: asher, a. d., duke, l. m., & wilson, s. (2012). paths of discovery: comparing the search effectiveness of ebsco discovery service, summon, google scholar, and conventional library resources. college & research libraries, 74(5), p. 464-488. retrieved from http://crl.acrl.org/content/early/2012/05/07/crl-374.full.pdf   reviewed by: michelle dalton liaison librarian university college dublin dublin, ireland email: michelle.dalton@ucd.ie      received: 17 jan. 2014   accepted: 22 may 2014      2014 dalton. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objectives – to explore the effectiveness of different search tools (ebsco discovery service (eds), summon, google scholar and traditional library resources) in supporting the typical research queries faced by undergraduate students and gain an understanding of student research practices.   design – mixed methods approach using quantitative data collected from grading of students’ selected resources combined with qualitative data from a search process interview with students.   setting – two university libraries in the united states of america (bucknell university (bu) and illinois wesleyan university (iwu)).   subjects – eighty-seven undergraduate students across a range of disciplines.   methods – participants were assigned to one of five test groups and required to find two resources for each of four standardised research queries using a specified tool: eds; summon; google scholar; library catalogue/databases; or “no tool” where no specific tool was specified and participants were free to choose. the resources submitted by students for each of the four queries were rated on a scale of 0-3 by four librarians using a rubric, to produce average ratings for each tool. the interview comprised two parts: the search task, followed by a reflective interview based on open-ended questions relating to search practices and habits. the search process interview was recorded using camtasia screen capture and audio software, and the urls used by participants were also recorded.   main results – quantitative results indicated that students who used eds selected slightly higher quality sources on average (scoring 2.54 out of 3), compared to all other groups. those who used eds also completed the queries in less time (747 seconds) than those using summon (1,209 seconds), google scholar (968 seconds), library databases (963 seconds) or where no tool was specified (1,081 seconds). academic journal articles also represented the relatively highest proportion of resources for this group (73.8% of resources chosen), whilst newspaper articles were chosen most frequently by those using summon (20.6% of resources chosen). the qualitative findings suggest that students may over-rely on the top results provided by search systems, rather than using critical analysis and evaluation.   conclusion – although eds performed slightly better overall, in some cases the tools produced relatively similar results, and none of the tools performed particularly poorly. indeed the reasonably strong performance of both google scholar and traditional library tools/databases in some aspects (such as the relative proportion of books and journal articles chosen by students), may raise questions regarding the potential benefit of acquiring a new discovery product, given the possibly significant costs involved. as the study finds that most students do not go beyond simple searches and the first page of results, regardless of the tool they are using, this suggests that discovery services do not substantially lessen the need for information literacy instruction, although it may provide some opportunity to redirect teaching time away from retrieval and towards higher-order skills such as evaluating information and critical thinking.     commentary   discovery often “tops the charts as the foremost issue” in library systems and automation (breeding, 2010, p. 31). faced with increasing competition from web-scale search tools such as google, many academic libraries have looked towards discovery services as a solution that can match their users’ needs and preferences. however, no clear consensus has yet emerged regarding the best product available in this space, or indeed whether these tools are ultimately more efficient than using traditional library tools such as individual databases. in this context, the study addresses an important and emerging question, by comparing the efficacy of a suite of tools in dealing with undergraduate information queries.   trying to make direct comparisons between search platforms that are used across two different institutions is both complex and potentially problematic due to the likely variation in holdings and resources. however, the authors clearly acknowledge this weakness and openly refer to aspects which may undermine any inferences. for instance, the eds product in iwu did not automatically index lexisnexis whilst summon in bu did. as elements and defaults can usually be customised to reflect institutional preferences, the results may have been more meaningful if both tools were configured in a similar way for the purposes of the study. other possible differences, including variation in the information skills levels of students between institutions, are also flagged.   however, the level of detail in the study is exceptional, offering break downs by page views, number of searches, time taken and resource type. this kind of information provides granular and detailed data which can inform usability analysis and information literacy instruction. that eds outperformed google scholar in terms of length of time taken (as well as the quality of material selected), may indicate that discovery platforms can potentially compete with the single search box experience of google when it comes to ease and speed of use, and indeed user experience more generally.   in those instances where significant differences were found between products (such as the typical proportion of resources selected which are scholarly articles, books or websites), it highlights the influence that the tools that libraries provide, promote and recommend may have on the information and content ultimately selected and used by students. websites were selected much more frequently by those given no explicit directions regarding which resource to use and this suggests that undergraduate students still require significant guidance on where to start searching.   the qualitative data in particular provides rich insight into students’ thought processes and how they select and evaluate sources – an aspect that is often overlooked in favour of retrieval. in this context, the study offers a valuable perspective that extends beyond many of the existing studies surrounding discovery which are purely quantitative (lown et al., 2013; chapman et al., 2013). from the comments included, in many cases there is a generally good awareness of what constitutes a quality source of information. notwithstanding this, the depth of knowledge in this respect appears limited, indicative that true understanding and deeper critical evaluation skills may be an opportunity for further development through library support. the intuitive nature and interface of many discovery products, means that time previously devoted to database instruction could be redirected towards other areas. indeed, this is perhaps one of the biggest potential benefits that discovery may open up for libraries: by simplifying our retrieval systems for users, it provides them with more time to explore the full range of our resources and services, as well as the information that they find.     references   breeding, m. (2010). the state of the art in library discovery 2010. computers in libraries, 30(1), 31-34. retrieved from http://www.librarytechnology.org/ltg-displaytext.pl?rc=14574   chapman, s., desai, s., hagedorn, k., varnum, k., mishra, s., & piacentine, j. (2013). manually classifying user search queries on an academic library web site. journal of web librarianship, 7(4), 401-421. http://dx.doi.org/10.1080/19322909.2013.842096   lown, c., sierra, t., & boyer, j. (2013). how users search the library from a single search box. college & research libraries, 74(3), 227-241. retrieved from http://crl.acrl.org/content/74/3/227.full.pdf   evidence based library and information practice evidence based library and information practice 2010, 5.4 1 evidence based library and information practice editorial can scholarly communication be evidence based? denise koufogiannakis editor-in-chief collections & acquisitions coordinator, university of alberta libraries edmonton, alberta, canada email: denise.koufogiannakis@ualberta.ca 2010 koufogiannakis. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. this issue of evidence based library and information practice includes three papers from the evidence based scholarly communication conference (ebscc) that took place in march 2010i . kroth, philips and eldredge have written a commentary that gives an overview of the conference, and introduces us to the research papers that were presented. as well, two research presentations from the conference appear in this issue, an article by donahue about a potential new method of communicating between scholars, and a paper by gilliland in our using evidence in practice section, detailing a library’s open access day preparations. kroth, philips and eldredge note that “the ebscc brought together librarians and information specialists to share evidencebased strategies for developing effective local scholarly communication support and training and, hopefully, form new coalitions to address this topic at a local and national level.” (p 108). this conference focused on translational medicine, and looked at how to promote new methods of scholarly communication, partially through the inclusion of research papers at the conference. the inclusion of these articles and the evidence based focus of the ebscc conference, made me ask myself, can scholarly communication be evidence based? at its core, scholarly communication is anything but a scientific issue. it is charged with emotion; from authors, publishers, librarians and others involved in the business of publishing. the recent shift to look at new models of scholarly communication has been a threat to many of the established models and sparked much debate in the academic world, especially in relation to open access. in her 2006 eblip commentary on evidence based practice and open access, morrison notes, “open access and evidence based librarianship are a natural combination” (p. 49), and outlines her perspective on many of the reasons why. debate continues to rage, however, regarding how authors should disseminate their work, the role of research funding bodies that use public funds, the added value brought to the work by publishers, the role of peer reviewers, the economics of various models, and simply what works best. research has been done on many of these issuesii and much of that work has then been mailto:denise.koufogiannakis@ualberta.ca� evidence based library and information practice 2010, 5.4 2 critically appraised and debated postpublication on mailing listsiii and social networking media such as blogs.iv the bmj is one scholarly publication that has committed itself to becoming an “evidence based publisher” and is carrying out research on aspects of scholarly publishing to help guide their processes (schroter, n.d.). research on scholarly communication is a hot topic indeed; and for librarians, an area of information overload if there ever was one. how to sort out the good from the bad; the research that is high quality from that which is biased? at this point in time, it is my view that the research does not yet provide a definitive answer for how libraries should approach new models of scholarly communication. we are in the middle of a debate, in the middle of a surge of research, and an ever-changing lens in which we view and approach this topic. but evidence based practice has always been about more than just research – it considers what is needed by our users, and is guided by our professional judgement. putting those elements together allows us to sort through the research and make informed decisions about our approach to collections, and how we do liaison work. for anyone looking for a research idea, there are certainly a couple of systematic reviews possible on these issues that would benefit practitioners immensely. the decision to start eblip was not an evidence based one. it was based in a desire to give the topic a home for discussion, and that in order to facilitate discussion, the widest audience possible must be reached. hence, barriers such as cost needed to be reduced, and the decision to be open access was made. this was a decision based on principles of access, sharing and broad communication. and so far it has worked, with a parade of volunteers who contribute their time, energy and talents. it was really about taking a chance on something that seemed worthwhile, but which the community of lis professionals would determine the worth of by their contribution. that contribution has exceeded expectations, both from the perspective of submissions received, and the ever-increasing number of volunteers who inspect and polish the content to make it shine. so, eblip is a case where open access has worked. it is one drop in the ocean of conflicting evidence on the topic. my conclusion, then, is not straightforward. decisions around scholarly communication can be evidence based, but the political, personal, economic, and social factors all make the issue a complicated one. this seems to be an area where the best research is not necessarily the main concern. ensuring that quality research remains a part of the conversation is what journals such as this one can do to help provide answers, but in the end those who contribute intellectual content to the process, and those who spend money to enable access to that content for their communities, will ultimately decide what happens. i hope research evidence will be part of that determination. references morrison, h. (2006). evidence based librarianship and open access. evidence based library and information practice, 1(2), 46-50. retrieved from http://ejournals.library.ualberta.ca/index.php/ eblip/article/view/49/117 schroter, sara. (n.d.) the bmj’s ongoing programme of editorial research. retrieved november 5, 2010, from http://resources.bmj.com/bmj/aboutbmj/evidence-based-publishing i evidence-based scholarly communications conference: empowering information professionals to unlock translational research for our communities. http://hsc.unm.edu/library/ebscconference/ind ex.shtml ii two sources noting recent research are the monthly sparc open access newsletter which emphasizes open access http://www.earlham.edu/~peters/fos/; and charles w. bailey, jr. produces a yearly bibliography on scholarly electronic http://hsc.unm.edu/library/ebscconference/index.shtml� http://hsc.unm.edu/library/ebscconference/index.shtml� http://www.earlham.edu/~peters/fos/� evidence based library and information practice 2010, 5.4 3 publishing http://www.digitalscholarship.org/sepb/annual/annual.htm. iii liblicense-l is one example of a mailing list with heavy discussion of scholarly communication issues, with contributions from a variety of perspectives. iv as examples, see , http://poeticeconomics.blogspot.com/, http://oalibrarian.blogspot.com/, http://www.earlham.edu/~peters/fos/, http://blogs.plos.org/mfenner/, http://www.digital-scholarship.org/sepb/annual/annual.htm� http://www.digital-scholarship.org/sepb/annual/annual.htm� http://poeticeconomics.blogspot.com/� http://oalibrarian.blogspot.com/� http://www.earlham.edu/~peters/fos/� http://blogs.plos.org/mfenner/� / evidence based library and information practice evidence based library and information practice evidence based library and information practice 2011, 6.2 36 evidence based library and information practice evidence summary users of virtual reference are more satisfied with the service they receive than the providers of that service think they are a review of: hansen, d., johnson, m., norton, e., & mcdonough, a. (2009). virtual provider pessimism: analysing instant messaging reference encounters with the pair perception comparison method. information research, 14(4). retrieved from http://informationr.net/ir/14-4/ paper416.html reviewed by: carol d. howe reference librarian/assistant professor gabriele library immaculata university immaculata, pennsylvania, united states of america email: chowe@immaculata.edu received: 23 feb. 2011 accepted: 29 mar. 2011 2011 howe. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to examine provider pessimism by comparing user and provider perception of the same instant messaging reference transaction. design – instant messaging users and providers completed post-reference transaction surveys which were analyzed using the pair perception comparison method. setting – a large research university in the united states. subjects – two hundred undergraduate journalism students (users of the instant messaging service) and 51 master of library science (mls) students enrolled in a reference services class (providers of the instant messaging service). methods – the authors created a research help webpage from which users could access the instant messaging service. prior to service availability, providers received reference instruction and demonstrated reference aptitude through in-class activities. the authors briefed providers on the project and provided a wiki containing resources they might need during reference transactions. http://informationr.net/ir/14-4/%20paper416.html� mailto:chowe@immaculata.edu� evidence based library and information practice 2011, 6.2 37 providers worked in two-hour shifts, and two providers were available during each shift. the service was available for one week while potential users completed a journalism assignment. the authors asked both users and providers of the service to complete an online survey at the conclusion of the reference transaction. users and providers completed different surveys, but both types included the following four elements: questions to aid in matching a user to a provider; questions about satisfaction with the service based on guidelines put forth by the reference and user services association (rusa); open-ended questions about the reference transaction; and questions regarding demographics, prior reference service usage, and knowledge of instant messaging. there were 55 valid reference transactions, and from those, the authors matched 26 pairs of user and provider surveys. the authors analyzed paired surveys to (a) compare the user’s perception of the reference transaction with the provider’s guess about the user’s perception and (b) compare the provider’s self-perception of the reference transaction with the provider’s guess about the user’s perception. the authors introduced the pair perception comparison method for the analysis using two-tailed paired t-tests and wilcoxon signed-rank tests. main results – analysis of background information showed that users were younger on average than providers and used instant messaging more frequently. even so, most users and providers felt comfortable with instant messaging. when providers were asked to guess how satisfied overall they thought the user was with the reference transaction, they reported on average that the user was less satisfied than the user actually was. these results were statistically significant. the authors found no significant difference between the providers’ overall satisfaction with the service they provided and their guesses about the users’ overall satisfaction. the authors also analyzed the matched pairs on 14 specific aspects of satisfaction gathered from surveys. when comparing the users’ satisfaction with the service they received and the providers’ guesses about the users’ satisfaction, the providers underestimated the users’ satisfaction on average for all 14 dimensions. the authors found statistically significant differences with regard to 7 of the 14 dimensions: tempo, ease of use, friendliness, understanding, accuracy, follow up, and spelling. when comparing the providers’ satisfaction with a given reference transaction and their guesses about the users’ satisfaction with the same reference transaction, the authors found significant differences for 3 of the 14 dimensions: interest, resource type, and accuracy. conclusion – this study has shown for instant messaging reference what other studies have shown for face-to-face reference—that provider pessimism exists. whatever the environment, providers of reference tend to judge themselves more harshly than the people they are helping judge them. based on a review of the literature, the authors further note that both expert and novice reference providers experience such pessimism. the authors are hopeful that providers will view these results as evidence of their own competence during instant messaging reference transactions. the results of this study provide valuable information for training instant messaging providers. for example, the fact that providers thought users were less satisfied with the tempo of the reference transaction than they really were suggested to the authors that instant messaging providers need not be so concerned about giving a quick answer. an accurate answer is more desirable. in the same vein, providers thought that users were more concerned with spelling than they really were. both of these cases, and others gleaned from the results, provide insight into what aspects of the reference transaction providers should spend their time and effort on. finally, the authors introduced the pair perception comparison method to compare feedback from matched pairs on individual evidence based library and information practice 2011, 6.2 38 reference transactions, a methodology not used in any earlier studies. they deemed this method to be an effective way to uncover biases and false assumptions. commentary the results of this study will be of interest to librarians who provide instant messaging reference service, or train others to do so, for several reasons. first, they indicate what users value in a virtual reference transaction. second, the study results may serve to bolster providers’ confidence. third, the authors provide an insightful discussion of the pair perception comparison method. much of the value of this study lies in its unique approach. while other studies have compared user and provider feedback for reference transactions, most of them have looked at face-to-face reference, not instant messaging, and have not matched user and provider responses for a given transaction. also novel was the assessment of provider satisfaction on two levels: satisfaction with their own performance as well as their guesses about user satisfaction. the study also has limitations that must be addressed. first is the size of the sample; the authors only examined 26 pairs of subjects. this was large enough to test for statistical significance between subjects, but likely too small to provide reliable results in other respects. for instance, are all users of instant messaging cavalier about spelling, or just the small number of users in this sample? additional studies are needed to validate their results. another limitation comes from the authors’ hand-picking their subjects. the providers were mls students in a reference services course. could this particular class of students generally lack self-confidence? with regard to the users, is it possible that some personality characteristic of journalism students in general skewed the results? it would be interesting to see the results from randomly selected samples in future studies of this sort. the authors introduced another limitation by virtue of using inexperienced providers. they note that provider pessimism has been observed for both new and expert reference providers, but it is not clear whether inexperienced reference providers exhibit more (or less) provider pessimism than their expert colleagues. additional research to compare new and expert providers is needed. the authors acknowledge that theirs is only one, small-scale study and welcome others to use the pair perception comparison method on different populations for further research. as a whole, this study is a valuable contribution to the existing literature on virtual reference service. the results will hearten any librarian who has experienced self-doubt during an instant messaging reference transaction. evidence based library and information practice evidence based library and information practice 2009, 4.4     evidence based library and information practice         editorial responsibilities    © 2009. this is an open access article distributed under the terms of the creative commons attribution   license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and   reproduction in any medium, provided the original work is properly cited.      editor‐in‐chief: denise koufogiannakis    associate editor (articles): alison brettle    associate editor (evidence summaries): lorie kloda    associate editor (classics, using evidence in practice): scott walter, denise koufogiannakis    associate editor (features, commentaries, ebl101, news): denise koufogiannakis    production editor: katrine mallan    editorial intern: andrea baer    copyeditors: priscilla stephenson (lead copyeditor), richard hayman, lisa shen, dale storie, mary  virginia taylor, elizabeth zeeuw    indexing support: pam morgan    3 research article   assessment of the institute for research design in librarianship (irdl): impact on the research productivity and careers of academic librarians   frans albarillo reference and instruction librarian brooklyn college, city university of new york brooklyn, new york, united states of america email: falbarillo@brooklyn.cuny.edu   marie r. kennedy serials & electronic resources librarian loyola marymount university los angeles, california, united states of america email: marie.kennedy@lmu.edu   kristine r. brancolini dean of the library loyola marymount university los angeles, california, united states of america email: kristine.brancolini@lmu.edu   received: 21 jan. 2022                                                                    accepted: 31 aug. 2022      2022 albarillo, kennedy, and brancolini. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30094     abstract   objective – this article reports the findings of a survey that is the initial phase of a mixed methods study to assess the first six years of a continuing education program designed for academic and research librarians, the institute for research design in librarianship (irdl). the study is designed to assess the effectiveness of irdl in meeting short-term and long-term programmatic objectives related to the research productivity, job performance, and professional identities of the participants in the program.   methods – in this first part of a two-phase study, the authors surveyed all 124 librarians who completed the in-person summer research workshop and year-long online follow-up program. the authors then analyzed the participant cvs and created research productivity scores as part of the evaluation of research productivity. the results of the second phase of the study, using focus groups and in-depth interviews, will be reported in a subsequent article.   results – eighty-nine participants responded to the survey, for a 72% response rate. as it relates to research productivity, there is a statistically significant correlation between the participants who began their irdl projects and then continued to do research, with higher research productivity. participants chose to publish more in book chapters and share research findings in fewer conference presentations after their participation in irdl. regarding the impact irdl may have had on any job-related factors, over 70% of respondents believed that irdl contributed to them extending their personal learning networks. a significant proportion of participants also noted a change in their self-identification as a librarian-researcher, before and after participating in the program. this article is unique as one of the first to operationalize the variables that look at librarians as practitioner-researchers and their research persistence, while building on the work of the past literature on research productivity.   conclusion – two notable findings are that irdl scholars persist in their research by continuing to pursue research projects beyond their irdl project, and that the program had an impact on their self-identification as librarian-researchers. these findings are unique in the evolving literature on librarians and research productivity. in examining a variety of factors, we believe that irdl is meeting its programmatic objectives as they relate to research productivity, job performance, and professional identities of the academic and research librarians participating in the program.     introduction   most academic librarians are practitioners and researchers, but they usually enter the profession unprepared for their research role. as a result, many lack knowledge and confidence in their ability to conduct research. the institute for research design in librarianship (irdl) was created to provide focused research training, coupled with a support network, to guide novice librarian-researchers through conducting a research project of their own design. the program was supported by two grants from the institute of museum and library services from 2013 to 2019. until irdl, there was no formal u.s.-based professional development program designed to support novice librarian-researchers in becoming confident and successful researchers.   using an explanatory sequential mixed methods approach, we investigate two research questions in this study: what are the short-term (rq1) and long-term impacts (rq2) of the irdl professional development program on the research productivity, job performance, and professional identities of irdl participants? our two-phase mixed methods study includes a survey, focus groups, and in-depth interviews to assess the effectiveness of irdl over those six years in meeting the short-term and long-term objectives of the program. the short-term objectives address teaching social science research methods, increasing research confidence, and helping each irdl scholar (hereafter “scholar”) complete their research project. the long-term objectives include continuing to pursue research opportunities after the irdl program, job performance gains attributed to irdl, like promotions and raises, and associating scholars’ professional identity as a researcher. this article reports only on the survey results. due to the size of the dataset, the results from the focus groups and in-depth interviews will be reported in a forthcoming article.    we selected a mixed methods design to derive “new insights that go beyond separate quantitative and qualitative results ... to gain new knowledge that is more than just the sum of the two parts” (creswell & plano clark, 2018, p. 13). the purpose was to explore both the quantitative measures of research success and productivity and the qualitative impact of the program on the research development of irdl scholars. the complete picture could not be gleaned from a survey alone; the quantitative data gathered from the survey and responses to open-ended questions and comments informed the qualitative phase of the project. from the annual assessments and follow-up with scholars during their year-long irdl experience, the co-directors realized that each scholar’s journey is different. with this study, the research team wanted to delve more deeply into these experiences and the research success factors associated with irdl.   irdl brought together committed cohorts of about 20 librarians each year who met in-person at a summer research training workshop and then continued to communicate throughout the next year as they worked on their research projects. irdl connected librarians to a growing community of like-minded researchers and collaborators through the development of personal learning networks (kennedy et al., 2017). during their irdl year, participants mentored and encouraged one another; these relationships often persisted long after the formal experience of irdl. in the second phase of irdl from 2016-2019, we added a formal mentoring component to the program, with the mentors tasked to further support and provide advice and assistance when the inevitable obstacles arise in completing a research project.   our results suggest that research training for motivated participants affords many benefits to librarians. brancolini & kennedy's (2017) past research on self-efficacy shows that the program increases researcher confidence. in this paper, our survey results show that participants perceive that irdl also helped them achieve positive gains in research productivity, earn tenure and promotion, expand their learning communities and research collaborators, and affirm their identity as researcher-practitioners. our analysis of the survey data shows no statistically significant barriers to finishing their irdl project. regarding research productivity success and irdl project completion, we found that completing the irdl project did not correlate to higher research productivity. instead, we found that the better indicator of research productivity is that scholars persevered and continued to do research regardless of the outcome of their irdl project.   literature review                 librarians as practitioner-researchers   librarians who conduct research have been described as practitioner-researchers (watson-boone, 2000), or librarian-researchers. practitioner-researchers exist in many disciplines, including social work, medicine, nursing, and teaching. this designation comes from the work of educator peter jarvis. in the practitioner-researcher: developing theory from practice, jarvis (1999) documented the emergence of researchers in the workplace, new researchers who use established research methods to solve practical problems. jarvis (1999) regards practitioner research as an outgrowth of rapid technological change and the need for continuous learning in professional life (p. xi). jarvis (1999) states that “practitioner-researchers are able to report aspects of practice at a depth that traditional forms of research might well not capture, precisely because they are practitioners” (p. 24). in his book, jarvis identified five types of research most frequently employed by practitioner-researchers: case studies, action research, collaborative research, using documents, and small-scale surveys.   inspired by jarvis’s book, watson-boone (2000) analyzed 24 articles authored by academic librarians published in the journal of academic librarianship between 1985 and 1995. she found that this group of practitioner-researchers who are academic librarians employed similar research methods: case study, evaluation, experimental, secondary data analysis, and survey research. watson-boone elaborates upon the importance of librarians conducting research and sharing their results. like jarvis, she observes that professional practice requires continuous learning and that employing research methods to problem-solving improves practice. the authors whose articles she analyzed first problematized and studied issues related to everyday academic library practice, then shared what they found through publications, advancing knowledge of librarianship. virginia wilson (2013) draws a parallel between practitioner research in healthcare and librarianship. patient-oriented clinician researchers facilitate the development of clinically relevant research and the dissemination of evidence based treatments in clinical practice (yanos & ziedonis, 2006). a librarian-researcher plays a similar role in library and information science (lis), bridging the lis faculty research community and the practitioner research community.   institute for research design in librarianship (irdl)   irdl is designed to meet the needs of the novice academic librarian-researcher. the rationale for irdl is rooted in the observation that most academic librarians enter the profession without realizing that librarians conduct and share the results of their research (matusiak & bright, 2020). for librarians on the tenure track, scholarship is usually required for tenure and promotion, but even at research libraries where librarians are not eligible for tenure, they are usually expected to conduct research (sassen & wahl, 2014). librarians author the majority of articles in lis journals (chang, 2016), including our profession's most highly-regarded journals (galbraith et al., 2014), such as the journal of academic librarianship (luo & mckinney, 2015). in a recent study of singleand co-authorship in selected lis research journals, chang (2021) found no statistically significant differences in the citation rates of librarian-researchers compared with lis academics. however, numerous studies have revealed the barriers many librarians face in conducting research, often starting with inadequate research training during graduate school (luo, 2011; vilz & poremski, 2015). over nearly 20 years, studies have found a declining belief among academic librarians that their lis master’s programs have prepared them to conduct research, from 30% to 17% (kennedy & brancolini, 2018, p. 834; powell et al., 2002, p. 70). a recent study of research methods curricula in master’s level lis programs found that the courses focus on teaching students to understand and evaluate research literature, rather than preparing them to conduct their own studies (matusiak & bright, 2020).   irdl was developed from the results of a national survey of librarians working in an academic setting conducted in late 2010, designed to gather evidence on librarian attitudes, involvement, and capabilities for engaging in research activities (kennedy & brancolini, 2012). two of the most significant findings directly influenced the form and design of irdl. first, the researchers found that most academic librarians do not believe that their mlis programs prepared them to conduct research. the second significant finding was that most librarians lack confidence in their research abilities. this is important because the 2012 survey also found that research confidence may be a predictor of research success. the co-directors designed irdl to maximize two of the four main sources that influence self-efficacy, as described by the psychologist albert bandura (1993): mastery experiences and social persuasion. this theoretical foundation influenced the decision to focus on a hands-on workshop format to teach social science research methods (gaining mastery of a process through practice) within a supportive community environment (receiving positive verbal feedback throughout the steps in the research process) and the objective of completing a research study within a year. irdl includes research support mechanisms such as monthly communication with the cohort and co-directors through online check-in sessions, a closed facebook group that includes scholars from all cohorts, and during the last three years of the program, formal mentoring by an experienced lis researcher. the centerpiece of irdl is the summer research workshop; it lays the foundation for the rest of the year-long research support. the nine-day workshop focuses on social science research design and methods training through expert instruction, small-group hands-on activities, and one-on-one proposal consultations with the workshop faculty and other researchers. the curriculum steps through the process of conducting a social science research study, with a focus on the three most commonly used data collection methods: survey, in-depth interviews, and focus groups. see appendix a for the course learning objectives.   assessing learning and research self-efficacy in irdl   the effectiveness of the workshop curriculum is assessed through two measures: a preand post-workshop research self-efficacy scale and a rubric-based evaluation conducted by academic librarians to measure improvements in the research proposals. upon completion of the research workshop, librarians’ confidence increased in all the areas covered in the curriculum: turning a topic into a research question, designing a project to address their question, performing a literature review, gathering quantitative and qualitative data, analyzing data, reporting the results, and identifying appropriate places to share the results (e.g., publications and conferences; brancolini & kennedy, 2017). the results of these assessments resulted in revisions to the workshop for the first three years of the program.   to ensure that irdl continues to address the research concerns of the academic library community, the 2010 survey, with slight modifications, was re-administered in 2015 (kennedy & brancolini, 2018), and the data used to make updates to the program. the most significant change to irdl-2 (2016–2019) was the addition of formal research mentoring, which paired each scholar with an experienced researcher (jason et al., 2021).   research success factors and productivity   the development of irdl has been influenced by two studies by canadian researchers who have looked at research success factors among academic librarians (hoffmann et al., 2014; hoffmann et al., 2017). in the first article of its kind, hoffmann et al. (2014) conducted a comprehensive literature review of empirical studies focused on research success factors, both in librarianship and in other applied fields. they found that research success requires a number of interrelated conditions. they grouped 16 factors into three categories: “individual attributes, peers and community, and institutional structures and supports” (hoffmann et al., 2014, p. 19), with many factors occurring in more than one category. individual attributes included research confidence (self-efficacy), which had already been incorporated into irdl and the assessments of its effectiveness. within the category of peers and community, mentoring was the only factor identified as showing a positive effect on research productivity. however, none of the 12 papers that measured the effect of mentoring was focused on librarians. this finding provided the impetus for the co-directors to create a formal mentoring program for irdl-2 (jason et al., 2021). hoffmann et al. (2017) conducted a follow-up study to measure the relationship between the research success categories and individual factors they found in the literature and the research productivity of canadian academic librarians. they surveyed 1,653 librarians and received 453 complete responses. respondents were asked to provide data on their research output over the previous five years in eight categories. the research productivity scores were then correlated with answers to “yes/no” statements that were mapped to the 11 success factors within the three categories. the authors found that all categories have an impact on research productivity, including the scores across all types of research output and for peer-reviewed articles, which were analyzed separately. no single category was dominant, suggesting that research productivity is affected by a constellation of factors.   the completion of six years of irdl provided this research team with a unique opportunity to study the 124 academic and research librarians who participated in irdl (2013–2016) and irdl-2 (2016–2019), examining both the short-term effects of the program on all participants and the longer-term effects on the first cohorts of scholars. hoffmann et al. (2017) noted that the follow-up to their quantitative study would include qualitative data. their work and earlier studies conducted by the irdl co-directors (brancolini & kennedy, 2017; kennedy & brancolini, 2012; kennedy & brancolini, 2018; kennedy et al., 2017; kennedy et al., 2020) influenced the decision to conduct a mixed methods study, starting with a survey and following up with focus groups and in-depth interviews.   methods   in fall 2019 and summer 2020, we conducted a mixed methods assessment of the short-term and long-term impacts of the irdl continuing education program. we used survey, focus group, and semi-structured interview instruments to gather data to measure the impact of irdl on three primary areas of concern: research productivity, job performance, and identity as a researcher.  we employed an explanatory sequential mixed methods approach; survey data was collected first, and the interview data was collected after the survey. “the overall intent of this design is to have the qualitative data help explain in more detail the initial quantitative results; thus, it is important to tie together or to connect the quantitative results to the qualitative data collection” (creswell & creswell, 2018, p. 13). in this section, we describe the process of developing, administering, and analyzing the data collected from the survey, the first part of the assessment. the findings from the focus groups and in-depth interviews from the second part of the study will be reported in a separate article.   study population   we included 124 scholars in the study. of those, 10 were employed at four-year colleges, 9 at two-year community colleges, 3 at research institutions (non-academic), and 102 at universities. the scholars are employed in a variety of functional areas within their libraries and archives and represent a range of levels of academic rank. about thirty-five percent of the scholar population identifies as hispanic and/or non-white.   recruitment and survey dissemination   after receiving approval of the survey protocol from the institutional review boards from our two institutions, two scholars pilot tested it. after incorporating their feedback, we sent an email to the population of scholars from cohorts 1-5 in september 2019, informing them about the project and asking them to look for a forthcoming email with the invitation from albarillo to participate in the assessment. albarillo contacted the group with the survey invitation in october 2019, with one follow-up email sent. albarillo contacted scholars from cohort 6 in july 2020, after they had completed their full irdl program. we offered a $50 usd gift card to each scholar who completed the survey and supplied their cv.   survey design and measures   we designed the survey around three areas of interest related to the short-term and long-term objectives of irdl on the professional lives of the scholars:   research productivity, as it relates to both progress on completion of their irdl research project and additional research completed and shared in some format; job performance, including how they believe participating in irdl may have impacted their professional trajectories; and identity as a researcher, including how irdl may have impacted how they think of themselves as researchers.   sections of the survey   research productivity   we posed several questions to capture data about the progress of the scholar’s irdl research project and projects completed since, such as: how much of your irdl project were you able to complete? choose the step that best describes the current state of your irdl project; how did you disseminate or attempt to disseminate your irdl project findings? have you conducted other research projects since irdl? and how did you disseminate or attempt to disseminate your post-irdl project(s) findings? we included a question for the scholar to identify any barriers encountered during their irdl research project, as well as supports promised and available from their library and institution in completing their project. we collected and analyzed scholars’ cvs to create research productivity scores to use as continuous variables.   job performance   in this section of the survey, we included questions about job performance indicators, such as status (whether employed in a tenure-track, continuing appointment, or promotion-eligible position), academic rank, and research support options provided by the library and institution. we included a question that asked if the scholar believed that irdl contributed to any of a list of job-related factors.   professional identity   in this section, we asked five questions, including two to identify if the scholar was pursuing or had achieved an additional degree. we had heard from scholars over the years that their participation in irdl had prompted them to pursue continuing education and wanted to better understand how many had pursued it. we included three questions to explore the concept of what it means to be a “librarian-researcher.” in the last question of the survey, we requested that respondents tell us anything about the impact (both short-term and long-term) of their experience as scholar on their research productivity, job performance, or identity as a researcher that we had not asked, but that they thought was important for us to know. at the close of the survey, the scholar could choose to enter their email address to be contacted for participation in a focus group or interview.   the survey and recruitment email are included as appendices b and c.   results   the survey did not collect cohort and demographic data since the lead author already had access to those data, collected previously as part of participation in the irdl program. all three authors made an earlier commitment to our study population that any analysis presented would be at the aggregate level; presenting results by cohort and demographic levels could compromise the anonymity of participants. this section presents the results of the survey in the same order as they appear in the survey, except for the analysis of the cvs (survey question 13), which will appear last. we have included inferential analysis in the results to complement the descriptive statistics where the survey design contained variables at the appropriate level of measurement and the survey data met the assumptions of the chosen statistical test with a p < .05.   we present here the results of the survey, followed by a brief analysis of the cvs and the measures of research productivity. the statistical analysis was performed using ibm spss (statistical package for the social sciences) statistical software, version 27. we calculated descriptive statistics (counts and percentages for categorical/nominal responses, means and standard deviations for continuous measures) for survey items.   a total of 89 scholars (of 124 recruited) completed the survey and provided their cvs, for a 72% survey response rate, with an equal distribution of respondents across the six cohorts. to check for possible cohort bias in the response rate, we conducted a cochran-armitage test of trend (laerd statistics, 2016) to see if response rates to the survey were uneven across the six cohort populations. the responses by cohort are cohort 1 (n= 15), cohort 2 (n = 9), cohort 3 (n = 15), cohort 4 (n= 13), cohort 5 (n = 18), and cohort 6 (n = 19). the test did not show a statistically significant linear trend between survey responses in the proportion of respondents from each cohort, p = .05.   research productivity   as part of the curriculum, the scholars hone the design of their research project at the summer research workshop and execute it in the following year. in answer to question 1 of the survey, “what research method(s) did you use in your irdl project,” 42 scholars reported that they chose a qualitative research design (47.2% of the projects), 22 chose a quantitative design (24.7%), 20 used a mixed methods design (22.5%), and 5 did not report their research design. in responding to question 2 of the survey, “how much of your irdl research project were you able to complete?” we see that 47.2% reported or published the results (as shown in table 1); for our purposes we are classifying these 42 projects as complete at the time the survey was administered.   table 1 stages of irdl project completion question 2: how much of your irdl project were you able to complete? choose the step that best describes the current state of your irdl project. n percent i completed the revised written proposal 6 6.7 i received irb approval 3 3.4 i started data collection 5 5.6 i completed data collection 4 4.5 i started organizing the data i collected for analysis 11 12.4 i completed a preliminary analysis of the data i collected 11 12.4 i completed a full analysis of the data i collected 7 7.9 i reported the results of the data i collected 16 18.0 i published the results of the data i collected 26 29.2 total 89 100   knowing that some scholars were likely continuing work on their irdl projects, we asked a follow-up question. of the projects that had not yet reached the stage of completion at the time the survey was administered, we wanted to know how likely they were to be completed. of the 18 scholars who reached the stage of completing a preliminary or full analysis of the data they collected, 14 noted that they were extremely likely to disseminate the results of their irdl project through presentation or publication.   question 7 prompted those scholars who reached the step of data analysis or completion to report how they shared or attempted to share their irdl project findings. submitting a proposal to present results as a presentation, paper, or panel at a regional, national, or an international conference was the most frequently mentioned, with 36 attempts noted. presenting results at their own library or institution was the second most frequently noted mechanism, with 35 presentations.   with question 8 we were interested in learning what barriers the scholars may have encountered while conducting their irdl research project. in table 2, we report the number of responses received for each barrier listed and how the scholars rated them in severity, using a four-point likert scale: not a barrier (1); somewhat of a barrier (2); moderate barrier (3); extreme barrier (4). overall, the scholars responded to the list of possible barriers as “not a barrier” or “somewhat of a barrier.” it is notable that two barriers related to job performance, “given new job responsibilities in the same position” and “changed jobs,” received the largest number of reports of being an extreme barrier in completing their project.   we were interested in determining if any of the self-reported irdl project barriers listed in table 2 had a linear, statistically significant correlation with the scholars' ability to complete their irdl project. using data from the second survey question, we created a dependent dichotomous variable, the irdl project completed variable, hereafter irdlprojcomp. the variable was constructed by collapsing the last two survey response options, "i reported the results of the data i collected" and "i published the results of the data i collected," into a new value called "completed"; the other response options were collapsed into a value called “not completed.” the values for this dichotomous variable show 42 scholars (47.2%) did complete their irdl project, and 47 scholars (52.8%) did not complete their project. we conducted a binomial logistic regression, as outlined in laerd statistics (2017). with irdlprojcomp as the dependent variable and table 2 barriers as dummy variables, we found no statistically significant relationship between any of the table 2 barriers and irdl project completion.     table 2 barriers to completing the irdl project, measured on a 4-point likert scale: not a barrier (1); somewhat of a barrier (2); moderate barrier (3); extreme barrier (4) question 8: of the barriers listed below, please indicate how intense that barrier was in completing your irdl project   n not a barrier somewhat of a barrier moderate barrier extreme barrier lack of experience using specific research methods 89 26 45 17 1 given new job responsibilities in the same position 89 51 12 13 13 changed jobs 89 64 2 7 16 lack of access to needed research-based literature 89 79 10 receiving institutional review board (irb) approval 88 70 11 7 access to the study population 89 54 20 9 6 access to online survey software 88 82 5 1 knowing how to use online survey software 88 75 8 4 1 organizing and preparing quantitative data for analysis 84 49 19 14 2 analyzing quantitative data 84 45 18 15 6 organizing and preparing qualitative data for analysis 87 41 33 11 2 analyzing qualitative data 87 37 33 12 5 low response rates from study participants 86 56 14 6 10 i collected unusable data 87 65 15 4 3 reporting results in a written format 88 53 27 4 4 reporting results in a webinar format 86 80 4 1 1 reporting results as a poster presentation, presentation, or panel in a meeting, conference, or workshop format 86 77 7 1 1   we were also interested to learn from the scholars which research supports had been made available to them, as promised in a letter of support from the dean or director that was required during the application process. of the five options noted in question 9, moral support from a supervisor was the most frequently received support, with 76% of the respondents reporting that it had been made available to them. sixty-four percent received at least half a day per week release from work duties to conduct their irdl projects. using the irdlprojcomp variable again, and now testing with supports as dummy variables, we found no statistically significant relationship between any of the supports and irdl project completion.   almost three-fourths of the respondents (n = 65, 73%) reported in question 10 that they continued to do research after their irdl project, either currently conducting a project or having conducted one or more other projects. twenty-three reported having neither attempted nor completed another research project since their irdl project. of those 47 who reported sharing or attempting to share the results of new research projects since irdl, the most often reported mechanism was submitting a proposal to present results as a presentation, paper, or panel at a regional, national, or an international conference (with scholars having reported 30 proposals submitted).   given the significant number of scholars continuing to do research after their irdl project, we wanted to explore whether continuing to do research may be a better indicator of program effectiveness than completing the irdl project. as library practitioners, we understand that research makes up only a small percentage of our professional responsibilities, with those research projects overlapping to some extent with job responsibilities. we also maintain that research is subject to the ebb and flow of the work year priorities familiar to evidence based library and information practice readers. therefore, using data from question 10, we created a dichotomous variable on scholars who continued to do research, hereafter called rescont. it collapses four response options that indicate the scholar continued to do research into one value, “continuing,” and the other response option into another value, “not continuing.” a crosstabulation revealed that there were 34 scholars (38.2%) who didn’t finish their irdl project but continued to do research.   we then created another variable, called avgsum. it is a continuous research productivity score that represents the sum total number of weighted research outputs from 2010–2020, as indicated on the scholar cvs, divided by the number of years they have been actively disseminating research. using the avgsum as the dependent variable and rescont and irdlprojcomp as independent variables, we conducted a multiple linear regression, as outlined in cohen et al. (2003, pp. 64–99), to test if continuing to conduct research or completing the irdl project were correlated with higher research productivity.   rescont and irdlprojcomp statistically predicted research avgsum with f(2,85) = 5.031, p <.001. the r2 for the model was 10.6 %, with an adjusted r2 of 8.5%, a small effect size. examining the coefficients of our dichotomous dependent variables shows that the rescont variable is statistically significant (n = 65, p = .002) with a b slope coefficient = 7.430, while the irdlprojcomp was not statistically significant (n = 42, p = .747). we interpret the results of this test as evidence that looking at whether or not a scholar continues to do research after their irdl project, as shown by rescont scholars (n = 65, 73%), is a better measure of irdl's long-term impact on research productivity than completion of the irdl project, since the variable is positively correlated to higher research productivity scores. on the other hand, irdlprojcomp scores show no positive and statistically significant correlation with research productivity.   job performance   in this section of the survey, we explored job performance indicators and the current research environments of the scholars. in responding to question 14 about which research support options are provided at the scholar’s institution or library, 48 respondents (53%) noted that travel funds with full reimbursement are available to them. workshops or other forms of continuing education were also prevalent, with 39 scholars noting this type of support. the least available support offered by their institutions or library was formal research mentors, with only seven scholars noting it as an option.   to understand promotion and tenure pressures, question 15 asked the job status of each scholar during their irdl year. about half of the scholars (n = 49) were in positions on the tenure track, with the possibility of promotion. using the variables created previously (irdlprojcomp, rescont, and avgsum), we used a binomial logistic regression (laerd, 2017) to examine whether completing the irdl project, continuing to do research after the irdl project, or the average research output was correlated with scholars on the tenure track, and found that the resulting model was not statistically significant, χ2(3) = 5.04, p < .17. being on the tenure track did not significantly correlate with higher research productivity, irdl project completion, or the scholar continuing to pursue research projects. we asked if the scholars attained a continuing appointment, tenure, or promotion at the institution where they were employed during their irdl year, and 13 noted that they attained a continuing appointment, but not promotion. five noted that they achieved tenure during their irdl year. the current academic rank attained by the scholars is dispersed among the options presented in the survey, with assistant professor (n = 22) being the most reported rank.   in order to consider the impact irdl may have had on any job-related factors as perceived by the scholars, we asked them to choose applicable factors from a list (as shown in table 3). the most frequently selected factor in question 18 was extending their personal learning networks, with 70.8% of the scholars choosing it. fifty-five scholars (61.8%) chose that they believe irdl contributed to them advising other librarians on their research projects. forty-three (48.3%) said that due to irdl they gained research collaborators.   table 3 factors contributing to job performance question 18: do you believe irdl contributed to any of the following job-related factors? choose all that apply. n percent i earned a promotion in rank  17 19.10 i received a promotion to a higher-level position at my institution  7 7.90 i achieved tenure  9 10.10 i received a certificate of continuous employment or a similar guarantee of job security  6 6.70 i received a new job title at my institution  7 7.90 i received a one-time monetary salary award (e.g., a merit increase)  5 5.60 i received a permanent raise in my base salary   11 12.40 i acquired new job responsibilities related to my skills as a social science researcher at my institution (e.g., assessment)  14 15.70 my research skills helped me secure a job at a new institution   12 13.50 i started a local writing or research group  4 4.50 i have advised other librarians about their research projects  55 61.80 i made presentation(s) based on the skills i gained during the irdl workshop   32 36.00 i have given workshops based on the skills i gained during the irdl workshop  16 18.00 i extended my personal learning network  63 70.80 i gained research collaborators  43 48.30   professional identity   the third section of the survey covered professional identity. the first set of questions asked about additional formal education and the second set asked about identity as a researcher.   questions 19 and 20 asked about pursuing additional formal education. we found five were working on a phd/edd/jd and five were working on a non-thesis-based master’s degree at the time they responded to the survey. eight noted that they had completed additional degrees since their irdl year.   we were interested to learn how participating in irdl may have affected their perception of librarians as researchers and of themselves as librarian-researchers. question 21 asked them to describe what it means to be a "librarian-researcher." although we did not formally code the responses to this question, the 86 responses reflect many aspects of research conducted by practitioners. the open-text question elicited a wide range of responses, with two major themes emerging: librarian-researchers 1) conduct research to improve the quality of their work, including the effectiveness of library programs and services, such as support for the academic researchers at their institutions; and 2) contribute to advancing knowledge in library and information science. although most responses were positive about the relationship between librarianship and research, some noted the difficulty of juggling job responsibilities and the many activities associated with conducting and sharing the results of research. these are four representative examples:   a librarian-researcher wears two symbiotic hats -one is the librarian who implements services and practices that contribute towards access to information, and the other conducts research that informs the most effective ways to do so. each should facilitate the other.   being a librarian-researcher means that i have first-hand experience conducting my own research and employing various research methods. this gives me the opportunity to better relate to the community of researchers i serve, while allowing me to advance the body of knowledge in my profession.   i identify as a "librarian-researcher" and for me that means that my work as a practitioner is as valuable as my work as a researcher, particularly because my practice is influenced by my research and my research is inspired by my practice.   a librarian-researcher has the dual task of conducting research and performing their daily work as a librarian. librarian researchers take the initiative to start new research projects, advise on others' research projects (e.g., other librarians doing library-related research, users that need methodological help), and see the potential for collaborations across their institutions as well as new ideas for research to be involved in.   twenty-eight of the respondents (31.5%) noted that they identified as a librarian-researcher before their irdl year (question 22) and seventy (78.7%) reported that they currently identify as a librarian-researcher (question 23). we used mcnemar's test procedures as outlined in laerd (2015a) and found that there was a statistically significant difference (p = .001) in the proportion of scholars who self-identified as librarian-researchers pre-irdl (.19) and post-irdl (.29).   the last question of the survey (question 24) was an open-text prompt to tell us anything “about the impact (both short-term and long-term) of your experience as a scholar on your research productivity, job performance, or identity as a researcher that we have not asked but you think is important for us to know.” fifty-five respondents offered their comments. responses to this question included multiple comments about an increase of confidence leading participants to stretch their professional goals, and many comments about the satisfaction of finding connections to other like-minded librarians in building a community. the most common word in these comments was “confidence” or “confident,” noted by 20 respondents. this representative example addresses multiple objectives of irdl; other responses are included in the discussion section.   irdl has instilled in me a greater sense of confidence. i feel more comfortable asking my colleagues for help and reaching out to others to seek collaborative research opportunities. i have participated in conversations with non-library faculty in which i have felt part of the discussion (regarding data analysis) because i have conducted original research (via my irdl project). i now feel more creatively inspired and can generate ideas for potential research projects more readily since attending irdl, and i feel that i have gained enough experience from irdl to attempt such projects. my irdl experience – as well as projects that develop post-irdl – can serve as talking points with non-library faculty, with librarians, and with future potential employers. irdl has opened the door to a professional pathway that did not feel attainable to me prior to my participation. i am grateful for the opportunity.   the scholars could opt in to participating further, in a focus group or one-on-one interview, by inserting their email address to acknowledge interest.   analysis of cvs and research productivity scores   we used the cv of each scholar as evidence of research productivity. albarillo reviewed each cv to count the type of output produced. since our focus is to examine research output, we did not include in our count the following written outputs, which may be scholarly in nature but not necessarily a result of original research: blog posts, conference reports, newsletters, book reviews, blurbs, editorials, zines, and articles in trade journals (examples of which include college & research libraries news, library journal, american libraries, and magazines for libraries). we also did not include in our count oral presentations that are work-related and part of job responsibilities (such as database demonstrations and career presentations to lis classes) or presentations that are very brief in nature (lightning talks, for example). in table 4, we show a summary of the scholars’ research output over the past ten years (2010–2020), the type of output mechanism used, and the number of each. overall, the most popular mechanism for sharing the results of their irdl projects is presentations (61.3%), then peer review articles (18.3%), followed by book chapters (6.6%).   we examined the research output of the scholars from before their participation in irdl and then compared their output to after their irdl year. we found that there appears to be a mean increase for some outputs: books, edited volumes, peer-reviewed papers, and book chapters. we also found a mean decrease for other outputs, as shown in table 5. to help us understand this shift in the choice of output mechanisms, we consulted the weighted research output scheme developed by hoffmann et al. (2017, p. 107). in that scheme, the four most heavily weighted outputs are book chapters, edited books, peer-reviewed articles, and authored books. this article builds on the use of hoffmann’s research output scheme, which was also applied in kennedy et al. (2020), as a way to quantify research productivity of accomplished librarian-researchers. from that scheme we see that following their irdl participation, scholars began disseminating their research through more sophisticated and prized outputs.   table 4 scholars’ research output over the past ten years (2010–2020, n = 89) output type min max mean median sd total number reported % of output reported poster 0 14 0.89 0.00 1.91 80 5.68 presentation 0 141 9.79 6.00 15.74 864 61.32 conference proceeding 0 15 0.57 0.00 1.82 51 3.62 non-peer-reviewed article 0 5 0.51 0.00 1.04 46 3.26 book chapter 0 6 1.05 1.00 1.29 94 6.67 edited book 0 5 0.12 0.00 0.59 11 0.78 peer-reviewed article 0 19 2.89 2.00 3.40 258 18.31 authored book 0 2 0.05 0.00 0.27 5 0.35 totals           1,409 100   table 5 research productivity formats: measures of central tendency, pre and post irdl (n = 89) output types, before and after irdl total number reported min max mean median sd variance range poster (pre) 49 0 6 0.55 0 1.138 1.296 6 poster (post) 31 0 14 0.35 0 1.538 2.366 14 presentation (pre) 458 0 75 5.15 2 8.725 76.126 75 presentation (post) 406 0 66 4.56 2 8.316 69.158 66 conference proceeding (pre) 31 0 5 0.35 0 0.943 0.889 5 conference proceeding (post) 20 0 10 0.22 0 1.136 1.29 10 non-peer-reviewed article (pre) 27 0 3 0.3 0 0.775 0.6 3 non-peer-reviewed article (post) 19 0 3 0.21 0 0.593 0.352 3 book chapter (pre) 29 0 5 0.33 0 0.75 0.563 5 book chapter (post) 65 0 4 0.73 0 1.053 1.108 4 edited book (pre)  4 0 2 0.04 0 0.257 0.066 2 edited book (post) 7 0 3 0.08 0 0.405 0.164 3 peer-reviewed article (pre) 103 0 9 1.16 0 1.876 3.52 9 peer-reviewed article (post) 155 0 13 1.74 1 2.259 5.103 13 authored book (pre) 2 0 1 0.02 0 0.149 0.022 1 authored book (post) 3 0 1 0.03 0 0.181 0.033 1   table 5 shows an increase in scholarly output by irdl participants in the form of book chapters and peer-reviewed articles. we wanted to run an inferential statistical test to see if this increase in publications after irdl was statistically significant. initially our original plan was to run a paired-sample t-test, comparing the research productivity scores of irdl participants before and after irdl.   as part of our calculations, we realized that the research productivity scores alone as represented in table 5 did not account for the variation in years that scholars have been active researchers. for example, some scholars have been very productive researchers from right after library school. other cvs showed that there were many scholars who have worked and steadily published and presented in the library profession for a longer period, and of course there were many new scholars who were just beginning their research and publication journey. we developed a formula to account for the variation in scholar publishing years before participation in irdl:   let a = represent the number of publications multiplied by the publication weight assigned in hoffmann et al. (2017, p. 107). let b = represent the year the scholar participated in irdl. let c = represent the date of the first scholarly output indicated in the cv as determined by the lead author. scholarly output in the form of posters, presentations, and publications before 2010 is not counted. nb, the "+1" accounts for the year the scholar participated in irdl   we created the following formula to calculate the research productivity score after irdl, which we call post avg:   let d = represent the year and the last date scholarly output indicated in the cv as determined by the lead author. publications after 2020 are not counted. let e = represent the year after the scholar participated in irdl. nb, if the scholar attended the summer research workshop in 2018, then only research output in the beginning of 2019 to the end of 2020 was counted.   by dividing scholars' research output scores, as represented by the variable “a”, by the number of active research years, represented by scholarly output in their cvs, we were able to test the hypothesis: is there a statistically significant difference in research productivity before and after irdl?   unfortunately, our pre avg and post avg scores were not normally distributed, an assumption our data needed to meet to be able to conduct a paired samples t-test according to laerd (2015b); the shapiro-wilk statistic (0.77, p = 0) and (0.82, p = 0) for the pre avg and post avg respectively. we decided to employ the wilcoxon signed-rank test, a nonparametric test equivalent to the paired samples t-test. we examined the procedure and criteria for running the test (pett, 2016, pp. 113–114) and found that the pre avg and post avg research productivity scores met all three criteria.   in addition to the total productivity scores, we ran the test to look for paired differences between the main research output formats like peer-reviewed articles, book chapters, and professional conferences. table 6 shows the results of the wilcoxon signed-rank test. the first result shows that there were no statistically significant differences in research productivity scores for scholars before and after irdl. the second result showed that there were no statistically significant differences before and after irdl for the format peer-reviewed papers. there were small, statistically significant increases in the median scores for book chapters, and a small and statistically significant decrease in conference presentations at p < .05. we believe that these scores, while statistically significant, represent too small a value to be meaningful.   table 6 results of the wilcoxon signed-rank test wilcoxon signed-rank test md null hypothesis test sig decision 0.602 the median of differences between the pre avg and the post avg total research productivity scores equals 0. related samples wilcoxon signed-rank test .167 retain the null 0.711 the median of differences between the pre avg peer review papers and the post avg peer review papers equals 0. related samples wilcoxon signed-rank test 0.072 retain the null 0.4621 the median of differences between the pre avg book chapters and the post avg book chapter equals 0. related samples wilcoxon signed-rank test 0.004 reject the null -0.4713 the median of differences between the pre avg conferences and the post avg conferences equals 0. related samples wilcoxon signed-rank test 0.004 reject the null   discussion   the present study focuses on the short-term and long-term impacts of irdl on the research productivity, job performance, and professional identity of the librarians who participated in the program. the first part of the study was a survey, intended to measure the impact of various personal and environmental factors on the ability of the scholar to complete their irdl research study and to go on to new research endeavors.   short-term and long-term research productivity   the first objective for each scholar is to complete their proposed research project, defined as reaching the stage of disseminating research results, either through presentation or publication. the results of the survey indicate that 42 respondents (47%) were successful. of the 18 who had completed preliminary or final data analysis, 14 reported that they were extremely likely to complete their project. the longer-term goal of irdl is to set participants on a path of research success. seventy-three percent of respondents reported that they had completed another research project, either instead of or in addition to their irdl project, indicating a disposition toward conducting and disseminating research in the future. we believe that our survey data demonstrates a new concept we call “research persistence.” this variable, based on the rescont (research continuer) data, is an important finding related to the outcome of attending irdl. we looked at many factors, and we believe that this data shows that despite past failures or successes — for example, the 34 scholars (38.2%) who didn’t finish their irdl project but continued to do research — many irdl scholars have chosen to persist and seek community and opportunities to improve their ability to conduct research. more research needs to be done to better understand this concept of “research persistence.”   the analysis of cvs to determine the number and types of research output for each respondent revealed that the effects on research productivity have persisted following the single irdl year. although the average productivity of scholars increased slightly post-irdl, the important finding is that dissemination patterns changed. the number of books, edited volumes, peer-reviewed articles, and book chapters increased, while the number of posters, presentations, conference proceedings, and articles in non-peer-reviewed publications decreased. this is unsurprising but also encouraging, as many scholars expressed a desire to disseminate their research in peer-reviewed journals, books, and other high-impact research formats. we did not find any meaningful, statistically significant increases or decreases in the inferential and non-parametric tests that we conducted related to research productivity. this was very surprising because the descriptive data in table 5 showed positive increases for particular formats. in our efforts to understand tenure pressure, we found it unusual that there was no significant correlation between being on the tenure track, higher research productivity, irdl project completion, or the scholar continuing to pursue research projects. we believe that this is due to the variability of tenure requirements across institutions. the cross-institutional variability of tenure requirements would make an interesting future study and contribute to our understanding of librarians and tenure status.   research barriers   the second objective of irdl is to remove or reduce barriers to completing their irdl research project. the results support the effectiveness of irdl in this regard. asked about barriers to completing their study that may have been encountered during their irdl year, the most frequent response was “not a barrier.” the most prevalent research barrier was lack of experience using specific research methods, with 45 reporting it as “somewhat of a barrier,” 17 as a “moderate barrier,” and 1 as an “extreme barrier” (nb: 26 report it as “not a problem”). all participants are novice researchers, so the irdl curriculum includes an introduction to three of the most prevalent quantitative and qualitative research methods: surveys, in-depth interviews, and focus groups (see appendix a for course learning objectives). however, irdl encourages participants to try new and more innovative research methods. for example, two scholars used vignettes in their studies (benedetti et al., 2018), which participants were not taught to use in the workshop; they discovered this method during their literature reviews and wanted to try it. despite 63 scholars reporting that lack of familiarity with specific research methods was a barrier, the barrier did not stop them from conducting their studies. sixty-five conducted their studies, 16 presented their results, and 26 published their results. twenty-nine scholars, however, responded that two work-related factors were extreme barriers: being given new responsibilities on the job and changing jobs. as fully summarized in the results section, we did not find any meaningful, statistically significant correlations between the barriers and the completion of the irdl project. barriers and strategies to overcome them were explored more fully in the qualitative phase of the research project.   research supports   the third objective of irdl is to ensure that scholars are provided with appropriate research support by their library or institution while conducting their irdl project. in addition to removing barriers, irdl provides a number of supports designed to foster research success. the most important support is time to focus on research, without work or family pressures, during the summer research workshop. however, it is also important to ensure that scholars have ample research support during the coming project year, including a half-day per week to devote to research activity. as part of the application process, librarians supplied a letter from their library dean or director outlining the availability of a variety of research supports. the survey found that these supports were provided as promised in the vast majority of cases. sixty-four percent received at least a half-day per week for research. respondents also reported on the availability of additional research supports. the least prevalent research support was formal mentoring, available to only seven of the respondents. however, this finding affirms the addition of formal research mentoring during the last three years of irdl. as similarly reported above, we did not find that any research supports were statistically related to completing the irdl project.   professional identity   following their irdl experience, a number of librarians have pursued additional education related to research proficiency, including five who at the time of the survey were currently working on a phd and three others who were applying to a phd program or considering doing so. one commented that “irdl was critical” in giving them the confidence to pursue a phd. another commented, “had i not attended irdl, i would not have pursed a phd.”   eighty-six participants responded to question 21, which asked for a definition of what it means to be a “librarian-researcher.” these responses reflect deep thinking about the relationship between their roles as academic and research librarians and their scholarship. before participating in irdl, 31.5% identified as a librarian-researcher, but 78.7% stated that they currently identify as a librarian-researcher, an increase of more than 47%. as noted in the results section, this change in professional identity is statistically significant and is especially meaningful since other research suggests that identifying as a researcher may be related to productivity (brew et al., 2016).    irdl provides an opportunity for librarians to form relationships and become part of a research community. we believe that the cohort model and the cross-cohort collaborations contribute greatly to the sense of identity and community. while we found no statistically significant correlations related to research productivity, these descriptive statistics suggest that building a research community is a complex phenomenon that merits more research. participation in irdl had positive social consequences for many librarians. about 71% of the scholars noted that they extended their personal learning networks (see table 3 for a summary of question 18). fifty-five scholars (61.8%) reported that they believed irdl contributed to them advising other librarians about their research projects. forty-three (48.3%) said that they gained research collaborators thanks to irdl. the development of a research community, through cohorts, is an important aspect of the irdl experience. there is evidence that some scholars are also part of a cross-cohort research community. the survey did not explore this phenomenon, but we believe that the qualitative interviews may give us a clearer picture of irdl-based research relationships. in response to question 24, a number of scholars commented about research community, as illustrated in the following four quotes:   i have found other like-minded librarians through irdl. a couple of them have become close research partners.    i think i would have found a way to educate myself about research methods, but finding like-minded people was one of the most valuable aspects of attending irdl. i found that i wasn’t alone and that there were other research nerds like me, which gave me the confidence to continue my work.   i’ve established networks of researchers who are amazing collaborators and we will be exploring more projects together.   the connections to other librarians who do research has [sic] been invaluable – even more so than the mentor relationship (though that was valuable too). this has been particularly important for me since i have no such connections at my own institution or even in my region.   limitations   we acknowledge some limitations to this research, the main one being the special population studied. the population of scholars is a selected group of librarian-researchers. they were chosen for the program based upon a number of factors, including their desire to learn and their enthusiasm for research. therefore, scholars do not represent the range of attitudes toward research among academic librarians. exner (2019) found in her qualitative study of novice researchers that even in libraries with supportive research environments, some librarian researchers are fearful and uncertain about conducting research, characterized as “extreme dislike” (pp. 64–65). the scholars do, however, represent the range of work environments and types of jobs in the higher education arena. cohorts are built from librarians and archivists working in traditional academic library settings as well as special libraries; their job functions demonstrate the full range of library and archive activities.   we also acknowledge a time bias built into the research design, using a cross-sectional design (data gathered at one point in time) of the six cohorts. this design naturally favors the responses that the scholars from the earlier cohorts are able to give, for example, to the question about their progress in their irdl research project. participants from the earlier cohorts should be further along in their projects than the later cohorts. we waited to survey the sixth cohort until they had completed their full irdl year, and that delay pushed their survey to during covid lockdown, which may have impacted some of the responses from that group.   the last limitation to note is based on our review of the cvs submitted by the scholars. the cvs were not standardized in any way prior to submission; they varied in formatting and categories. so that we were internally consistent in categorizing the types of research output, we often turned to locating the original article, presentation, or program and verifying the nature of the research output in ulrich’s periodicals directory (proquest) and the serials directory (ebsco).   our survey produced significant findings, many of which are aligned with the findings of hoffmann et. al (2017), that research productivity is affected by overlapping and reinforcing factors. we have attempted to operationalize these factors and acknowledge their limits. we also believe that limits are good occasions to suggest directions, where appropriate, for future research into this complex topic of librarians as research-practitioners. a survey alone cannot describe the full impact a program like irdl may have on its participants. the findings here are presented in aggregate, which likely masked differences in the individual experiences of scholars. in the next phase of the study, we examine the impact on the personal level, as it relates to research productivity, job performance, and researcher identity. we look forward to presenting the results of the following phase of this mixed methods study, based upon the individual in-depth interviews and focus groups with the participants.   conclusion   this phase of our study, designed to assess the effectiveness of the irdl program, provides compelling evidence that a continuing education program designed like irdl can have an impact on the research productivity, job performance, and professional identity of its participants. significant findings from this study demonstrate research persistence: an ongoing commitment to conducting and sharing the results of research beyond the program. we found that scholars chose more sophisticated and desirable research output formats after irdl, enjoyed an expanded peer and community group related to research activities, and demonstrated a meaningful increase in their professional identity as a librarian-researcher.   author contributions   frans albarillo: investigation, formal analysis (lead), data curation, methodology (lead), writing – review & editing marie r. kennedy: conceptualization, formal analysis (supporting), funding acquisition (lead), investigation, methodology (supporting), project administration, supervision, writing – original draft, writing – review & editing kristine r. brancolini: conceptualization, funding acquisition (supporting), methodology (supporting), writing – original draft, writing – review & editing   acknowledgment   this research was made possible in part by the institute of museum and library services laura bush 21st century librarian program grant re-40-16-0120-16. we would also like to thank our reviewers for their thorough critique of the manuscript, which improved the manuscript in significant ways. we also want to say thank you to the irdl community who made this research possible.   references   bandura, a. (1993). perceived self-efficacy in cognitive development and functioning. educational psychologist, 28(2), 117–148. https://doi.org/10.1207/s15326985ep2802_3   benedetti, a., jackson, j., & luo, l. (2018). vignettes: implications for lis research. college & research libraries, 79(2), 222–236. https://doi.org/10.5860/crl.79.2.222   brancolini, k. r., & kennedy, m. r. (2017). the development and use of a research self-efficacy scale to assess the effectiveness of a research training program for academic librarians. library and information research, 41(124), 44–84. https://doi.org/10.29173/lirg760   brew, a., boud, d., namgung, s. u., lucas, l., & crawford, k. (2016). research productivity and academics’ conceptions of research. higher education, 71, 681–697. https://doi.org/10.1007/s10734-015-9930-6   chang, y.-w. (2016). characteristics of articles co-authored by researchers and practitioners in library and information science journals. the journal of academic librarianship, 42(5), 535–541. https://doi.org/10.1016/j.acalib.2016.06.021   chang, y.-w. (2021). academic impact of articles by practitioners in the field of library and information science. college & research libraries, 82(1), 59–74. https://doi.org/10.5860/crl.82.1.59   cohen, j., cohen, p., west, s. g., & aiken, l. s. (2003). applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). lawrence erlbaum associates; routledge. https://doi.org/10.4324/9780203774441   creswell, j. w., & creswell, j. d. (2018). research design: qualitative, quantitative, and mixed methods approaches (5th ed.). sage.   creswell, j. w., & plano clark, v. l. (2018). designing and conducting mixed methods research (3rd ed.). sage.   exner, n. (2019). development of research competencies among academic librarians [doctoral dissertation, university of north carolina]. unc chapel hill university libraries carolina digital repository. https://doi.org/10.17615/p159-ya91   galbraith, q., smart, e., smith, s. d., & reed, m. (2014). who publishes in top-tier library science journals? an analysis by faculty status and tenure. college & research libraries, 75(5), 724–735. https://doi.org/10.5860/crl.75.5.724   hoffmann, k., berg, s. a., & koufogiannakis, d. (2014). examining success: identifying factors that contribute to research productivity across librarianship and other disciplines. library and information research, 38(119), 13–28. https://doi.org/10.29173/lirg639   hoffmann, k., berg, s., & koufogiannakis, d. (2017). understanding factors that encourage research productivity for academic librarians. evidence based library and information practice, 12(4), 102–128. https://doi.org/10.18438/b8g66f   jarvis, p. (1999). the practitioner-researcher: developing theory from practice. jossey-bass.   jason, d. p., iii, kennedy, m. r., & brancolini, k. r. (2021). mentoring academic librarians for research success. in l. j. rod-welch & b. e. weeg (eds.), academic library mentoring: fostering growth and renewal (pp. 241–262). association of college and research libraries.   kennedy, m. r., & brancolini, k. r. (2012). academic librarian research: a survey of attitudes, involvement, and perceived capabilities. college & research libraries, 73(5), 431–448. https://doi.org/10.5860/crl-276   kennedy, m. r., & brancolini, k. r. (2018). academic librarian research: an update to a survey of attitudes, involvement, and perceived capabilities. college & research libraries, 79(6), 822–851. https://doi.org/10.5860/crl.79.6.822   kennedy, m. r., brancolini, k. r., & kennedy, d. p. (2020). an exploratory study of accomplished librarian-researchers. evidence based library and information practice, 15(1), 179–217. https://doi.org/10.18438/eblip29655   kennedy, m. r., kennedy, d. p., & brancolini, k. r. (2017). the evolution of the personal networks of novice librarian researchers. portal: libraries & the academy, 17(1), 71–89. https://doi.org/10.1353/pla.2017.0005   laerd statistics. (2015a). mcnemar's test using spss statistics. statistical tutorials and software guides. https://statistics.laerd.com/   laerd statistics. (2015b). paired-samples t-test using spss statistics. statistical tutorials and software guides. https://statistics.laerd.com/   laerd statistics. (2016). cochran-armitage test of trend. statistical tutorials and software guides. https://statistics.laerd.com/   laerd statistics. (2017). binomial logistic regression using spss statistics. statistical tutorials and software guides. https://statistics.laerd.com/   luo, l. (2011). fusing research into practice: the role of research methods education. library and information science research, 33(3), 191–201. https://doi.org/10.1016/j.lisr.2010.12.001   luo, l., & mckinney, m. (2015). jal in the past decade: a comprehensive analysis of academic library research. the journal of academic librarianship, 41(2), 123–129. https://doi.org/10.1016/j.acalib.2015.01.003   matusiak, k. k., & bright, k. (2020). teaching research methods in master’s level lis programs: the united states perspective. journal of education for library and information science, 61(3), 357–382. http://doi.org/10.3138/jelis.61.3.2020-0001   pett, m. a. (2016). nonparametric statistics for health care research (2nd ed.). sage. https://us.sagepub.com/en-us/nam/nonparametric-statistics-for-health-care-research/book240591   powell, r. r., baker, l. m., & mika, j. j. (2002). library and information science practitioners and research. library & information science research, 24(1), 49–72. https://doi.org/10.1016/s0740-8188(01)00104-9   sassen, c., & wahl, d. (2014). fostering research and publication in academic libraries. college & research libraries, 75(4), 458–491. https://doi.org/10.5860/crl.75.4.458   vilz, a. j., & poremski, m. d. (2015). perceptions of support systems for tenure-track librarians. college & undergraduate libraries, 22(2), 149–166. https://doi.org/10.1080/10691316.2014.924845   watson-boone, r. (2000). academic librarians as practitioner-researchers. the journal of academic librarianship, 26(2), 85–93. https://doi.org/10.1016/s0099-1333(99)00144-5   wilson, v. (2013). formalized curiosity: reflecting on the librarian practitioner-researcher. evidence based library and information practice, 8(1), 111–117. https://doi.org/10.18438/b8zk6k   yanos, p. t., & ziedonis, d. m. (2006). the patient-oriented clinical researcher: advantages and challenges of being a double-agent. psychiatric services, 57(2), 249–253. https://doi.org/10.1176/appi.ps.57.2.249   appendix a: learning objectives for irdl summer research workshop   course objectives   at the end of this 10-day workshop, you will be able to: write effective research questions and hypotheses choose an appropriate research design for a library science study explain the conceptual logic behind various data collection approaches and describe the rationale for selection of specific methods identify appropriate sampling strategies for research projects use and apply commonly used qualitative data collection methods – focus groups and in-depth interviews  assess and apply different qualitative data analysis options design and implement a survey understand survey data management  explain various analytic options for surveys   appendix b: survey instrument   there are three thematic sections: research productivity, job performance, and professional identity. the survey should take around 30 40 minutes. you can return to the survey at any time by clicking on the invitation link to your email. please note that the survey will save the last answer you submit by clicking "ok" if it appears and advancing to the next question.   section 1: research productivity   1.       what research method(s) did you use in your irdl project? [text box response] skip to 2   2.       how much of your irdl project were you able to complete? choose the step that best describes the current state of your irdl project. [multiple choice] o    i completed the revised written proposal (skip to 3) o    i submitted an irb application (skip to 3) o    i received irb approval (skip to 4) o    i started data collection (skip to 4) o    i completed data collection (skip to 5) o    i started organizing the data i collected for analysis (skip to 5) o    i completed a preliminary analysis of the data i collected (skip to 6) o    i completed a full analysis of the data i collected (skip to 6) o    i reported the results of the data i collected (skip to 7) o    i published the results of the data i collected (skip to 7)   3.       how likely are you to receive irb approval for your project? [dropdown] –    extremely unlikely (skip to 8) –    unlikely (skip to 8) –    likely (skip to 4) –    extremely likely (skip to 4)   4.       how likely are you to complete the data collection for your project? [dropdown] –    extremely unlikely (skip to 8) –    unlikely (skip to 8) –    likely (skip to 5) –    extremely likely (skip to 5)   5.       how likely are you to complete the data analysis for your project? [dropdown] –    extremely unlikely (skip to 8) –    unlikely (skip to 8) –    likely (skip to 6) –    extremely likely (skip to 6)   6.       how likely are you to disseminate the results of your irdl project through presentation or publication? [dropdown] –    extremely unlikely (skip to 8) –    unlikely (skip to 8) –    likely (skip to 7) –    extremely likely (skip to 7)   7.       how did you disseminate or attempt to disseminate your irdl project findings? please use the adjacent column to indicate the number of times you disseminated your results for that particular format. (skip to 8)   how many times?   i presented my results at my library or institution [dropdown] 0-10; more than 10 i published results at my library or local institution (for example in a library meeting, newsletter, report, or another kind of local publication) [dropdown] 0-10; more than 10 i submitted a proposal to present results in a webinar (adobe connect, webex, etc.) [dropdown] 0-10; more than 10 i presented results in a webinar (adobe connect, webex, etc.) [dropdown] 0-10; more than 10 i submitted a proposal to present results as a poster at a regional, national, or an international conference [dropdown] 0-10; more than 10 i presented results as a poster at a regional, national, or an international conference [dropdown] 0-10; more than 10 i submitted a proposal to present results as a presentation, paper, or panel at a regional, national, or an international conference [dropdown] 0-10; more than 10 i presented results as a presentation, paper, or panel at a regional, national, or in an international conference [dropdown] 0-10; more than 10 i submitted an article to a non-peer reviewed journal or non-peer reviewed professional publication [dropdown] 0-10; more than 10 i published an article in a non-peer reviewed journal or a non-peer reviewed professional publication [dropdown] 0-10; more than 10 i submitted an article to a peer-reviewed journal [dropdown] 0-10; more than 10 i published an article in a peer-reviewed journal [dropdown] 0-10; more than 10 i submitted a chapter for inclusion in an edited volume [dropdown] 0-10; more than 10 i published a book chapter [dropdown] 0-10; more than 10 i submitted an edited volume proposal to a publisher [dropdown] 0-10; more than 10 i published an edited volume [dropdown] 0-10; more than 10 i submitted a book or monograph proposal to a publisher [dropdown] 0-10; more than 10 i published a book or monograph [dropdown] 0-10; more than 10   8.       of the barriers listed below, please indicate how intense that barrier was in completing your irdl project. (skip to 9)   not a barrier   somewhat of a barrier   moderate barrier     extreme barrier   lack of experience using specific research methods         given new job responsibilities in the same position         changed jobs         lack of access to needed research-based literature         receiving institutional review board (irb) approval         access to the study population         access to online survey software         knowing how to use online survey software         organizing and preparing quantitative data for analysis         analyzing quantitative data         organizing and preparing qualitative data for analysis         analyzing qualitative data         low response rates from study participants         i collected unusable data         reporting results in a written format         reporting results in a webinar format         reporting results as a poster presentation, presentation, or panel in a meeting, conference, or workshop format           9.       in relation to the letter of support written by your dean or director, did you receive any of the following supports from your library or institution while conducting your irdl project? (skip to 10)     [dropdown] at least half a day a week release from work duties to conduct your irdl project yes/no financial support yes/no moral support from my supervisor yes/no mentorship from my supervisor yes/no mentorship from someone other than my supervisor yes/no   10.    have you conducted other research projects since irdl? [multiple choice] o    yes, i am currently conducting one other research project (skip to 11) o    yes, i conducted one other research project (skip to 11) o    yes, i am currently conducting more than one other research project (skip to 11) o    yes, i conducted more than one other research project (skip to 11) o    no (skip to 13)   11.    have you disseminated or attempted to disseminate the results of your non-irdl research? [dropdown] o    yes (skip to 12) o    no (skip to 13)   12.    how did you disseminate or attempt to disseminate your non-irdl project(s) findings? please use the adjacent column to indicate the number of times you disseminated your results for that particular format.   how many times?   i presented my results at my library or institution [dropdown] 0-10; more than 10 i published results at my library or local institution (for example in a library meeting, newsletter, report, or another kind of local publication)   [dropdown] 0-10; more than 10 i submitted a proposal to present results in a webinar (adobe connect, webex, etc.) [dropdown] 0-10; more than 10 i presented results in a webinar (adobe connect, webex, etc.) [dropdown] 0-10; more than 10 i submitted a proposal to present results as a poster at a regional, national, or an international conference [dropdown] 0-10; more than 10 i presented results as a poster at a regional, national, or an international conference [dropdown] 0-10; more than 10 i submitted a proposal to present results as a presentation, paper, or panel at a regional, national, or an international conference [dropdown] 0-10; more than 10 i presented results as a presentation, paper, or panel at a regional, national, or in an international conference [dropdown] 0-10; more than 10 i submitted an article to a non-peer reviewed journal or non-peer reviewed professional publication [dropdown] 0-10; more than 10 i published an article in a non-peer reviewed journal or a non-peer reviewed professional publication [dropdown] 0-10; more than 10 i submitted an article to a peer-reviewed journal [dropdown] 0-10; more than 10 i published an article in a peer-reviewed journal [dropdown] 0-10; more than 10 i submitted a chapter for inclusion in an edited volume [dropdown] 0-10; more than 10 i published a book chapter [dropdown] 0-10; more than 10 i submitted an edited volume proposal to a publisher [dropdown] 0-10; more than 10 i published an edited volume [dropdown] 0-10; more than 10 i submitted a book or monograph proposal to a publisher [dropdown] 0-10; more than 10 i published a book or monograph [dropdown] 0-10; more than 10   13.    please upload your current cv.   section 2 job performance   14.    please select all that apply from the following research support options that your current institution or library provides for librarians. [checkboxes] (skip to 15) �    release time during the work week �    short-term or pre-tenure research leave �    sabbaticals �    travel funds (full reimbursement) �    travel funds (partial reimbursement) �    research design consultant or statistical consultant �    research grants �    research assistants (e.g., student workers or short-term project hires) �    formal research mentors �    workshops or other forms of continuing education �    no research support �    other (please specify)   15.    during your irdl year, were you employed in a tenure-track, continuing appointment, or promotion-eligible position? [multiple choice] o    tenure track with the possibility of promotion (skip to 16) o    continuing appointment with the possibility promotion (skip to 16) o    continuing appointment without the possibility of promotion (skip to 16) o    promotion only (skip to 16) o    neither continuing appointment, tenure, or promotion (skip to 17) o    other, please let us know about your institution’s appointments and promotion status during your irdl project year. (skip to 17)   16.    did you attain a continuing appointment, tenure, or promotion at the institution where you were employed during your irdl year? [multiple choice] o    yes, tenure, but not promotion (skip to 17) o    yes, tenure and promotion (skip to 17) o    yes. a continuing appointment, but not promotion (skip to 17) o    yes. a continuing appointment and promotion (skip to 17) o    yes, promotion (skip to 17) o    no (skip to 17)   17.    what is your current academic rank? (skip to 18) –    librarian 1 –    librarian 2 –    librarian 3 –    librarian 4 –    librarian 5 –    instructor –    lecturer –    assistant professor –    associate professor –    professor –    assistant librarian –    associate librarian –    librarian –    senior librarian –    adjunct or visiting instructor –    adjunct or visiting lecturer –    adjunct or visiting assistant professor –    adjunct or visiting associate professor –    adjunct or visiting professor –    visiting assistant librarian –    visiting librarian –    visiting associate librarian –    visiting senior librarian –    other   18.    do you believe irdl contributed to any of these following job-related factors? choose all that apply. (skip to 19) �    i earned a promotion in rank �    i received a promotion to a higher-level position at my institution �    i achieved tenure �    i received a certificate of continuous employment or a similar guarantee of job security �    i received new job title at my institution �    i got a one-time monetary salary award (e.g., a merit increase) �    i got a permanent raise in my base salary �    i acquired new job responsibilities related to my skills as a social science researcher at my institution (e.g., assessment projects, or projects that measure the impact of services) �    my research skills helped me secure a job at a new institution �    i started a local writing or research group �    i became a member of the irb �    i have advised other librarians about their research projects �    i made presentation(s) based on the skills i gained during the irdl workshop �    i have given workshops based on the skills i gained during the irdl workshop �    i extended my personal learning network �    i gained research collaborators �    other (please specify)   section: 3 professional identity   19.    are you currently working towards a certificate or an additional degree? check all that apply. [checkboxes] �    yes, a certificate �    yes, a thesis-based masters �    yes, a non-thesis-based masters �    yes, a phd, edd, or jd �    no, i am not working towards an additional certificate or degree   20.    have you completed an additional certificate or degree since your irdl year? check all that apply. [checkboxes] �    yes, a certificate �    yes, a thesis-based masters �    yes, a non-thesis-based masters �    yes, a phd, edd, or jd �    no, i am not working towards an additional certificate or degree   21.    in your own words, please describe what it means to be a “librarian-researcher.” [textbox]   22.    before your irdl year did you identify as a librarian-researcher? [multiple choice] o    yes o    no   23.    do you currently identify as a librarian-researcher?” [multiple choice] o    yes o    no   24.    is there anything about the impact (both short-term and long-term) of your experience as an irdl scholar on your research productivity, job performance, or identity as a researcher that we have not asked but you think is important for us to know? [textbox]   25.    are you interested in participating in a focus group or an interview for a further monetary incentive? the focus group and in-depth interview incentive is a $100 amazon gift card. focus groups and in-depth interviews will take place between january 6 to may 3, 2020. [multiple choice] o    yes o    no   26.    please provide an email where [name deleted] may contact you to schedule your participation in a focus group or interview. these will take place from january 6 to may 3, 2020. [textbox]   end of survey   appendix c: recruitment email   dear irdl scholar:   i am inviting you to participate in a survey of past participants of the institute for research design in librarianship.   the purpose of this survey is to measure the impact of irdl on your research productivity, job performance, and professional identity. i designed this survey in collaboration with irdl co-directors kristine brancolini and marie kennedy. we plan to publish and present the results of this study.   only frans albarillo, the lead principal investigator, will have access to the raw data, which he will anonymize. all data in reports, publication, and presentation of the data will be anonymous and analyzed in aggregate.   what will happen during the study   we will ask you to take two actions: 1. upload or send your current cv to frans albarillo at falbarillo@brooklyn.cuny.edu, so that we may examine your scholarly productivity since irdl. 2. complete a web-based survey. in the survey we will ask you to click through a series of questions with options for response. the survey is expected to take 15 to 20 minutes to complete.   your privacy is important we will make every effort to protect your privacy. no sensitive information will be gathered as part of this survey. any information you provide will remain confidential. only frans albarillo will view the results of the survey in their raw form.   your rights your participation in this study is completely voluntary and no risks are anticipated for you as a result of participating. if you decide to be in the study, you will have the right to stop participating at any time.   incentive when [name deleted] has confirmed that your cv has been received and the survey completed, you will be sent a $50 amazon electronic gift card.   institutional review board approval this is an irb-approved study, hrpp file number 2019-0747. brooklyn college, city university of new york is the irb of record. the irb coordinator is twyla tate, research compliance manager, and can be reached by email at twyla.tate@brooklyn.cuny.edu or by telephone at 718.951.500 ext. 3829. please don’t hesitate to contact me or the irb if you have any questions or concerns about the survey.   if you agree with all of the above statements, provide your consent to participate by clicking on the survey link below.   sincerely, frans albarillo 2014 irdl scholar reference and instruction librarian associate professor brooklyn college, city university of new york email: falbarillo@brooklyn.cuny.edu phone: (718)758-8213     research article   election voting and public library use in the united states   brady d. lund phd student emporia state university emporia, kansas, united states of america email: blund2@g.emporia.edu   beth l. hendrickson mls student emporia state university emporia, kansas, united states of america email: hendricksonbethb@gmail.com   matthew walston mls student emporia state university emporia, kansas, united states of america email: mwalston@g.emporia.edu   received: 19 aug. 2020                                                             accepted: 28 oct. 2020      2020 lund, hendrickson, and walston. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29824     abstract   objective this study examines whether a correlation exists between state-wide voting in federal elections and state average per capita visits to public libraries in the u.s. in so doing, it provides insight into the extent to which library patronage is affiliated with political leaning.   methods an analysis of data from the 2010, 2012, 2014, and 2016 public libraries survey and election results from the 2010, 2012, 2014, and 2016 presidential and house of representatives elections (by state) is performed with the assistance of tableau, a data visualization program. scatter plots provide a visual representation of the data, while correlation coefficients indicate the strength of relationship between voting and library visits per capita.   results the findings reveal no significant relationship between public library use and the vote share of a political party in elections among a state's population.   conclusions the political leaning of a state appears to have no correlation with the frequency of library usage among that state’s population.     introduction   in recent decades, several studies have examined the demographics of library patrons and which associated factors help to predict library use and non-use. ethnicity, age, and educational attainment, for instance, all appear to have a modest effect on the extent to which individuals utilize their public library (sin, 2012; sin & kim, 2008). these findings are compelling for library administrators, who may use them to inform outreach and services, as well as to argue for increased spending allocations. one central demographic attribute among many modern americans is their political identity. political leaning has not yet been well-examined in relation to public library usage. however, as with factors such as ethnicity and age, it is important to know whether political leaning impacts library usage, so that libraries may better conduct outreach and advocate for their role within their communities.   this study addresses the gap in literature relating to political leaning and library usage by performing an analysis of data from the 2010, 2012, 2014, and 2016 elections for the house of representatives, and the 2012 and 2016 presidential elections in the u.s., in relation to public library usage statistics from the 2010, 2012, 2014, and 2016 institute of museum and library services’ (imls) public library survey. correlation analyses of state-wide election results and median per-capita library visits are performed. the findings of these analyses may give public library employees, administrators, and researchers a point of advocacy in terms of the political diversity of library patrons.   literature review   studies involving the relationship between library usage and various population demographics individuals have existed for well over a century. some of the earliest library-related research studies worked to develop a profile of the “typical” library patron and the community in which the library served (wheeler, 1924; gaskill, dunbar, & brown, 1934). over time, these descriptive studies of library patrons have become an increasingly perfected art. researchers like sin (2012) and sin and kim (2008 have recently used advanced statistical techniques like logistical regression to analyze the relationships between a variety of demographic variables and library use and non-use.   several recent studies examined the concept of “motivation” to use libraries (aabo & strand, 2004; lee, 2007) or deflated motivation or interest that contributes to library non-use (mcnicol, 2004; nackerud, fransen, peterson, & mastel, 2013). often, the strongest motivating factors are found to be related to outcomes rather than demographic background, for example the motivation to address an information need. for the purposes of funding libraries, these findings are informative, as they show that libraries serve the information needs of diverse populations, not significantly skewed to one group of individuals. in the perspective of politicians, this likely makes funding libraries a less partisan issue.   a popular source of secondary data for studies of library usage is the imls’ public library survey (pls) data (https://www.imls.gov/research-evaluation/data-collection/public-libraries-survey). huang and tahamtan (2018) identified a variety of factors that predict library usage by the service population, such as the total expenditures of the library. joo and cahill (2019) also used the pls to analyze the relationship between library expenditures and usage by children and young adults. additionally, burke (2018) used this data source to identify connections between service availability and usage of public libraries. kim and yu (2011) even suggested using the pls data to develop an alternative model of public library management. this data can be used in conjunction with other data sets, such as state-wide voting statistics, to create new insights about library services and use.   starting in 2008, the online computer library center (oclc) in conjunction with the american library association published a report entitled “from awareness to funding: a study of library support in america.” this report examines a variety of factors related to library funding and support (de rosa & johnson, 2008). major sections of this report are dedicated to the demographics of who supports public libraries and why. while politically liberal individuals have been shown in these reports to be the group most likely to support public libraries even if they do not personally use them, all political groups have been shown to use the library regardless of stated political affiliation. furthermore, members of city councils were surveyed and indicated high levels of library support (even in comparison to the general population). the authors of the report conclude that “library funding support is an attitude, not a demographic” (de rosa & johnson, 2008); support cuts across groups, rather than being directly affiliated with any one group in particular.   while a few studies have looked at relationships between library usage and political leanings as part of larger studies of library use predictors, there are no specific longitudinal analyses on this topic. carlozzi’s (2018) study, for instance, examined a myriad of factors that may contribute to usage of a specific public library system. using regression analysis, a model that indicated a small effect attributable to political leaning was provided. this study, however, examined only one specific context (public libraries in massachusetts), rather than a broader scope such as state-wide correlations across the u.s.   aims   the aim of this study is to examine whether any correlation exists between election outcomes and public library usage in the u.s. the findings of this study may inform advocacy or outreach to underserved populations, as well as potentially serving as a counter or justification against politicized library funding decisions. this study facilitates a more complete understanding of the typical library user, by providing an indication for whether political affiliation is a key distinguishing factor related to the frequency of library use. this study is therefore guided by the following research question:   does a correlation exist between state-wide voting share in federal elections and state average per capita visits to public libraries in the u.s.?   methods   this study analyzes data acquired from 3 publicly-available data sets: 1) the imls’ pls data for the fiscal years 2010, 2012, 2014, and 2016, 2) united states census data for the same 4 years, and 3) the election results by state for the house elections in 2010, 2012, 2014, and 2016, the presidential elections in 2012 and 2016, and the composition of the state senates for the 2016 legislative sessions.   for the house data, a percentage was found by dividing the number of house positions won by democratic representatives against the total number of seats available. for the presidential data, the percentage of votes for the democratic candidates were collected by state. for the senate composition, the number of democratic seats held was compared to the total number of seats on the states’ senates.   only democratic party voting data were considered, as an assumption was made that given the u.s.’ two-party system, a direct, inverse relationship exists between democratic vote share and republican vote share. however, there are some third-party votes in every election, for example the particularly high third-party share of 2016. the data analysis methods may therefore lead to some discrepancy in the relationship between the vote share of the two major parties, which is a limitation that may be examined further in future studies.   the data were analyzed using tableau, a free data visualization and business intelligence software. from the pls, data on the number of visits for each state were accumulated by adding the total visits for each public library system within it. the u.s. census data provides the most accurate measure of the population of each state. the average number of visits per person was then calculated using a simple excel function. visits per person by state was the first factor in every comparison, as calculating in this way rather than using the number of visits alone eliminates the influence of population size on the totals.   election data were gathered from data.gov and transformed into percentages. subsequently, these data were transferred from excel to tableau, where they could best be manipulated. first, a scatter plot was developed for each set, with the percent democratic vote on the x-axis and average library visits per person on the y-axis. a trend line was placed in this plot using the slope-intercept formulation. spearman correlation coefficients and corresponding p-values were also retrieved from tableau.     figure 1 correlation between 2010 house vote and number of library visits.   figure 2 correlation between 2012 presidential vote and number of library visits.     results   figure 1 displays the data visualization for the 2010 house election. average library visits per person range from 3.14 in texas to 7.65 in ohio, with an average among all states of 4.67. the percentage of democratic representatives ranges from 0% in nine states to 100% in seven states, with an average of 12.5%.  among the states with 0 seats won by democratic candidates, visits per person range from 3.6 in north dakota to 6.9 in wyoming. among those with 100% of seats awarded to democrats, visits per person range from 3.92 in hawaii to 6.98 in connecticut. the correlation for these data sets is 0.29 with a p-value of 0.04. there is a statistically significant increase in visits per person based on the percentage of democratic vote, but the relationship is very weak.   the data visualization for the 2012 presidential election is shown in figure 2. in the 2012 election, utah has the lowest percentage of democratic votes at 24.8%, but the third highest number of visits per person. hawaii has the highest percentage of democratic votes at 70.6, though just an above average number of visits per person. the correlation for these data is 0.3 with a p-value of 0.03. there is a very weak relationship between the presidential vote by state and the number of library visits per person.   shown in figure 3 is the data plot for the correlation between visits per person and the number of democratic representatives elected in the 2012 house election. ten states elected no democratic representatives, ranging from 3.2 visits per person in north dakota to 6.2 visits per person in wyoming. eight states elected 100% democratic representatives ranging from 3.8 visits per person in hawaii to 6.8 visits per person in connecticut. the correlation for these data is 0.34 with a p-value of 0.01.   figure 4 displays the data for the 2014 house election. overall visits in 2014 ranged from 2.7 in texas to 7.1 in ohio. in this year, 6 states elected democratic representatives to all open positions while 11 states elected no democratic representatives. among the former group, the number of visits per person ranged from 3.4 in hawaii to 6.2 in massachusetts. the latter group ranged from 2.9 in west virginia to 6.3 in wyoming. the correlation for this comparison is 0.28 with a p-value of 0.05. the results show only a slight significance with a very weak positive correlation for democratic-favoring states.     figure 3 correlation between 2012 house vote and number of library visits.     figure 4 correlation between 2014 house vote and number of library visits.     three different data sets were compiled for the 2016 elections. the first, the 2016 presidential election, is shown in figure 5. this presidential election seemed to be very unique and polarizing compared to preceding elections; this is reflected to a small extent in the data findings. wyoming had the lowest democratic vote percentage at 21.6% but continued to have very high visits per person at 6.1 (ranking only behind ohio for the most). hawaii continued to have the highest democratic percentage at 62% and continued to have a low visit per person rate at 3.15. there continued to be a small positive relationship between percentage of democratic votes and visits, but the correlation is the weakest so far at 0.22 with a p-value of 0.11. this means that, for the first time in the data, the positive correlation falls well within the margin of error.   the results for the 2016 house election are displayed in figure 6. eleven states elected no democratic candidates, with visits per person ranging from 2.53 (arkansas) to 6.1 (wyoming). six states elected all democratic candidates, with visits per person ranging from 3.15 (hawaii) to 5.97 (vermont). the correlation among these data is 0.32 with a p-value of 0.02.     figure 5 correlation between 2016 presidential vote and number of library visits.     figure 6 correlation between 2016 house vote and number of library visits.     the third set of data for the 2016 election cycle is the composition of the state senates, shown in figure 7. as state and local governments are mostly responsible for funding public libraries, one might project that these data would show the strongest correlation yet, however the exact opposite is true. wyoming has the smallest composition of democrats within its state senate at 10%, with the second largest number of visits per person. regarding hawaii, 100% of the state senate identifies with the democratic party, while the state averages only 3.15 visits per person. although 3 of the 5 states with the smallest democratic compositions fall in the top 10 of states by visits per person, this is identical to the 5 states with the highest democratic compositions. the correlation for these data is only 0.2 with a p-value of 0.18. the slope of the trend line is 0.011 visits per person per percent point of democratic senators. in other words, a 20% shift in a state’s senate composition in favor of democrats would only correspond to a 0.2 increase in library visits per person.     figure 7 correlation between composition of state senate and number of library visits.     figure 8 correlation between 2012 presidential election vote and number of library visits, with the 10 biggest outliers removed.     figure 8 depicts the relationship for the 2012 presidential election, if the 10 biggest outliers such as wyoming and hawaii, were removed from the data. in this case, the correlation between the two variables rises to 0.71, with a p-value <0.001. this indicates a strong positive relationship between library visits and the percentage of democratic vote, however it also removes 20% of the overall data. so, while there is an underlying relationship among some states, it is not evident in the complete data set.   finally, figure 9 displays longitudinal data for library visits in the entire u.s. alongside the percentage of votes for the democratic party in each house election from 2010 to 2016. there is a very weak, non-significant correlation (0.26; p-value of 0.73) for this data set. while there was a small but steady drop in visits per person over this six-year period, the percentage of democratic votes rose by five percentage points in 2012, before dropping in 2014 and 2016. overall, there seems to be very little statistical evidence that politics influences library visits, or even that there is a marked relationship among these variables.   discussion   this study indicates that, in general, no correlation exists between election vote share and public library visits within a state. one important potential reason for this lack of relationship is that library use statistics remained relatively consistent from 2010 to 2016, while vote shares did not. for instance, minnesota saw a 15% jump in democratic vote share from 2010 to 2012, though the average number of library visits remained relatively stable. notably, the “rust belt” states of wisconsin, michigan, and pennsylvania, which voted majority democratic in 2012, flipped to majority republican in 2016, however they did not experience a proportional drop in library use. political opinions, at least in the short term, appear to be simply too volatile. longer-term shifts, like that of the southeastern u.s. from majority democratic to majority republican from the 1960s to 1990s, may be more likely to reveal a significant trend.   state senate race outcomes seem to be particularly indicative of the lack of correlation between voting outcomes and library use, as state senate voting occurs at a more local scale. for instance, in kansas, there are 40 senate seats, including over 20 divided among the state’s 3 major metropolitan areas (kansas city, wichita, and topeka). however, this approach is complicated by partisan gerrymandering, which is an approach to drawing congressional districts used by states dominated by a single political party, in order to increase the likelihood of their party retaining power during elections. this makes the composition of state senates one of the least reliable indicators of political sentiment. as noted in the limitations below, municipal (city council, mayoral) voting may be the best outcomes for future studies to utilize.     figure 9 house vote percent democrat for all states and average number of library visits.     figure 10 united states census regions: west, midwest, northeast, and south.     perhaps the most compelling findings in this study may be in figure 8, where the 10 biggest outliers from the general trend are removed. in the case of the 40 remaining states, a significant positive relationship does exist. however, this is true of many datasets; if enough data that disagree with a trend are removed, eventually that trend will emerge from the remaining data. this is one of the major shortcomings of sampling methods in empirical research and why a sampling of states was not used in this study.   furthermore, the potential for hidden variables that explain variation in the data must be considered. one such potential variable evident in figure 8 is geography. the states in the upper-right corner with high democratic share and high library visits are located in the northeast: vermont, new york, rhode island, massachusetts, new jersey, and connecticut. most states in the lower-left corner with low democratic share and low library visits are located in the south: oklahoma, west virginia, arkansas, kentucky, and alabama. so, even if a relationship did exist, the geographic alignment suggests that this might be the hidden variable that explains both politics and library visit frequency.   given the above observation about geographic relationships in both politics and public library visit frequency, an interesting follow up to this study may be to statistically evaluate relationships among geographic regions, such as the u.s. census regions shown in figure 10. regarding the 2016 presidential election data and library visit data, for instance, a significant difference can be found using a kruskal-wallis h test (non-parametric anova) for regional differences in both vote share (h = 4.06, p = .02) and public library visits (h = 5.55, p < .01). particularly, the south (in purple in figure 10) experiences low levels of both variables, while the midwest (yellow) has a lower democratic vote share compared to the west (red), but a higher number of library visits. this suggests that general cultural effects (including both library use and political leaning) attached to geography are a much more likely cause of variation.   limitations   a few limitations should be noted for this study. as mentioned earlier, only democratic party voting data were considered, which may lead to some discrepancy in the relationship between the vote share of the two major parties. this is a limitation that may be examined further in future studies. furthermore, this study, while technically longitudinal (using data from 2010 to 2016), selected a fairly short period of time in terms of politics; as noted above, the examination of an extended period of transition in american politics may be more insightful. also, this study used states as the unit of analysis, but most public libraries in the u.s. are municipally supported. looking at city politics, such as voting for city councils, and use of specific public libraries may offer some unique insight. carlozzi (2018) did examine this variable as a possible explanatory factor for libraries’ municipal appropriation and did find that a small effect existed. finally, there are always some limitations with incomplete data sources. if a library did not include its visits data in the pls data, then it had to be excluded from this analysis.   conclusion   this study investigated whether a correlation exists between the state-wide outcomes of elections and public library visits. the results demonstrate that state-wide voting share in the u.s. generally has no measurable relationship with public library usage in these states. regional variation is a much greater predictor of both of these variables. this finding separates political diversity from other variables like educational diversity, that may be used to estimate the rate of library visits. populations encompassing all different combinations of political affiliations, from the most conservative to the most liberal, utilize public libraries at similarly high levels. public library administrators may find this result useful for advocacy and outreach purposes, as it demonstrates to political decision makers that library use itself is not a political issue and that libraries may be treated as a neutral public good. these findings contribute to a growing body of literature that examine correlates of public library usage in the u.s., indicating that political polarization is not a factor that has a meaningful influence on library use.   references   aabo, s., & strand, j. (2004). public library valuation, nonuse values, and altruistic motivations. library & information science research, 26(3), 351-372. https://doi.org/10.1016/j.lisr.2004.01.003   burke, i. (2018). understanding public library services and use (doctoral dissertation). university of denver, denver, co.  https://digitalcommons.du.edu/etd/1494   carlozzi, m. (2018). the socioeconomic profile of well-funded public libraries: a regression analysis. evidence based library and information practice, 13(2), 13-26. https://doi.org/10.18438/eblip29332   de rosa, c., & johnson, j. (2008). from awareness to funding: a study of library support in america. dublin, oh: online computer library center. retrieved from https://www.oclc.org/research/publications/2008/funding.html   gaskill, h. v., dunbar, r. m., & brown, c. h. (1934). an analytical study of the use of a college library. the library quarterly: information, community, policy, 4(4), 564-587. retrieved from https://www.jstor.org/stable/4302123   huang, l. m., & tahamtan, i. (2018). why do people come? the factors influencing public library visits. proceedings of the association for information science and technology, 55(1), 832-833. https://doi.org/10.1002/pra2.2018.14505501136   joo, s., & cahill, m. (2019). the relationships between the expenditures and resources of public libraries and children’s and young adults’ use: an exploratory analysis of institute of museum and library services public library statistics data. journal of librarianship and information science, 51(2), 307-316. https://doi.org/10.1177/0961000617709057   kim, g., & yu, s. y. (2011). an exploratory study to develop an alternative model of public library management using the institute of museum and library services’ public library statistics. the library quarterly, 81(4), 359-382. https://doi.org/10.1086/661656   lee, s. (2007). vroom’s expectancy theory and the public library customer motivation model. library review, 56(9), 788-796. https://doi.org/10.1108/00242530710831239   mcnicol, s. (2004). investigating non-use of libraries in the uk using the mass-observation archive. journal of librarianship and information science, 36(2), 79-87. https://doi.org/10.1177/0961000604047134   nackerud, s., fransen, j., peterson, k., & mastel, k. (2013). analyzing demographics: assessing library use across the institution. portal: libraries and the academy, 13(2), 131-145. https://doi.org/10.1353/pla.2013.0017   sin, s. c. j. (2012). modeling the impact of individuals’ characteristics and library service levels on high school students’ public library usage: a national analysis. library & information science research, 34(3), 228-237. https://doi.org/10.1016/j.lisr.2012.01.002   sin, s. c. j., & kim, k. s. (2008). use and non-use of public libraries in the information age: a logistic regression analysis of household characteristics and library services variables. library & information science research, 30(3), 207-215. https://doi.org/10.1016/j.lisr.2007.11.008   wheeler, j. l. (1924). the library and the community. chicago, il: american library association.   feature   eblip7 closing keynote address: evidence based everything   charlene sorensen assistant dean (services to libraries) university library, university of saskatchewan saskatoon, saskatchewan, canada email: charlene.sorensen@usask.ca      2013 sorensen. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   the closing keynote speaker for eblip7 was canadian journalist, author, and lecturer, dan gardner.  this feature article summarises his keynote address which was based on a chapter of his upcoming book about forecasting, and explained how difficult it can be to think in an evidence based way. for thousands of years the concept of “evidence” did not exist and human life expectancy was dismal. life expectancy only improved in the 20th century when physicians started to question, inquire, and doubt some of the medical interventions instead of being so very sure that a treatment – like bleeding a patient – was effective. gardner posited that the ability to doubt, and in turn demand evidence to prove or disprove a theory, brought real progress to the field of medicine.   this all sounds great – except that our brains do not naturally function in an evidence based way. drawing upon the work of daniel kahneman, gardner provided an overview of our two modes of thought: system 1, which is unconscious, automatic, and effortless; and system 2, which is conscious, slow, and takes effort. because we have no conscious access to system 1, we often have strong intuitive responses that are not based on any real truth. for example, we are prone to confirmation bias, where people only seek out evidence that confirms their beliefs, and disconfirmation bias, where people set higher standards for evidence that contradicts their beliefs. these biases, among others, lead us to come to quick, confident conclusions based on scant evidence.   to demonstrate how difficult it can be to doubt and to force system 2 thinking in order to fight biases and automatic responses, gardner provided the example of archie cochrane (1909-1988), the pioneer of evidence based medicine. this man brought about lasting changes to the field of medicine by asking the question “how do you know that?” he spent his career challenging subjective opinions and he struggled to keep physicians from saying “i just know” instead of taking an evidence based approach. cochrane, however, was not immune to the power of system 1 thinking. he was referred to a surgeon because of suspected cancer and that surgeon performed an invasive surgery because it really seemed like cochrane had cancer. it was only after the surgery that the pathologist’s report came back stating that there was no cancer. the crazy thing is that cochrane did not doubt the surgeon and agreed to the surgery even though he knew the pathologist had not yet reported his results.   it really does seem like there is no hope for the rest of us if even someone so entrenched in evidence based medicine could have failed to insist upon an evidence based approach to his own health care. gardner did, however, provide some hope by highlighting that medicine has advanced and continues to advance by using the scientific method and valuing inquiry and doubt. therefore, other fields could potentially make similar advances in the future. the popularity of the phrase “evidence based” does point to some progress. in the end, we can all consider the question – why do i believe what i do – and attempt to use evidence to find the answer.   there were several questions from the audience related to ethical issues around non evidence based public health activities that may not be doing any harm, meta-cognition, prioritizing areas of society that should become evidence based, how to personally cope when we know how biased our brains can be, and some self-reflection about a tendency to the confirmation bias.   gardner’s comment that we naturally recognize the biases in others more than we see them in ourselves gives me hope for evidence based library and information practice. i believe that the high level of collaboration that we have established in many aspects of our work will serve us well to overcome these biases. as we learn more about how our brains work, as we practice asking the question “how do you know that”, and as we continue to be inspired by the eblip conferences, we will be better prepared for an evidence based future.   approaches to negotiating change through evolving library management styles in australian university librariesapproaches to negotiating change through evolving library management styles in australian university libraries evidence summary   approaches to negotiating change through evolving library management styles in australian university libraries   a review of: gunapala, m., montague, a., reynolds, s., & vo-tran, h. (2020). managing change in university libraries in the 21st century: an australian perspective. journal of the australian library and information association, 69(2), 191-214. https://doi.org/10.1080/24750158.2020.1756598   reviewed by: david dettman associate professor and library instruction program coordinator university of wisconsin-stevens point stevens point, wisconsin, united states of america email: ddettman@uwsp.edu   received: 1 sept. 2022                                                               accepted:  20 oct. 2022      2022 dettman. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30239     abstract   objective – investigating current change management practices to create a conceptual management framework for the 21st century.   design – interviews using a qualitative constructivist approach.   setting – australian university libraries.   subjects – chief university librarians of 18 public universities out of 37 in australia.   methods – chief university librarians in more than half of the public university libraries in australia were interviewed. the interviews were completed face-to-face using a semi-structured questioning approach, based on themes and concepts derived from the literature review. observation data were also gathered through physical visits to the libraries. the data analysis was conducted using two microsoft excel matrices, one grouped thematically and the other populated with relevant literature review commentary when it aligned with interviewee commentary. the conceptual framework used to guide the research is made up of six fundamentals of performance improvements to effectively manage change: resources, relevance, stakeholders, strategy, government policy, and university infrastructure. the research focused on current change management practices of chief librarians as they address these issues.   main results – the research revealed that the influence of, or the relationship between, the factors affecting changing university library environments creates a complex administrative environment where decision making addressing one of the fundamentals can have negative unintended consequences in one or more of the other key areas. the authors note that the literature and views of the informants show a change in the objectives of the future academic library characterized by, but not limited to, initiatives that are designed to meet changing needs of a diverse group of stakeholders. these objectives must be “innovative” and “add value to the university business rather than continue to do what was traditionally done” (gunapala et al., 2020, p. 203).   conclusion – the authors profess that the study provides theoretical insight to help library leaders address the many challenges currently in place and emerging across the australian university library landscape. they assert that the research reveals the need to shift focus from a more traditional transactional oriented model to an engagement orientated model, due to the introduction of market forces coupled with declining public funding. they conclude by claiming to provide a theoretical framework that when practically implemented will allow library leaders to successfully navigate and negotiate emerging changes across the spectrum of higher education.   commentary   positioning academic libraries to best deal with change is a topic that is covered significantly in the library leadership literature. although not a new topic, the article excels in the area of giving a multi-faceted view of the change landscape as opposed to focusing specifically on one aspect of change management, albeit those articles also have their place. this holistic approach allows the reader to get a sense of the delicate interplay between the forces that cannot be as fully appreciated in articles that focus on one aspect of change.   many expert views are shared regarding the change factors that make up the conceptual management framework, along with corresponding information gathered from the interviewed library leaders. although interesting and compelling, the data is presented in an informal style that fails to give a detailed picture of which of the different change factors take priority and are most top of mind for library leaders. a useful addition would be the inclusion of the matrix containing how often and to what lengths library leaders responded to questions regarding specific components that make up the framework. in short, having a more quantitative or mixed methods approach would have been preferable to the qualitative research method used.   the article contains a welcome summary bullet list of eight change objectives for the future of academic libraries, identified in the literature and addressed by chief university librarians during the interviews. it does not, however, contain the interview questions themselves, which negates the possibility of examining them through the lens of a qualitative critical appraisal tool. other limitations of the article include the lack of use of deidentified supporting quotes from the interview participants to underpin the themes arising from the data collection. basic demographic information about the 18 participants and the libraries they are employed by (for example, breakdown by gender, years in the profession, library size, university size) is also lacking. a final potential shortcoming is the omittance of a discussion section. this would have given the authors the opportunity to reflect upon the potential limitations of their research and potentially address the issues raised above.   the authors note that the data were collected in 2014, six years before the article’s publication. in defense of the current relevance of the data, the authors simply write that “the data collected remains valid as the change process continues rapidly” (gunapala et al., 2020, p. 196). many of the references that are foundational to the paper are well over a decade old and the paper makes multiple references to how the library landscape has changed since the 1990s. ultimately the reader will decide for themselves if the lapse of time has rendered some of the observations obsolete or imbued them with an urgency that may no longer be present.   this reviewer feels that despite the age of the references and six-year lag in between data collection and article publication, the conclusions arrived at by the analysis of the interview data are still applicable and provide a useful blueprint for academic library leaders in australia, and other countries around the world as they face many of the same change issues, constraints, and opportunities afforded by unprecedented transformation in higher education in general, and in academic libraries specifically.   references   gunapala, m., montague, a., reynolds, s., & vo-tran, h. (2020). managing change in university libraries in the 21st century: an australian perspective. journal of the australian library and information association, 69(2), 191-214. https://doi.org/10.1080/24750158.2020.1756598   evidence summary   ethnographic study at a music library found students prefer short stopovers and longer solitary study   a review of: hursh, d. w. & avenarius, c. b. (2013). what do patrons really do in music libraries? an ethnographic approach to improving library services. music reference services quarterly, 16(2), 84-108. doi:10.1080/10588167.2013.787522   reviewed by: dominique daniel humanities librarian for history and modern languages oakland university rochester, michigan, united states of america email: daniel@oakland.edu    received: 23 nov. 2013 accepted: 25 feb. 2014      2014 daniel. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to identify patterns of patron behaviour in the library in order to improve space utilization.   design – ethnographic data-gathering, including observations and a qualitative survey.   setting – music library of a large public university.   subjects – library patrons, primarily music students but also music faculty, other students and faculty, and regional music professionals and amateurs.   methods – in the exploratory phase, complete (i.e., incognito) participant observers recorded patron characteristics and behaviours in four zones of the library (the technology lab, the stacks, the reference area, and study carrels). they conducted a series of five-minute-long visual sweeps of these zones at five-minute intervals. observers were not given any checklist, but were told to record anything they saw regarding the personal characteristics, behaviours, and activities of patrons. the data collected resulted in what the investigators called “flip books” (a series of images recorded in close succession, which, when flipped, could give the illusion of movement). the data was analyzed using the grounded theory approach, a qualitative method to identify recurring themes on space use. a statistical analysis based on these themes was then conducted. in the second, explanatory phase, observers conducted new “sweeps,” or observations of the same library zones, this time using checklists to indicate the occurrence of specific activities identified in the first phase (solo vs. group activity, social interaction vs. study discussion, and use of technology). in addition, observers recorded patron entry and exit on “time cards,” and had all exiting patrons answer five brief questions about the types and volume of activities they had conducted in the various zones of the library.   main results – the vast majority of the patrons were students. most (at least three-quarters) engaged in solitary activity, and a large majority used electronic technology. according to data from the flip books, 44% engaged in multitasking, which was therefore significant but not preferred. it was more likely to occur when electronic technology was involved. patrons were most likely to be present in the library for less than 5 minutes or more than 20 minutes. patrons who stayed in the library for only a short time were more likely to engage in leisure activities than those who stayed longer, but leisure activities overall were as prevalent as study time. the technology lab and the reference area were the most popular zones. users stayed in the technology lab and stacks for short times only, whereas the reference area and carrels were favored for long visits. users engaged in multitasking mostly in the carrels and reference area.   conclusion – the patrons’ preference for solitary study is at odds with academic libraries’ current interest in collaborative learning spaces, but can be explained by the specific nature of music studies (artistic creation is a solitary activity), and is in line with previous ethnographic studies of public libraries. music students presumably use the technology labs for short visits between classes. they favor the study carrels for longer stays where they can multitask, using their own laptops and ipods. these findings can be used to help redesign the library. design recommendations include placing the technology lab by the entrance to enable quick coming and going, increasing the number of carrels, placing them in quiet parts of the library, and equipping them with electrical outlets.   commentary   this article adds to the growing body of qualitative research in library science using ethnographic methods. as the erial project has shown, ethnographic methods can provide in-depth information about users’ information behaviour. such methods are “inductive and hypothesis generating” (asher & miller, n.d., p. 3). accordingly, this study rightly aims to base space design on students’ learning needs, rather than operational considerations (bennett, 2005, p. 15).   the study was meticulously designed and implemented with the help of an ethnographer. the use of three different instruments allows for not only a detailed and nuanced analysis of patron behaviour in the library, but also a comparison of the validity and fruitfulness of the instruments. the research yielded a wealth of solid evidence about space use, which has interesting implications for the design of library spaces and service points. in particular, findings regarding the length of visits, the prevalence of solo activities, and the frequency of leisure activities complicate the widely held assumption of increasing demand for collaborative learning spaces in academic libraries. the authors mention some practical changes to improve space use at their institution, but more conclusions could be drawn from the findings.   surprisingly, the article’s literature review covers neither research on music libraries nor non-ethnographic studies of library spaces. without a review of general research on library space use, it is not clear whether this article’s findings are specific to music or even other departmental libraries. to better assess their finding of preference for solitary activity, for example, the authors could have relied on the extensive literature on “library as place” in addition to the ethnographic studies they mention. regardless of methodology, studies of undergraduates’ solo vs. group study preferences have had mixed results (applegate, 2009; fox & doshi, 2013; treadwell, binder & tagge, 2012; whitmire, 2001). it may be, as fox and doshi (2013) have concluded, that students value flexible space that allows for either group or individual study. music students may well be different from others, but the authors’ explanation that it is due to the nature of artistic creation does not rest on any evidence.   another consequence of the weak literature review is that the authors occasionally present findings without discussing their significance. a good example is the importance of leisure activities in the library, which the authors point out but do not analyze, although it could affect space design.   furthermore, as the authors admit, ethnographic research is time-consuming and complex to implement. it allows for “detailed and accurate results,” but some of the findings could be ascertained without such an elaborate method, like the need for more carrels located in quiet area and equipped with electrical outlets. the authors say little of the culture of study in the music department and their university as whole, although library facilities are best designed when considered in their broader institutional context (freeman, 2005, p. 7).   all in all, this study provides an excellent model for librarians interested in conducting an ethnographic study of space use. it presents different methods step by step and discusses benefits and drawbacks. but the article would benefit from in-depth analysis of the findings, rather than just the methodology.     references   applegate, r. (2009). the library is for studying: student preferences for study space. the journal of academic librarianship, 35(4), 341-346. doi:10.1016/j.acalib.2009.04.004   asher, a. & miller, s. (n.d.). so you want to do anthropology in your library? a practical guide to ethnographic research in academic libraries. erial project. retrieved 25 feb. 2014 from http://www.erialproject.org/wp-content/uploads/2011/03/toolkit-3.22.11.pdf   bennett, s. (2005). righting the balance. in library as place: rethinking roles, rethinking space. (pp. 10-24). washington, d.c.: council on library and information resources. retrieved 25 feb. 2014 from http://www.clir.org/pubs/reports/pub129/pub129.pdf/view   fox, r. & doshi, a. (2013). longitudinal assessment of “user-driven” library commons spaces. evidence based library and information practice, 8(2), 85-95. retrieved 25 feb. 2014 from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/19544/15214   freeman, g. t. (2005). the library as place: changes in learning patterns, collections, technology, and use. in library as place: rethinking roles, rethinking space. (pp. 1-9). washington, d.c.: council on library and information resources. retrieved 25 feb. 2014 from http://www.clir.org/pubs/reports/pub129/pub129.pdf/view   treadwell, j., binder, a., & tagge, n. (2012). seeing ourselves as others see us: library spaces through student eyes. in l. m. duke & a. d. asher (eds.), college libraries and student culture: what we now know. (pp. 127-142). chicago: american library association.   whitmire, e. (2001). a longitudinal study of undergraduates’ academic library experiences. the journal of academic librarianship, 27(5), 379-385. doi:10.1016/s0099-1333(01)00223-3   evidence based library and information practice editorial responsibilities      2021. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     editor-in-chief: ann medaille   associate editor (evidence summaries): stephanie krueger, heather macdonald, christina wissinger   associate editor (research articles): erin owens, lisl zach   associate editor (classics & reviews): jane schmidt   associate editor (commentaries, using evidence in practice): ann medaille   communications officer (news): kim mackenzie   production editor: rachel hinrichs   editorial intern: melissa cober   indexing coordinator: melissa cober   editorial advisors: alison brettle, michelle dunaway, lorie kloda, denise lafitte, heather pretty   copyeditors: heather healy (lead copyeditor), ève bourbeau-allard, kirstin duffin, julie evener, emily kingsland, christine moffatt, jane monson, jane morgan-daniel, elizabeth stregger, maria tan, nikki tummon, elaina vitale, ross wilson   editorial assistant: samantha sheplawy news/announcements   cara bradley receives robert h. blackburn distinguished paper award for paper published in eblip journal    2014. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   the canadian library association / association canadienne des bibliothèques has awarded cara bradley, teaching and learning librarian, university of regina, the 2014 robert h. blackburn distinguished paper award for her 2013 paper entitled, “information literacy articles in science pedagogy journals” published in evidence based library and information practice, vol 8, no 4 (2013) available from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/20230/15972.   the paper analyzed the prevalence of articles related to information literacy (il) in the 15 highest-impact science pedagogy journals during the ten year period 2002 – 2011 and found that librarians need to “redouble their efforts to publish and raise the profile of information literacy in science pedagogy journals, either alone or collaboratively with subject faculty.”   the r. h. blackburn award jury felt that bradley’s research built on past research, while opening a fresh perspective on a topic that should be of interest to academic librarians across canada.   the robert h. blackburn distinguished paper award is named for dr. blackburn, the first president of cacul (1963-64). the award annually recognizes notable research published by canadian library association members. the full text of the press release is available at http://www.cla.ca/am/template.cfm?section=home&contentid=15163&template=/cm/contentdisplay.cfm apply for the research training institute ‘22 news   apply for the research training institute ‘22    2021. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30083     are you a library professional who wants to take your research skills to the next level? do you have a brilliant idea or research topic that you want to pursue but do not know where to start and need help from your peers and experts  to carry it through?   the medical library association (mla) research training institute (rti) is a unique, highly effective, and collaborative online research training and support program. rti assessment findings show the rti learning model is sound, effective, and increases the research skills, confidence, and productivity of participants. the rti ‘22 immerses practicing librarians in scholarly research, inquiry, and publishing. librarians of all levels of professional experience and types of work environments who provide health information, services, and support; who have an interest in increasing their research skills and confidence; and who want to improve library and health care outcomes are encouraged to apply to the rti program.   the institute is a one-year online program that consists of a series of online modules in advanced research methods, mentoring by faculty experts and peer coaches, preparing and  implementing a research project, and an opportunity to present findings at the mla ’23 annual conference. rti ‘22 features an expanded research curriculum and greater affordability and flexibility for participants. learn more about rti program details.   rti ’22 applications will be open until january 12, 2022.   applicants are now being accepted through january 12, 2022, for the 2022 cohort of rti research fellows. accepted applicants will be notified in march 2022. see the rti submission process for eligibility requirements and selection  criteria. the institute offers many scholarship opportunities, including scholarships for professionals working in small libraries and academic health libraries, those engaging in diversity, equity, and inclusion (dei) research, and those with limited financial means.   discover and nurture your research talents! learn how to conduct and lead quality research projects to improve your library and support quality health care. meet like-minded research colleagues, make lifelong friends, gain research skills and confidence, form collaborations with other researchers, and help end users improve health and wellness! apply today!   read more about the rti success story.   the project is made possible in part by a grant from the u.s. institute of museum and library services (imls). applicants must be u.s. citizens or permanent residents.   if you would like to learn more about the institute, please visit the rti website or contact rti project director susan lessick, ahip, fmla, at @slessick@uci.edu.     evidence based library and information practice evidence based library and information practice 2010, 5.4 5 evidence based library and information practice acknowledgement of editorial advisors 2010. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. in 2010, the following people volunteered their time as editorial advisors, providing blind peer review of articles and evidence summaries that were submitted to evidence based library and information practice. the editorial team thanks the editorial advisors for their time, knowledge, and dedication which have contributed to the success of this publication. erin alcock, memorial university of newfoundland, canada greg bak, library and archives canada, canada anthony bernier, san jose state university, united states of america alissa black-dorward, fordham university school of law, united states of america kate boddy, peninsula college of medicine and dentistry universities of exeter and plymouth, united kingdom andrew booth, university of sheffield, united kingdom cara bradley, university of regina, canada marcy brown, silverchair science & communication, united states of america jeanette buckingham, university of alberta, canada sandy campbell, university of alberta, canada deborah charbonneau, wayne state university, united states of america amanda click, american university in cairo, egypt lisa cotter, newcastle university, australia sandy degroote, university of illinois at chicago, united states of america lani draper, stephen f. austin state university, united states of america jonathan eldredge, university of new mexico, united states of america juliet eve, university of brighton, united kingdom sue fahey, memorial university of newfoundland, canada alison farrell, memorial university of newfoundland, canada nancy fawley, virginia commonwealth university in qatar, united arab emirates bill fisher, san jose state university, united states of america lindsay glynn, memorial university of newfoundland, canada carol gordon, rutgers school of communication & information, united states of america genevieve gore, mcgill university, canada k. alix hayden, university of calgary, canada tony horava, university of ottawa, canada joanne jordan, keele university, united kingdom anthi katsirikou, university of piraeus, greece evidence based library and information practice 2010, 5.4 6 catherine king, national center for immunisation research and vaccine preventable diseases, australia laura kuo, hunter college, united states of america deborah lee, mississippi state university, united states of america suzanne lewis, gosford hospital library, australia michael lines, university of victoria, canada yazdan mansourian, tarbiat moallem university, iran sara marcus, queens college, cuny, united states of america marcia mardis, wayne state university, united states of america christine marton, university of toronto, canada paula mcmillen, university of nevada las vegas, united states of america ann medaille, university of nevada reno, united states of america dan mirau, concordia university college of alberta, canada obianuju mollel, alberta health services libraries, canada mac nason, algonquin college, canada cleo pappas, university of illinois at chicago, united states of america t. scott plutchak, university of alabama at birmingham, united states of america asim qayyum, charles sturt university, australia helen robertson, university of calgary, canada ann roselle, phoenix college, united states of america robert russell, northern state university, united states of america julie rustad, college of st. scholastica, united states of america pam ryan, university of alberta, canada alvin schrader, university of alberta, canada allison sivak, university of alberta, canada mark spasser, palmetto health, united states of america emily symonds, university of louisville, united states of america donna timm, louisiana state university, united states of america ingrid tonnison, northern sydney central coast health, australia alison yeoman, aberystwyth university, united kingdom li zhang, university of saskatchewan, canada / evidence based library and information practice anthi katsirikou, university of piraeus, greece evidence based library and information practice evidence based library and information practice 2010, 5.4 102 evidence based library and information practice evidence summary information rx program requires more promotion, more support and some adjustment a review of: meeks, k. (2009). information rx: promotion and utilization by georgia librarians and the georgia american college of physicians. journal of consumer health on the internet, 13, 129-134. reviewed by: matthew thomas collection librarian, northern ontario school of medicine thunder bay, on, canada email: matthew.james.thomas@gmail.com received: 31 aug. 2010 accepted: 16 oct. 2010 2010 thomas. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to determine the level of awareness of the information rx program by georgia librarians and georgia american college of physicians (gacp) members, and the use of information rx pads, with which physicians would “prescribe” information for their patients. design – descriptive (surveys and interviews). setting – georgia, u.s. surveys were distributed and responded to online. the faceto-face interview locations were not specified. subjects – one survey, which was provided to the georgia american college of physicians (gacp) membership including internal medicine physicians and medical students interested in internal medicine, had 46 respondents. the second survey was sent to librarians who were members of the georgia public library service (gpls) and the georgia health sciences library association (ghsla). there were 72 public librarians, 14 hospital librarians and 13 academic medical librarians who responded (as well as 6 not specified in the article). a select group of four medical librarians was chosen for more in-depth interviews. the number of surveys sent out was not provided. methods – two online surveys, one for physicians and one for librarians, were administered. no information concerning response rate was provided. face-to-face interviews with four academic medical librarians were conducted. no further information about the interviewing process was provided such as who conducted the mailto:matthew.james.thomas@gmail.com� evidence based library and information practice 2010, 5.4 103 interviews, methods used to ensure objectivity or consistency, or where the interviews were conducted. main results – out of 46 gacp survey respondents, only 4 were familiar with the information rx program and only 2 of those had used the information rx pads, neither of whom had referred anyone to a library for further assistance. the two who had not used the pads were either too busy or didn’t understand the program well enough. of 105 librarian survey respondents, 46 had heard of information rx, 37 had received information rx promotional materials, and 12 reported helping patrons look for information on medlineplus ‘‘prescribed’’ to them by their doctors. responses to the open-ended interview questions given to the four interviewed librarians were mixed regarding receipt of program materials, negative regarding the effectiveness of the program, and reported no awareness of any patrons having been helped with information “prescriptions.” to improve the program’s success, the author suggested steps such as providing promotional information on the medlineplus site, better integration between medlineplus and information rx, involving librarians somehow in the process as a whole given their ability to help users navigate and understand medlineplus, and marketed more to nurses given their front-line responsibilities. conclusion – although the program is somewhat useful, information rx has not been promoted or supported sufficiently. information needs to be linked on the medlineplus website, clarification of the program and that it is available at no charge should be emphasized. librarians should be involved and the target audience may need to be changed to include nurses. commentary the results of this study would be of obvious interest to those behind the information rx program and to any health librarians aware of the program, especially those in the georgia area. the results would be of general interest to any librarian supporting projects requiring physician participation in that it can be seen as a negative case study of such participation. the general conclusion that knowledge and understanding of the information rx program is not sufficiently widespread and that more promotion and education is necessary is rather clear but also not surprising. additional analysis and discussion should have been included to more firmly connect the results with the conclusion and recommendation. more detailed conclusions and specific recommendations could have been provided. the author should have provided additional information on the program itself and other tools involved. for example, the article mentioned that “physicians that participated in this project were provided information rx tools, such as ‘information rx pads’” but no other tools were described. were there other tools? if so, what were they? what did the program involve beyond the provision of the pads? a copy of the pads themselves was not included and no description given other than that they were to be used to “prescribe” information solutions to patients. according to an article referenced in this paper (siegal, logan, harnsberger et al. 2006), the pads are very simple and are apparently the central part of the program but this is not clear from the article. one other article (adams & de bont, 2007) appears to refer to this program (although not by name, despite the use of the phrase “information rx” in the title) but provides no more information than siegal and colleagues (2006) or the current one. the use of surveys and interviews were certainly an appropriate choice for type study given the stated objectives. no other work identified on the subject of information rx looked at the program from such a pragmatic vantage point and level of detail. unfortunately, copies of the two surveys conducted were not included. some survey questions and answers are discussed but it is not clear if what is mentioned is the entire survey content. on the other hand, the questions and responses from the four inevidence based library and information practice 2010, 5.4 104 depth interviews conducted were included in the article word for word. the focus on the responses of 4 interviewed subjects and not on the 151 survey respondents seems unbalanced. a few details which were missing in the description of the study would have enhanced the reader’s understanding. because the article does not identify how many individuals received the study, the rate of response is unknown. in the description of the gacp survey respondents, one was missing. the author states that there were “46 respondents to the gacp members’ survey” and that “41 were not familiar with information rx.” the author then discusses the “four respondents who were familiar with the program.” finally, there is no mention of how the four individuals in the “select group of medical librarians” were chosen. these are minor details in the study but including them would have been appropriate and important for the reader’s opinion of the worth of the study. references adams, s. & de bont, a. (2007). information rx: prescribing good consumerism and responsible citizenship. health care analysis, 15(4), 273-290. siegel, e. r., logan, r. a., harnsberger, r. l., craverdi, k., krause, j. a., lyon, b., hajarian, k., uhl, j., ruffin, a., & lindbergm, d. a. b. (2006). information rx: evaluation of a new informatics tool for physicians, patients, and libraries. information services and use, 26(1), 1–10. / evidence based library and information practice evidence summary evidence summary   survey respondents suggest that some academic library professionals without a graduate degree in librarianship have prior library experience and do not plan to pursue a library degree   a review of: oliver, a., & prosser, e. (2018). academic librarianship without the degree: examining the characteristics and motivations of academic library professionals. the journal of academic librarianship, 44(5), 613-619. https://doi.org/10.1016/j.acalib.2018.07.006   reviewed by: eugenia opuda health & human services librarian assistant professor dimond library university of new hampshire durham, new hampshire, united states of america email: eugenia.opuda@unh.edu   received: 5 aug. 2019                                                                   accepted:  4 feb. 2020      2020 opuda. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29617     abstract   objective – to examine the motivations and career paths of professionals outside the field of library science who work in academic library settings, including their reasons for not pursuing a graduate degree in librarianship.   design – multiple-choice survey questionnaire.   setting – not clearly stated.   subjects – 193 adults without graduate degrees in librarianship employed in professional positions in academic libraries.   methods – a nineteen-item multiple-choice questionnaire hosted on select survey and a previous study by the same authors. filtering excluded survey respondents who did not currently work in academic libraries, who had graduate degrees in librarianship, or who do not identify as an academic library professional.   main results – most of the survey respondents (n=115, 59.9%) had positions in a library prior to pursuing a professional academic library career. of those with prior library experience, most (n=98, 85.2%) had gained experience in academic library settings. the two top reasons cited for becoming an academic library professional were an interest in employment in academic library settings (n=59, 52.2%) and meeting  position requirements (n=54, 47.8%). a fifth of respondents both met the requirements for their position and had an interest in working in academic libraries (n=23, 20.4%). most respondents had less than five years’ experience (n=41, 36.6%) or six to ten years’ experience (n=43, 38.4%) in an academic library. less than half of respondents had became academic library professionals after applying as an external candidate (n=83, 44.6%) and a number of respondents had applied as an internal candidate (n=52, 28%). several respondents had become academic library professionals because they were promoted, appointed, or recruited within their academic libraries (n=35, 18.8%). few respondents were actively working on a graduate librarianship degree (n=21, 11.3%) and most expressed that they did not need such a degree (n=112, 67.9%). those who were pursuing a graduate degree in librarianship did so because of their desire to advance their careers (n=17, 81%). respondents’ current positions were mostly categorized in areas such as administration (n=77, 31.2%), scholarly communications (n=34, 13.8%), technical services (n=27, 10.9%), and information technology (n=20, 8.1%).   conclusion – having prior experience working in an academic library served as a notable motivating factor for entry into the position of academic library professional. two main pathways towards obtaining such positions included positions without graduate library degree requirements and the transition of paraprofessionals into professional-level jobs. most survey respondents noted their lack of interest in pursuing an advanced degree in librarianship, as they did not see the significance of having one. these findings may help library education programs to better understand growing needs in librarian education and prepare the future library workforce to meet these new demands.   commentary   previous literature (gremmels, 2013; simpson, 2013; zhu, 2012) about library professionals without graduate library degrees has tended to highlight non-mls or paraprofessional job responsibilities and staffing trends among a wide range of libraries. this research article builds upon a prior study by the authors (oliver & prosser, 2017) and remedies a gap in the literature by examining the motivations of individuals pursuing specific positions within academic libraries that do not require a library graduate degree. the authors, however, do not provide a clear purpose or motivation for conducting such a study, which raises concerns regarding the study’s significance.   this study was appraised using the “ebl critical appraisal checklist” (glynn, 2006). while the methodology was appropriate for the research question and target population, the authors did not detail how they recruited and selected study participants were recruited or whether informed consent was obtained. other methodological details missing from the study included the exact geographic locations of study participants, data collection methods, and the precise time frame during which the surveys were conducted. given the number of omitted details, replication of this study would be difficult. due to the number of survey respondents (193 of 233), the authors state that they were able to obtain a fairly reliable set of responses with a 7.2% margin of error with a 95% confidence interval. however, the authors cite a 2016 acrl population statistic that there are at least 6,900 academic library professionals in acrl libraries, which raises concerns that this study is not representative of a larger population and that the results are overstated. it was unclear whether the content of the instrument was validated by an expert panel or pilot tested to ensure the survey would truly assess what the authors wanted to measure, which may have some potential to impact the accuracy of the survey’s responses. additionally, survey respondents were able to select multiple answers for some questions, which caused some of the response totals to exceed 100%.   the results of this study may be inflated and may not represent a large portion of the demographic. however, there may be implications for individuals seeking careers in academic libraries, potential graduate students choosing between library programs, library hiring committees, professional associations, and the educational standards and curricula of accredited library programs. as new trends in academic libraries develop and emerge, skill sets and knowledge not addressed in traditional library education programs may be leading to new roles that do not require advanced degrees. this raises important questions regarding how graduate library programs might need to adapt in order to prepare students for future interdisciplinary roles and to justify the financial investment students make in pursuing advanced professional degrees. though responses to the survey indicate that there may be an increasing number of new professional positions within academic libraries that do not require a graduate degree in librarianship, the small response rate indicates that there is a need for more rigorous research in this area.   references   association of college & research libraries (2016). summary data tables – acrl 2016. retrieved from: https://www.acrlmetrics.com   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. https://doi.org/10.1108/07378830610692154   gremmels, g. s. (2013). staffing trends in college and university libraries. reference services review, 41(2), 233–252. https://doi.org/10.1108/00907321311326165   oliver, a., & prosser, e. (2017). choosing academic librarianship: an examination of characteristics and selection criteria. the journal of academic librarianship, 43(6), 526–531. https://doi.org/10.1016/j.acalib.2017.08.009   simpson, b. (2013). hiring non-mls librarians: trends and training implications. library leadership & management, 28(1). retrieved from https://journals.tdl.org/llm/index.php/llm/article/view/7019   zhu, l. (2012). the role of paraprofessionals in technical services in academic libraries. library resources & technical services, 56(3), 127–154. https://doi.org/10.5860/lrts.56n3.127     evidence based library and information practice evidence based library and information practice 2009, 4:3  evidence based library and information practice       ebl 101     looking to the literature: open access and free sources of lis evidence    virginia wilson   shirp coordinator, health sciences library,  university of saskatchewan   saskatoon, saskatchewan, canada   email: virginia.wilson@usask.ca    received: 06 august 2009        accepted: 10 august 2009      © 2009 wilson. this is an open access article distributed under the terms of the creative commons attribution  license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and  reproduction in any medium, provided the original work is properly cited.    in the last ebl 101 column, the discussion  centered upon assigning your question to one  or more domains in order to determine where  to look for research evidence. a rule of thumb  is to check out the lis resources first, and then  move to other bodies of evidence that may  inform your question, such as the business,  education, marketing, or management  literature, to name a few. one of the barriers to  evidence based library and information  practice is a lack of access to the research  evidence. some library acquisition budgets do  not include a line for professional  publications, or if they do, the funds allocated  to such collections can be relatively small.     in light of this barrier, i have compiled a list of  open access and free sources of library and  information science resources. according to  peter suber, open access literature is “digital,  online, free of charge, and free of most  copyright and licensing restrictions” (qtd. in  morrison: 46). anyone can access and utilize  open access resources. stephanie hall and i  compiled the following list for the evidence  based librarianship toolkit for public  libraries <http://ebltoolkit.pbworks.com/>,  and have updated it for this column.    databases    library, information science & technology  abstracts (lista)  <http://www.libraryresearch.com>  from ebsco publishing, this free bibliographic  database provides coverage on subjects such  as librarianship, classification, cataloging,  bibliometrics, online information retrieval,  information management and more.    journals    directory of open access journals (doaj)  <http://www.doaj.org/>  under the social sciences heading, there are 96  open access library and information science  journals in various languages.    evidence based library and information practice  <http://ejournals.library.ualberta.ca/index.php/ eblip>  an open access journal containing research,  evidence summaries, and commentary on  75 mailto:virginia.wilson@usask.ca http://ebltoolkit.pbworks.com/ http://www.libraryresearch.com/ http://www.doaj.org/ http://ejournals.library.ualberta.ca/index.php/eblip http://ejournals.library.ualberta.ca/index.php/eblip evidence based library and information practice 2009, 4:3  evidence based library and information  practice. published quarterly. canadian.    partnership: the canadian journal of library and  information practice and research  <http://journal.lib.uoguelph.ca//index.php/perj >  this open access practitioners’ journal  contains lis research, conference  presentations, profiles, reviews, commentary,  and more. published twice yearly. canadian.    libres: library and information science research  electronic journal <http://libres.curtin.edu.au//.  an open access international refereed  electronic journal devoted to new research in  library and information science. published  twice yearly. australian.    ariadne <http://www.ariadne.ac.uk/>  this is a web magazine for information  professionals in archives, libraries and  museums. published every three months,  beginning in january. uk.    e‐jasl: the electronic journal of academic and  special librarianship  <http://southernlibrarianship.icaap.org/>   an independent, professional, refereed  electronic journal dedicated first and foremost  to advancing knowledge and research in the  areas of academic and special librarianship  (from website). open access. canadian.    information research  <http://informationr.net/ir/>  this open access, international, peer‐reviewed,  scholarly journal is dedicated to making  accessible the results of research across a wide  range of information‐related disciplines (from  website).    archives    e‐lis: e‐prints in library and information  science <http://eprints.rclis.org/>  an open access archive for scientific or  technical documents, published or  unpublished, on librarianship, information  science and technology, and related areas  (from website). there are more than 9,400  items in the archive.    dlist: digital library of information science  and technology <http://dlist.sir.arizona.edu/>  an open access archive for the information  sciences. the archive has a global advisory  board.    oclc research publications repository  <http://www.oclc.org/research/publications/se arch.htm>  this repository contains works produced,  sponsored, or submitted by oclc research.  in general, the works are research‐oriented  and are in the subject area of library and  information science. many items describe  oclc research projects, activities, and  programs and were originally published by  oclc, while others are from peer‐reviewed  scholarly journals. it currently contains links  to the full text of 303 items (from website).    opendoar: the directory of open access  repositories <http://www.opendoar.org/>  an authoritative directory of academic open  access repositories, each opendoar  repository has been visited by project staff to  check the information that is recorded there.  this in‐depth approach does not rely on  automated analysis and gives a quality‐ controlled list of repositories (from website).    if you lack access to a range of lis resources,  these should get you started. make sure to  check with professional organizations to see if  there are resources available to you as part of  the membership fee. if you have an open  access or free lis resource to add to the list,  please contact me: virginia.wilson@usask.ca. i  will make any additions available in future  ebl 101 columns.    but what happens if you cannot find any  published research evidence to aid in your  decision‐making process? next time,  conducting your own research: something  to consider.      76 http://journal.lib.uoguelph.ca/index.php/perj http://libres.curtin.edu.au/ http://www.ariadne.ac.uk/ http://southernlibrarianship.icaap.org/ http://informationr.net/ir/ http://eprints.rclis.org/ http://dlist.sir.arizona.edu/ http://www.oclc.org/research/publications/search.htm http://www.oclc.org/research/publications/search.htm http://www.opendoar.org/ mailto:virginia.wilson@usask.ca evidence based library and information practice 2009, 4:3  works cited  morrison, heather.  “evidence based  librarianship and open access.”  evidence based library and  information practice 1.2 (2006): 46‐ 50. 6 aug. 2009  <http://ejournals.library.ualberta.ca/ index.php/eblip/article/view/49/11 7>.  77 http://ejournals.library.ualberta.ca/index.php/eblip/article/view/49/117 http://ejournals.library.ualberta.ca/index.php/eblip/article/view/49/117 http://ejournals.library.ualberta.ca/index.php/eblip/article/view/49/117 evidence based library and information practice evidence based library and information practice 2010, 5.4 21 evidence based library and information practice article using acrl standards to assess the information literacy of graduate students in an education program amy jo catalano assistant professor of library services hofstra university new york, united states email: amy.catalano@hofstra.edu received: 14 july 2010 accepted: 23 oct. 2010 2010 catalano. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective this study investigates the information literacy of graduate education students, including those in doctoral cohorts. the association for research and college libraries information literacy standards were used a baseline for measurement. methods a survey was sent to all graduate students in the school of education; it asked a combination of questions measuring students’ perceptions of their information literacy skills and testing their knowledge of information literacy. results – a total of 172 surveys were returned. the results indicated that while there is a heavy reliance on internet sources, many students were able to determine which sources were reliable and which were not. after attending information instruction sessions, students were more familiar with library services and more inclined to use them. conclusion it was determined that a one credit course or multiple sessions of library instruction would better serve graduate students completing capstone projects. introduction with the increased emphasis within academia on student-centered learning and student learning outcomes assessment by disciplinebased and regional accrediting bodies, many accrediting agencies are also beginning to stress the importance of standards-based information literacy skills for students in higher education. this article focuses on the mailto:amy.catalano@hofstra.edu� evidence based library and information practice 2010, 5.4 22 assessment of standards-based information literacy skills for graduate students at hofstra university, a large private institution located on long island, new york, usa. specifically, survey methodology was utilized to determine a baseline for graduate students’ information literacy skills, compared with the association of college and research libraries (acrl) standards for information literacy (il) competency for higher education. the middle states commission on higher education (msche), along with other key accrediting agencies, are proponents of the integration of information literacy as part of the standards of accreditation (thompson, 2002). towards that end the msche has adopted the association of college and research libraries (acrl) information literacy competency standards, which were developed to measure a learner’s ability to "recognize when information is needed and have the ability to locate, evaluate, and use effectively the needed information" (thompson, 2002, p. 222). in this study, the focus is on education graduate students. these students often complete capstone projects such as dissertations or master’s theses. these future teachers must be information literate themselves if they are to teach the same competencies to their students. in addition, professors in education expect their students to use research-based practices in their teaching as well as research, therefore it is essential that students know how to find and evaluate relevant sources. it is of concern to academic librarians that students who are engaged in the research process are often unaware of library resources, find them difficult to use, or use them ineffectively. since the acrl standards for il provide us with a framework for assessing the competencies necessary to become an information literate learner both in and out of a student’s academic pursuits, it then follows that we would use these standards to guide us toward the main research question of this study: “using the acrl standards for il as an assessment tool, to what extent are graduate students in education information literate?” literature review a review of studies measuring the information literacy, research, or library skills of graduate students revealed that few of the studies use standards as the basis for assessment and even fewer use the acrl il standards. three notable exceptions are studies conducted by ferguson, neely, and sullivan (2006) at the university of maryland, emde and emmett (2007) at the university of kansas, and berg and grant (2003) at san diego state university. while there is abundant literature on the information literacy skills of pre and inservice teachers, this literature review focuses on skills of graduate education students rather than those of undergraduate education majors, in line with the aims of the study. oakleaf and kaske (2009) offered guidance on assessing il in higher education for academic librarians. they focused on the variety of measures available and how to choose the best one for whichever goals an institution wants to achieve. while they did mention briefly the “information literacy competency standards for higher education,” the authors did not discuss the incorporation of the acrl il standards into assessments. a study that reported a positive relationship between graduate education students and information literacy was conducted by berg and grant (2003). the research discussed incorporating information literacy into a doctoral program using the acrl standards for il, and evaluating the contents of existing courses to identify where the five standards were covered and to what degree of adequacy. their findings showed that doctoral students in all courses engaged in activities corresponding to each standard. similarly, green (2006) discussed a librarian-created education doctoral level course that incorporates information literacy skills into its content. the course is constructed in such a way that students are expected to demonstrate that they are information literate through the searching, identifying, and evaluating of information collected for literature reviews. as students progress in the program they build evidence based library and information practice 2010, 5.4 23 on those information literacy skills as they learn to use them in more specialized ways. emde and emmett’s (2007) research focused on an assessment tool design based on desired learning outcomes using the acrl standards as a framework. when designing their assessment tool, they asked “does the question asked provide a valid measure of what the instructor wants to know about the students’ skills?” (2007, p.212). the authors discerned that the strength of studies such as theirs lies in the formation of questions created to measure real problems that would be encountered in a graduate student’s research. thus questions used in their assessment tool, which was implemented during a one credit library instruction course for graduate students in chemistry, used the acrl standards to create activities or questions that would allow the student to demonstrate the desired learning outcome. unfortunately, the authors did not enumerate their findings as they corresponded to the standards. they generalized their findings by whether students passed or failed pre and post tests. perrett (2004) surveyed graduate students’ information literacy skills using a similar methodology to the study described below. the students first evaluated their own skills and then performed specific tasks in order to compare the two. perrett found that 47% of the students accurately evaluated their skill level. more specifically, many students overestimated their internet searching skills. based on students’ performance they were provided with recommendations for further training to improve their skills; 64% of those students were advised to enroll in a bibliographic instruction course. earp (2008) studied the information source preferences of education graduate students and concluded that graduate students prefer information that is easily accessible even if it may be unreliable; they prefer electronic access; and they are unaware of many library resources and services such as interlibrary loan. these characteristics of student information seekers, whether graduate or undergraduate, are often shared throughout the disciplines. unsurprisingly, doctoral students tended to be more diligent in their information seeking. zaporozhetz (1987) reported that doctoral candidates stated that their faculty advisors expected them to possess advanced bibliographic skills that the advisors themselves did not have. an extension of the issue is illustrated by both zaporozhetz and labaree’s (2003) argument that because doctoral students in the field of education are accomplished professionals they may have difficulty admitting that they lack library search skills. therefore, despite the acknowledged need by doctoral students of possessing advanced library research skills, it is up to the institution and its faculty to ensure that students get this training. morner (1993), testing the library research skills of doctoral students of science education, concluded that these students were not well equipped for doctoral-level research. similarly in alire’s (1984) study of education doctoral students’ attitudes regarding the importance of library usage and the need for bibliographic instruction (bi), she found that over two-thirds of the students thought that knowledge of the library and its resources was important to their academic success. more than half of the students thought that they had deficiencies in library skills and felt that they could benefit from bi. finally, the majority of the students surveyed stated that a course in library research methodology should be required in programs where dissertations were mandatory and that they would take such a course. methods the 24 question survey (appendix a) was modeled after the one created at the university of maryland, baltimore county (ferguson, neely & sullivan, 2006) and based on the acrl information literacy competency standards for higher education. many of the questions were adapted from the evidence based library and information practice 2010, 5.4 24 neely test of relevance, evaluation, and information literacy attitudes. the first part of the survey posed questions about progress within a program as well as questions relating to the respondents’ occupation and experience using library sources. one question allowed for comments about major sources of frustration during students’ course of study. the second half of the survey presented a combination of questions designed to measure information literacy skills and the students’ perceptions of their information literacy. the author found self-reporting methodology the best way to illustrate the gap between real and perceived information literacy. this is important because, as described above, graduate students in education are often experienced and exceptionally capable performers in their field and reluctant to admit a deficiency in knowledge. the purpose of this study was to inform decision makers in schools of education of common phenomena among education graduate students. the survey was sent to all graduate students enrolled in the school of education and health and human services (soehhs) as well as graduate students in the speech-language and hearing program and the clinical psychology program, a total of 1,770 students. the latter two departments were selected because they both have a required 6 hour workshop on the use of library resources whereas soehhs does not require their doctoral students to complete such a course. comparisons of the abilities of those who do participate in such a workshop will be made in the results and discussion sections of this paper. the survey was emailed twice within a two week time period in the spring of 2008. in between each mailing, the directors of doctoral and master’s programs were informed of the study and asked to encourage their students to participate. a total of 172 surveys were returned (9.7% response rate). of the 172 respondents, 34 were doctoral students while the remainders were graduate students working on either their first or second master’s degree or graduate certificate. findings and discussion overview the level of academic and professional experience of the respondents was noted: 41% of the respondents had completed 5 courses or less while 100% of those in the education doctoral program were either school administrators, teachers or were professionals in the field of education. during the course of this study and based on the literature review, it was discovered that ed.d or ph.d education students are often very different from other types of ph.d students in that they often have well-established, full-time, careers as professionals and go to school part-time or during a summer cohort program. this may indicate different skill levels among these students as compared to graduate students in other disciplines (beile & boote, 2005). in response to the question, “what is your biggest challenge when looking for information?” 49% stated that writing in and following apa style is their biggest challenge while 45% of respondents said it was the inability to find relevant literature for their research; 30% expressed that they can’t effectively use the online databases, while 27% stated identifying an original research topic is a challenge. use of library services and experiences the majority (80%) of respondents reported they accessed library resources remotely, and 50% of them accessed these resources at least once a week. when using library resources, whether in person or remotely, 83% accessed online databases while only 53% borrowed books or other materials. interlibrary loans were used by 29%, but less than 1% used the research libraries program (rlp). the rlp allows students to borrow materials from neighboring academic libraries. the document delivery service was used by 9% of respondents and 36% use the library’s print evidence based library and information practice 2010, 5.4 25 journal collection. librarians came to classrooms for library instruction for 44% of respondents and 87% of those that have had instruction claimed that such instruction helps them to better navigate the library’s resources. while this study reports a relatively low use of interlibrary loan, as doctoral students and faculty learn about the availability and purpose of interlibrary loan, they tend to rely on it to complete their literature reviews. based on past experiences, some reasons why there is low use of interlibrary loan at the master’s level may include the lack of knowledge of its existence or how to access and use it. many students are also surprised at how easy it is to place an interlibrary loan request. other reasons for low use include the fact that libraries have more full-text available and the increasing availability of open access journals. acrl standards for information literacy competencies the following section reports the results of the questions pertaining to each of the five acrl standards for il competencies. no numerical passing score was attributed to this survey, as many of the questions were attitudinal. a discussion of the results follows each standard. standard 1: “the information literate student determines the nature and extent of the information need.” the survey included 4 questions to measure this standard; the first two related to students comfort levels with information sources. the majority (92%) of students felt equally comfortable finding information from a search engine and their professor, while many (77%) felt comfortable with the library webpage or the library itself (73%). in summary 85% reported that they were either very comfortable or comfortable identifying potential sources of information. generally this response was confirmed by examining the types of sources students reported to rely on, such as their professors. while this finding doesn’t directly address the performance indicator for standard 1: “confers with instructors and participates in class discussions, peer workgroups, and electronic discussions to identify a research topic, or other information need” (acrl information literacy standard 1.1), it could be inferred that a professor of a course would be consulted by a student when attempting to satisfy an information need presented by that course. however, while reporting on graduate education students, earp (2008) found that the majority of students surveyed rarely relied on professors, classmates or librarians. the third question asked “given the topic ‘students with adhd in the mainstream classroom’ as the subject of a research project, which of the following steps would you do first to begin your project?” almost half (44%) of students reported that they would begin their search with the internet using the terms “adhd” and “mainstream classroom,” whereas 22% said that they would search subject specific databases. only 14% would formulate questions based on the information need. none of the students would use a reference source on special education. the results of question three strongly indicate the move from traditional reference sources to the internet. research findings from martin (2008) concurs with this finding. while students are avoiding reference books, replacing them with internet searches in order to increase knowledge of topic would support the performance indicator “exploring general information sources to increase familiarity with a topic” (acrl information literacy standard 1.1 ). conversely, ferguson, neely and sullivan (2006), reported that students consulted a reference source at some stage in their research but rarely in the beginning. evidence based library and information practice 2010, 5.4 26 table 1 standard 2 n= 161 standard 2 q. 1.how frequently do you use the following search strategies always often sometimes rarely never don't know what that is truncation >1% 3 10 8 6 72 boolean operator: not 2 6 9 22 21 40 boolean operator: or 4 11.5 18.5 15 12 39 proximity operators 2.3 2.3 6.9 8.4 9.1 80 library of congress subject headings 3 6.9 10.7 14.5 31.3 33.6 eric descriptors 13 20.6 25.3 7.6 13.7 19.8 cross and multiple-field searching (such as by date) 7 26 28 10 14 15 these responses suggest that students rely less on the library, possibly indicating negative experiences with using library sources, such as difficulty navigating complex databases. this result conflicts with the 87% of students who say they perform better after librarian instruction. standard 2: “the information literate student accesses needed information effectively and efficiently.” the survey included 4 questions used to measure this standard. the first question asked “how frequently do you use the following search strategies: boolean operators, truncation, eric descriptors and subject headings?” a performance indicator demonstrating competency of this standard includes using effective search strategies. for this aspect of the standard of information literacy, many students were deficient. for all search strategies listed on the survey, at least 20% had no knowledge of the term (see table 1). another question asked, “during which events do you ask a librarian for assistance?” most respondents knew when to consult a librarian (ie. advice for where to look for information, or how to use a resource), while 10% did not know when a librarian’s expertise would be helpful. the third question asked, “where would you go or what would you do to find current information on the following topic: ‘terrorism on college campuses’?” this question allowed for multiple answers. nearly 20% said they would go to the internet for the needed information; 14% said they would go to newspaper archives even though the word “current” was emphasized in italics and 12% said they would go to databases. the answers to question one appeared to illustrate that students were deficient in this area. even though 71% claimed to be comfortable or very comfortable with developing successful search strategies, most students did not demonstrate knowledge of a variety of search tools such as the library of congress subject headings, truncation, or eric descriptors and thesaurus. similarly, reporting on undergraduate students, ferguson et al. (2006) indicated that 66% of students surveyed had never used proximity operators or truncation and nearly 74% rarely or infrequently used library of congress subject headings, eric descriptors or other controlled vocabulary. perrett (2004) reported in her audit of graduate students that less than half of were able to use boolean operators and only 66% used truncation searching. the lack of knowledge on how to use search tools and strategies may cause in a graduate student to miss out on relevant bodies of work for their research problem. boote and beile (2005) evidence based library and information practice 2010, 5.4 27 discussed this issue in their article about literature reviews in the education doctoral dissertation. it is essential to the quality of the dissertation that the literature review be complete. often, without thorough bibliographic training, one cannot do a complete search of all relevant literature. this includes finding literature, retrieving it, and evaluating its value and relevance to the student’s work. one weaknesses of the current study is the formation of questions about search strategies. many search engines, including google and some subscription databases, already use boolean logic. while the student may not have heard of the term “boolean logic” they may already be implementing it or having it done for them automatically by the database while constructing a search query. this may also apply to truncation, date limiters, and relevance rankings. saunders (2008) addressed the issue of information retrieval systems and il standards by concluding that increasingly more and more information retrieval systems (e.g. subscription databases, search engines, or online library catalogs) “include design enhancements meant to improve the precision and recall of the user’s search results by expanding or refining the original query with related terms… [in order to] expand a narrow search” (p. 92). similarly, other systems provide relevance rankings, relevance feedback, date ranking, as well as thesaurusenhanced searching. these design enhancements have implications for il standards as they can enhance the user’s “abilities” to search effectively for the needed information. further, these enhancements allow instruction librarians to focus on the other aspects of information literacy that comes after information retrieval (such as evaluating research, identifying bias etc.). those in the education program, as well as those who major in school psychology, tend to rely heavily on the eric database. knowledge of eric descriptors and limiters or use of the eric thesaurus can be helpful search tools. ignorance of possible search strategies may cause undue frustration with the research process. standard 3: “the information literate student evaluates information and its sources critically”. the survey included 3 questions to assess this standard. the first asked “can one evaluate an article for bias before reading it?” the majority (76%) of respondents provided answers that demonstrated knowledge of indicators of bias whereas 24% did not know the answer to the question. the next question asked, “would you find any of the following resources credible or reliable for your research?” choices were given that ranged from new york times and cnn to saturday night live’s weekend update and the national enquirer. the large majority of the respondents appeared aware of which sources listed in this question were satirical sources (e.g. saturday night live weekend update, the onion, and jon stewart’s daily show) and which are generally reliable sources (e.g. new york times and washington post). eight percent thought that these newspapers were never credible. the results of this question were surprising as there appeared to be no consistent or predictable answers. for example, 11% of respondents found that the national enquirer was at least sometimes credible and 24% said world news tonight was never credible.(see table 2 for full results). the majority (71%) claim to be comfortable or very comfortable with evaluating information, claims which are supported by the results. however, ferguson et al. (2006) reported different results for this same question. for example 58% of students, in the ferguson et al. study, stated that time or newsweek was always reliable and that 63% said that the ny times was reliable. twelve percent of the ferguson study found the saturday night live weekend update credible. this discrepancy may indicate a shift in the perception of credibility from undergraduate to graduate students. evidence based library and information practice 2010, 5.4 28 table 2 standard 3 n= 157 standard 3 q. 2 when would you find any of the following sources reliable or credible ? always sometimes never national geographic 36.22% 48.82% 14.96% new york times or washington post 25.98% 66.14% 7.87% cnn 24.41% 61.42% 14.17% time or newsweek 22.83% 65.35% 11.81% wikipedia 15.75% 40.16% 44.09% world news tonight/cbs evening news 14.96% 61.42% 23.62% the daily news 3.94% 48.82% 47.24% people 3.15% 31.50% 65.35% jon stewart's the daily show 3.15% 20.47% 76.38% sports illustrated 2.36% 41.73% 55.91% today show/good morning america 2.36% 54.33% 43.31% the onion (website) 0.00% 18.90% 81.10% rolling stone 0.00% 41.73% 58.27% national enquirer 0.00% 11.02% 88.98% saturday night live's weekend update 0.00% 11.02% 88.98% the last question that measured this standard asked the respondent to indicate which of seven statements were true when it came to information found on the internet. the majority (90%) said material on the web comes from many varied sources such as business, the government, organizations, or the public, 3% thought the web is more reliable than books and magazines, 2% thought material on the web is factual because the web is monitored by international organizations. standard 4: ”the information literate student uses information effectively to accomplish a specific purpose”. this standard addresses the ability of the student to organize information and communicate the information for a specific purpose. the survey included two attitudinal type questions corresponding to this standard. this is a difficult standard to assess without demonstration of the competency such as a research paper. an attempt to make an assessment was made by asking students to indicate their comfort levels with specific skills: integrating new information into an existing body of knowledge and organizing information for practical application. the majority (74%) claimed to be very comfortable or comfortable with integrating new information into an existing body of knowledge whereas, 12% were neutral, and 4% percent were uncomfortable or very uncomfortable. similarly, 77% were very comfortable or comfortable with organizing information for a practical application, whereas 20.5% were neutral, and 2.5% were uncomfortable or very uncomfortable. it is recommended that a method be devised for students to model these skills for future research. standard 5: “the information literate student… uses information ethically and legally” this standard addresses knowledge of freedom of speech and the use of copyrighted information and was assessed with two questions. the first asked, “what is the best definition of ‘intellectual freedom’?” a range of evidence based library and information practice 2010, 5.4 29 responses were given and multiple answers could be selected. “it is the right of every individual to both seek and receive information from all points of view without restriction;” was selected by 68% of respondents, 28% defined intellectual freedom as the encouragement of open and public sharing of ideas; and 3% believed it is to support the bill of rights. however, a small minority (3%) believes it means to prevent cheating by students or to limit access to ideas that some people find objectionable and dangerous. when asked to define intellectual freedom, 94% were able to answer correctly. the second question relating to this standard sought to measure a respondent’s knowledge about plagiarism and the web. when asked if it were true that all information taken from the web must be cited properly only 45.5% answered yes, while 13.5% answered that one may use all text and graphics freely unless they were copyrighted. similarly, ferguson et al.’s (2006) respondents also were unsure of when they could use items freely from the internet; 22% either did not know or responded incorrectly about using information from a website. the results of this survey indicate a lack of awareness about several necessary aspects of bibliographic knowledge. on the one hand, the majority of respondents demonstrated knowledge about the reliability of web and other sources, yet there is a heavy reliance on internet sources. perrett (2004), martin (2008), and earp (2008) all confirmed this finding. students often lack an understanding of the many sources available to them (such as print indexes and subscription databases) and how to use them effectively. given many students rely on the web rather than library sources, it must be questioned whether students would be more inclined to use library sources more frequently, if they knew their value. martin (2008) found that even with library instruction undergraduate education students from his study still relied on the web and avoided the databases. one reason cited is the complexity of databases. earp (2008) also pointed out that since students often rely on full-text of articles readily available, faculty and librarians need to stress “sometimes the best information is still located in print” (p. 84). as previously noted, the study conducted by the university of maryland (2006) concluded that while students often go to the web first, they perceive they are comfortable using sophisticated research methods and search strategies when they in fact demonstrate a lack of familiarity. this current study supports this conclusion. comparison between students receiving library instruction the survey was also administered to students in the speech, language and hearing program and the psychology doctoral program. the results are intended to be used as a basis for comparison as the students in those programs are required to attend mandatory library instruction sessions. the instruction consisted of two 3 hour sessions that included an orientation to the physical library, the library website, library services, finding books both in the library’s online catalog, as well as catalogs of other libraries (such as worldcat), navigating the research databases that are specific to psychology and speech, and how to use apa citation style. the sessions were taught by a psychology and speech subject specialist and allowed for hands-on searching both in the physical library and online. the subject specialist librarian also demonstrated the use of websites specific to the professions of psychology and speech. preand post-tests were given during the sessions in order to measure the effectiveness of the instruction. while the sample is too small to be statistically significant the data is worth noting. fifteen surveys were returned from students in these two programs. even though these students are required to attend library instruction, 2 of the 15 respondents did not. some of the most notable differences between the entire pool of respondents and those in the group mandated to take library instruction include: 73% use interlibrary loan as opposed to 29% of the whole group, and nearly 94% knew during which events to go to a librarian while only 89% of the whole group knew. table 3 evidence based library and information practice 2010, 5.4 30 shows that those receiving library instruction, on the whole, were more familiar with terminology surrounding search strategies, with the exception of eric limiters (see table 3). surprisingly, 18% answered the question incorrectly as to whether information needs to be cited when it is taken from the internet as opposed to 13.5% of the whole group getting that question incorrect. generally these results suggest those who attended the mandated library instruction session perceived themselves to be more confident about using different sources or identifying different potential uses of information. they also tend to be more knowledgeable about the services the library provides such as reference services, interlibrary loan, and databases that are appropriate for their field of research. it is also more likely that they are taught about specific search strategies using these databases. these students tend to rely less on internet search engines and websites. the entire group demonstrated weaknesses in their understanding of the reliability of internet resources, identifying bias, and reliance on the web for background information or news. these results suggest at least one library instruction session should be offered to all graduate students who must complete capstone projects. it would also be prudent to offer such information sessions to teachers-intraining, if means of assessing these competencies is not already in place in the teacher education program. this is particularly pertinent as information literacy competency is a requirement of both the teacher education accreditation council (teac) and the national council for accreditation of teacher education (ncate) (birch et al. 2008). four of the six ncate standards are parallel to the acrl standards. as the results of this survey indicate, students often think they are more capable than they are able to demonstrate. many librarians, teachers, and professors experienced with working with students know that they often go to the internet first for information, and they aren’t quite familiar with sophisticated search terms, strategies, and library jargon. students often don’t know the full range of library services and some don’t even know what a librarian does. for example, many students confuse e-books or e-libraries for article databases, thus often resulting in fruitless, inefficient, or frustrating searches, further resulting in gaps in their literature reviews. the results of other studies on the subject of information literacy of graduate students such as those completed at the university of maryland (2006), zaporozhetz (1987), boote and beile (2005), alire (1984), and martin (2008). each study corroborates these results and confirm that regular assignment based bibliographic instruction sessions better facilitates success in graduate studies. table 3 comparisons of the entire group of student surveyed vs. speech/psych students n=15 comparison of groups of students whole group speech/psych students don't know what it is don't know what it is truncation 72% 61% boolean not 40% 20% boolean or 39% 23% library of congress subject headings 80% 55% proximity indicators 33.6% 36% eric limitors 19.8% 38% cross and multiple-field searching 15% 8% evidence based library and information practice 2010, 5.4 31 an orientation program for graduate/doctoral students, that incorporates a library resources workshop is another suggestion that has been implemented at several institutions. the experiences of northwestern university in implementing such a program have been documented by lightman and reingold (2005). limitations of this study throughout the course of this research, several limitations and weaknesses of the survey were identified. the return rate of surveys was only 9.7% whereas an optimal rate would have been at least 22%. further, different results may have been determined had the differences between an online or internet source and database been more clearly defined. it is very possible when students said that they are consulting online sources they are referring to databases. while this study used the terms “search engines” and “websites” to refer to non-library, non-subscription databases, it is impossible to know what students actually understood those terms to mean. an additional limitation is that the use of boolean operators is somewhat outdated as some search products don’t use them anymore. in the section comparing the psychology/speech language group who received library instruction to the entire sample it was impossible, given the survey tool used, to remove individual respondents and triangulate the data. thus the responses of the psychology/speech language group are included in the whole group responses, distorting the comparison of the results. however, the results were worth including as they did indicate a difference in the quality of information learned specifically around database specific search strategies. lastly, while this study strove to measure students’ perception of their abilities as compared to what they could actually demonstrate, this goal would have been better measured with more questions that asked students to apply the il skills applicable to the standards. the study does not examine information use, but the mechanics of information processing (searching, finding, and evaluating). data on more complex information processes, such as organization and communication rely on respondents selfreporting, which is not the same as establishing that they can perform these tasks. therefore, opportunities to demonstrate information processing would have been more illustrative than self-reporting perceived abilities examples of questions, such as those designed by emmette and emde (2007) were based on acrl guidelines as well as being specific to their subjects’ major, chemistry, are a good model for a future survey. conclusion and implications for future research this article demonstrates the need to inform library instruction practice based on evidence based research. while professors and librarians may assume that graduate students have limited abilities when their research skills are measured against il standards, the research has shown that graduate students are far better at evaluating information sources than they may be given credit for. in contrast, they do need advanced bibliographic search skills and strategies that are often not addressed in traditional “one-shot” library instruction sessions. furthermore, multiple sessions maybe necessary throughout a student’s graduate career, as information needs develop and change. further suggestions for future research include: define the learning outcomes for the graduate programs as they relate to information literacy and construct more quantifiable ways to assess the acquisition of il and research skills. also, individual interviews with students would allow them to exhibit their knowledge of database use and assessments that ask students to demonstrate their ability to construct searches, find full text of articles, use interlibrary loan, and find statistical sources that are appropriate to their research. it is considered best practice to tailor evidence based library and information practice 2010, 5.4 32 questions meant to assess information literacy standards to the student’s subject major. further, employing inferential as opposed to descriptive statistics (as used in this study) would provide an evaluative basis on which to evaluate library instruction specifically created for doctoral students. future research also needs to be conducted on the differences on information literacy and search abilities of doctoral students completing their dissertations in comparison to students studying at other graduate levels, as the level of research needed to complete a dissertation is far more complex and ongoing than that of a master’s student. isolation of the abilities of doctoral students would allow for tailored evidence based means of instructing students at this level. references alire, c. a. (1984). a nationwide survey of education doctoral students’ attitudes regarding the importance of the library and the need for bibliographic instruction (doctoral dissertation, university of northern colorado). retrieved from dissertation abstracts. association of college and research libraries (acrl). (2000). information literacy competency standards for higher education. berg, m., & grant, m. (2003). information literacy integration in a doctoral program. in d. m. shinew & s. walters (eds.), information literacy instruction for educators (115-128). new york: haworth information press. beile, p., & boote, d. n. (2005). scholars before researchers: on the centrality of the dissertation literature review in research preparation. educational researcher, 34(6), 3-15. birch, t., greenfield, l., janke, k., schaeffer, d., & woods, a. (2008). partnering with librarians to meet ncate standards in teacher education. education, 128(3), 369-379. brown, c. m. (1999). information literacy of physical science graduate students in the information age. college & research libraries, 60(5), 426-38. earp, v. (2008). information source preferences of education graduate students. behavioral & social sciences librarian, 27(2), 73-91. emmett, a., & emde, j. (2007). assessing information literacy skills using the acrl standards as a guide. reference services review, 35(2), 210-229. ferguson, j.e., neely, t.y., & sullivan, k. (2006). a baseline information literacy assessment of biology students. reference & user services quarterly, 46(2), 61-71. green, r. (2006). fostering a community of doctoral learners. journal of library administration, 45(1), 169-183. labaree, d. f. (2003). the peculiar problems of preparing educational researchers. educational researcher, 32(1), 13-22. lightman, h., &. reingold, r. n. (2005). martin, j. (2008). the information seeking behavior of undergraduate education majors: does library instruction play a role?. evidence based library and information practice, 3(4). retrieved from http://ejournals.library.ualberta.ca/ind ex.php/eblip/article/view/1838/3696 morner, c.j. (1993). a test of library research skills for doctoral students. phd diss.(doctoral dissertation, boston college). retrieved from dissertation abstracts. oakleaf, m., & kaske, n. (2009). guiding questions for assessing information evidence based library and information practice 2010, 5.4 33 literacy in higher education. portal: libraries and the academy, 9(2), 273-286. perrett, v. (2004). graduate information literacy skills: the 2003 anu audit. the australian library journal, 53(2), 161-171. saunders, l. (2008). exploring connections between information retrieval systems and information literacy standards. library & information science research, 30, 86-93. singh, a. b. (2005). a report on faculty perceptions of students’ information literacy competencies in journalism and mass communication programs: the acejmc survey. college & research libraries, 66(4), 294-310. thompson, g. b. (2002). information literacy accreditation mandates: what they mean for faculty and librarians. library trends, 51(2), 218-241. zaporozhetz, l. e. (1987). the dissertation literature review: how faculty advisors prepare their doctoral candidates. phd diss. (doctoral dissertation, university of oregon, eugene). retrieved from dissertation abstracts. evidence based library and information practice 2010, 5.4 34 appendix a survey: information literacy and library services for graduate/doctoral students in education programs 1. what program are you currently enrolled in? ____________________________________ 2. how many courses have you completed so far?___________________________________ 3. what is your current occupation? _______________________________________________ 4. for how long? __________________ 5. do you ever access any library services (such as databases or ebooks) remotely (from home)? yes/no 6. if yes, approximately how often do you access these resources? a. more than two times per week b. about once per week c. 1 or 2 times per month d. less than once per month 7. which library services have you used so far (check all that apply)? a. borrow books, videos, or other materials b. ask reference librarians for assistance c. use course reserves or reference materials d. use online databases e. use online course guides or selected web links f. get interlibrary loan (ill) materials g. use microfilm materials h. use print (paper) journals i. use microsoft office software on library computers j. use library space to meet with classmates or professors k. use library space to read or work alone l. use document delivery services m. get research library program (rlp) card n. pay fines o. other (please specify) _____________________________________________ 8. have any of your classes included an information instruction session (when a librarian comes into your classroom to explain and demonstrate library resources)? yes / no 9. after these sessions do you feel as though you can better navigate databases and the library catalog or other research resources? yes / no 10. what are your biggest challenges to finding information during your graduate/doctoral studies? (select all that apply) a. coming up with an original topic b. finding supporting literature/peer reviewed literature for your topic c. following apa citation and writing style d. effectively using databases evidence based library and information practice 2010, 5.4 35 e. other _________________________________________________________________ 11. what information or skills would be useful for you to learn at this time? _______________________________________________________________________________ 12. what methods do you use to begin research? a. google b. databases (which ones?) ___________________________________________________ c. library catalog d. reference books such as encyclopedias e. other ____________________________________________________________________ 13. how comfortable/confident do you feel when seeking information from: (please place a check mark under the appropriate sentiment?) very comfortable comfortable neutral uncomfortable very uncomfortable a. an internet search engine b. a library web page c. a friend d. a professor or teaching assistant e. a faculty or class website f. the library 14. please indicate your comfort level with the listed skills very comfortable comfortable neutral uncomfortable very uncomfortable a. formulate questions based on information needs b. identify potential sources of information c. identify potential sources of information d. develop successful search strategies e. access sources of information, including computerbased and other technologies f. evaluate information g. organize information for practical application h. integrate new information into an evidence based library and information practice 2010, 5.4 36 existing body of knowledge i. use information in critical thinking and problem solving 15. given the topic “students with adhd in the mainstream classroom” as the subject of a research project, in what order would you perform the following steps? (use 0 if you would not take a particular step) __ browse a current printed magazine index __ browse the most recent issue of an education journal __ search the internet using the keywords “adhd” and “mainstream classroom” __ look at reference material that provides an overview of violence and teenagers __ brainstorm the concept, using the terms in the topic __ formulate questions based on the information needed to begin the research __ search subject-based and other related databases 16. other than books and journals, what other types of information are you familiar with or might use for a research project/paper? (select all that apply by circling your answers.) a. websites b. newspapers c. magazines d. interviews e. images/pictures f. speeches g. videos/dvds h. television/broadcasts i. television/radio transcripts j. diaries/letters k. manuscripts l. music m. radio shows/broadcasts n. dissertations/theses o. conference proceedings p. other (internet, journals, scholarly journals) q. none of the above 17. how frequently do you use the following search strategies? always often sometimes rarely never don’t know a. truncation b. boolean operator “not” c. boolean operator “or” d. proximity operators e. library of congress subject headings f. eric descriptors (in the eric database) evidence based library and information practice 2010, 5.4 37 g. cross and multiple-field searching (such as searching by date) 18. where would you go or what would you do to find current information on the following topic: “terrorism on college campuses?” (select all that apply by circling) a. online/internet b. reference works such as encyclopedias c. newspaper archives d. magazines e. television news f. friends/colleagues g. radio news h. a librarian i. faculty/professors j. television/radio transcripts k. abstracts and indexes online (databases) l. abstracts and indexes in print 19. during which events do you ask a librarian for assistance? (select all that apply) a. when i need advice about where to look for information b. when i don’t know how to use an information source c. when i need help choosing the best information source d. all of the above e. i don’t know when i would consult a librarian for assistance 20. can one evaluate an article for bias before reading it? a. no. i need to reaed an article to find bias b. yes. the abstract usually evaluates the article and notes any bias c. yes. if the article is reporting research, it should be unbiased d. i don’t know e. yes. the reputation of a journal or publisher may indicate bias 21. when would you find any of the following resources credible or reliable for your research? always sometimes never a. time or newsweek b. new york times or washington post c. the daily news d. wikipedia e. the onion (website) f. rolling stone g. sports illustrated h. people i. national georgraphic j. national enquirer k. cnn l. today show/good morning america m. saturday night live’s weekend update n. jon stewart’s the daily show o. world news tonite/cbs evening news evidence based library and information practice 2010, 5.4 38 22. circle all that are true: when it comes to information found on the world wide web: a. material on the web is far more reliable than books and magazines. b. material on the web is factual because the web is monitored by international organizations. c. material on the web comes from many varied sources such as business, the government, organizations, or the public. d. material on the web is accurate, timely, and appropriate. e. material from the web does not have to be cited under any circumstances. f. you may use the text or graphics freely unless they are specifically labeled as being copyrighted. g. you can assume that all of the data or text must be cited properly. 23. what is the best definition of “intellectual freedom” a. the right of every individual to both seek and receive information from all points of view without restriction b. the prevention of cheating by students c. the encouragement of open and public sharing of ideas d. the limiting of access to ideas and information that some people find objectionable or dangerous e. the support of the bill of rights 24. you need to find an editorial on president franklin delano roosevelt’s (1882-1945) new deal program. how would you be most likely to find one from that time period? a. read the newspaper b. search a print periodical index c. search government documents d. use an encyclopedia / evidence based library and information practice/ / evidence based library and information practice evidence based library and information practice 2011, 6.1 59 evidence based library and information practice evidence summary free e-books may increase print sales: a study with mixed results a review of: hilton, j. iii, & wiley, d. (2010). the short-term influence of free digital versions of books on print sales. journal of electronic publishing, 13(1). reviewed by: heather r. williams strategist for content processes & services, robert w. woodruff library emory university atlanta, georgia, united states of america email: hrwilli@emory.edu received: 29 nov. 2010 accepted: 15 feb. 2011 2011 williams. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to determine whether the availability of free digital versions of books impacts print sales. design – quantitative data comparison. setting – university instructional psychology department. subjects – a total of 41 books, each with a free digital version and a traditional print version. methods – this study used nielson bookscan data to track print book sales during a 16week period, 8 weeks before a free digital version of the book became available and 8 weeks after the availability of the free digital version. the authors tracked 41 books and organized them into four categories. the first included 7 nonfiction books, the second consisted of 5 science fiction/fantasy books, the third included 5 science fiction/fantasy books released together by random house, and the fourth group consisted of 24 science fiction/fantasy books released by tor books. the books released by tor books, unlike the other books in the study, were available by free download only if a person registered for tor’s newsletter and the downloads were only available for one week. when a free digital book from any of the other three groups was released, it remained available for several weeks, and more often, indefinitely. main results – combined print sales of the nonfiction titles in the first group increased 5% after the release of a free digital copy. the majority of the science fiction/fantasy books in mailto:hrwilli@emory.edu� evidence based library and information practice 2011, 6.1 60 the second group also had an increase in postfree release sales, with a combined increase of 26%. the combined sales of the random house titles increased by 9% after the release of the free digital versions. however, in stark contrast to the results of the first three groups, the fourth group of tor books had a combined decrease in print sales of 18%. while the authors were not able to explain this difference with certainty, they point out that the tor model for releasing the free digital books (making the free books available for only one week and requiring registration in order to download the books) was substantially different from the models used by the other publishers. conclusion – the study suggests a positive relationship may exist between free digital books and short-term print sales. however, the availability of free digital books did not always lead to increased print sales. the authors acknowledge a number of factors not fully accounted for, including the timing of the free digital release, the promotion it received, and the differences in the size of the audiences for the various books studied. ultimately, however, the authors believe the data indicates that when free digital books are offered for a period of time longer than a week, without requiring registration, print sales will increase. commentary this study attempts to answer the question of whether the availability of a free digital version of a book helps or hinders print book sales. however, as the authors point out in their introduction, the question of causation is difficult, if not impossible, to answer experimentally, because it is impossible to both release and not release a free digital version of a book. though true causality may not be determinable, the results of this study did appear to show a positive relationship between the availability of free digital books and print sales. the results of this study should be viewed with some caution. the study had a relatively small subject population, 41 books. how those 41 books were chosen is unclear. were they simply the only books for which the authors could clearly identify the release date of the free digital version? did the genre of the book influence its inclusion in the study? the study consisted of books from two different genres; three of the groups included science fiction/fantasy books and one consisted of nonfiction books. potential disparities in the marketing and promotion of the various books were not considered, nor as the authors acknowledge, were the differences in potential audience size for the books. in looking at the specific sales figures for the various titles, some clearly outsold others by several orders of magnitude. how did the possible differences in marketing effort and audience size impact the results? this study will be of interest to those concerned about issues of open access, open publishing, and the sustainability of traditional publishing models. this study was not meant to be an exhaustive analysis, and more studies of this type should be conducted to determine if the results are replicable for other categories of books. it is unclear whether a study of more academically oriented books or children’s books, for example, would yield the same results. ultimately, this study does contribute to our understanding of the relationship between the availability of free digital versions of books and print sales, and appears to give credence to the argument that the availability of free digital versions will increase print sales. additional studies showing such a correlation among various genres of books would strengthen the argument. / evidence based library and information practice evidence summary evidence based library and information practice using evidence in practice   using assessment tools to develop a workshop for library staff: establishing a culture of assessment   jung mi scoulas clinical assistant professor and assessment coordinator university of illinois at chicago chicago, illinois, united states of america email: jscoul2@uic.edu   received: 10 jan. 2020                                                               accepted: 8 apr. 2020      2020 scoulas. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29709   setting   the university of illinois at chicago is a public research university in the united states serving more than 33,000 students enrolled in its 16 colleges. the university library provides collections (2.2 million print volumes) and services in support of campus instructional and research programs on the chicago campus and at regional health sciences campuses in peoria, rockford, and urbana. the richard j. daley library in chicago holds collections in the arts, humanities, social sciences, and engineering. the library of the health sciences supports all of the health sciences colleges with materials for medicine, dentistry, nursing, public health, pharmacy, and applied health sciences. the university library employs 150 faculty and staff members as well as more than 100 student workers.   problem   the university of illinois at chicago library is committed to enhancing its outcome-oriented initiatives. to that end, the university library’s steering committee developed a strategic plan using a logic model, which is a visualization of a program or project showing the relationships between investments, activities, and intended results (w.k. kellogg foundation, 2006). the assessment coordinator, who is a member of the steering committee, has the primary responsibilities of planning and implementing library assessment initiatives, leading the assessment coordinator advisory committee, conducting surveys, and consulting with library units to determine their assessment needs. another responsibility of the assessment coordinator is creating professional development opportunities for library staff related to assessment. while the assessment coordinator plays a role in providing resources and expertise in assessment, it is not practical for one person to look into every project. at the university library, some librarians who possess strong assessment skills are involved in committees and working groups related to assessment in the library. assessment responsibilities must involve team members across departments to overcome assigning responsibilities to one specific group. therefore, this paper aims to demonstrate how assessment tools were used to create professional development opportunities to contribute to an assessment culture.   evidence   a fundamental task of librarians with assessment responsibilities is to “help librarians demonstrate their library’s value to the institution” (ala, 2017). another key responsibility is to provide mentoring, training, and coaching in order to “build a culture of assessment and organizational capacity for assessment” (ala, 2017).   to foster an assessment culture, the assessment coordinator at the university library decided to develop and offer a workshop designed for library staff to illustrate the use of a logic model followed by individual consultations. the logic model was chosen as the workshop topic because logic models have been used as a framework in grant proposals (e.g., imls), strategic planning (dubicki, 2011), library assessment (stoddart & lajoie, 2014), program evaluation (markless & streatfield, 2017), and program intervention (kletter, mendelez-torres, lilford & taylor, 2018). additionally, logic models are an important training topic (e.g., carli counts, 2019).   during september october 2018, the assessment coordinator had several informal interviews with department heads and senior staff to determine the best way to develop the logic model workshop. based on the feedback from the department heads, the assessment coordinator created two pilot workshops during regularly scheduled monthly department meetings prior to launching the logic model workshop to all staff. two department heads at the university library agreed to offer 30 minutes of their meeting time for the pilot workshop.   three days before the pilot workshop, the assessment coordinator distributed to participants a pre-workshop survey and pre-test asking them about their previous experience and knowledge of logic models. according to the pre-workshop survey results, more than 65% of participants were not aware of logic models, and more than 80% of participants had no experience using a logic model. therefore, the workshop focused on the basic concepts of the logic model, as well as emphasizing the benefits of adapting logic models to their projects. to develop the workshop content, the assessment coordinator reviewed logic model training guides developed by abdi and mensah (2016), taylor-powell and henert (2008), community tool box (n.d.), and the w.k. kellogg foundation (2006).   during the workshop, the assessment coordinator shared the results from the pre-survey, played a video from the dean about the importance of learning the logic model and her expectations, and led activities to enhance the participants’ engagement. after the pilot workshop, library staff and faculty received a post-workshop survey and post-test. the preand post-assessment tools aimed to answer five major questions:   1.      what were the participants’ previous knowledge and experience in using logic models? 2.      what is the overall evaluation of the pilot logic model workshop? 3.      what were the most interesting aspects of the workshops? 4.      did the participants’ knowledge and awareness of logic models increase after the workshops? 5.      which areas of the workshops could be improved?   a total of 26 library staff and faculty attended two pilot workshops (a 65% attendance rate). the first pilot workshop was conducted with the resource acquisition and management department in november 2018 (n = 16), and the second pilot workshop was conducted with the research services and resources department in december 2018 (n = 10). one half of participants completed the workshop evaluation survey. the key findings from the workshop evaluation survey (figure 1) demonstrate that participants generally found the workshop to be helpful. all of the participants (n = 13) replied “strongly agree” and “agree” with respect to the content’s usefulness, the workshop’s level and format, the speaker’s understanding of the material, and their willingness to recommend the workshop to others. only the workshop’s length was an issue. more than 15% of the respondents (n = 2) “disagreed” that the length of the workshop was appropriate, and 20% of the participants (n = 3) in the open-ended questionnaire provided further information saying that the workshop was too short.   figure 1 the evaluation results of the pilot workshops (n = 13)   participants were asked to name the most interesting aspect of the workshop. participants indicated that the most interesting aspect was the game activity where groups worked together to identify the basic concepts of the logic model and put them in the correct (n = 5). one respondent commented that “the video from the dean was great to show she supports these efforts.” the results from the tests indicate that the mean scores of participants’ knowledge of the logic models in the post-test (m = 6.14) were higher than in the pre-test (m = 4.25). this result confirms the effectiveness of the workshop.   implementation   based on the findings from the two pilot workshops, the assessment coordinator has been providing the logic model workshop on an ongoing basis to groups or individuals involved in a project that requires measuring outcomes. to date, the workshop has been conducted with three departments and one committee. because participants reported that the length of the pilot workshop was not sufficient, the workshop was expanded from 30 to 60 minutes. additionally, depending on the group’s previous experience and knowledge as assessed through the survey and pre-test, the assessment coordinator tailored the workshop activities and the level of contents.   during the logic model workshops, many participants understood the concepts; however, when applying the logic model to their work or a project, it was not easy to develop on their own. as a result, follow-up support was offered through one-on-one consultations. after presenting the logic model workshops, several participants requested assistance with either developing logic models for their programs and projects or reviewing their previous logic models to check if the measurable outcomes were appropriate and to verify how the intended outcomes are met.   outcome   within half a year, this initiative had positive results. immediately after the logic model workshop for the library’s human resources department in february 2019, the director of human resources requested a follow-up consultation and expressed their desire to develop an onboarding program using a logic model. since then, the assessment coordinator has been collaborating with the human resources department as they develop an onboarding program for new employees. as a result of the assessment coordinator’s six-month consultation with this department, the onboarding program using the logic model was completed and presented to the library steering committee.   another example is the undergraduate engagement program of the research services and resources department, which was developed by the outreach engagement faculty using the logic model prior to the logic model workshop. during the consultations, the assessment coordinator and the outreach engagement faculty revised the outcome statements and discussed the possible measures that enable faculty to measure the desired outcomes. the updated logic model was also shared with the library steering committee and in a library faculty meeting. some faculty who attended the presentations commented to the assessment coordinator that it was helpful for them to better understand the program and the program goals, and they wanted to develop their projects using logic models.   the last example is a faculty member who participated in the logic model workshop and wanted to write a grant proposal using the logic model. after the workshop, the assessment coordinator and the faculty member reviewed the draft grant proposal and focused on how to articulate measurable outcomes using the logic model. afterwards, the faculty member submitted her grant proposal and received the $20,000 grant.   reflection   developing valuable library staff training requires innovative strategies and a significant investment in time and resources. offering logic model workshops to the university of illinois at chicago library staff was one example of a successful training to build a culture of assessment. it was a useful place to start because it shows participants the value of considering assessment from the very beginning of designing a project or initiative. additional workshops are needed to provide librarians with skills for actually measuring whether the outcomes set in the logic model are being achieved. in the end, each member of the library staff who participates in the workshop will have the ability to assess and evaluate their own project and program which, in turn, establishes and reinforces the culture of assessment.   acknowledgements   this article would not be complete without acknowledging the guidance and support of several colleagues at the university of illinois at chicago library: sandra de groote, linda naru, paula dempsey, gwen gregory, glenda insua and carl lehnen.   references   abdi, s., & mensah g. (2016). focus on: logic models-a planning and evaluation tool. toronto: queen’s printer for ontario. retrieved from https://www.publichealthontario.ca/en/pages/default.aspx     american library association (2017, january). acrl proficiencies for assessment librarians and coordinators, retrieved from http://www.ala.org/acrl/standards/assessment_proficiencies, document id: 498fde98-4c4b-fc94-a5f7-9b6decd8a2e9   community tool box (n.d), chapter 2. other models for promoting community health and development. retrieved from https://ctb.ku.edu/en/table-of-contents/overview/models-for-community-health-and-development   consortium of academic and research libraries in illinois (carli) (2019), carli counts overview. retrieved from https://www.carli.illinois.edu/products-services/carli-counts   dubicki, e. (2011). strategic planning in college libraries. [chicago]: college library information packet committee, college libraries section, association of college and research libraries.   institute of museum and library services (n.d). retrieved from https://www.imls.gov/research-evaluation/evaluation-resources   kletter, m., mendelez-torres, g. j., lilford, r., & taylor, c. (2018). a library of logic models to explain how interventions to reduce diagnostic errors work. journal of patient safety, 00(00), 1. https://doi.org/10.1097/pts.0000000000000459   markless, s., & streatfield, d. (2017). how can you tell if it’s working? recent developments in impact evaluation and their implications for information literacy practice. journal of information literacy, 11(1), 106–119. https://doi.org/10.11645/11.1.2201   stoddart, r., & lajoie, e. w. (2014). the possibilities are assessable: using an evidence based framework to identify assessment opportunities in library technology departments. evidence based library and information practice, 9(4), 29–43. https://doi.org/10.18438/b8mk6g   taylor-powell, e., & henert, e. (2008). developing a logic model: teaching and training guide. retrieved from https://fyi.extension.wisc.edu/programdevelopment/files/2016/03/lmguidecomplete.pdf   w.k. kellogg foundation (2006). w.k. kellogg foundation logic model development guide. retrieved from https://www.wkkf.org/resource-directory/resource/2006/02/wk-kellogg-foundation-logic-model-development-guide   evidence summary   academic health sciences librarian job descriptions do not frequently reflect emerging skillsets and changing research needs   a review of: reed, j. b., & carroll, a. j. (2020). roles for health sciences librarians at college and university libraries. issues in science and technology librarianship, (94). https://doi.org/10.29173/istl42   reviewed by: eugenia opuda health & human services librarian assistant professor dimond library university of new hampshire durham, new hampshire, united states of america email: eugenia.opuda@unh.edu   received: 18 dec. 2020                                                             accepted:  22 jan. 2021      2021 opuda. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29898     abstract   objective – to examine job postings for academic health sciences libraries to determine if they reflect the changing research needs of institutions of higher education and to compare these postings to similar, existing positions.   design – mixed methods data analysis of job advertisements collected through relevant job boards and mailing lists. the authors conducted qualitative content analysis using a modified grounded theory approach, completed two cycles of coding using nvivo 12, and calculated statistical significance using fisher’s exact test.   setting – college and university library and association of academic health sciences libraries job boards and mailing lists between september 1, 2018 and march 1, 2019.   subjects – 104 unique posted job descriptions.    methods – the authors conducted a thorough search of posted position descriptions (ppds) for academic health sciences librarians across a number of job boards and mailing lists between september 1, 2018 and march 1, 2019. in addition to searching ala joblist, mla find a job, association of college & research libraries health sciences interest group (acrl hsig), medlib-l, and acrl science and technology section (sts), the authors also hand searched alumni and general library job electronic mailing lists using relevant keyword searching. inclusion criteria for ppds included research support and other research-related responsibilities for the health sciences. the authors excluded any ppds describing administrative or non-professional positions. following review, the irb determined that the research design did not qualify as human subjects research. after data collection, the authors categorized the ppds using the national network of libraries of medicine (nnlm) geographic regions and by the type of institution—college and university libraries (c&ul) or association of academic health sciences libraries (aahsl). using modified grounded theory, the authors identified emergent themes from the ppds and applied descriptive coding. then, the authors merged categories to create overall themes. using nvivo 12 to facilitate the mixed methods content analysis, the authors ran text queries to identify major themes in the position roles and responsibilities, required and preferred education, and required and preferred qualifications sections. they also noted themes they expected to see that did not emerge in the ppds, as well as emerging roles for health sciences libraries that are identified in the literature but did not appear as major themes in the included ppds. finally, the authors utilized fisher’s exact test to calculate statistical significance.   main results – in the quantitative analysis, the authors identified 60 aahsl and 44 c&ul ppds out of the 104 total job postings. positions were available from all 8 nnlm regions and across 32 states, though they were not all equally distributed. most of the positions (64 of the 104) were located in the nnlm middle atlantic, southeastern/atlantic, and greater midwest regions. the southeastern/atlantic and greater midwest regions made up nearly half of the included ppds. however, the new england region had the most postings per capita. in the qualitative analysis, an ala-accredited mlis or equivalent degree emerged as a near-universal requirement across all ppds. the authors noted that the few ppds that did not require this degree typically referenced it in the preferred education section or described a proxy to the mlis. furthermore, 57% of c&ul positions compared to 27% of aahsl positions listed preferred education (p=0.0004) that was usually related to health and science disciplines that the position supported.   there was significant overlap of required qualifications for aahsl and c&ul postings. the authors also identified a list of hard and soft skills noted in the ppds’ required qualifications sections, including experience with specific tools, expertise in library services, and interpersonal skills. however, reportedly emerging skills in data sciences, open science, grant experience, and research impact assessment were absent in many ppds. the authors found statistically significant differences between two themes in the ppd roles and responsibilities including collection management (p=0.0004) and systematic reviews (p=0.03). additionally, the authors found no statistically significant differences for required qualifications between aahls and c&ul ppds. they did find statistically significant differences for two preferred qualifications including the academic of health information professionals (ahip) credential (p=0.0042) and experience with systematic reviews (p=0.0009). the ahip credential and experience with systematic reviews were absent in the c&ul ppds and referenced rarely in aahsl postings. though diversity, equity, and inclusion (dei) qualifications were frequently referenced in c&ul ppd requirements, the authors noted that research libraries have failed to make meaningful change in diverse candidate hiring and retention, but also pointed to the rapid adoption of dei qualifications in ppds within a short period of time.   the authors highlighted that the roles and responsibilities reflected traditional librarian duties and referenced more emerging skills and research needs than any other section of the ppd. assessment and systematic reviews appeared more often in the roles and responsibilities sections of aahsl and c&ul ppds in comparison to the combined required and preferred qualifications sections of all the ppds. a more traditional responsibility, collection management, also appeared more frequently in the roles and responsibilities section of ppds than in the experience section, suggesting that most hiring committees feel confident that librarians who fill positions will be successful in performing collection management tasks despite experience. the authors noted that collection management, one of the most common themes that emerged from the data analysis, appeared more frequently in c&ul ppds and theorize that aahls may have dedicated collection management departments.   conclusions – while the research literature documents new roles and emerging skills for academic health sciences librarian positions, the authors noted that ppds do not frequently reflect those emerging roles and skills, and maintain traditional health sciences librarian skillsets. the authors concluded that library administrators should design position descriptions that are user centred and match the changing research needs of the local community. ppds should reflect changing priorities by including less weight towards the mlis degree, shifting traditional skillsets from required experience sections to preferred experience sections, adapting the language of ppds to be more inclusive and welcoming for a diverse pool of candidates, and adding an emphasis on dei responsibilities. by creating position descriptions that are user focused, library administrators and hiring committees make meaningful investments for their communities and their strategic priorities.   commentary   reed and carroll contribute to the research literature on analysis of library job advertisements, which includes an examination of acrl standards reflected in job advertisements (gold & grotti, 2013), a content analysis of leadership terms in scholarly communication librarian positions (hackstadt, 2020), and a content analysis of assessment responsibilities in librarian positions (passoneau & erickson, 2014). content analysis of job postings is a common method of research, so much so that a digital collection of position descriptions exists in part to facilitate this research (keith et al., 2017). this research article expands beyond the traditional position description analysis in an attempt to include in-depth, critical content analysis of ppds. the approach is unique in that it examines education and qualification requirements as well as roles and responsibilities in order to identify themes across the included job descriptions.   the ebl critical appraisal checklist (glynn, 2006) was used to appraise this study. the methodology, specifically the use of inferential statistics, is not appropriate for meeting the authors’ objectives. the authors state in their limitations that the selected six-month sample reflects a small snapshot in time and that the study findings may not be generalizable due to the constantly changing trends in ppds as well as geographic limitations. these limitations prevent the authors from making any meaningful inference about ppds overall. instead, the authors should take a descriptive approach, simply summarizing and displaying the findings of their limited data.   while data collection and analysis are meticulously described and replicable given the inclusion of search terms and exact text queries, the inappropriate methodology raises some concerns about study replication. additionally, the authors note that their search strategy introduced several opportunities for bias that favoured more postings from aahsl over c&ul institutions, as well as institutions that were able to afford job description posting fees. in order to account for missing ppds, the authors attempted to conduct a thorough hand search through mailing lists. furthermore, the variable formatting of ppds and inconsistent use of language to describe common library responsibilities presented additional challenges to data analysis.   as the authors noted, more research is needed, specifically a larger and more geographically diverse sample across a longer time period, for generalizability of the study findings. their limited data suggests that an incomplete snapshot of academic health sciences ppds within a short time period does not reflect emerging research trends and does not prepare librarians to support interdisciplinary research teams. however, the authors raise important questions. academic libraries should examine their institutional needs closely in order to align strategic priorities and create meaningful ppds that bring value to the research community. the authors also underscore that the inclusion of dei responsibilities in ppds must be matched with meaningful action in order to recruit and retain diverse librarians. this research study provides a robust, but flawed, study methodology and clear suggestions for conducting a fuller investigation and improving academic health sciences librarian job descriptions.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387–399. https://doi.org/10.1108/07378830610692154   gold, m. l., & grotti, m. g. (2013). do job advertisements reflect acrl’s standards for proficiencies for instruction librarians and coordinators?: a content analysis. the journal of academic librarianship, 39(6), 558–565. https://doi.org/10.1016/j.acalib.2013.05.013   hackstadt, a. (2020). leadership, development, and expertise: a qualitative content analysis of scholarly communication librarian position announcements. journal of librarianship & scholarly communication, 8(1), ep2376. http://doi.org/10.7710/2162-3309.2376   keith, b. w., smith, b. j., & taylor, l. n. (2017). building a collaborative position description archive as a community of practice. portal: libraries and the academy, 17(2), 419–434. https://doi.org/10.1353/pla.2017.0026   passonneau, s., & erickson, s. (2014). core competencies for assessment in libraries: a review and analysis of job postings. library leadership & management, 28(4), article 4. https://journals.tdl.org/llm/index.php/llm/article/view/7080       evidence based library and information practice evidence based library and information practice 2010, 5.4 70 evidence based library and information practice article google wave: have ctsa-minded institutions caught it? amy donahue special projects librarian university of minnesota bio-medical library st. paul, minnesota, united states of america email: adonahue@umn.edu received: 26 july 2010 accepted: 02 nov. 2010 2010 donahue. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract background google wave was touted as the next big communication tool— combining e-mail, social networking, and chat within a single “wave”—with the potential to create a new world for collaboration. information professionals who are knowledgeable of this tool and its capabilities could become uniquely situated to use it, evaluate it, and teach it. this seemed especially true for those working within clinical and translational science award (ctsa)-minded institutions, given the promise of interdisciplinary collaboration between investigators and the potential for creating new authorship models. this case study on google wave users who are affiliated with ctsa-minded institutions, was designed for and presented at the evidence-based scholarly communication conference held by the university of new mexico health sciences library and information center. it provides an early evidence based evaluation of google wave’s potential. methods two “waves” were created. the first consisted of five survey questions designed to collect demographic data on the respondents’ roles, a general impression of wave, the specific tools within wave that might be useful, and potential collaborators with whom the respondents might use wave. the second wave was a private, guided discussion on wave’s collaboration potential. individuals from ctsaminded institutions were invited to participate with messages on twitter, forums, blogs, and electronic mail lists, although there were difficulties reaching out to these institutions as a group. mailto:adonahue@umn.edu� evidence based library and information practice 2010, 5.4 71 results by the conclusion of the study, only a small number of people (n=11, with a viable n=9) had responded to the survey. given this small result set, it made sense to group the responses by the respondents’ roles (ctsa staff and researchers, support staff, medical librarian, or general public) and to treat them as individual cases. most of the respondents were librarians and support staff who felt that wave might have potential for collaboration; there were no ctsa researcher respondents. for the second part of the study, the discussion wave, only one participant explicitly expressed interest in joining. all were invited to join, but there was no participation in the discussion wave at the conclusion of the study. conclusions -the results of this study implied that google wave was not on the forefront of ctsa-minded institutions’ communication strategies. however, it was being used, and it did demonstrate new collaboration and authorship capabilities. being generally aware of these capabilities may be useful to information professionals who seek to be current and informed regarding developing technology and to those interested in scholarly communication practices. in addition, the difficulties encountered during this case study in attempting to reach out to ctsa-minded institutions raised the question of how members currently communicate with each other as institutions and as individuals. there was a lesson learned in the usefulness of doing case-study research to evaluate new technologies; the cost in terms of time was relatively low, and knowledge about the technology itself was gained while establishing a base level of evidence to potentially build on in the future. background google announced google wave to the world on 28 may 2009 (google, 2009a). after a few months of additional development, it was released to a group of more than 100,000 people on 30 sept. 2009 (rasmussen, 2009). in the months after the initial release, the number of users on wave grew substantially. google first extended additional invitations, and then opened the service to anyone who had or was interested in registering for a google account. many touted the service (google, 2009b; mccracken, 2009; siegler, 2009) as the next big communication tool, combining e-mail, social networking, and chat with other online tools. potentially, it seemed google wave could change the face of collaboration, greatly impacting the world of scholarly communication. what is google wave? explaining google wave is complicated by its jargon-rich terminology and a lack of the usual google intuitiveness; the following explains some of the major key terms. the main components of google wave are its “waves.” in writing, the standard is to capitalize the “w” when referring to the service, and to use the lowercase when referring to the objects. the waves are blank canvases for text and embedded objects such as videos, pictures, or documents. the waves are an answer to one of e-mail’s biggest problems: the number of ever growing and ever changing versions that occur when e-mails go back and forth. in other words, a wave can be thought of as a single conversation that can be edited and added to by participants; everyone is looking at the same copy. all revisions are tracked and visible in a playback mode. a few more key terms are worth noting: a “blip” is a reply within a wave, “gadgets” are small applications that add interactive content to a wave, and “bots” are automated robots that execute commands within a wave. a wave is “public” if it can be found and joined by anyone, and a “private” wave is a wave where only invited participants can join and interact. trapani and pash (2009) go into further detail on wave’s capabilities and the vocabulary used to describe them in their book, the complete guide to google wave. evidence based library and information practice 2010, 5.4 72 google wave, information professionals, and clinical and translational science librarians and information professionals may want to be aware of wave and its capabilities as it can be argued that they are uniquely situated to teach it, use it, and evaluate it. this position is supported by the fact that librarians currently provide assistance and offer training on other research tools, from google documents to refworks, frequently in interdisciplinary settings (e.g., within a health sciences library that serves multiple research and academic departments). librarians often use these tools themselves to become knowledgeable in order to answer their users’ questions, either through continuing education (rethlefsen, piorun, & prince, 2009) or by self-directed learning (whisner, 2009). google wave could be integrated into workflows in a similar fashion, giving both librarians and information professionals opportunities to demonstrate value as cutting edge information technology adopters and teachers, especially if they are able to help with evaluating when its use might be appropriate. librarians who serve the population of clinical and translational science researchers may find all of these roles particularly relevant, given wave's promise for interdisciplinary collaboration and the subsequent potential for new authorship models. clinical and translational science researchers span a breadth of disciplines including everything from basic chemistry to family practice to public health in the effort to speed up the process from bench to bedside to practice and back again (woolf, 2008). between 2006 and 2010, 55 clinical and translational science centers in the united states received federal funding awards, and there will be a total of 60 of these official programs within the next few years (national center for research resources, 2010). in addition, there are numerous other institutions in the u.s. and around the world doing similar research. the collaboration and authorship models that wave makes possible seem potentially useful to this user community. however, these models raise concerns such as tracking large numbers of authors (who may be inter-institutional as well as interdisciplinary), managing copyright and ownership, and deciding how and where to preserve and make accessible work done in the wave environment. all of these concerns fall into areas where librarians and information professionals have expertise. the case for a case study demonstrating the potential for google wave’s use in the translational science realm and describing information professionals’ possible roles is an interesting thought exercise, but “potential” does not itself necessarily translate into real life adoption. the author's interest in exploring stakeholders’ actual use and thoughts on google wave coincided with the evidencebased scholarly communication conference call for original research on scholarly communication topics, specifically within the clinical and translational science realm. the conference provided an opportunity to collect preliminary evidence and to present it to a librarian and information professional audience directly involved in clinical and translational science support. given the time limit of the months between the proposal's acceptance and the actual conference, along with the fact that google wave was still in a preview or developmental stage, and it had not been widely adopted, the author determined that the case study would be the most practical method for establishing some initial evidence regarding the use of wave in the clinical and translational science. without a large number of users who, 1) have had significant experience with wave and, 2) who are affiliated with a translational science program, a bigger experiment or trial was not feasible. observational studies are an important part of the evidence base for this very reason (ahn, bhandari, & schemitsch, 2009; black, 1996; hoppe, schemitsch, morshed, tornetta, & bhandari, 2009). in addition, most likely because of the newness of both google wave and the field of clinical and translational science, there did not evidence based library and information practice 2010, 5.4 73 appear to be any available evidence on their intersection. there have been a few recently published articles that discuss the potential of google wave as a web-based laboratory record (neylon, 2009) and as a tool to help pediatricians (sethi, 2010), but neither of these provided actual evidence on the use of wave. another article discussed the concrete development of a google wave “synbiowave” for synthetic biologists, but offered no evaluation of its adoption (staab, walossek, nellessen, grünberg, arndt, & müller, 2010). looking for articles on communication solutions for clinical and translational research yielded more results. a study at the university of pittsburgh described the process of creating and evaluating an “electronic management – clinical translational research system” that combined information, communication, and data hubs (cecchetti et al., 2009). cecchetti and his colleagues concluded that such systems “are a valuable support to ctr [clinical translational research] that can be provided throughout the ctr community, including the consortia of institutions that hold clinical translational science awards” (p. 455). all of these studies together lend support to looking deeper into wave’s potential. however, rather than immediately integrating wave into clinical and translational science workflows, it may be prudent to follow koufogiannakis’s (2007) suggestion “to plan an innovative project with research and assessment in mind” (p. 109) and to heed booth’s (2007) plea that “[o]nly by putting in place mechanisms for capturing evaluative data as we introduce new technologies will we provide others with the opportunity to build upon what we are currently learning” (p. 301). therefore, this case study, which can also be viewed from the action research and collaborative inquiry perspective (blichfeldt & andersen, 2006), provides an early evidence based evaluation of google wave’s potential, lays the groundwork for continued evaluation, and may provide a model for the early evaluation of other technologies that could impact the scholarly communication of clinical and translational research. population this study focused on google wave users from the united states who are involved with or affiliated in some capacity with clinical and translational science institutions, especially those who have received or are applying for the national institutes of health ctsas (collectively identified here as “ctsa-minded institutions”). this population became the primary focus, because it was also the target population for the evidence-based scholarly communication (ebsc) conference, where the results were presented. targeted users included investigators, support staff, and affiliated (liaison) librarians and information professionals of these institutions. members of the general public were also able to participate in the public survey, although they were not directly recruited. individuals from ctsa-minded institutions were asked to self-identify and were invited from public waves on related topics (e.g., research collaboration, scholarly communication, health technology, and biomedical informatics) found by searching wave tags and text. a blip was added to each appropriate wave, inviting individuals with affiliation to take the survey by clicking on a direct link. because this recruitment was all done publicly, it became possible for members of the general public to participate. additionally, “calls” for participants went through other available means, including twitter, a health informatics forum (donahue, 2010b), and a guest post on a ctsa-affiliated medical librarian’s blog (donahue & dettmar, 2010). attempts were also made to contact the ctsa communications key function committee in order to gain access to ctsawide electronic mail lists, but were not successful. since individuals were recruited using social networking technologies (after attempts to reach the communication committee failed), this population was also assumed to already have an interest in using these technologies. in addition, the focus on recruiting within wave lent itself to attracting users with evidence based library and information practice 2010, 5.4 74 familiarity with the tool. while on one hand this practice could bias the study towards those with positive feelings about wave, it also ensured that respondents were at least somewhat knowledgeable about the tool as they evaluated it. methods this case study explored google wave using qualitative survey and discussion tools. two waves were created to house these tools separately, and both are still available in google wave. the decision was made to do the study within wave, because not only did wave have the ability to easily capture any data collected, but also because it demanded that everyone who responded have had at least some experience with the tool. this may have been a significant limitation, in that it created the assumption that those reading the waves were literate in their use, but it also created a mechanism to capture data within a test implementation, along the lines of booth and koufogiannakis’s advice. the first wave was and is public and consists of several brief survey questions (using a polling gadget created specifically for wave). it was designed to collect demographic data on the respondents’ roles (i.e., researcher, librarian, support staff, or general public), if and what specific gadgets and bots might be useful, and potential collaborators for using wave. survey questions used both explicit options and an option to add “other” choices, if desired (appendix a). the survey format was selected because it could be easily implemented within wave and because it would be easy to manage and collect data should a large number of people respond. the second wave was a private, guided discussion on collaboration in wave. the purpose of this discussion was two-fold: 1) to collect more in-depth feedback as well as quotations on wave’s potential, and 2) to explore that potential by creating a space where respondents could experiment and practice using wave’s features. in fact, in order to take advantage of wave’s collaboration capabilities, a novel authorship model was proposed for this project: anyone who contributed to the discussion wave would have the option of being considered an author on this paper. in theory, this would create a real world test case for scholarly communication through wave. formal data collection on the two waves began friday, 8 jan. 2010 (following confirmation that the proposal was appropriate and accepted for the ebsc conference). data collection prior to the conference ended on friday, 19 feb. 2010. however, the waves are still and will remain open indefinitely, and a few additional responses are reported in the results below that were not reported at the ebsc conference. the last response recorded here is from 28 june 2010. the waves will remain a living place for the indefinite future, to collect future work and discussion in response to this article. the private discussion wave has also been made public. both of the waves can be accessed through their urls and then by either signing in with or creating a google account (donahue, 2010a; donahue, 2010c). results as of 28 june 2010, eleven individuals had joined the public survey wave in addition to the author. these individuals did not each answer all the survey questions, but each question had multiple responders. one of the participants explicitly expressed interest in joining the private discussion wave, and all were eventually invited to join for the sake of numbers. however, there was no participation in the discussion wave by the conclusion of the study, and no additional authors for this paper, as hoped for in the authorship model outlined above. a full tally of the number of responses for each survey question can be found in appendix b. however, given the low number of respondents, the results are reported here on a case-by-case basis and grouped by the labels with which the respondents selfidentified. the individual cases were isolated evidence based library and information practice 2010, 5.4 75 using the google wave playback features, and only those individuals who answered the initial “does wave have potential?” question are described below. (for this particular question, 3 responded “yes,” 5 responded “maybe,” and 1 responded “no"; total n=9). anonymity has been preserved in the paper, and no identifying information was requested in the survey, although individuals might have been identifiable by usernames used in the wave. librarian responses of the respondents who answered the “does wave have potential?” question (4/9) were librarians. the librarians mostly felt that wave might have potential for ctsa-minded institutions, although one librarian (librarian 4) did respond with a definitive “yes.” (however, this was the only question to which librarian 4 responded.) librarians 1 and 2 responded that “fnordlinky,” a bot that takes a pubmed identifier (a unique number assigned to each citation in pubmed) and returns the citation directly into the wave, would be of use to researchers. however, they did not suggest any additional gadgets or bots. librarians 1-3 agreed that there was potential in wave for collaboration between researchers within institutions and with librarians. librarians 1 and 2 also thought that there was potential for collaborating with researchers outside of the home institution, but only librarian 2 felt that wave would be helpful for coordinating with project managers. the final question on potential uses also showed some variation among the librarian responders. librarian 1 felt that sharing resources and collaborating on papers both within and between institutions, and asking questions within an institution were all potential uses. librarian 2 agreed that sharing resources both within and between had potential, and also felt that there was potential for coordinating project teams. librarian 3 saw potential for collaborating on papers within an institution, and also saw potential in coordinating project teams. ctsa-minded institution support staff responses three of the respondents identified themselves as clinical and translational science institute support staff. staff 1 indicated that wave was not useful for ctsa-minded researchers, yet they also indicated that there was potential within wave to collaborate with researchers both inside and outside the home institution, as well as with project managers and librarians. staff 1 also saw the coordination of project teams as a potential use. so although staff 1 was the only individual to go with the absolute “no” on wave's potential usefulness, there was still an indication of some potential, given the responses to the other questions. staff 2 was the most positive responder, choosing “yes” when asked whether wave had potential and choosing both fnordlinky and “watexy” (a bot that allows for the use of the mathematical latex language within a wave) as useful bots and gadgets. staff 2 also indicated that all collaboration suggestions and all use options could have potential. however, staff 2, like all other respondents, did not add any additional suggestions, even though it was possible to do so on every question. staff 3 also chose “yes” for the question of whether wave had potential, but did not answer any additional questions. unknown and general public responses the final two respondents were a member of the general public, and an individual who chose not to identify with a demographic group. the unknown individual responded only to the potential usefulness question, and indicated that wave might be of use. the member of the general public also chose “maybe” when asked if wave had the potential to be useful. this individual also chose watexy as a potentially useful bot. the unknown individual further indicated that wave could be used to work with any of the given collaborators (i.e., researchers both from the home and outside institutions, project managers, and librarians), and felt that wave's potential uses included coordinating teams, sharing resources both within and between evidence based library and information practice 2010, 5.4 76 institutions, and collaborating on papers both within and between institutions. discussion some of the study's limitations have already been mentioned, including the recruitment of respondents and the method of placing the study within wave. in the future, ideally these limitations could be addressed by expanding recruitment (possibly through incentives and additional channels) and by choosing an alternative method of data collection (perhaps after some initial test on the ability to use the tool being evaluated). it might also be helpful for future studies to put the limitations of the low response rate and the short time period (given the timeframe of the ebsc conference) in context and to discuss them in more depth. one deterring factor against more participation might have been wave’s prebeta, preview status. not only was it a new tool, it was also a tool not yet available to everyone. it also lacked some desired functionality. in addition, wave often ran slowly and frequently crashed. another important issue was wave’s lack of privacy. for example, it was apparent who responded to the poll questions; every blip has clear authorship, perhaps a reason why some of the respondents chose not to take the complete survey. in addition, google’s terms and conditions indicate that google has the right to use anything posted in wave, public or private. this is a major deterrent for researchers and investigators, although this would not be the same issue on an institutioncontrolled wave server. the other side of the privacy issue was a lack of authority control. it would have been possible --although perhaps unlikely, given the amount of effort-for example, for someone to give false information by impersonating a ctsa-minded individual. another communication issue became apparent as efforts to reach out to participants continued unsuccessfully: there did not seem to be a standard method for communicating with ctsa-minded investigators and affiliated staff in order to effectively promote the survey. any future studies might benefit from planning for additional time to find appropriate communication channels. the communication channels that were used for this study likely also affected the participant demographics. while it was acknowledged that participants would be wave users and therefore only a small percentage of ctsaminded institution staff, the author's reliance on a network of medical librarians (after the ctsa communications key function committee failed to respond), almost certainly contributed to the larger number of librarian respondents. the small result set of this study implies that google wave is not on the forefront of clinical and translational science communication, despite some initial positive feedback. these results do not provide solid evidence either for or against the tool’s potential (although the lack of response and interest certainly seems to indicate a lack of potential, despite the small amount of positive feedback). however, it is clear that wave was being used and that it has opened up new collaboration and authorship capabilities. simply being aware of these abilities may be useful to information professionals serving ctsa-minded institutions. also, while the number of respondents was very small, the results may provide a starting place should a ctsaminded institution show interest in using tools similar to wave. for instance, most of the individual cases supported using wave as a collaboration tool, but there was not much support for using it as more of a communication tool (e.g., asking questions). building on this early evidence might help focus internal marketing efforts and early evaluations. in addition, although there was little knowledge gained in terms of whether wave is perceived as potentially useful for ctsaminded institutions, and no information whatsoever on the use of google wave as a multi-author tool for affiliated investigators evidence based library and information practice 2010, 5.4 77 and staff, there are examples in other fields, such as journalism and business, where wave is used for collaboration (google, 2010). there is also the takeaway lesson of the overall research process. in this case, the process demonstrated the usefulness of doing preliminary case-study research to evaluate a new technology. the cost in terms of time (designing, creating, and promoting the survey and discussion waves) was relatively low, and knowledge was gained on how to use the technology while establishing a base level of evidence to potentially build on in the future. a single individual did the groundwork for the project over the course of less than three months; a similar undertaking might be a useful strategy (e.g., for a lone liaison librarian) for attempting to gather evidence on the use and potential of a new tool such as an open-access platform or an institutional repository. keeping in mind the lesson learned on the lack of ctsa-minded institution communication, it may be prudent to first focus on use in a home institution to start, or to first confirm that the selected method of communication with the ctsaminded institutions will work. conclusion on 4 aug. 2010, the official google blog announced that development on google wave as a standalone product had been canceled, and that the wave site itself will likely only be supported through the end of the year (hölzle, 2010). since that announcement, plans for the completion of a “wave-in-a-box” project have finalized around the creation of an application to be hosted and used on individual servers (north, 2010). while wave is not exactly gone for good, the ways in which ctsa-minded institutions might use it seem dead in the water, or at the very least, different from the inter-institutional communication, collaboration, and authorship ideas this project was exploring. perhaps it is fitting, then, that this case study did not provide significant evidence to support the idea that google wave is of value to members of the ctsa-minded institution community. however, that is not to say that it was inconsequential. it raises questions for future research, including exploring the communication needs of ctsa-minded researchers, such as what (if any) information they communicate to each other, given their highly interdisciplinary backgrounds; what communication channels they use; and what features are the most important or missing from these tools. there may also be future research in understanding why there was so little uptake for this study. in addition to the reasons given in the discussion, there may be issues of communication tool “overload,” a lack of technological skills, or a lack of interest. comparative research studies with groups using different tools would be an appropriate next step as well. these directions all fall within the realm of scholarly communication, where librarians and information professionals have a vested interest. it may be prudent for us to address these research questions, so that we might find places where our skills are needed within the communication flow. to take a step in a different direction, knowledge about communication methods may help with future studies similar to this one, where a new tool may have some potential and evidence is needed on what is already working. some of this project's respondents did indicate that wave may yet be useful, and there were conversations occurring within wave about its potential usefulness in research and for scholarly communication (eresearch community wave, 2010). the “wave-in-abox” project could produce cases regarding interesting uses. some of the technologies developed are indeed already being put to use, including “the in-browser rich text editor, the wave model, and the concurrency control mechanism” (zamfirescu & baxter, 2010). there may be continuing research for information professionals in understanding how these technologies could affect everything from workflows to personal information management, and comparison studies with already existing technologies evidence based library and information practice 2010, 5.4 78 would again be viable. this may be a long way off, but perhaps knowing about tools such as wave and recognizing that these new technologies will be bringing unique challenges is a lesson itself. ctsa-minded institutions may never “catch the wave,” but perhaps the lessons learned from this project can help the librarians and information professionals who serve them prepare for whatever is next on the horizon. acknowledgements the author thanks the medical librarians on twitter who responded to and forwarded her requests for participants in her wave survey, with special thanks to nikki dettmar, who also hosted a guest post on her blog. many thanks as well to jon eldridge, philip kroth, the participants at the ebsc conference, and to my peer reviewers, whose comments and suggestions for improving this paper were immensely helpful. this research was supported in part by an appointment to the nlm associate fellowship program sponsored by the national library of medicine and administered by the oak ridge institute for science and education. the university of minnesota bio-medical library provided additional support as an nlm associate fellowship program second year host. references ahn, h., bhandari, m., & schemitsch, e. h. (2009). an evidence-based approach to the adoption of new technology. the journal of bone and joint surgery (american), 91(supp. 3), 95-98. doi:10.2106/jbjs.h.01593 black, n. (1996). why we need observational studies to evaluate the effectiveness of health care. bmj (clinical research ed.), 312(7040), 1215-1218. blichfeldt, b. s., & andersen, j. r. (2006). creating a wider audience for action research: learning from case-study research. journal of research practice, 2(1), article d2. retrieved from http://www.jrp.icaap.org/index.php/jrp/a rticle/viewarticle/23/43 booth, a. (dec. 2007) blogs, wikis, and podcasts: the ‘evaluation bypass’ in action? health information and libraries journal, 24, 298-302. doi: 10.1111/j.14711842.2007.00739.x cecchetti, a. a., parmanto, b., veccio, m. l., ahmad, s., buch, s., zgheib, n. k., groak, s. j. jr., vemuganti, a., romkes, m., sciurba, f. donahoe, m. p., branch, r. a. (dec. 2009). team building: electronic management-clinical translational research (em-ctr) systems. clinical and translational science, 2(6), 449-455. doi: 10.1111/j.1752-8062.2009.00157.x donahue, a. (4 feb. 2010a). google wave & ctsis: a discussion [blog post]. retrieved 27 nov. 2010 from https://wave.google.com/wave/waveref/g ooglewave.com/w+oxg4xnhba donahue, a. (10 jan. 2010b). google wave in translational research. health informatics forum. retrieved 27 nov. 2010 from http://www.healthinformaticsforum.com/ forum/topics/google-wave-intranslational donahue, a. (8 jan. 2010c). the potential for wave at ctsa-minded institutions/ctsis (2010) [blog post]. retrieved 27 nov. 2010 from https://wave.google.com/wave/waveref/g ooglewave.com/w+poq-vaphb donahue, a., & dettmar, n. (4 feb. 2010). other aeries: ctsa/ctsis in ebsc [blog post]. retrieved 26 nov. 2010 from http://eagledawg.net/0210otheraeries/ eresearch community wave. (15 oct. 2009) [blog post]. retrieved 26 nov. 2010 from evidence based library and information practice 2010, 5.4 79 https://wave.google.com/wave/waveref/g ooglewave.com/w+khigfzxkd google. (28 may 2009a). google wave developer preview at google i/o 2009 [video file]. retrieved 20 nov., 2010 from only google. (2009b). went walkabout. brought back google wave. retrieved 26 nov. 2010 from http://googleblog.blogspot.com/2009/05/ went-walkabout-brought-back-googlewave.html google. (2010). how people are using google wave. retrieved 20 nov., 2010 from http://wave.google.com/using-wave.html hoppe, d. j., schemitsch, e. h., morshed, s., tornetta, p.,3rd, & bhandari, m. (2009). hierarchy of evidence: where observational studies fit in and why we need them. the journal of bone and joint surgery (american), 91(suppl. 3), 2-9. doi:10.2106/jbjs.h.01571 hölzle, u. (4 aug. 2010). update on google wave [blog post]. retrieved 20 nov. 2010 from http://googleblog.blogspot.com/2010/08/u pdate-on-google-wave.html koufogiannakis, d. on evidence based practice and innovation. evidence based library and information practice, 2(4), 108111. retrieved 27 nov. from http://ejournals.library.ualberta.ca/index. php/eblip/article/view/787/665 mccracken, harry. (28 may 2009). forget google apps: google wave is the new epicenter of the google-microsoft war [blog post]. retrieved october 18, 2010 from http://technologizer.com/2009/05/28/forge t-google-apps-google-wave-is-the-newepicenter-of-the-google-microsoft-war/ national center for research resources. (summ. 2010). clinical and translational science awards. ncrr fact sheet. retrieved october 29, 2010 from http://www.ctsaweb.org/docs/ctsa_fact sheet.pdf neylon, c. (2009). head in the clouds: reimagining the experimental laboratory record for the web-based networked world. automated experimentation, 1(3). doi:10.1186/1759-4499-1-3 north, a. (2 sept. 2010). wave open source next steps: “wave in a box” [blog post]. retrieved 25 nov. 2010 from http://googlewavedev.blogspot.com/2010/ 09/wave-open-source-next-steps-wave-inbox.html rasmussen, l. (29 sept. 2009). surf's up wednesday: google wave update [blog post]. retrieved 20 nov. 2010 from http://googleblog.blogspot.com/2009/09/s urfs-up-wednesday-google-waveupdate.html rethlefsen, m. l., piorun, m., & prince, j. d. (oct. 2009). teaching web 2.0 technologies using web 2.0 technologies. journal of the medical library association, 97(4), 253-259. doi:10.3163/15365050.97.4.008 sethi, s. k. (2010). when google wave hits the pediatricians! indian pediatrics, 47(5), 455. retrieved 27 nov. 2010 http://www.indianpediatrics.net/may2010 /455.pdf siegler, m. g. (28 may 2009). google wave drips with ambition: a new communication platform for a new web [blog post]. retrieved october 18, 2010 from http://techcrunch.com/2009/05/28/googlewave-drips-with-ambition-can-it-fulfillgoogles-grand-web-vision/ staab, p. r., walossek, j., nellessen, d., grünberg r., arndt k. m., müller, k. m. (1 nov. 2010). synbiowave: a real-time communication platform for molecular evidence based library and information practice 2010, 5.4 80 and synthetic biology. bioinformatics, 26(21), 2782-2783. doi:10.1093/bioinformatics/btq518 trapani, g., & pash, a. (2009). the complete guide to google wave: how to use google wave. retrieved 20 nov. 2010 from http://completewaveguide.com/ whisner, m. (2009). learning new applications. law library journal, 101(1), 115-120. retrieved 27 nov. 2010 from http://www.aallnet.org/products/pub_llj_ v101n01/2009-06.pdf woolf, s. h. (9 jan. 2008). the meaning of translational research and why it matters. jama: the journal of the american medical association, 299(2), 211-213. doi:10.1001/jama.2007.26 zamfirescu, j.d., baxter, a. (24 june 2010). wave federation keeps growing: rich text editor is open source [blog post]. retrieved 20 nov. 2010 from http://googlewavedev.blogspot.com/2010/ 06/wave-federation-keeps-growing-richtext.html evidence based library and information practice 2010, 5.4 81 appendix a – “the potential for wave at ctsa-minded institutions/ctsis” (survey questions) 1. does wave have the potential to be helpful to researchers at ctsis? a. yes b. no c. maybe 2. who are you? a. ctsi researcher b. librarian c. ctsi support staff d. ctsi-unaffiliated/general public e. (add option) 3. gadgets and robots that might have potential for research (vote for all that appeal to you): a. fnordlinky (converts pmid to citation info) b. watexy (use latex mathematical language in waves) c. none or n/a d. (add option) 4. who you would work with: potential wave collaborators (vote for all that appeal to you): a. researchers at home institution b. researchers at other institutions c. project managers d. librarians/information professionals e. (add option) 5. potential uses for wave (vote for all that appeal to you): a. sharing resources within institution b. sharing resources outside of institution c. coordinating project teams d. collaboration on papers within an institution e. collaboration on papers with other institutions f. asking questions within institution g. asking questions outside of institution h. (add option) evidence based library and information practice 2010, 5.4 82 appendix b – survey results 1. does wave have the potential to be helpful to researchers at ctsis? yes 3 no 1 maybe 5 total n 9 2. who are you? ctsi researcher 0 librarian 4 ctsi support staff 3 ctsi-unaffiliated/general public 2 (add additional option) 0 total n 9 3. gadgets and robots that might have potential for research (vote for all that appeal to you): fnordlinky 3 watexy 2 none or n/a 0 (add additional option) 0 total votes 5 4. who you would work with: potential wave collaborators (vote for all that appeal to you): researchers at home institution 6 researchers at other institution 5 project managers 5 librarians/information professionals 6 (add additional option) 0 total votes 22 5. potential uses for wave (vote for all that appeal to you): sharing resources within institution 4 sharing resources outside of institution 4 coordinating project teams 5 collaboration on papers within an institution 4 collaboration on papers with other institutions 3 asking questions within institution 2 asking questions outside of institution 1 (add additional option) 0 total votes 23 / evidence based library and information practice ebl 101   research methods: triangulation   virginia wilson director, centre for evidence based library and information practice (c-eblip) university library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   received: 11 feb. 2014   accepted: 16 feb. 2014      2014 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     triangulation refers to using more than one particular approach when doing research in order to get richer, fuller data and/or to help confirm the results of the research. denzin via flick (2002) points out four different types of triangulation:   data triangulation: using different sources of data. this includes different times for data collection, difference places from which to collect the data, and different people who could be involved in the research study. flick (2002) indicates that “the starting point is to purposively and systematically involve personas and study groups, local and temporal settings in the study” (p. 226). investigator triangulation: using several people (or at least more than one) in the data gathering and data analysis processes. this would be a “systematic comparison of different researchers’ influences on the issue and the results of the research” (flick, 2002, p. 226). theory triangulation: approaching the data with multiple theories or perspectives in mind to “extend the possibilities for producing knowledge” (flick, 2002, p. 227). methodological triangulation: two subtypes are noted – within-method and between-method. using more than one method to gather data.   triangulation initially was undertaken as a way to increase the validity of research results. however, flick (2002) states that “triangulation is less a strategy for validating results and procedures than an alternative to validation which increases scope, depth and consistency in methodological proceedings” (p. 227). triangulation can be used in quantitative and qualitative research and it even seems as though triangulation is just another term for mixed-methods research. the journal of mixed methods research has a special issue devoted to analyzing and exploring the various ways triangulation is used in mixed-methods research, obviously differentiating the two terms. this topic was chosen for the special issue “based on the claims made by many scholars in the field that triangulation provides a justification for the use of mixed methods” (mertens & hesse-biber, 2012, p. 76).   triangulation has been taken to task on a couple of occasions. given the philosophical and epistemological nature of the various positions taken regarding triangulation, it is too complex to delve into in this relatively brief column. in brief, triangulation has been criticized for “subscribing to a naive realism that implies that there can be a single definitive account of the social world” as well as for assuming that “sets of data deriving from different research methods can be unambiguously compared and regarded as equivalent” (bryman, 2004, p. 3). despite possible controversies, triangulation in whatever form it takes (although the most common is probably methodological triangulation), has become a staple in social science research.   examples of studies using triangulation:   wahl, d., avery, b., & henry, l. (2013). studying distance students: methods, findings, actions. journal of library & information services in distance learning7 (1-2), 183-209. doi: 10.1080/1533290x.2012.705656   zuze, h. & weideman, m. (2013). keyword stuffing and the big three search engines. online information review37(2),268-286. doi: 10.1108/oir-11-2011-0193   bitso, c. & fourie, i. (2012). an investigation of information-seeking behaviour of geography teachers for an information service intervention: the case of lesotho. information research17(4). retrieved 23 feb. 2014 from http://www.informationr.net/ir/17-4/paper549.html#.uwo1i-zfdiu     resources:   o’cathain, a., murphy, e., & nicholl,j. (2010). three techniques for integrating data in mixed methods studies. bmj 341 doi: http://dx.doi.org/10.1136/bmj.c4587 (published 17 september 2010).   denzin, n. k. (19708). strategies of multiple triangulation. in n. k. denzin (ed.), the research act: a theoretical introduction to sociological methods. (pp.). chicago, il: aldine.     references   bryman, a. (2004). triangulation and measurement. retrieved 11 feb. 20134 from http://www.referenceworld.com/sage/socialscience/triangulation.pdf   flick, u. (2002). an introduction to qualitative research (2nd ed.). london: sage publications.   mertens, d. m. & hesse-biber, s. (2012). triangulation and mixed methods research: provocative positions. journal of mixed methods research 6(2), 75-79. doi:10.1177/1558689812437100   evidence based library and information practice evidence based library and information practice 2009, 4.4  evidence based library and information practice       evidence summary    library and informatics training may improve question formulation among public  health practitioners    a review of:  eldredge, jonathan d., richard carr, david broudy, and ronald e. voorhees. “the effect of training  on question formulation among public health practitioners: results from a randomized  controlled trial.” journal of the medical library association 96.4 (2008): 299‐309.     reviewed by:  heather ganshorn  librarian, health information network calgary  calgary, ab, canada  email: heather.ganshorn@ucalgary.ca     received: 31 august 2009        accepted: 14 november 2009      © 2009 ganshorn. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution,  and reproduction in any medium, provided the original work is properly cited.     objective – to determine whether providing  library and informatics training to public  health professionals would increase the  number and sophistication of work‐related  questions asked by these workers.     design – randomized controlled trial.    setting – new mexico department of health.    subjects – public health professionals from a  variety of professions, including  “administrators, disease prevention  specialists, epidemiologists, health educators,  nurses, nutritionists, physicians, program  directors, and social workers” (301). only staff  from the new mexico department of health  were eligible to participate.    methods – all subjects received a three‐hour  training session on finding evidence based  public health information, with a focus on  using pubmed. two sessions were offered,  two weeks apart. participants were  randomized to either an intervention group,  which received instruction on the first date, or  a control group, which received instruction on  the second date.     the intervening two weeks constitute the  study period, in which both groups were  surveyed by e‐mail about their work‐related  question generation. three times per week,  subjects received e‐mail reminders asking  them to submit survey responses regarding all  questions that had arisen in their practice,  along with information about their attempts to  answer them. questions were tallied, and  totals were compared between the two groups.     questions were also analysed for level of  sophistication, and classified by the  71 mailto:heather.ganshorn@ucalgary.ca http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2009, 4.4  investigators as either “background”  questions, which are asked when one has little  knowledge of the field, and can usually be  answered using textbooks or other reference  sources, or “foreground” questions, which are  often asked when an individual is familiar  with the subject, and looking for more  sophisticated information that is usually  found in journals and similar sources. this  scheme for classifying questions was  developed by richardson and mulrow.    main results – the investigators found  differences in both the number and  sophistication of the questions asked between  the control and intervention groups. the  control group averaged only 0.69 questions  per participant during the two‐week  observation period, while the intervention  group averaged 1.24 questions. investigators  also found that a higher percentage of the  questions asked by the intervention group  were foreground questions (50.0%, versus  42.9% for the control group). however, when  two‐tailed t‐test analysis was performed on  both the frequency of questions and the level  of sophistication, the findings were not  statistically significant within a 95%  confidence interval.     conclusion – this study suggests that library  and informatics training for public health  professionals may increase the number of  questions that they ask on work‐related topics,  and also the sophistication of these questions.  however, more studies need to be done to  confirm these findings. the authors suggest  that replication of the study would be useful,  particularly as hurricanes katrina and rita  interfered with their ability to recruit and  retain participants. they also suggest that  studies be conducted on other training  methods to see which are most effective at  motivating users to seek information. finally,  the authors suggest that a prospective cohort  study might be a useful method for predicting  the effect of training on participants’  motivation to pursue answers to their  questions.       commentary    this trial was unfortunately not powerful  enough to draw conclusions about the impact  of instruction on question formulation. the  authors had a much smaller number of study  subjects than they had hoped for, in part  because hurricanes katrina and rita caused  much of new mexico’s public health  workforce to be deployed to the gulf states to  provide emergency assistance. the authors’  goal was to recruit 130 participants; they were  only able to recruit 75, who were evenly  allocated between control and intervention  groups. a larger sample size would likely  have generated more conclusive results.    the two‐week window for data gathering was  arguably too short to demonstrate behaviour  change. a longer study period might have  allowed the authors to observe whether the  intervention had an effect over time.    the authors followed good practices in the  planning and carrying out of the trial. a web  service was used to randomize the subjects,  and the investigator responsible for coding the  data was blinded to the allocation of each  subject. efforts were also made to minimize  hawthorne effect (the possibility that the act  of observation may change subjects’  behaviour) by running a two‐week survey  period prior to the first training session, in  order to get the subjects accustomed to e‐mail  reminders and the need to submit their  questions. the instruction sessions were  carried out identically to the best of the  instructor’s ability.     the authors include the survey instrument  they used as an appendix to the article, which  is very useful, and something that more  authors should do as part of publication.  however, they do not indicate whether this  instrument was validated. the survey  instrument also asked whether or not subjects  attempted to find an answer; why they may  not have attempted to do so; and whether any  information‐seeking activity successfully  answered their question. however, the results  of these survey questions were not reported in  72 evidence based library and information practice 2009, 4.4  73 the article. although the stated objective of the  study was to look at question formulation, it  would have been interesting to also see  whether the instruction actually enabled  subjects to answer their own questions.    the content of the instruction session could  have been better described. the authors state  that it:   consisted of a three‐hour‐long  training session covering basic  ebph [evidence based public  health] principles, such as  definitions and types of ebph  questions, levels of evidence,  evaluations of both  information and statistics  websites relevant to public  health, pubmed training  tailored to public health  practitioners’ needs, and free  peer‐reviewed web‐based  journals. (303)     however, the authors do not mention whether  any pre‐assessment of participants’ skills and  needs was carried out to ensure that  instruction was tailored appropriately; nor do  they indicate whether the instructional format  was demonstration only, or whether there was  an  interactive element.  without such details,  it is difficult to replicate this study, or to  hypothesize about how the session’s content  and method of delivery may have influenced  the results. the need for improved reporting  of studies on information skills training has  been noted by brettle.    as the authors themselves note, the lack of  statistical significance in their findings makes  generalization of their results problematic, but  they suggest interesting directions for research  and practice nonetheless. their most  important suggestion is that anyone seeking to  replicate the study do so with a larger sample  size. i would also add that this type of clinical  trial could be carried out with other groups of  health‐care practitioners, and it would be  interesting to see if there are differences  between groups both at baseline and after an  instructional intervention. those looking to  conduct such a study should not only plan to  sample a larger group, but carry out multiple  follow‐ups  over a longer period of time.    this study suggests that training increases  information‐seeking behaviour. this appears  to validate the use of library instruction for  public health practitioners (and presumably  for other groups, as well). instructional  librarians should be conscious of this, and  market any reference services or point‐of‐care  tools that make it easier for practitioners to  obtain the answers they seek.      works cited    brettle, alison. “information skills training: a  systematic review of the literature.”  health information & libraries  journal 2.s1 (2003): 3‐9.    richardson w. scott, and cynthia d. mulrow.  “lifelong learning and evidence‐ based medicine for primary care.”  textbook of primary care medicine.  ed. john noble. st. louis, mo: mosby,  2001. 2–9.  evidence summary   interviews with practitioners in the united kingdom reveal effective strategies for open access outreach to researchers   a review of: dawson, d. (2018). effective practices and strategies for open access outreach: a qualitative study. journal of librarianship and scholarly communication, 6(1), ep2216. https://doi.org/10.7710/2162-3309.2216   reviewed by: brittany richardson web services librarian, assistant professor university of tennessee at chattanooga library chattanooga, tennessee, united states of america email: brittany-richardson01@utc.edu   received: 1 mar. 2020                                                               accepted:  28 apr. 2020      2020 richardson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29747     abstract   objective – to discover effective outreach methods used by academic libraries to promote open access (oa) publishing to researchers.   design – semi-structured interviews   setting – 7 large research universities in the united kingdom (uk)   subjects – 14 individuals responsible for oa outreach at their institution, including librarians and other oa practitioners   methods – purposive sampling was used to select universities based on their membership in the uk’s russell group, designation in the top 20 of the research council uk’s oa grant-size ranking, and suggestions from other professionals. the author contacted individuals responsible for oa at these institutions by email to inform them of the study and solicit their participation. the subsequent semi-structured interviews occurred in person. areas of focus in the interview included: job responsibilities and overview of offered scholarly communications services; sources of oa services at the institution; evolution and effectiveness of oa outreach activities; support and scholarly communication knowledge needed by researchers; and advice for fellow practitioners conducting oa outreach. interviews were recorded, transcribed, and coded using the qualitative software nvivo. inductive analysis was conducted to identify key themes.   main results – the author identifies four primary themes in the coded interviews: “the message”; “key contacts and relationships”; “qualities of the oa practitioner”; and “advocacy versus compliance” (p. 1). participants advocated for straightforward, frequent messaging tailored to the audience. the author identifies relationships as important to outreach – especially support from influential administrators and buy-in from key researchers – highlighting that face-to-face interaction is helpful when cultivating these types of relationships. participants emphasized important qualities for oa practitioners to possess, including expertise, diplomacy, and perseverance. establishing credibility as an expert was identified as important to generating buy-in from researchers. finally, the author discusses the library’s role in oa advocacy vs. compliance. some participants suggested an overemphasis on compliance to meet funder requirements may overshadow promotion of the inherent value of oa in academic publishing.   conclusions – the author suggests that because uk open access efforts are robust and have been in progress for many years, oa practitioners from the uk may possess useful insights for north american librarians with growing initiatives. the study highlights implications for practice including the identification of effective outreach strategies, evidence of the need for balanced messaging, and observations on why faculty may be resistant to outreach from librarians. the author recommends further research to determine what type of messaging is most valuable and when, suggesting less complex policies in north america may allow for more focus on the public good of oa. successful outreach is predicated upon having enough time to gain the necessary depth of knowledge, and the study acknowledges that librarians with diverse job responsibilities may have less time to invest in this way. the author also suggests more research is needed to evaluate the impact of the dynamics between librarians and classroom faculty. the article posits that librarians who participate in research and tenure processes may find communication easier and have opportunities to promote oa through participation in university governance. finally, the author observes that librarians are poised to have a positive impact on the scholarly publishing system through outreach to researchers who can drive systemic change.   commentary   studies related to library oa outreach often focus on lessons learned from efforts at specific institutions, mainly in the united states (otto, 2016; vandegrift & colvin, 2012). fruin (2017) conducted an environmental scan of oa initiatives in the united kingdom, including some exploration of outreach activities, although this was not the primary focus of the study. moving open access implementation forward provides some suggestions for advocacy based on the experiences of uk practitioners (blanchett & degroff, 2017). the author’s study expands on the literature by focusing specifically on outreach at multiple large uk universities, synthesizing these efforts into useful recommendations for practitioners.   the study demonstrates strengths when evaluated using the qualitative study critical review tool created by letts, et al. (2007). the author clearly articulates the context and value of the study, highlighting existing oa challenges, the maturity of efforts in the united kingdom, and trends in the current literature. additionally, the research focus is clearly stated, the qualitative research design is appropriate, the process for selecting participants is articulated, and findings are well-presented. useful recommendations for practitioners based on study findings are provided.   there are also several areas for study improvement. first, it is possible that interviewing multiple practitioners from the same institution led to less diverse results. furthermore, although the author states that the purpose of the study was not to assess individual oa practitioner characteristics, the inclusion of some demographic factors could have provided more nuance in the interpretation of results (e.g. faculty status, highest degree earned, years of experience, etc.). additional details on the coding process and inductive analysis used to identify themes would have further bolstered validity. information on the perspective and background of the researcher would have also provided greater context for results, because as braun and clarke (2006) note, “data are not coded in an epistemological vacuum” (p. 12). finally, an evaluation of potential study limitations would have been valuable.   the results of this study may be of interest to librarians and other practitioners involved in advocating for oa initiatives to researchers. librarians at academic institutions in countries where initiatives are still being developed and barriers to buy-in exist may find the study’s recommendations particularly applicable. further research inclusive of institutions of varying types, sizes, and locations could provide insight into outreach methods relevant to diverse contexts. survey research may be a complementary way to evaluate outreach efforts more broadly across many institutions.   references   blanchett, h., & degroff, h. (2017). moving open access implementation forward: a handbook for open access good practice based on experiences of u.k. higher education institutions. jisc: bristol, u.k. retrieved from http://repository.jisc.ac.uk/6565/1/jisc_oagp_outputs_handbook_final.pdf   braun, v., & clarke, v. (2006). using thematic analysis in psychology. qualitative research in psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa   fruin, c. (2017). organization and delivery of scholarly communication services by academic and research libraries in the united kingdom: observations from across the pond. journal of librarianship and scholarly communication, 5(1), ep2157. https://doi.org/10.7710/2162-3309.2157   letts, l., wilkins, s., law, m., stewart, d., bosch, j., & westmorland, m. (2007). critical review form – qualitative studies (version 2.0). retrieved from http://www.peelregion.ca/health/library/eidmtools/qualreview_version2_0.pdf   otto, j. j. (2016). a resonant message: aligning scholar values and open access objectives in oa policy outreach to faculty and graduate students. journal of librarianship and scholarly communication, 4, ep2152. https://doi.org/10.7710/2162-3309.2152   vandegrift, m., & colvin, g. (2012). relational communications: developing key connections. college & research libraries news, 73(7), 386–389.  https://doi.org/10.5860/crln.73.7.8790   writing about research—the good, the bad, and the challenging editorial   writing about research—the good, the bad, and the challenging   lisl zach, phd managing partner  information insights llc philadelphia, pennsylvania, united states of america email: lisl@marsez.com      2022 zach. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30098     as i reflect on my eight years as an associate editor for research articles, and look forward to my new role as associate editor for review articles, i mostly think about the many wonderful manuscripts that i have already had the privilege and the pleasure to read, review, and help shepherd through the publication process. many of these manuscripts addressed research questions that i never considered in my own time as an active researcher, and many have made contributions to the field by looking at questions from new perspectives and/or analyzing data through the lens of experiences very different from my own. to all of the authors with whom i have worked, as well as our wonderful peer reviewers and dedicated eblip team, i send my thanks and appreciation for their continuing efforts.   that said, i can’t help noticing changes, not always for the better, in the practice of research and especially in the craft of writing about research for publication. back in the last century, when i was a doctoral student, we were required to take courses that focused on how to conduct a research study and how to write a paper reporting its results. the process always began with a clearly articulated research question, the answer to which would fill an identifiable gap and contribute to the knowledge base of the field. at the end of the process, one wrote a report that showed what had (or had not) been accomplished to meet the goals set out for the study and demonstrated how the answer to the original research question could make an impact on the field going forward. the reports tended to follow the simple yet powerful rhetorical triptych, “tell ’em what you’re going to tell ’em; then tell ’em; then tell ’em what you told ’em” (and by the way, make sure that you have the appropriate evidence to support what you are telling them).   this is still a standard that we at eblip use when evaluating a research manuscript submission. however, we often find that good research is not always well reported, and our peer reviewers frequently make comments about the need for clarity in the description of the goals and theoretical framework of a study as well as in the conclusions that are drawn from the findings. in the worst examples, the conclusions drawn have little apparent connection with the stated goals of the study, or they are not supported by the evidence provided, however interesting that evidence may be in its own right. in most cases, however, the problems tend to be in the organization of the material and its presentation—perhaps reflecting a lack of attention to the writing stage of the research process. as i often told authors in my editorial decision letters, “the strength of evidence-based research rests largely on the rigor with which a study is conducted, and it is essential to provide readers with a clear description of the process so that they can evaluate the validity and generalizability of the findings.”   writing for publication requires more than an important research question or interesting data, it requires discipline when putting the story together and making the case for why the results of a given research study are valuable to the field. especially when addressing a question that has been asked before, the conclusions need to demonstrate how the findings from a particular research study can be used by others in their own situations and how they can be applied to influence practice and support decision making—important outcomes of evidence-based research. each element of the research article must relate to this goal, and the clarity of the presentation is essential to the success of the process. strong writing skills are a part of any good researcher’s tool kit and should be honed. the ability to communicate research findings in an interesting—even memorable—way increases the likelihood that the study will be regarded as a significant contribution to the knowledge base of the field. so i urge authors to take up the challenge of focusing as much attention on the writing stage of their research process as they do on the other stages and to think of their manuscripts as an opportunity to put their research findings in a broader context and demonstrate how they can make an impact on the field going forward.   feature   eblip7 conference award winners      2013. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   during the 7th international evidence based library and information practice conference (eblip7), delegates were encouraged to vote for their favourite paper and poster. congratulations to the following people’s choice award winners!   eblip7 people's choice awards best paper first place looking for infolit: using syllabi to map strategic information literacy instruction katherine boss, long island university, united states emily drabinski, long island university,  united states a two-way second place tie putting a sacred cow out to pasture: assessing the effect of removing fines and reducing barriers at an academic library kathleen reed, vancouver island university, canada jean blackburn, vancouver island university, canada daniel sifton, vancouver island university, canada eblip in a public library: assessment and research at edmonton public library soleil surette, edmonton public library, canada laura winton, edmonton public library, canada   eblip7 people's choice awards best poster first place determining the direction of an information literacy program: data analysis leads to the creation of online learning objects gail strachan, seneca college, canada kathryn klages, seneca college, canada maria bordignon, seneca college, canada second place culling the herd in hard times: implementing an evidence-based "big deal" cancellation support tool at vancouver island university jean blackburn, vancouver island university, canada kathleen reed, vancouver island university, canada dana mcfarland, vancouver island university, canada a three-way third place tie navigating uncharted territory: introducing eblip into a department laura newton-miller, carleton university, canada studying customers' use of the physical space at edmonton public library beth wortman, edmonton public library, canada   collaborating on demonstrating evidence of student learning in undergraduate literacy instruction nancy goebel, university of alberta, canada michelle edwards thomson, red deer college, canada jessica knoch, grant macewan university, canada sara sharun, mount royal university, canada   evidence based library and information practice evidence based library and information practice 2011, 6.1 92 evidence based library and information practice using evidence in practice determining user needs for an academic health sciences library renovation susan powelson director health sciences library, university of calgary calgary, alberta, canada email: susan.powelson@ucalgary.ca marcus vaska librarian health information network, university of calgary calgary, alberta, canada email: mmvaska@ucalgary.ca received: 08 nov. 2010 accepted: 16 dec. 2010 2011 powelson and vaska. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/bync-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. setting the health sciences library (hsl) at the university of calgary (u of c) opened in the late 1970’s. the library serves the faculties of medicine, nursing, veterinary medicine, and a bachelor of health sciences program. in addition, hsl provides services to the foothills medical centre, the largest acute care hospital in calgary. although some minor renovations had been done at hsl in the past, including new carpet and relocation of the administration offices, the library remained relatively unchanged for more than 30 years. hsl formerly occupied 2600 square metres on the second floor of the university of calgary health sciences centre. reference and circulation operated from two separate service desks, and there was a large area for the display of periodicals, a few easy chairs, some study tables, and more than 100 carrels. there were only two small group rooms and three secure or lockable carrels. despite taking up a substantial portion of the library, the stacks remained crowded. the university of calgary library system was moving to an online preferred collection model. as a result, print journals were being cancelled. a renovation and collection rationalization was needed. problem in 2008, the faculty of medicine found it necessary to increase the enrolment of undergraduate medical students from 150 to 180. to meet accreditation standards, the faculty had to add two 50-seat classrooms, six mailto:susan.powelson@ucalgary.ca� mailto:marcus.vaska@ucalgary.ca� evidence based library and information practice 2011, 6.1 93 small group rooms, and a 250-seat lecture theatre. the only possible expansion space was in the library. the u of c provost and the dean of medicine reached an agreement in september 2008 for a total library renovation, giving the faculty approximately 25% of the existing library square footage to create spaces for joint use. complete renovation of the remaining health sciences library space would better meet current student needs. timelines were tight, as the faculty required the two classrooms and six small group rooms available for instructional purposes in less than a year. library staff had neither the time for a formal library needs consultation process, nor the creation of focus groups with faculty, students, and foothills medical centre staff. however, staff needed to determine what users wanted to see in a new library space. evidence a literature review revealed that library spaces need to be flexible, accessible, and attractive, with noisy and quiet areas carefully planned and distinct (fitzgerald, 2004; nelson, 2003). natural light should be maximized and fluorescent bulbs replaced with full spectrum lights. attractive and comfortable areas should be included (fitzgerald, 2004; twait, 2009). in addition, libraries can facilitate collaborative learning by planning for group study rooms as well as secure individual study rooms. study tables must be wired (marcus, covert-vail, & mandel, 2007). the literature provided excellent advice, but hsl staff wished to determine if u of c users had any unique or different needs. they wanted to know how much and what type of seating was required, as well as identify issues and irritants with the current space to ensure that the same problems were not recreated upon renovation completion. two hsl librarians created a brief survey. they chose surveymonkey™ because it is electronic, cumulates results automatically, is fast and easy to use, and it was a product they were most familiar with. although some provinces and institutions face surveymonkey™-related constraints, this is not a problem at the university of calgary. staff expected a faster and wider distribution with an electronic tool than with a paper questionnaire. they wanted to keep the survey brief and focused. the survey included five open-ended questions, asking respondents about their preferences for the new library, their use of information resources, and their sources of information. three additional demographic questions and two questions not specifically related to the library redesign completed the survey. staff posted the survey on the hsl web page, and emailed the survey link to faculty, student, and staff email lists. the survey was open from february 12 to 25, 2009. within the first 24 hours, 136 participants completed the survey with an eventual total of 170 survey responses. the majority (83.8%) of the responses were from students. their preferences included carrels at the back of the library for quiet study, a desperate need for outlets to plug in their laptops, more technology, better furnishings, and a more comfortable and attractive library. since 17% of those surveyed stated that they only browsed the collection, and students claimed they turned to google rather than the library catalogue, staff confirmed the decision to move a significant portion of the collection to high density storage (vaska, chan, & powelson, 2009). implementation the survey was revealing, valuable, and resulted in changes to the renovation plans. new plans ensured an electrical outlet for every seat in the library. a self-check machine will be installed in the near future. staff revised plans to eliminate most of the carrels and replace them with tables; however, more carrels were added in the back by the windows overlooking the atrium, replacing previous study tables. the new carrels are wired and have individual study lights. all study chairs are ergonomic and adjustable. there will be twenty comfortable chairs, evidence based library and information practice 2011, 6.1 94 placed in groups of four around coffee tables, and eight rolling footstools. these changes ensured that specific needs were addressed and that library patrons would be happy with the end result. outcome the health sciences library renovation will not be complete until early 2011. library users face a steady stream of construction traffic, several areas within the library that remain cordoned off, and temporary entrance doors which still need to be replaced. however, some of the new carrels and study tables are already in use, three of the six new group rooms are functional, as are the four secure carrels. despite the construction, user response has been overwhelmingly positive. carrels and small group rooms are heavily used, and faculty approach library staff both in and out of the library to compliment the changes. reflection although quickly designed and implemented, our survey demonstrates that evidence for decisions can be readily obtained and easily used. the survey results supported the findings from the literature; namely, that faculty, staff, and students desire attractive spaces with comfortable seating, windows, better lighting, and areas for both collaborative and quiet work. by enabling users to comment on their preferences, planners could incorporate their unique needs into the design. although hsl staff would have preferred to conduct a rigorous and detailed survey, followed up by focus groups and individual consultations, important input was nonetheless gained within the time constraints. obtaining evidence is not difficult, provided one is willing to be flexible and creative. evidence ensures better, more appropriate decision making even when it cannot be gathered using a rigorous, structured protocol. references fitzgerald, d. (2004). health sciences library renovation plans: update for the health sciences library users committee. hamilton, ontario: mcmaster university. marcus, c., covert-vail, l., & mandel, c. a. (2007). nyu 21st century library project: designing a research library of the future for new york university. new york: new york university libraries. retrieved october 28, 2010 from http://library.nyu.edu/about/kplreport. pdf nelson, p. p. (2003). current issues in the design of academic health sciences libraries: findings from three recent facility projects. journal of the medical library association, 91(3), 347-351. twait, m. (2009). if they build it, they will come: a student-designed library. college and research libraries news, 70(1), 21-24. vaska, m., chan, r., & powelson, s. (2009). results of a user survey to determine needs for a health sciences library renovation. new review of academic librarianship, 15(2), 219-234. http://library.nyu.edu/about/kplreport.pdf� http://library.nyu.edu/about/kplreport.pdf� / evidence based library and information practice evidence based library and information practice 2011, 6.3 1 evidence based library and information practice evidence summary implementing web 2.0 design patterns in an institutional repository may increase community participation a review of: cocciolo, a. (2010). can web 2.0 enhance community participation in an institutional repository? the case of pocketknowledge at teachers college, columbia university. the journal of academic librarianship, 36(4), 304–312. doi:10.1016/j.acalib.2010.05.004 reviewed by: yvonne hultman özek senior librarian faculty of medicine, lund university lund, sweden email: yvonne.hultman_ozek@med.lu.se received: 24 feb. 2011 accepted: 15 july 2011 © 2011 özek. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to investigate whether web 2.0 can enhance participation in institutional repositories (irs) and whether its widespread use can lead to success in this context. another purpose was to emphasize how an ir with a web 2.0 approach can connect individuals in their creative and intellectual outputs, no matter what form of shared material is contributed. design – comparative study. setting –two irs at teachers college, columbia university, which is a graduate and professional school of education in new york city. subjects – students, faculty, and staff using the pocketknowledge and cpc irs. methods – cocciolo compared two different irs called pocketknowledge and community program collections (cpc). pocketknowledge had the following web 2.0 design patterns: users control their own data; users should be trusted; flexible tags are preferred over hierarchical taxonomies; the attitude should be playful; mailto:yvonne.hultman_ozek@med.lu.se� evidence based library and information practice 2011, 6.3 75 software gets better the more people use it. the pocketknowledge ir design patterns were compared with the traditional design of the cpc ir. the crc ir organized information based on taxonomy (e.g., programs and departments), lack of user control of their own content, and centrality of authority. data were collected during a 22‐month period. the pocketknowledge ir was studied from september 2006 to july 2008, compiling information on both contributions and contributors. contributions made by library staff to aid availability in archival collections were excluded from the data sets, because the study was focused on community participation in the learning environment. the cpc was studied between november 2004 and july 2006. data collected included the contributions made to the system and information on the role of the contributor (e.g., student, faculty, or staff). main results – participation was much greater in the web 2.0 system (pocketknowledge) than in the non‐web 2.0 system (cpc). involvement in the latter, the cpc, was noted primarily for faculty (59%), with a smaller proportion of students (11%) contributing. this trend was reversed with the web 2.0 system, in which 79% of the contributions came from students. however, as a group, faculty were better represented than the student body as contributors to the web 2.0 system (23% and 8% respectively). faculty members who created an account (without contributing) represented 30% of the population. these observations suggest that web 2.0 is attractive to students as a space to share their intellectual creations, and at the same time it does not alienate the faculty. notwithstanding, although 31% of the student body had created a user account for pocketknowledge, the web 2.0 system, only 8% of the students actually contributed to this ir. the study examined only the participation rates and was not concerned with what motivated contributions to pocketknowledge. accordingly, the results can be extrapolated by observing that the limitation of previous irs is that they focused primarily on the library goals of collecting and preserving scholarly work, and did not consider what prompted faculty to contribute. despite the satisfactory participation in the two irs of interest, the author argued that the incentive is associated more extensively with the role as teacher than with the role as researcher. this is related to the ambition of faculty to improve classroom‐based experience by ensuring that their students are as engaged as possible in the teachers’ areas of expertise. in other words, a faculty contribution is motivated by knowing that students will become familiar with what is contributed. conclusion – this study suggests that irs can achieve greater participation by shifting the focus from the library goals to the objective of building localized teaching and learning communities by connecting individuals through their respective intellectual outputs. creation of a system like the cpc that supports such exchange will advance library goals by storing faculty’s scholarly work, whereas web 2.0 offers a set of approaches and design patterns for establishing systems that help promote community participation. greater student participation in an ir may prompt increased faculty participation, because the ir will be more extensively focused on the teaching and learning community than on the research community. thus, the major finding of the study is that greater community participation resulted from a web 2.0 design pattern approach. commentary the study highlighted different strategies to motivate community participation in irs. in particular, it emphasized how student participation (not contribution) can be increased by using the web 2.0 approach in the pocketknowledge ir. it would have been more suitable to compare the results if the two irs had been investigated during the same time period. the phenomenon of web 2.0 started evidence based library and information practice 2011, 6.3 76 around 2006, but gathering of the data on the “traditional” cpc ir was begun in 2004. one could speculate that the launch of the web 2.0 ir pocketknowledge was timely with the web 2.0 hype and therefore attracted special attention. what complicates matters even more is that contributing to and participating in an ir are not the same thing. in table 1 of the article, the student contributions appear to have been more or less the same, regardless of whether pocketknowledge or cpc was considered (11% and 8%, respectively, but the number of contributions was much greater in pocketknowledge). however, creating an account on (not contributing to) the former ir was noted for as many as 31% of the entire student population. it was not indicated whether the individuals who created an account were active and actually participated in the teaching and learning community, which explains why the proportions of contributions were essentially the same for the two irs. another limitation of the investigation was that an experimental design was not used. another issue is the quality of the contributions from students. the author did not define what types of contributions were made to the web 2.0 ir. if a study compares faculty members who deposit work in an ir, the objective is probably to expose the contributions that have been through the peer‐review process, thus indicating a certain degree of quality. however, it is unclear what type of contributions was made by the faculty, which is also the weakness of the method. one could speculate that adapting the web 2.0 concept for researchers depositing their peer‐reviewed published articles in an ir may not be as urgent as for the teaching and learning environment. by mentioning that further research is needed to ascertain whether community trust in an ir can replace regulations, the author implied that the issue of quality can be developed. this observation might be strongly associated with the potentially questionable quality of the contributions from students. as already mentioned, there was no clear differentiation regarding the type of contributions made by the faculty in their role as researchers as compared to those made in their role as teachers. in sum, the comparison of these two irs have diffuse comparable parameters and not entirely comparable. for example, purpose of the repositories, the marketing of the repositories and the type of support that was available are not clarified. the question of whether these parameters would have had an impact on participation rates remains. finally, it should be noted that the majority of the publications in the reference list discussed what motivates researchers’ behaviour in relation to irs, whereas only a few dealt with students’ behaviour and the use of irs in the learning environment. clearly, it would have been valuable if the author had included more investigations about students and their contribution of intellectual work to an ir learning environment or a learning management system in order to support the stated objectives and conclusions. it also would have been helpful if the author had stated in the introduction the definition of the term “institutional repository” in relation to the study. nevertheless, the study has captured some user activity in the “real world” of two different irs, and the results may indicate new directions for increased community participation in the learning environment. / evidence based library and information practice evidence based library and information practice evidence based library and information practice 2010, 5.3 20 evidence based library and information practice article learning in simulations: examining the effectiveness of information literacy instruction using middle school students’ portfolio products terrance s. newell assistant professor university of wisconsin-milwaukee school of information studies milwaukee, wisconsin, united states of america email: newellt@uwm.edu received: 30 mar. 2010 accepted: 13 aug. 2010 2010 newell. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective –this study compared the effectiveness of simulation-based and didactic instructional approaches in improving students’ understanding of information literacy (il) concepts and practices. methods – the instructional approaches were implemented with two groups of middle school students (i.e., seventh and eighth grades) over a 4-week period. during the implementation period, all students were required to maintain a portfolio of their work. the portfolios were designed to capture students’ actions as they engaged in a common set of information-based problems. the contents of the portfolios were analyzed to examine the research questions that guided the study. contingency tables demonstrated observed patterns of difference from week 1 to week 4. chi-square analysis helped to determine whether a significant relationship existed between instructional approach and shifts in il proficiency levels at the .05 level. results – there was a significant relationship between the simulation-based approach and increases in students’ ability to 1) recognize the need for information, 2) formulate specific questions that would help in finding needed information, 3) identify a range of information sources for meeting needs, 4) explain successful strategies for accessing needed information, 5) judge the accuracy, relevance and completeness of sources and 6) analyze information from a variety of sources to determine its applicability to a specific problem. four major distinctions are believed to have caused the students mailto:newellt@uwm.edu evidence based library and information practice 2010, 5.3 21 within the simulated instructional environment to experience more proficiency level shifts: situated practice, authenticity, community of practice and an expanded landscape of resources. conclusion – the results of this study suggest that simulation-based instructional approaches have the potential to augment il learning. the technology-based approaches may provide powerful learning environments (virtual worlds) that allow students to engage in the activities and practice of information specialists, instead of simply learning the facts associated with the discipline. introduction educational scholars are only beginning to understand the benefits and pitfalls of learning via simulated learning encounters. scholars are currently conducting research, building prototypes, and developing theoretical frameworks to assist educators in understanding how students learn within these technology-based environments and how effective the environments are within particular disciplinary contexts (e.g., history). the term digital simulation refers to a computer-based, problem solving situation with multiple interacting variables and a landscape of information resources (gredler, 2004). there are various types of simulations (e.g., social process, system, diagnostic, data management and laboratory). however, all simulations have four basic characteristics: 1) learners are placed within an environment that mirrors a complex real-world situation; 2) roles are assigned to learners; 3) an expanded information landscape is represented; and 4) feedback mechanisms are employed. the research literature indicates a potential in the role that simulations could play in mediating learning, and as a vital part of the k-12 instructional mission, u.s. school libraries are beginning to explore the potential role of digital simulations. it is imperative that school librarians gain evidence based insights into the effectiveness of simulation-based approaches in teaching information-oriented concepts and practices. literature review over the last three decades, the use of digital simulations has increased within the general educational landscape. gredler (2004) illuminated five dominant types of simulations tied to educational environments: social process, diagnostic, data management, system, and laboratory simulations. a social process simulation is a complex, disciplinespecific situation that provides learners with a finite field of contingencies for the occurrences inscribed within the simulation. a diagnostic simulation presents learners with realistic problem scenarios and requires them to make optimal, information-based decisions at sequential points. data-management simulations are collaborative exercises that require learners to manage financial variables under varying conditions. laboratory simulations place learners within disciplinespecific, virtual worlds, and learners then fill the roles of researchers within the simulations. system simulations also place learners within discipline-specific, virtual worlds. in system simulations learners play the role of problemsolvers. theoretical impetus for the use of digital simulations the use of simulations within the educational landscape is directly tied to ecological (e.g., gibson, 1986; shaw & bransford, 1977), constructivist (e.g., dewey, 1933; inhelder & piaget, 1958; kelly, 1963; bruner, 1977) and situative perspectives of learning (e.g., brown et al., 1989; kirshner & whitson, 1998). “ecological perspectives” of learning illuminate the role that contextually sensitive activity (e.g., interacting with visual, auditory, and tactile data in the environment) plays during the learning process (e.g., gibson, 1986). “constructivist perspectives” of evidence based library and information practice 2010, 5.3 22 learning advance the notion that learning is an active process of personal meaning-making, a concept grounded in experience (dewey, 1916). “situative perspectives view learning as a process of 1) active meaning-making within a student’s mind and 2) enculturation into the disciplinary practices of literate people within a particular domain of study (cobb, 1996). these three perspectives situate learning (or cognitive functioning) within environmental and experiential frameworks, which create a theoretical grounding for the use of digital simulations. virtual reality as a dominant technology for simulation design simulations can be brought to life using a variety of technologies. schools serving students from kindergarten through twelfth grade are primarily exploring the use of desktop virtual reality (vr). desktop vr is a relatively inexpensive computer-based technology that allows learners to examine three-dimensional, highly interactive, multisensory, simulated environments using a computer screen and a keyboard (mclellan, 2004; focier, 1999). educational settings are currently using vr in various ways: 1) feedback on student performance, 2) training systems, 3) experiential learning spaces, 4) virtual field trips and 5) rehabilitation spaces (mclellan, 2004). a number of vr environments have been employed within educational settings such as the virtual european schools project (bouras, fotakis, kapoulas, koubek, mayher & rehatscheck, 1999), intelligent distributed virtual training environment (bouras, triantafillou & tsiatsos, 2001), educational virtual environments project (bouras, giannaka & tsiatsos, 2003), quest atlantis (barab, thomas, dodge, carteaux & tuzun, 2005), active worlds educational universe (corbit, 2002), second life (andreas, tsiatsos, terzidou & pomportsis, 2010), and c-visions (san chee & meng hooi, 2002). effectiveness of simulations in educational environments empirical research studies show that simulation approaches are just as effective as traditional teaching approaches in teaching basic math and reading comprehension skills (rosas et al., 2003; laffey et al., 2003), mathematical problem solving skills (van eck & dempsey, 2002), basic logic (costabile et al., 2003), geographical content knowledge (wiebe & martin, 1994; virvou, katsionis & manos, 2005) and vocabulary skills (malouf, 1988). the simulation and learning literature has also illuminated particular advantages associated with these learning environments, such as the development of critical thinking skills (rieber, 1996), problem solving abilities (rieber, 1996; gorriz & mediana, 2000; prensky, 2001), visual/spatial skills (greenfield et al., 1994), cognitive strategies (gredler, 1996), discovery learning (prensky, 2001), interactivity (rosas et al., 2003; price, 1990) and motor skill growth. the research literature indicates the potential role that simulations may play in mediating learning. this article examines the persuasive, academic position that constructs digital simulations as dynamic educational spaces that could facilitate learning (shaffer, squire, halverson & gee, 2005). it also contributes to the current empirical record on learning via simulations within both the general educational landscape and the school library context by providing evidence into the effectiveness of these approaches in improving students’ understanding of il concepts and practices. aims this study compared the effectiveness of simulation-based and didactic instructional approaches in improving students’ il proficiency levels by examining the following questions: 1. after a 4 week course, does a significant relationship exist between instructional approach (simulation and didactic) and the number of students who were able to achieve a higher proficiency level in: evidence based library and information practice 2010, 5.3 23 a. recognizing the need for information b. formulating questions c. identifying a variety of potential information sources d. developing and using location strategies e. determining the accuracy, relevance, and comprehensiveness of information f. selecting appropriate information 2. what are the patterns of difference between the instructional approaches and number of shifts across six information-oriented variables (i.e., needs recognition, question formulation, identifying information sources, information location, information evaluation, and resource selection)? methods research site this study was conducted in a middle school (i.e., a school serving grades 7 and 8) in a u.s. midwestern city of approximately 200,000 residents. the school’s educational philosophy was based on two guiding principles: curricular integration and the use of technology to support learning. the school had three computer labs, a library computer center, and fully networked classroom computers. the school’s student-to-computer ratio was 4 to 1, with a total enrollment of 240 students. sample a total of 54 students recruited from the school’s four computer literacy classes participated in this study. the didactic group was composed of 27 students (12 seventh grade and 15 eighth grade students) and 27 students (11 seventh grade and 16 eighth grade students) in the simulation group. controlling for confounds there are, potentially, many confounding variables (e.g., teacher qualities and educational backgrounds) in a study attempting to compare instructional approaches. two primary strategies were used in an attempt to control for confounding variables and to minimize their role in observed effects: a) using random assignments, and b) keeping educators and instructional tasks constant for both instructional approaches. students within the research site were unsystematically assigned— except for grade level—to the classes participating in this study. there were various learning styles within each class -computer skill levels, genders, races, and intelligences. the four computer literacy classes were given unique numbers and a random numbers table was used to assign classes to instructional approaches. random assignment to learning models increased the probability that differences were due to the intended variables. in an attempt to keep some factors the same for every group, only one teacher and one school librarian were used for the study. therefore, any observed differences could not be attributed to differences in educator experience, personality, or educational background. furthermore, although the learning approaches differed for the two groups, each used the same informationoriented learning problems, tasks, and problem solving strategies. students engaged a common set of problems and thinking strategies for information-based problem solving. development of learning tasks both instructional approaches used the same information-oriented learning problems/tasks. the initial development of all informationoriented, learning tasks was performed by the researcher (figure 1). a variety of information literacy and information problem solving textbooks were analyzed and used in the construction of tasks. however, the school librarian and the technology teacher at the evidence based library and information practice 2010, 5.3 24 research site reviewed and added tasks prior to the beginning of the study. approaches to instruction simulation-based approach a three-dimensional (3d) simulation was used as one of the instructional approaches (figure 2). the 3d interface of the simulation was developed using authorware similar to the design tools utilized in second life (secondlife.com), the online virtual world. the interface was highly interactive and allowed students to navigate the 3d environment from every perspective using the computer screen, mouse, and keyboard. the simulated environment consisted of a middle school library (figure 3), high school library (figure 4), informal environments (see figure 5) and electronic environments (figure 6). the various parts of the simulated enwere designed to represent a small town. students could virtually walk from the middle school library to the high school library within the 3d simulation. the 3d simulation technology also enabled the construction of virtual information objects, artifacts, and resources (e.g., books, computers, televisions and people). within the simulated environment, students could improve freely, communicate using chat, and use a variety of information resources. within the simulation, students learned information-oriented practices through simulated experiences, interactions, and communities of practices. moreover, students learned by actively participating as information literacy (il) apprentices within a computer-generated community of information professionals. as il apprentices, students assisted the simulated information professionals in meeting the needs of other computer-generated characters within the simulation (i.e., people within the small simulated town) using novice informationoriented practices (wenger, 1998). through participation students gradually developed information-seeking skills. moreover, student learning was guided using three techniques: scaffolds, communities of practice, and cognitive process frameworks. instructional scaffolding aided students as they learned through active participation. , these learning supports (scaffolds) existed in the form of tutorials (gee, 2007), information on-demand (gee, 2007), just-in-time pop-ups (gee, 2007), modeling (collins et al., 1989), exploration (collins et al., 1989), and questioning (gallimore & tharp, 1990). a community of practice (cop) also aided the development of practices. information professionals—a group that is bounded by best practice approaches to information problem solving—constituted the community of practice. within the 3d simulation, students could work with members of this cop (e.g., computer-generated librarians, real librarians, and other student apprentices) to solve problems. third, as students engaged tasks, they used a cognitive process framework to cognitively structure the stages of information problem solving and the thinking strategies related to the different stages. fig. 1. sample portfolio task. thanks for your help! i am jaime. i am thinking about buying a vehicle like this one, but i don’t know how to ask the right questions or how to find the answers to those questions. i think that i need information related to the price, warranty and safety. i may also need information related to consumer and expert reports. could you help me answer these questions and find comprehensive visual information on the vehicle? could you also develop and answer four additional questions for me? evidence based library and information practice 2010, 5.3 25 fig. 2. 3d simulation. fig. 6. simulated accessing an electronic resource. didactic approach the second instructional approach was primarily a didactic format of teaching incorporating problem-based activities for students to solve. the didactic teaching context consisted of a real-world computer lab, middle school library, and the information objects within them (e.g., library catalogs, books, and computers). the middle school library and computer lab were connected, and the computer lab housed enough computers for each student to have a personal computer during the learning periods. during instruction, the technology teacher and the school librarian supported student learning using lectures and problem-based activities. students engaged in the problem-based activities in pairs, and used a generic, processoriented framework to guide them through the stages of information problem solving and thinking strategies related to various stages. fig. 3. simulated middle school library. fig. 4. simulated high school library. fig. 5. simulated informal information environment. evidence based library and information practice 2010, 5.3 26 the framework was the same as the one used in the simulation approach. assessment of student portfolios although the learning approaches differed for the two groups, both learning environments used the same information-oriented learning tasks. therefore, portfolios could be used to capture students’ activities as they engaged this common set of problems. the portfolios were not simply folders filled with work samples after the 4-week implementation period. instead, the portfolios facilitated the collection of students’ information problem solving activities across seven areas (i.e., task identification, search strategy initiation, information access, information evaluation, information use, information communication, and problem solving evaluation). the portfolio document was printed for students before the instructional periods began. the documents had 10 sections, representing the 10 required tasks for the 4-week period. as students addressed each of the tasks, they were required to record their information problem solving activities in designated parts of the portfolio. with the exceptions of the first, second, and last tasks, the tasks could be completed in any order. the second and last tasks were predetermined for content analysis purposes. data analysis a content analysis performed on week 1 and week 4 portfolio products examined the seven research questions guiding the study. content analysis is an empirical procedure for quantifying a field of representation, and the procedures of content analysis provided the researcher with a framework for classifying and quantifying the student portfolios as content. the empirical procedures also allowed the researcher to make statements about aspects of representation (e.g., shifts in information evaluation abilities) that are based on observable evidence. variables and values a content analysis classifies selected texts along specified dimensions known as variables and values. variables refer to dimensions of representation that are of the same type. values are mutually exclusive and there are exhaustive classification options for variables. for example, a school library (variable) could be classified as: elementary school (value 1), middle school (value 2), high school (value 3), alternative school (value 4), charter school (value 5), or private school (value 6). for the purposes of this study, the identification of variables and values was based on the standards for information literacy learning (aasl & aect, 1998). standards for information literacy learning advances three il standards (i.e., access, evaluation, and use standards), and thirteen demonstration indicators of information literacy across the standards (aasl & aect, 1998). the k-12 standards also provide three levels of proficiency (i.e., basic, proficient, and exemplary) for each demonstration indicator. using the standards for information literacy learning (aasl & aect, 1998), the variables were constructed using the content area indicators, and values were defined using the levels of proficiency tied to each indicator. variables and values were placed in a coding protocol to help guide the coders in applying consistent criteria as they assessed the portfolios. the variables and values are listed in appendix a. comparison procedures and reliability the researcher and another university level professor served as coders for this content analysis following a typical content analysis procedure (leedy & ormrod, 2001). first, the coders practiced using the coding protocol on five student work samples that were similar to—but not part of—the official field of bound-texts to refine clues, signifiers and definitions of variables/values. second, interrater agreement of the coded work samples was assessed. inter-rater agreement was initially 92 percent; however, a discussion of disagreements and further training resulted in evidence based library and information practice 2010, 5.3 27 a final inter-rater agreement of 100 percent. third, the coders used the coding protocol to consistently evaluate student portfolios along the identified dimensions (variable and values). each coder evaluated half of the didactic portfolios and half of the simulation portfolios. each portfolio was coded separately, and coders concealed student names during coding to minimize bias. fourth, after coding all portfolios, the coders produced a numerical set of results. contingency tables presented observed patterns of difference from week 1 to week 4. chi-square analysis determined whether a significant relationship existed between instructional approach and shifts in il proficiency levels at the .05 level. results the results of the content analysis follow. first, are the results from week 1. second are the results from the comparison of week 1 portfolios to week 4 portfolios according to the six variables that guided this study. the 27 didactic student portfolios from week 1 were compared to the same 27 portfolios from week 4 (the end of the course) to identify whether proficiency had changed across the six variables that guided this study. the 27 simulation portfolios from week 1 were also reexamined at the end of the course to evaluate changes in proficiency across the six variables. week 1 the content analysis clearly showed six major findings during the first week of implementation that demonstrated effectiveness (table 1). first, 33% of the didactic students and 37% of the simulation students did not consider whether additional information—beyond their own knowledge— was needed during the problem solving process. second, 93% of the didactic students and 100% of the simulation students could form only one broad question (or none at all) to guide them through the problem solving process. third, 74% of the didactic students and 93% of the simulation students could generate only two sources of information that could potentially be used to answer their question(s). fourth, 93% of the didactic students and 100% of the simulation students could either 1) not list any ideas for identifying and finding needed information, or 2) list only vague ideas about their potential strategies for locating information. fifth, 96% of the didactic students and 100% of the simulation students sparingly considered the concepts of accuracy, relevance, and comprehensiveness—if at all. sixth, 86% of the didactic students and 96% of the simulation students either did not identify information to address the problem or used only one or two information sources to address the problem. evidence based library and information practice 2010, 5.3 28 table 1 week 1 didactic (n=27) and simulation (n=27) groups didactic group: number of students across proficiency levels not represented basic proficient exemplary variables variable 1. recognizing need for information 9 (33%) 16 (59%) 2 (7%) variable 2. formulating questions based on information needs 4 (15%) 21 (78%) 2 (7%) variable 3. identifying a variety of potential sources of information 5 (18%) 15 (56%) 7 (26%) variable 4. developing and using successful strategies for locating information 10 (37%) 15 (56%) 2 (7%) variable 5. determining the accuracy, relevance, and comprehensiveness of information 21 (78%) 5 (18%) 1 (4%) variable 6. selecting information appropriate to the problem or the question at hand 13 (49%) 10 (37%) 4 (15%) simulation group: number of students across proficiency levels not represented basic proficient exemplary variables variable 1. recognizing need for information 10 (37%) 17 (63%) variable 2. formulating questions based on information needs 4 (15%) 23 (85%) variable 3. identifying a variety of potential sources of information 8 (30%) 17 (63%) 2 (7%) variable 4. developing and using successful strategies for locating information 15 (56%) 12 (45%) variable 5. determining the accuracy, relevance, and comprehensiveness of information 25 (93%) 2 (7%) variable 6. selecting information appropriate to the problem or the question at hand 17 (63%) 9 (33%) 1 (4%) recognizing the need for information in the didactic group, the proficiency levels of 6 students experienced a positive change, whereas, the proficiency levels of 19 students experienced a positive change in the simulation group (table 2). a chi-square test was performed to determine if a significant relationship existed between instructional approach and the number of students who were able to recognize the need for information at a higher proficiency level. this indicated a significant relationship for the simulation approach in improving proficiency levels at recognizing a need for information, [x2 (2, n=54) = 12.58, p < .05], but not the didactic approach (table 2). evidence based library and information practice 2010, 5.3 29 formulating questions based on information needs in the didactic group, the proficiency levels of 2 students experienced a positive change and the proficiency levels of 26 students experienced a positive change in the simulation group, (table 3). a chi-square test indicated that there was a significant relationship between the simulation approach and the ability to improve students’ proficiency levels in formulating questions [x2 (2, n=54) = 42.725, p < .05] but not the didactic approach (table 3). identifying a variety of potential sources of information proficiency levels of 6 students experienced a positive change in the didactic group and 26 students experienced a positive change in the simulation group (table 4). a chi-square test indicated this was a significant relationship for the simulation approach [x2 (2, n=54) = 30.682, p < .05] but not the didactic approach and the ability to improve proficiency levels in identifying sources of information (table 4). developing and using successful strategies for locating information in the didactic group, the proficiency levels of 4 students experienced a positive change whilst the proficiency levels of 22 students experienced a positive change in the simulation group (table 5). a chi-square test indicated that this was a significant change in the simulation group [x2 (2, n=54) = 24.033, p < .05] (table 5). table 2 recognizing the need for information: proficiency level improvements between weeks 1 and 4 variable 1: recognizing the need for information didactic group 6 didactic students improved from a basic to a proficient level 21 didactic students retained their week 1 proficiency levels simulation group 8 simulation students improved from a basic to a proficient level 11 simulation students improved from proficient to exemplary 8 simulation students retained their week 1 proficiency levels table 3 formulating: proficiency level improvements between weeks 1 and 4 variable 2: formulating questions based on information needs didactic group 2 didactic students improved from a “not represented” proficiency to a basic level 1 didactic student regressed from a proficient level to a basic level 24 didactic students retained their week 1 proficiency levels simulation group 3 simulation students improved from “not represented” to a basic level 8 simulation students improved from a basic to a proficient level 15 simulation students improved from a basic to an exemplary level 1 student retained her week 1 proficiency level evidence based library and information practice 2010, 5.3 30 table 5 locating information: proficiency level improvements between weeks 1 and 4 variable 4: developing and using successful strategies for locating information didactic group 4 didactic students improved from a “not represented” proficiency to a basic level 1 didactic student regressed from a proficient to a basic level 22 didactic students retained their week 1 proficiency levels simulation group 1 simulation student improved from a “not represented” proficiency to a basic level 11 simulation students improved from “not represented” to a proficient level 1 simulation student improved from a basic to a proficient level 9 simulation students improved from a basic to an exemplary level 5 simulation students retained their week 1 proficiency levels table 6 accuracy, relevance and comprehensiveness: proficiency level improvements between weeks 1 and 4 variable 5: determining the accuracy, relevance and comprehensiveness of information didactic group 6 didactic students improved from a “not represented” proficiency to a basic level 1 didactic student regressed from a proficient to a basic level 20 didactic students retained their week 1 proficiency levels simulation group 12 simulation students improved from a “not represented” proficiency to a basic level 5 simulation students improved from “not represented” to a proficient level 10 simulation students retained their week 1 proficiency levels determining the accuracy, relevance and comprehensiveness of information the proficiency levels of 6 students in the didactic group and 17 students in the simulation group experienced a positive change (table 6). a chi-square test indicated that this was a significant improvement in the simulation group [x2 (2, n=54) = 9.164, p < .05] for determining the accuracy, relevance and comprehensiveness of information (table 6). selecting information appropriate for the problem the proficiency levels of 8 students in the didactic group and 20 students in the simulation group experienced a positive change (table 7). a chi-square test determined a significant relationship for the simulation approach and the number of students who were able to select appropriate information at a higher proficiency level [x2 (2, n=54) = 10.681, p < .05]. there was no significant relationship between the didactic approach and the ability to improve proficiency levels (table 7). evidence based library and information practice 2010, 5.3 31 table 7 selecting information: proficiency level improvements between weeks 1 and 4 variable 6: selecting information appropriate to the problem or question at hand didactic group 2 didactic students improved from a “not represented” proficiency to a basic level 6 didactic students improved from a basic to a proficient level 19 didactic students retained their week 1 proficiency levels simulation group 4 simulation students improved from “not represented to a proficient level 13 simulation students improved from “not represented to an exemplary level 5 simulation students improved from a basic to an exemplary level 7 simulation students retained their week 1 proficiency levels discussion a comparison of students’ proficiency between weeks 1 and 4 indicates that the simulation was more likely to improve students’ ability levels across all 6 variables. this means that there was a significant relationship between the simulation approach and increases in students’ abilities to:  recognize the need for information,  formulate specific questions to help find needed information,  identify a range of information sources,  explain successful strategies for accessing needed information,  judge the accuracy, relevance, and completeness of sources, and  analyze information from a variety of sources to determine its applicability to the specific problem. these findings suggests that while there are surface level theoretical homologies between different types of information literacy and instructional environments that seem to assign similar roles to information educators, teachers, and students during the course of learning, there are profound distinctions. moreover, both instructional approaches featured the same content, information-based problems, and educators. however, the ways in which the content was experienced by the students suggested significant distinctions between the approaches. those distinctions are believed to have caused the students in the simulated instructional environment to experience more proficiency level shifts. the distinctions can be organized around 4 dimensions:  situated practice  authenticity  community of practice  an expanded landscape of resources. situated practice both the didactic and simulation environments shared the same informationoriented content and practices; however, the delivery of content within the didactic environment was more abstract and decontextualized. moreover, units of information content were primarily transmitted from educators to students via lectures. although the educators incorporated learning activities that required students to engage in complex problems, the problems did not become the context for teaching and learning. in other words, teaching and learning did not occur primarily within the problem contexts. rather, students went through the traditional process of classroom learning and then demonstrated abilities during an activity session. within the simulation environment, students learned practices through the process of solving complex problems and participating in a virtual community of practice (shaffer, squire, halverson & gee, 2005). in other words, evidence based library and information practice 2010, 5.3 32 students actively participated as members of a disciplinary community (information specialists) while addressing communityspecific tasks/problems. as students worked toward the completion of tasks and problems, they were building an epistemic frame that was specific to the community. in addition, students worked toward the completion of tasks with distinct ways of acting, interacting, valuing, feeling, and knowing that constituted an epistemic frame. the simulation approach situated knowledge development within legitimate participation and practice. authenticity learning environments could be considered authentic to educators or students (brown et al., 1989). both the didactic and simulation environments were considered authentic by the educators because the content, practices, problems, and resources were reflective of real world, disciplinary contexts and activities. however, for students, authenticity refers to the nexus between content or practices and the degree to which they value those practices during activities that require their use (barab & duffy, 2000). the student portfolios suggest that the simulation learners perceived a stronger relationship between information -oriented practices and their value during use whereas portfolios from the didactic learners suggested that they did not value the cognitive process framework (a set of information problem solving practices) that they were taught. instead, the didactic portfolios show that students eliminated many practices (e.g., information evaluation, selection, and organization) from the information problem solving equation. they began to focus more upon quick solutions generated outside of the problem solving framework and its practices, reflected in portfolio products that focused less on the process of information problem solving and more on the creation of products. in other words, they became more productoriented and less process-oriented. portfolios from the simulation learners suggested that they valued the cognitive process framework (a set of information problem solving practices) that had been taught. instead of cutting practices, simulation students began to augment the initial 7-stage cognitive framework with an additional chain of practices: 1) defining the problem through a multimodal and keyword generating process; 2) questioning the current information environment for needed information; 3) searching for different versions of information skills; and 4) thinking about resources that were not in their current environment. within the immersive environment, the improvement from conscious information practices to automatic (almost unconscious) operations was disrupted because of perceived limitations in the information problem solving process and its chain of practices. simulation students not only valued the content during activities that required their use, but they also attempted to improve the practices. expanded landscape of resources the didactic portfolios did not reflect a strong acknowledgment or understanding of different types of information and multiple information environments. instead, they were internet focused, and this was possibly the result of the learning context. moreover, the classroom and its computers served as the primary context for information problem solving activities within the didactic environment. as didactic students encountered problem based activities that required them to deconstruct both the problem and its surrounding context and to solve the problem using information tools, they also encountered a simplification of realworld information complexities. no matter what scenario, the classroom and computer represented the primary informational landscape for problem solving. simulation students, on the other hand, contextualized the use of processes within a vr space that recreated the information complexities of the real world. the simulation reinforced the idea that school libraries are preparing students to navigate, evaluate, and use diverse print and electronic environments, some directly located evidence based library and information practice 2010, 5.3 33 within or linked to institutions, and others independent from time and place constraints. therefore, simulation students were afforded the opportunity to employ processes, practices, and knowledge across information environments and artifacts. the diversity within the simulation manifested itself in student products that acknowledged different types of formats and information environments. community of practice the didactic approach created a teachercentered or lecture-based environment. the simulation environment, on the other hand, created a simulated community of practice (cop). a cop is a disciplinary community of expert practitioners that are bound by: a common task; a shared set of standards; and a common understanding of cognitive and social practices. a cop shares knowledge and practices with students during an instructional process that requires inexperienced members of a discipline (e.g., students) to learn under the guidance of specialists within the discipline (e.g., experienced information specialists) during their legitimate participation in authentic, disciplinary activity. the same content and processes were delivered by both the cop and the lecturebased environment. however, with a few exceptions, those processes were represented only within the simulation portfolios. although this was a simulated cop that could not replicate all real life nuances, the participatory storyline that inscribed authentic, purposeful interactions between students and experts across multiple tasks, artifacts, and environments made the simulated cop more authentic and may have increased the representation of content and processes in the portfolios of the simulation group. limitations there are benefits and pitfalls for all types of assessment, including portfolio assessment. portfolio assessment allows for the purposeful collection of students’ work, and it illuminates their demonstrated efforts and abilities. this form of assessment aligns well with the major strength of learning via simulation —that is, the development of skills through practice and engagement. however it may not have illuminated the major strengths of didactic instruction, the recall of disciplinary content. in other words, didactic students could have developed an exemplary level of understanding of content across the six information literacy domains (i.e., needs identification, question formulation, source identification, information seeking, information evaluation, and source selection) and yet failed in the actual application of the content. on the other hand, the simulation group could have developed an exemplary level of application and failed to answer complex questions about content (e.g., “what is the difference between external, internal, and negotiated needs?”). fixed assessment designs such as multiple choice and short answer questions are more appropriate for the assessment of content knowledge. conclusion the results of this study suggest that simulation approaches have the potential to augment information literacy teaching and learning practices. these technology-based approaches may provide powerful learning environments (virtual worlds) that allow students to engage in the activities and practices of information specialists, instead of simply learning the facts associated with the discipline. digital simulations also allow for the creation of simulated environments that allow students to engage discipline-specific, literacy activities requiring distinctive ways of using language, objects, processes, information, and knowledge. within such simulated learning environments students can inhabit roles such as engineers or information specialists, that may be inaccessible within the traditional teaching and learning landscape. these simulated learning environments can also use their distinct features (e.g., situated practice, authenticity, community of practice, and expanded resource landscapes) to aid in creating a nexus between the understanding of evidence based library and information practice 2010, 5.3 34 abstract ideas within roles and the use of concrete practices to produce meaning within roles (shaffer et. al., 2005). references american association of school librarians & association for educational communications and technology. (1998). information power: building partnerships for learning. chicago:, il american library association. andreas, k., tsiatsost, t., terzidou, t., & pomportsis, a. (sept. 2010). fostering collaborative learning in second life: metaphors and affordances. computers & education, 55(2), 603-615. barab, s., & duffy, t. (2000). from practice fields to communities of practice. in d. jonassen & s. land (eds.) theoretical foundations of learning environments (pp. 25-56). mahwah, nj: erlbaum. barab, s., thomas, m., dodge, t., carteaux, r., & tuzun, h. (2005). making learning fun: quest atlantis, a game without guns. educational technology research and development, 53(1), 86-107. bouras, c., fotakis, d., kapoulas, v., koubek, a., mayher, h., & rehatscheck, h. (1999). virtual european school-ves. in proceedings of the ieee multimedia systems ’99: special session on european projects, 1055–1057. bouras, c., triantafillou, v., & tsiatsos, t., (2001). aspects of collaborative learning environment using distributed virtual environments. in proceedings of the world conference on educational multimedia, hyper-media & telecommunications, 173–178. bouras, c., giannaka, e., & tsiatsos, t., (2003). virtual collaboration spaces: the eve community. in proceedings of the 2003 symposium on applications and the internet, 48–55. brown, j. s., collins, a., & duguid, p. (jan. feb. 1989). situated cognition and the culture of learning. educational research, 18(1), 32-42. bruner, j. (1977). the process of education. cambridge, ma: harvard university press. chen, s. 1993. a study of high school students’ online catalog searching behavior. school library media quarterly 22: 3340. cobb, p. (1996). where is the mind? a coordination of sociocultural and cognitive constructivist perspectives. in c. t. fosnot (ed.), constructivism: theory, perspectives, and practice (pp. 34-54). new york, ny: teachers college press. collins, a., brown, j., & newman, s. (1989). cognitive apprenticeship: teaching the crafts of reading, writing and mathematics. in l. b. resnick (ed.), knowing, learning and instruction: essays in honor of robert glaser (pp. 223–53). hillsdale, n.j.: erlbaum. corbit, m. (2002). building virtual worlds for informal science learning in the active worlds educational universe (awedu). presence 11, 55–67. costabile, m., de angeli, a. roselli, t., lanzilotti, r. & plantamura, p. (2003). evaluating the educational impact of a tutoring hypermedia for children. information technology in childhood education annual, 289-308. dewey, j. (1933). how we think. lexington, ma: heath. dewey, j. (1916). democracy and education: an introduction to the philosophy of education. new york, ny: macmillan. evidence based library and information practice 2010, 5.3 35 dunn, k. (2002). assessing information literacy skills in the california state university: a progress report. journal of academic librarianship, 30, 26-35. focier, r. c. (1999). the computer as an educational tool: productivity and problem solving. columbus, oh: merrill. gallimore, r., & tharp, r. (1990). teaching mind in society: teaching, schooling and literate discourse. in l. c. moll (ed.), vygotsky and education: instructional implications and applications of sociohistorical psychology (pp. 175–205). cambridge, u.k.: cambridge. gee, j. (2007). what video games have to teach us about learning and literacy. new york, ny: palgrave macmillan. gibson, j. j. (1986). the ecological approach to visual perception. hillsdale, nj: erlbaum. goniz m, & medina c.. (2000). engaging girls with computers through software games. communications of the acm, 43(1), 42-49. gredler, m. e. (2004). games and simulations and their relationships to learning. in d h. jonassen (ed.), handbook of research on educational communications and technology (pp. 571-581). mahwah, nj: erlbaum. hirsh, s. (1998). relevance determinations in children’s use of electronic resources: a case study. proceedings of the 61st asis annual meeting, 35, 63-72. inhelder, b., & piaget, j. (1958). the growth of logical thinking: from childhood to adolescence: an essay on the construction of formal operational structures. new york, ny: basic books. kelly, g. a. (1963). a theory of personality: the psychology of personal constructs. new york, ny: norton. kirshner, d., & whitson, j. a. (1997). situated cognition: social, semiotic, and psychological perspectives. mahwah, nj: erlbaum. laffey, j., espinosa, l., moore, j., & lodree, a. (2003). supporting learning and behavior of at-risk young children: computers in urban education. journal of research technology in education, 35, 423-440. loerke, k. (1994). teaching the library research process in junior high. school libraries in canada, 14, 23-26. malouf, d. (win.) 1987-1988). the effect of instructional computer games on continuing student motivation.” the journal of special education, 21(4), 2738. mark, b., & jacobson, t. (1995). teaching anxious students skills for the electronic library. college teaching, 43, 28-31. mclellan, h. (2004). virtual realities. in d h. jonassen (ed.), handbook of research on educational communications and technology (pp. 461-497). mahwah, nj: erlbaum. pitts, j. (1995). mental models of information: the 1993-1994 aasl/highsmith research award study. school library media quarterly 23, 177-184. price, r. 1990. computer-aided instruction: a guide for authors. pacific grove, ca: brooks/cole. prensky, m. (2001). digital game-based learning. new york, ny: mcgraw-hill. rieber, l. (1996). seriously considering play: designing interactive learning evidence based library and information practice 2010, 5.3 36 environments based on the blending of microworlds, simulations, and games. educational technology research and development (44), 43-58. rosas, r., nussbaum, m., cumsille p., marianov,v., correa, m., flores p., grau v., lagos f., lópez, x., lópez v., rodriguez p., & salinas, m. (jan. 2003). beyond nintendo: design and assessment of educational video games for firstand second-grade students. computers & education, 40(1), 71-94. shaffer, d., squire, k., halverson, g., & gee, j. (2005). video games and the future of learning. phi delta kappan, 87, 104–11. san chee, y., & meng hooi, c. (2002). c visions: socialized learning through collaborative, virtual, interactive simulations. in proceedings of conference on computer support for collaborative learning, 687–696. shaw, r. e., & bransford, j. (1977). introduction: psychological approaches to the problem of knowledge. in r. e. shaw & j. bransford (eds.), perceiving, acting, and knowing (pp. 1-39). hillsdale, nj: erlbaum. van eck, r., & dempsey j. (2002). the effect of competition and contextualized advisement on the transfer of mathematics skills in a computerbased instructional simulation game. educational technology research and development, 50(3), 23-41. virvou, m., katsionis, g., & manos, k.. (2005). combining software games with education: evaluation of its educational effectiveness. educational technology & society, 8(2), 54-65. wiebe, j., & martin, n. j. (1994). the impact of a computer-based adventure game on achievement and attitudes in geography. journal of computing in childhood education, 5(1), 61-71. evidence based library and information practice 2010, 5.3 37 appendix a variables and values variable 1. recognizing the need for information value 1.0 not represented. value 1.1 basic—when faced with the information problem, the student did not consider whether additional information (beyond one’s own knowledge) was needed to resolve it. value 1.2 proficient—when faced with the information problem, the student recognized the need for additional information as a central feature in the problem solving process. value 1.3 exemplary—the student identified the primary problem and a range of connected problems. s/he also assessed whether the problems could be resolved based on one’s own knowledge or whether additional information was required. variable 2. formulating questions based on information needs value 2.0 not represented value 2.1 basic—the student stated one or two broad questions that would help in finding needed information. value 2.2 proficient—the student stated both broad questions and sub-questions that would help in finding needed information. value 2.3 exemplary—the student used a flexible questioning style that allowed him/her to revise, add, and delete questions within a given problem situation. variable 3. identifying a variety of potential sources of information value 3.0 not represented. value 3.1 basic—the student listed one or two sources of information and generally explained the kind of information found in each. value 3.2 proficient—the student brainstormed a range of sources of information that would meet an information need. value 3.3 exemplary—the student used a full range of information sources to meet differing information needs. variable 4. developing and using successful strategies for locating information value 4.0 not represented. value 4.1 basic—the student listed basic ideas for how to identify and find needed information. value 4.2 proficient—the student articulated and applied a plan to access needed information. value 4.3 exemplary—the student formulated and revised plans for accessing information for a range of needs and situations. variable 5. determining the accuracy, relevance, and comprehensiveness of information value 5.0 not represented. value 5.1 basic—the student generally understood the concept of accuracy, relevance, and comprehensiveness, but considered them sparingly. value 5.2 proficient—the student compared and contrasted sources related to the primary problem to determine which were more accurate, relevant, and comprehensive. value 5.3 exemplary—the student judged the accuracy, relevance, and completeness of information in relation to a range of problems and sub-problems. evidence based library and information practice 2010, 5.3 38 variable 6. selecting information appropriate to the problem or question at hand value 6.0 not represented value 6.1 basic—the student identified information (one or two types) applicable to a specific information problem or question value 6.2 proficient—the student analysed information from a variety of sources to determine its applicability to the specific information problem. value 6.3 exemplary—the student integrated accurate, relevant, and comprehensive information to resolve the information problem. evidence based library and information practice evidence based library and information practice 2010, 5.2  evidence based library and information practice       evidence summary    sabbatical options for academic librarians in the u.s. vary widely    a review of:  flaspohler, m. r. (2009). librarian sabbatical leaves: do we need to get out more? journal of academic  librarianship, 35(2), 152‐161.     reviewed by:  heather ganshorn  librarian, health information network calgary, holy cross site  calgary, alberta, canada  email: heather.ganshorn@ucalgary.ca      received: 2 mar. 2010          accepted: 14 apr. 2010       2010 ganshorn. this is an open access article distributed under the terms of the creative commons‐ attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐ sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original  work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is  redistributed under the same or similar license to this one.    abstract   objectives – to gather data on what  proportion of u.s. academic libraries provide  sabbatical opportunities to librarians, and to  explore library directors’ perceptions of the  effectiveness of sabbaticals and barriers to  sabbatical‐taking among librarians at their  institutions.  design – online questionnaire.    setting – academic libraries in the u.s..    subjects – directors of 403 academic libraries.    methods – the author reviewed the literature  on sabbatical leaves in the library profession.  she then developed an online survey using the  university of washington’s catalyst system (a  tool similar to surveymonkey). the survey  contained both closed and open‐ended  questions, in order to generate quantitative  data as well as to gather more in‐depth  information on respondents’ views.     a sample of american academic library  directors was generated by choosing every  eighth entry on a list of 3037 academic  libraries generated by lib‐web‐cats, an online  directory of libraries  (http://www.librarytechnology.org/libwebcats /). the survey was sent to 403 academic library  directors based on this selection method. the  author received 101 successfully completed  surveys for a response rate of 25%.    main results – the author found that just  over half of respondents (53 libraries, or 52%)  indicated their institutions offered sabbatical  leaves to librarians. thirty‐six per cent  indicated they did not, while 12% indicated  “other” (many of these respondents  93 mailto:heather.ganshorn@ucalgary.ca evidence based library and information practice 2010, 5.2  commented with clarifications about what  other leave programs were available to  librarians). of the 53 institutions that reported  offering leave programs, only half (27  respondents) indicated that library staff  members had taken a sabbatical leave.    open‐ended questions generated some insight  into the barriers to sabbatical leaves at  academic libraries. differences between  institutions in terms of availability of  sabbatical leaves appear to be due to a  combination of librarian status (whether or not  librarians have full faculty status); funding  issues (in some institutions, the library, and  not the college administration, has to cover the  costs of a sabbatical); and availability of other  staff to cover the duties of the individual  taking the leave.     respondents also noted a discrepancy  between the length and timing of librarian  sabbaticals compared to other faculty (i.e., the  professoriate), with librarians more often  required to begin their leaves in the summer.  librarian sabbaticals were also sometimes  shorter than those of other faculty; in some  institutions a summer‐length sabbatical was  available, but not a six‐month or year‐long  sabbatical, even though these options were  available to other faculty.    in terms of impacts of sabbaticals, most  respondents who had experienced a staff  member taking sabbatical felt that the  sabbatical benefited the staff member and the  institution; positive results include improved  morale, publications that raised the profile of  the library, and learning that was applied in  the workplace. some respondents, however,  had negative experiences to report, the most  common being that the sabbatical had had no  effect. some respondents noted staff who had  taken sabbaticals had failed to meet the goals  that had been set for the sabbatical. when  asked what could be done to enhance  sabbatical programs, respondents at  institutions with these programs had some  interesting suggestions, such as aligning  sabbatical programs more closely with  institutional goals, or promoting the pursuit of  more collaborative research while on  sabbatical.    conclusion – the author notes that while it’s  dangerous to over‐generalize from such a brief  survey, many of the issues raised in the  responses, such as faculty status, funding  shortfalls, and staff shortages echo themes  raised elsewhere in the library literature.  these issues probably need to be addressed if  we are to see any increase in the number of  librarians taking sabbatical leaves.    the author’s other conclusion is that librarians  must be more accountable for demonstrating  how a sabbatical could add institutional value,  and for meeting the goals set in their  sabbatical plans. the author conducted this  study while on sabbatical herself, and  concludes it “provides one example of how a  librarian might create a manageable, research‐ based project that more closely marries  academic rigor to personal experience” (p.  160).      commentary    the  author  of  this  evidence  summary  used   lindsay  glynn’s  critical  appraisal   checklist  to  estimate  the  overall  validity  to   be  around 50%;  therefore no generalizable  conclusions can be drawn from this study,  though it raises some interesting issues for  future research.    the author has chosen an interesting topic; as  she demonstrates in her literature review,  there isn’t much research on librarian  sabbaticals. what research is available does  not analyze barriers and enablers, outcomes  for the individual and institutions, or even the  actual uptake of sabbatical leaves in  institutions that offer them. in that sense, the  author’s study has the potential to add to our  knowledge of these issues, and provide  directions for future study.    the author doesn’t address some important  first premises in her article, such as the  purpose of sabbaticals (among teaching  94 evidence based library and information practice 2010, 5.2  95 faculty, they are for research, and not for  burnout or continuing education, as she seems  to imply). it is also unclear whether sabbaticals  are in fact universally offered to teaching  faculty at all post‐secondary institutions.  there is no comparison in this study of how  the library’s sabbatical policy may differ from  the same institution’s policy for other faculty.    the author is to be applauded for including  her survey instrument as an appendix  (available in the online but not the print  version of the article). this makes it much  easier to appraise her article, and also  provides an example for other researchers  who may wish to adapt her study.  unfortunately, the author does not indicate  that her survey instrument was pre‐tested or  validated.    most of the survey questions address only  those institutions where there is a leave  program; given that the author’s literature  search had identified a lack of sabbatical  programs in many academic institutions, it  would have been useful if the survey had  included more questions specifically for those  without a program. the author mentions  wanting to keep the survey short to increase  response rates among busy library directors,  but these additional questions would not have  increased the length of the survey as  respondents could have been routed to  different series of questions depending on  whether or not their institution offers  sabbatical leave.    there were also issues with the sampling  method used in this survey. the author used  the directory of libraries at lib‐web‐cats. there  is no way to determine how accurately this  directory reflects the actual number of  academic libraries in the u.s., though it does  seem quite comprehensive. still, one might  question why the author did not instead  contact the association of college and  research libraries for a list of member  institutions, or perhaps try to obtain separate  lists in niche areas (for example, a list of all the  tribal colleges in the u.s.).     while not strictly randomization, the author  did employ a reasonable approach to  obtaining a study sample, choosing every  eighth library from the alphabetical lib‐web‐ cats list. it might also have been worthwhile to  use a stratified sampling method to ensure  representation from different types of colleges.    the selection of sample size was problematic.  the author states, “from the start i planned to  survey about 400 libraries, hoping for a strong  response rate while maintaining a data set that  i felt was still manageable for one person” (p.  154). there are problems with this approach.  most researchers with experience in survey  design would advise against hoping for a  large response rate. it’s better to prepare for a  low response rate by increasing your sample  size. sample size should not be determined by  what the researcher feels she can handle as a  data set, but rather by statistical  considerations of what size is required to  provide usable data. perhaps the author could  have considered sending the survey to more  libraries when it became clear her response  rate was low. given that sabbatical programs  are not something that changes rapidly over  time, a delay in survey administration should  not have affected the integrity of the data.    there was one small mistake in the use of  figures.  the author used a line graph rather  than a bar chart in figure 5 (p. 156) to illustrate  the number of librarian sabbaticals over the  last five years by carnegie institution type.  since this is nominal and not continuous data,  a bar chart would have been more  appropriate.    the open‐ended responses raised some  interesting questions that need to be  answered. how do we define a successful  sabbatical from both the employer and  employee perspective? how can programs be  structured to ensure that the candidate and the  employer have the same understanding of the  goals of the sabbatical, and to ensure  accountability on both sides (support from the  institution and the meeting of goals by the  candidate)? to what degree does faculty status  determine the availability and nature of   evidence based library and information practice 2010, 5.2  sabbaticals at an institution? given that many  institutions have a sabbatical program in  theory, but in practice nobody ever applies to  it, what are the barriers and enablers to  librarian sabbaticals? this is fertile ground for  future survey research, and could also  possibly benefit from some more qualitative  research methods that explore the reasons  behind some of the more successful programs.      references    glynn,  l. (2006). a critical appraisal tool  for library and information research.  library hi tech  24(3), 387‐99.       96 evidence summary   first year medical students use library resources emphasized during instruction sessions   a review of: rafferty, ryan s. (2013). the impact of library instruction: do first-year medical students use library resources specifically highlighted during instructional sessions? journal of the medical library association, 101(3), 213-217.   reviewed by: laura newton miller collections assessment librarian carleton university library ottawa, ontario, canada email: laura.newtonmiller@carleton.ca   received: 25 nov. 2013 accepted: 17 jan. 2014      2014 newton miller. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to determine if library instruction has an effect on resources cited in student reports.   design – citation analysis.   setting – the study took place in the medical school of a large american university.   subjects one hundred eighteen of 120 first-year medical student reports were analyzed. two reports did not include any works cited and were excluded from the study.   methods over the course of 3 years, 15 20-minute library instruction sessions were conducted. the sessions, based on five clinical cases presented each year were conducted approximately two weeks before each report due date. eighty-five case-specific resources were demonstrated, with teaching plans being modified from year to year based on the frequency of citation of a particular resource cited the prior year. a libguide online course guide also directed students to specific resources shown in the class, with content updated every year based on citation trends from the previous year. every citation referenced in a report was then categorized into a) those that were discussed during an instruction session, b) those found on a course guide, c) those accessible through the library, d) those available from course material (i.e., powerpoint presentation, lecture notes), or e) those which did not fall under any of the other categories. a citation could be included in multiple categories.   main results – the 118 reports included 2983 citations. over the 3 year period, an average of 77.51% of all citations were from library resources, 49.55% of the citations from a resource demonstrated in the class, and 21.68% from resources found in the course guide. although citations from sources discussed in class did not increase significantly from year to year, the percent of citations from resources on the course guide significantly increased from 19.40% to 25.63%.   conclusion – medical students cite library resources emphasized during instruction sessions.     commentary   although there is a growing number of citation analysis studies related to the effects of library instruction, there are relatively few papers that report on the effect of instruction on medical students. this paper fills that particular niche.   using the evidence based library and information practice (eblip) critical appraisal checklist (glynn, 2006), it was determined that citation analysis was an appropriate tool for this study. the methodology is explained very clearly and those interested would be able to replicate the study relatively easily.   percentages of citations linked to various sources were presented in averages. because there were five different cases throughout a particular year for three years, it would have been beneficial to see more detailed analysis. for example, were there any differences in where sources came from based on a particular case? in other words, were students more likely to use demonstrated resources for some case study reports, but use more online course guide resources for other case studies? it also would have been useful to give closer examination to citations found in “nonlibrary” and “other” categories (approximately 20% each).   the author’s literature review describes how verbal encouragement from professors or librarians does not seem to be as effective as actually having clearly defined guidelines for students (p. 214). it was interesting to see that there were no significant changes in percentages of cited resources discussed in instruction over the three year period, yet percentages of cited items found in course guides steadily increased from year to year. this is partially due to the fact that popular resources cited from previous years were added to subsequent course guides. one can argue that citation analysis can be a valuable teaching assessment tool that helps librarians make modifications to teaching plans and guides based on popular citations from previous years.   reading this paper makes one think of the bigger philosophical questions of providing instruction: should we be giving students the fish or teaching them how to fish? should the students simply be given the titles of books/papers where they can find their answers or should they be shown how to find the information through informed search techniques? it is certainly easier to conduct a citation analysis if there are specific titles for which to look, but it may not help students’ information seeking skills in the long term. in this study, the author finds some balance by providing instruction on general navigation of key databases, but it is important for those devising future impact studies to consider the bigger philosophical question.   this study shows that the library is making an impact on a specialized group of students with very specific resource needs. conducting a follow-up study with those students by examining use of resources in other assignments could inform any long-term impact of instruction. for other student groups where a broad range of journals and books are needed, it may be interesting (or depressing) to know how many of those library resources are also retrievable through google scholar. in other words, future studies which include checking google scholar as part of the methodology may help determine if students would have found these resources regardless of whether library instruction was provided. this paper is useful for librarians interested in using citation analysis as an assessment tool for their own teaching – to be able to modify teaching plans based on what students cite in their papers. it is also of interest to those needing to show the library’s value and impact for specialized groups of students.     reference   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399.   evidence based library and information practice evidence based library and information practice 2011, 6.2 55 evidence based library and information practice evidence summary translation of hedges in medical databases to other platforms’ syntax may cause significantly different search results a review of: bradley, s. m. (2010). examination of the clinical queries and systematic review “hedges” in embase and medline. journal of the canadian health libraries association, 31, 27-37. reviewed by: heather ganshorn health sciences librarian university of calgary calgary, ab, canada email: heather.ganshorn@ucalgary.ca received: 28 feb. 2011 accepted: 21 apr. 2011 2011 ganshorn. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to determine whether the methodological search filters in ovidsp medline and ovidsp embase also known as clinical queries hedges had been modified from the originals which were written by the mcmaster university health information research unit hedges group (the haynes group) and whether the translations of these hedges by the national library of medicine used in pubmed and ebsco medline were reliable. the hedges examined are for the clinical categories of diagnosis, therapy, etiology, prognosis, clinical prediction guides, and reviews. the author also examined the translated national library of medicine (nlm) systematic reviews hedges in ovidsp medline and ebsco medline. design – validity of hedges used in various databases. setting – ovidsp medline, ovidsp embase, ebsco medline and pubmed were studied. subjects – the clinical queries hedges designed to facilitate enhanced retrieval of particular types of studies in the abovementioned databases were compared. methods – the author ran the clinical queries hedges in ovidsp medline, ovidsp embase and pubmed. next, she manually entered the original haynes group published hedge search strings for each clinical query in these databases, and compared the results to the clinical queries. the author also mailto:heather.ganshorn@ucalgary.ca� evidence based library and information practice 2011, 6.2 56 compared the results obtained from the ovid medline clinical queries versus the hedges in pubmed and ebsco medline. the percentage difference in number of hits between the ovid platform and the other platform was calculated. where the difference was greater than 10%, the author modified the search string and re-tested it. there was no gold standard for comparison, so it was not possible to make calculations such as sensitivity, specificity, precision, or accuracy. for the testing of the review hedges, the author used the cochrane database of systematic reviews as a gold standard to compare search results. she also compared the results in ovidsp medline to the results in ebsco medline and pubmed. main results – comparing the 27 ovidsp clinical queries limits to the equivalent haynes search strings, the author found identical results, suggesting that the ovidsp hedges have not been changed from haynes’ original search strings. however, when the ovidsp medline hedges were compared to pubmed and ebsco, there were discrepancies. if the hedges were translated exactly, one should expect the result sets to be nearly identical, with the exception of records that had not yet been uploaded to ovidsp and ebsco (pubmed contains records that are not yet fully indexed). however, other problems became evident. while the majority of searches yielded similar numbers of records, there were discrepancies of >10% in the number of hits for five of the clinical queries. some of the hedges involved truncated search terms that, in pubmed, generated a message indicating that only the first 600 variations of the word root would be used. the author modified these hedges in order to obtain potentially more accurate results, though as she does not have a gold standard set for comparison, the modified hedges could not be thoroughly evaluated. three of the ebsco medline clinical queries hedges also generated significantly different results from ovidsp medline. the author was able to modify these hedges to generate similar results to those found in pubmed. the author’s examination of the various systematic review hedges identified other problems. for these hedges, it was possible to use the cochrane database of systematic reviews as a simple gold standard to assess the reliability of these filters. the haynes clinical queries review hedge is used in ovidsp embase. the author found that this hedge’s sensitive filter retrieved 100% of the cochrane reviews, while the optimized filter retrieved all reviews but one. however, the specific filter retrieved only 16% of the cochrane reviews. the author notes that the haynes hedges were developed using a subset of journals that did not include the cochrane database of systematic reviews. the clinical queries review hedge in medline appeared to have better results. in ovidsp, the sensitive and optimized hedges found all but one record, while the specific hedge found 83% of the records, a result that was mirrored in ebsco medline and pubmed. conclusion users of ovidsp medline can be confident that the clinical queries limits are true translations of the hedges published by haynes et al., as they were found to give identical results to manual entry of these hedges. however, users cannot be confident that these queries will give the same results in pubmed, due to differences in syntax between the two interfaces. users of ebsco medline can be less confident that the clinical queries have been perfectly translated from the original haynes queries, as three of these queries were found to yield significantly different results from the ovidsp medline search. the author recommends that ovidsp be the search interface of choice when using these hedges in medline. the national library of medicine’s (nlm) systematic reviews hedge has been translated into ovidsp and ebsco, but has never been validated. the author found significant errors in this hedge in the ovidsp version, which evidence based library and information practice 2011, 6.2 57 were rectified after she contacted ovid. however, ovid was reluctant to share its translation of the hedge, as this is proprietary information. the author recommends that for this reason, it is best to use pubmed to search for systematic reviews, as the search string for its hedge is publicly available. the author also notes that this issue of proprietary information is very problematic for librarians, as it makes it impossible for them to assess the hedges they are using from vendors, or to identify the source of the problem when they get unusual results. commentary this study raises some important concerns about the reliability of hedges in the various medline and embase interfaces when searching the biomedical literature. the author’s approach to comparing these hedges is methodical and easily replicated. she has included several tables comparing results across interfaces in the print version of the article, and further supplementary data is available in the form of three supplementary appendices to the online version of the article. this makes is easy to view and analyze her work. the author also found some problems with ovidsp’s subject subheadings, which were reported to ovid and subsequently rectified, so this research has already had a demonstrable beneficial effect. this research demonstrates that the closedaccess nature of most proprietary database platforms can be problematic. the author contacted ovid when she identified problems with their translation of the nlm systematic reviews hedge, requesting to view their version of it, and was told that the information was “proprietary,” though they eventually agreed to provide her with the hedge, only for use with her student coursework (p. 34). she also found problems with two other subheadings, which she reported to ovid. however, her research clearly shows that users would benefit from a more open system in which the translations of these filters could be scrutinized by librarians, and improved upon/corrected if necessary. this study raises several issues that are worth pursuing further. the author only had access to embase in the ovidsp version. it would be valuable if the original elsevier version of embase could also be included in a future comparison. since pubmed and ovidsp have both updated their platforms since this work was done, it would be difficult to replicate the author’s work, but it is likely that many of the same issues persist, or that new ones may have arisen. regular analysis of this sort should be carried out and reported on, ideally by more than one person or organization, and according to agreed-upon guidelines. this is information that medical librarians need on an ongoing basis. one would hope that the vendors themselves would shoulder some of this work. the author notes that she modified some of the problematic hedges, and that this appeared to improve her results, but without a gold standard set of articles, it is impossible to be certain. it would be interesting to see this study expanded to include such a gold standard set, but as this was a student project, and the work of one individual, it is understandable that the author chose not to expand the project in this direction. the author also notes that “although discrepancies in the number of hits would indicate a difference in the search results, the same number of hits would not necessarily mean identical search results” (p. 31). a gold standard set would go some way toward resolving the question of how similar different vendors’ result sets are. the use of the cochrane database of systematic reviews as a gold standard set for evaluating the systematic reviews hedges was very informative; it is doubtful that most searchers are aware that this hedge may not find relevant cochrane reviews. in general, when developing or modifying search strategies for testing, the best practice would be to have another information professional review and, if necessary, suggest evidence based library and information practice 2011, 6.2 58 revisions to one’s work. the author does not expressly indicate that this was done, though as it was a student project, and she credits a faculty member in the acknowledgments, one can infer that there was some review of the author’s work. this study is a well executed evaluation of some key tools for medical librarians, and raises many issues that merit further investigation. / evidence based library and information practice evidence summary evidence summary   library staff morale correlates with having a sense of respect and value for their work, relationship to direct supervisors and colleagues, and autonomy and flexibility in their work environments   a review of: glusker, a., emmelhainz, c., estrada, n., & dyess, b. (2022). “viewed as equals”: the impacts of library organizational cultures and management on library staff morale. journal of library administration, 62(2), 153–189. https://doi.org/10.1080/01930826.2022.2026119   reviewed by: eugenia opuda health & human services librarian dimond library university of new hampshire durham, new hampshire, united states of america email: eugenia.opuda@unh.edu   received: 1 sept. 2022                                                               accepted:  21 oct. 2022      2022 opuda. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30240      abstract   objective – to explore what library organizational factors influence library staff morale.   design – semi-structured interview, grounded theory.   setting – academic libraries across the united states during the early months of the covid-19 pandemic.   subjects – 34 academic library staff, defined by the authors as employees whose positions do not require an mlis degree and do not include the title “librarian”, from 23 private and public colleges and universities across 16 states, mostly representative of the west and midwest regions.   methods – in 2020, the authors emailed a call for study participants to library listservs and state library associations across the us, selected a convenience sample of 34 library staff from academic institutions, and conducted structured interviews by phone or by google meet over the course of may through june 2020. the authors note that the sample over-represents public and larger institutions in the west and midwest regions. a student worker transcribed the audio recordings and de-identified transcripts underwent iterative, thematic coding in maxqda, a qualitative data analysis tool. the authors used a grounded theory approach to conduct open coding, then identified relationships between themes, and elaborated upon each theme based on its relationship to a theoretical model of morale impact avenues in library organizational structures, which was developed by one of the authors.   main results –the authors uncovered that most study participants (n = 21) reported having high levels of morale, a surprise to the research team who expected that participants with lower levels of morale would participate in the study. most participants (n = 27) worked in public and larger institutions, and the majority were female (n = 24), though only 5 were black, indigenous, and people of color (bipoc). participants mostly had mlis degrees or other advanced degrees. the results of the study expanded beyond the original research questions to comprise a broader set of factors that impact morale levels including relationships with colleagues and direct managers, opportunities for advancement, respect, work autonomy, and funding. respondents emphasized that staff morale was significantly impacted by their relationship with direct managers, noting that micromanagement, defensiveness, and lack of accommodations contributed to lower levels of morale and a sense of disconnection. managers who were supportive, advocated for staff needs, and were good listeners improved morale. relationships between staff and their librarian colleagues also impacted morale, with the librarian–staff divide and treatment of staff by librarians being major contributors to influencing morale. additionally, staff felt that having or lacking respect from librarians and administration and having autonomy and flexibility in their work made a big impact on morale. having opportunities to meaningfully engage, to advance in the workplace, to receive professional development funding, to participate in decision-making processes, and to feel valued by the institution contributed to higher levels of staff morale.   conclusion – library staff morale is impacted mostly by staff members’ sense of connection, respect, and value within the institution and among their librarian colleagues, direct managers, and library administration. having pathways for advancement and professional development, meaningful opportunities to contribute to institutional decision-making, and autonomy over their professional and personal lives contributed to a higher sense of staff morale. the authors highlight several practical recommendations for improving staff morale including fostering a respectful environment, advocating for more flexible and better work environments, and providing opportunities for professional development and growth.   commentary   this research article contributes to a significant gap in the library research literature on the factors that impact library staff morale. issues of library worker morale and burnout existed prior to the covid-19 pandemic, though the pandemic exacerbated these existing problems (corrado, 2022; hodge & williams, 2021; hudson-vitale & miller waltz, 2020; miller waltz, 2021). hodge and williams (2021) describe the significant harm of the pandemic in combination with anti-black racism that was finally examined in 2020 and call for library institutions to identify and address structural inequities. corrado (2022), hudson-vitale and miller waltz (2020), and miller waltz (2021) offer practical suggestions on mitigating the effects of low morale and burnout due to the pandemic. furthermore, katopol (2016) describes the impact of burnout on developing compassion fatigue and the negative impacts on library workers’ wellbeing and work satisfaction.   this article was appraised using an adapted version of russell and gregory’s qualitative research appraisal questions from 2003 (suarez, 2010). while the authors identified specific and clear research questions, they acknowledge that they modified the research questions due to the richness of the qualitative data. additionally, the authors received irb approval of their study and selected appropriate methods for conducting the study. the appendix includes a helpful collection of all forms, questionnaires, and protocols used in the study. the authors conducted the sampling thoughtfully, though they state that bipoc library staff members, private institutions, and the south and the northeast were underrepresented. they indicate that more targeted research is needed to be more representative of library staff nationally. additionally, the authors do not identify the roles of the library staff participants or explore the impact of having an mlis or another advanced degree, which could have further contextualized the findings of the interviews. the authors pilot tested their interview questions and collected, managed, and analyzed data systematically with de-identified information.   interview data and summary of findings are rich and thoughtfully capture the significance of strong interpersonal relationships, mutual respect, and placing value on library staff members despite their positions within an organization’s hierarchy. documenting their coding data in the findings to fully illustrate the grounded theory methodology could have increased transparency of the results. though the authors design a theoretical model, the visual diagram illustrating the model falls short of representing the elements and relationships that they hope to convey. the overarching message of the study simply underscores the importance of treating library staff workers as valuable and equal members of a library’s organization and investing in their growth and development as much as librarians. the authors uncover ways in which library staff experience low morale due to feeling undervalued, micromanaged, and treated as dispensable.   the timing of this research is unique as the authors conducted interviews in the early months of the covid-19 pandemic. themes that emerged from data collection were likely enhanced or exacerbated by the pandemic, particularly the theme of autonomy and flexibility of the work environment. as library staff juggled new work environments and budgetary uncertainties, they also juggled new professional and personal responsibilities. the authors identify impactful and helpful recommendations to improve staff morale including advocating for work flexibility and fair pay, meaningfully incorporating staff voices in decision-making processes, and investing in staff professional development and career advancement opportunities. several years after the start of covid-19-related shutdowns, the takeaways from the authors’ research are arguably more imperative for library institutions to consider and to apply as more emphasis is being placed on workplace burnout, worker mental health, and organizational empathy. it highlights inequities within library organizational cultures and contributes to amplifying the voices of library staff who are underrepresented in this area of research.   references   corrado, e. m. (2022). low morale and burnout in libraries. technical services quarterly, 39(1), 37–48. https://doi.org/10.1080/07317131.2021.2011149   farkas, m. (2021, march/april). building morale in a pandemic: how to support library workers as whole people. american libraries, 52(3/4), 48. https://americanlibrariesmagazine.org/2021/03/01/building-morale-pandemic-library-workers/   glusker, a., emmelhainz, c., estrada, n., & dyess, b. (2022). “viewed as equals”: the impacts of library organizational cultures and management on library staff morale. journal of library administration, 62(2), 153–189. https://doi.org/10.1080/01930826.2022.2026119   hodge, t., & williams, j. (2021, january/february). call to action: envisioning a future that centers bipoc voices. american libraries, 52(1/2), 54. https://americanlibrariesmagazine.org/2021/01/04/call-to-action/   hudson-vitale, c., & waltz, r. m. (2020). caring for our colleagues: wellness and support strategies for remote library teams. college & research libraries news, 81(10), 494–497. https://doi.org/10.5860/crln.81.10.494   katopol, p. (2015). enough already: compassion fatigue. library leadership & management, 30(2). https://doi.org/10.5860/llm.v30i2.7177   miller waltz, r. (2021). in support of flourishing: practices to engage, motivate, affirm, and appreciate. international information & library review, 53(4), 333–340. https://doi.org/10.1080/10572317.2021.1990564   suarez, d. (2010). evaluating qualitative research studies for evidence based library and information practice. evidence based library and information practice, 5(2), 75–85. https://doi.org/10.18438/b8v90m     evidence summary   students value asynchronous instruction, individual projects and frequent communication with the instructor in an online library science classroom   a review of: hajibayova, l. (2017). students’ viewpoint: what constitutes presence in an online classroom? cataloging & classification quarterly, 55(1), 12–25. https://doi.org/10.1080/01639374.2016.1241972   reviewed by: heather macdonald health and biosciences librarian macodrum library carleton university ottawa, ontario, canada email: heather.macdonald@carleton.ca   received: 21 aug. 2020                                                             accepted:  30 oct. 2020      2020 macdonald. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29826     abstract   objective – determine student perceptions of online learning.   design – survey questionnaire.   setting – an online class in the school of library and information science at a midwestern us public university.   subjects – 45 graduate students in an abstracting and indexing class.   methods – class participants filled in an online questionnaire at the end of the semester. the survey covered topics related to collaboration, communication, modes of instruction, and assessment. the researcher calculated frequency counts for questions and did a correlation analysis.   main results – for collaboration the author found that 62% of students expressed no or limited interest in participation in collaborative projects. factors for successful completion of group projects included member commitment, instructor involvement, technology tools (discussion boards, wikis, blogs), group size (3–5 people preferred), and the nature and design of the project.    preference for communication frequency via email ranged from daily to never with the highest percentage (28.57%) preferring once a week. communication frequency through the learning management system (lms) was similar. the largest percentage of students preferred communication 2–3 times per week for virtual (38.89%) and face-to-face (41.67%) office hours. the correlation between communication via lms and virtual office hours was r = 0.89, p < 0.05. of students completing the questionnaire, 47.22% found the instructor’s presence effective. while most students disagreed with using social media in an online course, many friended or followed the instructor or the class social media page.   students preferred asynchronous over synchronous lectures and activities. preference for frequency was once a week. there was a correlation between synchronous lectures and synchronous activities (r = 0.77, p < 0.05).   student preferences for the frequency of overview and discussion of class materials were roughly equal in distribution (daily, 4–6 times/week, 2–3 times/week, weekly, or never). there was a correlation between synchronous overview and asynchronous overview of class materials (r = 0.93, p < 0.05). in terms of assessment, students found group discussion, individual projects, research papers, quizzes, and tests the most effective class assignments. several correlation analyses were done between assignment types.   conclusion – this study found students had limited interest in collaborative projects. it was also found that regular communication with the teacher was important. students preferred asynchronous instruction and activities. they also preferred individual assignments for evaluation.   commentary   this study is assessed using the cat: a generic critical appraisal tool (perry & rathbun-grubb, 2014). the author lists four research questions (two are combined under one bullet point). the author also provides background on factors influencing the quality of online teaching and learning and then introduces the community of inquiry (coi) framework with three core overlapping presences (social, cognitive, and teaching). the author references the validated coi instrument (arbaugh, 2008) that measures each of the presences and their interrelationships.    although this study is based on coi, it did not use the coi survey instrument. the instrument is not referenced in the methods section, and no questionnaire is provided in an appendix. the author asked students about instructor presence, but the reader does not know how it was defined in the questionnaire. the study includes student quotes in the discussion, but it does not indicate what questions the author asked or if the author sought ethics approval or if approval was required. the author presents a large amount of quantitative data in six tables in the results related to communication, social media, modes of instruction, and assignment frequency, delivery, and assessment. the author also includes several correlation analyses but does not reference the correlations in the discussion. this begs the question why they were done, especially as there is no hypothesis related to these correlations. the author finds a correlation between synchronous lectures and activities but indicates that most students prefer asynchronous modes and yet does not provide a correlation analysis for this conclusion.    the discussion addresses each of the research questions asked and ties in the results to the literature. the author acknowledges the limitation of this study: a sample size of 45 graduate students. the author connects the results to the coi framework with respect to social presence (communication frequency and tools) and teaching presence (modes of instruction) in the discussion.   this is a timely topic with many institutions providing online instruction for the 2020–21 school year. despite its flaws, this exploratory study provides a generous amount of data about graduate student preferences for communication in an online environment as well as preferences for learning modes, activities, and assessment. it points to flexibility and engagement being critical for online learning, which is a valuable insight for instructors designing and delivering online courses.    references          arbaugh, j., cleveland-innes, m., diaz, s., garrison, d., ice, p., richardson, j., & swan, k. (2008). developing a community of inquiry instrument: testing a measure of the community of inquiry framework using a multi-institutional sample. the internet and higher education, 11(3–4), 133–136. https://doi.org/10.1016/j.iheduc.2008.06.003   garrison, d. r., & akyol, z. (2013). the community of inquiry theoretical framework. in m.g. moore (ed.), handbook of distance education (pp. 104–119). philadelphia, pa: routledge.   kozan, k., & richardson, j.c. (2014). interrelationships between and among social, teaching, and cognitive presence. internet and higher education, 21, 68–73. https://doi.org/10.1016/j.iheduc.2013.10.007   perryman, c., & rathbun-grubb, s. (2014). the cat: a generic critical appraisal tool. in jotform – form builder. retrieved from http://www.jotform.us/cp1757/thecat     a case study on how reference staffing and visibility models impact patron behaviors evidence summary   a case study on how reference staffing and visibility models impact patron behaviors   a review of: holm, c.e. & kantor, s. (2021). reference is not dead: a case study of patron habits and library staffing models. portal: libraries and the academy, 21(2), 299–316. https://doi.org/10.1353/pla.2021.0017   reviewed by: matthew bridgeman, mlis information and education librarian robert wood johnson library of the health sciences rutgers, the state university of new jersey new brunswick, new jersey, united states of america email: mcb226@libraries.rutgers.edu   received: 1 dec. 2021                                                                accepted:  25 jan. 2022      2022 bridgeman. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30084     abstract   objective – to determine if reference staffing models are a predictor of reference question rates and if academic library patrons’ reference behaviors are linked to reference staffing models and desk visibility.   design – a retrospective case study.   setting – two academic libraries at a large r3 public university in the state of georgia, united states of america.   subjects – 10,295 service transactions (chat and in-person, including non-reference transactions related to directional and technology questions) from the 2016 fiscal year and 6,568 service transactions (chat and in-person, including only chat non-reference transactions) from fy 2017.   methods – analysis of two years of service transaction data (july 2015 to june 2017) recorded by librarians using the reference analytics module of springshare’s libanswers at three locations (virtual 24/7 chat and two libraries with different physical locations, such as centrally-located or harder-to-find service points) for three kinds of reference service modes: chat, fully-staffed in-person services, and occasional “on-call” services. “reference” transactions were classified using the reference & user services association (rusa) definition. email, sms/text, and facebook inquiries were excluded from this study. one library, which had the same service model for the 2016-2017 fiscal years, served as the study’s “control” so that an analysis of service model alterations could be conducted. main results – the rate of chat reference remained steady, independent from the desk model employed. there was also an overall decline in reference questions from fy 2016 to fy 2017. for fy 2016, the average daily chat transaction rate was 16.1 inquiries (range: 0 inquiries for some days and up to 51 for others) compared to an average 20.5 inquiries at the two physical service locations (range: 0 to 95 inquiries per day). in fy 2017, the average daily chat transaction rate was 13.9 inquiries (range: 0 to 46 inquiries per day) compared to 6.8 transactions for the physical locations (range: 0 to 19 inquiries per day). for fy 2016, when the model shifted to on-call, the average daily chat transaction rate was 14 inquiries compared to the physical locations with 0 and .67 inquires per day. in fy 2017, the averages were 19.33 for chat compared to .33 and .33 for the physical locations.   conclusion – for the two fiscal years studied here, question rates and reference behaviors seemed to be linked to staffing models. patrons in this study preferred a staffed and visible desk and 24/7 chat, while “on-call” services were not favored. by replacing the visible desk with an on-call model, the library created a situation where chat was the only consistent reference service offering. as a result, patrons may have viewed the visible desk as being unreliable. the on-call service model appears to have negatively affected patron behavior since, according to the data presented, patrons’ reference needs were best met by chat and a visibly staffed desk service model.   commentary                this article opens by asking the question “is reference dead?” it is one of the most pressing issues in librarianship (bowron & weber, 2019; seal, 2011). as the authors mention in their literature review, librarians have been prophesizing the collapse of the reference service since the dawn of the internet in the 1990s. if anyone can search the internet for information, why seek assistance from a librarian? this question persists and evolves through time.   to review this study, the critical appraisal tool design by glynn (2006), appropriate for evaluating a retrospective case study involving quantitative data, was used. data collection methods and the target population for the study were clearly stated, and the large number of reference transactions from multiple locations provided a substantial pool of data for analysis, though from a single institution, making it difficult to generalize and validate the findings without comparisons to others studies. future investigations would be strengthened by improved sampling mechanisms and data from multiple institutions.   the authors used springshare’s libanswers to gather the information, which provided an effective tool for data collection. however, the accuracy of data entry into the system by librarians may have influenced the integrity of the data, a limitation the authors address to the best of their ability (p. 305). furthermore, the inclusion/exclusion criteria for reference questions were based on the rusa guidelines, a matter of interpretation also addressed by the authors (p. 305). despite these limitations, the trends presented are clear, though the authors do not adequately address the overall decline in reference transactions during the years investigated. the decline could be a result of the staffing model alterations, but this is conjecture.   are reference services dead? this study, while not answering this question fully, provides a backdrop for institutions considering this question, providing a methodology other libraries might use in analyzing service and visibility models, crucially important to current practice in the wake of the covid-19 pandemic, where many libraries are struggling to find the closest equivalent to a “visible” reference desk in the virtual space.   references   bowron, c. r., & weber, j. e. (2019). the evolving reference desk: a case study. journal of academic librarianship, 45(5), 102057. https://doi.org/10.1016/j.acalib.2019.102057   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387–399. https://doi.org/10.1108/07378830610692154   kostagiolas, p., & katsani, a. (2021). the management of public libraries during covid-19 pandemic: a systematic literature review through prisma method. library management, 42(8), 531–549. https://doi.org/10.1108/lm-08-2021-0068   seal, r. a. (2011). trends, issues, and innovation in academic library service: introduction. journal of library administration, 51(3), 255–258. https://doi.org/10.1080/01930826.2011.556953   microsoft word news_asist.doc evidence based library and information practice 2007, 2:2  127 evidence based library and information practice     news    asist sig iii’s international paper contest raffle      dr j. k. vijayakumar  assistant director, health sciences library  american university of antigua  antigua, west indies  e‐mail: vijay@auamed.net      © 2007 vijayakumar. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    the american society for information  science and technology’s sig for  international information issues’  fundraising raffle for this year’s  international paper contest is in full swing.  the grand‐prize winner will go home with a  sophisticated dell inspiron digital  notebook. the fundraiser kicked off on  april 1, 2007, and will conclude with a draw  during the 2007 international reception at  the asist annual meeting in milwaukee,  wisconsin in october 2007. only 250 tickets  will be sold, thus each entrant has an  excellent chance of winning the grand prize.  tickets are sold on a first come, first served  basis. monies raised will promote  international research by benefiting the  winners of the 2007 international paper  contest.    tickets are available at  http://www.asist.org/sig/sigiii/fundraising /fundraising2007/    the purposes of sig iii are:    • to promote better awareness among  asist members and information  professionals of the importance of  international cooperation  • to facilitate and enhance better  communication and interaction among  asist members and their foreign  colleagues on information issues  • to develop an international network of  digital scholars and experts on digital  libraries and information technology in  developing countries  • to provide a forum for exploring and  discussing international information  issues and problems    sig iii membership includes most non‐u.s.  asist members, and a true cross‐section of  u.s. asist members.    mailto:vijay@auamed.net http://creativecommons.org/licenses/by/2.0 http://www.asist.org/sig/sigiii/fundraising evidence based library and information practice 2007, 2:2  128 more information about the paper contest  can be found on the award winning sig iii  website under paper contest:   (http://www.asis.org/sig/sigiii/index.htm)                http://www.asis.org/sig/sigiii/index.htm microsoft word art1_855_wpage evidence based library and information practice 2008, 3:1  3    evidence based library and information practice         article    developing a comprehensive search strategy for evidence based systematic  reviews      julia b. deluca   information specialist  centers for disease control and prevention  national center for hiv/aids, hepatitis, std and tb prevention  division of hiv/aids prevention  prevention research branch  atlanta, ga, united states of america  email: zxz4@cdc.gov    mary m. mullins  information specialist  centers for disease control and prevention  national center for hiv/aids, hepatitis, std and tb prevention  division of hiv/aids prevention  prevention research branch  atlanta, ga, united states of america  email: asu8@cdc.gov    cynthia m. lyles  mathematical statistician  centers for disease control and prevention  national center for hiv/aids, hepatitis, std and tb prevention  division of hiv/aids prevention  prevention research branch  atlanta, ga, united states of america  email: cml6@cdc.gov    nicole crepaz  behavioral scientist  centers for disease control and prevention  national center for hiv/aids, hepatitis, std and tb prevention  division of hiv/aids prevention  prevention research branch  atlanta, ga, united states of america  email: ncc9@cdc.gov  evidence based library and information practice 2008, 3:1  4   linda kay  behavioral scientist  centers for disease control and prevention  national center for hiv/aids, hepatitis, std and tb prevention  division of hiv/aids prevention  prevention research branch  atlanta, ga, united states of america  email: lsk0@cdc.gov    sekhar thadiparthi  business analyst  centers for disease control and prevention  national center for hiv/aids, hepatitis, std and tb prevention  division of hiv/aids prevention  prevention research branch  atlanta, ga, united states of america  email: zkv9@cdc.gov      received: 11 december 2007      accepted: 10 february 2008      © 2008 de luca et al.  this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective ‐ within the health care field it becomes ever more critical to conduct  systematic reviews of the research literature to guide programmatic activities,  policy‐making decisions, and future research. conducting systematic reviews  requires a comprehensive search of behavioural, social, and policy research to  identify relevant literature. as a result, the validity of the systematic review findings  and recommendations is partly a function of the quality of the systematic search of  the literature. therefore, a carefully thought out and organized plan for developing  and testing a comprehensive search strategy should be followed. this paper uses the  hiv/aids prevention literature to provide a framework for developing, testing, and  conducting a comprehensive search strategy looking beyond rcts.    methods ‐ comprehensive search strategies, including automated and manual  search techniques, were developed, tested, and implemented to locate published  and unpublished citations in order to build a database of hiv/aids and sexually  transmitted diseases (std) literature. the search incorporated various automated  and manual search methods to decrease the chance of missing pertinent  information. the automated search was implemented in medline, embase,  psycinfo, sociological abstracts and aidsline. these searches utilized both index  evidence based library and information practice 2008, 3:1  5 terms as well as keywords including truncation, proximity, and phrases. the  manual search method includes physically examining journals (hand searching),  reference list checks, and researching key authors.     results ‐ using automated and manual search components, the search strategy  retrieved 17,493 articles about prevention of hiv/aids and stds for the years 1988‐ 2005. the automated search found 91%, and the manual search contributed 9% of  the articles reporting on hiv/aids or std interventions with behavioural/biologic  outcomes. among the citations located with automated searches, 48% were found in  only one database (20% medline, 18% psycinfo, 8% embase, 2% sociological  abstracts).    conclusions ‐ development of a comprehensive review of the literature requires  searching multiple databases and methods of manual searching in order to locate all  relevant citations. understanding the project needs, recognizing the limitations and  strengths of specific electronic databases, and being aware of other methods for  developing and refining a search are vital in planning an effective and  comprehensive search strategy. reporting standards for literature searches as part of  the broader push for procedurally transparent and reproducible systematic reviews  is not only advisable, but good evidence based practice.      introduction    within evidence based health care it  becomes ever more critical to conduct  systematic reviews of research literature to  guide programmatic activities, policy‐ making, and future research. this increasing  reliance of the health care field (e.g.,  medicine, nursing, radiology, and social  work) on evidence based research highlights  the importance of a comprehensive and  systematic search and examination of the  best scientific literature (perry 3). research  synthesis, including systematic reviews and  meta‐analysis, is a valuable tool in tracking  down the strongest scientific evidence to  move prevention into program practice and  to answer clinical questions (sackett 72).    high quality research synthesis consists of a  systematic process for identifying, critically  appraising, synthesizing, and translating the  information. the first important step in  systematic reviews is to develop and  implement a search strategy for identifying  the relevant scientific literature available  (jenuwine and floyd 349; howes et al. 101;  bruce and mollison 14; mcmanus et al. 1562;  chalmers, dickersin, and chalmers 786;  crumley and blackhall 167). without  identifying the relevant literature in a  systematic, comprehensive, and  reproducible way, the information retrieved  in the search may not be complete and may  potentially lead to differing and potentially  conflicting recommendations (conn et al.  181; counsell 384). despite the critical  implications of a flawed or incomplete  search, published systematic reviews and  meta‐analysis often provide a limited  explanation of the search methods used to  capture the literature (booth 422; weller 163;  zhang, sampson, and macgowan 5).  determining which electronic databases will  be searched; knowing how to apply  indexing terms, key words, and key phrases  to refine a search; deciding whether both  automated and manual components are  necessary; and knowing how to combine  various components are all important  evidence based library and information practice 2008, 3:1  6 aspects of producing a high quality  systematic search.    the hiv/aids and std prevention field has  recently begun to rely on research synthesis  to prioritize funding program efforts. the  u.s. center for disease control and  prevention’s (cdc) prevention research  synthesis (prs) project has conducted many  systematic reviews and meta‐analyses to  provide important recommendations for  developing efficacious prevention  programs. high risk groups, such as men  who have sex with men (msm), hiv‐ positive, african american, and hispanic  populations have been critically reviewed to  evaluate the efficacy of behavioural  interventions and to identify research gaps  (crepaz et al. “prevention;” crepaz et al.  “efficacy;” herbst et al. “meta‐analytic  msm;” herbst et al. “hispanic;” herbst et  al. “effectiveness msm;” johnson et al.;  simoni et al.). other types of systematic  reviews have identified specific hiv  behavioural interventions demonstrating  strong evidence of efficacy in reducing hiv  risk (lyles et al. “evidence‐based;”evidence  based interventions). the findings of these  systematic reviews have promoted the us  national dissemination of evidence based  hiv intervention (collins et al.; neumann  and sogolow), and have been incorporated  into the hiv prevention strategic plans of  state health departments across the usa  (peterson and randall; shea et al.).    systematic reviews of intervention research  have typically focused on randomized  controlled trials (rcts) (berry et al.;  boynton et al.; crumley et al. “resources”;  dickersin, scherer, and lefebvre; helmer et  al.; higgins and green; jadad, moher, and  klassen; robinson and dickersin; savoie et  al.; watson and richardson). in hiv  prevention research, however, a wealth of  intervention research exists that utilizes  nonrandomized evaluation designs. some  recent articles have called for an expanded  review of the scientific evidence by  including these types of studies in order to  provide a more integrated picture of the  existing state‐of‐science and broader  evidence based recommendations for public  health practice (atkins, fink, and slutsky;  victora, habicht, and bryce). incorporating  evidence from a broader range of literature  beyond rcts requires a more general  systematic search of the literature and, thus,  a more complex search strategy.    this paper is intended to provide a  framework for developing, testing, and  conducting a comprehensive search strategy  looking beyond rcts, using the prs project  and its research into hiv/aids behavioural  prevention literature between 1988 and  2005. the contributions of automated and  manual searches, as well as the citations  identified in each electronic database used  in the automated search, were analyzed to  assess their overall importance in  information retrieval within the hiv/aids  behavioural intervention research field.    cdc’s hiv/aids prevention research  synthesis project    an understanding of the context and  purpose of a research synthesis project is  essential to guide the development of the  project’s search strategy. the prs project  was initiated in 1996 by the division of  hiv/aids prevention (dhap) at the cdc  to translate the cumulative scientific  literature into evidence based  recommendations for guiding  programmatic activities, policy‐making, and  future research. the specific goals of the  project are to:    • systematically identify and catalog  the hiv/aids or sexually  transmitted diseases (std)  behavioural prevention research  literature;  evidence based library and information practice 2008, 3:1  7 • conduct systematic reviews and  meta‐analyses to understand the  overall efficacy of different types of  behavioural interventions and to  identify efficacious components of  those interventions;  • identify behavioural interventions  with evidence of efficacy; and  • assess gaps in the existing literature  for future research (sogolow et al.).    an internal database of hiv/aids  behavioural prevention literature was  created to serve as the resource for all prs  research synthesis activities. this database  has been the foundation for many peer‐ reviewed publications (crepaz et al.  “prevention;” crepaz et al. “efficacy;”  crepaz et al. “hiv‐positive;” passin et al.;  herbst et al. “meta‐analytic msm;” herbst  et al. “hispanic;” herbst et al. “effectiveness  msm;” lyles et al. “best‐evidence”).    due to the varied goals of the prs project,  several unique requirements for establishing  the prs database had to be considered when  developing a systematic search. the focus of  the prs project is to evaluate the efficacy of  hiv behavioural interventions. although  this focus is relatively narrow, the goals of  the prs project required the inclusion of all  hiv/aids and std behavioural prevention  research literature in the prs database.  thus, developing a broader scope in the  systematic search allowed for reviews of  background literature or formative research  in hiv behavioural prevention. it was also  important to understand that hiv  prevention literature cuts across multiple  research areas including medicine, public  health, nursing, psychology, sociology,  anthropology, and social work. searching  across a wide range of research fields is  necessary to ensure the comprehensiveness  of the prs database.  these features of the  project and the literature made this an ideal  topic area for illustrating a framework for  developing a systematic search strategy that  may be of use in a range of health and social  care fields.  methods    developing a systematic search to  accommodate the unique aspects of the prs  project required extensive planning and  testing. to identify all relevant citations  across multiple sources and within a broad  scope, the search strategy consisted of  automated and manual search components.  the national library of medicine (nlm)  estimates 13,000 to 14,000 biomedical  journals are currently being published  worldwide, and that these journals can be  searched through various electronic  databases (nlm factsheet). no single  electronic database offers complete indexing  of all available citations; therefore multiple  databases need to be searched (crumley and  blackhall 167; alpi 98). manual searching  was included to complement automated  searches, because relying on electronic  databases alone would have left the searches  vulnerable to missing information or to  errors in indexing.     automated search    focusing the topic area    the focus of the prs project – evaluating the  efficacy of hiv behavioural interventions –  was separated into three unique and equally  important domains to help structure and  guide the development of the automated  search:    • hiv, aids, or sexually transmitted  disease (std)  • prevention, intervention, or  evaluation  • behavioural or biologic outcomes  measuring potential or actual hiv  or std risk reduction.     citations referring to hiv/aids/std  behavioural prevention research were  evidence based library and information practice 2008, 3:1  8 considered within prs as “in‐scope”  citations, whereas, citations that  encompassed all three domains (i.e.,  hiv/aids/std behavioural intervention  citations with behavioural or biologic  evaluation data) were the primary focus of  the prs project.     database selection    the prs project searched four electronic  databases to cover a broad range of health  related literature: medline, embase,  psycinfo, and sociological abstracts using  the ovid platform. these are key databases  for biomedical, psychological, behavioural  science, and public health literature.  aidsline, a database specifically dedicated  to hiv/aids research, was also included in  the prs search until its discontinuation in  december 2000. the search was initially  developed, tested, and implemented in  medline, “the most widely used and  studied database in health care” (jadad,  moher, and klassen 812). core indexing  terms, keywords, and key phrases were the  main elements of the automated search  component. each step in developing the  automated search component was  conducted simultaneously for each of the  three key domains.     any group conducting systematic searching  and review will ultimately realize there is a  limited amount of time and resources  available (crumley and blackhall 167; jadad,  moher, and klassen 812). databases were  selected according to the capabilities, time  constraints, and resources available.  databases considered and tested for  inclusion, but ultimately not used were  cinahl, current contents, eric, and isi  web of knowledge (social science citation index  and science citation index expanded). results  of those tests concluded that citations from  cinahl would have duplicated results and  would not contribute significant findings  beyond those identified in medline.  current contents was not selected due to  the focus of the database – it did not offer  specialized or significant insight into public  health areas outside the findings of the other  included databases. the focus of eric is on  educational and youth populations, an area  researched by another cdc entity, the  division of adolescent and school health  (dash). isi’s web of knowledge provides  valuable linking information, but its  indexing system does not use a controlled  vocabulary. isi databases were used in the  manual search process to verify references  between related articles.    testing each database    testing of each database consisted of using  the same indexing, keywords, and phrases  to retrieve and analyze the citations  collected. a sample search strategy was  created based on 75 hiv/aids or std  behavioural intervention evaluation  citations. the search was used to determine  if that group of articles would be retrieved  by each database. this test sought to answer  certain questions:   • how many citations were retrieved?   • are these articles within the scope  of the project?   • how many extraneous citations  were collected with this search?    testing was performed in all databases  considered, but only the selected databases  were fully searched.    developing an automated search    the first step in developing the automated  search component was to identify medical  subject headings (mesh) terms in  medline. the 75 previously mentioned  evaluation citations were examined to create  a list of mesh terms by conducting inverse  searching – identifying terms by analyzing  the subject indexing in order to expand the  search (boynton et al. 141). each potential  mesh term was evaluated in the national  evidence based library and information practice 2008, 3:1  9 library of medicine’s mesh database,  examining the entry term, scope note,  subheadings, related terms, and previous  indexing for each of the terms from the 75  citations. the list of mesh terms was  compiled separately for each of the three  primary domains of the search. initially, 116  possible mesh terms were identified. after  cross‐checking the indexing of all 75  citations and then sampling citations  retrieved with these terms, the pool  narrowed to 42 mesh terms for the  automated search component for medline.    once completed for medline, the list of  indexing terms was tested, and adapted  where necessary, to accommodate  differences in the remaining electronic  databases. matching subject terms between  databases was complex in some cases.  sometimes similar indexing terms do not  exist between databases, creating gaps in  search components. the scope note for each  indexing term contains both a definition and  also lists of “used for” and “related terms”  to help pinpoint the specific intent and  usage of that subject heading. each term  was checked individually and then its  related terms were also tested, making it  possible to narrow or broaden the search as  needed.    with indexing terms identified, the  automated search was further refined by  using subheadings (i.e., subcategories of  descriptors within indexing terms).  subheadings, available only in medline,  aidsline and embase, were used to  refine an indexing term when the concept  was too broad to be an effective search term  (greenhalgh 181). after carefully reviewing  the scope notes, the “prevention and  control” subheading was selected to  increase the precision of the search. adding  this subheading effectively reduced the  number of irrelevant citations and  substantially decreased the volume of  citations.  refining the automated search    another method for refining searches was to  use the “explode” and “focus” options  provided by the ovid search platform. for  the hiv/aids/std domain, using the focus  option (indicated by an asterisk in ovid)  ensured the search would return citations  specific to a particular subject heading or  subheading (e.g., “hiv infection/pc*”). for  the intervention evaluation and outcome  domains, focusing seemed to limit the  automated search too much, while  exploding (“exp” in ovid) captured a large  number of irrelevant citations. using a  variety of relevant and sufficiently narrow  indexing terms (e.g., “evaluation studies,”  “intervention studies,” “sexual behavior”),  without further focusing or exploding these  terms, proved to be the most effective route  for searching.      while the use of indexing terms is critical in  locating relevant citations, indexing can be  limited by numerous factors, including  poorly written abstracts, misclassification, a  lack of appropriate index terms, and lag  time in indexing (boynton et al. 139).  because of these limitations, a search  devised with mesh or indexing terms alone  may fail to capture all essential information.  therefore, inverse searching was conducted  within the title and abstract of the 75  citations to identify relevant keywords and  phrases for each of the domains to expand  to the automated search strategy. examining  the results produced by the test search also  helped to determine which indexing and  keyword terms were important to the final  project search.    once keywords and phrases were identified,  two techniques – truncation and proximity –  were used to refine this aspect of the  automated search. truncation (expressed as  “$” in ovid) was used in many cases to  find all spelling variants of a root term.  however, using truncation alone sometimes  evidence based library and information practice 2008, 3:1  10 retrieved an unmanageable number of  citations. to narrow the search, specific  keyword phrases were used when needed  (i.e., “controlled trial,” “control group”).  proximity operators for concept and phrase  searching (“adj” in ovid) were used to  identify two keywords near each other or  variations in a particular phrase (e.g.,  “infect$ adj4 rate$”). after all aspects of the  automated search were identified (electronic  databases, indexing terms, keywords, and  phrases), and after applying all search  refinements (subheadings, exp, focus,  truncation, proximity) for each of the three  key domains of the prs project, the  automated search strategy was finalized  using boolean operators to combine  elements within domains (“or”) and cross‐ referencing between domains (“and”).   (see the complete prs automated search  strategies in appendixes a‐c.)    keeping the search up to date    an important aspect of the prs project is  keeping the internal database current. due  to the volume of citations retrieved with  each search, it is only possible to update the  search annually to identify the most recent  literature. however, the annual search is  completed for a three‐year time period – one  initial search for each specific year, and two  subsequent searches in following years. for  instance, the annual search in 2006 included  publications for years 2004, 2005, and 2006.  this search provided the second update for  2004 publications, the first 2005 update, and  the initial 2006 search. to ensure consistency  over time, the database indexing terms are  regularly checked for additions, deletions,  changes, or replacements. changes in  subject jargon and vernacular also evolve  with time. any needed adjustments to  indexing terms or subject language are then  incorporated into the search strategy, so that  retrieval of the literature is kept up‐to‐date  (brettle et al. “searching” 166).    manual search    three manual search methods were used to  complement the automated search  component and to provide additional search  coverage for a comprehensive search  strategy. while a time‐consuming endeavor,  hand searching has proved to be a more  direct and explicit way of retrieving relevant  articles omitted by databases due to  indexing inaccuracies or because of lag time  in the indexing process (avenell, handoll,  and grant 509; counsell 386; jadad, moher,  and klassen 813). a manual review of the  issues from 35 key journals is conducted  biannually to identify citations relevant to  the focus of the prs project. the key  journals are selected by querying the prs  database for those journals producing the  greatest number of hiv behavioural  intervention citations with behavioural or  biologic outcomes. because this number  may vary across publication year, the  journal selection is re‐evaluated annually.  (appendix d contains the list of journals  currently searched). the second manual  search method involves networking with  researchers to obtain relevant published and  unpublished reports. finally, the prs  project team continuously checks other  sources to identify additional citations.  these include relevant electronic mail lists,  clinical trial databases (e.g., cochrane  library, crisp database), conference  proceedings, and reference lists of relevant  hiv behavioural prevention research  literature.    assessing the search    downloading citations into a general database  (prs)    all automated and manual searches are  conducted by year and by database (e.g.,  medline 2002). this approach narrows the  number of citations to a manageable number  for checking and downloading. the citations  evidence based library and information practice 2008, 3:1  11 identified from the same database in a  specific year are assigned to a unique batch  number, thus allowing the examination of  the contribution of each database by year.  using the ovid platform, the citations are  downloaded in batches from the each  electronic database to biblioscape reference  information manager™ software. the prs  database is designed to check for  duplications between imported citations and  those already existing in the database. this  process identifies the number of times a  citation is found and in which database,  leading to a calculation of the percentage of  overlap between databases.    calculation of search component contributions    using the batch number information, three  indexes were calculated to evaluate the  contributions of each search component (i.e.,  automated versus manual searches) and of  each electronic database. these indices  quantified overall contribution, unique  contribution, and overlap. the indices are  defined below for a search of two electronic  databases, where a and b represent the  number of citations found in database a and  b, respectively, and where c is the number  of citations found in both databases (or the  overlap).     the overall contribution (%) of  electronic database a is calculated as:    100 )( x cba a ⎥ ⎦ ⎤ ⎢ ⎣ ⎡ −+     the unique contribution (%) of  electronic database a is calculated as:    ( ) 100 )( x cba ca ⎥ ⎦ ⎤ ⎢ ⎣ ⎡ −+ −     the percent overlap between  electronic databases a and b is calculated  as:    100 )( x cba c ⎥ ⎦ ⎤ ⎢ ⎣ ⎡ −+     these same formulas were used when  comparing manual to automated methods  and were generalized when comparing  three or more electronic databases.     results    as of december 2006, after completing the  comprehensive search strategy for the years  1988‐2005, a total of 17,493 citations have  been identified via automated searches of  the electronic databases medline,  embase, psycinfo, and sociological  abstracts (table 1). the manual search  component, including hand searching  journals, reference lists, electronic mail lists,  and networking with researchers, have  identified 1,232 citations. after checking for  overlap between automated and manual  search results, there were 615 citations  unique to the manual search component.  the comprehensive systematic search from  1988‐2005 identified a combined total of  18,108 citations.  aidsline identified an  additional 302 citations for the years 1988‐ 2000, but these citations are not included in  the calculations below. evidence based library and information practice 2008, 3:1  12     all citations    overall contribution  unique contributiona      n     %b      n    %c  database            medline  8,279  47% 4,875  28%    psycinfo  7,343  42% 4,659  27%    embase  5,716  33% 2,677  15%    soc abstracts  1,776  10% 754  4%  automated total†  17,493 ‐ 12,965  74%d          search method         automated total  17,493  97% 16,876  93%   manual total    1,232   7%     615  3%  combined total††  18,108  ‐ 17,491  97%d    table 1. all citations retrieved by database and method, 1988‐2005    a citations that were identified by only one database or method.      b denominator is the overall contribution total. combined percentages total more than 100% due  to overlap.  c denominator is the overall contribution total. combined percentages total less than 100% due to  overlap.  d percent of total citations that were identified by only one database or method.  † the same citation was identified in 2 or more databases resulting in multiple records of the  same citation. duplicated records were removed from overall contribution totals. for all citations,  4,528 citations (26%) were identified by 2 or more databases, resulting in 5,621 duplicate records.  †† the same citation was identified by both methods, resulting in duplicate records that are not  reflected in the combined total. for all citations, 617 citations (3%) were identified by both  methods.                                    evidence based library and information practice 2008, 3:1  13 overall contribution of each electronic  database    among electronic databases from the  automated search component, medline,  psycinfo, and embase generated the  greatest retrieval of all citations (n=8,279,  47%; n=7,343, 42%; and n=5,716, 33%,  respectively, of 17,493 citations) (table 1).  sociological abstracts contributed 10%  (n=1,776) of the total number of citations  identified by the automated search  component. this pattern for overall  contribution remained similar for particular  types of citations, where medline,  psycinfo, and embase continued to  identify the most in‐scope citations (44%,  55%, 33%, respectively) (table 2) and  behavioural/biologic citations (65%, 50%,  46%, respectively) (table 3). psycinfo  identified the greatest number of in‐scope  citations (n=5,085, 55%); medline  identified the greatest number of  behavioural/biologic citations (n=712, 65%);  and sociological abstracts identified the  fewest of both (14% and 10%, respectively).     unique contribution of each electronic  database    of the 17,493 citations identified from the  automated search, 74% (n=12,965) were  identified by only one electronic database  (table 1). in other words, only 26% of the  citations were identified by two or more  databases. when considering the citations  more specific to the prs’ primary focus (i.e.,  behavioural/biologic citations), a large  portion (48%) was still identified by only  one database.    the unique contributions were greatest for  medline and psycinfo across all citation  types: 28% (n=4,875) of all citations were  identified by medline alone, and 27%  (n=4,659) were identified only by psycinfo  (table 1). embase alone accounted for 15%  (n=2,677) of all citations, and sociological  abstracts alone accounted for 4% (n=754).  the unique contribution for in‐scope  citations was greatest for psycinfo (31%);  the unique contribution for  behavioural/biologic citations was greatest  for medline (20%). although most of the  behavioural/biologic citations were  identified by either medline or psycinfo,  another 10% were identified only by  embase (8%), or by sociological abstracts  (2%).     percent overlap among electronic databases    since medline, psycinfo, and embase  contributed the greatest number of citations  across all citation types, these databases  were used to examine overlap. the percent  overlap was calculated for each pair‐wise  comparison as well as for the overlap across  all three electronic databases (table 4). the  percent overlap between medline and  psycinfo was 12% and varied by year. a  similar percent overlap was observed  between embase and psycinfo (11%  overall. interestingly, even though  medline and embase both contain a  biomedical focus, the percent overlap was  only 20%, which is only one‐fifth of a total  11,697 citations identified from either  database. when searching all three  electronic databases, the percent overlap  was 23%, where 19% of the citations were  identified in 2 of the 3 databases and only  4% were identified in all 3 databases (figure  1a).   evidence based library and information practice 2008, 3:1  14   in‐scope citations    overall contribution   unique contributiona      n     %b      n    %c  database            medline  4,073  44%  1,713  19%    psycinfo  5,085  55%  2,908  31%    embase  3,040  33%  932  10%    soc abstracts  1,271  14%  431  5%  automated total†  9,249  ‐  5,984  65%d            search method           automated total  9,249  94%  8,657  88%   manual total  1,162  12%  570  6%  combined total††  9,819  ‐  9,227  94%d    table 2. in‐scope citations by database and method, 1988‐2005      a citations that were identified by only one database or method.      b denominator is the overall contribution total. combined percentages total more than 100% due  to overlap.  c denominator is the overall contribution total. combined percentages total less than 100% due to  overlap.  d percent of total citations that were identified by only one database or method.  † the same citation was identified in 2 or more databases resulting in multiple records of the  same citation. duplicated records were removed from overall contribution totals. for in‐scope  citations, 3,265 citations (35%) were identified by 2 or more databases, resulting in 4,220 duplicate  records.  †† the same citation was identified by both methods, resulting in duplicate records that are not  reflected in the combined total. for in‐scope citations there were 592 (6%) citations identified.  evidence based library and information practice 2008, 3:1  15   intervention citations with  behavioural/biologic outcomes    overall contribution  unique contributiona      n     %b      n    %c  database            medline  712  65%  218  20%    psycinfo  548  50%  200  18%    embase  508  46%  91  8%    soc abstracts  110  10%  21  2%  automated total†  1,097  ‐  530  48%d            search method           automated total  1,097  91%  906  75%   manual total  300  25%  109  9%  combined total††  1,206  ‐  1,015  84%d    table 3. intervention citations by database and method, 1988‐2005    a citations that were identified by only one database or method.      b denominator is the overall contribution total. combined percentages total more than 100% due  to overlap.  c denominator is the overall contribution total. combined percentages total less than 100% due to  overlap.  d percent of total citations that were identified by only one database or method  † the same citation was identified in 2 or more databases, resulting in multiple records of the  same citation. duplicated records were removed from overall contribution totals. for  intervention citations with behavioural or biologic outcomes, 567 citations (52%) were identified  by 2 or more databases, resulting in 781 duplicate records.  †† the same citation was identified by both methods, resulting in duplicate records not reflected  in the combined total. for intervention citations with behavioural or biologic outcomes, 191 (16%)  citations were identified.  evidence based library and information practice 2008, 3:1  16   all citationsa  intervention citations with  behavioural/biologic outcomesb  total  uniquely  identifiedc  overlap  total  uniquely  identifiedc  overlap  n % n % medline & psycinfo  13,937  12,252  1,685  12%    977  694  283  29%  medline & embase  11,697  9,399  2,298  20%    861  502  359  42%  embase & psycinfo  11,764  10,469  1,295  11%  847  638  209  25%  medline, psycinfo,  & embase  16,739  12,819  3,920*  23%  1,076  543  533**  50%    table 4. overlap of citations between medline, psycinfo, and embase, searching the 1988‐ 2005 literature    a the number of citations identified by each database: medline (n=8,279),  psycinfo (n=7,343), embase (n=5,716)  b the number of intervention citations with behavioural/biologic outcomes  identified by each database: medline (n=712), psycinfo (n=548), embase  (n=508)  c citations identified by either one, but only one, database  *   3,241 citations (19%) were identified in 2 of the 3 databases; 679 (4%) were  identified in all 3  ** 374 citations (35%) were identified in 2 of the 3 databases; 159 (15%) were  identified in all 3  evidence based library and information practice 2008, 3:1  17 for behavioural/biologic citations, the  percent overlap between databases  increased substantially compared to the  percent observed among all citations (table  4). the greatest overlap was again between  medline and embase (42%). the percent  overlap between medline and psycinfo  was 29%, and between embase and  psycinfo the overlap was 25%. the percent  overlap for behavioural/biologic citations  among all 3 databases was 50%, where 35%  of the citations were identified in 2 of the 3  databases. only 15% were identified in all 3  databases (figure 1b).   evidence based library and information practice 2008, 3:1  18 contribution of the search methods: manual  and automated    overall, the manual search method added  7% of the citations to the prs database, in  contrast to a contribution of 97% for the  automated search method (table 1). the  contribution of the manual search method  increased to 12% for in‐scope citations and  to 25% for behavioural/biologic citations,  while the contribution for the automated  search method decreased slightly to 91% for  behavioural/biologic citations. the increase  in overall contribution by the manual search  method was accompanied by an increase in  overlap with the automated search  component, an increase in unique  contribution of the manual search, and a  decrease in unique contribution of the  automated search. of the 18,108 citations  identified overall, only 3% were identified  by both methods (table 1). there was  greater overlap for in‐scope citations (6%)  (table 2) and for behavioural/biologic  citations (16%) (table 3). as far as unique  contributions, 6% (n=570) of the in‐scope  citations (table 2) and 9% (n=109) of the  behavioural/biologic citations (table 3) were  identified exclusively by the manual search  component. these 109 uniquely identified  behavioural/biologic citations, which are the  primary focus of the prs project, would  have been missed if only automated  searches were conducted.     discussion          developing, testing, and implementing a  systematic search for research synthesis is  an extremely complex and time‐intensive  task. librarians should be considered an  integral part of the systematic review  process in the evidence based healthcare  field (beverley, booth, and bath 66;  mcgowan and sampson; sampson and  mcgowan 1057; helmer et al. 347; harris 86;  zhang, sampson, and mcgowan 5).  although the specifics of this search strategy  focus on the prs project needs, the general  framework for developing a systematic  search strategy as described within this  paper may be valuable to other research  synthesis projects in a variety of disciplines,  and it adds to the methods for search  strategy development already presented  (matthews et al; mcnally and alborz). a  thorough evaluation of any systematic  review cannot happen without explicit  documentation of the process used by  researchers. additionally, it is difficult to  place confidence in the findings of a  systematic review, if the systematic search  used to generate the pool of literature for  that review is not clearly reported (booth  426; sampson and mcgowan 1160; weller  163; patrick et al. 199).    during the development, testing, and  refinement of the prs search strategy,  various methods and techniques were  selected to ensure the breadth and depth of  the search. using both indexing terms and  keyword searching can improve the  retrieval of relevant citations. these  techniques complement each other and will  help overcome potential differences in use  of terminology among authors, as well as  the limitations of indexing. the prs project  focuses on completeness of purpose, rather  than precision or specificity. the  effectiveness of an automated search  strategy can be improved by testing the  strategy and evaluating the results.     monitoring indexing terms over time is also  necessary, since databases add, change and  delete terms routinely. new terms can  narrow or broaden the meaning and  relevance of existing terms, altering the  outcome of an automated search. one  potential problem is that these changes may  not be applied consistently to all citations  evidence based library and information practice 2008, 3:1  19 published prior to the changes. for example,  the term “unsafe sex” was added as a  mesh term to medline in 2005. this term,  however, was added to the indexing of  citations published prior to the change,  because they were not indexed until 2005.  similarly, keyword terms and concepts may  also change over time, necessitating  additional modification and retesting of the  automated search. while the search must  evolve as new terms enter the subject matter  vernacular, it is also important to be aware  of the potential issues associated with any  modification.    consistent with previous findings (alpi 99;  avenell, handoll, and grant 508; brettle and  long 354; minozzi, pistotti, and forni 721),  examinations of the overlap between  databases and the unique contributions of  each database suggest that multiple  databases should be used for systematic  searching. in addition, the automated search  should be supplemented by a manual search  component, including a hand search of  journals, conference proceedings and  abstracts, reference chasing, and personal  communications with key researchers in the  field. a combination of these two  approaches will increase the likelihood of  capturing all the relevant citations for  research synthesis activities.     it is important to point out that the amount  of overlap between databases and the  unique contributions of each database  presented here should be interpreted with  caution. this evaluation was performed  within the context of the hiv behavioural  prevention research literature and may not  be generalizable to other research fields. it is  likely that systematic searches in other  behavioural science or social science fields  could produce similar findings and  conclusions; however, further testing would  be required to make that determination. it is  also inappropriate to use this information to  select a database with the highest yield for  sole use in conducting a systematic review.  within the hiv behavioural prevention  research field, all four electronic databases  (medline, embase, psycinfo, and  sociological abstracts) and the manual search  component made important unique  contributions to the retrieval of relevant  literature.    conclusion    as the hiv prevention field becomes  increasingly more reliant on systematic  reviews of the literature for making  evidence based recommendations, it  becomes ever more critical to improve the  quality of searching the research literature.  developing and implementing a systematic  and comprehensive search strategy for  identifying all the relevant research  literature is a vital first step in conducting a  systematic review. creating a search  strategy requires a well thought out process  for planning, testing, and refining, but this  process is necessary to ensure that all  relevant information is included in any  research synthesis activity.  these standards  are important in moving evidence based  public health and information practice  forward.    acknowledgments    the findings and conclusions in this report  are those of the authors and do not  necessarily represent the views of the  centers for disease control and prevention.  this work was supported by the prevention  research branch, division of hiv/aids  prevention, us centers for disease control  and prevention and was not funded by any  other organisation.the authors thank past  and present members of the cdc’s  hiv/aids prevention research team –  tanesha griffin, jeffrey h. herbst, angela  k. horn, angela hutchinson, elizabeth d.  jacobs, laura v. lloyd, paola marrero‐ gonzalez, warren passin, jocelyn patterson,  evidence based library and information practice 2008, 3:1  20 sima rama, r. thomas sherba, lev  zohrabyan – who contributed to the  development and maintenance of the prs  database.       works cited    alpi, kristine m. ʺexpert searching in public  health.ʺ journal of the medical  library association 93.1 (jan. 2005):  97‐103.  atkins, david, kenneth fink, jean slutsky,  and the agency for healthcare  research and quality; north  american evidence‐based practice  centers. ʺbetter information for  better health care: the evidence‐ based practice center program and  the agency for healthcare research  and quality.ʺ annals of internal  medicine 142.12 pt. 2 (21 june 2005):  1035‐41.  avenell, alison, helen g. handoll, and  adrian m. grant. ʺlessons for  search strategies from a systematic  review, in the cochrane library, of  nutritional supplementation trials  in patients after hip fracture.ʺ  american journal of clinical  nutrition 73.3 (mar. 2001): 505‐10.  berry, elizabeth, steven kelly, john hutton,  keith m. harris, and michael a.  smith. ʺidentifying studies for  systematic reviews. an example  from medical imaging.ʺ  international journal of technology  assessment in health care 16.2  (spring 2000): 668‐72.  beverley, catherine, andrew booth, and  peter a. bath. ʺthe role of the  information specialist in the  systematic review process: a  health information case study.ʺ  health information and libraries  journal 20.2 (june 2003): 65‐74.  biblioscape reference information  manager™ . vers. 6.6. june 2007  <http://www.biblioscape.com/index. html>.  booth, andrew. ʺbrimful of starliteʺ:  toward standards for reporting  literature searches.ʺ journal of the  medical library association 94.4  (oct. 2006): 421‐9, e205.  boynton, janette, julie glanville, david  mcdaid, and carol lefebvre.  ʺidentifying systematic reviews in  medline: developing an objective  approach to search strategy  design.ʺ journal of information  science 24.3 (1 june 1998): 137‐57.   brettle, alison j., and andrew f. long.  ʺcomparison of bibliographic  databases for information on the  rehabilitation of people with severe  mental illness.ʺ bulletin of the  medical library association 89.4  (oct. 2001): 353‐62.  brettle, alison j., andrew f. long, maria j.  grant, and joanne greenhalgh.  ʺsearching for information on  outcomes: do you need to be  comprehensive?ʺ quality in health  care 7.3 (sept. 1998): 163‐7.  bruce, julie, and jill mollison. ʺreviewing  the literature: adopting a  systematic approach.ʺ journal of  family planning and reproductive  health care 30.1 (2004): 13‐6.  chalmers, ian, kay dickersin, and thomas  c. chalmers. ʺgetting to grips with  archie cochraneʹs agenda.ʺ bmj  305.6857 (3 oct. 1992): 786‐8.  evidence based library and information practice 2008, 3:1  21 collins, charles, camilla harshbarger,  richard sawyer, and myriam  hamdallah. ʺthe diffusion of  effective behavioral interventions  project: development,  implementation, and lessons  learned.ʺ aids education and  prevention 18.4 suppl. a (aug.  2006): 5‐20.  conn, vicki s., sang‐arun isaramalai,  sabyasachi rath, peeranuch  jantarakupt, rohini wadhawan,  and yashodhara dash. ʺbeyond  medline for literature searches.ʺ  journal of  nursing scholarship 35.2  (2003): 177‐82.  counsell, carl. ʺformulating questions and  locating primary studies for  inclusion in systematic reviews.ʺ  annals of internal medicine 127.5  (sept. 1997): 380‐7.  crepaz, nicole, warren f. passin, jeffrey h.  herbst, sima m. rama, robert m.  malow, david w. purcell, and  richard j. wolitski. ʺmeta‐analysis  of cognitive‐behavioral  interventions on hiv‐positive  personsʹ mental health and  immune functioning.ʺ health  psychology 27.1 (2008): 4‐14.  crepaz, nicole, cynthia m. lyles, richard j,  wolitski, warren f. passin, sima m.  rama, jeffrey h. herbst, david w.  purcell, robert m. malow, ron stall,  and the hiv/aids prevention  research synthesis team. ʺdo  prevention interventions reduce  hiv risk behaviours among people  living with hiv? a meta‐analytic  review of controlled trials.ʺ aids  20.2 (9 jan. 2006): 143‐57.  crepaz, nicole, angela k. horn, sima m.  rama, tanesha griffin, julia b.  deluca, mary m. mullins, sevgi o.  aral, and the hiv/aids prevention  research synthesis team. ʺthe  efficacy of behavioral interventions  in reducing hiv risk sex behaviors  and incident sexually transmitted  disease in black and hispanic  sexually transmitted disease clinic  patients in the united states: a  meta‐analytic review.ʺ  sexually  transmitted diseases 34.6 (june  2007): 319‐32.  crumley, ellen t., and karen blackhall.  ʺsetting up search strategies for  systematic reviews (or, how many  ways can you spell diarrhea?).ʺ  bibliotheca medica canadiana 24.4  (2003): 167‐8.  crumley, ellen t., natasha wiebe, kritie  cramer, terry p. klassen, and lisa  hartling. ʺwhich resources should  be used to identify rct/ccts for  systematic reviews: a systematic  review.ʺ bmc medical research  methodology 5.24 (10 aug. 2005).  doi: 10.1186/1471‐2288‐5‐24  dickersin, kay, renslow scherer, and carol  lefebvre. ʺidentifying relevant  studies for systematic reviews.ʺ  bmj 309.6964 (12 nov. 1994): 1286‐ 91.  ʺevidence‐based interventions.ʺ centers for  disease control and prevention.  2007. hiv/aids prevention  research synthesis project (prs). 13  oct. 2007.  <http://www.cdc.gov/hiv/topics/res earch/prs/evidence‐based‐ interventions.htm>.  ʺfaq: journal selection for medline®  indexing at nlm.ʺ national library  of medicine. 2007 national library  of medicine, bethesda, md. 13 oct.  evidence based library and information practice 2008, 3:1  22 2007.  <http://www.nlm.nih.gov/pubs/fact sheets/j_sel_faq.html >.  greenhalgh, trisha. ʺhow to read a paper.  the medline database.ʺ bmj  315.7101 (19 july 1997): 180‐3.  harris, martha r. ʺthe librarianʹs roles in  the systematic review process: a  case study.ʺ journal of the medical  library association 93.1 (jan. 2005):  81‐7.  helmer, diane, isabellel savoie, carolyn  green, and arminée kazanjian.  ʺevidence‐based practice: extending  the search to find material for the  systematic review.ʺ bulletin of the  medical library association 89.4  (oct. 2001): 346‐52.  herbst, jeffrey h., r. thomas sherba, nicole  crepaz, julia b. deluca, lev  zohrabyan, ron d. stall, cynthia m.  lyles, and the hiv/aids prevention  research synthesis team. ʺa meta‐ analytic review of hiv behavioral  interventions for reducing sexual  risk behavior of men who have  sex with men.ʺ jaids journal of  acquired immune deficiency  syndromes 39.2 (1 june 2005): 228‐ 41.  herbst, jeffrey h., linda s. kay, warren f.  passin, cynthia m. lyles, nicole  crepaz, barbara v. marín, and the  hiv/aids prevention research  synthesis team. ʺa systematic  review and meta‐analysis of  behavioral interventions to reduce  hiv risk behaviors of hispanics in  the united states and puerto rico.ʺ  aids and behavior 11.1 (jan. 2007):  25‐47.  herbst, jeffrey h., carolyn beeker, anita  mathew, tarra mcnally, warren f.  passin,  linda s. kay, nicole  crepaz, cynthia m. lyles, peter  briss, sajal chattopadhyay,  robert  l. johnson, and the task force on  community preventive services.  ʺthe effectiveness of individual‐,  group‐, and community‐level hiv  behavioral risk‐reduction  interventions for adult men who  have sex with men: a systematic  review.ʺ american journal of  preventive medicine 32.4 suppl. 1  (apr. 2007): s38‐s67.  higgins, julian p., and sally green eds.  cochrane handbook for systematic  reviews of interventions 4.2.6  [updated september 2006].. 24  february 2008  <http://www.cochrane.org/resource s/handbook/index.htm>.  howes, faline, jodie doyle, nicki jackson,  and elizabeth waters. ʺevidence‐ based public health: the  importance of finding ʹdifficult to  locateʹ public health and health  promotion intervention studies for  systematic reviews.ʺ journal of  public health 26.1 (mar. 2004): 101‐ 4.  jadad, alejandro r., david moher, and  terry p. klassen. ʺguides for  reading and interpreting  systematic reviews: ii. how did the  authors find the studies and  assess their quality?ʺ archives of  pediatrics and adolescent medicine  152.8 (aug. 1998): 812‐7.  jenuwine, elizabeth s., and judith a. floyd.  ʺcomparison of medical subject  headings and text‐word searches  in medline to retrieve studies on  sleep in healthy individuals.ʺ  evidence based library and information practice 2008, 3:1  23 journal of the medical library  association 92.3 (july 2004): 349‐53.  johnson, blair t., michael p. carey,  stephanie chaudoir, and allecia e.  reid. ʺsexual risk reduction for  persons living with hiv: research  synthesis of randomized  controlled trials, 1993 to 2004.ʺ  journal of acquired immune  deficiency syndromes 41.5 (15 apr.  2006): 642‐50.    lyles, cynthia m., linda s. kay, nicole  crepaz, jeffrey h. herbst, warren f.  passin,  angela s. kim, sima m.  rama, sekhar thadiparthi, julia b.  deluca, mary m. mullins, and the  hiv/aids prevention research  synthesis project team. ʺbest‐ evidence interventions: findings  from a systematic review of hiv  behavioral interventions for u.s.  populations at high risk, 2000‐ 2004.ʺ american journal of public  health 97.1 (jan. 2007): 133‐43.  lyles, cynthia m., nicole crepaz, jeffrey h.  herbst, linda s. kay, and the  hiv/aids prevention research  synthesis project team. ʺevidence‐ based hiv behavioral prevention  from the perspective of the cdcʹs  hiv/aids prevention research  synthesis team.ʺ aids education  and prevention 18.4 suppl. a (aug.  2006): 21‐31.  matthews, elaine j., adrian g. k. edwards,  jackie barker, michael bloor, judith  covey, kerenza hood, roisin pill,  ian. russell, nigel stott, and clare  wilkinson. “efficient literature  searching in diffuse topics: lessons  from a systematic review of  research on communicating risk to  patients in primary care.” health  libraries review 16.2 (1999): 112‐20.  20 feb. 2008.  doi:10.1046/j.1365‐ 2532.1999.00219.x     mcgowan, jessie, and margaret sampson.  ʺsystematic reviews need  systematic searchers.ʺ journal of the  medical library association 93.1  (jan. 2005): 74‐80.  mcmanus, richard j., sue wilson, brendan  c. delaney, david a. fitzmaurice,  c. j. hyde, r. s. tobias, s. jowett,  and richard hobbs. ʺreview of the  usefulness of contacting other  experts when conducting a  literature search for systematic  reviews.ʺ bmj 317.7172 (5 dec.  1998): 1562‐3.  mcnally, rosalind, and alison alborz.  “developing methods for systematic  reviewing in health services  delivery and organization: an  example from a review of access to  health care for people with learning  disabilities. part 1. identifying the  literature.” health information and  libraries journal 21.3 (sept. 2004):  182‐92.  minozzi, silvia, vanna pistotti, and marco  forni. ʺsearching for rehabilitation  articles on medline and embase.  an example with cross‐over  design.ʺ archives of physical  medicine and rehabilitation 81.6  (june 2000): 720‐2.  neumann, mary spink, and ellen sogolow.  ʺreplicating effective programs:  hiv/aids prevention technology  transfer.ʺ aids education and  prevention 12.suppl. 5 (2000): 35‐48.  passin, warren f., angela s. kim, angela b.  hutchinson, nicole crepaz, jeffrey  h. herbst, cynthia m. lyles, and  the hiv/aids prevention research  evidence based library and information practice 2008, 3:1  24 synthesis project team. ʺa  systematic review of hiv partner  counseling and referral services:  client and provider attitudes,  preferences, practices, and  experiences.ʺ sexually transmitted  diseases 33.5 (may 2006): 320‐8.  patrick, timothy b., george demiris, lillian  c. folk, david e. moxley, joyce a.  mitchell, and donghua tao.  ʺevidence‐based retrieval in  evidence‐based medicine.ʺ journal  of the medical library association  92.2 (apr. 2004): 196‐9.  perry, gerald j., and michael r. kronenfeld.  ʺevidence‐based practice: a new  paradigm brings new  opportunities for health sciences  librarians.ʺ medical reference  services quarterly 24.4 (winter  2005): 1‐16.  peterson, amy s., and lisa m. randall.  ʺutilizing multilevel partnerships to  build the capacity of community‐ based organizations to implement  effective hiv prevention  interventions in michigan.ʺ aids  education and prevention 18.4  suppl a (2006): 83‐95.  robinson, karen a., and kay dickersin.  ʺdevelopment of a highly sensitive  search strategy for the retrieval of  reports of controlled trials using  pubmed.ʺ international journal of  epidemiology 31.1 (2002): 150‐3.  sackett, david l., william m. c. rosenberg,  j. a. muir gray, r. brian haynes,  and w. scott richardson. ʺevidence  based medicine: what it is and  what it isnʹt.ʺ bmj 312.7023 (13 jan.  1996): 71‐2.  sampson, margaret and jessie mcgowan.  ʺerrors in search strategies were  identified by type and frequency.ʺ  journal of clinical epidemiology  59.10 (2006): 1057‐63.  savoie, isabelle, diane helmer, carolyn j.  green, and arminee kazanjian.  ʺbeyond medline: reducing bias  through extended systematic  review search.ʺ international  journal of technology assessment  in health care 19.1 (winter 2003):  168‐78.  shea, madeline a., barry p. callis, hope  cassidy‐stewart, kevin cranston,  and naomi tomoyasu. ʺdiffusion of  effective hiv prevention  interventions‐‐lessons from  maryland and massachusetts.ʺ  aids education and prevention  18.4 suppl a (aug. 2006): 96‐107.  simoni, jane m., cynthia r. pearson, david  w. pantalone, gary marks, and  nicole crepaz. ʺefficacy of  interventions in improving highly  active antiretroviral therapy  adherence and hiv‐1 rna viral  load: a meta‐analytic review of  randomized controlled trials.ʺ  journal of acquired immune  deficiency syndromes 43.suppl 1  (dec. 2006): s23‐s35.  sogolow, ellen, greet peersman, salaam  semaan, darcy strouse, cynthia m.  lyles, and the hiv/aids prevention  research synthesis project team.  ʺthe hiv/aids prevention  research synthesis project: scope,  methods, and study classification  results.ʺ journal of acquired  immune deficiency syndromes  30.suppl. 1 (2002): s15‐s29.  evidence based library and information practice 2008, 3:1  25 victora, cesar g., jean‐pierre habicht, and  jennifer bryce. ʺevidence‐based  public health: moving beyond  randomized trials.ʺ american  journal of public health 94.3 (2004):  400‐5.  watson, r. j., and philip h. richardson.  ʺidentifying randomized  controlled trials of cognitive  therapy for depression: comparing  the efficiency of embase, medline  and psycinfo bibliographic  databases.ʺ british journal of  medical psychology 72.pt. 4 (1999):  535‐42.  weller, ann c. ʺmounting evidence that  librarians are essential for  comprehensive literature searches  for meta‐analyses and cochrane  reports.ʺ journal of the medical  library association 92.2 (apr. 2004):  163‐4.  zhang, li, margaret sampson, and jessie  macgowan. ʺreporting of the role  of the expert searcher in cochrane  reviews.ʺ evidence based library  and information practice 1.4 (dec.  2006): 3‐16. 18 feb. 2008  <http://ejournals.library.ualberta.ca/ index.php/eblip/article/view/57/15 7>.  evidence based library and information practice 2008, 3:1  26 appendix a – ovid search strategy –  medline     $   = truncation    * = focus  ab  = abstract      ti = title  /ut = utilization subheading  /pc = prevention and control subheading      hiv/aids/stds mesh    1. *hiv infections/pc    2. *aids/pc  3. *sexually transmitted  diseases/pc  4. *sexually transmitted diseases,  bacterial/  5. *sexually transmitted diseases,  viral/  6. aids serodiagnosis/ut  7. *hiv seropositivity/    8. or/1‐7    prevention/intervention mesh and  keywords    9. primary prevention/  10. preventive health services/  11. health promotion/  12. program evaluation/  13. randomized controlled trials/  14. evaluation studies/  15. contact tracing/  16. case management/  17. needle‐exchange programs/  18. intervention studies/  19. follow‐up studies/  20. longitudinal studies/  21. multicenter studies/  22. random allocation/  23. (control group).ti,ab  24. (control trial).ti,ab  25. (controlled trial).ti,ab  26. (rct or rcts).ti,ab  27. (case management).ti,ab  28. (contact tracing).ti,ab  29. (counseling or counselling).ti,ab  30. (detox or detoxification).ti,ab  31. (drug$ adj4 treatment$).ti,ab  32. education$.ti,ab  33. (effect or effects or effective or  effectiveness).ti,ab  34. efficacy.ti,ab  35. evaluation$.ti,ab  36. impact$.ti,ab  37. intervention$.ti,ab  38. (needle exchang$).ti,ab  39. network$.ti,ab  40. outreach$.ti,ab  41. (partner notification).ti,ab  42. (partner$ adj4 contact$ adj4  referral$).ti,ab  43. (notif$ adj4 partner$).ti,ab  44. prevention$.ti,ab  45. program$.ti,ab  46. random$.ti,ab  47. rehab$.ti,ab  48. skill$.ti,ab  49. (syringe exchang$).ti,ab  50. methadone.ti,ab  51. test$.ti,ab  52. training.ti,ab  53. trial$.ti,ab    54. or/9‐53    behavior/outcomes mesh and keywords    55. behavior/  56. behavior therapy/  57. health behavior/  58. risk reduction behavior/  59. risk‐taking/  60. contraception behavior/  61. coitus/  62. sexual abstinence/  63. sexual behavior/  64. sexual partners/  65. safe sex/  66. unsafe sex/  67. heroin dependence/pc  68. needle sharing/  69. condoms/ut  evidence based library and information practice 2008, 3:1  27 70. condoms, female/ut  71. contraceptive devices, male/ut  72. contraceptive devices,  female/ut  73. substance abuse,  intravenous/pc  74. substance‐related disorders/pc  75. cocaine‐related disorders/pc  76. health services/ut  77. heroin.ti,ab  78. cocaine.ti,ab  79. opiate$.ti,ab  80. opium.ti,ab  81. paraphernalia.ti,ab  82. (treatment$ adj2 entry).ti,ab  83. (treatment$ adj2 enter$).ti,ab  84. (abstin$ or abstain$).ti,ab  85. drug$.ti,ab  86. substance.ti,ab  87. (idu or idus or ivdu or  ivdus).ti,ab  88. ((behavior$ or behaviour$ or  activit$ or access$ or utiliz$ or  use$ or using$ or test$ or risk$  or outcome$) adj4 (reduc$ or  declin$ or chang$ or effect$ or  increas$ or decreas$ or impact$  or modif$ or lower$ or  maintain$ or maintenance)).ti,ab  89. bleach$.ti,ab  90. clean$.ti,ab  91. condom$.ti,ab  92. contracept$.ti,ab  93. crack.ti,ab  94. disclos$.ti,ab  95. incidence.ti,ab  96. inject$.ti,ab  97. intention$.ti,ab  98. intercourse.ti,ab  99. needle$.ti,ab  100. infect$ adj4 (new$ or rate$ or  declin$ or reduc$ or prevent$ or  lower$ or decreas$).ti,ab  101. partner$.ti,ab  102. seroconver$.ti,ab  103. sex$.ti,ab  104. syring$.ti,ab  105. test$.ti,ab    106. or/55‐105    107. 8 and 54 and 106    limits: english language, publication types,  date limits  evidence based library and information practice 2008, 3:1  28 appendix b – ovid search strategy –  psycinfo    $  = truncation      * = focus  ab = abstract      ti = title    hiv/aids/stds subject headings    1. *hiv/  2. *aids/  3. *sexually transmitted diseases/    4. or/1‐3    prevention/intervention subject headings  and keywords    5. prevention/  6. aids prevention/  7. hiv testing/  8. primary health care/  9. health promotion/  10. program evaluation/  11. case management/  12. needle‐exchange programs/  13. followup studies/  14. longitudinal studies/  15. random sampling/  16. control group.ti,ab  17. control trial.ti,ab  18. controlled trial.ti,ab  19. (rct or rcts).ti,ab  20. case management.ti,ab  21. contact tracing.ti,ab  22. (counseling or counselling).ti,ab  23. (detox or detoxification).ti,ab  24. (drug$ adj4 treatment$).ti,ab  25. education$.ti,ab  26. (effect or effects or effective or  effectiveness).ti,ab  27. efficacy.ti,ab  28. evaluation$.ti,ab  29. impact$.ti,ab  30. intervention$.ti,ab  31. needle exchang$.ti,ab  32. network$.ti,ab  33. outreach$.ti,ab  34. partner notification.ti,ab  35. (partner$ adj4 contact$ adj4  referral$).ti,ab  36. (notif$ adj4 partner$).ti,ab  37. prevention$.ti,ab  38. program$.ti,ab  39. random$.ti,ab  40. rehab$.ti,ab  41. skill$.ti,ab  42. syringe exchang$.ti,ab  43. methadone.ti,ab  44. test$.ti,ab  45. training.ti,ab  46. trial$.ti,ab    47. or/5‐46    behavior/outcomes subject headings and  keywords    48. behavior/  49. behavior therapy/  50. health behavior/  51. risk‐taking/  52. sexual risk taking /  53. sexual abstinence/  54. sexual partners/  55. safe sex/  56. psychosexual behavior/  57. behavior change/  58. condoms/  59. contraceptive devices/  60. drug abuse/  61. intravenous drug usage/  62. cocaine/  63. heroin addiction/  64. heroin/  65. needle sharing/  66. at risk populations/  67. heroin.ti,ab  68. cocaine.ti,ab  69. opiate$.ti,ab  70. opium.ti,ab  71. paraphernalia.ti,ab  72. (treatment$ adj2 entry).ti,ab  73. (treatment$ adj2 enter$).ti,ab  74. (abstin$ or abstain$).ti,ab  75. drug$.ti,ab  76. substance.ti,ab  evidence based library and information practice 2008, 3:1  29 77. (idu or idus or ivdu or  ivdus).ti,ab  78. ((behavior$ or behaviour$ or  activit$ or access$ or utiliz$ or  use$ or using$ or test$ or risk$  or outcome$) adj4 (reduc$ or  declin$ or chang$ or effect$ or  increas$ or decreas$ or impact$  or modif$ or lower$ or  maintain$ or maintenance)).ti,ab  79. bleach$.ti,ab  80. clean$.ti,ab  81. condom$.ti,ab  82. contracept$.ti,ab  83. crack.ti,ab  84. disclos$.ti,ab  85. incidence.ti,ab  86. inject$.ti,ab  87. intention$.ti,ab  88. intercourse.ti,ab  89. needle$.ti,ab  90. infect$ adj4 (new$ or rate$ or  declin$ or reduc$ or prevent$ or  lower$ or decreas$).ti,ab  91. partner$.ti,ab  92. seroconver$.ti,ab  93. sex$.ti,ab  94. syring$.ti,ab  95. test$.ti,ab    96. or/48‐95    97. 4 and 47 and 96    limits: english language, publication types,  date limits    evidence based library and information practice 2008, 3:1  30 appendix c – ovid search strategy –   embase      $   = truncation    * = focus  ab  = abstract      ti = title  /pc = prevention and control subheading    hiv/aids/stds subject headings    1. *human immunodeficiency  virus infection/pc    2. *acquired immunodeficiency  syndrome/pc  3. *sexually transmitted  diseases/pc    4. or/1‐3     prevention/intervention subject headings  and keywords    5. primary prevention/  6. preventive health service/  7. health promotion/  8. health care quality/  9. patient care/  10. randomized controlled trials/  11. evaluation/  12. contact examination/  13. types of  study/  14. follow up/  15. longitudinal study/  16. multicenter study/  17. randomization/  18. control group.ti,ab  19. control trial.ti,ab  20. controlled trial.ti,ab  21. (rct or rcts).ti,ab  22. case management.ti,ab  23. contact tracing.ti,ab  24. (counseling or counselling).ti,ab  25. (detox or detoxification).ti,ab  26. (drug$ adj4 treatment$).ti,ab  27. education$.ti,ab  28. (effect or effects or effective or  effectiveness).ti,ab  29. efficacy.ti,ab  30. evaluation$.ti,ab  31. impact$.ti,ab  32. intervention$.ti,ab  33. needle exchang$.ti,ab  34. network$.ti,ab  35. outreach$.ti,ab  36. partner notification.ti,ab  37. (partner$ adj4 contact$ adj4  referral$).ti,ab  38. (notif$ adj4 partner$).ti,ab  39. prevention$.ti,ab  40. program$.ti,ab  41. random$.ti,ab  42. rehab$.ti,ab  43. skill$.ti,ab  44. syringe exchang$.ti,ab  45. methadone.ti,ab  46. test$.ti,ab  47. training.ti,ab  48. trial$.ti,ab    49. or/5‐48    behavior/outcomes subject headings and  keywords    50. behavior/  51. behavior therapy/  52. health behavior/  53. infection risk/  54. high risk population/  55. risk reduction/  56. coitus/  57. abstinence/  58. sexual behavior/  59. safe sex/  60. diamorphine/  61. drug abuse/  62. condom/  63. drug dependence/  64. intravenous drug abuse/  65. health service/  66. “drug use”/  67. substance abuse/  68. addiction/  69. opiate addiction/  70. cocaine/  evidence based library and information practice 2008, 3:1  31 71. cocaine dependence/  72. heroin.ti,ab  73. cocaine.ti,ab  74. opiate$.ti,ab  75. opium.ti,ab  76. paraphernalia.ti,ab  77. (treatment$ adj2 entry).ti,ab  78. (treatment$ adj2 enter$).ti,ab  79. (abstin$ or abstain$).ti,ab  80. drug$.ti,ab  81. substance.ti,ab  82. (idu or idus or ivdu or  ivdus).ti,ab  83. ((behavior$ or behaviour$ or  activit$ or access$ or utiliz$ or  use$ or using$ or test$ or risk$  or outcome$) adj4 (reduc$ or  declin$ or chang$ or effect$ or  increas$ or decreas$ or impact$  or modif$ or lower$ or  maintain$ or maintenance)).ti,ab  84. bleach$.ti,ab  85. clean$.ti,ab  86. condom$.ti,ab  87. contracept$.ti,ab  88. crack.ti,ab  89. disclos$.ti,ab  90. incidence.ti,ab  91. inject$.ti,ab  92. intention$.ti,ab  93. intercourse.ti,ab  94. needle$.ti,ab  95. infect$ adj4 (new$ or rate$ or  declin$ or reduc$ or prevent$ or  lower$ or decreas$).ti,ab  96. partner$.ti,ab  97. seroconver$.ti,ab  98. sex$.ti,ab  99. syring$.ti,ab  100. test$.ti,ab    101. or/50‐100    102. 4 and 49 and 101    limits: english language, publication types  date limits  evidence based library and information practice 2008, 3:1  32 appendix d – journal list for manual  search, updated february 2007    addiction  aids  aids and behavior  aids care  aids education and prevention  aids patient care and stds  american journal of community psychology  american journal of drug and alcohol abuse  american journal of preventive medicine  american journal of psychiatry  american journal of public health  annals of behavioral medicine  archives of pediatrics & adolescent medicine  clinical infectious diseases  drug and alcohol dependence  health education & behavior  health education research  health psychology  international journal of std & aids  jaids journal of acquired immune  deficiency syndromes  jama journal of the american medical  association  journal of adolescent health  journal of consulting and clinical psychology  journal of drug issues  journal of psychoactive drugs  journal of substance abuse treatment  journal of urban health  lancet  morbidity and mortality weekly report  psychology of addictive behaviors  public health reports  research in nursing & health  sexually transmitted diseases  sexually transmitted infections  substance use & misuse  microsoft word news_riwa.doc evidence based library and information practice 2007, 2:1  161 evidence based library and information practice     news    winner of the 2006/2007 research in the workplace award (riwa*) announced      maria j grant  chair ‐ research in the workplace award 2006/2007  centre for nursing, midwifery and collaborative research  university of salford, salford, uk  email: m.j.grant@salford.ac.uk      © 2007 grant. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    it is with great pleasure that the research in  the workplace award (riwa)* assessment  panel announces the winning proposal in  2006/2007 to be for a multi‐site randomised  controlled trial to determine the impact of  providing a virtual reference service (access  specialist knowledge ‐ ask) to the local  primary care and mental health trusts  within the uk national health service.     the project will be led by rachel southon,  royal surrey county hospitals nhs trust,  in collaboration with vicki veness, also of  royal surrey county hospitals nhs trust  and john loy, avon & wiltshire mental  health partnership nhs trust.     the project will commence in april 2007  and the assessment panel believes it will  yield measurable outcomes and provide an  evidence base for developing services to  primary care. the project is due for  completion by march 2008 and will be  followed by a comprehensive programme of  dissemination.  a copy of the winning proposal can be  accessed via the riwa* web site at  http://ifmh.org.uk/riwa.html.  the  proposal was considered to be well planned  and an exemplary model of a proposal in  terms of identifying an important question  and an appropriate methodology with  which to address it.       riwa* is a biennial award and details of  projects which have previously been funded,  together with news of future awards, can be  found on the riwa* web site at  http://ifmh.org.uk/riwa.html.      riwa 2006/2007 is managed by ifm  healthcare and is sponsored by national  library for health cpd forum, ifm  healthcare, the health libraries group, the  university medical school librarians group,  and the university health sciences libraries  and libraries for nursing.      http://creativecommons.org/licenses/by/2.0 http://ifmh.org.uk/riwa.html http://ifmh.org.uk/riwa.html evidence based library and information practice editorial responsibilities      2020. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     editor-in-chief: lorie kloda   associate editor (evidence summaries): stephanie krueger, christina wissinger   associate editor (research articles): ann medaille, lisl zach   associate editor (classics & reviews): jane schmidt   associate editor (commentaries, using evidence in practice): lorie kloda   associate editor (feature): lorie kloda   communications officer (news): tatiana bryant   production editor: rachel hinrichs   editorial intern: kimberly mackenzie   editorial advisors: lindsay alcock, alison brettle, michelle dunaway, denise koufogiannakis, heather pretty   copyeditors: heather healy (lead copyeditor), georgianne bordner, julie evener, emily kingsland, alison moore, jane morgan-daniel, stacey l. penney, elizabeth stregger, nikki tummon   indexing support: kate shore   news/announcements   c-eblip fall symposium: librarians as researchers wednesday, october 15, 2014 university library, university of saskatchewan      2014. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   the centre for evidence based library and information practice (c-eblip) is located in the university library, university of saskatchewan, canada. c-eblip’s mission is to support university library librarians and archivists as researchers and evidence based practitioners, and to raise the profile of librarians as researchers on campus and beyond.   call for proposals    librarians are regularly conducting research and disseminating the results. whether it’s a requirement for tenure and promotion, part of evidence based library and information practice, curiosity, or contributing to the field of librarianship, librarians often approach research from the unique perspective of practice.   the centre for evidence based library and information practice (c-eblip) fall symposium: librarians as researchers is a one-day conference (with complimentary registration, including breaks and lunch) designed to facilitate sharing, collaboration, and networking with the focus on librarians in their researcher roles.  the symposium will consist of a series of single-track sessions, along with a keynote address, and time for networking. the sessions will focus not only on the research being done by librarians, but also on why librarians are doing research – their motivations or inspirations.   sessions are 20 minutes in length which includes time for questions. session topics should broadly relate to one of the focus areas of the symposium:   research being done by librarians (i.e. results of research projects, a project in its interim state, methodology, experiences with conducting research, etc.) why librarians are conducting research (what inspires, motivates you to conduct research? are there external factors? do you have a philosophy of research?)   session proposals of no more than 250 words should include the following:   1.            description of the presentation 2.            how the proposal links to a focus areas (research being done by librarians; why librarians are doing research) 3.            outcomes or takeaways   applicants will be advised of the status of their proposals by the week of august 11, 2014.     submit proposals to virginia wilson, director, centre for evidence based library and information practice, virginia.wilson@usask.ca by thursday, july 31, 2014.    http://library.usask.ca/ceblip/index.php     evidence based library and information practice evidence based library and information practice 2011, 6.1 78 evidence based library and information practice evidence summary wikipedia and the national cancer institute website appear to offer similar osteosarcoma information for patients a review of: leithner, a., werner, m., glehr, m., friesenbichler, j., keithner, k., & windhager r. (2010). wikipedia and osteosarcoma: a trustworthy patients' information? journal of the medical informatics association, 17(4), 373-374. reviewed by: kate kelly royal college of surgeons in ireland lower mercer street, dublin 2. ireland email: katekelly@rcsi.ie received: 29 nov. 2010 accepted: 2 feb. 2011 2011 kelly. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to compare the completeness and accuracy of information about osteosarcoma in wikipedia to information found on the patient and health professional versions of the u.s. national cancer institute (nci) website. design – comparative study, test against 20 item questionnaire and expert opinion. setting – n/a subjects – n/a methods – the authors developed a 20-item questionnaire to test the completeness and accuracy of information on osteosarcoma in wikipedia and on the "patient version and the health professional version of the national cancer institute's website as 'official' reference websites" (p. 373). three independent observers, two surgeons specializing in musculoskeletal tumour surgery and a medical student, tested the english language version of wikipedia and the nci “websites” on april 3, 2009. answers to the 20 questions found on the websites were scored from zero to three and were discussed with a member of the "german board for guidelines in musculoskeletal surgery" (p. 373) and verified against international guidelines published by the world health organization. data was analyzed using spss and group comparisons were performed using mann-whitney u test with p-values of less than 0.05 significance. mailto:katekelly@rcsi.ie� evidence based library and information practice 2011, 6.1 79 main results – the quality of information about osteosarcoma found in the english language version of wikipedia was good but inferior to the patient information from nci. out of a total of 60 points wikipedia scored 33, nci patient information 40 and nci professional information 50. there was no significant difference between the nci patient information and wikipedia but a significant difference (p=0.039) between wikipedia and nci professional information. conclusion – non-peer reviewed websites providing health information, such as wikipedia, should include links to sites such as nci and other more definitive sources such as professional and international organizations. frequent checks should be used to ensure external links are of the highest quality. commentary this is a very short, interesting article, which suffers from lack of detail. the authors state they compared the "patient version and the health professional version of the national cancer institute's (nci) website" (p. 373) with the wikipedia osteosarcoma page but what this means is unclear. the nci does not have separate websites for patients and health professionals. the physicians data query (pdq) cancer information summaries, produced by the nci, are available in separate versions for patients and professionals and can be accessed via several routes on the nci website. presumably it is the two versions of the pdq summaries that the authors compared to wikipedia but this is not stated. the reference to three websites throughout the article is confusing. the lack of detail on scoring criteria is also confusing. the one table in the article is the osteosarcoma questionnaire presenting the "points for each answer for each of the three different websites" (table 1, p. 374). however, there is no information about the scoring criteria, for example, it is unclear why wikipedia scores 0 points for “do you find web links to study centres (euramos?)” (table 1, p. 374) while nci patient and nci professional each score 2 points. while the version of wikipedia current at the time of the test, last edited on march 19, 2009, did not have a link to euramos, it did link to the nci and the mayo clinic. are these not "study sites"? euramos is the european and american osteosarcoma study group and the only reference to it this reviewer could find on the nci website was in reports detailing the activities of all cooperative study groups, i.e., there was no direct link from either the patient or professional pdq summary on osteosarcoma so it is unclear why it scored two points. likewise there is no exploration of the reported result that "nci professional" scored only 50 out of 60 points (it is the gold standard site after all) nor that wikipedia outscored "nci professional" on three items and matched it on eight. detail on the scoring criteria would have been a helpful inclusion. the value of the little statistical data reported is debatable given the acknowledged small sample size and a similar lack of detail on what exactly was analyzed. a statement that the three reviewers preferred wikipedia when asked about ease of use and explaining the page's oversight appears in the middle of the article and is not explained. questions about ease of use were not included in the reported questionnaire used to compare websites and no other reference is made to the reviewers' opinions being part of the study. one of the conclusions states that sites such as wikipedia should "include links to more definitive sources" (p. 374) which seems a little odd when the version of wikipedia current at the time of the test did link to definitive sources such as those noted above plus the american association of family physicians and a university of minnesota research study. to be fair, the authors themselves draw attention to some of the most obvious limitations of the study such as small sample size in terms of items tested; potential bias in reviewers (the three "independent observers" are also three of the authors) and the "creation evidence based library and information practice 2011, 6.1 80 of the questions without external review board" (p. 374). on that basis the validity and reliability of the questionnaire is questionable. the conclusion that "our study shows that the quality of osteosarcoma-related information found in the english wikipedia is good but inferior to the patient information provided by the nci" (p. 374) is not supported based on the evidence presented in the article. the genuine concern with accuracy of information in nonpeer reviewed websites is clear but this particular study has the feel of something put together quickly rather than rigorously. it also uses a study design of clinician evaluation of consumer health sites. as the intended audience of these sites are patients, caregivers and other healthcare professionals the study may have benefited from including these potential users of information as reviewers. it is the authors desire that this study generate discussion among professionals and maybe also lead to a larger international study on various bone and soft tissue described in wikipedia and this study serves that purpose well. they recommend that frequent checks be made on external links on wikipedia to ensure they are of the highest quality. maybe the authors have done what they suggest other professionals do and checked the wikipedia osteosarcoma page the editorial history shows it has been frequently edited and the external sites flagged several times for compliance with wikipedia policy on linking to external sites. it is a shame that the methodology is not replicable as those with subject knowledge may find it interesting to adapt and repeat in other domains. as it stands, this study has no direct applicability to library practice but those interested in a test methodology should maybe consult the 2008 study on the accuracy of drug information in wikipedia upon which this study was based (clauson, polen, boulos & dzenowagis, 2008). references clauson, k. a., polen, h. h., boulos, m. n., & dzenowagis, j. h. (2008). scope, completeness and accuracy of drug information in wikipedia. annals of pharmacotherapy, 42(12), 1814-1821. / evidence based library and information practice evidence summary evidence based library and information practice evidence based library and information practice 2010, 5.4 90 evidence based library and information practice evidence summary music information seeking behaviour poses unique challenges for the design of information retrieval systems a review of: lee, j. h. (2010). analysis of user needs and information features in natural language queries seeking music information. journal of the american society for information science and technology, 61, 1025-1045. reviewed by: cari merkley librarian mount royal university calgary, alberta, canada email: cmerkley@mtroyal.ca received: 1 sept. 2010 accepted: 25 oct. 2010 2010 merkley. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to better understand music information seeking behaviour in a real life situation and to create a taxonomy relating to this behaviour to facilitate better comparison of music information retrieval studies in the future. design – content analysis of natural language queries. setting – google answers, a fee based online service. subjects – 1,705 queries and their related answers and comments posted in the music category of the google answers website before april 27, 2005. methods – a total of 2,208 queries were retrieved from the music category on the google answers service. google answers was a fee based service in which users posted questions and indicated what they were willing to pay to have them answered. the queries selected for this study were posted prior to april 27, 2005, over a year before the service was discontinued completely. of the 2208 queries taken from the site, only 1,705 were classified as relevant to the question of music information seeking by the researcher. the off-topic queries were not included in the study. evidence based library and information practice 2010, 5.4 91 each of the 1,705 queries was coded according to the needs expressed by the user and the information provided to assist researchers in answering the question. the initial coding framework used by the researcher was informed by previous studies of music information retrieval to facilitate comparison, but was expanded and revised to reflect the evidence itself. only the questions themselves were subjected to this iterative coding process. the answers provided by the google answer researchers and online comments posted by other users were examined by the author, but not coded for inclusion in the study. user needs in the questions were coded for their form and topic. each question was assigned at least one form and one topic. form refers to the type of question being asked and consisted of the following 10 categories: identification, location, verification, recommendation, evaluation, ready reference, reproduction, description, research, and other. reproduction in this context is defined as “questions asking for text” and referred most often to questions looking for song lyrics, while evaluation typically meant the user was seeking reviews of works (p. 1029). sixteen question topics were outlined in the coding framework. they included lyrics, translation, meaning (i.e., of lyrics), score, work, version, recording (e.g., where is an album available for purchase), related work, genre, artist, publisher, instrument, statistics, background (e.g. definitions), resource (i.e. sources of music information) and other. the questions were also coded for their features or the information provided by the user. the final coding framework outlined 57 features, some of which were further subdivided by additional attributes. for example, a feature with attributes was title. the researcher further clarified the attribute of title by indicating whether the user mentioned the title of a musical work, recording, printed material or related work in their question. more than one feature could appear in a user query. main results – overall, the most common questions posted on the google answers service relating to music involved identifying works or artists, finding recordings, or retrieving lyrics. the most popular query forms were identification (43.8%), location (33.3%), and reproduction (10.9%). the most common topics were work (49.1%), artist (36.4%), recording (16.7%), and lyrics (10.4%). the most common features provided by users in their posted questions were person name (53%), title (50.9%), date (45.6%), genre (37.2%), role (33.8%), and lyric (27.6%). the person name usually referred to an artist’s name (in 95.6% of cases) and title most often referred to the title of a musical work. another feature that appeared in 25.6% of queries was place reference, almost half of which referred to the place where the user encountered the music they were enquiring about. while the coding framework eventually encompassed 57 different features, a small number of features dominated, with seven features used in over 25% of the queries posted and 33 features appearing in less than 10%. the seven most common features were person name, title, date, genre, role, lyric, and place reference. lee categorized most of the queries as “known-item searches,” even though at times users provided incorrect information and many were looking for information about the musical item but not the item itself (p. 1035). other interesting features identified by the author were the presence of “dormant searches,” long standing questions a user had about a musical item, sometimes for years, which were reawakened by hearing the song again or other events (p. 1037). multiple versions of musical works and the provision of information gleaned third hand by users were also identified as complicating factors in correctly meeting musical information needs. conclusion – while certain types of questions dominated among music queries posted on the google answers service, there were a wide variety of music information needs expressed by users. in some cases, the features provided by the user as clues to answering the query were very personal, and related to the context evidence based library and information practice 2010, 5.4 92 in which they encountered the work or the mood a particular work or artist evoked. such circumstances are not currently or adequately covered by existing bibliographic record standards, which focus on qualities inherent in the music itself. the author suggests that user context should play a greater role in the testing and development of music information retrieval systems, although the instability and variability of this type of information is acknowledged. in some cases this context could apply to other works (film, television, etc.) in which a musical work is featured. another potential implication for music information retrieval system development is a need to re-evaluate the terminology employed in testing to ensure that it is the language most often employed by users. for example, the 128 different terms used in this study to describe how a musical item made the user feel did not significantly overlap with terms employed in a previous music information retrieval task involving mood classification conducted through mirex, the music information retrieval evaluation exchange, in 2007. the author also argues that while most current music information retrieval testing is taskspecific – e.g., how can a user search for a particular work by humming a few bars or searching for a work based on its genre, in real life, users come to their search with information that is not neatly parsed into separate tasks. the study affirms a need for systems that can combine tasks and/or consolidate the results of separate tasks for users. commentary this study reaffirms the value of evaluating information retrieval systems with data gleaned from empirical studies of users in their natural habitat. as the author of the study rightly points out, what is particularly valuable in this instance is that the queries used in this study were not shaped by their interaction with a particular database or existing bibliographic records, but rather contained the information that users thought would be most helpful in tracking down the answer to their question. the types of questions logged may suggest that in many cases the information need users were attempting to satisfy was personal rather than academic, which may play a role in the potential applicability of the results to certain contexts. however, the high level at which the data is presented in the study and the potential overlap between these spheres make it difficult to achieve a clear determination on this issue. the google answers service may have attracted a particular type of music seeker, but the fact that users are expressing their questions in free form makes it a particularly rich source of data on how users articulate their information needs. the field of music information retrieval research is complex, and involves experts from a variety of fields, of which information science is one (downie, 2008). throughout the study, the author draws on existing work on information retrieval while clearly making the case for the unique challenges faced by individuals working to facilitate user access to the rich body of music information objects in existence. another source of research on this particular issue is the proceedings of the annual conference of the international society of music information retrieval (2010). nonmusic specialists may also find value in the methodology employed to answer other types of research questions. the author provides considerable detail on the coding framework created to support content analysis of the google answers questions and addresses some of the advantages and challenges posed by use of web resources as artifacts of information seeking. the author highlights the advantages of content analysis as a methodology, such as the ability to express results both numerically and qualitatively. the author also clearly addresses the issue of the representativeness of the data sample, and refrains from making sweeping generalizations based on the data. finally, the author’s call for more empirical studies on user behaviour and less reliance on anecdotal evidence when creating information systems evidence based library and information practice 2010, 5.4 93 will strike a chord with information professionals generally, not just those working with music. references downie, j. s. (2008). the music information retrieval evaluation exchange (20052007): a window into music information retrieval research. acoustical science and technology, 29(4), 247-255. international society of music information retrieval. (2010). ismir the international society for music information retrieval. retrieved november 22, 2010 from http://www.ismir.net/ / evidence based library and information practice evidence summary feature editorial   eblip7: the possibilities are endless!   virginia wilson guest editor director, centre for evidence based library and information practice university library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca      2013 wilson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   the 7th international evidence based library and information practice conference (eblip7) was held in saskatoon, saskatchewan, canada, on july 15-18, 2013. eighteen months of planning came to fruition over four sunny summer days in saskatchewan. following in the illustrious footsteps of sheffield, edmonton, brisbane, chapel hill-durham, stockholm, and salford, the university library, university of saskatchewan welcomed 100 delegates to the evidence-based party.   welcome to the eblip7 feature in evidence based library and information practice. in this issue, you can get a good sense of the eblip7 conference if you were unable to join us in saskatoon. and if you were a delegate in july, hopefully these pieces will bring back good memories of an intellectually stimulating and fun time. in this issue you’ll find dr. denise koufogiannakis’s opening keynote address, in which she talks about different kinds of evidence and the idea that evidence based practice is a mindset. also included in this issue is charlene sorensen’s summary of dan gardner’s closing keynote talk based on his chapter in a forthcoming book on forecasting.    the eblip7 lightning strikes were five minute, 20-slide talks on the topic “how i’ve been inspired by eblip.” the twist there was that the slides were set to automatically advance every 15 seconds. while it’s difficult to replicate the experience of actually being there to hear those exciting talks live, here you’ll find 6 out of the 8 lightning strikes talks. you’ll have to imagine the “edge of your seat” feeling of wondering when that slide will advance! also in this issue, several conference delegates have jotted down their impressions of the conference, which are collected in a commentary piece. and then of course there are the people’s choice awards for best paper and best poster.   an international conference relies on many people generously putting in many hours of their time. as chair of the local organizing committee for elbip7, i can safely say that eblip7 would not have happened without the efforts of the organizing committee members from the university library, university of saskatchewan: lyn currie, angie gerrard, karim tharani, carolyn doi, rachel sarjeant-jenkins, jen murray, and christine neilson (who had to leave us just prior to the conference). our many volunteers and session facilitators did a tremendous amount of work as well, and were crucial to the success of eblip7.   of course, eblip7 assumes there will be an eblip8! the call for expressions of interest to host the 8th international evidence based library and information practice conference in 2015 went out in the fall and the location of eblip8 will be announced before the end of 2013. i can relate from personal experience that while planning a conference does take hard work and commitment, the rewards are astonishing.   every conference has its own flavour and its own feeling when you look back on it. for me, eblip7 was this: research and evidence, colleagues and friends, learning and exploring, hootenannies and mosquitoes, dainties and pie, and laughter and collegiality. i hope that if you’ve never attended an eblip conference, there’s one in your future. i will see you at eblip8!       university of saskatchewan   evidence based library and information practice using evidence in practice   using inventory data to enhance music collections   joel roberts music librarian university libraries university of memphis memphis, tennessee, united states of america email: jcrberts@memphis.edu   rachel scott ils librarian university libraries university of memphis memphis, tennessee, united states of america email: rescott3@memphis.edu     received: 8 aug. 2019                                                                    accepted: 6 nov. 2019      2019 roberts and scott. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29620   setting   the university of memphis is a public university in an urban setting in the united states. the university libraries use innovative interfaces, inc.’s sierra integrated library system (ils). using sierra, a comprehensive inventory was conducted in one of the university’s branch libraries, the music library. the music library houses reference materials (2,803); non-circulating scores (6,189); circulating scores of different sizes—standard (19,503), oversize (307), and   miniature (2,115); desk items such as cds and dvds (10,263); and standard circulating monographic items (14,357) in a collection totaling around 56,000 items. the school of music at the university of memphis has roughly 400 undergraduate students, 100 graduate students, and 45 faculty members who comprise the main patron base of the music library. the music library nonetheless assists a variety of patrons, including undergraduate students, graduate students, faculty, and staff from all university programs, as well as community members who are not affiliated with the university. the personnel involved in this project included two faculty librarians, two full-time staff members, and one graduate assistant. it had been several years since the collection had been partially inventoried and the status of the physical collection was increasingly disordered. although this workflow represents an example from a music library, the steps are applicable to most academic library settings because the branch includes a variety of item locations, types, and statuses.   problem   collections are not static and require ongoing analysis of their physical condition, availability, and the degree to which they meet the needs of their users. that an inventory of the music library collections was long overdue had become increasingly evident over the last few years. library staff and student workers were spending more time with patrons in the stacks attempting to locate known items. some items were merely shelved incorrectly, but other items were found to have call numbers, locations, or item statuses that differed from what was indicated in the catalogue. music library personnel kept a tally of items that required a search of the shelves and found that in roughly 25% of searched items there was an inconsistency between the information contained in the ils and the physical item. this finding underscores a significant problem: if our catalogues do not accurately describe what is available to users, we will not meet their information needs. consequently, we were interested in aligning the physical collections in the music library with the data in the ils, and in using inventory data to support collections-related decisions.   evidence   there is detailed literature on inventorying academic library collections, including studies that leverage the ils to partially automate inventory processes (ernick, 2005; sung, whisler, & sung, 2009; womack, 2010; loesch, 2011; greenwood, 2013). a local workflow for inventorying collections using sierra has been recently established and is described in barton & scott (2020). much of the literature on conducting inventories, however, is not focused on the particular needs of music library collections, which have relatively diverse formats and a host of complications not encountered with collections that are predominated by books. for example, music scores often contain multiple parts. the existence of multiple parts complicates inventory processes in that an item can appear to be on the shelf in the sense that its barcode is accounted for, but the item can still be missing parts, making it essentially useless. for this reason, we had to combine our local workflow for inventory processes with some music-specific processes, such as separately accounting for parts in multi-part items and assessing handwritten markings on notated music.   with this in mind, music library personnel scanned into .txt files the barcodes of all items available, which the ils librarian uploaded into the backend of sierra. using sierra’s “compare inventory to shelf list” program, the inventory date fields in item records were updated and inventory reports were generated. this data is our primary evidence; it reflected both the physical order of the scanned items and the representation of the scanned items in the ils. the data also uncovered items that were not in the catalogue; that is, their barcodes were not associated with records in the ils. most importantly, the data pinpointed items that were shelved incorrectly. this was useful in two distinct ways. first, the reports identified items that were simply misshelved. second, several items listed as misshelved in the reports were in fact in the correct place. further investigation, however, revealed that the call numbers printed on spine labels did not always match the call numbers in the ils. examination of these items on a case-by-case basis allowed us to determine whether the spine label contained an error, or whether the data in the call number field in sierra contained errors.   implementation   during the inventory project, we implemented strategies to benefit our collection in a variety of ways. to do this, we conducted shelf-reading to count pieces in multi-part scores, put items in shelf order, and identify damaged materials. by physically handling every item, we were able to establish the physical condition and extent to which our collections matched data in the ils. by verifying that all multi-part scores had all parts and were in usable condition, the project led directly to weeding and replacement decisions. inspection of scores during the project also led to the identification of items that needed care, whether it was erasing markings or basic repair. implementation falls into three categories: system updates, care or weeding, and acquisitions.   system updates   using the reports generated by the ils, we identified items that were inaccurately listed in the ils. we noted, for example, that some recital recordings and lps that were listed as being available in the music library were in fact in the university libraries’ storage facility. the item record location was accordingly updated globally via sierra’s global update feature. inventory scanning was done in sections based on item location, and doing so allowed us to globally update those items whose location was not accurately represented. for example, all music monuments have the item location mu51, but after scanning this section, we found items with the location for music reference or scores. we updated various item record fixed fields, including location, agency, and status, with each inventory scanning session.    we also ran reports and confirmed that several items marked as lost or missing had been scanned during the inventory process. these items could then be updated globally to reflect that they were available in the music library. relatively few items were identified that had been listed as being available in the ils, but had not been inventoried. after searching to confirm that these were not missed or skipped somehow, these items could be updated with an unavailable status in the ils and suppressed from public view. oclc holdings were also removed from these titles, as we would be unable to fulfill interlibrary loan requests for them.      care & weeding   before we started scanning the collection for inventory, each of the multi-part scores was inspected to ensure that all of the pieces were in the appropriate folder and that they were in good physical condition. scores that were missing parts were considered for weeding, or if previous use merited such, replacement. several items had extensive markings and were carefully erased when possible. if the markings had rendered the work unreadable or had otherwise damaged the score, it was considered for weeding or replacement, depending on usage and other holdings. we added prompts to remind employees to count the number of parts for those multi-part scores that did not already include a pop-up message indicating the number of pieces. after inventory scanning, the music librarian noted some instances in which several copies of lower-quality editions were available and decided to weed some of these.     acquisitions   we weeded damaged materials first. then we identified titles that were missing and no longer on the shelf. with this information, we determined titles that needed to be replaced. we ultimately did not need to purchase a great deal of content to fill these gaps. with a large-scale space reclamation project in another of the university’s branch libraries, we were able to compare their music holdings to our recently-inventoried collection and use some of their materials, which were to be weeded, to plug holes in our music collection.     outcome   the implemented changes ultimately made the representation of our physical collection more accurate. specifically, over 2,500 records were updated. some of these were items that were able to be updated en masse, such as 1,713 recital recordings that had been moved to storage years ago without the location having been changed. another 243 items were found to be missing and were deleted as a result. in addition, 16 items were found to have been sitting on the shelf with barcodes and call numbers, yet without records in the ils; these were catalogued and added to the library’s collection in the ils. around 100 monuments previously had an item status indicating that they were to be used in the library only; these were updated to show a status of available. the remainder of the items were updated with respect to their call numbers, status, or location.   as a result of this project, not only are the shelves in better physical order, but call number discrepancies have been fixed in the ils. in some cases, the call number that was in the system had not matched the call number on the spine of the book for decades, making the task of finding that particular book nearly impossible. similarly, item locations for hundreds of items, particularly for reference materials that had previously been listed in the regular stacks, were updated. there is now a sense of confidence that when an item is listed as “available,” it is on the shelf. prior to conducting inventory, when walking back into the stacks to assist a patron, there was a lack of certainty as to whether the item would be there or not.    reflection    this process was straightforward, and we encountered no major obstacles. by physically handling every item in the collection, we gained great insight to its physical condition and availability. updating our ils by running a systematic progression of reports and global updates is also a process with which we were familiar. by successfully executing this project in a relatively short time period of four months, we realized that it is indeed doable and should be prioritized on a regular and ongoing basis. because there are so many aspects to a project of this nature—erasing, scanning, reading reports, processing bulk updates to bibliographic records, and others—there are opportunities for all music library employees, including student workers, graduate assistants, library assistants, and librarians, to contribute. consequently, this large-scale project ultimately proved to be a valuable team-building venture involving close collaboration between personnel in the music library, systems, and collection development departments.   references   barton, g. p., & scott, r. e. (2020). relocating, downsizing, and merging: inventory projects to manage change in a digital environment in s. marien (ed.), library technical services: adapting to a changing environment. west lafayette, in: purdue university press.   ernick, l. (2005). floating bibs and orphan bar codes: benefits of an inventory at a small college. library resources & technical services, 49(3), 210–216. https://doi.org/10.5860/lrts.49n3.210   greenwood, j. t. (2013). taking it to the stacks: an inventory project at the university of mississippi libraries. journal of access services, 10(2), 77–89.   loesch, m. f. (2011). inventory redux: a twenty-first century adaptation. technical services quarterly, 28(3), 301–311. https://doi.org/10.1080/07317131.2011.571636   sung, j. s., whisler, j. a., & sung, n. (2009). a cost-benefit analysis of a collections inventory project: a statistical analysis of inventory data from a medium-sized academic library. the journal of academic librarianship, 35(4), 314-323. https://doi.org/10.1016/j.acalib.2009.04.002   womack, j. (2010). inventory or stockcheck? the christian librarian, 53(3), 111–113. retrieved from https://digitalcommons.georgefox.edu/tcl/vol53/iss3/4     microsoft word es_haddow.doc evidence based library and information practice 2006, 1:4      70 evidence based library and information practice       evidence summary    a combination of citation analyses can reveal the nature of a journal’s scholarly  communication, its influence in a scientific community, and the geographic location  of its authors and citers.    a review of:  bonnevie‐nebelong, ellen. “methods for journal evaluation: journal citation identity, journal  citation image, and internationalisation.” scientometrics 66.2 (jan. 2006): 411‐24.      reviewed by:  gaby haddow  senior librarian, humanities, research services, library & information service  curtin university of technology  perth, western australia, australia  e‐mail: g.haddow@curtin.edu.au      received: 28 august 2006            accepted: 12 october 2006      © 2006 haddow. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to conduct a number of citation  analyses of the journal of documentation  (jdoc), comparing the results with analyses  of the journal of information science (jis),  and the journal of the american society of  information science and technology  (jasist) to illustrate features of jdoc.     design – bibliometric study.    setting – library and information science  journal literature.    sample – citations given by and given to  the jdoc were analysed and compared to  those from the jasist and the jis. author  affiliation data were analysed from articles  published in jdoc and from articles citing  jdoc. the data were drawn from three time  periods: 1975‐2003, 1980‐2003, and 1990‐2003.    methods – journal citation identity was  examined for the period 1990‐2003. the  analysis involved calculating the number of  different journals represented by citations  given by a journal in a publishing year. the  resulting citation/citee ratio is indicative of  diversity and extent of scholarly sources  used by a journal. journal citation identity  was also examined by calculating the  number of journal self‐citations (in the  period 1980 to 2003) as a proportion of the  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:4      71 total number of citations given by the  journal. a high rate of journal self‐citations  suggests introspection or isolation from  other journals in its field. the content of the  three journals was examined for the period  1973‐2003 to determine the proportion of  scientific content (i.e. articles, notes, reviews,  and letters).     journal citation image was examined by  calculating journal self‐citations as a  proportion of the citations given by other  journals to the sample journal. the result  signifies the degree of a journal’s visibility in  its field. a second aspect of journal citation  image was investigated using the new  journal diffusion factor (n jdf). the n jdf  was carried out for each year between 1975  and 2003 and calculated the average number  of different journals that cite an article in a  sample journal. a high number of different  citing journals implies influence in the field.  the n jdf for the sample journals was  compared with their journal impact factor  (jif) over the same period.    two further analyses of data gathered from  1990 to 2003 were conducted for the journal  of documentation. the first identified  journals most frequently co‐cited with jdoc,  an analysis that can locate a journal within  or outside its field. lastly,  internationalisation analyses were carried  out for jdoc. internationalisation relates to  the geographic affiliations of authors of  jdoc articles and of authors citing jdoc.  geographic affiliation was analysed using  three classifications: affiliation in north  america, western europe, or other  geographic locations.    main results –   journal citation identity: jasist was found to  have the highest citation/citee ratio at 1.88,  while jdoc and jis had similar ratios of  1.50 and 1.44 respectively. this finding  suggests jasist draws its citations from  fewer journals than jdoc and jis. the  scientific content of jdoc ranged from 18%  to 50% in the period analysed, the lowest  proportion of the three journals. all journals  had seen a reduction in the proportion of  journal self‐citations over 23 years. average  journal self‐citations for the period were  4.3% for jasist, 3.9% for jdoc, and 3.4%  for jis.    journal citation image: the number of journal  self‐citations as a proportion of the total  number of citations given to a journal was  relatively stable for the three journals in the  period 1991‐2003. jasist had a slightly  higher rate at around 30%, the rate for jdoc  was approximately 15%, and jis showed the  greatest variation ranging from 25% to 12%.  in the years 1980 to 1990 jis was found to  have a much higher proportion of journal  self‐citations, spiking to over 85% in 1986.  jdoc and jasist self‐cited at a rate that  differed little over the full 1980 to 2003  period.     the average n jdf for jdoc increased from  just over 0.3 in 1975 to almost 0.5 in 2003. jis  had a steady average n jdf around 0.2 and  jasist an average of just over 0.3 in the  same period. a comparison of the journals’  jif in these years shows jis is the only  journal with an average jif that is  decreasing. the average jif for jdoc and  jasist increased.     jasist was the most frequently co‐cited  journal with jdoc, followed by information  processing and management and jis.     internationalisation: a large proportion of  authors publishing in jdoc were affiliated  with western european institutions with a  general trend showing decreasing numbers  of authors from north america and other  geographic areas. authors citing jdoc were  predominantly from north america in the  1990s, but by 2002 authors from western  europe were citing jdoc in larger numbers.  the proportion of citing authors from other  evidence based library and information practice 2006, 1:4      72 regions remained steady at around 10% over  the thirteen year period.    conclusions – in comparison with jasist,  the journal citation identity of jdoc shows  a broader scientific base with less  dependence upon articles from its previous  issues. jdoc is cited by a larger number of  other journals than jasist and jis,  indicating a higher degree of visibility in the  scholarly community. the journals most  often cited alongside jdoc mark it as firmly  grounded in the field of library and  information science. jdoc is attractive to  western european authors both as a  publishing channel and as a journal to  which they make reference.     commentary    in this study, well‐known analyses are  imaginatively applied to three important  research journals in library and information  science. the findings provide an overview  of the journals’ citing practice and the extent  to which the journals are cited by others.     most of the analyses conducted in this study  are common in the bibliometrics literature. it  is the combination of analyses and the new  journal diffusion factor analysis, a  newcomer to the field, that makes this study  different. the journal diffusion factor was  first proposed in 2002 and modified in 2004  by frandsen (hence the name new journal  diffusion factor). it is this analysis that  could capture attention, if as the author  suggests, the njdf emerges as a possible  alternative to the journal impact factor for  evaluating journals.     there are several shortcomings in the way  in which the study is reported. while the  methods follow a pattern like many others  in the field of bibliometrics ‐ from which it  might be assumed that thomson scientific  isi data were used as units of analysis ‐ the  source of citation data is not discussed. the  study focuses on journal‐to‐journal citations  and yet citations to non‐journal literature  would also exist in the sample journals. if  citing characteristics of journals are  indicative of their scientific base and reach  in a field, the number and nature of citations  to non‐journal literature is an important  factor to consider. no mention is made of  these citations. also confusingly, four  different time spans are used for different  analyses with no explanation as to why.  interested readers might find answers to a  number of these concerns in a paper co‐ authored by bonnevie‐nebelong, also  published in 2006, which reports the same  study in much greater detail.     the analyses discussed in the paper are not  without theoretical interest and could form  the basis of further research, particularly in  regards to the new journal diffusion factor  and internationalisation. however, it is not  immediately clear how this information  could be utilised by library practitioners.  bibliometric analyses can be useful in  collection management, although possibly  less so as publishers and database vendors  increasingly package journals.  interdisciplinary citing patterns and the age  of citations are also aspects of bibliometrics  that might have application for collection  managers making decisions about  acquisition, retention, or disposal of  resources.    the author’s conclusions seem tenuous  based on the information provided in the  paper and do not add a great deal to our  knowledge about the three journals sampled.  they are core research journals in the field  with established reputations. due to the  (assumed) dependence upon thomson  scientific isi data, it is doubtful that the  alternative methods of journal evaluation  proposed could be applied effectively to a  broader and more diverse journal set.  furthermore, it is unlikely we will see these  methods used widely in the profession  evidence based library and information practice 2006, 1:4      73 unless they are routinely calculated and  published as is the case for the journal  impact factor.     works cited    bonnevie‐nebelong, ellen and tove faber  frandsen. “journal citation identity  and journal citation image: a portrait  of the journal of documentation.”  journal of documentation 62.1 (2006):  30‐57.    frandsen, tove faber. “journal diffusion  factors – a measure of diffusion?”  aslib proceedings 56.1 (2004): 5‐11.      evidence based library and information practice evidence based library and information practice 2010, 5.4 99 evidence based library and information practice evidence summary potential fit to the department outweighs professional criteria in the hiring process in academic libraries a review of: wang, z. & guarria, c. (2010). unlocking the mystery: what academic library search committees look for in filling faculty positions. technical services quarterly, 27, 66–86. reviewed by: yvonne hultman özek senior librarian lund university, faculty of medicine, lund, sweden email: yvonne.hultman_ozek@med.lu.se received: 1 aug. 2010 accepted: 8 nov. 2010 2010 özek. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. objective – to identify key factors affecting the probability of obtaining an interview and being hired for an academic library position. design – an online survey was distributed via the following electronic mail lists: acrl, lita, collib, metro, acqnet, colldv, uls, equilibr, and alf. the questionnaire was posted via studentvoice, an assessment survey provider. setting – academic libraries in the united states. subjects – the 242 academic library search committees that responded to the online survey. methods – the authors reviewed the literature on the hiring process in academic libraries. a questionnaire for an online survey was developed. the instrument contained closed questions with the option to add comments. the survey was available for completion june 3 to june 15, 2008. main results – skills and performance of job requirements were rated as the most important criteria by 90% of the 242 academic library search committees that responded to the survey. previous academic library experience was rated as essential by 38%. the findings also showed that committees are positive towards hiring recent graduates, and over 90% check references. in addition, 75% of the respondents emphasized the importance of skills in bibliographic instruction (bi), mailto:yvonne.hultman_ozek@med.lu.se� evidence based library and information practice 2010, 5.4 100 particularly when choosing staff for public services. furthermore, of the 242 respondents, 47.52%, answering the corresponding question indicated that a relevant cover letter, correct spelling, and declaration of the candidate’s activities over all time periods are crucial aspects. those in favour of using a weighted scoring system, 37% of 218 respondents, felt that it served as a tool to level the playing field for gathering accurate information, and it also helped to improve the efficiency as well as speed of the hiring process. however, 62.84% of the respondents commented that a weighted scoring system is too prescribed, and some universities did not allow the use of this method. of 218 respondents, 65% employed evaluation forms after an interview, 38% reported that they would go beyond the applicant’s given references, and 61% felt that the applicant’s potential to fit into the department was important. the “potential fit” criteria scored the highest of these criteria: demonstrated performance of job requirements; cover letter; and knowledge of trends in latest developments in library science (p. 74). of 211 respondents, 47.39% reported that the average length of the search process was 4 to 6 months. most respondents perceived the search process as slow. conclusion – in general, the survey offered an overview of current practices of academic library search committees, which can aid those on the hiring side as well as those who are seeking a job. based on the results, the authors state that, in addition to all of the job requirements, it is vital to consider the potential fit of the applicant within the department. the hiring of candidates with less experience emphasizes the significance of fitting into the department and can be weighed against selection of individuals with more experience. this conclusion is encouraging for those who have recently graduated from library school. commentary using glynn’s (2006) critical appraisal checklist, the study raises some questions which are highlighted below. this study examined the key factors that can determine successful recruitment and also aid jobseeking librarians in deciding what to focus on in an application and what to prepare for in an interview. the analyses revealed that certain key factors have a particularly positive impact on the hiring process, and hence the main findings fulfilled the stated objectives of the investigation. nevertheless, the study does have some weaknesses. the study was based on 242 respondents, but it is not clear how many people actually received the questionnaire. the results were presented only in figures combined with the questions, and it would have been helpful for the readers if the entire data collection instrument had been presented as an appendix. including the questionnaire also would have facilitated interpretation of the authors’ statement about how the respondents’ comments aided the study. the article’s presentation of the figures directly following the related survey questions in the text was also inconsistent. thus, reading the survey findings section was difficult. this study does provide additional information about key factors that promote successful selection of new staff, but the authors could have extended the survey questions to include a broader perception of the hiring process. for example, the authors’ concluding remark in the literature review section indicates that it is critical to recruit and retain competent staff in order to support the requirements of teaching, learning, and research, and therefore search committees need to re-examine the hiring process. questions aligned with this statement were lacking in the survey. thus, it is difficult to determine how representative the results are of the studied group and the results cannot entirely be generalized. one should also keep in mind that the setting is limited to the u.s., where the hiring and tenure-track processes evidence based library and information practice 2010, 5.4 101 differ compared to many other countries. the authors could have included some background information on how the hiring and tenure-track processes work in academic libraries in the u.s. in the conclusion section, the authors assert that search committees should adjust their procedures to meet the needs of the institution, but this was not fully explored in the survey. for instance, is the institution’s or the library’s mission statement or vision taken into consideration as a context for the hiring process? it is not evident whether the question of a candidate’s potential “departmental fit” is derived from the needs of the institution. yet, according to the authors, this is an important component in the hiring process, as more than 9 out of 10 said that potential fit was extremely important. thus, based on the literature review, the authors could have created more in-depth questions related to the institution’s or the library’s goals and mission statement with the aim of gaining more extensive knowledge about the hiring process. the authors state that the hiring process is too long, but it is not clear why the respondents had this perception. the context for these statements is lacking. it is not demonstrated which type of libraries had this perception. finally, the authors conclude that 90% of the respondents viewed “demonstrated performance of job requirements” as most important, but the survey shows 58.68%. “potential fit to the department” scored higher, 60.74%. thus, the authors’ conclusion is not aligned with the survey findings. this study sheds light on existing practical factors that are highly important because they can facilitate the hiring process for both the search committees and for candidates applying for available positions. however, the following question remains: how can search committees re-examine the hiring process to meet the needs of their institutions? references glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. / evidence based library and information practice evidence summary main results – skills and performance of job requirements were rated as the most important criteria by 90% of the 242 academic library search committees that responded to the survey. previous academic library experience was rated as essential by 38%. ... evidence based library and information practice 2011, 6.3 1 evidence based library and information practice evidence summary perceptions of shared leadership within academic libraries suggest room for improvement a review of: cawthorne, j. e. (2010). leading from the middle of the organization: an examination of shared leadership in academic libraries. the journal of academic librarianship, 36(2), 151-157. doi:10.1016/j.acalib.2010.01.006 reviewed by: genevieve c. gore liaison librarian life sciences library, mcgill university montreal, quebec, canada email: genevieve.gore@mcgill.ca received: 25 feb. 2011 accepted: 02 aug. 2011 © 2011 gore. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to survey middle managers’ beliefs regarding their participation in shared leadership and their libraries’ practices of shared leadership. design – cross-sectional survey. setting – twenty-two academic libraries within four-year public master’s level institutions in the pacific northwest of the united states. subjects – a total of 115 middle managers were invited to participate; 77 completed the survey for a response rate of 66.9%. methods – selected middle managers were contacted by email a total of five times and were invited to complete a web-based survey composed of three sections. the first section contained 10 statements for rating shared leadership within their own institutions, which they were asked to rate on an eight-point likert scale from 1 (strongly agree) to 7 (strongly disagree), with 8 as an option for no opinion. the second section used the same scale to rate their levels of agreement with conceptual mailto:genevieve.gore@mcgill.ca� evidence based library and information practice 2011, 6.3 83 definitions of shared leadership from jackson’s framework. jackson’s framework consists of four components for ascertaining levels of shared leadership from both managerial and staff perspectives: accountability, equity, partnership and ownership. the third section invited subjects to provide their own definition of shared leadership. a three-part pretest served to validate the survey instrument. mean scores were calculated for each statement. main results – in the first section, there was the highest overall level of agreement (1.52) with the statement “i am accountable for the decisions within the scope of my responsibility” followed by “i share information with the senior library administration” (1.71). there was the lowest overall level of agreement (3.65) with the statement that “ideas presented at all levels of staff in the library are equally considered.” in the second section, respondents’ mean scores for jackson’s definitions of four concepts of shared leadership were 2.62 for ownership, 2.68 for both partnership and equity, and 2.77 for accountability. in the third section, respondents most often linked their definitions of shared leadership with communication, learning and collaboration. conclusion – examining middle managers’ perceptions of shared leadership may help us understand organizational trends and capacity for leadership within libraries. future research could examine shared leadership at levels below middle management as well as the relationship between accountability and shared leadership throughout the organization. commentary this study presents an impressive literature review of shared leadership, including significant cross-disciplinary references to information on the topic. however, references to lis scholarship are limited, and the author does not explicitly state whether this is due to a lack of literature. the questionnaire design also presents a number of challenges. the problem statement speaks in part of actual levels of shared leadership, but questionnaires can only gauge respondents’ perceptions of shared leadership – an important distinction that seems to be somewhat blurred although acknowledged. the definition of “middle manager” is not entirely clear, which makes it difficult to replicate the study. the actual questionnaire is not included as a supplementary file, which makes it difficult to evaluate its design or replicate the study. generally, it is preferable to use previously validated instruments when possible, particularly given the inherent difficulties of validating a survey instrument (pre-testing – which, to the author’s credit, was done – does not guarantee validity or reliability, although it does contribute to face validity). the author references what appears to be an assessment tool of shared leadership and it is unclear why the tool was not used in this study. it is also unclear whether, in section two, the questionnaire sought to establish respondents’ level of agreement with the definitions of jackson’s four components of shared leadership, or whether the respondents were being asked to rate the presence of each component within their own institution. seeing a copy of the questionnaire may answer that question. there were also some discrepancies between the mean scores reflected in the text and the mean scores reflected in the figures when referring to the same statements, which added an element of confusion to the reading, as did the lack of consistency in the way the statements were presented in the figures. the author does not discuss the geographical sampling used in the study as a source of a possible lack of external validity; however, he does make explicit the study population used. the concept of shared leadership within libraries appears to be quite a challenging area of study. it would be very interesting to know how organizational models are changing and whether shared leadership is on the rise or decline relative to other organizational models, something to which the author draws attention. / evidence based library and information practice evidence based library and information practice evidence based library and information practice 2010, 5.4 121 evidence based library and information practice using evidence in practice evaluating open access day 2009 at ohio state university anne gilliland head, health sciences copyright management office prior health sciences library and center for knowledge management ohio state university columbus, ohio, united states of america email: anne.gilliland@osumc.edu received: 16 july 2010 accepted: 29 sept. 2010 2010 gilliland. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. setting ohio state university (osu) is a large, landgrant university in the united states with a total enrollment of about 63,000 and with about 55,000 students at the main campus in columbus, ohio. the university libraries have fifteen locations across the main campus. librarians working in the areas of health sciences, mathematics, chemistry, and biology began planning for osu’s first open access day programming in january, 2009. the group chose october 19, 2009 to hold the program because it was during the week when universities and other organizations would be holding open access week events across the world (sparc the scholarly publishing & academic resources coalition, 2010). the planning group employed a wide variety of approaches to advertise and publicize the event, including messages on social media sites, submissions to campus media outlets, radio spots, posters, and others. the program for open access day 2009 was a panel discussion about open access publishing that featured faculty members who publish in open access journals, editors of open access journals, and commercial and society publishers’ representatives. a vice president from osu’s office of research set the stage with opening remarks that gave a background and overview of current issues in the open access movement. all the speakers gave substantive talks that held the audience’s attention. debate was lively but remained civil. some speakers showed a high level of advocacy toward open access, while others were more skeptical. video of the program was streamed live over the internet to osu’s regional campuses and to other remote viewers. a spirited question and answer session followed the speakers’ presentations. the open access day planning group also held a lunch after the program in order for invited participants to discuss open access day issues in greater detail. although engaging mailto:anne.gilliland@osumc.edu� evidence based library and information practice 2010, 5.4 122 conversation continued, attendance was sparse at this event. problem interest in open access publishing was high on campus because state budget cuts had prompted the ohio library and information network (ohiolink) to discontinue funding for ohio universities’ authors’ fees for some open access publications (l. hartel, personal communication, october 25, 2008). as a result, a number of researchers at osu had become more aware of how these fees had been assessed and paid, and of the amount of money involved. nevertheless, there was a sense that more awareness was needed, and the planning group hoped that the open access day 2009 program would broaden the discussion about open access publishing, both within the libraries on campus and throughout the university community. for the first open access day program, the planning group wanted a balanced presentation, one that described both the promise and the challenges of open access publishing and allowed for healthy debate and a variety of viewpoints without acrimony. the group also wanted balance in the disciplines represented, including the sciences, social sciences, and humanities. because the budget was limited, the format was a panel of local speakers and publishers’ representatives. the planning group hoped the audience for the program would come from all over campus, not only from the libraries, and believed that a panel of their peers would most appeal to faculty members on campus. the intended audience was faculty, staff, and students from both the health sciences side of campus and from the main campus community. goals for the program included making faculty, staff, and students aware of choices in publishing, increasing awareness of scholarly communications issues, and providing information on the mechanisms and costs of open access publishing. in addition, there was a desire to open up a dialogue about scholarly publishing between librarians and other faculty, staff, and students. a few weeks after the program, the planning group gathered to discuss survey results and their own reactions and to begin discussing what kind of programming would be appropriate for 2010. evidence between 45 and 50 people attended the program (the number varied because people entered and left the room during the event). the planning group gave each attendee a survey to complete, but only 11 surveys were returned. most respondents rated the panelists and moderator highly, appreciated the physical facility where the program was held, and felt that open access would be an important subject for the university in the coming year. anecdotal evidence suggested that many attendees were from one of the libraries on campus, and so it is likely that most of the survey responders were affiliated with one of the libraries as well, though the survey did not ask for that information. many of the responders to the survey commented extensively. responders were particularly interested in the panelists’ remarks about how open access might change the way universities use and evaluate impact factors and how open access publishing might change decision-making for promotion and tenure. there was also interest in information about available green (self-archiving) open access options. in the months after the program, the author conducted interviews with a variety of stakeholders to learn their perceptions of the last year’s program and to discuss what might be offered in the future. these stakeholders included the five librarians and a library staff member who had been members of the open access day 2009 planning group, a librarian who works with ohio state university’s institutional repository and chairs the university libraries lecture committee, the evidence based library and information practice 2010, 5.4 123 head of outreach at the prior health sciences library, and staff members associated with the center for clinical and translational science at ohio state university. the author used the following questions in interviews with the core open access day planning group: o what went well this year? o what should we change next year? o what topics or formats make sense to try for 2010? o are you interested in being involved with planning for 2010? the planning group members generally agreed that the program had been successful and that many library staff members had increased their understanding of the issues of open access publishing as a result. at the same time the group was also aware that very few people from outside the libraries had attended the program and that the best way to engage the wider university community was still an open question. the group discussed holding meetings focused on individual departments in 2010, but rejected the idea because several group members believed that these meetings would be poorly attended. for a larger program, planning group members suggested the following subjects: scoap3, a new model for publishing in high energy physics; open access journal “failures” (journals that were not sustainable); the importance of open access in patient information and education; and the importance of open access publishing for the use of researchers in developing countries. the subject of the effect of open access on promotion and tenure and impact factors had been of high interest to 2009 attendees, particularly because osu’s president had called for an overhaul of the promotion and tenure system (“gordon gee,” 2010). nevertheless, the planning group concluded that any in-depth discussion of promotion and tenure on campus was beyond the group’s scope. given the low turnout in 2009, the group had a lack of interest in sponsoring another lunch with invited guests. the author interviewed a library faculty member involved with the libraries’ lecture series and the institutional repository using the same questions used in the planning group’s interviews. this individual indicated an interest in being more closely involved with open access week planning for 2010 in order to provide a tie to other library programming and to contribute more specific information on the university’s self-archiving options. the author also conducted an interview with the prior health sciences library head of outreach, which centered on the likelihood of receiving grant funding and the logistics of applying for it. during this meeting and in subsequent ones, several additional sources of grant funding were identified. the final interview was with staff from the center for clinical and translational studies in order to assess the center’s interest in scholarly communications issues and programming in 2010. the interviewer started by describing the 2009 program and then asked about what sort of interest the center staff or their researchers had in open access publishing and what sort of programming would be effective. this interview was less fruitful. beyond making sure researchers comply with the nih public access policy when required, the center staff did not have a pressing interest in authors’ rights or open access. they were, however, willing to publicize programs about open access devised by others. implementation all planning group members have continued to work on programming for open access week 2010, and the group has added several new members. based on the survey and interview results, 2010 programming will expand to a full week and will include three keynote speakers’ programs and two smaller seminars. one keynote speaker will be a professor and journal editor who will speak on evidence based library and information practice 2010, 5.4 124 sustainable publishing models in mathematics. the other keynote speaker will discuss the importance of open access publishing for patient and consumer health information. she will give one lecture for the university community and one lecture for the general public. university libraries’ staff members will lead the seminars on copyright, authors’ rights, self-archiving, and the nih public access policy. in 2010 the planning group will redesign the survey to take into account the larger variety of programming to be offered and to capture information about whether or not attendees are affiliated with one of the libraries on campus. to boost the return rate, the group will offer small premiums to those who complete the survey. it would be desirable to measure whether knowledge of the issues of open access publishing improves as the result of open access week programming. the planning group considered administering an additional survey to attendees before each program but rejected the notion as too intrusive. outcome expanded programming for open access week 2010 will emphasize the themes that last year’s attendees found meaningful and that the planning group believes will draw a wider and more diverse audience. more people are involved in the planning process. at this writing, acting on advice from one of the interviews, the group has obtained grant money from two additional sources for 2010 programming. one of these successful grant applications has facilitated a partnership with learning technology groups on campus for help with publicity and outreach. the full impact and efficacy of the 2010 changes are unknown until the programs take place. to help measure that impact, in 2010 the group will use a re-designed survey and more assertive methods to improve survey results. if those results are more reliable, they may help the group learn whether the decisions they made were effective and may provide more complete guidance for future prospects in this area. reflection using open access day 2009 survey results and interviews to make decisions about 2010 programming was a straightforward process. methodical use of interviews was particularly useful to help widen planning group participation, to obtain more grant money, and to make new partnerships. what is less straightforward to determine is whether use of that evidence will help the 2010 programming meet the continuing challenge of making faculty, staff, and students outside the libraries aware of the role open access and other scholarly communications issues plays in research and teaching today. this is one of the planning group’s more difficult goals. paradoxically, most of the evidence used to plan 2010 open access week programming came from library faculty and staff, and so it is likely to reflect that group’s interests and concerns primarily. the evidence to determine the best ways to reach members of the university community who are not part of the libraries may need to come from other methods and instruments. acknowledgements the author thanks the members of the 2009 open access day planning group at ohio state university and the organizers of and attendees at the evidence-based scholarly communications conference, university of new mexico, albuquerque, nm, march 11-12, 2010 for their help and support. references gordon gee wants to talk about tenure criteria. (2010). in inside higher ed. retrieved 1 oct. 2010 from http://www.insidehighered.com/news /2010/02/05/qt/ gordon_gee_wants_to_talk_about_ten ure_criteria evidence based library and information practice 2010, 5.4 125 sparc the scholarly publishing & academic resources coalition. (2010). in open access week. retrieved 1 oct. 2010 from http://www.openaccessweek.org / evidence based library and information practice librarians working in the areas of health sciences, mathematics, chemistry, and biology began planning for osu’s first open access day programming in january, 2009. the group chose october 19, 2009 to hold the program because it was during the week wh... the program for open access day 2009 was a panel discussion about open access publishing that featured faculty members who publish in open access journals, editors of open access journals, and commercial and society publishers’ representatives. a vic... problem interest in open access publishing was high on campus because state budget cuts had prompted the ohio library and information network (ohiolink) to discontinue funding for ohio universities’ authors’ fees for some open access publications (l. hartel, ... for the first open access day program, the planning group wanted a balanced presentation, one that described both the promise and the challenges of open access publishing and allowed for healthy debate and a variety of viewpoints without acrimony. the... the intended audience was faculty, staff, and students from both the health sciences side of campus and from the main campus community. goals for the program included making faculty, staff, and students aware of choices in publishing, increasing aware... a few weeks after the program, the planning group gathered to discuss survey results and their own reactions and to begin discussing what kind of programming would be appropriate for 2010. evidence based library and information practice evidence based library and information practice 2010, 5.2  evidence based library and information practice       news/announcements    library and information science research coalition in the uk launches first national  conference        2010. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which  permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly  attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the  same or similar license to this one.    the one‐day conference, evidence, value and  impact: the lis research landscape in 2010  <http://lisresearch.org/conference‐2010/>, will  take place on 28 june at the british library  conference centre.     it will consider perspectives on the library  and information science (lis) research  landscape. it will tackle issues relating to  identifying lis research opportunities;  translating research outcomes into practice;  growing research capacity amongst lis  professionals; and developing the future uk  lis research agenda.    the programme will appeal to all lis research  stakeholders; from funders of lis‐related  research activity through to those who publish  lis research output, as well as practitioner  researchers and academics. with first‐hand  access to expert speakers and peers on the day,  delegates will develop their knowledge of the  lis research landscape, including the work of  the lis research coalition, while also  increasing their awareness of:      o the diversity of lis research  opportunities  o research funding sources  o potential research collaborators  o means of increasing the relevance of  their research efforts  o avenues for publication of their  research output   o research development opportunities  for individuals and groups  o techniques for integrating research  activities into everyday work practice    the conference includes keynote presentations  and break‐out sessions, with further  opportunities for delegates to participate  actively in the discussions  <http://lisresearch.org/conference‐2010/one‐ minute‐madness/> on the day, including a  ʺone minute madnessʺ session.     the opening keynote speaker, professor  andrew dillon of the university of texas ‐ an  lis researcher of international repute ‐ will  speak on international perspectives of uk lis  research. the closing keynote speaker,  loughborough universityʹs professor charles  oppenheim ‐ also an lis researcher of  international standing ‐ will discuss how an  examination of the evidence, value and impact  of current lis research can inform the future  lis research agenda.     afternoon break‐out sessions will be led by a  number of experts with research experience in  116 http://lisresearch.org/conference-2010/ http://lisresearch.org/conference-2010/one-minute-madness/ http://lisresearch.org/conference-2010/one-minute-madness/ evidence based library and information practice 2010, 5.2  public, academic, special and corporate  libraries; the health service; business;  publishing; consulting; training; charities and  higher education.     delegates will have their chance to take the  stage in the conferenceʹs ʺone minute  madnessʺ session immediately before the  lunch break. one minute madness is a hard‐ hitting, rapid‐fire, 60‐second opportunity way  of presenting ideas in an informal and fun  way. the format allows speakers to put across  a core idea much more successfully than is  often the case for formal conference papers.  delegates who wish to participate in the one  minute madness session sign up when they  register for the conference.    the conference venue is the british libraryʹs  state of the art conference centre. located  in  central london, the british library conference  centre is within easy walking distance of three  mainline railway stations ‐ st pancras  (eurostar terminal), euston and kingʹs cross ‐  and six tube lines, and is thus at the heart of  the uk and european transportation network.    the conference fee is £100 inclusive of vat  (£85.10 + £14.90 vat). six sponsored places for  phd students are available thanks to the  generosity of three lis recruitment firms: glen  recruitment, sue hill recruitment, and tfpl.    for further information:    find out more and book your place at lis  research coalition conference web page:  http://lisresearch.org/conference‐2010/  <http://lisresearch.org/conference‐2010/>      full explanation of ʺone minute madnessʺ:   http://lisresearch.org/conference‐2010/one‐ minute‐madness/  <http://lisresearch.org/conference‐2010/one‐ minute‐madness/>      conference secretariat: event logistics:  http://event‐logistics.co.uk <http://event‐ logistics.co.uk>      conference hashtag: #lisrc10    follow the lis research coalition on twitter:  @lisresearch  117 http://lisresearch.org/conference-2010/ http://lisresearch.org/conference-2010/ http://lisresearch.org/conference-2010/one-minute-madness/ http://lisresearch.org/conference-2010/one-minute-madness/ http://lisresearch.org/conference-2010/one-minute-madness/ http://lisresearch.org/conference-2010/one-minute-madness/ http://event-logistics.co.uk/ http://event-logistics.co.uk/ http://event-logistics.co.uk/ evidence summary   reading faculty’s research publications helps to determine which professors to target for data services   a review of: williams, s.c. (2013). using a bibliographic study to identify faculty candidates for data services. science & technology libraries 32,(2). 202-209. doi:10.1080/0194262x.2013.774622   reviewed by: giovanna badia liaison librarian, schulich library of science and engineering mcgill university montreal, quebec, canada email: giovanna.badia@mcgill.ca   received: 2 dec. 2013     accepted: 10 jan. 2014      2014 badia. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – the research project examined university faculty’s publications in order to find professors with previous data experiences. the professors could then be approached with an offer of the library’s data services. design – bibliographic study. setting – department of crop sciences in the college of agricultural, consumer, and environmental sciences at the university of illinois at urbana-champaign. subjects – a total of 62 assistant, associate, and full professors.  methods – the author searched web of science and faculty web pages to find each of the subjects’ two most recent research or review articles.  altogether, 124 articles were read to check whether data sources were used and shared.    data sources were defined as sources other than traditional citations to literature for information or ideas, such as data repositories, supplementary files, and weather stations.  data sharing was defined as publicly sharing data beyond that published in the journal article, such as providing supplementary files with the article or submitting data sets to a disciplinary repository (p. 205).      main results – thirty of the 124 articles, which were written by 20 different professors, referred to additional data that was made openly accessible.  the analysis of the articles uncovered a variety of data experiences, such as faculty who utilized repository data, published supplementary files, submitted their own data to repositories, or posted data on their university’s website.  these 20 faculty members were contacted and asked for a meeting “to discuss their data sharing thoughts and experiences and to ask whether they [saw] a role for the library in facilitating data sharing” (p. 206).  the author received a positive response from seven of the faculty members and had a successful meeting with each of them.   conclusion – a bibliographic study can be employed to select which professors to target for data services.  while this method is time-consuming, it allows librarians to gather rich data about faculty research that will help them to create customized, relevant messages to professors about the library’s data services. it also allows them to become more knowledgeable about data practices and resources in a particular discipline.   commentary   according to the author, this study describes a new application of bibliographic study design that is to help librarians determine which professors within a university are likely candidates for their library’s data services.  the project description in the article is detailed enough for readers to reproduce the study in their own institutions if they wish, and discusses the pros and cons of using the method described.  however, this reviewer feels that there is an important piece missing from the article.  the method described in the article is a means to an end, but sufficient details about the end are not provided in the article.  once the author identified the faculty to contact via the bibliographic study, how did the author create the customized messages that asked faculty to meet?  what details were in the author’s messages to convince them to meet?  only 7 out of the 20 faculty members contacted (35%) responded so it is difficult to judge whether the author was successful or not in the absence of details about the author’s communication with her audience.  also, what transpired during the meetings with the faculty who responded?  the author states that “a detailed analysis of the interviews is planned for future research” (p. 206), but inclusion of this information would have made the current article stronger and helped convince readers that the effort expended in the bibliographic analysis was indeed worthwhile.   evidence summary   model correlates many factors to undergraduates’ perceived importance of library and research activities, but low explanation power suggests more research needed   a review of: soria, k. m. (2013). factors predicting the importance of libraries and research activities for undergraduates. journal of academic librarianship, 39(6), 464-470. http://dx.doi.org/10.1016/j.acalib.2013.08.017   reviewed by: diana k. wakimoto associate librarian california state university, east bay hayward, california, united states of america email: diana.wakimoto@csueastbay.edu   received: 19 mar. 2014  accepted: 22 may 2014      2014 wakimoto. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – the purpose is to analyze characteristics and perceptions of undergraduate students to determine factors that predict the importance of library and research activities for the students.   design – student experience in the research university (seru) survey questionnaire.   setting – nine large, public, research universities in the united states of america.   subjects – 16,778 undergraduates who completed the form of the survey that included the academic engagement module questions.   methods – the researcher used descriptive and inferential statistics to analyze student responses. descriptive statistics included coding demographic, collegiate, and academic variables, as well as student perceptions of the importance of library and research activities. these were used in the inferential statistical analyses. ordinary least squares regression and factor analysis were used to determine variables and factors that correlated to students’ perceptions of the importance of libraries and research activities.   main results – the response rate for the overall seru survey was 38.1%. the results showed that the majority of students considered having access to a “world-class library collection,” learning research methods, and attending a university with “world-class researchers” to be important. the regression model explained 22.7% of variance in the importance students placed on libraries and research activities; factors important to the model covered demographics, collegiate, and academic variables. four variables created in factor analysis (academic engagement, library skills, satisfaction with libraries and research, and faculty interactions) were significantly correlated with the importance students placed on libraries and research activities. the most important predictors in the model were: student satisfaction, interest in a research or science profession, interest in medical or health-related profession, academic engagement, and academic level.   conclusion – based on the results of this study, librarians should be able to tailor their marketing to specific student groups to increase the perception of importance of libraries by undergraduates. for example, more success may be had marketing to students who are hispanic, asian, international, interested in law, psychology or research professions as the study found these students place more importance on libraries and research activities than other groups. these students may be targeted for being peer advocates for the libraries. further research is suggested to more fully understand factors that influence the value undergraduate students place on libraries and find ways to increase the value of libraries and research activities for those demographic groups who currently rate the importance lower.     commentary   this study is part of the growing body of literature concerned with determining factors associated with students valuing the academic library and demonstrating the libraries’ value to students and administrators (oakleaf, 2010). the study’s topic and suggestions for practical application of the findings to showcase the importance of libraries and increase the perception of importance in various student demographic groups will be of interest to academic librarians. for many librarians, the lack of accessibility of the statistical methods and results combined with the relatively low predictive power of the regression model, may make them cautious about applying the study’s results in outreach and marketing efforts.    the study provides an interesting look at correlating factors with student perceptions of the importance of library and research activities. the researcher notes the limitation of the low response rate, which is not a unique issue with online surveys (sax, gilmartin, & bryant, 2003) and the large number of responses may compensate for the potential nonresponse bias. using the appraisal checklist by glynn (2006), the article is valid if the reader makes assumptions about the seru questionnaire being validated; regardless of this lack of clarity, the analysis of the data is clear.   the main concern with the study is basing conclusions for marketing and outreach to specific demographics on a regression model that only explains 22.7% of the importance placed on libraries and research activities. the study found some factors that are statistically significant in predicting the importance of libraries and research activities, but many important factors were not uncovered via the seru questionnaire and the researcher calls for more research to create a robust model.   furthermore, while the statistical analyses are appropriate to the research questions, lack of clarity in communicating these methods and results for a non-statistician audience limits understandability. librarians without strong statistics backgrounds will have trouble evaluating whether the results are valid to apply leaving them to rely only on the researcher’s interpretations. the researcher is very capable of communicating statistics and limitations to a non-statistician audience as seen in another recent article written with her colleagues (soria, fransen, & nackerud, 2013). hopefully her future articles will be written as clearly, as these topics are of great interest and importance to academic librarians.   as academic librarians seek ways to assess their impact on undergraduate education and ways of marketing their value to students, research in these areas becomes increasingly important. in order to deepen the research base, as noted by the researcher, librarians should work with other academic units to collect and evaluate appropriate data to move beyond student perceptions and into library and research skills to correlate library activities with student performance.     references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154   oakleaf, m. (2010). value of academic libraries: a comprehensive research review and report. chicago, il: association of college and research libraries. retrieved from http://www.ala.org/acrl/sites/ala.org.acrl/files/content/issues/value/val_report.pdf   sax, l. j., gilmartin, s. k., & bryant, a. n. (2003). assessing response rates and nonresponse bias in web and paper surveys. research in higher education, 44(4), 409-432. http://link.springer.com/journal/11162   soria, k. m., fransen, j., & nackerud, s. (2013). library use and undergraduate student outcomes: new evidence for students’ retention and academic success. portal: libraries and the academy, 13(2), 147-164. http://dx.doi.org/10.1353/pla.2013.0010   evidence based library and information practice       editorial responsibilities      2013. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.       editor-in-chief: alison brettle   associate editor (articles): wayne jones   associate editor (evidence summaries): lorie kloda   associate editor (reviews and articles): denise koufogiannakis   associate editor (commentaries, using evidence in practice, news): alison brettle   production editor: michelle dunaway   layout support: katrine mallan   copyeditors: heather pretty (lead copyeditor), georgianne bordner, molly des jardin, linda ferguson, richard hayman, stacey penney, mary virginia taylor , alison yeoman   indexing support: pam morgan   evidence based library and information practice evidence based library and information practice 2010, 5.4 137 evidence based library and information practice news/announcements call for papers: ala/aasl educators of library media specialists section (elmss) of the american association of school librarians this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. the ala/aasl educators of library media specialists section (elmss) of the american association of school librarians invites submissions for papers to be reviewed through a blind, juried process. selected papers will be presented at the 15th annual conference of the american association of school librarians to be held in minneapolis, minnesota, 27-30 oct. 2011. educational reform of instructional practice is on the cusp of restoring a balance between authentic teaching and assessment for accountability. the common core standards movement and a new generation of assessments from the u.s. department of education will replace teaching to multiple choice tests that narrow curriculum content and constrain critical thinking, creativity, and collaboration. papers that are selected position school libraries in the context of educational reform that addresses issues such as, but not limited to, emerging adolescent literacy (grades 4-12); digital literacy; differentiation and intervention to address individual learning needs; collaboration for teaching and learning; critical thinking, innovation, and creativity; and learning content and competencies through inquiry. papers should be original research that focuses on school library practice and related subjects, or school librarianship, and be within a 5,000 word limit using apa style. research in progress that reports preliminary findings is accepted. please submit papers in electronic form to carol gordon, carol.gordon@rutgers.edu. deadline for submission is 3 jan. 2011 at 11:59 est. papers for presentations will be accepted and evaluated on an on-going basis until the deadline. authors of the selected papers will be contacted on 1 feb. 2011. / evidence based library and information practice evidence based library and information practice evidence based library and information practice 2009, 4:3     evidence based library and information practice         commentary    the call for evidence based practice: speaking louder than words    ray lyons  independent consultant  cleveland, ohio, united states of america  email: raylyons@gmail.com    received: 10 august 2009        accepted: 11 august 2009      © 2009 lyons. this is an open access article distributed under the terms of the creative commons attribution  license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and  reproduction in any medium, provided the original work is properly cited.      ʺwhen i use a word,ʺ humpty  dumpty said, in rather a scornful  tone, ʺit means just what i choose it to  mean – neither more nor less.ʺ   “the question is,” said alice,  “whether you can make words mean  so many different things.” (carroll  through the looking glass 99)                                the appearance of evidence based medicine  and practice has been heralded as a new  paradigm for professional practice in health  care, social work, human resources, and other  fields. since thomas kuhn’s monumental  book the structure of scientific revolutions  brought attention to the word “paradigm”, the  term has taken on a range of meanings, from  the latest management fad to a moderately  different viewpoint to a profoundly new  scientific model. kuhn’s main thesis is that  science does not advance by a “process in  which more accurate conceptions gradually  replace less accurate ones under the impetus  of experiment” (buchwald and smith 365).  rather, other processes are at play that lead  (usually reluctant) scientists to abandon  current theories and adopt completely new  ones. yet, it is not these processes that are  most pertinent to the development of evidence  based practice (ebp). instead, i suggest that  what happens between paradigm shifts ‐ what  might be called “scientific business as usual” ‐  is a more fruitful aspect of kuhn’s work for  any discussion of evidence‐based practice. in  the sciences, business as usual consists of  continued experimentation and measurement.    in its essence, experimentation is reality  testing. most familiar to us as the notion of  trial and error, the core idea of  experimentation is the systematic and  objective confirmation of theories in the real  world. based on the scientific method,  experimentation and related observation  techniques seek to produce evidence in the  form of data that is empirical (objectively  measurable) and verifiable (can be replicated  by others).    for more than a century, valid data has been  central to the function of managerial control in  business and public administration. its  importance can be traced from data collection  techniques devised by railroad managers in  the 19th century to the appearance of scientific  management and later incarnations as  decision‐support and data‐driven decision  63 mailto:raylyons@gmail.com evidence based library and information practice 2009, 4:3  making. the need for verifiable data is also  expressed in the modern business ideal of due  diligence, in accepted standards of auditing  practice, and in the requirement for  transparency in public management  accountability. in its most sophisticated form,  the idea of using data to adjust or refine  operations or conditions appeared as the  concept of “feedback” in general systems  theory in the 1950’s.    these two ideas ‐ verifiable data and feedback  mechanisms for decision and control processes  ‐ are central to all business improvement  domains including operations research,  systems analysis, total quality management,  business excellence, performance  management, process re‐engineering, program  evaluation, project management,  organizational learning, and evidence‐based  management. thus, in the field of business  management there has been widespread  acceptance of the ideas that valid data ought  to be part of any decision making process, and  that consequences of decisions should be  examined, for the greater part of the previous  century and perhaps even longer.     for this reason it is difficult to conceive how  the core ideas in ebp could be considered to  be radical or revolutionary. granted, by some  definitions, “paradigm shift” can refer to the  acceptance of new theories and principles by a  profession, rather than to the principles and  theories themselves. among the helping  professions the ideology of ebp might well  entail enough change to warrant the  designation “new paradigm” (although i  suppose we would need evidence to confirm  this).  still, the main innovation promoted by  ebp remains the application of established  procedures, namely, conducting incremental  experimentation and refining knowledge  based on results.    more broadly, these procedures are  components of a standard problem solving  process that has been taught for decades to  students in management, engineering, systems  analysis, and operations research.  the process  can be summarized as follows:  1. define the problem and identify  possible contributing factors.  2. specify a desired state of affairs.  3. identify and evaluate alternative  solutions for reaching the desired state  of affairs.  4. assess the feasibility and suitability in  terms of costs, benefits, and  constraints, of each alternative  solution.  5. select the most desirable solution and  confirm that its specification is  complete and well‐understood.   6. prepare a plan to implement the  solution, and then implement it.  7. collect valid data to monitor results,  comparing both the implementation  and the results to the specifications in  step 2 and step 5.  8. if needed, adjust the implementation  plan and repeat steps 6 to 8.    for several decades, a similar problem solving  process has been embraced by most helping  professions, including medicine, nursing,  social work, psychotherapy, counseling, and  rehabilitation. standard professional practice  in these fields has required that each client’s  presenting problems be assessed and  diagnosed, that a suitable, professionally  informed treatment plan be developed prior to  treatment, that treatment be meticulously  applied, that client progress be monitored  closely, and that appropriate adjustments to  the delivery of treatment be made based on  the results that are observed.    evidence based medicine (ebm) was  developed to improve treatment planning by  remedying a perceived gap between scientific  knowledge and professional practice  (gambrill 339).1 the distinctive aspect of both  ebm and ebp is therefore the objective that  evidence in the form of accurate, valid,  relevant, complete, and balanced  representations of clinical research findings be  used to inform decisions about diagnostics  and treatment planning.2    arguably, the call for evidence, particularly  the conviction that assuring the welfare of  64 evidence based library and information practice 2009, 4:3  clients requires clinicians to use “the current  best evidence”, is the source of authority for  the evidence based movements. because of  this directive, use of biased information in  clinical care decisions is considered risky and  professionally irresponsible. to avoid bias,  ebm and ebp have developed rigorous  procedures to assure that systematic reviews  are complete, balanced, and impartial (trinder  9‐10).    this call for evidence can also be heard in  evidence based library and information  practice (eblip), especially as it has been  applied to school librarianship, where the call  has been amplified into a “manifesto” (todd,  “evidence‐based manifesto”). amid the  clamour, however, school librarianship has  forsaken two central tenets of ebp. first, it has  neglected the requirement for impartiality in  the collection and interpretation of data.  second, it has diverted the focus of the  evidence based analysis away from client  needs. the result is a topsy‐turvy practice of  collecting evidence for the express purpose of  promoting school libraries as an effective  educational intervention.     todd writes, “evidence based advocacy is  clearly an intent of such [school library impact  study] documents, seeking to establish a  strong argument for the support and  continuous improvement for school libraries  … as well as to draw attention to current  issues facing school libraries” (“school  librarianship” 85).3  this is akin to saying that  “evidence based medical advocacy seeks to  establish a strong argument for the support  and continuous improvement of the medical  profession … as well as to draw attention to  the difficulties the profession faces.”    advocacy has no place in ebp and ebm. in  fact, avoiding bias and prejudiced agendas is  the main reason these practices developed in  the first place. even so, school librarianship  has resorted to “evidence” from a plethora of  advocacy research studies as a way to attempt  to rescue its libraries. referring to a  promotional publication by publisher and  distributor scholastic, todd writes, “the  research studies articulate the range of  dimensions that underpin this [positive]  impact [of school libraries on student  learning], specifying an evidence based  framework for decision making about school  libraries and their continuous improvement”  (“school librarianship” 86).4     rather than the impartial systematic reviews  that ebm and ebp insist upon, we are  presented with evidence compiled by a school  library vendor! this alone should serve as a  signal that eblip as practiced in the realm of  school librarianship is straying far from the  foundational ideas of ebm. further, todd’s  statement reveals the actual agenda of eblip  in school librarianship to be the decision about  the survival of the libraries themselves, not  about any survey of alternative interventions  on behalf of students. this misplaced focus is  obvious in todd’s contention that the central  question for evidence based school  librarianship is, ʺwhy do school libraries  matter today?” (“evidence‐based manifesto”  41).    neither formulating a biased question that  aims to confirm the relevance and efficacy of  an intervention nor compiling impact studies  or other campaigns in answer to this question  is evidence based practice. these efforts are  pure advocacy and promotion. they are  neither impartial nor client centered. ebp  requires that the effectiveness of school  libraries in meeting specific student needs be  evaluated in comparison with relevant  alternative educational interventions.   unfortunately, the importance of this type of  comprehensive evaluation is missing from the  “evidence based framework” which school  librarianship has adopted (todd “school  librarianship” 86).     since i happen to believe that reading books is  beneficial to students, i view efforts to support  school libraries as vital. nevertheless, using  research findings to promote this singular  cause is properly labeled “advocacy research”  or “action research”, not evidence based  practice. our profession’s misappropriation of  this new buzzword makes us appear careless  65 evidence based library and information practice 2009, 4:3  and uninformed. misuse of the term has  already transferred into mainstream  librarianship, for example in webjunction’s  new model of “library management  competencies” which urges librarians to “use  evidence‐based management to demonstrate  the value of the library” (gutshe 2).  in this era  of slogans, factoids, and fads librarianship  ought to be especially aware of the potential  for oversimplification and distortion of  information like this.    naturally, ebp is struggling with how its key  principles are operationalized and how they  might properly evolve. some founding  doctrines may well change or disappear in this  process. nonetheless, i believe eblip is too  quick to abandon core ebp principles in favor  of grander, if fuzzier, views of this approach.  booth recommends that librarianship not  allow “the ideal [core tenets of ebp] … to be  the enemy of the good” (54). he therefore  advocates more lenient definitions of evidence  and downplays the need for systematic  reviews of empirical research. he favours  expanded methods of data collection and  collation to “gain a broader understanding of  the issues and options involved [in a  problem]” (53).  if we want to devise the  dioramic data collection systems that booth  envisions, we might turn to the field of  operations research to explore mind maps,  rich picture diagrams, and other problem  modeling tools (53‐54). we might also call on  experts in information architecture and  knowledge management to learn how to  organize the information we collect.   however, i do not think that these activities  would rightly fall under the rubric of eblip,  nor should they use that label.    certainly, librarianship should experiment  with any tools and methods that are  reasonably likely to help accomplish its  objectives. whatever our choices, when we  expand our bag of evidence based decision  making tricks, i suggest that the tools we  assemble retain their original names, so that  we do not misunderstand their functions and  capabilities. in the long run, we might decide  that it is our bag itself that needs to be  renamed.      notes    1 a few years earlier a similar deficiency had  surfaced in the field of program evaluation:  the need for clearer understanding of  “program theory,” that is, the specific  rationale for designed program interventions,  precisely what programs should be composed  of, to which specific client populations they  were intended, and so on. see rossi, lipsey,  and freeman (165).    2  besides the “emerging giant of research  information” in ebp, the movement has also  underscored the importance of professional  judgment and client needs and characteristics  (kitson 1). see also haynes, devereaux and  guyett.    3  i suspect that the phrase “continuous  improvement” appears in todd’s statement  for its positive connotations. the phrase is  otherwise unnecessary because no “strong  argument” is needed. continuous  improvement is already part of ebp, as  continued experimentation carried out to  build up a research base and as outcome  monitoring conducted in order to refine how  interventions are applied.    4 as the previous note indicates, a decision  about whether school libraries need  continuous improvement is not under  consideration. nevertheless, this is an  interesting research question since it is  possible that these libraries are maximally  effective and efficient as they currently  operate. changes might actually diminish  their efficacy, for all we know.      works cited    booth, andrew. “the unteachable in pursuit  of the unreadable?” evidence based  library and information practice 1.2  (2006): 51‐6. 30 august 2009  66 evidence based library and information practice 2009, 4:3  67 <http://ejournals.library.ualberta.ca/i ndex.php/eblip/article/view/48/118>.    buchwald, jed z., and george e. smith.  “thomas s. kuhn 1922 – 1996.”  philosophy of science 64 (1997): 361‐ 76.     carroll, lewis. through the looking glass  and what alice found there. new  york: rand mcnally & co., 1917.    gambrill, eileen. “evidence based practice  and policy: choices ahead.”  research on social work practice 16.3  (2006): 338‐57.    gutshe, betha. ed. competency index for the  library field. dublin, oh: oclc  online computer library center,  2009. 30 august 2009  <http://www.webjunction.org/c/docu ment_library/get_file?folderid=67024 497&name=dlfe‐16500008.pdf>.    haynes, r. brian, p.j. devereaux, and gordon  h. guyett. “clinical expertise in the  era of evidence‐based medicine and  patient choice.” evidence‐based  medicine 7 (2002): 36‐8.    kitson, alison l. “the uncertainty and  incongruity of evidence based  health care.” international journal of  evidence based healthcare 6 (2008):  1‐2.    kuhn, thomas. the structure of scientific   revolutions. chicago: university of  chicago press, 1962.    rossi, peter henry, mark w. lipsey, and  howard e. freeman. evaluation: a  systematic approach. thousand  oaks, ca: sage publications, 2004.    todd, ross. “the evidence‐based manifesto: if  school librarians can’t prove they  make a difference, they may cease  to exist.” school library journal 4  (2008): 39‐43. 30 august 2009  <http://www.schoollibraryjournal.co m/article/ca6545434.html>.    todd, ross. “school librarianship and  evidence based practice: progress,  perspectives, and challenges.”  evidence based library and  information practice 4:2 (2009): 78‐96.   30 august 2009  <http://ejournals.library.ualberta.ca/i ndex.php/eblip/article/view/4637/53 18>.     trinder, liz. “introduction: the context of  evidence‐based practice.” evidence‐ based practice: a critical appraisal,  ed. liz trinder. oxford: blackwell  science, 2000.1‐16.    http://ejournals.library.ualberta.ca/index.php/eblip/article/view/48/118 http://ejournals.library.ualberta.ca/index.php/eblip/article/view/48/118 http://www.webjunction.org/c/document_library/get_file?folderid=67024497&name=dlfe-16500008.pdf http://www.webjunction.org/c/document_library/get_file?folderid=67024497&name=dlfe-16500008.pdf http://www.webjunction.org/c/document_library/get_file?folderid=67024497&name=dlfe-16500008.pdf http://www.schoollibraryjournal.com/article/ca6545434.html http://www.schoollibraryjournal.com/article/ca6545434.html http://ejournals.library.ualberta.ca/index.php/eblip/article/view/4637/5318 http://ejournals.library.ualberta.ca/index.php/eblip/article/view/4637/5318 http://ejournals.library.ualberta.ca/index.php/eblip/article/view/4637/5318 evidence based library and information practice using evidence in practice   evidence based practice for virtual reality spaces and services: a service design case study   doralyn rossmann associate professor and head of digital library initiatives montana state university library bozeman, montana, united states of america email: doralyn@montana.edu   scott w. h. young associate professor and user experience & assessment librarian montana state university library bozeman, montana, united states of america email: swyoung@montana.edu   received: 20 jan. 2020                                                               accepted: 15 may 2020      2020 rossmann and young. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29714   setting   montana state university (msu) is a public, land-grant university located in bozeman, montana, the fastest growing micropolitan city in the united states (u.s. census bureau, 2018). msu is one of only two universities nationwide classified both as a “very high research” and “very high undergraduate,” and it holds the carnegie community engagement classification. it is montana’s largest university with an enrollment of 16,766 undergraduate students, 1,949 graduate students, and 619 tenured or tenure-track faculty, as of december 2019. the university’s   undergraduate enrollment grew 36 percent in the last decade. the msu library is centrally located on campus and is msu’s second-most visited building. students use the msu library as a home-base to study between classes and to meet with their peers. the msu library provides many technology resources including laptops, gopro cameras, digital cameras, scanners, as well as technology-enhanced spaces like classrooms and group study rooms. one technology space in the msu library, the cyberdiscovery space, was initially established with the capability for data visualization and collaboration with other researchers at remote locations. because of low usage and support, efforts were made to make the space more appealing with small, phased upgrades to include the addition of virtual reality hardware and software for use by msu students, faculty, and staff, and increased oversight by msu library employees. unfortunately, the space continued to see little use. in an effort to better understand the service and to increase usage, a task force was formed to reimagine the space.    this project demonstrated the use of service design, an effective method of re-envisioning the purpose of a space, which engaged multiple stakeholders to gain a shared mission and vision and cleared steps to act upon this mission and vision. ultimately, this method can be employed by other libraries seeking to improve use of a space or service.   problem   in 2016, the msu library was encouraged to put in a request to fund a large, 4-monitor computer visualization wall which would allow msu faculty, staff, and students to connect with remote users to share large data sets, explore data visualizations, and collaboratively view and edit these data. this setup was to be modelled on a space at another university with similar software and hardware. in that other configuration, though, the computer visualization wall was in a room dedicated to the hardware and had on-site staffing to assist users. in contrast to this model space, the msu situation exhibited problems that needed to be addressed in order to design an operational service. these problems included lack of a mission and vision for the space, absence of a clear definition of the boundaries and purpose of the space, a lack of staff present in the space, and no clear responsibility or ownership for the space and its technologies.   the space examined in this study, named the cyberdiscovery space, is housed on the library’s first floor. in its initial setup, the cyberdiscovery space was loosely defined by four large computer monitors mounted on a wall – two monitors on top and two below, along with chairs facing the display wall. this monitor setup allowed for users to display content across all four monitors as if they were one, big monitor. the seating in the area could accommodate about 20 people in chairs facing the monitors. the space was not cordoned off, so users could also sit in the chairs without interacting with the computer visualization setup in any way. a high-performance computer was hooked up to the monitors, running sage software for visualization and sharing with remote users at other institutions also running sage. this hardware and software were maintained by msu university information technology (uit) personnel, who are housed off-campus. while the space was intended for any msu faculty, staff, or students to use, the on-boarding process for a new user was complex. for people to use the space, they would need an initial orientation and possible ongoing, in-person technical support. uit staff were willing to meet with users on-demand, but were not present for any walk-in users, which was especially challenging given the cyberdiscovery space’s central, highly visible location on the msu library’s first floor.   some people saw the cyberdiscovery space as having the potential to connect remote users for collaborative work. however, there were few people using the sage software at msu or elsewhere, and the openness of the space was not ideal for that type of work. most other universities using the sage software were doing so in labs or classroom spaces. the msu library did not have personnel available to support the space or the complexity of the software, so it was difficult to know how to manage or promote the space and understand its purpose.   as a result of the lack of resources and clear purpose, the cyberdiscovery space saw little use and largely languished in its first year and a half. additionally, msu’s rapid growth in enrollment has not been accompanied by a growth in the size of the library. given the library’s central location and its use by students across disciplines, the library needs to ensure all library spaces are meeting the needs of its users. the cyberdiscovery space was observably and evidentially not being used and was not clearly defined in its purpose. eventually, the uit staff who did provide some support the space left the msu or were assigned to new units. this situation and the desire to have better use of the space offered the opportunity for the msu library to assume responsibility of the space and explore other modes of use.   experience from the initial launch showed that the entry into using the sage software was too complex for the library to provide enough tech support. so instead, the library decided to focus on the visualization wall and large presentation capabilities of the space using the native windows os environment. this made it much easier for walk-up use of the space without the need for extensive orientation and training. the space got some use for library-sponsored talks by speakers who incorporated the visualization display as part of their presentations, but it still did not get much use beyond that.   in another attempt to use the space and explore emerging technologies, the library purchased a virtual reality (vr) headset to use in the cyberdiscovery space. digital library initiatives (dli) staff were the primary users of and support for vr and, in this phase, were only available to work with users on vr by appointment.   still, a lack of clear vision and mission for the space led to general confusion about its purpose, who provided support for and could use it, and what you could do in the space. with limited signage and few champions, usage remained low.   evidence   in an effort to apply an evidenced-based approach to improving the new space, the library collected and analyzed data from the following five sources: usage statistics, web analytics, conversations with library personnel, a literature review related to organizational mission and vision, and a participatory service design activity.   when the cyberdiscovery space was initially created, the library set up a reservation system on the space’s accompanying website for users, which was advertised through promotional materials and signage. while there had been observably low use of the space, this low use was verified by looking at reservation statistics. in the first semester of use, august 15-december 31, 2017, there were only nine reservations. in the same timeframe the following year, 2018, there were only 13 reservations. similarly, hits on the website for the same time periods were low with 27 sessions (2017) and 34 sessions (2018). in addition, regular walkthroughs of library to get counts of when people were using spaces at various times of the semester showed that people were regularly sitting in the cyberdiscovery space seats to study, but the cyberdiscovery space technologies were rarely turned on or in use. additionally, staff repeatedly expressed concern with a lack of understanding of purpose of the space and how to support it when a user asked about it. library staff felt these low numbers of reservations, low web traffic, and qualitative staff comments were adequate prompts for us to reconsider use of the space.   the literature suggests that creating a mission and vision for library spaces in alignment with the library’s overall mission and vision will create a shared understanding of their purpose and will lead to clearer messaging to library users (kuchi 2006; rossmann, 2019; welch & wyatt-baker, 2018). the literature also encourages involving stakeholders in grassroots efforts to develop a plan, mission, and vision for library spaces and services (casey, 2015; wójcik, 2019). msu library leadership decided to form a group to re-envision the mission and vision of the space and a plan for how that information would be conveyed to library users and staff.   task force   data collected from usage statistics, web analytics, informal staff interviews, and the literature indicated that the new space would benefit from improved operational cohesion and a more intentional service design that included input from the various staff members who deliver the service. with this evidence, in may 2019, the head of dli brought together a group of potential stakeholders in the cyberdiscovery space to form a task force with a mandate to make the space more useful and to have more known value. members of the task force were asked to commit to a weekly meeting where they would initially set a mission and vision, then identify tasks that needed to be completed, and finally, report progress on completing those tasks. the task force’s work would end in mid-august 2019 in time for the beginning of the fall semester with an official re-launch and re-branding of the space. the task force consisted of 12 people, including instruction librarians, learning and research services (i.e., circulation and reference), faculty and staff, dli faculty and staff, library marketing staff, and the user experience & assessment librarian.   service design   a service design lens was applied to help guide the collaborative work of the task force. service design is a holistic, human-centered approach to service co-creation (marquez & downey, 2016). the service design process is interested in evaluating and improving all aspects of a library service operation. importantly for a diverse group of library and it staff, service design can help build empathy and shared understanding across departments and functional units (gasparini, 2015). in applying the service design framework to the cyberdiscovery space, the aim was to build empathy for the various people involved in delivering and using the service, as well as to generate operational evidence that supports decision-making to improve the service.   the service design process was focused around a participatory workshop. workshop participants included library staff who support and deliver the service. the goal of the workshop was to produce a shared understanding of the cyberdiscovery space that could then inform a clear mission and vision statement for the service. participants were formed into small groups of three, and were asked to respond to the following prompts:   “how does the cyberdiscovery space help us achieve our mission and strategic plan?” “write a values statement for the cyberdiscovery space” “list 3 use cases for faculty in the cyberdiscovery space” “list 3 use cases for students in the cyberdiscovery space”   participants then shared their responses in their small groups and selected the top answers to each of these questions. each small group reported back to the whole group about their top choices. as a part of this exercise, the current msu library vision statement, mission statement, values statement, and strategic plan were provided to participants. those statements and plan had recently been developed with participation from all msu library employees, so the exercise was well-suited to adopt ideas from the statements and plans into language for the cyberdiscovery space.   these results were then given to the head of dli to synthesize into mission and vision statements for the space and to provide use-cases which could be listed on the cyberdiscovery space website as talking points with faculty and students and msu library employees. the service design workshop resulted in improved team cohesion through the participatory process, and a strengthened message for communicating the service to internal and external audiences. ultimately, the service design process helped generate a shared mission and vision about the purpose and use of the space.    implementation   using the new mission and vision statement as a guide for the purpose of the space, the task force identified what needed to be done to get the space ready in time for the fall semester. the tasks were then distributed among members of the task force and, each week, members reported their progress on, and eventual completion of each task. the completed tasks include:   ●       ordered and installed partitions with translucent panels in the upper part to define the space and provide more privacy for vr users; ●       renamed the space to virtual discovery space; ●       entered vr equipment into the library’s checkout system; ●       setup a location for storing vr equipment when not in use; ●       established circulation policies; ●       created new signage and updated website; ●       purchased a second vr headset for further testing; ●       hired a student assistant to staff the space for events, orientations, promotion, troubleshooting; ●       developed a central landing page for all vr at the university to reflect spaces, resources, courses, and personnel involved with vr.   the group met their goal of finishing all these projects in time for the start of the fall semester.    outcome   the user experience & assessment librarian and the head of dli established some initial measures of assessment of the space. an immediate measure was user feedback of the space. a flip chart was presented to users as they entered and exited the space, along with sticky-notes and a pen. the flip chart offered questions like “what did you like most about your experience here?", "what did you like least about your experience here?", "what was most surprising about your visit today?", "what would you want in the future for this space?", "describe your experience here today in one word or phrase", "what did you learn here today?", "what are you still confused about in this space?", "are you more prepared than when you arrived here (smiley or frowny face)?"   based on feedback to these questions, some modifications were made to the space:   ●       added an odor-absorbing container during warmer months ●       added an oscillating fan ●       created a user manual ●       created a user-orientation tutorial video ●       added signage with brief instructions and contact information ●       started checking the space daily for technical issues ●       added some new vr apps based on user requests ●       provided vr headset cleaning wipes   while there had been observed low use of the space prior to this project, a significant increase in use of the space was verified by tallying the number of reservations made through the library’s website. comparing the use over the same timeframe of august 15-december 31 in 2017, 2018, and now in 2019, there were 9, 13, and 135 uses, respectively. similarly, hits on the website for the same periods increased significantly with 27 sessions (2017), 34 sessions (2018) and 143 sessions (2019). in addition to other use of the space, four faculty members incorporated use of the space in their classes in spring 2020.   reflection   this project was fairly straightforward and would be easy to replicate with future projects and services. ideally, when launching a new space or service, there is a clear purpose, adequate staffing, and consistent messaging. following the principle and practices of service design can help create an inclusive, operational service. these elements were not initially present in the cyberdiscovery space. bringing together people who had ideas for and interest in the space through the formation of a task force allowed for the creation of a shared mission and vision. stakeholders were willing to participate knowing that they were part of a task force which had a clear end-date for their commitment. signage and webpages now clearly reflect the mission of and vision for the space, and library employees can now easily articulate the purpose of the space. the biggest challenge was making a short, but intense, time commitment to get this project done over the summer months. in the end, the project was worthwhile given the significant increase in use of the space by the library community. there is hope that the library will continue to build on this success.   conclusion   library services are often complex, multi-stakeholder operations. applying a service design lens to library operations can support collaboration across organizational units, leading to more opportunities for empathy-building and shared understanding. the case study in this article involves a new virtual reality space. we describe the complexity of such a service and the usefulness of implementing the new service with the intentionality and inclusivity of service design. as a result of our service design process, we improved the operation of our new virtual reality space to the benefit of both our staff and our users. we recommend service design as a methodology for other libraries looking to implement or modify a complex library service like a virtual reality space.   references   casey, a. m. (2015). grassroots strategic planning: involving library staff from the beginning. journal of library administration, 55(4), 329–340. https://doi.org/10.1080/01930826.2015.1038935   gasparini, a. a. (2015, february). perspective and use of empathy in design thinking. paper presented at achi 2015: the eighth international conference on advances in computer-human interaction, lisbon, portugal. retrieved from http://www.thinkmind.org/index.php?view=article&articleid=achi_2015_3_10_20121   kuchi, t. (2006). constant change and the strategic role of communication: a selective annotated bibliography. library management, 27(4/5), 218–235. http://doi.org/10.1108/01435120610668160   marquez, j., & downey, a. (2016). library service design: a lita guide to holistic assessment, insight, and improvement. lanham, md: rowman & littlefield publishers.   rossmann, d. (2019). creating an organizationally embedded strategic communication plan for libraries. library leadership & management, 33(2). https://doi.org/10.15788/201902llm     united states census bureau. (2018). metropolitan and micropolitan statistical areas population totals: 2010-2018. https://www.census.gov/data/tables/time-series/demo/popest/2010s-total-metro-and-micro-statistical-areas.html   welch, a. n., & wyatt-baxter, k. (2018). beyond metrics: connecting academic library makerspace assessment practices with organizational values. library hi tech, 36(2), 306–318. https://doi.org/10.1108/lht-08-2017-018   wójcik, m. (2019). how to design innovative information services at the library? library hi tech, 37(2), 138–154. https://doi.org/10.1108/lht-07-2018-0094 evidence based library and information practice evidence based library and information practice 2010, 5.2  evidence based library and information practice       article    the balanced scorecard: a systemic model for evaluation and assessment of learning outcomes?    tom bielavitz  assistant university librarian/assistant professor  portland state university library  portland, oregon, united states of america  email: bielavit@pdx.edu    received: 1 dec. 2009          accepted: 4 may 2010       2010 bielavitz. this is an open access article distributed under the terms of the creative commons‐ attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐ sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original  work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is  redistributed under the same or similar license to this one.    abstract    objective – the goal of this paper is to explore using kaplan and norton’s balanced  scorecard methodology as a systemic model for outcomes assessment. the  expectations of academic accrediting agencies have shifted from measurement of  inputs and outputs to that of the library’s impact on learning and demonstrating  accountability. recent literature has presented methods for performing specific  aspects of outcomes assessment. however, the scorecard methodology may provide a  systemic advantage beneficial to library administrators and managers.    methods – this paper provides a selective review of outcomes assessment in  academic libraries and a description of the balanced scorecard methodology, focusing  on its relevance to assessment and demonstration of accountability.     results – a theoretical scenario is outlined, including examples of a scorecard used  for outcomes assessment. for each example, the benefits of using a systemic approach  are examined.    conclusions – using a systems‐thinking approach to outcomes assessment may  provide significant advantages to library administrators and managers. as the model  includes traditional methods of outcomes assessment, the scorecard approach adds  elements of process improvement, identification of the inputs and outputs that create  outcomes, and a tool for communicating accountability for resources.    35 evidence based library and information practice 2010, 5.2  introduction    each year, academic libraries employ  librarians to teach students research skills, to  purchase materials and license databases, and  to create and maintain the technological and  physical infrastructure which provides access  to their resources. statistics show the number  of students attending instructional sessions,  using the catalog and library resources, or  how many are entering the building, but none  of these measure how the students’ research  skills have improved. the association of  college and research libraries (acrl) task  force on academic library outcomes  assessment report (1998)defined outcomes as  “the ways in which library users are changed  as a result of their contact with the libraryʹs  resources and programs” (section ii, para. 2).  how do we know if the library’s human and  financial resources are actually contributing to  student learning outcomes? how do library  managers know, with any precision, which  factors actually influence student learning?  how does the library demonstrate their  impact on learning to accrediting agencies and  campus administrators?     in the conclusion to dugan and hernon’s  “outcomes assessment: not synonymous  with inputs and outputs” (2002) they write,  …the profession needs to develop  knowledge, measures, and data‐ collection techniques that cut across  perspectives. the result is a more  complete view of a “jig‐saw” puzzle  entitled “the library as a partner and  contribution to achievement of the  institutional mission.” (p. 380).   to extend this metaphor, a holistic, systems  approach can view the entirety of the  assessment puzzle rather than a specific piece  of the puzzle. such a technique can be found  in kaplan and norton’s balanced scorecard  technique, first outlined in “the balanced  scorecard: measures that drive success”  (1992), which can be used for identifying the  cause and effect relationships between inputs  (budget allocations, training, personnel),  outputs (projects, instructional sessions), and  the performance drivers that have an impact  on learning outcomes. the balanced scorecard  process can help administrators make data‐ driven decisions regarding how to allocate  resources and structure programs to create  learning outcomes. this paper will provide a  theoretical scenario of a balanced scorecard  used to structure an academic library’s  information literacy program with the goals of  creating and measuring learning outcomes.    literature review    this paper is concerned with two areas of  research: the balanced scorecard and learning  outcomes assessment. as background, this  paper will first provide a brief review of the  balanced scorecard, then examine the use of  the balanced scorecard in academic libraries,  and finally, provide an overview of the current  assessment environment in academic  librarianship.     the balanced scorecard was developed  through david kaplan and robert norton’s  (1992) research into corporations’ primary  reliance on financial data as a measure of  success, and that financial measures indicate  past success, but do not assist in predicting  future success (pp. 71‐79). the common  metaphor is that using financial measures to  steer a corporation is like driving by looking  through the rear view mirror.      kaplan and norton created a “scorecard”  system based on evaluating the organization’s  performance in four perspectives: financial,  customer, internal business processes, and  learning and growth. measuring the  organization in this manner provides a  “balanced” view, rather than the traditional  financial summary. the balanced scorecard  also calls for a balance between types of  measures, and creating a scorecard entails  identifying the cause and effect relationships  between inputs, outputs, and outcomes.     kaplan and norton (1996) place great  emphasis on identifying performance drivers,  also called lead indicators, which are the  inputs and outputs that have the effect of  driving an organization towards its strategic  36 evidence based library and information practice 2010, 5.2  goals; i.e., performance drivers create  outcomes (pp. 31‐32). the balanced scorecard  literature generally includes user satisfaction  as an outcome. while the acrl task force  report (1998) agrees that satisfaction is an  outcome, it downplays the importance of  satisfaction as “a facile outcome…too  unrelated to more substantial outcomes that  hew more closely to the missions of the  libraries and the institutions they serve”  (section ii, para. 2). in this paper’s application  of the scorecard to learning outcomes,  satisfaction measures are included, but  reinforced by complementary outcome  measures.      the balanced scorecard’s four perspectives  give managers a holistic view of a complex  organization, allowing them to see the  interactions and interdependencies of the  component parts and how they contribute to  the whole. like any system, the scorecard  provides a closed‐loop cycle of creating  objectives and measures, implementing them,  gathering and analyzing data, and then  refining (and perhaps discarding and  replacing) the objectives and measures.    originally designed for the corporate sector,  kaplan and norton have cited calabro (2001)  that half of all us fortune 500 companies use  the balanced scorecard, as do 40‐45 percent of  european fortune 500 companies (para. 9). a  key element of the scorecard’s appeal to  corporate executives and non‐ profit/government administrators is its utility  in clearly communicating an organization’s  accountability. administrators in the  governmental and non‐profit sectors such as  the american institute of certified public  accountants (united states), the society of  certified management accountants (canada),  recommended the technique after the passing  of the 2002 sarbanes‐oxley act (sox), citing  the scorecard’s utility for demonstrating  accountability. sox is a legislative reaction to  the accounting scandals concerning enron,  adelphia, and others, the goals of which are to  improve corporate accounting controls,  increase regulations, and restore consumer  confidence (niven, 2003; office of economic  analysis, united states securities and  exchange commission, 2002).      adopters of the balanced scorecard in  academic libraries have implemented kaplan  and norton’s process differently,  experimenting with how it could be used.  lloyd (2006) outlined the scorecard’s  applicability to academic libraries and  summarized the process for creating a  scorecard. lloyd identified several critical  success factors in implementing a scorecard,  including establishing a clear mission;  ensuring strong leadership support, selecting  the correct measurements, communicating the  performance results to constituencies,  strategically motivating staff, and making  evidence based decisions based on scorecard  results.      poll (2000) aimed to “establish an integrated  controlling system and to collect and evaluate  performance as well as cost data for  management decisions” (p. 709). poll focused  on standardized data sets, such as iso/dis  2789 and iso 11620 that would allow libraries  to benchmark their scorecards. self (2003)  implemented a scorecard at the university of  virginia library with the goals of gaining  control of their data‐collection efforts, setting  measurable goals, and for the “intelligibility”  of the methodology (para.6). kettunen (2009)  used the scorecard concept as a framework for  evaluating the strategic plans for twenty‐nine  finnish libraries, though not for the actual  creation of the libraries’ strategic plans.  gerryts and pienaar (2000) utilized the  scorecard as a tool for managing  organizational renewal. hernon and dugan’s  “an action plan for assessment in your  library” (2002) discussed the scorecard as a  tool for process improvement (p. 68).      in the last quarter century, higher education  accreditation has moved towards measuring  institutional quality in terms of learning  outcomes. the acrl task force (1998) stated  that “the association has no statement on  outcomes assessment, and that its standards,  largely written as output measures, are out of  step with the practices and philosophy of  37 evidence based library and information practice 2010, 5.2  regional and accrediting agencies and state  higher education agencies” (section i, para. 2).  ten years later, academic libraries still seek  improved means of communicating the impact  of their work, and are at risk if they are not  able to substantiate the impact of their  resources on the individual users. indeed, in  lakos’s (2007) interviews with library  directors, he found that “some directors  identified interest from their campus  administrators in impact data related to  learning outcomes but acknowledged that the  library is not yet viewed as central to these  outcomes” (p. 445).       the 1998 task force made recommendations  to the acrl board:   1. policy: …the acrl board adopt an  explicit policy of endorsing the  development and use of outcomes  assessment among academic libraries.  2. accreditation: …acrl foster  continued cooperation with the  regional and specialized accrediting  agencies. (section v)    despite the recommendations of the acrl  task force, libraries continue to largely rely  on input and output measures. the acrl  task force states, “…the purpose of all  inputs…is to achieve outcomes” (section ii,  para. 6). inputs may identify the resources that  create outcomes, and outputs may illustrate  the use of resources which are used in creating  outcomes, but neither is effective in actually  measuring learning outcomes. librarians  should continue to collect the standard  input/output data for various purposes, such  as ongoing longitudinal studies and surveys,  but must be more effective in illustrating the  correlation between inputs and how they  manifest in useful outputs and outcomes.        in the united states, the prominent  professional library organizations have  pursued very different avenues of research in  assessment. whereas the acrl task force  focused on assessment of learning outcomes  for the purposes of accreditation, the arl  new measures initiatives do not address  student outcomes, learning outcomes, or  connect any of the measures to accreditation.  rather, arl’s statsqual program includes  libqual+, which focuses on user satisfaction;  arl statistics, which collects data on the  input and output measures of the member  libraries; e‐metrics, measuring electronic  resources, and climatequal, which measures  organizational climate and diversity.     aims    dugan and hernon (2002) identified the  relationship between evaluation and  assessment:  outcomes assessment measures the  contributions that the library makes to  the institution’s educational mission  as a whole. evaluating the process of  conducting assessment on student  learning outcomes is designed to  improve library services through the  application of the feedback loop  within a typical systems planning  model. it is intended to identify areas  in which the library could improve the  methodologies deployed as the means  to affecting learning changes in the  individual. (p. 378)  using a theoretical scenario, this article will  propose an approach using the balanced  scorecard that helps library administrators  identify the inputs (financial and human  resources) that impact learning outcomes,  establish a system for improving the  instructional process, and provide a tool to  communicate the value that the library  contributes towards the institution’s learning  goals. those interested in evidence based  practice in librarianship should find the  balanced scorecard relevant for its focus on  gathering and using data to produce desired  outcomes. this paper will illustrate how the  scorecard facilitates data‐driven evaluation of  the learning process, leading to improving the  learning outcomes of users of academic  libraries.    this article will rely on the definitions of  outcomes, inputs, outputs and standards as  defined in the acrl task force report (2002):  38 evidence based library and information practice 2010, 5.2  39 outcomes: the ways in which library  users are changed as a result of their  contact with the library’s resources  and programs.    inputs: the raw materials of a library  program‐ the money, space, collection,  equipment, and staff out of which a  program can arise.     outputs: quantify work done, ie:  number of books circulated, number  of reference questions answered.  (section ii)    components of a balanced scorecard     a brief description of the components of  scorecard will be useful in understanding the  scorecard methodology. objectives and  metrics are created for each of the four  perspectives of the balanced scorecard:  financial, internal business processes,  learning and growth, and the customer.  essentially, the goal is to identify those  measurable elements, which when improved  upon, advance objectives. as an example,  consider training for a marathon and the  training regimens published in runners’  literature: in the 1st week, one runs an average  of 5 miles a few days a week. by the 8th week,  one is averaging 12 miles, and by the 13th one  is running 26 miles. the regimen increases  weekly distance, and types of training  sessions, until one is theoretically prepared to  run a marathon. in this example:    • strategy statement: the strategic purpose –  to complete a marathon  • objective: a specific action plan – engage in  a daily training program in order to  compete in this year’s new york  marathon  • initiatives: the projects, tasks, and ongoing  work undertaken to achieve the objective  – the daily running regimen   • input measures: resources devoted to  initiatives: time, money (running shoes,  etc), and energy put into training  • output/lag measure: the tangible results of  input resources – miles run  • lead indicator: an indicator that signals  future success – achieving weekly “miles  run” targets increases the possibility of  completing a marathon  • outcome measure: the resulting change in  an individual due to the impact of inputs –  one’s ability to run great distances;  compete in a marathon; increased  cardiovascular health    it is a simple metaphor, but illustrative of the  subtle differences between types of measures  as well as how the same metric is used for  different purposes. measured alone, “miles  run” is a simple output of specific inputs:  time, money, energy allocated to running. in  the context of the strategy statement “i want to  complete a marathon,” the “miles run” metric  becomes a lead indicator of potential  marathon completion; achieving the metric  targets are mileposts by which the potential  for future success may be measured.     scorecards created for the for‐profit sector are  structured so that all the objectives are linked  in cause and effect relationships with the goal  of improving finances. in assessing student  learning, the focus is instead on the customer  perspective (niven, 2003). creating a balanced  scorecard for student learning outcomes is a  matter of framing hypotheses which are  proven, over time, by observation of the  correlation of inputs, outputs, and whether or  not they are resulting in the desired outcomes.     a “strategy map” is a visual representation of  the hypothesized cause and effect  relationships between performance objectives  of a balanced scorecard. the function of the  map is to provide “the missing link between  strategy formulation and strategy execution”  (kaplan & norton, 2004, p. 10). a good map  clarifies exactly what needs to happen, in a  logical order, to achieve outcomes. it visually  tells a story about how the library creates  value for the library user. funding, in the  financial perspective, translates into an ability  to provide training and educational  opportunities for staff (the learning and  growth perspective). an adequately trained  staff can perform higher‐level services, utilize   evidence based library and information practice 2010, 5.2        fig. 1. strategy map    new skills, and instruct users to a greater  degree (the internal business process  perspective). these three perspectives describe  the logical cause and effect progression of  adding value to services, and produce learning  outcomes (the customer perspective). figure 1  provides a strategy map which illustrates the  cause and effect between the scorecard’s four  perspectives for a scenario which relates to an  academic library’s information literacy  program. this scenario is discussed  throughout the remainder of the paper.      a theoretical scenario of using the  scorecard as a system for outcomes  assessment and evaluation    the following expository section describes a  theoretical scorecard designed to systemically  assess the learning outcomes of an academic  library’s information literacy program. the  scenario is a combination of commonly‐ accepted practices presented together for the  purpose of illustration. in reality, such a  transition is not a simple process and would  indeed require a full scale project cycle of  planning, implementation, and revision.  additionally, creating and implementing an  information literacy program implies a  significant degree of organizational change.  this article will not address issues of  organizational change and development in  implementing an information literacy  program.    the scenario is as follows: a large university  library must respond to the requirements of  administration to quantify how the library  supports university strategic priorities.  campus administration and faculty have  recently voted upon and adopted a set of  undergraduate campus‐wide learning  outcomes, one of which is information literacy.  the librarians feel that there is a strong  alignment between the information literacy  outcomes of their instructional program, the  campus‐wide learning outcome requirement,  and the university’s strategic priority to  “improve undergraduate learning and  retention.”  a task force is charged to review the research  on the current information literacy literature,  techniques, and tools and to create a proposal  for an information literacy program.  hypothesizing that a student who is  information literate will be better prepared to  succeed and earn a degree, the team  determines that improving the library’s  information literacy program is the best way  to meaningfully contribute to the university’s  strategic priority. the team decides to use the  40 evidence based library and information practice 2010, 5.2  balanced scorecard as a management system  for evaluating and assessing the program, and  adopts the university priority to “improve  undergraduate learning and retention” as  their scorecard’s strategy statement.      customer perspective    with a ratio of 1400 students per instructional  librarian, an essential characteristic of the  program must be scalability. with this in  mind, the committee focuses on creating an  instructional program with two initiatives:   1) the instructional librarians will  focus on a departmental outreach  program with the goal of building on  existing library/departmental liaison  relationships, wherein the librarians  will work with course instructors to  integrate information literacy concepts  into curriculum.  2) create online information literacy  content available via course  management software and the  library’s website to supplement the  library’s departmental liaison efforts  and provide instruction for distance  learners.    a significant hurdle will be to gain the  cooperation of faculty to allow the librarians to  add content to their course management  system sites.     the ability to measure and report the actual  impact of information literacy on student  learning is necessary to establish the library’s  contribution towards the university’s  academic priority. the library uses  libqual+, a web‐based qualitative survey  tool which uses 22 questions to measure  library users’ minimum, perceived, and  desired levels of service quality in three  dimensions: information control, affect of  service, and library as place, including five  questions regarding information literacy  outcomes. using libqual+ in an outcomes‐ based scorecard is not necessary; other  methods are also possible, such as an in‐house  designed survey for example. the task force,  agreeing with acrl’s opinion that satisfaction  is a minor outcome, decides to use libqual+  results, but supplemented by qualitative  analysis of the submitted comments, and  followed‐up by a series of targeted focus  groups which can provide important feedback  for improving the program (association of  research libraries, 2010).    in order to actually measure student learning  of information literacy, the task force decides  to evaluate and implement a web‐based  information literacy standardized test. there  are a variety of tests currently available, each  with its own features, and some, such as  project sails, are valid and reliable  (dependent on sample size). the sails test  allows pre‐ and post testing on cohorts, but  does not provide individual student test scores  (project sails validity and reliability, 2008).    example scorecard for the customer  perspective    objective: improve undergraduate  information literacy skills.  hypothesis: teaching information  literacy skills will provide  undergraduates with the necessary  skills for academic success.   initiative: libqual+ provides a  measure of user satisfaction with the  library’s information literacy efforts.  monitor the libqual+ “information  literacy outcomes questions” scores;  perform qualitative analysis of  information literacy outcomes  survey comments; and perform focus  group interviews to gather specific  feedback on the information literacy  program.    measure: libqual+ information  literacy outcomes undergraduate  mean scores versus previous survey  results.  type of measure: lag/outcome.  target: 10% increase in mean  libqual+ information literacy  outcomes undergraduate scores  versus previous survey results.    objective: improve undergraduate  information literacy skills.  41 evidence based library and information practice 2010, 5.2  hypothesis: teaching information  literacy skills will provide  undergraduates with the necessary  skills for academic success.   initiative: choose and implement a  standardized information literacy test  (e.g., sails) to be administered by  librarians working with departmental  faculty.  measure: test scores by acrl skill  set; this will provide targeted  feedback for improving specific  aspects of the program.  type of measure: lag/outcome.  target: x% improvement over  previous year (baseline to be  established).    internal business process perspective    successfully integrating information literacy  into the curriculum is the result of a  coordinated effort between administrators,  faculty and librarians. administration and  faculty have shown leadership and have set  expectations through the adoption of the  campus‐wide learning outcomes. the library  intends to position itself as a leader and  collaborator in the university’s information  literacy efforts.     as they seek to establish leading indicators,  they see that their best leverage points are the  liaison relationships established with course  instructors to incorporate information literacy  into the curriculum. a performance driver is  created: each instructional librarian will seek  to establish a certain number of liaison  relationships per term, building a community  of instructors that are actively incorporating  information literacy into their classes. another  measure is established: a certain percentage of  faculty relationships actively teaching  information literacy as part of appropriate  courses.    the committee also discusses the need for  balancing academic freedom versus the need  for a level of consistency within the  information literacy program. while  consistency among the librarian’s techniques  for integrating information technology into  curriculum is not necessary to achieve desired  outcomes, consistency is necessary for  determining which methods are effective and  which are not. the balanced scorecard is a  closed‐loop system focused on process  improvement. if each librarian uses a different  approach, it is difficult to determine which are  adding value to the program and which are  not. to this end, when working with  departmental faculty, librarians would use the  same materials, provide the same services, and  teach the same concepts based on acrl’s  information literacy skill sets (acrl 2000).    example scorecard for the internal business  processes perspective    objective: librarians develop services  and materials to help departmental  faculty integrate information literacy  concepts into curriculum.  hypothesis: working with instructors  to integrate information literacy is the  most effective and scalable method.  initiative: instructional librarians to  work with departmental faculty to  integrate information literacy into  curriculum; each instructional  librarian will set goals to contact a  number of departmental faculty per  term; a certain percentage of those  relationships will be targeted for  partnering on information literacy.  measure: quantity of partnerships.  type of measure: lead indicator.  target: x per term.    objective: provide online information  literacy tutorials.  hypothesis: online tutorials will  supplement departmental faculty’s  information literacy instruction,  support distance learners, transfer  students, and provide easily‐accessible  instruction for all members of the  community.  initiative: create a series of multi‐ media information literacy tutorials  for each acrl information literacy  skill set.  42 evidence based library and information practice 2010, 5.2  measure: test scores by acrl skill  set; this will provide targeted  feedback for improving specific  aspects of the program.  type of measure: lag/outcome.  target: x% improvement over  previous year (baseline to be  established).    objective: raise awareness among  university faculty, staff, and students  of the information literacy program  and objectives.  initiative: create a marketing plan to  support the information literacy  program.  measure 1: quantity of library  information literacy webpage “hits”.  type of measure: lag.  target: x% improvement over  previous year (baseline to be  established).  measure 2: number of students that  have taken the standardized  information literacy test per term.  type of measure: outcome.  target: x% improvement over  previous year (baseline to be  established).      learning and growth perspective    the task force realizes that they must develop,  amongst instructional librarians, a shared  understanding of techniques and methods to  disseminate information literacy teaching that  have the greatest impact on undergraduate  student learning in their campus environment.  the task force presents its research on the  current information literacy literature,  techniques, and tools, and creates a proposal  for an information literacy program. the  measure of this objective, which is both an  input and lead indicator, is the number of  instructional librarians that ultimately adopt  and promote the instructional program.        example scorecard for the learning and  growth perspective    objective: create a shared  understanding of options and  methods to disseminate information  literacy teaching that have the greatest  impact on undergraduate student  learning in our campus environment.  hypothesis: though instructional  faculty are the campus information  literacy experts, a training program  will ensure a certain amount of  consistency in methods used by  librarians to provide services, tools,  and instruction to departmental  faculty and users.   initiative: create and implement a  standardized information literacy  program.   measure: number of instructional  librarians working within information  literacy program parameters.  type of measure: lead.  target: all library instructional  faculty will be working within the  information literacy program  parameters.    financial perspective    resources are required to accomplish  objectives, and identifying and allocating the  budgetary and human inputs draws a direct  connection from these resources to the  outcomes. visibly, this is best seen in the  strategy map that connects the cause and  effect relationships between the library’s  allocation of funds and personnel, the value  that is added by the library’s intellectual  capital, and the impact on student learning of  information literacy. it is also very useful to be  able to demonstrate to university  administration the direct effect of allocated  funds.            43 evidence based library and information practice 2010, 5.2  example scorecard for the financial  perspective    objective: allocate budget for  standardized information literacy  testing.  initiative: choose and  purchase/license a standardized  testing system.  measure: a standardized testing  system has been purchased/licensed.  type of measure: input.  target: tasks completed on schedule  per project plan.     objective: allocate budget and  staffing for implementing a project to  create online tutorials, content, and  license subject guide software.  initiative: draft a project plan to  create content, tutorials; license  software for creating subject guides.  measure: progress made towards the  project plan timeline.   type of measure: input.  target: tasks completed on schedule  per project plan.    discussion    the benefits of using the scorecard become  clear when progress towards the objectives is  cyclically measured and one can begin to test  the hypotheses that structure the scorecard.  chosen measures may or may not be useful,  and others may require several assessment  cycles before a conclusion can be reached.  even when targets are not met, the scorecard  is valuable to identify the initiatives that are  ineffective, and to provide a basis for process  improvement.  evaluation of the program is  an iterative process.    in the customer perspective, the focus is on  measuring the library’s contribution to  learning outcomes. the data collected in this  perspective are useful to communicate library  effectiveness to stakeholders. within the  university community, library administration  may use these data as metrics to support the  university’s academic priorities, as evidence of  the library’s accountability in its use of  funding, and to advocate for increased  institutional funding. for external  stakeholders, the library may use these data as  metrics of learning outcomes for accreditation.    measures in the internal business processes  perspective combine the resources described  in the financial perspective with the results of  the initiatives of the learning and growth  perspective to create the outcomes in the  customer perspective. this perspective  provides many opportunities for  organizational introspection: will librarians  meet their outreach goals? what financial or  learning and growth inputs are factors? is the  information literacy marketing plan effective  in spreading awareness? the data collected in  the internal business processes perspective  allow managers to establish the validity of the  supporting initiatives and measures from the  other perspectives.    the learning and growth perspective  combines the raw inputs of time, effort, and  funding to produce learning outcomes among  staff. it can produce a staff which is better  prepared to meet the challenges of the future  and achieve organizational goals. success of  the scenario’s learning and growth objectives  hinge upon library faculty using a consistent  pedagogy in teaching information literacy. if  the targets are not met, what were the factors?  were the librarians divided in the chosen  approach? what was the reason for lack of  participation? for example, if lack of time to  participate in the effort is cited as a cause, a  library manager may then make informed  decisions about how to solve the problem;  perhaps the target measure would be achieved  with more substantial inputs?    the adequacy of the initiatives and targets set  for the financial perspective is determined in  the other perspectives; if the targets of the  other perspectives are not achieved, it may be  because of insufficient allocations in the  financial perspective. if the inputs in the  financial perspective are correct, the data may  help to make convincing arguments to  stakeholders, such as: with greater funding,  44 evidence based library and information practice 2010, 5.2  could the program be expanded to better meet  goals, or broadened in scope to include  graduate students? conversely, during  challenging economic times, a library manager  could reasonably predict the impact of  reducing inputs.    even when the data indicate that targets have  not been met, the scorecard is useful as a  diagnostic tool for process improvement. or, if  it is determined that the hypothesized cause  and effect relationships that were supposed to  drive the outcomes are false, the scorecard  provides administrators with an opportunity  to make informed corrections to their strategy.  an examination of the results may also  prompt further research. for example, is there  a correlation between the libqual+ survey  participants’ perceived satisfaction with the  library’s information literacy program and  the sails test scores?    conclusion    this article used a theoretical scenario to  illustrate the balanced scorecard’s use as a  systemic model for evaluating and assessing  an academic library’s learning outcomes.  starting with defining the desired outcomes  (in this scenario, improving undergraduate  information literacy skills) the scenario  described how the task force worked  backwards, and created hypotheses of the  cause and effect relationships between the  initiatives and financial and human resources  that are required to achieve outcomes. the  task force established performance drivers,  which are the measures that create outcomes.  development of the instructional program  through the scorecard’s four perspectives  helps to ensure that all of the factors for  success are identified, and if not, the scorecard  is useful as a diagnostic tool to facilitate data‐ driven evaluation of the instructional process.  creation of a scorecard to assess learning  outcomes simultaneously breaks down the  process into its elemental parts and aggregates  the parts for the completed “jig‐saw” puzzle  picture called for by dugan and hernon  (2002).     designed for the private for‐profit sector, the  scorecard was developed to create the  outcome of financial success. soon, the public  sector began to adopt the balanced scorecard  as a system to achieve and communicate their  effectiveness in serving the public. accrediting  agencies are placing increased demands on  institutes of higher education – and their  libraries – to prove that they are creating  learning outcomes. the example illustrates  that the balanced scorecard can serve as a  “typical systems planning model” to evaluate  and assess an academic library’s learning  outcomes.      references    association of college and research libraries   (1998). the acrl task force on   academic library outcomes assessment  report. chicago, il: american library  association.  retrieved 19 may 2010  from http://www.ala.org/ala/  mgrps/divs/acrl/publications/whitepa pers/taskforceacademic.cfm    association of college and research libraries     (2000). information literacy competency   standards for higher education. chicago,  il: american library association.  retrieved 19 may 2010 from  http://www.ala.org/ala/mgrps/divs/ac rl/standards/standards.pdf     association of research libraries (2010).  libqual+ survey faq. retrieved 19  may 2010 from  http://www.libqual.org/about/about_s urvey/faq_survey    calabro, l. (2001). on balance. cfo magazine,  17(2). retrieved 19 may 2010 from  http://www.cfo.com/article.cfm/29916 08     dugan, r.e. & hernon, p. (2002). outcomes   assessment: not synonymous with  inputs and outputs. journal of academic  librarianship, 28(6), 376‐380.    45 evidence based library and information practice 2010, 5.2  46 gerryts, e., & pienaar, h. (1999). a key to the  new library. abstracts and fulltext  documents of papers and demos at  the international association of  technological university libraries  conference, may 17‐21, 1999: the  future of libraries in human  communication. chania, greece.    hernon, p. & dugan, r.e. (2002). an action  plan for outcomes assessment in your  library. chicago, il: american library  association.    kaplan, r. s., & norton, d. r. (1992). the  balanced scorecard: measures that  drive performance. harvard business  review, 70(1),, 71‐79.     kaplan, r.s., & norton, d. r. (1996). the  balanced scorecard: translating strategy  into action. boston, ma: harvard  business school press.    kaplan, r.s., & norton, d. r. (2004). strategy  maps: converting intangible assets into  tangible outcomes. boston, ma:  harvard business school press.    kettunen, j. (2007). the strategic evaluation of  academic libraries. library hi tech,  25(3), 409‐421. doi:  101108/07378830710820989.    lakos, a. (2007). evidence‐based library  management: the l challenge. portal:  libraries and the academy, 7(4), 431‐450.   lloyd, s. (2006). building library success using   the balanced scorecard. library  quarterly, 76(3), 352‐361.    niven, p.r. (2003). balanced scorecard step‐by‐ step for government and nonprofit  agencies. hoboken, nj: john wiley &  sons, inc.    office of economic analysis, united states    securities and exchange commission   (2009). study of the sarbanes‐oxley act  of 2002 section 404 internal control over  financial reporting requirements.  retrieved 19 may 2010 from  http://www.sec.gov/news/studies/200 9/sox‐404_study.pdf       poll, r. (2001). performance, process, and  costs: managing service quality with  the balanced scorecard. library trends,  49(4), 709‐717.     project sails validity and reliability (2008).  retrieved 19 may 2010 from  https://www.projectsails.org/abouttest /validation.php?page=abouttest    self, j. (2003). using data to make choices: the   balanced scorecard at the university  of virginia library. arl bimonthly  report, 230/231. retrieved 19 may 2010  from http://www.arl.org/bm~doc/  balscorecard.pdf                                    evidence summary   surveying north american academic library websites for instructional outreach and delivery reveals a broad range of approaches employed   a review of: yang, s. q., & chou, m. (2014). promoting and teaching information literacy on the internet: surveying the web sites of 264 academic libraries in north america. journal of web librarianship, 8(1), 88-104. http://dx.doi.org/10.1080/19322909.2014.855586   reviewed by: heather coates digital scholarship & data management librarian indiana university-purdue university indianapolis (iupui) university library indianapolis, indiana, united states of america email: hcoates@iupui.edu   received: 25 aug. 2014 accepted: 11 nov. 2014      2014 coates. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine the extent to which academic libraries have used the web to market and deliver information literacy both as a service and as a concept.   design – survey of web content.   setting – websites of north american academic libraries.                                                                        subjects – a random sample of 264 libraries selected from peterson’s four-year colleges.   methods – the investigators reviewed and analyzed content on academic library websites by recording the presence of various types of information. presence was recorded for the term information literacy, tutorial content, guides and tests, and delivery of information literacy instruction. the frequencies of tutorials and guides were also reported.   main results – approximately 65% of the libraries used their website to promote instruction, while 30% did not mention information literacy or library instruction. a wide range of terminology was used to denote library instruction, but information literacy was not highly used. approximately 5% of libraries had no public web presence. research guides, tutorials, or both were provided by 64% of libraries. more than 300 tutorials in a variety of formats, including adobe flash videos, static web pages with little or no animations, webcasts, documents, and presentations were offered by 111 libraries. the tutorials addressed general research topics, databases, concepts and technical skills, among others.   conclusion – while the majority of academic libraries sampled have incorporated information literacy and library instruction into their web presence, it is unclear why nearly one third did not mention these activities. further study is needed to benchmark how libraries are using the web for instruction and outreach.     commentary   the way academic libraries use their websites to promote information literacy and library instruction is poorly documented. this is due, in part, to the way librarians discuss web and instructional technologies. such discussions tend to focus on functional aspects of particular platforms (e.g., course management systems, libguides, etc.) or delivery mechanisms (e.g., videos, games, etc.). another challenge in studying this area is that contact with patrons happens across many campus settings, so this broad context is difficult to measure as a whole. this study attempts to document the use of a particular type of web presence (i.e., library websites), but implications for use are unclear due to limitations of the selected method.   in applying the ebl critical appraisal checklist (glynn, 2006), several concerns arose regarding study validity. the primary limitation of this study is the assumption that library websites provide an accurate gauge of a library’s information literacy outreach and instructional activities. for example, the methods employed in this study would not capture instruction that is described or delivered in other sites such as course management systems and libguides. it is difficult to assess the quality of the data reported due to a lack of rationale for the selected methods and insufficient procedural detail for data collection and coding. although the data collection spreadsheet is included in the appendix, it is not clear how the authors gathered data from the library websites. was library content on other public platforms or websites included? how did the investigators browse or search the web content? how did the investigators ensure inter-coder reliability? the answers to these questions have a significant impact on the validity of the study, which is questionable based on the available information.   further research on the use of various web platforms to promote and deliver information literacy instruction is necessary to identify effective outreach and instructional strategies for various student populations. first, we must clarify the distinction between raising awareness of library services and information literacy advocacy and instruction. second, in examining the library use of various platforms, we also need to be cognizant of the fact that very few students start their research at the library homepage (timpson & sansom, 2011). unfortunately, this study does not deliver immediately usable results for academic librarians. it does provide valuable lessons for future research, including the importance of developing focused and answerable research questions. there is also a need for longitudinal surveys to characterize the broad landscape of library technology use for instruction and outreach.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399.   timpson, h. & sansom, g. (2011). a student perspective on e-resource discovery: has the google factor changed publisher platform searching forever? the serials librarian: from the printed page to the digital age, 61(2), 253-266.  http://dx.doi.org/10.1080/0361526x.2011.592115 evidence based library and information practice evidence based library and information practice 2011, 6.2 64 evidence based library and information practice evidence summary public librarians with the highest retention rate are more likely to choose their entire career path in public libraries a review of: noh, y. (2010). a study analyzing the career path of librarians. the journal of academic librarianship, 36(4), 329–346. reviewed by: nazi torabi reference and instructional librarian university of western ontario london, ontario, canada email: ntorabi@uwo.ca received: 01 mar. 2011 accepted: 04 may 2011 2011 torabi. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objectives – the main objectives of this study were the following: • to analyze the career path and career movement of librarians in korea • to identify and compare factors influencing the career movement path of chief librarians in public libraries and other librarians • to determine library positions’ turnover rates, average career retention, career reinstatement, proximity between careers, and proximity between different libraries design – survey questionnaire. setting – one survey conducted in college libraries, public libraries, special libraries, school libraries, and library-related service providers in korea, and another in public libraries in korea, targeting chief librarians only. subjects – librarians were identified from the 2008 korean library yearbook published by the korean library association. also, more survey recipients in the ‘other category’ were identified through internet search, directory search, and library ads. a total of 816 librarians participated in the survey. the breakdowns of participants based on the type of library they were working at are the following: mailto:ntorabi@uwo.ca� evidence based library and information practice 2011, 6.2 65 first survey: • 282 public librarians • 268 university librarians • 24 special librarians • 25 school librarians • 15 other librarians second survey: • 202 chief librarians at public libraries methods – a total of 2179 questionnaires were distributed twice in may 2009 via mail to different libraries. postage paid envelopes were provided. a phone call reminder was made to increase the response rate. 614 copies were returned. the total response rate for the survey was 28.18%. the highest response rate was from academic libraries with a total of 37.17% (table 2 in the article). six hundred and forty three copies of the questionnaire were sent out to chief librarians and the response rate was 31.42%. the sas statistical package was used for conducting statistical analysis of the data. the content areas covered in the two questionnaires are listed below in table 1. in order to investigate the career movement path of librarians, participants were asked to identify their current job position plus the first three positions before the current position. each position equals one phase in the librarians’ career path analysis. the jobs were broadly categorized into public libraries, university libraries, special libraries, schools and school libraries, database and content builders, library automation system developers, marc companies, agencies, internet portals, library supply companies, and others. main results – the data obtained from those items italicized in table 1 were used to compare factors influencing the career movement path of chief librarians in public libraries and librarians. the survey results on differences in gender show that most chief librarians (58.9%) were men compare to women (41.1%), while the ratio of women to men librarians is 2:5. therefore, it is more likely that male librarians decide to become chief librarians. analyzing the age of respondents indicates that it takes 20 years of library experience before one becomes a chief librarian. survey results on differences in the final academic degree between librarians and chief librarians show that more librarians (96.7%) held higher graduate degree compare to chief librarians (87.6%) (results calculated form table 7 p. 333). likewise, there are more librarians who studied library and information science (83.3%) compare to chief librarians (55.9%) (table 8). comparing the type of certificates held by two groups, it is evident that 26.7% of chief librarians do not have any library related certificates compare to 5.9% of librarians. the survey results suggest that both librarians and chief librarians make effort to increase their knowledge and skills. while librarians spend more time to improve operational aptitude and personal management, chief librarians identify themselves as hard working individuals. the study conducted a simple analysis on factors influencing the career movement of librarians and chief librarians. those factors differ greatly between librarians and chief librarians. while 25.5% of librarians rank compensation and working conditions as the most important factors, 19.2% of chief librarians report organizational culture as the main motivating factor. based on the percentage of initial career selected in each job category, the most frequent career paths taken by librarians and chief librarians were identified. a total of 92.9% of public librarians reported that their whole career path was indeed in public libraries. in addition, the average length of service by career movement phase was identified to get shorter from the initial to fourth career for librarians while they intent to stay at their current position the longest (157.8 months). on the other hand, chief librarians have one main career in which they have stayed the longest (116 months). this main career is the position immediately prior to the current job position. the main results of this article are summarized in table 21, based on numbers presented in the tables 12 to 20 in the article. evidence based library and information practice 2011, 6.2 66 the lowest and highest turnovers were among public librarians (7.1%) and library automation system developer and information professionals at marc companies (100%). conversely, the retention rates for all other jobs were also determined. 6.2% of school librarians returned to school libraries after working in other types of libraries. this is the highest rate for returning to the previous job. the proximity between different types of careers was analyzed based on combining the numbers of instances of career movement from the initial to the forth career. the highest rate of proximity is always the movement between the same types of career. the two highest (not same career type) career movements are university library and public library (2.65%) and university library and special library (2.32%). table 1 content areas covered in the two questionnaires for librarians for chief librarians personal background gender gender age age final academic degree final academic degree field of study field of study ownership of certificate of qualification ownership of certificate of qualification current job position governing institution career movement / career development age at the time of the initial career age at the time of the initial career career movement path career movement path organizational and environmental factors influencing career movement factors influencing career movement personal factors influencing career movement number of times changing department number of times changing department before becoming a chief librarian efforts made to move careers efforts made to become a chief librarian initial job time of becoming a chief librarian career movement path after the current job position educational opportunities after becoming a chief librarian evidence based library and information practice 2011, 6.2 67 conclusion – it is more likely that male librarians will take on leadership responsibilities in public libraries. usually, it takes 20 years of library experience before one becomes chief librarian. more librarians hold higher graduate degrees compared to chief librarians. this study also analyzes factors influencing the career movement of librarians and chief librarians. those factors differ greatly between librarians and chief librarians. the lowest retention rate was 0% for library automation system developers and information professionals at marc companies, whereas the highest retention rate was for public librarians followed by university libraries. the highest rate for return-to-first-job belongs to school libraries. it is noteworthy to mention that chief librarians have one main career in which they have stayed the longest. this main career is the position immediately before to the current job position. it usually occurs in the midcareer phase and it is when the individual has spent more time developing their skills and expertise and has been getting ready for their leadership role. commentary this article is a great start for a topic that has been neglected. it provides new information regarding the most popular career paths taken by librarians in korea. also, understanding the retention and turnover rates of different library positions can be useful for managers and human resources, and it provides ideas for professional development. glynn’s eblip critical appraisal checklist (2006) was used as the critical appraisal tool. using this tool, several significant flaws call into question the results and applicability of this work. the main problem is the lack of adequate statistical analysis. the methodology is based on descriptive statistics while the conclusions are presented in the form of inferences. results are presented in simple percentages rather than investigating standard deviation, p-values, or confidence intervals to identify any correlation or significant difference between variables and groups. sample size might be insufficient for some groups in this study. only 24 special librarians and 25 school librarians participated in the survey. also, selection bias might be the main source of error for the first section of the article. the data obtained for chief librarians in public libraries was compared with librarians in other types of libraries. one should assume that those who choose to work as public librarians are more likely to become a chief librarian in a public library, as this assumption can be supported by 92.9% of cases that have selected public library as initial and final job option (p. 339). therefore, it is more meaningful to compare the career path and career movement of chief librarians in public libraries with public librarians only. it is indicated that the questionnaires were written based on research papers on career development (p. 334). unfortunately, the author does not provide further information about those research papers. there is no report on whether a pilot study was conducted to evaluate alpha-reliability and validity of the instrument. in addition, not all factors have been considered (e.g., all academic degree held by the participants, the positions of authority held in previous careers, or job market condition). also, there is no rationale presented for those factors that have been selected for the study. throughout the article there is no mention of any missing data. missing data and the approach that is taken by the researcher to handle the missing data can greatly compromise the results of a study. the author claims that because 26.7% of chief librarians do not have any library related certificates compared to 5.9% of librarians, it is more likely that people without a degree in library and information science become chief librarians compared to librarians (p. 335). there is an indirect relationship between the likelihood of becoming a chief librarian and the lack of a degree in library and information science. one can argue that they held several other degrees in management, business, or evidence based library and information practice 2011, 6.2 68 even the age of the individual could be important but not accounted for in the questionnaires or analysis. the author shows that gender differences might be a factor that causes male librarians to assume leadership responsibilities in public libraries. bello (1996) has conducted a more indepth analysis of classical factors that influence choosing librarianship as a profession. in bello’s study, gender difference was considered as a confounding variable. also, it is known that “opportunities for career change begin decreasing with increasing age” (johnson, 2002, p. 623). this means that age should be considered as another confounding variable in the analysis. while career mobility path has been investigated throughout the article, the career movement factors have not been investigated adequately in the article. the factors influencing career movement and the effort that is put into career movement vary greatly between librarians and chief librarians. a more adequate statistical analysis is required to further investigate whether the difference between variables are statistically significant. references bello, musa a. (1996). choosing a career: librarian? librarian career development, 4(4), 15–19. glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. johnson, t. j. (2002). making it to the major leagues: career movement between library and archival professions and from small college to large university libraries. library trends, 50(4), 614–630. / evidence based library and information practice evidence summary   content analysis of reference transactions provides guidance regarding staffing of library service points   a review of: bishop, b. w., & bartlett, j. a. (2013). where do we go from here? informing academic library staffing through reference transaction analysis. college & research libraries, 74(5), 489-500. retrieved from http://crl.acrl.org/content/early/2012/05/07/crl-365.full.pdf   reviewed by: annie m. hughes reference librarian wilson dental library university of southern california los angeles, ca, united states of america email: amhughes@usc.edu    received: 22 feb. 2014   accepted: 25 apr. 2014      2014 hughes. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to identify the quantity of location-based and subject-based questions and determine the locations where those questions are asked in order to inform decision-making regarding optimal placing of staff.   design – content analysis of location-based and subject-based reference transactions or transcripts collected using libstats at 15 face-to-face (f2f) service points and via virtual services.   setting – virtual and f2f service points at university of kentucky (uk) campus libraries.   subjects – 1,852 location-based and subject-based reference transactions gathered via a systematic sample of every 70th transaction out of 129,572 transactions collected.   methods – using libstats, the researchers collected data on location-based and subject-based questions at all service points at uk libraries between 2008 and 2011. the researchers eliminated transcripts that did not include complete data or questions with fields left blank. if all question fields were properly completed, identification and coding of location-based or subject-based questions took place.   usable transcripts included 1,333 questions that contained sufficient data. for this particular content analysis only the question type, reference mode, and location of question were utilized from the data collected. unusable transactions were removed prior to content analysis, and reliability testing was conducted to determine interrater and intrarater reliability. interrater reliability was high (krippendorff’s alpha = .87%) and intrarater reliability was acceptable (cohen's kappa = .880).   main results – from the usable transcripts, 83.7% contained location-based questions and 16.3% were subject-based, and a little over 80% of location-based questions and 77.2% of subject-based questions were asked face-to-face (f2f). of the location-based questions, 11.5% were directional questions and many of these questions were related to finding places inside the libraries. “attribute of location” questions related to library services and resources, such as finding an item, printing, circulation, desk supplies, and computer problems, made up 72.8% of total question transactions. researchers found that subject-based questions were difficult to categorize and noted that other methods would be needed to analyze the content of these questions. professional librarians and library staff are better equipped to answer these questions, and the location where the question asked is irrelevant. the researchers addressed the issue of where questions were asked by recording the reference mode (chat, e-mail, phone, or f2f) and location service point at uk libraries. overall, 79% of questions were asked f2f, rather than via chat or e-mail. researchers think that this is due to a lack of marketing efforts regarding those services, noting that most questions were asked in the system’s large main library, which also receives the most subject-based questions.   conclusion – this study can inform the uk libraries system as to where their resources are most needed and allow for more strategic decision-making regarding staffing. the study could also prompt development of a mobile application to answer location-based questions, though more investigation is needed before moving forward with development of a mobile app. due to the findings of this study, uk libraries will deploy their professional library staff to locations where subject-based questions were most frequently asked. because staffing of libraries is one of the “most expensive and valuable resources,” academic libraries can use this method to validate their current staffing strategies or justify the allocation of staff throughout their systems (p. 499).     commentary   while the researchers have published on this topic in other instances, it is surprising that they do not focus their literature review toward desk staffing and service point allocations in academic libraries. they do cite articles in their introduction related to their research questions, though the literature review is focused primarily on mobile technologies such as mobile applications, qr codes, and augmented reality. these topics have less to do with the research questions at hand, and so it is odd that they discuss mobile applications and augmented reality in their larger review.   the authors do not provide sufficient background as to why they are conducting the study, nor do they provide a description of their current staffing or service model. we know that the staff exist at various service points within the library, but more information about their location, the setup of the service points (e.g., whether physical service points have a separate circulation desk/reference desk or a reference office) would be helpful.   the authors of this study clearly state the limitations of their data collection. collecting complete data at busy service points proved to be a challenge when using libstats. the researchers put forth that simplifying data collection and increasing training on systems used for collection would be a future approach. the uk libraries system also plans to implement the libanswers product by springshare, which will allow for recording of f2f transactions, telephone, e-mail, text message and chat transactions and which also provides a faq section that allows patrons to instantly access location-based and subject-based answers. the researchers plan to replicate this study using data gathered via this new system.   the researchers report that a mobile application could help to answer location-based questions, such as finding physical items, which would allow for staff to stay at their service points instead of abandoning their stations to aid a patron. the researchers do indicate that without further study they are speculating regarding the use of mobile applications.   one major outcome of this study is that 80% of questions asked at desk or virtual service points were location-based questions, not subject-based or reference questions. the implication here is that professional librarians or reference librarians are using their skills to answer directional questions rather than focusing on the more complex reference questions for which they were hired. in this case, the use of a tiered reference service whereby staff or library student workers triage questions at key services points and then pass on more complex questions to a reference librarian would be ideal.   overall, the article provides evidence that location-based questions are asked most frequently at various service points throughout the library and that subject-based questions, while asked less often, require a skilled professional to answer them. this type of study can inform library administration about how to go forward with staffing allocations as well as introduce technologies that provide anytime, anywhere service to alleviate the volume of location-based questions.   evidence based library and information practice evidence based library and information practice 2011, 6.2 19 evidence based library and information practice article a survey of primary school libraries to determine the availability and adequacy of services for universal basic education (ube) in oyo state, nigeria belau olatunde gbadamosi college librarian emmanuel alayande college of education oyo, nigeria email: tundeseyi58@yahoo.com received: 01 dec. 2009 accepted: 02 june 2011 2011 gbadamosi. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective -as a first step in gathering evidence, this study surveyed school libraries and examined the services those libraries provide in relation to the universal basic education (ube) program at a primary level. the purpose of this paper is to explore these ube factors in relation to school libraries in oyo state, nigeria. methods -a questionnaire was sent to the 33 schools which have a library and information centre staffed by a teacher-librarian, in order to examine library services and the views of those providing them. the questionnaire covers library space, categories of library resources, student activities during library visits, mobile library services, and the teacher-librarians’ perception of the adequacy of the collection and the challenges in the use of library. results -questionnaires were returned by 30 teacher-librarians out of 33, a response rate of 91%. although pupils across each local government area have the chance to visit school libraries, the system of one short visit per week is inadequate for developing a universal interest in reading and in study skills. staff responsible for the libraries consider many of the resources to be inadequate or outdated and stock is depleted as pupils borrow books, leaving fewer for those who follow. conclusions -as a first step to implementing evidence based practice, this paper describes the primary school library system in nigeria and provides evidence on how it operates in oyo state. teacher-librarians who staff the libraries confirm the mailto:tundeseyi58@yahoo.com� evidence based library and information practice 2011, 6.2 20 inadequacies in the system. they perceive collections to be inadequate and confirm that some pupils in the state have only limited access to library resources. respondents also believe that they need further training, particularly in computer and information literacy. introduction in nigeria, successive governments have recognized education as a vehicle for national development and as a powerful instrument for social change. however, introducing basic education at all levels across nigeria has been problematic. over the years, attempts have been made to make education universally free for all children of school age. the universal primary education (upe) program was launched in september, 1976. at the state level, bendel, lagos, ogun, ondo and oyo proclaimed free universal education at the primary, secondary and tertiary levels in 1980, but the upe program was stopped with the takeover of the government by the military in december 1983. according to the universal basic education commission (2007), there was a general decay in all aspects and levels of the educational system. teachers were poorly trained and poorly motivated. the funding for education was low since that sector was relegated to the background. it was not until may 1999 that the nigerian federal government decided to provide universal basic education (ube) to all children of school age up to junior secondary level. almost at the same time, the governments of lagos, oyo, ogun, osun, ondo and ekiti states, then controlled by different parties outside the federal government, launched free universal basic education for all children of school age up to the secondary education level. ube is a policy aimed at reforming basic education in nigeria, and has been sponsored by the world bank, the federal government of nigeria, state and local governments, and intends to cover all neglected groups such as poor rural children, street children, nomads, migrants, native and indigenous peoples, minorities and the physically disabled. the ube has six basic components with a formal educational system being just one of them. other components relate to early childhood literacy, life skills for adults, services to the nomadic population, and non-formal education or apprenticeship training for youths outside the formal education system (federal republic of nigeria, 2000). funding for library development is dependent on the project priorities of the individual participating states. there is not a fixed percentage of funding allocated to library development. while some states have library development in their project priority list, others may not (unagha, 2008). oyo state took a step to address the provision of library and information centres in primary schools in 2005. government efforts to promote education and literacy in nigeria have not translated to high quality school graduates or high literacy rates amongst the population. this is due to factors which include under-funding, poor school infrastructures, poorly trained and poorly remunerated teachers and poorly equipped libraries (unagha, 2008). school libraries and universal basic education the most successful feature of the upe or ube, according to gbadamosi (2006) is the increase in the population of children wanting to enroll in education at all levels in nigeria. school libraries and library services are essential to the successful implementation of ube. ifla (2006) states that “the school library provides information and ideas that are fundamental to functioning successfully in today's information and knowledge-based society. the school library equips students with life-long learning skills and develops the imagination, enabling them to live as responsible citizens.” as an educational support service, the school library adds value evidence based library and information practice 2011, 6.2 21 to classroom instruction by providing additional resources and information that expands pupils’ reading horizons, upgrades the intellectual scope and depth of teachers, and provides conducive environments for pupils that are not found elsewhere in the school. according to unagha (2008) the school library helps children learn to enjoy reading, use library resources as a lifelong habit, and gives them a chance to evaluate and use information. the importance of libraries in the successful implementation of ube was acknowledged in the national policy on education in which libraries are described as one of the most important educational services. the policy also suggested that training should be provided for librarians and library assistants (federal republic of nigeria, 2004). nevertheless, federal, state, and local governments as well as private owners establish schools without making provision for school library services. it is essential to appraise and evaluate the impact that school libraries have on the academic success of students. however, because of the limited provision of libraries in primary schools in nigeria, it was considered useful, as a first step in gathering evidence, to survey school libraries and examine the services they provide in relation to the ube program at a primary level. the purpose of this paper is to explore these factors in relation to school libraries in oyo state, nigeria. literature review the place of school libraries in universal basic education (ube) worldwide cannot be overemphasized. traveline (1997) stresses that the central importance of the library and its learning resources has not diminished. these learning resources facilitate the acquisition of the physical and intellectual skills necessary to assist the individual to develop literacy for lifelong education. the education for all (efa) framework of action for meeting basic education needs recognize libraries as invaluable information resources which must work in partnership with school and community workers. lance, welborn, & hamilton-pennell (1993) report that the school library is one of the few factors whose contributions to academic achievement has been documented empirically, and is a contribution that cannot be dismissed by other powerful influences on student performance. lonsdale (2003) agrees, claiming that existing research in australian schools shows that school libraries can have positive impacts on student achievement whether measured in terms of reading scores, literacy or learning in general. similarly, the ontario library association showed that there is a positive relationship between professionally-staffed school libraries and student achievement and suggests the importance of teacher-librarians in fostering engagement in reading, a factor that is instrumental in fostering literacy and lifelong learning for students (“school libraries”, 2006). in a related study published by scholastic library publishing, (school libraries work!, 2008) the importance of school libraries to students’ education is clearly demonstrated. the study finds that whether student achievement is measured by standardized reading achievement tests or by global assessments of learning, a well-stocked library overseen by a certified librarian has a positive impact on student achievement regardless of the socio-economic or educational level of the community. the role of librarians and teacher-librarians in dispensing library services and impacting positively on students’ achievement has been researched extensively. lance (2001) found that the level of development of the library media program was a predictor of student performance and that staffing levels correlated with test scores. he also points out that there was higher performance at all educational levels in schools with librarians than in schools without librarians. it is important to bear in mind that the value and importance of the library and its impact evidence based library and information practice 2011, 6.2 22 on the achievement of students as reported in developed countries may not be the same in developing countries like nigeria. aguolu and aguolu (2002) note that african students enter universities without ever using any library either in the primary or secondary schools. libraries are unavailable in most schools and where they are, use is not necessarily promoted to students. the situation in oyo state is no different which is why adediran (2002) argues that every primary school in oyo state should have a library. this paper, therefore, surveys school libraries in relation to the ube program at primary level in oyo state nigeria to explore the experience and perceptions of those providing the school library service. primary schools and provision of library services within oyo state in oyo state, there are 2,116 public primary schools (as of july 2010). of these, only 33 have functional libraries. across the state there are over one million pupils and more than 28,000 teachers. of these teachers, 256 were selected and trained as teacher-librarians and 60 of received further training. one school in each of the 33 local government areas (lga) has a library and information centre, staffed by a teacher-librarian. the education trust fund (etf) in nigeria has allocated n33 million naira (nigeria currency) to purchase books, and each (lga) is alloted 296 volumes of books. public primary schools without library and information resource centres are expected to visit, on a rotational basis, the school with a library in their area. in each of the schools within an lga, only one or two classes, particularly primary four or five stream, are enlisted to visit schools where library and information centres are located. however, the implementation of this policy is not feasible in rural areas where schools are scattered and the schools with libraries are based some distance away from the schools which they serve. furthermore, where there is no transport, pupils must walk to the school where the library is located. in oyo state, mobile libraries are intended to benefit those schools that are unable to visit the library in their area. under this arrangement, the books are boxed and boxes are taken to the rural schools and then moved to other schools at three weeks intervals (adediran, 2002) objectives this study seeks to establish: 1. which library resources and services are available at the primary schools under the universal basic education program in oyo state. 2. the views of personnel in primary school libraries in oyo state regarding the adequacy of the resources and the obstacles impeding effective use of library resources. methods a questionnaire (appendix a) was sent to the 33 schools that have a library and information centre staffed by a teacher-librarian. before the questionnaire was distributed it was checked by a senior colleague for content validity. the questionnaire covers library space, categories of library resources, student activities during library visits, mobile library services, and the teacher-librarians perceptions of the adequacy of the collection and the challenges in the use of library. results questionnaires were returned by 30 teacherlibrarians out of 33, a response rate of 91%. the adequacy of instructional and book resources was studied by assessing the availability and currency of 13 types of resources. each of the instructional materials was assessed using the 4-likert scale rating as shown in table 1 below. ha = highly adequate; a = adequate; fa = fairly adequate; na = not adequate. evidence based library and information practice 2011, 6.2 23 table 1 assessing the adequacy of available instructional/book resources (n=30) instructional resource available response frequency frequency % 1 children’s encyclopedia ha a fa na 22 08 73.33 26.6 2 dictionaries ha a fa na 06 24 20 80 3 historical/cultural/geographical charts ha a fa na 30 100 4 science charts ha a fa na 07 23 23.33 76.66 5 computer sets ha a fa na 04 26 13.33 86.66 book resources available 6 mathematics ha a fa na 10 20 33.33 66.66 7 english language ha a fa na 15 15 50 50 8 elementary science ha a fa na 16 10 04 53.33 33.33 13.33 9 verbal & qualitative reasoning ha a fa na 18 10 02 60 33.33 06.66 10 physical & health education ha a fa na 30 100 11 yoruba language ha a fa na 25 04 01 83.33 13.34 03.33 12 literature/story books ha a fa na 28 02 93.33 06.66 13 newspapers/magazines ha a fa na 15 15 50 50 evidence based library and information practice 2011, 6.2 24 table 1 shows the perceptions of the adequacy of the resources in relation to the number of students using them during their visits. table 1 shows that instructional and book resources that are rated highly adequate include children’s encyclopedias, verbal and qualitative reasoning, yoruba language and story books. these are rated foremost and considered highly adequate in terms of the total collection in relation to the number of pupils using them at a time. table 2 provides information on the operation of the library services including library mobile services. from table 2 above, it can be seen that although pupils have free access to library resources when they visit the library, they are not formally registered or given registration cards. the majority were able to borrow books for one week. school teachers can borrow on behalf of the schools. this implies that the borrowing facility by either the pupils or teachers will deplete the books further as each school visits the centre. in consequence, there are fewer books available when students visit the library. since library and information centres service many schools, pupils and teachers use of the borrowing facility will render the book collections inadequate. mobile library services are not well established, occurring in only 3 (10%) of the total number of library and information centres in the state. teacher-librarians who have transport such as motor-bikes are assigned to act as supervisors for schools in rural areas which have no access to library services. table 3 illustrates the student-activities that take place during library visits. all respondents agree that schools visit library and information centres on an average of once per week, however those running the library believe this is inadequate. this is because each school selects one or two streams of primary four or five or both as designated classes to visit the library and information centres. it is only this stream that visits the centre; other classes do not access the library at all. all libraries provide reading and comprehension in local yoruba and english language but are table 2 types of library services and mobile library services (n =30) services yes % no % 1 pupils are registered and issued with registration cards 30 100 2 pupils have free access to books in the library 30 100 3 pupils can borrow books 30 100 4 loan policy: one book for one week two books for two weeks four books for two weeks 28 93.33 02 06.66 5 teacher can borrow on behalf of their school 28 93.33 02 6.66 6 mobile library service is available for schools in rural areas 03 10 27 90 7 teacher-librarians that has mean of transportation is appointed as mobile library teacher supervisors 29 96.66 01 33.33 evidence based library and information practice 2011, 6.2 25 not used for reading personal notes or carrying out homework. table 4 provides more detail regarding teacher-librarian perceptions of the adequacy of the library services. these findings suggest that respondents are happy with the amount of space in the library but not the furniture within it. in addition the majority suggest that book resources are in short supply and obsolete. furthermore, respondents believe that reference books are insufficient. twentyeight (93.32%) of the respondents agree that a change needs to be made to the ledger system of loaning books to users, where the controlling records of the borrowed books are kept in a book (or ledger) form. the books are not catalogued or organized according to any of the known classification schemes and only an accession number is used to identify each library book, making it difficult to track the books which are loaned. table 4 shows that 28 (93.3%) of the respondents strongly agree that library personnel, particularly teacher-librarians and library assistants, need training and retraining while two respondents (6.6%) disagree. the actual use of the library resources is very low and all respondents agreed that is because a library is not located within each of the schools. instead, some classes visit a local library on a weekly basis. the table also shows that the majority of respondents (83.33%) strongly agree that there is the need to introduce information and communication technology (internet services) to complement the conventional library services in school libraries. table 5 shows respondents’ perceptions of the challenges relating to effective use of the library. all respondents agree that one library resource centre is inadequate to effectively serve all the schools in a lga. the table shows that 26 respondents (86.6%) strongly agree that the movement of the pupils and by extension, the security of the pupils is hindered because the library resource centre is located far away. the security of the pupils and teacher-librarians alike is endangered because in some cases pupils and staff have to cross busy roads. of the total number of respondents, 20 (66.6%) strongly agree and 8 (26.6%) agree that books and other current sources should be procured to strengthen the library collections (suggesting that the respondents perceive book collections as inadequate). table 5 also shows that respondents agree that there is need for the appointment of qualified librarians and library officers who are computer and information literate. table 3 student activities during library visits (n = 30) activity yes % no % 1 school visits library for one period of 45 minutes per week 30 100 2 do you consider one period of 45 minutes per week adequate 30 100 3 reading of personal notes, and carrying out home assignments 30 100 4 reading and comprehension exercises in local yoruba and english language texts under the guidance of a teacherlibrarian 30 100 evidence based library and information practice 2011, 6.2 26 table 4 assessing the adequacy of library services and mobile library services (n = 30) response frequency relative frequency % 1 library space is adequate sa a d sd 22 08 73.33 26.66 2 library seating furniture is adequate sa a d sd 10 10 10 33.33 33.33 33.33 3 many of the books are obsolete sa a d sd 30 100 4 books are in short supply sa a d sd 21 09 70 30 5 numbers of school visits are very low sa a d sd 30 100 6 reference books are sufficient sa a d sd 10 20 33.33 66.66 7 organization of books follow specific scheme sa a d sd 12 18 40 60 8 accession number is used to arrange the library books sa a d sd 29 01 96.66 3.33 9 personnel need training and retraining sa a d sd 28 02 93.33 06.66 10 ledger system of loan needs to be changed sa a d sd 08 20 02 26.66 66.66 06.66 11 fact that library is not located within each of the schools hinders the effective use of the resources sa a d sd 30 100 12 effective mobile library service is predicated upon the availability of mobile vans and personnel sa a d sd 29 01 96.66 03.33 13 need to introduce internet service to complement conventional library services in schools sa a d sd 25 05 83.33 16.66 key: sa= strongly agree; a= agree; d= disagree; sd= strongly disagree evidence based library and information practice 2011, 6.2 27 discussion the questionnaire identifies perceptions of a range of inadequacies in relation to school libraries in oyo state. the data shows that although pupils across each local government area have the chance to visit school libraries, the system of one short visit per week is inadequate for developing a universal interest in reading and in study skills. further, those who are able to visit the library are discouraged by long walks or busy traffic. this means that libraries and librarians are not well integrated into the teaching and learning process. mobile libraries are in their infancy and this means that schools in rural areas do not have access to library services. table 5 perceptions on the challenges relating to effective use of the library (n = 30) response frequency frequency % 1 one library resource centre cannot effectively service all the schools in a local government area sa a d sd 30 100 2 because the library resource centre is located far away, movement of pupils to the centre hinders the use sa a d sd 26 04 86.66 13.33 3 in some local government areas where pupils have to cross major roads, road crossing poses serious danger to the pupils sa a d sd 28 02 93.33 6.66 4 there is the need to introduce ict in primary school library resource centre sa a d sd 30 100 5 there is the need to strengthen collection of books and current sources in the library sa a d sd 20 08 02 66.66 26.66 06.6 6 need for the appointment of qualified librarians and library officers sa a d sd 28 02 93.33 06.66 7 need for training and retraining of teacherlibrarians to be computer and information literate sa a d sd 28 02 93.33 06.06 key: sa= strongly agree; a= agree; d= disagree; sd= strongly disagree evidence based library and information practice 2011, 6.2 28 staff members responsible for the libraries consider many of the resources to be inadequate or outdated and stock is depleted as pupils borrow books, leaving fewer for those who follow. traditional classification schemes are not used to organize books and inadequate ledger systems are used as a means of tracking the movement of books in the library. effective delivery of library services for ube is contingent on resources such as books, personnel, space and new information and communication technologies (ict). for school libraries to provide environments that enable teaching and learning, they must have qualified staff. this survey shows that qualified librarians and library officers would be welcomed for staffing the school library and there is a need for training and retraining of teacher-librarians to be computer and information literate. introducing ict in the libraries is also desired by those providing library services. usoro and usanga (2007) argue that children must be encouraged to use resources effectively for lifelong education. primary school children must develop a reading habit that will encourage literacy. one way to overcome the neglect of school libraries in nigeria and oyo state may be to include them in the basic education program. obanya (2002) maintains this has the effect of encouraging extensive reading development in the middle years, through systematic exposure to a variety of carefully graded reading materials. conclusion as a first step to implementing evidence-based practice, this paper describes the primary school library system in nigeria and provides evidence on how it operates in oyo state. teacher-librarians who staff the libraries confirm the inadequacies in the system. they perceive the collections to be inadequate and confirm that some pupils in the state have only limited access to library resources. respondents (teacher-librarians) also believe that they need further training, particularly in computer and information literacy. acknowledgement the author would like to acknowledge the helpful comments of hope lehman and carol waseleski who provided writing assistance. references adediran, s.a. (2002, may 15). oyo state moves to provide libraries in primary schools. this day. retrieved from http://www.thisdayonline.com/archiv e/2002/05/15 aguolu, c.c., & aguolu, i.e. (2002). libraries and information management in nigeria. maiduguri, nigeria: ed-linform services. federal republic of nigeria. (2004). national policy on education (4th ed.). lagos, nigeria: nerdc press. federal republic of nigeria (2000). implementation guidelines for the ube programme. abuja, nigeria: federal ministry of education. gbadamosi, b.o. (2006). demographic information dissemination, utilization and educational policy making in south western nigeria. (unpublished doctoral dissertation). university of ibadan, ibadan, nigeria. international federation of library associations and institutions, school libraries and resource centres section. (2006). ifla/unesco school library manifesto: the school library in teaching and learning for all. retrieved from http://archive.ifla.org/vii/s11/pubs/ma nifest.htm evidence based library and information practice 2011, 6.2 29 lance, k.c. (2001) proof of the power: recent research on the impact of school library media programs on the academic achievement of u.s. public school students. retrieved from eric database (ed456861) lance, k.c., welborn, l., & hamilton-pennell, c. (1993). the impact of school library media centers on academic achievement. denver, co: colorado department of education. retrieved from eric database (ed353989). lonsdale, m. (2003). impact of school libraries on student achievement: a review of the research. retrieved from eric database (ed482253) obanya, p. (2002). revitalizing education in africa. nigeria: stirling-horden. school libraries and student achievement in ontario. (2006). ontario library association. retrieved from http://www.accessola.com/data/6/rec_ docs/137_eqao_pfe_study_2006.pdf school libraries work! (2008). scholastic library publishing. retrieved from http://www.fundourfuturewashington .org/resources/slw3_2008.pdf traveline, k.a. (1997). explore the playground of books: tips for the parents of beginning readers. newark, de: international reading association. unagha, a.o. (2008). implementing universal basic education (ube) through the strategic provision of school library services. library philosophy and practice. retrieved from http://unllib.unl.edu/lpp/amanze.htm universal basic education commission. (2007). universal basic education (ube) programme national assessment. abuja; ubec. usoro, i.m.p., & usanga, e.e. (2007). the role of nigerian primary school libraries in literacy and lifelong learning. library philosophy and practice. retrieved from http://www.webpages.uidaho.edu/~m bolin/ima-eboro.htm evidence based library and information practice 2011, 6.2 30 appendix a questionnaire survey of school libraries in primary schools in oyo state universal basic education (ube) introduction please complete this questionnaire as truthfully as to assist in bringing about improvements to library services in primary schools in oyo state, nigeria. your response is essentially for research purpose and shall be made as confidential as possible. dr. b. o. gbadamosi emmanuel alayande college of education, oyo, oyo state, nigeria. 1. i. name of the school personal/institutional data ii. name of the teacher-librarian iii. local government area iv. your present status – classroom teacher, teacher-librarian, assistant teacher-librarian, teacher-librarian supervisor 2. each school visits the designated library (mark as appropriate) provision of library space yes no once in a week twice in a week every day of the week 3. library resources available highly adequate fairly adequate adequate not adequate 1. children encyclopedia and their volumes 2. dictionaries (please give numbers) 3. historical/cultural/geographical charts (give the numbers) 4. science chart (give the numbers) 5. available computers (give the numbers) 6. mathematics (please give the numbers) 7. english language 8. elementary science 9. verbal & qualitative reasoning (give the numbers) 10. physical & health education (please assessing the adequacy and recency of available instructional/book resources evidence based library and information practice 2011, 6.2 31 give the numbers) 11. yoruba language 12. literature/story book (give the numbers) 13. newspaper/magazine 4. student activities during library visit yes no 1. school visits library for one period of 45 minutes per week 2. do you consider one period of 45 minutes per week adequate. 3. identify student activities when they visit the library (mark as many as relevant) a. private reading of textbooks b. private reading of english/yoruba literatures c. reading of english/yoruba literatures under the guidance of a teacher librarian d. reading of personal lesson notes 4. student doing their home assignments 5. library services available mark the various services provided for the students otherwise leave blank yes no 1. pupils are registered and issue registration cards 2. pupils have free access to books in the library 3. pupils can borrow books 4. identify number of books a student can borrow and loan period a. one book for one week b. two books for two weeks c. four books for two weeks 5. loan service to teachers a. one service to teachers b. two books for one week c. two books for two weeks d. four books for one week e. teacher can borrow on behalf of the school. 6. mobile library services mark up as many relevant options as possible yes no a. mobile library is available for schools in rural areas b. mobile librarian are appointed among mobile teacher supervisor evidence based library and information practice 2011, 6.2 32 7. assessing the effectiveness of library services and mobile library services s/n strongly agree agree disagree strongly disagree i. library space is adequate ii. library sitting furnitures are adequate iii. many of the textbooks are obsolete iv. books are in short supply v. school visits are very low vi. reference books are sufficient vii. organization of books follow specific scheme viii. accession number is used to arrange books ix. personnel need training and retraining x. leger system of borrow need to be changed xi. because the library is not located within each of the schools affect the effective use of the resources 8. access the problem confronting and solutions being profer to the effective use of the library s/n strongly agree agree disagree strongly disagree i. one library resource centre cannot effectively service all schools in a local government ii. because library resource centre is located far away, movement of pupils to the centre hinders the use iii. because library resource centre is located far away, movement of pupils have to cross major road, crossing the road poses serious danger to the pupils iv. there is the need to introduce ice in primary library resource centre v. there is need to strengthen the books and current sources collections in the library vi. effective mobile library is predicated upon the availability of mobile vans vii. need to introduce mobile internet service even to the rural area evidence based library and information practice 2011, 6.2 33 viii. need for the appointment of trained librarians and library officers ix. need for training and retraining of teacher-librarians to be computer and internet compliance / evidence based library and information practice research article   digital literacy skills for family history research   jaci wilkinson head, discovery and user experience herman b. wells library indiana university bloomington, indiana, united states of america email: wilkinj@iu.edu   natalie bond government information librarian maureen and mike mansfield library university of montana missoula, montana, united states of america email: natalie.bond@mso.umt.edu   received: 29 oct. 2020                                                               accepted: 21 feb. 2021      2021 wilkinson and bond. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29624     abstract   objective – in this case study, an archivist and librarian teamed up to teach an introductory course on family history research for adult learners at their university’s lifelong learning centre. in response to students’ relative lack of digital skills, the instructors developed a new set of introductory skills that they believe are essential for genealogy research.   methods – authors conducted preand post-course surveys to determine student expectations and the extent to which the course met those expectations. authors coded one of these surveys.   results – course assessment and class activities exposed the need for a set of digital skills that go beyond a literacy framework to assist family history researchers. after analyzing key themes found in preand post-course assessment, authors developed a new tool for genealogy instructors titled introductory digital skills and practices in genealogy (idsg).   conclusion – archivist/librarian collaborations are an excellent way to cultivate needs-based teaching and outreach opportunities in our wider communities, particularly for adult learners. the introductory digital skills and practices in genealogy tool is meant to inspire and assist other library professionals who want to teach family history research, serving as a reminder to centre teaching tangible digital skills as a focal point of instruction.     introduction   genealogy, both as a hobby and industry, has seen a renaissance over the past two decades in the united states, most recently boosted by renewed interest in the potential for genealogy and genetic testing to help solve cold crime cases (greener, 2014; payne, 2020; rodriguez, 2014; sachs, 2019). in this case study, the authors—a librarian and an archivist—report on the implementation of an adult education class focusing on genealogy and digital literacy, and lessons learned through course assessment. they share foundational digital skills needed to conduct effective genealogy work, represented in a new document: introductory digital skills in genealogy (idsg; appendix a). this document has its genesis in students’ queries about “the basics” of digital research skills and how they relate to genealogy. idsg pulls components of digital literacy, archival intelligence models, and domain knowledge into a cluster of foundational digital skills needed for effective genealogical research.   neither author is a certified genealogist or a member of a genealogy organization. in approaching this course and subsequent paper, the authors derived their tools and perspectives from librarianship and archival praxis, rather than the practice of genealogy itself. the field of genealogy retains its own pedagogical approach and praxis, but a critical analysis of these methods would necessitate a more thorough examination, which lies outside the confines of this paper. according to the national genealogical society, “genealogy is often used to describe a line of descent, traced continuously from an ancestor, often also called a lineage. there is some expectation that a genealogy is a formal or scholarly study of ancestral family lines” (national genealogical society, n.d.). this paper thus addresses the pursuit of family research and refrains from turning a critical eye to methodology or praxis. the authors used the terms “genealogy” and “family history” interchangeably throughout their course as well as in this paper; any perceived differences in meaning have been deemed minute enough by the majority of genealogy practitioners as to be rendered moot.   literature review   the popularity of genealogy   given the popularity of genealogy and the utility of libraries and archives for free access to related materials, services, and resources, professionals working in these spaces have created resources to help serve the specific needs of users. in 1996, the american library association’s reference and adult services division’s (rasd’s) board of directors first published guidelines for instruction in genealogy at library schools that are still maintained today (rasd history section genealogy committee, 1996). two papers published in 2003 and 2014 studied the information-seeking behaviours of genealogists with the purpose of helping librarians and archivists understand the distinctive needs of this group of researchers whom they might encounter in reference transactions (duff & johnson, 2003; friday, 2014). more recently, multiple texts have been published to help libraries grow their corpuses of genealogy materials and expertise (schultz, 2018; smallwood & gubnitskaia, 2018). in genealogy and the librarian: perspectives on research, instruction, outreach and management, cheri daniels discusses the notion of genealogy literacy in her chapter titled “genealogy literacy: helping patrons build stable trees through information literacy standards,” and creates a definition by mapping each section of the information literacy standards (a precursor to the current association of college & research libraries (acrl) framework for information literacy for higher education) onto concepts of genealogy reference help (smallwood & gubnitskaia, 2018, pp. 176–179).   applicable literacies   digital literacy is widely recognized and implemented within the broader fields of education and policy, but its meaning remains somewhat nebulous, usually applied to scholarship and projects related to computer, information, and media literacy (nichols & stornaiuolo, 2019). while incorporating information literacy into library instruction and practice has become de rigueur, practitioners do not generally include technical skills, thus giving rise to the necessity of mapping information literacy concepts to computer (digital) skills.   the early to mid-2000s saw a shift away from prescriptive, skill-based competencies towards more descriptive narratives of how digital mediums are used to create knowledge in local communities (for example, boyd, 2014; gee, 2003; hull & katz, 2006). nichols and stornaiuolo (2019) argue that it “might mean reclaiming from the past an attention to the internal complexities of technical systems, and providing both descriptive accounts and prescriptive strategies that can illuminate and guide activities in these domains” (p. 20).   in 2013, the american library association assembled a task force to define digital literacy and make policy recommendations about the role of libraries in fostering digital literacy skills. they defined digital literacy as “the ability to use information and communication technologies to find, understand, evaluate, create, and communicate digital information, an ability that requires both cognitive and technical skills” (american library association, 2013). heuristics like the craap test are often used to teach users how to evaluate the validity of digital content (blakeslee, 2004); however, it has been argued that “most literacies are heavily domain-dependent, and based not on skills, but on a body of knowledge that comes from mindful immersion in a context” (caulfield, 2016). caulfield suggests that more concrete tools should be given to students when teaching them how to evaluate information resources.   archival and primary source literacies have been created by the archival community to identify and assess instruction in archives and special collections, most often in a higher education setting (e.g., carini, 2016). a key reason for developing this specialized literacy was a fear that younger demographics were less equipped to critically analyze and interpret non-electronic documents (carini, 2016). an early model identified three areas for archival researchers: archival intelligence, subject knowledge, and artifactual literacy (yakel & torres, 2003). carini (2016) fleshed out this model into a six-standard information literacy for archives and special collections with outcomes for each standard. the society of american archivists (saa) and acrl’s rare books and manuscripts section (rbms) jointly published their own primary source literacy guidelines in 2018 that includes 22 learning objectives spread across five categories: conceptualize; find & access; read, understand, & summarize; interpret, analyze, & evaluate; and use & incorporate (saa-acrl/rbms, 2018). like the acrl framework, carini and the saa-acrl/rbms frameworks are broad and do not follow any certain order of complexity or information-seeking journey.   information literacy tools and research often focus on formal education settings, yet adult information literacy has distinct indicators and its utility extends far beyond classrooms. information literacy has been at the heart of united nations’ initiatives and goals related to public health, employment, and civil participation for over a decade (unesco institute for statistics, 2018). the alexandria proclamation of 2005 stated that information literacy was essential not just for educational purposes but also lifelong personal, social, and occupational goals (participants in the high-level colloquium on information literacy and lifelong learning, 2005). the elements of information literacy defined by the united nations educational, scientific and cultural organization’s information for all programme in their 2008 report do not include technical skills, but they are implied through elements like “store and retrieve information” (catts & lau, 2008). information literacy allows for the creation of a full “knowledge chain” where citizens take in information, create knowledge, and use that knowledge to create or disseminate new information in their communities. research shows that adults learn in different ways than children and that they respond better to training that focuses on tools and resources that fill a targeted need, as opposed to the cumulative, complex educational models that are more common in formal education (stern & kaur, 2010). this research is rooted in the model of andragogy, which, as opposed to pedagogy, is the theory that adults learn better when a learning experience is tailored to their experiences and interests.   archivist and librarian classroom collaboration   while instructional collaboration between archivists and librarians is a naturally synergistic alliance, there is a dearth of literature surrounding such collaborations, particularly on the creation and teaching of whole courses. within literature focusing on archival outreach, there has been a call for an “integrative approach”: archivists finding and focusing on spaces where potential constituents already spend time and energy, and integrating instructional practices with established habits and interests (rettig, 2008), or the “archives 2.0” approach, which involves archives proactively attracting new users (theimer, 2011). in her review of crowdsourcing projects at the british library, ellis (2014) concludes that collaboration with the community helps “create a sense of pride and ownership in cultural and information institutions” (p. 4).   academic librarians regularly engage in outreach and instruction with a defined audience: that of the institution’s students and faculty. acrl (2011) guidelines state that libraries need to empower librarians to “collaborate with faculty and other academic professionals in planning, implementing, and assessing information literacy programming”. one instance of local practitioners facilitating a cross-institutional partnership saw a map librarian from the university of minnesota and a special collections librarian from the local county library offering free classes on resources about the history of neighborhoods in minneapolis (lawton & block lawton, 2009). the authors concluded that the “everyday people” who participated were given the tools to foster a deeper appreciation of the places they lived (lawton & block lawton, 2009).   aims   as a result of teaching a family history research course for adult learners, the authors developed the following research question: what tool(s) can facilitate the combined teaching of family history research with digital literacy? bringing together the viewpoint and expertise of a web/user experience librarian and an archivist, the resulting introductory digital skills and practices in genealogy (idsg) document is rooted in the experiences of teaching a full course on the subject, data collected in class activities and post-course evaluations, and related literature about digital and primary source literacies. the authors sought to create a body of tangible skills and practices for instructors of family history research to negate the need for expertise in literacy standards and learning outcomes. in describing the facilitation of this course and the idsg, the authors aim to inspire similar instructional collaborations that include teaching fundamental digital skills.   this work occupies a particular niche in the literature, combining aspects of adult information literacy, digital literacy, genealogy, librarian/archivist teaching collaboration, and digital preservation best practices; yet, its results are broadly applicable. genealogy will likely remain a popular hobby, librarians and archivists will continue to develop and implement closely related areas of expertise, and the digital realm in which genealogy research occurs will continue to evolve, requiring technological agility and acumen. the authors’ goal is to inspire other archivists and librarians to cultivate needs-based teaching opportunities in their wider communities and to draw on the idsg artifact as a starting point in family history instruction.   methods   despite the popularity of genealogy research, its key constituents—older adults—often lack confidence in their technological skills, and have trouble identifying factual information online (anderson & perrin, 2017; gottfried & grieco, 2018). recognizing this potential contradiction in skills and interest, the authors—an archivist and a web services librarian—came together to offer the first-ever family history course at the osher lifelong learning center at the university of montana (molli). given their combined expertise in local history, primary source research, and information/digital literacy, the authors titled the course “conducting family history research: digital literacy & research methods.” they predicted that a class on genealogical research might be popular in their local community of adult learners, and they were right: once listed, the course was at capacity (30 students) weeks before it began.   two bodies of data were used to inform the creation of idsg. the first data corpus consisted of a survey of students’ learning goals, taken from an activity conducted during the first class wherein students were asked about their goals for the course. instructors wrote these goals on a white board for general discussion and consensus building, with the aim of empowering the class through self-directed learning. the authors photographed the white board and later coded these answers based on their relevance to common skill-based values: discovery and navigation of online resources, subject-specific knowledge, and genealogy-specific skills.   though a fairly basic introductory classroom activity, this goal setting and subsequent mapping of skill-based values proved to be influential in the construction of the idsg, particularly in how students articulated both their learning goals and what they didn’t know. for example, although “digital literacy” was in the course title, less than a third of student learning goals were related to online research/resources. the majority of students sought instead to improve their domain knowledge and skills in historical research, such as gaining subjector area-specific knowledge around a particular person or place. a full list of learning goals collected on the first day of class is detailed in appendix b. besides responding to these collected learning goals, the instructors spent much of the course refining students’ expectations around genealogy research.   post-course evaluations, created by molli, provided valuable insight not only into what students learned but also what they didn’t learn, and how they felt about this disparity. out of 30 students, 22 (73%) submitted at least some answers to questions from these course evaluations. a list of questions asked in the evaluation is located in appendix c. data from both preand post-class surveys were compared side by side to identify and code skill-based learning themes. finally, reflection and analysis of the syllabus and post-course discussions between the authors also informed the development of the idsg.   findings and results   it only took two class sessions for the authors to hit upon the observation that inspired and redirected the remainder of the course, and this paper. in order to help facilitate students’ research, the authors had included a resource section on the course syllabus titled “beyond literacy: good research habits.” this document introduced an assortment of unofficial research and technical tips. while this list was meant to be informal, it sparked a lengthy discussion amongst the students that exposed a lack of foundational technology concepts—ones that the instructors had mistakenly assumed most students had already mastered. once the class dove into these “tech tips,” however, the questions flowed freely: what is a hard drive? what is “the cloud”? how do i hold down two buttons at once on my keyboard? from then on, every class session contained a 15-minute “tech tips” section, where the instructors shared one technology-based skill or tool and gave students time to practice. each tech tips section was informed by student input. at the end of the course, instructors refined and compiled these “tech tips” into an artifact that they then shared with students (found in appendix d). while three students indicated on course evaluations that they felt they had learned a good deal about conducting online research, five students noted that they still felt overwhelmed by either the online components of the class or the computers that they had to use for classwork (the class was held in a computer classroom to accommodate some students who didn’t have a laptop or tablet to bring to class). the idsg was created as a strategic tool to prevent students from being overwhelmed in similar future courses.   survey results   seventy-three percent of students (22 of 30) responded to the course evaluation. in comparing these responses to students’ original learning goals, the authors were not surprised to see that, while students generally expressed satisfaction with the course itself, they remained frustrated with their perceived ability to access and utilize digital resources for genealogical research. one stated, “i was a very beginning novice in research and computer skills, so could have used more basic information/skills.” another wrote that “the first classes were pretty esoteric and over my head. i think more research facts and how-tos would be more helpful.” students’ responses indicated that many were looking for a true introductory course to genealogy, which ultimately was not the stated directive of the course: molli specifically asked the instructors to provide something more nuanced than an introductory course on genealogy, as the learning centre had provided a number of those in the past. as such, the authors proceeded to develop a class plan and syllabus specifically tailored to digital literacy, predicated on the incorrect assumption that students would arrive to class already equipped with basic digital literacy skills. while the negative comments were disheartening, the authors are cognizant that they primarily stemmed from frustrations around technology: the inability to keep up with in-class activities on computers, delayed or rushed class instruction due to the instructors spending class time troubleshooting students’ computer questions, and confusion around various topics. as one student noted, “explanations, for me anyway, could have been adjusted a little more to a ‘not very digitally literate person’ like myself! certain things escaped me!” this feedback ultimately proved, however, to both validate instructors’ observations around digital literacy throughout the course and inform the development of idsg. many students did express satisfaction with the complexity and nuance of class discussions, readings, and information resources, and the authors felt that one of the most successful aspects of the course lay in broadening students’ horizons within the vast realm of genealogical research. students expressed enthusiasm around topics like the ethics of dna testing and proprietary genealogy websites, the seemingly endless amount of freely available digital genealogical resources, and the very notion of “digital literacy.” “everything about the course was tailored to getting students to understand digital literacy and family history—the very intimate, urgent relationship between them,” one student wrote.   introductory digital skills and practices in genealogy (idsg)   the idsg document is located in appendix a and is meant to be used by instructors of all kinds to summarize introductory digital literacy skills for family history researchers. these skills are structured within three frames, which are further organized into components and competencies. responding to caulfield’s (2016) ideas about digital literacy, this collection of skills and practices was created with the belief that researchers of family history need to combine digital domain knowledge with historical domain knowledge in order to efficiently and enjoyably conduct family history research. idsg is split into three frames: “discovery and access,” “discerning the value of information,” and “information storage and organization.” these frames were informed by students’ responses to the learning goals survey implemented during the first class session, as well as observations noted during the implementation of “tech tips” throughout the course. this portion of the paper details the ways in which librarian and archivist expertise came together to formulate the idsg section by section, in concert with applicable research and the preand post-class surveys from students.   frame #1: discovery and access   the first frame of idsg, “discovery and access,” is derived from the hybrid of online and physical resources that define, and complicate, genealogy research. public libraries are a natural centre of discovery, as these institutions serve as services and collections access points for members of the general public. many public libraries, too, retain their own genealogy sections and experts. previous research has also shown that novice archives users have trouble distinguishing libraries from other public institutions like archives, historical societies, local museums, and newspaper archives (hensley et al., 2014). in a time when users often feel adrift in huge swathes of digital information, it can be reassuring for overwhelmed beginners to have instructors emphasize the importance of nearby institutions that facilitate local research. in the course evaluations, two students highlighted the knowledge they gained around local spaces and institutions, as well as a desire for more guest lectures from leaders in local historical and genealogical organizations.   based on their experiences providing library and archival reference assistance, the authors also identified five foundational skills and practices for the discovery of online genealogy resources. these competencies are primarily concerned with defining and distinguishing groups of needed information and focus specifically on library catalogs, archival finding aids, and digital asset management systems (dams) as the primary online discovery and access tools. these tools provide access to information in different ways and are often inconsistent across institutional platforms. providing examples of these different groups of online discovery and access tools in context is thus invaluable for teaching genealogical research concepts as well as introducing individual resources. the importance of distinguishing between these types of platforms is also identified in the saa-acrl/rbms’s guidelines for primary literacy: “distinguish between catalogs, databases, and other online resources that contain information about sources, versus those that contain digital versions, originals, or copies of the sources themselves” (saa-acrl/rbms, 2018, p. 5).   idsg frame #1 is also influenced by the course’s original tech tips document (appendix d). these tools and practices represent simple mechanisms to increase the efficiency of online research. within the computer classroom used for this course, some students’ technological insecurities were heightened by using an unfamiliar computer or operating system. because of this, directions for technical skill are given for use with both pc and macs in the tech tips document. the need for simple, authoritative technology tutorials is one that extends beyond older adults: according to internal google research, only 10% of people know how to execute ctrl+f, the find function that allows you to search a web page or document (marks, 2011).   frame #2: discerning the value of information   returning to michael caulfield’s (2016) observations on the shortcomings of source evaluation heuristics without domain knowledge, a key takeaway from both teaching this course and subsequent auxiliary research lies in the necessity of background historical knowledge for family history researchers. duff and johnson (2003) call knowledge of local history and context “vital” and assert that it improves searching behaviours. rather than asking students in genealogy courses to grapple with abstract literacy frameworks, idsg specifies competencies like the “ability to identify and locate authoritative texts to build historical domain knowledge” within its second frame, “discerning the value of information,” as a primary component of genealogical education. the craap test is also used within this frame to introduce the concept of digital literacy. while the craap test has limitations, it is a helpful introductory heuristic to teach students how to identify basic elements of web documents.   one frame of the acrl framework for information literacy (2015) states that “authority is constructed and contextual.” genealogy instructors should emphasize the importance of researchers familiarizing themselves with the professional genealogical organizations responsible for creating structures of authority and standardization as a way to both direct and contextualize their research. for this course, the instructors asked students to explore the certified board of genealogists (cbg) website and discuss some of the primary functions of that institution. unlike their local group (the western montana genealogical society), cbg has extensive research standards, a code of ethics, and even a process for discipline and dispute resolution. while none of the students were ultimately interested in cbg membership or certification, this activity allowed them to explore the broader world of commercial and legal genealogy and asked them to consider the functional purpose of professional practices (for instance, the standard of proof) that might seem arbitrary within their less formal family history research.   frame #2’s scaffolding was directed by the librarian author, who had experience teaching information literacy skills in a classroom setting. but by specifically identifying local societies and cultural heritage institutions, this frame also counters the limitations of frameworks and heuristics: finding local experts and plugging into an existing community is invaluable for the family history researcher. the archivist author was much more “plugged in” to this human network than the librarian author, due to her frequent interfacing with such resources through reference and research work.   frame #3: information storage and organization   “information storage and organization” constitutes the most technically arduous frame of the idsg document. students seemed to understand some of the competencies housed within this frame, but only as they applied to their current computing practices. they did, however, identify the need for these skills in their learning goals. one student noted that they wanted to learn how to “organize information” more efficiently, while another indicated a desire to more effectively “us[e] [resources] and writing” (appendix b). the instructors observed that, when placed in a new environment (the computer classroom) or on an unfamiliar device, students lacked the technological resilience to fulfill practical tasks. the skills captured in this component of the idsg cover the how, where, and why of storing information, from personal note-taking to oral histories and public records. foundational to building this technological resilience is understanding common file types, knowing how to store them, and how to convert them. best practices in digital archiving and preservation served as the basis for identifying the skills necessary to fulfill the “information storage and organization” frame.   despite the course’s focus on text-based files, most genealogical research necessarily involves both image and audio files. many family history researchers are involved in the care and preservation of familial historic documents and photographs, often taking photographs or scans of these analog materials. this means that, in addition to any text-based documents, they are also likely creating, managing, and storing digital image files, which requires specific digital skills. while many digital collections are publicly available, not all are immediately available for convenient download. in the event that digital items are not openly available for immediate download, the instructors noted the importance of understanding how to download and save digital images from a restricted digital collection.   oral histories continue to be increasingly prevalent in public repositories of historic resources, and family history researchers should understand how to both access existing oral histories and, potentially, conduct and preserve their own (many students in the course expressed a desire to pass their research on to children and grandchildren). public institutions like the american folklife center at the library of congress retain helpful checklists for planning and conducting oral history interviews, and many other organizations (most notably npr’s storycorp) have launched phone applications for this purpose. the authors also provided a brief overview of the landscape of available recording equipment, applications, and documentation for oral histories, with the archivist providing much of the digital preservation expertise.   finally, the authors regularly used terms like “hard drive,” “google docs,” and “the cloud” throughout the course, a practice that was to become a major source of confusion for students. it is thus imperative that instructors facilitate a clear discussion of file storage options at the outset of instruction. defining nebulous technical terminology, and incorporating activities that demonstrate these concepts, can go a long way towards building confidence in students’ own technical abilities. as a user experience librarian and content strategist, the librarian author quickly noted that simple and precise technical language, along with clear explanations, was key to alleviating anxiety in students.   discussion   summary and findings   while both instructors believed at the outset of the course that an archivist/librarian collaboration would prove to be a compelling and particularly fruitful partnership, neither anticipated the ways in which this alliance would inform discoveries around digital literacy skills for family history research. they were forced to adapt both the syllabus and individual class structure as the course progressed and it became apparent that students required additional guidance for technological tasks. the resulting framework, the idsg document, serves as an artifact that melds threads of primary source literacy with digital and information literacy to form a tool that has the potential to enhance genealogical and archival instruction, empower and equip adult learners and beginner family history researchers with basic and necessary digital skills, and contextualize genealogical research within traditional information literacy frameworks.   the authors found that the platforms and practices that defined their work as librarian and archivist were very different when it came to discovery and access. it took a good deal of learning about each other’s practices for them to be able to identify and articulate the ways in which discovery tools and information artifacts commonly used by archivists were different from those used most often by a librarian, particularly with regard to user experience and behaviour. for example, why is a finding aid different from a research guide, and why do they often live on separate platforms? it was necessary to obtain a full understanding of both unique information landscapes in order to create a complete picture of discovery and access that students could learn to navigate. both authors occasionally experienced frustration with the significant differences between these information landscapes, particularly for the sake of their students.   digital literacy frameworks prioritize the skills required to create online content and perceive oneself as the creator (de george-walker & tyler, 2014). asking students to do so in an introductory course went beyond the parameters of the authors’ goals. instead, the course set the stage for creating a local “participatory genealogy culture,” one in which students could begin taking part (jenkins, 2009). building out from a digital literacy and family history research course, instructors envisioned facilitating other librarian–archivist collaborations like wikipedia edit-a-thons (for example, sliger krause et al., 2017) or planning a second course incorporating more sophisticated research and digital literacy skills. the construction of a locality guide, a reference document used extensively by family history researchers requiring regular maintenance, would provide another opportunity for a cohort of graduates from an introductory genealogy course to enact the skills and practices from idsg.   expanding family history researchers’ online expertise, resilience, and well-being not only improves the genealogy research experience but has potential for making a positive civic impact, as well. the digital skills on which this case study focuses are transferable to many other online behaviours that define everyday life (e.g., reading the news or researching a new car) and can help adult learners better navigate an ever-expanding body of online resources. further opportunities for research could include studying the impact of teaching digital skills for family history research on adult learners’ success at online tasks, like identifying trustworthy news sources.   exclusions and limitations   the methods and instruments used to collect data in this case study have limitations. first, the instructors did not implement matching preand post-course assessment tools in order to track either students’ achievement of learning outcomes or their ability to locate and assess digital information sources before and after the implementation of the course. instead, the instructors believed they would have subsequent opportunities to teach further iterations of this course and more carefully craft assessments based on what they learned. this was not to be the case, as both instructors departed the university of montana within a year of teaching this initial course. second, neither had experience developing or teaching a full course and did not have the tools to scaffold a new course and assessment around learning outcomes. still, the insights gleaned from these methods directly informed the creation of the introductory digital skills and practices in genealogy in invaluable and informative ways.   in deciding what to include in the idsg, the authors chose to omit content that is covered in other genealogy or digital literacy resources or that could not be scoped to skill-based practices. the idsg also excludes any discussion of popular family research platforms and services like familysearch and ancestry, despite students’ identification of common goals around learning how to use such specific genealogical resources. in spite of this desire to hew away from discussing specific information resources, the authors did prioritize resources from local cultural heritage institutions in their instruction, as opposed to proprietary websites. they faltered in articulating this intention to students, however, and recommend that fellow librarians and archivists engaging in genealogical instruction clearly communicate to students the extent to which they will (or will not) engage with popular online genealogy resources throughout the course. in the final course evaluations, three students explicitly expressed disappointment that they did not receive step-by-step tutorials for online tools like ancestry.com.   conclusion   this case study describes the context and creation of an evidence and practice based set of digital skills to answer the research question: what tools can we build to assist instructors in teaching the basics of family history research that combine digital skills with research skills? existing literature discussing related literacies often describes abstract concepts, not practical skills. the introductory digital skills and practices for genealogy document is a melding of archivist and librarian expertise that is meant to enhance students’ practical skills and domain knowledge in both historical research and digital literacy.   for adult learners, participating in a creative and challenging hobby like family history research is enjoyable, deepens a sense of familial belonging, and has the potential to improve overall health and wellbeing (conner et al., 2018). the authors saw this positive impact first hand in the form of a subset of students who, after the course completed, continued to gather at the local public library each month to both conduct their individual genealogical research and to support one another’s work. one of the authors had the opportunity to attend one of the regular meetings and witnessed the camaraderie that stemmed from the former students’ collaboration.   for the authors, collaborating to implement a full course provided a rare opportunity to bring together complementary expertise in order to provide a unique educational opportunity for community members. these types of collaborations are infrequently described in existing literature. the artifact that resulted from this case study, introductory digital skills and practices in genealogy, is meant to inspire and assist others who want to teach family history research, and to encourage them to make tangible digital skills a focal point of their instruction.   author contributions   jaci wilkinson: conceptualization, formal analysis, investigation, methodology, project administration, writing – original draft natalie bond: formal analysis, investigation, project administration, writing – review & editing   references   anderson, m., & perrin, a. (2017, may 17). tech adoption climbs among older adults. pew research center. https://www.pewresearch.org/internet/2017/05/17/tech-adoption-climbs-among-older-adults/   american library association office for information technology policy’s digital literacy task force. (2013). conclusions & recommendations for digital literacy programs and libraries. http://hdl.handle.net/11213/16262   association of college & research libraries. (2011). guidelines for instruction programs in academic libraries. http://www.ala.org/acrl/standards/guidelinesinstruction   association of college & research libraries. (2015). framework for information literacy for higher education. http://www.ala.org/acrl/standards/ilframework   blakeslee, s. (2004). the craap test. loex quarterly, 31(3), 6–7.   boyd, d. (2014). it's complicated: the social lives of networked teens. yale university press.   carini, p. (2016). information literacy for archives and special collections: defining outcomes. portal: libraries and the academy, 16(1), 191-206.   catts, r., & lau, j. (2008). towards information literacy indicators. united nations educational, scientific and cultural organization, information for all programme. https://unesdoc.unesco.org/ark:/48223/pf0000158723   caulfield, m. (2016, december 19). yes, digital literacy. but which one? hapgood. https://hapgood.us/2016/12/19/yes-digital-literacy-but-which-one/   conner, t. s., deyoung, c. g., & silvia, p. j. (2018). everyday creative activity as a path to flourishing. the journal of positive psychology, 13(2), 181–189.   de george-walker, l., & tyler, m. a. (2014). connected older adults: conceptualising their digital participation. in i. buchem, g. attwell, & g. tur (eds.), the ple conference 2013: learning and diversity in the cities of the future (pp. 107–114). logos verlag berlin.   duff, w., & johnson, c. (2003). where is the list with all the names? information-seeking behavior of genealogists. the american archivist 66(1), 79–95. https://doi.org/10.17723/aarc.66.1.l375uj047224737n   ellis, s. (2014) a history of collaboration, a future in crowdsourcing: positive impacts of cooperation on british librarianship. libri: international journal of libraries and information studies, 64(1), 1-10.   friday, k. (2014). learning from e-family history: a model of online family historian research behaviour. information research, 19(4), paper 641. http://informationr.net/ir/19-4/paper641.html#.ydgaiehkjiu   gee, j. p. (2003). what video games have to teach us about learning and literacy. palgrave macmillan.   greener, g. n. (2014, february 5). empirical evidence of the popularity of family history using digital traces. family search. https://www.familysearch.org/blog/en/empirical-evidence-popularity-family-history-digital-traces/   gottfried, j., & grieco, j. (2018, october 23). younger americans are better than older americans at telling factual news statements from opinions. pew research center. https://www.pewresearch.org/fact-tank/2018/10/23/younger-americans-are-better-than-older-americans-at-telling-factual-news-statements-from-opinions/   hensley, m.k., murphy, b.p., & swain, e.d. (2014). analyzing archival intelligence: a collaboration between library instruction and archives. communications in information literacy, 8(1), 96-114. doi:10.15760/comminfolit.2014.8.1.155   hull, g. a., & katz, m.l. (2006). crafting an agentive self: case studies of digital storytelling. research in the teaching of english, 41(1), 43–81.   jenkins, h. (with purushtoma, r., weigal, m., clinton, k., & robison, a.). (2009). confronting the challenges of participatory culture: media education for the 21st century. mit press.   lawton, j.r., & block lawton, h. public-academic library collaboration: a case study of an instructional house and property history research program for the public. the american archivist, 72(2), 496-514.   marks, p. (2011, november 23). google usability chief: ideas have to be discoverable. new scientist. https://www.newscientist.com/article/mg21228400-400-google-usability-chief-ideas-have-to-be-discoverable/   national genealogical society. (n.d.) interested in family history? retrieved february 25, 2021, from https://www.ngsgenealogy.org/family-history/   nichols, t. p., & stornaiuolo, a. (2019). assembling “digital literacies”: contingent pasts, possible futures. media and communication, 7(2), 14–24. http://dx.doi.org/10.17645/mac.v7i2.1946   the participants in the high-level colloquium on information literacy and lifelong learning. (2005). beacons of the information society: the alexandria proclamation on information literacy and lifelong learning. international federation of library associations and institutions. retrieved february 26, 2021, from https://www.ifla.org/publications/beacons-of-the-information-society-the-alexandria-proclamation-on-information-literacy   payne, k. (2020, march 6). genealogy websites help to solve crimes, raise questions about ethics [radio broadcast]. on morning edition. national public radio. https://www.npr.org/2020/03/06/812789112/genealogy-websites-help-to-solve-crimes-raise-questions-about-ethics   reference and adult services division history section genealogy committee. (1996). guidelines for a unit or course of instruction in genealogical research at schools of library and information science. rq, 36(1), 73–77. http://www.jstor.org/stable/20863048   rettig, p. j. (2008). an integrative approach to archival outreach: a case study of becoming part of the constituents' community. journal of archival organization, 5(3), 31–46. https://doi.org/10.1080/15332740802174175   rodriguez, g. (2014, may 30). how genealogy became almost as popular as porn. time. https://time.com/133811/how-genealogy-became-almost-as-popular-as-porn/   sachs, h. (2019, december 13). the dark side of our genealogy craze: understanding our roots may foster empathy—or division. the washington post. https://www.washingtonpost.com/outlook/2019/12/13/dark-side-our-genealogy-craze/   schultz, j. l. (2018). helping patrons find their roots: a genealogy handbook for librarians. ala editions.   sliger krause, r., rosenzweig, j., & victor, p., jr. (2017). out of the vault: developing a wikipedia edit-a-thon to enhance public programming for university archives and special collections. journal of western archives, 8(1), article 3. https://digitalcommons.usu.edu/westernarchives/vol8/iss1/3/   society of american archivists–association of college & research libraries/rare books and manuscripts section joint task force on the development of guidelines for primary source literacy. (2018). guidelines for primary source literacy. https://www2.archivists.org/sites/all/files/guidelinesforprimarysourceliteracy-june2018.pdf   smallwood, c., & gubnitskaia, v. (eds.). (2018). genealogy and the librarian: perspectives on research, instruction, outreach and management. mcfarland & company.   stern, c., & kaur, t. (2010). developing theory-based, practical information literacy training for adults. international information & library review, 42(2), 69–74. https://doi.org/10.1080/10572317.2010.10762847   theimer, k. (2011). what is the meaning of archives 2.0? the american archivist, 74(1), 58–68. https://doi.org/10.17723/aarc.74.1.h7tn4m4027407666   unesco institute for statistics. (2018). a global framework of reference on digital literacy skills for indicator 4.4.2 (information paper no. 51). united nations educational, scientific and cultural organization. http://uis.unesco.org/sites/default/files/documents/ip51-global-framework-reference-digital-literacy-skills-2018-en.pdf   yakel, e., & torres, d. (2003). ai: archival intelligence and user expertise. the american archivist, 66(1), 51–78. https://doi.org/10.17723/aarc.66.1.q022h85pn51n5800     appendix a introductory digital skills for genealogy (idsg): a guide for instructors frame #1: discovery and access discovering local organizations and resources ○       is able to gain access and/or membership to a local public library and accompanying websites and catalogs. ○       understands how and when to use interlibrary loan. ○       can identify local institutions with publicly available online resources or documents focusing on local history (e.g., church of jesus christ of latter-day saints libraries, local historical societies).   finding and evaluating online resources ○       is able to navigate online finding aids; understands how finding aids are constructed. ○       has an increased capacity for distinguishing between digitized and non-digitized archival resources. ○       understands the importance of using online resources to discover resources not available online. ○       knows key national government organizations that provide freely available digitized resources. ○       understands the kind of information contained within online library catalogs, as well as both the capacity and limitations of this information. ○       understands what oral histories are and how to access transcripts.   streamlining the online research experience ○       understands and can execute opening a link in a new browser tab versus a new window. ○       knows how to zoom in and out on text and images within a browser window. ○       can search for words or phrases in a long page or document. ○       is able to take a screenshot.   frame #2: discerning the value of information selecting and evaluating sources ○       can determine the “provenance” of an online resource using information contained within the document and accompanying metadata. ○       is able to apply the craap test to online resources that are not historical documents. ○       has an increased capacity for critically reading and evaluating the utility of academic articles. ○       is able to seek out subject experts at local cultural heritage institutions for assistance in selecting and evaluating resources.   building domain knowledge ○       can identify pertinent historical and cultural events that influence personal family history research. uses this domain knowledge to construct strong search terms. ○       is able to identify and locate authoritative, canonical texts to build historical domain knowledge that will assist family history research. ○       has knowledge of professional genealogical organizations and the resources and services they provide.   citing and attributing works ○       is able to implement a consistent citation style. ○       has a basic understanding of citation conventions for genealogical research. ○       can create in-line, hyperlinked text.   frame #3: information storage and organization identifying file types and their utility and conversion ○       can differentiate between pdf, docx, and html files. ○       understands the difference between tiff and jpg/png files. ○       is able to convert docx and html files to pdf, and understands why this matters. ○       is able to save a web page as a pdf. ○       is able to save an image from a web page in the absence of a “download” button.   recognizing file storage options ○       understands the difference between storing information on a hard drive and “in the cloud.” ○       has basic knowledge of common cloud-computing applications and services. ○       understands how to use external storage devices, such as an external hard drive or flash drive.   creating and saving oral histories ○       has basic knowledge of the equipment needed for conducting and capturing oral history interviews. ○       knows the recommended file types for preservation (wave) and sharing (mp3).     appendix b pre-course survey: what did students want to learn?   discovery and navigation of online resources subject-specific knowledge genealogy-specific skills “user-friendly resources” “native american genealogy (flathead & north dakota)” “the basics” “online resources” “irish immigrants” “dna versus family tree” “assess credibility of websites” “military records” “identifying people in photographs” “ancestry.com navigation” “homesteads” “develop family tree from scratch” “national academic library digital resources” “fort lewis” “using [resources] and writing” “enough info to get a reduced rate at missoula cemetery” “translation of foreign documents” “finishing family projects” “organizing information” “civil war”   “library of congress and national archives” “international research: first-generation immigrant”   “resources that can correct incorrect information” “story behind ancestors’ common names: who has the correct one?”       appendix c course evaluation questions the course as a whole was: ❏     excellent ❏     very good ❏     good ❏     fair ❏     poor ❏     very poor   the course content was: ❏     excellent ❏     very good ❏     good ❏     fair ❏     poor ❏     very poor   the instructor’s effectiveness in teaching the subject matter was: ❏     excellent ❏     very good ❏     good ❏     fair ❏     poor ❏     very poor   how would you rate the instructor’s explanations? ❏     excellent ❏     very good ❏     good ❏     fair ❏     poor ❏     very poor   class sessions were interesting and engaging: ❏     excellent ❏     very good ❏     good ❏     fair ❏     poor ❏     very poor   class sessions were well organized: ❏     excellent ❏     very good ❏     good ❏     fair ❏     poor ❏     very poor   did this course meet your expectations based on the description in the brochure? ❏     yes ❏     no   would you take another course from this instructor? ❏     yes ❏     no   what did you like about this class? [open response field]   do you have suggestions about how this course could be better? [open response field]   do you have any suggestions for other molli instructors/courses? [open response field]     appendix d tech tips note: when two keys are listed with a “+” in between, hold down the first key listed and then press down the second key so that both keys are held down simultaneously.   action keystrokes for computer running an apple operating system (mac) keystrokes for computer running on a non-apple operating system (pc: examples include windows, linux) what it does find command + f ctrl + f a search box will appear, and you can type in keywords and phrases to see if that page or document has any of those words. new tab command + t ctrl + t opens a new tab in your browser (e.g., firefox, safari, or chrome). new window command + w ctrl + w opens a new window in your browser (e.g., firefox, safari, or chrome). open link right click with your mouse or, using a trackpad, hold down the control key and then click with your trackpad. a list of actions will appear. select “open link in new tab.” ← same opens a link in a new tab in your browser so that your current screen doesn’t disappear. action keystrokes for computer running an apple operating system (mac) keystrokes for computer running on a non-apple operating system (pc: examples include windows, linux) what it does hyperlink text highlight text you want to have hyperlinked, right-click, select “link” from menu, and paste a url in the field. after link is created, make sure there is a strong visual difference between linked and unlinked text (usually blue and underlined). ← same allows you to “hide” a url in text, especially in a citation. zoom in the options at the top of your browser, click view >> zoom >> zoom in. ← same makes text or images larger inside your browser. these steps may vary slightly from browser to browser. save web page from the browser’s main menu, click file, then print. in the print pop-up, find near the bottom “open a pdf in preview.” once pdf is open in the preview application, click file in the main menu, then save. ← same saves a full webpage (even what isn’t on your screen if you have to scroll to see the full page) as a pdf so all text and images are preserved. this is a great idea in case the website disappears unexpectedly. action keystrokes for computer running an apple operating system (mac) keystrokes for computer running on a non-apple operating system (pc: examples include windows, linux) what it does take a screen shot press and hold down command, shift, and the 4 key. when your cursor (usually a black arrow) turns into a plus mark with a circle at the centre, let all three keys go. hold down your cursor and draw a box around where you want a screen shot taken. the screenshot will be saved as a jpg in your desktop folder. press the prtscn button on your keyboard. this will take a screenshot of your whole screen and copy it to your computer’s clipboard. open microsoft paint, go to file, and click paste. the screenshot will appear in paint. save the image as a png. saves an image of what is on your screen (usually as a jpg or png).   microsoft word feat_conference_abstracts.doc evidence based library and information practice 2006, 1:1      112 evidence based library and information practice       feature     abstracts of papers and poster sessions of the 3rd international evidence based  librarianship conference ‐ evolution of evidence: global perspectives on linking  research with practice    brisbane, australia  16 – 19 october 2005    reprinted with the permission of the conference organizing committee.       presentation types    research papers are full length papers (not  to exceed 10 pages) that critically discuss  completed work demonstrating the  application of research in practice.  discussion papers critically exploring issues  in the application of evidence based practice  to inform professional practice were also  welcome. accepted research papers are  published on the conference website at:  http://conferences.alia.org.au/ebl2005/    hot topics are shorter papers (1000 words)  that critically outline research‐in‐progress.  hot topics provide an opportunity to  discuss, and receive constructive feedback  on an ongoing research project. practical  papers that identify and facilitate  professional dialogue on key issues within  ebl were also welcome. accepted hot  topics are published on the conference  website at:  http://conferences.alia.org.au/ebl2005/    posters will allow for the presentation of  new developments in evidence based  practice and work in progress. posters offer  an excellent opportunity for new conference  presenters.  accepted posters abstracts are  available on the conference website at:  http://conferences.alia.org.au/ebl2005/  speaker abstracts    legend:    research paper   hot topic  keynote    persuasive evidence: improving  customer service through evidence‐based  librarianship  wendy abbott  bond university, australia    as a key service provider within a small private  tertiary institution, the library at bond  university excels in customer service. this has  been achieved through the dedication and  professionalism of staff and by developing good  relationships with the students and staff of the  university (our customers). it also relies on a  range of quality processes including the use of  surveys, benchmarking and performance  indicators. there has been a natural progression  from using quality processes towards adopting  the much broader approach of evidence‐based  library practice as a means of finding ways to  improve customer service.     http://conferences.alia.org.au/ebl2005/ http://conferences.alia.org.au/ebl2005/ http://conferences.alia.org.au/ebl2005/ evidence based library and information practice 2006, 1:1      113 this paper will use case studies to illustrate how  evidence‐based practice has contributed to  informing decisions and resolving issues of  concern in service delivery at bond university  library. one case study involves research to  assist in deciding the most appropriate opening  hours for the library. a second example is a  project to research and implement best practice  for managing collections of feature films on dvd  and video in a university setting. both these  projects were initiated in response to concerns  raised by customer groups. a third case study  will report on progress of a collaborative project  between the library and academic staff to  expand the teaching of endnote for  undergraduate students.     drawing on the experience of these research  projects the discussion will highlight the benefits  of evidence‐based information practice in finding  solutions to problems and maximising the value  of library services. the need for practitioners to  develop the tools and methods for undertaking  research will also be discussed.        where there’s a way there’s a will!  the royal college of nursing’s  information needs survey of nurses and  health professionals  ros bertulis, jackie lord  royal college of nursing, uk    in 2004 the royal college of nursing library and  information services carried out a survey into the  information needs of nurses and other allied  health professionals in the uk. we wanted a  good evidence base in order to develop our  services and play an influencing role to improve  nurses’ access to information. the survey  focussed on:   • information or knowledge to improve  practice in the clinical area   • information needed to support lifelong  learning and formal study.    the research was carried out by means of a hard  copy questionnaire which was distributed by  rcn activists and health librarians. this  methodology was selected to ensure that we got  a good response from all our target audience, not  just those health care professionals with good  access to information technology. we managed to  get a total of 1715 completed questionnaires and  a good cross‐section of our target audience,  making this exercise one of the largest and most  robust ever undertaken in the uk. the data was  analysed using spss.     the results of the survey were encouraging both  for nursing and information professionals.  having good access to information – via the  internet and via a physical library and via  information skills – appears to have a direct effect  on putting evidence into practice. those with the  best access to information were more likely to  both search for evidence and to change their  practice as a result of research. and respondents  who search for evidence or report changing their  practice as a result of research are much more  likely to find a range of information resources  useful for both improving practice and lifelong  learning. this paper will discuss the research  methodology and the key findings of the survey  which has national and international  implications.      evidence based information  practice: an abc and y?  andrew booth1 and anne brice2  1. university of sheffield, uk  2. university of oxford, uk    this keynote will endeavour to provide an  overview of the most significant developments  within evidence based information practice to  date. it will also attempt a brief circumspection of  parallel initiatives within related fields of  evidence based practice (such as education, social  care and management) – what can we learn and  what pitfalls should we avoid? rather than  uncritically adopt a paradigm that has been  developed in response to other professions’  acknowledged difficulties with information  retrieval and knowledge management the  information profession has the opportunity to  stamp its own distinctive contribution on  evidence based practice. at the same time there  are particular challenges for a profession that  does not itself have a tradition of research  interpretation and utilisation. key to the  development of evidence based information  practice is the need to stimulate a new generation  of reflective practitioner who responds to the  external environment – whether the actual  stimulus be their own observations, feedback  from users or the more formal contribution of  research findings.     evidence based library and information practice 2006, 1:1      114  evidence to support strategic  decision making for health care  information services: the effective  methods of providing information for  patient care (empiric) project  alison brettle, claire hulme, paula ormandy  health care practice r&d unit, university of salford, uk    background and aim  throughout the north west (and the uk) the  services provided to health professionals who  need information for patient care are varied.  some libraries provide mediated searches, some  provide information skills training, some employ  dedicated library trainers and others provide a  range of services. with limited resources, should  libraries be investing in some services rather than  others? in order to provide evidence for strategic  decision making for north west uk health  library services, the empiric project aims to  investigate the effectiveness and cost  effectiveness of these approaches.     methods  a baseline data collection survey to all health  libraries in the region (n=102) and literature  review were undertaken to contextualise the  research. following this a questionnaire survey  was carried out on 115 library staff and 15 library  trainers at 26 sites who agreed to participate and  a random sample of health professionals (n=780)  who had received services from the participating  sites. the questionnaires presented both  quantitative and qualitative data and these were  analysed using spss software, thematic analysis  and economic analysis. the findings from the  questionnaires were drafted into a report and  confirmed via two expert workshops and a range  of telephone interviews before presentation in the  final report.     findings  initial findings indicate that there is evidence to  demonstrate that information skills training  improves health professional’s ability to find  information for patient care, however mediated  searches are also effective in providing evidence  for patient care, research and professional  development. dedicated library trainers are as  effective as other library staff in providing  information skills training but time and lack of  confidence prevent health professionals from  using their information skills.     conclusions  the final results, conclusions and how these can  be used for strategic decision making will be  presented.        evidence and practice: evolving a post‐ graduate program for teacher‐librarians  raylee elliott burns  queensland university of technology    this session charts and appraises the designing  and implementation of a post‐graduate program  stimulated by research evidence. the ‘performing  hybridity’ project provides research evidence to  support the designing and design of the master  of learning innovation (teacher librarianship).  the mli (tl) is a foundation course for teachers  entering the profession and also offers masters  level professional development options for  graduated teacher‐librarian practitioners.         exploring evidence‐based  information literacy  catherine clark  university of western australia, australia    significant resources are spent in higher  education libraries on designing and  implementing programmes to ensure that  graduates are information literate. an emerging  challenge for libraries is to determine how  effective these programmes are, and to assess  whether the resources supporting them are being  well used.     the recent publication of the information skills  survey (catts, 2003) has provided an evidence‐ based test instrument for evaluating law,  education and social science students’  information literacy skills. however, there is  currently no equivalent instrument for the  sciences, including medicine.     this paper reports on research conducted with  medical students at the university of western  australia that aimed to validate the information  skills survey for medical information literacy  programmes. discussion includes the research  process and how the implications of the research  will affect information literacy programmes in  the future.     evidence based library and information practice 2006, 1:1      115  adding spice to our library intranet  site: a recipe to enhance usability  lisa cotter, larnich harije, suzanne lewis,  ingrid tonnison  central coast health, australia    gosford hospital (northern sydney central  coast health), in new south wales, is a teaching  hospital of the university of newcastle,  australia. this partnership in health education  and training has been particularly successful in  the area of library services, with a strong,  mutually beneficial relationship maintained  between central coast health library services  and the university of newcastle libraries.     in 2004 the central coast health (cch) library  services staff decided that their section of the  cch intranet had outgrown its original structure  and purpose, and a thorough revision of both  architecture and content was required. the  library’s intranet site also needed to reflect more  strongly the close relationship with the  university of newcastle’s library service,  evidenced by a recent migration of the cch  library holdings onto the university’s catalogue,  newcat.     the goals of the project were to produce a highly  usable library intranet site; to undertake the  project according to evidence‐based librarianship  principles; and to present the project to the  library community. accordingly, we reframed  the problem – a user‐unfriendly intranet site – in  terms of the following research question:     how can the usability of the central coast  health library services’ intranet site be  improved to enhance to siteʹs effectiveness as a  gateway to the library’s services and resources,  for the staff and students of the organisation?     a four‐stage project plan was developed:   • stage one – bring the site up to an  acceptable level to be used as a starting  point from which further evidence‐ based improvements can be made;   • stage two ‐ locate, retrieve and examine  the evidence to identify what  development is required. the evidence  was derived from a number of sources  including research‐based web design  and usability guidelines; and locally  based research carried out at central  coast health employing both  qualitative (interviews) and quantitative  (usability testing and analysis) methods;   • stage three – redevelopment of the  intranet site, a review of content and  creation of guidelines for ongoing  maintenance; and   • stage four – establishment of an  internet site for the cch library  services, based upon findings of the  intranet site study.    this paper will outline the progress and findings  of this research project.      four dimensional evidence based  management: calculating, comparing,  contrasting and containing   j. eric davies   library and statistics information unit, department of  information science, loughborough university, uk    managers face increasing pressures to develop  and deliver appropriate services of quality in a  world of limited resources. innovation and  change go hand in hand with the quest for  efficiency gains and continuous improvement.  there is a range of performance evidence – local  and global ‐ at the managerʹs disposal, but it has  to be used appropriately and intelligently. this  paper explores the various dimensions, or  applications, of evidence based management  (ebm) in library and information services.    four aspects are identified:   • [calculating] local performance data on  a system and its individual service  points   • [comparing] pooled data gathered and  benchmarking against similar systems   • [contrasting] performance against  published standards and national  examples of best practice   • [containing] and assimilating research  based intelligence and macro data that  informs service design and decisions.     each aspect is briefly discussed with examples  drawn from the various sectors of library and  information services provision. they are chiefly  based on lisuʹs experience in consulting for  library and information services, collaborating  with various interest groups, and undertaking  applicable research into service related topics.    evidence based library and information practice 2006, 1:1      116 the future is promising as ebm gains  prominence in the professional agenda. in the uk  , lisu has a long history of developing  appropriate tools and skills; lirg, now  assimilated into cilip, has consistently  championed the cause of practice‐based research;  and cilipʹs ‘newʹ policy and strategy for r&d  exemplifies its commitment to this area.    the author concludes that there is still some way  to go both in the acquisition, interpretation and  application of evidence in the real situation. the  link between theory and practice has to be  strengthened and the competencies of managers  (at all levels) in ebm has to be enhanced.        evidence‐based public health  informatics training for public health  practitioners  jonathan d. eldredge, richard carr  the university of new mexico health sciences library and  informatics center, usa    question  does evidence‐based public health training  increase the frequency and sophistication of  practice‐related questions among public health  professionals?     background  this training program includes a needs  assessment of professional‐level department of  health employees, the development of evidence‐ based public health training for these  predominantly rural practitioners,  implementation of the training itself, followed by  both immediate and two‐week post‐training  evaluation instruments.     methods  randomised controlled trial. the intervention  group will receive the training early in the  program whereas the control group will receive  identical training at a later date. the frequency  and sophistication of the questions generated by  both intervention and control groups will be the  basis for the comparison.     results  we hypothesize that the intervention group will  articulate a statistically significant greater  number of practice‐based questions and that  these questions will be far more sophisticated  following the ebph training session than the  questions generated by the still untrained control  group.     conclusions  at the time of the ebl conference during  october 2005 we will be able to report  preliminary results based upon our having  trained and compared the questions articulated  by 90 of the 120 participants scheduled for  training in this program        decisions, decisions – libraries,  bandwagons and clinical decision support  systems (cdsss)  cheryl hamill1, cecily gilbert2  1. fremantle hospital and health service, australia  2. sir charles gairdner hospital, australia    can a collection of electronic library resources  and services constitute a decision support  system? should health librarians hitch their value  wagon to the star of clinical decision support?     in recent years, many health libraries have  bought bundles of electronic resources, as a way  of providing online access to clinical information  throughout their health service. such collections  may include citation databases, drug resources,  fulltext journals and textbooks.     with networked access to these products often  available throughout a health service, clinicians  are encouraged to integrate quick searches into  their process of deciding a patient’s care. projects  in clinical librarianship are also fostering active  use of electronic resources to resolve  uncertainties in patient care. the clinical librarian  researches questions of specific patient care and  prepares a summary of findings for the medical  team, often within a 24‐hour time frame.     computerised clinical decision support systems  (cdss) are also becoming mainstream  applications in many health services. these  programs are designed to flag an individual  medical record when the patient passes a critical  boundary, such as falling due for a routine test,  or registering a raised blood pressure reading.  the treating physician is prompted to initiate  preventive or early interventions to avert a likely  problem, or alerted when incompatible drugs or  unsuitable tests are ordered.     the attraction of a cdss is that – if followed – it  saves treatment dollars and may improve patient  evidence based library and information practice 2006, 1:1      117 health. there are suggestions that serving up  online information resources will give some of  the same benefits. as more knowledge emerges  about the impact of library resources on patient  care, recent systematic reviews of decision  support systems are highlighting the features  which are key to their effectiveness. in this paper,  we examine the evidence for the relative value of  electronically delivered library resources and  truly integrated decision support systems, and  make a case for a broader definition of the value  library resources and services add to clinical care.        transaction log analysis @ state  library of queensland  scott hamilton1, helen thurlow2  1. queensland university of technology, australia   2. state library of queensland, australia    the state library of queensland’s voyager  online catalogue (opac) is a key service in our  agenda to develop a smart library network. the  opac receives the highest number of hits on the  state library’s website and the logs recorded  over 1.6 million hits in 2004. in 2004, an industry  project undertaken by a queensland university  of technology (qut) student analysed the opac  transaction logs. the project’s objectives were to  review the log files for the following:   • most common search terms used by  patrons   • most common search strategies used by  patrons   • success or failure rate relating to  searches undertaken.    the log files comprised the following:   • 2002 : 19/8 to 30/12 80mb 368k rows   • 2003 : 1/1 to 11/10 172mb 795k rows   • 2004 : 20/4 to 20/8 40mb 185k rows    this paper, co‐presented by qut and the state  library, details the research process and findings.  data was analysed using qlikview, a business  intelligence application. the results detailed who  was searching (based on ip address), when they  were searching (by hour, day, week), which  search options were selected and what search  terms were entered. a literature review provided  discussion about what constitutes a successful  search and, according to some criteria,  approximately two thirds of the state library’s  voyager opac searches would be considered  ‘unsuccessful’. the state library is now using  this research to inform decision making,  specifically in the following areas:   • collection development – most/least  popular subject areas in the collection;  collection gaps identified by  unsuccessful searches   • help screen content   • staff training – assess staff  understanding of opac search options  and develop appropriate training   • cataloguing – practices that may  improve search success rates;  relationship between keywords and  subject headings   • opac design – search options  presented on search screens and  terminology used.    the qut industry project provided the state  library with significant research analysis and  results which are being used to support changes  to opac design and content – evidence of a  successful collaboration between researchers and  practitioners.        measuring value: a comparison of  performance quality measures and  outcomes identified by australian and  north american libraries  steve hiller1, cathie jilovsky2,   1. library assessment coordinator university of  washington, usa   2. caval collaborative solutions, australia    nearly ten years ago, stoffle and her colleagues  at the university of arizona stressed that  “librarians must be sure that their work,  activities and tasks add value to the customer.”  to accomplish this, libraries should “collect data  and use them as the basis for decision making  rather than rely on subjective impressions and  opinions.” in other words, libraries needed  evidence to plan and manage their resources and  services in ways that added effective value to the  customer. however, many libraries while  recognizing the need to use data as good  management practice, are unsure what evidence  was needed, how to collect and analyze it, and  finally how to use it to improve libraries. the  association of research libraries (arl) in north  america and caval collaborative solutions in  australia have worked to address these issues by  developing programs and training to assist  libraries in identifying and measuring what is of  evidence based library and information practice 2006, 1:1      118 value to customers and the outcomes of library  activities.     how alike are american and australian libraries  in their perception of what data is critical for  library performance? this paper compares  evidence and outcomes identified by australian  and north american librarians as important to  measure for library service quality. data for this  study comes from information provided by  participants in programs and workshops run  separately by arl and caval in 2004‐05. the  arl program, “making library assessment  work: practical approaches for developing and  sustaining effective assessment” evaluates  measurement and assessment activities in  academic research libraries. seven libraries are  participating during 2004‐05, and another twelve  in 2005‐06. caval sponsored six half‐day  seminars on “practical approaches to effective  library assessment in 2004 and has a number of  measurement and assessment related workshops  scheduled for 2005 including six full‐day  workshops on “statistics, measurement and  performance indicators”.        effective methods for teaching  information literacy skills to  undergraduate students: what does the  library research literature reveal?  denise koufogiannakis  university of alberta libraries, canada    the objective of this study is to assess which  library instruction methods are most effective for  improving the information skills of students at an  introductory, undergraduate level. a systematic  review of the library research literature was  conducted. 15 databases were searched, and  more than 4000 potentially relevant articles were  retrieved. included studies met the following  criteria: instruction had to be librarian or library  assistant led as part of a class or stand‐alone  session (any method), the population had to be  undergraduate students at a post‐secondary  academic institution, and the research studies  had to have an evaluative component that  measured the cognitive outcome effect on  student learning via some test of information  literacy. more than 100 articles met the inclusion  requirements, went through the data extraction  process and were critically appraised. results  will be presented according to the analysis of  various data extraction components, including:  quality of the research, standards from the  association of college & research libraries’  information literacy competency standards for  higher education addressed by research  outcomes, the level of cognitive learning assessed  (based upon bloom’s taxonomy), instructional  topic, the various modes of instructional delivery  used, and the teaching methods shown to be  most effective based upon research results. this  systematic review will aid academic librarians  who teach information literacy skills to  undergraduate students make more informed  decisions regarding their teaching methods. this  study entails one piece of the complex puzzle  relating to information literacy, based upon  cognitive outcomes.        why don’t mental health staff use  library services? a qualitative and  quantitative investigation  john loy  avon & wiltshire mental health partnership nhs trust,  uk    while the importance of making practice  evidence‐based has become firmly established  within healthcare, the research literature  identifies a gap in information provision for  mental health staff1. the dispersed nature of  mental health services means that electronic  resources are becoming increasingly important as  a means of delivery of information to staff.  however, not everyone has the skills, support,  desire, or in some cases even the hardware to  make effective use of these new services2     this paper will present the results of a two‐part  research project conducted within the avon &  wiltshire mental health partnership nhs trust  (awp) during the summer of 2004. the first  stage takes the form of semi‐structured  interviews with a purposeful sample of staff.  thematic analysis of the interviews feeds into the  design of the questionnaire, distributed to a  representative sample of 500 staff and forming  the second stage of the study.     the combination of qualitative and quantitative  data provides a rich picture of how staff within a  mental health trust interact with library services.  do staff working within a hospital setting, and  with access to library services on site, make more  use of services than their community colleagues?  which professional group are the keenest users?  have the national core content services  evidence based library and information practice 2006, 1:1      119 provided across england made access to the  healthcare literature a reality?     when preliminary results from this study were  presented at the uk’s health libraries group  conference in belfast, september 2004 they were  cited on lis‐medical as “the highlight….that  sticks in my mind”3. now that a more detailed  analysis of results has been possible we are able  to view a more complete picture. we will  consider how the research results are shaping the  delivery of library services within awp and the  challenges of promoting and integrating  nationally procured services with locally  delivered resources.     1. blackburn, n. 2001 building bridges: towards integrated  library and information services for mental health and  social care. health information and libraries journal 18  pp 203‐212   2. adams, a. and blandford, a. 2002. acceptability of  medical digital libraries. health informatics journal 8  pp 58‐66   3. glover, stuart. lis‐medical 20 september 2004     the doctoring of evidence based  librarianship  peter macauley   deakin university, australia    in librarianship we often lend things to others,  but when it comes to research we tend to borrow  more than we lend. we borrow theory, we  borrow pedagogic and management practices  and, of course, we have borrowed evidence  based practice and turned it into evidence based  librarianship. in a multidisciplinary, trans‐ disciplinary and global world, this is  understandable and acceptable. however,  originality and creativity are positive traits which  can demonstrate the progressiveness and  maturity of a profession. when those from other  professions and disciplines borrow our practices  and evidence—based on our research—such  recognition acknowledges we are making  original and innovative contributions to the  knowledge economy. while our profession has  an abundance of highly educated and trained  practitioners, our formal training doesnʹt  normally include higher order skills such as  research methodologies or theoretical  frameworks. we often espouse the deficiency of  our usersʹ information literacy yet we too have  deficiencies and particularly with regard to  research. these deficiencies can be overcome by  research training. the universally accepted form  of research training is the doctorate; either the  phd or a professional doctorate.     doctoral qualifications provide training in  theory, methods, ethics, writing, scholarly  communication, information literacy, project  management and in‐depth subject knowledge.  they also establish credibility within our  community of practice and in communities  external to our own. to pass, a doctorate must be  adjudged as an original and substantial  contribution to knowledge. to move forward as a  profession we should be creating new  knowledge—original knowledge—which  reinforces the integrity of the profession and us  as individual practitioners. in other words, we  should create the evidence rather than basing our  practice on the evidence of others. this  presentation critiques the role of doctoral  research in creating an evidence base for our  professional practice and challenges librarians to  become credentialed by undertaking a phd or a  professional doctorate. becoming a credentialed  researcher is not without problems however. for  instance, you will question and critique the status  quo; you will relate theoretical constructs to  practice which may frustrate some colleagues;  and you may query bureaucratic mechanisms  within your working environment. your research  will inform practice and vice versa. you will base  decision making on substantiated evidence and  not just the opinions of others.     whose role is it to undertake research in  librarianship? where does the creativity come  from within our profession? what is good  research and what is not? do library practitioners  have a voice and an audience when it comes to  research and what does the doctorate provide for  you—the practitioner—and the profession as a  whole? these questions will be addressed, and  this presentation will debunk many, if not most,  of the myths about the irrelevance of the  doctorate to the professional practice of  librarians. we cannot rely on practice alone;  rather, our practice should also be based on  research: research that we create. should we  follow, or should we lead?        information seeking behaviours of  business students and the development of  an academic business digital library  joyline makani, kelli wooshue  dalhousie university libraries, canada     evidence based library and information practice 2006, 1:1      120 there is a wealth of research on the development  of digital libraries, particularly regarding  information retrieval and technical  considerations. however, little has focussed on  how information is sought and transformed into  knowledge and how this should influence digital  library design. digital libraries are not simply  collections of digital versions of existing  resources. covi and kling (1996) observed that  just building a digital library would not be  enough; the digital collections, storage, and  transmission should be useful to people who use  them. in other words, effective digital library  design requires an understanding of how users  do their work, how they use information, how  they create knowledge, and how digital libraries  support these processes. research on  undergraduate students indicates that they are  increasingly able to use digital resources even  though they are not necessarily information  literate (lombardo and miree, 2003). it is  critically important that digital libraries help  improve the transferable information literacy of  students. this research project will examine and  analyse how business students learn, gather, and  use information individually and collectively,  and how they share their work across geographic  boundaries and time zones. the students studied  will be drawn from the school of business at  dalhousie university. the study will be based on  qualitative data collected through in‐depth semi‐ structured interviews, task‐analysis and  observational study of business students at  dalhousie university as they perform group  research tasks. as the students will be involved  in a highly interactive and contextually sensitive  decision‐making process, qualitative techniques  are appropriate for highlighting the themes,  processes, and cognitive behaviours of the  students. (hepworth, 1998) the  recommendations emerging from this study will  form the basis for best practice guidelines for  effective business digital library design in the  academic environment.     evidence‐based librarianship:  linking research to practice in the north  american context  joanne gard marshall   school of information and library science, university of  north carolina at chapel hill, usa    ebl has emerged as a truly international  endeavour mirroring the adoption of evidence‐ based practice by health professionals around the  globe. this paper will review the evolution of  ebl with an emphasis on the u.s. and canada.  particular attention will be paid to the role of  professional associations in diffusing ebl as an  innovation in practice and as a tool for  transforming the profession. the medical library  association and the special library association  have both had research statements in place for  some time. sla has revised their research policy  within the last three years, retitling it ʺputting  our knowledge to workʺ. mla currently has a  task force reviewing their policy statement,  ʺusing scientific evidence to improve  information practiceʺ. other professional  associations such as the american library  association and the association of college and  research libraries, have included the ability to  understand and conduct research as one of the  competencies expected of professional librarians.  challenges and barriers to the full  implementation of ebl will be reviewed and  strategies suggested for ongoing development.        employing evidence: does it have a  job in vocational libraries?  cecily martina, brad jones  southbank institute of tafe, australia    given that librarians are tertiary educated –  occasionally with supplementary qualifications  covering research and statistics – it is not  surprising evidence based librarianship, albeit a  relatively recent development, has an academic  appeal.     in the vocational sector past energies have been  occupied with a greater librarian / clientele  disparity and matters pragmatic rather than  research. nevertheless, as several commentators  have pointed out, such a distinction may be  considered harsh, as librarians generally have  been doing research integral to their everyday  role. this also applies in the vocational setting.     in more recent times the vocational librarian has  been functioning within the most evidence based  sector in the educational environment: national,  transportable, prescriptive, competency based  and outcome driven training packages. the  vocational librarian is now well embedded  within the educational process due to  developments such as a more varied clientele,  online learning, evolving pedagogical styles, and  changing liaison and collaborative roles.  measurement and evidence have become  evidence based library and information practice 2006, 1:1      121 increasingly significant in an environment that  requires accountability, quality assurance, and  return on investment.     this paper examines some of these developments  together with some tentative examples, with a  focus on information literacy from the southbank  institute library experience. we suggest that  vocational and other librarians have been doing  research but need to be more systematic about  design and analysis, librarians need to develop  ‘evidence literacy’ as one of their professional  evaluation skills, and that evidence based  librarianship will need to deliver practical  outcomes to establish credibility in a vocational  context.        the just‐in‐time librarian consultation  service  j. mcgowan, j. w. hogg, c. campbell, d.  salzwedel, d. worster, t. rader, e. soto  institute of population health, university of ottawa,  canada.    objective  the purpose of this study is to investigate if  collaboration between librarians and family  physicians, nurses and nurse practitioners  improves access to information for high‐quality  decision making in primary care. it will  investigate whether or not a librarian  consultation service reduces the workload of  clinicians and/or saves them time. as well, it will  examine whether or not a librarian consultation  service is cost‐effective.     study intervention/implementation  this project provides a librarian service to family  health networks (fhns) to answer short  reference questions. family physicians, nurses  and other members of the primary care team are  taught how to use this service, including how to  create a clinical question and which types of  questions they may request from the librarian.  the questions are asked using a hand‐held device  or a web‐based form. the questions are answered  by the librarians within twenty minutes to  provide “just‐in‐time” information.     a detailed librarian flow chart was developed to  assist librarians in selecting electronic evidence‐ based sources for answering questions. evidence‐ based sources were appraised using a validated  assessment tool. physicians and a research  librarian in the department of family medicine  provided training in evidence‐based medicine for  the project librarians and support staff. as well,  the project librarians received detailed training  about the information needs and resources in the  family medicine discipline.     research design  this project conducts an outcome and process  evaluation of the service using an rct design  and a mixed methods approach (survey,  interviews).     early findings  the project has been introduced into several  primary care practices. qualitative results from  small group discussions will be shared. examples  of participant questions and librarian answers  will also be discussed.     implications  it is anticipated that the results of this project can  be applied widely to other fhns, including  urban and rural fhns, either locally, provincially  or nationally.        can the quality of literature searches  be measured and improved? the ehtas  project  j. mcgowan1, margaret sampson2, carol  lefebvre3  1. institute of population health, university of ottawa,  canada,   2. childrenʹs hospital of eastern ontario research institute,  canada  3. uk cochrane centre, uk.    objective   the quality of a health technology assessment  (hta) report is dependent on the accuracy and  completeness of the evidence base and this is  dependent on the quality of the search. however,  a validated process for evaluating the quality and  completeness of the evidence base for systematic  reviews does not exist. this project will see the  development of a validated checklist to verify the  methodological quality of the electronic search  strategy used in the process of hta  development.     methods  a search of the literature to identify the evidence  base for the importance of checklist items to the  validity of the search will be conduced. a draft  checklist has been created based on the review of  105 cochrane reviews. stakeholders will be  evidence based library and information practice 2006, 1:1      122 involved in the development and validation of  this checklist through a consensus‐building  forum. from this forum we will develop criteria  and guidelines for rating the items on the  checklist. these will then be tested based on a  test collection of published search strategies from  hta, the cochrane collaboration and specific  journal articles. this process will be refined and  retested with a senior librarian and less  experienced searchers.     outcomes  the major project deliverables from this include a  peer review checklist and rating guide, a  consensus‐building forum, journal publications  and knowledge translation initiatives.     implications  taking an identified a gap in the quality of  published searches, we propose a cost‐effective  mechanism for ensuring and protecting the  evidence on which policy formulation and  decision‐making rests. this project will advance  the methods for undertaking health technology  assessments and systematic reviews.        evidence‐based morsels – a taster of  strategic decisions based on operational  data  marian morgan‐bindon, leona jennings  gold coast city council, australia    this paper discusses a trio of case studies where  evidence‐based practice, using data from a  variety of council systems and sources, was used  in the planning and evaluation of library services  at gold coast city council’s 18 service points.     in the first example population density mapping  from data derived from council’s gis was  correlated with residential population estimates  and analysis of library membership data to assist  planning and reviewing the locations of library  facilities and broader infrastructure planning.     the second case study outlines how patterns of  use, observation and customer consultation was  utilised in the review of general access to library  facilities, and in particular, library opening  hours.     finally the paper discusses ways in which data  from the library management system is being  used on almost a daily basis daily to plan and  evaluate library programming, benefits  realisation, strategic planning, the impact of new  services or changes to library policy.        but does it work? – building the  evidence base for information literacy  development  gill needham  the open university library and learning resources  centre, uk    while the last ten years have seen exciting  developments in information literacy (il)  worldwide, this is still a relatively new area for  research with consequent gaps in our knowledge  about what works best for which groups of  learners. the stronger the available evidence base  on the effectiveness of il interventions the easier  it will be to   1. compete for resources in an increasingly  cash‐strapped environment   2. persuade our further and higher  education institutions to integrate il  into policy and the curriculum, and   3. convince our students that il  development is a worthwhile use of  their precious study time.    the majority of people carrying out innovative  work in il are busy practitioners with little time  to pursue research opportunities. collaborative  initiatives sharing resources and expertise across  national and domain boundaries could benefit  the entire il community.     although the open university’s information  literacy unit has been successful in developing a  range of materials and programmes, these have  been largely based on informal and anecdotal  evidence of both the needs of our students and  the most effective approaches. like many of our  colleagues in other institutions, we collect both  demographic and survey data on our students, as  well as records of their performance, and we are  interested in the extent to which this data might  be usefully pooled to inform future development.  we have recently concluded a project to develop  a robust online diagnostic tool to measure the  level of skills of our learners, and to ascertain the  effectiveness of our interventions. we are keen to  explore the potential of this tool for use in cross‐ sectoral research and we would like to discuss  possibilities with interested colleagues.     in summary this paper presents an opportunity  to open dialogue about the following:   evidence based library and information practice 2006, 1:1      123 • the research questions which the il  community would like to address and  possible models of collaboration   • the availability of routinely collected  data which could inform the research  agenda   • the possible use of diagnostic tools in  partnership research       teaching information skills for  evidence based practice in occupational  therapy – what is effective?  anne parkhill  monash institute of health services research and aptly  information design, australia    overview  the author, in her role as information specialist  at the monash institute of health services  research, will report on a program to teach  information management skills for evidence  based practice to the southern health hospital’s  occupational therapy department.     introduction  the research is being conducted following an  ebip (evidence based information practice)  mnemonic known as spice:     setting – southern health and st vincent’s  hospitals (melbourne)   perspective ‐ occupational therapy staff uptake  and use of evidence based practice in their work   intervention – involvement of an information  professional in training, preparation and  attendance at monthly journal clubs for six  months   comparison – current practice   evaluation – information skills in evidence based  practice after 6 months     a review of the literature in education,  librarianship and allied health is being conducted  to identify current practice for programs such as  the study session under review. questions to be  considered include;   • in relation to evidence based practice,  how is it applied in the discipline of  occupational therapy and what is  current best practice?   • in relation to evidence based  information practice, how is such  teaching undertaken in the information  profession?   program description   the author is designing an ebp training program  for the hospital ot department whereby staff  will be offered face‐to‐face sessions to teach  search strategy formulation and retrieval of  evidence from relevant online databases. the  author is also helping with preparation for and  attending monthly journal club meetings.     discussion and outcomes    ot staff will be assessed for information skills  before and after the teaching sessions and after a  6 month period to test their search and retrieval  knowledge. the research in this topic will be  compared with best evidence based practice for  both the information practice (ebip) and  occupational therapy fields (ebot). results from  the study will lead to a hospital guideline for  future teaching programs.       issues and problems for librariansʹ  conducting research ‐ an example of a  randomised controlled trial comparing the  effect of e‐learning, with a taught  workshop, on the knowledge and search  skills of health professionals.  nicola pearce‐smith  department of knowledge and information sciences,  supporting public health (nhs), uk.    objectives  to establish whether there is a difference in terms  of knowledge and skills, between self‐directed  learning using a web‐based resource, compared  with a classroom based interactive workshop  directed by librarians, for teaching health  professionals how to search.     background  training health professionals to search databases  is thought to improve their skills and knowledge,  but there is little evidence in the literature to back  this up. health librarians regularly teach  searching workshops, but do not know whether  these social cognitive learning methods are  effective for health professionals. self‐directed  online, or e‐learning, is becoming more  widespread, but again there is little evidence to  show whether this is an effective training  method. librarians need to be confident that  group sessions to teach search skills are  worthwhile, and developing e‐learning resources  for search skills training is justified.   evidence based library and information practice 2006, 1:1      124 methods  the study design is a randomised controlled trial,  involving health professionals working for the  oxfordshire radcliffe trust. participants will be  randomised into two groups – one group will  receive access to a search skills web resource (e‐ group) and the other group will receive a search  workshop taught by a librarian (workshop  group). the e‐group will be shown a  demonstration of the e‐learning resource and  asked to complete as much of the module as they  can over the next week; the workshop group will  receive a 1.5 hour taught workshop. participants  will complete pre‐ and post‐intervention tests  involving multiple choice questions and practical  searching using clinical scenarios. a follow up  test will be sent 4 weeks later. the test results will  be evaluated using a score chart/checklist.     results and conclusion  the main hypothesis for this trial is that the two  educational interventions will be equally  effective in transferring knowledge and skills.  when results are obtained and analysed in the  summer, it will be possible to assess the validity  of this hypothesis.       can evidenced based research  methods assist library managers in  becoming better managers?  bob pymm, damian lodge  charles sturt university, australia    evidenced based librarianship (ebl) has tended  to focus on how effectively librarians deliver  information to their clients. one area less  developed, is the use by library managers of  evidenced based research methods to improve  their own management skills and abilities. this  paper will focus on a case study, undertaken in  2005, that assesses how effective library  managers believe themselves to be, compared to  how they actually rate against a series of  measures, using a number of objective and  subjective evaluation techniques such as  questionnaires, performance evaluation and 360  degree feedback. this work is based, in part, on  the research of psychologists kruger and  dunning (1999), indicating the gap that exists  between people’s perceptions of their skills and  their actual skill level. it is anticipated that the  evidence drawn from this case study will provide  a preliminary basis for identifying gaps in  management education and training and form  the basis for a broader study of management  competency in libraries.       evidence based instruction and  development the daubert approach  alan t. schroeder, jr  california state university, usa.  in 1993 the united states supreme court handed  down the daubert decision (daubert v. merrill‐ dow pharmaceuticals, 509 u.s. 579, 113 s. ct.  2786, 125 l.ed.2d 469 (1993)). the author  proposes using the scientific and non‐scientific  criteria elucidated in daubert for successful  teaching and learning: whether a particular  theory has been tested; peer review; error rate;  verified standards; and acceptance within the  larger library community. librarians must be  aware of the need to properly document their  findings and present them in a logical, systematic  and scientifically valid manner to avoid a  daubert type challenge. the criteria put forth in  daubert offer a conceptual framework for the  future and an analysis of the evidence  base which underpins practice. examining  evidentiary findings through the daubert lens  offer a new instrument in proposing new  information services.        what’s the point of it all if no one uses  your library? using and creating evidence  to promote a library service – ongoing  research  jane shelling  national resource centre, alcohol and other drugs council  of australia, australia    aim  library promotion, as part of an overall library  marketing strategy is an essential part of library  survival. research into this topic began with the  question; “what is the most effective promotional  method1 for the national resource centre2 to  undertake, in order to achieve a positive,  promotional outcome3”. to answer this question  i decided to gather evidence, integrate my own  acquired knowledge, increase my local  knowledge, apply the results, measure the  outcome and attempt to create further evidence.     method   • i conducted literature searches in health,  library and marketing databases and  evidence based library and information practice 2006, 1:1      125 websites to gather, initially, background  knowledge and then foreground evidence to  specifically answer the question on;   o promotional ideas and methods,  and their successfulness with  libraries similar to the national  resource centre (nrc).  • i attended relevant workshops/conferences  to network, gather ideas and learn more  about promotional methods.   o the initial research allowed me to  get the best from these workshops.  • i organized a telephone survey of the  resource centre’s clients. this was to gather  quantitative data and also qualitative data  (gathered from speaking directly to clients). i  wanted to find out;   o who they were,   o what, if any, problems they were  having with the present service,   o what they liked best about us,   o what they wanted from us and   o what their biggest information  problems were.  • i devised an individual “success”  measurement tool to apply to all  promotional activities for future use.   o this allowed me to define the  “successfulness” of promotional  activities in a number of positive  ways.     results  i have a number of planned, promotional  activities at various stages at present and as they  are completed i am measuring their impact. at  the end of this year i want to evaluate the whole  process i have undertaken, assess the range of  promotional activities completed, write up my  results and recommendations for the future, and  write a paper which i hope will add evidence to  the knowledge base of library promotion.     1. what promotional activities are there and how  have other libraries ranked their effectiveness.   2. evidence needs to be specific to my service, and  have applicability to what makes my library  unique so evidence needs to apply to either a drug  and alcohol library or a small health library or at  least a special library.   3. what is a positive outcome, this needs to be  measured. a quantitative measurement could be  an increase in; membership, enquiries/contacts,  loans. how could increases in; awareness of  service, profile of organization, educational  benefits, credibility be measured.       virtual evidence: analyze the  footsteps of your users  win shih  university of colorado health sciences center, usa.    this paper will discuss a practice of utilizing  web logs and usage statistics to maximize library  resource and improve services at the university  of colorado health sciences center library. like  many of its peers, in 2004 the library switched  most of its print journals to online subscription.  in this transformational process, the library was  faced with unique issues related to digital  resources and services. questions concerning  reliable usage measurements, proficient access  management and control, adequacy of user  licenses, robust system performance and it  infrastructure, strong compliance of copyright  laws and contractual agreements lead to the  development of a decision‐support mechanism  based on system‐based usage logs, combined  with data collected from qualitative  measurements.     by systematically and periodically mining the  data collected from library servers, click‐through  tracking script, and electronic resource vendors,  usage patterns are identified and studied.  statistical reports provide evidence for collection  development, resource allocation, budget and  funding requests, system performance, and it  capacity planning. in addition, log analysis can  also reveal unusual activities and potential  threats to network security. from the log entries,  we have observed our website was cloaked by  massive and exploitative web crawlers from  google and other search engines. at times, we  are alerted by attempts of unauthorized access  and excessive downloading. the warnings lead  us to take corrective actions before problems  further escalate. such practice further fosters a  culture of evidence–based assessment on services  and resources in the library.     in a networked academy, libraries, together with  their parent organizations, are striving to serve  their less‐visible, but more diverse customers  with ever‐growing virtual resources.  methodically monitoring and studying network  traffic and usage statistics lay a foundation to  better serve our users with adequate digital  resources and reliable it infrastructure.       evidence based library and information practice 2006, 1:1      126  developing and validating a palliative  care subject search filter  r. m. sladek1, j. tieman1, b. fazekas2, a.  abernethy3, d. currow1  1. flinders university, australia  2. southern adelaide palliative services, repatriation  general hospital, australia   3. duke university medical center, usa    background  efficient access to relevant information underpins  efforts to translate research evidence into  practice. palliative care clinicians and researchers  face many informational challenges including the  diverse areas in which palliative care information  may be published and a lack of descriptiveness  and consistent terminology to support searching.     objective  to formulate and evaluate palliative care search  filters for use in the general medical literature.     methods  four general medical journals (bmj, lancet,  annals of internal medicine, and jama from  1999 to 2001) were handsearched by palliative  clinicians and researchers to identify articles and  items relevant to palliative care, forming a ”gold  standard” reference set (gs). a series of search  strategies comprising mesh terms and textwords  were created for use in ovid medline (om) by an  expert librarian and palliative clinicians and  researchers.   retrieved references from the om searches were  compared to the gs references using a specially  written computer program that automatically  calculated the sensitivity, specificity, accuracy  and precision of the om searches.     results and conclusions  773/20501 (3.8%) relevant articles were identified  by hand‐searching. a master search combining  nine mesh descriptors with three textwords  achieved 45.4% sensitivity, 99.3% specificity, 73%  precision and 97.3% accuracy. efforts to increase  the sensitivity by modifying three relevant  published but unvalidated searches did not  improve the yield, except in one case which  resulted in an improved sensitivity of 56.9%, but  offset by reduced specificity (92.1%), precision  (22%) and accuracy (90.8%). we confirmed that  palliative care literature is difficult to identify in  the general literature. whilst our filter represents  the best trade‐off between sensitivity, specificity,  accuracy and precision, the sensitivity is  unacceptably low. further research, such as  frequency analysis of textwords and mesh  terms, is required to increase the sensitivity of  searching.       developing library marketing  strategies based on statistics  yoo‐seong song  university of illinois at urbana‐champaign, usa    the university library at the university of  illinois at urbana‐champaign consists of over 40  departmental libraries with more than 10 million  volumes, making it the largest public university  library and the third largest among all university  libraries in north america. as a departmental  library within the university library, the  business & economics library (bel) primarily  serves over 3,500 students and faculty at the  college of business for their research needs.     starting 2002, the bel sought to reshape its  image from a “house of books” to an electronic  business information center. the author  conducted extensive research to collect data on  user experiences and expectations of library  services and developed marketing strategies to  promote the bel’s services. the author sought to  assess the effectiveness of information literacy  and different patterns of information seeking  behavior among business students. in 2003 and  2004, two surveys were taken to study how  business students perceived library services in  terms of electronic library services and library  instruction. based on the results, the author  formulated marketing strategies to promote a  new image of the bel through partnership with  other campus units and relationship marketing to  faculty members. this study illustrates how data  and statistics on library services can be applied to  formulating and implementing marketing  strategies. methods of evaluating the impact of  those strategies will also be discussed        e‐learning versus workshops to teach  critical appraisal to health professionals: a  randomised controlled equivalence study  linda ward  university hospitals of leicester nhs trust, uk    health professionals need skills in critical  appraisal to interpret the research evidence base  and implement results. research studies have  evidence based library and information practice 2006, 1:1      127 made explicit links between evidence based  practice (ebp) and ebp education, although it is  unknown which educational methods best  promote ebp. since 2000 university hospitals of  leicester (uhl) nhs trust libraries have  offered critical appraisal workshops to practicing  health professionals. these packages are also  provided in an interactive elearning format,  using the trust virtual learning environment  (vle).     this presentation will illustrate the content of the  modules and the approach that has been taken to  evaluating the effectiveness of elearning to teach  research methods and critical appraisal to health  professionals. the methodology of the research  will be presented and the opportunity for  contributions from the delegates will be  welcomed.     study design  randomised controlled equivalence study.     objectives  to test   • the null hypothesis of non‐equivalence  i.e. that there are educationally  significant differences in learning  between groups   • change from baseline knowledge, skills  or behaviour   • relationships between learner  characteristics and learning outcomes    intervention  three on‐line elearning lessons cover research  methods, critical appraisal and introductory level  statistics. the package includes graphics,  interactive features and self‐tests. access is via  the trust vle. comparison: six face to face  group workshops offer practical experience in  understanding research methods and critical  appraisal techniques; interpreting the  methodology and results of typical published  research studies     outcomes  analysis of pre‐ and post‐test validated  questionnaires testing knowledge, skills and  behaviour will inform the development of the  teaching packages in the most effective way. this  might be exclusively on‐line, exclusively via  group workshops, or a combination of both. for  the libraries, for the trust and for future course  participants this will be a major benefit.        evidence based librarianship in the  humanities: an application for second  language learners  glenna westwood  university of lethbridge, canada    in this presentation glenna westwood will  discuss the practice of evidence based  librarianship (ebl) from the perspective of a  humanities librarian. she will present the results  of the first phases of a research program  investigating the information needs of second  language learners. while the research was  originally designed to answer questions arising  from her own professional practice, the findings  contribute to the scholarship of ebl and  challenge some of our basic perceptions of  academic libraries.     ms. westwood will talk about her current  research experience applying an ebl model to  investigate the information needs of second  language learners. her hypothesis is that  academic libraries fail to adequately meet the  information needs of second language students  enrolled in academic institutions and in  particular, the needs of students in ‘lower level’  or foundation courses. beginning with a study of  her own institutional history and practice, ms.  westwood continued her research during a  year’s study leave at the language school of the  university of guanajuato, guanajuato, mexico  from july 2004 to july 2005. she will discuss her  work in mexico and share the results of a follow  up user survey conducted at the language  school self access centre in march 2005. the  presentation will conclude with some ideas for  further research and their implications for  academic libraries in the 21st century.       escholarship as socio‐technical  change: theory, practice and praxis  susan p. williams1, fides datu lawton2  1. school of business, university of sydney, australia  2. the library, university of technology, sydney, australia    developments in electronic publishing in higher  education are leading to significant changes in  the information supply chain and institutional  arrangements for the creation, organisation,  dissemination and preservation of scholarly  communications such as journal papers,  conference papers, working papers and other  scholarly information products. despite the rich  evidence based library and information practice 2006, 1:1      128 and growing literatures about scholarly  epublishing there is still much uncertainty about  its impact on the social dynamics of learning, on  universities and on the providers and consumers  of scholarly publications.     much of the current literature focuses on  technical issues such as: the design of systems for  storage and retrieval; the development of tools  for resource discovery, user authorisation and  authentication; the development and selection of  standards for resource description, encoding,  interoperability and digital preservation as well  as on publishing issues such as the changes to the  peer review process. whilst this provides  important technical guidance to the designers of  epublishing systems it is less helpful in  translating these systems into useful services.  further, there are few longitudinal studies  assessing how these systems are being adopted  and accepted by content providers and  consumers.     in this paper we present a program of critical and  evaluative research to investigate scholarly  epublishing systems in use and over time. the  aim of the research is to investigate the impact of  epublishing on the work practices of individuals,  the responsibilities of professional groups and  institutional structures, policies and practices.     the paper is structured as follows. first we argue  for socio‐technical theoretical perspectives that  make the work of scholarly epublishing visible.  second, to assist us in understanding the  interplay of theory and practice, we discuss the  need for longitudinal studies and participatory  research designs involving academic and  practitioner researchers. these support the  development of evidence‐based policies and  practices and our understanding of scholarly  epublishing praxis.       what can students’ bibliographies  tell us? ‐ evidence based information skills  teaching for engineering students  fei yu, jan sullivan, leith woodall  university of queensland, australia.    in order to target the needs of the students, and  to deliver the information skills workshop more  effectively, bibliographies submitted from 1st  year engineering, and 2nd year and 4th year  chemical engineering students’ project report  were analysed. by examining and comparing  bibliographies, evidence of the students’  strengths and weaknesses in locating, retrieving  and citing information would be found. the  bibliographies were analysed for: the number of  items cited; the variety of items cited and the  citation style. the topic of the project report was  also looked at to see the relationship between the  topic and the items cited. the results showed that  there was a clear pattern that higher year  students cited more items in total than lower year  students did in their bibliographies. second and  4th year engineering students cited more books  and journal articles than 1st year students, but  websites were still used extensively by all three  groups of students. some 1st year design project  topics students used the internet resources more  than other sources of information. students from  all year groups had difficulties with citation style.        poster abstracts    contribution of checking reference lists to  systematic reviews  t. armour, o. dingwall, m. sampson  chalmers research group, children’s hospital of eastern  ontario research institute, canada    background  handbooks advise systematic reviewers to check  reference lists (crl) of primary and/or secondary  studies to identify additional material for  systematic reviews. evidence‐based practice  would weigh the sensitivity and specificity of this  approach against the resource implications and  the evidence regarding the potential to protect  against or introduce bias into the review.    objectives  to systematically review the research evidence  regarding crl, examining these key questions:  what is the marginal yield of crl? is it most  productive to focus the crl search on certain  publication types? does crl a) help protect the  review against bias? or b) potentially introduce  bias? what is the evidence regarding cost‐benefit  of crl? what is the utility of using reference lists  for developing electronic searches, or validating  their recall?    methods  we searched the cochrane methodology  register, medline, library literature and lisa to  identify published or unpublished primary  evidence based library and information practice 2006, 1:1      129 research examining crl. case reports of the  yield from crl in a single systematic review  were excluded. we also determined the  prevalence of crl in a sample of 105 cochrane  reviews.     results  crl was reported by 86.7% of cochrane reviews  and was the most common search technique,  after searching electronic databases.     initial screening has identified three eligible  studies. preliminary findings suggest crl may  identify novel unpublished material.1,2  one study  observed positive bias within reference lists  (citation bias).3     discussion   recommendations to check reference lists are not  evidence‐based. the practice has not been  studied systematically and evidence is largely  from expert opinion and case reports (grade d).4     1 vickers aj, smith c. incorporating data from  dissertations in systematic reviews. int j technol  assess health care 2000; 16(2):711‐713.  2 royle p, waugh n. literature searching for clinical  and cost‐effectiveness studies used in health  technology assessment reports carried out for the  national institute for clinical excellence appraisal  system. health technol assess 2003; 7(34):1‐64.  3 gotzsche pc. reference bias in reports of drug trials.  br med j (clin res ed) 1987; 295(6599):654‐656.  4 phillips b, ball c, sackett d, badenoch d, straus s,  haynes rb, and dawes m. oxford centre for  evidence‐based medicine levels of evidence.  available at:  http://www.cebm.net/levels_of_evidence.asp      working with what works  janine chandler  australian institute of criminology, australia    criminology is turning to the evidence base in an  effort to inform policy about “what works” in  criminal justice interventions. a feature of this  evolution is the campbell collaboration, a sister  group to the cochrane collaboration, which aims  to make systematic research reviews widely  available. this poster examines the growth of  evaluation and evidence‐based research over  time; raises awareness of the campbell  collaboration; gives examples of the application  of evidence‐based policy from a practitioner’s  view; and discusses how this shift in practice is  affecting the work of the jv barry library at the  australian institute of criminology  rehabilitating the stroke collection  mary grimmond, sharna carter  central coast health, australia    does the central coast health library service  monograph collection adequately support the  health service’s specialist area of stroke as  identified by the nsw health greater  metropolitan clinical taskforce? this is the  question library staff set out to answer by  designing and applying a collection analysis  methodology.    stroke patients treated in the central coast  health specialist stroke units are seen by a  multidisciplinary team of clinicians including  doctors, nurses, physiotherapists, nutritionists,  speech pathologists, occupational therapists, etc.  a hospital library collection should provide  adequate resources to support all these specialist  areas.   one of the challenges in completing a collection  analysis on a specific subject area was  determining a means of locating relevant  resources within the collection, prior to  determining factors such as coverage and age of  these items. focus on ddc call numbers as a  starting point was soon found to be superficial,  with risk of overlooking relevant resources  dispersed throughout the collection. use of lc  and mesh subject heading tree searches was  expected to encapsulate a broader range of  resources. however, how could library staff be  confident that ‘relevant’ resources were still not  being overlooked?    it was decided that interaction with the  multidisciplinary stroke team members was an  essential part of the process. a questionnaire was  developed and dispersed amongst staff  representing different specialties on the stroke  team. this questionnaire listed the subject  headings identified by library staff as relevant to  treatment of stroke. team members were asked  to add their own ideas of relevant subject  headings, with consideration of broader skills/  knowledge required for the treatment of stroke.  the questionnaire also allowed for team  members to comment on types of materials and  specific titles they felt were useful / essential in a  basic collection on stroke management. use of  the questionnaire had several benefits –  education of the recipients about subject  searching, promotion of the library as a valuable  resource; and confirmation at a primary level of  the broad or narrower focus of subject searching  http://www.cebm.net/levels_of_evidence.asp evidence based library and information practice 2006, 1:1      130 and resource use within the library collection by  the stroke team.     results of the questionnaire will be used to assess  the collection of stroke resources and identify  titles for purchase to enhance the collection. it is  expected that use of this methodology can be  repeated amongst other specialty groups using  the library, to facilitate further planning and  decision making for effective collection  management.    today’s student is the registered nurse of  tomorrow:  the university of queensland  library’s role in preparing nurses to apply  evidence in clinical decision making.  jenny hall, stephanie nunan, ruth foxlee, lisa  kruesi  university of queensland library, australia    in recent years senior management have driven  the need to establish a culture of evidence based  practice (ebp) in queensland hospitals.  ebp was  first established in the medical arena. now in  2005 there is a growing emphasis for nurses to  use the evidence to support clinical decision  making.      information literacy is seen as a critical  prerequisite for evidence based practice.  in the  past, assumptions have been made about the  base level of nurses’ information literacy skills.  it  has been identified that without at least moderate  information literacy skills the ability to practice  using the evidence based model is severely  hampered.  this paper discusses a study which  examined the information literacy skills of  students entering the bachelor of nursing, a  newly offered program at the university of  queensland and those of graduate registered  nurses entering employment at the princess  alexandra hospital located in brisbane,  australia.  a pre test and post test survey was  used to ascertain base level of information  literacy and evaluate the intervention, a librarian  led workshop on searching and locating journal  articles, use of the university of queensland  catalogue, boolean operators and identifying  relevant health databases. the results of the  workshop indicated the students do not fully  understand major information retrieval  principles.  the paper will report on the study  process, report on related studies to date, the  issues identified and the means to raise  information literacy levels of nurses, in order to  adequately prepare them for the ebp  environment.       the human in human information transfer  – the wider implications of gatekeeping  yang lu  university of california at los angeles, usa    as humans inescapably embraced by social  networks, we exhibit a fundamental reliance on  interpersonal relations for the acquisition of  information. gatekeepers, who help link people  with unknown information, are the “humans” in  this interpersonal information transfer web. this  paper pulls together ideas from diverse  disciplinary research literatures to provide a  synthetic view of the gatekeeping process. it  reveals that gatekeepers arise in social groups by  different means, as noted and verified by  empirical research, are cultural certification,  informal nominations and by virtue of social  positions. constituting key elements of our  information environment, gatekeepers are very  influential in that they help to define social  reality in our mind and affect decisions we make.  however, while relying and trusting our  gatekeepers, we still need to be aware of the  biases and information gaps gatekeepers could  possibly have presented. more importantly,  gatekeeping research helps reveal the  information transfer mechanism through social  networks and prompts us to social network  analysis which can further promote our  understanding of human information behavior as  reflections of social relations, and the perennial  issue of information inequity.     getting to first base:  exploring evidence  based sexual health  melissa maclean  fpa health, australia    how does a library work together with a  telephone information and referral service to  provide up to date evidence based information to  health professionals and the general community?     fpa health (formerly family planning nsw)  runs a confidential information and referral  telephone service providing information on a  wide range of sexual health topics. this service,  fpa healthline, is staffed by clinical nurse  specialists who answer client’s questions via  telephone or email. the librarian is a very  important part of the question answering  process, providing current, relevant and timely  evidence based library and information practice 2006, 1:1      131 medical and nursing research for fpa healthline  nurses, which is used in conjunction with the  sexual health expertise of the organisation to  answer questions at hand and to create new  frequently asked questions (faqs) for the  website.     a case study will be explored using the example  of recent discussion in the medical press about  the possible effect of depot medroxyprogesterone  acetate (depo provera) on bone density. this  study will be used to illustrate ways in which a  small organisation providing a unique service in  nsw, can manage resources, knowledge and  information in order to provide an evidence  based information service that aims to be  instrumental in improving the reproductive and  sexual health of the people of nsw.     the smartal (st martin’s college  health students access to learning  resources whilst on placement) project  liz mailer  st martin’s college of higher education, uk    smartal is a practitioner based research  project centred in the library service of a  distributed higher education college in the  united kingdom aiming to improve the service  that health students on placement receive from  library services.     the smartal project was run by one part‐time  researcher over a 15 week period (december 2004  to march 2005).  a literature review was  conducted at the beginning of the project to  provide background information and awareness  of similar research in the area. the methodology  (questionnaires to students and interviews with  clinical support staff) was developed by  northumbria university’s hensal (health and  education northumbria students access to  learning resources) project, which reported in  2002. using an existing methodology enabled  smartal to be run within the time scale and  triangulate the experience gained by the  hensal project team. it also enabled  comparisons to be made with the findings of the  hensal project to determine national patterns  and trends.  the questionnaire data was analysed  using spss software.    the findings of the smartal project identified  various barriers faced by placement students in  accessing learning resources1. particular  difficulties concerned access to a library for  students working away from a large hospital,  access to networked computers and study space  and inconvenient library opening hours. the  project report will make recommendations to the  college and to the local nhs strategic health  authority with a view to providing an improved  service to placement students from the college  library and the nhs libraries.      the research has been supported by the college  sabbatical leave scheme, which for the first time  in 2004 invited support staff to apply for funding  alongside academics. library services was keen  to take advantage of this opportunity since  library‐led projects and research will contribute  to the local evidence‐base to support the  implementation of the library services strategic  plan 2005‐2009.     1 learning resources covered printed and electronic  books and journals, web‐based information, it  and study space.    if a client asks about evidence‐based  practice, what does it mean for you?  andrew meier  centre for automotive safety research, university of  adelaide    the evidence‐based movement continues to  spread across sectors ‐ from health to areas such  as education, criminology and transport. this  means that information professionals serving in  an increasing number of sectors need to be aware  of not only the growing importance of the  concept but the practicalities involved in  providing information on interventions  evaluated with higher methodological standards.  even more so given that in some sectors clients  may increasingly be requesting information on  methodologies, the actual subject of the  experiment being of secondary importance.   information professionals areas such as health  are well aware of the importance of  randomisation and control groups in indexing  and retrieval but this is not likely to be the case  for many other sectors.     the issue of evidence‐based information retrieval  in selected sectors has been examined in the  literature, including for librarianship. these few  papers have highlighted the problems of  database indexing and locating older material of  high methodological standard. they are also a  few years old and these problems not only still  exist but continue to grow. material continues to  be catalogued by information professionals  evidence based library and information practice 2006, 1:1      132 unaware of that the methodology of the  intervention may be as important as the subject  of the study itself. urgent action to promote  awareness of evidence‐based concepts, and their  importance regardless of one’s place in the  information profession is required. this paper  reinforces these issues with reference to the  transport and transport safety sector.    using library stocktaking data to make  evidence‐based decisions about library  procedures and policies  ailsa parker  whitireia community polytechnic library, nz    a stocktake prior to a move to a new building  ensures that holdings on the library catalogue are  accurate and items on the shelves arranged  correctly. whitireia community polytechnic at  porirua, new zealand, followed this procedure.  regular stocktakes had been done, but some  investigation into best practice was required as  the stocktake was to be electronic rather than  manual. a loss rate was identified as well as  problem areas, and evidence gained from the  stocktake was then appraised and evaluated in  terms of other data from within the library  system e.g. the previous stocktake, item loan  records and intra‐campus loan data. results were  benchmarked against other academic libraries,  although new zealand data was difficult to  obtain. a set of recommendations and best  practice guidelines was then drawn up. probably  one of the most useful sources of benchmarking  data and guidelines was theft and loss from uk  libraries: a national survey. this 1992 paper from  the crime prevention unit series by john  burrows and diane cooper, details definition,  frequency, extent and nature of loss from british  libraries.     the library is now in its new building and the  results of any future stocktake will need to be  examined in terms of changes to the physical  layout of the library e.g. non‐library staff can exit  the building without going through security  gates, the staff information desk is now a  considerable distance from the security gates and  users have access to an external deck. a separate  maori collection has also been created. in  addition, recent well‐publicised thefts from new  zealand libraries has generated some literature  on the subject of stocktaking and theft, so this can  also be incorporated into future benchmarking  and planning.    100 searches: refining the process of  identifying articles for systematic reviews  m. sampson, j. mcgowan, k. shojania  chalmers research group, ottawa health research  institute, canada    objective  identifying the appropriate studies for inclusion  in systematic reviews is important for the both  the completion of the initial review and well as  for carrying out updates.  this poster presents an  iterative process involving close collaboration  between librarians and clinical experts.    methods  a team of three librarians and one clinical expert  worked together to update a large project of  related systematic reviews. 100 different medline  search strategies were developed.  different  searches were created to focus on specificity and  sensitivity. search performance was tested using  known items. the search results were imported  into a web‐based systematic review software  database and the results were screened.  an  initial consensus double rating established inter‐ rater consistency. subsequent screening was  undertaken by one librarian and was confirmed  by clinical expert review.    results  including librarians in the screening process  allow for cross‐topic consistency and the  identification of false hits. as well, the librarian’s  knowledge of the subject was increased and this  assisted in the refinement of the search strategies.  close collaboration with a clinical expert allowed  for the identification of known articles and  responsive follow‐up to search specificity results.    conclusions  the results of this project will allow for the  development of a searching and screening  process for completing and updating systematic  reviews. iterative involvement of librarians in the  searching and screening process may be  particularly effective in compound systematic  reviews involving a large number of searching  elements.    evolution of the practice of evidence‐based  dentistry sessions at westmead hospital –  an important role for medical librarians  ratnes singham, linda mulheron  westmead hospital library, australia    evidence based library and information practice 2006, 1:1      133 evidence–based practice program (ebp) was first  introduced to the dental students at sydney  university in 2000. this subject was first  introduced in their curriculum so that they are  able to look for evidence in their clinical practice  of dentistry.     the reference and reader services librarian’s  role (from westmead hospital, a teaching  hospital of sydney university) to this program  was to participate in the group tutorial by  facilitating the students searching strategies.    each session consisted of 3 tutorials. during the  first half of the tutorial, the tutor would try and  get the students to identify a clinical problem  from which they needed to develop and refine a  search question. the librarian became involved  in the second half of the tutorial in showing the  students to search the various resources such as  medline, embase, ebmr databases. search  techniques involving subject headings versus  keywords, boolean logic and the use of  maximum precision and maximum recall filters  for retrieving articles on therapy, etiology,  prognosis, diagnosis, clinical trials and  systematic reviews.     after its first year of implementation of the  program, the supervisor has reported positively  to the university on the program and its great  value to the students. this is evidenced in the  fact that the librarian has continued in this role  as facilitator for the last 4 years. these sessions  have further evolved and now the students are  encouraged to first formulate a well structured  question based on their clinical situation using  the pico model. in the pico model p represents  patient or problem, i for intervention, c for  comparison, and finally o the outcome(s) of  interest.    these ebp sessions, apart from introducing the  students to life long learning skills, was also  meaningful to them as they were encouraged to  look for evidence to back their clinical treatment  of their patients.    evidence‐based medicine: a review of  medical librarian’s role  lisa song  hong kong baptist university library, hong kong    evidence‐based medicine (ebm), emerged into  the medical practice for over ten years, is about  the integrating individual clinical expertise and  patient value with best available research  evidence. practicing ebm involves the  fundamental skill of finding, critically appraising  and applying the evidence to the clinical decision  making.    ebm extends medical librarian’s role beyond  identifying and locating the literatures to  involvement in clinical practicing and teaching  quality information filtering and critical appraisal  of medical literature. these activities require  librarians to acquire new knowledge and develop  new skills. therefore, librarians need to rethink  their roles in providing medical information.  medical librarians have the skills and are well  positioned to respond to ebm’s information  needs as trainers or mediators. however, there  are many obstacles and challenges for medical  librarians in ebm practice. increasing literatures  state that continuing education should always be  a part of the professional life because librarians  are facing excellent opportunity to develop and  expend their roles.    in this poster, i will represent why the librarians  are important in ebm practice, and illustrate the  medical librarian’s role in the areas of ebm  teaching, learning, clinical practice and medical  research. the obstacles for the librarians in ebm  practice will also be identified.     a health evidence support service for  health professionals  caroline storer  bradford health informatics service, uk    a health evidence support service (hess) has  been available to all health service staff in the  bradford nhs health partnership for three  years. hess was implemented as an extension of  the existing library and information services to  work proactively with clinical and healthcare  teams in primary and secondary care to make  more effective use of health evidence obtained  from research.     the key roles of the health evidence support  specialist are to package evidence into easily  accessible formats and to match individuals with  relevant information they are unaware of but  likely to require. a number of services are  provided in order to achieve this and ultimately  to get evidence into practice, these are listed  below:    • weekly new guidance bulletins  • an information request service  evidence based library and information practice 2006, 1:1      134 • tool kits for evidence‐based practice  (intranet and hard copy)  • hands on evidence‐based practice  teaching sessions  • summaries of evidence for education  events    the health evidence support specialist works  closely with research partners within the health  partnership. an evaluation project using  questionnaires and focus groups is currently  ongoing. initial feedback suggests health  professionals have welcomed the service and  find it extremely useful. the health evidence  support service was a new post both to northern  and yorkshire and we believe country wide.  feedback suggests the post has ensured that  relevant evidence‐based information reaches  frontline staff. as a result of the success of this  post, another health evidence support specialist  has been appointed at another health community  in west yorkshire.    tool kits for evidence‐based practice   caroline storer  bradford health informatics service, uk    in order to fulfil clinical governance  requirements, all staff require accessible  information tailored to their area of practice.  many frontline staff do not have the time,  confidence, skills or knowledge to learn about  and implement evidence‐based practice or to  search for information. the bradford health  informatics service library wanted to make  specialty specific evidence‐based information  readily available in all areas of practice.  a  resource pack was produced to meet these  identified needs.    the tool kits for evidence‐based practice have  been provided for primary and secondary care,  higher education libraries and hospices. the  content is also available on trust intranet sites. in  undertaking this project, we hoped to achieve the  following:    • a resource, easily accessible to all health  care staff.  • increase awareness of traditional and  electronic evidence‐based practice  resources available.  • increase staff confidence in using  information and their ability to question  practice and link it to delivery of care.  • promote use and understanding of the  intranet/internet.  • inform staff of key points of contact  internally and externally who could  help with further development of  evidence‐based practice and research.  • ensure care provided is always based  on best evidence.    evaluation of the project within acute trusts has  shown that staff are aware of the tool kit and, in  most cases they are placed near a computer  where all staff have access to them. in some areas  the tool kits are used at induction. the  introduction of the tool kits has led to an increase  in requests for information from the libraries.  this is an ongoing project and the updating of  the tool kits is time consuming and it is hoped  departments will be able to update their own tool  kits from the intranet version. further evaluation  of the project is planned over the next few  months.    reflections on how evidence‐based  information practice can be applied in an  evidence‐based health care centre  catherine voutier, anne parkhill  centre for clinical effectiveness, monash institute of health  services research, australia    this poster will explore ways in which evidence‐ based information/library practice (ebip) can be  incorporated into the work practices of  information professionals at an evidence‐based  health care centre (ebhc). does this influence  the way in which we work? does knowledge of  ebhc principles help in the practice of ebip?     while evidence based practice at ebhcs has been  applied for a number of years, ebip is quite new.  information professionals working in ebhc  centres have a relatively firm grasp of the  principles and practices evidence based practice.   these include five steps which can be put simply  as: ask, search/retrieve, appraise, implement  and review. traditionally, information   professionals work in the first two, and  sometimes the third, step. ebip takes on ebhc  attributes and also includes that of evidence via  experience, collection and application of valid  statistical data, and user feedback.  we ask “how  can this be implemented in current practice?”,  and “are we already doing this without realising  it?!”      as information professionals at the monash  institute of health services research we  investigate best practice for our main information  activities; search strategy development, retrieval,  evidence based library and information practice 2006, 1:1      135 current awareness and education and some  implementation and review.   if it is  advantageous for any organisation that all its  employees work at best practice levels and  regularly evaluate their performance, these  reflections will benefit the implementation of  improved workflows and the review of our  services currently collected in regular user  surveys. evidence summary   editors view the continuous publication model as a satisfactory alternative for open access lis journals   a review of: cirasella, j., & bowdoin, s. (2013). just roll with it? rolling volumes vs. discrete issues in open access library and information science journals. journal of librarianship and scholarly communication, 1(4). http://dx.doi.org/10.7710/2162-3309.1086   reviewed by: richard hayman assistant professor & digital initiatives librarian mount royal university calgary, alberta, canada email: rhayman@mtroyal.ca   received: 10 jun. 2014   accepted: 19 aug. 2014      2014 hayman. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract    objective – to understand the prevalence of, motivations for, and satisfaction with using a rolling-volume publishing model, as opposed to publishing discrete issues, across open access academic journals in library and information science.   design – a 12 question survey questionnaire.   setting – english-language, open access library and information science (lis) journals published in the united states of america.   subjects – a total of 21 open access lis journals identified via the directory of open access journals that were actively publishing, and that also met the authors’ standard of scholarliness, which they established by identifying a journal’s peer-review process or other evidence of rigorous review. based on responses, 12 journals published using discrete issues, while 9 published as rolling volumes or as rolling volumes with some discrete issues.   methods – in late 2011, the study’s authors invited lead editors or primary journal contacts to complete the survey. survey participants were asked to identify whether their journal published in discrete issues, rolling volumes, or rolling volumes with occasional discrete issues, with the latter two categories combined as one for ease of results analysis. survey logic split respondents into two groups, either discrete-issue or rolling-volume. respondents in both categories were posed similar sets of questions, with the key difference being that the questions directed at each category accounted for the publication model the journals themselves identified as using. editors from both groups were asked about the reasons for using the publication model they identified for their journal: within the survey tool, authors provided 16 potential reasons for using a discrete-issue model, and 13 potential reasons for using a rolling-volume model. respondents from both groups were asked to mark all reasons that applied for their respective journals. the survey also included questions about whether the journal had ever used the alternate publishing model, the editor’s satisfaction with their current model, and the likelihood of the journal switching to the alternate publishing model in the foreseeable future.   main results – the authors collected complete responses from 21 of the original 29 journals invited to participate in the study, a response rate of 72%. for the 12 journals that identified as using discrete issues, ease of production workflow (91.7%), clear production deadlines (75.0%), and journal publicity and promotion (75.0%) were the three most common reasons for using a discrete-issue model. for the nine journals using rolling volumes, improved production workflow (77.8%), decreased dependence on production deadlines (77.8%), and increased speed of research dissemination (66.7%) were the three most common reasons cited for using a rolling-volume model.   findings show that overall satisfaction with a journal’s particular publication model was a common factor regardless of publishing model in use, though only the rolling-volume editors unanimously reported being very satisfied with their model. this high satisfaction rate is reflected in editors’ positions that they were very unlikely to switch away from the rolling-volume method. while a majority of editors of discrete-issue journals also reported being very satisfied or somewhat satisfied with their current model, the mixed responses to whether they would contemplate switching to the alternate model suggests that awareness of the benefits of rolling-volume publishing is increasing.   conclusion – researchers discovered a greater incidence of rolling-volume model journals with open access lis journals than anticipated, suggesting that this is an area where additional research is necessary. the relative newness of the rolling-volume model may be a contributing factor to the high satisfaction rate among editors of journals using this model, as journal editors are likely to be more deliberate in selecting this model over the traditional discrete-issue publishing model. workflow and production practices were identified as key characteristics for selecting a publishing model regardless of the model selected, and therefore this is another area in need of further investigation.     commentary   this study is timely, especially as open access, copyright, and intellectual property considerations all present new challenges to existing models of traditional academic publishing. the rolling-volume publication model, also known as continuous publication, is a relatively recent trend, but is very relevant within the modern context of electronic publishing and born digital scholarship. various scholarly publications have adopted the rolling-volume model, including plos one, the bmj, and all royal society publications, to name a few. however, there is very little scholarship studying the adoption of this model at disciplinary levels or across open access publishing as a whole, and what few publications do exist primarily take the form of editorials or rely upon anecdotal evidence whereby individuals present their own experience with using a continuous publication model (e.g., duriez, 2013).   the authors of this study provide a concise explanation of the inclusion criteria, such as english-language lis publications from the united states, and these criteria sufficiently limit the scope of the study to permit a thoughtful analysis. the authors readily admit that these limitations mean that readers should not generalize these findings to other open access publications, so instead they suggest numerous opportunities for further research. there is no way for readers to identify specifically which journals were included in the study, nor whether the authors identified all journals meeting the inclusion criteria. however, since the study aimed to discover motivations and reasons for selecting a particular publishing model, thus ensuring that participants were able to respond confidentially, this ultimately provides for a more fulsome discussion of the topic under consideration.   it is worth noting that inviting individual editors to represent their journal as a whole does not necessarily provide an accurate picture of that journal’s overall experience with a publication model. instead, this information may only represent that editor’s perspective of his or her journal at the time of data collection. for example, the question addressing satisfaction with the current publishing model asks for the editor’s opinion, and the authors report these results as the individual editor’s level of satisfaction, not the overall level of satisfaction as might be expressed by the entire journal editorial team. some clarification of these questions would help increase the validity of the study (glynn, 2006), as it is unclear whether an editor’s survey responses represent his or her personal opinions or are the position held by the journal’s full editorial team. the article could be strengthened by statistical analysis, rather than just comparison of raw results, as this would help clarify whether differences noted by the authors are statistically significant. the survey instrument is included as an appendix.   overall, this research is an important step in filling the gap in the literature regarding use of the rolling-volume publication model. the information reported will be beneficial to editorial teams and publishers who are considering adopting this model, whether for existing publications or for new startup journals. the evidence in favour of adopting one or the other model is particularly impactful, as workflow implications and production timelines are pressing considerations for all publications, no matter which publication model they currently use. those interested in scholarly communication or who provide expertise and guidance for open access publishing endeavours at their respective institutions will also benefit from this research.   references   duriez, h. (2013). 350 years at the cutting edge of scientific publishing the royal society moves to continuous publication. insights: the uksg journal, 26(2), 190-197. http://dx.doi.org/10.1629/2048-7754.61   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154   evidence based library and information practice evidence based library and information practice 2010, 5.4 96 evidence based library and information practice evidence summary learning through reflective writing: a teaching strategy a review of: sen, b. a. (2010). reflective writing: a management skill. library management, 31(1/2), 79-93. reviewed by: kristen l. young liaison services librarian a. alfred taubman health sciences library, university of michigan ann arbor, michigan, united states of america email: kristeny@umich.edu received: 1 sept. 2010 accepted: 11 nov. 2010 2010 young. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to explore student thought on both reflection and reflective writing as a process, and to analyze the writing by the application of clearly defined and identifiable outcomes. design – mixed method approach consisting of a qualitative analysis of 116 written reflections from master’s level students as well as a quantitative statistical analysis. setting –the university of sheffield masterslevel librarianship program’s course inf6005, “management for lis.” subjects – of the 31 students registered the course during the 2007-2008 academic year, 22 (71%), allowed their reflections to be used for the purposes of research. of these, 7 students identified themselves as male, and 15 were female. all students included were over 21 years of age and had previous library experience, with varying degrees of management experience in supervisory roles. not all supervisory experience was gathered within the library domain. methods –a total of 116 reflective journal entries were submitted by the participating students during the eight month period from october 2008 to may 2009. in order to identify themes, qualitative analysis was applied to the reflective writing responses. descriptive statistics were also applied to test the hypothesis, illustrate the relationships between reflective writing and outcomes, and locate identifiable outcomes. main results – practising reflection demonstrated benefits for individuals and mailto:kristeny@umich.edu� evidence based library and information practice 2010, 5.4 97 groups both in and outside of the workplace. on the whole, individuals gained the most from reflection and saw it in the most positive light when it was practised as a daily activity. quantitatively, when students began to master the practice of reflection, they demonstrated an increase in their ability to learn and an overall improvement of self-development and critical thinking skills, and gained a defined awareness of personal mental function. when decision making became easier, students understood they had begun to master the art of reflective practice and analytical reflective writing. qualitatively, when the students’ reflections were assessed, ten different themes emerged: (1) nature of reflection (2) reflection seen as useful in providing support for a career and professional development (3) reflective writing – benefits (4) reflective writing – potential in future employment and workplace (5) encouraging others to use reflective practice (6) reflecting positively (7) reflection applicable to both individuals and groups (8) reflection in support of personal awareness (9) exploration of different methods of reflection (10) difficulties in focusing enough to be able to reflect deeply conclusion – reflection is a skill that can be practised and developed. initially, not all students enrolled in the class and participating in the study knew what reflective writing was or what it entailed. students were given support to educate them in this area. support included academic reading, lectures, reflective writing workshops and an overall opportunity to develop their skills further. reflective writing was demonstrated to have a very positive relationship with several key outcomes. the areas impacted include academic learning, self-development, and critical review, with key outcomes including an increased awareness of personal mental function and increased support for decision making, as well as empowerment and emancipation. the clearest benefit was represented when students wrote about their analytical reflections. commentary the purpose of the study was to gather students’ thoughts on the process of reflection and reflective writing and analyze the thoughts in relation to identifiable outcomes. students quickly demonstrated a need for self development and recognized the value of developing skills in working as a team in relation to reflection. the author of the study acknowledges that reflection in any form is not easily understood, and is perhaps even more difficult to teach. the study is well written, provides a comprehensive review of the literature, and clearly demonstrates the benefits of reflective writing after a short period of time. those benefits include but are not limited to: an increased sense of awareness of self and others, and how that relates to work situations. both reflection and reflective writing play an important part in how we develop personally and professionally. one methodological strength of the original article is that a second researcher is used to independently assess “a sample of the written statements,” which increases the study’s reliability (p. 84). however, it would have been good to know how large the sample that had been independently assessed was. this paper differs from previous literature by presenting statistical evidence “to confirm the relationships between reflective writing and a range of potential outcomes” (p. 79). it should be seen as a valuable asset within the field. although the analysis provided results that were statistically significant across seven of the eight outcomes tested, the evidence from student reflections and further analysis contained in table 4 should be read with some evidence based library and information practice 2010, 5.4 98 caution. as with all self-reporting and further analysis, personal bias may have influenced the results. this is particularly possible given that the students were completing the reflective writing tasks as a part of assignments worth 30% of their overall assessment mark for that course (p. 80). this introduces the possibility that students may have written the tasks to be in line with what they anticipated the lecturers may have wanted to see. this topic could be of interest to all practitioners in the profession, whether new or experienced. after all, the practice of reflection is part of lifelong learning. / evidence based library and information practice evidence summary evidence based library and information practice evidence based library and information practice 2011, 6.1 99 evidence based library and information practice ebl 101 disseminating your research virginia wilson client services librarian, murray library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca received: 22 dec. 2010 accepted: 05 jan. 2011 2011 wilson. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. in terms of looking at all the steps of evidence based library and information practice, we're almost home! the final thing to consider once you have gone through the process is disseminating your work. whether you have used the evidence base to inform your practice or decision making, or you have conducted your own research study to do the same, disseminating the results is beneficial for a number of reasons. crumley and koufogiannakis (2002) state that "perhaps the largest obstacle in finding library research is that librarians generally do not publish their research" (p. 69). when thinking of dissemination, keep three things in mind: ease of comprehension, ease of access, and multiple forums. strive for accessible language and format, and consider publishing in an open access journal so that the information is freely available to your colleagues. in terms of multiple forums, there are several venues to consider when thinking about getting the word out. first up is publishing. why might you want to publish your own work on your own practice? an important reason is to increase the evidence base. in order to practice ebl, we need to have the knowledge base. and that means publishing research activities in peerreviewed scholarly journals. eldredge (2006) contends that we as a profession have "an obligation to ensure that results are communicated, even if these results are not dramatic or new, to build a more solid foundation to our knowledge base" (351). but publishing in scholarly journals is not the only way to get the word out. publishing online on blogs, wikis, and personal or professional websites, just to name a few online avenues, also serves to get the message out. another way to disseminate your message is by educational or professional development means. present a paper or take part in a poster session at a conference. run a workshop, session, or webinar based on the topic of your research. present your findings at a journal mailto:virginia.wilson@usask.ca� evidence based library and information practice 2011, 6.1 100 club, discussion, group, or any other gathering in a professional setting. the research that librarians undertake to inform their practice can be integrated into library management and policy in a few ways. making sure that the ultimate decision-makers in the organization are aware of the research activities, or conversely, aware of a lack of evidence for certain issues, can prompt administrators to go forward with an evidence based approach. presenting evidence either self-generated or found in the literature at library meetings or other internal venues can be the spark needed to push new ideas forward, and can contribute to an evidence based culture shift within the organization. and finally, personal communication is an excellent way to get the word out about your results and to share what you have found with colleagues. listservs and email work, as do social networking tools such facebook, twitter, wikis, and blogs. chapter 12 in evidence-based practice for information professionals: a handbook by crumley and koufogiannakis, fleshes out all of these examples and is a must-read if you are thinking of getting the results from the evidence based process out there. it is difficult to practice in a vacuum. and in evidence based library and information practice, just as we need to reach out to find the evidence, so does everyone else who is endeavouring to practice in that way. there needs to be evidence in the evidence base. i believe it is safe to assume that because you had a question you needed to explore, others have or will have the same question. eblip is a way to work collaboratively without even meeting face to face. sharing the evidence; making it available to inform practice, helps everyone in librarianship. now that i have finished taking you through the steps of ebl, next time i will embark on a new and exciting adventure in ebl 101. i will be looking at various research methods, qualitative and quantitative, and will be giving overviews of the methods including types of research studies for which they would be appropriate. first up, an overview of qualitative and quantitative research. references crumley, e. and koufogiannakis, d. (2002). developing evidence-based librarianship: practical steps for implementation. health information and libraries journal 19(2), 61-70. crumley, e. and koufogiannakis, d. (2004). disseminating the lessons of evidence based practice. in a. booth & a. brice (eds.), evidence-based practice for information professionals: a handbook (pp. 138-143). london: facet. eldredge, j. (2006). evidence-based librarianship: the ebl process. library hi tech 24(3), 341-354. / evidence based library and information practice microsoft word art_warner.doc evidence based library and information practice 2007, 2:2  55 evidence based library and information practice     article    measuring the effectiveness of queen elizabeth ii library document delivery  operations before and after the implementation of relais international’s enterprise  document delivery software      patrick warner  head of document delivery  queen elizabeth ii library  memorial university of newfoundland  st. john’s, nl canada  e‐mail: pwarner@mun.ca      received: 05 july 2006    accepted: 18 april 2007      © 2007 warner. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    abstract    objective ‐ to compare the performance of the queen elizabeth ii document delivery  operation before and after the implementation of relais international’s enterprise document  delivery software.    methods ‐ this paper employs methodology established in the association of research  libraries’ 1998 publication, “measuring the performance of interlibrary loan operations”  and repeated in arl’s “assessing ill/dd services: new cost‐ effective alternatives,”  published in 2004. in both studies, three measures were used to evaluate the efficiency of  document delivery operations: fill rate, turnaround time and direct costs. both studies offer  arl benchmark or mean scores for each efficiency measure. this paper compares queen  elizabeth ii document delivery (qeii/dd) scores for each efficiency measure with those  reported in both arl studies.    results ‐ data for the two periods under review, 1999‐2000 and 2004‐2005, indicate that  borrowing fill rates remained relatively stable, showing only a 3% drop in the latter year,  while lending fill rates showed a significant increase (11%). turnaround times for filled  qeii/dd borrowing returnable requests were faster on average by 4.2 days or 24%.  turnaround times for qeii/dd non‐returnable borrowing requests also show improvement:  a filled non‐returnable request was faster on average by 1 day or 12%. the average cost of a  mailto:pwarner@mun.ca http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:2  56  qeii/dd borrowing request has remained stable: $22.82 in 1999‐2000 and $22.61 in 2004‐ 2005. in contrast, the average cost of a qeii/dd lending request has increased slightly: from  $11.08 in 1999‐2000 to $13.12 in 2004‐2005.  .     conclusion ‐ both the implementation of relais document delivery software and the  delivery of returnables (loans) by courier between consortium members have allowed the  qeii/dd unit to post modest gains in both fill rates and per unit costs and more substantive  gains in turnaround time.     introduction    document delivery services at memorial  university of newfoundland comprises  interlibrary loan, intercampus loan and  document supply to distance users. this  paper will focus on the part of document  delivery services known traditionally as  interlibrary loan. interlibrary loan has both a  borrowing and a lending component.  borrowing refers to the practice of locating  and obtaining documents from other  institutions on behalf of students, faculty  and staff of memorial university. lending  refers to the practice of supplying  documents from memorial university  libraries to other institutions.     this paper describes a two‐part study  conducted at memorial university of  newfoundland’s queen elizabeth ii library  document delivery (qeii/dd) unit which  measured the efficiency of the operation  before and after the implementation of  relais enterprise document delivery  software. the ultimate aim of this study was  to test the hypothesis that the  implementation of relais enterprise  software would result in greater efficiency  in local document delivery operations. a  secondary aim of this study was to compare  the performance of the qeii/dd operation  with the performance of association of  research libraries in north america (arl).  in order to achieve these aims, an efficiency  study of the qeii/dd operation carried out  in 2001 was repeated in 2005 and the scores  compared to benchmarks reported for the  arl group. the study, reported in this  paper, adds to a growing body of literature  on performance measurement in interlibrary  loans and document delivery operations.     relais enterprise software was implemented  at memorial university as part of a growing  consortium of atlantic canadian university  libraries known as the east coast relais  consortium (ecrc). both the consortium  and the adoption of new software arose in  response to a demand for greater efficiency  both in resource sharing and in document  delivery following a decade of cuts to  journal subscriptions.  also contributing to  the demand was an increased availability of  both electronic discovery tools and  electronic journals which fuelled user  expectations of rapid access to a wide  variety of literature. relais software was  implemented at memorial in two phases:  relais lending (version 2.3) was introduced  in the fall of 2000, and relais borrowing  (version 4.1) was introduced in the fall of  2004. though the software has been fully  operational since fall 2004, there are  frequent upgrades and enhancements. the  most recent upgrade, to relais enterprise v.  2006, took place in december of 2006.     relais enterprise document delivery  management software employs a variety of  technologies and protocols to semi‐ automate the document delivery process.  relais enterprise uses control number,  author, title, and phrase searching to  automatically generate a z.39.50 (an  information search and retrieval protocol)  evidence based library and information practice 2007, 2:2  57 query to one or more library catalogues. the  system communicates with other document  delivery management systems through the  iso 10160/1 communication protocol,  through ‘generic script format’ (a cgi script  that allows data to be processed through a  form), as well through other non‐standard  means, such as e‐mail. the system allows for  the automatic creation of routing lists for  each request.  staff use free text e‐mail or a  series of ‘canned’ messages to communicate  with external libraries and local  patrons.  patrons can monitor the status of  requests via relais access web forms. a key  component of the system is the integration  of scanning software with the document  delivery management software which  allows for the seamless retrieval and  completion of requests. relais enterprise  also facilitates the delivery of documents in  a variety of formats, including ariel, fax,  electronic mail and post‐to‐web.     literature review    in 2001, thomas nisonger published a  review of literature evaluating interlibrary  loan and document delivery services in  which he stated, “the most frequently  evaluated issues are ‘speed,’ fill rate and  cost, yet a host of other questions have been  addressed: the cost effectiveness of access  versus ownership, user satisfaction,  concentration and scatter in the requested  materials, and the percentage of items  already held in the collection” (nisonger 1).   nisonger concludes that while a  “voluminous amount has been written”  only a “fraction of the published literature  reports the results of evaluative studies”  (nisonger 1). those interested in a  comprehensive review of publication  around performance measurement between  the years 1986 and 1998 should see stein  who points out that research in this area had  been “hindered by the lack of  standardisation of measurement devices  and definitions of terms” (stein 11).  for the purposes of this paper, the author  conducted a literature survey of articles  published between 2000‐2006 on the subject  of performance measurement and efficiency  and found a similar pattern to that outlined  by nisonger: a high number of articles  focussing on issues related to interlibrary  loan and document delivery services, but  relatively few articles reporting on  evaluative studies of those services  (nisonger 2). literature reviews published  by connolly (2000, 2001), gould (2000, 2001,  2002) and mcgrath (2002‐2007), in the  journal interlending and document supply,   highlight continued interest both in the  access versus ownership debate and in  resource sharing. these reviews also  highlight a growing body of literature on  automation, unmediated document delivery,  copyright, electronic resources, consortia, as  well as licensing and pricing as it relates to  the so‐called ‘big deals”  in recent years,  many articles have been published on the  subjects of open access, e‐books, digitization  and google.     mary jackson’s studies of interlibrary loan  and document delivery operations in north  america continue to generate evaluative  literature (jackson, 1998; and jackson, et al.  2004). a number of other authors around  the world have recently performed  benchmarking studies. these include fisher  in new zealand (2000), vattulainen in  norway (2000), søndergaard in denmark  (2001), stabler in the usa (2002), and bailey‐ hainer in the usa (2001 and 2004).  the  national resource sharing working group  reports on an australian benchmarking  study based on jackson’s ill/dd study  (2001). papers emanating from the  australian study include ruthven (2001)  and ruthven and magnay (2002).     between 2000 and 2006, both interlending and  document supply and the journal of  interlibrary loan, document delivery and  electronic reserves published articles on user  evidence based library and information practice 2007, 2:2  58 satisfaction related to the automation of  interlibrary loan and document delivery  operations. porat (2001) reflects on user  satisfaction and ill/dd automation at the  university of haifa. sutherland and wanat  (2000) discuss how automated statistics can  be used to monitor and improve ill/dd  customer service.  burke (2001) reports on  turnaround times for the commercial  document delivery service, dissertations  express.     elsewhere, efficiencies related to the  implementation of specific document  delivery technologies are discussed: weible  and robben (2002) report on the benefits of  ill/dd automation using prospero; tonn  (2003) investigates patron satisfaction  related to the implementation of illiad at  ellender memorial library; and connell  (2006) offers a comparison of turnaround  times between illiadʹs odyssey and ariel  delivery methods. among recent papers  that discuss the implementation of specific  commercial document delivery software  and technologies, only two focus on relais  enterprise: cornish (2000) and guadagno  (2005). both papers are descriptive rather  than evaluative, and the latter (guadagno)  was produced on behalf of the vendor. this  paper is the first to offer a quantitative  analysis of operational efficiencies resulting  from the implementation relais enterprise  software.    methods    this study uses methodology employed in  the association of research libraries’  (arl)1998 publication, “measuring the  performance of interlibrary loan  operations” and repeated in arl’s  “assessing ill/dd services: new cost‐  effective alternatives,” published in 2004. in  both studies, three measures were used to  evaluate the efficiency of document delivery  operations: fill rate, turnaround time and  direct costs. both studies offer benchmark or  mean scores for each efficiency measure  based on scores for arl libraries. this  paper will compare 1999‐2000 qeii/dd  mediated borrowing and lending scores for  each measure with the mean scores reported  for mediated ill/dd in the 1998 arl study.  a further comparison will be made between  2004‐2005 qeii/dd mediated borrowing  and lending scores for each measure with  the mean scores reported for mediated  ill/dd in the 2004 arl study. finally, this  paper will compare 2004‐2005 qeii/dd  scores with the mean scores reported in the  2004 study for arl high‐performing  mediated ill/dd. comparing 1999‐2000  qeii/dd scores for all three measures with  those for 2004‐2005 will show whether the  implementation of relais enterprise  software resulted in operational efficiencies.  comparing qeii/dd scores for both years  with the benchmarks for arl libraries will  show how the qeii/dd service compares, in  terms of the three efficiency measures, with  document delivery services in north  american research libraries.     fill rate  fill rate “represents the percentage of  requests successfully filled” (jackson 5). for  document delivery lending, fill rate means  the total number of returnables (loans) and  non‐returnables (photocopies) supplied to  external libraries. the lending fill rate is  expressed as a percentage of the total  number of items requested by external  libraries. for document delivery borrowing,  fill rate means the total number of  returnables and non‐returnables supplied to  the qeii/dd unit by external libraries or  document suppliers. the borrowing fill rate  is expressed as a percentage of the total  number of document delivery requests  submitted by patrons. in calculating the  number of requests submitted by patrons,  qeii/dd staff members exclude those  requests that are found to be available in our  local collection, a number which amounts to  evidence based library and information practice 2007, 2:2  59 between 15% and 20% of the total number of  requests submitted each year.    turnaround time  turnaround time is calculated as “the  number of calendar days from the date the  user submitted the request to the date the  library notified the user that the item was  ready to be picked up or the date that the  ill unit sent the item to the user” (jackson  51). this includes the initial processing time,  the time it took to obtain the item and any  post‐receipt processing. following arl’s  reasoning that filled document delivery  requests are the “product” of ill/dd units,  this study calculates the turnaround time of  filled request only, i.e. it does not include in  its calculations requests that could not be  filled or requests that users submitted which  were subsequently found to be held in the  local collection (jackson 32). furthermore,  this study focuses on the turnaround time  for borrowing requests only. qeii/dd  turnaround time statistics were calculated  on requests received during two periods:  february 2000 and february 2005. statistics  for february 2000 were manually generated.  statistics for february 2005 were generated  from the relais database using crystal  reports reporting software.     direct costs  direct costs are calculated using the  following categories: general operations  (material and supplies, printing, courier  charges, interlibrary loan charges, student  assistant salaries, reimbursements, external  contracts and operating leases), staff salaries,  software, hardware and equipment. the cost  of hardware and equipment is calculated at  25% of the replacement cost and is based on  a four year upgrade cycle. general library  functions such as collections development,  serials and acquisitions are excluded from  cost calculations. jackson makes the case for  excluding these costs as follows: “collection  development, acquisitions, and circulation  are necessary preconditions for an ill  service and would be a cost to the library  whether or not the library provided ill  services (jackson 5). wherever possible,  direct costs are assigned to either borrowing  or lending. this is easily done with some  costs, e.g. staff costs, courier charges, and  not so easily done with others. costs shared  between units—supplies, some equipment,  some software charges—were divided  proportionately between borrowing and  lending, based on the percentage of the total  number qeii document delivery requests  filled by each unit. that works out to be 59%  for lending and 41% for borrowing. all costs  are reported in canadian funds.    results    fill rate   tables 1 and 2 show that qeii/dd borrowing  fill rates remained relatively stable   between 1999‐2000 and 2004‐2005, showing  only a 3% drop in the latter year. borrowing  fill rates also remain very close to the mean  fill rates reported by arl in 1998 and again  in 2004.     qeii/dd lending fill rates, on the other hand,  show a significant increase (11%) between  1999‐2000 and 2004‐2005.  qeii/dd lending  fill rates are also considerably higher than  arl’s reported mean rates: 13% higher in        arl (mean)  qeii/dd  borrowing requests (filled)  85%  87%  lending requests (filled)  55%  68%    table 1. fill rate: qeii/dd fill rate in 1999/2000 compared to arl libraries (1998 study)  evidence based library and information practice 2007, 2:2  60     arl(mean)  qeii/dd  borrowing requests (filled)  86%  84%  lending requests (filled)  58%  79%    table 2.  fill rate: qeii/dd fill rate in 2004/2005 compared to arl libraries (2004 study)      arl (mean) # of days  qeii/dd # of days  turnaround time for returnables   16.9  18.0  turnaround time for non‐returnables   14.9  8.1    table 3. borrowing turnaround time: qeii/dd turnaround in 1999 compared to arl libraries (1998)      arl (mean) # of days  qeii/dd # of days  turnaround time for returnables   9.3  13.8  turnaround time for non‐returnables   6.1  7.1    table 4. borrowing turnaround time: qeii/dd turnaround in 2004‐2005 compared to arl libraries (2004)      arl (mean)   qeii/dd   cost of a borrowing request   $22.69*  $22.82  cost of a lending  request  $12.78  $11.08    table 5. direct costs: qeii/dd direct costs in 1999/2000 compared to arl libraries (1998)  *a conversion rate of 1.3613 was used to convert usd to cdn (arl 1988)      arl (mean)   qeii/dd   cost of a borrowing request   $27.44*  $22.61  cost of a lending  request  $14.53  $13.12    table 6. direct costs: qeii/dd direct costs in 2004‐2005 compared to arl libraries (2004)  *a conversion rate of 1.5687 was used to convert usd to cdn (arl 2004)      1999‐2000 and 21% higher in 2004‐2005. it is  notable that arl mean fill rates for lending  show only a 3% increase between the 1998  study and the 2004 study.    turnaround time   tables 3 and 4 show improved turnaround  times for filled qeii/dd borrowing  returnable requests during the two periods  under review, 1990‐2000 and 2004‐2005. a  filled request for a returnable was faster on  average by 4.2 days or 24%. turnaround  times for qeii/dd non‐returnable  borrowing requests also show improvement:     a filled non‐returnable request was faster on  average by 1 day or 12%.    direct costs   tables 5 and 6 show that the average cost of a  qeii/dd borrowing request has remained  stable: $22.82 in 1999‐2000 and $22.61 in  2004‐2005. in contrast, the average cost of a  qeii/dd lending request has increased  slightly: from $11.08 in 1999‐2000 to $13.12  in 2004‐2005.  when compared to the mean  for arl libraries, the unit cost for a  qeii/dd lending request in 2004‐2005 was  $1.83 or 10% cheaper.  evidence based library and information practice 2007, 2:2  61     unit  cost  fill rate  turnaround (#days)  returnables  turnaround (#days)  non‐returnables  mean for arl (14)  $24.39*  90%  5.4   7.6**  qeii/dd (04/05)  $22.61  84%  13.8  7.1    table 7. qeii/dd (2004‐2005) compared to arl best borrowers (2004): all measures  *a conversion rate of 1.5687 was used to convert usd to cdn (arl 2004)  **notably the mean turnaround time for non‐returnables for the 14 high‐performing borrowers was 22%  slower than the mean for all 58 arl units studied.      comparison with high performance borrowing   table 7 shows that qeii/dd borrowing per  unit request costs and fill rates compare  favourably with high‐performing arl  borrowing operations. turnaround times  compare less favourably. the following  tables offer a comparison between the  qeii/dd scores for each efficiency measure  and the scores of the 14 high‐performing  arl borrowing units.    discussion    the results show that the implementation of  realis enterprises document delivery  software has allowed the qeii/dd operation  to post modest gains in both fill rates and  per unit costs and more substantive gains in  turnaround times.  the results also show  that qeii/dd scores for all three efficiency  measures compare favourably with the  mean scores reported for arl libraries.    fill rates: lending statistics for the qeii/dd  are not consistent with arl statistics in all  areas. the qeii/dd lending unit posts much  higher fill rates when both the total number  of lending requests is lower than the mean  for arl libraries (11,927 as compared to  41,088 in 2004‐2005), and the ratio of  returnables to non‐returnables is higher (in  both years under review, the qeii/dd unit  filled slightly higher number of requests (7%)  for returnables than did arl libraries).    one factor that might explain the difference  in lending fill rates is that 75% of arl  libraries involved in the study place  requests through worldcat, oclc’s union  catalogue (jackson 45). jackson goes on to  say that many libraries do not regularly  update holdings on oclc and that  borrowing staff do not always check oclc  union lists before placing requests (jackson   45‐46). in contrast, the qeii/dd lending unit  fill rates may benefit from a library policy  which dictates regular updating of the  library’s holdings in amicus, the national  library of canada’s union catalogue,  resulting in more accurate holdings  statements and fewer cancelled lending  requests. as well, ecrc members search  each member’s catalogue before placing a  request.       a second factor that might explain the  difference in lending fill rates is staffing  levels. in its 2004 study, arl reports the  mean number of filled lending requests per  lending fte staff as 10,297. for the qeii/dd,  the number of lending requests per lending  fte staff is 7,016. in this regard, the  performance of qeii/dd lending  corroborates another statistically significant  relationship reported by jackson: “as total  lending staff increases, the lending fill rate  increases (jackson 40).    turnaround times: courier delivery is the  single most important factor contributing to  improved turnaround time of returnables  evidence based library and information practice 2007, 2:2  62 between consortium members.  approximately 18% of returnable requests  are supplied to the qeii library by ecrc  members, and turnaround times are  generally fast, e.g. requests supplied from  within the consortium are usually filled  within 48 hours. the mean turnaround time  for all returnables (13.8 days) is still slow,  however, when compared with the mean for  arl libraries (9.3 days).  the most  significant factor contributing to slow  turnaround for returnables is the time a  mailed item spends in transit. based on this  study, it is recommended that qeii/dd unit  move to courier delivery of all borrowing  requests for returnables.       given the possibility of increased  automation of workflow, including the  functionality to create routing lists and  automatically route requests to suppliers, a  greater improvement in turnaround time for  non‐returnables was expected. in reality,  however, the majority of borrowing requests  are still mediated. while most requests are  submitted using relais access web forms,  users still submit incomplete or abbreviated  bibliographic citations, i.e. requests which  require staff editing. also, because relais  software is configured to search at the issn  level and not the volume/issue level, staff  members will, whenever possible, check  holding statements in amicus before  submitting a request.       one development that may have a positive  effect on turnaround time is the  implementation of a virtual union catalogue  through the atlantic scholarly information  network (asin) portal. document delivery  requests from the asin portal will send a  full bibliographic citation from the relevant  catalogue or index to the relais database.  this will have a significant impact on the  amount of editing staff are required to do on  any one request and will facilitate automatic  routing of requests.      it is notable that in the most recent period  under review, february 2005, the relais  borrowing module was still a relatively new  implementation. a more recent in‐house  review of turnaround time for qeii/dd  borrowing requests, comparing the periods  october 1, 2004 to march 1, 2005 and  october 1, 2005 to march 1, 2006, shows  significant improvement in turnaround time  for non‐returnables: from 7.1 days to 4.7  days per request, an improvement of  approximately 30%. interestingly, there  were no major changes to workflow rules or  to the policies during this period. it is likely  that staff familiarity and confidence in the  system were contributing factors.      direct costs: staff salaries remain the single  biggest factor on cost per unit transaction  for both borrowing and lending requests. in  the 2004 arl study, jackson reports that  staff costs account for 58% of the cost of a  borrowing transaction and 75% of the cost of  a lending transaction (jackson 33). by  comparison, a cost analysis of qeii/dd  lending and borrowing transactions for the  year 2004‐2005 shows 81% of the cost of a  borrowing transaction and 66% of the cost of  a lending transaction are attributable to staff  salaries.    both borrowing and lending units each had  one less fte position in 2004‐2005 than they  did in the academic year 1999‐2000.  additional staff costs were incurred with  the hiring of two full‐time relais system  administrators, a portion of whose salaries is  charged to the qeii/dd budget. because  software charges and the salary costs for  system administrators are attributed  proportionate to the total number of filled  lending and borrowing requests, it is  possible that the unit cost for a lending  request is slightly inflated and the unit cost  for a borrowing request slightly  underestimated.        evidence based library and information practice 2007, 2:2  63 high performance borrowing operations  while the qeii/dd scores for the three  efficiency measures compare favourably  with the mean for arl borrowing units, it is  worth comparing qeii/dd scores with those  of low cost, high‐performing borrowing  operations reported in the 2004 arl study.  as stated earlier in the results section,  qeii/dd borrowing per unit request costs  and fill rates compare favourably with high‐ performing arl borrowing operations,  though turnaround times compare less  favourably. as noted above, a more recent  in‐house review of turnaround time for non‐ returnables shows a major improvement.  the local operation posted a significantly  better turnaround time than the mean  turnaround time reported for arl  borrowers. further improvements are  possible with increased automation and a  more streamlined workflow. a significant  improvement in turnaround time for  returnables is to be expected if the  recommendation of this study to implement  courier delivery of all returnables meets  with approval.      in arl’s 2004 study, jackson points to a  number of areas that may be exploited to  improve the efficiency of document delivery  operations. these include monitoring the  performance of lenders, posting articles on  secure web sites for users to access, and  increasing the use of user‐initiated services  (jackson 100). the qeii/dd unit is currently  working with other consortium members to  develop reports which will monitor the  performance of lenders. the east coast  relais consortium has also committed to  print‐to‐web article delivery by fall 2007.  print‐to web is new technology which  allows for a scanned document to be posted  to a web server. once notified by e‐mail that  the requested item is available, the user will  view the cover page online and then print a  single copy of the document. the user is  prevented from keeping an electronic copy.  since 2002, the qeii/dd has used cisti  source, the canadian institute for scientific  and technical information’s document  delivery service, to provide user‐initiated  document delivery services to selected  faculty and graduate students. cisti source  is fast, cost‐efficient and popular with users  (warner 217) in addition, memorial  university libraries are currently taking a  leading role in the development of the  atlantic scholarly information network,  which will see all atlantic canadian  university libraries using a single virtual  union catalogue and a single document  delivery system. this development will lead  to more user‐initiated document delivery  services of the kind jackson describes as  “direct consortial borrowing” (jackson 100).    conclusion    this paper confirms that that the  implementation of relais enterprise  software resulted in greater efficiency in  local document delivery operations. this  paper also offers evidence based on three  key measures: fill rate, cost and turnaround  time— that that qeii/dd performance  compares favourably with the performance  of association of research libraries in  north america (arl).    over the past six years, the queen elizabeth  ii document delivery unit has moved from  being a paper‐based, stand‐alone document  delivery operation to a semi‐automated  document delivery unit operating in a  consortial environment. the academic year  2004‐2005 was a transition year, i.e. the first  full year of operation using both relais  borrowing and lending modules. that 2004‐ 2005 was a transition year is corroborated by  the fact that there was a significant  improvement in turnaround times in 2005‐ 2006, an improvement which shows the  qeii/dd borrowing unit out‐performing the  arl borrowing operations documented in  jackson’s 2004 study.       evidence based library and information practice 2007, 2:2  64 further improvements in turnaround time  are to be expected in the next few years as  the degree of mediation in request  processing decreases and courier delivery of  all returnables is implemented. per unit  costs should also decrease when the east  coast relais consortium moves to a hosted  option with relais international for systems  and system’s services. once these  technological enhancements and policy  changes are in place, it is likely that  subsequent efficiencies will be achieved  only by implementing unmediated  document delivery services.     works cited    jackson, mary e, bruce kingma and tom  delaney. assessing ill/dd services:  new cost‐effective alternatives.  washington, dc : association of  research libraries, 2004.  nisonger, thomas e. “accessing  information: the evaluation research.”  collection management 26.1 (2001): 1‐ 23.   stein, j.e. ʺmeasuring the performance of  ill and document supply: 1986  to1998ʺ, performance management  and metrics 2.1 (2001): 11‐72.   warner, patrick. “cisti source and journal  use at memorial university of  newfoundland.”  interlending &  document supply 32.4 (2004): 215‐218.     other works mentioned    bailey‐hainer, brenda k. “swift:  coloradoʹs statewide electronic  interlibrary loan service.” colorado  libraries 27.3 (2001): 33‐35.     bailey‐hainer, brenda k. “multi‐type  statewide resource sharing: the  colorado swift experience.” journal  of library administration 40.1/2: (2004)  5‐22.     burke, kimberly a. “express at any cost?  designing a pilot project for improving  turn‐around time for ill of  dissertations.” journal of interlibrary  loan, document delivery &  information supply 11. 4: (2001) 89‐101.     connell, ruth s., karen l. janke.  “turnaround time between illiadʹs  odyssey and ariel delivery methods:  a comparison.” journal of interlibrary  loan, document delivery &  information supply 16. 3 (2006) : 41‐55.     connolly, pauline a. “interlending and  document supply: a review of recent  literature—xxxvii.” interlending &  document supply 28.1 (2000) : 28‐35.     ‐‐‐.  “interlending and document supply: a  review of recent literature‐‐xxxix.”  interlending & document supply 29.1  (2001.) : 28‐35.     cornish, alan. “the relais document  delivery system: an innovative  model for resource sharing.” journal  of interlibrary loan, document  delivery & information supply,  vol.3  (2000): 77‐84.    fisher, t. ʺbenchmarking document  delivery at the university of otago  libraryʺ,  http://www.conference.co.nz/lianza 2000/    gould, sara. “interlending and document  supply: a review of recent literature‐‐ xxxviii.” interlending & document  supply 28.3 (2000): 35‐42    ‐‐‐. “interlending and document supply: a  review of recent literature‐‐xl.”  interlending & document supply 29.3  2001: 133‐139      http://www.conference.co.nz/lianza evidence based library and information practice 2007, 2:2  65 ‐‐‐. “interlending and document supply: a  review of recent literature‐‐xxxviii.”  interlending & document supply 30.1  (2002): 35‐42    guadagno, elena. “taking interlibrary loan  and document delivery to new  frontiers using relais ill.” journal of  interlibrary loan, document delivery  & electronic reserve 15 4: (2005) 83‐87    jackson, mary e. measuring the  performance of interlibrary loan  operations in north american  research and college libraries.  washington, dc : association of  research libraries, 1998.    mcgrath, mike. “interlending and  document supply: a review of recent  literature‐‐xlii.” interlending &  document supply 30. 3 (2002): 145‐151.     ‐‐‐. “interlending and document supply: a  review of recent literature‐‐xlii.”  interlending & document supply 30.  4 (2002): 203‐209    ‐‐‐. “interlending and document supply: a  review of recent literature‐‐xliv.”  interlending & document supply 31.  1 (2003): 35‐41    ‐‐‐. “interlending and document supply: a  review of recent literature‐‐xlv.”  interlending & document supply 31.  2 (2003): 142‐146    ‐‐‐. “interlending and document supply: a  review of recent literature‐‐xlvi.”  interlending & document supply 31.  3 (2003): 201‐207    ‐‐‐. “interlending and document supply: a  review of recent literature‐‐xlvii.”  interlending & document supply 31.  4 (2003): 270‐276    ‐‐‐. “interlending and document supply: a  review of recent literature‐‐xlviii.”  interlending & document supply 32.  1 (2004): 50‐54    ‐‐‐. “interlending and document supply: a  review of recent literature‐‐xlix.”  interlending & document supply 32.  2 (2004): 142‐146    ‐‐‐. “interlending and document supply: a  review of the recent literature ‐ 50.”  interlending & document supply 32.4  (2004): 244‐254.     ‐‐‐. “interlending and document supply: a  review of the recent literature ‐ 51.”  interlending & document supply 33.1  (2005): 42‐48.    ‐‐‐. “interlending and document supply: a  review of the recent literature ‐ 52.”  interlending & document supply 33.3  (2005): 162‐171    ‐‐‐. “interlending and document supply: a  review of the recent literature ‐ 53.”  interlending & document supply 33.4  (2005):214‐220.    ‐‐‐. “interlending and document supply: a  review of the recent literature ‐ 54.”  interlending & document supply 34.1  (2006): 36‐43    ‐‐‐. “interlending and document supply: a  review of the recent literature ‐ 55.”  interlending & document supply 34.2  (2006): 84‐92    ‐‐‐. “interlending and document supply: a  review of the recent literature ‐ 56.”  interlending & document supply 34.3  (2006): 140‐147    ‐‐‐. “interlending and document supply: a  review of the recent literature ‐ 57.”  evidence based library and information practice 2007, 2:2  66 interlending & document supply 34.4  (2006): 194‐203    national resource sharing working group  (2001), interlibrary loan and  document delivery  bench marking  study. national library of australia.  <http://www.nla.gov.au/initiatives/nrs wg/benchmarking.html>      porat, lynne.  “automation of interlibrary  loan services: effects on the patron  and the library.” interlending &  document supply 29. 3: (2001) 108‐113.    ruthven, t., ʺwhat makes high performance:  the ill/dd benchmarking studyʺ,  incite, 22 (2001) 10‐15.    ruthven, tom, susan magnay. “top  performing interlending operations:  results of the australian  benchmarking study.” interlending &  document supply 30.2 (2002): 73‐79.     søndergaard, p. ʺill performance  measurementʺ, df‐revy, 23.10 (2000):  320‐2.     stabler, karen. “benchmarking inter library  loan and document delivery services:  lessons learned at new mexico state  university.” journal of interlibrary  loan, document delivery &  information supply 12.3 (2002): 57‐73    sutherland, n. and susan wanat.  “marshaling statistics: enhancing  interlibrary loan service with oclc  management software.”  journal of  interlibrary loan, document delivery  & information supply 11.1 (2000):87‐ 98             tonn, anke. “interlibrary loan internet  accessible database (illiad): a  patron focused satisfactory  interlibrary loan service study.”  journal of interlibrary loan,  document delivery & information  supply 14. 2 (2003): 49‐63.     vattulainen, p. “interlending performance  measurement in the nordic countries:  a project plan.” df‐revy 23.10 (2000)  307‐8    weible, cherié l and christine robben.  “calming the tempest: the benefits of  using prospero for electronic  document delivery in a large  academic library. journal of  interlibrary loan, document delivery  & information supply 12.4 (2002) 79‐ 86          http://www.nla.gov.au/initiatives/nrswg/benchmarking.html http://www.nla.gov.au/initiatives/nrswg/benchmarking.html evidence based library and information practice evidence based library and information practice 2011, 6.1 61 evidence based library and information practice evidence summary the presence of web 2.0 applications is associated with the overall service quality of library websites a review of: chua, a. y. k., & goh, d. h. (2010). a study of web 2.0 applications in library websites. library & information science research, 32(3), 203-211. reviewed by: leslie bussert reference and instruction librarian university of washington bothell/cascadia community college campus library bothell, washington, united states of america email: lbussert@uwb.edu received: 29 nov. 2010 accepted: 16 jan. 2011 2011 bussert. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to determine the prevalence and use of web 2.0 applications in library websites and to determine whether or not their presence enhances the quality of the sites. design – content analysis. setting – public and academic libraries in north america, europe, and asia. subjects – a total of 120 academic and public library websites in english. methods – this study identified and selected library websites from academic and public libraries using a combination of directories, ratings reports, and ranking lists from three geographic regions. over a four-month period in 2009, three coders conducted a three-step content analysis of the selected library websites. this analysis was conducted after having established inter-coder reliability using cohen’s kappa and analytic procedure familiarity, using a randomly selected pilot set of 30 library websites. the remaining 90 websites were evenly distributed among the three coders for analysis. they determined whether web 2.0 applications were present, examined how those applications were used, and gave an overall appraisal of website quality. coders inspected library website links, conducted site searches, used search engines, and searched within major social networking sites to determine the presence of web 2.0 applications. a quality framework classification scheme was used during coding mailto:lbussert@uwb.edu� evidence based library and information practice 2011, 6.1 62 to identify how the web 2.0 applications were being used within library websites. this framework was established around four dimensions of library services: information acquisition (blogs and wikis), information dissemination (rich site summary, or “rss”), information organization (social tagging), and information sharing (social networking and instant messaging). a five-point likert scale was also used in concert with a website quality evaluation framework to assess the quality of the library websites. this framework included three aspects of website quality: system quality, information quality, and service quality. a combination of statistical techniques such as chi-square analysis, cramer’s v, analysis of variance, tukey’s statistic, and multiple regression were then used to analyze the findings. main results – web 2.0 applications have been adopted by libraries across north america, europe, and asia. the most popular web 2.0 applications were blogs (56.6%), rss (50%), and instant messaging services (46.6%) while less prevalent were social networking services (20%), wikis (16.6%), and social tagging applications (16.6%). the extent of libraries’ adoption of web 2.0 applications also varied according to region. north american libraries used all web 2.0 applications most consistently and were more attuned to heavier users, particularly with regard to information sharing applications (e.g., instant messaging, social networking). european libraries lagged behind those in asia in embracing information acquisition applications (e.g., blogs, wikis) but were comparable to other regions. social networking services and instant messaging were strongly associated with region; rss, blogs, and social tagging showed moderate to moderately strong associations; and only wikis did not demonstrate a statistically significant association with region. this study also identified how web 2.0 applications were being used. blogs were used to generate interest, engage users, and endear users to library personnel, while wikis culled resources from users thematically. rss feeds communicated news, events or resource updates, and were also used in combination with library blogs. social tagging invited users to save, organize, and share information, while some websites used librarian-generated tags for search and discovery or included them as tag clouds in library blogs. instant messaging was used to assist users synchronously during scheduled timeslots; meanwhile, social networking sites offered alternate channels to communicate and build connections with users. the authors found a relationship between website quality and the presence of web 2.0 applications based on the criteria presented in their quality evaluation framework. applications facilitating information sharing (i.e., social networks and instant messaging) had a stronger influence on a website’s overall quality than those being used for information acquisition and dissemination (i.e., rss, wikis, and blogs). web 2.0 applications among academic and public libraries shared a similar level of presence on the library websites; therefore, the type of library is not associated with the quality of the library website. north american websites tended to be higher in quality than european or asian library websites, and differences in quality between european and asian sites were insignificant. conclusion – this study reveals that libraries in various geographic regions adopt web 2.0 applications differently. web 2.0 applications in library websites enhance users’ experience with library resources and support their interests. library websites are implementing web 2.0 applications in a myriad of ways, including using these applications in concert with one another to increase user engagement. the presence of web 2.0 applications strongly affects service quality but only weakly influences information quality. commentary this study is one of the first to investigate the extent of web 2.0 applications as implemented in library websites and whether their presence contributes to website quality. based on the evidence based library and information practice 2011, 6.1 63 web 2.0 and website evaluation literature, the authors propose two valuable frameworks: one for identifying web 2.0 applications, and another for evaluating quality within library websites. this information can assist web designers and administrators in planning, implementing, managing, and obtaining support to employ web 2.0 applications, and for creating policies for their implementation. the methodology was well designed though not without weaknesses. the authors recognized the limitations of their study, such as only including english-language websites and seeking out just six specific web 2.0 applications within those sites. there are additional flaws not addressed by the authors, however, and these include the geographic areas from which the sample websites were taken; the methodology used to select the websites; a lack of accounting for the influence of culture on the adoption of web 2.0 applications; and an error in the data analysis. the geographic categories examined are overly broad and uneven. the asian websites analyzed included libraries in australia and new zealand, while north american websites examined omitted canada. this may be due to the methods that were used to select the websites. using web rating and ranking lists biases the sample in favour of those more apt to be included in such lists: in other words, large and affluent public or academic libraries. such libraries are likely to have a solid web presence with large user bases, influencing the presence of web 2.0 applications in their websites. thus, the findings may not apply to smaller or less affluent libraries. cultural nuances affecting the adoption of web 2.0 applications were not accounted for, nor discussed. web 2.0 applications are well known in the united states, but may not be as familiar or popular in the other regions included in the study. varying rates of internet penetration were also not addressed in depth. a library’s user population may also influence the presence of web 2.0 applications. an inaccurate generalization is made by including social tagging (i.e., information organization) within the variables identified to have strong influence on overall website quality when their data show it not to be a significant variable in relation to the dependent variable of information quality. despite its limitations, however, this study offers a baseline for future studies to build upon, and the authors themselves call for further research into this topic. as web 2.0 applications rapidly emerge, it will be prudent to revisit these questions in order to understand trends in their adoption for use in library websites. / evidence based library and information practice evidence summary evidence based library and information practice evidence based library and information practice 2011, 6.2 34 evidence based library and information practice evidence summary lack of annual reports make it difficult to analyze library strategic credibility a review of: staines, g. (2009). towards an assessment of strategic credibility in academic libraries. library management, 30(3), 148-162. reviewed by: kirsty thomson subject librarian library university of the west of scotland, united kingdom email: kirsty.thomson@uws.ac.uk received: 28 feb. 2011 accepted: 21 apr. 2011 2011 thomson. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to investigate whether libraries achieve strategic credibility by assessing if strategic planning goals match the achievements described in annual reports. design – content analysis of annual reports and strategic plans from a sample of association of research libraries (arl). setting – academic libraries in canada and the united states of america. subjects – a random sample of 12 canadian and 16 american academic libraries. all libraries were members of arl. methods – the researcher contacted the directors of 28 arl libraries and asked for copies of their strategic plans and annual reports. she also visited the websites of libraries to obtain the reports. the contents of the strategic plans and annual reports were analyzed, and trends in the canadian and american strategic plans were identified. main results – this study found that only 39% of arl libraries produce annual reports, making it difficult to assess if libraries have strategic credibility, as their strategic plans cannot be assessed against annual reports. the strategic plans gathered in this study were analyzed and emerging themes were identified. these included physical library space (renovations, expansions or new buildings); offsite storage; assessment (both of the libraries’ services, and of information literacy training); development activities such as fundraising and marketing; and personnel mailto:kirsty.thomson@uws.ac.uk� evidence based library and information practice 2011, 6.2 35 issues. cultural differences also were found in the strategic plans, with american libraries being more focused on trends such as digitization and institutional repositories, and canadian libraries’ plans being more focused on users’ needs. trends in annual reports were not reported due to the small number of annual reports in the sample. conclusion – this study gives a snapshot of the trends in strategic plans of arl members. it shows that many arl members do not produce an annual report, and that it is therefore difficult to assess if their strategic plans are implemented successfully. the article hypothesizes that the communication of achievements may now be part of development and marketing efforts, rather than traditional annual reports. commentary this study set out to investigate the connection between library strategic plans and annual reports, and to use this comparison to assess strategic credibility. no information is provided about the methods used for selecting the random sample of libraries, but the author should be commended for locating information about the availability of strategic plans and annual reports for all of her selection. the study does not include matched pair comparisons of strategic plans and annual reports from the same library, which would have helped address the key research question of "whether strategic planning goals align with annual reports" (p. 148). the lack of annual reports from her sample libraries, and therefore the lack of comparisons, means that this research question is not fully addressed; perhaps the study would have been better reframed as a content analysis of strategic plans, rather than tackling the strategic credibility of libraries. the results section of this study gives information about how many libraries produce strategic plans and annual reports. some of the figures given are problematic. table i (p. 155) shows a total of 11 libraries publishing annual reports, but produces this total from the sum of seven for canada and five for the united states. the text states that “only 11 libraries ... publish annual reports” (p. 155), but table iii also gives the total as 12 (p. 157). the article does not contain any obvious explanations for these discrepancies, and this does cast a shadow on some of the quantitative analysis. the researcher used association of college and research libraries criteria to carry out content analysis of the strategic plans and identified widespread trends, but the low number of annual reports meant trends could not be identified in the annual reports and little comparative work could be carried out. although the sample size is small, this exploratory study gives an insight into the themes found in the strategic plans of arl libraries, and highlights the difficulty in assessing strategic credibility in the absence of library annual reports. there is scope for further research into other methods for assessing strategic credibility, especially as this study suggests alternative communication routes may be being used. / evidence based library and information practice received: 28 feb. 2011 accepted: 21 apr. 2011 abstract setting – academic libraries in canada and the united states of america. commentary evidence based library and information practice evidence based library and information practice 2010, 5.4 4 evidence based library and information practice editorial responsibilities 2010. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. editor-in-chief: denise koufogiannakis associate editor (articles): alison brettle associate editor (evidence summaries): lorie kloda associate editor (classics): jonathan eldredge associate editor (features, commentaries, ebl101, news): denise koufogiannakis production editor: katrine mallan editorial intern: andrea baer copyeditors: priscilla stephenson (lead copyeditor), georgianne bordner, richard hayman, heather pretty, lisa shen, dale storie, mary virginia taylor indexing support: pam morgan writing assistance: molly des jardin, hope leman, tracy powell iwaskow, carol waseleski / evidence based library and information practice editorial responsibilities evidence summary   plagiarism curricula may reduce need for punitive plagiarism education   a review of: george, s., costigan, a., & o’hara, m. (2013). placing the library at the heart of plagiarism prevention: the university of bradford experience. new review of academic librarianship, 19(2), 141-160. doi:10.1080/13614533.2013.800756   reviewed by: robin e. miller assistant professor and research & instruction librarian mcintyre library university of wisconsin-eau claire eau claire, wisconsin, united states of america email: millerob@uwec.edu   received: 2 dec. 2013     accepted: 21 jan. 2014      2014 miller. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to describe the development and implementation of two courses designed to help university students avoid plagiarism.   design – quantitative and qualitative analysis.   setting – a university in the united kingdom.   subjects – an unknown number of university students who took a plagiarism awareness program (pap) course between 2008 and 2011, and approximately 3,000 university students enrolled in a plagiarism avoidance for new students (pans) course delivered via a virtual learning environment (vle) between october and december 2012. the authors attempted to collect rates of continued plagiarism among students who had taken plagiarism education courses. the authors also surveyed 702 university students about plagiarism in 2011.   methods – data collected from pap participants informed revision of the authors’ approach to plagiarism education and led to development of the second course, pans. at the end of the course, students completed a test of their knowledge about plagiarism. authors compared scores from students who took a course supervised by a librarian to the scores from students who took the course independently.   main results – students reported that many aspects of citation and attribution are challenging (p. 149). the authors discovered that 93% of students who completed the pans course facilitated by a librarian in-person passed the final exam with a grade of 70% or higher, while 85% of students who took the same course independently, without a librarian instructor, in an online vle scored 70% or higher (p. 155). the authors report that referrals of students who plagiarized declined significantly (p-value < 0.001) since the implementation of a plagiarism avoidance curriculum.   conclusion – as reported by the authors, first-year university students require more extensive education about plagiarism avoidance. a university plagiarism avoidance program instructed by librarians reduces the total number of students caught plagiarizing and mitigates the need for punitive plagiarism education programs. in discussing the challenges and implementation of plagiarism awareness curricula, the authors contribute to the dialogue about effective approaches to addressing this critical issue in higher education.     commentary   though concerns about academic integrity simmer at institutions of higher education, few universities have perfected plagiarism education programs. the authors situate their research in the context of higher education’s effort to detect, categorize, and punish students for acts of plagiarism. the authors also outline development of the university of bradford’s (uk) approach to plagiarism education, unfolding through two courses.   the first course (pap) was aimed at students caught plagiarizing, and students enrolled in the course reported positive feelings about the course experience. while this commentary assumes that the selection method included all students enrolled in the pap course, the authors do not report the number of subjects. the second course (pans) was designed to train all first-year university students to avoid plagiarism. this program development was informed by a survey of 703 students, though the authors do not provide a survey response rate. survey responses indicate that students are concerned about plagiarism and that they misunderstand basic concepts. the authors did not append the survey instrument, limiting opportunities for validation or replication.   the most significant findings of this research result from a 3-month period in 2012, when the authors collected data from 3,000 first-year students enrolled in the plagiarism course. the authors report that 93% of students enrolled in the pans course led by a librarian scored 70% or higher on the course test, while 85% of students enrolled in an asynchronous pans course without a librarian instructor achieved a score of 70% or higher. the authors do not comment about whether this difference is significant enough to warrant expansion of librarian-led courses, though the difference in achievement is only 8%. beyond final test scores, the authors do not report data about student performance in the final, online iteration of the course. since the pans course was administered in a virtual environment, and therefore the authors had ample opportunity to collect student achievement data from specific learning objects and activities, the lack of reporting on said data is a significant oversight. by failing to evaluate areas of the course material where students performed better or worse than expected, the authors leave the reader with little knowledge of the learning outcomes or specific skills developed by students in the course.   the authors argue that data collected annually from 2008 to 2012 suggest that the number of students caught and referred for plagiarizing declined significantly following the implementation and growth of pans, especially in the 2011-2012 academic year. this claim is supported with a footnote reporting “significance is chi-squared 46.7182 p-value 0.000” (p. 156). although this p-value is significant, the authors should not have reported it as they did. as explained in the statistics tutorial at university of glasgow: “never report that p=0, this is a side effect of the limited accuracy of some software. if p is less than 0.001, then report p<0.001 rather than the exact p value” (p-values section).   all first-year students took the course in 2012, but the authors do not report the number of students enrolled in the years spanning 2008 to 2011, or what percentage of the student body took the course prior to 2012. a minimally labeled bar graph may also support the claim that 2011-2012 outcomes were better than any previous year (p. 155), but this decline only indicates that students were less likely to plagiarize in that academic year and does not project likelihood of future plagiarism.   the validity of this research is significantly limited by its design and by the data reported or omitted by the authors. data were collected through multiple phases of the program’s four year development. however, the authors inconsistently reported number of subjects or characteristics for all phases of the multi-year study. the authors utilized different methods of data collection and analysis, and no method is compared to other inquiries or research in the field. while the authors report notable findings for the final year of the study, these derive from data collected from a large subject pool over a brief period, preventing the authors from drawing a clear link between the intervention and long-term outcomes.   for academic library practitioners, this research serves as one example of how to develop a large-scale plagiarism avoidance curriculum for all students at a university. as exploratory research, this work may justify continued development of the anti-plagiarism program at the university of bradford. the researchers must clarify their methods, report data more clearly, and collect longitudinal quantitative data about student achievement as a result of the plagiarism interventions described. future research should control for additional variables and the authors should attempt to clearly compare the performance of their subjects over time.   reference   statistics tutorial: p-values and t-tables. (n.d.). in university of glasgow statistical understanding made simple. retrieved 11 feb. 2014 from http://www.gla.ac.uk/sums/users/jdbmcdonald/prepost_ttest/pandt1.html evidence summary   google scholar retrieves twice as many relevant citations as pubmed and provides greater full-text access for quick, clinical nephrology searches   a review of: shariff, s. z., bejaimal, s. a. d., sontrop, j. m., iansavichus, a. v., haynes, r. b., weir, m. a., & garg, a. x. (2013). retrieving clinical evidence: a comparison of pubmed and google scholar for quick clinical searches. journal of medical internet research, 15(8). doi:10.2196/jmir.2624   reviewed by: diana k. wakimoto online literacy librarian california state university, east bay hayward, california, united states of america email: diana.wakimoto@csueastbay.edu   received: 15 oct. 2013   accepted: 6 feb. 2014      2014 wakimoto. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to compare recall and precision of results retrieved by searches in pubmed and google scholar for clinical nephrology literature.   design – survey questionnaire, comparative.   setting ­– canada.   subjects – practicing nephrologists with average age of 48 years and who have practiced nephrology for an average of 15 years.   methods – the researchers identified 100 systematic reviews in renal therapy published between 2001 and 2009. the primary studies cited in the systematic reviews served as the reference standard for relevant articles; 1,574 unique citations were identified and used to measure recall and precision. the researchers created a unique clinical question from each of the objective statements of systematic reviews and sent one question to a random sample of practicing nephrologists to determine the search strings they would use to search for clinical literature; the researchers collected 100 usable responses. using the search string in both google scholar and pubmed, the researchers analyzed the first 40 retrieved results in each for recall of relevant literature and precision. the researchers also analyzed the availability of full-text articles in each database. a pilot study to test the methodology preceded the main study.   results – google scholar’s recall for the first 40 records was 21.9% and pubmed was 10.9%. each database contained 78% of the relevant literature/reference standard set from the systematic reviews. however, 15% of the articles were in neither database. precision results were similar (7.6% for google scholar and 5.6% for pubmed). google scholar had more full-text available at 15% of articles versus 5% for pubmed. google scholar and pubmed had similar numbers of relevant articles when all retrieved records were analyzed, but google scholar still provided more access to free full-text articles.   conclusion – google scholar provides better recall and provides more access to full-text than pubmed; however, search strings provided by nephrologists used in both databases failed to retrieve 80% of relevant articles. therefore improving nephrologists’ ability to effectively search could enhance their ability to implement research in practice helping patients. the researchers suggest future studies should be conducted to determine the generalizability of the findings on recall and precision in other medical disciplines.     commentary   this study adds to the growing body of literature showing multiple professions’ interest in the usefulness of google scholar for searching literature as compared to more established databases such as pubmed (bourbakhsh, nugent, wang, cevik, & nugent, 2012). as more people – both researchers and practitioners – become aware of, and start to use, google scholar, it becomes even more important to analyze google scholar’s recall, precision, and access to full-text articles in relation to other databases. this study’s results will be of great interest to librarians who work with clinicians in medical disciplines and for others who may want to replicate the study’s methodology in their own area of practice.   this is a very strong study methodologically, is well-written, and is useful for both researchers and practitioners. it builds on prior research conducted by the research team on search habits and techniques of nephrologists (shariff et al., 2011); a critique of another study by the researchers in this area appears in an earlier volume of evidence based library and information practice (kelly, 2012). the current study is considered valid using the critical appraisal checklist by glynn (2006). the methodology is described in a way that can be replicated and the results are clearly reported. the researchers also document the limitations of the study, which could be tested in future studies. the researchers also provide suggestions for future research to expand the generalizability of the findings.   a minor possible weakness of the article is this: for readers to fully understand the survey questionnaire used, and the characteristics reported on the nephrologists’ searching habits, they would need to read the research team’s previous article which more fully analyzes the survey data (shariff et al., 2011). however, this is quite minor and the researchers still report adequate information on nephrologist survey respondents for the readers to contextualize the results and discussion.   this research article is significant in its rigorous comparison of google scholar and pubmed and provides valuable insight for medical librarians, and possibly librarians who work in other fields as well. as more people use google scholar for quickly finding research to use in clinical therapy, it is important that librarians understand the strengths and weaknesses of the database and its place in literature searching. as the researchers note that most nephrologists view fewer than 40 search results, it would be interesting to replicate the study given pubmed’s recently introduced relevancy ranking, a feature that may have given google scholar the edge in retrieving more relevant results in the first 40 records. the researchers’ call for helping to improve the searching capabilities of clinicians is one area in which librarians can apply this study’s findings to potentially impact patient care.     references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi:10.1108/07378830610692154   kelly, k. (2012). applying the narrow forms of pubmed methods-based and topic-based filters increases nephrologists’ search efficiency. evidence based library and information practice, 7(3), 95-97. retrieved 17 feb. 2014 from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/17687/14307   nourbakhsh, e., nugent, r., wang, h., cevik, c., & nugent, k. (2012). medical literature searches: a comparison of pubmed and google scholar. health information and libraries journal, 29, 214-222. doi:10.1111/j.1471-1842.2012.00992.x   shariff, s. z., bejaimal, s. a. d., sontrop, j. m., iansavichus, a. v., weir, m. a., haynes, r. b., … garg, a. x. (2011). searching for medical information online: a survey of canadian nephrologists. journal of nephrology, 24(6), 723-732. doi:10.5301/jn.2011.6373   evidence based library and information practice evidence based library and information practice 2010, 5.3 68 evidence based library and information practice commentary a voyage of discovery: identifying barriers to eblip in the caribbean andrew booth reader in evidence based information practice school of health and related research (scharr) university of sheffield sheffield, united kingdom email: a.booth@sheffield.ac.uk jonathan d. eldredge associate professor, school of medicine and interim coordinator learning design center, health sciences library and informatics center university of new mexico albuquerque, new mexico, united states of america email: jeldredge@salud.unm.edu received: 19 july 2010 accepted: 12 aug. 2010 2010 booth and eldredge. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/bync-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. in fourteen hundred and ninety two columbus sailed the ocean blue when christopher columbus first arrived in santo domingo, now the dominican republic, he encountered an already diverse and vibrant local culture. as successive waves of europeans have journeyed to the caribbean they too have come to recognise the value of a fusion of cultures, of a unity in diversity. clearly there are lessons for any sphere of practice, including librarianship, regarding the pitfalls of attempting to impose models, albeit those proved successful elsewhere, on local inhabitants within their existing dutch, english, french or spanish traditions. such attempts not only face the likelihood of limited success, but also carry the risk of missed opportunities in failing to engage with a wealth of skills, expertise, experience and cultural heritage. in twenty hund-er-ed and ten a voyage of discovery took place again thus it was that invited plenary speakers at the 40th association of caribbean university, research and institutional libraries (acuril) conference 2010, joanne marshall, jonathan eldredge and andrew booth came to recognise that it would not simply be enough to present existing models of evidence based library and information practice, albeit successfully employed elsewhere across the globe. it would be equally important to engage with the richness of caribbean library mailto:a.booth@sheffield.ac.uk mailto:jeldredge@salud.unm.edu evidence based library and information practice 2010, 5.3 69 culture with all its diverse traditions, origins, and language groups. furthermore, after more than three days of presentations and workshops outlining the current state of eblip thinking, it was opportune for 300 plus participants from across the region to vocalise and share their reactions to this potential new addition to an already weighty library and information sciences agenda. for a friday morning session, designed to feature a joint “conversatorio” between jonathan eldredge and andrew booth, we decided to initiate an interactive process with the conference delegates. we sought to provide delegates with an opportunity to share, in a non-threatening environment, their personal responses to the featured eblip process. furthermore, we wanted to derive a unique picture of the barriers and constraints facing caribbean librarians when seeking to implement eblip within their own contexts. this would be both for the particular, i.e., the caribbean region, and, given that this context features language, skills, and resource constraints, as an exemplar for such issues more generally (booth, 2008). in this sense we could contribute further to the conference by facilitating, whilst in our turn benefiting individually from, this interactive process. finally, we believed, and this was confirmed through consultation with members of the acuril executive, that such interaction would provide a valuable snapshot within which to tackle eblip implementation issues. after a brief explanation of the intended purpose and schedule for the session we invited participants to complete their response to the following statement: “eblip would be easier in my library if...”. the emphasis on “my” was intentional, with this process being aimed at local level implementation and not on issues removed from day-to-day experience. this statement was simultaneously translated into both french and spanish. participants were requested to record two copies of their response; the first for their personal use in the subsequent discussion and the second on cumulative data collection sheets for our subsequent analysis and use by acuril. once participants had been given time to record their responses they were invited to form small groupings by language in order to share their issues with one another. separate language activities are not generally a feature of the cosmopolitan acuril activities. rather, we made this particular decision based upon practical considerations, such as the limited availability of translators for small group activity, and in recognition that, notwithstanding significant diversity, there was greater commonality between groups sharing the same language. the resultant buzz groups were very lively and largely self-directing. the two facilitators employed “light touch” monitoring and guidance, the latter only when requested by the groups, to ensure that there were no difficulties with understanding of either instructions or process. after between fifteen and twenty minutes the facilitators encouraged the groups to share at least one issue that had arisen within their discussion, with anonymity being preserved through use of a single rapporteur for each group. simultaneous translation was again provided for this plenary activity. the facilitators then invited the groups to reconvene and to engage in a brief solution-generating activity (8-10 minutes) to provide a degree of resolution to the issues that had been raised and shared. the intention was very much that “a problem shared is a problem halved” with colleagues often being able to suggest potential solutions that were not within the immediate view of the one generating the original issue. finally jonathan eldredge shared with participants published examples, from the research section of the medical library association (eldredge et al, 2009) and the swedish library association (maceviciute & wilson, 2009), of recent delphi exercises on research priorities. it was suggested that similar methods might be usefully transferred and adopted, while recognising that the focus of the exercise in the caribbean had been on impediments to evidence based library and information practice (i.e., related to implementation) and not on research priorities per se. evidence based library and information practice 2010, 5.3 70 a flavour of the caribbean the authors intend to work with members of acuril in formally recording and analysing all responses, following grouping and verification, and to publish a more complete report, with french and spanish language versions as appendices. for the present we feel that it is useful not only to share some initial data from the exercise in this commentary, but also to describe this process to the readership of evidence based library and information practice. we believe that this process is feasible and practical for a wide range of library interest groups, especially those facing particular constraints to the spread of eblip. what did we find? our informal survey elicited 83 separate responses, with 44 in spanish, 33 in english, and 6 in french. while many concerns, particularly those relating to constraints of time and the need to acquire the technical skills of evidence based practice, are common to other areas of the library world, delegates identified particular issues of perceived importance within the region. the paramount concern, receiving greater emphasis than in other parts of the library world, related to the need for the support from managers and the institution as a whole. there appeared to be the perception that managers would not be familiar with the evidence based process and so a major task would be to educate managers and to get them to provide a supportive framework within which staff could bring about changes: [eblip would be easier in my library if...] those responsible for the service give latitude to those who propose to do projects within an agreed framework. [french] [eblip would be easier in my library if...] in my library on a day to day basis we take decisions based on evidence for our actions, we have clear policies and, above all, we have the support of management. [spanish] related to this was the fact that, unlike in other parts of the world where there is major “buy in” to the evidence based decisionmaking model, this new way of doing things would need to compete with, and survive against, well-established alternative models for decision-making (such as existing project management methodologies and programmed budgeting): [eblip would be easier in my library if...] the professional staff agree that it is the common approach to problem solving or project management. [english] uptake and widespread adoption of the evidence based practice model is, therefore, by no means a certainty. another concern receiving particular emphasis was that existing roles of professional staff members, and perceptions of what were and what were not their areas of influence, would place unwelcome constraints on the initiation and promotion of new projects and programmes of change: eblip would be easier in my library if...] i received the support of my superiors to carry out all activities that promote the library with my labours. [spanish] throughout the week the three keynote speakers emphasized the need to work in teams. they also incorporated small group work activities within their sessions. participants’ responses, perhaps expectedly, reflected this perspective. yet, it could be that such a perspective also reflects a particular reality in this region of the world. in other words, the various components of the evidence based process, such as knowledge of research processes, familiarity with english language literature and depth of professional experience, were unlikely to be possessed by a single individual and, therefore, team work, rather than being an optimal feature, was very much a necessity: evidence based library and information practice 2010, 5.3 71 eblip would be easier in my library if...] we form permanent teams of librarians to implement the [eblip] methodology. [spanish] finally we encountered a perception, commonly repeated in other parts of the world but particularly acute in this region, that a lack of resources and qualified staff make it a considerable challenge to deliver even a minimal standard of service: our fundamental problem is the lack of qualified personnel and economic resources. [spanish] [eblip would be easier in my library if...] we have an adequate budget to cover acquisition of library material, technologies, personnel, maintenance of the physical area, and everything to do with the information unit. [spanish] under such circumstances evidence based library and information practice might be considered a patently unattainable, or even unwelcome, distraction. such a perception may need to be addressed in a similar way to that utilised by the cochrane collaboration with a focus on “quick wins” using low-level accessible technologies and services. only once such an approach has addressed the “primary” issues of concern, in terms of urgency, severity and coverage, can one proceed to more technocentric secondary concerns: it is promoted (e.g. examples of success stories). [english] these are significant and serious challenges. an emerging consensus from the comments suggests that through education and continued dialogue, such challenges might be overcome. other comments suggest that the three keynotes speakers’ own voyages of discovery coincided with the prospect of a new hope and vision for others in attendance: we have previously used eblip informally, but now we have the methodology to more easily formalize and apply this process. [spanish] we encourage experimental research in our libraries. [spanish] we need to be trained more deeply in using the tools... to address problems and situations on a day to day basis. [spanish] this last comment particularly suggests future voyages for collaboration and mutual benefits in advancing this newly shared vision. more to be discovered the main activities of the conference took place, appropriately enough, in rooms named after the ships of christopher columbus. for example the venue for the above process was the main santa maria auditorium. for the invited speakers, as for the assembled participants, this was equally a voyage of discovery. for the speakers this conference presented an opportunity to gain valuable insights into constraints for eblip in, what were for them, previously unexplored territories of the library world. for the conference participants this represented an initial foray into hitherto uncharted realms of the eblip method. this rich fusion of shared experience and expertise, of method and context, will thus benefit speakers, participants and, indeed, future readers of the resultant outputs alike. the foyer of the hotel venue displays a large picture with the caption “it appears that more yet remains to be discovered”. though celebrating the discovery of a physical “new world” this caption possesses even greater relevance when applied to the professional development of individual attendees at the conference: the subsequent pursuit of eblip across the entire caribbean region and, indeed, for the advancement of the eblip movement as a whole! evidence based library and information practice 2010, 5.3 72 references acuril [association of caribbean university, research, and institutional libraries]. (2010). “innovation and impact of information in the caribbean: evidencebased practice of the information professional.” 40th acuril conference (june 10th-15th 2010). retrieved august 26, 2010, from: http://acuril.uprrp.edu/ and conference blog, retrieved august 26, 2010, from http://cibernotasacurilcybernotes.wordp ress.com/acuril-xl/ booth, a. (2008). evidence-based practice and the developing world. health information and libraries journal 25(1), 74-77. eldredge, j.d., harris, m.r., & ascher, m.t. (2009). defining the medical library association research agenda: methodology and final results from a consensus process. journal of the medical library association 97(3), 178-85. maceviciute, e. & wilson, t.d. (2009) a delphi investigation into the research needs in swedish librarianship. information research 14(4), paper 419. retrieved july 15, 2010, from http://informationr.net/ir/144/paper419.html http://acuril.uprrp.edu/ http://cibernotasacurilcybernotes.wordpress.com/acuril-xl/ http://cibernotasacurilcybernotes.wordpress.com/acuril-xl/ http://informationr.net/ir/14-4/paper419.html http://informationr.net/ir/14-4/paper419.html evidence based library and information practice evidence based library and information practice 2010, 5.3 62 evidence based library and information practice commentary musings on collection analysis and its utility in modern collection development jane schmidt head, collection services ryerson university library toronto, ontario, canada email: jschmidt@ryerson.ca received: 03 aug. 2010 accepted: 12 aug. 2010 2010 schmidt. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. introduction academic librarians are frequently called upon to provide library reports in support of new program proposals and for periodic program reviews. accreditation and quality assurance bodies want to know that students have access to library services and collections that are sufficient to aid them in the successful completion of their degrees. the individuals responsible for the completion of these reports vary from institution to institution, but the initial request almost always goes to the liaison librarian first. inevitably, a request for collection analysis makes its way to the collections department. as a collection manager in a university that serves a large undergraduate population and an ever growing graduate cohort, i have often been dismayed at the lack of specificity from various accreditation bodies that request library reports, and further bewildered by the level of detail about physical collections desired by librarians charged with completing these reports. in the five years that i have been engaged in collection analysis, i have formed definite opinions about the traditional methods and the lack of quality and utility of the data gathered from these processes. but as we know, old habits die hard. this commentary is an attempt to articulate what i feel academic libraries should prioritize when demonstrating their ability to support a new or existing program with their services and collections. what defines library support? the scholar’s view of the library has been changing steadily over time. according to an ithaka report, that change has not been in a favourable direction. last year, only 24% of faculty surveyed used the library’s online catalogue as a starting point for their research (kolowich, 2010). why is it, then, that the librarian’s first inclination is to study the holdings of library catalogues in extraordinary detail in order to demonstrate that the library can support “x” program? mailto:jschmidt@ryerson.ca evidence based library and information practice 2010, 5.3 63 my institution, ryerson university, will soon be undergoing accreditation review for the association to advance collegiate schools of business (aacsb). wanting to be proactive, i did some investigation about what would be required from the library. in the aacsb 2010 standards and i found two references to the library: services: does the organization provide services (e.g., library, academic services, residence life, parking, maintenance of grounds and facilities, human resource management policies and services, information technology infrastructure, etc.) necessary to sustain the activities of the applicant academic unit? are the costs of those services charged back to the applicant academic unit? (p. 7) a school with only campus-based undergraduate programs would normally be expected to provide sufficient classroom and computing facilities for students, student advising, faculty development including instructional enhancement, library and other information access, technology assistance, and support for faculty intellectual contributions. with the addition of master’s level programs would come expectations for applied research support. (p. 27) my next step was to see if i could find what other institutions had done in this regard, and if their reports were available online. one of the first references i found was a pennsylvania state university libraries study, published in the peer-reviewed journal collection building. the study sought to determine “if libraries which sustain accredited mba programs possess a significantly greater number of selected titles from the harvard business school core collection of titles than those that do not sustain such accredited programs.” (mcguigan, crawford, & kubiske, 2004, p. 78). the methodology was meticulous in its detail and execution. not surprisingly, the authors’ hypothesis was proved accurate, and the conclusion of the study upheld, that there is a “link between the strength of library collections and aacsb accreditation of mba programs” (mcguigan et al., 2004, p. 81). i was disappointed in the extent of the study. to summarize the results of the study, the libraries of aacsb accredited schools have more books from the hbs core list than do libraries from schools not accredited by the aacsb. what does this tell us about these libraries? in my opinion, not much. what about research support? what about electronic resources and federated searches? what about outreach and just-in-time support from qualified professionals? what about data curation and the availability of historical time series financial data? is there an information commons available to undergrad students seeking support in numeracy and writing skills? are there qualified it support providers available to offer technical support? one would hope that the aacsb would be far more interested in these aspects of the library than in the number of business books they have. to be fair, this was a stand-alone study and was not intended to be a report used for aacsb accreditation; however, for those seeking guidance in the process, it supports the predilection to chase down information that reveals little substance. thankfully, the other reports that i found during my search, such as those listed on the aacsb resource guide, presented by the business reference in academic libraries committee (2001), were more fulsome and talked about the library as a whole. that being said, each one of the studies included a micro-collection analysis that included subject heading book counts. as i hope to demonstrate, this is no longer an effective measurement of the quality of a library’s collection. progressive collection analysis is not bean counting collection analysis means many different things. from usage statistics to impact factors to cost-per-use analysis, there are a myriad of tools available to help librarians understand evidence based library and information practice 2010, 5.3 64 the strengths and weaknesses of their collections more fully. the only problem is, we seem to have the tools, but we aren’t exactly sure what to do with what they tell us. in a way, tools (such as those just mentioned) that are focused more on journal literature and databases are easier to use. they can help us determine what our users need in order to advance their research, and they can tell us what is being used frequently, and what is not being used at all. in tough budget times, these figures can help us determine priorities and assist us in defending our collections budgets through demonstration of community need. ulrich’s serials analysis (http://www.ulrichsweb.com/ulrichsweb/anal ysis/default.asp) reveals which peer-reviewed publications are missing from your collection in a few easy steps. unfortunately, when it comes to monograph collection analysis, the answers are not so straightforward. a literature review of methods used for monograph collections analysis yielded many articles about different ways to determine the strength of one’s collection; everything from the classic brief tests of collection strength (white, 1995) to the new and improved version called better than brief tests (white, 2008). hyödynmaa, ahlholm-kannisto, and nurminen used a collection mapping technique in their study of finnish libraries (2010), and oclc’s worldcat collection analysis (http://www.oclc.org/collectionanalysis/) offers another means of analysis. what seemed to be missing from this intensive study were definitive recommendations for action following a collection analysis. one exception is the article by henry, longstaff and kampen (2008), which focused on outcomes. however, the reported outcome was a new system of collection management integrating faculty involvement with a more focused approach in selection. rather than creating innovative solutions such as interlibrary loan/acquisitions partnerships for just-in-time acquisitions (i.e. purchasing on demand), this approach seems almost regressive rather than progressive. at ryerson, we have found that while faculty involvement in collection development is always encouraged, it represents only a small percentage of acquisitions. increased faculty involvement in selection would put our collection management practice back decades, not to mention the lost productivity gained through the use of sophisticated approval plans and streamlined acquisition workflows. in this era of just-in-time acquisitions, digitizing on demand, and new discovery technology such as serials solutions’ summon (http://www.serialssolutions.com/summon/) and oclc’s worldcat local, even if an analysis reveals that a physical collection is lacking in a particular subject area, does it make sense to spend scarce resources filling the shelves with titles that may possibly be needed by someone, sometime in the future? a recent study by courant and nielsen (2010) found that it costs $4.26 per year to maintain one book in open stacks in an academic library. wholesale purchasing of physical volumes to improve a library collection doesn’t seem to be the smartest way to expend resources. that money could be invested in people and technology to improve digitization efforts, and to hire additional staff to more effectively deliver information on demand. space would be saved to better equip libraries to build user-focused common spaces outfitted with services to better help them understand what information is available to them and how best to use it. the association of college and research libraries (acrl) encourages peer comparison for assessing library operations and the provision of library services. a close look at acrl’s standards for libraries in higher education (2004) reveals that microcomparisons, such as using subject headings to determine the number of books in one’s library catalogue compared to another, are not mentioned. yet, this is often the first request that i get from liaison librarians. acrl places higher value on comparing the distribution of resources by student and faculty fte (full time equivalent) and departments within the library. the problem is that this information is much more difficult to obtain than volume http://www.oclc.org/collectionanalysis/ http://www.serialssolutions.com/summon/ evidence based library and information practice 2010, 5.3 65 counts in various opacs. there is still a great deal of fixation on subject-heading based volume counts as collection analysis, despite the fact that it is statistically flawed in many ways (e.g., variant cataloguing practices and different integrated library systems’ (ils) display conventions). furthermore, we rarely identify the overlap or omissions in our collections upon completion of these peer comparisons, thus negating any potential utility of the information gathered. worldcat collection analysis (wca) was meant to be the ideal alternative to this method. although it has been available to libraries for several years, it is still in the fledgling stages of adoption and implementation, much to the chagrin of those who hoped to standardize its use. it would reveal what titles were missing from your collection, and would also tell you how current your collection was compared to your peers. it would accomplish this through sophisticated algorithms that would hold up to statistical scrutiny. my institution invested in wca enthusiastically, only to discover that it was only as good as the cataloging standards and currency of the participating libraries available for comparison. wca allows the generation of title lists for analysis on a one-to-one comparison, but individual permissions must be obtained from each library. we recently obtained permission for a one-to-one comparison from a peer library, only to discover that the institution in question was three years behind in its holdings updates, with no commitment from them as to when that update would occur. the program review due date had long passed by the time they had contacted us. old fashioned subject heading volume counts were submitted as the collection analysis portion of the report instead. not only are librarians lacking the proper tools to collect collection analysis data, we are unable to decide what to do with it, or how to act on the information once we get it, accurate or not. rather than focusing our energies on describing what we can provide in services, electronic resources, and access, we spend an inordinate amount of time counting beans. a more literal interpretation of peer comparison, as per the acrl standards, would produce a more useful and readable report. we don’t actually need to make more work for ourselves, when chances are the omission of this micro-analysis would not even be missed. how should we be analyzing our collections? ideally, collection analysis should be an ongoing process to inform decision making at renewal time for electronic resources and serials, and during regular approval plan review. approval plan revision in and of itself is action-oriented collection analysis, and should be treated as a routine task for subject librarians. this helps to keep librarians aware of not only what is in the collection, but also how (or if) it is being used. many methods have been discussed in this commentary including white’s brief tests, worldcat collection analysis, ulrich’s serials analysis and various electronic resource usage statistics. the appropriateness of each of these methods depends on several factors. good collection analysis should be action-oriented. if space is at a premium at your institution, and a digital-preferred strategy is in place, perhaps using worldcat collection analysis to generate lists of titles not held in your library isn’t the best approach. you could still use wca to demonstrate the extent of your holdings and how they compare to peer institutions, and then explain the processes in place to provide access to titles not held (i.e., interlibrary loan). you could further showcase the strength of your electronic resources and extol the virtue of 24/7 access. that being said, in the print-heavy world of the social sciences and humanities, perhaps you require more funding to expand your monographs budget. you could work with your vendor to generate a retrospective “wish list” of titles compiled, for example, from the output of a peer institution’s approval plan. this is far more useful than simply stating that another library evidence based library and information practice 2010, 5.3 66 has more volumes in a particular subject heading, as is the case in so many library reports i’ve seen. in areas such as science, technology, and medicine, where the focus is on current journal literature and electronic access, you would be wise to highlight subscriptions to top tier journals and demonstrate value for investment in the library through usage statistics and access to the number of publications in which faculty have published. furthermore, if you have an institutional repository (ir), you could campaign for an open access (oa) policy and encourage hosting oa journals locally, highlighting the benefit of increased exposure to publications loaded on the ir. the library has the power to heighten the reputation of the university with these services – an attention-grabbing (and budget friendly) feature, most definitely. the bottom line is that the number of books in a library does not indicate a “good” library. “what about good old-fashioned serendipity?” is a common refrain from traditionalists. libraries are not retail outfits. we do not simply toss titles that are “not moving” from the shelves. there is something to be said for having information at your fingertips that you didn’t even know you needed until you did a keyword search in the catalogue. this portal to knowledge discovery has long been one of the library’s key functions. however, as it has been demonstrated, it is not very efficient or realistic to strive for comprehensive research collections in each and every library, nor is that the highest valued function expected of us from faculty and students. as martin, roblessmith, garrison, and way (2009) astutely observed, the necessity of direct support for curriculum and the desire to maintain core collections to also support research requires “an approach to collection management that is highly elastic and sensitive even to minor program changes in course offerings and program structure” (p. 214). this means that libraries in comprehensive universities must be prepared to be nimble in service provision. services include access to collections, but access does not have to equal ownership. part of the problem lies in lack of consistency in the directives from various accreditation bodies and their reviewers. it is our responsibility to create guidelines and best practices for demonstrating library support. continuing to invest time in poring over micro-collection analysis in library reports instead of focusing on what the library offers as a whole is a disservice to us as busy professionals and to the expectations of accreditation bodies. libraries need to do better in sharing information as recommended in the acrl standards (2004) for peer comparison. transparency in resource allocation and a commitment to improving resource sharing needs to be a priority. in almost every other way, librarians have progressed far beyond being merely the “keepers of books.” if we, as collection managers, continue to perpetuate the notion that having more books is the best way to support programs, we will not advance alternative models of access. we should be focusing instead on more advanced discovery tools and on lobbying for more skilled staff to assist in facilitating the delivery of information. references association to advance collegiate schools of business. (31 jan. 2010). eligibility procedures and accreditation standards for business accreditation. retrieved 10 aug. 2010, from http://www.aacsb.edu/accreditation/aa acsb-standards-2010.pdf brass business reference in academic libraries committee. (11 may 2001). aacsb resource guide. retrieved from http://www.ala.org/ala/mgrps/divs/rusa/s ections/brass/brassprotools/aacsbresourc/ aacsbresource.cfm association of college and research libraries. (2004 june). standards for libraries in higher education. retrieved from http://www.ala.org/ala/mgrps/divs/acrl/st andards/standardslibraries.cfm evidence based library and information practice 2010, 5.3 67 courant, paul n., & nielsen, m. b. (june 2010). on the cost of keeping a book. in the idea of order: transforming research collections for 21st century scholarship (pp. 81-105). washington, dc: council on library and information resources. retrieved 10 aug. 2010 from http://www.clir.org henry, e., longstaff, r., & van kampen, d. (2008). collection analysis outcomes in an academic library. collection building, 27(3), 113-117. hyödynmaa, m., ahlholm-kannisto, a., & nurminen, h. (2010). how to evaluate library collections: a case study of collection mapping. collection building, 29(2), 43-49. kolowich, s. (7 apr. 2010). eroding library role? insider higher ed. retrieved 8 aug. 2010 from http://www.insidehighered.com/news/20 10/04/07/survey martin, h., robles-smith, k., garrison, j., & way, d. (2009). methods and strategies for creating a culture of collections assessment at comprehensive universities. journal of electronic resources librarianship, 21(3/4), 213-236. mcguigan, g. s., crawford, g. a., & kubiske, j. l. (2004). accreditation and library collections: the monographic holdings of academic libraries that support aacsb accredited and non-accredited mba programs in the state of pennsylvania, usa. collection building, 23(2), 78-86. white, h. d. (1995). brief tests of collection strength: a methodology for all types of libraries. westport, cn: greenwood press. white, h. d. (2008). better than brief tests: coverage power tests of collection strength. college & research libraries, 69(2), 155-174. microsoft word es_hall.doc evidence based library and information practice 2006, 1:1  89 evidence based library and information practice       evidence summary    e‐book trial using handheld devices yields mixed reactions from public library staff  and users in essex county, uk      a review of: dearnley, james, cliff mcknight, and anne morris. ʺelectronic book usage in public libraries: a  study of user and staff reactions to a pda‐based collection.ʺ journal of librarianship and  information science 36.4 (december 2004): 175‐82.    reviewed by: stephanie hall  chief librarian, powell river public library  powell river, british columbia, canada  e‐mail: shall@powellriverlibrary.ca     received: 20 december 2005  accepted: 21 february 2006   © 2006 hall. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to assess e‐book delivery on  handheld devices provided to public library  patrons, particularly housebound or  visually impaired patrons and the users of a  mobile library.    design – product trial with a post‐trial  questionnaire for patrons and a post‐trial  focus group discussion among participating  library staff.    setting – mid‐to‐large size public library  system in the united kingdom.    subjects – 23 patrons participated; the  number of library staff participating is not  identified. although the target population  initially identified was housebound people,  patrons reliant on the mobile library, and  visually impaired people, the project team  determined that it did not have sufficient  funds to upgrade the pdas to be compatible  with their preferred software for the visually  impaired, and therefore this group was  eliminated from the study.  lack of funds  was also cited as a factor in the team being  unable to provide assistive technology for  those users suffering from arthritis, though  this group was not excluded from the study.  just over half the patrons were aged 60+,  and more than half were female.  two  patrons self‐identified as housebound.    methods – e‐books were downloaded onto  hewlett packard ipaq 1910 pdas by  library staff and were loaned to participants  between january and april of 2004.   although the authors of the study state that   http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:1  90   “creating staff champions was an important  objective in this project” (p.176), it is unclear  whether staff were screened for positive  attitude towards e‐books. prior to the trial,  staff members were provided with training  and orientation, and participants were asked  for their font and reading preferences.  support for participating patrons was  available from staff or via an accompanying  user manual, e‐mail, or telephone help line.  in a questionnaire administered following  the trial, participants were asked to respond  to questions focusing on functionality of the  handheld devices and e‐book formats,  positive and negative reactions to reading  from the devices, and differences in the  reading experience using the pda as  opposed to a book. a group discussion with  participating library staff was held in april  of 2004.  the feedback from both groups  was compared.    main results – patrons: the devices were  generally found to be usable, with a few  exceptions: one patron with arthritis had  difficulty operating the device and another  developed hand cramps.  positive reactions  regarding the novelty of using the devices,   portability (the ability to store several books  on one small device) and readability (the  ability to customize font size and to read in  low light conditions) were offset by  frustration with low battery life, small  screen size, limits on usage (i.e. not to be  used in the bath), difficulty paging back and  forth (‘getting lost on the ipaq’), and the  inferior sensation of using a pda as  opposed to the ‘tactile’ quality of books.  in  addition, some patrons voiced fears that e‐ books might supplant paper books and  libraries themselves. in all, thirteen patrons  indicated that the e‐books had some  advantages over books, while eighteen  found that there were aspects of using e‐ books that they disliked as compared to  books.    staff: the staff shared some concerns with  users, identifying portability as a strength     and low battery life (including the need to  reformat devices after batteries ran out) as a  weakness. in addition, some staff felt that  some patrons preferred the e‐book format  for books on sensitive topics, as they  provided more privacy in borrowing.  other  staff concerns included the potential for  users loading other software on the devices  and the (presumably in the case of a full  roll‐out where users would download their  own e‐books from home) lack of a  broadband connection for some users.    conclusion – this study should be read as a  case study of a trial of the adobe book and  palm e‐book formats on hewlett packard  ipaq 1910 pdas amongst a small group of  public library (primarily mobile library)  patrons. while the findings generally  indicated that both staff and users found the  technology too problematic to be adopted,  the trial was confined to the hardware and  software specified.  furthermore, since the  sample surveyed was small and not  randomly selected, it would be difficult to  apply the study’s findings to the larger  population (exact number not specified in  the study) of housebound and mobile  library users of the essex county system.     for those who are considering the  introduction of similar technology in a  public library setting, the final report on the  larger project of which this study was a part  is available on the british library website:  http://www.bl.uk/about/cooperation/laser‐ pubs.html     commentary    the study design allowed for patrons to use  the pdas in their home environments,  which provided for greater realism.  at the  same time, this meant that patrons using the  devices were not subjected to the close  behavioural observation characteristic of a  usability study or an in‐depth qualitative  approach. since the method of data  collection was a questionnaire designed to     http://www.bl.uk/about/cooperation/laser%e2%80%90pubs%00 evidence based library and information practice 2006, 1:1  91 elicit ‘reactions’ to the technology, sample  size and question formulation become of  interest when determining the reliability of  the results.  it is unfortunate, therefore, that  the authors indicate neither the size nor the  specific characteristics of population the  sample was drawn from, nor do they  establish whether the sample is  representative of that larger population.   selection bias is not accounted for in the  choice of patrons, while it is unclear  whether selection of staff was deliberately  biased in favour of those who demonstrated  enthusiasm for the e‐books and their  attendant technology. the questionnaire  used for the study is not provided within  the article, nor does it appear on the web‐ based final report of the larger study,  available on the british library website.      despite the above‐noted issues, the topic  under study is increasingly relevant, for, as  more and more jurisdictions invest  substantial quantities of cash in e‐book  technology (for example, the 2005 state‐wide  launch of an e‐book service in ohio:  http://winslo.state.oh.us/publib/oep_project. html), libraries that wish to make these  services available to patrons on both sides of  the digital divide will be seeking workable  solutions.  however, as the researchers  discovered in this study, the limitations of  current handheld devices, coupled with  their cost, make it tempting to “[leave] e‐ book formats useable on pdas…to  individuals who own their own devices and  are comfortable using them” (p.181).   meanwhile, although “it is easy to speculate  that pc tablets and enticing‐looking future  technologies will solve all the problems that  were noted [in the study],…in essence this is  the crux – waiting for tomorrow means that  more effort will continue to be put into e‐ book research rather than actual  implementation” (p.181).            the research team met with various  technical problems as the study progressed  (the need for upgrades to accommodate  specialized software for the visually  impaired, a change in software format from  the e‐book provider mid‐study and, on  another portion of the study, major licensing  woes). these glitches, as the authors note,  can be instructive in themselves. in  particular, they note that “settling on a  format, and building a collection therefore  offers public librarians a specific and crucial  problem to grapple with – which format to  run with and develop” (p.176). a list of e‐ book implementation guidelines based on  the researchers’ experiences are given in the  project’s final report.    finally, the authors note that a full cost‐ benefit analysis in introducing such  technology into library collections is beyond  the scope of the study.  nonetheless, for  public library decision makers thinking  critically about the allocation of collection  development funds, such an analysis would  be of real practical value.   http://winslo.state.oh.us/publib/oep_project evidence based library and information practice evidence based library and information practice 2010, 5.3 4 evidence based library and information practice editorial responsibilities 2010. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. editor-in-chief: denise koufogiannakis associate editor (articles): alison brettle associate editor (evidence summaries): lorie kloda associate editor (classics): jonathan eldredge associate editor (features, commentaries, ebl101, news): denise koufogiannakis production editor: katrine mallan editorial intern: andrea baer copyeditors: priscilla stephenson (lead copyeditor), georgianne bordner, susan dewar, richard hayman, heather pretty, lisa shen, dale storie, mary virginia taylor, elizabeth zeeuw indexing support: pam morgan writing assistance: molly des jardin, hope leman, tracy powell iwaskow, carol waseleski evidence summary   mental model construction in medlineplus information searching involves changes and developments in cognition, emotion, and behaviour   a review of: zhang, y. (2013). the development of users' mental models of medlineplus in information searching. library & information science research, 35(2), 159-170. doi:http://dx.doi.org/10.1016/j.lisr.2012.11.004   reviewed by: antonio derosa reference librarian memorial sloan-kettering cancer center new york, new york, united states of america email: derosaa1@mskcc.org   received: 6 aug. 2013    accepted: 11 oct. 2013      2013 derosa. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract objectives – to explore the construction of mental models as a dynamic process and how users understand a consumer health information system, medlineplus, during a search session.   design – face-to-face interview.    setting – large university.   subjects – a total of 38 undergraduate students participated in the study. all majoring in non-medical fields, such as art history, psychology, business, and communication studies.    methods – participants were randomized into two groups: the simple task group and the complex task group. simple task group members were asked to perform 12 simple tasks while the complex group members performed three more-involved tasks. simple tasks were defined as succinct questions with finite answers while complex tasks were open-ended and required more cognitive activity and synthesizing on the part of the individual. participants in both groups were then given four simple tasks and two complex tasks to perform. data was derived by video recording search sessions with individuals and interview-like questions for the tasks performed. participants were given a brief introduction to the search session design and sessions took place in a private lab. since the aim of the study was to track participants’ mental modeling processes over time, coding of data was caught at three different times throughout the search sessions: t1 (mm1) after five minutes of free exploration, t2 (mm2) after the first search session, and t3 (mm3) after the second search session.     main results – the author discusses the demographic specifics of the population participating in the study. although participants were split into two groups, the results were combined to be more meaningful. out of the 38 participants, 20 were female and 18 were male with ages ranging from 18 to 22. they had, on average, 10 years of computer experience and their average spatial ability score was 12.71. also on average, they spent about 20 minutes completing the first search session and 12 minutes completing the second search session. the results show that participant-developed mental models of the medlineplus web space can be clustered into the following five theoretical components (this information is quantified in tables throughout the paper): system, content, information organization, interface, and procedural knowledge.   conclusion – the study allowed participants to articulate their mental models and representations while conducting predefined searches during private sessions using medlineplus. the study also illustrates how users’ mental models of a system developed during interactions with an online system, on a theoretical level. little is actually known about how mental models are developed when users interact with an information system. the study serves to explore this arena and reveals that the mental model construction involves changes and developments in three parallel dimensions: cognition, emotion, and behaviour. also, these dimensions are accompanied by three mental activities: assimilating new concepts, phasing out previously perceived concepts, and modifying existing concepts. the mental model construction process could be a useful tool to build user models and make better design decisions for information systems.      commentary   the aim of the study was to better understand how a set of users establish mental models while searching an online information resource, namely medlineplus. participants were randomly assigned into two groups and asked to perform a series of tasks (different task-types for each group). although results were recorded in a regimented manner, it is difficult to tell whether or not the population chosen is free of bias. also, there is no discussion of inclusion/exclusion of participants or individual session results in the paper, if any.   the author does not clarify exactly how the face-to-face interviews were conducted. for instance, were there other observers in the room while the sessions were taking place? if so, the author must ensure that the possibility of observer bias was reduced. perhaps one of the most important omissions is that the author does not supply the questionnaire or instrument used to review and capture the responses during the task-performing sessions in the study.   the author clearly defines her step-by-step process for conducting the study. the methodology is well laid out and has potential to be reproduced by other investigators, even without the actual questions asked during the sessions. the results of the study are effectively outlined as they pertain to the design and data collection processes. from beginning to end, there is a clear path that the investigator has taken to perform the study.   this research shows the need for information professionals to better understand mental models: what they are, how users create them, and how they are used to more fluidly navigate a database or other online information system. further consideration of mental models can help database and web developers make more informed decisions about the designs of their interfaces and hierarchy of their content, which can aide in producing more intuitive databases for their users (hvorecký, 2010).   finally, the author gives some advice for further research in the field of user-developed mental models. since most users create mental models based on a database from previous, similar resources they have used, it would behoove researchers to look into providing users with familiar metaphors or systems to help them learn a new system. this cognitive familiarity already exists in the literature and would strengthen the argument of this paper (johnson-laird, 2013).   the appraisal of this mental model development study was conducted using the evidence-based library and information practice (eblip) critical appraisal checklist from memorial university of newfoundland (glynn, 2006).     references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi:10.1108/07378830610692154   hvorecký, j., drlík, m., & munk, m. (2010). enhancing database querying skills by choosing a more appropriate interface. iee educon conference 2010: annual global engineering education conference. (pp. 1897-1905) madrid, spain. doi:10.1109/educon.2010.5492434   johnson-laird, p.n. (2013) mental models and cognitive change [special issue]. journal of cognitive psychology, 25(2) p.131-138. doi:10.1080/20445911.2012.759935 evidence summary   a survey of electronic serials managers reveals diversity in practice   a review of: branscome, b. a. (2013). management of electronic serials in academic libraries: the results of an online survey. serials review, 39(4), 216-226. http://dx.doi.org/10.1016/j.serrev.2013.10.004   reviewed by: laura costello head of library materials and acquisitions teachers college columbia university new york, new york, united states of america email: lac2184@columbia.edu   received: 11 jun. 2014   accepted: 13 aug. 2014    2014 costello. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to examine industry standards for the management of electronic serials and measure the adoption of electronic serials over print.   design – survey questionnaire.   setting – email lists aimed at academic librarians working in serials management.   subjects – 195 self-selected subscribers to serials email lists.   methods – the author created a 20 question survey that consisted primarily of closed-ended questions pertaining to the collection demographics, staff, budget, and tools of serials management groups in academic libraries. the survey was conducted via survey monkey and examined using the analytical features of the tool. participants remained anonymous and the survey questions did not ask them to reveal identifiable information about their libraries. main results – collection demographics questions revealed that 78% of surveyed librarians estimated that print-only collections represented 40% or fewer of their serials holdings. the author observed diversity in the factors that influence print to digital transitions in academic libraries. however 71.5% of participants indicated that publisher technology support like ip authentication was required before adopting digital subscriptions.   a lack of standardization also marked serials workflows, department responsibilities, and department titles. the author did not find a correlation between serials budget and the enrollment size of the institution. participants reported that they used tools from popular serials management vendors like serials solutions, innovative interfaces, ebsco, and ex libris, but most indicated that they used more than one tool for serials management. participants specified 52 unique serials management products used in their libraries. conclusion – in surveying academic librarians engaged in serials management, the author sought to identify trends and standards in the field, but instead found significant variation in serials budgets and processes amongst the responding libraries. while it is clear that electronic subscriptions are a significant development and now a permanent feature of serials management, decisions to move from print to digital are complex and definitive conclusions about best practices for serials transitions could not be drawn from this study.   the survey revealed that institutions have invested in staff and tools for the management of electronic serials, but staffing configurations and tool combinations are also extremely diverse. the author concluded that the lack of standardization in these areas and the disconnect between institution and serials budget size indicated a serials landscape that was highly individualized and customized to each institution’s unique needs.     commentary   this survey was ambitious and covered several important factors in modern serials management, but it may have raised more questions than it answered about the way libraries choose, conceptualize, moderate, and maintain their serials departments and collections in an increasingly digital landscape. the conclusion that serials management processes are very individualized and institution-specific seems accurate, but it is important to consider the unprecedented choice that libraries have enjoyed in vendors, access types, and package configurations since electronic serials first became accessible. perhaps, rather than the assertion the author makes that electronic serials management processes are on a trajectory from disordered to standardized, library collections are moving from standardization enforced through lack of options towards the customized collections they have always tried to cultivate.   the case for disordered-by-design electronic resources management is shown through earlier surveys. both srivastava and taglienti (2005), and collins (2008) showed similar diversity in practice and departmental organization. the trajectory from earlier surveys to the present work does not seem to indicate significant desire or action towards greater standardization in serials management processes.   the diversity between libraries can also be seen in the institutional factors and term definition differences observed in the study. the author reported that he received feedback from participants that some of the terms suggested in the multiple choice questions were confusing or specific to the point of unnecessary exclusion. this is particularly apparent in the section devoted to titles and team organization. the survey seemed to operate under the assumption that staff members and departments were uniquely dedicated to serials management, but this was not always the case. in some institutions, serials management and electronic services maintenance was a task shared across multiple teams or the entire staff.  the fact that several specific tools, like serials solutions 360 core, were common, but particular combinations of serials management tools were unique to each institution may reflect the relatively recent appearance of electronic serials in libraries. tool configurations may have been developed to suit existing staff workflows, rather than libraries recruiting new teams and team members to operate specific management systems.   the lack of correlation between budget and institutional enrollment seems provocative, though this data could vary for many reasons such as a lack of well-defined terms for reporting. if it does stand it may only suggest an individualized balance at each institution between serials spending and spending on other formats.   to clarify the landscape of electronic serials management, more specific research is needed including more intensive data collection strategies like interviews with serials professionals and a more vigorous examination of tool use and staff organization. the author acknowledges that because the survey was voluntary and institutions did not provide identifying information, further study is needed before general conclusions can be drawn about serials management across academic libraries. this research is important and should be studied broadly. serials departments have had time to determine efficient workflows for the management of electronic serials and the study of this diverse landscape has potential implications for other types of collections, like monographs, that are only now embracing digital access on a large scale.     references   collins, m. (2008). staffing trends and issues in e-resource management. in m. collins, & p. l. carr (eds.), managing the transition from print to electronic journals and resources: a guide for library and information professionals. (pp. 109–128). new york: routledge.   srivastava, s., & taglienti, p. (2005). e-journal management: an online survey evaluation. serials review, 31(1), 28-38. doi:10.1016/j.serrev.2004.12.002   editorial   the evidence of change   alison brettle editor-in-chief school of nursing, midwifery and social work university of salford, united kingdom email: a.brettle@salford.ac.uk      2014 brettle. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     welcome to the march 9(1) issue of eblip; our first issue for 2014.  spring is traditionally the time of year associated with change and the eblip journal has been no exception.  we would like to welcome on board a number of new members to our editorial team and contributors to the journal.  first of all we would like to welcome lisl zach, our new associate editor (articles). lisl is from drexel university, philadelphia, usa, a library and information science educator with a range of research interests, who will work alongside lorie kloda.  finally we are welcoming on board a number of new evidence summary writers; there are too many to mention individually but we are pleased that they will be making an excellent contribution to the journal.  we also have to say some goodbyes.  first of all to pam morgan, our indexer who has been with us since the inception of the journal.  i would like to thank pam for all the work she has undertaken over the last eight years in making eblip and its contents visible to the wider audience.  secondly derek rodriguez, who was with us for a short time as associate editor (articles), had to re-evaluate his position at the journal due to increasing demands in his day job. thanks go to derek for his contribution in 2013.   change and evaluation are both consistent with eblip itself, as both are associated with the latter stages of the eblip process (apply findings to change practice and assess to evaluate the impact of that change).  the idea of translating or implementing evidence into practice is something that can be seen as a challenge or quite scary.  so i thought it may be useful to share some words of wisdom from evidence based health care writers which i think are equally applicable in libraries.  in relation to implementing evidence into practice “even with the best laid plans, if something can go wrong, it will” (hagedorn et al., 2006, s21) and “there’s no magic bullet for translating what is known from research into practice” (titler, 2008, p. 12).  fortunately the same writers offer some solutions “it is vital to be able to assess barriers to implementation quickly and correct for them” (hagedorn et al., 2006) and “to move evidence based interventions into practice, several strategies may be needed as what works in one context may not in another” (titler, 2008).  i prefer to sum these solutions up with my own words of wisdom for implementing evidence into practice “know your enemy” and “if at first you don’t succeed, then try and try again”.  so with these words of wisdom in mind, i challenge you to implement some evidence into your practice and evaluate the results.     references   hagedord, h., hogan, m., smith, j.l., bowman, c., curran, g.m., espadas, b., kimmel, b., kochevar, l., legro, m.w., sales, a.e. (2006). lessons learned about implementing research evidence into clinical practice: experiences from va queri. j gen intern med; 21(suppl. 2): s21-24. doi:10.1111/j.1525-1497.2006.00358.x   titler, m.g., (2008). the evidence for evidence-based practice implementation. in: r.g. hughes (ed.)., patient safety and quality: an evidence-based handbook for nurses. (chapter 7). rockville, md: agency for healthcare research and quality.   news/announcements   opportunities within the eblip journal      2013. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   the evidence based library and information practice journal would like to announce opportunities to fulfil three roles within the eblip journal: associate editor (articles), index support and evidence summary writers.  evidence based library and information practice (eblip) is a non-profit, open access journal and all positions are voluntary and unpaid.   associate editor (articles)   we are looking for a second associate editor (articles) to share the workload with the current associate editor (articles).   the associate editor (articles) is responsible for:   ·         overseeing the complete editorial process for items submitted to the articles section (this includes assignment of peer reviewers, monitoring the peer review process, communicating with authors and peer reviewers, making acceptance decisions, ensuring required changes to manuscripts are made, communicating with copyeditors and the production editor to ensure the final copy is as expected). ·         communicating and consulting with the editor-in-chief on a regular basis. ·         attending editorial team meetings, via skype, on a monthly basis and participating in the overall governance of the journal. ·         ensuring that all necessary deadlines are met. ·         communicating with potential authors and responding to queries.    the ideal candidate will be well-versed in evidence based practice and research methods. this position requires dedicated time on a regular basis, and it is therefore essential that interested persons ensure available time to devote to this position prior to applying. it is estimated that the workload is approximately 10-15 hours per month.   the current associate editor (articles) is happy to answer specific queries about the role on request.  please email lorie.kloda@mcgill.ca.   interested persons should send a statement of interest, summarizing experience and suitability for the role, as well as a brief resume to editor-in-chief alison brettle (a.brettle@salford.ac.uk) by january 13, 2014.     indexing support role   eblip is looking for a library and information professional with an interest in cataloguing and indexing to provide indexing support.   the eblip indexing support role involves 5-7 hours indexing per issue (4 per year), using author assigned keywords. the role would suit an lis professional with experience in cataloguing or indexing, familiarity with the doaj, comfort level with computers and the internet, knowledge of the various parts of a citation, and an eye for detail. although previous experience in cataloguing or indexing would be an asset, it is not a requirement.   there is also an opportunity to undertake some development work, which could include: a review of where the journal is indexed and adjustments as necessary and retrospective standardized indexing for previous issues and application of this to subsequent issues.   interested persons should send a statement of interest, summarizing experience and suitability for the role, as well as a brief resume to editor-in-chief alison brettle (a.brettle@salford.ac.uk) by january 13, 2014.   evidence summary writers   eblip seeks to add six writers to the evidence summaries team. evidence summaries provide critical appraisal syntheses for specific research articles. these research synopses provide readers with information regarding the original research article's validity and reliability, thus providing information on the presence or absence of evidence with which to make informed decisions. evidence summaries team members are required to write two evidence summaries per year, with a two year commitment to the journal. evidence summaries cover all areas of library and information studies and we encourage applications from information professionals in areas such as school, public, and special libraries, as well as academic settings. interested persons should send a statement of interest, indicating areas of strength they would bring to the role, as well as a brief resume to heather pretty (associate editor, evidence summaries) hjpretty@mun.ca by january 13, 2014. applicants who are shortlisted will be asked to submit a sample evidence summary.   ** for all positions only those applicants who are selected or shortlisted will be contacted by the editors. about the journal: published quarterly by the university of alberta, this peer reviewed, open access journal is targeted at all library and information professionals interested in an evidence based model of practice. by facilitating access to library and information studies research via original research articles and evidence summaries of relevant research from the library literature, evidence based library and information practice enables information professionals to practice their profession in an evidence-based manner. please visit the evidence based library and information practice website (http://ejournals.library.ualberta.ca/index.php/eblip) for further information about the journal. microsoft word es_furlan.doc evidence based library and information practice 2007, 2:2  97 evidence based library and information practice       evidence summary    virtual racism rears its head: uncovering librarian bias in e‐mail reference services    a review of:  shachaf, pnina, and sarah horowitz. ʺare virtual reference services color blind?ʺ library &  information science research 28.4 (sept. 2006): 501‐20.    reviewed by:   wendy furlan  liaison librarian, university of queensland library  brisbane, queensland, australia  e‐mail: w.furlan@library.uq.edu.au      received: 27 february 2007  accepted: 24 april 2007      © 2007 furlan. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to examine whether librarians  provide equitable virtual reference services  to diverse user groups.    design – unobtrusive method of defined  scenarios submitted via e‐mail.    setting – twenty‐three association of  research libraries (arl) member libraries  from across the united states. all arl  member libraries were invited to participate,  with the 23 acceptances providing 19%  participation.    subjects – anonymous librarians from the  23 participating libraries’ virtual e‐mail  reference services. up to 6 librarians from  each library may have been involved.    six fictitious personas were developed to  represent particular ethnic or religious  groups, whereby the ethnic or religious  affiliation was only indicated by the name  chosen for each user and the corresponding  e‐mail address. names were selected from  lists of names or baby names available  online: latoya johnson (african‐american),  rosa manuz (hispanic), chang su (asian ‐  chinese), mary anderson  (caucasian/christian), ahmed ibrahim  (muslim), and moshe cohen  (caucasian/jewish). these personas were  used to submit reference queries via e‐mail  to the virtual reference services taking part  in the study.    methods – five different types of reference  queries were developed for use in this study.  three were based on prior published  research as they were deemed to be  answerable by the majority of libraries. they  included a dissertation query, a sports team  query, and a population query all designed  to be tailored to the target institution. the  mailto:furlan@library.uq.edu.au http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:2  98 other 2 queries were developed with  participating institutions’ virtual reference  guidelines in mind, and were thought to not  be answered by the target institutions when  submitted by unaffiliated users. they  consisted of a subject query on a special  collection topic that asked for copies of  relevant articles to be sent out, and an article  query requesting that a copy of a specific  article be e‐mailed to the patron.    the study was conducted over a 6 week  period beginning the second week of  september, 2005. each week, 1 fictitious  persona was used to e‐mail a reference  query to the virtual reference service of each  of the 23 participating institutions. five of  each type of query were sent by each  persona. during september and october  2005, a total of 138 queries were sent. each  institution received a different query for  each of the first 5 weeks, and in the sixth  week they received a repeat of a previous  request with details of title or years altered.  all other text in every request sent was kept  consistent. each institution only received 1  request from each persona during the study.    in order to eliminate any study bias caused  by an informed decision regarding the order  in which personas were used, they were  randomly arranged (alphabetically by  surname). furthermore, to avoid suspicions  from responding librarians, queries were e‐ mailed on different days of the week at  different times. this created some  limitations in interpretating response times  as some queries were submitted on  weekends.    all queries were analysed by nvivo  software in order to identify attributes and  patterns to aid qualitative analysis. each  transaction (a single query and any related  responses) was classified according to 12  attributes and 59 categories based on  various associations’ digital reference  guidelines. transactions were coded and  then 10% re‐coded by a different coder. this  led to the clarification and refinement of the  coding scheme, resulting in the number of  categories used being reduced to 23. coding  was then performed in 3 iterations until 90%  agreement between the 2 coders was  reached. the final inter‐coder reliability was  92%. the study did not support cross  tabulation among user groups on most  content categories due to the small sample  size.    main results – response times varied  greatly between users. moshe  (caucasian/jewish) received an average  turn‐around of less than a day. at the other  end of the spectrum, ahmed’s (muslim)  responses took an average of 3.5 days. both  ahmed and latoya (african‐american) sent  queries which took over 18 days to receive a  response. the length (number of words) of  replies also indicated a differing level of  service with mary (caucasian/christian)  and moshe receiving far lengthier responses  than the other 4 personas. number of replies  (including automatic replies) was examined  in comparison with the number of replies  which answered the question, and again  indicated mary and moshe were receiving a  better level of service.     the way in which the user was addressed  by the librarian was examined as another  measure of service, i.e. first name, full name,  honorific. this again mirrored the low level  of service received by ahmed. the  professional endings used by librarians in  their replies also reinforced the high quality  of service received by moshe across other  categories.    results for rosa (latino) and chang (asian ‐  chinese) were average for most categories  presented.    conclusion – in this study, a discriminatory  pattern was clearly evident, with the  african‐american and muslim users  evidence based library and information practice 2007, 2:2  99 receiving poor levels of service from virtual  reference librarians across all dimensions of  quality evaluated. the caucasian (christian  and jewish) users also noticeably received  the best level of service. it is noted, however,  that the sample size of the study is not large  enough for generalisations to be drawn and  that future, more statistically significant  studies are warranted. many other  questions are raised by the study for  possible future research into racism  exhibited by library staff and services.    commentary      this study provides a relatively well  planned and considered look into the  discriminatory tendencies of reference  librarians when operating in the virtual  environment. the authors provide strong  linkages to professional association  guidelines from the international federation  of library associations (ifla) and the  reference and user services association  (rusa) together with a large body of  relevant research literature. the  methodology was relatively sound, with  most study aspects explained and  limitations stated and discussed. the  reference queries submitted to the target  library services are cleverly designed and  take advantage of prior reference service  research. the process of selecting library  services to be involved in the study drew on  a large pool of potential participants with all  arl libraries invited to participate but  reference librarians operating the service  being unaware of queries relating to the  study, enabling an unobtrusive approach. a  thorough collection of tables and graphs are  included in the article to outline the  procedure and results, which greatly assist  the reader in following the study.    while the study was generally well  conducted and backed‐up with references to  prior literature, it is not without fault. many  of the limitations and possible pitfalls  apparent in this study are identified and  discussed by the authors. the major  limitation of the study is the small sample  size, which the authors plainly state when  describing their methodology and talk about  further in the article’s discussion. the  discussion also suggests possible reasons for  negative service results for some factors  measured, including gender confusion of  user names, difficulties in distinguishing  first and last names of users, and variations  of query reply signatures due to  institutional policy. it is interesting to note  that although these limitations were openly  discussed, this is not reflected in the  concluding statements which, if read on  their own, indicate that virtual reference  librarians discriminate against muslims and  african‐americans and favour caucasians.     there are, however, several other factors to  do with the design, methodology and  reporting of the study results that are either  not specified or unclear. when stating their  research question at the conclusion of the  introduction, the authors indicate that they  are also examining possible gender bias;  however, this is not mentioned or reported  upon anywhere else in the paper. the  authors do not state whether the analysis of  e‐mail responses was blind or not, a possible  source of bias left unknown. the  methodology is obtusely written, which,  combined with a lack of specifics on  ordering of e‐mails and who sent what to  whom, makes the study quite difficult to  replicate. the selection and evaluation of the  ethnical representation of user names for the  study is also questionable, with no  validation for this process presented. the  authors delayed making linkages between  some factors evaluated in the study and  quality of service until the paper’s  discussion. several limitations of factors  measured are also not mentioned until the  discussion, although quite strong inferences  are drawn from them in the results section.  such relationships and limitations should  evidence based library and information practice 2007, 2:2  100 have been outlined earlier to enable a better  reading of results.    the way in which some data are graphically  presented and described in the results  section of the paper is questionable, and  raises concerns of possible author bias. for  example, the measure of response time is  quite well broken down in the second figure,  with the messages for each persona  displayed as percentages of messages within  each given response time (i.e. 70% of  chang’s messages were responded to on the  same day). however, the first figure used to  represent this data simply displays the  average response times for all messages sent  by each persona. presenting the data in this  way enabled possible response anomalies to  greatly skew the average response times  represented by the graph. furthermore, the  authors have omitted any comment on the  response times for the latino and asian  personas, which scored quite well. these  two personas seem largely unaddressed, as  are negative results for the jewish persona  and positive results for the muslim persona.  this results in a very strong impression of  service bias to the reader if graphs are not  closely examined.  direct links to existing professional  guidelines and research is extremely evident  in this paper. furthermore, the authors give  substantial consideration to how this  research can be utilised by the profession  and expanded upon in future studies. five  points are clearly included towards the end  of the discussion with direct ideas of what  libraries and lis schools can do to improve  equality of physical and virtual services,  including training, service evaluations,  performance evaluations, emphasising the  importance of equitable services and  employing minorities. further research  ideas include using a more statistically  significant sample, examining other library  services, studying services in other library  sectors, considering geographical location  and its relation to bias, or comparing public  and privately run higher education  institutions.     overall, this study certainly raises some  questions about whether or not bias is  evident in library services, and gives some  practical ideas for how a library can  evaluate and improve equality of services.  research article   library usage study, the how and what: a survey of space usage at a mid-sized research library   aaron f. nichols library associate professor, access and media services librarian david w. howe memorial library university of vermont burlington, vermont, united states of america email: aaron.nichols@uvm.edu   paul p. philbin library associate professor emeritus david w. howe memorial library university of vermont burlington, vermont, united states of america email: paul.philbin@uvm.edu   received: 9 feb. 2022                                                                     accepted: 10 oct. 2022      2022 nichols and philbin. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30103     abstract   objective – the research was conducted to understand better how and what spaces are used in a mid-size academic library. also, the authors were interested in their users' spatial likes and dislikes and why they gravitated to or avoided specific spaces or floors. the authors also found an opportunity to examine recent renovations that added a connector bridge to a first-year student dorm and the subsequent increase in foot traffic to evaluate its success in meeting users' needs for varied and productive study spaces across the building.   methods – the study used a survey to gauge user satisfaction with the library's space and environment for research, study, and collaborative work. the authors hand-distributed a survey with five multiple-choice and three open-response questions to users over three days (monday-wednesday) between 10 am 4 pm, the busiest days and times in a typical week. the collected surveys were sorted and coded in an excel spreadsheet and uploaded and analyzed in jmp pro.   results – the 298 completed responses came from undergraduate students (n=281) who visited the first floor, identified as a collaborative study space (n=144). respondents showed that they visit the library daily (58%, n=173) and weekly (34%, n=104). most of the survey participants (98%, n=293) indicated that they pursued academic work in quiet spaces they occupied (75%, n=224). interestingly enough, the noisiest and quietest floors are the areas most avoided, the first floor-collaborative, noisiest space (54%, n=161) and the third floor-designated as quiet space (18%, n=55). the final survey question invited the respondents to "sound off," with 135 responding; 107 (79%) of them opined on improvements to existing study spaces within the library.   conclusion – this research demonstrated that students value the library as a place to study but are critical of excessive noise and overcrowding in the designated collaborative study areas. academic libraries should consider balance when designing library study spaces. librarians and space designers should strive to strike an appropriate balance between seating quality and quantity, acceptable noise levels in designated collaborative and quiet study spaces, and the impacts of environmental factors such as printers, food services, exhibits, art displays, restrooms, and walkways through library study spaces within the library.     introduction   the university library provides information, services, and support to the university’s 12,327 students, with about half living on campus. undergraduates make up 10,700 of the student body, and graduate programs bring in 1,627 graduate students. the medical library, which is not a part of this research study, serves a small and robust medical school and research centre. the university is a public land grant university located in the northeastern united states. the vast majority of the students attend full-time and are under 25 years old. the library building also houses non-library entities such as the instructional support services, a writing centre, classroom technology services, student accessibility services, and a coffee and snack bar with seating.   the four-floor building is used heavily throughout the academic year, with peak daily gate counts exceeding 9,000 during the 2018 and 2019 academic years. the library uses the traf-sys people counting systems to count entries and exits. when the pandemic reduced building capacity, we upgraded to their safecount system to monitor live occupancy.   problem statement   since 2016, circulation staff conducts an annual density count during the first or second week in november to determine where library users congregate. six years' worth of data leads us to this study to determine who, what, and why individuals use library spaces. in the interim, the university built a new residence hall for first-year students adjacent to the library with a connecting bridge and renovated space in the library. the addition and renovation created a second entrance and exit to the library and relocated a service point from the lower level. the renovations included a more spacious center for multimedia development and a reconfigured gallery for multi-cultural art. the reconfigured spaces altered traffic flow and increased seating capacity on the first floor.   like many multi-level academic libraries, the first floor of the library hosts a collaborative social learning environment including a blend of group seating and soft chairs, printers, computing stations, three public service points, one group study room, and a coffee and snack bar. when the study was conducted, the library was open seven days each week during the academic year with the library opening at 8 am and closing at 12 am on most weekdays. total seating on the first floor was 325 and head-count data revealed that seating was often full or overflowing during peak day-time hours, with 352 counted. the second floor is a designated quiet study floor with 358 total seats including large multiple seat desks, soft seating, and individual study carrels. also included on the second floor are six sound-proof group study rooms, a large silent study room, book and journal stacks, current periodicals, and computer workstations. head-count data reveal that peak usage occurs during day-time hours, with 258 filled seats for a peak count. the third floor is designated for quiet study and is nearly silent with little foot traffic. the third floor contains 403 total seats, book stacks throughout the centre of the floor, one group study room, and a suite of administrative offices. seating consists of a few large multi-seat tables, individual study carrels, and is heavily dominated by simple single-seat desks, which are placed side-by-side along the perimeter of the floor. head-count data reveal more seating than demand, with a peak head-count of 124 seats filled during peak operating hours. wi-fi is excellent throughout the building.   since the construction of the connector bridge and the subsequent increase in foot traffic, no formal evaluation of the library space has been conducted. while we have population density studies that tell us the number of users distributed throughout the various areas in the building, we had no qualitative data from our users. previous libqual studies provide excellent general information about the library. still, they do not give enough detail to pinpoint the aspects our users like and dislike about the spaces in the building. anecdotes from librarians and staff painted a dismal picture of the first floor's overcrowded, chaotic, and boisterous environment. opinions from librarians and staff were firmly in favour of creating a strict noise policy to facilitate a quieter and less crowded first floor, but we needed to hear directly from the students who use the spaces throughout the building to determine if they share the same assessment of the library spaces as our librarians and staff.   to determine new space configurations and policies that work for our users, we needed to hear what they see as good and bad aspects of the library’s study environment. we decided to conduct a qualitative study with several guiding research questions:   1)       which spaces do users avoid? why? 2)       which spaces are users drawn to? why? 3)       what activities do users engage in their chosen space? 4)       do these spaces work well for library users? what can be improved?   literature review   over the past three decades, the academic library has evolved from the traditional library, a mausoleum of knowledge and tranquil study, to the technology-enhanced and highly collaborative learning commons model. many academic libraries have provided library space to non-library entities (lux et al., 2016). they have emphasized the creation of a technology-rich environment and social learning spaces (bostick & irwin, 2014). throughout the early 2000’s, academic libraries redesigned their spaces to incorporate technology, group study spaces, areas for socializing and informal study, and eating (freeman, 2005; jamieson, 2006).   these trends in social study spaces and non-library services relocating to the library are not universally accepted. for example, james (2013) documented a conflict that emerged when faculty who were displeased with the learning commons established at east carolina university voiced their concerns about the future of the library. gayton (2008) argued that library users value the traditional communal academic library model and that the developing social library model threatens the communal spirit and harms the experience. with these significant changes in academic libraries, we have seen many opportunities for research and debate.   research and user feedback studies demonstrated that library redesign efforts were viewed positively by library users (jamieson, 2006) and that libraries have experienced increased use of physical spaces following a redesign (shill & tonner, 2004). spatial preferences and the attributes that determine users’ spatial choices are important factors in determining the overall quality of library study spaces. cha and kim (2015) discovered that attributes such as abundance of space, noise level, crowding, comfortable furnishing, and cleanliness were primary factors in users’ spatial choices. lux et al. (2016) found that library users overwhelmingly use library spaces for study and that individual study spaces are very much in demand. similarly, applegate (2009) found that the library is primarily used as a study space and suggested a blend of study environments to accommodate a wide range of student needs. noise levels play a factor in the quality of library spaces. pierard and baca (2019) found that a moderate noise level is ideal for collaborative study areas and stemmer and strawser (2019) observed that students highly value quiet study spaces. research regarding user preferences and their views on the quality of library spaces provides valuable insights for successful remodeling and spatial design efforts. this study adds to the current body of research by examining user activities, preferences, and dislikes for spatial choice and study environment.   the groundbreaking ethnographic study by foster and gibbons (2007) at the university of rochester library inspired many ethnographic studies in academic and public libraries across the united states. the ethnographic research in illinois academic libraries project involved five illinois academic libraries. it used interviews, photo journals, and mapping diaries to better understand students’ work behaviours in the libraries and applied those findings to library design initiatives (asher et al., 2010). pierard and lee (2011) used ethnographic methods such as photographic observation, photo diaries, and flipcharts in conjunction with a traditional user survey to learn how library spaces were being used and what users considered to be an ideal library space. the researchers used their findings to change public spaces while staying within a modest budget. bedwell and banks (2013) made important discoveries in how the library environment impacted student behaviour and study patterns when they employed students from a sociology and social anthropology class to observe how students use space in the dalhousie university library. these studies have all been instrumental in understanding what users are doing in library spaces and how they use them.   long-range multiyear studies such as those conducted by gerke and teeter (2017) and harrop and turpin (2013) effectively employed mixed methods to measure user interactions in their physical library spaces. lux et al. (2016) combined an exit survey with observation and noted that focus groups might have added more depth to their study. hillman et al. (2017) conducted a comprehensive mixed methods study involving librarians, students, and sociology faculty to conduct seating sweeps to map patron activity, conduct student-led focus groups, and implement a survey based on the findings of the seating sweeps and focus groups. at the university of iowa, thomas et al. (2015) paired an online survey with observational headcounts to assess the purpose and demographics of their learning commons use. at penn state, lynn (2011) conducted a needs assessment in advance of a building renovation using an online survey tool and student and faculty focus groups.   not all studies need to be labour intensive or complex to give quality data. short and simple studies have yielded excellent data for many academic libraries. bailin (2011) conducted short interviews with students to better understand their space needs, how the library was meeting those needs, and to evaluate the success of the library remodel. stemmer and strawser (2019) implemented a survey to gauge the needs and desires of students in advance of a library renovation project and used the data to inform the remodeling process. cha and kim (2015) and gardner and eng (2005) also had success collecting useful qualitative and quantitative data on library space attributes users value and need with paper-based surveys. hedge et al. (2018) collected insightful data on patron activities, elements of an ideal study space, and preferences for environmental factors such as noise and lighting in the library, using an online survey instrument. the effectiveness of these simple study methodologies coupled with a lack of time and resources to plan and execute a more exhaustive study led us to employ a simple survey to harvest user feedback.   methods   during three consecutive days for the week of october 7, 2019, a survey was conducted to gather information about the current space usage of the library. the authors designed an eight-item survey instrument based on the study's goals. it consists of five multiple-choice questions and three open response questions. the complete survey is found in the appendix. institutional review board exemption for the study was received in august 2019. survey collection began on monday, october 7, 2019. it ended wednesday, october 9, 2019. from 10 am to 4 pm during those days, the authors hand distributed the surveys, attempting to distribute at least 50 surveys per pre-identified zone. we selected dates in early october because it was roughly five weeks into the sixteen-week semester. this is a period of time when library business is steadily busy but not a period, like final exam or mid-term periods, in which library use is unusually high. time and resources did not allow for survey distribution during evening and weekend hours, and we therefore selected the 10 am – 4 pm time period to distribute surveys because it was the busiest time of day and would yield the most survey results.   on monday, october 7, 2019, the authors distributed at least 350 surveys to library users on the first floor in seven identified zones. on tuesday, october 8, 2019, 300 surveys were handed out on the second floor, covering six zones. finally, on wednesday, october 9, 2019, 150 surveys were distributed on the third floor in three zones. some spaces were so quiet that the authors left surveys on tables with printed instructions. survey participants were instructed to deposit completed surveys in locked ballot boxes placed strategically throughout the library. each survey was identified with a floor and zone designation in the footer of the survey document. the completed surveys were sorted and coded in an excel spreadsheet. the spreadsheet data were uploaded to jmp pro for analysis.   results   demographic of participants   responses came primarily from undergraduate students (n=281). graduate students (n=12), community users (n=3), faculty (n=1), and staff (n=1) were all underrepresented in relation to their numbers across campus. however, data from past libqual studies as well as anecdotal evidence suggest that undergraduates are indeed the dominant user group of the library.   responses were spread across all areas of the library. library users from all three main floors participated in the survey. table 1 shows the floors with the most respondents. overall, we were pleased to see participants from a wide range of floors and seating areas.   table 1 number of participants floor surveyed participants 1 (collaborative) 144 2 (quiet) 74 3 (quiet) 80 total 298   visit frequency   after demographic data questions, the survey asked participants how frequently they use the library area/space they are in. participants tended to use the area/space they occupied with regular frequency, with 58% (n=173) responding that they visit the library area/space daily and 34% (n=104) responding that they use the library area/space on a weekly basis. a small minority of participants responded that they use the library area/space infrequently, with only 15 participants responding that they visit the area/space monthly or less than monthly, and six responded with “other.”   visit frequency was similar for participants across all three floors. participants who responded that they visit the area/space daily were proportionately represented across the collaborative study first floor (58%), quiet study second floor (55%), and quiet study third floor (60%). weekly users saw a similar, equal distribution across the first (34%), second (38%), and third (34%) floors. table 2 shows participant visit frequency broken down by the floor they were surveyed on.   table 2 visit frequency floor surveyed daily weekly monthly less than monthly other questionnaire responses 1 (collaborative) 84 49 4 4 3 144 2 (quiet) 41 28 2 1 2 74 3 (quiet) 48 27 3 1 1 80 total 173 (58%) 104 (34%) 9 (3%) 6 (2%) 6 (2%) 298   the high rate of visit frequency demonstrates that the library attracts and retains a loyal following and that participants tend to return to the spaces they use. while these data alone do not point directly to a satisfied user base, they do demonstrate that the library offers resources and/or services sought by undergraduate students.   what users are doing in the library   one of the driving research goals of this study is determining what library users are doing while they spend time in the library. the survey presented a simple multiple response questions “what are you doing in this space?” participants were given several response options, including an open response, from which they were invited to select all responses that apply.   most of the participants (98%, n=293) responded that they were pursuing academic work while in the library. personal work is the next most popular activity with 37% (n=109) of participants selecting this option. surprisingly, almost a quarter of all participants (23%, n=69) responded that they were socializing while in the library. leisure reading (7%, n=22) and the open response option (5%, n=16) were the least frequently selected options.   the high number of participants engaging in social activities makes more sense when looking at the data broken down by floor. participants surveyed on the first floor, which encourages collaborative study and sees significant foot traffic, all but owned the social activities with 34% (n=49) of first-floor participants responding that they engage in social activities while in the library. participants on the second floor, which is a designated quiet floor, responded with a surprising 19% (n=14) indicating they indulged in social engagement. only 7.5% (n=6) of participants surveyed from the third floor claimed they engaged in social activities while in the library.   table 3 participant activity floor surveyed academic work personal work social leisure reading other questionnaire responses 1 (collaborative) 142 50 49 10 10 144 2 (quiet) 71 29 14 3 2 74 3 (quiet) 80 30 6 9 4 80 total 293 (98%) 109 (37%) 69 (23%) 22 (7%) 16 (5%) 298   why users choose the space they occupy   our next question asked participants why they chose to use the space they occupied. the survey asked participants “why do you use this space/area?” and allowed participants to select multiple responses including an open-ended “other” option.   many participants (75%, n=224) selected “quiet” as the reason for selecting the space they occupied. other popular selections included “furniture” (49%, n=146) and “outlets” (42%, n=125), with “lighting” (27%, n=81), “fewer noise restrictions” (25%, n=76), “other” (21%, n=64), and “access to technology” (20%, n=61) as less popular yet compelling and statistically significant reasons.   broken down by floor, the distribution of responses is somewhat different. participants (n=144) on the collaborative and often noisy first floor selected “furniture” (56%, n=81) most frequently and, quite surprisingly, they chose “quiet” (55%, n=79) with almost the same frequency. all other reasons for using spaces on the first floor were selected at similar, statistically significant frequencies: “outlets” (35%, n=50), “fewer noise restrictions” (32%, n=46), “access to technology” (30%, n=43), and “other” (29%, n=42).   table 4 why participants choose the space they occupy floor surveyed quiet furniture outlets lighting fewer noise restrictions other access to technology questionnaire responses 1 (collaborative) 79 81 50 39 46 42 43 144 2 (quiet) 69 37 44 24 14 9 11 74 3 (quiet) 76 28 31 18 16 13 7 80 total 224 (75%) 146 (49%) 125 (42%) 81 (27%) 76 (25%) 64 (21%) 61 (20%) 298   participants on the second floor (n=74), which is a designated quiet floor, selected “quiet” (93%, n=69) with overwhelming frequency. “outlets” (59%, n=44), “furniture” (50%, n=37), and “lighting” (32%, n=24) were also strong attributes that attracted participants to second-floor spaces. not surprisingly, with the second floor offering a small number of computer terminals without printers or specialized technology, responses such as “fewer noise restrictions” (19%, n=14), “access to technology” (15%, n=11), and “other” (12%, n=9) were not significant factors for most second-floor participants.   third-floor participants (n=80) sought “quiet” (95%, n=76) above all else. “outlets (39%, n=31) and “furniture” (35%, n=28) were also important reasons participants chose third-floor spaces. “lighting” (n=18), “fewer noise restrictions” (n=16), and “access to technology” (n=7) were not factors for most participants.   user needs   our results showed that participants come to the library for a variety of needs. quiet study (84%, n=250), printing (66%, n=197), and collaborative study (46%, n=136) were the needs most frequently selected by participants. standard technologies such as computers (19%, n=58) and scanning (12%, n=35) were selected with modest frequency, while more specialized technologies such as the 3d printer (2%, n=5) and microform scanners (1%, n=3) were in exceptionally low demand. participants cited other needs such as additional electrical/usb outlets (2%, n=7) and more comfortable seating (2%, n=5) in the open-ended comment section.   the first floor of the library is the central printing hub for all of campus and, as we expected, printing (72%, n=104) was the most frequently selected need by first-floor participants. however, we were surprised to see that participants surveyed on the collaborative and often noisy first floor selected quiet (70%, n=101) with remarkably high frequency, even more so than collaborative study (58%, n=83). one possible explanation is that the survey was taken at a time of year when collaborative projects are not frequently worked on. it is also possible that participants on the first floor think of quiet in relative terms, as one participant remarked “it [the library] is quieter than my dorm.” it is also possible that the first floor is quiet enough for most students during certain times of the day and of the year. other than printing, participants saw modest interest in technological offerings such as scanning (14%, n=20), computers (26%, n=37), 3d printing (2%, n=5), and microform scanners (1%, n=3).   participants on the second floor overwhelmingly selected quiet (95%, n=70) as a need. collaborative study was selected at a moderately high rate (38%, n=28). printing (59%, n=44) dominated technological needs, while computers (21%, n=16) and scanning (12%, n=9) were modestly selected. all other options were selected infrequently.   all respondents on the third floor selected quiet (99%, n=79) as a need. like the first and second floors, printing was selected with strong frequency (61%, n=49) by third-floor participants. collaborative study (31%, n=25) was the next most frequently selected need. scanning, computers, microform readers, and 3d printing were selected with little frequency.   table 5 participant needs floor surveyed quiet printing collaborative study computers scanning other microform scanner 3d printing questionnaire responses 1 101 104 83 37 20 17 2 4 144 2 70 44 28 16 9 7 1 1 74 3 79 49 25 5 6 5 0 0 80 total 250 (84%) 197 (66%) 136 (46%) 58 (19%) 35 (12%) 29 (10%) 3 (1%) 5 (2%) 298   the next question, which was designed as a follow up to the multiple response question 5, asked participants “are we meeting those needs?” with an open-ended response option. there was also room for participants to add commentary to their response. of the 298 total participants, 251 (84%) answered “yes,” the library is meeting their needs. only four (1.3%) participants responded with “no” and 43 (14%) neither responded yes nor no. most of the comments centered around improved study space needs (n=74) such as more or larger tables, more collaborative study space, more seating, more outlets, comfortable seating, better lighting, and complaints about crowding and noise. we saw complaints (n=25) about printing, which is understandable given a rash of mechanical failures we dealt with due to worn out machinery during the survey.   areas avoided   the collaborative first floor is the most avoided floor with 54% (161 of 298) of all participants identifying the first floor, or specific areas within the first floor, as an area they avoid at some point. first-floor areas such as the coffee and snack bar (n=11), front lobby (n=7), and reference computer area (n=11) were pointed out as places avoided by participants. surprisingly, 39.6% (57 of 144) of participants surveyed from the first floor avoid the first floor at some point. the most frequently cited reason participants avoid the first floor is that the space is too loud (66%) and crowded (28%).   the quiet third floor is the next most avoided space with 18% (n=55) of participants noting that they avoid that floor. most of those who avoid the third floor do so because they find it is too quiet (78%, n=43) for their liking. other reasons participants avoid the third floor do not show up with much frequency and are therefore statistically insignificant. all the participants who avoid the third floor were either surveyed from the first floor (n=43) or the second floor (n=12) and none of the participants who were surveyed on the third floor avoided the third floor.   the second floor is the least avoided area with only 9% (n=27) of participants claiming that they avoid the area. most of the survey takers who avoid the second floor were surveyed from the first floor (n=19) and claimed that they avoid the floor (n=12) because it is too quiet. only one participant surveyed on the second floor and seven surveyed from the third floor avoid the second floor.   table 6 areas avoided floor surveyed avoid floor 1 (collaborative) avoid floor 2 (quiet) avoid floor 3 (quiet) avoid ground floor questionnaire responses 1 57 19 43 5 144 2 42 1 12 1 74 3 62 7 0 4 80 total 161 (54%) 27 (9%) 55 (18%) 10 (3%) 298   sound off   the final survey question invited participants to “sound off” in an open-ended response about anything they wish. of the 135 responses, 107 (79%) opined on improvements that can be made to existing study spaces within the library. demands for more spaces and furnishing types such as more tables (n=23), more soft seating (n=10), more seating in general (n=14), more collaborative study space (n=5), and more private study space (n=8) were made clear by participants. participants also asked for improvements to the quality of study spaces such as better lighting and more natural light (n=12), more electrical outlets (n=16), a more updated look (n=7), and better temperature regulation (n=5). participants did not mention many grievances about noise (n=6) or crowding (n=9), and complaints regarding printing (n=7) seem to have been exhausted in previous sections.   discussion   the study sought to determine why and how users are using study spaces in the library and whether the spaces they are using are satisfying their needs.   noise level and crowding on the collaborative study floor   the first floor is the largest open space in the library and serves as the only open study space that encourages collaborative study without noise restrictions. anecdotal observations from librarians and staff that first-floor users are using the first floor as a recreational area are largely untrue. our study revealed that nearly all of the first-floor participants were engaged in academic work and that these users value both quiet and collaborative study. indeed, some of the first-floor participants did reveal that they are there to socialize in addition to pursuing their studies, but very few indicated that they were there strictly for socializing. while the perception that first-floor users are primarily interested in socializing is patently wrong, complaints from librarians that noise is an issue on the first floor are strongly supported by the participants who were surveyed on the first floor. participants surveyed from all floors, including the collaborative and noisy first floor, find the first floor too loud for their liking. crowding was another problem that was widely identified by first-floor participants and indeed, many library staff from the first-floor service points have identified crowding as a consistent problem in the first-floor lobby, walkway, and study areas. research by stemmer and strawser (2019) and vondracek (2007) found similar patterns at their libraries, with significant numbers of respondents indicating that the learning commons were too loud, crowded, and distracting for productive work.   the first floor provides much-needed and wanted collaborative study space for students. to improve the user experience, crowding and excessive noise must be addressed. research conducted by pierard and baca (2019) and mehta et al. (2012) suggests that a moderate ambient noise level, about 70 db, is ideal for collaborative learning spaces. with this in mind, we recommend studying the noise levels on the first floor with a decibel meter to gain an accurate assessment of when and where noise may be problematic.   we do not see a single approach as being effective in mitigating excessive noise on the first floor. indeed, research has demonstrated that a multiple approach strategy is most effective in mitigating excessive noise levels. these approaches include expectation-setting, self-monitoring through noise monitoring displays such as noisesign, gentle and consistent staff intervention, the development of positive marketing campaigns to help users find appropriate study spaces in and outside of the library, and addressing building and seating design problems that enable excessive noise and crowding (pierard & baca, 2019).   creating barriers or physical delineations between study areas and the hectic and highly social lobby and thruway areas would be a good first step. many of the desks on the first floor are crammed together and the computer pods in the reference section frequently see groups of students on top of one another. freeing up space on the first floor to create higher quality collaborative study areas with reasonable space between group tables might also be helpful in reducing overcrowding. pierard and baca (2019) noted that such rearrangement of furnishings and redesign can be successful in solving noise problems with design rather than policy.   participants on the designated quiet study second and third floors were satisfied with the level of noise on their respective floors. there were only a few complaints about noise occasionally becoming an issue on these floors. while crowding was not frequently cited as a major issue for these floors, some users did claim that the second floor is often too crowded. observations from circulation staff and population density data bolster the crowding claim, especially in the middle and end of each semester. in related studies, applegate (2009) found that students involved in quiet individual study prefer not to sit next to one another, that a seat or more apart is ideal for focused study. stemmer and strawser (2019) found that students desire more room and more quiet study space. the user experience might be enhanced by creating more space for users to spread out and have a bit more self-space. while adding more seats might be tempting, with the limited available seating on the quiet floors in the library, we should focus more on the quality of the individual study spaces by creating larger, more spacious work areas with more room between adjacent tables, desks, and carrels. and while users tend to self-enforce noise on these quiet floors, signage on the large multi-seat tables might be helpful to remind groups of students not to engage in conversation or intense whispering.   technology   printing is the technology most used by survey participants. this is certainly backed up by data from our pharos print management system and local printer leasing vendor, which indicate that printers in the library are some of the most used in the entire state. while some academic libraries have made efforts to reduce printing and printing waste due to environmental and cost-recovery concerns (ashmore & morris, 2002; calloway & callahan, 2003), the university library has made strides to make printing more accessible to students by dropping printing fees from ten cents per page to five cents per page and offering free printing days at the end of each semester. although students and faculty are actively engaged in environmental campaigns such as banning the sale of bottled water on campus and divestment from fossil fuel industries, the demand for paper printing has not attracted much attention. in fact, the student government has asked that printing become more accessible by reducing or eliminating printing fees. a campus study targeted at students and faculty’s printing habits and needs could be useful in understanding the importance that this service plays for the campus community and whether financial and environmental waste are concerns.   computers and scanning equipment saw modest rates of reported use. in fact, logins for library-provided computers have seen a significant decline over the past three years in the library. yet the computer pod seating in the reference area is frequently overcrowded during peak library hours throughout the academic year. this is substantiated, albeit anecdotally, by reports from public services staff as well as commentary from this survey that computer seating is frequently occupied by patrons who are not making use of the library-provided computers.   the declining need for library-supplied computers coupled with a need for more seating should prompt a redesign of the collaborative first-floor computing area. eliminating collaborative study floor computers and replacing the densely packed pod-type desks with large tables with generous egress between them will enhance the user experience by reducing overcrowding. it could reduce noise levels as groups will not feel compelled to talk over one another. the existing library-supplied computers on the first and second floors that are intended for printing and quick look-up access should remain as their need is still strong enough to justify their location in those spaces.   using data for reopening during the covid-19 pandemic   social distancing guidelines and a strict building capacity limit dictated by the state made reopening the library a challenge. in-person socializing and group study were not feasible under the social distancing guidelines in place between august 2020 and may 2021. a building capacity limit of 225, where we regularly see over 800 at peak hours, required removing desks, chairs, and soft seating. data collected and analyzed from this study were used to inform the functionality of the "new normal" library for reopening at the start of fall 2020 and through spring 2021 semesters.   mask wearing and social distancing were mandated by the university and state officials. but it was up to us to reopen the library and implement policies designed to encourage social distancing and face coverings, discourage group work, and foster an environment for research and study that would benefit most students on campus. this study's data revealed that many participants do not like the excessive noise and social atmosphere on the collaborative first floor. in response to this, we implemented a quiet study policy for the entire library. the designated "quiet" second and third floors remained silent study areas. the previously noisy and collaborative first floor was transformed into a mostly quiet work area where users could talk or participate in online courses at a low conversational level. based on our students' observational data and verbal gratitude, we can see that this new pandemic-specific policy was well-received. items identified in the study as having low value to users, such as computers and small tables, were removed from the library to give users access to things participants think have a higher value such as larger desk spaces and printers.   using data for post-pandemic (vaccination) reopening   with the removal of state-mandated social distancing guidelines and building capacity restrictions, campus administration planned for a fully in-person campus experience for the fall 2021 semester. we were instructed by campus officials to open the building to all visitors free of nearly all of the previous year’s covid-19 restrictions, with mask wearing being the exception. despite an overwhelming vaccination rate among students, faculty, and staff, campus administration mandated a strict mask-wearing policy for all occupants in campus buildings, including the library. with the removal of the past year’s restrictions, students were welcome to engage in collaborative in-person study and enjoy eating in the library building once again.   data from the study revealed that many students come to the library as a destination for collaborative study. at the same time, data revealed that overcrowding and noise were major problems for most students. to accommodate the overwhelming desire for less crowding and less noise, in addition to the need for collaborative study, we made efforts to improve the quality of the study spaces by sacrificing the number of seats in the building. for the fall 2021 semester, we reduced seating in the building from 1288 seats to 670 seats. large tables on the first floor that once had eight seats crammed into a table now have four seats but offer a much larger table surface to work on and a less crowded study experience.   changes to our technological offerings also helped to reduce crowding issues in the building. many of our printers and scanners were in the middle of prime collaborative study space. printers are noisy machines that often see significant queues build up and require frequent maintenance. to address the noise and crowding around printing and scanning activities, we moved all printers and scanners to the already noisy coffee and snack bar. the result, coupled with the reduction of first-floor computers from 120 to 50, allowed us to make room for more open desk space on the collaborative first floor. data gathered from diminishing login numbers coupled with a new university mandate that all students purchase a laptop tell us that the reduction in machines will not be missed by our students.   at this point, it is too early to draw conclusions on the impact of these changes. we have observed significantly less crowding in the building. groups of students are studying on the collaborative first floor and there appears to be more productive study and less frivolous socializing. this is especially true in the areas of the first floor that offer large multi-seat tables. we have observed some of the computer pods on the first floor tend to be more social and over-crowded than the large group tables. while the reduction in seating and relocation of printers may have helped to reduce noise and crowding, there are other factors at play. the campus-wide mask mandate and fear of covid-19 might have kept some students away from the building. the university also has a large and academically proficient first-year class and a shift of focus on academic achievement may also be a factor in the changes we have observed. another, more tangible factor is the closing of the coffee and snack bar. in pre-pandemic times, the coffee and snack bar could attract over 45,000 visitors per semester. we need at least one complete academic year post-pandemic, without restrictions, to determine how students use spaces optimally. keeping the food and coffee service closed might also contribute to less congestion, noise, and socializing and contribute to a more productive study and academic experience.   limitations and future research   this study provided useful qualitative information, but some limitations should be noted. while the results are similar to other studies, as indicated in the “discussion,” the findings of this study are limited to one institution. a more exhaustive mixed methods approach that employs focus groups, ethnographic, exit, and observational methods could have enhanced our findings and given us an opportunity to gain more insights from participants. we also recognize that the survey was distributed during day-time hours, and therefore the data may not apply equally to evening use. finally, this study only surveyed students who were in the library and did not represent those students who do not come to the library. discovering why students choose not to use library spaces could be just as useful as gaining insights from regular users.   conclusion   this space study proved to be extremely useful for an event we did not anticipate when we first developed the research questions and implemented the research. the data from this study provided sound guidance for the reopening of the library and its study spaces during the pandemic. we know that students want variety and balance in library study spaces and that our past efforts spun our collaborative spaces out of balance. when the building was renovated in 2017, we prioritized fitting as many seats in the building as possible, especially on the first floor. we realize now that, in emphasizing seating quantity and fostering an “anything goes” approach, we created a library space that lacked balance between productive work and social interactions. academic libraries should take balance into consideration when designing library study spaces. librarians and space designers should strive to strike an appropriate balance between seating quality and quantity, acceptable noise levels in designated collaborative and quiet study spaces, and the impacts of environmental factors such as printers, food services, exhibit and art displays, restrooms, and walkways on library study spaces.   the pandemic was a helpful inconvenience. without it, we might never have experienced reduced occupancy and seating levels, reduced or changed hours, and the closing of the coffee and snack bar. we have a nagging suspicion that the loss of 45,000 visitors attracted by the coffee and snack bar might contribute to less congestion and noise. a future survey to follow up on any new seating arrangements and where we might deploy technologies is undoubtedly in our future. we are currently working with a vendor to purchase portable indoor zonez soundproof privacy booths in our libraries, informing a follow-up study. the covid-19 pandemic spurred the rise of remote communications technologies such as microsoft teams and zoom that expand the way students, faculty, and staff work in the post-covid era. academic libraries should be prepared to accommodate and facilitate the use of these technologies so that library users can engage in remote classes, meetings, and other activities like tele-health appointments with privacy and environmental impact considerations in mind.   author contributions   aaron nichols: conceptualization, data curation, formal analysis, investigation, methodology, software, visualization, writing original draft, writing review and editing paul philbin: conceptualization, investigation, methodology, writing original draft, writing review and editing   references   applegate, r. (2009). the library is for studying: student preferences for study space. the journal of academic librarianship, 35(4), 341–346. https://doi.org/10.1016/j.acalib.2009.04.004   asher, a., duke, l., & green, d. (2010). the erial project: ethnographic research in illinois academic libraries. academic commons, 8. https://academiccommons.org/may-2010/the-erial-project-ethnographic-research-in-illinois-academic-libraries/   ashmore, b., & morris, s. e. (2002). from scraps to reams: a survey of printing services in academic libraries. college & research libraries, 63(4), 342–352.   bailin, k. (2011). changes in academic library space: a case study at the university of new south wales. australian academic & research libraries, 42(4), 342–359. https://doi.org/10.1080/00048623.2011.10722245   bedwell, l., & banks, c. (2013). seeing through the eyes of students: participant observation in an academic library. partnership: the canadian journal of library information practice and research, 8(1), 1–17. https://doi.org/10.21083/partnership.v8i1.2502   bostick, s., & irwin, b. (2014, june 2–5). library design in the age of technology planning for a changing environment [plenary paper]. 35th international association of scientific and technological university libraries (iatul) conference, helsinki, finland. https://docs.lib.purdue.edu/iatul/2014/plenaries/3   calloway, m., & callahan, d. (2003). paper use and recycling in academic libraries. e-jasl: the electronic journal of academic and special librarianship, 4(2–3), 1–8. https://digitalcommons.unl.edu/ejasljournal/27/   cha, s. h., & kim, t. w. (2015). what matters for students’ use of physical library space? the journal of academic librarianship, 41(3), 274–279. https://doi.org/10.1016/j.acalib.2015.03.014   foster, n. f., & gibbons, s. (eds.). (2007). studying students: the undergraduate research project at the university of rochester. association of college and research libraries. https://www.ala.org/acrl/sites/ala.org.acrl/files/content/publications/booksanddigitalresources/digital/foster-gibbons_cmpd.pdf   freeman, g. t. (2005). the library as a place: changes in learning patterns, collections, technology, and use. in library as place: rethinking roles, rethinking space (pp. 1–9).  council on library and information resources. https://www.clir.org/pubs/reports/pub129/freeman/   gardner, s., & eng, s. (2005). what students want: generation y and the changing function of the academic library. portal: libraries and the academy, 5(3), 405–420.   gayton, j. t. (2008). academic libraries: “social” or “communal”? the nature and future of academic libraries. journal of academic librarianship, 34(1), 60–66. https://doi.org/10.1016/j.acalib.2007.11.011   gerke, j., & teeter, k. (2017). counting heads: building space assessment into your library assessment plan. in d. m. mueller (ed.), at the helm: leading transformation — conference proceedings (pp. 152–159). association of college and research libraries. https://www.ala.org/acrl/sites/ala.org.acrl/files/content/conferences/confsandpreconfs/2017/countingheads.pdf   harrop, d., & turpin, b. (2013). a study exploring learners’ informal learning space behaviors, attitudes, and preferences. new review of academic librarianship, 19(1), 58–77. https://doi.org/10.1080/13614533.2013.740961   hegde, a. l., boucher, t. m., & lavelle, a. d. (2018). how do you work? understanding user needs for responsive study space design. college & research libraries, 79(7), 895–915. https://doi.org/10.5860/crl.79.7.895   hillman, c., blackburn, k., shamp, k., & nunez, c. (2017). user-focused, user-led: space assessment to transform a small academic library. evidence based library and information practice, 12(4), 41–61. https://doi.org/10.18438/b83x00   james, r. m. (2013). culture war in the collaborative learning commons. journal of learning spaces, 2(1). http://libjournal.uncg.edu/jls/article/view/502    jamieson, h. (2006). food for thought: learning services’ approach to food and drink policies in the learning and resource centre. sconul focus, 39, 13–14.   https://www.sconul.ac.uk/sites/default/files/documents/sconul%20focus%2039.pdf   lux, v., snyder, r. j., & boff, c. (2016). why users come to the library: a case study of library and non-library units. the journal of academic librarianship, 42(2), 109–117. https://doi.org/10.1016/j.acalib.2016.01.004   lynn, v. a. (2011). a knowledge commons needs assessment: building for the future at penn state. college and research libraries news, 72(8), 464–467. https://doi.org/10.5860/crln.72.8.8618   mehta, r., zhu, r., & cheema, a. (2012). is noise always bad? exploring the effects of ambient noise on creative cognition. journal of consumer research, 39(4), 784–799. https://doi.org/10.1086/665048   pierard, c., & baca, o. (2019). finding the sonic sweet spot: implementing a noise management program in a library learning commons. journal of access services, 16(4), 125–150. https://doi.org/10.1080/15367967.2019.1649985   pierard, c., & lee, n. (2011). studying space: improving space planning with user studies. journal of access services, 8(4), 190–207.   shill, h. b., & tonner, s. (2004). does the building still matter? usage patterns in new, expanded, and renovated libraries, 1995–2002. college and research libraries, 65(2), 123–150. https://doi.org/10.5860/crl.65.2.123   stemmer, j., & strawser, m. g. (2019). silence in a noisy world: using student feedback to enhance library silent study space. evidence based library and information practice, 14(3), 128–134. https://doi.org/10.18438/eblip29581   thomas, b., van horne, s., jacobson, w., & anson, m. (2015). the design and assessment of the learning commons at the university of iowa. the journal of academic librarianship, 41(6), 804–813. https://doi.org/10.1016/j.acalib.2015.09.005   vondracek, r. (2007). comfort and convenience? why students choose alternatives to the library. libraries and the academy, 7(3), 277–293. https://doi.org/10.1353/pla.2007.0039   appendix sample survey instrument qualitative survey of space usage in the library   1. what is your status at the university? (circle one choice only). (a) undergraduate (b) graduate (c) faculty (d) staff (e) community user   2. how often do you use this area/space? (circle one choice only). (a) daily (b) weekly (c) monthly (d) less than monthly (e) other:   3. what are you doing in this space? (circle all that apply). (a) academic work/study (b) social (c) leisure reading (d) personal work  (e) other:   4. why do you use this space/area? (circle all that apply). (a) furniture (b) quiet (c) fewer noise restrictions (d) access to technology (e) lighting (f) access to outlets (g) other:   5. when using the library what are your needs? (circle all that apply). (a) quiet atmosphere (b) collaborative atmosphere (c) computers (d) 3-d printing (e) microfilm readers (f) scanning technology (g) printing (h) other:   6. are we meeting those needs?   7. which space(s) in the library do you avoid? explain:   8. sound off (what else is on your mind in terms of library space usage).   date: october 2019/atrium1 microsoft word es_hook.doc evidence based library and information practice 2006, 1:3      67 evidence based library and information practice       evidence summary    study of search engine transaction logs shows little change in how users use search  engines   a review of:  jansen, bernard j., and amanda spink. “how are we searching the world wide web? a  comparison of nine search engine transaction logs.” information processing &  management 42.1 (2006): 248‐263.    reviewed by:  david hook  manager, operations information and configuration management, mda   brampton, ontario, canada  e‐mail: david.hook@mdacorporation.com    received: 02 june 2006            accepted: 18 june 2006      © 2006 hook. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to examine the interactions  between users and search engines, and how  they have changed over time.     design – comparative analysis of search  engine transaction logs.    setting – nine major analyses of search  engine transaction logs.    subjects – nine web search engine studies  (4 european, 5 american) over a seven‐year  period, covering the search engines excite,  fireball, altavista, bwie and alltheweb.    methods – the results from individual  studies are compared by year of study for  percentages of single query sessions, one‐ term queries, operator (and, or, not, etc.)  usage and single result page viewing. as  well, the authors group the search queries  into eleven different topical categories and  compare how the breakdown has changed  over time.    main results – based on the percentage of  single query sessions, it does not appear that  the complexity of interactions has changed  significantly for either the u.s.‐based or the  european‐based search engines. as well,  there was little change observed in the  percentage of one‐term queries over the  years of study for either the u.s.‐based or  the european‐based search engines. few  users (generally less than 20%) use boolean  or other operators in their queries, and these  percentages have remained relatively stable.  one area of noticeable change is in the  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:3      68 percentage of users viewing only one results  page, which has increased over the years of  study. based on the studies of the u.s.‐based  search engines, the topical categories of  ‘people, place or things’ and ‘commerce,  travel, employment or economy’ are  becoming more popular, while the  categories of ‘sex and pornography’ and  ‘entertainment or recreation’ are declining.    conclusions – the percentage of users  viewing only one results page increased  during the years of the study, while the  percentages of single query sessions, one‐ term sessions and operator usage remained  stable. the increase in single result page  viewing implies that users are tending to  view fewer results per web query. there  was also a significant difference in the  percentage of queries using boolean  operators between the us‐based and the  european‐based search engines. one of the  study’s findings was that results from a  study of a particular search engine cannot  necessarily be applied to all search engines.  finally, web search topics show a trend  towards information or commerce searching  rather than entertainment.     commentary    the authors have undertaken a study of  some very interesting and important  questions: as the internet and search engines  evolve, and as more and more people use  the internet, how are users’ search patterns  and techniques changing?     one thing that the researchers of this study  do particularly well is to clearly identify  their research questions – searching for  trends and differences in the number of one‐ query sessions, one‐term queries and results  pages viewed, and the changes in the search  topics themselves. their methodology is  equally well described, complete with  definitions and full explanations. the  authors detail an extensive background  literature search and identify the three  different types of methodologies used to  study web searching in the past. although  there have been many previous studies of  search engine use, the authors point out that  to date there have been few studies that  compare the findings across the different  studies. thus, this study fills a void in the  research, complementing existing research.     it is unfortunate that the number of search  engines covered by the various studies was  very limited. this appears to have been  necessary, due to the limited amount of  usable data from the other search engines,  but meant that trend analysis of u.s‐.based  search engines was based on only two  search engines. notably missing from the  search engines was google, not only  because it is currently the most popular  search engine, but also because its  development has changed user expectations  of search engines and web searches. the  limited number of studies and search  engines included puts into question the  usefulness of the results, but it appears that  the researchers were working with what  they had available to them.    for the most part, the authors do a thorough  job of analyzing and explaining the results  of their comparison studies. there were two  cases (percentage of one‐query sessions and  percentage of users viewing only one search  results page) where the results of the 1998  study of the alta vista search engine had to  be discounted because the user sessions  were artificially limited to 5 minutes,  whereas studies have shown that the  average user session is typically 15 minutes  in length. this factor presumably affected  the way the search engine was used and  therefore the results were skewed. while the  authors do identify this variable, one  wonders if there are other variables in the  studies that may have also had an impact on  the study results. it is unclear whether or  not the authors have investigated the  evidence based library and information practice 2006, 1:3      69 impacts of other potential variations among  the different studies, as this example is one  of the few described.     in some cases, the conclusions drawn from  the results do not appear to have a base. the  article states that “[u]sers have a low  tolerance of viewing any results past the  first page. they prefer to reformulate the  web query rather than wade through result  listings.”  yet the percentages of single  query sessions are almost as high as those of  the single‐page views.     this study does shed some light on the  changes in how users are using search  engines. in some cases, the authors have  found little change in the patterns of search  engine use, which, although not as exciting  as discovering a change, is just as important  to report. although more research is needed  to better understand the variables and  comparisons among different search engine  studies, this study presents research that is  useful to the library and information science  profession. certainly, the trends in search  topics are useful for reference and collection  planning, but the use of one‐term queries  among search engine users might also give  some insight as to how library users  approach reference questions.                                     knowledge of users’ search techniques can  also be beneficial for those providing library  instruction. knowing that users are typically  viewing only the first page of search results,  for example, indicates that instructors  should be educating users on how to  identify and discern the legitimate sources  that may be contained on the page. as well,  knowing that few users are using advanced  queries, should instructors put more  emphasis on teaching the use of boolean  operators ‐ or should boolean searching be  taught at all? similarly, for those designing  library search interfaces, knowing that few  users use boolean operators may mean that  there is less need to build advanced search  capabilities into their features, or perhaps a  greater need to simplify them.      article   the possibilities are assessable: using an evidence based framework to identify assessment opportunities in library technology departments   rick stoddart head, user & research services university of idaho moscow, idaho, united states of america email: rstoddart@uidaho.edu   evviva weinraub lajoie director, emerging technologies & services oregon state university libraries & press corvallis, oregon, united states of america email: evviva.weinraub@oregonstate.edu   received: 26 july 2014    accepted: 21 nov. 2014      2014 stoddart and weinraub lajoie. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – this study aimed to identify assessment opportunities and stakeholder connections in an emerging technologies department. such departments are often overlooked by traditional assessment measures because they do not appear to provide direct support for student learning.   methods – the study consisted of a content analysis of departmental records and of weekly activity journals which were completed by staff in the emerging technologies and services department in a u.s. academic library. the findings were supported by interviews with team members to provide richer data. an evidence based framework was used to identify stakeholder interactions where impactful evidence might be gathered to support decision-making and to communicate value.   results – the study identified a lack of available assessable evidence with some types of interaction, outreach activity, and responsibilities of staff being under-reported in departmental documentation. a modified logic model was developed to further identify assessment opportunities and reporting processes.   conclusion – the authors conclude that an evidence based practice research approach offers an engaging and illuminative framework to identify department alignment to strategic initiatives and learning goals. in order to provide a more complete picture of library impact and value, new and robust methods of assessing library technology departments must be developed and employed. introduction   demonstrating value and impact is an ongoing and evolving challenge for academic libraries. often value is determined by the impact library services and resources have with stakeholders and one of the prominent populations served by academic libraries is students. university administrators who determine library budgets and set organizational goals place importance on how academic libraries meet student needs, contribute to student learning, and advance institutional strategic teaching efforts. thus, identifying the points where a library’s resources and services intersect with students provides potential opportunities to insert assessment measures that will aid in articulating library value to stakeholders such as students and university administrators. this article is about identifying assessment opportunities and stakeholder connections using an evidence based research framework.   academic libraries offer many obvious intersections with students and other stakeholders through their services points such as at the reference or circulation desk, or through instruction and workshops that deal directly with students and patrons. assessing the impact of departments engaged in library instruction, reference, and circulation is relatively well established. potential assessment of these areas can occur using transactional metrics provided through circulation, reference, and attendance statistics. additional value can be determined by follow-up surveys, evaluating student work, or connecting these transaction statistics to institutional data such as grade-point-average or retention rates. however, academic libraries have units such as cataloguing, technology departments, digitization units, and others that may not have direct contact with students and patrons. this lack of direct contact with stakeholders does not excuse these library departments from library assessment efforts as libraries devote significant human, financial, and technology resources to these areas and need to articulate a return on these investments.   an essential contributor to library value is found within library systems, web services, and emerging technologies departments. often these areas of library services are overlooked by traditional assessment measures and efforts because they do not appear to provide direct support for student learning. this is unfortunate, as technology units are a “crucial” contributor to service organizations that deal primarily in information, such as libraries (braun, 1998, p. 64). library technology departments offer important services and expertise that certainly influence student learning, researcher productivity, and library innovation but documenting this impact remains an ongoing challenge. in order to provide a more complete picture of library impact and value, new and robust methods of assessing library technology departments must be developed and employed. however, care should be taken to ensure that these library assessments be thoughtfully and effectively integrated within existing workflows and structures. libraries are encouraged to take the time for a thorough self-examination before embarking on an assessment project. this reflection is necessary in order to assist in identifying the potential uses of the data, as well as to build a sustainable assessment cycle.   the emerging technologies & services (ets) department at oregon state university libraries & press (osulp) provides an example of this departmental self-examination. in the fall of 2012, ets engaged in a qualitative research project to identify stakeholder intersections with ets activities and services. the primary intended outcome from this study was to determine stakeholder intersection points from which ets could insert future assessment measures to articulate value and impact. this research project was inspired and guided by the evidence based library and information practices outlined by booth (2009) and koufogiannakis (2011). this study consisted of a content analysis of departmental reports and weekly activity journals of ets members. these items were analyzed for interactions with stakeholders such as students, faculty, and library professionals. an additional evaluation criterion was applied by examining how departmental reports and activity journals contributed to the advancement of the osulp strategic plan, and alignment to national library standards as these both serve to outline library value and impact.   from these results ets was able to draw informed inferences about the department’s role in student learning, advancement of the osulp strategic plan, and alignment to national library standards. this data was further enhanced by interviews with team members that provided greater detail, leading to the discovery of under-realized connections between the library technology department and its stakeholders. the outcomes of this research project for ets included a series of next steps to better capture evidence to convey impact, and a stronger commitment to assessment efforts within the library. interestingly, the assessment activities highlighted areas in which the department was taking on a leadership role that had not been identified previously. the outcomes renewed the department’s focus on outreach beyond osulp. from a library assessment perspective, this project yielded a greater understanding of the potential all library departments have to contribute to student learning, library and university prestige, as well as providing meaningful value to library stakeholders.   emerging technologies and services   the ets department at osulp encompasses more than maintaining the computing and technology infrastructure for the library. the vision of the ets department is to: pioneer efforts that transform access to content and collections. forge partnerships to expand current services and explore new frontiers of library technology .this vision is translated by the ets department into innovative undertakings such as contribution to larger open-source projects like hydra; a community driven, digital asset management solution (dams), translating osulp press books into dynamic educational mobile sites, transforming library spaces with student-centered technology, an in-house developed study room reservation system, a friend-finder tool to aid study groups, and collaborating with government and non-government entities on the oregon explorer data portal. these innovative projects are reminiscent of the stance some library administrators take that libraries have “an obligation to drive technological change” (bengtson & bunnett, 2012, p. 702). this is the stance embraced by ets at osulp.   ets bridges both the management and the creation of the technology environment within the osulp library, or as braun (1998) describes it as “technology in context.” braun points out, “whereas the application, creation, design, maintenance, and improvement of technology itself are, of the domain of the engineer and scientist, managing technology in the context, and for the benefit, of a firm, is the domain of the technology manager” (p. 5). ets is made up of equal parts of each side of braun’s technology dynamic of technology managers and designers. at the time of this research study, the unit included two tenure track librarians who performed research about technology application, instruction, and use; two professional faculty, one of whom acted in the role of overall “technology manager”; six staff including programmers developing new applications and software; and various student workers.   ets is a very productive and dynamic department that allows the library to remain innovative and relevant to users. after collections and staffing costs, technology is the largest expenditure line in the library. if you exclude salaries, technology costs are the library's largest expenditure after collections. such a significant resource allocation makes clear that osulp sees value in the work and resources ets provides not only to the library but also its users. despite the apparent value the library places on this department, ets remained challenged to convey its impact in a meaningful way to campus stakeholders, such as university administrators, who place an emphasis on assessing student learning and success in the classroom. the most common way libraries articulate their contribution to student learning is through their instruction program and workshops. ets has not traditionally participated in teaching and instruction activities. however, that is not to say that ets does not interact with students, contribute to student learning, or offer education opportunities to students. in fact, a recent report (grajek, 2014) identified “improving student outcomes” through “strategically leveraging technology” as a top ten issue for educational technologists. with this need to strategically leverage technology to improve student outcomes in mind, an environmental scan was undertaken to determine how ets’s accomplishments might contribute, whether directly or indirectly, to student learning, university priorities, and national standards.   the eblip cycle as a research framework   library assessment is interested in evidence that can convey impact. this research project is tasked with identifying stakeholder interactions where impactful evidence might be gathered to inform decisions and communicate value. one framework that supports evidence based processes is found within the literature of evidence based librarianship (ebl). crumley and koufogiannakis (2002) frame evidence based librarians as “a means to improve the profession of librarianship by asking questions as well as finding, critically appraising, and incorporating research evidence from library science (and other disciplines) into daily practice. it also involves encouraging librarians to conduct high quality qualitative and quantitative research” (p. 62). eldredge (2000) suggested ebl is “an applied rather than theoretical science. ebl merges scientific research with pressing needs to solve practical problems…. ebl provides a framework for self-correction as new information becomes available that suggests new directions or methods” (p. 290). the nature of ebl as being applied, practical, and informing daily practice, suggestive of new directions, and new evidence resonated with the authors as a framework to construct the ets assessment project.   recently ebl practices and models have become more inclusive of answering day-to-day library management questions not simply targeted research projects. booth (2009) points out that “(i)t is simplistic to assume that a complex managerial situation will yield a single question as in the classic (evidence based practice) formulating or framing a question” (p. 342). indeed, our project demonstrates the need to take a wider more iterative and reflective approach to understanding the problem to be addressed. booth (2009) concurs as “a management problem may be more effectively tackled by achieving a wider, more holistic perspective. within the context of team working     and collaboration it is extremely valuable for a team to arrive at a shared understanding of the problem to articulate this collectively” (booth, 2009, p. 342). this is the very outcome ets and the assessment librarian sought in uncovering the assessment possibilities within the ets department. as booth notes, “each team member has a contribution to make, which itself needs to be valued and carried forward within the decision-making process (p. 342). booth notes further, “(g)iven that library services are human mediated, a significant contributor to the success of any service change is the motivation, involvement, and commitment of the team (p. 343). the more collaborative ebl model as booth proposes thus provides an appropriate framework to evaluate the assessment opportunities within ets. this newer model is made up five steps (booth 2009).   ●        articulate: articulating the problem ●        assemble: assembling the evidence base ●        assess: assessing the evidence ●        agree: agreeing on the actions ●        adapt: adapting the implementation   this model is what guided the process to determine the assessment potential and capability of the ets department at osulp. this framework was further enhanced by the set of questions koufogiannakis (2011) provides regarding gathering practiced based evidence in libraries.   this ebl framework not only provides a pathway to begin to gather evidence of assessment and impact, it also provides a tool to help identify the places and types of evidence that may need to be gathered in a more deliberate or strategic manner. this elegantly mirrored our project goals: determine what library assessment evidence and opportunities are already being leveraged as well as where gaps may exist that might be filled in with additional research effort or assessment tools.   articulate: articulating the problem   the overarching question driving this research study was:   ●        what types of interactions does ets have with stakeholders and students?   answering this question would help ets identify opportunities to target assessment interventions that strategically gather evidence to convey impact to stakeholders. this broad research question was refined further into three more specific questions to help better understand the ets department’s impact and contribution to library and university outcomes valued by stakeholders.   ●        where/how is ets impacting student learning? ●        where/how is ets advancing the library’s strategic plan? ●        where/how is ets contributing to meeting national library standards?   assemble: assembling the evidence base   once the research questions were articulated, the next step in the ebl framework was to gather available evidence about the ets department’s actual and potential contributions to the libraries, the university, and to the national assessment guidelines. the process was guided by the evidence gathering questions suggested by koufogiannakis (2011): what do i already know?; what local evidence is available?; what does the literature say?; and what other information do i need to gather? what do i already know?   the ets department was well integrated into the 2012 2017 osulp strategic plan (oregon state university libraries and press, n.d.). there were a large number of strategic activities within the plan that were spearheaded directly by ets. these items have been identified in internal documents and progress was reported through quarterly library reports. these reports captured the traditional criteria of departmental accomplishments, projects, personnel issues, and challenges. beyond these reports, there was little in the way of formal assessment activities such as user feedback surveys, return on investment projects, or other evidence gathering procedures to tie ets efforts to library and campus-wide outcomes. most projects were documented as “completed” or “in-progress” on reports but were lacking in their ability to articulate project impacts on student success, faculty productivity, or university advancement.   based on departmental reports and quick scan of known activities, ets offers many opportunities for student interaction. this occurs through direct interactions with student ets employees and the hosting of student internships. students are also recruited by ets to test new technologies through usability testing. interactions also occur with users when troubleshooting access issues to library resources. ets also has indirect user interactions through the development and maintenance of the technology students interact with, as well as through the acquisition and support of the educational technologies that underpin the library’s instruction efforts. consultations also transpire with constituents such as library staff, university departments and faculty, and stakeholders outside osulp. however, despite knowing about these interactions, details of the frequency, quality, and assessability of these interactions were unclear. thus, ets continued to have trouble translating these interactions into impacts within the currently available and accepted assessment reporting structure and was therefore not accurately conveying the department’s overall contribution to library and student success.   what local evidence is available?   osulp gathers local evidence from user feedback tools like libqual+, occasional surveys of patrons on technology use, as well as statistics gathered on library equipment in public/teaching areas. this data provides ets with indirect evidence of contribution to library value. however, these measures are not comprehensive in nature, and fail to articulate the full array of interactions ets has with stakeholders. further potential evidence of ets’s impact is articulated in library departmental reports but these documents often provide a summary of activities not a full scope of data for additional analysis. in summary, local evidence available to demonstrate ets’s impact was limited and highlights the need for additional assessment practices in this area.   what does the literature say?   the body of literature concerning library assessment of student learning, space evaluation, and collection usage is growing at healthy rate. unfortunately, one area that needs some additional development is capturing the contribution to assessment efforts from under-represented library units such as ets. little (2013) points out, “academic libraries, especially those with research missions and relatively large budgets, have also not paid as much attention as might be desired to the assessment and evaluation of library technologies…” and the accompanying infrastructure and services (p. 596). most studies found that the library literature focused on evaluating specific technologies rather than the overall services and impact library technology departments might provide. such is the case with dougherty (2009) who suggests that strategically evaluating library information technology is an even more critical need as a result of recent economic troubles. dougherty suggests “measuring performance,” examining usage statistics, and soliciting constituent feedback as a few strategies to consider when thinking about managing technology costs. ergood, neu, strudwick, burkule, and boxen. (2012) echo dougherty’s concerns: “in adapting to the many changes facing us today, the development of an effective strategy for identifying and evaluating emerging technologies is vital” (p. 122). despite these recommendations by dougherty (2009), ergood et al. (2012), and little (2013) to examine, evaluate, and assess emerging technologies and library technology units; the library literature is lacking in research to aid in such projects. ergood et al. (2012) found the same research gap noting: “while the professional literature covers emerging technologies and social media in libraries pretty well, there is a gap in coverage of specific planning and staffing approaches to such technologies, whether in libraries or elsewhere” (p.124). this lack of literature also extends to works addressing overall library technology units such as ets.   a review of the academic literature revealed a few broad themes associated with technology units in organizations and value assessment. in general, the library literature focused on the technology tools themselves, not the assessment of departmental impact. for example, little (2013) writes about using google analytics and the usefulness of usability studies as a way to assess how useful the technology is. dash and padhi (2010) write about the various library assessment tools available and how they assess library technologies, but again, ignore the department supporting those tools. the information technology literature is focused in much the same way, with an emphasis on it infrastructure, purchasing, and the importance of customer service but little about broader outcomes such as organizational success and external stakeholder impact.   though not specifically about libraries or library technology, the area of technology assessment offers some research of general interest. for example, braun (1998) provides a well thought-out examination of technology assessment in the broader sense concerning the potential impacts on society, government, and businesses. braun defines technology assessment as “a systematic attempt to foresee the consequences of introducing a particular technology in all spheres it is likely to interact” (p. 28). this definition provides some guidance to the assessment of library technology departments in suggesting that all spheres of interaction with their services be considered including student learning or other stakeholder impacts.   examining the business literature for articles about assessing organizational unit value and contribution to external stakeholder value also yielded limited results. in fact some research suggests, that in addition to ignoring assessment measures altogether, focusing too heavily on certain performance measurements for assessment can be detrimental to an organization’s measures of impact and effectiveness (meyer & gupta, 1994). for example, meyer and gupta talk of a process called “perverse learning” wherein individuals learn which metrics are emphasized by administrators and only put efforts into those activities that are being measured and ignore those activities that are not. such “perverse learning” can damage the accuracy of performance measures as well as create disconnect between an organization’s purpose and the actions it actually emphasizes (pp.339-340). this speaks to the need to revisit performance measures periodically, align them to organizational goals, and refresh or develop new metrics as needed. in the case of osulp, the need was apparent to balance evidence gathering across the organization not simply the areas that were traditionally leveraged to gather stakeholder impact such as student learning via library instruction or through student computer usage statistics in library learning commons.    little (2013) reminds us that, “assessment should be built in to everything thing we do, including our technology programs, planning, and services” (p.597). nguyen and frazee (2009) emphasize that strategic technology planning is required within higher education to avoid haphazard implementation of tools and resources. braun (1998) in talking about the wider assessment of technology in society, the environment, and within organizations suggests that the “purpose of technology assessment is to look beyond the immediately obvious and analyze the ramifications of given technology in as wide-ranging and far-sighted manner as possible” (p. 1). in assessing library technology units, one must look beyond the “immediately obvious” criteria of cost and use, and expand the analysis to more “wide-ranging and far-sighted” impacts such as student learning and stakeholder value. these “wide-ranging and far-sighted” impacts are often articulated in a library’s strategic plan or mission and suggest criteria for library technology units to consider assessing value. cervone (2010) concurs “when evaluating emerging technologies or innovative new practices and services, it is critical to ensure that the path your library is going down is in sync with the mission of the parent institution” (p. 240). how library technology departments align their efforts and services to advance the library’s strategic plan or mission is one avenue to examine value and determine criteria for assessment. within information technology literature, this congruence of technology efforts and organizational goals is known as “alignment” or “fit” (bergeron, raymond, & rivard, 2004). such alignment theories posit that organizations “... whose strategy and structure are aligned should be less vulnerable to external change and internal inefficiencies and should thus perform better” (p. 1004).   similarly, as within the relationship of technology assessment and strategic management, braun (1998) points out: “the firm needs strategic management for long-term survival and prosperity. technology is vital to the life of the firm and is one of the most important tools available for taking up a certain strategic stance. thus technology needs specialist strategic management. strategic management of technology requires an information input in the form of technology assessment” (p. 55). such assessments are essential to determine the strategic fit of technology units within libraries as they represent potential areas of tension within the overall organization. as units like ets are often synonymous with innovation and experimentation, cervone (2010) points out that, “innovation without demonstrable value being added to processes or services is not something that is typically valued by an organization’s leadership” (p. 240). ergood et al. (2012) agree with regards to emerging technologies that, “given the strong culture of assessment in libraries, an integral last step is to consider the metrics to be used in determining the effectiveness of the tools our libraries implement” (p. 125). the examination of the “effectiveness” of these tools on the student learning valued by library stakeholders and articulated in library strategic plans is what framed the research questions used to guide this study. this study attempts to examine the strategic alignment of the ets department at osu libraries, as a first step in building more meaningful assessments that will assist in articulating value and measuring performance. these assessments will further align the ets department within osulp library’s overall strategic plan that emphasizes student success.   what other information do i need to gather?   as part of the process, the researchers reviewed ets departmental quarterly reports and noted a scarcity of detailed assessment and impact evidence being reported. additionally, there was an absence of assessment processes built into departmental projects. combining these two factors with a lack of literature evaluating library technology departments, it was determined that there was a real research need to examine the assessment possibilities associated with library technology departments such as ets.   the researchers’ initial step was to perform an informal research project to gather examples of ets activities for potential assessment. ets members were asked to maintain a weekly journal of activities. similar “diary” studies have been successfully used with library patrons to better understand information seeking behavior (xu, sharples, & makri. 2011; lee, paik, & joo 2012). sheble & wildemuth (2009) in describing the potential of diary studies as a library research methodology note: “diary methods are more likely to capture ordinary events and observations that might be neglected by other methods because participants view them as insignificant, take them for granted, or forget them (p. 213). the ets member activity journals were undertaken to not only capture potentially assessable activities, but to also gain a better understanding of the day-to-day activities staff were performing that might be viewed as “insignificant” but which in reality are a high impact practice of value to library stakeholders. the review of these day-to-day tasks yielded a clearer understanding of what actions were necessary to accomplish projects recorded in the quarterly reports. the initial data from this quick project provided a starting point to uncover the library assessment opportunities within the department.   assess: assessing the evidence   while there was a paucity of assessment data readily available in the ets department, this is not to say there was no evidence to assess. as mentioned in the previous section, a brief assessment project was undertaken to gather and list some of the daily activities of ets members. these daily activity journals, as well the past two years of departmental quarterly reports offered a body of evidence to evaluate for alignment with library goals and strategic plan. below is a summary of the data available to be analyzed:   ●        ets quarterly reports (n=9) ●        partial fy2011 to fy2013 ●        ets member activity journals (n=9) ●        staff members kept hourly journals for a one week. this activity was performed twice in a 20-week period. ●        ets staff members were asked to note each activity, time spent performing the task, who/what department it impacted, and with whom they may have collaborated to accomplish each task.   these departmental quarterly reports and ets member journals were analyzed using content analysis to identify activities, tasks, and accomplishments that aligned with strategic library documents. these documents included the osulp strategic plan (oregon state university libraries and press, 2010), the osu learning goals for graduates (oregon state university provost, 2010), and the association of college & research libraries (acrl) standards for libraries in higher education (association of college and research libraries, 2011). these documents were selected because they articulated library and university-wide strategic goals for student learning, technology, and library efficiencies. the data was coded based on four sets of criteria developed from these documents, which yielded sixty codes for the content analysis. these codes were grouped in these general areas:   1.       activities that involved stakeholders or collaborators  (18) 2.       activities that contributed to osulp learning goals for graduates (7) 3.       activities that contributed to osulp strategic plan (4) 4.       activities that related to specific ets responsibilities in osulp strategic plan (11) 5.       activities that met one of the acrl standards for libraries in higher education (9) 6.       activities that related to specific technology aspects of the acrl standards for libraries in higher education (11)   these content areas were identified because they contained potential contexts in which assessment could occur to derive impact or value ets has at a library-wide, campus, and national level. the intended outcome for this content analysis was to identify gaps and strengths within ets. the ultimate goal was to establish where stakeholder value intersected with ets projects and services. problematizing the goal of strategic alignment of ets raised these specific research questions:   ●        where does the ets department have direct student contact? ·         what areas of the osulp learning goals are being advanced? ·         how are they being assessed, if they are at all? ●        is the ets department aligned with the library’s needs and strategic plan? ·         what activities demonstrate a contribution to the osulp strategic plan? ●        how does the ets department contribute to library success in terms of national standards of excellence? ·         what activities demonstrate a contribution to the acrl standards for libraries in higher education?   the ets employee activity journals and department quarterly reports yielded 302 excerpts for content analysis. for example, one of the activity journals noted: "continue working on classroom build update (library faculty)/all classroom users. on and off all day. 4 days". this excerpt was coded for the stakeholders this activity impacted, in this case instruction librarians and students as an indirect interaction. this activity was then coded as contributing to the osulp strategic plan goal of enriching academic impact and educational prosperity, as well as the acrl standards for libraries in higher education principle of educational role. codes derived from the osulp learning goals for graduates were only applied if the ets activity was a direct student interaction. because the departmental reports did not emphasize this interaction in the osulp reporting template, there were few ets activities coded with set.   one initial finding that became apparent during the content analysis was that the departmental reports did not provide an accurate representation of all the activities ets undertook. furthermore, the ets member daily journal exercise was guided by a worksheet without any formal training in how to capture personal activity data. this lack of formal training resulted in each employee providing differing levels of detail about their daily activities. that said, this evaluation of ets activities, efforts, and projects was intended to be a starting point for future assessment activities. with that limited goal in mind, this research project was viewed as successful by the ets department as this study yielded actionable data to inform future decision-making.   agree: agreeing on the actions   the next stage of the evidence based practice model is agreeing on what the evidence shows and what proposed actions may result from the assembled evidence. in the case of the limited available evidence from the literature review and the data generated in the content analysis of departmental reports and member activity journals, ets was presented with a variety of results to consider.   renewed emphasis on assessment, evidence gathering, and reporting   one of the major areas of consensus was the need to have more evidence for assessment purposes. this consensus point is the inspiration for this project but it is also demonstrated and reiterated in the lack of available assessable evidence. for example, one of the areas where the researcher knew that ets had strong direct impact was with student employees. student employees, after receiving specialized training, are assigned a project that they manage from beginning to end. this type of student engagement, surprisingly, was not adequately articulated or captured in the departmental quarterly reports or daily activity journals. another example supporting the gap in available local evidence is found in how library space interactions are documented by ets members in the daily library activities such as in support of various teaching and public services( i.e. computers in the classrooms and labs, printers, scanners, tablet computers, etc.). while time spent on issues relating to library space was “known” by ets staff, and the responsibilities themselves were written into the job descriptions of at least three individuals in the department, this hadn’t been adequately documented and was seen as “missing” from the ets content analysis. as a result of this insight, it was agreed that a continued effort to build better evidence-gathering practices and reporting within ets would be emphasized.   to better articulate ets’s assessment opportunities, a modified logic model was developed based on the outcomes of the content analysis (see appendix). logic models are used in performance management and program evaluation to clearly lay out an organization’s inputs, outputs, activities, and outcomes (mclaughlin & jordan 1999; millar, simeone & carnevale 2001). logic models serve to build common understanding, identify priorities, and articulate performance indicators for ongoing assessment (mclaughlin & jordan, 1999, p.66). the ets logic model was seen as one way to further identify assessment opportunities and reporting processes. the ets logic model provided a visual representation of the inputs (money, time, expertise) and outputs (code, project completions) that the department generated. further, these inputs and outputs were connected both to the stakeholders impacted as well as to the overall intended outcomes articulated in the osulp strategic plan such as student success and faculty productivity. the ets logic model serves as a blueprint to help begin thinking more deeply about the assessments needed to connect what ets does to the overall outcomes of the library and university.   embracing library outreach and collaboration   as a result of this study, a number of “i knew this but now we have evidence” moments came to light, such as documenting ets efforts around supporting library technology. however, the most well documented strategic alignment was around collaborative and outreach ets activities. suddenly, there was a major area of documented impact, articulated in the osulp strategic plan and values, university learning goals, and national learning standards that ets could own and build on. collaboration is one of the core values of osulp and is featured prominently throughout the libraries’ strategic plan. these collaborative outreach activities were well documented in ets departmental quarterly reports, and it became clear through the reporting activity, that those projects were also a major part of the day-to-day activities of members of the department. staff members worked directly with branch libraries and state agencies, as well as collaborating on shared services with other universities. despite the robust evidence of collaboration and outreach, ets staff recognized that the quarterly reports were still under-reporting the breadth and depth of the department’s activities and impact on stakeholders thus suggesting that there was additional evidence of collaboration and outreach that was still not being documented. agreement was reached that ets would build on this newly articulated strength of collaboration and outreach as one way to demonstrate strategic alignment with the library and university, as well as continue to develop ways to gather evidence to support these endeavours.   to better support ets’ strategic alignment with library collaboration and outreach goals, and to further emphasize this role to stakeholders, ets revamped their mission and vision statements to highlight the collaborative work the department engages in to provide outstanding service and outreach (oregon state university, n.d.).   potential next steps   at the conclusion of this ets content analysis project, the researchers agreed that for the data collected to have real impact within the library, it would be useful in the future to undertake a library-wide content analysis of all library department reports as a point of comparison. such a cross-departmental analysis of departmental quarterly reports would better articulate collaboration across units as well as impact on stakeholders within the library. additionally, this kind of library-wide content analysis of reporting might likely uncover other hidden areas of impact such as with collaboration and outreach in ets, as well as gaps in evidence gathering and reporting.   the researchers concluded that holding a training session in the library about reporting best practices might be one way to address gaps in reporting content. quarterly reports in the library are inconsistent, and the details reported through those reports differ from department to department. such a training session would highlight the importance of consistent reporting for evidence gathering as a tool for decision-making and demonstrating strategic alignment to stakeholders. now that the osulp strategic plan has been in place for over two years, a technique like this reporting model offers an easy way to assess effort, impact, and advancement of strategic initiatives. the data collected through reporting exercises like this, could have significant impact on the way that future strategic planning sessions move forward for the library; on how the library conveys its value to university administrators, and can help with fundraising activities though highlighting impact in areas that aren’t obvious immediately.    adapt: adapting the implementation   in the evidence based practice model booth (2009) proposed, adapting the implementation is the final step. this stage acknowledges that local application of evidence based interventions may involve modification, flexibility, and retooling. this step emphasizes that evidence based practices, like library assessment, are iterative. the researchers in this initial examination of ets embraced this fact, as the gathering of evidence and resulting content analysis were seen as first steps to building a sustainable, flexible, and personalized ets assessment model. the implementation of suggested outcomes is still a work in progress and will continually be assessed and reassessed as ets builds their departmental assessment resources and culture. a content analysis of departmental reports may not be necessary for every library technology unit to undertake, neither may displaying outcomes in a departmental logic model; but the acts of articulating the question, assembling evidence, assessing the evidence, agreeing on outcomes based on the evidence, and finally the adaption of outcome implementation to local needs suggest a template for all libraries to consider for assessment, strategic fit, and demonstrating value. this final step is about ‘owning’ the application of your ebp research, or similarly developing and embracing your own assessment process to meet organizational needs. this examination of ets suggests that an evidence based practices model offers an adaptable research approach to begin iterative assessment activities within libraries.   conclusion   the possibilities are assessable. the challenge for libraries is to identify these assessment opportunities and take advantage of them as means to gather evidence to support change and convey impact. demonstrating value to stakeholders and documenting evidence of contributing to a university’s mission and strategic initiatives are essential undertakings for all units of an academic library. bengston and bunnett (2012) posit that, “libraries, if they want to be seen as vital, relevant, and positioned as key players in the information handling of the future, must actively engage with technology on every level” (p. 705). library technology units, such as ets at osulp, demonstrate such a commitment. at the same time, the managing of library innovation and technology must also be assessed for value to the organization. as cervone (2010) reminds us, these efforts must be in “sync” with the parent organization. an evidence based practice research approach offers an engaging and illuminative framework to identify department alignment to strategic initiatives and learning goals. the ebp process proposed by booth (2009) provides a step-by-step process for library departments, such as ets, to begin gathering and assessing evidence of impact and value (see table 1). in this project’s instance, the first step was a self-assessment of departmental impact and potential impact areas. this self-reflection proved invaluable resulting in refocus of the department’s mission and re-emphasis on developing new assessment practices and reporting.     table 1 application of the ebp process (booth, 2009) to the ets department ebp model emerging technologies & services department articulate the question what intersections does ets have with stakeholders? where/how is ets impacting student learning? where/how is ets advancing the library’s strategic plan? where/how is ets contributing to meeting national library standards? assemble the evidence strategic documents departmental reports weekly activity journals literature assess the evidence content analysis of department reports content analysis of activity journals agree on the actions renewed emphasis on assessment, evidence gathering, and reporting embracing library outreach and collaboration adapting the implementation results from evidence gathering to inform future ets assessment activities.     the evidence based practices framework provides libraries with guidance, as well as with a suggested research process in gathering evidence that may inform library-wide assessment practices. kloda (2013) sees a clear connection between assessment and evidence based practices as iterative cycles that support rigorous inquiry and change.   most academic libraries want to be innovative in their practices and culture. innovation is desired across many library departments but is especially embedded in library technology units. bengston and bunnett (2012) note, “organizations that wish to support innovations cannot hope to do so by merely stating that they support innovation or by inviting their employees to innovate” (p. 700). library assessment and evidence gathering are key to conveying innovation, as well as identifying and marketing contributions to student success or organizational mission. the ebl process offers a robust framework for project management, iterative development, and collaboration engagement for identifying and developing assessment opportunities. the possibilities are, indeed, assessable. libraries only need to build evidence-gathering processes within their ongoing activities and efforts in order to realize this opportunity in full.   references   bengtson, j., & bunnett, b. (2012). across the table: competing perspectives for managing technology in a library setting. journal of library administration, 52(8), 699-715. http://dx.doi.org/10.1080/01930826.2012.746877   bergeron, f., raymond, l., & rivard, s. (2004). ideal patterns of strategic alignment and business performance. information & management, 41(8), 1003-1020.   booth, a. (2009). eblip five-point-zero: towards a collaborative model of evidence based practice. health information and libraries journal, 26(4), 341–344. http://dx.doi.org/10.1111/j.1471-1842.2009.00867.x   braun, e. (1998). technology in context: technology assessment for managers. london: routledge.   cervone. h. f. (2010). emerging technology, innovation, and the digital library. oclc systems & services, 26(4), 239 –242.   crumley e., and koufogiannakis d. (2002). developing evidence based librarianship: practical steps for implementation. health information and libraries journal 19(2), 61-70. http://dx.doi.org/ 10.1046/j.1471-1842.2002.00372.x   dash, n.k., & padhi, p. (2010). quality assessment of libraries. desidoc journal of library & information technology, 30(6), 12-23. retrieved from http://publications.drdo.gov.in/ojs/index.php/djlit/index   dougherty, w. c. (2009). managing technology during times of economic downturns: challenges and opportunities. the journal of academic librarianship, 35(4), 373-376. http://dx.doi.org/10.1016/j.acalib.2009.04.007   eldredge j.d. (2000). evidence-based librarianship: an overview. bulletin of the medical library association, 88(4), 289-302. retrieved from http://www.ncbi.nlm.nih.gov/pmc/journals/72/   ergood, a., neu, r. c., strudwick, j., burkule, a., & boxen, j. (2012). five heads are better than one: the committee approach to identifying, assessing, and initializing emerging technologies. technical services quarterly, 29(2), 122-134. http://dx.doi.org/ 10.1080/07317131.2012.650611   grajek, s. (2014). top 10 it issues, 2014: be the change you see. educause review. march/april. 10-46. retrieved from http://www.educause.edu/ero/article/top-ten-it-issues-2014-be-change-you-see   kloda, l. (2013). from eblip to assessment. presentation at 2013 evidence based library & information practice conference. may. university of saskatchewan, saskatoon, canada. retrieved 27 nov. 2014 from http://www.slideshare.net/lkloda/kloda-eblip-7-lightning   koufogiannakis. d. (2011). considering the place of practice-based evidence within evidence based library and information practice (eblip). library and information research 35(111), 41-58. retrieved from http://www.lirgjournal.org.uk/lir/ojs/index.php/lir   lee, j. y., paik, w., & joo, s. (2012). information resource selection of undergraduate students in academic search tasks. information research, 17(1), 5. retrieved from http://www.informationr.net/ir/   little, g. (2013). squaring the circle: library technology and assessment. the journal of academic librarianship, 396, 596-598. http://dx.doi.org/10.1016/j.acalib.2013.09.002   mclaughlin, j. a., & jordan, g. b. (1999). logic models: a tool for telling your programs performance story. evaluation and program planning, 22(1), 65-72. http://dx.doi.org/10.1016/s0149-7189(98)00042-1   meyer, m. w., & gupta, v. (1994). the performance paradox. research in organizational behavior, 16, 309-369.   millar, a., simeone, r. s., & carnevale, j. t. (2001). logic models: a systems tool for performance management. evaluation and program planning, 24(1), 73-81. http://dx.doi.org/10.1016/s0149-7189(00)00048-3   nguyen, f., & frazee, j. p. (2009). strategic technology planning in higher education. performance improvement, 48(7), 31-40. http://dx.doi.org/10.1002/pfi.20093   sheble, l., & wildemuth, b.m. (2009). research diaries. in b.m. wildemuth (ed.), applications of social research methods to questions in information and library science (pp. 211-221). westport, conn.: libraries unlimited.   xu, s., sharples, s., & makri, s. (2011). a user-centred mobile diary study approach to understanding serendipity in information research. information research,16(3), 7. retrieved from http://www.informationr.net/ir/     appendix ets logic model news/announcements   share buttons      2013. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     eblip’s pages now include share buttons which allow authors and readers to distribute eblip content to facebook, twitter, and more than 300 other social networking services. the share buttons, which are part of a suite of tools available through the addthis platform (www.addthis.com), represent eblip’s first step towards utilizing alternative metrics (altmetrics) to measure the impact of the journal. the share buttons track the number of times that eblip’s items are tweeted and shared, and because these metrics are displayed at the bottom of each page, authors can see how frequently their articles have been shared with others and track the social impact of their work. please use the new share buttons to share eblip’s content with your followers! evidence based library and information practice evidence based library and information practice 2011, 6.1 74 evidence based library and information practice evidence summary choosing the right book: factors that affect children’s reading a review of: maynard, s., mackay, s., & smyth, f. (2008). a survey of young people’s reading in england: borrowing and choosing books. journal of librarianship and information science, 40(4), 239-253. reviewed by: gayle bogel assistant professor, educational technology, graduate school of education and allied professions fairfield university fairfield, connecticut, united states of america email: gbogel@gmail.com received: 15 nov. 2010 accepted: 13 feb. 2011 2011 bogel. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to analyze factors affecting book choice: reading recommendations, reasons for choosing series books, book reviews, and school libraries. design – this secondary analysis of data is based on a selection of findings from a larger 2005 survey that monitored trends in youth reading habits. the large scale 2005 study was designed as a follow up to a 1996 survey. the 2005 survey used online questionnaires and formal statistical analysis to compare gender and age groups. the data on factors affecting book choice were derived from the original questions, responses, and analyses. setting – questionnaires were administered in 22 primary and 24 secondary schools in the uk with access to computers and internet. subjects – almost 4,200 students from 4 to 16 years of age. methods – study authors invited approximately 150 schools to participate in the survey. forty-six schools (31%) responded. a total of 22 primary and 24 secondary schools participated in the original study between april and june 2005. this study used comparative analysis to examine factors affecting book choice between gender and age groups. statistical significance was defined as one percent. other demographic information was collected, such mailto:gbogel@gmail.com� evidence based library and information practice 2011, 6.1 75 as ethnicity, language spoken at home, and religion, but was deemed insufficient for any meaningful analysis. there were 4,182 responses to the survey, separated into three age groups: ages 4 to 7 (ks1), ages 7 to 11 (ks2), and ages 11 to 16 (ks3&4). students were asked to describe themselves as readers by responding to multiple choice questions, and then to provide specific information on the places or people most frequently used as book borrowing sources. participants were considered “enthusiastic” readers if they “read a lot with pleasure” and “average” readers if they “read an ordinary amount”. participants responded to additional multiple choice questions on specific factors related to borrowing books and book choices, the process of choosing series, fiction, and nonfiction books. main results – readers: the number of children who described themselves as “enthusiastic” readers decreased with age, and approximately half of the children between 7 and 16 years of age described themselves as “average” readers, average rating increasing slightly through this age group. investigators found a marked difference in gender within the 4 to 7 year olds: 49.7% percent of girls in this age group considered themselves “enthusiastic,” compared to 37.3% of boys. only 18.5% of girls considered themselves “reluctant” readers, compared to 28.1% of boys. the longitudinal comparison to the earlier 1996 study found that although the percentage of boys from ages 7 to 16 who described themselves as reluctant readers stayed about the same, the percentage of girls who described themselves as reluctant increased. there was also a dramatic drop in the number of girls who described themselves as enthusiastic; from 51% in 1996 to 17% in 2005. borrowing books: libraries of all types (school, classroom and public) were the prime sources for borrowing books. girls borrowed more books from schools than boys, and girls also borrowed more books from non-library sources (family members and friends) than boys. both boys and girls increasingly reported borrowing more books from friends than from libraries as they grew older. over half of each age group rated the school library or classroom book corner as having “enough” books. the perception of “too many books” decreased with age. the perception of quality of the classroom or school library also decreased with age. the number of students in the younger age groups who rated the quality of books in the classroom or school library as “very good” was significantly higher than students in the older age groups. the number of student who rated the quality as “okay” doubled from the lower to the higher age group. choosing books: children were asked six questions related to whether the physical book itself provided motivation to read, and six questions related to other factors for book recommendation. younger readers were more likely to choose a book for its visual appeal, although this factor (interesting cover or illustrations inside) was more consistent for boys of all age groups than girls. the author’s name and book blurb were stronger factors for girls in the 7 to 16 age group than for boys. the study authors sought to explore the idea of “shared reading” and asked children in the 7 to 11 and 11 to 16 age groups how often they chose a book based on a recommendation from a friend or family member, a public or school librarian, or other adults. friends were the strongest recommendation source (43.2% for 7 to 11 year olds and 38.4% for 11to 16 year olds). recommendations from school or public librarians rated only three to six percent for both age groups. all age groups were asked about choosing series books, and the questions were simplified for the younger age group. all students reported that appealing factors were the consistency of characters, familiar storylines, and familiar writing styles. some evidence based library and information practice 2011, 6.1 76 students also noted that the availability of series books positively affected their choices. print or online book reviews were used “hardly ever or never” by over 40% of the 7 to16 year old age groups, while television or magazine reviews or recommendations were highly rated by over 50% of respondents in the same age groups. participants chose informational or non-fiction books because of personal interests, hobbies, or recommendations from friends, while they selected fiction or poetry for the blurb, title, or appearance and design of the cover. celebrity recommendations and books about celebrities were popular reasons for choosing books. when asked who helped them choose books, 53.2% in the 7 to11 age group and 66.1% in the 11to 16 age group stated that no one ever helps them. of the students who had help choosing books, “family members” was the most common response. local librarians were not used as a source for recommendations. commentary secondary analysis looks more deeply at existing data and assembles additional information from longitudinal studies such as this one. the secondary analysis must match the variables in the original study with those of the secondary study (clark & cossette, 2000). it is unfortunate that the additional demographic data collected regarding ethnicity and language spoken at home was not available. both of these factors could provide interesting analysis of the data on reading in libraries versus home, and on the data regarding recommendations from family members. the researchers do not mention the socioeconomic status of the study population, but poverty level has been identified as a significant factor in children’s perception of and access to reading materials (krashen, 2004). the lack of availability of books in the home due to economic pressures could dramatically affect children’s responses to whether they borrow books from other family members. the number of single parent homes could also affect responses to the questions of whether the participant most often looks to father or mother for book recommendations, and whether the child uses his or her mother’s recommendations. the researchers note as “worrisome” the increasing number of those describing themselves as “reluctant readers,” with a concurrent decrease in those describing themselves as “enthusiastic” readers. while the goals of this study were not to evaluate reading instruction methods, these methods and the current emphasis on prescriptive, skills-based reading in elementary schools could affect how children see themselves as readers. instructional methods could also contribute to the decline in enthusiasm and perception of being good readers (cody, 2010). the conclusions echoed existing research that indicates boys in general are more likely to consider themselves reluctant readers and be viewed by others as reluctant readers. boys in the study were less likely to rely on book reviews or book displays for reading recommendations. the researchers noted that boys were also more likely to rate the books in their local libraries as “not very good.” the researchers suggest that school library selection policies could explain this finding. they support their hypothesis with research on boys’ reading that contradicts the opinion that “boys don’t like reading.” boys like reading if selectors find books that appeal on a cultural and emotional level to boys’ interests (jeffery, 2009; love & hamston, 2003). one weakness in the study is that the researchers did not differentiate between school libraries and classroom book corners when asking children about books available at school. the researchers found the percentage of participants believing that there were “too many books” was surprising, and perhaps resulted from the inability to locate particular reading matter quickly and easily. this reviewer has observed that school libraries are organized in ways that promote easy access to reading materials through organized collections and online catalogues, while evidence based library and information practice 2011, 6.1 77 classroom libraries do not. it is unclear whether students were referring to the school library or classroom book corner in many of their answers. perhaps the answer “too many books” is not surprising if the students’ primary contact was with classroom book corners that are not organized for easy access and that do not offer enough books at each reading level to accommodate the needs of students during self-selected reading (sewell, 2003). the researchers note disappointing answers to questions about whether children receive help in choosing books. participants rarely used both school and public librarians, and the researchers found that libraries themselves were considered valuable for borrowing books but children often don’t ask for professional assistance in the library. although any number of factors could account for the lack of assistance from librarians (staffing levels, training, availability), readers are left with the dichotomous finding that although school and public libraries were the two most cited sources of books for students of all ages, the librarians were not considered important factors in book selection. this study successfully attempted to determine some of the factors in children’s book choices, but the results pinpointed areas for further research. references clarke, s. p., & cossette, s. (2000). secondary analysis: theoretical, methodological, and practical considerations. canadian journal of nursing research, 32(3), 109–129. jeffery, d. (2009). reaching reluctant readers (aka books for boys). alan review, 36(2), 56. krashen, s. (2004). the power of reading: insights from the research. (2nd ed.). portsmouth, nh: heinemann publishing co. and westport, ct: libraries unlimited. love, k., & hamston, j. (2003). teenage boys' leisure reading dispositions: juggling male youth culture and family cultural capital. educational review, 55(2), 161. love, k., & hamston, j. (2004). committed and reluctant male teenage readers: beyond bedtime stories. journal of literacy research, 36(3), 335. cody, a. (2010, may 10). stephen krashen: children need food, health care, and books. not new standards and tests [web log comment]. retrieved from, http://blogs.edweek.org/teachers/livingindialogue/2010/05/stephen_krashen_fix_ poverty_an.html sewell, e. j. (2003). students' choice of books during self-selected reading. edrs opinion papers (eric document reproduction service no. ed476400). worthy, j., & mckool, s. s. (1996). students who say they hate to read: the importance of opportunity, choice, and access. yearbooknational reading conference, 45, 245-256. http://blogs.edweek.org/teachers/living-indialogue/2010/05/stephen_krashen_fix_poverty_an.html� http://blogs.edweek.org/teachers/living-indialogue/2010/05/stephen_krashen_fix_poverty_an.html� http://blogs.edweek.org/teachers/living-indialogue/2010/05/stephen_krashen_fix_poverty_an.html� / evidence based library and information practice evidence based library and information practice evidence based library and information practice 2011, 6.2 61 evidence based library and information practice evidence summary the library as a preferred place for studying: observation of students’ use of physical spaces a review of: applegate, r. (2009). the library is for studying: student preferences for study space. the journal of academic librarianship, 35(4), 341-346. reviewed by: annie m. hughes reference librarian wilson dental library, university of southern california los angeles, california, united states of america e-mail: amhughes@usc.edu received: 28 feb. 2011 accepted: 21 apr. 2011 2011 hughes. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to determine students’ utilization of physical spaces in the library, excluding computer labs or stacks. design – observational research, unobtrusive method. setting – areas of space in the university library, as well as within adjoining areas at indiana university-purdue university indianapolis, such as carrels, tables, soft chairs, and study rooms. subjects – students using the library’s space. methods – the researcher chose to collect data via observation of individuals and groups in a particular space in the library, noting the gender of the individuals using the space and whether or not they were using laptops. areas of space examined were carrels, group study rooms, chairs and sofas, tables and chairs in the academic commons, and benches and chairs within corridors. the unit of analysis used was equal to an individual seat. the research excluded stack space as well as any space with fixed computer stations. the time periods chosen to study the spaces were selected based on the author’s previous research. due to higher daytime usage than evening, data was collected at two time periods during the day: 12-1 p.m. and 3-4 p.m., monday through friday. the researcher recorded the time of the semester as well, choosing weeks 14-17 in fall 2007 and weeks 10-17 in spring 2008. space diagrams for mailto:amhughes@usc.edu� evidence based library and information practice 2011, 6.2 62 collecting data were created, and each area had different collection times. all data was entered into a database in which each area was recorded with the number and type of users. each area had a different capacity as to how many individuals it could hold. if the percentage of capacity was higher than 50%, the usage was considered to be notable. main results – the researchers observed a few patterns from their data collection. gender analysis provided information regarding the use of laptops; men were more likely to use them than women. while men were a smaller part of the overall university demographic while this research took place, they utilized the library spaces most. as expected, library usage increased as the end of each semester neared, suggesting that the spaces are used mainly for study purposes. the author also chose to collect data regarding library usage by semester, which is questionable because the student population declined from fall to spring and a campus center opened, providing another study space. the most attractive spaces in the library were study rooms, and for the most part, groups, as opposed to individual students, utilized these rooms. the chair and sofa areas of the library were the next most popular areas, but the study carrels were also popular, especially toward the end of a semester. conclusion – according to the researcher, the data collected points to the library as a preferred place for studying, as opposed to other activities. by observing the use of areas such as study carrels, soft chairs, and group study rooms, one can derive data that will allow for future space planning, as well as gain an understanding of how a current space is being used. commentary the data collected for this study is important in that it provides evidence of how certain spaces within the library are used, and this data could be useful at the beginning of designing a new space. if charged with spaceplanning for a university library, it would be beneficial to know, for example, that study carrels and group study rooms are utilized most frequently. the researcher chose to evaluate several aspects of the use of space in the library, but did not develop any clear research goal or question other than whether library space is used for studying. gender, or rather “sex”, the preferred term in academia, was one aspect the researcher chose to consider. however, while it is interesting to consider sex when planning spaces, it is also presumptuous to assume that female students would want open spaces instead of closed spaces for safety issues. a separate study surveying students and analyzing preferences by sex would likely be more beneficial than making broad generalizations. again, it would be interesting to survey these students and get more indepth information on why men choose to utilize the study space more frequently than women. the author also makes an assumption that the spaces are being used only for studying purposes. she does not consider that students with laptops often use softer spaces to watch online media, participate in social networking activities, browse the internet, and email friends. since no evidence was gathered concerning these activities, it is hard to say that the spaces are used only for studying. the researcher also did not make it clear how she came to the conclusion that the library is the preferred space for studying, as she did not directly observe other areas on campus, specifically the campus center. also, in order to get a clear view of the spaces studied, it would be beneficial to include floor plans or images of the library space instead of simply a textual description. applegate wrote “build it and what? measuring the implementation and outcomes of an information commons” in 2007. her paper examined the expectations, pros and evidence based library and information practice 2011, 6.2 63 cons, and issues that confront libraries in designing an information commons. the study evaluated for this particular summary does not pose a clear research question as did her previous paper. a follow-up on utilization of library space since the redesign and development of the academic commons would be more effective and provide more evidence as to the importance of space planning. references applegate, r. (2007) build it and what? measuring the implementation and outcomes of an information commons. in sailing into the future: charting our destiny. proceedings of the thirteenth national conference of the association of college and research libraries, march 29april 1, 2007, baltimore, maryland (pp. 167-173). chicago: association of college and research libraries. / evidence based library and information practice conference paper   striving for excellence: organizational climate matters   shelley phipps retired (currently consultant) university of arizona tucson, arizona, united states of america email: shelley_phipps@yahoo.com   brinley franklin vice provost for university libraries university of connecticut storrs, connecticut, united states of america email: brinley.franklin@lib.uconn.edu   shikha sharma social sciences team leader university of connecticut libraries storrs, connecticut, united states of america email: shikha.sharma@lib.uconn.edu      2013 phipps, franklin, and sharma. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to describe steps undertaken by the university of connecticut libraries to respond to the results of an organizational climate assessment. more than 80% of the libraries’ staff members completed the climatequal® survey instrument in the spring of 2007. an organizational development consultant designed a format for focus groups to provide anonymous, but more detailed, experience-based information to help the libraries discover, understand, and respond to the root causes of “problem” areas indicated by the survey results.   methods – in november 2007, the consultant conducted five 90-minute, on-site focus group sessions, each with 7-15 participants. two of the sessions were open to all staff members, while the others focused on underrepresented minority group members, team leaders, and the staff of one specific team.   results – a summary report based on compiled data and including recommendations was submitted and discussed with the libraries’ leadership group. in line with organizational development practice, recommendations were made to engage those closest to the “problems” (i.e., the staff) to design and recommend improvements to internal systems. the consultant advised the formation of six teams to address internal systems, and an initial three teams comprised of staff members from across the library were formed. these teams were charged with formulating a set of recommended actions that will contribute to a healthier organizational climate in three areas: leadership and team decision making; performance management; and hiring, merit, and promotion. the findings, recommendations, and progress-to-date of each team are summarized.   conclusion – the climatequal® results and the follow-up with the organizational development consultant helped in identifying potential problem areas within the libraries’ internal systems. the consultant made recommendations that led to the development of concrete roadmaps, benchmarks, and associated strategies. the libraries’ progress on its strategic plan will serve as the barometer for gauging the effect of these changes.   introduction   the university of connecticut (uconn) libraries began doing organizational climate assessments in 1999 with the original intent of measuring whether articulated organizational values were achieved following a library-wide reorganization in 1996. the uconn libraries’ original organizational climate assessment was influenced by the balanced scorecard approach (kaplan & norton, 1996). the balanced scorecard is one of the more recent multidimensional approaches to organizational assessment. earlier, among others, georgopolous and tannebaum (1957) cited multiple effectiveness measures, kanter and brinkerhoff (1981) researched the topic, and cameron (1978) published multidimensional organizational assessment literature specifically related to higher education.   the uconn libraries’ organizational assessment was directly influenced by the learning and growth component of the balanced scorecard’s vision and strategy process that asked “to achieve our vision, how will we sustain our ability to change and improve?” the other three balanced scorecard processes were addressed by the libraries through user satisfaction studies, a statistical data information system, and a workflow study.   eighteen assessment criteria for the libraries’ reorganization and fifteen assessment criteria for the seven newly created functional areas had been developed by the libraries, mostly in response to concerns voiced by staff during the 1995 strategic planning and 1996 reorganization processes. in 1999, three years after the reorganization into functional areas and teams was completed, the libraries conducted its first organizational climate assessment using a staff survey based on articulated measures of success including:   empowering staff with respect to operational responsibilities; reflecting shared leadership and mutual respect among the staff; displaying strong inter-area cooperation; fostering good communications throughout the libraries; making the decision-making structure clear to staff; and showing consideration for individual differences.   an organizational development (od) consultant followed up on issues raised in the staff survey and outlined a number of actions that the libraries subsequently pursued, including: team training, staff interactions (e.g., developing good communication skills and, mutual respect, understanding how mental models and the ladder of inference contribute to positive problem solving), leadership development training, employee recognition and rewards, improved communication by opening leadership council meetings to library staff, and implementing a more open process in developing the libraries’ periodic strategic plan/shared vision updates.   the 1999 organizational assessment was repeated in 2002 and 2005. the overall average score measuring the libraries success in achieving organizational values increased by 4.5% between 1999 and 2002 and by 5% between 2002 and 2005. the overall average score at the functional area level increased by 3.7% between 1999 and 2002 and by 2.9% between 2002 and 2005. beginning in 2002, the libraries sponsored a library-wide annual “day of learning” in part to address issues like organizational boundaries, the value of teamwork, and managing change. simultaneously, between 1996 and 2006, the libraries’ average user satisfaction improved by 12% and between 2000 and 2008 the libraries libqual+® satisfaction results improved by 5%.   although the uconn libraries experience with organizational assessment studies was valuable, it was not based on a standardized instrument like the litwin and stringer organization climate questionnaire which was developed in harvard university’s graduate school of business administration research division in the 1960s and was based on nine a priori scales: structure; responsibility; reward; risk; warmth; support; standards; conflict; and identity (sims & lafollette, 1975).   in so far as the uconn libraries initially used an instrument that was not developed in conjunction with other libraries, the libraries were enthusiastic about participating in the first group of arl libraries to pilot the organizational climate and diversity assessment (ocda) methodology in 2007. soon to become climatequal® (http://www.climatequal.org/), the survey methodology was developed by paul hanges in the industrial/organizational psychology program at the university of maryland in conjunction with the university of maryland libraries beginning in 2000. the key organizational climate concepts that ocda assessed in 2007 were: climate for diversity; climate for continuous learning; climate for innovation; climate for justice/fairness; job satisfaction; and team climate as they related to customer service. ocda also assessed whether library policies, practices, and procedures were effectively supporting the library’s mission as well as employee’s perceptions of what behaviors were expected, supported and rewarded.   more than 80% of the libraries’ staff completed the ocda survey instrument in the spring of 2007. when the summary draft findings were presented to the library staff at its 2007 fall forum, it was clear that the libraries needed assistance in determining how to interpret and respond to the survey results. this became even more evident when the qualitative results, based on open-ended staff comments, were made available several months later. the uconn libraries engaged one of the same organizational development consultants who had assisted the libraries with its earlier organizational climate assessments. this consultant had also gained considerable trust and credibility with the library staff through her earlier work.   consultant’s design/methodology/approach   the consultant designed a format for focus groups to provide anonymous but more detailed, experience-based information, which helped the libraries discover, understand and appropriately respond to the root causes of “problem” areas indicated in the climatequal® survey. a summary report based on compiled data and including recommendations was submitted and discussed with the libraries’ leadership group. assisting that group in understanding that problems were embedded in the libraries' systems, policies or practices, and should be divorced from “personal blame” was an important part of the “helping” role of the consultant. in line with organizational development practice, recommendations were made to engage those closest to the “problems,” the staff, in designing and recommending improvements to internal systems.   background and preparation for the consultation   to begin this latest assignment to help the uconn libraries achieve their organizational goals, the consultant read and re-read the rich but complex data description of climate factors and analysis provided to the uconn libraries by the ocda staff. also, to get a picture from a different set of data of issues present in the environment as the staff completed the ocda survey, she studied other consultant reports which had recently been submitted to the libraries, and inquired about follow-up to each. she also examined the 2007 strategic plan and its updates.   after interviews with the director and the libraries organizational and staff development librarian, the consultant determined that a) there was serious interest in understanding how climate factors were affecting the staff's trust and commitment to the organization, and b) that there was a genuine commitment to implement systems changes to improve the climate. there was willingness as well to embrace the yet to be proven concept that there was a direct correlation between staff perceptions of climate and customers' perceptions of quality service (kyrillidou & baughman, 2009). as has been described, uconn has one of the longest surviving team-based organizations, with an embedded history of staff empowerment, a clear value to support diversity, and an overt commitment to continuous learning and improvement. it is worth noting that they were in a minority of the ocda test group participants who, almost immediately, shared the results of the survey with their staff on an all staff website and held an all staff meeting to discuss results.   the specific assignment of the consultant was to:   conduct focus groups on all climate factors of the survey that indicated a strong need for improvement – as indicated by less than 50% agreement on a particular factor; summarize, analyze, and report the focus group results to the leadership council; and advise on next steps.   conducting the focus groups: selecting participants   in november 2007, six months after the administration of the survey, the organizational and staff development librarian invited staff to attend 1 of 5 90-minute, on-site focus groups. a general invitation went to all staff to attend one of two “mixed staff sessions” and to the entire staff of one specific team and all underrepresented minority groups, where a significant difference in team/group responses was noted when compared to the overall libraries responses to specific climate factors in the ocda report. a fifth focus group was held with the team leaders to round out the “view” and learn more about probable causes. anywhere from 7-15 participants attended each focus group session.   believing that confidentiality was not an issue, a list was kept of the attendees. this later proved slightly problematic, since those attending first agreed to verbatim summaries, but withdrew that agreement when they saw the summarized transcripts, even though all personal identification had been excluded from the summaries. this experience matched the consultant's experience that although staff want to feel that their actual thoughts, as expressed, would be the most helpful to those in power, there is a prevalent reluctance to believe, in the end when one sees honest expressions of concerns, that there is no possibility of negative personal consequences. this dilemma was averted by having the consultant further summarize and abstract and, where possible, generalize the feedback from the groups, submit the summaries to the group for their approval, and further edit the final summaries appended to the final report.   conducting the focus groups: approach   in a previously published article (phipps, 2004) the consultant had detailed the importance of looking at systems to discover causes of climate problems. this approach provided a conceptual framework to further analyze the climate factors included in the ocda survey. each factor in the survey can be seen as evaluating the success or failure of an organizational system (i.e., distributive justice encompassed the systems of performance management and rewards). thus questions for the focus groups were designed to enable the mapping of results to particular system improvements that might be called for.   root cause analysis as embraced by dean gano (1999) was the best “question” methodology to use in the focus groups in order to distinguish between possible system causes (called “conditions” in gano's book) and event-driven causes. each session began with a reminder of the purpose of the ocda survey and a description of the climate factor(s) for which input was being sought. agreement on ground rules for the session, including one on confidentiality, was sought before proceeding with the questions. the consultant took relatively “verbatim” notes on posted chart paper and reviewed those notes with the individuals providing the comment and with the whole group prior to ending the session. (although this is not the formally recommended way to track focus group input, it was done to save the time of transcribing audio tapes and helped the consultant gain a better understanding of issues referenced by the group.)   using gano's approach, each group was asked to think of an event (something that happened in the libraries prior to the administration of the survey in may) or a condition (the way things worked, what policies and procedures were in place or how they were implemented, the culture in the libraries or a team) which might have contributed to the 50% disagreement with the positive statement of the climate factor in the survey. using this approach, it would be possible to distinguish between past events, which were beyond the libraries' control to “change” and conditions or systems which were amenable to change in order to positively affect the climate. causal events were not dismissed as unimportant, but were noted and recommendations made for becoming aware of their impact. such events could be acknowledged openly as “mistaken” or “naïve,” clarified or given further context, and avoided in the future in order to avert a negative consequence on climate.   conduct of the focus groups: reports   after completing drafts, gaining feedback, re-writing summaries and gaining the agreement of the focus groups to share the summaries with the organizational and staff development librarian and the vice provost for libraries, a draft final report with appended focus group summaries was submitted. the final report outline was as follows:   background process identification of categories (climate factors) which were discussed in the focus groups summary recommendations appendix: mixed staff (combined summary from the two groups), underrepresented minorities, staff from one team; team leaders’ separate reports, including analysis of themes and summary recommendations for each.   summary recommendations   follow-up approach   because the uconn libraries is a team-based learning organization committed to organizational development strategies, it was recommended that all change efforts include substantial involvement of the staff. best practice recommended including the staff in further research, since only two data sets (the survey results report and the consultant's report) existed. staff involvement can point the way to substantial and successfully implemented changes that can lead to actual culture change. since results of the survey indicated a gap between understandings of the staff and perceptions of the administration and a significant amount of time had passed and many changes had already been implemented since the original survey administration, the appointment of staff teams would reinforce the commitment to shared leadership, the development of organizational competencies, and the collaborative spirit needed for future success of the libraries in this environment of constant change. as organizational development practice has shown, staff understanding of issues and involvement in addressing them, can increase the effectiveness of planned actions and contribute to overall cost-efficiency.   overall strategic understanding   the libraries had already begun a new strategic planning process in april/may 2007 which included a review of "staffing to vision and plan 2010," which was originally developed in october 2006 by the planning group and administrators, and a retreat of 45 staff members to review and understand staffing needs and develop areas of emphasis and de-emphasis for the future. a new retreat was planned to begin the 2015 strategic plan. the consultant recommended moving deliberatively forward on this approach, continuing broad and deep communications and using several methods for learning about the environment and customers' changing needs.   focused climate improvements   based on her expanded view of the results and interpretations of the ocda survey responses the consultant provided the vice provost for libraries a comprehensive list of recommendations for specific aspects to consider as further, more internally grounded recommendations are developed by staff teams. while the strategic planning process proceeded, the consultant recommended that selected, representative teams of 4-6 staff members be appointed to research and recommend changes in the following organizational systems:   the leadership and team decision-making systems, including the structure and role of the diversity team; the performance management system – with a focus on what improvements can be added to the currently prescribed procedures that would increase constructive approaches to developing staff; the hiring, merit and promotion systems – with an emphasis on the processes used, the involvement of peers and the clarity of the goals and criteria; the communication system – with a focus on leadership council, the strategic planning process and the “staffing to vision” approach; the approaches to learning, training, and innovation.   each team was clearly charged to:   understand the current situation (i.e., what is prescribed, what is practiced, what is changeable from internal policy and procedure documentation and interviews with campus sources); understand what is desirable and will impact improved customer service using the arl/ocda and the consultant's report’s data and analysis; develop and evaluate potential actions which can be taken, using the recommendations of the consultant as a starting point; and, consult with the organizational and staff development librarian and the leadership council to decide which high-benefit/acceptable cost actions can be implemented over the next two years.   in order to insure that teams worked from actual data and did not rely too much on perceptions gleaned from the survey and the focus group summaries, the consultant also recommended that the following be collected by the organizational and staff development librarian and shared with each team as appropriate:   staff salary improvement and turnover data staff promotion data affirmative action/equal employment opportunity hiring data exit interview data trend data regarding budget allocation to the libraries (personnel, capital and operations) an outlined history of organizational changes in structure and staffing over the past 18 years.   this combined set of data would enable the libraries to understand what is actually happening to the organizational infrastructure that may or may not be contributing to the development of a healthy organization committed to customer service and continuous improvement.   in addition the works of roosevelt thomas, especially beyond race and gender: unleashing the power of your total work force by managing diversity, were recommended as a starting point to better understand how workforce diversity and complexity play out in the seemingly simple questions posed in the ocda survey. the consultant also reminded the uconn libraries administration to consider the sensitivity of minority responses to questions regarding diversity and discrimination in the ocda survey, and give less attention to “average” or total responses, since the underrepresented members of the staff comprised a very small minority.   delivery of the report and follow-up   after sending the written report to the vice provost for libraries, the consultant met with the vice provost; the organizational and staff development librarian; and the leadership council and discussed their reactions, answered questions and provided further clarifications. much of this conversation helped to familiarize the leadership council members with the systems view of organizations and to help them not personalize the information provided by the report and the focus groups. she then addressed an all staff meeting which was attended by almost the entire staff. in this meeting she again reminded the staff of the purpose and timing of the original survey. she stressed the commitment of the vice provost for libraries and leadership council to discover root causes and move toward making positive changes in the climate of the uconn libraries, and of their agreements to appoint staff teams to pursue further research and the development of recommendations for action.   library actions based on the consultant’s work   the organizational development consultant advised the formation of six teams to address internal systems and an initial three teams, prioritized by the consultant and comprised of staff members from across the library were formed. these teams were charged to “formulate a set of recommended actions that will contribute to a healthier organizational climate that promotes enhanced customer service by improving the libraries’: (1) leadership and team decision-making systems; (2) performance management system; and (3) hiring, merit, and promotion systems.” the findings, recommendations, and progress-to-date of each team are summarized below.   leadership and team decision-making system project team (ltdmspt)   this team was charged with formulating a set of recommended actions that would improve the libraries’ decision-making system with a focus on clarifying leadership roles of the libraries’ various stakeholders including leadership council members and team leaders, and studied the design, structure, and expectations from cross-functional, area, and project/task teams in the library.   to accomplish its work, the leadership and team decision-making system project team reviewed relevant policy documents and reports, and administered three in-house staff surveys. the first two surveys were targeted toward team leaders and team members and focused on empowerment and decision making; the third survey was administered to all staff and focused on leadership issues. the survey results indicated that most teams used consensus as a decision making tool and shared leadership emerged as the primary team model. however, there was a lack of agreement on and consistent practice of a clearly defined leadership model. there was also a lack of clarity in some areas about the model of consensus and the leadership roles in a learning organization.   based on its findings, ltdmspt made the following recommendations to clearly define both the roles and the responsibilities of leaders and individual staff:   carefully define various leadership models to determine if the library will be led from the top down, from the middle, or by shared leadership; individual staff be held responsible for their participation under the libraries’ new leadership and decision making structure; consider restructuring leadership council as part of reorganization process to possibly include some team leaders; all members of leadership council attend leadership training together and periodically participate in team building exercises; provide ongoing mandatory training to all team leaders on subjects like communication, facilitation skills, project management, team building, and managerial skills; reduce the number of standing cross-functional teams; and modify the libraries’ current meeting structure.   all of these recommendations were addressed by the libraries during its reorganization. the university of connecticut libraries reorganization project team (2009) recognized that to achieve a dynamic organization:   leaders must lead from within the organization, not from above; authority must be vested in the appropriate staff throughout the organization, rather than held only at the top; there must be clearly stated measurable goals, but also the ability to adapt and make timely changes to achieve those goals; and there must be a unified purpose and a determined focus on work that advances that purpose.   the model also clearly identifies the leadership roles of individuals within the organization:   the vice provost for university libraries is ultimately responsible for the overall success of the university of connecticut libraries. in administering the libraries, the vice provost consults with many constituent groups including other university administrators, the provost’s library advisory committee, the director’s council, the libraries’ planning team, the libraries’ team leaders and external entities. the assistant vice provost for university libraries and the program area directors are responsible for the success of their program areas, and together with the vice provost, the overall success of the libraries. team leaders are responsible for the success of their teams. the team leaders meet informally twice a month to provide support to each other on management and leadership issues, discuss important developments, explore opportunities for collaboration, and to have honest discussion about larger library issues such as staff morale, trust, and communication. attendance is entirely voluntary and there is no formal agenda. every other bi-weekly meeting is also attended by the vice provost of libraries. this “face time with the vp” gives team leaders an opportunity to raise any issues of interest or concern. library staff members are responsible for successfully carrying out their individual position duties, their team assignments, and for suggesting and implementing process improvements to better serve library users. the new libraries structure supports matrix relationships among staff members outside established program area teams. each staff member has one supervisor, but many staff members have both a primary reporting relationship to their program area team and, by virtue of their position duties, a secondary (non-reporting) relationship to a team outside their home program area.   a “decision table” clarifying the decision-making authority of various entities under the new leadership model was developed and shared with all staff. this table identifies the key entities including the vice provost, the director’s council, program area directors, collections budget team, planning team, area team leaders, and cross program teams and indicates their decision making role. the table identifies such core operations as charging and populating standing and project teams; allocating budgets; personnel decisions; setting library hours; hiring and promotion and clearly identifies who is in-charge of making decisions for each such function.   leadership council was renamed the directors’ council and while it still consists of the vice provost for libraries, the assistant vice provost, and the program area directors, its role and charge have been modified. the directors’ council advises the vice provost on the overall administration of the university libraries, charges standing teams, and approves operating budgets, staffing requests, and library-wide policies.   a new planning team that also reports to the vice provost for libraries was established. the planning team facilitates collaboration among staff members both within and outside their program areas. the team is charged with setting the libraries’ strategic priority goals, charging and populating cross program area project teams, updating the libraries’ strategic plan at least once every five years and administering the carry forward budget designated to fund strategic initiatives. five members, one from each program area, with staggered two-year term are eligible to serve on the planning team. members are elected by the library staff area by area.   a diversity advisory team was charged to coordinate the libraries’ diversity related initiatives. reporting directly to the vice provost for libraries, the team is comprised of rotating members reflecting staff diversity both in terms of human identity (e.g., ethnicity, national origin, gender, sexual orientation, and age), as well as program areas teams, and regional campuses within the libraries. the diversity advisory team serves as a resource to the vice provost for university libraries and the university libraries staff. it also works with the library student advisory council to seek continual student input on library collections, services, and diversity-related projects and initiatives.   as recommended by the leadership project team, leadership council members attended intensive interactive leadership training (burnham rosen group, 2009) together and also received individual counseling. leadership council members have subsequently tried to schedule one meeting a month to follow up on interactive leadership principles.   the libraries also instituted a mandatory training program for all team leaders and team members. the interaction and leadership training program covers topics such as providing feedback to others, resolving conflict, interaction skills, etc. all team leaders and members attended workshops in 2009 and refresher sessions were held in 2010 for team leaders.   as part of the reorganization, over 15 cross functional project teams were decommissioned and the libraries’ meeting structure was modified. facilitating communication among all staff is an important aspect of the planning team’s work. each month, the team organizes town hall meetings where staff can share ideas; propose and discuss new initiatives; learn about important developments within the library and beyond; and provide and seek feedback on projects. these meetings have been well attended by staff and have contributed to increased awareness of institution-wide issues.   in addition, the planning team conducts a “strategic goal development fair” every six months. this offers staff an opportunity to submit proposals that would enhance user services and contribute to the library’s five-year strategic plan. the fair offers staff a venue to brainstorm and develop ideas, garner feedback, and identify interested partners for collaboration. subsequently, staff members submit written proposals to the planning team, which reviews them against a set of criteria. besides increasing staff involvement in planning and decision-making, this process has generated goodwill and good-spirited competition among staff to forward ideas that would increase user satisfaction and contribute to the library’s strategic plan.    performance management project team   the performance management project team was charged to investigate performance management issues such as goal setting, coaching, performance evaluation, staff development, and progressive discipline.   to carry out its charge, the team consulted existing forms used for goal setting and reporting of annual activities and achievements. it also studied the current performance evaluation practices including training opportunities available to supervisors. feedback regarding the existing goal statement form indicated that most staff members found the form confusing. lack of explanations led to varying interpretation of terms such as strategic, operational, and individual goals, and outcome measures. the team also identified a need for quarterly performance reviews to ensure that performance evaluation became a yearlong exercise rather than once-a-year activity. it pointed out that quarterly reviews would facilitate regular dialogue between employees and supervisors regarding progress made on mutually agreed upon goals, assist in clarifying work priorities, set expectations and remove any surprises in the end. mandatory performance review training for all supervisors emerged as another issue requiring attention.   the performance management project team made the following recommendations to improve the libraries’ performance management systems:   revise and clarify the goal setting and the report of activity and achievements forms: to eliminate staff confusion, the team proposed revising the forms by clarifying terms, streamlining categories, and including additional identifying information. for example, the team recommended replacing strategic, individual and/or personal goals with a performance goal category. the performance goal is defined as a mutually agreed upon goal by supervisor and employee about what the employee is going to achieve. it is based on an employee’s current work assignment and is aligned to the area and strategic plans. training on setting smart goals: employees and supervisors receive training on how to set specific, measureable, attainable/accountable, results oriented, and timely (smart) goals. performance evaluation training for all supervisors must be made mandatory, and must be made available for new supervisory staff arriving after performance management period training has ended. mandatory contributions by all team leaders to team members’ evaluations. quarterly reviews of all staff to help ensure performance management is an ongoing process throughout the year rather than a brief, once-per-year discussion. development of an intranet site to serve as a repository for documents related to performance management including best practices.   five of the six performance management project team’s recommendations (excluding the intranet site) have been implemented. the revisions proposed to the goals setting form and the report of activities and achievements forms were accepted and adopted by the libraries in 2009. smart goals setting training for all employees is offered and mandatory performance management training for supervisors is required. team leaders who have team members with assignments on multiple teams must receive input from non-supervising team leaders when completing performance evaluations. quarterly reviews were adopted as standard practice and the quarterly review form developed by the team is being used by the entire library system. the quarterly reviews allow staff and their supervisors to touch base on a regular basis and monitor progress on mutually agreed goals, adjusting them if needed throughout the year to address competing priorities, new developments, and workload issues.   hiring, merit, and promotion systems team   the hiring, merit, and promotion systems team was charged to formulate a set of recommended actions that would improve the libraries’ hiring, merit, and promotion system. the team was to focus on the processes used, the involvement of peers, and the clarity of the goals and criteria. it is important to note that the frameworks of the libraries’ hiring, merit and promotion systems are set by the university or by collective bargaining agreements, therefore, any changes to the libraries’ policies or practices must fit within those frameworks. the team collected and reviewed data in the following areas: approximate average search costs in dollars and in staff time; uconn office of diversity & equity, uconn human resources and uconn libraries search policies and practices; uconn and uconn libraries promotion policies and practices; merit policies and practices; historical data from 2001-2006 on libraries merit awards; the ocda final report; comments from the ocda focus groups; and the results of a questionnaire sent to uconn libraries staff. the team also held a joint meeting with the performance management systems team to discuss shared concerns.   fewer opportunities to interact with and provide feedback about job candidates emerged as one of the major staff concerns. the advancement opportunities available to the libraries’ staff were found to be adequate, nevertheless, the hiring, merit, and promotion systems team recommended various enhancements to the current systems including additional educational opportunities for staff. the team also made a set of recommendations to make the university’s discretionary merit system more fair and transparent. listed below are various recommendations of this team:   hiring   ·         search committee composition: the immediate supervisor and at least one member of each job class in an open position’s team membership should be on the search committee. whenever possible diversify the search committee as needed by including, for example, departmental faculty, a staff member in a comparable position, or a counterpart from regional campus libraries, etc. ·         form search committee early enough in the hiring process that committee members can review job duties, job qualifications, and job postings before they are submitted to the human resources department. a shared understanding of the job expectations for the position would make the search committee’s work easier and consistent. ·         improving the search process: provide additional avenues for feedback from staff members not serving on search committees. ·         departures: revise the current exit interview questions. assess and prioritize the vacant position’s duties if they are to be assigned to one or more staff, including what will not get done. solicit volunteers, system-wide if possible and allow people to build on skills and interests. if the vacant position’s duties are not going to be distributed among existing staff but still need to be carried out in the short term, hire an end-date or special payroll employee to cover those duties until a final decision is made about filling the position.   merit   ·         establish a standard framework for university merit: align “library language” with “university merit language.” ·         communicate criteria for university merit effectively to staff: supervisors should clarify criteria for merit in conjunction with annual goal setting meetings. ·         make a clear case for merit recommendations: direct supervisor should clearly explain on the university merit form how an employee’s achievements are merit-worthy.   promotion   ·         educational opportunities: libraries’ union representatives should arrange for annual brown bag sessions to help library staff understand their options for promotion or reclassification. supervisors should understand the promotion options available to each staff member they supervise. supervisors should encourage their staff to pursue promotion and provide timelines. ·         new career ladders: investigate and implement a tiered promotion ladder for non-university librarian/non-university library assistant ucpea (university of connecticut professional employees association) and classified staff. ·         first time candidates for promotion, regardless of rank, should be assigned a mentor to guide them through the process.   considerable progress has been made related to hiring, merit, and promotions. new search committees charged since 2009 have incorporated several of the team’s recommendations including committee members with diverse background and forming search committees early enough to allow committee members to participate in drafting job postings and job description. all new hires are assigned mentors and are provided an “orientation checklist” to ensure that they are introduced to the libraries’ services and collections in a systematic way. in coordination with their supervisor, new staff schedule one-on-one meetings with relevant teams, areas, and library staff associated with their responsibilities to learn more about local policies, procedures, and issues of concerns.   a standard framework that aligns library examples with university merit language was established and communicated in 2010. this framework provides more guidance to supervisors on how to evaluate and rank staff performance for merit.   conclusion   the climatequal® results and the follow-up with the od consultant helped in identifying potential problem areas within the libraries’ internal systems. the od consultant made recommendations that led to the development of concrete roadmaps, benchmarks, and associated strategies to improve the libraries’ leadership, organizational structure and decision-making models; hiring, merit, and promotion systems; and the performance management system. the libraries progress on its strategic plan which includes relevant libqual+® metrics will serve as the barometer for gauging the effect of these changes.   the uconn libraries participated in libqual® again in 2010 and will likely re-administer the climatequal® survey to assess the staff's perceptions of actual progress toward creating a healthy climate that is in congruence with its values as a team-based learning organization. the libraries hope to continually improve and contribute to customers' success by providing a supportive climate where teamwork, diversity, and justice are reflected in their policies, procedures, and practices.     references   burnham rosen group. (2009). interactive leadershiptm: a workshop for developing exceptional leadership. boston: burnham rosen group.   cameron, k. (1978). measuring organizational effectiveness in institutions of higher education. administrative science quarterly, 23(4), 604-632.   deming, w. e. (1994). the new economics for industry, government, education (2nd ed.). cambridge, ma: mit press.   gano, d. l. (1999). apollo root cause analysis: a new way of thinking. yakima, wa: apollonian publications.   georgopolous, b. s. & tannebaum, a. s. (1957). a study of organizational effectiveness. american sociological review, 22(5), 534-540.   kanter, r. m., & brinkerhoff, d. (1981). organizational performance: recent developments in measurement. annual review of sociology, 7, 321-349. retrieved 20 may 2013 from http://www.annualreviews.org/doi/pdf/10.1146/annurev.so.07.080181.001541     kaplan, r. s. & norton, d. p. (1996). using the balanced scorecard as a strategic management system. harvard business review, 74(1), 75-85.   kyrillidou, m., & baughman, m. s. (2009). climatequal®: organizational climate and diversity assessment. college & research libraries news, 70(3), 154-157. retrieved 20 may 2013 from http://crln.acrl.org/content/70/3/154.full.pdf+html   phipps, s. (2004). the system design approach to organizational development: the university of arizona model. library trends, 53(1), 68-111.   scholtes, p. r. (1998). the leader’s handbook: a guide to inspiring your people and managing the daily workflow. new york: mcgraw-hill.   sims, h. p., jr., & lafollette, w. (1975). an assessment of the litwin and stringer organization climate questionnaire. personnel psychology, 28(1), 19-38. doi:10.1111/j.1744-6570.1975.tb00388.x   university of connecticut libraries reorganization project team. (2009). reorganizing the university of connecticut libraries: report of the plan 2014 reorganization planning team. unpublished report.       microsoft word es_furlan.doc evidence based library and information practice 2006, 1:4      60 evidence based library and information practice       evidence summary    library users expect link resolvers to provide full text while librarians expect  accurate results    a review of:  wakimoto, jina choi, david s. walker, and katherine s. dabbour. ʺthe myths and realities of  sfx in academic libraries.ʺ the journal of academic librarianship 32.2 (mar. 2006): 127‐ 36.    reviewed by:  wendy furlan  liaison librarian, university of queensland library  brisbane, queensland, australia  e‐mail: w.furlan@library.uq.edu.au      received: 30 august 2006            accepted: 15 october 2006      © 2006 furlan. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to determine how successful  the link resolver, sfx, is in meeting the  expectations of library users and librarians.    design – analysis of an online user survey,  library staff focus groups, retrospective  analysis of system statistics, and test  searches.    setting – two california state university  campus libraries in the united states:  northbridge, with over 31,000 students on  campus, and san marcos, with over 7,300  students on campus.    subjects – a total of 453 online survey  responses were submitted from library users,  421 from northbridge and 32 from san  marcos. twenty librarians took part in the  focus groups conducted with library staff  consisting of 14 of the 23 librarians from  northbridge (2 from technical services and  12 from public services), and 6 of the 10 san  marcos librarians (3 from technical services  and 3 from public services). no further  information was provided on the  characteristics of the subjects.     methods – an online survey was offered to  users of the two campus libraries for a two‐ week period in may 2004. the survey  consisted of 8 questions, 7 fixed response  and 1 free text. survey distribution was  enabled via a different mechanism at each  campus. the northbridge library offered the  survey to users via a pop‐up window each  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:4      61 time the sfx service was clicked on, while  the san marcos library presented the survey  as a link from the library’s home page.  survey responses from both campuses were  combined and analysed together. focus  groups were conducted with librarians from  each campus library on april 20th, 21st, and  29th, 2004. librarians attended focus groups  only with others from their own campus.  statistics were gathered from each campus’  local sfx system for the 3‐month period  from september 14, 2004, to december 14,  2004. statistics from each campus were  combined for analysis. the authors also  conducted 224 test searches over the 3‐ month period from july to september, 2004.    main results – analysis of the surveys  revealed that 80% of users expected to see a  full‐text article online when they clicked on  the sfx button; 20% expected to rarely or  never see one. responses also gave an  almost equal split when users were asked if  the sfx service met their expectations with  49.5% saying it did and 50.5% saying it did  not. the free text survey question asking for  comments on the user’s overall opinion of  sfx received 174 responses, 26% of which  were positive, 40% negative, and 34% mixed.  the primary theme expressed in 49% of all  comments received was disappointment in  not gaining full‐text access. thirty‐three  percent of other comments were classed as  having a general theme, while the remaining  19% of comments regarding the sfx service  were categorised with themes of complexity,  technical problems, efficiency, or confusion.    results from the librarian focus groups  differed between the two campuses.  northbridge librarians had 50% to 85%  confidence in the accuracy of sfx and were  generally impressed with the service. san  marcos librarians had 60% to 100%  confidence that sfx would work, however it  was also discovered that several participants  had suspicions regarding the accuracy of the  system.   the sfx usage statistics obtained covered  188,944 individual uses of sfx at both  campuses. statistics showed that 48% of  these uses resulted in the user clicking on an  option provided, either linking to full‐text,  catalogue look‐up, or inter‐library loan form  access. of total occurrences, 39.7% had a  link to full‐text displayed; this link was  accessed 65.2% of the time. forty‐seven  percent of sfx uses provided a catalogue  link (23.8% of which were accessed) and  37.9% of uses provided an inter‐library loan  link (8.4% of which were accessed).    the test searches revealed anomalies to take  into account when analysing the sfx usage  statistics, including that about 15% of sfx  requests display multiple full‐text links. of  the test searches conducted, 22.2% of full‐ text availability results ended in either  technical or accuracy errors and 8.8% of  catalogue look‐ups produced errors. in  those cases where errors did not result there  were also significant percentages of  instances where the library did not have  access to the desired resource: 35.3% of  searches correctly indicated that no full text  was available, and 57.6% correctly linked to  the catalogue to show that the periodical  was not held locally. while these are  correctly generated system results they are  still likely to be seen as unfavourable  outcomes by users.    conclusion – the results of the study  indicated that both library users’ and  librarians’ expectations of sfx were slightly  higher than their actual experiences.  librarians’ primary concerns related to the  need for more accurate results while library  users wanted more full text. it was noted  that many complaints associated with sfx  were likely to actually be problems with  systems that sfx links into rather than the  software itself. although imperfect, sfx,  and link resolvers in general, are noted to be  a vast improvement on the many separate  searches required in the past to locate full  evidence based library and information practice 2006, 1:4      62 text and undoubtedly user expectations and  demand for 100% seamless accessibility will  grow.    commentary    this study can provide limited insight and a  learning opportunity to any library, which  has or is considering implementing link  resolver software and wishes to examine the  likely expectations their users and staff will  have of such a system.     the sample population for the user survey  had no defined criteria and responses were  only received from library users who chose  to access the online survey. although the  number of responses for the northbridge  campus survey (421) was 13 times that of  the san marcos campus (32), if the estimated  student populations of each campus are  considered as an estimate of total potential  responses, response rates can be estimated  as 1.4% for northbridge and 0.4% for san  marcos. both figures fall well below  statistical significance and results could not  be said to be representative of the entire  population. the fact that the number of  responses for san marcos was so small  brings into question whether those results  should have been analysed at all. the  authors did not report any statistical  significance tests on response numbers prior  to analysis of the results, which would have  certainly revealed how representative the  study was. furthermore, as the surveys  were made accessible via different avenues  from each campus library, data collection  differences between the two campuses  would have had a major impact on not only  the number of responses but also on the  attributes of the responding population.  given these differences it would probably  have been wiser to analyse campus figures  independently of one another. the user  survey was attached as an appendix for  readers to review the content, but the two  questions included to gain information  about the population were not discussed at  all by the authors. this is unfortunate as  trends of the different user groups may have  provided an interesting insight. responses  to the open‐ended question in the survey  were categorised by two of the authors by  their positive or negative tone. this is a  valid qualitative analysis process, but the  calculation of percentage figures from this is  highly unscientific and therefore all analysis  based on these figures is flawed. when  concluding, the authors do state that  generalisations can not be made from the  results and the “non‐randomized” nature of  the survey distribution is acknowledged.    the focus groups conducted with librarians  at each campus incorporated a good  population sample with 61% of northbridge  and 60% of san marcos librarians taking  part. documentation of further  characteristics of the focus group subjects,  for example number of years in the  profession, could also have enhanced  analysis and showed trends in the results.  while comments documented from these  focus groups are interesting, the absence of  the list of questions asked of the groups  makes it difficult for readers to put  comments into context or to replicate the  process. the authors again categorised  comments and calculated percentages on  this basis, and repeated the unscientific  practice from the user surveys.     the system statistics discussed provide an  insight into the use of sfx, although as with  the results from the test searches, do not  directly help to meet the study objective of  how successful sfx is in meeting the  expectations of library users and librarians.  although it poses some possibilities as to  why the system may not produce a  favourable outcome, it is impossible for  system‐generated statistics and author‐ conducted searches to predict or replicate  the actual experiences and expectations of  users and librarians. the validity of the test  evidence based library and information practice 2006, 1:4      63 searches is also questionable as searches  conducted by information professionals are  not necessarily comparable with those  performed by the average library user. the  lack of detail regarding the searches again  makes results difficult for readers to put into  context and impossible to replicate.    to obtain more reliable and representative  results, future studies should consider  several factors: maintaining consistency in  survey delivery methods; achieving  significant response rates from users;  gathering, collating and analysing  demographical information of subjects (i.e.  student or staff level and status, field of  study, number of years at the institution);  utilising test searches and system statistics  retrieved prior to user studies in the  formation of relevant and effective survey  and focus group questions; not deriving  statistics from qualitative data; and ensuring  all support documentation are included in  final publications.                                                  overall, the study provides an interesting  insight into the expectations library users  and librarians have for link resolvers and  the disparity of these expectations, as library  users expect full‐text accessibility and  librarians expect an accurate system.  however, due to the limitations of the  methodology and analysis, further evidence  is warranted if decision‐making regarding  the implementation of link resolver software  and how to best meet or investigate user  expectations is to be based on such research. evidence based library and information practice evidence based library and information practice 2010, 5.3 73 evidence based library and information practice ebl 101 evaluating the results of evidence application, part one virginia wilson client services librarian, murray library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca received: 03 aug. 2010 accepted: 06 aug. 2010 2010 wilson. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. after the question has been formulated and the evidence has been found, appraised, and applied, the next step is evaluation. it is easy to move on to something else and skip the evaluation. lack of time and the desire to get on with what needs to be done next can thwart our best intentions. but evaluation is an important step in evidence based library and information practice (eblip) and it should be carried out on two levels: the practitioner level, where the person undertaking eblip looks at his or her own performance in the process; and the practice level, where what has been implemented is assessed. part one of ‚evaluating the results of evidence application‛ will deal with evaluation at the practitioner level. in the past few years, the idea of reflective practice has been written about in conjunction with eblip. cultivating the practice of reflection is helpful in all realms of professional practice, and especially helpful in terms of eblip, because it helps practitioners continue to learn and grow in daily practice. booth (2004) writes about reflection in evidence-based practice for information professionals: a handbook. there, he talks about the notion of reflective practice as first discussed by donald a. schön in the reflective practitioner: how professionals think in action. schön (1983) differentiates between ‚reflection-on-action‛ and ‚reflection-inaction‛: essentially reflecting after the fact and reflecting as you go. grant (2007) published a systematic review which found that published reflections among library and information professionals have shifted from ‚reminiscence and retrospective accounts of careers and organizational development‛ lacking much analytical insight to ‚analytical accounts of reflection both ‘on’ and ‘in’ action‛ (p. 164). and in the last issue of evidence based library and information practice, koufogiannakis (2010) offers some thoughts on reflection in practice where she states that ‚research knowledge only takes us so far‛ and that through reflection, we can gain ‚a better understanding of how and why mailto:virginia.wilson@usask.ca evidence based library and information practice 2010, 5.3 74 we do what we do as librarians and information professionals‛ (p. 2). evaluating your own performance as an evidence based practitioner involves contemplating and analyzing what you did, or in other words, reflecting on action. booth (2004) suggests some questions to ask yourself:  did i ask a specific focused question?  did i find efficiently the best evidence to answer my questions?  did i evaluate the evidence reliably according to validity and usefulness?  did i apply the results of the research appropriately to a specific user or group of users? (p. 127) of course, you can reflect in more (or less) detail on your own performance. booth (2004) claims that ‚*reflective practice+ can be as simple, or as complicated, as you want it to be‛ (p. 130). reflection should also take place throughout the eblip process – reflection-in-action. while it may take awhile to remember to stop and reflect, especially if the project has momentum and you are worried that stopping to ponder might jeopardize it, getting into the habit of thinking about daily practice as it happens will help strengthen reflection as a continuous process in practice. thoughtfully considering what is going on as it happens brings an element of consciousness to professional practice, and can help get away from the ‚we’ve always done it this way‛ trap. so how might an evidence based practitioner reflect in action? booth (2004) suggests several ways to get started. solo reflection might take place in a diary or with a mentor on an ongoing basis. group reflection can be looked at as action learning, where the group collaboratively, self-reflectively, and critically documents it at all states of the process. in the era of web 2.0, reflection can take place on a blog or a wiki, on a social networking site such as facebook with a group of like-minded professionals, or however else you might currently utilize 2.0 technologies. it would be beneficial to create communities of reflective practice where experiences and knowledge could be shared. try to make time for reflecting upon and evaluating your own performance as an evidence based librarian or information professional. the more you self-assess, the easier the process will become and the more you will learn about using evidence in practice. next time, i’ll look at evaluating the results of evidence application at the practice level. references booth, a. (2004). evaluating your performance. in a. booth & a. brice (eds.), evidence‐based practice for information professionals: a handbook (pp. 127-137). london: facet. grant, m.j. (2007). the role of reflection in the library and information sector: a systematic review. health information and libraries journal, 24(3), 155-166. koufogiannakis, d. (2010). thoughts on reflection. evidence based library and information practice, 5(2), 1-3. schön, d.a. (1983). the reflective practitioner: how professionals think in action. new york: basic books. evidence based library and information practice 2011, 6.3 1 evidence based library and information practice evidence summary master’s students in history could benefit from a greater library sensitivity and commitment to interdisciplinarity, and from more efficient document delivery a review of: sherriff, g. (2010). information use in history research: a citation analysis of master’s level theses. portal: libraries and the academy, 10 (2) 165-183. doi: 10.1353/pla.0.0092. reviewed by: r. laval hunsucker information and collection specialist emeritus university libraries, universiteit van amsterdam amsterdam, breukelen, the netherlands email: amoinsde@yahoo.com received: 01 mar. 2011 accepted: 15 july 2011 © 2011 hunsucker. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – this study sought to determine the characteristics of research materials used by history students in preparing their master’s theses. of which information resources formats did such students make use, and in what proportions? what was the age distribution of resources used? what was the dispersal over journal titles and over subject classification, i.e., the degree of interdisciplinarity? to what extent did the master’s students make use of nonenglish-language materials? to what extent did their institution’s library hold the resources in question? the investigator was especially interested in finding quantitative support for what he terms two “hypotheses.” the first of these is that historical research depends to a high degree on monographs, journal articles being far less important to it than they are to research in, especially, the natural sciences and technology. the second is that the age distribution of resources important to historical research is much flatter and longer than that of resources upon which researchers in the natural sciences and technology rely. design – citation analysis, supplemented with comprehensive catalogue searches. mailto:amoinsde@yahoo.com� http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/� evidence based library and information practice 2011, 6.3 65 setting – southern connecticut state university (scsu), a mid-sized public university located in new haven, connecticut, u.s.a. subjects – ma and ms theses (n=47) successfully submitted to the department of history over the period from academic year 1998/1999 through academic year 2007/2008, inclusive. methods – the investigator initially identified the theses through a search of the online catalogue (“consuls”) of the connecticut state university system, and retrieved all of them in either electronic or hard-copy form. he then subjected all citations (n=3,498) listed in the references sections of these theses to an examination in order to identify for each cited resource the format, the age, the language, and, in the case of scholarly journal articles, the journal of publication. he carried out bibliographic searches in order to rectify any citations which he had noted to be faulty or incomplete. the study took no account of possible additional citations in footnotes or endnotes or in the text, and did not measure citation intensity (whether, for instance, a thesis referred only once, or perhaps many times, to a given resource). duplicates “were ignored.” he furthermore performed systematic searches in consuls and in the library of congress (lc) online catalogue in order to establish, insofar as possible, into which assigned lc classification class each resource fell, and whether it belonged to the holdings of the scsu library. “holdings,” as used here, includes physical resources owned, as well as those resources to which the library has licensed access. not marked as either “held” or “not held” were: resources available online without restriction or charge, items not identified in either consuls or the lc catalogue, and all government documents. ages of cited resources were calculated based on the edition or version date actually given in a student’s citation, without any consideration of a possible earlier date of the original version of the publication or document concerned. main results – format, age distribution, and journal frequency. the local citation analysis found that 53.2% of all cited resources were monographs, 7.8% were scholarly articles, 5.3% were contributed chapters in books, and 0.6% were dissertations or theses. non-scholarly periodicals accounted for 15.7%, government documents for 6.7%, and freely available web documents for 4.1%. the remainder, approximately 6.5%, comprised archival papers, judicial documents, directories, interviews, posters, audiovisual materials, and 13 other formats. cited resources, measured back from the date of acceptance of the citing thesis, ranged from 0 to 479 years old; the mode was 3 years, but the median was “25” (p. 170) or “26” (p. 177) years. just over 70% (i.e., 2,500 cited resources) were more than ten years old. almost one thousand of the cited resources were fifty or more years old. the 274 scholarly journal articles included in the references sections were spread over 153 distinct journal titles, of which 105 titles made only one appearance, and 136 titles three or fewer appearances. the mean was 1.8 appearances. subject dispersal and language. of the 2,084 cited resources for which lc classification was locatable, 51.5% had a classification other than history, i.e., other than class c, d, e, or f. nearly two thirds (66.0%) of the cited scholarly journal articles had appeared in journals with a focus other than history. (note: table 4 is incorrect, precisely reversing the actual ratio.) of all cited items, 98.5% were in the english language. half (27) of the non-english-language resources cited were in korean, all cited in the same thesis. books (i.e., monographs plus compilations from which contributed chapters were cited) accounted for 87.0% of foreign-language citations. more than four fifths of the examined theses (83.0%) cited not a single non-englishlanguage resource. local holdings. of all 3,498 cited items, 3,022 could be coded as either “held” or “not held” by the scsu library. of the items so coded (not, as evidence based library and information practice 2011, 6.3 66 indicated on p. 180, of all cited items), scarcely two fifths (41.0%) belonged to the library’s holdings. the holdings percentage was highest (72.6%) for the 274 scholarly journal articles cited, followed by the 186 contributed chapters (50.0%), the 550 non-scholarly periodical items (49.5%), and the 1,861 monographs (46.8%). for other cited formats, the percentage was much lower, and in some cases, e.g., for the 55 archival and the 44 judicial documents, it was 0.0%. of the 54 foreign-language resources cited, the institution’s library held only two. conclusion – the investigator concludes that his study’s findings do indeed lend quantitative support to his two “hypotheses.” this outcome will surprise few, if any, librarians; it is in accord with what koenig (1978) long ago saw as a matter of “intuition” and “all conventional wisdom,” something that many subsequent studies have confirmed. sherriff accordingly recommends, firstly, that collections which strive to support historical research should, in matters of acquisition policy and budget allocation, take serious account of that field’s relatively strong dependence on monographs. secondly, the data on age distribution carry obvious implications for librarians’ decisionmaking on matters such as de-accessioning and weeding, relegation to remote storage, and retrospective acquisitions. this finding should also be considered, for instance, in connection with preservation policy and the maintaining of special collections. he even suggests that librarians “need to teach students the value of reviewing literature historically and showing them how to do so effectively” (p. 177). sherriff considers a number of further (tentative) conclusions to be warranted or suggested by the results of this study. first of all, that historical research is now characteristically an interdisciplinary matter, in the sense that it requires extensive access to information resources, including journals, which libraries have traditionally not classified as belonging to the discipline of history itself. for a library supporting such research, this phenomenon “has implications for matters including collection budgets, reference work, bibliographic instruction, and the location of collections and departmental libraries” (p. 168). it also means “that librarians working with history students and history collections need to be aware of the relevant resources in other disciplines. this can improve reference work, research assistance, and bibliographic instruction; it may also help the coordination of acquisitions across departmental lines” (p. 179). secondly, one may conclude that “there is no ‘core’ collection of journals for history” (p. 178) which will be able to satisfy a large proportion of master’s students’ research needs. thirdly, the fact that a library such as scsu’s holds significantly less than half of what master’s students require for preparing their theses “may exercise a narrowing effect on students’ awareness of the existing literature on their topics” (p. 180), “increases the importance of departmental faculty, reference librarians, and subject specialist librarians drawing students’ attention to resources beyond the library’s catalogues and collections” (p. 180), and requires that the library give serious attention to effective document delivery arrangements. finally, this study’s finding that only a small percentage of master’s students in history made use of nonenglish-language materials, but then in certain cases used them rather extensively (27 korean items cited in one thesis, ten italian in another, nine spanish in yet another), suggests that acquisition, or at least proactive acquisition, of such materials needn’t be a priority, as long as, once again, the students concerned have easy access to efficient and affordable document delivery services. sherriff does concede, however, that his finding could indicate “that students are unaware of relevant resources in other languages or are aware of them but lack the language skills necessary to use them” (p. 179). evidence based library and information practice 2011, 6.3 67 commentary sherriff did not set out to answer a limited, well constructed, explicitly stated, and practiceoriented research question. his results are not of the kind that practitioners can put directly or easily to use. the significance for us of his study is that it does now provide us with some welcome hard data on certain features of thesis citation practice in a specific context (a small history program, without doctoral component, at a young non-research-extensive state university), and in particular on: the high degree of interdisciplinary and journal-reference dispersal; the low degree of local availability of cited resources; and the extremely low frequency of foreign-language resource citation. this is a good beginning. what we now need, to give such data genuine evidentiary value for others, is further and comparable data from similar programs elsewhere, as well as findings, preferably longitudinal, on the same variables from studies of history research at other types of institutions and on other levels. this reviewer was therefore indeed surprised to note sherriff’s strong implication that his conclusions from this single study should be taken as generalizable to graduate research in history at all types of institutions, or indeed even to scholarly research by historians in general. to buttress such a view, he cites no relevant earlier research beyond the tentative findings of a study of citations to journal articles in the fields of geology and biology (zipp, 1996). it is in fact possible, and, if so, quite significant, that his quantitative results regarding interdisciplinarity are representative for historical research more broadly (buchanan & hérubel, 2011; dalton & charnigo, 2004; delgadillo & lynch, 1999; hellqvist, 2010). so too, perhaps, those regarding age-distribution of cited resources (lowe, 2003; smyth, 2011). but that they are in further respects representative is open to significant doubt, in view of findings reported by other researchers (broadus, 1985; dalton & charnigo; delgadillo & lynch; haycock, 2004; jones, chapman, & woods, 1972; knievel & kellsey, 2005; lowe 2003; pancheshnikov, 2007). even his assumption that master’s research in its citation practice and patterns does not as such significantly differ from doctoral research, conflicts not merely with conventional wisdom but also with what numerous investigations have shown, with regard not only to monograph/journal article ratio but also to the usage of foreign language resources, of non-scholarly periodicals, and of primary source documents (barrett, 2005; eckel, 2009; feyereisen & spoiden, 2009; kushkowski, parsons, and wiese, 2003; pancheshnikov, 2007; smyth, 2011). in at least all of these other respects one can, unfortunately, not view sherriff’s study as contributing meaningfully to the evidence base for practitioners concerned with library support for history researchers more generally. the potential usefulness of sherriff’s findings suffers as well from certain inherent, research design, and methodological shortcomings. the population studied is unexemplary for the academic discipline at large, since many categories of historical scholarship are poorly represented, or not represented at all. more than two thirds of the theses dealt with north american history. there is no apparent sensitivity to differing sub-disciplines, or diverse historiographic or theoretical/critical approaches, though these may vary considerably in resource requirements and citation practice (hellqvist, 2010). his commitment to an entirely quantitative and unobtrusive approach meant that he – notably unlike, for example, junni (2007) in her study of master’s students; or fuchs, thomsen, bias, and davis (2006) in their application of “behavioral citation analysis” to doctoral students – had no contact with any students or faculty members. interviewing or surveying them could have yielded crucial relevant information on prevailing citation culture, guidelines, or requirements; on specific advisor or departmental expectations; and so on. having opted for strictly quantitative methods, he can, of course, also tell us nothing about how the evidence based library and information practice 2011, 6.3 68 students discovered or accessed – or why they may have failed to discover or to access – relevant materials; nor, for example, can he report on the extent to which they made use of resources which they eventually did not cite, or perhaps mentioned only in the text or in notes. the report, furthermore, provides no demographic information. measurement bias is also present: the theses were submitted in the years 1998-2008, but sherriff apparently coded all of their cited resources for local availability on the basis of catalogue searches which he performed at a later point, presumably in 2009. this of course compromises the validity of the holdings data reported. to what extent, we can only guess – but it seems likely that, during the period from 1998 to 2009, much will have changed as far as local resource availability (and citability?) was concerned. such developments will moreover have exercised a diachronically confounding influence on all of the variables investigated by sherriff, in some cases perhaps a fairly significant influence. the above-mentioned study by fuchs et al. (2006) incorporated this same measurement bias, and they frankly stated that this was “problematic,” and “a clear limitation.” sherriff, on the other hand, leaves entirely unmentioned the same deficiency in his own research design. indeed, he is in general insufficiently forthcoming about his investigation’s limitations. it takes hardly any account of the role of electronic formats (or the concomitant blurring of boundaries between traditional formats), of emerging new scenarios for scholarly communication, or of the uncertain outlook for monographic publishing. a more fundamental defect is that his article displays only perfunctory sensitivity to the inherent weaknesses of citation analysis – or, as in his case more specifically, references list analysis – as a research method. much has been written concerning its weaknesses and validity problems, but sherriff makes no reference even to publications as relevant as those of line (1977, 1978), smith (1981), macroberts and macroberts (1989, 2010), kelland and young (1994), nor even to haycock’s (2004) specific warnings regarding potential graduate student citing tendencies. we should by now all be aware that inclusion in a references list is not necessarily the same thing as use, and that use – of whatever variety – is something quite different from need. yet sherriff, having eschewed any qualitative research component, nonetheless clearly implies that his study is a contribution toward understanding not only history researchers’ resource and library use, but even their information resources needs (or simply “information needs,” as he at one point puts it). this makes all the more striking his apparent obliviousness, in his literature review and references, to many relevant previous studies, as well as his paucity of explicit suggestions concerning where further research is necessary or would be useful. finally, we may note that, although the article presents (in six tables and one figure) only selected statistics from the research study, the author gives no indication whether full data are also available to interested readers. line (1977, 1978) found that citation analyses were of no value, and were indeed irrelevant, to practical librarianship. kelland and young (1994) concluded that the relationship between citation activity and library materials use was for numerous reasons “inherently problematical,” and saw the two measures as no more than “moderately correlated.” still, the question remains: even if we do in principle accept such a correlation, what should this imply for our actual library and information services practice? sherriff’s study contributes only modestly toward answering that question, even with regard to library support for research in history. it does, though, at least suggest a few specific considerations which librarians can bring to bear, preferably in combination with other, in particular qualitative, indicators, in attempting to make their services more evidence based library and information practice 2011, 6.3 69 appropriate for a particular category of student researchers in a particular kind of institution. references barrett, a. (2005). the information-seeking habits of graduate student researchers in the humanities. the journal of academic librarianship, 31(4), 324-331. broadus, r. n. (1985). a proposed method for eliminating titles from periodical subscription lists. college & research libraries, 46(1), 30-35. buchanan, a. l., & hérubel, j.-p. v. m. (2011). taking clio's pulse . . .: or, examining characteristics of monographic publications reviewed by american historical review. journal of scholarly publishing, 42(2), 160-181. dalton, m. s., & charnigo, l. (2004). historians and their information sources. college & research libraries, 65(5), 400-425. delgadillo, r., & lynch, b. p. (1999). future historians: their quest for information. college & research libraries, 60(3), 245-259. eckel, e. j. (2009). the emerging engineering scholar: a citation analysis of theses and dissertations at western michigan university. issues in science & technology librarianship, 56. retrieved 7 may 2011 from http://www.istl.org/09 winter/refereed.html feyereisen, p., & spoiden, a. (2009). can local citation analysis of master's and doctoral theses help decision-making about the management of the collection of periodicals? a case study in psychology and education sciences. the journal of academic librarianship, 35(6), 514-522. fuchs, b. e., thomsen, c. m., bias, r. g., & davis, d. g., jr. (2006). behavioral citation analysis: toward collection enhancement for users. college & research libraries, 67(4), 304-324. haycock, l. a. (2004). citation analysis of education dissertations for collection development. library resources & technical services, 48(2), 102-106. hellqvist, b. (2010). referencing in the humanities and its implications for citation analysis. journal of the american society for information science and technology, 61(2), 310–318. doi: 10.1002/asi.21256 jones, c., chapman, m., & woods, p. c. (1972). the characteristics of the literature used by historians. journal of librarianship and information science, 4(3), 137-156. doi: 10.1177/096100067200400301 junni, p. (2007). students seeking information for their masters' theses: the effect of the internet. information research, 12(2), paper 305. retrieved 7 may 2011 from http://informationr.net/ir/12 2/paper305.html kelland, j. l., & young, a. p. (1994). citation as a form of library use. collection management, 19(1&2), 81-100. doi: 10.1300/j105v19n01_04 knievel, j. e., & kellsey, c. (2005). citation analysis for collection development: a comparative study of eight humanities fields. the library quarterly, 75(2), 142 168. doi: 10.1086/431331 koenig, m. e. d. (1978). citation analysis for the arts and humanities as a collection evidence based library and information practice 2011, 6.3 70 management tool. collection management, 2(3), 247-264. doi: 10.1300/j105v02n03_06 kushkowski, j. d., parsons, k. a., & wiese, w. h. (2003). master's and doctoral thesis citations: analysis and trends of a longitudinal study. portal: libraries and the academy, 3(3), 459-479. doi: 10.1353/pla.2003.0062 line, m. b. (1977). on the irrelevance of citation analyses to practical librarianship. in w.e. batten (ed.), eurim ii: a european conference on the application of research in information services and libraries: 23-25 march 1976, rai international congrescentrum, amsterdam, netherlands: proceedings (pp. 51-56). london: aslib. line, m. b. (1978). rank lists based on citations and library uses as indicators of journal usage in individual libraries. collection management, 2(4), 313-316. doi: 10.1300/j105v02n04_04 lowe, m. s. (2003). reference analysis of the american historical review. collection building, 22(1), 13-20. doi: 10.1108/01604950310457168 macroberts, m. h., & macroberts, b. r. (1989). problems of citation analysis: a critical review. journal of the american society for information science, 40(5), 342-349. doi: 10.1002/(sici)1097 4571(198909)40:5<342::aid asi7>3.0.co;2-u macroberts, m. h., & macroberts, b. r. (2010). problems of citation analysis: a study of uncited and seldom-cited influences. journal of the american society for information science and technology, 61(1), 1–12. doi: 10.1002/asi.21228 pancheshnikov, y. (2007). a comparison of literature citations in faculty publications and student theses as indicators of collection use and a background for collection management at a university library. the journal of academic librarianship, 33(6), 674-683. smith, l. c. (1981). citation analysis. library trends, 30(1), 83-106. smyth, j. b. (2011). tracking trends: students’ information use in the social sciences and humanities, 1995–2008. portal: libraries and the academy, 11(1), 551 573. doi: 10.1353/pla.2011.0009 zipp, l. s. (1996). thesis and dissertation citations as indicators of faculty research use of university library journal collections. library resources & technical services, 40(4), 335-342. / evidence based library and information practice feature   eblip7 lightning strikes: how i’ve been inspired by evidence based library and information practice (eblip)      2013. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   eblip7 lightning strikes were 5 minute/20 slide talks on the topic “how i’ve been inspired by evidence based library and information practice (eblip).” the twist was that the slides automatically advanced every 15 seconds! the goal was to hear individuals’ stories and perspectives, and conference organizers encouraged a broad interpretation of the topic. in the following talks, slides are described in italics.   thank you to the lightning strikes participants who provided a version of their talks for this issue. thank you as well to lorie kloda and joanne gard marshall who also presented lightning strikes. their talks are finding other venues as published research papers.   people of influence margaret haines, university librarian carleton university, ottawa, canada*   when virginia asked me if i would talk about the impact that evidence based library and information practice (eblip) has had on my career, i realized that it was the actions of the people who promoted evidence based medicine (ebm) which had the most influence on me. so, this is a brief tribute to them and it begins when i became the first national health service (nhs) library adviser in the uk in 1994.   that year, the evidence based practice “bug” got me at a canadian medical association conference when i heard about the work of the mcmaster gurus gordon guyatt, brian haynes and andy oxman. i was surprised that no other librarians were there at the conference because it seemed relevant to the work that librarians did when searching the literature to find the best evidence to support medical decision making. this was confirmed by a general practitioner in the audience who complained that he had no time to search the literature because he did not have his own personal librarian. i was later thrilled to discover that the mcmaster gang did have their very own personal librarian, ann mckibbon, who demonstrated just what librarians could do by playing a key role in sifting and appraising the research evidence and in conducting systematic reviews.   when i came back to the uk, i became very interested in the role of librarians in evidence based medicine (ebm) and wrote an editorial for health libraries journal challenging the uk health library community to get involved in some way. as the nhs library adviser, i felt i ought to know more about what i was preaching and decided to research archie cochrane, who was said to be the “father” of ebm. that led me to professor iain chalmers, the head of the uk cochrane centre who was zealous about the need to base medical practice on evidence, and who asked me to help interview librarians for a key role in the work of his centre.   together we chose carol lefebvre who proved her weight in gold standards through her work on refining medline search strategies. carol has taught so many of us over the years how to do it right and with her quiet confidence was a perfect complement to iain, the passionate promoter. but the real driver in getting all librarians into ebm was dr. muir gray, director of r&d in oxford who supported me at the national r&d directors’ table to pitch how nhs librarians could be trained to do systematic reviews. muir put money where his mouth was and funded his regional librarian to do something about it.   enter dr. judy palmer, nhs regional librarian for oxford, and her deputies including david stewart and anne brice, who started a training program for librarians to improve their searching skills and to learn critical appraisal skills. it was the first in the uk and had a profound influence on all health librarians and on professional practice everywhere.   muir did something else which was significant – he brought dr. dave sackett to the uk to start the centre for evidence based medicine in oxford.  dave of course is another father of ebm and helped to introduce this paradigm more widely in the nhs. soon there were advocates everywhere: anna donald, andrew haines (no relation), alison hill, vikki entwhistle, trevor sheldon, jeremy wyatt, ruairidh milne, to name but a few. but what sets these ones apart is that they actively sought collaboration with librarians whether in getting research into practice (gripp) programs or in evidence based purchasing or patient advocacy.   and of course there were the librarians in these collaborations and here are some familiar faces –  andrew booth and anne brice, as well as some less familiar to you in north america such as julie glanville from the york centre for reviews and dissemination, bob gann and gill needham who worked on evidence based patient care, robert kiley from the wellcome centre, david stewart from the nhs, and my successor as nhs library adviser, veronica fraser.   at this point, i want to pay tribute to two mentors – sir john pattison, nhs r&d director and barbara stocking, chief executive of the nhs southeast for seeing research dissemination as a form of knowledge management and for taking a chance on putting a librarian, me, in charge of research and development for the nhs south east region. it was my biggest challenge.   so all of these people taught me about evidence based practice, getting research into practice and the key role of librarians in managing the knowledge base of health care, but at the same time, some librarians were applying this approach to our own profession of librarianship. enter anne brice and andrew booth who were among the first to promote evidence based library and information practice (eblip) and who wrote the seminal work on the subject and who have shaped the debate internationally for years. and andrew is just as fond of the pun now as when i hired him at the king’s fund centre nearly 25 years ago.   there were many others who shaped the eblip movement such as the three here from north america – joanne marshall, jonathan eldredge and denise koufogiannikis, all of whom are actually here at the conference and need no introduction. but i owe special thanks to joanne who shaped my career by always being someone i wanted to be, and tried to emulate.   and of course, there were other research leaders. in the uk, for example, there were chris urquhart, alison brettle and maria grant. without their continual efforts to provide the research evidence for our profession and to embed eblip in library education, the eblip movement would not have moved forward as fast as it did.   and eblip truly is an international phenomenon, thanks to many other librarians and researchers such as those listed here – gilliam hallam, helen partridge, ann ritchie, rowena cullen, lotta haglund, and many others around the world. i apologize for only mentioning a few of you and i would specially like to acknowledge the contribution of niels ole pors who sadly died this year.   then, there were librarians who saw eblip as part of being a professional. as president of the chartered institute for library and information professionals (cilip) in the uk, i worked with my presidential team (margaret watson, debby shorley) and staff at cilip like marion huckle, to embed eblip into cilip's qualifications framework so that every member had to demonstrate how they used evidence to shape their learning and practice.   in north america, the canadian association of research libraries (carl), the association of research libraries (arl), and the medical library association (mla) are also ensuring that librarians have the research skills and understanding to be evidence based practitioners. i particularly enjoyed working with dr. vicki williamson, our host here at saskatchewan, on carl's competencies statement and also loved seeing how she put what we were preaching into practice in her own library.   and now i am learning from the younger generation, people such as you and people in my own team, like laura newton miller, who really get what it is all about and who are adding to our knowledge base and influencing their peers through their research and professional practice.   so, what have i learned? i learned that behind every great “evidence based” practitioner, there is a librarian finding the evidence, and that behind every great librarian, there is another librarian generating the evidence.  thank you to all the evidence based practitioners, librarians and others, who have helped me along the way.   *margaret haines worked in the united kingdom from 1989 to 2006, where she held five positions: director of the king’s fund centre library (1989-1994), nhs library adviser (1994-6), chief executive of the library and information commission (1996-2000), director of research and knowledge management for the nhs southeast (2000-2003), director of information systems and services at king’s college london (2003-2006). she returned to canada in 2006 to become university librarian at carleton university in ottawa, her hometown.   how i’ve been inspired by eblip virginia wilson, director centre for evidence based library and information practice (c-eblip) university library, university of saskatchewan, canada   in library school, i wrote a paper exploring how physicians found the answers to clinical questions and was horrified to find out that often the research literature was not consulted. it was also in library school that i was hired by three librarians to assist on an evidence based librarianship project.   an image of the original charlie’s angels labelled denise koufogiannakis, linda slater, and ellen crumley those three women inspired me to not only look for research to inform my practice, but also to do research. i’m intrigued by research, see the value in research, and figure if it’s out there and it’s useful, why don’t we use it?   research cartoon i’ve never understood why some people seem threatened by the thought of using research evidence to inform decision making. and those people are out there. i’ve met them. but don’t we need all the help we can get to get it as close to right as we possibly can?   a vintage advertisement for hair rental (“you know it makes sense”) it just makes sense! yes, we have our professional knowledge and skills. but if we can get more information, it’s better, right? more is more. by looking to the research we might find a piece of the puzzle that we were missing or overlooking.   an image of fabio so i got the fever, the eblip fever, and i started going to eblip conferences and writing for the eblip journal. i was using research to inform my practice. i was dreaming about andrew booth in my sleep.   wait a minute – that wasn’t andrew.   “everybody panic!” cartoon then all of a sudden i was applying to host the international conference, and then planning the conference, and starting up a centre for evidence based library and information practice. and i thought, wow i’m on a bit of a roller coaster!   image of a roller coaster but no one is ever alone on a roller coaster, unless it’s early in the morning at the amusement park. there are lots and lots of other library and information professionals out there also with the eblip fever. it’s a little bit of a pandemic!   image of dr. johnny fever from 70s tv sitcom wkrp in cincinnati and you come to the conferences, and use evidence in practice, and conduct your own research, and disseminate your own research, and actively participate, and as seth godin says, we have a tribe.   image of seth godin there’s an eblip community. yes, it’s loose and there’s no formal organization just yet, but when you look at the things that have come out of this community in only 15 years, it’s very exciting. the journal, the conference series, the connections and partnerships.   image of a big bag of evidence (yes, it’s difficult to describe) so, i’m in. i’m eblip all the way, baby! and now i’m the director of the centre for evidence based library and information practice here at the university library. the centre idea surfaced almost 2 years ago and came to fruition in december 2012.   image of the comfortable seating in the newly furnished centre for evidence based library and information practice it was then that the university council approved the creation of c-eblip and i began to shop for office furniture for the centre space. i also began gathering interested colleagues to be centre members and together we will explore research and evidence based practice.   image in the style of “keep calm and carry on”: demand evidence and think critically the u of s encourages the establishment of centres to enhance the academic interests of the university and its faculty, and to meet the needs of the community at large. centres are intended to strengthen, coordinate or facilitate scholarly activities, and to offer new areas of activity consistent with the university's strategic direction.   image of c-eblip’s goals, also found here http://library.usask.ca/ceblip/about/index.php   the centre consists of three components that are interrelated: supporting librarians as researchers, promoting evidence based practice, and overseeing established activities relating to the university library’s culture of research.   personal nature photography, dandelions and while the bare bones of the centre are in place, the exciting part will be building the centre into something that meets the needs of the librarians and archivists at the university library both in terms of research and of evidence based practice. and it will be built with the active participation of centre members.   personal nature photography, geese (i was running out of slide ideas!) and if i can reach into the future for a minute here, i also see the centre as having the potential to grow into something that transcends the bounds of the university library. i see the centre being a hub or a connector for eblip across canada and possibly internationally.   yet another photography of nature but i don’t want to get too far ahead of things. i’ve only been the director for 2 weeks. still, i see lots of potential for the centre. and there’s lots of exciting things ahead for eblip, too.   image, evidence to the rescue there’s eblip8 to think about! and more great eblip journals issues. more collaborations and connections to be made amongst librarian colleagues. more ways to use evidence, and more definitions of evidence, as we heard from denise k. in her opening keynote.   image of christopher walken from saturday night live: “i got a fever. and the only prescription that can cure it is more cowbell!” so the eblip fever continues for me. it’s exciting to belong to a tribe who believes in a way of practicing librarianship that makes sense. it’s exciting to belong to a movement that continues to make great strides and continues to inspire librarians to not only look to the research, but to conduct research.   ryan gosling “hey girl” meme image, “hey girl, it’s a good thing i have my library card...’cause i’m checkin’ you out!” having a network of eblip friends and colleagues around the world is inspiring as well. i just have to look to these people to see that the possibilities are endless when it comes to eblip and being a librarian practitioner-researcher.   image of a diverse array of fruit and finally, being here with all of you for the past three days has really filled up the inspiration reservoir. seeing a conference come together and meeting such a great group of people will continue to inspire me for a long time to come.     library intelligence: developing a culture of evidence-based decision making karim tharani, it librarian university library, university of saskatchewan, canada   with the recent establishment of the centre for evidence based library and information practice (c-eblip) at the university od saskatchewan library, i think it is very timely for me, and perhaps also for other members of the centre, to start thinking about fresh, innovative and practical ideas to contribute to the research agenda and activities of the centre. i would like to talk about one such idea, which is the concept of library intelligence.   i define library intelligence as the art and science of bringing people, processes, and tools together to identify, obtain, and synthesize data in a systematic and sustainable manner, to assist decision makers in gaining insights into the current state of library operations. the concept as well as the term library intelligence is inspired by the well-established practices of business intelligence or bi in the commercial sector. bi is often defined as solutions that assist decision makers in understanding the current state of their organization. of course there are many other terms associated with bi such as data warehousing, decision support systems, key performance indicators or kpis, etc.   in essence, library intelligence or li is nothing more than the application of business intelligence practices in libraries. and in this sense, the concept of library intelligence is already gaining some traction. “the academic library community faces some tough decisions with regard to business intelligence. first, this is not an assessment project, but a matter of technical and staff infrastructure, on the level of our commitments to ils technology and similar it supported functions.”    in fact in 2011, i had the opportunity to present a paper at the electronic resources & libraries (er&l) conference with two of my colleagues, charlene sorensen and craig harkema. in this paper we discussed a prototype that we built using some of the bi techniques to manage electronic resources. one of our motivations in building the prototype was to promote decision making, which was evidence based rather than vendor driven. given the reality of distributed decision making in libraries, we also wanted to use bi to help enhance our culture of collaboration and accountability as custodians of electronic resources.   why should libraries even bother about library intelligence? from an operational point of view, libraries generate and gather a fair amount of data. but how much of this data is analyzed or used to inform our decision making on a regular basis? if data is important enough to gather, why is not important enough for us to analyze and utilize? library intelligence provides a way for us to gather, organize, and share information from our existing systems in ways that can help us gain insights for better decision making and planning. in my opinion, library intelligence is a worthwhile capability for libraries to build and strengthen to promote informed and insightful decisions based on evidence.   so how do we go about building library intelligence capability? well, having a structure around any new endeavor is always helpful. one such structure is the transformational life cycle (tlc) framework (zucca, 2013), which helps organization develop new capabilities from a life cycle perspective. it takes into account the various progressive phases of a new capability such as envisioning, defining, designing, developing and deploying.    the tlc framework also defines critical high-level process areas such as capability development, ownership building, and program stewardship that collectively enable organizations to achieve maturity and expertise in any given capability. the final aspect of the tlc framework, and perhaps the most tangible and operational one, defines the key enablers of any organization endeavoring to build new capabilities or expertise. these enablers include people, process, technology and infrastructure that an organization uses to fulfil its mission.   with the establishment of c-eblip at the university of saskatchewan library, the eblip community is fortunate to have the physical and intellectual space to entertain new ideas and ways to implement these ideas. it’s a place for all of us to engage, enlighten and explore!   how i’ve been inspired by eblip ashley ireland, director of user and instruction services murray state university libraries, murray, kentucky, united states of america   hello, my name is ashley ireland. i am director of user and instruction services at murray state university libraries in murray, kentucky.    this is my lightning strike presentation about how i have been inspired by evidence based methods in libraries, in which i have subtitled “how i learned to stop worrying and love the things i can quantify.”   the truth is, i wouldn’t really say i’ve been “inspired” by evidence based methods. inspired sort of implies butterflies and rainbows, right? it implies that we’re on the road to fix the problem, and that now we have the energy and enthusiasm to get there.   evidence based practice, though, sheds light on the nooks and crannies of librarianship that aren’t necessarily pretty and haven’t always been up for discussion. it says, “yeah, but why?” and “prove it” to our long-standing assumptions about philosophy and practice.    the truth is, not everything in librarianship is an upward trend, and we have this innate fear that exposing any of that would mean the imminent end to our profession. door counts dropping?  use of print collections slowing? don’t tell me we’ve seen fewer questions at our ref desk!   ebp allows us to say, “ok. if we try this, then what happens?” it is the application of the scientific method on a profession that has occasionally, in the past, been driven by habits and ritual. transitioning to an evidence based practice means that we don’t rest our laurels regarding what has always been done and why.    i think that ebp allows us to move forward within the realm of uncertainty surrounding our profession. it allows us to experiment without fear because it gives us the tools to articulate our impact, from as small as a dollar spent to lasting impacts about how time spent with students results in increased retention in higher education institutions.    some might say that ebp reduces our profession to some overly simplistic cycle, or even forces us to quantify what shouldn’t be quantified. there might be some truth there, if you only think that ebp should be used to measure the things we already recognize as being true.    that perspective is a shame, because evidence based practice also gives us, hold on to your seats, people, a place to fail. safely. this is the scientific method, right? we don’t necessarily have to find correlation and causation. but if we find that there isn’t a relationship between things we’ve examined, then we know more now than we did before. that is progress.     i have high expectations for this profession, and as i hope to be in it as few as 30 years from now, it better hold strong. i know that i’m preaching to the choir, but i say we start demanding the evidence and reducing the fear. (all different types of evidence, of course, in support of the opening keynote).   let’s add evidence based practice to our toolbox of other things that scare the crap out of librarians, use these things to show and test and experiment our way to a newly defined librarianship.    i guess you could say that i have been inspired to challenge the status quo. for instance, ebp has given me the foundation i need to build a giant dataset of library data simply to “see what we find.”   evidence based librarianship, despite its label that is somewhat boring and to some, frightening, is about curiosity. and libraries are about curiosity. as far as i’m concerned, it’s a perfect fit.     how i’ve been inspired by eblip gillian nowlan, liaison librarian dr. john archer library university of regina, canada   i am incredibly excited to be here at eblip7 and just thrilled to be able to share my own research at the poster session and hear about the amazing research that everyone is engaged in. so, i wanted to take the next five minutes to tell you why evidence based library and information practice is inspiring to me.   i think evidence based library and information practice and research is a lot about muddling through, not knowing what to expect or find and working your way through it and i think that’s part of what makes it so exciting to work with.   i’m kind of a muddler. i do enjoy jumping into things and trying to find my way through and learning through trial and error and discovery. so evidence based library and information practice is really a nice fit for me and the way i conduct research projects i engage in.   evidence based library and information practice is based on the very through selection and analysis of literature or original research and it’s really all about being engaged in what is going on around you and being aware of the different possibilities as you work through each step.   the opportunity to engage in what others are doing and reading about their perspectives on different topics and ideas. finding out what they are doing, why, and how really helps me guide my research and how i approach each project.   i love reading articles and getting those “ah-ha” moments, when i realize my thoughts are either right on track or that i’m way off and have to rethink my approach. although it can be frustrating at times it’s certainly eye opening and it can spark new ideas and ways of thinking about how you conduct your research.   i think a huge part of evidence based library and information practice is about exploring the world around you. find out how other people and places are doing things and how that fits into your environment.   evidence based practice is all about thinking critically. critical thinking and critical reflection are skills that need to be practiced and developed and this type of inquiry helps to solidify these important skills for life.   evidence based library and information practice means asking questions. questions really drive the eblip process. it’s what moves us through the process and challenges us to think about things differently and from different perspective.   it’s one of the best ways to find out more about a topic or research area. i think this kind of fits well with the muddling part, where you can just jump into the unknown and greatly increase your knowledge in an unknown area.   asking questions can lead to different ideas or ways of looking at things and it can certainly sprout new projects and connections with either individuals or topic areas that you find interesting. there are lots of networking opportunities when you participate in this research practice.   as the name suggests, evidence based library and information practice is all about finding evidence to provide the best information for your current practice. evidence based practice has been incredibly helpful to me when pitching a new idea or changing the way we do things at the library.   one of my first experiences using evidence based library practice was during my first year as an academic librarian when i set out to collect data from the university community in order to build a mobile app for the library.   using the evidence based approach i collected a large amount of data and assessed, evaluated, and compared it to current literature.   the results of the mobile applications project were incredibly helpful and the process led to an efficient mobile application design that provided the university community with what they needed.   even if you are working alone you are still interacting with other individuals through their research and making important connections that you can draw on later as the project progresses or as you advance in your research.   it provides the opportunity to collaborate and network with others who are interested in the same materials as you and to find a network of people that have similar interests and are building in areas that you are working on.   working in this type of environment can really be inspiring and you can develop new skills that can be used in your future projects, research, and work environment.   i really hope you have a wonderful conference and go sprout some new ideas and skills and get inspired by evidence based library and information practice. thank you for your attention.   eager and enthusiastic enroller tasha maddison, science liaison librarian university library, university of saskatchewan, canada   i am quick to agree. i am known as someone who always volunteers; sometimes with few details about what i am volunteering for.  i am happy to be that person, but recently i had cause to seek some further information before continuing with a task. this experience inspired my “lightning strike presentation” at the eblip7 international conference this past summer in saskatoon.    when initially asked to participate, i welcomed the challenge of a 5-minute presentation utilizing only 20 slides. really the topic and the fact that i was presenting at an eblip conference is somewhat ironic, as i wholeheartedly jumped in and got right to work, thinking that i had all the details i needed. the topic seemed, i had assumed, open to multiple interpretations and i was keen to create something unique. alas, when i handed in the first draft of my presentation i immediately discovered that i had not followed all of the instructions.   and what was the critical detail i had missed? all the slides had to be equally timed and be set up to automatically forward. sadly, there was no way that i could make my initial presentation work as i had rewritten “the twelve days of christmas” to reflect topics of discussion from the conference itself. as we all know, the song gets longer with each verse.   my next idea was to shape the lesson i had learned into a tale of the princess penelope who was known throughout the land for making disastrous assumptions. along the way she meets a cast of characters, including the viscountess virginia (does that name sound familiar?) who help her to see the error in her ways. the moral of this tale: examine the evidence, expand your knowledge and inform your opinions, yet remain thoroughly enthusiastic about all complex conundrums that come your way.   in retrospect, my initial mistake, led to a learning opportunity and ultimately a more interesting and entertaining presentation.  if you are interested in reading a transcript from my presentation entitled the ill conceived supposition or learning more about my experience preparing a lightning strike presentation, please visit my blog at http://whatilearnedinlibraryschooltoday.blogspot.ca/search/label/eblip.      references   zucca, j. (2013). business intelligence infrastructure for academic libraries. evidence based library and information practice, 8(2), 172-182. ebl 101   research methods: systematic reviews   virginia wilson director, centre for evidence based library and information practice (c-eblip) university library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   received: 12 aug. 2013 accepted: 17 aug. 2013      2013 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     a systematic review is not merely a literature review. while we can be systematic in preparing a literature review, a systematic review is a research method used to address a specific research question. systematic reviews “present a comprehensive summary of research based knowledge that can aid both practitioners and policy makers in decision making” (brettle, 2009, p. 43). meta-analysis, “the use of statistical methods to combine results of individual studies,” may or may not be conducted in a systematic review (the cochrane collaboration, 2002). systematic reviews in lis research were practically unheard of just over a decade ago. however, as denise koufogiannakis found, they “have become an important source of information because they both synthesize the existing research on a topic, as well as critically appraise it and try to draw conclusions from the total body of quality research evidence” (2012, p. 91).   the cochrane collaboration and the campbell collaboration are two organizations which oversee the conducting of systematic reviews based on specific guidelines. the cochrane collaboration focuses on the health sciences, while the campbell collaboration works in the areas of crime and justice, education, international development, and social welfare. additionally, an individual researcher or research team can undertake a systematic review to attempt to answer a research question. khan, kunz, kleijnen, andantes explain that a “review earns the adjective systematic if it is based on a clearly formulated question, identifies relevant studies, appraises their quality and summarizes the evidence by use of explicit methodology. it is the explicit and systematic approach that distinguishes systematic reviews from traditional reviews and commentaries” (2003, p. 118).   the process by which to undertake a systematic review has several steps:   1.       develop a research question   as with any research project, the development of a clear, explicit, and concise research question is the bedrock upon which the project rests. spend time thinking and planning.   2.       identify relevant work   do an extensive search for research studies. you may be looking at a particular date range. once you have exhausted the literature, decide which papers to include and exclude based on criteria that have come out of your research question. you will probably get a lot of results from your initial massive searches and the criteria will help to sort out which studies belong in your systematic review. a test run of the inclusion and exclusion criteria will show if more or is needed or changes need to be made. record the reasons for inclusion or exclusion.   3.       critically appraise the included studies   you need to look for quality and rigour. quality is difficult to define but thorough critical appraisal can help to determine if the study results are sound. critical appraisal will also help to determine whether or not meta-analysis will be a part of your systematic review.   4.       extract the data   your research question will guide the types of data you will want to extract from the included studies. create a standardized data extraction form to keep track of everything you pull from the studies. methodically review the articles in order to fill out the extraction form.   5.       synthesize and analyze the findings   “the goal of data synthesis is to go beyond simply summarizing but to also include ‘an analysis of the relationships within and between studies and an overall assessment of the robustness of the evidence’” (centre for reviews and dissemination, 2009, 48 as quoted in phelps & campbell, 2012, p. 13).   the final and best thing i can pass along is that denise koufogiannakis along with several contributors put together a comprehensive list of all the systematic reviews undertaken in lis to date. the lis systematic reviews wiki includes the reviews listed alphabetically by author and classified by topic. there is also a nice bibliography where you can find much more information on the systematic review.     references   brettle, a. (2009). systematic reviews and evidence based library and information practice. evidence based library and information practice, 4(1): 43-50.   khan, k.s., kunz, r., kleijnen, j., & antes, g. (2003). five steps to conducting a systematic review. journal of the royal society of medicine, 96: 118-121.   koufogiannakis, d. (2012). the lis systematic review wiki. retrieved 4 sept. 2013 from http://lis-systematic-reviews.wikispaces.com/welcome   koufogiannakis, d. (2012). the state of systematic reviews in library and information studies. evidence based library and information practice, 7(2): 91-95.   meta-analysis. (2002). in the cochrane collaboration open learning material. retrieved 4 sept. 2013 from http://www.cochrane-net.org/openlearning/html/mod12-2.htm      phelps, s. f., and campbell, n. (2012). systematic reviews in theory and practice for library and information studies. library & information research, 36(112): 6-15. evidence based library and information practice evidence based library and information practice 2010, 5.4 53 evidence based library and information practice article personal growth, habits and understanding: pleasure reading among first-year university students melanie parlette library technician conestoga college, kitchener ontario, canada email: m.parlette@dal.ca vivian howard assistant professor school of information management, dalhousie university, canada email: vivian.howard@dal.ca received: 9 june 2010 accepted: 23 oct. 2010 2010 parlette and howard. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/bync-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objectives – this study examines the reading habits and experiences of first-year undergraduate students at dalhousie university and the university of king’s college in halifax, nova scotia, canada. methods – first-year undergraduate university students (aged 18 to 20) were recruited to take part in focus group discussions and responses were analysed to examine the following topics: (1) the role of reading in their lives, both academic and personal; (2) the development of reading habits from childhood; (3) reading engagement strategies; and (4) selection strategies. results – this study suggests that reading for pleasure is a well-established habit amongst many first-year undergraduate students. first-year undergraduates primarily read for pleasure in order to relax but also recognize that pleasure reading can play a positive role in their academic performance, enhancing their range of background knowledge as well as their active vocabulary. mailto:m.parlette@dal.ca� mailto:vivian.howard@dal.ca� evidence based library and information practice 2010, 5.4 54 conclusions – the conclusions of this research provide recommendations for librarians and university administration to engage students and increase rates of retention in postsecondary institutions. in particular, recommendations related to the importance of pleasure reading collections, campus reading programs, book clubs, readers’ advisory services and quiet and comfortable reading areas in academic libraries are provided. introduction first-year undergraduate university students (aged 18 to 20) are at a pivotal moment in their lives. they are often moving away from home for the first time and may find university life difficult, experiencing both social and academic challenges (kantanis, 2000). evidence suggests that when first-year university students read for pleasure, they are taking part in an activity that allows them to continue to develop their literacy skills and to refine their sense of self (hughes-hassell & rodge, 2007; jarvis, 2003; salter & brooke 2003). positive experiences with pleasure reading may assist in the transition to adulthood during the university experience and may even enhance student retention. in this study, the pleasure reading habits of firstyear university students are explored to answer the following research questions: 1. how do these young adult readers see themselves as readers? what is the importance of reading, both in their personal and academic lives? 2. how has their reading habit developed since childhood? 3. how do they engage with what they read? 4. what do they read? how do they select their reading material? on a practical level this study generates information useful for informing the development of a variety of strategies to encourage the pleasure reading habit amongst undergraduate students including the establishment of pleasure reading collections, readers’ advisory services, and campus book clubs. literature review pleasure reading has been the focus of several national studies in the first decade of the 21st century. the national endowment for the arts (nea) described the state of pleasure reading in the united states as “an imminent cultural crisis” in its 2004 report, reading at risk, which found a 10% decline in the number of adult readers in the past twenty years. this report also found that reading had declined for those in the 18-24 year old age group more precipitously than for other age categories. in contrast, the reading and buying books for pleasure survey (createc, 2005) in canada indicated that the reading habits of young people, aged 16 to 24, were not experiencing a similar decline. the canadian study indicated that 84% of respondents between the ages of 16 and 24 had purchased a book to read for pleasure within the last 12 months and that reading habits for this age group remained largely unchanged over the past two decades. pleasure reading and postsecondary education college and university attendance is an important step for many young people as they come of age and become independent adults. in a 2004 report conducted by statistics canada and human resources and skills development canada, lambert, zeman, allen and bussiere observe that canada has the highest level of per capita post-secondary enrolment in the world (p. 6). however, the experience of moving from high school to post-secondary education can be challenging, and shaienks and gluszynski (2007) emphasize that maintaining a “positive attitude and [feeling a] sense of belonging during [the] first-year of postsecondary education is critical in the decision to continue or drop out” (p.4). postsecondary institutions evidence based library and information practice 2010, 5.4 55 are developing new strategies to engage students in a variety of educationally purposeful activities in an effort to attain higher levels of achievement and retention. the national survey of student engagement (2009) cites a positive correlation between student engagement in such activities and whether or not students return to a second year at a postsecondary institution (p.7). the most risky time for most students is between their first and second year of study, when, statistically, most dropouts occur, so engagement activities are often focused on first-year students. the academic benefits of pleasure reading are emphasized by babbitt bray et al. (2004), who argue that the more college students read for pleasure the more likely they are to develop a strong vocabulary and cultural literacy. clark and rumbold (2006) state that reading for pleasure is positively linked with reading achievement and writing capability as well as comprehension and grammar. they emphasize that a positive attitude towards reading also relates to stronger standardized testing scores. ross (1999) observes that pleasure reading can provide valuable insights into personal relationships and experiences and can have a substantial impact on readers’ understanding of themselves and the world. a similar argument is made by richardson and eccles (2007), who observe: the reading that children and adolescents engage in for its own sake may also provide ‘self-generated learning opportunities’ that in turn serve to nurture and support educational aspirations, achievement motivation, occupational choices, as well as ways of understanding oneself and others (p.342). the national survey of student engagement (2009) finds that the vast majority of american college students (74-80%) read less than four books for pleasure during the academic year (p.36). this study suggests that homework and assigned readings are not entirely responsible for low levels of pleasure reading among this group. falk-ross (2001) suggests that a significant number of first-year students at colleges and universities in the united states enter postsecondary education with inadequate reading skills and are underprepared to meet the requirements of their coursework. garfield (2008) explores strategies to support and encourage student reading in a uk university. she advocates the development of a university reading strategy to aid and improve student learning in the university environment, and concludes that those who attend university are often not prepared for academic study and that “deep” reading must be “nurtured, encouraged, and enabled” (p.13) in academia and that technology, programming and services should be used to assist in this process. elliott (2007) and trott and elliot (2009) argue for the reintegration of readers’ services in the academic library as one option to address the declining literacy rates of post-secondary students. pleasure reading collections and reader’s advisory in academic libraries pleasure reading collections and readers’ advisory are two methods through which academic institutions might attempt to engage students with pleasure reading; however, few academic institutions use either of these strategies. smith and young (2008) recommend such services to broaden the relationship between the library and the larger community of the institution, including faculty, staff and students. o’english et al. (2006) and cubbage (2010) explore the value of graphic novels in academic library recreational reading collections, suggesting that graphic novels can provide support to the academic curriculum while circulating as a part of a pleasure reading collection. in this way, academic libraries can promote literacy and provide a rewarding pleasure reading experience, while also creating collections that may be used in the classroom. sanders (2009) examines the popular reading collections at three universities in the united states and discovers that undergraduate students use these collections actively for their recreational reading. evidence based library and information practice 2010, 5.4 56 reading engagement in groups salter and brooke (2007) maintain that programs and activities that promote pleasure reading are important to support reader development among college students. reading for pleasure in a group, as a member of a book club or through participating in a common reading program, is another strategy to encourage reading for pleasure among university students. ferguson (2006) and boff et al. (2005) suggest that common reading experiences provide an opportunity to promote reading habits that will follow participants throughout their lives. they further suggest that such experiences may bring together students who may not have interacted with each other before, perhaps because they are from different disciplines, and furthermore, through participating, students will exchange ideas, thus improving their knowledge and critical thinking skills. boff et al. (2005) thus conclude that common reading programs “build community among students, [and] also build bridges from the library to the rest of the university” (p.282). reading groups can also be useful in promoting the discussion of books and reading amongst those who prefer sharing their reading with people outside of their friendship circle (howard, 2008). reading and academic major there is limited research on the relationship between pleasure reading and a student’s choice of subject major in college or university. chen (2007) examined data from two national surveys of the pleasure reading habits of university students in taiwan, one addressing first-year students and the other fourth year students (chen, 2007). the results of these surveys show that “humanities majors were reading more extracurricular material than natural science and education majors, but not necessarily more than math/computer science and engineering majors” (p.651). karim and hasan (2006) examined the academic and pleasure reading habits and attitudes of students at the international islamic university malaysia. they studied students in both it-based and arts-based programs and found that it students were likely to “rely more on the web site as their reading materials and resources” (p.296). this study also found that students in the arts-based program used the library at a much greater rate than their counterparts in it-based programs (karim & hasan, 2006, p.295). readers in a multimedia world wilson and casey (2007) attribute declines in reading among secondary students to video and computer-generated games. lankshear and knobel (2007) state that research in education should include digital literacy, such as gaming. smith and young (2008) suggest that pleasure reading is increasingly complex in a world of blogs, webrings, fan sites, and wikis: while students may be reading a great deal, their reading may be “short, unfiltered, unedited bites” (p.521) rather than sustained and immersive. mokhtari, reichard and gardner (2009) examined the impact of the internet and television on the reading habits of college students at a university in the united states and found that while watching television was a common activity among the students studied, the internet and recreational reading were described as more enjoyable (mokhtari, reichard, & gardner, 2009, p.618). they also found that the students do not feel that television viewing affects time spent on the internet but that they do feel that it decreases the amount of time they spend reading both academically and recreationally (p.618). the national endowment for the arts (2007) report to read or not to read suggests that on average television consumes approximately half of daily leisure time among americans ages 15 and older (p.38). in 2006, pleasure reading among 15 to 24-yearolds in the united states accounted for 2.6% of their leisure time (p. 40). this age group is also likely to multi-task while reading, with 35% of 7th to 12th graders using other media including television, music, or a computer while reading (national endowment for the arts, 2007, p. 44). thus, while there has recently been a renewed evidence based library and information practice 2010, 5.4 57 interest in research into pleasure reading by various groups, few studies have specifically studied undergraduate university students even though the recreational reading habits of this age group have been identified by two large scale studies by the national endowment for the arts (2004, 2007) as undergoing a precipitous decline. likewise, few previous studies have analyzed undergraduate students’ perceptions of group reading activities, the potential of pleasure reading to enhance student engagement and retention, or the appeal of pleasure reading collections within the academic library. this study addresses this gap in the research literature by examining the role of pleasure reading in the lives of first-year undergraduate students. methods using focus group methodology, this study examined five groups of firstyear undergraduate students at dalhousie university and the university of king’s college in halifax, nova scotia. the participants in the focus groups were all 18 to 20 years of age and were all in their first-year of postsecondary education. focus groups allowed for the gathering of qualitative data in a setting that promoted discussion among multiple participants. focus groups were chosen to allow for more in-depth discussion than could have been achieved through a questionnaire. gibbs (1997) states that through focus groups, attitudes, feelings, and experiences may be revealed through the interaction in a group setting as opposed to other methods such as questionnaires and one-to-one interviewing. however, gibbs (1997) also makes clear that focus groups have certain limitations: the researcher may have less control over the data than in quantitative research and not all participants may participate equally. furthermore, it is always possible that participants will fail to express their personal opinions and will conform to a popular opinion (p. 4). the primary research instruments consisted of a set of focus group questions, developed from the research questions (see appendix). this standard set of questions was used to facilitate comparison between groups. during discussions, deviation from the questions was allowed in order to pursue topics that spontaneously arose through the discussions and to clarify ideas. recruitment of participants once ethics approval was obtained from the dalhousie university social sciences and humanities research ethics board, first-year undergraduate students were recruited through two different methods. the first approach involved soliciting participants directly through class visits to selected firstyear undergraduate classes with enrolments of over 100 students in a variety of faculties. during class visits, those that were interested in the study provided their name and email address for follow up contact. secondly, students were recruited through posters placed in high-traffic areas on the dalhousie university and university of king’s college campuses. recruitment posters were also placed in libraries, classroom buildings, and the student union building. students were encouraged to participate with the understanding that the discussion would only take 60 – 90 minutes of their time. while they were not compensated monetarily, pizza and refreshments were provided and those that participated were entered in a draw for a $50 gift certificate to a local shopping mall. readers of all types were encouraged to participate. sixty-four students initially indicated an interest in the study. all 64 individuals were contacted and 26 ultimately participated in focus groups. five focus groups were conducted and two to eight individuals participated in each. focus group discussions were digitally recorded, transcribed, and coded using qsr nvivo qualitative analysis software. using a grounded theory approach, transcripts were independently analyzed by both researchers and emerging themes were identified. to assure confidentiality, evidence based library and information practice 2010, 5.4 58 pseudonyms were used in place of participants’ real names in the transcripts. neither sentence structure nor grammar was corrected in the transcription process. thirteen participants were from rural communities and thirteen were from urban communities.1 there were slightly more male participants (14) than female (12). twelve students were registered in the faculty of arts and social sciences, an equal number were registered in the faculty of science, and two were registered in the faculty of management. results the role of reading (research question 1) in each focus group the participants were asked if pleasure reading is important in their academic and personal lives. all except one member of the focus groups indicated that they were enthusiastic and active readers who read for pleasure primarily for relaxation and enjoyment. even the one self-identified reluctant reader mentioned that she was reading a book for a class and that she was enjoying it and discussing it with her friends. however, it must be taken into account that those who participated in the focus groups were self-selected and therefore might be more likely to have been enthusiastic about reading because they volunteered to discuss their reading habits and experiences. some participants found it almost impossible to separate their academic reading from their pleasure reading experiences. several participants attributed reading to helping them become more intelligent and improving their vocabulary. four participants commented on the fact that reading allowed them to develop intellectually and gain perspective on different topics. the possibility of learning new things and broadening perspectives was identified as a strong 1 urban communities were defined as census metropolitan areas (cma) as determined by statistics canada. http://www12.statcan.ca/english/census01/pro ducts/reference/dict/geo009.htm motivation for reading. for example, ashley expressed the opinion that “[reading] expands your world. it exposes you to places you wouldn’t normally be exposed to in everyday life.” several of the students in the focus groups described how pleasure reading provided them with a more thorough understanding of course material; as indicated in the following quote: isabella: i’d say [when you read] you just expand your abilities in general. in reading you develop better vocabulary for writing and if you read like that’s more for pleasure reading but even if you read you know information, it gives you greater knowledge, like worldly things….if you don’t read then you can’t have the base in order to build on. you know what i mean? you can go to lectures and listen but if you don’t really understand what they are talking about from the beginning, then you don’t have that basic knowledge. the development and encouragement of reading (research question 2) during the focus group sessions participants were encouraged to recall their first reading experiences. fifteen of the individuals cited the positive influence of their parents and their role in creating positive reading habits and experiences. madison, for example, stated that “my room just always had books and that was my parents naturally said “you should go read” so i’ve just always been around it.” many of the participants recalled in great detail how they learned to read and the pride they felt in this accomplishment: jacob: the first thing i remember was i started with a calvin and hobbes comic and i would always flip through it because i liked the pictures and the artwork and everything like that. but i evidence based library and information practice 2010, 5.4 59 could never quite read the words in the bubbles and things like that and it was really frustrating till one day i was reading a panel and i was able to discern the words ‘yes’ and ‘no.’ and then it started; words just seemed to fit together after that and they started making sense. that’s about the earliest thing i can remember about reading. participants also discussed how their pleasure reading habits had changed since entering university. several students noted that they had an increased understanding of what they were reading and that their preferred genres had changed. these two factors may be interconnected, as jacob explains: jacob: when i got older i actually started reading a lot more because when i was younger i would read like simple stuff like comic books and things like that but once i got into high school i started becoming familiar with more words and increased my vocabulary i found i could read more fantasy, more complex fantasy novels and things like that, so i started reading more because i was like really interested. andrew explains that his reading habits have changed significantly since starting university and that he now only reads serious, high quality reading material because he finds it most rewarding and productive. his academic reading and his pleasure reading now intersect: andrew: i think university has kind of like shoved this idea on me that my time is very valuable and when i choose a book i kind of discriminate against anything i think is trash. if i read something i want to get something out of it. like i think what you guys read for class i read for pleasure so it’s like i’m the opposite situation as you. reading engagement (research question 3) the participants were asked to describe whether any books they had read for pleasure had been particularly influential on their lives. sixty-five percent of participants felt that pleasure reading had helped them to better understand themselves and the world around them and many mentioned specific titles that had been highly influential. in the following quotes, participants describe their feelings of personal identification with their reading and the way in which their reading has given them an unexpected perspective on their own lives: grace: well, i read hamlet last year and i can’t believe i’m saying a piece of classical fiction changed my life but it did.… in this case it was just two words; it was just one of hamlet’s speeches. just the two words “let be.” i read that and something clicked. i don’t understand, i don’t know if it was the intention for that to click with every person; it was just the way i was when i was reading that it’s really interesting that something meant to be read a certain way can affect people so many different ways. brad: ummm i read david copperfield by charles dickens and he had a really crappy life and yet he still maintains … this positive look on life that things are going to look up eventually and things eventually do go his way. it just kinda gives you that [thought], you know, when things all fall to pieces to just keep moving on. it’s just that whole theme. that makes you think “my life isn’t so bad.” i mean things like my cell phone died and my girlfriend left me. this guy just lost his house. evidence based library and information practice 2010, 5.4 60 the participants of the focus groups described various “epiphany moments” in which reading a particular text had caused them to re-think how they understood certain issues. much like grace and brad, madison discusses the effect of the joy luck club and how it changed her view of mother/child relationships: madison: i remember being really profoundly moved by the joy luck club. and having such a deep gratitude for my mother after that. um, just because you realize just how much mothers will do for their children. it’s, it was, profound. the books that some of the participants described as influencing them were not only those that they were reading solely for pleasure. samantha mentioned that in fyp [foundation year program] some of the books were helping her better understand human nature. madison also highlighted how she related to both sense and sensibility, a book she read purely for pleasure, and another that she had read for fyp: samantha: not all of [our reading] in fyp [foundation year program]. but a lot of it. a lot of what we learn about is human nature. madison: sense and sensibility my best friend and i are those sisters. exactly. it was really funny reading it. i read it and i told my friend and then she read it and she was like oh my goodness. i am also st. augustine as a teenager. it’s kinda scary. several participants felt that their academic reading had given them opportunities for personal development. richardson and eccles (2007) explain that reading can encourage adolescents to “think about and contemplate their possible selves – who they want to be, who they are, and who they fear being” (p.354). these important connections encourage the participants to move outside the text and relate to other elements in their own lives, creating a strong sense of engagement with what they are reading. ross (1999) argues that readers are actively engaged in constructing their own unique meaning from texts based on their own experiences and that “the affective dimension is involved throughout the process, from choosing a book according to mood to valuing a book for its emotional support in providing confirmation, courage or self-acceptance” (p.797). the participants in the current study identified with books because of their prior experiences and created their own unique meaning based on these experiences. they also felt that their reading helped them to understand their own relationships more clearly. two-thirds of the participants stated that they frequently discussed both their pleasure and academic reading with friends, classmates or family members. somewhat surprisingly, however, no participants indicated that they had taken part in dal reads!, a campus-wide reading program initiated at dalhousie university in the summer of 2009 and targeted at first-year undergraduates. the dal reads! initiative requested that students read the book of negroes by lawrence hill prior to beginning university; book discussion groups based on the novel were offered in the fall and some first-year classes integrated the novel or related material in lectures. this lack of participation appears connected to students’ desire to be independent readers and their dislike of being “told” what to read for pleasure, as the following quotes illustrate: sarah: if someone tells me to read a book, i just don’t want to. justin: i feel pressured and i can’t enjoy it. the participants frequently discussed the importance of “discovering” a book and their evidence based library and information practice 2010, 5.4 61 desire to be independent in their reading choices. in her study of young adolescent readers, howard (2008) describes three broad types of avid readers: social (those who prefer to read the same books as their close friends), detached (those who share their reading with non-friendship-based groups such as book clubs) and solitary (those who do not discuss their reading with their peers). both detached and solitary readers “tend to view reading recommendations from friends as unwanted pressure … [they] see themselves as trendsetters or opinion leaders and enjoy dispensing reading recommendations to others” (p.112). in this study, participants victoria and carter expressed very similar views as both emphasized how they felt that they would rather “discover” a book independently than follow a recommendation provided to them: victoria: i know how you guys were saying you don’t like reading a book that someone was like “here, read it!” i don’t mind if someone is like “hey you should read it sometime” and i might randomly pick it up. what i don’t like is when books are so huge like harry potter. i like them but i don’ t love them because i think that if i had stumbled across it on my own i would have really liked them but because there is so much expectation behind it it’s almost like pressure like you’re forced to enjoy it. i’m always thinking “do i like this right now?” i don’t know [the pressure] kinda ruins it. carter: it could be that psychological thing where we want to find that diamond in the rough. i would assume that people who like reading are pretty independent to begin with. these individuals indicate that they are less likely to be interested in books that are deemed to popular to the masses or that others recommend to them. they identify what they read with their sense of individuality. this finding implies that programs such as dal reads! in which students are told to read a popular and acclaimed book might not be appealing to students such as carter and victoria or other “detached avid readers”, as it goes against their desire to be unique and independent in their reading choices. despite the fact that they did not participate in dal reads, when they were asked if they had previously participated in a book club, 46% of participants indicated that they had done so either through their school or public library and had found it a positive experience: aaron: i find that in a book club, and everyone’s talking about it and discussing it and your opinion and your knowledge grows and expands out. grace: and also everyone, depending on where you are in your life and what kind of lives they have, will read a book differently. so it’s really interesting to get more than one mind together and just hear their different readings. they’ll pick up on things and they will have read more strongly things that you glossed over. just because of a difference in life experience. so it’s really interesting to then be able to reflect on the book [together]. selection and reading (research question 4) in the focus groups the participants discussed what they were currently reading; 50% were reading general fiction, 21% were reading various types of nonfiction (biography, autobiography, science, business), and 15% were reading science fiction/fantasy. the evidence based library and information practice 2010, 5.4 62 remaining participants were reading other types of genre fiction. most participants obtained their reading material from commercial sources including chain, independent, and campus bookstores. the next most popular response was that they had received reading material from a friend or relative, and the least common response was that they had borrowed reading material from a library. this is consistent with koch and kendall (2003), who studied 16 to 19yearolds in the united kingdom and found that few individuals had borrowed fiction books from the library and that the majority were purchasing them or borrowing fiction books from friends. participants in the current study explained that they often found it difficult to find appealing reading material at the public library and didn’t have a clear strategy for selecting books to read for pleasure. eighty-eight percent of participants indicated that they spend time reading online. figure 1 describes their online reading habits and reveals participants do a wide variety of online reading with blogs, facebook, web comics and anecdotal humour sites appearing to be the most popular. interestingly, no participants indicated that they owned an ebook reader or stated that they read books online. participants were asked to describe their impressions of the academic library. they indicated that they were impressed with the research resources, but did not feel that the academic library was a place to find recreational reading materials. however, most participants indicated that while they remotely accessed the electronic databases provided by the academic library, they had not yet visited the library to borrow books, and when they went to the library it was typically for other activities such as working on group projects or studying. the participants were asked how they would feel about a separate pleasure reading collection within the academic library. a large portion of participants, 83% of those who responded to the question, indicated that they would use such a collection if it existed. these participants indicated that such a collection should be browsable and organized like a large chain bookstore with book displays and genre groupings in order to make book selection easier and faster. fig. 1. online reading habits. evidence based library and information practice 2010, 5.4 63 discussion limitations and assumptions this study was restricted to first-year undergraduate students attending dalhousie university and the university of king’s college. the research goal was to gain a broad understanding of some of the recurring attitudes and behaviours regarding the pleasure reading experience and first-year undergraduate students. the findings can not necessarily be generalized to first-year undergraduate students in other geographic settings although the findings may be useful to those providing programming and services to this group. the following limitations should also be noted: • focus group participants were selfselected. thus, the students that chose to participate in focus groups discussing their pleasure reading experiences may not be representative of the general first-year student population at either dalhousie university or the university of king’s college. • because participants were selfselected, it was difficult to attract an equal number of active and non-active pleasure readers. it was also difficult to attract an equal number of participants from each discipline. • in conducting this research, it was assumed that those who participated in the focus groups answered questions honestly. however, it is possible that participants did not always express their personal views and gave responses which conformed to popular opinion. while the focus group participants might not be representative of the entire first-year undergraduate population, the habits and experiences that they described provide useful insight into how they view themselves as readers and the value of reading within their everyday lives. pleasure reading collection the individuals who participated in this focus group were overwhelmingly enthusiastic about the concept of a separate pleasure reading collection within one of the academic libraries on campus. establishing a pleasure reading collection in academic libraries is recommended as one option to encourage leisure reading among students in an academic environment. o’english et al. (2006) and cubbage (2010) specifically recommend graphic novel collections because of their popularity with university students. cubbage (2010) has also developed a pleasure reading collection which can be used as an academic resource for the study of popular culture. campus reading programs participants appeared to resist the dal reads! program because they viewed it as too prescriptive; as independent readers, they wanted to be free to choose their own reading material. this resistance to a common reading initiative might be reduced by having students take part in the selection process, therefore creating a sense of ownership over the program. campus reading programs could also offer more than one option, so that students could select from a shortlist of three to four titles reflecting a range of subjects, genres and formats. book clubs many students indicated that they had participated in and enjoyed book clubs. participation in book clubs had encouraged them to meet new people and to broaden their understanding of what they read. to encourage students’ further enjoyment and engagement with what they are reading for pleasure, book clubs and online reading communities could be organized by the university. evidence based library and information practice 2010, 5.4 64 readers’ advisory a substantial number of participants indicated that they were currently reading material that they had purchased, received as a gift or borrowed from friends, relatives, and other acquaintances. very few borrowed books from either the public or academic library to read for pleasure. several participants indicated that they rarely borrowed books to read for pleasure from the public or academic library because they did not have an effective strategy for choosing appealing reading materials and did not find library collections easy to browse. the academic library could use a variety of readers’ advisory strategies to encourage pleasure reading during university. smith and young (2008) suggest that readers’ advisory can be an opportunity to “highlight the collections [and] to broaden the relationship between the library and the university community” (p.522). by facilitating all types of reading in the library, academic librarians have the opportunity to encourage literacy and reading enjoyment. readers’ advisory services need not be limited to librarians; students should be encouraged to share recommendations with each other through online communities or bulletin boards placed in high traffic areas of the library. quiet reading area developing a quiet, comfortable reading area, perhaps near a separate pleasure reading collection, is another way to create a positive experience for university students. the creation of a comfortable and relaxing area for students to read for pleasure would encourage students to use the library for recreational reading, not just for studying or group work. conclusions this study suggests that reading for pleasure is a well-established habit amongst many firstyear undergraduate students. first-year undergraduates primarily read for pleasure in order to relax but also recognize that pleasure reading can play a positive role in their academic performance, enhancing their range of background knowledge as well as their active vocabulary. participants read books in print format although they read blogs, comics and news online. most participants indicated that they obtained their books from bookstores, borrowed them from friends or family, or received them as gifts. participants rarely borrowed books from either the public or academic library and indicated that they often had difficulty finding appealing pleasure reading materials at the public library whereas they felt that the academic library was only to be used for research resources, not for pleasure reading materials. many participants also indicated that they enjoyed discussing their pleasure reading with their peers and found such discussions enhanced their understanding of their reading and also provided them with useful suggestions for further reading. the findings of this study suggest that there is much that academic libraries could do to serve the recreational reading needs of undergraduate students. at a time when academic libraries are witnessing dramatic changes in patterns of use and are increasingly being asked to demonstrate their value and their contribution to student recruitment, retention and engagement (association of college and research libraries, 2010), this study finds that the academic library could expand its role and its visibility in the university by fostering pleasure reading in addition to academic research and information literacy skills. this could be achieved through a variety of strategies including the introduction of a pleasure reading collection, the establishment of on-campus reading groups or book clubs, and the introduction of a range of readers’ advisory services. future research this research study suggests several avenues of future research. first, longitudinal studies are needed to monitor changes in reading habits over time. how do the reading habits of first-year undergraduates change as they progress through subsequent years of study and do these changes vary by field of study? evidence based library and information practice 2010, 5.4 65 second, further research could explore the correlation between pleasure reading, academic discipline and academic achievement. third, future research could investigate the role of reading for pleasure in the lives of undergraduate students at a broader range of post-secondary institutions in other geographic locations. finally, if academic libraries do introduce any of the pleasure reading strategies outlined above, the success of these initiatives should be carefully assessed through follow-up studies. references association of college and research libraries (2010). the value of academic libraries: a comprehensive research review and report. researched by megan oakleaf. chicago association of college and research libraries. retrieved from http://www.acrl.ala.org/value/ babbitt bray, g., pascarella, e.t., pierson, c.t. (2004). postsecondary education and some dimensions of literacy development: an exploration of longitudinal evidence. reading research quarterly, 39(3), 306-330. boff, c., schroeder, r., letson, c., & gambill, j. (2005). building uncommon community with a common book: the role of librarians as collaborators and contributors to campus reading programs. research strategies, 20(4), 271-283. chen, s.y. (2007). extracurricular reading habits of college students in taiwan: findings from two national surveys. journal of adolescent & adult literacy, 50(8), 642-653. clark, c., & rumbold, k. (2006). reading for pleasure: a research overview. united kingdom: national literacy trust. retrieved from http://www.literacytrust.org.uk/resear ch/nlt_research/271_reading_for_pleas ure_a_research_overview createc. (2005). reading and buying books for pleasure: 2005 national survey. ottawa: canadian heritage. retrieved from http://dsppsd.pwgsc.gc.ca/collection/ch44-612005e.pdf cubbage, c. (2010). selection and popular culture in large academic libraries: taking the temperature of your research community. in r. g. weiner (ed.), graphic novels and comics in libraries and archives: essays on readers, research, history and cataloging (pp. 72-80). jefferson, nc: mcfarland. elliott, j. (2007). academic libraries and extracurricular reading promotion. reference & user services quarterly, 46(3), 34-43. falk-ross, f. c. (2001). toward the new literacy: changes in college students' reading comprehension strategies following reading/writing projects. journal of adolescent & adult literacy, 45(4), 272-284. ferguson, m. (2006). creating common ground: common reading and the first-year of college. peer review, summer, 8-10. garfield, d. (2008). a reading strategy for a uk university: reviewing the literature on reading, literacy and libraries, with particular regard to the he sector. journal of information literacy, 2(2), 18-31 gibbs, a. (1997). focus groups. social research update. 19(winter). retrieved from http://sru.soc.surrey.ac.uk/sru19.html howard, v. (2008). peer group influences on avid teen readers. new review of evidence based library and information practice 2010, 5.4 66 children's literature and librarianship, 14 (2), 103-119. howard, v., & jin, s. (2004). what are they reading? a survey of the reading habits and library usage patterns of teens in nova scotia. canadian journal of information & library sciences, 28(4), 25-44. hughes-hassell, s. & rodge, p. (2007). the leisure reading habits of urban adolescents. journal of adolescent & adult literacy, 51(1), 22-33. human resources development canada, canada council of ministers of education, and statistics canada. (2006). measuring up canadian results of the oecd pisa study: the performance of canada’s youth in science, reading and mathematics. ottawa, ontario: statistics canada, human resources social development canada. retrieved from http://www.statcan.gc.ca/bsolc/olccel/olc-cel?catno=81-590x&chropg=1&lang=eng jarvis, c. (2003). desirable reading: the relationship between women students’ lives and their reading practices. adult education quarterly, 53(4), 261-276. kantanis, t. (2000). the role of social transition in students’ adjustment to the firstyear of university. journal of institutional research (south-east asia), 9(1), 100-109. karim, n.s.a., & hasan, a. (2006). reading habits and attitude in the digital age: analysis of gender and academic program differences in malaysia. electronic library, 25(3), 285-298. koch, l., & kendall, m. (2003). older teenagers’ attitudes to a further education college library’s fiction collection. new review of children’s literature and librarianship, 9(1), 21-40. lambert, m., zeman, k., allen, m., & bussiere p. (2004). who pursues postsecondary education, who leaves and why: results from the youth in transition survey. ottawa, ontario: statistics canada, human resources social development canada. retrieved from http://www.statcan.gc.ca/pub/81-595m/81-595-m2004026-eng.pdf lankshear, c., & knobel, m. (2007). researching new literacies: web 2.0 practices and insider perspectives. elearning, 4(3), 224-240. mokhtari, k., reichard, c, & gardner, a. (2009). the impact of internet use and tv watching on the reading habits of college students. journal of adolescent and adult literacy, 52 (7), 609619.national endowment for the arts. (2004). reading at risk: a survey of literary reading in america. washington, dc: national endowment for the arts. retrieved from http://www.nea.gov/pub/readingatri sk.pdf national endowment for the arts. (2007). to read or not to read: a question of national consequence. washington, dc: national endowment for the arts. national survey of student engagement. (2009). assessment for improvement: tracking student engagement over time – annual results 2009. bloomington, in: indiana university centre for postsecondary research. retrieved from http://nsse.iub.edu/nsse_2009_result s/pdf/nsse_ar_2009.pdf o’english, l., gregory matthews, j., & blakesley lindsay, e. (2006). graphic novels in academic libraries: from evidence based library and information practice 2010, 5.4 67 maus to manga and beyond. journal of academic librarianship, 32(2), 173-182. peters, t. (2009). the future of reading. library journal, 134(18), 18-22. richardson, p.w. & eccles, j.s. (2007). rewards of reading: toward the development of possible selves and identities. international journal of educational research, 46(6). 341-356. ross, c. s. (1999). finding without seeking: the information encounter in the context of reading for pleasure. information processing and management, 35, 783799. salter, a. & brook, j. (2007). are we becoming an aliterate society? the demand for recreational reading among undergraduates at two universities. college & undergraduate libraries, 14(3), 27-43. sanders, m. (2009). popular reading collections in public university libraries: a survey of three southeastern states. public services quarterly, 5(3), 174-183. shaienks, d. & gluszynski t. (2007). participation in postsecondary education: graduates, continuers, and drop-outs, results from yits cycle 4. ottawa, ontario: statistics canada, culture, tourism and the centre for education statistics division. retrieved from http://www.pisa.gc.ca/pdf/81-595-59e.pdf shaienks, d.t., gluszynski, t., & bayard, j. (2008). postsecondary education – participation and dropping out: differences across university, college and other types of postsecondary institutions. ottawa, ontario: statistics canada, culture, tourism and the centre for education statistics division. retrieved from http://www.statcan.gc.ca/pub/81-595m/81-595-m2008070-eng.pdf smith, r. & young, n.j. (2008). giving pleasure its due: collection promotion and readers’ advisory in academic libraries. journal of academic librarianship, 34 (6), 520-526. trott, b. & elliott, j. (2009). barriers to extracurricular reading promotion in academic libraries. reference & user services quarterly, 48(4), 340-346. wilson, j d., & casey, l.h. (2007). understanding the recreational reading patterns of secondary students. reading improvement, 44(1), 40-49. evidence based library and information practice 2010, 5.4 68 appendix –focus group questions 1. icebreaker • what program are you in and where are you from? (removed from screening measures form) 2. what are your first memories of reading as a child? • what do you remember about learning to read? • how has your reading habit changed since childhood? • as a child, did you feel encouraged to read for pleasure? • who encouraged you to read? 3. what are you reading right now? • what do you read for pleasure? • do you read books, magazines, blogs? • are you reading this book for pleasure? • how did you choose to read this book • where did you get this book? • do you visit the public library? • tell me about your experiences going to the academic libraries on campus. • if there were a recreational reading collection on campus, would you use it? 4. tell me about your pleasure reading experiences. • when do you usually read for pleasure? • how would you describe your reading frequency? • why do you choose to read for pleasure? 5. why is reading important to you? • why is it important in your personal life? • why is it important in your academic life? 6. how do you engage with what you read? by engage, i mean how you interact, perhaps to learn more about a topic or discuss with others. • how do you explore what you have read and use this knowledge to understand the world around you? • do you re-read media that is of interest you? do you research topics / ideas that you come across as you read? • do you discuss what you read with others? if so, who? • have you ever participated in a common reading program or book club? • if yes, did you enjoy this experience? • what did you feel you gained from it? • if you have never participated in a common reading program or book club, why not? • have you had an experience where a book helped you to develop a better sense of yourself? tell me about this. • did you participate in the dal reads! initiative? why or why not? 7. what motivates you to read? • do you have goals for your own reading? • what prevents you from achieving these goals? • what would help you to achieve your reading goals? evidence based library and information practice 2010, 5.4 69 8. how do you think that your reading habits and interests have changed since you left high school? • do you see them continuing to change or stay the same? 9. how would you describe your online reading experience? • what do you read online? • do you interact with what you read online? 10. how often would you like to be able to read for pleasure? • does anything prevent you from doing so? • if something does, what prevents you from reading for pleasure? 11. is there anything else that you would like to comment on about your own personal reading experience or habits? / evidence based library and information practice / personal growth, habits and understanding: pleasure reading among first-year university students literature review pleasure reading and postsecondary education pleasure reading collections and reader’s advisory in academic libraries reading engagement in groups reading and academic major readers in a multimedia world methods recruitment of participants the role of reading (research question 1) the development and encouragement of reading (research question 2) reading engagement (research question 3) selection and reading (research question 4) pleasure reading collection campus reading programs book clubs readers’ advisory quiet reading area conclusions future research garfield, d. (2008). a reading strategy for a uk university: reviewing the literature on reading, literacy and libraries, with particular regard to the he sector. journal of information literacy, 2(2), 18-31 appendix –focus group questions evidence based library and information practice evidence based library and information practice 2010, 5.3 5 evidence based library and information practice article the impact of the acquisition of electronic medical texts on the usage of equivalent print books in an academic medical library pamela s. morgan information resources librarian, health sciences library memorial university of newfoundland st. john's, newfoundland and labrador, canada email: pmorgan@mun.ca received: 08 apr. 2010 accepted: 17 july 2010 2010 morgan. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objectives – this study examines whether acquiring a text in electronic format effects the usage of the print version of the text, focusing specifically on medical texts. studies in the literature dealt specifically with general collections and it was not clear if they were applicable to medical collections. it was also not clear if these studies should play a role in determining whether a medical library should purchase electronic texts or whether reserve collections are still needed for print texts. methods – four usage studies were conducted using data from the circulation system and the electronic vendor systems. these were 1) trends of print usage; 2) trends of electronic usage; 3) a comparison of electronic usage with print usage of the same title in the reserve collection; 4) a comparison of electronic usage with print usage of the same title in the general collection. results – in comparison to print, substantial usage is being made of electronic books. print is maintaining a level pattern of usage while electronic usage is increasing steadily. there was a noticeable difference in the usage levels of the electronic texts as regards to the package in which they are contained. usage of print texts both on reserve and in the general collection has decreased over time, however the acquisition of the electronic version of a medical title had little impact on the usage of the equivalent print version. mailto:pmorgan@mun.ca evidence based library and information practice 2010, 5.3 6 conclusion – there is a demand for medical texts in medical libraries. electronic versions can replace print versions of texts in reserve. further investigation is needed of current patterns of print collection usage, with particular emphasis on trends in reserve collection usage. introduction libraries are still confronting the issue of electronic resources and their place in the library. while most have transitioned to some electronic holdings, questions still arise as to the degree to which a library should acquire electronic resources and whether electronic versions should replace print or whether print and electronic versions should exist concurrently. remote accessibility and the ability to link from electronic bibliographic citation indexes through to full-text articles encouraged user demand for increased electronic access to the journal literature. the same cannot be said of e-books. despite the same ease of remote access and linking ability, e-books have not yet received the same degree of acceptance as ejournals, and their presence in libraries has been slower to grow. reading full-text onscreen remains an unacceptable model for most users and printing from e-books is not easily accomplished. like many libraries, memorial university of newfoundland’s health sciences library has been implementing electronic resources and establishing management guidelines. the library has switched its print subscriptions to electronic ones wherever possible and implemented a policy indicating a preference for electronic versions of a journal over print versions. but it is faced with difficult decisions regarding the development of the monograph collection, with particular reference to e-books and the reserve collection. while the library has been experimenting with e-books, the time has come to make a policy decision on whether to fully develop an e-book collection and to determine the relationship between the electronic and print monographs. as a collection that, by its general definition, receives high use, the reserve collection is an easy target for replacing the print version with an electronic version that can be accessed at all hours by multiple users. this study compares the usage trends of the print and electronic versions of the same standard medical texts in order to enhance the monograph collections policy by seeking to answer two questions: 1) should a health sciences library acquire medical e-books? 2) can the electronic versions of standard medical texts replace the print versions that are traditionally kept on reserve? literature review e-book usage studies in the literature examine e-books in general collections, with most focused on netlibrary collections. no studies were found examining specialised medical collections of e-books. few scholars read print books in their entirety, rather preferring to scan the book, read specific sections in detail, look for precise information, read the introduction and conclusion to get the general flavour, or make notes or copy specific items of interest (summerfield, 1998, p. 317-318; hughes, 2001, p. 116). levine-clark (2006) reported that only 7% of users read the entire book online, with the majority only reading a chapter (p. 14), while nicholas (2008) reported 6% reading the entire book online, with most dipping in and out of chapters (p. 323). following a similar pattern to e-book reading, hughes (2001) reported that 40% of users of reserve print collections indicated that they read less than 50 pages of the reserve texts (p. 116). this could open the way to replace print reserves with electronic texts. studies of e-book collections show that they are used at least as much as print collections. use of titles within the e-book collections evidence based library and information practice 2010, 5.3 7 ranged from 58% (mandel & summerfield, 1998, section 3.2.1.2.), through 70% (dillon, 2001, p. 115) to a high of 94% (langston, 2003, p. 25; grigson, 2009, p. 67). in contrast, christianson (2005) found that most electronic books in the collection were not used at all, and that the high usage could be attributed to a small number of titles (p. 354-5). this was corroborated by littman & connaway’s discovery that that if a title were unpopular in print it was also unpopular in electronic format (2004, p. 261). williams & best (2006) compared titles available in both formats and found that only 7% of the electronic titles circulated compared to 79% of the titles in print format. the remaining 14% of titles were checked out in both electronic and print format (p. 477). chan (2005) reported “...that subject areas that were heavily used in print were also heavily used in electronic form... similarly, subject areas that were not well used in print also received little use in electronic format” (p. 215). dillon (2001) speculated that subjects which “...lend themselves to the quick reference-style lookup that are already part of web behaviour... may indicate that these subjects are particularly suited to web-based ebooks” (p. 119). medicine consistently fell into the top ten subject areas using e-books (langston, 2003, p. 26-27; chan (2005, p. 214; bailey, 2006, p. 58; wilkins, 2007, p. 249). despite this higher use of electronic books in certain subject areas, both ramirez & gyeszly (2001, p. 163) and fernandez (2003, p. 28-29) found an overall preference of print for all subject areas. woo (2005) found that medicine preferred print to electronic at a rate of two-toone (p. 132). wilkins (2007) also noted, however, that e-books are used more heavily when the faculty promote their use and include them on reading lists (p. 249). snowhill (2001) surveyed academic libraries with known e-book collections to determine their experiences (section 8, academic institutions’ experience, bullet 2). all agreed that the acquisition of electronic books had little to no impact on print usage. this pattern was also borne out in other studies (dillon, 2001, p. 124; hughes, 2001, p. 117). online titles, however, appear to be used to a higher degree than the print titles, possibly because of the variance in the type of reading undertaken. electronic versions lend themselves to jumping into a section and then hopping around following links, whereas print versions are more likely to be read extensively. christianson & aucoin (2005) found that fewer e-books were used than print books but that the circulation for those e-books was higher than for the print (p. 75). both hughes (2001, p. 117) and mandel & summerfield (1998, section 3.2.1.2) reported that online titles were used three times more than print titles, while williams & best (2006) reported average use of 2.11 circulations for print compared to 1.30 for electronic titles (p. 477). joint (2009) stated that a digital library is used forty times more than a print library (p. 66), while littman & connaway (2004) described 11% higher usage of e-books than of the print equivalent (p. 260). as can be seen, trying to find a correlation of print and electronic use is complex. on the one hand, popular print titles are also used in electronic format, and certain subject areas and types of materials are used more than others in electronic format. on the other hand, there remains a preference for print format across all subjects. in addition, the electronic texts are used more than print, with the variations ranging from double to forty times, but have had little impact on the use of the print format. difficulties in comparing print usage to electronic usage are compounded by the lack of a standard definition as to what exactly constitutes an electronic “use” and how this compares to a print “circulation”. background memorial university of newfoundland, canada, is a comprehensive university offering a wide range of programs at undergraduate, graduate, and post-graduate levels. the health sciences library is one of evidence based library and information practice 2010, 5.3 8 four libraries and serves the faculty of medicine, the school of nursing, and the school of pharmacy, comprising some 1500 undergraduates, 285 graduates, and 130 permanent faculty. interns and residents of the faculty of medicine rotate throughout the hospitals in the province, and there are many part-time faculty who are practitioners in various regions of the province. the school of nursing has a large distance education program with international enrolment. students, faculty, and staff of the university may obtain remote access privileges to the electronic resources purchased by any of the university libraries via a proxy server. the health sciences library began experimenting with e-books in the online format rather than e-book readers and other formats. the library purchased an electronic copy of the standard harrison's principles of internal medicine as well as the stat!ref package in 2001, and added md consult through a consortial arrangement with other atlantic provinces health libraries in 2002. these resources were supplemented by taking advantage of the university’s main library’s netlibrary and ebrary subscriptions, which include materials in a variety of health-related areas. based on preliminary results of this study, a subscription to selected texts within books@ovid began in 2005, and a subscription to the canadian electronic library: canadian health research collection was added in 2007. a brief description of the electronic packages mentioned is available in appendix a. all course textbooks required by the faculty of medicine are placed in the reserve collection, in addition to instructor-requested texts and those identified by the library as high demand items. normally the library acquires every edition, with the latest placed on reserve and the two previous editions placed in the stacks. books in the stacks circulate for two weeks for undergraduates and four months for graduate students and faculty, with unlimited renewals. the majority of reserves circulate for 2 hours with no renewals. methods four studies were conducted: an analysis of trends of both print and electronic usage, a comparison of print reserve titles relative to the health sciences library’s subscribed electronic versions, and a comparison of print titles in the general collection relative to the health sciences library’s subscribed electronic versions. electronic data the three e-book packages (md consult, stat!ref, and books@ovid) each provided statistics in a different manner. for md consult, the statistics for the consortium as a whole were emailed to each member of the consortium. however, because individual title statistics were not available from md consult, this database was used only to determine comparable print titles. while stat!ref allows compilation of statistics based on a specified time frame, limited historical statistics are available. stat!ref statistics for the trend analysis were obtained for eleven-month periods, february through december, since february 2003 was the earliest date for which stat!ref statistics were available at the time of compilation. the statistic “document use by title” was used. trend analysis statistics for books@ovid were downloaded from their website using the report “monthly book usage report by customer”. neither system contained statistics for unique users, which would be most comparable to print circulation statistics. nor did the systems count usage in exactly the same way, an issue which will be addressed later in the discussion section. print data the titles held by the health sciences library through these three e-book packages were checked against the library catalogue to determine comparable print titles. texts were eliminated from the study if the editions did not match or if the print edition was in the non-circulating reference collection. titles for which the current print edition was on reserve evidence based library and information practice 2010, 5.3 9 were set aside for a separate comparison of the reserve collection. reports were generated from the library’s sirsi unicorn library management system using the “transaction statistics” report to identify whether the call number for the selected reserve titles, and all earlier editions, had a “charge item”, “charge reserve”, or “renew item” transaction during the specified time period. individual transaction reports were compiled for periods of one calendar year, from 1995 to 2009. each edition was specified by its unique call number. in the case of the trend analysis for the circulating collection, the transactions were determined for the base call number, rather than by specific edition, over the specified period. this provided total circulation figures for all editions that are classed in the same call number, including those editions that have since been withdrawn from the collection. the call number transactions were compiled for periods of one calendar year, from 1995 to 2009, which allowed for a trend of print usage to be established in order to compare usage before and after the electronic subscriptions began. the starting date of 1995 was chosen because the unicorn logs only contain data beginning in that year. that year is therefore the earliest data that could be collected for the print editions, regardless of the date of the actual edition held by the library. additionally, statistics are not available for when a print title is off the shelf to be consulted or copied, but not actually signed out. the data was entered into spreadsheets in order to manipulate the data and compare patterns of usage. results and analysis print trends eighty-two titles were examined in the print collection. these titles were selected specifically because they were available electronically in the packages being examined, were in the circulating collection, and circulation figures were available for the three years before and after acquisition of their electronic counterpart. twelve titles that were in the reserve collection were excluded from this section of the study. usage was highest in the first year studied and declined every year thereafter with the exception of 2003 and 2004 when there was a modest increase. similarly, average use of titles in the collection declined except for those same two years (fig. 1). overall average use of a title during the twelve years of the study was 3.8 uses. fig. 1 average print usage 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 evidence based library and information practice 2010, 5.3 10 the highest annual use for any title over the course of the study was for a pharmacy title. its usage dropped substantially in 2000, from 111 uses to 39 uses, and dropped again the following year to 13 uses. there were twenty titles with zero use from 1998-2000, accounting for an average of 27% of the titles studied. the number of print titles never used has been increasing steadily, reaching an average of 60% of the titles studied by 2008-2009. e-book trends stat!ref showed substantial use from its first introduction and usage increased steadily until 2007 (see fig. 2). pharmacy texts are consistently amongst the highest used texts in the collection, and one partial reason for the high usage is that the pharmacy titles have been regularly used in a laboratory class. total usage declined for the first time in 2007 and 2008, but is showing signs of recovery in 2009, though there is no apparent reason for the decline. while one of the most highly used titles changed name in 2007 and its total usage dropped substantially, from some 2000 uses to just over 200 uses, another well-used title also dropped, from some 1800 uses to under 300 uses, with no change in either the name or means of access. thus while some decline can be attributed to the title change, the decline cannot be fully attributed to this one title, and indeed removing both of these titles from the calculations still resulted in a decline in overall usage. thus, the decline is spread over all titles, with 80% of the titles showing a decline in usage from 2006 to 2007. the database vendor has indicated that there was no change in their method of collecting statistics. please note that actual titles are not named at the request of the vendor. as can be seen in fig. 3, the number of e-books in the stat!ref package has increased every year since its purchase. the titles are selected annually by the librarians. while some lowuse titles have been cancelled by the librarians, other titles have been removed by the publisher, including some moderately used titles. with the addition of new titles, one would expect that total usage of the collection would increase, but as already noted, total usage has declined and is only slowly increasing in 2009. this 2006 to 2007 decline in usage, combined with the addition of titles, had a serious effect on the average use per title, from 291 uses per title to 114 (see fig. 3). additional titles in 2008 further decreased average use, but as total usage is recovering, the average use per title is also improving in 2009. fig. 2 stat!ref total usage 0 2000 4000 6000 8000 10000 12000 14000 16000 2003 2004 2005 2006 2007 2008 2009 evidence based library and information practice 2010, 5.3 11 a new subscription to books@ovid was begun partway through 2005, with a package deal for 25 titles as selected by the librarians. seven titles were added in 2006, and a further 29 titles were added in 2007, doubling the size of the collection. in 2008, the library purchased the doody core book collection, for a total of 144 titles currently in the books@ovid collection. as can be seen in fig. 4, total use of the collection has risen each year until 2009, when there is a slight decline in usage. with the addition of so many new titles, increased usage was to be expected. reserve titles, print and electronic the current edition of a title is kept on reserve and two earlier editions are kept in the general circulating collection. five titles in the reserve collection are also in the stat!ref collection, one is in the md consult collection, and five titles are in the books@ovid collection. one title was initially available online from the publisher and is now available in the stat!ref collection. these twelve reserve titles were analyzed to determine if the acquisition of the electronic version had an effect on the use of the print version on reserve. fig. 3 stat!ref average use per title 0 50 100 150 200 250 300 350 2003 2004 2005 2006 2007 2008 2009 number of titles average use per title fig. 5 books@ovid average use per title 0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 2005 2006 2007 2008 2009 average use per title number of titles evidence based library and information practice 2010, 5.3 12 one title was acquired in print at approximately the same time as the electronic version was acquired, and thus there is no earlier circulation pattern for comparison, leaving eleven titles to study. of these 11, two have been cancelled electronically since the study began (one in 2005 and one in 2009), and another two have an earlier edition on reserve than is available electronically. only two titles showed a decline in the year of or following the year in which the electronic version was acquired. however, both of these titles had begun a decline in usage several years earlier, thus the decline in print usage cannot be attributed solely to the introduction of the electronic version. furthermore, this decline in print usage has continued. in another three cases, usage declined the year of acquisition of the electronic version, but increased in the subsequent years. in two cases, usage of the print went up the year the electronic version was acquired, and then returned to normal levels, while in the remaining four cases, there was no noticeable difference in the circulation patterns. note, however, that in all cases, the electronic version is used more frequently than the print version (see fig. 6). the sharp decline in electronic usage for these texts corresponds to the previously mentioned decline in usage from 2006 to 2007. as could be expected, print usage for the reserve texts is highest for the latest edition on reserve, and lower for the older editions in the general collection. however, there is an overall decline in the use of all editions of the reserve texts, including those that are on reserve (fig. 7). it is difficult to ascertain why reserve usage is in decline, particularly as there is no obvious increase in electronic usage of these particular titles. it could be that other electronic titles or evidence based summary databases such as uptodate and essential evidence plus are meeting the needs of users. fig. 6 average usage of reserve titles 0.0 50.0 100.0 150.0 200.0 250.0 300.0 350.0 400.0 450.0 500.0 2 0 0 3 2 0 0 4 2 0 0 5 2 0 0 6 2 0 0 7 2 0 0 8 2 0 0 9 total print electronic evidence based library and information practice 2010, 5.3 13 general collection, print and electronic an examination of the print titles in the general collection, for which the library also had an electronic version via either md consult (37 titles) stat!ref (26 titles), or books@ovid (19 titles) reveals that the acquisition of the electronic versions had virtually no effect on the usage of the print version. this holds true even in eight test cases where the electronic version was a more recent edition than the print version. circulations for the three years previous to the acquisition of the electronic version were compared to the circulations for the three years after the acquisition of the electronic version. in only five cases does it appear that the circulations dropped subsequent to the electronic purchase (see fig. 8). in 59 instances, there is no discernable change in the pattern of use. fourteen cases are questionable, in that the usage changed, but the usage had been variable enough in the previous years to make attributing the decline to the acquisition of the electronic version problematic. surprisingly, there are four instances where usage of the print increased in the year of acquisition of the electronic subscription, or the subsequent year. overall print versus electronic an overall comparison of the average use of the resources clearly reveals that the electronic books are used more than the print books (fig. 9). each print text receives an average of 3 uses per year, compared to 8 uses per year for books@ovid, and 173 uses per year for stat!ref, over the last five years when all three resources were available. even if the print usage were doubled to account for inlibrary usage, it is still less than the books@ovid numbers. it is also quite apparent that stat!ref is much more popular than books@ovid, although a large portion of this can be attributed to the different mix of titles. there are no titles in the books@ovid collection that have major usage when compared to the other titles in the collection, unlike with the stat!ref collection, where exceptional use of a handful of titles skew the statistics. stat!ref contains the popular drug reference texts such fig. 7 total print usage versus reserve usage 0.0 10.0 20.0 30.0 40.0 50.0 60.0 1 9 9 6 1 9 9 7 1 9 9 8 1 9 9 9 2 0 0 0 2 0 0 1 2 0 0 2 2 0 0 3 2 0 0 4 2 0 0 5 2 0 0 6 2 0 0 7 2 0 0 8 2 0 0 9 reserve total print evidence based library and information practice 2010, 5.3 14 as the usp di and ahsf drug information, as well as the merck manual and harrison’s principles of internal medicine. in addition, it must be noted that the different resources do not count usage in exactly the same way making a direct comparison invalid. of greater interest is the comparison of trend lines. it can be seen that books@ovid usage is holding steady, while stat!ref usage has dropped substantially after years of constant increase. one might expect average use of a text to go down as more titles are added, thus distributing the usage across more titles, however titles have been added to both of these electronic packages in the past few years. with added titles, one would also expect that total usage would remain stable or would increase. this is the case for books@ovid, indicating stable use of the collection. stat!ref, however, is showing a substantial drop in 2007 and is now only beginning to show signs of recovery. it would appear that the titles added to books@ovid are receiving usage that is more consistent with the overall average use of the database, thus the added titles are not dissipating the previous average. the 2007 name change of the most highly used title in the collection has had an effect on the stat!ref usage. combined usage for the old and new titles accounted for only half the previous usage, and it halved again in 2008, a total drop of over 2200 uses, or 387%, from 2006 to 2008. as users become familiar with the title change, usage is recovering and is over 1000 uses in 2009. however the stat!ref decline in usage cannot be attributed solely to this title change, as removing this title from the statistics still results in a decline in usage. the trend is still consistent, but not as sharp as with this title included (fig. 9). new titles added to stat!ref are receiving considerably lower usage than the previous average use, indicating that some high-use titles have been removed from the collection, and that their replacements are not as popular. this is often the result of a publisher pulling its title from an aggregator such as stat!ref for exclusive access via their own database. fig. 8 change in print usage with acquisition of electronic version 6.1% 17.1% 72.0% 4.9% decline questionable stable increase evidence based library and information practice 2010, 5.3 15 discussion it is clear that substantial use is being made of the electronic monographs. the trend demonstrates higher use for the electronic version than for the print, and electronic usage figures are increasing steadily from year to year. these numbers indicate that e-books have their place and that they are becoming more popular. as e-books become more prevalent and students become more familiar with them, their usage and acceptance can be expected to grow. this is particularly true if their use is promoted in class, as can be seen from the high use of the pharmacy texts at this university. this would imply that librarians’ promoting e-books through information literacy instruction classes and reference services would positively affect the use of electronic resources. however, since professors have direct control over assigned grades, consideration must also be given to whether a professor’s opinions would have a greater impact on student use of resources than those of a librarian. as with previous studies, the acquisition of electronic versions has had little impact on the use of the print version of a given text in the general collection. only five of the titles examined from the general collection have a downward trend that may be correlated to the acquisition of the electronic version, and the change is minor. with print usage staying consistent over time, there is yet a place for print titles in the health sciences fields. similarly, it appears that electronic versions are not having an effect on the usage of print titles in the reserve collections. however, overall usage of print reserves is lower than might be anticipated. texts are generally placed on reserve due to a high anticipated demand, but with the exception of two titles, demand for reserve titles has dropped considerably since 2004. given the small number of titles examined and the low circulations for those titles, a comprehensive review of titles on reserve is in order, as it appears that many titles are not being used. since the acquisition of the electronic edition has had little impact on the usage of the print edition, further study is needed of overall usage patterns of both the reserve and general print collections where there is no electronic fig. 9 total use of resources 0 2000 4000 6000 8000 10000 12000 14000 16000 1 9 9 8 1 9 9 9 2 0 0 0 2 0 0 1 2 0 0 2 2 0 0 3 2 0 0 4 2 0 0 5 2 0 0 6 2 0 0 7 2 0 0 8 2 0 0 9 stat!ref books@ovid print stat!ref (changed title removed) evidence based library and information practice 2010, 5.3 16 equivalent. this can determine if the usage of print monographs is in decline. moreover, given the low overall usage of the print titles, further study is also needed to determine what constitutes good usage of a print monograph. it is not believed that the non-inclusion of inhouse statistics skewed the results significantly. hardesty’s study on patterns of book usage cited several studies which his own study confirmed, that “recorded circulation is a good indicator of the total use of books...” (1988, p. 75). in his study, in-house usage was approximately 40% that of total use and unused books remained unused over time (p. 67). given the low average of print usage in this study, doubling this average does not impact significantly on the results. further studies of electronic usage, such as the netlibrary or ebrary titles in subject areas of interest to our users, and the recently added springer e-book collection could bolster the case for increasing electronic collections. more detailed statistics, as well as statistics collected over long time frames are needed. several vendors can only supply data for a rolling twelve month period, which is not helpful for detailed statistical analysis. counter standards (www.projectcounter.org) are helpful in ensuring that electronic resources from different vendors are counted consistently, but counter-compliant reports for monographs are slow in becoming available and are not available for historic data. without a method of ensuring that different systems are counting usage in the same manner, it is not possible to do direct comparisons of packages. as well, the problem of matching electronic usage to print circulation remains. first is the problem of defining a “use”. print titles are tracked by charge out, with no reference to the uniqueness of the user, renewals, the use being made, or the number of chapters consulted during the circulation. electronic titles are tracked by clicks into a document/chapter with no reference to the motives behind the click, for example whether the wrong item was accidently clicked, a quick skim to see whether or not the item is useful, or an in-depth perusal (sottong, 2008, p. 45). and there is no consideration to whether it was one person with multiple accesses or many people accessing one document/chapter. while a usage by chapter count gives a better idea of the actual usage of a text than a circulation count, it also exaggerates the usage of a text when compared to the print circulation count, as the circulation count cannot measure the number of times a user consulted a chapter while they had the book checked out, or if a colleague consulted the text another had signed out. a more accurate measure might be to count on the basis of ip address, counting one usage for each uninterrupted access to a resource. the difficulty would be in determining if the access was uninterrupted or whether a user had changed at a public station, and would still not account for one person continually using the resource over the same time period for which the print version was checked out, or a person who has had to log back in because of a system error or time-out. a qualitative study such as a focus group or survey could address this difficulty and provide insight into how a user utilizes electronic resources. conclusion the study posed two questions: 1) should a health sciences library acquire medical ebooks? 2) can electronic versions of standard medical texts replace the print versions on reserve? in the first instance this study has found that medical and health sciences libraries should acquire electronic books. the high and increasing use being made of electronic texts indicates that medical e-books are being sought by users of medical collections. while the difference in counting can account for some differences in the numbers of e-book usage versus print book circulation, it cannot negate the static trend for print titles and the positive trend for electronic resources. electronic versions can provide multiple user access to the most recent edition, which is generally updated on a regular basis. in contrast to printed volumes, users do not have to settle for an older edition because the http://www.projectcounter.org/ evidence based library and information practice 2010, 5.3 17 current edition is signed out for a significant period of time. however the selection of titles is critical to usage, as seen from the differing use of the two electronic packages. simply because a text is available electronically does not mean that it will be used any more than all print titles on the shelf are used. titles that are more popular in print will also be more popular in electronic format. also critical is the required use or promotion of electronic texts by faculty. this can be supplemented by librarians using electronic resources in instruction and reference services. secondly, in relation to replacing the print reserve collection with an electronic one, the higher use of the electronic texts that have print versions on reserve, coupled with the low print circulations of reserve texts, suggests that libraries can be more selective in the items that are placed on reserve and could replace many print reserves with electronic texts. users are becoming more familiar with electronic texts and more comfortable with reading text on screen, particularly when reading selectively as is more usual with reserve reading. the text is readily available at all times of the day and users do not have to settle for a short reserve loan period. reserve desks are now pointing to electronic versions of journal articles rather than photocopies, and students are coming to expect that reserve items will be available electronically. however, the importance of the print collection cannot be ignored. the acquisition of the electronic version has little impact on the circulation levels of print versions, and so any decision to move from one to the other cannot be based on statistics alone. qualitative studies will be needed to inform and reinforce any decision. however the apparent low overall use of the print collection and the accessibility of e-resources strongly suggest that electronic texts should replace much of the print collection. an overall examination of the print collection would prove helpful in confirming that the low use is consistent throughout the collection and provide the impetus to fully develop the electronic monograph collection. references bailey, t. p. (2006). electronic book usage at a master's level i university: a longitudinal study. journal of academic librarianship, 32, 52-59. doi:10.1016/j.acalib.2005.10.004 chan, g.r.y.c., & lai, j. k. (2005). shaping the strategy for e-books: a hong kong perspective. library collections, acquisitions, & technical services, 29, 205-219. doi:10.1016/j.lcats.2005.04.002 christianson, m. (2005). patterns of use of electronic books. library collections, acquisitions, & technical services, 29, 351-363. doi:10.1016/j.lcats.2006.03.014 christianson, m., & aucoin, m. (2005). electronic or print books: which are used? library collections, acquisitions, and technical services, 29, 71-82. doi: 10.1016/j.lcats.2005.01.002 dillon, d. (2001). e-books: the university of texas experience, part 1. library hi tech, 19, 113-124. doi: 10.1108/07378830110394826 fernandez, m. (2003). a usage comparison for print and electronic books in the university of north carolina at chapel hill. unpublished master’s thesis, university of north carolina, chapel hill, nc. grigson, a. (2009). evaluating business models for e-books through usage data analysis: a case study from the university of westminster. journal of electronic resources librarianship, 21, 62-74. doi: 10.1080/19411260902858623 hardesty, l. (1988) use of library materials at a small liberal arts college: a replication. collection management, 10(3/4), 61-80. hughes, c. a. (2001 april). the myth of evidence based library and information practice 2010, 5.3 18 obsolescence: the monograph in the digital library. portal: libraries & the academy, 1, 113-119. retrieved 29 april 2009 from http://muse.jhu.edu/journals/portal_li braries_and_the_academy/v001/1.2hu ghes.pdf joint, n. (2009). choosing between print or digital collection building in times of financial constraint. library review, 58, 265-271. doi: 10.1108/00242530910952819 langston, m. (2003 spring). the california state university e-book pilot project: implications for cooperative collection development. library collections, acquisitions, and technical services, 27, 19-32. doi:10.1016/s14649055(02)00305-6 levine-clark, m. (2006). electronic book usage: a survey at the university of denver. portal: libraries & the academy, 6, 285-299. retrieved 28 april 2009 from http://muse.jhu.edu/journals/portal_li braries_and_the_academy/v006/6.3lev ine-clark.pdf littman, j., & connaway, l. s. (2004) a circulation analysis of print books and e-books in an academic research library. library resources & technical services, 48, 256-262. mandel, c., & summerfield, m. (1998 january). books online: potentialities and realities. retrieved 27 june 2010 from http://www.arl.org/resources/pubs/sp ecscholmono/mandel.shtml nicholas, d., rowlands, i., clark, d., huntington, p., jamali, h. r., & ollé, c. (2008). uk scholarly e-book usage: a landmark survey. aslib proceedings, new information perspectives, 60, 311334. doi: 10.1108/00012530810887962 ramirez, d., & gyeszly, s. d. (2001). netlibrary: a new direction in collection development. collection building, 20, 154-164. doi: 10.1108/eum0000000005995 snowhill, l. (2001). e-books and their future in academic libraries. d-lib magazine, 7(7/8). retrieved 17 march 2006 from http://www.dlib.org/dlib/july01/snow hill/07snowhill.html sottong, s. (2008). the elusive e-book. american libraries, 39, 44-48. summerfield, m. (1998). online books: what role will they fill for users of the academic library? in c. m. laguardia & b. mitchell (eds.), finding common ground: creating the library of the future without diminishing the library of the past. (pp. 313-325). new york: nealschuman publishers. wilkins, v. (2007) managing e-books at the university of derby: a case study. program: electronic library & information systems, 41, 238-252. doi: 10.1108/00330330710774129 williams, k. c., & best, r. (2006) e-book usage and the choice outstanding academic book list: is there any correlation? the journal of academic librarianship, 32, 474-478. doi:10.1016/j.acalib.2006.05.007 woo, h. (2005) the 2004 user survey at the university of hong kong libraries. college & research libraries, 66, 115135. http://muse.jhu.edu/journals/portal_libraries_and_the_academy/v001/1.2hughes.pdf http://muse.jhu.edu/journals/portal_libraries_and_the_academy/v001/1.2hughes.pdf http://muse.jhu.edu/journals/portal_libraries_and_the_academy/v001/1.2hughes.pdf http://muse.jhu.edu/journals/portal_libraries_and_the_academy/v006/6.3levine-clark.pdf http://muse.jhu.edu/journals/portal_libraries_and_the_academy/v006/6.3levine-clark.pdf http://muse.jhu.edu/journals/portal_libraries_and_the_academy/v006/6.3levine-clark.pdf http://www.arl.org/resources/pubs/specscholmono/mandel.shtml http://www.arl.org/resources/pubs/specscholmono/mandel.shtml http://www.dlib.org/dlib/july01/snowhill/07snowhill.html http://www.dlib.org/dlib/july01/snowhill/07snowhill.html evidence based library and information practice 2010, 5.3 19 appendix a: electronic book packages books@ovid http://www.ovid.com/site/products/books_landing.jsp books@ovid is an online collection of clinical texts using the ovid interface. lippincott, williams & wilkins is the main publisher represented and it has a strong medical/nursing clinical collection. canadian health research collection http://www.canadianelectroniclibrary.ca/cdn_health_research_collection.html the canadian health research collection is an electronic collection of health monographs and reports published by canadian research institutes and universities as well as various government agencies. the documents are made available on the ebrary platform. ebrary http://www.ebrary.com/corp/libraries.jsp ebrary is an e-book vendor / platform. it offers packages as well as hosting services for electronic titles purchased elsewhere. it has over 150,000 e-books available from over 400 publishers. medical texts are generally not the current edition of clinical texts. essential evidence plus http://www.essentialevidenceplus.com/ formerly known as infopoems, essential evidence plus is a clinical reference tool, providing point-of-care summaries for clinicians. md consult http://www.mdconsult.com/php/208599885-2/homepage md consult is a full-text database offering a package of elsevier journals, monographic series, and books online through its own search interface. netlibrary http://library.netlibrary.com/home.aspx netlibrary is an e-book vendor. it has over 200,000 e-books available from a wide selection of publishers. until recently, medical texts were generally not the current edition of clinical texts. springer e-book collection http://www.springer.com/librarians/e-content/ebooks?sgwid=0-40791-0-0-0 the springer e-book collection provides online access to all texts published by springer publishing. titles can be purchased individually or as subject or yearly packages. stat!ref http://www.statref.com/ stat!ref is a full-text aggregator database of many of the top clinical monographs from a wide variety of publishers. uptodate http://www.uptodate.com/home/index.html uptodate is a clinical reference tool, providing point-of-care summaries for clinicians. http://www.ovid.com/site/products/books_landing.jsp http://www.canadianelectroniclibrary.ca/cdn_health_research_collection.html http://www.ebrary.com/corp/libraries.jsp http://www.essentialevidenceplus.com/ http://www.mdconsult.com/php/208599885-2/homepage http://library.netlibrary.com/home.aspx http://www.springer.com/librarians/e-content/ebooks?sgwid=0-40791-0-0-0 http://www.statref.com/ http://www.uptodate.com/home/index.html evidence summary   academic library administrators perceive value in their librarians’ research   a review of: perkins, g.h. & slowik, a.j.w. (2013). the value of research in academic libraries. college & research libraries, 74(2), 143-158. retrieved from http://crl.acrl.org/content/74/2/143.full.pdf+html   reviewed by: elaine sullo coordinator, information and instructional services himmelfarb health sciences library the george washington university washington, district of columbia, united states of america email: elainej@gwu.edu   received: 11 jun. 2014   accepted: 15 aug. 2014      2014 sullo. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to explore academic library administrators’ perceived value of their librarians’ research, specifically the importance to the profession and the library community.   design – qualitative, exploratory study using a survey questionnaire.   setting – academic libraries in the united states of america.                        subjects – 23 library administrators.   methods – during the summer of 2010, one of the authors conducted 20-30 minute telephone interviews with 23 academic library administrators. interviews were recorded and transcribed for coding. interview questions 1-3 and 8-19 were content-analyzed; the authors described common themes for each of these questions. items 4-7 had likert scale response formats, and a mean and standard deviation were computed for each of these items.   main results – the benefits of librarians’ research included fulfilling tenure-track requirements, enriching relationships with teaching faculty, library faculty recognition, improved services and programs, collaboration with others, research result application to daily issues, development as librarians, and improved knowledge of the research field.   the perceived current changes and future issues for university libraries included increased digitization of collections, scholarly communication, and expanded instructional engagement of faculty and students, as well as future economic downturn and budget cuts. administrators noted several methods that influenced their thinking: professional meetings, reading professional journals, informal discussions with colleagues, and social media such as facebook and twitter.   academic library administrators used a variety of methods to support their librarians’ research. these included tenure-track requirements, research incentives, travel funds, grants, sabbaticals, release time, and shared communication about research. additionally, there was a substantial perceived interrelationship between how librarians’ research benefited the librarian, the library, the university, and the profession. recognition and new programs and services were thought to benefit all four areas, and monetary rewards were considered benefits for the first three areas.   conclusion – based on the sample of 23 academic library administrators, the authors conclude that librarians’ research is perceived as valuable to both the academic and library communities.     commentary   while early history of research in academic libraries did not show much value for librarians, the association of college and research libraries (acrl)’s first “standards for college libraries” in 1957 marked the beginning of setting research standards and valuing academic librarians’ research. most literature since this point written about librarians’ research roles has emphasized advantages or disadvantages of library research, as well as the level of institutional support for such undertakings and its effect on research activities. the authors of this study state that they hope to add to the dialogue on academic research by assessing the opinions of library administrators for the benefit of the research community.   the article was appraised using the critical appraisal skills programme qualitative research checklist (casp, 2013). the structured interview format was appropriate for the exploratory nature of this study. the steps taken by the authors for data collection, analysis, reporting, and explicit description of findings meet the criteria specified in the casp checklist. the survey questionnaire used in the study is included as appendix a in the article.   in addition to the qualitative data, the survey also asked participants several demographic questions, such as title, number of library volumes, highest academic degree earned, whether the institution is public or private, ala accredited, or arl affiliated. this data is clearly presented in table format throughout the article.   the authors use several methods to ensure trustworthiness and credibility of their research. both authors reviewed the interview transcripts and agreed upon themes. additionally, by using several participant quotes, the authors were able to share the rich, thick description of the context with the reader. however, while the authors offered to send a copy of the finished study to the interviewees, they did not do member checks with the participants to verify that the interview transcripts had indeed captured the ideas that were intended to be conveyed.   this study was well-designed and could serve as a model of how to conceptualize and report the findings of qualitative research. as such, the article is of potential interest to a wide audience. furthermore, as the study methodology is well constructed and explained in detail, it could be replicated by other researchers. the results are clearly presented, and provide the audience with a wealth of information which could lead to additional research ideas. finally, because the article includes demographic information about the administrators and their associated libraries, the data could be relevant to other libraries that share the same qualities as these institutions.   considering the sample size for this study, there may be some concern regarding the issue of confidentiality. details from the research results could lead to the identification of institutions; however the answers to specific interview questions would be more difficult to associate with a particular library.   other than stating a confirmation that the research suggests that library administrators perceived multiple values of their librarians’ research, the authors mention that what was discovered during this research project was similar to what was already mentioned in the literature. they also suggest that additional research is needed to shed further light on this topic. perhaps a more semi-structured interview guide, along with additional participant quotations, would lead to a richer understanding of the value of research and the specific institutional policies that either support or hinder its progress.   references   critical appraisal skills programme (casp). (2013). qualitative research checklist. retrieved from http://www.caspinternational.org/index.asp?o=1076   evidence based library and information practice evidence based library and information practice 2009, 4.4  evidence based library and information practice       ebl 101    conducting your own research: something to consider    virginia wilson  shirp coordinator, health sciences library   university of saskatchewan    saskatoon, saskatchewan, canada    email: virginia.wilson@usask.ca    received: 01 november 2009        accepted: 11 november 2009      © 2009 wilson. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    in the last issue we looked at a variety of free  and open access lis resources that you can use  to find evidence pertaining to your practice  question. consulting the evidence is obviously  the cornerstone of evidence based practice and  evidence based librarianship (ebl). there are  occasions, though, when a search of the  available evidence just isn’t fruitful. the  articles found may not be directly applicable  to your situation. the articles might increase  your body of knowledge about a particular  subject, but perhaps are not practical enough.  or, you might not be able to find anything at  all. this ebl 101 column asks you to consider  collecting your own evidence in order to  inform your decision‐making process, i.e.,  undertake some research of your own.    of the three definitions proposed for ebl  early in the millennium, only one specifically  mentions the notion of librarians conducting  research. crumley and koufogiannakis’  definition describes ebl as including  “encouraging librarians to conduct high  quality qualitative and quantitative research”  (62). for academic librarians on the tenure  track, such research is often essential for  professional movement, and will most likely  (although not always) be tied to professional  practice. however, there are more benefits to  be derived from conducting research than  getting tenure or promotion, and ideally more  than tenure‐track, academic librarians will be  conducting it.    what are the benefits of practitioners  conducting research? the benefits are many,  and i would suggest that they outweigh the  barriers that might hamper librarians and  information professionals in their quest to  conduct research:  • research aids in the improvement of  problem solving and decision making  in the workplace.   • research by library and information  professionals creates new knowledge  and contributes to the growth of lis  as a profession and a discipline.   • undertaking research can help make  library professionals critical  consumers of the research literature.   • librarian‐researchers bring an  additional set of skills to the table in  95 mailto:virginia.wilson@usask.ca http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2009, 4.4  order to provide optimal information  services to researchers in other fields.   • conducting a research project can  improve an individual’s ability to  think critically and analytically.  (adapted from versoza)    it’s important not to let the thought of a  research project strike fear into your heart.  one of the barriers to the practice of ebl is the  real or perceived notion held by librarians that  they do not possess the necessary skills to  undertake research. however, projects can be  simple and practical. the process of ebl is  intended to help improve your decision  making and problem solving at work. thus,  research questions that emerge from practice  can be powerful motivating factors in the  conducting of research because they are found  in your everyday work and can aid you with  it.     work has been done of compiling research  questions that need to be answered. in 2001,  jonathan eldredge published an article  entitled “the most relevant and answerable  research questions facing the practice of  health sciences librarianship.” in 2006, lewis  and cotter looked at that issue again in their  article, published in evidence based library and  information practice, entitled “have the most  relevant and answerable research questions  facing librarians changed between 2001 and  2006?” both articles are useful in terms of  getting an idea of what research topics are  important to librarians, health or otherwise,  and can help set the stage for thinking about  your own practice questions.    professional associations can be a good source  of both research topics and research funding.  often, associations will put together a research  agenda, a way to focus resources on research  topics that would benefit members.  • cla/acb (canadian library  association) library research and  development grants  http://www.cla.ca/am/template.cfm? section=grants&template=/cm/htm ldisplay.cfm&contentid=5758  be sure to check with your cla  division or interest group for research  funding opportunities as well.  • acrl (association of college and  research libraries) research and  scholarship committee  http://www.ala.org/ala/mgrps/divs/ac rl/about/sections/is/committees/resear ch.cfm  this page contains the acrl research  agenda for library instruction and  information literacy, a bibliography of  research methodology resources, and  much more for the librarian‐ researcher.  • mla (medical library association)  research  http://www.mlanet.org/research/  this page contains grant information,  research resources, and projects and  studies.  • sla (special libraries association)  research grant  http://www.sla.org/content/resources/ scholargrant/resgrant/index.cfm  this grant description suggests  research topics and the focus is on  evidence‐based practice.    there are a variety of books and websites that  can assist you in becoming familiar with and  more adept at various research methods.  workshops, webinars, and online courses can  help you spruce up your skills as well.    a few resources for the librarian‐ researcher  online activities:  • the education institute often has  sessions that deal with research  methods.  http://www.thepartnership.ca/partner ship/bins/index_ei.asp  some books to get you started:  • leedy, paul d., and jeanne ellis  ormrod. practical research: planning  and design. 8th ed. upper saddle  river, nj: prentice hall, 2005.  • powell, ronald r., and lynn silipigni  connaway. basic research methods  96 http://www.cla.ca/am/template.cfm?section=grants&template=/cm/htmldisplay.cfm&contentid=5758 http://www.cla.ca/am/template.cfm?section=grants&template=/cm/htmldisplay.cfm&contentid=5758 http://www.cla.ca/am/template.cfm?section=grants&template=/cm/htmldisplay.cfm&contentid=5758 http://www.ala.org/ala/mgrps/divs/acrl/about/sections/is/committees/research.cfm http://www.ala.org/ala/mgrps/divs/acrl/about/sections/is/committees/research.cfm http://www.ala.org/ala/mgrps/divs/acrl/about/sections/is/committees/research.cfm http://www.mlanet.org/research/ http://www.sla.org/content/resources/scholargrant/resgrant/index.cfm http://www.sla.org/content/resources/scholargrant/resgrant/index.cfm http://www.thepartnership.ca/partnership/bins/index_ei.asp http://www.thepartnership.ca/partnership/bins/index_ei.asp evidence based library and information practice 2009, 4.4  97 for librarians, 4th ed. westport, ct:  libraries unlimited, 2004.  online tools for research activities:  • survey monkey  http://www.surveymonkey.com/  • sample size calculator:  http://www.surveysystem.com/sscalc. htm  • webpages that perform statistical  calculations: http://statpages.org/  • bank of canada inflation calculator:  http://www.bankofcanada.ca/en/rates/ inflation_calc.html  • outcomes toolkit:  http://ibec.ischool.washington.edu/sta tic/ibeccat.aspx@subcat=outcome%20t oolkit&cat=tools%20and%20resources &tri=toolkitabout.htm   (these links are from the evidence based  toolkit for public libraries created by  steph hall and me:  http://ebltoolkit.pbworks.com/.)  conducting your own research from a front  line perspective will assist you in your own  decision‐making process. but make sure you  give thought to broadcasting the results  beyond your own library. by disseminating  your results via scholarly journal, conference  presentation, personal communication, or  other dissemination mode, you will be making  it possible for other practitioners to learn from  and build upon your work. a barrier to the  practice of ebl is lack of buy in by managers.  the research process, including the publishing  of results, is not seen as important or  worthwhile in some instances. however,  genoni, haddow, and ritchie state that  “practitioners must increase their  contributions to research literature. … the  writing‐up of research needs to be recognized  and rewarded by employers as a legitimate  professional task” (57). research projects can  enhance the library’s status within its  community, both locally and professionally,  which will help to generate the buy in. the  dissemination process helps build and  strengthen the evidence base, and helps to  build a culture of research amongst librarian  and information professionals.     i’m always on the lookout for good quality  books/websites/online sources and courses  dealing with research, and i know i’ve only  scratched the surface in this column. please  pass along your favourites to me and i will  compile them for a future column  (virginia.wilson@usask.ca). coming up next  time in ebl 101, an introduction to critical  appraisal.      works cited    crumley, ellen, and denise koufogiannakis.  “developing evidence‐based  librarianship: practical steps for  implementation.” health information  and libraries journal 19.2 (2002): 61‐ 70.    eldredge, jon. “the most relevant and  answerable research questions  facing the practice of health sciences  librarianship.” hypothesis 15.1  (2001): 9‐17.    genoni, paul, gaby haddow, and ann  ritchie. “why don’t librarians use  research?” evidence based practice  for information professionals: a  handbook. eds. andrew booth and  anne brice. london: facet, 2004. 49‐ 60.    lewis, suzanne, and lisa cotter. “have the  most relevant and answerable  research questions facing librarians  changed between 2001 and 2006?”  evidence based library and  information practice 2.1 (2007): 107‐20.    verzosa, fe angela m. “motivating librarians  to conduct research.” philippine  association of academic and research  librarians (paarl) national  seminar‐workshop on empowering  information professionals to make a  difference. ateneo de naga  http://www.surveymonkey.com/ http://www.surveysystem.com/sscalc.htm http://www.surveysystem.com/sscalc.htm http://statpages.org/ http://www.bankofcanada.ca/en/rates/inflation_calc.html http://www.bankofcanada.ca/en/rates/inflation_calc.html http://ibec.ischool.washington.edu/static/ibeccat.aspx@subcat=outcome%20toolkit&cat=tools%20and%20resources&tri=toolkitabout.htm http://ibec.ischool.washington.edu/static/ibeccat.aspx@subcat=outcome%20toolkit&cat=tools%20and%20resources&tri=toolkitabout.htm http://ibec.ischool.washington.edu/static/ibeccat.aspx@subcat=outcome%20toolkit&cat=tools%20and%20resources&tri=toolkitabout.htm http://ibec.ischool.washington.edu/static/ibeccat.aspx@subcat=outcome%20toolkit&cat=tools%20and%20resources&tri=toolkitabout.htm http://ebltoolkit.pbworks.com/ mailto:virginia.wilson@usask.ca evidence based library and information practice 2009, 4.4  university, bicol, philippines, 24 october  2006. 1 nov. 2009  <http://www.slideshare.net/verzosaf /motivating‐librarians‐to‐conduct‐ research>.   98 http://www.slideshare.net/verzosaf/motivating-librarians-to-conduct-research http://www.slideshare.net/verzosaf/motivating-librarians-to-conduct-research http://www.slideshare.net/verzosaf/motivating-librarians-to-conduct-research evidence based library and information practice editorial responsibilities      2013. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.       editor-in-chief: alison brettle   associate editor (evidence summaries): heather pretty   associate editor (articles): lorie kloda; derek rodriguez   associate editor (reviews): denise koufogiannakis   associate editor (classics): jon eldredge   associate editor (commentaries, using evidence in practice, news): alison brettle   production editor: michelle dunaway   editorial intern: archana deshmukh   copyeditors: richard hayman (lead copyeditor), georgianne bordner, molly des jardin, linda ferguson, richard hayman, stacey penney, maria tan, mary virginia taylor, alison yeoman   indexing support: pam morgan   feature   the 7th international evidence based library and information practice conference (eblip7): conference commentaries and reflections      2013. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   rick stoddart, assessment librarian, oregon state university libraries & press, oregon state university, usa   can you have a crush on a methodology? well, i am little embarrassed to say i left the university of saskatchewan quite enamoured with evidence based library and information practice. as an assessment librarian, i am attracted to all aspects of librarianship that elegantly bridge library practice and evidence based decision making. the eblip7 conference was definitely jam packed with this sort of smartly done librarianship. each conference presentation, lightning talk, and poster session incorporated the beauty of evidence based practice with library content such as instruction, space planning, outreach, and research methodologies. the conference program itself was also spiced with verve and pluck especially in the poster madness and lightning strikes sessions. likewise, let’s not forget about whimsy and mischief at the evidence based hootenanny either. these events keep the conference lively like the bubbles in a glass of champagne. what was not to love about eblip7?   the keynote by denise koufogiannakis was quite inspirational and i took away her framework for the differing types of evidence and discussion of an amended evidence based model for librarianship to share with my colleagues. i attended presentations about the intersection of evidenced based practice and librarian research agendas, virtual reference, library leadership, and student performance. librarian research agendas and virtual reference were particularly timely topics for my library, and i gained some new perspectives in these areas to put to use right away when i returned home.   personally, i was encouraged that eblip7 was inclusive of librarians of all stripes, including medical, public, school, and academic librarians. i guess i am not the only librarian charmed by the potential of evidence based practices to influence effective decision-making in libraries. i was grateful for the opportunity the conference afforded to network with other assessment librarians. i think one of my most important takeaways was being able to build deeper connections between library assessment and evidence based library and information practice. lorie kloda’s lightning strike session the last day really helped put the library assessment puzzle piece into the overall evidence based librarianship picture.       generally, there was a lot of energy and camaraderie at eblip7 that was certainly contagious throughout the conference. i definitely left saskatoon excited about the idea of evidence based practices in libraries. i attribute this ongoing conference enthusiasm to the planning and efforts of virginia wilson, the conference chair, and the rest of the conference planning committee. continued kudos on a conference well done!   perhaps it is the lingering buzz from the cheaply priced beer at the evidence based hootenanny or the sugar rush from the saskatoon berry pie at the ribbon cutting ceremony at new centre for evidence based library & information practice, but let it be known that eblip7 was a blast, and i can’t wait for the next one.   daniel hsieh, princeton junction, new jersey, usa   i attended eblip7 in july, 2013 at university of saskatchewan (uofs) as a delegate’s companion and stayed in the campus dormitory. the selection of the uofs as the conference site was an excellent choice. the campus is so beautiful and the university offers the perfect infrastructure for a conference. the simple lifestyle on campus was interruption-free, which helped me focus on what i wanted to accomplish: i finished writing the major part of my scientific manuscript during my stay in the dormitory.    eblip7 organizers arranged two exciting tours for the delegate’s companions: one was a visit to a local brewing company and the other a visit to the famous canadian light source synchrotron at the uofs. as a chemical engineer working in the pharmaceutical industry, i found these tours very interesting and educational.   tasting all kinds of beer was the first step of visiting the local brewery. fortunately after this tasting event, i was still fully awake. all kinds of grains suitable for the beer making process were displayed at the entrance, which were also available for purchase by local beer makers. the basic tools and recipes for beer making were also displayed to help us understand the beer making process. one interesting part of this process is the liquid seal associated with the fermentation reactor.     this liquid seal has two functions:  first, this seal prevents oxygen from getting into the fermentation process, because oxygen will ruin the taste of the beer. second, this seal can be used as an indicator of the end of fermentation process. during beer making, carbon dioxide is produced as a product and can escape from the fermenter through the liquid seal. when the fermentation process is complete, no carbon dioxide will bubble through the seal. this signals the end of the fermentation process and the beer is almost ready to be harvested.   a senior research scientist at the canadian light source introduced us to the famous canadian synchrotron—an excellent introduction on how the moving particles are generated at a speed close to light speed and how the synchrotron can be applied to real life. i was particularly impressed with two amazing applications: the identification of the origin of diamond from extremely low levels of mineral impurities, and the reason why arthritis is so painful. this is because there are numerous tiny spikes from the bone at the joints penetrating into skin. these kinds of spikes cannot be detected using regular x-rays, but they can be observed from the results generated using the synchrotron.    during the conference, we had a chance to attend a food festival in downtown saskatoon, a taste of saskatchewan, where we also enjoyed a live concert. on the way back to the university, we walked along the south saskatchewan river, a beautiful view.   the arrangements for eblip7 were excellent and i brought back a souvenir given by eblip7, a book entitled prairie dreams, which contained photographs from photographer courtney milne.  the photos in this book are a fond reminder of my trip to saskatoon.   iffat ahmad, university library, university of saskatchewan, canada   i work at the university library, university of saskatchewan, and this year i was a delegate at the eblip7 conference hosted by the university library. i was able to attend this conference as a recipient of the marjorie clelland scholarship. this is an annual scholarship created in the memory of a highly respected colleague who is no longer with us. with her dedication and hard work, marjorie clelland left a legacy that inspires us to continue to better ourselves. the scholarship in her name annually allows library staff to attend a conference related to academia and library work. upon being awarded this scholarship, i gladly chose to attend a conference right here at the university of saskatchewan.   i attended a variety of sessions on a broad range of topics that were both interesting and informational. the papers presented covered subjects such as copyright as it relates to eblip, as well as noise control in libraries. all of the presenters were knowledgeable and enjoyable to listen to. i truly felt that all of the sessions were well worth attending. i cannot stress enough what a fun experience it was to attend eblip7 here at my place of work. having never been a delegate at a conference, i enjoyed the novelty of being one. it gave me a new perspective of the university – as viewed by a visitor to the campus.   this gave me a new appreciation as to what a truly beautiful campus it is. having never attended a conference before, i had nothing to compare it to. however, i am in agreement with fellow delegates who said that is was incredibly efficient and smoothly run. virginia wilson, her organizing team, and the many volunteers and session facilitators did a splendid job in ensuring that ebpli7 was a huge success.   sandra stubbs, university of the west indies mona, western jamaica campus, west indies   the caribbean was represented by four librarians, all from the university of the west indies (uwi), including the university librarian from the st. augustine campus in trinidad and tobago, and the campus librarian from the mona campus of uwi in kingston, jamaica.   i previously attended eblip6 in 2011 in manchester, england, and a conference of the association of caribbean university research and institutional libraries (acuril) in 2010 in the dominican republic. at the latter, the conference was themed around evidence based library and information practice, and presentations were led by some of the chief proponents of eblip, and so i was eager to investigate how colleagues worldwide had embraced and advanced the evolving concepts and models of evidence based discussion and application in libraries.   i was not disappointed. the opening keynote address by dr. denise koufogiannakis, entitled “what we talk about when we talk about evidence,” set an excellent tone for eblip7. she discussed the evidence based practice movement with its origins in the field of medicine, to what is meant by evidence in the context of library and information practice and how such evidence might be utilized in practical ways to inform decision making. she spoke of four types of evidence, namely anecdotal, causal, expert, and experiential, and then examined categories of hard versus soft evidence. this was especially useful for those who were still trying to understand the paradigms and models in what is now commonly known as to as the international eblip movement.     in making selections of sessions to attend from the many concurrent choices available, i was happy to be able to participate in presentations of research and discussions of current practice in my areas of interest, including noise management in libraries, copyright, virtual reference, prevailing versus ideal research environments, and changes in academic library space. as expected, the poster sessions were very informative. a number of poster presenters incorporated the useful facility whereby participants were able to use our mobile devices to scan the posters for instant access to both the presented and additional information via their online web pages.        evadne mclean and sandra stubbs from jamaica in foreground   my caribbean colleagues and i especially enjoyed the lightning strikes, which saw presenters very creatively presenting evidence based projects in the allotted time of just five minutes each. these strikes were much more engaging and information filled than one might anticipate. audience reactions and applause underscored their utility. congratulations to all presenters and the organizing committees for this and the successful staging of the entire conference.   venue appreciation   the venue chosen for eblip7 was picturesque. i was among the eblip7 delegates who chose accommodation in the halls of residence on the beautiful campus of the university saskatchewan. this afforded walking access to the conference sessions, which were all held at the campus.  i was particularly fascinated with the rocks in the permanent exhibition displayed in the geology building.   hospitality was superb. the generosity of the people of saskatoon carried over into all social programmes. i don’t remember going to a conference before where the consensus among attendees was that we were perhaps overfed.   centre for evidence based library & information practice   during the conference we were pleased to be part of the historic opening of the centre for evidence based library & information practice in the murray library. this function included a ribbon-cutting ceremony (followed by lunch) with university library dean vicki williamson officiating.   social programmes   part of what makes conferences memorable is the opportunities to engage with the local and international populace outside of the conference setting, and to explore parts of the host city. i had an enjoyable bus ride into the country side and back to the barn playhouse for an event that was dubbed “evidence based hootennany” and yes, it did include lots of wine, food (with too much dessert  personally prepared by the chair of the local organizing committee), barn dancing, prizes, and surprises!   a taste of saskatchewan & kiwanis park   a group of us joined saskatoon librarians from the organizing committee on a very enjoyable walk along the river to the event dubbed “taste of saskatchewan”. this was the 18th anniversary of this festival, held in the beautiful kiwanis park. as a kiwanian, i was especially excited to visit this memorial park and observe the many monuments, pavilions, fountains, statues and information billboards showcasing historical saskatoon and world events. this park is located between the broadway bridge and the university bridge on the banks of south saskatchewan river.   the taste of saskatchewan festival featured over 30 of saskatoon’s finest restaurants serving their favourite dishes, including entrees, desserts, and speciality items. my eblip7 colleagues and i rocked and danced to music of live bands in performance and basked in the atmosphere. the festival was said to be featuring over 55 of saskatchewan’s premier bands and performers.   overall, the conference was thoughtfully organized with relevant information available throughout from courteous volunteers at the information desk. i look forward to using and sharing valuable information garnered from the many sessions attended with my colleagues in the west indies and beyond.   cheers!     saskatchewan hall   crystals on display in the geology building   participants at the opening ceremony of centre for evidence based library & information practice   participants liming outside the barn playhouse ahead of post-dinner activities       sharon murphy, head, academic services division, queen’s university library, kingston, on, canada   as my plane touched down in saskatoon on july 15th i was greeted with a tornado warning. welcome to the prairies, and welcome to the 7th international evidence based library and information practice conference! the meteorological tornado never occurred but the intellectual and social burst of energy started from day one and continued building throughout the conference. lest my metaphor be taken in any negative connotation, i have to add that this was one positive storm of energy.   my first foray into the eblip community was two years ago at the university of salford during the 6th eblip conference. little was my understanding of evidence based librarianship at that time, and i felt like i had stumbled in as a visitor from a strange and foreign land, i.e., from librarianship outside of the health sciences. the eblip community welcomed me then and continues to inform and inspire me now. so the opportunity to attend this year, in a sister canadian university, was irresistible.   reviewing my notes as i prepared to write this reflection reminded me that every session i attended was a stimulating presentation of solid research – truly, every single session. i participated in sessions on copyright, fines, citation management, citation data analysis, scholarly communication, system dynamic modeling, action-learning for writing, leadership development, and assessment. certainly the scope of research shows how eblip is an approach to understanding and practicing librarianship and not a discipline specific pursuit. i’m only sorry i missed other brilliant presentations – such is the conundrum of concurrent sessions.   we did all get to enjoy the posters, keynotes, and socializing. the posters alone were worth the price of admission and had the most brilliant introductions i have ever witnessed during poster madness. we were enticed to view the posters by the creators giving introductions that included skits, costumes, dialogues and singing – yes indeed, thank you, laura newton miller! the opening keynote was especially thoughtful, with denise koufogiannakis enlarging our understanding of what we mean by evidence and how we use it as we work together. i think about that a lot in my own work.   there were many assessment librarians in attendance and i wandered around musing aloud about the relationship between assessment and ebl. lorie kloda took up that question in her lightning strikes contribution. she gave us a very useful conceptualization of the relationship between the two (no spoiler alert needed here as i will wisely leave it to lorie to elaborate, which i’m sure she will).   well, no great conference experience is really great without community and fun. i was nineteen years old the last time i stayed in residence and have judiciously avoided it ever since (i loved it when i was 19, but as one gets older…). however, my eblip stay in residence turned out to be an unexpected highlight. the university of saskatchewan campus is beautiful and residence was a place to visit and to get to know colleagues from canada and all over the world. i especially loved our breakfast gabs all together in the “caf”.  of course this brings to mind the joyous and delicious banquet on a prairie farm under that prairie sky.   this was my first visit to saskatoon and i will be back. it was fun to get to know the city and to get to know colleagues from around the globe. with maria grant from salford, i roamed the city and perused the galleries after the conference closed. i will certainly travel again to saskatoon and hope for the opportunity to experience another eblip conference.   thank you to all who made it happen, i can hardly wait for the next one!   jon eldredge, associate professor, university of new mexico, usa   to the best of my knowledge, i believe that i am one of only two people who have participated in all seven eblip conferences to date. each eblip conference has showcased its own local attractions and has provided a window into regional practices.   each eblip conference, beginning with the first in sheffield, has offered its own memorable elements. sheffield during 2001 featured the promise of a dramatic evidence-based revolution within our profession. edmonton in 2003 opened the movement beyond the boundaries of the health sciences, where it had derived much of its initial inspiration. edmonton brought in academic, school, and public library representatives.   brisbane during 2005 suggested the codification of eblip and announced the birth of this peer reviewed journal. then north carolina brought many us colleagues on board to our international movement during 2007. north carolina also featured the perspectives of research faculty members charged with training future members of our profession.   stockholm offered a distinct turning point of eblip toward a genuine international movement rather as primarily an english-language-centered series of conferences. the nordic countries demonstrated their sophisticated perspectives on eblip at this 2009 conference, and contributed a vision of integrating eblip as a team-centered pursuit into mainstream practice.   eblip6 brought this biennial conference to salford near manchester, not far from the first-ever conference hosted in sheffield. the conference was held during 2011 on the salford university campus with hotel accommodations distributed around manchester. the program emphasized an applied research focus and multiple perspectives on the future of eblip. attendees enthusiastically cheered on the poster madness innovation.   how shall eblip7 during july 15-18, 2013 be remembered? i will recall it in a most unexpected way. i arrived almost two days early to complete a research project and begin a manuscript far away from the distractions of my office. eblip7 offered on-campus lodging in the dorms so i elected to stay in a dorm room instead of a hotel separated by some distance from the on-campus conference venue. others also arrived early to attend pre-conference workshops or to relax away from their offices.   when i left eblip 7 the research project and the manuscript were still in the same incomplete state. instead, i felt invigorated by my interactions with the many other conferees who also had stayed in the dorm and who had eaten at the adjacent dining hall. during our time together on campus we created a community of eblip enthusiasts. we talked on the lawns, while walking around the beautiful university of saskatchewan campus or throughout the city of saskatoon, at late night eateries, more conventional restaurants, in the hallways, and in the dining halls. the conversations almost always were stimulating. some conversations catalyzed me to pursue new directions in eblip. i recall one breakfast dining hall discussion in which no one wanted to stop in order to attend the formal program. the late night hallway conversations about eblip in the dorms examined some underlying assumptions and introduced us to interesting variations of eblip at other types of libraries.   library conferences often take place at large hotel conference centers. in contrast, many specialized academic societies in the usa sponsor their conferences on university campuses with attendees housed in dormitory housing. there is something about communal living in this kind of smaller conference context that many other conferees and i obviously found to be very stimulating. based on the positive experiences at eblip7, i would recommend that all future eblip conferences use campus venues to foster the same kinds of informal discussions and networking. such communal conference living could very well expand and strengthen our international eblip community.   each eblip conference gets better and better. the organizing committee made all of us comfortable and fully entertained throughout eblip7. i find it increasingly difficult to choose between the tantalizing presentations at different concurrent eblip sessions. that trend continued at eblip7. i found the presentation by pors and johannsen about the diffusion of findings from systematic reviews in the mass media, for example, to be an intriguing research direction worth pursuing in other countries. the poster madness program, once again, provided informative and entertaining presentations such as one sung to the theme of “yesterday” and another presented as a cheer exercise.   evidence summary   critical thinking exercises in the classroom are helpful but not sufficient for improving critical thinking test scores   a review of: wallace, e. d., & jefferson, r. n. (2013). developing critical thinking skills for information seeking success. new review of academic librarianship, 19(3): 246-255. http://dx.doi.org/10.1080/13614533.2013.802702   reviewed by: heather coates digital scholarship & data management librarian indiana university-purdue university indianapolis (iupui) university library indianapolis, indiana, united states of america email: hcoates@iupui.edu   received: 7 mar. 2014    accepted: 30 apr. 2014      2014 coates. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine whether a series of workbook exercises contributed to improved critical thinking test scores.   design – post-test design with a quasi-experimental control group.   setting – military college in the united states of america.   subjects – 76 undergraduates enrolled in a required freshman orientation seminar.   methods – approximately one third of the enrolled participants (n=26) were provided with a copy of the book critical thinking: building the basics. a subset of exercises was completed independently over three to four class sessions during the first three weeks of the semester. the control group (n=50) did not receive any critical skills thinking instruction. the icritical thinking skills test, an online exam provided by educational testing service (ets), was administered to both groups during a class session. the exam consists of 7 types of tasks:  define, access, evaluate, manage, integrate, create, communicate, evaluated using 14 tasks based on real-world scenarios.   main results – approximately 20% (15) of all students passed the test, 9 from the intervention group and 6 from the control group. significant differences were detected between the groups on the integrate and manage subtests. the range for individual subtests and total scores was wide. scores for two of the seven subtests, create and evaluate, showed the greatest amount of variability; the communicate subtest scores had the least.   conclusion – limitations of the study include potential motivational differences between the groups. students who completed workbook exercises appeared to be motivated to do well on the test, while those who did not seemed less motivated. the effectiveness of exercises in developing critical thinking skills in this study will persuade administrators to consider using such exercises in the classroom.     commentary   despite their perceived importance for student learning outcomes, critical thinking skills are rarely taught explicitly in the college classroom. in part, this is because few approach an operational definition than can be used to inform instruction and evaluation. those definitions included in a recent systematic review (behar-horenstein & niu, 2011) describe critical thinking as an attitude, application of skills, and a process. similarly, many instructional methods have been used, but are not sufficiently characterized in the literature. this, along with the preponderance of preand quasi-experimental studies using small samples sizes, threatens internal validity, thus limiting applicability to other instructional settings because causal effects for the interventions cannot be asserted.   the study reviewed here examines this relatively understudied area, the role of critical thinking skills in information seeking and use. glynn’s (2006) tool developed for library and information science was used to appraise the article. unfortunately, the article does not include sufficient detail to support a comprehensive appraisal, though it mirrors many of the design weaknesses described above.   the major strengths of the study by wallace and jefferson (2013) are the use of explicit and active learning strategies. however, it is difficult to establish the quality and generalizability of the evidence reported due to incomplete description of the study. critical elements of the study are not described, including the population, sample, and the details of the instructional intervention. without this information, it is difficult to determine the relevance to professional practice.   another area of confusion is which of two testing instruments were used. the authors report using the “icritical thinking skills test” offered by ets. no such test is listed by ets, but they do offer the icritical thinking certification (ets, 2010). both tests include the same seven task areas, but the iskills test was designed and validated to assess information literacy skills. since the authors’ citation does not match the narrative description, it is impossible to determine whether the instrument used was appropriate.   there are further concerns relating to the testing situation regarding validity of the test results. the class period was not long enough to allow for completion of the registration process (15 minutes) and the test (60 minutes). if students perceived that the exercises and/or the test were irrelevant to the course, it is likely that their performance does not reflect their actual abilities. a lack of motivation may be a confounding variable for test performance. furthermore, the accuracy of the anova results for subtest differences is questionable. the reported anova results are inconsistent with the table values, but consistent with the narrative. more generally, the authors do not adequately discuss the implications of the findings, particularly the study limitations and how they may be addressed in future research. overall, the missing and conflicting information presented in this article raise significant concerns as to the validity and applicability of the findings.   despite the methodological concerns, this study contributes to a gap in the literature. given the increasing demand to demonstrate the value of higher education, this is an area ripe for further study. however, future studies should address the design limitations outlined by behar-horenstein and nui (2011), by using carefully design quasi-experimental or experimental studies that combine quantitative and qualitative approaches for measuring change in critical thinking ability.     references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi:10.1108/07378830610692154   behar-horenstein, l. & niu, l. (2011). teaching critical thinking skills in higher education: a review of the literature, journal of college teaching & learning, 8(2), 25-42. retrieved from http://journals.cluteonline.com/index.php/tlc/article/view/3554/3601   educational testing service. (2010). icritical thinking certification. retrieved from http://www.certiport.com/portal/common/documentlibrary/icritical_thinking-datasheet.pdf   what has changed since 2015? a new and expanded update on copyright practices and approaches at canadian post-secondaries research article   what has changed since 2015? a new and expanded update on copyright practices and approaches at canadian post-secondaries   rumi graham university copyright advisor and graduate studies librarian university of lethbridge library lethbridge, alberta, canada email: grahry@uleth.ca   christina winter copyright and scholarly communications librarian university of regina library regina, saskatchewan, canada email: christina.winter@uregina.ca   received: 7 sept. 2020                                                                    accepted: 19 oct. 2021      2021 graham and winter. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30033     abstract   objective – the aim of this study is to update our understanding of how canadian post-secondary institutions address copyright education, management, and policy matters since our last survey conducted in 2015. through the new survey, we seek to shed further light on what is known about post-secondary educational copying and contribute to filling some knowledge gaps such as those identified in the 2017 statutory review of the canadian copyright act.   methods – in early 2020, a survey invitation was sent to the person or office responsible for oversight of copyright matters at member institutions of five canadian regional academic library consortia. the study methods used were largely the same as those employed in our 2015 survey on copyright practices of canadian universities.   results – in 2020, respondents were fewer in number but represented a wider variety of types of post-secondary institutions. in general, responsibility for copyright services and management decisions seemed to be concentrated in the library or copyright office. topics covered and methods used in copyright education remained relatively unchanged, as did issues addressed in copyright policies. areas reflecting some changes included blanket collective licensing, the extent of executive responsibility for copyright, and approaches to copyright education. at most participating institutions, fewer than two staff were involved in copyright services and library licenses were the permissions source most frequently relied on “very often.” few responded to questions on the use of specialized permissions management tools and compliance monitoring.   conclusion – copyright practices and policies at post-secondary institutions will continue to evolve and respond to changes in case law, legislation, pedagogical approaches, and students’ learning needs. the recent supreme court of canada ruling on approved copying tariffs and fair dealing provides some clarity to educational institutions regarding options for managing copyright obligations and reaffirms the importance of user’s rights in maintaining a proper balance between public and private interests in canadian copyright law.     introduction   this study builds on a national survey undertaken in 2015 to probe how canadian universities managed their copyright practices in light of major shifts that had occurred in the copyright sphere (graham & winter, 2017). the 2015 study updated a study by horava (2010) that explored copyright communication in canadian university libraries. we wanted to understand how major developments in canadian copyright since 2015 have impacted post-secondary copyright practices. to that end, in early 2020 we conducted a follow-up survey. the new survey was distributed to an expanded pool of potential participants to seek current information on copyright practices across a wider variety of types of institutions beyond just universities.   since 2015, we have witnessed several significant developments in canada’s educational copying landscape. four of these developments are legal proceedings involving post-secondary institutions, one being the outcome of the class action sought by copibec against université laval after the university exited its blanket license in 2014 (copibec, 2014). the other three are court rulings arising from the 2013 access copyright (ac) lawsuit alleging that fair dealing guidelines adopted by york university authorize and encourage unlawful copying and that the university must operate within the approved interim tariff (access copyright, 2013).   three further developments are the 2017 statutory review of the copyright act, the 2018 signing of a new north american free trade agreement requiring canada to extend its term of copyright, and copyright board approval in 2019 of two ac post-secondary tariffs. for an overview of each of these developments in terms of their relevance to copying practices of post-secondary institutions, see appendix 1.   literature review   copyright practices of post-secondary institutions have been the subject of several studies published since 2015. zerkee (2017) surveyed copyright administrators at canadian universities about their copyright education programs for faculty. most respondents said they offered some type of copyright education for instructors, but few conducted formal evaluations of their effectiveness. patterson (2017) interviewed canadian university staff holding copyright positions about their backgrounds and duties. most interviewees were librarians and few had formal legal or copyright training. a notable finding was that almost half of the interviewees said they did not know which office or individual was the final authority on their institution’s copyright decisions. patterson advocated for faculty status and academic freedom for copyright staff as their work may involve questioning or critiquing administrative decisions (2017, p. 8).    di valentino (2016) examined copyright practices of canadian universities through a content analysis of university copyright policies, guidelines, and publicly accessible copyright web pages and by surveying teaching faculty about their copyright practices and awareness of copyright policies and training. findings supported the thesis that while canadian statutory and case law provide educational institutions with sufficient grounds for unauthorized but lawful uses of copyrighted works, university copyright policies tend to be overly restrictive and risk averse. di valentino warned that although a blanket copying license might be less expensive, “universities must ask themselves about the implications of asking for permission where none is needed, or ‘agreeing’ to licence terms that claim posting a link is copying” (2016, p. 184).   beyond canada, lewin-lane et al. (2018) conducted a literature review and environmental scan to learn about copyright services offered by u.s. higher education institutions and their libraries. noting a high degree of variability in their findings, the researchers concluded that clear patterns in copyright service models have yet to emerge in u.s. academic libraries. they suggested it would be useful to develop a centralized repository of copyright best practices. secker et al. (2019) used findings from a multi-national european panel discussion on copyright literacy levels of copyright librarians to reflect on underlying rationales for copyright education. recognizing that many librarians lack confidence in their knowledge of copyright law, the researchers proposed that library associations take a lead role in offering copyright education programs for their members and outlined a five-part framework for critical copyright literacy. fernández-molina et al. (2020) examined the websites of 24 high-ranking universities with copyright offices in the u.s., canada, australia, netherlands, and the u.k. to gather information about services offered and copyright staff profiles. they found that services offered by copyright offices had expanded beyond guidance on using copyrighted materials for teaching purposes to address scholarly communications topics and services that included author rights and publication agreements. the study identified fair dealing/fair use and other infringement exceptions to be among the most important user-related topics addressed by the copyright offices examined. the researchers concluded that “the needs of professors, researchers and students are nonetheless similar in all countries. in other words, copyright and scholarly communication offices are a dire necessity worldwide” (2020, p. 11).   methods   our 2020 survey employed methods and questions that were similar to those used in our 2015 survey (graham & winter, 2017). both authors obtained ethics approval for the 2020 survey protocols from their respective institutions. the survey questions were developed and pre-tested in english and were then translated into french. the survey was created and managed on the qualtrics platform (https://www.qualtrics.com/). participants were offered the option of responding in english or french and after the survey closed, textual responses submitted in french were translated into english for data analysis purposes. a professional translator completed all needed translations to and from french.   one methodological difference introduced in the 2020 survey was wider survey distribution. as we desired a more inclusive picture of how copyright is managed across canadian post-secondaries, we expanded the pool of potential respondents by including not only universities, but colleges and institutes as well. this was an area for further research identified in our 2015 study (graham & winter, 2017). all universities that received a 2015 survey invitation were again invited to complete the 2020 survey. these institutions comprised all members of caul (council of atlantic university libraries), bci (bureau de coopération interuniversitaire), and ocul (ontario council of university libraries), as well as all university (full) members of coppul (council of prairie and pacific university libraries). in addition, we extended an invitation to participate in the 2020 survey to affiliate members of coppul, most of which are colleges and institutes, and to members of clo (college libraries ontario). a total of 119 institutions were invited to participate in the 2020 survey.   another methodological adjustment we made in 2020 was to send survey invitations directly to the copyright office or copyright specialist instead of the university librarian or library director for each institution included in our study. in most cases we were able to locate contact information for the copyright office or copyright staff through institutional websites or staff directories and in the few instances where this approach was unsuccessful, the invitation was directed to the head of the institution’s library. just as we did in the 2015 survey, 2020 invitation recipients were asked to forward the invitation to another employee at their institution if that individual was better suited to respond. no individual received more than one survey invitation.[1]   although the 2020 survey questions (see appendix 2) were predominantly the same as those used in our 2015 survey (see appendix 3), we made some minor changes and added some new questions. the new questions probed the size of each institution’s copyright staffing complement, whether specialized software is used to manage permissions and licensing, the extent to which permission sources such as licensing or statutory provisions are relied on for copyright clearance work, how institutions cover transactional licensing costs, and whether a formal process for monitoring copyright compliance in the institutional learning management system (lms) has been implemented.   the 2020 survey opened in mid-january 2020 and remained open for one month. about one week prior to the closing date, a reminder notice was sent to invited participants who had not responded. a total of 54 responses were received, of which 39 represented finished surveys. for the purposes of this study we analyzed only the 39 finished surveys.   results   responding institutions   the 39 finished responses to the 2020 survey represent a 19% drop from the 48 responses received to the 2015 survey despite the larger pool of potential participants in 2020. table 1 summarizes the 2020 survey response rates by consortium. participation was highest within coppul (about 46%), followed by caul (about 39%). the 33% overall participation rate for the 2020 survey is about 46% lower than the 61% participation rate obtained in our 2015 survey (graham & winter, 2017). compared to 2015 survey results, in 2020 the number of responses from caul and coppul institutions remained at similar levels, but significantly fewer responses were received from bci and ocul member institutions.   in terms of institutional size (full-time equivalent, or fte, students), the numbers of 2020 survey respondents from large and small institutions were noticeably lower than those obtained in 2015, while the number of 2020 responses from medium-sized institutions was somewhat higher than that of 2015.   regarding the position title of survey respondents, the 2020 survey results suggest that between 2015 and 2020 the locus of responsibility for copyright-related services, activities, and decisions shifted away from executive and second-tier executive positions, with a counterbalancing shift toward position titles containing the term “copyright” (figure 1). of the three responses in 2020 that were other than executive, second-tier executive, or copyright positions, two indicated that no single position or unit was solely responsible for copyright and the third said the responsible position was held by a library technician.   table 1 survey respondents by consortium, 2020   member/affiliate libraries 2020 respondents response rate caul 18 7 39% bci 18 2 11% ocul 21 5 24% clo 23 7 30% coppul 37 17 46% no response   1   total/average 117 39 33%   figure 1 survey respondents by position title, 2020 and 2015.   figure 2 responsibility for permissions clearance, 2020 (n=39).   responsibility for copyright   copyright education   in 2020, responsibility for education in the use of copyrighted works was concentrated in two campus units – the library or copyright office – whereas responsibility for this activity in 2015 was dispersed over a wider array of units that included central administration and a copyright committee. about 54% of 2020 survey participants said copyright user education was the responsibility of the copyright office alone or in a shared capacity and 38% said it was the responsibility of the library alone or shared. these two units together accounted for 92% of responding institutions. the remaining 8% of 2020 participants did not respond to this question.    we also found differences in the distribution of responsibility for copyright education for authors and other creators who are often the first owners of copyright in their works. compared to the 2015 results, a greater proportion of 2020 respondents (about 41%) said the locus of responsibility for copyright owner education was the copyright office and a somewhat smaller proportion (about 36%) said responsibility lay in the library, in both cases acting alone or in a shared capacity with other units. but the overall picture remained constant: between 77% and 79% of respondents to both surveys said responsibility for copyright education for creators rested with the copyright office or the library, each acting alone or in partnership with other units.   permissions clearance   on the whole, the locus of responsibility for permissions clearance in 2020 was similar to the 2015 survey findings for materials distributed via the institutional lms, e-reserve (electronic reserve), reserve (print reserve), and coursepack (figure 2). at the same time, some unique 2020 responses prompted us to refine the permission categories. we split bookstore/copyshop into two categories – campus bookstore and external commercial entity – and converted the faculty category to faculty alone/assisted. we also split not applicable/no response into separate categories to permit more detailed comparisons.   delivery mode-specific comparisons of 2015 and 2020 responses regarding permissions responsibility are summarized in figures 3 to 6. the 2015 survey results indicated the units most often responsible for clearing lms, e-reserve and coursepack permissions were the library, alone or shared, followed by the copyright office, alone or shared, but this pattern was reversed in the 2020 survey results. in 2020, the library and copyright office, alone or shared, were collectively responsible for 60% or more of the permissions work for lms and reserve materials. they were also collectively responsible for about 50% of permissions work for e-reserve and coursepack materials. the proportion of respondents who did not answer the permission responsibility questions more than doubled in 2020.   the library’s lead role in reserve permissions work in the 2015 survey results remained evident in the 2020 results, albeit considerably attenuated. another finding observed in the 2015 and 2020 survey results is that e-reserve was inapplicable or not offered at more than 30% of responding institutions. interestingly, no 2020 survey respondent said that coursepack permissions clearance work was inapplicable at their institution. permissions work for coursepack material is the only category that involved significant levels of participation from entities or campus units outside of the library and copyright office (figure 6). from 2015 to 2020, the level of campus bookstore involvement in coursepack permissions remained constant while the involvement of external entities increased.   figure 3 responsibility for lms permissions clearance, 2020 and 2015.   figure 4 responsibility for e-reserve permissions clearance, 2020 and 2015.   figure 5 responsibility for reserve permissions clearance, 2020 and 2015.   figure 6 responsibility for coursepack permissions clearance, 2020 and 2015.   blanket licensing decisions   between 2015 and 2020, responses to the question about responsibility for blanket licensing decisions shifted significantly (figure 7). from 2015 to 2020, the proportion of respondents who said central administration was responsible for blanket licensing decisions fell from 50% to 31%. this shift was counterbalanced by an increase from 39% to 59% in the proportion of 2020 respondents who identified the library or copyright office as the responsible unit. in addition, one 2020 respondent indicated that an external entity was responsible for their institution’s blanket licensing decisions.   copyright policies   shifts also took place between 2015 and 2020 in the locus of responsibility for policies governing the use of copyrighted materials. while the library was, by far, the most frequently identified as the responsible unit for copyright user policies in 2015, by 2020 the locus of responsibility for copyright user education was most frequently identified to be the copyright office, followed closely by the library. central administration was not far behind, with all three units acting alone or in a shared capacity.   in 2020, central administration was also less frequently involved in policies governing copyright ownership. the 2020 survey responses revealed a three-way tie for this area of copyright responsibility among the copyright office, library, and central administration, alone or shared. as well, the proportion of respondents who said copyright policies for authors and other copyright owners was the responsibility of the research office or faculty association in each case increased slightly from 2015 to 2020.   figure 7 responsibility for blanket licensing decisions, 2020 and 2015.   copyright staffing   the 2020 survey asked respondents to estimate the number of staff involved in providing copyright assistance or services at their institution, a question that did not appear in the 2015 survey. we received 37 responses providing a numeric estimate. as might be expected, the responses were quite varied, ranging from 0.1 to 15 staff positions involved in copyright work at a single institution. the mean average was just over 3.5 staff, the median was 2, and the mode was 1. by far the most common response (about 43%) among responding institutions was fewer than 2 staff (figure 8).   copyright education   topics and methods   in general, education for users of copyrighted works appeared to remain more or less unchanged across the two surveys in terms of topics addressed and educational approaches. in 2015 and 2020, webpages and information literacy sessions were the most frequently used methods to deliver copyright education. the topics and issues most often addressed in user education remained exceptions to infringement such as fair dealing and requests for individual assistance.   a slight shift was observed between 2015 and 2020 in the topics most often addressed in education aimed at creators of copyrighted works. of 43 responses received in 2015, owner/creator rights (53.5%) was identified most often, followed by negotiating publisher contracts or addenda and open access (37.2%). in the 2020 survey, those 2 topics were reversed—of 28 responses received, negotiating publisher contracts or addenda was the most frequently identified topic of copyright education for creators (46.4%) followed by owner/creator rights (25%).   figure 8 number of staff responsible for copyright work at responding institutions, 2020.   changes in copyright education   in the 2020 survey, of 35 responses received, slightly over half (51%) indicated that significant changes had occurred in the way their institution addressed copyright education within the previous 5 years. some changes responded to shifts in areas of interest within the institutional community (e.g., “faculty are more interested in the alternatives such as oer, library licensed e-resources and e-reserves. also, some faculty are interested in copyright education and resources for their students”) while other changes responded to an exit from blanket licensing (e.g., “our institution increased its copyright education program significantly to coincide with the end of the institutional licence with access copyright”). several respondents noted that changes stemmed from greater integration of copyright staff involvement in library and institutional activities (e.g., “new people in the position have become more involved in planning processes and faculty meetings, greater involvement in oers and electronic reserves”).   respondents were also asked about ways in which their copyright education efforts could be enhanced. among the 33 responses, themes that arose most frequently included staff and resources (including time), outreach, and increased efforts directed at copyright online education.   copyright policy   policy adoption and issues covered   in the 2020 survey, 92% of the 39 responding institutions indicated they had a copyright policy or copyright guidelines and only 1 respondent said their institution had neither. over 87% of respondents noted that their copyright policies or copyright guidelines were publicly available.   the most commons topics addressed in copyright policies of responding institutions included fair dealing (51.4%), copyright guidelines (37.1%), and copyright policy (22.8%). respondents were asked to identify the policy date of establishment and, if applicable, the most recent revision date (table 2). almost half of the respondents who provided a copyright policy establishment date said their institution’s policy had been reviewed or was currently under review.   six respondents indicated the areas in which policy revision were made. at two responding institutions, major revisions had been made in the areas of policy scope. at the other four responding institutions, revisions were either minor or they addressed one or more of the following topics: fair dealing, name changes, and inclusion of students.   table 2 copyright policy year of establishment and last revision, 2020 time period policy established: frequency of response (n=30) policy last revised: frequency of response (n=16) 1990-1999 12.8%   2000-2009 7.7% 2.6% 2010-2020 56.4% 33.3% under review   5.1% not applicable 2.6%   no response 20.5% 59%   possible enhancements   the 2020 survey asked respondents to identify ways their institutional copyright policies could be enhanced. this question received 26 responses but there were no clearly discernable patterns among the themes or topics mentioned. between three and four respondents mentioned potential policy enhancements in one or more of the following areas: general usefulness, impact, clarity, faculty engagement, education or support, and more visibility and promotion.   blanket licensing   more than three-quarters of 37 respondents to the 2020 survey indicated that their institutions were operating outside of a blanket licensing environment, with the remaining respondents indicating their institutions were covered by a blanket license with access copyright or copibec. this finding represents a significant change, as just over half of 2015 survey respondents said they were operating under a blanket license.   in the 2020 survey, of the eight institutions that reported having a blanket license, five were from ontario (clo) and (ocul), two were from quebec (bci), and one was from western canada (coppul). a cross-tabulation of blanket licensing with institutional size suggests the likelihood of institutional reliance on blanket licensing decreases as institutional size increases. six of the eight respondents at institutions with a blanket license provided the date on which their access copyright or copibec license was initiated. these dates spanned 2000 to 2019 and each date was unique.   copyright permissions and licensing   assessment of applicable library licenses   in 2015 and 2020, we asked respondents to indicate whether their institution typically checked for the existence of an applicable library license as a part of permissions clearance work. this question asked about permissions work for five categories of materials: coursepacks produced in-house, coursepacks produced by copyshops, materials on reserve, materials on e-reserve, and materials distributed via the lms. on the whole, the response patterns in both surveys were comparable (figure 9). in 2015 and 2020, more than half of survey respondents said library licenses were checked as part of permissions work for readings made available via in-house coursepacks, the institutional lms, and reserves.   there are also some differences between the 2015 and 2020 responses. the proportion of respondents who indicated that library licenses were checked for reserve readings fell from 71% in 2015 to 56% in 2020, but at the same time the proportion who said doing so was not applicable more than doubled from 2015 (4.2%) to 2020 (10.3%). there was also a downward shift from 58% in 2015 to about 44% in 2020 in the proportion of responses indicating library licenses were checked for e-reserve readings paired with a small upward shift from 27.1% in 2015 to 33.3% in 2020 in the proportion who said doing so was not applicable.   managing permissions and licensing   a new question in the 2020 survey asked if a respondent’s institution used a software application or platform to manage copyright permissions and licensing. of the 37 responses received, 70% indicated no management software or platform was in use and 27% responded affirmatively. the remaining 3% of responses indicated that plans were in place to begin using licensed software in the near future.   respondents who indicated their institution uses a software application to manage permissions and licensing were asked to name the adopted system. of the nine responses received, several indicated multiple tools were in use (figure 10).   figure 9 assessment of applicability of library licenses during permissions work, 2020 and 2015.   figure 10 application or platform used for permissions or licensing management, 2020 (n=9).   permission sources   another question unique to the 2020 survey asked respondents how often their institution relied on particular sources for permissions clearance in a 12-month period. of sources relied on “very often,” the top three were library licensed databases (so identified by about 78% of respondents), followed by fair dealing (46%) and users’ rights for educational institutions (31%) (figure 11).   figure 11 permission sources relied on, sorted by “very often,” 2020, n=39   when responses to the same question were sorted by sources relied on “often,” the top three sources were open licensing (44%), user’s rights for educational institutions under the copyright act (39%) and fair dealing under the copyright act (32%). in addition, more than two-thirds of respondents indicated that a blanket institutional copying license was “never” relied on for permissions and more than 30% of respondents said their institution “rarely” relied on permission granted without payable fees and on transactional licenses for business cases.   transactional licensing costs   for the first time in the 2020 survey, we asked respondents how their institutions cover transactional licensing fees for reproduction or distribution of excerpts such as book chapters as well as business cases (figure 12). they were invited to choose as many of the five listed options as were applicable at their institution. a total of 37 respondents answered the questions on licensing costs by making 61 response selections for book chapters and 54 response selections for business cases.   for book chapters, the most frequent response was coverage by a centralized fund (33%) followed by indirect cost recovery via fees charged to students by the bookstore (25%). in contrast, the most frequent response for business cases were fees charged directly to students (24%) followed by “other” (22%).   explanations offered for a response of “other” to the question regarding business cases included uncertainty about how fees are covered and indications that such fees are covered by the business school. other explanations described situation-dependent approaches. an example provided by one respondent is that students pay permission fee costs directly unless a case is used in a seminar, in which case the business school covers the fees.   respondents provided fewer explanations when they chose “other” in response to the question about licensing fee coverage for excerpts such as book chapters. in general, however, the explanations ranged from indications of case-by-case decisions to statements that the requesting department or unit is responsible for permissions and licensing fees.   copyright compliance in the lms   the majority (75%) of the 36 respondents who answered the compliance monitoring question indicated their institution does not have a regularly conducted process for monitoring copyright compliance in the lms. a few (8%) responded affirmatively and several comments that accompanied responses of “other” (17%) mentioned the availability of informal or on-request review processes.   three survey participants responded to questions about their institution’s process of monitoring compliance in the lms. they indicated it takes place collaboratively with the library or copyright office and one or more of the following additional campus units: information technology, legal counsel, copyright committee, faculty association. all three indicated monitoring compliance is conducted by the copyright office or copyright coordinator and involves between 5% and 50% of the work hours of the responsible office or position.   regarding the extent to which faculty are involved in ensuring compliance in the lms, one respondent said their institution asks faculty to provide copyright staff with course materials not posted on the lms along with any permissions obtained by faculty. another respondent said random surveys are employed, and the third indicated that instructors are responsible for ensuring copyright compliance of materials used in the lms. as for the usual process for monitoring lms copyright compliance, all three responses indicated a random sampling approach is used.   participants who said no lms compliance monitoring was currently in place were asked if their institution had plans to implement a formal monitoring process in the near future. of the 27 responses received, 52% said no, 11% said yes, and 37% chose “other.” of those who chose “other,” about half indicated that development of a monitoring process is a potential option or is under discussion.   additional comments   in response to the 2020 survey’s concluding invitation to provide any further comments on copyright compliance or copyright management in general, the following topics were mentioned:   ·         promotion of links and suggestions for e-book purchases in lieu of reproduction or distribution of protected works ·         intention to promote oer materials more strongly ·         existence of, or plans to introduce, a formal statement of responsibility regarding copyright compliance in the lms ·         desire for more staff to assist with copyright matters or development of a more regularized, proactive approach   discussion   the indu report on the first statutory review of canada’s copyright act expressed concern that “despite the volume and diversity of evidence submitted throughout the review, the committee observed a problematic lack of authoritative and impartial data and analysis on major issues” (canada, house of commons, standing commmittee on industry, science and technology, 2019, p. 25). as one way of alleviating the problem of unreliable data, recommendation 4 of the report suggested that statistics canada be mandated to collect authoritative data on the economic impacts of copyright law on canadian creators and creative industries (2019, p. 25).   while our 2020 survey results may not align with the kinds of systematically and comprehensively gathered data the indu committee had in mind, our aim was to shed light on current day-to-day copyright practices of canadian post-secondary institutions and what may have changed in those practices over the past five years. in this section we consider the extent to which the results of our survey may alleviate some data gaps by providing updated information on what actually happens within the realm of educational copying on higher education campuses across canada as reported by staff responsible for copyright at their institutions.   survey participation   by the mid-february closing date, our 2020 survey had achieved a 33% participation rate, representing just over half of the 61% participation rate attained in the 2015 survey. about one month later, closure of canadian post-secondary institutions commenced in what would become a more than year-long period of delivering almost all classes online due to the covid-19 pandemic (e.g., canadian press, 2020; small, 2020). had the pandemic closures and disruptions not occurred, we would likely have considered seeking research ethics approval to reopen the survey later that year to encourage more survey completions. this is a step we took in our 2015 survey due to a similar low response rate on the initial closing date (graham & winter, 2017).   the switch from in-person to online delivery of classes for the latter half of the winter 2020/spring 2021 term precipitated by covid-19 closures raised many copyright-related issues needing to be addressed quickly. our copyright roles at our respective institutions meant we had little time to devote our survey research until fall 2020, when high levels of stress and uncertainty continued to permeate the daily life of post-secondary students and staff, creating an inauspicious environment in which to seek additional responses by reopening the survey. thus, due to a significantly lower response rate, the 2020 results may not be as representative of copyright practices across canadian post-secondary institutions as the results of the 2015 survey.   we were successful in achieving participation from a wider mix of types of institutions than those included in the 2015 survey, however. all five academic library consortia included in the distribution of the 2020 survey were represented in the survey results, which included seven respondents from clo, the only consortium composed entirely of colleges and institutes. as the member institutions of the other four library consortia included in the 2020 survey were exclusively or predominantly universities, the majority of 2020 respondents were from canadian universities, as was the case in the 2015 survey.   areas of continuity   for the 2020 survey we retained most of the questions included in the 2015 survey, which allowed us to compare the results of the two surveys to look for patterns suggesting stability or change. copyright education topics and methods, copyright policy, and permissions licensing are three areas of practice in which we perceived significant levels of continuity to be evident between 2015 and 2020.   education topics and methods   the results of both surveys suggest that the methods used to provide copyright education and the topics covered in copyright education for users remained, on the whole, quite similar. fair dealing and other statutory user rights remained the most frequently addressed topics. a slight change occurred in the topics covered in copyright education for creators, with the ordinal position of the 2015 top two topics ending up reversed in 2020. the top two topics of copyright creator education in 2020 were negotiating publisher contracts or addenda, followed by owner/creator rights. these findings on the topics most frequently covered in copyright education for users and creators align with the results of the study by fernández-molina et al. (2020) on copyright services and staffing at a selection of international universities.   copyright policy   the existence of institutional copyright guidelines or policy was confirmed by just over 80% of the 2015 respondents, while the confirmation level rose to above 90% among 2020 respondents. this finding suggests a strengthened commitment to upholding the rights granted under the copyright act throughout the post-secondary sector. a single 2020 survey respondent said their institution did not have a copyright policy, compared with eight participants in the 2015 survey who responded to this question in the negative.   permissions and content delivery modes   within the 2015 and 2020 survey results, we noticed similar patterns in the locus of responsibility for copyright permissions and licensing work which continued to be performed mainly by library and copyright office staff. in 2020, the copyright office played the lead role in clearing permissions for three of the four content delivery modes explored—the lms, e-reserve, and coursepacks. the one exception was the library’s lead role in permissions work for materials placed on reserve.   when respondents answered “not applicable” to any of the questions on responsibility for permissions clearance for a particular content delivery mode, we took this to be an indication that the institution most likely did not use that delivery mode. with this understanding in mind, comparisons of responses from both surveys suggest that from 2015 to 2020, the availability of e-reserve as a delivery mode option declined slightly and institutional reliance on content delivery via an lms increased slightly. in fact, it is possible that by 2020, all survey participants were using an lms, as no respondent said permissions clearance for materials distributed via the lms was “not applicable.”   similar response patterns to the questions probing whether library licenses are considered during permissions work suggest that library licensing continues to play a key role in institutional management of copyright. for content delivered via coursepacks produced in-house, lms, and reserve, responses to both surveys indicated the applicability of library licenses was checked at more than half of participating institutions. for e-reserve content, the proportion who indicated permissions work included consideration of library licensing dipped below half of the respondents in 2020, but may be due, at least in part, to the slight increase in the proportion of institutions that we infer did not offer e-reserve service.   areas of change   changes in some areas of post-secondary copyright practices appear to have evolved between 2015 and 2020, in a few cases perhaps reflecting key events that unfolded within this timeframe. the locus of these observed shifts in copyright practices occurred in the extent to which responsibility for copyright was held by senior administrative positions, the prevalence of blanket collective licensing, and approaches to copyright education.   executive responsibility for copyright   between 2015 and 2020, the level of central administration involvement in copyright matters declined even further. in response to the question about the institutional office or position title responsible for copyright, the proportion of 2020 respondents who named an executive or second-tier executive position fell by about 9% with a corresponding increase in the proportion naming an office or position that included the term “copyright”. this finding extends a trend first observed in our 2015 survey, which, in turn, followed up on a 2008 survey by horava (2010).   in the area of policies pertaining to copyright owners, central administration held this responsibility in 2015 at about one-third of responding institutions but by 2020, central administration’s role was significantly diminished, as it was named as the responsible unit at less than one-fifth of institutions. similarly, central administration was named as the unit responsible for blanket licensing decisions by half of 2015 respondents, but this fell to fewer than one-third of respondents to the 2020 survey.   in most cases, the shift away from executive responsibility for copyright was counterbalanced chiefly by responsibility more frequently held by copyright or library staff. for example, by 2020, the library was the unit most often named as holding responsibility for blanket licensing, and the library or copyright office was named as the responsible unit for blanket licensing by about three-fifths of participating institutions. this finding suggests that over the period spanning 2015 to 2020, significant specialization and maturation of copyright expertise and knowledge has taken place within the library or copyright office across canadian post-secondary institutions.   approaches to copyright education   although the topics covered and methods used in copyright education for users and creators remained similar in 2015 and 2020, more than half of the respondents to the 2020 survey said significant changes had occurred in how their institution addressed copyright education. written responses explaining those changes mentioned growing institutional awareness of alternatives to traditional commercially published works such as oers, increased focus on using institutional site licenses to online content, adaptation to operating outside of blanket collective licensing, and greater integration of copyright staff in institution-wide concerns such as scholarly communication.   participation in blanket collective licensing   in april and may, 2012, new model blanket copying licenses were successfully negotiated by ac and two associations representing canadian universities, colleges, and institutes at a time of great uncertainty about how the scc would rule in the fair dealing case involving ac and k-12 schools outside of quebec (access copyright, 2012a, 2012b). as things turned out, the scc judgment was released in july 2012 shortly after a number of institutions had signed up for the ac blanket license (alberta (education) v access copyright, 2012). the 2015 survey revealed that just over half of responding institutions held an ac blanket license that would expire at the end of 2015.   in contrast, more than three-quarters of institutions participating in the 2020 survey indicated they did not hold a blanket license. this strongly suggests that by 2020, most post-secondary institutions outside of quebec did not find sufficient value in blanket collective licensing. confirmation from the scc that copyright board-approved tariffs do not bind institutions to pay tariff fees if they use even a single work within a collective society’s repertoire (york university v access copyright, 2021) means institutions remain free to determine how best to ensure that their educational copying complies with canadian copyright law, which may or may not involve blanket licensing.   new areas explored   we introduced some additional questions in the 2020 survey that looked at three broad areas: copyright staffing, management of permissions clearance processes, and compliance monitoring in the lms. information gleaned in these areas help to enrich our understanding of current institutional copyright management practices and operations.   copyright staffing   the 2020 survey responses indicated that although the number of staff having copyright responsibilities at canadian post-secondaries varied widely, the largest proportion of institutions had fewer than two staff who were responsible for copyright services and the median number of copyright staff was two. institutional size was not always a predictor of the number of copyright staff, however, as one large institution (> 25,000 fte) reported having only a single person responsible for copyright while at the opposite end of the spectrum, one small institution (< 10,000 fte) said 10 staff were involved in providing copyright services.   managing permissions clearance   as we received only nine responses to the question that asked if software applications or platforms are used in permissions work, the extent to which post-secondaries have adopted tools specifically designed for this activity remains uncertain. the responses we did receive suggest that the tools for managing various aspects of permissions work in use in 2020 included a mix of commercially available permissions management tools, locally developed tools, and common office productivity applications.   library licenses were, by far, the permissions source most frequently relied on “very often” by more than three-quarters of survey participants. this finding corroborates what many post-secondary institutions and research organizations told the indu committee via witness testimony and submitted briefs during the 2017 copyright act review regarding the central importance of institutional site licenses. such site licenses are negotiated directly with rights owners for online access to scholarly content and obviate the need for mediation by copyright collectives (e.g., canadian association of research libraries, 2018; canadian association of university teachers, 2018).   fair dealing was the permissions source next most frequently relied on “very often” by close to half of respondents. the scc decision in the case launched by ac in 2013 provides robust reassurance that fair dealing truly is a user’s right available to all students (york university v access copyright, 2021). as the scc noted in its unanimous judgement,   contrary to the federal court of appeal’s view, in the educational context it is not only the institutional perspective that matters. when teaching staff at a university make copies for their students’ education, they are not “hid[ing] behind the shield of the user’s allowable purpose in order to engage in a separate purpose that tends to make the dealing unfair” (2021, para 102).   largely on the basis of this principle, the scc drew the following conclusion:   it was therefore an error for the court of appeal, in addressing the purpose of the dealing, to hold that it is only the “institution’s perspective that matters” and that york’s financial purpose was a “clear indication of unfairness” . . . . funds “saved” by proper exercise of the fair dealing right go to the university’s core objective of education, not to some ulterior commercial purpose (2021, para 103).   in light of the pivotal 2021 scc ruling and the fact that the period of ac blanket licensing within the public education sector spanning the 1990s and 2000s was founded on an agreement to disagree about the scope and applicability of fair dealing to educational copying (graham, 2016, p. 337), perhaps we will see greater use of fair dealing—an “always available” user’s right (cch v lsuc, 2004, para 49)—as a statutory source of permissions clearance by post-secondary institutions in the future. after all, a mere one-tenth of 2020 respondents said they relied “very often” on blanket collective licensing in permissions work.   another permissions-related issue included for the first time in the 2020 survey asked how transactional licensing costs are covered for excerpts such as book chapters and business cases. we included separate questions about these two kinds of course materials because while many book publications fall within the repertoire of works for which ac offers blanket licensing, as far as we are aware, commercially published business cases from organizations such as ivey publishing and harvard business publishing have never been part of any repertoire covered by blanket collective licensing in canada.   the survey results revealed no single dominant means of covering transactional licensing costs. about one-third of the 37 participants who answered these two questions indicated their institution uses multiple methods (respondents were invited to indicate all approaches that applied). the most frequent responses were a centralized fund for book chapters and directly charging permission fees to students for business cases. the finding that institutions use a variety of ways to manage licensing costs underlines a general observation that no single approach (such as blanket collective licensing) is likely to be an effective or efficient way of addressing the copyright compliance and management needs of all canadian post-secondary institutions.   compliance monitoring although only three respondents said their institution had implemented a process for regular monitoring of copyright compliance in the lms, there were commonalities across the responses in terms of how the process is structured, which units are responsible for leading the activity, and the involvement of several other campus units or offices. at institutions currently without a monitoring process, about half of the respondents indicated there were no plans to introduce one in the future.   that a “substantial shift to digital content” (canada, house of commons, standing commmittee on industry, science and technology, 2019, p. 58) has taken place within higher education is noncontroversial. but many witnesses from the canadian publishing sector alleged that post-secondary institutions engage in inappropriately uncompensated “mass and systematic use of their works” (canada, house of commons, standing commmittee on industry, science and technology, 2019, p. 57), while educational institutions “denied claims of rampant copyright infringement” (canada, house of commons, standing commmittee on industry, science and technology, 2019, p. 60), instead arguing that most of their uses of protected works are copyright compliant.   as the lms had become an essential teaching and learning tool even before covid-19 (peters, 2021), it represents the space in which the publishing sector alleges wide-spread copyright infringement uses take place, out of view to all except instructors and their students. the indu committee noted the lack of reliable data makes it “difficult to determine whether the education sector has adopted adequate measures to prevent and discourage copyright infringement” (canada, house of commons, standing commmittee on industry, science and technology, 2019, p. 64).   given the scc’s reconfirmation that proper fair dealing analyses are no longer to be guided by earlier “author-centric” approaches focusing on “the exclusive right of authors and copyright owners to control how their works [are] used in the marketplace” (york university v access copyright, 2021, para 90), the time may now be ripe for canadian post-secondary institutions to review their fair dealing guidelines and policies to ensure they align with the scc’s most recent guidance. institutions may also consider ways in which they can collaborate to assemble “new and authoritative information” (canada, house of commons, standing commmittee on industry, science and technology, 2019, p. 65) about their educational copying that upholds principles of academic freedom, privacy and confidentiality, as well as evidence-based professional practice.   conclusion   copyright continues to be a public policy matter that is in considerable flux in canada and around the world. canadian post-secondary institutions have been acutely aware of, and in some cases, have actively participated in, major events that have unfolded in the copyright sphere in the past five years. some of those events have spotlighted long-held, strongly divergent views on the fundamental purpose of copyright and the scope and appropriate application of educational fair dealing, which, in turn, have had implications for the copyright practices of canadian post-secondary institutions.   the survey we conducted in early 2020 provided an updated understanding of where responsibility for various copyright services and decision-making processes lies across canada’s universities, colleges and institutes. it also further illuminated some specific areas of practice, policy, and management including copyright education, participation in collective blanket licensing, permissions assessment processes, and copyright staffing complements.   our study results suggest that since 2015, continued consolidation has taken place in the locus of copyright expertise on post-secondary campuses. aspects of our new findings were likely influenced not only by developments in the educational copying environment that have been taking shape over the past decade, but also by growing interest in open educational resources and open access to the scholarly literature which have gained prominence over the past five years within the scholarly communications ecosystem.   some limitations identified in our 2015 study remain applicable to the present study. for example, due to the nature of survey-based research, the 2020 survey provides only a snapshot of institutional practices at a single point in time. as well, the extent to which the study has yielded a representative indication of copyright practices across canadian post-secondary institutions is not as strong we had hoped for, due to the lower than desired response rate, which was impacted by the unforeseen arrival of the covid-19 pandemic and subsequent major disruptions.   fruitful areas for future exploration that could potentially build on our research and those of others (e.g., fernández-molina et al., 2020; patterson, 2017) include closer examinations of copyright staff profiles and the nature of shared institutional responsibility for copyright-related services, operations, policies, and decision-making processes. copyright practices of post-secondary institutions will undoubtedly continue to evolve and respond to changes in case law, copyright legislation, and the ways in which information sources needed by students and researchers are created, made available, and used.   we are hopeful that the crucially important scc judgment (york university v access copyright, 2021) confirming the voluntary nature of blanket collective licensing and the court’s prior guidance on the scope and proper assessment of fair dealing in post-secondary settings provides a solid legal foundation on which to refine robust, copyright-compliant practices going forward.   author contributions   rumi graham: conceptualization, formal analysis, writing – review & editing, investigation, methodology, funding acquisition. christina winter: conceptualization, formal analysis, writing – review & editing, investigation, methodology, funding acquisition.   acknowledgements   this study was funded by a 2019 practicing librarian grant from the canadian association of research libraries. the authors again thank tony horava for permission to adapt the survey instrument used in his 2008 study for our 2015 survey, which was largely reused again in the present study.   references   access copyright. (2012a, may 29). access copyright announces agreement with the association of community colleges of canada on a model license [news release]. https://web.archive.org/web/20150607015005/http://www.accesscopyright.ca/media/23761/access_copyright_announces_agreement_with_the_association_of_community_colleges_of_canada_on_a_model_licence.pdf access copyright. (2012b, april 16). access copyright signs model licence with the association of universities and colleges of canada [news release]. https://web.archive.org/web/20150706161636/https://accesscopyright.ca/media/22883/access_copyright_aucc_model_agreement_media_release.pdf access copyright. (2013, april 8). canada's writers and publishers take a stand against damaging interpretations of fair dealing by the education sector [news release]. https://web.archive.org/web/20150905101600/http://accesscopyright.ca/media/35670/2013-04-08_ac_statement.pdf access copyright. (2019, june 5). access copyright statement regarding standing committee on industry, science and technology’s copyright act review report. https://www.accesscopyright.ca/media/announcements/access-copyright-statement-regarding-standing-committee-on-industry-science-and-technology-s-copyright-act-review-report/ alberta (education) v canadian copyright licensing agency (access copyright), 2012 scc 37. https://scc-csc.lexum.com/scc-csc/scc-csc/en/item/9997/index.do bains, n., & joly, m. (2017, december 17). [letter to dan ruimy, chair, standing committee on industry, science and technology, house of commons]. government of canada. https://www.ourcommons.ca/content/committee/421/indu/webdoc/wd9706058/421_indu_reldoc_pdf/indu_deptindustrydeptcanadianheritage_copyrightact-e.pdf berne convention for the protection of literary and artistic works. (1886, september 9).  http://www.wipo.int/edocs/lexdocs/treaties/en/berne/trt_berne_001en.pdf canada, house of commons, standing committee on canadian heritage. (2019). shifting paradigms. https://www.ourcommons.ca/documentviewer/en/42-1/chpc/report-19/ canada, house of commons, standing commmittee on industry, science and technology. (2019). statutory review of the copyright act: report of the standing committee on industry, science  and technology. https://www.ourcommons.ca/content/committee/421/indu/reports/rp10537003/421_indu_rpt16_pdf/421_indu_rpt16-e.pdf canada, united states, & mexico. (2018). canada-united states-mexico agreement. https://www.international.gc.ca/trade-commerce/trade-agreements-accords-commerciaux/agr-acc/cusma-aceum/text-texte/toc-tdm.aspx?lang=eng canadian association of research libraries. (2018, july 4). brief to the indu committee as part of the review of the copyright act. https://www.ourcommons.ca/content/committee/421/indu/brief/br10003061/br-external/canadianassociationofresearchlibraries-e.pdf canadian association of research libraries. (2019, june 11). carl statement on the indu report on the copyright act. canadian association of research libraries. https://www.carl-abrc.ca/wp-content/uploads/2019/06/copyright_review_statement_en.pdf canadian association of university teachers. (2018, july). caut submission to the standing committee on industry, science and technology statutory review of the copyright act. https://www.ourcommons.ca/content/committee/421/indu/brief/br10006790/br-external/canadianassociationofuniversityteachers-e.pdf canadian copyright licensing agency (access copyright) v york university, 2017 fc 669. https://reports.fja-cmf.gc.ca/fja-cmf/j/en/item/331683/index.do canadian council of archives, association des archivistes du québec, & association of canadian archivists. (2019, july). canadian archivists applaud the indu report on the statutory review of the copyright act. canadian council of archives. http://www.archivescanada.ca/uploads/files/news/communique_indureport_july19-2019_en.pdf canadian federation of library associations/fédérations canadienne des associations de bibliothèques. (2019, june 14). canadian libraries commend the parliamentary review of the copyright act. cfla-fcab. http://cfla-fcab.ca/en/advocacy/canadian-libraries-commend-the-parliamentary-review-of-the-copyright-act/ canadian press. (2020, march 18). canadian colleges, universities tell students to vacate dorms amid coronavirus pandemic. global news. https://globalnews.ca/news/6695575/canada-coronavirus-colleges-universities-dorms/  cch canadian ltd v law society of upper canada, 2004 2004 scc 13. https://scc-csc.lexum.com/scc-csc/scc-csc/en/item/2125/index.do copibec. (2014, november 10). $4 million class action lawsuit against université laval for copyright infringement [news release]. http://www.copibec.qc.ca/portals/0/fichiers_pdf_anglais/news%20release-copibec%20novembre%2010%202014.pdf copibec. (2018a, june 19). copibec and université laval reach out-of-court settlement on copyright royalties [news release]. https://www.copibec.ca/en/nouvelle/179/copibec-et-l-universite-laval-concluent-une-entente-hors-cour-en-matiere-de-droits-d-auteurs copibec. (2018b, june 21). settlement agreement with the representatives of an authorized class action. copibec. https://www.copibec.ca/en/settlement-agreement copyright board of canada. (2010). statement of proposed royalties to be collected by access copyright for the reprographic reproduction, in canada, of works in its repertoire: post-secondary educational institutions (2011-2013). https://cb-cda.gc.ca/sites/default/files/inline-files/2009-06-11-1.pdf copyright board of canada. (2013). statement of proposed royalties to be collected by access copyright for the reprographic reproduction, in canada, of works in its repertoire: post-secondary educational institutions (2014-2017). https://cb-cda.gc.ca/sites/default/files/inline-files/supplement_18_may_2013.pdf copyright board of canada. (2019). statement of royalties to be collected by access copyright for the reprographic reproduction, in canada, of works in its repertoire: post-secondary educational institutions (2011-2014), post-secondary educational institutions (2015-2017). https://decisions.cb-cda.gc.ca/cb-cda/decisions/en/item/453965/index.do copyright modernization act, (2012, s.c. 2012, c. 20). https://laws-lois.justice.gc.ca/eng/annualstatutes/2012_20/fulltext.html di valentino, l. (2016). laying the foundation for copyright policy and practice in canadian universities [doctoral dissertation, western university]. scholarship@western. http://ir.lib.uwo.ca/etd/4312 fernández-molina, j.-c., martínez-ávila, d., & silva, e. g. (2020). university copyright/scholarly communication offices: analysis of their services and staff profile. the journal of academic librarianship, 46(2), 102133. https://doi.org/10.1016/j.acalib.2020.102133  flynn, j., giblin, r., & petitjean, f. (2019). what happens when books enter the public domain? testing copyright’s underuse hypothesis across australia, new zealand, the united states and canada. unsw law journal, 42(4), 1215-1253. http://www.unswlawjournal.unsw.edu.au/wp-content/uploads/2019/11/3-flynn-giblin-and-petitjean.pdf  geist, m. (2017, september 14). why copyright term matters: publisher study highlights crucial role of the public domain in ontario schools. michael geist. https://www.michaelgeist.ca/2017/09/copyright-term-matters-publisher-study-highlights-crucial-role-public-domain-ontario-schools/ graham, r. (2016). an evidence-informed picture of course-related copying. college & research libraries, 77(3), 335-358. https://doi.org/10.5860/crl.77.3.335 graham, r., & winter, c. (2017). what happened after the 2012 shift in canadian copyright law? an updated survey on how copyright is managed across canadian universities. evidence based library & information practice, 12(3), 132-155. https://doi.org/10.18438/b8g953  horava, t. (2010). copyright communication in canadian academic libraries: a national survey. canadian journal of information and library science, 34(1), 1-38. https://doi.org/10.1353/ils.0.0002  innovation science and economic development canada. (2021). a consultation on how to implement an extended general term of copyright protection in canada. government of canada. http://www.ic.gc.ca/eic/site/693.nsf/eng/00188.html  lewin-lane, s., dethloff, n., grob, j., townes, a., & lierman, a. (2018). the search for a service model of copyright best practices in academic libraries. journal of copyright in education and librarianship, 2(2), 1-24. https://www.jcel-pub.org/jcel/article/view/6713  music canada. (2019, june 4). music canada statement on the release of the standing committee on industry, science and technology report. music canada. https://musiccanada.com/news/music-canada-statement-on-the-release-of-the-standing-committee-on-industry-science-and-technology-report/ patterson, e. (2017). the canadian university copyright specialist: a cross-canada selfie. partnership: the canadian journal of library and information practice and research, 11(2), 1-9. https://doi.org/10.21083/partnership.v11i2.3856  peters, d. (2021, january 13). learning management systems are more important than ever. university affairs. https://www.universityaffairs.ca/features/feature-article/learning-management-systems-are-more-important-than-ever/  savage, s., & zerkee, j. (2019). language and discourse in the copyright act review. abc copyright annual conference, saskatooon, sk. https://harvest.usask.ca/handle/10388/12166 secker, j., morrison, c., & nilsson, i.-l. (2019). copyright literacy and the role of librarians as educators and advocates: an international symposium. journal of copyright in education & librarianship, 3(2), 1-19. https://doi.org/10.17161/jcel.v3i2.6927  small, k. (2020, march 15). alberta orders all classes cancelled, daycares closed as covid-19 cases rise to 56 in the province. global news. https://globalnews.ca/news/6681118/alberta-covid-19-coronavirus-school-closures-funding/  société québécoise de gestion collective des droits de reproduction v université laval, 2016 qccs 900. https://www.canlii.org/fr/qc/qccs/doc/2016/2016qccs900/2016qccs900.html société québécoise de gestion collective des droits de reproduction v université laval, 2017 qcca 199. https://www.canlii.org/fr/qc/qcca/doc/2017/2017qcca199/2017qcca199.html society of composers, authors and music publishers of canada v. bell canada, 2012 scc 36. http://scc-csc.lexum.com/scc-csc/scc-csc/en/item/9996/index.do supreme court of canada. (2020, october 15). applications for leave: york university, et al. v. canadian copyright licensing agency ("access copyright"), et al. https://decisions.scc-csc.ca/scc-csc/scc-l-csc-a/en/item/18507/index.do york university v access copyright, 2021 scc 32. https://decisions.scc-csc.ca/scc-csc/scc-csc/en/item/18972/index.do york university v canadian copyright licensing agency (access copyright), 2020 fca 77. https://decisions.fca-caf.gc.ca/fca-caf/decisions/en/469654/1/document.do zerkee, j. (2017). approaches to copyright education for faculty in canada. partnership: the canadian journal of library and information practice and research,, 11(2), 1-28. https://doi.org/10.21083/partnership.v11i2.3794    appendix a copyright developments, 2015-2020   copibec v. laval   in 2016, the quebec superior court declined a motion from the société québécoise de gestion collective des droits de reproduction (copibec) to authorize a class action against université laval for alleged copyright infringement while operating outside of a blanket license (société québécoise de gestion collective des droits de reproduction v université laval, 2016). a year later, the quebec court of appeal (qca) authorized copibec’s class action by overturning the lower court’s decision. the qca ruling thus comprised the first step in a class action proceeding (société québécoise de gestion collective des droits de reproduction v université laval, 2017).   the class action did not proceed, however, as the parties settled the disagreement out-of-court (copibec, 2018a). details of the settlement agreement included suspension of université laval’s copyright policy and guidelines, retroactive payment for a copying licence covering 2014-2017, laval’s return to province-wide post-secondary blanket licensing, and additional payments by laval for infringement of moral rights and other fees (copibec, 2018b). this settlement thus marked a return to the status quo throughout the quebec educational copying environment, with all post-secondary institutions operating under a blanket collective license.   ac v. york   canada’s federal court (fc) delivered its decision in the ac and york university (york) case on july 12, 2017 after a 19-day hearing that took place over may and june 2016 (access copyright v york university, 2017). the two key matters at issue were whether it was mandatory for york to operate under the approved interim tariff and whether copying by york under its fair dealing guidelines was lawful under the provisions of the copyright act. the fc decision found in favour of ac on both matters. on the mandatory tariff question, the fc said “the interim tariff is mandatory and enforceable against york. to hold otherwise would be to frustrate the purpose of the tariff scheme of the act . . . and to choose form over substance” (2017, para 7). on york’s fair dealing guidelines, the fc said “york’s own fair dealing guidelines   . . . are not fair in either their terms or their application. the guidelines do not withstand the application of the two-part test laid down by supreme court of canada” (2017, para 14).   york v. ac appeal   both york and ac appealed the 2017 fc decision to the federal court of appeal (fca). following a two-day hearing in march 2019, the fca released its decision on april 22, 2020 (york university v access copyright, 2020). the fca ruled against ac by overturning the fc ruling on the mandatory effect of approved tariffs. the fca’s extensive review of the legislative history of the copyright act’s licensing and tariff regimes and relevant case law led the court to conclude that   the fact that the collective society/tariff regime is a means of regulating licensing schemes which, by definition, are consensual. . . . [and] continuous references to licensing schemes and the retention of the key elements of the 1936 act leave little doubt that tariffs are not mandatory which is to say that collective societies are not entitled to enforce the terms of their approved tariff against non-licensees (2020, para 202).   on the issue of the fairness of york’s fair dealing guidelines, however, the fca ruled against york by upholding the fc finding of unfairness. the fca’s conclusion was that “york has not shown that the federal court erred in law in its understanding of the relevant factors or that it fell into palpable and overriding error in applying them to the facts” (2020, para 312).   york v. ac appeal to the supreme court   a third and final chapter of the tariff/fair dealing dispute between ac and york opened with both parties seeking leave to appeal to the supreme court of canada (scc), which was granted to both parties in october 2020 (supreme court of canada, 2020). following a half-day hearing in may 2021, the scc released its unanimous judgment a mere two months later on july 30, 2021 (york university v access copyright, 2021). on the key question of whether it is mandatory for york and other post-secondary institutions to pay tariff fees if they copy any materials in ac’s repertoire, the scc dismissed ac’s appeal, thereby upholding the fca decision that tariffs are not mandatory.   among other things, the scc noted that   access copyright’s interpretation of s. 68.2(1) is not only unsupported by the purpose of the [copyright] board’s price-setting role, it is, respectfully, also in direct conflict with that purpose. instead of operating as a part of a scheme designed to control collective societies’ potentially unfair market power, access copyright’s interpretation would turn tariffs into a plainly anti-competitive tool, boosting collective societies’ power to the detriment of users (2021, para 71).   the scc declined to grant york’s request for a declaration regarding its fair dealing guidelines because the determination that tariffs are not enforceable on non-licensees meant there was no live issue between the parties. the scc agreed with york, however, that institutional guidelines are an important way to help students actualize their fair dealing rights. the scc further noted that it did not endorse the fair dealing analyses conducted by the fc and fca and offered the following corrections to the reasoning from those courts.   ·         throughout their fairness analyses, the fc and fca incorrectly adhered to an institutional perspective—york’s purported commercial purpose—without proper regard for fair dealing as a component of user’s rights that are integral to copyright’s balance between private and public interests. the scc affirmed that “a proper balance ensures that creators’ rights are recognized, but authorial control is not privileged over the public interest” (2021, para 93). ·         regarding their assessment of the “purpose” fair dealing factor, both courts repeated the same error made by the copyright board when it distinguished between the purposes of teachers and their students, which the scc had corrected in an earlier decision (alberta (education) v access copyright, 2012). the scc reiterated that “the purpose of copying conducted by university teachers for student use is for the student’s education” (2021, para 103). ·         the fc’s assessments of the “amount” factor was incorrect when it criticized york’s guidelines on the basis that they allow different excerpts of a work to be used by different groups of students such that in the aggregate, the whole work could be used. the scc stated the fc should instead have followed the guidance offered on this matter in its earlier ruling (alberta (education) v access copyright, 2012). ·         the fc’s assessment of the “character” factor was also incorrect in that it overlooked the guidance in socan v bell,  (2012) that large-scale, systemic dealings are not inherently unfair. in their 2021 ruling, the scc noted that “the character of the dealing factor must be carefully applied in the university context, where dealings conducted by larger universities on behalf of their students could lead to findings of unfairness when compared to smaller universities. this would be discordant with the nature of fair dealing as a user’s right” (2021, para 105). post-secondary copying tariffs   a development closely related to the ac and york case was the copyright board’s long-awaited certification of two ac post-secondary tariffs in 2019 (copyright board of canada, 2019). ac had filed its first proposed tariff for post-secondary institutions in march 2010 covering the period from 2011-2013 (copyright board of canada, 2010), followed three years later by a second post-secondary tariff proposal for 2014-2017 (copyright board of canada, 2013). the copyright board consolidated its approval process for ac’s first two proposed post-secondary tariffs in july 2015. the copyright board hearing in january 2016 involved ac and only a single individual intervenor, as all other parties and intervenors had withdrawn from the tariff approval proceedings (copyright board of canada, 2019, p. 5).   the royalty rates set by the copyright board for 2011 to 2014 and 2015 to 2017 were based on licenses offered by ac to universities and colleges during these two time periods, which the board used as proxies for the market value of ac’s licenses (2019, p. 2). one issue considered by the board was the “legal landscape respecting the notion of fair dealing as it applies to the education sector” (2019, p. 34). despite ac’s contention that educational institutions had misinterpreted how fair dealing may apply to educational copying, the board deemed it unnecessary to assess the role of fair dealing since the two licenses used as proxies “already incorporate a market-generated allowance for the current instability surrounding the fair dealing landscape” (2019, p. 35).   review of the copyright act   another major development in the canadian educational copying environment was the 2017 statutory review of the copyright act. conducted over 2018 and first half of 2019 by the house of commons standing committee on industry, science and technology (indu), the review was mandated by amendments passed in 2012 (copyright modernization act, 2012). in their mandate letter to the chair of the indu committee, the minister of innovation, science and economic development and minister of canadian heritage acknowledged that “market disruption has often driven copyright reform,” but went on to note, “we respectfully suggest that the copyright act itself might not be the most effective tool to address all of the concerns stemming from recent disruptions” (bains & joly, 2017).   during the statutory review of the act, the indu committee solicited broad stakeholder feedback through a series of nation-wide public consultations, in-person witness presentations, and written briefs. the library and education sectors were actively engaged in this process. through their analyses of written briefs submitted to the indu committee as part of the statutory review, savage and zerkee (2019) found that the post-secondary education sector submitted the highest proportion (42%) of  all submitted briefs.   the review process culminated in june 2019 with the release of the indu report containing 36 recommendations (canada, house of commons, standing commmittee on industry, science and technology, 2019). feedback on the report from educational and cultural stakeholder groups was mostly favorable regarding the extent to which it balanced the needs of creators and users (e.g., canadian association of research libraries, 2019; canadian council of archives et al., 2019; canadian federation of library associations/fédérations canadienne des associations de bibliothèques, 2019). reactions to the report from copyright collectives tended to be less enthusiastic, however (e.g., access copyright, 2019; music canada, 2019).   invited by the indu committee to contribute to the statutory review, in march 2018 the house of commons standing committee on canadian heritage (chpc) commenced a study of thremuneration models for artists and creative industries. the chpc study used an information-gathering process similar to that employed by indu but was more narrowly focused in scope. in june 2019, the chpc committee independently released the results of its investigations, which included 22 recommendations (canada, house of commons, standing committee on canadian heritage, 2019).   in the end, the indu and chpc statutory review reports reached divergent conclusions on a number of issues. for example, the indu committee said it “cannot endorse the proposal to limit educational fair dealing to cases where access to a work is not ‘commercially available,’ as defined under the act” (canada, house of commons, standing commmittee on industry, science and technology, 2019, p. 64), whereas the chpc committee’s recommendation 18 proposes that the “government of canada amend the act to clarify that fair dealing should not apply to educational institutions when the work is commercially available” (canada, house of commons, standing committee on canadian heritage, 2019, p. 43). both reports were presented to parliament, but no actions were undertaken before parliament was dissolved just prior to the 2019 canadian federal election.   copyright term extension   the last development in the copyright realm of particular note was trigged by a fall 2018 agreement reached by canada, the u.s., and mexico to replace the 1994 north american free trade agreement with a new agreement known in canada as cusma (canada et al., 2018). in the new agreement which came into force in july 2020, the chapter dealing with intellectual property rights contains a requirement in section h, article 20.63 stating that:   each party shall provide that in cases in which the term of protection of a work, performance, or phonogram is to be calculated: (a) on the basis of a natural person, the term shall not be less than the life of the author and 70 years after the author’s death” (canada et al., 2018, p. 20-33).   since canada is the only contracting party to cusma whose current copyright laws do not already provide a general term of copyright protection that is at least life plus 70 years, canada must implement a 20-year extension to its current term of copyright—life plus 50 years—which is the minimum term of protection specified in the berne convention (1886). in march 2021, the government of canada released a consultation paper and invited comment on how best to meet its copyright term extension obligations (innovation science and economic development canada, 2021). it is notable that the 2019 copyright act review reports prepared by indu and chpc contain contrasting recommendations on the term of copyright:   indu report recommendation 6: that, in the event that the term of copyright is extended, the government of canada consider amending the copyright act to ensure that copyright in a work cannot be enforced beyond the current term unless the alleged infringement occurred after the registration of the work (canada, house of commons, standing commmittee on industry, science and technology, 2019, p. 4). chpc report recommendation 7: that the government of canada pursue its commitment to implement the extension of copyright from 50 to 70 years after the author’s death (canada, house of commons, standing committee on canadian heritage, 2019, p. 1).   many canadian post-secondary institutions and educators believe that copyright term extension does not further the purpose of copyright as it provides no additional incentives to create new works (e.g., canadian association of research libraries, 2018; canadian association of university teachers, 2018). term extension can also hinder educational efforts that are dependent on public domain works (geist, 2017). additionally, an examination of what happens when books enter the public domain in australia, new zealand, the u.s., and canada found that “where copyright has been extended, libraries are being obliged to pay higher prices in exchange for worse access” (flynn et al., 2019, p. 1246). thus, to maintain copyright’s balance between public and private interests, it is essential for term extension implementation to include adequate measures to offset repercussions that would otherwise adversely affect the ability of educators and students to access copyrighted works.   appendix b 2020 survey questions   copyright practices and approaches at canadian post-secondaries: an expanded follow-up national survey   introduction   1.       in which consortium is your institution a member? a.       council of atlantic university libraries b.       bureau de coopération interuniversitaire c.        ontario council of university libraries d.       council of prairie and pacific university libraries e.       college libraries ontario   2.       what is the approximate size of your institution? a.       very small (up to 2,000 fte) b.       small (2,001 to 10,000 fte) c.        medium (10,001 to 25,000 fte) d.       large (25,001+ fte)   3.       what is the title of the position or office responsible for copyright at your institution? a.       university librarian/library director b.       copyright advisor c.        copyright coordinator d.       copyright librarian e.       copyright manager f.        copyright officer g.       copyright specialist h.       other (please explain)   responsibility for copyright   4.       at your institution, which position(s), department(s) or office(s) are responsible for the following activities associated with copyright?  if responsibility is shared, please indicate the position, department or office of all that are involved.  if an activity is not applicable, please enter "n/a". a.       education on the use of copyrighted materials b.       education on exercising and protecting owner rights under the copyright act c.        permissions clearance for coursepacks produced by your institution (print or electronic) d.       permissions clearance for works made available on library reserve (print) e.       permissions clearance for works made available on library reserve (electronic) f.        permissions clearance for works used in your institution’s learning management system g.       decisions on blanket licensing matters h.       development of institutional policies for users of copyrighted materials i.         development of institutional policies for copyright owners of copyrighted materials   5.       what is the approximate number of staff involved in to providing copyright assistance or services at your institution?  [comment box: please explain]   copyright education   6.       what are your institution’s main methods of providing copyright education a.       for users of copyrighted materials? b.       for creators of copyrighted materials?   7.       what are the topics most frequently covered in copyright education a.       for users of copyrighted materials? b.       for creators of copyrighted materials?   8.       over the past five years, have there been significant changes in how copyright education is addressed at your institution?  a.       yes (please briefly explain) b.       no   9.       what, if any, are some ways in which the usefulness of your institution’s copyright education efforts could be enhanced?   copyright policy   10.    does your institution have policies or guidelines on copyright?  a.       yes [please explain] b.       no   11.    [if  you responded yes to question 10 please provide the following information for your institution’s copyright policy: a.       specific issues addressed in the policy b.       date of establishment c.        most recent revision date, if applicable d.       main areas revised, if applicable e.       is the copyright policy publicly accessible?   12.    is your institution presently covered by a blanket license with access copyright or copibec? a.       yes b.       no   13.    if you responded yes to question 12, please indicate the date on which the license was initiated.   14.    what, if any, are some ways in which the usefulness of your institution’s copyright policies could be enhanced?     copyright permissions & licensing 15.    at your institution, is the potential applicability of a library licence for a full-text resource assessed when readings are distributed in the following ways? a.       coursepacks produced by your institution (print or electronic) ·   yes ·   no ·   uncertain ·   not applicable b.       coursepacks produced by a commercial copyshop (print or electronic) ·   yes ·   no ·   uncertain ·   not applicable   c.        copyrighted materials used in your institution’s learning management system ·   yes ·   no ·   uncertain ·   not applicable d.       copyrighted materials placed on library reserve (print) ·   yes ·   no ·   uncertain ·   not applicable e.       copyrighted materials placed on library reserve (electronic) ·   yes ·   no ·   uncertain ·   not applicable   16.    does your institution use a software application or platform to assist with managing copyright permissions and licensing? a.       yes b.       no c.        other (please explain)   17.    [if you answered yes to question 16] please identify the application(s) or platform(s) used.   18.    how often does your institution rely on the following sources to clear permissions for course materials?  5-point likert scale:  1 (never) to 5 (very frequently) a.       fair dealing under the copyright act (s. 29  s. 29.2) b.       user’s rights for educational institutions under the copyright act (e.g., s. 29.5, s. 30.04) c.        library license agreements for full-text electronic resource subscriptions d.       blanket institutional license from a copyright collective e.       openly licensed content such as those released under creative commons licenses f.        author or publisher permission granted without payable fees g.       transactional licensing for use of excerpts such as book chapters h.       transactional licensing for business cases i.         public domain (material not protected by copyright) j.         other (please explain)   19.    how does your institution cover the cost of transactional licensing for use of excerpts such as book chapters? (choose all that apply) a.       costs are directly charged to students b.       costs are indirectly charged to students via student fees c.        costs are factored into purchase or digital access fees managed by the bookstore d.       costs are factored into purchase or digital access fees managed by a unit other than the bookstore e.       costs are covered by a centralized fund managed by a unit such as the library or copyright office f.        other (please explain) g.       not applicable at my institution   20.    how does your institution cover the cost of transactional licensing for business cases? (choose all that apply) a.       costs are directly charged to students b.       costs are indirectly charged to students via student fees c.        costs are factored into purchase or digital access fees managed by the bookstore d.       costs are factored into purchase or digital access fees managed by a unit other than the bookstore e.       costs are covered by a centralized fund managed by a unit such as the library or copyright office f.        other (please explain) g.       not applicable at my institution   21.    other aspects of your institution’s practices and approaches regarding copyright permissions clearance you wish to comment on?   copyright compliance 22.    has your institution formally implemented a regularly conducted process for monitoring copyright compliance in its learning management system (lms)? a.       yes b.       no c.        other (please explain)   23.    [if response to question 22 is “yes”] how was the process for monitoring compliance in the (lms) developed, and by whom?   24.    [if response to question 6 is “yes”] who is responsible for monitoring copyright compliance in your institution’s lms? 25.    [if response to question 6 is “yes”] for each position having at least some responsibility for monitoring compliance in the lms, about what proportion of their normal work hours is taken up with compliance monitoring activities?   26.    [if response to question 6 is “yes”] to what extent are instructors of lms courses involved in the compliance monitoring process?   27.    [if response to question 6 is “yes”] please briefly describe the usual process for monitoring copyright compliance in your institution’s lms.   28.    [if response to question 6 is “no”] does your institution plan to implement a formal compliance monitoring process in the near future? a.       yes b.       no c.        other (please explain)   29.    are there other comments you wish to provide on your institution’s practices and approaches regarding copyright compliance or copyright management in general?   appendix c 2015 survey questions copyright practices and approaches at canadian universities: a national survey   introduction   5.       in which consortium is your institution a member? a.       council of atlantic university libraries b.       bureau de coopération interuniversitaire c.        ontario council of university libraries d.       council of prairie and pacific university libraries   6.       what is the approximate size of your institution? a.       small (up to 10,000 fte) b.       medium (10,001 to 25,000 fte) c.        large (25,001+ fte)   7.       what is your position title? i.         university librarian/library director j.         copyright advisor/officer k.       other (please explain)   responsibility for copyright   8.       at your institution, which position(s), department(s) or office(s) are responsible for the following activities associated with copyright?  if responsibility is shared, please indicate the position, department or office of all that are involved.  if an activity is not applicable, please enter "n/a". a.       education on the use of copyrighted materials b.       education on exercising and protecting owner rights under the copyright act c.        permissions clearance for coursepacks produced by your institution (print or electronic) d.       permissions clearance for works made available on library reserve (print) e.       permissions clearance for works made available on library reserve (electronic) f.        permissions clearance for works used in your institution’s learning management system g.       decisions on blanket licensing matters h.       development of institutional policies for users of copyrighted materials i.         development of institutional policies for copyright owners of copyrighted materials   copyright education   9.       what are your institution’s main methods of providing copyright education a.       for users of copyrighted materials? b.       for creators of copyrighted materials?   10.    what are the topics most frequently covered in copyright education a.       for users of copyrighted materials? b.       for creators of copyrighted materials? 11.    over the past five years, have there been significant changes in how copyright education is addressed at your institution?  a.       yes (please briefly explain) b.       no   12.    what are the most significant copyright education challenges at your institution?   copyright policy   13.    does your institution have a policy or guidelines on copyright?  c.        yes d.       no   14.    [if  you responded yes to question 9] please provide the following information for your institution’s copyright policy/guidelines: a.       specific topic addressed b.       date of establishment c.        most recent revision date, if applicable d.       main areas revised, if applicable e.       methods for communicating the policy/guidelines to your institution’s community   15.    is your institution presently covered by a blanket (access copyright or copibec) licence? a.       yes b.       no   16.    has your institution ever opted out of blanket licence coverage? a.       yes (please indicate the opt-out period start and end dates, if applicable) b.       no   17.    what are the most significant copyright policy challenges at your institution?   copyright permissions 18.    at your institution, is the potential applicability of a library licence for a full-text resource assessed when course readings are distributed in the following ways? a.       coursepacks produced by your institution (print or electronic) ·   yes ·   no ·   uncertain ·   not applicable b.       coursepacks produced by a commercial copyshop (print or electronic) ·   yes ·   no ·   uncertain ·   not applicable c.        copyrighted materials used in your institution’s learning management system ·   yes ·   no ·   uncertain ·   not applicable d.       copyrighted materials placed on library reserve (print) ·   yes ·   no ·   uncertain ·   not applicable f.        copyrighted materials placed on library reserve (electronic) ·   yes ·   no ·   uncertain ·   not applicable   19.    has your institution developed tools to assist institutional community members in clearing copyright permissions? a.       yes b.       no   20.    [if you answered yes to question 15] please identify and briefly describe each permissions clearance tool your institution has developed.   21.    what are the most significant copyright permissions challenges at your institution?   22.    are there other comments you wish to provide on your institution’s practices and approaches regarding copyright? [1] one clo member was excluded whose director was also the director of an ocul member who received a survey invitation. research article   public libraries and health promotion partnerships: needs and opportunities   noah lenstra associate professor of library & information science university of north carolina at greensboro greensboro, north carolina, united states of america email: lenstra@uncg.edu   joanna roberts graduate research assistant university of north carolina at greensboro greensboro, north carolina, united states of america email: jyroberts871@gmail.com   received: 21 sept. 2022                                                             accepted: 17 nov. 2022      2023 lenstra and roberts. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30250     abstract   objective – across north america, public libraries have increasingly served their communities by working with partners to connect patrons to essential healthcare services, including preventative. however, little is known about the extent of these partnerships, or the need for them, as seen from the perspective of public library workers. in this study, we set out to address the following research question: what needs and opportunities are associated with health promotion partnerships involving public libraries?   methods – using snowball sampling techniques, in september 2021, 123 library workers from across the state of south carolina in the united states (us) completed an online survey about their health partnerships and health-related continuing education needs; an additional 19 completed a portion of the survey.   results – key findings included that library capacity is limited, but the desire to support health via partnerships is strong. there is a need for health partnerships to increase library capacity to support health. public libraries already offer a range of health-related services. finally, disparities exist across regions and between urban and rural communities.   conclusion – as an exploratory study based on a self-selecting sample of public library workers in a particular state of the us, this study has some limitations. nonetheless, this article highlights implications for a variety of stakeholder groups, including library workers and administrators, funders, and policy makers, and researchers. for researchers, the primary implication is the need to better understand, both from the public library worker’s perspective and from the (actual or potential) health partner’s perspective, needs and opportunities associated with this form of partnership work.     introduction   on november 21, 2022, the canadian journal of community mental health published the research article “supporting mental health in a public library context: a mixed methods brief evaluation” (oudshoorn et al., 2022). the authors explored the potential for a collaboratively constructed mental health wellness hub situated in a canadian urban library, finding that this form of co-location was desired by patrons, mental health hub staff, and library staff.   as discussed below, the topic of public libraries and public librarians as health promotion partners has been increasingly explored by a range of scholars representing a range of disciplines. despite an increase in calls to better understand the current potential of public library participation in health promotion (flaherty & miller, 2016), most studies on this topic have either been case studies of particular communities (as in oudshoorn et al., 2022), or focused exclusively on single topics, such as consumer health at the library reference desk (e.g., arnott smith, 2011) or mental health (oudshoorn et al., 2022).   to give one example: there is a growing literature on social workers in public libraries (ogden & williams, 2022), which suggests that this integration is neither easy, nor inevitable, but instead requires different actors and stakeholders getting to know each other and find common ground (wahler et al., 2022). library workers recognized the need for someone like a social worker, but perhaps that need could also be filled by other forms of partnership not being explored in any sort of broad-scale way. for instance, baum et al. (2022), the authors of a recent study on the topic of social work-public library partnerships from the u.s. state of florida, found that:   all seven branch managers [interviewed] expressed enthusiasm when discussing the trend of social workers in libraries, noting patron struggles with food insecurity, homelessness, immigration, substance misuse, mental health challenges, and overall economic disadvantage as major motivating factors behind their support for including them in public libraries. (p. 14)   the range of social and health needs identified by these public librarians pointed to the need for a complementary range of partnerships, not only with social workers, but also with others in the health and social services sectors. nevertheless, we lack a broader understanding of the current state of needs and opportunities associated with public library participation in community health partnerships. with this study, we sought to begin to fill this gap.   literature review   discussions of health promotion in public libraries are as old as the profession of public librarianship itself (mon, 2021; rubenstein, 2012). most public library-based health promotion partnerships have been highly localized, involving librarians working with local health partners to develop innovative solutions to local problems, such as the example of a bookmobile in rural georgia transporting a county nurse in the 1940s (rubenstein, 2012). in the 1960s, some urban libraries started developing community information and referral systems to refer patrons in need to the services of other agencies, including health agencies (arnott smith, 2011). more recently, in 1992, a public librarian in stratford, ct, developed a partnership with a local teen counseling group and an aerobics instructor to develop a physical and mental health support group for teenagers at the library (lenstra, 2018).   through this literature review, we identified three themes in recent literature on this topic:   the library as community space for access to health-related services the library as a space for social workers and health workers the critical, if understudied, role of library workers in these partnerships. the library as community space for access to health-related services   the idea of the public library as a community space has become more prominent (e.g., klinenberg, 2018; mattern, 2007), shaping discussions of how health promotion activities occur in public libraries. as shown in a state-wide survey of pennsylvania library directors, health services in public libraries include, in different places, access to social workers, summer meals, bathrooms, a respite from the elements for individuals experiencing homelessness, nutrition classes, telehealth, and a range of other health and social services (whiteman et al., 2018). this trend continued during the covid-19 pandemic, when libraries were framed as convenient spaces to distribute tests, host immunization clinics, support access to telehealth, and even assist in efforts to address food insecurity (e.g., state of wisconsin, 2022; virginia department of health, 2021).   an additional facet of the literature on library as space has been research on libraries as crucial nodes in disaster response (liu et al., 2017; tu-keefner, 2016), particularly research on libraries in areas prone to hurricanes (hamilton, 2011; jaeger et al., 2006; mardis et al., 2020; veil & bishop, 2014). this trend has continued during the covid-19 pandemic, with research published on the roles of public libraries and librarians during this emergency (smith, 2020).    another notable trend was that of the public library as a support for child and family health. studies have been done on libraries as hosts of summer meal programs (de la cruz et al., 2020; sandha & holben, 2021), nutritional education classes (freedman & nickell, 2010), physical activity classes (bedard et al., 2020), oral health programs (woodson et al., 2011), and more generally as institutions that support health, including mental health, among vulnerable teenagers and youth (banas et al., 2020; campana et al., 2022; grossman et al., 2021; winkelstein, 2019).   providing direct medical support for adults through telehealth was a newer option being explored. santos (2021) provided a case study of this effort in a small rural library in pottsboro, texas. deguzman et al. (2021) found great potential for libraries to become hubs for providing health access to rural populations with little or no broadband access.   the library as a space for social workers and health workers   there has also been interest in placing social workers and other health workers in public libraries since the san francisco public library began the practice in 2009 (esguerra, 2019). this work emerged in part due to an increasingly public realization that public libraries were sites of public health incidents, including drug overdoses. feuerstein et al. (2022) surveyed five states (n=356) for information on instances of substance abuse on library property, and how libraries planned and prepared for this occurrence. the researchers found that alcohol and drug use was common on library property, but most libraries did not have on-site medical help, such as naloxone. they also found that librarians would like more training on how to handle these situations.   giesler (2021) studied perceptions of social workers in public libraries, finding differences in how the position was utilized across library systems. social workers might be primarily focused on training other library staff to recognize and empathize with specific patron populations, such as homeless populations. they could also interact directly with library patrons to offer services. gross and latham (2021) also found this benefit in staff training by the six library administrators who already employed social workers in the southeast us (n=52). johnson (2021) and wahler et al. (2022) described the components of “readiness” required for both a public library and the participating university when considering a social work student internship at a library.   other health liaisons have been found to be helpful for library staff assisting patrons with complex needs. homeless patrons often used the library for various reasons, as described by adams and krtalić (2021). they found that the presence of a community health worker focused on the needs of the homeless population helped the library and its staff to better understand and provide for those needs by reducing barriers to services.   other researchers considered alternative models of placing health liaisons in libraries. both interprofessional student internship models and the training of library staff in health information have been the focus of research. pandolfelli et al. (2021) considered the lessons from different experiential learning opportunities for students (n=21) from a variety of professions: general health courses at the undergraduate level, and masters’ level social worker, library science, and public health students. this study of a joint training experience for the students allowed for both support and building common ground between professions and the students involved when working in a library setting. this sort of inter-professional training model has been deployed at the university of missouri, where library & information science students took courses on public health as part of an experimental, federally funded project (bossaller et al., 2022).   draper (2021) conducted a feasibility study to explore the potential for collaboration between nutrition educators and public libraries. draper found that while there were extensive overlapping goals between u.s. department of agriculture’s supplemental nutrition assistance program education program (snap-ed)—which is an evidence-based program that helps people lead healthy, active lives by partnering with state and local organizations—and public libraries, library staff had little knowledge of the federal program, with only 1 of the 14 participants having any understanding of snap-ed and how it could support libraries.   in spring 2022, the texas-based st. david’s foundation announced a new $1.5 million initiative to support what they are calling libraries for health. the foundation is collaborating to broaden access to mental health services for rural residents by placing non-clinical mental health workers at the public libraries in central texas. the non-clinical mental health worker initiative is modeled on peer navigator programs found in some urban libraries across the country. crucially, the program includes a strong evaluation component, led by the rand corporation (carey, 2022).   the under-studied role of library workers   as public libraries were increasingly seen as opportune spaces for health promotion services, the roles of library workers in administering these services were sometimes overlooked. the absence of library workers from these discussions could sometimes lead to burnout and staff feeling overwhelmed, as they felt they were being asked to take on more and more in their daily work (freeman & blomley, 2019). this perception increased during the covid-19 pandemic, when many librarians suffered trauma as front-line workers (comito & zabriskie, 2022). the question of what capacity public library workers have to participate in community health partnerships remains under-studied.   when library workers were mentioned in research on this topic, researchers tended to focus on how to most effectively train library workers to field reference questions related to consumer health information in ways analogous to the work done by medical or health sciences librarians. derosa et al. (2021) described a partnership between weill cornell medicine library and the city of brooklyn, new york, focused on using a train-the-trainer model for library workers by providing training in how to better serve patrons requesting information about health issues. other researchers have focused on the preparedness of public library workers to support patrons in crisis. wong et al. (2021) reported on the ability of pennsylvania public librarians (n=100) to provide health information on substance abuse issues over the phone and found that there was a wide variation between libraries. brus et al. (2019) surveyed public library staff in australia and found a lack of confidence in dealing with patrons with complicated social issues such as mental health and homelessness. malone and clifton (2021) explored this idea in a five-year study of the oklahoma public library system (n=106 staff, n=67 libraries) to certify existing public library staff in specialized training from the medical library association. fewer researchers have focused on how to foster cross-sector collaborations most effectively between public librarians and those working in the community health sector (lenstra & mcgehee, 2022).   aims   in this study, we set out to address the following research question: what needs and opportunities are associated with health promotion partnerships involving public libraries?   we framed this question from the perspective of public library workers. future research on this topic could investigate the question from the perspective of actual or potential public library partners.   methods   to understand the needs and opportunities associated with south carolina public library participation in health initiatives, we designed a survey through a collaborative process that included the following steps:   1.      a review of survey instruments used in previous surveys of the topic of public libraries and community health, including those in bertot et al. (2015), feuerstein-simon et al. (2020), and whiteman et al. (2018). 2.      codifying the range of health partnerships involving public libraries discussed in previous literature to ensure the survey inquired about different partnership configurations. 3.      an alignment of the research instrument with the priorities of the south carolina center for rural & primary healthcare (sc crph), a partner in this study. 4.      coordination with the state library of south carolina around framing this topic.   after development and testing, the research methods were approved by the institutional review board of the university of north carolina at greensboro (study #irb-fy22-71).   there is no comprehensive directory of public library employees in south carolina, either at the state or local levels. according to the u.s. institute of museum & library services (imls), in fy2019 – the most recent year for which data was available at the time of this writing – the total staff of all public libraries in south carolina is 2,112 (pelczar, 2021). this number includes 514 credentialed librarians and 1,598 other employees, including paraprofessionals, groundskeepers, and security staff, among others. there are 42 public library systems in the state. although we were most interested in hearing from librarians, the survey was designed such that it was open to any employee of a public library in south carolina.   to reach these employees, the researchers used a form of snowball sampling in which individuals and institutions that were pillars of the public library community in south carolina were asked to distribute the survey to their networks on behalf of the researchers. these institutions included the state library of south carolina, the network of the national library of medicine, the south carolina library association, and the sc crph itself.    the survey was distributed over four weeks in september 2021, an extremely difficult moment in south carolina and in the world. for logistical reasons, the survey had to be distributed during this moment in time. september 2021 was in the middle of the global pandemic. these logistical reasons centered around the temporal constraints of the south carolina center for rural & primary healthcare, which wished to better understand this topic prior to releasing financial awards in spring 2022 to south carolina public libraries wishing to embark on novel health partnerships in their communities.   during the four weeks the survey was open, the researchers monitored the response rate, generating a weekly map of where respondents were coming from, at the county level. this response rate informed subsequent snowball sampling techniques, which focused on attempting to secure complete saturation across all counties in the state of south carolina. more information on recruitment and sampling can be found in lenstra and roberts (2022).   analysis   descriptive statistics were calculated for all closed-ended survey responses, while thematic coding was conducted to analyze open-ended responses. to further analyze the data and to generate regional and other trends, the researchers used the demographic information respondents provided about their job titles and library locations to generate comparisons. following federal practices established by the imls, rural and urban differences were established using the procedures set by the national center for education statistics, a unit of the u.s. department of education.   we organized this article around the sections of the survey that centered on partnerships that included public librarians and actors in the health and social service sectors, including results that helped indicate why such partnerships would or would not be desirable. the survey included a range of questions on the broader topic of needs and opportunities associated with public libraries as institutions embedded within community health ecosystems. readers interested in accessing broader survey results, including the dataset itself and the results of the thematic coding of open-ended responses, may do so at the open access white paper published by lenstra and roberts (2022).   limitations   as with any nonprobability sampling technique, there were limitations to this approach, which centered around the fact that statistical generalization to the broader population studied is impossible. nonetheless, we chose nonprobability sampling as the best way to secure a broad sample of the south carolina public library community within the timeframe of the project.   additional limitations derived from the survey format itself. it was possible that different respondents may have interpreted some of the survey’s prompts in different ways. for instance, the survey did not specify what was meant by “access to health literacy,” and thus this prompt and others like it may have been interpreted in different ways. despite these limitations, this survey and its results provided an unprecedented window into perceptions, needs, and opportunities associated with public library and health partnerships.    results   in general terms, the sample of respondents roughly aligned with the distribution of public libraries across south carolina. figure 1 shows that the distribution of the 123 respondents who fully completed the survey roughly aligned with the distribution of south carolina libraries. the number of “completed responses” refers to the number of library workers who totally completed the survey. there were 21 additional respondents who gradually dropped out of the survey after completing only a portion. all those who dropped out did so after completing at least a full page of questions. more information on the sample appeared in lenstra and roberts (2022), and the number of respondents for individual questions can be found in the appendix.   figure 1 survey response distribution compared to the distribution of public libraries across the state of south carolina.   the vocabulary used in figure 1 corresponds to the three ways that the imls used to assess the geographical distribution of public libraries across the nation: 1) library branches (outlets, in the nomenclature of the imls) referred to library branches and bookmobiles, 2) library systems (administrative entities mod in the nomenclature of the imls) referred to the geographic spread of multi-branch library systems, and 3) library headquarters (administrative entities add in the nomenclature of the imls) referred to the locations of the headquarters of multi-branch library systems.   the desire to collaborate with health workers is strong, but capacity is limited   nearly every respondent reported a need for a health worker or a health liaison to help them serve the public in their library: over 90% said that if outside help were available, they could see a need for a health or social worker at their libraries.   however, when respondents were asked if they would like to have specific types of health workers or health liaisons available at their libraries, interest diminished. only 74% of respondents were interested in, or currently had available, social workers at their libraries. social workers were the most desired type of health liaison (table 1).   table 1 public library worker interest in having health professionals available to the public at their libraries health professional (n=126) not interested offered interested – not offered social workers 26% 23% 51% nurses 37% 12% 51% health educator 25% 27% 48% medical students 48% 4% 48% community health workers 29% 24% 47% social work students 44% 13% 43% americorps or other volunteers 40% 21% 39% other health-related professional 69% 8% 23%   one reason for this perceived need may relate to the prevalence of health-related incidents that occur on library property. nearly 80% of respondents reported that people experiencing homelessness used their public libraries as day shelters, and between 10-50% reported a range of other incidents on library properties, including drug deals, physical violence, and overdoses. librarians also wrote open-ended comments about health-related incidents they had witnessed at their libraries, including seizures and heart problems (figure 2).   figure 2 medical and health related incidents occurring on public library property. (n=127)   city librarians were most likely to report all the incident types asked about, except for drug deals (figure 3). data suggested, however, that these sorts of incidents occurred in public libraries across the state. less than 30% of rural respondents said no health-related incidents had occurred at their properties. as open public spaces, health issues occurring in communities tended to also occur in public libraries.   figure 3 medical and health related incidents occurring on public library property, by type of community served. (n=127)   in any case, one reason for the difference between perceived need for outside help in general, and perceived need for specific forms of outside help, related to limited library capacity to develop new initiatives. as one respondent wrote in an open-ended comment: “we do not have enough staff and really cannot handle any more programs. even when partnering with others, it takes staff time, and we just cannot do it anymore due to not enough staff.”   a need for health partnerships to increase library capacity to support health   respondents did not always have the partnerships that would enable them to bring other types of health services to their libraries, or to refer library patrons to appropriate health or social service agencies. public libraries typically had close connections with agencies that support the social determinants of health (sdoh), including educational institutions, parks & recreation units, and non-profits. these reported close relationships could position public libraries to effectively facilitate community conversations on health needs in ways that would bring more voices into local health planning and policy making.   librarians also reported offering a range of services that support addressing the sdoh, including access to technology, literacy, education, food, legal aid, and employment. across the state, many public libraries have hosted a wide array of services that support public health and the sdoh, with more than 40% reporting they have hosted everything from food drives to fitness classes, farmers’ markets, summer meals, health fairs, and blood drives.   less robust were the relationships between public libraries and agencies specifically in the health sector, and less common were library services that directly supported access to healthcare. less than 50% of respondents reported close relationships with any organization in the health sector (table 2).   table 2 closeness of relationships between public libraries and potential partners (n=127) department/institution very close or somewhat close not very close k-12 schools 89% 11% early education providers, including daycares 88% 12% local non-profit organizations 85% 15% parks & recreation unit 61% 39% colleges or universities 60% 40% health department 44% 56% hospital or healthcare system(s) 41% 59% health coalition or alliances 41% 59% snap-ed implementing agency 41% 59% department of justice / department of corrections 31% 69% wic clinics 29% 71%   despite being less common currently, there existed a sizable number of early adopters and health champions within the south carolina public library sector who reported already working closely with health partners. around one quarter of respondents said they have had health liaisons and telehealth services available at their libraries. around one-third of respondents reported the presence of a health champion employed within their libraries, someone who championed health services and partnerships and could be utilized as an entry point for programs and partnerships.   responsive health services in south carolina public libraries   nevertheless, most respondents thought that individuals in their communities look to the library as a safe and trusted space, used both to access health literacy and to access health services, and most librarians saw health equity as a priority for their libraries.    the most common way in which public librarians themselves directly supported health centers was around information access, with 75% of respondents saying their libraries supported access to health information in general, 63% supporting health literacy, and 57% reporting they provided help identifying and using local health resources. less commonly reported were informational referrals to appropriate health or social service agencies (43%). more than 60% of respondents said their libraries supported access to related services during the covid-19 pandemic, including 42% who offered immunization clinics for covid-19, and 29% who offered covid-19 testing services. in the context of the ongoing opioid crisis, over 20% of urban librarians, and over 10% of rural librarians, reported having naloxone available at their libraries.   regional and rural/urban disparities   in general terms, rural librarians were broadly interested in doing more to support health, and compared to their more urban peers, have had fewer opportunities, and less capacity, to do so. for instance, most rural librarians reported interest in offering mental health first aid trainings, while most urban librarians had already offered these trainings.   rural librarians were also those least likely to have had formalized health partnerships, with 50% reporting no partners in programmatic or funded health initiatives, meaning they were less likely to have partnerships to support health. given this situation, rural respondents were broadly interested in whatever resources they may be able to bring to their communities. thinking about what kind of health liaison would be the best fit for a public library, respondents across the state articulated a preference for fully credentialed health liaisons, rather than for students, volunteers, or other health workers in training. rural librarians, however, were broadly interested in whatever health liaisons they could bring to their libraries, regardless of credentials.   continuing education and support needs   only 10% of respondents reported no barriers to supporting health at their libraries, suggesting a need for more robust continuing education and sustained support. top priorities for continuing education as reported by survey respondents included how to get started supporting health at public libraries, how to sustain these efforts, and how to build partnerships around this topic. major barriers to supporting health included a perceived lack of expertise and funding. librarians reported wanting to learn more about this topic from other librarians who have directly dealt with these issues at their libraries.   looking to continuing education needs, urban librarians had markedly different continuing education priorities, with sustainability and evaluation coming out on top (table 3). in contrast, for all other parts of the state, there was more interest in introductory topics, with how to get started and how to partner rated as top priorities for continuing education. it appeared that urban libraries had, in general, already started these partnerships, and were looking to better sustain and evaluate them, while all other libraries were looking to get started with these types of partnerships.   although it was not always identified as a top priority for continuing education, evaluation emerged as a significant obstacle. most respondents indicated that they were not doing anything to evaluate or track the impacts of their libraries on health. one respondent wrote:   we don't really have a way to track this info. we did weight loss programs, but the partner tracked progress and no long-term info available. we have done nutrition and health programs with our hospital targeting diabetes and heart disease, distributed food during 2020, have had exercise programs for seniors, walking programs, etc. we have sponsored cpr training courses for the public.   an effective evaluation system would need to consider the myriad and evolving ways in which public libraries support health. due to an absence of evaluation systems, the contributions of public libraries to community health were often invisible, and thus underappreciated and under-supported.   table 3 top priorities for library continuing education, by community (n=121) urban (n=20) suburban (n=24) town (n=50) rural (n=27) sustainability (70%) how to get started (54%) how to get started (62%) how to get started (59%) how to evaluate (65%) how to partner (tied) (54%) how to partner (56%) how to partner (tied) (59%) how to get started (60%) sustainability (tied) (54%) marketing (48%) marketing (56%) how to partner (50%) marketing (38%) sustainability (46%) sustainability (48%) marketing (30%) how to evaluate (21%) how to evaluate (36%) how to evaluate (tied) (48%)   discussion   most respondents to this survey saw a role for their public libraries in health promotion, equity, and access. nevertheless, obstacles large and small prevented the south carolina public library workforce from doing as much as they would have liked to support health. in urban south carolina, funding, sustainability, and evaluation were major challenges, while more rural areas were challenged in discovering how to get started and how to build partnerships. throughout the state, respondents saw a need for help weaving health into the operations of a public library without overwhelming or over-burdening the library staff. librarians needed technical assistance, as well as support for funding and evaluation, to make their community-based health initiatives sustainable and impactful over the long-term.   through this survey, we also found that public library workers across the state were already assisting the most vulnerable, including individuals experiencing homelessness, drug addiction, mental health crises, and more. while the survey results suggested that some libraries were working to prevent staff burnout, there was an opportunity for health partners to help libraries with current burdens and burnout issues, while also creating relationships that could be leveraged to increase health in both the shortand long-term. previous researchers (brus et al., 2019; gross & latham, 2021) have indicated that social workers and other health liaisons could provide the library workforce with the skills needed to practice self-care while also caring for the most vulnerable. in addition, this type of training could provide library staff and medical staff with a greater understanding of the benefits of each group to the wider community. this, in turn, could help build relationships that would make coordination between them more strategic when focused on health.   in this study, we identified a handful of library systems that have embraced health services and partnerships at their libraries, including in their strategic plans. the charleston county public library is one example (ccpl, 2021). their strategic plan explicitly called for the library to “empower learners of all ages to manage their lifelong physical and mental health,” “empower individuals with the knowledge to make healthy food choices,” and “empower individuals to obtain and understand basic health information” (ccpl, 2021). finding ways to meaningfully enable these early adopters and their leadership teams to share their successes and challenges with other libraries could potentially drive innovation forward.    to extend this trend, these library health champions could share best practices, advocate for promising partnerships, and share common successes and challenges through the peer-to-peer infrastructure that exists for professional development and continuing education among public librarians.   there is a strong tradition of training programs for public librarians focused on increasing their comfort and confidence with health information. this training has been, historically, offered by medical and academic health science librarians (e.g., malone & clifton, 2021). the successful deployment of peer-to-peer training among public librarians, perhaps in a learning cohort, could provide public librarians with a different type of training program, one focused less on comfort and confidence with health information sources, and more on comfort and confidence working collaboratively with community health partners.   our goal should be to find ways to enable the health and public library workforces to mutually build each other up, with the two workforces adding value to each other, and adding capacity to their abilities to support the communities they serve together.   a second promising practice would be to find ways to better connect library directors to local health leaders and to other library leaders. these connections could be made not only at the library executive director level, but also at the deputy director and branch/division manager levels. survey results suggested these library middle managers were less connected to local health partners than library directors. in any case, at the leadership level, the focus is less on cultivating library health champions, and more on how we make these partnerships work, administratively.   library directors and leaders need help understanding how to integrate health into library services in ways that avoid the burnout of their staff, and that are sustainable over time. they also need help integrating timely topics, such as telehealth, into their libraries. evaluation is a perennial issue in public libraries, and thinking strategically about health in public librarianship is another need.   evaluation of how public libraries support health is also essential. a starting point for developing this type of evaluation should be a discussion between health organizations and public libraries that promotes understanding of the different structures and needs of each group. finding ways to embed documentation into these partnerships is crucial for their long-term viability.   conclusion   in this exploratory study, we highlighted implications for a variety of stakeholder groups, including those working in the health sector at both local and state levels, as well as library workers and administrators, funders and policy makers, and researchers.   given the limited and self-selecting sample, comparisons between rural and urban public library workers remained tentative. additional research using a randomized sampling model that employs cluster sampling to ensure a strategically selected distribution of public library workers representing the rural-urban continuum could enable a more nuanced understanding of the unique needs of public library workers within different types of communities.   we are only beginning to understand needs and opportunities associated with public library participation in community health initiatives. additional research must also consider this topic from the perspective of community health partners. how ready are community health workers and social workers to partner with public librarians? this topic also needs to be addressed to holistically understand this topic.   more generally, many of the findings of this survey deserved more nuanced explanation through interview-based research. the survey results showed what was happening in south carolina’s public libraries; it cannot answer why things were the way they were. for instance, the survey found that in one-third of respondents’ libraries, a health champion was employed. how did these health champions within the public library workforce come to be? what policies, practices, and community forces led to health champions working at these libraries? these are topics interview-based and case study research could help to illuminate.   although tentative, the findings from this project unambiguously demonstrated interest within the south carolina public library workforce to support health, particularly though partnerships that would bring health workers to their libraries. although additional research is needed to build up our understanding of this topic, this survey showed a great potential for impacts associated with public library health partnerships.   author contributions   noah lenstra: conceptualization, formal analysis, funding acquisition, investigation, methodology, supervision, writing – review & editing joanna roberts: data curation, formal analysis, investigation, project administration, validation, visualization, writing – original draft   acknowledgment   this research was made possible in part by the south carolina center for rural & primary healthcare grant #22-0035. we would also like to thank our reviewers for their thorough critique of the manuscript, which improved the manuscript in significant ways. we also want to say thank you to the south carolina public library and public health communities, who shared feedback on early findings from this project at a south carolina state library webinar and the south carolina public health association conference, respectively.   references   adams, c., & krtalić, m. (2021). i feel at home: perspectives of homeless library customers on public library services and social inclusion. journal of librarianship and information science, 54(4), 779-790. https://doi.org/10.1177/09610006211053045   arnott smith, c. (2011). “the easier-to-use version”: public librarian awareness of consumer health resources from the national library of medicine. journal of consumer health on the internet, 15(2), 149–163. https://doi.org/10.1080/15398285.2011.573339   banas, j. r., oh, m. j., willard, r., & dunn, j. (2020). a public health approach to uncovering the health-related needs of teen library patrons. the journal of research on libraries and young adults, 11(1). https://neiudc.neiu.edu/hpera-pub/9/   baum, b., gross, m., latham, d., crabtree, l., & randolph, k. (2022). bridging the service gap: branch managers talk about social workers in public libraries. public library quarterly. https://doi.org/10.1080/01616846.2022.2113696   bedard, c., bremer, e., & cairney, j. (2020). evaluation of the move 2 learn program, a community-based movement and pre-literacy intervention for young children. physical education and sport pedagogy, 25(1), 101-117. https://doi.org/10.1080/17408989.2019.1690645    bertot, j. c., real, b., lee, j., mcdermott, a. j., & jaeger, p. t. (2015). 2014 digital inclusion survey: survey findings and results. information policy & access center (university of maryland); american library association. https://web.archive.org/web/20170403022504/http://digitalinclusion.umd.edu/sites/default/files/uploads/2014digitalinclusionsurveyfinalrelease.pdf   bossaller, j. s., adkins, d., pryor, c. n., & ward, d. h. (2022). catalysts for community health (c4ch). https://c4ch.missouri.edu/   brus, j., fulco, c, hornibrook, l., monypenny, k., & niranjan, p. (2019). social issues in public libraries: supporting our staff. https://www.plv.org.au/wp-content/uploads/2020/04/social-issues-in-public-libraries-supporting-our-staff-2019.pdf   campana, k., mills, j. e., kociubuk, j., & martin, m. h. (2022). access, advocacy, and impact: how public libraries are contributing to educational equity for children and families in underserved communities. journal of research in childhood education, 36(4), 561-576. https://doi.org/10.1080/02568543.2021.2017375    carey, l. (2022, march 24). texas rural libraries will help with mental health access. the daily yonder. https://dailyyonder.com/texas-rural-libraries-will-help-with-mental-health-access/2022/03/24/   ccpl: charleston county public library. (2021). ccpl's strategic vision for 2021-2024. https://www.ccpl.org/strategicvision   comito, l., & zabriskie, c. (2022). urban library trauma study: final report. urban librarians unite. https://urbanlibrariansunite.org/ults/   de la cruz, m. m., phan, k., & bruce, j. s. (2020). more to offer than books: stakeholder perceptions of a public library-based meal programme. public health nutrition, 23(12), 2179-2188. https://doi.org/10.1017/s1368980019004336   deguzman, p. b., jain, n., & loureiro, c. g. (2021). public libraries as partners in telemedicine delivery: a review and research agenda. public library quarterly, 41(3), 294-304. https://doi.org/10.1080/01616846.2021.1877080   derosa, a. p., jedlicka, c., mages, k. c., & stribling, j. c. (2021). crossing the brooklyn bridge: a health literacy training partnership before and during covid-19. journal of the medical library association, 109(1), 90–96. https://doi.org/10.5195/jmla.2021.1014   draper, c. l. (2021). exploring the feasibility of partnerships between public libraries and the snap-ed program. public library quarterly, 41(5), 439-455. https://doi.org/10.1080/01616846.2021.1906615   esguerra, l. (2019, january 4). providing social service resources in a library setting. public libraries online. http://publiclibrariesonline.org/2019/01/providing-social-service-resources-in-a-library-setting/   feuerstein-simon, r., lowenstein, m., dupuis, r., dolan, a., marti, x. l., harvey, a., ali, h., meisel, z. f., grande, d. t., lenstra, n., & cannuscio, c. c. (2022). substance use and overdose in public libraries: results from a five-state survey in the us. journal of community health, 47, 344-350. https://doi.org/10.1007/s10900-021-01048-2   flaherty, m. g., & miller, d. (2016). rural public libraries as community change agents: opportunities for health promotion. journal of education for library and information science, 57(2), 143-150. https://doi.org/10.3138/jelis.57.2.143   freedman, m. r., & nickell, a. (2010). impact of after-school nutrition workshops in a public library setting. journal of nutrition education and behavior, 42(3), 192-196. https://doi.org/10.1016/j.jneb.2009.07.003   freeman, l. m., & blomley, n. (2019). enacting property: making space for the public in the municipal library. environment and planning c: politics and space, 37(2), 199–218. https://doi.org/10.1177/2399654418784024   giesler, m. (2021). perceptions of the public library social worker: challenges and opportunities. the library quarterly, 91(4), 402–419. https://doi.org/10.1086/715915 gross, m., & latham, d. (2021). social work in public libraries: a survey of heads of public library administrative units. journal of library administration, 61(7), 758–775. https://doi.org/10.1080/01930826.2021.1972727   grossman, s., agosto, d. e., winston, m., epstein, n. e., cannuscio, c. c., martinez-donate, a., & klassen, a. c. (2021). how public libraries help immigrants adjust to life in a new country: a review of the literature. health promotion practice, 23(5), 804-816. https://doi.org/10.1177/15248399211001064   hamilton, r. (2011). the state library of louisiana and public libraries' response to hurricanes: issues, strategies, and lessons. public library quarterly, 30(1), 40-53. https://doi.org/10.1080/01616846.2010.525385   jaeger, p. t., langa, l. a., mcclure, c. r., & bertot, j. c. (2006). the 2004 and 2005 gulf coast hurricanes: evolving roles and lessons learned for public libraries in disaster preparedness and community services. public library quarterly, 25(3-4), 199-214. https://doi.org/10.1300/j118v25n03_17   johnson, s. c. (2021). innovative social work field placements in public libraries. social work education, 41(5), 1006-1017. https://doi.org/10.1080/02615479.2021.1908987   klinenberg, e. (2018). palaces for the people: how social infrastructure can help fight inequality, polarization, and the decline of civic life. crown.   lenstra, n. (2018). let’s move! fitness programming in public libraries. public library quarterly, 37(1), 61-80. https://doi.org/10.1080/01616846.2017.1316150   lenstra, n., & mcgehee, m. (2022). public librarians and public health: how do partners perceive them? journal of library outreach and engagement, 2(1), 66-80. https://doi.org/10.21900/j.jloe.v2i1.883   lenstra, n. & roberts, j. (2022). south carolina public libraries & health: needs and opportunities. https://libres.uncg.edu/ir/listing.aspx?id=37560   liu, j., tu-keefner, f., zamir, h., & hastings, s. k. (2017). social media as a tool connecting with library users in disasters: a case study of the 2015 catastrophic flooding in south carolina. science & technology libraries, 36(3), 274-287. https://doi.org/10.1080/0194262x.2017.1358128   malone, t., & clifton, s. (2021). using focus groups to evaluate a multiyear consumer health outreach collaboration. journal of the medical library association, 109(4), 575-582. https://doi.org/10.5195/jmla.2021.987     mardis, m. a., jones, f. r., pickett, s. m., gomez, d., tenney, c. s., leonarczyk, z., & nagy, s. (2020, october 13). librarians as natural disaster stress response facilitators: building evidence for trauma-informed library education and practice [paper presentation]. 2020 association of library and information science educators conference. http://hdl.handle.net/2142/108818   mattern, s. c. (2007). the new downtown library: designing with communities. university of minnesota press.   mon, l. (2021). the fight against enzy: us libraries during the influenza epidemic of 1918. dttp: documents to the people, 49(1), 12-17. https://doi.org/10.5860/dttp.v49i1.7538   ogden, l. p., & williams, r. d. (2022). supporting patrons in crisis through a social work-public library collaboration. journal of library administration, 62(5), 656-672. https://doi.org/10.1080/01930826.2022.2083442   oudshoorn, a., van berkum, a., burkell, j., berman, h., carswell, j., & van loon, c. (2022). supporting mental health in a public library context: a mixed methods brief evaluation. canadian journal of community mental health, 41(2), 25-45. https://doi.org/10.7870/cjcmh-2022-013    pandolfelli, g., hammock, a., topek-walker, l., d’ambrosio, m., tejada, t., della ratta, c., lasala, m. e., koos, j. a., lewis, v., & benz scott, l. (2021). an interprofessional team-based experiential learning experience in public libraries. pedagogy in health promotion, 9(1), 54-63. https://doi.org/10.1177/23733799211048517   pelczar, m., frehill, l. m., nielsen, e., kaiser, a., & li, j. (2021). public libraries survey: fiscal year 2019. institute of museum & library services. https://www.imls.gov/research-evaluation/data-collection/public-libraries-survey   rubenstein, e. (2012). from social hygiene to consumer health: libraries, health information, and the american public from the late nineteenth century to the 1980s. library & information history, 28(3), 202-219. https://doi.org/10.1179/1758348912z.00000000016   sandha, p., & holben, d. h. (2021). perceptions of the summer food environment in a rural appalachian mississippi community by youth: photovoice and focus group. journal of hunger & environmental nutrition, 17(6), 834-849. https://doi.org/10.1080/19320248.2021.1994082   santos, m. c. (2021). libraries as telehealth providers. texas library journal, 97(1), 92–94.   smith, j. (2020). information in crisis: analyzing the future roles of public libraries during and post-covid-19. journal of the australian library and information association, 69(4), 422-429. https://doi.org/10.1080/24750158.2020.1840719   state of wisconsin. (2022, march 30). gov. evers announces $5 million investment to expand access to telehealth services [press release]. https://content.govdelivery.com/accounts/wigov/bulletins/3110131   tu-keefner, f. (2016). the value of public libraries during a major flooding. in a. morishima, a. rauber, & c. l. liew (eds.), digital libraries: knowledge, information, and data in an open access society (pp. 10-15). springer cham. https://doi.org/10.1007/978-3-319-49304-6_2   veil, s. r., & bishop, b. w. (2014). opportunities and challenges for public libraries to enhance community resilience. risk analysis, 34(4), 721-734. https://doi.org/10.1111/risa.12130    virginia department of health. (2021). supporting testing access through community collaboration. https://www.vdh.virginia.gov/coronavirus/protect-yourself/covid-19-testing/stacc/   wahler, e. a., ressler, j. d., johnson, s. c., rortvedt, c., saecker, t., helling, j., williams, m. a., & hoover, d. (2022). public library-based social work field placements: guidance for public libraries planning to become a social work practicum site. public library quarterly. https://doi.org/10.1080/01616846.2022.2044264   whiteman, e. d., dupuis, r., morgan, a. u., d’alonzo, b., epstein, c., klusaritz, h., & cannuscio, c. c. (2018). public libraries as partners for health. preventing chronic disease, 15. https://doi.org/10.5888/pcd15.170392   winkelstein, j. a. (2019). the role of public libraries in the lives of lgbtq+ youth experiencing homelessness. in b. mehra (ed.), lgbtq+ librarianship in the 21st century: emerging directions of advocacy and community engagement in diverse information environments (pp. 197-221). emerald publishing limited. https://doi.org/10.1108/s0065-283020190000045016   wong, v., cannuscio, c. c., lowenstein, m., feuerstein-simon, r., graves, r., & meisel, z. f. (2021). how do public libraries respond to patron queries about opioid use disorder? a secret shopper study. substance abuse, 42(4), 957–961. https://doi.org/10.1080/08897077.2021.1900980   woodson, d. e., timm, d. f., & jones, d. (2011). teaching kids about healthy lifestyles through stories and games: partnering with public libraries to reach local children. journal of hospital librarianship, 11(1), 59-69. https://doi.org/10.1080/15323269.2011.538619   appendix survey instrument   survey questions number of respondents do you understand the consent information provided above and agree to participate in the study?   part 1. demographics                  what is the zip code* where your library is located? *having this                information will allow us to incorporate data from other sources,                including the us census. 142                what is the name of the library or library branch where you work? 142                what is your job title? 142                how long have you worked at your library? 142 part 2. health services at the library                  we would like to know how your library supports            health. please                indicate what types of health-related services or programs your library                has, to your knowledge, offered, as well as what types of topics you                would like to learn more about in the future. (select all that apply)                                 access to health information in general 142                               access to health literacy 142                               access to primary healthcare 142                               access to preventative health services 142                               access to health insurance 142                               access to mental health or behavioral health 142                               access to reproductive health 142                               access to services for substance use disorders 142                               access to covid-19 related services 142                               access to food 142                               access to nutrition 142                               access to physical activity 142                               access to support with chronic disease(s) 142                               access to services related to healthy aging 142                               access to reentry services for those previously incarcerated 142                               access to housing 142                               access to transportation 142                               access to employment 142                               access to early childhood services 142                               access to education (adult) 142                               access to education (pre-k) 142                               access to education (k-12) 142                               access to legal aid 142                               access to economic development opportunities 142                               access to technology 142                               access to literacy 142 part 3. library in the community                  how would you rate the following: [options included strongly agree, slightly agree, slightly disagree, strongly disagree]                                 my library routinely offers off-site programs or services 136                               library staff often participate in community meetings or coalitions 136                               organizations in general typically look to the library as a partner 136                               health organizations, specifically, look to the library as a partner 136                               individuals in the community typically see the library as a safe and                                              trusted space to access health literacy 136                               individuals in the community typically see the library as a safe and                                              trusted space to access health services 136                               my library sees health equity as a priority 136                               my library serves as a space where people can meet new people                               in the community.. 136                               my library serves as a space where social connections are affirmed 136                               individuals in the community typically see the library as a safe and                                              trusted space for all ages 136                               individuals in the community typically see the library as a safe and                                              trusted space for all ages 136                               library staff are typically well versed in the pressing issues facing                                              the community 136                               library staff are typically able to work collaboratively with other                                                               individuals and organizations to address pressing community                                    issues 136 part 4. the library and community health                  have library staff and/or partners ever offered any of the following at                your library, or off-site with library participation? (select all that                apply). "partners" here includes all individuals or organizations that                are not directly affiliated with the library                                  immunization clinics, in general (e.g. for vaccinations) 129                               immunization clinics, specifically for covid19 129                               covid-19 testing 129                               health screening services: blood pressure 129                               health screening services: obesity 129                               health screening services: mammography 129                               health screening services: other 129                               assistance with mental health issues (e.g. social, behavioral,                                                      emotional needs) 129                               referrals to appropriate health and/or social service agencies 129                               locating and evaluating free health information online 129                               using subscription health database(s) 129                               identifying health insurance resources 129                               understanding specific health topics 129                               identifying or using local health resources 129                               offering fitness classes 129                               offering nutrition classes 129                               summer meals 129                               other ways of distributing free food (community fridge, food boxes) 129                               health fairs 129                               farmer’s markets 129                               blood drives 129                               food drives 129                               mental health first aid trainings 129                               telehealth services 129                have any of the following health-related groups ever met at your                                                                           library? (select all that apply)                                 health coalitions 127                               health department task forces 127                               area agency on aging 127                               other health-related groups (please describe) 127                               none of the above 127                to your knowledge, have any of the following ever occurred at your                                       library, or on property owned by your library (e.g. parking lot)?                                 drug overdose 127                               drug deal 127                               alcohol or tobacco use against library policy 127                               individuals experiencing homelessness using library as de facto                                              day shelter 127                               violence that requires intervention from security staff or law                                                     enforcement 127                               other health related incidents (please describe) 127                               none of these 127                do any of your library staff have access to the following on-site at your                                                 library? (select all that apply)                                 naloxone 127                               epipen 127                               automated external defibrillator (aed) 127                               other health-related equipment (please describe) 127                               none of the above 127 part 5. staffing for health                  does your library currently, or has your library ever had, any of the                                        following types of individuals available to the public?                                 social workers 126                               social work students 126                               community health workers 126                               health educator 126                               nurses 126                               americorps or other volunteers 126                               other health-related professional (describe) 126                if your library has any health-related professionals currently available                     to the public, about how often do these individuals typically                                       provide services at your library?                                 daily 126                               weekly 126                               monthly 126                               less than once a month 126                               not applicable 126                if your library could have any health-related professionals available to                                   the public, about how often do you think the        services of such                                              individual(s) would be needed at your library?                                 daily 126                               weekly 126                               monthly 126                               less than once a month 126                               not applicable 126                to your knowledge, does your library have someone on staff who you                                   would characterize as a “champion” for health-related programs,                                              services, or partnerships?                                 yes 126                               no 126                if yes, could you please briefly describe what your library’s health                                                          champion(s) do to support health-related programs, services, or                                       partnerships?     part 6. health partnerships and funding                  has your library ever worked with or received funding from any of the                following, specifically to offer health related services or programs?                                 sc center for rural and primary healthcare 123                               hands on health sc 123                               national network of libraries of medicine (nnlm) 123                               institute of museum and library sciences 123                               regional healthcare systems 123                               foundations 123                               food share sc 123                               south carolina state library 123                               clemson cooperative extension 123                               other organizations (please describe) 123                               none of the above 123                thinking about your local community, how would you characterize the                relationship between your library and the following organizations?                                 health department 123                               hospital or healthcare system(s) 123                               health coalition or alliances 123                               snap-ed implementing agency 123                               local non-profit organizations 123                               colleges or universities 123                               k-12 schools 123                               parks & recreation unit 123                               early education providers, including daycares 123                               wic clinics 123                               department of justice / department of corrections 123 part 7. health priorities                  what barriers, in your opinion, stand in the way of your library being                able to participate in efforts to support health? (select all that apply)                                 no barriers 122                               not sure where to start 122                               no one has asked us to help, or to participate in community efforts 122                               funding 122                               lack of expertise on topic 122                               lack of partners 122                               lack of space 122                               doesn't fit within the mission of our library 122                               other (please specify) 122                thinking of future continuing education opportunities, what are priorities for you in terms of library support for health? (select all that         apply)                                 how to get started with health-related services or programs 121                               how to market the availability of health-related services or                                                                       programs 121                               how to sustain health-related services or programs 121                               how to expand health-related services or programs 121                               how to partner with community collaborators 121                               how to evaluate health-related services or programs 121                               other (please describe) 121                               none of the above 121                thinking of future continuing education opportunities, how would you                most like to learn more about the topics addressed in this                questionnaire (select one)                                 from a sc public library worker who has directly worked on these                                              topics at their library 122                               from a sc public library administrator who has supervised work                                             on these topics at their library 122                               from a medical or health sciences librarian with expertise on this                                              topic 122                               from a staff member at the south carolina center for rural and                                              primary healthcare 122                               from a person in your community (e.g. local health department) 122                               from someone else (please specify) 122                thinking about the topics addressed in this questionnaire, is there                                           anything else you would like us to know?                    if you would be potentially interested in participating in an interview or                focus group about these topics, please insert your email address                here                    please include your email address to receive a $10 amazon gift card     thematic “repackaging” of the evidence summaries section and an initial focus on evidence based reference practices editorial   thematic “repackaging” of the evidence summaries section and an initial focus on evidence based reference practices   stephanie krueger associate editor (evidence summaries) senior consultant national library of technology in prague prague, czech republic email: stephanie.krueger@techlib.cz   ann medaille editor-in-chief professor, director of research & instructional services university of nevada, reno libraries reno, nevada, united states of america email: amedaille@unr.edu      2022 krueger and medaille. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30105     since the publication of the first issue in 2006, evidence based library and information practice (eblip) has published evidence summaries, which are brief critical appraisal reviews of current research articles from the library and information science (lis) literature. with the first issue of 2022 and in the coming issues, the editorial board would like to respond to a study of the eblip community (kloda et al., 2014) assessing the impact of evidence summaries (es) on helping to bridge the gap between research and day-to-day library practice. kloda et al. concluded that their survey highlighted “a potential need to improve marketing or packaging of the evidence summaries to ensure that they reach the intended audience and achieve maximum impact on practice” (p. 43).   in 2022 and 2023, we will be “repackaging” the evidence summaries section by providing critical appraisals to readers using the six domains of eblip that koufogiannakis et al. (2004) identified in their content analysis of lis research: reference, education, collections, management, information access and retrieval, and professional issues/lis education. packaged together, the value of evidence summaries as tools for practice becomes even more apparent. not only do they make collections of evidence more accessible to librarians and other information practitioners, but they also reaffirm the value of evidence that is of high quality—in other words, evidence that has been gathered through careful attention to study design and rigorous data collection and analysis. themed collections of evidence summaries emphasize the need for grounding library and information practices in solid research and assessment, which is even more critical during the rapidly changing times in which we find ourselves. when grouped together, these works emphasize the different types of evidence that can be used to answer similar questions, while also revealing patterns, contradictions, and areas for further investigation.   the five evidence summaries published in volume 17, issue 1 are focused on reference, probing the relevance of reference services today and placing a spotlight on different avenues of current research and practice in this domain: the impact of different reference staffing models, responses to covid-19, possible racial bias in virtual reference services, administrative views about such services, and the very definition of such services themselves. given the overall decline seen in recent years in traditional reference services at many (but not all!) institutions and the challenges to in-person reference services caused by recent events, the studies appraised here—several focused on virtual reference services—provide a variety of perspectives that may help librarians consider and experiment with possible changes to reference services in the coming year. they also prompt many questions. how do we best provide virtual reference services and reach more members of our communities? how do we continue to articulate the value of such services to decision makers? how might we work even more closely together to incorporate findings from others into our activities? we hope the research highlighted in this issue’s evidence summaries will ignite your curiosity and spur you to consider your practice—be it in an academic, corporate, or public library setting—in a new light.   as we implement this new approach to the evidence summaries section, we welcome your feedback on thematic groupings as well as on any other ideas you may have to improve the transfer of research to practice. on a personal level, as two instruction and reference practitioners who navigated the many (often rapid) transitions associated with covid-19 on our physical and virtual campuses, we sincerely hope that the year ahead provides us all with greater ability to contemplate our practice, using the lessons learned from the past two years to consider the broader importance of our profession to those we serve directly and to civic society more broadly.   references   kloda, l. a., koufogiannakis, d., & brettle, a. (2014). assessing the impact of evidence summaries in library and information practice. library and information research, 38(119), 29–46. http://eprints.rclis.org/24749/1/644-2873-1-pb.pdf   koufogiannakis, slater, l., & crumley, e. (2004). a content analysis of librarianship research. journal of information science, 30(3), 227–239. https://doi.org/10.1177/0165551504044668   evidence summary   academic library websites show heavy use of web 2.0 applications   a review of: boateng, f., & liu, y. q. (2014). web 2.0 applications’ usage and trends in top us academic libraries. library hi tech, 32(1), 120-138. doi:10.1108/lht-07-2013-0093   reviewed by: diana k. wakimoto associate librarian california state university, east bay hayward, california, united states of america email: diana.wakimoto@csueastbay.edu   received: 28 july 2014   accepted: 11 nov. 2014      2014 wakimoto. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to explore web 2.0 application use in academic libraries through determining: web 2.0 applications used, the purpose of using these applications, and how the use of web 2.0 is changing.   design – exploratory survey of academic library websites using content analysis of websites, blogs, and social networking service platforms.   setting – websites of academic libraries in the united states, blog platforms, and social networking services.   subjects – 100 academic libraries.   methods – the researchers based their selection of academic library websites on the us news & world report’s 2013 list of the top 100 best colleges in the united states. the authors created a checklist to determine which web 2.0 technologies were used by the academic libraries on their websites and for what purposes. the researchers searched for web 2.0 applications on the main page and one subpage down from the main page. the researchers also used keyword searches on the library’s website to find web 2.0 applications and searched blog platforms and social networking sites.   main results – the authors found that facebook and twitter were the most popular web 2.0 applications and that all of the libraries analyzed used social networking services. blogs were the second most popular web 2.0 tool at 99% participation rate, followed closely by rss (97%) and instant messaging (91%). libraries used these web 2.0 tools for information sharing including: outreach, promotion, providing online reference services, subject guides, tutorials, highlighting resources, and posting announcements.   conclusion – the academic libraries analyzed in this study use web 2.0 applications to a much greater extent than previous research had shown. the researchers expect to see increased use of web 2.0 applications by academic libraries in the coming years. they suggest that future research focus on web 2.0 use by historically black colleges in the united states and on collaboration between academic libraries and other academic units when offering web 2.0 services.     commentary   as web 2.0 and social networking tools become more commonly used in libraries, more studies are being published on the trends of usage and attitudes surrounding these tools (chu & du, 2013; del bosque, leif, & skarl, 2012). this study provides an overview of web 2.0 and social networking usage in academic libraries, similar to other contemporary studies (chu & du, 2012), and could be used as a basis for comparative studies of web 2.0 use in academic libraries in other countries and best practices for community engagement through web 2.0 tools.   overall the study’s methodology is clearly outlined, but more details about how the checkpoints were constructed and details about which blog platforms were searched would allow easier replication of the study and assessment of the results’ reliability. the study is valid according to the eblip critical appraisal checklist by glynn (2006) in part because it provides information on the sample population and the checklist used to evaluate web 2.0 usage. however, more information about how the researchers ensured reliability in the content analysis of the websites – for example, the resolution of inter-rater disputes and greater clarity on the procedure for browsing the libraries’ websites – would strengthen the article.   the study’s design allowed it to answer two out of the three research questions posed, namely of which web 2.0 applications are in use and how the academic libraries are using them. however, information about the third research question – the direction in which web 2.0 use by libraries is developing – remained largely undiscussed. additional research is needed to answer this question through a longitudinal comparison of libraries’ use of web 2.0 applications, something the authors do not suggest as an avenue for future research.   furthermore, it would be of great value to undertake a study that looks into whether the use of web 2.0 and social media is actually engaging users in order to show that libraries’ online presences are indeed social and dialogical in nature. studies on the effectiveness of web 2.0 would answer the authors’ underlying assumptions that use of web 2.0 by academic libraries is effective, which this study does not have data to support. as libraries increase their use of web 2.0 tools, studies investigating the use and effectiveness of such tools become increasingly important. this study provides an overview that may be useful for academic librarians determining the best web 2.0 tools for their particular needs. as research continues to increase on web 2.0 and social networking tools, best practices should emerge that will further assist librarians in employing the best tools for engaging their users online.   references   chu, s. k-w., & du, h. s. (2013). social networking tools for academic libraries. journal of librarianship and information science, 45(1), 64-75. doi:10.1177/0961000611434361   del bosque, d., leif, s. a., & skarl, s. (2012). libraries atwitter: trends in academic library tweeting. reference services review, 40(2), 199-213. doi:10.1108/00907321211228246   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi:10.1108/07378830610692154   article   the scholarly communications needs of faculty: an evidence based foundation for the development of library services   diane (dede) dawson science liaison librarian university of saskatchewan saskatoon, saskatchewan, canada email: diane.dawson@usask.ca   received: 7 apr. 2014     accepted: 21 oct. 2014      2014 dawson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objectives – this exploratory research seeks to broadly understand the publishing behaviours and attitudes of faculty, across all disciplines, at the university of saskatchewan in response to the growing significance of open access publishing and archiving. the objective for seeking this understanding is to discover the current and emerging needs of researchers in order to determine if scholarly communications services are in demand here and, if so, to provide an evidence-based foundation for the potential future development of such a program of services at the university library, university of saskatchewan.   methods – all faculty members at the university of saskatchewan were sent personalized email invitations to participate in a short online survey during the month of november 2012. the survey was composed of four parts: current research and publishing activities/behaviours; open access behaviours, awareness, and attitudes; needs assessment; and demographics. descriptive and inferential statistics were calculated.   results – the survey elicited 291 complete responses – a 21.9% response rate. results suggest that faculty already have a high level of support for the open access movement, and considerable awareness of it. however, there remains a lack of knowledge regarding their rights as authors, a low familiarity with tools available to support them in their scholarly communications activities, and substantial resistance to paying the article processing charges of some open access journals. survey respondents also provided a considerable number of comments – perhaps an indication of their engagement with these issues and desire for a forum in which to discuss them. it is reasonable to speculate that those who chose not to respond to this survey likely have less interest in, and support of, open access. hence, the scholarly communications needs of this larger group of non-respondents are conceivably even greater.   conclusion – faculty at the university of saskatchewan are in considerable need of scholarly communications services. areas of most need include: advice and guidance on authors’ rights issues such as retention of copyright; more education and support with resources such as subject repositories; and additional assistance with article processing charges. the university library could play a valuable role in increasing the research productivity and impact of faculty by aiding them in these areas. introduction   the association of research libraries (arl) defines scholarly communications as "the system through which research and other scholarly writings are created, evaluated for quality, disseminated to the scholarly community, and preserved for future use" (association of research libraries, n.d.). the scholarly communications landscape has arguably changed more in the last two decades than in the entire history of the academic journal (see soloman, 2013). the rise of the internet has not only enabled the rapid shift from print to online, but has also enabled the development of new tools, new formats, and even new business models for open access journal publishing.   “open access literature is digital, online, free of charge, and free of most copyright and licensing restrictions” (suber, 2004). researchers can make their articles open access by publishing in an open access journal (“gold”) or by self-archiving a copy of their manuscript in an open repository (“green”). the budapest open access initiative of 2002 (chan et al., 2002) is widely viewed as the defining event when this movement was born, and since then it has grown rapidly. in fact, lewis (2012) argues that gold open access will be the dominant mode of publishing within the next decade. the transition to an open access environment is perhaps one of the central topics in scholarly communications at present and permeates many related aspects such as impact metrics, peer review, and copyright. additionally, many institutions and major funding agencies are now mandating that their researchers and funding recipients make the products of their research openly available. researchers need to adapt to these changes and their implications quickly.   academic librarians are uniquely positioned to assist faculty in navigating this complex and rapidly evolving scholarly communications landscape. librarians deal with publishers on a routine basis as part of their professional practice and also increasingly as publishing researchers themselves. the missions of academic libraries largely involve supporting the academic and research agendas of their institutions. in light of these conditions, many academic libraries are extending their support services to encompass various scholarly communications initiatives such as hosting and managing institutional repositories, education and outreach on open access issues, establishing author’s funds to pay the article processing charges of some gold open access journals, and supporting campus-based open access journal publishing activities. at the present time, the university library, university of saskatchewan offers no services of this kind for faculty.   the university of saskatchewan is the largest university in the province of saskatchewan, canada, with more than 21,000 students and over 1000 faculty. it is a public medical-doctoral institution offering a wide range of programs and courses including many professional and post-graduate degrees. in 2011, the university of saskatchewan joined the u15 group of canadian research universities, a group of 15 research intensive universities that advocates for public policies to advance research and innovation in canada. since joining the u15, the university of saskatchewan has greatly enhanced its focus on increasing research output and metrics, and increasing performance in tri-agency funding (see promise and potential: the third integrated plan http://www.usask.ca/plan/index.php).   the three main federal funding bodies in canada are often collectively known as the “tri-agency” or “tri-council.” this group released a draft open access policy in october 2013 (see nserc, 2013) that will require fundees to make publications resulting from their funded research open access by either the green or gold route. this policy is expected to be launched in late 2014 or early 2015.   literature review   surveys of authors for opinions and awareness of open access   since the origin of the open access movement the opinions, concerns, and levels of awareness of authors have been tracked in numerous studies. although little is known locally regarding faculty attitudes on open access, many such surveys have been carried out at other institutions and more broadly by government agencies, publishers, and various interest groups over the years.   xia (2010) used a longitudinal approach to analyze these numerous surveys of researchers’ attitudes and behaviours on open access covering a period of 20 years beginning in the early 1990s. unsurprisingly, this meta-analysis discovered a steady increase over time in the awareness of researchers, as well as an increase in author participation in open access publishing. however, researchers’ concerns on the quality/reputation of open access journals, and perceived lack of peer-review in these journals, remained constant over this time (xia, 2010).   recently, two large-scale international and cross-disciplinary studies were also conducted, both between 2009 and 2011: the soap and peer surveys.   the soap (study of open access publishing) survey was financed by the european commission and is the largest study of its kind conducted to date, with almost 54,000 respondents – most of whom are active researchers. the majority of these respondents (89%) have a favourable view of open access and indicate that openly available articles are beneficial to their fields. the most significant barrier to publishing in an open access venue is the availability of funding to pay article processing charges, followed closely by the perceived lack of quality open access journals in the researcher’s discipline (dallmeier-tiessen et al., 2011).   in contrast, the peer (publishing and the ecology of european research) survey studied the perceptions, motivations and behaviours of authors and readers specifically regarding open repositories. the final report of the study concludes that although researchers have a favourable view of open access and general awareness of it, few of them associate it with self-archiving and many are confused about the different types of repositories and versions of articles posted in them (fry et al., 2011). another key conclusion of the peer study is that “academic researchers have a conservative set of attitudes, perceptions and behaviours towards the scholarly communication system and do not desire fundamental changes in the way research is currently disseminated and published” (fry et al., 2011, p. 76).   scholarly communications services offered by academic libraries   many academic libraries have responded to the quickly changing scholarly communications environment by developing a range of services to support researchers. a 2007 arl spec kit (#299) surveyed arl libraries about the nature of library-initiated scholarly communications educational activities. of the 73 responding libraries, 75% indicate that they offer such education at their institutions while 18% do not but plan to. only five responding libraries do not offer these services or another unit on campus has this responsibility (newman, blecic, & armstrong, 2007). a more recent spec kit (#332), the organization of scholarly communication services, reports that 93% of the 60 arl libraries responding to the spec kit survey offer scholarly communication services; of these, 76% indicate that the library is the main leader in this area at their institution. among the services offered, all libraries are active in advising and educating authors about copyright and retaining their copyright, and “76% of the responding libraries offer services related to hosting and managing digital content, 71% offer campus-based publishing services, and 55% provide the services associated with supporting research, publishing, and creative works” (radom, feltner-reichert, & stringer-stanback, 2012, p. 13). libraries at non-arl institutions are also offering the same kinds of services, but at somewhat lower rates of adoption (thomas, 2013).   a 2009 survey of 21 members of the canadian association of research libraries (carl) found that nearly all maintain an institutional repository for faculty self-archiving, and a majority are involved in open access educational activities and have designated individuals or teams with related responsibilities (greyson, vezina, morrison, taylor, & black, 2009). in another survey of 18 carl libraries 12 of the respondents reported having dedicated funds to support open access, nine of which include money to fund faculty article processing charges in gold open access journals (fernandez & nariani, 2011).   there are a wide variety of leadership structures currently in place in libraries to carry out these initiatives – from single individuals to committees or entire departments (burpee & fernandez, 2014; radom et al., 2012). at other institutions, scholarly communications activities have been incorporated directly into liaison responsibilities (see malenfant, 2010; and wirth & chadwell, 2010). although it is conceivable that other units on campus, such as research offices, may also provide these services to faculty, in practice it is librarians who often feel a greater mandate in the education and promotion of open access. research offices are more likely to focus on assisting researchers in successfully achieving grant funding (greyson et al., 2009).   aims   the main aim of this exploratory study is to discover the current and emerging needs of university faculty in an effort to determine if scholarly communications services are in demand and, if so, to provide an evidence based foundation for the potential future development of such a program of services. no previous research of this kind has been carried out at the university of saskatchewan. results from this study will therefore also provide a benchmark from which to compare any future data collected here.   methods   an online survey was created using fluid surveys software. the survey consisted of 18 questions in all; 4 questions involved a possible follow-up question depending on the answer given by the participant. therefore, the maximum number of questions a participant could encounter was 22. the full survey instrument is available in the appendix.   the first question of the survey: “in the last ten years have you disseminated the results of your research/artistic work?” was the only required question. respondents who answered “no” to this would be excluded from the study. this enabled the survey to collect responses only from actively publishing researchers. all other questions in the survey were not required.   questions were divided into four broad areas: current research and publishing activities/behaviours; open access behaviours, awareness, and attitudes; needs assessment; and demographics. considerable effort was made to ensure that the language in the survey questions could apply to the scholarly communications practices in a wide range of disciplines. the survey included 11 comment boxes that were distributed throughout in an effort to collect additional qualitative data; none of these boxes were required.   no incentives were offered for participation, so the survey was kept brief in order to encourage participants to complete it once started. the average time actually taken to complete the survey was just under 13 minutes.   an email invitation to participate was sent to all faculty members, in all disciplines, at the university of saskatchewan. access could not be obtained to a pre-existing email distribution list for all faculty, so instead an email list was manually constructed in excel by visiting departmental webpages. however, each department manages their own faculty lists on their webpages, so there is no consistency across campus on clearly and accurately identifying the status of individuals listed; and the lists were not always up-to-date. therefore, no effort was made to limit this survey to faculty of a particular rank or status; and it is likely that some individuals outside of faculty (e.g., sessionals or lecturers) might have been inadvertently invited to participate as well. a more practical and efficient means for creating an accurate email list for faculty could not be devised.   the email list was imported into the survey software which then generated personalized invitations for each faculty member. in total, 1327 invitations were sent. the survey remained open for the month of november 2012; two reminder emails were sent. the survey responses were anonymous.   statistical analysis of the results was performed within the survey software itself and in the statistical software package spss.   this study was granted ethical approval by the behavioural research ethics board of the university of saskatchewan.   results   of the 1327 survey invitations that were sent out, 338 responses were received of which 291 were complete. this is an overall response rate of 21.9%.   the results outlined in this section are taken only from the 291 complete responses; responses of those who did not fully complete the survey were excluded from the analysis. some respondents did not answer all of the questions in the survey – this explains why the total count for individual questions may be less than 291.   only the key findings are reported and are herein organized according to dominant themes that emerged; they do not necessarily follow the original sequence of the survey. a more complete account of the results of this study is openly available (see dawson, 2014).   open access awareness, support, and participation   participants were provided with peter suber’s (2004) definition of open access and they were then asked to assess their understanding of this term. in this study, “understanding” is being considered equivalent to “awareness.”   all 291 participants responded to this question with 91% indicating that they either understand the concept well, or have some knowledge of it (table 1). this is a high level of general awareness. only four individuals indicated that they were not aware of the concept.   although faculty claim a reasonably high level of awareness of open access, their knowledge of the details of open access options available is lower. only 33% indicate that they are aware of a subject repository in their discipline (table 2). it is unclear, however, if this seemingly low level of awareness might actually be due to the lack of these outlets for some disciplines.   faculty knowledge of hybrid journals is higher with 53% answering that they are aware if this option and a further 18% “somewhat aware.” still, nearly a third of the respondents do not know about the hybrid journal option (table 3).   the next question was designed to assess the individual’s level of support for the overarching philosophy of open access. the first paragraph of the budapest open access initiative (chan et al., 2002) was included above the question to clarify what was meant by “philosophy” of open access.     table 1 please rate your level of understanding of “open access” (n = 291) response count percentage i understand it well 95 33% i have some knowledge of it 169 58% i have heard of it but i am not sure what it is 23 8% i was not aware of it 4 1%   table 2 are you aware of a subject repository* in your discipline? *an online archive available for researchers/creators in your discipline to post copies of their works (n = 291) response count percentage yes 97 33% no 156 54% not sure 38 13%   table 3 are you aware of “hybrid journals”*? *traditional journals that offer an option to authors to make their individual articles open access for a fee (n = 290). response count percentage i am aware of this option 154 53% i am somewhat aware of this option 54 18% i was not aware of this option 82 28%     a strong majority (94%) of respondents either strongly support or somewhat support the philosophy of open access as described in the budapest open access initiative (figure 1).   respondents’ opinions on open access were also gauged by providing a number of statements and asking respondents to decide to what extent they agreed or disagreed (figures 2 & 3). respondents once again display their strong support for open access with 92% either strongly agreeing or agreeing to the statement “results of publicly-funded research should be made available for all to read without barriers.” although the respondents to this survey appear to predominantly be open access supporters, a majority (83%) also do not want to pay article processing charges with their grant money.   the level of respondents’ prior participation in open access publishing or archiving was assessed. all 291 participants responded to this question with 101 indicating that they have never made their works open access; the remaining 190 respondents have made their works open access in the past (figure 4). these 190 individuals also indicated how they did this: through an open access journal or book, self-archiving in a repository or personal website, through a hybrid journal, or “not sure how.” this last option was included for those respondents who may have delegated publishing and archiving responsibilities to co-authors. of all of these “yes” options, there were a total of 275 responses – indicating that many of the 190 “yes” respondents have participated in open access in several different ways. a follow-up comments box was provided to the 101 individuals who had not made their works open access to allow them to explain their reasons. of the 81 responses here, 43% indicated that the cost of article processing fees were too high, 20% had concerns regarding the quality of journals (i.e. no peer review, low impact factors), and 17% felt that they did not know enough about open access to be confident publishing this way (figure 4).   authors and copyright   faculty should have freedom to choose outlets to publish in. however, they should also be informed and empowered to negotiate their publication agreements in order to retain rights important to them – such as the right to deposit a copy of the manuscript in an open repository to comply with funder’s requirements. several questions investigated faculty opinions and behaviours regarding author transfer of copyright to publishers. the majority of respondents (77%) either agree or strongly agree with the statement “researchers should retain the copyright to their published works” while 79% also indicate that they do not have the time/interest/expertise to negotiate the copyright terms (figures 5 & 6).     figure 1 how would you characterize your support for the philosophy of open access as outlined in the paragraph above? (strongly support = 56%; somewhat support = 38%; somewhat oppose = 4%; strongly oppose = 0%; don’t know = 2%.   n = 289).   figure 2 results of publicly-funded research should be made available for all to read without barriers (strongly agree = 59%; agree = 33%; disagree = 5%; strongly disagree = 1%; don’t know = 2%. n = 289).   figure 3 i do not want to spend my grant funds on publishing fees (strongly agree = 36%; agree = 47%; disagree = 12%; strongly disagree = 1%; don’t know = 5%. n = 289).   figure 4 have you ever made any of your publications or artistic works available on an open access basis? how? check all that apply. (n = 291).   figure 5 researchers should retain their copyright (strongly agree = 25%; agree = 52%; disagree = 10%; strongly disagree = 1%; don’t know = 12%. n = 289).   figure 6 i do not have the time/interest/expertise to negotiate copyright terms (strongly agree = 21%; agree = 58%; disagree = 13%; strongly disagree = 2%; don’t know = 6%. n = 288).     another question asked more specifically about how the respondents handle their copyright transfer agreements from publishers. an overwhelming majority (99%) usually sign the agreement “as is” (table 4). five of the 11 remarks left in the comments box after this question center on the belief that these terms are not negotiable or participants indicate they did not know they were negotiable. of the 287 respondents to this question only 4 (1%) indicate that they modify copyright transfer agreements. a follow-up question asked these four respondents how they have modified their agreements. three have replaced the publisher’s terms with their own and one has attached an addendum.   support for possible library initiatives   when asked about possible major library scholarly communications initiatives the majority of respondents either strongly support or somewhat support (between 70% and 80%) all of them (table 5; figure 7). although the university library has an institutional repository, it is currently only available for electronic theses and dissertations and librarian research output. a repository for research publications, available to all faculty on campus, is the major initiative most favoured by respondents (78% strongly/somewhat support). hosting and support for online publications is the next most popular major initiative (76% strongly/somewhat support). a publications fund, administered by the university’s research services unit, is already in existence and will support up to $1000 of article processing charges for open access publishing. some respondents referred to this fund in their comments and remarked that there was no need to duplicate services on campus in this regard.   finally, participants were asked how they would like to learn more about, and stay up-to-date on scholarly communications topics (table 6). the top three answers, each with more than 50% of the responses, are: online guides, discipline-specific seminars, and occasional newsletters. these are relatively minor library initiatives that would require significantly less funding and staff time than those major initiatives discussed above.     table 4 how do you usually handle the copyright terms in your publishing contracts? (n = 287).   response count percentage i may or may not examine the copyright terms of the contract – i just sign it as is 111 39% i examine the copyright terms of the contract and usually sign it as is 172 60% i modify the copyright terms of the contract before signing 4 1%     table 5 how strongly would you support the following possible university library initiatives? possible initiatives strongly support somewhat support somewhat oppose strongly oppose don't know total responses (n) institutional repository for publications 97 (34%) 127 (44%) 17 (6%) 10 (3%) 37 (13%) 288 institutional repository for research data 88 (31%) 119 (41%) 24 (8%) 11 (4%) 46 (16%) 288 fund for open access authors’ fees 110 (38%) 101 (35%) 34 (12%) 14 (5%) 30 (10%) 289 hosting/support for open access journals 105 (36%) 115 (40%) 20 (7%) 7 (2%) 41 (14%) 288     figure 7 a visual representation of table 5. (“strongly support” and “somewhat support”, and “strongly oppose” and “somewhat oppose” responses are combined. n = 288, 289)   table 6 which possible university library initiatives would you find useful in order to learn about, and stay up-to-date on, scholarly communications topics (such as open access)? check all that apply. (n = 272). response count percentage online guide to resources and information 199 73% seminars/workshops tailored for your discipline/department 165 61% occasional newsletters 144 53% individual consultations with a librarian 126 46% seminars/workshops open to all  119 44% blog postings 62 23% open discussion group 49 18% all of the above 30 11%     demographics of respondents   the majority of participants in this survey conduct research in the health sciences (figure 8), have been involved in research and publishing for either 10-19 years (33%) or 20 or more years (45%), and have been awarded tenure (68%)   manova analyses were carried out in spss to ascertain if there were differences in survey responses based on any demographic criteria. it was determined that, for the most part, there were not enough responses in each demographic area to make any clear conclusions between groups of respondents.       figure 8 what is your broad discipline/research area(s)? check all that apply. (n = 288).     discussion   limitations: nonresponse bias   in an effort to increase survey response rates the invitation emails included a short but descriptive subject line: “survey on open access: invitation to participate”. the unintended result of this may have been encouraging the participation of faculty who already have an interest in this topic, and discouraging the rest. the high levels of support for open access seen throughout this survey may be indicative of this effect: the pool of faculty that responded may represent those that already have a favourable attitude in this regard. those with little interest or no opinions on the topic simply may not have responded to the survey at the same rate. therefore, it is likely that this study experienced nonresponse bias. for this reason, the results are likely skewed and cannot be viewed as generalizable to all faculty. however, keeping this in mind, several interesting themes emerged in this study.   the contradictions: authors’ rights and article processing charges   there are two striking contradictions in these results. although the pool of respondents to this survey seems to predominantly include those faculty members already supportive and knowledgeable on open access, it is startling to see their almost complete lack of action regarding authors’ rights issues such as maintaining their copyright, and their strong resistance to paying article processing charges for gold journals from grant funds. it is sobering to consider the greater extent to which these concerns might exist among the larger group of non-respondents on campus.   similar surveys of faculty have also noted this contradiction regarding copyright. moore (2011) found a very high percentage of university of toronto faculty (93%) usually sign publisher’s copyright transfer agreements as-is despite also agreeing (58%) that managing copyright is important. the university of california’s survey reports comparable results and they note that “the disconnect between attitude and behavior is acute with regard to copyright” (university of california, 2007, p. 1). this seems to be a widespread phenomenon since all responding libraries in the 2013 arl spec kit survey offer services to “advise and educate authors about copyright, retaining rights, etc.” (radom et al., 2012, p. 13). by far, the strongest scholarly communications need exhibited by university of saskatchewan faculty is in education and guidance on authors’ rights issues such as how to modify and negotiate copyright transfer agreements from publishers.   author reluctance to pay article processing charges is not a new issue, in fact some of the earliest studies of authors’ opinions on open access noted this resistance to paying fees (rowlands, nicholas, & huntington, 2004; schroter, tite, & smith, 2005). more recently, a survey of canadian researchers’ publishing behaviours found strong support for open access in principle (83%) but considerably less agreement that it is worth the financial cost (43%); and even fewer (14%) agree that funding for article processing charges is readily available (phase 5 research, 2014). the majority of gold open access journals funded in this way are in the field of biomedicine, and this is also where the highest article processing charges are (soloman & björk, 2012). the majority of respondents to the present survey are from the field of health sciences. due to the prevalence of such charges in this field it might be expected that authors are becoming accustomed to using their grant funds for this purpose – but the results herein suggest that this is not the case. it has been shown that providing authors with funds specifically to pay article processing charges offers an incentive for faculty to publish in gold open access journals (nariani and fernandez, 2012).   the university of saskatchewan publications fund is available for any costs associated with publishing – including author fees for open access journals. however, the fund is a limited pot of money so applications for this kind of support are in direct competition with other requests (such as for pages charges or reproduction of colour prints). a specific fund to pay article processing charges in addition to this fund may encourage more researchers to publish in gold journals. additionally, services to assist authors in locating open access journals that don’t charge fees would also be helpful. more investigation and discussion on how best to support authors in this area is required.   awareness vs. detailed knowledge   the results of the survey indicate that this group of faculty already has a high level of basic awareness of open access. however, more detailed knowledge may be lacking – and it is this detailed knowledge which may be necessary to enable researchers to actually follow-through and make their publications open access. it is logical to speculate that this lack of detailed knowledge is even greater among the larger group of non-respondents on campus.   one area where faculty seem to lack knowledge is in locations to archive their works: only 33% of respondents knew of a subject repository in their discipline. this could be due to the fact that not all disciplines have such repositories yet, but likely also relates to greater awareness of gold open access in comparison to green open access. the peer study found that few researchers associate open access with self-archiving (fry et al., 2011) and björk et al. (2010) reported that the gold option is more dominant in life and health sciences compared with other disciplines where the green option is more well-known. since the majority of respondents to the present survey were from the health sciences this disciplinary factor may be at play here.   other studies have reported a similar discrepancy between open access awareness and detailed knowledge. for example, moore’s (2011) survey of university of toronto faculty found that awareness of open access is very high and the principle is strongly supported but the actual understanding of the different options is more limited. morris and thorn’s (2009) research found that there is substantial support among researchers for the principle of open access, though it is unclear how many actually fully understand the issue and less than half know what self-archiving is. and swan and brown (2007) noted that researchers may assess their level of awareness and understanding of open access higher than it actually is; while they may be familiar with the concept they are not knowledgeable about how to actually carry through and make their publications open access.   it is clear that the university of saskatchewan researchers who responded to this survey are aware of open access but may need assistance in clarifying the details and options available.   engagement with open access   although a 21.9% response rate to this survey may at first glance seem to be low, it is actually similar to or higher than those attained in other comparable online surveys of university researchers (coonin & younce, 2010; kocken & wical, 2013; mischo & schlembach, 2011; moore, 2011). the university of california even states that their response rate of 22.9% is “relatively high” and that this, in addition to lengthy comments left by respondents, indicates that “faculty are strongly interested in issues related to scholarly communication” (university of california, 2007, p. 2). a similar conclusion is reached in the present study.   in total, 347 comments were left in the 11 optional textboxes distributed throughout this survey. some of these comments voiced very impassioned opinions on open access. combined with the relatively high survey response rate, this extensive use of comments boxes suggests a high level of engagement with this topic on campus, and a desire for further discussion. the university library could provide a forum to enable and facilitate these discussions in an interdisciplinary setting.   recommendations   develop authors’ rights support services. a clear outcome of this study is that there is almost no awareness or action on the part of faculty when negotiating with publishers to retain some of their rights as authors. librarians, perhaps in collaboration with the university’s copyright office, could support faculty in this area. this would require some professional development for librarians, but since research is a requirement for librarians at the university of saskatchewan we already encounter copyright transfer agreements during our activities as publishing authors. encouraging more awareness of authors’ rights issues for librarians as authors would be a reasonable first step in this direction.   expand initiatives to support authors in paying article processing charges. another clear conclusion of this study is that, even among this group of open access supporting faculty, there is strong resistance to paying article processing charges for gold journals from their grant funds. the university library could investigate options to supplement the publications fund with a fund that is specifically designed just for supporting authors publishing in gold open access journals, or by supporting emerging economic models for open access journal publishing such as peerj (by purchasing institutional publishing plans). an additional related initiative could be assisting authors in locating alternate open access outlets that do not charge fees, and raising their awareness of the green route to open access.   invest in an ongoing program of open access educational and awareness-raising initiatives. results of this study show that responding faculty have a high level of support and awareness of open access but may lack some detailed knowledge such as the tools available and practical steps to take in making their publications open access. this need is likely even greater among the larger group of non-respondents on campus. faculty indicated in this survey that online guides, discipline-specific seminars, and occasional newsletters are their preferred means to learn and stay up-to-date on scholarly communications issues.   implementation of any or all of these recommendations would require the reallocation of library financial and human resources to support them. many libraries have created a scholarly communications librarian position specifically to coordinate and lead such initiatives; and other institutions have established teams to share in these responsibilities. either way, if the university library chooses to act on these recommendations, librarian expertise and resources will need to be assigned to get these initiatives off the ground and in order to make an effective difference in supporting researchers on campus.   conclusion   the results of this study indicate that faculty at the university of saskatchewan are in considerable need of scholarly communications services. the faculty who responded to the survey are already strong supporters of open access and highly aware and engaged in the topic. however, it is likely that this survey experienced non-response bias: those individuals with prior interest and knowledge of open access were possibly more inclined to participate than those without. it is therefore reasonable to speculate that the scholarly communications needs of this larger group of non-respondents may be even greater. areas of most need include: advice and guidance on authors’ rights issues such as retention of copyright; assistance paying article processing charges or seeking alternate publishing outlets; and education and support with resources that enable open access. the need for such services is likely to increase with the implementation of the upcoming tri-agency open access policy.   librarians are the logical professionals on campus to provide such a suite of programs and services, indeed many academic libraries already offer scholarly communications services as part of their mandate to support the research mission of their institutions. the university library could play a valuable role in increasing the research productivity and impact of faculty by aiding them in these areas.   acknowledgements   thank you to g. braganza, g. ferguson, h. jacobs, b. pratt, and v. wilson for helpful comments and advice during the creation of the survey instrument; and a. liang for compiling the faculty email list. thank you to j. disano, k. clavelle, j. mccutcheon, and the social sciences research laboratories (ssrl) at the university of saskatchewan for assistance in statistical analysis of the survey results. thank you to c. hampson and c. sorensen for thoughtful comments on early drafts of this paper, and to c. polischuk and v. wilson for additional supportive and insightful advice.   this research was financially supported by the university of saskatchewan new faculty start-up fund.   references   association of research libraries. (n.d.). scholarly communication. retrieved from http://www.arl.org/focus-areas/scholarly-communication   björk, b.-c., welling, p., laakso, m., majlender, p., hedlund, t., & gudnason, g. (2010). open access to the scientific journal literature: situation 2009. plos one, 5(6), e11273. doi:10.1371/journal.pone.0011273   burpee, k. j., & fernandez, l. (2014). scholarly communication at canadian research libraries: conversations with librarians. journal of librarianship and scholarly communication, 2(2), 1–21. doi:10.7710/2162-3309.1121   chan, l., cuplinskas, d., eisen, m., friend, f., genova, y., guedon, j., ... velterop, j. (2002). budapest open access initiative. retrieved from http://www.budapestopenaccessinitiative.org/read   coonin, b., & younce, l. m. (2010). publishing in open access education journals: the authors’ perspectives. behavioral & social sciences librarian, 29(2), 118–132. doi:10.1080/01639261003742181   dallmeier-tiessen, s., darby, r., goerner, b., hyppoelae, j., igo-kemenes, p., kahn, d., … van der stelt, w. (2011). highlights from the soap project survey: what scientists think about open access publishing. retrieved from http://arxiv.org/abs/1101.5260   dawson, d. (2014). internal report: results of u of s faculty survey on scholarly communications behaviours and needs. retrieved from: http://ecommons.usask.ca/handle/10388/6290   fernandez, l., & nariani, r. (2011). open access funds: a canadian library survey. partnership: the canadian journal of library and information practice and research, 6(1), 1–24. retrieved from https://journal.lib.uoguelph.ca/index.php/perj/article/view/1424/2083   fry, j., probets, s., creaser, c., greenwood, h., spezi, v., & white, s. (2011). peer behavioural research: authors and users vis-à-vis journals and repositories. retrieved from http://www.peerproject.eu/fileadmin/media/reports/peer_d4_final_report_29sept11.pdf   greyson, d., vezina, k., morrison, h., taylor, d., & black, c. (2009). university supports for open access: a canadian national survey. canadian journal of higher education, 39(3), 1–32. retrieved from http://spectrum.library.concordia.ca/6566/1/university_supports_for_oa.pdf   kocken, g., & wical, s. (2013). “i’ve never heard of it before”: awareness of open access at a small liberal arts university. behavioral and social sciences librarian, 32(3), 140–154. doi:10.1080/01639269.2013.817876   lewis, d. w. (2012). the inevitability of open access. college and research libraries, 73(5), 493–506. doi:10.5860/crl-299   malenfant, k. j. (2010). leading change in the system of scholarly communication: a case study of engaging liaison librarians for outreach to faculty. college and research libraries, 71(1), 63–76. doi:10.5860/crl.71.1.63   mischo, w. h., & schlembach, m. c. (2011). open access issues and engineering faculty attitudes and practices. journal of library administration, 51, 432–454. doi:10.1080/01930826.2011.589349   moore, g. (2011). survey of university of toronto faculty awareness, attitudes and practices regarding scholarly communication: a preliminary report. retrieved from https://tspace.library.utoronto.ca/handle/1807/26446   morris, s., & thorn, s. (2009). learned society members and open access. learned publishing, 22(3), 221–239. doi:10.1087/2009308   nariani, r., & fernandez, l. (2012). open access publishing: what authors want. college and research libraries, 73(2), 182–195. doi:10.5860/crl-203   newman, k. a., blecic, d. d., & armstrong, k. l. (2007). spec kit 299: scholarly communication education initiatives. washington, dc: association of research libraries. retrieved from http://publications.arl.org/scholarly-communication-spec-kit-299/   nserc. (2013, october 15). draft tri-agency open access policy. retrieved from http://www.nserc-crsng.gc.ca/nserc-crsng/policies-politiques/tri-oa-policy-politique-la-trois_eng.asp   phase 5 research. (2014). canadian researchers’ publishing attitudes and behaviours. retrieved from http://www.cdnsciencepub.com/learning-centre/impact-and-discovery/researcher-survey-results.aspx   radom, r., feltner-reichert, m., & stringer-stanback, k. (2012). spec kit 332: organization of scholarly communication services. washington, dc: association of research libraries. retrieved from http://publications.arl.org/organization-of-scholarly-communication-services-spec-kit-332/   rowlands, i., nicholas, d., & huntington, p. (2004). scholarly communication in the digital environment: what do authors want? learned publishing, 17(4), 261–273. doi: http://dx.doi.org/10.1087/0953151042321680   schroter, s., tite, l., & smith, r. (2005). perceptions of open access publishing: interviews with journal authors. british medical journal, 330(7494), 756–759. doi:10.1136/bmj.38359.695220.82   solomon, d. j. (2013). digital distribution of academic journals and its impact on scholarly communication: looking back after 20 years. the journal of academic librarianship, 39(1), 23–28. doi:10.1016/j.acalib.2012.10.001   solomon, d. j., & björk, b.-c. (2012). a study of open access journals using article processing charges. journal of the american society for information science and technology, 63(8), 1485–1495. doi:10.1002/asi.22673   suber, p. (2004). a very brief introduction to open access. retrieved from http://legacy.earlham.edu/~peters/fos/brief.htm   swan, a. & brown, s. (2007). researcher awareness and access to open access content through libraries: a study for the jisc scholarly communications group. retrieved from http://www.jisc.ac.uk/whatwedo/topics/opentechnologies/openaccess/reports/researcherawareness.aspx   thomas, w. j. (2013). the structure of scholarly communications within academic libraries. serials review, 39(3), 167–171. doi:10.1016/j.serrev.2013.07.003   university of california. (2007). faculty attitudes and behaviours regarding scholarly communication: survey findings from the university of california. retrieved from http://osc.universityofcalifornia.edu/2007/08/report-on-faculty-attitudes-and-behaviors-regarding-scholarly-communication/   wirth, a. a., & chadwell, f. a. (2010). rights well: an authors’ rights workshop for librarians. portal: libraries and the academy, 10(3), 337–354. doi:10.1353/pla.0.0105   xia, j. (2010). a longitudinal study of scholars attitudes and behaviors toward open-access journal publishing. journal of the american society for information science and technology, 61(3), 615–624. doi:10.1002/asi.21283     appendix: the survey instrument   open access publishing and faculty at the university of saskatchewan: an exploratory study   part a: current research & publishing activities/behaviours   in the last ten years have you disseminated the results of your research/artistic work? yes no in the last ten years how have you disseminated the results of your research/artistic work? please estimate the number of items in each category.     none 1-10 items 10 + items published a peer-reviewed journal article published a book contributed a chapter to an edited book published a paper in a conference proceedings given a conference presentation or poster displayed work in an exhibition, or installation other (please specify):           how do you usually handle the copyright terms in your publishing contracts?   i may or may not examine the copyright terms of the contract – i just sign it as is i examine the copyright terms of the contract and usually sign it as is i modify the copyright terms of the contract before signing comments:       in what ways have you modified the terms in your contracts with publishers? check all that apply.   i have replaced the publisher’s terms with my own i have attached an addendum (such as the sparc author addendum) other (please specify): ______________________     do you produce a large amount of data in digital format* in your research/artistic work?  *for example: analyses, measurements, counts, images, music, film, etc   yes no sometimes   do you have concerns about storing and managing this data and/or providing access to this data to other researchers/creators?    yes no sometimes comments:           part b: open access behaviours, awareness, and attitudes   open access definition: open-access (oa) literature is digital, online, free of charge, and free of most copyright and licensing restrictions. what makes it possible is the internet and the consent of the author or copyright-holder. there are two primary vehicles for delivering oa for scholarly works: oa journals (or books), and oa archives or repositories. oa journals perform peer review and then make the approved contents freely available to the world. oa archives or repositories do not perform peer review, but simply make their contents freely available to the world. (based on peter suber’s “a very brief introduction to open access” http://www.earlham.edu/~peters/fos/brief.htm)   please rate your level of understanding of “open access."   i understand it well i have some knowledge of it i have heard of it but i am not sure what it is i was not aware of it comments:       "philosophy" of open access   the budapest open access initiative (paragraph 1): an old tradition and a new technology have converged to make possible an unprecedented public good. the old tradition is the willingness of scientists and scholars to publish the fruits of their research in scholarly journals without payment, for the sake of inquiry and knowledge. the new technology is the internet. the public good they make possible is the world-wide electronic distribution of the peer-reviewed journal literature and completely free and unrestricted access to it by all scientists, scholars, teachers, students, and other curious minds. removing access barriers to this literature will accelerate research, enrich education, share the learning of the rich with the poor and the poor with the rich, make this literature as useful as it can be, and lay the foundation for uniting humanity in a common intellectual conversation and quest for knowledge. (http://www.soros.org/openaccess/read)   how would you characterize your support for the philosophy of open access as outlined in the paragraph above?   strongly support somewhat support somewhat oppose strongly oppose don’t know comments:     are you aware of “hybrid journals”*?  *traditional journals that offer an option to authors to make their individual articles open access for a fee    i am aware of this option i am somewhat aware of this option i was not aware of this option   have you ever made any of your publications or artistic works available on an open access basis? check all that apply.    yes, through self-archiving (in an online repository or personal website) yes, through publishing in an oa journal or book yes, through a hybrid journal  yes, but i’m not sure how no (please comment below on any particular reasons you may have for not making your publications available on an open access basis) comments:     to what extent do you agree or disagree with the following statements?      strongly agree agree disagree strongly disagree don’t know results of publicly-funded research should be made available for all to read without barriers open access leads to the publication of poor quality research open access will increase the citations to, and impact of, my publications researchers should retain the copyright to their published works open access publications are not properly peer-reviewed i do not have the time/interest/expertise to negotiate the copyright terms in my publishing contracts i do not want to spend my grant funds on publishing fees i have trouble telling the scam publishers apart from the legitimate open access publishers my current tenure and promotion standards discourage me from making my publications open access comments:       are you aware of a subject repository* in your discipline?  *an online archive available for researchers/creators in your discipline to post copies of their works   yes no not sure   do you currently serve as an editor for a traditional (non-open access) publication?  yes no   do you currently serve as an editor for an open access publication?  yes no   many universities are now implementing open access mandates requiring researchers to deposit copies of their publications in open online repositories. if the university of saskatchewan established an institutional repository, how strongly would you support a similar mandate here?   strongly support somewhat support somewhat oppose strongly oppose don’t know comments:       scholarly communication costs money. whom do you think should be responsible for the publication costs? check all that apply.   note: the first two options comprise the majority of the current model the university library through subscriptions to for-profit publishers  the university library and researchers through subscriptions and membership fees to scholarly societies funding agencies your department/school/college authors readers other (please specify): ______________________ comments:       part c: needs assessment   many academic libraries have developed services to support the scholarly communications activities of researchers at their institutions. the questions in part c are intended to ascertain the level of support for the development of similar services at the university of saskatchewan.   how strongly would you support the following possible university library initiatives?     strongly support somewhat support somewhat oppose strongly oppose don't know an institutional repository for open archiving of publications an institutional repository for open archiving of digital research/artistic data a library-administered fund to help pay authors’ fees in open access or hybrid journals hosting and support services for online publications   comments:     which possible university library initiatives would you find useful in order to learn about, and stay up-to-date on, scholarly communications topics (such as open access)? check all that apply.   seminars/workshops open to all  seminars/workshops tailored for your discipline/department occasional newsletters blog postings online guide to resources and information individual consultations with a librarian open discussion group all of the above other suggestions? please comment below.     part d: demographics   questions in this section are intended to ascertain any trends related to respondents’ discipline, experience, and rank.   what is your broad discipline/research area(s)? check all that apply:   agriculture and bioresources business/management/finance education engineering fine arts health sciences (medicine, vet. medicine, dentistry) humanities/music kinesiology law library and information science life sciences mathematics/computer science physical sciences social sciences other (please specify): ______________________   how many years have you been actively involved in research and publishing (or the creation and display of artistic work)?    0-9 years 10-19 years 20 years or more   have you been awarded tenure at the university of saskatchewan?    yes no other (please specify): ______________________   how many years has it been since you were awarded tenure at the university of saskatchewan?   0-9 years 10-19 years 20 years or more   please hit submit to complete the survey. microsoft word es_kloda.doc evidence based library and information practice 2006, 1:1  83 evidence based library and information practice       evidence summary    undergraduate students do not understand some library jargon typically used in  library instruction      a review of:   hutcherson, norman b. “library jargon: student recognition of terms and concepts commonly  used by librarians in the classroom.” college and research libraries 65.4 (july 2004): 349‐54.    reviewed by:   lorie a. kloda  instructional technology librarian, life sciences library, mcgill university  montreal, quebec, canada  e‐mail: lorie.kloda@mcgill.ca    received: 20 december 2005  accepted: 13 february 2006      © 2006 kloda. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to determine students’ level of  recognition for 28 commonly used terms in  library instruction.    design – survey, multiple‐choice  questionnaire.    setting – large state university library in  the united states (this is assumed from the  author’s current affiliation).    subjects – 300 first‐ and second‐year  university students enrolled in a library  skills course between september 2000 and  june 2003.    methods – two 15‐question multiple‐choice  questionnaires were created to verify  students’ understanding of 28 terms  commonly used in library instruction, or  “library jargon”. each questionnaire  included 12 unique terms and, in order to  ensure consistency between questionnaire  results, three common terms. for each  question, a definition was provided and  four terms, including the correct one, were  offered as possible answers.  four variants  of each survey were developed with varied  question and answer order.  students who  completed a seven‐week library skills lab  received one of the two questionnaires. lab  instructors explained the objective of the  survey and the students completed them in  10 to 15 minutes during class time. of the  300 students enrolled in the lab between  september 2000 and june 2003, 297 returned  completed questionnaires. the researcher  used microsoft excel to calculate descriptive  statistics, including the mean, median, and  standard deviation for individual  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:1  84 questionnaires as well as combined results.   no demographic data were collected.    main results – the mean score for both  questionnaires was 62.31% (n=297).  that is,  on average, students answered 9.35 out of 15  questions correctly, with a standard  deviation of +‐4.12. students were able to  recognize library‐related terms to varying  degrees.  terms identified correctly most  often included:  plagiarism (100%), reference  services (94.60%), research (94.00%),  copyright (91.58%), and table of contents  (90.50%).  terms identified correctly the  least often included:  boolean logic (8.10%),  bibliography (14.90%), controlled  vocabulary (18.10%), truncation (27.70%),  and precision (31.80%).  for the three terms  used in both questionnaires, results were  similar.      conclusion – the results of this study  demonstrate that terms used more widely  (e.g. plagiarism, copyright) are more often  recognized by students compared with  terms used less frequently (e.g. boolean  logic, truncation).  also, terms whose  meanings are well‐understood in everyday  language, such as citation and authority,  may be misunderstood in the context of  library instruction.  for this reason, it can be  assumed that students may be confused  when faced with this unfamiliar  terminology.  the study makes  recommendations for librarians to take  measures to prevent misunderstandings  during library instruction such as defining  terms used and reducing the use of library  jargon.      commentary    the study sample was presumably drawn  from a single unidentified institution in the  united states.  no demographic data were  collected from the participants, making it  difficult to generalize results beyond first‐  and second‐year undergraduate students at  that institution.  the response rate of 99% is  large enough to make estimates, however  the sample size in relation to the size of the  population is not known, making the  statistical significance of the data unclear.   the author does not include information  about the sample group’s enrolment in the  information skills lab – was participation  voluntary or a requirement of the program?   possible confounding factors include self‐ selection of participants in the lab, previous  library instruction, and students’ areas of  study.    the author states that the questionnaire was  piloted in the first semester of its use, and  some terms were replaced on the  recommendation of faculty and students.   the consistency of the three common terms  used in both variations of the questionnaire  also confirms the survey’s internal reliability.   however, the lack of inclusion of the  instrument, including the definitions  employed for each of the library jargon  terms, makes it difficult to assess the  external validity of the instrument.  there is  no way to determine whether students’  correct answers confirm their  comprehension of the meaning of a  particular term, or conversely, whether their  incorrect answers translate to their lack of  comprehension.  the article does not  provide an overview of the content or  outcomes of the lab, and it is unclear at what  point surveys were administered ‐‐ during,  before, or after the skills lab.  it is assumed  that the lab included the use of most, if not  all, of the terms in the survey, but it is  unclear as to whether definitions were  provided for any of these.  the study does  not attempt to measure the success of an  educational intervention.    this study employs a quantitative  methodology to answer a question about  understanding or meaning of terminology   used in library instruction.  a qualitative  research design, such as one using focus  evidence based library and information practice 2006, 1:1  85 groups, would have also been appropriate  for discovering more information about the  students’ perceptions of library jargon.  do  students want to learn the correct  definitions, or would they prefer that  librarians select simpler terms?  for those  terms which they identified incorrectly,  what do they think the terms mean?  these  questions could not be addressed using the  multiple‐choice questionnaire format  employed.    this study attempts to answer an important  question for academic librarians: how well   do our students understand terms and  concepts presented during orientation and  information literacy instruction sessions?                                                               while many librarians already suspect that  some students do not know the meaning of  some library jargon, this study attempts to  quantify the proportion of students who can  and cannot correctly recognize 28 terms.   these results suggest that those involved in  the design and delivery of library  instruction as well as those involved in the  design of websites and library signage need  to consider providing alternative  terminology or definitions for commonly  misunderstood terms.    evidence based library and information practice evidence based library and information practice 2010, 5.2  evidence based library and information practice       news/announcements    call for studies on quality improvement for inclusion in systematic review       2010. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which  permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly  attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the  same or similar license to this one.    the north west quality improvement  systematic review group in the u.k., are  currently undertaking a systematic review  on quality improvement in health libraries.   we would welcome any contributions from  colleagues in alerting us to unpublished  studies or reports which examine / evaluate  quality improvement systems (performance  indicators, standards, benchmarking or  other quality systems / tools) in health  libraries.      please e‐mail victoria kirk, clinical  librarian at victoria.kirk@nhs.net.     on behalf of the north west quality  improvement systematic review group  (uk)  118 mailto:victoria.kirk@nhs.net microsoft word es2_917_final_final_final evidence based library and information practice 2008, 3:1  57 evidence based library and information practice       evidence summary    computer‐assisted library instruction and face‐to‐face library instruction prove  equally effective for teaching basic library skills in academic libraries    a review of:    zhang, li, watson, erin m. and banfield, laura.  ʺthe efficacy of computer‐assisted instruction  versus face‐to‐face instruction in academic libraries:  a systematic review.ʺ  the journal  of academic librarianship 33.4 (july 2007):  478‐484.      reviewed by:  stephanie walker  acting chief librarian & executive director of academic information technologies   brooklyn college – city university of new york   brooklyn, ny, usa  e‐mail:  swalker@brooklyn.cuny.edu       received: 28 november 2007      accepted: 23 january 2008      © 2008 walker. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    abstract    objective – to conduct a systematic review  of several studies comparing the efficacy of  face‐to‐face versus computer‐assisted  instruction (cai) for teaching basic library  skills to patrons of academic libraries.    design – systematic review of existing  studies (randomised controlled trials and  controlled trials).      setting ‐ college and university libraries     subjects – the subjects studied were  patrons of any type of academic library,  whether university, college, or other post‐ secondary institution, receiving instruction  in basic library skills. ten studies were  included in the review, of which seven were  done in the united states, two in australia,  and one in canada. the total number of  subjects in all of the studies under review  was 1283. nine of the studies focused on  undergraduates enrolled in specific courses  (undergraduate courses ranging widely in  subject area, or in one case a first year  experience program); the other study  focused on library instruction methods  taught to students in a graduate research  methods course, yet the study was still  intended to measure the efficacy of library  evidence based library and information practice 2008, 3:1  58 instruction methods, yet the study was still  intended to measure the efficacy of library  instruction methods.      methods – one included study was a  randomised controlled trial; the other nine  were controlled trials. the date range under  consideration was for studies done between  1990 and 2005. all original studies were  required to compare the efficacy of face‐to‐ face versus cai instruction. both  information skills and students’ reactions to  receiving the instruction were considered.  to identify appropriate studies, searches  were done across the following library and  education‐related databases: lisa, eric,  and library literature. the authors screened  the 728 unique studies’ bibliographic  information for relevance against four  criteria: studies had to be of a particular  type of design (randomised controlled trials,  controlled trials, cohort studies, and case  studies), with a sample size greater than one  and with pre‐ and post‐test measurements;  study participants had to be academic  library patrons; the study needed to  compare cai and face‐to‐face instruction;  and both the students’ information skills  and reactions to the instruction had to be  measured. this left 40 unique studies, which  were then retrieved in full text. next, studies  were selected to meet the inclusion criteria  further using the quorom format, a  reporting structure used for improving the  quality of reports of meta‐analyses of  randomised trials (moher et al 1896‐1900).   evaluation of methodological quality was  then done using a dual method: authors  watson and zhang assessed the studies  independently, each using the “checklist for  study quality” developed by downs and  black (downs and black 377‐384), adapted  slightly to remove non‐relevant questions.   after analysis, when additional information  was needed, original study authors were  contacted.  finally, ten studies were  included in the analysis.      the instruction sessions covered many  topics, such as catalog use, reading citations,  awareness of library services and  collections, basic searching of bibliographic  databases, and more. but all could qualify as  basic, rather than advanced, library  instruction.  all studies did pre‐ and post‐ tests of students’ skills – some immediately  after instruction, and others with a time  lapse of up to six weeks. most authors  created their own tests, though one adapted  an existing scale. individual performance  improvement was not studied in many cases  due to privacy concerns.      main results ‐ nine of the ten studies  found cai and face‐to‐face instruction  equally effective; the tenth study found face‐ to‐face instruction more effective. the  students’ reaction to instruction methods  varied – some students felt more satisfied  with face‐to‐face instruction and felt that  they learned better, while other studies  found that students receiving cai felt more  confident.  some found no difference in  confidence.      it was impossible to carry out a meta‐ analysis of the studies, as the skills taught,  methods used, and evaluation tools in each  case varied widely, and the data provided  by the ten studies lacked sufficient detail to  allow meta‐analysis. as well, there were  major methodological differences in the  studies – some studies allowed participants  the opportunities for hands‐on practice;  others did not. the cai tutorials also varied  – some were clearly interactive, and in other  studies, it was not certain that the tutorial  allowed for interactivity.      the authors of the systematic review  identified possible problems with the  selected studies as well.  all studies were  evaluated according to four criteria on the  modified downs‐black scale: reporting,  external validity, and two measures of  internal validity (possible bias and possible  evidence based library and information practice 2008, 3:1  59 confounding).  a perfect score would have  been 25; the mean score was 17.3. areas  where authors lost points included areas  such as failure to estimate data variability,  failure to report participants lost to follow‐ up, failure to have blind marking of pre‐  and post‐tests, failure to allocate  participants randomly, and a variety of  other areas. as well, few studies examined  participants’ confidence level with  computers before they participated in  instruction.      conclusion – based on this systematic  review, cai and face‐to‐face instruction  appear to be equally effective in teaching  students basic library skills. the authors of  the study are reluctant to state this  categorically, and issue several caveats: a)  only one trial was randomised; b) seven of  the studies were conducted in the usa, with  the others being from canada and australia,  and learning and teaching styles could be  very different in other countries; c) the  students were largely undergraduates, and  the authors are curious as to whether results  would be similar with faculty, staff, or older  groups (though of course, not all  undergraduates are traditional  undergraduates); d) the tests ranged widely  in design, and were largely developed  individually, and the authors recommend  developing a validated test; and e) if the  pre‐ and post‐tests are identical and given in  rapid succession, this could skew results.      commentary – the fact that the trials  included in the systematic review were not  randomised is of some concern, but this  does not negate the usefulness of the work.   it would be of greater concern if the lack of  randomisation included students being  permitted to choose their method of  instruction; but this does not appear to have  been the case. lack of randomisation was  more a factor in the choice of the test  subjects. studies were done on particular  groups of people, such as classes of  students. it is often difficult or impossible to  have randomisation in studies like this due  to budgetary constraints or lack of access to  other potential participants. given this, if  the objective is simply to evaluate the  effectiveness of cai vs. face‐to‐face  instruction, controlled trials should suffice,  as no one wishes to disadvantage any  particular group by randomising who does  or does not receive instruction.  that nine of  the ten studies selected were controlled  trials speaks to this ethical dilemma.      as well, studies conducted across the usa,  canada, and australia are perhaps less  likely to vary as widely as studies  conducted in the usa compared to studies  conducted in countries which are less well‐ supplied with technology. this is an issue to  consider when evaluating the usefulness of  this work. if one conducted similar studies  in developing countries, results might differ  substantially.        students were indeed largely  undergraduates, but that is where libraries  hope to ‘catch’ them – to inculcate library  skills at an early stage in their academic  careers. so evaluating the effectiveness of  cai vs. face‐to‐face instruction seems most  important for academic librarians who wish  to determine how best to spend their time  and resources – on developing cai to  relieve us of the often‐heavy burden of  dozens of sections of identical or near‐ identical library instruction, or on face‐to‐ face instruction, should that have been  absolutely proven to be more effective.      the final two concerns of the authors, as  expressed in their conclusions, seem to merit  the greatest discussion. to take them in  reverse order:  how much is it possible that  test results were biased in some studies if  the pre‐ and post‐tests were identical, and,  worse still, given in close time proximity?  and how much effect did diversity in the  tests have on the results?  given that most  evidence based library and information practice 2008, 3:1  60 found that cai and face‐to‐face instruction  were similarly effective, this latter point  may not be a huge concern, as there was a  level of consistency across many studies.  however, the authors are correct when they  state that there is more to be done.  recommended changes would be a  validated and consistent test, tried across  multiple institutions, with pre‐ and post‐ tests that were both different and given at  set times in the course. also, an interactive  version of cai, with randomised and  validated quiz questions generated to test  learning, could be useful. as the authors  stated, it is not clear that all cai was  interactive; yet interactivity enhances the  user experience.      one additional concern is that while the  studies reviewed all fall within the date  range of the systematic review (1990‐2005),  the aim of the study was to evaluate current  practices in cai. there have been great  technological advancements in the latter  portion of the date range, and the usage of  the web has expanded enormously.  additionally, user comfort with technology  has likely grown in recent years. additional  research focusing on latter years could be  useful.     this systematic review contributes to our  understanding of the state of knowledge  about cai vs. face‐to‐face instruction for  library instruction, and points to directions  for further development. this article is also  useful for instruction librarians interested in  developing cai; it would also be helpful to  libraries struggling with the question of  whether cai can indeed substitute for face‐ to‐face instruction in some cases. cai is  often seen as a way to reduce teaching load  for instruction librarians, though it can have  huge up‐front time and financial investment  requirements; this study may help  instruction librarians and others decide  whether such investment is likely to be  worthwhile for their situations.      works cited  downs, sara h. and black, nick. ʺthe  feasibility of creating a checklist for  the assessment of the methodological  quality both of randomised and non‐ randomised studies of health care  interventions.ʺ journal of  epidemiology and community health  52.6 (1998): 377‐384.   moher, david et al. ʺimproving the quality  of reports of meta‐analyses of  randomised controlled trials:  the  quorom statement.ʺ lancet 354.9193  (1999): 1896 ‐ 1900.   research article   university students’ changing library needs and use: a comparison of 2016 and 2018 student surveys   jung mi scoulas clinical assistant professor and assessment coordinator university of illinois at chicago, university library chicago, illinois, united states of america email: jscoul2@uic.edu      sandra l. de groote professor and head of assessment and scholarly communications university of illinois at chicago, university library chicago, illinois, united states of america email: sgroote@uic.edu   received: 10 aug. 2019                                                                  accepted: 6 jan. 2020      2020 scoulas and de groote. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29621     abstract   objective – this study examines differences in university students’ library use and satisfaction (e.g., in-person library visits, online and print resource use, space satisfaction, and library website use) between 2016 and 2018 based on local survey data. it also discusses how these findings provided guidance for future planning and action.       methods – the academic university library developed the surveys for undergraduate and graduate students and distributed them in spring 2016 and 2018. both student surveys focused on examining students’ needs relative to library resources and services, although the 2018 student survey also attempted to quantify students’ library visits and their use of library resources. while the surveys were not identical, the four questions that appeared in both surveys (i.e., library visits, resource use, library space satisfaction, and library website use) were recoded, rescaled, and analyzed to measure the differences in both surveys.   results – the survey results reveal that students’ library visits and use of library resources in 2018 were higher than in 2016. in particular, undergraduate students’ use of library resources in 2016 were lower than those in 2018, whereas graduate students’ use of library resources remained similar in both years. another key finding indicates that the mean score of students’ library quiet study space satisfaction in 2018 was higher than in 2016. however, when compared to the 2016 survey, there was a decrease in students’ ease of library website use in the 2018 survey.   conclusion – assessing students’ behavior and satisfaction associated with their use of library physical spaces, resources, and services should be conducted on an ongoing basis. over time, the survey findings can be used as evidence based data to communicate patterns of users’ behavior and satisfaction with various stakeholders, identify areas for improvement or engagement, and demonstrate the library’s impact. survey results can also inform further strategic and assessment planning.     introduction   academic libraries have utilized various assessment measures to understand users’ needs, improve their services, and further demonstrate the value of the library. surveys are a popular assessment tool widely used for exploring users’ needs in academic libraries (liebst & feinmark, 2016; matthews, 2007). while various user surveys are conducted by academic libraries, two major types of user surveys are standardized surveys (e.g., libqual+, measuring information service outcomes) and local surveys developed by individual academic libraries (e.g., montgomery, 2014; scoulas & de groote, 2019). regardless of the type of user surveys, many academic libraries attempt to use an evidence based approach by reviewing the survey findings, using those findings for making decisions, and monitoring changes over time (dennis, greenwood & watson, 2013; greenwood, watson & dennis, 2011; mccaffrey & breen, 2016; mccaffrey, 2019; montgomery, 2014; norton, tennant, edwards & pomputius, 2018; taylor & heath, 2012).   the university of illinois at chicago (uic) library began using surveys as a way to understand users’ perceptions and needs related to the university library’s spaces, services and resources. the findings have allowed for an evidence based approach to identify areas for change or improvement. the principle involved in this effort is to establish the culture of assessment within the library, “an organizational environment in which decisions are based on facts, research, and analysis, and where services are planned and delivered in ways that maximize positive outcomes and impacts for customers and stakeholders” (lakos & phipps, 2004, p. 352). this has been done using a standardized survey like libqual+ (2002, 2006, and 2012), as well as local surveys focused on specific aspects (e.g., library website, space, services) as needed. the university library decided to discontinue use of libqual+ because it was not always possible to apply the findings to decision making and because of complaints that the survey was too complicated to take. locally focused surveys had been conducted at various times at the uic library to help guide changes related to space (e.g., adding furniture, remodeling spaces, installing a coffee shop). however, a major challenge was a lack of an instrument to benchmark and monitor users’ perceptions of library resources, services, and space over time and to measure the impact of students’ library use on their academic success. beginning in 2015, the assessment coordinator advisory committee (ac2) at the uic library developed user experience surveys for students (2016 and 2018) and faculty (2017 and 2019).   the purpose of this study is to examine any reported measurable differences in university students’ experiences using the library between 2016 and 2018. the study also explores which survey findings provide actionable data for the library to use, and it also describes the actions the university library took as a result of the survey data. this paper will be a useful guide for librarians who:   are considering developing their own instrument to capture users’ needs and to track those changes over time; already have a locally developed instrument but are considering revising it to collect more meaningful data; are not sure how to take action based on their own findings from previous user surveys; want to strengthen the culture of assessment within their library.   literature review   academic libraries have exerted great effort to establish a culture of assessment using evidence based approaches to explore what library users want and how to provide easy access to library services and resources. a survey of academic libraries in the united states (u.s.) showed that more than 70% of academic libraries used “assessment data to improve practice” (farkas, hinchliffe & houk, 2015, p. 157). most important, the goal is to improve the quality of the library’s resources and services for users based on the results of the assessment efforts. improving user services and demonstrating the value of the library to its stakeholders are among the primary missions of academic libraries. in a fast-paced academic environment, academic libraries cannot play a large role in students’ efforts to accomplish their academic goals without understanding students’ needs and preferences.   user surveys at academic libraries and use of findings   many academic libraries have implemented user surveys to understand users’ behaviors, attitudes toward and satisfaction with library services and resources so as to improve current practices based on the results. one of the popular and widely used standardized user surveys used by academic libraries is libqual+. libqual+ is a web-based survey tool administered by the association of research libraries (arl). since 2000, more than 1,300 academic and public libraries have used this survey tool (arl, n.d.). libqual+ is regarded as a useful tool for librarians and administrators to effectively gather users’ feedback, a stable instrument to continuously track users’ behavior over time, and a benchmark to help libraries gauge their success against other institutions (arl, n.d., hinchliffe, 2015; mccaffrey, 2019; taylor & heath, 2012). several academic libraries using libqual+ explored how users’ perceptions of library services and resources changed over time and how the academic libraries implemented changes in response to the survey results (dennis et al., 2013; greenwood et al., 2011; mccaffrey, 2019; mccaffrey & breen, 2016; taylor & heath, 2012). in spite of the popularity of the libqual+ survey, there are reports of limitations with the tool. challenges of using libqual+ include participants having difficulty in completing the survey due to too many questions and similarity among questions (voorbij, 2012); difficulty in understanding the fixed survey questions, such as minimum, perceived, and desired levels of service quality (thompson, cook, & health, 2000); difficulties in connecting the libqual+ data with locally collected statistics (e.g., gate counts); a lack of flexibility to customize questions; and difficulty reading  results (dennis et al., 2013). dennis and colleagues (2013) suggested that alternative survey methods besides libqual+ are needed to measure changes within the library.   several academic libraries have developed their own local user surveys to gauge users’ behaviors and needs and used the findings for improvement (e.g., montgomery, 2014; ojennus & watts, 2017). benefits of using a local survey are that academic libraries can customize their questions and response options and focus on a specific area that they intend to investigate (e.g., space). for example, ojennus and watts (2017) conducted an online survey in 2015 of all students at whitworth university in order to examine how they used the library (e.g., space and technology) and identify possible areas for improvement. after comparing their findings with trends identified in the literature, the authors concluded that their findings revealed local needs and interests. as a result, the library director at whitworth university made several changes to library space and amenities (e.g., offering free coffee and making more private rooms available during finals week) and addressing problems with wireless access in collaboration with the it department (ojennus & watts, 2017). ojennus and watts (2017) further stated that they plan to continue surveying users to collect longitudinal data to monitor the trends and evaluate “the efficacy of our responses to them” (p. 333).   while many surveys may be one-time investigations into users’ needs, other researchers have studied user needs over extended periods of time using local surveys. for instance, montgomery (2014) at the olin library at rollins college conducted surveys two years in a row using the same survey questionnaire to determine how users’ perceptions of their learning behavior changed before and after library space renovations. the author commented that they were satisfied with users’ feedback and were able to better understand users’ learning behaviors at the library. the university of florida health science center library used a survey developed by the university of southern california health sciences libraries staff and examined how health science users’ awareness of technology (e.g., mobile apps) and their interest in using technology to access library resources and services (e.g., citation tools and library electronic resources) from their mobile devices changed from 2012 to 2016 (norton et al., 2018). the authors stated that “the annual review of survey results has allowed librarians to identify the local users’ needs and interests as they changed over time and has led to incremental changes in services offered” (p. 329).   in spite of the advantages of conducting local surveys, some drawbacks were raised in the literature: a lack of national benchmarking data and challenges related to reliable and valid survey questions (hinchliffe, 2015). in addition, few studies addressed repeating user surveys over time to monitoring users’ needs in various areas, such as space, usage of resources, or technology.   patterns of students’ library experience   with the internet and advanced technology, college students can easily access books and journals remotely using their own computer or mobile device without entering the library. the ability to remotely access e-books, journals, databases, and services has shaped users’ library use patterns over 15 years. previous studies showed that easy access to online resources and students’ expectations of the library influenced students’ library use pattern (de groote, hitchcock, & mcgowan, 2007; lee, ritterbush, & sivigny, 2010). in two separate studies conducted during periods of 14 years and 15 years, students’ in-person library visits declined, whereas their reference questions via email and phone increased (de groote et al., 2007; lee et al., 2010). according to statistics from arl, the patterns of students’ library visits also confirm that among 123 member libraries the number of users accessing the library (gate counts) declined at 42% of academic institutions in the u.s. from 2015 to 2018. in addition, in a recent article, cohen (2019) stated that the trend in students checking out print books decreased over the past decade, whereas the availability of online articles and e-journal downloads increased.   academic libraries efforts to meet users’ needs related to library space (e.g., floor renovation, extending hours, reorganizing spaces) have shifted from being collection oriented to user focused by incorporating users’ feedback. as a result, students’ perceptions of library space have evolved over time; library space is becoming less traditional and more conducive to social learning. data from 2007 to 2014 from the university of limerick in ireland and an international consortium of arl and the society of college, national and university libraries indicated that quiet space is increasingly considered important for library users (mccaffrey & breen, 2016). this pattern is also confirmed by a recent study done by mccaffrey (2019) showing that users’ perceptions of quiet space have improved from 2007 to 2016. other academic libraries have also observed trends related to students using library space, indicating that students used library space not only for working alone but, depending on their learning needs, also for working with their peers (montgomery, 2014; scoulas & de groote, 2019). in one library, after rearranging furniture in the library spaces, the overall library space usage from 2015 to 2018 increased 15% and use of group study tables and a new group study area increased about 270% (oberlander, miller, mott & anderson, 2019).   assessing students’ needs on the library website is critical for academic libraries: “library websites are a gateway to library resources, services, contact information, and events” (anderson, 2016, p. 19). while previous studies show that usage of library websites has declined over time (allen, baker, wilson, creamer & consiglio, 2013; anderson, 2016), libraries continue to improve their websites for students’ use. mierzecka and suminas (2017) examined which features of the library website are most important for students at the university of warsaw in poland and the vilnius university in lithuania via open-ended responses. they found that the top five important features of the library website were (ranked in order): the online library catalogue, information about the location and opening hours, login account access, the online collection, and a floor map showing reading rooms. students’ library website experiences were also examined after changes were made to the libraries’ websites.   aims   the aim of this study is to examine the patterns of students’ library use and satisfaction (in-person library visits, resource use, space satisfaction, and library website use) based on responses to surveys distributed to students in 2016 and 2018. this study also describes how these results were used at evidenced based data to provide guidance for a plan of action.    methods   institutional setting   uic is a large public research university classified by the carnegie classification as having highest research activity. more than 30,000 students are enrolled in all of its 15 colleges, and students have access to two large libraries in chicago (an arts, humanities, sciences, social sciences, and engineering library and a health sciences library) and three smaller health sciences libraries located at uic’s regional areas (peoria, rockford, and urbana).   survey development   the ac2, which consisted of library faculty representing various units (research and instruction, collections, website, administration, assessment and scholarly communications) in the university library, developed the surveys for undergraduate and graduate students. the first locally developed survey, consisting of 19 questions, was distributed to the institution’s students in 2016 (see appendix a). prior to distribution, the surveys were piloted with 6 to 8 students who read the questions and shared their thoughts aloud to allow the research team to observe if there were any issues with interpreting the questions. at the time of the data analysis, it was determined that effectively analyzing all of the data was not possible and that changes to the survey would be needed. for the 2018 student survey, the ac2 reviewed the 2016 student survey questions and findings and revised some scales (e.g., converting dichotomous to interval scales for frequency of in-person or online library visits and library resource use), wording, and format of the survey (see appendix b). for further details of how the 2018 student experience survey was revised, refer to aksu dunya and de groote’s article (2019). as a result of these changes, the surveys were not identical and comparisons with all data points were not possible. however, several questions did remain in both surveys and comparisons of the results to these questions were conducted and reported in this paper.   measures   student library visits in person and use of library resources   the response scales in the 2016 survey regarding students’ in-person library visits and use of library resources were different from those used in the 2018 survey. for example, in the 2016 survey in the past year, have you visited the library at your campus site for study or research? and in the past year, have you used [library resources]? had a nominal scale of yes (1) and no (0), whereas in the 2018 survey last semester, how often did you visit the university library? and last semester, how often did you use [library resources]? had an ordinal scale of never (0) to daily (4). these items were recorded, as follows: any responses from (1) to (4) in the 2018 survey data were coded as yes (1) and the rest as no (0) to match those used in the 2016 data.   student library space satisfaction   the scales for the questions related to student library space satisfaction (i.e., quiet study space and collaborative/group space) in the 2016 survey were coded very satisfied (1), satisfied (2), neutral (3), dissatisfied (4), very dissatisfied (5), and i do not use [this space] (6). on the other hand, the same questions in the 2018 survey were coded from i don’t use this space in the library (0), very dissatisfied (1), dissatisfied (2), satisfied (3), and very satisfied (4). given that the scales in the 2016 survey were reverse coded in comparison to the 2018 survey scale, the 2016 data was recoded so that, for example, very satisfied (1) becomes very satisfied (5) and i do not use [this space] (6) to i do not use [this space] (0) to match with the 2018 data. prior to rescaling, the frequency of i do not use this space (0) response in the 2016 and 2018 surveys was analyzed. afterwards, the i do not use this space (0) response was dropped from both the 2016 and 2018 survey results because this response affects the calculation of the mean scores. because the 2016 survey used the 6-point likert scale format including neutral (3) and the 2018 survey was a 5-point likert scale format, it was not possible to directly compare the mean scores. therefore, the data was rescaled by using the formula developed by preston and colman (2000): (rating – 1) / (number of categories – 1) x 100. this method is used as a way to compare survey results when the surveys used different scales.   to rescale the results of the survey, the 5-point likert scale data (1, 2, 3, 4, 5) in the 2016 survey was converted to a continuous one (0, 25, 50, 75, 100), and the 4-point likert scale data (1, 2, 3, 4) in the 2018 survey was converted to a continuous one (0, 33.33, 66.67, 100). another adjustment was made for the library space satisfaction questions. the space satisfaction questions in the 2016 survey included quiet study space and collaborative/group space, but the 2018 survey questions included quiet study space, collaborative space, and group study room, separately. to compare the library space satisfaction in both surveys, the responses of collaborative and group space in the 2018 survey were calculated as a mean score. student use of library website   the scales for the question related to how easy it was to use features of the library website (e.g., finding a book, searching for journal articles) were originally coded as a 6-point likert scale, from very difficult (1) to i have not used this (6) in the 2016 survey. however, these scales were coded as a 5-point likert scale in the 2018 survey from i don’t use this service (0) to very easy (4). for the procedure of recoding and rescaling data for students’ library website usage, the authors followed the same methods as described for library space satisfaction.   preferred location for studying   students were also asked why they studied in places other than the library by selecting all of the responses that apply. while the options were not the same for both years, several options were relatively similar: more study space, quieter study space, food/drink availability, and equipment or software that i need are available.   how to spend funding   students were also asked how they would spend funding to improve the library. they could select two options in the 2016 survey and three options in the 2018 survey. not all of the options were the same in both surveys, but the options from both surveys included more computers, more quiet study spaces, and more electric outlets. however, more food and drink options was included only in the 2018 survey.     participants   the total number of respondents in 2016 was 1,087 (response rate of 4%), whereas the number of respondents in 2018 was 2,277 (response rate of 8%). this reflects a 109% increase over the response rate of the 2016 survey. both surveys contained key demographics, including gender, age, class level (e.g., undergraduate and graduate), first generation status, and commuter status. as shown in table 1, it appears that the ratio of respondents from the 2016 and 2018 surveys was similar to the ratio of the 2018 university population (a difference of less than 5%) in all of the demographic variables, except for two categories: female respondents and graduate students were overrepresented in both surveys. as such, on the whole, the survey respondents were representative of the university’s population.   there are slight differences between the 2016 survey and 2018 survey respondents’ demographics. the percentage of student respondents in the age group between 16 and 25 increased almost 10% from 2016 (59.52%) to 2018 (69.43%), whereas the rest of the age groups slightly decreased. the percentage of undergraduate respondents increased about 9% from 2016 (48.30%) to 2018 (56.96%), and the first generation respondents increased more than 2% from 2016 (10.86%) to 2018 (13.22%). some of the respondents’ demographics were similar in both surveys. for example, in both surveys, female students (61.64% in 2016 and 63.68% in 2018) were more likely to participate in the survey than male students (38.36% in 2016 and 36.14% in 2018). in additional, the majority of the respondents in both surveys (85.19% in 2016 and 85.68% in 2018) were commuters, meaning they did not live on campus.   data collection   students’ demographic information was obtained from the office of institutional research (oir) with the participating students’ consent for both the 2016 and 2018 surveys. the demographic information requested for both surveys included gender, age, class level, first generation status, and commuter/resident/online status (see table 1). the oir uploaded a “panel” in qualtrics that contained all students’ email addresses and demographic information. students were sent an email from qualtrics requesting their participation in the survey. the procedures for collecting survey responses remained the same for both surveys and are outlined in detail in the study of scoulas and de groote (2019). as an incentive, all survey respondents were able to enter a drawing for one of three ipads in the 2018 survey. no incentives were offered with the 2016 survey.   table 1 comparison of students’ demographic information: 2016 and 2018   sample total 2018 student population (n = 28,725)   2016 (n = 1,087) 2018 (n = 2,277) gender, n (%)            female 670 (61.64%) 1,450 (63.68%) 15,201 (52.92%)      male 417 (38.36%) 823 (36.14%) 13,408 (46.68%)      unknown 4 (0.18%) 116 (0.40%) age group, n (%)            16-25 647 (59.52%) 1,581 (69.43%) 20,598 (71.71%)      26-35 297 (27.32%) 509 (22.35%) 6,206 (21.60%)      above 35 143 (13.16%) 187 (8.21%) 1,921 (6.69%) class, n (%)            undergraduate 525 (48.30%) 1,297 (56.96%) 18,886 (65.75%)      graduate 536 (49.31%) 980 (43.04%) 9,839 (34.25%) first generation, n (%) 118 (10.86%) 301 (13.22%) 4,801 (16.71%) transfer, n (%) 211 (19.41%) 463 (20.33%) 6,890 (23.99%) residency, n (%)            commuters 926 (85.19%) 1,951 (85.68%) 24,584 (85.58%)      resident 109 (10.03%) 276 (12.12%) 3,114 (10.84%)      online 52 (4.78%) 50 (2.20%) 1,027 (3.58%)   data analysis   all data was analyzed using spss 25. descriptive statistics were used in response to q1, student library visits and resource use. to test whether a statistically significant difference in student library space satisfaction and use of the library website existed in the 2016 and 2018 survey data, an independent sample t-test was used. before conducting any statistical tests, the requirements of the assumptions of each test were checked. the 2016 response and 2018 response distributions were sufficiently normal for the purposes of conducting a t-test (e.g., skew < |2.0| and kurtosis < |9.0|; schmider, ziegler, danay, beyer & bühner, 2010). in addition, the assumption that homogeneity of variances in spread scores is equal in different groups of cases was tested and was not met via levene’s f test. therefore, adjusted degrees of freedom were used.   results   comparison of student in-person library visits and resource use   table 2 shows the student library visit and library resource use in the 2016 and 2018 survey data. with respect to their frequency of library visits, the results indicate that student library visits slightly increased from 2016 to 2018. in additional, the frequency of student library resource use (journal articles, books, and databases) moderately increased from 2016 to 2018.   students’ in-person library visits and library resource use were further organized by class level (undergraduate students and graduate students) in order to show whether or not there were differences in their library use between surveys. figure 1 shows that overall undergraduate students visited the library more than graduate students in both years. undergraduate students’ in-person library visits in 2018 (51.30%) were higher than in 2016 (46.00%), whereas graduate students’ in-person library visits in 2018 (35.31%) were slightly lower than in 2016 (38.91%).   table 2 comparison of student in-person library visits and use in both surveys   2016 (n = 1,087) 2018 (n = 2,277)   n % n % in-person visits 923 84.91 1,972 86.61 use of library resources 850 78.20 1,946 85.46   regarding students’ use of library resources, as shown in figure 2, the patterns by class level across the two surveys were different. for example, undergraduate students’ use of library resources in 2016 (37.63%) was lower than that in 2018 (44.62%), whereas graduate students’ use of library resources remained similar in both years (40.57% in 2016 and 40.84% in 2018). undergraduate students’ use of library resources (37.63%) was lower than graduate students’ (40.57%) in 2016, but in 2018 undergraduate students’ use of library resources (44.62%) was higher than graduate students’ use (40.84%).   library space usage and satisfaction   prior to analyzing the comparison of students’ library space satisfaction, the percentages of the respondents answering that they did not use a space were analyzed (see figure 3). this information allowed us to see whether or not the patterns of students’ library space usage have changed over time. as shown in figure 3, the respondents who indicated not using the space for both quiet study space and collaborative/group space (11.32% and 19.69%) in 2016 were higher than those in 2018 (9.00% and 17.13%). the findings suggest that the respondents in 2018 were more likely to use the quiet study space and collaborative/group space than respondents in 2016.   figure 1 students’ in-person library visits by class level.   figure 2 students’ use of library resources by class level.   figure 3 percentage of students responding that they do not use quiet study space or collaborative/group space in 2016 and 2018.   an independent sample t-test was conducted to compare further students’ library space satisfaction in 2016 and in 2018. as shown in table 3, the results show that there was a statistically significant difference in the scores of quiet study space for the 2016 surveys (m = 67.47, sd = 26.65) and 2018 surveys (m = 73.84, sd = 24.15); t(1356) = -5.72, p < .001, d = 0.25. this result suggests that the average student satisfaction in the quiet study space in the 2018 survey was higher than in the 2016 survey. however, there was no significant difference in the mean scores of collaborative/group study space for 2016 (m = 65.93, sd = 25.46) and 2018 (m = 68.25, sd = 23.47); t(1322) = -1.97, p = .05, d = 0.09.   library website use and ease of use   to see the pattern of students’ library website use in 2016 and 2018, the responses indicating that students did not use a service were compared. the findings show that the percentages of respondents who did not use services such as asking a librarian for assistance on im/chat; finding films, videos, or online images; and booking a group study room in 2016 were higher than those in 2018, meaning that students in 2018 tended to use those services more than students in 2016 (see figure 4). on the other hand, the percentages of respondents who did not use services such as finding a print book, requesting a print book, logging into my account, and subject and course guides in 2016 were lower than those in 2018, suggesting that students in 2016 were more likely to use those services than students in 2018.   next, an independent sample t-test was conducted to investigate whether there were differences in students’ ease of using the library website between the 2016 and 2018 surveys. as shown previously in table 3, the results of the t-test indicate that there were statistically significant survey differences in the ease of library website use, except for the finding media and booking a group study room features, when comparing the 2016 survey with the 2018 survey at the level of p < .05.   in 2016, the top reasons for studying in places other than the library included more study space (42.41%), quieter study space (37.72%), and food/drink availability (32.84%). however, in the 2018 survey, quieter study space was not in the top three responses. instead, i can find a seat was the top selection (46.90%), an option that was not available in the 2016 survey. studying in other places because of the availability of food and drink increased (44.80%) and studying in other places with more study space decreased (33.90%). in 2016, students indicated they would spend funding on more online resources (24.38%), more computers (21.25%), and more quiet study spaces (21.16%). in 2018, students indicated they would spend funding on more food and drink options (32.37%), an option which was not available in the 2016 survey. they also indicated the desire for more quiet study space (31.14%) and more electric outlets (30.52%); compared to the previous survey, both of these preferences increased compared to the previous survey.   table 3 comparison of average student library space satisfaction and library website ease of use in both surveys   2016 (2018)a     n m sd t df p 95% confidence interval (lower vs. higher) cohen’s db student library space satisfaction                   quiet study space 777 (1,799) 67.47 (73.84) 26.65 (24.25) -5.72 1356 *** -8.55 -4.18 0.25 collaborative/group space 686 (1,231) 65.93 (68.25) 25.46 (23.47) -1.97 1322   -4.64 -.01 0.09 ease of use of library website features                   searching for journal articles 748 (1,813) 79.08 (72.79) 23.32 (22.53) 6.27 1350 *** 4.32 8.26 0.27 finding a print book 601 (1,391) 74.63 (68.18) 25.47 (25.42) 5.19 1136 *** 4.01 8.89 0.13 requesting a print book 460 (1,114) 78.37 (66.76) 26.08 (26.26) 8.02 862 *** 8.77 14.46 0.44 logging into my library account 477 (1,027) 81.13 (70.66) 24.04 (25.83) 7.68 991 *** 7.80 13.15 0.18 asking a librarian for assistance on im/chat 281 (1,038) 80.61 (76.56) 22.99 (23.84) 2.60 456 * .98 7.11 0.17 using library subject and course guides 515 (1,051) 75.78 (69.84) 24.94 (23.24) 4.53 960 *** 3.36 8.51 0.24 finding films, videos, or online images 202 (845) 66.46 (67.26) 28.40 (25.62) -.37 284   -5.10 3.50 0.03 booking a group study room 282 (950) 71.37 (69.61) 26.97 (26.34) .96 452   -1.82 5.33 0.07 aresults from the 2018 survey are provided in parentheses. bcohen’s d: 0.2= small effect, 0.5= moderate effect, 0.8= large effect. * p < .05. *** p < .001.   figure 4 percentage of students responding that they do not use a website feature in 2016 and 2018.   discussion   patterns of students’ library use   using the 2016 and 2018 survey results, the current study examined if there were differences in students’ library use in four areas: library visits, resources, library space, and the website. the findings of this study illustrate that, even within a short period of time between surveys (in this case, 2 years), there were differences in users’ library use. from 2016 to 2018, there was an increase in students’ library visits, resource use, and satisfaction, and there was a decrease in the ease of library website use. finding an increase in library visits is different from the decrease in students’ library visits indicated in the literature (arl [statistics from 2016 to 2018]; de groote et al., 2007; lee et al., 2010). however, students’ resource use was similar to what has been shown in the literature with an increase in students’ online resource use such as e-journals (arl [statistics from 2016 to 2018]; cohen, 2019). continued efforts to provide space for students may have contributed to the increase in undergraduate use of the library. the findings about the increased use of library resources may be associated with libraries providing a higher number of resources. in fact, according to arl statistics, from 2016 to 2018 the volumes and e-books in the university’s library collection have increased 37.16% and 60.44%, respectively. use of the library decreased between the survey period for graduate students. it could be that the heavy presence of undergraduate students was a deterrent to graduate student use.   from analyzing findings to taking action   the second goal of this paper is to discuss how the findings from the survey were used to take further action. the findings from the surveys and comparative data were presented to the ac2, to the steering committee, and to all library staff to share the overall patterns of students’ library experiences in various areas and demonstrate how the library is doing. the findings were also shared with external stakeholders like the faculty advisory committee at the uic to demonstrate the role and investment of the library in efforts to support students’ academic success and to seek further insight and feedback into the findings. the results of both surveys and their comparisons were also used as evidence based data to further shape the strategic plan and the university library’s assessment plan.   the quantitative results regarding increases in certain areas (library visits, resource use, and space satisfaction) did not lead to new decisions or actions but rather acknowledgement that there were areas where the library was doing well. on the other hand, declining satisfaction results related to services were indicators that further information should be sought and potential action taken. discussion of some of these findings and the actions taken by the library follow.   independent of findings from the 2016 survey, the library’s website had undergone a redesign. unfortunately, the respondents to the 2016 student survey found using the library website to be easier than the respondents to the 2018 student survey. this pattern of library website user satisfaction was very important. anecdotal evidence from librarians suggested that users were encountering some difficulties when navigating the library website. the responses to the 2018 survey provided clearer evidence that additional changes are needed. it is possible that students who participated in the 2016 survey used the library website before the re-design, so they found the old library website easier to use. however, even newer students reported challenges with the website. no immediate actions were taken to revise the webpage, but longer-term planning includes further redesign of the webpage.    according to lakos and phipps (2004), one example of whether a culture of assessment exists in a library is that the organization has “relevant data and [that] user feedback are routinely collected, analyzed, and used to set priorities, allocate resources, and make decisions” (p. 353). incorporating users’ feedback into the decision-making process and making improvements based on the evidence are part of the effort to strengthen a culture of assessment within the library. some of the key findings from the 2016 survey suggested that students were not satisfied with quiet study space, access to computers, and electrical outlets. to address these concerns, additional outlets were installed in several areas in the library. to increase students’ quiet study space, in spring 2017 the main library of uic increased the availability of designated quiet study space from one floor to two floors in addition to providing more single study furniture. further, and again independent of 2016 survey results, a complete renovation was done at the library of the health sciences between 2017 and 2018, which provided additional study space, more electrical outlets, a coffee shop, additional computers, additional seating, extended hours, and—for a short period of time—microwaves.   it would appear that as a result of the space renovations, overall satisfaction with the library spaces increased. however, access to space remained a challenge. this is likely due in part to a 5% increase in undergraduate enrollment and because space issues were already a problem in 2016. without increasing the actual square footage of the library, the overcrowding issue remains a challenge to solve. there are also not many more places to add additional outlets. findings related to the increase in students’ quiet space satisfaction are similar to the patterns observed in other studies. for example, in spite of the issue of insufficient space, other academic libraries that continued to exert efforts to improve their space by reorganizing study areas and dealing with noise problems based on users’ feedback did find that students’ perceptions of quiet space improved over time (mccaffrey, 2019; mccaffrey & breen, 2016).   comparing two surveys to understand the patterns of users’ library experiences was useful; however, it is not always possible to understand why an increased or decreased result occurred. mccaffrey (2019) argues that “detailed comparisons between two surveys can be problematic, particularly when analyzed at a question, user group or dimension level, as scores can increase or decrease for reasons that may be unknown or difficult to explain” (p. 72). however, comparisons between two surveys provide a valuable lesson. libraries must keep asking core questions and monitoring users’ responses regularly to monitor their experiences and satisfaction with resources and measure the impact of the library on students’ academic success over time. in order to capture students’ responses and more accurately measure the library’s impact on students’ success, the ac2 decided to revise the response scales, wording, and content (aksu dunya & de groote, 2019). however, this revision was a very important step in repeating user surveys over time that provide more meaningful evidence to the university library staff. in addition, “the effectiveness of the revision” was supported by the significant increase in the response rate (aksu dunya & de groote, 2019, p. 54). although the revisions of both surveys resulted in adjustments to the scales within those surveys to accurately compare them, the results provide reliable, informative, and meaningful evidence. this is important because libraries are expected to follow the steps of the methods rigorously tested and used by researchers (e.g., holmes, & mergen, 2013; preston & colman, 2000).   while the adjustments to the coding and scaling that were made to compare both surveys were not the primary focus of this paper, it is important to mention that adjustments to surveys may be needed in subsequent years to collect the most useful data. this paper demonstrates that institutions should be prepared from the beginning and be open to adjusting their tools as needed in order to develop the ideal benchmarking tools that can be used consistently over time. if a library has data from slightly different surveys (e.g., different scales and wording) that requires minor adjustments such as rescaling, this process should be done carefully by following the methods used by researchers (e.g., holmes, & mergen, 2013; preston & colman, 2000) to make the findings reliable and accurate. the authors also want to highlight that developing a tool for benchmarking does not mean that questions need to be identical in each survey. a certain number of questions can be used for assessing users’ needs related to new services or how to address different needs based on changes in technology, student enrollment, or other influences.   this study is unique compared to other studies reported in the literature because few of those studies have examined how libraries track and report their impact over time by using local user experience surveys. by monitoring user trends over time, this study expands current research on academic libraries’ efforts to use evidence based data for improvements, decision-making, and future library assessments. those practices include reviewing the assessment tool, revising it as needed, and taking action based on the findings; these are integral to reinforcing a culture of assessment. these practices provide valuable and meaningful information that guide librarians who plan to use local surveys to monitor users’ experiences over time and determine what to address in their next assessment plan.   conclusion   assessing users’ behaviors and satisfaction associated with their use of library physical spaces, resources, and services should be conducted on an ongoing basis in order to observe and respond to evolving trends. determining how often this type of survey will be conducted and monitored to explore these trends over time will vary based on the availability of data at libraries and how the libraries will use the data. the uic library plans to conduct student surveys every other year. a specific set of questions that be asked each time so that the library can better understand and react to changes in user needs and interests.     equally important to conducting assessment is what academic libraries do in response to the assessment findings. over time the findings from surveys can be used as evidence based data to communicate with various stakeholders for indicating the patterns of users’ behavior, identify areas for improvement, demonstrate the library’s impact, and develop a deeper understanding of users. findings can also be used when developing strategic plans and a library assessment plan. to accomplish this, the findings from the surveys need to be carefully reviewed and, when feasible, used as the basis of responding to users’ needs in order to improve the library spaces, resources, and services. not only that, it is critical to re-assess users’ experiences by comparing present and future survey results with the findings of previous assessments. this cycle of assessment will be critical for customizing and targeting services that are useful for the diverse student body served by an academic library. establishing a culture of assessment in academic libraries begins with assessing how users perceive the services and resources provided by the library and improving users’ experiences based on these findings.   acknowledgements   we greatly appreciate the support of our colleagues of the university of illinois at chicago’s ac2, who contributed their expertise to the assessment. we also thank carl lehnen and mary shultz for reviewing the manuscript.   references   aksu dunya, b., & de groote, s. (2019). revision of an academic library user experience survey. performance measurement and metrics, 20(1), 48–59. https://doi.org/10.1108/pmm-10-2018-0026   allen, l., baker, n., wilson, j., creamer, k., & consiglio, d. (2013). analyzing the miso data: broader perspectives on library and computing trends. evidence based library and information practice, 8(2), 129–138. https://doi.org/10.18438/b82g7v   anderson, l. (2016). library website visits and enrollment trends. evidence based library and information practice, 11(1), 4–22. https://doi.org/10.18438/b8f918   association of research libraries. (n.d.). arl statistics. retrieved from https://www.arlstatistics.org/repository    association of research libraries (n.d.). libqual+ 2018 survey. retrieved from http://www.libqual.org/publications   cohen, d. (2019, may 26). the books of college libraries are turning into wallpaper. the atlantic. retrieved july 25, 2019, from https://www.theatlantic.com/ideas/archive/2019/05/college-students-arent-checking-out-books/590305/    de groote, s. l., hitchcock, k., & mcgowan, r. (2007). trends in reference usage statistics in an academic health sciences library. journal of the medical library association: jmla, 95(1), 23–30. https://www.ncbi.nlm.nih.gov/pmc/articles/pmc1773032/   dennis, m., greenwood, j., & watson, a. (2013). libqual revisited: further analysis of qualitative and quantitative survey results at the university of mississippi. journal of academic librarianship, 39(6), 512–516. https://doi.org/10.1016/j.acalib.2013.08.009   farkas, m., hinchliffe, l., & houk, a. (2015). bridges and barriers: factors influencing a culture of assessment in academic libraries. college & research libraries, 76(2), 150–169. https://doi.org/10.5860/crl.76.2.150   greenwood, j. t., watson, a. p., & dennis, m. (2011). ten years of libqual: a study of qualitative and quantitative survey results at the university of mississippi 2001-2010. journal of academic librarianship, 37(4), 312–318. https://doi.org/10.1016/j.acalib.2011.04.005   hinchliffe, l. j. (2015). osservatorio: library assessment and user surveys in academic librarianship in the united states. aib studi, 55(2), 247–257. https://doi.org/10.2426/aibstudi-11198   holmes, d. s., & mergen, a. e. (2013). converting survey results from four-point to five-point scale: a case study. total quality management and business excellence, 25(1–2), 175–182. https://doi.org/10.1080/14783363.2013.799330   lakos, a., & phipps, s. (2004). creating a culture of assessment: a catalyst for organizational change. portal: libraries and the academy, 4(3), 345–361. https://doi.org/10.1353/pla.2004.0052   lee, m., ritterbush, j., & sivigny, r. (2010). reference at the commons: a case study. reference services review, 38(1), 81–89. https://doi.org/10.1108/00907321011020743   liebst, a., & feinmark, d. (2016). tools of academic library assessment: the user survey. journal of library administration, 56(6), 748–755. https://doi.org/10.1080/01930826.2016.1199144   matthews, j. r. (2007). library assessment in higher education. westport, conn.: libraries unlimited.   mccaffrey, c. (2019). transforming the university library one step at a time: a ten year libqual + review. new review of academic librarianship, 25(1), 59–75. https://doi.org/10.1080/13614533.2018.1511438   mccaffrey, c., & breen, m. (2016). quiet in the library: an evidence-based approach to improving the student experience. portal, 16(4), 775–791. https://doi.org/10.1353/pla.2016.0052   mierzecka, a., & suminas, a. (2018). academic library website functions in the context of users’ information needs. journal of librarianship and information science, 50(2), 157–167. https://doi.org/10.1177/0961000616664401   montgomery, s. e. (2014). library space assessment: user learning behaviors in the library. journal of academic librarianship, 40(1), 70–75. https://doi.org/10.1016/j.acalib.2013.11.003   norton, h. f., tennant, m. r., edwards, m. e., & pomputius, a. (2018). use of annual surveying to identify technology trends and improve service provision. journal of the medical library association, 106(3), 320–329. https://doi.org/10.5195/jmla.2018.324   oberlander, c., miller, b., mott, e., & anderson, k. (2019, may 3). how data improved our library space. the chronicle of higher education. retrieved from https://www.chronicle.com/interactives/20190505-oberlander   ojennus, p., & watts, k. a. (2017). user preferences and library space at whitworth university library. journal of librarianship and information science, 49(3), 320–334. https://doi.org/10.1177/0961000615592947   preston, c. c., & colman, a. m. (2000). optimal number of response categories in rating scales: reliability, validity, discriminating power, and respondent preferences. acta psychologica, 104(1), 1–15. https://doi.org/10.1016/s0001-6918(99)00050-5   schmider, e., ziegler, m., danay, e., beyer, l., & bühner, m. (2010). is it really robust? reinvestigating the robustness of anova against violations of the normal distribution assumption. methodology, 6(4), 147–151. https://doi.org/10.1027/1614-2241/a000016     scoulas, j. m., & de groote, s. l. (2019). the library’s impact on university students’ academic success and learning. evidence based library and information practice, 14(3), 2–27. https://doi.org/10.18438/eblip29547   taylor, m., & heath, f. (2012). assessment and continuous planning: the key to transformation at the university of texas libraries. journal of library administration, 52(5), 424–435. https://doi.org/10.1080/01930826.2012.700798   thompson, b., cook, c., & health, f., (2000) the libqual+ gap measurement model: the bad, the ugly, and the good of gap measurement. performance measurement and metrics, 1(3), 165–178.  https://doi.org/10.1108/eum0000000007216   voorbij, h. (2012). the use of libqual+ by european research libraries. performance measurement and metrics, 13(3), 154–168. https://doi.org/10.1108/14678041211284704     appendix a 2016 student survey   i have read the “agreement to participate” document and agree to participate in this research. o  yes o  no   in the past year, have you visited the library at your campus site for study or research? o  yes o  no o  i am an online student only   how satisfied are you with the quiet study space at your library? o  very satisfied o  satisfied o  neutral o  dissatisfied o  very dissatisfied o  i do not use quiet space in the library   how satisfied are you with the collaborative/group space at your library? o  very satisfied o  satisfied o  neutral o  dissatisfied o  very dissatisfied o  i do not use group space in the library   how satisfied are you with the computers available at your library? o  very satisfied o  satisfied o  neutral o  dissatisfied o  very dissatisfied o  i do not use the computers in the library   if you study in places other than your library, what do you like about those spaces? [check all that apply.] o  more study space o  quieter o  food/drinks are sold here o  equipment or software that i need are available o  other (please specify):___________________ o  i do not study in spaces other than my library       in the past year, have you used the library books, e-books, databases, journal articles, or other library resources?   o  yes o  no   how satisfied are you with the library resources you have used in the past year?     have not used this very satisfied satisfied neutral dissatisfied very dissatisfied textbooks on reserve in the library o    o    o    o    o    o    books (other than required textbooks) o    o    o    o    o    o    e-books (other than required textbooks) o    o    o    o    o    o    e-journals o    o    o    o    o    o    databases o    o    o    o    o    o    streaming videos o    o    o    o    o    o    online patient care tools o    o    o    o    o    o    dvds on reserve o    o    o    o    o    o    other, please specify o    o    o    o    o    o      what two specific library resources have you used the most in the past year?                 please rank up to 3 resources you use most often for research projects, 1 being the most important (where you go first) o  search box on the library website o  library subject/research guides o  library of the health sciences website (chicago, peoria, rockford, or urbana) o  the health sciences gateway o  database a-z list o  google or some other search engine (bing, yahoo, etc.) o  wikipedia o  academic search engine such as google scholar o  blackboard o  other (please specify):_____________________   think about your satisfaction with the library services you have used in the past year. how satisfied were you with each service?     have not used this very satisfied satisfied neutral dissatisfied very dissatisfied a class session where a librarian taught research skills for a specific assignment o    o    o    o    o    o    a library workshop that teaches how to use online resources o    o    o    o    o    o    assistance from staff in the library o    o    o    o    o    o    ask a librarian by im/chat, phone, or email o    o    o    o    o    o    a research consultation with a librarian (scheduled appointment) o    o    o    o    o    o    request a book or article from another library o    o    o    o    o    o    specialized research assistance from the subject librarian for my department/college o    o    o    o    o    o    group study rooms in the library o    o    o    o    o    o      think about how you used the library website in the past year. how easy was it to use each feature below?     have not used this very easy easy neutral difficult very difficult find a book at the university o    o    o    o    o    o    login to my account to renew a library book or check the status of a request o    o    o    o    o    o    search for journal articles on my topic o    o    o    o    o    o    user a library subject/research guide to find material by subject o    o    o    o    o    o    find films, videos, or online images at the university o    o    o    o    o    o    request a book or article from another library o    o    o    o    o    o    ask a librarian for assistance by email or im chat o    o    o    o    o    o    make an appointment for research help o    o    o    o    o    o    book a group study room online o    o    o    o    o    o    some other activity (please specify): [                               ] o    o    o    o    o    o        as part of your online course(s), have you used the university library e-books, databases, journal articles or other library resources or services for assignments or other course-related activities? o  yes o  no   in the past year, have you used the library website to find e-books, databases, e-journals, or other library resources?   o  yes o  no   think about the library services you used in the past year in your online program. how satisfied are you with each service?     have not used this very satisfied satisfied neutral dissatisfied very dissatisfied requesting an article from the university library or another library o    o    o    o    o    o    online access to full-test articles and e-books o    o    o    o    o    o    online im/chat research help o    o    o    o    o    o    telephone research help o    o    o    o    o    o    virtual consultation with a librarian o    o    o    o    o    o    online workshops about library research (e.g., finding books, journals, requesting electronic materials, finding literature and journal articles) o    o    o    o    o    o    self-paced tutorials about library research o    o    o    o    o    o      when should the library offer virtual consultations or online workshops with a librarian to provide assistance with library research? select you most preferred time.   o  8am to 12 pm cst (monday-friday) o  12-4 pm cst (monday-friday) o  4-8 pm cst (monday-friday) o  8 pm to 12 am cst (monday-friday) o  weekends during the day o  live virtual services not needed   how would you spend money to improve the library?  please select the two most important items. o  longer hours o  more comfortable furniture o  more computers o  more online resources (ebooks, ejournals, databases, etc) o  more electrical outlets o  more individual desks o  more print books o  more quiet/silent study spaces o  more whiteboards o  other (please specify):_______________     thinking about your overall experience with the library, what is one thing that you would like us to know to improve your experience?               appendix b 2018 student experience survey   i have read the “agreement to participate” document and agree to participate in this research.   o  yes o  no   last semester, how often did you visit the university library?   daily multiple days in a week once a week once a month never in person o    o    o    o    o    online o    o    o    o    o      how satisfied are you with the library spaces below at your library?   very satisfied satisfied dissatisfied very dissatisfied i don’t use this space in the library quiet study spaces o    o    o    o    o    collaborative study spaces o    o    o    o    o    group study rooms o    o    o    o    o    computer areas o    o    o    o    o      if you study in places other than the university library, what do you like about those spaces? [check all that apply]   o  more study space o  quieter study space o  food/drink availability o  software availability o  equipment (e.g., computer, printer, scanner, etc.) availability o  longer hours o  more comfortable furniture o  i can find a seat o  i prefer to study at home o  other (please specify) [                                 ]   last semester, how often did you use each of the library resources below?     daily multiple days in a week once a week once a month never journal articles o    o    o    o    o    subject specific databases o    o    o    o    o    print books from the stacks o    o    o    o    o    textbooks on reserve o    o    o    o    o    electronic books o    o    o    o    o    library subject & course guides o    o    o    o    o    special collections & university archives o    o    o    o    o    digital images o    o    o    o    o    streaming media o    o    o    o    o    dvds on reserve o    o    o    o    o    patient care tools o    o    o    o    o      please indicate the relative importance of each of the library resources/services for your research or coursework.     very important important somewhat important not at all important i don’t use this tool/service journal articles o    o    o    o    o    subject specific databases o    o    o    o    o    print books from the stacks o    o    o    o    o    textbooks on reserve o    o    o    o    o    electronic books o    o    o    o    o    library subject & course guides o    o    o    o    o    special collections & university archives o    o    o    o    o    digital images o    o    o    o    o    streaming media o    o    o    o    o    dvds on reserve o    o    o    o    o    patient care tools o    o    o    o    o    library instruction arranged by your professor o    o    o    o    o    library workshops that you self-selected to attend o    o    o    o    o    other (please specify) o    o    o    o    o      how easy is it to use the university library website for the services below?     very easy easy difficult very difficult i don’t use this service finding journal articles using the search box on the library home page o    o    o    o    o    finding an e-book using the search box on the library home page o    o    o    o    o    finding a print book using the search box on the library home page o    o    o    o    o    accessing a database to search for articles and other scholarly materials o    o    o    o    o    requesting a print book from another library o    o    o    o    o    requesting an article from another library o    o    o    o    o    logging into my library account to renew a book o    o    o    o    o    asking for help from a librarian by im/chat o    o    o    o    o    using library subject & course guides to access materials by subject o    o    o    o    o    finding media (e.g., films, videos, online images, etc.) o    o    o    o    o    booking a group study room online o    o    o    o    o    other (please specify) o    o    o    o    o      how likely are you to recommend the following library services to another student?     very likely likely unlikely very unlikely i don’t use this service one on one research consultation with a librarian o    o    o    o    o    library workshops about library research (e.g., finding resources, requesting materials, etc.) o    o    o    o    o    im/online chat research help o    o    o    o    o    e-mail research help o    o    o    o    o      suppose you had funding to improve the university library. please select up to three of your most important priorities from the list below. o  access to more online journals o  access to more books (e-books, print books, textbooks) o  more computers o  more quiet study space o  more group study space o  more electrical outlets o  more whiteboards o  more drink/food options o  additional comfortable furniture o  other (please list) [                              ]   think about your overall library experience at the institution, please tell us about your experiences with the library that positively impacted your coursework or research.                       evidence based library and information practice evidence based library and information practice 2010, 5.1  evidence based library and information practice       commentary    virtual peer mentoring (vpm) may facilitate the entire eblip process    jonathan d. eldredge  associate professor and interim coordinator  learning design center, health sciences library and informatics center  university of new mexico  albuquerque, new mexico, united states of america  email: jeldredge@salud.unm.edu    received: 16 jan. 2010          accepted: 18 jan. 2010       2010 eldredge. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution,  and reproduction in any medium, provided the original work is properly cited.     introduction    rapid changes in information management  bring increased urgency for all to engage in  continuous lifelong learning. common  methods for keeping our individual skills and  knowledge bases current include regularly  scanning and reading targeted core journals;  participation in conferences and continuing  education workshops; coursework in subjects  such as research methods; and, piloting new  approaches in the workplace.    this commentary features the involvement of  mentors as guides to professional growth and  development. it devotes particular attention to  an emerging information technology‐based  variation of mentoring known as virtual peer  mentoring (vpm). the author then employs  the five‐step eblip process to illustrate how  vpm might contribute to one’s continuous  professional vitality.    traditional mentoring    mentoring occurs in many variations across   different contexts (gabriel & kaufield, 2008;  johnson, 2007; mullen, 2005). traditional  mentoring consists of a colleague with far  more experience and knowledge guiding a  less experienced and knowledgeable colleague  in gaining full membership into a profession.  studwell (2002) describes mentors as wise and  trusted counselors or teachers. a traditional  mentor “guides, teaches, and develops a  novice…[who] can help with day‐to‐day tasks  …or they can help guide more substantial  decisions such as career planning” (zerzan,  hess, schur, phillips, & rigotti, 2009, p. 140).    traditional mentoring normally yields great  benefits for the protégé (or “mentee”)  receiving this guidance. de janasz, ensher, and  heun (2008) note that traditional mentoring  frequently results in protégé career success. in  fact, traditional mentoring has been linked to  successes for alfred nobel and his prize  winners (kantha, 1999; bentivoglio, 2006). in  on becoming a mentor, johnson (2007) provides  extensive inventories of benefits that can  accrue to protégés due to appropriate  mentoring, including psychosocial support,  7 http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2010, 5.1  commitment to the profession, professional  growth, self‐confidence, encouragement,  support, and exciting challenges. those who  have had multiple mentors (and multiple  protégés later in their careers) might add that  mentoring also offers valued professional  friendships.     traditional mentoring does have weaknesses.  the imbalanced power differential between  mentor and protégé may cause this otherwise  revered institution to develop pathologies.  barnett (2008) documents two such  vulnerabilities: boundary issues and multiple  relationships. boundaries are the basic ground  rules that structure these professional  friendships. when these boundaries are  breached in ethical ways, no such pathologies  are likely to occur. barnett relies upon the  american psychological association’s (apa)  code of ethics (2002) principles of beneficence,  nonmaleficence, fidelity, autonomy, and  justice to distinguish between ethical and  unethical breaches of boundaries in mentor‐ protégé relationships. neither the american  library association (1997) nor the medical  library association’s (1994) codes of ethics  elaborate to the same extent on such  dimensions to ethical behavior so these apa  ethics guidelines might serve our profession  well for safeguarding these mentor‐protégé  relationships.      barnett (2008) examines situations in which a  mentor might have multiple relationships with  a protégé. these might involve the mentor’s  serving as the supervisor or in another  superior position within the organization.  other potentially complicating relationships  might include a professional association  appointment that could influence the protégé’s  career or an editorial position in a leading  journal. as these roles, or relationships,  multiply so does the potential exploitation of  the protégé by compounding the inherent  power differential in a traditional mentoring  relationship. barnett emphasizes that the  existence of boundary crossings or multiple  relationships does not automatically condemn  such mentoring relationships to pathological  outcomes, provided that the participants   uphold ethical principles.  traditional mentoring may be vulnerable to an  entirely different weakness, related to supply  and demand for mentors. lublin (2003)  observes that even the highest level managers  need qualified mentors to guide their careers.  yet, most hierarchal organizations have few  high‐level professionals with time to mentor  more junior professionals (brice et al., 2002).  within five years of graduating from a library  or information science degree program, most  professionals reach a level of specialization  within their institutions that they cannot rely  on colleagues in the same institution to  provide guidance appropriate to their  specialties.     virtual peer mentoring (vpm)    over many years traditional mentoring has  harnessed successively more sophisticated  information and communications technology.  twenty years ago, when still early in his  career, the author had several traditional style  mentors who mentored him mainly via  telephone. email eventually supplemented the  telephone for these mentoring relationships as  well as his growing role as a traditional  mentor to others. in recent years free web‐ based conferencing through services such as  skype and dimdim have further enhanced  mentoring at a distance. e‐mentoring research  suggests that electronic mediation can  beneficially focus mentor and protégé  attention on shared goals and values rather  than geographic, institutional or demographic  differences (de janasz et al., 2008; single &  single, 2005). e‐mentoring may also help build  professional communities that can span  continents (gunawardena et al., 2008).    johnson (2007) and mullen (2005) observe the  emergence of  alternatives to traditional  mentoring relationships. peer mentoring,  sometimes known as “reciprocal mentoring”  or “co‐mentoring”, represents a form of  mutual mentoring between colleagues who  share approximately the same level of  professional development or status.  kram and  isabella’s (1985) pioneering research  discovered many career‐enhancing and  psychosocial benefits of peer mentoring.   other researchers identify many advantages  8 evidence based library and information practice 2010, 5.1  9 over traditional mentoring for experienced  professionals (dahl, 2005; jipson & paley,  2000; mullen, 2000; kochan & trimble, 2000;  gabriel & kaufield, 2008). bryant and  terborg’s (2008) field‐based experimental  study suggests that peer mentoring might be a  superior method for information‐intensive  professionals to contribute more productively  to knowledge creation and sharing within  their organizations.    virtual peer mentoring (vpm), an emerging  hybrid of e‐mentoring and peer mentoring,  offers some solutions for eblip practitioners  concerned about the potential pathologies in  traditional mentoring or the shortage of  appropriate mentors in a traditional  mentorship arrangements. the author has  been engaged in vpm with various colleagues  for the past two years. table 1 offers guidance  on how to maintain a vpm relationship. this  guidance is based on the nascent literature and  personal experiences and has not been  subjected to rigorous tests. nevertheless it is  supported by research evidence on distance  learning and collaborations involving similar  situations.     two themes from the research on distance  learning emerge to inform newcomers to  vpm. first, participants in vpm should make  reasonable efforts to establish “social  presence” in their interactions (short,  williams, & christie, 1976). social presence  involves several avenues for presenting vpm  participants as lifelike, trustworthy, and  authentic as possible at a distance in order to  approximate face‐to‐face communication.  establishing social presence can be a strong  predictor of participant satisfaction  (gunawardena & zittle, 1997) in virtual  interactions.  while not essential, it appears to  help vpm participants to maintain social  presence where they meet in person prior to  engaging in distance co‐mentoring (morris,  nadler, kurtzberg, & thompson, 2002).  thompson and nadler (2002) have identified  possible miscommunications that might occur  in vpm types of situations. a second theme  involves sensitivity to cultural differences  between participants. hofstede (1991) and  more recently others (knight, gunawardena,  & aydin, 2009) note the dimensions of cultural  differences. power differentials, the degree of  communitarianism, and extent of femininity or  masculinity expressed within an individual’s  society can affect her or his virtual  interactions. the author recommends  gunawardena and lapointe’s (2008) concise  book chapter for readers interested in social  presence and cultural sensitivity.     the eblip process    the use of the eblip process for decision  making has been described in detail  previously (eldredge, 2006 & 2008) so it will  be summarized with an emphasis upon  aspects related to vpm. figure 1 offers a visual  depiction of the five‐step eblip process for  readers to reference during the following text.     step one: formulate an important and  answerable question    the ability to plan and therefore enact  decision making processes appears to be a  uniquely human trait only partially shared  with certain primates (osvath, 2009). within  the human species, professionals in particular  are required to make important decisions  (bennett & gibson, 2006). decision making  inevitably leads to raising a central question.  the eblip process consequentially begins  with an important and answerable question  related to our professional practice. because  we aid users in answering their questions on a  regular basis, we already possess skills  particularly attuned and adapted to this first  question formulation step (booth, 2006).  formulating an important question that will  appropriately lead to an informed decision  typically presents a challenge even for  experienced practitioners, however. it is easy  to become distracted or diverted from  articulating the central question. a mentor  will actively listen to a protégé’s description of  the context of both decision and central  question and will assist her or him in  articulating and refining the central question.  evidence based library and information practice 2010, 5.1  table 1  tips for virtual peer mentoring (vpm)  when in the virtual peer mentor role:  • actively listen to your virtual protégé without interrupting.  • provide any needed information.  • establish personal boundaries on what you are willing or not willing to share or disclose.  • refer your protégé quickly to a colleague who possesses knowledge or a skill needed by the  protégé in order to succeed. be sure to follow‐up on the referral.  • only offer advice if your protégé asks for guidance.  • search for new ways to provide useful guidance or encouragement for your protégé.  when in the virtual peer protégé role:  • initiate contact when confronted with a decision or in guidance with applying the eblip  process to one’s practice.  • respect your virtual mentor’s personal boundaries.  • always credit your virtual mentor’s contributions either through co‐authorship or  acknowledgements.  for both virtual peer protégés and mentors:  • do not assume that either one of you will always be able to answer all of the other’s  questions.  • arrange the best dates/times to meet virtually. meet as often as desired.  • seek clarification whenever one of you might not understand the other (gunawardena &  lapointe, 2008).  • if your virtual peer mentoring relationship becomes far more of a personal rather than a  professional relationship then you might want to cease the peer mentoring aspect.  • recognize that misunderstandings will occur due to cultural differences. embrace the  challenge of overcoming these cultural differences and appreciate learning about your peer’s  background.  step two: searching for the best available  evidence    members of our profession additionally  possess the skills needed for locating the best  available evidence for answering the central  question. paradoxically, our documented  knowledge base presents complex challenges  for even the determined searcher. our  profession has a peculiar blend of incentives to  emphasize professional communication within  the gray literature (eldredge, 2008) while  contending with negative incentives that  frequently work against communicating in the  more accessible and searchable peer reviewed  literature. denise koufogiannakis and ellen  crumley (2002; 2004) have developed six  subject domains for librarianship research that  inform this second step of searching the  published literature: reference, education,   10 evidence based library and information practice 2010, 5.1  fig. 1. evidence based library and information practice (eblip) process chart.    collections, management, information access  and retrieval, and marketing/promotion.  three of these subject domains (reference,  collections, information access) are well  suited for searching the library and  information sciences literature databases. the  other three subject domains (education,  management, marketing/promotion) are  better searched via specialty databases for  each field’s literatures. beverley (2004) and  winning (2004) have diagnosed challenges to  searching, and provide various solutions for  eblip searchers. a mentor may advise their  protégé on how to overcome such challenges.    step three: critically appraising the best  available evidence    searching for the best available evidence  typically produces an impressive stack of  articles from peer reviewed journals as well as  printouts from other authoritative sources.  the eblip practitioner must then review and  determine which articles offer the best  available evidence. as it turns out, the kind of  question asked in the first step in the eblip  process provides important insights on  selecting the best evidence for making a  decision. most eblip questions can be readily  categorized into three major types: prediction,  intervention, and exploration. by categorizing  our questions by one of these three types, we  can evaluate which evidence has the greatest  weight compared to other evidence related to  our central question in the decision making  process. systematic reviews reside at the  highest level of evidence for all three types of  questions. systematic reviews have addressed  central questions such as those related to  clinical medical librarian services (winning &  beverly, 2003; wagner & byrd, 2004) and  information skills training (brettle, 2003). at  this time, with few systematic reviews within  our field, we must identify alternative  evidence for each question category.  intervention questions normally are answered  best by an experimental design known as a  randomized controlled trial, prediction  questions by cohort studies and exploration  questions by qualitative studies, provided that  no relevant systematic reviews exist. the  eblip levels of evidence are explained in  more detail elsewhere (eldredge, 2002). booth  and brice (2004) have developed a  tremendously helpful (and freely accessible)  checklist of questions to consider while  critically appraising evidence for answering  eblip questions. glynn (2006) also has created  a checklist drawn from multiple sources. a  mentor can help the protégé to navigate the  different types of evidence, and with selecting  appraisal tools appropriately.    sometimes the eblip practitioner cannot find  authoritative evidence to make an informed  decision. figure one reflects this possible  scenario between steps three and four. in this  instance, the vpm mentor may encourage the  practitioner to articulate a call for further  research or to participate directly in  conducting the research needed to answer  appropriately the eblip question.    step four: make a decision    eblip assists practitioners in making  informed decisions. although many of us are  experienced decision makers, we can no doubt  describe decision making processes  undertaken by groups that have resulted in  unsatisfactory decisions. the behavioral and  social sciences have researched “cognitive  biases” as obstacles to making sound decisions  even where all of the best evidence is available  to decision makers (eldredge, 2008). the  eblip practitioner must therefore remain  eternally vigilant to biases contaminating the  eblip process. a good mentor will make  11 evidence based library and information practice 2010, 5.1  themselves available as a sounding board to  discuss and review the rationale(s) for the final  decision.    step five: evaluate performance    booth (2004) conceptualizes this final step at  individual practitioner, institutional, and  professional levels. we must always appraise  our individual performances as reflective  professionals. peer review processes, whether  job performance reviews within our  institutions or review processes at journals or  by professional associations, assist us in  assessing our strengths and weaknesses as  individual practitioners. many institutions  have become increasingly aware of, and  aligned with, evidence‐based practice in recent  years, making it relatively easy to ask   collectively if the library practices eblip in  making major decisions. once identified,  deficiencies can be addressed within the  institution. the mentor’s role may include  identifying other colleagues who may help a  protégé to address such deficiencies. at the  professional level, colleagues need to evaluate  whether their professional association  identifies the most important and answerable  questions facing our profession. recently the  medical library association in the u.s.  (eldredge, harris, & ascher, 2009) and the  swedish library association (maceviciute,  wilson, lalloo, & lindh, 2009) have  commissioned such studies using the delphi  method to identify key questions facing the  members of these respective associations. a  mentor may encourage a protégé and others to  contribute to such efforts.      table 2  vpm throughout the eblip process  eblip process  step  protégé role   mentor role  formulate  initiates contact with colleague for  mentoring assistance when  confronted with a decision in one’s  practice.  actively listens to protégé’s description  of context of decision and central  question.  assists in articulating and refining  question.  search  offers feedback on successes or  deficiencies of different search  approaches.  suggests databases and other sources of  authoritative evidence.  proposes possible controlled vocabulary  or keyword approaches.  appraise  applies critical appraisal skills to  different evidence produced from  search.  listens and suggests inclusion and  exclusion criteria.  decide  final responsibility for making the  decision.  available to discuss and review  rationale(s) for final decision.  evaluate  self‐assesses own performance in the  process.  contributes to institutional and  profession‐wide efforts to evaluate  the relevance of central questions  raised and how eblip might be  practiced by others.  refers protégé to other colleague(s) who  might possess expertise needed by  protégé.  contributes to profession‐wide efforts to  evaluate the relevance of central  questions raised and how eblip might  be practiced by others.  12 evidence based library and information practice 2010, 5.1  applying vpm to the eblip process    table 2 summarizes how colleagues might use  vpm throughout the eblip process by  assuming the roles indicated.  interchangeability of roles should be borne in  mind since, at any specific moment, either  member of a vpm dyad might be in either  mentor or protégé roles. table 2 further shows  how vpm might be interpreted and adapted  to guide each step of the eblip process. by  employing vpm in the eblip process, we may  even build an international eblip community  of practice (wenger, 1998) when used in  conjunction with maintaining a peer reviewed  journal and offering international workshops  and conferences.  implications for practice  • virtual peer mentoring (vpm) is one  possible means for maintaining one’s  continuous professional vitality.  • mentoring relationships can thrive  despite boundary crossings or  multiple relationships provided both  participants uphold ethical principles.  • virtual peer mentoring is a possible  response to the short supply of high  level professionals with time to  mentor junior staff.  implications for research  • professional associations must take  the lead in identifying the most  important and answerable questions  facing our profession.  • where authoritative evidence does not  exist for an informed decision, a  mentor may advise a protégé on the  potential for further research  • despite scarce research on virtual  peer mentoring, we can infer guidance   from similar research on distance  learning and collaborations.  references  american library association. (1997). code of  ethics. chicago: american library  association. retrieved 20 feb. 2010  from  http://www.ala.org/ala/aboutala/office s/oif/statementspols/codeofethics/code ethics.cfm  american psychological association. (2002).  ethical principles of psychologists and  code of conduct. washington, dc:  american psychological association.  retrieved 20 feb. 2010 from  http://www.apa.org/ethics/code2002.h tml   barnett, j. e. (2008). mentoring, boundaries,  and multiple relationships:  opportunities and challenges.  mentoring & tutoring: partnership in  learning 16(1), 3‐16.  bennett, m. d., & gibson, j. m. (2006). a field  guide to good decisions. westport, ct:  praeger.  bentivoglio, m. (2006). giuseppe levi: mentor  of three nobel laureates. journal of the  history of the neurosciences 15(4), 358‐ 368.  beverley, c. (2004). searching the library and  information science literature. in a.  booth & a. brice (eds.), evidence‐based  practice for information professionals: a  handbook (pp. 98‐103). london: facet  publishing. retrieved 20 feb. 2010  from  http://ebliptext.pbworks.com/f/booth+ %26+brice+2004+ebp+for+info+profes sionals+‐+a+handbook.pdf  booth, a. (2004). evaluating your  performance. in a. booth & a. brice  (eds.), evidence‐based practice for  information professionals: a handbook  (pp. 127‐137). london: facet  publishing. retrieved 20 feb. 2010  from  13 http://www.ala.org/ala/aboutala/offices/oif/statementspols/codeofethics/codeethics.cfm http://www.ala.org/ala/aboutala/offices/oif/statementspols/codeofethics/codeethics.cfm http://www.ala.org/ala/aboutala/offices/oif/statementspols/codeofethics/codeethics.cfm http://www.apa.org/ethics/code2002.html http://www.apa.org/ethics/code2002.html http://ebliptext.pbworks.com/f/booth+%26+brice+2004+ebp+for+info+professionals+-+a+handbook.pdf http://ebliptext.pbworks.com/f/booth+%26+brice+2004+ebp+for+info+professionals+-+a+handbook.pdf http://ebliptext.pbworks.com/f/booth+%26+brice+2004+ebp+for+info+professionals+-+a+handbook.pdf evidence based library and information practice 2010, 5.1  http://ebliptext.pbworks.com/f/booth+ %26+brice+2004+ebp+for+info+profes sionals+‐+a+handbook.pdf  booth, a. (2006). clear and present questions.  library hi tech 24(3), 355‐368.  booth, a., & brice, a. (2004). appraising the  evidence, in a. booth & a. brice  (eds.), evidence‐based practice for  information professionals: a handbook  (pp. 104‐118). london: facet  publishing. retrieved 20 feb. 2010  from   http://ebliptext.pbworks.com/f/booth+ %26+brice+2004+ebp+for+info+profes sionals+‐+a+handbook.pdf  brettle, a. (2003). information skills training: a  systematic review of the literature.  health information and libraries journal,  20(s1), 3‐9.  brice, a., brown,  c., hickman, m., &  thorburn, l. (2002). helin pilot  mentoring scheme. library trends,  50(4), 651‐664.  bryant, s. e., & terborg, j. r. (2008). impact of  peer mentor training on creating and  sharing organizational knowledge.  journal of managerial issues, 20(1), 11‐ 29.  crumley, e., & koufogiannakis, d. (2002).  developing evidence‐based  librarianship: practical steps for  implementation. health information and  libraries journal, 19(2), 61‐70.  dahl, j. (2005). peer mentoring builds faculty  satisfaction. distance education report,  9(23), 1‐2.  de janasz, s. c., ensher, e. a., heun, c. (2008).  virtual relationships and real benefits:  using e‐mentoring to connect business  students with practicing managers.  mentoring and tutoring: partnership in  learning, 16(4), 394‐411.  eldredge, j. (2002). evidence based  librarianship: levels of evidence.   hypothesis, 16(1) 10‐13.  retrieved 20  feb. 2010 from  http://research.mlanet.org/hypothesis/  eldredge, j. (2006). evidence‐based  librarianship: the ebl process. library  hi tech 24(3), 341‐354.  eldredge, j. d. (2008). evidence‐based practice.  in m. s. wood (ed.), introduction to  health sciences librarianship (pp. 241‐ 269). new york: haworth press.   eldredge, j., harris, m., & ascher, m. (2009).  defining the medical library  association research agenda:  methodology and final results from a  consensus process. journal of the  medical library association, 97(3), 178‐ 85. retrieved 20 feb. 2010 from  http://www.ncbi.nlm.nih.gov/pmc/arti cles/pmc2706444/?tool=pubmed  gabriel, m. a., & kaufield, k. j. (2008).  reciprocal mentorship: an effective  support for online instructors.  mentoring & tutoring: partnership in  learning, 16(3), 311‐327.  glynn, l. (2006). a critical appraisal tool for  library and information research.  library hi tech, 24(3), 387‐399.  gunawardena, c. n., & lapointe, d. (2008).  social and cultural diversity in  distance education. in t. evans, m.  haughey, & d. murphy (eds.),  international handbook of distance  education (pp. 51‐69). bingley, uk:  emerald.  gunawardena, c. n., lapointe, d., linder‐ vanberschot, j., skinner, j.k.,  richmond, k.b., barrett, k., & cardiff,  m.s. (2008). e‐mentoring to guide  inquiry‐based online learning across  cultures. 24th annual conference on  distance teaching and learning.   retrieved 23 feb. 2010 from          14 http://ebliptext.pbworks.com/f/booth+%26+brice+2004+ebp+for+info+professionals+-+a+handbook.pdf http://ebliptext.pbworks.com/f/booth+%26+brice+2004+ebp+for+info+professionals+-+a+handbook.pdf http://ebliptext.pbworks.com/f/booth+%26+brice+2004+ebp+for+info+professionals+-+a+handbook.pdf http://ebliptext.pbworks.com/f/booth+%26+brice+2004+ebp+for+info+professionals+-+a+handbook.pdf http://ebliptext.pbworks.com/f/booth+%26+brice+2004+ebp+for+info+professionals+-+a+handbook.pdf http://ebliptext.pbworks.com/f/booth+%26+brice+2004+ebp+for+info+professionals+-+a+handbook.pdf http://research.mlanet.org/hypothesis/ http://www.ncbi.nlm.nih.gov/pmc/articles/pmc2706444/?tool=pubmed http://www.ncbi.nlm.nih.gov/pmc/articles/pmc2706444/?tool=pubmed evidence based library and information practice 2010, 5.1  http://www.uwex.edu/disted/conferen ce/resource_library/handouts/08_1367 6_p.pdf  gunawardena, c. n., & zittle, f. j. (1997).  social presence as a predictor of  satisfaction within a computer‐ mediated conferencing environment.  american journal of distance education,  11(3), 8‐26.  hofstede, g. h. (1991). cultures and  organizations: software of the mind. new  york: mcgraw‐hill.  jipson, j., & paley, n. (2000). because no one  gets there alone: collaboration as co‐ mentoring. theory into practice, 39(1),  36‐42.  johnson, w. b. (2007). on being a mentor. new  york, lawrence erlbaum associates.  kanta, s. s. (1999). alfred nobel’s unusual  creativity: an analysis. medical  hypotheses, 53(4), 338‐344.  knight, e. e., gunawardena, c. n., & aydin,  c. h. (2009). cultural interpretations  of the visual meaning of icons and  images used in north american web  design. educational media international,  46(1), 17‐35.   kochan, f. k., & trimble, s. b. (2000). from  mentoring to co‐mentoring:  establishing collaborative  relationships. theory into practice,  39(1), 20‐28.  koufogiannakis, d., & crumley, e. (2004). a  content analysis of librarianship  research. journal of information science,  30(3), 227‐239.  kram, k. e., & isabella, l. a. (1985). mentoring  alternatives: the role of peer  relationships in career development.  academy of management journal, 28(1),  110‐132.  lublin, j. s. (2003, july 1). even top executives  could use mentors to benefit careers.  wall street journal eastern edition, b1.  maceviciute, e., wilson, t., lallo, i., & lindh,  m. (2009). a delphi study of research  needs for swedish libraries. science for  the professions, 5, 3‐67.  medical library association. (1994). code of  ethics for health sciences librarianship.  chicago: medical library association.  retrieved 20 feb. 2010 from       http://www.mlanet.org/about/ethics.ht ml.  morris, m., nadler, j., kurtzberg, t., &  thompson, l. (2002). schmooze or  lose: social friction and lubrication in  e‐mail negotiations. group dynamics:  theory, research, and practice 6(1), 89‐ 100.  mullen, c. a. (2000). constructing co‐ mentoring partnerships: walkways  we must travel. theory into practice,  39(1), 4‐11.  mullen, c. a. (2005). mentorship primer. new  york: peter lang.  osvath, m. (2009). spontaneous planning for  future stone throwing by a male  chimpanzee. current biology, 19(5),  r190‐191.  short, j., williams, e., & christie, b. (1976). the  social psychology of telecommunications.  new york: john wiley & sons.   single, p. b., & single, r. m. (2005). e‐ mentoring for social equity: review of  research to inform program  development. mentoring and tutoring,  13(2), 301‐320.  studwell, w. e. (2002). mentorship and the  future of library and information  science. technicalities, 22(4), 3‐6.  15 http://www.uwex.edu/disted/conference/resource_library/handouts/08_13676_p.pdf http://www.uwex.edu/disted/conference/resource_library/handouts/08_13676_p.pdf http://www.uwex.edu/disted/conference/resource_library/handouts/08_13676_p.pdf http://www.mlanet.org/about/ethics.html http://www.mlanet.org/about/ethics.html evidence based library and information practice 2010, 5.1  16 thompson, l., & nadler, j. (2002). negotiating  via information technology: theory  and application. journal of social issues,  58(1), 109‐124.  wagner, k.c., & byrd, g. d. (2004). evaluating  the effectiveness of clinical medical  librarian programs: a systematic  review of the literature. journal of the  medical library association, 92(1), 14‐ 33.  wenger, e. (1998). communities of practice:  learning, meaning, and identity.  cambridge, uk: cambridge  university press.  winning, a. (2004). identifying sources of  evidence. in a. booth, & a. brice,  evidence‐based practice for information  professionals: a handbook (pp. 77‐88).  london: facet publishing. retrieved  20 feb. 2010 from  http://ebliptext.pbworks.com/f/booth+ %26+brice+2004+ebp+for+info+profes sionals+‐+a+handbook.pdf  winning, m. a., & beverley, c. a. (2003).  clinical librarianship: a systematic  review of the literature. health  information and libraries journal, 20(s1),  10‐21.  zerzan, j. t., hess, r., schur, e., phillips, r. s.,  & rigotti, n. (2009). making the most  of mentors: a guide for mentees.  academic medicine, 84(1), 140‐144.  http://ebliptext.pbworks.com/f/booth+%26+brice+2004+ebp+for+info+professionals+-+a+handbook.pdf http://ebliptext.pbworks.com/f/booth+%26+brice+2004+ebp+for+info+professionals+-+a+handbook.pdf http://ebliptext.pbworks.com/f/booth+%26+brice+2004+ebp+for+info+professionals+-+a+handbook.pdf table 1tips for virtual peer mentoring (vpm) implications for practice implications for research references evidence summary   graduate students may need information literacy instruction as much as undergraduates   a review of: conway, kate. (2011). how prepared are students for postgraduate study? a comparison of the information literacy skills of commencing undergraduate and postgraduate studies students at curtin university. australian academic & research libraries, 42(2), 121-135.   reviewed by: robin e. miller assistant professor and research & instruction librarian mcintyre library university of wisconsin-eau claire eau claire, wisconsin, united states of america email: millerob@uwec.edu   received: 8 apr. 2014    accepted: 12 aug. 2014      2014 miller. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine whether there is a difference in the information literacy skills of postgraduate and undergraduate students beginning an information studies program, and to examine the influence of demographic characteristics on information literacy skills.   design – online, multiple choice questionnaire to test basic information literacy skills.   setting – information studies program at a large university in western australia.   subjects – 64 information studies students who responded to an email invitation to participate in an online questionnaire, a 44% response rate. of those responding, 23 were undergraduates and 41 were postgraduates.   methods – over the course of two semesters, an online survey was administered. in order to measure student performance against established standards, 25 test questions were aligned with the australian and new zealand information literacy framework (anziil) (bundy, 2004), an adapted version of the acrl information literacy standards for higher education (association of college & research libraries, 2000). in the first semester that the survey was administered, 9 demographic questions were asked and 11 in the second semester. participants were invited to respond voluntarily to the questionnaire via email. results were presented as descriptive statistics, comparing undergraduate and postgraduate student performance. the results were not tested for statistical significance and the author did not control for confounding variables.   main results – postgraduate respondents scored an average of 77% on the test questionnaire, while undergraduates scored an average of 69%. the 25% of respondents who had previous work experience in a library achieved average scores of 79%, in contrast to 69% among those who had not worked in a library. average scores for undergraduates in the 20-30 age group were 81%, while those in the 30-40 age group averaged 65%. among both undergraduate and postgraduate students, scores may indicate deficiencies in information literacy skills in several areas, including parsing citations, strategies for locating specific content, and defining an information need.   conclusion – the study concludes that postgraduate students’ information literacy skills may be marginally better than the skills of undergraduates. age was found to be associated with higher performance among undergraduate students, and a variety of “basic” information literacy skills may elude many respondents. these findings might prompt librarians and instructors to look closely at gaps in information literacy knowledge among students at both the undergraduate and postgraduate level.     commentary   librarians and instructional faculty may assume that new postgraduate students possess a high level of information literacy, instilled during their undergraduate careers. the author attempts to test this assumption.   rather than testing the information literacy skills of only postgraduate students, conway chose to take a comparative approach by testing two populations, postgraduate and undergraduate students of information studies at curtin university. the literature review notes that few studies have compared basic information literacy competencies of postgraduate and undergraduate students. the author suggests that undergraduates are the focal point of literature about information literacy skill level; however, the literature review omits recent publications focused on graduate students (siegel, 2009; catalano, 2010; mcmillen, garcia, & bolin, 2010).   conway acknowledges the limitations of multiple choice tests for the higher order thinking associated with information literacy. presumably due to cost and other practical considerations, the author did not use a standardized information literacy skills test like sails, which is proprietary and not open access. the survey instrument was developed using questions from a curtin university library online information literacy tutorial that is no longer available (curtin university library, 2010), and questions that have appeared in previously administered tests of information literacy skills (mittermeyer & quirion, 2003), though some tests cited by the author are not readily available for consultation (stokes, 2005; van zijl, bennett, darling, shields & bennett, 2006). the choice to use previously administered questions may lend validity to some of the questions and findings. the author reports that tests using similar questions yielded lower average scores than the curtin university test, speculating this finding is due to inclusion of postgraduates in the subject pool and the significant population of subjects who had previous library work experience. unfortunately, the survey instrument was not appended.   a serious limitation of this study is that the author did not test the results for statistical significance. as conway notes, this research cannot be generalized because of the small respondent pool; however, the test was also not administered to a random sample of students. choosing to test information studies students means that these findings have no point of comparison to postgraduates and undergraduates in other disciplines.   while postgraduates generally performed better than undergraduates, the author notes that the overall performance of postgraduates was only 8% higher than undergraduates. the author particularly notes a correlation between age and skills performance, with respondents in their twenties scoring higher on some questions than respondents in their thirties. of interest to librarians, respondents had higher scores if they had previous library work experience, but scores were not improved by previous information literacy instruction. however, as reported, these results do not control for confounding variables.   for practitioners, the most essential conclusion of this research is that both postgraduates and undergraduates who took the skills test appeared to struggle with concepts that librarians and classroom instructors may view as basic. serious limitations of research design notwithstanding, this research may remind librarians of the need to integrate information literacy instruction throughout undergraduate and postgraduate experiences.     references   association of college & research libraries. (2000). information literacy competency standards for higher education. chicago, il: american library association. retrieved from http://www.ala.org/acrl/sites/ala.org.acrl/files/content/standards/standards.pdf   bundy, a. (ed.). (2004). australian and new zealand information literacy framework: principles, standards and practice, 2nd ed. adelaide: australian and new zealand institute for information literacy. retrieved from http://archive.caul.edu.au/info-literacy/infoliteracyframework.pdf   catalano, a. j. (2010). using acrl standards to assess the information literacy of graduate students in an education program. evidence based library and information practice, 5(4), 21-38. retrieved from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/8878   curtin university library. (2010). infotrekk. retrieved from http://library.curtin.edu.au/studyand-research-tools/online-tutorials/infotrekk/     mcmillen, p. s., garcia, j., & bolin, d. a. (2010). promoting professionalism in master’s level teachers through research based writing. journal of academic librarianship, 36(5), 427-439. doi:10.1016/j.acalib.2010.06.007   mittermeyer, d. & quirion, d. (2003). information literacy: study of incoming first-year undergraduates in quebec. quebec: conference des recteurs et des principaux des universites du quebec.   siegel, g. (ed.). (2009). libraries and graduate students: building connections. london: routledge.   stokes, n. (2005). report on the list (library & information skills training) research project. perth: central tafe.   van zijl, p., d. bennett, v. darling, g. shields & e. bennett. (2006). information literacy survey report. dunedin: otago polytechnic.   ebl 101   research methods: mixed methods research   virginia wilson librarian university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   received: 13 may 2013  accepted: 6 june 2013      2013 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   research paradigms. qualitative research. quantitative research. the firm adherence to one or the other can cause a lot of upheaval, as can be seen in the long-standing paradigm wars between the two. researchers from both sides of this ontological divide have been convinced of the “rightness” of their methodologies and beliefs around their specific approach to research. the side taken often depends upon from which disciplines the researchers are coming, as some disciplines work strictly quantitatively while some work from the qualitative perspective. increasingly, however, we are seeing mixed methods used in research studies. in a 2004 article, johnson and onwuegbuzi argued from the position that “there is now a trilogy of major research paradigms: qualitative research, quantitative research, and mixed methods research” (p. 24).   as always, i can only scratch the surface of any topic in this column, and mixed methods research (also referred to as mixed research) is no exception, especially as its history is steeped in philosophical understandings of research and knowledge acquisition in general. the mixed methods paradigm attempts to get in the middle of the two other approaches (quantitative and qualitative), seeking to respect both by using both in a research study. according to johnson, onwuegbuzie, and turner (2007), “mixed methods research is, generally speaking, an approach to knowledge (theory and practice) that attempts to consider multiple viewpoints, perspectives, positions, and standpoints (always including the standpoints of qualitative and quantitative research)” (p. 113). these researchers encourage a broad interpretation of “methods” in mixed methods research to allow for “inclusion of issues and strategies surrounding methods of data collection (e.g., questionnaires, interviews, observations), methods of research (e.g., experiments, ethnography), and related philosophical issues (e.g., ontology, epistemology, axiology)” (johnson, onwuegbuzie, and turner, 2007, p. 118).   looking at literature dealing with mixed methods research is a bit like navigating a maze: there’s a lot of it and it all looks much the same at the start. the johnson, onwuegbuzie, and turner (2007) article is particularly helpful, as it not only gives a history of this methodology, but also lists key players in the mixed methods mix, and looks specifically at the definitions that have emerged in the past 2 decades. the definitions are especially useful for a column such as this. the following are from the johnson, onwuegbuzie, and turner (2007) article:   mixed methods research is a research design (or methodology) in which the researcher collects, analyzes, and mixes (integrates or connects) both quantitative and qualitative data in a single study or a multiphase program of inquiry. (creswell, p. 119)   mixed methods research is a systematic integration of quantitative and qualitative methods in a single study for purposes of obtaining a fuller picture and deeper understanding of a phenomenon. mixed methods can be integrated in such a way that qualitative and quantitative methods retain their original structures and procedures (pure form mixed methods). alternatively, these two methods can be adapted, altered, or synthesized to fit the research and cost situations of the study (modified form mixed methods). (chen, p. 119)   mixed methods means the combination of different qualitative and quantitative methods of data collection and data analysis in one empirical research project. this combination can serve for two different purposes: it can help to discover and to handle threats for validity arising from the use of qualitative or quantitative research by applying methods from the alternative methodological tradition and can thus ensure good scientific practice by enhancing the validity of methods and research findings. or it can be used to gain a fuller picture and deeper understanding of the investigated phenomenon by relating complementary findings to each other which result from the use of methods from the different methodological traditions of qualitative and quantitative research. (kelle, p. 120)   mixed methods research refers to the use of data collection methods that collect both quantitative and qualitative data. mixed methods research acknowledges that all methods have inherent biases and weaknesses; that using a mixed method approach increases the likelihood that the sum of the data collected will be richer, more meaningful, and ultimately more useful in answering the research questions. (preskill, p. 121).   there are many more definitions of mixed methods research out there. while there are some variations in the content, most contain similar core ideas of using both qualitative and quantitative methods in order to more fully explore a research question. mixed methods research requires balance and an open mind to the benefits of various approaches to research.   in terms of mixed methods in library and information science (lis) research, raya fidel asks, “are we there yet?” fidel’s article is a must read, as she explores mixed methods research from an lis perspective, coming to the conclusion after an analysis of 465 articles published in four major lis research journals that only 5% of these articles used mixed methods, and that the methodology was not mentioned by name in the articles (p. 269). for her study, fidel uses the definition of mixed methods research written by tashakkori and creswell (2007) because of its inclusive nature: mmr is ‘research in which the investigator collects and analyzes data, integrates the findings, and draws inferences using both qualitative and quantitative approaches or methods in a single study or program of inquiry’ (p. 4)” (fidel, 2008, p. 269). however, given that the definitions of mixed methods research vary, that fidel found mixed methods research even though the terminology was not explicitly used in the research article, and that her research project examined only 4 lis research journals, it is not surprising to find that mixed methods research is being undertaken in lis. for example, see the following:   brettle, a., hulme, c., & ormandy, p. (2006) the costs and effectiveness of information-skills training and mediated searching: quantitative results from the empiric project. health information & libraries journal, 23(4), 239–247. doi: 10.1111/j.14711842.2006.00670.x   kwon, n. (2008). a mixed-methods investigation of the relationship between critical thinking and library anxiety among undergraduate students in their information search process. college & research libraries, 69(2), 117-131.   digging into mixed methods research would be beneficial to lis researchers. coming at a research question from multiple methodological perspectives in the same study will add a depth and breadth to the findings, and open up options for data collection and analysis.   resources:   on the web   the twitter hashtag #mixedmethods will yield tweets having to do with mixed methods research. developing mixed methods research with dr. john w. creswell, video, http://www.youtube.com/watch?v=psvsd9fax38 (an excellent encapsulation in about 6 minutes) what is mixed methods research? john creswell, video. http://www.youtube.com/watch?v=1oanitlpyx8 (15 min) creswell, j. (2012) introduction to mixed methods research, keynote address for the caqd conference, university of marburg, germany, march 8, 2012. http://prezi.com/qsksml6l-_vi/introduction-to-mixed-methods-research best practices for mixed methods research in the health sciences, national institutes of health, http://obssr.od.nih.gov/mixed_methods_research   journal of mixed methods research, sage publications, http://mmr.sagepub.com   books   creswell, j. w. & plano clark, v.l. (2011). designing and conducting mixed methods research. los angeles: sage.   hesse-biber. s.n. (2010). mixed methods research: merging theory with practice. new york: guilford press.   tashakkori, a., & teddlie, c. (eds.). (2003). handbook of mixed methods in social and behavioral research. thousand oaks, ca: sage.   teddlie, c. & tashakkori, a. (2009) foundations of mixed methods research: integrating quantitative and qualitative approaches in the social and behavioral sciences. los angeles: sage.     references   fidel, r. (2008). are we there yet?: mixed methods research in library and information science.  library and information science research, 30, 265-272.   johnson, r.b. & onwuegbuzie, a.j. (2004). mixed methods research: a research paradigm whose time has come. educational researcher, 33(7), 14-26. stable url: http://www.jstor.org/stable/3700093   johnson, r.b., onwuegbuzie, a.j. & turner, l.a. toward a definition of mixed methods research. journal of mixed methods research 1, 112-133.  doi:10.1177/1558689806298224 evidence based library and information practice evidence based library and information practice 2011, 6.2 5 evidence based library and information practice article science information literacy tutorials and pedagogy ping li assistant professor graduate school of library and information studies queens college, city university of new york flushing, new york, united states of america email: pli@qc.cuny.edu received: 01 dec. 2010 accepted: 21 may 2011 2011 li. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – this study examined information literacy tutorials in science. the goals of the research were to identify which of the information literacy standards for science, engineering and technology were addressed in the tutorials, and the extent that the tutorials incorporated good pedagogical elements. methods – the researcher chose for review 31 of the tutorials selected by members of the acrl science & technology section (sts) information literacy committee. she carefully analyzed the tutorials and developed a database with codes for the topic of each tutorial, the sts information literacy standard(s) addressed by the tutorial, and whether good pedagogical elements were incorporated. the entire analysis and coding procedure was repeated three times to ensure consistency. results – the tutorials analyzed in this study covered various subjects and addressed all the (sts) information literacy standards. the tutorials presented information clearly and allowed users to select their own learning paths. the incorporation of good pedagogical elements was limited, especially in relation to active learning elements. conclusions – web tutorials have been accepted as effective information literacy instruction tools and have been used to teach all elements of the sts information literacy standards. yet, ensuring they provide a real learning experience for students mailto:pli@qc.cuny.edu� evidence based library and information practice 2011, 6.2 6 remains a challenge. more serious thought needs to be given to integrating good pedagogy into these instructional tools in order to attain deep learning. introduction user instruction has been considered one of the essential functions of librarianship for many years. it has been described in a number of ways, including library orientation, library instruction, bibliographic instruction (bi), user education, and information literacy instruction. the concept of information literacy instruction has been discussed and promoted since the 1980s. information literacy instruction goes beyond the basic need for library orientation and introduction to include research tools provided within a search strategy framework (grassian & kaplowitz, 2009). the movement has developed in response to the increasing amount of information available to people and the growing complexity of information technology. information literacy refers to a set of abilities requiring individuals to “recognize when information is needed and have the ability to locate, evaluate, and use effectively the needed information” (american library association, 1989, para. 3). the framework for information literacy instruction in academic settings is described in the information literacy competency standards for higher education, which outlines five standards linked to twenty-two performance indicators (association of college and research libraries, 2000). these standards help address the unique disciplines of science, engineering, and technology, which pose unique challenges in identifying, evaluating, acquiring, and using information. to help guide information literacy instruction in these disciplines, information literacy standards for science and engineering / technology was developed by the ala/acrl/sts task force on information literacy for science and technology. these standards acknowledge the difficulty of obtaining peer reviewed literature and grey literature, the interdisciplinary nature of science, engineering and technology, and the variety of formats that require increasing expertise in information technology (slebodnik & zeidman-karpinski, 2008). the role of libraries and librarians in helping achieve information literacy standards has been extensively researched and discussed. it is a widely shared belief that libraries, particularly those funded by educational institutions, should be the lead agencies in articulating, promoting, and developing the community’s information literacy (stanger, 2009). while various types of information literacy instruction have been developed for library users, there has been an increasing demand for more rapid, anytime-anyplace sharing of information (bawden, devon, & sinclair, 2000). in response to this demand, many libraries are now offering information literacy instruction via the internet. online tutorials are one of the common instructional tools for this purpose (donaldson, 2000). compared to other instructional modes, online tutorials have the advantage of being available 24 hours a day and 7 days a week to anyone with an internet connection. online tutorials combined with face-to-face (f2f) sessions can save instructors’ time. “tutorial maintenance can take time but may take less time in the long run than teaching numerous f2f one-shot sessions with a limited librarian staff” (grassian, & kaplowitz, 2009, p.150). knowing how to achieve learning goals through good pedagogy is always a main consideration for educators. in face-to-face instruction, active learning can be seen as an important pedagogical technique. while instructing asynchronously, web tutorials seem to be at a disadvantage in incorporating good pedagogical elements. a common complaint about web tutorials is their lack of sufficient interactivity to allow active learning experiences (dewald, scholz-crane, booth, & levine, 2000). as early as 1999, dewald noted evidence based library and information practice 2011, 6.2 7 the importance of incorporating good pedagogy into such web-based library instruction as online tutorials. she argued that with online and asynchronous instructional mode, good pedagogy and active learning should be equally important in achieving instructional objectives (dewald, 1999b). it is believed that “real learning will only take place when students are asked to understand the material in their own terms and use it to accomplish a meaningful task” (dewald et al., 2000, p. 38). the inclusion of active learning in a tutorial clearly improves teaching effectiveness (hrycaj, 2005). in discussing a project to develop an online library skills tutorial at the university of akron, franks, hackley, straw, and direnzo (2000), also stressed the importance of active learning in their proposed tutorial. the feasibility of incorporating good pedagogy, such as active learning elements, into web-based library tutorials has been discussed in the literature. according to dewald, librarians have the opportunity to combine good pedagogy with the unique capabilities of the web, and new technologies should allow various methods of accomplishing active learning (dewald, 1999b; dewald et al., 2000). dewald proposed a set of pedagogical guidelines for web-based library instruction (including tutorials) that included active learning elements feasible in this (dewald, 1999b). these guidelines have been incorporated successfully in various tutorials. the design team for an online library skills tutorial at toronto’s seneca college of applied arts and technology reviewed a number of completed tutorials. they noted that many of the best tutorials consistently incorporated the use of active learning and “provided students with opportunities to interact and test new ideas as the lessons progressed” (donaldson, 2000, p. 241). one salient feature of the texas information literacy tutorial (tilt), winner of the 2000 acrl instruction section's “innovation in instruction award,” was its incorporation of various active learning elements, including quiz questions and exercises (hrycaj, 2005). in the areas of science, engineering and technology, the sts information literacy committee has been compiling a list of teaching ideas and techniques related to the performance indicators of the information literacy standards (http://wikis.ala.org/acrl/index. php?title=teaching_tips&oldid-42295; revision as of 7 apr. 2011, at 06:26). by examining which of the science, engineering, and technology information literacy standards can be addressed by web-based tutorials, this study sought to provide evidence that contributes to the further development of the sts information literacy committee’s list of teaching ideas and techniques. literature review analysis of web-based tutorials several studies have been reported in the literature analyzing web-based tutorials in an attempt to determine their characteristics, often based on a set of pre-established indicators. one of the earlier studies of this type was conducted by dewald (1999a), in which she evaluated a sample of nineteen tutorials of the library instruction round table against seven pre-established fundamental indicators of good library instruction: course-related and specifically assignment-related; active learning; collaborative learning; multiple media presentations; well-clarified objectives; concept teaching; and offering a librarian's follow-up help. the study concluded that tutorials could not replace face-to-face training, and that they should be used in connection with academic classes, rather than in isolation. tancheva (2003) conducted another study of this type. her study aimed at determining to what extent an online library instruction tutorial addressed the principles of accepted learning theory in distributed environments and the accepted principles of effective library instruction. tancheva analyzed over 40 tutorials and examined the experience of creating an interactive tutorial at the albert r. mann library, cornell university. she enumerated features she considered essential evidence based library and information practice 2011, 6.2 8 to the ideal online library instruction tutorial: preliminary assessment, branching capabilities, problem-based, concept-based, interactive, assessment, and feedback. she concluded that online library instruction tutorials were ongoing projects under constant and extensive revisions (tancheva, 2003). paul hrycaj (2005) analyzed 65 tutorials created by association of research libraries (arl) member libraries. he followed dewald’s (1999a) methodology, focusing on the active learning elements. those elements were operationally defined as having quizzes at the end of tutorial modules, questions and exercises integrated within tutorial modules, quizzes that required use of a separate browser window, and the capability to send quiz results to an instructor. the results of the study demonstrated an increase in the use of active learning elements, as compared to those reported in dewald’s study (1999a). anderson, wilson, livingston, and locicero (2008) evaluated 274 tutorials used by the medical libraries of the association of american medical colleges. the preestablished indicators for their analysis included whether the tutorials were created by the libraries or by third parties, the subjects of each tutorial, the software used to create them, active learning elements, including the level of interactivity (i.e., whether users were required to perform searches, complete exercises, click on appropriate boxes for additional information, open up the database or software product in a new window, and follow along with the steps in the tutorial), the availability of quizzes or tests, requests for feedback or an evaluation survey, as well as having user profiles. the study found that in spite of the existence of many self-produced web tutorials, few of the tutorials incorporated active learning elements. somoza-fernández and abadal conducted a recent tutorial analysis (2009). they analyzed 180 academic library tutorials using 30 indicators grouped in 5 main areas: general characteristics, content, educational features, browsing and design, and technological features. the indicators for educational features were timing (whether there was a statement of the time estimated for studying each section), pretest, teaching methodology (e.g., presentation of content, guided demonstration, or problem-solving), presence and type of exercises, evaluation or feedback for the exercises, contact with the librarian, a method to evaluate the tutorial (i.e., whether the tutorial could be evaluated or provided the possibility to suggest improvements). results of the study revealed that only 12 of the 180 tutorials satisfied most of the indicators. the authors concluded that web-based tutorials offered by academic libraries are still at an early stage of development (somozafernández & abadal, 2009). pedagogical elements elements of good pedagogy have been proposed by a number of authors. according to the guidelines proposed by dewald (1999b), web-based library instruction combined with good pedagogy should allow users to select their own paths through information, should provide clearly presented information to help learners develop their own understanding of material, and should provide interactivity for learning and assessment. among the previous studies of online tutorials, hrycaj’s research (2005) specifically focused on the extent of the use of active learning in online "stand-alone" library skills tutorials. hrycaj used collins and o'brien’s (2003) definition of active learning, describing it as "the process of keeping students mentally, and often physically, active in their learning through activities that involve them in gathering information, thinking, and problem solving". this definition fits well into dewald’s (1999b) proposed pedagogical guidelines for learning and assessment interactivity and is practical for the analysis and evaluation of web tutorials. however, there is a lack of research to establish how web tutorials address information literacy standards using good pedagogy, especially in the area of science, evidence based library and information practice 2011, 6.2 9 engineering, and technology. this study sought to address this gap by determining which of the information literacy standards for science and engineering/technology were addressed by science information literacy tutorials and by determining the extent to which good pedagogical elements are incorporated in those tutorials. methods sample the sample was created from the list of tutorials selected by sts information literacy committee members (http://wikis.ala.org/acrl/index.php?title=scien ce_information_literacy_tutorials&oldid=421 10; revision as of 7 apr. 2011, at 02:17). each tutorial in this list was selected to address one or more of the information literacy standards for science and engineering/technology (ala/acrl/sts task force, 2006). the list was carefully examined; those that fell into the following categories were excluded from the final sample: • tutorials where the sole purpose was to demonstrate or instruct in the use of a specific resource (e.g., how to search pubmed or cinahl). • resources in the format of guides, handouts, “traditional” web pages, or (most) tilt-based tutorials • collections of tutorials that cover diverse topics or have multiple instructional purposes • tutorials that were no longer accessible • tutorials created outside north america. the final sample was comprised of 31 tutorials, of which 28 were from the united states and 3 from canada. information literacy standards for science and engineering/technology this study used the information literacy standards developed by the ala/acrl/sts task force on information literacy for science and technology: • standard one: the information literate student determines the nature and extent of the information needed. • standard two: the information literate student acquires needed information effectively and efficiently. • standard three: the information literate student critically evaluates the procured information and its sources, and as a result, decides whether or not to modify the initial query and/or seek additional sources and whether to develop a new research process. • standard four: the information literate student understands the economic, ethical, legal, and social issues surrounding the use of information and its technologies and either as an individual or as a member of a group, uses information effectively, ethically, and legally to accomplish a specific purpose. • standard five: the information literate student understands that information literacy is an ongoing process and an important component of lifelong learning and recognizes the need to keep current regarding new developments in his or her field. (ala/acrl/sts task force, 2006) good pedagogical elements the pedagogical elements used for the analysis of tutorials in this study were derived from the studies described in the literature review above (dewald, 1999b; hrycaj, 2005; anderson, wilson, livingston, and locicero, 2008; somoza-fernández and abadal, 2009). they include: • an option for users to select their own paths through information • information presented clearly to help learners develop their own understanding of material • active learning elements o quizzes at the end of tutorial modules evidence based library and information practice 2011, 6.2 10 o questions integrated within tutorial modules o exercises used within tutorial modules o quizzes requiring use of a separate browser window o option to send quiz results to an instructor o surveys for feedback analysis of the tutorials for each tutorial the researcher used an excel™ coding sheet to indicate the presence of the standards outlined above. the columns of the coding sheet indicated the name of the institution that created the tutorial, the tutorial’s url, its subject areas as identified by sts information literacy committee members, sts information literacy standard(s) addressed by the tutorial, and whether each of the selected pedagogical elements was incorporated. the entire coding and analysis procedure was repeated three times to ensure consistency, a research procedure recommended by babbie (2007). results the project analysis included a total of 31 tutorials in various subjects (table 1). table 1 tutorials by subject subject number of tutorials science, general 9 biology 5 biomedicine 4 (public) health 3 medicine 2 nursing 2 engineering 2 patent 2 pharmacy 1 chemistry 1 the categorization of the subject areas followed the original practice of the acrl science & technology section (sts) information literacy committee. results showed that there were comparatively more general science tutorials than other subjects. table 2 illustrates that the majority of tutorials aimed to address standards i, ii, and iii relating to information needs, acquiring information, and critically evaluating the sources of information. further analysis of the tutorials addressing standard i showed that their focus was identifying an information need and the variety of potential sources for information. one clear example of this was the tutorial on finding online health statistics created by university of michigan school of public health (http://www.sph.umich.edu/miinfo/03-hs/index.html). one of the three learning objectives of that tutorial was to “identify the types of statistics and data sets that are available on the internet.” it instructed users on the categories of statistical information available on the internet, and it also provided specific guidance in showing users how to define their information needs and formulate their questions. the majority of tutorials in the sample addressed standard ii, the effective and efficient acquisition of needed information. an example was a tutorial for nurses, “nursing: refining searches”(http://www.lib.ua.edu/ content/tutorials/nursing/ges/?id=200), created by rodgers library at the university of alabama. this tutorial examined the basic ways to refine searches and the use of boolean logic. similar objectives were addressed by the tutorial created by the university of hawaii’s leeward community college library (http://wwwleeward.hawaii.edu /lib/tutorials/bioskills/biolibraryskills_ files/v3_document.htm). the leeward tutorial also taught keyword search strategies and use of boolean operators to narrow or broaden search results. many of the tutorials in the sample addressed standard iii how to critically evaluate information and its sources. the tutorial (http://www2.roosevelt.edu/library/libraryurel oaded/introduction.htm), created by chicago’s evidence based library and information practice 2011, 6.2 11 roosevelt university library, presented training on the evaluation of web information. of the few tutorials that focused on standard iv, most aimed at presenting an understanding of intellectual property, copyright, plagiarism, and how to appropriately cite information sources. an example was a tutorial “searching the pharmacology literature” created by mcgill university’s library (http://mmiweb.mmi. mcgill.ca/dev/loriekloda/healthlibinstruction.htm). the mcgill tutorial had a section, “academic integrity & citing sources” that focused on the importance of ensuring that the content in a paper should be original material, and if not, attribution to its original source must be made clear to the reader. the tutorial instructed users on ways to reference information sources in a paper and how to follow the vancouver style for writing in the biomedical sciences. the “science information tutorial,” developed by the libraries of the university of california – irvine (http://www.lib.uci.edu/ how/tutorials/science_info_tutorial/tutorial.ht ml) is another example of a tutorial focusing on standard iv. an entire section of this tutorial focused on “reading, evaluating and citing information,” where the importance of citing sources, recognizing and avoiding plagiarism, basic citation elements, and citation styles were clearly explained. only two of the tutorials examined in this project addressed standard v, which emphasizes the importance of maintaining current awareness of information literacy through lifelong learning. a tutorial developed by the massachusetts institute of technology (mit) libraries addressed this standard. “scientific publication cycle” (http://techtv.mit.edu/collections/mitlibraries/ videos/3636-scientific-publication-cycle) focused on the scientific publication cycle and scientific research process. it emphasized the value of keeping current with new developments in the literature of the field. another example of a standard v tutorial was “mi-info” (michigan informatics), created by the university of michigan public health library. the section “staying informed” (http://www.sph.umich.edu/mi-info/01-si/) aimed to identify strategies and resources to help users stay informed of news and developments related to their area(s) of interest within public health. the tutorial also helped learners develop an action plan for incorporating these current awareness strategies and resources into their regular work routines. the tutorial analysis also disclosed the incorporation of pedagogical elements in the tutorials (table 3) and the extent to which active learning elements were integrated (table 4). to determine whether a web tutorial provided information clearly, the researcher followed dewald’s guidelines (1999b) to determine whether the instructional design of table 2 standards addressed by tutorials information literacy standard addressed number / percentage tutorials i – determining information need 24 77.42% ii acquiring information effectively and efficiently 25 80.65% iii – evaluating the procured information and its sources critically 24 77.42% iv – understanding the economic, ethical, legal, and social issues 12 38.71% v – understanding the ongoing process of information literacy 2 6.45% note. percentages do not total 100%, as each tutorial could address multiple standards. evidence based library and information practice 2011, 6.2 12 the tutorial followed a logical progression of step-by-step skills needed to accomplish the learning objectives. the specific techniques for this purpose included use of page design elements such as arrows, color, highlighting, white space, small icons, and various font sizes used to structure information for better learner understanding; text that was succinct, broken up, and arranged for maximum clarity; and graphics used to clarify points or to maintain the student’s interest (dewald, 1999b). to decide whether a web tutorial provided the option for users to select their own paths through information the researcher carefully examined the tutorial to determine whether a linked table of contents would be available for users to have a continual overview of the material, and whether there were internal navigational aids (e.g., buttons, icons, or text links) to allow users to review material, move between and within sections as needed, and to receive additional explanations if desired (dewald, 1999b). for hrycaj (2005), the use of a quiz at the end of a module or instructional unit was one of the most common types of active learning in an online tutorial, and the results of this study correspond to hrycaj’s (2005) observations. fifteen tutorials analyzed in this study contained this active learning element. one example was the tutorial, “how do i find … science information” created by the university of california, irvine (uci) libraries (http://www.lib.uci.edu/how/how-do-i.html). this tutorial had a brief quiz at the end of each module, and users could immediately see if their answers were marked correctly. if the user gave a wrong answer, the reason was given. another tutorial with self-assessments was developed by the library at the university of hawaii’s leeward community college. their tutorial, “a guide to biological information and resources” (http://www.leeward.hawaii.edu/lib/bioguide/ index.html) provided a snap quiz at the end of each module. the users were encouraged to answer the questions on their own before going to the answer page at the end of each tutorial. twelve tutorials analyzed for this research had embedded questions within the modules. for example, the engineering students’ tutorial “ges131 library tutorial” (http://fc.eng.ua.edu/ges/) created by rodgers library for science and engineering at the university of alabama, included several selftable 3 pedagogical elements in tutorials pedagogical element number of tutorials having the element percentage provision of clearly presented information 31 100.00% option for users to select their own paths through information 27 87.10% a ctive l earning e lem ent quizzes/exercises at the end of tutorial modules 15 48.39% questions integrated within tutorial modules 12 38.71% exercises used within tutorial modules 15 48.39% quizzes requiring use of separate browser windows 2 6.45% options to send quiz results to instructor 3 9.68% surveys for feedback 5 16.13% note. percentages do not total 100%, as each tutorial could incorporate multiple pedagogical elements. evidence based library and information practice 2011, 6.2 13 assessment questions throughout each of the tutorial modules. another example was the tutorial on reading scientific papers developed by purdue university libraries (http://www.lib.purdue.edu/phys/assets/scipa pertutorial.swf). questions in this tutorial were integrated throughout, to initiate users’ deep thinking and to advance the lesson. exercises in tutorials often give users a search exercise. some of them involve a dual screen one screen provides directions and the other screen provides the resource for the user to carry out the directions (hrycaj, 2005). an example of the dual screen approach was the tutorial created by the university of massachusetts amherst libraries, “herbs, spices & medicinal plants” (http://umalws1.library.umass.edu/instruction /tutorials/herbs/index.html) (see figure1). according to hrycaj (2005), quizzes that require use of a separate browser window test users’ skills, because the user is expected to complete a research exercise in a separate window, find the results, and then return to the tutorial to enter the answer. among all the sample tutorials analyzed, only two had this element. one was the tutorial “evaluating web sites” created by chicago’s roosevelt university library (http://www2.roosevelt.edu/library/libraryurel oaded/introduction.htm). users were instructed to follow the link to a site (which opened in a new window), view the site, close the new browser, and return to the page when they were finished in order to take the quiz (http://www2.roosevelt.edu/library/libraryurel oaded/siteone.htm). an option to send quiz results to an instructor may not be active learning technique in itself, figure 1 example of dual screen for exercises in tutorials. (http://umalws1.library.umass.edu/instruction/tutorials/herbs/books/activity.html#). evidence based library and information practice 2011, 6.2 14 yet it can motivate users to be learning actively. three of the analyzed tutorials offered this option. one example was the university of alabama rodgers library tutorial “ges131 library tutorial” (http://fc.eng.ua.edu/ges/). this tutorial included homework quizzes at the end of some modules. the users were instructed to record their responses to each question and then submit the responses to the instructor. a survey for feedback is a technique that can initiate users’ critical thinking. five tutorials analyzed for this project offered this option. an example was the tutorial “introduction to evidence-based medicine” from duke university medical center library. this survey included both close-ended and openended questions to collect user feedback (http://www.hsl.unc.edu/services/tutorials/eb m/evalthis.htm). table 4 demonstrates the extent to which active learning elements were incorporated into the science, engineering, and technology information literacy tutorials analyzed. the majority of the tutorials incorporated some active learning element(s). indeed, 50% contained one or two active learning elements. almost a quarter of the tutorials analyzed contained three or four active learning elements, but none contained more than four. seven (almost a quarter of those studied) did not contain any active learning element. discussion the results of this study show that these tutorials address the sts information literacy standards, particularly standards i, ii, and iii relating to information needs, acquiring information, and critically evaluating information sources. those that illustrate these features particularly well have been noted as examples, where appropriate, within the text. designers of web tutorials in science, engineering, and technology have generally seen the importance of good pedagogy. the tutorials presented information clearly, and in most cases they provided the option for users to select their own paths through information. over three quarters of the tutorials in the sample contained some sort of active learning component. yet the level to which good pedagogy and especially active learning elements were incorporated into web tutorials was unsatisfactory. almost one quarter of the sample tutorials disregarded the need for active learning elements and another quarter had only the minimum of such elements. science, engineering, and technology disciplines are rapidly changing, and the body of knowledge in these fields is carried by a wide variety of information sources and formats (ala/acrl/sts task force, 2006). the study showed that libraries are able to design instructional materials to help improve users’ critical thinking and lifelong learning table 4 number of active learning elements in tutorials number of active learning elements incorporated number of tutorials percentage of tutorials 0 7 22.58% 1 8 25.81% 2 8 25.81% 3 4 12.90% 4 4 12.90% 5 0 0% 6 0 0% evidence based library and information practice 2011, 6.2 15 skills, instead of solely demonstrating or instructing in the use of a specific resource. web tutorials, while teaching asynchronously, can be available 24 hours a day and 7 days a week. they should especially appeal to those who cannot come to campus for face-to-face instructions. good pedagogical elements are the basis for effective information literacy instruction. with web tutorials, clearly presented information and the provision of the option for learners to select their own paths for information can help learners develop their own understanding of material and acquire deep learning of the content (dewald, 1999b). active learning elements are critical to the successful achievement of the instructional purpose, and real learning cannot possibly happen without these elements (dewald et al., 2000). this study provided examples of web tutorials that address information literacy standards with satisfactory incorporation of good pedagogical elements. one example is the “science information tutorial” created by the university of california, irvine libraries (http://www.lib.uci.edu/how/tutorials/science_ info_tutorial/tutorial.html). it was one of the best among the tutorials evaluated here in terms of addressing information literacy standards by incorporating good pedagogical elements. aiming at addressing sts information literacy standard i, ii, iii, and iv, it clearly stated its purpose and objectives in observable behaviors from the beginning. it provided the option for users to select their own paths through information via a main menu on the left frame of the screen on all the web pages and used “previous” and “next” buttons at the bottom of each page. it provided clearly presented information via various techniques, e.g., color, highlighting, white space, small icons, different fonts and font sizes, text that was succinct, broken up, and arranged for maximum clarity, and used graphics to clarify points or to maintain students’ interest. for each module, it had a pre-test, review, and a quiz at the end. questions and activities were integrated in all the tutorial modules to initiate users’ critical thinking. users were motivated by a completion certificate and were encouraged to give their feedback and comments when successfully completing each module. another tutorial that successfully incorporated good pedagogical elements was one created by the university of massachusetts amherst libraries (http://umalws1.library.umass/instr uction/tutorials/herbs/index.html). that tutorial instructed users to search for, evaluate, and find information in the field of herbs, spices, and medicinal plants. it shared various features of the “science information tutorial” from the university of california, irvine libraries. both provided the option for users to select their own paths through information via a main menu on the left of the screen on all the web pages and the “previous” and “next buttons” at the bottom of each page. they clearly presented information via various techniques. the amherst libraries tutorial offered exercises that involved dual screens (figure 1). it used both pre-assessment and post-assessment tools. users were encouraged to contact librarians for more ideas on how to find the precise data or resource that they needed. they were urged to join in an anonymous evaluation of the tutorial and could even receive a response if they provided an email address. it is both important and feasible to base information literacy instruction on the sts standards with the incorporation of good pedagogical elements. although there are several successful examples in this area, there is clearly more work to be done, particularly in relation to sts standards iv and v which cover ethical and legal issues and the need for keeping up to date with lifelong learning skills there are two distinct approaches to learning: surface learning, in which the learner memorizes the information, and deep learning, in which the learner tries to understand the information or to seek meaning (alexander, 1995). surface approaches are generally associated with poor learning outcomes, while deep approaches evidence based library and information practice 2011, 6.2 16 tend to yield higher quality learning outcomes (entwistle & ramsden, 1983; watkins, 1983). numerous web tutorials have been designed to address sts information literacy standards, which have gone beyond the teaching ideas and techniques compiled by the sts information literacy committee (http://wikis.ala.org/acrl/index.php?title=teac hing_tips&oldid=42295; revision as of 7 apr. 2011, at 06:26). however, it is important to have active learning elements integrated into all web tutorials, to ensure that users will acquire deep learning from them. this study achieved its objective of identifying which of the sts information literacy standards were addressed by web tutorials. future research is needed to expand the findings of this study and to determine which performance indicators of specific literacy standards are addressed by web tutorials. this recommended study might provide a better idea of what web tutorials can do to help learners acquire information literacy skills and become lifelong learners. feedback from learners themselves may provide a more realistic picture of how web tutorials are accepted as information literacy instruction tools in science, engineering, and technology. it may also reveal what content and pedagogical elements appeal to learners as truly effective in helping them develop their information literacy skills. a survey is needed to elicit constructive comments and suggestions from users of web tutorials in these disciplines. this study excluded tutorials that focused on the usage of a specific resource, collections of tutorials covering diverse topics, and those with diverse instructional purposes. the tutorials included in this project originated only from north america and excluded many tutorials from other countries. these exclusions ensured that the study was feasible, however they may have led to bias or skewing of the research findings. the focus on web tutorials addressing the information literacy standards in science, engineering, and technology also meant that the study was narrow in scope. future research may be needed to analyze the web tutorials that deal with the acrl information literacy competency standards for higher education. such studies could provide insight into a broader picture of academic libraries’ efforts in teaching information literacy standards via web tutorials and the extent to which they demonstrate the importance of good pedagogy in this kind of instruction. conclusion many librarians have accepted web tutorials as effective information literacy instruction tools for science, engineering, and technology. yet ensuring that they are real learning experiences for learners remains a challenge. the study shows that previous guidelines on good pedagogy and standards for information literacy have been integrated into web tutorials. however, there is still work to be done, particularly with regard to standards that focus on ethical and legal issues, using information literacy as a component of lifelong learning, and ensuring that a range of active learning elements is incorporated. if tutorial developers further considered employing these pedagogical techniques, learners might better experience deep learning. references acrl science information literacy tutorials. (7 may 2011). retrieved 2 june 2011 from http://wikis.ala.org/acrl/index.php? title=science_information_literac y_tutorials&oldid=42110 ala/acrl/sts task force on information literacy for science and technology. (n.d.). information literacy standards for science and engineering/technology. retrieved 1 june 2011 from http://www.ala.org/ala/mgrps/divs/ac rl/standards/infolitscitech.cfm alexander, s. (1995). teaching and learning on the world wide web. southern cross university, retrieved 1 june 2011 from evidence based library and information practice 2011, 6.2 17 http://ausweb.scu.edu.au/aw95/educat ion2/alexander/ american library association. (1989). presidential committee on information literacy: final report. retrieved 1 june 2011 from http://www.ala.org/ala/mgrps/divs/ac rl/publications/whitepapers/presidenti al.cfm anderson, r. p., wilson, s. p., livingston, m. b., & locicero, a. d. (2008). characteristics and content of medical library tutorials: a review. journal of medical library association, 96(1), 6163. doi:10.3163/1536-5050.96.1.61 association of college and research libraries. (18 jan. 2000). information literacy competency standards for higher education. retrieved 1 june 2011 from http://www.ala.org/ala/mgrps/divs/ac rl/standards/standards.pdf babbie, e. r. (2008). basics of social research. 4th ed. belmont, ca: thomson/ wadsworth publishing. bawden, d., devon, t. k., & sinclair, i. w. (2000). desktop information systems and services: a user survey in a pharmaceutical research organization. international journal of information management, 20(2), 151-160. doi:10.1016/s0268-4012(99)00062-6 collins, j. w. iii, & o'brien, n. p. (eds.) (2003). the greenwood dictionary of education. westport, ct: greenwood press. dewald, n. h. (1999a). transporting good library instruction practices into the web environment: an analysis of online tutorials. the journal of academic librarianship, 25(1), 26-31. doi:10.1016/s0099-1333(99)80172-4 dewald, n. h. (1999b). web-based library instruction: what is good pedagogy? information technology and libraries, 18(1), 26-31. dewald, n., scholz-crane, a., booth, a., & levine, c. (2000). information literacy at a distance: instructional design issues. journal of academic librarianship, 26(1), 33-44. doi:10.1016/s0099-1333(99)00121-4 donaldson, k. a. (2000). library research success: designing an online tutorial to teach information literacy skills to first-year students. the internet and higher education, 2(4), 237-251. doi:10.1016/s1096-7516(00)00025-7 entwistle, n. j., & ramsden, p. (1983) understanding student learning. london: croom helm. franks, j. a., hackley, r. s., straw, j. e., & direnzo, s. (2000). developing an interactive web tutorial to teach information competencies. journal of educational media & library sciences, 37(3), 235-255. grassian, e. s., & kaplowitz, j. r. (2009). information literacy instruction: theory and practice (2nd ed.). new york, ny: neal-schuman. hrycaj, p. l. (2005). elements of active learning in the online tutorials of arl members. reference services review, 33(2), 210-218. doi: 10.1108/00907320510597417 slebodnik, m., & zeidman-karpinski, a. (summer 2008). science and technology resources on the internet: resources for information literacy instruction in the sciences. issues in science and technology librarianship. retrieved from http://www.library.ucsb.edu/istl/08 summer/internet.html evidence based library and information practice 2011, 6.2 18 somoza-fernández, m., & abadal, e. (2009). analysis of web-based tutorials created by academic libraries. the journal of academic librarianship, 35(2), 126-131. doi:10.1016/j.acalib.2009.01.010 stanger, k. (2009). implementing information literacy in higher education: a perspective on the roles of librarians and disciplinary faculty. libres: library and information science research electronic journal, 19(1): 1-6. retrieved 1 june 2011 from http://libres.curtin.edu.au/libres19n1/ tancheva, k. (2003). online tutorials for library instruction: an ongoing project under constant revision. april 10-13, 2003, charlotte, north carolina acrl eleventh national conference: learning to make a difference. retrieved 1 june 2011 from http://www.ala.org/ala/mgrps/divs/ac rl/events/pdf/tancheva.pdf watkins, d. a. (1983). depth of processing and the quality of learning outcomes. instructional science, 12(1), 49-58. doi: 10.1007/bf00120900 / evidence based library and information practice science information literacy tutorials and pedagogy sample the sample was created from the list of tutorials selected by sts information literacy committee members (http://wikis.ala.org/acrl/index.php?title=science_information_literacy_tutorials&oldid=42110; revision as of 7 apr. 2011, at 02:17). each tutoria... standard one: the information literate student determines the nature and extent of the information needed. standard two: the information literate student acquires needed information effectively and efficiently. standard three: the information literate student critically evaluates the procured information and its sources, and as a result, decides whether or not to modify the initial query and/or seek additional sources and whether to develop a new research pr... standard four: the information literate student understands the economic, ethical, legal, and social issues surrounding the use of information and its technologies and either as an individual or as a member of a group, uses information effectively, et... standard five: the information literate student understands that information literacy is an ongoing process and an important component of lifelong learning and recognizes the need to keep current regarding new developments in his or her field. (ala/acrl/sts task force, 2006) evidence based library and information practice evidence based library and information practice 2011, 6.1 81 evidence based library and information practice evidence summary physicists and astronomers use google as a starting point for specific queries, but do not intentionally use it to search for articles a review of: jamali, h. r., & asadi, s. (2010). google and the scholar: the role of google in scientists’ information seeking behaviour. online information review, 34(2), 282-294. reviewed by: laura newton miller science & engineering librarian carleton university ottawa, ontario, canada email: laura_newtonmiller@carleton.ca received: 15 nov. 2010 accepted: 16 feb. 2011 2011 newton miller. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to determine how google’s general search engine impacts the information-seeking behaviour of physicists and astronomers. design – using purposive stratified nonrandom sampling, a mixed-methods study was conducted which included one-on-one interviews, information-event cards, and an online questionnaire survey. setting – department of physics and astronomy at university college london. subjects – the researchers interviewed 26 phd students and 30 faculty members (23% of the department’s 242 faculty and students), and 24 of those participants completed information-event cards. a total of 114 respondents (47.1% of the department members) participated in the online survey. methods – the researchers conducted 56 interviews which lasted an average of 44 minutes each. these were digitally recorded, fully transcribed, and coded. the researchers asked questions related to information-seeking behaviour and scholarly communication. four information-event cards were given to volunteer interviewees to gather critical incident information on their first four information-seeking actions after the interview. these were to be completed preferably within the first week of receiving the cards, with 82 cards completed by 24 participants. once initial analysis of the mailto:laura_newtonmiller@carleton.ca� evidence based library and information practice 2011, 6.1 82 interviews was completed, the researchers sent an online survey to the members of the same department. main results – this particular paper examined only the results related to the scholars’ information-seeking behaviour in terms of search engines and web searching. details of further results are examined in jamali (2008) and jamali and nicholas (2008). the authors reported that 18% of the respondents used google on a daily basis to identify articles. they also found that 11% searched subject databases, and 9% searched e-journal websites on a daily basis. when responses on daily searching were combined with those from participants who searched two to three times per week, the most popular method for finding research was by tracking references at the end of an article (61%). this was followed by google (58%) and toc email alerts (35%). responses showed that 46% never used google scholar to discover research articles. when asked if they intentionally searched google to find articles, all except two participants answered that they do not, instead using specific databases to find research. the researchers noted that finding articles in google was not the original intention of participants’ searches, but more of a by-product of google searching. in the information-event card study, two categories emerged based on the kinds of information required. this included participants looking for general information on a specific topic (64%, with 22 cases finding this information successfully), and participants knowing exactly what piece of information they were seeking (36%, with 28 cases finding information successfully). there was no occurrence of using google specifically to conduct a literature search or to search for a paper during this information-event card study, although the researchers say that google is progressively showing more scholarly information within its search results. (this cannot be ascertained from these specific results except for one response from an interviewee.) the researchers found that 29.4% of respondents used google to find specific pieces of information, although it was not necessarily scholarly. conclusion – physics and astronomy researchers do not intentionally use google’s general search engine to search for articles, but, google seems to be a good starting point for problem-specific information queries. commentary oh, google. librarians certainly have a love/hate relationship with you. instruction librarians are continually faced with students who regularly use google as a research tool. its simplicity of use may not necessarily lead to the best research. but google’s popularity is not limited to students. it is important for librarians to know how faculty and other researchers are finding information in order to better meet their research needs. jamali and asadi’s study aims to learn more about the information-seeking behaviour of physicists and astronomers in terms of using search engines for web searching. they have written a very good literature review on the debate surrounding the use of google in academia. this paper is an extrapolation of results from a bigger study of information-seeking behaviours of physicists and astronomers. details of further results are examined in jamali (2008) and jamali and nicholas (2008). although jamali and asadi have written detailed results, something seems to be missing in writing this paper from a bigger study. it’s difficult to pinpoint exactly what is missing, but the conclusions don’t necessarily work together with the results. perhaps it is the reliance on one or two interviewee statements to make claims that are not necessarily apparent from the results. for example, the researchers state that google is progressively showing more scholarly information within its search results. this may be true, but it cannot be ascertained from the results except for one response from an interviewee. evidence based library and information practice 2011, 6.1 83 perhaps more concrete definitions need to be put in place. for example, a definition would have been beneficial in explaining what is meant by “identifying an article.” the researchers show that 18% of respondents use google on a daily basis to identify articles, and when this is combined with those who search two to three times per week, google is used by 58% of respondents as the second most popular method for finding an article (behind tracking references at the end of an article). some clarification would be beneficial in this case. when asked if researchers intentionally searched google to find articles, all except two people answered that they do not (using particular databases instead). it looks like physicists and astronomers are mostly just looking for specifics, perhaps bibliographic information for a reference, but the numbers are not completely clear. it would be valuable for readers to see a copy of the questions asked in this case. it would also be useful to know what researchers use to track a reference at the end of an article. it is not entirely apparent if they use google, a subject database, or some other method for this task. results are gleaned from all three methods used in the study. the information-event component of the study can easily be replicated because the authors show a copy of an information-event card and clearly explain the procedures given to participants. however, although the authors stated that the interview and survey questions were piloted before the study was executed, it is difficult for readers of the paper to replicate the study for further research because there are no copies of the interview or survey questions. this research used both phd students and faculty as participants. however, it does not separate the results of the students versus faculty, except in mentioning that three student interviewees liked the google brand. it would have been interesting to note any discrepancies within the two groups, as differences were described for other results in jamali and nicholas (2008). the researchers raise an interesting point about the reliance of google within specific subfields of physics and astronomy in terms of the high availability of open access materials for this information. it would be useful to repeat this study now that there is a proliferation of open access materials in this and other science fields. this study was conducted in 2005 and 2006. in web years, this feels like a lifetime ago. the researchers acknowledge that although google scholar was not popular among these scientists at the time of the study (46% had never used it), this is an area worthy of further investigation five years after the original study. since google scholar now plays a much more predominant role in academia, it is difficult to determine what actions should be taken from this particular research. anecdotal evidence from academic librarians will inform us that google scholar is the first choice for many patrons. what should libraries do to connect google to the library collections? how do we get patrons to come back to the library website (assuming that they visited the website in the first place, which many have not)? some libraries have a google scholar search option directly on the home page, in the hopes that if one is going to use google scholar, accessing through the library website will at least lead patrons back to library eresources. others might argue against this, reasoning that if it is a less reliable source, the library should not put a link to google scholar on the homepage when other, more reliable, library resources are vying for exposure on precious web real estate. a compromise to this is to list google scholar with the list of other library databases. it is questionable whether patrons will take the time to go this route unless specifically told about the benefits of linking up to the library’s proxy server. it is certainly a predicament with which many librarians struggle. the authors of this paper acknowledge that these are results from one department of one institution and that caution should be taken when generalizing these results. regardless, studies such as this are helpful to librarians in evidence based library and information practice 2011, 6.1 84 order to better understand the needs of their science patrons. knowing how faculty and students find information can help in planning future instruction sessions and website redesign projects. because this study is somewhat disjointed in its results, it would be beneficial for librarians to read the other paper by jamali and nicholas (2008) in order to see a fuller picture of this particular research. references jamali, h. r. (2008). information-seeking behaviour of physicists and astronomers: an intradisciplinary study. university college london. unpublished phd thesis. jamali, h. r., & nicholas, d. (2008). information-seeking behaviour of physicists and astronomers. aslib proceedings, 60(5), 444-462. / evidence based library and information practice evidence summary evidence based library and information practice evidence based library and information practice 2011, 6.2 51 evidence based library and information practice evidence summary pubmed is slightly more sensitive but more time intensive to search than ovid medline a review of: katchamart, w., faulkner, a., feldman, b., tomlinson, g., & bombardier, c. (2011). pubmed had a higher sensitivity than ovid-medline in the search for systematic reviews. journal of clinical epidemiology, 64(7), 805-807. doi:10.1016/j.jclinepi.2010.06.004 reviewed by: cari merkley associate professor library, mount royal university calgary, alberta, canada email: cmerkley@mtroyal.ca received: 01 mar. 2011 accepted: 25 may 2011 2011 merkley. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective — to compare the results of searching the medline database through ovid and the free online version of pubmed administered by the national library of medicine for randomized controlled trials on the subject of the drug methotrexate (mtx) for patients suffering from rheumatoid arthritis. design — comparative analysis of search results. setting — searches conducted by researchers affiliated with mahidol university in bangkok, thailand, and the university of toronto and the university health network in toronto, ontario. subjects — a total of 3966 search results obtained from ovid medline and pubmed. methods — this study employs an ovid medline search strategy originally created for a published systematic review that identified randomized controlled trials on mtx and rheumatoid arthritis (katchamart, trudeau, phumethum, & bombardier, 2009). two of the authors of the original systematic review (katchamart and bombardier) are among the authors of this current study. appropriate medical subject heading (mesh) terms and their synonyms were identified for the three main concepts (rheumatoid arthritis, mtx, and randomized controlled trials). the search was performed in ovid medline, seeking articles in any language that met the mailto:cmerkley@mtroyal.ca� evidence based library and information practice 2011, 6.2 52 search criteria, from the earliest date covered by medline to january 2009. each mesh or keyword term within a concept was searched separately, and then combined with other like terms using the boolean operator or. the searches for the three concepts were finally combined using and. the ovid medline search was then translated for use in pubmed by an information professional. the formatting and terminology used in some of the original ovid medline search statements had to be changed so they would work in the new database environment, but the researchers tried to ensure that the two searches were as similar as possible. the translated search was then executed in pubmed. the final results, as well as the number of articles retrieved for each key search concept (rheumatoid arthritis, mtx, and randomized controlled trials), were then compared. the final results were further analyzed for measures of sensitivity, precision, and number needed to read. sensitivity is calculated by the number of eligible studies found in a database divided by the “total number of eligible studies in the review” multiplied by 100 (katchamart, faulkner, feldman, tomlinson, & bombardier, p. 806). eligible studies were identified using the inclusion/exclusion criteria developed by katchamart et al. the figure for “total number of eligible studies in the review” is taken from that same study, which forms the “gold standard” for this analysis (katchamart et al., p. 806). precision is calculated by dividing the total number of eligible citations from a database by the total number of citations returned by the database for the search multiplied by 100 (katchamart et al., p. 806). the number needed to read (nnr) formula used by the authors is 1/precision, taken from a study by bachman, coray, estermann, and ter riet (2002). main results — the pubmed search found more results than ovid medline for each of the three key concepts – rheumatoid arthritis, mtx and randomized controlled trials. once the three concepts were combined, pubmed found 106 more articles than ovid medline (2036 vs. 1930). once the review eligibility criteria were applied to the search results from pubmed, 18 eligible articles were identified, one more article than in ovid medline. the authors indicated that the additional article located in pubmed was from a journal that was not yet indexed by medline at the time the relevant article was published. to determine database sensitivity, these numbers were then divided by 20, the total number of eligible studies located in the katachamart et al. 2009 review, which employed tools like embase and strategies like hand searching in addition to medline in order to identify relevant studies. because of the additional study it located, the sensitivity of pubmed was determined to be slightly higher than ovid medline (90% vs. 85%). there was little difference between the two databases in terms of precision and nnr. precision for ovid medline was calculated at 0.881% and at 0.884% for pubmed. the nnr was 114 for ovid medline and 113 for pubmed. conclusion — the authors state that while pubmed had a higher calculated sensitivity than ovid medline in the context of this particular search because it contained content not indexed by ovid medline that proved to be relevant for this topic, its precision and nnr were almost equal to medline’s. some technical limitations of the pubmed interface were experienced by researchers during the study, such as periodic instability and the inability to save and modify searches and their results line by line. these same issues did not arise while using ovid medline. the need for a skilled translation of ovid medline searches for use in the pubmed interface was also emphasized by the authors, as differences in syntax and formatting that are not properly addressed could impact pubmed’s sensitivity and precision. evidence based library and information practice 2011, 6.2 53 commentary as the study opens, the authors remark on the fact that researchers are turning to pubmed because it is free. the results suggest that, in the context of this particular search, the free tool slightly outperformed the subscription database. the broader content of pubmed proved to be very significant for this topic area, and future researchers may want to compare the date coverage of the indexing of key journals in both databases because of this experience. however, the fact that pubmed returned higher results and a relevant article missed by medline is tempered by the list of the technical challenges the researchers experienced in using it. reference to the pubmed user experience outside of the results themselves is very relevant to readers who are making decisions around the best use of their time and dollars, although the stability issues experienced by the authors may have been temporary in nature or addressed by the u.s. national library of medicine in subsequent updates. one wonders if the same critical eye could be applied to the ovidsp platform, even though the authors indicate that they did not experience the same challenges when using ovid as they did with pubmed. were there issues specific to ovid medline that they did not encounter using pubmed, other than cost? further research, as the authors point out, is also needed to compare pubmed’s and ovid medline’s performance across a variety of topics, not just the one searched here. this study is inextricably linked with the systematic review which provided the authors with their search strategy and inclusion/exclusion criteria, and at times it relies heavily on that review to fill in details that are not provided here. this is problematic, particularly when figures are directly imported from the systematic review for use in calculating sensitivity or when it may be useful for readers to know the process by which citations were reviewed for relevance, something that is covered by a single sentence in this study. the inclusion/exclusion process is relevant to the pubmed and ovid medline discussion as it feeds into measures of sensitivity, precision, and nnr. other details glossed over here include the date coverage of the searches. one assumes that the ovid medline search covers 1950 to january 2009 as it did in the systematic review, and that the same dates were covered in pubmed, but it is not explicitly stated. finally, the author presented the formula for nnr, but did not explain what its value means in this particular study. for example, bachmann et al. (2002) state that in the context of their study the “number needed to read figure shows how many abstracts have to be read to identify one diagnostic study” (p. 656). how the 1/precision formula was determined for nnr is not clear in bachmann et al. or in this study, and subsequent studies have used this same calculation but with slightly different understandings of what it means. for example, mckibbon, wilczynski, and haynes (2004) define nrr as “number of articles that are needed to be read to obtain one that is clinically relevant and has high-quality methods” (p. 13). it has also been proposed as an alternative to journal impact factor as an indicator of the quality of a particular journal (toth, muir, & brice, 2005). the use of the existing methodology from the systematic review provides the authors with a model to compare their results to, but readers may wish to extend their reading to the original publication to get the most out of this more recent study. references bachmann, l. m., coray, r., estermann, p., & ter riet, g. (2002). identifying diagnostic studies in medline: reducing the number needed to read. jamia, 9(6), 653-658. doi: 10.1197/jamia.m1124 katchamart, w., trudeau, j., phumethum, v., & bombardier, c. (2009). efficacy and toxicity of methotrexate (mtx) monotherapy versus mtx combination therapy with nonevidence based library and information practice 2011, 6.2 54 biological disease-modifying antirheumatic drugs in rheumatoid arthritis: a systematic review and meta-analysis. annals of the rheumatic diseases, 68, 11051112. doi: 10.1136/ard.2008.099861 mckibbon, k. a., wilczynski, n. l., & haynes, r. b. (2004). what do evidencebased secondary journals tell us about the publication of clinically important articles in primary healthcare journals? bmc medicine, 2(33), 1-13. doi:10.1186/1741-7015-2-33 toth, b., muir, j. a., & brice, a. (2005). the number needed to read—a new measure of journal value. health information & libraries journal, 22(2), 81-82. doi:10.1111/j.14711842.2005.00568.x / evidence based library and information practice evidence summary microsoft word es7_970_final_final evidence based library and information practice 2008, 3:1  75 evidence based library and information practice       evidence summary    too few articles in the journal literature on instruction in academic libraries are  research‐based    a review of:  crawford, gregory a., and jessica feldt. “an analysis of the literature on instruction in  academic libraries.” reference & user services quarterly 46.3 (spr. 2007): 77‐87.      reviewed by:  kurt blythe  serials access librarian, university of north carolina‐chapel hill  chapel hill, north carolina, united states of america  e‐mail: kcblythe@email.unc.edu      received : 14 december 2007    accepted : 11 february 2008      © 2008 blythe. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to discover which journals  publish a preponderance of the literature on  instruction in academic libraries, and to  further learn what topics are discussed in  the literature and what methods of inquiry  are used to explore those topics.     design – comparative analysis.     setting – the eric database.    subjects – journal articles published  between 1971 and 2002, found in the eric  database via the descriptor ‘library  instruction.’     methods – journal articles published  between 1971 and 2002 were searched in the  eric database using the term ‘library  instruction,’ because the term has been used  by eric since the inception of the database.  article characteristics such as year of  publication, journal in which a given article  was published, and types of articles were  identified. after normalizing the  identification of article types by comparing  individual codifications of a random sample  of articles from the results list, the  researchers coded all of the articles  according to the articles’ main topics. these  article topics were then condensed into  broader topics (e.g. articles coded initially as  evidence based library and information practice 2008, 3:1  76 addressing the topics of instruction to  “graduate students” or “high school  students” were condensed under the  heading “instruction of specific  populations”). those articles deemed to  have a research component were further  analyzed in terms of research methodology,  scope, and use of statistics. the questions  guiding the authorsʹ study consisted of:    • in which journals are articles on  instruction in academic libraries  published?    • what are the topics of the articles  that have been published?    • how has the literature of instruction  in academic libraries changed over  the years?    • what is the nature of research  [authorsʹ emphasis] articles on  instruction in academic libraries?    • for research articles, what are the  research methods used and what  types of statistics are utilized? (77‐ 78).    main results – the search found 791 articles  appearing in 90 journals. however, only six  journals were responsible for publishing  60% of the articles on the results list. based  on the categorization into topics or types,  five article types – “general essays,” articles  on “instruction for searching online catalogs  and databases,” “articles on course‐ or  assignment‐related instruction,”  “programmatic and management issues,”  and “specific institution instruction” –  accounted for 54% of the total articles;  eleven other topics are represented in the  remaining articles. only 24.5% of the articles  could be considered “research‐based.”  while nine types of research methods were  used, 62.3% of the research‐based articles  relied on surveys or questionnaires.  however, taking into account what could be  an expected decline in interest for studying  instruction for microforms, and an increased  interest in studying instruction for the  internet, the types of articles published each  year between 1971 and 2002 remained  generally static.       conclusion – the authors conclude that,  unsurprisingly, a select few journals are  responsible for publishing the majority of  articles on instruction in academic libraries,  and note that these journals are “closely  associated with academic librarianship and  instruction” (85). the authors’ findings  concerning the types of articles published  over the course of the date range examined,  although interesting, are equally  unsurprising.  what is surprising and  alarming, is that only a quarter of all articles  published are research‐based. furthermore,  those articles that were found to be research‐ based did not often employ sophisticated  research methods, preferring for example,  the use of descriptive statistics over  inferential statistics. ideally, this article will  spur the development and publication of  more, and more rigorous, research‐based  scholarship in addition to pointing authors  both to the journals most likely to publish  their research and to the topics that have  been historically (since 1971)  underrepresented in the literature.    commentary     the authors look at a sample of articles  covering various topics in instruction in  academic libraries. they do well to lay out  the parameters of their study: that their  sample was arrived at solely by searching  the eric database, that their search was  performed using only the descriptor “library  instruction,” and that the search was limited  to articles published between 1971 and 2002.  readers may remain curious as to how and  by whom the descriptor was generated in  the first place, and assigned to articles in the  second, just as they may question whether  eric was the right choice in database, but  neither an explanation in the case of the  former questions, nor a comparison with  other databases in the case of the latter is  evidence based library and information practice 2008, 3:1  77 provided. however, the authors  satisfactorily explain the reason for limiting  their search in the fashion they did, and also  admit the imprecision inherent in any  analysis of article subjects. in fact, the  methodology of the study is admirably laid  out, and given the constraints of the search,  the resulting article provides a good analysis  of what types of scholarship have been done  in the realm of library instruction, and  therefore will prove interesting to most  librarians, educators, and scholars engaged  in that arena.      in addition to the major findings of the  study, many tidbits of information are  provided, such as the fact that only three  articles on instruction for audiovisual  materials, multimedia, or microforms have  been published since 1994; or that articles  concerning internet instruction began being  published in 1993 and peaked in 1996. it is  the major findings though that will prove  the most useful to students, scholars, and  practitioners of library instruction. these  findings effectively point authors to the  journals most likely to publish articles on a  given topic; illustrate which topics are the  most popular, therefore indicating trends on  the one hand and areas that may need  further exploration on the other; and  provide what amounts to a wish list on the  part of the authors for the types of research  they feel the field needs in order to grow.  this last element, concerning research or the  lack thereof in the scholarship on instruction  in academic libraries, is really the thrust of  the article. the authors assert that data has  been collected regarding issues in library  instruction but that it goes unused. in order  for library instruction to continue to  contribute to the larger missions of  education and information literacy, scholars  must do a better job of more rigorously  researching their ideas and supporting their  conclusions with properly applied statistical  evidence.     using evidence in practice   evidence for removal of a reference collection in an academic health sciences library   linda seale john w. scott health sciences library 2k3.28 walter c. mackenzie health sciences centre university of alberta edmonton, alberta, canada email: linda.seale@ualberta.ca   trish chatterley john w. scott health sciences library 2k3.28 walter c. mackenzie health sciences centre university of alberta edmonton, alberta, canada email: trish.chatterley@ualberta.ca   marlene dorgan john w. scott health sciences library 2k3.28 walter c. mackenzie health sciences centre university of alberta edmonton, alberta, canada email: marlene.dorgan@ualberta.ca     received: 21 oct. 2013   accepted: 5 feb. 2014      2014 seale, chatterley, and dorgan. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   setting   the survey was conducted at the john w. scott health sciences library, a large academic health sciences library, at the university of alberta in edmonton, alberta, canada.  the library serves undergraduate and graduate students, faculty, and researchers in five faculties (medicine and dentistry, nursing, pharmacy and pharmaceutical sciences, rehabilitation medicine and school of public health), as well as members of a number of affiliated research institutes, university of alberta affiliates, members of the neos consortium (a resource sharing collaboration of hospital, government, and academic libraries in the province of alberta), and the general public. problem   one goal of the library is to maintain the relevancy of and provide optimum access to the library’s print resources. in the fall of 2011 it was noted that observational and anecdotal data gathered over the previous two years had indicated that use of the print reference collection was declining. less re-shelving was required of titles removed from the shelves in the reference room, and fewer requests were made at the service desk for print reference titles. it is assumed the decline can be at least partially attributed to a change in the library’s collections policy which now stipulates preferential purchasing of reference titles in electronic format to enhance access. a large deselection project was completed in the summer of 2011, with approximately half the titles from the reference collection transferred to the stacks so they could circulate. four hundred and nine titles remained. evidence was needed for further decisions on the future of the collection, and on its possible elimination.   evidence   a literature search was conducted, revealing that very little recent literature exists on the deselection of reference collections or on their replacement by electronic or virtual collections. a shelf list of the reference collection was printed and a spreadsheet created to allow recording of the use of each title. a staff member verified that all titles on the shelf list were present and missing titles were located. with the cooperation of the shelving supervisor, the student shelving staff was instructed to record each title re-shelved fora 12-month period (january-december 2012). notices were placed in the reference room asking clients not to re-shelve books.   in january 2013, the results were summarized and analyzed. while the evidence is somewhat imperfect as some students may have re-shelved their own books, it is the best available indicator of usage.   the evidence indicated that only 33% (n=136) of titles were used during that year.  of these, eight had more than ten uses and constituted 39% of overall use. the two top-used titles in this group together accounted for 22% of overall use. a further 14 titles had five to ten uses and constituted 20% of overall use. both high-use categories were almost entirely comprised of pharmacy materials. pharmacy titles (lc classification rm and rs) represented 24.4% of the collection but 59% of usage. titles with one to four uses accounted for 23% of titles overall, and those with no uses for 67%.   usage peaked in the latter half of the fall and winter terms, when papers were due and examinations were pending, with two pharmacy titles dominating.   implementation   overall the evidence indicated that levels of use did not warrant the maintenance of a separate reference collection. all the reference titles were transferred to circulating stacks, the majority with a standard circulation period. six high use titles and eleven in the five to ten uses category were transferred but with a shorter (four day) loan period; all but three were pharmacy titles.  (this selection was based on use, on the pharmacy librarian’s knowledge of titles important in the pharmacy curriculum, and the availability or lack of electronic versions). notices were placed in the reference room indicating that the books had been integrated into the regular collection. a few instructors who list reference titles with call numbers on student assignments were informed of the change so that assignments could be updated.   outcome   circulation statistics for the period february-september 2013 indicate moderate circulation of short loan titles and low or no use of the majority of regular loan titles.  however, this does not reflect in-house use, which would require a separate survey. virtually no feedback from clients has been provided at the service desk.   the reference room is now used for quiet study and as a location for receptions for special events.  several classes have been held in the area and after the installation of glass doors at the entrance, it will be even more usable as a teaching space, a facility otherwise lacking in the library.   reflection   the survey, the analysis of the results, and the implementation of the evidence all proved to be a straightforward process.  the shelving supervisor was helpful in instructing student shelving staff in the procedure, and in ensuring that the new student shelving staff for each academic term were informed. a library associate staff member collated the statistics and wrote the summary report. another associate staff member changed the location and loan status of each title in the catalogue, removed reference stickers, and applied stickers indicating short loan as appropriate.  re-shelving was done on a daily basis by the student shelving staff over a period of three weeks, as part of normal operations. although a formal evaluation was not conducted, no negative comments were received by subject librarians or by service desk staff. given that feedback received from health sciences library clientele usually expresses a preference for electronic content, an assessment of the change was not deemed necessary.   the evidence obtained from this survey provided a sound basis for elimination of the reference collection, for physical re-location of the books, and for re-purposing of the reference room. it achieved the goals of providing optimum access to the library’s print resources and maintaining their relevancy.    evidence summary   undergraduate use of library databases decreases as level of study progresses   a review of: mbabu, l.g., bertram, a. b., & varnum, k. (2013). patterns of undergraduates’ use of scholarly databases in a large research university. journal of academic librarianship, 39(2), 189-193. http://dx.doi.org/10.10.1016/j.acalib.2012.10.004   reviewed by: kimberly miller research & instruction librarian for emerging technologies albert s. cook library towson university towson, maryland, united states of america email: kimberlymiller@towson.edu   received: 11 jun. 2014   accepted: 6 aug. 2014      2014 miller. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to investigate undergraduate students’ patterns of electronic database use to discover whether database use increases as undergraduate students progress into later stages of study with increasingly sophisticated information needs and demands.   design – user database authentication log analysis.   setting – a large research university in the midwestern united states of america.   subjects – a total of 26,208 undergraduate students enrolled during the fall 2009 academic semester.   methods – the researchers obtained logs of user-authenticated activity from the university’s databases. logged data for each user included: the user’s action and details of that action (including database searches), the time of action, the user’s relationship to the university, the individual school in which the user was enrolled, and the user’s class standing. the data were analyzed to determine which proportion of undergraduate students accessed the library’s electronic databases. the study reports that the logged data accounted for 61% of all database activity, and the authors suggest the other 39% of use is likely from “non-undergraduate members of the research community within the [university’s] campus ip range” (192).   main results – the study found that 10,897 (42%) of the subject population of undergraduate students accessed the library’s electronic databases. the study also compared database access by class standing, and found that freshman undergraduates had the highest proportion of database use, with 56% of enrolled freshman accessing the library’s databases. sophomores had the second highest proportion of students accessing the databases at 40%; juniors and seniors had the lowest percentage of use, with 38% of enrolled students at each level accessing the library’s databases. the study also found that november was the peak of database search activity, accounting for 37% of database searches for the fall 2009 semester. database use varied by the schools or colleges in which students were enrolled, with the school of nursing having the highest percentage of enrolled undergraduates using library databases (54%). the authors also report that the college of literature, science, and the arts had the fourth highest proportion of users at 46%, representing 7,523 unique students, more than double the combined number of undergraduate users from all other programs. since the college of literature, science, and the arts accounts for more than 60% of the total undergraduate enrollment, the authors suggest that information literacy instruction targeted to these programs would have the greatest campus-wide impact.   conclusion – although the library conducts a number of library instruction sessions with freshman students each fall semester, the authors conclude that database use patterns suggest that the proportion of students who continue to use library databases decreases as level of study progresses. this finding does not support the study’s hypothesis that database use increases as students advance through their undergraduate studies.     commentary   with academic library expenditures for electronic database and journal subscriptions continuing to rise, combined with students’ reported preference for internet-based resources, investigating use of electronic library resources remains important for academic libraries of all sizes and classifications. previous research suggests that electronic resource use is positively associated with higher student grade point averages and student retention (davidson, rollins, & cherry, 2013; haddow, 2013; soria, fransen, & nackerud, 2013). the current study is an informative addition to the existing literature in its attempt to understand whether undergraduate students’ rate of access to scholarly resources through library databases changes as they advance through their academic career.   utilizing glynn’s (2006) critical appraisal checklist, the study’s strengths lie in choice of population, data collection method, and study design. unobtrusively gathering authenticated access logs throughout an entire semester allows for collecting data on the majority of users who accessed electronic resources, reducing sampling bias and providing a robust sample size. measuring actual database use, rather than relying on self-report methods, lends support to the reliability of the findings. students do have the option to opt out of logged activity tracking at the university, but it is unclear how many students may opt out. additionally, although the authors suggest that the reported unauthenticated use is likely from non-undergraduate researchers, it is unclear how much error this may introduce into the study. for instance, if an undergraduate student accessed resources via one of the library’s non-authenticated workstations, their search activity would be lost in the unauthenticated data.   the article’s literature review places the study within the context of information literacy in the curriculum and librarian-faculty collaboration. since the study’s aim and findings highlight patterns of undergraduates’ resource use, a more thorough review of current literature tying undergraduates’ electronic resource use to information literacy instruction would help elucidate the connection between information literacy instruction and study findings. although the introduction includes a discussion of students’ online search preferences, the cited literature is somewhat outdated and is not integrated with the information literacy instruction material.   in discussing their findings, the authors suggest that the momentum of first-year students’ information literacy seminars does not continue throughout advanced years of undergraduate study. it is unclear from the article how much information literacy instruction is provided in upper division courses or whether upper-level students in the current study would have experienced the same type or amount of library instruction during their freshman years. in addition to the current study’s one semester snapshot, future longitudinal research tracking change over time with cohorts of students throughout their undergraduate career may provide additional support for the study’s conclusions.   the study’s findings are useful for academic instruction librarians attempting to integrate information literacy throughout the undergraduate curriculum. although many institutions focus their efforts on library instruction during an undergraduate’s first year of study, students’ information needs are also expected to require more interaction with library resources as they engage in advanced disciplinary study. the study’s findings suggest that upper division students may not continue to engage with scholarly sources as hypothesized, implying that freshman interventions alone do not promote long-term growth and resource usage. academic librarians should continue identifying strategic places within the disciplinary curriculum to target on-going information literacy development beyond the first year, encouraging students to utilize resources necessary for deep engagement within an academic discipline.     references   davidson, k. s., rollins, s., & cherry, e. (2013). demonstrating our value: tying use of electronic resources to academic success. serials librarian, 65(1), 74-79. http://dx.doi.org/10.1080/0361526x.2013.800630   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24, (3). 387-399. http://dx.doi.org/10.1108/07378830610692154   haddow, g. (2013). academic library use and student retention: a quantitative analysis. library & information science research, 35(2), 127-136. http://dx.doi.org/10.1016/j.lisr.2012.12.002   soria, k. m., fransen, j., & nackerud, s. (2013). library use and undergraduate student outcomes: new evidence for students’ retention and academic success. portal: libraries and the academy, 13(2), 147-164. http://dx.doi.org/10.1353/pla.2013.0010 transforming academic libraries into information commons: a proposed model research article   transforming academic libraries into information commons: a proposed model   kavita chaddha assistant librarian, indian institute of management lucknow noida campus noida, delhi national capital region, india phd scholar, indira gandhi national open university (ignou) maidan garhi, new delhi, india email: kavita@iiml.ac.in   uma kanjilal faculty of library & information science, school of social sciences pro vice-chancellor indira gandhi national open university (ignou) maidan garhi, new delhi, india email: ukanjilal@ignou.ac.in   received: 1 july 2021                                                                 accepted: 25 oct. 2021      2022 chaddha and kanjilal. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30004     abstract   objective – the main objective was to create an information commons (ic) model for the existing library with minimum structural changes to achieve maximum benefit. the subdivisions of the main objective were:   •                 to find out students' expectations and perceptions of an ideal learning environment; •                 to find out the factors which influence the satisfaction level of the students for the library; •                 to find out how satisfied are students from the existing library; and •                 to find out the current library usage pattern of the students.   methods – based on the available literature on the topic, an online questionnaire survey was constructed with google forms and sent to current cohorts studying at the institute through e-mail, along with the study's rationale and a request for participation. we contacted 294 students, of which 199 responded. the data were analyzed and presented using microsoft excel.   results – the findings of the study showed the keen interest of the students in library resources and services. it also showed that the students were not fully satisfied with the current library space and working hours. they wanted enhanced quiet areas and collaborative spaces where information experts help them use the current technology to improve their learning experience. based on the gathered data analysis, an ic model for redesigning the existing library has been recommended.   conclusion – the present study was the first step in research on ics in the indian context. this pilot study captured the perception and expectations of all levels of students: postgraduate, working executives, and senior-level executives. most of the suggestions have been incorporated into the plan. with very few construction changes and new furniture, this model can be easily implemented in a small academic library without discarding the old furniture.     introduction   the advancement of information technology and increased internet use have significantly changed academic libraries' nature and students' information behaviour. the academic library has become an extension of the classroom, where students learn by collaborating. the students use the library not only for seeking information but also to socialize intellectually. electronic resources have significantly increased in recent years, but faculty members continue to use both printed material and electronic resources. the library has become the centre of the intellectual community in an organization.   to make libraries more user-friendly, librarians in the united states (u.s.) conducted surveys on students' needs and expectations from libraries in their colleges and universities. after analyzing the surveys on users' preferences for library spaces and services, they could redesign their libraries. the users wanted a place to study using desktop and laptop computers with a wireless internet connection. they needed a place for collaborative and group study with a lab for using multimedia and myriad software. they required a quiet area for serious research work. they wanted a seminar room with a projector and screen for lectures and presentations. they needed a cafe and a lounge for light refreshments and socializing. in a nutshell, the users perceived the library a place away from home and classroom – as the third place: an information commons (ic).   according to loertscher and koachlin (2014), traditional libraries are resource-oriented, whereas ics are learning-oriented. an ic aims to enhance student, faculty, and researcher learning, teaching, and research experience. the library’s physical space (the first level) with new arrangements, technologies, and services is the prominent feature to organize workspace and service delivery. it is a collaborative physical and virtual environment that invites and ignites participatory learning (loertscher & koachlin, 2014).   an ic is a virtual space at the second level, where students can search, access, and retrieve relevant information instantly for their studies and project work, and connect with other students, researchers, and faculty working in the same area. according to beagle (2006): "a virtual commons is a pervasive online environment in which a wide variety of electronic resources and services can be accessed through a single graphical user interface (gui) and potentially searched in parallel with a single search engine from any networked workstation" (p. 4).   at the third level, ic becomes a cultural commons. at this third level are the social and cultural arenas of free speech, shared knowledge, and creative expression in the digital age, as in the surrounding envelope of laws, regulations, commercial practices, and popular traditions (beagle, 2006).   many universities and colleges in the u.s. and europe have converted their libraries into ics, and the students and faculty are using the ic for information sharing and knowledge creation. other countries, such as australia, china, japan, and new zealand, have also converted college and university libraries into ics.    the indian institute of management lucknow   the indian institute of management lucknow (iim lucknow) is a leading management school in india, established by india's government as one of the four centres of excellence in business management education. the first one was iim calcutta, which opened in 1961 in the east, the second was iim ahmedabad, which opened in 1961 in the west. the third, iim bangalore, opened in 1973 in the south, and the fourth, iim lucknow, opened in 1984 in the north. at present, there are 20 iims all over the country.   iim lucknow, spread over 200 acres of land, is beautifully surrounded by lush greenery and artistic landscaping. three regular programs—post graduate program (pgp), post graduate program in agri-business management (pgp-abm) and fellow program in management (fpm)—run from lucknow campus along with management development programs (mdp).   figure 1 iim lucknow campus.   the iim lucknow library, gyanodaya, has a two-story building spread in 30,000 square feet. it holds 60,000 learning resources in different formats in management and related subjects.   figure 2 the iim lucknow library.   in 2005, iim lucknow opened another campus at noida (iiml noida) to provide higher education to management executives who already work in the industry and wish to upgrade themselves. the major programs run by iiml noida are a two-year working managers' program (wmp), one-year international program for management executives (ipmx), two-year post graduate program in sustainable management (pgpsm), and four-year executive fellow program in management (efpm). various management development programs for corporates are also conducted throughout the year. the iiml noida campus is spread over 20 acres of land (1/10th of the area of the lucknow campus).   figure 3 the iiml noida campus.   the library is situated in the right wing of the second floor of the administrative block, with a floor space of 1,000 square meters. the library's learning resources include a print collection of around 6,500 books, 1,500 bound journals, 30 current journals, and electronic resources of around 500 e-databases, 1,500 e-journals, and 1,500 e-books. the library portal, gyanoday, is accessible through the intranet.   figure 4 the iiml noida library.   statement of the problem   the libraries of higher education institutes in india are in a transition period. the decreased use of print resources and increased demand for online resources have compelled the librarians to adjust budgets accordingly. the library's declining footfall has impacted the justification for the size of the library workforce. the emphasis is now on skills: technology, media, and research. the librarians are trying to cope with the changes by upgrading their skills. still, it is challenging to convince the administration to spend money on redesigning and revamping the libraries, let alone asking for a new ic building. even the libraries' role in the students' overall learning and performance is not researched and documented correctly. without accurate research data, librarians cannot justify the demand for skilled staff, the latest technology-based equipment, and library building or space renovation. in short, we can say that indian libraries are facing the same problems which libraries in western countries faced a couple of decades ago.   the present case study attempted to determine whether the needs, expectations, and perceptions of indian students of higher education are different from or the same as their counterparts in other countries of the world. based on the results, the authors have tried to convert the existing library space into a model of an ic, suitable for a small/medium academic library, that is low cost, requires minor structural modifications, and is in sync with the users' needs and expectations.   scope and limitations of the study   we limited the study to only one institute because it is a pilot study. the authors have created a plan for space utilization of a small academic library based on the results. other small academic libraries can utilize the findings of the research. librarians can conduct similar studies for their institutes and make a stronger case with accurate research data for repurposing and reconstructing their institute libraries as ics.   literature review   a detailed literature search was done to understand the concept of ic, what models are available, how they are planned and implemented in various institutes and universities, assessment techniques, the benefits of introducing ic in colleges and universities, and various case studies conducted by librarians. the concept of ic came into existence in the mid-1990s. it was the time when librarians started using ic to improve library services. the rationale behind ic was to enhance student learning and scholarship. however, there was a need to create a rapport between the librarians and students and faculty.   beagle (1999) has done significant work in the field of ic, developing the concept of ic at three levels: physical, virtual, and cultural (2006). most colleges and universities in the u.s. have adopted beagle’s model in designing and developing their libraries.   bailey and tierney (2008) further developed the concept of ics and learning commons (lcs) at four levels. the first two levels are ics and the next two levels are lcs.    macwhinnie (2003) called the library the third place for students, with the first place being home and the second place being the classroom. students spent almost one-third of their time in libraries (macwhinnie, 2003). therefore, libraries should be inviting and comfortable.   mcmullen (2008) emphasized analyzing students' needs before planning an ic on a campus. the library's space utilization, the technology used, and the library staff required in the ic should accord with the students' needs. libraries work for improvement and to support the learning outcomes of the students. therefore, their opinions matter the most.   sinclair (2009) discussed the changing role of academic librarians due to increased technology use in libraries. this new role is focused not only on overall expertise, guidance, and instructions to students in using both offline and online information, but also on the institution's overall goals. the librarians must keep upgrading their skills according to the demands of the time.   massis (2010) emphasized that libraries' new role has become a one-stop-shop for campus teaching and learning. the ic is full of students working in groups, discussing topics, and preparing projects. compared to traditional libraries, the renovated or reconstructed ics match the current teaching pedagogy, and the students find everything they need under one roof.   peterson (2013) conducted a case study on the central library at a midwestern u.s. university to determine its students’ current usage behaviour, preferences, and expectations from the library. results showed that students know what they want from a modern library. they still needed the quiet spaces provided in the traditional libraries, and they required group study spaces separately (peterson, 2013). this case study affirmed that students required both individual and group study spaces.   dryden and goldstein (2013) discussed assessment models which may help evaluate lcs. they used various assessment methods like technology surveys, space assessment surveys, and focus groups. the assessment of ics validated the need to spend time, effort, and money on library infrastructure and provided cost-benefit analysis data.   woo et al. (2019) wrote about the impact of lcs on the students' learning behaviour and skills development. their study was significant as it measured the impact of using an lc on students’ learning and skills development, which is the primary goal of creating an ic.   the concept of ic is new to india. no significant research has been conducted, published, or documented related to ic for indian libraries. few practising librarians have written papers on either ics or lcs. in one study, singh (2019) compared the infrastructure, facilities, and services of banaras hindu university, located at varanasi, uttar pradesh, india, with an lc. the handful of articles on lcs presented in conferences and seminars in india are based on research conducted in western countries and not on locally conducted research.   there are many differences between indian and western countries' socio-cultural values, economic conditions, and educational systems. in the west, institutes of higher learning for students are well maintained with good infrastructure. even in the most ordinary universities, the library forms the centre of life for teachers and students. librarians enjoy a high status as their contribution to academic life cuts across academic disciplines. they work closely with teachers and students in the various tasks involved in procuring books and journals, keeping the library quiet and friendly, and ensuring speedy access.   the indian case is the opposite. the library exists on the margins of the classroom. in many universities, undergraduate students are not allowed to use the university library. subscriptions to journals and magazines have dwindled over the years, and maintaining past volumes is now seen as an ancient practice because e-storage is available. the reading rooms carry an unkempt, hapless look, with noisy ceiling fans and worn-out books waiting to be removed. the faculty members also do not trust the librarians to help in teaching, learning, or research activities. there is always a lack of funds to spend on the library’s infrastructure and skilled staff. with the indifferent attitude of the administration, librarians also do not take the initiative to improve their libraries. that is why there is a need to conduct and publish research-based studies in the indian context.   aims   in this study, our main aim was to create an ic model for an existing library with minimal structural changes to achieve maximum benefit. the subdivisions of the main aim were:   ·        to find out students' expectations and perceptions of an ideal learning environment; ·        to find out the factors which influence the satisfaction level of the students for the library; ·        to find out how satisfied are students from the existing library; and ·        to find out the current library usage pattern of the students.   methods   the population   the post graduate course (pgpsm), executive course (ipmx), and working managers course (wmp) students have comprised the target population for this pilot study.   research instrument   a structured online questionnaire was constructed using google forms to collect primary data (appendix 1). it was sent to current cohorts studying at the institute through e-mail. two hundred ninety-four students were contacted, of which one hundred ninety-nine responded. likert's 5-point rating scale was used in most of the questions. the data were then analyzed and presented using microsoft excel.   results, analysis, and discussion   the questionnaire had 17 questions. the first two questions were related to demographic data. question numbers 3 to 7 were related to students' library usage patterns. question numbers 8 to 10 were related to students' opinions about the quality of library services. the next four questions (11-14) captured the students' preferences for various library space features and furniture. the subsequent two questions (15-16) captured satisfaction and desirability levels for various library components. the last question was an open-ended question inviting suggestions to improve the library services or experience.   the captured data showed that the student needs and expectations differed course-wise. the ipmx, being a one-year course, was compact, and after attending classes for the whole day, the students used the library during the evening and night. the other course, wmp (two years' program), had working executives who resided on the campus for three days (friday evening to sunday evening), twice a month. they used the library only six days per month. all the ipmx and wmp students owned laptops, so they used the library for individual and group study. another course, pgpsm, was a two-year residential course, where students were fresher, comparatively younger, and used the computers in the lab for accessing e-resources.   demographic data   a total of 199 students, out of 294 contacted, responded to the survey, or 67.69% of the total sample. the course-wise participation (in percentage) is given in figure 5.   figure 5 course representation.   the percentage of women candidates opting for management courses has increased over the years in india. in this study, the total percentage of men was 73.37% and women was 26.63% combined for all courses (see figure 6).   figure 6 gender.   library usage pattern   the data revealed that most of the students (58.79%) came to the library only when required; 18.59% came several times a month; 11.06% several times a week; 10.55% several times per semester; and only 1.01% came to the library daily.   however, all students used e-resources almost every day. the students did not sit and study inside the library, showing that the library space was not utilized correctly and was more like a traditional library. these data also revealed that the circulation of print books has not been reduced, and the acquisition, technical, and circulation processes have not been impacted. the entry register kept at the entrance showed that the user footfall has decreased over the years. only 3-4 students entered the library daily, largely for the issue or return of books. during examination times, however, the number of users increased.   in response to the question regarding the time students spent in the library, the data showed that many students (34.67%) spent 1-3 hours, followed by 29.65% who spent less than 30 minutes, 23.62% spent 30 minutes to 1 hour, 11.06% spent 3-6 hours, and 1.01% spent more than 6 hours in the library.   the inference revealed from these data was that most students (64.32%) spent 1 to 3 hours or less than 30 minutes inside the library whenever they used it.   most of the students (67%) preferred using the library from 6:00 p.m. to 12:00 a.m. the main reason was that the students have classes from 9:00 a.m. to 5:30 p.m. the best time was the evening time to study for them. the course-specific details showed that 78% of ipmx students, 66% of wmp 2nd year students, 63% of wmp 1st year students, 54% of pgpsm 1st year students, and 48% of pgpsm 2nd year students fell into this category. the next most popular time slot—from 12:00 p.m. to 6:00 p.m.—was used by 20% of students. these data indicated that the students' most preferred time to visit the library (87%) is a 12 hour window between 12:00 p.m. to 12:00 a.m.   figure 7 frequency of visiting the library.   figure 8 time spent in the library.   figure 9 preferred time.   the main reasons students used the library were identified by analyzing the case studies previously conducted on this topic, and the answers for the survey question: "why do you go to the library?" the students could choose as many appropriate options as applicable.   the data showed that most students go to the library to take print-outs (i.e., print documents), followed by   ·        search information for assignments, ·        read for the test or exam, ·        study alone, ·        issue or return books, ·        use a computer for studies, ·        read print journals or magazines, ·        study in a group, and ·        socialize with other students.   a few other reasons are to read newspapers and for peace of mind.   figure 10 reasons for visiting the library.   legends (reasons):   a.    search information for my assignment b.    read for test or exam c.    study alone d.    study with a group e.    use a computer for studies f.     issue/return books g.    read print journals/magazines h.    take print-outs (i.e., print documents) i.     socialize   table 1 reasons for visiting the library sr. no. reason percentage sr. no. reason percentage 1. take print-outs 70.85% 6. use a computer for studies 42.71% 2. search information for my assignment 62.31% 7. read print journals /magazines 38.19% 3. read for test or exam 55.79% 8. study with a group 26.63% 4. study alone 49.75% 9. socialize 4.02% 5. issue/return books 43.22%         in the survey, the students identified their preferred section of the library, choosing between the library's three sections based on the functionality: the main library reading area, the informal reading area, and the computer lab. the data showed that the informal reading area was preferred by the most students (39.70%), followed by the computer lab (33.17%), and the main library reading area (27.14%). the informal reading area and the computer lab are open 24/7. that was why most of the students preferred to use those two areas. the second reason was that they can use this area for group study or discussions. alternatively, this area can be used for silent study with their laptop and reading materials. however, problems may arise when different students want to use the informal reading area for both types of activities simultaneously.   figure 11 preferred section.   the students were asked to rate the speed and quality of wi-fi connectivity in the library using likert's five-point scale. the most students (34.67%) rated it "good", followed by "very good" (31.16%), "fair" (16.58%), "excellent" (10.05%), and "poor" (7.54%). these results indicates that there is still room for improvement in wi-fi connectivity. the users would prefer to study in the library if the internet speed and quality were much better than other areas on the campus.   figure 12 speed and quality of wi-fi.   figure 13 web portal, iim lucknow library.   the library portal (see figure 13), called “gyanoday” (meaning enlightenment) is accessible through the intranet at both campuses and via remote access through open athens. all the e-resources are available there. the data collected about the usability and accessibility of the library portal show that the most students (42.71%) have rated it "very good," followed by "good" (33.17%), "excellent" (10.05%), "fair" (9.55%), and "poor" (4.52%). the data indicate that the portal should be more user friendly and provide more information related to library resources and services.   figure 14 usability and accessibility of library portal.   students were asked to rate the quality of the library based on various physical attributes on a scale from one to five: the library's size, natural light, light, air conditioning, noise level, and ambience. the data showed that the students were overall satisfied with the artificial lights, natural light, air conditioning, and somewhat with the library’s ambience. however, they were not happy with the library's size or the noise level in the library. the author has tried to overcome these two shortcomings in the proposed model plan by creating research carrels for silent study and soundproof rooms for group study.   figure 15 quality of library.   students' preference in library space and furniture   the students were asked to rate the library's attributes, which, if included in the library, may influence their learning experience. the data showed that the most crucial feature, which was "extremely influential" in students' learning experience, was the level of sound in the library (42.21%), followed by comfortable furniture (38.19%). the next feature, which was "very influential," was artificial lights (41.71%), followed by air conditioning (41.21%), ambience (40.20%), and natural light (39.70%). the least influential features were colourful walls (15.08%) and café/food joints (15.58%).   figure 16 impact on learning.   students liked the idea of keeping the print materials in the library, but moving them to compact shelving; 39.20% of students found it "likely" that by doing this, the space of the library would be better utilized. other students were "neutral" (28.64), found it "extremely likely" (20.60%), "unlikely" (7.54%), or "extremely unlikely" (4.02%). the proposed model has the provision of compact shelving in the left wing.   figure 17 compact shelving.   while collecting data to find out students' desire to include social spaces in the library, we found that the most students (36.68%) considered it "very desirable" to have social spaces, followed by 28.64% considering it "desirable," 16.58% listing their feelings as "neutral," 11.06% considering it "very undesirable," and 7.04% considering it "undesirable." it is interesting to note that 41.46% of ipmx students and 41.38% of wmp 2nd year students rated social space in the library as “very desirable.” these students were senior and middle-level executives, respectively. they were interested in knowing and discussing various social topics in a comfortable and stimulating environment with their peer group.   figure 18 social space.   when we asked students their reasons for visiting the library, only 1% of students responded to socialize, but while asking about the desirability for social space in the library, almost 37% rated it as very desirable, and 29% rated it as desirable. it showed that if the library provided social space on its premises, 66% of students would prefer to use it.   the next set of data were about the type of furniture preferred by the students. most of the students rated all the features included in the survey as "desirable," particularly large index tables with 4-6 chairs, sofas with coffee tables, small round tables with 2-3 chairs, and foldable and movable tables and chairs.   figure 19 choice of furniture.   satisfaction with the library   the students were asked to rate their satisfaction with various library services and operations on a five-point scale. most students were "satisfied" with study spaces, computers, printing facilities, library services, and library orientation. however, they were "least satisfied" with the library’s hours of operation. if there were a sufficient number of staff members, the library could be opened for more time. however, the reading area and the computer centre is open 24/7.   figure 20 satisfaction level.   desirable components   the three “most desirable” components students would want included in an ideal learning environment were: "silent study area/research carrels" (50.25%), “scanners" (44.22%), and "color printers" (44.22%). figure 16 and table 2 describe the percentage distribution of each component on a scale of five. the “least desirable” component was a 3d printer with software (12.56%).   figure 21 desirable components.   table 2 desirable components   components very desirable desirable neutral undesirable very undesirable 1 café/ refreshment joint 32.66% 26.13% 20.10% 13.07% 8.04% 2 video conferencing room 27.64% 25.13% 25.13% 15.58% 6.53% 3 flexible spaces/movable furniture 25.13% 31.16% 25.63% 12.06% 6.03% 4 silent study area/study carrels 50.25% 29.15% 15.08% 3.52% 2.01% 5 group study zone 40.70% 31.66% 18.09% 5.53% 4.02% 6 e-learning classrooms 33.67% 32.66% 23.62% 8.04% 2.01% 7 smartboards 34.17% 32.16% 23.12% 7.54% 3.02% 8 big screens 26.13% 26.13% 29.15% 13.57% 5.03% 9 small seminar room 37.69% 32.16% 19.10% 8.04% 3.02% 10 video making/media production 27.64% 32.66% 27.64% 8.54% 3.52% 11 3d printer with software 24.12% 24.12% 25.15% 14.07% 12.56% 12 library website 42.21% 30.65% 18.09% 5.53% 3.52% 13 mobile app for the library 43.22% 25.63% 20.60% 6.03% 4.52% 14 scanners 44.22% 28.64% 17.09% 7.04% 3.02% 15 color printers 44.22% 27.14% 15.58% 8.54% 4.52% 16 makerspace 31.66% 28.14% 30.15% 8.04% 2.01%   suggestions for improving library facilities   students were asked to provide suggestions for making the library an ideal learning environment. for this purpose, an open-ended question was included in the survey. a few key points, selected from the responses, are listed below:   ·        24/7 functional library, full of print, digital and online resources, with the most suitable internet speed and comfortable furniture. ·        the perfect ambience would be with natural light and other furniture/décor. ·        a quiet study space, along with separate group discussion/presentation practice spaces, is required. ·        self-checkout/check-in kiosk. ·        a cozy place with air-conditioned rooms and lots of reading materials and journals. ·        an informal cubicle type place for group study with sofa and table is a must. ·        common area with bean bags and positive experience. ·        cafe serving tea and coffee. ·        an audio-visual room would be welcome to watch videos. ·        a perfect library must be a quiet space where all reading material is at hand. audio and video books are to be streamed on wi-fi so that students can use their headphones to analyze the topic better.   research findings   the research findings showed that students were not frequent visitors to the library. they used it only when required. the average usage was for 1-3 hours. most of them used it between 6:00 p.m. to 12:00 a.m. the two primary reasons for visiting the library were taking printouts and searching for information for assignments.   the students were satisfied with the wi-fi speed and quality and the library portal's usability and accessibility. most of them rated the library as "very good" based on various physical attributes like ambience, air conditioning, noise level, lights, natural light, and library size. most of the students were not satisfied with the hours of operation of the library. however, they were satisfied with the study space, library instruction/orientation session about library resources, library services, and computer workstations in the lab and printing facilities. they were unsure about the library's range of learning resources (books/journals/media).   natural light, artificial lights, sound, air conditioning, ambience, comfortable furniture, colourful walls, and cafe/food joint are all factors that would enhance the students' learning experience. most students favoured keeping the print books and bound journals in compact shelving as the library space would be better utilized.   the students preferred large index tables with 4-6 chairs over small round tables with 2-3 chairs, sofas with coffee tables, and foldable and movable tables and chairs. most of the students wanted to have social spaces in the library. being a small campus, they needed a centrally located place, open 24/7, that was adequately lit, full of ambience, and comfortable furniture.   the choices of the students for the features to be included in the ic (in order of preference) were as follows:   table 3 preferred features s. no. ic features incorporated in plan remarks 1 silent study area/study carrels yes   2 scanners yes   3 colour printers yes   4 mobile app for the library no the author published research on the topic in 2016. yet to be implemented 5 library website yes though it needs improvement 6 group study zone yes   7 small seminar room yes   8 smart boards yes   9 e-learning classrooms yes   10 café/ refreshment joint yes   11 makerspace no not chosen by the students 12 video making/media production no no space is left for this facility 13 video conferencing room yes   14 big screens yes   15 flexible spaces /movable furniture yes   16 3d printer with software no not chosen by the students   the existing plan of the library   right wing   the library is situated on the second floor of the administration block. it has two wings. the library operates from the right-wing. the existing library has a stack area, print journals display, reading area, and bound volumes in one portion. this portion is open from 9:00 a.m. to 6:30 p.m. the other has a computer lab and an informal reading area. this portion is open 24/7. a glass partition divides the library into two parts (refer to figure 17 in appendix b).   left wing   the left-wing is currently not used for the library. though the whole second floor was constructed for the library, the institute's left-wing is being used for student examination purposes. while creating a model for ic, the author has used both wings. with minimal changes in the existing infrastructure, maximal components have been incorporated to convert the library into an ideal learning place for students (refer to figure 19 in appendix b).   connecting lobby   the lobby area that connects the left wing to the right wing is currently not being used. there is a room, which the institution's visiting doctor is using. two makeshift rooms have been constructed recently for the computer centre (cc) manager and cc staff at the lobby entrance (refer to figure 21 in appendix b).   model plan of ic   based on the data analysis, a model plan for the existing library has been designed, which is the study's primary aim. the research findings have been incorporated into the model plan of the library. the plan has been designed so that with minimum expenses, maximum improvement can be achieved.   the students used the library mainly to take printouts as printers were available only in the library’s computer lab. likewise, books were available only in the library, so students had to visit the library for issuing and returning books. if its services and spaces were enhanced according to the students’ requirements, they could utilize the library’s full potential. at the same time, using the library space and resources for a better learning experience would improve learning outcomes.   right wing   based on the data analyzed, the right wing has been remodelled accordingly. the bound volumes and book stacks have been shifted to the left wing, and the area thus vacated is now used for video conferencing, a small seminar room (as desired by 36.7% of students), and research carrels (as desired by 46.6% of students).   four group study rooms (as desired by 37.9% of students) and the cc manager’s office have been placed in this wing. these rooms will be made soundproof.   the reading tables have been shifted to the left wing. sofa sets, chairs, and bean bags have been moved to the left wing. workstations in the computer lab have been rearranged. this arrangement will allow the lab to be used as an e-classroom also, as desired by 35% of students. a smart tv with a large monitor has also been added to the lab for the same purpose (refer to figure 18 in appendix b).   left wing   the left wing has been planned as a quiet zone. the books have been kept in compact shelving (as desired by 38.6% of students) to utilize the space for seating purposes. the bound volumes of journals are also kept on compact shelves. the acquisition room has been shifted to the left wing. a storeroom is also in this wing. current periodicals, magazines, and newspapers have been moved from the right to left wing. research carrels (as desired by 46.6% of students) for quiet reading have been planned for this wing also. sofa sets, comfortable chairs, and bean bags have been put in proper places (refer to figure 20 in appendix b).   connecting lobby   the lobby area was initially earmarked for the circulation service of the library. the model plan has the same purpose for space. a circulation desk has been proposed to be built on the opposite side of the entrance. the option for self-check-in and check-out of library documents is also there. the room, which the visiting doctor currently uses, has been proposed to be the librarian's room. the cc manager and staff have been shifted to the right wing. their offices have been repurposed into a photocopy/scanner/printer room and locker-room, respectively. tea and coffee vending machines (as desired by 33.9% of students) and snack machines have been placed in the lobby area. some bean bags and chairs have been put in this area (refer to figure 22 in appendix b).   conclusion   a few top indian higher education institutes are currently taking library infrastructure transformations seriously. these institutes have recently renovated their libraries with the help of corporate social responsibility (csr) funds from various companies. still, it was a long journey before dedicated ics for indian colleges and universities were the main focus.   the main obstacle in front of librarians is to convince stakeholders that they play an important role in achieving the institute's learning, teaching, and research goals. to overcome this obstacle, they must have research-based data. the available data is based on international research. as said earlier, there is much difference between indian and western countries' socio-cultural values, economic conditions, and educational systems.   the present study is the first step in research on ic in the indian context. this pilot study has captured the perception and expectations of all levels of students – postgraduate, working executives, and senior-level executives. most of the suggestions gathered through the survey have been incorporated into the library’s space plan. with very few construction changes and new furniture, this model can be easily implemented in a small academic library without discarding the old furniture.   author contributions   kavita chaddha: conceptualization, methodology, analysis, interpretation, visualization, software, writing uma kanjilal: guidance, review, supervision   references   bailey, d. r., & tierney, b. g. (2008). transforming library service through information commons: case studies for the digital age. chicago: american library association.   beagle, d. (1999). conceptualizing an information commons. the journal of academic librarianship, 25(2), 82-89. https://doi.org/10.1016/s0099-1333(99)80003-2   beagle, d. r. (2006). the information commons handbook. new york: neal-schuman.   dryden, n. h., & goldstein, s. (2013). regional campus learning commons: assessing to meet students needs. journal of library administration, 53(5-6), 293-322. https://doi.org/10.1080/01930826.2013.876822 loertscher, d. v., & koechlin, c. (2014). climbing to excellence: defining characteristics of successful learning commons. knowledge quest, 42(4), 14-15.   macwhinnie, l. a. (2003). the information commons: the academic library of the future. portal: libraries and the academy, 3(2), 241-257. https://doi.org/10.1353/pla.2003.0040   massis, b. e. (2010). the academic library becomes the academic learning commons. new library world, 111(3/4), 161-163. https://doi.org/10.1108/03074801011027664   mcmullen, s. (2008). us academic libraries: today's learning commons model. peb exchange, 2008(4), 1-6. https://www.oecd-ilibrary.org/content/paper/245354858154   peterson, n. k. (2013). the developing role of the university library as a student learning commons: implications to the interior spaces within [master's thesis, iowa state university]. iowa state university digital repository. https://lib.dr.iastate.edu/etd/13186   singh, p. (2019). establishing library learning commons in universities of india: a case study at bhu library system. library philosophy and practice (e-journal). https://digitalcommons.unl.edu/libphilprac/2477/   woo, e. m., serenko, a., & chu, s. k. w. (2019). an exploratory study of the relationship between the use of the learning commons and students' perceived learning outcomes. the journal of academic librarianship, 45(4), 413-419. https://doi.org/10.1016/j.acalib.2019.05.007   appendix a survey questionnaire   1.      name of the course   2.      gender   3.      how often do you visit the library? a.      only when required b.      several times per semester c.      several times a month d.      several times a week e.      daily   4.      how much time do you spend in the library? a.      more than 6 hours b.      3-6 hours c.      1-3 hours d.      30 minutes to 1 hour e.      less than 30 minutes   5.      what time do you prefer to use the library? a.      6.00 a.m. to 12.00 p.m. b.      12.00 p.m. to 6.00 p.m. c.      6.00 p.m. to 12.00 a.m. d.      12.00 a.m. to 6.00 a.m.   6.      why do you visit the library? you may opt for as many as applicable. a.      search information for my assignment b.      read for test or exam c.      study alone d.      study with a group e.      use a computer for studies f.       issue/return books g.      read print journals/magazines h.      take print-outs i.       socialize   7.      which section of the library do you use most and why? a.      main library reading area b.      informal reading area c.      computer lab   8.      please rate the speed and quality of wi-fi connectivity inside the library on a scale of 5, where 1 stands for the "least" and 5 stands for the "most."   9.      please rate the usability and accessibility of the library portal on a scale of 5, where 1 stands for the "least" and 5 stands for the "most." 10.   please rate the quality of reading space in the library based on given features on a scale of 5, where 1 stands for "least" and 5 stands for "most." a.      size of the library area b.      natural light c.      lights d.      noise level e.      air conditioning f.       ambience   11.   please rate the impact of the library's following features on students' learning experience on a scale of 5, where 1 stands for "least" and 5 stands for "most." a.      natural light b.      artificial lights c.      sound d.      air conditioning e.      ambience f.       comfortable furniture g.      colourful walls h.      café/food joint   12.   do you think the library space would be better utilized if the print books/journals were kept in compact shelving? please give your response on a scale of 5, where 1 stands for "least likely," and 5 stands for "most likely."   13.   what kind of furniture do you prefer in the library? please give your response on a scale of 5, where 1 stands for "least desirable" and 5 stands for "most desirable." a.      large index tables with 4-6 chairs b.      small round tables with 2-3 chairs c.      foldable and movable tables and chairs d.      sofas with coffee tables   14.   do you think that the library should have social spaces/lounge/casual settings? please give your response on a scale of 5, where 1 stands for "least required" and 5 stands for "most required."   15.   please rate how satisfied you are with the library services on a scale of 5, where 1 stands for "least satisfied" and 5 stands for "most satisfied." a.      hours of operation b.      study space c.      range of learning resources in the library (books/journals/videos) d.      library instruction/orientation session about library resources e.      computer workstations in the lab f.       printing facility g.      library services, i.e., circulation, reference   16.   please assign numbers (1-5) to each component; our library should have, where 1 stands for "least desirable" and 5 stands for "most desirable."   s. no. feature 1. café/refreshment zone 2. video conferencing room 3. flexible spaces/movable furniture 4. e-learning classrooms 5. group study zones 6. silent study areas/study carrels 7. smartboards 8. big screens 9. small seminar room/s for presentation practice 10. video making/ editing suites/ media production 11. 3d printer with software 12. library website 13. library mobile app 14. scanners 15. colour printers 16. makerspace   17.   please give your suggestions for improving the library to enhance your learning experience.    appendix b   figure 22 existing plan (right wing).   figure 23 proposed plan (right wing).   figure 24 existing plan (left wing).   figure 25 proposed plan (left wing).   figure 26 existing plan (connecting lobby).   figure 27 proposed plan (connecting lobby).   evidence based library and information practice evidence based library and information practice 2010, 5.2  evidence based library and information practice       ebl 101     applicability: what is it? how do you find it?    virginia wilson   shirp coordinator, health sciences library   university of saskatchewan   saskatoon, saskatchewan, canada   email: virginia.wilson@usask.ca    received: 21 apr. 2010          accepted: 05 may 2010      © 2010 wilson. this is an open access article distributed under the terms of the creative  commons‐attribution‐noncommercial‐share alike license 2.5 canada  (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and  reproduction in any medium, provided the original work is properly attributed, not used for commercial  purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.    we’ve come a long way over the past year in  this column. there’s been the formulation of  the well‐built question, the seeking and  finding of evidence in the published literature,  the consideration of conducting research  yourself, and the appraisal of research  evidence. now that you have some valid  evidence, you need to determine its  applicability to the situation at hand.    applicability “relates to the extent to which  the results are likely to impact on practice”  (booth & brice, 2004). whether or not a  particular research study is applicable to your  situation is subjective. what works for one  person’s situation, may not work for another  because “libraries show considerable variation  with regard to environment, context and  institutional values” (booth, 2009).  determining applicability is an essential step  in evidence based practice, and it is important  to note that there are different levels of  applicability to look for.     the evidence you find that is relevant to your  situation will usually be one of four things:  1.  directly applicable  2.  needs to be locally validated (i.e.  replicate the study at the local level)  3.  needs to be adapted (derivation)  4.  improves your understanding of the  situation  (koufogiannakis & crumley, 2004 (1, 2 and 4),  and booth, 2004 (3))    ideally, a piece of research evidence you find  will match all or many of the particulars of  your situation, so you can apply it directly and  move on to evaluating the results. how do  you decide if evidence is directly applicable?  there are several variables to consider:     • user group: does the user group in  the study compare to your user  group?  • timeliness: is the research current  enough? technology research, for  example, should be the most current  111 mailto:virginia.wilson@usask.ca evidence based library and information practice 2010, 5.2  research available, while research on  collections can be somewhat older.  • cost: is it fiscally feasible to apply the  evidence in your current  environment?  • politics: is there resistance to change  in your work environment? will the  new concept be accepted? will an  adversarial relationship be created?  • severity: how critical is it that a  solution to your situation be found,  and quickly?  (koufogiannakis and crumley, 2004)    with a bit of thought, you can determine  whether or not the research evidence you have  found can be directly applied to the decision  you need to make or the problem that needs to  be solved in your practice.     because, at present, the body of evidence for  library and information studies is smaller  than, for example, medicine, finding research  that is directly applicable can be difficult. you  will more likely find evidence that resembles  your situation, but that needs to be replicated  and validated at the local level.   koufogiannakis and crumley state that “when  librarians locally validate existing evidence,  they are building the evidence base” (2004). it  is worthwhile keeping in mind that if you go  the route of validating the evidence you have  found by replicating it at your level, the  greater lis community will benefit if you  write up your efforts and find a way to  disseminate the information. similarly, taking  evidence that illustrates larger issues and  applying it at a local level can result in rich  and useful material with which to work and  can be used as an example by other libraries.    additionally, part of the evidence you find  could be adaptable to your own situation.  booth (2004) calls this derivation. derivation is  possible when “some aspect of the  methodology or perhaps the instrument  (questionnaire or interview schedule) can be  adapted to your own practice, though  different” (booth, 2004). in other words, to  avoid reinventing the wheel you can borrow a  part or parts of a research study to construct  your own. as an added benefit, “deviation  from a reported intervention, tailoring it  according to detailed knowledge of the  library’s clientele, may lead to an enhanced  chance of success” (booth, 2009).    the most common form of evidence found is  evidence that helps to improve a librarian’s  understanding of a situation. while not  directly applicable to what is going on, this  kind of evidence can be applied in ways that  increase knowledge and provide a larger  context. keeping up with issues important to  your practice is a central part of evidence  based practice, and reading widely is a good  habit to develop.    evidence‐based practice for information  professionals: a handbook contains an excellent  chapter on applicability (chapter 10). the book  is currently out of print but a pdf of the pre‐ publication manuscript has recent been made  available at http://ebliptext.pbworks.com/.  also useful is an applicability checklist  compiled by librarians at the university of  newcastle in australia. the checklist contains  questions about user group, timeliness, cost,  politics, and severity that will help to  determine the applicability of research  evidence. you can find the checklist here:  http://www.newcastle.edu.au/service/library/g osford/ebl/toolkit     only you can decide if a piece of research is  applicable to your situation. with these tools,  categorizing the research should be  straightforward, and you will be well on your  way to applying the evidence you have found.  in the next issue, ebl 101 will focus on  evaluating the results of the research  application.      references    booth, a. (2004). what research studies do   practitioners actually find useful?  health information and libraries journal,  21, 197‐200.      112 http://ebliptext.pbworks.com/ http://www.newcastle.edu.au/service/library/gosford/ebl/toolkit http://www.newcastle.edu.au/service/library/gosford/ebl/toolkit evidence based library and information practice 2010, 5.2  booth, a. (2009). eblip five‐point‐zero:   towards a collaborative model of  evidence based practice. health  information and libraries journal, 26,  341‐344.    booth, a., & brice a. (2004). appraising the   evidence. in a. booth & a. brice  (eds.), evidence‐based practice for  information professionals: a handbook  (pp. 104 ‐ 118). london: facet.    koufogiannakis, d., & crumley, e. (2004).   applying evidence to your everyday  practice. in a. booth & a. brice (eds.)  evidence‐based practice for information  professionals: a handbook (pp. 119‐126).  london: facet.  113 research article   if you build it, will they (really) come? student perceptions of proximity and other factors affecting use of an academic library curriculum collection   madelaine vanderwerff librarian, assistant professor mount royal university library calgary, alberta, canada email: mvanderwerff@mtroyal.ca   pearl herscovitch librarian, associate professor mount royal university library calgary, alberta, canada email: pherscovitch@mtroyal.ca   received: 5 nov. 2020                                                               accepted: 2 apr. 2021      2021 vanderwerff and herscovitch. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29875     abstract   objective – this study investigated student perceptions of an undergraduate university library’s curriculum collection, before and after a move to a new library building. the objective was to identify how factors such as proximity to program classrooms and faculty offices, flexible seating, accessibility, and other physical improvements to the space housing the collection impacted students’ perceptions.   methods – this longitudinal study conducted between 2016 and 2017 used a combination of methods to examine whether library use of a specialized academic library collection was impacted by a significant space improvement and change in location. a cohort of education students was surveyed before and after the construction of a new building that housed both the library and their department and co-located the curriculum collection with departmental teaching spaces. the students were surveyed about their use of the space and resources. the researchers then compared the survey results to circulation data. the researchers ground this study in lefebvre’s spatial triad theory, applying it to library design and collection use (lefebvre, 1992).   results – researchers identified proximity to classrooms and general convenience as the dominant factors influencing students’ use of the collection. survey results showed an increased awareness of the collection and an increase in use of the collection for completion of assignments and practicum work. circulation data confirmed that between 2016-2019, there was a steady increase in use of the curriculum collection.   conclusion – students’ responses revealed that physical characteristics of the space were less important than proximity, the major factor that impacted their use of the curriculum collection. this revelation confirms lefebvre’s idea that spatial practice, i.e., how users access and use the space, is more significant and identifiable to students than the design and physical characteristics of the space.     background in 2009 mount royal university (mru) transitioned from a college to a university, and in 2011, a university transfer program in education became a full bachelor of education degree. based on a recommendation by the provincial approval body, campus alberta quality council (caqc), the education librarian was granted one-time funds to transform the collection, which had focused on pedagogical theory and children’s literature, to support students in their academic work and their practicum placements in k-7 settings. this transformation required the acquisition of physical objects such as kits, realia, games, manipulatives, puppets, musical instruments, teacher support material, and textbooks. special funding for the collection was expended by 2014. after 2014, the curriculum collection was supported through an annual collection budget allocation.   the provincial government committed funding to support the building of the riddell library and learning centre (rllc), a free-standing, four-story facility, which opened in 2017. features of the rllc include: data and touch-screen visualization spaces, a makerspace, a 360-degree immersive studio, an xr experience lab, 2 flexible teaching classrooms that can accommodate up to 70 students, a temperature-controlled archive, audio-productions suites, 31 bookable group rooms outfitted with screens and white boards, 2 presentation practice rooms, silent study areas, computer commons, and group study areas, and more. the building is also home to the academic development centre, the institute for the scholarship of teaching and learning, student learning services (mount royal university’s student writing centre) and the department of education. the curriculum collection was relocated from its dusty, dark corner in the old library to a bright space with flexible furniture and shelving that is both adaptable and appropriate. the collection is adjacent to the department of education where bachelor of education students attend classes in the majority of their core courses, and is now essentially embedded in the department.   in anticipation of the move, the authors were interested in examining whether improved library facilities would have an impact on the use of the curriculum collection. we supposed that the curriculum collection was not well-used in the old library because of the unfavourable location and predicted that an improved environment would have a positive impact on use. the collection was in a remote corner that had very poor lighting and on shelving that could barely accommodate the oversized materials and larger kits. the space did not provide students or other potential users with an inviting place to explore the collection. we were interested in investigating what effect co-location or proximity to classrooms might have on students’ use of the collection.    terms used:   use: use has been defined in many ways in library literature. fleming-may (2011) identified multiple applications of the word “use” through content analysis, which could include an interaction with all library resources (things, people, services, space) measured by door counts, occupation of physical space, bibliographic analysis measuring instances in which library resources are applied or referred to as an abstract concept such as process, or utility. in the context of this study, use is defined as access to items in a physical collection. use refers to transactional instances in which individuals check physical items out of the library or interact with physical collection items.   co-location: researchers applied the definition provided by bodolay et. al (2016) as a location convenient to users across separate campus units. this does not imply the creation of new services that leverage the joint expertise of the library and campus partners.   curriculum collection: acrl’s curriculum materials committee has developed guidelines for curriculum materials that define curriculum collections as physical locations for instructional resources for preschool through grade 12 students. materials are used by education students and faculty to develop curricula and lesson plans and to complete course assignments. these collections or branch libraries are often referred to as a curriculum materials center or instruction materials lab. curriculum collections may be housed in a main campus library or the building housing the faculty or department of education program. (curriculum materials committee, education and behavioral sciences section, acrl, 2017)   the literature   facility improvement and impacts on library use   libraries completing facility improvements have reported an increase in use of library space and library collections post-renovation (albanese, 2003; martell, 2008; shill &tonner, 2004). certain factors impacting use of facilities or collections in academic libraries have been identified in the literature over the past 20 years. these include amount of space, noise level, crowdedness, comfort, type and flexibility of furniture, cleanliness, access to services and technology, and availability of collaborative space (bailin, 2013; cha and kim, 2015; gardner & eng, 2005; given & leckie, 2003; holder & lange, 2014).  proximity to collections also affects how students make choices in the selection of information to support their assignments and coursework as well as where they physically choose to sit in the library (julien & michels, 2004; may & swabey, 2015). mccreadie and rice’s (1999) examination of how and why users access information included physical constraints such as geography, space, distance, and proximity. time factors, convenience, and ease of use have been identified as significant considerations in the context of information seeking behaviour (connaway, dickey & radford, 2011; savolainen, 2006). literature on the importance of a student-centred approach to library access suggests that library co-location with a student’s home department contributes to the development of a more student-focused environment, increasing access to both services and discipline-specific resources (defrain & hong, 2020).   convenience and proximity   the theoretical grounding for this study was based on henri lefebvre’s spatial triad theory applied to library design and subsequent user perception and use. lefebvre was a marxist philosopher, well known for his work on spatial theories. in lefebvre’s view, space “cannot be separated from social relations and is the product of ideological, economic, and political forces (the domain of power) that seek to delimit, regulate, and control the activities that occur within and through it” (zieleniec, 2013, para. 9). the spatial triad theory is introduced in lefebvre’s, the production of space (la production de l’espace) (lefebvre, 1992). this is a complex theory that has the potential for wider application in the study of library spaces as it seeks to “uncover the social relations involved in the production of space and the significance this has for a comprehensive knowledge of space” (zieleniec, 2007, p. 70).  the relevance to libraries becomes apparent in lefebvre’s work when we consider the importance of social relationships in the production of space—space transformed to place as it is imbued with significance and meaning assigned by the everyday practice of its users (zieleniec, 2007). the three elements of the triad are:   ●       representations of space (conceived space) interpreted as the actual characteristics of library space as developed by architects, planners, and engineers, ●       spatial practices (perceived space) interpreted as the user’s perception of the built space, ●       representational spaces (lived space) interpreted as library users’ access and use of the space (ilako et al., 2020; leckie & given, 2010).   in lefebvre’s view, “spaces become places when individuals and groups assign meaning and social significance to them”. without meaning, space remains and exists in the realm of the abstract, defined by architects and planners (zieleniec, 2013, p. 953). our application of lefebvre’s spatial triad theory aligns with mccreadie and rice’s (1999) description of constraints, such as geography, demographics, environmental arrangement, space, distance, and proximity which can lead to perceived availability or convenience. the physical attributes of a library space can serve to influence or constrain access to information along dimensions of distance and proximity, openness and security, and clarity or obstruction. this investigation provides an opportunity to explore how user experience impacts use of or access to a discipline specific collection. applying lefebvre’s theory allows for a better understanding of the meaning and significance users assign to this area of the library as it transitions from space to place. an understanding of students’ perception of the space, and their everyday practice within it, will help the authors identify elements of control and regulation that may hinder or contribute to how students might assign significance to the space.   savolainen’s (2006) work aligns with mccreadie and rice (1999), reinforcing the importance of space and time on the use of information and spatial factors related to physical distance between the information seeker and information sources. savolainen’s idea that distance and time factors serve as a context that informs choice about information seeking is detailed by connaway et al. (2011), who view convenience as a situational criterion in people’s actions, and together with ease of use, as determining factors in how individuals make their information seeking decisions.   feedback gathered through student consultations on library redesign often reflects a preference for discipline specific libraries near their department (mccullough & calzonetti, 2017; teel, 2013). students may protest or organize petitions as they did in response to a proposed stem branch library consolidation at university of akron (mccullough & calzonetti, 2017). mru library’s curriculum collection is primarily a physical collection, consisting of print materials, manipulatives, juvenile literature, kits, and models that users need to physically access. guidelines for curriculum materials centres (2017), developed by an ad hoc committee of acrl, suggest that these libraries are often located in the same building as the department of education. this preference for a library’s proximity to a department is reinforced in an article reviewing curriculum collections in australian universities, where a change in use patterns was identified when curriculum collections moved from the building housing the education department to the main library:   ...moving into the library often changed the focus of collection use, from being an active teaching and learning area that replicated classroom and school library spaces, to being simply another library collection distant from the students’ learning environments. hence, the collections were not used as much or in the same way. for example, academics did not bring groups into the collection as much as they had previously, when the collection may have been adjacent to their lecture rooms. nor did students use the collections located in the library in the same way (locke, 2007, p.4).   in a study by teel (2013), student consultations revealed the need for improvement in physical space and technology in their curriculum materials centre and importantly, a preference for the centre to relocate to the faculty of education building. a more recent study by stoddart and godfrey (2020) examined space usage in a newly renovated curriculum centre housed in the education building. they identify the most frequently used spaces in order to better understand the centre’s contribution to “campus learning”, and emphasize the importance of connecting library design to program and university learning outcomes. these authors refer to van note chism’s discussion of the creation of spaces that have been intentionally designed to impact student learning. many of the elements described by van note chism were considered in the design of mru’s curriculum collection area, including flexibility that allows for group work, comfortable seating, natural and task appropriate lighting and de-centeredness where learning spaces flow (van note chism, 2006, as cited in stoddard & godfrey, 2020). the curriculum collection area at mru’s new library was designed to serve as an extension of the education department’s classrooms with flexible and comfortable seating and an open space that doubles as an informal gathering area or a classroom. instructors sometimes teach in the space or provide students with class time to walk down the hall and retrieve items to bring back to class. library classes are often taught in this area, requiring students to apply critical evaluation and literacy skills as they examine resources in groups. the goal of this research is to examine the impact of a very significant and intentional change in environment and space allocated to the collection and surrounding area. researchers formulated survey questions to identify the importance of location and other space related factors influencing collection use before and after the move to the new building.   methodology   this longitudinal study employed exploratory mixed methods research to examine possible changes in use of this collection over time. the goal was to try and establish meaningful connections between two sets of data collected by comparing qualitative survey responses with physical item circulation data (chrzastowski &joseph, 2005; creswell, 2003; hiller & self, 2001). ethics approval was granted by mount royal university’s human research ethics board (hreb). a survey was sent to students enrolled in a third-year education course (educ 3361) in 2017, prior to the move to a new building. the same student cohort was surveyed in a fourth-year education course (educ 4020) in winter 2018 after the library collection was moved to the new building. the survey responses were kept anonymous, as individual changes in use were less of a concern to the researchers than growth or patterns in use from the entire cohort. the rationale behind anonymizing the survey was to reduce the impulse to provide pleasing or socially desirable responses. the education librarian works closely with students in this program and has built a rapport with many of the students surveyed. as a result of this established relationship, the researchers felt that an anonymous survey would encourage honest responses regarding library use. recruitment of participants was based on their enrolment in these courses, as they are core courses in the bachelor of education program, and was conducted by both investigators during an in-person class visit. students were encouraged to complete a short, 7-8 question online survey on the survey monkey platform.   the survey questions were developed with spatial triad theory in mind. the three elements of lefebvre’s theory representations of space/conceived space, spatial practices/perceived space, and representational spaces/lived space, provided a grounding for our survey questions and data analysis (ilako et al., 2020). we attempted to determine how the design of the new library space occupied by the curriculum collection (representations of space) affected students’ use. the survey asked students which factors contributed to increased use, to determine their perceptions of the space. through the analysis of qualitative and quantitative data, the authors assessed whether the students’ perceptions of use (spatial practices) and actual collection use (representational spaces) were aligned. to further understand students' experience of the space and collection, we asked about the purpose of their collection use. while the triad identifies three elements of space, the interaction of these elements in the production of space is important to our interpretation of lefebvre’s theory. the survey included a question regarding student perceptions of the location and its impact on their use of the curriculum collection. the questionnaire also had a series of multiple-choice questions related to how students first learned about the collection, the purpose of their collection use (with children, for assignments etc.) and a demographic question about their minor. the 2018 iteration of the survey included an additional question about what factors, if any, impacted their use of the collection after the move.   a visual representation of the survey questions (excluding demographic questions) in relation to each element of the triad theory has been provided in figure 1. interpretations of the theory and its elements vary in the literature, making the process of categorizing questions difficult. the intent was not to separate them as they are interrelated. the decision to locate the collection close to the department was part of the planning process, but clearly influenced student perceptions (perceived space) and their use of the space (lived space). the impact of the architecturally conceived space on students’ perceptions and the influence of these perceptions on their daily practice or lived spatial experience demonstrates a fluid process in the production of space. the area was designed to create meaningful connections between departmental classrooms and the collection area. furniture was selected to create an informal classroom and meeting area and the hope was that the survey questions would prompt students to comment on furniture and lighting as details that influenced their perception of the area. the goal was to increase understanding of student perceptions of the space and the majority of the survey questions were concerned with spatial practice and how students’ perceptions contributed to their actual experience of the space.   in addition to the survey, physical item circulation data between the period of 2013-2019 was gathered and analysed. with the assistance of a staff member in the library’s information systems unit, and a staff member in the collections unit, data was extracted from the two integrated library systems (voyager and alma), in use during the study period. transactions which qualified for inclusion included any item that was signed out by a patron.     figure 1 survey questions mapped to lefebvre’s spatial triad.     results   survey results   the sample size was small, with 62 students enrolled in educ 3361 in 2017 and 65 students enrolled in educ 4020 in 2018. in 2017, 59 students completed the online survey (n=59 or 95% response rate) and in 2018, 38 students completed the survey (n=38 or 58% response rate). responses were migrated from surveymonkey into a spreadsheet, multiple choice answers were tallied, and a content analysis schema was applied to the text of the short answer/open text responses by two independent coders guided by pre-established codes and themes.   general knowledge of the collection   out of the 59 students surveyed in 2017, 56 (95%) were aware that the collection existed. again, in 2018, 34 out of 36 (95%) students surveyed said that they knew about the collection. if students answered no to this question, they were not asked subsequent questions and the survey ended. this question helped eliminate responses from students who could not answer the rest of the survey, so sample size changed to n=56 in 2017 and n=36 2018.   in 2017, 24 (41%) students indicated that they learned about the library through the education librarian, while 27 (47%) learned about it through instructor endorsement and 7 (12%) discovered the collection through their peers. in 2018, responses to this question shifted as 23 (61%) students indicated that they learned about the collection through an instructor, followed by the librarian at 32% (12) and 8% (3) from their fellow classmates. the “power users” of the collection, who used it 10 or more times, were minoring in indigenous studies, humanities, math, general sciences and teaching english as a second language (tesl), a pattern consistent in both years.   student perceptions of use   students were asked to select a number range reflecting their use of the curriculum collection during the course of their program (table 1) as well as the purpose of their use (table 2).   factors that impacted use   students were asked a direct question about whether location had an impact on their use of the curriculum collection. in 2017 (pre -move), 33% of students said that location did impact use, and 67% of students surveyed indicated location did not have an impact. in 2018, 44% of students responded that location had an impact, while 55% said that it did not affect their use of the collection. students were asked to list other factors that impacted their use.  the following themes were identified in their responses:          ●          types of material available in the collection (suggestions of what we need more of, or what was useful to them, by way of subject or format)        ●          characteristics of space (dark corner, “squished aisles”, accessibility, location within the library)        ●          knowledge of the collection        ●          proximity to practicum        ●          proximity to classes        ●          cost savings (having access to the collection meant that they did not have to purchase their own materials)   in response to an open-ended question at the end of the survey in 2017, several students indicated that they did not learn of the curriculum collection until later in their degree. as mentioned previously, the location of the curriculum collection in the old library was not highly visible, housed at the back corner of the library with very little lighting and not many workspaces or seating directly adjacent to the collection. in 2018, almost all qualitative responses were related to the location of the collection. the primary focus of those responses was on how convenient it was to access the collection now that it was on the same floor as their classes and how the space more organized. the data indicates that planning the new building to locate the curriculum collection adjacent to the collection, so that students pass it every day to get to their classrooms, has had a positive impact on their perceived use of the collection.     table 1 frequency of student use of curriculum collection number of times you have used the collection throughout the course of your studies to date 2017 responses 2018 responses increase/decrease 0 5 2 -1.6% 1-5 35 16 -18.23% 6-10 12 14 +16.15% 10 or more 6 6 +5.45% total responses 58 38       table 2 purpose of student use of curriculum collection for what purpose have you used the curriculum collection? 2017 responses 2018 responses with children outside the program 31% 42% class work 78% 81% completion of assignments 80% 97% practicum 64% 44% other 1% 3%     circulation data analysis   mru library employs a liaison delivery model for library instruction and maintains a similar model for collection development. each program is assigned a subject librarian with an annual collection budget allocation. the collection allocation formula considers several factors including number of enrolled students, full time faculty, and circulation data in the determination of each disciplinary budget. subject budgets align with the overall acquisitions budget and, due to the current economic climate, the library has not seen an increase in the acquisition budget for some time. annual acquisitions in all disciplines have primarily attempted to maintain library collections to support current programs.   analysis of circulation patterns reflects steady growth of the curriculum collection and an increase in the use of the collection between 2016 (before the move) and 2019. (figure 2). items circulated refers to physical item transactions (charges, recalls, renewals, holds). we compared curriculum collection circulation of physical items (figure 2), with overall circulation of the entire library (figure 3). from 2016-2019 all library circulation statistics remained relatively consistent before and after the move; 40% of the collection circulated (either browsed or borrowed) in 2016, 37% in 2017, 33% in 2018, and 42% in 2019. the curriculum collection saw a significant increase in use immediately after the move to the new building with 36% of items circulating in 2016,62% in 2017, 85% in 2018, and 75% in 2019.     figure 2 total items versus circulations in curriculum collection. figure 3 total items versus circulations in library.     discussion   student survey responses   of the bachelor of education students surveyed, 95% were aware of the collection, once the collection was moved next door to their classrooms. of particular interest was the way in which students learned about the collection. in 2018, survey responses indicated that students learned of the collection more often from their instructors. when students were asked about how they used the collection, data reflected increases in use for supporting in-class work and completing assignments. in both years surveyed, students who identified as power users (those users who used it ten or more times), indicated that they also used the collection beyond the classroom and used materials from the collection for practicum related work or for purposes directly involving children. students who used the collection less, generally responded that they used the collection to support class work or assignments and remained consistent both years surveyed. the new location is a few metres from department of education classrooms, allowing students to use the space for group work, study, and completing assignments. the program employs a cohort model and group assignments are common in many of the required courses (mount royal university, 2020). the results of the survey indicate that those who use the collection often are taking advantage of the collection and bringing materials off campus to support their practicums.     figure 4 curriculum collection circulation by user type.     mru library applies a liaison librarian model with a single librarian assigned to a subject area or department in order to provide teaching, research and collections support. from 2016-2019 the education librarian delivered an average of 6 library sessions per semester. library instruction is always assignment based and as the bachelor of education program is relatively new, assignments change regularly. there was no change in the education liaison librarian, so the general level of promotion for the curriculum collection did not vary preand post-move. however, after the move, library instruction delivered to education students took place in department classrooms, library labs, or the curriculum collection area located directly adjacent to the department. before the move, the department of education, and many of the classrooms where library instruction occurred, took place in campus locations that were a 5–10-minute walk from the library. it is interesting that after the move there was a shift in how students learned of the collection from librarian to instructor, which could be indicative of an increase in faculty knowledge of the collection, faculty use of the collection, or faculty integration of the curriculum collection into course assignments. due to data collection and retention policies at the university, the library collects limited personal data related to patrons, so it is difficult to identify who is using the collection and what program they are connected to. we looked at changes in use according to patron type before and after the move to the new library and noted that faculty circulation transactions doubled in the course of four years and that students are the primary users of the collection, with the greatest increase occurring after the move to the new space (see figure 4).   there was a noticeable decrease in staff use of the collection post-move which reinforces the importance of collection proximity. the library moved from a location in the heart of the main campus building to a freestanding building on the edge of the campus. while convenience was enhanced for education students through co-location, convenience decreased for many staff on campus who, we can surmise based on the data, were deterred by the walk across campus required to access the collection in the new building.   while the collection is available to patrons outside of the department of education, we focused on the curriculum collection's intended user group to understand the impact of co-location and other factors on students' use. considering the observable growth in faculty use, and the increase in student responses indicating faculty endorsement of the collection, it would be worthwhile to investigate how often the collection is incorporated into assignments. the increase in community borrower and alumni use is also noteworthy. the rllc is a free standing, 4 story building, where the previous library was a single level space located in the main campus building. the move and new adjacencies with building partners such as the department of education increased convenience and accessibility for these students, and circulation data also suggests a positive impact on access and convenience for members of the public and alumni. we built it and they came.   we asked students about their minor to determine if there were patterns in subject area use with the goal of providing direction for future collection development. correlating minors with use levels was a challenge because the survey question asked students to respond based on a numbered range of uses. students who identified the largest range provided, (10 or more times), were minoring in both arts and humanities-based disciplines such as indigenous studies, humanities, and teaching english as a second language (tesl), and stem disciplines like math and general sciences. previous studies have indicated that science students are less likely to access library collections in person while arts students are more likely to use print and on-site materials (chrzastowski & joseph, 2006; whitmire, 2002). the responses to this survey could be indicative of faculty endorsement, disciplinary norms, or requirements for use of the collection in coursework and assignments.   students’ perceptions of use: proximity is everything   student responses regarding the impact of proximity are both a reflection of the work of architects, designers, and planners (representations of space) and the perceived value students place on convenience and easy access (spatial practice). other comments refer to the usefulness of the collection and its relevance to practicum or professional practice (representational space). comments illustrate the relationship between the three elements of the triad. they are inextricably tied to one another as the meaning students assign to the space evolves from an initial response to the planners’ location choice, leading to an ease of access for course work. students proclaimed “love” and appreciation for the space and collection led to its incorporation into practicum and course work contributing to the production of space.    “the new location for the curriculum collection is easy to access and organized in a fashion that is easy to navigate. the central location and organized sections have made it more accessible and easier to utilize.”   “i enjoy going to the new location better, so i find myself near the curriculum collection more often.”   “easier access”   “classes were all in the library building so (sic) was never out of my way to visit.”   “before it moved, i did not use it because i was unaware of where it was”.   “it has because it is in closer proximity to where i study.”   proximity emerged as the most significant factor in students’ increased use of the curriculum collection. it was apparent that after the move to the new location adjacent to their classrooms, students were using the collection with greater frequency (table 1). because of the change in proximity, the collection became more visible to its target user group which had a positive impact on awareness and use of the collection. this reinforces the idea that physical proximity can have a positive effect on academic libraries’ ability to serve their users (freiburger et al., 2016). circulation data verified a substantial increase in use between 2016 and 2018. this increase aligns with student responses and with the literature on library space improvement and increased use of an academic library collection.      figure 5 student comments mapped to lefebvre’s spatial triad theory.     survey responses, however, suggest some confusion on the part of students about consistent definition of terms. convenience was used interchangeably with proximity, which speaks to the likeliness that in the lives of students these terms may be equivalent. students frequently referred to space as a limiting factor in accessing the collection. the physical space in the new library has been identified as a significant improvement by visitors, but students made few references to the space itself as a factor in their increased use of the collection. an open and bright space with tables, carrels, comfortable seating, and group rooms contrasts significantly with the crowded, dark corner previously used to house the collection. the survey questions did not prompt students to consider these specific factors in their assessment of increased use of the collection. while students in the pre-move survey indicated that location impacted use, questions were not specific enough about whether it had a negative or positive effect. certain questions on the survey could have been asked differently and might have elicited more informative and specific responses related to space.   limitations   during the process of coding qualitative responses, it was discovered that there were some omissions and minor flaws in the wording and specificity of the survey questions. while maintaining a likert scale, the same cohort could have been asked to describe their range of perceived use in the year surveyed, not the duration of their studies, assuming that as students progress through their degree, their library use would only have increased. also, there was growth in the collection over the 3-year period from which circulation data was extracted and analysed, and a larger or more improved collection could have contributed to the increase in students’ use. analysis of circulation data showed an increase in use specific to user type, but privacy restrictions mean the program to which students and faculty are attached cannot be determined. without that data it is not possible to assign the increase in circulation to education students with perfect certainty. a review of transactions by patron type preand post-move also reveals that other borrowers are using the collection. librarians’ definitions of terms may differ from students. providing definitions at the start of the survey for terms like “use” ensures clearer and more meaningful responses (kidston, 1985). there was also an expectation that students would have elaborated in their responses regarding the improved space. if the survey was redeployed, questions would provide details specific to lighting, furniture, and study spaces to determine if these were additional factors that impacted use. some students mentioned these factors within their responses, but not to the extent anticipated.   other considerations academic branch and specialized collections   recent branch closures and consolidation in academic libraries underscore the importance of identifying the value of locating discipline-specific collections close to the departments they support. in 2004, hiller reported on a series of measures used at university of washington to evaluate the viability of branch libraries. he predicted the acceleration of branch library closures and mergers with the exception of those serving programs that are “dependent on print collections and that provide space that supports students work in a collaborative teaching and learning environment” (p. 131). curriculum collections fall into the category of libraries that rely on print and physical objects, but this has not protected them from mergers. more recently, mccullough (2017) identified branch consolidation as a long-term trend in the context of academic libraries’ response to budget reductions, the shift to electronic collections, and campus space concerns. evidence relating to use patterns and the integration of library material into course assignments and curriculum are crucial, particularly in light of de-funding, and budget cuts. when assessing the closure of branch libraries, budget concerns and low circulation statistics inform part of those decisions. branch closures or amalgamations with larger libraries can have a variety of negative impacts on university library systems, including a decline in overall use of print or physical resources, a negative perception of service, and a decrease in requests for information literacy instruction (lange et al., 2015). sometimes the notion of “library as a place” or the intrinsic value of a physical space offers value despite low circulation statistics or gate counts. however, even high-use branches that serve large student populations are subject to closures. university of alberta coutts library, a branch library serving the faculties of education and kinesiology, was recently closed due to budget cuts (lachacz, 2020). high circulating collections that consist of physical books and manipulatives are clearly not exempt from this trend. curriculum collections, or other specialized collections that rely heavily on the circulation of physical resources and student use of physical spaces, have been identified as vulnerable to branch consolidation (zdravkovska, 2011). budget concerns are driving branch consolidations in the face of evidence presented by many studies suggesting that these high-use branches serve their users more effectively when they are in close proximity to their corresponding department or faculty (locke, 2007; hiller, 2004).   conclusion   this study demonstrates that a particular user group’s use of a collection and space, in this case undergraduate bachelor of education students, is significantly impacted by how they perceive the space that houses the collection. participants in this study demonstrated a change in their perception of a discipline specific collection after a significant improvement was made to the library space housing the collection. the curriculum collection, which was in an unfavourable and inconvenient library location, distant from classrooms and education department offices, was used less frequently prior to the library’s move to a new building.  once the curriculum collection was relocated, adjacent to the department of education, where the collection’s primary, intended user group gathered for classes, circulation statistics increased. in their survey responses, students identified proximity to the collection as having a positive impact on their use of the collection. this reinforces lefebvre’s spatial triad theory describing how conceived space is directly related to perceived and lived space. a question remains regarding the particular meanings or social significance assigned to the current space and how these may be controlled or prompted by course curriculum or assignment requirements. an exploration of the incorporation of the collection into the education curriculum will provide a more comprehensive understanding of factors contributing to student use of the space and collection. the investigators are currently collecting data in the second part of this study, investigating how education faculty use the collection in their teaching and assignments.   readers may find it useful to consider the power of lefebvre’s theory to provide a lens through which to understand how library space planning contributes to the production of space where users assign meaning in the completion of their course and professional work. leckie and given (2010) state that “the relationship between perceived, conceived and lived are not linear and not stable but rather are fluid and dynamic” (pp. 228-229). the curriculum area examined in this study is not a static space and will continue to evolve to meet users’ curricular and professional needs. a future study may provide opportunities to understand how the space and collection can serve as a more effective extension of the classroom and education program curriculum, allowing users to challenge our original design and create a more meaningful lived space. lefebvre’s theory has provided a context for the cyclical nature of space production as challenges provide users with the opportunity to produce and reproduce space.   important issues came to the attention of the researchers indirectly during this study.  responses from students suggested that there could be a connection between increased student use and the incorporation of the collection into assignments and course curriculum. after the move, faculty increasingly recommended the collection to students and developed assignments that required the use of curriculum resources. the researchers will endeavour to explore use patterns among user groups and survey faculty about changes in how they incorporate the curriculum collection into teaching and assignments. a future study that investigates the relationship between student collection use with curriculum integration could provide deeper insight into how the collection is being used. this point of inquiry was identified through the triangulation of survey and circulation data, which provided a more complete picture of how the collection was being used, or in lefebvre’s terms, how the space was produced. knowing how faculty and students are integrating physical collections into their course work and assignments will inform space planning and librarians’ collection development and teaching practices to meet users’ needs more effectively. there is also a growing number of branch and specialized collection closures and consolidations occurring in academic libraries. evidence of the importance of collection proximity to academic programs and integration with student learning may inform future management of these spaces and difficult decisions related to closures.   acknowledgments   thanks to jenny joe, ils programmer analyst, mount royal university for assisting with data ils data extraction; and margy macmillan, & richard hayman for feedback and comments.   author contributions   pearl herscovitch: conceptualization (equal), formal analysis (equal), investigation (equal), visualization  (equal), writing – original draft (equal), writing – review & editing madelaine vanderwerff: conceptualization (equal), data curation, formal analysis (equal), investigation (equal), methodology, visualization (equal), writing – original draft (equal)   references   albanese, r.a. (2003). deserted no more: after years of declining usage statistics, the campus library rebound. library journal, 128(7), 34-36.   bailin, k. (2011). changes in academic library space: a case study at the university of new south wales. australian academic & research libraries, 42(4), 342-359. https://doi.org/10.1080/00048623.2011.10722245   bodolay, r., frye, s., kruse, c., & luke, d. (2016). moving from co-location to cooperation to collaboration. in space and organizational considerations in academic library partnerships and collaborations (pp. 230–254). https://doi.org/10.4018/978-1-5225-0326-2.ch011   cha, s. h., & kim, t. w. (2015). what matters for students' use of physical library space? the journal of academic librarianship, 41(3), 274-279. https://doi.org/10.1016/j.acalib.2015.03.014   chrzastowski, t. e., & joseph, l. (2006). surveying graduate and professional students' perspectives on library services, facilities and collections at the university of illinois at urbana-champaign: does subject discipline continue to influence library use? issues in science & technology librarianship, 45, 12.   connaway, l., dickey, t., & radford, m. (2011). “if it is too inconvenient i’m not going after it:” convenience as a critical factor in information-seeking behaviours. library & information science research., 33(3), 179–190. https://doi.org/10.1016/j.lisr.2010.12.002   curriculum materials committee, education and behavioral sciences section, acrl.  (2017). guidelines for curriculum materials centers.   creswell, j. w., plano clark, v. l., gutmann, m. l., & hanson, w. e. (2003). advanced mixed methods research designs. handbook of mixed methods in social and behavioral research, 209, 240.   defrain, e. & hong, m. (2020). interiors, affect, and use: how does an academic library’s learning commons support students’ needs? evidence based library and information practice, 15(2), 42-68. https://doi.org/10.18438/eblip29677   fleming-may, r. (2011). what is library use? facets of concept and a typology of its application in the literature of library and information science. the library quarterly: information, community, policy, 81(3), 297–320. https://doi.org/10.1086/660133     freiburger, g., martin, j. r., & nuñez, a. v. (2016). an embedded librarian program: eight years on. medical reference services quarterly, 35(4), 388-396. https://doi.org/10.1080/02763869.2016.1220756   gardner, s., & eng, s. (2005). what students want: generation y and the changing function of the academic library. portal: libraries and the academy, 5(3), 405-420. https://doi.org/10.1353/pla.2005.0034   given, l., & leckie, g. (2003). “sweeping” the library: mapping the social activity space of the public library. library & information science research, 25(4), 365–385. https://doi.org/10.1016/s0740-8188(03)00049-5   hiller, s., & self, j. (2001). a decade of user surveys: utilizing and assessing a standard assessment tool to measure library performance at the university of virginia and university of washington [conference session]. 4th northumbria conference. washington, dc. https://digital.lib.washington.edu/researchworks/bitstream/1773/19920/1/northumbria%202001%20pm4.pdf   hiller, s. (2004). measure by measure: assessing the viability of the physical library. the bottom line, 17(4), 126-131. https://doi.org/10.1108/08880450410567400   holder, s., & lange, j. (2014). looking and listening: a mixed-methods study of space use and user satisfaction. evidence based library and information practice, 9(3), 4-27. https://doi.org/10.18438/b8303t   ilako, c., bukirwa, m. & maceviciute, e. (2020). relevance of the spatial triad theory in (re) designing and planning of academic library spaces.  qualitative and quantitative methods in libraries, 9(1), 65-75.   julien, h., & michels, d. (2004). intra-individual information behaviour in daily life. information processing & management, 40(3), 547-562. http://doi.org/10.1016/s0306-4573(02)00093-6   kennedy, j. (2008). anonymity. in p. j. lavrakas (ed.), encyclopedia of survey research methods (pp. 27-28). sage. http://doi.org/10.4135/9781412963947   kidston, j. s. (1985). the validity of questionnaire responses. the library quarterly, 55(2), 133-150. https://doi.org/10.1086/601589   lachacz, a. (2020, june 4). breaking: u of a closes coutts library; marks second library closure due to budget cuts. the gateway. https://thegatewayonline.ca/2020/06/breaking-u-of-a-closes-coutts-library-marks-second-library-closure-due-to-budget-cuts/   lange, j., lannon, a., & mckinnon, d. (2014). the measurable effects of closing a branch library: circulation, instruction, and service perception. portal: libraries and the academy, 14(4), 633-651. https://doi.org/10.1353/pla.2014.0031   leckie, g. & given, l. (2010). henri lefebvre and spatial dialectics. in g. leckie, l. given & j. buschman (eds.), critical theory for library and information science: exploring the social from across the disciplines (pp. 221-236). libraries unlimited.   lefebvre, h. (1992). the production of space (d. nicholson-smith, trans.). wiley blackwell. (original work published 1974)   locke, r. (2007) more than puppets: curriculum collections in australian universities. australian academic and research libraries, 38 (3), 192-215. https://doi.org/10.1080/00048623.2007.10721296   martell, c. (2008). the absent user: physical use of academic library collections and services continues to decline 1995–2006. the journal of academic librarianship, 34(5), 400-407. https://doi.org/10.1016/j.acalib.2008.06.003   may, f., & swabey, a. (2015). using and experiencing the academic library: a multisite observational study of space and place. college & research libraries, 76(6), 771-795. https://doi.org/10.5860/crl.76.6.771   mccreadie, m. & rice, r.e. (1999). trends in analyzing access to information. part i: cross-disciplinary conceptualizations of access. information processing & management, 35(1), 45-76. https://doi.org/10.1016/s0306-4573(98)00037-5   mccullough, i.  & calzonetti, j.  (2017). are they different? an investigation of space and learning in a stem branch library. in s. montgomery (ed.), assessing library space for learning (pp. 143-166).  rowman & littlefield.   mount royal university. (2020). department of education practicum & field experience. https://www.mountroyal.ca/programscourses/facultiesschoolscentres/healthcommunityeducation/departments/education/practicum-field-experience/index.htm    shill, h. b., & tonner, s. (2004). does the building still matter? usage patterns in new, expanded, and renovated libraries, 1995–2002. college & research libraries, 65(2), 123-150. https://doi.org/10.5860/crl.65.2.123   savolainen, r. (2006). spatial factors as contextual qualifiers of information seeking. information research: an international electronic journal, 11(4), 2.   stoddart, r., & godfrey, b. (2020). gathering evidence of learning in library curriculum center spaces with eb gis. evidence based library and information practice, 15(3), 21–35. https://doi.org/10.18438/eblip29721   teel, l. (2013). transforming space in the curriculum materials center. education libraries, 36(1), 4-14.   whitmire, e. (2002). disciplinary differences and undergraduate’s information‐seeking behavior. journal of the american society for information science and technology, 53(8), 631-638. https://doi.org/10.1002/asi.10123   zdravkovska, n. (2011). academic branch libraries in changing times. elsevier. http://doi.org/10.1016/b978-1-84334-630-2.50005-8     zieleniec, a. (2013). space, social theories of. in b. kaldis (ed.), encyclopedia of philosophy & the social sciences (vol. 1, pp. 953-955). sage. http://dx.doi.org/10.4135/9781452276052   zieleniec, a. (2007). space and social theory. sage. http://dx.doi.org/10.4135/9781446215784   news/announcements   acrl uls evidence-based librarianship discussion at ala annual      2013. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   interested in performing better research as a librarian practitioner? want to learn how to inform decision making in your library with research and evidence? then join the acrl uls evidence based discussion group at ala annual. we will be hosting a discussion on sunday 30 june from 8:30 am 10:00 am in mccormick place n230a. invited guests for the discussion are:   ·         john budd, professor, school of information & learning technologies, university of missouri ·         sharon weiner, professor of library science and w. wayne booker chair in information literacy, purdue university libraries ·         nancy adams, associate director/coordinator of education & instruction, penn state hershey, george t. harrell health sciences library   this is great opportunity to talk and ask questions about research in librarianship with other interested professionals.   evidence summary   information literacy (il) intervention workshop has positive, but limited, effects on undergraduate students’ il skills   a review of: gross, m. & latham, d. (2013). addressing below proficient information literacy skills: evaluating the efficacy of an evidence-based educational intervention. library & information science research, 35(3), 181-190. http://dx.doi.org/10.1016/j.lisr.2013.03.001   reviewed by: lisa shen business reference librarian sam houston state university huntsville, texas, united states of america email: lshen@shsu.edu   received: 9 mar. 2014    accepted: 30 apr. 2014      2014 shen. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to evaluate the impact of an educational intervention workshop on students’ information literacy (il) skills and self-perception of their own il knowledge.   design – quasi-experimental design with control groups and semi-structured interviews.   setting – two community colleges in the united states of america, one in a rural setting and one in an urban setting.   subjects – ninety-two students enrolled in an entry-level english course, who scored below proficiency (65%) on the information literacy test (ilt).   methods – one hundred students from each college took the pre-session ilt and an il self-assessment survey at the beginning of the spring 2011 semester. the ilt used was developed and validated by james madison university (wise, cameron, yang, & davis, n.d.) and measures understanding of all the association of college and research libraries (acrl) information literacy competency standards (acrl, 2000, pp. 2-3) except standard 4. for motivation, students each received $20 for their efforts and were told those who scored in the top 15% would enter a draw to win one of two additional prizes of $50. those who scored below the ilt proficiency level of 65% were invited to participate in the quasi-experiment.   forty-nine participants were assigned to the workshop group and 43 to the control group. the two groups were comparable in demographic characteristics, prior il learning, and ilt scores. those in the workshop group were ask to attend one of five workshops designed around the analyze, search, evaluate (ase) process model for il interventions (gross, armstrong, & latham, 2012). the workshops were offered on both campuses and taught by the same instruction librarian.   the workshop participants completed questionnaires, which included a second ilt, self-assessment, and ase-based questions, before and after the il workshops. each workshop participant received $30. the control group participants took the same post-session questionnaire after the workshops were completed and received $20. the same $50 incentive was offered to both groups. two weeks after the workshops, semi-structured individual interviews were conducted with 30 participants to analyze their learning experiences.   results – participants’ self-assessment of il skills showed significant downgrading after they took the ilt for the first time. this downward calibration held true for both the control (t (41) = 4.077, p < 0.004) and the workshop (t (45) = 4.149, p < 0.000) groups. subsequent self-ratings from the control group showed this downward recalibration of self-assessment was sustained over time.   for participants in the workshop group, their average self-rating of il ability rose from a pre-ase workshop rating of 2.79 out of a maximum score of 5, to a post-workshop rating of 3.83. however, the same participants’ post-workshop ilt scores did not show any significant improvement. attending the ase workshop did not help participants to achieve the “proficient” il skill level (an ilt score of 65% or higher).   nonetheless, the workshop group’s performance on the ase focused questions, also administered preand post-session, indicated that participants did gain some il skills during the workshop. on the ase questions, which had a maximum score of 25 points, the workshop group’s average score increased from 10.62, pre-session to 13.40, post-session, while the control group had an average score of 10.91 pre-session and 10.77 post-session.   in the follow-up interviews, most participants reviewed the workshop positively and felt that their peers would benefit from attending. however, the skills participants reported learning primarily focused on the search stage of the ase model, such as exact phrase, truncation, and the advanced search options in google.   conclusion – this quasi-experiment examined the impact of a one-hour ase model-based workshop on first-year english students with below-proficiency il skill levels. self-assessment ratings indicated that workshop attendance increased students’ confidence in their skill level, although this upward recalibration of self-view significantly overestimated participants’ actual skill gain. preand post-test questionnaires indicated that, while students did gain some new il knowledge, attending the workshop was insufficient to improve their il skill to the proficient level.     commentary   the design of this study appears sound. the authors also provided either copies of, or citations for all the assessment instruments. nonetheless, the study scored an overall validity of 70%, slightly below the acceptable validity measure of 75%, on the evidence-based librarianship (ebl) critical appraisal checklist (glynn, 2006).   the overall rating was negatively affected by some missing procedural information, such as ethics approval and participant consent, and minor study design flaws, such as asking the control group questions about the ase workshop in the post-session questionnaire. while it is important to keep all conditions comparable between the control and workshop groups, including three survey questions participants cannot answer might be unnecessary. for instance, the researchers noted 35% of the control group participants “failed to respond” (p. 186) to the final self-assessment question, likely because it was about the workshop. however, it would be more interesting to know how the other 65% of participants responded to a question requesting comments to a workshop they did not attend.   one omission of concern was the workshop group’s post-session ilt scores. the authors had stated there was “no evidence of improved scores… for any of the participants” (p. 187), though evidence was also unavailable to support this observation. the ilt is a validated 65-question instrument with four subscales. therefore, participants’ performance on this test demands some elaboration, especially since extensive discussion was made on the same group’s performance on the post-session ase questionnaire.   some of these issues in the article may be attributed to editorial decisions. the authors had included extensive information about the ase-based il workshop, which took up over 10% of the article, leaving less space for other study details. the workshop development was an extensive project and deserves due attention, however, this information had already been captured in another article by the authors (gross, armstrong, & latham, 2012). therefore, it would have been more effective for the authors to refer readers to the other ase publication, than attempt to describe two complex studies in one article.   despite these issues, this study demonstrated the limited impact of a one-hour workshop on students’ actual il skills and the false positive self-assessment such workshops could generate. the study provides a timely and valuable contribution to current il research and its findings provide strong practical implications for the current trends in reconsidering the usefulness of student self-perception reports and effectiveness of one-time il workshops.     references   association of college and research libraries (2000). information literacy competency standards for higher education. retrieved from http://www.ala.org/ala/acrl/acrlstandards/ informationliteracycompetency.htm   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi:10.1108/07378830610692154    gross, m., armstrong, b., & latham, d., (2012). the analyze, search, evaluate (ase) process model: three steps toward information literacy. community & junior college libraries 18(3-4), 103-118. http://dx.doi.org/10.1080/02763915.2012.780488   wise, s. l., cameron, l., yang, s.-t., & davis, s. (n.d.). information literacy test: test development and administration manual. harrisonburg, va: institute for computer-based assessment center for assessment and research studies, james madison university.   call for applicants for eblip journal: communications officer news   call for applicants for eblip journal: communications officer    2022. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30108     evidence based library and information practice (eblip), a peer reviewed open access journal published since 2006, seeks to fill the position of communications officer. the successful candidate will be expected to begin their role by june 2022 and serve a three-year term (with possibility of renewal).   the communications officer will be responsible for:   developing and implementing a communications plan for the journal. developing, designing, writing, and editing material to promote the journal in collaboration with the editor-in-chief. this can include email messages to mail lists, postings on social media, and announcements in the eblip journal. communicating and consulting with the editor-in-chief on a regular basis. attending editorial board meetings via zoom on a quarterly basis. ensuring that all deadlines are met.   the ideal candidate for communications officer will have skills and knowledge in areas of communications, preferably in the scholarly/academic realm. excellent written communication skills in english are required. experience with social media, developing and executing communication plans, and an interest in evidence based librarianship and open access are considered assets. this position requires dedicated time on a regular basis, and it is therefore essential that interested persons ensure available time to devote to this position prior to applying. it is estimated that the workload is approximately 5-10 hours per month.   interested persons should send a resume/cv and cover letter indicating areas of strength they would bring to the role to ann medaille, editor-in-chief, at amedaille@unr.edu by april 15, 2022. please email ann with any questions.   **please note that evidence based library and information practice is a non-profit, open access journal, and all positions are voluntary and unpaid.   about the journal:   published quarterly and hosted by the university of alberta, this peer-reviewed, open access journal is targeted at all library and information professionals interested in an evidence based model of practice. by facilitating access to librarianship research via original research articles and evidence summaries of relevant research from the library literature, evidence based library and information practice enables librarians to practice their profession in an evidence based manner. please visit the evidence based library and information practice web site (https://journals.library.ualberta.ca/eblip/index.php/eblip) for further information about the journal.   evidence based library and information practice evidence based library and information practice 2009, 4:4     evidence based library and information practice         news/announcements    canadian cochrane network and centre 8th annual symposium   evidence in uncertain times: meeting the challenge      © 2009. this is an open access article distributed under the terms of the creative commons attribution   license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and   reproduction in any medium, provided the original work is properly cited.      the canadian cochrane network and centre  is please to announce that our 8th annual  symposium will be held in ottawa, may 19‐20,  2009. as always, our mission is to support  evidence‐based medical decision making and  to promote and disseminate evidence‐based  information as far and wide across canada as  possible. this symposium is no exception,  with our theme being evidence in uncertain  times: meeting the challenge.    the ccnc recognizes that these are uncertain  times for many people in many ways:  canadians are dealing with a struggling  economy – we need the most ‘bang for our  buck’ when it comes to healthcare; the country  faces a looming pandemic – we need to ensure  we have the best possible evidence to make  informed decisions to treat widespread  ailments; and our population, young and old,  faces a myriad of diseases and conditions  daily – we need to be sure our healthcare is  being handled appropriately.    the canadian cochrane network and centre  will investigate these questions, and solutions,  at our symposium. we are inviting national  and international experts to explore these  topics and help to make sense of them with  you, while discussing the best direction for the  future.   please join us on may 19‐20 to be a part of this  important event. we also invite you to attend  our pre‐symposium workshops from 17‐18  may. this is an ideal time to come to ottawa  and explore evidence in uncertain times while  taking some time to explore the nation’s  capital during its famous tulip festival. let  the ccnc stimulate your mind while the city  heightens your senses. symposium guests will  stay at the marriott hotel on kent street for a  special conference rate.    please continue to visit  http://ccncsymposium.com for program  updates and event information. feel free to  contact us at ccnc.symposium@uottawa.ca if  you have any questions.    we’d love to see you here.    the canadian cochrane network and centre     are you cochrane library user? please fill out  our three‐minute user survey by visiting  http://tiny.cc/cochranesurvey or  http://ccnc.cochrane.org. we need your  feedback to help make a case to extend the  national library license beyond december  2009.     109 http://ccncsymposium.com/ mailto:ccnc.symposium@uottawa.ca http://tiny.cc/cochranesurvey http://ccnc.cochrane.org/ public libraries help patrons of color to bridge the digital divide, but barriers remain evidence summary   public libraries help patrons of color to bridge the digital divide, but barriers remain   a review of: pun, r. (2021). understanding the roles of public libraries and digital exclusion through critical race theory: an exploratory study of people of color in california affected by the digital divide and the pandemic. urban library journal, 26(2). https://academicworks.cuny.edu/ulj/vol26/iss2/1/   reviewed by: kimberly mackenzie research data and scholarly communications librarian university of massachusetts medical school worcester, massachusetts, united states of america email: kimberly.mackenzie@umassmed.edu   received: 30 mar. 2021                                                             accepted:  24 may 2021      2021 mackenzie. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30035     abstract   objective – this study explored the role of the public library in the support of patrons of color who experience digital exclusion.   design – in-person and telephone interviews, grounded theory, and critical race theory.   setting – public libraries in california.   subjects – persons of color who were active public library technology resource users due to experiencing the digital divide.   methods – in-person, 60to 90-minute interviews were conducted with participants referred to the author by public librarians at select libraries in california. sixteen open-ended questions were asked, relating to demographics, access to technology at home, library technology access and use, technology skills, and thoughts on how libraries could change or improve technology services. a 20to 30-minute follow-up interview was conducted during the phase of the covid-19 pandemic when public libraries were closed. interview transcripts were analyzed by the author, who created a codebook of common themes. responses were analyzed through the lens of grounded theory and critical race theory.   main results – nine participants were recruited; six consented to the first interview and two of the six consented to the second interview. four of the participants self-reported as asian, one as black/african american, and one as hispanic/latino american. none of the participants had internet access in their homes, though some reported having laptops or inconsistent cellular service.   common uses of library technology included job search activities (resume building, job searching, applications); schoolwork; research and skill development; and legal or housing form finding. leisure activities including social media and youtube were also mentioned.   access limitations included inconvenient library hours, particularly for those attending college or holding a job with daytime hours, and physical distance from the library. a common complaint was the time limit on computer access set by the library; “the concept of time” was mentioned “over 70 times collectively by all participants” (p. 14).   language was another barrier to access, mentioned by three of the participants. most reported being more likely to ask for help from a library staff person who shared their language or had a similar background. participants also reported wishing more technology workshops were offered, especially workshops in languages other than english.   the two participants who took part in the second interview “expressed frustration and sadness” about the lack of library access during the covid-19 pandemic (p. 16). one participant reported having to get internet access at her home for her children to attend school. the second participant expressed her difficulty in conducting research or printing information with only the small screen of her phone to provide access.   conclusion – library patrons of color living within the digital divide make use of public library technology but experience multiple barriers. libraries can alleviate these barriers by examining their hours, policies, and staffing models to be more accessible to patrons of color lacking internet access at home.   commentary   letts et al.’s (2007) questionnaire, critical review form for qualitative studies, was used for this analysis. the study’s purpose was clearly explained through an extensive literature review and discussion of critical race theory (crt), the theoretical framework employed. the interview questions were, for the most part, constructed so that participants could answer with as much or as little information as they felt comfortable sharing. however, question 11 could have been leading as written: “what are the benefits of using the library computer compared to home?” this question would be less leading if posed as “are there any benefits to using the library computer compared to home?” since many participants mentioned language barriers in response to question 13, “if there’s a library staff member who shares a similar background as you, are you more willing to ask for help? why or why not?”, further research should include language as a variable.   while the interview method is an appropriate choice for the questions asked in this study, the full study design was not clear as described. it is not clear how the (very small) participant group was chosen for this study. the methods section explains that participants were chosen “through a referral process by public librarians in select libraries in california,” though there is no explanation of how the libraries were chosen or how the librarians referred participants to the researcher (p. 7). in the author’s discussion of study limitations, he states that there may have been participants who “did not see the opportunity to participate in this study,” but it is not explained how they would have learned about the study (p. 18). there is also no discussion, beyond that it was irb approved, as to where the interviews took place or whether participants were compensated for their time. another concern is that the original interviews took place prior to the start of the pandemic, which indicates that the study design was altered to include the second interviews. additionally, the interviews were transcribed and the codebook was developed after both sets of interviews had been conducted, which could have potentially affected how the codebook was created (p. 8).   while the evidence provided from these interviews is of enough interest to drive further studies and should certainly be used by library systems as an impetus to survey their own users and communities at large, the sample size of six is not enough to afford generalization to all populations. the interview questions used in this study could be adapted by other library systems to learn more about their communities and their service needs, particularly in terms of staffing and technology needs. the results can help libraries focus on areas that need improvement and how to remove barriers to access and service.   references   american library association. (2006, july 7). access to digital resources and services: an interpretation of the library bill of rights. http://www.ala.org/advocacy/intfreedom/librarybill/interpretations/digital   letts, l., wilkins, s., law, m., stewart, d., bosch, j., & westmorland, m. (2007). critical review form – qualitative studies (version 2.0). http://www.peelregion.ca/health/library/eidmtools/qualreview_version2_0.pdf     evidence summary   public libraries could better serve older adults by having more programming specifically directed toward them   a review of: bennett-kapusniak, r. (2013). older adults and the public library: the impact of the boomer generation. public library quarterly, 32(3), 204-222. doi: 10.1080/01616846.2013.818814   reviewed by: ann glusker reference/consumer health librarian business, science and technology department the seattle public library seattle, washington, united states of america email: ann.glusker@spl.org   received: 28 jul. 2014    accepted: 9 oct. 2014      2014 glusker. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine whether programming at public libraries in the united states is aimed at older adults, and whether the programs help older adults maintain their health and well-being.   design – web site evaluation.   setting – public libraries in the united states of america.   subjects – the main library of each public library system located in the capital city of each of the 50 states in the united states of america.   methods – a scan of each of the web sites of the selected public library branches was performed by the author, to determine the number of programs specifically directed towards older adults.   main results – the scan of sites indicated that there was very little programming specifically aimed at older adults and their needs. mainly, offerings for older adults took the form of mixing in with adults of all ages. computer technology class offerings were particularly lacking. the majority of libraries had programs to teach how to access library resources and electronic media (although not necessarily aimed at older adults), programs for those who are less mobile, as well as some adaptive   technology for those with sensory disabilities. in addition, the majority of libraries had adult literacy programs, and active collaboration with community organizations.   conclusion – public libraries can do more to develop programs specifically for older adults. they should take into account the wide diversity of older adults’ information and other needs. in particular, they should consider offering programs that focus on technological skills, and also should offer assistive technology for older patrons. there also needs to be more research on the needs of older adults, from the perspective of the patrons themselves and that of library staff.     commentary   the issue of public library programming for older adults is timely and important. the population of older adults is increasing as the baby boomer generation ages, recent changes in the economy have affected retirements, and never has the need for technical skills been more essential. according to the author, prior research has dealt with what older adults would like to see in library services, but has not focused on programs aimed at this group. older adults have different needs than younger adults and children, and even within the older adult population there are variations in health status, ability and movement, and technological comfort. these days, mobile services and large print books are not enough.   there are some significant issues with the methodology of this study that make the results applicable only in a very general sense. in this evidence summary, these issues were systematically assessed, using the critical appraisal checklist by glynn (2006). in the study, the web sites constituting the data sources from which conclusions were made, were surveyed by only one person, who may have had a particular bias. there was no report of comparison assessment by another researcher. the time frame considered was very narrow. in addition, there was no report of how programs listed on web sites were categorized, or any specific terms or vocabulary that were used to assess the applicability of programs to older adults. the author herself states that the library sample selection criteria (the main branches of library systems in united states state capitals) underrepresent rural areas, and demographically, they exclude the largest systems as well. furthermore, only the main branches of the systems were examined, which may limit the sample to downtown areas with low residential density and usage. it may be that programming activities for older adults are concentrated in the areas in which they actually live, where they are served by branch rather than main libraries.   however, in a broader sense, the article has important insights to offer, even if the specific findings are not representative. the report on programming offered nationwide, taken qualitatively, can suggest best practices and new programming initiatives. for example, the discussion of frequency of program offerings may be a new concept to some, and the descriptions of services such as therapeutic and reminiscing kits might spark librarians’ ideas for new programs. in addition, the discussion section relates a number of concrete steps libraries can take to increase their services for older adults. these include ideas for: grant funding, website enhancements such as having a link or tab for older adults, setting aside a computer lab for technology practice, organizing programs aimed at people with health or disability issues, and rethinking possibilities for collaboration with community organizations serving older adults. taken with other recent research which also explores improving services to senior citizens (charbonneau, 2014; perry, 2014), this article’s broader contributions are worth considering.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi:10.1108/07378830610692154   charbonneau, d. (2014). public library websites and adherence to senior-friendly guidelines. public library quarterly, 33, 121-130. doi:10.1080/01616846.2014.910722   perry, c. (2014). information services to older adults: initial findings from a survey of suburban libraries. library quarterly: information, community, policy, 84(3), 343-386. doi:10.1086/676491 evidence summary   survey of the information-seeking behaviour of hospital professionals at a public cancer hospital in greece proves the value of hospital libraries   a review of: kostagiolas, p. a., ziavrou, k., alexias, g., & niakas, d. (2012). studying the information-seeking behavior of hospital professionals: the case of metaxa cancer hospital in greece. journal of hospital librarianship, 12(1), 33-45. doi: 10.1080/15323269.2012.637871   reviewed by: antonio derosa assistant reference librarian memorial sloan-kettering cancer center new york, new york, united states of america email: derosaa1@mskcc.org   received: 22 feb. 2013   accepted: 21 apr. 2013      2013 derosa. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to study the information-seeking practices of hospital staff and weigh the impact of hospital libraries on effective information-seeking.   design – survey questionnaire.   setting – large public cancer hospital in greece.   subjects – the authors surveyed 49 physicians, 43 nursing staff members, 25 administrative staff members, 23 paramedical staff members, and 5 technical staff members, totaling 145 health professionals.   methods – participants were given a questionnaire comprised of five parts: general information (including gender, age, education, position, and professional experience); questions on computer and internet accessibility; questions regarding individual information needs; questions on information-seeking obstacles; and a question regarding the satisfaction with the current degree of information availability in the hospital. the last question was ranked using a 5-point likert scale. each questionnaire was distributed with a cover letter explaining the anonymity and consent of the respondent. hospital members were randomly selected using a number generator and respondents returned completed surveys to the hospital personnel office in a sealed envelope within a specified time frame. the sampled group was representative of the overall population of the hospital.   main results – the authors discuss demographic data of respondents: 65.7% were women; 56.7% were over 40 years old; 29.0% were graduates of higher technological institutes; 28.3% were university graduates; 9.7% held a postgraduate degree; 8.3% had a phd; and 1.4% had only secondary education. as for the remainder of the survey questions: 64% of respondents had access to the internet both at home and at work, while only 8.2% had no access to the internet at all; most respondents noted using the internet for seeking scientific information (83.0%) and e-mail communication (65.3%); the main obstacle respondents noted experiencing when seeking information was the lack of time (3.7 score on likert scale). a lack of information services was second to the time issue with a likert scale score of 3.6.   conclusion – based on the research results, the authors affirmed the invaluable role of hospital libraries. the hospital library can help to eliminate the obstacles faced by healthcare professionals by providing support in the areas of weakness based on the survey results. this can be made possible through the hospital library’s involvement in educational activities, investigation of information technologies, and development of information services to accommodate the difficulties regularly experienced by hospital staff. researchers revealed that funding for hospital libraries in greece is an issue preventing many new initiatives, that there is no association to represent hospital libraries in greece, that the few libraries operating in hospitals in greece are understaffed with no administrative control, and the majority of greek hospitals do not have adequate library facilities. these drawbacks contribute to the information-seeking challenges experienced by greek healthcare professionals.     commentary   this study was conducted as a means to evaluate the importance and shortcomings of the hospital library in relation to its user base. it is important to note that the researchers did not provide details about how they developed their categories for their survey, that no standard framework was used to create and analyze the questionnaire, and that the method of distribution of the questionnaire is unclear.   aside from the ill-transparent survey categories, questions, and framework used, the study at metaxa in greece clearly defined the target population and percentages of respondents. also, by supplying participants with a cover letter stating confidentiality and instructions on returning the survey (in a closed envelope) to non-study researchers, the researchers ensured that both the results and participant identities were blinded to them, though the questionnaire is not completely anonymous as submissions were coded upon receipt. the authors also chose a population representative of the hospital’s total community (20% of total population of medical, nursing, and administrative/technical staff). partnering with the personnel department on this effort was an effective way of achieving an unbiased questionnaire distribution.   the authors did not include a version of the final survey with their paper, meaning readers do not have the opportunity to ensure that the questions asked were clearly stated and able to elicit exact answers from respondents. it would be helpful to see the survey itself while reading the results of the study. the use of the likert scale is evident and this study could be replicated by other institutions if desired (with the exception of the official questionnaire used).   the conclusions reflect the analysis of the study in that the authors comment on the information seeking challenges faced by professionals and give suggestions for librarians to eliminate these obstacles. though these are briefly mentioned by the authors, there is no further detail about implementing certain programs into the hospital library workflow. furthermore, some examination into ways of delivering information services with little administrative support would be helpful.   this research shows the gaps in education and opportunities for training on efficient information seeking in the hospital. new technology allows for remote training as well as more dynamic in-person instruction. hospital librarians should know the tools their clients prefer using and show them how to enhance their experience and get the most out of research. the lack of time reported by many hospital professionals is a sign that the library can offer information services such as performing literature searches and delivering results to clients as requested, for instance. also, evaluating and suggesting key resources that are the most intuitive for hospital professionals is a way of showcasing the librarian’s special skills and helping the client maximize their time when conducting research.   the appraisal of this information-seeking behaviour study was conducted using the evidence-based library and information practice (eblip) critical appraisal checklist from memorial university of newfoundland (glynn, 2006).     references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi: 10.1108/07378830610692154   news/announcements   eblip seeks editorial intern and copyeditors      2014. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   evidence based library and information practice (eblip) is looking to fill a number of volunteer positions.   we are looking for a new editorial intern to assist our editorial team. ideally candidates will be in place by november 2014 and will commit to serving a two year term.   the role of the intern is to:   provide a final check of proofed copy before publication, using the pre-developed checklist check and edit the item metadata to ensure title, author, and abstract correspond to the submitted manuscript assist the editor-in-chief with calls for papers, or calls for volunteers contact potential authors, asking them to consider submission to the journal assist with investigation and implementation of new projects related to the journal participate in monthly editorial team meetings   the ideal candidate will be an mlis student interested in evidence based practice and research methods, and who possesses strong attention to detail. the position requires dedicated time on a regular basis, and it is therefore essential that interested persons ensure available time to devote to this position prior to applying. it is estimated that the workload is approximately 10 hours per month.   interested persons should send a statement of interest, indicating the areas of strength they would bring to the role, as well as a brief resume to alison brettle, editor-in-chief, at a.brettle@salford.ac.uk by 1st october 2014.   specific queries about the role should be addressed to michelle dunaway, production editor, at mdunawa@gmail.com.   evidence based library and information practice (eblip) is looking for two new copyeditors to join our copyediting team. ideally candidates will be in place by november 2014, in time to begin work on the december issue.   the role of copyeditors is to:   ·         accept or reject copyediting requests in timely manner ·         utilize eblip copyediting guidelines, alongside apa style guidelines ·         ensure that proper grammar and spelling conventions are addressed ·         verify citations and references to the extent possible with available bibliographic resources, and advise editor of any not checked   ·         meet copyediting deadlines set by editors ·         identify inconsistencies or changes in format/style as necessary   the ideal candidate will have copyediting or other detail-oriented editing experience, possess working knowledge of apa style, and have an interest in evidence based practice.   the position requires dedicated time to ensure eblip publishing deadlines are met, and it is therefore essential that interested persons ensure available time to devote to this position prior to applying. it is estimated that the workload is approximately 5-10 hours per issue, and much of this work falls over a one-month period, on a quarterly basis, prior to the publication of each issue.   interested persons should send a statement of interest, indicating the areas of strength they would bring to the role, as well as a brief resume to alison brettle, editor-in-chief, at a.brettle@salford.ac.uk by 1st october 2014.   specific queries about the role should be addressed to richard hayman, lead copyeditor, at rhayman@mtroyal.ca. applicants for these positions will please note that evidence based library and information practice is a non-profit, open access journal and all positions are voluntary and unpaid.   about the journal:   published quarterly, and hosted by the university of alberta, this peer-reviewed, open access journal is targeted at all library and information professionals interested in an evidence based model of practice. by facilitating access to librarianship research via original research articles and evidence summaries of relevant research from the library literature, evidence based library and information practice will enable librarians to practice their profession in an evidence based manner.   please visit the evidence based library and information practice web site (http://ejournals.library.ualberta.ca/index.php/eblip) for more information about the journal.   microsoft word art_grimmond.doc evidence based library and information practice 2006, 1:2  3 evidence based library and information practice     article    rehabilitating the stroke collection      mary grimmond  northern sydney central coast area health service  gosford, new south wales, australia  e‐mail: mgrimmond@nsccahs.nsw.gov.au    sharna carter  northern sydney central coast area health service  gosford, new south wales, australia    susanne lewis  northern sydney central coast area health service  gosford, new south wales, australia  e‐mail: slewis@nsccahs.nsw.gov.au      received: 30 november 2005    accepted: 4 may 2006      © 2006 grimmond, carter, and lewis. this is an open access article distributed under the terms of the  creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits  unrestricted use, distribution, and reproduction in any medium, provided the original work is properly  cited.    abstract    objective ‐ the aim of this project was to complete an analysis of monograph and  audiovisual items held in the central coast health service (cchs) libraries and containing  information relevant to the treatment of acute stroke. acute stroke is treated by  multidisciplinary teams of clinicians based at two hospitals within the cchs. the adequacy  of the library collection was measured by subject coverage and age.    methods ‐ the methodology used consisted of three main steps: a literature review; design,  administration, and analysis of a questionnaire to members of the cchs acute stroke team;  and an analysis of the libraries’ collections. the research project utilised project  management methodology and an evidence based librarianship framework.    results ‐ the questionnaire revealed that electronic resources were by far the most  frequently used by participants, followed in order by print journals, books, interlibrary loan  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:2  4  articles, and audiovisual items. collection analysis demonstrated that the monograph and  audiovisual collections were adequate in both scope and currency to support the  information needs of acute stroke team members, with the exception of resources to  support patient education.    conclusion ‐ the researchers developed recommendations for future collection  development in the area of acute stroke resources. conducting this project within the  evidence based librarianship framework helped to develop library staff members’  confidence in their ability to make future collection development decisions, informed by the  target group’s information needs and preferences. the collection analysis methodology was  designed to be replicated, and new specialist groups within the client base of the library will  be targeted to repeat the collection analysis process.     objective    the central coast health service (cchs)  library in new south wales, australia,  provides information services to meet the  clinical, research and learning needs of the  central coast sector (803 beds) of northern  sydney central coast health (nscch). the  primary client groups include the cchs  staff (3,665 fte) and health students based  at cchs. the main site, gosford hospital, is  a teaching hospital of the university of  newcastle, australia.      acute stroke patients treated in the central  coast health service specialist stroke units  are seen by a multidisciplinary team of  clinicians including doctors, nurses,  physiotherapists, nutritionists, speech  pathologists, and occupational therapists,  collectively known as the acute stroke  team.     the hospital library collection is expected to  provide adequate resources to support all  these specialist areas. library staff set out to  answer the question:     does the central coast health service library’s  monograph and audiovisual collection, in the  context of all information sources currently  available, adequately meet the information needs  of the multidisciplinary acute stroke teams at  gosford and wyong hospitals, as measured by  subject coverage, and publication age?     the aim of this small project was to  complete an analysis of monograph and  audiovisual items providing information  relevant to the treatment of acute stroke that  are held at the central coast health  libraries.    literature review    the research team conducted database  searches using all locally available  information science and multidisciplinary  databases, including library and  information science abstracts (lisa),  emerald, expanded academic,  sciencedirect, and proquest 5000. in  addition, researchers hand‐searched several  library science journals: hypothesis, health  information and libraries journal, the  journal of academic librarianship and the  journal of the medical library association.    a combination of the following search terms  was used in the search strategy:    • multidisciplinary  • interdisciplinary  • library (truncated)  • collection development  • collection assessment     evidence based library and information practice 2006, 1:2  5 initially the literature search also included  the search terms ‘stroke’, ‘cerebrovascular  disease’, and ‘aphasia’.  however when  these terms were combined with either  ‘collection development’ or ‘collection  assessment’, no relevant results were  obtained.    the literature review also revealed a lack of  published material on assessment of a  multidisciplinary collection.  although there  is a large amount of material in the library  and information science literature on  collection development, the database  searches did not identify any articles that  specifically discussed developing or  assessing a collection to support a  multidisciplinary health team such as the  acute stroke teams.    jacoby et al. developed a collection  development methodology to assess print  and digital resources for social work.  jacoby’s research was useful in formulating  the methodology for the central coast  health service project, because social work  is an interdisciplinary area of practice,  drawing from a wide range of fields. as  such, it was a useful model for collection  assessment in the multidisciplinary clinical  field of acute stroke treatment. jacoby’s  research team of liaison librarians used a list  of forty‐five library of congress subject  headings they selected to reflect the subject  needs of the social work program at their  university. (395)    in their article on interdisciplinary collection  evaluation, dobson, kushkowski, and  gerhard noted that interdisciplinary fields  pose particular problems for collection  evaluation (282).  because traditional  methods of collection evaluation are based  on tightly focused subject areas, evaluation  methods based on call number analysis may  not be suitable for inter‐ or multidisciplinary  fields. for interdisciplinary collection  analysis, dobson’s group recommended  using keyword and subject headings to  identify relevant areas of the collection (282).  this use‐centred and materials‐centred  evaluation method was adapted for use by  the cchs research team.     chu’s research on librarian‐faculty relations  in collection development provided a  theoretical background on lateral relations,  defined as relationships and communication  “in a horizontal layer of an organisation  between people who do not share the same  vertical hierarchy” (15). this provided a  useful model to describe the library staff’s  interaction with the acute stroke teams  while carrying out this project.    methods    the research team utilised project  management principles to ensure successful  and timely completion of the project. the  cchs library uses a standard project  methodology for all projects.  the  methodology sets out the stages required for  a project, including identification of  fundamental variables (time, cost and  performance); stakeholder analysis; scoping  the project; identifying the project  deliverables; specifying boundaries,  constraints, and anticipated outcomes; and  constructing a work breakdown structure.    incorporating the project methodology  within the framework of evidence based  librarianship (ebl) added to the value of the  research.  the challenge of mapping the ebl  process to the work breakdown of the  project methodology was worthwhile, in  that it introduced a greater rigour into the  standard project methodology.              evidence based library and information practice 2006, 1:2  6     standard project methodology    evidence based librarianship framework  identify and document parameters of the  project.  formulate an answerable question.  define the scope of the project and perform a  literature review.  search the library and information science  literature.  administer the questionnaire.  identify the libraries’ holdings on acute  stroke.  analyse the libraries’ collections.  identify sources of evidence.  analyse the results.  appraise the evidence.  develop recommendations.  apply the evidence to everyday practice.  document and evaluate the methodology.  evaluate performance.  disseminate results to colleagues by  preparing a conference presentation or  publishing a paper.   disseminate the results to other librarians.    table 1: standard project methodology and ebl framework        in particular, the evidence based  librarianship process emphasises the  importance of both evaluating performance  and disseminating results. for this reason,  the collection analysis methodology was  documented to facilitate future collection  assessment projects. the methodology was  designed to be replicated and is freely  available to colleagues for evaluation and  dissemination.    the methodology consisted of three main  stages: the literature review described above;  design, administration, and analysis of a  questionnaire to acute stroke team  members; and collection analysis based on a  keyword search of the catalogue.    questionnaire design and administration    a questionnaire was designed and  administered to multidisciplinary acute  stroke team members to obtain qualitative  data on recent information needs and to  assist in determining the required subject  coverage of monographs and audiovisual  items for collection analysis.  ethics  permission was not required for this project  as it did not involve patients or members of  the public.  the researchers and acute  stroke team leaders agreed that participant  confidentiality was not an issue.    the questionnaire was designed with the  aim of optimising both the response rate  and the usefulness of the information  obtained for analysis. the questionnaire was  kept as succinct as possible, and consisted of  five questions on two pages. the first  question reviewed frequency of monograph  use compared to other information  resources available to the stroke team. the  remaining four questions covered recent  information needs of participants, keywords  used by participants for online information  searches of resources recommended by  participants as being useful in the treatment  of acute stroke, and additional comments  from participants.      the questionnaire was discussed and  demonstrated in person to the acute stroke  evidence based library and information practice 2006, 1:2  7 team leaders at both gosford and wyong  hospitals to ensure the willing cooperation  of these key clinicians. team members  provided participants’ contact details, and  they encouraged the clinicians to participate  in the survey.  questionnaires were emailed  to team members and follow up email  contact was made with participants,  encouraging completion of the  questionnaire during the second week after  initial distribution. participants were  encouraged to contact project leaders with  questions about the questionnaire or the  project.     the questionnaire results were compiled  and entered into an excel spreadsheet for  ease of analysis. the researchers identified  common themes in the responses, and they  made recommendations for both immediate  actions and future initiatives in collection  development.    collection analysis    the next stage of the project was to compile  a list of relevant monographs in the  collection using a combination of keywords  identified by library staff and questionnaire  respondents and terms derived from  medical subject headings (mesh).    the task of identifying items in the  collection relevant to the information needs  of the multidisciplinary acute stroke teams  was challenging. it was not possible to  review the holdings within a targeted  dewey decimal classification range due to  the multidisciplinary nature of acute stroke  diagnosis and treatment. detailed  examination of bibliographic records in the  catalogue revealed that not all records had  been assigned subject headings. as a result,  the librarian researchers had to rely on  keyword searching to locate relevant items  in the collection. a review of chapter  headings from the identified monographs  prompted the inclusion of a range of  keywords based on symptoms and  treatment of stroke.   prior to administration of the questionnaire,  cchs librarians identified items relevant to  acute stroke by using the following  keywords:     • (stroke or cerebrovascular) and  (assessment or diagnosis or imaging or  therapy or treatment or rehabilitation  or recovery or medicine or nursing or  vision or speech or nutrition or  psychology or social or cognition or  movement or paralysis or ataxia)    • hemiplegia    • apraxia    • aphasia    utilising participants’ responses from the  questionnaire, the following keywords were  later added, increasing the holdings list  considerably:    • (brain and anatomy) or neuroanatomy    • (patient or client) and (education or  training)    some items retrieved from the searches with  these keywords were not relevant to acute  stroke, and those were not included in the  final holdings list. subjective decision  making was required to omit these holdings.     records identified by keyword searches of  the innovative millennium catalogue were  exported to an excel spreadsheet. this was  time consuming, however, the spreadsheet  allowed the librarians a means of providing  feedback to participants regarding holdings  following completion of the questionnaire.  the spreadsheet included the following  information about each item: title, author,  year of publication, barcode number, call  number, and name of holding library.  evidence based library and information practice 2006, 1:2  8 the listed items were reviewed by year of  publication, scope, and subject coverage;  results were compared with the holdings of  other libraries participating in the shared  catalogue. these libraries include several  nsw health system hospital libraries and a  university library serving the region’s  medical, nursing, and allied health students.    the collection analysis methodology was  documented to facilitate future collection  assessment projects, both within the cchs  library and in other libraries.     results    questionnaire    in total, 17 questionnaires were distributed  and 10 were returned, giving a response rate  of 58.8%. questionnaires were returned by  each of the specialty groups in the  multidisciplinary teams except social  workers.    question 1: please indicate the type of  information resources you use in relation to  your work with the acute stroke team.  electronic resources were the resources most  frequently used by participants, followed in  order by print journals, books, interlibrary  loan articles, and audiovisual items. these  results suggested that an online resources  subject guide for neurology on the library’s  intranet site would be a useful tool for the  acute stroke team members.    question 2: please describe your information  needs in relation to acute stroke over the last  year. (i.e., specific treatment techniques,  anatomy and physiology, evidence based  studies, refresher information via general  textbooks )     the majority of responses related to the  specialty skills of team members (for  example ’exercises and strategies for  management of dysphasia’ was most  relevant to speech pathologists).  however,  several respondents identified a range of  topics as being of current interest, including  evidence based medicine, anatomy and  physiology, patient education (visual  displays, 3d models, instructor guides), and  current treatment techniques.  one  respondent specifically emphasised that  journals were more useful than books and  that online resources were particularly  useful.  another respondent stated that  internet‐based language therapy resources  were of particular interest.        resources used  frequently  occasionally  rarely  never  electronic resources  8 (80%)  1 (10%)  1 (10%)  0 (0%)  print journals  6 (60%)  3 (30%)  1 (10%)  0 (0%)  books  1 (10%)  5 (50%)  4 (40%)  0 (0%)  audiovisual items  1 (10%)  3 (30%)  4 (40%)  2 (20%)  interlibrary loan books  0 (0%)  2 (20%)  6 (60%)  2 (20%)  interlibrary loan  journal articles  1 (10%)  4 (40%)  4 (40%)  1 (10%)    table 2: types of information resources used      evidence based library and information practice 2006, 1:2  9 question 3: when searching for resources (of  any type), what keywords would you  choose?     the selected keywords tended to reflect the  specialty skills of the respondents. many of  the selected keywords were too specific for  use in a general library catalogue search,  being more valid for use in bibliographic  database searches. this may reflect the  respondents’ most frequently used  information sources (i.e. electronic / print  serials in preference to monographs). more  respondents identified the term ‘stroke’ as a  keyword, as opposed to the mesh term  ‘cerebrovascular accident’. additional  keywords and phrases not previously  identified by the project leaders’ pre‐ questionnaire included ‘educational tools’,  ‘design of educational resources’, ‘patient  education’, ‘general anatomy’, ‘physiology’,  and ‘neuroanatomy’.     question 4: please list any specific titles  which you feel are valuable in providing  information on acute stroke.    the response to this question was limited,  with almost half of the respondents not  offering a response to this question.  of  those that did respond, many suggested  titles that were already available in some  format within the library collection.      question 5: additional comments    four additional comments were provided  by respondents, although six of the ten  respondents offered no comments at all.  one respondent thanked the librarians for  their efforts to provide updated stroke  resources. other comments from  respondents included:    • lack of time to pursue educational / non‐ clinical tasks  • online information resources are most  useful   • journals relating to stroke and nutrition  are most useful    the questionnaire results helped to identify  additional keywords used to analyse the  collection.  results also demonstrated that  some respondents were not searching the  library catalogue effectively, because they  recommended purchase of some items  already held in the collection.     librarians developed a feedback package  and distributed it to questionnaire  respondents. the package included a list of  the current holdings identified as relevant to  acute stroke; a summary of the  recommendations obtained from the  questionnaire and collection analyses; and  an education package on searching the  catalogue by keyword and subject headings  specifically tailored to the respondents’  needs.    collection analysis     results of the collection analysis indicated  that the monograph collection provided  broad subject coverage and adequate  currency when compared with other  libraries’ holdings. there was a noticeable  gap in the availability of simple and current  information targeting acute stroke suitable  for use in patient or career education.     discussion  use of the questionnaire had several benefits  – education of the recipients about the  subject; promotion of the library as a  valuable resource; and confirmation at a  primary level of the broad or narrower focus  of subject searching and resource use within  the library collection by the stroke team.     originally, the questionnaire was to be  administered face to face with participants  to allow for questions to be answered  evidence based library and information practice 2006, 1:2  10 immediately or to provide basic library  training, if required. however, after meeting  with the gosford and wyong hospitals’  acute stroke team leaders, it was evident  that this would not be practical due to time  restraints on clinicians.  the team leaders  indicated that emailing questionnaires was  likely to produce the highest response rate.    the main problem encountered by the  researchers was the clinicians’ lack of time  to complete a questionnaire.  this is an  ongoing problem for librarians undertaking  research with clients working in the health  field.  the small sample size was  determined by the size of the acute stroke  teams.  while this meant that the  researchers were using a sample of  convenience, it gave the project clear  purpose and focus to work with these  multidisciplinary patient‐centred teams and  provided useful information for the library.    conclusion    although this was a small project, the  results of the acute stroke team survey and  subsequent collection analysis provided  valuable evidence on collection  development and information needs in the  area of multidisciplinary acute stroke  treatment.  the results indicate that there  was some use of print library resources in  the area of acute stroke treatment, but that  online resources were the preferred format  for information access by this client group.  there was a noticeable gap in the  availability of resources to support patient  education.    the following recommendations arose from  this project for ongoing collection  development in the multidisciplinary area of  acute stroke treatment:    • continue to purchase the latest editions  of core neurology texts in print format  to maintain the scope and currency of  this part of the collection. it is not  necessary to increase the total number of  print resources.     • purchase a small number of new titles to  provide simple information to support  patient information and education in the  area of acute stroke.    • notify acute stroke team members of  new library purchases using an email  notification system and a link on the  library’s intranet site.    • review access to print and online  journals that focus on the subject areas  of stroke and neurology, with the aim of  increasing access to online resources.    • include an online neurology resources  subject guide on the library’s intranet  site.    conducting this project within the  framework of evidence based librarianship  means that library staff are now confident  their ongoing collection development  decisions regarding acute stroke resources  will be informed by evidence of the target  group’s information needs and preferences.   although undertaken in a small, specialist  area, relevant to a specific team on the  hospital staff, the collection analysis  methodology was designed to be replicated.  new specialist groups within the client base  of the library will be targeted to repeat the  collection analysis process.     further evaluation may be possible through  feedback from colleagues who choose to use  the methodology in their own libraries.  evaluation obtained from the acute stroke  team leaders at gosford hospital and  wyong hospital indicated that this project  was useful to their teams in increasing  awareness of the library and its services.  they also learned more about the library  evidence based library and information practice 2006, 1:2  11 resources available to support clinical  practice in the area of acute stroke.      acknowledgement    this paper is based on a poster presented at  the 3rd international evidence based  librarianship conference held in brisbane,  australia (october 2005).  winner of the  qut best poster award.  see:  <http://conferences.alia.org.au/ebl2005/confe renceawards.html>    works cited    chu, felix t. ʺlibrarian‐faculty relations in  collection development.ʺ journal of  academic librarianship 23.1 (1997): 15‐ 20.                                                          dobson, cynthia, jeffrey d. kushkowski,  and kristen h. gerhard. ʺcollection  evaluation for interdisciplinary fields:  a comprehensive approach.ʺ journal of  academic librarianship 22.4 (1996): 279‐ 84.    jacoby, beth e., jane murray, ina alterman,   and penny welbourne. ʺresource  selection for an interdisciplinary field:  a methodology.ʺ journal of the medical  library association 90.4 (oct. 2002):  393‐99. 4 june 2006  <http://www.pubmedcentral.gov/picren der.fcgi?artid=128955&blobtype=pdf>.        http://conferences.alia.org.au/ebl2005/confe http://www.pubmedcentral.gov/picren evidence based library and information practice evidence based library and information practice 2010, 5.4 105 evidence based library and information practice evidence summary adherence to rusa’s guidelines for virtual reference services is below expected in academic libraries a review of: platt, j. & benson, p. (2010) . improving the virtual reference experience: how closely do academic libraries adhere to rusa guidelines? journal of library and information services in distance learning, 4(1-2), 30-42. reviewed by: annie m. hughes reference librarian, herman ostrow school of dentistry university of southern california los angeles, ca, united states of america email: received: 14 sept. 2010 accepted: 20 nov. 2010 amhughes@usc.edu 2010 hughes. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to evaluate the quality of academic libraries’ virtual reference services and measure compliance to the reference & user services association’s (rusa’s) guidelines for virtual reference & user services. design – qualitative research study evaluating virtual reference chat sessions using rusa’s guidelines for virtual reference & user services. setting – virtual reference environments in public academic libraries in the united states. subjects – twenty virtual reference providers from public academic libraries. methods – initially researchers selected 1 academic library out of each of the 50 states to evaluate for quality virtual chat reference services, however because of factors including time and availability of virtual chat services to unaffiliated institutions; the sample included only 20 academic libraries. after selecting the 20 academic libraries for evaluation, researchers posed as virtual chat reference patrons using emails and aliases that had no affiliation to any particular institution. researchers then asked the librarian or library mailto:amhughes@usc.edu� evidence based library and information practice 2010, 5.4 106 staff a two-part question making sure to leave out any library jargon or anything that would lead the virtual chat reference operator to recognize that they are also affiliated with a library or library school. using the rusa guidelines for virtual reference & user services, researchers then evaluated their virtual chat reference experience for the following: approachability; interest; listening/inquiring; searching; follow-up; suggests patron call or visit the library. main results – when evaluated for jargonfree websites and overall usability in finding all types of reference services, 80% of the library’s websites were easy to use and jargon free, reflecting overall high usability. evaluation of library staff’s ability to maintain “word contact” by writing prompts to convey interest in the patron’s question left some room for improvement. sixty percent of researchers coding their virtual reference experience thought the level of contact was below expected. information regarding question and answering procedures, question scope, types of answers provided and expected turnaround time for questions was only available in 30% of examined websites. thirty-five percent of researchers felt that library staff members gathered enough information to answer the question without compromising privacy, however, 25% thought that staff members gathered a very small amount of information on the patron’s need, although privacy never felt compromised. when researchers evaluated the library staff member on their ability to explain how to utilize resources properly, 50% thought the instruction provided was below average. although 15% believed they received “superior instruction.” seventy-five percent of the researchers were not asked by a library staff member if the question received an adequate answer, 50% of reference transactions library staff did not consult a librarian or expert, and in 55% of transactions the staff member did not suggest that the patron visit or call the library. conclusion – while the researchers received some valuable information about the need to improve virtual reference services in academic libraries, there were some flaws in their research. the question they developed was almost too clear and made it difficult for the individual answering the chat reference to adequately perform a reference interview or ask probing questions. it is possible that because researchers carefully planned out their question they set themselves up to create an interaction that would not normally occur in a virtual chat reference environment. also, because researchers were unable to evaluate what was occurring in the environment surrounding the virtual chat reference providers it was impossible to make a judgment on the speed or length of the interaction. the researchers did come away from the study with results that point to a need to utilize the rusa guidelines in order to conduct effective reference interviews, maintain appropriate contact with the user when engaging in chat reference, provide instruction and point patrons to quality resources as well as consult an expert on the topic if needed. they surmised that if libraries utilized these guidelines, virtual chat reference services would be improved. commentary the researchers identify limitations of the study and give those who would like to explore virtual reference quality further a few ideas to consider. the two-part reference question posed by the users to the library staff did not prompt an in-depth reference interview because the question was very clear. evidence based library and information practice 2010, 5.4 107 if the question posed was ambiguous it is possible that the library staff would have asked more questions of the user. by selecting only one question to ask of the virtual chat reference providers, the data available was limited. if the researchers had utilized several questions of various levels of difficulty, perhaps quality of service would be easier to clearly evaluate. researchers also evaluated their virtual chat reference experience by looking at whether or not library staff or librarians consulted experts to effectively answer the question. experts need not be consulted for “manageable” or simple reference questions, hence another reason to use several questions of varying levels of difficulty. another important aspect missing from this study is that it is not known how much training library staff had on providing virtual reference services. also, because the user had to appear to be a regular patron who is unaware of library jargon, the level of expertise of the respondent could not be recorded. one of rusa’s guidelines discusses the importance of maintaining “word contact,” but what researchers did not consider is the type of chat client used. chat clients that do not provide sound alerts or other prompts could affect responsiveness especially if another patron interrupts the service. usability of the chat client and the library staff’s comfort level and knowledge of how to use the client could affect maintenance of contact. while there are identified limitations provided by the researchers and the sample size was not particularly representative of the academic libraries in the united states, the study does reflect on what aspects of virtual reference need improvement and provides some evidence of what users expect from virtual reference service. unfortunately, it would be difficult to point other researchers to this paper as an example since there are so many flaws with the research. there are other higher quality studies available that evaluate virtual chat reference services against the rusa guidelines. / evidence based library and information practice evidence summary evidence based library and information practice evidence based library and information practice 2010, 5.4 132 evidence based library and information practice news/announcements eblip6 news 2010. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. the eblip6 conference will take place in the u.k. at the university of salford, 27-30 june 2011. the call for abstracts has now closed and the programme committee will be informing potential participants of the outcome of their submission by mid january. the programme is beginning to take shape with keynotes from professor martin hall (university of salford), dr hazel hall (library and information science research coalition), dr ross todd (rutgers state university, new jersey) and professor peter brophy (recently retired from the centre for research in library and information management, manchester). andrew booth will be closing the conferences with his perspectives on eblip, eblip6 and eblip in the future. we are also investigating the possibility of bringing an interactive element to the conference, enabling those who are unable to participate in the full conference, to experience some of the conference online. a number of u.k. organizations will be providing bursaries to enable participants attend the conference and a list of these and more general conference bursaries is available on the frequently asked questions section of the conference website. http://www.eblip6.salford.ac.uk/ to keep up to date with eblip6 news, register your interest on the conference website http://www.eblip6.salford.ac.uk/register.php, join us on facebook, http://bit.ly/99ey96 or follow us on twitter http://twitter.com/eblip2011 we look forward to seeing you at eblip6! alison brettle and maria grant – eblip6 conference chairs http://www.eblip6.salford.ac.uk/� http://www.eblip6.salford.ac.uk/register.php� http://bit.ly/99ey96� http://www.facebook.com/group.php?gid=141896412498612&ref=ts� http://www.facebook.com/group.php?gid=141896412498612&ref=ts� http://twitter.com/eblip2011� / evidence based library and information practice marketing and assessment in academic libraries: a marriage of convenience or true love?   lynne porat head, interlibrary loans and assessment younes and soraya nazarian library university of haifa haifa, israel email: porat@univ.haifa.ac.il      2013 porat. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – this paper describes the process of cooperation between the marketing and assessment teams at the university of haifa in israel, from initial apprehension about working together to the successful marketing of a suite of user studies.   methods – the first step was a formal meeting in which the leader of the assessment team explained the aims of assessment. for each assessment activity, the assessment team submitted a formal request for assistance to the marketing team, conducted team meetings on how to market each assessment, and met with the marketing team to explain the survey and receive their input on how it should be marketed. over a 3-year period, 5 joint activities were undertaken: a 1-day, in-library use survey; a wayfinding study, in which 10 new students were filmed as they searched for 3 items in the library; 5 focus group sessions regarding upcoming library renovations; a libqual+® survey measuring perceptions of service quality among the entire campus population; and an online survey of non-users of the library. the success of the assessment/marketing projects was measured by the response rates, the representativeness of the results, and the number of free-text comments with rectifiable issues.   results – although the response rates were not very high in any of the surveys, they were very representative of the university population. with over 40% or respondents filling in free-text comments, the information received was used and applied in making service changes, including the creation and marketing of additional group study rooms, improved signage, and the launch of a “quiet” campaign. in addition, a “you said – we did” document was compiled that outlines all of the changes that were implemented since the first four surveys were conducted; this document was published on the library’s blog, facebook page, and website.   conclusion – the number of issues that appear in the first “you said – we did” document is a testament to the close and ongoing collaboration between the two teams, from the planning stages of each survey until publication of results and notification of the changes that were implemented.   introduction   the last decade has seen an increase in the interdependence of library marketing and assessment (alire, 2007; wright and white, 2007) – a relationship whose importance was recognized earlier in the private sector by souder (1981) who found that “r & d/marketing interface problems was a major factor in the failure of new products” (p. 67). and it is not uncommon in the private sector for marketing managers to carry out both assessment, or market research, and marketing tasks (baker & hart, 2008). this interdependence was first mentioned in the library literature by john sumsion (2001) when he stated that “’marketing’ and ‘user studies’ may employ different terminology but, in reality, they are two sides of the same coin” (p. 221) and was followed by kearns' (2004) more explicit explanation:   marketing and assessment are converging in libraries…librarians are recognizing that assessment and marketing are intertwined so that libraries cannot be marketed without knowing what users want or need, and that libraries cannot be assessed if users do not know what they can or do offer. (p. 51)   this “symbiotic relationship between marketing and assessment” was further reinforced in a paper given at 2006 library assessment conference by becher & mintz (2007, p. 80).   promoting library assessment activities is not an easy task for most libraries and the difficulty is compounded by a dearth of professional literature on the subject, though one useful publication is “suggestions for marketing your libqual+® survey” (libqual+®, 2010). however, coordinated marketing and assessment efforts prior, during, and after survey administration can lead to increased response rates, greater visibility, and increased library use – as seen at the university of haifa and some american university libraries (becher & mintz, 2007).   since the 1980s, many academic libraries have been aware of the need to market their resources and services “in order to facilitate the achievement of important organizational goals” (spalding & wang, 2006, p. 502). however, library marketing is still not widely practiced outside of the us (enache & simona, 2008), partly due to the substantial effort and budget required to do so effectively. and even in the us where many libraries do market their services, very few have an independent marketing unit – more often marketing is linked to a library development or fundraising department (dewey, 2006). by the same token, library managers are becoming increasingly aware of the importance of assessing their activities in order to accomplish their strategic goals (joubert & lee, 2007). as a result, marketing teams are now faced with the need to market not only their library’s resources and services, but also to promote surveys and other assessment activities, which require the participation and goodwill of users (and non-users); and to publicize the results of these surveys, which may not show the library in an entirely positive light.   goals of university of haifa’s marketing and assessment teams   in 2006, the university of haifa library made a strategic decision to create two teams to address marketing and assessment issues. a six-member marketing team was chosen by the library management with members from each of the main departments. the criteria for inclusion were based on professional abilities and leadership and interpersonal skills. for example, one member was the library’s graphic designer, one was the library’s web site manager, and one was in charge of the blog. the team leader had very good connections within the university, such as with the university’s central public relations department. the team was given the following goals:   •     to promote awareness of existing and new library resources and services •     to increase accessibility, awareness and use of library resources and services •     to increase visibility of the physical and digital library   a year later, a nine-member assessment team – two of whom were also members of the marketing team – was formed. the original team leader was the head of reference; but she retired a year later and the job was assigned to the head of interlibrary loans. this individual had just finished writing a doctoral dissertation in information science and was one of the few librarians within the organization with experience in survey administration and statistics. the other members were talented librarians – most in non-managerial positions – who were familiar with the library’s strategic plan and vision. the team was given the goal to create a “culture of assessment” defined as:   an organizational environment in which decisions are based on facts, research and analysis, and where services are planned and delivered in ways that maximize positive outcomes and impacts for customers and stakeholders. a “culture of assessment” exists in organizations where staff care to know what results they produce and how those results relate to customers’ expectations. organizational mission, values, structures, and systems support behavior that is performance and learning focused. (phipps & lakos, 2004, p. 352)   it was later summarized by covey (2005) as “a set of beliefs, behaviors, and assumptions that drive an ongoing cycle of data gathering, analysis, interpretation, organization, presentation, and use to achieve planned objectives” (p. 83).   •     to assess the extent to which the library is meeting the needs of its customers. •     to assess the extent to which library customers are satisfied with library services. •     to recommend the implementation of changes in the library based on the surveys that are conducted.   initially, neither the marketing nor the assessment teams were very enthusiastic about cooperating with each other, but the mutual benefits of doing so soon became apparent. the marketing team could benefit from the assessment team’s market research which would assist in priority-setting: and the assessment team could benefit from the marketing team’s assistance in promoting its activities. in addition, as both teams were quite large and consisted of members who had primary job responsibilities outside of marketing and assessment, the workload and expertise could be distributed among more people. some of the difficulties encountered and how we overcame them are outlined below.   reservations about conducting joint marketing/assessment projects   the main reason for the assessment team’s reservations about working with the marketing team stemmed from a fear of loss of autonomy regarding decision-making. for example, decisions about the wording on invitations, posters, blog posts, etc. would be made in conjunction with the marketing team. there was also a concern about the practicalities of working with so many people across two large teams.   there were several reasons why the marketing team was concerned about marketing assessment activities:   first, marketing is a time-consuming, expensive and labor-intensive process; and it is far more difficult to market assessment activities that have no immediate benefits to the user, than it is to market essential library resources, services, and products.   second, it is necessary to enlist the goodwill of users who need to be convinced of the future benefits of devoting their time to a survey and inevitably need to be offered expensive incentives.   third, there was a belief among some librarians that assessment is redundant (unlike marketing which was unanimously accepted as necessary). for example, when the university of haifa library carried out its wayfinding study, some librarians commented that they knew what the problems were, so why bother conducting a survey? and prior to conducting an in-library use and libqual+® surveys, some librarians argued that the majority of survey respondents are either gripers or hold exceptionally pro-library views – “library cheerleaders or disgruntled users” as kearns (2004, p. 52) describes – so why bother surveying them?   fourth, there was a belief that marketing would not increase survey response rates for some assessment activities. for example, a non-user survey conducted by the library was believed to be redundant by several librarians because it should be in a student’s own interest to use the library and not the library’s responsibility to market to disinterested students.   fifth, the difficulty of publicizing negative results was cited. for example, libqual+® results from 2009 showed a gap between the perceived and desired levels of noise in the library, which proved challenging to publicize as there was going to be even more noise during upcoming renovations.   sixth, the results of library assessment activities may show that current marketing activities are deficient. recent studies show that more than 40% of students lack knowledge about library services and resources (george et al., 2006; washington-hoagland & clougherty, 2002); and several of the university of haifa library’s surveys showed that many of the desired services already existed, such as home delivery of books and articles, electrical sockets for laptop computers, etc.   seventh, there are very few courses in library schools on marketing or assessment, so librarians had to learn these skills on the job. at the university of haifa library, none of the marketing or assessment librarians had received formal training apart from a few professional development courses, guidance from a faculty member, and training opportunities offered by the association of research libraries (arl).   finally, the difficulty of depending on the efforts of others was cited. for example, a graphic designer and the university’s central public relations staff had many other commitments and time constraints which would inevitably delay progress.   despite all of these reservations, several assessment projects were successfully marketed during the past three years, following a request for assistance from the assessment team and with the approval of the library administration.   the first step was a formal meeting in which the leader of the assessment team explained the aims of assessment to the marketing team. next, an assessment logo with the slogan “you can impact the library!” was created, which appears on all assessment materials, followed by the creation of a library assessment website (http://lib.haifa.ac.il/libinfo/assessment/index.php/en/). for each assessment activity the assessment team submitted a formal request for assistance from the marketing team, conducted team meetings on how to market each assessment, and met formally with the marketing team to explain the survey and receive their input on how it should be marketed. the two teams never met in person, but there was a lot of e-mail correspondence to/from all team members. many of these e-mail discussions were lively and agreement was not always reached easily. for example, the marketing team usually wanted catchy attention-getting phrases, whereas the assessment team preferred messages that conveyed the true intention of the surveys.   in may 2008, following the advice of arl consultants, steve hiller and martha kyrillidou, the university of haifa library ran its first assessment, an in-library use survey, which was a hebrew translation of the university of washington’s one-page questionnaire on what users did in the physical library on a particular day. the survey was distributed during two-hour periods for two weeks in the middle of the second semester. posters were hung up around the campus; updates and photographs were posted on the library blog, the library and university web sites, and on the plasma television screens within the library. balloons were hung at the entrance to the library and student employees with library t-shirts approached potential participants and handed out questionnaires and sweets. boxes decorated with the assessment logo were used for returning completed forms.   after the survey closed, a summary of the results was published on the library blog, on the library’s facebook page, and presented at library staff meetings and at national and international conferences.   the next joint marketing project was a wayfinding study, conducted in november-december 2008. attractive invitations were sent by e-mail to a sample of 110 new students who were asked to choose a convenient day and time to participate in a study in which they would be filmed in return for $15. of the 20 positive responses 10 students turned up on the assigned day. the participants were asked to find three items in the library and were observed attempting to complete the assignment. after the assessment team viewed all the sessions, the results were transcribed, analyzed and published as a report on the library blog and discussed at staff meetings.   from march-june 2009 five focus group sessions on the subject of the upcoming library renovations were conducted. attractive invitations were sent by e-mail to a sample from five specific user groups: ba, ma, and phd students; research assistants; and faculty members. fifteen dollars were offered to each student participant as an incentive.   in may 2009, libqual+® was administered for the first time. marketing began by sending a letter to the entire population of 22,000 students and faculty. posters were hung up around the campus, and the survey was advertised on the library and university web sites. student employees roved the campus with laptops and offered assistance in completing the survey. a raffle of food, drink, and bookshop vouchers were offered as incentives.   a more recent assessment was a non-user survey, conducted in may 2010. a one-page, online questionnaire was sent by e-mail to 5,000 students and faculty who had not borrowed a book or accessed the electronic offprints’ database during the previous academic year via qsia (question sharing, information and assessment), the library’s software for creating online exams, assignments and surveys. the patterns of use questionnaire aimed to increase the library’s understanding of how people obtain academic information so that respondents would not know that they had been targeted as non-users. no incentive was offered and no marketing was done due to problems targeting the desired group, such as obtaining active e-mail addresses. as expected, the response rate was very low, possibly due to the lack of marketing, but more likely because of technical problems during the first few days of survey administration and the inherent difficulty of getting non-users to respond to such a survey.   measuring success   the success of the assessment/marketing projects was measured by: the response rates, the representativeness of the results, and the number of free-text comments with rectifiable issues. although the response rates were not very high in any of the surveys, they were very representative of the university of haifa population; and with over 40% of respondents filling in free-text comments the information received was used and applied in making service changes. based on the survey results and comments received, several changes were implemented, such as the creation and marketing of additional group study rooms, improved signage, and the launch of a “quiet” campaign – all of which have improved services for users.   for the quiet campaign posters, screen savers for the public workstations, and colorful bookmarks were produced, which seemed to have a very positive effect on the noise level in the library. in addition, a “you said – we did” document was compiled which outlines all the changes that were implemented since the first four surveys were conducted and was published on the library blog, the library’s facebook page, and the library website. some of the changes were already in the planning stages before the surveys were conducted, and these were more easily implemented once confirming information was received. however some issues such as the widespread dissatisfaction with the noise level were not previously known and the quiet campaign was implemented as a direct result of the in-library use survey. the high priority given to implementing these changes is directly connected to library management’s vision of improving service and putting the user at the center.   the following list shows examples from the 2010 “you said – we did” document which was published on the library website:   •     quieter library: a member of the library staff now roves the library during peak hours and asks people to turn off their mobile phones •     group study areas: an additional group study room was opened with desks, computers, and a drinks machine was installed •     help locating books on shelves: an internal telephone helpline was installed in various places in the library, and temporary library staff members now wear purple shirts for easy identification if help is need among the stacks •     improved signage in the library: signs were added at the entrance/exit and to the media and periodicals departments that help orientation within the stacks •     entrance to the library with bags: entrance with bags is now permitted together with plans for installing lockers in the new library wing •     one place to search for all library information: the federated "onesearch" database was launched allowing simultaneous searching for books, articles, images, maps, videos, and databases. •     easy to use library web site: a new user-friendly web site was launched •     simplified remote connection to the library: a web-based single sign-on method has replaced the old method of individual proxy authentication •     one-on-one reference services: advanced reference services are now available for graduate students and faculty   in addition, the following improvements were made:   •     more electrical outlets for laptops •     more public workstations and loaner laptops •     a more liberal loan policy regarding dvds •     more access to full-text articles in hebrew •     widespread notification of library resources via electronic notice boards, monthly email newsletters, the library blog, facebook, and twitter •     drinks machines were installed •     a more comfortable and welcoming physical space •     shorter queues at the reference desk •     reduced cost of interlibrary loans •     better assistance with technical questions   conclusion   the number of issues that appear in the first “you said – we did” document is a testament to the close and ongoing collaboration of the assessment and marketing teams, from the planning stages of each survey until publication of the results and notification of the changes that were implemented. these activities have also been supplemented with additional assessment efforts, most notably an in-house survey and usability testing. in this respect, the relationship is not just one of convenience; it is based on true love.     references   alire, c. a. (2007). word-of-mouth marketing: abandoning the academic library ivory tower. new library world, 108(11/12), 545-551. doi:10.1108/03074800710838272   baker, m. j., & hart, s. (2008). the marketing book. oxford: butterworth-heinemann.   becher, m., & mintz, m. (2007). a leap in the right direction: how a symbiotic relationship between assessment and marketing. in f. defranco, s. hiller, l. j. hichliffe, k. justh, m. kyrillidou, j. self, & j. stein (eds.), proceedings of the library assessment conference: building effective, sustainable and practical assessment (pp. 69-82). washington, dc: association of research libraries.   covey, d. t. (2005). using data to persuade: state your case and prove it. library administration & management, 19(2), 82-89. retrieved 28 may 2013 from http://repository.cmu.edu/lib_science/62   dewey, b. i. (2006). fund-raising for large public university libraries: margin for excellence. library leadership and management, 20(1), 5-12. retrieved 28 may 2013 from http://journals.tdl.org/llm/index.php/llm/article/download/1608/888‎   enache, i., & simona, g. (2008). marketing and evaluation in libraries. reference services.the new information and communication technologies. library and information science research (studii de biblioteconomie şi ştiinţa informării), 12, 92-99. retrieved 28 may 2013 from http://www.lisr.ro/en12-enachegheorghe.pdf   george, c. a., bright, a., hurlbert, t., linke, e. c., st clair, g., & stein, j. (2006). scholarly use of information: graduate students' information seeking behaviour. information research, 11(4), 1-27. retrieved 28 may 2013 from http://informationr.net/ir/11-4/paper272.html   joubert, d. j., & lee, t. p. (2007). empowering your institution through assessment. journal of the medical library association, 95(1), 46-53. retrieved 28 may 2013 from http://www.ncbi.nlm.nih.gov/pmc/articles/pmc1773025/   kearns, s. k. (2004). marketing library service assessment. technical services quarterly, 22(2), 49-61. doi:10.1300/j124v22n02_05   libqual+®. (2010). suggestions for marketing your libqual+® survey. retrieved 28 may 2013 from http://old.libqual.org/publications/all.cfm?pubtype=12   phipps, s. e., & lakos, a. (2004). creating a culture of assessment: a catalyst for organizational change. portal: libraries and the academy, 4(3), 345-361. retrieved 28 may 2013 from http://www.escholarship.org/uc/item/0843106w   souder, w. e. (1981). disharmony between r&d and marketing. industrial marketing management, 10(1), 67-73. doi: 10.1016/0019-8501(81)90029-8   spalding, h. h., & wang, j. (2006). the challenges and opportunities of marketing academic libraries in the usa. library management, 27(6/7), 494-504. doi:10.1108/01435120610702477   sumsion, j. (2001). library statistics for marketing. ifla journal, 27(4), 221-231. doi:10.1177/034003520102700404   washington-hoagland, c., & clougherty, l. (2002). identifying the resource and service needs of graduate and professional students: the university of iowa user needs of graduate professional series. portal: libraries and the academy, 2(1), 125-143. retrieved 28 may 2013 from http://muse.jhu.edu/journals/portal_libraries_and_the_academy/v002/2.1hoagland.html   wright, s., & white, l. s. (2007). spec kit 303: library assessment. washington, dc: association of research libraries.     article   data-driven decision making: an holistic approach to assessment in special collections repositories   melanie griffin special collections librarian university of south florida tampa, florida, united states of america email: griffinm@usf.edu   barbara lewis coordinator for digital collections university of south florida tampa, florida, united states of america email: bilewis@usf.edu   mark i. greenberg dean of libraries western washington university bellingham, washington, united states of america email: mark.greenberg@wwu.edu   received: 15 mar. 2013   accepted: 10 apr. 2013      2013 griffin, lewis, and greenberg. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – in an environment of shrinking budgets and reduced staffing, this study seeks to identify a comprehensive, integrated assessment strategy to better focus diminished resources within special collections repositories.   methods – this article presents the results of a single case study conducted in the special and digital collections department at a university library. the department created an holistic assessment model, taking into account both public and technical services, to explore inter-related questions affecting both day-to-day operations as well as long-term, strategic priorities.   results – data from a variety of assessment activities positively impacted the department’s practices, informing decisions made about staff skill sets, training, and scheduling; outreach activities; and prioritizing technical services.  the results provide a comprehensive view of both patron and department needs, allowing for a wide variety of improvements and changes in staffing practices, all driven by data rather than anecdotal evidence.   conclusion – although the data generated for this study is institutionally specific, the methodology is applicable to special collections departments at other institutions. a systemic, holistic approach to assessment in special collections departments enables the implementation of operational efficiencies. it also provides data that allows the department to document its value to university-wide stakeholders.     introduction         like many academic libraries, the university of south florida (usf) tampa library reports all manner of statistics but has not come effortlessly to assessment (university of south florida tampa library, 2012). over the last decade, it has participated several times in libqual. librarians involved in instruction use a variety of assessment tools, such as preand post-tests, but as of yet there is no library-wide assessment program such as standardized assessment of information literacy skills (project sails, 2012; rumble & noe, 2009; mery, newby & peng, 2012; sutherland, 2009).  a greater emphasis on accountability in higher education, sobering fiscal realities at usf, and considerable attention over the last decade paid to library assessment activities have driven the usf tampa library to action.   in late 2009, the usf provost convened a campus-wide student success task force to recommend a fundamental transformation to the student experience. in its 160-page report, the task force made three recommendations: institutionalize student success as a permanent usf priority, integrate student success into usf’s institutional culture, and build the research capacity to support student success initiatives (university of south florida, 2010). in 2010, the university formed the office of student success (oss). for the last three years, oss has engaged nearly every unit at usf in order to enhance academic progress and student satisfaction, improve graduation and graduate school admission rates, and increase student competitiveness in the marketplace. the library quickly became central to oss goals. library administrators extended hours of operation to 24/5 and welcomed tutoring and learning services, a writing center, stem teaching lab, student employment center, and the office of undergraduate research into the library.   the emphasis on accountability for improved student success coincided with deep cuts to higher education in florida. usf’s appropriation from the state legislature fell from nearly $371.91 million in 2007-2008 to $305.25 million in 2011-2012 (university of south florida system, 2011). during the same period student headcount grew from 28,578 to 45,290 (university of south florida office of decision support, 2012). in the tampa library, the number of professional and non-professional staff declined from 103.59. fte in february 2008 to 81.59 fte in february 2013, and in may 2011 the graduate assistant program in cooperation with the school for library and information science was discontinued. the loss of six part-time graduate students placed a particular strain on public services activities throughout the tampa library, including the special collections reading room. compounding loss of staff between 2007 and 2011, total library expenditures decreased 8% from $11.86 million to $10.91 million while print and electronic resources costs at usf increased an average 4.2% annually (university of south florida tampa library, 2012). these cuts required the library to redouble its efforts to work smarter, assess operations and services, and utilize scarce resources most effectively.   the rising importance of assessment within the library profession, as evidenced by attention to the topic in professional literature, also motivated the usf tampa library to take assessment more seriously. an august 2012 survey of publications on assessment indexed in ebsco library literature & information science full text revealed 236 peer-reviewed articles published 1990-1999 and 676 peer-reviewed articles the following decade. from january 2010-august 2012, alone, 376 articles appeared in publication.   aims   within an environment of rising expectations, decreasing resources, and the profession’s growing interest in assessment, the tampa library formally revisited strategic goals set in 2008 in order to adapt them to the dramatically shifting terrain. following a lengthy process that involved the entire staff, a written report in may 2011 “reset” the strategic direction begun three years earlier. the report confirmed special & digital collections’ (sdc) significant role in cultivating a research culture within the library. specifically, sdc was asked to redouble its efforts to build several collections of national distinction (albeit with fewer resources), develop and refine research tools and services to support these collections, and expand its outreach. library investment of staff time and financial resources prioritized strategic projects (university of south florida tampa library office of the dean, 2011).   sdc staff quickly realized it could not meet its obligations under the library’s strategic plan nor continue to improve public services and collections in an environment of diminishing human and financial resources without greater attention paid to assessment. department librarians and staff also understood that the questions it sought to answer, though focused primarily on public services, were interrelated and thus required an approach that addressed a variety of activities in a comprehensive and integrated manner. specifically, sdc’s assessment plan asked the following:    1) what are the department’s staffing needs? 2) what staff skill sets and training are required to meet researchers’ expectations, and what personnel skills and functions are most needed by the department in the future? 3) where should the department target its outreach efforts? 4) how can the department streamline and prioritize technical services to support patron needs? 5) how can collection development and intellectual access activities best align with strategic goals and patron needs?   literature review   the professional literature includes a rapidly growing number of publications on assessment for academic libraries in general, but discussions of assessment methodologies for special collections and archives tend to be sparse and to focus on answering specific questions, usually related to technical services (bancroft library, 2011; philadelphia area consortium of special collections, 2013). common types of assessment studies in special collections literature include methodologies for computing the time (abraham, balzarini & frantilla, 1985) or money (ericksen & shuster, 1995) required to process archival materials, reducing the backlog of hidden collections (jones, 2004), measuring the impact of special collections cataloging (lundy, 2008), and performing condition assessments (green, 2004). while many of these studies, particularly those discussing minimal standards processing, consider access and user implications (greene & meissner, 2005), very few as yet focus specifically on establishing metrics for defining “good” public services in special collections or archives. the archival metrics project is an outlier, providing toolkits for assessing various parts of a special collections or archives department, including public services web tools. although the toolkits are important resources for special collections and archival repositories, they are not exhaustive. they do not, for example, provide mechanisms for assessing technical services in relationship to public services. more recent literature, particularly the fall 2012 special rbm: a journal of rare books, manuscripts, and cultural heritage issue on assessment, focuses on a varied array of hypothetical assessment strategies for special collections and archives. articles from this issue establish the framework for an evidence-based practice approach to assessment (chapman & yakel, 2012), outline methods for conducting archival collections assessment (conway & proffitt, 2012) and instruction assessment (bahde & smedberg, 2012), and considerations for assessing online finding aid and website design (hu, 2012). only one article offers a practical case study, detailing how assessment methodologies might be applied in a specific circumstance (gustainis, 2012).   methods   as sdc was unable to find an existing assessment methodology that considers the entire special collections environment, sdc librarians and staff created an holistic assessment model that takes into account the needs, requirements, and standards of public services, technical services, and administration. this paper presents the results of qualitative and quantitative assessment activities, which, when taken as a whole, provided sdc with a comprehensive view of patron and department needs. from january 2011 to december 2012, staff collected quantitative data on collection use, reading room activity, and website traffic using circulation and reading room statistics, reader registration profiles, and web analytics. to accomplish the necessary quantitative data collection, the department utilized a variety of free and library-wide licensed systems to automate existing manual processes and thereby create operational efficiencies.  although there are a variety of tools available to facilitate assessment activities, sdc chose google analytics, desk tracker, libguides, and sql queries in content management systems, such as fedora commons repository software.  to address challenges specific to managing special collections, the department licensed aeon. patron surveys and usability testing provided qualitative information on the patron experience in the reading room and with the department’s web tools.    aeon   aeon, a product of atlas systems, inc., is a material request and workflow management software specifically designed for special collections libraries and archives. the data collected in aeon provides staff with detailed patron information as well as reading room and material usage statistics. patron data includes status (undergraduate, graduate, faculty, staff, community user, visiting scholar, etc.), discipline (humanities, social sciences, etc.), research interests (optional), and the day, time, and duration of each visit. material request data includes the type of material (e.g., monograph, archival material, etc.), collection name, day and time a user received and returned materials, the patron’s user id, and standard bibliographic information.   from january 2011-december 2012, 4,547 material transactions and 1,355 reading room visits were recorded. at the end of each semester, staff generates and analyzes a standard set of reports (see figure 1). the combination of data collected and reported enables sdc staff to identify when the reading room is most active, what type of patrons use the reading room and when, and what collections are used and by whom. aeon also tracks which staff members are involved in each step of each transaction. analysis of this data provides insight into staff members’ proficiency in their use of aeon and identifies potential training needs.   desk tracker   aeon offers valuable data on reading room and collections use, but the software is not designed to record all patron contact. sdc librarian and staff interaction with patrons takes many forms, including face-to-face communication, email, letter, fax, and telephone calls, and serves a variety of purposes, including research consultations, program planning, collection development, and donor relations. in order to better assess the use of librarian time and the knowledge required by reading room staff, sdc needed a system to capture data on all types of patron interactions.   desk tracker is a web-based library statistics system offered by compendium library services llc. the program enables library staff to record general patron transaction activities, generate reports via a standardized reporting process, and customize desk tracker windows to capture both individual and public service point information. the customization features make it possible for staff at each service location to collect unique data, but also to standardize across service points how patron transaction information is recorded and tracked and the type and level of data that is collected.    as with aeon, staff members record user type. in addition, they also note the purpose of the visit, the specific request(s) made and/or question(s) asked, and the outcome of the interaction (see figure 2). in the case of material requests, the interaction is noted in desk tracker, but all details of the request appear in aeon.   figure 1 average # of reading room visits by hour january 3, 2011 – december 9, 2011 (n=1,355 visits)     figure 2 desk tracker reading room staff form     reading room patron survey   rather than developing an independent instrument to measure patron satisfaction in the reading room, sdc adapted existing instruments created by the archival metrics project. sdc modified its instrument to be as short, simple, and as meaningful to the institutional context as possible. all patrons who request materials in the reading room are asked to fill out a paper survey, which is provided to them with their requested materials. a staff member then enters survey data into a surveymonkey form to facilitate data analysis. while ideal circumstances would require each patron to complete the questionnaire in a web-based form during each visit, some patrons decline to receive or complete the survey, and the physical layout and limited computer availability in usf’s reading room preclude a web-based option.   the one-page, one-minute survey asks users to rate their satisfaction in the reading room in six concrete, easily measured areas: the helpfulness of staff, time spent waiting for materials, hours of operation, noise levels, website functionality, and photocopying / duplication services. two additional questions ask patrons to rate their overall experience and their progress towards meeting research goals for the visit. the survey collects limited demographic information about the patron: status (undergraduate student, graduate student, faculty member, visiting scholar, community user) and the purpose of his/her visit to special collections (class assignment, dissertation or thesis, publication, family history, etc.). the survey ends with an open-ended comment field, asking for ways in which the reading room experience could be improved.   website and digital collections usability study   based on lackluster response rates to web-based usability testing at the usf library, during the spring 2011 semester sdc opted to conduct face-to-face website usability testing with a small sampling (n=10) of representative user types: undergraduate students, graduate students, and faculty members. unfortunately, no community users were available or willing to participate, resulting in a small but significant gap in the population sampled. this usability testing focused on two of sdc’s web tools: its main website and its digital collections user interface (coral).   the only demographics captured during usability testing were user status and preferred browser. during the test, sdc staff asked participants to find information on the department’s website and to perform a series of tasks using coral. a staff member observed the user during the test, created screen captures, recorded any verbal questions or comments, but did not provide help. after completing the series of tasks, staff asked each user a series of open-ended questions. web analytics   sdc uses a variety of content management systems to organize its web presence, including wordpress, libguides, and omeka. the department utilizes google analytics to track total and unique page views, bounce rates, exit rates, average time on pages for all wordpress and omeka web pages, as well as the browser and operating systems used to access these websites. in addition to its main website content, sdc also maintains a number of libguide-based subject pages, and the department uses the software’s built-in statistics tools to track individual page and guide views, device type, browser, and operating system.   fedora commons   sdc currently utilizes an internally developed digital asset management system built using the fedora commons repository software to store and access its own text, image, and audio/video digital content. searches, hits, views, and downloads are recorded in the database so that regular and ad hoc reports can be generated to identify digital collection and item usage. reports also detail the number of items in each collection and the size in megabytes for each item and collection.   results and discussion   after implementing the tools outlined above and initiating data collection, sdc librarians and staff analyzed the results of each process separately and as part of a long-term assessment strategy to inform changes in departmental practices. staff focused particularly on analyzing intersecting data points from multiple tools and devoted its time to improving services, rather than highlighting a list of problems that, for a variety of institutional or budgetary reasons, could not be fixed. with two years of ongoing, integrated data collection complete, the assessment outcomes described below have offered an excellent starting point for data-driven decision making. over time, the department plans to refine its continuous assessment strategies, learn more from data collected, and improve its operations accordingly.   staffing needs      prior to 2009, two department employees, often at least one librarian, staffed the public services desk in the reading room during all hours of operation (monday friday, 9 am – 6 pm). the department’s “just in case” model ensured that someone capable of answering any type of reference question would always be available, just in case they were needed. budget and staffing cuts necessitated changes. a single staff member, often a temporary student employee, began working solo shifts at the public service desk during reduced hours (monday – friday, 10 am – 5 pm), paging materials, answering basic reference questions when possible, and providing a librarian’s phone number or email address when greater knowledge or a reference consultation was needed.   librarians and administration worried about the implications of the new reading room model on quality service. department staff used several of the tools described above to assess the impact of these service changes on patron satisfaction and, most importantly, prioritized data rather than anecdotal evidence in its discussions.   first, the reading room survey provided simple quantitative data on factors such as patrons’ satisfaction with hours of operation and wait time, as well as qualitative information on their experience in the reading room. staff discovered that their perceptions of inadequate staffing levels and excessive wait times were exaggerated. during the first 2 years of data collection, only 1 of the 223 respondents expressed dissatisfaction with the service provided in the reading room. patrons were not shy in expressing concerns about other matters, particularly sdc’s inadequate photo duplication services, limited hours of operation, and sometimes confusing website. staff worried about collections expertise and research consultations “on demand” in the reading room, but patrons’ survey responses revealed that they did not mind receiving a librarian’s email address or phone number in lieu of an immediate answer. in fact, instead of the anticipated complaints, users routinely offered compliments about staff knowledge, availability, and helpfulness. sample responses to the question “what can we do to make improve your experience in the special collections reading room” include: “nothing! :)” and “nothing; clone your staff & send them downstairs to first floor reference desk. attitudes are so helpful up here!” constructive criticism comments included “extended evening hours,” “at work & class from 9 am to 5 pm daily. i have trouble getting to s.c. during the open hours,” and “make copier accessible to reader[s].” based on the collected data, staff decided that major changes to the service model were not needed, but that operational modifications would be beneficial.   two significant changes occurred due to results from the reading room survey. first, patrons confirmed the inadequacy of photo duplication services. the department relied on a single flat bed photocopier, inaccessible to patrons, and staff denied many copy orders either on account of materials’ size or fragility. staff offered use of an inexpensive digital camera, but it was not a popular solution, as lighting levels and limited camera functionality frequently resulted in blurry images, especially of textual materials. staff believed that an overhead scanner with a book cradle provided a better solution, but the cost seemed prohibitive given the library’s declining operating budget. using the qualitative and quantitative data generated by the patron survey, sdc partnered with the academic services unit at the tampa library to write a successful student technology fee grant to install three overhead scanners in the building, including one in the reading room. with the scanner installed, patron complaints about reproduction services have drastically decreased, although they have not disappeared entirely, and many patrons have offered positive feedback on the change from limited photocopies to self-service scanning.   second, in response to the dissatisfaction expressed by users over reading room hours, staff looked for simple ways to modify hours of operation without accruing additional costs. using data exported from aeon, staff analyzed traffic patterns in the reading room. they isolated high demand during lunch and early afternoon and more limited use late in the afternoon. they noted the frequency with which the department opened to waiting patrons, which ultimately led staff to open the reading room an hour earlier. current staffing levels preclude evening and weekend hours, despite repeated requests for “different” hours on the reading room survey and in phone calls to the department, documented in desk tracker. the data also confirmed that a second staff member at the public service desk is generally not needed before lunch and at the end of the day, but additional support is required for three hours in the afternoon monday through thursday. today, the department’s reading room manager, with additional help readily available, covers these hours. additionally, data from 2011 on limited late afternoon use during summer semesters led the department to close an hour earlier from may to august in 2012, making better use of staffing resources.   data derived from desk tracker provide granular information on patrons’ needs in the reading room. early afternoon hours tend to be the busiest, but undergraduates with known item retrieval requests constitute a disproportionate number of users during these hours. visiting researchers, graduate students, and usf faculty, for whom more time consuming transactions usually occur, tend to arrive much earlier in the day, and they often have communicated with a librarian liaison before their visit. for patrons who have called ahead or already completed a research consultation, item retrieval requests tend to be more predictable and thus less time consuming for desk staff.   training and supervision   prior to aeon’s adoption by usf in spring 2010, special collections staff did not uniformly adhere to written procedures regarding information expected on reader registration forms and call slips or the order in which tasks were to be completed at the reading room public service desk. as a result, the department knew little about some of its patrons for purposes of outreach and security, could not accurately count collection use from illegible or incomplete call slips, and faced unacceptably high numbers of misplaced materials with no way but memory to trace the last staff member to touch an item.   with multiple librarians overseeing the reading room but no single person in charge, effective training and supervision proved difficult. new students or staff working the public service desk struggled to remember and follow policies and procedures, and a few recalcitrant longtime employees remained wedded to old ways of doing things. juggling multiple responsibilities, the department’s director and librarians did the best they could to address issues as they occurred, but the collaborative approach to training and supervision proved increasingly ineffective. amidst other changes underway in public services, department members decided to fill a line vacated by an administrative assistant with an operations manager to oversee staff and student training and supervision, revise and implement new reading room procedures, coordinate security, collect and analyze statistics, and maintain public services software management systems.   aeon offers uniform, required workflows that limit the ability to provide or accept incomplete patron registration or materials request information or skip steps in the request, retrieval, and re-shelving of items. aeon reports provide information on transaction types and about individual staff members’ performance with the product, thereby identifying areas in need of additional training. for example, analysis of the data on users signed into and out of the reading room revealed that some staff members were not always diligent about signing patrons out. remediation and enhanced supervision ensured that staff more accurately recorded reading room traffic data.   web presence   sdc’s website is often the first point of contact between patrons and the department, and it serves as an essential outreach tool. phone calls and in-person questions from puzzled or frustrated patrons suggested that sdc’s website navigation structure was not always intuitive and its content occasionally incomplete. early results from the reading room patron survey, which asks users how easily they navigated the department’s website, confirmed this suspicion. usability testing with patrons and bounce rates derived from google analytics provided concrete information on specific and suspected navigation difficulties, confusing terminology, and technical barriers to accessing information.   when sdc first conducted usability testing on its web pages, the library used libguides as its content management system. this cms necessitated a tabbed structure, but staff built pages without consistent, hierarchical navigation accessible from every page of the site. not surprisingly, patrons most often experienced difficulty with basic navigation. undergraduates, in particular, struggled to find the special & digital collections portion of the library’s main landing page. once they arrived, however, most users understood the site’s terminology and successfully located basic information such as department hours and a librarian to help with a project on a specific topic.   to make the website a more effective outreach tool, two sdc librarians collaborated with the newly hired webmaster to improve navigation. in addition, one librarian participated in the library’s website redesign team to ensure that the group considered sdc needs. technical limitations in the libguides platform prevented the design of flexible navigational structures, thus negatively impacting the user experience.  as a result, staff moved considerable amounts of content to wordpress, a cms which accommodates the flexible design of uniform navigation. as users more often experienced problems with navigation, not content or vocabulary, the department asked the webmaster to provide only structural and design support, and it retained control over its content management. these significant changes to the department’s website require additional assessment, planned for summer 2013, to measure the efficacy of the modifications and determine what improvements might still be made.   further usability testing, in conjunction with statistics from google analytics, highlighted a known issue in sdc’s digital collections user interface (coral): 36 percent of the site’s users accessed the collections in internet explorer, and those users experienced greater difficulty in performing simple and complex searches. in addition, users requested several enhancements, including boolean operators, the ability to limit searches by format (text, audio, and/or video), and access to higher resolution downloadable images. they identified visual changes to the interface to make it more intuitive, reduce click-throughs, and permit reproduction of metadata. sdc librarians, administration, and the coral developer documented, discussed, and prioritized all user concerns. as with assessment of the reading room, staff focused on issues that could be changed rather than those without a practicable solution. during spring and summer 2012, the developer made programmatic modifications and enhancements to coral in order to address these issues. in addition, google analytics and libguides statistical data indicated an increase in the use of smartphones and tablet devices to access sdc web content.  this change in our users’ profile highlighted the need to develop user interfaces that are device and browser independent.  to that end, current programming projects, such as sdc’s oral history player interface (ohpi), are being developed in html5 to replace flash-based systems currently in place.   data derived from page hits, combined with collection use patterns from coral and aeon, provided insights into new avenues for outreach. some of the department’s most heavily used collections correlate with its most frequently used web pages, but some collection pages have extremely high hit rates despite sporadic collection use. low bounce rates suggest that page visits do not result from false hits, but staff have not been able to discover why, in some instances, webpage usage coincides with collection usage and other times it does not. librarians are currently experimenting to see if high web hits for collections with lower usage statistics offers an opportunity for targeted outreach and instruction efforts to translate interest into use.   technical services and collection development   the same budget cuts that necessitated changes to sdc’s public services staffing policies also resulted in slower rates for cataloguing monographs and serials, processing archival collections, and acquiring or creating collections. in spring 2012, the department’s dedicated cataloguer resigned, leaving much of the work to a single remaining paraprofessional. sdc staff utilized the reading room patron survey, aeon, desk tracker, and coral statistics to reset some of its technical services priorities, make targeted acquisitions, and establish more responsive cataloguing, archival processing, and digitization priorities.   in terms of stacks maintenance, sdc has thirty-six distinct location codes in usf’s catalog for monographic collections and an additional two codes for archival and manuscript collections. locations are further delineated in a separately maintained stacks guide, which indicates the range and shelf number for each collection. as in many repositories, space is at a premium, and in recent years staff members have spent considerable time shifting collections to accommodate new acquisitions. now, with data on which collections patrons most heavily use, stacks management decisions are more thoughtful. infrequently used collections, for example, now reside in quasi-remote storage, freeing space near the reading room for heavily used materials.   collection use data has also driven decisions about whether to pursue or accept specific donations and to make particular purchases. for example, materials related to the cigar industry and its ethnic communities in ybor city and west tampa comprise one of sdc’s most heavily used collection areas. as a result, one sdc librarian has devoted additional effort to working with potential donors to assess and, where appropriate, accept donations of manuscripts and monographs. with the holocaust & genocide studies center’s collections receiving growing use by faculty and students, sdc librarians have expanded relationships with targeted rare book and manuscript vendors in the u.s. and abroad to purchase published and unpublished materials. given increased demand for the subject area by users, these items receive priority cataloguing and processing.     reading room patron surveys, google analytics, and aeon and desk tracker statistics now play a greater role in determining digitization priorities. for example, based upon extensive use of a local african american newspaper, the department embarked upon a preservation/access project to digitize paper copies of the previous five years.  sdc librarians track disproportionate hits to subject pages on the department web site, the high use of specific collections, and individual digitization requests. patrons’ needs have joined a parallel production track within the digitization lab. longer-term, internally directed projects designed to grow usf’s reputation as a research library occur alongside externally driven, more immediate, smaller scale digital collection building.    conclusions   during the last two years, special & digital collections has focused considerable energy on developing and implementing a systematic, holistic assessment strategy to improve a range of services in the department. with data derived from several assessment tools, staff has better aligned reading room hours and staff skill sets with patron needs, utilized limited financial and human resources to build the physical and digital collections demanded by its patrons, and more thoughtfully targeted its outreach efforts.   despite significant improvements to department operations, sdc’s assessment efforts are not complete. most sdc assessment activities are continuous, but not all. the department needs to conduct more frequent usability studies of its web tools and several content management systems. since mid-2012, sdc’s digital collections have resided in usf’s institutional repository as well as in coral. as yet, sdc has not gauged patrons’ satisfaction with the repository or determined their preferences between systems. in coming months, the department plans to implement new photo duplication request processes, and those too will require careful analysis and patron feedback. recent changes to cataloguing workflows, shifting from a dedicated professional cataloguer to greater reliance on paraprofessionals and carefully supervised students, need both quantitative and qualitative study. although there were no national metrics for assessing special collections in place when sdc began its assessment projects, a task force of the rare book and manuscript section of acrl is currently working to address this gap (acrl rare books and manuscripts section, 2013). once the task force establishes metrics and assessment standards, sdc plans to integrate them into its methodology and to track self-improvement from year to year and to compare itself to peer and aspirant institutions.   despite these challenges, something transformative has occurred over the last two years at usf. where once sdc librarians and staff aspired to assessment, today the department has adopted a systemic, holistic strategy that has become part of its working culture. this culture of assessment has enabled the department to improve operational efficiencies and to maintain or, in some instances, increase levels of service during lean financial times. data derived from assessment activities clearly demonstrate the impact of sdc activities and allow the department to illustrate its alignment with institutional priorities to library and university administration on a routine basis as well as during formal reviews, such as university-wide reaccreditation processes. the greatest hurdle to continuous improvement has been overcome, leaving usf’s special and digital collections department well situated for continued relevance and success.   acknowledgements   a version of this paper was first presented at the library assessment conference in charlottesville, virginia, united states of america october 29-31, 2012. the authors would like to thank session participants for their feedback.     references   abraham, t., balzarini, s.e., & frantilla, a. (1985). what is backlog is prologue: a measurement of archival processing. american archivist 48(1): 31-44.   acrl rare books and manuscripts section. (2013). metrics and assessment task force. retrieved 9 may 2013 from http://www.rbms.info/committees/task_force/metrics_assessment/index.shtml.   archival metrics: promoting a culture of assessment in archives and special collections. retrieved 9 may 2013 from http://archivalmetrics.org/.   bahde, a. and smedberg, h. (2012). measuring the magic: assessment in the special collections and archives classroom. rbm: a journal of rare books, manuscripts, & cultural heritage 13(2): 152-174.   bancroft library. (2011). the bancroft library manuscripts survey project. retrieved 9 may 2013 from http://bancroft.berkeley.edu/info/mss_survey/index.html.   chapman, j. & yakel, e. (2012). data-driven management and interoperable metrics for special collections and archives user services. rbm: a journal of rare books, manuscripts, & cultural heritage 13(2): 129-151.   conway, m.o. & proffitt, m. (2012). the practice, power, and promise of archival collections assessment. rbm: a journal of rare books, manuscripts, & cultural heritage 13(2): 100-112.   ericksen, p. & shuster, r. (1995). beneficial shocks: the place of processing-cost analysis in archival administration. american archivist 58(1): 32-52.   green, p. (2004). a method for undertaking a full conservation audit of special collections of books and manuscripts. collection management 28(4): 23-42. doi: 10.1300/j105v28n04_03   greene, m.a. & meissner, m. (2005). more product, less process: revamping traditional archival processing. american archivist 68(2): 208-263.   gustainis, e.r.n. (2012). processing workflow analysis for special collections: the center for the history of medicine, francis a. countway library of medicine as case study. rbm: a journal of rare books, manuscripts, & cultural heritage 13(2): 113-128.   hu, r. (2012). methods to tame the madness: a practitioner’s guide to user assessment techniques for online finding aid and website design. rbm: a journal of rare books, manuscripts, & cultural heritage 13(2): 175-190.   jones, b.m.j. (2004). hidden collections, scholarly barriers: creating access to unprocessed special collections materials in america’s research libraries. rbm: a journal of rare books, manuscripts, & cultural heritage 5(2): 88-105.   lundy, m. (2008). provenance evidence in bibliographic records: demonstrating the value of best practices in special collections cataloging. library resources & technical services 52(3): 164-172.   mery, y., newby, j. and peng, k. (2012). why one-shot information literacy sessions are not the future of instruction: a case for online credit courses. college & research libraries 73(4): 366-377.   philadelphia area consortium of special collections. (2013). pacscl hidden collections processing project. retrieved 9 may 2013 from http://clir.pacscl.org/.   project sails. (2012). about project sails. retrieved 9 may 2013 from https://www.projectsails.org/aboutsails.   rumble, j. & noe, n. (2009). project sails: launching information literacy assessment across university waters. technical services quarterly, 26(4): 287-298. doi: 10.1080/07317130802678936   sutherland, k. (2009). librarians as literacy sponsors: a critique of information literacy assessment tools. progressive librarian 33: 18-25.   university of south florida. (2010). student success task force report, 2009-2010. retrieved 9 may 2013 from http://usfweb3.usf.edu/studentsuccess/about/student-success-task-force-final-report.pdf.   university of south florida office of decision support. (2012). student headcount. retrieved 9 may 2013 from http://usfweb3.usf.edu/infocenter/.   university of south florida system. (2011). operating budget, 2011-2012. retrieved 9 may 2013 from http://usfweb2.usf.edu/bpa/ob/11-12/1112_operating_budget_web.pdf.   university of south florida tampa library. (2012). public services statistics. retrieved 9 may 2013 from http://www.lib.usf.edu/academic-services/public-services-statistics/.   university of south florida tampa library office of the dean. (2011). resetting strategic direction: final report. retrieved 9 may 2013 from http://guides.lib.usf.edu/dean-usf-libraries.       evidence based library and information practice editorial responsibilities      2013. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     editor-in-chief: alison brettle associate editor (evidence summaries): lorie kloda associate editor (articles and reviews): denise koufogiannakis associate editor (commentaries, using evidence in practice, news): alison brettle feature editor: martha kyrillidou production editor: michelle dunaway editorial intern: archana deshmukh copyeditors: heather pretty (lead copyeditor), georgianne bordner, molly des jardin, linda ferguson, richard hayman, stacey penney, mary virginia taylor, alison yeoman indexing support: pam morgan   evidence based library and information practice evidence based library and information practice 2011, 6.2 39 evidence based library and information practice evidence summary residents and medical students correctly answer clinical questions more often with google and uptodate than with pubmed or ovid medline a review of: thiele, r. h., poiro, n. c., scalzo, d. c., & nemergut, e. c. (2010). speed, accuracy, and confidence in google, ovid, pubmed, and uptodate: results of a randomised trial. postgraduate medical journal, 86(1018), 459-465. doi:10.1136/pgmj.2010.098053 reviewed by: theresa arndt associate director for research & instructional services waidner-spahr library, dickinson college carlisle, pennsylvania, united states of america email: arndtt@dickinson.edu received: 01 mar. 2011 accepted: 21 apr. 2011 2011 arndt. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to determine which search tool (google, uptodate, pubmed or ovidmedline) produces more accurate answers for residents, medical students, and attending physicians searching on clinical questions in anesthesiology and critical care. searcher confidence in the answers and speed with which answers were found were also examined. design – randomized study without a control group. setting – large university medical center. subjects –subjects included 15 fourth year medical students (third and fourth year), 35 residents, and 4 attending physicians volunteered and completed the study. one additional attending withdrew halfway through the study. the authors were unsuccessful in recruiting an equal number of subjects from each group. methods – a set of eight anesthesia and critical care questions was developed, based on their commonality and importance in clinical practice and their answerability. four search tools were employed: google, uptodate, pubmed, and ovid medline. in part i, subjects were given a random set of four of the questions to answer with the search tool(s) of their choice, but could use only one evidence based library and information practice 2011, 6.2 40 search tool per question. in part ii, several weeks later, the same subjects were randomly assigned a search tool with which to answer all 8 questions. the authors state that “for data analysis, pubmed was arbitrarily chosen to be the “reference standard.”” statistical analysis was used to identify significant differences between pubmed and the other search tools. main results – part i: subjects choosing a search tool were more likely to find a correct answer with google or uptodate. there were no statistically significant differences in confidence with answers between any of the search tools and pubmed. part ii: though subjects were assigned a search tool, some questions were repeated from part i. for repeated questions, ovid users (compared to pubmed users) were significantly less likely to find the correct answer for repeated questions. otherwise, there was no statistically significant difference in questions answered correctly. confidence did not differ. when asked to answer new questions, subjects using google and uptodate were significantly more likely to find a correct answer than pubmed users. uptodate users were more confident. there was no statistical difference in primary outcome (correct answer with high confidence) between google, ovid, and pubmed. pooled data from parts i and ii, removing repeated questions: subjects using google and uptodate were more likely to find correct answers. confidence was highest among uptodate users. average search time per question (limited to 5 minutes per question) in ascending order of time spent was: uptodate, google, pubmed, and ovid. conclusion – while the number of participants is small, the results suggest that the popular search engine google and the commercially produced secondary online source uptodate are more useful and efficient for finding answers to questions arising in anesthesiology and critical care practice than tools focused exclusively on indexing the primary literature. commentary this study provides further evidence of the value of general internet and specialized secondary online sources to clinical practice. while expert searchers may chafe at the use of google in medical settings and the reliance on digested, commercial products such as uptodate over the “premier biomedical database” medline, it is becoming more difficult to deny the effectiveness and efficiency of the new tools. the literature review by the authors of this study places their research in context with other similar studies. while many studies have focused on the search process, this study’s authors developed a set of realistic questions for which there were available evidence-based answers, and tested the ability of clinicians who were non-expert searchers to actually find the correct answer. less compelling was the measure of selfreported confidence in the answer found. the authors express surprise at the lack of correlation between confidence and ability to correctly answer a question, although this phenomenon has been demonstrated in many studies across various disciplines including medical specialties. user choices of tools (in part i) and expressions of confidence were also contradictory, with google being the most popular search tool despite users expressing most confidence in uptodate. the study methods are quite complex, with multiple statistical analyses conducted. while the authors present various bar charts, the actual number of subjects behind the percentages presented is quite small. in addition to these charts and p values, it would have been helpful for the author to present an analysis of variance (anova) table to help the reader judge the significance of the statistical findings. in addition to the small number of subjects, a significant study weakness, acknowledged by evidence based library and information practice 2011, 6.2 41 the authors, is their failure to record the final source of the answer given for each question. the search tools used in this study are not directly comparable, nor are they discrete. google is never an end source, but rather leads the user to other sources, which can include specific pubmed records or full-text openaccess primary literature, such as that archived in pubmed central. a google search may also link directly to content within uptodate, a web-based subscription source which itself cites the primary literature and incorporates medline records. this does not, however, challenge the finding of google or uptodate as the most efficient means to find an answer to a clinical question compared to pubmed, or to the commercial medline interface ovid, which fared poorly in this study. studies such as this one are useful for shedding light on the ways in which clinicians actually approach searching for answers to clinical questions and the high priority that is placed on efficiency in practice settings. such understanding calls into question the emphasis many health librarians place on traditional databases, and can help librarians improve the ways in which they support this population of clients with research tools, training, and other services relevant to their needs. / evidence based library and information practice evidence summary microsoft word news_proded.doc evidence based library and information practice 2007, 2:3    134 evidence based library and information practice     news    evidence based library and information practice seeks production editor        © 2007 glynn. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    evidence based library and information  practice (eblip) is seeking a production  editor to join its editorial team.      the production editor is responsible for the  publishing function of the journal, including  manuscript layout and formatting of content.      the ideal candidate will be able to work  within short deadlines and dedicate focused  time on issue production according to a set  schedule.  solid knowledge of ms word,  adobe acrobat and basic html coding is  essential as well as a demonstrated aptitude  and comfort working in an online  environment. this is an unpaid position.    interested persons should send their  resumes by october 1, 2007, to:    lindsay glynn  editor‐in‐chief  lglynn@mun.ca  (709) 777‐6026                    http://creativecommons.org/licenses/by/2.0 mailto:lglynn@mun.ca feature   eblip7 keynote: what we talk about when we talk about evidence   denise koufogiannakis collections and acquisitions coordinator university of alberta libraries edmonton, alberta, canada email: denise.koufogiannakis@ualberta.ca   received: 22 aug. 2013  accepted: 17 oct. 2013      2013 koufogiannakis . this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   the following text is a summary of the opening keynote address at the 7th international evidence based library and information practice conference, given on july 16, 2013 at the university of saskatchewan, in saskatoon, canada.   introduction   this morning i want us all to start thinking about evidence! now i know that doesn’t seem too revelatory given that we’re at an evidence based practice conference -but i think that we don’t usually take the time to think about what evidence actually is in the profession of librarianship, or how we use it.  so, my talk is going to explore those issues of evidence, based on the findings from my doctoral research. what is evidence?   the oxford dictionary says that evidence is: “the available body of facts or information indicating whether a belief or proposition is true or valid” (evidence, 2010). looking at the wider body of literature about the nature of evidence, key elements of evidence are revealed and can be applied to the field of library and information studies (lis). in keeping with the previous definition, evidence is commonly thought of as something constituting a form of proof to enhance a claim (hornikx, 2005; upshur, vandenkerkhof & goel, 2001; reynolds & reynolds, 2002; twinning, 2003).  that evidence serves as a proof, differentiates it from information – information must be relevant to the question at hand, and be used to prove a hypothesis in order to be considered evidence. evidence is generally seen as having three major properties: relevance, credibility, and inferential force or weight (schum, 2011). relevance looks at how the information bears on what is attempted to be proven; credibility asks whether what is reported actually occurred; and, inferential force or weight considers how strong the evidence is in comparison to other evidence.   types of evidence noted in the literature are wide ranging. rieke and sillars (1984) consider there to be four types of evidence: anecdotal (a specific instance), statistical (numerical representation of multiple instances), causal (explanation for the occurrence of effect), and expert (testimony of an expert) evidence. in a similar vein but considering a different categorization of evidence, glasby, walshe, and harvey (2007) created a typology with three types of evidence: theoretical (ideas, concepts and models to explain how and why something works), empirical (measuring outcomes and effectiveness via empirical research), and experiential (people’s experiences with an intervention). they say that “we need to embrace a broad definition of evidence, which recognises the contribution of different sorts of knowledge to decision making” (p. 434). evidence must always be used in context, whether in the context of a particular situation, or context of a wider body of professional knowledge. in my recent phd research, a grounded theory study, i studied a group of 19 participants. all were academic librarians in canada, working in a variety of settings and positions. they kept online diaries in the form of a private blog, for the period of one month each, in which they wrote down the problems or questions that they encountered in their practice during that time period, and what they did about them. basically, they were tracking their thought and decision making processes for me. i then interviewed each participant to dig deeper into the detail of their decision making, and learn why they made the decisions they did, as well as what kinds of evidence they used to help them in making that decision. i wanted to learn about what sources academic librarians use as evidence and how they use that evidence. driving the study was a desire to base the model of evidence based library and information practice (eblip), which promotes the use of research evidence in practice, on research itself. with the exception of a study by thorpe, partridge and edwards (2008) (see also partridge, edwards and thorpe, 2010), no research had been conducted on the actual eblip model and whether it was useful or appropriate for librarians. since eblip was adapted from evidence based medicine (ebm), it was a legitimate question to ask whether the same model that worked for physicians really works for librarians. it was time to explore whether the model was valid and if changes were needed. the goal was to approach the study with a view to learn and to listen to academic librarians and how they use evidence in daily practice. after doing a thorough examination of the model of eblip as it has been presented in the literature, it became clear to me that “evidence” in the context of eblip refers to published research articles. booth’s definition, as noted here, does account for other aspects, but the focus is on research derived evidence, and what we have pursued within the eblip movement since the time of this definition, points mostly to research evidence:   “an approach to information science that promotes the collection, interpretation and integration of valid, important and applicable user-reported, librarian observed, and research-derived evidence. the best available evidence, moderated by user needs and preferences, is applied to improve the quality of professional judgements.” (booth, 2000) the process of eblip, as with other forms of evidence based practice, is one which advocates searching the literature to find research articles, appraising those research articles to ensure they are valid, and then integrating the findings into one’s practice.  while professional knowledge and user preferences are accounted for in the definition of eblip, the conversation about those elements stops there. there have also been criticisms that the eblip model does not account for other forms of knowledge that are a vital part of professional practice. booth himself has more recently tried to clarify that eblip requires more than research, and that “the best available evidence and insights derived from working experience, moderated by user needs and preferences” are essential (booth, 2012). the more basic question of “what is evidence?” has not yet been debated or tested to any degree within the literature of eblip. there has been no research to show that in lis evidence only consists of research; this treatment of evidence was simply adapted from the evidence based medicine model.   “i’m clueless how to speak evidence”   when participants in my study were directly asked what they considered to be evidence, most were a bit taken aback by the question often noting that they had not thought about what evidence was before, or admitting that it was “a difficult question”. after thinking about it, most participants took a very broad view of evidence they were very open to the possibility of what evidence might be. most participants named several sources of evidence, and usually put those in context. for example, different evidence sources depending upon the type of problem faced.   as for what the participants actually used as evidence my research revealed that academic librarians use a wide breadth of evidence sources in their decision making. actual evidence sources used were numerous and detailed. in order to best convey this information, the evidence sources have been grouped into two main types, and within those types there are a total of nine main categories of evidence (koufogiannakis, 2012).   hard evidence sources are types of evidence that are usually more scientific in nature. they may focus on numbers, or are tied to traditional publishing outputs. sources are usually quantitative in nature, although qualitative research and non-research publications also fall into this category. ultimately, there is some written, concrete information tied to this type of evidence. a librarian can point to it and easily share it with colleagues. it is often vetted through an outside body (publisher or institution), and adheres to some set of rules. these types of evidence include the published literature (research and non-research articles), facts, documents, statistics and data, as well as local research and evaluation projects that are documented. these sources are generally acknowledged as acceptable sources of evidence, and are what a librarian would normally think of as evidence in library and information studies.   the other type of evidence can be thought of as “soft” evidence. as opposed to the “hard” evidence, soft sources of evidence are non-scientific. they focus on experience and accumulated knowledge, opinion, instinct, and what other libraries or librarians do. this type of evidence focuses on a story, and how details fit into a particular context. soft evidence provides a real-life connection, insights, new ideas, and inspiration. such types of evidence include input from colleagues, tacit knowledge, individual feedback from users, and anecdotal evidence. these types of evidence are more informal and generally not seen as deserving of the label evidence, although they are used by academic librarians in their decision making as a form of proof.   i want to illustrate this use of multiple types of evidence through the example of information literacy. let’s just pretend that librarians at a university need to develop a new information literacy program for engineering students. what are the evidence sources they are likely to draw upon (in broad terms, without knowing the specific context)? this is to illustrate how we go about gathering evidence, and where we find it.   first the librarian might reflect on his or her own past knowledge or experience in teaching information literacy skills to reflect upon what has worked for him or her in the past. he or she will likely speak with colleagues at their own institution to learn what has worked in other subjects or courses. the librarian may then speak with faculty – looking to discover what students in this course need to learn; what are the course objectives?     they may then branch out to see whether any other universities offer a similar course; start looking at the literature for examples of what worked elsewhere; look on the internet at other universities’ websites for documentation or other types of information relating to such a course. then they may decide to follow up that initial investigation of what others are doing by arranging to speak with specific librarians at other universities and learn what they did and what was successful. at the same time, they may probe deeper into the literature to look for any research studies that show what is most effective. in addition, they may look at documents such as the acrl (2011) information literacy guidelines, for guidance. the librarian will then go back to local information is there any internal data or research on information literacy or student needs? what needs to be evaluated once the course starts? the librarian may begin planning evaluation and feedback mechanisms, and may even think about whether there is a research project they can carry out within this new endeavour.   ultimately, the librarian must make a decision on how to teach the course. which of the sources consulted might be best to help with that decision? it’s complicated! which do we place more weight in? which should we place more weight in? how do we know what is best? how do we combine the various pieces of what we learn through the evidence gathering process? this is still a very big gap in our knowledge, and should be of utmost importance to those of us interested in evidence based library and information practice. what i’ve learned about evidence in lis, is that it can come from many sources. evidence is much more than research – and depending upon the type of question or problem we are trying to address, research will not always be the best source of evidence. the role of eblip is about using evidence and figuring out what is the best evidence in your particular situation. evidence use is not easily prescriptive, and must consider local circumstances. eblip’s focus to date has been on research evidence and how to read and understand research better. this is a good thing (i certainly do not want to diminish the importance of the work that has been done in this respect) but it is not the only thing and we must begin to explore other types of evidence. and finally, i think that librarians would be better served by a having greater understanding of the best types of evidence to use in particular situations. the question, then, is how do we best weigh different sources of evidence? i do not yet have that answer, nor has it been explored in our research literature to date. how do librarians use evidence?   i now want to turn our thoughts to how we, as librarians, use evidence. as previously mentioned, the focus of eblip to date has been on research evidence, and when we look at the model of eblip, which was adopted from medicine, it is clearly targeted toward the individual practitioner. this figure illustrates the 5a’s you are likely all familiar with it outlines the process of evidence based practice, and is what we have adopted within eblip.       figure 1 the 5a’s process of evidence based practice (hayward, 2007).     this model is meant to be used by an individual practitioner a physician, nurse, and in our case, librarian. the librarian works through each of these steps to practice in a more evidence based way. the findings from my doctoral study illustrated one very basic but game-changing thing to me that academic librarians (and i think this is largely transferable to other types of librarians) work in groups. all of our major decisions are made in groups, or require approval from others. there are some smaller decisions that we make on our own, but for the most part, our professional decisions rely upon others.   now, maybe some of you at this point are saying, “well, yeah, of course” but even though i have been an academic librarian for more than 15 years, and have been involved with the eblip movement for more than 13 years, this was a complete revelation to me. the fact is that our work in groups changes how we make decisions as opposed to when we make decisions on our own; and the fact is also that we make more decisions in groups than we do as individuals. this must have an enormous impact on the discourse within eblip, because if we want evidence to truly permeate and improve librarian decision making, we need to look at eblip from the context of group decision making.   this leads me then to how we actually use evidence, which has not been very well explored through research prior to my study. we use evidence for convincing, and i will explain this in more detail. there are two main aspects to convincing (koufogiannakis, 2013a, 2013b).   first of all, evidence is used for confirming. my research found that one of the main reasons librarians use evidence is to confirm that the decision they are making is correct. confirming generally applies in situations where an individual decision is being made, or when the librarian is part of a well-functioning group that she or he feels comfortable with.   confirming is nearly always positive because in doing so, a librarian is seeking to better understand something and add to their knowledge as a professional. what emerged very clearly in the data from participants is that academic librarians confirm to feel better and more confident that they are doing the right thing while remaining open to new possibilities. they may have initial thoughts, reactions and instincts, but they want to confirm those instincts with more concrete sources of evidence in order to proceed with their decision in a more confident manner. this is another way that the librarian brings together the soft evidence of their initial gut instinct or their own knowledge, with harder sources of evidence that corroborate the soft evidence, or else make the librarian re-think their initial position on the matter due to new evidence that was not previously known or considered. the quotes below, from participants in my study, illustrate some of the reasons and ways that librarians use evidence to confirm. participants felt that they could not base decisions solely on their existing knowledge because best practices are constantly changing and they need to continually learn. from those librarians just starting out, to those that were quite experienced, there was a common feeling throughout that they did not know everything and wanted some form of reinforcement whether it be from the literature, input from colleagues, or some other source of evidence.   i tend to use that [the literature] as confirmation for interesting ideas that i read about. (librarian 16, interview)   i find it interesting when the outcome matches/supports my initial gut reaction and instincts. for me this is one of the ways i test for validity when making decisions, a little private “ah-ha” moment – i can say, with confidence: ‘i knew it, i knew i was right’. if the info collected informs a decision or action different from my initial thought – i chalk it up to experience and put it under the category of: ‘good thing i double checked this’. (librarian 6, diary)   i just think that way and i feel more confident about what we’re doing if i know that we have – that we’ve tried to collect evidence, we’ve tried to assess what we’re doing and to me it’s just more confidence in going forward with other things. (librarian 17, interview) confirming is done for oneself. it is an act that reassures, and corroborates instinct or tacit knowledge. the participants’ actions show that they do not just gather evidence for external purposes, but that they gather and use evidence as part of their own professional development and regular practice of keeping current.     although not usually the case, confirming can occasionally be negative, if a librarian consciously discredits or avoids evidence that does not support their preconceived notion of what is the best.   secondly, evidence is used for influencing. as previously mentioned, while some decision making by librarians is individual, often decisions are made in a group setting, especially when they will have a major impact on library users or staff. my research shows that group decision making leads librarians to try and influence the final decision. influencing can be positive or negative. when in a positive work environment, participants often first go through the confirming stage for themselves, but when working with others, they bring evidence to the table in order to enable the group to make the best decision possible. in a positive situation individuals feel free to speak and be heard, and will reach a consensus. what an individual brings to the table, in this environment, is a positive form of influencing. when participants were in a negative environment, they often felt they were not being listened to, or their concerns not heard. they then adopted strategies to deal with this. one such strategy was to bring research evidence to the table in support of their viewpoint, where someone with an opposing viewpoint may not have done this. research is generally well regarded in an academic environment and therefore cannot be as easily dismissed as a person's own opinion. any form of evidence that shows “what other libraries do” is also seen in a very favourable light, as libraries may be more likely to make changes based on what is happening around them at other institutions. other strategies were to convince individuals and bring them on-side prior to any decision, or to stress particular points depending upon what the decision maker needs to hear in order to be persuaded. in all cases, the individuals want to influence the final result, and where a work environment is negative, they will use evidence as a “weapon”, to quote thorpe, partridge and edwards (2008) as they describe in the findings of their research regarding librarians’ experiences of evidence based practice, which is in keeping with my findings. different levels of control regarding decision making emerged from the data in this study. it became clear that librarians do not always have control over their own decisions. when an individual librarian makes his or her own decision, influencing is not required. in situations where a group makes the final decision, or where someone else makes the final decision, influencing is widely used. and the following quotes from participants illustrate the use of evidence for influencing:   where the group setting makes a difference, i think, is that depending upon whether or not i’m a champion for a particular project, i may present, you know i may frame the evidence in a way that i think would speak to the needs of the people in the group. (librarian 2, interview)   i will have to sell this to the university librarian. (librarian 18, diary)   i think you have to be very strategic because you have to recognize what the other person’s concerns are in order to address them and that’s the strategic part; and also being able to address the mandates of the library and all those other conflicts, right? (librarian 5, interview) the overall concept of convincing includes the two sub-categories i just discussed, confirming and influencing. confirming focuses on the self. it concerns a librarian’s knowledge and positioning as a professional. in this case, librarians look to the evidence in order to confirm and reassure themselves that they are on the right track with their decision making. they turn to the literature or to input from colleagues in order to verify their initial instincts. this process is a positive one because it is self-inflicted and builds confidence. generally, the librarian comes to the process of looking for and using evidence to confirm in a very open minded and forthright manner. influencing focuses on others and what a librarian needs to do to contribute to what would be a positive outcome from their perspective. influencing concerns transmitting what an individual thinks the decision should be to others that are involved in making the final decision, in order to convince them to come to the same conclusion. influencing can be a positive or negative experience depending upon the work environment. evidence in this situation can become simply a means to an end, and used differently depending upon the circumstances and the people involved. work environment largely determines the convincing strategy. for example, in co-worker relationships, how much control one holds, what is likely to convince someone, past experiences in dealing with particular people, and the perception of being heard in the workplace are all factors that impact the use of evidence and the reasons for using evidence.   depending upon the work environment, evidence is used differently. if it is a positive work environment, academic librarians are more forthcoming with ideas, listen to others, and are open to what the evidence says. if the work environment is negative, there is often secrecy, a withholding of information, evidence is used selectively to make a case, situations are approached differently depending upon personalities, there are feelings of hopelessness, and power-plays and strategizing are common.   generally, librarians want to contribute to organizational decision making, but if they feel that they are not being listened to, they will be disempowered and look for other ways to influence the outcome (or some may simply give up). ultimately, individual academic librarians are not in control of most final decisions. therefore, they do what they can to influence and impact the decision indirectly.  our workplaces have a huge impact on how we use evidence.   shifting the eblip paradigm   this research has shown me that two key parts of the current eblip model need to be reworked. 1) we need to look at a wider breadth of evidence sources within eblip, and move our discussions beyond research. librarians use many forms of evidence. this is legitimate and the eblip movement needs to catch up. 2) we need to consider how librarians do their work, and reframe the model so that it makes sense within our institutional, group-driven decision making, as opposed to independent, individual decision making. i propose to you the following points:   1)  we are not health care professionals: sources of evidence in health do not necessarily transfer into sources of evidence in librarianship. it is time to look at ourselves rather than model another profession.   2)  we have unique types of evidence within our profession.   3) we rarely act alone – we work in institutions and make decisions in groups.   4) we almost always act locally.   5) we care about what we do and want to influence outcomes.   6) we don't know enough about ourselves as decision makers.   7) we don't know enough about what are the most important evidence sources to help us.   keeping these points in mind, i want to propose we start to follow a new model of eblip which is not radically different, but which suits us better.   in 2009, booth proposed a new 5as of eblip which focus more on collaboration. this model is a better representation of the eblip process as it applies to librarians and fits very well with what i found in my study. it accounts for multiple sources of evidence; focuses on group decision making; and places evidence within the overarching problem and environment. it also encourages consensus building and adaptation as part of a cyclical process towards successful implementation, and gives more consideration to the areas of apply and assess, in the newly named ‘agree’ and ‘adapt’ stages. this version of the 5as is more holistic and encompassing of the complex process of evidence based decision making, as well as more practical. booth himself noted that his model was a work in progress; a prototype which had potential to be modified.   my doctoral study results fit very well with this model for eblip. in my thesis i build upon booth’s work to enhance this model further. booth had based his new model on threads of discussions happening at the eblip5 conference in stockholm. while not research, it arose from keen insightfulness of the discussions within a community of practice. my research has now confirmed that this model is a better fit for librarians that the original model.   in addition to booth’s alternative model, work i previously published, based on a presentation at the eblip6 conference which grew out of an earlier phase of this study, is drawn upon for the new model (koufogiannakis, 2011). this work focuses on questions that a practitioner should ask themselves when making professional decisions in an evidence based manner. these questions account for both hard and soft sources of evidence, with a focus on continually asking questions and improving practice.   my work combines well with that of booth’s to create a more holistic approach to practicing librarianship in an evidence based way.  a key point however, is that we shouldn’t focus on the model we need to do what works. a model itself can serve as a guide but should be flexible. this then is a new model for evidence based library and information practice:   1)       articulate – come to an understanding of the problem and articulate it.   questions: what do i/we already know about this problem? clarify existing knowledge and be honest about assumptions or difficulties that may be obstacles. this may involve sharing background documents, having an honest discussion, and determining priorities. consider the urgency of the situation, financial constraints, and goals.   actions: set boundaries and clearly articulate the problem that requires a decision.   2)       assemble – assemble evidence from multiple sources that are most appropriate to the problem at hand.   questions: what types of evidence would be best to help solve this problem? what does the literature say? what do those who will be impacted say? what information and data do we have locally? do colleagues at other institutions have similar experiences they can share? what is the most important evidence to obtain in light of the problem previously articulated?   actions: gather evidence from appropriate sources.   3)       assess – place the evidence against all components of the wider overarching problem. assess the evidence for its quantity and quality.   questions: of the evidence assembled, what pieces of evidence hold the most weight? why? what evidence seems to be most trustworthy and valid? what evidence is most applicable to the current problem? what parts of this evidence can be applied to my context?   actions: evaluate and weigh evidence sources. determine what the evidence says as a whole.   4)       agree – determine the best way forward and if working with a group, try to achieve consensus based on the evidence and organisational goals.   questions: have i/we looked at all the evidence openly and without prejudice? what is the best decision based on everything we know from the problem, the context, and the evidence? have we considered all reasonable alternatives? how will this decision impact library users? is the decision in keeping with our organisation’s goals and values? can i explain this decision with confidence? what questions still remain?   actions: determine a course of action and begin implementation of the decision.   5)       adapt –revisit goals and needs. reflect on the success of the implementation.   questions: now that we have begun to implement the decision, what is working? what isn’t? what else needs to be done? are there new questions or problems arising?   action: evaluate the decision and how it has worked in practice. reflect on your role and actions. discuss the situation with others and determine any changes required.   a model for eblip needs to look at all evidence, including evidence driven by practice as well as research. librarians need to take a different view of how evidence may be used in practice, and tie research and practice together rather than separating them. practitioners bring evidence to the table through the very action of their practice. the local context of the practitioner is the key, and research cannot just be simply handed over for a practitioner to implement. the practitioner can use such research to inform themselves, but other components are also important. concepts related to practice theory, focusing on the practitioner and his or her knowing in practice – both local evidence and professional knowledge – help to provide a more complete picture of decision making within the profession of librarianship.   the eblip model must be revised so that the overall approach addresses other aspects of evidence. all forms of evidence need to be respected and the lis professional, with his or her underlying knowledge (a part of soft evidence), is at the centre of the decision making process. different types of evidence need to be weighed within the context in which they are found, and only the practitioners dealing with that decision can appropriately assign value and importance within that context.   there must be an emphasis on applicability, because decision making is ultimately a local endeavour. in every situation, we must work within restrictions. these elements are facts of life and cannot be ignored. within such boundaries librarians need to weigh appropriate evidence and make contextual decisions. an evidence based library and information practitioner is someone who undertakes considered incorporation of available evidence when making a decision. an evidence based practitioner incorporates research evidence, local sources of data, and professional knowledge into their decision making.  all three must be present. moving forward   there are two large areas of research i would like to see the eblip community address in the next few years:   1. what are the best evidence sources based on the type of question? it would be beneficial for researchers to explore and recommend the best evidence sources based on the type of question. this would not be a hierarchical list, but would serve as a guideline on what sources of evidence a librarian should consult for that type of question. for example, if one has a collections problem, the research literature should be consulted, but other sources of evidence that would provide good information include usage statistics for e-products, circulation statistics, faculty priorities, output of tools such as oclc collection analysis, interlibrary loan and link resolver reports, as well as the publication patterns of faculty. researchers could determine what the sources are for each area of practice, and in what circumstances they are best used.   2. how do we “read” the results of different types of evidence sources? it would also be very beneficial for practitioners to have guidance on how to “read” the results of different evidence sources. for example, what a practitioner needs to consider when looking at reference statistics, or what elements a librarian should consider when conducting an evaluation of their teaching. some of this information will be found in existing literature, and a scoping review of what has already been documented would be a good start. we have already done this with the development of critical appraisal tools for research studies and it would be beneficial to extend this work to other types of evidence sources.   evidence helps us find answers   to close, i’d like to encourage you all to keep thinking about evidence and how you use it. ultimately, evidence, in its many forms, helps us find answers. however, we can’t just accept evidence at face value. we need to better understand evidence otherwise we don’t really know what ‘proof’ the various pieces of evidence provide. eblip has already made great strides towards better understanding research evidence, and while we need to continue to improve our research literature, we also need to extend that effort towards understanding other types of evidence that is used in librarianship.   i think we can only do this if we question, test and allow ourselves and one another to make mistakes while learning and exploring.   what excites me about all this is very much in keeping with the theme of this conference – “the possibilities are endless”. there are endless questions, endless ideas, and we all have something to contribute. in fact, we all need to contribute. above all, eblip is a mindset a way of approaching practice with openness and curiosity take time during this conference to listen, be inspired and discover your possibilities.     references   association of college and research libraries. (2011). guidelines for instruction programs in academic libraries.  retrieved 6 november 2013 from http://www.ala.org/acrl/standards/guidelinesinstruction   booth, a. (2000, july). librarian heal thyself: evidence based librarianship, useful, practical, desirable? 8th international congress on medical librarianship, london, uk.   booth, a. (2009). eblip five-point-zero: towards a collaborative model of evidence-based practice. health information and libraries journal, 26(4), 341-344. doi: 10.1111/j.1471-1842.2009.00867.x   booth, a. (2012). evidence based library and information practice: harnessing professional passions to the power of research. new zealand library & information management journal, 52(4). http://www.lianza.org.nz/resources/lianza-publications/nzlimj/evidence-based-library-and-information-practice-harnessing-prof   evidence. (2013). in oxford dictionaries pro. retrieved 6 nov. 2013 from http://www.oxforddictionaries.com/definition/english/evidence?q=evidence   glasby, j., walshe, k., & harvey, g. (2007). making evidence fit for purpose in decision making: a case study of the hospital discharge of older people. evidence & policy, 3(3), 425-437. doi: 10.1332/174426407781738065   hayward, r. s. (2007). evidence-based information cycle. centre for health evidence. retrieved 8 nov. 2013 from http://www.cche.net/info.asp   hornikx, j. (2005). a review of experimental research on the relative persuasiveness of anecdotal, statistical, causal, and expert evidence. studies in communication sciences, 5(1), 205-216.   koufogiannakis, d. (2011). considering the place of practice-based evidence within evidence based library and information practice (eblip). library & information research, 35(111), 41-58. retrieved 8 nov. 2013 from http://www.lirgjournal.org.uk/lir/ojs/index.php/lir/article/view/486/527   koufogiannakis, d. (2012). academic librarians’ conception and use of evidence sources in practice. evidence based library and information practice, 7(4), 5-24. retrieved 8 nov. 2013 from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/18072/14468   koufogiannakis, d. (2013a). academic librarians use evidence for convincing: a qualitative study. sage open, 3(april-june). doi: 10.1177/2158244013490708   koufogiannakis, d. a. (2013b). how academic librarians use evidence in their decision making: reconsidering the evidence based practice model. (unpublished doctoral dissertation). aberystwyth university, wales, u.k.   partridge, h., edwards, s., & thorpe, c. (2010). evidence-based practice: information professionals’ experience of information literacy in the workplace. in a. lloyd, & s. talja (eds.). practising information literacy: bringing theories of learning, practice and information literacy together. wagga wagga, nsw: charles sturt university.   reynolds, r. a., & reynolds, j. l. (2002). evidence. in j. p. dillard, & m. pfau (eds.). the persuasion handbook: developments in theory and practice. (pp. 427-444) thousand oaks, ca: sage.   rieke, r., & sillars, m. o. (1984). argumentation and the decision making process. new york: harper collins.   schum, d. (2011). classifying forms and combinations of evidence: necessary in a science of evidence. in p. dawid, w. twining, & m. vasilaki (eds.). evidence, inference and enquiry. (pp. 11-36). oxford: oxford university press.   thorpe, c., partridge, h., & edwards, s. l. (2008). are library and information professionals ready for evidence based practice? paper presented at the alia biennial conference: dreaming 08, alice springs, nt, australia. retrieved 8 nov. 2013 from http://conferences.alia.org.au/alia2008/papers/pdfs/309.pdf   twinning, w. (2003). evidence as a multi-disciplinary subject. law, probability & risk, 2(2), 91-107. doi: 10.1093/lpr/2.2.91   upshur, r. e., vandenkerkhof, e. g., & goel, v. (2001). meaning and measurement: an inclusive model of evidence in health care. journal of evaluation in clinical practice, 7(2), 91-96. doi:10.1046/j.1365-2753.2001.00279.x       microsoft word art_yu.doc evidence based library and information practice 2006, 1:2  12 evidence based library and information practice     article    what can students’ bibliographies tell us? evidence based information skills  teaching for engineering students      fei yu  liaison librarian  dorothy hill physical sciences and engineering library  university of queensland  st. lucia, queensland, australia  e‐mail: f.yu@library.uq.edu.au    jan sullivan  liaison librarian  dorothy hill physical sciences and engineering library  university of queensland  st. lucia, queensland, australia  e‐mail: j.sullivan@library.uq.edu.au    leith woodall  liaison librarian  dorothy hill physical sciences and engineering library  university of queensland  st. lucia, queensland, australia  e‐mail: l.woodall@library.uq.edu.au      received: 20 april 2006   accepted: 16 may 2006      © 2006 yu, sullivan, and woodall. this is an open access article distributed under the terms of the creative  commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    abstract    objective ‐ this project sought to identify students’ strengths and weaknesses in locating,  retrieving, and citing information  in order to deliver information skills workshops more  effectively.      http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:2  13  methods ‐ bibliographies submitted from first‐year engineering and second‐ and fourth‐ year chemical engineering students’ project reports were analysed for the number of items  cited, the variety of items cited, and the correct use of citation style.  the topics of the  project reports were also reviewed to see the relationships between the topics and the items  cited.      results ‐ the results show that upper level students cited more items in total than did lower  level students in their bibliographies.  second‐ and fourth‐year engineering students cited  more books and journal articles than first‐year students cited.  web sites were used  extensively by all three groups of students, and for some first‐year students these were the  most frequently used sources.  students from all three groups had difficulties with citation  style.      conclusion ‐ there was a clear difference in citation frequency between upper and lower  level engineering students.  different strategies of information skills instruction are needed  for different levels of students.  librarians and department faculty members need to include  good quality internet resources in their teaching and to change the emphasis from finding  information to finding, interpreting, and citing accurately.      introduction    one of the core duties of academic librarians  is to help university students develop  information skills.  in designing their  training programs, librarians are faced with  making practical decisions about the content  and delivery of the training.  eldredge  claimed “evidence‐based librarianship offers  a decision‐making framework” (290).  this  study used the principles of evidence based  librarianship in its expectation of finding  evidence from students’ bibliographies to  help teaching librarians develop improved  information skills classes for engineering  students.      the dorothy hill physical sciences and  engineering library at the university of  queensland offers many tailored  information skills workshops to students at  various levels of study and in different  disciplines in the faculty of engineering,  physical sciences and architecture.   tailored workshops were presented to  project design students in first‐year  engineering and second‐year and fourth‐ year chemical engineering programs.  the  primary goal of the workshops was to  develop the students’ ability to locate and  retrieve the information needed for their  projects.  time was also spent teaching  citation techniques to be used in their  project reports.   although general use of internet search  engines was not usually encouraged, it is  acknowledged as an important source for  certain types of information.  anecdotal  evidence suggests that students tend to use  the internet to find information, rather than  using scholarly databases for journal articles  or searching library catalogues for books.   however, no study has examined the  information sources used by engineering  students for their design projects, or  determined what differences exist between  first‐year students and upper level students.      another increasing concern among  librarians, as well as teaching academics, is  that students have very little knowledge of  or skill in citing the information correctly.   again, few studies have examined this topic.   the question was whether instruction  librarians should spend more time in  training students how to cite information.  evidence based library and information practice 2006, 1:2  14 because the university is taking an  increasingly hard line on plagiarism,  students need to be aware of the necessity of  acknowledging other people’s work and  ideas by citing them in the bibliographies of  their research work.    this study was designed to find evidence  showing students’ strengths and  weaknesses in locating, retrieving, and  citing information, and the study also  sought to explore the implications for future  improvements in information skills teaching.       literature review    in order to help design information skills  programs, academic librarians have used a  range of research methods to identify the  information sources used by students in  their assignments.  these techniques include  survey questionnaires, pre‐testing and post‐ testing, focus groups, and interviews.      majid and tan used a questionnaire‐based  survey in their research and determined that  although books are students’ preferred  source of information, internet search  engines such as google and lecture notes  were the next favourite sources. use of  computer databases and electronic journals  was surprisingly low (318).  palmer and  tucker conducted an evaluation of the  information skills literacy program at  deakin university (geelong, victoria,  australia) using a combination of pre‐ and  post‐tests with questionnaires.  they  concluded that the majority of first‐year  engineering and technology students were  familiar with basic types of information  sources such as books, journals, and internet  websites.  palmer and tucker used  evaluation questionnaires to obtain  students’ feedback on improving  information literacy skills activities.   students recommended that more time “be  directed to developing the concepts of  complex reference types, such as chapters in  books, papers in conference proceedings,  and other forms of academic monographs  with multiple authors” (28).     tenopir’s study found that engineers read  fewer journal articles than scientists or  medical researchers (157).  another study  found that engineers rely heavily on  interpersonal communication with  colleagues and social networks (fidel and  green 564).  similar information using  behaviour may be reflected in engineering  students’ bibliographies for their writing  assignments, however little literature has  been identified on this topic.    citation analysis is another technique used  in research studies.  it is widely used in  areas such as library collection policy  development, knowledge base structure  analysis, and academic research assessment.   however, it has had limited use in relation  to design and evaluation of information  skills programs in engineering.  one study  used citation analysis to identify the  information sources biology honours  students used for assignments (kraus 175).   citation analysis has also been used to  examine the accuracy of students’ citations  of information sources in their assignments.   in testing an online tutorial tool to track  students’ referencing errors, kendall found  many students experience difficulties with  citing and referencing bibliographic sources  (131).  when referencing both books and e‐ documents, first‐year students made more  errors than postgraduates (62% for books vs.  44%, 85% for e‐documents vs. 80%).  the  errors included missing place of publication,  missing publisher details, missing date of  publication, incorrect format for author’s  name,  incorrect location of title in citation,  and incorrect format for title (kendall).    in summary, a range of techniques such as  questionnaires and pre‐ and post‐tests have  been used in information skills research  studies. only a few studies have been found  evidence based library and information practice 2006, 1:2  15 where citation analysis was used to find  evidence of how students use information  sources and the problems students have  when citing the references in their  assignments. few studies have analysed  undergraduate engineering students’  bibliographies to see what information  sources they use and how they cite the  sources.      the current study was designed to analyse  undergraduate engineering students’  bibliographies to confirm the findings from  previous research using techniques other  than citation analysis and to help determine  the components of information skills  teaching and training.    the aim of the study    the ultimate purpose of this study is to use  evidence found in research to plan, prepare,  and systematically design appropriate  information skills training for different  levels of engineering students.  in order to  achieve this purpose, we looked to see if  there was a relationship between the  research topic and the type of information  sources used.  past experience has shown  that for some topics, students tend to use  one source of information more than others.   however, no research has confirmed or  refuted this observation from anecdotal  evidence.    the librarian instructors involved in this  research examined the bibliographies to see  if the items were correctly cited following a  known citation format.  the librarian  instructors regarded the ability to compile a  comprehensive and correctly cited  bibliography as a skill that all university  students should acquire.      methodology    in this study, the bibliographies of design  project reports from first‐year engineering  students and second‐ and fourth‐year  chemical engineering students were  analysed.  in total, 134 first‐year, 15 second‐ year and 8 fourth‐year bibliographies were  collected and analysed.  (third‐year  chemical engineering students were not  required to do project reports.)   the authors  examined each bibliography and tallied the  cited references.  the results were entered in  an excel spreadsheet to identify:     • the number of items cited in each  bibliography and the total number  cited for each class;    • the number of information sources  used and the relationship between  project titles and information  sources used; and     • the correct use of citation style (e.g.,  harvard or vancouver).      for information sources used, the authors  examined the variety of sources and the  total number of items used from each source.   to determine the correct use of citation style,  the authors examined the layout of the  bibliography and evaluated the accuracy of  citation format, including the presence of  appropriate parts of the citations,  consistency of citation style, capitalisation,  italics, and punctuation.    statistical analysis involved the use of a two  tailed t‐test to show the degree of  significance in a comparison of the three  groups to see if there were any patterns,  both in terms of using information sources  and in citing them.    results     the research identified three categories of  results: the total number of items cited; the  number of information sources used, and  the relationship between project titles and  the type of information sources used. evidence based library and information practice 2006, 1:2  16     table 1: number of items in first‐year, second‐year and fourth‐year student bibliographies      total number of items cited    there was a clear pattern that higher year  students cited more items in total than lower  year students with first years citing an  average number of 17.6 items per  bibliography and fourth year students citing  an average of 77.1 (table 1).    a detailed examination of the references  showed that 4 first‐year bibliographies  contained less than 5 items; 84 contained  between 5 and 20 references; 38 contained  between 21 and 40 references; 8 contained  between 41 and 60 items; and there were  none containing more than 60 items.  in  contrast, there were no fourth‐year students  with less than 5 items in their bibliographies,  and there were 4 with more than 60 items  (table 2).        number of information sources used    five types of information sources were  examined: web sites, journal articles, books,  personal communications and lecture notes,  and other specialised information such as  standards, patents, and theses.  references  to each type of source were tabulated for  each class, and percentages were calculated.   a one‐tailed t‐test was also calculated to  document the significant differences  between the three groups of students’  bibliographies in terms of the sources used  by each group.      as students progress in years, they tend to  rely less on web sites as information sources  (table 3).  use of books shows a reverse  trend: first‐ year students use books less  than students in upper level classes.  none  of the three groups showed an obvious  preference in the use of journal articles.         table 2: detailed breakdown of items in first‐year, second‐year and fourth‐year student bibliographies    bibliography items  first‐year  (n=134)  second‐year   (n=15)  fourth‐year  (n=8)  total number of items  2356    312  615    average (mean) number of  items in each bibliography    17.6  20.8  76.9      first‐year  (n=134)  second‐year   (n=15)  fourth‐year  (n=8)     1 ‐ 4  4  0  0    5 ‐ 20  84  8  1  21 ‐ 40  38  6  1  41 ‐ 60  8  1  2    total number of items in  each bibliography  60 ‐  0  0  4  evidence based library and information practice 2006, 1:2  17   table 3: analysis of information sources used in first‐year, second‐year and fourth‐year students’  bibliographies      a two‐tailed t‐test showed there were  significant differences between three of the  student groups with regard to the number  of books used as resources for the project  bibliographies:    • first‐year and second‐year  students    • first‐year and fourth‐year  students  • second‐year and fourth‐year  students    there was also a significant difference  between second‐ and fourth‐year and  between first‐ and fourth‐year student  groups in relation to the total number of  items cited in the project bibliographies  (table 4).     table 4: mean number of items cited in first‐year, second‐year and fourth‐year student bibliographies   information sources  first‐year  (n=134)  second‐year  (n=15)  fourth‐year  (n=8)    #  %  #  %  #  %  web sites  1582  67.1  173  55.5  141  22.9  books    536  22.7  102  32.7  363  58.8  journals    194    8.2    22    7.1    71  11.5  personal communications and  lecture notes      14    0.6      4    1.3      2    0.3  other specialized information  (e.g., patents, cds, theses,  standards)      30    1.3     11    3.5    38    6.2  total   2356     312     615    information sources  first‐year   second‐year  fourth‐year  significance  web sites  11.8  11.5  17.6    books  4.0   6.8  45.5  a, b, c  journals  1.4   0.3    8.9    personal communication  and lecture notes  0.1     0.29      0.25    other specialized  information (e.g., patents,  cds, theses, standards)  0.2   0.7    4.5    total  17.6  20.8  77.5  b, c    a =  significant difference between first‐ and second‐year p<0.05  b =  significant difference between second‐ and fourth‐year p<0.05  c =  significant difference between first‐  and fourth‐year p<0.05  evidence based library and information practice 2006, 1:2  18   table 5: relationship between project topics and first‐year students’ bibliographies   note: numbers below grey line are per cents (%)    first‐year students did not use many  textbooks.  second‐year students used  encyclopaedias and handbooks extensively,  and most of them cited the three core  encyclopaedias recommended by the  lecturer.    relationship between project titles and the type  of information sources used    the relationship between project titles and  information sources was examined for the   first‐year students, because the project  topics for that class dealt with a wide range  of engineering issues.  for example, project  1 involved detailed mechanical design  processes in a very specific field, whereas  project 3 required students to consider many  social, economic, and financial issues while     evaluating solutions to traffic problems.   the authors assumed, therefore, that  different types of information sources  would be used for different projects.  the  students received few recommendations for  information resources from their lecturers.    results shown in table 5 confirm that  students used fewer web sites for project 1  (27.5% of the total items in their  bibliographies), compared to students  completing project 3 (80.2% of the total  items in their bibliographies).  project 1  students used more books (67.5%) than  students completing project 3 (2.9%).   project 1 students used more books than any  other group.   project    1  (n=19)  2  (n=23)  3  (n=17)  4  (n=18)  5  (n=22)  6  (n=11)  7  (n=16)  8  (n=6)  total no. of  items  120  470  379  460  360  246  195  92                    web sites   27.5  59.8  80.2  77.6  70.0  77.2  50.3  50.0    journals,  newspapers     4.2  10.4    7.7    8.0  10.0    3.3    8.2    4.3    books   67.5  29.1    2.9  13.3  18.9  14.6  37.9  42.4    personal  communication  and  lecture  notes   0  0    0.3  0.7  0    2.0    2.6  0    other  specialized  information  (e.g., patents,  cds, theses,  standards)     0.8    0.9    2.1  0  0    2.4    1.0    3.3  evidence based library and information practice 2006, 1:2  19       correct use of citation style    there was no apparent evidence that  second‐ and fourth‐year students performed  better than first‐year students in terms of  bibliography layout or accuracy of citations  (table 6).  the fourth‐year students had the  lowest percentage scores on the assessment  of their use of appropriate parts of citations.    discussion and implications     analysis of the bibliographies showed  students in all years were able to find a  variety of relevant information.  they were  not so successful in citing the information.    the analysis showed second‐year and  fourth‐year students tended to use web sites  less and books and journal articles more  than first‐year students.  both second‐ and  fourth‐year students used core chemical  engineering materials, such as  encyclopaedias handbooks, and prescribed  textbooks, because their project topics were  all on process design in chemical  engineering.  this corresponds with the  findings of majid and tan who found that  books and the internet are undergraduate  students’ favourite sources (318). it also  supports tenopir’s findings that engineers  read fewer journal articles (157).    citation    analysis  first‐year (n=134)  second‐year  (n=15)  fourth‐year (n=8)      #  %  #  %  #  %  layout of  bibliography  list not divided by  reference type    118  88.1  14  93.3  7  87.5    list in alphabetic order  82  61.2  n/a  n/a  n/a  n/a    list not numbered or  bulleted    104  77.6  n/a  n/a  n/a  n/a  accuracy of  citation  presence of  appropriate parts of  citation    109  81.3  13  86.7  5  62.5    consistent use of a  citation style    107  79.9  14  93.3  6  75    capitalisation  22  16.4  n/a  n/a  n/a  n/a    italics  80  59.7  n/a  n/a  n/a  n/a    punctuation  87  64.9  n/a  n/a  n/a  n/a    table 6: citation style analysis of first‐year, second‐year and fourth‐year students’ bibliographies  note: n/a ‐ students in second‐ and fourth‐year did not receive a recommended style, so these analyses  were not always relevant.  evidence based library and information practice 2006, 1:2  20 first‐year students, on the other hand, have  not yet specialised in a particular subject  area, and their project topics are more  general.  the purpose of the students’  research projects was to give them an  opportunity to learn about the engineering  profession.  there were few required  readings from lecturers.  since these  students had recently graduated from high  school and were not aware of the core  resources for university courses, it is likely  that they turned to the internet for  information.  however, there were a few  projects for which the lecturers thought web  sites provided the best sources for  information.  in these cases, the focus of  information skills teaching could be about  ways to evaluate internet information, how  to cite the information in the text, and how  to reference information in the bibliography.      the comparison between types of  information sources used by students  showed that all year levels tend to use ‘basic  reference types’ ‐ such as books, journal  articles, and the internet ‐ more than  specialised information sources such as  standards, theses, and patents.  this  supports palmer and tucker’s findings that  students are familiar with basic types of  information sources such as books, journals,  and internet web sites, but they are not  familiar with more complex resources.  as  suggested by palmer and tucker, our  information skills teaching should focus  more on those complex reference types for  upper level students.    we observed that some students, especially  first‐year students, could not differentiate  between a journal and a journal article, and  they did not understand why they needed  journal articles.  this knowledge will help  us redesign our information skills  workshops for first‐year students, to include  more explanation of the content and value  of books, journal articles, and web sites.      there was clear evidence that all year  groups had problems in citing information  accurately.  this was consistent with  kendall’s findings (131).  one common  citation error among all year groups’  bibliographies in this study was  inconsistency in listing author names.  there  were missing parts in author names and in  some instances surname and first name  were not in the right order.  students at all  year levels also had difficulties citing  personal communications and specialised  documents. it seemed that they had trouble  deciding what information should be  included in the citation and what could be  omitted.      many first‐year students received low  marks for the accuracy of their citations  (table 6 ).  only 3% achieved full marks for  citation accuracy, while 44% achieved full  marks in citation layout.   the central  problem we observed from the fourth‐year  students’ project bibliographies was  inconsistency.  the fourth‐year design  project report consisted of three volumes,  and each volume was authored by a  different student from the group.  each  submitted a list of references which often  followed different citation styles.   inconsistency within a bibliography  included different layout, different citation  styles, and sometimes repeated references.      in order to address the citation accuracy  problems mentioned above, exercises to  help students recognise different parts of a  citation and to familiarise them with  different citation styles should be designed  and included in information skills  workshops.  librarians also need to work  with students to better understand the  dynamics of student groups and in finding  the person in their group best suited to  having overall responsibility for producing  the bibliography.    evidence based library and information practice 2006, 1:2  21 the most important finding of this study is  that citing information was clearly a  weakness for all student groups in this  project.  at all year levels, more time should  be spent on teaching students how to cite  information accurately.  the importance of  correctly citing information should be  explained and stressed in all information  skills training.      academic staff also expressed concern that  the in‐text citations at all year levels did not  always match the bibliographies.  this  aspect needs to be examined more closely.   future information skills training may need  to incorporate in‐text citing for students at  all levels.    the findings indicate that systematic  planning of information skills teaching and  training should be conducted, and learning  outcomes need to be revisited for all levels  of students.  with first‐year students, we  need to concentrate on the basics and build  on this progressively in later years.  for  example, with the first‐year students we  should emphasise finding, interpreting, and  citing books, journal articles, and web sites.   for second‐year students, the emphasis  would be on chapters of books and  encyclopaedias.  personal communications  and specialised publications such as  conference papers, patents, and standards  would be covered in the fourth‐year.  in  later years, to help produce the bibliography  in the correct citation style, reference  management software such as endnote  could also be introduced.      conclusion    this study showed that there was a clear  difference in citation frequency between  upper and lower level engineering students.  upper level students cited more total items  in their bibliographies than did first‐year  students.  regarding the variety of items  cited, second‐ and fourth‐year engineering  students cited more books and journal  articles than first‐year students, but websites  were still used extensively by students in all  years.   some evidence showed that for a  few first‐year design project topics, students  used more internet sources than other  sources of information.  librarians and  department faculty members need to select  good quality internet resources to include in  information skills teaching and training.    students in each group had difficulties  citing personal communication and other  specialised information sources.  they also  had various problems in consistency of  citation style and inclusion of parts of  citations such as author names.  these  findings indicate that different strategies in  information skills teaching and training are  needed.  the emphasis needs to change  from merely finding information to finding,  interpreting, and citing it accurately.   systematic planning is needed to gradually  build on students’ information skills  portfolios.      acknowledgement  a version of this paper was presented at the  3rd international evidence based  librarianship conference held in brisbane,  australia (october 2005).  see:  http://conferences.alia.org.au/ebl2005/yu.pd f    works cited    eldredge, jonathan d. ʺevidence‐based  librarianship: an overview.ʺ bulletin of  the medical library association 88.4  (2000): 289‐302. 28 may 2006  <http://www.pubmedcentral.gov/article render.fcgi?tool=pubmed&pubmedid=1 1055296>.    fidel, raya, and maurice green. ʺthe many  faces of accessibility: engineersʹ  perception of information sources.ʺ  http://conferences.alia.org.au/ebl2005/yu.pd http://www.pubmedcentral.gov/article evidence based library and information practice 2006, 1:2  22 information processing & management  40.3 (2004): 563‐81.    kendall, margaret. ʺtackling student  referencing errors through an online  tutorial.ʺ aslib proceedings: new  information perspectives 57.2 (2005):  131‐45.    kraus, joseph r. ʺcitation patterns of  advanced undergraduate students in  biology, 2000‐2002.ʺ science and  technology libraries 22.3/4 (2002): 161‐ 79.    majid, shaheen, and ai tee tan. ʺusage of  information resources by computer  engineering students: a case study of  nanyang technological university,  singapore.ʺ online information review  26.5 (2002): 318‐25.    palmer, stuart, and barry tucker. ʺplanning,  delivery and evaluation of information  literacy training for engineering and  technology students.ʺ australian  academic and research libraries 35.1  (2004): 16‐35. 28 may 2006  <http://alia.org.au/publishing/aarl/35.1/f ull.text/palmer.tucker.html>.    tenopir, carol, and donald w. king.  communication patterns of engineers.  hoboken, nj: ieee press; wiley  interscience, 2004.      http://alia.org.au/publishing/aarl/35.1/f evidence based library and information practice evidence based library and information practice 2010, 5.1  evidence based library and information practice       article    evaluating the development of virtual communities of practice that support evidence  based practice    christine urquhart  senior lecturer  department of information studies  aberystwyth university, united kingdom  email: cju@aber.ac.uk    anne brice  department of knowledge & information sciences  public health resource unit, oxford, united kingdom  email: anne.brice@dphpc.ox.ac.uk    janet cooper  department of information studies, aberystwyth university  aberystwyth, wales, united kingdom    siân spink   department of information studies, aberystwyth university  aberystwyth, wales, united kingdom     rhian thomas  department of information studies, aberystwyth university  aberystwyth, wales, united kingdom  email: ret@aber.ac.uk    received: 01 dec. 2009          accepted: 12 feb. 2010       2010 urquhart, brice, cooper, spink and thomas. this is an open access article distributed under the terms of  the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada  (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and  reproduction in any medium, provided the original work is properly attributed, not used for commercial  purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.    abstract    objective – the aim of this paper is to examine how virtual community of practice  principles might be used by information professionals with emphasis on the work of  the specialist libraries for health professionals in england, uk. an evaluation  conducted in 2004‐2005 examined the operation of the specialist libraries, which the  48 mailto:anne.brice@dphpc.ox.ac.uk mailto:ret@aber.ac.uk evidence based library and information practice 2010, 5.1  49 national library for health had contracted out to various organisations, and assessed  their stage of development as communities of practice.     methods – evaluation methods included observation of a meeting of information  specialists, interviews with clinical leads and information specialists, and evaluation  of the content and format of the specialist library websites. the evaluation framework  was based on a systematic review of the literature to determine the critical success  factors for communities of practice and their role in supporting evidence based  practice. an updated literature review was conducted for this paper.    results – operational structures varied but were mostly effective in producing  communities of practice that were at an “engaged” stage. some specialist libraries  wished to move towards the “active” stage by supporting online discussion forums, or  by providing question and answering services or more learning activities and  materials. although the evidence from the literature suggests there are few clear  criteria for judging the effectiveness of communities of practice, the evaluation  framework used here was successful in identifying the state of progress and how  information professionals might approach designing virtual communities of practice.    conclusions – structuring library and information services around community of  practice principles is effective. careful and participative design of the information  architecture is required for good support for evidence based practice.     introduction    the nhs evidence health information  resources website (formerly the national  library for health, nlh and national  electronic library for health, nelh) aims to  deliver health library services to nhs staff,  students, patients and carers in england, and  to support and promote evidence based  decision making. at the time of the empirical  research described in this paper, the aim was  to offer a range of services that mixed  traditional services, such as professional  librarian support, with new web based  services such as a comprehensive search  engine, clinical question answering, and  current awareness services.    a major component of the nlh portal, the  specialist libraries section, was established in  2003 to make it easier for distinct health care  communities to access and transfer  knowledge. specialist libraries were based  around three types of specialty: disease  groupings (e.g., cancer, cardiovascular or skin  diseases), populations (e.g., children or older  people), and different types of healthcare  activity (e.g., complementary medicine or  clinical genetics). specialist libraries (now  renamed specialist collections) aim to identify  and engage all key stakeholders and partners.    definitions of communities of practice vary,  but one main proponent, wenger (1998),  emphasises how communities of practice  foster social learning – learning that takes  place within practice, based on observation  and interactions (p. 96). the nlh’s aim, via  the specialist libraries, was to ensure that  practitioners in different specialties could  make sense of evidence based practice in ways  that were meaningful to them, and apply  lessons learned by others in their specialist  area. while many private sector organisations  have embraced communities of practice as a  way of preserving and fostering knowledge,  large public sector organisations such as the  national health service (uk) seem to have  been slower in adopting these knowledge  management solutions. the specialist  libraries were designed on evidence based  principles to ensure that as far as possible they  worked as communities of practice should.  the findings of an evaluation conducted by  evidence based library and information practice 2010, 5.1  the aberystwyth university in 2004‐2005 were  intended to identify the barriers and enablers  to further the development of the specialist  libraries as communities of practice. this  paper outlines how those principles were  derived, how the evaluation was conducted,  and the main findings. discussion and  conclusions emphasise lessons that may be  transferred to other information services and  health organisations, using the findings of a  review of the literature as well as the study  results.    aims, objectives, and context    the aim of the evaluation was a mid‐term  review of the development of the specialist  libraries. the primary objectives were to  assess whether the 19 specialist libraries were  operating as intended, whether they were  developing as communities of practice, and  what stage they had reached. secondary  objectives included examining how the  information specialists supporting the  specialist libraries were themselves operating  as a community and to examine the links  between the specialist libraries and other  health organisations such as professional  bodies and patient support groups (their  advisory group structure).    some of the specialist libraries had been  formed in 2004, others had been in existence  for at least three years (operating as “virtual  branch libraries” within the nelh). their  structures varied slightly according to their  host organisation, but the usual management  team for a specialist library had a clinical  lead, an information specialist, support by an  editorial board to help appraise quality and  approve content development strategies, and a  reference group of other organisations and  groups (including patient support groups)  with an interest in the clinical speciality. some  information specialists supported more than  one specialist library.    methods    the methods used for the evaluation included:  • revalidation of framework criteria for  appraising the websites of the  specialist libraries (details below)  • evaluation of the 19 websites against  the framework criteria to determine  the stage of development   • interviews with specialist library  teams – clinical leads (1 face‐to‐face  interview, 3 telephone interviews) and  information specialists (4 face‐to‐face  interviews, 2 telephone interviews,  plus one set of replies emailed back).    interviews were intended to complement the  website evaluations and provide answers to  some of the questions concerning future  development plans. questions (appendix 1)  explored perceptions of clinical leads and  information specialists about the functioning  of their specialist library, how the  management structures operated, and what  their priorities were. interviews were  transcribed and coded according to the  evaluation framework themes.    in addition, one member of the research team  attended the information specialists’ meeting  in bath from 8‐9 december, 2004 for  observation and background fact finding. in  total, 10 of 19 specialist libraries were  approached for more formal interviews, and  10 supplied information. a selection matrix  was used to ensure that the sample included  old and new libraries, various organisational  set‐ups, and clinical communities (e.g., chronic  conditions, acute specialities). several of the  information specialists service more than one  specialist library and the interviews with  those information specialists sought  comparisons with the work they did for both  specialist libraries.    developing and revalidating the framework  criteria    we believed two key concepts for  communities of practice were important for  the health sector. these were: 1) legitimate  peripheral participation (lave & wenger,  1991): and 2) the possible membership an  individual might have in several communities  50 evidence based library and information practice 2010, 5.1  of practice (wenger, 1998, p. 158). any health  community of practice needs to allow for new  members, such as those on clinical rotations to  participate on the fringes initially, but still to  recognise them as legitimate members of the  virtual community. in addition, health  professionals may belong to several  communities, representing their various  professional and research interests. an earlier  study had developed an evaluation  framework, based on a systematic review of  the literature (urquhart, yeoman, & sharp,  2002). the review identified community of  practice evaluations through web sites (white,  2001), literature searches (1995‐2001) on  databases (isi social citation index/web of  knowledge) using search terms such as  knowledge management, communities,  organisational learning and social learning. it  established that most evaluations of  community of practice projects have used a  variety of methods to provide triangulation of  the findings (mostly qualitative). one  longitudinal five‐year study by gongla &  rizzuto (2001) used participant observation,  activity measurement and structured  interviews. other approaches include the case  study approach (hildreth, kimble, & wright,  1998); focus groups plus quantitative methods  (moreno, 2001); action research (catino, 1999);  the american productivity quality council  benchmarking methodology (o’dell, hubert,  & mcdermott, 2001); and a social capital  framework (lesser & storck, 2001). although  many of the studies included for further  review used triangulation to support  conclusions drawn, none included sufficient  raw data to enable verification of the  interpretations and conclusions made.     the key themes to emerge from this synthesis  of the research evidence were:    • virtual communities may be  constituted formally or informally.  • trust often needs to be fostered  through face‐to‐face communication,  and trust is often consolidated  through working on a shared  document or task.  • communities of practice may evolve  through stages, but not necessarily in  true life cycle format. (moreno, 2001)     the stages of evolution identified in the major  longitudinal study in ibm (gongla & rizzuto,  2001) are:    • potential (connecting individuals)  • building (allowing individuals to  learn more about each other, share  experiences and knowledge, create  shared norms)  • engaged (emphasis on access and  learning to provide support to new  members and add to add to the  knowledge base)  • active (emphasis on collaboration and  shared work tasks)  • innovation and generation (to  develop new products and services  and even to spawn new communities  of practice).    by 2001, there had been some community of  practice trials (e.g., lau & hayward, 2000) in  the health sector. the limited evidence from  these indicated the importance of building  trust. the infrastructure or learning  architecture for a community of practice  (wenger, 1998, p. 230) should allow for  engagement (e.g., shared repositories,  storytelling, joint tasks), imagination (e.g.,  scenario development, explanations and  examples), and alignment (e.g., feedback and  audit mechanisms, mediation mechanisms).  the review demonstrated that the activities,  documents, and processes within a  community of practice would need to cater for  the differences in perspective (markus, 2001)  between novices, expert practitioners,  researchers and those involved in secondary  knowledge production (e.g., in systematic  reviews). a synthesis by nutley & davies  (2001) of various approaches to organisational  learning, as applied to the nhs, contrasts the  emphasis on the codification of knowledge, as  shown in standardised reporting schemes, and  publication of guidelines with the  “unlearning” processes required to change  established practice in the light of new  51 evidence based library and information practice 2010, 5.1  evidence. for the health sector, as in other  communities of practice, power and politics  cannot be ignored (fox, 2000), and  assumptions about norms of behaviour and  responsibilities for changing practice need to  be considered.     most of the published qualitative studies on  which the appraisal framework (below) was  based would not meet all the criteria for  reporting qualitative research, as details were  lacking. nevertheless, the framework was  successful in identifying that the “virtual  branch libraries” were functioning as  communities of practice (urquhart et al., 2002)  and therein to inform the specification for  commissioning the specialist libraries in 2003.    updated literature review    the literature review was updated for this  evaluation (2004‐2005) to identify trends in  social learning theories, recent community of  practice evaluations in the health sector, and  other related systematic reviews. no changes  to the original framework were deemed  necessary.     more recent literature, published since the  original (2001) and updated (2004) synthesis  for the evaluation work was identified  through a literature search of medline  (using pubmed related article searching) and  web of knowledge and by following up  references in relevant articles. this time, the  inclusion criteria were narrowed to systematic  reviews of communities of practice in the  health sector, comparative case studies,  research studies that included virtual and face‐ to‐face communities of practice for health  professionals, and theoretical frameworks that  might be applied to virtual health  communities of practice (2004‐ mid 2009  publication date).    several systematic reviews were identified. li  et al. (2007) compared business and health  communities of practice, but found no  evaluation studies in the health sector that  fitted their inclusion criteria. their synthesis of  literature (1991‐2005) identified four important  characteristics: social interaction among  members, knowledge sharing, knowledge  creation, and identity building – but these are  not consistently present in all communities of  practice. eysenbach, powell, englesakis, rizo,  & stern (2004) systematically reviewed the  effects of online peer‐to‐peer interactions in  virtual health communities of practice and  electronic support groups, but found no  robust evidence. an ongoing australian study  (braithwaite et al., 2009) did a content analysis  (greenfield, travaglia, nugus, & braithwaite,  2007) of the 624 publications on health systems  communities of practice identified in their  systematic review, which will contribute to an  evaluation methodology with evidence based  outcome measures for communities of practice  and social networks. a systematic review  (fung‐kee‐fung et al., 2008) of regional  collaboratives in surgery identified five  enablers for effective learning for best practice  (establishment of trust, reliable data, clinical  leadership, institutional support, and an  infrastructure for quality management).     dubé, bourhis, & jacob (2005), in a qualitative  analysis of 14 organisations (comparative case  study) developing virtual communities of  practice found that relevance, and a  facilitating environment were vital. although  intentionally formed communities may fail  and face‐to‐face communication is important  for initial socialization, it is not necessary to  build on existing informal networks. in  addition, the benefits of voluntary versus  mandatory participation are unclear.     some of the themes that emerge from these  studies – the evolving definitions of  communities of practice and the difficulty of  evaluating complex adaptive systems that do  not behave in a linear fashion, are echoed by  qualitative research by ferlie, fitzgerald,  wood, & hawkins (2005), whose analysis of  eight comparative case studies indicated that  tight professional boundaries may inhibit the  spread of innovations in evidence based  practice. innovation is not necessarily a neat  linear process, particularly as some  professional networks may operate  horizontally (between colleagues) and others  52 evidence based library and information practice 2010, 5.1  vertically (senior to junior staff) (greenhalgh,  robert, macfarlane, bate, & kyriakidou,  2004).    from the literature retrieved in the initial and  updated review, evidence was collated to  inform questions for this paper about the  design approach for communities of practice,  development risks, and how effectiveness  might be evaluated. the key themes are, with  the type and level of evidence indicated, are:    • contingent approaches to design are  desirable for trust and knowledge  sharing to develop.   it‐led communities may be more  difficult to engineer than it‐supported  communities (kling & courtright,  2003), and even in carefully  constructed knowledge networks  exposing gaps in knowledge can seem  risky (falkman, gustafsson, jontell, &  torgersson, 2008). benefits of  knowledge sharing need to be clear  and norms of behaviour in virtual  communities apparent (giordano,  2007). russell, greenhalgh, boynton,  & rigby (2004) indicate that the  success of an informal email network  to support evidence based care  depended on careful targeting and  filtering of messages.  level of evidence: mainly qualitative  analysis  • the ways of working and professional  practice and learning may adversely  affect multidisciplinary communities  of practice (ferlie et al., 2005).   different types of informal learning  exist, and the concept of “situated  learning” needs to be reviewed (eraut,  2000). forums which are ostensibly  multi‐professional are in fact medical  (dopson, fitzgerald, ferlie, gabbay, &  locock, 2002). group dynamics, and  the different ways in which doctors  and nurses may value important  knowledge may affect a  multidisciplinary community of  practice in unpredictable ways  (gabbay et al., 2003).  level of evidence: theoretical opinion,  and mainly qualitative analysis   • legitimate peripheral participation  and power relations are tricky.  experts can also learn from the  novices (fuller, hodkinson,  hodkinson, & unwin, 2005), but  power relations (fox, 2000; li et al.,  2007) may complicate matters, and  marginal membership may  disadvantage some (mallinson, popay,  kowarzik, & mackian, 2006).  level of evidence: systematic review,  plus qualitative analysis   • criteria for success of communities of  practice are elusive (eysenbach et al,  2004; li et al., 2007), perhaps because  they are complex adaptive systems  (greenhalgh et al., 2004; sheffield,  2008).   rigorous experimental evidence is  sparse, although one randomised  controlled trial (barwick, peters, &  boydell, 2009) and a pre‐test, post‐test  design (tolson, booth, & lowndes,  2008) suggest that members of a  community of practice group are more  positive in attitude towards a change  in practice, more knowledgeable, and  willing to take decisions  independently.  level of evidence: systematic reviews,  plus single randomised control trial  web site appraisal    for each of five critical success factors  identified by the nlh team commissioning  the research, appraisal questions were  developed (figure 1, for an outline list). these  questions were based on the evidence from the  original systematic review (urquhart et al.,  2002) and were also relevant to the processes  and enabling technology of interest to the  nlh. questions probing the stage of  development were based on the most rigorous  evaluation of communities of practice  identified (gongla & rizzuto, 2001) as figure 2  illustrates for the usability critical success  factor.  53 evidence based library and information practice 2010, 5.1  54 website appraisals were completed for the 19  specialist libraries in existence in late 2004.  copies of the individual completed appraisal  forms were sent to the relevant information  specialists for verification by their core team,  and amendments made as necessary.    results    stage of development    the stages of development are characterised  as follows:    1. building stage (constructs communal  memory and context: learning about  one another; building a common  vocabulary; creating roles; beginning  repertoires)  2. engaged stage (promotes access and  learning: building trust, loyalty, and  commitment; providing outreach;  telling community stories;  encouraging contribution to the  knowledge base)  3. active stage (supports collaboration:  engaging members in work groups or  collaborative working with others;  using analytical and decision‐making  tools)  4. adaptive stage (creates new  products: founding of more  communities; responding to  environmental changes)    critical success  factor  questions  functionality  1. are the purpose, aim, and identity clear?  2. are there ways of identifying and locating community members?  3. is there a clear knowledge management framework, common repository?  4. what functions support newcomers, or visitors?  5. how are evaluation, audit, and community “sensing” achieved?  6. how are links with other groups, and organisations presented?  7. how might policy making in the department of health be supported?  usability  1. how are individuals brought together?  2. are the roles of participants and the norms of behaviour clear?  3. is the organisation of knowledge appropriate to the community?  4. does the process of feedback work transparently?  content  1. are there directories of members or equivalent?  2. does the range of content include document and library systems, community “stories”,  record of collaborative work efforts?  3. are there decision making and analytical tools to support application of the content?  4. are there links with other systems in the workplace, such as the electronic patient  record?  stakeholder  involvement  1. what types of participation are possible, and can participants (professional societies,  patient groups, research workers, charities, commercial organisations) move to a level of  participation appropriate to their needs?  2. how is personal identity and communal identity supported?  3. are the rhythm of events, news for the workplace reflected?  4. what rewards of membership are apparent?  5. what types of collaborative interaction might be supported?  6. what type of mentoring is available?  7. how is primary care taken into account?  8. are patients stakeholders, and what might be the relationship with nhs direct online or  similar?  project  management  1. how is “senior management” represented?  2. is there a core of community leaders?  3. how is leadership interpreted?  4. what evaluation mechanisms exist, and how are they acted on?  5. how are diverse roles identified and represented?  6. what mechanisms are there for building trust among community members?  fig. 1. website appraisal framework.  evidence based library and information practice 2010, 5.1  rationale  stage characteristics  1. how are individuals brought together?  potential stage. some evaluations stress need for  relationship development, some outreach from existing  teams, and virtual working an important consideration  for professionals used to face‐to‐face communication  2. are the roles of participants and the norms of  behaviour clear?  building stage. need for consideration of potential  computer mediated communication problems in the  health sector, different genres of communication among  different professional groups  3. is the organisation of knowledge appropriate to  the community?  building to engaged stages. some possible uncertainty  about the formats appropriate for different professional  groups  4. does the process of feedback work transparently?  engaged to active stages. helping to support change in  practice, and organisational learning  fig. 2. mapping of web site usability criteria to stages identified in the gongla & rizzuto review.  these stages of development are the same as  used in the previous evaluation (urquhart et  al., 2002). some of the libraries were found to  be mid‐way between stages, making it difficult  to indicate how many were at each stage. for  example, some of the older established  libraries were, unsurprisingly, more  advanced than those newly established in 2004  (e.g., cancer, oral health, later life). and as  might be expected, there were signs of change  with different parts of the same library  showing signs of operating at different stages.  in one specialist library the user group was at  the building stage, but their uk‐wide steering  group was well into development of the active  and adaptive stages of community of practice  development.     all the specialist libraries fulfilled the criteria  for the building stage and most were at the     engaged stage, but were developing in  different ways. the way feedback was  handled differed across the libraries, although  the value of contributions from users was  acknowledged as a virtual circle:     we send items out to our stakeholders who  are the experts in the field and say ‐  should this still be in the library? the  more that we can get the stakeholders and  the user involved... you want people to be  looking at the site, and they will say ‐ look,  this is now out of date…and that’s  starting to happen.    most of the specialist libraries were stuck at  the engaged stage, as the resource  management system (rms) introduced by the  nlh for use by all the specialist libraries for  the architecture of the virtual community of  practice did not provide the required  functionality to support some types of  collaborative working and discussion as easily  as it might. unlike many of the communities  of practice in the private sector, the specialist  libraries did not “tell community stories” (as  might be expected of the engaged stage) but  they did provide access to clinical guidelines  and other procedural knowledge – perhaps a  very formalized way of documenting “how  things are done around here and why.”    building trust and loyalty (in the engaged  stage) was characterised by concern about  providing a quality product valued by the  community, and one that the community  would contribute to, in some way. as libraries  expand the content it is likely that the number  of clinical “leads” or at least the need for  varied clinical input increases, particularly if  there is a great emphasis on quality assurance  processes.    i think we’ve invested quite a lot of time in  the quality assurance of things…of  guidelines that we’re going to put on and  that’s why it’s taken us.... 18 months  developing the clinical library…recently  i’ve had more clinical input, we’ve divided  the library up so we’ve got other clinicians  55 evidence based library and information practice 2010, 5.1  involved more. so i’m doing far more of  the sort of overview.    one of the more established specialist  libraries viewed their role in the active stage  as the integration of the smaller “communities  of practice” in their area. this included  specialist interest groups and professional  groups, represented through their advisory  group structure. they stress collaboration but  “we try hard not to be seen as competition with the  specialist societies.”     some specialist libraries viewed their role as  helping their community make decisions  quickly, which included convenient provision  of digests or summaries of the content that  might already be on the nlh site (such as  nice guidelines or cochrane reviews) but is  dispersed and therefore harder to locate.     if a nice guideline is difficult for  somebody to use because it’s an 80 page  word document then there’s not much we  can do about that…i suppose what  specialist libraries are doing, is that we  are trying to present critically appraised  abstracted versions of content to give  people a way into some of those more  difficult to use resources. that’s  something we’re working on as well.    benefits and drawbacks of a community of  practice “brand”    several of the interviews revealed the tension  between providing a recognisable nlh‐ branded service, with a standardised resource  management system, and a community of  practice that was relevant to the needs of that  community. for some specialist libraries, this  spelt problems for the trust they had built up  with their users: “you’re faced with a community  of users who have seen the functionality go with no  extra, no real extra content.” experimentation  with rss (really simply syndication) feeds  was acknowledged as useful, if not vital, in  supporting current awareness services, but  there were some misgivings at the time that  some user‐led needs analysis had not been  fully integrated in the timescale expected.   opinions varied about the desirability of  standardisation. some want complete  standardisation, others want control of their  front page, or some degree of flexibility. the  resource management system (rms)  provides some advantages for the information  specialists in helping to crosslink to other  libraries’ information or other nlh evidence  resources.    there was a perception that the rms and the  rss feeds were very good at helping with the  information management, but that the support  of the community of practice would need  more emphasis on learning. some  interviewees noted that the current  configuration of the rms prevented them  from putting on material that would suit the  newly qualified or visiting staff – those who  might be “legitimate peripheral participants.”    sub‐groups of information specialists, under  the aegis of the nlh, were tasked to formulate  guidance on standards and presentation,  training and development, evaluation, and  publication types on behalf of all the libraries.  the interviews confirmed that there had been  problems with technical guidance and  support, but that an appointment of a co‐ ordinator at the nlh end, and easier  reporting (and better response rate) for  technical problems had helped improve the  situation.     the information specialists were helped  through the problems of implementing the  rms by their own network. ”we can actually  have an electronic community of all the specialist  libraries’ staff so we can collaborate and have  online discussions, file sharing, post events, have  some statistics on the electronic community.”  those based near each other geographically  have close working relationships: “and that’s a  kind of group therapy session as much as anything  else. we get together and talk about some of the  difficulties involved in producing these libraries.”  this community of practice of information  specialists, working to support the specialist  libraries, was operating very successfully.      56 evidence based library and information practice 2010, 5.1  support of social learning    at the time of evaluation, several information   specialists and clinical leads discussed how to  get their communities involved in activities  that would be viewed as social learning,  although only a minority offered access to  training materials. the organisational  framework for the social learning is the  reference group or advisory group of  professional organisations, relevant charities,  and patient support groups. within that large  group there may be a core group of people  who contribute regularly to the work of the  specialist library. activities such as sharing in  appraisal of material for the web site would  build the community, but would also require  some intrinsic rewards: “my personal view is  that we’re unlikely to get anyone with the right  expertise to quality appraise anything for us for no  academic credit or reward whatsoever.” on the  other hand involving clinical students who are  trained in critical appraisal in this type of  content selection as “something that is done”  may help as “they are the people we can influence  most and that could be done cheaply and  easily…they will be our main users in the future.”     routes to capitalise on what people already  do, to use that as a scaffold to extend the work  of the community of practice in the specialist  library, include working with existing journal  clubs: “at the moment we’re trying to get links  into things like journal clubs in teaching  hospitals.” some specialist libraries see the job  of supporting the specialist libraries as  something professionals should accept as part  of their role, and would become ingrained  during professional education and training.     what we want them to do is to let us  know they read something interesting and  to provide us with a quality assessment of  that. they’re doing it all the time, they  should be doing it to the people they’re  training, their junior staff, they’re doing it  for journals, what we do is provide them  with materials so they can provide us with  their opinion online.    discussion    most evaluations of communities of practice  have been conducted, as this one was, using  qualitative research methods, to understand  how the communities were operating. the  framework devised was effective in  highlighting some of the development  problems and successes, and was objective in  identifying the main differences between some  specialist libraries, although as indicated  earlier the level of evidence on which the  framework was based was not high. the first  evaluation of the libraries highlighted the  problems in moving from the engaged stage to  the active stage (yeoman, urquhart, & sharp,  2003). this second evaluation also found this  to be a problem, suggesting that moving  between some stages may be more difficult for  some types of communities than others.  unfortunately, there are few independent  longitudinal evaluations of genuine  communities of practice to assess whether the  stage framework needs expansion, or whether  the problem is really one of describing  appropriate phases for system states in  complex systems, and identifying triggers for  change.     the problems surrounding the rms  implementation indicated some of the  problems of becoming an it‐led rather than an  it supported venture, and that it‐led  communities can fail to thrive (kling &  courtright, 2003). it should support existing  social networks, but there may be problems  ahead for the specialist libraries in fostering  multidisciplinary working (dopson et al.,  2002; ferlie et al., 2005; greenhalgh et al.,  2004). perhaps multidisciplinary, virtual  health communities of practice will flourish  better when there is a real identified clinical  need for sharing and creation of new  knowledge, such as in pain management for  children (curran‐smith, abidi, & forgeron,  2005). evidence from the services’  collaboratives set up by the nhs to support  knowledge management showed a failure to  foster the knowledge sharing and creation that  might be typical of a community of practice  (bate & robert, 2002). such sharing is hard to  57 evidence based library and information practice 2010, 5.1  achieve unless participants trust each other  enough to ask questions and share knowledge  (falkman et al., 2008; giordano, 2007). the  community of practice ideas stress the  importance of “legitimate peripheral  participation”, and the importance of  welcoming newcomers was acknowledged by  the specialist libraries. an informal email  network on evidence based healthcare  (chain) provided filtered access to people  with know‐how and was very popular, but  those who were acting at the periphery also  appreciated what they could learn from  reading the postings (russell et al., 2004).  there sometimes seems to be an assumption  among online community developers that  “lurking” is a problem and indicative of lack  of interest. the advantage of using a  community of practice framework is that such  lurking is legitimate, not a problem, but  participants need to be able to move to more  active roles if they wish (mallinson et al., 2007)  and more social network analysis might  illuminate the patterns of social interactions to  be expected (curran‐smith & abidi, 2007).    more importantly, the ideas behind social  learning and communities of practice may  need to be reviewed before devising it‐ supported frameworks to support such  learning (li et al., 2007; cox, 2005). eraut  (2000) distinguishes three types of learning  within social learning and three types of tacit  knowledge: tacit understanding of people and  situations, routine actions, and the rules for  intuitive decision‐making. one framework  that has been tried in the health sector is the  healthcare scenario composer (cheah &  abidi, 2001). this would provide an electronic  forum to allow healthcare experts to respond  to a given scenario (which may be “already  solved”), a challenge scenario (atypical, which  elicits tacit knowledge), or solved‐challenge  scenario (to assess degree of consensus or  disagreement) with a proposed solution. this  approach seems to have some similarities with  some of the principles of case‐based reasoning  (becerra‐fernandez, gonzalez, & sabherwal,  2004, p. 160). falkman et al. (2008) have also  designed a community for clinical case  discussions. this may be successful in  secondary care, but ethnographic research on  knowledge management in primary care  points to the very fluid nature of tacit  knowledge in primary care, the collectively  constructed “mindlines” (the internalised tacit  guidelines which were negotiated with a  variety of people) according to organisational  and time constraints (gabbay & le may, 2004).  to complicate matters further, several groups  in the health sector have used the term  “communities of practice” as an aspirational  term rather than descriptive of actual practice  (as noted by gabbay et al., 2003).     the community of information specialists  appeared to working effectively as a  community of practice within the general  specialist library service structure. within  each specialist library the management  structure generally worked with the reference  advisory group contributing advice and  guidance that should help satisfy the needs of  a professional and lay audience.     the evaluation is limited in that only around  half of the specialist libraries provided  interview data. it is possible that some of those  not interviewed had examples of good  practice that were not identified. however, the  community of information specialists was  operating well, and it is likely that news about  developments in other libraries would be well  known and brought to the attention of the  evaluators.     conclusions    the evaluation indicated that the specialist  libraries were operating as communities of  practice although not at a very advanced  stage.    librarians developing virtual communities of  practice should pay close attention to the  design of learning activities that might be  expected of the engaged stage of development  and the type of collaborative work typical of  the active stage of development. librarians  need to do much more than collection  management for communities of practice to  develop successfully – the information  58 evidence based library and information practice 2010, 5.1  professional has to interact with users far  more intensively. the emphasis should be on  encouraging contributions from community  members and engaging them in activities  relevant to their practice.     the nlh acted on many of the  recommendations of the evaluation. for  example, specialist collections now have  access to the content management system for  their own home page (a move supported by  the updated literature review on the  importance of contingent approaches to  design). in future more personalisation might  be expected from members of a community of  practice accustomed to downloading  applications (apps) for their own electronic  devices.    structuring library and information services  around community of practice principles  appears effective, and within the uk, nhs  scotland is following these principles in  design of their knowledge networks  (caldwell, davies, stewart, thain, & wales,  2008). such services build on the social  learning that happens naturally among  professionals (although the updated literature  suggests that power relations can be tricky to  handle, with different professional groups  having different values and approaches to  learning and expectations of their  responsibilities). some of the difficulties of  power relations may be lessened through  design of structures such as the reference  advisory groups for the specialist libraries,  which encouraged input from a wide range of  stakeholders.     a community of practice is a win‐win scenario  for supporting evidence based practice,  provided appropriate and relevant solutions  for the professional community and their  problems are designed. as the updated  literature review suggests, we still need to find  good criteria to judge the effectiveness of  communities of practice, but ongoing  systematic reviews may provide that evidence.      acknowledgements  the authors thank the nlh team and the nhs  information authority for funding this  research. we are very grateful to all those who  consented to be interviewed, or who  contributed to the research in other ways. we  are also very grateful to the reviewers for their  constructive comments which have improved  the paper.     an earlier version of this paper was presented  at the eblip5 conference in stockholm,  sweden, june 29‐july 3, 2009, and was  awarded best oral presentation.    declaration of interests    christine urquhart is part of a team working  under the auspices of the cochrane effective  practice and organisation of care group to  systematically review the evidence for the  effectiveness of communities of practice on  health care delivery and professional  performance.      references    barwick, m. a., peters j., & boydell, k. (2009).  getting to uptake: do communities of  practice support the implementation  of evidence‐based practice? journal of  the canadian academy of child and  adolescent psychiatry, 18(1), 16‐29.    bate, s. p., & robert g. (2002). knowledge  management and communities of  practice in the private sector: lessons  for modernizing the national health  service in england and wales. public  administration, 80(4), 643‐663.    becerra‐fernandez, i., gonzález a., &  sabherwal r. (2004). knowledge  management: challenges, solutions, and  technologies. upper saddle river, nj:  pearson education.     braithwaite, j., westbrook, j. i.,  ranmuthugala, g., cunningham, f.,  plumb, j., wiley, j., . . . debono d.  59 evidence based library and information practice 2010, 5.1  (2009). the development, design,  testing, refinement, simulation and  application of an evaluation  framework for communities of  practice and social‐professional  networks. bmc health services  research, 9, 162. doi:10.1186/1472‐6963‐ 9‐162    caldwell, l., davies, s., stewart, f., thain, a.,  & wales, a. (2008). scottish toolkit for  knowledge management. health  information & libraries journal 25(2),  125‐134.    catino, m. (1999). learning and knowledge in  communities of work practice: the  case of the pilot plants at cer‐montell.  in m. easterby‐smith m., l. araujo, &  j. burgoyne (eds.), organizational  learning. 3rd international conference,  lancaster. lancaster: university of  lancaster.    cheah y.‐n., & abidi, s. s. r. (2001). the role  of information technology in the  explication and crystallization of tacit  healthcare knowledge. health  informatics journal, 7(3), 158‐167.    cox, a. (2005). what are communities of  practice? a comparison of four  seminal works. journal of information  science, 31(6), 527‐540.    curran‐smith, j., & abidi, s. s. r. (2007).  evaluation of an online discussion  forum for emergency practitioners.  health informatics journal 13(4), 255‐ 266.    curran‐smith, j., & abidi, s. s. r., & forgeron,  p. (2005). towards a collaborative  learning environment for children’s  pain management: leveraging an  online discussion forum. health  informatics journal, 11(1), 19‐31.    dopson, s., fitzgerald, l., ferlie, e., gabbay, j.,  & locock, l. (2002). no magic targets!  changing clinical practice to become  more evidence‐based. health care  management review, 27(3), 35‐47.    dubé, l., bourhis, a., & jacob, r. (2005). the  impact of structuring characteristics  on the launching of virtual  communities of practice. journal of  organizational change management,  18(2), 145‐166.    eraut, m. (2000). non‐formal learning and  tacit knowledge in professional work.  british journal of educational  psychology, 70(1), 113‐116.    eysenbach, g., powell, j., englesakis, m., rizo,  c., & stern, a. (2004). health related  virtual communities and electronic  support groups: systematic review of  the effects of online peer to peer  interactions. british medical journal  (bmj), 328(7449), 1166‐1171.    falkman, g., gustafsson, m., jontell, m., &  torgersson, o. (2008). somweb: a  semantic web‐based system for  supporting collaboration of  distributed medical communities of  practice. journal of medical internet  research, 10(3), e25. doi:  10.2196/jmir.1059    ferlie, e., fitzgerald, l., wood, m., &  hawkins, c. (2005). the nonspread of  innovations: the mediating role of  professionals. academy of management  journal, 48(1), 117‐134. retrieved 26  feb. 2010 from  http://www.mangematin.org/mcoi/t extes%202006‐7/amj2005%20ferlie.pdf    fox, s. (2000). communities of practice,  foucault and actor‐network theory.  journal of management studies, 37(6),  853‐867.    fuller, a., hodkinson, h., hodkinson, p., &  unwin, l. (2005). learning as  peripheral participation in  communities of practice: a  reassessment of key concepts in  60 evidence based library and information practice 2010, 5.1  workplace learning. british educational  research journal, 31(1), 49‐68.    fung‐kee‐fung, m., goubanova, e., sequira,  k., abdulla, a., cook, r., crossley, c.,  stern, h. (2008). development of  communities of practice to facilitate  quality improvement initiatives in  surgical oncology. quality management  in health care, 17(2), 174‐185.    gabbay, j., le may, a., jefferson, h., webb, d.,  lovelock, r., powell, j. & lathlean, j.  (2003). a case study of knowledge  management in multi‐agency  consumer‐informed “communities of  practice”: implications for evidence‐ based policy development in health  and social services. health, 7(3), 283‐ 310. doi:10.1177/1363459303007003003    gabbay, j., & le may, a. (2004). evidence  based guidelines or collectively  constructed “mindlines?”  ethnographic study of knowledge  management in primary care. british  medical journal (bmj), 329(7473), 1013‐ 1017.    giordano, r. (2007). an investigation of the  use of a wiki to support knowledge  exchange in public health. in  conference on supporting group work:  proceedings of the 2007 international  acm conference on supporting group  work, november 04‐07, sanibel island,  florida. retrieved 26 feb. 2010 from  http://delivery.acm.org/10.1145/13200 00/1316664/p269‐ giordano.pdf?key1=1316664&key2=19 95027621&coll=guide&dl=guide&c fid=77838471&cftoken=88621490    gongla, p. & rizzuto, c. r. (2001). evolving  communities of practice: ibm global  services experience. ibm systems  journal, 40(4), 842‐862.    greenfield, d., travaglia, j., nugus, p., &  braithwaite, j. (2007). health sector  community of practice: final report. a  content analysis and systematic  review of the health community of  practice research literature. in  university of new south wales, faculty  of medicine. retrieved 26 feb. 2010  from  http://www.med.unsw.edu.au/medwe b.nsf/resources/2007articles1/$file/he alth+sector+cop+literature+review.p df    greenhalgh, t., robert, g., macfarlane, f.,  bate, p., & kyriakidou, o. (2004).  diffusion of innovations in service  organizations: systematic review and  recommendations. the milbank  quarterly, 82(4), 581‐629.    hildreth, p. m., kimble, c., & wright, p.  (1998). computer mediated  communication and international  communities of practice. in computer  mediated communications and  communities of practice. proceedings of  ethicomp’98, march 1998, erasmus  university, the netherlands. retrieved  26 feb. from  http://papers.ssrn.com/sol3/papers.cf m?abstract_id=681184    kling, r. & courtright, c. (2003).  characterizing collective behavior  online: the social organization of  hangouts, clubs, associations, teams  and communities. the information  society, 19(3), 221‐235.    lau, f., & hayward, r. (2000). building a  virtual network in a community  health research training program.  journal of the american medical  informatics association, 7(4), 361‐377.    lave, j., & wenger, e. (1991). situated learning:  legitimate peripheral participation.  cambridge: cup.    lesser, e. l., & storck, j. (2001). communities  of practice and organizational  performance. ibm systems journal,  40(4), 831‐841.  61 evidence based library and information practice 2010, 5.1  62 li, l. c., grimshaw, j. m., nielsen, c., judd,   m., coyte, p. c., & graham, i. d.  (2007). use of communities of practice  in business and health care sectors.  implementation science, 4, 27.  doi:10.1186/1748‐5908‐4‐2    mallinson, s., popay, j., kowarzik, u., &  mackian, s. (2006). developing the  public health workforce: a  “communities of practice”  perspective. policy & politics,, 34(2),  265‐285.  markus, l. m. (2001). toward a theory of  knowledge reuse: types of knowledge  reuse situations and factors in reuse  success. journal of management  information systems, 18(1), 57‐93.    moreno, a. (2001). enhancing knowledge  exchange through communities of  practice at the inter‐american  development bank. aslib proceedings,  53(8), 296‐308.    nutley, s. m., & davies, h. t. o. (2001).  developing organizational learning in  the nhs. medical education, 35(1), 35‐ 42.    o’dell, c., hubert, c. & mcdermott, r. (2001).  building and sustaining communities of  practice: continuing success in knowledge  management. houston, tx: american  productivity & quality center.    russell, j., greenhalgh, t., boynton, p. &  rigby, m. (2004). soft networks for  bridging the gap between research  and practice: illuminative evaluation  of chain. british medical journal  (bmj), 328(7449), 1174‐1179.    sheffield, j. (2008). does health care for  systemic development? systems  research and behavioral science, 25(2),  283‐290.  tolson, d., booth, j. & lowndes, a. (2008).  achieving evidence‐based nursing  practice: impact of the caledonian  development model.” journal of  nursing management, 16(6), 682‐691.    urquhart, c., yeoman, a., & sharp, s. (2002).  nelh communities of practice evaluation  report. aberystwyth: university of  wales aberystwyth. retrieved 26 feb.  2010 from  http://users.aber.ac.uk/cju/nelhcop1.6‐ revisedfinal.doc    wenger, e. (1998). communities of practice:  learning, meaning and identity.  cambridge: cup.    white, m. (2001). communities of practice.  freepint, 101. retrieved 26 feb. 2010  from  http://www.freepint.com/issues/29110 1.htm#feature    yeoman, a., urquhart, c., & sharp, s. (2003).  moving communities of practice  forward: the challenge for the  national electronic library for health  and its virtual branch libraries.  health informatics journal, 9(4), 241‐ 252.                                                evidence based library and information practice 2010, 5.1  appendix 1     interview schedule     participants were sent the questions for the interview in advance. the order of questions varied  according to the focus of the interview and whether the interviews were being conducted with the  clinical lead or the information specialist.     background and organisation    1. please tell me a bit about the background to your specialist library.  2. how does the specialist library team operate and what contact is there with developers of other  specialist libraries?  3. are there any particular problems with the development of the specialist library (e.g., innovation  versus conformity with nelh norms)?   4. how do you see the link between your specialist library and the nelh?   5. what links do you have to the nks and/or the map of medicine?     supporting the community of practice/stakeholders    7. do you have a perception of the specialist library as serving an identifiable community of  practice?  8. do you see a role for the specialist library in encouraging collaborative working within the  community of practice?  9. are there areas of unmet information need in your specialism, and will the specialist library help  to address them?   10. do you see a training or mentoring role for the specialist library?  11. how do you ensure that the information available via the specialist library is current and  appropriate to the members of the community?  12. are there other organisations within the specialism that provide a similar service to the specialist  library?     audit and future development    13. do you collect usage statistics and, if so, what do you do with them?  14. what evaluation mechanisms are in place?  15. do you know what will happen when your grant ends – have you made any contingency/future  plans?    nb: at the time of the interviews, the specialist libraries were still part of the national electronic  library for health (nelh). migration to the national library of health occurred in 2006.      63 christine urquhart methods discussion conclusions evidence summary   varying student behaviours observed in the library prompt the need for further research   a review of: paretta, l. t., & catalano, a. (2013). what students really do in the library: an observational study. the reference librarian, 54(2), 157-167. doi:10.1080/02763877.2013.755033   reviewed by: maria melssen medical librarian, independent contractor port clinton, ohio, united states of america email: mariamelssen@gmail.com   received: 25 oct. 2013   accepted: 10 jan. 2014      2014 melssen. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine if the behaviours of students studying in the library are primarily study or non-study related, the extent to which these behaviours occur simultaneously, what types of study and non-study behaviours are most common, and if the time of day or use of social media have an effect on those behaviours.   design – observational study.   setting – two university libraries in new york.   subjects – a total of 730 university students.   methods – two librarians at 2 separate university libraries observed and recorded the behaviours of 730 students. observations were conducted over the course of several weeks during the fall of 2011 in the designated study or quiet areas, reference room, and at computer terminals of the libraries. observations were made by walking past the students or by observing them from a corner of the room for between 3 to 10 seconds per student. student activities were recorded using a coding chart. the librarians also collected data on the perceived age, gender, and ethnicity of the students and whether the students were using a computer at the time of observation. if students displayed more than one behaviour during a single observation, such as talking on the phone while searching the library’s online catalogue, the first behaviour observed or the behaviour that was perceived by the observer to be the dominant behaviour was coded behaviour 1.the second behaviour was coded behaviour 2.   main results – the behaviours of 730 students were observed and recorded. two librarians at separate universities were responsible for data collection. kappa statistical analysis was performed and inter-rater reliability was determined to be in agreement. data was analyzed quantitatively using spss software.   over 90% of students observed were perceived to be under 25 years of age and 56% were women. the majority were perceived to be white (62%).   of the 730 observations, 59% (430) were study related and 37% (300) were non-study related. the most common study related behaviours included reading school-related print materials (18.8%) and typing/working on a document (12.3%). the most common non-study related behaviours included facebook/social media (11.4%) and website/games (9.3%). the least common study related behaviour was using the school website (1.2%) and the least common non-study related behaviour was “other on the phone” (0.1%).   second behaviours were observed in 95 of the 730 students observed. listening to music was the most common second behaviour (35.8%) and educational website was the least common (1.1%).   most study observations were made on mondays and most non-study observations were made on thursdays and fridays. throughout the entire day, study related behaviours were observed between 62-67% of the time regardless of the time of day. students working on computers were more likely to be observed in engaging in non-study related behaviour (73%) than those not working on a computer (44%).   conclusion – students display a variety of study and non-study behaviours throughout the day with the majority of the behaviours being study related. students also blend study and non-study activities together, as evident in their switching between study and non-study related behaviours in a single observation and their ability to multitask. data gathered from this study provides evidence that students view the library as not only a place for study but also a place for socialization.   several limitations of this study are acknowledged by the authors. first, behaviours that appear to be non-study related, such as watching videos on youtube, could be study related. many faculty members utilize social media tools such as facebook, twitter, and youtube to support their course content. a student observed watching youtube videos could be watching a professor’s lecture, not a video for entertainment purposes only. this lack of knowing definitively why students are utilizing social media while in the library may have led the authors to mistake non-study behaviour for study behaviour.   an additional limitation is the short duration of time spent observing the students as well as the proximity of the observer to the student. observations lasting longer than 3 to 10 seconds and made at a closer range to the students could provide more accurate data regarding what type of behaviours students engage in and for how much time. in addition to the before mentioned limitations, the authors acknowledge that they had no way of knowing if the individuals being observed were actual students: the assumed students could have been faculty, staff, or visitors to the university.   due to the study’s limitations, further research is needed to determine in greater detail what students are doing while they are studying in the library. this data would allow librarians to justify the need to provide both study and non-study space to meet the diverse needs of students. conducting a cohort study would allow researchers to observe student behaviour longitudinally. it would minimize the limitations of short-term student observation as well as the proximity of the observer to the student. research on the use of mobile technologies by students, such as smart phones, to access study related material while they are in the library would also yield valuable data regarding student study behaviours.     commentary   research indicates that students do not accurately report their study behaviours. a student may claim to study for three consecutive hours in the library; however, two of the three hours could be spent texting friends or checking twitter. the authors sought to determine if their own students spend more of their time studying or engaging in non-study related behaviours.   critical appraisal of this study was completed using the evidence based library and information practice critical appraisal checklist (glynn, 2006). the study’s validity was analyzed in four content areas: population, data collection, study design, and results. while the results are valid, the validity of the population selection, study design, and data collection methods are questionable.   the selection of study participants is problematic due to lack of information on the student population size. the authors are also not clear if an equal number of observations were made at both libraries. knowing the student population size and if a comparable number of observations were made at both institutions would help determine if 730 participants is a large enough sample size and if all possible study participants are represented.   an additional concern is the study design. the authors acknowledge that it is possible some of the students observed could have been faculty, staff, or visitors. this confounding variable could have been addressed if inclusion and exclusion criteria of study participants were clearer or if a different study design that allowed for identifying students from non-student library users was chosen. also of concern is the author’s lack of explanation as to why data on ethnicity, age, or gender was significant to the study.   one of the main objectives of the study was to determine if such factors as time of day affect study behaviours. detail regarding the types of behaviours is provided, however, minimal information on time of day is given. having access to a copy of the coding instrument used in addition to a clearer description of the data collection methods would strengthen the face validity of the study and allow other researchers to replicate the study.   despite these issues, results were clearly explained. ethics approval was obtained and informed consent was not necessary for this study. kappa analysis minimized inter-rater bias and the study was validated through pilot testing. opportunities for further research were identified by the authors.   accurate data regarding student behaviours in the library allows librarians to better understand the diverse needs of students and provide library resources as well as services to meet those needs. the challenge is how librarians can determine what exactly students are doing in the library. this study inspires further research on the use of observational studies.   references   glynn, l. (2006) a critical appraisal tool for library and information research. library hi tech, 24 (3), 387-399. doi:10.1108/07378830610692154   evidence summary   children display seven distinct roles when searching online at home   a review of: foss, e., druin, a., brewer, r., lo, p., sanchez, l., golub, e., & hutchinson, h. (2012). children’s search roles at home: implications for designers, researchers, educators, and parents. journal of the american society for information science and technology, 63(3), 558-573. doi:10.1002/asi.21700   reviewed by: diana k. wakimoto online literacy librarian california state university, east bay hayward, california, united states of america email: diana.wakimoto@csueastbay.edu     received: 5 apr. 2013    accepted: 4 jul. 2013      2013 wakimoto. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to explore children’s internet searching at home in order to make recommendations to designers, researchers, educators, and parents on how to assist children in becoming search literate through understanding children’s search roles.   design – qualitative, exploratory study.   setting – children’s homes in the urban areas of maryland, virginia, delaware, and pennsylvania.   subjects – 83 children (28 children were age 7, 29 were age 9, and 26 were age 11). 41 of the children were female and 42 were male. parents of the children were also included in the study. 77% of the parent interviews were carried out with mothers, 15% were with fathers, and 8% were with both parents together.    methods – the authors conducted qualitative interviews both with the parents and the children. parents were interviewed first and the interviews were audiotaped and transcribed. the interviews covered computer rules, children’s experience in searching, searching habits, and areas of frustration. interviews with the children covered questions about frequency of computer use and reasons for searching. these interviews were video recorded and transcribed. after the interview, the children were asked to complete five search tasks, which were video recorded, and were asked if they had successfully completed the task and why they clicked the link results. the researchers also took notes throughout the interviews and search tasks.   the researchers were able to analyze 80 transcripts from the children and 75 transcripts from the parents. the interview transcripts were coded using inductive, qualitative coding starting with open coding to identify categories of children’s search roles. the transcripts from the children interviews were coded three times by one researcher and the coding was verified by another researcher. the transcripts then were coded again using the code book developed by the first researcher. the researchers completed axial and selective coding to refine their search role categories. the researchers also analyzed the data in order to identify behaviours that distinguished the categories from each other. the same coding process was used for the parent interview transcripts. the results from the analysis of the parent interviews were used to verify findings from the children transcripts.   main results – children searching at home show seven different searching roles: developing, domain-specific, power, nonmotivated, distracted, rule-bound, and visual, with each search role being delineated by specific behaviors and/or abilities. triggers for searching change as children age, with younger children searching based on personal interests while older children search for school-related information. children rely on summaries shown on the results page, as well as familiarity with known websites, in deciding which links to click. children are interested in both moving and still image results, with visual searchers, power, and distracted searchers frequently mentioning images in their interviews. power searchers, those with the ability to use keywords and with an understanding of search engines, discussed less influence on their searches than others. parents have more influence over younger children while school has more influence over older children. parents helped and influenced their children’s searching in varied ways including demonstrating and offering advice for searching and setting rules for searching. children often reported frustration with their searches, which was also reported by parents. most of the children were unable to complete the complex search task as they were unable to separate the query into multiple parts. few gender differences in searching were found, although researchers reported that games were a trigger for boys more often than girls, and boys declined to search more than girls. girls were more influenced in their searching than boys and stopped searching due to boredom more often than boys.   conclusion – the authors suggest that the findings can help search engine designers, researchers, educators, and parents to assist children in becoming search literate. designers should enable scaffolded, assisted searching in order to help searchers, especially with separating out multiple parts of a complex question and with encouraging fact-checking. educators and parents can coordinate their efforts to more effectively help children overcome searching frustrations and challenges. researchers could replicate the study to validate the search roles discovered by the authors and also extend the study to focus on searching in regards to gender and use of other devices, such as smartphones and tablets.     commentary   this study adds to the large body of literature on online searching and contributes to a greater understanding of children’s searching in the home. as the ability to search effectively for information online becomes more necessary at a younger age due to the increased use of computers and mobile devices in grade school and the need to search in order to complete homework assignments, understanding children’s searching behaviours and roles becomes crucial in order to assist them in becoming search literate. this study should be of great interest for educators and parents who work with the children, as well as designers who are charged with improving search engines for use by children.   this study is well-written and overall well-designed, providing valuable information on the various roles children conform to while searching at home. this study’s design and findings are valid according to an evaluation using the critical appraisal checklist by glynn (2006). having multiple researchers code the transcripts increased the reliability of the results. the search roles found by the authors do appear to be supported by the data and the authors explicitly stated research limitations and data loss issues that allowed for the transcription and use of approximately 90% of the transcripts. the inclusion of the interview questions/guide would have strengthened the study; however, the inclusion of the specific search task questions should allow others to replicate the searching portion of the study if not to replicate exactly the background interview questions. also, as the interview questions and search tasks appear to have not been previously used or validated, the use of a pilot study would have strengthened the study.   the only puzzling part of the study is that, although it obviously followed a grounded theory approach, the authors never explicitly state the use of grounded theory. however, from the references to the work of corbin and strauss (2008), the construction of the literature review in relation to the results of data collection and analysis, and the procedures of data analysis, the influence of grounded theory can be seen clearly. grounded theory is a reasonable approach given the exploratory nature of the study and its use in other studies of search behaviour and information-seeking (ostrander, 2008).   understanding the various search roles of children is an important contribution to the profession’s knowledge of children’s searching behaviours and will assist librarians, teachers, and parents in helping children learn to search more productively. as the researchers note, the findings may also help designers improve search interfaces. continued research in this area, including the authors’ follow-up study on adolescent searchers, will strengthen the research base, which practitioners can draw on as they create lessons, activities, and guidelines for helping children become search literate.     references   corbin, j., & strauss, a.l. (2008). basics of qualitative research: techniques and procedures for developing grounded theory (3rd ed.). thousand oaks, ca: sage publications.    glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi: 10.1108/07378830610692154   ostrander, m. (2008). talking, looking, flying, searching: information seeking behaviour in second life. library hi tech, 26(4), 512-524. doi: 10.1108/07378830810920860 rubrics may be a useful tool for assessing mlis student learning experiences evidence summary   rubrics may be a useful tool for assessing mlis student learning experiences   a review of: adkins, d., buchanan, s. a., bossaller, j. s., brendler, b. m., alston, j. k., & moulaison sandy, h. (2021). assessing experiential learning to promote students’ diversity engagement. journal of education for library and information science, 62(2), 201–219. https://doi.org/10.3138/jelis.2019-0061 reviewed by: jessica koos senior assistant librarian/health sciences librarian stony brook university stony brook, new york, united states of america email: jessica.koos@stonybrook.edu   received: 22 feb. 2022                                                                  accepted:  4 apr. 2022      2022 koos. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30109     abstract   objective – to develop a rubric to assess diversity awareness and professional socialization through in-person or online experiential learning for online mlis students.   design – exploratory case study.   setting – school of information science & learning technologies, university of missouri.   subjects – six experiential learning projects designed to promote diversity and professional socialization for online mlis students.   methods – the authors developed a rubric in order to evaluate the characteristics of several experiential learning projects. the major themes that were measured in the rubric were identified through a comprehensive literature search, and these included professional socialization, service orientation, values orientation, and diversity & inclusion. the authors also added three original accessibility factors that they considered relevant from a practical approach: time, money, and geographic mobility.   main results – the rubric was successfully applied to several ongoing experiential learning projects, as well as to a new project. the authors concluded that it provided a useful framework for assessing the accessibility and estimated value of these experiences.   conclusion – the rubric seems to be a useful start to assessing experiential learning. however, more research is needed to ensure that it is actually measuring the domains that it is intended to measure. this study only focused on whether the rubric could be applied, whereas future studies should assess its accuracy. the rubric may be useful for curriculum evaluation and planning, accreditation, tenure/promotion, and instructor self-assessment.   commentary   the quality of this study was appraised using “the cat: a generic critical appraisal tool” created by perryman & rathbun-grubb (2014). overall, the article was found to be of high quality based on this assessment. the authors are faculty members from the university of missouri school of information science & learning technologies. the research questions are clearly defined and match the method used.   the authors carefully explain the importance of experiential learning in mlis programs and define the three diversity theories they utilized when designing experiential learning projects at the university of missouri: contact theory, diversity levers, and the inclusive excellence framework. they also discuss literature that demonstrates an increase in student learning resulting from similar experiences. finally, they draw upon the literature to include in their rubric several domains of importance in lis education: professional socialization, service orientation, values orientation, and diversity & inclusion. they also add accessibility factors to the rubric, such as time, cost, and ability to travel, all of which may significantly impact a student’s ability to participate in these activities.   the rubric clearly outlines all of these factors. however, the category for “values orientation” appears to be somewhat difficult to measure. listing the specific values within the rubric itself, or including an appendix containing a list, would potentially increase the accuracy of the rubric.   one point of confusion in this article is that the authors state that the current projects being examined using the rubric must already contain specific components. if the general components are already prescribed, this seems to greatly diminish the utility of the rubric. not surprisingly, the rubric scores for all projects only ranged from 16 to 18 out of 24. it may have been more beneficial if the authors examined experiential learning projects from other mlis programs in order to more rigorously test its efficacy.   the authors state that service learning projects can be beneficial by “allowing instructors to add reasonable and short-term opportunities for interaction within the structure of a class, rather than depending on students gaining this instruction through practicum, internship, assistantship, or work experience” (p. 216). while it is helpful to increase these types of learning experiences in virtual programs, it should not diminish the need for other types of experiences, such as internships and volunteer positions. internships and volunteer roles are mutually beneficial to both students and libraries alike and have the capacity to provide a deeper dive into professional socialization, particularly in the area of diversity. with the recent increase in virtual work due to the covid-19 pandemic, there is likely an increase in the number of such positions that are available virtually. also, these types of positions may lead to full time employment after graduation.    lastly, the authors point out the limitation that this rubric does not assess student learning, only the experience itself. this seems to be a crucial component that should be addressed in future research. overall, the existing rubric may be of great benefit to educators who are initially designing learning experiences for their students, assessing existing experiences, and facilitating accreditation, self-evaluation, and promotion/tenure assessment.   references   perryman, c., & rathbun-grubb, s. (2014). the cat: a generic critical appraisal tool. http://www.jotform.us/cp1757/thecat   evidence based library and information practice evidence based library and information practice 2010, 5.1  evidence based library and information practice       commentary    an evidence based checklist for the peer review of electronic search strategies (press  ebc)     jessie mcgowan  senior information scientist, adjunct professor  faculty of medicine, university of ottawa  institute of population health, university of ottawa  ottawa hospital research institute  ottawa, ontario, canada  email: jmcgowan@uottawa.ca    margaret sampson  manager of library services   children’s hospital of eastern ontario research institute  ottawa, ontario, canada  email: msampson@cheo.on.ca    carol lefebvre  senior information specialist   uk cochrane centre, national institute for health research  oxford, united kingdom  email: clefebvre@cochrane.ac.uk    received: 18 jan. 2010          accepted: 18 jan. 2010       2010 mcgowan, sampson and lefebvre. this is an open access article distributed under the terms of the  creative commons‐attribution‐noncommercial‐no derivative works license 2.5 canada  (http://creativecommons.org/licenses/by‐nc‐nd/2.5/), which permits distribution provided the original work is  properly attributed, not used for commercial purposes, and not altered or transformed.    the quality of literature searches is extremely  important, especially for health technology  assessments (htas) and systematic reviews.  high quality searches of information resources  are essential components in the efforts  towards accuracy and completeness of the  evidence base. ascertaining whether a high  quality search has been conducted, however,  can be difficult. factors contributing to search  quality include the skills set of the person (or  persons) who  develop/s the search strategies  and conduct/s the searches, and the use of a  systematic approach (mcgowan & sampson,  2005). in the past, however, there has been  little consistency in and poor quality of  reporting of searches, so it is often difficult to  evaluate the actual searches that were  conducted (sampson, mcgowan, tetzlaff,  cogo, & moher, 2008; sampson & mcgowan,  2006).   149 mailto:jmcgowan@uottawa.ca mailto:msampson@cheo.on.ca mailto:clefebvre@cochrane.ac.uk evidence based library and information practice 2010, 5.1  we received funding from the canadian  agency for drugs and technologies in health  (cadth) to identify and assess the impact of  errors in the elements of electronic search  strategies associated with accuracy and  completeness of the evidence base and to  propose enhancements in the methods used  for creating and evaluating search strategies.  the project was called peer review of  electronic search strategies (press).  the  research consisted of three main components:  a systematic review of the evidence relating to  quality issues and errors in electronic search  strategies; a web‐based survey of expert  opinion on the impact of search elements on  search results and the importance of these  elements in the peer review of electronic  search strategies; and a peer review forum to  test the procedural aspects of receiving and  responding to peer review requests  electronically.     the systematic review was used to find out if  any existing checklists could be identified, and  adapted if necessary, but no validated  checklists were found. we then reviewed the  information retrieval literature to develop an  evidence based set of elements that could be  used to evaluate electronic search strategies.  this draft set of items was additionally  informed by expert opinion and a final set of  peer review elements was developed. these  peer review elements represent only those  aspects of an electronic search strategy that  can be demonstrated to reduce precision or  recall (sensitivity) of the search if not skillfully  executed. these elements, therefore, represent  those aspects of the search that should be  subjected to peer review.    the full report of the methods and results of  this research was published by the canadian  association for drugs and technology in  health (cadth) (sampson, mcgowan,  lefebvre, moher, & grimshaw, 2008).  additionally, a more concise journal article  summarizing the findings was published  (sampson et al., 2009).    we presented our findings to groups of  librarians at meetings and workshops. it was  by the dissemination and interaction with the  stakeholders of this work (i.e., expert searchers  / librarians) that we realized that to translate  this work fully into practice a checklist with  concise guidance and an active web‐based  forum were required. based on this experience  and the findings described above, we  developed an evidence based checklist that  librarians and other searchers can use to guide  and inform the peer review of electronic  search strategies. we present this evidence  based assessment checklist here (table 1) with  an explanation of each of the elements (table  2).    we present these two tools for use in  accordance with a creative commons  attribution ‐ noncommercial ‐ no derivative  works 2.5 canada license  (http://creativecommons.org/). we have  selected a non‐commercial license to place  these tools in the public domain, to be used in  the spirit of indirect reciprocity or generalized  reciprocity (also known as ’pay it forward’)  (ohtsuki & iwasa, 2004; ʺpay it forwardʺ,  2009). rather than directly reciprocating the  benefit of receiving peer review of one’s own  search, one would reciprocate by undertaking  conscientious peer review of another search,  not necessarily a search by the person who  undertook the peer review of your search.  while the evidence on peer review is mixed, if  most grant applications, or most manuscripts,  are peer reviewed by two reviewers, than for  each submitted grant application or article, a  researcher or author should be willing to  undertake two peer reviews of other grant  applications or articles in order to sustain the  scientific community.  peer review of journal  articles and grant applications is often  anonymous. we suggest that if peer review of  electronic search strategies were not  anonymous, it might enhance the reviewer’s  reputation if done well. such a gain in  reputation should act as positive  reinforcement for the reviewer, increasing the  likelihood that their future reviews would also  be of high quality (mashima & takahashi,  2008). obtaining good quality peer reviews  should also strengthen the sense of  community among those using the press  150 evidence based library and information practice 2010, 5.1  151 forum, by advancing community objectives of  achieving the highest possible search quality  (mashima & takahashi, 2008). we have  selected a ‘no derivative work’ license, as the  elements of the checklist are evidence based.  we assert that they should be used without  amendment or local adaptation. if new  evidence is generated, the authors undertake  to update the checklist accordingly.    finding a suitable and willing peer reviewer is  an additional challenge for anyone wishing to  have their search peer reviewed. we are  currently exploring a press forum where  librarians/ expert searchers can submit their  own searches for peer review and identify  searches from other librarians/expert searchers  and peer review them. a prototype forum was  developed to define the important elements  for an effective, interactive peer review forum.  it was premised on the existence of strong  community and the values of reciprocity that  characterize the library and information  profession. the pilot forum presaged the  phenomenal rise of web‐based social  networking and used a programmed database  as a back‐end. a new press forum is being  developed using newer social networking  tools to ensure sustainability.     table 1  evidence based assessment form for the peer review of electronic search strategies (press ebc)  press ebc assessment form 1. translation of the research question: adequate adequate with recommendations needs revision provide an explanation or example _______________________________________________ _______________________________________________ 2. boolean and proximity operators: adequate adequate with recommendations needs revision provide an explanation or example _______________________________________________ _______________________________________________ 3. subject headings: adequate adequate with recommendations needs revision provide an explanation or example _______________________________________________ _______________________________________________ evidence based library and information practice 2010, 5.1  4. natural language / free-text: adequate adequate with recommendations needs revision provide an explanation or example _______________________________________________ _______________________________________________ 5. spelling, syntax and line numbers: adequate adequate with recommendations needs revision provide an explanation or example _______________________________________________ _______________________________________________ 6. limits and filters: adequate adequate with recommendations needs revision provide an explanation or example _______________________________________________ _______________________________________________ 7. search strategy adaptations: adequate adequate with recommendations needs revision provide an explanation or example _______________________________________________ _______________________________________________ © sampson m, mcgowan j, lefebvre c                   152 evidence based library and information practice 2010, 5.1  153 table 2  elements for the peer review of electronic search strategies (press ebc elements)    references acknowledgements     mashima, r. & takahashi, n. (2008). the  emergence of generalized exchange by  indirect reciprocity. in a. biel, d. eel, t.  gärling, & m. gustafsson (eds.), new  issues and paradigms in research on social  dilemmas (pp. 159‐176). new york, ny:  springer.  we wish to acknowledge and thank david  moher, jeremy grimshaw and elise cogo for  their work on the original grant from the  canadian agency for drugs and technologies  in health. we also wish to acknowledge and  thank janet joyce, raymond daniel, tamara  rader, nick barrowman, isabelle gaboury and  alex korablin for their roles in this project.  this research was supported by a capacity‐ building grant from the canadian agency for  drugs and technologies in health.   mcgowan, j. & sampson, m. (2005). systematic  reviews need systematic searchers.  journal of the medical library association,  93, 74‐80.   evidence based library and information practice 2010, 5.1  ohtsuki, h. & iwasa, y. (2004). how should we  define goodness? reputation dynamics  in indirect reciprocity. journal of  theoretical biology, 231, 107‐120.  sampson, m. & mcgowan, j. (2006). errors in  search strategies were identified by  type and frequency. journal of clinincal  epidemiology, 59, 1057‐1063.  sampson, m., mcgowan, j., cogo, e.,  grimshaw, j., moher, d., & lefebvre,  c. (2009). an evidence‐based practice  guideline for the peer review of  electronic search strategies. journal of  clinical epidemiology, 62, 944‐952.  sampson, m., mcgowan, j., lefebvre, c.,  moher, d., & grimshaw, j. m. (2008).  press: peer review of electronic search  strategies. cadth technical report.  retrieved 20 feb. 2010  from:  http://cadth.ca/index.php/en/publication/78 1.appendices  sampson, m., mcgowan, j., tetzlaff, j., cogo,  e., & moher, d. (2008). no consensus  exists on search reporting methods for  systematic reviews. journal of clinical  epidemiology, 61, 748‐754.  pay it forward. (2009). in wikipedia. retrieved  november 24, 2009, from  http://en.wikipedia.org/w/index.php?ti tle=pay_it_forward&oldid=327729147                154 http://cadth.ca/index.php/en/publication/781.appendices http://cadth.ca/index.php/en/publication/781.appendices http://en.wikipedia.org/w/index.php?title=pay_it_forward&oldid=327729147 http://en.wikipedia.org/w/index.php?title=pay_it_forward&oldid=327729147 commentary an evidence based checklist for the peer review of electronic search strategies (press ebc) we received funding from the canadian agency for drugs and technologies in health (cadth) to identify and assess the impact of errors in the elements of electronic search strategies associated with accuracy and completeness of the evidence base and to propose enhancements in the methods used for creating and evaluating search strategies. the project was called peer review of electronic search strategies (press). the research consisted of three main components: a systematic review of the evidence relating to quality issues and errors in electronic search strategies; a web-based survey of expert opinion on the impact of search elements on search results and the importance of these elements in the peer review of electronic search strategies; and a peer review forum to test the procedural aspects of receiving and responding to peer review requests electronically. the systematic review was used to find out if any existing checklists could be identified, and adapted if necessary, but no validated checklists were found. we then reviewed the information retrieval literature to develop an evidence based set of elements that could be used to evaluate electronic search strategies. this draft set of items was additionally informed by expert opinion and a final set of peer review elements was developed. these peer review elements represent only those aspects of an electronic search strategy that can be demonstrated to reduce precision or recall (sensitivity) of the search if not skillfully executed. these elements, therefore, represent those aspects of the search that should be subjected to peer review. the full report of the methods and results of this research was published by the canadian association for drugs and technology in health (cadth) (sampson, mcgowan, lefebvre, moher, & grimshaw, 2008). additionally, a more concise journal article summarizing the findings was published (sampson et al., 2009). press ebc assessment form evidence summary   a pilot to initiate research data management services within academic libraries helps librarians to learn about, engage with, and enhance skills within their research communities   a review of: read, k. b, koos, j., miller, r. s., miller, c. f., phillips, g. a., scheinfeld, l., & surkis, a. (2019). a model for initiating research data management services at academic libraries. journal of the medical library association, 107(3), 432–441. https://doi.org/10.5195/jmla.2019.545   reviewed by: joanne m. muellenbach library director and associate professor california health sciences university clovis, california, united states of america email: jmuellenbach@chsu.edu   received: 11 nov. 2020                                                             accepted:  5 jan. 2021      2021 muellenbach. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29879     abstract   objectives – to initiate or expand research data management (rdm) services within the participating libraries serving health sciences populations.   design – case report.   setting – six institutions consisting of three academic health sciences and three university libraries within the national network of libraries of medicine middle atlantic region in the united states of america.   subjects – between two and eight librarians participated from each institution, for a total of twenty-six librarian participants.    methods – pre-pilot phone interviews were conducted and included open-ended questions about rdm services, the library’s motivation for participating, and their degree of institutional commitment. to deepen their understanding of rdm, the participants were required to complete eight educational modules that included text, videos, and quizzes. the participating institutions received data interview questions to connect with their research community to be better informed about their attitudes, language, and practices. the participants also received a teaching toolkit, complete with slides, a script, and an attendee evaluation form. the participants were provided with a data series, consisting of branded classes for teaching over a designated period with instructors from within and outside of the library. collaboration with library partners was encouraged as was the use of a focused marketing plan. in fact, a major component of the pilot was the expert support, provided through biweekly meetings that included marketing tips and presentations on such topics as clinical research data management and data visualization.  finally, post-pilot program interviews were conducted, and the open-ended questions covered the pilot program as a whole and its individual components.   main results – of the six participating institutions, five institutions rated the rdm educational modules very positively. conducting data interviews was valuable for all six institutions because it allowed the librarians to meet with researchers, build relationships, and use what they learned to develop rdm services for the future. the teaching toolkit was rated positively by the six institutions, especially for its adaptability, the time saved over developing the content from scratch, and its usability. finally, the two institutions that held the data series courses stated that the series succeeded in further marketing the rdm services developed by the library.    conclusion – the pilot project met its objectives: the librarians at the participating institutions completed the educational modules, administered the data interviews, and taught an rdm foundations class based on the teaching toolkit. in addition, a data series was hosted at two institutions. the components of the pilot project had the intended results at each institution, and the classes were reviewed favorably. based on the pilot participants’ positive outcomes, the authors are certain that the freely available program materials would achieve success elsewhere.   commentary   rdm training provides individuals with the ability to review data, build on other researchers’ results, and reproduce their experiments. librarians have an opportunity to provide support in this area and to expand rdm services within the biomedical research community. this study builds on a curriculum that was designed by and for librarians to train researchers, as well as an online course developed by harvard university librarians (martin & goldman, 2017, 2019). this study’s program is unique in that it provides in-depth librarian training and a customizable teaching toolkit for delivering rdm training institution wide.      application of “the cat: a generic critical appraisal tool” (perryman, 2014) found the study to be concise and well organized, with links to detailed supplementary materials, including the well-designed preand post-pilot interview questions. the authors have in-depth knowledge about rdm services, as evidenced by the fact that their project received support from the national network of libraries of medicine (nnlm), and more than ten peer-reviewed papers on this topic are included in their reference list. the project provided the librarian cohort with customizable materials and, importantly, the expert mentoring needed to launch their own rdm services, and this resulted in positive ratings by the pilot participants.    one potential weakness of this study is that the pilot project relied on expert librarian mentors from just one institution, and this time commitment is not sustainable. perhaps a future initiative could involve a community of data librarian mentors that could be available via chat, email, phone, a slack channel, and web conferencing. the lead author is already working to address this issue as the coauthor of a commentary that describes a rationale to create the medical library association (mla) data services competency to further data and open science skills development (federer et al., 2020). the authors also highlighted that an mla data special interest group has been developed, as well as nnlm rdm class cohorts. in addition, librarians from smaller libraries could partner with instructors from outside the library and thereby expand course offerings. this would also serve to enhance interprofessional education initiatives, provide opportunities for teaching, and strengthen the program overall.   the authors succeeded in providing a cohort of librarians with rdm training and a customizable toolkit for initiating and enhancing rdm services within their institutions, and this report makes a significant contribution to the literature. this implies that the institutions of this librarian cohort must provide resources to help foster their development as rdm experts. given that the study was published in 2019, expanding the original toolkit to include the rdm materials from librarians that were tailored to the needs of their institutions would benefit future librarian participants from a wide variety of institutional settings and ensure that the materials are up to date. in addition, since rdm principles and practices are the same worldwide, future studies may wish to focus on researchers from different countries, disciplines, ethnic groups, and settings who receive this training and demonstrate its value in furthering research productivity.     references   federer, l., foster, e. d., glusker, a., henderson, m., read, k., & zhao, s. (2020). the medical library association data services competency: a framework for data science and open science skills development. journal of the medical library association, 108(2), 304–309. https://doi.org/10.5195%2fjmla.2020.909   martin, e., & goldman, j. (2019). best practices for biomedical research data management. canvas network. https://www.canvas.net/browse/harvard-medical/courses/biomed-research-data-mgmt   martin, e., & goldman, j. (2017). new england collaborative data management curriculum. lamar soutter library. https://library.umassmed.edu/resources/necdmc/index   perryman, c., & rathbun-grubb, s. (2014). the cat: a generic critical appraisal tool. https://www.jotform.us/cp1757/thecat         microsoft word es_loy.doc evidence based library and information practice 2007, 2:2  87    evidence based library and information practice         evidence summary    new search strategies successfully optimize retrieval of clinically sound treatment  studies in embase       a review of:  wong, sharon s‐l, nancy l. wilczynski, and r. brian haynes. “developing optimal search  strategies for detecting clinically sound treatment studies in embase.” journal of the  medical library association 94.1 (jan. 2006): 41‐47. 14 may 2007  http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=1324770.    reviewed by:   john loy  learning resources manager, avon & wiltshire mental health partnership nhs trust  callington road hospital  brislington, bristol, united kingdom  e‐mail: john.loy@awp.nhs.uk      received: 01 march 2007  accepted: 24 april 2007      © 2007 loy. this is an open access article distributed under the terms of the creative commons attribution  license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and  reproduction in any medium, provided the original work is properly cited.      abstract    objective – to develop and test the  sensitivity and specificity, precision and  accuracy of search strategies to retrieve  clinically sound treatment studies in the  embase database.    design – analytical study.    setting – methodologically sound studies of  treatment from 55 journals indexed in  embase for the year 2000.    subjects – embase and hand searches  performed at the health information  research unit of mcmaster university,  ontario, canada.     methods – the authors compare the results  of embase searches using their search  strategies with the “gold standard” of  articles retrieved by hand search. research  assistants initially hand searched each issue  of 55 selected journals published in 2000 to  identify articles detailing studies on  healthcare treatment of humans. subject  coverage of the journals was wide ranging  and included obstetrics and gynaecology,  psychiatry, oncology, neurology, surgery  and general practice. studies were then  assessed to ensure they met the qualifying  criteria: random allocation of participants to  groups, outcome assessment of at least 80%  of participants who began the study, and  analysis consistent with study design.  http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=1324770 mailto:loy@awp.nhs.uk http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:2  88 initially, 3850 articles on treatment were  identified, of which 1256 (32.6%) were  methodologically sound. to construct a  comprehensive set of search terms, input  was sought from librarians and researchers  in the us and canada.  this initially  produced a list of 5385 terms, of which 4843  were unique and 3524 produced hits.  individual search terms with sensitivity  greater then 25% and specificity greater then  75% were incorporated into search strategies  for use within the ovid interface for the  embase database to retrieve articles  meeting the same criteria. these strategies  were developed using all 27,769 articles  published in the 55 journals in 2000. this all‐ inclusive approach was used to test the  search strategies’ ability to identify high‐ quality treatment articles from a larger pool  of material.    main results – the single term which  achieved best sensitivity was “random:mp,”  with a sensitivity of 95.1%. this same term  achieved a high specificity of 92.5%. the  best‐performing single term for specificity  was “randomized:tw” at 96.7%, but this did  reduce sensitivity to 63.2%. the single term  to achieve the best balance between the two  was “clinical trial:mp,” with a sensitivity of  88.3% and specificity of 88.0%. combining  terms produced varied results, and table 3  within the article details terms used to give  the best combinations for sensitivity,  specificity and optimisation of both. the  best three‐term search strategies for  sensitivity achieved a rate just shy of 99%  with a specificity of 72.0%, while the  optimum three‐term strategy for specificity  achieved 96.7% but with a trade off of  lowering the rate of sensitivity to 51.7%. the  best‐performing combination of search  terms to optimise sensitivity and specificity  produced values exceeding 92% for both.    conclusion – the authors present search  strategies which can successfully be used to  retrieve methodologically sound studies on  the prevention and treatment of disease and  health complications indexed on the  embase database. a clear outline of the  trade‐off between sensitivity and specificity  of the strategies is included.    commentary      as the authors highlight in their  introduction, keeping up to date with a  healthcare knowledge base which is  expanding at the rate of over 2 million new  articles a year is a labour‐intensive business.  separating the wheat from the chaff in order  to focus on high quality evidence based  literature is an increasingly important role  for library services in supporting the busy  clinician. with this objective, the authors set  out to develop optimal search strategies in  order to identify high‐quality treatment  studies in embase. the work is believed to  be the first time such strategies have been  developed for this database and the authors  most certainly achieve their objective.     this is one of several articles by wilczynski  and colleagues on search strategies, all of  which are referenced in the original article,  and as their earlier work has been adapted  for use as clinical queries search filters in  pubmed, we can be assured that we are in  safe hands. in many ways this is a follow‐up  piece to an article the authors conducted on  the medline database (haynes 2005) and  which itself was the subject of an evidence  summary published in eblip (brown  2006) .that medline study was praised in  eblip as “evidence‐based practice at its  best” and this companion piece is of an  equally high calibre.     the authors are able to build on their  previous work in the field, being confident  to test the strategies on a comparatively  small set of 55 journals whittled down from  an initial 135 titles. their previous work  using medline had demonstrated that  using a smaller dataset had produced no  evidence based library and information practice 2007, 2:2  89 statistically significant differences from  using a larger one, whilst significantly  decreasing computation time.     a possible shortcoming of the study, and  one not addressed by the authors, presents  itself when contemplating the 55 titles  selected. these 55 are all “big‐name”  mainstream journals which also are indexed  in medline. as the overlap between  medline and embase is 30‐50%, one  wonders whether the filters would work as  well if applied to a broader selection of  journal titles and subject headings which are  available within embase but not medline.    one of the great features of this article is the  exemplary use of tables throughout, each  one clearly laid out and easy to interpret.  the complex search strategies are detailed in  a straightforward table where at a glance it  is easy to compare sensitivity, specificity,  precision and accuracy. considering the  numerous figures involved and the potential  for confusion, the result is refreshingly  simple to absorb, and other authors should  take note of how to present complex data in  an easily digestible manner.    the authors are also prepared to highlight  the limitations of their study. we are made  aware that there is a real trade off between  specificity and sensitivity of search results.  using their best specificity strategy, the  sensitivity of the search drops dramatically  to nearly 50%, meaning that almost 1 in 2  clinically relevant articles would be missed.  searchers need to ponder which is the most  appropriate filter for their needs.     it never hurts to remind ourselves that while  rcts are a robust methodology, not all  research lends itself to the format of a  randomized controlled trial. while an rct  will be the research methodology of choice  for high‐quality treatment research, its use is  not always possible or appropriate. a great  deal of rigorous and methodologically  sound research takes place using, for  example, cohort studies, which is an equally  valid methodology. even if we do embrace  the rct as king we must remember that not  all rcts included on embase will have  been conducted to the same standards,  coming as they do from a wide variety of  journals with differing publication criteria.     an excellent illustration is included of what  all this means in practical terms, with a  worked example of a search on herbal  medicine in table 4. taking the initial  results of 5696 articles, and combining them  with the search strategy to optimise  sensitivity and specificity, the results are  reduced to 427 hits or around 7.5% of the  original number. while completists may be  prepared to hunt through nearly 6000  references, i for one will be happy to use the  filters presented here and be confident that  any research of note is unlikely to be  missing.      works cited    haynes, r. brian, k. ann mckibbon, nancy  l. wilczynski, stephen d. walter, and  stephen r. were. “optimal search  strategies for retrieving scientifically  strong studies of treatment from  medline: analytical survey.” british  medical journal 330.7501 (21 may 2005):  1179‐1182. 14 may 2007   <http://www.bmj.com/cgi/content/abst ract/330/7501/1179#otherarticles>.    brown, marcy l. “new search strategies  optimize medline retrieval of  sound studies on treatment or  prevention of health disorders.”  evidence based library and  information practice 1.2 (2006):  23‐25.  14 may 2007  <http://ejournals.library.ualberta.ca/ind ex.php/eblip/article/view/45/111>.  http://www.bmj.com/cgi/content/abstract/330/7501/1179#otherarticles http://www.bmj.com/cgi/content/abstract/330/7501/1179#otherarticles http://ejournals.library.ualberta.ca/index.php/eblip/article/view/45/111 http://ejournals.library.ualberta.ca/index.php/eblip/article/view/45/111 microsoft word es_haley.doc evidence based library and information practice 2006, 1:4      57 evidence based library and information practice       evidence summary    analysis of print and electronic serials’ use statistics facilitates print cancellation  decisions   a review of:  gallagher, john, kathleen bauer, daniel m. dollar. “evidence‐based librarianship: utilizing data  from all available sources to make judicious print cancellation decisions.”  library  collections, acquisitions & technical services 29.2 (2005): 169‐79.      reviewed by:  pamela haley  associate librarian, access/technical services  william r. lederman law library  queen’s university  kingston, ontario, canada  e‐mail: haleyp@post.queensu.ca      received: 23 august 2006            accepted: 29 september 2006      © 2006 haley. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to apply the principles of  evidence‐based librarianship to the decision‐ making process regarding the cancellation  of print serials.    design – quantitative analysis of local and  national data from various sources.    subjects – data sources included 1249  current unbound print journals, 3465  medline‐indexed electronic journals,  statistics from the association of research  libraries and american association of  health sciences libraries, as well as  traditional library statistics.    setting – the study was conducted in the  yale university’s cushing/whitney medical  library located in new haven, connecticut  u.s.a.    methods – several sources were targeted for  data.  a three‐month periodical usage study  of the current issues of the library’s 1249  actively received print titles was undertaken.   excel‐generated alphabetical listings of titles  were used by shelvers to indicate, with a  check mark, which issues were shelved   http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:4      58 during a specified week.  the workflow was  adjusted to ensure only items under study  were counted.  signs asking patrons not to  re‐shelve journal issues were posted.  usage  data were collected weekly and entered into  an excel spreadsheet where the total use of  the journals was tracked. in‐house  circulation, photocopy, and gate count  statistics were also used. in addition to the  survey, sfx statistics for the library’s  electronic journals indexed in medline  (3465) were gathered during the same 3  month period covered by the print usage  survey.  medline was chosen as the  delineating factor to ensure consistent  subject coverage with the print journal  collection.  for perspective and trends,  statistics from the association of research  libraries and the american association of  health sciences libraries were considered.    main results – based on the study’s  findings, 53% of the print collection (657  titles) received no use during the study  period; 7.1 % (89 titles) were used more than  once per month; and 1.28% were used one  or more times per week. further, only 10%   (125 titles) of the collection represented  60.7% of the total print collection use.  there  was also a direct correlation between the  drop in patrons coming to the library and  the decrease in print periodical use.   sfx  statistics revealed that of the 3465 medline  indexed titles 14.8% (513 titles) were not  accessed at all and 10% of the journals  represented 56.8% of all sfx usage.  these  results were consistent with statistics from  the association of research libraries and  the american association of health sciences  libraries.    conclusion – titles that were used the most  in print were also used the most  electronically. further, the study revealed  that print journals are used only a fraction  as often as their electronic counterparts.   indeed, in both the case of print and  electronic journals the largest use came from  a small number of subscribed titles.  print  collection maintenance is more labour  intensive and costly than electronic.   consequently, resources spent supporting  53% of the print collection that is not used  seriously impacts efficiency.  with  constraints on acquisitions budgets, funding  unused collections does not make sense.  examination of the print serial collection is  only part of ensuring effective collections.   as this study has indicated, unused  electronic titles are also a drain on resources  and further analysis of electronic packages  is warranted.       commentary    this study looks at print and electronic  journal usage statistics to provide indicators  as to what print collection titles should be  cancelled.  the study will be of interest to all  librarians wrestling with the challenges of  maintaining a relevant and cost‐effective  journal collection, and is straightforward  enough to be readily duplicated.  the raison  d’etre behind the analysis was reasonably  well articulated, although greater  elucidation as to why the selected data  sources were chosen would have added to  the reader’s understanding of the process.    the methodology used compared 3 months  worth of use statistics for print titles in the  collection and electronic journal titles that  covered the same subject areas in sfx.  the  use of sfx slightly skews the results as sfx  accounts for only a small portion of the  library’s electronic usage.  still, trends are  identifiable and useful.  no margin of error  was calculated to allow for shelving errors.   a brief description of the alternative study  methods that were considered would have  been helpful.    overall this approach is reasonable, given  the financial constraints facing many  libraries, and the analysis adds insight to the  decision‐making process behind print  evidence based library and information practice 2006, 1:4      59 journal cancellations.  conducted in a  medical library, the question remains as to  whether the findings are valid in the field of  social sciences and humanities.  further  study is warranted to answer this question.   the indication that some electronic journals  received little or no use should also be                                                                                     investigated in terms of how to keep the  electronic collection relevant in light of the  difficulties of unbundling.  further  investigation into whether the fact that sfx  data represented only 8.5% of access in this  study is consistent with the experience of  other libraries could be quite revealing.  evidence summary   first year university students arrive with some search skills, but struggle with scholarly sources   a review of: salisbury, f., & karasmanis, s. (2011). are they ready? exploring student information literacy skills in the transition from secondary to tertiary education. australian academic & research libraries, 42(1), 43-58. http://dx.doi.org/10.1080/00048623.2011.10722203   reviewed by: cari merkley associate professor mount royal university library calgary, alberta, canada email: cmerkley@mtroyal.ca   received: 9 mar. 2014    accepted: 25 apr. 2014      2014 merkley. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine what existing information literacy skills first year students possess upon entering university.   design – quantitative survey questionnaire.   setting – a research university in australia.   subjects – 1,029 first year students in the health sciences.   methods – first year students enrolled in the health sciences were asked to complete a paper questionnaire in their first week of classes in 2009. the 20 question survey was distributed in student tutorial groups. the first 10 questions collected information on student demographics, expected library use, and existing information seeking behaviour. the remaining 10 questions tested students’ understanding of information literacy concepts. data collected from the survey were analyzed using the statistical software spss.   main results – most of the students who responded to the questionnaire were between the ages of 16 and 21 (84.3%) with only 2.2% over the age of 40. approximately 15% of respondents had completed some postsecondary university or vocational education prior to enrolling in their current program.   the students ranked google, a friend, and a book as the top three places they would go to find information on something they knew little about. google was also the most popular choice for finding a scholarly article (35% of respondents), followed by the library catalogue (21%).   a large proportion of students correctly answered questions relating to identifying appropriate search terms. for example, one third of the students selected the correct combination of search concepts for a provided topic, and 77% identified that the choice of search phrase could negatively impact search results. students also demonstrated prior knowledge of the boolean operator and, with 38% correctly identifying its use in the related question. most students were also able to identify key markers of a website’s credibility.   questions relating to ethical information use and scholarly literature proved more challenging. almost half (45%) of the students said that they did not know the characteristics of a peer reviewed journal article. twenty five percent of respondents indicated that citing an information source was only necessary in the case of direct quotes, with only 28% correctly identifying the need for citing both quotes and paraphrasing. only 23% were able to select the example of a journal citation from the list presented.   conclusion – students enter university with existing strengths in concept identification and basic search formulation, but require the most assistance with locating and identifying scholarly literature and how to cite it appropriately in their work. the findings will inform the development of an online information literacy assessment tool to assist incoming students in identifying areas where they may require additional support as they transition to university.     commentary   from the beginning of the article, it is clear that the authors are firmly rooted in a constructivist approach to learning, even if the theory itself is never named. while the pre/post test method has long been a part of library research as a way to measure the impact of educational interventions, the authors’ focus on the pre-test results here suggests an attempt to further demonstrate that a student is not an empty vessel to be filled with information literacy knowledge, but an individual with strengths that can be harnessed as part of the learning process. the constructivist approach to learning with its focus on individual meaning making, building on existing knowledge, interactivity, and tasks that reflect real life concerns, continues to be a dominant pedagogical force in information literacy instruction today (cooperstein & kocevar-weidinger, 2004).     the article itself is a brief snapshot of a larger study that included a post-test with the same cohort at the end of the academic year. it largely stands on its own, but the relevant data tables and discussion of some of the survey question results were left out of this work, perhaps in the interest of brevity. for example, readers may be interested to know that almost 60% of the students who completed the pre-test reported that they had encountered information literacy instruction in their previous studies (fisch, karasmanis, salisbury, & corbin, 2009). the study’s strengths include an excellent response rate (63%) and the use of a previously validated survey instrument (glynn, 2006). the survey was based on one initially used by mittermeyer and quirion with incoming students at quebec universities in 2003. this survey has been used by a number of other researchers internationally since its original publication, allowing the authors to compare their responses to the work of others. however, even at the time the authors used the tool in 2009, the instrument was showing its age. the authors noted that several questions needed to be reworded because of references to outdated information tools. there was also a surprising emphasis in the survey on how to use the library catalog, with four questions addressing this particular tool, and only one specific question relating to websites. this may seem more jarring to readers in 2014, when the use of discovery services by many university libraries has eroded the traditional boundaries between library catalogues and databases.   the survey instrument is also based on a set of information literacy competencies that are currently being rewritten by the library community. the 2004 australian and new zealand information literacy framework referenced by the authors and the original 2003 survey are both based on the 2000 american college and research libraries’ (acrl) information literacy competency standards for higher education (mittenmeyer & quirion, 2003). the first draft of what is being described as a new framework for information literacy for higher education was released by acrl in february 2014, with part two of the draft set for release in april 2014. the release of the acrl’s framework marks a significant shift in direction away from the existing standards’ “limited, almost formulaic approach to understanding a complex information ecosystem” (acrl, 2014, p. 3). the authors’ constructivist approach to learning still resonates within the acrl framework draft, but the task-focused nature of several questions asked in the pre-test survey seems to reflect some of the concerns expressed about the standards on which they were based. while it remains important to consider students’ existing knowledge in order to design appropriate and useful information literacy support, the development of new research instruments with which to better capture the complexity of students’ understanding of their information environment is necessary.     references   association of college and research libraries. (2014). framework for information literacy for higher education, draft 1, part 1. retrieved from http://acrl.ala.org /ilstandards/wp-content/uploads/2014/02/framework-for-il-for-he-draft-1-part-1.pdf   cooperstein, s. e., & kocevar-weidinger, e. (2004). beyond active learning: a constructivist approach to learning. reference services review, 32(2), 141-148. doi:10.1108 /00907320410537658   fisch, e., karasmanis, s., salisbury, f., & corbin, j. (2009). library pre and post experience survey: final report. retrieved from http://arrow.latrobe .edu.au:8080/vital/access/manager/repository/latrobe:20691   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi:10.1108/07378830610692154    mittermeyer, d., & quirion, d. (2003). information literacy: study of incoming first-year undergraduates in quebec. retrieved from http://www.crepuq.qc.ca/documents /bibl/formation /studies_ang.pdf   evidence summary   searching on health information databases: a search interface including thesaurus term and tree browsers is more effective than a simple search interface   a review of: mu, x., lu, k., ryu, h. (2014). explicitly integrating mesh thesaurus help into health information retrieval systems: an empirical user study. information processing and management, 50(1), 24-40. http://dx.doi.org/10.1016/j.ipm.2013.03.005   reviewed by: joanne l. jordan research information manager arthritis research uk primary care centre, keele university keele, staffordshire, united kingdom email: j.jordan@keele.ac.uk   received: 12 june 2014  accepted: 30 oct. 2014      2014 jordan. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objectives – to compare the effectiveness of a search interface with built-in thesaurus (mesh) terms and tree browsers (meshmed) to a simple search interface (simplemed) in supporting health information retrieval. researchers also examined the contribution of the mesh term and tree browser components towards effective information retrieval and assessed whether and how these elements influence the users’ search methods and strategies.   design – empirical comparison study.   setting – a four-year university in the united states of america.   subjects – 45 undergraduate and postgraduate students from 12 different academic departments.   methods – researchers recruited 55 students, of which 10 were excluded, using flyers posted across a university campus from a wide range of disciplines. participants were paid a small stipend taking part in the study.   the authors developed two information retrieval systems, simplemed and meshmed, to search across a test collection, ohsumed, a database containing 348,566 medline citations used in information retrieval research. simplemed includes a search browser and a popup window displaying record details. the meshmed search interface includes two additional browsers, one for looking up details of mesh terms and another showing where the term fits into the tree structure. the search tasks had two parts: to define a key biomedical term, and to explore the association between concepts. after a brief tutorial covering the key functions of both systems, avoiding suggestion of one interface being better than the other, each participant then searched for six topics, three on each interface, allocated randomly using a 6x6 latin square design.   the study tracked participants’ perceived topic familiarity using a 9-point likert scale, measured before and after each search, with changes in score recorded. it examined the time spent in each search system, as recorded objectively by system logs, to measure engagement with searching task. finally, the study examined whether participants found an answer to the set question, and whether that response was wrong, partially correct, or correct. participants were asked about the portion of time they spent on each of the system components, and transaction log data was used to capture transitions between the search components. the participants also added their comments to a questionnaire after the search phase of the experiment.   main results – the baseline mean topic familiarity scores were similar for both interfaces, with simplemed’s mean of 2.01, with a standard deviation 1.43, compared to meshmed’s mean of 2.08 with a standard deviation of 1.60. the mean was taken for topic familiarity change scores over three questions on each interface and compared using a paired sample two-tailed t-test. this showed a statistically significant difference between the mean change in topic familiarity scores for simplemed and meshmed.   only 46 (17%) of the questions were not answered, 34 (74%) when participants were using simplemed and 12 (26%) when using meshmed. researchers found a chi-squared test association between the interface and whether the answer was correct, suggesting that meshmed users were less likely to answer questions incorrectly. the question-answer scores positively correlated to the topic familiarity change scores, indicating that those participants whose familiarity with the topic improved the most were more likely to answer the question correctly.   the mean amount of time spent overall using the two interfaces was not significantly different, though researchers do not provide data on mean times, only total time and test statistics. on the meshmed interface, on average participants found the term browser feature the most useful aspect and spent the most amount of time in this component. the tree browser feature was rated as contributing the least to the searching task and the participants spent the least amount of time in this part of the interface.   patterns of transitions between the components are reported, the most common of which were from the search browser to the popup records, from the term to the search browser, and vice versa. these observations suggest that participants were verifying the terms and clicking back and forth between the components to carry out iterative and more accurate searches. the authors identify seven typical patterns and described four different combinations of transitions between components.   based on questionnaire feedback, participants found the term browser helpful to define the medical terms used, and for additional suggested terms to add to their search. the tree browser allowed participants to see how terms relate to each other, and helped identify related terms, despite many negative feedback comments about this feature. almost all participants (43 of 45) preferred meshmed for searching, finding the extra components helpful to produce better results.   conclusion – meshmed was shown to be more effective than simplemed for improving topic familiarity and finding correct answers to the set questions. most participants reported a preference for the meshmed interface that included a term browser and tree browser to the straightforward simplemed interface. both meshmed components contributed to the search process; the term browser was particularly helpful for defining and developing new concepts, and the tree browser added a view of the relationship between terms. the authors suggest that health information retrieval systems include visible and accessible thesaurus searching to assist with developing search strategies.     commentary   health literacy is a fast-growing area of research and this study looks to contribute to this area. the study evaluates the usefulness of providing easy, visible access to thesaurus and tree browsers to enhance retrieval of health information from a bibliographic health database. on the whole, the study is well designed and conducted when evaluated against glynn’s critical appraisal tool (2006). one shortcoming is that effect sizes should be reported, such as the mean change in topic familiarity scores, rather than only the statistical tests and significance (sullivan & feinn, 2012).   the participants in this study are well-educated university students familiar with research. it is worth questioning whether they serve as a representative sample of consumers looking for health information online who may be unfamiliar with medical terminology. participants were also volunteers and paid for their involvement, which may have biased the sample, although this is unlikely as the remuneration is very small. the study findings may not be generalizable to all health consumers or even all university students due to the small sample size.   the new systems under study searched across a subset of the medline database records, whose intended audience is health professionals and researchers. therefore, this is not the most accessible source of health information for a general consumer. there are many reliable, pre-appraised, synthesised, evidence-based health resources available on the internet, such as medlineplus (http://www.nlm.nih.gov/medlineplus/) and patient.co.uk (http://www.patient.co.uk/).  these resources are specifically written without medical terminology for a lay audience, and do not need thesaurus browsers to be able to find the information that is required. health consumers, particularly lay people, require skills in judging if online health information comes from a reliable source and should be directed toward trustworthy resources by health professionals or professional organisations.   although the study results may only directly apply to a small proportion of health consumers, many health professionals and medical students often struggle with retrieving answers to clinical questions from databases such as medline. search engines and interfaces have tended to simplify the searching functions in recent years, but this study suggests that people need more tools that increase personal understanding of a topic. mesh browsers are available on pubmed and other interfaces, but generally users have to know what thesaurus terms are, how to use them, and where to find them in the system, to be able to use them effectively.   searchers who may not already be familiar with a topic tend to use general search engines and online dictionaries to help find and define related terms before searching a database. the built-in mesh ‘scope notes’ do not always give adequate definitions of the thesaurus terms. more resources, such as thesaurus and tree browsers, that are easily accessible and visible should be built into search interfaces.   often thesaurus searching is seen as too advanced and not taught with simple searching techniques. therefore, this study has implications for how searching skills are taught to health professionals, and suggests the importance of search skills instruction that highlights the added benefit of using the in-built thesaurus from the beginning. the findings of this study are particularly useful for librarians and other information professionals who teach search skills, and may benefit and influence developers of search interfaces.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154   sullivan, g. m., & feinn r. (2012). using effect size—or why the p value is not enough. journal of graduate medical education, 4(3), 279-282. http://dx.doi.org/10.4300/jgme-d-12-00156.1 library staff need more support in order to alleviate teaching anxiety evidence summary   library staff need more support in order to alleviate teaching anxiety   a review of: lundstrom, k., fagerheim, b. & van geem, s. (2021). library teaching anxiety: understanding and supporting a persistent issue in librarianship. college & research libraries, 82(3), 389-409. https://doi.org/10.5860/crl.82.3.389   reviewed by: jessica a. koos senior assistant librarian/health sciences librarian stony brook university stony brook, new york, united states of america email: jessica.koos@stonybrook.edu   received: 14 sept. 2021                                                             accepted:  21 oct. 2021      2021 koos. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30041     abstract   objective – to determine academic librarians’ attitudes towards their teaching, how teaching anxiety manifests itself, and how teaching anxiety affects these attitudes.   design – online survey.   setting – the survey was distributed through various library science listservs.   subjects – any library staff with a teaching component in their role were invited to respond. there was a total of 1,035 initial responses.   methods – the survey questions were based on a previously published survey about teaching anxiety by davis (2007). however, the survey for this study added questions about formal and self-diagnosis of other types of anxieties, physical and psychological anxiety symptoms, and how teaching anxiety impacts other areas of the respondents’ lives. there were also questions on potential supports to reduce teaching anxiety, as well as potential barriers to these supports.   main results – it was found that approximately 65% of respondents experience teaching anxiety. approximately 40% of those respondents were formally diagnosed with anxiety, and approximately 42% were self-diagnosed. there was a significant association between a formal diagnosis of anxiety, and teaching anxiety. there were also significant associations between past training, preparation, and teaching anxiety, with anxiety occurring less with increased training and preparation.   conclusion – teaching anxiety is a significant issue among library staff. supports in the form of workshops on teaching as well as coping with anxiety can possibly help to reduce this phenomenon.   commentary   the quality of this study was appraised using “the cat: a generic critical appraisal tool” created by perryman & rathbun-grubb (2014). overall the article was found to be of high quality based on this assessment. the first two authors are research and instruction librarians at a university, and the third author is a sociology faculty member. the research questions were clearly defined, and matched the method used. the single limitation that was mentioned in the article was the fact that the respondents were self-selected, as opposed to being randomly selected. the authors stated that they “attempted to mitigate” this factor by obtaining a high number of responses, however the response rate does not seem high enough to have achieved this goal.   a copy of the survey itself was included as an appendix, however a significant limitation was that it was not piloted. it was not clear if the survey was open to librarians or library staff, since all the respondents were referred to as “librarians” throughout the article. also, geographical information was not obtained, which is a significant piece of information, especially since the survey was distributed through both national and international listservs.   one of the major findings of this article is that teaching anxiety in library staff is linked to other formally diagnosed anxiety disorders, which is not surprising. what is shocking is that there is such a large number of respondents (367 out of 925, or 39.68%) that reported having a formal diagnosis. the authors include statistics showing that the rate of anxiety diagnosis in the general population both internationally and in the u.s. is much lower. further research is necessary to clarify this finding. is it that people with anxiety are drawn to the profession, or are there certain aspects of it that may be anxiety provoking, such as teaching? could this be related to the fact that the respondents were self-selected, and therefore those with anxiety were more likely to participate in the survey?   it is unclear as to why the authors chose to include an option for self-diagnosis of anxiety disorders in this survey. 41.65% of those not formally diagnosed with anxiety claimed to be self-diagnosed. it is highly questionable that the respondents would be able to accurately diagnose themselves with these conditions, as this is typically done by trained mental health professionals. this should be taken into account when examining the data for these respondents.   some of the factors that contributed to teaching anxiety were found to be, “fear of looking foolish or unknowledgeable in front of others” (13.70%), “fear of being unprepared” (12.73%), and “no previous contact or rapport with students” (11.26%). not surprisingly, it was found that respondents that had previous coursework in information literacy instruction were less likely to experience teaching anxiety. additionally, those who felt prepared to teach were less likely to have anxiety than those that were not. thus, it seems that training and preparation may play a vital role in mitigating this issue.   respondents also indicated that they would be willing to enroll in a course or workshop on teaching, coping with teaching anxiety, as well as coping with anxiety in general. they also indicated that a structured group of colleagues, or peer observations and feedback on teaching may be helpful as well. libraries need to acknowledge that teaching anxiety is a significant issue for many, and multiple supports should be put in place to for those with this issue. additionally, libraries should consider alternative staffing patterns and workloads for those that prefer not to teach.   references   davis, k.d. (2007). the academic librarians as instructor: a study of teacher anxiety. college & undergraduate libraries. 14(2). 77-107. https://doi.org/10.1300/j106v14n02_06   perryman, c. & rathbun-grubb, s. (2014). the cat: a generic critical appraisal tool. in jotform – formbuilder. http://www.jotform.us/cp1757/thecat   evidence summary   relevancy trumps format when teaching information literacy   a review of: tewell, e. c. (2014). tying television comedies to information literacy: a mixed-methods investigation. the journal of academic librarianship, 40(2), 134-141. doi:10.1016/j.acalib.2014.02.004   reviewed by: giovanna badia liaison librarian, schulich library of science and engineering mcgill university montréal, québec, canada email: giovanna.badia@mcgill.ca   received: 8 sep. 2014     accepted: 18 nov. 2014      2014 badia. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract objective – this study assessed the effects of showing television comedy clips to demonstrate information literacy concepts when teaching one-shot instruction sessions. more specifically, it examined whether the students’ retention and understanding increased when television comedy clips were used and whether students preferred instruction that included popular culture examples. design – a mixed-methods investigation that employed multiple-choice questionnaires and focus group interviews. setting – a small liberal arts college in the united states of america. subjects – a total of 211 freshmen students enrolled in a first-year studies course. the students were divided into 16 class sections. the author collected a total of 193 valid responses to the pretests and posttests in his study. methods – half of the class sections (103 respondents) were taught selected information literacy concepts using television comedy clips and a group discussion led by the instructor. the other half (90 respondents) were taught using only an instructor-led discussion. the classes were randomly selected to belong to the experimental group (with tv comedy clips) or the control group (without tv comedy clips). an online pretest questionnaire, consisting of 10 multiple-choice questions, was administered at the beginning of the 90-minute library instruction session for both groups. an online posttest questionnaire, consisting of the same questions as the pretest but in a randomized order, was completed by the students at the end of the session. about a month later, one-hour focus group interviews were conducted with a small subset of the study’s subjects who volunteered to participate in the focus groups. the experimental focus group consisted of five study participants who had attended a library instruction session that involved showing the television comedy clips and the control focus group consisted of six study participants who had attended a library instruction session that did not include showing the television comedy clips.   main results – the experimental group scored higher than the control group on the posttest with an average “increase of 1.07 points from preto posttest compared to a 0.13 mean increase in the control group” (p. 139), which means that the experimental group answered one more question correctly. four out of the five participants in the experimental focus group also discussed the television comedy clips even though they were not asked about them. conversely, when asked about what they enjoyed in the class, the majority of participants from both focus groups discussed the content covered in the session rather than any teaching methods employed. “the quantitative results suggest that student test results either increased, as in the experimental group, or remained relatively level, as in the control group, due to the type of instruction received” (p. 137).   conclusion – the author states that the results from the test questionnaires and answers from focus group sessions indicate that using television comedy clips may be a successful way of improving students’ retention of course content. however, the study’s results could not demonstrate that students liked classes with popular culture examples more than classes without them, since the majority of focus group participants found the course content more interesting than the manner in which the content was taught. the relevancy of the content presented in an information literacy session appears to make more of an impact on the students than the format in which it is presented.   commentary   this well-conducted study supplements the small pool of existing literature on the use of popular culture as one method of supporting information literacy by seeking to answer research questions that build upon the literature. glynn’s (2006) ebl critical appraisal checklist was applied to this study, which ranks highly for the data collection, study design, and results questions in the checklist. the author clearly describes the study’s methods and results for the readers to form their own interpretation of the data, which will support the logic of the author’s conclusions. in addition to the description in the methods section, the appendices contain the questions used in the pretest, posttest, and focus groups, making it easy for readers to reproduce the study. the only missing detail is how the groups were randomized. in this reviewer’s opinion, this article can serve as one possible example of how an assessment study should be written and conducted.   this study does not rank as highly for the population section of questions in the critical appraisal checklist, due to the author’s use of a small “nonrandom convenience sample” (p. 140). the author indicates the sample size as a limitation of the study, since the results cannot be generalized to the entire undergraduate population. he cautions readers to take into account their own student population and organizational characteristics when deciding how to apply the study’s findings.   librarians teaching information literacy sessions will be able to apply the study’s major finding, “that the fundamental difference that encourages student learning appears to lay not in the specific format but in making information literacy more relevant and accessible to students’ lives” (p. 140), to their own teaching methods. the choice of whether to employ popular culture in information literacy sessions is up to the instructor, since there was only an average increase of 1.07 points between the students’ scores on the pretests and posttests. while this is statistically significant, it may not be a large enough difference to be practically significant such that librarians would be convinced to start incorporating television comedy clips into their own information literacy sessions if they are not already doing so. references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi:10.1108/07378830610692145   conference paper   measuring the value of library resources and student academic performance through relational datasets   margie jantti university librarian university of wollongong wollongong, new south wales, australia email: margie@uow.edu.au   brian cox manager, quality and marketing university of wollongong library wollongong, new south wales, australia email: brian_cox@uow.edu.au      2013 jantti and cox. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – this article describes a project undertaken by the university of wollongong library (uwl) to identify whether a correlation exists between usage of library resources and academic performance.   methods – a multidimensional approach to systems design was implemented, requiring collaboration between among the library, university administration, performance indicator project team (pip), and information technology services. the project centers on the integration and interrogation of a series of discrete datasets containing student performance, attrition, demographic, borrowing, and electronic resources usage data. pip built a cube for the library that links usage of library resources to student demographic data and academic performance (the “library cube”). other cubes will be linked later.    results – while initial reports are rudimentary and do not yet incorporate data on e-resource usage, results are favourable in demonstrating the value of using the library information resources in coursework. based on the data generated to date, students who borrow library resources do outperform students who do not. early trend data shows up to a 12-point difference in grades.   conclusion – the library cube signals a new milestone in the uwl’s quality assessment journey. well-established measures of effectiveness and efficiency will be further complemented by measures of impact and value, allowing the library to step even closer to the goal of having effective and valued partnerships with the university community to realize teaching, learning, research, and internalization goals.   introduction   when the university of wollongong library (uwl) first commenced its quality assessment journey in 1994 there was a paucity of measures within the library and information sector to guide the evaluation of quality and effectiveness, to supplement the data demonstrating efficiency. performance indicators and measures primarily consisted of those mandated by government agencies or professional associations. the emphasis, typically, was on inputs and outputs. this situation is somewhat different now. a quality and service excellence program (qse), conceived in 1994, provided the catalyst to critically review and evaluate uwl’s capacity to deliver services of value to its clients and stakeholders. the qse encapsulated the improvement goals of the library; an emerging commitment to total quality management and a recognised need for an overall planning and management framework to replace the well-intentioned, but somewhat fragmented improvement efforts of the past.   to complement the qse program, uwl adopted the australian business excellence framework (abef) as a change management model (mcgregor, 2004). the abef provides descriptions of the essential features, characteristics, and approaches of organisational systems that promote sustainable and excellent performance, with emphasis on determining and evaluating customer needs, expectations and perceptions of excellent service. the ‘customer focus’ category of the abef encourages organisations to assess their ability to understand the needs and expectations of its customers, how customer relationships are managed, and customer perception of value. at uwl, the term client is used to describe the individuals seeking to and/or utilising services and resources.   early forays into assessment indicated that clients’ perceptions of library services were mostly favourable, however, success was difficult to measure and promote due to the lack of robust performance indicators and measures. to address this deficit, the collection, and interpretation of information and data was essential to facilitate and sustain the vision for transformational change. a performance indicator framework (pif), mapped to stakeholders’ needs and expectations was developed, providing a foundation for the systematic review of services and processes using quantitative and qualitative measures. through the reporting mechanisms embedded in the pif, it became possible to systematically measure and evaluate performance (i.e., how effectively and efficiently we manage and improve processes) and to assess clients’ satisfaction with services and resources. this represented a significant shift in the way that data and information was viewed and used; the emphasis was starting to change from inputs and outputs to measures of outcomes.   the introduction of a new element within the abef, customer perception of value, revealed an area addressed less rigorously by uwl; that is, how clients perceived the library’s competency in meeting their value goals or whether clients believed they received fair value for the ‘investment’ or cost of engaging with a service. while surveys and feedback systems provide data and information on a range of service elements, they are limited in their capacity to provide information and insight into the perceived value gained by engaging with the library (i.e., the return on the client’s effort for using services and resources).   measuring the value of using library resources   while the processes for evaluating expectations, performance, and satisfaction with available resources are robust and sustainable; measures of impact or affect are less well addressed. for uwl the critical impact question is: what is the value to the student when they use library information resources? this question cannot be answered adequately through satisfaction indices, or by de-identified usage rates of resources.   typically, information resources funds represent a significant proportion of the total allocation to libraries. in academic libraries, millions of dollars are committed annually to the acquisition of and subscription to information resources to meet the research, teaching and learning needs of their clientele. conversely, anecdotal evidence and local research (cooper, 2010) shows that many students bypass the library and almost exclusively use commercial browsers or resources (e.g., google, wikipedia) to fulfil their information needs.   the challenge for this library (and others) is to maintain visibility and relevance as a reputable interface for coursework and research resources in the context of an expanding information market. what is needed is a credible hook to show the value of engaging with library resources. we need to produce evidence that shows by using library resources students can improve academic performance that students who use the library get better grades. the approach chosen to measure the impact or value of library information resources differs from more traditional approaches to measuring return on investment (roi). roi can be defined as income received as a percent of the amount invested in an asset (luther, 2008). a positive roi indicates that more benefit than cost has been generated by the process/investment/result; a negative roi indicates less benefit was generated than the resource provided (white, 2007). the approach chosen at uwl has focused not purely on monetary return or loss. rather, we have sought a way to unambiguously demonstrate to students why using library resources is worth their time and effort (holt, 2007).   it turns out that there is a lot of useful information already being collected that can potentially speak to the value generated by the library. this information is managed by the library, and by other units on campus. internally, we have our library management system (lms). this system, like all lmss, contains a large amount of information about our clients, both borrowing and demographic data. there are also other systems on campus used to manage students’ university experience; systems that contain information collected before, during and after student enrolment. these systems include information managed by the recruitment arm of the university, information managed by campus administration, and information managed by the campus it department; and includes details on enrolment, academic performance, demographics, attrition, equity, alumni, and usage of the library’s resources. each of these information silos is useful to this assessment effort; collectively, they have allowed the library to make more informed decisions about the services and resources it provides, and the communication styles it has adopted. however, the real power of this information can only be unlocked by joining these data silos together. separated, these information silos tell a small and fragmented story about one facet of the student experience. together, the joined datasets tell a richer story (beckerle, 2008). without a joined dataset, for example, we can only know the demographic composition of the overall student population. however, if, for example, the student demographic data was joined to data on relating to usage of our resources, then we would be in a position to know both the demographic profile of library users, and be able to compare this profile to the demographic profile of non-library users.   the project we have embarked on involves joining as many datasets as is ethically, politically, and technically possible to join; with the aim of producing data that will allow the library to:   identify the value it contributes to the university; improve usage through targeted promotions; provide more outcome focussed key performance indicators (kpi); and drive deeper improvements.   the main requirement for joining any two datasets together is that that each must contain a common unique identifier. all of the systems mentioned above do contain a unique personal identifier, the student number. the political, ethical, and technical accessibility of the datasets varies from system to system. as an absolute minimum, we needed to be able to join information about the usage of our resources to student demographic and academic performance. anything less would not deliver a worthwhile return on effort. the joined datasets are encapsulated in a “cube,” (romero & abelló, 2009) and managed via business intelligence software.   the university performance indicator project team has built a cube for the library that links usage of library resources to student demographic data and academic performance (the “library cube”). other cubes that will be linked later in the year to the library cube include course and subject and student attrition. later plans include linking to the student satisfaction, equity, recruitment, and admission cubes. the library cube is currently still under development, and should be completed by the end of 2010.   converting data about usage of our resources into a usable form proved to be one of the more challenging aspects of the project. information about usage of our resources is held in two places. information about anything that is borrowed from our physical collection is held in the lms. unfortunately, the information contained in the lms is locked inside a “black box” that for the most part only allows access to aggregated data or individual records. we can, however, export a flat file containing a snapshot of all current clients and the books they have borrowed to date. this is not as much information as we need, but it is information we can use. we export this ‘snapshot’ each week, and the difference between two snapshots represents the amount borrowed by each client over the period between the snapshots.   like most libraries, demand for our physical collection is diminishing, while demand for our electronic resources is rising. consequently, the long-term success of the project hinges upon being able to access information about usage of our electronic resources. fortunately, this information is captured in logs as part of the authentication process. these logs do not contain all the information we need, but it does contain information we can use.   each time a user accesses any of our electronic resources a record is written to our ezproxy log. this log contains the student’s unique id, the electronic resource they accessed, and the time they accessed the resource. the number of log entries generated depends upon the content and code of the website that contains the resource the client is accessing. consequently, the number of log entries is arbitrary; so there is no value in counting the number of entries. however, we do know which database platform they used, and in many cases the actual database. so, in the spirit of pragmatism (i.e., take what you can use) we decided to convert the logs into meaningful data as follows:   the day is divided into 144 ten-minute periods; if a user accessed a database during a ten-minute period, then the name of that database is captured; any further accesses made to the same database during the ten-minute period are not recorded. (the user either accessed a given database during a ten minute period, or they did not.)   using these rules, we will be able to identify how many different electronic resources a user accessed during the day, and for how many ten-minute periods they accessed these databases. the number of ten-minute periods can be converted into a score (count) with a maximum score of 144 for a day for a given database. this method will provide a proxy measure for sessions—which despite its limitations should give a reasonably reliable and valid indication of the depth and scope usage.   aside from the technical challenges, there were also ethical, legal and political issues to resolve.   privacy   the primary ethical and legal consideration was privacy. the university of wollongong’s privacy information sheet outlines the 12 principles to which the university must comply regarding the collection, storage, access, use, and disclosure of personal information (”privacy information sheet,” 2010). fortunately, there are no legal barriers, as uow has obtained consent to use personal information for the project, via its privacy policy to which students must agree as part of their enrolment.   at an ethical level, the additional privacy risks potentially posed by the project have been eliminated by the way the personal information will be managed. privacy is only an issue to the extent that it involves the use, disclosure, etc. of personal information. information is only personal if it is possible to uniquely identify an individual from the information in question. the project will result in the construction of a cube built by joining several datasets, all of which will contain personal information. however, the library will not be able to use the cube to drill down to see a specific individual’s personal information. in other words, the data that the library can view in the cube will always be aggregated, which means we will not be able to identify a specific individual’s usage, except in the highly unlikely situation where a very small number of individuals belong to the variable contained within a dimension in the cube (e.g., hypothetically, if we only have five students from botswana, then it may be possible to identify those individuals from manipulating various aggregated views filtered to citizenship) (aggarwal & yu, 2008). in all cases, the personally identifiable data that could be gleaned from the cube is significantly less than that which can already be ethically and legally obtained by the library from its lms, usage logs, and access to student management systems. moreover, access to the cube will be even more restricted than is the case for the other systems that contain the same information.   executive support   the project involves doing something that is quite different for a library, and it requires the support of other units, and their executives. consequently, it is only healthy and expected that the project should encounter resistive inertia in some places. the library senior executive provided full and enthusiastic support for the project from the beginning. without this support, the project could not have succeeded.   the library has been very fortunate in the sense that the campus vice-principal (administration), has been and continues to be a major force behind improving performance measures at the university, notably through the creation of the performance indicators project team (pip). our goal to improve our ability to measure our performance sits very well with the vice-principal’s vision. the pip team’s vision is “to improve university performance through enhancing business decision-making by offering a seamless and secure architecture that provides business users with access to accurate, meaningful and shared data in a timely manner” (performance indicators project team, 2009). through carefully planned communication and demonstrated goal alignment, we were easily able to obtain the external senior executive support we needed for the project to succeed.   other libraries considering pursuing a similar project may not be as fortunate as we have been in obtaining support, and may benefit from reading lombardo and eichinger’s writings on political savvy and organisational agility (2009). from a practical point of view, anyone considering such a project should allow their library executive at least a month to absorb, understand, and commit to undertaking such a project; and allow at least six months to obtain support from all the necessary units. most importantly, undertaking such a project is only feasible if most of your student data is housed in online analytical processing (olap) cubes, or managed by other business intelligence software with similar functionality. our project could not have got off the ground without pip; they are the team that built the library cube.   there are three broad uses for which the library plans to use the information: to improve accountability; to support process improvement; and to support marketing.   accountability   uow makes a significant investment in its library. in 2009, the library had a budget of over $12m (aud), representing 4% of the campus budget (“library annual report,” 2009). the campus expects, and is entitled to know, the return it is obtaining from investing in the library. it is highly unlikely that the library will ever be able to provide a hard answer to this question, given that many of our activities generate real but largely unquantifiable value. for example, what value could be placed on rekindling an individual’s interest in learning? how much of that value can be attributed to the library? nevertheless, the project will allow us to provide better performance data than we have in the past.   we actually have seen a positive correlation between borrowing activity and academic performance for the data we have put into the cube so far. but we have not yet put in all the desired data elements (e.g. e-resources use) for that correlation to have much meaning. most importantly, the library understands and recognises that it cannot claim all the credit for increased academic performance. clearly, students would not perform nearly as well without the guidance, support, research, and teaching activities of academic staff. but it is also equally true that a student could fail their degree if they do not read anything. this point cannot be overemphasised. academic learning is about exploration and intellectual growth, and there are many paths to this destination. however, despite all the technological changes, the best way to grow academically is still by reading from and engaging with the body of knowledge generated by scholarly enquiry (levy & levy, 2005). students read from many places, and we hope to show that students are better off reading material from our collection.   the data we obtain from this project will allow us to demonstrate that those students that do not use our resources are at a disadvantage academically, and we will be able to quantify the degree of disadvantage. we will be able to quantify this disadvantage both in terms of lower academic performance and higher attrition rates.   process improvement   the library cube will provide the information we need to further support continuous improvement in three areas: collection development; academic relationships; and marketing.   the library spends a significant proportion of its budget subscribing to electronic databases. we are able to obtain information on the number of downloads associated with subscriptions; and we combine this with cost data to create rough indices, such as cost per download. the library uses this information, in consultation with academic staff, to continually improve and develop its collection. there are, however, two major limitations of this data: it is not linked to academic performance; and it takes far too long to get the data.   the library cube will be updated weekly, which will allow us to view in a much more timely fashion how our electronic resources are being used. we will also be able to see at the end of each session, which resources had a significant impact on academic performance, and which resources did not. we will be able to use this information to make more informed decisions about electronic resource collection development and to identify and replicate the processes that led to specific resources facilitating higher academic performance.   on this last point, we hope and expect that the cube will provide information that will support the library in taking a more holistic systems-based approach to improving the contributions the library makes to academic learning. for example, we will have enough information to be able to differentiate between those courses that have a higher proportion of library users, and those that do not. we will know which academics run those courses; so we will be in a position to begin to investigate what specifically some academics are doing differently that results in their students being more likely to use the library. this will allow us to identify what behaviours and practices support greater library usage, which in turn will provide the information we need to champion and support the rollout of best practices across the campus.   marketing   the library cube will also allow us to integrate marketing more closely with our core business activities, and to do so with surgical precision. for example, we will be able to provide academics with the evidence they need to effectively promote the library to their students. we will also be able to draw on this information in our own teaching activities, to convincingly demonstrate the research behaviours that led to academic success. we will know which specific group we should target to improve take-up. most importantly, we will know almost immediately whether our marketing efforts succeeded, which in turn will help us to make informed decisions about whether to change tack, or continue with more of the same.   conclusion   the ability to demonstrate the value of libraries and their collections is becoming all the more important and undeniably challenging in a period of generational change embodied in a fundamental shift in students’ attitudes to using information. not only do we need to convince the university executive and faculty of the value of libraries, our most challenging audience is increasingly that of the student body. we needed to garner evidence that would unequivocally demonstrate that academic performance can improve by using a library’s information resources.   to address this problem, a multidimensional approach to systems design was implemented, requiring not inconsiderable collaboration and cooperation between the library, university administration, pip, and information technology services (its). the project centred on the integration and interrogation of a series of discrete datasets (e.g., student performance, attrition, demographic, borrowing, and electronic resources usage data). although the time required to establish the problem statement, business rules, and reporting requirements has been lengthy, the genesis of the library cube is proving worthwhile. while initial reports are rudimentary, and do not yet incorporate data on e-resource usage (e.g., online journals), results are favourable in demonstrating the value of using library information resources in coursework. based on the data generated to date, students who borrow library resources do outperform students who do not. early trend data shows up to a 12-point difference in grades. such improved performance could influence a student’s decision to stay at university or leave; the overall quality of the learning experience; or the capacity to produce students who embody the university’s graduate qualities, notably that of being an independent learner who values scholarly information resources. importantly, the library cube will help to identify those students who use the library’s resources infrequently, or not at all. through this knowledge, highly tailored and tightly focused promotion and marketing strategies can be deployed, with immediate feedback on the effectiveness of chosen strategies.   the library cube signals a new milestone in the uwl’s quality assessment journey. well established measures of effectiveness and efficiency will be further complemented by measures of impact and value, allowing us to step even closer to the goal of having effective and valued partnerships with the university community to realise teaching, learning, research and internationalization goals.     references   aggarwal, c. c., & yu, p. s. (2008). privacy-preserving data mining: models and algorithms. new york: springer.   beckerle, m. (2008). how business intelligence systems deliver value: interview with mike beckerle of oco, inc. journal of digital asset management, 4(5), 277-291. doi:10.1057/dam.2008.22   cooper, lynda. (2010). online education program for transitioning students. incite, 31(6), 25. retrieved 20 may 2013 from http://archive.alia.org.au/incite/2010/v31.06.pdf   holt, glen. (2007). communicating the value of your libraries. the bottom line: managing library finances, 20(3), 119-124. doi:10.1108/08880450710825833   levy, p., & levy, s. (2005). developing the new learning environment: the changing role of the academic librarian. london: facet.   library annual report 2009. (2009). in university of wollongong. retrieved 20 may 2013 from http://www.library.uow.edu.au/content/groups/public/@web/@lib/documents/doc/uow077099.pdf   lombardo, m. m., & eichinger r. w. (2009). fyi: for your improvement, a guide for development and coaching. (5th ed). lominger international: a korn/ferry company.   luther, j. (2008). university investment in the library: what’s the return? a case study at the university of illinois at urbana-champaign. san diego: elsevier.   mcgregor, f. (2004). excellent libraries: a quality assurance perspective. in a. a. nitecki (ed.), advances in librarianship. (pp. 17-53). san diego: elsevier academic press.   romero, o. and abelló, a. (2009). a survey of multidimensional modelling methodologies. international journal of data warehousing and mining, 5(2), 1-23. doi:10.4018/jdwm.2009040101   privacy information sheet – general. (2010). in university of wollongong. retrieved 20 may 2013 from http://www.uow.edu.au/content/groups/public/@web/@fin/@lcu/documents/doc/uow061010.pdf    performance indicators project team. (2009). vision. unpublished document, university of wollongong, wollongong, new south wales, australia.   white, l. n. (2007). an old tool with potential new uses: return on investment. the bottom line: managing library finances, 20(1), 5-9. doi:10.1108/08880450710747407   microsoft word comm_cleyle.doc evidence based library and information practice 2006, 1:3  95 evidence based library and information practice     commentary    ebl is hot, help it sizzle!  su cleyle  associate university librarian  memorial university of newfoundland  st. john’s, newfoundland, canada  email: scleyle@mun.ca        © 2006 cleyle. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      ebl is hot in our profession.  its concepts  and theories are the subject of articles,  journals, and books in our literature.   workshops and conferences are embracing  the idea and “evidence” is quickly become  the new standard for our research processes.   with all this support why does ebl need  help?  while it seems to be accepted at a  theoretical level, ebl needs help on a more  practical level with implementation – and  that is where library managers and directors  come in.     library administrators at all levels need to  embrace the concept and become the driving  force for its acceptance into the daily  routines and practices of our professional  work.  they can accomplish this as they are  the ones who set the agendas.  it is the  directors and managers who approve  sabbaticals and research projects, and who  provide resources (human or otherwise) for  all library initiatives.  i predict that without  the practical support of library  administrators, the theoretical support for  ebl will start to dwindle.     making decisions based on the best possible  evidence or information at hand is what  administrators do naturally. however, those  decisions are not always empirically based.   decisions are often based on conversations  held around the water cooler or with other  administrators at meetings and conferences.   it is essential that decision makers become  more aware of what kind of information is  driving their decision‐making.  it would be  much better for our libraries and our  profession if those administrative decisions  were based on evidence based research.   a  library director making a case to a senior  administrator would be in a much better  position to argue for funding to provide  additional services if there was research to  back up the argument.  in the same way, a  librarian would be in a much better position  to serve students if he/she was aware of  student use patterns based on research.  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:3  96 linda watson noted in her presentation to  the 2nd international evidence based  librarianship conference in edmonton,  alberta that there are four main players in  ebl: lis programs, associations, librarians,  and library administrators. she outlines  five points for library administrators and  their role in ebl: recruit individuals with  innate curiosity; create an environment of  inquiry and expectation for reflective  practice, presentations, and publications;  provide resources for learning and  practicing research skills; seek out  opportunities within the institution for  research partnerships; and more than  anything, set an example! (watson 2003)    building on watson’s suggestions, here are  some practical ways administrators can  make ebl happen in their libraries:    1. encourage proposals for new services  that follow the steps of ebl.    2. offer institutional support for  research. this is easier in an academic  library where there is an expectation  that librarians (who are often  members of the local faculty  association) will conduct research.  encourage leave time for sabbaticals  or research leaves.    3. create financial support mechanisms  for research, such as a research fund  that will be available to researching  librarians on leave or not.     4. encourage question formulation in all  policy discussions using pico or  booth/brice’s spice (setting,  population, intervention, comparison  and evaluation) model. (booth and  brice 2003)     5. support ebl activities such as journal  clubs, research committees, research  registries, and brown bag sessions.   6. support collaborative initiatives for  research both within libraries and  among different libraries and faculty.    7. encourage the creation of repeatable  research that encourages a more  robust body of knowledge. remove  language from collective agreements  or agreements with librarians that  promotes only “unique or original  research.”     8. be willing to change. making  decisions based on the best possible  information can produce many  changes, and administrators need to  be prepared for and ready to work  with these changes.     professionals are excited about ebl because  they have found a way to bring research  into practice.  the nature of librarianship is  service.  with ebl librarians can enhance  their research practice in a way that also  enhances service to the user. no wonder the  profession is excited.   to ensure it becomes  a common part of our research and practice,  library directors and managers need to  embrace the idea and help with the  implementation.  it is the right thing to do  for both the staff they administer and users  they serve.       works cited    watson, lisa. “evidence‐based  librarianship: current players/future  aspirations.” 2nd international  evidence based librarianship  conference. lister hall, university of  alberta, edmonton, ab. 6 june 2003.  the university of sheffield. 3 august  2006  <http://www.shef.ac.uk/scharr/eblib/co nf2003.htm>.    http://www.shef.ac.uk/scharr/eblib/co evidence based library and information practice 2006, 1:3  97 booth, andrew and anne brice. “evidence  based librarianship: the first steps.”  2nd international evidence based  librarianship conference. lister hall,  university of alberta, edmonton, ab. 6  june 2003. the university of sheffield.  3 august 2006  <http://www.shef.ac.uk/scharr/eblib/co nf2003.htm>.     booth, andrew. “formulating answerable  questions.” evidence‐based practice for  information professionals: a handbook.  ed. andrew booth and anne brice.  london: facet publishing, 2004. 61‐70.                          http://www.shef.ac.uk/scharr/eblib/co news/announcements   early bird registrations now open for eblip8      2015. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   the organising and program committees of the 8th international evidence based library and information practice conference are pleased to announce that registrations are now open. eblip8 will be held july 6-8, 2015, in brisbane, australia.   keynote speakers include virginia wilson, director, centre for evidence based librarian & information practice, university of saskatchewan (canada); neil carrington, ceo, act for kids (australia), and kim tairi, university librarian, swinburne university of technology (australia). the draft program includes 39 presentations from local and international speakers. these presentations will provide an opportunity for delegates to hear about cutting edge research in the field as well as scholarly discussion about the future of evidence based practice within the profession. the program also includes a choice of seven interactive workshops of relevance to evidence based practice that will provide delegates with the chance to interact and engage with other members of the library and information community.   early bird registration closes may 1, 2015. for more information, visit the conference website at http://eblip8.info/.   article   a survey of graphic novel collection and use in american public libraries   edward francis schneider assistant professor university of south florida, school of information tampa, florida, united states of america email: efschneider@usf.edu   received: 27 dec. 2013   accepted: 16 july 2014      2014 schneider. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – the objective of this study was to survey american public libraries about their collection and use of graphic novels and compare their use to similar data collected about video games.   methods – public libraries were identified and contacted electronically for participation through an open us government database of public library systems. the libraries contacted were asked to participate voluntarily.   results – the results indicated that both graphic novels and video games have become a common part of library collections, and both media can have high levels of impact on circulation. results indicated that while almost all libraries surveyed had some graphic novels in their collections, those serving larger populations were much more likely to use graphic novels in patron outreach. similarly, video game collection was also more commonly found in libraries serving larger populations. results also showed that young readers were the primary users of graphic novels.   conclusion – responses provided a clear indicator that graphic novels are a near-ubiquitous part of public libraries today. the results on readership bolster the concept of graphic novels as a gateway to adult literacy. the results also highlight differences between larger and smaller libraries in terms of resource allocations towards new media. the patron demographics associated with comics show that library cooperation could be a potential marketing tool for comic book companies.   introduction   the inclusion of graphic novels and video games in libraries is not a new phenomenon. by 1934, issues of max gaines’ funnies on parade and famous funnies: a carnival of comics found their way onto library shelves across america, and by 1982, video games had done the same (holston, 2010, p.10; emmens, 1982). as the 80th anniversary of the publication of the first comic book nears, arguments for and against their presence in libraries have stayed fairly consistent throughout. discussion about comic books and video games being collected by libraries primarily revolves around the value for patrons, allocation of resources, and organization. to this day, patrons can find librarians on both sides of the divide when it comes to whether graphic novels and video games belong in collections.   comic books and graphic novels present interesting questions for librarians. most comic books are published in a format that resembles a periodical, but graphic novels more closely resemble novels. for example, the bound version of watchmen by moore, gibbons & higgins (1986) was named by time magazine as one of the 100 best english-language novels published since 1923 (grossman & lacayo, 2010), but watchmen was originally published as a series of monthly comics. besides the periodical/novel question, some popular cataloguing systems lump all comic books together based on format rather than publisher or creator. there is plenty of evidence that comic books are in libraries in various forms, from studies on academic library collection strength, to books advising how to collect and use comics in libraries. information on how libraries use and archive comics and graphic novels, as well as their impact on library patrons is less readily available.   this study investigated how american public libraries archive and use comic books and graphic novels, and asked librarians across the united states about the role of these materials in their systems. as a point of comparison, the professional librarians who participated in this survey were also asked similar questions about the use, archiving, and role, of video games. both comic books and video games are newer media forms, have unique cataloguing issues, and have some history of being frowned upon by parents and educators. the intent of this study was to: a) create a portrait of how comic books and graphic novels are being used by working librarians today; and b) compare the status of comic books and graphic novels within libraries to the status of video games, as a historical benchmark of new media form adoption.   literature review   one of the earliest and most consistent arguments concerning the collection of comic books and graphic novels is their legitimacy. nyberg (2009) notes that upon discovering comic books in the 1930s, many librarians launched “a crusade to turn young readers from the lurid ‘funny’ books to more wholesome fare”, and they were seen as a “problematic form of juvenile sub-literature” (p. 26-27). this was despite the fact that many early comic books, such as those by max gaines, were collections of strip comics that had previously run in daily american newspapers with little surrounding controversy.  it was these comics being printed for the first time in the format commonly thought of as a comic book that brought criticism. in his 1998 paper, a practicing comic book librarian surveys his collection and his craft, randall scott, a comic art specialist at michigan state university, indicates that even academic libraries that now emphasize comic book collection had collections influenced by anti-comics sentiment only a few decades prior. while there was some early opposition to libraries collecting comics, there is also some history of librarians defending comic book collection against concerned patrons. a survey of patron challenges to materials in massachusetts turned up documents from 1948 detailing a librarian’s defence of her library’s comic book collection against a catholic nun requesting the collection’s removal (musgrave, 2013).   many of the reasons presented to support arguments against the inclusion of comics within a library collection seemed reasonable at the time. firstly, the format of combining illustrations with written words was associated too closely with children’s picture books, leading many to believe that readers of comics were not really reading. secondly, many felt that the violence and sexuality depicted in some comics would threaten the morals and decency of young readers. in the early 1950s, psychiatrists such as dr. fredric wertham joined the debate, arguing that “reading such fare did psychological damage to children” (nyberg, 2009, p. 27). wertham’s work was used as the basis for the comics code authority, an industry-imposed set of comic decency standards (nyberg, 2009). the third argument against legitimacy has been that the “genre” of comics and graphic novels does not hold up as true forms of literature, particularly when compared to the classics. to this day, “because the words ‘comic’ or ‘graphic novels’ still have the stigma of being hack literature many librarians consider this type of reading material to be inappropriate for the library and resist its acceptance” (sheppard, 2007, p.12).   similar statements have been made regarding the appropriateness of including video games in library collections. us surgeon general c. everett koop (1982) expressed serious concerns about the impact of video games on children’s mental and physical health. comic books have also been treated as a public health menace; they were the subject of a 1948 symposium at the new york academy of medicine titled, “the psychopathology of comic books” (thrasher, 1949). more current books such as stop teaching our kids to kill: a call to action against tv, movie, & video game violence (grossman & degaetano, 2007) represent common concerns about video games causing delinquent behaviour. the concerns about video games often resemble the concerns about comic books in the 1950s. for example, there are many parallels between grossman and degaetano’s anti-video game book and dr. frederic wertham’s famous anti-comics book seduction of the innocent: the influence of comic books on today’s youth (1954).  both feature lurid chapter titles, and both present context-free examples that attempt link increases in real world violence to the emergence of the new media.   in the 2011 supreme court case, brown v. ema (2011), the state of california claimed that “[v]ideo games present special problems because they are ‘interactive’, in that the player participates in the violent action onscreen and determines outcome” (p. 9). judge posner states in the courts opinion:   “all literature is interactive. the better it is, the more interactive. literature when it is successful draws the reader into the story, makes him identify with the characters, invites him to judge them and quarrel with them, or experience their joys and sufferings as the readers own.” (brown v ema, 2011, p. 11)   justice scalia pointed out that this is not the first time this kind of argument has been made, noting, “…past confusion and alarm about possible harm to minors by penny dreadfuls (lurid novels), movies, and comic books” (brown v. ema, 2011, p. 11).   in the last few decades, artists and authors of graphic novels have begun tackling more serious concepts “…such as homosexuality, racism, and aids” (sheppard, 2007, p. 13). prime examples are art spielgman’s maus: a survivors tale, craig thompson’s blankets, and marjane satropis’s persepolis. reading comics is a complex task of decoding information and making “…relevant social, linguistic, and cultural conventions” (tilley, 2008, p. 23). research has shown that graphic novels not only help bridge the gap for reluctant readers, but are “linguistically equal to other works of literature” (sheppard, 2007, p. 13). beyond comprehension of the text at hand, graphic novels can help readers “…decode mood, tone... facial and body expressions, the symbolic meanings of certain images and postures, metaphors and similes, and other social and literary nuances” (sheppard, 2007, p. 13). video games, like graphic novels, range in depth and sophistication. games such as september 12th and beyond good and evil are discussed in buchanan and vanden elzen’s (2012) “beyond a fad: why video games should be part of 21st century libraries”. these games allow users to explore, through a combination of story, visuals, and interactions, relevant, provocative topics such as “…patriotism, paranoia, and the role of dissent in a democracy” (buchanan & elzen, 2012, p. 18). some games such as alice: the madness returns and batman: arkham asylum are based on books and graphic novels, while some games inspire fiction of their own, such as resident evil and halo. in a recent article, levine (2009) cites kutner and olson, stating not only that direct links between real world violence and video games have been debunked repeatedly, but that “…games actually help children learn valuable skills such as collaboration, problem-solving, teamwork, and coping with negative emotions” (p. 34).   as video games become more prominent, some librarians question what role they have in the library and how the library’s duties to the public are affected. in a chapter written for the 2008 journal of access services, the author dismissed library efforts that included video games and likened them to prostitution of the library (annoyed librarian, 2008, p. 620-621). the author considers adding video games to collections as a ploy to increase circulation numbers. but in that same year, the president of the american library association spoke positively of gaming as an social and educational good, and the conclusion of a 2009 american library association journal wrote that “…gaming is just one more way libraries can continue to offer a mix of recreation, social, and communal activities in a safe, non-commercial space” (levine, 2009, p. 35).   for libraries that include graphic novels and video games in their collections, a number of issues arise, including cost, as well as where and how materials should be shelved and catalogued. widespread economic issues have impacted libraries, which have led to concerns about collection development funds being used for video games and graphic novels. macdonald (2013) cites research done by zabriskie in her recent article, “how graphic novels became the hottest section in the library”, regarding the cost per circulation when compared with other popular titles including harry potter, twilight, and ged guides. zabriskie found that graphic novels had higher circulation rates but were also 10 cents less per circulation, making them considerably more cost-efficient (macdonald, 2013, p. 22). online services, like comics plus: library edition and comixology, allow libraries to utilize an even more cost-effective model for serving materials to patrons (macdonald, 2013, p. 25).   these services can also assist with the other major concern regarding archiving of these materials, finding space in already crowded library stacks. hartman (2010) describes the challenge with comic books as an intersection of limited resources, limited shelving space and growing patron demand. the library of congress and dewey systems do little to help librarians in this matter, clumping all of these materials into the same call numbers regardless of topic. for graphic novels, dewey uses 741.5 while library of congress splits them between ps and nc. video games find themselves clumped either in gv1469 for library of congress or in the 794s for dewey. “steve raiteri noted that graphic novels should be treated like any other material, taking into account content and intended audience” (nyberg, 2009, p. 37). one of the first things that becomes a problem for graphic novels is the assumption that it is a genre rather than a format. once that is sorted out librarians must determine whether they should be integrated with the topics they cover or grouped together. hartman’s opinion and experience at the toledo-lucas county public library in ohio is that patrons prefer not to search through the catalogue and then the general collection to find what they are looking for. they prefer to browse the graphic novel section to see what was new and would interest them (hartman, 2010, p. 59). additionally, librarians must determine if they separate collections by age ranges. collections are commonly split between juvenile, young adult, and adult. these can be good for general guidelines but many items blur the line between young adult and adult content. for the cleveland public library, materials are not only in the central library but also in literature and young adult sections. while this may increase visibility it may also mean that users have to find their way to multiple areas of the library to find what they want (pyles, 2012, p. 34). secondly, with the mixture of series and single volumes, individual titles may get lost in the shelves. to resolve this, some libraries choose to separate series and single volume items. this allows for individual titles to not become lost between larger series that sometimes have thirty or more volumes (segraves, 2010, p. 70).   many librarians, even those who did not grow up with graphic novels and video games, are “willing to concede legitimacy” due to explosive circulation stats and maturing content (hartman, 2010, p. 59). as these materials become more prevalent, librarians should focus on improving formal classifications systems that allow for better access and organization, programming, and collection development policies that allow librarians to keep collections relevant and enticing to patrons.   a librarian looking to begin or expand their library’s collection in graphic novels has a wide range of potential resources. information can be found in books like serchay’s the librarian’s guide to graphic novels for adults (2010) and graphic novels now: building managing, and marketing a dynamic collection (goldsmith, 2005). there are also a host of articles for librarians who serve different audiences. for example, school librarians can access the articles, “graphic novels and school libraries” (rudiger & schliesman, 2007) and “graphic novels in libraries: supporting teacher education and librarianship programs” (williams & peterson, 2009) for reference. collections specialists could read the article, “comic books and graphic novels for libraries: what to buy” (lavin, 1998) and academic librarians could begin with “graphic novels in academic libraries: from maus to manga and beyond” (o’english, matthews, & lindsay, 2006).   although there is a considerable amount of literature for librarians looking for practical guidance in collecting comic books, there is not as much information on the overall adoption rate of comic books by public libraries. the most recent account on the adoption of comics in libraries comes from a survey of librarians done by the american library association’s office of intellectual freedom in 2005. however, it is not a detailed account or analysis of comics and graphic novels in libraries. the results were disclosed in a press release titled, graphic novels: suggestions for librarians (2006). of 185 public library employees surveyed, 97% indicated comics or graphic novels were in their libraries. however, no information on the author or methodology behind the study was included. only the number of participants and three resulting statistics were included. efforts to find more details behind the study were unsuccessful.   other researchers have assessed library collection of comic books by measuring the adoption rate of historically important titles. in a study surveying the holdings of all academic members of the association of research libraries (werthmann, 2010), it was discovered that a wide discrepancy exists regarding the amount of graphic novels collected. werthmann based his investigation on a list of 77 comic books that are highly regarded by critics, and then investigated which libraries had them as part of their collections. he discovered that most libraries had some graphic novels, but the amount collected generally correlated with the size of their collection, and some of the libraries surveyed had none of the titles on the list.   research goals & objectives:   comic books and video games represent two different generations of new media forms entering libraries. the primary goal of this research study was to assess the collection, use, and impact of comic books and graphic novels in libraries. the secondary goal was to study the role of video games in libraries as a point of comparison for the role of comics in the library, as both media forms have a similar media history. the central objectives of this research were:   to assess the frequency with which american public libraries keep graphic novels as part of their collection to assess the frequency with which american public libraries keep console video games as part of their collection to assess the demographics of patrons who utilize graphic novel and video game collections to assess the impact of graphic novel collections on youth engagement and library promotion to assess the shelving decisions libraries make regarding graphic novels to assess the relationship between graphic novel and console video game collections   methodology   the first step was the development of a survey instrument based on research goals. input from public library employees was used to frame the questions in the working context of a public library system. the survey instrument was hosted publicly on google drive, and was tested by a small number of pre-service librarians in a graduate mlis program. the responses to the survey were automatically collected into a database also hosted on google drive.   once the survey was constructed and tested, a list of public libraries was obtained. a list of over 9,000 american public libraries and library systems was retrieved from the us government data clearinghouse (institute of library and museum services, 2011).  four hundred and fifty libraries were selected at random. the government database contains the web address of each library, and the individual websites of the selected libraries were used to identify the collections specialist for each library. the request to participate in the study was then sent by email to the collections specialist. if no collections specialist or person with a similar role was identifiable, the request was sent to the e-mail listed for general inquiries. somewhat similar research has used freedom of information act (foia) requests to force libraries into participating in a study on patron challenges (musgrave, 2013). in this case it was decided not to use the leverage of foia requests to ensure a completely voluntary participation.   the requests to participate included an explanation of the purposes of the research and a link to the survey instrument. appeals for participation were also made over public librarian-related social media. to ensure the participants were from public libraries and had roles within libraries that gave them enough perspective to answer our survey, participants were asked for their position title and what kind of library they serve. survey participants were also told that they could receive a copy of the final outcome of the survey if they included their e-mail address, and 85.8% of participants gave their address. the targeted nature of the appeals for participation and the specific nature of the professional questions were intended to help to ensure an appropriate sample. results from participants who did not indicate they worked in a public library or did not provide any professional details were removed. the survey was kept online for two months from the time of the last request for participation.   results   at the end of the surveying period, the final data included responses from 106 participants. participants represented 31 states, and had an average library service population of 121,792 users. participants included libraries that serve populations as small as 600 users, and as large as 3 million users; the median size was 16,000 users. of the 106 participants, 104 indicated they had comics or graphic novels in their collections and only two indicated they did not. these results closely mirrored the results of the ala/comic book legal defense fund study from seven years earlier (national coalition against censorship, 2006). this survey showed 98.1% of respondents had comic books, the earlier ala/comic book legal defense fund study (2006) found 96.7% did; statistically, this is not considered an increase.   the most apparent factor in motivating libraries to add graphic novels was user demand. participants were asked in a free-writing format why their library began adding graphic novels to the collection, and 66.9% of libraries with comics indicated patron demand as the primary motivating factor. twelve participants (11.6%) indicated that collections were started by library staff with an interest in comics, and nine participants (8.7%) stated that their libraries purchased comics in an attempt to encourage patronage.    participants were presented with age brackets of patrons, and asked which groups most commonly use comics and graphic novels. as seen in figure 1, the most commonly indicated user for library comics and graphic novels was between the ages of 12 and 15 (making up 23% of graphic novel circulation), closely followed by users aged 10 to 12 (16%) and 15 to 18 (15%). overall, 52% of graphic novel circulation could be attributed to people 18 years of age and younger.   figure 1 percentage of graphic novel collection circulation by patron age group.     the results indicated that 29 out of 106 (27.4%) libraries participating had clubs or events relating to comics. out of the 29 libraries with comic-themed clubs or events, 6 libraries had both clubs and events, 9 libraries had a comic book/manga related club but no events, and 14 only had events related to comics. this activity seemed to favor larger libraries, as the average constituent population size of the libraries that had comic book activities (195,811) was more than double that of the libraries that did not host comics-related activities (95,793).   participants were also asked questions about collecting console video games within their respective libraries. while not as prevalent as comic books and graphic novels, this study found a number of public libraries stocking console video games. forty-eight out of 106 respondents (45.3%) indicated that they had console games in their library’s collections. games for nintendo’s wii system were the most common, but games for nintendo ds, microsoft xbox and other systems were found. collecting console video games seemed to be more common in larger libraries, as the average service population size of the libraries with games was over three times larger (186,112) than the libraries with no games (53,521). interestingly, only a low positive correlation was found between collecting video games and having comic related clubs and events (pearson’s r = 0.164). participants were asked to rate the popularity of their graphic novel and video game holdings with users, on a scale of one to five. comic books were indicated as being more popular (μ = 3.81) than video games (μ = 3.31).   as mentioned, there has been discussion amongst librarians around the world about the classification of graphic novels and comic books (masuda, 1989). survey participants were asked where the comic books and graphic novels were shelved in their library.   25.2% put them in the young adult or children’s section 14.5% placed all comics/graphic novels in a separate section 60.1% had them in multiple areas of the library   while the results showed that only 14.5% of public libraries put all their graphic novels and comic books into a separate section, a portion of the librarians who indicated they kept these materials in multiple sections also indicated that they had a featured area for these materials. overall, 23.3% of responding libraries indicated that there was a part of their shelving that included an area exclusively dedicated to comic books and graphic novels.   pro-comics and graphic novel material for librarians often contains advice for working with comic books as a potentially problematic material. of the 106 respondents, 6 indicated that they had policies for restricting materials based on age, in response to parental request, in their libraries, and none of these policies had anything particularly to do with comics or graphic novels.   summarizing the results in the context of the research goals and objectives provides a portrait of activity that could potentially be useful for both librarians and researchers. the results:   confirmed that overwhelmingly, american public libraries keep graphic novels as part of their collection. revealed that less than approximately 40% of american public libraries keep console video games as part of their collection. revealed that librarians indicated that both comic books/graphic novels and video games are highly circulated items, but comic book are more popular than video games. indicated that people under the age of 18 are the primary users of comic books from the library. show that there are a range of shelving options being used by libraries when it comes to comic books and graphic novels. the majority of libraries have them in multiple sections, and almost one in four has some section of their library specifically dedicated to displaying comic books.  found that libraries serving larger communities are more likely to collect video games and have comic book based clubs. however, there is little correlation between collecting video games and hosting comic book events.   discussion   the first comic book debuted in 1934 (nyberg, 2009), and the first commercial video game, in 1971 (emmens, 1982). thus, at the time of this writing, comic books are approximately 80 years old, and video games are just over 40 years old. the results of this study show that comic books and graphic novels have become a part of library culture, and video games are on their way to doing the same.   future research could involve performing database searches to learn more about the small percentage of libraries that do not collect comic books or graphic novels. werthmann (2010) found that, although rare, there recently were academic research institutions with virtually no comic books in their libraries. there is the possibility of more public libraries with no comic books or graphic novels than identified in this study. libraries with limited resources may have been less inclined to participate in our survey, although our results are in line with previous ala sponsored research.   a potential next step would be to look at the details of collection. for example, comic books can be subscribed to as periodicals or collected as bound volumes. it is not known if libraries collect periodical comics, and if these are handled differently. comic books from japan, known as manga, are popular in the comic book genre. it would be interesting to know what percentage of libraries’ collections nationwide are dedicated to japanese comic books, as a measure of globalization of culture. this study found that libraries collecting console video games was fairly common, but we did not ask if the libraries also carried and checked out the systems needed to play these games. now that library collection of comic books and video games has been established, more details on the decisions surrounding that collection are warranted. this study relied on voluntary participation; it only asked library staff to give broad numerical estimates on the ages of people who check out comic books and their popularity in circulation. asking a staff member what percentage of their comics are japanese in origin or what video game console is checked out the most would be better served by a separate study where libraries’ collection databases are accessed directly.   it is currently a tough climate for many public libraries; over 40% of states handed budget cuts to public library systems in the last three years, and 45% of public libraries state that they still don’t have sufficient internet speeds (ala, 2012). on the surface, it might seem somewhat intuitive that larger libraries would be more likely to collect video games due to larger operating budgets. however, in reality, the retail price of video games tends to be between $20 usd and $60 usd, meaning that they are not tremendously more expensive than books. anecdotally, the two libraries that stated they did not have comics and graphic novels in their collections each indicated an average service population size of under 5,000 users. the total service population of the combined libraries of survey respondents was over 11.9 million.  while the results showed that over 98% of library systems surveyed had comics, the percentage of the american population that has a local library with comic books or graphic novels could certainly be higher.   the results could potentially be interesting to literacy specialists, as more than half of comic book and graphic novel checkouts are to people under 18 years of age, and these are generally considered difficult years for male students and reading (young & brozo, 2001). the results are also potentially interesting on a commerce level, as the survey results showed that the demographics of people who check out comics from the library are substantially different from recent research on the demographics of the people who purchase comics. dc comics recently paired with nielsen to do a demographic analysis of comic book readers in coordination with the launch of their comic book line titled the new 52 (dce editorial, 2011). the results showed that less than 2% of their sales were to people under 18 years of age. in the same study, dc found that half of their readers were 35 years of age and older, and that their most active demographic is aged 25-35 years. comic book companies should understand that these results indicate that comic books in the library are not a threat to sales from comic book stores. further research could specifically investigate if there is a link between library behaviours and comic purchasing behaviours.   conclusion   the clear takeaway from this study is that comic books have become an ingrained part of public library holdings in the united states. while it is not impossible to find a library with no comic books or graphic novels, this is now the second study to show that upwards of 97% of public libraries have them in their collections. video games, being approximately half the age of comic books, are still only found in less than half of libraries, showing that they are far from being fully embraced by libraries as a whole. the results also show that larger libraries are using graphic novels and comics for community engagement with a far greater frequency than smaller libraries. the collection discrepancy is even wider in the adoption of video games.   a public librarian considering resource allocation can be assured that collecting comics and graphic novels is a widespread library phenomenon, and that their collection can promote young adult circulation. publishers of comic books can also use the results of this study to understand that libraries collecting comics is a potential pathway to comic book sales later in life. while the differences between large libraries and small libraries in the collection and use of graphic novels and video games might seem troubling on the surface, overall trends show an american public library system that is adapting to changes in patron demand and changes in the overall media landscape.     references
   american libraries association. (2012). state of america’s libraries report. retrieved from http://www.ala.org/news/mediapresscenter/americaslibraries/soal2012/public-libraries   annoyed librarian. (2008). public librarians: why do they bother? journal of access services, 5(4), 611-621.   brown v. ema, 564 u.s. (2011). slip opinion. retrieved from http://www.supremecourt.gov/ opinions/10pdf/08-1448.pdf   buchanan, k. & vanden elzen, a. (2012). beyond a fad: why video games should be part of 21st century libraries. education libraries, 35, 15-33.   institute of library and museum services (2011) public library survey (pls) 2011. retrieved from http://catalog.data.gov/dataset/public-library-survey-pls-2011   dce editorial. (2012) dc comics the new 52 product launch research results. retrieved from http://www.dccomics.com/blog/2012/02/09/dc-comics-the-new-52-product-launch-research-results   elzen, a. (2012). beyond a fad: why video games should be part of 21st century libraries. education libraries, 35, 15-33.   emmens, c. (1982). the circulation of video games. school library journal, 29(3), 45.   goldsmith, f. (2005). graphic novels now: building, managing, and marketing a dynamic collection. ala editions. chicago, il.   grossman, l. c. d., & degaetano, g. (2009). stop teaching our kids to kill: a call to action against tv, movie & video game violence. random house llc. new york.   grossman, l., & lacayo, r. (2010). all-time 100 novels. originally published january 11, 2010. retrieved from http://entertainment.time.com/2005/10/16/all-time-100-novels/slide/watchmen-1986-by-alan-moore-dave-gibbons/   hartman, a. (2010). creative shelving: placement in library collections. in r. g. weiner (ed.), graphic novels and comics in libraries and archives: essays on readers, research, history and cataloging (pp. 63-67). jefferson, nc: mcfarland & company.   holston, a. (2010). a librarian’s guide to the history of graphic novels. in r. g. weiner (ed.), graphic novels and comics in libraries and archives: essays on readers, research, history and cataloging (pp. 9-16). jefferson, nc: mcfarland & company.   koop, c. e. (1982, november 9). surgeon general sees danger in video games. new york times, p. a16.   lavin, m. r. (1998). comic books and graphic novels for libraries: what to buy. serials review, 24(2), 31-45.   levine, j. (2009). gaming, all grown up. in american libraries. august-september, pp. 34-35.   macdonald, h. (2013, may 3). how graphic novels became the hottest section in the library. publishers weekly, 20-25.   masuda, c. (1989). gakko toshokan to manga [school libraries and manga]. gakko toshokan, 468, 9-18.   moore, a., gibbons, d., & higgins, j. (2005). watchmen. dc comics. new york.   musgrave, s. (2013). librarian rebukes nun over comic books. muckrock, retrieved from https://www.muckrock.com/news/archives/2013/oct/25/librarian-rebukes-nun-over-comic-books/   national coalition against censorship, american library association and comic book legal defense fund (2006), “graphic novels: suggestions for librarians”, available at: http://www.ncac.org/graphicnovels (accessed january 31, 2007).   nyberg, a. (2009) how librarians learned to love the graphic novel. in r. g.   weiner (ed.), graphic novels and comics in libraries and archives: essays on readers, research, history and cataloging (pp. 26-40). jefferson, nc: mcfarland & company.   o'english, l., matthews, j. g., & lindsay, e. b. (2006). graphic novels in academic libraries: from maus to manga and beyond. the journal of academic librarianship, 32(2), 173-182.   pyles, c. (2012). it’s no joke: comics and collection development. public libraries, 51(6), 32-35.   rudiger, h. m., & schliesman, m. (2007). graphic novels and school libraries. knowledge quest, 36(2), 57-59.   satrapi, m. (2007). the complete persepolis. new york: pantheon books.   scott, r. w. (1998). a practicing comic-book librarian surveys his collection and his craft. serials review, 24(1), 49-56.   segraves, e. (2010). teen-led revamp. in r. weiner (ed.), graphic novels and comics in libraries and archives: essays on readers, research, history and cataloging (pp. 68-71). jefferson, nc: mcfarland & company.   serchay, d. s. (2010). the librarian's guide to graphic novels for adults. neal-schuman publishers. new york.   sheppard, a. (2007). graphic novels in the library. arkansas libraries, 64(3), 12-16.   spiegelman, a. (1995). maus: a survivor's tale. proteus, 12(2), 1-2.   thrasher, f. m. (1949). the comics and delinquency: cause or scapegoat. the journal of educational sociology, vol. 23, no. 4. 195-205.   thompson, c. (2004). blankets: an illustrated novel. top shelf.   tilley, c. l. (may 2008). reading comics. school library media activities monthly, 24(9), 23– 26.   wertham, f. (1954). seduction of the innocent: the influence of comic books on today's youth. new york: rinehart & company.   werthmann, e. j. (2010). graphic novel holdings in academic libraries: an analysis of the collections of association of research libraries members. in r. g. weiner (ed.), graphic novels and comics in libraries and archives: essays on readers, research, history and cataloging (pp. 242-59). jefferson, nc: mcfarland & company.   williams, v. k., & peterson, d. v. (2009). graphic novels in libraries: supporting teacher education and librarianship programs. library resources & technical services, 53(3), 166-173.   young, j. p., & brozo, w. g. (2001). boys will be boys, or will they? literacy and masculinities. reading research quarterly, 36(3), 316-325.   evidence summary   digital libraries that demonstrate high levels of mutual complementarity in collection-level metadata give a richer representation of their content and improve subject access for users   a review of: zavalina, o. l. (2013). complementarity in subject metadata in large-scale digital libraries: a comparative analysis. cataloging & classification quarterly, 52(1), 77-89. http://dx.doi.org/10.1080/01639374.2013.848316   reviewed by: aoife lawton systems librarian health service executive regional library & information service, dr. steevens’ hospital dublin, ireland email: aoife.lawton@hse.ie   received: 29 jul. 2014    accepted: 14 oct. 2014      2014 lawton. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine how well digital library content is represented through free-text and subject headings. specifically to examine whether a combination of free-text description data and controlled vocabulary is more comprehensive than free-text description data alone in describing digital collections.   design – qualitative content analysis and complementarity comparison.   setting – three large scale cultural heritage digital libraries: one in europe and two in the united states of america.   methods – the researcher retrieved xml files of complete metadata records for two of the digital libraries, while the third library openly exposed its full metadata. the systematic samples obtained for all three libraries enabled qualitative content analysis to uncover how metadata values relate to each other at the collection level. the researcher retrieved 99 collection-level metadata records in total for analysis. the breakdown was 39, 33, and 27 records per digital library. when comparing metadata in the free-text description metadata element with data in four controlled vocabulary elements, subject, geographic coverage, temporal coverage and object type, the researcher observed three types of complementarity: one-way, two-way and multiple-complementarity. the author refers to complementarity as “describing a collection’s subject matter with mutually complementary data values in controlled vocabulary and free-text subject metadata elements” (zavalina, 2013, p. 77). for example, within a temporal coverage metadata element the term “19th century” would complement a description metadata element “1850–1899” in the same record.   main results – the researcher found a high level of one-way complementarity in the metadata of all three digital libraries. this was mostly demonstrated by free-text data in the description element complemented by data in the controlled vocabulary elements of subject, geographic coverage, temporal coverage, and object type. only one library demonstrated a significant proportion (19%) of redundancy between free-text and controlled vocabulary metadata. an example of redundancy found included a repetition of geographic information in both a description and geographic coverage metadata elements.   conclusion – the author reports high levels of mutual complementarity in the three cultural heritage digital libraries studied. the findings demonstrate that collection-level metadata which includes both free-text and controlled vocabulary is more representative of the intellectual content of the collections and improves subject access for users. the author maintains that there is no standard for collection-level metadata descriptions, and that this research may contribute to best practice guidelines in this area. it is unclear whether the digital libraries studied had written policies in place on how to describe collections and if those policies were adhered to in practice. the author expresses a need for further research to be conducted on collection-level metadata in other domains, such as science and interdisciplinary digital libraries, and on other scales (e.g., regional or state collections) and geographic regions beyond europe and the united states.     commentary   although there is an abundance of general research and literature on metadata, digital libraries, interoperability, and standards, there is little specific research on complementarity of collection-level metadata. this research highlights this gap and adds value to the current body of research on this topic.   the methods of data collection and presentation of results score high on glynn’s critical appraisal checklist (glynn, 2006). this work builds on a previous study by the same author, which compared the free-text description metadata field in multiple digital libraries (zavalina, 2011). the current work extends the analysis to include three digital libraries of similar content and size and the comparative analysis of free-text description element with four controlled vocabulary subject metadata fields. the sample size of metadata records from each library is similar, making the analysis sufficiently precise. consent was obtained from developers of two of the digital libraries who agreed to provide the records for the content analysis. there is a detailed description of the level of comparative analysis carried out. however, there is no description of any tools that may have been used to compare the metadata records and retrieved digital content samples. a more detailed description of the methodology used would have been useful.   the results are well presented with five figures that visually demonstrate the level of complementarity between metadata elements. the extensive descriptive detail provided by the author enables further replication of this study on other digital library collections. the author thanks individuals at two universities for feedback on her study; however it is unclear whether the method of the content analysis used has been externally validated beyond this feedback. the author refers to the importance of complementarity of metadata in digital collections as an enabler of subject based access to collections. this study would be strengthened with the inclusion of evidence about how users of cultural heritage collections browse or search collections by subject.   based on the evidence presented in this paper, digital library managers, repository managers and cataloguers should consider the collection-level metadata that they are currently implementing and whether this is something that needs attention. specifically, the potential for levels of redundancy observed in this study is an important finding. cataloguers could save time by avoiding entering duplicate data, such as identical geographic information in both description and geographic coverage elements. as the growth of digital library collections is set to continue, adherence to standards of metadata descriptors and best use of descriptive content including free-text and controlled vocabulary elements will become ever more important to discoverability. the author signposts two guidelines of relevance to cataloguers: the framework of guidance for building good digital collections established by niso, and the guidelines for digital libraries being prepared by the international federation of library associations and institutions in conjunction with the world digital library project.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154   zavalina, o. l. (2011). free-text collection-level subject metadata in large-scale digital libraries: a comparative content analysis. in t. baker, d. i. hillman, & a. issac, (eds.), metadata harmonization: bridging languages of description. proceedings of the dcmi international conference on dublin core and metadata applications (pp. 147.157). dcmi. retrieved from http://dcpapers.dublincore.org/pubs/index evidence based library and information practice evidence based library and information practice 2011, 6.1 64 evidence based library and information practice evidence summary flexible scheduling may have a positive impact on school library circulation a review of: gavigan, k., pribesh, s., & dickinson, g. (2010). fixed or flexible schedule? schedule impacts and school library circulation. library and information science research, 32(2), 131-37. reviewed by: ann medaille reference & instruction librarian university of nevada, reno reno, nevada, united states of america email: amedaille@unr.edu received: 14 sept. 2010 accepted: 7 dec. 2010 2011 medaille. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to determine whether a significant, positive relationship exists between flexible scheduling in elementary school library media centers and per-pupil circulation statistics. design – online survey. setting – library media centers in public elementary schools in two states of the united states. subjects – a total of 88 elementary school library media specialists completed the survey. methods – a 22-question online survey was created using inquisite software. a link to the survey was sent via e-mail to a 600-person random sample of public school library media specialists whose names were drawn from the memberships of the north carolina school library media association and the virginia educational media association, with combined memberships totalling approximately 2,000. a random sample of 600 was chosen to provide a 95% confidence level with a confidence interval of plus or minus three points. the survey included questions about school schedules (flexible, partially flexible, or fixed), collection size, circulation statistics, total student enrolment, school type (elementary, middle, or high), school location (urban, rural, or suburban), percentage of students eligible for free and reduced price lunch, numbers of full-time professional library media specialists employed at the school, access procedures, and library closure mailto:amedaille@unr.edu� evidence based library and information practice 2011, 6.1 65 information. the survey response rate was 29.3% (176 respondents). because fixed versus flexible scheduling is an issue that primarily affects elementary schools, the authors further narrowed their subject pool to 88 elementary school respondents. data were analyzed using spss 16.0 statistical software. correlation analysis, including the analysis of covariance (ancova), was used to compare differences in the three scheduling types and to control for other variables (such as location, socioeconomic status, collection size, staffing, days closed, and others) that may affect circulation. main results – of the 88 elementary school responses, 33 (38%) had fixed schedules, 44 (50%) had partially flexible schedules, and 11 (13%) had totally flexible schedules. fiftythree schools supplied data regarding perpupil check-out, and the average number of books checked out per student per year was 52, across all types of schools. the number of book check-outs per pupil differed according to schedule type. students in schools with fixed schedules checked out an average of 51 books per year, those in schools with partially flexible schedules checked out 46 books per year, and those in schools with totally flexible schedules checked out 71 books per year. when the authors adjusted the data for other factors affecting circulation (such as location, socioeconomic status, collection size, staffing, and others), they found that students in schools with fixed schedules checked out 57 books per year, those in schools with partially flexible schedules checked out 68 books per year, and those in schools with totally flexible schedules checked out 102 books per year. the authors concluded that schedule accounts for 21% of variation in the rate of book check-out per pupil. conclusion – these results suggest that the type of schedule used in elementary school library media centers does have a significant relation to circulation statistics. specifically, when library media centres employ flexible scheduling, students are likely to check out more books per year on average. although these results are not generalizable, this study provides much-needed research into the relationship between scheduling and circulation, and establishes a basis for further studies in this area. commentary the debate about whether flexible scheduling is superior to fixed scheduling has been going on for decades among school library media specialists (creighton, 2007). fixed scheduling occurs when classes are regularly scheduled in the library media center each week, while flexible scheduling means that access to the library is available throughout the day, and classes meet in the library on the basis of curricular need rather than a predetermined schedule (hurley, 2004; creighton, 2007). with fixed scheduling, classes may be regularly scheduled in the library media center to give teachers a free or planning period (creighton, 2007). as of november, 2010, the american association of school librarians (aasl) strongly opposes library scheduling that is designed to provide teacher release time, asserting that "the integrated library media program philosophy requires that an open schedule must be maintained." despite the aasl’s strong stance, some elementary school librarians oppose flexible scheduling (johnson, 2001), and many elementary school libraries used fixed schedules instead. while most public secondary school libraries use flexible scheduling, fewer than half of elementary school libraries use flexible or partially flexibly scheduling (creighton, 2007). although several research studies have examined the benefits of flexible scheduling on libraryteacher collaboration and instruction, there is a lack of statistical research that demonstrates a connection between flexible scheduling and student achievement (hurley, 2004). the current study makes an important contribution to this debate by linking scheduling to circulation statistics, a relationship that is critical because higher numbers of book check-outs may be connected to improved reading abilities of students. evidence based library and information practice 2011, 6.1 66 however, this study contains some methodological errors in regard to sampling. the authors explain that they chose a random sample size of 600 from a population of 2,000 school library media specialists, although they do not divulge how the participants in their sample were selected (e.g., simple random, systematic random, or stratified random). they authors sent their survey to 600 participants so as to ensure a 95% confidence level with a plus or minus three-point confidence interval. unfortunately, the authors calculated their confidence interval without anticipating a high refusal rate. since their survey response rate was only 29.3% (176 respondents), their confidence interval is actually much higher (for a discussion of population sampling and sample sizes, see beck & manuel, 2008). the authors further reduced their sample by 50% because they were only interested in responses from elementary schools. the result is a much smaller sample size (88) than originally planned, with only 11 responses coming from schools with totally flexible schedules. in addition, circulation data have been reported from only 53 schools. thus, the limitations of the sample suggest that the findings of this study may not be representative of the target population – that is, elementary school librarians in two school library associations. although the authors tested the survey instrument in advance, the sampling errors raise questions regarding the validity and reliability of the study. however, the authors themselves point out that their study is exploratory in nature and cannot be generalized to other public elementary schools in the united states. because the authors used statistical modeling to control for other variables that may have an impact on circulation, their results are still noteworthy, even if their findings are not statistically generalizable. this study provides useful guidance for further research into the impact of scheduling type and its relation to free voluntary reading and literacy development. free voluntary reading is the idea of reading for pleasure. krashen (2006) explains that "the secret of its effectiveness is simple: children become better readers by reading" (p. 43). free voluntary reading includes people’s ability to choose the materials they read without regard to completing assignments or selecting from a predetermined list. proponents of free voluntary reading point to a large amount research showing its connection to higher literacy skills (krashen, 2004, 2006). to help improve children’s literacy, school library media centres should provide as much access to books as possible and encourage children to read for pleasure. this study provides evidence that flexible scheduling may provide children with improved access to reading materials, and in turn, may be an important factor in children’s literacy development. continued research in school library media centers will attempt to articulate the factors that contribute to enhanced literacy skills in students. because there is a lack of statistical data showing support for the aasl’s position in regard to flexible scheduling, this study makes an important contribution to the debate by suggesting that flexible scheduling may play a critical role in improving student achievement. references american association of school librarians. position statement on flexible scheduling. retrieved 30 november 2010 from http://www.ala.org/ala/mgrps/divs/aa sl/aaslissues/positionstatements/ flexiblescheduling.cfm beck, s. e., & manuel, k. (2008). practical research methods for librarians and information professionals. london: nealschuman publishers. creighton, p. m. (2007). just how flexible are we? the current state of scheduling in school libraries. library media connection, 26(3), 10, 12-14. http://www.ala.org/ala/mgrps/divs/aasl/aaslissues/positionstatements/� http://www.ala.org/ala/mgrps/divs/aasl/aaslissues/positionstatements/� http://www.ala.org/ala/mgrps/divs/aasl/aaslissues/positionstatements/� evidence based library and information practice 2011, 6.1 67 hurley, c. a. (2004). fixed vs. flexible scheduling in school library media centers: a continuing debate. library media connection, 23(3), 36-41. johnson, d. (2001). it’s good to be inflexible. school library journal, 47(11), 39. krashen, s. d. (2004). the power of reading: insights from the research (2nd ed.). westport, ct: libraries unlimited. krashen, s. (2006). free reading. school library journal, 52(9), 42-45. / evidence based library and information practice evidence summary   increased size of e-book collection positively impacts usage but may reach critical mass   a review of: lamothe, a. r. (2013). factors influencing the usage of an electronic book collection: size of the e-book collection, the student population, and the faculty population. college & research libraries, 74(1), 39-59. retrieved 12 july 2013 from http://crl.acrl.org/content/74/1/39.full.pdf      reviewed by: eamon c. tewell reference & instruction librarian long island university, brooklyn campus brooklyn, new york, united states of america email: eamont@gmail.com     received: 20 mar. 2013  accepted: 5 jul. 2013      2013 tewell. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to investigate the impact of collection size, student population, and faculty population on the use of an e-book collection.   design – longitudinal quantitative analysis.   setting – mid-sized public university located in ontario, canada.   subjects – data from 79,821 e-books related to searches and viewings; data regarding number of e-books held, students enrolled, and faculty employed at institution.   methods – numbers of e-books purchased individually and in packages were calculated, followed by the acquisition of annual student and faculty numbers through the university institutional planning office. searches for and viewings of e-books conducted via vendor websites were obtained directly from vendors. data for all variables represent years 2002-2010.   main results – very high pearson’s correlation coefficients of r = 0.96 for searches performed and r = 0.91 for viewings were found in relation to the number of e-books held. while the annual increase in number of viewings was at a rate similar to that of e-books available, a 7% decrease in searches and viewings occurred in 2010. in terms of user populations, doctoral students exhibited the strongest association with e-book collection size followed by undergraduate students and faculty.   conclusions – based upon examination of correlation coefficients, the study concludes that the e-book collection size is closely associated with the level of e-book usage. the author notes that the data suggests use of the collection may possibly have leveled off, implying that additional large increases in the e-book collection could incur unnecessary expenditure. “viewings per e-book” and “searches per e-book” ratios were highest when e-books were obtained on an individual title-by-title basis, though the author cautions that this does not necessarily prove that selective purchasing results in increased use. a deeper quantitative analysis into e-book usage and academic program size is considered for future research, as well as a comparison between electronic reference books and monographs. the author recommends that similar research be performed at other institutions of varying size to determine whether the study’s results would be replicated.     commentary   as patrons continue to utilize online resources, as libraries face challenges accommodating ever-expanding physical collections, and as electronic book access and interfaces improve, significant questions arise regarding maximizing e-book usage. beginning in the mid-2000s methods for acquiring and evaluating e-books have been discussed in the literature with increasing frequency. however, library e-book collections are less often considered in relation to size, as naylor (1987) first accomplished with physical collections by correlating the collection size of public libraries in new york with average circulation. the lamothe study’s unique contribution to the literature is that it considers e-book use in relation to the size of the collection offered to users.   this study of e-books benefits from a clear description of data collection methods and an extensive sample size. the results are thoroughly expounded upon and numerous visual indicators of the results accompany the text. the author’s description of the historical development of the university’s e-book collection and its phases of growth provide valuable context for practitioners. appropriate tests for statistical significance were carried out and fully reported, minimizing the potential of random error.   issues regarding the study’s validity include a lack of research questions, a minor omission with implications for replication, and loss of potential data. the study’s objective is not stated, possibly resulting in readers being unsure of the purpose for study. the addition of research questions could provide the study with a stronger framework and clear answers to stated questions in the conclusion. in respect to methodology it is not apparent whether the data was obtained retrospectively or from 2002-2010 as the e-book collection grew, which may be problematic for other researchers interested in replicating the study. searches for e-books conducted via the library catalogue were not available, a considerable loss of potential data seeing that, by the author’s admission, faculty and graduate students rely heavily on library catalogues to locate and access e-books. despite these concerns the study’s findings remain valid overall.   the author does not conclude with any implications for practice, limiting recommendations to future studies to be conducted. evidence that e-book use is correlated to size of collection may be particularly significant for libraries with smaller electronic collections that are seeking to increase e-book adoption among users, yet many additional factors such as marketing (e.g., torabi, 2011) and ease of access do not allow for decision-making based on evidence related to collection size alone. in the most recent year that data was captured for this study, the number of e-books purchased surpassed the number of searches performed for the first time in five years. this is an important fact that is only briefly speculated upon, and may have further implications for practice. when might an e-book collection’s size outpace use? in what ways can librarians determine how to best accommodate both print and electronic collections within their budgets? further research could explore the effect of patron-driven acquisitions on e-book use, or comparing use of electronic reference materials to print. additional qualitative studies on the adoption of e-books would lend much-needed user perspectives to this constantly evolving topic.     references   naylor, r. j. (1987). the efficient mid-size library: comparing book budget to population to collection size. library journal, 112(3), 119-120.   torabi, n. (2011). academic libraries should consider a strategic approach to promotion and marketing of e-books. evidence based library and information practice, 6(4), 130-133. retrieved 12 july 2013 from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/11625/ evidence based library and information practice evidence based library and information practice 2011, 6.2 75 evidence based library and information practice commentary formulating the evidence based practice question: a review of the frameworks karen sue davies assistant professor, school of information studies university of wisconsin–milwaukee milwaukee, wisconsin, united states of america email: daviesk@uwm.edu received: 17 jan. 2011 accepted: 04 apr. 2011 2011 davies. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. introduction questions are the driving force behind evidence based practice (ebp) (eldredge, 2000). if there were no questions, ebp would be unnecessary. evidence based practice questions focus on practical real-world problems and issues. the more urgent the question, the greater the need to place it in an ebp context. one of the most challenging aspects of ebp is to actually identify the answerable question. this ability to identify the question is fundamental to then locating relevant information to answer the question. an unstructured collection of keywords can retrieve irrelevant literature, which wastes time and effort eliminating inappropriate information. successfully retrieving relevant information begins with a clearly defined, well-structured question. a standardized format or framework for asking questions helps focus on the key elements. question generation also enables a period of reflection. is this the information i am really looking for? why i am looking for this information? is there another option to pursue first? this paper introduces the first published framework, pico (richardson, wilson, nishikawa and hayward, 1995) and some of its later variations including eclipse (wildridge and bell, 2002) and spice (booth, 2004). sample library and information science (lis) questions are provided to illustrate the use of these frameworks to answer questions in disciplines other than medicine. booth (2006) published a broad overview of developing answerable research questions which also considered whether variations to the original pico framework were justifiable and worthwhile. this paper will expand on that work. mailto:daviesk@uwm.edu� evidence based library and information practice 2011, 6.2 76 question frameworks in practice pico the concept of pico was introduced in 1995 by richardson et al. to break down clinical questions into searchable keywords. this mnemonic helps address these questions: p patient or problem: who is the patient? what are the most important characteristics of the patient? what is the primary problem, disease, or co-existing condition? i – intervention: what is the main intervention being considered? c – comparison: what is the main comparison intervention? o outcome: what are the anticipated measures, improvements, or affects? medical scenario and question: an overweight woman in her forties has never travelled by airplane before. she is planning an anniversary holiday with her husband including several long flights. she is concerned about the risk of deep vein thrombosis. she would like to know if compression stockings are effective in preventing this condition or whether a few exercises during the flight would be enough. p – patient / problem: female, middle-aged, overweight i – intervention: compression stockings c – comparison: in-flight exercises o – outcome: prevent deep vein thrombosis the pico framework and its variations were developed to answer health-related questions. with a slight modification, this framework can structure questions related to lis. the p in pico refers to patient, but substituting population for patient provides a question format for all areas of librarianship. the population may be children, teens, seniors, those from a specific ethnic group, those with a common goal (e.g., job-seekers), or those with a common interest (such as a gardening club). the intervention is the new concept being considered, such as longer opening hours, a reading club, after-school activity, resources in a particular language, or the introduction of wi-fi. lis scenario and question: art history master’s students submit theses with more bibliography errors than those from students of other faculties. the dean of art history raised this issue with the head librarian. the head librarian suggested that database training could help. p – population: art history master’s students i – intervention: database searching training c – comparison: students with no training or students from other faculties o – outcome: improved bibliographic quality table 1 illustrates the different components introduced in several pico framework variations. fineout-overholt and johnson (2005) considered the questioning behavior of nurses. they suggested a five-component scheme for evidence based practice questions using the acronym picot, with t representing timeframe. this refers to one or more time-related variables such as the length of time the treatment should be prescribed or the point at which the outcome is measured. a picot question in the lis field is: in a specialist library, does posting the monthly library bulletin on the website instead of only having printed newsletters available result in increased usage of the library and the new resources mentioned in the bulletin? in this question, the timeframe refers to a month. petticrew and roberts (2005) suggested picoc as an alternative ending to picot, with c representing context. for example, what is the context for intervention delivery? in lis, context could be a public library, academic library, or health library. a variation similar to picot is picott. in this instance, neither t relates to timeframe. the ts refer to the type of question and the best type of study design to answer that particular question (schardt, adams, owens, keitz, and fontelo, 2007). an example lis question is: in a specialist library, does instant messaging or e-mail messaging result in the greatest customer satisfaction with a virtual reference service? this type of question is user analysis, and a relevant type of study design is evidence based library and information practice 2011, 6.2 77 a questionnaire. the picott framework may be too restrictive when searching. if you are searching for effective websites then transaction log analysis would be a reasonable type of study design. by limiting to that study type you would miss user observation studies, focus groups, and controlled experiments. these frameworks should focus the search strategy, while not excluding potentially useful and relevant information. specifically developed for building and adapting oncology guidelines is pipoh (adapte collaboration, 2009). the second p refers to professionals (to whom the guideline will be targeted) and h stands for health care setting and context (in which the adapted guideline will be used). an example of this in the lis setting would be: what is appropriate training for fieldwork students working on the library’s issue or circulation desk? p – population: library users i – intervention: training p – professionals: fieldwork students o – outcome: s – setting: issue or circulation desk dawes et al. (2007) developed pecodr and undertook a pilot study to determine whether this structure existed in medical journal abstracts. e refers to exposure, replacing table 1 components of the different pico-based frameworks pa ti en t / p op ul at io n in te rv en tio n c om pa ri so n o ut co m e t im ef ra m e c on te xt t yp e of q ue st io n t yp e of s tu d y d es ig n pr of es si on al s h ea lt h c ar e se tt in g e xp os ur e d ur at io n r es ul ts e nv ir on m en t st ak eh ol d er s si tu at io n richardson et al., 1995 fineoutoverholt & johnson, 2005 petticrew & roberts, 2005 schardt et al., 2007 adapte collaboration, 2009 dawes et al., 2007 schlosser & o'neil-pirozzi, 2006 dicenso, guyatt, & ciliska, 2005 evidence based library and information practice 2011, 6.2 78 intervention to allow the inclusion of different study types such as case control studies and cohort studies. the d stands for duration, either the length of time of the exposure or until the outcome is assessed. the r refers to results. here is a sample lis question: does teaching database searching skills to postgraduate students in a hands-on workshop compared to a lecture result in effective skills to utilize throughout two or more years of study? duration would be the length of the postgraduate course (2+ years), and results could be defined as effective searching skills. schlosser and o'neil-pirozzi (2006) proposed pesico which applied to the field of fluency disorders and speech language pathology. e refers to the environment or the context in which the problem occurs, and s stands for stakeholders. stakeholders are an important consideration in certain library settings. lis scenario and question: each year, library staff accompany new university students on an introductory library tour. the tour is timeconsuming and may not be appropriate for new students who have much information to absorb in their first few days. library staff and student instructors suggested that staff post a virtual library tour on the website. it can be accessed at a time and place to suit the student, and may improve their understanding of library services. p – population: new university students e – environment: library s – stakeholders: library staff and student instructors i – intervention: virtual library tour c – comparison: physical library tour o – outcome: improved understanding of library services many of the adapted pico frameworks introduce terms worth consideration depending on the subject, area, topic, or question. the elements which are additions to the original pico framework could serve as filters to be reviewed after gathering the initial pico search results. they can help determine the relevance of initial search results. for example, consider filtering on context when determining if the results from a rural public library service are directly applicable to a large endowed university library. dicenso, guyatt, and ciliska (2005) suggested that questions which can best be answered with qualitative information require just two components. such questions may focus on the meaning of an experience or problem. p – population: the characteristics of individuals, families, groups, or communities s – situation: an understanding of the condition, experiences, circumstances, or situation this framework focuses on these two key elements of the question. an lis example is: in a public library, should all library staff who have face-to-face, telephone, or e-mail contact with users attend a customer awareness course? p population: library staff with user contact s situation: customer awareness course eclipse pico and its variations were all developed to answer clinical questions. within the medical field there are other types of questions which need to be answered. eclipse was developed to address questions from the health policy and management area (wildridge and bell, 2002). e – expectation: why does the user want the information? c client group: for whom is the service intended? l – location: where is the service physically sited? i – impact: what is the service change being evaluated? what would represent success? how is this measured? this component is similar to outcomes of the pico framework. p – professionals: who provides or improves the service? se – service: what type of service is under consideration? evidence based library and information practice 2011, 6.2 79 lis scenario and question: there have been user complaints about the current interlibrary loan (ill) service. what alternatives might improve customer satisfaction? e – expectation: improve customer satisfaction c client group: library users who request ills l – location: library i – impact: improve the ill service p – professionals: ill staff se – service: ill spice the previous frameworks can all be adapted to answer lis questions. one framework, spice, was developed specifically to answer questions in this field (booth, 2004): s – setting: what is the context for the question? the research evidence should reflect the context or the research findings may not be transferable. p – perspective: who are the users, potential users, or stakeholders of the service? i – intervention: what is being done for the users, potential users, or stakeholders? c – comparison: what are the alternatives? an alternative might maintain the status quo and change nothing. e – evaluation: what measurement will determine the intervention’s success? in other words, what is the result? the spice framework specifically includes stakeholders under p for perspective and is therefore similar to the pesico framework. lis question: in presentations to library benefactors, does the use of outcome-based library service evaluations improve their perceptions of the importance and value of library services? s – setting: library presentation to funders p – perspective: library benefactors i – intervention: outcome-based evaluations of library services c – comparison: other evaluations e – evaluation: improved perception of the importance and value of library services some of these additional concepts are related. context, environment, and setting have similar connotations, and duration is similar to timeframe. this suggests that the options for constructing well-defined questions are not as numerous as table 1 suggests. combining comparable and related terms would provide the following concepts: p – population or problem i – intervention or exposure c – comparison o – outcome c – context or environment or setting p – professionals r – research – incorporating type of question and type of study design r – results s – stakeholder or perspective or potential users t – timeframe or duration conclusion these frameworks are tools to guide the search strategy formation. a minor adaption to the medical question frameworks, usually something as simple as changing patient to population, enables the structuring of questions from all the library and information science domains. rather than consider all of these frameworks as essentially different, it is useful to examine the different elements: timeframe, duration, context, (health care) setting, environment, type of question, type of study design, professionals, exposure, results, stakeholders, and situation. these can be used interchangeably when required. maintaining an awareness of the different options for structuring searches broadens the potential uses of the frameworks. detailed knowledge of the frameworks also enables the searcher to refine strategies to suit each particular situation rather than trying to fit a search situation to a framework. evidence based library and information practice 2011, 6.2 80 references the adapte collaboration. (2009). the adapte process: resource toolkit for guideline adaption (version 2). retrieved from http://www.g-in.net/document-store/adapteresource-toolkit-guideline-adaptationversion-2 booth, a. (2004). formulating answerable questions. in a. booth & a. brice (eds.), evidence based practice for information professionals: a handbook (pp.61-70). london: facet publishing. booth, a. (2006). clear and present questions: formulating questions for evidence based practice. library hi tech, 24(3), 355-68. doi:10.1108/07378830610692127 dawes, m., pluye, p., shea, l., grad, r., greenberg, a., & nie, j.y. (2007). the identification of clinically important elements within medical journal abstracts: patient population problem, exposure intervention, comparison, outcome, duration and results (pecodr). informatics in primary care, 15(1), 9-16. dicenso, a., guyatt, g., & ciliska, d. (2005). evidence-based nursing: a guide to clinical practice. st louis, mo: elsevier mosby. eldredge, j. d. (2000). evidence-based librarianship: an overview. bulletin of the medical library association, 88(4), 289-302. fineout-overholt, e., & johnson, l. (2005). teaching ebp: asking searchable, answerable clinical questions. worldviews on evidence-based nursing, 2(3), 157-60. doi: 10.1111/j.17416787.2005.00032.x nollan, r., fineout-overholt, e., & stephenson, p. (2005). asking compelling clinical questions. in b. m. melnyk & e. fineout-overholt (eds.). evidence-based practice in nursing and healthcare: a guide to best practice (pp.25-37). philadelphia: lippincott, williams & wilkins. petticrew m., & roberts, h. (2005). systematic reviews in the social sciences: a practical guide. malden, ma: blackwell publishing. richardson, w. s., wilson, m. c., nishikawa, j., & hayward, r. s. a. (1995). the well-built clinical question: a key to evidence-based decisions. acp journal club, 123, a12-13. schardt, c., adams, m. b., owens, t., keitz, s., & fontelo, p. (2007). utilization of the pico framework to improve searching pubmed for clinical questions. bmc medical informatics and decision making, 7, 16. doi:10.1186/1472-6947-7-16 schlosser, r. w., & o'neil-pirozzi, t. (spring, 2006). problem formulation in evidence-based practice and systematic reviews. contemporary issues in communication sciences and disorders, 33, 5-10. wildridge, v., & bell, l. (2002). how clip became eclipse: a mnemonic to assist in searching for health policy/management information. health information and libraries journal, 19(2), 113-115. doi: 10.1046/j.14711842.2002.00378.x http://www.g-i-n.net/document-store/adapte-resource-toolkit-guideline-adaptation-version-2� http://www.g-i-n.net/document-store/adapte-resource-toolkit-guideline-adaptation-version-2� http://www.g-i-n.net/document-store/adapte-resource-toolkit-guideline-adaptation-version-2� http://www.g-i-n.net/document-store/adapte-resource-toolkit-guideline-adaptation-version-2� http://dx.doi.org/10.1108/07378830610692127� / evidence based library and information practice microsoft word news_call.doc evidence based library and information practice 2007, 2:1  160 evidence based library and information practice     news    call for papers: evidence based library and information practice      © 2007 eblip. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    evidence based library and information  practice seeks papers on all areas  of ebl/ebp including, but not limited to:     • ebl application   • qualitative and quantitative research   • management and administration issues  related to ebp   • research tools (statistics, data collection  methods, etc.)   • collaborative and interprofessional ebp   • research education in library schools   • evidence‐based practices from other  disciplines applicable to ebl   • harnessing evidence to support new  innovations   • developing and applying evidence  based tools   • future prospects for the evidenced  based information profession   • maximizing the value and impact of our  information services      submission deadlines for 2007:  march 1st (for june 15th issue)  june 1st (for september 15th issue)  september 1st (for december 15th issue)    further information regarding author  guidelines and the submission process can  be found on the submissions section of the  eblip website.    http://creativecommons.org/licenses/by/2.0 evidence summary   there is no association between subject liaisons’ perception of their work and faculty satisfaction with their liaisons   a review of: arendt, j. & lotts, m. (2012). what liaisons say about themselves and what faculty say about their liaisons, a u.s. survey. portal: libraries and the academy, 12(2), 155-177. doi:10.1353/pla.2012.0015   reviewed by: lisa shen business reference librarian sam houston state university huntsville, texas, united states email: lshen@shsu.edu   received: 22 aug. 2013 accepted: 18 oct. 2013      2013 shen. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine the relationship between librarians’ self-assessment of their liaison responsibilities and faculty’s satisfaction with their liaison’s performance, and the factors influencing these perceptions.   design – web-based survey questionnaire.   setting – the survey was conducted over the internet through email invitations.   subjects – 354 librarians and 140 faculty members from selected universities and colleges in the united states.   methods – 602 colleges and universities were selected based on institution size, degrees offered, and financial status using u.s. department of education’s 2008 institution data. each institution was randomly assigned one of three subject designations: chemistry, psychology, or english. a randomly selected faculty member from the designated subject department and their corresponding subject liaison librarian (“liaison”) were contacted for the survey.   institution websites were used to locate faculty and liaisons. if a list of liaisons could not be found, then a librarian from the website’s available contact list was randomly selected instead. the chosen individuals were invited via email in april 2010 to participate in the online survey. before the survey closed in mid-may, up to two follow-up emails were sent to those who had neither responded nor asked to be removed from the contact list. the survey questionnaire was delivered through the lime survey platform and consisted of 53 items in 15 questions.    main results – the survey had an overall response rate 41.0%: 58.8% from librarians and 23.3% from faculty. three hundred and four of the 354 librarians surveyed (85.9%) were self-identified liaisons, although researchers were unable to identify 61 of them through their library websites.   most liaisons surveyed had responsibilities in the areas of collection development (96.1%), instruction (87.2%), and reference (82.6%). they provided an average of eight types of liaison services, some of which fall under these categories. the liaisons worked with an average of four academic departments (m=4.12, sd=2.98) and spent approximately 10 hours per week (m=10.36, sd=9.68) on their subject responsibilities.   the majority of liaisons felt they were successful (62.5%) or very successful (13.8%) in their liaison services and were either satisfied (50.7%) or very satisfied (12.2%) with the liaison relationship with their departments. e-mail (97.2%) was the liaisons’ most frequently cited communication channel. the frequency of contact with their departments had the highest correlation (gamma = -0.567, p < 0.05) with liaisons’ perception of their own performances.   of the 140 faculty surveyed, 104 indicated that their library had liaisons and 66.3% of them had had some contact with the liaison within the previous 6 months. faculty who knew their liaison by name (gamma = 0.668, p < 0.05) or who had recent contact with the liaison (gamma = -0.48) were more satisfied with the liaison services than those who did not. faculty who received more services from their liaisons (gamma = 0.521) also indicated greater satisfaction than those who received fewer services.   faculty assigned higher importance than liaisons did to three liaison services: faculty participation in collection development, new publication notices, and copyright information. on the other hand, liaisons ranked the importance of information literacy-related services, including in-class library instruction sessions and integration of library instruction into the curriculum, much higher than did faculty.      furthermore, 66 pairs of liaisons and their corresponding subject faculty completed the surveys. forty-nine of the faculty members out of those matched pairs knew their liaisons and were more satisfied with the liaison services than those who did not. however, no other relationships, such as correlations between faculty satisfaction of their liaisons and liaisons’ assessment of their own performance, could be found between responses of these matched faculty and liaison pairs.   conclusion – this study highlighted the disparity between faculty’s and librarians’ perceptions of library liaison programs. most notably, there were no statistically significant relationships between liaisons’ perception and satisfaction of their work and their faculty members’ satisfaction of the liaison services. faculty and liaisons also differed in their assigned importance to various types of liaison services.   moreover, while faculty’s satisfaction with liaison services correlated with the frequency of their contact with and the number of services received from their liaisons, their satisfaction did not translate into approval of the library. no statistically significant relationship could be found between faculty’s familiarity or interaction with their liaisons and their satisfaction with their libraries overall.     commentary   since much of the research on library liaison services is limited to the perspectives of librarians or to programs within a single institution, this article provides a timely contribution to current scholarship. the study scored an overall rating of 84% based on the evidence-based librarianship (ebl) critical appraisal checklist (glynn, 2006). rating for each ebl sub-section was also equal to or greater than 75%, indicating both an overall and section validity.   however, there is one notable flaw in the study design: the authors chose english, chemistry, and psychology as the representative academic disciplines without providing any sound rationale for their choices. this selection included two subjects in the humanities, but excluded all of fine arts, engineering, education, medicine, and business. since faculty members’ information needs vary across disciplines, such unbalanced subject selection reduces the representativeness of the study findings. possible variations between responses from the three disciplines were also unexamined, even though the findings could be valuable to liaisons working with multiple academic departments.   in addition, while selected survey questions are described in the results, readers would benefit from complete copies of the questionnaires. in particular, it is unclear whether the faculty survey effectively defined, or differentiated between, liaison and general library services. for instance, a few faculty members indicated that they received copyright information as a liaison service, even though their corresponding liaisons did not provide copyright consultations.   libraries commonly offer faculty services through channels in addition to subject liaisons. therefore, faculty respondents may have mistakenly attributed all services from the library as services from their liaisons, or vice versa. since the authors aim to differentiate between faculty satisfaction of their liaison and of the library, providing the actual questionnaires would aide readers in determining whether a lack of clearly defined service channels in the faculty survey had significantly impacted the validity of selected findings.   nonetheless, despite a few areas for improvement, this study provides a timely examination of liaison services and highlights the lack of evidence based research to support the effectiveness of liaison programs and their values to academic libraries. furthermore, the findings not only provide practical implications for liaison librarians to evaluate and prioritize the type of services offered based on faculty feedback, but also offer directions for future scholarship, such as comparisons of liaison service perceptions across multiple disciplines, or interaction between liaisons and multiple faculty members from the corresponding departments.     references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi:10.1108/07378830610692154   ebl 101   research methods: action research   virginia wilson director, centre for evidence based library and information practice (c-eblip) university library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   received: 31 oct. 2013   accepted: 15 nov. 2013      2013 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   action research, also known as participatory action research, is a type of enquiry that emerged from the field of education and is primarily used by teachers. various other disciplines also use action research, such as public health and nursing (hannigan, 2008). however, any practitioner in a social practice who conducts research could find action research a useful tool. action research is undertaken at the local level and often involves the participation of those who might otherwise be considered subjects of the research. it’s a practical type of research designed to inform local issues of a practical nature. so, the teachers are looking at questions from their classrooms and librarians as researchers are looking at issues arising from their library. action research’s focal point is a problem to be solved and it is undertaken by practitioners.   a definition of action research:  “action research is the study of a social situation carried out by those involved in that situation in order to improve both their practice and the quality of their understanding” (munn-giddings & winter, 2002, p. 8). the notion of “social situation” here is a broad one, and while it may suggest mingling at a cocktail party, it should be taken to mean that any social interaction that a practitioner or a practitioner’s organization has with people (users, clients, students, patients, etc.) is a candidate for action research.   action research is a process that helps practitioners develop a better understanding about the particulars of a specific practice-based situation. it has a personal and a social aim: there is ideally improvement in the practitioner’s learning and an improvement in the situation that is being investigated. these two aims are interdependent. (mcniff, lomax, & whitehead, 2003).   in terms of the steps one takes for an action research project, there are a variety of ways in which the process is described. some say there are three steps although each of those steps has several sub-steps. sometimes the steps are referred to as phases with a variety of numbers there. i’m going to list five basic steps in action research with the caveat that this process is an iterative, cyclical, and reflexive process rather than a linear one.   action research steps   identify – problem, question, baseline data, and professional development plan – the intervention, assessment, and time to assess collect data analyze data reflect – what was learned, what will be improved, and making improvements (teachers’ pd inc, 2012-13)   critics of action research are bothered by the notion that the process involves the local level and input from stakeholders, claiming that these are case studies, or the “how i done it good in my library” study, and are most likely not generalizable to other situations or institutions (beck & manuel, 2008, p. 196). beck and manuel (2008) state that “while not claiming that its results are completely representative or generalizable, action research does include a number of methodological steps to ensure that it is rigorous in obtaining its results, and these steps help to ensure that results are at least somewhat representative or generalizable” (p. 196). as with any other methodology, care should be taken to ensure that it is the best methodology to deal with a particular research question.   action research has emerged in the library literature around the topic of information literacy and elsewhere. here are some examples.   greenan, e. (2002). walking the talk: a collaborative collection development project. school libraries in canada, 21(4), 12-14.   kendall, m. (2005). tackling student referencing errors through an online tutorial. aslib proceedings, 57(2), 131-145.   rious, k. (2013). teaching social justice in an information literacy course: an action research case study. catholic library world, 83(3), 191-195.   vezzosi, m. (2006). information literacy and action research: an overview and some reflections. new library world, 107(7-8), 286-301.   wilson, t.d. (2000). recent trends in user studies: action research and qualitative methods. information research 5(3). retrieved 31 oct. 2013 from http://informationr.net/ir/5-3/paper76.html   there are lots of books and articles available that describe the process of action research from various perspectives and in varying levels of detail. here are some examples:   chevalier, j.m. & buckles, d.j. (2013). participatory action research: theory and methods for engaged inquiry. new york : routledge.   cook, d. & farmer, l. (eds.). (2011). using qualitative methods in action research: how librarians can get to the why of data. chicago, il: association of college and research libraries.   howard, j.k. & eckhardt, s.a. (2005). action research: a guide for library media specialists. worthington, oh: linworth.   mcniff, j.  & whitehead, j. (2011). all you need to know about action research (2nd ed). los angeles, ca: sage.   mills, g.e. (2014). action research: a guide for the teacher researcher (5th ed). boston : pearson.   pavlish, c.p. & pharris, m.d. (2012). community-based collaborative action research: a nursing approach. sudbury, ma : jones & bartlett learning.   for a research topic with very practical, practice-based origins, action research may be the best way to explore it and obtain useful, reflective answers.     references   beck, s.e. & mauel, k. (2008) practical research methods for librarians and information professionals. new york: neal-schuman.   hannigan, g.g. (2008). action research: methods that make sense. medical reference services quarterly 16(1), 53-58. doi: 10.1300/j115v16n01_06   mcniff, j., lomax, p., & whitehead, j. (2003). you and your action research project. taylor & francis. retrieved 31 oct. 2013 from <http://www.myilibrary.com?id=3669>   winter, r. & munn-giddings, c. (2002). a handbook for action research in health and social care. routledge. retrieved 31 oct. 2013 from <http://www.myilibrary.com?id=6016>   teachers’ pd inc. (2012-13). action research model. retrieved 31 oct. 2013 from http://www.csuchico.edu/teachergrants/actionresearch/ar_model.shtml   evidence based library and information practice 2011, 6.3 1 evidence based library and information practice evidence summary students are confident using federated search tools as much as single databases a review of: armstrong, a. (2009). student perceptions of federated searching vs. single database searching. reference services review, 37(3), 291-303. doi:10.1108/00907320910982785 reviewed by: deena yanofsky liaison librarian walter hitschfeld geographic information centre mcgill university montreal, quebec, canada email: deena.yanofsky@mcgill.ca received: 4 mar. 2011 accepted: 15 july 2011 © 2011 yanofsky. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to measure students’ perceptions of the ease-of-use and efficacy of a federated search tool versus a single multidisciplinary database. design – an evaluation worksheet, employing a combination of quantitative and qualitative questions. setting – a required, first-year english composition course taught at the university of illinois at chicago (uic). subjects – thirty-one undergraduate students completed and submitted the worksheet. methods – students attended two library instruction sessions. the first session introduced participants to basic boolean searching (using and only), selecting appropriate keywords and searching for books in the library catalogue. in the second library session, students were handed an evaluation worksheet and, with no introduction to the process of searching article databases, were asked to find relevant articles on a research topic of their own choosing using both a federated search tool and a single multidisciplinary database. mailto:deena.yanofsky@mcgill.ca� evidence based library and information practice 2011, 6.3 54 the evaluation worksheet was divided into four sections: step-by-step instructions for accessing the single multidisciplinary database and the federated search tool; space to record search strings in both resources; space to record the titles of up to five relevant articles; and a series of quantitative and qualitative questions regarding ease-of-use, relevancy of results, overall preference (if any) between the two resources, likeliness of future use and other preferred research tools. half of the participants received a worksheet with instructions to search the federated search tool before the single database; the order was reversed for the other half of the students. the evaluation worksheet was designed to be completed in one hour. participant responses to qualitative questions were analyzed, codified and grouped into thematic categories. if a student mentioned more than one factor in responding to a question, their response was recorded in multiple categories. main results – participants indicated a slight preference for using a federated search tool over a single multidisciplinary database. of the 31 students who completed the evaluation worksheet, 16 (51.6%) found that their search results were more relevant in the federated search tool; 10 (32.3%) students reported that the articles they found were more relevant in the single database. three students stated that both search tools produced equally relevant results and two students responded that neither resource produced relevant results on their topic. when asked to state which resource they would be likely to use in the future, 22 students (71%) indicated that they would use the federated search tool and 21 (67.7%) students answered that they would use the single multidisciplinary database. of the participants who expressed potential use of the single database to look for articles in the future, 43% referred to the ease of use or efficiency of the search tool in their responses. similarly, more than half of the students who stated that they would use the federated search tool in the future (54.5%) also cited ease of use or efficiency in their answers. in total, 11 students (35.5%) stated that they would be unlikely to use the federated search tool for future research. in their responses, students referred to the inefficiency or complexity of the research tool. of the 12 participants (38.7%) who stated that they would be unlikely to use the single multidisciplinary database, 50% cited a lack of relevant results and 42% referred to the overall complexity and inefficiency of the database. conclusions – the results of this study do not support a significant preference among undergraduate students for either search tool. though some participants struggled with terminology or various features of each resource, more students expressed confidence and satisfaction with the search process no matter which tool they opted to use. given student confidence and comfort level in both research environments, the author suggests that librarians should place equal weight on both types of resources in library instruction. commentary the aim of this study was to measure undergraduate student search preferences for using a federated search tool compared to navigating a single multidisciplinary database. unlike other usability studies of metasearch tools, interpretations of effectiveness, efficiency and user satisfaction were based solely on student first impressions; the students themselves determined the usefulness and quality of search results after only initial and brief exposure to both resources. although the author does acknowledge some of the limitations of this approach, the overall quality of the study is uneven and does not provide a significant contribution to the literature on metasearch environments. evidence based library and information practice 2011, 6.3 55 while the study was designed to measure the functional abilities of students using two different metasearch tools for the first time, it is unclear how many students were in fact firsttime users or whether students were indeed novice researchers. participants were enrolled in two sections of english 161, a required first-year course at the uic. the author states that “most” of the participants were first-year college students; she does not provide data to support this statement or clarify how many students fall into this category. the total response rate is also not reported. given that the underlying assumption of this study is that first experiences indicate future use of a search tool, the fact that the study does not make clear each student’s level of education or verify their previous experience with each search tool is a substantial omission. the author reports that of the 31 students who took part in this study, 21 indicated that they might use the single multidisciplinary database to look for articles in the future and 22 students answered that they would be likely to use the federated search tool. while it is clear that several students responded with confidence in both resources, the study does not separate the number of students who responded positively to both search tools from the number of students who experienced difficulties or frustrations with one or both tools. a comparison would have given more weight to the study’s findings, making it possible to determine whether the responses indicated an actual preference for one tool or feature over the other. it is worth noting that although the results of this study are limited by several significant methodological errors, the goal was simply to discover whether subjects would be as comfortable, confident and functional using a federated search tool as using individual databases. it is safe to conclude that students feel self-sufficient and confident regardless of which search tool they opt to use. these findings indicate that information literacy models should focus on transferability, one of the desired outcomes of information literacy instruction and a core component of digital literacy. the tendency for students to self-report educational success and confidence using online information and communication tools is well documented in the literature. in order to provide adequate information literacy skills for today's students, librarians must encourage the development of critical thinking skills through activities that emphasize the nature of information and the information-seeking process as a whole, regardless of the tools being used or the topic being studied. / evidence based library and information practice evidence summary students are confident using federated search tools as much as single databases a review of: armstrong, a. (2009). student perceptions of federated searching vs. single database searching. reference services review, 37(3), 291-303. doi:10.1108/00907320910982785 received: 4 mar. 2011 accepted: 15 july 2011 abstract objective – to measure students’ perceptions of the ease-of-use and efficacy of a federated search tool versus a single multidisciplinary database. design – an evaluation worksheet, employing a combination of quantitative and qualitative questions. setting – a required, first-year english composition course taught at the university of illinois at chicago (uic). subjects – thirty-one undergraduate students completed and submitted the worksheet. methods – students attended two library instruction sessions. the first session introduced participants to basic boolean searching (using and only), selecting appropriate keywords and searching for books in the library catalogue. in the second librar... the evaluation worksheet was divided into four sections: step-by-step instructions for accessing the single multidisciplinary database and the federated search tool; space to record search strings in both resources; space to record the titles of up to... participant responses to qualitative questions were analyzed, codified and grouped into thematic categories. if a student mentioned more than one factor in responding to a question, their response was recorded in multiple categories. main results – participants indicated a slight preference for using a federated search tool over a single multidisciplinary database. of the 31 students who completed the evaluation worksheet, 16 (51.6%) found that their search results were more rele... when asked to state which resource they would be likely to use in the future, 22 students (71%) indicated that they would use the federated search tool and 21 (67.7%) students answered that they would use the single multidisciplinary database. of the ... in total, 11 students (35.5%) stated that they would be unlikely to use the federated search tool for future research. in their responses, students referred to the inefficiency or complexity of the research tool. of the 12 participants (38.7%) who sta... conclusions – the results of this study do not support a significant preference among undergraduate students for either search tool. though some participants struggled with terminology or various features of each resource, more students expressed con... given student confidence and comfort level in both research environments, the author suggests that librarians should place equal weight on both types of resources in library instruction. commentary the aim of this study was to measure undergraduate student search preferences for using a federated search tool compared to navigating a single multidisciplinary database. unlike other usability studies of metasearch tools, interpretations of effectiv... while the study was designed to measure the functional abilities of students using two different metasearch tools for the first time, it is unclear how many students were in fact first-time users or whether students were indeed novice researchers. par... the author reports that of the 31 students who took part in this study, 21 indicated that they might use the single multidisciplinary database to look for articles in the future and 22 students answered that they would be likely to use the federated s... it is worth noting that although the results of this study are limited by several significant methodological errors, the goal was simply to discover whether subjects would be as comfortable, confident and functional using a federated search tool as us... how affective and emotional labor impede or facilitate new teaching practices among information literacy instruction librarians is inconclusive evidence summary   how affective and emotional labor impede or facilitate new teaching practices among information literacy instruction librarians is inconclusive   a review of: galoozis, e. (2019). affective aspects of instruction librarians’ decisions to adopt new teaching practices: laying the groundwork for incremental change. college & research libraries, 80(7), 1036–1050. https://doi.org/10.5860/crl.80.7.1036   reviewed by: eugenia opuda health & human services librarian assistant professor dimond library university of new hampshire durham, new hampshire, united states of america email: eugenia.opuda@unh.edu   received: 28 feb. 2022                                                                  accepted:  4 apr. 2022      2022 opuda. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30112     abstract   objective – to investigate the correlation between emotional and affective labour and instruction librarians’ willingness to adopt and implement novel teaching and educational practices.   design – semi-structured interview, grounded theory.   setting – academic libraries in the greater western library alliance consortium.   subjects – 12 information literacy librarians from the greater western library alliance consortium of 38 research libraries from the us.   methods – in 2016, the author shared a call for study participants in the greater western library alliance (gwla) consortium, selected a convenience sample of 12 information literacy instruction (ili) librarians, and conducted a structured interview over adobe connect, a virtual video-interviewing tool. interview transcripts underwent initial thematic coding using a grounded theory approach. additionally, the author used walker and symons’ theories of motivation to code interview responses related to emotional and affective labour.   main results – the author identified three thematic categories in the interview transcripts: barriers and influences for adopting new teaching practices, and practices implemented by ili librarians. the author mapped these response themes to walker and symons’ (1997) conditions of human motivation: autonomy, competency, feedback, affirmation, and setting meaningful goals. some major barriers to adopting new teaching practices are burnout and emotional exhaustion due to understaffing, time demands, the sheer quantity of instruction sessions, and the lack of post-instruction feedback to reinforce pedagogical decisions. a sense of competency, autonomy, and support when designing library instruction sessions encourages librarians to adopt new teaching practices. the author explored what practices ili librarians applied to implement new teaching practices. having plenty of time to prepare prior to an instruction session encourages ili librarians to build new teaching practices into sessions. the respondents noted that building relationships with faculty, students, and library colleagues enables them to receive helpful feedback.   conclusion – though there is some correlation between affective and emotional labour and the motivation of ili librarians to adopt and implement new teaching practices, the author notes that the results are not generalizable to a larger context based on the small sample size. it is clear there are many opportunities to investigate other factors that impact librarian motivation and emotional labour including the dynamics of race, gender identity, and disability, or the managerial and structural factors and workplace culture that impede or facilitate the adoption of new teaching practices.     commentary   this research is a direct response to celene seymour’s (2012) call to explore the emotional labour of ili librarians in library instruction and contributes to the larger body of education and library information science research on motivation and the impact of affective and emotional labour on adopting new teaching practices. currently, existing literature also examines ili and student affect in the context of online instruction (fernández-ramos, 2019; rapchak, 2019) as well as other teaching modalities (baragash & al-samarraie, 2018), but does not address motivation or emotional and affective labour of ili librarians. this article addresses the gap in the literature to identify emotional and affective labour in library instruction contexts, and highlights barriers and factors to positive motivation.   this research article was appraised using an adapted version of russell and gregory’s (2003) qualitative research appraisal questions (suarez, 2010). though a specific research question is not articulated, the author provides a description of her research aims. the selected methodology is appropriate for the identified outcomes though it is unclear whether the interview questions were validated, how interviews were conducted, how data was collected, and what modality of instruction was investigated. however, the research methodology does not clearly identify how the questionnaire was distributed to 38 research libraries in the gwla, how informed consent was obtained, and what inclusion and exclusion criteria were used to select ili librarians for the interview. a convenience sample was selected due to the author’s institution participating in another research study with the same pool of gwla libraries. it is unclear whether sampling methods were appropriate considering there is no demographic information of study participants or an explanation of why demographic data was not shared. the author conducted two phases of data analysis of interview transcripts, thematic coding using grounded theory and the theories of motivation, which were appropriate to the research study.   the author is incredibly descriptive of the research findings and upfront about some of the limitations of the study. she emphasizes that the findings are not intended to be generalizable. the author’s conclusions accurately reflect the two-phased analysis and address the objectives of the study. additionally, the author calls for more research examining how race, gender identity, disability, structural factors, and workplace culture contribute to adoption of new teaching practices.   based on the results from the structured interview as well as the existing literature, the author makes practical recommendations for ili librarians and library leaders and managers to identify and address emotional labour. the results of this research provide strategies to incrementally change workplace culture in order to build motivation to adopt new teaching practices in library settings. this research also demonstrates the potential value of using qualitative research in uncovering thoughts and experiences of library workers to address library-related issues. additionally, the author brings attention to the importance of recognizing and proactively addressing emotional and affective labour in the library workplace, which has historically remained invisible, and unacknowledged elements that contribute to librarianship. these types of invisible labour can contribute to burnout, lack of satisfaction, and reluctance to explore new teaching practices among ili librarians. future research could investigate how student engagement contributes to ili librarian motivation, as well as strategies to meaningfully collaborate, advocate, and negotiate with teaching faculty outside of the library that can lead to changes to the larger university culture.   references   baragash, r. s., & al-samarraie, h. (2018). an empirical study of the impact of multiple modes of delivery on student learning in a blended course. the reference librarian, 59(3), 149–162. https://doi.org/10.1080/02763877.2018.1467295   fernández-ramos, a. (2019). online information literacy instruction in mexican university libraries: the librarians’ point of view. the journal of academic librarianship, 45(3), 242–251. https://doi.org/10.1016/j.acalib.2019.03.008   rapchak, m. (2019). when online instruction doesn’t measure up: how can you tell, and what should you do? journal of library & information services in distance learning, 13(1–2), 150–158. https://doi.org/10.1080/1533290x.2018.1499248   russell, c. k., & gregory, d. m. (2003). evaluation of qualitative research studies. evidence-based nursing, 6(2), 36–40. http://dx.doi.org/10.1136/ebn.6.2.36   seymour, c. (2012). ethnographic study of information literacy librarians’ work experience: a report from two states. in c. w. wilkinson & c. bruch (eds.), transforming information literacy programs: intersecting frontiers of self, library culture, and campus community. association of college and research libraries.   suarez, d. (2010). evaluating qualitative research studies for evidence based library and information practice. evidence based library and information practice, 5(2), 75–85. https://doi.org/10.18438/b8v90m   walker, c. j. & symons, c. (1997). the meaning of human motivation. in j. l. bess (ed.), teaching well and liking it: motivating faculty to teach effectively (pp. 3–18). johns hopkins university press.     article   looking and listening: a mixed-methods study of space use and user satisfaction   sara holder head librarian schulich library (science, engineering & medicine) mcgill university montreal, canada email: sara.holder@mcgill.ca   jessica lange business librarian humanities & social sciences library mcgill university montreal, canada email: jessica.lange@mcgill.ca   received: 4 june 2014     accepted: 7 aug. 2014         2014 holder and lange. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – this study was designed to assess users' reactions to two newly re-designed spaces – one intended for quiet study and the other for group study – in the busiest library branch of a large research university. the researchers sought to answer the following questions: for which activity (group work, quiet study, and lounging or relaxing) do the users feel the space is most effective? which furniture pieces do users prefer and for which activities? how are these spaces being used?   methods – researchers used a mixed-methods approach for this study. two methods – surveys and comment boards – were used to gather user feedback on preference for use of the space and users’ feelings about particular furniture types. a third method – observation – was used to determine which of the particular areas and furniture pieces occupants were using most, for which activities the furniture was most commonly used, and what types of possessions occupants most often carried with them.   results – user opinion indicated that each of the spaces assessed was most effective for the type of activity for which it was designed. of the 80% of respondents that indicated they would use the quiet study space for quiet study, 91% indicated that the space was either "very effective" or "effective" for that purpose. the survey results also indicated that 47% of the respondents would use the group study space for that purpose. the observation data confirmed that the quiet study space was being used primarily for individual study; however, the data for the group study space showed equal levels of use for individual and group study. users expressed a preference for traditional furniture, such as tables and desk chairs, over comfortable pieces for group work and for quiet study. one exception was a cushioned reading chair that was the preferred item for quiet study in 23% of the responses. the white boards were chosen as a preferred item for group study by 27% of respondents. the observations showed similar results for group study, with the three table types and the desk chair being used most often. the lounge chairs and couch grouping was used most often for individual study, followed by the tables and desk chairs.   conclusion – by combining user feedback gathered through surveys and comment boards with usage patterns determined via observation data, the researchers were able to answer the questions for which their assessment was designed. results were analyzed to compare user-stated preferences with actual behaviour and were used to make future design decisions for other library spaces. although the results of this study are institutionally specific, the methodology could be successfully applied in other library settings. introduction   library spaces are increasingly transforming from those designed to house collections to those concerned with user comfort and support for activities beyond the use of the collection. when planning these user-centered spaces, it is common practice to assess the preferences and needs of the users who will ultimately occupy them. some libraries have gathered user feedback both pre-design and post-design, prior to construction or renovation (norton, butson, tennant, & botero, 2013). but what of the users’ opinions of these new or updated spaces once they are completed? is it necessary to gather these opinions? an argument could be made that it is not essential if the designers of the space have been conscientious in polling users and applying their feedback. feedback could also be risky in a situation where it would be difficult to change elements of the design should the user feedback be negative. what if, however, there were opportunities to duplicate the design, or use elements of it, in other spaces? this consideration was at the root of the project described in this article.   the humanities & social sciences library at mcgill university is comprised of two adjoining structures, the mclennan and redpath library buildings (known in combination as the mclennan-redpath complex). the public spaces in these buildings have been updated at various times throughout their lifespans; however, there had not been any targeted efforts to determine whether the spaces were meeting students' needs. the first project was to assess a recently renovated quiet study area on the third floor of the mclennan building. upgrades included new lighting and furniture: specifically, long, electrified tables with dividers, large and small tables on wheels, simple desk chairs, and comfortable reading chairs.   the second project involved a deal with a local company to provide furniture pieces on a trial basis that the library could either purchase or switch out for different types. the trial furniture was largely of the comfortable seating type but also included two configurations of tables and desk chairs as well as moveable and fixed white boards. in order to showcase these pieces, the library opened up an area of recently vacated staff space to create a large group work zone (including two enclosed, bookable group study rooms), dubbed the “furniture lab.” with these two projects in place, the library administration tasked the mclennan-redpath space planning working group with gathering student feedback. given the large number of new furniture types and pieces being used in these spaces, and the trade-in agreement for the furniture lab pieces, the administration was particularly interested in gauging student reaction to the individual types and pieces and finding out how the students were using the furniture and the spaces. if these spaces were well-received by users, they could be duplicated, either in whole or in part, in other areas of the complex and in other branches. the nature of these renovations also left some latitude for change if users were not satisfied.   literature review   in the past decade there has been an increasing body of library research devoted to space planning and space assessment in libraries. as webb, schaller, and hunley (2008) note, “the proliferation of digital formats, the options for high density storage, and the increased ease of resource sharing have reduced the need for on-site collection storage thus opening up space for other types of services” (p. 407). “library as place” has emerged as librarians look for ways to accurately measure how users are engaging with their spaces, what users want from their spaces, and what the space demands will be for the future. future space demands are particularly hard to predict, especially with changing technology. for example, increasing the number of electrical outlets has been identified as a major space need in many studies given the rise of the laptop computer, something that may have been difficult to imagine even 10 or 15 years ago (brown-sica, 2012; halling & carrigan, 2012; norton et al., 2013; vaska, chan, & powelson, 2009).   in the space planning literature, obtaining information on user preferences and space demands is addressed in various ways. one common method is to engage students directly about their desires for changes to library spaces. many studies rely on traditional feedback methodologies such as surveys, focus groups, whiteboards, and comment boards to obtain information; however, more innovative strategies such as photo diaries and mediated drawing exercises are also being explored. for example, crook and mitchell (2012) had several of their students keep an audio diary to reflect on their study habits and behaviour. similarly, hobbs and klare (2010) provided students with disposable cameras and asked them to photograph their interpretation of various pre-defined subjects such as their favourite place to study. other studies focus on what students actually do in library spaces; these studies rely on observational methods to ascertain how users are engaging with their spaces. bedwell and banks (2013) partnered with an anthropology class at their university to make direct observations of students’ habits, activities, and behaviours in the library. others have employed a mixed methodology, using several of these approaches to answer their research questions. for example, pierard and lee (2011) employed photo diaries, flipcharts, and a traditional user survey in their study, and crook and mitchell (2012) employed observation, audio diaries, and focus groups, while foster and gibbons (2007) used interviews, maps, photographs, and flipcharts. these less traditional means for obtaining feedback are considered ethnographic approaches. as asher, miller, and green  (2012) write,  “ethnographers typically describe a particular situation or process by asking multiple people about it, and by analyzing multiple types of data, such as interviews, direct observation, photographs, journals, or cultural artifacts” (p. 3). through combining various feedback methodologies, researchers hope to obtain more well-rounded and comprehensive information about the population which they are studying.   beyond library literature, the fields of urban planning and architecture provide insights into further feedback methodologies for public spaces. in addition to interviews and observation methods, doxtater (2005) employed an online virtual recreation of a university residence to understand user experiences with the space. hua, göçer, and göçer (2014) used interviews and surveys to understand user satisfaction with a newly leed (leadership in environmental energy and design) certified university building. this data was then combined with objective measurements such as temperature and humidity and mapped spatially to create a visual representation of how effective the renovations had been. within architecture literature, post-occupancy evaluations provide additional examples of obtaining user feedback about spaces. the san francisco public library (2000) administered focus groups, staff and user surveys, observations, and interviews to evaluate library spaces. in her article, cranz (2013) outlines the effects of this post-occupancy evaluation on the san francisco public library. preiser and wang (2008) provide an additional example of a post-occupancy evaluation of a library space by architects.   likewise, literature on urban planning involving citizen participation can also provide additional feedback approaches. shipley and utz (2012) provide a good overview of these methods, such as public meetings, focus groups, citizen juries, visioning, and scenario workshops.   one component discussed in several space planning research articles is furniture preference: do students prefer couches, carrels, booths, or other types of furniture? research conducted by halling and carrigan (2012), hobbs and klare (2010), pierard and lee (2011), and webb et al. (2008) identified a preference or desire for soft or comfortable furniture. while most of the aforementioned studies relied on student comments or surveys to determine this preference, webb et al. (2008) supported this through direct observation as well. they found there was a “higher than expected usage for soft furniture and computer stations” and a lower than expected usage for more traditional types of furniture such as large tables and chairs (webb et al., 2008, p. 415). foster and gibbons (2007) came to this same conclusion. however, the preference for “soft” or comfortable is not consistent across all studies and there is often a difference between students’ stated preference and their behaviour.   contrary to the above, vaska et al. (2009) discovered in their survey that carrel areas in the library were the most popular spaces, while applegate (2009) noted through observation that study rooms were the most frequently used spaces (followed by “soft spaces”). brown-sica (2012) also noted through observation that traditional furniture such as tables and chairs were popular, reflecting a “need to ‘get down to work’ as opposed to socializing” (p. 223). this contrast in findings may best be explained by the diversity of functions that students wish their library to fulfill. bailin (2011) found that students wanted more of everything out of their library space (more individual study spaces, more group study spaces, more computers) and that the breakdown of what spaces students said they used is fairly evenly divided across all options (e.g., group, individual, lounge, and others). webb et al. (2008) noted through observation that 70% of the students were engaged in individual study (p. 416). crook and mitchell (2012) observed that approximately 50% of students were engaged in individual study while the rest were engaged in conversation of some variety (p. 128). this diversity of activities in libraries may best be summed up by montgomery (2011), who ascertained that students “want to study alone but still need space to meet in groups” (p. 84). as such, a variety of furniture is required in order to meet those needs.     based on the literature consulted, initial student feedback, and general observations, the working group had several assumptions about what the study would find. given the ubiquity of laptops as well as the literature reviewed, they anticipated that students would desire more outlets. additionally, even though the humanities & social sciences library is intended primarily for students in the faculty of arts, given the central location and size of the branch, the group anticipated that students from all disciplines would make use of the space. with regards to the furniture lab, they hypothesized that it would be used primarily for group study, as it is located in a high traffic area. finally, given several studies which outlined student preference for furniture, and some initial student feedback they had received, the working group expected that students would prefer “comfy” or “soft” furniture in the furniture lab space. however, since this study was partly exploratory, the group hoped to obtain additional information beyond the assumptions outlined.   methodology   the working group chose to use a combination of methods to obtain data about these spaces: surveys, observation, and comment boards. they designed a survey instrument with questions that focused on elements that could be changed (such as furniture) and questions that prompted the respondents to offer their opinion of what type of activities the space was best suited for. similarly, they used the comment boards to solicit feedback on particular furniture pieces and general satisfaction with the spaces. to account for the potential difference in students’ stated preferences and their actual behaviour, the researchers also employed the observation method (goodman, 2011). using survey and observation methods together provided a more complete picture of user satisfaction with the spaces, as well as user preference for particular areas and furniture types. this mixed-methodology approach and combination of survey and observation data was inspired by webb et al. (2008), who combined video surveillance footage with surveys and web polls to obtain information on students’ library space use.   although webb et al. (2008) inspired this mixed methodology approach, due to privacy regulations, video surveillance was not an option for observing student behavior at mcgill. for this reason, the working group modeled their observation method on given and leckie (2003), who describe how research teams at two canadian public libraries used “an unobtrusive patron-observation survey, called ‘seating sweeps’” to answer questions about the use and functionality of central libraries as public space (p. 373). this observation method collects minimal user demographics (sex, estimated age) and data on user activity (what they are doing) and possessions (what they have with them) in a specified space at a specified time. the observation criteria used was also adapted from given and leckie (2003), particularly their list of possessions and activities. this method also allowed the group to compare the students’ survey responses and comments with their behaviour. since there were two separate spaces being evaluated, all of the data collection elements had two parts – one for the furniture lab and one for the mclennan building third floor space.   the working group's use of comment boards had its roots in two places. several members of the group had prior experience with this method and had found that it complimented the use of surveys. use of this approach was also inspired by halling and carrigan (2012) who utilized whiteboard voting in their study as one method for obtaining student feedback.   survey design   both the mclennan building third floor survey (appendix a) and the furniture lab survey (appendix b) instruments included seven questions: six multiple choice and one open-ended. two of the questions were demographic (type of patron and faculty/department affiliation) and four were designed to obtain the students’ opinions about the effectiveness of the space and furniture pieces for particular activities (group work, quiet study, and lounging or relaxing). on both surveys, the final question was open-ended to allow for any additional comments or suggestions regarding the space or furniture. all of the questions on both surveys were optional and the surveys were completely anonymous.   data collection   surveys   both of the surveys were made available in paper and online format. the paper surveys were offered to students using a container attached to the boards through which comments were being solicited. a second container was used to collect the completed surveys. the group members also used these boards to indicate the web address where students could access the online version of the surveys, which were offered via surveymonkey. both versions of the surveys were available for approximately two weeks.   comment boards   in order to solicit comments on the boards, group members used a combination of open questions about the space and about specific pieces of furniture. using the bulletin boards, the group members attached pictures of specific pieces of furniture spaced evenly throughout the board with the following solicitation across the top of the board: “we want to know what you think of the new group study space.” additional prompts were posted as well, such as: “which is your favorite?” and “love it/love it not.” sticky notes and markers were available so students could write comments and attach them near the relevant furniture picture. the group members used the whiteboards to solicit comments about the space by writing: “what do you think of this space?” or simply: “comments?” group members visited the boards several times each day to collect the completed surveys and to take pictures of (and refresh) the comment boards.   observation   the group members set up two online forms (one for each space) using google drive to record and analyze the data from their observations. this gave the observers the choice of recording their observations on paper and entering the results in the online form at their leisure or using a laptop or tablet to record the data in the online form as they performed their observations. the observation forms (appendix c) were designed using given and leckie's (2003) as a template. the group members decided to record the number of male and female users but not to estimate the users’ age as this was not relevant to the study. they used some of the same variables as given and leckie (2003) in the possessions and activities categories and made some additions. they also added four categories to the form: interaction (students working alone/students working collaboratively/other), position (sitting/standing/other), whiteboard use (no whiteboard/not using/using individually/using interactively/there is writing on whiteboard but not clear if it is from current occupant/other), and adequate space provided for possessions (yes/no/other).   the group members mapped out both spaces in order to break them down into locations that would be observed. the mclennan building third floor space includes several different types of seating in repeated groupings throughout the floor (appendix d: third floor area map). the group members assigned numbers to each of these similar seating groups (e.g., tables with blue dividers, area #1, #2, etc.), as well as the four group study/seminar rooms, and added them as locations. a total of 14 locations were included in the observation form for the mclennan building third floor space. the form for the furniture lab space also included 14 locations; however, on this form each of the locations corresponded to individual furniture types (appendix e: furniture lab pictures). the group members planned 12 observations of each space at corresponding times spread over one week (table 1). in total, 10 observations were completed for the furniture lab space and 11 for the mclennan building third floor space.   results   surveys   third floor   the working group received 41 completed surveys (38 paper and 3 online) for the mclennan building third floor space and 88 (78 paper and 10 online) for the furniture lab space. the respondents to both surveys were primarily undergraduates (85% and 90%) and the largest number indicated that they were part of the faculty of arts (39% and 46%). this was not surprising as the humanities & social sciences library houses many of the materials the arts students would need to complete their assignments, as well as the offices of the liaison librarians for the departments in the faculty. however, it was notable that the second largest number of respondents to both surveys indicated they were part of the faculty of science (24% and 22%). most respondents (77%) to the mclennan building third floor space survey indicated that they use the space for quiet study, and 91% rated the space either very effective (54%) or somewhat effective (37%) for this type of studying (figure 1).   the comments regarding the mclennan building third floor space design were very positive, in particular regarding the lighting, colour scheme, and designation of zones for quiet study. several respondents suggested that the space could be improved if more electrical outlets were added and several others suggested that library staff should enforce the quiet study concept for those zones. temperature is often an issue in the large buildings on the mcgill campus (especially in the winter) so it was not a surprise that numerous respondents mentioned that the space was too cold.   furniture lab   the responses for space use preference in the furniture lab were more surprising, considering that the space was designed for group work. the largest group of responses (47%) indicated the intent to use the space for group work; however, 30% of respondents indicated that they intended to use the space for quiet study, and 23% indicated that they intended to use the space for lounging or relaxing (figure 2).     table 1 observation times for furniture lab and mclennan building third floor space (week of december 9, 2012) day time monday 10 a.m. tuesday 10 a.m., 2 p.m., 4 p.m. thursday 10 a.m., 2 p.m., 4 p.m. friday 10 a.m., 2 p.m., 4 p.m. saturday 5 p.m. sunday 8 p.m.   figure 1 effectiveness of the third floor study space for quiet or individual study.   figure 2 preference for space use, furniture lab.     the survey form (appendix b) offered a selection of 12 furniture pieces so that respondents could indicate their top preference for the three types of activity: group study, quiet study, and lounging or relaxing. for quiet study, 36% of respondents chose the y-shaped divided table as the top furniture item, with the red desk-arm chair a close second at 23%. the other highly-rated item was the desk chair (17%), which is used with the y-shaped divided table and the u-shaped table (figure 3). the top-rated item for group work was the portable whiteboard (27%), followed by the desk chair (13%) and the u-shaped table (11%) (figure 4). the remaining 49% of the responses for this question were divided among the other nine furniture items.     figure 3 furniture preference, quiet study.   figure 4 furniture preference, group study.   there was a similar breakdown in responses for the top-rated item for lounging or relaxing. the question mark lounger was chosen by 23% of respondents, the reading chair with wooden arms by 20%, and the low-slung reading chair by 17%. the remaining 40% of responses were divided among the remaining nine items. the final question on both survey forms was an open-ended solicitation for comments or suggestions. the furniture lab survey respondents most commonly suggested that the space should have more tables, electrical outlets, and whiteboards. they also suggested that the whiteboard markers be replaced more frequently. the comments were generally positive toward the space, especially its design and designation as a group study space, though there was a mixed response to the furniture colours.   comment boards   the bulletin board and whiteboard comments were a mix of positive and negative; however, several items received consistently positive comments. these items included the moveable whiteboards ("more please"), the round and u-shaped tables and desk chairs ("the best"; "beautiful"), the low-slung reading chair ("this chair is pure happiness"), and the question mark lounger ("love it so sassy"). it was notable that the three-sided table that had been the top choice in the furniture lab survey for quiet study received comments that confirmed it was not well-suited for group work ("chairs too close to each other"; "more appropriate for individual study space").   observation   third floor   during the 11 observations completed for the mclennan building third floor space, the working group members observed a total of 1,565 occupants. with the exception of two of the group study rooms, observations of each area showed a much higher instance (80% or greater) of occupants working alone than working together. in the areas where a whiteboard was present, all observations showed it was either being used or had been used (i.e., there was writing on it). the occupants in the third floor space were most commonly observed carrying out the following activities: using laptops or tablets, reading, writing, and using earphones (figure 5). the most common possessions observed were laptops or tablets, books, paper, backpacks or totes, and earphones. many of the occupants were observed in possession of beverages and most often (>60% of the time) these were in closed containers. eating was observed infrequently (<20% of the time).   furniture lab   in the 10 observations completed for the furniture lab space, a total of 490 occupants were observed. though the space was designed for group study, observations showed occupants using the space equally for independent study and for collaborative work. collaborative work was observed most often in the group study rooms, at the y-shaped divided table, and at the u-shaped and round tables. the whiteboards paired with the u-shaped tables were in use most often, followed closely by the whiteboards in the group study rooms. the main activities taking place in the furniture lab space were virtually the same as those most commonly observed in the mclennan building third floor space; however, drinking was slightly more common than using earphones (16% vs. 13%). eating was indicated in approximately 18% of the observations, most commonly at the round and u-shaped tables. occupants at the round tables, the u-shaped tables, and on the rounded chairs were most often observed reading. the most common possessions observed were identical to the mclennan building third floor space. occupants in the furniture lab space were in possession of drinks (most often in closed containers) on average 35% of the time.     figure 5 occupant activities, third floor.   figure 6 occupant activities, furniture lab.     discussion   two of the working group's initial assumptions proved to be accurate: that students would desire more outlets and that students from all disciplines would make use of the space.   it was not surprising, given the consensus in the literature that one of the students’ most frequent suggestions was for additional electrical outlets (brown-sica, 2012; halling & carrigan, 2012; norton et al., 2013; vaska et al., 2009). this was doubly confirmed via the observations, during which it was noted that both in the third floor quiet space and in the furniture lab, the most commonly observed item in the occupants’ possession was a laptop or tablet. these results, together with the studies mentioned earlier, provide evidence for including ample access to electricity in the design of any library space.   in reviewing the results from the surveys and comment boards, some were as expected, particularly the use of the spaces by students from a wide range of disciplines. even though the mclennan-redpath library complex serves primarily students in the faculty of arts, its location at the centre of campus makes it a hub for students in all faculties. this was demonstrated in the survey responses showing that all faculties were represented; notably, students in the faculty of science made up almost one quarter of those surveyed (figures 7 and 8).   two of working group's other assumptions proved to be inaccurate: that students would prefer “comfy” or “soft” furniture and that the furniture lab would be used primarily for group study.   both the survey responses and observations revealed a desire among users for more traditional furniture such as tables and desk chairs. the observation data showed that tables were the most commonly used item in the furniture lab for group study and that the lounge chair and couch grouping was only slightly more commonly used than the tables for individual study. in the surveys, the tables, desk chairs, and moveable whiteboards were the most preferred items. one cushioned reading chair was the only "comfy" item to show as preferred (23% for quiet study). as libraries are more and more becoming a “home away from home” for students, the working group members had anticipated users would express a greater preference for “comfy” furniture. there is also considerable evidence for this furniture type preference in the literature (halling & carrigan, 2012; hobbs & klare, 2010; montgomery, 2011; pierard & lee, 2011; webb et al., 2008). this divergence from the existing body of evidence indicates potential for further investigation; however, it may be attributable to the difference in survey design. the furniture lab survey instrument (appendix b) provided the opportunity for users to rate furniture based on its intended use (i.e., individual study, group study, or relaxing). other furniture preference studies asked more generally what type of furniture students would prefer without providing the option for selecting furniture based on different use scenarios. additionally, given that this study dealt with particular furniture pieces, it is possible that the respondents and occupants choices may indicate a lack of truly comfortable options rather than a true preference for desks and tables.   foster and gibbons (2007) discuss in their chapter on library design and ethnography that in their experience, library “zones” are “neither determined nor enforced by the library staff. rather the students develop and enforce them” (p. 20). given that assessment, the researchers should not have been surprised to discover that the furniture lab space was not being used as they had initially intended. the furniture lab is located in a busy, high-traffic area of the library; however, almost one-third of survey respondents indicated that they used the furniture lab for individual or quiet study. this was also confirmed through the observation results (figure 9) that show occupants working collaboratively just under 50% of the time.     figure 7 survey respondents by faculty, furniture lab.   figure 8 survey respondents by faculty, third floor.     several other studies have found similar results (bryant, matthews, & walton, 2009; crook & mitchell, 2012; harrop & turpin, 2013), which suggests this could be a common pattern in the use of space designed for group work. it would require further analysis to determine if students were willingly choosing to do their quiet study in that area or if this was not so much a choice as a necessity, given the lack of sufficient quiet space elsewhere in the library.   beyond validating or contradicting initial assumptions, the multi-method approach allowed the working group to discover additional information. in both the furniture lab and the third floor quiet space, the most commonly observed activities were the same: using laptops or tablets, reading, writing, and using earphones. this is consistent with other studies utilizing the observation method. given and leckie's (2003) results, gathered over ten years ago when laptops were less prevalent, found that reading and writing were the most popular activities, followed by computer use. bryant et al. (2009) found similar results, as did lehto, toivonen, and iivonen (2012).   however, it was encouraging to learn both through observation and through survey analysis that the third floor space was being used for its intended purpose (i.e., quiet study) and that it was generally regarded to be effective in fulfilling that objective.   limitations   in embarking on this project, the working group members’ objective was to get a better sense of what users liked and did not like about the re-designed spaces and how they were using the spaces. with this in mind, the group did not set out to be exhaustive in their data collection; they focused instead on using several methods to gather sufficient data to answer their questions without overextending staff time or annoying users. this approach limits the analysis and the strength of the conclusions that can be drawn from the data. the library is open to the public and the survey and comment boards were made freely available, thus the population size is unknown and the response rate cannot be defined. for this same reason there was no way to control for duplication or  multiple responses from the same individual. the results from the observations do not provide a complete picture, as data was not collected during the late night or early morning hours. finally, as is the case with any study done in a single site involving a particular population, the results of this investigation cannot be assumed to be typical or indicative of the opinions and preferences of other university populations. however, the authors feel that the methods could be successfully applied in other library settings.     figure 9 observed occupant interaction, furniture lab.     conclusion   as library spaces continue to adapt to meet the changing needs and expectations of their users, it is important for library administrators to gather feedback on user preferences and usage patterns. the past twenty years have seen radical changes in the physical layouts and use of space in libraries and there is no doubt that library spaces will continue to adapt and evolve over the course of the next several decades.   the authors found that a mixed method analysis was particularly useful for this project to determine both what users want out of their library spaces and how they are currently using them. observation data demonstrated usage patterns that may have been overlooked by traditional survey methods. conversely, survey responses provided important user feedback and comments. by combining the methods, this study illuminates some key issues, notably, the desire for traditional furniture (tables, chairs), as well as the need for more electrical outlets in all areas of the library, and the positive return on investment (high incidence of usage and user satisfaction) for the relatively low-cost addition of whiteboards. it also confirms that some library spaces are satisfying their anticipated need: the third floor quiet study area is in fact being used for that purpose and a majority of respondents find it effective in that respect.   the results of this project have been used to inform purchasing decisions to outfit other spaces in the mclennan-redpath complex as well as in other libraries on campus. the furniture lab space is being expanded such that it will more than double in size. following the findings of this study that the space was used for both group and individual work, the expanded space has been laid out accordingly and filled with the furniture items identified as most popular for each type of work. the most popular items from the furniture lab have also been installed in another branch’s new group space, and whiteboards have been added in several branches. the positive student response to the third floor space has been a factor in renovation design decisions for the first and second floors of the mclennan library building. the furniture in both areas has been updated to include long wood-finish tables (some with dividers, some without), similar to the ones observed to be popular in this study. all re-designed spaces and new tables will have multiple power outlets per seat (plug and usb). going forward, the library plans to continue obtaining user feedback to inform space planning decisions and to adapt the results of the research undertaken here to other library spaces on campus.     references   applegate, r. (2009). the library is for studying: student preferences for study space. the journal of academic librarianship, 35(4), 341-346. http://dx.doi.org/10.1016/j.acalib.2009.04.004   asher, a. d., miller, s., & green, d. (2012). ethnographic research in illinois academic libraries: the erial project. in l. m. duke, & a. d. asher (eds.), college libraries and student culture: what we now know (pp. 1-14). chicago, il: american library association.   bailin, k. (2011). changes in academic library space: a case study at the university of new south wales. australian academic & research libraries, 42(4), 342-359. http://dx.doi.org/10.1080/00048623.2011.10722245   bedwell, l., & banks, c. (2013). seeing through the eyes of students: participant observation in an academic library. partnership: the canadian journal of library & information practice & research, 8(1), 1-17. retrieved from http://condor.lib.uoguelph.ca/index.php/perj/article/view/2502/2905   brown-sica, m. s. (2012). library spaces for urban, diverse commuter students: a participatory action research project. college & research libraries, 73(3), 217-231. retrieved from http://crl.acrl.org/content/73/3/217.full.pdf   bryant, j., matthews, g., & walton, g. (2009). academic libraries and social and learning space: a case study of loughborough university library, uk. journal of librarianship and information science, 41(1), 7-18. http://dx.doi.org/10.1177/0961000608099895   cranz, g. (2013). how post-occupancy evaluation research affected design and policy at the san francisco public library. journal of architectural & planning research, 30(1), 77-90.   crook, c., & mitchell, g. (2012). ambience in social learning: student engagement with new designs for learning spaces. cambridge journal of education, 42(2), 121-139. http://dx.doi.org/10.1080/0305764x.2012.676627   doxtater, d. (2005). living in la paz: an ethnographic evaluation of categories of experience in a 'new urban' residence hall. journal of architectural & planning research, 22(1), 30-50.   foster, n. f., & gibbons, s. (2007). studying students: the undergraduate research project at the university of rochester. chicago, il: association of college and research libraries.   given, l. m., & leckie, g. j. (2003). ‘‘sweeping’’ the library: mapping the social activity space of the public library. library & information science research, 25(3), 365–385. http://dx.doi.org/doi:10.1016/s0740-8188(03)00049-5   goodman, v. d. (2011). qualitative research and the modern library. oxford, england: chandos.   halling, t. d., & carrigan, e. (2012). navigating user feedback channels to chart an evidence based course for library redesign. evidence based library & information practice, 7(1), 70-81. retrieved from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/10207   harrop, d., & turpin, b., (2013). a study exploring learners' informal learning space behaviors, attitudes, and preferences. new review of academic librarianship, 19(1), 58-77. http://dx.doi.org/10.1080/13614533.2013.740961   hobbs, k., & klare, d. (2010). user driven design: using ethnographic techniques to plan student study space. technical services quarterly, 27(4), 347-363. http://dx.doi.org/10.1080/07317131003766009   hua, y., göçer, o., & göçer, k. (2014). spatial mapping of occupant satisfaction and indoor environment quality in a leed platinum campus building. building and environment, 79, 124-137. http://dx.doi.org/10.1016/j.buildenv.2014.04.029   lehto, a., toivonen, l., & iivonen, m. (2012). university library premises: the evaluation of customer satisfaction and usage. in j. lau, a. tammaro, & t. bothma (eds.), libraries driving access to knowledge (pp. 289-314). boston, ma: de gruyter saur. http://dx.doi.org/10.1515/9783110263121.289   montgomery, s. e. (2011). quantitative vs. qualitative do different research methods give us consistent information about our users and their library space needs? library & information research, 35(111), 73-86. retrieved from http://www.lirgjournal.org.uk/lir/ojs/index.php/lir/article/view/482   norton, h. f., butson, l. c., tennant, m. r., & botero, c. e. (2013). space planning: a renovation saga involving library users. medical reference services quarterly, 32(2), 133-150. http://dx.doi.org/10.1080/02763869.2013.776879   pierard, c., & lee, n. (2011). studying space: improving space planning with user studies. journal of access services, 8(4), 190-207. http://dx.doi.org/10.1080/15367967.2011.602258   preiser, w. f. e., & wang, x. (2008). quantitative (gis) and qualitative (bpe) assessments of library performance. archnet ijar: international journal of architectural research, 2(1), 212-231. retrieved from http://archnet.org/publications/5108   san francisco public library. (2000). post-occupancy evaluation of main library. retrieved from http://sfpl.org/index.php?pg=2000043301   shipley, r., & utz, s. (2012). making it count: a review of the value and techniques for public consultation. journal of planning literature, 27(1), 22-42. http://dx.doi.org/10.1177/0885412211413133   vaska, m., chan, r., & powelson, s. (2009). results of a user survey to determine needs for a health sciences library renovation. new review of academic librarianship, 15(2), 219-234. http://dx.doi.org/10.1080/13614530903240635   webb, k. m., schaller, m. a., & hunley, s. a. (2008). measuring library space use and preferences: charting a path toward increased engagement. portal: libraries and the academy, 8(4), 407-422. http://dx.doi.org/10.1353/pla.0.0014     appendix a third floor survey instrument       appendix b furniture lab survey instrument                 appendix c observation data collection form   appendix d third floor area map       appendix e furniture pictures     audio feedback project: a project to increase social presence in a virtual library and knowledge service research article   audio feedback project: a project to increase social presence in a virtual library and knowledge service   matt holland library manager library and knowledge service for nhs ambulance services in england (lks ase) bolton, united kingdom email: matt.holland@nwas.nhs.uk received: 4 july 2021                                                                 accepted: 23 feb. 2022      2022 holland. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30006     abstract   objective – this research project sought to determine if audio feedback in literature searches can increase the social presence of the library and create a positive view of the library service. it also explored the process of recording and sending audio feedback; tested its practicality, sustainability, and accessibility; and ascertained whether audio feedback enhanced the library’s communication, thereby creating a positive attitude toward the library and its services.   methods – the research was conducted in a small virtual library and information service. the research sample consisted of all library users and clinicians who requested a mediated literature search between july 2019 and july 2020. all participants were sent an audio commentary on their search results, recorded by the librarian, and were asked to respond to an online questionnaire. the questionnaire consisted of five statements. the study participants indicated their agreement or disagreement with each statement on a five-point likert scale.   results – the researcher sent out 96 audio commentaries, generating 31 responses to the questionnaire. the results indicated that users felt the audio feedback improved their understanding of the results of their inquiry, made them feel more comfortable about using the library, enhanced their experience of communicating with the library and provided a better experience than just receiving an email. the responses broadly supported the contention that audio commentaries created social presence and generated a positive view of the library.   conclusion – the researcher found that delivering audio feedback was both practical and sustainable. some consideration was given to individual learning styles and how these made audio or text feedback more or less effective. specifically, audio feedback enhanced communications better than an email alone.     introduction   during the recent covid-19 pandemic, requirements for social distancing and home working have accelerated a trend toward virtual communication between users and librarians.  this was underpinned by improved meeting software and its wide availability on smartphones, tablets, and laptop computers. however, these forms of networked communication are synchronous, requiring both users and librarians to be present at the same time. this may be the preferred channel for some users, but there are factors that mitigate the simultaneous availability of users and librarians. these could include time pressures caused by increased workload, shifts in flexible and home working patterns, and the systemic pressures caused by unplanned events such as covid-19. for these reasons, some users may prefer asynchronous forms of communication, which is not time dependent, and both sender and receiver do not have to be present simultaneously. examples of asynchronous communication include email and web-forms for search requests.   social presence   in this project, the author drew on the theory of social presence. originally proposed in the 1970s to apply to what were then new forms of computer-mediated communication (short et al., 1976),  calefato and lanubile (2010) defined social presence as:   the degree to which one perceives the presence of participants in the communication. social presence theory argues that media differ in the ability to convey the psychological perception that other people are physically present, due to the different ability of media to transmit visual and verbal cues (e.g., physical distance, gaze, postures, facial expressions, voice intonation, and so on). (p. 287)   different technologies have different capacities for enhancing social presence. video conferencing, with a rich range of cues, conveys a higher degree of social presence than a telephone call. synchronous communications, with the capacity to immediately interact, interrogate, and clarify meaning, convey a greater degree of social presence than asynchronous communications, such as email.   digital audio   the author’s objective in initiating the audio feedback project, described in this paper, was to explore a simple and effective way to increase the library’s social presence in a way that was sympathetic to asynchronous communication between the library and its users. digital audio was selected because it is a widely used and understood technology that exists in all modern smartphones, tablets, and computers. audio is also low cost, as potential creators and users already have the technology they need. in addition, audio files are generally small and easy to play or stream on internet-connected devices.    audio feedback   the researcher used audio feedback on mediated search results, emailed to users. this was an area where complex information was conveyed to users. enhanced delivery was an added benefit. no literature exists on audio feedback in a library context. however, audio feedback given to students on assignments was used in higher education and has generated a small body of research literature that has informed this project. there were broad practical similarities in the process of giving feedback on a search versus an assignment. however, there were also pedagogical differences which were explored briefly in the literature review. in the case of the library, feedback was typically delivered through a digital audio file (.wav), either sent via email, or by sending a unique link to a file available in the cloud.   library and knowledge service for nhs ambulance services in england (lks ase)   the audio feedback project was conducted by the librarian, who is the only member of staff. lks ase is a virtual service operated via a website, email, social media, and phone. lks ase has a national footprint that covers eight of the ten ambulance services. lks ase users typically do not engage in synchronous communication with the library, for example, arranging to meet in person or talk on the phone, because of the challenges of distance and working busy shifts on ambulances or in emergency call centers. most communication is asynchronous, via email. the origin of this project was a desire to provide better service than can be achieved by email alone.   review of the literature   this project was mainly informed by research in the educational literature that focused on audio feedback to students. these were mostly small-scale studies, in which researchers looked at student and instructor attitudes to audio feedback, its effectiveness when compared to written feedback, and its role in delivering formative and summative assessment. while the pedagogical discussions of assessment were not applicable to this project, the educational studies provided both practical examples of implementation of audio feedback and valuable theoretical insights. it is noted that the majority of studies included here were published between 2002 and 2017, with fewer recent studies, possibly indicative of the integration of these technologies into virtual learning environments.   time and efficiencies of audio feedback   the majority of researchers found that delivering audio feedback did not take more time than delivering feedback as text  (brearley & cullen, 2012; cann, 2014; rotheram, 2009; sarcona et al., 2020). there were some qualifications. rotheram  (2009) noted that speed of delivery of feedback increased with experience in using the technology and recording audio feedback. one group of researchers found that delivering feedback through virtual learning environments (vle) added more time and complexity to the procedure than sending a file or a link to an audio file (carruthers et al., 2015). however, other researchers using a different software found no effect, finding that vle were highly efficient and especially suited to dispersed student communities (lunt & curran, 2010). in three studies, researchers compared audio feedback to written feedback. they found that in one case, one minute of audio feedback equated to six minutes spent on written feedback (lunt & curran, 2010). in another study (cann, 2014), the researcher estimated that it took five minutes to record a 500 word report. ice et al. (2007) found that written feedback took 13.43 minutes and audio feedback took 3.81 minutes, a 75% time savings. the broad implication of these studies was that audio feedback delivered more content in less time than written feedback.   audio feedback and learning styles   researchers in two studies make a connection between learning style and audio feedback. learning styles describe individual preferences for receiving information based on cognitive strengths and weaknesses. typically, learners are grouped in four categories: visual learners, auditory learners, reading and writing learners, and kinesthetic learners (university of kansas, 2021). students who had a preference for an auditory style of learning preferred audio feedback, while students who had a predominantly visual style preferred written feedback. according to sarcona et al. (2020): “consideration of students perceived predominant learning styles was reported in this study and found a significant association between students’ perceived learning style and type of feedback preference” (p. 57). in another study, a student aware of their learning styles reported this as a reason for a preference written feedback: “i am a visual learner and prefer the written word” (morris & chikwa, 2016, p. 6). wolstencroft and de main (2021) looked at the way students engage with written feedback. they found that many students failed to engage with feedback in a written form. in their study, audio feedback significantly increased the number of students who engaged with feedback.    higher information content of audio feedback   a number of researchers reported that audio feedback provided a greater level of detail than written feedback (carruthers et al., 2015; gould & day, 2013; parkes & fletcher, 2017, 2019; rawle et al., 2018; rodway-dyer et al., 2011). this was a quality recognised by students and instructors. students recognized this as a strength, attributing a greater depth of understanding to audio feedback. instructors reported that they were able to provide “more detailed and bespoke feedback to students” (lunt & curran, 2010, p. 764).   personal nature of audio feedback   researchers who used students as subjects reported that audio feedback felt more personal to the recipient (carruthers et al., 2015; lunt & curran, 2010; merry & orsmond, 2008; munro & hollingworth, 2014; parkes & fletcher, 2019; rasi & vuojärvi, 2018; rawle et al., 2018; rotheram, 2009).) wolstencroft and de main (2021) argued that audio feedback created emotional engagement and personal connectivity between instructors and students. students reported that hearing the voice of a tutor or instructor was a positive experience, and felt comforting, reducing feelings of isolation in an online environment (parkes & fletcher, 2017). in one study, students reported that they felt instructors who used audio feedback were more caring (ice et al., 2007). students felt the feedback was less generic and more tailored to their individual piece of work  (hennessy & forrester, 2014). researchers also noted that the human voice is more nuanced than written feedback, conveying greater meaning and emphasis (hennessy & forrester, 2014; ice et al., 2007) students, in turn, reported that audio feedback provided greater clarity than written feedback (parkes & fletcher, 2019; rawle et al., 2018).   social presence and audio feedback   several researchers make specific reference to the idea of presence in audio feedback. in their study, moore and wallace (2012) found that 30% of the students they surveyed identified the presence of the tutor’s voice as a benefit of audio feedback. ice at al. (2007), in their study,  argued that enhanced social presence increased student satisfaction with the course or programme: “we believe that audio feedback should be considered a means by which to increase positive perceptions of the quality of instructor interactions and, by extension, social presence in aln [asynchronous learning network]” (p. 19).  parkes and fletcher (2019) also argued that social presence created by audio feedback provides instructors with “the opportunity to engender a greater sense of connectedness with their students” (p. 452).  although library-based studies were absent, it might be reasonable to assume that audio feedback to users would create both a sense of social presence and greater user satisfaction in encounters with the library service.   the library literature   research on asynchronous communication with library users’ centers on the delivery of asynchronous instruction. in higher education, this research responded to the demands of increasing student numbers and the challenges of reaching students through face-to-face instruction. recent studies in healthcare have cited similar changes in working practice relating to service delivery during covid-19, as physical libraries closed to users. there was no substantial literature on asynchronous audio feedback in libraries. however, one researcher, taking a similar approach to the audio feedback project, used screen casting to provide answers to user inquiries (bailey, 2012). similarly, social presence in virtual communication with users represents a gap in the library literature.   aims   in this project, the researcher aimed to explore the process of recording and sending audio feedback to test its practicality sustainability, and accessibility. the researcher also sought to understand whether audio feedback enhanced a library’s communications, creating a positive attitude toward the library and its services where the alternative was just receiving an email.    in this small study, the researcher asked two questions: does audio feedback in library consultations improve user understanding of the results of their search? does audio feedback in library consultations increase users' impressions of social presence better than email feedback alone?    methods   the research was conducted by the author who is the sole member of professional staff for lks ase. the researcher was entirely responsible for all stages of the project. prior to the project all communication about search results was delivered via email.   the research sample is a convenience sample, defined as “nonprobability sampling in which people are sampled simply because they are ‘convenient’ sources of data for researchers” (battaglia, 2008, p. 149). in this survey, the convenience sample was all library users who requested a mediated literature search from lks ase between july 2019 and july 2020.  library users in this instance are ambulance staff and researchers employed by eight ambulance services in england. no demographic data was collected. it was not recorded whether users had used the library prior to the project. typically, inquiries were from users who were working and also studying, undertaking continuing professional development, or clinical inquiries. two of the ten ambulance services, south western ambulance service and london ambulance service, were excluded because they were not part of the lks ase project. the sample generated 96 audio feedback commentaries sent to users with links to a questionnaire. of those 96 users who received a commentary, 31 returned the questionnaire, giving a response rate of 30%. non-respondents may not have listened to the commentary, or listened to the commentary and chosen not to complete the questionnaire. the results of this project were particular to lks ase and were not generalizable.   data collection methods   inquirers received an email reply from lks ase containing a word document with results of their search as an enclosure, and a brief textual commentary. inquirers also received a unique link to a .wav file with a spoken audio commentary on their search. the spoken audio commentary was recorded on a work-supplied android smartphone using a free voice recording application. commentaries varied between one and a half to three minutes depending on the search complexity. they shared a basic format: an introduction, a description of the search, an analysis of the search results, an offer to supply full text copies of documents, and a request to complete the online questionnaire. some discretion was used in the format to make it appropriate to the inquirer and the search. the .wav files were shared with inquirers from a public space on the microsoft onedrive cloud platform. in addition, the email contained a request for feedback with a link to an online questionnaire.   the online questionnaire was a simplified adaptation of the satisfaction scale (gunawardena & zittle, 1997). the questionnaire had five statements in which audio commentary recipients were asked to rate their level of agreement on a five-point likert scale from strongly agree to strongly disagree. the questionnaire was created, delivered, and analyzed using the web based libguides survey tool.   results   most respondents strongly agreed or agreed with the statement “listening to the audio feedback improved my understanding of the results of my search or enquiry,” with a small neutral response. the three questions that addressed social presence, asking if users were more comfortable in using the library, felt their experience had been enhanced by audio feedback, and was better than email alone, also elicited mostly strongly agreed or agreed.  however, a slightly higher number of respondents chose neutral on these questions.   of the 96 questionnaires sent out, 31 were returned. it is unknown why the remaining 65 questionnaires were not returned. a limitation of the study is that non-return may have introduced a bias in the results if, for example, non-returners tended to disagree with the five statements. two respondents indicated they experienced technical difficulties in accessing the audio files. several respondents contacted lks ase directly. however, as the questionnaire was anonymous it is not known if they were these respondents.   learning points from the author, as a participant in the project, are summarised in the good practice in audio feedback section. from the perspective of lessons learned during the execution of the project, the time taken to record and send audio files reduced significantly with practice; the first recording took longer than the 50th. reflection during the course of the project, in particular on the key points covered in the recording, enabled improvements that simplified the check list, leaving more time to focus on a personalized individual feedback on the search.   the results of the survey are summarised in table 1.   table 1 summary of responses to audio feedback questionnaire a statement strongly agree   respondents / percentage agree   respondents / percentage neutral   respondents / percentage disagree   respondents / percentage strongly disagree   respondents / percentage listening to the audio feedback improved my understanding of the results of my search or enquiry. 10 / 32.26% 18 / 58.06% 3 / 9.68% 0 / 0% 0 / 0% hearing the librarians voice made me feel more comfortable about using the library in the future. 17 / 58.04% 10 / 32.26% 5 / 16.13% 0 / 0% 0 / 0% listening to the audio feedback enhanced my experience of using the library. 15 / 48.39% 12 / 28.71% 4 / 12.90% 0 / 0% 0 / 0% overall receiving audio feedback gave me a better experience than communicating via email alone.  16 / 51.61% 10 / 32.26% 5 / 16.13% 0 / 0% 0 / 0% i was able to listen to the audio file without difficulty.   21 / 67.74% 7 / 22.58% 1 / 3.23% 2 / 6.45% 0 / 0% n = 31. a these statistics were collected between july 2019 and july 2020.   discussion   practical, sustainable, and accessible   the audio commentaries were recorded with no additional cost to the library service as the hardware and software were already available. the average time taken to record a commentary, transfer to the cloud, copy the unique link to the recording, and to prepare and send an email with the search results, was about ten minutes. the recordings were made in a private office space, but any space with low ambient sound would be practical. the process required no additional training, as the skills required were essentially those required to operate a smartphone. with the exception of the studies that used specialized vle software, most higher education studies cited in the literature review used a .wav audio file either distributed via email or the cloud. the iteration of these studies over the course of an academic program, and this project sustained over a year, were evidence that audio feedback is both practical and sustainable.   audio is the most accessible form of electronic media. many people listen to podcasts and stream music. the skills, software, and hardware required to listen to audio files are ubiquitous. no studies reported significant problems in end users listening to audio. problems may be a result of issues beyond the end users’ control, such as corrupt files or hardware and software problems. in any communication with users, it is important to offer assistance with accessing files in the rare instances where this is an issue.   better than email alone   the research took place within a specific context of a virtual library and knowledge service trying to create a stronger connection with users where communications are asynchronous via email. the studies cited in the literature review indicated several benefits that accrue from audio feedback. these included the higher information content of audio and a personal connection when hearing a human voice, which gives a perception of presence in communication, thereby creating a more positive experience of the library.   the statements in the questionnaire were intended to test whether library users felt a greater sense of presence once they had listened to the feedback. the majority of those who responded indicated they strongly agreed or agreed with statements in the questionnaire. of the respondents, 27 (90%) agreed that “hearing the librarian’s voice made me feel more comfortable about using the library.” without overstating this single piece of evidence, the addition of audio added a different and personal dimension to the library experience because the librarian was more present through his or her voice. in addition, 27 respondents (90%) felt they had a better experience using the library after listening to the audio feedback.    underpinning the project was the challenge of how to enhance communication where the alternative format is email. for physical libraries providing virtual services, email does not reflect the richness of a physical environment. for virtual libraries, email does not project the full character of virtual services. technology already offers audio and video, not covered in this research. we may look forward to future technologies and software to enhance our communication toolkit. proactively exploring these response mediums will broaden our repertoire and accommodate the preferences of more users.   of all respondents, 84% agreed that audio feedback was better than email alone. in addition, their understanding of their search results was improved by listening to the audio.    learning styles   communicating with users face-to-face in person, or even face-to-face online, provides a rich set of cues for users to gather information that can be supported by signposting guides and support materials that typically populate a library website. where synchronous communication is not possible, there is a risk of an email monoculture which may suit many, but not all, users. this is reflected in two unsolicited free text comments received from participants in the project that highlight the varying effects of feedback delivery on recipients with different learning styles. one respondent said, “i like the addition of the voice debrief, helps for those of us that struggle with reading.” another respondent said, “i would still very much need the accompanying email to support the message – i’m very much a visual and read/write person.”   learning styles have the potential to affect how library users receive information. the predominance of one form of communication, text, may favour certain styles. the inclusion of multiple forms, text and audio, favors a larger community of library users than either text or audio alone. the unsolicited feedback received in the audio feedback project reflected that some prefer to hear rather than read information. it should be noted that speech to text technology is available with microsoft and google word processors. this was not explored in the audio feedback project but provides a quick solution to providing speech and text.   good practice in audio feedback   some studies provided summaries of good practice. while many recommendations address pedagogical issues relating to summative and formative feedback, some practical recommendations apply to all audio feedback. these include having a rubric, criteria, or script to guide feedback and ensure consistency (gould & day, 2013; lunt & curran, 2010). another researcher advised keeping recordings short (cann, 2014) to keep file sizes small, although the length should also be guided by the task in hand. keeping a reflective record of the experiences of using audio feedback was suggested to guide improvements and modifications later (carruthers et al., 2015). the following guidelines for good practice reflect suggestions from the literature and direct experiences from the audio feedback project:   1.      a simple check list that picks up the main headings of your feedback is useful. these act as an aide-memoire to ensure that you cover the main points. the checklist should be designed to fit the specific situation in your library and its users. it should probably be reviewed after repeated use. 2.      make notes of any specific points you want to address in your feedback to prompt you when you make a recording. 3.      keep your recordings short. try not to exceed three minutes.  4.      operate at the minimum level of technical complication, typically a smartphone and recording app generating a .wav file. this saves you time and ensures that your users can access your files easily. 5.      the tone of your recording should be that of an intelligent conversation between adults. a natural and straightforward approach to your recording is best. avoid humour, long words, and jargon, if you can. 6.      do not edit recordings, record in one take. as the recordings are short it is easy to stop and start again if you make a mistake. this is something that becomes easier the more recordings you do. the first recoding is more challenging than the 96th recording. 7.      listen back to a few seconds of each recording to check volume levels and any interference from background noise that you may not have noticed the first time. recordings do not need to be perfect, but they must be audible to the user. 8.      you may choose to store your recordings so that you can refer to them later. use a naming convention that makes files easy to retrieve for future reference. 9.      offer an opportunity for users to give your feedback on your recordings. this is generally good practice when evaluating a new service and may generate useful comment and feedback. 10.   in your email to users, offer to support any user who experiences technical problems.   limitations   this study was a small-scale convenience sample and cannot be generalized to cover all healthcare library and knowledge services. however, the conclusions fit in with similar studies on audio feedback in higher education supporting their general conclusions. there is no information from non-respondents to the questionnaire who may have experienced unreported technical problems or simply not listened to the audio feedback. a further larger scale study looking specifically at a healthcare library and knowledge service context would provide a useful validation of this approach. studies that include an examination of the impact of learning style or the way information is received would be useful.   conclusion   the shift to being a virtual library service, whether caused by short term factors, such as covid-19, or longer-term effects of changing working patterns, presents both a challenge and an opportunity to rethink how we communicate with our users. while synchronous communication using video conferencing software provides part of the answer, for asynchronous communication many will use email. the challenge for virtual services is to think creatively about how to use email to build better communication with our users. in this small project, using audio feedback for literature searches enhanced communication with users and delivered richer content that projects the librarian’s presence in a novel but accessible way at very low cost. audio feedback also provided a choice to users who may prefer audio to text.    acknowledgment   the author would like to thank the reviewers, the editor and copyeditor, and anne norman who read and commented on a version of this paper.   references   bailey, j. (2012). informal screencasting: results of a customer‐satisfaction survey with a convenience sample. new library world, 113(1/2), 7-26. https://doi.org/10.1108/03074801211199013   battaglia, m. (2008). convenience sampling. in p. j. lavrakas (ed.), encyclopedia of survey research methods (pp. 149). sage. https://doi.org/10.4135/9781412963947   brearley, f. q., & cullen, w. r. (2012). providing students with formative audio feedback. bioscience education, 20(1), 22-36. https://doi.org/10.11120/beej.2012.20000022   calefato, f., & lanubile, f. (2010). communication media selection for remote interaction of ad hoc groups. in advances in computers: improving the web (pp. 271-313). elsevier. https://doi.org/10.1016/s0065-2458(10)78006-2   cann, a. (2014). engaging students with audio feedback. bioscience education, 22(1), 31-41. https://doi.org/10.11120/beej.2014.00027   carruthers, c., mccarron, b., bolan, p., devine, a., mcmahon-beattie, u., & burns, a. (2015). "i like the sound of that"--an evaluation of providing audio feedback via the virtual learning environment for summative assessment. assessment & evaluation in higher education, 40(3), 352-370. https://doi.org/10.1080/02602938.2014.917145   gould, j., & day, p. (2013). hearing you loud and clear: student perspectives of audio feedback in higher education. assessment & evaluation in higher education, 38(5), 554-566. https://doi.org/10.1080/02602938.2012.660131   gunawardena, c. n., & zittle, f. j. (1997). social presence as a predictor of satisfaction within a computer‐mediated conferencing environment. american journal of distance education, 11(3), 8-26. https://doi.org/10.1080/08923649709526970   hennessy, c., & forrester, g. (2014). developing a framework for effective audio feedback: a case study. assessment & evaluation in higher education, 39(7), 777-789. https://doi.org/10.1080/02602938.2013.870530   ice, p., curtis, r., phillips, p., & wells, j. (2007). using asynchronous audio feedback to enhance teaching presence and students' sense of community. journal of asynchronous learning networks, 11(2), 3-25. https://www.learntechlib.org/p/104047/   lunt, t., & curran, j. (2010). ‘are you listening please?’ the advantages of electronic audio feedback compared to written feedback. evaluation in higher education, 35, 759-769. https://doi.org/10.1080/02602930902977772   merry, s., & orsmond, p. (2008). students’ attitudes to and usage of academic feedback provided via audio files. bioscience education, 11(1), 1-11. https://doi.org/10.3108/beej.11.3   moore, c., & wallace, i. p. h. (2012). personalizing feedback for feed-forward opportunities utilizing audio feedback technologies for online students. international journal of e-education, e-business, e-management and e-learning, 2(1), 6-10. http://www.ijeeee.org/papers/072-z00055f00014.pdf     morris, c., & chikwa, g. (2016). audio versus written feedback: exploring learners' preference and the impact of feedback format on students' academic performance. active learning in higher education, 17(2), 125-137. https://doi.org/10.1177/1469787416637482   munro, w., & hollingworth, l. (2014). audio feedback to physiotherapy students for viva voce: how effective is "the living voice"? assessment & evaluation in higher education, 39(7), 865-878. https://doi.org/10.1080/02602938.2013.873387   parkes, m., & fletcher, p. (2017). a longitudinal, quantitative study of student attitudes towards audio feedback for assessment. assessment & evaluation in higher education, 42(7), 1046-1053. https://doi.org/10.1080/02602938.2016.1224810   parkes, m., & fletcher, p. (2019). let's talk assessment: an exploration of student perceptions of audio feedback for assessment. international journal on e-learning: corporate, government, healthcare & higher education, 18(4), 441-460. https://www.learntechlib.org/primary/p/181292/   rasi, p., & vuojärvi, h. (2018). toward personal and emotional connectivity in mobile higher education through asynchronous formative audio feedback. british journal of educational technology, 49(2), 292-304. https://doi.org/10.1111/bjet.12587   rawle, f., thuna, m., zhao, t., & kaler, m. (2018). audio feedback: student and teaching assistant perspectives on an alternative mode of feedback for written assignments. the canadian journal for the scholarship of teaching and learning, 9(2). https://doi.org/10.5206/cjsotl-rcacea.2018.2.2   rodway-dyer, s., knight, j., & dunne, e. (2011). a case study on audio feedback with geography undergraduates. journal of geography in higher education, 35(2), 217-231. https://doi.org/10.1080/03098265.2010.524197   rotheram, b. (2009). sounds good: using digital audio for evaluation feedback. world of learning, 2, 176-179. https://doi.org/10.22329/celt.v2i0.3224   sarcona, a., dirhan, d., & davidson, p. (2020). an overview of audio and written feedback from students' and instructors' perspective. educational media international, 57(1), 47-60. https://doi.org/10.1080/09523987.2020.1744853   short, j. a., williams, e., & christie, b. (1976). the social psychology of telecommunications. wiley.   university of kansas. (2021, june 28). different learning styles what teachers need to know [blog post]. https://educationonline.ku.edu/community/4-different-learning-styles-to-know   wolstencroft, p., & de main, l. (2021). ‘why didn’t you tell me that before?’ engaging undergraduate students in feedback and feedforward within uk higher education. journal of further and higher education, 45(3), 312-323. https://doi.org/10.1080/0309877x.2020.1759517   appendix copy of questionnaire   the questionnaire asked respondents to respond to five statements recording their level of agreement.    listening to the audio feedback improved my understanding of the results of my search or enquiry. strongly agree / agree / neutral / disagree / strongly disagree   hearing the librarians voice made me feel more comfortable about using the library in the future. strongly agree / agree / neutral / disagree / strongly disagree   listening to the audio feedback enhanced my experience of using the library.                strongly agree / agree / neutral / disagree / strongly disagree   overall receiving audio feedback gave me a better experience than communicating via email alone.                strongly agree / agree / neutral / disagree / strongly disagree   i was able to listen to the audio file without difficulty. strongly agree / agree / neutral / disagree / strongly disagree     evidence based library and information practice editorial responsibilities      2014. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     editor-in-chief: alison brettle   associate editor (evidence summaries): heather pretty   associate editor (articles): lorie kloda, lisl zach   associate editor (reviews): denise koufogiannakis   associate editor (classics): jon eldredge   associate editor (commentaries, using evidence in practice, news): alison brettle   production editor: michelle dunaway   editorial intern: archana deshmukh   copyeditors: richard hayman (lead copyeditor), georgianne bordner, linda ferguson, stacey penney, maria tan, mary virginia taylor, alison yeoman   indexing support: meral alakus   evidence summary   perceived and actual search behaviors may provide markers for healthcare utilization and severity of illness   a review of: white, r. w., & horvitz, e. (2014). from health search to healthcare: explorations of intention and utilization via query logs and user surveys. journal of the american medical informatics association, 21(1), 49-55. http://dx.doi/org10.1136/amiajnl-2012-001473   reviewed by: lindsay alcock head, public services health sciences library memorial university of newfoundland st. john’s, newfoundland, canada email: lalcock@mun.ca   received: 12 june 2014  accepted: 16 oct. 2014      2014 alcock. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to gain an understanding of the relationship between online health information searching behaviour and healthcare utilization.   design – survey and log data analysis.   setting – a software development campus and health information websites with servers in the united states of america.   subjects – two separate subject groups were used for this study. for the search log analysis, participants were randomly selected english-speaking users of a microsoft toolbar who had consented to provide their anonymous log data. 489 volunteers who indicated they could recall their last visit to a medical facility were invited to participate in the survey.   methods – to determine search behaviour, four months of data from 2011 were collected and analyzed from search engine logs. a unique user identifier allowed for analysis of individual search behaviour across multiple sessions, which then provided the opportunity to identify search behaviour changes over time. search queries were labelled and annotated as symptoms, serious illnesses, and benign explanation based on curated lists identified in a related study. erroneous synonymous entries were removed to increase labelling precision (e.g., astrology-related terms were removed for “cancer”). the researchers specifically noted searches signifying health utilization intent (hui). initial queries indicating hui for each user were identified to determine whether or not there were changes in search behaviour prior to and following searches indicating hui.   perceptions of motivators related to healthcare utilization (hu) were gathered through a validated, anonymous electronic survey. through fifty open and closed questions, participants were asked how they search for medical information online, how they locate medical facilities and scheduled appointments, and how their search behaviour might differ before and after hu. survey results were compared with search log data to identify and explain trends.   main results – from log data, search queries focusing on symptoms increased prior to the first indication of hui and decreased afterwards. the authors suggest that this increase may reflect a “heightened state of concern or uncertainty” (p. 51). as well, searches on relatively benign symptoms were observed to spike dramatically three weeks after the first identified hui search, reflecting what the authors suggest may be related to users having been reassured through a visit with a health professional. the increase in benign symptom searching is supported by survey data. the number of symptom-related searches is shown to correlate with the number of hui searches using pearson’s correlation coefficient (r=0.64, t(78)=14.43, p<0.001).   nearly 40% of survey participants searched online for information about a medical facility prior to a visit, and facility visits normally occurred within one (78%) or two (94%) weeks of the hui search. those visiting a facility for the first time were more likely to search for information related to the facility prior to the visit than those who had visited the facility previously. knowledge level was observed to contribute to the results as well in that searchers with self-reported low domain knowledge were not only more likely to search for a type of facility rather than a specific facility, but were also more likely to visit the facility sooner after an hui search than those with high domain knowledge. low-domain knowledge participants were also more likely to self-diagnose, more prone to alarmist behaviour related to symptom severity, and were more concerned with medical insurance.   survey respondents indicated that the focus of their searches prior to hu was primarily on symptom checking and potential diagnosis. following facility visits participants’ searches focused more on specific conditions or treatments. in addition, respondents noted that the frequency of their medical-related searching for serious conditions reduced after they had been to see their physician, indicating that the initial perceived severity of illness was potentially alarmist.   conclusion – search activity, both perceived and actual, may act as a marker to hui and as an indication that hu has occurred as well as the severity of the hu outcome. information gleaned from user logs could be used to adapt and model search engine output for users both before and after hu. further analysis on potential search engine output and geolocation is suggested to determine the full application of such data analysis.     commentary   aside from a few studies (shuyler & knight, 2003; white & horvitz, 2010; white & horvitz 2013), little research has been done to determine how to tailor search results more effectively to a user based on web searching behaviour. while a literature review is provided, the lack of disclosure regarding search strategies or resources consulted raises questions regarding the comprehensiveness of the review. this article attempts to fill the gap between perceived and actual searching behaviour and how it relates to hu. using the critical appraisal checklist (glynn, 2006), the study is determined to be valid.                 this complex study is clearly written. the subjects for the log data analysis and survey are different, and the two methodologies provide separate but related results. therefore, it is important to note that trends can only be described. that said, similar trends did indeed emerge, namely the spike in benign symptom checking after hu. the authors identify that hui intent and hu cannot be determined with certainty in user logs, which does cast a question of validity on the inferences made.   the sample size appears reasonable and the participants were randomly selected although little information is provided regarding randomization procedures and sample size power. consent was obtained from both populations. it is unclear whether both groups were similar, as demographics were not obtained for all participants. therefore, a precise comparison between the two groups’ behaviours is not possible. there may be some inherent bias with the log-user data population due to the user’s knowledge that their log data was being analyzed, which may have affected their search behaviour. the authors recognize that the survey participants may not be representative of the broader population given that they were all drawn from microsoft and were therefore, likely microsoft employees.   data collection methods were clearly described and could be replicated. the user log data collection was based on similar validated studies and the survey was tested on volunteers.   given the study limitations, the results from each data set were clearly described and reflected in the accompanying figures and tables. especially interesting were the suggested explanations provided for the fluctuations in search logs/queries and the possible correlations observed between user logs and survey responses. that two different user groups providing two different data sets are shown to exhibit similar online behaviours with respect to hu is intriguing and fodder for future research. the addition of inferential data analysis would have added insight to the study results, particularly with the addition of demographic data as independent variables.   health searching behaviour and health utilization are inextricably linked. to garner searching behaviour in order to provide more relevant and tailored information to users is a logical leap for providers of healthcare and health information. this study provides the link between perceived and actual behaviour and also the initial groundwork for further research.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154   shuyler, k. s., & knight, k. m. (2003). what are patients seeking when they turn to the internet? qualitative content analysis of questions asked by visitors to an orthopaedics web site. journal of medical internet research, 5(4), e24. http://dx.doi.org/10.2196/jmir.5.4.e24     white, r. w., & horvitz, e. (2010). web to world: predicting transitions from self-diagnosis to the pursuit of local medical assistance in web search. amia annual symposium proceedings / amia symposium. amia symposium, 2010, 882-886.   white, r., & horvitz, e. (2013). from web search to healthcare utilization: privacy-sensitive studies from mobile data. journal of the american medical informatics association, 20(1), 61-68. http://dx.doi.org/ 10.1136/amiajnl-2011-000765   evidence based library and information practice evidence based library and information practice 2011, 6.1 95 evidence based library and information practice classic patricia knapp’s landmark project to develop a plan of curriculum-integrated library instruction a review of: knapp, p. b. (1966). the monteith college library experiment. new york, ny: scarecrow press. reviewed by: carol d. howe reference librarian/assistant professor gabriele library, immaculata university immaculata, pennsylvania, united states of america email: chowe@immaculata.edu received: 22 july 2010 accepted: 16 nov. 2010 2011 howe. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to create a college-level, four-year plan of library instruction in which assignments directly relate to students’ course work. to develop tools to assess the plan’s effectiveness in improving students’ library skills and contributing to their overall academic success. design – exploratory longitudinal cohort study employing pilot library assignments, interviews, and questionnaires. setting – monteith college, one of eleven colleges at wayne state university. monteith was a small liberal arts college established in 1959 which stressed innovative teaching methods such as team-teaching, small-group discussion, and independent study (worrell, 2002). subjects – teaching faculty from all three college divisions—social sciences, natural sciences, and humanities—and students at monteith college. over the course of the study the college employed between 15 and 30 faculty members and enrolled 300 to700 students. methods – the project team consisted of project director patricia knapp, a project librarian, and a project research analyst. the team worked with the teaching faculty to develop course-related library assignments. students completed a series of assignments over several semesters as part of their course requirements. the assignment series changed over the course of the project. students who mailto:chowe@immaculata.edu� evidence based library and information practice 2011, 6.1 96 entered in the fall of 1959 or the spring of 1960 completed sequence a consisting of six assignments. students who entered in the fall of 1960 completed sequence b, six assignments that were a mixture of original and revised assignments. students who entered in the spring or fall of 1961 completed two revised library assignments. in the summer of 1961, the investigators conducted the first of two small studies. they interviewed a random sample of 21 monteith students about their experiences with the library and the required library assignments. the students also completed library performance tests such as choosing a subject heading to match a topic or deciphering an entry in a periodical index. this allowed the investigators to compare different measures of library competence and get feedback on the library assignments. in the summer of 1962, the investigators conducted a second small study of 40 monteith students. the investigators evaluated the tests and other tools used in the first study. the investigators then analyzed student and faculty data collected from sequences a, b, and c, and from the two sample studies. data included faculty interviews and feedback from student participants in the sample studies. the investigators also analyzed questionnaire data and the completed student assignments. they analyzed data using nonparametric, small sample statistics. main results – knapp’s results helped shape the final plan of instruction and assessment presented in her book-length published report the monteith college library experiment. it should be stated again that the project objective was not to implement a plan of instruction and assessment but simply to develop one. one of the most important findings was that small sample studies can effectively test the reliability of library assignments. the sample studies allowed the team to “…define and measure library competence and to identify factors associated with its achievement” (knapp, 1966, p. 17). on a different level, the project offered insight into the faculty-librarian relationship. the investigators found that faculty resisted librarian input into their courses. they also discovered that the most effective group size for developing library assignments was a small group of two to four people, but this sized group was conducive to informal meetings in which key players, often the librarian, were left out. when faculty did not share in decision-making, project morale was low. the project team reorganized and reassigned roles, and the project ran more smoothly. knapp also learned about the faculty-student relationship. knapp felt that some faculty simply passed on their knowledge to students rather than teaching students how to acquire it for themselves (worrell, 2002). she found that student enthusiasm mirrored faculty enthusiasm about library assignments. early in the project, faculty members presented library assignments to their students. the investigators discerned that both students and faculty were more amenable to the assignments when a librarian presented them and explained their purpose. knapp (2000) agreed with bruner who stated in the process of education that context is important when teaching any skill; students need to be able to relate the skills they are learning to the importance of why they are learning them (1960). finally, knapp learned that students need more than to understand library organization (such as cataloging and classification systems). students also need to understand “the organization of scholarly communication” to foster true library competence (knapp, 1966, p. 81). whereas library organization concerns itself with subject and form, the organization of scholarship “reflect[s] discipline, ‘school,’ concept, and method” (knapp, 2000, p. 10). conclusion – the monteith college library experiment ended in 1962 with a thoughtfully evidence based library and information practice 2011, 6.1 97 planned and tested program of library instruction. the final proposed program included 10 library assignments that were: of increasing complexity and aligned with the curriculum; intellectual with a focus on problem-solving; and feasible within the library’s parameters. students would complete one or more of the assignments each semester for four years as part of specific course requirements. knapp noted the program could be adapted to any college curriculum. it would require six years for implementation and assessment. this includes an initial year for planning in which teaching faculty and librarians would collaboratively develop course-related library assignments, four years for student completion of assignments, and a sixth year for assessment. knapp outlined three levels of assessment. investigators would assess the appropriateness of individual assignments through interviews and questionnaires collected from faculty and students, as well as completed student assignments. knapp outlined two ways to assess library competence. first, monteith faculty members would assess literature reviews in their subject specialties written by second semester seniors. next, faculty from other wayne state colleges would review papers from both monteith and non-monteith students to comparatively assess the students’ use of sources. knapp proposed that faculty judgment would be the most valuable measure of the relationship between library competence and overall academic success. knapp was prepared to implement her plan of instruction using all of her findings, but her proposal to move into phase two of the project was rejected by both the office of education, whose members cited economic reasons, and the council on library resources, whose members were not satisfied that faculty were invested in the idea of curriculum-integrated library instruction (worrell, 2002). commentary the idea of integrating library instruction into the curriculum was not new when knapp started her work at monteith college. according to hernon (1982), librarians have advocated building relationships with faculty and aligning instruction with coursework as far back as the late nineteenth and early twentieth centuries. knapp was the first, however, to formally study such issues with a large-scale research project (worrell, 2002). the value of the monteith college library experiment goes well beyond a plan of library instruction and assessment. many contemporary librarians can relate to and learn from the trials, errors, and frustrations knapp experienced during the monteith project. as knapp stated,“the college librarian has learned from experience that students use the library very little because very little library use is required of them in their course work” (knapp, 1959, p. 1). knapp’s honest and detailed record of findings is invaluable to any librarian embarking on developing such a plan. progress since the monteith college library experiment is heartening, and a salute to knapp’s efforts. the graduate department of education at california state university, northridge, developed a partnership between teaching faculty and librarians due in part to faculty consensus that students lacked skill at finding and evaluating information (brasley, 2008). when the nursing faculty at ball state university found their students needed basic information-seeking skills, they partnered with librarians to create a successful information literacy program (brasley, 2008). the university at albany, suny, created an information literacy program in which different skills are learned at different points in their college careers (mackey & jacobson, 2004). major accrediting bodies proposed library instruction of the sort that knapp championed. the middle states commission on higher education encourages the teaching of subject-specific information literacy skills that increase in complexity as students move evidence based library and information practice 2011, 6.1 98 through four years of college (mackey & jacobson, 2004). the monteith project also had weaknesses that cannot be ignored, not the least of which was its lack of a formal research design. while the data collected provided valuable information, much of it was subjective data from interviews and questionnaires. in his 1967 review of the monteith college library experiment, gwynn acknowledged the groundbreaking nature of knapp’s project but described it as an “untidy exploration of unknowns combined with a somewhat fumbling struggle to develop a methodology where none has existed…” (p. 299). the project also had an unrealistically ideal setting. knapp worked under the best possible circumstances—a newly formed college with many young faculty who were willing to try innovative teaching methods. few librarians will ever experience these circumstances. nevertheless, librarians should commend knapp for her landmark study. in many ways, her insight seemed ahead of her time. she accomplished 50 years ago what so many librarians strive for today—the development of a model program of library instruction integrated into a four-year curriculum. knapp also stressed the importance of an assessment plan, a popular concept in modern librarianship. the 2001 continuing education committee of ala’s library instruction round table stated it well when it said of knapp’s monteith project, “in outlining her vision of the academic librarian’s role, one sees today what we dub intensive training through course integrated instruction. thirty-five years after its initial appearance, it is illuminating to encounter these foundation stones of the discipline” (as cited in worrell, 2002, p. 185). references brasley, s. s. (2008). effective librarian and discipline faculty collaboration models for integrating information literacy into the fabric of an academic institution. new directions for teaching and learning, 114, 71-88. doi: 10.1002/tl.318 bruner, j. s. (1960). the process of education. cambridge, ma: harvard university press. gwynn, s. e. (1967). the monteith college library experiment (review). library quarterly, 3, 299-302. hernon, p. (1982). instruction in the use of academic libraries: a preliminary study of the early years as based on selective extant materials. the journal of library history, 17(1), 16-38. knapp, p. b. (1959). college teaching and the college library (acrl monograph number 23). chicago, il: american library association. / evidence based library and information practice microsoft word es_mckenna.doc evidence based library and information practice 2006, 1:1  107 evidence based library and information practice       evidence summary    evidence based research activities, interests and opportunities exist for practitioners  in all library sectors in the british isles    a review of:   mcnicol, sarah. “is research an untapped resource in the library and information profession?”  journal of librarianship and information science 36.3 (september 2004):119‐26.      reviewed by:   julie mckenna  services assessment librarian, university of regina  regina, saskatchewan, canada  e‐mail: julie.mckenna@uregina.ca    received: 04 january 2006  accepted: 13 february 2006      © 2006 mckenna. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to scope the range, nature and  challenges of present, planned and future  research by practitioners within libraries in  the british isles.    design – a series of survey questionnaires  sent by mail.      setting – public, academic, health, special  and school libraries of the british isles.    subjects – a total of 2384 questionnaires  were sent out and 334 responses were  received.  62 academic libraries, 83 health  libraries, 78 public libraries, 63 school  libraries and 48 special libraries participated  in the study.     methods – this study was undertaken in  2003 by a research team at the university of  central england.  survey questionnaires  were sent by mail to library directors in all  public library authorities, academic libraries,  health libraries and special libraries in  england, wales, scotland, northern ireland  and the republic of ireland.  in addition,  questionnaires were sent to the librarians in  all state and independent schools within a  sample of nine local education authorities.    each participant was asked to identify past  and future research issues of interest and the  barriers to research practice within their  library.  research was defined to include  work on both externally funded and in‐ house projects and examples of the types of  activities that this might include were  provided.    main results – half the respondents  reported that they had been involved in  some form of research in the past two years,   http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:1  108   with the lowest level of involvement from  school libraries and the highest in public  libraries.  generally, only the library directly  involved in conducting the research made  use of the findings and a gap in the  dissemination of results was identified  across sectors.  user surveys were the most  common form of research undertaken across  libraries and slightly fewer respondents  intended to carry out research in the coming  twelve months than had in the past (the area  most commonly mentioned was user  surveys).  information and communications  technology (ict) was an area of planned  future research in all libraries, as were user  needs and user behaviour. the most  frequently cited barriers to research activity  across all sectors were lack of time and  financial resources.  staff skills and the lack  of focus on practical problems to solve were  indicated as a barrier in health, public and  academic libraries.    libraries reported a range of common uses  for the research findings including:  informing strategic and service planning;  providing benchmarking data and  measuring the effectiveness of services;  identification of marketing and public  relations opportunities; discovery of staff  training needs; and use of the results to  demonstrate the value of libraries to  funding bodies.    conclusion – this study provides insight  into practitioner‐focused areas of research  interest and possible areas for future  investigation.  as the author reports in her  conclusion, the survey results cannot be  considered representative of the wider  population.      since research interests often overlap, a  sector wide or cross‐sectoral research  approach should be considered to allow  library staff to identify and resolve common  problems.  wide dissemination of research  results within the practitioner community  would be of benefit to all.  greater       communication between practitioner and  information science communities is also  encouraged, as these communities’ work is  mutually beneficial.    commentary    this study is a tremendous first step  towards the identification of evidence based  research activities, interests and  opportunities for practitioners in all library  sectors in the british isles.  this article  should encourage those carrying out  research practice in the british isles, since it  highlights the amount of research activity  happening within the library community.    key problems identified by this study were  the lack of recognition and poor  dissemination of practitioner‐led research  findings within the research community, as  well as the tendency for practitioners to  rarely see themselves as active members of  the research community.      further information about the design of  survey instruments, mechanics of data  collection, pre‐testing of instruments,  research ethics approval, how  confidentiality was maintained, mechanisms  for distributing the surveys, and follow‐up  with those invited to participate would help  provide greater insight into the results. it  was difficult to assess the validity or  reliability of the results without the survey  instruments.  for example, the author notes  that librarians did not report some research  activity because they did not feel it would  be of interest outside of their organization,  yet there was no indication as to how this  conclusion was reached.  this research is  timely, and access to the survey instruments  could peak interest from other researchers  who may wish to apply this study across  other geographic regions.      the survey focused upon library directors in  all sectors, with the exception of school       evidence based library and information practice 2006, 1:1  109   libraries, where it was directed to a sample  group of practitioners.  no indication was  given about the collection of demographic  information for either the institution or the  individual completing the survey.  these  demographic variables could reveal factors  that may have influenced participation and  could provide a rich context for the results  reported.  finally, throughout the article  there was a mixture of reporting about  libraries and librarians.  it was not clear  whether the survey was aimed at recording  activities of institutions or individual  practitioners, and the results were presented  as if the study had investigated both.        a further study of this subject would be of  great interest to both library practitioners  and researchers who wish to know how  evidence based practice is being carried out  in the british isles and beyond.  a  comprehensive survey should be  undertaken and reported in a manner that  assures the reader of a valid and authentic  research approach.  further study with  recommended strategies for improving the  perception of practitioners as a part of the  research community, the dissemination of  research within the practitioner community  and communication of research and results  between the academic and practitioner  communities would be of value.  microsoft word es_stephens.doc evidence based library and information practice 2007, 2:2  94 evidence based library and information practice       evidence summary    information literacy skills:  teacher and student viewpoints    a review of:  herring, james e. ʺa critical investigation of students’ and teachers’ view of the use of  information literacy skills in school assignments.ʺ school library media research 9  (2006). 14 may 2007  <http://ala.org/ala/aasl/aaslpubsandjournals/slmrb/slmrcontents/volume9/informationlite racy.htm>.    reviewed by:   julie stephens  media specialist, calhoun educational complex  calhoun, georgia, united states of america  e‐mail:  stephensj@calhounschools.org      received: 28 february 2007  accepted: 25 april 2007      © 2007 stephens. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to examine student and teacher  views of information literacy skills in school  assignments in order to determine:  1) to  what extent did students value the use of a  research model booklet (plus)?  2) how  confident were the students about doing a  good assignment and did the plus booklet  affect their confidence?  3)what benefits and  limitations did students identify from  individual brainstorming and concept  mapping in relation to learning more about  their topic and producing a good  assignment?  4) to what extent did students  see value in doing preliminary reading to  revise their initial keywords and concept  maps?  5) what reading and note‐taking  strategies did students adopt when using  print and electronic resources?  6) to what  extent (and why) did students prefer to use  electronic rather than print resources?    7) what are the implications for teachers  and school library media specialists (slms)?    design – qualitative, action research;  collaborative inquiry.    setting – ripon grammar school, yorkshire,  united kingdom (high school/co‐ educational).    subjects – fifty‐two students in the second  year of high school (year 8) enrolled in a  science class studying sound technology; the  school library media specialist (slms);  science teachers.     methods – students in this study had  previously been taught a variety of  http://ala.org/ala/aasl/aaslpubsandjournals/slmrb/slmrcontents/volume9/informationliteracy.htm http://ala.org/ala/aasl/aaslpubsandjournals/slmrb/slmrcontents/volume9/informationliteracy.htm mailto:stephensj@calhounschools.org http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:2  95 information skills and had been introduced  to a research model called plus (purpose,  location, use, self‐evaluation).  students  were given a plus model booklet, were  required to select a topic in the area of  sound technology, and were expected to do  brainstorming and concept mapping and to  produce a 600‐word essay.  after the  assignment was completed, three methods  of data collection were employed to  determine students’ and teachers’ views:   1) post‐assignment questionnaire  2) group  interviews with students and teachers  3)  semi‐structured interview with the school  librarian.    main results – responses indicated that  students were “mostly” satisfied with the  use of the plus model, although there were  18 students who did not respond to  questions regarding the use of the booklet.   it was also clear from the questionnaire that  the majority of the students did not feel  confident in their abilities to produce a  satisfactory assignment prior to beginning  the research; however, 48% of the students  indicated that the plus booklet made them  more confident.  a comparable number of  students said the booklet had no effect on  their confidence and one student said it  made them less confident.      students responded very positively about  the use of individual brainstorming and  concept mapping as a way to organize and  focus on their topic.  (there was a split  between those who felt a written concept  map was useful and those who felt a mental  concept map was just as helpful.)  the  majority of students felt that group  brainstorming was helpful, while a few  indicated the behavior of other students  during group brainstorming was a  hindrance.      questions about preliminary reading were  not open‐ended but were multiple choice.   there was no response to indicate whether  the preliminary reading was helpful or not,  but rather how it was helpful.  most students  indicated it had helped to identify the right  keywords for further research and it helped  them in finding the right resources.      students were questioned on the format of  their note‐taking.  sixty‐five per cent  preferred to hand write their notes in a  notebook; fifteen per cent preferred to take  notes electronically in some type of word  processing program; twelve per cent  preferred to cut and paste into a word  processor; and eight per cent preferred  “other methods.”  note‐taking styles ranged  from bulleted lists to spider diagrams, to  using headings with categorized notes.     when asked to indicate the percentage of  information derived from web sites versus  information from books and journals,  responses showed that over 65.5 % of the  information came from web sites while only  35.5 % came from print material.  when  asked why, students responded that web  sites were easier to read, more useful, easier  to use, more interesting, faster to use, had  more pictures, and could be accessed from  home.  students who indicated a preference  for print material identified reasons such as  not trusting web sites and slow access.    interviews with the teachers and the school  librarian indicated agreement that the  benefits of using the plus model were: it  kept students from rushing into the  assignment; it allowed for better thinking  and analytical skills; it helped students  improve their note‐taking skills; and  students were transferring skills.    in the interviews, teachers were very  positive concerning the collaborative efforts  of the school librarian. responses from the  slms’s focused on how to improve student  use of resources, and concern that teachers  and librarians should engage students in  activities that foster critical thinking.  evidence based library and information practice 2007, 2:2  96 conclusion – the study suggested that  students are capable of reflecting on their  use of information literacy skills.  it also  indicated that students saw the value in  brainstorming, concept mapping, and the  use of a research model such as the plus  booklet, even though these strategies did  not necessarily suit the learning styles of all  students.  there was overwhelming  evidence that students prefer electronic  resources over print resources and reasons  why they prefer electronic resources were  articulated.  implications for teachers and  school library media specialists include:  collaboration, the importance of seeking and  analyzing student feedback, examining  transfer of information literacy skills across  subjects, and exploring student use of print  and electronic resources.    commentary      the topic of this research is significant and  of value to anyone working with students,  such as teachers and school librarians. all  charts, graphs and questionnaires were  included and were appropriate and clear.  there was not much explanation of the  plus model beyond a simple flowchart and  a definition of each component (an example  of the booklet the students had “in hand”  would have been helpful).  an extensive  review of literature was included.  this  particular study was very limited in that it  only involved a small number of students in  one age group, at one school involved with  one assignment.  however, student  responses were probably very typical and  could have been the same in a larger, more  varied group of subjects. herring was very  limited in his explanation of the content of  the project (he did note that his research  design was qualitative rather than  quantitative, but more description of the  actual process would have been beneficial.)  little information was given as to how  much time was spent on the project, how  much actual instruction in research skills  was delivered by the teachers and/or media  specialist, and how the results of this one  assignment compared to previous  assignments.  future research could  examine the use of information literacy  skills across a range of ages and curriculum  areas and could compare the use of the  plus model to research done without the  model or with an alternative research model.   the outcomes of this research were clearly  stated.  these findings indicate that some  students definitely benefit from the use of  research models such as the plus model  used in this study.   the fact the students  rely so heavily on, and prefer electronic  resources, also has definite implications for  the school media specialist, the technology  teacher and the classroom teacher as they  design and implement learning experiences  and assessments.    microsoft word news_eblip5.doc evidence based library and information practice 2007, 2:4  122 evidence based library and information practice     news    eblip5 website up and running        © 2007 glynn. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    the university library at karolinska  institutet would like to welcome you to the  5th evidence based library and information  practice conference in stockholm, sweden,  june 29th ‐ july 3rd 2009. further details,  including information about the venue and  the conference organizers, can be found by  visiting the new eblip5 website at  http://eblip5.kib.ki.se.   http://creativecommons.org/licenses/by/2.0 http://eblip5.kib.ki.se evidence based library and information practice evidence based library and information practice 2010, 5.4 39 evidence based library and information practice article letting students take the lead: a user-centred approach to evaluating subject guides kimberley hintz humanities & social sciences librarian university of british columbia library vancouver, british columbia email: kimberley.hintz@ubc.ca paula farrar art + architecture + planning librarian university of british columbia library vancouver, british columbia email: paula.farrar@ubc.ca shirin eshghi japanese language librarian university of british columbia library vancouver, british columbia email: shirin.eshghi@ubc.ca barbara sobol learning services librarian (research) university of british columbia library kelowna, british columbia email: barbara.sobol@ubc.ca jo-anne naslund education librarian university of british columbia library vancouver, british columbia email: joanne.naslund@ubc.ca teresa lee health sciences librarian university of british columbia library vancouver, british columbia email: teresa.lee@ubc.ca tara stephens circle 2010 olympics and paralympics librarian university of british columbia library vancouver, british columbia email: tara.stephens@ubc.ca aleha mccauley community business services librarian irving k. barber learning centre at the university of british columbia vancouver, british columbia email: aleha.mcauley@ubc.ca received: 21 june 2010 accepted: 23 oct. 2010 2010 hintz et al. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – what do students need and want from library subject guides? options such as web 2.0 enhancement are now available to librarians creating subject-specific web pages. librarians may be eager to implement these new tools, but are such add-ons a priority for mailto:kimberley.hintz@ubc.ca� mailto:paula.farrar@ubc.ca� mailto:shirin.eshghi@ubc.ca� mailto:barbara.sobol@ubc.ca� mailto:joanne.naslund@ubc.ca� mailto:teresa.lee@ubc.ca� mailto:tara.stephens@ubc.ca� mailto:aleha.mcauley@ubc.ca� evidence based library and information practice 2010, 5.4 40 students? this paper aims to start a dialogue on this issue by presenting the findings of the university of british columbia (ubc) library’s subject guides working group (sgwg), which was tasked with assessing current library subject guides in order to make recommendations for the update and future development of ubc library subject guides. methods – the working group solicited feedback through a questionnaire that was distributed to undergraduate and graduate students from a variety of disciplines at ubc. the questionnaire included an evaluation of ubc subject guides, as well as guides from other academic libraries that used various platforms such as libguides and subjectsplus. results – respondents to the student questionnaire indicated that a simple and clean layout was of primary importance. students also desired succinct annotations to resources and limited page scrolling. meanwhile, few students identified web 2.0 features such as rating systems and discussion forums as being important for their needs. the working group used the questionnaire data to create a “top ten” list of student recommendations. conclusions – the “top ten” list of student recommendations was combined with stakeholder feedback from faculty, liaison librarians and library systems and information technology representatives to create the sgwg’s final recommendation for subject guide revision and enhancement. for the sgwg these findings called into question the necessity of web 2.0 technologies within subject guide pages and highlighted the need for further research on the topic of subject guide usability and effectiveness. introduction academic libraries often employ the term “subject guides” to describe web pages created by liaison librarians that include subjectspecific content meant to assist students or faculty in their research. subject guides often contain links to online resources, information about print resources, research tips, citation style guides, as well as other information relevant to a particular research area. subject guides may also potentially include web 2.0 interactive multimedia tools such as video tutorials, tagging and user polls, and incorporate blogs and social networking applications. although the content of a subject guide is of primary concern to liaison librarians, as with any website or online resource, the presentation of that content both positively and negatively impacts a student’s ability to access the information at their point of need. at the university of british columbia (ubc) library, subject guides have been considered an important element of liaison librarian work, and a critical means by which the library communicates information about its collections and services to its users. despite this perceived importance, the layout and functionality of ubc library subject guides had not been reviewed for at least 10 years, and many of the emerging web 2.0 technologies were not readily implementable within the existing platform. in the spring of 2008, ubc librarians expressed a desire to refresh the look of the library’s subject guides, improve their functionality and take advantage of recent web 2.0 developments that were key features of commercial subject guide platforms such as libguides. a libguides review working group was formed to assess libguides and its suitability for use in the ubc library system. this working group quickly came to the conclusion that in order to make an informed decision on the future direction of the subject guides, it was necessary to compare a variety of subject guide platforms, and more importantly, consult with our key stakeholders. believing that the assessment should be user-driven, the libguides review working group was evidence based library and information practice 2010, 5.4 41 disbanded, and the subject guides working group (sgwg) was formed. in july 2008, the newly formed sgwg developed a revised task list: review the current subject guide literature, investigate the various platforms available, gather feedback from the key stakeholders, and, finally, make a recommendation for the future development of ubc library’s subject guides. at the initial meeting of the sgwg much of the discussion revolved around how to gather feedback from key stakeholders. it was decided that input from liaison librarians and systems staff should be solicited through invitational meetings. however, the group determined that a more wide-reaching and evidence based approach was necessary for the student feedback component, as students have been and will continue to be the primary target audience for subject guides. with this in mind, the sgwg decided that a questionnaire distributed to a large sampling of students from various disciplines would be appropriate. using questionnaire data in conjunction with a "wish list" of needs from the liaison librarian feedback sessions, the sgwg met with systems staff to explore platform options, including proprietary subject guide software systems, open source tools and in-house solutions. at the end of this process the sgwg proposed a series of recommendations for the future directions of ubc library’s subject guides. these recommendations addressed the needs of library stakeholders, while at the same time providing the best possible resource for library users over the long term. this article provides a review of the literature, an explanation of the student questionnaire methodology, and an analysis and discussion of the data, including a “top ten” list of priorities. how this information was used towards the sgwg’s final recommendations is then discussed. literature review the research literature on subject guides can be divided into three prominent categories that demonstrate those areas which have received the most attention: first, the history or evolving purpose of subject guides; second, the challenges of subject guides, both technical and administrative; third, a focus on subject guide use. finally, the topic of web 2.0 technologies has emerged as a trend that crosses these categories and is something our research addresses. the history or purpose of subject guides is well covered within the literature. smith (2008) provides a comprehensive overview of the evolution of guides from the late nineteenth-century to the present, including descriptions of the many iterations subject guides have taken along the way. the origin of subject guides lies in descriptive bibliographies designed to facilitate the use of resources (smith, 2008, pp. 512-514). tchangalova and feigley (2008) list a plethora of terms for what basically falls into the category of subject guides: research guides, research tools, pathfinders, electronic library guides, e-guides, webliographies, subject portals, etc. while the style of delivery has changed substantially, the basic purpose of providing a starting point for the researcher remains unchanged. much of the literature on subject guides concerns the challenges that they pose to librarians. issues of creation (policy, technology), maintenance (division of labour, technology, technical expertise) and overall administration are common in the literature (darby, 2006; jackson, blackburn, & mcdonald, 2007; prentice, 2009; staley, 2007; wales, 2005). a trend within this category of the literature concerns the technical requirements of staff to create and maintain subject guides (buczynski, 2009). this issue is tied to the frequency with which guides need to be updated, and the common practice in most libraries of migrating print guides to the online environment through static webpages. recent technological developments, such as server side include (ssi) statements, allow for new or revised content to be “pushed” to web pages alleviating the need for manual updates. as these types of technologies are evidence based library and information practice 2010, 5.4 42 implemented in libraries, many of the technical concerns surrounding the updating of subject guides should be mitigated (buczynski, 2009; goans 2006; greene, 2008; northrup & ashmore, 2006). the administering of guides also emerges as a dominant trend in the literature with a focus on issues regarding standardization (tchangalova & feigley, 2008), time and consistency (jackson, blackburn, & mcdonald, 2007, p. 20, p. 33). the emergence of commercial and open source subject guide products such as libguides and library à la carte, respectively, directly address these types of concerns as they streamline the technical and administrative aspects of subject guides (griggs 2009; montgomery, 2009; smith, 2008). given that subject guides are innately userfocused, the literature indicates that subject guide use is less studied than one might expect. this sentiment is echoed by staley, who describes the state of research on subject guide usage as “still at an exploratory stage” (staley 2007, p. 122). an overview of the literature reveals that three types of questions are often asked: how are subject guides used? who uses them? and, are they useful? staley focuses on “actual use and perceived usefulness of subject guides” amongst specific groups of students (staley, 2007, p. 130). griggs details how library à la carte was created with user input integrated into its development (griggs, 2009, p. 14). courtois, higgins and kapur (2005) surveyed user satisfaction with subject guides. reeb and gibbons (2004) recommend delivering subject guide content at the course level to better match students’ mental models of information. these studies on subject guide use are insightful, but there is clearly more work to be done on these issues. woven through each of these dominant categories of subject guide research is the recurrent theme of incorporating emerging web 2.0 technologies within libraries (kellam, cox, & winkler, 2009, p. 350). various approaches to integrating web 2.0 features have been explored and these are frequently referred to in the literature as “subject guide 2.0”. while no official definition of this term exists, yang (2009) takes her meaning from the “unspoken consensus that subject guides 2.0 are those created with web 2.0 technologies” (p. 91). however, yang is quick to observe that other writers on the subject, such as meredith farkas and ellyssa kroski, place more emphasis on usage and interactivity than on the technologies themselves (p. 91). the diverse means of distributing librarianproduced content, as well as facilitating usergenerated content, therefore, take precedence in much of the subject guide 2.0 literature. yang (2009) provides an extensive catalogue of subject guide 2.0 characteristics, ranging from multimedia features to statistics reporting, and a comparison chart that lists those features included in common subject guide content management systems. allan (2009) offers a two page guide to increasing the “functionality and depth” of your subject guide with web 2.0. others advocate using web 2.0 tools as the very basis of subject guides (dodson, 2008; jackson, blackburn, & mcdonald, 2007). the appeal of platforms such as libguides is that these web 2.0 features are built-in (judd, 2009; moses, & richard, 2008). web 2.0 technologies are extensively discussed in the professional literature and the assumption is that by adding web 2.0 features to library resources, specifically subject guides, there will be a perceptible improvement viewed as such by students. curious about the validity of these assumptions, we asked students to assess an array of subject guides, some with web 2.0 features, and some without. the purpose of our questionnaire, therefore, was to find out what students really want from subject guides. methods a questionnaire was used to gather data from students as to what subject guide features, content, and design would be most helpful in serving their research and study needs (see appendix a). the sgwg obtained a minimal risk certificate of approval from the evidence based library and information practice 2010, 5.4 43 behavioural research ethics board at ubc’s office of research services before proceeding with the project. prior to distributing the questionnaire, it was piloted in two trial sessions: one with six participants and the other with four. these trial sessions provided an opportunity to test the length and format of the questionnaire, as well as to determine venue requirements. a total of 55 questionnaires were administered in the computer labs of six branch libraries during winter session term ii. the six branches chosen serve different disciplines (humanities, social sciences, and sciences) and user groups (ubc vancouver and ubc okanagan students). handbill flyers in each branch were used to advertise the study and $5 ubc ams food outlets gift certificates were offered as incentives for participants. these sessions were conducted during lunch hour and early afternoon time periods to take advantage of peak student traffic in the branch libraries. most respondents completed the questionnaire in twenty to thirty minutes. completion of the questionnaire required respondents to read a consent form and anonymously record their personal background information including: • student status (undergraduate, graduate, unclassified) • number of years attending ubc (0-5+) • program, department, or faculty • frequency of use of the ubc library website (never, sometimes, frequently, always). the first section of the questionnaire asked students the following open-ended question: if you could create a webpage that would help you with your research or course work, what are the kinds of things that you would put on it? the respondents could list, sketch, or draw what they would like to see on such a web page. the second section of the questionnaire involved three parts. the first part asked respondents to examine three sample subject guides selected from different libraries. to ensure that participants were exposed to a wide variety of features available on different subject guides while giving each participant a manageable number of samples to view in the time allotted, the subject guides were grouped and administered as two sets (28 of set a and 27 of set b). each set included the following: 1. a ubc subject guide (static web page) 2. a libguides subject guide 3. a subject guide created using open source software (eg. library à la carte or subjectsplus) the second part asked students to record their comments on printed copies of the guides. the third part asked students to rate each subject guide using a likert scale of 1-5 on the following: • comprehension (from 1-very unclear to 5-very clear) • visual appearance (from 1-boring to 5-gets my interest) • content (from 1-not useful to 5-very useful). the final question in this section had students select which subject guide they would most likely use if available in their subject area. they were also asked to provide a reason for their choice. the purpose of the data analysis of the completed questionnaires was to identify common elements in content, design and functionality, not to compare specific platforms. to decrease potential bias in the analysis of the qualitative components of the questionnaires, two members of the sgwg coded the data independently. the results were then compared and discrepancies in coding between the reviewers were discussed until reviewers came to a consensus. in the end a total of 40 separate codes were used to quantify the data. findings demographics from the 55 completed questionnaires we learned that undergraduate students represented 78% of the participants and the remaining 22% were graduate students. the evidence based library and information practice 2010, 5.4 44 surveyed population showed a relatively even distribution in terms of years spent at ubc, with 9% at less than 1 year; 28% at 1 year; 16% at 2 years; 16% at 3 years; 20% at 4 years; and 11% at 5 years. participants were asked to indicate their subject areas and again, a relatively even distribution was achieved with 42% self-identifying as students in the arts, 29% from the sciences, and 20% from education. a significant percentage of the students (46%) used the library website at least occasionally and only 9% reported that they never consulted the website for research or coursework. student preferences student feedback on the open-ended question ranged from scribbled keywords to more elaborate commentary about specific features. for example, respondent #19 commented: “highlight seminal articles in the field of research. i.e. the most cited, most discussed, on which other papers follow.” although students were encouraged to list or sketch ideas, nearly all chose textual description rather than visual representation. given the similarity between comments in this section and other narrative commentary provided in parts two and three of the second section, the reviewers decided to use one coding scheme to analyze all sections. the second section of the questionnaire asked participants to evaluate three separate subject guides. one of the most telling findings of the study showed that when asked which guide they would use if it were available in their subject area, 20 of the 55 respondents (36%) chose a ubc guide, a much higher number than expected. these results suggest that the ubc subject guides were not as outdated and unusable as we had assumed. we reviewed the responses of the 35 participants who chose other guides as their preferred format and found an interesting correlation. when respondents were asked to compare three subject guides based on comprehension, visual appearance, and content using the 1-5 likert scale, ubc’s guides rated high for content and comprehension but significantly low on visual appeal (fig. 1). 0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 ubc guide open source libguides ubc guide 3.97 2.58 4.02 open source 3.82 3.02 3.58 libguides 4.04 3.84 4.11 comprehension visual appearance content fig. 1. the mean values for comprehension, visual appearance and content are represented, indicating ubc’s guide received significantly lower marks for visual appearance. evidence based library and information practice 2010, 5.4 45 when asked to choose their favourite subject guide, participants generally chose the one they had ranked as highest in visual appearance. this relationship showed that while content and comprehension are important, visual appeal can be a deciding factor in determining which guides students would most likely use. to learn more about what constitutes visual appeal, we analyzed the commentary provided and responses to other parts of the questionnaire. the data analysis used a total of 40 codes to represent the general themes expressed in the narrative comments provided by participants. while all of the data was useful, importance was placed on ideas or features that received numerous comments. in the end, the working group decided to focus on the ten codes that had received the most comments—the resulting “top ten” list would form the basis for future development recommendations (fig. 2). student priorities in first place, by a large margin, students mentioned the importance of a simple or clean layout. a total of 47 of the 55 respondents made comments on the simple structure, layout and use of white space on the guide. as an example, when questionnaire respondent # 30 (a senior undergraduate student) was asked to choose his/her favourite guide and tell us why, the response was: “the format is very clear and neat. the first two webpages just make me feel dizzy. too many words on it – very messy.” the overwhelmingly positive response for this characteristic showed that basic principles of good web design must be followed. analysis also revealed students’ desire for more succinct, jargon-free resource descriptions. we also noted from our results that navigation was extremely important as three of the top ten features included the use of tabs, section headings and keeping pages to a manageable page length to limit the amount of scrolling—all pointing to common usability considerations. fig. 2. “top ten” list of student priorities indicates the number of times (frequency) a particular element was given positive feedback on the questionnaires. 47 31 31 23 22 22 21 19 18 16 codes fr eq ue nc y simple layout/clean annotations search feature embedded instruction content easy to understand librarian contact info tabs citation info section headings length evidence based library and information practice 2010, 5.4 46 at the same time, participants noted that short annotations describing resources were highly desirable. participants on 31 questionnaires responded positively to this feature. additional feedback regarding length of the page and length of annotations suggested that students are not willing to read an annotation more than a sentence or two long. participants also expressed a desire to see embedded instruction linked to resources in the form of brief tutorials. students wanted direction and help with these resources, indicating that they did not want to simply be pointed to a resource; they wanted to be told how best to make use of it. students also liked seeing a search box, the inclusion of librarian contact information, and help with citing their sources on the guides. they wanted immediate access to search results and assistance in finding someone to help them with their questions. their interest in having citation style information available on the subject guide is a reflection of students’ desire for a resource that provides guidance in each stage of the research process. this theme was also evident in their comments regarding embedded instruction features, including chat and online tutorials, each of which was coded separately. interestingly, while several codes were used to record comments about web 2.0 elements, none of them appeared in the “top ten” list. students appeared skeptical about rating systems, discussion forums, student recommendations, and they showed little interest in personalization features. discussion while the results of the student questionnaires provided valuable insight into the preferences and opinions of our primary stakeholders, the limitations of the study must be addressed. constraints in terms of staff time, resources, and recruiting participants resulted in a small sample size that could not be considered sufficiently representative of the diverse student population at ubc. however, the sgwg employed several strategies to increase participation from students across disciplines and levels of study: • six questionnaire sessions were conducted in six of the library’s busiest 21 branches, including the ubc okanagan library and in the irving k. barber learning centre, which houses a café, computer lab, and several academic support departments including the library. • sessions were conducted on six separate dates and at 3 separate times (11am, 12pm or by appointment) to encourage participation from students with disparate schedules and, therefore, different programs and levels of study. despite these measures to capture a variety of responses across the student body, it was not possible to guarantee accurate representation from all student groups, nor was it possible to determine whether participants were more experienced than the general student population as library users, and therefore more familiar with ubc library and other subject guides. another possible instance of familiarity introducing bias is the fact that though participants were encouraged to evaluate the merit of each guide independently of its institutional context, students were likely to be best acquainted with the ubc guides. with this in mind, the sgwg members conducting questionnaire sessions were provided with a standardized script instructing participants to consider the overall usefulness of the subject guide in terms of style, format and types of content rather than who developed the guide or its applicability to their area of study (see appendix b). given the limitations of our study, the data collected from the questionnaires were interpreted not as a definitive representation of what our students need or want from subject guides, but rather as a way to broaden our understanding of what elements appeal to students and why. not surprisingly, the questionnaire results confirm the importance of simplicity and clarity in the design and content of subject guides. students evidence based library and information practice 2010, 5.4 47 consistently emphasized a need for a clean and simple layout, with tabs, section headings, and other clear navigational signposts. respondents also expressed a desire for easyto-understand content that is comprehensible both in terms of navigational structure and clarity of language. additionally they emphasized a desire for careful selection over quantity. generally, the findings from our study confirm strutin's intuitions that students' compulsion to rely on google and wikipedia as starting points for research has to do with "speed and simplicity"; "familiarity and consistency"; and "know[ing] how to navigate quickly" (2008). on the other hand, our findings suggest that we have yet to fully understand users' needs with respect to subject guides. in the current literature, the inclusion of web 2.0 technologies in subject guides tends to be viewed uncritically as a positive development. yang noted, for example, that user-generated content is needed: "collaboration among faculty, students, and librarians is necessary in creating and maintaining subject guides; therefore subject guide 2.0 must have the capability to receive and incorporate input from users" (2009, p. 92). further, yang suggests that in being able to enter into online student territory, libguides, the most complete cms in terms of web 2.0 functionality, can only be beneficial: "by integrating with facebook, delicious.com, and twitter, libguides advertises and distributes the contents of the subject guides to internet social networking sites, places that students frequent" (p. 94). however, there appears to be little evidence to support the efficacy of this kind of exposure and integration. in fact, our findings suggest that students do not welcome or even understand all the web 2.0 features that can possibly be embedded into subject guides. despite our small sample, data from the questionnaires suggests that web 2.0 features need to be examined more critically. a pattern that appeared to emerge from our findings was the conceptualization of web 2.0 features into two broad categories: 1) features that promote peer-to-peer interaction and learning (student recommendations, forums and personalization/customization) and 2) features that enable students largely to receive authoritative advice (librarian chat window, embedded tutorials/handouts giving advice on how to use resources). although we initially suspected that collaborative elements such as forums and popularity functions would rank highly as enhanced features, students were largely unreceptive to these tools. in fact, responses often indicated that students found these features confusing. the following comments were received regarding the rating system functionality: “don’t understand its intention.”, “what do these mean? who rates them?”, and “stars lead researchers in certain ways which might be misleading.” both the quantitative data and the subtext of many of the participant comments revealed that students are more interested in finding authoritative information from accepted experts (librarians and faculty members) rather than in using subject guides as a site for their own knowledge production and interaction with peers. in short, students come to subject guides expecting to be firmly guided towards the materials and conventions of accepted scholarly practice. future research, therefore, needs to move beyond simply advocating for the integration of web 2.0 features into subject guides to a more critical and selective approach for adopting features and functionalities based on appropriate evaluation. these findings provided the sgwg with much to consider in revising ubc library’s subject guides. the working group’s recommendation to implement a locallydeveloped content management system (cms) rather than libguides, a popular commercial product, emerged directly from and reflects student feedback, internal contexts and priorities, as well as the needs expressed by librarians as primary content creators. evidence based library and information practice 2010, 5.4 48 conclusions the subject guide working group began this study with the goal of understanding what our primary and secondary stakeholders needed from subject guides with regard to both content and mode of delivery. in addition to a review of the literature and available subject guide platforms, we solicited feedback from librarians and library staff and developed a questionnaire for students. the insights gained from these questionnaires proved invaluable to the sgwg as we investigated future directions for ubc subject guides. while many of the comments and viewpoints expressed by students reinforced our own perceptions of what needed to be improved or revised on our subject guides, we could not adequately predict which available features would appeal most to students and why. once these findings were compared to the needs and preferences articulated by librarians and library staff, it became evident that an inhouse subject guide cms could be built with the features and functionality outlined in our “top ten” list. moreover, the potential for greater control over customization and integration with existing local systems provided by an in-house cms added an additional level of appeal over a commercial or open-source product despite the extensive list of subject guide 2.0 features currently available. yang’s comprehensive checklist includes an impressive selection of options for tools and functionality such as multimedia, multi-formats, ease of use, global change, search boxes, link checking, integration with social bookmarking sites, rss feed, tagging, interactivity and real time chat, user input, blogs, wikis, and statistics reporting (2009, p. 97). despite the vast possibilities for content production and delivery offered by these features, we discovered that the students who completed the questionnaire favoured those that enabled the provision of authoritative advice and instruction, such as librarian contact information and embedded tutorials. this unexpected response raised a number of questions about the benefits of web 2.0 features in subject guides and the need for a better understanding of appropriate contexts in which these features may best be used. there are an increasing number of platforms that are capable of integrating various web 2.0 features into library subject guides. we would like to see more studies that assess the effectiveness of subject guide 2.0 features in academic libraries from users’ perspectives. rigorous research is needed to help us learn more about the usage of subject guides, the resources and features utilized within them, and to better understand how to guide our users to resources that enable them to meet their research goals. as libraries re-evaluate and recreate subject guides to incorporate new technologies —many for the first time since transitioning to online from static print —we encourage decision makers to carefully consider local users’ perspectives, goals, needs, and real usage of subject guides before investing additional resources, money, and time into the direction of subject guide 2.0. references allan, c. (2009). evolve your research guides into the web 2.0 ecosystem. texas library journal, 85(1), 20-21. buczynski, j. a. (2009). online web development platforms enable all reference staff to work on subject guides. internet reference services quarterly, 14(3), 61-66. courtois, m. p., higgins, m. e., & kapur, a. (2005). was this guide helpful? users' perceptions of subject guides. reference services review, 33(2), 188-196. darby, a. (2006). implementing an open source application in a college library: ecu's pirate source. college & undergraduate libraries, 13(1), 41-52. farkas, m. (2007). subject guide 2.0. american libraries, 38(5), 33. evidence based library and information practice 2010, 5.4 49 goans, d., leach, g., & & vogel, t. (2006). beyond html: developing and reimagining library web guides in a content management system. library hi tech, 24(1), 29-53. greene, a. (2008). managing subject guides with sql server and asp.net. library hi tech, 26(2), 213-231. griggs, k. (2009). library information made to order: an open source project built for and with librarians. computers in libraries, 29(2), 12-47. jackson, m., blackburn, j. d., & mcdonald, r. h. (2007). media wiki open-source software as infrastructure for electronic resources outreach. reference librarian, 48(1), 19-36. judd, c., & montgomery, n. m. (2009). libguides and librarians: connecting content and community. kentucky libraries, 73(3), 14-17. kellam, l. m., cox, r., & winkler, h. (2009). hacking blackboard: customizing access to library resources through the blackboard course management system. journal of web librarianship, 3(4), 349-363. moses, d., & richard, j. (2008). solutions for subject guides. partnership: the canadian journal of library & information practice & research, 3(2), 1-9. northrup, l. a., & ashmore, b. (2006). creating easy to update subject guides without using a database. college & undergraduate libraries, 13(1), 53-57. prentice, k., gaines, j., & levy, l. (2009). new "starting points" for resources by subject. medical reference services quarterly, 28(1), 88-97. reeb, b. &. gibbons, s. (2004). students, librarians, and subject guides: improving a poor rate of return. portal: libraries and the academy, 4(1), 123-130. smith, m. m. (2008). 21st century readers' aids: past history and future directions. journal of web librarianship, 2(4), 511-523. staley, s. m. (2007). academic subject guides: a case study of use at san jose state university. college & research libraries, 68(2), 119-139. strutin, m. (2008, fall). making research guides more useful and more well used. issues in science & technology librarianship, 55. tchangalova, n., & feigley, a. (2008). subject guides: putting a new spin on an old concept. electronic journal of academic & special librarianship, 9(3), 1-1. wales, t. (2005). library subject guides: a content management case study at the open university, uk. program: electronic library & information systems, 39(2), 112121. wrosch, j. (2007, june 28). open source software options for any library. mla forum, 5(3). yang, s. q. (2009). subject guide 2.0: a dream or reality? journal of library and information science, 35(1), 90-98. evidence based library and information practice 2010, 5.4 50 appendix a student questionnaire ubc library subject guides questionnaire a a) background information 1. student status: undergraduate graduate unclassified 2. how many years have you been at ubc? 0 1 2 3 4 5+ 3. what program, department or faculty are you in? ________________________ 4. how often do you use the ubc library website for your research or course work? never sometimes frequently always b) questions 1. [estimated time: 3-5 minutes] if you could create a webpage that would help you with your research or course work what are the kinds of things that you would put on it? (hint: list ideas and/or sketch what the page would look like). 2. [estimated time: 10-15 minutes] a) take a few minutes to look at the three sample subject guides listed in group a at: http://www.library.ubc.ca/finearts/sgwg/samples.html evidence based library and information practice 2010, 5.4 51 b) using the printed copies of the guides, let us know what you like or dislike about each guide. feel free to highlight, circle, mark up, make notes, etc... c) please rate each subject guide by circling the appropriate number ranking: ------------------------------------------------------------------------------------------------ 1. university of british columbia – environmental studies/sciences comprehension: very unclear 1 2 3 4 5 very clear visual appearance: boring 1 2 3 4 5gets my interest content: not useful 1 2 3 4 5 very useful ------------------------------------------------------------------------------------------------ 2. university of victoria – history comprehension: very unclear 1 2 3 4 5 very clear visual appearance: boring 1 2 3 4 5 gets my interest content: not useful 1 2 3 4 5 very useful ------------------------------------------------------------------------------------------------ 3. dalhousie university – biology comprehension: very unclear 1 2 3 4 5 very clear visual appearance: boring 1 2 3 4 5 gets my interest content: not useful 1 2 3 4 5 very useful ------------------------------------------------------------------------------------------------ d) of the subject guides you have just reviewed, which one would you be most likely to use, if available in your subject area? 1 2 3 none of the above please tell us why you chose this answer: appendix b standardized script for administering the student questionnaire introduction script for student questionnaires preamble thanks for coming today to participate in this ubc library research project. my name is ____________ and joining me today are librarians from across campus: (introduce everyone present) the reason we’ve invited you here is to get your input on how to make the library’s subject guides better. subject guides are webpages created by the subject librarians to guide you in your research or course work. evidence based library and information practice 2010, 5.4 52 before we get started, we’d like to ask you to read the consent form. (2 min.) now you have the option to either complete the questionnaire here or take it with you, complete it and return it to _____________ at __________ library within the next three days (date: __________). if you would like to continue now, please fill out section a of the form. (2 min.) ________________________________________________________________________ brainstorming the session today consists of two main parts. in this first part, we’d like you to brainstorm a bit about what you would consider an ideal subject guide (and this can be about any research topic or subject). please sketch out or jot down how you would envision this guide. we’ll give you about 5 minutes to do this. (6 min.) responding to examples in this second part, we’re going to ask you to respond to three examples of subject guides, one from ubc and two from other university libraries. we’ve chosen subject guides in __________________________. although these topics may not be directly relevant for you, we’re interested in getting your feedback about how generally usefully they are. take a look at each webpage online and at the print outs we’ve given you. feel free to mark up the print outs by circling, highlighting or noting things that you find helpful or interesting or not useful. then, complete section c and d, using the rating scales provided. you have about 10 min. for this section. (12 min.) / evidence based library and information practice/ / / evidence based library and information practice evidence based library and information practice 2010, 5.1  evidence based library and information practice       commentary    shaping the information professional of the future    dr. birgitta olander  associate professor, library and information science  department of cultural sciences, lund university  lund, sweden  email: birgitta.olander@kultur.lu.se    received: 18 jan 2010          accepted: 18 jan 2010      © 2010 olander. this is an open access article distributed under the terms of the creative commons‐ attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐ sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original  work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is  redistributed under the same or similar license to this one.    a new arena and its areas of competence       during the last two decades we have seen  extensive changes in society and an explosive  development in information and  communication technologies (icts). those  changes have been both rapid and global, and  there are no indications that the rate of change  will slow down in the medium‐term future. in  the process, a new and different arena for  library and information services has  developed, one placing new demands on the  skills of information professionals.  information has never before been this widely  and easily accessible. even so, people demand  increasingly faster and easier access to  information and information services,  regardless of form, time or location.     library and information science education  must be both sustainable and renewable. the  skills that lis students acquire today should  be relevant also in 2040, since they can  anticipate a professional life of approximately  30‐35 years after completing their degrees.  this sounds slightly absurd given the rate of  change we have seen so far, and highlights the  fact that a persistent renewal of skills, i.e.,  continuing professional education, is required  in addition to the original lis degree. the  pedagogical issues involved in this include  both individual and organizational learning  and skills development. evidence based  library and information practice is a viable  strategy for renewal of one’s knowledge base,  while at the same time making one’s original  education sustainable.     in only 15 years the world wide web has  created a completely different information  landscape, penetrating into everyday life  much faster than television did in the 1950s.  the new lis arena comes as a result of this  and many other ict developments. other  factors, mainly socioeconomic, have been  instrumental in creating this new arena, but in  my opinion the web has been the single most  powerful driving force. the information sector  without digital technologies is completely  unthinkable today. digital technology not only  has vastly improved information searching,  but also provides us with an unsurpassed  1 mailto:birgitta.olander@kultur.lu.se evidence based library and information practice 2010, 5.1  diversity of information tools and media. in  this new arena, books, journals, and other old  media are reborn as digital information, often  directly available to users through online  venues. regardless of format, users demand  and expect that all the information and  entertainment on the web is fully and  instantly accessible.     who today needs a library when google and  amazon have replaced bibliographical  databases as search tools? access to  information about documents and other media  is no longer good enough, not when users  want the real thing ‐ be it the full text, music  file, or downloadable images ‐ instantly. a  vast majority of information seekers find  google sufficient for their needs, which  suggests that libraries and information  services face a real threat of being  marginalised due to an inability to fully  exploit the web’s potential. mobile phones, e‐ mail, blogs, twitter, chat, youtube, facebook,  and many more, are standard tools for  communication and information sharing  today. in the new arena there is tangible  competition for the attention of information  users, and library services must be attractive  enough to ensure that users remain actively  interested. libraries must position themselves  at the forefront and make relevant use of the  whole range of new technologies in order to  make information accessible in the formats  and ways that users prefer. in addition,  libraries have to be flexible and meet the  expectations of their users by being available  around the clock, and must be highly visible in  the social media forums where users operate.  many libraries have already come a long way  to meet the requirements of the new arena, are  increasingly accessible and interacting with  their users, but much remains to be done.     skills requirements in the new arena    it is essential that educators have a good  notion of what skills will be required in the  medium‐term future. the competencies  considered necessary for librarians today and  tomorrow might be defined on several levels,  from overall professional skills to personal  qualities. the underlying assumption is that  all professional skills must be theoretically  anchored and based on research evidence.   i have identified six areas of competence  required for the new arena that are applicable  in a wide range of tasks and contexts. multi‐ tasking capacity and social skills always have  been crucial for information professionals and  still are, but today the communication  initiative no longer rests with users alone.  now librarians have to be proactive in their  efforts to interact with users.     1. first and foremost, excellent social  skills are required. efficient teamwork  with colleagues and users goes  beyond merely getting along – it is  required for successful service  development. social skills are  increasingly important when  information services are organized for  project‐oriented work and to increase  user interactivity.     2. being able to communicate well  with users will continue to be vital for  information professionals. they must  push to develop interactive  communication tailored to user  preferences, and keep up‐to‐date with  new technologies and implement  multiple means of user  communication in order to keep users  engaged.    3. information management skills will  be equally important in the future as  today. increased accessibility is  pointless without relevant knowledge  organization and information  management tailored to meet users’  needs.     4. excellent ict skills are compulsory  for information professionals. most of  the new librarians were born digital,  placing them on an equal footing with  users their own age. the greatest  challenge is to continue to learn new  technologies and make them  professionally useful.  2 evidence based library and information practice 2010, 5.1    5. openness to change is a crucial  aspect, given that the rate of change is  not expected to slow over the  medium‐term. librarians whose  professional knowledge is theory‐ based rather than skills‐based tend to  be more flexible and willingly  embrace change. this is because a  theoretical knowledge base is focused  on content, which is sustainable,  rather than form and tools, which are  likely to change often.      6. finally, strategic competence is also  essential for applying a holistic,  process‐oriented view on all  professional activities. continued  success requires an understanding of  the goals, objectives, and future  development of the institution.    lis in theory and practice     in 1993, sweden’s former bias towards  vocational training was replaced by master’s  programmes in library and information  science, and integrated into the formal  national university system. the scientific basis  required in all areas of higher education is a  guarantee for knowledge that allows  generalization, making it sustainable. a fully  academic education in library and information  science has to place the emphasis on theory  rather than practice.  in preparing students for  future skills requirements, lis educators must  integrate theoretical and applied knowledge,  and ensure that student’ learning alternates  between the two. lis theory can be  supplemented by field studies and by visiting  lecturers who are active information  professionals. this interplay of theory and  practice will help students to develop a  professional attitude and identity. however, it  is not uncommon that lis students perceive  the practice oriented elements of their  education to be the most important. the  underlying assumption appears to be that  practice is where you really learn to be a  librarian.     many students seem to believe that the  theoretical knowledge they possess is not  valued in professional practice. if this belief is  widespread, and not just particular to swedish  lis students, we have a serious attitude  problem in our professional field, a problem  with clear implications for evidence based  practice. swedish librarians who graduated  before 1993 often have inadequate theoretical  knowledge in their professional subject area.  since many of the librarians that new lis  students meet belong to this category, this  could influence the students’ perceptions of  the importance of a theoretical foundation.  this is also an issue for career perceptions and  life‐long professional learning.    a proactive lis education should provide  future information professionals with what we  might call a “quintessential knowledge kit”.  the knowledge included in this kit is  applicable in a wide range of tasks and  contexts and constitutes core professional  competencies:    • comprehension of how information is  sought for, disseminated, and shared  between people and within  organizations;  • the ability to manage information and  render it meaningful and accessible  for potential users;  • the ability to devise systems or  methods for extracting new  knowledge from information sources;  • comprehension of the tactical and  strategic importance of information in  various contexts.    comparing the knowledge kit and the list of  six skills areas required in the new lis arena,  we find that they differ in the level of  abstraction but are otherwise reasonably  compatible. the first two items on the list,  social skills and communication skills, are  necessary for application of all the elements of  the knowledge kit, especially the first two.  item three on the skills list ‐ information  management skills ‐ correlates well with the  first two elements of the knowledge kit. item  four – ict skills – is required for applications  3 evidence based library and information practice 2010, 5.1  of the first three elements of the kit, and item  six – strategic competence – represents the  essence of the last element of the kit. item five  on the skills list, openness to change,  correlates to all and none. this item carries  very great importance, as an attitude that lis  education can foster in students, but it is  difficult to define as a knowledge element or  skill per se.     life‐long professional learning and library  careers    most people do not become librarians in order  to make spectacular careers. when they enter  library school, many lis students perceive the  physical library as the opposite of stressful  and busy, notions frequently linked to career  perceptions. they change opinion after having  met with the practice first‐hand, but their  attitudes on career planning often remain  unaltered. i see a clear correlation between  career planning and skills development. in  both cases it is essential for the development  of a professional identity to be explicitly aware  of one’s individual qualities, competencies,  and professional goals.     career consciousness requires the ability to  formulate succinctly what you know, what  you can do, and what you want to do. it is  useful to practice verbalizing one’s skills, by,  for example, trying to describe to non‐ information professionals how you handle  tasks at work. the ability to present a  comprehensible job description is necessary  when it comes to promoting oneself in a career  context, and it is important to include both  personal qualities and professional skills.  personal qualities like communication skills,  empathy, inventiveness and flexibility, and  many more are crucial – and very attractive –  in the job market.     discussion about skills development should  start from a holistic view of the competence  present, or missing, in the organization. the  visions that library leaders strive to implement  in order to profile their operations often  demand that individual staff and the  organization as a whole hone their skills. the  organization’s visions, goals and objectives  indicate what kind of librarians the library  needs, and highlight what areas of competence  should be developed further. strategic  planning for developing the services of a  library could be regarded as career planning  on an organizational level. skills development  on this level benefits from teamwork and  project‐oriented approaches, coupled with a  work environment that encourages learning.  continuous organizational and professional  development should be based on scientific  evidence in order to be sustainable.    future information professionals need not  only great social and ict skills, but also a  wide range of traditional library competencies.  a massive generation shift is under way in the  information sector as many librarians head for  retirement. the collective and individual  knowledge they possess is literally invaluable  for the professional field of practice, and we  need to find viable strategies to transfer this  knowledge to their successors. libraries and  information services designed as learning  organizations are better prepared to manage  this, whereas those that are not will face  difficulties.     this paper has addressed aspects that are  crucial in shaping the information  professionals we need in the medium‐term  future. in identifying the skills required in the  new arena of the information sector, we  discover implications for the practising  information professional in terms of skills  development and for lis education.     implications for practice    • individual and institutional levels of  professional skills development  constitute two types of library careers,  and go hand‐in‐hand.   • the more successful the organization,  the more room there is for individual  success. for the individual, the reward  for life‐long learning is a gratifying  career; for the institution, the reward  is a learning organization.   4 evidence based library and information practice 2010, 5.1  5 • openness to change is key among six  skills areas identified for the new  arena. those able to cope with change  have a solid theoretical lis knowledge  base, plan their work experience with  a clear vision of their future, and  possess an eagerness to learn.   • evidence based practice is a natural  mode for professionals who are open  to change.    implications for lis education    • library schools must take  responsibility for recruiting and  educating students to provide them  with academic, professional, and  personal skills that future practice  demands.   • carefully deliberated recruitment  criteria and integration of theoretical  and applied professional knowledge  are cornerstones for sustainable lis  education.   • it is essential to impress upon students  the importance of maintaining and  updating one’s knowledge throughout  professional life.    evidence summary   development of dealand journal-level metrics and methods assists librarians to evaluate big deals   a review of: blecic, d.d., wiberley, jr., s.e., fiscella, j.b., bahnmaier-blaszczak, s., & lowery, r. (2013). deal or no deal?: evaluating big deals and their journals. college & research libraries, 74(2), 178-193.   reviewed by: kathleen reed assessment & data librarian vancouver island university nanaimo, british columbia, canada email: kathleen.reed@viu.ca     received: 1 jan. 2014      accepted: 17 jun. 2014      2014 reed. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to assess the value of aggregated journal packages (big deals) and to select individual journal titles for continued subscription should a deal be cancelled.   design – case study.   setting – doctoral research university library in the united states of america.   subjects – three anonymous big deals.   methods – the authors define metrics at two levels (deal and journal) to evaluate big deal packages. the metrics rely heavily on the counter jr1 metric successful full-text article request (sftar).   main results – the authors found that while 30% of journals provide 80% of sftars, the cost of subscribing to these journals individually would not save significant sums of money. additionally, they speculate that library users would increase the number of interlibrary loan requests to access the 20% of sftars that would be inaccessible if a big deal was cut, amounting to increased costs.    conclusion – with no sign of publishers moving to change the price and conditions of big deals, these arrangements are becoming unsustainable for libraries. as this occurs, librarians require methods of assessing which deals to keep and which to cut, as well as evidence of to which individual journals they should subscribe. the authors of this paper set out one method of conducting these assessments that they have found to be useful at an academic library. they conclude by stating that even with sftar data, individuals must keep in mind the necessity of providing equitable access to all of a university community’s user groups.     commentary   in a climate of financial difficulties, there is a need for metrics and methods to assist librarians in evaluating big deals, “an online aggregation of journals that publishers offer as a one-price, one size fits all package” (frazier, 2001). since big deals utilize significant financial resources of academic libraries, they continue to be under close scrutiny for their value to the institution. the authors present an approach used at one research library to assess big deals and to select individual journal titles for continued subscription should a deal be cancelled.    as an introduction to the assessment of big deals, this article is a must-read. it contains practical instructions for conducting assessment, interwoven with discussion of many of the critical issues and challenges librarians face when examining big deals. the authors’ suggested assessment steps go beyond examining just a single metric like counter’s jr1 (successful full-text article requests), but are not so complex that they require difficult models and special training. there are two processes written about in detail: establishing deal-level metrics and journal-level metrics.   the authors give a thoughtful commentary on major pitfalls of their metrics, including devoting significant discussion to the importance of qualitative measures and individual judgment in addition to quantitative calculations. for example, using the three big deals as examples and breaking down each journal title into one of four subject areas (humanities, social sciences, stem, and health sciences), the authors show that if the sftar numbers alone were relied upon, almost all access to humanities journals would have been cut. however, the authors do not formalize these qualitative steps in the big deal assessment model as clearly as the quantitative metrics steps. a subsequent paper that captures the qualitative process of assessing the value of add-ons to the packages such as mobile accessibility or interface design, for example, would be of use to those thinking about database assessment.    additionally, the authors do not account for the value of having large numbers of resources discoverable to serve the long-tail of user searches. a resource may not be used for several years, and then be found to be useful by an individual. having a comprehensive collection that is discoverable adds value to a library, although it is difficult to assign a specific dollar value to this strength of a big deal.   a final gap in the authors’ analysis relates to the journal prices used in calculations. while the authors found that the savings projected by the analysis were not significant, they came to this conclusion based on the subscription price for individual journals listed on publishers’ websites. it is not known if a library would get a discount on the list prices. if this occurred, it might change the significance of the total savings.   despite a few minor aspects that could be strengthened, overall this article is a useful and thoughtful contribution to the literature on assessing big deals. the authors provide helpful examples of metrics and methods, as well as a roadmap through a potential minefield of mistakes and oversights that could befall librarians doing this type of assessment.    references   frazier, k. (2001). the librarians’ dilemma: contemplating the costs of the “big deal.” d-lib magazine, 7(3). retrieved from http://www.dlib.org/dlib/march01/frazier/03frazier.html   evidence based library and information practice evidence based library and information practice 2010, 5.3 49 evidence based library and information practice evidence summary labour costs for inventory control less expensive than repurchasing a review of: sung, j. s., whisler, j. a., & sung, n. (2009). a cost-benefit analysis of a collections inventory project: a statistical analysis of inventory data from a medium-sized academic library. journal of academic librarianship, 35(4), 314-323. reviewed by: laura newton miller science & engineering librarian, carleton university ottawa, ontario, canada e-mail: laura_newtonmiller@carleton.ca received: 27 apr. 2010 accepted: 4 aug. 2010 2010 newton miller. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to describe an inventory system that was created within the library and to show the cost-effectiveness of using the inventory system compared to the price of reacquiring mis-shelved books. design – bibliometric study and cost-benefit analysis. setting – medium-sized academic library in a rural community of the united states. subjects – approximately 300,000 books from lc classifications d through h, n, p and q, representing two thirds of the library’s entire monograph collection. methods – the library created its own electronic inventory and shelf-reading program, using a laptop computer equipped with a hand-held scanner, to scan barcodes in the stacks. library staff used the microsoft access database to update two files containing a shelf-list and an active-status list while the books were scanned. the program alerted the worker if books found had an active status (i.e., missing, renewed, overdue, charged), were not in the correct order, or were not in the system. each transaction created a log which contained a time stamp (to the second), the call number and the barcode number. it also took note of scanning errors, books that were out of order, and books that were not on a shelf-list. after a complete section was examined, a list was produced to reveal the distance of mis-shelved books from their correct location and the amount of time mailto:laura_newtonmiller@carleton.ca evidence based library and information practice 2010, 5.3 50 between each scan. the researchers used statistical analysis (using spss 15.0) to measure scan speed for each scan, misshelving rate and error distance of each misshelved book. in order to analyze the cost of labour to replace a book versus the cost of inventorying, the researchers estimated the salary costs of staff members involved in selection, acquisition and cataloguing. the library spent $440,000 usd in labour costs to purchase 15,000 monographs in one fiscal year (approximately $30.00 usd in labour costs per book). they multiplied this by 5300 books that were found to be “badly” mis-shelved (found beyond 25 books away from the proper position). labour fees were used to determine costs of inventorying by calculating average scanning speed and cost per hour to pay someone to scan the entire half-million monograph collection. main results – it took approximately 707 hours to scan 305,000 monographs. the average (mode) calculation for scans was 5 seconds for 80% of the barcodes, with an average (mean) of 8.35 seconds between scans. the longest average (mean) time for scanning barcodes was in the n section, followed by g, h, p, q, d, e and f. a total of 291 books were found on the shelves with an “active” status (i.e., charged (4), overdue (7), renewed (4), in transit (24), and missing (228)). twenty-four books with the status “miscellaneous” (i.e., at bindery, call slip, cataloguing review, damaged, and mending) were also found on the shelves. of the 15 active books in the categories “charged”, “overdue” and “renewed”, ten were found in the proper position on the shelf. of the 228 “missing” books, 30% were scanned in the correct location, 10% were found 26 to 100 books away, and half were located over 100 books from their proper location. in addition to the books already marked as “missing” in the catalogue, there were 516 books (.17% of the entire scanned section) still not found on the shelf after three searches over a period of 6 months. of the 291 active status books found on the shelves, 52% were reused as of july 2008. (the inventory was completed at the end of 2006). over 36% of books mis-shelved further than 25 books from their correct location were reused. however, among all books scanned, only 17% were reused during the same time period. the researchers noted that inconsistencies between the call number as shown on the book label and how it appeared in the catalogue occurred 565 times. of these discrepancies, 40% of the labels resulted in books being misplaced ten or fewer books away, 10% misplaced between 10 and 100 books away, and 35% misplaced more than 100 books away from the correct position. in general, 82% of mis-shelved books were found within 1 to 25 books away from their correct location. by calculating that 5300 books were mis-shelved beyond 25 books away from their proper position, labour costs were estimated to be at least $159,000 usd (5300 x $30.00 usd per book in labour costs). costs for interlibrary loan were calculated at approximately $30.00 usd per transaction, and patron’s time wasted trying to locate misplaced books was estimated at 30 minutes per book. this was much more than the labour costs associated with scanning books, which at an average speed of 8 seconds per book and $10.00 us per hour for scanning worked out to be 2.2 cents per book, or $11,000 usd to scan the entire half-million monograph collection. conclusion – the results appear to reveal that the labour costs for inventory control are less expensive than repurchasing or borrowing the same number of books. commentary stack services staff and student workers are usually those who have the important task of taking inventory of the library’s collection. shelf-reading is a common way of doing this, as well as by using a barcode reader and comparing to a shelf list. however, the “library stacks management system” that is described in this study goes one step further by providing a way to notify proper staff of any inventory discrepancies immediately instead of staff having to wait to go back to the stacks at a later time to correct a problem. it is evidence based library and information practice 2010, 5.3 51 also very useful to know how much time it takes to go through the inventory process. the researchers explain that past inventory studies are more descriptive than analytic. they are to be commended for the detailed analysis of inventory scanned, as this will help staff to better understand their collection and workflows. because bibliometric studies focus on a specific set of data, it is important to replicate and make comparisons with other similar studies. this analysis mostly compares itself to one study in particular (anderson, 1998). similar research taken on by other libraries will be valuable to the community as a whole, since these studies work very well when building on one another’s work. however, some of the data described in this paper is misleading. for example, the researchers discuss books being mis-shelved a certain number of books away from the proper position. one must look in the footnotes to discover that the researchers actually assume that if 27 inches of the average shelf is taken up with books and that the average thickness of a book is set to one inch, this can then be converted into 25 books. as bibliometric studies depend on building on past studies, it would have been far more valuable to maintain descriptions similar to those in anderson’s study, which described major errors in mis-shelving as those books being out of place beyond the shelves before or after the correct shelf (anderson, 1998). this would also make more sense for staff members who count in terms of bays and shelves as opposed to individual books. there are several examples in which data analyzed with this new inventory system can provide valuable information to libraries regarding workflows and procedures. the researchers focused on the 15 books with “active” statuses (charged, renewed, and overdue) but do not go into much detail regarding reasons for 24 “miscellaneous” books being found on the shelves. this includes books that say “at bindery”, “call slip”, “cataloguing review”, “damaged”, and “mending”. this is important information for departments that may want to examine their workflows to ensure that these particular books do not get onto the shelves without a change in their status being noted. the researchers also focus on the fact that label discrepancies are minor in that 40% of these led to only very slight mis-shelving of books. however they do not comment on the fact that 35% led to books being mis-shelved over 100 books away. again, this is an area in which libraries will want to study workflows in order to make sure mistakes like this are not a burden to the patrons. finally, of the 228 “missing” books, 30% were scanned in the correct location. this particular library may want to examine its procedures for labelling something in the catalogue as “missing”, as one would assume that it takes time for the staff to change the item’s status in the catalogue, when it may have been more efficient for a staff member to check first to find the book in the correct location. as part of the cost-benefit analysis, it would have been valuable to read commentary on the costs of various library purchasing programs (e.g., coutts, yankee book peddler) that in some respects have made selection and purchasing more streamlined. the researchers discuss the costs of interlibrary loan transactions and acknowledge that the study they were focusing on was over ten years old. more recent research regarding acquisition costs and interlibrary loan transaction costs would be beneficial to this and other studies. the cost-benefit analysis would be further improved by a description of work procedures. salaries should also be described more clearly within the paper as opposed to making reference to them in the footnotes. job descriptions would be helpful in determining the meaning of salary numbers. for example, assuming that collection development falls under reference services (or something similar), many hours are devoted to instruction, reference desk, committee work, and professional activities. because this is not sufficiently explained in the study, the costbenefit analysis is very difficult to replicate. the researchers also fail to describe the initial labour costs involved in developing the evidence based library and information practice 2010, 5.3 52 inventory system in-house. there would be a significant one-time cost, but maintenance and staff training expenses would also have to be taken into consideration. there is great value in having a library collection in order. having an in-house inventory system that gives detailed analysis is of great benefit to the library. perhaps one cannot sufficiently put a price on the value of such a system that forces libraries to reexamine their workflows. the cost-benefit analysis is complex because there is so much to take into consideration. perhaps comparing the labour costs of the current inventory system to another way of taking inventory (e.g., traditional shelf reading) would be more beneficial and less complicated for librarians trying to decide if they want to spend money on a detailed in-house inventory system. references anderson, d. r. (1998). method without madness: shelf-reading methods and project management. college & undergraduate libraries, 5(1), 1-13. evidence based library and information practice commentary   researching the economic contribution of public libraries   brian ashley director, libraries arts council england     manchester, united kingdom email: brian.ashley@artscouncil.org.uk   vivien niblett senior officer, policy & research (evaluation) arts council england manchester, united kingdom email: vivien.niblett@artscouncil.org.uk   received: 2 sept. 2014    accepted: 14 nov. 2014      2014 ashley and niblett. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   the economic contribution of arts and culture   arts council england champions, develops, and invests in artistic and cultural experiences that enrich people's lives. since 2011 we have been responsible for supporting and developing public libraries in england. to inform our wider contribution and to provide a focus for the development of public libraries in england, we undertook some substantial research into the role of public libraries in the future. the resultant study was called envisioning the library of the future (arts council england, 2013).   following on from this, and in a context of reduced public spending, arts council england invested in a strand of research to identify and measure the contribution arts and culture make to the national economy (centre for economics and business research, 2013). the study used economic impact assessment methods to estimate the value of the arts and culture sectors. however, the approaches used were not appropriate for the valuation of public libraries, whose service model does not have clear financial outputs measurable through national accounts. this left not only a gap in the evidence available at a national level about what library services contribute to the economy, but also a question of how to gather this evidence.   studies had been carried out that measured the economic impact of libraries, including some local studies in the uk and national abroad. however, we were not aware of work exploring the indirect ways in which libraries could make an economic contribution– for example, by reducing demand on health services through public health promotion, or providing internet and information access that helps people improve their job prospects. we wanted to explore whether these benefits could be valued.    we felt undertaking research to explore this issue would help us in our role to develop and advocate the public value of libraries and would enable both arts council england and the wider libraries sector to understand how libraries make an economic contribution.   developing a model of economic contribution   the first stage was to establish a need for and interest in a study of economic contribution. there are a number of different ways in which libraries can be seen to create economic value, so we wanted to focus the work by developing a theoretical model which could be tested.   partners, such as the society of chief librarians, the british library, and the local government association, came together for an initial scoping meeting with arts council england which aimed to:   a)       establish the level of interest and need for this work in the libraries sector; b)       establish where there are gaps in evidence around the economic contribution of libraries; c)       begin to develop a model of how we believe libraries make an economic contribution; and d)      identify any priority areas of investigation.   we used ideas generated in this discussion to develop models, based on the concept of logic models (w.k kellogg foundation, 2004), of how libraries might contribute to the economy. these became the lines of enquiry for a desk-based evidence review which aimed to identify and synthesize the existing evidence available and to further test and develop the models.   evidence review   the commissioned evidence review summarised the different approaches taken to the question of economic contribution and identified strengths and weaknesses in the evidence base for the educational and social impact of public libraries and its associated economic value (arts council england, 2014). a summary of these findings are reproduced below.   economic valuation studies of public libraries   as the traditional metrics for measuring the economic contribution of an industry are not appropriate in a public library context, researchers have used a number of different methods for quantifying the economic value of libraries. three different hypotheses as to how public libraries make a contribution to the economy were identified:   as economic actors in their own right (economic impact) as institutions that facilitate the creation of economic value in the adjacent area and local economy (place-based economic development) as organizations that deliver a wide range of services, most of which are valued by both users and non-users when set against their costs of provision (benefit-cost/total economic value approaches)   all three hypotheses require empirical methods based on the collection of primary data, which makes them both bespoke and relatively expensive. they are not designed to add up to an aggregate picture of the economic value of public libraries in england nor for their results to be comparable.   the economic impact literature shows that public libraries employ people and spend money, having a knock-on effect in the local economy, through supply chain expenditures and the wage expenditure of employees. a few large library facilities may also trigger significant ancillary spending in the local economy by visitors who are drawn to the area by the library. economic impact assessment is well tried and tested within other industry contexts in the uk and in other country contexts with regards to libraries (specifically the us). however, it has few merits for libraries within a u.k. public policy context because libraries simply do not have the characteristics to perform well in relation to the additional requirements that are embedded in public policy economic impact appraisals in the uk.   libraries’ contribution to wider place-based economic development is an area which suggests greater promise based on the existing current case studies both in the uk and internationally. libraries can be anchor tenants in mixed-use physical developments and regeneration initiatives, potentially boosting the footfall, buzz, image, and profile of a neighbourhood or area particularly if the library is new, large, or housed in an iconic building. finally, where specialist services are provided, libraries can also support local economic development through business advice and support for individuals, micro businesses, and smes.   however, the evidence base on libraries’ contribution to wider place-based economic development is at present under-developed. it is based on only a small number of case studies, which vary greatly in detail and lack longitudinal analysis and attempts to account for other factors that may also have contributed to identifiable regeneration and local economic development.   the most numerous are benefit-cost studies and, in the case of contingent valuation (cv) methods, have the potential to gain the most traction with national government stakeholders.   there is near universal consistency across these studies in reporting positive benefit-cost ratios for public libraries, demonstrating that societies value public libraries over and above what they pay for them collectively. however, this benefit-cost varies in a range across one-and-a-half, twice, or 10 times the cost of provision. these approaches, particularly cv methods, are both very expensive and complex to undertake to a credible standard, and their robustness around libraries can be compromised due to a range of methodological weaknesses. the choice of methods and assumptions appears to have a consistent bearing on benefit-cost ratios.   finally, there are three challenges to gain a comprehensive sense of economic contribution:   the various estimation techniques are all focused on producing a single figure for the impact, value, or return on investment from the service as a whole. they pay little attention to how the constituent library services, resources, and buildings generate this value (aside from some revealed preference studies that look at one or two services only, such as book and media lending). this emphasis on the aggregate quantification of value can also seem somewhat abstract. for instance, it does not generate the kind of evidence that facilitates detailed decision making and therefore it can seem remote from the day-to-day reality of service planning and budget setting. relatedly, using economic valuation methods alone means that it can be hard to communicate the benefits of libraries to non-economist audiences. all the methods require some degree of technical knowledge to properly understand, with cv studies being particularly complicated and specialist. lastly, studies really only measure the short term economic value of library services. this means that the wider value to society in the present is not captured and neither is the value of these in the future.   these structural weaknesses in economic valuation approaches mean that literature from disciplines other than economics is required in order to understand in more detail and more holistically, how public libraries make an economic contribution to society.   studies on libraries’ educational and social impact   the evidence review goes on to consider libraries’ educational and social impact within five key areas, chosen to cover the main activities currently taking place across the library service:   children and young people’s education and personal development   existing research provides compelling evidence that library usage is linked to reading levels among children and young people, and that library usage and reading, in turn, are important factors in literacy skill levels and general educational attainment. some research also suggests that the quality of public libraries’ space supports educational attainment. while there is no evidence of the direct financial benefits of libraries’ impact specifically, some recent studies indicate the private and public economic benefits that would be obtained by addressing low literacy and education levels.   adult education, skills and employability   while some surveys have established a link between adult reading habits and library usage, it remains difficult to confirm the direction of causation between the two. similarly, there is only limited available evidence of their impact on adult literacy levels. recent large-scale us-based surveys demonstrate that adults use libraries’ information and communications technology (ict) provision in particular to support their learning and for job searching activities. literature from the uk also provides evidence of the provision of job support services, if not evidence of impact. overall, research in the uk on adult learning, skills development, and library-based employment support remains largely focused on local, smaller-scale studies.   health and wellbeing   while there is no consistent data on the provision of library-based health and wellbeing activities, existing research suggests that such activities are increasingly becoming a core part of the public library offer in the uk. bibliotherapy activities are now widely available across public libraries in the uk and computer-based cognitive behavioural therapy (ccbt) activities are increasing. existing research also suggests strong correlations between reading and mental health benefits, as well as a link between literacy and health literacy – people’s ability to access and use health information. for all of these reasons, libraries contribute to the health “prevention agenda”. a small number of studies also suggest library-based health provision could contribute to reducing the current high costs of ill health. evidence is lagging behind practice in terms of demonstrating what benefits service providers may get from using libraries to deliver health activities, although several uk-based case studies suggest that libraries are valued for their non-clinical atmosphere and community reach.   community support and cohesion   current research demonstrates that a majority of library users and non-users consider libraries important for their community and suggests that libraries may play an important role in contributing to the social capital of communities. evidence exists in particular for libraries’ contribution to facilitating social contact and mixing within local communities, as well as through increasing levels of trust among people. existing research also demonstrates that people place a high trust in libraries themselves as institutions. while it might be argued that this may contribute to wider trust in institutions in general – a further marker of social capital – there is currently little evidence to support this. several studies further suggest that through increasing social capital, libraries contribute to enhanced community cohesion and thereby, to healthier, safer communities. although evidence of the impact of cohesion on reducing deprivation is limited, the cost of ill health and crime within communities illustrates the potential cost savings that could be realized through healthier and safer communities.   digital inclusion   studies demonstrate a high level of available ict provision across the public library networks, as well as the high usage of this provision by library users and the role libraries thus play in increasing digital inclusion. evidence also shows that library ict provision is frequently used by visitors to gain information important to their everyday life including education, work, and social networks. in addition, several recent studies highlight the personal benefits of digital inclusion by increasing employability in addition to the cost savings to the state through services increasingly being provided online under the “digital by default” agenda.   summary of the evidence   the existing literature on the educational and social benefits of libraries naturally contains limitations and weaknesses, in particular issues around:   a general lack of longitudinal studies and surveys/studies fulfilling the requirements in the health sector for clinical studies  a lack of management and impact data on libraries’ various local services and their users and partners that can be aggregated the difficulty of establishing causality between library usage and a range of outcomes a lack of evidence of the savings to the state generated by libraries regarding the downstream outcomes that libraries contribute toward.   however, evidence is already sufficient to conclude that public libraries:   provide positive outcomes for people and communities in many areas – far exceeding the traditional perception of libraries as just places from which to borrow books; contribute to long term processes of human capital formation, the maintenance of mental and physical wellbeing, social inclusivity, and the cohesion of communities; make a real economic contribution to the uk that is long term with financial benefits that arise downstream from libraries’ activities, and that takes place through multi-dimensional, complex processes of human and social development.   this suggests that attempting to derive a realistic and accurate overall monetary valuation for public libraries will be very hard to achieve. it also shows that measuring libraries’ short term economic impact provides only a very thin, diminished account of their true value.   in-depth study   the evidence review left no illusions about the complexity and challenges presented by the task of capturing the long terms contribution of public libraries to the national economy. however, arts council england remains committed to delivering evidence that will have a significant benefit in the active and live debates about the role of and future for public libraries.   we chose to continue this work by commissioning an in-depth study which focused on one of the five policy strands identified by the evidence review. following discussion with stakeholders, it was agreed that this area would be health and wellbeing. it was felt that this is an area in which libraries are increasingly active, and where robust evidence would assist in making the case when commissioning services.   the key research questions of this study are:   what is the £ value contributed by libraries through their impact on health and wellbeing at both a national and local level? how is this value created, and what is the specific role of library services, programmes, buildings, or staff in this process? to whom does this value accrue? how does value change for different end users, such as different parts of the public sector, or individuals and communities with different backgrounds?   this study has now begun and will be undertaken in the period up to february 2015 with an intention to publish the findings in the spring of 2015. the findings of this study should assist library services and stakeholders to articulate the value and contribution of libraries through robust and credible evidence.   references   arts council england. (2013). envisioning the library of the future.  manchester: arts council england. retrieved from  http://www.artscouncil.org.uk/what-we-do/supporting-libraries/library-of-the-future/   arts council england. (2014). evidence review of the economic contribution of libraries.  manchester: arts council england. retrieved from http://www.artscouncil.org.uk/advice-and-guidance/browse-advice-and-guidance/evidence-review-economic-contribution-libraries   centre for economics and business research. (2013). the contribution of the arts and culture to the national economy.  manchester: arts council england. retrieved from  http://www.artscouncil.org.uk/advice-and-guidance/browse-advice-and-guidance/contribution-arts-and-culture-national-economy   w.k kellogg foundation. (2004). logic model development guide. michigan: w.k kellogg foundation. retrieved from  https://www.wkkf.org/resource-directory/resource/2006/02/wk-kellogg-foundation-logic-model-development-guide   microsoft word art_kouf.doc evidence based library and information practice 2006, 1:3  3 evidence based library and information practice     article    effective methods for teaching information literacy skills to undergraduate students:  a systematic review and meta‐analysis      denise koufogiannakis  collections and acquisitions coordinator  university of alberta libraries  edmonton, alberta, canada  e‐mail: denise.koufogiannakis@ualberta.ca    natasha wiebe  research associate/statistician  university of alberta  edmonton, alberta, canada  e‐mail: natasha.wiebe@ualberta.ca      received: 31 may 2006    accepted: 02 august 2006      © 2006 koufogiannakis and wiebe. this is an open access article distributed under the terms of the  creative commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits  unrestricted use, distribution, and reproduction in any medium, provided the original work is properly  cited.    structured abstract    objective ‐ the objective of this systematic review was to assess which library instruction  methods are most effective for improving the information skills of students at an  introductory, undergraduate level, using cognitive outcomes (measuring changes in  knowledge). the study sought to address the following questions:  1) what is the overall state of research on this topic?  2) which teaching methods are more effective?    methods ‐ this project utilised systematic review methodology. researchers searched  fifteen databases and retrieved 4,356 potentially relevant citations.  they reviewed the titles  and abstracts for relevance, and of those, 257 complete articles were considered in‐depth  using a predetermined inclusion/exclusion form. there were 122 unique studies that met  the inclusion criteria and were subjected to an extensive data extraction and critical  appraisal process. of these studies, 55 met author‐defined quality criteria to provide  information on the effectiveness of different teaching methods. from this review there was  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:3  4 a final group of 16 studies with sufficient information to enable meta‐analyses and  calculations of standardized mean differences.    results ‐ the overwhelming majority of studies were conducted in the united states (88%).  experimental or quasi‐experimental research methods were used in 79 studies (65%).  teaching methods used in the studies varied, with the majority focused on traditional  methods of teaching, followed by computer assisted instruction (cai), and self‐directed  independent learning (sdil). studies measured outcomes that correlated with bloom’s  lower levels of learning (‘remember’, ‘understand’, ‘apply’).     sixteen studies compared traditional instruction (ti) with no instruction, and twelve of  those found a positive outcome. meta‐analysis of the data from 4 of these studies agreed  with the positive conclusions favouring ti. fourteen studies compared cai with traditional  instruction (ti), and 9 of these showed a neutral result. meta‐analysis of 8 of these studies  agreed with this neutral result. another group of 6 studies compared sdil with no  instruction, and meta‐analysis of 5 of these agreed that the result was positive in favour of  sdil.      conclusion ‐ based on the results of the meta‐analysis, there is sufficient evidence to  suggest that cai is as effective as ti. evidence also suggests that both ti and sdil are more  effective than no instruction. additional comparative research needs to be done across  different teaching methods.  studies comparing active learning (al), cai, and sdil would  greatly enrich the research literature. further studies utilizing appropriate methodologies  and validated research tools would enrich our evidence base, and contribute to the growth  of knowledge about effectiveness of particular teaching methods.      introduction    information literacy is a topic of great  interest in the field of library and  information studies, particularly among  academic librarians, who view teaching as  an important role (baruchson‐arbib and  bronstein; godwin; peacock). the national  forum on information literacy defines  information literacy as “the ability to know  when there is a need for information, to be  able to identify, locate, evaluate, and  effectively use that information for the issue  or problem at hand.”  librarians are  constantly looking to improve the methods  by which they teach information skills to  undergraduate students, in order to increase  the students’ competencies in this area. the  professional literature is populated with  articles on the topic, ranging from research  studies, new innovations, and tales of  implementation in various settings. most  librarians have their own stories of teaching  successes and failures which they pass on to  others. however, while there have been  attempts to systematically appraise the  literature as it pertains to health  professionals (brettle; garg and turtle), the  general library research literature of this  field has not been gathered or summarized  in a systematic way that would facilitate an  evidence based approach towards  undergraduate level instruction (i.e., using  evidence to inform and support information  literacy initiatives). this study was an  attempt to sort through the published  literature and move beyond conjecture  surrounding the most effective methods of  teaching information literacy skills, via a  systematic review of the library research  literature relating to undergraduate teaching.     the objective of this review was to assess  which library instruction methods are most  evidence based library and information practice 2006, 1:3  5 effective for improving the information  skills of students at an introductory,  undergraduate level, using cognitive  outcomes (measuring changes in  knowledge). cognitive outcomes may be at  varying levels, from simply remembering  facts, to applying what was taught in a new  situation, to creating new content.  it should  be noted that cognitive outcomes are only  one aspect that may be considered when  determining the success of an information  literacy program. behavioural (measuring  changes in actions) and affective (measuring  changes in attitudes or values) outcomes are  other aspects that may be considered. this  review, however, did not attempt to  encompass all areas of research, and focused  solely on cognitive outcomes.     the study sought to address the following  questions:  1) what is the overall state of research  on this topic?  2) which teaching methods are more  effective?    in addition, the following hypotheses were  postulated:  1) instruction that is taught by a  librarian face‐to‐face is more  effective than instruction that is  computer‐based.   2) instruction that encourages active  participation from students is more  effective than passive instructional  modes, such as lectures and  demonstrations.  3) using bloom’s taxonomy, the higher  the level of learning outcomes  measured, the more difficult it will  be to link effectiveness directly to  library instruction, and the less  likely it is that the library  instruction will result in a positive  outcome.     in terms of evidence based library and  information practice, the goal of conducting  this systematic review was to find evidence  on information literacy instructional  methods that may have a direct impact on  the way academic librarians approach  information literacy instruction for  undergraduate students.    methods    the research methodology used was a  systematic review, including an extensive  literature search, an inclusion and exclusion  process for potential studies, extraction of  data from the included studies, and analysis  of that data.    once the study objectives and hypotheses  had been determined, inclusion and  exclusion criteria were developed.  categories of data required to address the  review objectives and hypotheses were also  noted for future data extraction. these  predetermined criteria helped frame the  search process.    fifteen databases (lisa, library literature,  eric, inspec, academic search premier,  educational research abstracts, cinahl, web  of science, dissertation abstracts, conference  papers index, sigle, ceruk, education‐line,  british education index, and australian  education index) were searched for relevant  articles in the fall of 2004. the searches  (appendix a) were updated in may 2005,  and results combined in a single  bibliography. researchers also checked  related bibliographies, literature reviews,  and references cited in these articles.     for a study to be included in the systematic  review, it had to meet the following criteria:  • instruction had to be led by a librarian  or library assistant (or with librarians as  part of the instruction team) for a class  or stand‐alone session utilising any  instruction method.   evidence based library and information practice 2006, 1:3  6 • study subjects had to be undergraduate  students at a post‐secondary academic  institution.   • the research study had to have an  evaluative component that measured  the cognitive outcome effect of  instruction on student learning via some  test of information literacy (e.g., pre‐  and post‐tests, graded papers, or  bibliographies).     included studies were not limited by  publication date, but were limited to the  english language. studies were not  excluded on the basis of quality or study  methodology.    as is shown in figure 1, 4,356 potentially  relevant citations were retrieved from the  literature search, and titles and abstracts  were reviewed for significance. of those, 257  complete articles were considered in‐depth  using a predetermined inclusion/exclusion  form, with 108 meeting inclusion criteria.  another 17 articles that met inclusion  criteria were identified by checking  reference lists and review articles on the  topic.    after eliminating three studies where  articles had been reported in two source  publications, there were 122 unique studies  that met the inclusion criteria and  underwent an extensive data extraction and  critical appraisal process. appendix b  contains the list of studies reviewed.   researchers entered data extraction  elements (appendix c) in an excel®  spreadsheet.    categories for data extraction were tested on  a subset of 8 articles included in the  systematic review. studies were critically  appraised using the checklist developed by  morrison et al. (891).  the checklist consists  of nine questions, focused on the validity  and applicability of the study being  appraised.  the 122 studies were used to  present the results relating to the first  objective, namely the overall state of  research on the topic.    following the data extraction and critical  appraisal process undertaken by dk, a  methodological quality filter was applied to  all 122 studies. those that had a  comparative study design and compared  two different teaching methods, and whose  outcomes were based on tests of statistical  significance (n=55) were analyzed to  determine the results of effectiveness for  different teaching methods.  a meta‐ analysis was conducted, by nw, on 16 of  these studies to substantiate the findings  relating to the effectiveness of different  teaching methods. the meta‐analysis section  of this paper describes this approach in  more detail.    results     the results are presented in three parts.  the  first part addresses the first research  objective:  to provide an overview of the  state of research on this topic with a  description of the studies included. part two  provides further analysis on a subsection of  the results of those studies to examine the  second study objective and determine which  teaching methods are most effective. part  three details the meta‐analysis on specific  aspects of effectiveness, where the data  enabled this method to be used.    i. status of research     description of included studies  the majority (78%, 97/125) of the studies  were published as journal articles (table 1),  with large numbers coming from three main  journals, college & research libraries, research  strategies, and journal of academic  librarianship. publications were spread over  a time period spanning from 1963 to 2005,  with most publications (35%, 44/125) coming  from the current decade. the 1980s   evidence based library and information practice 2006, 1:3  7 figure 1: flow diagram of the systematic review process      potentially eligible citations  identified by literature search  (n=4,356)  articles subjected to an in‐depth  review (n=257)  citations excluded at stage 1  (n= 4,099)  articles found outside of structured  literature search that met inclusion  criteria (n=17)  articles excluded at stage 2 (n=142);  unable to obtain (n=7)  total number of articles included  in the systematic review (n=125) articles found through literature  search that met inclusion criteria  (n=108)  total number of research studies  included in the systematic review  (n=122) publication overlap whereby  the same study was  published in more than one  source (n=3)  total number of research studies  that met quality filter criteria  (n=55)  total number of studies that were  suitable for meta‐analysis (n=16)  see results, section i  see results, section ii  see results, section iii  evidence based library and information practice 2006, 1:3  8 type of publication  number   journal articles               97  college & research libraries             (20)    research strategies                            (14)    journal of academic librarianship    (11)    other journals                                  (52)    eric documents  19  dissertations     8  book chapters     1  total publications  125    table 1: publication types      produced nearly the same amount of  research on the topic (33%, 41/125). in the  1990s the numbers declined (21%, 26/125),  but there was a resurgence in the present  decade.    the vast majority (88%) of studies (107/122)  were conducted in the united states. other  countries included were canada (6.5%,  8/122) studies and australia (4%, 5/122). the  uk, and trinidad and tobago each  contributed one study.  the dominance of  studies from the united states is  overwhelming. certainly other countries,  such as the united kingdom and australia,  have strong academic information literacy  programs operating. this may indicate that  the focus of research on the topic of  information literacy has been different in  other countries, and that the specific focus of  this systematic review has been dominated  by u.s. researchers. perhaps researchers in  other countries have focused on more  qualitative aspects, rather than trying to  measure effectiveness based on cognitive  outcomes.    study types  the research studies in this systematic  review all used quantitative research  methods to measure cognitive outcomes.  some studies also employed qualitative  methods to measure other types of     outcomes, but those are outside the scope of  this review. figure 2 provides a general  breakdown of study types. most studies  were quasi‐experimental, employing a  controlled study design, but without  randomising the students to teaching  groups. fifty‐nine of the 122 (48%) studies  fit into this categorization, including  controlled before‐and‐after studies, as well  as studies that were post‐intervention,  single time point with a control group.  twenty (16%) of the studies were  experimental, employing randomisation and  a control group. the majority of these used  cluster randomisation, since educational  groups were often pre‐formed, and the  randomisation of individuals was beyond  the researcher’s control. some studies were  pre‐experimental, since they did not have a  comparison or control group and focused on  exploring the change in one group of  students, using a pre‐ and post‐test (i.e.,  before‐and‐after study).  nine studies (7%)  were observational, including longitudinal  and cross‐sectional studies. the single  ‘other’ study was large and  multidimensional in nature, which could  not be classified into any of the other  categories.    the studies had varying evaluation periods,  with the majority (51%, 62/122) covering a  time period of one semester. evidence based library and information practice 2006, 1:3  9 20 59 9 1 33 0 10 20 30 40 50 60 70 experimental quasiexperimental pre-experimental observational other   figure 2: study types    one group (7.4%, 9/122) of studies took  place over the course of one year, and a  further 8/122 (6.6%) took place over 2  semesters. the remaining studies’  evaluation periods ranged from 1 day to 6  years. studies varied with respect to when  learning outcomes were measured. the  largest number (28%, 34/122) tested learning  outcomes at the end of the semester.  another group of studies (25%, 30/122)  tested immediately following an instruction  session. the remainder ranged from the next  class following instruction, to several weeks  after instruction, to 3 years after the course  was completed. there were 17 studies (14%)  that did not report this variable.       41 16 16 15 11 11 7 5 3 1 7 0 5 10 15 20 25 30 35 40 45 ar ts lib ra ry sc ien ce sc ien ce s he alt h s cie nc es so cia l s cie nc es un ive rs ity in tro du cti on ed uc ati on bu sin es s co mm un ica tio ns la w no t s ta te d / no t a pp lic ab le   figure 3: subject areas covered  evidence based library and information practice 2006, 1:3  10   teaching methods    active learning  students are actively engaged in their own learning, with  the instructor taking on a facilitation role.   computer assisted instruction  a computer is used to deliver the instruction directly to  the student.  learner‐centred instruction  focus is on the individual student’s unique learning  needs.  self directed, independent  learning  learning in which the individual has primary  responsibility for his or her education.  traditional instruction  instructional material is transmitted to students from  teachers, and is a passive method of learning for students.    table 2: definitions of teaching methods      academic subject areas covered  the 122 studies represented a wide range of  undergraduate subject areas. most covered a  single subject area, but others included more  than one discipline. the highest percentage  of studies came from instruction related to  courses in english (30%, 37/122), included in  the arts category. another subject area with  many studies was library science, which  were mainly courses devoted to library  skills and taught by librarians for credit  (13%, 16/122). figure 3 shows the number of  studies arranged by broad subject discipline.     teaching methods  most studies compared two or more  teaching methods. of the 122 studies, 88  (72%) were comparative, while the  remaining 34 (28%) studies did not have a  comparison group. these non‐comparative  studies were generally evaluating the  effectiveness of a single teaching method to  determine whether students’ scores  improved following the instruction.    the teaching methods (table 2) used in the  studies varied, with the majority focused on  traditional methods of teaching (e.g., lecture,  demonstration). other studies dealt with  computer‐assisted instruction (cai) (e.g.,  web‐based tutorials); self‐directed,  independent learning (sdil) (e.g.,     workbooks); active learning (al) (e.g.,  problem based learning); and learner‐ centred instruction (lci) (e.g., individual  term paper counselling). while many  teaching methods included a combination of  methods, for the purposes of this review  they were grouped according to the primary  teaching method used (figure 4). further  details of the studies meeting the quality  filter are in appendix d.    total librarian contact time with students  ranged from 15 minutes to 3 hours per week  over the course of a semester. information  on the amount of instructional contact time  was lacking in 27% (33/122) of the studies.    instructional topics varied among studies,  although many common themes were  identified. the most common areas for  instruction were conducting library research  and research strategies (33%, 40/122), using  the catalogue (28%, 34/122), using reference  tools (27%, 33/122), an overview of the  library and its resources (21%, 26/122  studies), literature searching (20%, 24/122),  and using computerized or electronic  resources (19%, 23/122).    evidence based library and information practice 2006, 1:3  11 58 22 21 14 3 1 3 0 10 20 30 40 50 60 70 traditional computerassisted instruction self-directed, independent learning active learning learnercentred various methods unable to determine   figure 4: primary teaching methods used      outcome measures  the vast majority of studies measured  cognitive outcomes via student scores on a  post‐test (79%, 96/122). other outcome  measures included the graded quality of the  students’ bibliographies (7%,9/122), an  assessment of search tasks (6.5%,8/122),  assignment scores (5%,6/122), scores on term  papers or essays (3%, 4/122), and scores on a  general library skills survey (3%, 4/122).  some studies used more than one cognitive  outcome measure.     bloom’s taxonomy of educational objectives  (as revised by anderson and krathwohl)  was used to determine the level of cognitive  learning outcomes measured for the 122  included studies. the anderson and  krathwohl revision was used because it  provides an important update of bloom’s  1956 taxonomy and incorporates new  knowledge into the framework. anderson  and krathwohl note the following structure  of the cognitive process, covering six levels  of learning:    1.0 remember – retrieve relevant  knowledge from long‐term  memory.  1.1 recognizing  1.2 recalling  2.0 understand – construct  meaning from instructional  messages, including oral,  written, and graphic  communication.  2.1 interpreting  2.2 exemplifying  2.3 classifying  2.4 summarizing  2.5 inferring  2.6 comparing  2.7 explaining  3.0 apply – carry out or use a  procedure in a given situation.  3.1 executing  3.2 implementing  4.0 analyze – break material into  constituent parts and determine  how parts relate to one another  and to an overall structure or  purpose.  4.1 differentiating  4.2 organizing  evidence based library and information practice 2006, 1:3  12 4.3 attributing  5.0 evaluate – make judgements  based on criteria and standards.  5.1 checking  5.2 critiquing  6.0 create – put elements together  to form a coherent or functional  whole; reorganize elements into  a new pattern or structure.  6.1 generating  6.2 planning  6.3 producing    (anderson and  krathwohl 31)      figure 5 shows the levels of learning  outcomes measured by the 122 studies in  this review. more than one level of learning  was often assessed within a single study,  with ‘remember’ being the most commonly  assessed area. most studies focused on one  of the three more basic levels of learning  outcomes. there is an evident lack of  cognitive assessment research in the higher  order areas ‐‐ ‘analyze’, ‘evaluate’, and  ‘create’. also of note is that 23 studies (18%)  did not provide enough information about  what they were testing to determine what  learning outcomes were being measured.  the research hypothesis for this study  anticipated that the higher the level of  learning outcomes measured (using bloom’s  taxonomy), the more difficult it will be to  link effectiveness directly to library  instruction, and the less likely it is that the  library instruction will result in a positive  outcome. results of this systematic review  indicate the hypothesis is partially  supported. the higher levels of learning  outcomes were less frequently measured,  suggesting a possible difficulty in testing  these areas. furthermore, the results in the  studies that did measure higher level  outcomes varied widely, with a greater  percentage of those studies having mixed  results. the levels ‘remember’,  ‘understand’, and ‘apply’ seem to be easier  to measure in terms of cognitive outcomes,  while ‘analyze’, ‘evaluate’, and ‘create’ are  more complex concepts that do not lend  themselves as well to quantitative  measurement. as a result, the focus of  outcome measures is more evident in areas  where demonstrable outcomes can be  achieved.    62 49 56 21 7 1 23 0 10 20 30 40 50 60 70 remember understand apply analyze evaluate create unable to determine   figure 5: number of studies assessing the bloom (revised) levels of learning outcomes  evidence based library and information practice 2006, 1:3  13 in the areas of ‘analyze’, ‘evaluate’, and  ‘create’ there are more variables that cannot  easily be separated and tested, so it is more  difficult for students’ learning to be  attributed directly to the library‐led  instruction they received.     ii. effectiveness of different teaching  methods    reported outcomes by teaching method  the second objective of this review was to  determine effective teaching methods. in  order to do this, studies were categorised  and reviewed with a quality filter.  based on  the primary teaching method, studies were  categorised into five teaching methods:  active learning (al), computer assisted  instruction (cai), learner‐centred (lc), self‐ directed independent learning (sdil), and  traditional instruction (ti). four studies  could not be classified according to teaching  method.     study results were recorded to indicate if  the intervention had a positive, negative,  neutral, or mixed result, based upon the  outcomes measured. studies were subjected  to two quality filters. the first determined  whether the study was comparative (a  design whereby the intervention is  compared to another teaching method or  control). the second quality filter  determined whether the study outcome as  noted by the authors was based on data  assessed for statistical significance. these  quality filters were selected to remove bias  due to confounding (control groups should  be similar in every respect to the  experimental group except for the variable  being tested) and to account for differences  due to chance (statistical significance).  a  total of 73 studies met these two criteria.   one additional study (ridgeway), while not  using tests of statistical significance, did  have enough information for such tests to be  calculated, so it was included as well,  bringing the total to 74.  however, 19 of  these compared different modes of delivery  or other aspects within the same overall  teaching method, so they were removed  from the analysis, since they did not  compare different teaching methods.      the 55 remaining studies compared the  main teaching intervention to a different  teaching method, or to no instruction, and  those 55 studies form the basis of the  following review and meta‐analysis in order  to draw conclusions about the effectiveness  of different teaching methods. selected  characteristics of the 55 studies are noted in  appendix d.    traditional instruction  the traditional method of instruction (ti)  was the main intervention for the highest  number of studies in general (n=58).  however, only 18 of these met the  aforementioned quality criteria. table 3  provides the number of study outcomes for  ti, with a specific breakdown by the  comparison teaching method.    traditional instruction (n=18)    comparison  positive  neutral  negative  mixed  total  no instruction (ni)  12  3  0  1  16  cai  0  0  0  1    1  placebo  0  0  0  1    1    table 3: study outcomes for traditional instruction evidence based library and information practice 2006, 1:3  14  computer assisted instruction (n=17)    comparison  positive  neutral  negative  mixed  total  traditional  3  9  2  0  14  self‐directed  0  2  0  0    2  no instruction  0  1  0  0    1    table 4: study outcomes for computer assisted instruction     the single study comparing ti against a  placebo (an inactive treatment given instead  of the treatment being evaluated, in this case  a non‐instructional film), had a mixed result.   although ti accounts for much of the  research literature, there is a lack of  comparative research to determine  effectiveness versus other teaching methods.   where the comparison is versus no  instruction, the studies reviewed here show  a mainly positive result, with 12 of the 16  reporting an affirmative outcome in favour  of ti, suggesting that this instructional  method is more effective than no instruction.  within this subset of data, there was  sufficient information in 4 studies  comparing ti with no instruction, to  perform meta‐analysis and further  substantiate this finding, as explained in  section iii.    computer assisted instruction  there were 22 studies for which computer  assisted instruction (cai) was the main  intervention.  of these, 17 met the study      criteria (see table 4), including 14  comparing cai against ti, 2 comparing cai  against sdil, and 1 where cai was  compared to no instruction.  the group of 14  studies comparing cai with ti is the largest  single grouping of studies found in this  systematic review. based on the reported  outcomes of these cai studies, the overall  neutral result points toward cai being just  as effective as traditional teaching methods.  eight of these were selected for meta‐ analysis (see section iii).     self directed, independent learning  a total of 21 studies focused on sdil as the  main intervention, with 13 meeting the  aforementioned criteria; 7 studies  comparing sdil with ti and 6 comparing  sdil with ni (see table 5). the overall  positive and neutral outcomes suggest that  sdil is as effective as ti and more effective  than ni. four of the studies comparing sdil  to no instruction met the study criteria for  the meta‐analysis, discussed in section iii.    self‐directed independent learning (n=13)    comparison  positive  neutral  negative  mixed  total  traditional  2  5  0  0  7  no instruction  5  1  0  0  6    table 5: study outcomes for self‐directed, independent learning  evidence based library and information practice 2006, 1:3  15    active learning (n=5)    comparison  positive  neutral  negative  mixed  total  traditional  1  1  1  0  3  no instruction  1  0  0  1  2    table 6: study outcomes for active learning      active learning  fourteen studies focused on active learning  (al). of those, 5 met the criteria for further  analysis ‐‐ 3 compared al with ti, and 2  compared al with ni (see table 6). results  varied widely and indicate the studies  pertaining to al are not conclusive and  require further research. the hypothesis,  “instruction that encourages active  participation from students is more effective  than instructional modes that are passive,  such as lecture and demonstration,” could  not be addressed and meta‐analysis could  not be performed.    learner centred instruction  finally, three studies focused on learner‐ centred instruction (lci) as the main  intervention. two met the criteria for further  analysis; one comparing lci to ti and the  other to no instruction, both with a neutral  result (see table 7). the small numbers of  research studies in this area prevent further  analysis. further research is required to  build the evidence base with respect to  effectiveness of lci.        iii. meta‐analysis of comparative studies    a meta‐analysis is the pooling of estimates  of effect from individual studies asking the  same basic question.  meta‐analysis  provides an estimate of overall effect as well  as measuring the variability between these  studies. additionally, the meta‐analysis will  weight the estimates based on how many  subjects each study tested and on how much   between‐subject variability existed.  the  purpose of meta‐analysis is to provide some  guidance (ruling out chance and as much  bias as possible) around discrepant studies  and amongst small studies with  indeterminate results. further information  about conducting meta‐analysis can be  found in chalmers et al, cooper and hedges,  and egger et al. meta‐analysis was  performed on 16 studies for these  comparisons:  • traditional instruction vs. no  instruction (n=4)  • computer assisted instruction vs.  traditional instruction (n=8)  • self‐directed independent learning  vs. no instruction (n=4)    learner‐centred instruction (n=2)    comparison  positive  neutral  negative  mixed  total  traditional  0  1  0  0  1  no instruction  0  1  0  0  1    table 7: study outcomes for learner‐centred instruction evidence based library and information practice 2006, 1:3  16 these three comparisons were selected  because the studies in these areas reported  enough information to calculate a  standardized mean difference (smd), a  measure by which the studies could be  compared directly despite the use of  different measurement tools. smd is the  difference in means of the two groups being  compared, divided by an estimate of  standard deviation (sd). the number of  participants in both the intervention and  control groups, the mean score for each  group, and the standard deviation was used  to calculate smd.    the meta‐analysis allows for a visual  representation of the research data in a  meta‐graph using revman software (see  figures 6, 7, and 8).  each row in the meta‐ graph constitutes one study. this includes  the raw data (here, the sample size, mean,  and standard deviation for each teaching  method) as well as the estimate of effect  (here, the smd and the confidence intervals).  the best estimate of effect for each study is  represented by the square, and the  horizontal line running through the square  represents the confidence interval. the  vertical zero line is the line of no effect. if  the confidence interval crosses this line, the  estimate of effect is not significant. the  diamond at the bottom of the graph  represents the overall pooled estimate. its  centre is the best estimate of effect, and its  width represents the pooled confidence  intervals.    this meta‐analysis used standardized mean  difference (smd), the difference in means of  the two groups being compared, divided by  an estimate of standard deviation (sd). the  pooled sd from both instructional groups  was used. the smd is in units of sd. for  example, as shown in figure 7, the smd for  alexander’s study is 0.27 sds. therefore,  alexander found a difference of 0.27 sds  between the two groups, favouring cai.  however, this result was not statistically  significant, since the 95% confidence interval  (‐0.15 to 0.69 sds) includes zero. this means  that although the best estimate is 0.27 sds,  the true difference may lie anywhere  between ‐0.15 and 0.69 sds, including zero,  the estimate of no effect. an smd of 0.2 is  said to be small, 0.5 to be moderate, and 0.8  to be large (cohen 25).      the three meta‐analyses show the  individual estimates of each study plus the  overall pooled result. there are different  statistical methods for pooling. we chose a  method that accounts for random effects  between individual studies. this means that  not only did we expect subject‐to‐subject  variation, but also study‐to‐study variation,  such as differences due to type of computer  programs, teaching styles, evaluation  periods, and so on. including random effects  in the overall estimate of effect widens the  confidence intervals, thereby increasing the  uncertainty of the estimate.     traditional vs. no instruction  of the studies comparing ti to no  instruction, 4 provided sufficient  information to calculate smd.  three of the  four studies (cooper moore, lechner, and  toifel) provided multiple ti vs. ni  comparison groups within their studies, and  these were included in the meta‐analysis as  separate comparisons. this information is  summarized in the meta‐graph in figure 6.   the meta‐graph shows that for this group of  studies, there is a statistically significant  difference between the ti and no instruction  groups, suggesting that the ti method is  more effective than no instruction.   additionally, since the confidence interval  stretches from 0.14 to 0.48, we would  conclude that the size of effect would be  somewhere between small and moderate,  using cohen’s guidelines.  evidence based library and information practice 2006, 1:3  17 figure 6: standardized mean difference of traditional instruction vs. no instruction      computer assisted instruction vs. traditional  instruction  the cai subset of data included 8 studies  that were able to be compared using meta‐ analysis, since all 8 compared cai with ti,  and they provided sufficient information to  calculate the standardized mean difference  (smd). this information is summarized in  figure 7. the pooled smd estimate is ‐0.09  sds, and the 95% confidence intervals, ‐0.47  to 0.29 sds, cross the zero line, allowing for  the possibility of no difference between  groups. note also that our best estimate, ‐ 0.09 sds, is very small. thus, provided that  these studies are representative (of typical  and quality instruction methods), we can  say there is no evidence to support any  difference between cai and ti methods.  however, we cannot say that there is     absolutely no possibility of a true difference,  since the confidence limits include estimates  (e.g. ‐0.47, 0.29) that cohen (25) and other  researchers would consider important.      the meta‐graph shows that for this group of  studies, there is no difference between cai  and ti, suggesting that cai is just as  effective as traditional teaching methods.  results of this research did not support the  hypothesis that, “instruction that is taught  by a librarian face‐to‐face is more effective  than instruction that is computer‐based”.     self‐directed, independent learning vs.  no  instruction  there were 4 studies comparing sdil to no  instruction and provided enough data to  calculate smd. the meta‐graph in figure 8       figure 7: standardized mean difference of cai vs. traditional studies evidence based library and information practice 2006, 1:3  18 figure 8. standardized mean difference of self‐directed independent learning vs. no instruction.      shows the result of these studies. note that  broadway studied 1 sdil group and 2 ni  groups. we divided broadway’s sdil group  (halving the sample size, so that each of the  no instruction groups could be included in  the meta‐analysis as a separate study.  phipps and dickstein studied 4 instruction  groups (2 were self‐directed, and 2 had no  instruction). the studies were included in  the meta‐analysis as 2 separate studies.    as the graph shows, the data in these  studies qualitatively favours sdil over no  instruction, and all the studies individually  favour self‐directed teaching over no  instruction. however, since the size of effect  is heterogeneous (the confidence intervals  are non‐overlapping), we cannot conclude a  precise estimate of effect. size of benefit will  vary based on the specific content of self‐ directed instruction. we can, however,  conclude that sdil is more effective than no  instruction, since all studies are statistically  significant in favour of sdil.  additionally,  since the lower overall confidence limit is  greater than 0.8 (as are most of the lower  confidence limits for the individual studies),  we can expect that sdil will confer a larger  benefit compared to no instruction.     discussion    the 122 studies included in this review  provide a broad picture of the state of the  research pertaining to effective library  instruction methods for improving the  information skills of students at an  introductory, undergraduate level, using  cognitive outcomes.    most studies have been conducted on  traditional modes of instruction, however  these varied in terms of quality, and the  studies often lacked a comparison group.  when there was a comparison, it was  usually to ‘no instruction’. the next most  frequent teaching area was ‘computer  assisted instruction’. this group of studies  largely focused on a comparison with  ‘traditional instruction’, perhaps an  indication of the way in which that type of  instruction has sought to prove its worth  against face‐to‐face teaching. ‘self‐directed,  independent learning’ had a number of  positive outcomes as a whole when  compared to either ‘no instruction’ or  ‘traditional instruction’ methods. neither  ‘active learning’ nor ’learner‐centred’  instruction yielded enough studies to make  any meaningful comparisons.      looking at the studies as a whole, there  were a variety of research methods used,  some more rigorous than others. of the 122,  79 (65%) were experimental or quasi‐ experimental studies. in trying to measure  effectiveness, such studies indicate more  reliable methods, if those studies are  conducted properly.   evidence based library and information practice 2006, 1:3  19 there were several issues with methodology  and gaps in the information reported. one  apparent area was the lack of validated  research instruments. of the 122, only 9 (7%)  studies gave a detailed description of how  the research instrument was validated. a  validated research tool increases the  strength of the study, allowing us to trust  that the questions actually measure what  they intended to measure. a further 22 (18%)  studies indicated some attempt to pilot test  the instrument. the remaining studies did  not touch upon the importance of the  research instrument at all.       sample size ranged in size from those with  less than 100 participants to very large with  more than 600 participants throughout the  122 studies. in 8 (6.5%) studies the sample  size was not mentioned, and in another  group of 5 (4%) studies the number of  participants was stated in only approximate  numbers. there were 37 (30%) studies with  100 or fewer participants, 35 (29%) studies  had 101‐200 participants, 22 (18%) studies  had 201‐600 participants, and 9 (7%) studies  had more than 600 participants.   more than 25 (20%) of the 122 studies did  not perform any statistical analysis of the  data they collected. a further 12 (10%) noted  that statistical tests were performed, but did  not report their data. other methodological  issues with the studies were the lack of  reporting of loss to follow‐up between pre‐ test and post‐test, and biases that were  introduced, most noticeably selection bias of  participants.    gaps in reporting were often a problem. in  many cases, it was difficult to determine  what a study was trying to measure, since  there was no clear description of the  research instrument or the learning  outcomes being measured. sometimes there  was no description of what was being  taught or the learning environment. such  elements are important to give the reader a  complete picture, in order to determine  whether the study may be applicable to the  reader’s own situation.  with regard to  bloom’s levels of learning, many studies did  not report sufficient information to  determine the outcomes being measured.  for the studies that could be measured,  focus rested on levels of learning that were  lower and easier to measure.     the 16 studies (13%) that met the criteria for  meta‐analysis were of higher quality, and  the meta‐analysis method accounted for  differences between studies. this analysis  suggests three key points, which could  affect practice:  • computer assisted instruction is as  effective as traditional instruction.  • traditional instruction is more  effective than no instruction.  • self‐directed, independent learning  is more effective than no instruction.    the current research in our field does not  highlight any particular teaching method as  being more effective than any other teaching  method. while individual studies have  compared different teaching methods  directly, not enough of them have compared  those teaching methods in a meaningful  way. the only direct comparison between  two teaching methods that could be made  statistically using meta‐analysis was with  computer assisted instruction vs. traditional  instruction.    the neutral result showing that cai is just  as effective as ti indicates that decisions to  implement cai should not be disregarded  because of the belief that in‐person  instruction is more beneficial to the students.  as the research points to equal outcomes,  other factors such as time, costs, staffing,  and ability to reach greater numbers of  students need to be weighed in the decision  of using one teaching method over another.  qualitative research designs may have more  to add to the discussion of which method  evidence based library and information practice 2006, 1:3  20 may be preferable, based upon other factors  such as user preference.    the result showing that ti is more effective  than no instruction, provides a measure of  the worth of library instruction as it has  most commonly been performed in the past.  teaching seems to provide some benefit,  over and above not teaching at all. while  this is positive, more research is needed to  determine whether or not ti is the best  method of instruction. studies comparing  traditional teaching methods to al and to  sdil, for example, should be considered, so  that we can more fully understand the  effectiveness of the ways in which we teach  information literacy skills.    the positive results for sdil versus no  instruction, should not be overlooked. as  with cai, this method of instruction  provides a solution that does not require  instruction to be face‐to‐face. it is a possible  solution for those wanting to implement  information literacy initiatives but not  having the resources to teach in person.  again, more research is required to test this  teaching method against other methods,  including traditional instruction and cai.    active learning and learner‐centred  instruction did not have enough studies to  reach any conclusions about their  effectiveness. these areas require further  research to create a body of studies  comparing these teaching methods to other  more established methods, such as  traditional and computer assisted  instruction.    possible confounders of this systematic  review and meta‐analysis include the  dominance of u.s. studies in the evidence  base, which may impact transferability of  these results. another factor is that only  studies reported in the english language  were included; results may not be able to be  generalized to practice in other countries.   the authors relied on study design as a  measure of quality, but study design in and  of itself does not constitute a good study.   just because a study has a design that  should be more rigorous, does not mean  that it has been well conducted. however,  for the questions of effectiveness that the  authors were attempting to answer, the  comparative study design was an  appropriate filter. finally, results were not  compared across subject areas, so results  may not necessarily be transferable from  one subject area to another. readers should  look at the detail provided in appendix d  for studies that apply specifically to their  subject areas.    conclusion    the goal of conducting this systematic  review was to find evidence about  information literacy instructional methods  that may have a direct impact on the way  academic librarians approach teaching  information literacy to undergraduate  students. this review provides a general  picture of the research that has been done in  this area, highlights some of the better  research, draws together current evidence  on what teaching methods are most effective,  highlights problems with this body of  research, and outlines what we can learn  from the current evidence base.     to determine effectiveness of teaching  methods, more comparative research needs  to be done across different teaching methods  using sound research methodologies and  validated research tools.  careful  consideration should be given to the most  appropriate research method, and  researchers should determine possible areas  of comparison before beginning the study.  the norm is to compare with no instruction,  or to the standard traditional methods.  studies comparing to active learning,  computer assisted instruction, and self‐ directed independent learning would  evidence based library and information practice 2006, 1:3  21 greatly enrich the research literature.  building upon existing studies with good  methodologies and validated research tools  would enrich our evidence base and  contribute to the growth of knowledge  about effectiveness of particular teaching  methods.    this systematic review of the literature  provides librarians who teach information  literacy skills with an overview of the  research literature in this area. despite  certain flaws in the studies which have been  outlined, this review provides pragmatic  evidence of the effectiveness of several  methods of teaching information literacy.   the lack of comparative studies, however,  does not allow us to conclude anything  about the effectiveness of teaching methods  in areas such as active learning and learner‐ centred instruction. in the areas of  traditional instruction, cai, and self‐ directed independent learning, only three  conclusions could be made. the systematic  method of arriving at those conclusions  provides us with clear evidence in those  specific areas. however, there is not enough  evidence to determine which teaching  method is best. we should approach  information literacy grounded in this  knowledge, and move forward to build  upon it, contributing to the evidence base in  the future.    nevertheless, this systematic review will aid  librarians teaching information literacy  skills to undergraduate students to be more  informed about the teaching methods they  use. while there is not enough current  evidence to persuade instructional librarians  to change their teaching practices from one  method to another, librarians should know  that the evidence base for effectiveness  based on cognitive outcomes is weak, and  make their decisions with that in mind. this  systematic review should be a call for those  involved with information literacy  instruction to contribute to the research  knowledge base in their field of interest, so  that better decisions can be made in the  future. researchers, particularly those in  countries other than the u.s., should  continue to build upon the research results  summarised here to provide more research  evidence of effective library instruction  methods.       works cited    anderson, lorin w., and david r.  krathwohl, eds. a taxonomy for  learning, teaching, and assessing: a  revision of bloomʹs taxonomy of  educational objectives. new york:  longman, 2001.    baruchson‐arbib, shifra, and jenny  bronstein. “a view to the future of the  library and information science  profession: a delphi study.” journal of  the american society for information  science and technology 53.5 (mar.  2002): 397‐408.  brettle, alison. “information skills training:  a systematic review of the  literature.” health information and  libraries journal 20.supp.1 (2003): 3‐9.    chalmers i., l.v. hedges, and h. cooper.  “a brief history of research  synthesis.” evaluation & the health  professions 25.1 (2002):12‐37.    cohen, j. statistical power analysis for the  behavioral sciences, rev. ed. hillsdale,  nj: lawrence erlbaum associates, 1988.     cooper, h., and l.v. hedges, eds. the  handbook of research synthesis. sage  publications, 1994.    egger, m., g. davey smith, and d.g.  altman. systematic reviews in health  care. london: bmj books, 2001.     evidence based library and information practice 2006, 1:3  22 garg, anupama, and kathleen m. turtle.  “effectiveness of training health  professionals in literature search skills  using electronic health databases – a  critical appraisal.” health information  and libraries journal 20.1 (2003): 33‐41.    godwin, peter. “learning and teaching –  the librarian’s contribution: an  introduction.” vine 31.1 (2001): 3‐4.    morrison, jillian m., frank sullivan,  elizabeth murray, and brian jolly.  “evidence‐based education:  development of an instrument to  critically appraise reports of  educational interventions.” medical  education 33 (1999): 890‐3.    national forum on information literacy.  “what is information literacy? “ 20  july 2006 <http://www.infolit.org/>.    peacock, judith. “teaching skills for  teaching librarians: postcards from  the edge of the educational  paradigm.” australian academic &  research libraries 32.1 (2001): 26‐42.   26 april 2006  <http://www.alia.org.au/publishing/aar l/32.1/full.text/jpeacock.html>.                                    acknowledgements    the authors would like to thank the  anonymous peer reviewers whose  comments improved the quality of this  paper.    denise koufogiannakis would like to thank  the university of alberta libraries for  granting her a one year professional leave,  during which time she conducted this  research. she would also like to thank  andrew booth, who generously provided  invaluable advice and support during the  research process.     a preliminary version of this paper was  presented at the 3rd international evidence  based librarianship conference in brisbane,  australia, october 2005. see:   <http://conferences.alia.org.au/ebl2005/kouf ogiannakis.pdf>. http://www.infolit.org/ http://www.alia.org.au/publishing/aar http://conferences.alia.org.au/ebl2005/kouf evidence based library and information practice 2006, 1:3  23 appendix a: literature search process    searches were conducted between september and november, 2004 and updated in may, 2005.    1. lisa (csa)  1969 ‐ present  search strategy:  ((de=(user training) or (information literacy) or (computer assisted instruction) or (information  literacy tutorial) or (education activities)) or ʺlibrary instruct*ʺ or ʺlibrary educat*ʺ or ʺlibrar* teach*ʺ  or ʺuser train*ʺ or ʺuser educat*ʺ or ʺonline tutor*ʺ or ʺlibrary skill*ʺ or ʺinformation skill*ʺ) and  (de=students or undergrad* or ʺfirst yearʺ or bachelor* or ʺpost secondaryʺ)  number of documents retrieved: 1,443.    2. library literature (silverplatter)  1984 ‐ present  search strategy:         #1 bibliographic‐instruction‐college‐and‐university‐students in de (1,285 records)       #2 bibliographic‐instruction‐junior‐and‐community‐college‐students in de (47 records)       #3  #1 or #2 (1,332 records)         #4 computer‐assisted‐instruction in de (909 records)       #5 end‐user‐searching‐teaching in de (750 records)       #6 bibliographic‐instruction in de (207 records)       #7 internet‐teaching in de (398 records)       #8 information literacy (593 records)       #9 library instruct* (535 records)     #10 librar* teach* (2,351 records)     #11 user train* (41 records)     #12 user educat* (213 records)     #13 online tutor* (16 records)     #14 library skill* (159 records)     #15 information skill* (206 records)     #16 #4 or #5 or #6 or #7 or #8 or #9 or #10 or #11 or #12 or #13 or #14 or #15 (5,708 records)     #17 undergrad* (508 records)     #18 first year (77 records)     #19 bachelor* (9 records)     #20 post secondary (8 records)     #21 postsecondary (12 records)     #22 #17 or #18 or #19 or #20 or #21 (605 records)     #23 #16 and #22 (108 records)     #24 #3 or #23 (1,383 records)  number of documents retrieved: 1,383.    3. eric (ovid) 1966 ‐ july 2004  search strategy:  1     exp information literacy/ (984)  2     exp course integrated library instruction/ (409)  3     exp library instruction/ (3,082)  4     exp information skills/ (1,950)  5     exp library skills/ (1,542)  6     library instruct$.mp. [mp=abstract, title, headings word, identifiers, eric digests full text] (3,231)  7     library educat$.mp. [mp=abstract, title, headings word, identifiers, eric digests full text] (3,133)  8     librar$ teach$.mp. [mp=abstract, title, headings word, identifiers, eric digests full text] (1,068)  evidence based library and information practice 2006, 1:3  24 9     user train$.mp. [mp=abstract, title, headings word, identifiers, eric digests full text] (255)  10     user educat$.mp. [mp=abstract, title, headings word, identifiers, eric digests full text] (224)  11     information literacy.mp. [mp=abstract, title, headings word, identifiers, eric digests full text]  (1,274)  12     online tutor$.mp. [mp=abstract, title, headings word, identifiers, eric digests full text] (55)  13     library skill$.mp. [mp=abstract, title, headings word, identifiers, eric digests full text] (1,749)  14     information skill$.mp. [mp=abstract, title, headings word, identifiers, eric digests full text]  (1,114)  15     or/1‐14 (9,234)  16     exp undergraduate students/ (5,855)  17     exp college freshmen/ (5,672)  18     exp college freshmen/ or exp college juniors/ or exp college seniors/ or exp college  sophomores/ (6,057)  19     undergrad$.mp. [mp=abstract, title, headings word, identifiers, eric digests full text] (27,073)  20     (postsecondary or post‐secondary or post secondary).mp. [mp=abstract, title, headings word,  identifiers, eric digests full text] (38,914)  21     bibliographic instruct$.mp. (632)  22     15 or 21 (9,278)  23     bachelor$.mp. [mp=abstract, title, headings word, identifiers, eric digests full text] (3,905)  24     (first year or first‐year).mp. [mp=abstract, title, headings word, identifiers, eric digests full text]  (8,997)  25     or/16‐20 (70,037)  26     or/23‐25 (79,802)  27     26 and 22 (885)  28     exp undergraduate study/ (7,066)  29     26 or 28 (79,802)  30     29 and 22 (885)  31     from 30 keep 1‐885 (885)  number of documents retrieved: 885    4. inspec (axiom/iop) 1968 – present  search strategy:  ((information literacy or information instruction or information searching skills or library instruct*  or library educat* or librar* teach* or user train* or user educat* or online tutor* or library skill* or  information skill* or bibliographic instruct*) and (undergrad* or first year or bachelor* or post  secondary or freshman or academic librar*)) <in> (ti,ab,ci,ui)  number of documents retrieved: 232    5. academic search premier (ebsco) 1975 ‐present  search strategy:  s1  su (information literacy or library education or  information services ‐ user education or computer‐ assisted instruction or user education)      5,366 s2  (bibliographic instruct* or library instruct* or  library educat* or librar* teach* or user train* or  user educat* or online tutor* or library skill* or  information skill*)      2,868 s3  (undergrad* or first year or bachelor* or  postsecondary or academic librar*)    35,765 evidence based library and information practice 2006, 1:3  25 s4  (s2 or s1)      8,071 s5  (s4 and s3)         767 number of documents retrieved: 767    6. educational research abstracts 1995‐present  search strategy:  ((keywords/ʺlibrary instruct*ʺ or keywords/ʺbibliographic instruct*ʺ or keywords/ʺinformation  literacyʺ or keywords/ʺuser train*ʺ or keywords/ʺlibrary educat*ʺ or keywords/ʺlibrary teach*ʺ or  keywords/ʺonline tutor*ʺ or keywords/ʺlibrary skill*ʺ or keywords/ʺinformation skill*ʺ) and  (keywords/undergrad* or keywords/ʺfirst yearʺ or keywords/bachelor* or keywords/postsecondary  or keywords/ʺpost secondaryʺ or keywords/ʺacademic librar*ʺ))  number of documents retrieved: 6    7. cinahl (ovid)  1982 – present  search strategy:  1     exp library user education/ (357)  2     (library instruct$ or library educat$ or librar$ teach$ or information literacy or library skill$ or  information skill$).mp. [mp=title, cinahl subject headings, abstract, instrumentation] (205)  3     1 or 2 (473)  4     exp students, college/ (2,911)  5     (undergrad$ or first year or bachelor$ or postsecondary or post secondary).mp.   [mp=title, cinahl subject headings, abstract, instrumentation] (4,314)  6     exp libraries, academic/ (928)  7     or/4‐6 (7,719)  8     3 and 7 (124)  9     from 8 keep 1‐124 (124)  number of documents retrieved: 124    8. web of science (all sections) 1945‐present  search strategy:  #1   1,665 ts=(librar* instruct* or bibliographic instruct* or library educat* or librar*  teach* or user train* or user educat* or information literacy or online  tutor* or library skill* or information skill*)  doctype=all document types; language=all languages; databases=sci‐ expanded, ssci, a&hci; timespan=1945‐2004  #2   56,264 ts=(undergrad* or first year or first‐year or bachelor* or postsecondary  or post‐secondary or academic librar* or university student* or college  student*)  doctype=all document types; language=all languages; databases=sci‐ expanded, ssci, a&hci; timespan=1945‐2004  #3  185 #1 and #2  doctype=all document types; language=all languages; databases=sci‐ expanded, ssci, a&hci; timespan=1945‐2004   number of documents retrieved: 185    9. dissertation abstracts / digital dissertations (proquest)   search strategy:  evidence based library and information practice 2006, 1:3  26 1   key(library instruction) or key(bibliographic  instruction) or key(information literacy)        138 2   key(library education) or key(user education) or  key(library skill?) or key(information skill?)        171 3   #2 or #1        296 4   key(undergrad?) or key(first year) or key(first‐ year) or key(bachelor?) or key(postsecondary)  or key(post‐secondary) or key(academic librar?)  or key(university student?) or key(college  student?)   40,957 5  #3 and #4        70 number of documents retrieved: 70    10. conference papers index (csa)  search strategy:  (ʺinformation literacyʺ or ʺlibrary instruct*ʺ or ʺlibrary educat*ʺ or ʺlibrar* teach*ʺ or ʺuser train*ʺ or  ʺuser educat*ʺ or ʺonline tutor*ʺ or ʺlibrary skill*ʺ or ʺinformation skill*ʺ) and (undergrad* or ʺfirst  yearʺ or bachelor* or ʺpost secondaryʺ or academic librar*)  number of documents retrieved: 2    11. sigle (system for information on grey literature in europe))  1980‐present  search strategy:  (librar? instruct? or bibliographic instruct? or library educat? or librar?  (3a)teach? or user train? or user educat? or information literacy or  online tutor? or library skill? or information skill?) and (undergrad? or  first year or first‐year or bachelor? or postsecondary or post‐secondary  or academic librar? or university student? or college student?)  number of documents retrieved: 0    12. ceruk: current education research in the uk  browsed keyword terms: librarians; library and information services; information skills  number of documents retrieved: 15    13. education‐line <http://www.leeds.ac.uk/educol>  query: (ʺlibrary instruct$ʺ or ʺbibliographic instruct$ʺ or ʺlibrary educat$ʺ  or ʺlibrar$ teach$ʺ or ʺuser train$ʺ or ʺuser educat$ʺ or ʺinformation  literacyʺ or ʺonline tutor$ʺ or ʺlibrary skill$ʺ or ʺinformation skill$ʺ) and  (undergrad$ or ʺfirst yearʺ or ʺfirst‐yearʺ or bachelor$ or postsecondary  or ʺpost‐secondaryʺ or ʺuniversity student$ʺ or ʺcollege student$ʺ or  ʺacademic librar$ʺ)  number of documents retrieved: 11    14.  british education index (dialog)  number of documents retrieved: 16    15.  australian education index (dialog)  number of documents retrieved: 30   http://www.leeds.ac.uk/educol evidence based library and information practice 2006, 1:3  27 appendix b: studies included     1. ackerson, linda g., jeanne g. howard, and virginia e. young, ʺassessing the relationship  between library instruction methods and the quality of undergraduate research.ʺ research  strategies 9.3 (1991): 139‐41. (results also published in ackerson et al., 1994.)  2. ackerson, linda g., and virginia e. young.  “evaluating the impact of library instruction  methods on the quality of student research (three‐year study at the university of  alabama).” research strategies 12 (1994): 132‐44. (results also published in ackerson et al.,  1991.)     3. alexander, linda b. libs 1000: a credit course in library skills at east carolina university.   1994. eric document reproduction service ed376818.  4. alexander, linda b.  “library skills instruction: a comparison of students in a web‐based  course versus a traditional instruction course.”  diss. univ. of louisville, may 2000.    (results also published in alexander and smith, 2001.)  5. alexander, linda b., and robert c. smith. ʺresearch findings of a library skills instruction  web course.ʺ portal 1.3 (2001): 309‐28.  (results also published in alexander, 2000.)  6. andretta, susie. “legal information literacy: a pilot study.” new library world  102.1166/1167 (2001): 255‐64.     7. arnold, julie, robert kackley, and stephen fortune. “hands‐on learning for freshman  engineering students.” issues in science and technology librarianship 37 (spring 2003). 19  may 2006 <http://www.istl.org/03‐spring/article3.html>.     8. axeen, marina esther. teaching library use to undergraduates‐‐comparison of computer‐ based instruction and the conventional lecture. final report. urbana: univ. of illinois, 1967.  eric document reproduction service  ed014316.     9. benefiel, candace r., and joe jaros. ʺplanning and testing a self‐guided taped tour in an  academic library.ʺ  rq 29.2 (winter 1989): 199‐208.     10. benham, frances. college library technology and cooperation grants program interim  performance report higher education act, title ii‐d. tuscaloosa: alabama univ.,1988. eric  document reproduction service ed346853.     11. bolt, janice ann havlicek. “a study of the effects of a bibliographic instruction course on  achievement and retention of college students.” diss. florida state university, 1986.    12.  borgman, christine l. “the user’s mental model of an information retrieval system: an  experiment on a prototype online catalog.” international journal of man‐machine studies  24.1 (1986): 47‐64.  (accessed via reprint in international journal of human‐computer studies  51.2 (1999): 435‐52).   13. bostian, rebecca, and anne robbins. ʺeffective instruction for searching cd‐rom indexes.ʺ  laserdisk professional 3.1 (1990): 14‐7.     http://www.istl.org/03%e2%80%90spring/article3.html%00%00 evidence based library and information practice 2006, 1:3  28 14. bradigan, pamela s., and carol a. mularski. “end‐user searching in a medical school  curriculum: an evaluated modular approach.” bulletin of the medical library association  77.4 (1989 oct.): 348‐56.     15. breivik, patricia senn.  open admissions and the academic library. chicago: ala, 1977.   “brooklyn college: a test case” p.49‐66.    16. bren, barbara, beth hillemann, and victoria topp. ʺeffectiveness of hands‐on instruction of  electronic resources.ʺ research strategies 16.1 (1998): 41‐51.     17. broadway, marsha denise. ʺself‐directed instruction in query formation and presentation  for college students (bibliographic instruction, reference interview, communication).ʺ  diss.  florida state university, 1985.    18. brown, cecelia, and lee r. krumholz. “integrating information literacy into the science  curriculum.” college and research libraries 63.2 (mar. 2002): 111‐23.       19. brown, lyn s., and g. jeremiah ryan. the relationship of time to effectiveness in research  skills instruction for students at philadelphia college of bible.  pennsylvania: philadelphia  college of bible, 1995. eric document reproduction service ed402931.     20. buchanan, nancy l., karen rupp‐serrano, and johanne lagrange. ʺthe effectiveness of a  projected computerized presentation in teaching online library catalog searching.ʺ college  and research libraries 53.4 (july 1992): 307‐18.      21. bushong, sara. utilization of powerpoint presentation software in library instruction of  subject specific reference sources.  kent state university, 1998. eric document  reproduction service ed423914.     22. cameron, lynn, and james hart.  “assessment of psyclit competence, attitudes and  instructional methods.”  teaching of psychology 19.4 (1992): 239‐42.       23. champion, brian. computer assisted instruction and bibliographic instruction: preliminary  data on the use of plato in the bi program of the humanities and social sciences library,  university of alberta.  university of alberta, 1986. eric document reproduction service  ed284567.  24. cherry, joan m., weijing yuan, and marshall clinton. “evaluating the effectiveness of a  concept‐based computer tutorial for opac users.” college and research libraries 55.4  (july 1994): 355‐64.      25. cherry, joan m., and marshall clinton. “an experimental investigation of two types of  instruction for opac users.” canadian journal of information science 16 (dec. 1991): 2‐22.     26. churkovich, marion, and christine oughtred. ʺcan an online tutorial pass the test for  library instruction? an evaluation and comparison of library skills instruction methods for  first year students at deakin university.ʺ australian academic and research libraries 33.1  (2002): 25‐38.      evidence based library and information practice 2006, 1:3  29 27. colaric, susan m. ʺinstruction for web searching: an empirical study.ʺ  college & research  libraries 64.2 (mar. 2003): 111‐22.     28. cooper moore, anne. ʺthe impact of hands‐on information literacy instruction on  learning/knowledge of information literacy concepts and mastery of the research process  in college courses: a quasi‐experimental study.ʺ diss. new mexico state university, 2001.    29. cudiner, shelley, and oskar harmon. ʺcomparing the effectiveness of different presentation  formats for workshops on introductory library skills.ʺ research strategies 18.1 (2001): 49‐61.    30. currie, margaret, elaine goettler, and sandra mccaskill. “evaluating the relationship  between library skills and library instruction.” canadian library journal 39.1 (feb. 1982):  35‐7.    31. damron, james a. ʺin‐coming provisional students: comparing pre‐ and post‐test scores  for a summer library program.ʺ virginia libraries 49.2 (2003): 21‐2.     32. daugherty, timothy k., and elizabeth w. carter.  “assessment of outcome‐focused library  instruction in psychology.”  journal of instructional psychology 24.1 (1997): 29‐33.       33. davis, dorothy f.  “a comparison of bibliographic instruction methods on cd‐rom  databases.”  research strategies 11.3 (1993): 156‐63.      34.  davis, h. scott. a comparison of three instructional approaches to online catalog  instruction: what students prefer may work best.  1989. eric document reproduction  service ed314083.     35. delong, edward james. an evaluative report of the richmond college freshman library  instruction program.  virginia: richmond univ., 1978. eric document reproduction service  ed157547.  36. dodgen, lynda, sarah naper, olia palmer, and adrian rapp. “not so sili: sociology  information literacy infusion as the focus of faculty and librarian collaboration.”  community & junior college libraries 11.4 (2003): 27‐33.       37. donegan, patricia morris, ralph e. domas, and john r. deosdade. ʺthe comparable effects  of term paper counseling and group instruction sessions.ʺ college and research libraries  50 (1989): 195‐205.      38. dorsch, josephine l., meenakshy k. aiyer, and lynne e. meyer. “impact of an evidence‐ based medicine curriculum on medical students’ attitudes and skills.” journal of the  medical library association 92.4 (2004): 397‐406.    39. edwards, sherri.  “effects of a self‐paced workbook on students’ skills and attitudes.”   research strategies 9.4 (1991): 180‐8.      40. ellsbury, susan h. feasibility study: library instruction in specific science disciplines using  the self‐paced workbook adapted to departmental needs, mitchell memorial library, fall  1981.  mississippi: mississippi state univ., 1981. eric document reproduction service  evidence based library and information practice 2006, 1:3  30 ed225589.    41. emmons, mark, and wanda martin. ʺengaging conversation: evaluating the contribution of  library instruction to the quality of student research.ʺ college and research libraries 63.6  (2002): 545‐60.      42. eyman, david h., and alven c. nunley, jr. the effectiveness of library science 1011 in  teaching bibliographical skills. tahlequah: northeastern oklahoma state univ., 1977. eric  document reproduction service ed150962.    43. facinelli, jaclyn. music students and bibliography instruction: a study.  ohio: university of  akron, 1982. eric document reproduction service ed242322.   44. fox, lynne m., judith m. richter, and nancy e. white. “a multidimensional evaluation of a  nursing information‐literacy program.” bulletin of the medical library association 84.2  (apr. 1996): 182‐90.      45. franklin, godfrey, and ronald c. toifel. ʺthe effects of bi on library knowledge and skills  among education students.ʺ  research strategies 12.4 (fall 1994): 224‐37.    46. frasca, michael a., josephine l. dorsch, jean c. aldag, and richard g. christiansen. “a  multidisciplinary approach to information management and critical appraisal instruction:  a controlled study.” bulletin of the medical library association 80.1 (jan.1992): 23‐8.    47. fry, thomas k., and joan r. kaplowitz. ʺthe english 3 library instruction program at ucla:  a follow‐up study.ʺ  research strategies 6.3 (1988): 100‐8.      48. germain, carol anne, trudi e. jacobson, and sue a. kaczor.  ʺa comparison of the  effectiveness of presentation formats for instruction: teaching first‐year students.ʺ college  and research libraries 61.1 (jan. 2000): 65‐72.    49. gutierrez, carolyn, and jianrong wang. ʺa comparison of an electronic vs. print workbook  for information literacy instruction.ʺ the journal of academic librarianship 27.3 (2001): 208‐ 12.    50. hales, celia. ʺbasic bi at the university of north carolina at charlotte: results of an  experiment.ʺ southeastern librarian  (fall 1985): 76‐7.      51. hall, virginia b., daniel krautheim, and baljit s. hansra. “a slide‐tape program for  beginning pharmacy students: effect on learning.” bulletin of the medical library  association 65.4 (oct. 1977): 443‐5.      52. hardesty, larry, nicholas p. lovrich, jr., and james mannon. ʺevaluating library‐use  instruction.ʺ college and research libraries 40.4 (july 1979): 309‐17.      53. hardesty, larry, nicholas p. lovrich, jr., and james mannon.  “library use instruction:  assessment of the long‐term effects.” college & research libraries 43.1 (1982): 38‐46.     54. hausrath, don, shuk‐chun auyeung, jo anne howell, and kaye bedell. “integrating  evidence based library and information practice 2006, 1:3  31 information competencies into the allied health curriculum at gavilan college.”  community & junior college libraries 11.2 (2003): 13‐49.     55. holman, lucy. ʺa comparison of computer‐assisted instruction and classroom  bibliographic instruction.ʺ reference and user services quarterly 40.1 (2000): 53‐60.    56. holt, joan s., and steven falk. “evaluation of library workbooks in a community college  setting.” reference librarian 11 (fall‐winter 1984): 321‐34.      57. hooks, james d. teaching library skills to academically unprepared college freshmen.  1986. eric document reproduction service ed296740.  58. howze, philip c., and dana e. smith. “library instruction as independent study: the  summer enrichment program experiment at iowa state university.” reference services  review; 23.4 (1995): 75‐82.      59. jacobsen, gertrude n., and michael j. albright. “motivation via videotape: key to  undergraduate library instruction in the research library.” journal of academic  librarianship 9.5 (nov. 1983): 270‐5.     60. jennerich, elaine zaremba, and bessie hess smith. ʺa bibliographic instruction program in  music.ʺ college and research libraries 40.3 (may 1979): 226‐33.     61. johnson, kathleen a., and barbara s. plake. “evaluation of plato library instructional  lessons: another view.” journal of acadmic librarianship 6.3 (1980): 154‐8.    62. julien, heidi, and stuart boon. ʺassessing instructional outcomes in canadian academic  libraries.ʺ library & information science research 26.2 (2004): 121‐39.    63. kaplowitz, joan, and janice contini. “computer‐assisted instruction: is it an option for  bibliographic instruction in large undergraduate survey classes?” college and research  libraries 59.1 (jan. 1998): 19‐27.     64. kaplowitz, joan r. ʺa pre‐ and post‐test evaluation of the english 3‐library instruction  program at ucla.ʺ research strategies 4 (1986): 11‐7.      65. keever, ellen h., and james c. raymond. “integrated library instruction on the university  campus: experiment at the university of alabama.” journal of academic librarianship 2.4  (sept. 1976): 185‐7.     66. kimsey, mary b., and s. lyn cameron. “teaching and assessing information literacy in a  geography program.” journal of geography 104.1 (jan.‐feb. 2005): 17‐23.    67. kirk, thomas.  “a comparison of two methods of library instruction for students in  introductory biology.”  college and research libraries 32 (1971): 465‐74.      68. kohl, david, and lizabeth wilson.  “effectiveness of course‐integrated bibliographic  instruction in improving coursework.”  rq 27 (1986): 206‐11.    evidence based library and information practice 2006, 1:3  32 69. kochtanek, thomas r. measuring the effectiveness of a library skills program. arlington  educational resources information center. 1980. eric document reproduction service  ed190157.  70. koufogiannakis, denise, jeanette buckingham, arif alibhai, and david rayner. “impact of  librarians in first year medical and dental student problem‐based learning (pbl) groups:  a controlled study.” health information and libraries journal 22.3 (sept. 2005): 189‐95.    71. langley, linda. “the effects of a credit course in bibliographic instruction.” technicalities 2  (1987): 3‐7.    72. lawson, mollie d. ʺassessment of a college freshman course in information resources.ʺ  library review 48.2 (1999): 73‐8.      73. lawson, v. lonnie. ʺusing a computer‐assisted instruction program to replace the  traditional library tour: an experimental study.ʺ rq 29 (fall 1989): 71‐9.     74. lechner, judith v. ʺbibliographic instruction evaluation: a study testing the correlations  among five measures of the impact of a bibliographic instruction program on  undergraduatesʹ information searching behavior in libraries.ʺ diss. university of california,  1989.    75. linton, anne m., patricia h. wilson, alexandra gomes, laura abate, and matthew  mintz. ”evaluation of evidence‐based medicine search skills in the clinical years.” medical  reference services quarterly 23.2 (summer 2004): 21‐31.     76. mackey, neosha, charlotte dugan, willa garrett, and c. lynne freeman. “teaching with  hypercard in place of a textbook.” computers in libraries 12 (oct. 1992): 22‐6.    77. macpherson, karen. “an information processing model of undergraduate electronic  database information retrieval.” journal of the american society for information science  and technology 55.4 (2004): 333‐47.   (results also published in macpherson, 2004, australian  academic and research libraries.)  78. macpherson, karen. “undergraduate information literacy: a teaching framework.”  australian academic and research libraries 35.3 (2004): 226‐42.  (results also published in  macpherson, 2004, journal of the american society for information science and technology.)  79.  madland, denise, and marian a. smith.  “computer assisted instruction for teaching  conceptual library skills to remedial students.”  research strategies 6 (1988): 52‐64.      80. magi, trina j. “whatʹs best for students? comparing the effectiveness of a traditional print  pathfinder and a web‐based research tool.” portal: libraries and the academy 3.4 (oct.  2003): 671‐86.     81. marcus, sandra, and sheila beck. ʺa library adventure: comparing a treasure hunt with a  traditional freshman orientation tour.ʺ college and research libraries  64.1 (2003): 23‐44.     82. marfleet, b. gregory, and brian j. dille. information literacy and the undergraduate  evidence based library and information practice 2006, 1:3  33 methods curriculum. 2003. eric document reproduction service ed479704.    83. meehan‐black, elizabeth c. the effects of two methods of course‐related library  instruction on undergraduatesʹ library skills and attitudes.  1981. eric document  reproduction service ed246918.   84. nichols, james, barbara shaffer, and karen shockey. ʺchanging the face of instruction: is  online or in‐class more effective?ʺ college and research libraries  64.5 (2003): 378‐88.     85. nielsen, brian, and betsy k. baker. ʺeducating the online catalog user: a model evaluation  study.ʺ library trends 35.4 (1987): 571‐85.      86. orme, william a. “a study of the residual impact of the texas information literacy tutorial  on the information‐seeking ability of first year college students.” college and research  libraries 65.3 (may 2004): 205‐15.    87. paglia, alison, and annie donahue. “collaboration works: integrating information  competencies into the psychology curricula.” reference services review 31.4 (2003): 320‐8.      88. palmer, stuart, and barry tucker. “planning, delivery and evaluation of information  literacy training for engineering and technology students.” australian academic and  research libraries 35.1 (mar. 2004): 16‐34.      89. parang, elizabeth, melinda raine, and trisha stevenson. “redesigning freshman seminar  library instruction based on information competencies.” research strategies 17.4 (2000):  269‐80.     90. pearson, penelope, and virginia tiefel. “evaluating undergraduate library instruction at the  ohio state university.” journal of academic librarianship 7.6 (jan. 1982): 351‐7.      91. phipps, shelley, and ruth dickstein. “the library skills program at the university of  arizona: testing, evaluation and critique.” journal of academic librarianship 5 (sept. 1979):  205‐14.    92. phillips, linda l., and e. ann raup. ʺcomparing methods for teaching use of periodical  indexes.ʺ journal of academic librarianship 4.6 (jan. 1979): 420‐3.     93. popa, opritsa, deborah a. metzger, and james a. singleton. ʺteaching search techniques on  the computerized catalog and on the traditional card catalog: a comparative study.ʺ  college and research libraries 49.3 (1988): 263‐74.     94. portman, chris a., and adrienne julius roush.  “assessing the effects of library  instruction.”  journal of academic librarianship 30.6 (nov. 2004): 461‐5.    95. prorak, diane, tania gottschalk, and michael pollastro. ʺteaching method and psychological  type in bibliographic instruction: effect on student learning and confidence.ʺ rq 33.4  (summer 1994): 484‐95.      96. reidelbach, marie a., dorothy b. willis, joan latta konecky, ruth j. rasmussen, and joan  evidence based library and information practice 2006, 1:3  34 stark. ʺan introduction to independent learning skills for incoming medical students.ʺ  bulletin of the medical library association 76.2 (1988): 159‐63.     97. ren, wen hua. ʺlibrary instruction and college student self‐efficacy in electronic  information searching.ʺ journal of academic librarianship 26.5 (2000): 323‐8.     98. riddle, john s., and karen a hartman. ʺbut are they learning anything?ʺ designing an  assessment of first year library instruction.ʺ college and undergraduate libraries 7.2 (2000):  59‐69.  99. ridgeway, trish. library orientation methods, mental maps, and public services planning.  1983. eric document reproduction service ed247942.     100.robinson, andrew m., and karen schlegl. ʺstudent bibliographies improve when  professors  provide enforceable guidelines for citations.ʺ portal: libraries and the academy    4.2 (2004):  275‐90.   101.salisbury, fiona, and jenny ellis. “online and face‐to‐face: evaluating methods for teaching  information literacy skills to undergraduate arts students.” library review 52.5 (2003):  209‐17.  102.schilling, katherine lynne. “information‐literacy skills development in undergraduate  medical education: a comparison study of the impact of training methodologies on  learning outcomes.” diss. boston university, 2002.       103.schipper, rachel ann. ʺcomputer‐assisted instruction, learning style, field orientation,  time measurement, and citizen status: bibliographic instruction and college freshmen.ʺ   diss. florida institute of technology, 2000.   104.selegean, john cornell, martha lou thomas, and mary louise richman. “long‐range  effectiveness of library‐use instruction.” college and research libraries 44 (nov. 1983): 477‐ 80.     105.steele, godfrey a., and ernesta greenidge. “integrating medical communication skills with  library skills curricula among first year medical students at the university of the west  indies, st. augustine.” health information and libraries journal 19.4 (dec. 2002): 206‐13.     106.stevens, robert. aimlo: auto‐instructional media for library orientation. final report. ft.  collins: colorado state univ., 1974. eric document reproduction service ed105882.  107.stewart, linda, and jan olsen. “compact disk databases: are they good for users?” online  12 (may 1988): 48‐52.  108.sugranes, maria, and james a. neal. “evaluation of a self‐paced bibliographic instruction  course.” college & research libraries 44 (nov. 1983): 444‐57.    109.suprenant, thomas t. ʺlearning, lecture, and programmed instruction text: an experiment  in bibliographic instruction.ʺ  college and research libraries 43.1 (jan. 1982): 31‐7.      evidence based library and information practice 2006, 1:3  35 110.swanson, troy a.  “a radical step: implementing a critical information literacy model.”   portal: libraries and the academy 4.2 (apr. 2004): 259‐73.  111.tabur, susanne, rea devakos, sylvia newman, jennifer bayne, joan bartlett, elizabeth a.  reid, linda devore, m. lynn russell, and elizabeth uleryk. “peer informatics tutoring for  medical students: the toronto experience.” bibliotheca medica canadiana 22.3 (spring 2001):  121‐7.       112. thaxton, lyn, mary beth faccioli, and anne page mosby. “leveraging collaboration for  information literacy in psychology.” reference services review 32.2 (2004): 185‐9.      113.tiefel, virginia may. ʺevaluating a library user education program: a decade of  experience.ʺ college and research libraries 50 (1989): 249‐59.      114.tierno, mark j., and joann h. lee. “developing and evaluating library research skills in  education: a model for course‐integrated bibliographic instruction.” rq 22 (1983): 284‐91.  115.toifel, ronald c., and godfrey franklin. “using technology to teach preservice students  about locating information in the academic library.” journal of educational technology  systems 27.2 (1998‐99): 133‐45.       116.vander meer, patricia f., and galen e. rike.  “multimedia: meeting the demand for user  education with a self instructional tutorial.”  research strategies 14 (1996): 145‐58.      117.vertrees, robert l., and marjorie e. murfin. “teaching the legislative process: an evaluation  of classroom and library instruction and a legislative history exercise.” government  publications review 7a.6 (1980): 505‐15.     118.wallace, margaret c., allison storten, and patrick a. crookes. ʺteaching information  literacy skills: an evaluation.ʺ nurse education today 20.6 (2000): 485‐9.    119.ware, susan a., and deena j. morganti, ʺa competency‐based approach to assessing  workbook effectiveness.ʺ research strategies  4 (winter 1986): 4‐10.     120.webster, janet, and loretta riely.  “a library instruction case study: measuring success  from multiple perspectives.”  research strategies 19.1 (2003): 16‐32.    121.wendt, paul.  a study to determine the extent to which instruction to university freshmen  in the use of the university library can be turned over to teaching machines. 1963. eric  document reproduction service ed003559.     122.wiggins, marvin e. hands‐on instruction in an electronic classroom. a final report to the  u.s. dept. of education of a research and development grant awarded to establish a fully‐ equipped electronic training room and test the effectiveness of hands‐on instruction in  learning the notis opac and silver‐platter eric. provo, utah: brigham young univ., 1994.   eric document reproduction service ed369391.    123.williams, mitsuko, and elisabeth b. davis. “evaluation of plato library instructional  lessons.” journal of academic librarianship 5.1 (mar. 1979): 14‐19.     evidence based library and information practice 2006, 1:3  36 124.wood, richard j. ʺthe impact of a library research course on students at slippery rock  university.ʺ the journal of academic librarianship 10.5  (1984): 278‐84.    125.zahner, jane elizabeth. ʺa cognitive strategies framework for domain‐integrated process‐ oriented library instruction: the effects on research process orientation, library anxiety,  attitudes, and research products of college students.ʺ  diss. florida state university, 1992.           evidence based library and information practice 2006, 1:3  37 appendix c: data extraction elements  author – names of authors on paper    date – date paper was published    country and institution – country and institution where research took place    study objective – the stated objective of the study, as relayed by the authors    subject area – at the departmental level (i.e., sociology, medicine, biology)    participants – number of participants in study, other factors that are known about the  participants (university level of students, average age, prior educational level)    evaluation period – duration of the research project (i.e., 4 years, 6 weeks)    learning objectives – aims of teaching related to information literacy    faculty collaboration – was the instruction based on a partnership between librarians and  faculty (collaboration/partnership/consultation), or was it solely librarian initiated. (yes/no)  did  the librarian and faculty member work together?    integrated or independent – was the instruction integrated as part of an existing subject‐based  course, or was it an independent course or session? (integrated/independent)    tie to student need – was the instruction tied directly to student need, wherein the instruction  had a direct tie to a student assignment; or was the instruction indirectly tied to student need,  wherein the instruction gave examples and assignments that were similar to what they would be  expecting in their course; or was the instruction not tied to student need, using examples and  assignments which were of supposed interest to the student for future need. (direct/indirect/no  tie)    contact time ‐‐  number of hours of instruction included within the study    teaching method used – way of presenting instructional materials or conducting instructional  activities     mode of delivery – specific elements regarding how the instruction was carried out (i.e., lecture,  small‐group, study guide, computer lab exercises, face‐to‐face, webct module, online tutorial)    instructional topic – specific focus of the instruction (i.e., library orientation, using bibliographic  databases, doing research for a term paper)    research methods used – type of study design.  also note details of the research methodology.    learning outcomes measured – information literacy learning outcomes the researchers were  testing (i.e., improved ability to search online database, increased knowledge of appropriate  evidence based library and information practice 2006, 1:3  38 information resources, and improved understanding of where to search for information).  list the  outcomes stated and their categories, based on bloom’s revised taxonomy (anderson and  krathwohl) of educational objectives for the cognitive domain (‘remember’, ‘understand’,  ‘apply’, ‘analyze’, ‘evaluate’, ‘create’).    acrl information literacy standard / performance indicator – cite the standard and  performance indicator numbers as given in the ala document: information literacy  competency standards for higher education  <http://www.ala.org/ala/acrl/acrlstandards/standards.pdf>.    point at which learning outcomes were measured – when the test of learning outcomes took  place (i.e., immediately following instruction, 1 week after instruction, 2 years after instruction)    results – results of research (include all numerical data)      reported outcome – outcome of research (positive, mixed, neutral, or negative), as reported by  authors (i.e., students using x method of instruction had higher test scores).      implications of research – as suggested by the authors (i.e., x method of teaching should be used;  more research needs to be done on x).    critical appraisal elements – using the glasgow checklist (morrison et al. 891).  http://www.ala.org/ala/acrl/acrlstandards/standards.pdf evidence based library and information practice 2006, 1:3  39 appendix d: characteristics of studies meeting quality filter     quality filter: 1) comparative studies, 2) compared 2 different teaching methods, 3) outcomes  based on data assessed for statistical significance.     **shaded rows are studies included in meta‐analysis    abbreviations    al = active learning  sdil = self‐directed independent learning  cai = computer‐assisted instruction  ti = traditional learning    study  teaching  method  used  subject area  research methods  used  outcome  measure  study  outcome  sample  size  active learning  bren 1998   al vs. ti  english  literature  post‐intervention,  single time point, with  control group.   score on  post‐test  positive  86  cudiner 2001   al vs. ti  with an al  component  first year  experience  controlled before‐ and‐after.    score on  post‐test  negative  13  frasca 1992   al with a  ti  component  vs. no  instruction  medicine  post‐intervention,  single time point, with  control group.   score on  post‐test  positive  92  koufogiannakis  2005   al vs. no  instruction  medicine and  dentistry  randomized  controlled trial  (before‐and‐after).   score on  post‐test;  score on final  exam  mixed  164  prorak 1994   al vs. ti  english  composition  controlled before‐ and‐after.   score on  post‐test  neutral  246  computer assisted learning  alexander 2000  & 2001  cai vs. ti  library science  post‐intervention,  single time point, with  control group.    final exam  grade   neutral  88  axeen 1967   cai vs. ti  library science  controlled before‐ and‐after.   score on  post‐test  neutral  66  cherry 1991   cai vs. ti  vs. no  instruction  english  controlled before‐ and‐after.   score on  post‐test  search tasks  positive  53  cherry 1994   cai vs. no  instruction.   n/a  randomized  controlled trial  (before‐and‐after).    score on  post‐test  search tasks  neutral  30  churkovich  2002   cai  vs.  cai  with  mediation  vs. ti with  an al  component  sociology  randomized  controlled trial  (before‐and‐after).    score on  post‐test  negative  174  germain 2000   cai  vs. ti   with an al  component  first year  experience  controlled before‐ and‐after.   score on  post‐test  neutral  303  evidence based library and information practice 2006, 1:3  40 gutierrez 2001   cai   with  a ti  component  vs. sdil   with a ti  component.  not stated   randomized  controlled trial  (before‐and‐after).   score on  post‐test  neutral  134  holman 2000   cai  vs. ti  vs. no  instruction  english  composition   controlled before‐ and‐after.   score on  post‐test  neutral  125  hooks 1986   cai  vs. ti  library science   post‐intervention,  single time point, with  control group.    score on  post‐test  positive  19  johnson 1980   cai  vs. ti   (2 different  modes) vs.  no  instruction  english  post‐intervention,  single time point, with  control group.   score on  post‐test  neutral  68  kaplowitz 1998   cai  with a  sdil  component  vs. ti with  a sdil  component  biology    controlled before‐ and‐after with follow‐ up.   score on the  post‐test  neutral  423  lawson 1989   cai  vs. ti  english  controlled before‐ and‐after.   score on  post‐test  positive  172  madland 1988   cai  vs. ti  vs. no  instruction  remedial  english  controlled before‐ and‐after.   score on  post‐test  negative  not stated  nichols 2003   cai  vs. ti  english  composition  controlled before‐ and‐after.   score on  post‐test  neutral  64  orme 2004   cai  vs. ti  vs. cai  and ti vs.  no  instruction  business  post‐intervention,  single time point, with  control group.   scores on  post‐test;  searching  task; and  transcription  task  neutral  128  schilling 2002   cai vs. ti  medicine  randomized  controlled trial  (before‐and‐after).   score on  post‐test  neutral  128  vander meer  1996   cai  vs.  sdil  with  a ti  component  university 101  randomized  controlled trial  (before‐and‐after).   score on  post‐test  neutral  186  learner‐centred instruction  donegan 1989   learner‐ centred  vs. ti  vs. no  instruction  management   post‐intervention,  single time point,  with control  group.   score on  post‐test  neutral  156  tabur 2001  learner‐ centred vs. no  instruction  medicine  before‐and‐after  study.   score on  post‐test  neutral  137  self‐directed, independent learning  broadway 1985   sdil vs. no  instruction  library science  controlled  before‐and‐after  with follow‐up.   score on  post‐test  positive  48  evidence based library and information practice 2006, 1:3  41 ellsbury 1981   sdil  vs. no  instruction  agricultural and  biological  engineering,  biological  sciences,  entomology  controlled  before‐and‐after.   score on  post‐test  positive  155  holt 1984   sdil vs. ti vs.  no instruction  english  composition  controlled  before‐and‐after.  score on  post‐test  neutral  approximately  112  kirk 1971   sdil vs. ti  biology  post‐intervention,  multiple time  points, with  control group.   score on  bibliography;  essay grades  neutral  approximately  190   marcus 2003   sdil vs. ti  introduction to  college life  randomized  controlled trial  (after‐only).    score on  post‐test  neutral  203  meehan‐black  1981   sdil vs. ti vs.  no instruction  educational  psychology and  human  development  controlled  before‐and‐after.   scores on  two post‐ tests  positive  50  phillips 1979   sdil  vs. ti   communications  randomized  controlled trial  (after‐only).   score on  post‐test   neutral  161  phipps 1979   sdil vs. no  instruction  english  composition  controlled  before‐and‐after.   score on  post‐test  positive  487  ridgeway 1983  sdil vs. ti  vs.  no instruction  writing  randomized  controlled trial  (after‐only).  score on  post‐test  neutral  149  stevens 1974   sdil,  comparing two  different  modes of  delivery vs. no  instruction  psychology  post‐intervention,  single time point,  with control  group.   score on  post‐test  positive  502  suprenant 1982   sdil vs. ti vs.  no instruction  english  randomized  controlled trial  (before‐and‐ after).   score on  post‐test  positive  1,234  wendt 1963   sdil vs. ti vs.  no instruction  english  randomized  controlled trial  (before‐and‐ after).   score on  post‐test  neutral  193  wood 1984   sdil with a ti  component vs.  no instruction  library science  controlled  before‐and‐after.   score on  post‐test  positive  86  traditional instruction   bolt 1986  ti  with an al  component vs.  no instruction  library science   longitudinal  retrospective  cohort with  control group.   grade point  average  neutral  572  bostian 1990   ti, comparing  3 different  modes vs. no  instruction  introduction to  the academic  community;   business   post‐intervention,  single time point,  with control  group.   assessment  of search  strategies   positive  56  breivik 1977   ti with an al  component vs.  ti vs. no  instruction  english  controlled  before‐and‐after.   score on  final term  paper  positive  130  evidence based library and information practice 2006, 1:3  42 buchanan 1992   ti  vs. no  instruction  english  composition  post‐intervention,  single time point,  with control  group.   score on  searching  exercises  positive  137  cooper moore  2001   ti with an al  component vs.  no instruction  english courses  accounted for  75% of the  sample.   controlled  before‐and‐after.   score on  post‐test and  bibliography  assessment  positive  820  currie 1982   ti  (instruction  plus  compulsory  assignment vs.  instruction  only) vs. no  instruction  biology;  sociology  post‐intervention,  single time point,  with control  group.   score on  post‐test  positive  406  davis 1993   ti (compares 3  different  modes of  delivery), vs.  cai  freshman  orientation;  english  composition;  education   randomized  controlled trial  (after‐only).    score on  post‐test  mixed  220  dodgen 2003   ti with an al  component vs.  no instruction  sociology  controlled  before‐and‐after.   score on  post‐test  mixed  294  eyman 1977   ti  vs. no  instruction  library science  controlled  before‐and‐after.   score on  post‐test  neutral  242  fox 1996   ti with an al  component vs.  no instruction  nursing  multidimensional  program  evaluation.    scores on  post‐tests;  graded  assignments  positive  a) not stated;  b) 276;   c) 112  hardesty 1979   ti with a sdil   component vs.  no instruction  english  composition  controlled  before‐and‐after.    score on  post‐test  positive  162  hardesty 1982   ti with a sdil  component vs.  no instruction  english  composition  longitudinal  cohort with  control group.   scores on  skills tests  positive  403  lechner 1989   ti with a sdil  component  vs.  no instruction  education   randomized  controlled trial.   score on  post‐test;  score on  performance  test  neutral  199  nielsen 1987   ti with a sdil  component,  comparing the  order in which  the modes of  delivery were  given, vs.  placebo  n/a  randomized  controlled trial  (before‐and‐ after).    score on  post‐test  mixed  90  robinson 2004   ti  comparing  different  motivations  for an  assignment vs.  no instruction  political science  post‐intervention,  single time point,  with control  group.   analysis of  bibliography  citations  positive  84  selegean 1983   ti  with an al  component vs.  no instruction  library science   longitudinal  cohort study with  control group.   grade point  average;  student  persistence;  and  positive  468  evidence based library and information practice 2006, 1:3  43 graduation  rates  toifel 1999   ti and al vs.  ti  with an al  component vs.  no instruction  education  controlled  before‐and‐after.   score on  post‐test  positive  145  wallace 2000   ti with an al  component vs.  no instruction  nursing  controlled  before‐and‐after.   score on  post‐test  positive  127                announcing and advocating: the missing step in the eblip model commentary   announcing and advocating: the missing step in the eblip model   clare thorpe director, library services southern cross university gold coast, queensland, australia email: clare.thorpe@scu.edu.au   received: 15 sept. 2021                                                            accepted: 29 sept. 2021      2021 thorpe. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30044     introduction   the evidence based library and information practice (eblip) model “has been described as a structured approach to decision making” (hallam, 2018, p. 456) and a method for problem solving (howard & davis, 2011). it consists of five sequential stages that step a library and information science (lis) professional or team through the eblip process. the five stages are articulate, assemble, assess, agree and adapt, colloquially known as “the 5as” (koufogiannakis, 2013). the model has iteratively evolved over the past 17 years. yet it fails to include one of the most important characteristics of evidence based practice. this article argues that the model needs to evolve again to explicitly highlight the importance and relevance of communicating eblip outcomes and process to the local community and the professional evidence base. a sixth “a” of announcing or advocating is proposed.   evolution of the eblip model   the first version of the model by booth (2004) proposed five steps and established the foundational principles of eblip. it emphasized a reliance on research literature as the only source of evidence and focused on an individual practitioner’s approach to a research task. the steps included:   defining the problem (ask) finding the best evidence in the research literature (acquire) appraising the evidence (appraise) applying the evidence to practice (apply) evaluating the change, performance, or impact (assess)   booth (2009) subsequently reflected on the five stages and proposed an amended version whereby an evidence based practitioner would:   articulate the problem assemble the evidence base assess the evidence agree the actions adapt the implementation   in the evolved model booth suggested that a feedback loop existed between the agree-adapt steps and identified that decisions in libraries are often made by teams, rather than individual practitioners. the revised model began to acknowledge research literature and locally collected data as equally valid sources of evidence.   koufogiannakis (2013) validated booth’s model in her doctoral thesis, in which she argued for a broader definition of evidence that included professional knowledge and local evidence alongside published research. the final iteration of the five-step model was published in koufogiannakis and brettle’s 2016 book, being evidence based in library and information practice, in which they stated that the five steps were cyclical in nature and could be applied to both individual and group decisions (p. 14). figure 1 the eblip model (koufogiannakis & brettle, 2016, p. 14).   this version of the model drew on a range of different evidence sources and was described as a holistic and realistic depiction of the eblip process. the model has been widely adopted and applied by individuals and teams, with hallam (2018) noting that the koufogiannakis and brettle version allows practitioners to take ownership of the process, and fosters critical reflective practice among lis professionals.   alternative frameworks   alongside the evolution of the eblip model, a small number of related frameworks were proposed and documented in the literature. howard and davis (2011) melded design thinking with booth’s original 2004 version of the model. their approach combined the philosophies of the two frameworks to produce a hybrid model of evidence based practice (ebp) and design thinking. the model proposed six stages:   define the problem undertake and appraise research prototype and test implement the solution evaluate the outcomes engage in storytelling   howard and davis’s (2011) hybrid ebp and design thinking model was the first to include a step that explicitly identified the role of communication as a characteristic of eblip. the sixth step—engage in storytelling—is described as “a process to close the loop and contribute to the evidence base” (p. 19). howard and davis argued that when solutions to complex workplace problems have been implemented and evaluated, it is important to tell the story through informal and formal channels. they suggested there are benefits to the individual, organization, and the broader profession in documenting the process, the inputs (or evidence), and the learnings of the eblip process in order to add to the evidence base that can be drawn on by other lis practitioners in the future.   howlett (2018) proposed a four-phase framework to describe how eblip may be undertaken by academic libraries as a strategic engagement activity. howlett challenged the unidirectional nature of the eblip model, arguing that various stages of the model are multi-directional, iterative in nature, and interconnected in practice when applied to complex organizations. the proposed “lens” reduced the steps or phases of eblip to four:   interpret the organizational context and strategic priorities apply the library’s strategy measure the outcomes communicate the impact of the library’s strategic contribution   this model emphasized the application of evidence based practice through which academic libraries “[tell] the story of how the library contributes to student and institutional success” (howlett, 2018, p. 76). howlett argued that the communication step empowers library leaders to generate influence and advocate for what the library is and what it achieves within their university.   in thorpe and howlett’s (2020) evidence based library and information practice capability maturity model, the way in which a library reported or communicated evidence was identified as an indicator of maturity. more mature organizations focused on communicating for influence and making evidence easily understood by the target audience (p. 97). interview respondents demonstrated varying degrees of appreciating and applying the power of communication to demonstrate value and impact to local stakeholders. staff from libraries that showed a high level of eblip maturity could also articulate the benefits of contributing to the lis evidence base.   the alternative frameworks view evidence based practice from different perspectives. however, all explicitly feature a stage in which lis practitioners communicate their findings, processes, and outcomes. communication is emphasized as a key step that informs future research, documents methodologies and processes, articulates the role of the library and its staff, demonstrates value and impact, and builds the profession’s evidence base.   implicit or explicit   neither booth’s original models nor koufogiannakis and brettle’s widely adopted version explicitly identified a step in which the lis practitioner communicates their evidence based practice to their stakeholders, clients, or peers. while koufogiannakis and brettle did not include mention of communicating (or advocating or announcing) as a step in their model, they have written about the importance of communication within eblip. as early as 2004, crumley and koufogiannakis (2004) argued that:   dissemination of research results is vital to the progress of the profession as well as helping to improve practice. it involves not only making your research available, but also ensuring that it is accessible to others and presented in a manner that is easy to understand. (p. 127)   they promoted communication within the library, to its parent organization, and externally to the profession via informal and formal methods of dissemination, such as conference presentations, journal clubs, scholarly publication, reports to management, and personal networking (crumley & koufogiannakis, 2004).   koufogiannakis and brettle (2016, pp. 165–166) recommended that lis professionals engaging in eblip should:   share their “learn[ing] with others in order to improve the knowledge of the profession.” “use … new knowledge or evidence to convince or influence others of the best way forward or to prove the value of their services.”   they suggested that the importance of communication was implied throughout the contributed chapters of their book and acknowledged that it was an aspect of eblip which, at the time of publication, had not been well considered in the literature (koufogiannakis & brettle, 2016, p. 166).   one way to consider how to explicitly embed communication as a stage in the eblip model is to consider the relationship between eblip and research processes. hallam (2018) drew parallels between the eblip model and research processes, stating that one of the goals of evidence based practice is to inspire librarians to conduct research. nguyen and hider (2018) also linked eblip with the benefits of undertaking research as a librarian. they surmised that research is a key tool for eblip, particularly in the academic library sector, where practice-oriented research can be “harnessed by [library] management to implement improvements and innovation” (p. 16). writing, publishing, disseminating and sharing the completed work is the final step in the research process (hallam, 2018, p. 457). communicating research findings is a critical and often required stage of the research process, particularly when publishing research findings is mandated by funding bodies. if eblip is accepted as a form of practitioner research, then it follows that communicating findings and results should be a logical and explicit requirement of being evidence based lis practitioners.   the omission of a communication step as an endorsed and prioritized part of eblip could be why scholarship and practitioner research are not widely accepted as a part of lis professionals’ work. lamond and fields’s (2020) review of 20 years of eblip in new zealand reported that it was difficult to find examples of eblip application and development in the literature. lamond and fields (2020) assumed that the published outputs were not representative of the eblip work undertaken across the country. they purported that eblip in new zealand was primarily undertaken as an information gathering activity to solve workplace problems, with little or no consultation of published literature or theory, and subsequently not reported in the published literature as research outcomes (p. 31). less formal examples were found in presentations, blog posts, product reviews for vendors and were observed anecdotally at meetings and in conversations. the failure of lis practitioners to announce, report, and publish their work makes it challenging to determine how widespread eblip adoption is by individuals, teams, and organizations. todd (2015) highlighted the perceived invisibility of school librarians’ impact on student learning due to a lack of research and an evidence base to support advocacy efforts in australia. for lis practitioners committed to being evidence based, it should be concerning that the impact of libraries engaging in eblip continues quietly and remains mostly invisible to libraries’ funding organizations, clients, and the profession. figure 2 shows how the sixth step could be added to the eblip model.   announce and advocate—the missing as   why should announcing, advocating, and communicating be made an explicit part in the eblip model? i propose four benefits that may apply to individuals, libraries, and the profession:   to advocate and influence to contribute to the profession’s evidence base to demonstrate professional expertise to build organizational capacity and maturity figure 2 the proposed evolution of the eblip model.   to advocate and influence   libraries are commonly reliant on funding from the organization they serve, be it a university, government, or for purpose or for profit corporations. using evidence to influence decisions and decision makers is a key reason that librarians adopt evidence based approaches in their work (partridge et al., 2010, p. 285). howlett’s (2018) organizational lens model argued that one purpose of eblip is to effectively communicate the library’s contribution and value to its parent organization or funding body. lamond and fields (2020) stated that evidence based reports have an increased chance of getting funding for projects. being able to articulate clearly the evidence supporting a project, initiative, or business case is more likely to influence stakeholders. in reporting evidence for advocacy and influence, it pays to strategically consider the target audience. crumley and koufogiannakis (2004) argued that eblip needs to be user-friendly and understandable by those to whom library staff report, as well as to colleagues. while the message and the method of communication should have a clear purpose and be easy to follow, evidence should be communicated in ways that might influence the decision made by those in power. the eblip model can be strengthened by emphasizing this activity in order to empower lis practitioners using evidence based practices to achieve success.   to contribute to the evidence base   issues with the quality and quantity of the lis evidence base have acted as a barrier to adopting and implementing eblip from the beginning of the movement (haddow, 1997; koufogiannakis & crumley, 2006). this reason alone should be enough to explicitly add a communication focused step to the eblip model. howard and davis (2011) included storytelling in their model, stating that sharing what has been learned adds to the evidence base locally within a library, at its parent institution, and in the broader lis profession. koufogiannakis and crumley (2006) argued that   every librarian has a part to play in building up an evidence base that is directly relevant to our decision-making needs. … librarians need to start filling the gaps and mending the seams of our professional body of knowledge in order for our profession to advance. (p. 338)   increasing the quality, quantity, and diversity of work contributed to the evidence base should also foster inclusion and diversity of opinion, inviting more voices and alternative perspectives into the profession. like the critical librarianship movement, eblip is contextualized to local, social, political, and economic environments (drabinski, 2019). a model that endorses and promotes the communication of eblip empowers the development of critical librarianship in which evidence can challenge and be challenged. when librarians use evidence to advocate, they bring an awareness to organisational behaviour that can be named and professionally discussed in order to expose bias in decision making (koufogiannakis, 2013, p. 197). the critical nature of questioning that starts with the articulate stage should reach a logical conclusion with advocacy. in doing so, evidence based practice is well aligned with the critical librarianship movement to document, uncover, and challenge assumptions in library structures, systems, and services. for eblip to fully support the development of a community of practice that “changes the profession for the better” (koufogiannakis & brettle, 2016, p. 166), the model must promote the importance of contributing to the profession’s evidence base.   to demonstrate professional expertise   at its heart, eblip promotes and develops “the mind-set of a critically reflective practitioner” (hallam, 2018, p. 457). in order for eblip to be an embedded and valued part of everyday professional practice, it must be visible. communicating research findings promotes the benefits of being evidence based. it encourages and supports practitioners who wish to develop their skills and expertise in this space (hallam, 2018). appleton (2021) argued that the lis professionals should exhibit pride in their work, and should actively and deliberately promote their research based achievements. one strategy suggested by appleton (2021) is to engage in scholarly writing and presenting as a way to build the reputation of both individual contributors and the library service. crumley and koufogiannakis (2004) stated that disseminating evidence based practice contributes to how librarians understand and define their role. by announcing outcomes and achievements to the community, evidence based practitioners can document their expertise in reaching milestones and developing innovations, time-stamping projects for future reference. if lis professionals want to be evidence based, then the communication and sharing of their achievements and enthusiasm should be a defining feature of their professional expertise and identity.   to build organizational capacity and maturity   library services are human centred and human mediated. a culture of evidence based practice within an organization requires a shared approach and participation from all staff. booth’s revision of the original model was partly influenced by his observations of eblip applied within teams. booth (2009) noted that “a significant contributor to the success of any service change is the motivation, involvement and commitment of the team” (p. 343). lamond and fields (2020) viewed eblip as a way of developing staff. they described how eblip benefits the library producing evidence based outcomes and also develops the potential and performance of staff through the process. the way in which evidence was communicated to influence organizational decision making and to demonstrate value and impact was a key indicator of eblip maturity in thorpe and howlett’s (2020) model. staff who communicated eblip within their libraries and to external audiences contributed to growing the maturity of the library as an evidence based organization. the ability to effectively communicate to different audiences via different channels is a core professional skill for all lis workers. nguyen and hider (2018) identified many benefits for libraries in fostering a culture of research communication. the benefits included “more efficient ways of working, better informed staff, the production of evidence that can be used for advocacy, and professional kudos for the library and individuals” (nguyen & hider, 2018, p. 16). hallam (2018) argued that employers should “provide opportunities and resources for their staff to engage in ebp, including the dissemination of research findings to the wider profession” (p. 460). in the covid pandemic environment where libraries have benefited from sharing knowledge, evidence, and experiences with each other, it makes sense for the eblip model to demonstrate a commitment to communication in order to build organizational capacity, resilience, and maturity.   conclusion   koufogiannakis and brettle (2016) stated that their eblip model was “more about approaching practice with a particular mindset, rather than about checking off steps in a process” (p. 165). regardless of the authors’ intent, it is easy to default to using the model as a step-by-step guide, especially for professionals beginning to engage with eblip as a way of working and being. this makes the absence of a step that promotes the communication of eblip activities a challenge for the future of the profession. if a generation of lis professionals learn to engage in eblip without announcing, advocating, and communicating their work, then criticisms of the validity of the profession’s evidence base will endure. communicating in an evidence based way should be an explicit part of the eblip professional identity. by adding advocate and announce to the model as the “6th a,” lis professionals who are doing and being evidence based in their practice will be well placed and valued for their expertise. they will be well equipped to influence decision makers, grow in maturity, and contribute to the evidence base of the profession. the eblip model must be strengthened with an explicit step that promotes actively contributing to the evidence base for the betterment of libraries and the profession.   acknowledgements   the author acknowledges and pays respect to the people of the yugambeh nation on whose land this work was created.   references   appleton, l. (2021). editorial – academic librarians and engaging with scholarship. new review of academic librarianship, 27(2), 145–148. https://doi.org/10.1080/13614533.2021.1944747   booth, a. (2004). formulating answerable questions. in a. booth & a. brice (eds.), evidence-based practice for information professionals: a handbook (pp. 61–70). facet publishing. booth, a. (2009). eblip five-point-zero: towards a collaborative model of evidence-based practice. health information & libraries journal, 26(4), 341–344. https://doi.org/10.1111/j.1471-1842.2009.00867.x   crumley, e., & koufogiannakis, d. (2004). disseminating the lessons of evidence based practice. in a. booth & a. brice (eds.), evidence-based practice for information professionals: a handbook (pp. 138–143). facet publishing.   drabinski, e. (2019). what is critical about critical librarianship? art libraries journal, 44(2), 49–57. https://doi.org/10.1017/alj.2019.3   haddow, g. (1997). the nature of journals of librarianship: a review. libres: library and information science research, 7(1). https://doi.org/10.32655/libres.1997.1.4   hallam, g. (2018). being evidence based makes sense! an introduction to evidence based library and information practice (eblip). bibliothek forschung und praxis, 42(3), 453–462. https://doi.org/10.1515/bfp-2018-0067   howard, z., & davis, k. (2011). from solving puzzles to designing solutions: integrating design thinking into evidence based practice. evidence based library and information practice, 6(4), 15–21. https://doi.org/10.18438/b8tc81   howlett, a. (2018). time to move eblip forward with an organizational lens. evidence based library and information practice, 13(3), 74–80. https://doi.org/10.18438/eblip29491   koufogiannakis, d. (2013). how academic librarians use evidence in their decision making: reconsidering the evidence based practice model [doctoral dissertation, aberystwyth university]. aberystwyth research portal. http://hdl.handle.net/2160/12963   koufogiannakis, d., & brettle, a. (eds.). (2016). being evidence based in library and information practice. facet publishing.   koufogiannakis, d., & crumley, e. (2006). research in librarianship: issues to consider. library hi tech, 24(3), 324–340. https://doi.org/10.1108/07378830610692109   lamond, h., & fields, a. (2020). evidence based library and information practice: a new zealand perspective. new zealand library and information management journal, 57(2), 29–36. https://doi.org/10.6084/m9.figshare.12725387   nguyen, l. c., & hider, p. (2018). narrowing the gap between lis research and practice in australia. journal of the australian library and information association, 67(1), 3–19. https://doi.org/10.1080/24750158.2018.1430412   partridge, h., edwards, s. l., & thorpe, c. (2010). evidence-based practice: information professionals’ experience of information literacy in the workplace. in a. lloyd & s. talja (eds.), practising information literacy: bringing theories of learning, practice and information literacy together (pp. 273–297). centre for information studies. https://doi.org/10.1016/b978-1-876938-79-6.50013-3   thorpe, c., & howlett, a. (2020). understanding eblip at an organizational level: an initial maturity model. evidence based library and information practice, 15(1), 90–105. https://doi.org/10.18438/eblip29639  todd, r. j. (2015). evidence-based practice and school libraries: interconnections of evidence, advocacy, and actions. knowledge quest, 43(3), 8–15. https://files.eric.ed.gov/fulltext/ej1048950.pdf   classics   mcclure and samuels’ study on information sources used for decision making and the connection to organizational climate still resonates today   a review of: mcclure, c. r., & samuels, a. r. (1985). factors affecting the use of information for academic library decision making. college & research libraries, 46(6), 483-498.   reviewed by: denise koufogiannakis collections and acquisitions coordinator university of alberta libraries edmonton, alberta, canada email: denise.koufogiannakis@ualberta.ca   received: 1 aug. 2014    accepted: 28 nov. 2014      2014 koufogiannakis. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective to investigate the use of information sources for decision making within academic libraries; specifically looking at what sources of information are used, whether information use is related to organizational climate, and what organizational factors lead to optimal information use in decision making.   design cross-sectional survey on a random sample of libraries.   setting 18 medium to moderately large academic libraries from across the united states.   subjects 356 academic librarians holding a variety of positions and levels of responsibility within their organizations.   methods a questionnaire was mailed to participants in order to measure relationships between four main variables: information acquisition, information dissemination, information evaluation, and library climate. all instruments were validated and tested for reliability. participants were given 10 library decision situations to consider, together with a list of potential information sources to inform the decision, and then choose which information source they would use primarily in each situation. participants’ perception of their library climate was measured with five scales covering innovation, support, freedom, democratic governance, and esprit.   main results the study found that academic librarians prefer internal sources of information, such as interpersonal communication with library staff, and library committees, for making decisions. however, paraprofessional staff members were not seen as meaningful sources of information within this grouping. the participants rarely chose to consult external information sources, such as other professionals outside of the library, or library users. information sources such as conducting research, continuing education, past experience, or personal opinion were not found to be important to the participants’ decision making. written documents such as articles, books, and brochures were also seldom used. democratic governance was the organizational climate dimension found to be most closely linked to information dissemination.   conclusion the authors conclude that the study suggests that academic librarians are not using a full complement of information sources to assist with their decision making, and that the “information that is used tends to be ‘opinion-based’ rather than empirically based” (p. 495).  proximity of information plays a role, with information that is closer and easier to obtain being used more frequently. the authors strongly stress, with concern, that, “current academic library decision-making processes encourage ineffective activities since they preclude or limit clientele input, empirical research, and additional environmental input” (p. 495).   commentary   this study by mcclure and samuels, while 30 years old, examines some of the same issues that the academic library community is struggling with today in terms of evidence as part of decision making, and how organizational climate impacts librarians’ use of evidence in practice. this study deserves to be read and recognised as a “classic” by anyone who is interested in these issues within our profession, and to be recognised by the eblip community in particular, as its insights, based on a well-designed, large scale, and broadly based study that used validated tools, were precursors to the eblip movement that began 15 years later. unfortunately, the body of research related to decision making theory and organizational behaviour within libraries is still scant today.   the specific study summarized here is but one piece of a wider body of work by mcclure and samuels. mcclure led several studies that examined the use of information and data in decision making within libraries. he first reviewed the literature relating to management of organizational information and how information is used for decision making and concluded that when people who are information rich are included in decision making, the organization will be more productive in meeting its goals (mcclure, 1978). later, mcclure worked with samuels to study the utilization of information for decision making in both public (samuels & mcclure, 1983) and academic (mcclure & samuels, 1985) libraries, the academic portion of which is summarized above. both papers, stemming from the same large study, found that librarians preferred internal sources of information, such as personal communication and internal documents. patron involvement in decision making was practically non-existent, and very few decisions used information from empirical research.   samuels and mcclure also found that organizations where decision making is shared and the environment is more open with its communication facilitate greater use of information in decision making. mcclure (1986) went on to recommend professional and organizational strategies to increase the use of data in decision-making:   1.       review existing management styles and organizational climates within the academic library. 2.       increase the knowledge level of the importance and potential applications of cost and performance measurement data. 3.       develop administrative systems that support the identification, collection, organization, analysis, and reporting of cost and performance measure data. 4.       establish reward structures for librarians who use cost and performance measurement methodologies for library decision-making. (p. 332–333)   the factors raised by mcclure and samuels in the 1980s have come to light again in the 21st century, with evidence based practice and assessment have emerging as topics of interest within library and information studies. authors have written about the importance of research and data in decision making within libraries, and that such use requires an environment where openness, integrity, and trust are enabled (lakos & phipps, 2004; hiller, kyrillidou & self, 2008). involvement of staff in decision making and developing clear communication systems help to facilitate this type of organizational change where evidence can become part of the culture of the organization (davies, 2007). davies also stresses the importance that “evidence is used honestly and that data is acquired and presented in as transparent a fashion as possible” (2007, p.6). such transparency and honesty allow for staff to participate more fully and contributes to the integrity of the decision making process. hiller, kyrillidou and self (2008) determined that evidence alone is not enough to create a research culture where decisions are grounded in data. ultimately, organizational culture and leadership within the organization are crucial to the integration of evidence as a normal part of decision making within academic libraries. the issues raised by mcclure and samuels still resonate today and much can be learned by looking back to their impressive research.   references   davies, j. e. (2007, aug.). culture, capability and character in applying evidence to service enhancement and development: an exploration. world library and information congress: 73rd ifla general conference and council, durban, south africa. retrieved from http://archive.ifla.org/iv/ifla73/papers/154-davies-en.pdf   hiller, s., kyrillidou, m., & self, j. (2008). when the evidence is not enough: organizational factors that influence effective and successful library assessment. performance measurement and metrics, 9(3), 223-230.   lakos, a., & phipps, s. e. (2004). creating a culture of assessment: a catalyst for organizational change. portal: libraries and the academy, 4(3), 345-361.   mcclure, c. r. (1978). the information rich employee and information for decision making: review and comments. information processing & management, 14(6), 381-394.   mcclure, c. r. (1986). a view from the trenches: costing and performance measures for academic library public services. college & research libraries, 47(4), 323–336.   mcclure, c. r., & samuels, a. r. (1985). factors affecting the use of information for academic library decision making. college & research libraries, 46(6), 483-498.   samuels, a. r., & mcclure, c. r. (1983). utilization of information for decision making under varying organizational climate conditions in public libraries. journal of library administration, 4(3), 1-20. classics   setting the course: revisiting the dawn of nursing information literacy scholarship in the work of fox, richter and white   a review of: fox, l. m., richter, j. m., & white, n. (1989). pathways to information literacy. journal of nursing education, 28(9), 422-425.   reviewed by: bridget faricy-beredo medical librarian/instructor, college of medicine mulford library, university of toledo toledo, ohio, united states of america email: bridget.faricy@utoledo.edu   received: 18 april 2013  accepted: 01 dec. 2013      2013 faricy-beredo. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective to use a multi-dimensional approach to evaluate the effectiveness of a nursing information literacy program (pathways to information literacy) delivered to undergraduate nursing students. assessment sought to track progress in both affective and cognitive spheres.    design – this program evaluation focuses on the pathways to information literacy (pil) curriculum, which was delivered from 1988-1992. it consisted of 6 hours of librarian-delivered instruction, divided over 4 sessions. to evaluate the impact of this curriculum, the authors gathered five different data sets: informal feedback; the results of a survey measuring the affective domain of confidence; the results of a longitudinal cohort survey of graduates; and two different sets of data gathered from distinct samples but utilizing the same information literacy assessment tool. all five data sets served the greater purpose of assessing students’ mastery of information literacy.   setting an undergraduate bachelor of science in nursing (bsn) program within a state university, the university of northern colorado.   subjects in general, the subjects were different cohorts of nursing students between 1988 and 1992. class size hovered at just over 100. for most measures, response rate was high and the dropout rate was low. it is worth noting that one of the assessments was a longitudinal cohort survey of graduates. as the mobility of graduates often decreases the number of responses, the omission of the response rate for this measure is of concern.   methods the methods are one of the most striking aspects of this study.  the authors employed no less than five methods of assessment:     from 1988-1992, investigators gathered informal feedback from both students and faculty members about the written assignments of the pil program. the specific method for gathering feedback was not reported. from 1988-1990 pil students in their junior year took preand postpil program confidence surveys. the survey tool, which was included, contained 6 likert-like questions which assessed their affective domain of confidence as related to their ability to perform information literacy related tasks, such as using a bibliographic index. during the 1990/91 academic year the authors administered a general, university-wide information literacy assessment tool, which was included. it measured both self-perceived progress and objectively measured skill attainment in the cognitive domain. the survey tool lacked validation, but had been previously published in the library literature (greer, 1991). students from the pil program were extracted from the data for comparison to the general student population. during 1991/92 pil students completed the above mentioned information literacy assessment tool as a pre-and post test.   both pil (1990/91) and non-pil (1988/89) cohorts completed a longitudinal post-graduation survey which was included. comparison of the two cohorts aimed to examine the effect of the pil program on subsequent scholarly professional activities.   while the methodologies were all tied to the overall purpose of program evaluation, they were not tied to specific pedagogies or content units.    main results fox et al. generally utilized simple, descriptive statistical data. the data derived from the information literacy assessment tool was the exception, producing ordinal data which was analyzed using a chi-squared approach.   all outcomes supported the positive effect of the pil program. the soft technique of gathering informal feedback from students and faculty resulted in positive feedback. faculty reported that their students became independent in information-seeking and the quality of their papers increased. students also reported that the assignment and instruction gave them confidence and that the written assignments were a nice break from the traditional examinations. the affective confidence survey noted a substantial improvement: pre-program only 26% reported confidence when performing information-seeking strategies compared to 76% post-instruction. when the information literacy assessment tool was administered both to 68 pil participants and to 208 general students, the pil students both believed themselves to be more successful and demonstrated greater knowledge. 70% of pil students answered cd-rom index questions correctly, compared to 49% of general students. when the same tool was given to only pil students as a pre-and posttest, it showed statically significant increases in the use of the library and mastery of several specific search techniques: p<0.05 for 7 of 20 measures. the post-graduation survey showed that 45% of the pil students had engaged in some scholarly activity, as compared to 10% of non-pil graduates.    conclusion the authors concluded that the multidimensional assessment efforts delivered a comprehensive view of the effectiveness of the program, demonstrating student benefit in cognitive (knowledge attainment) and affective (confidence levels) domains as well as in subsequent professional behavior.   commentary   a multidimensional evaluation of a nursing information-literacy program (menilp) (fox, richter & white, 1996) contains a description of the pathways to information literacy (pil) program. this program consisted of four instruction sessions, which were integrated across two required nursing courses in the junior year of study. though the instructional pedagogies of the sessions were not detailed, the authors did provide the goals for the instruction and mentioned several assessment methods. the goals for the instruction included:   develop an understanding of library organization and services acquire skills in forming research questions and locating and evaluating accurate, relevant information for problem solving apply appropriate information-seeking strategies; and emulate the scholarly activities of professional nurses (fox et al., 1996, p.183)   the first information literacy instruction session targeted the skills of information location and synthesis. mastery of the course was measured by an exam and an assignment which required the students to synthesize general sources to create a topical summary of a condition and its treatment. session two introduced more advanced and nursing-specific research techniques such as locating and using specialized handbooks and citation indexes. librarian-led group discussion surrounding conflicting research findings was utilized as a synthesizing methodology. session three presented the process of topic selection and research question refinement. it offered the students in-class time to work on their topics using a topic selection grid tool. session four focused on computer search strategies, including cinahl on cd-rom, which was a relatively new addition to the library’s collection. students were taught effective search strategies for cinahl and asked to demonstrate their knowledge by submitting a print out of their strategy. a detailed description of the program’s rationale and creation can be found in an earlier work by the same authors (fox, richter, & white, 1989). though well done, that project description has not had as great an impact as menilp which focuses on program evaluation.   the program evaluation plan for the pil program pre-dated many of the current, widely-used frameworks for program evaluation, such as the framework for program evaluation in public health (centers for disease control and prevention, 1999). the authors did however utilize an appropriate evaluation framework, selecting staropoli and waltz’s model (1978) which is specific to health education programs.  the model takes a familiar five question format, asking first, who will be involved in the evaluation? the faculty and the librarian. what are the purposes in conducting an evaluation? to evaluate the effectiveness of the program and measure the degree of skill acquisition. what is to be evaluated: curriculum, objectives, faculty or students? students’ mastery of information literacy in the cognitive and affective domains. how is the evaluation to proceed?  the authors summarize the past practices and detail plans for on-going evaluation “every two-three years” (fox et al., 1996, p. 184). while this older model pre-dates the now commonly seen emphasis on evidence, the authors were ahead of their time in broadly defining their purpose as creating “[o]bjective procedures [which] should be used to facilitate the collection of dependable, unbiased data… to determine value” (fox et al., 1996, p. 184).    menilp significantly impacted subsequent scholarship.  a three-pronged approach using the cited reference search functions of isi web of knowledge, cinahl, and google scholar identified 39 subsequent citations spanning 17 years. although most of the publications were within the field of library and information science, approximately 25% were from nursing/allied health. google scholar, as expected, identified the greatest diversity of formats (books, thesis, articles and reports) as well as the most non-english language publications, six in total (kleibel & mayer, 2005; meneses placeres, 2008a, 2008b; qun, 2011; nodarse rodríguez, 2005; sundin, 2003).     figure 1 number of menilp citations by year   the five-prong evaluation strategy was one of the main drivers of the article’s popularity.  each of the evaluation methods designs can be appraised in terms of evidence quality. the informal feedback, while illuminating, did not have the power to influence practice. the other four methodologies involved either survey or tests; none of which utilized validated tools. the full text of the tools was included so the reader is able to make some judgments about the content and face validity of the measures. when the information literacy assessment tool was used as a preand post-test, the results were determined statistically to be significant, with reported p values. when it was used to compare pil students to the larger under-graduate population, there are concerns. the pil students differed significantly from the general student population. no controls were put in place for the nursing students’ higher average gpas, greater average age and their skewed gender, though the authors noted these differences. the confidence survey’s descriptive statistics were impressive but are not analyzed for statistical significance. the post-graduation survey suffers from the same limitations. it does however have the most interesting design of all the methodologies: a cohort, longitudinal survey. when considering an evidence base, this type of study design is considered strong and is especially well-suited to look at questions of etiology (mackibbon, wilczynski, & eady, 2009). in this design, the pil program was the exposure which was controlled for in two different cohorts of graduates.  the pil graduates, when compared to their non-pil peers using simple descriptive statistics, reported more professional reading and greater participation in scholarly activity. fox and colleagues do not discuss the internal/external validity, reliability or limitations of their methodologies at any point.   though the individual results dim under an analysis of their strength of evidence, they do synergistically work together to form a larger preponderance of evidence. when considered as a whole, the methodologies create a body of data that delivers its own internal triangulation of results and meet the authors’ objective of a collection of data which determines value. the thoroughness and ultimate success of this work as a program evaluation goes a long way in explaining why this article has had such an impact. it was of particular relevance to the subsequent researchers who were picking up on the growing evidence-based trend that was occurring both in program evaluation and in nursing practice.     these subsequent researchers have used and continue to use this work in divergent ways.  the multiple methodologies and conclusions are often considered in piecemeal fashion. this study provides support or evidence for:   the definition of information literacy (il) the positive effect of il instruction in general as well as within the affective and cognitive domains evaluation of il instruction (including post-graduation impact) nursing faculty/librarian collaborations evidence-based practice   each one of these areas will be considered separately.    definition of information literacy   because menilp was a vanguard of information literacy scholarship within nursing, its first section was aptly titled why information literacy? (fox et al., 1996). this section defined the term and justified the need for il, specifically within a nursing context.  this provided fertile ground for subsequent authors to mine the section when defining information literacy for their readers (cobus, 2008; frier, 2009; hopkins, callister, mandleco, lassetter, & astill, 2011; jacobs, rosenfeld, & haber, 2003; nayda & rankin, 2008).   positive effect of il instruction   menilp reported positive findings, supporting the effectiveness of information literacy instruction. these positive findings provided justification for scores of curriculum-integrated nursing il programs and worked their way into many subsequent studies (barnard, nash, & o'brien, 2005; brettle, 2003; brettle & raynor, 2013; carter-templeton, 2011; craig & corrall, 2007; eimas & barton, 2001; grant & brettle, 2006; jette, tribble, gagnon, & mathieu, 2010; meneses placeres, 2008a, 2008b; morgan, fogel, hicks, wright, & tyler, 2007; nodarse rodríguez, 2005; rush, 2008; saranto & hovenga, 2004; shorten, wallace & crookes, 2001; tronstad, phillips, garcia, & harlow, 2009; weinert & palmer, 2007; wright & mcgurk, 2000). authors occasionally described the work with words like “seminal” (craig & corrall, 2007).   evaluation of il instruction   menilp’s true focus was the evaluation of the pil program.  in this they were also vanguards; ahead of current library trends like measuring value and assessing outcomes. many subsequent studies have focused on the evaluation methods specifically (argüelles, 2012; baro & ebhomeya, 2013; carlock & anderson, 2007; carter-templeton, 2011; davies, urquhart, smith, & hepworth, 1997; martin, 2008; meldrum & tootell, 2004; powell & case-smith, 2003; saranto & hovenga, 2004; wallace, shorten, crookes, mcgurk, & brewer, 1999). once again, the variety of menilp assessment methods has spawned a diversity of applications. for some authors the multidimensional approach has been the feature of note (davies et al., 1997; wallace et al., 1999), for others it was the librarian/faculty team approach (martin, 2008). one of the most striking evaluation methods of menilp was the longitudinal, cohort survey of graduates. four subsequent studies focus on the selection of this study design (baro &  ebhomeya, 2013; brettle, 2003; eldredge, 2002; powell & case-smith, 2003).    nursing faculty/librarian collaborations   while not as note-worthy within the library literature, several works from the nursing perspective addressed the strength of the collaboration between librarians and nursing faculty described in menilp (carter-templeton, 2011; honey, 2007; schloman, 2001).  the library literature was much more likely to focus on the aspect of curriculum integration.   evidence-based practice   obviously the world has changed since 1996. of tremendous import to the practice of health science librarianship has been the advent of evidence-based practice (ebp). though ebp was not an explicit concern of fox et al., their work does eloquently speak of a nurse’s need to be able to acquire and appraise information, which are critical steps in the ebp process. by clearly addressing these concerns their study drew the interest of later ebp investigators (durando & oakley, 2005; jacobs et al., 2003; rush, 2008; shorten, wallace & crookes, 2001; urquhart, 1998).   a multidimensional evaluation of a nursing information-literacy program could have been sprawling mess of an article. it has the clear aim of a program evaluation but presents five different evaluation strategies, spanning a five year period. the evaluation strategies themselves vary from simple informal feedback to a complex cohort design. it tries to measure both affective and cognitive changes. with such a broad focus in terms of time, methodology, and approach it certainly had the potential to fail. instead it is a powerful, enduring article with a global impact. its forward-thinking focus on evaluation and evidence continues to serve the needs of scholars both inside and outside of librarianship. it is well-deserving of the descriptor “classic”.     references   argüelles, c. (2012). program-integrated information literacy (piil) in a hospital's nursing department: a practical model. journal of hospital librarianship, 12(2), 97-111. doi: 10.1080/15323269.2012.665717   barnard, a., nash, r., & o'brien, m. (2005). information literacy: developing lifelong skills through nursing education. journal of nursing education, 44(11), 505-510.   baro, e. e., & ebhomeya, l. (2013). information needs and seeking behaviours of nurses: a survey of two hospitals in bayelsa state, nigeria. health education, 113(3), 183-195.   brettle, a. (2003). information skills training: a systematic review of the literature. health information & libraries journal, 20(suppl. 1), 3-9. doi: 10.1046/j.1365-2532.20.s1.3.x   brettle, a., & raynor, m. (2013). developing information literacy skills in pre-registration nurses: an experimental study of teaching methods. nurse education today, 33(2), 103-109. doi: 10.1016/j.nedt.2011.12.003   carlock, d., & anderson, j. (2007). teaching and assessing the database searching skills of student nurses. nurse educator, 32(6), 251-255. doi: 10.1097/01.nne.0000299477.57185.ba   carter-templeton, h. d. (2011). nurses’ information appraisal within the clinical setting. (doctoral dissertation the university of tennessee). retrieved 4 dec. 2013 from http://etd.uthsc.edu/world-access/carter-templeton/2011-040-carter-templeton.pdf      centers for disease control and prevention. (1999). framework for program evaluation in public health. mmwr: recommendations and reports, 48(rr. 11), 1-40.   cobus, l. (2008). integrating information literacy into the education of public health professionals: roles for librarians and the library. journal of the medical library association, 96(1), 28-33. doi: 10.3163/1536-5050.96.1.28   craig, a., & corrall, s. (2007). making a difference? measuring the impact of an information literacy programme for pre-registration nursing students in the uk. health information and libraries journal, 24(2), 118-127. doi: 10.1111/j.1471-1842.2007.00688.x   davies, r., urquhart, c. j., smith, j., & hepworth, j. (1997). establishing the value of information to nursing continuing education: report of the evince project. (44). west yorkshire: british library research and innovation centre.   durando, p., & oakley, p. (2005). developing information literacy skills in nursing and rehabilitation therapy students. journal of the canadian health libraries association, 26(1), 7-11. doi: 10.5596/c05-007   eimas, r., & barton, h. (2001). web-based instruction for undergraduate nurses. in b. i. dewey (ed.), library user education: powerful learning, powerful partnerships (pp. 283-287). lanham, md: scarecrow press.   eldredge, j. (2002). cohort studies in health sciences librarianship. journal of the medical library association, 90(4), 380-392. retrieved 4 dec. 2013 from http://www.ncbi.nlm.nih.gov/pmc/articles/pmc128954/     fox, l. m., richter, j. m., & white, n. (1989). pathways to information literacy. journal of nursing education, 28(9), 422-425.   fox, l. m., richter, j. m., & white, n. e. (1996). a multidimensional evaluation of a nursing information-literacy program. bulletin of the medical library association, 84(2), 182-190. retrieved 4 dec. 2013 from http://www.ncbi.nlm.nih.gov/pmc/articles/pmc299399/   frier, m. d. (2009). comments on information literacy interventions in nursing and engineering courses at the cape peninsula university of technology, south africa. retrieved 4 dec. 2013 from http://athlonenursingcollege.wikispaces.com/   grant, m. j., & brettle, a. j. (2006). developing and evaluating an interactive information skills tutorial. health information and libraries journal, 23(2), 79-88. doi: 10.1111/j.1471-1842.2006.00655.x   greer, a. (1991). assessment of learning outcomes: a measure of progress in library literacy. college and research libraries, 52(6), 549-557. retrieved 4 dec. 2013 from http://hdl.handle.net/2142/41432   honey, m. l. (2007). teaching and learning with technology as enabler: a case study on flexible learning for postgraduate nurses. (doctoral dissertation). the university of auckland, auckland, new zealand. retrieved 4 dec. 2013 from https://researchspace.auckland.ac.nz/bitstream/handle/2292/2383/02whole.pdf?sequence=4      hopkins, b., callister, l. c., mandleco, b., lassetter, j., & astill, m. (2011). librarians as partners of the faculty in teaching of scholarly inquiry in nursing to undergraduates at brigham young university. science & technology libraries, 30(3), 267-276. doi: 10.1080/0194262x.2011.593416   jacobs, s. k., rosenfeld, p., & haber, j. (2003). information literacy as the foundation for evidence-based practice in graduate nursing education: a curriculum-integrated approach. journal of professional nursing, 19(5), 320-328. doi: 10.1016/s8755-7223(03)00097-8   jette, s., tribble, d.s., gagnon, j., & mathieu, l. (2010). nursing students' perceptions of their resources toward the development of competencies in nursing informatics. nurse education today, 30(8), 742-746. doi: 10.1016/j.nedt.2010.01.016   kleibel, v., & mayer, h. (2005). literaturrecherche für gesundheitsberufe. wien: facultas.   mackibbon, a., wilczynski, n., & eady, a. (2009). pdq: evidence-based principles and practice. shelton, ct: people's medical publishing house.   martin, s. (2008). reflections on a user education session with nursing students. health libraries review, 15(2), 111-116. doi: 10.1046/j.1365-2532.1998.1520111.x   meldrum, a. m., & tootell, h. (2004). integrating information literacy into curriculum assessment practice: an informatics case study. journal of university teaching & learning practice, 1(2), 49-58. retrieved 4 dec. 2013 from http://ro.uow.edu.au/jutlp/vol1/iss2/2   meneses placeres, g. (2008a). aproximaciones teóricas a la evaluación de la alfabetización informacional en la educación superior. acimed, 18(1). retrieved 4 dec. 2103 from http://hdl.handle.net/10760/12314   meneses placeres, g. (2008b). la evaluación en la alfabetización informacional en el contexto de la educación superior: aproximación teórica. biblios, (31), 1-11. retrieved 4 dec. 2013 from http://eprints.rclis.org/12314/   morgan, p. d., fogel, j., hicks, p., wright, l., & tyler, i. (2007). strategic enhancement of nursing students information literacy skills: interdisciplinary perspectives. abnf journal, 18(2), 40-45.   nayda, r., & rankin, e. (2008). information literacy skill development and life long learning: exploring nursing students' and academics' understandings. australian journal of advanced nursing, 26(2), 27-33. retrieved 4 dec. 2013 from http://pandora.nla.gov.au/pan/81890/20090327-0014/www.ajan.com.au/vol26/26-2_nayda.pdf   nodarse rodríguez, m. (2005). la enseñanza de las ciencias de la información en el currículum de los estudiantes de medicina y de otras especialidades afines. acimed, 13(6). retrieved 4 dec. 2103 from http://eprints.rclis.org/7963/   powell, c. a., & case-smith, j. (2003). information literacy skills of occupational therapy graduates: a survey of learning outcomes. journal of the medical library association, 91(4), 468-477. retrieved 4 dec. 2103 from http://www.ncbi.nlm.nih.gov/pmc/articles/pmc209513/   qun, w. (2011). [information literacy education involved in research and innovation activities]. medical information (china), 21. retrieved 4 dec. 2013 from http://www.cqvip.com/qk/98226b/201121/39906379.html   rush, k. l. (2008). connecting practice to evidence using laptop computers in the classroom. cin-computers informatics nursing, 26(4), 190-196. doi: 10.1097/01.ncn.0000304803.38746.ad   saranto, k., & hovenga, e. j. s. (2004). information literacy what it is about? literature review of the concept and the context. international journal of medical informatics, 73(6), 503-513. doi: 10.1016/j.ijmedinf.2004.03.002   schloman, b. f. (2001). information literacy: the benefits of partnership. online journal of issues in nursing, 6p. retrieved 4 dec. 2013 from www.nursingworld.org/mainmenucategories/anamarketplace/anaperiodicals/ojin/columns/informationresources/informationliteracy.aspx   shorten, a., wallace, c. & crookes, p. (2001). developing information literacy: a key to evidence‐based nursing. international nursing review, 48(2), 86-92. doi: 10.1046/j.1466-7657.2001.00045.x   staropoli, c., & waltz, c. f. (1978). developing and evaluating educational programs for health care providers. philadelphia, pa: f.a. davis.   sundin, o. (2003). informationssökning i övergången från utbildnings-till yrkespraktik en kunskapsöversikt. borås, sweden: department of library and information science, university of borås.   tronstad, b., phillips, l., garcia, j., & harlow, m. a. (2009). assessing the tip online information literacy tutorial. reference services review, 37(1), 54-64.                 doi: 10.1108/00907320910934995   urquhart, c. (1998). personal knowledge: a clinical perspective from the value and evince projects in health library and information services. journal of documentation, 54(4), 420-442. doi: 10.1108/eum0000000007176   wallace, m. c., shorten, a., crookes, p. a., mcgurk, c., & brewer, c. (1999). integrating information literacies into an undergraduate nursing programme. nurse education today, 19(2), 136-141. doi: 10.1054/nedt.1999.0621   weinert, d. j., & palmer, e. m. (2007). influence of an information literacy course on students' information search behavior. journal of allied health, 36(1), e1-e12.   wright, l., & mcgurk, c. (2000). integrating information literacy: university of wollongong experience. in bruce, c. & candy, p. (eds), information literacy around the world: advances in programs and research.(pp. 83-97). wagga wagga, new south wales: centre for information studies, charles sturt university.   editorial   evidence based practice: an opportunity?   alison brettle editor-in-chief school of nursing, midwifery and social work university of salford, united kingdom email: a.brettle@salford.ac.uk      2013 brettle. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   welcome to the december 2013 issue of the eblip journal. looking back it is hard to believe that i have been editor-in-chief for two years. my first issue contained a feature of the eblip6 conference, and i’m delighted that this issue contains a feature of the eblip7 conference, which was held in saskatoon in july 2013.   i’ve spent some time reflecting and looking back over the last few weeks; not because i had nothing better to do, but because i was asked to do a keynote speech about my research over the last 10 years (brettle, 2013). the event was to mark a 10 year anniversary of a health library at coventry university hospital in the uk and my speech was entitled “evidence based practice: an opportunity for health librarians?”    over 10 years ago (and before eblip was conceived), ebp was heralded as an opportunity for health librarians to promote their library and expand into new roles (palmer, 1996; scherrer & dorsch, 1999). when i critically appraised the literature on this (brettle, 2009), i was quite disappointed to find that although ebp had provided the opportunity for promoting the health library, for the most part, librarians’ roles hadn’t really changed, and the work that they were engaging in, in relation to evidence based practice, was mainly supporting literature reviews and training clinicians to undertake their own searching.    i classified the roles that health librarians may play in relation to evidence based practice into four domains:   1.             a supportive role, using traditional skills and expertise to train clinicians to find evidence for practice, to find evidence on behalf of clinicians, or to provide expert searching and guidance in relation to systematic reviews 2.             a supportive role, using their knowledge of critical appraisal to teach critical appraisal or to critically appraise search results to support health professionals in their ebp 3.             an active role, using research or critical appraisal to conduct systematic reviews or engage in eblip in their own domain (i.e., libraries or health information) 4.             an active role, using the ebp skills (critical appraisal or research) outside the library but in the health and social care domain (brettle, 2009, 2012)   at the time, i found few examples within the literature of librarians using their ebp skills outside their own domain, but there were quite a few examples, perhaps not surprisingly, of librarians engaging in eblip to improve the evidence base of their own library practice (brettle, 2009). one of the problems is that there are few descriptions of librarians engaging in different roles within the literature. i am guilty of this myself, having engaged in systematic reviews outside the library (e.g., brettle, hill, & jenkins, 2008; dugdill, brettle, hulme, mccluskey, & long, 2008), i did not describe my role, which in both cases involved leading the systematic review itself as well as critically appraising the health literature; pretty active and outside the comfort zone of health libraries! for me, engaging in evidence based activities inside library and information practice gave me the confidence and skills to branch outside my familiar domain, and then on to new roles and opportunities. other librarians have done similar things (brettle & urquhart, 2012). so, i urge you to engage in eblip, as you never know where it may take you – to saskatoon and beyond?     references   brettle, a. (2013 nov. 13). evidence based practice: an opportunity for health librarians? [keynote lecture] doing what it says on the tin? 10 year celebration of health librarianship: past, present & future.  university hospital coventry. uk   brettle, a. (2012). the librarian within research and evidence based practice.  in brettle, a., & urquhart, c. (eds.). (2012). changing roles and contexts for health library and information professionals. london: facet. (pp.135-160).   brettle, a. (2009). exploring the roles, effectiveness and impact of health information professionals within evidence based practice. salford: university of salford. retrieved 21 nov. 2013 from http://usir.salford.ac.uk/12960/   brettle, a., & urquhart, c. (eds.). (2012). changing roles and contexts for health library and information professionals. london: facet.   brettle, a., hill, a., & jenkins, p. (2008). counselling in primary care: a systematic review of the evidence. counselling & psychotherapy research, 8(4), 207-214.   dugdill, l., brettle, a., hulme, c., mccluskey, s. & long, a. f. (2008). workplace physical activity interventions: a systematic review. international journal of workplace health management, 1(1), 20-40.   palmer, j. (1996). effectiveness and efficiency: new roles and new skills for health librarians. aslib proceedings, 48(10), 247-252. doi:10.1108/eb051433   scherrer, c. s., & dorsch, j. l. (1999). the evolving role of the librarian in evidence-based medicine. bulletin of the medical library association, 87(3), 322-328. microsoft word es_perryman.doc evidence based library and information practice 2007, 2:2  81 evidence based library and information practice       evidence summary    does a social network based model of journal metrics improve ranking?      a review of:  bollen, johan, herbert van de sompel, joan a. smith and rick luce. “toward alternative  metrics of journal impact: a comparison of download and citation data.” information  processing and management 41.6 (2005):1419‐40.    reviewed by:   carol perryman  trln doctoral fellow, school of information & library science  university of north carolina at chapel hill  chapel hill, north carolina, united states of america  e‐mail: cp1757@gmail.com      received: 01 march 2007  accepted: 09 april 2007      © 2007 perryman. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to test a new model for  measuring journal impact by using  principles of social networking. research  questions are as follows:  1. can valid networks of journal  relationships be derived from reader article  download patterns registered in a digital  library’s server logs?  2. can social network metrics of journal  impact validly be calculated from the  structure of such networks?  3. if so, how do the resulting journal impact  rankings relate to the isi impact factor (if)?    design – bibliometric, social network  centrality analysis    setting – los alamos national laboratory  (lanl), new mexico    subjects – 40,847 full‐text articles  downloaded from a large digital library by  1,858 unique users over a 6 month period.    methods – full‐text article downloads from  a large digital library for a six‐month period  were examined using social networking  analysis methods. issns for journals in  which the retrieved articles were published  were paired based upon the proximity of  use by the same user, based on the  supposition that proximal downloads are  related in some way. reader‐generated  networks (rgns) were then tested for  small‐world characteristics. the resulting  rgn data were then compared with  author‐generated networks (agns) for the  same journals indexed in the institute of  scientific information (isi) annual impact  factor (if) rankings, in the journal citation  mailto:cp1757@gmail.com http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:2  82 reports (jcr) database. next, a sample of  the agn‐derived pairings was examined by  a team of 22 scientists, who were asked to  rate the strength of relationships between  journals on a five‐point scale.  centrality  ratings were calculated for the agn and  rgn sets of journals, as well as for the isi if.     main results – closeness and centrality  rankings for the isi if and the agn metrics  were low, but significant, suggesting that  centrality metrics are an acceptable impact  metric. comparison between the rgn and  isi if data found marked differences, with  rgn mirroring local population needs to a  much higher degree, and with a non‐ significant correlation between the isi if  and rgn ranking, while agn and rgn  centrality rankings show significant  centrality and closeness and betweenness  correlations. rgn network ranking  identified highly localized foci of interest for  the lanl, as well as “interest‐bridging”  subject areas pointing to possible emerging  interests among the scientists.    conclusion – the study results appear to  successfully demonstrate an alternative to  existing journal impact ranking that can  more validly and accurately reflect the  practices of a local community. the authors  suggest that the social network‐derived  methodology for identification of impact  rankings avoids biases intrinsic to isi if as a  result of frequentist metrics collected from a  global user group. although the authors  resist the idea of generalizability due to the  local nature of their data, they suggest that  the methodology can be successfully used in  other settings, and for a more global  community. finally, the authors propose the  automated creation of an open‐source rgn  whose data could be localized for smaller  communities, with potentially large  implications for the existing publishing  industry.      commentary      bollen et al. build a compelling case, as  others have done, for the need to find  alternative methods for journal impact  rankings in an era where more and more  grey literature is being published. a  particularly salient point is the concern that  isi’s impact ranking is based upon a global  community, which does not necessarily  mirror the practices of a local population –  and nor can it do so, based upon its  structure. similar to isi, the proposed  methodology concentrates on a subset of the  published literature (specifically, that which  is accessed via a digital library). although  mention is made of isi’s exclusion of grey  literature in its rankings, bollen, et al. do not  discuss how their new journal impact model  might incorporate online or other sources.      one assumption made by the authors is that  downloaded full‐text articles are analogous  to works cited.  in somewhat the same way  that a google search often retrieves  thousands of false hits, the searcher may  frequently download texts in excess of need.  one could justifiably argue that selection  equals preference, but preference does not  equal final selection.  the result might be  more akin to a “related items” retrieval such  as pubmed performs – useful, interesting,  but not necessarily pertinent.  the authors  address this issue by having scientists  examine journal‐to‐journal connections for  articles downloaded together, asking them  to make judgments about the strength of  thematic similarities. ultimately, the derived  data are generated from a localized  frequentist model, which is not a quality  judgment. this assumption, operationalized  into a journal impact ranking system, may  introduce error of its own. additionally, no  mention is made of interrater reliability for  this part of the process, although the expert  examination itself was used as a check  system.     evidence based library and information practice 2007, 2:2  83 the authors state that the download  frequencies for individual users are not  particularly significant in terms of journal  impact operations.  further questions might  be, “at what level is significance reached?”  and whether rankings constructed within  smaller, more localized networks (or those  with less than voluminous downloads) are  less valid than a larger network. co‐retrieval  frequencies were found to be highly skewed,  with the highest journal retrievals at 22 and  18, but fully 5,250 of the articles were  retrieved only one time.  comparison with  unique ip addresses might have shown  persistent use by a single user, meaning that  a heavy user could significantly skew the  impact rankings, which might be of concern  to tenure and promotion committees, but no  mention is made of this issue.    the findings of this study about the use of  social networking methodologies to  construct a new journal impact ranking that  is potentially scalable will be of interest to  researchers and those who work with the  existing isi if in an institution or other  locality. the authors have advanced the  applicability of small‐world and social  networking metrics to other data, and  provide a reproducible, carefully rigorous,  and clearly written explanation of their  processes.  other new knowledge provided  by bollen et al. is a taxonomy of journal  impact rankings, a valuable addition to the  literature on this problematic topic. finally,  a social networking‐structured impact  methodology has the potential to identify  emerging research trends among local  populations, an important contribution in  itself.                  librarian expertise is under-utilized by students and faculty in online courses evidence summary   librarian expertise is under-utilized by students and faculty in online courses   a review of: steele, j.e. (2021). the role of the academic librarian in online courses: a case study. journal of academic librarianship, 47(5), 102384. https://doi.org/10.1016/j.acalib.2021.102384 reviewed by: kimberly mackenzie research data and scholarly communications librarian university of massachusetts medical school worcester, massachusetts, united states of america email: kimberly.mackenzie@umassmed.edu   received: 1 mar. 2022                                                                    accepted:  28 apr. 2022      2022 mackenzie. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30114     abstract   objective – to examine the role of academic librarians in online courses in a university setting.   design – survey questionnaire.   setting – a multi-campus university in the southern united states.   subjects – students, faculty, and librarians who had taken, taught, or assisted in fully online courses.   methods – email addresses for potential survey participants were provided by the university office of institutional research. the researchers tailored survey questions weto specific subject groups. the surveys took roughly 15 minutes to complete and were open for 1 week following the original email. surveys included 12 – 16 questions, depending on the version, and included questions relating to the use of librarians in online courses, the type of assistance they provided, and how assistance was provided (e.g., in person, email, live chat). question types included yes/no, check-all-that-apply, and open-ended-answer.   main results – of the student responders, 23.24% reported asking a librarian for help with research or an assignment. this help included finding resources (34.48%), database searching (28.57%), and searching the library catalog (20.69%). help was given over email (28.03%), live chat (31.82%), and in person (17.42%), which was reported to be most helpful by several students. only 10.61% reported using video-conferencing software such as zoom.   only 5.88% of faculty reported including a librarian for synchronous instruction in online courses, while 19.12% made use of asynchronous tutorials created by a librarian. the majority of respondents (93.1%) had not worked with an embedded librarian in their courses, and many reported not knowing that it was an option. instead, faculty perceived librarians to be an outside resource.   both faculty members and students reported a desire for more video tutorials from librarians. several faculty mentioned wanting a library module that could serve as an introduction to the library, library resources, and basic instruction topics such as citation styles.   conclusion – while some students and faculty have worked with librarians in online courses and welcomed their involvement, there is room for improvement in library outreach, including how the library communicates with and supports this growing population.   commentary   boynton and greenhalgh’s (2004) guide to questionnaire research appraisal was consulted for this analysis. a survey questionnaire was the appropriate method to capture the information needed, and this survey used a combination of statements with tick boxes and open-ended questions, which allowed participants to give concrete answers when necessary and voice their opinions and experiences further on given topics. the questions were clearly written and directed to each subject group, for example, “the online course(s) i have completed have been …” for students versus “the online course(s) i have taught have been…” for faculty.   there were a number of issues with the presentation of data. first, it was not clear how many respondents there were from each group or what the response rate was. while one could infer that there were 370 student respondents based on the responses to an early question in the student survey, there is no way to definitively tell. when presenting the results of a survey, the number of participants and the response rate should be included to help the reader determine whether there was an appropriate sample size.   appropriate sample size was a second problem, in that librarians were included in the study design as potential survey respondents but were not discussed in the results section. while the significantly lower number of librarian respondents was mentioned as a limitation of the study, the data still should be reported as results and the response rate reported. the views of the librarians who did respond to the survey could have added further meaning to the responses of students and faculty.   the survey results make clear that faculty did not realize librarians could be an embedded resource for their classes. an interesting question could have been whether they were contacted in the past by a librarian who offered to help with classes. it would also be interesting to know how students knew to contact a librarian for help, as they were not explicitly asked whether they knew about the library from a course or another experience.   despite the issues mentioned, academic librarians can use this research to determine the needs of their students and faculty regarding online courses. this type of survey could be adapted by any library to gauge interest in and need for embedded library instruction. faculty outreach to increase awareness of library resources, both in-person and online, could improve engagement. outreach to online students can also allow for better understanding of available library resources. as more schools are embracing online classes, librarians will have to adapt their methods and may find new opportunities to directly support these classes.    references   boynton, p. m., & greenhalgh, t. (2004). selecting, designing, and developing your questionnaire. bmj, 328(7451), 1312–1315. https://doi.org/10.1136/bmj.328.7451.1312   evidence summary   academic historians in canada report both positive and negative attitudes towards e-books for teaching and research   a review of: martin, k. & quan-haase, a. (2013). are e-books replacing print books? tradition, serendipity, and opportunity in the adoption and use of e-books for historical research and teaching. journal of the american society for information science and technology, 64(5), 1016-1028. doi:10.1002/asi.22801   reviewed by: heather coates digital scholarship & data management librarian indiana university-purdue university indianapolis (iupui) university library indianapolis, indiana, united states of america email: hcoates@iupui.edu   received: 26 aug. 2013 accepted: 25 oct. 2013      2013 coates. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to understand academic historians’ attitudes towards, and perceptions of, e-books for use in teaching and research.   design – qualitative analysis of semi-structured interviews using a grounded theory approach.   subjects – ten faculty members in departments of history at academic institutions in southwestern ontario participated.   methods – participants were recruited using flyers and email distribution lists. the authors conducted semi-structured interviews lasting 30-60 minutes, between october 2010 and december 2011. after 10 interviews, the authors determined saturation had been reached and ceased recruitment. interviews were recorded and transcribed for coding. analysis was conducted using grounded theory procedures incorporating roger’s innovation decision model.   main results – the authors elicited participants' perceptions of e-books without providing a common definition for the concept. consistent with previous studies, participants were confused about what constituted an e-book, particularly the distinction between e-books and electronic journals and databases. several comments included illustrate this confusion, indicating the responses collected may represent perceptions of e-resources more generally, rather than e-books in particular. the authors mention that at least one participant who initially responded that they had not used e-books later changed their response as the interview progressed. unfortunately, the exact number of participants who did so is not reported.   participants reported both negative and positive attitudes towards e-books. attitudes varied depending on the characteristic discussed. the characteristics identified focused primarily on the delivery mechanism, rather than the content, of e-books. the authors identified four factors each as contributing to positive and negative attitudes. factors associated with a negative attitude included availability, serendipity, cost, and tradition. these factors stemmed from concerns about changing student research behaviours resulting from the differences between e-books and print books. factors associated with a positive attitude included convenience, teaching innovations, research practices, and cost benefits. these factors largely reflected benefits to students, such as the ability to access e-books easily (convenience), increased access in general, and the perceived relatively low cost of student e-books. the factor directly benefitting respondents was improved speed and accuracy in their work, enabled by particular technological features. while participants were eager to use e-books in the classroom, there were concerns about implications for research practices. participants worried that the benefits of browsing and serendipitous discovery would be lost as students chose materials based on convenience rather than other factors, such as quality. finally, the perceived lack of digitized historical documents available for use as primary sources was also of concern.   conclusions – the authors state that confusion regarding the nature of e-books slows adoption. while participants were exploring ways to incorporate e-books into their norms, values, and research practices, they are unlikely to rely solely on e-books as primary sources. this stems from two perceptions. first, current e-book formats and platforms do not authentically represent all the characteristics of print books. second, there are insufficient primary sources available as e-books. the validity of these perceptions is not addressed in this article.     commentary   literature on humanists’ information seeking behaviour and use of technology suggests that they are often early adopters if it improves the efficiency of their research. this belies the popular notion of historians as reluctant adopters of technology. in fact, historians seem to approach new technologies pragmatically. while there is substantial literature on technology use of humanists in general, there is little literature on information and communication technology (ict) adoption practices in particular disciplines. while examining e-book use as an ict enables separation of content from the package and delivery mechanisms, confusion remains about the nature of e-books. the confusion centers on the distinctions between the content and the delivery mechanism and the e-book format versus other e-resources. what is the difference between an e-book and another long-form document available electronically?   the article was evaluated using the critical appraisal skills programme qualitative research checklist (casp, 2013). the choice of semi-structured interviews and analysis based in grounded theory are appropriate for the exploratory nature of the study. the rationale for the study is generally strong and includes careful consideration of roger’s innovation decision model. the choice to limit questions to two phases of the model – knowledge and persuasion – limits the usefulness of the results and introduces uncertainty regarding the conclusions. a second concern is the lack of detail provided for sampling and recruitment methods. more specifically, demographics of the population sampled, recruitment strategies, development of the interview guide, and the process for developing the coding scheme need to be described in greater detail. given limited information about the population and sample, it is difficult to determine the relevance of these results for other populations of historians. future studies should include questions addressing all phases of the adoption process. however, considering the lack of evidence regarding the uptake of e-books by particular communities, this study begins to fill a gap in our understanding of historians’ considerations for adopting new ict.   participant confusion about the distinctions between e-books and other e-resources likely is reflected in the data. it is unclear how comments on other types of e-resources were handled, which raises questions about validity of the data. at times, the authors themselves conflate issues related to content versus delivery. the authors conclude that participants were in the beginning stages of adoption. since the interview questions did not include later phases of the model, this conclusion is unsubstantiated.   this confusion speaks to the diversity of functions and features provided by online reading tools, digital rights management software, reading devices, and search interfaces (e.g., google books). since the adoption of a standardized set of features is unlikely in the near future, librarians can increase researchers’ understanding of the benefits and drawbacks of e-books through educational programs and individual consultations. these opportunities could include discussions about balancing the risk of lost serendipity in the discovery process with the increased ease in discovery, access, and analysis, thus addressing the relative advantage of e-books. additionally, collaborating to develop ways to replicate browsing experiences and serendipitous discovery would engage historians more deeply with the technology and provide useful insights into their research practices. as mentioned by the authors, evidence quantifying e-book availability for this group would be valuable information for both historians and librarians. librarians are often challenged to make collection decisions based on little or no evidence about the practices of our patrons. evidence on the adoption of e-books by various disciplines, along with attitudes and relevant practices, would allow librarians to make better informed collection decisions and use of increasingly limited funds. finally, researchers addressing this topic going forward should clearly define the scope of the e-book for participants, perhaps even differentiating consumption by device or platform.     references   critical appraisal skills programme. (2013). qualitative research checklist. retrieved 14 jul. 2013 from http://www.casp-uk.net/wp-content/uploads/2011/11/casp-qualitative-research-checklist-31.05.13.pdf   conference paper   coding practices for libqual+® open-ended comments   karen neurohr assessment librarian oklahoma state university stillwater, oklahoma, united states of america email: karen.neurohr@okstate.edu   eric ackermann head, reference services and library assessment radford university radford, virginia, united states of america email: egackerma@radford.edu   daniel p. o'mahony director of library planning and assessment brown university library providence, rhode island, united states of america email: dpo@brown.edu   lynda s. white associate director of library assessment university of virginia library charlottesville, virginia, united states of america email: lsw6y@virginia.edu      2013 neurohr, ackermann, o’mahony, and white. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – this paper presents the results of a study of libraries’ practices for coding open-ended comments collected through libqual+® surveys and suggests practical steps for facilitating this qualitative analysis.   methods – in the fall of 2009, survey invitations were sent to contacts at 641 institutions that had participated in the libqual+® survey from 2003 to 2009. of those invited, there were 154 respondents, for an overall response rate of 24.0%.   results – nearly 87% of the respondents indicated that their library had performed a qualitative analysis of the comments from their most recent libqual+® survey. of these, over 65% used computer software to organize, code, sort, or analyze their comments, while 33.6% hand-coded their comments on paper. of the 76 respondents who provided information on software, 73.7% used excel, 18.4% used atlas.ti, and 7.9% used nvivo. most institutions (55.8%) had only 1 person coding the comments; 26.9% had 2 coders, and very few had 3 or more. of those who performed some type of analysis on their comments, nearly all (91.9%) indicated that they developed keywords and topics from reading through the comments (emergent keywords). another common approach was to code the comments according to the libqual+® dimensions; 55.0% of respondents used this strategy. nearly all of the institutions (92.7%) reported using their libqual+® comments internally to improve library operations. libraries also typically incorporated the comments into local university reports (75.5%) and used the comments in outreach communications to the university community (60.9%).   conclusion – comments obtained from the libqual+® survey can be useful for strategic planning, understanding users, identifying areas for improvement, and prioritizing needs. a key suggestion raised by respondents to this survey was for practitioners to consider sharing the fruits of their labor more widely, including coding taxonomies and strategies, as well as broader discussion of qualitative analysis methods and practices.   introduction   since its launch in 2000, libqual+® has become the most prevalent library assessment instrument for measuring service quality.   libqual+® has been used to collect service quality assessment perceptions from 1,294,674 participants at 1,164 institutions around the world. libqual+® has been implemented in 28 language variations: afrikaans, chinese, danish, dutch, english (american, british, dutch, finnish, france, norwegian, swedish, swiss), finnish, french (british english-be, belge, canada, france, swiss), german (and german swiss), greek, hebrew, japanese, norwegian, spanish, swedish (and swedish be), and welsh (kyrillidou, thompson, & cook, 2011, p. 3).   a key component of the libqual+® survey data is the file of respondents’ free-text comments that accompanies the quantitative data – almost 40% of libqual+® respondents typically include narrative comments (green & kyrillidou, 2010, p. 26).   “[t]he open-ended comments gathered as part of libqual+® are themselves useful in fleshing out insights into perceived library service quality. respondents often use the comments box on the survey to make constructive suggestions on specific ways to address their concerns” (cook et al., 2008, p. 14). thus, systematic analysis of a library’s qualitative data from libqual+® can be extremely valuable in assessing the library’s performance and identifying areas for improvement. to better understand libraries’ current practices in analyzing and using libqual+® comments, the authors conducted a survey of all u.s. and canadian libraries that administered at least one libqual+® survey from 2003 through june 2009. survey questions asked respondents to describe what they did with the open-ended comments received from their libqual+® survey and probed aspects including coding methods, local resources for coding, and the use of comments for various purposes. this paper presents the survey findings as well as suggestions for practical steps to help facilitate qualitative analysis of libqual+® comments. the questionnaire can be found at http://www.library.okstate.edu/dean/neurohr/codingsurvey10-26-09.pdf.   literature review/bibliography   a search of the published, peer-reviewed library literature found 12 articles and conference papers produced by 11 academic libraries: bowling green state university (haricombe & boettcher, 2004); northeastern university (habich, 2009); notre dame (jones & kayongo, 2009); texas a&m (guidry, 2002; clark, 2007); university of arizona (begay, lee, martin, & ray, 2004); university of british columbia (friesen, 2009); university of idaho (jankowska, hertel, & young, 2006);  university of massachusetts-amherst (fretwell, 2009); university of pittsburgh (knapp, 2004); vanderbilt university (wilson, 2004); western michigan university (dennis & bower, 2008). these articles covered libqual+® surveys administered during the period from 2001 to 2007 and for the most part described the methodologies, experiences, and findings of individual libraries that performed some type of systematic analysis of their survey’s comments.   all 1 institutions represented in the literature review were doctorate-granting universities. seven of these 11 libraries were members of arl (begay et al., 2004; guidry, 2002; clark, 2007; jones & kayongo, 2009; fretwell, 2009; friesen, 2009; knapp, 2004; wilson, 2004). ten of the 11 institutions are located in the united states: 3 in the northeast, 3 in the south, 2 in the midwest, and 2 in the west, while the eleventh institution is located in canada.   the amount of detail reported in the literature review by libraries about the management of their coding projects was relatively sparse and inconsistent. only 3 of the 11 libraries represented in the literature review reported any project structure, all of which were ad hoc or informal (begay et al., 2004; habich, 2009; jankowska et al., 2006). three of the libraries reported the number of coders they used: one reported using one coder (habich, 2009), and two reported using two coders (dennis & bower, 2008; jones & kayongo, 2009). two non-librarians were involved in the coding (dennis & bower, 2008; guidry, 2002). only one of the libraries reported providing formal training for their coders by way of a consultant (begay et al., 2004) while another library’s coder was self-taught (habich, 2009). the remaining nine libraries did not provide any information on coder training.   all 11 of these libraries reported performing qualitative analysis on either all or a representative sample of the comments they received from the libqual+® surveys they conducted, which was part of the criteria for selecting these 11 articles. the average number of comments received by these 11 libraries was 1,031. seven of the 12 authors reported using computer software to help in the analysis (begay et al., dennis & bower, 2008; friesen, 2009; guidry, 2002; habich, 2009; haricombe & boettcher, 2004; jones & kayongo, 2009) while 5 did not report what coding method (by computer or by hand) they used (clark, 2007; fretwell, 2009; jankowska et al., 2006; knapp, 2004; wilson, 2004). of the seven libraries that reported using software, three used atlas.ti (dennis & bower, 2008; friesen, 2009; guidry, 2002), two used excel (habich, 2009; jones & kayongo, 2009), one used nud*ist – now called nvivo – (begay et al., 2004), and one used access (haricombe & boettcher, 2004).   the 11 libraries covered in the literature review varied in the way they developed a coding system for use in the analysis of their libqual+® comment data. five of the 11 reported basing their codes on the 3 libqual+® dimensions affect of service, information control, and library as place (friesen, 2009; habich, 2009; jankowska et al., 2006; jones & kayongo, 2009; wilson, 2004). three of the 11 libraries also based their coding on the individual libqual+® and/or local questions (friesen, 2009; habich, 2009; jones & kayongo, 2009). three of the libraries reported using a predetermined set of concepts or keywords (begay et al., 2004; haricombe & boettcher, 2004; jones & kayongo, 2009), while nine reported using keywords and concepts developed from the content of the comments (begay et al., 2004; clark, 2007; dennis & bower, 2008; fretwell, 2009; friesen, 2009; guidry, 2002; habich, 2009; haricombe & boettcher, 2004; jankowska et al., 2006). nine of the 11 libraries reported coding the distinct topics found within each comment in lieu of using 1 code for the entire comment (begay et al., 2004; dennis & bower, 2008; fretwell, 2009; friesen, 2009; guidry, 2002; habich, 2009; haricombe & boettcher, 2004; jones & kayongo, 2009; wilson, 2004). seven of the libraries also coded a comment “positive” or “negative” if it expressed such an experience with an aspect of the library (begay et al., 2004; dennis & bower, 2008; fretwell, 2009; friesen, 2009; guidry, 2002; habich, 2009; wilson, 2004). note that the use of each of the elements discussed above was not exclusive. each of these libraries reported using a different combination in developing their coding system. only one did not include any report of the elements it used to create its coding schema (knapp, 2004).   only 2 of the 11 libraries reported any detailed information about the steps they took to encourage or enforce coding consistency and reduce coding subjectivity during their projects. both reported that their coders worked using an understanding gained through prior discussion of how to apply the codes (begay et al., 2004; jones & kayongo, 2009), but only one had their coders work independently on randomly assigned sets of comments (begay et al., 2004). none of these libraries reported documenting their coding procedures.   all 11 of the libraries also reported using the results to communicate with other professionals in the field (begay et al., 2004; clark, 2007; dennis & bower, 2008; fretwell, 2009; friesen, 2009; guidry, 2002; habich, 2009; haricombe & boettcher, 2004; jankowska et al., 2006; jones & kayongo, 2009; knapp, 2004; wilson, 2004). few of the 11 libraries reported any further plans to use the results of their qualitative analysis. one library reported plans to incorporate some of their findings into their annual reports and other intra-university administrative reports (dennis & bower, 2008). only three planned to include the findings in outreach communications to their university (dennis & bower, 2008; habich, 2009; haricombe & boettcher, 2004) or to external groups (e.g., donors or potential donors; habich, 2009).   the libraries represented in the literature review reported several benefits from analyzing their comment data. two of the libraries gained a better understanding of library users’ needs and priorities (jones & kayongo, 2009; fretwell, 2009). one found a new source of ideas for new services (begay, 2004). three libraries found a new source for improving existing services (clark, 2007; friesen, 2009; wilson, 2004). one found a new source for maximizing the impact of limited resources (habich, 2009). three of the 11 libraries reported that they had developed a new tool for analyzing other data sets (begay et al., 2004; dennis & bower, 2008; jankowska et al., 2006). two discovered that the findings from analyzing the libqual+® comment data complemented and enhanced the findings from the quantitative data (dennis & bower, 2008; jones & kayongo, 2009).   only one of these libraries indicated the nature of the biggest challenge they encountered during the project, which was devising a method for comment analysis that did not require learning a new software program (habich, 2009). none of the libraries represented reported on what support from their institutions, vendors, or others they wished they had during the project. only one mentioned a resource they found helpful: the survey research expertise available in their university’s office of institutional research (habich, 2009).   methodology   libqual+® quantitative measures have been thoroughly investigated and validated, but what about the qualitative data? each survey includes an open-ended statement: “please enter any comments about library services in the box below.” how do libraries analyze and use the data received in response to this statement?   in the fall of 2008, a small working group began to study this question. the study was initially informed by feedback obtained by one of the authors new to libqual+® who queried the libqual-l discussion list in february 2008 by asking, “can anyone share information about how they coded the open-ended comments from the libqual+® survey?” the wide variety in the responses received led to the ad hoc formation of a luncheon affinity group to discuss coding at the 2008 library assessment conference in seattle. over 15 librarians participated in the affinity group and there was much interest in coding methodologies and practices. next, the authors met to discuss ways to explore coding, drafted a survey and planned for the survey’s distribution.   in september 2009, the survey questionnaire was piloted to a small group of 30 colleagues who had responded to the listserv query or participated in the affinity group. they assisted the authors in clarifying the wording and structure of the questionnaire by answering these questions about the draft:   1.       how long did it take to complete the survey? (the goal was 10 minutes or less.) 2.       can you answer the questions quickly/easily? 3.       are the questions clear? which are not? do you have suggestions for clarification? 4.       are the questions generic enough to cover most possible situations at your institution or others you are familiar with? 5.       other comments.   the association of research libraries (arl) provided generous assistance by emailing survey invitations to all of the contacts at north american institutions that participated in the libqual+® survey from 2003 through spring 2009. there were 641 institutions: 110 arl members (84 from the united states and 16 from canada) and 531 non-members (515 in the united states and 28 in canada). the first invitation was sent on october 27, 2009, followed by four reminders at one-week intervals. of those invited, there were 154 respondents for an overall response rate of 24.0%.   survey results   the survey asked what kind of institution the respondent was affiliated with by using the carnegie classifications for higher education. of the 151 responses to this question, 9.3% were from baccalaureate colleges, 36.4% from master’s colleges and universities, and 54.3% were from doctorate-granting universities (see figure 1). there were no responses from other types of institutions.   arl members comprised 35.1% of the respondents to the survey (figure 2). arl members were over-represented in the response, since only 17.2% of the 641 libraries in the sample were arl members.   a large majority of the 154 respondents (85.1%) were from the united states with the remaining libraries from canada (figure 3). nonetheless, canadian libraries were over-represented in the response, at 14.9%; only 9.4% of the 641 libraries in the sample were canadian.   a little more than 33% of the u.s. respondents were from the northeast section of the country, closely followed by the south and midwest. only 11.5% were from the western states (figure 4). for 60.2% of respondents, administration of the libqual+® survey was handled by a formal or standing group within the library. or by someone whose position included survey administration. thus, among these respondents, there appeared to be some permanent responsibility in their library for assessment (figure 5). nearly 40% implemented libqual+® through an informal or ad hoc team or project group.   figure 1 what is your institution type?   figure 2 does your library belong to the association of research libraries (arl)?       figure 3 country   figure 4 sections of the united states   nearly 87% of the respondents indicated that their library had performed a qualitative analysis of the comments from their most recent libqual+® survey (figure 6), where “qualitative analysis” was described as any process that organized or categorized or tagged/coded the free-text comments so that they might be used by library staff or others in assessing and/or improving library services. of those who did not perform analysis on their survey comments, the most frequently mentioned reason was lack of staff time. the average number of libqual+® comments received by responding libraries was 379. the median was 293 but the number of comments ranged from one to 1,420.   the survey asked those who had performed a qualitative analysis of their comments about the tools and methods they used in their approach. of the 114 responding libraries that provided answers, over 65% used some sort of computer software to organize, code, sort, or analyze their comments, while 33.6% hand coded their comments on paper (figure 7).   the survey revealed that coders primarily used excel to analyze the comments: of the 76 respondents that provided information on software, 73.7% used excel (figure 8). atlas.ti was the most common qualitative data analysis software used (18.4% for atlas.ti versus 7.9% for nvivo).   figure 5 libqual+® administrators   figure 6 did you perform qualitative analysis of the open-ended comments?   figure 7 coding methods   figure 8 software used (respondents could choose more than one option)   most respondents (58 out of 104 libraries, or 55.8%) had only 1 person coding the comments (figure 9). twenty-eight (26.9%) had 2 coders, but very few had 3 or more. thus, at over 80% of the responding libraries, either 1 or 2 people performed the coding. only 18 libraries (17.3%) had 3 or more people who did coding.   staff who performed the coding at respondents’ libraries were typically professional librarians: 84.2% of respondents indicated that librarians were coders while 25.4% used non-librarian staff (figure 10).   training for coders came from several venues, primarily libqual+® workshops run by arl (69.6%), but there was also a large contingent that was self-taught or who had taken formal courses in assessment methods (figure 11). “other” tended to be consultants from other areas of the local institution.   respondents used a number of approaches to code the comments (table 1). of those who performed some type of analysis on their comments, nearly all (91.9%) indicated that they developed keywords and topics from reading through the comments (emergent keywords). another common approach was to code the comments according to libqual+® dimensions (55.0% of respondents used this strategy). less common was coding according to the 22 individual libqual+® questions (done by only 27.0%). a couple of respondents specifically mentioned that creating a word cloud to visually display the key concepts that emerged from their libqual+® comments was an effective tool, especially in communicating their findings to others.   figure 9 number of institutions with n coders   figure 10 coder status (respondents could choose more than one option)   figure 11 training activities (respondents could choose more than one option)   table 1 basis for coding the comments basis for coding the comments:* % n emergent keywords or concepts (e.g., “service hours”) developed from reading the comments? 91.9% 102 whether or not it expressed a “positive” or “negative” perspective/experience of the library? 67.6% 75 the libqual+® dimensions: affect of service, information control, & library as place? 55.0% 61 the number of distinct topic(s) in a single respondent’s comment? 46.8% 52 a pre-set list of keywords or concepts (e.g., “service hours”)? 41.4% 46 the 22 individual libqual+® questions and/or the 5 local questions? 27.0% 30 other 10.8% 12   in order to enhance consistency and objectivity, a number of steps were often implemented, including training, using previous coding schemes, and having others check the work of a single coder (33% of “other”). see table 2.   roughly half (51.4%) of those responding to the survey did not document the process they used to code/analyze their libqual+® comments (table 3). the most common documentation produced was lists of tags/codes with definitions and descriptions of the procedure or methodology used.   nearly all (92.7%) of the responding libraries reported using their libqual+® comments internally to improve library operations (table 4). libraries also typically incorporated the comments into local university reports (75.5%) and used the comments in outreach communications to the university community (60.9%). notably, roughly half (46.4%) of respondents said they either did or planned to include their libqual+® comments in communications with professional communities (e.g., in conference presentations or professional publications).   benefits   the survey asked, “for your library, what was the best benefit of coding the comments?” the two most frequently mentioned benefits were (1) that the comments helped to identify action items for improvement, and (2) that the comments helped the library better understand its users (figure 12). other benefits included providing results and examples that can be communicated to various library constituents such as the provost or potential donors, identifying and analyzing specific needs and issues raised by users, identifying trends and patterns, and corroborating the quantitative survey data.   table 2 consistency in coding consistency in coding was assured by:* % n training and/or discussion was conducted ahead of time for all participants to ensure a common understanding of the application of the codes/tags 44.6% 37 coding schemes and definitions from previous survey administrations were consulted 44.6% 37 other (please specify) 43.4% 36 each comment was coded independently by at least two people 27.7% 23 comments were randomly assigned to people doing the coding 12.0% 10 *respondents could choose more than one option.   table 3 documentation type documentation type* % n none; did not document the process 51.4% 55 code book (list of tags/codes, definitions, examples, etc) 27.1% 29 description of procedure and methodology 25.2% 27 other (please specify) 17.8% 19 *respondents could choose more than one option.   table 4 uses of comment data uses of comment data* yes no plan to do internally within the library for operational improvements 92.7% 0.9% 6.4% incorporated into administrative reports to the university community (e.g., in annual report, budget request, etc.) 75.5% 7.3% 16.4% included in outreach communications to the university community (e.g., in announcements for new services) 60.9% 18.2% 17.3% included in communication with professional community (e.g., in conference presentations or professional publication) 25.5% 43.6% 20.9% included in outreach communications to external audiences such as donors or potential donors (e.g., demonstrate satisfaction with funded gifts or express need for funds, etc.) 22.7% 38.2% 27.3% other 3.6% 20.9% 0.9% *respondents could choose more than one option.   challenges   when it came to the biggest challenges of coding the comments, time constraints were mentioned most frequently (figure 13). time here referred not only to the duration of coding itself, but also included the time it took to learn new software, and the time to manage multiple coders. closely related to lack of time was the expressed challenge of lack of people/staff to perform the coding and analysis. another resource-related challenge was the lack of appropriate software.   respondents also described a number of challenges related to the process of performing the actual coding and analysis, including developing categories/groupings for coding schemes. other less frequently mentioned challenges included dealing with multiple concepts, maintaining consistency throughout the coding process, the difficulty in maintaining objectivity, and the need for assistance in analyzing and interpreting the data. some respondents also commented on the sheer volume of the qualitative data (the average number of comments per responding library was 379, with each comment likely to contain numerous concepts to be coded separately).   support needed   the survey asked, “what kind of support (from your library, institution, arl, software vendor, etc.) would be most helpful to you in doing qualitative analysis of libqual+® comments?” software purchase and software training were  cited most frequently (figure 14). respondents also made a number of suggestions regarding sharing information, experiences, and work products in conducting the coding of libqual+® comments, as well as sharing the results of the qualitative analysis. for example:   ·         “perhaps the sharing of the index terms that others have used” ·         “it might be interesting for a group … to draft a thesaurus and research commonalities and trends across universities." ·         “it would be great to share comments or types of comments, for informal benchmarking, similar to how we can compare our scores on items through the notebooks."     figure 12 best benefits of coding   figure 13 biggest challenges of coding   arl was gratefully acknowledged for their many workshops and training/sharing sessions on libqual+® generally, but there was also an expressed interest in online training/webinars on coding. in addition, a desire for basic training in qualitative research theory/methodologies was mentioned, as well as training for the actual coding and analysis. more staff to help with coding was desired by several respondents.   recommended resources   finally, the survey asked the respondents to recommend helpful resources for someone new at starting a coding project. the resource mentioned most often was arl with its myriad activities which include publications, the library service quality academy, the library assessment conference and proceedings, the libqual+® website and workshops, and the assessment listserv/blog (table 5). other resources mentioned included experts on campus, software vendors’ workshops and websites, and formal research courses. the works of two institutions were mentioned specifically: the brown university guide (http://old.libqual.org/documents/admin/brownu_2005_lq_qual_method.pdf) and articles from notre dame (see, for example, jones & kayongo, 2009).   figure 14 most helpful support for coding   table 5 recommended resources recommended resources: n arl activities 20 none or unsure 12 online resources 9 software manuals, training, tutorials, websites 7 articles, books 6 suggestions 5 formal and informal coursework 4 institutional, campus resources 3 manuals, guides 3   several specific resources were listed by survey respondents as helpful starting points for conducting qualitative research:   corbin, j. & strauss, a. (2008). basics of qualitative research. los angeles, ca: sage. (or another book on grounded theory generation)   richards, l. (2005). handling qualitative data: a practical guide. london: sage publications.   lapelle, n. (2004). simplifying qualitative data analysis using general purpose software tools. field methods, 16(1), 85-108.   online qda. (2012). school of human & health sciences, university of huddersfield. retrieved 30 may 2013 from http://onlineqda.hud.ac.uk/introduction/index.php                 šauperl, a. (n.d.). qualitative research methods in information and library science: an annotated bibliography of sources, in university of ljubljana, faculty of arts, department of library and information science and book studies.. retrieved 30 may 2013 from http://uisk.ff.cuni.cz/dwn/1003/1725cs_cz_qualitative%20research%20methods-bibliography.rtf   conclusion   comments obtained from the libqual+® survey can be useful for strategic planning, understanding users, identifying areas for improvement, and prioritizing needs. clearly, the survey results indicated a strong interest in systematically analyzing the open-ended comments from the libqual+® survey: nearly 87% of respondents performed qualitative analysis on their most recent libqual+® comments, and of that group more than 65% utilized a computer software tool in conducting that analysis. in more than half of the responding libraries, libqual+® analysis was conducted by individuals or groups with permanent responsibility for assessment. however, nearly 33% of respondents indicated they had no training and were self-taught regarding qualitative analysis.   overall, respondents expressed a strong desire for assistance in learning how to code and for knowing the best practices used by other libraries. far and away, microsoft excel was the tool of choice as nearly 75% of respondents used it for some aspect of their analysis. there appeared to be some confusion about the capabilities of text analysis software packages, presumably by those who had not used such a tool (e.g., several respondents commented on not using any software that “automatically” assigned codes to the text).   a key suggestion raised by respondents to this survey was for practitioners to consider sharing the fruits of their labor more widely (including coding taxonomies and coding strategies) as well as broader discussion of qualitative analysis methods, strategies, approaches, and practices. to this end, it was encouraging that more than half of the survey respondents indicated that they either already had or planned to include their libqual+® comments in communications with professional communities (e.g., in conference presentations or professional publications). such sharing of information, methods, and results should be welcomed given that the literature review performed as part of this study revealed very few items that focused on performing a systematic analysis of libqual+® comments.   administering a libqual+® survey typically results in a wealth of data, and librarians want to know how best to use it. performing qualitative analysis of the open-ended comments is a typical practice with multiple benefits accompanied by multiple challenges. a variety of tools and methods are utilized by libraries.   acknowledgement   the authors gratefully acknowledge the input of the librarians who responded to the arl libqual-l query, participated in the library assessment conference affinity group, tested and provided input for the pilot survey, and took the “libqual+® comment coding survey.” also, the authors thank arl for their work and cooperation in sending the survey invitations. we hope that this exploratory study helps describe the current state of practice of qualitative analysis among libqual+® libraries and provides a basis from which the emerging community of interest might grow.     references   begay, w., lee, d. r., martin, j., & ray, m. (2004). quantifying qualitative data: using libqual+® comments for library-wide planning activities at the university of arizona. journal of library administration, 40(3/4), 111-119. doi:10.1300/j111v40n03_09.   clark, d. (2007). practical assessment at texas a&m libraries: using libqual+® comments to enhance reference services. in f. defranco, s. hiller, l. j. hinchliffe, k. justh, m. kyrillidou, j. self, & j. stein (eds.), proceedings of the library assessment conference: building effective, sustainable, practical assessment, (pp. 91-94). washington, dc: association of research libraries. retrieved 30 may 2013 from http://libraryassessment.org/bm~doc/proceedings-lac-2006.pdf   cook, c., heath, f., thompson, b., davis, m., kyrillidou, m., & roebuck, g. (2008). libqual+® 2008 survey results notebook– arl. washington, dc: association of research libraries. retrieved 30 may 2013 from http://www.libqual.org/documents/admin/arl_notebook_2008.pdf   dennis, b. w., & bower, t. (2008). using content analysis software to analyze survey comments. portal: libraries and the academy 8(4), 423-437. doi:10.1353/pla.0.0015   fretwell, g. (2009). examining the overlooked: open-ended comments from 6,108 invalid 2007 libqual+® survey responses. in s. hiller, k. justh, m. kyrillidou, & j. self (eds.), proceedings of the library assessment conference:building effective, sustainable, practical assessment (pp. 443-448). washington, dc: association of research libraries. retrieved 30 may 2013 http://libraryassessment.org/bm~doc/proceedings-lac-2008.pdf   friesen, m. (2009). applying atlas.ti and nesstar webview to the libqual+® results at ubc library: getting started. in s. hiller, k. justh, m. kyrillidou, & j. self (eds.), proceedings of the library assessment conference: building effective, sustainable, practical assessment (pp. 449-455). washington, dc: association of research libraries. retrieved 30 may 2013 from http://libraryassessment.org/bm~doc/proceedings-lac-2008.pdf   green, d., & kyrillidou, m. (2010). libqual+® procedures manual, including the libqual+® lite feature. washington, dc: association of research libraries.   guidry, j. a. (2002). libqual+® spring 2001 comments: a qualitative analysis using atlas.ti. performance measurement and metrics 3(2), 100-107. doi:10.1108/14678040210429008   habich, e. c. (2009). analyzing libqual+® comments using excel: an accessible tool for engaging discussion and action.  in s. hiller, k. justh, m. kyrillidou, & j. self (eds.), proceedings of the library assessment conference: building effective, sustainable,practical assessment (pp. 417-423).washington, dc: association of research libraries. retrieved 30 may 2013 from http://libraryassessment.org/bm~doc/proceedings-lac-2008.pdf   haricombe, l. j., & boettcher, b. j. (2004). using libqual+® data in strategic planning: bowling green state university. journal of library administration, 40(3/4), 181-195. doi:10.1300/j111v40n03_14.   jankowska, m. a., hertel, k., & young, n. j. (2006). improving library service quality to graduate students: libqual+®  survey results in a practical setting.  portal: libraries and the academy, 6(1), 59-77. doi:10.1353/pla.2006.0005   jones, s., & kayongo, j. (2009). are they really that different?: identifying needs and priorities across user groups and disciplines at the university of notre dame through libqual+® user comments. in s. hiller, k. justh, m. kyrillidou, & j. self (eds.), proceedings of the library assessment conference: building effective, sustainable, practical assessment (pp. 425-441).  washington, dc: association of research libraries. retrieved 20 may 2013 from http://libraryassessment.org/bm~doc/proceedings-lac-2008.pdf     knapp, a. e. (2004). we asked them what they thought, now what do we do? the use of libqual+® data to redesign public services at the university of pittsburgh. journal of library administration, 40(3/4), 157-171. doi:10.1300/j111v40n03_12   kyrillidou, m., thompson, b., & cook, c. (2011, aug.). regrounding libqual+® for the digital library environment: an analysis of the digiqual data. paper presented at the 9th northumbria international conference on performance measurement in libraries and information services, york, england.   wilson, f. (2004). libqual+® 2002 at vanderbilt university: what do the results mean and where do we go from here? journal of library administration, 40(3/4), 197-240. doi: 10.1300/j111v40n03_15. microsoft word es_herron.doc evidence based library and information practice 2006, 1:4      64 evidence based library and information practice       evidence summary    students and graduates learn library educational content from interactive  multimedia tutorials     a review of:  markey, karen, annie armstrong, sandy de groote, michael fosmire, laura fuderer, kelly  garrett, helen georgas, linda sharp, cheri smith, michael spaly, and jonie. warner.  “testing the effectiveness of interactive multimedia for library‐user education.” portal:  libraries & the academy 5.4 (oct. 2005): 527‐54.      reviewed by:  david herron  scholarly developer, karolinska institutet, university library  stockholm, sweden  e‐mail: david.herron@kib.ki.se    lotta haglund  head of information and public relations, karolinska institutet, university library   stockholm, sweden  e‐mail: lotta.haglund@kib.ki.se      received: 31 august 2006            accepted: 11 october 2006      © 2006 herron and haglund. this is an open access article distributed under the terms of the creative  commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective –to demonstrate the effectiveness  of interactive multimedia tutorials in  delivering library educational content, and  to evaluate librarian experiences of  developing multimedia tutorials, both as  part of the lumens (drabenstott) project.    design – user study (questionnaire and  interviews) using pretest‐posttest design.  setting – four academic libraries in the  united states. one library dropped out  during the course of the project.    subjects – ninety university students from  the university of illinois chicago (uic),  purdue university, and the university of  notre dame participated in the main study  to evaluate three of the tutorials: “doing  research an introduction to the concepts of  online searching,” “how to read a scientific  paper,” and  “hungry for information?”  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:4      65 another group of 15 subjects from uic,  consisting of 10 graduate students, 2 faculty,  2 librarians, and one fellow, assessed a  fourth tutorial “keeping current in your  field.”  librarians were interviewed about  their experiences producing the interactive  multimedia tutorials.    methods – the 90 students were given a  pretest containing questions about library  educational content and five demographic  questions. the students used the  multimedia tutorial for 15‐30 minutes and  immediately afterward were given a  posttest containing comparable questions to  the pretest in terms of content and difficulty.  the students were also asked to rate their  experiences of using the tutorials in various  ways on a scale from 0‐10. at uic, the  experiences of the subjects using the  multimedia tutorial were assessed by  personal interviews. librarians producing  the multimedia tutorials were asked about  their experiences of developing multimedia  tutorials through e‐mail, listserv discussion,  phone calls, and face‐to‐face personal and  group interviews.    main results – all three libraries measured  a significant increase (using a one sample t‐ test, p<0.001) in marks when comparing the  pretest and posttest results. the changes in  mean marks were uic, 7.1 to 9;  purdue, 7.7  to 9.4; and notre dame, 6.2 to 9.   the results show that the majority (>75%) of  students were familiar with tutorial content  before start. despite this, most of the  students found the tutorials useful and  enjoyable, and the majority were fairly likely  to recommend the tutorial to a friend.  interviews with subjects at uic revealed  similar experiences, except that the subjects  were less familiar with the tutorial content  at the beginning, and they were more likely  to return to the tutorial for a refresher. the  tutorial with the highest amount of  interactivity was the most popular. the  librarians found it difficult to find time to  learn macromedia flash and to work within  the lumens project generally. eight out of  15 librarians remained with the project over  the entire period.     conclusion – students learned library  educational content by using multimedia  tutorials and seemed to enjoy the experience,  and educational librarians should lead  multi‐expert project teams in tutorial  production. finally, the educational value of  multimedia tutorials must be offset from the  time and effort needed to produce them.     commentary     the article presents some interesting  educational and managerial results which  should be of use to the library community.  educationally, the article demonstrates that  both undergraduates and graduates found  the interactive multimedia tutorials useful  and that both groups could improve their  information literacy by using tutorials. in a  recent systematic review, brettle showed  that the evidence base for the effectiveness  of library training is weak. this study  provides additional positive evidence that  training can be effective in the form of  interactive multimedia tutorials. from the  management point of view, the article  indicates that multi‐professional teams  would be more effective in tutorial  production.     the educational discussion is truncated by  the omission of the expected learning  outcomes for the tutorials and the library  content questions; these could have been  added in an addendum. this is not helped  by the virtual lack of references in the article  to other published studies. the high score  on the pretests also raises some doubt about  the intellectual challenge of the tutorials.     the total numbers of students at the  participating universities are not included  and therefore it is unclear how  evidence based library and information practice 2006, 1:4      66 representative the samples of 30 subjects are.  although the lumens project was carried  out between october 2001 and september  2003, it is unclear in this article exactly when  these studies were conducted.  the authors  use a one sample t‐test, which is a  parametric test assuming normality.  whether or not the authors checked for  normality is unclear. clarity would also be  improved if the specific statistical tests used  to test for differences in the demographic  variables were given in the text.      the results could be presented in fewer  figures and tables, and the authors could be  somewhat more consistent with their use of  units in the text. author use of a scale from  0‐10 leads to some confusion in the figures  since the first category contains three grades  while the others contain two. clarity might  also be improved if the figure axes had been  labeled.     the subjects seemed very positive about the  “keeping current” tutorial. however, one  wonders if they were also biased by  payment to take part in the interviews. it  would improve our understanding of the  representativeness of the uic sample if the  selection criteria for the 15 participants in  this part of the study were more clearly  stated.    although four libraries began the project,  one library (earlham college library)  dropped out for a number of reasons: the  time‐consuming nature of the project, the  amount of work involved, other work  commitments, changing work assignments,  and ample opportunities to deliver content  through “live instruction” (which decreased  motivation). this provides important  management information for libraries  interested in launching similar projects,  especially since the others libraries had  similar concerns in the final evaluation of  the project. this result underlines the need  for management to make realistic  predictions concerning time, human  resources, and (presumably) costs before  embarking on multimedia productions.     finally, the studies are fairly valid and  reliable (but not replicable in the present  form of the article) and they provide  reasonable evidence to support the practice  of educational librarians, multimedia  experts, and library managers in a  university library setting.     works cited    drabenstott, karen markey. “interactive  multimedia for library‐user  education.” portal: libraries and the  academy 3.4 (2003): 601‐13    brettle, alison. “information skills training:  a systematic review of the literature.”  health information & libraries journal  20.s1 (jun. 2003): 3‐9.    evidence summary   learners with low self-efficacy for information literacy rely on library resources less often but are more willing to learn how to use them   a review of: tang, y., & tseng, h. w. (2013). distance learners’ self-efficacy and information literacy skills. the journal of academic librarianship 39(6): 517-521. doi:10.1016/j.acalib.2013.08.008   reviewed by: dominique daniel humanities librarian for history and modern languages oakland university rochester, michigan, united states of america email: daniel@oakland.edu   received: 6 may 2014    accepted: 19 aug. 2014      2014 daniel. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objectives – to determine whether there is a relationship between self-efficacy (i.e., confidence) regarding information literacy skills and self-efficacy for distance learning; and to compare the use of electronic resources by high and low information literacy self-efficacy distance learners and their interest in learning more about searching.   design – online survey.   setting – a small public university in the united states of america.   subjects – undergraduate and graduate students enrolled in one or more online courses. most respondents were in their twenties, 76% were female, 59% were undergraduates, and 69% were full time students.   methods – students were asked six demographic questions, eight questions measuring their self-efficacy for information literacy, and four questions measuring their self-efficacy for online learning. all self-efficacy questions were adapted from previous studies and used a one to five likert scale. the response rate was 6.2%. correlational analysis was conducted to test the first two hypotheses (students who have higher self-efficacy for information seeking are more likely to have higher self-efficacy for online learning and for information manipulation). descriptive analysis was used for the remaining hypotheses, to test whether students who have higher information literacy self-efficacy are more likely to have high library skills (hypothesis three) and are more interested in learning about how to use library resources (hypothesis four). among respondents high information literacy self-efficacy and low self-efficacy groups were distinguished, using the mean score of information literacy self-efficacy.   main results – there was a significant correlation between self-efficacy for information seeking and self-efficacy for online learning (r = .27), as well as self-efficacy for information manipulation (r = .79). students with high information seeking self-efficacy were more likely to use library databases (28.72%), while low self-efficacy respondents more often chose commercial search engines (30.98%). however those respondents were more likely to be interested in learning how to use library resources.   conclusion – distance students with higher self-efficacy for information seeking and use also had higher self-efficacy for online learning. it is important to encourage such self-efficacy since studies have shown that it relates to better information literacy skills and a higher ability to be self-regulated learners. confident learners process information, make effective decisions, and improve their learning more easily. furthermore many respondents in this survey had little or false knowledge of how to use appropriate resources for their learning needs. this points to the need for effective library instruction. this study also shows that low self-efficacy students would like to have library instruction, especially to help them plan specific research assignments.     commentary   research about self-efficacy, namely people’s judgments about their ability to complete tasks and succeed, has long been of interest to librarians who seek to engage students in information literacy (il) instruction. this study examines the self-efficacy levels of online learners, and especially the relationship between self-efficacy in il and in online learning – a form of learning that requires strong self-regulation from students. particularly interesting is the finding that lower il self-efficacy students are generally more motivated to learn about it. thus self-efficacy may have important implications for students’ motivations to learn, and for librarians’ design of online instructional objects. in their survey, tang & tseng measured self-efficacy by asking respondents to agree or disagree with statements about their own abilities. this is an easy method, but the reliability of the scales used is not mentioned.   this study suffers from several weaknesses. first, it relies on a dated and incomplete literature review. for example, it uses a 2000 study to provide evidence that instruction sessions boost self-efficacy regarding the use of electronic resources (ren, 2000). furthermore it ignores the numerous studies that indicate negative relationships between self-efficacy and achievement, including recent research from the attaining information literacy project on student self-views, which shows that self-efficacy is stronger among students with lower abilities and is not a predictor of success (gross & latham, 2012). although they do not target distance learners, such findings complicate the picture shown by the studies cited in the literature review, and even contradict it.   one limitation of this study is that it connects self-efficacy to actual skills or improved performance without testing the skills. for hypothesis three, the authors assume that higher self-efficacy students have superior library skills because they selected library resources more often in the survey question about their go-to resources. while this choice is intriguing, a self-selected answer about preferred resources does not measure library skills, rather awareness of library resources.   furthermore, the authors perhaps overstate the link to research that finds that individuals with higher self-efficacy learn more easily. a statement from kurbanoglu (2003) that high self-efficacy results in self-regulated learning – quoted once in their article and later repeated in conjunction with their own study results (p. 519, 520) –– leads the authors to conclude that improving self-efficacy leads to better learning outcomes, but this is not demonstrated by the study.   gross & latham’s recent findings (2012) from the attaining information literacy project are bolstered by psychology research that showed that lesser skilled people tend to overestimate their abilities more than more highly skilled individuals, because the former lack the metacognitive competence to effectively evaluate their own skill level (ehrlinger, johnson, banner, dunning, & kruger, 2008; kruger & dunning, 1999). librarians should look more deeply at the connection between self-efficacy, motivation to learn, and effective learning, to design online instruction that motivates both low and high self-efficacy students.     references   ehrlinger, j., johnson, k., banner, m., dunning, d., & kruger, j. (2008). why the unskilled are unaware: further explorations of (absent) self-insight among the incompetent. organizational behavior and human decision processes, 105(1), 98–121. doi:10.1016/j.obhdp.2007.05.002   gross, m., & latham, d. (2012). what’s skill got to do with it?: information literacy skills and self-views of ability among first-year college students. journal of the american society for information science and technology, 63(3), 574–583. doi:10.1002/asi.21681   kruger, j., & dunning, d. (1999). unskilled and unaware of it: how difficulties in recognizing one’s own incompetence lead to inflated self-assessments. journal of personality and social psychology, 77(6), 1121–1134. doi:10.1037/0022-3514.77.6.1121   kurbanoglu, s. s. (2003). self-efficacy: a concept closely linked to information literacy and lifelong learning. journal of documentation, 59(6), 635-646.   ren, w.-h. (2000). library instruction and college student self-efficacy in electronic information searching. the journal of academic librarianship, 26(5), 323–328. doi:10.1016/s0099-1333(00)00138-5 evidence summary   print books are cheaper than e-books for academic libraries   a review of: bailey, t. p., scott, a. l., & best, r. d. (2015). cost differentials between e-books and print in academic libraries. college & research libraries, 76(1), 6-18. doi: 10.5860/crl.76.1.2   reviewed by: laura newton miller assessment librarian carleton university ottawa, ontario, canada email: laura.newtonmiller@carleton.ca   received: 29 may 2015  accepted: 23 jul. 2015      2015 newton miller. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine the difference in cost (if any) between print and e-book titles for an academic library.   design – case study.   setting – library system of a small, regional university in the southern united states of america.   subjects – 264 titles requested by faculty (out of 462 total requests) that were available in both print and electronic format.   method – using baker & taylor’s title source 3 (now title source 360), the researchers compared pricing between the print version (paperback preferred) and electronic version (single user only) of titles requested by faculty during the fall 2012 semester.   main results – as a whole, print titles had a mean price of $53.50 and electronic equivalent titles had a mean price of $73.50 (a $19.17 difference). only 44 of the 264 e-book titles were less expensive than their print equivalents. when broken down by lc classification, e-books were generally more expensive than print across all subjects except for religion and philosophy (bj-by) and the social sciences (h-hv). average prices for both print and electronic were cheaper for university press publications versus non-university press publications. (this was true for both arithmetic and weighted means.) humanities books were the least expensive (mean cost/print title), but the average e-book cost was slightly higher than the social sciences. science books were most expensive (average) both in print and electronic.   conclusion – on average, print books are cheaper than e-books for academic libraries.   commentary   many e-book articles explore usage and attitudes, but – with the exception of those that discuss demand-driven acquisition models (downey, 2014) – only mention actual costs of e-books in passing. influenced by gray and copeland (2012) who looked at cost differentials between print and electronic books in public libraries, this article looks specifically at price differences in academic libraries.   the glynn’s critical appraisal checklist (2006) was used to determine that a case study was an appropriate tool for this research. the authors are commended for the clear, easy-to-follow way this research design can be repeated in other university library contexts.   the authors gave a very robust literature review for anyone interested in learning more about general e-book issues, libraries’ transition to electronic format for book collections, pricing models, demand-driven acquisition, and price comparisons. criticisms of this paper are minor. the library used a paperback-preferred acquisition model for their print books, which are often cheaper than hardcover. results might be slightly different if comparing to mostly hardcover instead of paperback copy, although it is not known how many print books actually are hardcover versus paperback.   in analyzing the results of the study, the authors discussed the physical processing and storage costs for print versus ongoing maintenance costs for continued access to e-versions. in the literature review they mentioned staffing costs for print, but there was no mention of staffing costs in maintaining e-books. staffing time for maintaining electronic resources (including e-books) is very significant (samples & healy, 2014), and this is certainly an area for further future study.   the only real criticism about this article has nothing to do with the article itself, but with the time lapse between the study taking place and the publication of the article. e-book publishing has always been and continues to be in a state of flux. although a pre-print has luckily been available since 2013, the “official” record of the publication is 2015; the data are already 3 years old. those researching similar “in flux” topics should be attentive to time-sensitivity when writing for publication.   with decreasing budgets, an increasing need for study space, and varying needs of users, this research is useful in helping academic libraries to make sometimes difficult acquisition decisions. academic libraries could easily be inspired by this research to conduct their own smaller-scale study in order to investigate possible local differences.   references   downey, k., zhang, y., urbano, c., & klinger, t. (2014). a comparative study of print book and dda ebook acquisition and use. technical services quarterly, 31(2), 139-160. doi: 10.1080/07317131.2014.875379   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi: 10.1108/07378830610692154   gray, d. j. & copeland, a. j. (2012). e-book versus print: a per-title cost and use comparison of a public library’s popular titles. references & user services quarterly, 51(4), 334-39. http://dx.doi.org/10.5860/rusq.51n4.334   samples, j. & healy, c. (2014). making it look easy: maintaining the magic of access. serials review, 40(2), 105-117. doi: 10.1080/00987913.2014.929483     evidence summary   library users attempt to access a wide range of information beyond books and articles through a single search box   a review of: lown, c., sierra, t., & boyer, j. (2013). how users search the library from a single search box. college & research libraries, 74(3), 227-241. http://dx.doi.org/10.5860/crl-321   reviewed by: sara sharun assistant professor mount royal university library calgary, alberta, canada email: ssharun@mtroyal.ca   received: 12 june 2014  accepted: 4 nov. 2014      2014 sharun. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to identify how users use a single search box.   design – transaction log analysis.   setting – a large research university in north carolina, united states of america.   subjects – search results from a customized single search box on the library’s home page, consisting of 739,180 searches and 655,388 hyperlink selections.   methods – the authors used custom logging software to generate transaction logs of all searches that took place over two semesters (august to december 2010 and january to may 2011) in quicksearch, a custom-made, combined search application. the study tracked search queries and hyperlink selections, noting which modules in the discovery interface (articles, catalogue, databases, and others) were selected and, from these modules, which hyperlinks were clicked.   main results – transaction log analysis was conducted on over 739,000 searches during the two semesters and over 655,000 hyperlink selections from the results interface. the 20 most frequent queries made in quicksearch were primarily for specific resources (database titles or journal titles), administrative information, and library services. the 153 most popular searches made up 10% of all searches. hyperlinks to full-text articles (41.5%) and the catalogue (35.2%) accounted for about 76% of the links selected. about 23% of links selected were for other modules (e.g., faqs, “best bets,” and journal titles). hyperlinks that led directly to specific items were selected more frequently than hyperlinks to a full list of results.   conclusion – analysis of user transaction logs suggests that users do not understand what is being searched in a combined search box and that search applications need to direct users more effectively to resources beyond the catalogue and article databases. users attempt to access a wide range of information from a single search box, and the most commonly used modules in quicksearch do not serve many of the most frequent queries. many of the most common queries can be defined and addressed with a predefined list of results, improving the quality of results and the search experience for users. ongoing evaluation and analysis of the search interface and subsequent optimization for the most frequent queries can improve user experience.     commentary   this article is one of a number of recent articles on combined search applications and web-scale discovery services. whereas most of these recent articles discuss administrative concerns related to the acquisition and implementation of discovery services or the functionality and design of specific tools, this article attempts to examine user behaviour in relation to one of these tools. this article is relevant to libraries seeking evidence of how users actually engage with a single search box on a library website.   this article builds on an existing body of literature on user behaviour in the context of library catalogues and web search engines. it employs a methodology whose strengths and weaknesses have been thoroughly examined (jansen, 2006) and that has been demonstrated to be valid in these other contexts. in addition, this study meets the criteria for validity established by the ebl critical appraisal checklist (glynn, 2006). by means of a comprehensive literature review, the authors situate their study firmly within this body of research and make a solid argument for the significance of their study and their findings.   transaction log analysis (tla) is a useful methodology for studies of user behaviour with a defined information system, and it provides clear evidence of interactions that can be used to identify some aspects of user behaviour and improve the design and function of information retrieval systems (jansen, 2006, p. 409). further research on user behaviour with single search applications could effectively employ tla in combination with other quantitative and qualitative data gathering techniques (e.g., surveys, questionnaires, interviews, and focus groups) to address some of the limitations of this methodology, provide a more complete picture of the users behind the behaviours, and inform the design of discovery platforms.   overall, this study is well designed and is particularly strengthened by the large sample size obtained by the authors. it features detailed discussion of specific actions taken by users, changes implemented to address problems, and the impact of these changes on the use of the search application. this study provides evidence of the impact and importance of continuous gathering of data, regular adjustments and revisions, and occasional substantive changes to the software to direct users to the most appropriate resources. it also indicates that data from tla may uncover “latent demand for resources” that are not found in a single search   box, which is information that could be applied not only to improving system design but also to collection development and even user education (p. 240).   the authors make several observations about user behaviour that can inform the development of discovery services and single search boxes in other libraries. the utility of this study for other libraries is somewhat limited because it describes custom-made discovery software that few libraries have the capacity to create and uses custom analysis software to gather specific data from transaction logs. libraries using vendor-supplied commercial products may be reliant on the vendor to share this type of data or conduct this type of research on their behalf. the ability to make significant changes to the search interface based on evidence of user behaviour may also be beyond many libraries’ capabilities. nonetheless, this study provides solid evidence to inform the user-centred design of library websites and the employment of discovery services to provide a single point of access to a variety of library resources via multiple platforms.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154   jansen, b. j. (2006). search log analysis: what it is, what's been done, how to do it. library & information science research, 28(3), 407-432. http://dx.doi.org/10.1016/j.lisr.2006.06.005 microsoft word news_ifla.doc evidence based library and information practice 2007, 2:4  114 evidence based library and information practice     news    ifla health and biosciences libraries section: call for papers for satellite session         lindsay glynn  editor‐in‐chief    public services librarian and instruction coordinator, health sciences library  memorial university of newfoundland  st. john’s, newfoundland, canada  e‐mail: lglynn@mun.ca      © 2007 glynn. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    ʺrole of evidence‐based research in  medical librariesʺ     the one‐day session will have two sub‐ themes:     1. library efforts in support of evidence‐ based research.   2. research conducted by health science  libraries and librarians.     the audience of this session is likely to  include health care professionals, educators,  researchers and librarians.     you are invited to submit an abstract for one  of the two themes. it is hoped that papers  will cover a wide range of areas, for  example:     • how libraries train library staff and  users in using evidence.  • partnerships and collaborations that  support evidence‐based research.  • research that has made a difference to  library services.  • the librarian’s role in critical appraisal  of the evidence.   • clinical librarians ‐ walking the talk.   • tools to support evidence‐based  medicine.  • research methods.   • promoting the evidence.     presentations are suggested to be no more  than 30 minutes, including 10 minutes for  questions.     submission guidelines     the proposals must be submitted in an  electronic format and must contain:   mailto:lglynn@mun.ca http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:4  115   • title of paper;   • summary of paper (250 ‐ 350 words  maximum);   • speakerʹs name, address, telephone and  fax numbers, professional affiliation,  email address, and biographical note (40  words maximum).     submissions must be received no later than  december 24th, 2007, preferably by email to:     heather todd   university of queensland library   st lucia campus   st lucia, queensland 4072   e‐mail: h.todd@library.uq.edu.au     important dates     satellite session (preceding the ifla  conference):   august 10th‐14th, 2008  quebec city, quebec, canada     deadline for submission of abstracts:  december 24th, 2007:             notification of acceptance/rejection:  january 25th, 2008     deadline for submission of text:   may, 2008    information for speakers:    regrettably, iflaʹs sections do not have  funds available to pay for speakers’  expenses (e.g. registration fees, travel  expenses, or accommodation costs).  however, there may be limited funding  available through other ifla channels,  especially for people from developing  countries.     for further information:   heather todd   executive manager, engineering and  sciences library service  university of queensland library, st lucia  campus   st lucia qld 4072 australia   phone: +61 7 334 64394   e‐mail: h.todd@library.uq.edu.au                     mailto:todd@library.uq.edu.au mailto:todd@library.uq.edu.au conference paper   value of libraries: relationships between provision, usage, and research outcomes   michael jubb director research information network london, england, united kingdom email: michael.jubb@researchinfonet.org   ian rowlands research services manager and bibliometrician university of leicester leicester, england, united kingdom email: ir46@le.ac.uk   david nicholas director ciber research ltd newbury, england, united kingdom email: dave.nicholas@ciber-research.eu      2013 jubb, rowlands, and nicholas. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to explore the relationships in the united kingdom between library expenditures, levels of usage, and research outcomes, focusing on the provision and use of e-journals.   methods – the project used a mixture of top-down and bottom-up approaches. it involved a close study of the behaviors of researchers in eight universities and two research institutes across a range of six subject areas, along with a parallel gathering and analysis of data for all u.k. universities and colleges, covering various library indicators together with data on article downloads and a range of measures of research performance. the work was undertaken in two stages and was completed in 2010. the first stage involved detailed mining of the publishers’ logs from elsevier’s science direct and from oxford journals to generate fine-grained insights into the information-seeking behavior of scholars from the case study institutions, together with an initial analysis of the u.k.-wide data. the second stage involved a survey and interviews with a wide range of researchers as well as librarians from the case study institutions, together with further analysis of the u.k.-wide data.   results – strong variations were found between users, not only in different disciplines but also in different institutions. some, but not all, of the variations seemed to be related to the size and research intensity of the institution. analyses of the u.k.-wide data show that levels of library expenditure influence subsequent levels of use of e-journals. while the modeling does not show strong direct linkages in either direction between library expenditure and research performance, it does show a strong positive feedback loop between the use of e-journals and research performance.   conclusion – there is a need both to broaden the focus beyond e-journals and for more detailed work to test hypotheses and understand the dynamics of the relationships between different variables over time.   introduction   expenditure in university and college libraries in the uk amounted in 2008 to £630 million (society of college, national and university libraries, 2009), a figure that had grown by 30% in real terms over the previous ten years. growth in expenditure was even faster – at nearly 48% – for the libraries of the research-intensive universities represented by research libraries uk (rluk). numbers of staff and students also grew, however, and income and expenditure on research grew even faster. hence the proportion of total university expenditure that went to support libraries fell: from 3.4% to 2.8% across all uk universities; and from 3.2% to 2.6% across the rluk libraries. so libraries represent a declining share of university budgets, and they will have to fight hard to avoid further falls in that share as universities face significant cuts in the income they receive from public funds.   in that context, it is particularly important that libraries should be able to show not only that they are operating efficiently, but that they provide services with demonstrable links to success in achieving institutional goals. return on investment is thus an increasingly important issue. in order to address these issues, libraries need to do more to understand user behaviour and workflows; and to rigorously analyse and demonstrate the value of what they do in terms of improving students’ experience, and supporting teaching, learning, and research.   there has been a tendency, in the uk at least, for performance indicators to focus on inputs and outputs that are relatively straightforward to measure, as distinct from the much harder issues relating to impact and value. in current circumstances, however, it is important that more is done to analyse the relationships between library activities on the one hand, and learning and research outcomes on the other.   work of this kind is in its relatively early stages, and it is fraught with difficulties. gathering and analysing evidence of value is notoriously difficult; a number of different approaches have been adopted, and there is no single answer. a key question is “value for whom?” in relation to libraries, approaches to gathering evidence of value for students or academic staff may well differ from approaches to value for funders or for universities. similarly, approaches to the value of existing services may not be appropriate in gathering evidence of possible changes (positive or negative) either to the nature or to the level of those services. and there are notorious difficulties in assessing changes in value over time.   this paper focuses on one element in that set of issues: the provision of information content, particularly e-journals, that libraries make from within their budgets, and the use that is made of that content. it reports in particular on the findings of a study commissioned and overseen by the research information network (rin), and undertaken by the centre for information behaviour and the evaluation of research (ciber) at university college london. the study was undertaken in two stages, and was completed in 2010.   project design and methodology   the study started with the aim of providing a detailed portrait of the information-seeking behaviour of uk researchers, of how they make use of e-journals and of the benefits that flow from that use. more detailed objectives were to ·         investigate researchers’ behaviour, in terms of levels and patterns of usage, content viewed, navigational preferences, and routes used to access e-journal content ·         ascertain how researchers’ behaviours vary by subjects and disciplines, and in relation to the universities and other institutions in which they work ·         gather and analyse any evidence of relationships between researchers’ behaviours and usage, and institutional expenditure on e-journals, and ·         gather and analyse any evidence of relationships between researchers’ behaviours on the one hand and research productivity, outputs and outcomes on the other, including such measures as numbers of publications produced, citations attracted, and the results of research evaluations.   the project used a mixture of top-down and bottom-up approaches. it thus involved a close study of the behaviours of researchers in eight universities and two research institutes across a range of six subject areas; and a parallel gathering and analysis of data for all uk universities and colleges, covering various library indicators together with data on article downloads and a range of measures of research performance. the work was undertaken in two stages. the first stage involved detailed mining of the publishers’ logs from elsevier’s science direct and from oxford journals to generate fine-grained insights into the information-seeking behaviour of scholars from the case study institutions, together with an initial analysis of the u.k.-wide data (research information network, 2009) the second stage involved a survey and interviews with a wide range of researchers as well as librarians from the case study institutions, together with further analysis of the u.k.-wide data (research information network, 2011).   expenditure and usage of e-journals   expenditure on information content of all kinds represents about 35% of all library expenditure across the u.k. university library sector (figure 1), and that proportion has been relatively stable over the past decade. but there are significant differences between individual libraries – proportions vary between under 30% and over 40% – and groups of libraries. the proportion tends to be lowest in small colleges and specialist institutions, and highest in the older universities.   the relatively stable proportion of expenditure on content implies, of course, increases in actual expenditure in real terms. but here experiences differ across the sector. in the research-intensive universities expenditure rose by 52%; but in the newer universities, after rising by 5% in the years up to 2002, expenditure on content has actually declined in real terms since then, and in 2008 was actually 2% lower than it was in 1998.   figure 1 expenditure on information provision as a percentage of overall library expenditure 1998-2008   the lion’s share of that expenditure goes to serials, which now account for nearly 20% of total library expenditure across the u.k. higher education (he) sector. that marks a significant change over the past ten years. in 1998 books accounted for just over 12% of library expenditure, and serials just over 15%; but by 2008 the percentages had diverged rapidly, to 9% and 19% respectively (figure 2). in several older universities, serials account for over a quarter of the total library budget.   growth in expenditure on serials has of course been accompanied, as a result of the adoption of big deals, by a huge increase in the number of titles available (figure 3). overall, the number of titles has increased by over 153% across all uk university libraries between 1998 and 2008. within this, there is considerable variation, both in the rate of change and in the overall number of titles available. rluk members, while showing one of the smaller overall increases at 56%, has a consistently larger number of titles available than any other group. other he colleges, also showing a lower rate of change at 39%, have noticeably fewer titles available than preand post-92 universities. nonetheless, the overall story is one of rapid and significant change.   and the increase in provision has been accompanied by huge increases in usage. our estimates of the number of downloads of e-journal articles as reported by libraries in accordance with the counter protocols are shown in table 1. they show an increase of over two and a half times across the sector as a whole between 2004 and 2008, with even higher rates of growth among the research-intensive russell group of universities.   one simple approach to value is to ascertain the unit cost per download and its variation between different institutions or over time. as shown in table 2, since the rise in usage has been faster than the rise in expenditure on serials, the cost per download fell sharply between 2004 and 2008: from £1.19 to £0.70 (thus by 41%) across the sector as a whole, with an even sharper fall of 62% among the research-intensive russell group universities.   figure 2 serials expenditure as a percentage of overall library expenditure 1998-2008   figure 3 number of serial titles per institution 1998-2008   when levels of usage are put alongside expenditure on e-journals in individual universities across the uk, again the results are intriguing. they show a very strong correlation between volumes of downloads and expenditure, with only a few outliers; as shown in figure 4, a matrix scatterplot based on downloads for all downloads (counter (adjusted), elsevier sciencedirect, and oxford journals (fitted linear regression trendlines with 95% confidence intervals). only the plots for oxford journals show a wide scatter, reflecting the relatively small number of journal titles involved, and their concentration in a relatively small range of subject areas. overall, however, our findings seem to indicate that universities as a whole are spending their money wisely.   variations between subjects and institutions   it is well known that there are significant variations between the usage behaviours of researchers in different disciplines, as well as in the provision of information resources and services directed towards them. this is borne out by the detailed analysis of the usage logs for science direct and oxford journals in our case study subjects and institutions. table 3 shows that economists differ from both life scientists and physical scientists in the degree of concentration on a small number of titles, in the numbers of pages viewed per session, in their use of abstracts, and in their use of external gateways such as google or google scholar to get to content.   table 1 annual counter downloads (ciber estimates based on sconul) mean for sector (huber’s m-estimator) year 2004 2005 2006 2007 2008 russell group 783,870 1,377,603 1,846,121 2,211,245 2,795,825 pre-1992 institutions 439,813 632,144 655,926 819,335 1,001,521 post-1992 institutions 283,760 332,251 443,027 521,350 592,253 total 432,693 632,758 772,600 930,415 1,134,165                     index 2004=100 year 2004 2005 2006 2007 2008 russell group 100 175.7 235.5 282.1 356.7 pre-1992 institutions 100 143.7 151.4 186.3 227.7 post-1992 institutions 100 117.1 156.1 183.7 208.7 total 100 146.2 178.6 215.0 262.1   table 2 direct cost per download at constant prices (sconul/counter/ciber estimates) mean for sector                                                           (huber’s m-estimator) index 2004=100 year 2004 2005 2006 2007 2008 2004 2005 2006 2007 2008 russell group £1.73 £0.99 £0.82 £0.74 £0.66 100 57.2 47.4 42.8 38.2 pre-1992 institutions £1.20 £0.96 £0.98 £0.91 £0.81 100 80.0 81.7 75.8 67.5 post-1992 institutions £1.01 £0.85 £0.73 £0.68 £0.65 100 84.2 72.3 67.3 64.4 total £1.19 £0.91 £0.83 £0.77 £0.70 100 76.5 69.7 64.7 58.8   but there are significant variations also between different areas of the sciences. in physics and chemistry, for example, there are big differences in the degree of concentration on specific journal titles. the total number of titles viewed was broadly similar in the two disciplines; but the most popular 5% of titles accounted for 39.5% of use in chemistry, as compared with 26.6% in physics.   figure 4 uk higher education libraries expenditure and usage of e-journals   table 3 information-seeking behaviour readers in different subjects behave differently   journal titles viewed most popular 5% of journals accounted for % use page views (average per session) abstract views (% sessions) gateways (% page views arriving via gateways) chemistry 196 39.5 3.2 23.3 49.2 environmental sciences 248 29.6 3.6 22.7 41.4 economics 132 46.9 3.8 30.4 19 life sciences 531 38.1 2.0 19.5 65.9 physics 204 26.6 2.5 20.1 57.8   there are similar variations as to the average number of page views per session. it is not obvious, for example, why environmental scientists should view nearly twice as many pages during a session as life scientists do, though it may be related to the latter’s much higher usage of external gateways, including services such as pubmed. there is more consistency with regard to the use of abstracts: only economists stand out as using them much more than scientists do.   perhaps more intriguing are the variations between users in the same discipline at different institutions. our analysis shows, for example, significant variations in intensity of usage at our case study institutions. the following two charts compare usage (in this case numbers of page views in the subject area concerned as shown in the science direct logs) with the size of the institution in two subject areas. the measure of size is the number of staff submitted to the 2008 research assessment exercise (rae), the exercise that has been undertaken roughly every five years in the uk since 1986, to assess the quality of research in each university in the uk. the number of staff submitted to the rae provides only a rough indication of size, since it does not take account of numbers of research students or of staff (such as research assistants) who were ineligible or who were not chosen for submission by their institution. nevertheless, it provides a reasonable indication of the weight of research effort in each institution. in each of the graphs, the data are indexed to the institution with the largest number of research-active staff in the subject area.   what is intriguing here is that intensity of use does not appear to be closely correlated with size or with the quality of the research that is undertaken at the universities concerned. in physics, for example, the quality ratings achieved in the 2008 rae by cambridge, edinburgh, manchester, and university college london were fairly similar. the striking variation – by a factor of four – in the ratios between levels of use and of size at edinburgh and ucl on the one hand, and manchester on the other, is not explained simply by either the volume or the quality of the research being produced at those institutions. there are similar variations, by as much as a factor of six, in the age of the articles that are viewed in different subjects and institutions; and again these do not seem to be related to levels or quality of research performance at individual institutions.   variations in the titles viewed at different institutions seem to show, however, a more understandable pattern. table 4 shows the average impact factor of the journals viewed at the case study institutions. since impact factors vary considerably between disciplines, we have sought to normalise for the range of disciplines at each institution, by calculating a “relative impact” factor, which matches each journal viewed against the average for that discipline. thus a value of 1 means that the journals viewed at that institution are typical—in terms of their citation impact—of the journals for that range of disciplines as a whole, worldwide. a value greater than 1 means that users at that institution are viewing articles in journals with an impact factor higher than the average in that range of disciplines. what is notable here is that users at the most research-intensive universities (cambridge, edinburgh, manchester and ucl) are using journals that are more heavily cited than the global average in their disciplines. users at other institutions, including the two government-funded research institutes (the centre for ecology and hydrology and the rothamsted agricultural research institute) tend to use journals where the impact factor clusters around the average.   figure 5 usage comparison – size of institution in two subject areas   such variations may well be related to differences in how users in different institutions get to content. users at the more research-intensive universities tend to make more use of gateways such as google scholar and pubmed, and then to spend less time on a journal site than their colleagues in other institutions. figure 6 shows the average session length in science direct for users at each of our case study institutions in the life sciences, mapped against the research rating of the authors at each institution as measured by the hirsch index. the percentages indicate how many science direct sessions originated from an external gateway service, and the diameter of the circles is scaled to that value.   we can also derive similar patterns when we look at usage of navigation facilities within the science direct platform, with users at the less-research-intensive institutions making much more use of menus and search facilities, especially citation search. they also make more use of value-added services such as alerts, and articles in press.   the conclusions from this part of our work are that there are strong variations between users not only in different disciplines but also in different institutions, and that some – but not all – of the variations seem to be related to the size and research-intensity of the institution. such variations also raise questions, of course, about the utility and value of some of the services provided by libraries and publishers, particularly when services such as advanced search are used only infrequently. one conclusion from our findings is thus the familiar one that one size does not fit all. it is already well understood that researchers in different disciplines behave differently and have different needs. what has perhaps been less well covered in the literature has been the differences in behaviours, and presumably needs, between users in different institutions.   relationships between usage and value   we have already noted that there are close relationships between expenditure on and usage of e-journals; and those relationships remain strong even when we control statistically for institutional size. trying to assess the impact or value of usage is more difficult. for the linkages between use of information resources provided by libraries on the one hand, and research or learning outcomes on the other are difficult to pin down, and chains of reasoning may raise as many questions as they seek to answer. one approach is to try to calculate the return on investment (roi) for individual libraries. recent studies (tenopir, king, mays, wu, and baer, 2010) suggest that the roi varies from between 15.5: 1 to under 0.64:1 (i.e., a negative return), depending on such factors as the balance between teaching and research, and the subject mix, at each university.   table 4 average impact factor of the journals viewed at the case study institutions case study average impact factor of journals viewed relative impact aberdeen 3.0 1.2 bangor 2.3 0.9 cambridge 5.0 2.0 centre for ecology and hydrology 2.6 1.0 edinburgh 3.7 1.5 manchester 3.9 1.6 rothamsted 2.6 1.0 strathclyde 2.7 1.1 swansea 2.5 1.0 ucl 4.1 1.7     figure 6 session length and gateways   we have taken a rather different approach, seeking to investigate the relationships between levels of usage on the one hand, and a range of measures of research activity on the other. we first of all identified from our analysis of the data across the uk sector three groups of universities in terms of the volume of downloads: moderate, high and super users. in table 5, we match these groups with various measures of research activity as well as a calculation of cost per download.   these figures suggest that there might be a relationship between e-journal usage and research performance: the differences in performance between the groups are statistically significant, although differences in cost per download are much less so.   we then moved to a more detailed mapping of article downloads in individual universities plotted against similar measures of research performance, as shown, for example, in figure 7.   table 5 usage groups and research outcome measures   moderate users (n=80) high users (n=25) super users (n=10) research papers per academic 0.4 0.8 1.0 research grants and contracts per academic (£000’s) 12.7 29.0 39.7 phd awards per 100 academics 9.1 17.5 17.4 cost per download £0.89 £0.74 £0.60     figure 7 use and outcomes publications   it is clear that the fit is very close, with only a few outliers. nevertheless, correlations do not necessarily imply causal relationships; and still less do they provide a clear indication of the direction in which cause and effect might run. we have therefore adapted a technique using partial least squares regression and path modelling, a predictive technique that is particularly useful when predictor variables are highly correlated. we have thus built a model that seeks to predict levels of three variables – expenditure, usage and research outcomes – on the basis of the other two; and to quantify how good they are as predictors of each other. expenditure is represented by the total spending on journals; usage by downloads as reported in accordance with the counter protocols; and research outcomes by numbers of ph.d. awards, income from research grants and contracts, numbers of articles published and field-normalised citation impact. we used data from 113 u.k. universities for the two years 2004 and 2007, so that the models could include a two-year time lag, and we could test whether 2004 independent variables predict 2007 dependents.   we used the model to test six hypotheses:   levels of library expenditure influence subsequent levels of use of e-journals levels of e-journal use influence subsequent levels of library expenditure levels of library expenditure influence subsequent research performance successful research performance influences subsequent levels of library expenditure levels of e-journal use influence subsequent research performance successful research performance influences subsequent levels of use of e-journals   the criteria used for determining whether or not a hypothesis is supported were a path co-efficient equal to or greater than 0.3, and a t-statistic equal to or greater than 2.02 (the threshold for significance at the 5% level).   the results of the analysis are summarised in table 6 and figure 8. results show that the first hypothesis is supported: expenditure drives use; indeed it is a precondition for it. the reverse hypothesis, that use drives subsequent expenditure, is not supported, probably because the relationship is complicated by the big deals, and journals are not priced according to usage.   the modelling does not show strong direct linkages in either direction between library expenditure and research performance. the two variables here are of course conceptually distant from each other. any relationships between them may therefore tend to be indirect rather than direct; and any direct relationship may involve a time lag longer than two years.   the modelling does, however, show a strong positive feedback loop between the use of e-journals and research performance. indeed, the model shows that use is a powerful predictor of subsequent research success, and this linkage is by far the strongest in the model.   table 6 testing of hypotheses on journal expenditure, usage, and research outcomes hypothesis path coefficient t-statistic p 1. investment drives use 0.492 2.94 <0.05 2. use drives research success 0.846 6.46 <0.01 3.expenditure drives research success 0.125 0.91 <0.40 4. use drives expenditure 0.256 0.49 <0.40 5. research success drives use 0.479 3.01 <0.05 6. research success drives expenditure 0.416 0.80 <0.40     figure 8 relationships between levels of expenditure, usage of e-journals, and research outcomes   these findings focus on e-journals, and they are suggestive rather than conclusive. there is a need both to broaden the focus beyond e-journals and for more detailed work to test hypotheses and understand the dynamics of the relationships between different variables over time. it is important that such work should be continued so that we help libraries to show not only how effectively (or not) they are operating, but the extent to which they are providing services with demonstrable links to success in achieving institutional goals. in difficult economic times, we need a deeper understanding of user behaviour and workflows; and rigorous analyses of the value of library and information services and activities in improving students’ experience and in supporting teaching, learning and research. there is a need to go beyond performance indicators that focus on inputs and outputs, and to address the much harder issues relating to impact and value. that implies detailed investigations of the relationships between library activities on the one hand, and learning and research outcomes on the other. in current circumstances, senior managers in many universities will be seeking such evidence if they are asked to sustain current levels of expenditure to support library and information services.     references   research information network. (2009). e-journals: their use, value and impact. london, uk:rin. retrieved 18 may 2013 from http://www.rin.ac.uk/system/files/attachments/e-journals-report.pdf‎   research information network. (2011). e-journals: their use, value and impact: final report. london, uk: rin. retrieved 18 may 2013 from http://www.rin.ac.uk/our-work/communicating-and-disseminating-research/e-journals-their-use-value-and-impact   society of college, national and university libraries. (2009). annual library statistics. london, uk: sconul.   tenopir, c., king d. w., mays r., wu l., & baer a. (2010). measuring value and return on investment of academic libraries. serials: the journal for the serials community, 23(3), 182-190. retrieved 18 may 2013 from http://uksg.metapress.com/content/k477161477t9573g/fulltext.pdf evidence based library and information practice editorial responsibilities      2015. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   editor-in-chief: lorie kloda   associate editor (evidence summaries): heather pretty   associate editor (articles): rebekah (becky) willson; lisl zach; with additional assistance from denise koufogiannakis   associate editor (classics): jonathan eldredge   associate editor (reviews): denise koufogiannakis   associate editor (features, commentaries, using evidence in practice, ebl 101, news): lorie kloda   guest editor (feature): martha kyrillidou   production editor: michelle dunaway   editorial intern: melissa griffiths   editorial advisor: alison brettle   copyeditors: richard hayman (lead copyeditor), georgianne bordner, julie evener, linda ferguson, heather healy, stacey penney, maria tan, mary virginia taylor, alison yeoman   indexing support: meral alakus   evidence summary   use of library services can be associated with a positive effect on first-year students’ gpa and retention   a review of: soria, k. m., fransen, j., & nackerud, s. (2014). stacks, serials, search engines, and students’ success: first-year undergraduate students’ library use, academic achievement, and retention. the journal of academic librarianship, 40(1), 84-91.http://dx.doi.org/10.1016/j.acalib.2013.12.002   reviewed by: eamon c. tewell reference & instruction librarian long island university brooklyn, new york, united states of america email: eamon.tewell@liu.edu   received: 14 sep. 2014   accepted: 26 jan. 2015      2015 tewell. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to investigate the degree of relation between first-year undergraduate students’ library use and their academic achievement (measured by cumulative gpa) and firstto second-year retention.   design – quantitative data obtained from library systems combined with regression analyses.   setting – a large public university located in the united states of america.   subjects – the study included 5,368 non-transfer first-year students, with a total of 5,162 students retained for the final sample.   methods – data on 10 library usage variables were collected using student logins to library databases and websites and analyzed using spss. these variables included logins to databases, use of electronic books and journals, chat reference questions, and workshops signed up for, among others. there were 2 separate regressions utilized to predict students’ cumulative gpa by these 10 types of library use. two separate logistic regressions were utilized to predict firstto second-year retention by the same library usage variables.   main results – 81.9% of first-year students used at least one library service. overall, students who used their academic library’s services and/or resources once or more during an academic year had a higher average retention rate and gpa compared to their peers who had not used the library. it was found that four library use areas, including book loans, database logins, electronic journal logins, and library workstation logins, were positively associated with students’ gpa. database logins and library workstation logins were positively associated with retention. each of the models used to predict either student gpa or retention by library use were found to be statistically significant.   conclusions – the study suggests that there is a positive and significant relationship between a number of library activities and students’ gpa and retention. the effect size of these activities upon the primary outcome variables of gpa and retention is small, though this is logical considering the one-time use of a library service is unlikely to meaningfully influence one’s academic success. other non-library factors in the student experience must be considered.   commentary   academic libraries are under increasing pressure to demonstrate their contributions to achieving institutional goals and student success. student success can be conceived in a number of ways. most germane to this evidence summary, two previous studies tying the library to student success examine the variables of staffing and funding. emmons and wilkinson (2011) found that the ratio of professional library staff to students had a positive relationship with retention, while teske, cahoy, and dicarlo (2013) found that different types of expenditures at multiple institutions could be associated with retention and graduation. the study at hand is guided by the following question: is library use, including the one-time use of specific services, associated with first-year students’ gpa and retention when multiple characteristics are controlled for?   among the work’s many strengths, the authors include a number of relevant studies that contextualize the research questions. the population is clearly defined and of a sufficient size for the research’s intent. by controlling for multiple variables the authors reduce the possibility of introducing omitted variable bias, a shortcoming common in research investigating the library’s contributions to student success. the many steps of data analysis are described in detail, thus increasing the possibility of replication. finally, the results and their implications are fully discussed, with numerous recommendations for researchers and practitioners interested in demonstrating the fiscal value of their department.   some factors in the study negatively impact the strength of the evidence presented. many of these factors are recognized by the authors in the limitations and recommendations section, including the limit that institution size has on generalizability and the fact that some library activities could not be measured because they were not associated with a user id. one noteworthy factor acknowledged is that of causality, which is that correlation does not imply causation. likely due to the scope of variables involved, the description of the data collection process lacks sufficient detail to allow for full replication. the assumption made that gpa is equivalent to academic achievement also requires additional consideration.   the study does not include a description of the ethics approval process and whether the authors received irb approval or notified students that their data was being procured for the purposes of the research. the authors identify their work as one that does not infringe on privacy due to its use of aggregated data, and encourage other libraries to begin collecting data on student use of the library. however, it is not clear what steps were taken to protect patron privacy, a concern not to be quickly dismissed at a time when many companies and organizations are collecting increasingly granular private data without user awareness.   the primary significance of this research lies in its attempt to discern the library’s impact upon two variables often identified as key to institutional effectiveness: academic achievement and retention. implications of this study include the possibilities of replicating the data analysis process, which can be used to inform the provision of the services measured. future work in this area, as recommended by the authors, should account for the use of the library as a social and physical space. while it is no doubt challenging to collect these types of data, they will allow library researchers and professionals alike to better represent the myriad functions of the library and depict students as the complex, multifaceted learners that they are.   references   emmons, m., & wilkinson, f. c. (2011). the academic library impact on student persistence. college and research libraries, 72(2), 128–149. retrieved from http://crl.acrl.org/content/72/2/128   teske, b., cahoy, d., & dicarlo, m. (2013). libraries and student persistence in southern colleges and universities. reference services review, 41(2), 266-279. http://dx.doi.org/10.1108/00907321311326174   news/announcements registration now open for eblig mini-conference during cla 2010 registration is now open for the cla 2010 conference and trade show to be held at edmonton's shaw conference centre, june 2-5. please plan to attend the evidence-based librarianship interest group's (eblig) preconference session on wednesday, june 2nd, from 1-5 p.m. at the stanley a. milner branch, edmonton public library.   becoming evidence based: a research in practice mini-conference this half day conference focuses on evidence based practice and its incorporation into professional decision making. with featured keynote speaker lisa m. given (associate professor, school of library and information studies, university of alberta), becoming evidence based will introduce attendees to the current dialogue and research in ebp through contributed papers and discussions. registration fee: eblig members: $35.00 cla members: $45.00 non-members: $70.00 (includes afternoon refreshment break) for more information and to register, please visit http://www.cla.ca/conference/2010/ article   the impact of leadership development on the organizational culture of a canadian academic library   jill crawley-low librarian university archives & special collections, university of saskatchewan saskatoon, saskatchewan, canada email: jill.crawley@usask.ca   received: 19 june 2013   accepted: 30 oct. 2013      2013 crawley-low. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine the perceived impact of leadership development on the behaviours and competencies of employees and the organizational culture of the university library, university of saskatchewan, canada.   methods – using grounded theory methodology, the study was conducted in an academic library serving a mid-sized medical-doctoral university in western canada. twenty-one librarians and support staff who had completed the university library’s library leadership development program (lldp) participated in one-on-one interviews of 40-60 minutes duration. interview transcripts were prepared by the researcher and reviewed by the participants. after editing, those source documents were analyzed to reveal patterns and common threads in the responses. the coding scheme that best fits the data includes the following four headings: skill development, learning opportunities, strategic change management, and shared understanding of organizational vision and values.   results – according to the responses in interviews given by graduates of the library leadership development program, the library’s investment in learning has created a cohort of employees who are: self-aware and engaged, committed to learning and able to develop new skills,  appreciative of change and accepting of challenges, or accountable and committed to achieving the organization’s vision and values.   conclusion – competencies and behaviours developed through exposure to leadership development learning opportunities are changing the nature of the organization’s culture to be more collaborative, flexible, open and accepting of change and challenge, supportive of learning, able to create and use knowledge, and focussed on achieving the organization’s vision and values. these are the characteristics commonly associated with a learning organization.   introduction   academic libraries operate in complex and continuously changing information environments as members of the library community and their parent academic institutions. often, large organizations have a conservative culture and may be rigid in their hierarchical structure: traits that do not promote flexibility in responding to shifting environmental conditions. however, libraries can leverage the actions of their most influential and important resource, their employees, to loosen the organizational culture. the university library at the university of saskatchewan has found that a well-designed and effectively delivered leadership development program empowers employees to learn new skills to develop behaviours that allow them to respond positively to change, to accept challenges in their daily lives, and to become flexible enough to solve problems and create knowledge.   in this project, graduates of an in-house leadership development program were asked to consider the value of the learning opportunity and whether they perceived changes in their own behaviours and that of their colleagues that were reflected in changes to the organizational culture. a significant investment by library administration over time has been channelled into the leadership program and, although surveys gathered information about the effectiveness of the program, there was no investigation of the impact of employees’ leadership behaviours on themselves, their colleagues, and the organizational culture. the research presented here supplies evidence that the organizational culture has changed as employees have developed and applied their leadership skills.   with the arrival of a new library dean in 2006, the university library at the university of saskatchewan committed to developing its employees as leaders and innovators. the vision, as described in the university library’s strategic plan (2013b), is supported by several initiatives that are designed to transform library collections, facilities, services, and the organizational culture. as expressed by the dean, “at the heart of the plan was our vision to be leaders and innovators and to create a dynamic learning organization” (williamson, 2013, p. 134). fully realizing this vision would involve developing an engaged workforce operating in a supportive learning environment. one of the action items in the plan was to develop a process for growing management leadership and expertise within the library and rolling it out to all levels of library employees. from this action item the library leadership development plan (lldp) was launched in 2009.   the lldp was designed and delivered by two organizational development specialists, neither of whom were librarians. organizational development is “an ongoing, thoughtfully planned effort by all members of an organization to improve how that organization operates, serves its stakeholders, fulfills its mission and approaches its vision” (stephens & russell, 2004, p. 241). in developing the content of the program, the organizational development specialists gathered input from library employees through focus groups. the first lldp was presented to 18 librarians with or without administrative responsibilities. the second lldp was presented to support staff with supervisory or technical responsibilities. later iterations, as lldp3 and lldp4, were open to all library employees.   the lldp consists of six modules offered in sessions of two consecutive days in a pleasant off-campus location over several months. class size is limited to 20 employees. each module highlights a leadership competency supported by group discussion and individual study. each module includes a behavioural assessment instrument to encourage self-awareness and better understanding of colleagues. midway through the program, participants form three teams and each team selects a “leadership action challenge” from a list of leadership topics of interest to academic libraries. this encourages practical application of leadership skills by participants, and recommendations from the challenge reports have found their way into the library’s strategic plan as action items. the lldp program at a glance is reproduced in appendix a.  more details about the program can be found in williamson (2013).   although the future of the lldp as an ongoing development program was unknown in 2009, when the inaugural program was offered, it has been well received by employees and library administration has remained committed to the program. the stable program content over time has resulted in a valuable collection of longitudinal data about satisfaction with the program. from an analysis of data from the four lldp offerings, braganza(2012) reports “we can infer participants in each cohort gained valuable leadership knowledge and skills” (p. 16).   the university library collects data from other sources as well. since 2005, there has been an annual employee opinion survey including questions to measure employee engagement. as described by mierke (2013), “in 2005 … only 54% of library employees were considered to be engaged. however, between 2005 and 2011 this score steadily increased to 72% and held steady in 2012.” the lldp and its companion sustaining leadership program is the primary leadership development program in the library. although there are other training and development opportunities for employees, the lldp focusses on leadership, which suggests that it has contributed to the rising engagement scores.   the current study takes a different route and links the development of employees’ leadership competencies and behaviours with changes in the library’s organizational culture based on exposure to leadership development programs.   literature review   the leadership and organizational development literature is multi-disciplinary and voluminous compared to the more modest literature focussing specifically on leadership and organizational development occurring in libraries. a special issue of library trends on organizational development and leadership is edited by russell and stephens (2004) and is very helpful. selected topics discussing the application of organizational development and leadership theory in a library setting include the learning organization by giesecke and mcneil (2004), organizational culture by kaarst-brown, nicholson, von dran and stanton (2004), and organizational development by stephens and russell (2004). leadership in libraries is an important ongoing area of interest because there is a requirement to mentor the next generation of library leaders (branin, 2012) and to transform libraries through leadership development programs (fallon, maxwell, mccaffrey & mcmahon, 2011; jordan, 2012; merrill & lindsay, 2009; michalak, 2012; williamson, 2013).   since the research in leadership and leadership development, organizational culture, and the learning organization has occurred mostly in the business disciplines, it is useful to mention the concepts here. there are many definitions of leadership. a simple, but a useful one, suggested by day and antonakis (2012), is: “leadership is purpose-driven action that brings about change or transformation based on values, ideals, vision, symbols and emotional exchanges,” as (p. 5). thus, leadership becomes embedded in the social interactions among employees who are working towards a common goal. another definition was developed by the first lldp cohort in 2009:   [leadership is] using our behaviour to influence others to willingly follow an idea, process or vision for the common good. everyone can be a leader, not just by position. there is leadership at all layers and levels within the library. (university of saskatchewan, 2009).   for individuals and groups, leadership development is the expansion of an individual’s capacity to be effective in leadership roles and process. “leadership is a social process… ld [leadership development] can be something we do along the way as part of our work, not something additional we do on the side if we have time” (gaines, 2012, p. 9).  an effective leadership development program sets the conditions for direction, alignment, and commitment at the individual, group and organizational levels.   organizational culture has been widely studied and the findings applied in a variety of work settings where organizations want to maximize effectiveness in complex situations.  according to schein (1985):   the culture of a group is a pattern of shared basic assumptions – invented, discovered, or developed by a given group as it learns to cope with its problems of external adaptation and internal integration that has worked well enough to be considered valid and, therefore, to be taught to new members as the correct way to perceive, think and feel. (chapter 1, p. 9). in fact, the strength and stability of a culture is derived from its group-based nature. therefore, culture is deeply entrenched in organizations and results from the behaviours and attitudes of employees of the organization. although the culture is always evolving, it takes time to change basic assumptions, so change occurs first with adjustments in employees’ behaviours and attitudes in response to environmental and other factors impacting the organization.   employees developing leadership skills need the supportive environment of a learning organization. the learning organization is a concept that is characterized by five learning disciplines, one of which is systems thinking. according to senge (1990), in writing about the learning organization, the practice of systems thinking underlies the other four learning disciplines. further, the systems thinking discipline is a way of dealing with complexity in organizations by seeing interrelationships rather than linear cause-effect chains. the lldp introduced library employees to the theory underpinning the five disciplines of a learning organization: shared vision, personal mastery, mental models, group learning, and systems thinking (senge, 1990, pp. 6-11). a learning organization is one which is “skilled at creating, acquiring and transferring knowledge and modifying its behavior to reflect new knowledge and insights” (giesecke & mcneil, 2004, p. 55). meanwhile, a leadership development program provides the resources by which employees can develop leadership competencies and behaviours and, in turn, influence the organizational culture.   aims   the goal of this study was to describe library employees’ perceptions of the impact of the lldp on themselves, their colleagues, and the organization. the research questions guiding this study were:   which leadership behaviours do the lldp graduates perceive having developed as a result of their exposure to library leadership develop opportunities? how do the lldp graduates rate the effect of library leadership programs on their leadership development? which effects of leadership learning have the lldp graduates observed in their colleagues and in the organization?   methods   the study took place in an academic library serving a mid-sized medical/doctoral university in western canada using grounded theory methodology. this research project was approved by the university of saskatchewan research ethics board in june 2012.   grounded theory, as described by glaser and strauss (1967), is the methodology used for analyzing the qualitative data collected in this study. it is a systematic approach to data collection and analysis that is, according to kenealy (2012), increasingly used in organizational and managerial research. grounded theory is explicitly emergent with the focus is on determining which theory accounts for the research situation, rather than on hypothesis testing. the starting point is data collection using unstructured interviews (alvesson & ashcraft, 2012), which is supplemented by observational data, findings from the literature, and additional sampling of relevant data.   in this study, the researcher invited 44 library employees who were graduates of the lldp to participate in the study. twenty-one consented, and this number of research participants was sufficient to reach saturation of the data, and therefore the researcher recorded no new responses as the last transcripts were coded. the lldp graduates participated in one-on-one interviews of 40-60 minutes duration in a neutral study location.   the researcher administered a set of interview questions that included both standardized questions and open-ended questions designed for full exploration of the participants’ perceptions of the effect of leadership learning on themselves, their colleagues, and the organization. the interview questions are included in appendix b. within a few days of each interview, the researcher prepared an interview transcript to be reviewed by the participant. these source documents were imported into nvivo software for analysis.   nvivo was used to organize verbatim segments from the interview transcripts that expressed the participants’ responses to interview questions. the researcher coded segments of the transcripts using the following explanatory headings: skill development, learning opportunities, strategic change management, and shared understanding of organizational vision and values. these headings became the leadership learning model. the researcher assigned levels of importance to the data by frequency or originality of coded responses, as recommended by sinkovics and alfoldi (2012) and by bazeley (2007). the researcher organized interviewees’ comments using the categories of the leadership learning model. it was evident that participation in the lldp had had a positive effect on individuals and the organization.   results   the study population                 although 70 employees graduated from the lldp from 2009 to 2012, the available study population in summer 2012 was restricted to 44 potential participants because of leaves, vacations, retirements, and resignations. the final study sample consisted of librarians, library assistants, and administrative employees, twenty of whom were women. the majority of the 21 participants were library assistants (9/21) or librarians (9/21), and a minority (3/21) were administrative employees. the librarians and library assistants had different profiles for years of service as shown in figure 1. the majority of librarians (5/9) were relative newcomers with 0 to 5 years of service, whereas, the majority of library assistants (5/9) were senior employees with 20 years of service or more. although it may seem that the administrative employees were under-represented, the proportion of study participants reflects the profile of the library as a whole.   figure 1 participants’ classification and years of service.   whether library employees work directly with clients, as in branch libraries, or work with clients internal to the library, as in service units, their perspectives on service may differ. thus, lldp organizers took care to include both branch and unit employees in the sessions. all seven branches and all four units in the university library were represented in the population, with 52% of the participants from branches (11/21) slightly outnumbering 48% of the participants from the units (10/21).   self-rating leadership development   during the interviews, respondents were asked to rate their own leadership development in three time periods: before lldp, six months after lldp, and currently. the majority of participants (15/21) perceived an increase in their leadership skill level from the time period, before lldp, to the current time period. they cited three factors for their improved leadership skills:   ·         participating in ongoing leadership learning opportunities such as the library’s leadership community of practice or leadership reading club; ·         personal motivation was used to keep learning; and  ·         consciously and routinely applying new skills in their work.   those participants (6/21) whose leadership development did not increase after attending the lldp cited the following reasons:   ·         work demands leaving no time for participating in continuing learning opportunities; ·         substantial leadership experience and learning prior to the lldp; ·         changing work roles; and ·         failing to use the skills they had developed.   overall, the majority of respondents (15/21) answering this question saw their self-identified leadership rating improve as result of attending the library leadership development program.   leadership learning model   five open-ended questions asked for fuller exploration of the participants’ perceptions of the value of leadership learning and the effect of leadership learning on themselves, their colleagues, and the organization (see appendix b). as the responses were coded and different organizational schemes were tried, the following categories kept reappearing: skill development, learning opportunities, strategic change management, and shared understanding of organizational vision and values. these categories formed the leadership learning model, which was devised by the researcher as a way to organize the interview responses. upon reviewing the literature, the researcher found that these categories often appeared in combination with other categories in descriptions of the essential components of leadership learning programs. the model has two categories in common (skill development and shared understanding of vision and values) with the discipline of systems thinking, which includes shared vision, personal mastery, mental models, and group learning (senge, 1990). the four categories are discussed in greater detail with verbatim comments from interview participants.   skill development   in the category of skill development, the behaviours resulting from their exposure to leadership learning and development opportunities that were most often cited by respondents were increased self-awareness (14/21 participants) and increased self-confidence (10/21 participants). the assessment and feedback instruments offered by the lldp and other leadership opportunities allowed participants to explore their own strengths and weaknesses. for example, participant 16 noted that, “the idea of not always focussing on what’s wrong, this was much more positive, building on what’s good, let’s recognize what is working which is really nice”.    some interviewees perceived leadership learning to be of value personally, while others cited professional growth. participant 16 noted, “[i] remember it being said that leadership can happen in your personal life. i hadn’t thought of that”. participant 21 analyzed leadership learning as,   any learning, but specifically leadership learning to me has made me a better person, professionally. a lot of that has to do with the confidence it’s given me, the fact that i will seek opportunities, it’s given me a bit more of a polish that i can now seek out these opportunities and be a team member and take on more of a leadership role on committees, volunteering more to take on tasks associated with committees.     figure 2 leadership learning model.   according to participant 2, leadership learning is, “allowing me to stay fresh and excited because leadership in a managerial or supervisor role can be tiring and exhausting and difficult from time to time.” according to participant 9:   showing leadership isn’t necessarily being the ceo running an office. it’s being the ceo of your life. you have the tools to decide where you want to go with this. you can change the way you think about your career, your life, how you handle situations or you can stay the same. it’s stirring something up that’s making you face some of these things that you’ve never uncovered.   participant 21 noted:   the majority of people i speak to have been glad they went through lldp, they have confidence. roles and responsibilities have changed because they are now applying for positions, because they feel they have the skills and confidence. they are moving outside that box that they were in before. i think we are seeing more progression in career paths.   in addition to developing self-awareness and self-confidence, participants commented on their enhanced understanding of the perspectives and actions of others. in the lldp, cohort members shared the results from their personality instruments with one another, leading to a better understanding of themselves and also why colleagues behave as they do. interviewees commented that as a result of their deeper understanding of colleagues’ behaviours they were more accepting of others. they recognized and appreciated diversity more and felt greater respect for their colleagues, which resulted in positive behaviours such as being less critical and more tolerant, patient, and empathetic toward colleagues.   participant 14 spoke about “recognizing that just because they [other employees] are doing things differently doesn’t mean it’s wrong, recognizing behavioural styles and different personalities, it really helps to understand where a person is coming from”. meanwhile, participant 11 noted:   learning about myself is a big benefit, but learning about others is bigger which leads to being more patient and empathetic”. participant 14’s attitude towards colleagues was, “being a better coach and mentor, being less critical, being more open and giving the person more time to grow, and just being more tolerant.   with improved understanding of others came more effective and efficient personal and professional relationships.   team-building is an activity that results from individual skill development and awareness of others. the lldp presented theory about the characteristics of high-functioning teams which was subsequently applied in practice as teams were formed and members worked collaboratively for program-end presentations. participants noted that teams seemed to work better after the lldp because there was a common understanding of the attributes of teams and how effective teams operate. according to participant 2, “lldp has certainly helped the team approach that shared roles and responsibilities and staff contributing and being part of teams and saying ‘i’ll try that,’” while participant 12 noted, “before i used to think who broke it, now i think how can we fix it? it’s a mind shift. if i do one thing better today than before then it’s ok.”   employees commented on the shared experience and language of leadership learning that helped improve their relationships with colleagues and boosted collaboration. participant 12 explained:   [lldp has affected] the majority of people who have gone through; it’s become a good conversation piece among people who don’t associate usually. there is more communication and a shared experience to build on. it’s creating an atmosphere that we are all one group.   participant 21 agreed:   indeed, because the teams and individuals are the foundation of the organization, and if the individuals and the teams are working effectively, then this will be reflected in the organization and it will work better.   the idea of “leading from where you are” is a tagline from the lldp, and 4/21 interviewees included that specific phrase in their answers. according to participant 16,   “there is wider recognition that you can lead from wherever you are, [you] didn’t need to be in a formal leadership position; it just raised my awareness that i did have some influence that i didn’t think i had previously”.   overall, the skills that participants developed and the behaviours learned as individuals and on teams have created a cohort of self-aware and engaged employees.   learning opportunities   the learning opportunities category relates to the development of holistic leaders working in a supportive learning organization. learning by doing was observed in the library’s environment of leadership development. interviewees expressed a desire to make time in their busy lives to hone and practice leadership skills and apply them until they became automatic. participant 4 commented that, “a lot of this stuff you hear it over and over again and it eventually becomes natural as opposed to a conscious act, while participant 21 noted, “i think we learned that for the future we will always be able to apply [learning] and make the organization better and serve our users.”.   interviewees showed their commitment to learning with their comments about sustaining leadership in the library. they noted that self-directed groups such as the leadership community of practice, the leadership reading club, and other sustaining opportunities, all foster ongoing learning. they also noted that library management’s commitment to leadership development has been reflected in higher numbers of employees participating in library-sponsored leadership learning opportunities. participant 19 noted that, “more people are showing up to events and training, more of the library staff goes to sessions.”   arising from the value of the shared learning experience, the lldp graduates showed their commitment to the learning organization by lobbying library managers to continue to support learning opportunities that are available for all employees. according to one, “any investment that an organization makes in training of any kind is always worth it” (participant 22). another noted, “i became an advocate for lldp; i encouraged several people personally who were thinking of taking the next session after mine” (participant 10), while a third commented, “there is also a little bit more of a realization that you risk being left behind, if you don’t change” (participant 12).   in terms of developing diverse skills, participants cited cross-training opportunities as one way to help improve relationships and dissolve real or perceived barriers or silos. participant 10 noted, “staff don’t need to move around but it’s useful if there is a cross-training opportunity and you get a different perspective”. overall, participants valued a learning environment where they could participate actively in their own learning, develop new skills, and share learning with colleagues.   change management   organizations use strategic change management tools such as improved communication and alignment of priorities to help build change-related skills. thus, an objective of the lldp was “to develop a plan with change management and communications strategies, including success factors, to increase probability of positive leadership skill development” (clarke, 2010, p. 3). the lldp and other university library-sponsored leadership learning opportunities have addressed the theory of change in organizations and the importance of clear and open communication at all levels in the organization.   there is evidence from participants’ comments that:   ·         they acknowledge that change is inevitable; ·          they feel more confident about embracing change including feeling more flexible in their approaches to work; and ·         some have expressed a personal desire for change.   employees have been introduced to change management tools in the lldp and that has taken away some of the fear of change. they have embraced change as inevitable and potentially beneficial and are beginning to perceive ways of working in the organization that are more flexible and experimental. participant 11 noted, “because of our history here, people don’t accept change too well, so it was good to get people introduced into it.” another interviewee mentioned that, “staff are adapting to changes now, they know change is inevitable” (participant 12), and according to participant 10, “i see people changing and ideas are forming and people are more willing to talk about those things and look at things from different perspectives.” as noted by participant 5,   i have said this to lots of people that i feel fortunate to work in an organization that supports leadership. it takes a lot of time and effort and it’s a big investment of resources. i don’t think everyone recognizes how huge the opportunity is.   participants felt that improved communication helped make them more aware of the details of change occurring in the organization and more aware of the effects of those changes on the organization. one noted, “we are not implementing this change because we want to do something to you; it’s just better for the organization to do it this way” (participant 11), while “communication with staff is better, there is more transparency” (participant 18). some felt an expectation to give opinions and a new freedom to question decisions, although others felt that feedback was not always welcomed. participant 11 would, “step out of my comfort zone and do things i wouldn’t have done before and speak more, give my opinion”.   an important aspect of change management is succession planning, which appears as initiatives in both the university of saskatchewan’s people plan (2013a) and the library strategic plan (2013b). the strategic plan was created in 2006 by a large team with broad library representation. it makes public the vision, mission, and values of the university library which are supported by four strategies. the plan is reviewed annually and action items related to the strategies may be added or removed. a report card keeps track of the progress of all action items. central themes in the strategic plan recognize the critical role of library employees in all aspects of library services, collections, and facilities development. the people plan was created using a similar process that developed the strategic plan. the four core people strategies are: learn and develop, relationships and collaboration, appreciate and celebrate, and conversation and communication. by providing employees with the means to develop leadership skills and fostering the learning environment with sustaining leadership development offerings, the organization expects that employees will develop leadership competencies and behaviours that contribute to transformation. this will also address, in part, managing the employee complement for optimal performance. participant 16 noted,   i don’t know if this is related directly to the library leadership program but i do feel a bit of pressure that the expectation is that the ones who are here will be expected to take on more leadership roles.   participant 17 echoed that thought, “you are not management, but we expect leadership from you”.   interviewees were well aware of the generational change occurring in the library with the departure of experienced, mature employees and the intake of new employees with marketable skills that the transformed library needs. as noted in the individual skill development section, the lldp has had a positive effect on team building, collaboration and positive behaviours including respect for each other. as a result, there is more sharing of knowledge and more flexible approaches to completing tasks. participant 18 commented,   with our clients, they come to the library and talk to people who are excited about the library and engaged with the library it makes a more positive experience with the library. newer staff are engaged, but the older staff after taking the program are a little bit more positive[ly].   interviewees indicated that certain job duties need to evolve to satisfy the career plans of employees who now see themselves as leaders. participant 2 noted,   it’s a challenge [for supervisors keeping control] because all of a sudden they’ve got people who want to participate in lots of things and want to show their initiative, they have great ideas they want to share them and apply them”.   overall there is a new acceptance of the inevitability of change, without fear, and a willingness to accept challenges and work together for the betterment of the organization.   shared understanding of the organization’s vision and values   shared understanding of the organization’s vision and values is a fundamental aspect of leadership development, in which the direction of the organization is spelled out in living strategic planning documents and employees are encouraged to align their activities to support the organization’s goals. a strong sense of community also arises from shared understanding of the organization’s vision and values.   the strategic plan and the people plan make public the values and dreams of the organization, and describe the initiatives required to move towards achieving the transformative vision. in terms of understanding the organization’s vision and values, participants noted that a grasp of the “big picture” view of the organization is more common than in the past and there is increased comprehension of progress towards the transformation as described in the library’s vision. participant 10 explained, “what i saw that i would really applaud [in] coming to this organization was the planning, the organization, strategic directions, and the focus on growth, training and change”. participant 12 noted,   i personally like to be someone who has a better knowledge of the big picture. some people want to come in and [they say], ‘let me do my job, i want to fly under the radar’. i like to know where i fit in with regards to the rest of the organization.   the library strategic plan and the people plan are also documents that guide personal development, as noted by participant 8:   leadership development speaks to aligned priorities whatever leadership looks like, whether it’s a formal role or not, it’s a sense of ownership of being part owner of this space and this place and this organization, so i don’t feel like just a number putting in time to do a certain amount of work every day. it feels like i have stake in this.    the environment of strategic planning in the university library is reinforced by the university’s open discussion and development of its integrated plan. there is the perception by the university that the library is a valued employer because they focus on developing employees. according to one interviewee, “we [the university library] are much more strategic in the way we work by aligning with the university and it seems to reinforce what we’re doing and why we’re doing certain things” (participant 1). this was echoed by participant 12, who noted “in the campus community, i interact with a number of people and the library is seen as a progressive employer. it’s not perfect, but the library is one of the more progressive units”.   another aspect of developing a shared understanding of the vision and values is a strong sense of community. this sense was fostered by the lldp in the intensive shared learning experience and has been built upon by other leadership learning opportunities. there was general recognition that participation in leadership development opportunities provided lldp participants with tools and information to behave as leaders, which in turn helped create a more positive mindset towards work, better relationships with colleagues, and a sense of their place in the organization. several interviewees concurred, such as participant 17, who noted, “for people who buy in, it’s affirming, it’s motivating,” while another commented “[there was] more collegiality later amongst the staff of four cohorts that have the same language” (participant 14). on a similar theme, “there’s much more acceptance of changes in roles and responsibilities, and more shared roles and responsibilities” (participant 2). participant 7 expressed confidence in the learning community, saying, “i am seeking out relationships that might have been intimidating to me before.” overall, the shared understanding of the vision and values underscores the importance of each employee’s contribution to the organization and creates momentum towards achieving the transformative vision.   the culture of the organization is being changed by the behaviours and attitudes of employees who have embraced leadership development. these are the kind of employees who will contribute to and sustain a learning organization. as noted by giesecke and mcneil (2004), employees “are engaged and accountable; they appreciate change; accept challenge; are able to develop new skills; and are committed to the organization’s vision and values” (p. 55).  university library employees see themselves as leaders and have expectations of themselves, of their colleagues, and of the organization to achieve a common vision.   discussion   using the leadership learning model as a framework for organizing the varied responses from the interviews, it is the perception of the lldp graduates that the program has helped them develop leadership behaviours and competencies. this was asked by the first research question.   for the category of individual skill development, participants reported increased self-confidence and self-awareness, and a deeper understanding of, and respect for, others. interpersonal relationships between individuals and among groups are more open and collaborative, and individuals are prepared to share knowledge and break down silos in the organization. this is evidence of a self-aware and engaged workforce.   in the area of learning opportunities, participants valued learning and recognized the importance of sustained leadership development for themselves and for all employees in the organization. they were prepared to lead from where they are and to continue learning, developing and practicing leader behaviours and competencies. this is evidence of commitment to learning and becoming a more skilled and forward-thinking workforce.   participants demonstrated an altered perspective of change in the workplace. they embraced change as inevitable and potentially beneficial, and no longer to be feared. they acknowledged that improved and increased communication at all levels has made decision-making seem more transparent and understandable throughout the organization. they have more flexible attitudes toward work, how it is accomplished, and by whom. this is evidence of a workforce that can appreciate change and accept challenges.   finally, interviewees reported a deeper understanding of the organization’s vision and values and an enhanced mental image of their place in the organization. knowing that their contributions to the organization were important translated into an enhanced service philosophy and heightened leadership expectations for everyone in the organization. a stronger sense of community has developed through shared learning experiences coupled with a deeper understanding of the organization’s vision and values. shared activity towards a common goal provides the momentum that moves the organization closer to achieving its vision. this is evidence of employees who are accountable with a commitment to the organization’s vision and values.   the second research question asked how lldp graduates rated the effect of the program on their development. overall, the majority of respondents (15/21) answering this question saw their self-identified leadership rating improve as result of attending the lldp. the success factors cited included participation in ongoing leadership activities, personal motivation to continue learning, and consciously applying new skills at work. learning was hampered for others by competing demands on their time, changing work roles, and failure to practice and develop leadership skills.   the third research question asked the lldp graduates for their perceptions of changes to the organizational culture. culture is deeply entrenched in organizations and cannot be imposed externally; it has to develop internally in response to conditions within the organization, and is always evolving. changes in the culture of an organization are first seen as changes in the behaviours and attitudes of employees, and occur over time as basic assumptions are changed. this study indicates that the library’s organizational culture is changing as employees develop their leadership competencies and behaviours. their competencies and behaviours are changing the organization’s culture to be more collaborative, flexible, open to change and challenge, supportive of learning, able to create and use knowledge, and focussed on achieving the vision and living the values.   the inherently conservative nature of the library organization resists opportunities to develop fully into a learning organization. there are a number of examples of deficits, including fear of risk-taking, failure to foster innovative thinking or reflection, discomfort with feedback from colleagues, and failure to translate learning into practice. all are opportunities to learn, but may fall prey to reverting to old ways of thinking and doing. the response to the conservative organization that is resisting change is to support and empower employees through a strong and sustainable leadership development program in a supportive learning environment that allows them to keep the transformative vision in view at all times.   conclusions   the university library’s vision to transform library collections, facilities, services and the organizational culture is activated by initiatives in its strategic planning documents. some of those initiatives focus on the development of leaders within the organization whose leadership competencies and behaviours will, by combined effort, change the culture of the organization. according to the responses in interviews given by graduates of the lldp, the investment in learning for employees has paid off by creating a cohort of workers who are self-aware and engaged. these employees appreciate change, accept challenges, are able to develop new skills, and are committed to achieving the organization’s vision and values.   their leadership behaviours and competencies developed by employees during the library leadership development program offerings have translated into changes in the organizational culture. participants perceive the culture to be more collaborative, operating more transparently, open to change, supportive of learning and creating new knowledge, and focussed on a common vision. the cycle involving leadership development occurring in a supportive learning organization leads to continuous progress towards the library’s transformative vision of leaders and innovators operating in a learning organization.      references   alvesson, m., & ashcraft, k. l. (2012). interviews. in g. symon & c. cassell (eds.), qualitative organizational research: core methods and current challenges (pp. 239-257). los angeles, ca: sage.   bazeley, p. (2007). qualitative data analysis with nvivo. los angeles: sage.   braganza, g. (2012). an evaluation of the library leadership development program (lldp). unpublished manuscript, university library, university of saskatchewan, saskatoon, canada.   branin, j. (2012). editorial: library leadership. college & research libraries, 73, 113.   clarke, j. (2010). the library leadership development program (lldp): report on cohort 1. unpublished manuscript, university library, university of saskatchewan, saskatoon, canada.   day, d. d., & antonakis, j. (eds.). (2012). the nature of leadership (2nd ed.). los angeles: sage.   fallon, h., maxwell, j., mccaffrey, c., & mcmahon, s. (2011). engaging with leadership development in irish academic libraries: some reflections of the future leaders programme. the australian library journal, 60(1), 8-20.   gaines, k. (2012). leadership development: let's all take a fresh look at the field. leadership excellence, 29(1), 9.   giesecke, j., & mcneil, b. (2004). transitioning to the learning organization. library trends, 53(1), 54-67.   glaser, b. g., & strauss, a. l. (1967). the discovery of grounded theory: strategies for qualitative research. chicago: aldine pub. co.   jordan, m. w. (2012). developing leadership competencies in librarians. ifla journal, 38(1), 37-46.   kaarst-brown, m. l., nicholson, s., von dran, g. m., & stanton, j. m. (2004). organizational cultures of libraries as a strategic resource. library trends, 53(1), 33-53.   kenealy, g. j. (2012). grounded theory: a theory building approach. in g. symon & c. cassell (eds.), qualitative organizational research: core methods and current challenges (pp. 408-425). los angeles, ca: sage publications.   merrill, a. n., & lindsay, e. b. (2009). growing your own: building an internal leadership training program. library leadership & management, 23(2), 85-87.   michalak, s. c. (2012). this changes everything: transforming the academic library. journal of library administration, 52(5), 411-423.   mierke, j. (2013). leadership development to transform a library. manuscript submitted for publication.   russell, k., & stephens, d., eds. (2004). organizational development and leadership [special issue]. library trends, 53(1), 1-264.   schein, e. h. (1985). organizational culture and leadership: a dynamic view (p. 9). san francisco, ca: jossey-bass.   senge, p. m. (1990). the fifth discipline: the art and practice of the learning organization. new york, ny: doubleday.   sinkovics, r., & alfoldi, e. a. (2012). facilitating the interaction between theory and data in qualitative research using caqdas. in g. symon & c. cassell (eds.), qualitative organizational research: core methods and current challenges (pp. 109-131). san francisco, ca: sage.   stephens, d., & russell, k. (2004). organizational development, leadership, change, and the future of libraries. library trends, 53(1), 238-257.   university of saskatchewan. (2009). university library definition of leadership. retrieved from http://library.usask.ca/intranet/learning_development/leadership   university of saskatchewan. (2013a). university library people plan 2012-16. retrieved from http://library.usask.ca/info/peopleplan.pdf   university of saskatchewan. (2013b). university library strategic plan. retrieved from http://library.usask.ca/info/strategicplan.pdf   williamson, v. (2013). leadership to transform our library: a case study from the university library, university of saskatchewan. in s. mossop (ed.), achieving transformative change in academic libraries (hong kong ed.,) pp. 133-161. oxford: chandos.       appendix a sample lldp program at a glance module 1 module 2 module 3 module 4 module 5 module 6 leadership & relationship building team building leading change planning & accountability leadership & organizational culture personal mastery & organizational effectiveness   ·    what is leadership? ·    personal style ·    communication ·    developing others   ·   stages of team development ·    team player style ·    team effectiveness ·    team charter   • leadership style • effective change process • mindset • impacts •appreciative inquiry   ·       nature of strategic management ·       aligning goals & objectives ·       defining & developing accountability ·       participation     ·      what is culture? ·      how does it form? ·      creating a learning & service culture   ·   personal growth ·   presentations re: organizational leadership challenge ·   ongoing leadership learning plans discussion topic: communication discussion topic: collaboration discussion topic: transformation discussion topic: transparency discussion topic: organizational culture discussion topic: mastery   assessment tool:   assessment tool:   assessment tool:   assessment tool:   assessment tool:   assessment tool:   leadership action challenge team status report 1/2 day   appendix b: interview questions   project title:       leadership learning and organizational culture in an academic library   the interview consists of nine questions in total. the first three are simple identification questions followed by a ranking question. there are three complete the sentence type questions. question #8 explores the organizational culture and the final question asks if you have any final comments on leadership or organizational culture. are you ready to begin?   please indicate your cohort from the library leadership development program (lldp) lldp1 lldp2 lldp3 lldp4   please indicate your bargaining unit a.       aspa b.       cupe c.        faculty association   please indicate your length of service in the university library a.       0-5 years b.       6-10 years c.        11-20 years d.       21+ years   the university library began a structured library leadership program in 2009 with lldp and has continued with sustaining leadership series 2 and training and development opportunities as part of ldc’s program. considering the leadership opportunities you have experienced since 2009, please rate your own leadership development as high, medium, or low before lldp 6 months after lldp currently   can you tell me more about your rating for each of these time periods?   here are three questions that ask you to complete a sentence with your thoughts   to me, library employees who develop and apply leadership skills are … a.       which behavioural changes have you observed? b.       is there a payoff for the investment in learning? what is it?   developing and applying leadership skills in my work means … which leadership skills have you developed? how have those leadership skills been applied in your work?   the value of leadership learning a.       to me is … b.       to my team is perceived as … c.        to the organization is perceived as …   this is the question about the organizational culture in the library and whether or not you have seen changes that you would relate to leadership development programs.   have you seen any changes in the organizational culture resulting from the leadership development program? have the roles and responsibilities of employees changed? how? has distribution of resources changed? how? have operations in your work area changed? how? have strategies for the library changed? how? which key behaviours have you observed?  what has been the effect of those behaviours on the organization? do you see the leadership learning program affecting the organization in the future? please tell me more about that?   we are closing in on the end of this interview so i would like to ask if you have   any other thoughts or comments on library leadership and organizational change? or anything else you would like to mention?   thank you again for you participation. microsoft word art_ makani.doc evidence based library and information practice 2006, 1:4  30 evidence based library and information practice     article    information seeking behaviours of business students and the development of academic  digital libraries      joyline makani  dalhousie university  killam memorial library  halifax, nova scotia, canada   e‐mail: makani@dal.ca    kelli wooshue  halifax public libraries  spring garden road memorial public library  5381 spring garden road  halifax, nova scotia, canada  e‐mail: k_wooshue@hotmail.com      received: 05 june 2006    accepted: 02 august 2006      © 2006 makani and wooshue. this is an open access article distributed under the terms of the creative  commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    abstract    objectives ‐ to gain insight into the extent to which user information‐seeking behaviours  should inform the design and development of digital libraries in an academic setting, a  study was carried out at dalhousie university, canada to explore the information‐seeking  behaviours of business students.    methods ‐ the students studied were drawn from the school of business administration at  dalhousie university, canada. the study was based on qualitative and quantitative data  collected through a survey, in‐depth semi‐structured interviews, observational study and  document analysis. qualitative case study data was coded using qsr n6 qualitative data  analysis software. the data was categorized using atkinson’s “model of business  information users’ expectations” and renda and straccia‘s personalized collaborative dl  model. atkinson’s model defines the expectations of business students in terms of cost, time,  effort required, pleasure and the avoidance of pain. renda and straccia’s model of a  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:4  31 personalized and collaborative digital library centres around three concepts: actors, objects,  and functionality. the survey data was analysed using the zoomerang software.    results ‐ the study results revealed that students tend to select resources based on cost  (free or for fee), accessibility, ease of use, speed of delivery (of results), and convenience.  the results showed that similar to atkinson’s findings, the business students’ information‐ seeking behaviour is influenced by the concepts of cost‐benefit and break‐even analyses that  underlie business education. concerning speed of delivery and convenience, the  organization of the resources was paramount. students preferred user‐defined resource lists,  alert services, and expert‐created business resource collections. when asked about the  usefulness of potential digital library functionalities, students valued a personalized user  interface and communal virtual spaces to share information and communicate in real‐time  with their peers.    conclusion ‐ this study reveals that when digital libraries are developed, user behaviours  and needs should be taken into consideration. it demonstrates that the activity as well as the  “user’s orientation and motivation” (here the business student training) can directly  influence the design and use of a digital tool. in other words, this study confirms a new  typology of a business digital information user or use behaviour, one that requires the  building of dedicated accredited library research systems. providing information and  information tools tailored to this specific audience is more likely to increase the appeal and  use of an academic business digital library.       introduction    there is a wealth of research on the  development of digital libraries, particularly  regarding information retrieval and  technical considerations. however, very few  researchers have focussed on how  information is sought and transformed into  knowledge and how this should influence  digital library design.  in this study a simple  definition of a digital library is used, that  defined in the dictionary of human  geography as, “a system providing the  services of a library in digital form.” digital  libraries therefore, are not simply collections  of digital versions of existing resources. as  covi and kling observed, effective digital  library design requires an understanding of  how users do their work, how they use  information, how they create knowledge,  and how digital libraries support these  processes. just building a digital library  would not be enough; the digital collections,  storage, and transmission should be useful  to people who use them. (672) research on  undergraduate students indicates that they  are increasingly able to use digital resources  even though they are not necessarily  information literate. (lombardo and miree,  6) it is critically important that digital  libraries help improve the transferable  information literacy of students.     this study was conducted to provide new  insights on information seeking behaviours  of business students, exploring implications  of these findings to the development of  business digital libraries.  the research  project examined and analysed how  business students learn, gather, and use  information individually and collectively,  and how they share their work.  the study  was based on qualitative and quantitative  data collected through a survey, in‐depth  semi‐structured interviews, document  analysis, and observational study of  business students at dalhousie university as  they perform group research tasks. the  recommendations that emerge from this  study could form the basis for best practice  evidence based library and information practice 2006, 1:4  32 guidelines for effective business digital  library design in the academic environment.    literature review    information‐seeking behaviour research    two information‐seeking behaviour models  from the library and information science  literature inform this study and provide the  theoretical foundation.    atkinson provides insight into the  motivations and expectations held by  business information users who seek  information. while it utilizes the key  elements identified in other traditional  information‐seeking models, atkinson’s  “model of business information usersʹ  expectations” (mbiue), addresses the  characteristic needs of users of business  information and their associated optimizing  behaviour. (61) he suggests that the  background and training that business  people undergo in cost‐benefit and break‐ even analyses motivates them to apply this  conceptual model to the search for  information. the business user seeks to  optimize the value of the information  retrieved in relation to cost, time, or effort  expended. atkinson emphasizes the value  of this model to designers of information  retrieval systems and reference librarians.  however, the shortfall of this model is that  it is based on hypothetical relationships and  not on any empirical research. as a result,  few authors have used or cited atkinson’s  mbiue. despite this, atkinson’s model is  one of the few that provides an analysis of  user behaviour specific to business  information users.    allan foster emphasizes the significance of  context to understand information‐seeking  behaviour. (228) foster observed that  “information behaviour is not isolated from  the context within the information seeker  works.” (232) he identified external factors  such as social and organizational, time, the  project, navigational issues, and access to  sources as influencing the information  seeking process. foster’s model, although  based on the study of interdisciplinary  information seekers, provides a concrete  platform for a holistic approach to the  understanding of information‐seeking  behaviour. “the nonlinear model of  information‐seeking,” with its identified  “core processes of opening, orientation and  consolidation”, illustrates the process of  information‐seeking in a way that reflects  the experience of information seekers and  offers applicability to business information  user studies. (foster, 228)  as maureen  mackenzie noted in her study of the  behaviours of line managers in information‐ gathering, ʺmuch of managerial work is not  linear. managers will make rapid decisions,  jump from task to task and bypass a formal  search for information when problem  solving.ʺ (1) this is a useful observation  especially when one seeks to understand the  behaviour of academic business information  users who are in general undergoing  training to be future managers.    digital library research    earlier approaches to the development of  digital libraries have been characterized by a  “build it and they will come” philosophy.  (greensten and thorin) this approach was  viewed as acceptable at the time since most  of the digital library projects were carried  out as “road tests” for bigger projects to  come. recent studies have emphasized the  need to involve the user from the initial  stages of a digital library development.  greensten and thorin noted that “as the  integration of the new technologies begins  to transform the libraries and the  possibilities for constructing innovative  network services, libraries see a pressing  need to engage users and to reassess their  interests and needs.”      evidence based library and information practice 2006, 1:4  33 in the same line of thought, renda and  straccia observed that thus far, digital  libraries have been oriented towards a  generic user and provide no or poor support  to individuals or a defined community of  users. (5) to them, digital libraries “answer  queries crudely rather than, for instance,  learn the long‐term or short‐term  requirements idiosyncratic to a specific user  or, more general, specific to an information  seeking task.” thus, they concluded that,  “the growing diversity of digital libraries  (dl), the communities accessing them, and  how the information is used requires the  next generation of dls to be more effective  at providing information that is tailored to a  personʹs background knowledge, skills,  tasks, and intended use of the information.”   renda and straccia have developed a model  of a personalized collaborative dl. their  model defines a dl “not only as an  information resource where users my  submit queries to satisfy their daily  information need, but also as a collaborative  working and meeting space of people  sharing common interests.” (5)    david nicholas et al. introduced new ways  to characterize and categorize the  information‐seeking behaviour of the  “digital information consumer.” (24)their  study dealt with large populations of web  users from which they developed a  “typology of digital information users or use  behaviour.” they presented a new form of  information‐seeking behaviour which they  characterized as bouncing or flicking. their  results show that “users seldom penetrate a  site to any depth, tend to visit a number of  sites for any given information need, and  seldom return to sites they once visited.”  for this type of user, nicholas et al.  concluded, information providers should  not be thinking of building dedicated  “accredited” systems or gateways. is this  true for business information users or does  this group of users need dedicated,  personalised gateways they can identify  with as business specific for their research?    methods    group study     the data was collected at dalhousie  university in 2005. the study recognized  that business program students at dalhousie  are often required to work in groups or in  teams and thus this collaborative dimension  is fundamental to the design of a digital  library. we were interested in exploring the  physical characteristics of collaborative  information seeking behaviour by studying  students as they performed group research  tasks. thus a qualitative case study was  conducted. as the students were involved in  a highly interactive and contextually  sensitive decision‐making process,  qualitative techniques were determined to  be appropriate for highlighting the themes,  processes, and cognitive behaviours of the  students. (hepworth, 695)  the primary  methods of data gathering were in‐depth  semi‐structured interviews, observational  study and document analysis.     as defined by patton the study sample for  the group study was purposively selected  since the focus was to understand and  illuminate the behaviour of business  students as a case rather than to generalize  from a sample to the general population.  (foster, 228) the authors drew up a  checklist to determine which type of courses  the student sample could be drawn from.  the first requirement was that the students  were enrolled in courses that had group  assignments that required considerable  research and the processing of large  amounts of information using information  technology. groups of students with  different years and levels of academic study  were of interest in order to gather a variety  of perspectives on the behaviours of     evidence based library and information practice 2006, 1:4  34 the business student, thus the inclusion of   students from both undergraduate and  graduate courses. recruitment of the  participants was by requesting for  volunteers in selected business classes (that  met the above requirements). participants  who volunteered to take part in this study  were guaranteed anonymity.    the group study was carried out from  january to april 2005 and followed three  groups of business students working on  group project assignments. the participants  in the group study included ten  undergraduate students in the  commerce/management program and five  graduate students in the mba program. the  undergraduate students worked in two  groups of five while the graduate students  were all in one group. for both the  undergraduate and graduate courses the  project assignment was a mandatory part of  the course and was carried out in 10 weeks,  from the end of january to the first week of  april. the assignments involved  formulating a topic, searching for  information and writing and submitting a  group project report.     interviews were conducted with each group  of students at the beginning and end of the  project. the purpose of these interviews was  to focus on the issues that could not be  addressed through observation only. this  offered the investigators an opportunity to  “explore the experience of the participants  and to elicit, by probing, new themes as they  emerge.” (foster, 228) for example, the  participants were asked about how they  conduct research or share research  information in groups. the interviews were  not tightly structured but used a set of  outlined guidelines and points to ensure all  important factors were captured. (appendix  a) the interviews were tape recorded. the  tape recordings, which were transcribed  verbatim, served as the primary aid to our  memory.   the investigators were present as observers  when each group met to work on the project.  to ensure consistency in observation data  the primary investigator was present at each  of the groups meetings. based on the  insights gained in the first group interviews  an observation item checklist was created.  (appendix b) the purpose of the checklist  was to ensure observers would not miss  important items and also to establish a  certain level of consistency in the items  observed. field notes were taken during  each meeting. also, all emails sent by group  members relating to the project were copied  to the primary investigators.       document analysis    the data from the documents collected in  the group study, i.e. field notes, email  documentation and interview transcripts,  was coded and analysed using qsr n6  qualitative data analysis software. in order  to analyze the documents data a set of tags  or categories were created which would  allow each of the comments or responses to  be coded individually with as many tags as  appropriate. we categorized the data using  atkinson’s “model of business information  users” and renda and straccia‘s  personalized collaborative dl model.  atkinson’s model defines the expectation of  business students in terms of cost, time,  effort required, and pleasure and the  avoidance of pain. renda and straccia’s  model centres around three concepts: actors,  objects, and functionality.  in our study, the  actors were the group of business students at  dalhousie and their community of peers,  professors and experts; the objects were the  information resources used by the students  and the organization of those same  resources; and the functionality was how the  students use, communicate and/or share the  information.  documents were first analysed  in the context of the rest of the same  observation notes, emails or transcripts.  connections between documents were then  evidence based library and information practice 2006, 1:4  35 examined. finally relationships between  and among categories were analysed.    survey     to corroborate the information gained from  the qualitative data and fully understand  the behaviour of the business information  user in an academic setting, the data from  the group study was compared to data  collected from a web‐based survey of full‐ time students enrolled at dalhousie in the  mba and commerce programs. the survey,  which ran from january 5 through march 2,  2005, addressed a number of topics: their  computer experience and use, their  experiences locating and organizing  information, their research skills, their  preferences for potential digital library  features, and demographics. 105 of the 212  students who visited the survey page  completed the questionnaire. five partially  completed surveys were also submitted.   data from the survey was analysed using  the zoomerang software.    results    demographics    105 business students completed the survey  and the group study had 15 participants.  the age of the participants (survey and  group study) ranged from 18 – 45 years. 74%  (n = 89) of all participants were  undergraduates and 26% (n= 31) were  graduates. most of the undergraduates were  between 18 and 25 years of age (95% n = 85),  while 56% (n =17) of graduate students fell  within the same age range.  in terms of  computer experience/use, the majority of the  participants had used computers and the  internet for more than 7 years [92% (n = 110)  and 63% (n = 76) respectively].    most of the graduate students (63%, n = 19)  had received prior library research  instruction. among the undergraduates,  about half indicated that they had received  library research instruction. the instruction  was delivered by librarians or obtained  through self‐guided methods. most of the  participants rated their information research  skills as good, although they sometimes  experienced difficulty finding the  information they need.    actors    the study results show that business  students frequently interact with others  when seeking information. in the survey  and group study, students mentioned that  they often consult with their peers,  professors, teaching assistants (tas), and  experts in relevant fields during the  information seeking process. the document  analysis showed that among these contacts,  peers (classmates) are considered  particularly important (mentioned in 4.5%  of all lines coded). survey results  corroborate this trend as shown in table 1  below.       not important  somewhat  important  important  very important  importance of  peers  2%  13%  38%  47%    table 1. importance of peers when completing assignments  evidence based library and information practice 2006, 1:4  36     graduate  undergraduate  consult your textbook/class notes  16.7%  15.6%  browse the business section of the library stacks   0%  0%  talk to a librarian  0%  0%  talk to your classmates  0%  3.9%  search google (or another similar search engine)  37.5%  63.6%  go to the libraryʹs business subject page  4.2%  1.3%  search the libraryʹs online catalogue (novanet)  12.5%  5.2%  search a business database  20.8%  9.1%  other, please specify  8.3%  1.3%    table 2. starting point for locating information    objects and functionality    as already mentioned above objects in this  study refers to the information resources  used by the students and the organization of  those same resources; and the functionality  denotes how the students use, communicate  and/or share the information.   the strategies for seeking and sharing  information and the resources accessed by  the students showed some interesting  similarities among the participants, as  demonstrated by their preferences for  specific information sources. in the group  study regardless of their status (i.e. graduate,  undergraduate), business students were  most likely to search google (or another  similar search engine) as the initial step in  their information seeking process.     table 2 shows the survey participants’  responses when asked how they would  begin locating information for their topic.  the preferred starting point for both  undergraduate and graduate students is  google or another similar search engine  although it is significantly higher for  undergraduate students. although google  is selected as the starting point by 37.5% of  the graduate students the library and its  resources continue to play a significant role  for this user group. a significant proportion  of the graduate students selected library  related resources as their starting point, i.e.  business databases (20.8%) and the library’s  online catalog (12.5%).textbooks and class  notes are similarly used by both groups.  interestingly neither group talks to  librarians when first seeking information. it  is also important to highlight the  relationship between actors and objects in  the table below by noting that for  undergraduates peers or classmates play a  somewhat important role as a starting point  in their information seeking process.     when asked how they communicate with  their professors and peers, the participants  mentioned e‐mail more than any other  mode in both survey and group study. in  the qualitative data, e‐mail was found in  15% of the lines coded as shown in the  report summary (fig. 1) below.   email and printing were identified as the  main functionality tools used by students to  organize and share the information they  collect during the research process.    evidence based library and information practice 2006, 1:4  37 report summary from qsr6 document analysis      (f 12)                  //free nodes/communication (e‐mail|email|mail)  +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++  +++ total number of text units retrieved = 201  +++ retrievals in 7 out of 7 documents, = 100%.  +++ the documents with retrievals have a total of 1341 text units, so text units retrieved in these  documents = 15%.  +++ all documents have a total of 1341 text units, so text units found in these documents = 15%.      figure 1. report summary     it should also be noted that in 13% of all  document text lines, objects and  functionality were discussed in the context  of collaboration, indicating a noteworthy  relationship among these concepts. survey  data also supports this observation. (see  table 3) it is evident that students strongly  associate the usefulness of an object with  functionalities that facilitate collaboration.  it was evident from both survey and group  study data that librarians continue to play a  significant role in the information seeking   process despite not being perceived as a  useful starting point when looking for  information (table 2), the data presented in       tables 3 & 6 show that librarians are a  valued resource in the digital information  seeking process.  in table 3, “online reference  service that allows you to communicate with a  librarian in real time” was selected as a useful  dl feature by 66% respondents. and table 6  shows that “organized resource collections, (e.g.  by subject, topic, or librarian selections)” were  ranked as very useful by 55% of the students.     cost    cost is a major factor determining the  students’ preference and use of information  sources. this was evident from students’  statements in the group study interviews.     feature  useful – very useful  discussion forums for exchanging information with others  66%  community folders that allow you to share data and saved searches  80%  online reference service that allows you to communicate with a  librarian in real time (e.g., chat/instant messaging)  66%  online meeting rooms that allow you and your members to collaborate  in real time  71%  web‐based collaborative software that allows you and your group  members to read/edit web content  63%    table 3. usefulness of digital library features       evidence based library and information practice 2006, 1:4  38   not important  somewhat  important  important  very important  no fees  associated with  the resource    0%    5%    18%    77%    table 4: importance of cost when selecting resources      for example,    “i use ... as i said, i use proquest  and google, which everyone uses.  i  think google scholar i tried a  couple times cuz i heard from  professor x about it.  i though it  was pretty interesting especially  because it was for free and  proquest, we only use it because  itʹs provided by dalhousie.  if we  had to pay for it, i wouldnʹt use it.   iʹd go for google scholar most.”    from the survey, 77% of the respondents  regarded “no fees associated with a  resource” as a “very important” factor in  their selection of information resources. (see  table 4)    time    for business students, time is an overriding  consideration at all stages of the information  seeking process as revealed in both the  group study and survey results. a transcript  analysis identified inferences to time in most  of the documents. time affects the format  and type of resource used to search for  information, the way in which the  information is disseminated and organized,  and the creation of knowledge. for example,  the students in the group study noted a  number of shortcomings associated with e‐ mail (e.g., slow response times) which  accounted for their continued use of other  communication modes, namely telephone,  chat, and discussion forums. (see figure 2)  effort expended / pleasure and the avoidance of  pain    the amount of effort expended and  perceived ease of use are two important  criteria for students when selecting research  tools as captured in the following quotations.    “to tell you the truth, i always  get what i wanted out of google  without putting the quotes and all  that stuff.”    “...  but it’s not quite as  convenient as [???] or whatever.   you go in to webct, then you  have to go into the forum, then  you have to [???].  i don’t know,  it’s kind of a hassle.  just like  course mail, it’s not as  convenient…”    “a strong and easy to use search  engine. articles linked to search  engine using keywords rather  than only titles. this would make  searching for articles with a  specific theme very fast and easy.”    these sentiments are corroborated by the  survey results. (see table 5). as previously  mentioned, google is preferred by most  participants.  when asked about this  preference, the students explained that  when they try to use library databases, they  often have difficulty identifying appropriate  search terms.  with google, they can enter  the little bit of information that they have  evidence based library and information practice 2006, 1:4  39 0.00 0.50 1.00 1.50 2.00 2.50 3.00 email phone chat discussion forum communication mode p er ce nt ag es (% )     figure 2. modes of communication        not  important  somewhat  important  important  very  important  online availability of an article in  fulltext  0%  5%  28%  67%  ability to access electronic resources  remotely (e.g., from home)  3%  5%  13%  79%  ease of use  0%  9%  28%  63%    table 5. important factors when selecting information research tools      and immediately get numerous relevant  results. below are statements made by two  students.    “i usually need a trigger to get  started.  just something … i don’t  know.  if i’m trying to do a  research paper on something but  it’s kind of a general area, i’ll need  a specific word to trigger it so then  i’ll use the word in a search and  that will get me all the information  i need.”      “or even some place, some kind of  database or something where you  can go and there would be all these  different         main topics that are obvious but  also talk about subjects that talk  about say ethics and business. say  for example i’ve got to do a paper  on some ethical issue in business,  and then maybe there could be  some kind of database that says  advertising but then it goes off  into each … all these ideas about  advertising like about cigarette  like tobacco advertising, or  advertising beer, or you know  what i mean, stuff like  [???]  advertising so you have all these  subcategories so you can kinda  break it down and find the actual  topic.”         evidence based library and information practice 2006, 1:4  40   not useful  somewhat  useful  useful  very  useful  a user interface that you can personalize  according to your preferences  10%  24%  32%  33%  discussion forums for exchanging  information with others   7%  28%  41%  24%  the ability to save/archive your searches  for future use   0%  5%  34%  61%  community folders that allow you to  share data and saved searches   3%  18%  46%  33%  personal folders that allow you to  organize your own information space   1%  4%  24%  71%  an alert service that sends you  information based on your predefined  preferences   4%  23%  40%  33%  organized resource collections, (e.g. by  subject, topic, or librarian selections)   1%  7%  37%  55%  user‐created resource collections (e.g.,  favorites/bookmarks)   1%  13%  44%  42%  online reference service that allows you to  communicate with a librarian in real time  (e.g., chat/instant messaging)   8%  24%  42%  26%  online meeting rooms that allow you and  your members to collaborate in real time   8%  21%  31%  39%  web‐based collaborative software that  allows you and your group members to  read/edit web content   6%  30%  41%  23%    table 6. useful digital library features      useful digital library features    when asked about the usefulness of  potential digital library features, students  overwhelmingly valued having personal  folders and communal virtual spaces that  would allow the students to share  information and communicate in real‐time  with their peers. concerning speed of  delivery and convenience, the organization  of the resources was paramount. students  tended to prefer user‐defined resource lists,  alert services, and expert‐created business  resource collections. some of the dl  properties desirable to business students are  identified in the table below.  discussion    this study reveals that for effective business  digital library design, an understanding of  how the targeted users do their work, how  they use information, and how they create  knowledge is essential. to ensure a dl’s  maximum utilization by business students,  the dl should be built not only as an  “information resource where users may  submit queries to get what they are  searching for, but also a collaborative  working and meeting space” (renda and  straccia, 719.).       evidence based library and information practice 2006, 1:4  41 also, in this study, business user behaviour  concepts of “optimization, cost/benefit and  break‐even analysis” as modeled by  atkinson are affirmed.  similar to  atkinson’s findings, the business students’  information‐seeking behaviour is influenced  by the concepts of cost‐benefit and break‐ even analyses that underlie business  education. the students’ first step in the  information seeking process ‐‐ whether they  realize it’s their first step or not ‐‐ is  identifying the best resources in terms of  cost, effort, convenience, and time. thus, a  business digital library can assist students  by providing tools such as the “community‐ based or social filtering technology” [that]  harness the collective knowledge of all  [participants] to make predictions about  preferences” for their peers who share the  same information needs. examples of such  tools include “multi‐user virtual  environments (i.e., muves, a.k.a. muds  and moos); and user‐profiling and  recommendation engines now used by  amazon.com”. (center for innovation in  learning technologies)    it should also be noted that this study has  pointed to a very different typology of a  digital information user or use behaviour  than the paradigm of the digital information  consumer developed by david nichols et al.,  that of bouncer or flicker, who doesn’t need  dedicated, personalized systems.  for the  business information user, information  providers should think about building  dedicated, personalized systems the user  can identify with as specific to their business  research needs.  providing information and  digital tools tailored to this specific audience  is more likely to increase the appeal and use  of an academic business digital library.    however this study has some limitations.  the focus of this study on undergraduate  and graduate full‐time students means that  the results can not be generalized across all  business academic dl users. studies  exploring faculty and distance learners are  needed to fully validate our findings. also  work task factors in relation to group  members’ information‐seeking behaviour  need to be further explored. more  investigation is needed into the specific  tasks performed as a group versus  individual tasks in the information seeking  process i.e. identifying, analysing, defining  their information problems, searching for  information and retrieving the information.    conclusion    this study emphasizes that when  developing digital libraries user behaviour  and needs should be taken into  consideration. it demonstrates that the  activity as well as the “user’s orientation  and motivation” context (here the business  student training) can directly influence the  design and use of a digital tool. both models,  atkinson’s “model of business information  users” and renda and straccia‘s  personalized collaborative dl model, have  been validated. from this study future  digital library designers have a reference.  the quotation below sums it up perfectly:    “i find the idea of the ʺrealʺ  library very time consuming and  inefficient. i look forward to a  virtual library that not only  facilitates quick and easy access to  information but also team and  peer collaboration.”    works cited     artkinson, joseph. journal of interlibrary  loan, document delivery and  information supply 8.1 (1997): 61‐79.     auster, ethel, and chun wei choo.  ʺenvironmental scanning by ceos in  two canadian industries.ʺ journal of  the american society for information  science 44.4 (1993): 194‐204.   evidence based library and information practice 2006, 1:4  42 ‐‐‐. ʺhow senior managers acquire and use  information in environmental  scanning.ʺ information processing &  management 30.5 (1994): 607‐18.     bilal, dania, and joe kirby. ʺdifference and  similarities in information seeking:  children and adults as web users.ʺ  information processing & management  38.5 (2002): 649.     bouthillier, france. ʺaccess to information  for small business managers:  examination of some strategies and  values.ʺ canadian journal of  information and library science 27.3  (2003): 5.     covi, lisa, and rob kling. ʺorganizational  dimensions of effective digital library  use: closed rational and open natural  systems models.ʺ journal of the  american society for information  science 47.9 (1996): 672‐90.     fidel, raya, et al. ʺa visit to the information  mall: web searching behavior of high  school students.ʺ journal of the  american society for information  science 50.1 (1999): 24‐37.     fisher, karen e., joan c. durrance, and  marian bouch hinton. ʺinformation  grounds and the use of need‐based  services by immigrants in queens,  new york: a context‐based, outcome  evaluation approach.ʺ journal of the  american society for information  science and technology 55.8 (2004):  754‐66.     foster, allen. ʺa nonlinear model of  information‐seeking behavior.ʺ journal  of the american society for  information science and technology  55.3 (2004): 228‐37.     greenstein, daniel, and suzanne e. thorin.  the digital library:  a biography. second edition ed. n.p.:  council on library and information  resources, 2002. . council on library  and information resources. october  2005  <http://www.clir.org/pubs/reports/pub 109/contents.html>.     hepworth, mark. ʺa framework for  understanding user requirements for  an information service: defining the  needs of informal careers.ʺ journal of  the american society for information  science and technology 55.8 (2004):  695‐708.      johnson, j. david e., et al. ʺfields and  pathways: contrasting or  complementary views of information  seeking.ʺ information processing &  management 42.2 (2006): 569‐82.     kuhlthau, carol collier. ʺaccommodating  the userʹs information search process:  challenges for information retrieval  system designers.ʺ american society  for information science.bulletin of the  american society for information  science 25.3 (1999): 12‐6.      lombardo, shawn v., and cynthia e. miree.  ʺcaught in the web: the impact of  library instruction on business  studentsʹ perceptions and use of print  and online resources.ʺ college &  research libraries 64.1 (2003): 6.     mackenzie, maureen l. the accumulation  of information by line managers  within a business environment and the  resulting cognitive savings account.  ph.d long island university, c. w.  post center, 2003 united states ‐‐ new  york: online. proquest august 2005  <http://proquest.umi.com/pqdweb?did http://www.clir.org/pubs/reports/pub http://proquest.umi.com/pqdweb?did evidence based library and information practice 2006, 1:4  43 =765118601&fmt=7&clientid=65345&r qt=309&vname=pqd>.     mckenzie, pamela j. ʺpositioning theory  and the negotiation of information  needs in a clinical midwifery setting.ʺ  journal of the american society for  information science and technology  55.8 (2004): 685‐94.     nicholas, david, paul huntington, and  peter williams: tom dobrowolski. ʺre‐ appraising information‐seeking  behavior in a digital environment:  bouncers, checkers, returnees and the  like.ʺ journal of documentation 60.1  (2004): 24‐43.     renda, m. elena, and umberto straccia. ʺa  personalized collaborative digital  library environment: a model and an  application.ʺ information processing &  management 41.1 (2005): 5‐21.     shih, hung‐pin. ʺextended technology  acceptance model of internet  utilization behavior.ʺ information &  management 41.6 (2004): 719‐21.     song, yoo‐seong. ʺa comparative study on  information‐seeking behaviors of  domestic and international business  students.ʺ research strategies 20.1/2  (2004): 23‐34.      van scoyoc, anna m., and caroline cason.  ʺthe electronic academic library:  undergraduate research behavior in a  library without books.ʺ portal :  libraries and the academy 6.1 (2006):  47‐59.     whitmire, ethelene. ʺdisciplinary  differences and undergraduatesʹ  information‐seeking behavior.ʺ journal  of the american society for  information science and technology  53.8 (2002): 631‐8.   wu, yuhfen diana, and susan lee kendall.  ʺteaching faculty’s perspectives on  business information literacy.ʺ  reference services review vol. 34.no.  1 (2006):  86‐96.     zhang, dongming, et al. ʺuser information‐ seeking behavior in a medical web  portal environment: a preliminary  study.ʺ journal of the american society  for information science and  technology 55.8 (2004): 670.     ʺdigital libraryʺ. the dictionary of human  geography (2000). xreferplus.  september 2006  <http://www.xreferplus.com/entry/734 102>    http://www.xreferplus.com/entry/734 evidence based library and information practice 2006, 1:4  44 appendix a  group interview guide    1.  how do you approach the task of researching on a new topic? (how do you define your    topic in the beginning? how do you draw together ideas?    2.  how would you characterize the approach you take to solve the information problem?    3.  describe for me things that you do to find information? (probe at each step for what you    do/need/feel/think, where you look.)    4.  what resources do you use to find business information? (e.g. databases, library shelves,  web)    5.  what factors play a role in your decision to use various sources?    6.  how do you identify new or useful information sources? (when looking at a range of    sources, how do you decide which ones will be worth using? / when looking at the    results of a search how do you decide which results are relevant?)    7.  what difference do you think there is between working on a research assignment in a    group or individually? if it is different, why do think that is?    8.  what are the biggest barriers/obstacles to gaining access to information?    9.  how would you describe your process of information seeking: is it as clearly defined    stages or as many smaller parts or something else?    10.  how do you communicate with each other? (which media do you use? how do you    collaborate with each other?     11.  what would you recommend to someone starting on a similar assignment to improve his    or her chances of finding relevant information?    12.  age, gender, academic status, degree concentration, academic background.  evidence based library and information practice 2006, 1:4  45 appendix b  group study observation checklist    use of information sources  note which sources they are using and mark  which one they pick as a starting point for getting  information for the project by using the number 1 e.g. google, abi/inform global, books,  librarian, peers (people outside the group) etc    sharing of information/ collaborating?  note whether they email, print, or create web pages, save citations in refworks    communication among them selves or with people outside the group e.g. professors,  librarians or tas?  note whether they email, chat, instant messaging, telephone, in‐person    aim of use of information sources,  note whether it was for clarification, to understand a topic, to explore, or for final information  search    how often are they in contact with other people outside their group?  note frequency and with whom    the information seeking process?  note whether they have defined stages, perform research tasks as a group or they assign each  other research tasks i.e. dividing the research process into individual portions etc.     the use of time, effort expended pleasure?  note how much time they spend on each resource, frustration, joy (indicate words used to  express these feelings)                  article   assessing the fitness of an academic library for doctoral research   susan edwards head, psychology, education and social welfare libraries university of california, berkeley berkeley, california, united states of america email: sedwards@library.berkeley.edu   lynn jones reference coordinator doe library university of california, berkeley berkeley, california, united states of america email:  ljones@library.berkeley.edu     received: 10 march 2014               accepted: 23 may2014      2014 edwards and jones. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – at the university of california, berkeley (ucb), researchers compared how well the library collections supported doctoral research in the three related disciplines of education, psychology, and social welfare. the goal of this project was to gather empirical data to answer questions about materials cited in dissertations, including ownership, age of materials and disciplinary usage.   methods – researchers analyzed the bibliographies of doctoral dissertations from three academic departments at ucb: education (2009-2010), psychology (2009-2010), and social welfare (2009-2011). the sampling methodology used a systematic sample with a random start. to achieve a 95% (+/-4%) confidence interval, the sample included a total of 3,372 citations from 107 dissertations. researchers consulted with a statistician to determine the statistical significance of the results. the test for the age of citation used a signed ranks test, which is typical for ordinal data or skewed interval data. the test for ownership was a chi-square test, which is typical for nominal data or dichotomous data.   results – researchers determined that a very high percentage of the cited journals were owned or licensed by the library. the ownership rate for cited journals was 97% for both education and social welfare, and 99% for psychology. there was a statistically significant difference between the three disciplines, with psychology better supported than either education (p=.02) or social welfare (p=.01). however, since ownership rates for journals in all three disciplines were extremely high, this was not a meaningful difference. for books, the researchers found a significantly smaller percentage of books owned in social welfare compared to either education (p=.00) or psychology (p=.00). we found no significant difference between the percentages of books owned in psychology versus education (p=.27). psychology students cited the highest percentage of journals while education students cited the highest percentage of books. psychology students cited almost no free web resources, but education and social welfare students did cite free web resources (primarily government documents, working papers, or non-governmental organization reports). all three disciplines cited older material than anticipated.   conclusions – the citation analysis, while time-consuming, provided new and important information about the use of the library’s collections and the level of support the collections afford doctoral students in the three related disciplines of education, psychology and social welfare. this data has informed collections-related decisions including format purchases and fund allocations.   introduction   the reputation of research libraries and their collections have long been entwined, with the size of the latter often held as a key indicator of the quality of the former. however, traditional measures of collection strength -dollars spent, number of volumes added, percent of scholarly publishing acquired, range of languages collected, and even circulation, interlibrary loan requests, and usage data-are no longer enough. it is not that, as the star wars character yoda says, "size matters not," (kurtz & kershner, 1980) but that shrinking buying power and a growing need to demonstrate return on investment requires we understand how well the library supports the research of its students and faculty.   university of california – berkeley’s (ucb) state-funded collections budget has not increased since 2001. inflation, the decreasing value of the united states dollar abroad, the rising cost of journals, and the need to handle new formats such as data have increased pressure on collections budgets. this challenging budget situation has made it more important than ever to understand how well the collections support the disciplines, and whether some disciplines, or some format types, have been disproportionately impacted.   citation analysis of dissertations offers a reliable source of data to assess how well the collections meet the needs of doctoral students, a core user group of the library. citing a source in a dissertation indicates that the source was of value to the student; this is a conclusion that cannot be made from usage statistics alone. since education, psychology, and social welfare all offer doctoral programs at berkeley, data derived from dissertation citations allows for meaningful comparisons between these disciplines. a large literature supports the notion that doctoral student research, in addition to being valuable in its own right, serves as a reasonable proxy for faculty research patterns (zipp, 1996).   with the current emphasis on resource access over resource ownership, citation analysis of ownership and licenses may seem anachronistic. the authors are well aware of the importance of access, and understand that no library can meet all the needs of doctoral level researchers. however, decisions must be made about how to allocate funds, both between and within disciplines, and the researchers wanted scholarly practice at the home institution to inform collections assessment and analysis.   aims   the goal of this project was to gather empirical data to answer the following questions:   what percentage of cited material is owned or licensed by the library? does it vary by format and/or discipline? how does berkeley compare to peer research institutions in its level of support for graduate research in these disciplines? what is the ratio of books to journals cited, by discipline? is there an emerging pattern of free web resources being cited, by discipline? how old is the material being cited? does it vary by discipline or format? what are the most frequently cited journals overall, and by discipline?   literature review   a substantial literature exists on the topic of citation analysis, although not all publications in that pool were relevant to our research due to differences in methodology, disciplinary areas, and assumptions. sources used to inform this study primarily focus on dissertation citation analysis in research-level library collections in the social sciences, in particular in the fields of psychology, education, and social welfare. hoffmann & doucette (2012) extract the methodological details of 34 citation analysis studies and compare them. among the variables most commonly analyzed by these studies are: percent of holdings owned or licensed, citation age, frequency of journals cited, citation by title, and type of document cited. the article is quite valuable for those designing a citation analysis study.   the earliest relevant citation analysis research found was the seminal work by peritz and sor (1990). their original findings supporting the use of dissertation citations as a basis for collection decision-making have been replicated and substantiated by many others over the past 23 years. zipp's (1996) research reinforces the claim that the research of graduate students and faculty are sufficiently similar that analysis of dissertations can serve as an adequate proxy for analyzing the bibliographies of faculty publications.   a common goal of citation analysis studies is to identify core journals for selection (or deselection) purposes. waugh and ruppel (2004) summarize reasons why citation analysis of dissertations is one of the most reliable ways to determine journal usage. they also make the point that a list of most heavily used journals can be helpful to a department in several other ways, such as suggesting places for authors to publish, to help weigh the value of a publication in the academic review process, and to help decide which library fund should pay for which journals. wirth and mellinger (2012) present innovative ideas for using citation analysis data to improve online subject guides. thomas's (2000) research on the citation behaviour of 1,024 social work master's students is typical in its research questions, including which journals were cited, and the median age of citations, which new titles should be purchased, and which funds were used to purchase cited materials.   an ongoing controversy in the use of citation analysis of student work is the so-called 'convenience bias'. this is the argument that students will use journals available locally rather than using 'the best' articles which might require going into the stacks or even using interlibrary borrowing (sexton, 2006). this argument was not a concern for the present project for two reasons: 1) the researchers assume that it would be equally true (or not) across the three disciplines we studied, therefore making the comparative analysis still meaningful, and 2) the researchers want to know what types of resources students are using (e.g., the mix of books to journals by discipline, or the age of material cited), even if these are the most convenient materials.   some question the premise that citation analysis of dissertation references is an appropriate basis for collection building (haycock, 2004). beile, boote, and killingsworth (2004) ask, "is it reasonable to conclude… that research collections that contain the majority of cited items are sufficient for doctoral level research?" (p. 348). on the contrary, they claim that basing collection decisions on usage by a single institution's students will lead to a "skewed list of journals" because "doctoral students simply do not possess sufficient knowledge of information resources, expertise in mining the literature of the field, or the ability to… create quality bibliographies" (beile, boote & killingsworth, 2003, p. 12). the researchers agree that there are limitations to the value of citation analysis for deciding which journals to add or cancel, but were not using this data to make individual title decisions.   based on the literature review for this study, there appear to be differences between the citation behaviour of masters' level students and doctoral candidates. for example, when comparing the citation behaviour of faculty and masters level students in biology, pancheshnikov (2007) found that the journals cited by each group are quite different from one another and that faculty cite a broader array of sources than master's students. pancheshnikov concluded that faculty work should be the basis of collection decisions. as noted above, doctoral research more closely matches that of faculty. thus, the current study focuses only on doctoral student dissertations, not a wider range of graduate student work. while there may be disagreement over how well student dissertation research mimics that of faculty, it is the researchers' contention in this article that the doctoral research of graduate students is valuable in its own right and must be supported by research library collections.   methods   the researchers analyzed the bibliographies of all doctoral dissertations from ucb from three disciplines: education (2009-2010), psychology (2009-2010), and social welfare (2009-2011). social welfare has a significantly smaller number of doctoral students (see table 1) so an additional year of data were used to make the sample sizes more comparable. in this study we consulted an evaluator to determine the sampling methodology, and determined that a systematic sample with a random start would be appropriate. to achieve a 95% (+/-4%) confidence interval the researchers sampled a total of 3,372 citations from 107 dissertations. we also consulted a statistician to determine statistical significance of the results. the test for the age of citation used a signed ranks test, which is typical for ordinal data or skewed interval data. the test for ownership was a chi-square test, which is typical for nominal data or dichotomous data.   the researchers gathered lists of students and their dissertations from each department's administrative office. most dissertations were located in the library catalogue or digital dissertations database (http://search.proquest.com/dissertations/). eleven dissertations were unavailable and therefore were not sampled, possibly introducing a sampling bias for which we have not corrected. as of 2009 all of berkeley’s dissertations are available in electronic format only, and students can choose to make them available immediately, or to embargo them for two years or more with justification.   for each discipline the researchers selected a random number from 1 to 5. the citation corresponding to that number, and every fifth citation after, was selected from each bibliography and entered into an excel spreadsheet created for that discipline. we used numbers in place of authors' names in project spreadsheets to identify dissertations in order to preserve the anonymity of graduate students. every sampled citation was listed under the appropriate assigned student number; each citation was also assigned a unique identifying number to allow us to sort and resort the data and still be able to restore the original sequence as needed.            table 1 number of citations in doctoral dissertations by discipline discipline total dissertations sampled number of citations in final sample education 57 1,340 psychology 28 644 social welfare 22 1,388 totals 107 3,372     student library employees conducted preliminary searching of citations in berkeley's online catalogue and subscription databases licensed locally, as well as in resources of the library’s consortial partners to which berkeley’s students have access. they entered ownership or licensing status into the spreadsheets, along with the information for titles which were freely available on the web, or for those not owned or licensed. citations that students were unable to verify were later researched by the authors and added to the spreadsheets.   researchers then analyzed the collected data to determine the following:   the percentage of cited journals and monographs owned or licensed by ucb library median number of citations per dissertation in each department the percentage of books vs. journals cited, sorted by department journal titles ranked by frequency cited, by department the top ten most-cited journals, by department, annotated with ulrich's subject designation the median age of citations (grouped by journals or books) in dissertations in each department. this was calculated as the difference between the year of submission of the dissertation and year of publication of the item cited.   results   our research confirmed that the journals owned or currently licensed by ucb library provided a very high percentage of articles cited by students in all three disciplines. as shown in table 2, the ownership rate for cited journals was 97% for both education and social welfare, and 99% for psychology. there was a statistically significant difference between the three disciplines, meaning it was unlikely to have occurred by chance in the samples, with psychology better supported than either education (p=.02) or social welfare (p=.01). however, since ownership rates for journals in all three disciplines were extremely high, this was not a meaningful difference.   for books, the researchers found a significantly smaller percentage of books owned in social welfare compared to either education (p=.00) or psychology (p=.00) but found no significant difference between the percentages of books owned in psychology versus education (p=.27 ).   the mix of journals to books varied significantly by discipline, with psychology dissertations citing the highest percentage of journals, and education dissertations citing the highest percentage of books. as shown in table 3, psychology students cited almost no free web resources, but education and social welfare students did cite free online resources with some frequency, approximately 7% of their total citations. the free web resources used were primarily government documents or reports from non-governmental or advocacy organizations, while these types of materials were rarely cited by students in psychology students, who overwhelmingly used peer-reviewed articles.   one surprising finding was the age of material students cited, shown in table 4. not only did all three disciplines cite material that was older than anticipated, the assumption that psychology would rely almost exclusively on recent research was not supported. on the contrary, half the sources cited by psychology doctoral students were more than nine years old. in general, education students cited older journals than psychology, but we found no statistically significant difference for books between these two disciplines. social welfare and education students cited older journals than psychology, but surprisingly we found that psychology students cited older books than social welfare. the difference in citation age between education and social welfare was not statistically significant for books or journals.     table 2 journals and books owned or licensed by the library as cited in doctoral dissertations discipline journals books education 97% 86% psychology 99% 87% social welfare 97% 72%     table 3 type of sources cited in doctoral dissertations discipline journals books web sources [government documents, etc.] education 46% 47% 7% psychology 84% 15% <1% social welfare 59% 33% 8%   table 4 median age of citations in doctoral dissertations discipline journals books combined education 11 years 13 years 11 years psychology 8 years 14 years 9 years social welfare 10 years 11 years 10 years     one of the most surprising findings was the strong cross-disciplinary nature of the research in these three disciplines. as shown in table 5, social welfare and education doctoral students frequently cited journals in psychology instead of the high impact journals in their respective fields, while psychology doctoral students relied on journals classified as psychiatry and neurology, and sciences: comprehensive works   according to the ulrichsweb global serials directory (http://ulrichsweb.serialssolutions.com/) subject classifications. the neuroscience orientation of ucb's psychology department was well known to the selector for that area, but the heavy reliance on psychology journals by education and social welfare students was a surprise both to the selector and the faculty who serve as doctoral advisers. this finding has implications for fund allocations and for the importance of collaborative review in serials cancellations projects since the discipline which funds the title may be different from the one(s) that are using it.   discussion   as part of a major research institution, the ucb library has a tradition of outstanding collections and a mandate to support doctoral level research. in an era of rising serial costs and deepening fiscal constraints, it is imperative to analyze how well the collections still support doctoral student research and whether departments were equitably supported. the results show that ucb library met virtually all of the journal demand in psychology, education, and social welfare, the three disciplines being studies. however, the library does not provide as high a percentage of monographs as it does journals. in fact, the data show weaker support for monograph budgets for all three disciplines, which leads the researchers to ask an important question: has the library sacrificed monographs to maintain journal coverage, and is this wise?     table 5 most frequently cited journals in doctoral dissertations by discipline rank education psychology social welfare 1st journal of personality and social psychology neuroimage child development 2nd child development journal of personality and social psychology developmental psychology 3rd journal of educational psychology journal of neuroscience children and youth services review 4th   developmental psychology nature   neuropsychologia   nature neuroscience american psychologist   development and psychopathology 5th   journal of research in science teaching   journal of the learning sciences   reading research quarterly neuron   science   future of children   child abuse & neglect 6th   american educational research journal   applied psychological measurement   educational psychologist journal of neurophysiology   journal of cognitive neuroscience   child welfare   american sociological review   journal of consulting & clinical psychology   pediatrics   social science and medicine note: italicized titles are from outside the discipline.        one of the researchers’ initial goals was to learn how well the ucb library supports the research of doctoral students and how well that compares with library support provided by its peer institutions. the literature review did not reveal data from peer research libraries in these disciplines with a comparable methodology, so we were unable to make this comparison. nevertheless, this data is important for internal benchmarking within ucb library, as one measure for comparing for how well the library supports the research of different departments. we note that 86-87% of monographs used by education and psychology doctoral students are available at ucb library while only 72% of monographs for social welfare are included in the ucb library collection. this discrepancy is prompting meaningful discussions about budgetary equity between disciplines.   with this data, the researchers hope to begin a conversation within our own institution and with peer libraries about how much support of research is enough. what is an acceptable level of doctoral research support? two other findings were significant for collection development decisions. first, graduate students in all three disciplines cited material significantly older than expected. the median age of citations in education and social welfare was about ten years, meaning a full 50% of citations were to material more than ten years old. for psychology, the median age of citations was nine years old. this finding contradicted the conventional wisdom that sciences such as psychology do not cite older materials. with this new information, ucb library's selectors in these disciplines intend to purchase more online back files, and to consider more carefully our deselection decisions for older material.   citations from these three disciplines showed more usage of psychology journals by education and social welfare than we had assumed. three of the top four journals cited by education students and four of the top five journals cited by social welfare students are classified in psychology by ulrich's. decisions to cancel journals in psychology based on the usage behaviour only of psychology graduate students would hamper the research of students in education, social welfare, and perhaps many other departments. this finding demonstrates the need for data from multiple departments and multiple sources in making the best collection development decisions, and has implications for disciplinary budget allocations as well.   next steps   this study produced enough actionable information that the citation analysis will be continued for dissertations in business, economics, political science, and history. this next phase received a research grant to hire students for data entry and a library assistant for bibliographic verification. using the same project design and statistical methodology will enable a comparison between all seven departments.   a careful analysis of the citation formatting in the bibliographies provides a fertile topic for outreach to departments, and possible workshops for graduate students. citation errors were common, in particular how to cite edited works and primary sources. understanding citation errors— both the rate and types of citation errors, which were not tracked in this study— by department or advisor would help librarians design appropriate and targeted instruction.   four immediate uses of this data are planned. the ucb library has begun to purchase additional journal back files in psychology, given the strong reliance on journals versus monographs in that department. some monographic funds are being re-allocated from psychology to social welfare to correct the imbalance found. titles cited but not owned or licensed by ucb library are being evaluated to determine whether they should be acquired, and we are working closely with faculty and students to better understand what types of monographs they need and what areas of the collection need strengthening.   in the next iteration of this research, more detailed information on types of materials cited will be collected, not just monographs, journals, and websites, but also dissertations, news sources, archives, data sources and other genres. tracking changes in what material types are being cited by graduate students provides insight into changing trends in scholarship and local research practice. this project did not assess how well doctoral student citation behaviour matches that of faculty. since there is a controversy in the literature over this point, it would be useful to investigate this correlation for the design of future citation research at ucb library.   conclusion   this pilot project provided new and important information about the use of collections by doctoral students in education, psychology and social welfare ucb. the researchers learned that our journal coverage is excellent in all three disciplines, but that our monographic coverage for all three disciplines is less complete, particularly in social welfare. cross-disciplinary use of journals is greater than expected, and all three disciplines use older materials than assumed. this data will be used to inform budget reallocation, collection development and management.   in addition, designing and conducting a quantitative social sciences research project has made us much more sensitive to the challenges and complexities that face students. our research project has showed us that the literature review, which librarians primarily focus on, is just one element of a much larger research process. becoming deeply engaged with our own research has equipped us to have different and more sophisticated conversations with students, fellow librarians, faculty, and academic deans and chairs about academic research.     references   beile, p. m., boote, d. n., & killingsworth, e. k. (2003). characteristics of education doctoral dissertation references: an inter-institutional analysis of review of literature citations. presented at the american education research association annual conference, chicago, il: eric.   beile, p. m., boote, d. n., & killingsworth, e. k. (2004). a microscope or a mirror?: a question of study validity regarding the use of dissertation citation analysis for evaluating research collections. journal of academic librarianship, 30(5), 347-353. http://dx.doi.org/10.1016/j.acalib.2004.06.001   haycock, l. a. (2004). citation analysis of education dissertations for collection development. library resources and technical services, 48(2), 102-106. http://alcts.metapress.com/content/k47n44/   hoffmann, k., & doucette, l. (2012). a review of citation analysis methodologies for collection management. college & research libraries, 73(4), 321-335. http://crl.acrl.org/   kurtz, g. (producer), & kershner, i. (director). (1980). star wars v: the empire strikes back [motion picture]. united states: lucasfilm.   pancheshnikov, y. (2007). a comparison of literature citations in faculty publications and student theses as indicators of collection use and a background for collection management at a university library. journal of academic librarianship, 33(6), 674–683. http://dx.doi.org/10.1016/j.acalib.2007.09.011   peritz, b. c., & sor, d. (1990). the use of libraries by graduate students in psychology as indicated by citations. collection management, 12(3/4), 11-23. http://dx.doi.org/10.1300/j105v12n03_02   sexton, e. (2006). journal use by graduate students as indicated by master’s theses bibliographies at an urban commuter college, 1991-2004. behavioral & social sciences librarian, 24(2), 93-111. http://dx.doi.org/10.1300/j103v24n02_04   thomas, j. (2000). never enough. behavioral & social sciences librarian, 19(1), 1–16. http://dx.doi.org/10.1300/j103v19n01_01   vallmitjana, n., & sabaté, l. g. (2008). citation analysis of ph.d. dissertation references as a tool for collection management in an academic chemistry library. college & research libraries, 69(1), 72 -82. http://crl.acrl.org/   waugh, c. k., & ruppel, m. (2004). citation analysis of dissertation, thesis, and research paper references in workforce education and development. journal of academic librarianship, 30(4), 276-284. http://dx.doi.org/10.1016/j.acalib.2004.04.003   zipp, l. (1996). thesis and dissertation citations as indicators of faculty research use of university library journal collections. library resources and technical services, 40(4), 335-342. http://alcts.metapress.com/content/k47n44/           news/announcements   call for applications: institute for research design in librarianship      2013. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     the william h. hannon library has received a three-year grant from the institute for museum and library services (imls) to offer a nine-day continuing education opportunity for academic and research librarians. each year 21 librarians will receive instruction in research design and a full year of support to complete a research project at their home institutions. the summer institute for research design in librarianship (irdl) is supplemented with pre-institute learning activities and a personal learning network that provides ongoing mentoring.  the institutes will be held on the campus of loyola marymount university in los angeles, california.  for more information about the project, including the project partners, the school of library and information science at san josé state university and the statewide california electronic library consortium (scelc), please see the project website:  http://irdlonline.org.   the first institute will be held on june 16-26, 2014, with arrival on campus on sunday, june 15, and departure on friday, june 27.   we are now issuing a call for applications for the irdl 2014. we are seeking novice librarian researchers who are employed by academic libraries or research libraries outside an academic setting in the united states to participate in the institute. we define novice researchers as those who may have conducted research but have not yet had an article published or a presentation accepted by a peer-reviewed publication or conference; librarians who have presented peer-reviewed poster sessions will be eligible. librarians of all levels of professional experience are welcome to apply.   we seek librarians with a passion for research and a desire to improve their research skills. the project is designed to bring together all that the literature tells us about the necessary conditions for librarians to conduct valid and reliable research in an institutional setting. the cohort will be chosen from a selective submission process, with an emphasis on enthusiasm for research and diversity from a variety of perspectives, including ethnicity and type and size of library.   the advisory committee, comprised of librarians with research experience in a variety of settings, will use an open review process to select participants who will travel to los angeles, california, participate in all institute activities, and conduct a research project during the 2014-2015 academic year. the online application requests a proposed research project that will be revised at the institute and will be the basis for the study to be completed in the coming year.   selection criteria:   significance of the research problem to the operational success of the applicant’s library or to the profession of librarianship; clarity of the proposed methodology; enthusiasm for research and desire to learn; commitment to the year-long process of communicating with other participants and conducting the proposed study.   we will be accepting applications from december 1, 2013 to february 1, 2014.   scholars accepted to the institute will be notified by march 1, 2014.   please contact project directors with any questions about the institute or the application process: kristine brancolini, dean of the library, loyola marymount university: brancoli@lmu.edu   marie kennedy, serials & electronic resources librarian, loyola marymount university: marie.kennedy@lmu.edu   evidence based library and information practice evidence based library and information practice 2010, 5.3 46 evidence based library and information practice evidence summary college students in an experimental study took longer to achieve comprehension when instant messaging while reading a review of: bowman, l. l., levine, l. e., waite, b. m., & gendron, m. (2010). can students really multitask? an experimental study of instant messaging while reading. computers & education, 54, 927931. reviewed by: megan von isenburg associate director of public services, duke medical center library durham, north carolina, united states of america email: megan.vonisenburg@duke.edu received: 11 may 2010 accepted: 7 july 2010 2010 von isenburg. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. objective – to examine the effects of multitasking while doing school work. the experiment specifically measured total time spent reading a simulated textbook passage and tested comprehension in students who received instant messages before reading, while reading, or not at all. design – experimental design in which one group of students read an online text while receiving and responding to instant messages. comparison groups either received instant messages (ims) prior to reading the text passage or did not receive any ims during the task. setting – general psychology department at central connecticut state university, united states. subjects – eighty-nine college students enrolled in general psychology courses. the participants included 43 women and 46 men and were between 17 and 46 years old. most students were full time students (91%), most were european / white (74%) and in their first (46%) or second (33%) year of college. participants’ academic majors represented all the schools in the university. methods – researchers created a simulated environment in which a passage from a psychology textbook was displayed on five consecutive screens. for the experimental group, an im appeared on each of the five screens preceded by an alert sound. messages were written to reflect the types of questions students might ask each other when they first mailto:megan.vonisenburg@duke.edu evidence based library and information practice 2010, 5.3 47 meet, such as “what do you like to do in your spare time?” subjects were randomized to three situations: receiving ims before reading, receiving ims during reading, or not receiving any ims. subjects were told that they would either receive ims before reading, while reading, or not at all. messages received during reading appeared one per screen after a specified time spent on each page (after 17, 15, 29, 20 and 26 seconds, respectively.) students could take as long as necessary to read the passage and to respond to ims. after reading the passage, students were given a multiple choice test with 25 questions to determine reading comprehension and retention. students also completed a demographic questionnaire to measure their typical instant messaging behaviour, including the amount of time they spend each week instant messaging, how often im software is on when their computer is on, and how often im software is on when they are studying. both of these activities took place on the same computers used for the reading experiment. students were additionally asked to comment on the clarity of instructions, the representativeness of the task to their typical im experiences, and the interest and similarity to normal coursework of the reading itself. these questions were asked on paper rather than on the computer. software recorded the lengths of time each student spent in reading the passage, reading and responding to ims, and answering the online questions. for those students who received ims during reading, the time spent from receipt of each im to each response was subtracted from the total reading time. main results – there were no differences in test performance between the three groups. statistically significant differences were found in the amount of time that students took to complete the reading: students who instant messaged during reading took significantly longer to read the online text than those students who instant messaged before reading and those who did not im, even when time spent receiving and responding to ims was subtracted from the totals. students who instant messaged before reading took the least amount of time in the exercise. further statistical analysis revealed no significant differences in the time spent instant messaging between the two im groups. responses to the demographic questions indicate that students spend a mean 7.5 hours instant messaging per week, that 67% of students have im software on “sometimes,” “often,” or “very often” while the computer is on and 62% of the time while studying. analysis indicated that none of the im use variables were correlated with test performance or reading time and that there were no significant differences between the experimental groups according to prior im use. responses from the 77 students who answered the questions about the experiment itself are also included, though not all of these students answered each question. seventy students (99%) agreed or strongly agreed that instructions were clear. seventy-one percent of the 52 students that received ims agreed or strongly agreed that they were realistic, and 75% agreed or strongly agreed that they responded to ims in a typical manner. sixtytwo students (82%) agreed or strongly agreed that the text was similar to those assigned for actual coursework, and 39 students (51%) agreed or strongly agreed that the passage was interesting. students commented on the authenticity of the experiment in free text responses such as, “i responded how i would have to anyone,” and “they were questions that anyone i don’t know might ask.” conclusion – this experimental study suggests that students who im while reading will perform as well but take longer to complete the task than those who do not im while reading or those students who im before reading. evidence based library and information practice 2010, 5.3 48 commentary the study was well designed and executed, but questions about how closely the experimental setting simulated the real world study environments and ims of college students could impact the applicability of its findings. three possible theories are offered for the finding that the im-during-reading group took the longest to complete the task: that the anticipation of receiving an im was distracting, that there was a psychological refractory period that occurred when students switched between reading and instant messaging, and that students re-read passages after being interrupted by ims. researchers suggest that re-reading passages could have reinforced student understanding of the content and improved test performance for this group. researchers also offer explanations for why the im-prior-to-reading group took the least amount of time to complete the task. they suggest that instant messaging before the reading could serve as a “warm up exercise” that facilitated comfort with the task and the equipment. since each participant was told that ims would arrive before reading, while reading, or not at all, those that got ims at the beginning could also have concentrated on the reading with the knowledge that they would not be interrupted further. there are a number of dissimilarities between the experimental setting and real life. students in the experiment were given unlimited time to read and respond to ims, while in reality, studying is often done in a time-limited context. the types of questions contained in the ims and the frequency with which the ims were sent were also not realistic. while students noted that the questions in the ims were “questions that anyone i don’t know might ask,” it is notable that, due to the closed nature of most im systems, people generally do not get ims from people they do not know. it would be worth investigating if an extended im conversation with a friend or romantic partner would further impact reading time and comprehension. in the real world, students would likely not know that an im is coming. the unpredictable timing and arrival of ims similarly might further impact one’s ability to concentrate on academic reading. participant selection methods are not stated and it is not clear whether students enrolled in this study are representative of the overall student population or whether they knew they would be tested on the material. the study enrolled students in general psychology courses who were given course credit for their participation and it is possible that students sought to perform well on the test since their participation took place in an academic context. it is unclear what the implications of these findings might be in instructional and workplace settings. if classes are time-limited, perhaps multi-tasking students would not be able to achieve comprehension within the time constraints of the class. further distraction research in the classroom and workplace, not just in reading, is warranted. it would be worthwhile to repeat this study in more natural environments with ims that more closely approximate the types of conversations that students might have, in a time-limited context in which students are not being given academic credit for participation. article   assessing the library’s grants program   beth sandore namachchivaya associate university librarian for research, associate dean of libraries and professor university of illinois library urbana, illinois, united stated of america email: sandore@illinois.edu   jamie mcgowan assistant director, global collaborations committee on institutional cooperation (cic) champaign, illinois, united states of america email: jmcgowan@staff.cic   received: 17 mar. 2015   accepted: 13 may 2015        2015 namachchivaya and mcgowan. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – the authors analyzed seven years of sponsored research projects at the university of illinois library at urbana–champaign with the aim of understanding the research trends and themes over that period. the analysis was aimed at identifying areas of future research potential and corresponding support opportunities. goals included developing institutional research themes that intersect with funding priorities, demystifying grant writing and project management through professional development programs, increasing communication about grant successes; and bringing new faculty and academic staff into these processes. the review and analysis has proven valuable for the library’s institutional practices, and this assessment may also inform other institutions’ initiatives with grant-writing.   methods – the authors performed a combination of quantitative and qualitative analyses of the university library’s grant activities that enabled us to accomplish several goals: 1) establish a baseline of data on funded grants; 2) identify motivations for pursuing grants and the obstacles that library professionals face in the process; 3) establish a stronger support structure based on feedback gathered, and through collaborations with other groups that support the research process; and 4) identify strategic research themes that leverage local strengths and address institutional priorities.   conclusions – analysis of library data on externally funded grants from the university’s proposal data system provided insight into the trends, themes, and outliers. informal interviews were carried out with investigators to identify areas where the library could more effectively support those who were pursuing and administering grants in support of research. the assessment revealed the need for the library to support grant efforts as an integral component of the research process introduction   for over a decade the university of illinois library at urbana-champaign has sustained a track record of successful external grant funding. grants support many types of activities, including research by librarians in library and information science and other fields, collection acquisition and processing, preservation, new user service programs, digitization, digital library development, assessment and evaluation, and professional development and training programs. in difficult economic times, libraries rely increasingly on grants to fund innovation and research. the impetus for this assessment study stems from the library’s desire to identify ways to support librarians and professional staff who were successful at garnering grant funds, and to provide incentives and an ongoing support infrastructure that would encourage more librarians and staff to seek grants. this paper describes an analysis of the grants “landscape” in the library and the resulting data helped the library to better support librarians and other professionals to develop successful grants. in conducting this work, we sought answers to several core and thought-provoking questions:   ·         what are the recent funding trends for the university library? ·         what can the university library do to encourage success and minimize obstacles to grant submission? ·         what can the institution do to support success after the award? ·         in what strategic areas could the library expand its grant activities?   in today’s challenging economic climate, faculty and researchers are both motivated and expected to pursue external funding as a means of developing and sustaining institutional research and service functions. as cuillier and stoffle (2011) note, university libraries are no exception, with librarians seeking funding to support a variety of innovative new programs and to perform research. given these professional and economic drivers, libraries are positioned either to initiate or to be partners in grants and sponsored research. beyond a climate in which grant funding is good for the institution, grants support a number of the university of illinois library’s innovations. grant funds incubate initiatives that extend the library’s core activities, projects and programs, and this infusion of support is critical to their success. a 2004 arl spec kit survey (mook, 2004) on grant coordination reported that of 65 respondents, 62 libraries indicated that they pursued grants. roughly half of those 65 libraries reported an increase in grant funding within the previous 5 year period, and 40% reported that they had no change, and 10% reported a decrease in grant funding. further, nearly two thirds of the libraries reporting vested the responsibility for managing grants in the librarians who were the grant’s principal investigator (pi). to this scholarship we introduce a new thread -assessment of grant programs.  this study is unique from the standpoint that it has not been represented in the current literature.   method   in deciding to conduct a baseline evaluation, we were mindful of the value of assessment to our organization and processes.  extending a “culture of assessment” to grant funding is a signal of its importance in the broader scope of library work (lakos and phipps, 2004). at the organizational level, this initial assessment also signals a commitment within the institution and among its leadership to prioritize external funding for evaluation. while it appears that libraries seek grants increasingly to support programs, services and research, the literature revealed scant analysis of grant funding programs in libraries. the average number of grants and level of grant funding at the university of illinois library has risen steadily over the past decade. this trend suggested that grant funding is evolving into a mainstream program area for libraries, which, like other library programs, should clearly be subject to assessment. as lakos and phipps (2004) reiterate, “what gets measured gets managed.” quantitative and qualitative measures enable libraries to target support for individuals in their grant-writing, through enhanced infrastructure, and the development of a culture of institutional research support.   three common themes emerged from the literature on grant–writing and librarianship. first, there are works that are more or less instructional, guiding one through the steps of writing a grant proposal (landau, 2011; herkovic, 2004; zambare, 2004;). a second grouping outlines potential sources of funding (cuillier and stoffle, 2011; taylor, 2010). the third highlights the value of grants for career development (herkovic, 2004). the data analyzed were drawn from a university database that tracks grant proposal information, and from interviews conducted with a librarians and professional staff who are actively engaged in grants that support research and service programs.   the first source of data, from the university’s division of management information proposal data system, provided current and historic proposal data dating back to 1996. using this database, we accessed the university library’s proposal data to provide the primary quantitative data. the data maintained by this database are sponsored research processed by the office of sponsored programs and research administration, and they only represent grants submitted to external entities rather than institutionally based competitions. the database includes information about the status of grant proposals (awarded, declined, and pending), the principal investigators names and affiliations, the title of the proposals, the funder, and the amount of money proposed, awarded, and spent, and the length of the awards.   we initially sought to represent 10 years of grant data. however, the accuracy of the proposal database deteriorated with legacy data from a system migration that occurred eight years ago. hence, we focused on seven years of data, presented here. in analyzing these data, we opted to focus mostly on successful proposals, mapping between the award data, the library as an organization, and more nuanced data about each proposal’s focus or intent.   the second source of data was informal interviews with 10 library faculty who have written and/or are actively writing external grant proposals. the informational interviews offered rich qualitative data that added depth to our quantitative assessment. for instance, interviewees highlighted the professional and institutional value of grants, the context in which such grants emerge, and suggested avenues for improving the grant-writing process.   combined proposal data and interviews provide insights that guide institutional practices – such that the university library is well-placed to develop strategic research initiatives, support initiatives underway, and cultivate grant-writing interests and skills across the library. we present a summary of the quantitative data next followed by the qualitative data. following our analysis, we outline our responses to these findings. again, one of our core goals is to support the development and success of grant-funded initiatives. these steps, assessments, and our initial responses are described in greater detail below.   results: quantitative analysis   at the summary level, librarians and other professional staff in the university of illinois library submitted 146 grant proposals during the past 7 years. there were 85 of these grants awarded, yielding a success rate of 58.2%. the library’s track record of garnering external funding compares favorably with the university of illinois campus, which sustained a 48.4% success rate during the same period. for the library, new proposal success fluctuated from year to year (figure 1). however, when multi-year grants are factored in, the distribution of grants levels out by comparison (figure 2).     figure 1 number of grants awarded     figure 2 grants reflecting multi-year funding, 2004-2011     figure 3 number of awarded grants by sponsor, 2004-2011     figure 4 grant award amounts by sponsor, 2004-2011     on the whole, funding represents a well-balanced blend of sources with the largest number of grants coming from associations (e.g., membership organizations such as the digital library federation (dlf), and the council on library and information resources (clir), and professional associations such as the american library association (ala)), followed by federal agencies, philanthropic foundations, the state of illinois, and other lesser sources.   in contrast, looking at the breakdown of actual funding dollars, foundations and federal monies accounted for the vast majority of the grant monies generated (figure 4). associations, such as the ala, dlf, crl, lama, and the state of illinois offered many smaller grants that totaled 4% of the total amount. special contracts, funding mostly archival initiatives, accounted for a 7% portion of the total.   to get a sense of faculty participation in grant-funded initiatives, we looked at the number of people serving as principal investigators (pi) or co-principal investigators (co-pi) on grants. figure 5 represents these figures for the seven year period. on an annual basis, approximately 10% of librarians and professional staff serve as either a pi or co-pi on grants; however, over time, grant awards go to approximately 30% of the total library professional staff. the data indicated that a small and slowly growing number of librarians were repeatedly successful at getting grants.   through this analysis, we also sought to understand how funder’s strategic agendas influenced programs and research that were initiated with grant support, and how illinois’ institutional strengths were enhanced through programs that were consonant with the library’s and the institution’s strengths. to assess grant focus, we broke down the grant awards into several key categories of interest detailed in figure 6. the professional development and training grants support the university library’s mortenson center for international library programs, which provides training to librarians globally.   a further analysis of the access, management and preservation category reveals that grants were made across the board for several sub-categories of activities related to access, management, and preservation. these sub-categories included technology development, digitization and microfilming, and the specific area of access, management, and preservation. (figure 7)     figure 5 pis and co-pis on grants, 2004-2011     figure 6 funding amounts by grant categories, 2004-2011     figure 7 access, management, preservation grants: funding by sub-categories     to assess funding levels over time, we reviewed the average funding level in each of four categories per year over each of the seven years. the following graph represents average funding levels over time, divided up by the grant focus (figure 8).     figure 8 average funding levels from 2004-2011     these results illustrate the lower levels of funding for collection development as compared to access, management, and preservation or professional development and training. the external state-sponsored collection acquisition grants dwindled to nil by 2009, which reflects the reduction of funds from the lsta (library services and technology act) federal funding that is allocated to states. also the dip in funding in 2010 is striking across most areas of funding. this dip can signal multiple changes.   first, internally, several major grant initiatives ended in 2009. this meant that faculty were actively engaged in wrapping up their commitments to projects in 2009, and they were less involved in writing and submitting new project proposals. second, the global economic crises also led to increased competition for funds, and university of illinois library was one of many institutions competing for reduced federal and foundation dollars. library grant awards were smaller and the number of awards was also reduced. for each of the three years prior to this study, the university library faculty was awarded upwards of ten grant proposals; whereas in 2010, five proposals were funded. last, the number of grant competitions and the size of awards may have also been impacted by the economic crises, as funders had to react to the crises.   equally striking is the bounce-back in average funding in 2011, where the level exceeds previous levels in three of the four categories. the rebound in funding levels was due to a number of continuing grants, as well as an award in 2011 of one substantial grant.   in addition to comparing grant foci and funding over time, funding amounts were assessed from different types of funders over time (figure 9). the analysis revealed that higher funding levels came from federal agencies and philanthropic sources. also, state sources of funding were on the increase, but they were largely curtailed by budget cuts until 2014. associations’ funding support disappeared from the library’s portfolio during this time period.   it appears from these data that philanthropic foundations funded grants at consistently lower levels throughout the past few years of the economic downturn. however funding levels have increased in the past two years, with private foundations providing the library’s highest average funding. also, federal funding fell sharply in 2009 and 2010, but it has in recent years been on the increase. certainly, the funding levels do not reflect funding sources alone. grants coming to a close, application success rates, and levels of funding are primary contributors to funding fluctuations. the variables that lead to these conditions may be internal to the institutions, the competition, or the broader economic crises that led to a contraction of funding opportunities.     figure 9 average funding levels by source, 2004-2011     qualitative analysis   the baseline assessment also incorporated qualitative data obtained from informal interviews conducted with ten librarians and one academic professional at the university library to learn more about their perceptions of library grant-writing, the support provided, and processes. all of these individuals had participated in externally-sponsored grant projects, either as principal investigators, co-principal investigators, or as a part of a team. their comments can be classified into one of three categories – opportunities, challenges and concerns, and needs or issues that were specific to the context of a particular grant. in the cases where needs or issues applied to specific grants, the associate university librarian for research worked with the faculty and staff who expressed concerns to address them.   opportunities   expanding library strategic programs. one of the most frequently reinforced viewpoints articulated by the interviewees was that grant funding provided the opportunity to carry out research and to develop new services, technology, and training programs. in the mortenson center for international library programs, grants support a high percentage of the programs in that unit, supporting librarians world-wide. external funding is essential to the center’s programs, enabling librarians to participate in international collaborations and professional development. in other areas, several of the principal investigators pointed to the expansion of collections, services, access, preservation, cataloging, and technological innovations that resulted from grant funds. a specific example of this development is the “easysearch” locally-developed federated search system that supports searches by title, author, or keyword in a broad selection of freely-available as well as licensed e-resources. a healthy mix of private foundation and federal agency funding has supported the development and use of easysearch as a research tool to increase understanding of user interactions with federated search systems.   sense of accomplishment. another factor mentioned by librarians involved in sponsored research was that they enjoyed the autonomy and the sense of accomplishment that came with crafting and carrying out projects. participants noted, in particular, that faculty status of librarians is important to their role in securing grant funds, and they cited the status in securing external support. the library supports librarians and academic professional staff to initiate research projects that identify and build on institutional strengths. as a result, their grant activities are an important component of their professional identity and career trajectory. one librarian described her grant-funded projects as a “career highlight.”   professional advancement. those who participated in the interviews pointed to professional advancement as another important outcome. sponsored research contributed to skills development, research and publications, and everyone interviewed noted that they were recognized for their grant successes in their annual evaluations and in promotion and tenure reviews. a number of librarians also indicated that grant funding helped them to develop their research agendas in new directions, ranging from new approaches to managing collections to launching projects that resulted in new research findings. one interviewee described a situation where he developed an unsuccessful grant proposal into a case study that resulted in a publication.   enhancing reputation. another positive perspective on grant writing is that funded projects enhance the reputation of the library, on campus, nationally, and internationally. grants can help build awareness and support within and across professional networks, and the outcomes and services reach multiple audiences within those networks as well. grants can provide important services and outreach on campus, and many funded initiatives reach constituencies at other institutions. several of the grantees noted that their grants supported diverse communities including the university, academic, and public libraries, state and local government, k-12 schools, and the media.   positive feedback and community-building. most grants require an evaluation component, and periodic reports that provide useful feedback for the individual as well as the library. in the instance where the reports are publicly available, they increase awareness of the project and enhance the visibility of the institution within and beyond the library community. the data from the evaluation can generate informative baseline information and new tools for ongoing assessment. the university library also benefits from the grants as the funds support positions for visiting staff and students, who have the opportunity to build skills and experience, and to contribute to research, publications and conference presentations. many of the librarians interviewed noted the growth of stronger communities that emerge from the collaboration brought about through grant-supported projects. interviewees indicated that grant project collaborations with library and campus professionals produced positive outcomes. additionally, the processes involved in proposal submission, reporting, and budgeting draws on the expertise of support personnel as well. involving a wider community of library staff in proposal review and project implementation is an important avenue towards building wider professional relationships within the library community.   challenges and concerns   while most of those who were interviewed emphasized positive outcomes, a number of librarians expressed concerns. analysis of these concerns, and the suggestions to remedy them, can help to build successful future outcomes.   balancing grants with primary responsibilities. some who were interviewed expressed the concern that the institutional culture of the library does not promote grant-writing and the associated research. they commented that the pressure of their primary responsibilities detracts from the time available to pursue research. librarians at the university of illinois are required to undergo campus review and evaluation for tenure and promotion. research and publication are required elements of a librarian’s tenure and promotion review. for pre-tenure librarians, the enthusiasm to pursue a grant in support of research is tempered by the high initial effort required to prepare a grant proposal that may or may not be funded.   relationship to library strategic plan. some of the librarians interviewed expressed concern that the library should articulate areas that are priorities for institutional research in the library’s strategic plan. they suggested that the library articulate synergies between strategic directions and institutional research priorities so that librarians and professional staff would have the opportunity to align substantial efforts to obtain grants with strategic library research and development priorities. the authors note that at the time the interviews were conducted, the library was in the process of developing a three-year strategic plan, and these suggestions were considered in that process.    bottlenecks and silos. pre-tenure faculty in particular noted that they encountered bottlenecks in the grant development process that they felt could have been avoided if they had had sufficient access to their expert colleagues and business office staff. this group noted that they expended considerable effort up-front “learning the ropes” of successful grant-writing. they felt unprepared for what seemed to be unpredictable obstacles that occurred in the course of preparing and submitting a grant proposal. budget preparation was an area where most interviewees noted they were required to devote significant time. in particular, many commented that they were not prepared for the requirement to identify sources of “cost-sharing” in order to address an agency’s requirement for matching funds, and noted that this part of budgeting was complicated and time-consuming. yet another challenge articulated by those who were interviewed was the difficulty of identifying more experienced colleagues who could devote time to planning the grant, and reviewing drafts of the proposal narrative at various stages in its development, to provide advice on the impact of the proposed work and the clarity and completeness of the narrative. at the time of the survey, support for grant preparation was limited to the associate university librarian for research, and the research manager in the library’s business office. other colleagues with grant expertise provided advice and support on an informal basis.    internal submission timeline. another concern expressed was that institutional requirements for grant submission did not allow sufficient time for development of the narrative and plan. some grant opportunities have a brief turnaround time between the call for proposals and the submission deadline. the university library and the campus require that both the completed proposal narrative and the budget and submission package are reviewed at each level. this means that the narrative and budget must be completed roughly three weeks before the funder’s submission date. this time frame enables the university library to review the narrative and the budget, to complete required paperwork, and to ensure that any commitments made in the proposal can be supported. the office of sponsored programs reviews proposals to ensure that investigators comply with university regulations, as well as funder requirements. admittedly, there is little that can be done to address the internal review requirements for grant proposals. most proposals require iterative interaction between the pi, the library, and the campus prior to submission to modify the proposal budget and plan of work and to strengthen the narrative, based on feedback from the internal review process.   limited funding options for collections grants and specific research interests. several of those interviewed noted the discontinuation of state grant competitions that funded collection development. these collection enhancement grants, coordinated by the carli (consortium of academic research libraries of illinois) funds, channeled lsta funding to strengthen collections in targeted areas. other interviewees pointed out that funding to support either their collection or research interests is very limited. these barriers hamper individual’s grant submissions.  they also reflect the reality of a sponsored research environment that is driven by funders’ research interests.  while there are numerous opportunities, not all areas of lis research are not considered funding priorities.   discussion   interviewees made several suggestions aimed at better supporting proposal development. they requested that the library sponsor discussion sessions about grant proposal development, where knowledge and experience about grant preparation could be shared widely. they recommended involving successful grantees, who could share their expertise in proposal development. several librarians recommended hosting a two-part series, with one session focusing on cultivation of ideas, planning, and grant submission, while the second session could concentrate on how actual projects were implemented, and strategies for success. senior faculty suggested that working groups, organized around a research interest, could support internal proposal review and might be a rich avenue to pursue for several reasons. this suggestion was aimed at providing assistance with the development of the idea, literature reviews, and reviewing the final proposal. several interviewees noted that they relied upon a pool of experienced colleagues to review their proposals. they developed strong linkages to faculty based in their disciplinary units or with librarians at other campuses. one librarian indicated that she had received feedback from staff in the office of the vice chancellor for research, and attended grant-writing workshops led by an interdisciplinary campus unit. interviewees also suggested that the library provide web-based support for writing grants. finally, those who were interviewed wanted to see their grant projects promoted within the library, on campus, and to other constituents with a potential interest in their research, with press releases and information on the library’s web site. they suggested that this promotion could feature the initiative, itself, or specifically funded activities and outcomes, and information about the research outcomes. faculty felt that showcasing grant accomplishments could raise awareness of the project to a broader audience. those interviewed suggested that the library develop a web site that featured research and grant initiatives.   several key findings emerged from this assessment. historical trends in library grant funding were identified, along with areas where the library is positioned to enhance grant efforts. library faculty and staff identified core organizational issues that were perceived as obstacles to pursuing external funding to support research and innovative service development. the analysis revealed that faculty view grant opportunities as having extraordinary value within their careers and for the institution. finally, this work revealed a need for the library to cultivate an up-and-coming cadre of faculty and professional staff who can transform key research questions into compelling proposals. as part of this effort, several changes were made, including the development of professional forums aimed at faculty and staff who are interested in and ready to pursue external funding, the creation of a blog aimed at recognizing the research accomplishments of library professionals, and the institution of more frequent and consistent communication about grant and research opportunities.   these data support a number of findings. first, as an organization, the library now has a baseline of data about grant challenges and successes. as a result of this analysis the library has a clear idea of the number and thematic scope of grants received annually, as well as their strategic value to the institution. data were generated that describe in detail the breakdown of grants by strategic focus and funder. the library now has a method to assess changes over time that result in successes, and to pinpoint areas in which it ought to pursue future growth. for example, the steady stream of substantial grants awarded to the library’s mortenson center for international library programs to support international leadership training programs served as a strong indicator of the success of the mortenson program in the area of international library leadership training. similarly, several grants have been awarded to support the evaluation of federated search services, which has enabled the library to develop strong expertise in this area. the analysis also enabled us to identify areas where the library could consider seeking external funding to augment existing programs that could be of interest to the broader research library community. two such areas included the assessment of user-focused services, and international reference service.   improving these measures is important to the library, especially as it increases support to librarians who pursue grants to address institutional priorities. the library is reviewing the way it supports grant projects, so that it can enhance the success of future proposals. this assessment is also leading to opportunities that address people’s concerns and obstacles to success. the library implemented an internal review process to provide librarians with timely feedback on grant proposals. the office of the associate university librarian for research worked with the library’s research and publication committee to organize workshops on grant-writing for librarians and professional staff. one workshop involved experienced grant-writers who discussed the positioning of their research to obtain grant funds. a second workshop provided information on how to apply for internal competitive opportunities and introduced other campus resource units that support research. the library also implemented a blog called “recognizing library excellence” that promotes the research of the library’s faculty and professional staff, posting periodic updates on publications, presentations, research grants, and professional awards (recognizing excellence at the university of illinois library).   further strategies for supporting proposal writing include more presentations and web documentation on grant preparation and identification of grants to support strategic needs. two workshops were presented as part of the library’s savvy researcher series for graduate students and faculty, focusing on grant resources and search strategies for identifying funding opportunities. this material was expanded into a libguide on grants, fellowships, and scholarships that presents tools for finding grants and resources for writing successful proposals (grants, fellowships and scholarships libguide).   the issues raised by librarians and staff in the interviews helped to inform daily operations as well as strategic planning. new ideas that are incubated in grant projects have the potential to shape strategic directions. the national science foundation’s digital library initiative phase 1 program spawned numerous creative developments, including the creation of google. areas that are targeted for strategic development, either in a single library or within a large professional organization like the arl, can serve as guideposts for further exploration supported by grant funding.   the library has several long-standing internal competitive grant programs that support research, publication, and innovation, and serve to seed external grant proposals. the library makes available approximately $30,000 annually that is awarded on a competitive basis to librarians in support of research and publication, juried by the research and publication committee. further, the library supports an innovation fund that seeds the development of innovative ideas and programs. the library’s virtual reference system—the only tool that enables management of geographically dispersed virtual reference—was developed with seed funds from the innovation fund. the campus also supports research initiatives with funding for both research and travel, for which librarians are eligible to compete. these funds provide avenues for librarians to develop initiatives that can leverage external funding into large-scale demonstration or research projects. the analysis prompted us to recognize the important bridge role that such a group can play in an organization to assist a researcher in moving from a local idea to an externally-vetted and funded research initiative.   conclusion   as a result of this assessment, the library increased its efforts to provide effective internal support in the proposal preparation process, including help with budgets, support documentation, and the review of grant proposal narratives. several changes were initiated based on the feedback from the data analysis and the interviews. these included: collaboration with the library research and publication committee to develop and offer forums to engage more library professionals in initiating grant proposals; developing workshops through public-facing programs; establishing a library blog that recognizes research and professional accomplishments; developing a libguide that focuses on identifying grant opportunities; and providing reviews of grant proposals prior to submission.    the most important outcome of the assessment was that it revealed the need for the library to support grant efforts as an integral component of the research process.  although it appears obvious in retrospect, the assessment enabled the library to integrate support for grants into a more cohesive research infrastructure than it had previously supported. this evaluation of grants awarded to the library identified trajectories of funding in different areas, and opportunities that grants provide to librarians. it was clear from the interviews that librarians view grants as significant milestones in their research and program-building activities. the feedback from the interviews revealed additional ways to support funded research projects after they are awarded. periodic meetings including the pi and other project staff, the library’s manager for research, and the associate university librarian for research provide opportunities to review progress, confirm or revise goals, and to review the budget and spending rate of the project. participants in the interviews suggested that it was important for the library to recognize the efforts of those engaged in grant activities by communicating systematically the outcomes and successes to a broad audience. the analysis and the interviews also identified areas where the library could stimulate the development of new programs services, or new areas of research. this analysis was a key factor in the library’s decision to re-shape the position description of the library and information science librarian, incorporating substantive responsibilities for research support services into this role. continued monitoring of these data points, and periodic interviews with investigators are ongoing organizational goals.   the review and analysis of the library’s grants program has proven valuable for the library’s institutional practices, and this assessment may also inform other institutions’ initiatives with grant-writing. it can serve as a model to other academic and research libraries interested in two areas: 1) utilizing quantitative methods to understand and track the past and current trends related to research interests and grant funding and 2) using quantitative and qualitative data to design support systems for those in the library seeking grants.   references   cuillier, c., & stoffle, c. j. (2011). finding alternative sources of revenue. journal of library administration, 51(7-8), 777-809. http://dx.doi.org/10.1080/01930826.2011.601276   grants, fellowships and scholarships: find funding support for your research or studies—a guide aimed at faculty, researchers, and students. in libguides @ university of illinois library retrieved 1 june 2015 from: http://uiuc.libguides.com/content.php?pid=334382&search_terms=grants   herkovic, a. (2004). proposals, grants, projects and careers: a strategic view for libraries. library management, 25(8), 376-380. http://dx.doi.org/10.1108/01435120410562862   lakos, a., & phipps, s. (2004). creating a culture of assessment: a catalyst for organizational change. portal: libraries & the academy,4(3), 345-361. retrieved 1 june 2015 from: http://muse.jhu.edu/login?auth=0&type=summary&url=/journals/portal_libraries_and_the_academy/v004/4.3lakos.pdf     landau, h. b. (2011). winning library grants: a game plan. chicago: american library association.   mook, c. (2004). arl spec kit 283: grant coordination. washington, dc: association of research libraries.   “recognizing excellence at the university of illinois library,” retrieved 1 june 2015 from: http://publish.illinois.edu/library-excellence/   taylor, c. (2010). thinking out of the box: fundraising during economic downturns. serials librarian, 59(3-4), 370-383. http://dx.doi.org/10.1080/03615261003623120   zambare, a. (2004). the grant-writing process: a learning experience. college & research libraries news, 65(11), 673-676.   microsoft word es_haddow.doc evidence based library and information practice 2007, 2:2  84 evidence based library and information practice       evidence summary    level 1 counter compliant vendor statistics are a reliable measure of journal  usage    a review of:  duy, joanna and liwen vaughan. “can electronic journal usage data replace citation data as a  measure of journal use? an empirical examination.” the journal of academic  librarianship 32.5 (sept. 2006): 512‐17.    reviewed by:   gaby haddow  senior librarian, humanities, research services, library & information service  curtin university of technology  perth, western australia, australia  e‐mail: g.haddow@curtin.edu.au      received: 27 february 2007  accepted: 01 may 2007      © 2007 haddow. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to identify valid measures of  journal usage by comparing citation data  with print and electronic journal use data.    design – bibliometric study.    setting – large academic library in canada.    subjects – instances of use were collected  from 11 print journals of the american  chemical society (acs), 9 print journals of  the royal society of chemistry (rsc), and  electronic journals in chemistry and  biochemistry from four publishers – acs,  rsc, elsevier, and wiley. acs, elsevier, and  wiley journals in chemistry‐related subject  areas were sampled for journal impact  factors and citations data from the institute  for scientific information (isi).    methods – journal usage data were  collected to determine if an association  existed between: (1) print and electronic  journal use; (2) electronic journal use and  citations to journals by authors from the  university; and (3) electronic journal use and  journal impact factors.     between june 2000 and september 2003,  library staff recorded the re‐shelving of  bound volumes and loose issues of 20  journal titles published by the acs and the  rsc.     electronic journal usage data were collected  for journals published by acs, rsc,  elsevier, and wiley within the isi‐defined  mailto:haddow@curtin.edu.au http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:2  85 chemistry and biochemistry subject area.  data were drawn from the publishers’ level  1 counter compliant usage statistics.  these data equate 1 instance of use with a  user viewing an html or pdf full text  article. the period of data collection varied,  but at least 2.5 years of data were collected  for each publisher.     journal impact factors were collected for all  isi chemistry‐related journals published by  acs, elsevier, and wiley for the year 2001.  library journal utilization reports (purchased  from isi) were used to determine the  number of times researchers at the  university cited journals in the same set of  chemistry‐related journals over the period  1998 to 2002. the authors call this “local  citation data” (512).    the results from electronic journal use were  also analysed for correlation with the total  number of citations, as reported in the  journal citation reports, for each journal in  the sample.    main results – the study found a significant  correlation (p<0.01) between the results for  print journal and electronic journal usage. a  similar finding was reported for correlation  between electronic journal usage data and  local citation data (p<0.01). no significant  association was found between journal  impact factors and electronic journal usage  data. however, when an analysis was  conducted for the total number of citations  to the journals (drawn from the journal  impact factor calculations in journal citation  reports) and electronic journal use,  significant correlations were found for all  publishers’ journals.    conclusion – within the fields of chemistry  and biochemistry, electronic journal usage  data provided by publishers are an equally  valid method of determining journal usage  as print journal re‐shelving data. the results  of the study indicate this association is valid  even when print journal subscriptions have  ceased. local citation data (the citations  made by researchers at the institution being  studied) also provide a valid measure of  journal use when compared with electronic  journal usage results. journal impact factors  should be used with caution when libraries  are making journal collection decisions.    commentary      this study is an excellent example of how a  local project can inform the wider library  community, and encourage further thinking  and research on an issue which is relevant to  many in that community. although not  radically different to previous journal use  studies, the paper brings a range of methods  together and presents the research in a form  that is readable and easily replicated in  other settings. the research question and  study objectives are clearly stated; the  research methods are appropriate and  discussed in sufficient detail; and the  authors’ conclusions draw directly from the  study’s findings, while acknowledging a  number of limitations.    in their introduction, duy and vaughan  discuss different methods of measuring  journal use and the arguments for and  against employing these measures.  questions relating to the accuracy of journal  re‐shelving data are raised, as are the  limitations of using journal impact factors  in collection development decisions. this  study’s findings indicate journal re‐shelving  data are at least as valid a measure as  publisher data for electronic journal use.  two alternative inferences can be drawn  from the study’s results: re‐shelving data  and electronic journal use data are equally  useful methods to determine use; or neither  re‐shelving data nor electronic journal use  data are valid measures of journal use. the  authors’ conclusion supports the former,  stating their results suggest vendor supplied  evidence based library and information practice 2007, 2:2  86 electronic journal usage data can replace  journal re‐shelving studies.    for librarians involved in journal collection  decisions, the finding that level 1  counter compliant statistics are a valid  method to determine electronic journal use  is very good news indeed. it provides a  degree of confidence in using these data for  journal collection decisions and it also lends  weight to the argument that all publishers of  electronic journals conform to this standard.    the finding that local citation data are a  useful measure is encouraging and may  provide an alternative when publisher  usage data are not available or not level 1  counter compliant. it is an alternative  limited by the cost involved in acquiring the  data from isi and in the number of journals  indexed by isi. on the other hand, the  study’s results showed no correlation  between journal impact factors and  electronic journal use data which suggests  that some caution is required when  applying journal impact factors in local  collection decisions.   there is one perplexing aspect to the study.  the findings for the usage of electronic  journals published by the royal society of  chemistry are not reported in 2 of the  paper’s tables, although the authors list this  publisher in the methodology section. this  is a minor quibble in what is an  exceptionally relevant and useful study for  librarians grappling with the complexities of  electronic journal subscriptions and  collection management decisions.     as this study is confined to 1 institution and  a limited number of chemistry‐related  journals, it would be unwise to draw  generalisations from it for application in  practice. further research is needed to  provide a clearer understanding of the most  useful measures of journal usage in different  settings and across a range of subject areas.  for those interested in adding to this body  of knowledge, the study by duy and  vaughan is an excellent place to start.      evidence based library and information practice evidence based library and information practice 2011, 6.1 1 evidence based library and information practice editorial evidence based practice: science? or art? denise koufogiannakis editor-in-chief collections & acquisitions coordinator, university of alberta libraries edmonton, alberta, canada email: denise.koufogiannakis@ualberta.ca 2011 koufogiannakis. this is an open access article distributed under the terms of the creative commonsattribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-ncsa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. evidence based library and information practice (eblip) is a strategy to bridge research and practice. generally eblip is seen as a movement to encourage and give practitioners the means to incorporate research into their practice, where it previously may have been lacking. the widely accepted definition of eblip (booth, 2000) stresses three aspects that contribute to a practice that is evidence based: 1) "the best available evidence;" 2) "moderated by user needs and preferences;" 3) "applied to improve the quality of professional judgements." the area that the eblip movement has focused on is how to create and understand the best available research evidence. ce courses, critical appraisal checklists, and many articles have been written to address a need for librarian education in this area, and it seems that strides have been made. but very little in the eblip literature talks about how we make professional judgements, or moderate evidence based on our user needs and preferences. likewise, how do we make good evidence based decisions when our evidence base is weak. these things seem to be elements we just take for granted or can’t translate into words. it is in keeping with tacit knowledge that librarians just seem to have or acquire skills with education and on the job experience. tacit knowledge is "knowledge that is not easily articulated, and frequently involves knowledge of how to do things. we can infer its existence only by observing behaviour and determining that this sort of knowledge is a precondition for effective performance" (patel, arocha, & kaufman, 1999, p.78). it is something that is difficult to translate into an article or guideline for how we work. i think of this area as the "art" of evidence based practice. and the art is crucial to being an evidence based practitioner. science = systematized knowledge, explicit research, methodological examination, investigation, data art = professional knowledge of your craft, intuition, experience, tacit knowledge, reflection, creativity, values, people-skills mailto:denise.koufogiannakis@ualberta.ca� evidence based library and information practice 2011, 6.1 2 both science and art are important and i hope we do not measure their worth against one another. rather, we need to recognize that both elements contribute to good decision making and practice improvement. without taking research evidence into consideration, we would be lacking scientific knowledge that helps us gain a deeper understanding of our profession. this needs to be continually renewed and paid attention. however, without the art side of the equation, meaning and context may easily be lost. someone can gather all the best evidence, but not know how to effectively implement it. or if you are overly reliant on perfect evidence before you make a decision that is in the best interest of your community, you will likely lag behind and not serve people’s needs. no research study, no matter how perfectly conducted is going to provide all the answers for what we do in practice. we need to embrace both the science and the art of evidence based practice – otherwise, we will overlook important elements of the whole situation that practitioners work within. doing so is not neat and tidy, but does that really matter? lis is a social science, and the "social" implies "messy" because people and real-life situations are not easily controlled. the art of our craft allows us to embrace the messy situation, find ways to be creative, put our professional judgements to use and find the best solutions to meet the needs of individual users by applying the best of what we find in the research literature together with the best of what we know is likely to help this person. my purely unscientific judgement as a practitioner is that lis practice is probably 30% science and 70% art (more or less depending upon the specific topic). as such, i think that we cannot ignore the art of evidence based practice, and in fact should begin thinking about ways to be better artists as well as better scientists. within this journal, the using evidence in practice section brings together the science and art of evidence based practice. in this section, practitioners recount how they used evidence to assist with their practice decisions. they also provide details of non-research evidence or situations that impact their judgements, and at the end of the piece they reflect on the overall process and how their practice may be improved in the future. i hope these experiences inspire readers, and make the connections between science and art more obvious. references booth, a. (2000, july 2-5). librarian heal thyself: evidence based librarianship, useful, practical, desirable? proceedings from the 8th international congress on medical librarianship. london, uk. patel, v.l., arocha, j.f, & kaufman, d.r. (1999). expertise and tacit knowledge in medicine. in r.j. sternberg & j.a. horvath (eds.), tacit knowledge in professional practice: researcher and practitioner perspectives (pp. 75-99). mahwah, nj: lawrence erlbaum. / evidence based library and information practice article   what five minutes in the classroom can do to uncover the basic information literacy skills of your college students: a multiyear assessment study   ma lei hsieh assistant professor-librarian instruction and government documents librarian franklin f. moore library rider university lawrenceville, new jersey, united states of america email: mhsieh@rider.edu   patricia h. dawson associate professor-librarian science librarian franklin f. moore library rider university lawrenceville, new jersey, united states of america email: pdawson@rider.edu   dr. michael t. carlin associate professor psychology department rider university lawrenceville, new jersey, united states of america           email: mcarlin@rider.edu   received: 25 feb. 2013    accepted: 16 july 2013      2013 hsieh, dawson, and carlin. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – librarians at rider university attempted to discern the basic information literacy (il) skills of students over a two year period (2009-2011). this study aims to explore the impact of one-session information literacy instruction on student acquisition of the information literacy skills of identifying information and accessing information using a pretest/posttest design at a single institution. the research questions include: do different student populations (in different class years, honors students, etc.) possess different levels of il? does the frequency of prior il instruction (ili) make a difference? do students improve their il skills after the ili?   methods – the librarians at rider university developed the test instruments over two years and administered them to students attending the ili sessions each semester. the test was given to students as they entered the classroom before the official start-time of the class, and the test was stopped five minutes into the class. a pretest with five questions was developed from the 1st acrl il standards. a few demographic questions were added. this pretest was used in fall 2009. in spring 2010, a second pretest was developed with five questions on the 2nd acrl il standards. students of all class years who attended ili sessions took the pretests. in 2010-2011, the pretest combining the 10 questions used in the previous year was administered to classes taking the required cmp-125 research writing and the bhp-150 honors seminar courses. an identical posttest was given to those classes that returned for a follow-up session. only the scores from students taking both pretests and posttests were used to compare learning outcomes.   results – participants’ basic levels of il skills were relatively low. their skills in identifying needed resources (acrl il standards 1) were higher than those related to information access (acrl il standards 2). freshmen in the honors seminar outperformed all other rider students. no differences were found in different class years or with varying frequencies of prior il training. in 2010-2011, students improved significantly in a few il concepts after the ili, but overall gains were limited.   limitations – many limitations are present in this study, including the challenge of developing ideal test questions and that the pretest was administered to a wide variety of classes. also not all the il concepts in the test were adequately addressed in these sessions. these factors would have affected the results.   conclusions – the results defy a common assumption that students’ levels of il proficiency correlate with their class years and the frequency of prior ili in college. these findings fill a gap in the literature by supporting the anecdote that students do not retain or transfer their il skills in the long term. the results raise an important question as to what can be done to help students more effectively learn and retain il in college. the authors offer strategies to improve instruction and assessment, including experimenting with different pedagogies and creating different posttests for spring 2012.     introduction   rider university, located in lawrenceville, nj, is a private, coeducational university with 5,500 students, offering 69 undergraduate programs in business administration, education, liberal arts, the sciences, fine and performing arts, counseling, and leadership, plus 25 masters level degrees. librarians at the franklin f. moore library (also known as the moore library) have established an active library instruction program, working with teaching faculty to integrate information literacy (il) into their courses for the past decade. following the emphasis placed on assessment by the middle states commission on higher education, the accrediting body for rider university, the moore librarians have been involved in assessment since 2002.   the learning objectives for information literacy are based on the association of college and research libraries (acrl) information literacy competency standards for higher education (american library association, association of college and research libraries, 2000). this study reports on the moore library’s assessment program that measured students’ il levels in two academic years (2009-2010 and 2010-2011) on the first two acrl il standards, which include the same il learning objectives for students at rider university. these objectives contain the basic information literacy competencies and are appropriate for lower division undergraduates:   1.       the information literate student determines the nature and extent of the information needed. students will identify a variety of types and formats of potential sources of information. 2.       the information literate student accesses needed information effectively and efficiently. students will recognize controlled vocabularies; illustrate search statements that incorporate appropriate keywords and synonyms, boolean operators, nesting of terms, and truncation, refining the search statement when necessary; and determine the most appropriate resources for accessing needed information.   research questions   most of the ili sessions occur in the library’s two computer labs. the topics of the assignments and areas of study range widely from business to humanities, social sciences, sciences, and technology. the librarians teach sessions for their liaison departments and share teaching responsibilities for the core research writing and the honors seminar classes.   the current study assessed the knowledge of students of all class years for the first two il objectives in the academic year 2009-2010. for the 2010-2011 academic year, identical pretests and posttests combining these two il objectives were used to assess the impact of ili on students’ learning in the research writing course. our research questions for this study include:   do students’ possess different levels of knowledge and skills for the two il objectives? do different student populations [freshmen (1st year), sophomores (2nd year), juniors (3rd year), seniors (4th year), honors students, and students in different areas of study] possess different levels of il? are students’ performances on the il pretest associated with the frequency of prior ili received? do students improve their il skills after the ili?   literature review   several approaches to assessing undergraduate students’ acquisition of information literacy skills have been documented. this literature review will discuss these different methods used, including the pre and posttest system used by the authors.   small-scale, inexpensive assessment approaches    some assessment techniques that are on a smaller and less expensive scale include the one-minute paper (angelo & cross, 1993; choinski & emanuel, 2006; cunningham, 2006), an attitude survey, observational assessment, a faculty assessment survey (cunningham, 2006), short quizzes given during a one-credit course (hufford, 2010), online tutorials (fain, 2011; heimke & matthies, 2004; johnson, 2009; lechner, 2005; merz & mark, 2002; tronstad, phillips, garcia, & harlow, 2009), interviews (julien & boon, 2004) and, in one case, interviews conducted by anthropologists (kolowich, 2011).   authentic assessment   authentic assessment depends on students’ actual performance on tasks such as an annotated bibliography, submitted research papers, bibliographies, and worksheets as discussed by oakleaf (2011) and brown and kingsley-wilson (2010). rubrics are designed to assess these types of documents and provide a systematic way to determine how well students have achieved the learning objectives. mcculley (2009) and rogers (2001) discuss “reflective learning” where students think about their processes of learning through the use of portfolios and journals as a way to help them become better learners.    standardized testing   other types of assessment include standardized testing formats used to assess baseline competencies among undergraduate students. the educational testing service (ets) and kent state university have developed standardized tests for measuring student information research skills using scenarios: the iskills exam (katz, 2007; katz & macklin, 2007), and project sails, (radcliff, salem, o’connor, burhanna, & gedeon, 2007; rumble & noe, 2009) respectively. these tests allow large-scale aggregation of data amongst many institutions, but can be expensive, time consuming, and would be difficult to use for pretesting and posttesting. however, these tests can be used to assess gains over time to determine trends.   pre and posttest methods   it is evident that the library literature details a wide variety of mechanisms for assessing il skills. moore librarians selected the pretest and posttest method for obtaining a sense of students’ understanding of different resources and search strategy skills and confirm or disprove anecdotal evidence of such skills. these tests are easy to construct using online forms via google docs; they can be administered quickly, and data they generate are easily downloaded and analyzed (hsieh & dawson, 2010). this procedure allows librarians to document students’ il skills and to measure gains over time.   there are instances of the use of pretests and posttests in credit courses. in a one-credit course taught at texas technology university, students were given a pretest at the start of the semester and a posttest at the very end of the semester. both tests were identical and some questions had multiple answers. despite il skills taught over a 14 week period, students did not do as well as expected in the posttest (hufford, 2010). in another example, a three-credit class conducted at the university of rhode island used pretests and posttests to determine student learning of boolean operators (burkhardt, 2007). the librarians were disappointed in the small increase between these two test scores. gandhi (2004) detailed the assessment of a five-session model for library instruction and found students learned more than after a one-shot session. however, this may not be a very practical approach because of the many demands on the librarians’ times. further, faculty members are usually very reluctant to give up class time for such sessions.   about 85 % of 60to 90-minute course-integrated ili sessions taught by the moore librarians at rider university are single-sessions, typical ili offered in many academic libraries in the united states (merz & mark, 2002).  most of the literature reviewed for this paper involves one-shot instruction sessions. a survey developed by librarians at the university of northern texas used a software system that tracked websites used by students during their assessment; four questions were used as a pretest and posttest. this survey demonstrated that ili helped students and provided information on weaknesses such as subject searching in the online catalog (byerly, downey, & ramin, 2006). a review paper describing the pretest/posttest techniques raised concerns about using identical sets of questions for both tests, and the problem with the span of time placed between these two tests as the major factor in determining retention (emmett & emde, 2007). at cornell university, the posttests indicated improvement in il skills but the authors stated that a posttest later in the term would be needed to determine the amount of retention of the material (tancheva, andrews & steinhart, 2007). julien and boon (2004) did just that. the posttest was given immediately after a library session and a post-posttest given to students three to four months later. students showed a decline from the posttest to the post-posttest. this indicates that information is not retained well, and suggests the need for reinforcement of il concepts throughout the semester.   fain (2011) outlined a five-year longitudinal study using pretests and posttests administered by teaching faculty instead of librarians. the author enlisted the help of psychology faculty for the statistical analysis of the data, similar to what moore librarians have done. in addition, the study emphasized the impact of the assignment on teaching information literacy skills, i.e., if journal articles are required but not books, then assessment questions related to using an online catalog or types of books will not be taught by librarians. this would affect the outcomes of any il assessment that asks about resources such as books and skills using the online catalog.   surveys have frequently been used to assess students’ opinions or ask about their satisfaction with il instruction (matthews, 2007). freeman and lynd-balta (2010) and knight (2002) described studies using pretests and posttests to assess students’ confidence levels in information literacy skills. however, these types of studies do not demonstrate students’ knowledge or capability to apply learned il skills. as illustrated by dawson and campbell (2009), computer and information literacy skills are not equivalent. students may exhibit confidence in their search skills because of their familiarity with google and social media. this may be a consequence of students confusing their computer skills with information literacy skills.     method   participants   participants were undergraduate and graduate students at rider university sampled from the fall 2009 semester through the spring semester of 2011. for the first year of the study (academic year 2009-2010), all students attending ili sessions in the moore library computer labs were assessed to establish a baseline il level of all students. in the second year (academic year 2010-2011), instead of testing students in all ili sessions, the librarians narrowed the study population to students in the research writing and the honors seminar courses. this was done because the il objectives matched the courses’ objectives well. in addition, 7 of the research writing instructors requested a follow-up session after the first ili, allowing the use of posttests in 15 classes to determine learning outcomes from their previous ili.   numbers of participants are shown in table 1. participants for the fall 2009 and spring 2010 semesters included all who received ili at the moore library. of these 1,986 students, 560 were freshmen, 310 were sophomores, 420 were juniors, 338 were seniors, 313 were graduate students, and 45 were reported as “other.” students in subsequent cohorts (fall 2010 and spring 2011) included students from core writing courses only. students not in the honors program were enrolled in research writing (cmp125) course. the fall 2010 cohort consisted mainly of sophomores (129 of 177 students), whereas freshmen composed the majority in the spring 2011 cohort (362 of 436 students).   students in the baccalaureate honors program (bhp) have gpas of 3.5 or better. they scored higher in the standardized college entrance exam scholastic aptitude test (sat) with critical reading and math scores of at least 600 and writing score of at least 550 (bhp, n.d.). the honors students typically take the bhp-150 honors seminar in the spring semester during their freshmen year and therefore, almost all (121/122) were freshmen. this group did not receive follow-up sessions.              information literacy instruction sessions   all sessions took place in the computer labs of the moore library and were conducted by one librarian. classes typically included up to 25 students. the basic il concepts in the first two il objectives are applicable in any ili. some concepts might be introduced more thoroughly than others in a session depending on the requirements of the assignments and students’ topics. for example, when an assignment called only for journal articles, librarians would demonstrate searching for articles in selected databases but not searching for books in the online catalog. book sources might be merely mentioned in such a session. on the other hand, some concepts and skills such as search logic, methods of searching for articles using the library subscription databases, and locating the library’s periodicals were emphasized in almost all sessions.   the il content the librarians taught was not limited to the il concepts represented in the tests. for each session, the librarians provided handouts that included a combination of content outlines and step-by-step instructions for their sessions. in summer 2010, the library began subscribing to libguidestm for research guides, with moore librarians gradually developing their instruction content in these online guides and using them in ili. librarians’ teaching styles varied and individual librarians engaged students and faculty differently in their sessions. nevertheless, most ili consisted of lecture, search demonstration, and hands-on time during which librarians monitored and coached students in researching their topics. in the 2010-2011 academic year, of the 20 cmp-125 faculty members, 7 requested a follow-up session after the first ili to allow students more instruction, hands-on time, and coaching from the librarians. as the result, 15 of the 39 classes (38%) had follow-up sessions.     table 1 numbers of students and tests used for each cohort   n test pre1 pre2 post1 post2 fall 2009 1106 a x --- --- --- spring 2010 880 b --- x --- --- spring 2010 honors 55 b --- x --- --- fall 2010 177 ab x x --- --- fall 2010 pre-post* 44 ab x x x x spring 2011 436 ab x x --- --- spring 2011 pre-post* 115 ab x x x x spring 2011 honors 67 ab x x --- --- note: an x denotes administration of the test. pre1 = pretest for objective 1, pre2 = pretest for objective 2, post1 = posttest for objective 1, post2 = posttest for objective 2, test a includes 5 questions for objective 1 in appendix a; test b includes 5 questions for objective 2 in appendix b. pre-post* = preand posttest matching records.     test development   time is at a premium in the ili sessions if both instruction and hands-on time are to be included, therefore, the assessment instrument had to be short and easily accessible to students in the library labs. in fall 2009, an online test with five multiple-choice questions (see appendix a) was developed to assess the il abilities of students on the first il objective – identifying a variety of sources. the test was developed according to the best practices guidelines for generating tests/surveys outlined by radcliff, jensen, salem, kenneth, and gedeon (2007) and by adapting test questions used elsewhere (burkhardt, 2007; goebel & mandeville, 2007; mery, newby, & peng, 2011; schroeder & mashek, 2007; staley, branch, & hewitt, 2010). it was piloted on student workers at the moore library to ensure that the language in the test was clear to college level students. the librarians installed the test online using google docs. in spring 2010, a second test (see appendix b) was developed in the same fashion for the second il objective on constructing search queries.   in the second year of the study, the librarians aimed to measure student learning in the ili sessions by developing identical pretests and posttests. the two sets of questions used in 2009-2010 were combined into a single set of questions used in 2010. following use of this 10-item test in the fall of 2010, the moore librarians shared the fall 2010 results with a group of teaching faculty and received feedback on the test in january 2011. consequently, in spring 2011, the wording of several questions was modified to make them clearer without changing the il concepts assessed. the changes are noted under each question in the appendices.   in addition to the questions regarding il objectives, demographic questions were included on all versions of the firstand second-year tests. these questions included the course number, class year (freshmen, sophomore, junior, senior, graduate student, or other), major area of study (humanities, business, education, science, social science, undeclared, or other), the number of prior library instruction sessions attended (on spring 2010 and later versions), and a four-digit identifying code (e.g., id number) for matching pretests and posttests in the fall 2010 and spring 2011 cohorts.   procedure   as students arrived for their sessions at the library class labs, they were instructed to take the online pretest. students had until five minutes into the scheduled session time to take the test. those arriving after that time did not take the test. students then completed the ili session and departed. in the fall of 2010 and spring of 2011, 7 out of 20 instructors of research writing (cmp125) classes had their classes return for a follow-up library session to receive additional instruction and hands-on time. these students again were tested as they arrived for their session and up to five minutes into the scheduled session time. the time from the first session to the follow-up session varied, but averaged approximately three weeks.   design & analysis   analyses comprised a series of one-way and factorial analyses of variance anovas for the focal hypotheses. the regw-q multiple comparison procedure was used. in addition, chi-squared tests of association and mcnemar’s test were used for analyses involving nominal scale measures. the type i error rate for all tests was .05. eta, a measure of nonlinear relationship, was used as a measure of effect size.    independent variables were class year (freshmen, sophomore, junior, senior, graduate student), area of study (humanities, science, social science, education, business, undeclared, other), course type (core, honors), il objective (1, 2), and test (pretest, posttest). additionally, the number of prior il sessions was used as a correlate of performance for certain cohorts who were asked to report this information. the dependent variable was the number of questions answered correctly (of 5) for each learning objective.   results   in this section, a discussion of the impact of the question revision in spring 2011 leads to the combined data that revealed findings of the pretests in both years. performances on the two il objectives were examined separately. comparisons were made for the cohorts by semester, class year, course (including honors program), major area of study, and frequency of prior ili. following the pretests results were the comparisons of students’ matching pretest and posttest records in fall 2010 and spring 2011. findings include students’ learning outcomes on the whole, by objectives and by questions.     impact of revised questions in spring 2011   after receiving feedback from some class faculty on the test results, six questions were revised (q2, 3, 5, 6, 7, and 10) in spring 2011 to make the questions clearer to students without changing the il concepts (changes are indicated in the appendices under each original question). scores for the original test and revised test were compared. only the questions with scores that varied largely from the previous year were examined for the impact of the revision (see figure 1). only questions 2 and 3 yielded results warranting notice.   the decline in scores by almost 10% in q2 indicated that the change from “a library’s database/index” to “a database such as academic search premier” did not help more students to choose this answer. the reason behind the change was that perhaps some students did not understand the database/index reference, but they might be familiar with “academic search premier.” the lower scores in the revised version indicated that probably fewer participants recognized this database’s name and therefore fewer chose it.   question 3 asked which source one should use to search for background information on an unfamiliar topic. the question changed from “what's the first thing you should do to get started?” to “what's the best way to get an overview of this topic?” the correct answer changed from “find out some basics on watersheds from an encyclopedia” to the same answer but included both print and online versions. the revision moved 16% more participants in the spring to choose “encyclopedia (online or print),” but the majority (52.8%) still chose the web for their answer. the preference of most participants remained the same in both semesters. on the whole, the minor revisions to the questions in spring 2011 had minimal impact on the results because the highest scores received by the participants – q4, q2, q1, q5 in this order, and the lowest scores – q6, q8, q10, were the same in both semesters and not altered by the question revision. for this reason, subsequent analyses combined data from the two versions of the test.   pretest   il objectives. the pretest data revealed that the participants performed significantly better on il objective 1 (q1-q5) than on objective 2 (q6-q10), f(1,629) = 143.01, p < .001, eta = .43. this effect of objective did not interact with year, major, or cohort.                 differences across semesters. for objective 1, there were no significant differences across semesters, (f(2,1784) = 1.23, p = .29, eta = .03) . for objective 2, the spring 2011 cohort (mean (m) = 1.43, sd = 1.05) scored higher than the fall 2010 (m = 1.15, sd = 1.10) and spring 2010 (m = 1.29, standard deviation (sd) = 1.12) cohorts, f(2,1558) = 6.65, p = .001, eta = .09. however, this difference was small (eta = .09) and more a function of the large sample size than a meaningful difference across cohorts. for this reason, cohorts were combined in subsequent analyses.       figure 1 accuracy rates for the original and revised questions of the pretests’ participants.       class year comparisons. forty-six participants listed "other" for class year and therefore were omitted from these analyses. the full sample included 1,001 freshmen, 479 sophomores, 450 juniors, 367 seniors, and 324 graduate students. no differences were present for objective 1, f(4,1745) = 1.02, p = .40, eta = .04, nor for objective 2, f(4,1547) = 1.07, p = .37, eta = .05.   course comparisons. participants were coded as students in an honors course, the research writing course (cmp-125), any other undergraduate course, or a graduate course. for objective 1, honors students scored higher than the undergraduate students, f(3,1780) = 4.14, p = .006, eta = .08. graduate students scored equivalently to all other groups. for objective 2, the honors students scored higher than all groups, f(3,1555) = 16.59, p < .001, eta = .18. in addition, the cmp-125 students scored lower than all other groups. descriptive statistics for the course groups by objective are shown in figure 2 and table 2.   in fall 2010 and spring 2011, those completing both objectives in the pretests included only cmp125 students (n = 611) and honors students (n = 68). the difference in total number correct for the 10 questions was significant, f(1,677) = 26.67, p < .001, eta = .19. the honors group [m (mean) = 5.32, sd (standard deviation) = 1.88] scored higher than the cmp125 group (m = 4.16, sd = 1.75).   comparisons by major area of study. areas of study were self-identified by students but the areas do not necessarily correspond to their schools or their academic departments. for example, psychology and counseling are under the school of education but students in these programs may consider social sciences as their area of study. with the broadly defined areas, most participants of the tests were in business, followed by social sciences and education. the percentages of participants representing sciences, the humanities, undeclared and other were all in the single digits.       figure 2 mean number of correct responses (of 5) for each objective by course. error bars denote 95% confidence intervals.     table 2 number of correct responses for each group for objectives 1 and 2     honors cmp-125 other undergrad graduate objective 1 m 3.37 2.96 2.92 3.13   sd 1.13 1.19 1.20 1.05 objective 2 m 1.89 1.21 1.42 1.48   sd 1.19 1.05 1.02 1.16       there was a significant effect of area of study for objective 1, f(6,1777) = 7.544, p < .001, eta = .16. multiple comparisons indicated that business, undeclared, and other categories scored lower than did the other four groups. there was no difference across majors for objective 2. for the cmp125 students in fall 2010 and spring 2011 who took the tests for both objective 1 and objective 2, there was a significant effect present, f(6,673) = 5.33, p < .001, eta = .21.the humanities and science students scored higher than the business and other students. descriptive statistics are shown in figure 3 and table 3.   frequency of prior ili. the question regarding the frequency of prior ili from rider university first appeared in the test in the spring of 2010 and was included in the following year. from the self-report of the participants, 62% of freshmen, 43% of graduate students and 20% – 30% of undergraduates other than freshmen never had a prior ili session (see figure 4). the number of prior ili sessions was not significantly correlated with any outcome variable (i.e., numbers correct for objective 1, objective 2, or combined).   table 3 number of correct responses by major area of study (n = number)   mean sd n humanities 5.04 1.89 45 science 4.89 2.01 71 business 3.99 1.71 225 education 4.21 1.62 123 social science 4.50 1.84 132 undeclared 4.09 1.66 45 other 3.61 1.70 38     figure 3 mean number correct on the combined pretests by major area of study. error bars denote 95% confidence intervals.       pretest & posttest comparisons   performances of 159 cmp-125 students (44 fall 2010, 115 spring 2011) who took both pretests and posttests, were compared using a 2 (group: fall 2010, spring 2011) x 2 (test: pretest, posttest) x 2 (objective: 1, 2) factorial anova. this analysis found significant differences between the groups, f(1, 157) = 4.43, p = .037, eta = .13, and objectives, f(1, 157) = 311.17, p < .001, eta = .81. the spring 2011 group (m = 4.77, sd = 1.65) scored significantly higher than did the fall 2010 group (m = 4.17, sd = 1.44). participants scored higher on objective 1 (m = 3.20, sd = 0.97) than on objective 2 (m = 1.41, sd = 0.98). there was no change from pretest to posttest.   figure 4 percentage of students with each indicated number of prior ili session by class year.   figure 5 percentage of correct responses for each question on the pretest and posttest 2010-2011.     mcnemar's tests were performed to determine whether accuracy rates varied across tests (pretest vs. posttest) for each question. significant increases in accuracy were seen only for questions 6 (subject search in the catalog, p = .003) and 8 (truncation, p < .001), even though the posttest scores for both questions were still rather low. these increases were evident in both semesters. the differences between the pretests and posttests for the other questions were not significant (see figure 5).   for the first il objective on identifying a variety of sources (q1 to q5), participants performed the best in differentiating scholarly journals from popular magazines (q4). they did relatively well on the purposes of the catalog and the library’s databases (q1 & q2). more than half of the population knew how to find the library’s full-text journals (q5). less than half of the participants would use encyclopedias to search for background information of an unfamiliar topic (q3). most chose to use a web search engine for this purpose.   among the five questions on objective 2, participants performed the best on the combination use of boolean and and or (q7) with around 40% accuracy rate. they received the lowest scores on using subject search to find books on the critiques of shakespeare’s romeo and juliet in the online catalog (q6). more of them would search by title and by keyword, and very few chose the correct answer, “by subject.” the posttest scores of this question improved significantly from 10% to 20%.   the tests revealed further that a great majority of participants were unfamiliar with the use of truncation (q8) and the boolean connector or (q9). very few would consider using books for a reliable and thorough history on a common topic (q10). most chose to use scholarly journals.   discussion   do students’ possess different levels of knowledge and skills for the two il objectives?   participants had higher scores on the first il objective than the second il objective, indicating that although a majority of rider students could identify a variety of sources, few could construct their searches efficiently using these resources.   do different student populations (class years, honors students, area of study) possess different levels of il?   faculty members often assume that students have received their il training before entering their class, and expect students to know how to search for information (kolowich, 2011; lindsay et al., 2006). the finding that upperclassmen performed no differently than their lower division counterparts defies the assumption and raises an important question about the long-term effectiveness of the ili that students receive in and prior to college.   the honors group, composed mainly of second semester freshmen, demonstrated higher il levels than their peers. it is worth noting that two other studies found positive correlations between students’ il levels and their gpas (mcdermott, 2005; silvernail, small, walker, wilson, & wintle, 2008). do these findings suggest that the honors students are efficient in doing research and would not need il training? the average pretest scores of 67% on the first objective and 38% on the second objective for this group suggest that they have ample room for improvement, especially on the second objective (search queries), and could benefit from il instruction. the findings of johnson, anelli, galbraith, and green (2011) agree with those of the present study: their honors students demonstrated the same problems as the others in locating the library’s resources.   as explained earlier, the areas of study were broadly defined and do not necessarily correspond to the participants’ majors. the findings revealed that the humanities and science students scored higher than business students. additional research is needed to determine whether students perform better on discipline-specific il concepts than on general il concepts. one business librarian found no such correlation (campbell, 2011) for her business students. in the future, other researchers may wish to investigate students’ il in relation to their majors.   are students’ performances on the il pretest associated with the frequency of receiving prior library instruction?   participants did report having multiple ili sessions during their undergraduate years at rider university. whereas the majority (63%) of freshmen reported having no prior ili at rider, by senior year nearly 80% had experienced one or multiple ili sessions. the finding of no correlation between the frequency of participants’ prior ili and pretest scores suggests that students might not develop il skills through the “law of exposure” (matthews, 2007). what they learn may not be retained for or transferred to another research experience. julien and boon (2004) reported a similar finding of students not retaining their il learning three or four months after the ili. numerous other studies also came to the same conclusion that it is erroneous to assume that one or more ili sessions in the early college years will prepare students well for higher levels of research work, and students do not become il proficient from the single-session ili (stec, 2006; lechner, 2005; johnson, 2011; mokhtar & majid, 2011; wong & webb, 2011). it can be inferred from these findings that more intensified il training than the current single-session model needs to take place during the four-year period for students to retain basic il skills.   do students improve their il skills after the ili?   the pretest data revealed that the il level of all students was relatively low. the overall differences between the pretests and posttests in the cmp-125 students were not significant. this supports other findings that students’ learning from single-session ili is limited (mokhtar & majid, 2006; portmann & roush, 2004; hsieh & holden, 2010). while the credit il course is not the preferred or the primary way for il training on most college campuses (tancheva et al., 2007), some librarians have embedded themselves in the classroom over a period of time (steiner & madden, 2008), or have actually gained grading power (coulter , clarke, & scamman, 2007). moore librarians need to investigate other means for working more closely with the professors to integrate il into their courses in the future in order to increase the shortand long-term impact of ili.   reflections on constructing test items   analysis of responses to several individual test questions provided significant insight regarding students' knowledge and misconceptions. more participants chose to use “a web search engine for a complete list of references on the topic” instead of “an encyclopedia” to find background information on an unfamiliar topic (q3). the answer for encyclopedia was revised in spring 2011 to include online encyclopedias. as in the previous semester, the majority of participants chose to use the web over an encyclopedia. many faculty members agreed with students and considered a web search engine a better tool than encyclopedias for background information. even though librarians prefer encyclopedia sources for their reliability, it is hard for the library reference sources to compete with the easy access of web search engines. this preference of users, including faculty members, for using the web over traditional reference sources is an established trend that was documented a decade ago in rockman’s (2002) study. considering the development of the web sources over the past decade, the web could be considered acceptable for this question on most topics, if not all.   participants’ accuracy rates for the boolean operators (and/or) in q7 and q9 were in 30% to 40% range. biddix, chung and park (2011) observed that library databases, with their subject, thesaurus and boolean operators search systems, are too complicated and problematic for students. burkhardt (2007) was dissatisfied with students’ improvement on boolean operators over a three-credit il course. the authors of the current study agree with kolowich (2011) that the majority of students do not understand search logic and would have great difficulty finding good sources. to master the basic search logic, students need to learn the operators’ functions correctly and need practice to reinforce their learning.   few students reported they would search for critiques of literary works by subject in the catalog as indicated in q6. rider students were not exceptional when compared with those in the study of byerly et al., (2006) where only 1.6% of students chose the correct answer for the question on subject search. many more studies in the literature also found students’ lack of knowledge on subject or controlled vocabulary (matthews, 2007; brown & krumholz, 2002; lindsay et al., 2006; riddle & hartman, 2000). in discussions with rider faculty members about the question, some of them disagreed with the importance that librarians tend to place on subject searching; they preferred keyword searches instead. as experienced researchers in their fields, these faculty members may know how to use the appropriate keywords to find relevant items with considerable efficiency. they are also motivated to spend the time to sort through large numbers of returns for their studies. but the same knowledge and search mode cannot be expected of most freshmen and sophomores. even though librarians would like to teach students about the concept of subject and controlled vocabulary, various factors, including ingrained personal preferences and habitual research patterns may make learning this concept a challenge.   few participants had prior knowledge about the use of truncation (q8, figure 7). students in other institutions also had trouble with this concept (matthews, 2007; furno & flanagan, 2008). even though participants improved significantly in the posttest, it remains the case that a minority (30%) scored correctly on the question after the ili.   more participants chose journals than books as their source even when books may have been more appropriate (q10). other researchers also noticed that college students are overlooking the value of books and printed materials (rockman, 2002; head, 2012) and do not understand the limitations of scholarly journals (furno & flanagan, 2008; schroeder & mashek, 2007; stec, 2006).   the finding that students were weak in constructing search queries prompted the moore librarians to spend more time explaining the search concepts in the sessions. vocabulary may play a part in students’ understanding. defining terms such as “truncation” or avoiding library jargon may improve students’ search skills. video clips and tutorials were included in the class research guides to help students learn and review the search logic and processes. knowing that those students who had prior ili sessions might not remember or transfer that knowledge in more advanced classes, some librarians used the inquiry method to determine what these upper class students might already know about il. if a majority of students could answer the questions correctly, then the librarians would skip teaching those concepts. the assessment findings also helped librarians work more closely with the class faculty to include il concepts in their assignments.   limitations   moore librarians, in teaching the course integrated one-session ili, face serious limitations and obstacles that the teaching faculty does not. professors in a variety of disciplines request ili and have different requirements for their assignments. further, even though the librarians were aware of the common il objectives for students as well as the items on the tests, they necessarily taught to the assignments, not to the tests. some il concepts received greater emphasis than others during the ili sessions. this lack of uniformity and control by librarians in teaching il is common for the single-session ili in most colleges. it would help if, in the future, librarians record which il concepts they teach in each session. this will allow for more precision in assessment by relating what is taught to what students learn in the sessions.   owing to the limited class time in the session, the test instruments were very brief and used only multiple-choice questions. the number of questions was not large enough to provide a comprehensive picture of students’ il skills. multiple-choice questions are limited when it comes to assessing participants’ higher order thinking skills (oakleaf, 2008). nevertheless, the format is still widely used by researchers and educators because access, data gathering, tabulation, and analysis are comparatively easy (hsieh & holden, 2010). as suggested by suskie (2007) and oakleaf (2010), there is no single instrument that can provide a comprehensive picture of il competency. when practicable, it is best to use multiple instruments– at least enough to supplement the perspective gained from a single assessment tool. these could include students’ reflections on their learning, one-minute papers and librarians’ reviewing of students’ papers to evaluate students’ application of research concepts and methods in actual productions. in addition, performance measures, such as timing how long it takes students to complete searches, may be useful, even though this method would be highly sensitive to individual differences and learning styles.   the tests used in the current research were developed over the course of two years. the participants taking each test were the students in the ili sessions of each semester. there is a certain limitation in making comparisons among the different cohorts because each cohort may have experiences that are different from the other cohorts (furno & flanagan, 2008). nevertheless, the tests on the same concepts over the semesters captured accumulated snapshots of data that revealed the strengths and weaknesses of not only the specific cohorts at specific moments in time, but also over the longer-term, in this case two years.   the librarians found it challenging to develop perfect questions. even though the librarians encouraged students to use reference resources that are considered more reliable than the freely available but, qualitatively, highly inconsistent sources on the web (q3), and they would also like students to learn about subject searching in the catalog because it is an efficient search method (q6), these questions could arguably have more than one correct answer depending on user preferences and topics. the minor revision in the test questions in spring 2011 may have also affected test results albeit the impact seemed insignificant. the authors continue to improve the assessment instrument, including developing questions with multiple correct answers and the opportunity to select multiple responses for each question. these changes will reduce the impact of guessing and increase the psychometric quality of the items. the librarians also intend to develop alternate versions of the test so different pretests and posttests can be employed. this will reduce the effect of memory for prior responses on performance and provide a better estimate of learning.   conclusions   despite the described limitations, the brief tests yielded meaningful information about the students’ il abilities for the moore librarians, allowing for adjustment of il instruction and future assessment. among the findings, the il program reached over 70% of the undergraduates beyond their freshmen year. students’ il levels on the first two il objectives were relatively low, but significantly higher for il objective 1 than for il objective 2. this means their skills in identifying needed resources (acrl il standards 1) were higher than those related to information access (acrl il standards 2). more importantly, students’ basic il levels correlate positively with their academic levels (e.g. the honors group), but not with their class years (e.g. freshman, sophomore) or with the number of prior il instruction they received. the pretests and posttests of the research writing classes revealed that students’ gains over one-session ili were limited. participants did improve significantly after the instruction on two il concepts: truncation and subject searching in the catalog.   the finding that students did not improve their il skills significantly via single-session ili is not new to the literature. on the other hand, the literature contains very few investigations into students’ longer term learning outcomes in il. this study shows participants not demonstrating progress in il despite possibly multiple prior ili over a span of years. this finding suggests that students in all class years (including graduate students) need continued reinforcement of basic il concepts and skills.   the findings raise the important question as to what can be done to help students learn and retain il more effectively in college. more – and multiple – teaching strategies, including a combination of online, face-to-face, embedded librarian, credit course, and curriculum mapping may be considered in the library’s future instruction program as resources are made available. the librarians have also considered other curricular changes such as providing students and/or teaching faculty with answer sheets with rationales for each response for continual learning. in addition, the moore librarians planned with professors in experimenting different pedagogies to reinforce student learning of il in the following semester. one had classes preview il research guides and take a graded quiz before ili, another included interactive learning activities during the ili, and the other had multiple ili and follow-up sessions. comparisons of these approaches will help determine which teaching methods might be the most effective means for helping students improve their il skills.   acknowledgements   the authors wish to thank the instruction librarians at the franklin moore library of rider university for their support and dedication in administering the assessment instruments to students in the information literacy instruction sessions in the past few years. these colleagues include diane campbell, robert congleton, melissa hofmann, katharine holden, robert lackie, marilyn quinn and sharon yang. we also wish to express our appreciation to the moore library assessment committee for the ideas its members offered during the design phase of this research project and for its support throughout, and to colleagues kathy holden for proofreading and to hugh holden for polishing the manuscript.      references   angelo, t. a., & cross, k. p. (1993). classroom assessment techniques: a handbook for college teachers. san francisco, ca: jossey-bass.   american library association. association of college and research libraries. (2000). information literacy competency standards for higher education. retrieved 26 july 2013 from http://www.ala.org/ala/mgrps/divs/acrl/standards/informationliteracycompetency.cfm   bhp basic facts. (n.d.). information for prospective and current students. in rider university. retrieved 26 july 2013, from http://www.rider.edu/academics/academic-programs/honors-programs/baccalaureate-honors-program/bhp-basic-facts   biddix, j. p., chung, c. j., & park, h. w. (2011). convenience or credibility: a study of college student online research behaviors. the internet and higher education, 14(3), 175-182. doi: 10.1016/j.iheduc.2011.01.003   brown, c., & kingsley-wilson, b. (2010). assessing organically: turning an assignment into an assessment. reference services review, 38(4), 536-556. doi:10.1108/00907321011090719   brown, c., & krumholz, l. r. (2002). integrating information literacy into the science curriculum. college & research libraries, 63(2), 111-123. retrieved 26 july 2013 from https://abbynet.sd34.bc.ca/~dereck_dirom/035eee2e-002f4e0d.9/science.pdf   burkhardt, j. m. (2007). assessing library skills: a first step to information literacy. portal: libraries and the academy, 7(1), 25-49. retrieved 26 july 2013 from http://muse.jhu.edu/journals/portal_libraries_and_the_academy/v007/7.1burkhardt.html   byerly, g., downey, a., & ramin, l. (2006). footholds and foundations: setting freshmen on the path to lifelong learning. reference services review, 34(4), 589-598. doi: 10.1108/00907320610716477   campbell, d. k. (2011). broad focus, narrow focus: a look at information literacy across a school of business and within a capstone course. journal of business & finance librarianship, 16(4), 307-325. retrieved 26 july 2013 from http://www.tandfonline.com/doi/full/10.1080/08963568.2011.605622   choinski, e., & emanuel, m. (2006). the one-minute paper and the one-hour class: outcomes assessment for one-shot library instruction. reference services review, 34(1), 148-155. doi: 10.1108/00907320610648824   coulter, p., clarke, s., & scamman, c. (2007). course grade as a measure of the effectiveness of one-shot information literacy instruction. public services quarterly, 3(1), 147-163. doi:10.1300/j295v03n01-08   cunningham, a. (2006). using "ready-to-go" assessment tools to create a year long assessment portfolio and improve instruction. college & undergraduate libraries, 13(2), 75-90. doi:10.1300/j106v13n02_06   dawson, p. h., & campbell, d. k. (2009). driving fast to nowhere on the information highway: a look at a shifting paradigm of literacy in the twenty-first century. in v. b. cvetkovic & r. j. lackie (eds.), teaching generation m: a handbook for librarians and educators (pp. 33-50). new york city, ny: neal-schuman.   emmett, a., & emde, j. (2007). assessing information literacy skills using the acrl standards as a guide. reference services review, 35(2), 210-229. doi:10.1108/00907320710749146   fain, m. (2011). assessing information literacy skills development in first year students: a multi-year study. the journal of academic librarianship, 37(2), 109-119. doi: 10.1016/j.acalib.2011.02.002   freeman, e., & lynd-balta, e. (2010). developing information literacy skills early in an undergraduate curriculum. college teaching, 58(3), 109-115. retrieved 26 july 2013 from eric database. (ej893019).   furno, c., & flanagan, d. (2008), “information literacy: getting the most from your 60 minutes”, reference services review, 36(3), 264-270. doi: 10.1108/00907320810895350   gandhi, s. (2004). faculty-librarian collaboration to assess the effectiveness of a five-session library instruction model. community & junior college libraries, 12(4), 15-48. doi:10.1300/j107v12n04_05   goebel, n., neff, p., & mandeville, a. (2007). assessment within the augustana model of undergraduate discipline-specific information literacy credit courses. public services quarterly, 3(1-2), 165-189.   head, a. j. (2012), “learning curve: how college graduates solve information problems once they join the workplace”. project information literacy research report. retrieved 26 july 2013 from  http://projectinfolit.org/pdfs/pil_fall2012_workplacestudy_fullreport.pdf   heimke, j., & matthies, b. s. (2004). assessing freshmen library skills and attitudes before program development: one library’s experience. college & undergraduate libraries, 11(2), 29-49. doi: 10.1300/j106v11n02_03   hsieh, m. l., & dawson, p. h. (2010). a university’s information literacy assessment program using google docs. brick and click libraries: proceedings of an academic library symposium, 119-128.   hsieh, m. l., & holden, h. a. (2010). the effectiveness of a university’s single-session information literacy instruction. reference services review, 38(3), 458-473. doi: 10.1108/00907321011070937   hufford, j. r. (2010). what are they learning? preand post-assessment surveys for libr 1100, introduction to library research. college & research libraries, 71(2), 139-158. retrieved 26 july 2013 from http://crl.acrl.org/content/71/2/139.full.pdf   johnson, c. m., anelli, c. m., galbraith, b. j., & green, k. a. (2011). information literacy instruction and assessment in an honors college science fundamentals course. college & research libraries, 72(6), 533-547. retrieved 26 july 2013 from http://crl.acrl.org/content/72/6/533.full.pdf+html   johnson, w. g. (2009). developing an information literacy action plan. community & junior college libraries, 15(4), 212-216. doi:10.1080/02763910903269853   julien, h., & boon, s. (2004). assessing instructional outcomes in canadian academic libraries. library & information science research, 26(2), 121-139. doi:10.1016/j.lisr.2004.01.008   katz, i. r. (2007). testing information literacy in digital environments: ets's iskills assessment. information technology & libraries, 26(3), 3-12. doi: 10.6017/ital.v26i3.3271   katz, i. r., & macklin, a. s. (2007). information and communication technology (ict) literacy: integration and assessment in higher education. journal of systemics, cybernetics and informatics, 5(4), 50-55. retrieved 26 july 2013 from http://www.iiisci.org/journal/cv$/sci/pdfs/p890541.pdf   knight, l. a. (2002). the role of assessment in library user education. reference services review, 30(1), 15-24. doi: 10.1108/00907320210416500   kolowich, s. (2011, august 22). what students don't know [web log post], inside higher ed news. retrieved 26 july 2013 from: http://www.insidehighered.com/news/2011/08/22/erial_study_of_student_research_habits_at_illinois_university_libraries_reveals_alarmingly_poor_information_literacy_and_skills   lechner, d. l. (2005). graduate student research instruction: testing an interactive web-based library tutorial for a health sciences database. research strategies, 20(4), 469-481. doi: 10.1016/j.resstr.2006.12.017   lindsay, e. b., cummings, l., johnson, c. m., & scales, b. j. (2006). if you build it, will they learn? assessing online information literacy tutorials. college & research libraries, 67(5), 429-445. retrieved 26 july 2013 from http://crl.acrl.org/content/67/5/429.full.pdf+html   matthews, j. r. (2007). evaluation and measurement of library services. westport, ct: libraries unlimited.   mcculley, c. (2009). mixing and matching: assessing information literacy. communication in information literacy, 3(2), 171-180.   mcdermott, d. (2005). library instruction for high-risk freshmen: evaluating an enrichment program. reference services review, 33(4), 418-437. doi:10.1108/00907320510631553   mery, y., newby, j., & peng, k. (2011). assessing the reliability and validity of locally developed information literacy test items. reference services review, 39(1), 98-122. doi:10.1108/00907321111108141   merz, l. h., & mark, b. l. (2002). clip note 32: assessment in college library instruction programs. chicago, il: association of college and research libraries.   mokhtar, i. a., & majid, s. (2006). teaching information literacy for in-depth knowledge and sustained learning. education for information, 24(1), 31-49.   oakleaf, m. (2008). dangers and opportunities: a conceptual map of information literacy assessment approaches. portal: libraries and the academy, 8(3), 233-253. doi:10.1353/pla.0.0011   oakleaf, m. (2010). the value of academic libraries: a comprehensive research review and report. association of college and research libraries. retrieved 26 july 2013 from http://www.ala.org/ala/mgrps/divs/acrl/issues/value/val_report.pdf oakleaf, m. (2011). are they learning? are we? learning outcomes and the academic library. library quarterly, 81(1), 61-82.   portmann, c.a. & roush, a.j. (2004), “assessing the effects of library instruction”, the journal of academic librarianship, 30(6), 461-465. doi: 10.1016/j.acalib.2004.07.004   radcliff, c., jensen, m. l., salem, j. a., burhanna, k. j., & gedeon, j. a. (2007). a practical guide to information literacy assessment for academic librarians. westport, ct: libraries unlimited.   radcliff, c. j., salem, j. a., o'connor, l. g., burhanna, k. j. & gedeon, j. a. (2007). project sails skill sets for the 2011-2012 academic year [description of standardized assessment of information literacy skills (sails) test]. kent state university. retrieved 26 july 2013, from https://www.projectsails.org/skillsets   riddle, j. s., & hartman, k. a. (2000). but are they learning anything? designing and assessment of first year library instruction. college & undergraduate libraries, 7(2), 59-69. doi: 10.1300/j106v07n02_06   rockman, i. f. (2002). strengthening connections between information literacy, general education, and assessment efforts. library trends, 51(2), 185-198.   rogers, r. r. (2001). reflection in higher education: a concept analysis. innovative higher education, 26(1), 37-57. doi:10.1023/a:1010986404527   rumble, j., & noe, n. (2009). project sails: launching information literacy assessment across university waters. technical services quarterly, 26(4), 287-298. doi:10.1080/07317130802678936   schroeder, r., & mashek, k. b. (2007). building a case for the teaching library: using a culture of assessment to reassure converted campus partners while persuading the reluctant. public services quarterly, 3(1-2), 83-110.   silvernail, d. l., small, d., walker, l., wilson, r. l., & wintle, s. e. (2008). using technology in helping students achieve 21st century skills: a pilot study. center for education policy, applied research, and evaluation.   staley, s. m., branch, n. a., & hewitt, t. l. (2010, september). standardized library instruction assessment: an institution-specific approach. information research: an international electronic journal, 15(3), 1-22. retrieved 29 july 2013 from http://www.eric.ed.gov/pdfs/ej912761.pdf   stec, e. m. (2006). using best practices: librarians, graduate students and instruction. reference services review, 34(1), 97-116. doi:10.1108/00907320610648798   steiner, s. k., & madden, m. l. (eds.). (2008). the desk and beyond: next generation reference services. chicago, il: association of college and research libraries.   suskie, l. (2007). answering the complex question of “how good is good enough?” assessment update, 19(4), 1-2, 12-13. doi:10.1002/au.194   tancheva, k., andrews, c., & steinhart, g. (2007). library instruction assessment in academic libraries. public services quarterly, 3(1-2), 29-56. doi:10.1300/j295v03n01_03   tronstad, b., phillips, l., garcia, j., & harlow, m. a. (2009). assessing the tip online information literacy tutorial. reference services review, 37(1), 54-64. doi:10.1108/00907320910934995   wong, s. h. r., & webb, t. d. (2011). uncovering meaningful correlation between student academic performance and library material usage. college & research libraries, 72(4), 361-370. retrieved 29 july 2013 from http://crl.acrl.org/content/72/4/361.abstract     appendix a   test questions for il objective 1 (questions 1 5) [correct answers are italicized]   1. typically a library's online catalog contains: a. information about books, videos, and other nonprint items in the library b. the complete text of the journal articles in the library c. information about the college's courses d. full-text books e. don’t know   2. which of the following would be the best tool to use to obtain journal articles for your topic “autistic children”? a. the library’s online catalog b. a library’s database/index c. an encyclopedia d. google e. don’t know   * answer b changed to “a database such as academic search premier” in spring 2011.   3.  you have gotten an assignment on “watersheds” which you know very little about. what's the first thing you should do to get started? a. browse the library shelves for books on watersheds. b. type “watersheds” in a web search engine for a complete list of references on the topic. c. ask your friends if any of them know about your topic. d. find out some basics on watersheds from an encyclopedia. e. ask the professor if you can change topics.           * changed the question to: you have gotten an assignment on “watersheds” about which you know very little. what's the best way to get an overview of this topic? * answer d changed to “find out some basics on watersheds from an encyclopedia (online or print)” in spring 2011.   4.  which of the following are characteristics of scholarly journals? a. contain colorful, glossy pages and typically accept commercial advertising. b. mainly for the general public to read. c. report news events in a timely manner. d. articles include detailed references. e. don’t know.     5.  what is the easiest way to find out if the library has the 1998 issues of journal of communication? a. search the library’s periodical shelves. b. search “journal holdings” on the library web page. c. search google scholar. d. search noodlebib. e. don’t know.   * question changed to “what is the best way to find out if the rider university libraries have the full text articles of the ….”     appendix b   test questions for il objective 2 (questions 6 10) [correct answers are italicized]   to find the critiques on william shakespeare’s play romeo and juliet, in the online catalog, i would do a search: a. by title b. by keyword c. by subject d. by author e. don’t know   * question changed to “what is an efficient way to find critiques on william shakespeare’s …” in s6.   7.  which is the correct search strategy to combine terms with the operators (and, or)? a. death penalty or capital punishment and women b. death penalty or (capital punishment and women) c. (death penalty or capital punishment) and women d. (death penalty and women) or capital punishment” e. i don't know   * question changed to “which search statement is correct when you search for information on the topic ‘should women be exempt from death penalty?’” in spring2011.   8.  truncation is a library computer-searching term meaning that the last letter or letters of a word are substituted with a symbol, such as “*” or “$”. a good reason you might truncate a search term such as child* is that truncation will a. limit the search to descriptor or subject fields b. reduce the number of irrelevant citations c. yield more citations d. save time in typing a long word e. i don't know   9.  in order to find more documents on my topic i can include synonyms in my search statement. to connect those synonyms in my statement, i use: a. and b. + c. not d. or e. i don’t know   10.  choose the best place to find a reliable and detailed history of television in the us for a research paper. a. book b. website c. magazine/newspaper d. scholarly journal e. i don’t know   *question changed to “choose the best place to find a reliable and thorough history ….” in spring 2011.         evidence based library and information practice commentary   the librarian as practitioner/researcher: a discussion   lisa r. horowitz assessment librarian massachusetts institute of technology libraries cambridge, massachusetts, united states of america email: lisah@mit.edu   jason martin head of public services dupont-ball library, stetson university deland, florida, united states of america email: jmartin2@stetson.edu   received: 26 aug. 2013 accepted: 28 aug. 2013      2013 horowitz and martin. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     introduction   at the american library association’s (ala) 2013 annual conference, the evidence based practices discussion group (ebpdg), part of acrl’s university libraries section, met to discuss the state of research in librarianship. while the focus of the ebpdg is on informing practice with research, previous discussions revealed a dearth of good research in the literature and a lack of understanding of research methodology and practice amongst librarians. in order to address these shortcomings, the ebpdg invited three guests to answer pre-set questions for the first 30 minutes of the discussion and then opened the discussion to all the attendees for the last hour. the invited guests were: john budd, professor at the school of information & learning technologies, university of missouri sharon weiner, information literacy librarian at purdue university libraries nancy adams, health sciences librarian at penn state hershey george t. harrell health sciences library   these guests provided the perspectives of both practicing librarians and library school faculty, as practitioners who use research to influence practice, and as researchers for whom practice drives their research agenda. all three are prolific researchers. lisa horowitz (massachusetts institute of technology) and jason martin (stetson university) moderated the discussion.   research: choosing, practicing, and improving   for many librarian practitioners, the first obstacle to overcome in research is how to select a topic. the discussion’s guests provided several ways for librarians to unearth and choose research topics. one way is to see puzzles and patterns of interest in the environment. study the day-to-day problems of your library and look for topics that will not only better your organization, but that will apply, and be of interest, to more than your institution. research topics also spring from a librarian’s strong professional interest and innate curiosity, and from systematic, logical research to identify gaps in the published research. librarians also must be aware of what research will be funded. research funding is ever more crucial in these times of smaller budgets and limited resources.   for many librarians, a disconnect exists between the research performed by library school faculty and practical problems of the profession. bridging this gap is important for the librarian practitioners because it aids directly in developing evidence based practice. the best remedy for this disconnect is collaboration between library school faculty and librarian practitioners so that they may explore questions together. john budd regularly attends practitioner conferences like ala because he is interested in how librarians are trying to solve problems. librarian practitioners and library school faculty must listen to each other, and practitioners should reach out to teaching faculty in nearby library schools, talk to them about their interests, and work together with them.   to improve research, librarians must start with research in mind. too often, a librarian collects data to solve a problem or to answer a question, and then decides to write and publish an article. this leads to a plethora of “this is how we do it in our library” articles, which may or may not be helpful to others in the profession. next, librarians need to learn about, and improve upon, research methodology. sharon weiner has an online course, “practice into research: research into practice” available from the purdue university libraries website at http://www.lib.purdue.edu/infolit/practice. this self-paced course helps librarian practitioners understand the basic concepts of sound research. also suggested were workshops on evidence based practice and classes on qualitative and quantitative methodology. replication of published, well-executed research studies also helps to improve research skills while minimizing preparation time. nancy adams suggested the need for librarians to be familiar with the “pyramid of evidence,” (figure 1) and how research studies gain more internal and external validity as one moves up the pyramid.   sadly, library school students are not being taught the fundamentals of research. research methods courses are offered in library schools but are not universally required. instead, library schools prepare students to be consumers of research, which is not unreasonable given the work they will be doing. even non-researcher librarians, however, need to be able to understand research, to apply research results, to conduct literature reviews, and to understand what correct research methodologies are, in order to engage in evidence based practice. at the same time, without quality research, evidence based practice is not possible. the evidence used to inform decisions must be valid and reliable; therefore solid approaches to research are needed. good research starts with a problem statement or a question, and not with data. meaning is found in the analysis of data gathered to answer a question. in order to develop an informed research question, librarians must be well-read in a topic, which means reading until encountering repetition of citations and ideas. librarians should also read more broadly, outside of the library literature, in fields such as management, organizational design, and information technology. this broad reading can provide the groundwork for research in librarianship. in fact, simply reading about other study designs can actually stimulate research ideas. being well-read ties back to choosing a topic since it allows a librarian to be aware of the gaps in the research.       figure 1 pyramid of evidence (information systems department, 2006)       guests and audience members raised two philosophical conundrums: how do libraries balance the need to be nimble and innovative with the need to engage in thoughtful research? and, how do librarians view themselves? librarians who view themselves as practitioner/researchers are more involved in and concerned about research than librarians who view themselves as practitioners who publish to receive promotions or other extrinsic rewards.   the role of theory   a large part of the open discussion was spent discussing theory, both why to use it, and how to find one. librarianship is an atheoretical profession, but theory is important to librarianship. practitioners may seem annoyed by theory, but in reality every one of us uses theory; we are just not conscious of it. theory provides a framework that guides the research and gives it more credibility. theory allows the researcher to step outside of the profession and examine it critically. librarian practitioners must learn the importance of theory.   finding a theoretical framework involves reading the literature and, again, not just library literature. most research articles use a theoretical framework, and one can simply search in a database for “theoretical framework” or use a resource such as the book by leckie, given and buschman (2010).   one way of incorporating theory was discussed at length the potential use of action research in librarianship. action research offers the ability to ground your research in the needs of your own institution. it pulls together research and theory and may be a more viable model for librarian practitioners. in action research, the researcher intervenes in the research problem and aims to change what exists, rather than simply studying and writing about what exists. action research involves getting people to think and act differently, including the researcher. it takes formally structured research and moves it into policy and applicability and then into local institutions. in other words, action research puts analysis into action to solve a problem. action research takes into account the agency of all those affected by the research problem and any solutions to said problem. to learn more about action research, nancy adams suggested reading a text by kuhne and quigley (1997).   conclusion   libraries of all types are under more and more pressure to show their value to the community and student learning. further, libraries must adapt to an ever-changing information landscape and evolve to meet the needs and wants of a new generation of users. evidence based practice is one key way libraries can change and adapt, but in order for this approach to work it is imperative libraries and librarians engage in quality research. research can inform decision making and demonstrate value, but in order to do so, librarians must improve the quality of their research. librarians need to do a better job of choosing their research topics, to become more informed of research methods, and to make good use of theoretical frameworks. the challenge is great, but one that is surely surmountable.     references   kuhne, g. w., & quigley, b. a. (1997). “understanding and using action research in practice settings.” new directions for adult and continuing education, 73, 23–40. doi: 10.1002/ace.7302   leckie, g. j., given, l. m., & buschman, j. e. (eds.) (2010). critical theory for library and information science: exploring the social from across the disciplines, santa barbara, ca: libraries unlimited.   information services department of the library of the health sciences-chicago, university of illinois at chicago. (2006)  levels of evidence: evidence-based practice in the health sciences: evidence-based nursing tutorial. retrieved 8 sept. 2013 from http://ebp.lib.uic.edu/nursing/node/12       microsoft word news_equator.doc evidence based library and information practice 2007, 2:4  116 evidence based library and information practice     news    the equator network – a new web resource for good reporting of health research         lindsay glynn  editor‐in‐chief    public services librarian and instruction coordinator, health sciences library  memorial university of newfoundland  st. john’s, newfoundland, canada  e‐mail: lglynn@mun.ca      © 2007 glynn. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    too often, good research evidence is  undermined by the poor quality of scientific  reporting. the equator network  www.equator‐network.org is a new  initiative that seeks to improve the quality  and reliability of health research literature.     equator network has launched a new  website that provides unique access to  collated expertise and resources for good  reporting of health research. the resources  are aimed at researchers (authors of research  articles), journal editors, peer reviewers, and  developers of reporting guidelines.     the most important website feature is a  collection of reporting guidelines (see  http://www.equator‐ network.org/index.aspx?o=1032). the last  decade saw an expansion in the  development of guidelines that specify  minimum criteria for reporting various  study designs. of these guidelines,  consort, for reporting randomised  controlled trials (see http://www.consort‐ statement.org)  and quorom, for  systematic reviews (see www.consort‐ statement.org/mod_product/uploads/quor um statement 1999.pdf) are probably the  best known. the equator site brings all  available resources under ‘one roof’ and its  team is extensively networking and  searching to keep the collection up‐to‐date.     to become a truly useful resource for its  users, the equator network plans further  development and a set of key deliverables,  including:     • a web‐based resource centre with easy  access to reporting guidelines and other  information;   mailto:lglynn@mun.ca http://creativecommons.org/licenses/by/2.0 http://www.equator%e2%80%90network.org http://www.equator%e2%80%90network.org/index.aspx?o=1032%00%00 http://www.equator%e2%80%90network.org/index.aspx?o=1032%00%00 http://www.equator%e2%80%90network.org/index.aspx?o=1032%00%00 http://www.consort%e2%80%90statement.org http://www.consort%e2%80%90statement.org http://www.consort%e2%80%90statement.org http://www.consort%e2%80%90statement.org/mod_product/uploads/quor http://www.consort%e2%80%90statement.org/mod_product/uploads/quor http://www.consort%e2%80%90statement.org/mod_product/uploads/quor evidence based library and information practice 2007, 2:4  117 • training courses for authors, editors and  peer reviewers, to facilitate transparent  and accurate reporting and use of  reporting guidelines;   • support to the developers of reporting  guidelines;   • annual assessment of how journals  implement reporting guidelines;   • annual audit of reporting quality across  the health research literature.   professor doug altman, one of the key  movers of the consort initiative, is  leading the equator project. the  equator network international steering  group includes leading experts in the fields  of health research methodology, reporting  and editorial work.     funding   the uk nhs national knowledge service  provided the initial funding for the project.  however, in order to have any meaningful  impact on the quality of reporting of  research publications we need to secure  more funds from different organisations.  funding opportunities for various  equator activities are outlined on our  website.   for more information please visit the  equator network website at  http://www.equator‐network.org/  contact: iveta simera by e‐mail:  iveta.simera@cancer.org.uk     please feel free to link to the equator site  from your website to facilitate use of our  resources.   http://www.equator%e2%80%90network.org mailto:simera@cancer.org.uk article   availability of library and information services for rural community development in nigeria: a case study of ewatto, in esan south east lga of edo state   jerome idiegbeyan-ose technical services librarian centre for learning resources covenant university canaan land ota, ogun state, nigeria email: jerose2010@yahoo.com                 olalekan a. adekunjo faculty of pharmacy librarian kenneth dike library university of ibadan ibadan, oyo state, nigeria email: blessing_adekunjo@yahoo.com   promise ilo acquisition librarian centre for learning resources covenant university canaan land ota, ogun state, nigeria email: promise.ilo@covenantuniversity.edu.ng   frederick e. odion serial /research librarian university library ambrose alli university ekpoma, edo state nigeria email: fredrickman@yahoo.com   received: 21 jan. 2015    accepted: 21 july 2015      2015 idiegbeyan-ose, adekunjo, ilo, and odion. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – this paper examines the availability of library and information services in a rural community using as a case study the community of ewatto in esan south east local government area of edo state in nigeria   methods – questionnaires were distributed by hand in each of ewatto’s eighteen villages.   results – the survey found that the information needs of ewatto rural dwellers are in the areas of health, occupation, politics, as well as family and education. their major sources of information include town criers, oral information from neighbours and newspapers. the major challenges hindering information availability include absence of libraries and information service centres, lack of internet connectivity and poor attitude of rural extension workers.   conclusion – for any nation to develop maximally, rural communities must be provided with adequate, relevant, and up-to-date information, the authors recommend the establishment of a library and information centre in ewatto. introduction   the difference between developed and developing nations is generally attributed to factors such as per capita income, literacy, and living standard (saharil, 2015, dufrenot, mignon & naccache, 2009). good tax systems, good governance, and well-developed infrastructure are also associated with developed countries (goodspeed, martinez-vazquez & zhang, 2010).   kamba (2009), however, asserts that information also contributes positively to the social, political, educational, and economic development of any community. it creates awareness among the members of that community. it is a resource that all individuals use to improve their wellbeing. information enables individuals, societies, communities, and organisations to make rational decisions and reduce their level of uncertainty (idiegbeyan-ose & akpoghome, 2009).   ikhizama and okojie (2001) observed that libraries are influential in the dissemination of information, especially in rural areas. this is particularly important in developing nations like nigeria because the bulk of the nation’s population dwells in the rural areas (issa, omopupa & salman, 2012).     unfortunately, research findings and observation reveal that establishments and agencies whose responsibilities it is to ensure information dissemination in nigeria have an urban-biased orientation (issa et al., 2012). even though obinyan, obinyan and aidenojie (2011) observed that rural dwellers can derive maximum benefits from the use of library resources and services, libraries and information centres are not commonly found in rural areas.   social inequality is perpetuated when the rural population does not have access to up-to-date information (issa et al., 2012). the information needs of rural dwellers include those of education, health, agriculture, employment, and transportation among others (alegbeleye & aina, 1985, idiegbeyan-ose & akpoghome, 2009, mtega & ronald, 2013).   literature review   agha and akhtar (1992) characterise information as a strategic resource along with human, financial and natural resources while onu (2005) refers to information “as the oil that greases the wheel of governance programme and the activities of government and individuals” (p.85).   popoola (2007) enumerates the five factors of production as: land, labour, capital, entrepreneurship, and information. he further stresses that for meaningful development to take place anywhere, adequate information must be provided. berezi (1983 also calls information the fifth factor of production, saying it is essential for agricultural and rural development and that it is what brings about social and economic change. achugbue and anie (2011) observe that african countries have not devoted attention to providing their citizens with access to information, and especially not the rural dwellers who constitute a majority of the african population.   opeke (2004) states that information is widely accepted as the essential feature of production, consumption, and exchange and that information empowers all citizens. furthermore, information is needed for empowerment and decision making, and is necessary for development in rural communities (unagha & ibenne, 2011). seneviratne (2007) suggests that rural communities need information for living and for participation in social activities.    access to information is a basic right of citizens either in rural or urban areas. nevertheless, nigerian rural dwellers are sometimes perceived to be ‘illiterates’ who have nothing to contribute to the society. in 1987 aboyade reported that some librarians believed that until urban dwellers with capacity to use library services had been adequately served, it would be “a misdirection of efforts to serve the illiterate rural dwellers” (p. 200).   in this globalised and information-driven society, information has become a necessity (unagha & ibenne, 2011). harande (2009) points out that “any serious, active, conscious, sensitive government would not neglect rural communities in terms of provision of information” (p.1). he further observes that there is a significant relationship between slow or no development in african countries and neglect of the rural areas. in 1985 alegbeleye and aina stated that third world countries had come to realize that little meaningful development would take place unless the rural areas were provided with adequate and timely information. more recently, harande (2009) observes that there are still inadequate information services in rural areas.   statement of problem   rural dwellers cannot fulfill their potential and contribute to national development if they lack access to information.   the rural community of ewatto has several public and private primary and secondary schools, yet one wonders if members of this community also have access to the information required for the improvement of their lives and for contributing to national development. it is against this background that the researchers decided to investigate library and information services in ewatto community in esan south east lga (local government area) of edo state, nigeria.   aims   the general objective of this study is to examine information services in ewatto.   the specific objectives are to   ·         ascertain the information needs of ewatto community dwellers; ·         find out the sources of information available for ewatto community dwellers; ·         investigate the availability of library and information services in ewatto town; ·         examine the challenges hindering information availability to ewatto community dwellers.   the research questions are   ·         what are the information needs of ewatto community dwellers? ·         what are the sources of information available for ewatto community dwellers? ·         are library and information services provided in ewatto community? ·         what are the challenges hindering information availability in ewatto community?   brief description of ewatto in esan south east lga of edo state nigeria   ewatto is a community of eighteen villages located in edo state in southern nigeria. elevated at 155 metres above sea level, it lies less than seven degrees north of the equator. the land is agricultural. the population reference bureau points out that “census taking in developing countries can be …difficult” (lalasz, 2006), and one of the few available estimates of ewatto’s population places it at about 9600 (lalasz, 2006).   figure 1 ewatto, nigeria (mapcarta, 2015).     ubiaja is the nearest city, and is the location of the administrative headquarters of esan south east local government area as well as the location of the nearest public library. ubiaja is about nineteen kilometres north of ewatto.   while the population in ewatto’s eighteen villages engages in hunting, trading, traditional medicine, teaching, craft and health work, most people are farmers who produce crops such as yams, tomatoes, kola nuts and cassava.     figure 2 satellite view of ewatto’s villages (mapcarta, 2015).   there are village heads, or chiefs, in each of the eighteen villages. these chiefs are responsible to the onojie (literally, king: the administrative head of all the villages) who is, in turn, responsible to the edo state governor. ewatto has three major markets, a central post office, and a police station. cell phone service is available, but internet service is not.   esan and english are the predominant languages. there are two secondary schools and eight primary schools founded by the state government, as well as private primary and secondary schools. the literacy rate in ewatto is high because almost all the young people have completed secondary school.   methods   the survey method (a questionnaire) was adopted for collecting information in this study. a copy of the questionnaire can be found in appendix a. the gender and age of the respondents is summarized in the charts below.   development of questionnaire   survey taking is not a common experience in ewatto. it was felt that some respondents would find it difficult to respond to open-ended questions, and others might not respond at all. further, the entire questionnaire had to be interpreted for some respondents, and open-ended questions would have been especially difficult for them. therefore, the authors decided not to use this type of question and, instead, provided a list of answer choices for individual questions. the answer choices were based on information needs and challenges of rural dwellers as suggested by the literature. with respect to question 6, for example, kamba (2009) and mtega and ronald (2013) suggest that lack of a library and information centre is an impediment to information availability while harande (2009), zaid and popoola (2010) and mtega and ronald (2013) mention lack of internet access as an impediment. kamba (2009), zaid and popoola (2010) and mtega and ronald (2013) suggest that government insensitivity to rural information needs and insensitivity of rural information workers contribute as well.   an “others” option was provided in questions 3, 4, 5 and 6 in case respondents wished to articulate their own answers (none did).   data collection in ewatto’s villages   the researchers and research assistants used okadas (the motorbikes ubiquitous in nigeria) to get from one village to another over ewatto’s dirt roads. they walked around the villages distributing the questionnaires after 4pm when everyone was home from the work place or farm. anyone could participate, and respondents included farmers, students, retired and active teachers, principals, headmasters, politicians and civil servants. on arriving in each home, researchers would observe cultural protocols such as making proper greetings, introducing themselves, and conversing about the purpose of the visit. they waited for respondents to fill in the questionnaires and collected them back. farmers who could not read or write were assisted in completion of the questionnaires with the aid of local language interpreters. after the main exercise, the villagers would offer the visitors refreshments as the culture of ewatto demands. each visit could take up to an hour, or longer with non-literate respondents, because questions had to be interpreted for them in the esan language. as a result of these conditions, the process of distributing and administering the questionnaires took about two years.   in all, fifteen questionnaires were distributed in each of ewatto’s eighteen villages. this gave a total of 270. out of the 270 questionnaires distributed, 238 were returned of which 210 were properly completed and used for data analysis. this represents a 77.8% response rate.   results   items 1 and 2 on the questionnaire were used to collect data about gender and age.   in question 3, respondents were asked to indicate their information needs. based on their responses, these needs are categorized broadly as shown in table 3.   table 3 shows that health /family planning information is the most common information need of rural dwellers of ewatto as indicated by 183 (87.1%) of the respondents. this is followed by occupational information as indicated by 177 (84.3%) of the total respondents. information on politics and education were ranked third and fourth. the “others” choice, which ranked fifth, was selected by 66 respondents but none of these 66 responses indicated what was meant by “others.” the responses are further depicted in figure 3.     table 1. gender of respondents gender of respondents frequency percentage male 87 41.4 female 123 58.6 total 210 100    table 2. age of respondents age of respondents frequency percentage 10-15 years 13 6.2 16-20 years 17 8.1 21-25 years 33 15.7 26-30 years 30 14.3 31-35 years 23 11.0 36-40 years 32 15.2 41 and above 62 29.5 total 210 100   table 3. information needs of ewatto community dwellers category of information need number of respondents percentage (%) occupational information 177 84.3 health information/ family planning 183 87.1 information on education 89 42.4 information on politics 122 58.1 others (selected, but details not articulated). 66 31.4     figure 3. information needs of ewatto community dwellers.     in question 4, respondents were asked to indicate the sources of information available to them in ewatto. their responses are summarized in table 4.   table 4 shows that the major means of getting information in ewatto is through the town criers. town criers are appointed by the chief and village elders who provide them with information. an example of the news they might announce is “the chief said medical officers are coming to our village tomorrow to immunize our children under five years old. you are expected to bring your children to the chief compound tomorrow morning.” most town criers are men. they walk around the village using a gong to get people’s attention before they give the information.   in this study, 196 respondents (constituting about 93.3% of the respondents) indicated that they get news from town criers. another major source is oral information which is accounted for by 155 (73.8%) respondents. newspaper, posters, magazines, and church bulletin attracted 112 (53.3%) respondents, while radio and television were identified by 56 (26.7%) respondents.   in question 5, respondents were asked if library and information services are provided anywhere in ewatto community. the respondents categorically stated that there are no libraries and information services of any type provided in ewatto community, including in any of the primary and secondary schools owned by the community, state government, or individuals.     table 4. sources of information available in ewatto sources of information number of  respondents percentage (%) town criers 196 93.3 radio and television 56 26.7 oral information through friends and neighbours                    155 73.8 newspapers, posters, magazines and church bulletin 112 53.3     figure 4. pie chart showing sources of information in ewatto community.     in question 6, respondents were asked to indicate one or more challenges hindering the availability of information in ewatto community. responses are summarized in table 5.     table 5. challenges of information availability in ewatto community challenges number of responses percentage (%) unavailability of library and information centres 198 94.3 lack of internet connectivity 148 70.5 insensitivity of the government to rural information needs 134 60.8 poor attitude of  rural information extension workers 112 53.3 others 0 0     table 5 shows that 198 (94.3%) of the respondents stated that unavailability of library and information centres was the major barrier to information availability in ewatto. lack of internet connectivity is another challenge as indicated by 148 (70.5%) respondents. insensitivity of the government to rural information needs is further challenge as shown by 134 (60.8%) rural respondents. finally, 112 (53.3%) respondents indicated that lack of information is due to poor attitude of rural information extension workers.[1] an “others” option was provided for this question, but was not selected by any of the respondents.     figure 5. bar chart showing challenges of information availability in ewatto.     discussion   this study shows that rural dwellers in ewatto community have various information needs ranging from occupational to health, political, recreation, family planning, and retirement. this is consistent with the findings of mtega and ronald (2013), and seneviratne (2007), who stressed that communities need information in the areas of health, education, agriculture, politics and occupation.   the main source of information for ewatto rural dwellers is the town crier.    the study points to lack of a library and information centre, lack of internet connectivity and insensitivity of the government to rural information needs as impediments to information availability. this finding is consistent with mtega and ronald (2013) who listed lack of internet, lack of electricity, outdated information sources, and lack of government interest among the challenges of rural dwellers in being able to access information.   a library and information centre, including internet connectivity, would seem a likely remedy to these problems.   in their study of public libraries in edo state, obinyan, et al. (2011) pointed out that public libraries are "hardly used and when they are used, the majority of users are pupils who use them as reading rooms” (p. 4). obinyan et al. indicated that this low usage happened because the information needs of the local population were not investigated beforehand so as to know the right materials and services to be provided. the outcome was that the libraries (all of which lacked internet access) were of little relevance to potential users. why would the outcome in ewatto be different? one of the main purposes of this study was to discover the specific information needs of the ewatto community. a library based on the known information needs of the community could provide relevant materials in various formats including print, audio, and video. the library could also provide translation and information repackaging services for non-literate users. these materials could include information in the local dialect in the form of posters and other pictorial formats, drama, songs, and so on. user education programmes would be of key importance and could be supported by the village heads of ewatto, whose words are highly respected. the village heads could also direct the town criers to remind villagers of the need to use the library and announce special events such as programmes to teach farmers how to increase crop yield, pregnancy management for expectant mothers, etc. such a library could maintain a high level of patronage and remain relevant by considering and meeting the information needs of the entire ewatto community.   conclusion   information plays a vital role in the progress of any nation and nigeria is no exception. it is the responsibility of federal, state, and local government to provide adequate, relevant, and timely information, especially to the rural dwellers who make up the majority of nigeria’s population.   research cited in this study has shown that rural dwellers can derive maximum benefits from information services available in library and information centres. in the community of ewatto, however, there is no library and information centre, not even in any of the government-run or privately-owned schools, thus limiting the community’s access to needed information. although public libraries in africa are often underused, this study suggests that a library based on the known information needs of the community could employ existing cultural avenues of support and maintain high patronage and relevance to the community. therefore, the following recommendations are made:   solicit support and funding from those government ministries whose responsibility it is to provide internet and information services to rural dwellers, with the goal of creating and maintaining a library and information centre in ewatto. also solicit private support and funding to create and help maintain the library. select library materials and offer user education programmes based upon the specific information needs of the community. provide specialized materials and services to meet the needs of non-literate residents. use existing cultural avenues (influential role of village heads, services of town criers, etc.) to create awareness, promote library use and attendance at education programmes, and, in general, help ensure that the library maintains its relevance to the community.   it is expected that a library based on this model could lead to numerous individual life improvements and more community participation in national development.   acknowledgments: we would like to thank itamah jerome and glory jerome for their assistance in the distribution and collection of questionnaires, carol waseleski for her advice and the editor and reviewers of eblip for their valuable recommendations.   references   aboyade, b.o. (1987). the provision of information for rural development. ibadan; fountain publishers.   achugbue, e. i. & anie, s. o. (2011). icts and information needs of rural female farmers in delta state nigeria. library philosophy and practice, retrieved 6 sept. 2012 from http://digitalcommons.unl.edu/libphilprac/448   agha, s. s & akhtar, v. (1992). the responsibility and the response: sustaining information systems in developing countries. journal of information science, 18(4), 283-292. http://dx.doi.org/10.1177/016555159201800406   alegbeleye, g. o. & aina, l. o. (1985). library services and rural community development in nigeria: an introduction. ibadan: department of library studies.   berezi, a. (1983). information as a factor of production. business economics, xvi(1), 14–20.   dufrenot, g. mignon, v. & naccache, t (2009. the slow convergence of per capita income between the developing countries: “growth resistance” and sometimes growth tragedy. retrieved 6 apr. 2015 from www.nottingham.ac.uk/credit/documents/papers/09-03.pdf   goodspeed, t., martinez-vazquez, j. & zhang, l. (2010). public policies and fdi location: differences between developing and developed countries. retrieved 4 sept. 2015 from http://icepp.gsu.edu/files/2015/03/ispwp1001.pdf       harande, y. i. (2009). information services for rural community development in nigeria. library philosophy and practice. retrieved 2 oct. 2012 from http://www.webpages.uidaho.edu/mbolin/harande.htm   idiegbeyan-ose, j. & akpoghome, t. u. (2009). information as an effective tool in rural development. international journal of library and information science, 1(3) 022-028. retrieved 22 oct. 2012 from www.academicjournals.org/journal/ijlis/article-full-text.../9a5900e747   ikhizama, b. o. & okojie, v. o. (2001). library and information services for development and rural transformation. bookmobiles and outreach services, 4(2) 22-30.   issa, a. o., omopupa, k.t. & salman, a. a. (2012). rural information provision for national development. a study of kwara north senatorial district of kwara state nigeria. plna quarterly, 76(3), 171. retrieved 5 sept. 2012 from http://www.pnla.org/quarterly   kamba, m. a. (2009). an overview of the provision of information for rural development in nigeria. samaru journal of information studies. vol. 9 (1). http://dx.doi.org/10.4314/sjis.v9i1.55469     lalasz, r. (2006). in the news: the nigerian census. in population reference bureau. retrieved 1 june 2015 from http://www.prb.org/publications/articles/2006/inthenewsthenigeriancensus.aspx   mapcarta (2015). the interactive map. ewatto, nigeria. retrieved 1 june 2015 from http://mapcarta.com/.     mtega, w. p. & ronald, b. (2013). the state of rural information and communication services in tanzania: a meta-analysis.   international journal of information and communication technology research, 3(2), 64-73. retrieved 4 sept. 2015 from http://esjournals.org/journaloftechnology/archive/vol3no2/vol3no2_3.pdf   obinyan, g. a., obinyan, o. o., & aidenojie, e. (2011). use of information resources in four branches of a state public library in nigeria. chinese librarianship: an international electronic journal, 31. retrieved 3 dec. 2014 from http://www.white-clouds.com/iclc/cliej/     onu, b. c. (2005). using newspapers to satisfy the information needs of readers at the federal polytechnic library nekede owerri. nigerian library link. 3(1&2), 84-96.   opeke, r. (2004). information for education and development in nigeria. a compendium of papers presented at the 2004 nla annual national conference/agm, akure, june 2127.   popoola, s. o. (2007). information product and services: unpublished master note.   saharil, a. (2015). differences between developed, developing and underdeveloped countries. retrieved 6 apr. 2015 from http://docslide.us/documents/differences-between-developed-developing-and-underdeveloped-countries.html   seneviratne, w. (2007). framework to measure community information literacy among rural citizens in sri lanka: building of a cil model. sri lanka journal of librarianship and information management. 3(1), 14-24.   unagha, a. o. & ibenne, s. k. (2011). obstacles to provision and use of development information in             rural communities in nigeria. library philosophy and practice. retrieved 4 sept. 2015 from http://digitalcommons.unl.edu/libphilprac/   zaid, y. a. & popoola, s. o. (2010). quality of life among rural nigerian women: the role of information. library philosophy and practice retrieved 4 sept. 2015 from http://digitalcommons.unl.edu/libphilprac/     appendix a   questionnaire on availability of library and information services for rural community development in nigeria: a case study of ewatto, in esan south east lga of edo state.   please provide answers to the questions. the information is for research purpose.   thank you.   1.    sex male (   ) female (  )   2.    age  10-15 years (   ) 16-20 years (   )  21-25 years (   )  26-30 years (  ) 31-35 years (  ) 36-40 (  ) 41 and above (  )   3.    what is your information need? please tick the options that are applicable to you. information needs   occupational information   health/family information   information on education   information on politics   others     4.    what are the sources of your information? please tick the options that are applicable to you. sources of information   town criers   radio and television   oral information through friends and neighbor   newspapers, posters, magazines and church bulletin   others     5.    is there any library and information centre in ewatto? yes (  )  no (  )   6.    what are the challenges of information availability in ewatto? please tick the options that are applicable to you. challenges to information availability   unavailability of library and information centre   lack of internet connectivity   insensitivity of the government to rural information needs   poor attitude of rural information extension workers   others       [1] rural extension workers are government officers in different sectors, such as agriculture, health, and education. they serve as intermediaries between the government and the rural dwellers in their day-to-day activities. their responsibility is to liaise with the rural dwellers and find out their challenges and then to communicate these issues to the appropriate government agency for a solution. evidence summary   administrator interest is perceived to encourage faculty and librarian involvement in open access activities   a review of: reinsfelder, t.l., & anderson, j.a. (2013). observations and perceptions of academic administrator influence on open access initiatives. journal of academic librarianship, 39(6): 481-487. http://dx.doi.org/10.1016/j.acalib.2013.08.014   reviewed by: eamon c. tewell reference & instruction librarian long island university, brooklyn brooklyn, new york, united states of america email: eamon.tewell@liu.edu   received: 12 may 2014  accepted: 8 aug. 2014      2014 tewell. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to better understand the roles and influence of senior-level academic administrators, such as provosts, on open access (oa) activities at the institutional level, including whether librarians perform these activities regardless of administrative interest.   design – web-based survey questionnaire combined with multiple regression analysis.   settings – the research was conducted online using surveys emailed to potential participants at not-for-profit public and private academic institutions in the united states with a fte of greater than 1000.   subjects – academic library directors at selected colleges and universities.   methods – using directory information from the national center for education statistics (nces) and filtering institutions according to not-for-profit status, size, and special focus, a survey sample of 1135 colleges and universities was obtained. library websites were used to acquire contact information for library directors. in summer 2012 the 43-item survey questionnaire was distributed to respondents online using qualtrics. the four primary variables were each comprised of multiple questionnaire items and validated using factor analysis, and the data was explored using multiple regression.   main results – the survey received 298 respondents for a 26% response rate, though the number of incomplete responses is not stated. among four stakeholder groups (faculty, publishers, librarians, and senior academic administrators), library directors perceived librarians as having the greatest influence in regards to the adoption of open access (mean = .7056), followed by faculty (.3792), administrators (.1881), and publishers as having a negative impact (–.3684). a positive correlative relationship was determined between administrator attention to open access—a key variable operationalized by combining eight questionnaire items—and the variables librarian commitment to open access, faculty commitment to open access, and faculty proclivity toward open access, with the latter especially the case at lower levels of administrator support. regarding institution size, library directors perceived a higher likelihood of faculty adherence and librarian commitment to oa at large institutions (over 20,000). a given institution’s public or private status and geographic region were not significant predictors of faculty or librarian commitment or adherence to open access.   conclusions – the study finds that academic library directors perceive librarians to have the strongest influence upon adoption of open access, and senior academic administrator attention to open access is positively linked to the oa activities of faculty and librarians. larger institutions are considered to have greater commitment to oa, potentially due to differing missions according to institution type. the authors recommend that open access advocates consider administrator roles and target administrator support when seeking to increase participation in oa.     commentary   open access publishing is the subject of much interest and debate in the library, academic, and publishing communities. a considerable body of research addresses various stakeholders’ positions on the adoption of open access, and these works frequently provide insights into how certain populations approach oa concepts and practice. such studies have primarily focused upon the attitudes of either faculty and researchers (xia, 2010) or librarians (palmer, dill & christie, 2009). the only prior research considering academic administrators’ roles in the open access environment is that of reinsfelder (2012). the study at hand makes a valuable contribution to the literature on open access in that it addresses the understudied population of academic administrators and their perceived attention to oa. in general, the authors accomplish their goal of increasing understanding of administrator influence on faculty and librarian participation in oa.   the most significant strengths of this research include the well-explicated methodology, appropriate statistical procedures to validate the primary variables and test significance of the results, and the discussion and interpretation of the findings. regarding the sample, the inclusion and exclusion criteria are made clear and the survey response represents a sufficient sample size and response rate. the research instrument and accompanying answers are included in the appendix, increasing the feasibility of replicating the study.   despite a strong overall design and reporting of the results, some limitations impacting the strength of the evidence were identified. the implications for practice and future research are minimally considered. the authors recommend additional studies in this area using quantitative and qualitative methodologies but no specific suggestions are offered. one such approach might be a citation analysis of administrator publishing histories to identify publications appearing in oa titles. the study lacks a mention of its limitations, which would be useful information for readers wishing to interpret and evaluate the findings and conclusions. another point of consideration is the data source. only one of four stakeholder groups, library directors, is consulted. although possibly outside of the scope of this paper, it would be highly illuminating to compare directors’ responses with those of another group, such as academic deans and chairpersons, to distinguish where their perceptions intersect or diverge. despite these limitations, the methods, findings, and conclusions are sound and provide useful evidence regarding the research questions examined.   this work is a timely and insightful investigation of library directors’ perception of various stakeholders’ influence, particularly academic administrators, upon oa practices at colleges and universities in the united states. the most valuable aspect of this research is that it reflects on the role of administrators, an often-overlooked group, and identifies their importance in the dynamic and politically charged oa landscape. practical implications suggested by the authors include advising open access advocates to solicit support from administrators, who indeed must be involved in the oa conversation to improve progress in this vital movement that features prominently in the future of scholarship. additionally, the influence of library directors upon academic administrators, including how directors might educate and solicit support for oa, can and should be considered in practice. future work in this area should examine the perceptions of multiple groups to better identify how each comprehends oa efforts in relation to one another.     references   palmer, k.l., dill, e., & christie, c. (2009). where there’s a will there’s a way? survey of academic librarian attitudes about open access. college and research libraries, 70(4), 315-335. http://crl.acrl.org/content/70/4/315     reinsfelder, t.l. (2012). open access publishing practices in a complex environment: conditions, barriers & bases of power. journal of librarianship and scholarly communication, 1(1), ep1029. http://dx.doi.org/10.7710/2162-3309.1029     xia, j. (2010). a longitudinal study of scholars’ attitudes and behaviors toward open-access journal publishing. journal of the american society for information science and technology, 61(3), 615–624. http://dx.doi.org/10.1002/asi.21283 microsoft word es_haley.doc evidence based library and information practice 2007, 2:2  107 evidence based library and information practice       evidence summary    leadership is central to the organizational success of libraries    a review of:  mullins, john, and margaret linehan. “the central role of leaders in public libraries.” library  management 26.6/7 (2005): 386‐96.    reviewed by:   pamela haley  manager, library services  stormont, dundas and glengarry counties  cornwall, ontario, canada  e‐mail: phaley@sympatico.ca      received: 28 february 2007  accepted: 18 march 2007      © 2007 haley. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to provide an understanding of  the concept of leadership as perceived and  practised by a group of international public  library leaders, thereby contributing to the  existing body of literature on this topic.    design – qualitative study employing in‐ depth, face‐to‐face interviews with 3  subjects and 27 e‐mail interviews based on  30 structured questions formulated from a  review of the literature.    setting – nottingham and rotherham, two  cities in th public libraries in ireland, britain,  and the east coast of the united states  during 2003/04.    subjects – thirty senior public library  leaders.  methods – thirty top level library leaders,  50% of whom came from outside ireland,  were selected based on the following  rationale concerning their institutions: have  a history of providing publicly funded  public libraries; are under‐researched; and,  accessibility for the authors. irish librarians  were chosen based on their national profile,  geographic convenience to the researchers,  and availability/cooperation. only library  leaders or deputy leaders were considered.  interviews for 27 candidates where  scheduled via e‐mail and 3 were arranged  through face‐to‐face meetings. participants  were asked 30 structured questions drawn  from the current literature on leadership.  responses were recorded electronically and  later transcribed for analysis via a grounded  theory approach that categorized the data.  this method of analysis endeavours to  provide theory based on research  observations that begin from qualitative  mailto:phaley@sympatico.ca http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:2  108 data and then go through a process of sifting  and categorization, which leads to a  hypothesis.    main results – nine broad categories  emerged from the ʺgrounded theory  analysis.ʺ the paper deals with only 1 ‐ the  central role of the library leader. in response  to queries by the authors, 67% (20) agreed  that leadership took precedence over all  other factors as a determinant for  organizational success. ten (33%) disagreed  and were evenly split as to their reasons.  five said it was an important factor, and five  felt leadership did not take precedence over  other drivers of success. the study reveals  that leaders do have an impact in  determining organizational culture, with  63% saying the leader determines the  culture, and 37% stating the leader only  contributes to the culture of an organization.  national cultural differences affect this  point of view, for example 73% of the irish  public librarians, 60% of the british public  librarians, and 40% of the american public  librarians stated that it is the leader rather  than the staff who determine organizational  culture. another core finding reveals that  not all ceoʹs are leaders and not all  librarians are leadership material, which  points to the need for more classic  leadership training.     conclusion – based on the studyʹs findings,  the authors hypothesize that leadership is a  central component in the strategic and  operational practices and success of public  librarianship. transformational, rather than  ʺleader centricʺ methods of leadership  encourage growth of organizational change  and success.    commentary      the authors have achieved their stated  purpose, at least in part. their efforts have  certainly added to the paucity of literature  on the topic of leadership in libraries. the  research methods used raise a number of  concerns. despite the authors’ assertions, it  is unclear whether the methodology is based  on a “representative” sample. the criteria  for inclusion in the study appear very broad  and based on convenience. at no time do  the authors provide a concise statement of  what they believe the attributes of a leader  are to use as a basis of their selection of  interviewees. there appears to be an  underlying assumption that a leader is  someone who has risen to the top position in  a library. there is no evidence provided that  would attest to the subjects’ leadership  capabilities. a description of how  “grounded theory” was actually applied  would have clarified how the data was  analysed. in addition, the paper fails to  delineate the 9 broad thematic categories  identified from the results.  it would have  been helpful if answers to all the questions  asked of the interviewees had been  provided to give a fuller picture of the study  results. indeed, there is a sense stemming  from the lack of full disclosure of the results,  that the results examined have been “cherry  picked” to support a hypothesis. much of  the discussion is based on anecdotal  evidence. the authors purport to explore the  “central role of leaders” in their research  paper. their results, however, are not well  documented or elaborated. the authors state  that they have found a “central role of  leaders” which paraphrased is that “modern  leaders need to see their staff as colleagues  rather than as subordinates, and need to  encourage autonomy in order to manage  existing and future challenges.” while this is  an interesting supposition, none of the data  presented brings the reader to this same  conclusion. nor is there any explanation as  to how the authors arrived at their  hypothesis based on the gathered data. a  closer examination of the geographic  differences in leadership would have been  enlightening – does the overall culture of a  society affect the style of leadership? also, it  would have been interesting to know how  evidence based library and information practice 2007, 2:2  109 those working with the subjects viewed the  subjects’ leadership skills, and how these  subordinates themselves defined leadership.  this article is a starting point from which  further research will provide greater  insights into the role of leadership in public  libraries.    evidence based library and information practice commentary   building evidence of the value and impact of library and information services: methods, metrics and roi   carol tenopir chancellor’s professor university of tennessee knoxville, tennessee, united states of america email: ctenopir@utk.edu      2013 tenopir. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   those who work in libraries or have a close understanding of their historical and contemporary roles know that libraries bring value to the individuals and institutions they serve. yet, when there are decreasing resources and increasing alternatives for information and attention, libraries of all types find that they need to measure and demonstrate the value of all of their collections and services. and, sometimes, measuring value means choosing to eliminate some traditional roles in order to take on or re-emphasize new ones (tenopir, 2012).   the history of building evidence of the value of libraries is long, particularly for public and special libraries (griffiths & king, 1993, 2011; fraser, et al, 2002; matthews, 2002, 2007). in academic research libraries, the libqual+ initiative provides an international base of comparison for patrons’ expectations and how individual libraries meet those expectations (association of research libraries, 2013). oakleaf (2010) summarized years of work on value of academic libraries. she emphasized the need for academic libraries to demonstrate their value and offered multiple suggestions for how to do so, including surrogate measurements for value and impact. as important as demonstrating value, however, is being valuable: the line of inquiry about how valuable libraries are and how to increase their value derives from inquiring about the demonstration of value (oakleaf, 2010).   yet, much remains to be done in building a culture of measuring value in the specific context of each individual academic library. the libvalue project, sponsored by the u.s. institute of museum and library services, is a multi-year project focusing on testing multiple methods for measuring the value, outcomes, and return on investment (roi) of academic library products and services (libvalue.cci.utk.edu, 2013). principals in libvalue include the university of tennessee; university of illinois; syracuse university; byrant university; and the association of research libraries. other participants have included, in the uk, jisc collections; cranfield university; university of dundee; university of east anglia; durham university; imperial college london; and university of manchester. in the us, participants include seton hall university; the university of north carolina-wilmington; the university of colorado; baruch college; buffalo state university; and brooklyn college. in australia, participants include the john metcalfe memorial foundation; the university of new south wales; and university of queensland.   the value of an academic library is complex, because total value is composed of many separate values for each type of collection or service and because the value differs for different constituents and over time. the value of the entire library may be a composite of dozens (or even hundreds) of separate services and collections, each of which has a different value. value to first year undergraduates may be the library’s role in encouraging them to continue in school (thus factoring into retention rates), which is enhanced by having a welcoming physical space with friendly assistance. by the time those students are seniors, the value of the library may be more in helping them find high quality resources in a timely manner to improve their research papers. for a faculty member, the value of instructional services may be to help them improve the courses they teach and, at the same time, help their students do better in class assignments. for a graduate student or a faculty member’s role as researcher, the value of the library may be to get access to the widest range of resources or the use of an institutional repository as a place to deposit their research datasets for long-term preservation and shared access. value in all of these cases, however, revolves around outcomes – how the library contributes to the academic work and success of the people it serves.   in a broad sense, library and information resources value can be considered as either “exchange” value or “use” value (machlup, 1979).  in the information context, exchange (or “purchase”) value is what one is willing to pay for information in money or time, and use value is the favorable consequences derived from using the information. use value can also be looked at as outcomes from using information products or services. bruce kingma of syracuse university, principal in the libvalue project, describes three different types of library value, including: 1) economic (private) value, e.g., what is the value to an individual to use the library resources?; 2) social (public) value, e.g., what is the value to the institution of the library?; and 3) environmental (externality) value, e.g., what is the value of the environmental savings of library provision of electronic resources?   another way to categorize types of value measurements in libraries is to consolidate these measures into: 1) measures of “implied” value, such as the value that is measured from usage statistics (people download the library e-resources, therefore value of these collections is implied); 2) “explicit” measures of value that come directly from testimonials (for example, how access to a special collection helped an author complete a book) or value that constituents ascribe to the library and describe in interviews, focus groups, or questionnaires; and 3) “derived” measures such as roi or contingent valuation measures that are calculated on a combination of implied, explicit and other sources of evidence (tenopir, 2012). some examples of types of value and measurements are given below.   exchange value for a journals collection can be calculated using questionnaires to gather information on the amount of reading by academic staff and faculty members (or students), how much time they spend on average per reading, and extrapolating up to an annual basis and to the total university faculty population. for example, in six universities in the uk, faculty members report reading on average per month, 22 scholarly articles at 49 minutes per reading, 7 books or parts of books at 106 minutes per reading, and 10 other scholarly publications at 42 minutes per reading. this adds up to a commitment of 37 hours per month just on reading (not including the time spent identifying and obtaining the readings) (tenopir & volentine, 2012). the percent of that time that comes on readings from the library is one way to measure exchange value; in the case of the uk, respondents reported that 67.3% of article readings, 27.6% of book readings, and 15.2% of other publication readings come from the library. that means they spend over 16 hours per month or an estimated 192 hours per year just in reading items from their libraries. bringing in average faculty salaries and extrapolating to the total faculty numbers, and adding in time identifying and locating the readings, completes this exchange value calculation (king, 2012).   another value that can be derived from this type of study is contingent valuation (what faculty or students would expect or be willing to pay if the library collections or services were not available to them). at bryant university in the us, libvalue principal donald w. king conducted an in depth cost and value study of multiple library services. for journal collections he estimated that at the average salary of bryant university faculty members, the cost of not having access to library-provided articles would carry a total cost of nearly a half million dollars or a cost per faculty member of $1,200 or cost per reading of $27 (king, 2012).   return on investment (roi) in a strict sense is a quantitative measure expressed as a ratio of the value returned to the institution for each monetary unit invested in the library.  in other words, for every amount of money spent on the library, the university receives ‘x’ amount in return. roi can be used with contingent valuation or it can be used to demonstrate that library collections and services contribute to income-generating activities.   in a study funded by elsevier, the roi of research university libraries in eight countries in helping the research grants process succeed, through access to high quality materials to cite, was estimated to be between just under 1:1 for a humanities/social sciences based teaching institution to over 15:1 for a scientific research university system. most comprehensive research and teaching universities had a library roi in grants in the range of 3-5:1 (tenopir, 2010).    the national network of libraries of medicine has an roi calculator that helps libraries demonstrate: how much benefit does the institution, your user, receive for every dollar spent by the library?; value of benefits and costs for each service; and total value of library use (2013). simple public library roi calculators are also available (see, for example, http://www.lrs.org/data-tools/public-libraries/return-on-investment/ and http://www2.library.lapeer.org/library-return-on-investment-calculator.html).   in a softer sense, roi is also values of all types and outcomes that come to stakeholders and the institution from use of the library’s collections, services, and contribution to its communities. use or outcomes value can be gathered explicitly in a survey or interview by asking respondents to focus on a critical incident of information use. in the example of scholarly reading, we ask respondents to focus on the last scholarly article they read (whether or not it was particularly typical or important), thus giving us a second stage random sample of readings in addition to the first stage sample of readers. in the study in the uk, faculty identified many outcomes from article reading, including in rank order: 1) inspire new thinking or ideas; 2) improve the results of work; 3) narrow/broaden/or change the focus of the principal purpose; 4) resolve technical problems; 5) save time or other resources. negative outcomes were offered as choices, but were seldom selected.   open-ended questions are another source of explicit outcomes to support decision making (volentine & tenopir, 2013). comments include simple praise or complaints about the level of service provided or can offer hints on how to improve service. for example, from one u.k. academic: “electronic access to the university library system from off-site is crucial for swift access to articles to support my teaching and research activities.” another (this time from our studies in australia) offers: “i prefer reading hard copy to online books and articles.  i particularly dislike ebooks and articles which i’m only able to read for a limited time.”  comments can be used to gather personal evidence of the value of the library’s collections and services; they can also lead to specific courses of action such as negotiating contracts that include off-site access and better e-book lending policies.   putting together quantitative data and open-ended comments in surveys allows libraries to put a personal face on their value to constituents by building data-based personas (volentine, et al 2012). personas are fictional characters based on actual respondent data. for example, in the uk, segmenting the data to select the most successful academics (defined as those who have won an award in the last two years and published four or more items), builds a portrait of a successful academic, who: reads more of every type of material; spends more time per book and other publication readings; uses the library for articles; more often buys books and obtains other publications from the internet; occasionally participates and creates social media content. another persona in the u.k. study is an associate professor in physics who reads 30 articles, 2 books, and 11 other publications per month and has not visited a physical library in many years. he needs current issues of articles, off-site access to collections, access to search engines and online resources without a distinction between library and non-library resources, and a wide range of dates to see trends over time.   it should be clear that value of the library to its constituents can be demonstrated in many ways – by time invested, by value to purpose, by outcomes of use, and by roi. multiple methods should be used to measure value, including quantitative, qualitative, and a mixture of both. no one method stands alone and the choice of methods must be tied to the mission of each specific institution. quantitative data can show roi and trends, while qualitative data can tell a story or put a personal face on data. whatever methods are used, libraries need to focus on measuring outcomes, not inputs, and use this evidence to demonstrate the role of the library in helping with the success of faculty, graduate students, and undergraduate students.   this commentary is based on a presentation for the lis dream project conference, british library, 9th july 2012, uk.   acknowledgements: the studies reported here involve many librarians around the world, who are too numerous to mention specifically, but who collectively have made these projects successful. special thanks to paula kaufmann, dean of libraries, university of illinois-urbana-champaign; bruce kingman, professor at syracuse university; and donald w. king, research professor at bryant university and all of the libvalue researchers. libvalue is funded by a grant from the u.s. institute of museum and library services. additional funding for international participation comes from, in the u.k. jisc collections and, in australia, the john metcalfe memorial foundation. an international study on roi in grants was funded by elsevier.     references   association of research libraries (2013).  libqual+: charting library service quality.  retrieved from https://www.libqual.org/home   fraser, b.t., mcclure, c.r. & leahy, e.h. (2002).  toward a framework for assessing library and institutional outcomes.  portal: libraries and the academy, 2(4), p. 505-528.    griffiths, j.m. & king, d.w. (1993).  increasing the information edge.  special libraries research series. washington, dc: special libraries association.    griffiths, j.m. & king, d.w. (2011).  a strong future for public library use and employment.  chicago: american library association.    king, d.w. (29 october 2012).  assessment of the use, value, and roi of academic library services.  paper presented at the arl library assessment conference, charlottesville, va.   libvalue.cci.utk.edu (2013).  libvalue: value, outcomes, and return on investment of academic libraries.  retrieved from http://libvalue.cci.utk.edu   machlup, f. (1979).  uses, value, and benefits of knowledge.  science communication, 1(1), p. 62-81.    matthews, j.r. (2002).  the bottom line: determining and communicating the value of the special library.  westport, ct: libraries unlimited.   matthews, j.r. (2007).  the evaluation and measurement of library services.  westport, ct: libraries unlimited.   national network of libraries of medicine (2013).  library value calculator: what is your library worth?  retrieved from http://nnlm.gov/mcr/evaluation/roi.html   oakleaf, m. (2010).  the value of academic libraries: a comprehensive research review and report.  association of college and research libraries, chicago, il.   tenopir, c. (2010).  university investment in the library, phase ii: an international study of the library’s value to the grants process.  library connect white papers.  retrieved from http://libraryconnect.elsevier.com/university-investment-library-phase-ii-international-study-librarys-value-grants-process-2010   tenopir, c. (9 july 2012).  building evidence of the value and impact of library information services: methods, metrics and roi.  keynote presentation at the dream project conference, london, uk.   tenopir, c. (2012) beyond usage: measuring library outcomes and value.  library management, 33(1/2), p. 5-13. doi: 10.1108/01435121211203275.   tenopir, c. & volentine, r. (1 february 2012).  uk scholarly reading and the value of library resources: summary results of the study conducted spring 2011.  jisc report. http://www.jisc-collections.ac.uk/reports/ukscholarlyreadingreport/     volentine, r. & tenopir, c. (2013). “value of academic reading and value of the library in academics’ own words”.  aslib proceedings. 65 (4).   volentine, r., whitson, l., & tenopir, c. (2012). “portraits of success: building personas from scholarly reading patterns”. qqml 4th annual conference proceedings. microsoft word news_registration.doc evidence based library and information practice 2007, 2:1  164 evidence based library and information practice     news    registration for 4th international evidence based library and information practice  conference      © 2007 eblip. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    registration for the 4th international  evidence based library and information  practice (eblip4) is open.  the final day for  registration at the reduced rate is april 9,  2007.    the conference on may 6‐9, 2007 in chapel  hill‐durham, north carolina, will feature  themed sessions on evidence‐based practice  in academic libraries, school library media,  healthcare, special libraries, and evidence‐ based methodology. two days of continuing  education will follow. the conference  provides a forum for the presentation of  high quality papers and posters as well as  examples of how eblip is being  implemented in library and information  settings around the globe.    the registration form, along with further  information on eblp4, is available on the  conference website at  http://www.eblip4.unc.edu/index.html.              http://creativecommons.org/licenses/by/2.0 http://www.eblip4.unc.edu/index.html article   developing and applying an information literacy rubric to student annotated bibliographies   erin e. rinto undergraduate learning librarian and assistant professor university of nevada las vegas las vegas, nevada, united states of america email: erin.rinto@unlv.edu   received: 20 mar. 2013   accepted: 23 may 2013      2013 rinto. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – this study demonstrates one method of developing and applying rubrics to student writing in order to gather evidence of how students utilize information literacy skills in the context of an authentic assessment activity. the process of creating a rubric, training scorers to use the rubric, collecting annotated bibliographies, applying the rubric to student work, and the results of the rubric assessment are described. implications for information literacy instruction are also discussed.   methods – the focus of this study was the english 102 (eng 102) course, a required research-based writing course that partners the instructors with the university librarians for information literacy instruction. the author developed an information literacy rubric to assess student evaluation of information resources in the eng 102 annotated bibliography assignment and trained three other librarians how to apply the rubric to student work. the rubric assessed the extent to which students critically applied the evaluative criteria currency, relevance, accuracy, authority, and purpose to the information sources in their annotations. at the end of the semester, the author collected up to three de-identified annotated bibliographies from each of the 58 sections of eng 102. the rubric was applied to up to five annotations in each bibliography, resulting in a total examination of 773 annotations (some sections turned in fewer than 3 samples, and some bibliographies had fewer than 5 annotations). results – the results of the study showed that students struggle with critically evaluating information resources, a finding that supports the existing information literacy assessment literature. the overwhelming majority of annotations consisted of summative information with little evidence that students used any evaluative criteria when they selected an information source. of the five criteria examined, relevance to the student’s research topic and authority were the most commonly used methods of resource evaluation, while currency, accuracy, and purpose were criteria least-often used. the low average scores on the rubric assessment indicate that students are not adequately learning how to apply this set of information literacy skills.   conclusions – the library instruction sessions for eng 102 need to move beyond the skills of choosing and narrowing a topic, selecting keywords, and searching in a library database. students also need more targeted instruction on higher-order skills, particularly how to critically evaluate and question the sources they find. the results of this assessment are being used to refocus the learning outcomes of eng 102 library sessions so that instruction can better meet student needs. the results are also being used to make the case for further collaboration between eng 102 and the university library.     introduction   it has been well-documented in the library literature that academic libraries are responsible for assessing their services, especially library instruction, in order to communicate impact and better meet student needs (rockman, 2002; avery, 2003; choinski, mark, & murphy, 2003; oakleaf & kaske, 2009). in order to intentionally design, implement, and assess information literacy instruction, it is helpful to have information about how students apply information literacy skills in practice. in particular, how do students understand and articulate the concept of evaluating information resources? does library instruction influence the decisions students make during their research process? how can the assessment of student work help practicing librarians make the most of the ubiquitous single course period instruction session?                  this research project is informed by the assessment component of a new collaboration between the lied library and the english composition program at the university of nevada las vegas (unlv). the focus of the partnership was the english 102 (eng 102) course, a required research-based writing class. in prior years, the relationship between the library and the eng 102 course was informal in nature; there was no established set of learning goals for each library session that applied directly to the learning outcomes of eng 102 nor a shared understanding of how the library session related to the larger goals of the eng 102 curriculum. no regular assessment program showed how the library instruction sessions contributed to the information literacy needs of eng 102 students. one goal of this new partnership was to introduce and execute an assessment plan for the eng 102 information literacy instruction program. the assessment plan culminated with the collection and analysis of annotated bibliographies using a rubric designed by the author to assess students’ skills with evaluating information resources.   the purpose of this case study is to demonstrate one way that rubrics can be developed and applied to student writing to show how students apply information literacy skills in the context of an authentic assessment activity. this study contributes to the information literacy assessment literature by using a rubric to assess the information literacy skills evidenced by a sample of student work from a large, high-impact undergraduate composition course. the results of this research project will allow librarians to fine-tune the single course period library instruction sessions that accompany the research component of the eng 102 course.     literature review   the assessment literature indicates that an important and ongoing trend is authentic assessment; that is, using meaningful tasks to measure student learning (knight, 2006). performance-based assignments are key ways to gauge how students are internalizing what they are taught in class. unfortunately, unless librarians are teaching full-semester courses, they rarely see the outcome of what they teach. one way that librarians can become involved in authentic assessment is to collect work samples from the students who come to the library for instruction. librarians can then evaluate the samples based on the skills that they would expect to see in student work. the results of such an assessment can inform future decisions about instruction, identifying areas where students excel or struggle and designing instruction programs that better support student learning.                 one particular method that librarians have used to assess student information literacy skills is the rubric. rubrics are advantageous assessment tools because they can be used to turn subjective data into objective information that can help librarians make decisions about how to best support student learning (oakleaf, 2007; arter & mctighe, 2001). rubrics allow an evaluation of students’ information literacy skills within the context of an actual writing assignment, supporting the notion of authentic assessment.                  in the last ten years several studies that use rubrics to assess student information literacy skills have been conducted. in 2002, emmons and martin used rubrics to evaluate 10 semesters’ worth of student portfolios from an english composition course in order to evaluate how changes to library instruction impacted the students’ research processes. this study showed that while some small improvements were made in the way students selected information resources, closer collaboration between the composition program and the library was needed (emmons & martin, 2002). choinski, mark, and murphy (2003) developed a rubric to score papers from an information resources course at the university of mississippi. they found that while students succeeded in narrowing research topics, discussing their research process, and identifying source types, they struggled with higher-order critical thinking skills. knight (2006) scored annotated bibliographies in order to evaluate information literacy skills in a freshman-level writing course. the study uncovered areas where the library could better support student learning, including focusing more on mechanical skills (database selection and use) as well as critical-thinking skills (evaluating the sources found in the databases). these studies, which used rubrics to evaluate student writing, all share similar findings—students succeed in identifying basic information if they are directly asked to do so but have difficulty critically evaluating and using academic-level sources.                  while these articles help inform how students apply information literacy skills in authentic assessment tasks, they do not provide very detailed information on how information literacy rubrics were developed and applied to student work. studies that delve deeper into the rubric creation process rectify some of these issues. fagerheim and shrode (2009) provide insight into the development of an information literacy rubric for upper-level science students, such as collecting benchmarks for graduates, identifying measurable objectives for these benchmarks, and consulting with faculty members within the discipline, but there is no discussion of how scorers were trained to use the rubric. hoffman and labonte (2012) explore the validity of using an information literacy rubric to score student writing. the authors discuss the brainstorming of performance criteria and the alignment of the rubric to institutional outcomes but there is no description of the training process for raters. helvoort (2010) explains how a rubric was created to evaluate dutch students’ information literacy skills, but the rubric was meant to be generalizable to a variety of courses and assignments, making it difficult to transfer the processes described to a single course assignment.    perhaps the most in-depth descriptions of the rubric development and training process appear in two studies by oakleaf (2007; 2009) in which rubrics were used to score student responses on an online information literacy tutorial. oakleaf describes the process for training the raters on rubric application and the ways in which that training impacted inter-rater reliability and validity (oakleaf, 2007). oakleaf (2009) gives a description of the mandatory training session. the raters were divided into small groups; the purpose of the study, the assignment, and the rubric were introduced and discussed; and five sample papers were used as “anchors” and scored during a model read-aloud of how to apply the rubric. oakleaf used maki’s 6 step-norming process to have the raters score sample papers and then discuss and reconcile differences in their scores (oakleaf, 2009; maki, 2010). this process was repeated twice on sample papers before the raters were ready to score sets of student responses on their own. oakleaf’s explanation of how to train raters on an information literacy rubric was used as the model for rubric training for this study.                 though the literature on using rubrics to evaluate information literacy skills has grown over the last decade, oakleaf’s studies remain some of the only examples of how to actually apply the rubrics in an academic library setting. thus, there is still a need for localized studies that describe the application of information literacy rubrics. this study contributes to the literature by providing a case study of developing and using rubrics to evaluate how students apply information literacy skills in their class assignments.   context and aims   context   eng 102 is the second in a two-course sequence that fulfills the english composition requirement for degree completion at unlv. eng 102 is a high impact course that sees a very large enrollment; in the fall of 2012, there were 58 sections of eng 102, with 25 students in each section. the course has four major assignments, consisting of a summary and synthesis paper, an argument analysis, an annotated bibliography, and a researched-based argument essay. the third assignment, the annotated bibliography, was the focus of this study since the eng 102 library instruction sessions have traditionally targeted the learning outcomes of the annotated bibliography project.   aims                 the author had two aims for this research project: the first was to gather evidence of how students apply information literacy skills in the context of an authentic assessment activity, and to use that information to fine-tune information literacy instruction sessions for the eng 102 course. the second aim was to fill a gap in the literature by providing a case study of rubric development and application to student work. by offering a transparent view of how the rubric was created and how raters were trained, the author hopes to provide a localized case study of the practicalities of rubric usage.   methodology   developing rubrics for information literacy   a rubric is an assessment tool that establishes the criteria used to judge a performance, the range in quality one might expect to see for a task, what score should be given, and what that score means, regardless of who scores the performance or when that score is given (callison, 2000; maki, 2010). a scoring rubric consists of two parts: criteria, which describe the traits that will be evaluated by the rubric, and performance indicators, which describe the range of possible performances “along an achievement continuum” (maki, 2010, p. 219).                  the benefits of rubrics as assessment tools are widely recognized: they help establish evaluation standards and keep assessment consistent and objective (huba & freed 2000; callison, 2000); they also make the evaluation criteria explicit and communicable to other educators, stakeholders, and students (montgomery, 2002). the most commonly cited disadvantage of rubrics is that they are time consuming to develop and apply (callison, 2000; mertler, 2001; montgomery, 2002). the advantages of the descriptive data that come from rubrics should be weighed against their time-consuming nature, and proper time should be allotted for creating, teaching, and applying a rubric.                 there is much information in the assessment literature on the general steps one can take to develop a scoring rubric. the model adapted by the author for the study consists of seven stages and was developed by mertler (2001). other examples of rubric development models can be found in arter and mctighe (2001), moskal (2003), stevens and levi (2005), and maki (2010). mertler’s model (mertler, 2001).                 1. reexamine learning outcomes to be addressed. 2. identify specific observable attributes that you want to see or do not want to see students demonstrate. 3. brainstorm characteristics that describe each attribute. identify ways to describe above average, average, and below average performance for each observable attribute. 4. write thorough narrative descriptions for excellent and poor work for each individual attribute. 5. complete rubric by describing other levels on continuum.                 6. collect student work samples for each level.                 7. revise and reflect.   in accordance with mertler’s model, the author began the process of designing the eng 102 information literacy rubric by defining the learning outcomes that needed to be addressed. the learning outcomes that the composition program identified for the annotated bibliography assignment were used as a starting point for developing the rubric criteria. the annotated bibliography assignment has six information-literacy-centered learning outcomes, including choosing and narrowing a research topic, designing search strategies, conducting academic research, evaluating sources, writing citations, and planning a research-based argument essay. many of these outcomes require students to use higher-order critical thinking skills, which were identified as areas of difficulty in previous studies that used information literacy rubrics, so the author was particularly interested in assessing those areas. the author then mapped each of the six outcomes to the association of college and research libraries information literacy competency standards for higher education and used a set of sample annotated bibliographies from a previous semester to identify attributes in student work that represented a range of good and poor performances for each of the six criteria (acrl, 2000). next, the author created written descriptions of the aspects of performances that qualified them as good or poor, and filled in the rubric with descriptions of “middle-range” performances. this first draft resulted in three rubrics that were shared with other instruction librarians and the eng 102 coordinator during a rubric workshop led by an expert in the field who came to unlv’s campus to help support library assessment efforts.   the discussions during the workshop led to substantial revision of the rubrics’ content and format. language was standardized and clarified, with careful attention paid to using parallel structure. in addition, efforts were made to ensure that only one element was assessed in each criterion and that the performance indicators on the rubric were mutually exclusive. maki’s checklist for evaluating a rubric proved to be a useful tool for identifying areas of ambiguity and overlap (maki, 2010). the author also refocused the scope of the project, which was too broad for the first stage of the assessment project. instead of addressing all six information literacy learning outcomes identified for eng 102, the author decided to start with just one outcome: source evaluation.    for the source evaluation rubric, five criteria were selected to assess: currency, relevance, accuracy, authority, and purpose. these criteria were drawn from a unlv libraries’ handout that aids students in evaluating the credibility of a resource and walks them through how to decide if a source is useful for their project. the rubric had three performance indicators to represent the range of student work in terms of how the student applied the evaluative criteria: “level 0—not evidenced,” “level 1—developing (using evaluation criteria at face value),” and “level 2—competent (using evaluation criteria critically)” (see figure 1).   the goal of the rubric was to identify which evaluative criteria students were not using at all, which they were using in only a shallow way, and which criteria students were using as critical consumers of information.  in order to gather this level of detail, the author decided that the rubric would be used to evaluate the individual annotations in each bibliography, not the bibliography as a whole.  this meant that the rubric would be applied up to five times for each student’s paper, since students were to turn in at least five annotations.       figure 1 source evaluation rubric     applying the rubrics                 collecting and preparing student samples    the eng 102 coordinator, a faculty member in the composition department, had already established a method for collecting a sampling of student work every semester, so the author was able to receive copies of the annotated bibliography assignment from this sampling. the eng 102 coordinator uses a form of systematic sampling where the work from every 5th, 10th, and 15th student in each section is collected (creswell, 2005). this means that at least three papers were to be collected from each of the 58 sections. in all, the author received a total of 155 annotated bibliographies, representing 10% of the total eng 102 student population (not every section turned in the required 3 samples). in accordance with irb protocol, the eng 102 coordinator de-identified all papers before the author received them for this study.   the author read through the first 50 samples received in order to find sets of anchor papers to use during the rubric training sessions, as was recommended by oakleaf (2009).  anchor, or model, papers were selected as examples for the training session because they reflected a range of high, medium, and low scoring student work.  fifteen annotated bibliographies were selected and grouped into three sets so as to reflect a variety of student responses to the assignment.    preparing for the training session: issues of inter-rater reliability   the author selected three other librarians to help score the student work samples. the other three librarians were trained on how to apply the rubric in a series of two 2 hour sessions.    inter-rater reliability was an issue of interest for this project because four librarians were involved in the rating process. inter-rater reliability is the degree that “raters’ responses are consistent across representative student populations” (maki, 2010, p. 224). calculating inter-rater reliability can determine if raters are applying a rubric in the same way, meaning the ratings can statistically be considered equivalent to one another (cohen, 1960; moskal, 2003; oakleaf, 2009). because the sample of student work resulted in over 700 individual annotations, the author wanted to determine if this total could be equally divided between the four raters, resulting in each person having to score only a quarter of the samples. if, during the training sessions, the four raters could be shown to have a shared understanding of the rubric, as evidenced through calculating inter-rater reliability statistics, then only the recommended 30% overlap between papers would be needed (stemler, 2004).   in order to calculate inter-rater reliability for this study, the author used agreestat, a downloadable microsoft excel workbook that calculates a variety of agreement statistics. due to the fact that there were four raters, fleiss’s kappa and conger’s kappa were used as the agreement statistics for this study. these statistics are based on cohen’s kappa, a well-established statistic for calculating agreement between two raters. fleiss’s kappa and conger’s kappa modify cohen’s kappa to allow for agreement between multiple raters (stemler, 2004; oakleaf, 2009; fleiss, 1971; conger, 1980; gwet, 2010). the landis and koch index for interpreting kappa statistics was used to determine if sufficient agreement had been reached. a score of 0.70 is the minimum score needed on the index for raters to be considered equivalent (landis & koch, 1971; stemler, 2004).    table 1 kappa index kappa statistic strength of agreement <0.00 poor 0.00-0.20 slight 0.21-0.40 fair 0.41-0.60 moderate 0.61-0.80 substantial 0.81-1.00 almost perfect     the training session    the rubric training sessions followed maki’s 6-step process: first, the author introduced the annotated bibliography assignment and the learning outcomes that were to be assessed. she handed out copies of the rubric and explained the criteria and performance indicators. the author then conducted a “read-aloud” with two of the anchor annotated bibliographies, reading through the annotations and articulating how she would score them based on the rubric. once the librarians felt comfortable with the application of the rubric, each librarian individually scored a practice set of three bibliographies. differences in scoring were identified, discussed, and reconciled (maki, 2010; oakleaf, 2009).    at this point, the author used the statistical software agreestat to calculate the group’s inter-rater reliability. since the initial scores were very low on the landis and koch index, the scoring and discussing process was repeated twice more. the inter-rater reliability scores for each round were as follows:   inter-rater reliability greater than “fair” was not reached during three rounds of scoring; the numbers were well below the recommended 0.70 score. the author decided that, since the raters could not be considered equivalent in their application of the rubric, the papers could not be divided evenly between them. instead, each bibliography would be scored twice, by two different raters, and the author would reconcile any differences in scores, a suggested method for resolving dissimilar ratings known as “tertium quid,” in which the score of an adjudicator is combined with the closest score of the original raters, and the dissimilar score is not used (johnson, penny, and gordon, 2008, p. 241). the librarians had two weeks to score their set of bibliographies, at the end of which time the author recorded all scores.    results   a total of 773 annotations were scored for the study. the following table shows the number of annotations that applied each of the evaluative criteria, and to what degree:   for three of the five evaluative criteria, the majority of the annotations received a level 0 score, meaning they did not provide any evidence of using the criterion in question. for currency, 686 (88%) of the annotations received a level 0; for accuracy, 575 (74%) received a level 0; and for purpose, 615 (79.5%) of the annotations were scored level 0. though students did better applying the criteria relevance and authority, a substantial percentage of students did not use these criteria either—330 (42.6%) annotations received a level 0 for relevance and 268 (34.6%) annotations received a level 0 for authority. in fact, in every instance except authority, level 0 was the most frequent score out of the three possible performance levels.    the rubric was designed in such a way that a level 1 score indicates that students are aware that a particular evaluative criterion exists and attempt to use it to assess the usefulness and appropriateness of a source, and a level 2 score indicates that students apply these criteria in a critical way. when level 1 and 2 scores are combined, it is apparent that 87 annotations (11%) use the criterion currency to evaluate the source, 158 annotations (19.7%) use the criterion purpose to evaluate the source, 198 annotations (25.6%) use the criterion accuracy to evaluate the source, 443 annotations (57.2%) use the criterion relevance to evaluate the source, and 505 annotations (65.2%) use the criterion     table 2 interrater reliability from training rounds     table 3 scores for evaluative criteria     authority to evaluate the source. in the instances of relevance and authority, it is evident that the majority of students are at least aware that these criteria should be considered when selecting an information source.   however, there were no instances in which the majority of students applied the evaluative criteria in a critical way, the performance required to receive a level 2 score. for currency, 17 (2%) of annotations were scored level 2; for accuracy, 14 (1.8%) annotations were scored level 2; and even fewer for purpose—only 6 annotations, less than 1% , used the criteria critically. however, in the case of relevance and authority nearly a quarter of the annotations applied the criteria critically, at 180 annotations (23.2%) and 189 annotations (24.4%), respectively.    examining the annotations within the context of the bibliographies in which they appeared provides information on how students applied these evaluative criteria across their entire papers. each of the 155 papers examined contained up to five annotations, meaning each of the evaluative criteria could receive a score of up to ten points—two points per annotation. the author totaled the scores each paper received for the application of the criteria currency, relevance, accuracy, authority, and purpose. the following table contains the average total score for each evaluative criterion, the maximum score for each criterion, and the standard deviation—how closely students clustered to the average score.   it is interesting to note that the mean scores for currency, accuracy, and purpose were quite low (0.66, 1.35, and 1.15 out of 10, respectively), and that the standard deviations for these criteria were also low (1.47, 1.61., and 1.67, respectively). a small standard deviation means that most of the scores are grouped very close to the average score, with only a few outlying points (hand, 2008). thus, even though the range of scores given for these criteria was high, the high scores were outliers—the majority of scores given for each annotation was actually very near the low mean scores. this further reinforces the notion that students are consistently failing to critically apply the evaluative criteria currency, accuracy, and purpose in their annotations.    discussion   overall, it appears that the results of this study support the information literacy assessment literature in terms of students struggling with the application of critical thinking skills. in their rubric assessment of student writing, emmons and martin (2002) indicated that students had particular difficulties with identifying the purpose of different kinds of sources; knight (2006) also stated that source evaluation was a particular issue in student writing. the eng 102 annotated bibliographies reinforce these ideas—the only evaluative criteria students considered on a consistent basis was the author of their source and how that source related to their research topics. this study therefore contributes to the literature on student application of critical thinking skills by providing specific data about the degree to which students evaluate their information resources.   table 4 breakdown of scores for evaluative criteria in student papers   a practical implication of this research is an indication of where reinforced information literacy instruction might be beneficial to students. it is clear that students need more support in learning how to identify whether a source is appropriate and to articulate how it contributes to their argument. this is a complex skill set and needs to be taught in both the eng 102 classroom as well as in the library instruction session. the first step the author took was to share the results of the rubric assessment with the eng 102 coordinator and to initiate a conversation about what elements of source evaluation the composition program would like to emphasize in future semesters. the author also plans to share these results with the eng 102 instructors at their fall orientation meeting and to stress areas where the library can help improve student performance, as well as steps the eng 102 instructors can take in their own classrooms to help students become more critical of their information sources. the author had previously created an eng 102 instructor portal—a website available to eng 102 instructors that provides information literacy activities and assignments to support the learning outcomes of eng 102. the portal was highly utilized throughout the 2012-2013 school year, but the learning outcome of source evaluation was perhaps buried beneath a myriad of other learning outcomes, and was therefore not sufficiently highlighted. the author plans to streamline the instructor portal content to emphasize this skill set, and to discuss with eng 102 instructors at orientation the possible ways they can integrate these activities into their teaching.                  in addition, the library needs to target source evaluation more consistently during the instruction sessions for eng 102. according to a survey of the librarians who worked with eng 102, source evaluation was not covered across all sections that came to the library, and, when it was taught, it usually consisted of briefly showing the students the source evaluation handout at the end of the session. the results of this study clearly demonstrate that this is not enough instruction on source evaluation, and the author has met with the instruction librarians to discuss ways in which we can better teach source evaluation in our sessions. the group agreed that in order to more fully instruct on source evaluation, other learning outcomes that are currently taught in the library session should be moved out of the face-to-face classroom and into the virtual one. the author is in the process of creating a web tutorial that will have students work through selecting and narrowing a research topic prior to coming to the library for instruction. the class time formerly spent on topic exploration can then be better spent on helping students navigate through search results and apply evaluative criteria to their potential sources.    one limitation to this study is the restricted face-to-face time that librarians have with students; the library can only support teaching one 75 minute instruction session for each section of eng 102. the use of a web tutorial to free up class time, as well as providing eng 102 instructors with more resources on source evaluation is intended to alleviate some of this pressure. the library is also limited in terms of what individual eng 102 instructors emphasize when they assign the annotated bibliography project. since there is no shared rubric that eng 102 instructors have to use for the annotated bibliographies, it is plausible that there is a large amount of variability in what the classroom teachers instruct their students to do in their annotations. including eng 102 instructors in the conversation about how students struggle with source evaluation—via fall orientation and the instructor portal—is one step in improving this issue.    finally, the process of applying the information literacy rubric can also be improved in the future. the training sessions failed to establish sufficient inter-rater reliability, and this was in large part due to not enough time allotted for training, particularly in discussing differences in scores. two 2-hour training sessions (over two days) was all the group could commit to the project, but in the future, the author will emphasize the importance of a full day of training so that more time can be spent working through differences.   additional factors that might influence student performance should also be examined. the author did gather information about student performances based on their section type—regular, themed, or distance education—as well as differences in scores between eng 102 classes that came to the library for instruction versus those that did not; though it was required to schedule a library instruction session, ten sections did not come to the library. more in-depth statistical analysis needs to be done in order to interpret that information and the impact, if any, that these factors have on student performance.    conclusions   this study intended to accomplish two goals. the first aim of this study was to gather evidence about how students apply information literacy skills in the context of an authentic assessment activity, and to be able to use that information to establish a baseline from which to design future library instruction. the overall scores on the rubric indicate that there is considerable need for further instruction on the concept of how and why to apply evaluative criteria when selecting information resources. clearly, one instruction session is not enough time to sufficiently introduce this concept—teaching source evaluation is a responsibility that should be shared between the eng 102 instructors and the librarian in the instruction session. working with the eng 102 program to help instructors embed this information literacy skill set within their own classrooms is an important first step in helping students succeed.    a second aim of this study was to provide a case study of how to create and apply a rubric to evaluate student information literacy skills. to that end, the author provided a transparent view of the process of rubric development and training scorers to use the rubric. several lessons were learned during this process, such as how to move forward when inter-rater reliability was less than desirable. it is the author’s hope that others will build upon the successes and learn from the limitations of this study in order to continue the important discussion of authentic assessment of information literacy skills and the ways in which academic libraries can better support student learning.   acknowledgements   the author would like to thank anne zald, cheryl taranto, and samantha godbey for their assistance in scoring student work using the information literacy rubric; ruby fowler for providing the de-identified copies of annotated bibliographies; jen fabbi for her guidance throughout the project; and megan oakleaf for leading the assessment workshop that led to the revision of the first draft of the rubric.      references   avery, e.f. (2003). assessing information literacy instruction.   in e.f. avery (ed.), assessing student learning outcomes for information literacy instruction in academic institutions (pp. 1-5).  chicago, il: acrl.   arter, j. a. & mctighe, j. (2001). scoring rubrics in the classroom: using performance criteria for assessing and improving student performance. thousand oaks, ca: corwin press.   callison, d. (2000). rubrics. school library media activities monthly, 17(2), 34-36, 42.   choinski, e., mark, a. e., & murphey, m. (2003). assessment with rubrics: efficient and objective means of assessing student outcomes in an information resources class. portal: libraries & the academy, 3(4), 563.   cohen, j. (1960). a coefficient of agreement for nominal scales.  educational and psychological measurement, 20(1), 37-46.   conger, a.j. (1980). integration and generalization of kappas for multiple raters.  psychological bulletin, 88(2), 322-328.   creswell, j. w. (2005). educational research: planning, conducting, and evaluating quantitative and qualitative research (2nd ed.). upper saddle river, nj: merrill.   emmons, m., & martin, w. (2002). engaging conversation: evaluating the contribution of library instruction to the quality of student research. college & research libraries, 63(6), 545-60.   fagerheim, b. a. & shrode, f.g. (2009). information literacy rubrics within the disciplines.  communications in information literacy, 3(2), 158-170.   fleiss, j.l. (1971). measuring nominal scale agreement among many raters. psychological bulletin, 76(5), 378-382.   gwet, k. (2010). handbook of inter-rater reliability, 2nd ed. gaithersburg, md: advanced analytics.   hand, d.j. (2008). statistics: a very short introduction.  oxford: oxford university press.   van helvoort, j. (2010). a scoring rubric for performance assessment of information literacy in dutch higher education.  journal of information literacy, 4(1), 22-39.   hoffman, d. & labonte, k. (2012). meeting information literacy outcomes: partnering with faculty to create effective information literacy assessment.  journal of information literacy, 6(2), 70-85.   huba, m. e. & freed, j. e. (2000). learner-centered assessment on college campuses: shifting the focus from teaching to learning. boston: allyn and bacon.   information literacy competency standards for higher education. (2000). retrieved august 23, 2013 from http://www.acrl.org/ala/mgrps/divs/acrl/standards/standards.pdf   johnson, r. l., penny, j. a., & gordon, b. (2009) assessing performance: designing, scoring, and validating performance tasks. new york: the guilford press.   knight, l. a. (2006). using rubrics to assess information literacy. reference services review, 34(1), 43-55. doi: 101108/00907320510631571   landis, j. r., & koch, g. g. (1977). the measurement of observer agreement for categorical data. biometrics, 33(1), 159-174. retrieved august 23, 2013 from http://www.jstor.org/stable/2529310   maki, p. (2010). assessing for learning: building a sustainable commitment across the institution (2nd ed.). sterling, va: stylus.   mertler, c. a. (2001). designing scoring rubrics for your classroom. practical assessment, research & evaluation, 7(25).   montgomery, k. (2002). authentic tasks and rubrics: going beyond traditional assessments in college teaching. college teaching, 50(1), 34-39. doi: 10.1080/87567550209595870   moskal, b. (2003). recommendations for developing classroom performance assessments and scoring rubrics. practical assessment, research and evaluation, 8(14).   oakleaf, m. (2007). using rubrics to collect evidence for decision-making: what do librarians need to learn? evidence based library and information practice, 2(3), 27-42.   oakleaf, m. (2009). using rubrics to assess information literacy: an examination of methodology and interrater reliability.  journal of the american society for information science and technology, 60(5), 969-983. doi: 10.1002/asi.21030   oakleaf, m. & kaske, n. (2009). guiding questions for assessing information literacy in higher education. portal: libraries & the academy, 9(2), 273-286. doi: 10.1353/pla.0.0046   rockman, i. f. (2002). strengthening connections between information literacy, general education, and assessment efforts. library trends, 51(2), 185-198.   stemler, s. e. (2004). a comparison of consensus, consistency, and measurement approaches to estimating interrater reliability. practical assessment, research & evaluation, 9(4).   stevens, d. d. & levi, a. (2005). introduction to rubrics :an assessment tool to save grading time, convey effective feedback, and promote student learning (1st ed.). sterling, va: stylus. news/announcements   phd studentships at university of salford, uk      2014. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   the university of salford, school of nursing, midwifery and social work is currently advertising a number of full time phd studentships, in information related topics.  the school of nursing, midwifery, social work & social sciences is a dynamic school with a commitment to lifelong learning. with around 200 members of academic staff who are experts in a wide range of disciplines including nursing, midwifery, social work, social policy, counselling and psychotherapy, sociology and criminology, we are able to offer an excellent and comprehensive educational experience. the school has a lively and very successful research community with extensive academic and clinical collaborations and nearly 100 doctoral students. the rich knowledge and expertise gained from our research activities helps to inform health and social policy and practice both nationally and internationally.   the school of nursing, midwifery, social work and social sciences are looking for research proposals in a wide range of subject areas, including: information management for healthcare; digital health and crime and justice informatics.  studentships are open to uk nationals and the closing date for applications is 31st march 2014.  further information is available from http://www.salford.ac.uk/study/postgraduate/fees-and-funding/funded-phd-studentship/pathway-to-excellence-studentships/schools-and-colleges/school-of-nursing,-midwifery-and-social-work   natural language processing for virtual reference analysis research article   natural language processing for virtual reference analysis   ansh sharma emerging professional in application development university of toronto scarborough library toronto, ontario, canada email: ansh.sharma@mail.utoronto.ca   kathryn barrett social sciences liaison librarian university of toronto scarborough library toronto, ontario, canada email: kathryn.barrett@utoronto.ca   kirsta stapelfeldt head, digital scholarship unit university of toronto scarborough library toronto, ontario, canada email: kirsta.stapelfeldt@utoronto.ca   received: 29 july 2021                                                               accepted: 14 dec. 2021      2022 sharma, barrett, and stapelfeldt. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30014     abstract   objective – chat transcript analysis can illuminate user needs by identifying common question topics, but traditional hand coding methods for topic analysis are time-consuming and poorly suited to large datasets. the research team explored the viability of automatic and natural language processing (nlp) strategies to perform rapid topic analysis on a large dataset of transcripts from a consortial chat service.   methods – the research team developed a toolchain for data processing and analysis, which incorporated targeted searching for query terms using regular expressions and natural language processing using the python spacy library for automatic topic analysis. processed data was exported to tableau for visualization. results were compared to hand-coded data to test the accuracy of conclusions.   results – the processed data provided insights about the volume of chats originating from each participating library, the proportion of chats answered by operator groups for each library, and the percentage of chats answered by different staff types. the data also captured the top referring urls for the service, course codes and file extensions mentioned, and query hits. natural language processing revealed that the most common topics were related to citation, subscription databases, and finding full-text articles, which aligns with common question types identified in hand-coded transcripts.   conclusion – compared to hand coding, automatic and nlp processing approaches have benefits in terms of the volume of data that can be analyzed and the time frame required for analysis, but they come with a trade-off in accuracy, such as false hits. therefore, computational approaches should be used to supplement traditional hand coding methods. as nlp becomes more accurate, approaches such as these may widen avenues of insight into virtual reference and patron needs.     introduction   librarians strive to understand patron needs in order to provide efficient and accurate reference services. reviewing questions from different reference channels offers an opportunity to identify common question types and to ensure that library staff have sufficient training and familiarity with resources to answer them. since virtual reference software preserves a record of the interaction, librarians can analyze chat transcripts to understand the breadth and frequency of question topics (chen & wang, 2019). researchers have traditionally used qualitative methods involving hand coding to determine the most common question types on chat services, but these methods are time consuming and not suited to analyzing large datasets. consequently, some researchers have begun to explore computational analysis for large corpora (paulus & friend wise, 2019a).   automated topical analysis of chat transcripts has not received much attention in the library and information science literature (ozeran & martin, 2019). consequently, our research team at the university of toronto scarborough (utsc) library sought to explore the viability of automatic and natural language processing (nlp) methodologies for chat transcript analysis. the team hoped to uncover how these methods might streamline or complement other forms of topical analysis. as the researchers were all members of the utsc library’s data and digital scholarship committee, a secondary goal of the project was to better understand patron needs concerning data and digital scholarship topics. the project received support from the library’s emerging professionals program, which extends employment to students enrolled in utsc’s computer science co-op program to expose them to digital scholarship and information science.   utsc is the eastern campus of the university of toronto system, offering over 250 programs to 13,843 students, rendering it similar in size to a mid-sized canadian university. utsc is served by one library, which is part of the larger university of toronto libraries (utl) system made up of over 40 libraries. utl participates in the ask a librarian virtual reference service, which is offered by the ontario council of university libraries (ocul), a consortium representing the libraries of all 21 universities in ontario, canada. the service reaches approximately 400,000 full-time equivalent students across the province. it is managed by scholars portal (sp), the service arm of ocul. ask a librarian connects students, faculty members, and researchers with real-time assistance through chat for 67 hours per week during the academic year. staffing is managed through a collaborative model, in which participating libraries provide staffing hours relative to their student populations and service usage. on evenings and weekends, staffing is supplemented by part-time virtual reference operators hired by ocul. ask a librarian is a busy service point; it handles over 25,000 chats per year, with roughly a third of chats originating from the university of toronto, ocul’s largest member.   the high level of activity on the ask a librarian service offers a large dataset to test the viability of natural language processing. since a team of utl and sp researchers had recently hand-coded a sample of chats by question type as part of an assessment project, there was also an opportunity to compare the results of automatic and manual approaches to topic analysis on the same service. overall, the project sought to answer the following research questions:   can a toolchain using automatic and natural language processing effectively perform broad topic analysis of chat transcripts? what types of insights can be gathered from the processed data, regarding patrons, chat operators, and users’ questions? to what extent are patrons asking questions related to data and digital scholarship topics? how does natural language processing compare to traditional hand coding methods in terms of accuracy?   literature review   since the emergence of virtual reference services in the late 1990s, chat reference has grown in popularity and become a core library service (matteson et al., 2011; schiller, 2016). almost half of universities and colleges in north america offer chat services, with roughly a quarter providing them through a consortium (yang & dalal, 2015). many users prefer virtual reference to face-to-face reference due to the personalized nature of the service, as well as its convenience and immediacy (chow & croxton, 2014; connaway & radford, 2011). given the popularity of chat reference services, it is essential to continually evaluate and improve them to ensure service effectiveness and quality. chat reference software typically offers detailed metadata about chat interactions, granular usage statistics, and complete transcripts of conversations, all of which can be used for assessment purposes. chat transcripts are a particularly rich source of data, as they can illuminate the topics being discussed within chats, helping librarians to identify user needs and adjust reference services and staff training in response (chen & wang, 2019).   chat transcripts can be analyzed using qualitative or quantitative approaches. qualitative methods take an iterative approach to analysis; multiple readings of the data allow patterns to emerge, and the patterns are categorized to answer research questions (paulus & friend wise, 2019b). for example, one common qualitative method, thematic analysis, allows researchers to understand the topic of online conversations. researchers code categories and themes that emerge from the transcripts of chats using their own interpretations, rather than an a priori framework. in contrast, quantitative approaches require that a predetermined theoretical framework, such as a coding scheme, be chosen at the start of the project and used to find patterns in the data, transforming the chat into a numerical representation (paulus & friend wise, 2019a). once a quantitative picture of the chat has been generated, it can be transformed into variables or used for calculations. for instance, content analysis requires researchers to manually assign numerical values to messages based on the degree to which characteristics are present, such as the frequency of certain words, using a coding scheme. while these methods permit an in-depth understanding of chats, they require significant researcher training, and they are time-intensive and inefficient for large chat datasets (chen & wang, 2019).   new computational methods are being explored to conduct content analysis in an automated fashion at scale using natural language processing techniques (paulus & friend wise, 2019a). for example, topic modeling is a method of computational analysis in which a collection of documents, such as chat messages or transcripts, are represented as a set of topics. the topics are identified through statistical analysis by examining patterns of word co-occurrence within the corpus. there are several algorithms for topic modeling that can reveal the semantic structure of the collection, with latent dirichlet allocation being the most common (kherwa & bansal, 2019). topic modeling has been used across numerous domains, including healthcare (wang et al., 2016), education (afacan adanir, 2019; willis et al., 2017), and technology (bulygin et al., 2018). notably, it has been employed to understand communications with customers in fields such as banking (hristova, 2021; pronoza et al., 2018).   the body of scholarship on computational analysis in the library and information field is limited, as most researchers have approached chat transcript analysis through traditional hand coding methods (ozeran & martin, 2019). several studies have used a semi-automatic approach for chat transcript analysis. schiller (2016) explored the learning taking place on wright state university’s chat reference service using a two-step transcript coding process. first, transcripts with the most words (representing 1% of the data) were manually coded, resulting in a codebook with five main coding categories. then the remaining transcripts were automatically coded using a text mining software that queried and extracted text strings matching each code category. bungaro, muzzupapa, and tomatis (2017) used a semi-automatic approach for selection and categorization of chats at the university of turin. the researchers designed a script to extract blocks of text beginning with keywords related to the natural sciences to understand the extent of questions addressed to their psychology library, then manually categorized chats according to the read scale and reviewed how the chat evolved throughout the reference interview.   a limited number of studies have used purely algorithmic analysis methods for library chats. chen and wang (2019) used latent dirichlet allocation topic modeling to extract topics from chat transcripts from carnegie mellon’s library, then applied vosviewer, a network analysis tool, to generate a term map, and found that the most prominent chat topics discussed library resources. kohler (2017) examined chats from rockhurst university’s greenlease library using natural language processing techniques. they ran three different topic extraction algorithms to explore common questions and determined that non-negative matrix factorization (nmf) was the best topic extractor, due to the even distribution of its topical clusters and the clarity of topic descriptors. latent dirichlet allocation and latent semantic analysis had topical distributions that were much more skewed. ozeran and martin (2019) also tested different algorithms for topic modeling using chat data from the university of illinois at urbana-champaign, determining that latent dirichlet allocation, phrase-latent dirichlet allocation, and non-negative matrix factorization had the most promise for large datasets. the researchers called on other libraries to employ similar approaches on their own chat reference services. this study answers the call by using natural language processing techniques to identify the frequency of topics within a busy consortial chat reference service in the province of ontario.   methods   in order to access chat transcripts from the ask a librarian chat service, we submitted a formal research data request which was approved by ocul’s ask a librarian research data working group. the ask a librarian coordinator then provided 132,263 transcripts in the comma-separated value (.csv) file format, covering the period of august 2016 to january 2021. due to the possible presence of identifiable information within the dataset, transcripts had to be anonymized, and careful attention had to be paid to data custody throughout the project. all data was uploaded to a secure server operated by the utsc library’s digital scholarship unit (dsu) for processing using a python toolchain developed by one of the authors, ansh sharma. processed data was then exported to additional .csvs and used to generate a series of dashboards using the popular data visualization software tableau. note that private servers and fully licensed versions of tableau were used for this part of the process as per the agreements with ocul for use and protection of the data.   processing   each record in the chat transcript .csv files provided metadata (table 1) as well as the full-text interaction between patron and chat operator.   table 1 source chat metadata fields chat identification number (id)   a unique identifier applied to the chat. guest identification string   a unique string generated by the chat platform to identify the guest. protocol used to initiate the chat (web)   the network protocol type used to begin the chat. queue the queue through which the chat was initiated  (institutional affiliation). profile the back-end information displayed to operators during the chat; corresponds to the queue field. date and time date and time the chat was initiated.   wait duration time (in seconds) the patron waited for an operator to pick up the chat.   duration of the chat   how long the chat lasted, in seconds. operator data a string of text containing the operator’s name and institution.   ip address of the patron identifiable information about the patron’s location. referrer link   the specific url of the webpage used to initiate the chat. chat log html content of the interaction between patron and operator (the chat itself).   a toolchain developed by sharma processed the data and exported a .csv or spreadsheet of “hits” on relevant information, including additional metadata. the resulting metadata categories for the processed data are provided in table 2. new metadata added as a result of processing has been indicated.   table 2 output chat metadata fields chat identification number (id)   a unique identifier applied to the chat. guest identification string   a unique string generated by the chat platform to identify the guest. protocol used to initiate the chat (web)   the network protocol type used to begin the chat. queue the queue through which the chat was initiated  (institutional affiliation). profile the back-end information displayed to operators during the chat; corresponds to the queue field. date and time date and time the chat was initiated. wait duration time (in seconds) the patron waited for an operator to pick up the chat.   duration of the chat   how long the chat lasted, in seconds. referrer link   the specific url of the webpage used to initiate the chat. referrer domain a the web domain of the url used to initiate the chat. used to identify sources of traffic. operator institution a the institution of the operator. uoft operator role a if the operator is from uoft, their uoft-specific role. uoft operator campus a if the operator is from uoft, the campus they are affiliated with. hit type a the type of "hit," or identified text, observed. hit a the specific text identified within the transcript. hit context a the context where the hit is identified. sent by a the user who sent the text in which the hit was identified. may either be the patron or the operator of the chat. proper noun classification a if the hit is a proper noun, the type of proper noun it is as identified by the spacy library. a new metadata added as part of processing.   this approach allowed the team to anonymize most of the data, as the name and username of each operator, ip address of each patron, and all complete chat transcripts were removed upon export, leaving only the metadata and hits found within the chat transcripts. beyond anonymization, additional metadata was created as a result of the processing script. as chats could be launched from several parts of a website, obtaining the distribution of referral urls while ignoring any subdirectories allowed the team to better aggregate categories for understanding where patrons accessed the chat service. for example, the link https://onesearch.library.redacted.ca/ask was shortened to https://onesearch.library.redacted.ca/ and extracted as a separate column under the header "referrer domain." the institutional affiliations of all operators were also recorded in a separate column, based on a suffix in the operator’s username that corresponded to their home institution. a list of operators relevant to the university of toronto queue along with their roles (e.g., librarian, library technician, student worker) was precompiled. this key was used in processing so that any operators found in this list had their roles identified as part of the metadata. to enable differentiation between patron and operator, metadata about whether individual messages were sent by patron or operator was programmatically determined based on the location of the guest’s identification string.   the chat logs themselves were separated and programmatically analyzed. a chat of relevance would become one or more rows in the output spreadsheet, which was structured around "hits'' rather than "chats." each row identified its hit type, and the unique chat id was maintained, allowing for the team to preserve an understanding of the context of hits and the number of chats implicated in the results set. the specific term that was located was also recorded in metadata, as well as a small snippet of the text surrounding the term, to aid in contextual analysis of the hit and to assess the validity of the hit.   cleaning   a dictionary of "canned" or automatic phrases used by the operator to respond to the patron was developed and such responses were identified and removed from the corpus. system and automatically generated messages were excluded from all searches.   targeted searching (regular expressions)   to answer research question 2a, regarding the extent to which chat patrons are asking about data and digital scholarship topics over chat, a list of dataand digital scholarship-related query terms (identified in the appendix) was prepared. regular expressions were used to identify chat logs with these search terms. to reduce the occurrence of false positives in detection, terms were only recorded if appropriate sentence structure was used and the term was not part of a larger word. for example, the term git would register as a hit in the sentence "git version control" and be recorded, but such a hit would not occur in the sentence "legitimate.” hits from targeted searching were recorded with the identifying tag “query.” all queries except those for a master’s in science degree were conducted in a case-insensitive format. regular expressions also enabled the discovery of course codes, which follow a reliable pattern of concatenated letters and numbers. the script would check if any text had identical formatting to that of a course code from the university of toronto. the regular expression was adjusted to match course codes from all three campuses: st. george, mississauga, and scarborough. any hits found were recorded with the hit type “course code.”   the mention of potential file extensions was of interest when analyzing transcripts. a regular expression was used to search through each message and identify patterns like those of file extensions. the regular expression searched for two to five lowercase letters preceded by a "." and succeeded by a space or the end of a sentence. such extensions are also commonly seen in web domain names, so common domain names such as .com, .ca, and .org were not included as file extensions. any hits found were recorded with the hit type "file extension."   as part of the process, the query terms needed to be modified and qualified in the following ways:   ·        the terms "utsc.utoronto.ca" and "utoronto.ca" were intended to be recorded as parts of urls, so the regular expression pattern was adjusted when searching for these terms to unconditionally include any mention of these terms. ·        to group together matches of "library," "librarian," and any similar terms, the term "librar" was used for querying. ·        a specific string search was used to group together "reference", "references", and similar terms. a similar process was used for the terms "citation" and "protocol." ·        the terms "nvivo" and "nvivohub" were grouped together. ·        an alternative regular expression was used to query abbreviations of a master’s degree. ·        an alternative regular expression was used to query abbreviations of a master's in science degree.   natural language processing   while refining the approach, it was determined that it might be possible to use the popular spacy python library to do automated analysis of the chat transcripts and surface common topics in the corpus; it is believed that this helps in determining the ratio of common topics in chat transcripts. each transcript was analyzed by an english natural language processing pipeline (referred to as "en_core_web_sm" by spacy) to analyze potential proper nouns, and each proper noun was recorded as a hit with hit type "proper noun." additionally, the type of proper noun identified by spacy was also recorded. proper nouns were analyzed by the pipeline based on the full context of the chat transcript. the usage of an additional library (gensim) to analyze topic keywords was considered; however, its functionality was very similar to that of spacy in identifying proper nouns. all remaining data were exported to a tableau software workbook to be visualized in multiple interactive ways. the types of insight gleaned from the visualizations developed were broad in scope.   results   using the processed data, the team was able to derive simple counts like the volume of chats from the queues for each participating institution. the data also illustrated the sharp growth in chat activity at the beginning of the covid-19 pandemic across all queues. the team was also able to determine the percentage of chats that were answered by librarians, technicians, and student workers in the university of toronto system. tableau provides affordances for data to be reviewed by time period. this approach permits a deeper understanding of the chat activity of different types of staff, which has implications for training and scheduling.   the team also determined the volume of chats responded to by operators at each of the participating institutions across the consortium, as well as the alignment of the institutional affiliations of users and operators across the service. consequently, it was possible to understand the extent to which university of toronto questions were responded to by local operators.   the top referring urls for the service over time were also identified, with and without subdomains, as well as the most common course codes in chat logs over time. course code data were illuminating, revealing which departments inspired the largest number of course-specific questions, and from which campus. for example, users identifying themselves as coming from a particular course came overwhelmingly from biology departments over the time period represented by the corpus being analyzed.   top query hit terms were "apa" and "reference." top popular proper nouns from the chats included popular databases, such as "proquest," suggesting that many patrons are sourcing help with vendor-provided databases. most of the terms in our targeted search pertaining to data and digital scholarship topics were not discovered in the corpus. terms that did appear within conversations mainly related to specific software for which the library negotiates licensing or that is used in research, such as spss. in addition, the census was a topic of several chats, suggesting that sourcing government-published data is of interest to patrons using the chat service. interest in the census may have been increased because the 2020 census took place during the period of data analysis.   the file extension data derived suggested that most files discussed in chat related to articles and webpages, as well as citation formats, a conclusion that is borne out in the dominance of citation formats as a topic in the nlp processing results.   discussion   can a toolchain using automatic and nlp effectively perform broad topic analysis of chat transcripts?   the toolchain developed for the project was able to process the chat transcripts, identify query hits from the list of terms, and perform topical analysis using automatic and natural language processing. the process took significantly less time than hand coding and was able to handle over 130,000 transcripts, confirming the benefits of computational analysis for large datasets previously identified in the literature (chen & wang, 2019; paulus & friend wise, 2019a; ozeran & martin, 2019). however, the programmatic data processing needed human mediation to interpret the meaning of some terms and to fix inaccuracies. automatic processing simplified the anonymization of the chat transcripts. the nlp process was very successful in identifying potential data for scrubbing, but human intervention was required to interpret some hits. for example, we needed to identify where hits referred to the proper noun “eric” as a person (where the row for the hit should be deleted) or the “education resources information center” (often also abbreviated as "eric"). other "false hits" included misspellings and stray punctuation, which the script could not identify. for example, “.hi” was identified as a filetype, when context suggested this was more likely a greeting. for this tertiary stage of human cleaning, having the "snippet" of text around the hit was very useful. however, this data needed to be removed before the data was exposed in tableau, as the words around a “hit term” often contained chat patron and operator identifying information. the nlp processing also introduced inaccuracies of other types, such as misidentifying a phrase as a proper noun and incorrectly labelling the type of proper noun.   while lacking the precision of hand coding, the team believes that programmatic processing of transcripts can be a beneficial supplement to hand coding, as automatic processing requires substantially less time commitment and can therefore be performed more often and across a larger corpus, permitting teams managing chat services to respond quickly to emerging topic trends. moreover, this approach lends itself to adoption across digital reference formats, as it can be modified to address any structured data format. in this case, the team reviewed a second set of data emerging from the email ticketing system, which could also be exported in .csv format. the code for the analysis was generalized and is licensed as open source (sharma, 2021).   what types of insights can be gathered from the processed data?   automatic and natural language processing provided insights about activity taking place on the chat service, such as the number of questions submitted by patrons at each participating library, the number of questions responded to by the operator groups of each library, and the proportions of chats picked up by different library staff types. these insights provide evidence regarding service usage and can inform decisions about staffing and training. the processed data also showed the extent to which questions were answered by operators from the users’ home library.   this dimension of the findings is of significant value when assessing service quality, as awareness of mismatches between the institutional affiliations of the user and patron can lead to user dissatisfaction (barrett & pagotto, 2019). the processed data also captured the top referring urls for the service, which can inform decisions about where to place chat buttons on each library’s website to drive service usage.   these insights were a valuable complement to the statistical reports offered by ask a librarian’s chat platform, libraryh3lp, which allows library administrators to view limited information about chat volume and operator activity. the processed data permitted the observation of additional chat metadata not ordinarily visible in the reports, such as library staff type and institutional match between the operator and patron. additionally, the data visualizations in the tableau dashboard can help staff easily compare various fields.   in terms of topical analysis, valuable insights were gathered from course codes, file extensions, and query hits. for instance, the team learned that the biology department generated the largest number of course-specific questions. in terms of file extensions, it was found that students asked about .pdfs, a likely indication that they were searching for full-text journal articles, and citations, suggesting that they were asking questions about citation managers. for the query hits and proper nouns derived from nlp, common topics mentioned were related to citation and subscription databases, indicating that students needed help with formatting references according to different citation styles, as well as searching in library resources.   to what extent are patrons asking questions related to data and digital scholarship topics?   the processed data indicated that few patrons are asking dataand digital scholarship-related questions through the ask a librarian chat service. only query terms related to the census and licensed software, such as spss, returned hits. this supports recent work by mawhinney (2020), which suggests that reference interactions involving data and digital scholarship are not well suited to the virtual environment. data and digital scholarship questions require longer reference interviews and often generate follow-up meetings. getting assistance for these types of questions requires a substantive time commitment, but virtual reference users prioritize short interactions. patrons with data or digital scholarship questions may therefore choose a different reference medium to get assistance, such as email or in-person service.   how does nlp compare to traditional hand coding methods in terms of accuracy?   in 2016, a research team from the university of toronto libraries and scholars portal hand-coded a sample of ask a librarian chats by topic using a coding scheme developed for the service (for coding key, see maidenberg et al., 2012; for topic analysis, see pagotto, 2020). of the chats in the sample, 56% were research-based (see figure 1); of these, questions about locating known items and finding sources on a particular topic were most common. other recurrent question types concerned policy, library accounts, e-resources, and citations. the results of the automatic chat analysis substantiated the hand coding. the most common question types related to research and citations, with frequent topics related to searching databases, accessing pdfs, and formatting citations according to different styles.   figure 1 ask a librarian questions by type (hand-coded).   inquiries about data and digital scholarship did not account for many questions in the hand-coded data. questions related to data were coded under research help, in the general research assistance sub-category, which applied to any chat asking “how do i find this type of thing?”. while this sub-category accounted for 6% of chats overall, the percentage of data-related questions on the service was lower in practice, since the category also encompassed questions about government documents, theses, and other publication types. most digital scholarship questions would have fallen under the category of library computing, a category that accounted for only 2% of questions overall. since few questions were found related to data and digital scholarship in the processed data, the automatic and natural language processing approach substantiated the hand-coded topic analysis.   conclusion   automatic and natural language processing toolchains have limitations when it comes to accuracy, and benefits when it comes to speed and the relative size of a corpus that can be analyzed. the team believes these benefits mean that automatic and natural language processing methods are well positioned to supplement hand coding, but results should be correlated with the results from studies utilizing hand coding techniques. it is possible that the precision of automatic and nlp approaches will continue to improve in accuracy. this study also provided a great opportunity for transdisciplinary learning and better understanding of emergent research tools.   acknowledgements   the research team wishes to thank scholars portal, particularly guinsly mondésir, for providing access to chat metadata and transcripts to support this project. the team also wishes to acknowledge judith logan of the university of toronto libraries and sabina pagotto and amy greenberg of scholars portal for their work in hand coding chats by question type as part of the 2016 ask a librarian assessment project, which were used as a comparison point in this work. the team would also like to acknowledge david kwasny of the university of toronto scarborough library’s digital scholarship unit for his contributions to initial conversations about natural language processing in the library.   author contributions   ansh sharma: methodology, data curation, software, formal analysis, visualization, writing – original draft kathryn barrett: conceptualization, methodology, data curation, investigation, formal analysis, writing – original draft (lead), writing – review & editing kirsta stapelfeldt: conceptualization, methodology, data curation, formal analysis, writing – original draft   references   afacan adanir, g. (2019). detecting topics of chat discussions in a computer supported collaborative learning (cscl) environment. turkish online journal of distance education, 20(1), 96-114. https://doi.org/10.17718/tojde.522398   barrett, k., & pagotto, s. (2019). local users, consortial providers: seeking points of dissatisfaction with a collaborative virtual reference service. evidence based library and information practice, 14(4), 2-20. https://doi.org/10.18438/eblip29624   bulygin, d., musabirov, i., suvorova, a., konstantinova, k., & okopnyi, p. (2018). between an arena and a sports bar: online chats of esports spectators. arxiv. https://arxiv.org/abs/1801.02862   bungaro, f., muzzupapa, m. v., & tomatis, m. s. (2017). extending the live chat reference service at the university of turin – a case study. 38th international association of scientific and technological university libraries conference, free university of bozen-bolzano, italy. https://docs.lib.purdue.edu/iatul/2017/challenges/1   chen, x., & wang, h. (2019). automated chat transcript analysis using topic modeling for library reference services. proceedings of the association for information science and technology, 56(1), 368-371. https://doi.org/10.1002/pra2.31   chow, a. s., & croxton, r. a. (2014). a usability evaluation of academic virtual reference services. college & research libraries, 75(3), 309-361. https://doi.org/10.5860/crl13-408   connaway, l. s., & radford, m. l. (2011). seeking synchronicity: revelations and recommendations for virtual reference. oclc research. https://www.oclc.org/research/publications/2011/synchronicity.html   hristova, g. (2021). topic modeling of chat data: a case study in the banking domain. aip conference proceedings, 2333(1), 150014. https://doi.org/10.1063/5.0044139   kherwa, p., & bansal, p. (2019). topic modeling: a comprehensive review. eai endorsed transactions on scalable information systems, 7(24), e2, 1-16. https://doi.org/10.4108/eai.13-7-2018.159623   kohler, e. (2017, november 3). what do your library chats say?: how to analyze webchat transcripts for sentiment and topic extraction. 17th annual brick & click: an academic library conference, maryville, mi, united states. https://eric.ed.gov/?id=ed578189   maidenberg, k., greenberg, a., whyte-appleby, j., logan, j., & spence, m. (2012). reference query coding key. tspace. http://hdl.handle.net/1807/94126   matteson, m. l., salamon, j., & brewster, l. (2011). a systematic review of research on live chat service. reference & user services quarterly, 51(2), 172-189. https://doi.org/10.5860/rusq.51n2.172   mawhinney, t. (2020). user preferences related to virtual reference services in an academic library. the journal of academic librarianship, 46(1), 102094. https://doi.org/10.1016/j.acalib.2019.102094   ozeran, m., & martin, p. (2019). “good night, good day, good luck”: applying topic modeling to chat reference transcripts. information technology and libraries, 38(2), 49-57. https://doi.org/10.6017/ital.v38i2.10921   pagotto, s. (2020). online communication [unpublished presentation]. scholars portal. https://spotdocs.scholarsportal.info/download/attachments/179013243/online_communication.pdf?version=2&modificationdate=1604339039000&api=v2   paulus, t. m., friend wise, a., & singleton, r. (2019a). how will the data be analyzed? part one: quantitative approaches including content analysis, statistical modeling, and computational methods. in t. m. paulus & a. friend wise (eds.), looking for insight, transformation, and learning in online talk (pp. 127-159). routledge.   paulus, t. m., & friend wise, a. (2019b). how will the data be analyzed? part two: qualitative approaches including thematic, narrative, conversation, and discourse analysis. in t. m. paulus & a. friend wise (eds.), looking for insight, transformation, and learning in online talk (pp. 160-196). routledge.   pronoza, e., pronoza, a., & yagunova, e. (2018, october 22-27). extraction of typical client requests from bank chat logs. 17th mexican international conference on artificial intelligence, guadalajara, mexico. https://doi.org/10.1007/978-3-030-04497-8_13   schiller, s. z. (2016). chat for chat: mediated learning in online chat virtual reference service. computers in human behavior, 65(1), 651-665. https://doi.org/10.1016/j.chb.2016.06.053   sharma, a. (2021). digitalutsc/communication_analysis (version 1.03) [source code]. https://doi.org/10.5281/zenodo.4918306   wang, t., huang, z., & gan, c. (2016). on mining latent topics from healthcare chat logs. journal of biomedical informatics, 61, 247-259. https://doi.org/10.1016/j.jbi.2016.04.008   willis, a., evans, a., kim, j. h., bryant, k., jagvaral, y., & glass m. (2017). identifying domain reasoning to support computer monitoring in typed-chat problem solving dialogues. journal of computing sciences in colleges, 33(2), 11-19. https://dl.acm.org/doi/10.5555/3144645.3144647   yang, s. q., & dalal, h. a. (2015). delivering virtual reference services on the web: an investigation into the current practice by academic libraries. the journal of academic librarianship, 41(1), 68-86. http://dx.doi.org/10.1016/j.acalib.2014.10.003   appendix list of terms   library/librarian nvivo/nvivohub reference citation protocol grad/graduate m.a/master's msc data map mapping download collection software analyze google digital scholarship digital humanities license open source creative commons wikipedia wikidata online digital visualize visualization git github metadata [campus].[university].ca [university].ca arcgis python javascript undergrad phd covidence spss qualitative quantitative tableau rstudio microdata odesi powerbi census chass workshop bibliography works cited apa mla ieee asa chicago turabian vancouver zotero refworks endnote stata sas cse qgis latex overleaf icmje ojs prisma jira prospero omeka wordpress drupal cms website   evidence summary   students taking numerous honours courses in high school have higher information literacy levels   a review of: fabbi, j. l. (2015). fortifying the pipeline: a quantitative exploration of high school factors impacting the information literacy of first-year college students. college & research libraries, 76(1), 31-42. http://dx.doi.org/10.5860/crl.76.1.31   reviewed by: dominique daniel humanities librarian for history and modern languages oakland university rochester, michigan, united states of america email: daniel@oakland.edu   received: 29 may 2015  accepted: 23 jul. 2015      2015 daniel. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to assess the impact of students’ high school performances on the development of their information literacy (il) competency.   design – statistical analysis of test performance.   setting – a large public university in the united states of america.   subjects – 93 first-time college freshmen. of these, 46% had been admitted on a probationary status due to gpa under the required 3.0 (“alternate admits”), and 61% had not declared a major (“exploring majors”). 39% identified as caucasian, 25% as hispanic, 22% as african american, and 15% as asian. 84% declared that their best language was english only.   methods – participants were self-selected freshmen who enrolled into programs offered by the university’s academic success center. they took the iskills test, an online evaluation of information literacy competencies developed by the educational testing service, and provided background data on their high school experience. using hierarchical multiple regression analysis, the researcher evaluated predictors of iskills score variance among a range of high school experiences: core high school gpa, number of honours classes taken in high school, and number of research projects or assignments in high school. the analysis controlled for gender, best language, race, and admission status as either alternate admit or exploring major.   main results – participants’ mean iskills scores was below the minimum passing score for the test. there was a significant positive correlation between iskills scores and exploring major status, core high school gpa, and having taken 5 to 12 honours courses. there was a negative correlation between iskills scores and language other than english, asian race, alternate admission status, and having had 1 to 4 honours courses. among the background variables, the most significant predictor of a student’s iskills score was his or her best language, followed by race. after controlling for these variables, the most important factors were students’ high school gpas and the number of honors courses taken.   conclusion – the researcher discovered that the number of honours courses taken in high school is a strong predictor of information literacy competency as measured by the iskills test. this remains true when controlling for race and other background factors. this finding is consistent with the assumption that high school teachers of honours courses believe their students to be capable of learning higher-order skills and therefore adopt a constructivist pedagogy, and that such pedagogy promotes the development of information literacy skills. yet the number of high school research projects or assignments could not be statistically correlated to information literacy competency. in subsequent focus groups, students who had taken fewer honours courses expressed test anxiety, while students who had taken numerous honours courses expressed their determination to get the correct answer. this may inform one surprising result of the study: that students who took 13 or more honours courses in high school did not score significantly better on the iskills test than those who took 5 to 12 courses.   commentary   the literature on information literacy assessment is abundant, especially literature regarding the factors associated with il competencies. this study provides an original approach to this topic by examining the link between high school experience and il performance at the university level. other studies on the connection between high school and college il skills have generally not used a performance-based, high-order skill-testing instrument such as iskills. furthermore, previous research has not focused on special populations of students characterized by lower than average high school academic achievements.   the study is based on a strong conceptual framework informed by il best practices and education scholarship. the methodology is sound and is explained in detail. the sample was very small but the researcher checked its normality against the overall student population. the author identifies several limitations of this study, including the fact that students self-reported their high school experiences, and also that participants were self-selected, as the test was administered to those who volunteered for programs offered by the university’s academic success center.   the researcher also uses selected results from another study, introducing focus groups she conducted after the test administration. these results help confirm the unreliability of self-reporting and provide explanations for some of the quantitative findings. however, not enough is said about the focus groups study, although it informs the results presented in this article.   more information about the design and validity of the iskills test would have allowed readers to better evaluate its effectiveness compared to other forms of il assessment, such as portfolio-based tests (katz, 2007; katz et al., 2009). iskills was developed between 2002 and 2006, and is based on the acrl il standards, which are now being superseded by a framework relying on threshold concepts rather than learning outcomes (oakleaf, 2014). yet because it is associated with critical thinking and problem solving skills, iskills is likely to remain a relevant assessment tool even under the new il framework.   improving undergraduate students’ il skills is central to the mission of academic librarians, so a better understanding of the factors conducive to il development could help librarians improve learning environments. this may include reaching out to high schools and encouraging constructivist pedagogy. this study is particularly important because it focuses on incoming students with lower gpas, who may have greater difficulty completing college. unfortunately, the findings in this article do not point to specific practical strategies. for example, the study could not link high il levels to the number of research assignments students had completed in high school. the author suggests that more research is needed about high school students’ experiences and about constructivist il pedagogy. more research on which specific il skills are most affected by high school experiences would also be welcome.   references   katz, i. r. (2007). testing information literacy in digital environments: ets's iskills assessment. information technology and libraries, 26(3), 3-12. http://dx.doi.org/10.6017/ital.v26i3.3271   katz, i. r., elliot, n., attali, y., scharf, d., powers, d., huey, h., …, & briller v. (2009). multiple methods of assessing information literacy: a case study. ets research spotlight (2), 21-27.   oakleaf, m. (2014). a roadmap for assessing student learning using the new framework for information literacy for higher education. journal of academic librarianship, 40(5), 510-514. http://dx.doi.org/10.1016/j.acalib.2014.08.001   microsoft word es_haglund.doc evidence based library and information practice 2007, 2:2  101 evidence based library and information practice       evidence summary    digital resource use and non‐use in the humanities and social sciences academic  settings is multifaceted      a review of:  harley, diane. “why study users? an environmental scan of use and users of digital resources  in humanities and social sciences undergraduate education.” first monday 12.1 (jan.  2007). 7 may 2007 <http://www.firstmonday.org/issues/issue12_1/harley/index.html>.    reviewed by:   lotta haglund  head of information and public relations, university library  karolinska institutet, stockholm, sweden  e‐mail: lotta.haglund@ki.se    david herron  scholarly developer, university library  karolinska institutet, stockholm, sweden  e‐mail: david.herron@ki.se      received: 21 march 2007   accepted: 25 april 2007      © 2007 haglund and herron. this is an open access article distributed under the terms of the creative  commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – (1) to map the digital resources  available to undergraduate educators in the  humanities and the social sciences, (2) to  survey faculty about their use of digital  resources, and (3) to examine how  understanding use and users can benefit the  integration of resources into teaching.    design – a mixed‐methods approach,  which included a survey, conducting  discussion groups, and in‐depth interviews.    setting – academic institutions in the  united states.    subjects – (1) “various stakeholders”; (2) 31  instructors from three institutions, and 4500  full‐time and part‐time faculty and graduate  students (at california public research  universities, liberal arts colleges and  community colleges); and (3) 13 digital  resource providers and two other  stakeholders, and 16 site owners or user  researchers.    methods – (1) a literature review, combined  with discussions with various stakeholders.   http://www.firstmonday.org/issues/issue12_1/harley/index.html mailto:haglund@ki.se mailto:herron@ki.se http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:2  102 (2) four sessions of discussion groups with  31 instructors from three institutions formed  the basis for developing a faculty survey  instrument. the survey was distributed both  on paper and online. (3) collection of data  on cost and collaborative development  strategies, in‐depth interviews with 13  digital resource providers and two other  stakeholders, combined with a two day  workshop with 16 experts, both on the  subject of online educational resources.    main results – (1) concerning the  humanities and social sciences digital  resource landscape, the main results of the  literature study were the conclusions that  the field of online education studies is  complicated by a lack of common  vocabulary, definitions, and analyses; and  that different stakeholder interests and  agendas also influence the understanding of  how digital resources are used. with the  help of discussion groups, an attempt at  creating a typology for digital resources  available to undergraduates was made,  looking at type of resource, origin, and type  of role of the provider or site owner. from  the article, it is unclear whether or not this  attempt at classification was successful.     (2) concerning faculty use or non‐use of  digital resources, the most important result  was the insight that personal teaching style  and philosophy influence resource use more  than anything else, and this also seemed to  be the most important reason for not using  digital resources. faculty use digital  resources for a number of reasons, to  improve student learning, provide context,  and also because it is expected of them.  more than 70% of faculty maintain their  own collection of digital resources. however,  the lack of efficient tools for collecting,  managing, and using these resources in  teaching is seen as a problem. there is also a  variation between scholarly fields, where  faculty in different disciplines require  different types of resources and use them in  different ways, and for different educational  reasons.     (3) concerning how understanding use and  users can benefit the integration of resources  in teaching, the results of the interviews  show a lack of common terms, metrics,  methods, or values for defining use and  users; but a shared desire to measure how  and for what purpose digital resources were  being used. few of the providers had any  plans to evaluate use and users in a  systematic way.     conclusion – the digital landscape is  complicated. faculty use is determined by  personal teaching style and philosophy.  digital resource providers would like to  know more about how and for what  purpose digital resources are being used.  experts see a number of areas for further  research, the results of which might help  clarify the situation.    the only way to understand the value of  digital resources is to measure their impact  and outcomes, but further work is needed to  provide common vocabulary, metrics, and  methods for evaluation.    commentary      the article provides a comprehensive  overview of the problem area, and can be  especially valuable for someone new to the  subject of the use of digital resources in  teaching. for the reader working in an  academic library, the description of the  results of the faculty survey investigating  the reasons for use and non‐use of digital  resources will provide useful insights for  discussing potential support to faculty  concerning these issues. the reviewers,  however, highly recommend that readers  refer to the original report by harley et al.  where a very comprehensive description of  the project is provided. the following  comments therefore refer to the article as an  evidence based library and information practice 2007, 2:2  103 independent piece of writing. all of the  comments are answered by referral to the  original report.     the article describes the very complex  reality of faculty use of digital objects, and  the problems concerned with measuring use.     the article lacks important information  concerning the set‐up of the different parts  of the study, which seriously influence the  readers ability to determine the reliability of  the study, such as how the respondents for  the different studies were selected, the  questions used in the survey, how the  information from the group discussions was  collected, as well as how the collected  information was analysed.     with the information in the article, it is hard  to determine the validity of the study, but to  anyone working in an academic setting the  description of faculty use and non‐use  seems familiar. the results from the faculty  survey seem useful, especially the insight  into the impact of personal teaching style  and philosophy on the use of digital  resources.    insight into reliability and validity would  have been improved by a more scientifically  structured presentation, and would have  added value to readers not able to find time  to read the final report from the project (a  total of 326 pages).    the three objectives of the study and the  methods used to investigate them are all  described in the article. however, in the  conclusion the author states that there are  three interrelated questions that she chooses  to clarify: how to assess if education within  the humanities and social sciences requires  different educational technology solutions  than other subject areas, if investments in  digital resources are worth it, and how to  share knowledge about users effectively.  even though the questions are interesting,  and the discussion deepens the  understanding of the problems surrounding  the use of digital resources, one wishes that  the author had focused on the opening  objectives and discussed the findings of the  study.     the conclusion that there are many  challenges when it comes to supporting the  management and integration of digital  resources into teaching, but that it is  definitely worth working on, is potentially  interesting for libraries planning to increase  services in this direction. the australian  “mire” (multimedia interactive research  environment) project may also in the future  provide some solution to these challenges.    works cited    harley, diane, et al. “use and users of  digital resources: a focus on  undergraduate education in the  humanities and social sciences.”  research and occasional papers series.  cshe.11.06 (apr. 2006). 9 mar. 2007.  university of california berkeley  center for studies in higher education.  7 may 2007  <http://cshe.berkeley.edu/publications/ publications.php?id=211>.    jakubowicz, andrew. “bridging the mire  between e‐research and e‐publishing  for multimedia digital scholarship in  the humanities and social sciences: an  australian case study.” webology 4.1  (mar. 2007). 7 may 2007  <http://www.webology.ir/2007/v4n1/a3 8.html>.    http://cshe.berkeley.edu/publications/publications.php?id=211 http://cshe.berkeley.edu/publications/publications.php?id=211 http://www.webology.ir/2007/v4n1/a38.html http://www.webology.ir/2007/v4n1/a38.html evidence based library and information practice evidence based library and information practice 2011, 6.1 3 evidence based library and information practice editorial responsibilities 2011. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. editor-in-chief: denise koufogiannakis associate editor (articles): alison brettle associate editor (evidence summaries): lorie kloda associate editor (classics): jonathan eldredge associate editor (features, commentaries, ebl101, news): denise koufogiannakis production editor: katrine mallan editorial intern: andrea baer copyeditors: priscilla stephenson (lead copyeditor), georgianne bordner, marcy brown, molly des jardin, richard hayman, lisa shen, dale storie, mary virginia taylor indexing support: pam morgan writing assistance: hope leman, tracy powell iwaskow, carol waseleski / evidence based library and information practice article   using analytic tools with california school library survey data   dr. lesley farmer professor of librarianship california state university long beach, california, united states of america email: lesley.farmer@csulb.edu   dr. alan safer california state university long beach, california, united states of america email: alan.safer@csulb.edu   joanna leack california state university long beach, california, united states of america email: joannaleack@gmail.com   received: 2 dec. 2014      accepted: 4 feb. 2015           2015 farmer, safer, and leack. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective — california school libraries have new state standards, which can serve to guide their programs. based on pre-standard and post-standard library survey data, this research compares california school library programs to determine the variables that can potentially help a school library reach the state standards, and to develop a predictive model of those variables.   methods – variations of decision trees and logistic regression statistical techniques were applied to the library survey data in order to create the best-fit model.   results – best models were chosen within each technique, and then compared, concluding that the decision tree using the cart algorithm had the most accurate results. numerous variables came up as important across different models, including: funding sources, collection size, and access to online subscriptions.   conclusion – school library metrics can help both librarians and the educational community analyze school library programs closely and determine effective ways to maximize the school library’s impact on student learning. more generally, library resources and services can be measured as data points, and then modeling statistics can be applied in order to optimize library operations.  introduction   from preschools to college, every school’s mission is to provide their students with the very best education possible. to do this, schools have to provide many things, such as curriculum, instruction, resources, and an effective learning environment.   within this framework, school libraries have as their mission: “to ensure that students and staff are effective users of ideas and information” (aasl, 2007, p. 8). this mission involves both physical and intellectual access, and requires considering preconditions, such as providing as much material in as many different formats as possible, or being open during commonly accessible hours for students. having proper staff, enough funding, and a quality collection potentially positively impact the school community, and therefore can have a positive effect on student learning outcomes.   in order to keep libraries striving to provide the best program of resources and services possible, some states set standards for those conditions that benefit student learning. in 2011 the california department of education put into effect statewide model school library standards, which address many aspects of a library including resources, staff, services, and budget. the standards included both student performance standards and research-based standards for the school library programs themselves (farmer & safer, 2010).   this notion of standards transcends school libraries. academic, public, and special libraries also need to provide the resources and services to meet their communities’ information needs. library systems may have baseline standards, stating a minimum number of volumes, subscriptions, equipment, staff, and required services. at the least, libraries often compare their resources and services to those of their counterparts, so that normative measures emerge. by identifying key factors that impact the library’s operational effectiveness, and by developing a predictive model, libraries can optimize funding decisions and develop evidence-based standards and guidelines.   this research used several data analytic techniques to determine which aspects of a california school library program affect its ability to meet these statewide standards. these statistical methods can be applied to many other library settings.   literature review   in this age of added accountability and value-added impact, school librarians need to show how they contribute to the school’s mission. furthermore, in tough economic times, school librarians have needed to make their case in order to continue their programs.   hundreds of research studies have found significant positive correlations between aspects of school library programs and student achievement. mansfield university’s literature review (kachel, 2013), scholastic’s what works! (2008), farmer’s 2003 synthesis and the library research service website (http://www.lrs.org/data-tools/school-libraries/impact-studies/) provide compendia of school library impact studies.   traditionally, school library programs have based their worth on their input and processes, that is, their resources and services (hatry, 2006). those program elements, however, have to be used in order to have an impact on student success, so usage figures are also kept. in the final analysis, student work, test scores, grades, retention and graduation rates serve as more useful data points of impact, although the library’s contribution is generally harder to measure (except for analysis of research projects). often librarians resort to perception-based assessment methods such as anecdotal observations, surveys, and interviews or focus groups, which may be more subjective compared to data-driven analyses (loertscher, 2008; mardis, 2011).   these data help determine the baseline quantity and quality of resources and services needed in order to provide satisfactory library programs: in other words, standards. in 2009 the american association of school librarians (aasl) developed guidelines for school library programs, based on their 2007 standards for 21st century learners, membership surveys, and focus groups. kentucky and missouri have formally adopted these national standards, and california’s 2011 standards (see appendix a) were informed by the aasl’s work. three-quarters of states have state-based school library standards, which tend to focus on staffing and resource quantitative measures and do not reflect the aasl’s 2009 guidelines (council of state school library consultants, 2014). only montana’s state standards appeared to be research-based (bartow, 2009). only texas investigated the relationship between their state standards and student achievement as measured by state standardized achievement tests (smith, 2001), which informed their 2005 revision (texas state library and archives commission, 2005); their conclusions were based on pearson correlation statistics.   as the california model school library standards were being developed, the state library consultant saw the need to underpin the standards with research. to that end, she asked dr farmer to review the literature about school library standards and program factors that significantly impact student success. updating her 2003 literature review, and drawing upon other existing compendia, as noted above, dr farmer identified contributing variables that appeared consistently in the literature:   ·         staffing (full-time credentialed school librarian, full-time paraprofessional) ·         access (flexible access to the library throughout the day for groups and individuals) ·         services (instruction, collaboration, reading guidance and promotion, reference, interlibrary loan) ·         resources (large current diverse and relevant materials that are well organized) ·         technology (internet connectivity, online databases, online library catalogue, library web portal) ·         management variables (budget, administrative support, documented policies, and procedures, strategic plan with assessment).   the presence of the specific variables (shown in italics) became the basis for the california school library baseline standards. the variables that are quantitative in nature (e.g., budget size, currency of collection) were calculated to determine adequate levels of support, which also constituted part of the baseline library standards (california department of education, 2011).   as the california school library program standards were being approved, farmer and safer (2010) wanted to determine if a significant difference existed between those school libraries that met the standards and those who did not. using the state’s most recent school library data set (2007-2008), the researchers applied descriptive statistics to identify standards variables. to be so designated, at least half of the survey respondents had to meet that specific baseline standard variable (that is, the library didn’t have to meet all of the factors’ standards). next, the researchers divided the data set into two groups: one that met all baseline standards, and one that did not meet all baseline standards. a t-test determined that the two groups were significantly different relative to resource and service standards; the most significant difference relative to the baseline standards was the presence of a full-time school librarian. a logistic regression analysis found that several variables related to resources and services further differentiated the two groups: number of subscription databases, library web portal presence, information literacy instruction, internet instruction, flexible scheduling, planning with teachers, book and non-book budget size, and currency of collection.   objectives   when the california model school library standards were being developed in 2010, the state economy was in crisis, and as a result school librarian positions were being eliminated. the 2007-2008 data set analyzed by farmer and safer (2010) preceded this economic drop, which provided a good baseline. the same researchers used the following year’s data (2011-2012) for comparison, and developed four research questions:   has the number of school library programs meeting the standard changed since the standards were approved? how do the significant variables identified in the 2007-2008 data set compare with those in the 2011-2012 data set? which variables differentiate school library programs meeting state standards from those which do not meet the standards? what statistical model provides the best fit of school library programs meeting state standards?   methods   to answer the research questions, the current study used the 2011-2012 california school library survey data set, and referred back as needed to the 2010 farmer and safer study.   each year the california department of education requests all k-12 schools in california to complete the annual online public school library survey about the prior year’s library data. typically, the site library staff complete and submit the survey, although occasionally a school administrator responds to the survey. the researchers had access to the resulting data, and applied several statistical methods to determine a model that would describe the data in terms of meeting state school library standards.   data description   the california department of education received 4017 responses (out of a possible 8588 k12 public schools, excluding special education, continuation, and alternative schools) to its survey regarding data about site school library for the academic year 2011-2012. of the respondent schools, 387 (9.6%) did not have a library so they were removed from the analysis, leaving 3630 useable libraries (observations). since information about the survey was disseminated to county and district superintendents, and to the state’s school librarian listserv, it is reasonable to assume that non-respondents were less likely to have school libraries than respondents. thus, the resulting data set may be considered representative of school libraries in california.   most response variables were coded as binary: indicating whether or not the school’s library met the specific state standard, with 0 being no and 1 being yes. three independent variables were categorical: single or joint use of library, credentialed staff, and grade level. five other independent variables had continuous values: number of books, average copyright date, book budget, and non-book budget. there were a total of 64 variables, noted in appendix b. the value of each variable was calculated; frequency and percentage statistics were applied to compare school libraries that met state standards and those that did not.   the researchers used sas enterprise miner to identify which school libraries met all the standards, and that determination was coded (0 being no and 1 being yes) into the data set.   statistical predictive modeling   the researchers wanted to develop a statistical model to predict which school libraries could meet state standards, based on a set of variables. the underlying concept of predictive analysis entails “searching for meaningful relationships among variables and representing those relationships in models” (miller, 2013, p. 2). predictive analytics reveals explanatory variables or predictors those factors that can relate to a desired response or outcome. in other words, what is the probability of an outcome given a set of input data? in this study, variable values were compared for library programs that met standards versus programs that did not. was there a subset of variables that could predict a school library’s success at meeting the state standards?   two main statistical techniques are recognized for developing decision procedures: logistic regression and decision trees. logistic regression is used to predict a response based on input data. decision trees are used to predict a categorical response, such as meeting standards or not (miller, 2013). within these two types of techniques are several possible versions.   a decision tree diagram looks like a flow chart because it is essentially a sequenced set of if-then decisions based on questions. an example is computer troubleshooting a printer failure, starting with the question: does the computer have electrical power? depending on the answer (yes or no), different actions are taken (e.g., if no, is the computer plugged in? if yes, is the computer switched on? this branching continues through a series of decision points). decision trees are a very useful statistical tool because they make good visual aids that are easy to interpret, and help show the relative importance of variables. they also facilitate predictions if a strong tree is built, and help find profiles of variables that are either much more likely, or much less likely, to occur than the overall average. statistical programs such as sas and spss can generate tree-based classification models using algorithms.   two main types of decision tree techniques are cart (classification and regression trees) and c4.5. the cart method allows for just two splits at any node (for example, meeting the standard or not, or budget greater or less than a specific amount), which can work well in this study because of the large number of binary factors. the algorithm is set up to choose a split among all possible splits at each node; it depends on the value of just one predictor variable. the best split point is one in which the resulting variables are unlikely to be mixed in ensuring splits. using the example above, determining if the power is on or not is a good first node because the ensuing issues are likely to be dependent on that first choice. a competing algorithm to cart is c4.5, where a node may split into more than two branches (larose, 2005).   the other statistical technique employed was logistic regression, which is used to find a model that relates a dependent binary variable (here, meeting, or not meeting, the state standard) with a set of independent variables. there are several advantages to running regression. first, if one is able to build a strong regression, it can potentially be used as a predictive tool for future data. it is relatively easy to interpret the effect of changing one predictive variable on the  response variable, holding the other predictive variables constant. regression analysis can also yield valuable information about the data within it. the researchers applied three selection methods — backwards, forward, and stepwise — and compared results to determine which model best fitted the data (miller, 2013)     table 1 quantitative factors of school library programs meeting standards   2011-2012 data 2007-2008 data average number of books ~21,000 ~16,000 average copyright date 1994 1995 average book budget                       ~$8000 ~$5000 average non-book budget ~$4000 ~$4000     table 2 data set of school library programs meeting baseline standards level of school total n (2011-2012) n meeting standard % meeting standard 2011-2012 data total n (2007-2008) % meeting standard 2007-2008 data elementary 2303  12  0.5 3312 0.4 middle school  531  39  7.3 842 8.2 high school  500 161 32.2 688 44.9 total 3334 212 6.4 4832 7.4     to compare the fit of each model, and see error rates across models, roc (receiver operating characteristic) was used (larose, 2005). this kind of chart visualizes the effectiveness of a classification model, calculating how well a variable will be assigned the right category, in comparison to being assigned a category randomly. a good predictive model shows a steep incline and remains near the top of the graph, indicating that the model can distinguish between two (or more) groups easily. a model that has a line close to the diagonal would imply that the classification is close to random or guessing. thus, a good model indicates that the categories are well chosen, and can be used to predict high-quality school library programs. as with decision trees, statistical software programs can calculate the roc based on the model’s sensitivity of the classification schemes.   for each model, the data were partitioned into a training set, for model fitting, and a validation set, for empirical validation. this technique was used in order to generalize to better predict future values. the training set included a random sample of 70% of the observations, and the remaining 30% composed the validation set to see how well the models classified other sets, and to determine possible generalizations (larose, 2005).   results   each library’s data were compared with the california state standards (appendices a and b). only 212 school library programs met all of the standards. appendix b details those resource and service variables that were present and independently met the standards. table 1 contains quantitative values for only the libraries that met all the standards; it lists the mean for those variables having continuous values (rather than simply being available or not, such as access after school).   table 2 compares the percentage of school libraries that met all of the state standards in 2011-2012 with the percentage of school libraries in 2007-2008 that met the standards before those standards were officially approved and disseminated (farmer & safer, 2010).   decision trees   each of the following trees randomly used 70% of the observations as the training set and the remaining 30% for the validation (or test) set.   cart (classification and regression trees)   figure 1 shows the tree generating the smallest misclassification error (for example, a variable that is classified as meeting a standard when in actuality it does not, or vice versa). unfortunately, the data set has many variables so the full decision tree is very difficult to read when viewing it as a whole. the right side of the tree is bushier than the left. the root node for this tree is “funding from the state lottery” since about 90% of the libraries did not receive state lottery funding (value of 0). therefore, the tree is unbalanced, with most decision points appearing on the right branch for that root node.   this decision tree’s variable importance is shown in table 3.   table 3 variable importance in cart decision tree variable name importance automated catalogue 1.0000 state lottery funds 0.8274 access to online resources 0.8228 online subscriptions 0.6155 streaming video subscriptions 0.5047 budget 0.4978 librarian helps find resources outside the library 0.4174   c4.5   running the c4.5 algorithm with target criterion set to entropy (that is, the least probability of a variable result occurring), the tree with the lowest misclassification error (that is, put in the wrong category) was generated. compared to the tree gained from the cart method, this tree has several more branches and nodes (i.e., it is bushier), which could potentially lead to a higher misclassification error. upon examination, the researchers found the misclassification rate to be the same as the tree generated with the cart algorithm.   regression   the next statistical technique used was logistic regression. three selection methods were applied: backwards, forward, and stepwise. afterwards, a comparison determined which model best fitted the data.   main effects: backward selection   the first logistic regression model to be run used only the basic variables, called the main effects model. using backwards selection initially starts with all the variables and slowly removes the insignificant ones. fifty-two steps (iterations) occurred during the backwards selection process. the final model selected is shown in table 4. many resources emerged, such as an automated online catalogue and automated textbook circulation. even more so, the kind of funding a library receives also appears on the list frequently.   fit statistics showed a misclassification error of 15.4%, which is relatively high in comparison to the other analysis performed. the average squared error also gives a percentage of 12.5%, which is high. both of these rates corresponded to the validation set.     figure 1 full decision tree.     main effects: forward selection   forward selection was used next on the main effects design. forward selection starts with zero variables and adds significant variables until the model is complete. the estimated selection plot is a visual way of seeing which variables were selected at which step in the process. state lottery funding was the first variable selected for the model, not surprisingly. using forward selection, only 15 steps were needed to create the optimal model.   several variables associated with funding were used, much like the main effects model using backwards selection. this model had a 15.2% misclassification rate on the validation set, which is a slight improvement in comparison to the backwards selection model. however, a classification chart showed that the model incorrectly classified school libraries that did not meet school standards.     figure 2 decision tree detail.     table 4 final model – regression, main effects, backwards selection variable significance level book budget 0.013 automated catalogue < 0.01 integrated information literacy instruction <0.01 state block grants (from federal government) <0.01 state school library funding <0.01 librarian helps find resources outside the library 0.013 interlibrary loan 0.047 librarian does online publishing 0.026 librarian creates wikis <0.01 online subscriptions <0.01     figure 3 cart decision tree – excluding funding sources     main effects: stepwise selection   in this selection model a variable can be added or removed at each step, depending on which would make the model better. the process resulted in the same variables as the forward selection one.   discussion   with the introduction of the california model school library standards, the educational community has metrics by which to assess school library programs and specific targets to aim for in improving those programs. furthermore, since these metrics were based on the professional literature about significant factors that impact student learning, the standards provide a case for value-added school library programs – and areas that could optimize such value.    school libraries meeting standards   the first research question asked whether the number of school library programs meeting the standard changed since the standards were approved. the short answer is “no” for elementary and high schools, and “yes” for middle schools, at least in terms of percentages. a major confounding external factor was economics, somewhat exacerbated by politics. by fall 2011, the state and federal economy was precarious, and federal funding for school libraries was severely reduced. not surprisingly, elementary school librarian positions became scarcer. at the high school level, school librarians became more likely to split their time between two (or more) schools, so they no longer met the standard of a full-time librarian at the site. in that respect, it is actually a bit heartening to see that the percentage of full-time middle school librarians increased 2.2%, although it still left almost 90% of middle schools without a full-time school librarian. it will be interesting to see in future years the extent to which school library programs improve because of the standards – or to which they improve because of the economic outlook. the latter picture would then predict that money more than standards makes the difference, which could lead to an unstable program.   for those school libraries that met the baseline standards, some interesting comparisons emerged. the average number of books increased as did the book budget, but the average copyright date was one year older than for the 2007-2008 data set. the 2011-2012 data set included some libraries build since 2008, which would account for the budget increase (and core collections include classic titles so are not automatically newer). in addition, school libraries may be reluctant to weed their collections in fear of leaving subject gaps, resulting in larger but older collections. non-book budgets stagnated.   predictive variables and models   the next research question asked what factors (i.e., variables) of a school library program can help determine if any given school library will meet the state standards in california? the accompanying fourth research question asked what statistical model provides the best fit of school library programs meeting state standards. some variables that stand out are those pertaining to staff, budget, and student accessible resources. these variables make sense since budget often drives resources, and staff manage the school library program.   each decision tree generated similar sets of variables, even though the trees were formed using different algorithms. these variables included state lottery funding, online access, and average copyright dates. the optimal decision trees formed had an average misclassification rate of 14.4%. these additional variables speak to more advanced school library program efforts, going beyond baseline measures. for instance, not only does the number of materials matter, but their currency impacts their use – and reflects the school’s support of the collection.   in examining the cart decision tree, the root node of “funding source” seemed to skew the remaining branches and leaves. further investigation with the former state school library consultant revealed that lottery and state grant funds were inactive at the time, but it was possible to use carry-over money to help finance school library programs. the survey responder, who was usually library staff, either had to know about this “inside” money stream or naively check off that box; the data seemed to indicate the former scenario. in that respect, the 10% of librarians who indicated this funding source are likely to be “in the know” about budgets or have good communication with the fiscal agents; in either case, this knowledge reflects pro-active management. such a disposition could be generalizable to other factors of the library programs, such as the availability of resources and related services.   to sidestep this issue of funding sources, a second cart decision tree was generated that excluded the funding sources variables. the result was a more balanced tree, as viewed in figure 3.   the important variables that emerged included (in order of importance): budget for non-book materials, evening access, book budget, number of books, level of library, availability of dvds, having classified staff, having online subscriptions (including streaming), and providing textbook service.    decision tree model comparison.   roc charts (figure 4) visualized differences between the cart and c4.5 tree for the training and validation data sets. the highest line signifies the c4.5 tree, the next highest line signifies the interactive tree (i.e., manually built), and the third line signifies the cart tree. for the training set, the c4.5 algorithm shows better results (higher accuracy), but the validation set shows better results with the decision trees produced by cart. wanting a tree with good predictive power, the accuracy of the validation set is more important; the higher the line (that is, maximum area above the line), the more accurate the model. with the roc percentages so close and the misclassification rates the same, the tree with the smallest averaged squared error should be selected as the optimal tree (larose, 2005). the decision tree which was produced using the cart algorithm showed the best results for the validation set, and was chosen to be the optimal tree.   regression model comparison   several logistic regression models were also run, including ones such as main effects and polynomial degrees. final models show significant variables to include the amount budgeted for books in 2011 and state lottery funding. although the logistic regression models were formed using multiple selection techniques, their misclassification rates did not match up to other models gained using different techniques. to see which logistic regression model was the best at classifying, roc charts were analyzed (figure 4). when considering the training set, the model that had the greatest accuracy and was best at classifying existing data was the main effects design using backwards selection. the same regression model also showed the highest accuracy when it came to the validation set based on the roc chart. fit statistics had the highest roc index for the backwards selected model, but that model also had the highest misclassification rate of the regression models. the difference, however, was only .2%, and when drilling down to specific variables, the backwards selection model did not have outstanding single misclassifications as did the forward selection. therefore, the backwards selection regression model was chosen as the preferred regression model.   final model comparison   model comparisons were run to determine the best models under each statistical technique. the goal at this point was to choose the overall best model, regardless of the method. the cart decision tree received the lowest misclassification rate, but it also has the second lowest training set accuracy (roc index).the logistic regression model had the highest misclassification rate and average squared error percentage; it also consistently had the lowest accuracy for both the training and validation set. figure 5 shows the roc chart for the validation set. the blue squares indicate the line that represents the cart decision tree model. even though a few of the models had higher accuracies than this method, cart gave a model with the lowest misclassification rate and average squared error.     figure 4 roc chart – comparison of decision tree models     figure 5 roc chart – final model comparison     comparison of 2007-2008 and 2011-2012 variables   the second research question asked how the significant variables identified in the 2007-2008 data set compared with the 2011-2012 data set. in the 2007-2008 study, the distinguishing variables were: availability of subscription databases, internet instruction, flexible scheduling, library web portal existence, information literacy instruction, planning with teachers, book and non-book budget size, and currency of collection. using the cart decision tree, several variables remained the same: availability of subscription databases, book and non-book budget size. additional variables identified in the 2011-2012 data included evening access and availability of dvds (probably not included in the earlier data set because of the small n sample size), having classified employees (probably because they were scarcer in 2011-2012), number of books, and textbook service (largely a function of high schools, and may be influenced by changing staffing patterns and online textbook initiatives). instruction and planning tend not to correlate closely with budget or even resources. so they might occur even in poorer school libraries; what is probably a more significant factor since 2007-2008, however, is the increased importance (and consequences) of high-stakes testing, which has tended to reduce library instruction and co-planning time.   conclusions   this research study examined california school library programs in light of the state’s model school library standards. using the california school library survey data from 2007-2008, farmer and safer’s 2010 study helped form these standards, and they discovered a significant difference between school libraries that met state standards and those who did not. the current research used the 2011-2012 school library survey data, which asked the same questions. this study compared the standards data, and explored a number of statistical models to find a best fit for capturing data about school library programs that could be used as predictors of program quality in order to provide the conditions for optimum learning experiences and student academic success.   the current study could not uncover any visible impact of the approved state standards on the 2011-2012 data relative to the 2007-2008 data (research questions 1 and 2), but the time frame was very short to expect any such changes. more current data would be needed, substantiated by interviews with school librarians to explain possible reasons for changes. furthermore, the economic and political landscape changed in the interim between the two time frames, which could account for changes.   after conducting several types of analyses, the cart decision tree provided the best fit to explain the data (research question 4). funding overall, and use of a variety of funding sources, were major factors in school library program status (research question 3). the findings pointed out the need for librarians to be aware of these funding streams, and to take advantage of them, which may require pro-active communication and negotiation with decision-makers. resources and access to them constituted another important “leg” of school library programs. books, non-print and online resources are all needed, and some analysis seemed to indicate that both physical and intellectual access through instruction were needed in order to make a difference. in general, there seemed to be a sizable gap between the vast majority of school libraries providing basic resources and services and those stellar libraries with rich collections, innovative services, and expanded access. in that respect, there is a possible matthew effect (that is, the bad become worse) that shows up more clearly in bad economic times.   as a model and possible predictive tool, the cart decision tree has potential as a way to examine school library programs, and determine the most effective allocation of funding in order to have a high-quality school library. this statistical model can also be used to make funding decisions in other kinds of libraries as well. the same variables could be used when appropriate, but other likely variables could be used as well, such as free parking, self-checkout systems, story hours, thesis workshops, and so on.   much research remains to be done. california has parallel data sets from 2003 to 2013, which can be analyzed using the cart decision tree model to look for patterns over time, both in terms of meeting standards as well as comparing important variables that make a significant difference in school library programs. newer survey data can be analyzed to see if the model school library standards impact support of school libraries. the cart decision tree can also be used with data from other states, or compared with national data, to determine possible significant differences between populations – or if a different model should be used.   school library metrics can help both librarians and the educational community analyze school library programs closely and determine effective ways to maximize the school library’s impact on student learning. more generally, library resources and services can be measured as data points, and then modeling statistics can be applied in order to optimize library operations.    references   american association of school librarians. (2007). standards for the 21st century learner. chicago, il: american library association.   american association of school librarians. (2009). empowering learners: guidelines for school library media programs. chicago, il: american library association.   bartow, c. (2009). how one state established school library/technology standards. school library monthly, 26(3), 19-21.   california department of education. (2011). model school library standards for california public schools kindergarten through grade twelve. sacramento, ca: california department of education.   council of state school library consultants. (2014). standards. salem, or: council of state school library consultants. retrived from http://cosslc.wikispaces.com/standards   farmer, l. (2003). student success and library media programs. westport, ct: libraries unlimited.   farmer, l., & safer, a. (2010). developing california school library media program standards. school library media research 13. retrieved from http://www.ala.org/aasl/slr   hatry, h. (2006). performance measurement: getting results (2nd ed.). washington, dc: the urban institute.   kachel, d. (2013). school library research summarized: a graduate class project. mansfield, pa: mansfield university.   larose, d. (2005). discovering knowledge in data: an introduction to data mining. hoboken, nj: wiley-interscience.   loertscher, d. (2008). information literacy 20 years later. teacher librarian, 35(5), 42-43.   mardis, m. (2011). evidence or evidence based practice? an analysis of iasl research forum papers, 1998-2009. evidence based library and information practice, 6(1), 4-23. retrieved from http://ejournals.library.ualberta.ca/index.php/eblip/index   miller, t. (2013). modeling techniques in predictive analytics: business problems and solutions with r. upper saddle river, nj: pearson.   scholastic publishing. (2008). what works! new york, ny: scholastic.   smith, e. g. (2001). texas school libraries: standards, resources, services and students’ performance. austin, tx: texas state library and archives commission.   texas state library and archives commission. (2005). school library programs: standards and guidelines for texas. austin, tx: texas state library and archives commission.     appendix a school library program standards (california department of education 2011, 34-42)   full time teacher librarian (.5 for schools with enrollment between 350 and 785 students) full time paraprofessional librarian assistant library open to students at least 36 hours per week integrated library management system with online public access capability library web page internet access for students flexible scheduling at least 20 hours per week class set of networked computers (10 for elementary, 15 for middle school, 25 for high school) facility to accommodate one class for instruction and small group independent work collaborative planning and teaching for at least two grade levels or departments at least 20 hours of instruction per week at least 5 hours of management per week reading guidance current policies, procedures and library plan, including assessment at least two online subscription databases print magazines (25 for elementary, 20 for middle school, 15 for high school) at least two-thirds of the collection less than 15 years old at least 28 books per student one book per student added per year for elementary and middle school; one book per two students for high school     appendix b 2011-2012 school library program variables meeting state standards   total n=3628 (8 missing)  elementary n= 2591 (6 missing)  n= 533   hs n= 498 (2 missing) variable elemen tary ms hs total # meeting standard total % meeting standard credentialed full-time teacher librarian 299 (11.6%) 188 (35.2%) 330 (66.3%) 818 22.6 paraprofessional 2590 (99.7%) 447 (83.9%) 373 (74.6%) 2927 80.7 open before school 1069 (41.2%) 421 (79%) 428 (85.6%) 1917 53 open for classes 2479 (95.5%) 513 (6.2%) 482 (96.4%) 3620 95.9 open during breaks 1721 (66.3%) 440 (82.6%) 437 (87.4%) 2597 71.7 open during lunch 1483 (57.1%) 486 (91.2%) 447 (89.4%) 2414 66.5 open after school 1085  41.8%) 391 (73.4%) 420 (84%) 1895 52.3 open evenings 55 (2.1%) 16 (3%) 61 (12.2%) 132 3.6 open weekends 7 (0.3%) 1 (0.2%) 19 (3.8%) 27 0.7 open summers 51 (2%) 6 (1.1%) 50 (10%)     used instructional materials funds 81 (3.1%) 39 (7.3%) 44 (8.8%) 164 4.5 used state lottery funds 182 (7%) 49 (9.2%) 57 (11.4%) 287 7.9 used per pupil allotment funds 255 (9.8%) 57 (10.7%) 70 (14%) 383 10.6 used general funds 569 (21.9%) 190 (5.6%) 252 (50.4%) 1009 27.8 received block grant 296 (11.4%) 78 (14.6%) 68 (13.6%) 441 12.2 did fundraising 1398 (53.8%) 298 (55.9%) 128 (25.6%) 1825 50.3 used title i funding 23 (7.8%) 72 (13.5%) 56 (11.2%) 364 10.0 used title v funding 6 (0.2%) 5 (0.9%) 3 (0.6%) 14 0.4 use local bond funding 66 (2.5%) 13 (2.4%) 11 (2.2%) 91 2.5 received other grant funding 1 (0.2%) 1 (0.2%) 2 (0.4%) 7 0.2 received start-up funds 16 (0.6%) 1 (0.2%) 2 (0.4%) 19 0.5 received other funding 323 (2.4%) 75 (14.1%) 92 (18.4%) 490 13.5 did online publishing 165 (6.4%) 74 (13.9%) 135 (27%) 374 10.3 share photos online 96 (3.7%) 51 (9.6%) 89 (17.8%) 236 6.5 used a news feed 83 (3.2%) 46 (8.6%) 65 (13%) 195 5.4 generated digital images 61 (2.3%) 38 (7.1%) 61 (12.2%) 160 4.4 used social bookmarks 44 (1.7%) 28 (5.3%) 61 (12.2%) 133 3.7 used wikis 249 (9.6%) 113 (21.2%) 126 (25.2%) 488 13.5 used online productivity tools 451 (17.4%) 209 (39.2%) 298 (59.6%) 957 26.5 used online social libraries 95 (3.7%) 42 (7.9%) 68 (13.6%) 205 5.7 used online videos 402 (15.5%) 154 (28.9%) 218 (43.6%) 774 21.3 downloaded audio files 182 (7%) 72 (13.5%) 95 (19%) 349 9.6 used ebooks and audiobooks 421 (16.2%) 141 (26.5%) 188 (37.6%) 751 20.7 used learning management systems 107 (4.1%) 51 (9.6%) 86 (17.2%) 244 6.7 provided opac 2157 (83.1%) 488 (91.6%) 453 (90.6%) 3097 85.4 circulate textbooks 779 (30%) 318 (59.7%) 308 (61.6%) 1403 38.7 provided access to online resources 565 (21.8%) 174 (32.6%) 277 (55.4%) 1017 228 provided video streaming services 1294 (49.8%) 322 (0.4%) 295 (59%) 1909 52.6 provided dvds 1136 (43.7%) 304 (57%) 301 (60.2%) 1739 47.9 provided audiobooks 655 (25.2%) 232 (43.5%) 248 (49.6%) 1134 31.3 provided integrated searching portal 618 (23.8%) 168 (31.5%) 186 (37.2%) 972 26.8 conducted workshops 208 (8%) 104 (19.5%) 168 (33.6%) 481 13.3 offered integrated information literacy instruction 202 (7.8%) 95 (17.8%) 154 (30.8%) 452 12.5 informally instructed on resource use 1503 (57.9%) 375 (70.4%) 384 (76.8%) 2261 62.3 gave reference help 1884 (72.5%) 463 (86.3%) 442 (88.4%) 2789 76.9 helped find outside resources 947 (36.5%) 297 (55.7%) 358 (71.6%) 1603 44.2 facilitated interlibrary loan 854 (32.9%) 245 (46%) 186 (37.2%) 1285 35.4 helped parents realize lifelong learning importance 981 (37.8%) 160 (30%) 141 (28.2%) 1283 35.4 coordinated in-school production of materials 203 (7.8%) 68 (12.8%) 87 (17.4%) 359 9.9 collaborated to create av products 118 (4.5%) 67 (12.6%) 110 (22%) 296 8.2 did av programming 90 (3.5%) 51 (9.6%) 64 (12.8%) 206 5.7 coordinated computer networks 370 (14.2%) 153 (28.7%) 159 (31.8%) 683 18.8 provided access to opac 1717 (66.1%) 438 (82.2%) 411 (82.2%) 2564 70.7 provided student internet access 1594 (61.4%) 470 (88.2%) 437 (87.4%) 2499 68.9 provided access to resource sharing network 258 (9.9%) 121 (22.7%) 172 (34.4%) 551 15.2 communicated proactively with principal 1817 (70%) 415 (77.9%) 362 (72.4%) 2593 71.5 attended site council 2x (year or more 499 (19.2%) 161 (30.2%) 182 (36.4%) 842 23.2 provided online subscription db 956 (36.8%) 262 (49.2%) 330 (6%) 1547 42.6     editorial   peer review of the evidence base   lorie kloda editor-in-chief assessment librarian, mcgill university library montreal, quebec, canada email: lorie.kloda@mcgill.ca      2015 kloda. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   welcome to the first issue of the tenth volume of evidence based library and information practice (eblip). this issue is suitably full, reflecting the quantity and quality of submissions we continue to receive and publish. inside, you will find 4 articles reporting on original research, 10 evidence summaries, 1 report about using evidence in practice, as well as a news item.   this is also my inaugural editorial as editor-in-chief. i have been involved with the journal since its inception in 2006, first as an evidence summary writer, then as the associate editor for evidence summaries, and then for articles. the journal editorial board also welcomes rebekah (becky) willson (currently in australia), as an associate editor for articles, and melissa griffiths (in the united kingdom) as editorial intern.   one of the reasons the journal has been so successful at publishing quality papers is its reliance on peer review. every few years, it seems the subject of peer review arises again in the scholarly sphere as a topic of controversy. i have always found these discussions interesting and revealing, because, despite peer review’s longevity and pervasiveness in traditional scholarly publishing, it seems there is always the argument that peer review needs improvement.   and it probably does. i have written before about the importance of post-publication peer review (kloda, 2009), in which i argued that peer review is not an excuse for not engaging critically with published work. nevertheless, peer review is an important process by which submitted manuscripts are evaluated before being considered for publication. it is especially important that submissions to this journal are peer reviewed by experts not only in research methods and the subject content, but also those fluent in issues important to practitioners. this journal has a wide audience of librarians and information professionals, as well as instructors and researchers in library and information studies. in order to be useful for these practitioners, manuscripts should make very clear the applicability of research results to practice.   eblip is similar to many other journals, open access or otherwise, in that we employ double-blind peer review in the editorial decision making process. the process is straightforward: all original research articles, feature articles, evidence summaries, classics, and review articles that are submitted for consideration are first read by the assigned editor to ensure the manuscript is within the journal’s scope. once the editor has confirmed that the submission has been properly blinded, i.e., no information about the authors is discernable from reading the manuscript, at least two peer reviewers are invited to review the submission. peer reviewers are typically selected for their expertise, not just in the methods, but for all aspects of the manuscript. peer reviewers are provided with guidelines which include questions about the research question or objectives(s) of the study, the literature review and references, the methods, presentation of findings, discussion, and implications for practice. that last element is so important for eblip, given its intended audience and its goal of transferring evidence into practice.   after these initial steps, the peer reviewers submit their reviews to the editor, who then considers their comments when making a decision as to whether to accept or reject the manuscript for publication. in many cases, an acceptance is conditional on revisions, and sometimes authors are asked to make thorough revision requiring resubmission for a second round of peer review. the editor then communicates their decision back to the author(s) with their suggested revisions and includes the reviews, anonymously.   the purpose of peer review is not to identify fraudulent research or plagiarism, but to ensure that any manuscript that is published is relevant, has sound methodology, and is a clearly written report of research undertaken. the contribution of a peer reviewer is not trivial. peer reviewing research in librarianship is an important service to the profession, as well as a contribution to scholarship. it is the voluntary donation of one’s time and expertise for the sake of knowledge, and in the case of eblip, for the sake of influencing practice.   though we have a long list of peer reviewers here at eblip, it is often difficult to find individuals willing to review, especially in areas for which there are few experts. an editor for an engineering journal recently lamented on the subject of securing peer reviewers, “it is not rare to have 10 refusals to every acceptance” (dasco, 2014). fortunately, we do not have this challenging a ratio, though our associate editors frequently need to ask four or five individuals before two reviewers, the minimum number required, accept. peer reviewers for eblip routinely review four or more manuscripts in a given year, though we try not to overburden any one reviewer.   because the work of a peer reviewer is essential in supporting the work of journal editors and the manuscript authors in ensuring quality, we try to reward reviewers by sharing with them the editor’s decision (accept, reject, revise, resubmit), and the comments of the other reviewers. reviewers’ comments are shared anonymously, and our intention in doing this is to allow peer reviewers to learn from each other’s comments and understand how their review contributes to the editorial process.   peer review is an important part of dissemination of scholarly work, yet it is a type of labour that goes unpaid. librarians’ and information studies researchers’ time is valuable. i encourage readers of scholarly literature to offer their services as peer reviewers in areas where they may have expertise, as there is a lot to be learned from peer reviewing about the research and writing processes, and it can be very rewarding. in selecting journals to contribute to as a peer reviewer, i encourage you to think about open access journals that rely entirely on the labour of volunteers, such as eblip, when deciding where to offer your expertise and services.   references   dacso, c. c. (2014). some consequences of refusing to participate in peer review. ieee journal of transactional engineering in health and medicine, 2. http://dx.doi.org/10.1109/jtehm.2015.2392271   kloda, l. a. (2009). being a critical professional: the importance of post-publication peer review in evidence based library and information practice. evidence based library and information practice, 4(3), 72-74. http://ejournals.library.ualberta.ca/index.php   evidence based library and information practice commentary   the evolution of evidence based library and information practice part iii: revitalizing the profession through eblip   jonathan d. eldredge associate professor health sciences library and informatics center university of new mexico albuquerque, new mexico, united states of america email: jeldredge@salud.unm.edu   received: 5 jan. 2014      accepted: 15 feb. 2014      2014 eldredge. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     why did you enter this profession?   your answer to this question will likely be unique and nuanced. your answer will probably be complex because the answer involves multiple levels of analysis and self-reflection. and, much will depend upon your frame of reference. your answer might focus on your long-standing organizational abilities. perhaps from an early age, for example, others had observed your inherent ability to create order out of chaos? perhaps it was an aptitude in communicating clearly or in teaching? or, maybe you could ferret out important clues to solving real-world puzzles?    your answer might additionally reference your personal core values. most librarians and information professionals seek to connect members of their user communities with desired information or ideas that potentially can improve these users’ or others’ lives.  part ii in this series of commentaries suggested a functional definition of our profession, when noting that “librarians and other information professionals identify, organize, and make accessible authoritative information for specific user populations” (eldredge, 2013, p. 103). yet, this definition did not answer the question as to why a highly-educated individual would pursue this specific profession over another profession, such as law, medicine, engineering, or teaching.   on a broader, profession-wide scale, numerous members of various associations have answered this question of values through the development of their codes of ethics. the american library association’s (2008) code of ethics states that “[members] have a special obligation to ensure the free flow of information and ideas to present and future generations” (para. 3). the chartered institute of library and information professionals’ (2012) professional practice code states that “the behaviour of professionals who work with information should be guided by a regard for the interests and needs of the users” (responsibilities to information and its users section, para. 1). the canadian library association (1976) similarly states that professionals “facilitate access to any and all sources of information which may be of assistance to library users” (responsibility point 3). the canadian health libraries association (2007) elaborates upon this point, indicating that “the health sciences librarian believes that knowledge is the sine qua non of informed decisions in health care, education, and research, and the health sciences librarian serves society, clients, and the institution by working to ensure that informed decisions can be made” (goals and principles for ethical conduct section, para. 1). the medical library association (2010) uses the exact same language in the preamble to its code of ethics, and captures it succinctly with the association tagline: “professionals providing quality information for improved health” (http://www.mlanet.org/).   eblip as a social movement   eblip represents a social movement among library and information practitioners. this movement serves multiple purposes, among them principally providing a process for informed decision making. the steps in the eblip process were described in part i of this series of commentaries (eldredge, 2012). the eblip process closely resembles the evidence-based practice processes in other professions as diverse as education (davies, 1999; slavin, 2002), management (rousseau, 2012), and healthcare (dawes et al., 2005).   this commentary suggests that eblip less obviously serves the additional purpose of renewing the contract our profession has with society. this added purpose results in members of society viewing our profession anew, with respect for its expertise, accountability, and for its user-oriented decisions.   koufogiannakis (2012a, p. 91), koufogiannakis (2012b, p. 6), glynn (2007, p. 1), and others such as lewis (2011, p. 152), have all referred to eblip as a “movement” within the profession. blumer (1951/1995) classifies social movements according to three types: general, specific, and expressive. eblip clearly fits blumer’s classification criteria for a specific movement with an aim toward reform rather than revolution. eblip also largely exhibits blumer’s five mechanisms that movements harness to accomplish their goals. these mechanisms might be termed: awakening, camaraderie, persistence, shared worldview, and strategy. coincidentally, blumer’s core concept of  “institutionalization” (pp. 63-64) took a concrete form at the eblip7 conference of july 2013 when the university of saskatchewan library dedicated its new centre for evidence based library & information practice (http://library.usask.ca/ceblip/). admittedly, institutionalization has been occurring at many academic sites worldwide in australia, canada, sweden, the u.k., and u.s. for over a decade. this dedication then was not an entirely new development, although it did represent the most dramatic and clearly-articulated example.   previous commentaries have noted the key characteristics of eblip (eldredge, 2012) and have explored the deeper potential purposes of eblip beyond the obvious purpose of decision making (eldredge, 2013). this commentary discusses the changing characteristics of professionalism in the 21st century and how eblip can play a key role in renewing our professional identity.   professionalism: core features   as noted in part ii of this series of commentaries, for the past half-century sociologists have studied occupational groups that are either widely-recognized or that self-identify as “professional” in different societies. in addition, sociologists have found it difficult to define professionalism in a universal way, perhaps because professionals function in varying ways within different societies at different times. in other words, specific societies define professions and then govern these professions’ rights and responsibilities in relation to the needs of these specific societies. sociologists’ conceptualizations suggest a special expression of rousseau’s social contract (1983), in which society oftentimes extends extra rights and obligations to members who belong to the professions.   societal expectations of the professions, as well as the boundaries placed on professions by society, frequently find expression in the policies of organizations or within government laws and regulations. it should be emphasized that societies create professions to serve societies’ needs. conversely, societies can elect to remove or to modify professional roles, responsibilities, and privileges.   as noted in part ii, sociologists during the 1950-1990 era identified certain key features of traditional professionalism in english-speaking and western european countries:   expertise authority higher education autonomy specialized or esoteric knowledge   these core features have normally been present in professionals practicing in these societies. the extent of the presence of these features has varied according to the profession under study (etzioni, 1969), the historical epoch, and the specific society. in relation to the final bullet point above it is interesting that pfeffer (2011) has taken the management profession to task for deviating from professional standards when not using valid scientific research results or methodologies. sociologists have based these aforementioned core features mainly on in-depth studies of the legal and medical professions. sociologists have been particularly fascinated with what they have viewed as the “monopoly power” of these two specific professions that allow individual practitioners to control many of the conditions of their practices within a specific society.   a study of 91 health care employees at a chinese university suggested that, in the cultural context of china, the aforementioned key features of professionalism were still present, but that the ethical concept of integrity was more highly valued among chinese health practitioners (ho, yu, hirsch, huang, &yang, 2011). integrity also appeared on a list of the top-ranked 29 valued professional traits in a multi-regional study of 584 physicians from different continents. among the other traits held in common across the continents that related to either evidence based practice or the aforementioned issues of professionalism were: respecting patient autonomy; accountability; respect for others; managing conflicts of interest; possessing sound judgment and decision making skills; improving oneself; and, not using one’s position for personal gain (chandratilake, mcaleer, & gibson, 2012). by substituting the word “patient” with “user” in this list of core traits we could readily apply the same list to the professional traits of library and information practitioners. these studies suggest that while national or cultural variations might exist (booth & eldredge, 2010), it still appears then that there might be sufficient commonalities for discussing professions across time and cultural contexts.   changes in society’s expectations of the professions   the relationship between the professions and society has been undergoing fundamental changes over the past two decades. sociologists are attuned to these changes and have attempted to provide coherent explanations to the underlying forces churning beneath the surface of these fluctuating societal expectations of professionals. in addition, sociologists have speculated on the future implications of these trends.   the professions, once viewed by sociologists during the twentieth century as the epitome of professional autonomy, have watched that autonomy erode during the twenty-first century. as gleeson and knights (2006) have observed, “today, few professions have been able to avoid the erosion of their independence from employer organizations or the state as industrial growth, globalization, and an expansion of government interventions have occurred” (p. 280). a study involving 1,800 journalists in 18 countries reported an erosion of autonomy mainly due to the concentration of news organizations’ ownership within a few large corporations (reich and hanitzsch, 2013). this previous autonomy was viewed as a rampart essential for protecting the free flow of information to the citizenry.   the legal, medical, and accounting professions all have experienced losses of autonomy due to the fact that members of all three professions now tend to work for large organizations, such as corporations or government agencies, instead of working in solo or small-scale family-owned practices. the desire of large organizations to align these traditional types of professionals’ priorities with the organization’s priorities has sometimes caused conflict between the professions and those management professionals representing the interests of the parent institution. at the very least, negotiating the altered relationships between these large organizations and the professions are creating new forms of legal, medical, and accounting professions that are adapting themselves more closely to organizational structures and priorities (muzio & kirkpatrick, 2011). it should be remembered from a broader perspective that individual professionals do benefit from their work in large organizations through greater opportunities for specialization. professionals also benefit from their association with large organizations through the collectivization of both risks and benefits.   historically, many library and information practitioners have worked within large organizations. these large organizations have consisted of institutions of higher learning for academic library and information professionals, academic health sciences centres or hospitals for health sciences professionals, municipalities for public librarians, and a variety of large organizations for special library and information professionals. thus, our profession regardless of sector has a long-term collective experience of working within large organizations. in contrast to the aforementioned law, health care, accounting, and journalist professions, we have developed long-standing strategies for aligning ourselves with large organizations’ goals without compromising our core professional values. conveniently, this historic pattern has positioned us well to adapt to the new social contract involving professionals in the twenty-first century.   sociologists tend to avoid normative interpretations of the changing relationships between society and the professions. sociologists instead examine underlying power structures in society and adaptation strategies of the professions. sociologists agree that changes in the traditional norm of professional autonomy, or independence, are most often expressed in altered client interactions with the professions. they point out that the place of diminished autonomy has been replaced by new forms of professional authority that are emerging and that revolve around the locus of accountability.   professionals within the new societal expectations framework still seem to retain their expert skills and specialized knowledge that derive from their education and experience. the core characteristics of professionalism outlined above largely still appear to remain intact. the monopoly position of some professions expressed as individual practitioner autonomy as found traditionally in a profession such as medicine no longer seems to exist. yet, some autonomy continues to exist. autonomy has decreased markedly, to be sure, but it now takes different forms, according to sociologists.    professionals seem to be transitioning away from a central focus upon autonomy toward a new emphasis upon accountability (gleeson & knights, 2006; noordegraaf, 2011; timmermans, 2005). “professionalism, then, is perceived to be about applying general, scientific knowledge to specific cases in rigorous and therefore routinized or institutionalized ways,” according to noordegraaf (2007). this observation suggests potential areas of compatibility between evidence based practice (ebp) and the professions as they reconcile their efforts with the need for standardization within organizations. berg, horstman, plass, and van heusden (2000) have suggested, amidst these changes, that core ebp characteristics such as practice guidelines continue to be subject to the expert interpretation by professionals so that some autonomy exists amidst an environment in which society demands more accountability or transparency from the professions.   response to changes from the professions   noordegraaf (2011) has observed that “it is not easy to (re)organize professionalism…. as professionals are strongly socialized, they will not easily redefine their own images of professionalism” (p. 1365). while researching this commentary i immersed myself in the literatures of the professions to review what professionals were discussing among themselves about alterations in their status due to these changing societal expectations. on the whole, these inwardly turned discussions consisted of many complaints, lamentations, and even jeremiad-toned tracts on the end of their special professional status. physicians, in particular, have been worried about their inability to reclaim their autonomy. it can be a depressing read.   while exploring my hypothesis about eblip serving to restore professional status, i furthermore did not find much explicitly written in the literatures of other professions linking ebp with these new conceptualizations of professionalism. the shift from less autonomy toward greater accountability for some professions appears to be a parallel yet largely unrelated development vis-à-vis the advent of evidence based practice. such writings in the professional literature might, i reasoned, at least provide potential frameworks or roadmaps for our profession to adapt for its own purposes. these articles do exist, although the linkages between ebp and a new conceptualization of professionalism mainly are oblique or secondary to other principal concerns about either evidence based practice or professionalism.   denny (1999) represents a noteworthy exception. denny writes that, “although the discourse of ebm [evidence based medicine] appears to question the individual authority of medical doctors, it actually reinforces such authority by regulating the conditions under which a physician may speak authoritatively about health and illness… to define and clarify what it means to be a doctor in relation to those who are not” (pp. 247-248). denny’s perhaps cynical approach, however, relates more to the ethics of power relations between the medical physicians and the challenges to medical authority by advocates for alternative medicine or by health consumerists. denny does credit the work of library and information practitioners in making ebm possible (p. 260), which echoes a point made over the years by many health sciences librarians and informaticists. mykhalovskiy and weir (2004) reiterate some of denny’s points and expand their analysis to make the preliminary suggestions that ebm runs the risk of evidence authoritarianism or being co-opted by medical managers. they instead reach the contrasting conclusion that ebm ultimately reinforces the professional authority of physicians. armstrong (2002) predicts that evidence based medicine ultimately will retain authority and autonomy for the overall medical profession yet at the “expense” of the autonomy of the individual practitioner (p. 1772). wagner, hendrich, moseley, and hudson (2007) explored the meanings of “professionalism” to medical students, residents, academic faculty and patients. three themes that define characteristics of modern medical professionalism emerged from this research: knowledge/skills, patient relationship, and character virtues. williams (2004) explored the meanings of professionalism for psychiatry and concludes that, “modern approaches to professionalism require robust mechanisms for translating evidence into practice that propel individualized patient care that fully recognizes the importance of diversity of values and culture” (p. 242). the american board of internal medicine foundation charter on new medical professionalism (2002) resonates with ebp when it declares that, “physicians have a duty to uphold scientific standards, to promote research, and to create new knowledge and ensure its appropriate use. the profession is responsible for the integrity of this knowledge, which is based on scientific evidence and physician experience” (p. 245). some dentists also consider ebp to be mechanism to enhance professional status: “professions and professionals have a perceived autonomy vested in their claim to objective scientific truth, which promotes public trust” (cannavina, cannavina, & walsh, 2000, p. 306).   nurses have probably been the most insistent that evidence based practice enhances their professional status. adams and mccarthy (2007), reavy and tavernier (2008), and vanhook (2009) all make the connection between ebp and a new professionalism. lejonqvist, eriksson, and meretoja (2011) make the point forcefully when they write, “nursing should be grounded in evidence, not tradition” (p. 340). mackley, bollinger, and lynch (2012) emphasize the need for nurses to generate their own research evidence to enhance professionalism. even authors such as colyer & kamath (1999), who express skepticism about ebp, still recognize its power for the nursing profession, particularly in persuading decision makers already oriented toward ebp who administer institutions such as hospitals that employ nurses.   bonell (1999) offers the most compelling argument about linking ebp to a new model of professionalism. she moreover warns of the misguided “debate” of qualitative versus quantitative adherents that might divide the nursing profession, thereby neutralizing ebp efforts to enhance professionalism. she depicts some authors of taking a negatively “stereotyped view of quantitative/experimental methods” that will only lead to fruitless debates that will divide the profession and lead to greater “marginalization of nurses in research and evidence-based practice initiatives” (p. 18). most importantly, bonell warns:   if nurses do not involve themselves in developing evidence-based health care, it is possible that other groups will lead on the evaluation of nursing and on developing evidence-based guidelines for nursing. this may result in nurses’ work becoming routinized, and nurses losing rather than gaining, autonomy and authority. (p. 19)   could the same prediction be forecast for library and information practitioners who complacently rely on “someone else” to support or to even pursue rigorous research? happily for nursing, bonell predicts that ebp will lead to greater professional status for nurses.   some physical therapists link ebp to professionalism, specifically to a changed concept of professional autonomy (hardage et al., 2012). they agree with the sociologists that “autonomy is not a static all-or-none dichotomy, but rather a matter of degree based on the environment and opportunities that exist at a particular time” (p. 84). speech-language pathologists link a merging of both science and what could be termed a “craft” to ebp to form a new professionalism (justice, 2010).   the health sciences professions, of course, are not alone in linking ebp to new conceptualizations of professionalism. professions outside the health sciences have explored ebp as a new element of modern professionalism. these linkages are a bit more obscure because the literatures and the literature databases that track these professions do not use the same standardized terminology or classifications for the concept of ebp as found in pubmed or psycinfo. still, a quick glance at these professions outside the health sciences suggests parallel trends regarding the changes to professional status. faculty members serving in higher education, whether junior or quite advanced in their careers, link professionalism with many of the elements of ebp. kram, wasserman, and yip (2012), for example, classify faculty roles into either scholar or practitioner modes. school teachers also think about professionalism in these terms (bourke, ryan, & lidstone, 2012). life coaches similarly associate professionalism in the current era with ebp elements (george, 2013). a few existential therapists meanwhile discuss the possibility of using randomized controlled trials to enhance their professional practices (finlay, 2012), although these psychotherapists might represent a minority (brettle, 2012). interested eblip adherents might want to master the specific vocabularies or ontologies of other fields to explore in far greater detail (and with a broader subject reach than found in this commentary) the advantageous linkages between evolving conceptualizations of professionalism and ebp. such comprehensive investigations might suggest ways that eblip can be linked to the long-term success of the library and information profession.   eblip in professional practice   this commentary has touched on a number of abstract subjects so perhaps it would be helpful to explore how eblip could enhance everyday practice within the new professionalism emerging during the twenty-first century. our core ethical values should align us with our users’ actual or potential needs when making important decisions via the eblip process. with those values in mind, here are some brief vignettes of eblip in action:   vignette one. in your role as collection resources development librarian you need to ensure that most of your users’ needs for authoritative information are met most of the time, despite the constraints of a modest budget. you select collection resources using the eblip process knowing that you must be held accountable to others’ for your decisions as part of the new professionalism. this transparency converges well with long-standing values of openness held by our profession. when others such as administrators or users request an explanation for your decisions, you can readily point to a your eblip process that identified a body of applied research evidence found in the peer reviewed literature, past performance of the same types of resources by your user community, interlibrary loan request data on the same or similar titles, likely a cost-benefit analysis, and possibly even cohort or experimental studies.   vignette two. all teaching at your institution must undergo review by a curricular oversight committee. you are responsible for teaching all students about certain competencies in information literacy knowledge and skills. you design your educational interventions by assessing student needs, reviewing past student evaluations, and by using the best available evidence from both applied library research and educational research. when confronted by one method of teaching versus another, you use the eblip process to find the highest forms of appropriate replicable research evidence to decide on the best course of action. your professional decision demonstrates your expert knowledge, transparent for all on the curriculum oversight committee to review.   vignette three. an administrator above you in the institution speculates aloud that perhaps some of the publicly-used space at your library can be reassigned. this speculation leads you to ask, “why do some users utilize the physical space of the library whereas others do not?” you search the literature for replicable research on what has been learned elsewhere about the uses of library space. perhaps then you conduct focus groups of both actual users and potential users of library space to learn their views. you might even confirm or modulate the focus groups’ findings with a widely-canvassed survey of all potential users in the community that the library serves. this variation of the eblip process enables you to weigh the potential benefits of either enhancing the existing space or exploring other uses with the administrator with an open mind. your process and decided-upon recommendations, informed by your expertise gained from years of professional experience and your values of serving your users, will be on display transparently for the administrator to review.   these three vignettes illustrate a diminished autonomy coupled to an increased accountability for these library and information professionals. these vignettes highlight the central place of relying upon rigorously researched, replicable evidence from both our own profession as well as adjunct professions such as education or management. plutchak’s (2005) argument for the need for a profession to build a robust body of evidence based upon applied research probably deserves further elaboration, but that tangential exploration belongs in a future commentary.   conclusion   the relationship between society and the professions continues to change. society no longer accepts without critical scrutiny the exercise of professional autonomy. instead, society challenges traditional forms of professional autonomy, particularly when decisions are intermingled with individual professional judgment. not even those more traditionally autonomous professions such as medicine and law, which practiced for so many years with few challenges to their authority, are now exempt from society’s critical gaze. library and information practitioners have placed a longstanding value in the transparency of their professional decisions within large organizations so our profession potentially can adapt easily to this shifting societal expectation.   the eblip process enables library and information practitioners to enhance their professional status by displaying a value in serving users and larger society, expertise in the subjects related to decisions made, and critical appraisal of the best evidence available for making these transparent decisions. eblip thereby offers our profession an unprecedented opportunity to demonstrate our expertise and value to society.     references   adams, s., & mccarthy, a. m. (2007). evidence-based practice guidelines and school nursing. journal of school nursing, 23(3), 128-136. http://dx.doi.org/10.1177/10598405070230030201   american board of internal medicine foundation. (2002). medical professionalism in the new millennium: a physician charter. annals of internal medicine, 136(3), 243-246. http://dx.doi.org/10.7326/0003-4819-136-3-200202050-00012   american library association. (2008). code of ethics of the american library association. retrieved from http://www.ala.org/advocacy/proethics/codeofethics/codeethics   armstrong, d. (2002). clinical autonomy, individual and collective: the problem of changing doctors’ behavior. social science & medicine, 55(10), 1771-1777.   berg, m., horstman, k., plass, s., & van heusden, m. (2000). guidelines, professionals, and the production of objectivity: standardization and the professionalism of insurance medicine. sociology of health & illness, 22(6), 765-791. http://dx.doi.org/10.1111/1467-9566.00230   blumer, h. (1951/1995). social movements. in s. m. lyman (ed.), social movements: critiques, concepts, case-studies (pp. 60-83). new york, ny: new york university press.   bonell, c. (1999). evidence-based nursing: a stereotyped view of quantitative and experimental research could work against professional autonomy and authority. journal of advanced nursing, 30(1), 18-23.   booth, a., & eldredge, j. d. (2010). a voyage of discovery: identifying barriers to eblip in the caribbean. evidence based library and information practice, 5(3), 68-72.   bourke, t., ryan, m., & lidstone, j. (2012). reclaiming professionalism for geography education: defending our own territory. teaching and teacher education, 28(7), 990-998. http://dx.doi.org/10.1016/j.tate.2012.05.005   brettle, a. (2012). learning from others about research evidence. evidence based library and information practice, 7(2), 1-2.   canadian health libraries association/association des bibliothèques de la santé du canada. (2007). chla-absc code of ethics. retrieved from http://www.chla-absc.ca/node/27        canadian library association/association canadienne des bibliothèques. (1976). code of ethics. retrieved from http://www.cla.ca/am/template.cfm?section=position_statements&template=/cm/contentdisplay.cfm&contentid=3035        cannavina, c. d., cannavina, g., & walsh, t. f. (2000). effects of evidence-based treatment and consent on professional autonomy. british dental journal, 188(6), 302-306.   chandratilake, m., mcaleer, s., & gibson, j. (2012). cultural similarities and differences in medical professionalism: a multi-region study. medical education, 46(3), 257-266.   chartered institute of library and information professionals. (2012). code of professional practice for library and information professionals. retrieved from http://www.cilip.org.uk/cilip/about/ethics/code-professional-practice       colyer, h. & kamath, p. (1999). evidence-based practice. a philosophical and political analysis: some matters for consideration by professional practitioners. journal of advanced nursing, 29 (1), 188-193. http://dx.doi.org/10.1046/j.1365-2648.1999.00878.x   davies, p. (1999). what is evidence-based education? british journal of educational studies, 47(2), 108-121. http://dx.doi.org/10.1111/1467-8527.00106   dawes, m., summerskill, w., glasziou, p., cartabellotta, a., martin, j., hopayian, . . . osborne, j. (2005). sicily statement on evidence-based practice. bmc medical education, 5(1), 1-7. http://dx.doi.org/10.1186/1472-6920-5-1   denny, k. (1999). evidence-based medicine and medical authority. journal of medical humanities, 20(4), 247-263. http://dx.doi.org/10.1023/a:1022924404779   eldredge, j. d. (2012). the evolution of evidence based library and information practice, part i: defining eblip. evidence based library and information practice, 7(4), 139-145.   eldredge, j. d. (2013). the evolution of evidence based library and information practice, part ii: the broader professional purpose of eblip. evidence based library and information practice, 8(1), 102-110.   etzioni, a. (1969). preface. in a. etzioni (ed.), the semi-professions and their organization (pp. v-xvii). new york, ny: the free press.   finlay, l. (2012). research: an existential predicament for our profession? in l. barnett & g. madison (eds.,) existential therapy: legacy, vibrancy, and dialogue (pp. 183-191). new york, ny: routledge.   george, m. (2013). seeking legitimacy: the professionalization of life coaching. sociological inquiry, 83(2), 179-208. http://dx.doi.org/10.1111/soin.12003   gleeson, d., & knights, d. (2006). challenging dualism: public professionalism in ‘troubled’ times. sociology, 40(2), 277-295. http://dx.doi.org/10.1177/0038038506062033   glynn, l. (2007). defending evidence based practice. evidence based library and information practice, 2(2), 1-2.   hardage, j., zeigler, s., blackwood, j., hartley, g., heitzman, j., & libera, j. (2012). development of a statement on autonomous practice: practice committee, section on geriatrics. journal of geriatric physical therapy, 35(2), 82-85. http://dx.doi.org/10.1519/jpt.0b013e3182476883   ho, m-j., yu, k-h., hirsch, d., huang, t-s., yang, p-c. (2011). does one size fit all? building a framework for medical professionalism. academic medicine, 86(11), 1407-1414. http://dx.doi.org/10.1097/acm.0b013e31823059d1   justice, l. m. (2010). when craft and science collide: improving therapeutic practices through evidence-based innovations. international journal of speech-language pathology, 12(2), 79-86.   koufogiannakis, d. l. (2012a). the state of systematic reviews in library and information studies. evidence based library and information practice, 7(2), 91-95.   koufogiannakis, d. l. (2012b). academic librarians’ conception and use of evidence sources in practice. evidence based library and information practice, 7(4), 5-24.   kram, k. e., wasserman, i. c., & yip, j. (2012). metaphors of identity and professional practice: learning from the scholar-practitioner. journal of applied behavioral science, 48(3), 304-341. http://dx.doi.org/10.1177/0021886312439097   lejonqvist, g. b., eriksson, k., & meretoja, r. (2011). evidence of clinical competence. scandinavian journal of caring sciences, 26(2), 340-348. http://dx.doi.org/10.1111/j.1471-6712.2011.00939.x   lewis, s. (2011). evidence based library and information practice in australia: defining skills and knowledge. health information and libraries journal, 28(2), 152-153. http://dx.doi.org/10.1111/j.1471-1842.2011.00937.x   mackley a., bollinger m., & lynch s. (2012). clinical research nursing: evidence today, better practices tomorrow. nursing for women’s health, 16(4), 325-329. http://dx.doi.org/10.1111/j.1751-486x.2012.01751.x   medical library association. (2010). code of ethics for health sciences librarianship. retrieved from http://www.mlanet.org/about/ethics.html   mykhalovskiy, e. & weir, l. (2004). the problem of evidence-based medicine: directions for social science. social science and medicine, 59(5), 1059-69. http://dx.doi.org/10.1016/j.socscimed.2003.12.002   muzio, d., & kirkpatrick, i. (2011). introduction: professionalism and organizations a conceptual framework. current sociology, 59(4), 389-405. http://dx.doi.org/10.1177/0011392111402584   noordegraaf, m. (2007). from “pure” to “hybrid” professionalism: present-day professionalism in ambiguous public domains. administration & society, 39(6), 761-785. http://dx.doi.org/10.1177/0095399707304434   noordegraaf, m. (2011). risky business: how professional fields (must) deal with organizational issues. organization studies, 32(10), 1349-1371. http://dx.doi.org/10.1177/0170840611416748   plutchak, t. s. (2005). building a body of evidence. journal of the medical library association, 93(2), 193-195.   pfeffer, j. (2011). management a profession? where’s the proof? harvard business review, 89(9), 38.   reavy, k., & tavernier, s. (2008). nurses reclaiming ownership of their practice: implementation of an evidence-based practice model and process. journal of continuing education in nursing, 39(4), 166-172.   reich, z., & hanitzsch, t. (2013). determinants’ of journalists’ professional autonomy: individual and national level factors matter more than organizational ones. mass communication and society, 16(1), 133-156. http://dx.doi.org/10.1080/15205436.2012.669002   rousseau, d. m. (2012). oxford handbook of evidence-based management. new york, ny: oxford university press.   rousseau, j-j. (1983). on the social contract (d. a. cress, trans.). indianapolis, in: hackett press.   slavin, r. e. (2002). evidence-based education policies: transforming educational practice and research. educational researcher, 31(7), 15-21. http://dx.doi.org/10.3102/0013189x031007015   timmermans, s. (2005). from autonomy to accountability: the role of clinical practice guidelines in professional power. perspectives in biology and medicine, 48(4), 490-501.   vanhook, p. m. (2009). overcoming the barriers to ebp. nursing management, 40(8), 9-11. http://dx.doi.org/10.1097/01.numa.0000359200.14210.af   wagner, p., hendrich, j., moseley, g., & hudson, v. (2007). defining medical professionalism: a qualitative study. medical education, 41(3), 288-294.   williams, r. (2004). evidence into practice: the culture and context of modern professionalism in psychiatry. current opinion in psychiatry, 17(4), 237-242. http://dx.doi.org/10.1097/01.yco.0000133824.09887.23   microsoft word comm.doc evidence based library and information practice 2007, 2:4  108 evidence based library and information practice     commentary    on evidence based practice and innovation      denise koufogiannakis  collections & acquisitions coordinator, university of alberta libraries  edmonton, alberta, canada  e‐mail: denise.koufogiannakis@ualberta.ca      received: 10 october 2007    accepted: 18 november 2007      © 2007 koufogiannakis. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      since the eblip4 conference, held in north  carolina in may 2007, i’ve been thinking a  lot about innovation.  innovation is “new  knowledge incorporated into products,  processes and services” (afuah cited in  popadiuk and choo 303). i heard innovation  mentioned a few times at the conference,  and it was always noted as being at odds  with evidence based practice. people were  asking how does innovation fit into the  world of ebp, when decisions are supposed  to be based on existing research evidence?  how can anything new ever be  accomplished and still fit within this model  as it has been presented to date? i had never  really heard this expressed in such a way  before, so it somewhat caught me off‐guard.  i have always thought that innovation and  evidence based practice go hand‐in‐hand,  complementing one another for the  betterment of our profession. it is time to  ensure that we address this question within  evidence based practice to quell some of the  criticism that ebp deters innovation, and to  find better ways of incorporating innovation  into the eblip model of practice.  to date, evidence based library and  information practice has focused upon a  process of asking relevant questions, finding  research information, critically appraising  the quality of what exists, and implementing  that knowledge into practice. this focus has  been a necessary one, and still requires  much work in order to be fully used and  implemented in a practical way. the current  gap between research and practice requires  that eblip‐minded folks ask these questions,  find the best available answers and try to  implement them. to my mind, this is an  essential part of being a well‐informed  practitioner.     but what about when there are no answers  to be found? what about when you have an  idea that has not been tried or tested before?  when you have no evidence on which to  mailto:koufogiannakis@ualberta.ca http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:4  109 base your practice?  do you simply stop and  decide that there is not enough evidence to  move forward? wait until someone else  figures it out and tests it? of course not!     only a couple of published papers have  looked at innovation and how it fits with  evidence based library and information  practice. five years ago, booth advocated  caution when adopting innovative projects.  he noted that the very newness of a project  creates a bias that leads to favourable  outcomes and so we should look at positive  results with a sceptical eye. booth’s caution  is not that we shouldn’t move ahead with  innovative projects, but that “the impact of a  project over a comparatively short period  [should not be] viewed too favourably in  comparison with the longer‐term  sustainability of an established service”  (240).     in contrast, smith, in a paper for the 2003  ifla conference in berlin, takes the  viewpoint that being too cautious may stifle  innovation. he writes, “if there is over‐ reliance on the evidence of proven solutions,  there is an inherent risk of closing off  considerations of alternative approaches.  constraining innovation may be a low‐risk  strategy but it is also a constricting one;  falling back on the tested can quickly  become a slippery slope to relying on the  comfortable” (2).    i propose that we can be innovative while at  the same time being evidence based  practitioners. innovation is an essential part  of what moves our profession forward.  when there is a problem and someone  thinks of a way that the problem could be  solved, or a process be made more efficient,  or a patron better served, these are times  when we make progress. this is not at odds  with being an evidence based practitioner.  deiss notes that “the practice of observing  customers’ information‐seeking behaviours,  for instance, is likely to yield information  about where an innovation might really  have value” (22). knowledge gained from  research is crucial to understanding when  and why we should propose new  innovations that are appropriate and will be  embraced. the two concepts are not at odds  with one another, but rather are mutually  beneficial.    what is at odds with being an evidence  based practitioner is moving ahead with  innovative services blindly. it is not enough  to just implement a new service without first  determining whether another library has  done something similar and how it worked  for them. if no research studies exist, are  there any case reports that may provide  guidance? by reviewing what has gone  before, one obtains a sense of the evidence  base for that particular area of our  profession. other fields beyond  librarianship may also yield relevant  research studies and be transferable to our  environment. if the evidence is lacking, then  one can still move ahead even though there  is no directly relevant previous research.  this is generally where innovative  practitioners will find themselves, since it is  unlikely that evidence already exists for  something that is a new solution.    a next step would be to plan an innovative  project with research and assessment in  mind.  from the outset of a new service,  think about what the intended outcomes are  and implement measures to determine  success. at the university of alberta, i  ensured that assessment measures were in  place from the start of our new patron’s  choice project, an initiative to purchase  books requested via interlibrary loan in  order to fill gaps in our collection based on  user‐need (koufogiannakis 2007). this  project was innovative for our institution,  but we were able to find a small body of  literature to better inform our decision  making at the outset, while incorporating  data collection in order to track variables  evidence based library and information practice 2007, 2:4  110 such as goals for turn‐around time, use of  the materials ordered, and subject areas of  books purchased. this project will be  evaluated in the fall and winter of 2007‐08 as  it reaches the first full year of  implementation.    in implementing a new service, evidence  based practitioners should view it as their  responsibility to check the research evidence  for ideas, document and test the service  being implemented, then provide that  information to the wider library community.  in some cases you may be able to test a new  service alongside a traditional service, with  a comparative research project. combining  an innovative idea with a research project,  whenever possible, will allow for testing of  the new innovation and contribute to the  evidence‐base of the profession so that  others can learn from such successes or  failures. both successes and failures need to  be highlighted, so that a fuller picture can  emerge. we cannot be afraid to report when  a new innovation fails. i am confident that i  practice what i preach in this respect. the  university of alberta libraries’ attempt at  integrating librarians into first year medical  students’ problem based learning groups  was shown by our research not to have an  impact (koufogiannakis et al. 2005), so we  stopped the practice and published our less  than positive results!    smith challenged the profession of  librarianship, “to ensure that commendable  risk‐aversion does not lead inexorably to a  retreat into the comfort zone which will  become marginalized because of the scale of  the external context of change” (3). clearly  we live in times of tremendous change, and  innovative solutions are required for  managing the impact of that change upon  our libraries. our need for good proof upon  which to base our decisions may not easily  be found. we cannot sit back and wait for  evidence to fall into our lap, but we can  move ahead while at the same time  incorporating the collection of evidence so  that others can make more informed  decisions in the future.     librarians who move their organizations  forward will be evidence based innovators,  combining innovative ideas with evidence  based principles. they will draw upon the  research knowledge of the profession,  integrate that knowledge into their own  local circumstances and needs, and find  innovative solutions to problems where no  good solution exists. the critical appraisal  skills which allow an evidence based  practitioner to read the literature with a  questioning mind also bring them to  question the way things currently work at  their institution. an inquiring mind will  always be learning, questioning and  problem‐solving, only to go through the  entire process again when a better solution  is possible. innovation, together with  evidence based practice, can guide us  through whatever the future may hold for  our profession.    a colleague of mine always says “don’t  throw the proverbial baby out with the  bathwater.” let’s be innovative, but not to  the point of getting rid of or damaging what  we know is good. this is really common  sense that brings risk and evidence together  in a balanced way.      works cited    booth, andrew. “on a cautious adoption  of innovative projects.” health  information and libraries journal 19.4  (2002): 230‐42.    deiss, kathryn j. “innovation and strategy:  rick and choice in shaping user‐ centered libraries.” library trends  53.1 (2004): 17‐32.        evidence based library and information practice 2007, 2:4  111 koufogiannakis, denise. “establishing a  model for evidence based collection  management.” 4th international  evidence based library & information  practice conference. durham, north  carolina, usa. 7 may 2007. 28 nov.  2007  <http://www.eblip4.unc.edu/papers/ko ufogiannakis.pdf>      koufogiannakis, denise, jeanette  buckingham, arif alibhai and david  rayner. ʺimpact of librarians in first‐ year medical and dental student  problem‐based learning (pbl) groups:  a controlled study.ʺ health  information and libraries journal 22.3  (2005): 189‐95.    popadiuk, silvio, and chun wei choo.  “innovation and knowledge creation:  how are these concepts related?”  international journal of information  management 26.4 (2006): 302‐12.    smith, alan. “innovation – the creative  tension of risk and evidence.” world  library and information congress: 69th  ifla general conference and council.  berlin, ger. 5 aug. 2003. 28 nov. 2007  <http://www.ifla.org/iv/ifla69/papers/0 68e‐smith.pdf>            http://www.eblip4.unc.edu/papers/koufogiannakis.pdf http://www.eblip4.unc.edu/papers/koufogiannakis.pdf http://www.ifla.org/iv/ifla69/papers/068e%e2%80%90smith.pdf http://www.ifla.org/iv/ifla69/papers/068e%e2%80%90smith.pdf evidence summary   students and libraries may benefit from late night hours   a review of: scarletto, e. a., burhanna, k. j., & richardson, e. (2013). wide awake at 4 am: a study of late night user behavior, perceptions and performance at an academic library. journal of academic librarianship, 39(5), 371-377. http://dx.doi.org/10.1016/j.acalib.2013.02.006   reviewed by: richard hayman assistant professor & digital initiatives librarian mount royal university calgary, alberta, canada email: rhayman@mtroyal.ca   received: 23 nov. 2014 accepted: 10 feb. 2015      2015 hayman. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to assess late night library usage, including a demographic profile of students benefitting from late night hours, with an analysis of the services and resources they used, and whether the use of late hours is connected to student success.   design – a mixed-methods approach including quantitative demographic information alongside qualitative user feedback collected using a web-based survey.   setting – a large, public research university library in the united states of america using late night operating hours (11 p.m. to 7:30 a.m.) to create 24-hour library availability 5 days per week.   subjects – undergraduate and graduate students.   methods – using the university’s building monitoring database (bmd), researchers collected data on which students were using the library building when late night hours were in effect for fall and spring semesters. along with the date and time of entry, the bmd collected the university id number of the students and their email address. using student id numbers, information from the bmd was cross-referenced with anonymized demographic information from the university’s institutional planning office, enabling comparisons across a range of other data, including students’ discipline, gpa, and other information.   researchers emailed students the web-based survey, directly targeting users who had made use of the library’s late night operating hours. survey questions investigated when students used the library, explored student aims when in the library, and asked students to rank the tasks they were trying to accomplish while in the library. in addition, researchers sought student feedback on what services and resources they used during late hours, asking students to identify services and resources they would have liked to use but which were not offered during late night hours, and inquiring about students’ sense of safety and security when using the library late at night.   main results – in total, researchers report that 5,822 students, representing approximately 21% of the campus population, visited the library during the late hours, for 22,383 visits. researchers report that 57% of late night users took advantage of the extended hours on more than one occasion, with 39% returning three or more times. sundays were the most popular day, while thursdays were least popular. researchers also tracked entry times, with the most popular entry times occurring between 11 p.m. to 2 a.m., accounting for 80% of all late night visits. while survey respondents were drawn entirely from the late night users, 63% preferred using the library late at night versus standard daytime operating hours.   the overall survey response rate was less than 5%. survey respondents (n=243) reported participating in a variety of activities while visiting during late night library hours, with quiet study (87%), working on projects or papers (72%), and group study (42%) as the most frequently reported activities. respondents also ranked the top three activities they hoped to accomplish while in the library: quiet study (50%) and work on projects and papers (34%) remained top activities, though group study (20%) fell to fourth place, slightly behind the third-ranked activity of printing or copying documents (22%).   respondents reported their use of services during late night hours, indicating use of the university’s wireless internet access, library printers, computers, and online databases and electronic resources. the only staff service point available to students during overnight hours, circulation services, was used by 16% of respondents. regarding student responses about what services were not offered that they would have liked to have available, the researchers reveal that “[f]ood and drink were overwhelmingly the most frequently requested services” (p. 374), followed by a desire for more comfortable furniture and spaces. some respondents also requested that late night hours be extended to seven days a week. overall, 96% of those submitting user feedback reported a sense of security, an important consideration for late night hours.   regarding the sample, researchers found that the population of late night library users closely reflected the overall university population. an independent samples t-test comparing the differences between the average gpa of late night users to the average gpa of the overall university population was statistically insignificant. for undergraduates only, there exists a small difference between the retention rate of students using late night hours (84.6%) versus overall university retention rate (80.2%). a pearson’s chi-squared test revealed a statistically significant association between late night library use and retention rates for both undergraduate and graduate populations when compared to the retention rates of non-late night users across the same student groups.   conclusion – this study reveals that undergraduate students in particular make use of late night hours for studying or other academic activities, and positions the late night model as a successful customer service offering at kent state university. although researchers do conclusively connect the availability of late night hours to student retention and academic success, their study points to the need for further research exploring this question.   commentary   the study anticipates trends recently discovered elsewhere, and contributes to the broader conversation of library operating hours. for example, sanders and hodges’ (2014) related study found that 22% of the on-campus student population visited the library between 11 p.m. and 8 a.m. (p. 314). meanwhile, sowell and nutefall (2014) report that 90% of student survey respondents (n=825) agreed that the availability of late hours had a positive impact on their academic success (p. 108). while further study is needed to determine the connections, if any, between student success and retention and library operating hours, it is hard to deny the positive perceptions that students have regarding late night hours.   with a minimal staffing model and only one service point available, this research does not provide evidence regarding how academic libraries should staff late night operations. however, the qualitative evidence from this study provides valuable information on the types of services, resources, and technologies required by students who take advantage of late night hours. libraries looking to offer 24-hour access should be prepared to have basic university technology resources, such as wireless internet access and printing. the researchers note the minimal demand for technical support for these services, perhaps suggesting that late night users do not expect the same level of service as during the day. it is unsurprising that late night users require options for food and drink, and desire comfortable spaces when working late at night, and this study reminds us that libraries should account for this need if drawing on this study as evidence for their own operating hours decisions. finally, the inclusion of the survey instrument as an appendix would strengthen this article’s presentation of survey results. without knowing the questions posed and other details of the instrument, it is difficult to judge the validity of the data, including the reliability and replicability of the study.   the large quantitative sample size, comprehensive quantitative data collection, and comparisons to overall institutional data all help contribute to the validity of this research (glynn, 2006). regarding how data are reported, n-values are only given for some of the responses, while others are only reported as percentages, making direct statistical comparisons a challenge. this shortcoming is compounded by the fact that the authors report different n-values for the quantitative dataset throughout the manuscript. this data, objectively collected by the building monitoring system, should remain constant. if there are corrected n-values for reporting different results, the reasons for these corrections should be described in detail for the benefit of the reader. the combination of minimal statistical analysis and unexplained shifts in n-value reporting do call into question the validity of specific results being reported. this is particularly worth noting when it comes to an important research question informing this study, about the attempt to connect late night hours to academic success and student retention, which ultimately receives limited attention.   as university libraries look for additional ways to assess their student impact under tight budget constraints, and as students demand greater flexibility, operating hours will continually come under examination. even with the limitations of this study, libraries considering the adoption of late night hours should find this research to be useful evidence that extended hours can have positive customer service benefits, and that such hours address students’ need for study space at times most convenient for them.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154   sanders, m., & hodges, c. (2014). an overnight success?: usage patterns and demographics of academic library patrons during the overnight period from 11 p.m.-8 a.m. journal of access services, 11(4), 309-320. http://dx.doi.org/10.1080/15367967.2014.945121   sowell, s. l., & nutefall, j. e. (2014). mysteries in the night: an exploratory study of student use and perceptions of 24/5 hours. public services quarterly, 10(2), 96-114. http://dx.doi.org/10.1080/15228959.2014.904676 evidence based library and information practice evidence based library and information practice 2010, 5.2  evidence based library and information practice       editorial    thoughts on reflection    denise koufogiannakis  editor‐in‐chief  collections & acquisitions coordinator, university of alberta libraries  edmonton, alberta, canada  email: denise.koufogiannakis@ualberta.ca       2010 koufogiannakis. this is an open access article distributed under the terms of the creative commons‐ attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐ sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original  work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is  redistributed under the same or similar license to this one.    there has been some acknowledgement in the  published literature that reflection is a crucial  element of the evidence based library and  information practice (eblip) model we have  adopted (booth 2004, 2006; grant 2007;  helliwell 2007). as we work through a  problem and try to incorporate the best  available evidence into our decision making,  reflection is required at several stages,  including the very identification of the  problem through to our assessment of the  process itself and what we have learned in  order to inform future practice. however,  reflection and reflective writing have not fully  been integrated into the process we espouse,  and very little has been done to look more  closely at this element of the model and how it  can be integrated into professional learning.    in a recently published research article, sen  (2010) confirms the relationship between  reflection and several aspects of professional  practice. these include critical review and  decision making, two aspects that are tied  closely to the evidence based process. sen  notes:     students were more likely to show  evidence of learning, self‐ development, the ability to review  issues crucially, awareness of their  own mental functions, ability to make  decision [sic] and being empowered  when they had mastered the art of  reflective practice and the more  deeply analytical reflective writing.  (p.84)    eblip (the journal) tries to incorporate  elements of reflection within the articles we  publish. while we clearly believe in the need  for our profession to do quality research and  publish that research so that it can be  accessible to practitioners, we also know that  research cannot be looked at in isolation. our  evidence summaries are one way of reflecting  critically on previously published research,  and in the same vein, our classics bring older  research studies back to the foreground. this  work needs to continue to be discussed and  looked at for its impact on our profession.     more directly, the using evidence in practice  section uses a standardized format enabling  1 mailto:denise.koufogiannakis@ualberta.ca evidence based library and information practice 2010, 5.2  practitioners to share their experience of  integrating research evidence into their  practice. the final section of these brief articles  asks the writers to reflect on their experience.  although it is not research, the individual  reflection allies with what schön (1983) called  “reflection on action” and such reflections  over time form a practical, tacit knowledge  that we use to inform our work. within this  section of the journal, we hope readers will  become more aware of how different types of  evidence can be integrated into real‐world  decision making. not everything requires a  full blown research study, and this section  allows readers to see what other practitioners  are doing, and in turn it should enable them to  reflect upon what they are doing in their own  practice.    being aware of situations where things may or  may not have worked, and reflecting on the  reasons why, brings together our sense of  critical thought and practical experience that  go a long way in filling the “librarian  observed” and “professional judgements”  parts of the eblip definition (booth and brice  2004). acquiring professional knowledge does  not end when we complete a graduate  program, or have a certain number of years  experience under our belts. it needs to be  continually and consciously cultivated via  reflection on our practice, our research, and  simply what works and why.      research knowledge only takes us so far.  people often ask me, “what do i do when  there is no evidence? or when the research  evidence is weak?” does this stop us from  moving ahead? no. a decision still needs to be  made. evidence based practice is not only  about acting when there is good evidence.  enhancing our professional judgments via a  career built on analytical reflection, will  provide knowledge that goes a long way  towards making difficult decisions a little bit  easier; even (or perhaps, especially) in the  cases when there is already a large body of  research evidence.    perhaps eblip proponents need to start  thinking about the best ways to be reflective  practitioners. could we start a new type of  “journal club” where practitioners keep  journals as reflective tools, and meet regularly  to work through problems and give one  another feedback? or pair up with a reflective  partner to have discussions on critical  incidents that occurred within our practice? i  am sure some people are already doing this, or  are finding other mediums such as blogs,  wikis, and mailing lists to reflect on elements  of practice and engage others who are  interested in the topic. however, coming up  with concrete strategies for developing  reflection within our profession needs to be  more widely discussed and articulated. our  professional bodies would be a good starting  place, and via these bodies developing skills  through continuing professional development.  we can learn a lot from other fields such as  education and nursing, which have more  fully incorporated elements of reflection into  their practice. we also need to consider the  best way to integrate our reflection with  available research. i’m sure there are many  ideas just waiting to be put into action, or  already happening (in which case we would  love to hear about them). as with anything  worthwhile, there is no simple formula for  success, but there is a valuable goal; in this  case, improving our practice for the benefit of  our patrons and a better understanding of  how and why we do what we do as librarians  and information professionals.      references    booth, a. (2006). clear and present questions:   formulating questions for evidence  based practice. library hi tech, 24(3),  355‐368.    booth, a. (2004). evaluating your   performance. in: booth, a. & brice, a.  (eds). evidence‐based practice for  information professionals: a handbook  (pp. 127–137). london: facet  publishing.        2 evidence based library and information practice 2010, 5.2  booth, a. and brice, a. (2004). why   evidence‐based information  practice? in: booth, a. & brice, a.   (eds). evidence‐based practice for  information professionals: a handbook  (pp. 1‐12). london: facet  publishing.    grant, m. (2007). the role of reflection in the  library and information sector: a  systematic review. health  information and libraries journal,  24(3), 155‐166.    helliwell, m. (2007). reflections of a  practitioner in an evidence based  world: 4th international evidence  based library & information  practice conference. evidence based  library and information practice 2(2),  120‐122.    sen, b. a. (2010).  reflective writing: a   management skill. library  management, 31(1/2), 79‐93.    schön, d.a. (1983). the reflective practitioner:   how professionals think in action. new  york: basic books.  3 evidence summary   google scholar could be used as a stand-alone resource for systematic reviews   a review of: gehanno, j. f., rollin, l., & darmoni, s. (2013). is the coverage of google scholar enough to be used alone for systematic reviews. bmc medical informatics and decision making, 13(1): 7. doi: 10.1186/1472-6947-13-7   reviewed by: saori wendy herman education and liaison librarian hofstra north shore-lij school of medicine hempstead, ny, united states of america email: saori.w.yoshioka@gmail.com   received: 27 jan. 2015   accepted: 27 apr. 2015    2015 herman. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine if google scholar (gs) is sensitive enough to be used as the sole search tool for systematic reviews.   design – citation analysis.   setting – biomedical literature.    subjects – original studies included in 29 systematic reviews published in the cochrane library or jama.   methods – the authors searched medline for any systematic reviews published in the 2008 and 2009 issues of jama or in the july 8, 2009 issue of the cochrane database of systematic reviews. they chose 29 systematic reviews for the study and included these reviews in a gold standard database created specifically for this project. the authors searched gs for the title of each of the original references for the 29 reviews. they computed and noted the recall of gs for each reference.   main results – the authors searched gs for 738 original studies with a 100% recall rate. they also made a side discovery of a number of major errors in the bibliographic references.   conclusion – researchers could use gs as a stand-alone database for systematic reviews or meta-analyses. with a couple improvements to the rate of positive predictive values and advanced search features, gs could become the leading medical bibliographic database.   commentary   the number of studies examining the content coverage, accuracy, precision, and recall rate of google scholar (gs) compared to other medical bibliographic databases continues to increase. a majority of these studies conclude that although gs can be used as one of several bibliographic databases for literature retrieval, researchers should not use it as a stand-alone tool (bramer, giustini, kramer, & anderson, 2013; giustini & kamel boulos, 2013). the authors of this study conclude otherwise, indicating that gs is sensitive enough to use as a stand-alone resource when performing systematic reviews.   this reviewer disagrees with the authors’ conclusion. in applying the ebl critical appraisal checklist (glynn, 2006), several concerns arose about the study validity and applicability, focusing primarily on the methods used and the resulting conclusion.   the authors searched gs for the article titles of the references for 29 systematic reviews and retrieved a recall rate of 100%. although the authors explain the methods used to select the systematic reviews and the inclusion and exclusion criteria for the references they included in their gs search, the authors do not explain in depth why the recall rate was so high. the only explanation mentioned is the ability for gs to access “the ‘invisible’ or ‘deep’ web” through “agreements with publishers” (p. 4). the authors do not explain the method or algorithm, which can potentially affect the reproducibility of a gs search, used to access the “deep” web.   furthermore, the 100% recall rate merely indicates that gs was able to re-find articles that the authors knew already existed. this does not indicate whether gs’s search algorithm has better accuracy, precision, or recall rate compared to other medical bibliographic databases. researchers should review all of these factors along with content coverage to make a final decision about whether a database is strong enough to use as a stand-alone resource.   instead of re-finding existing references, the authors should have performed a search in gs and other medical bibliographic databases using a reproducible search string and compared the results. doing so would have better assessed the elements (content coverage, accuracy, precision, and recall rate) needed to determine whether gs is an effective stand-alone resource.   one conclusion the reviewer agrees with is that gs’s coverage is more extensive than previously thought. gs has improved to the point that researchers can consider it as a possible resource to use when performing literature searches, systematic reviews, or meta-analyses. the caveat to this conclusion is to use gs in combination with other medical bibliographic databases.   the study provides insight into the growing usage of gs and the importance of paying close attention to the methodology of similar studies. librarians performing literature searches, systematic reviews, or meta-analyses should be well informed about these types of research studies and utilize them to improve their own searching practices.   references   bramer, w. m., giustini, d., kramer, b. m. r., & anderson, p. f. (2013). the comparative recall of google scholar versus pubmed in identical searches for biomedical systematic reviews: a review of searches used in systematic reviews. systematic reviews, 2: 115. doi: 10.1186/2046-4053-2-115   giustini, d., & kamel boulos, m. n. (2013). google scholar is not enough to be used alone for systematic reviews. online journal of public health informatics, 5(2). doi: 10.5210/ojphi.v5i2.4623   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi: 10.1108/07378830610692154   editorial   barriers to research and evidence   alison brettle editor-in-chief school of nursing, midwifery and social work university of salford, united kingdom email: a.brettle@salford.ac.uk      2013 brettle. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   i often find attending conferences or workshops a source of reflection or inspiration for editorials, and today i attended an event that proved to be no exception. the healer network is a uk grouping of professionals interested in health library research.  it brings together those working in health information at an academic, practitioner or strategic capacity as well as those working in higher education, research and the nhs. (http://www.libraryservices.nhs.uk/healer/minutes.html)    there were a number of interesting presentations, but one (and the subsequent interactive discussions) left me with some worrying thoughts. hannah spring (2013) presented some of the findings from her phd that found when health librarians were asked about their barriers to research they reported that they didn't know what research questions to ask! alternatively if they had research questions they didn't think to engage with the literature or believed that there was no evidence to answer them!  if we really don't have any research questions, and we really don't think to look at the literature or there really is no evidence, this is worrying indeed for the future of eblip.  it's also a situation i don't recognize from being involved in the eblip journal and was left wondering whether it was the health librarians perceptions of “research” and “evidence” that was the issue; questions which are being examined in the lirg scan which was described in another presentation.  the scan is a review of the evidence on: what practitioners understand by research; what kind of research is relevant to lis practitioners?  how do they use research and what are the barriers and facilitators to using research in practice? (https://sites.google.com/site/lirgweb/home/awards/lirg-scan-award)  the results will be used to help inform the chartered institute for library and information professionals’ policy on research.   the barrier which i’m much more familiar in terms of engaging with research or evidence, is that of accessing the literature.  our strap line says "eblip is an open access, peer reviewed journal which provides a forum for librarians and other information professionals to discover research that may contribute to decision making in professional practice. this is an ethos of which we are proud because we aim to overcome the barriers to making research accessible.  for eblip, as a journal that strives to make lis research accessible to the practitioner, open access is the only way forward.   i'm pleased to say that this issue is full of both research and evidence, and i hope it will answer some of the questions that arise from your practice, or at least go some way towards doing so.  i'm particularly pleased that we have a feature section on assessment that comprises articles from the 2010 library assessment conference which took place in baltimore, maryland, usa.  the articles have been peer reviewed at abstract level for acceptance to the conference but appear here as “conference papers”. we are happy to provide an open access forum to this conference material, and welcome the addition of the assessment agenda to the eblip journal.    assessment in libraries is an area which isn't always labeled research or evidence based practice but nevertheless provides a wealth of evidence and helps answer questions and demonstrate the impact and value of our library services. these are areas that are crucial to and synonymous with evidence based practice.  at our healer day, demonstrating value and impact were discussed as areas of research interest that may not always be seen as research by librarians.  demonstrating value and impact is a recurring theme within this issue as we also have a conventional article that was presented at the 2012 assessment conference (griffin et al, 2013) as well as carol tenopir’s’ commentary on measuring value based on a keynote address at the dream conference that was held in the uk in july 2012.   from this issue onwards there are a number of changes within our editorial team.  wayne jones has left to take up new interests and i would like to thank denise koufogiannakis for covering in the interim period.  lorie kloda is moving to associate editor (articles) and she will be joined by derek rodriguez from university of north carolina, chapel hill as a second associate editor (articles) to help with our increasing workload.  heather pretty is moving to associate editor (evidence summaries) and michelle dunaway is our new production editor.  we have a new editorial intern, archana deshmukh from the university of brighton and richard hayman will be our new lead copy editor.    finally if you are going to be attending eblip7 in saskatoon in july, i look forward to meeting you there and providing further updates about the eblip journal.     references   griffin, m, lewis, b., greenberg, m.i. (2013) data-driven decision making: an holistic approach to assessment in special collections repositories, evidence based library and information practice, 8(2), 225–238.   spring, h. (2013, may.) barriers to and priorities for research development in health librarianship: the results and recommendations from phd research.  paper presented at health information and libraries for evaluation and research meeting, york st john university, uk.  retrieved 7 june from http://www.libraryservices.nhs.uk/healer/minutes.html   evidence summary   academics in the uk use social media to enhance traditional scholarly reading   a review of: tenopir, c., volentine, r., & king, d.w. (2013). social media and scholarly reading. online information review, 37(2), 193-216. doi:10.1108/oir-04-2012-0062   reviewed by: annie m. hughes reference librarian wilson dental library, university of southern california los angeles, california, united states of america email: amhughes@usc.edu   received: 26 aug. 2013 accepted: 25 oct. 2013      2013 hughes. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to investigate academic staff’s use and creation of social media for work-related purposes as well as analyze how this influences consumption of traditional scholarly resources.   design – a 60 question survey instrument in questionnaire format (instrument appended to the article).   setting – six universities in the united kingdom.   subjects – respondents include 2,117 academic staff or faculty.   methods – in 2011, the authors sent a 60 question survey to 6 universities in the united kingdom. library directors were asked to send out the survey to their university’s academic staff, and 12,600 invitations were sent out. an application to the institutional review board was also implemented and approved as long as the respondents could exit the survey at any time. there were 2,117 respondents with a maximum of 1,078 respondents to the questions regarding social media.   questions included in the survey asked the frequency of use of traditional scholarly resources with a focus on reading articles and books. respondents were also asked to answer questions regarding how they accessed resources and how they used what they were reading. they were asked to provide information regarding accessing other publications such as conference proceedings, government documents, and magazines or trade journals. the authors also recorded demographic information such as respondent’s field of study, position within the university, age, and gender. following questions regarding use of traditional resources and demographics, respondents were asked to answer questions regarding social media use and creation for work-related purposes.   main results – the authors asked respondents how much traditional scholarly reading they did in the last month. with regard to traditional scholarly reading activity, the authors found that academic staff in the united kingdom read about 22 articles, with medical and health sciences field consuming the most articles and social scientists consuming the least. book and book chapter reading is more prevalent in the humanities discipline as they, on average, read about 20 books or book chapters. the average across disciplines is seven books or book chapters, with the medical and health sciences academic staff reading the least. “other publications” were also accounted for such as government documents, trade journals, and conference proceedings, and on average, academics read about 11 in the last month. after data was collected, authors excluded outliers over three standard deviations from the mean to assure a more representative average.   respondents were asked how often they use social media such as blogs, online videos, rss feeds, twitter, user comments in online articles, podcasts, and other. the authors found that academics in the united kingdom use social media occasionally, but not on a regular basis. they also found that social media is less likely to be created than used. occasional use is recorded by half of the respondents who use only two of the resources listed in the survey. only 5% of the respondents said they use all of the social media tools listed. over half of the respondents said they do not “create” social media tools for work. participation and use of the tools is much more prevalent according to the results of this survey.   regarding demographic responses recorded, those who are in the humanities and medical and health sciences use more blogs for their work, and those in the medical and health sciences also participate most in user comments in articles. younger respondents (under 30-40 years of age) use more social media tools such as blogs, rss feeds, and twitter. those who are actively teaching tend to use social media more frequently and while they do not create tools more frequently than others, they do create the most online videos out of any of the tools mentioned. there was no significant association between use of social media and the respondent’s position, gender, or the number of awards earned. respondents who “tweet” or use twitter consume the most amount of scholarly material. overall, the authors found that those who participate in social media also consume a significant amount of traditional scholarly content.   conclusion – the authors conclude that while most academics in the united kingdom participate in use of social media for work-related purposes, the results show that usage is not as frequent as expected. creation of these tools is even less frequent, although the survey did show that academics who consume traditional scholarly resources tend to also consume social media more frequently. the use of social media is also not replacing traditional scholarly resources, but instead they are used alongside as part of the vast amount of information sources available to scholars. publishers and others who are tasked with creation of scholarly content should consider the addition of social media tools into products. the article also implies that when academics can easily access both traditional and social media tools and use them in conjunction, the use of social media tools in academia will grow.     commentary   the use of social media to enhance scholarly communication activities has increased in the last decade, and the authors have found evidence by surveying multiple academics at universities in the united kingdom. while traditional scholarly resources are still being utilized in the larger academic community, the use and creation of social media tools such as blogs, twitter, and rss feeds are increasingly prevalent and do not result in disuse of the traditional format.   the authors of this paper chose to focus on the frequency of use and creation of social media in relation to use of traditional scholarly resources. they found that academics are using both types of resources to gain information, however they did not necessarily address the “how” and “where” of the use and creation of social media. are they using these tools to enhance their teaching or are they using tools to enhance their own personal growth with regard to their work? what are the reasons for using these tools versus using traditional scholarly resources? also, the word “create” to describe tweeting or blog posting or rss feed creation is a bit misleading. academics are not creating tweets but are participating in twitter or “using” twitter. researchers create rss feeds far less than they consume them or subscribe to them. there are also various levels of creation with regard to social media. are the blogs that respondents create research related or are they used for classroom purposes? these questions left unanswered are, of course, an opportunity for further research, however the authors did ask how the respondents use traditional scholarly resources.   with regard to limitations of the study, the authors were thorough in mentioning that the survey was self-reported and there could be some recall bias in relation to how much reading or social media use the respondents use. they also make note that they made the assumption that the six universities included in the study are typical of all research universities in the united kingdom and elsewhere in the western world. the researchers also derive data from a convenience sample and therefore do not provide a representative sample of the population.   the authors collected information regarding demographics that proves to be interesting. the information could be useful to those interested in discipline specific use of social media tools and traditional resources. age is also an interesting factor with regard to use of social media versus traditional scholarly resources and the authors addressed this issue.   overall, the information presented in this paper is useful to those interested in researching the current use of traditional scholarly resources. the data shows that academics use both to enhance scholarly communication and have not dismissed traditional methods for new tools. the authors imply if publishers provide access to social media tools via traditional scholarly materials, use of these tools will grow.   the evidence based library and information practice (eblip) critical appraisal checklist was used to calculate validity (glynn, 2006). the study is found to be valid.     references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi:10.1108/07378830610692154   article   far from a trivial pursuit: assessing the effectiveness of games in information literacy instruction   eamon tewell reference & instruction librarian long island university, brooklyn campus brooklyn, new york, united states of america email: eamon.tewell@liu.edu   katelyn angell reference & instruction librarian long island university, brooklyn campus brooklyn, new york, united states of america email: katelyn.angell@liu.edu   received: 2 aug. 2014     accepted: 2 feb. 2015      2015 tewell and angell. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to determine whether playing library-related online games during information literacy instruction sessions improves student performance on questionnaires pertaining to selected research practices: identifying citation types and keyword and synonym development.   methods – 86 students in seven introductory english composition classes at a large urban university in the northeastern united states served as participants. each class visited the library for library instruction twice during a given semester. in the experimental group students received information literacy instruction that incorporated two online games, and the control group received the same lesson plan with the exception of a lecture in place of playing games. a six-item preand posttest questionnaire was developed and administered at the outset and conclusion of the two-session classes. the 172 individual tests were coded, graded, and analyzed using spss.   results – a paired sample t-test comparing the control and experimental groups determined that that there was a statistically significant difference between scores on pre-tests and post-tests in the experimental group but not the control group.   conclusion – students who played the online games improved significantly more from pre-test to post-test than students who received a lecture in lieu of playing online games, suggesting that participating in games related to the instruction they received resulted in an improved ability to select appropriate keywords and ascertain citation formats. these findings contribute to the evidence that online games concerning two frequently challenging research practices can be successfully applied to library instruction sessions to improve student comprehension of such skills. introduction   information literacy instruction plays a key role in the educational mission of many academic libraries. librarians employ a wide range of strategies for teaching members of their community regarding the many dimensions of information access and use. one such method of teaching draws upon games-based learning to achieve the fulfillment of learning outcomes and increase student engagement and motivation. in practice, games-based learning frequently consists of librarians either creating their own games, adapting existing games used by other libraries, or designing class sessions using gaming principles (gamification). as opposed to traditional instruction, games may provide students with opportunities to meaningfully engage with classmates and the instructor, participate in hands-on activities, and learn new skills using their preexisting knowledge as a basis.   despite the ongoing popularity of games in library instruction, little research has been done on whether playing games in academic library settings may in fact translate into learning. in the present study, the authors predicted that students in the classes that incorporated games would score higher on the pre-/posttest assessment tool than students in the classes without games. in contrast, the null hypothesis was that there would be no significant differences in scores between the two groups. using two games whose efficacy has been previously tested by their developers, this study seeks to build on this existing evidence and provide insight into the question of whether online games are a preferable method of instruction compared to lectures in terms of student comprehension of targeted concepts.   literature review   a review of the literature reveals that using games for information literacy instruction is increasing in terms of acceptance and popularity, but in many cases assessment beyond student interest has yet to be explored. the scholarly discourse on games as tools to improve literacy began in 2003, when arizona state university professor james paul gee published his seminal monograph on games-based learning titled what video games have to teach us about learning and literacy. gee expounds upon the many ways in which games facilitate learning through his 36 video game learning principles, including critical learning, encouraging exploration and discovery, just-in-time learning, and utilizing active learning methods (2007). regarding information literacy specifically, gumulak and webber (2011) found that the video game-playing activities of 28 teenagers closely corresponded to established information literacy models.   gaming in libraries made a national debut at the 2005 gaming, learning and libraries symposium, where presenters from various library settings discussed how and why games were being used in libraries (doshi, 2006). since the mid-2000s a significant amount of literature has been generated on the subject of games in library instruction. though gaming-related topics such as developing video game collections and providing outreach through gaming events appear with regularity, this review will focus on games-based learning for information literacy instruction. also important to note is that the educational literature contains a great number of studies regarding the use of educational games, and non-library educators have incorporated games into their pedagogy for a far longer time than librarians. however, for the purposes of considering only the most applicable research in terms of setting, class content, and other contextual factors, this literature review focuses on non-digital and digital game initiatives at academic libraries.   non-digital games   non-digital games have been implemented at a number of college and university libraries due to their easy-to-play nature and inherent capacity for personal engagement with others in the class. though the authors selected to use online, digital games for the research at hand, a brief review of the use of non-digital games will help provide additional context on game-based learning in academic libraries. leach and sugarman (2005) note that the success of a library instruction game is dependent upon several factors, including the type of game played, the incorporation of learning outcomes, and the instructor’s flexibility. the authors present best practices for designing games using their activity based on the quiz-show jeopardy! as a case study. similarly, walker (2008) used the jeopardy! format in eight one-shot sessions to reinforce concepts learned earlier in the class, reporting that students responded positively to the game. both articles suggest that the game’s highly familiar format is an important factor in student receptivity. smith (2007) developed games such as tic-tac-toe, word searches, and crossword puzzles that used library-based terms and concepts.   many non-digital games are developed in order to supplement or enhance library orientation sessions. being a type of information literacy instruction that typically occurs in the first semester of a student’s higher education enrolment and focuses on basic research practices, the research into the use of games in academic library orientation sessions provides useful related evidence to consider and build upon. marcus and beck (2003) compared a traditional orientation to one that sent freshmen on a library treasure hunt that required locating a series of clues. by conducting a brief post-orientation test the authors found that the treasure hunt received more positive student feedback than the traditional orientation and held increased educational benefits (p. 31). thorough reviews of the many types of information literacy games, including in-person and virtual games, have been conducted by margino (2013) and smale (2011). smale (2012) developed the internet resource evaluation game quality counts wherein students search for and critically evaluate websites. survey responses indicated that players enjoyed the game and felt that their skills levels increased (p. 140).   digital games   digital and online games to teach college and university students library skills appeared in the literature at an early juncture with koelewyn and corby’s 1982 report on a computer game requiring students to use the reader’s guide to periodical literature. in the arcade-inspired game citation students were randomly assigned one of ten topics and then must construct a bibliography of a predetermined number of sources as quickly as possible using the reader’s guide (p. 171). a great deal has changed technologically since koelewyn and corby’s study, but the reasons for incorporating digital games into instruction remain the same. while at least one academic library has opted to modify an existing commercial videogame to tailor its learning objectives to their needs (clyde & thomas, 2008), the vast majority of libraries using digital games have developed their own. the online board game the information literacy game (rice, 2008) was received positively by students, who played the game by rolling a digital die and correctly answering questions to move ahead on the board. gallegos and allgood (2008) describe a process that began with a board game and led to development of an online game, which ultimately indicated student receptivity to playing information literacy games.   librarians at james madison university created two online games to serve two distinct purposes (mccabe & wise, 2009). citation tic-tac-toe asks players to identify the type of a given citation while playing tic-tac-toe, and magnetic keyword uses virtual refrigerator magnets to help students practise identifying keywords. the authors assessed each game differently, using quantitative methods for citation tic tac toe and qualitative methods for magnetic keyword, finding that in both cases students had increased their skill levels (p. 13). armstrong and georgas (2006) developed and assessed an interactive tutorial titled “doing research” and discerned a statistically significant difference in university student skills using a preand posttest questionnaire. smith and baker (2011) describe the impetus and development of two online games at utah valley university. the authors surveyed 52 students, who responded to the games’ informative and entertaining nature (p. 638).   mary broussard (2010), a prominent researcher in games-based learning, created the online game secret agents in the library as an alternative to a traditional library orientation. groups work in teams to answer a series of questions requiring use of the library’s website and locating materials in the stacks. additionally, broussard (2012) reviewed 17 online library games and analyzed the traits of successful games, offering six suggestions for libraries seeking to develop their own digital games. most recently broussard (2014) makes a case for games as tools for conducting formative assessment in the classroom, arguing that both games and assessment of student learning during a session share significant synchronicities.   the literature demonstrates that librarians have considered it worthwhile to incorporate games for the purposes of library orientations, engagement in one-shot sessions, practising specific library skills, and more. because a wide variety of games exist in terms of format and objectives, generalizing research findings is challenging. a vast majority of researchers measured student receptivity to a particular game instead of whether playing a game contributed to student learning. furthermore, reviewing the literature of games in library instruction presented difficulties in that digital games have a lifespan that can be as brief as one semester. bibliobouts, one of the most promising research-oriented games in terms of gameplay and adaptability by other institutions, is no longer available due to its four-year grant funding reaching its end (university of michigan school of information, 2012), though the bibliobouts team completed a book on designing effective online information literacy games (markey, leeder & rieh, 2014).. gaming expectations and technologies change rapidly, and as such it is difficult to determine which games are being used or are available. after a review of the literature, the authors were prepared to select the games most appropriate to their setting and learning outcomes.   methodology   research design and participants   the study was a quasi-experiment, as the requests for library instruction by teaching faculty at long island university did not permit random assignment of the university’s undergraduate population. the specific design was two groups/nonrandom selection/pretest posttest. pretest/posttest models are commonly employed by educational researchers to investigate effects of a particular treatment on learning (freed, hess & ryan, 2002). eighty-six students enrolled in introductory english composition classes at a large, urban university in the northeast served as participants. the sampling technique employed was convenience, a type of nonprobability sampling frequently used in research involving college students. the participants comprised seven english classes in total. professors of these classes contacted the library of their own accord to request instruction for their students. all seven classes visited the library for group information literacy instruction (ili) classes at two points during the semester. the researchers were the sole ili instructors included in this study.   participant ages ranged from 16 to 40, with an average age of 19. thirty participants identified as male and 56 participants identified as female. participants were divided into two groups prior to instruction: a control group of 43 students and an experimental group of 43 students.   instruments and procedure   before beginning the experiment the researchers needed to secure institutional review board (irb) approval, the researchers were granted an exemption from formal review as this study qualified as “research conducted in established or commonly accepted educational settings, involving normal educational practices” (long island university, n.d.).   the researchers informed their coordinator of instruction that they would like to teach seven sessions of english composition classes, and were thus assigned all classes requested by faculty desiring two ili sessions. three of the sessions were taught in fall 2014 and four were taught in spring 2014. the seven classes were divided into two groups prior to the instruction: the control group and the experimental group. one researcher taught four classes in the experimental condition and the other researcher taught three classes in the control condition. there was a total of 43 students in the experimental classes and 43 students in the control classes. each researcher selected the classes which fit best into his or her schedule. students in the control group would not play any educational games, while students in the experimental group played a keyword development game in the first ili session and a citing game in the second ili session. the sessions were all one hour and fifteen minutes long and there was an average of three weeks between the first and second sessions.   lesson plans were created for first and second sessions of both the control and the experimental classes. the lesson plans were identical with the exception that students in the experimental condition played a game (see appendix a for a detailed lesson plan). apart from the games, the researchers collaboratively developed all classroom materials utilized in this study. at the very beginning of the first session each student was administered a six-question multiple-choice-paper pretest developed collaboratively by the two researchers and adapted from beile’s test of information literacy for education (beile o’neil, 2005). students were given five minutes to complete the quiz, and all participants finished on time. this instrument assessed their knowledge of basic keyword development and citing skills (see appendix b).   both groups of students were then given a presentation on basic keyword development and database strategy skills. afterwards the experimental groups were asked to play a freely available game called doing research, created by librarians at the university of illinois at chicago and available at: www.uic.edu/depts/lib/reference/services/tutorials/doingresearch.shtml (armstrong & georgas, 2006). players are presented with a topic, the representation of women in film, and asked to choose certain keywords that represent the topic before moving forward. in the next step several synonyms for the terms “women” and “film” must be selected. students were allowed fifteen minutes to play the game. both sessions concluded with the distribution to students of an activity in which they explored a research paper topic and located one article in an academic database.   for the second session, both classes began with a presentation on citing in both mla and apa formats. librarians then gave students a demonstration of proquest databases. the experimental group subsequently played a game created by james madison university librarians called citation tic-tac-toe, available at: www.lib.jmu.edu/tictactoe/ (mccabe & wise, 2009). citation tic-tac-toe asks players to correctly identify a format when presented with a citation, such as articles, book chapters, and website domains. students were given ten minutes to play the game. next, both groups were provided with a worksheet that entailed locating an article on their research paper topic and the documentation of this article in apa and mla styles. before the second session ended students were given a posttest, which presented them with the same questions as the pretest ordered differently to discourage memorization. therefore, the independent variable in this project was the online games, while the dependent variable was the measures of achievement on the assessment tool.   once all of the classes were taught the pretests and posttests were graded by the researchers. the standard 100 percentile grading method was employed, with each of the six questions representing 17 percentage points (rounded up from 16.66). if students skipped a question the item was automatically counted as incorrect.   data collection and analysis   all pretests and posttests were coded using a simple numerical coding system. although all of the tests were anonymous this system was used to keep track of the artifacts. participants in the experimental group received a number ranging from 1-43 and participants in the control group received a number ranging from 44-86. the pretests and posttests were then coded accordingly. statistical analysis was used to determine if there was any significant difference between scores on the pretests and posttests in both groups. a one-tailed paired (dependent) t-test was chosen to analyze the data. descriptive statistics were also generated to ascertain group means and standard deviations. these statistics provide average scores on the pretests and posttests in the experimental and control groups. individual preand posttest scores were not compared, as the researchers focused on assessment at the class (group) level.   results   a paired sample t-test comparing the control and experimental groups determined that that there was a statistically significant difference between scores on pretests and posttests in the experimental (games) condition: t(42)=-3.056, p = 0.002. there was not a significant difference between scores on pretests and posttests in the control (no games) condition: t(42)=-.506, p = 0.308. table 1 provides the full statistical breakdown of the t-test’s output.   additionally, descriptive statistics for the scores on the pretests and posttests in both the experimental and control groups were calculated (see table 2).   although both conditions saw students improve their scores over time, the experimental group experienced a much larger improvement, as scores improved by around two percentage points in the control condition and around ten points in the experimental condition. the standard deviations were very similar, with the greatest deviation occurring in the pretest experimental condition and the lowest deviation occurring in the posttest control condition.     table 1 output for paired samples t-test pair condition mean std. dev. std. error mean t df sig. (1-tailed) pair 1 pre no games post no games -2.326 30.138 4.596 -.506 42 .308 pair 2 pre games post games -10.488 22.508 3.432 -3.056 42 .002     table 2 means for pretest and posttest scores in games and no games conditions pair condition mean n std. deviation std. error mean pair 1 pretest no games 60.30 43 24.657 3.76 pair 1 posttest no games 62.63 43 23.815 3.632 pair 2 pretest games 58.33 43 25.735 3.924 pair 2 posttest games 68.81 43 25.057 3.821     discussion   statistical analysis revealed that the null hypothesis, which proposed there would not be a significant difference between test scores in the experimental and control groups, can confidently be overturned. the alternative hypothesis, which predicted that students taught with games would outperform students in a control group on a library skills test, was confirmed. these findings suggest that the trend within academic librarianship of incorporating games into instruction has not been in vain; rather, the present study offers educators evidence that games may have the potential to positively impact information literacy skill development.   currently, there is very little research within lis literature employing a two group pretest/posttest design to assess the effectiveness of games. mccabe and wise (2009) are an exception, as they piloted their game citation-tic-tac-toe with both a control and experimental group. similar to the present study, mccabe and wise learned that students who played the game performed better on a posttest than students in a control group who took an online citation tutorial instead. when combined with the findings of the present study there now exists increasing evidence that games can enhance the development of information literacy skills, most demonstrably of citing.   two additional empirical articles mentioned in the literature review support the findings that games can increase information literacy knowledge. armstrong and georgas (2006), creators of the doing research tutorial used in the present study, found that students scored significantly higher on a posttest following participation in this game than on a pretest. although the lack of a control group prevented valuable comparative opportunities, the experiences of the students in armstrong and georgas’s project fared similarly to the students in the present study’s experimental group. both initiatives demonstrated the ability of interactive computer activities to boost scores on information literacy tests.   marcus and beck (2003) conducted an innovative study which compared the learning outcomes and attitudes of first year students in two different ili groups: a self-guided treasure hunt orientation or a traditional library tour. the treasure hunt can certainly be considered an educational game, as students adventured around the facility completing interactive library-related tasks and were awarded prizes. all students were given a library skills multiple-choice quiz following the treasure hunt, and statistical analysis showed that students in the treasure hunt (experimental) group performed better than students in the traditional tour (control) group.   what all of these studies share in common is empirical evidence that games can play a part in helping students sharpen their il skills. the positive statistical results support greater inclusion of games into active learning pedagogies within the academic library classroom, as well as potentially increasing the allocation of additional time and money for the development of educational games.   limitations and future directions   despite the concerted effort of the researchers to control variables in the quasi-experiment there are several limitations deserving of attention, including: researcher assignment to classes; students receiving insufficient time to complete the questionnaires; and the potential for student skills gained independent of library instruction between classes. first, instead of assigning one researcher to teach all of the games classes and the other researcher all of the control classes, a future study would entail both researchers teaching both types of classes. this measure would maximize the potential of the treatment (games) to affect learning and to minimize possible confounding influences of individualized instruction techniques of the two researchers.   another limitation of this project is the potential for participants to have experienced procedural bias. in brief, this bias occurs when participants are given an instrument to complete in a set time limit under close supervision of the researcher(s). in this study students were administered the pretests and posttests with the knowledge that they had five minutes to fill out each questionnaire. some participants could have felt pressured and rushed through the questions, making mistakes that might have been prevented by allowing them additional time. a small body of psychological research spanning nearly fifty years indicates the negative impact that timed tests can have on some individuals. morris and liebert (1969) empirically demonstrated that college students who showed high levels of worry on a questionnaire performed worse on an intelligence test than both high-worry students in an untimed condition and low-worry students. many years later onwuegbuzie and daley (1996) conducted a study which measured the performance of graduate students on a statistics examination in both timed and untimed conditions. analysis revealed that on average students in the untimed conditions received higher scores. another study focused on a community college population, noting that untimed tests can be particularly beneficial to older and nontraditional students (hodges & kennedy, 2004).   a third limit worth noting is that the passage of time in between completion of the pretests and posttests in both groups could have caused an extraneous time-related variable. students in the first session did not return to the library for at least two weeks subsequent to the second session; therefore, during this time they ostensibly could have gained some information literacy skills outside of the classes taught by the researchers. for example, a student could have visited the reference desk for keyword development or citation help, or consulted with a librarian for a one-on-one tutorial. therefore, it is a possibility that some students scored higher on the posttests than the pretests not because of the incorporation of games into instruction (i.e. the treatment), but because they improved their research skills in other ways during the period between the two sessions.   future research could adopt a methodology similar to the study at hand by examining the educational impact of games-based teaching interventions using preand posttests, but might do so using a longitudinal analysis conducted over the course of multiple academic years or with the addition of a qualitative measure to expand upon the dimensions of the evidence being presented. additionally, the wide variety of game formats and their different educational capacities should be considered, including medium (in-person, digital, and hybrid) and duration (from part of a standalone instruction session to integration throughout a semester-long course). evaluating the effects of information literacy gameplay when practised individually versus in small groups would be another beneficial avenue for research and would contribute much needed research to the area of games and learning in the context of library instruction.   conclusion   the results of this study suggest that, when implemented in information literacy instruction sessions, brief online games addressing two common research processes—identifying keywords and synonyms in addition to categorizing citation types—can be successfully utilized to improve student comprehension of these skills. the instruction containing games was compared with instruction with additional lecture, the latter being a type of teaching that can be considered “traditional” information literacy instruction. these games represent a modest change to the content addressed in the instructors’ ili sessions, and as such might easily be adopted by other librarians interested in using participatory, game-driven methods to encourage engagement with information literacy practices. the effective use of games will vary according to student backgrounds, desired learning outcomes, and other classroom factors, but in the appropriate circumstances games-based learning may have the potential to enhance student engagement and learning in regards to instructional content.   an additional advantage to games-based learning, noted by several researchers but outside of this study’s scope, is the role of gameplay in affective elements that contribute to learning, such as student enjoyment of the session and intrinsic motivation. the authors have found anecdotally in their experiences as instructors that the elements of engagement and motivation can be greatly improved when games are a part of student learning experiences. it is the authors’ hope that this research adds to the evidence base concerning the efficacy of games in the library classroom, and will encourage additional research and reflection on games-based learning and other popular teaching methods to ensure that our practices as information literacy instructors are grounded in effective pedagogy, and in turn, instruction that places learners first and foremost.   references   armstrong, a., & georgas, h. (2006). using interactive technology to teach information literacy concepts to undergraduate students. reference services review 34(4), 491-497. retrieved from http://www.emeraldinsight.com/loi/rsr   beile o’neil, p. (2005). development and validation of the beile test of information literacy for education (doctoral dissertation). university of central florida, orlando, fl.     broussard, m. j. (2010). secret agents in the library: integrating virtual and physical games in a small academic library. college & undergraduate libraries 17(1), 20-30. http://dx.doi.org/10.1080/10691310903584759   broussard, m. j. (2012). digital games in academic libraries: a review of games and suggested best practices. reference services review 40(1), 75-89. retrieved from http://www.emeraldinsight.com/loi/rsr   broussard, m. j. (2014). using games to make formative assessment fun in an academic library. journal of academic librarianship 40(1), 35-42. http://dx.doi.org/10.1016/j.acalib.2012.12.001   clyde, j., & thomas, c. (2008). building an information literacy first-person shooter. reference services review 36(4), 366-380. retrieved from http://www.emeraldinsight.com/loi/rsr   doshi, a. (2006). how gaming could improve information literacy. computers in libraries 26(5), 14-17. retrieved from http://www.infotoday.com/cilmag/default.shtml   freed, m., hess, r. k., & ryan, j. m. (2002). the educator’s desk reference: a sourcebook of educational information and research. (2nd ed.) lanham, md: rowman & littlefield publishers.   gallegos, b., & allgood, t. (2008). the fletcher library game project. in a. harris & s. e. rice (eds.), gaming in academic libraries: collections, marketing and information literacy (pp. 149-163). chicago il: association of college and research libraries.   gee, j. p. (2007). what video games have to teach us about learning and literacy. new york: palgrave macmillan.   gumulak, s., & webber, s. (2011). playing video games: learning and information literacy. aslib proceedings 63(2/3), 241-255. retrieved from http://www.emeraldinsight.com/loi/ap   hodges, d. z., & kennedy, n. h. (2004). editor's choice: post-testing in developmental education: a success story. community college review, 32(3), 35-42. http://dx.doi.org/10.1177/009155210403200303   long island university. (n.d.). institutional review board application for exempt category review. retrieved february 15, 2015 from http://www.liu.edu/~/media/files/academicaffairs/sponresearch/forms/uc_humansubjects-exempt-0413.ashx    koelewyn, a. c., & corby, k. (1982). citation: a library instruction computer game. rq 22(2), 171-174.   leach, g. j., & sugarman, t. s. (2005). play to win! using games in library instruction to enhance student learning. research strategies 20(3), 191-203. http://dx.doi.org/10.1016/j.resstr.2006.05.002   marcus, s., & beck, s. (2003). a library adventure: comparing a treasure hunt with a traditional freshman orientation tour. college & research libraries 64(1), 23-44. http://dx.doi.org/10.5860/crl.64.1.23   margino, m. (2013). revitalizing traditional information literacy instruction: exploring games in academic libraries. public services quarterly 9(4), 333-341. http://dx.doi.org/10.1080/15228959.2013.842417   markey, k., leeder, c, & rieh, s. y. (2014). designing online information literacy games that students want to play. lanham, md: rowman & littlefield.   martin, j., & ewing, r. (2008). power up! using digital gaming techniques to enhance library instruction. internet reference services quarterly 13(2-3), 209-225.   mccabe, j., & wise, s. (2009). it’s all fun and games until someone learns something: assessing the learning outcomes of two educational games. evidence based library and information practice, 4(4), 6-23. retrieved from http://ejournals.library.ualberta.ca/index.php/eblip/index   morris l, & liebert, r. (1969). effects of anxiety on timed and untimed intelligence tests: another look. journal of consulting and clinical psychology, 33(2), 240-244. http://dx.doi.org/10.1037/h0027164   onwuegbuzie, a. j., & daley, c. e. (1996). the relative contributions of examination-taking coping strategies and study coping strategies to test anxiety: a concurrent analysis. cognitive therapy and research, 20(3), 287-303. retrieved from http://link.springer.com/journal/10608   rice, s. e. (2008). education on a shoestring: creating an online information literacy game. in a. harris & s. e. rice (eds.), gaming in academic libraries: collections, marketing and information literacy (pp. 175-188). chicago il: association of college and research libraries.   smale, m. a. (2011). learning through quests and contests: games in information literacy instruction. journal of library innovation 2(2), 36-55. retrieved from http://www.libraryinnovation.org/index   smale, m. a. (2012). get in the game: developing an information literacy classroom game. journal of library innovation 3(1), 126-147. retrieved from http://www.libraryinnovation.org/index   smith, a.-l., & baker, l. (2011). getting a clue: creating student detectives and dragon slayers in your library. reference services review 39(4), 628-642. retrieved from http://www.emeraldinsight.com/loi/rsr   smith, f. a. (2007). games for teaching information literacy skills. library philosophy & practice 9(2), 1-12. retrieved from http://digitalcommons.unl.edu/libphilprac/   university of michigan school of information. (2012). about the bibliobouts project. bibliobouts project. retrieved february 15, 2015 from http://bibliobouts.si.umich.edu/biblioboutsabout.html.   walker, b. e. (2008). this is jeopardy! an exciting approach to learning in library instruction. reference services review 36(4), 381-388. retrieved from http://www.emeraldinsight.com/loi/rsr     appendix a lesson plans   lesson plans for session #1   experimental group   1.       introduction and overview of class content (5 minutes) 2.       students take pretest (5 minutes) 3.       prezi presentation on keyword development and topic formulation (10 minutes) 4.       students play keyword game (15 minutes) 5.       demonstrate gale virtual reference library and points of view reference center (15 minutes) 6.       students complete keyword worksheet activity (25 minutes)   control group   1.       introduction and overview of class content (5 minutes) 2.       students take pretest (5 minutes) 3.       prezi presentation on keyword development and topic formulation (10 minutes) 4.       brief lecture on keyword selection (15 minutes) 5.       demonstrate gale virtual reference library and points of view reference center (15 minutes) 6.       students complete keyword worksheet activity (25 minutes)   lesson plans for session #2   experimental group   1.       introduction and overview of class content (5 minutes) 2.       prezi presentation on citing in apa and mla formats (15 minutes) 3.       students play citation tic-tac-toe (10 minutes) 4.       demonstrate proquest databases (10 minutes) 5.       students complete citation and database searching worksheet activity (25 minutes) 6.       students take posttest (5 minutes) 7.       concluding remarks (5 minutes)   control group   1.       introduction and overview of class content (5 minutes) 2.       prezi presentation on citing in apa and mla formats (15 minutes) 3.       brief lecture on citation styles (10 minutes) 4.       demonstrate proquest databases (10 minutes) 5.       students complete citation and database searching activity (25 minutes) 6.       students take posttest (5 minutes) 7.       concluding remarks (5 minutes)     appendix b assessment quiz   1. using the citation below, what does the item in bold text represent?   szajnberg, n. (2012). zombies, vampires, werewolves: an adolescent's developmental system for the undead and their ambivalent dependence on the living, and technical implications. psychoanalytic review, 99(6), 897-910. doi:10.1521/prev.2012.99.6.897   a. article title b. volume c. author d. journal title   2. you have a class assignment to investigate americans’ attitudes towards the iraq war. a keyword search in the library catalog on “iraq war” returns over 700 items. which of the following steps would give you the best search results?   a. change search to “what are some of the most popular american attitudes on the iraq war?” b. add “american attitudes” to your search c. search by author using the same keywords d. search by title using the same keywords   3. which is the article title in the following mla citation? bray, kate. “a week in the life of jay-z.” the independent [london] 25 sept. 2009: 20. proquest databases. web. 10 sept. 2013.   a. the independent b. proquest databases c. there is no title provided d. a week in the life of jay-z   4. select the keywords that best represent synonyms for the concept “college students.”   a. colleges, universities, community colleges b. millennials, generation y, generation x c. graduate students, freshmen, sophomores d. midterms, finals, break   5. the following citation is for: orians, gordon, and gene christman. a comparative study of the behavior of red-winged, tricolored, and yellow-headed blackbirds. berkeley: university of california press, 1968. print.     a. a book b. a chapter in a book c. a journal article d. none of the above   6. select the set of keywords that would provide the best search results for the following question: what incentives do people have to use facebook or other social media?   a. facebook, twitter, instagram b. facebook, social media, motivation c. facebook, psychology, friends d. incentives, choices, motives   evidence based library and information practice editorial responsibilities      2014. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.       editor-in-chief: alison brettle   associate editor (evidence summaries): heather pretty   associate editor (articles): lorie kloda   associate editor (reviews): denise koufogiannakis   associate editor (classics): jon eldredge   associate editor (commentaries, using evidence in practice, news): alison brettle   production editor: michelle dunaway   editorial intern: archana deshmukh   copyeditors: richard hayman (lead copyeditor), georgianne bordner, molly des jardin, linda ferguson, richard hayman, stacey penney, maria tan, mary virginia taylor, alison yeoman   indexing support: pam morgan   evidence summary   five-month print and electronic patron-driven acquisitions trial at a large university shows circulation advantages   a review of: tynan, m. & mccarney, e. (2014). “click here to order this book”: a case study of print and electronic patron-driven acquisition in university college dublin. new review of academic librarianship, 20(2), 233-250. http://dx.doi.org/10.1080/13614533.2014.906352   reviewed by: laura costello head of library materials & acquisitions gottesman libraries teachers college, columbia university new york, new york, united states of america email: lac2184@columbia.edu   received: 31 may 2015  accepted: 11 aug. 2015      2015 costello. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to evaluate the effectiveness of the first patron-driven acquisitions program in the republic of ireland and determine the effects of this acquisitions strategy on circulation, budget, and collection development.   design – case study.   setting – a large university on two campuses in the republic of ireland with a total of over 25,000 students.   subjects – patron-driven acquisitions including 1,128 electronic monographs and 1,044 print monographs.   methods – the authors evaluated titles purchased during a five-month patron-driven acquisitions trial conducted in 2013. patron-selected titles were compared to traditionally acquired (faculty and librarian-selected) titles acquired during the same time period based on subject area and circulation data. results from the trial were also compared to a literature review of patron-driven acquisitions trials conducted at other institutions. information on selectors was examined for patron-driven print acquisitions.   main results – the most frequently acquired subject areas included business, politics, english, drama and film, medicine, psychology, history, and law. these frequently acquired subject areas were consistent across print and electronic patron-driven acquisitions, traditionally acquired titles at the institution, and data from the patron-driven acquisitions trials of other institutions. patron-selected titles in art history and architecture subjects showed a significant print preference over electronic. patron-selected electronic titles were used 8.45 times compared to 3.27 uses for traditionally selected electronic titles. patron-selected print titles circulated 1.32 times compared to 1.04 circulations for faculty-selected titles and 0.63 circulations for librarian-selected titles. for patron-driven print acquisitions, 63% of selectors were students and 37% were faculty and staff.   conclusion – the trial was considered successful in circulation and subject area diversity. subject breakdown for patron-selected titles was consistent with expectations and mirrored traditional acquisitions strategies and expected demand. patron-selected titles showed a circulation advantage over traditionally selected titles, though this advantage was more significant for electronic titles. the library intends to continue with patron-driven acquisitions. considerations for future trials, including higher quality and more selective discovery records for print titles, more informative marketing, and better timing, could improve results.   commentary   this trial examined both print and electronic patron-driven acquisitions and found a circulation advantage to this acquisitions strategy in both formats. these results were more modest than those found by other patron-driven acquisitions case studies, including those at kent state (downey, zhang, urbano, & klinger, 2014) and the university of tennessee at chattanooga (dunn & murgai, 2014). higher demand for humanities and social sciences titles is consistent across all three studies.   this study addressed both print and digital patron-driven acquisitions, and the authors observed “collection development" behaviours among faculty and academic staff. these selectors were more likely to order many items in a single session and were less likely to check out their items when they arrived. this behaviour and the authors’ observations that their new program was poorly understood by students and faculty members potentially calls into question that the population is representative of all eligible users, as outlined by glynn’s (2006) critical appraisal tool. despite this, the observations and lessons included in the study will be informative to practitioners hoping to implement patron-driven acquisitions programs in their own libraries.   the electronic book discovery records included in the patron-driven program featured a unique trigger process that may have influenced results. the custom in patron-driven research is to exclude trigger views from circulation data, but the protocol in the study required electronic books to be clicked at least twice within a 24-hour period to trigger purchases. the exclusion of these uses, which patrons could have found satisfactory for their research needs, could potentially explain why the circulation results of this study are more modest than those of similar studies that feature less demanding trigger protocols.   despite these challenges, the authors found clear advantages to patron-driven acquisitions while working through the complexities of implementing a new acquisitions strategy. the authors describe their experiences with a nationally mandated and lengthy tender process which required vendors to compete for bids after meeting set criteria for inclusion. the study represents a holistic examination of the beginning stages of a patron-driven program and further and longer trials will provide evidence of the sustained benefits or long-term implications of this model. practitioners in the early stages of planning a patron-driven strategy, particularly in public university libraries with complex tender processes for new vendor agreements, will find this study a useful guide.     references   downey, k., zhang, y., urbano, c., & klinger, t. (2014). a comparative study of print book and dda e-book acquisition and use. technical services quarterly, 31(2), 139-160. http://dx.doi.org/10.1080/07317131.2014.875379   dunn, m. j. & murgai, s. (2014). buy not borrow: building a bridge to a patron-centric collection – a case study at the university of tennessee at chattanooga’s lupton library. the southeastern librarian, 62(1), 10-18.   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154   conference paper   building scorecards in academic research libraries: performance measurement and organizational issues   vivian lewis acting university librarian mcmaster university hamilton, ontario, canada email: lewisvm@mcmaster.ca   steve hiller director of assessment and planning university of washington libraries seattle, washington, united states of america email: hiller@u.washington.edu   elizabeth mengel associate director, scholarly resources and special collections the sheridan libraries johns hopkins university baltimore, maryland, united states of america email: emengel@jhu.edu   donna tolson director of strategic assessment services university of virginia library charlottesville, virginia, united states of america email: djt5k@virginia.edu      2013 lewis, hiller, mengel, and tolson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – this paper describes the experiences of four prominent north american research libraries as they implemented balanced scorecards as part of a one-year initiative facilitated by the association of research libraries (arl). the balanced scorecard is a widely accepted organizational performance model that ties strategy to performance in four areas: finance, learning and growth, customers, and internal processes.   methods – four universities participated in the initiative: johns hopkins university, mcmaster university, the university of virginia, and the university of washington. each university sent a small group of librarians to develop their scorecard initiatives and identified a lead member. the four teams met with a consultant and the arl lead twice for face-to-face training in using the scorecard. participants came together during monthly phone calls to review progress and discuss next steps. additional face-to-face meetings were held throughout the year in conjunction with major library conferences.   results – the process of developing the scorecards included the following steps: defining a purpose statement, identifying strategic objectives, creating a strategy map, identifying measures, selecting appropriate measures, and setting targets. many commonalities were evident in the four libraries’ slates of strategic objectives. there were also many commonalities among measures, although the number chosen by each institution varied significantly, from 26 to 48.   conclusion – the yearlong arl initiative met its initial objectives. the four local implementations are still a work in progress, but the leads are fully trained and infrastructure is in place. data is being collected, and the leadership teams are starting to see their first deliverables from the process. the high level of commonality between measures proposed at the four sites suggests that a standardized slate of measures is viable.   introduction   a strategy without measures is just a wish and measures that are not aligned with strategy are a waste of time. (matthews, 2008)   the balanced scorecard is a widely accepted organizational performance model that ties strategy to performance in four critical areas: finance, learning and growth, customers, and internal processes. while originally designed for the for-profit sector, the scorecard has been adopted by non-profit and government organizations, including some libraries. this paper focuses on the experiences of four prominent north american research libraries (johns hopkins university, mcmaster university, the university of virginia and the university of washington) as they developed and implemented scorecards as part of a one-year initiative facilitated by the association of research libraries (arl).   this paper is divided into four major sections: an introduction to the balanced scorecard and its key components; an overview of the arl initiative and the process used to develop scorecards at each library; an exploration of the concept of a standardized suite of measures for arl libraries based on a commonality of key objectives; and a review of organizational challenges faced by the pilot sites during their implementations. the authors hope that the lessons learned and strategies employed at their institutions will assist other academic libraries choosing to implement the balanced scorecard.   what is the balanced scorecard?   the balanced scorecard was developed by harvard business school professors robert s. kaplan and david p. norton in the early 1990’s as a reaction to the industrial age emphasis on financial measures as the sole indicator of success. in their groundbreaking book, the balanced scorecard: translating strategy into action, kaplan and norton argue that the economic realities of the information age require a more well-rounded set of measures to evaluate and drive an organization’s performance:   “the balanced scorecard is a new framework for integrating measures derived from strategy. while retaining financial measures of past performance, the balanced scorecard introduces the drivers of future financial performance. the drivers, encompassing customer, internal business process, and learning and growth perspectives, are derived from an explicit and rigorous translation of the organization’s strategy into tangible objectives and measures.” (kaplan & norton, 1996, p. 18)   the balanced scorecard model is premised on strong and very direct linkages between key planning elements. each measure is directly aligned to one or more strategic objectives. success in meeting targets is a clear indication that the organization is moving its mission forward. linkages between measures (both within and across the four perspectives) help ensure that the organization maintains a truly “balanced” approach. in the same way, strategic initiatives are directly linked to the measures: only projects that improve an organization’s success in meeting its targets are eligible for linkage to the scorecard (kaplan and norton, 1992).   who uses the balanced scorecard?   while originally designed for the commercial sector, non-profit organizations have also been attracted to the model. kaplan and norton (1996) note that, “while the initial focus and application of the balanced scorecard has been in the for-profit (private) sector, the opportunity for the scorecard to improve the management of governmental and not-for-profit enterprises is, if anything, even greater” (p. 179). the balanced scorecard was first recommended for adoption by u.s. federal government procurement agencies during the clinton administration. the city of charlotte, north carolina, and the united way of southeastern new england were also early adopters (kaplan & norton, 2001).   as the concept has matured, the pool of non-profit organizations exploring the use of the balanced scorecard has grown along with specialized expertise in the use of the model in specific settings. ascendant strategy management group, the consulting firm used for the pilot, has helped government and non-profit organizations like the u.s. federal bureau of investigation, the u.s. securities and exchange commission, the atlanta public schools, and the catholic charities archdiocese of boston apply the scoreboard to achieve change. many of these organizations have had to quickly adapt to game-changing events such as 9/11 or the recent mortgage meltdown, and have turned to the balanced scorecard to promote successful organizational change.   although the total number of libraries adopting the balanced scorecard is unknown, it is likely still only a few handfuls worldwide. in scorecards for results: a guide for developing a library balanced scorecard (matthews, 2008), examples of libraries with experience using the balanced scorecard include the singapore public library and the university of virginia library.   aside from virginia, which developed their scorecard in 2001 (self, 2003), only a small number of academic libraries are known to have adopted this approach.   libraries, measures, and the balanced scorecard   while relatively few have adopted the balanced scorecard, libraries have a long tradition of collecting statistical and other measures related to organizational performance. for the most part, libraries collect input measures, the amounts of resources invested or put into the development and delivery of collections and services. input measures traditionally deal with such categories as collections, facilities, staffing, budget, and more recently, technology. they count things such as the number of volumes, user seats, librarians, dollars, or computers. they form the basis of many of the regional or national statistical surveys where comparisons between libraries can be made. for example, the association of research libraries membership index tracked the variables of number of volumes held, the number of volumes added during the year, number of current serials received, total operating expenditures, and total number of professional and support staff.   while input measures track the investment in library collections and services over time, they do not indicate if these resources and services are actually being used or how effective they were in meeting user needs. the use factor can be handled with output measures that count uses or transactions associated with library activities. these might include number of items loaned, number of reference transactions, instruction sessions, gate counts, computer log-ins, and web site visits. output measures are often used as surrogates for library effectiveness (i.e., an effective library is one that is heavily used). while these metrics do incorporate the user, they do not actually measure the impact specific services or resources had on that user. they also are not necessarily tied to any strategy or set of objectives.   process measures, also used extensively in libraries, measure the activities related to turning inputs into outputs. sometimes they are called efficiency measures as they calculate the amount of time per activity or the cost of that activity (for example, the average length of time to catalog a book or the cost of staffing a service point). process measures can also have a customer component such as the average time it takes to order a book or answer a question.   finally, outcome measures represent the effect or impact of a particular service or resource on the customer or what that service or resource enables the customer to do. successful outcome measures are usually linked to objectives and goals, which may not be solely defined by the library. for example, if there is a learning objective for students to cite information correctly in their term papers, an outcome measure might be that 95% of citations are accurate. durgan, hernon, and nitecki note that another goal of performance measurement is, “how well does the library serve the institutional mission and serve as an effective partner or collaborator?” (2009, p.38).   brophy (2008) considers measuring library performance to have two basic goals: “how good is this library? how much good does this library do?” (p. 7). that said, performance measures in themselves are not sufficient to achieve these goals if they are not tied directly to overall organizational strategy and objectives. while an increasing number of libraries are developing and using measures that tie directly to achievement of strategic objectives (franklin (2011) estimates at least 10% of arl libraries are including metrics in their strategic plans), they are usually applied to specific areas and are neither balanced nor integrated. the balanced scorecard provides an opportunity, not only to integrate these library performance measures within a more structured planning process, but one that also connects to synergistic organizational performance.   at the international level, the potential impact of the balanced scorecard as an organizational performance model for libraries can be seen in poll’s and boekhorst’s second revised edition of measuring quality: performance measurement in libraries. the authors’ selection of 40 indicators is based on 4 criteria, 1 of which is: “to cover the different aspects of the service quality as described in the balanced scorecard, including indicators for the aspect of development and potentials” (poll & boekhorst, 2007, p. 9). poll and boekhorst use the term “indicators” rather than performance measures and, citing iso 11620, note that good indicators are informative, reliable, valid, appropriate, practical, and comparable.    the association of research libraries initiative   the association of research libraries (arl) advances the interests of the major research libraries in the united states and canada. it has established a strong and multifaceted assessment program to enhance understanding of current and future trends in academic libraries and to assist member institutions in meeting their strategic objectives. arl places a strong focus on evidence-based decision making, and creating a culture of assessment. it has facilitated the introduction and use of many tools for building this capacity, including libqual+® and mines for libraries®.   with the difficult economic climate and increased requirements for accountability throughout the higher education sector, the need to enhance member libraries’ capacity for driving change has become even more crucial. arl decided to explore the balanced scorecard as a key tool for measuring performance and leading change within member institutions. as noted in kyrillidou (2010), arl intended to accomplish two tasks:  “to assist, train and facilitate the use of the scorecard in a small number of arl libraries; and to test the value of a collaborative model for learning about and implementing the new tool” (p. 33).   in late 2008, arl put out a call to its members for expressions of interest in participating in a one-year exploration of the balanced scorecard. the initiative was described as “an investment in helping libraries make a stronger case for the value they deliver by developing metrics that are tied to strategy” (association for research libraries, 2009).   the initial november 2008 meeting ultimately produced four universities keen to participate: johns hopkins university, mcmaster university, the university of virginia and the university of washington. the four institutions brought a wide spectrum of experiences to the table. the university of virginia library had used the balanced scorecard for a number of years, but was interested in refreshing their implementation and providing assistance to the new sites. the university of washington had a strong assessment program, but no experience with the scorecard. johns hopkins and mcmaster had developing assessment programs and no past experience with the scorecard.   each university sent a small group of librarians to develop their scorecard initiatives, and identified a lead member. the four teams met with the consultant and arl lead twice for face-to-face training in using the scorecard. participants came together during monthly phone calls to review progress and discuss next steps. additional face-to-face meetings were held throughout the year in conjunction with major library conferences.    overview of the balanced scorecard process   as with many other prominent performance management models, the balanced scorecard process appears relatively straightforward. participants are directed to:   identify the organization’s strategic objectives. categorize these objectives into four perspectives (financial, customer, internal process, learning and growth); render these objectives to a “strategy map,” a one-page representation of the organization’s strategic objectives; construct metric(s) to measure progress on each objective; set ambitious but reachable targets for each metric; identify strategic initiatives to improve the chance of meeting targets; communicate scorecard results regularly – both to staff and stakeholders; review and adjust the full complement of objectives, measures, targets and initiatives on a regular basis. easily stated but, as each library discovered, the balanced scorecard is not a simple or quick undertaking. the process demands a significant investment of time and intellectual labor. to be successful, the model also requires strong commitment from executive leadership and mid-level managers to champion the process to staff, customers and other stakeholders. and the impact on the organization can be equally significant. the scorecard forces an organization to have new – sometimes challenging – conversations, and to analyze aspects of its current and future state that may have otherwise gone unexamined. ultimately, the scorecard may substantially shift an organization’s strategic direction or dramatically change how its human capital and other resources are allocated. the scorecard is, by its very nature, a change driver. and the change is relentless. the model commits the organization to continuous and regular reflection and to communicate the results of those reflections with a new level of discipline and precision.   getting started defining a purpose statement   once committed to the process, the four libraries began immersing their teams in the language and key concepts associated with the balanced scorecard. arl brought the participants together and facilitated the conversations. the consultant provided the training, homework, and content for learning the process.   the planning teams began by creating “purpose statements.” a purpose statement defines the extent of an organization’s business in one single statement. it articulates why an organization exists, the scope of its work, and the advantage it brings. the statement differentiates one organization from its peers and helps to put a fence around the more lofty and grander vision and mission statements. these purpose statements were not created for public consumption but, for some sites, proved to be useful internal tools when working on strategy.   identifying strategic objectives   prior to entering the pilot, all four libraries had strategic plans with defined mission, vision, and value statements. all four sites had concerns about the value of these plans to drive their organizations forward. all sites were maintaining formal lists of goals or objectives, but recognized that the links between these goals and their overall mission were sometimes fairly tenuous. each site engaged in ambitious slates of projects, but the alignment between these projects and the organization’s overall mission, goals and objectives was often weak. in addition, the teams discovered that their current slates of objectives were focused more on what happened last year than on what they needed to do in the coming years to achieve their missions.   the scorecard process forced the four teams to re-examine their current slates of objectives in light of a new “balanced” four-perspective framework. did their objectives adequately address the four perspectives or did they put too much emphasis on one or two?  did the objectives drive change or just describe and justify the current landscape? did the objectives sync with the priorities of the larger university? what story did the current strategies tell and what story did they want them to express? how can an organization tell if it is achieving its mission when the concepts are so intangible?   unlike in for-profit organizations, the teams discovered that their current slates of objectives tended to focus primarily on the customer (the users) and internal processes (administrative efficiencies) – with relatively little attention being paid to the staff learning and growth and financial health perspectives. interestingly, the changes that were happening in the overall economy in 2008 and 2009, forced a new and sharper focus on financial issues.    in some cases, existing objectives were mapped into the framework, while in other cases, new directions were required. the groups were encouraged to aim for a maximum of 15 objectives (preferably 2 or 3 per perspective), each framed using an active verb. the consultants strongly discouraged the teams from mistaking “projects” for objectives. given that libraries do not like to stop doing anything and consistently strive to be all things to all people, narrowing down the past goals and initiatives into this smaller, more defined subset caused some angst at all sites.   creating a “strategy map”   participants were encouraged to render their slates of objectives into a “strategy map.” the map is a one-page visual representation of an organization’s strategic objectives. the maps were expected to very clearly show the balance and interrelationship between the four perspectives. if done well, a staff member should be able to recognize their organization’s map because it accurately reflects what that organization is all about. leaders are known to carry their strategy maps with them at all times – to help tell their organization’s story to others.   some organizations with well-developed scorecards and access to graphic artists have devised very clever renditions of their strategy maps. (a search of google for “strategy maps” turns up very interesting results.) even very basic strategy maps can be extremely powerful if they effectively capture an organization’s strategic future.   with the assistance of the consultants, the four arl sites crafted very simple strategy maps early in the process – and then returned to rework them many times during the implementation period. in some cases, discussions with stakeholders revealed the need for fairly significant overhauls. in other cases, the changes were more minute (e.g., reworking the wording to improve clarity).   on occasion, the limitations of the initial strategy maps were not revealed until later stages – when organizations were trying to identify specific measures and targets.  the teams soon discovered that the choice of words was pivotal: if the word appears in an objective statement, it should be measured. for example if an objective is framed to “hire, retain, train, and develop highly motivated, productive, technologically fluent, diverse staff” then, ultimately, that organization will need to measure its hiring, retaining, training, developing, processes. in addition the organization may need to measure the motivation, productivity, tech fluency, and diversity of its staff. eight measures could be required to fully evaluate a single overly-wordy objective. this is one of the clear focusing mechanisms of the balanced scorecard: it forces an organization to reconsider their lovely, lofty, lyric objectives in favor of more precise statements.   commonalities between strategy maps   many commonalities are evident in the four libraries’ slates of strategic objectives (table 1). while the exact wording on the strategy maps may be slightly different, the intentions are strikingly similar. overlap is evident in each perspective, but is most noticeable in the customer and financial perspectives.   the following analysis of objectives and measures is a snapshot of what each library recorded at the time this paper was written. because this is a change process, objectives, measures, and even the look of the strategy maps changed fairly frequently.   table 1 number of strategic objectives by library and perspective organization customer financial learning & growth internal processes total # of objectives johns hopkins 4 3 2 2 11 mcmaster 4 1 3 2 10 u. of virginia* 3 3 3 9 13 u. of washington 3 3 3 5 13  total all libraries 14 10 11 18 53 * university of virginia numbers based on 2007/9 scorecard   common objectives across the perspectives   the financial perspective provided the strongest commonalities in objectives. of the 10 objectives, johns hopkins, virginia, and washington had 3 each, and mcmaster had 1. the themes in this perspective were clear – secure funding for operational needs (4), align resources strategically (2), and measuring and improving the impact of resources and services (2).   in the customer perspective there were a total of 14 objectives across the 4 libraries, 4 each from mcmaster and johns hopkins and 3 each from virginia and washington. commonalities included the following:   providing productive and user centered spaces, both virtual and physical (4) discovery, access, and preservation of collections for current and future scholars (3) providing access to library tools and services (3) becoming a world class teaching, research, and learning library (3) integrating into the university’s teaching and research mission (3)   the learning and growth perspective also displayed many commonalities. certain words appeared frequently to describe staff including “collaborative,” “innovative,” “dynamic,” “diverse” and “healthy.” of the 11 objectives logged, johns hopkins accounted for 2 while mcmaster, virginia, and washington each had 3. common themes are as follows:   developing workforces that are productive, motivated, and engaged (4) developing workforces that are based on alignment with their strategic plan (2) supporting diversity (2)   the unique objectives under this perspective are indicative of the local environment and organizational culture. from embedding flexibility into everyone’s job description to providing clear paths and processes to carry innovation into production, clearly these libraries are reexamining the type of staffing they will need in the upcoming years.   the internal processes perspective displayed the most divergence in content. there were a total of 18 objectives – 9 at virginia, 5 at washington, and 2 each at johns hopkins and mcmaster. the wide variation in sheer number of objectives is attributable to local preference: some locations chose to position traditional internal process objectives within the customer service perspective given the focus on users. common objectives included the following:   promoting the libraries resources, services, and value (3) advocating for scholarly communications (2) optimizing performance through efficiencies and effectiveness of programs (this is similar to objectives in the financial perspective from two of the libraries) (2) integration of resources into user environments, increasing access to content, and improving search and discovery (2)   the unique objectives under this perspective include:   identifying unique, rare, and valuable collections developing a world class preservation program building collection management strategies for materials needed by scholars retooling and expanding collection storage space.   identifying measures   once the slates of strategic objectives were set, the four teams moved on to develop measures. as noted earlier, all four sites had been collecting vast amounts of data for many years. two of the libraries had more robust assessment programs in place and so were more quickly able to map the measures they had to their respective objectives.   for those libraries with less advanced assessment programs, the consultant provided an exercise to facilitate measurement development. given that the objectives themselves are often large and intangible, groups were advised to ask a very simple question – if we want to achieve that then what do we have to do well? once an organization understands what it needs to do well, developing a measure is somewhat easier.   selecting appropriate measures   while there are a number of considerations in choosing measures for the scorecard, five critical ones became readily apparent to the four teams:   does the metric directly measure performance to achieve the objective? what data is needed for the measure? how often should the data be collected and used? how many measures are needed for each objective? how should the measurement be presented?   does the metric directly measure performance in achieving the objective?   most metrics operate as surrogates or indicators of performance measurement for objectives. if the objective is narrowly written and framed using quantitative data, then it should be possible to find direct measures for it. for example, an objective to “increase the amount of gift and endowment revenue” could be linked to a measure of current revenue against a baseline.  objectives at a broader level, such as “create world-class teaching and learning spaces,” would most likely use performance indicators such as user satisfaction with space or number of instructional spaces.  essentially, metrics should be able to measure or indicate an organization’s progress in achieving its objectives.   what data is needed for the measure?   as noted earlier, all four libraries were already collecting vast amounts of data for purposes of campus and professional association accountability. data that is already being collected should be reviewed first for use as measures. however, matthews  adds a cautionary note: “there is a tendency among libraries to consider only the measures that are currently being collected or that would be easy to collect” (2008, p. 67). yet, the time and costs involved in beginning new data collection processes can be substantial and should not be underestimated. the primary focus of time and effort should be on achieving the objective rather than coming up with the best method of measuring it.  above all, the data should be practical – obtainable with a reasonable amount of effort and easy to use and understand.   how often should the data be collected and used?   the issue of data frequency was a regular point of discussion. much of the balanced scorecard literature calls for frequent reporting of results from measures, many on a quarterly basis. the frequency may depend on how readily available the data is. data can be extracted from automated systems on demand, but survey data or other assessments may have a longer reporting cycle. academic libraries also need to consider the academic calendar: monthly or quarterly tracking will be less useful than term-to-term or year-to-year comparisons of the same academic term periods.   how many measures are needed for each objective?   the four groups struggled to determine the correct number of measures for each objective. the preferred number depends on the objective and the data. narrowly defined objectives generally require fewer measures than those broadly defined. for example, an organization with an objective around enhancing teaching and learning activities might consider tracking the number of instructional sessions and participants, session evaluations, number of academic programs reached, evidence in student work, survey responses, and faculty evaluations of usefulness. data availability and frequency may also have an impact on the number of measures. some data, such as satisfaction surveys, may be available only once every 2-3 years while other data is collected on an ongoing basis.  matthews notes that, “it is better to have fewer measures than too many” (2008, p.87). the number of measures per perspective should also be limited.  finally, the measures should be looked at holistically within the entire scorecard to ensure that they provide a balanced measure of overall performance and are not reliant on the same data sources.   how should the results be presented? choosing the best way to present the data was also a significant consideration. the four teams spent much time visualizing how specific measures would be displayed so as to capture what was most meaningful. understanding what each particular chart type can provide can clarify what you want to show even more.   bar charts compare the performance of different projects at points in time pie charts show the composition of a metric and are helpful for showing ratios stacked bar charts show the accumulation of a measure over time line charts show the performance of a metric over time   developing a standardized slate of measures for arl libraries   commonalities between pilot slates   we have seen some commonalities among the pilot sites. as a result a key question is whether arl can offer a menu of potential themes and associated metrics that libraries can target to excel in achieving their organizational mission and vision. while each library in this study is to some extent in the initial phase of measure and target development, some general observations can be made about the areas of overlap that can give insights as to whether an arl menu of metrics can be constructed.    table 2 identifies the measures per objective for the four pilot sites. the table indicates fairly wide variation in the number of measures – from a low of 26 measures logged by mcmaster and washington to a high of 48 measures logged by johns hopkins. the average number of measures per objective also varies significantly, from two to 4.3 measures.   table 2 average number of measures per objective, by library organization # of objectives # of measures average # measure/objective johns hopkins 11 48 4.3 mcmaster 10 26 2.6 university of virginia 13 36 2.7 university of washington 13 26 2   table 3 number of measures per perspective and library perspective institution number of measures customer 47 total measures jhu 19   mcmaster 11   university of virginia 9   university of washington 8       financial 24 total measures jhu 13   mcmaster 3   university of virginia 5   university of washington 3       learning and growth 27 total measures jhu 7   mcmaster 6   university of virginia 5   university of washington 9       internal 38 total measures jhu 9   mcmaster 6   university of virginia 10   university of washington 13   an analysis of common measures across the four perspectives (table 3) uncovers many trends. it is important to note that the libraries may have the same measure but align it with a different perspective.   in the customer perspective there were a total of 47 individual measures identified by the four libraries.  common measures include the following:   providing productive user-centered space (4) customer satisfaction survey either home grown or libqual+® (4) instruction (3) turnaround time of interlibrary loan (ill) or other delivery methods (3) collection, preservation, and discovery of collections, tools and services both currently and in the future (3) integrate the library into the university’s teaching and research (3) growth and use of the institutional repository (2)   in the financial perspective the following were common measures:   generation of funds from either development activities or other sources of revenue generation (4) cost and/or unit cost of e-resources (2) ranking in the arl investment index (2) library allocation as a percentage of the overall university budget (2)   unique financial measures include amount of grant funding, unit cost of specific functions such as ill, and measures surrounding how the library contributes to faculty research or how many journals the library holds based on citations by faculty authors.   in the learning and growth perspective the following were common measures:   employee satisfaction (three, with two specifically mentioning use of climatequal) diversity (2) completion or creation of training and development plans (2) retention rates (2) alignment with their strategic plans (2)   finally, common measures in the internal processes perspective include the following:   promoting the library and communicating its value (3) assessment of specific services (3) effectiveness of their liaison services (2) scholarly communications issues (2) resolution of information technology (it) related problems (2) circulation of new monographs in the past two years as a measure (2)   the unique measures in the internal perspective often deal with process improvements unique to each library.   creating a standardized slate   the high level of commonality between measures being proposed at the four pilot sites suggests that a standardized slate is viable. participants benefited greatly from sharing lists of measures. reviewing a peer’s slate sometimes suggested new areas for exploration. discussions around measures often saved considerable time: partners benefitted from the successes and the failures at the other institutions.   the concept of reviewing the arl statistics in light of common measurements also appears worthwhile. such a strategy would standardize the definitions being used across member institutions and allow benchmarking between peers.   but more work needs to be done. the four pilot sites have established their preliminary set of measures and have, in some cases, started to collect data. early experience suggests that libraries need to go through a full cycle of collecting and analyzing the data, then actually attempt to use it for discussion and decision making before determining if the framing is right. in some cases, a measure might seem to be useful until the first set of numbers appear. it is not until the collector tries to render the first set of charts or the analyst first puts it under the microscope that the true nature of the measure emerges. and sometimes the true picture is not really known until the data comes before the library’s leadership group for the first time. even the simplest measures turn out to be more complex than originally expected.   targets   once the measures were identified, the pilot sites began the very challenging task of setting targets. ultimately, measures have little context without clear expectations or targets. targets articulate the level of success needed in achieving the objective. targets are quantitative and should be attainable. they can be based on overall mission, benchmarked practices, historical performance, and baseline data. for many measures, some form of baseline data will already be available that can be used to set targets. in other cases, a best “guesstimate” will be needed at the beginning. targets should not be set so low that they are easily achieved without much effort. a higher target should be reachable with effort in a reasonable amount of time. setting targets too high may lead to staff frustration and a perception by those outside the organization that the libraries are not succeeding in meeting their mission and objectives. targets can be revised, especially if the initial effort was set without sufficient data.   the university of virginia library was in a unique position, having used a balanced scorecard approach for almost a decade. they had long used a “two target” approach (high target = full success, low target = partial success, no target = no success). low targets were usually set at a point slightly better than current performance, while high targets were set to encourage substantially improved performance. when possible, the value for current performance was based on historical data, but occasionally targets were based on the educated guesses of responsible staff as to current performance on a certain measure. virginia has analyzed their measures annually, noting their level of success for each measure (self, 2004).   organizational issues   the participating libraries faced a host of organizational issues that required considerable time and effort to address. in many cases, the issues were not well-covered in the balanced scorecard literature: the literature assumes that senior leadership will wholeheartedly champion the decision to implement the scorecard, that a senior team with authority to make decisions will oversee the process, and that staff throughout the organization will naturally understand and follow. the reality in a large academic library with a history of a more cautious approach to change, a strong emphasis on consensus and a suspicion of non-academic approaches, is often very different.   getting the senior leadership team’s attention not surprisingly, discussions of the balanced scorecard battled for attention with the immediate – the operational imperatives that suck the time out of typical leadership meetings. teams had to convince senior leaders that strategy needed to drive operations and that the scorecard presented a healthy mix of what the organizations have needed for some time. scorecard team members needed to find champions within the leadership group to support what amounted to an institutional leap of faith – that the scorecard would apply a level of discipline that would (ultimately!) simplify operational decisions, reduce waste and provide greater clarity around priorities.   all four teams reported success in engaging their leadership teams relatively early in the process. the leadership groups came to realize that the scorecard could raise the level of discussion at executive meetings, simplify decision making, and help steer budgetary decisions. the site with two associate university librarians on the scorecard team experienced the least difficulty in moving the initiative forward. this parallels the leadership involvement at the university of virginia library when they first adopted the balanced scorecard in 2001. teams with a more distant relationship to the senior leadership group encountered more difficulty during the early days in getting the leadership’s attention, securing time on leadership group agendas, and ultimately capturing interest in the scorecard.   overcoming resistance – the human dynamic   participating libraries found that some of their staff colleagues viewed the balanced scorecard with a degree of suspicion – in some cases, even cynicism. this response is not unique to libraries – with its emphasis on performance measurement, change, and accountability, employees of any organization are likely to offer resistance to the scorecard process, and countless articles and book chapters offer strategies on how to address this reaction. one presenter at a recent conference for mission driven organizations drew laughter from the audience when he spoke about how he handled “malicious compliance” within his organization.   the tension between strategy and operations was experienced by all teams. staff expressed concern about not seeing their specific work assignments explicitly linked to the strategy map, metrics and initiatives. the four teams conveyed a similar message back to their organizations. in some cases, the work being done by a particular unit is extremely important to supporting institutional priorities, but in itself, is not strategic at the organizational level.   in addition, the arl initiative was underway during a particularly dire economic crisis when library budgets were stretched to the limit and layoffs were a distinct reality at many institutions. at the individual level, the balanced scorecard can be threatening – though not generally directly tied to job evaluations, it can be seen as a form of public performance management, and often is used to focus attention on strategic goals at the expense of ongoing, perhaps outmoded, operations. even in flusher financial times, virginia experienced staff reluctance to set “stretch” targets in their areas for fear of failure.   it also may be that certain elements common to the organizational culture of academic libraries may contribute additional resistance to the balanced scorecard.   first, libraries and library staff are not widely known as change agents. on the contrary, the collective focus has traditionally been on preservation and stability. even key innovations have often been focused on maintaining continuity and access to historical material, albeit in new ways. academic libraries, in particular, like the colleges and universities of which they are a part, are just beginning to substantially change their basic physical and organizational structures. only in the last decade have libraries begun to prioritize digital over physical collections and hire programmers as they once hired bibliographers. in many ways, libraries still operate very similarly to their counterparts a century ago. libraries are working at change, but unlike their counterparts in the fast-paced commercial world, still change relatively slowly.   second, academic libraries and library staff are not predisposed to adopt business tools. often mirroring the views of the academic community they serve, libraries tend to think that the scholarly nature of their work precludes the successful use of business-based strategies. however, there are signs that this attitude is beginning to change, with outcome-based budgeting and return-on-investment models gaining traction at institutions of higher education.   finally, libraries tend to operate within silos. this too is reinforced by the larger structure of the college or university, where individual departments typically retain a great deal of autonomy. despite the contemporary focus on collaboration and interdisciplinary scholarship, academics still tend to work on their own. library staff are not used to coming together to talk about the organization as a whole. staff members tend to focus on their own areas of specialization (e.g., cataloguing, reference, etc.) and do not typically create forums to facilitate high-level discussions about the future of the entire organization.   by implementing the balanced scorecard, planning teams are asking staff and campus stakeholders to make several leaps of faith – not only that tracking progress will increase the probability that the library will achieve its collective goals, but also that change is necessary and good, that a solution developed by and for the business world may have value in the academic environment, and that by working together, library staff can achieve more than by working independently.   making decisions/authority   the balanced scorecard, and the strategy that underlies it, is a compendium of choices or decisions – many of them hard ones. the strategy map forces the organization to choose one priority or direction over another. the final choice of metrics reflects a collective decision about what truly matters and is worth counting. the specific projects or initiatives linked to the scorecard reflect hard decisions about where the organization will invest its time and limited resources.   some participants reported issues associated with governance and authority structures.  in some cases, decision-making structures overlapped, thus getting in the way of setting clear priorities. in other cases, the decision-making structure was not clear.   more often though, the issue was behavioral rather than structural. participants encountered hesitancy to commit to one plan over another and a reluctance to be the one to make the final decision. staff often reported that they did not have enough information to provide an opinion on a given tactic. groups tended to revisit the same issues over and over again and bring closure to the issues at hand.   this reluctance might be associated with the historic focus on consensus as a decision-making style within many academic libraries (and within the academy as a whole). achieving 100% consensus on a given issue can take time and sometimes results in weaker solutions.   the four teams used a variety of techniques to arrive at decisions. all sites used a blending of staff committees to work on various aspects of the scorecard. at the end of the day, in all four sites, final decisions were made by the senior leadership team. some sites had success with expressing the continuous nature of the review process. the strategies were framed as hypotheses, the best choice of action at the given time and with the given information. mid-year adjustments and regular review were part of the process.   integrating the balanced scorecard into the strategic planning process   the four groups recognized that incorporating the balanced scorecard into their libraries’ existing assessment program was relatively easy, but if that was the full extent of the integration, the implementation would be only partially successful. the scorecard is not, as might appear at first, simply a container for assessment data. rather, the scorecard is a management and change process first, and a metrics process second. the teams recognized that the scorecard required a robust planning and decision-making cycle. to be effective, the senior leadership group needed to be reviewing the metrics and the strategic initiatives on a regular basis. the review meetings needed to be deep and focused on achieving success. the scorecard becomes the catalyst for rich conversations and sometimes difficult decisions – not just another cluster of data to shelve from quarter to quarter.    communicating progress with staff   given the complexity and intense integration of the scorecard into the organizational fabric, regular communication of progress with staff proved to be essential – but not always easy.   the strategy maps provided a good graphical representation of the libraries’ strategic objectives – but the interrelationships between the objectives, the metrics and the initiatives were hard to explain. participants struggled to find the right visual to bring all the pieces together and some pieces were undoubtedly lost in translation.  the pilot leads struggled with providing the right information at the right time – without unnecessarily confusing their colleagues. in many cases, chunks of time past without noticeable progress – and the initiative moved to the back of people’s consciousness. and of course, the participants themselves were learning as they went along.   the participants tried a variety of approaches to share their stories with their colleagues and their campus communities. most teams sent out regular communiqués to library staff and held a variety of face-to-face sessions (e.g., presentations, hands-on workshops, etc.). some sites reported the best progress when blending the balanced scorecard information with other broader events.   conclusion   the yearlong arl initiative has met its initial objectives. the four local implementations are still a work in progress, but the leads are fully trained and the infrastructure is in place. the sites continue to refine their measures, set their targets, and occasionally circle back to their original objective statements. data is being collected. the leadership teams are starting to see their first deliverables out of the balanced scorecard process.   although still early in the game, the concept of identifying standard suites of objectives and measures that arl libraries can select or start from appears to hold merit. strong commonalities are evident in the four sites’ work. as well, the arl objective to test a collaborative approach to assessment has been fruitful. the opportunities to discuss concepts and wording with peers helped reduce barriers. the community of practice around the scorecard process has helped make each implementation richer.   the study has identified the challenges but also the tremendous opportunities for implementing the balanced scorecard in an academic library. the process requires a substantial allocation of time and intellectual effort. the process requires a significant and ongoing commitment from senior leadership to be successful. the strength of the scorecard is its linkages. the process, if done effectively, can help solidify the bond between the organization’s strategic objectives and the specific initiatives it elects to undertake. the balanced scorecard forces an organization to have new, sometimes challenging, conversations and to analyze aspects of its current and future state that may have otherwise gone unexamined. ultimately, the scorecard may substantially shift an organization’s strategic direction or dramatically change how its human capital and other resources are allocated. the scorecard is, by its very nature, a change driver. the model commits the organization to continuous and regular reflection and to communicate the results of those reflections with a new level of discipline and precision.   references   association of research libraries. (2009). e-news for arl directors. retrieved 30 may 2013 from http://old.arl.org/news/enews/enews-15jan09.shtml   brophy, p. (2008). telling the story: qualitative approaches to measuring the performance of emerging library services. performance measurement and metrics, 9(1), 7-17. doi: 10.1108/14678040810869387   durgan, r. e., hernon, p., & nitecki, d. a. (2009). viewing library metrics from different perspectives: inputs, outputs and outcomes. santa barbara, ca: libraries unlimited.   franklin, b. (2011). surviving to thriving: advancing the institutional mission. (2011). university of connecticut libraries published works, paper 36. retrieved 30 may 2013 from http://digitalcommons.uconn.edu/libr_pubs/36   kaplan, r. s., & norton, d. p. (1992). the balanced scorecard: measures that drive performance. harvard business review, 70 (1), 71-79.   kaplan, r. s., & norton, d. p. (1996). the balanced scorecard: translating strategy into action. boston: harvard business school press.   kaplan, r. s., & norton, d. p. (2001). the strategy-focused organization: how balanced scorecard companies thrive in the new business environment. boston: harvard business school press.   kyrillidou, m. (2010). the arl library scorecard pilot: using the balanced scorecard in research libraries. rli, 271, 33-35. retrieved 28 may 2013 from http://publications.arl.org/10opit.pdf   matthews, j. r. (2008). scorecards for results: a guide for developing a library balanced scorecard. westport, ct: libraries unlimited.   poll r., & te boekhorst, p. (2007). measuring quality: performance measurement in libraries..munich: k.g. saur.   self, j. (2003). from values to metrics: implementation of the balanced scorecard at a university library. performance measurement and metrics, 4(2), 57-63. doi: 10.1108/14678040310486891   self, j. (2004). metrics and management: applying the results of the balanced scorecard. performance measurement and metrics, 5(3), 101-105. doi: 10.1108/14678040410570111   article   effects of mentioning the incentive prize in the email subject line on survey response   robert janke                                                                                                                                        learning services librarian okanagan campus library, university of british columbia kelowna, british columbia, canada email: robert.janke@ubc.ca   received: 15 oct. 2013    accepted: 4 feb. 2014      2014 janke. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – this study examined the effects that mentioning the survey incentive prize in the subject line of a reminder email had on the response rate and data quality. to date, manipulation of the subject line, specifically in terms of mentioning the incentive prize, has received limited attention in the survey design literature.    methods – the delivery of the survey invitation is discussed in terms of the timing of the launch and reminder emails. particular emphasis is given to the design of the email subject line and justification of the format. weekly response rates from four libqual+tm surveys were compared. in addition, weekly responses for one year were analyzed using spss to investigate if there were any between means differences in terms of three elements of data quality. the three elements were: length of time it took to complete the survey, the number of core questions with an n/a response, and the number of illogical responses where minimum scores were higher than desired.   results – the response rates for the second week were grouped together based on the presence or absence of the subject line manipulation. there was a significant difference between these means (4.75%, p 0.033). there was no statistical difference in regards to the measures of data quality as determined by a one-way anova test.   conclusions – reminding survey participants with an email that mentions the incentive prize in the subject line appears to increase response rates with no deleterious effects on data quality. the results of this investigation are encouraging, and those running the libqual+tm survey in their universities should consider implementing this method to increase response rates. further research to replicate these findings in other contexts and using an experimental design would be beneficial.      introduction   the library at the okanagan campus of the university of british columbia (ubco) has surveyed all of its faculty and students on three occasions using libqual+tm.  libqual+tm is a standardized survey developed by the association of research libraries to measure the service quality perceptions of library users. the surveys have taken place in 2007, 2010, and 2013. as with many other libraries that survey their users, ubco offered lottery incentive prizes in the hopes that it would increase response rate. in survey design, lottery incentive prizes differ from paid incentives in that the individual has a chance to win the given prize, as opposed to paid incentives, either preor postsurvey completion, that guarantee a prize for participants. as is typically the case for lottery incentives, ubco library made participants aware of the incentive prize in the body of the email inviting them to complete the survey (see the appendix for a sample of the invitation email). the response rate to the 2007 survey was 17.9%. however, with the explosion in the popularity of smart phones and other mobile devices, the author became curious in the lead up to the launch of the 2010 survey about the extent to which students were reading the full invitation to become aware that an incentive prize was being offered. the reasoning behind this concern is that the smaller screens may make it less appealing to read long emails, or the configuration of some of the email programs may put a greater emphasis on the subject line in the decision to open or delete the full message. as a result, this study addressed the following research questions:   1) would giving the existence of the lottery incentive prize more prominence, by mentioning it in the subject line of the email invitation, increase the survey response rate?    2) if mentioning the incentive prize increased the response rate, would this have a negative effect on the quality of the survey responses?   response rates should be a concern to all survey administrators. according to manzo and burke (2012), response rates for all types of surveys have been declining over the last decade, and low response rates threaten the validity of surveys. this is because as a group, non-responders may share similar characteristics. by not capturing their data, the sample and survey results would be biased.            literature review   there is considerable interest in the use of lottery survey incentive prizes on university campuses, so much so that there have been surveys by institutional researchers (porter & whitcomb, 2003) and librarians (buck, nutefall, & bridges, 2012) to gauge the level of their use. this obvious interest aside, the evidence regarding the effects of lottery incentives on survey response rates is contradictory.    a meta-analysis by cook, heath, and thompson (2000) noted that surveys using an incentive seemed to be associated with a lower response rate. in contrast, another meta-analysis by göritz (2006) concluded that there was a significant odds ratio of 1.19 showing that incentives encourage individuals to start web surveys and complete them (odds ratio of 1.27). expressed differently, an incentive should increase the odds of a person beginning the survey by 19% and completing it by 27% over the odds without incentives. in terms of absolute percentage differences, göritz (2006) concluded that an incentive should increase the response rate by an average of 2.8% and retention by 4.2%.   these two meta-analyses studied the effects of incentive prizes on response rates. there is an implied assumption, however, that the responder is aware that a prize is offered when the incentive prize is mentioned in the body of the invitation. this assumption is particularly interesting in light of a report from a canadian post-secondary library, using constant contact (email tracking software), that as little as 33% of their users bothered to open the email inviting them to take their libqual+tm survey in 2012 (reed, 2012). one of the reasons that invitees may or may not open and read an email is the subject line. the subject line, in contrast to the surveyor’s name and email address, may be the most likely element of the invitation that encourages recipients to open and review its contents (manzo & burke, 2012).     research to date on the effects of the subject line of the invitation email has been sparse. in research with high school seniors as well as undergraduates, porter and whitcomb (2005) manipulated the reason for the email, survey sponsor, and whether or not the subject line included a plea for help. trouteaud (2004) also used a plea manipulation in the subject line (please help... vs. share your advice with...) in an experiment conducted on american subscribers to a large company’s daily email newsletter. concerning research into the effects of mentioning the incentive prize in the email invitation to a web-based survey, only two studies were found.    linegang and moroney (2012), in an experiment at the university of dayton (ud), manipulated the subject line to gauge its effects on response rates among undergraduates invited to take a survey. both experimental groups received a pre-survey notification, a survey notification, and a reminder notice. all recipients were entered into a draw for gift certificates from local restaurants, and this information was communicated in the text of the email. linegang and moroney reported neither the value of these gift certificates, nor the number that they would be giving away. one group received the subject line “ud computer survey” while the other received the subject line “free food!!! ud computer survey” in all of their communications. it was reported that the response rate for the group that received the email subject line that mentioned the incentive prize was 24.1% compared to 30.3% for the group where the subject line did not mention the incentive prize, a 6.2% lower response rate for the group invited to the survey with an emphasis on the incentive prize.    similarly, kent and brandal (2003) also found a decreased response rate among their experimental group that received an e-mail subject line that mentioned the incentive prize. the subjects for their experiment were taken from the customer database of loyalty cardholders from a norwegian company. kent and brandal do not specify the subject line that did not emphasize the incentive prize, only saying that it was a survey from the company, whereas the other group’s email had the subject line “win a weekend for two to nice.” the response rates for the two groups were 66% and 52% respectively, a 14% lower response rate for the group invited to the survey with an emphasis on the incentive prize.   these counterintuitive results suggest that perhaps the recipients of the email which emphasized the incentive prize believed the email was “spam” that is, unsolicited email from a dubious source where there may or may not be an actual incentive prize to be won.      methods   sample   this research is based on data collected at ubco using the libqual+tm web survey instrument. the first time libqual+tm was used at the campus was in 2007, and this serves as a baseline for purposes of comparison. to test the effect of mentioning the incentive prize on survey response the author manipulated the content of the subject lines in the 2010 and 2013 survey cycles.     all three survey cycles launched at the start of the fourth week of classes in the second term of the winter session. in all cases, there was a single incentive prize worth approximately $300 mentioned in the body of the email invitation. with minor exceptions, the email invitations were identical each year: an example is in the appendix. the reminder email contained all of the text of the invitation with the addition of a paragraph at the beginning apologizing to those individuals who already completed the survey. (as no personal data were collected, the library could not determine who completed the survey and remove them from the invitee list). in 2007, the incentive prize was a digital camera. the subject line of the invitation and reminder emails, sent at one week intervals, was neutral with respect to mentioning the incentive prize, and simply said library survey.   data collection   because of the negative effects found in other research that mentioned the incentive prize in the email invitation (kent & brandal, 2003; linegang & moroney, 2012), great attention was paid to ways to alleviate the spam effect that may have influenced the outcome of these studies. in both of the intervention years, 2010 and 2013, the invitation email had the following subject line: library survey please let us know what you think of our service. only the reminder email, sent one week after the invitation, mentioned the incentive prize: library survey – you could win an [name of incentive prize]. the purpose of this two-stage approach was to build trust and familiarity with the initial invitation and then mention the incentive prize with the follow-up. a final strategy employed was to construct the subject lines in a consistent manner that made it clear that the email was indeed an invitation to a reputable survey. this strategy avoided the use of excessive capitalization, as in the case with linegang and moroney, and mentioned the word survey, unlike in the research by kent and brandal.      the final reminder, sent a week after the first, had the following subject line: library survey – your last chance to win an [name of incentive prize]. in 2010, the incentive prize was an ipod touch, while in 2013 it was an ipad mini. enrolment services sent out the invitations and reminders, and the author’s institutional email address appeared as the sender in order to be the one to receive any replies with questions.     in addition to response rate data from 2007, data from another institution that also ran libqual+tm in 2013 was included for comparison purposes. the vancouver campus of the university of british columbia was chosen as they also ran the survey in the second term of the winter session and sent survey reminders at one week intervals. instead of surveying all undergraduate and graduate students, they sampled from their population. the subject line for their invitation and first reminder email mentioned the existence of incentive prizes but did not specify what they were. more importantly, there was not the specific manipulation of mentioning the incentive prize beginning with the reminder email. the subject lines were: initial invitation ubc library survey – tell us what you think & enter to win prizes; first reminder ubc library survey – provide your feedback & enter to win prizes; final reminder ubc library survey – one week left to provide your feedback. the body of the email invitation mentioned the incentive prizes of an ipad mini and six $25 gift cards.   data analysis   excel files containing all survey data from the various survey cycles, supplied by libqual+tm, were used to generate the weekly and overall response rates. data quality was assessed using the raw data from the 2013 survey and analyzed using spss, version 19, to see if there were differences in data quality between the responses submitted during each week using a one-way anova test. the raw data file from libqual+tm includes the date that the survey was submitted, facilitating the grouping of responses. variables supplied in the survey data that were analyzed for quality include stime, the length of time it took the respondent to fill out the survey in seconds, countna, the number of core items where there was an n/a response, and finally countinv, the number of illogical responses where minimum scores are higher than desired. countinv is unique to the core questions in libqual+tm, where for each question the respondent provides three responses on a scale of 1-9, their desired and minimum levels of service as well as where they perceive the service quality of the library to be. because an individual’s minimum score should not exceed their desired, instances where this occurs would indicate a responder who is not paying close attention to the actual content of the questions.         results   table 1 details the number of students surveyed in a given year and the number of valid surveys received after the initial invitation and after each of the two reminders, as well as corresponding weekly response rates. 2013v represents the response rate from the vancouver campus survey. the calculation of the weekly response rates was based on the premise that those individuals who had responded in the previous week(s) were unlikely to respond again and were therefore removed from the denominator. the final column reflects the overall response rate for the different years and was calculated by simply dividing the total number of responses received over the course of the survey by the number of students surveyed.    although there is an increase in overall response rate between the 2007 and the 2010 and 2013 iterations, 5.1% and 1.2% respectively, 2007 marked the only year that the long version of the libqual+tm metric was used. as a result, attention should be paid to the significant differences in response rates for week 2, following the reminder email, and the change in the subject line to emphasize the incentive prize in years 2010 and 2013, rather than a comparison of the overall response rate. this is because the much shorter libqual+tm lite survey was used in 2010 and 2013, which may confound inter-year comparisons of overall response rate. figure 1 compares the weekly response rate graphically.    table 1 valid surveys received per week expressed as a weekly response rate year number of students surveyed valid surveys week 1 response rate week 1 valid surveys week 2 response rate week 2 valid surveys week 3 response rate week 3 overall response rate 2007 4132 325 7.9% 239 6.3% 176 4.9% 17.9% 2010 6160 541 8.8% 599 10.6% 278 5.5% 23% 2013 8069 403 5% 778 10.1% 358 5.2% 19.1% 2013v 4376 376 8.6% 195 4.9% 112 2.9% 15.6%     table 2 focuses on the response rates for week 2 and groups them together based on whether or not the incentive prize was mentioned. the mean response rate for the baseline data (no manipulation) for 2007 and 2013v was 5.6%, while the mean response rate for the years where there was a manipulation of the subject line, 2010 and 2013, was 10.35%. when the incentive prize was mentioned in the subject line in the reminder email, the response rate for that week was significantly higher (4.75%, p 0.033).   with respect to data quality, göritz (2006) raises some concerns in regards to offering incentives, namely, individuals completing the survey multiple times or simply entering “rubbish” responses in order to get to the end of the survey and be eligible for the incentive. for this investigation, there were no statistically significant differences between group means as determined by a one-way anova (scores were all above .05) for stime, countna, and countinv, indicating there was no more “rubbish” entered when the incentive prize was mentioned than when it was not.      figure 1 comparisons of weekly response rates.     table 2 response rates for week 2   mean standard deviation incentive mentioned (2010 & 2013) 10.35% 0.35 incentive not mentioned (2007 & 2013v) 5.6% 0.99 difference 4.75%*   *one tail t-test of significant differences (p<.05)     discussion   the results of this study suggest that mentioning the incentive prize in the subject line of the reminder email yields, on average, a 4.75% higher response rate for the given week. this is contrasted with linegang and moroney (2012), who found a 6.2% lower overall response rate for the invited with emphasis group and kent and brandal (2003) who found a 14% lower response rate. one explanation for this difference could be how the subject lines were constructed in the current investigation, most notably the absence of excessive capitalization and inclusion of the word “survey.”    unfortunately, it is difficult to gauge the effects this study had on the overall survey response rate because of a lack of experimental comparisons in the design. the largest external baseline for overall response rates that could be found averaged the rates from 13 post-secondary libraries that used a 100% lite version of libqual+tm in 2010. that survey of libqual+tm administrators by buck et al. (2012) reported an average response rate of 17%, which is 6% lower than the response rate of 23% that ubco obtained in 2010. taking that 6% difference and the 3.5% difference observed in the 2013 response rates between the two campuses of ubc does indicate a positive trend line for the effects of mentioning the incentive prize. the decision to complete a survey is complex, with multiple variables at play, making these inter-institutional comparisons less than ideal. see below for a suggested course of research that would better establish the effects on the overall response rate.     in regards to data quality, analysis of the 2013 survey responses indicated no inter-week differences on the measures chosen for analysis. these results are similar to those reported elsewhere that found no statistically significant differences in regards to the response speed (heerwegh, 2006) and item non-response (heerwegh, 2006; sánchez-fernández, muñoz-leiva, & montoro-ríos, 2012) when comparing groups that were either offered or not offered an incentive prize to complete a survey.   a practical implication of this research for librarians and information professionals who are delivering libqual+tm surveys via email is that mentioning the incentive prize in the reminder email will increase response rates for the given week and may improve overall response rates. survey administrators should also have confidence that implementing the strategies outlined in this study will not have a negative impact on the quality of responses provided by the respondents.   limitations & future research   the post hoc design of this investigation does not permit strong conclusions with regard to the exact effects mentioning the incentive prize in the subject line has on overall response rates for the libqual+tm survey. however, when contrasted with the limited literature in this area (kent & brandal, 2003; linegang & moroney, 2012) and their findings of a negative influence on response rates, it does make a strong argument for further research. in the future, it would be beneficial to create an experimental design in which three groups are randomly generated. one group would be invited and reminded about a survey with an email that does not mention the incentive in the subject line. a second group would be invited with a neutral subject line, but reminded with a subject line that mentions the incentive prize. lastly, a third group would be invited and reminded with a subject line that mentions the incentive prize in both instances. of course, if such research were carried out on a single university campus, one would have to keep in mind the concern of between experimental group communication raised by porter and whitcomb (2003), communication that is all the more likely in this time of hyper connectedness and when there are potentially valuable incentive prizes available to be won.    related to survey data quality, another aspect that deserves further attention would be a comparison of click-through rates and completion rates. although out of the scope of this investigation, comparing these two rates for instances when the prize was mentioned and when it was not, would give a more complete picture as to whether mentioned incentives encouraged someone to click through to the survey but once they viewed the survey for whatever reason they declined to fill it out.   another interesting line of inquiry would be other aspects of subject line composition that may have an influence on response rate. one example would be moving the mentioning of the incentive prize to earlier in the subject line. for instance, it might be interesting to compare the effects of the following subject line library survey – you could win an [name of incentive prize] with you could win an [name of incentive price] – fill out the library survey.   conclusion   increasing response rates for surveys conducted on university campuses is an area of interest for both librarians and institutional researchers alike. a common approach to attempt to increase response rates is to offer a lottery incentive prize. this study demonstrated a beneficial way to increase the response rates for the libqual+tm survey following the reminder email by manipulating the email subject line. in contrast to earlier studies, this study found that mentioning the incentive prize in the subject line of the reminder email increased the response rate. further investigation would permit conclusions on the effect that this manipulation had on the overall response rate for the survey, not just for a given week that it was open. in regards to data quality, this study found no differences between the weeks where the subject line manipulation occurred and when it did not for the three variables chosen for investigation. these results echo research conducted elsewhere on incentives and data quality. results of this study should give survey administrators confidence that adopting the strategy outlined in this investigation should not only increase the response rate for the week following the reminder email but should also not attract an inordinate amount of careless responses used as a vehicle for entry into the draw for the incentive prize.   references   buck, s., nutefall, j. e., & bridges, l. m. (2012). “we thought it might encourage participation." using lottery incentives to improve libqual+tm response rates among students. journal of academic librarianship, 38(6), 400-408. doi:10.1016 /j.acalib.2012.07.004   cook, c., heath, f., & thompson, r. l. (2000). a meta-analysis of response rates in webor internet-based surveys. educational and psychological measurement, 60(6), 821-836. doi:10.1177/00131640021970934   göritz, a. s. (2006). incentives in web studies: methodological issues and a review. international journal of internet science, 1(1), 58-70. retrieved 10 feb. 2014 from http://www.ijis.net/   heerwegh, d. (2006). an investigation of the effect of lotteries on web survey response rates. field methods, 18(2), 205-220. doi:10.1177/1525822x05285781   kent, r., & brandal, h. (2003). improving email response in a permission marketing context. international journal of market research, 45(4), 489-503. retrieved 10 feb. 2014 from https://www.mrs.org.uk/ijmr   linegang, m. p., & moroney, w. f. (2012). effects of cover letter subject line and open-ended question response area on responding to an internet survey. proceedings of the human factors and ergonomics society annual meeting, 56(1), 1268-1272. doi:10.1177/1071181312561225   manzo, a. n., & burke, j. m. (2012). increasing response rate in web-based/internet surveys. in l. gideon (ed.), handbook of survey methodology for the social sciences (pp. 327-343). new york, ny: springer. doi:10.1007/978-1-4614-3876-2_19   porter, s. r., & whitcomb, m. e. (2003). the impact of lottery incentives on student survey response rates. research in higher education, 44(4), 389-407. doi:10.1023/a:1024263031800   porter, s. r., & whitcomb, m. e. (2005). e-mail subject lines and their effect on web survey viewing and response. social science computer review, 23(3), 380-387. doi:10.1177/0894439305275912   reed, k. (2012, november 27). getting ready for libqual 2013. message posted to libqual-canada electronic mailing list, archived at http://www.mailman.srv.ualberta.ca/mailman/listinfo/libqual-canada   sánchez-fernández, j., muñoz-leiva, f., & montoro-ríos, f. j. (2012). improving retention rate and response quality in web-based surveys. computers in human behavior, 28(2), 507-514. doi:10.1016/j.chb.2011.10.023   trouteaud, a. r. (2004). how you ask counts: a test of internet-related components of response rates to a web-based survey. social science computer review, 22(3), 385-392.  doi:10.1177/0894439304265650     appendix example of invitation email   you are invited to participate in a comprehensive survey of library service quality. the survey, known as libqual+tm, assesses satisfaction with collections, services, access, and space at participating academic libraries throughout the world. because so many libraries use libqual+tm, it allows us to compare how we are doing with other libraries in canada, as well as with our colleagues at the ubc vancouver campus. these results help us create the library you need in the future. we value your input. speak up!   past experience indicates it takes an average of only 5 minutes to complete the web-based survey. please keep in mind that all the core questions must be completed for your results to be tallied in the overall totals and that if you do not wish to respond to a particular questions, just select the "na" box in the right hand column.   to take the web-based survey, please click on: [survey url]   the survey will be open from [survey dates]   incentive draw: your time is important! thank you! the library is offering an ipad mini to a randomly selected participant in the survey. if you choose, you may enter the draw by entering your e-mail address at the end of the survey.   confidentiality: all responses are held in strictest confidence. no identifying links between responses and the individual are retained. the only identifying piece of information, (your e-mail addresses if you choose to enter the draw), is stored separately from the survey results and is discarded after the winner has been identified.   more information: to see more information about the survey and its goals, please see: [url for survey information]   whom to contact: if you have any difficulty accessing or taking the survey or have any other questions or comments about the libqual+tm survey at ubc okanagan, please contact [survey administrator] by e-mail at [administrator’s address]   thank you for your help. [name and rank of chief librarian]         evidence summary   developing countries lag behind the us and uk in contributing to institutional repository literature   a review of: bhardwaj, r. k. (2014). institutional repository literature: a bibliometric analysis. science &technology libraries, 33(2), 185-202. http://dx.doi.org/10.1080/0194262x.2014.906018   reviewed by: heather coates digital scholarship & data management librarian indiana university-purdue university indianapolis (iupui) university library indianapolis, indiana, united states of america email: hcoates@iupui.edu   received: 27 feb. 2015   accepted: 21 apr. 2015      2015 coates. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – quantify the ir literature across the world by identifying countries with relatively high concentration of articles, describing the distribution of the literature by language, author (institutional and individual), journal, and examining characteristics such as the transformative activity index, and authorship and citation patterns.   design – this exploratory study of the literature used several bibliometric research methods to describe patterns and identify highly represented articles, authors, institutions, and journals.   setting – the library and information science abstracts database.   subjects – 436 articles from 118 journals.   methods – research articles and review papers published through december 31, 2012, were identified by searching library and information science abstracts (lisa). citation data for the 436 articles selected was gathered from lisa and scopus.   main results – the 436 articles from 118 journals had publication dates from 2001 through 2012, originated from 68 countries in 19 languages, and had authors affiliated with 159 institutions. the greatest number of institutional repository articles were published in 2011 while year-to-year growth was greatest from 2005-2006. most highly represented were the united states and the united kingdom, followed by india, australia, and spain.   twenty publishers were responsible for nearly half of the selected articles. the top four journals included oclc systems & services, d-lib magazine, serials review, and library hi tech. d-lib magazine alone published seven of the top 20 most cited articles. while most articles were written by a single author, the majority of the multiple author articles came from developed countries. citation analysis reveals that the 436 articles were cited 2,071 times, for an average of 4.8 citations per article. however, 147 articles received no citations. the five most prolific authors were elizabeth yakel, kim jihyun, karen markey, jingfeng xia, and sarika sawant.   conclusion – the author concludes that developing countries lag behind in establishing and publishing on institutional repositories and suggests that more authors will deposit in ir in the future. a proposed role for lis professionals is to communicate the objectives, values, and principles behind institutional repositories.   commentary   this study offers an international perspective in a body of literature that tends to focus on western institutions and practices. the author attempts to broadly characterize the global ir literature. while the findings are novel, the reader is left to interpret their consequence without guidance from the author. repository management as an evolving area of librarianship, changing institutional contexts, dramatic improvements in storage and discovery technologies, and the relationship between ir literature and practice in establishing and managing ir are not explored. the ir literature is not an appropriate proxy for the establishment and usage of institutional repositories across diverse national, academic, and library settings. the author appears to confuse the growth of institutional repositories with the growth of the ir literature and presents no evidence for the growth and expansion of such repositories.   though the author provides an ambitious list of study objectives, the methods section is truncated. it excludes crucial information about the methods used that is necessary to ascertain the validity of these results. more specifically, key details of the search terms were omitted, such as the search date range and the timeframe for conducting the searches. the search string itself was overly basic and may have excluded articles about platforms other than dspace, greenstone, and e-prints. furthermore, it is difficult to perform a critical appraisal of the findings without knowing the criteria applied during the screening and selection process. these gaps leave the reader questioning how well this sample captures the institutional repository literature.   the results consist largely of descriptive statistics, which reveal some interesting trends. unfortunately, the author does not delve into relationships between the variables. by emphasizing the educational role of librarians in helping users to deposit and use repositories, the author minimizes significant shifts in the scholarly communication ecosystem that have driven the expansion of ir. although they serve other purposes than mechanisms for open access, the motivation and context for presenting ir has largely been to rectify unequal access and preserve the scholarly record. this gap is apparent in the literature review and bibliography, where key concepts and publications are missing. charles bailey’s thorough bibliography on institutional repositories (2011) is an excellent resource that could have provided much needed structure and context for exploring the practical implication of these findings.   despite the shortcomings, repository managers may find this broad account of the ir literature useful for positioning their own scholarship, within the global literature. future studies could extend this work to explore the particularly issues and challenges in repository management and associated services as the field has developed over time. such information would be invaluable in understanding how institutional repository services respond to emerging technologies and the challenges facing higher education.   references   bailey, c. w. (2011). institutional repository bibliography, version 4. houston: digital scholarship. retrieved from http://www.digital-scholarship.org/irb/irb.html   evidence summary   assessment related skills and knowledge are increasingly mentioned in library job postings   a review of: passoneau, s., & erickson, s. (2014). core competencies for assessment in libraries: a review and analysis of job postings. library leadership & management, 28(4):1-19.  https://journals.tdl.org/llm/index.php/llm/article/view/7080   reviewed by: carol perryman assistant professor texas woman’s university denton, texas, united states of america email: cp1757@gmail.com   received: 6 dec. 2014     accepted: 26 jan. 2015      2015 perryman. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – the authors sought to determine whether existing definitions of assessment agree with assessment-related skills sought in job postings, and to identify key assessment-related skills, needs for training, and trends in assessment.   design – content analysis.   setting – job postings from six library-specific websites: the american library association, the library & information technology association, the society of american archivists, the council on library and information resources, the association of research libraries, and library assessment job announcements at http://libraryassessment.info/?cat=13.   subjects – job titles and descriptions published during an 18-month period between summer 2012 and winter 2013 that met the inclusion criteria (n=231).   methods – job postings were searched and analyzed in two separate sets whose inclusion criteria is as follows: first, job postings with the term assessment in the position title or as the main focus of the position (n=44) were retrieved; of these, three postings were too old to contain descriptions, so were excluded from analysis. second, job postings were retrieved with the terms assessment, evaluation, metrics, and strategic in the descriptive text of postings with position titles that did not specifically mention assessment (n=187). the full text of both sets was downloaded to atlas.ti software for analysis using a grounded theory approach. mutually exclusive terms emerging from the coding process were documented and defined; from this analysis, networks of code “families” or co-relational groupings helped to create categories and sub-categories. the context of terms was closely examined to understand the meaning of assessment-related terms in job descriptions. following this step, microsoft excel was used to generate tables and pivot tables, aiding understanding and illustrating data.   main results – all 44 job posts containing the term assessment as part of the position title were from research universities or four year colleges; of these, most were arl member libraries. for these postings, the concept of assessment was more clearly aligned with definitions of assessment as an ongoing process. the positions described, requiring a minimum of three years’ of experience, ranged from entry-level to administrative in nature.   in the second set (187 postings), the interchangeable use of the terms “assessment” and “evaluation” was particularly evident in job postings unrelated to library instruction. no library types other than academic were recruiting for assessment librarians, but related skills, usually referred to as evaluation in public and special libraries, were mentioned in all areas of library practice including instruction, administration, public services, user behavior, and to a lesser extent, access services, archives, information technology, cataloging, and more.  while less prominent, these less often mentioned areas of practice also appear to be increasing their awareness of assessment.   key skills and knowledge areas needed for assessment in libraries emerged from content analysis of the job postings. these were grouped under eight main areas of competency and were augmented by the authors’ own experiences as assessment librarians: background in library assessment, research methods, statistical and analytic skills, visualization and presentation skills, and project management and people skills.   conclusions – based upon analysis of this set of documents, a culture of assessment in libraries appears to be emerging, demonstrating a possible upward trend when contrasted with the earlier research of walter and oakleaf (2010). overall, assessment related skills and knowledge were increasingly evident across all library types and positions. suggestions for aiding the development of an emerging culture of assessment include fostering liaisons between ala divisions and library schools to persuade the schools of the need for related coursework, workshops focused on assessment-related skills, certification programs, and a proposed minor in library assessment. opening avenues for discussion between library types could enhance the growth of an assessment culture beyond academic librarianship. additional research to better understand the diffusion of assessment culture and practice into non-academic libraries is also recommended.   commentary   this study is intended to augment the earlier work of walter and oakleaf (2010), who performed an analysis of assessment-related job postings and found that “soft” skills were predominant (e.g., awareness of data needs) over “hard” skills (e.g., ability to gather and analyze data). the authors place their work in the context of numerous library job posting analyses. what is missing here is any definition of “competencies” as a concept different from “skills.” throughout, the terms are used interchangeably, proving the authors’ point that consensus is sorely needed on terminology.   lis-specific critical assessment tools developed by glynn (2006) and perryman (2014) were used to evaluate the quality of the article.   a limited number of posts were examined within an 18-month time frame. it is unclear whether the data was sufficiently representative, particularly in connection with non-academic libraries. the authors mention a lack of familiarity with public and special library practices and may have failed to include job listings that serve these communities, insufficiently representing their needs. while detailed, insufficient information is provided for replication of the searching processes used, including the rationale for search terms used to retrieve and text-mine job posting documents. one of the posting sites (ala) has a search engine that includes categories such as “other,” not just library positions. it is not made clear whether “other” was excluded from retrieval. searching is not possible at the lita job posting site, so authors must have opened each listing, however this is not mentioned. among the sites searched, the library assessment site lists only two job postings within the specified period, both from late 2012. the arl job listings provide only job titles and are not searchable by key words. the searchers would have had to go at least one and perhaps two layers in to retrieve the full posting text, but the process is not described. information about the process used would have aided clarity and served to inform future research aimed at replication of the study. in addition, there is no description of any data cleaning process or of validation methods used, including inter-rater reliability measures.   given page limitations, providing data online would have benefitted readers since the information mentioned is not sufficiently backed by statistics. as an example, the authors mention briefly that three years “experience” are required by “most” postings, but no explanation is provided about specific types of experience required. asking a question related to experience required in future research would benefit job seekers and employers. it is surprising that in this study, there is no mention of whether “most” postings required an mls degree.   the results of analysis of co-occurring terms is less convincing than the other findings, due to term ambiguity and the nature of the job postings themselves. measuring the use of specific phrases such as “program improvement” and “culture of assessment” presumes a relatively consistent vocabulary, which is mentioned as a deficit of the study by the authors.   despite its limitations, the study is a commendable effort that adds to a small body of literature tracking trends in assessment hiring in library practice. future research built on this work can continue to examine the diffusion of assessment to practice in all library types, informing our future as it continues to adapt to measuring and documenting our return on investment to stakeholders. additionally, the expanded categories derived from content analysis provide a basis for training and education that can benefit library administrators, educators, and associations.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154   perryman, c. (2014). bibcat: evaluation tool for bibliometric studies. in jotform – form builder. retrieved from http://form.jotform.us/form/42397103556153   walter, s. and oakleaf, m. (2010, oct. 27). recruiting for results: assessment skills and the academic library job market. slides presented at the arl library assessment conference. baltimore, md. retrieved from http://libraryassessment.org/bm~doc/walter_scott.pdf evidence based library and information practice commentary   the evolution of evidence based library and information practice, part ii: the broader professional purpose of eblip   jonathan d. eldredge associate professor health sciences library and informatics center university of new mexico albuquerque, new mexico, united states email: jeldredge@salud.unm.edu   received: 22 feb. 2013   accepted: 25 feb. 2013      2013 eldredge. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     part i in this series of commentaries in the december 2012 issue provided a consensus definition and description of eblip. the five steps of the eblip process consist of:   1. formulating an answerable question 2. searching for the evidence 3. critically appraising the evidence 4. making a decision and applying it 5. evaluating performance   part i reviewed how answering different types of questions raised in step one require different types of evidence. competing evidence, conceptualized primarily in terms of applied research, study design, and quality of the evidence, will guide the third step of critical appraisal. the completed eblip process finally should lead the busy practitioner to an informed decision based on the best available evidence.   part ii in this commentary series delves into the broader purpose, or function, of eblip within the library and information professions. the deceptively easy answer to this question hinges simplistically upon defining eblip as a decision making process, and that would be a technically accurate answer. this commentary explores the deeper function of eblip that relates to professional identity and practice, however. by exploring these deeper meanings we might be able to chart our journey toward nurturing and sustaining eblip in the future.   professionalism in practice   the everyday use of the noun or adjective “professional” suggests that the word has a common, widely understood meaning. we often refer to ourselves as “professionals” or evaluate others’ actions by seemingly obvious standards of “professional” behavior.   sociologists have long found the concept of “professional” to be problematic. for this reason they routinely express difficulty in arriving at a standard definition. defining the term “professional” appears to be difficult because the meaning of the term changes across time and culture, and even might be further confounded with other intermingled variables such as social status or economic monopoly power.   some sociologists have offered some fairly durable definitions that can provide some reference points for this commentary. these definitions reflect the changing times and contexts of these sociologists. freidson (1973) depicts a profession as making “claim to special esoteric competence and to concern for the quality of its work and its benefits to society, [and] obtains the exclusive right to perform a particular kind of work . . .” (p. 22). he notes that professions date back to the guilds and crafts. larson (1977) and macdonald (1995) both focus upon the exclusive right, or monopoly power of professions. beckman (1990) distills two core characteristics of the professional: autonomy and required formal training (pp. 113-138).   brante (1990) chronicles the definition from a traditional to a more modern, cynical mode. in the traditional sense, brante tells us that “professions are distinguished from other occupations by prestigious attributes such as strict ethics and integrity, a universalistic and functionally specific relation to their clients, and, above all, by employing skills based on scientific knowledge” (p. 76). cynically, brante then depicts professions as highly educated keepers of monopolized knowledge: “professions are seen as instruments, as resources by which their members can gain higher income, power, and prestige – a kind of collective egoism” (p. 76). brint (1994) lists some common features of professions: a coherent ideology, higher education as a necessary condition, autonomy, credentialing, and an adequate degree of self-governance to thwart external managerial control (p. 6). gleeson and knights (2006) document how professions are adapting to changing circumstances that tend to reduce their independence due to oversight by external authorities. they conclude that professionals are nimbly adapting to and even thriving despite some societal restrictions on their professional autonomy.   looked at another way, we can define professions in terms of their enduring core functions in society. this elemental approach allows us to define physicians as those persons who diagnose, treat, and advise patients on matters of disease and health. lawyers interpret, apply, and advocate on matters related to the law. educators assess existing knowledge or skills in learners so they can design appropriate learning experiences for these learners. librarians and other information professionals identify, organize, and make accessible authoritative information for specific user populations. these professions have served these core functions for society for the past 10, 20, 100, or more years. similarly, these professions most likely will serve these functional roles in society for the foreseeable future.   a new discipline?   could the appearance of eblip be the harbinger of a new and distinct academic discipline? could eblip be currently unfolding in ways that resemble the origins of the now well-established disciplines of chemistry, political science, and psychology, just to cite a few examples? some limited evidence exists to support this thesis. several investigators have attempted to track the emergence of new disciplines. ben-david and collins (1966) trace the arc of psychology’s formation as a discipline distinct from other disciplines. they devote special attention to its break from philosophy, generally considered to be its main discipline of origin. they discover three conditions that define this event of individuals beginning to identify themselves as subject area researchers in psychology: (a) individuals engage in research specific to psychology; (b) they do not identify themselves with any disciplines in addition to psychology; (c) they come to recognize their group identity as scientific psychologists (pp. 453-454).   they furthermore identify with a new set of research methods specific to psychology. their group numbers swell over time as more and more adherents flock to the new discipline. most eblip adherents would agree that they experience both individual and group self identification with the framework and applied research methods of eblip. they would not identify themselves apart from their parent professions of library and information practitioners, in direct contrast to the early adherents of psychology. indeed, attendees at biannual eblip conferences often introduce themselves to one another by their type of professional sector such as academic, school, public, or special librarians. they also identify themselves by their areas of functional specialty such as web design, instruction, or collection resources.   mullins (1973a) articulates four stages for the formation of a separate disciplinary identity. the first “normal stage” consists of low (or no) levels of collaboration or communication as individuals work alone in isolation. the second “network stage” witnesses more frequent collaboration and communication among individuals. publication of consensus-based published articles likely occurs in the transition to the next stage. the third “cluster stage” leads colleagues at this stage to work together at the same institution. more broadly, colleagues mutually support each other’s efforts. their views meanwhile begin to diverge from their parent discipline. importantly, these colleagues have graduate students who both study and work under their guidance. clusters can form, then dissipate, and then re-form elsewhere as the new discipline takes root. the fourth “specialty stage” consists of greater institutionalization and permanence in full time positions of faculty at academic centers. adherents establish journals in the discipline while secondary sources such as textbooks codify the discipline. the new discipline has a theoretical orientation, a group consciousness, research sites, training centers for teaching students, and the aforementioned secondary publication sources (mullins, 1973a, pp. 12-35). mullins generates additional confirmation for his four stages of discipline formation in his studies of molecular biology (1972) and ethnomethodology (1973b). mullins continues to influence investigations of disciplinary formation. just last year, for example, alcock (2012) used mullins’ work to describe the establishment of the new discipline of evolutionary medicine.   feist (2006) adapts and applies mullins’ four stages of disciplinary formation with his own three stage disciplinary formation framework. feist applies his framework to the formation of several closely related disciplines: philosophy of science, history of science, sociology of science, and psychology of science. feist outlines his three stages as: isolation, identification, and institutionalization. the first “isolation stage” resembles mullins’ first stage closely with individuals having no clear self or group identification as members of a distinct discipline. the second “identification stage” consists of colleagues recognizing their shared interests and identification with the discipline. they organize semi-regular conferences and establish their own leading disciplinary journal. leaders in the field usually establish training centers for students during this stage. the third “institutionalized stage” leads to annual conferences, one or more leading disciplinary journals, and established professional societies devoted to the discipline (pp. 8-36).   the frameworks offered by mullins (1973a) and feist (2006) closely resemble one another. both researchers outline stages and the elements within those stages that eblip also has experienced during its own development. eblip certainly had the first stages of isolation with its early advocates operating as individuals within their own regions or nations and without knowledge of one another. this commentator certainly experienced this isolation until his uk colleague bruce madge took his medical library association continuing education course on eblip in chicago in 1999. mr. madge then connected this commentator with andrew booth in the uk, who also had been working largely in isolation. booth’s first international eblip conference held in sheffield during 2001 attracted other eblip adherents from elsewhere, thereby setting in motion an advance toward what would appear to be mullins’ second network stage comprised of regular communication and collaborations. consistent with mullins’ third cluster stage some colleagues interested in eblip have worked in mutually supportive ways at the same institutions such as sheffield university, the university of north carolina, the university of new mexico, the university of alberta at edmonton, the  university of salford, and queensland university of technology in brisbane. the creation in 2006 of the internationally oriented, peer reviewed journal evidence based library and information practice certainly complies with mullins’ third cluster stage. even booth and brice’s evidence based practice for information professionals (2004), while a bit dated in parts, continues to serve as an open access textbook of sorts. connor’s book evidence based librarianship (2007) serves as less a textbook than as a series of vignettes on how eblip can be applied in various practices.   there are critical differences between eblip and a discipline. eblip does not follow mullins’ stages in the articulation of a new eblip theory distinct from the parent library and information professions. nor has eblip attracted graduate students in droves as found in mullins’ third cluster stage. most eblip practitioners seem to apply the eblip process with its accompanying skills and knowledge to their own specific sectorial contexts such as academic or school settings. furthermore, practitioners apply the eblip process in their respective specialties such as collection resources, education, or information access.   similarly, eblip has followed feist’s first two stages of isolation and identification closely. eblip has not established training centers with numerous students as in feist’s second stage, however. nor do eblip practitioners and researchers identify themselves as apart from their parent professions as predicted by feist. thus, while the stages outlined by mullins and feist seem tantalizingly close to describing the evolution of eblip, there are important and irrefutable elements that diverge from the formation of a separate discipline. at the risk of attributing motivations to eblip adherents, there appears to be no present or emerging desire among eblip researchers and practitioners at this time to distinguish themselves from their parent professions.   an invisible college?   eblip might not apply to the stages or characteristics of a new discipline since it lacks the accompanying physical world manifestations such as academic or professional school edifices. nor do eblip practitioners adhere to identities or roles apart from their parent professions. could eblip then instead be functioning more like an “invisible college” that exerts tremendous influence without presenting many physical clues of its existence to those outside of the professional community? an invisible college pertains to the “intentional cooperative work of a group of scientists who work on the same problems, not necessarily in the same place” (kantorovich, 1993, p. 190).   diana crane (1972) employed the term “invisible college” as a way of explaining the difficult-to-decipher social interconnections of scientific researchers involved in instances of accelerating production of knowledge as measured by dramatic surges in the volume of publications. garfield, sher, and torpie (1964) along with de solla price and beaver (1966) already had begun to observe invisible colleges in action. crane studied two specific invisible colleges in great depth and built upon others’ work to illuminate this phenomenon otherwise unnoticed to others outside those in a group of researchers. crane discovered that these social networks thrived on publications so that “not only can a scientist be influenced by publications written by authors whom he has never met, but he can also receive information second-hand through conversation or correspondence with third parties” (pp. 13-14). as the phrase “invisible college” might suggest, crane noted that “there is no formal leadership in a social circle although there are usually central figures” (p. 14). even when they have never met certain members of their invisible college, the members nevertheless know of and about one another. this “social interaction facilitates the diffusion of ideas that in turn makes possible cumulative growth of knowledge in a research area” (p. 26). crane tracks the historic life cycles of some invisible colleges that grew so exponentially that she compared their growth to a contagion. after some time, crane notes that collective interest tapered, and then the members of invisible colleges acquired new interests and soon joined other invisible colleges to pursue their new interests. interest in invisible colleges continues to thrive, particularly with the application of new information technology (howard, 2011).   eblip strongly resembled an invisible college during its first five or so formative years. once the eblip process had been discussed and codified, however, the invisible college dispersed as most of its members joined other invisible colleges. the numerous new invisible colleges now are applying the eblip process and the associated skills and knowledge in varied ways to advance our profession. when this commentator reflected upon his own membership in contemporary invisible colleges he enumerated his membership in at least five such social networks. undoubtedly, many other eblip readers will be able to link their own research interests to membership in at least a few invisible colleges. eblip as a group activity comprised of practitioners who read about eblip and attend eblip conferences has grown too large and has become too diverse to be described as a single invisible college.   a new paradigm?   could eblip be a new paradigm for library and information practice? several authors in this journal and elsewhere have described eblip as a “paradigm” within the past few years. carol gordon in her monumental two part series of articles has elaborated most extensively upon eblip as a paradigm in her own theory building in regards to school libraries engaged in “evidence based information literacy instruction” (2009a, 2009b). gordon conceptualizes eblip as the dominant paradigm that “serves a social and cultural purpose in molding a culture of inquiry for information literacy instruction” (2009a, p. 69). she adds that eblip facilitates the inclusion of evidence in the cycle of improvement within teaching (2009a, p. 69). later she describes eblip as the paradigm that provides the “sets of beliefs and values” for her theory-building to occur (2009a, p. 73). in her second article gordon describes eblip as a paradigm that “consists of the beliefs, assumptions, and values and techniques accepted by a community of practitioners” (2009b, p. 23). gordon then introduces the paradigm-related concepts of anomaly, criteria for selecting questions, and puzzle solving. gordon’s focus remains fixed ultimately on the use of the eblip paradigm simply as a platform for her own theory of library literacy instruction. gordon does not focus on the durability of using the concept of a paradigm to explain the purposes of eblip. thus, gordon does not elaborate upon eblip as a paradigm itself.   andrew booth and anne brice (2007) had previously described eblip as a paradigm in their tracing the development of eblip. yet, their focus also turns out to be elsewhere. booth and brice use the term “paradigm” instead as a reference point in their analysis of how eblip has changed during the years 2001-2007. they assert at the outset that eblip serves the role as a paradigm. booth and brice seem to assume that eblip readers already understand what they mean by their use of the term “paradigm.” given the great many eblip readers who are well-versed in scientific research, their assumption has some basis. their unreferenced and unexplained assertion about an eblip paradigm, while intriguing, still leaves unaddressed the central question of this commentary.   thomas kuhn in his classic book the structure of scientific revolutions (1970) popularized the use of the term “paradigm” with his description of how scientists conduct their research. kuhn devotes more than 200 pages in his book to defining and describing the concept of a “paradigm” in great detail. the following summary only touches upon those aspects of kuhn’s work most crucial to answering the question as to whether eblip represents a paradigm.   kuhn (1970) notes that in everyday language paradigm refers to “an accepted model or pattern” (p. 23). the oxford english dictionary (oed) (paradigm, 2005) traces the word paradigm from the year 1483 forward. the oed defines a paradigm in contemporary terms as “a pattern or model, an exemplar; (also) a typical instance of something, an example.” in contrast, kuhn asserts that scientists recognize a paradigm as consisting of those “universally recognized scientific achievements that for a time provide model problems and solutions to a community of practitioners” (p. viii) to guide their future inquiry. within a group of scientists a paradigm can represent “the entire constellation of beliefs, values, techniques, and so on shared by the members of a given community” (p. 175). the concrete models or examples (“exemplars”) of solutions to those recognized scientific problems occupy specific regions of the paradigm. scientists adhere to a paradigm based on the strength of its representation of reality. incidentally, scientific research can exist prior to the establishment of a paradigm in a condition kuhn depicts as “early fact-gathering” and as a “nearly random activity” (p. 15). kuhn seems at this juncture to be referring to activities such as john stuart mill’s inductive exercises (wilson, 2012). despite this proto-paradigmatic possibility, kuhn asserts that a paradigm offers a far stronger framework for identifying and solving problems compared to these pre-paradigmatic conditions.   the everyday (i.e., non-scientific) use of the term paradigm defined by kuhn (1970) as “an accepted model or pattern” (p. 23) does seem to fit eblip closely. the vast majority of readers of this journal and attendees at international eblip conferences undoubtedly would agree that eblip serves as a sequential process (or “model” or “pattern”) for reliably reaching informed decisions. the first installment in this commentary attempted to outline the other consensus-based characteristic features of eblip.   kuhn (1970) devotes his book instead to describing a scientific research context in his use of the word paradigm. gordon (2009a, 2009b) does cite and quote kuhn repeatedly, so her definition most likely aligns with kuhn. the deliberate use of the term paradigm employed by booth and brice (2007) suggests that they most definitely mean it in its more scientific, kuhnian interpretation despite the fact they never reference kuhn. otherwise, booth and brice likely would have used instead synonyms such as “framework,” or “model,” or “pattern” with great regularity when referring to eblip. as already noted, many in the eblip readership are familiar enough with scientific methods to an extent that they at least recognize the kuhnian connection.   kuhn (1970) clearly displays his reluctance to depict any activity outside of the physical sciences as influenced by paradigms. virtually all of his examples stem from the history of the physical sciences, perhaps because of his background as a theoretical physicist. at several junctures in his book he refuses even to accept that social scientists might employ paradigms. beyond those specific topical boundaries he states “though scientific development may resemble that in other fields more closely than has often been supposed, it is also strikingly different” (p. 209).   one might protest that a single philosopher or historian’s restriction of the term “paradigm” to the physical sciences should not be sufficient reason to restrict the concept to only the physical sciences. indeed, the pre-paradigm state described by kuhn does seem to resemble how applied research in librarianship fitfully lurched forward prior to the establishment of eblip. and, eblip seems to possess some of the same features of a paradigm as in the realm of the physical sciences. many in eblip seem to share the same values of scientific rigor, an almost overbearing skepticism, transparency, and the recognized superiority of some forms of evidence over others. the burden of proof does seem to fall on someone within the ranks of library or information practice to identify 20-30 examples that illustrate how eblip resembles at least one paradigm in the physical sciences. until such an exposition appears, we have to reserve judgment on categorizing eblip as a kuhnian paradigm.   this commentary might appear to leave the reader empty handed, as it suggests that eblip fits neither the description nor the stages of an emerging subject discipline. eblip might have been an invisible college during its early formative and codification years, but has since dispersed into numerous other invisible colleges. eblip does resemble a paradigm in the everyday sense of the word. whether eblip serves as a paradigm in the kuhnian sense bears closer examination, although this would require an ambitious intellectual undertaking to address adequately. at least this commentary has discounted some plausible purposes – or functions – of eblip within our profession. part iii of this commentary will discuss, among other explanatory pursuits, whether or not eblip represents a movement within our professions.     references   alcock, j. (2012). emergence of evolutionary medicine: publication trends from 1991-2010. journal of evolutionary medicine, 1(1): 1-12. doi:10.4303/jem/235572   beckman, s. (1990). professionalization: borderline authority and autonomy in work. in m. burrage & r. torstendahl (eds.), professions in theory and history: rethinking the study of the professions. (pp. 115-138). newbury park, ca: sage publications.   ben-david, j. & collins, r. (1966). social factors in the origins of a new science: the case of psychology. american sociological review, 31(4): 451-465.   booth, a. & brice, a. (eds.) (2004). evidence-based practice for information professionals. london: facet publishing. retrieved 28 feb. 2013 from http://ebliptext.pbworks.com/f/booth+%26+brice+2004+ebp+for+info+professionals+-+a+handbook.pdf    booth, a. & brice, a. (2007). “prediction is difficult, especially the future”: a progress report. evidence based library and information practice, 2(1): 89-106. retrieved 28 feb. 2013 from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/99/242   brante, t. (1990). professional types as a strategy of analysis. in m. burrage & r. torstendahl (eds.), professions in theory and history: rethinking the study of the professions. (pp. 73-85). newbury park, ca: sage publications.   brint, s. (1994). in an age of experts: the changing role of professionals in politics and public life. princeton, nj: princeton university press.   connor, e. (2007). (ed.) evidence-based librarianship: case studies and active learning exercises. oxford: chandos publishing.   crane, d. (1972). invisible colleges: diffusion of knowledge in scientific communities. chicago, il: university of chicago press.   de solla price, d. j. & beaver, d. (1966). collaboration in an invisible college. american psychologist, 21(11): 1011-1018.   feist, g. j. (2006). the psychology of science and the origins of the scientific mind. new haven, ct: yale university press.   freidson, e. (1973). professions and the occupational principle. in e. freidson (ed.), the professions and their prospects. (pp. 19-38). beverly hills, ca: sage publications.   garfield, e., sher, i. h. & torpie, r. j. (1964). the use of citation data in writing the history of science. philadelphia, pa: institute for scientific information. retrieved 21 december 2012 from http://www.garfield.library.upenn.edu/papers/useofcitdatawritinghistofsci.pdf   gleeson, d. & knights, d. (2006). challenging dualism: public professionalism in ‘troubled’ times. sociology, 40(2): 277-295.   gordon, c. a. (2009a). an emerging theory for evidence based information literacy instruction in school libraries, part 1: building a foundation. evidence based library and information practice, 4(2), 56-77. retrieved 28 feb. 2013 from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/5614/5320   gordon, c. a. (2009b). an emerging theory for evidence based information literacy instruction in school libraries, part 2: building a culture of inquiry. evidence based library and information practice, 4(3), 19-45. retrieved 28 feb. 2013 from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/6449/5559   howard, j. (2011, sept. 11). citation by citation, new maps chart hot research and scholarship’s hidden terrains. chronicle of higher education. retrieved 1 mar. 2013 from http://chronicle.com/article/maps-of-citations-uncover-new/128938/   kantorovich, a. (1993). scientific discovery: logic and tinkering. albany, ny: state university of new york press.   kuhn, t. s. (1970). the structure of scientific revolutions. 2nd ed. chicago: university of chicago press.   larson, m. s. (1977). the rise of professionalism: a sociological analysis. berkeley, ca: university of california press.   macdonald, k. m. (1995). the sociology of the professions. thousand oaks, ca: sage publications.   mullins, n. c. (1972). the development of a scientific specialty: the phage group and the origins of molecular biology. minerva, 10(1): 51-82. retrieved 1 mar. 2013 from http://link.springer.com/article/10.1007%2fbf01881390?li=true   mullins, n. c. (1973a). theories and theory groups in contemporary american sociology. new york: harper & row.    mullins, n. c. (1973b). the development of specialties in social science: the case of ethnomethodology. science studies, 3(3): 245-273. retrieved 5 mar. 2013 from http://www.jstor.org/stable/284495 doi:10.1177/030631277300300302   paradigm. (2005) oxford english dictionary (3rd ed.). retrieved 1 mar. 2013 from http://www.oed.com/view/entry/137329?redirectedfrom=paradigm#eid        wilson, f. (2012). john stuart mill. stanford encyclopedia of philosophy. retrieved 1 mar. 2013 from http://plato.stanford.edu/entries/mill/   evidence based library and information practice evidence based library and information practice 2010, 5.3 59 evidence based library and information practice evidence summary public libraries can play an important role in the aftermath of a natural disaster a review of: welsh, t. s. & higgins, s. e. (2009). public libraries post-hurricane katrina: a pilot study. library review, 58(9), 652-659. reviewed by: virginia wilson client services librarian murray library, university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca received: 18 may 2010 accepted: 7 july 2010 2010 wilson. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. objective – this paper analyzes hurricane katrina-related narratives to document the challenges faced by public libraries after the disaster and the disaster-relief services these libraries provided. design – a qualitative thematic analysis of narratives obtained by convenience sampling. setting – narratives were collected and analyzed in 2005 and 2006 across the gulf coast area of the united states. subjects – seventy-two library and information science students enrolled in the university of southern mississippi’s school of library and information science. many worked in local libraries. methods – in this pilot study, students volunteered to participate in a confidential process that involved telling their stories of their post-hurricane katrina experiences. data was collected in a natural setting (the libraries in which the students worked), and inductive reasoning was used to build themes based on these research questions: what post-disaster problems related to public libraries were noted in the students’ narratives? what post-disaster public library services were noted in the narratives? nvivo7 qualitative analysis software was used to analyze and code the narratives. passages related to public libraries were coded by library location and student. these passages were analyzed for themes related to postdisaster challenges and disaster-recovery services pertaining to public libraries. mailto:virginia.wilson@usask.ca evidence based library and information practice 2010, 5.3 60 main results – ten of the 72 narratives contained passages related to public libraries. the libraries included four in alabama, one in louisiana, and five in mississippi. results related to the first research question (what post-disaster problems related to public libraries were noted in the students’ narrative?) were physical damage to the building, from light damage to total destruction (reported in 8 or 80% of the students’ narratives), and inundation by refugees, evacuees, and relief workers (reported in 8 or 80% of the narratives). results pertaining to the second research question (what post-disaster public library services were noted in the narratives?) included providing information for things such as providing information via the use of computers and the filling out of the federal emergency management agency and red cross aid forms (6 or 60% of the narratives included this), listening and providing comfort (5 or 50% of the narratives), and volunteering and donating, both from others and of the students’ own time, money, or materials (noted by 5 or 50% of the narratives). conclusion – the researchers concluded that while public libraries suffered devastation during the hurricane, after the hurricane, those libraries that could open provided essential services to people in need. these services included providing access to computers and access to information via computers, aid in filling out necessary relief aid forms, listening and providing comfort, and volunteering time, money, and materials. the public library clearly played a role in both providing information and facilitating communication. documenting such contributions serves to illustrate the value of public libraries, especially in a post-disaster setting, and helps to demonstrate the value of public libraries in their communities. commentary this pilot study serves to educate readers on how a public library can participate in relief efforts, and it illustrates how people in need may turn to the public library for assistance. although the students who responded to the call for participation were volunteers, and thus self-selected rather than part of a random sample, the researchers hold the responses as “valid and reliable” by holding them up to fisher’s narrative theory, which states that “humans are rational beings and faithful to their own experiences” (p. 656). the researchers do acknowledge that the results of this study are not generalizable but can act as a starting point for further studies, although they give no indication of how they might take this work forward. this study provides interesting additional information. for example, the introduction contains details of the devastation experienced by gulf coast communities during and in the aftermath of hurricane katrina. the authors provide a glossary, defining such terms as narrative and pilot study, so that the reader has a clear understanding of how these terms are used in this study. the methods section discusses qualitative research in theoretical terms, but fails to fully establish exactly how the researchers conducted their research. they state that “narrative texts” were obtained from the 72 student volunteers. it is unclear if the students wrote down their experiences and handed them in, or if the term “text” refers to transcriptions of personal interviews. was a voice recorder used? did each researcher interview half the participants? were notes taken? one can assume that because these details are missing, the students indeed wrote down their experiences. still, it is not specified. as well, the researchers look specifically at two research questions, while only 10 out of the 72 responses contained passages that pertain directly to those questions. it is not stated what else the students were asked to relate about their own post-hurricane experiences. the presentation of the results is segmented. the authors give an abbreviated report of the results of the two pertinent research questions but do not provide any analysis of the results. evidence based library and information practice 2010, 5.3 61 additionally, after the conclusion and recommendations portion of the article, there is a section entitled “selected narrative excerpts.” there, the authors provide verbatim selections from the narratives, but again, do not provide analysis and fail to integrate the quotations with the earlier results. if the quotations are not meant to be part of the results, the authors should have made it clear as to what the quotations were meant to signify. this research may be useful for advocacy purposes. being able to illustrate the role of the public library in many different situations is valuable when having to speak to decision makers about funding. this research demonstrates how libraries can affect quality of life and how, even when taken to the extreme, community needs are met. the authors recommend further and more comprehensive qualitative studies to be undertaken to document the challenges faced by libraries and the role of libraries particularly in the aftermath of a disaster. conference paper   longitudinal assessment of “user-driven” library commons spaces   robert fox dean, university libraries university of louisville louisville, kentucky, united states of america email: bob.fox@louisville.edu   ameet doshi head, user experience department georgia institute of technology library atlanta, georgia, united states of america email: ameet.doshi@gatech.edu      2013 fox and doshi. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to conduct a longitudinal assessment of library spaces at the georgia tech library and to determine the satisfaction of students with the most recent commons renovation. the library has completed three commons area renovations. the library west commons (lwc) opened in 2002 with an individual productivity lab, multimedia studio, and presentation rehearsal studio, while the library east commons (lec) and the 2nd floor west commons (2 west) opened in 2006 and 2009, respectively, with flexible, user-centered environments designed to promote collaborative activities. this analysis focuses on the renovated collaborative spaces, while also investigating and commenting on how renovation impacts usage of other spaces in the library.   methods – usage of all library spaces was measured during one-week periods in fall 2008 and spring 2010. observations were made of each student floor in the library at four times during the day; measures included space utilization by groups, group sizes, and laptop utilization. in addition, a qualitative instrument was administered during spring 2010 to 103 students using the 2 west commons space to confirm whether the renovation met their needs.   results – overall, there was a 64.5% increase in group utilization of the library from 2008 to 2010, driven primarily by the 2 west renovation. the greatest concentration of group usage was in the lec and 2 west, though the number of groups using the lec declined. laptop use in the 2 west commons more than doubled (33.6% to 70.5%), and laptop use in the entire library increased from 40.5% to 49.0%. in the qualitative survey, scores ranged between 4.0 and 5.0 on a 5-point scale for items regarding four design themes for the 2 west renovation: power/data, lighting, aesthetics, and the creation of a “defined yet open” space.   conclusion – findings suggest that the 2 west commons is attracting more students and groups following its renovation, that it is attracting students and groups away from the previously renovated lec, and that overall usage of the library increased subsequent to the 2 west renovation.   introduction   the georgia tech library serves over 26,000 students, staff, and faculty. the main physical facility consists of two separate libraries (east and west) joined by a bridge. this facility is open 135 hours during the week closing only friday and saturday nights. georgia tech’s most recent libqual+® survey administration, conducted in 2010, demonstrated heavy use of the facility by both undergraduate and graduate students with 88% of undergraduates and 86% of graduate students indicating at least monthly use (cook et al., 2010; see figure 1). furthermore, over 60% of undergraduates indicated daily or weekly use of the facility. the fact that over 80% of graduate students indicate regular use of the library facility was particularly surprising, leading us to conclude that “library as place” remains a vibrant part of student life across user groups.   the georgia tech library has completed three commons area renovations. the first, the library west commons (lwc) opened in 2002 with a large individual productivity lab, a multimedia studio, and a presentation rehearsal studio. building on the success of the lwc, the library embarked on planning for the library east commons (lec) which opened in 2006. designed to promote collaborative activities in a flexible, user-centered environment, this renovation was particularly successful due to the depth of user feedback gathered throughout the design process. the third and most recent renovation, the 2nd floor west commons (“2 west”), was completed in fall 2009. the 2 west project continued and increased the level of user design input to the extent that it is often described as a “student designed” library space. fox and stuart (2009) provide more comprehensive information on the planning and design of these spaces.   figure 1 georgia tech library facility use among students (libqual+® 2010)   while the 2 west project shares a design component with the lec, namely that of providing user-informed collaborative spaces, it differs in two significant ways. first, 2 west provides more open spaces for collaborative activities than the lec, promoting greater adaptability for group size variations and, in general, a sense of more flexibility. second, other than four quick-use walk-up terminals, 2 west does not provide library-owned desktop computers. in 2007, georgia tech updated its student computer ownership policy to require that all incoming first-year students own a personal laptop computer. this requirement is significant because it helped drive the decision not to include desktop computing spaces in 2 west; but instead make the space more amenable to personal laptop use through abundant power outlets and wired data ports, an enhanced wireless network, wireless printing capability, and 42-inch monitors that can attach to multiple laptops simultaneously. rather than continuing to create expensive, financially unsustainable and less flexible “computer lab” commons with stand-alone computer terminals, the 2 west renovation embraces the laptop and mobile-device oriented culture of the larger institution.    purpose of the study   the purpose of this study was to conduct a longitudinal assessment of library spaces at the georgia tech library and to determine the satisfaction of students with the most recent commons renovation. our analysis focuses on the renovated collaborative spaces, while also investigating and commenting on how renovation impacts usage of other spaces in the library. by assessing the impact of this most recent renovation over time, we hope to provide justification for future renovations and inform these projects with the most appropriate user-sensitive design.   this longitudinal study seeks to answer the following key questions: to what degree does renovation impact utilization of the renovated space? how does usage of renovated spaces change over time, particularly in light of subsequent renovations to library spaces with similar functions? and, what effect did these renovations have on overall utilization of library spaces? based on previous observations and gate count data, we anticipated that the creation of the new 2 west commons would substantially increase utilization of that particular space, while also leading to an overall increase in library usage. we also expected to find that the increased utilization of 2 west would come at some expense to usage of the lec. both of these commons areas provide collaborative spaces; but the lec was often very crowded prior to 2 west construction, and we expected that the renovated 2 west would provide a “relief valve” for the student pressure to increase collaborative space within the library. as the final part of our study, we also seek to determine how satisfied students are with the renovation of the 2 west commons based on the original user-centered design criteria for that space.   although there have been numerous commentaries and research articles written about “library as place” and commons spaces, no published work examines the effect of renovation on library space utilization over time. in a landmark study, whitmire (2001) examined the library-use patterns of over 1,000 undergraduate students over their first three years in college. although whitmire’s research is very useful in providing a holistic understanding of how and why undergraduates use academic library resources, services and facilities, it does not specifically investigate the impact that renovation has on facilities usage. scott bennett, writing in the 2005 council on library and information resources (clir) report, library as place: rethinking roles, rethinking space, notes the importance of fostering social components of learning by creating a sense of community among students. we have found at georgia tech that creating a sense of community ownership empowers users to modify and govern the space based on their evolving needs. a major consideration for the 2 west renovation was to create a space where students felt comfortable moving furniture around to meet their needs and expectations. the success of this user participation dimension was assessed using a qualitative questionnaire, which supplements the space utilization data. potthoff, weis, montanelli, and murbach (2000) illustrate a behavioral sciences approach to evaluating library spaces called the role repertory grid procedure. while there is some overlap between our qualitative instrument and the comprehensive approach adopted by potthoff et al., our instrument focuses on evaluating four specific themes that emerged from student focus groups involved with the initial co-design phase for 2 west. however, the role repertory grid procedure may be useful for future iterations of the qualitative part of this study. somerville and collins (2008) write about the importance of collaborative, user-centered design principles in the planning process for the renovation of library learning spaces. though they discuss important components of user-driven library commons renovation, their work does not fill the research gap regarding longitudinal assessment of these spaces. furthermore, there have not been any formal studies to investigate the internal impact that renovation has on other spaces within the library.   methodology   the methodology for this longitudinal study involves both quantitative and qualitative components. initial observations were first performed during fall 2008 by measuring usage of all library spaces for one week. observations included counts of patrons using each space or zone at four times during the day. the number of groups in each space was recorded. some, but not all, of the 2008 observations also noted group sizes. one of the 2008 observations collected data on laptop utilization.   a second set of comparative observational data was collected during spring 2010 to more definitively determine the longitudinal impact of opening a new space on overall building usage, group usage versus individual usage by zone, and laptop utilization, after the opening of the 2 west commons. as in 2008, observations were made of each student floor in the library at four times during the day. these times were labeled as morning, afternoon, evening, and night and were taken at approximately the same time each day monday through thursday. the observations for fridays included only three data points as the library closes at 6:00 p.m. on fridays during most of the semester. for each count, the observer noted the number of individuals and groups in the zone, the sizes of each group, and the number of laptops being used. the 2010 observations were timed to coincide with the same period of the semester as when the 2008 observations were conducted.   additionally, feedback gathered from students during the initial design phase for 2 west identified specific areas for improvement. these areas included a desire for improved power and data access, improved lighting and aesthetics, and flexible spaces that could be “student-owned.” a qualitative instrument was administered during spring 2010 to students using the 2 west commons space to confirm if the renovation met their needs. this survey included the following questions:   on a scale of 1-5, how well does the power in this space meet your needs? why? on a scale of 1-5, how well does the lighting of this space meet your needs? why? on a scale of 1-5, how well do the aesthetics, furniture and ambience of this space meet your needs? why? during the initial planning for the design of this space, students noted a desire for a “defined yet open” space. they described a space that included well-defined areas for group study, while not limiting the option to move furniture around for their individual needs. on a scale of 1-5, how successful is this space in striking this balance of “defined yet open”? why?   in addition to the questions outlined above, the survey also included an open-ended question asking for additional comments or suggestions for the library. these qualitative comments are useful to describe the “lived experience” of the students within the 2 west space. they also provided context for the quantitative statistics and provided a better overall picture of how and why our users interact with the newly renovated areas in the library.   findings and observations   table 1 shows the percentage change in the numbers of individuals using the lec and 2 west, and the change in the total usage of all zones (floors 1 through 6), from the 2008 and 2010 observational data. the 2 west space saw higher usage for each 2010 data collection when compared with the same period in the 2008 observation. total usage of 2 west increased 94.0% between the 2008 and 2010 observations.   for the lec, some 2010 observations revealed higher usage while others declined when compared to 2008. overall usage of the lec during the observations increased only 2.7% thus lagging considerably behind the increase of 2 west. the data suggests that on the busiest days (monday through thursday), 2 west is attracting students away from the lec space. total usage of library spaces on all floors during the observations increased 25.1%, considerably higher than the increase in georgia tech student population between 2008 and 2010.   space utilization by groups   table 2 provides the total number of groups observed in each space and the percentage change for group utilization of each area between 2008 and 2010.   by far, the greatest concentration of group usage is in the lec and 2 west, as these are the only areas of the library that have been renovated specifically to provide collaborative space. these spaces are also located on the “talking allowed” floors rather than floors dedicated for quiet study. the increase of collaborative use of 2 west both in terms of raw numbers and percentage change is quite high, reflecting the popularity of this freshly renovated space. while still high, the number of groups using the lec declined. interestingly, the lec was the only space to experience a decline in number of groups from 2008 to 2010, though it should be noted that the percentage change in other areas is based on smaller counts as these spaces are primarily dedicated to quiet study. overall, the data illustrates that the 65.4% increase in group utilization of the library from 2008 to 2010 is driven primarily by the 2 west renovation.   group sizes while the 2008 observations recorded the number of groups using each zone, only eight of the 2008 observations noted the sizes of each group. these eight observations were the evening and night observations, monday through thursday. group sizes were noted during each of the 2010 observations, but we can only make a direct comparison between the 2008 and 2010 data for the evening and weekend observations conducted monday through thursday. the number of group members in 2 west increased 67.4%; but as the total number of groups more than doubled, the average group size decreased from 3.4 to 2.8 members. the number of group members in the lec declined 1.8% with group size declining slightly from 2.7 to 2.5 members. with the exception of the lec and one other zone, all observed spaces recorded increases in the total number of group members between the 2008 and 2010 observations, while average group sizes fluctuated with some zones experiencing increases and some decreases.   when reviewing all 19 observations made during 2010, including the morning and afternoon times excluded from the comparisons in the discussion above, variations in group sizes by zone seem to be minimal. average group sizes by zone ranged from 2.2 to 2.9 with no apparent pattern by size of the space, floor level (i.e., floors closer or further away from the main entrance), whether a quiet or talking space, or whether the space had been renovated. one variable that may have provided some impact on group size is the availability in certain zones of tables somewhat larger than in others, or specifically in the renovated 2 west area, of small tables that can easily be moved together to form larger groups.   table 1 lec, 2 west and total attendance % change 2008-2010 monday 1st floor east commons (lec) 2nd floor west commons (2 west) total all library zones % change (2008-2010) morning -18.1% +11.5% -9.8% afternoon +13.2% +134.5% +66.1% evening -5.9% +55.6% +25.8% night -17.6% +46.2% +12.8%         tuesday lec 2 west total library morning -16.7% +29.7% +10.1% afternoon -24.5% +87.7% +18.4% evening +10.6% +43.6% +38.2% night +17.5% +57.3% +65.0%         wednesday lec 2 west total library morning +24.4% +79.3% +9.3% afternoon -10.6% +103.7% +7.9% evening +16.1% +52.6% +28.7% night -5.4% +72.4% +41.7%         thursday lec 2 west total library morning -20.0% +260.0% +31.7% afternoon -4.0% +130.9% +45.9% evening -35.1% +25.3% -1.4% night -45.7% +15.7% -12.1%         friday lec 2 west total library morning +107.1% +246.2% +44.5% afternoon +22.2% +45.5% +18.0% evening +42.9% +286.7% +35.6% overall usage % change (2008-2010) +2.7% +94.0% +25.1%   laptop utilization table 3 illustrates how laptop utilization has changed since the 2 west renovation. the number of students utilizing laptop computers was noted during each observation in 2010.   table 2 space utilization by groups (% change 2008-2010) floor total # of groups (2008) total # of groups (2010) % change (2008-2010) 1 west (lwc) 49 74 +51.0% 1 east (lec) 237 218 -8.7% 2 west 207 450 +117.4% 2 east 36 46 +27.8% 3 west 8 23 +187.5% 3 east 36 37 +2.8% 4 west 30 63 +110.0% 4 east 16 22 +37.5% 5 18 31 +72.2% 6 16 25 +56.2% group utilization total % change, 2008-2010 +65.4%   table 3 laptop % utilization (2008-2010) floor 2008 2010 % change (2008-2010) 1 west (lwc) 0.0% 6.9% +6.9% 1 east (lec) 27.0% 35.6% +8.6% 2 west 33.6% 70.5% +36.9% 2 east 68.2% 52.2% -16% 3 west 80.0% 62.0% -18% 3 east 21.4% 68.2% +46.8% 4 west 100.0% 68.9% -31.1% 4 east 75.0% 62.4% -12.6% 5 88.0% 65.7% -22.3% 6 93.3% 72.6% -20.7% total laptop % utilization 40.5% 49.0% +8.5%   in 2010, laptop utilization varied significantly based on zone, but the lowest rates were observed in the lwc and lec with rates of 6.9% and 35.6% respectively. this result was expected for these areas since most seating areas in the lwc and about half those in the lec are outfitted with desktop computers. other spaces in the library saw laptop utilization rates from just over 50% to just over 70% in the 2 west commons, which is specifically designed to support laptops. total laptop utilization for all library spaces during the study was 49.0%. as there was only one observation in 2008 that noted laptop usage, it is not possible to fully report trends in this area. still, it can be noted that from the 2008 observation to those in 2010, laptop use in the 2 west commons more than doubled (33.6% to 70.5%) and that laptop use in the entire library increased from 40.5% to 49.0%. both of these results would be expected given the laptop-friendly renovation to 2 west and the a new freshman class subject to the institutional laptop requirement.   2 west qualitative survey   also significant are the results of the survey regarding the four core design themes for the 2 west renovation: power/data, lighting, aesthetics, and the creation of a “defined yet open” space. for this survey, the scale was centered so that a response of “3” indicated satisfaction with the renovation efforts for that theme. a “4” indicated that the renovation more than met the desired outcome for that space while a “5” indicated that the student felt the renovation effort had been great. as noted in table 4 below, over 100 students using the 2 west commons space participated in the survey. with all theme scores ranging between 4.0 and 5.0, it appears that students are quite satisfied with each aspect of the renovation.   convenient and ample power and data access was a primary concern because the 2 west renovation would not include desktop computers, but rather be marketed as a “laptop friendly” commons space. specific comments from the qualitative survey reflect student satisfaction with regards to power and data access:   “it’s real easy to plug in almost anywhere.” “not having to search/fight for outlets makes the library much easier to study in.” “plenty of power outlets scattered throughout.” “points are well placed.”   table 4 qualitative survey results themes average score (n=103) power/data 4.28 lighting 4.49 aesthetics, furniture, and ambience 4.44 "defined yet open" 4.43 total average 4.41   scale: 5 = great   4 = more than adequate   3 = meets needs   2 = not very well   1= totally inadequate   prior to the renovation, lighting levels in 2 west were described as unbalanced and generally harsh. the survey results show that students reacted positively to the refreshed lighting for the space:   “perfect for computers and work.”  “outside light and inside light work well together to create an aesthetically pleasing environment.”  “love the bright lights! doesn't feel like a prison like before.” “i feel like the lighting is great for reading, studying, etc. lighting is subtle as to not distract from work but sufficient enough to function. it almost seems that there is a lot more of natural lighting.”   we asked students how well the “aesthetics, furniture, and ambience” of this space met their needs. their scores and comments reflect an enthusiasm for the comfortable furniture, contemporary look and feel, and practical aesthetics for enhancing collaborative activities:   “oh my god, it is the perfect studying chair ever.” “effective for both group studying and studying alone.” “good comfortable chairs, nice tables, good group work atmosphere.” “furniture is nicer; doesn't have the feel of a dungeon.” “comfortable yet can focus.” “love the new set up, especially white boards. booths are comfortable.” “the modern and minimalist style helps me to concentrate on my work in a relaxed environment.” “very nice contemporary feel.” “simply much more appealing than before.”   the final theme we assessed was the flexibility of the space. during the co-design phase, students described a space that included well-defined areas for group study, while not limiting the option to move furniture around for their individual needs. the comments from the 2010 survey demonstrate that the space allows for such flexibility and openness:   “good, can easily move furniture to meet group needs.” “the present environment is one of the best places to study on campus due to how easily it can adapt to a student's needs.” “the objective is well met. the central space and other long tables are good for group studies, and the corners are quiet enough for individual studiers.” “you have your own space, but can still not be isolated from the rest of the library.” “this really is the perfect place to do group work, because there is so much freedom to move around and use various resources.” “the white board areas are great for group study, but the option remains open to rearrange furniture to an extent to accommodate larger groups of people.” “the white boards are a wonderful feature and it helps that most of the furniture is lightweight and moveable. it strikes a great balance.” “the spaces are less cubicle-like and are open. the rolling chairs make it easy to add more people to a group.”   finally, the survey provided an opportunity to gather feedback about improving services overall, and included an open-ended question about how to improve the library, generally. many students indicated a shortage of dry erase markers and erasers. this information was communicated to the commons coordinator who increased supplies during final exams. other students asked for improved power access in other library spaces. a power audit was conducted by the library facilities manager, and though it is not currently feasible to overhaul the entire electrical grid for the building, broken or non-functioning outlets can be repaired. a very common request was to “keep renovating up to the next floors.” although the present budget climate will not allow for an immediate comprehensive renovation, the quantitative and qualitative data suggests that adopting a user-driven approach for future facilities refreshments correlates well with student satisfaction.   conclusions   analysis of the longitudinal data collected suggests the following:   the 2 west commons is attracting more students and groups subsequent to its renovation. the 2 west commons is attracting students and groups away from the previously renovated lec. overall usage of the library increased subsequent to the 2 west renovation. the need for collaborative spaces in the library continues to grow. even with the most recent renovation of 2 west, the number of groups and group members continues to increase in other areas of the library including those designated as quiet space. laptop utilization is up somewhat for the whole library and significantly for 2 west.   data on student usage indicates that the most recently renovated spaces (2 west) are successful. it appears that the most recent renovation increased use of that commons space, as well as overall usage of the library. results of the qualitative survey regarding the 2 west renovation indicate a very high degree of satisfaction with the project results across each of the core design themes. this level of satisfaction is most likely attributable to the intensive user engagement process undertaken prior to renovation. the high scores on the survey corroborate the large increase observed in usage data for the 2 west commons space. the 2010 data also support the concept that students will embrace a laptop-friendly commons renovation and that all commons renovations do not require library-supplied desktop computers. future iterations of this longitudinal study should prove illuminating and practical for space planning and budgeting. in order to conduct a successful inquiry, we have found it useful to adopt the following practices to help ensure smooth data collection and analysis. as with all longitudinal research, using a consistent survey instrument and communicating data gathering guidelines is important to maintain integrity and consistency of results. in addition, it is vital to recognize the need to have knowledge transfer mechanisms in place to deal with changes in personnel. finally, a method for archiving raw data and results, preferably in an institutional repository or other centralized digital warehouse, can make the data analysis process more efficient and robust.   this study is unique because it assesses how renovating spaces impacts overall usage of the library over time. based on our literature review, this type of longitudinal study of library space utilization has not yet been published. this research also illustrates how renovating one space has the potential to attract users away from other library spaces. the data suggests that user-centered refreshment or renovation of library commons spaces can have a profound impact on utilization, and that this utilization can increase with the addition of financially sustainable laptop friendly spaces and not just the addition of commons spaces providing desktop computers. results from this study will be used to guide and inform future renovations at the georgia tech library. additionally, future observations may be able to more fully assess changes in the utilization of laptop computers. although this study concerns the georgia tech library, our experience may provide a useful roadmap for other institutions as they seek to transform spaces or assess existing ones.     references   bennett, s. (2005). righting the balance. in library as place: rethinking roles, rethinking space (pp. 10-24). washington, dc: council on library and information resources.   cook, c., heath, f., thompson, b., green, d., kyrillidou, m., & roebuck, g. (2010). libqual+® 2010 survey: georgia institute of technology. washington, dc: association of research libraries.   fox, r. e. & stuart, c. c. (2009). creating learning spaces through collaboration: how one library refined its approach. educause quarterly, 32(1). retrieved 16 may 2013 from           http://www.educause.edu/ero/article/creating-learning-spaces-through-collaboration-how-one-library-refined-its-approach   potthoff, j. k., weis, d. l., montanelli, d. s., & murbach, m. m. (2000). an evaluation of patron perceptions of library space using the role repertory grid procedure. college & research libraries, 61(3), 191-202. retrieved 16 may 2013 from http://crl.acrl.org/content/61/3/191.full.pdf+html   somerville, m. m. & collins, l. (2008). collaborative design: a learner-centered library planning approach. the electronic library, 26(6), 803-820. doi:10.1108/02640470810921592   whitmire, e. (2001). a longitudinal study of undergraduates’ academic library experiences. the journal of academic librarianship, 27(5), 379-385. doi:10.1016/s0099-1333(01)00223-3 news/announcements   c-eblip fall symposium: librarians as researchers – call for proposals      2015. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   the centre for evidence based library and information practice (c-eblip) is situated in the university library, university of saskatchewan (u of s), canada. c-eblip supports u of s librarians as researchers and promotes evidence based practice by:   acting as a central resource for mentorship and collegiality building on and sustaining personal learning and research support networks showcasing the research productivity of the university of saskatchewan librarians hosting national and international conferences, workshops, seminars, and events fostering long-term collegial relationships locally, nationally and internationally   the c-eblip fall symposium: librarians as researchers is looking forward to its second iteration on wednesday, october 14, 2015 at the university of saskatchewan. building on the success of last year’s inaugural event which saw 53 librarians attend from across canada and the us, this year’s symposium promises to be an active and engaging gathering place for librarians interested in conducting research. this one-day conference (with complimentary registration, including breakfast, breaks, and lunch) is designed to facilitate sharing, collaboration, and networking with the focus on librarians in their researcher roles.  whether it’s a requirement for tenure and promotion, part of evidence based library and information practice, curiosity, or contributing to one’s own practice or to the field of librarianship, librarians approach research from the unique perspective of practice.   the c-eblip fall symposium: librarians as researchers consists of a series of single-track sessions, along with a keynote address, posters (new this year!), and time for networking. the sessions and posters will focus not only on the research being done by librarians, but also on how librarians do research (tips, tricks, methodology, collaborative efforts, etc.) and why librarians do research (motivations or inspirations).   submit your proposal in the following areas:   sessions   sessions are 20 minutes in length which includes time for questions. session topics should broadly relate to one of the focus areas of the symposium:   research being done by librarians – results of research projects, a project in its interim state, experiences with conducting research, etc.   how and/or why librarians are conducting research – methods, techniques, tips, tricks, etc. the inspirations, motivations, philosophical underpinnings of conducting research, etc.   session proposals of no more than 250 words should include the following:   title and description of the presentation how the proposal links to a focus areas (research being done by librarians; how/why librarians are doing research) outcomes or takeaways   posters   the poster component of the symposium will include dedicated time for poster viewing and conversation, as well as an opportunity for poster presenters to give a one-minute overview of their posters to all symposium attendees. as above, posters should broadly relate to one of the focus areas of the symposium.   poster proposals of no more than 250 words should include the following:   title and description of the poster how the poster links to a focus area (research being done by librarians; how/why librarians are doing research) outcomes or takeaways   applicants will be advised of the status of their proposal no later than the week of august 24, 2015.   submit proposals to virginia wilson, director, centre for evidence based library and information practice, virginia.wilson@usask.ca, by friday, july 31, 2015.   watch for more details about the c-eblip fall symposium: http://library.usask.ca/ceblip/c-eblip-fall-symposium/about.php   download a pdf of the call for proposals: http://library.usask.ca/ceblip/c-eblip-fall-symposium/call-for-proposals.php   follow c-eblip on twitter: @ceblip     editorial   time for reflection and planning   alison brettle editor-in-chief school of nursing, midwifery and social work university of salford, united kingdom email: a.brettle@salford.ac.uk      2013 brettle. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.       welcome to the september 2013 issue of the eblip journal. for academics this time of year presents time to consolidate, reflect and plan in preparation for the year ahead. from an eblip journal point of view, a presentation that i prepared for the eblip7 conference at the university of saskatchewan, canada in july provided this opportunity for consolidation and reflection. the conference was a welcome opportunity to meet with some of our users in person, as well as to provide them with a journal update. as many of our users were not able to attend the conference i want to use this editorial as an opportunity to share the information that was presented in saskatchewan.   the journal has continued to grow and develop, however it remains a zero budget publication that is firmly rooted in practice. all our editorial team members contribute to the journal on top of their day jobs, so to meet with the growing editorial workload the team has been expanded to nine people (http://ejournals.library.ualberta.ca/index.php/eblip/about/editorialteam). to ensure quality and consistency the copyediting team has also expanded to eight individuals with a lead copyeditor forming part of the editorial team.   although the journal is hosted by the university of alberta, and has a strong canadian influence from its roots, it is a truly international affair. the editorial and copyediting teams have representatives from the usa and the uk as well as from canada. there are 19 evidence summary writers from a range of countries and the editorial advisory team (or peer reviewers) comprises 97 library practitioners or academics from countries ranging from botswana to portugal with many others in between. the advisors undergo a selection process which ensures that the team has a balance of expertise across library disciplines and research methodologies. this in turn ensures that the highest quality and most appropriate feedback is given to authors.   the journal is now in its eighth year of publication and is indexed in lisa, lista, library literature, google scholar and scopus. it has a combination of peer reviewed and non peer reviewed sections, with an average of 13 peer reviewed items per issue (articles and evidence summaries). the acceptance rate is 67% and our turnaround for peer reviewed items is fast: on average 31 days for review and 102 days to publication.   our most popular items (based on the number of downloads) are about research methods. this includes the ebl 101 column as well as our most-downloaded paper goddard (2007) “getting to the source: a survey of quantitative data sources available to the everyday librarian”. downloads for articles are high, with the top most-downloaded paper (goddard, 2007) having 12,375 downloads and 11 other papers having more than 5,000 downloads each (http://ejournals.library.ualberta.ca/index.php/eblip/announcement/view/149link). the most downloaded paper in 2011 was davies (2011) “formulating the evidence based practice question: a review of the frameworks”, and in 2012 the top download was raven (2012) “bridging the gap: understanding the differing research expectations of first-year students and professors”.   on average, articles are downloaded 1,491 times; individual authors can use open journal systems (ojs) to see how frequently their papers are viewed. in 2011 we introduced an html format in addition to pdf downloads and this has proved popular with an average view of 169 per item, a top view of 1,842 and 10 items with more than 1,000 views.   we also try to bring an evidence based approach to the journal and thus far have focussed on examining the evidence summaries and how librarians use them in their practice (kloda, koufogiannakis & mallan, 2011). the editorial team members who authored this paper received the robert h blackburn award in 2012. a follow up project which looks at the impact of evidence summaries on library practice has been presented at three international health library conferences: medical library association, us 2012; health libraries group, uk 2012 and eahil 2012 (europe). this project is currently being written up for publication.   i began this editorial by saying that reflection enabled planning for the future and so it is timely to end by briefly describing some future developments for the journal. the december 2013 issue will feature the eblip7 conference, including a commentary from the keynote speaker and summaries of the lightning strikes! presentations. given the popularity of the items which focus on research methods, we are currently compiling them into a wiki-based resource which will mean that users can access all the research methods material from one place (thank you to one of our users for this idea). finally, we are revising our reviewer and author guidelines and these will be available by the end of 2013.   i continue to feel proud and honoured to be editor of the eblip journal and thoroughly enjoy working with an inspirational team. i hope you continue to find this issue of use to your practice; feel free to submit your ideas for future features or improvements to ensure its continued development and relevance to your needs.     references   davies, k.s. (2011). formulating the evidence based practice question: a review of the frameworks. evidence based library and information practice, 6(2), 75-80. retrieved 4 sept. 2013 from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/9741   goddard, l. (2007). getting to the source: a survey of quantitative data sources available to the everyday librarian: part ii: data sources from specific library applications. evidence based library and information practice, 2(1), 68-88. retrieved 4 sept. 2013 from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/152     kloda, l., koufogiannakis, d. & mallan, k. (2011). transferring evidence into practice: what evidence summaries of library and information studies research tell practitioners. information research, 16(1). retrieved 4 sept 2013 from http://informationr.net/ir/16-1/paper465.html   raven m. (2012). “bridging the gap: understanding the differing research expectations of first-year students and professors. evidence based library and information practice, 7(3) retrieved 4 sept. 2013 from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/17172 microsoft word es_bogel.doc evidence based library and information practice 2007, 2:2  90 evidence based library and information practice       evidence summary    students in nova scotia schools without teacher‐librarians are not achieving  department of education expectations for information literacy skills    a review of:  gunn, holly, and gary hepburn. “seeking information for school purposes on the internet.”  canadian journal of learning and technology 29.1 (winter 2003): 67‐88. 24 may 2007   < http://www.cjlt.ca/content/vol29.1/04_gunn_hepburn.html>    reviewed by:   gayle bogel  director, learning resources and information technology, john read middle school  redding, connecticut, united states of america  e‐mail: gbogel@gmail.com      received: 27 february 2007  accepted: 14 may 2007      © 2007 bogel. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – this study investigated whether  the expectations for internet searching  strategies outlined in provincial curriculum  goals are being met in nova scotia schools.   twelfth‐grade students in representative  schools were surveyed as to their internet  information seeking strategies and their  perceptions of the effectiveness of those  strategies.  the results are presented as six  themes based on the survey questions.    design – survey questionnaire consisting of  yes/no, multiple‐choice, likert style, and  open‐ended responses.      setting – twelfth‐grade students from four  high schools in one district in nova scotia.  total participants: 198.  subjects – questionnaires were analyzed  from 243 general practitioners, practice  nurses, and practice managers in four  nottingham primary care trusts as well as  practices in the rotherham health authority  area.     methods – four research questions guided  this study:    1. what strategies and techniques do  students use that are helpful for  information‐seeking on the internet?  2. what knowledge do students have of  the different world wide web search  engines?  3. how do students perceive their ability  to locate information for school  purposes on the internet?  http://www.cjlt.ca/content/vol29.1/04_gunn_hepburn.html mailto:gbogel@gmail.com http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:2  91 4. how do students learn how to seek  information on the internet for school‐ related assignments?    the survey was developed through a  literature review of previous research.  each  survey item reflected a theme and one of the  four research questions. the survey was  field tested in a pilot study with two  twelfth‐grade students, and two twelfth‐ grade english classes.     the sample was assembled by asking  principals at the four schools to identify two  classes in each of their schools that  represented mixed academic abilities.  three  schools chose english classes, and one  school chose math classes participate in the  study. all students had agreed to be a part  of the study and only students present in  class on the day the questionnaire was given  were represented.  no effort was made to  include students who were absent.     results were tabulated as percentages of  responses, and presented in tables related to  the themes of the four research questions.    main results – throughout the study,  students reported very few strategies for  effective internet searching.  they cited  friends and family members rather than  teachers as their main sources for support,  and reported self‐taught trial and error as  the most common method of learning search  strategies.  despite their lack of effectiveness,  most students considered themselves  “good” or “very good” at finding the  information they need for school purposes.    most of the students used very few of the  strategies associated with effective searching  that have been stated in prior research  studies.     • research question one: use of  strategies and techniques for  information‐seeking on the internet  only 15% of students used boolean  operators regularly.    over 70% of students did not know how to  eliminate commercial sites, use particular  features, limit searches to recently updated  pages or limit searches to the title section of  a web page.    • research question two: knowledge of  world wide web search engines.     google was the overwhelming choice, with  66.7% percent of students  reporting that  they used it regularly.  other search engines  were used from 0 to 22%.    • research question three:  students’ perception of their information‐ seeking ability on the internet    81.3 % of students reported their abilities as  good or very good.    only 5% felt their abilities were poor.    • research question four: how students  learn what they know about  information‐seeking on the internet    72.7% reported self‐teaching strategies.    39.8% relied on friends or classmates, 36.8 %  relied on teachers.    2.5%  reported librarians as a source    of the students who reported self‐teaching,  53% used trial and error, 6.6% used help  screens and 4% searched for assistance.  80.8% of students who reported teachers as  a source for learning information strategies  were taught in computer‐related classes,  rather than in content area classes across the  disciplines.     although only 72% of students reported  having internet access at home, 64% stated  evidence based library and information practice 2007, 2:2  92 that they used the internet more at home  than at school to find information for  school‐related assignments.   46.3% of  students with no internet access at home  rated their perception of searching ability as  poor, compared to only 8.3% of students  who did have internet access at home.    conclusion – the researchers state that  actual practice in nova scotia schools does  not reflect the standard instructional  strategy of modeling as recognized by the  nova scotia department of education.  they  feel that the results of this study show that  very little modeling is being done by  classroom teachers; that the modeling is  instead being done by peers and family at  home.  this magnifies the disparity in  effective skills for those who do not have  internet access at home.   they also note that  the goal of integrating search strategy  instruction across the disciplines is not being  reached.     the researchers suggest two ways to offer  the needed instruction: compulsory classes  in information seeking for all students, or  the hiring of teacher‐librarians to support  instruction in the schools, working  collaboratively in all disciplines.  research  supporting the presence of teacher‐ librarians in teaching effective information  literacy skills, including internet searching,  is noted.      commentary      the 1994 reassignment of teacher‐librarians  to classrooms delegated the instructional  role of teaching information‐seeking  strategies to teachers in each discipline, with  the goal of integrating the instructional role  across the disciplines.  the vision for the  integration of information technologies within  the nova scotia public school system (1999)  explicitly outlines the need for teacher  support in helping students achieve stated  expectations for information‐seeking skills.   the researchers state that assumptions  about teachers’ expertise in online  information‐seeking, and their ability to  support students in this area, are erroneous ‐ ‐ that “the lack of qualifications of library  staff, the range of qualifications of library  staff, and the sometimes limited expertise in  internet searching techniques of many  teachers who are to support students” may  have a dramatic effect on meeting  expectations for graduation.     the researchers have confirmed at a local  level results that are similar to the initial  findings of the ets (educational testing  service) information literacy skills test that  has been developed and tested with high  school and college freshmen in the united  states. the ets testing clearly indicates that  students, although savvy with technology,  are not arriving at college with the search  skills necessary to successfully initiate and  complete inquiry based research  assignments integral to all college  disciplines (katz 2007).      a wealth of research is available on the  information‐seeking behaviors of students  in school. gross (2003) documented the  effect of imposed queries on children’s  searches at school and hirsch (2003)  explored the effect of children’s domain  knowledge on search behaviours.  shelton  (2007) outlined the causes of information‐ seeking failure in students, and provided  insight that can be used to shape effective  instructional practices.  there is a need for  teachers to understand how to structure  both targeted and open‐ended inquiry  lessons in search strategies, including  specific compensations for the cognitive and  affective needs of children at different  developmental stages and contexts.   professional development courses for school  librarians and teachers as pre‐service  instruction or as instruction guided by the  school librarian in a professional  development role would benefit from  evidence based library and information practice 2007, 2:2  93 including this research and its practical  applications in daily practice and  collaborative lesson design.    this user study addresses a clearly focused  issue, and the approach the researchers have  taken is substantiated by earlier related  research. the data collection was validated  with a pilot study and both the response  rate and the population chosen were  satisfactory. results were clearly reported  and interpreted. the survey was given only  in schools that did not have a teacher‐ librarian on staff, and since there are a small  number of schools in nova scotia that do  have a teacher librarian, it would perhaps  have been helpful to see the results of the  survey given at one of those schools, as a  basis of comparison.      the researchers have successfully  documented the lack of effective  information skills in students in the nova  scotia schools.  based on earlier studies that  cite correlations between the presence of  teacher librarians and the demonstration of  effective searching skills, they propose that  the lack of teacher librarians is a major  contributing factor to the lack of skills.   further study is needed to substantiate the  correlations.       works cited    gross, melissa. “children’s information  seeking at school: findings from a  qualitative study.” youth information  seeking behavior: theories, models  and behaviors. ed. mary k. chelton  and colleen cool. lanham, md:  scarecrow press, 2003. 211‐40.    hirsch, sandra. “domain knowledge and  children’s search behavior.” youth  information seeking behavior:  theories, models and behaviors. ed.  mary k. chelton and colleen cool.  lanham, md: scarecrow press, 2003.  241‐70.    katz, irvin. “ets research finds college  students fall short in demonstrating  ict literacy.” college& research  libraries news 68.1 (jan. 2007): 35‐37.  24 may 2007  <http://www.ala.org/ala/acrl/acrlpubs/c rlnews/backissues2007/january07/ets.ht m>    nova scotia. department of education and  culture. vision for the integration of  information technologies within the  nova scotia public school system.  halifax: the department, 1999.    shenton, andrew k. “causes of information  seeking failure: some insights from an  english research project.” youth  information seeking behavior ii:  context, theories, models and  behaviors. ed. mary k. chelton and  colleen cool. lanham, md: scarecrow  press, 2007. 313‐64.              http://www.ala.org/ala/acrl/acrlpubs/crlnews/backissues2007/january07/ets.htm http://www.ala.org/ala/acrl/acrlpubs/crlnews/backissues2007/january07/ets.htm http://www.ala.org/ala/acrl/acrlpubs/crlnews/backissues2007/january07/ets.htm evidence based library and information practice evidence based library and information practice 2010, 5.2  evidence based library and information practice       article    developing the role of a health information professional in a clinical research setting    helen m. seeley  research associate  dept. of academic neurosurgery  university of cambridge  cambridge, united kingdom  email: hms35@medschl.cam.ac.uk    christine urquhart  senior lecturer  university of wales  aberystwyth, united kingdom   email: cju@aber.ac.uk    peter hutchinson  honorary consultant neurosurgeon  dept. of academic neurosurgery  university of cambridge  cambridge, united kingdom  email: pjah2@cam.ac.uk    john pickard  professor of neurosurgery  dept. of academic neurosurgery  university of cambridge  cambridge, united kingdom  email: jdp1000@medschl.cam.ac.uk       received: 19 jan. 2010          accepted: 8 may 2010     2010 seeley, urquhart, hutchison and pickard. this is an open access article distributed under the terms of  the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada  (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and  reproduction in any medium, provided the original work is properly attributed, not used for commercial  purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.    abstract     objective ‐ this paper examines the role of a health information professional in a  large multidisciplinary project to improve services for head injury.  47 mailto:hms35@medschl.cam.ac.uk mailto:cju@aber.ac.uk mailto:pjah2@cam.ac.uk mailto:jdp1000@medschl.cam.ac.uk evidence based library and information practice 2010, 5.2  48   methods ‐ an action research approach was taken, with the information professional  acting as co‐ordinator. change management processes were guided by theory and  evidence. the health information professional was responsible for an ongoing  literature review on knowledge management (clinical and political issues), data  collection and analysis (from patient records), collating and comparing data (to help  develop standards), and devising appropriate dissemination strategies.     results ‐ important elements of the health information management role proved to be  1) co‐ordination; 2) setting up mechanisms for collaborative learning through  information sharing; and 3) using the theoretical frameworks (identified from the  literature review) to help guide implementation. the role that emerged here has some  similarities to the informationist role that stresses domain knowledge, continuous  learning and working in context (embedding). this project also emphasised the  importance of co‐ordination, and the ability to work across traditional library  information analysis (research literature discovery and appraisal) and information  analysis of patient data sets (the information management role).      conclusion ‐ experience with this project indicates that health information  professionals will need to be prepared to work with patient record data and synthesis  of that data, design systems to co‐ordinate patient data collection, as well as critically  appraise external evidence.    introduction     the role of the health information  professional (hip) is shaped and expanded by  the current healthcare environment and  system.  their unique set of skills is  increasingly being used as the need for  evidence based healthcare increases (gray,  2001). there are several types of health  information management role and  delineations of role boundaries may vary from  country to country. in north america, the  “informationist” role (davidoff, rankin, &  oliver, 2000) has developed. the term  informationist, popularized in the late 1980s  (debons, home, & cronenweth, 1988;  oswitch, 1990), was used by davidoff &  florance (2000) to describe a new health  profession which encompasses a wide  diversity of roles and a wide variety of  attributes and skills in a variety of healthcare  settings. responsiblities of the informationist  encompasses the socio‐economic, cultural,  scientific, and technical information systems.  scholars emphasised that this new  informationist should be an integral part of a  group with specialized expertise that can  contribute vitally to clinical situations rather  than information ‘servers’ in an auxiliary  capacity. the medical library association  (mla) explored the concept resulting in a  national implementation strategy, including  renaming the role information specialist in  context (isic) as a broader more inclusive  term emphasising the importance of working  within context (homan & mcgowan, 2002;  shipman, 2007).     this shift required existing hips to market  their services to health professionals (lewis,  urquhart & rolinson, 1998). however, and  health librarians have been surprised to find a  gap between what they offer and users’  expectations and impressions of services  offered by “information specialists”  (publicover et al. 2006). health professionals  often seem unaware of the new roles health  librarians  are fulfilling in work with  multidisciplinary teams for the production of  evidence bulletins (mann, sander &  weightman, 2006), or in clinical librarian roles.  (beverley, booth & bath, 2003; harrison &  sergeant, 2004; urquhart et al. 2007; wagner &  byrd, 204; winning & beverley, 2003).  evidence based library and information practice 2010, 5.2  bailey & rudman (2004) argued for the hip to  have knowledge and skills to lead in the  design, process and implementation of  collaborative research projects. beverley,  booth, and bath (2003) identified 11 roles  when tracing the evolvement of the  information specialist, the most recent being  primary researcher. bury, lindsey, and  roberts (2006) also noted a change in focus  from teaching by librarians to the concept of  the hip as a learning facilitator in the  literature.     conclusions from the literature are that hips  need to respond to changes, build on the past  and re‐engineer themselves to meet the  information‐intensive demands of healthcare  of the future. one major problem seems to be  credibility in knowledge transfer among  members of a healthcare team (jacobson &  goering, 2006), and another is the need for  specialist educational programmes for some  informationist careers (oliver et al., 2008).    in the united kingdom, according to the  national health service (nhs) careers  website, individuals in information  management “are responsible for the retrieval,  analysis, interpretation and presentation of  health data and information, to a high  standard.” the health librarian role has been  subsumed into knowledge management; hips  support health professionals and managers in  their education, training, and practice, by  locating, retrieving and organizing the  necessary evidence from a vast array of  resources, as well as training professionals in  critical appraisal.  in the uk, therefore, a  general distinction is made between those  information professionals who deal with the  processing and analysis of local patient data  (information managers), and those who seek  evidence from elsewhere for local application  (knowledge managers).    such distinctions may not always be helpful  for the health service or the hips involved.  the aim of this paper is to describe and  examine the role of a hip (the first author, hs)  in a multidisciplinary longitudinal research  project aimed at improving the care of head‐ injured patients in the east of england region,  uk. the project was collaborative, involving  university researchers, a large and renowned  teaching hospital, other local hospitals,  primary care providers, and other agencies  such as voluntary support groups.   setting    the eastern head injury group (ehig) is a  collaborative research partnership set up in  2000 as a response to recommendations in  royal college of surgeons of england (rcse),  the society of british neurosurgeons (sbns)  reports, and the house of commons’ 3rd  report (2000). this ehig study used a  collaborative action research approach to  identify and address deficiencies in regional  service provision for head injured patients.  the aim was to develop a framework for  regional service across tertiary, secondary,  primary care and social care. head injured  patients have diverse and complex needs,  which require a co‐ordinated service response  from a wide range of specialties, disciplines,  and organizations. for those commissioning  services (a responsibility of primary care  trusts, pcts) or delivering services, making  changes to service provision requires an  understanding of the relationships between  different parts of the system, and how changes  in one part can affect many other parts of the  system. therefore, a whole systems approach  was necessary. figure 1 shows the scope and  process of the research study.  the ehig began as a working group, but as  the project progressed through phases, it  developed into a partnership of academic  researchers, managers, and commissioners to  enable every aspect of service change to be  addressed (figure 2). this research  partnership acted as a steering group,  providing strong leadership in strategic  planning of the study, meeting regularly as  part of the action research cycles of planning,  action, and review or reflection, and was seen  as essential to successful implementation.  49 evidence based library and information practice 2010, 5.2  survey of current regional hi service provision detailed audit of regional epidemiology of hi mapping of services multidisciplinary ward round development of rehabilitation codes, standards, & potential care pathways pilot mhi clinic examination of role of hi co-ordinator development of hi coordination template neurotrauma clinic phase 2 examining & addressing issues phase 3 development of strategy & framework phase 4 implementation phase 1 survey evaluation of service planning tools development of cross-sectoral collaborative partnerships development of new rehabilitation facilities 2000 2008+ fig. 1. the ehig study action research cycles showing developmental process    nhs (national health service) management/commissioners/funders clinicians/practitioners/providers professional bodies/ multi-specialty/ dept of health/nhs government ehig collaborative research model • all key stakeholder groups involved (academic /management/clinical/ hip professional) • comprehensive skill mix • whole systems approach • responsive / flexible • collaborative continuum • contextual leadership 2000 head injury group steering group collaborative research partnership regional hi network 2008 multi collaborative research partnership h e a l t h i n f o r m a t i o n p r o f e s s i o n a l phase 1: initial membership • professor of neurosurgery (academic) representing north of region • consultant neurosurgeon (representing south of region) • eastern specialised commissioning group clinical director (nhs and commissioning) • health information professional: principal researcher phase 2: additional membership • clinical director of a&e (acute care and services) • consultant in rehabilitation medicine (rehabilitation post acute stages) • consultant paediatrician phase 3: additional membership • lead in head injuries: consultant neurosurgeon (an honorary fellowship supported by the academy of medical science and health foundation) phase 4: implementation stage • new escg representation changed with role and remit to liaise with strategic health authorities, primary care trust (pcts) and managers • representative from nhs management of services (neurosciences • service centre manager) • representative from pct primary care commissioning (chief commissioner) fig. 2. the composition and development of the eastern head injury group    50 evidence based library and information practice 2010, 5.2  the hip (hs) had several years experience  working within the neurosurgery department  and took on the role of the research co‐ ordinator / principal researcher. hs has a  librarianship qualification, a postgraduate  qualification in health information  management, and registered for a doctorate at  the start of the project under the co‐author  (cu). the post was jointly funded (nhs  research & development, eastern specialised  commissioning group, nhs, and university  of cambridge), thus creating shared  ownership and strong commitment to the  research work. the responsibilities of the role  stretched widely across the research study  (table 1).     methods    as the project was multidisciplinary and  aimed at changing practice through  collaborative learning, a participatory action  research approach was used together with soft  systems analysis techniques to explore the  explicit and tacit knowledge management.  adopting a theory‐driven approach to health  services research is desirable (brazil et al.  2005), but has challenges in terms of time,  capacity of the researcher’s and co‐ordination  of the research process. the dedicated role for  the hip freed clinical professionals from  conducting some of the data collection  “fieldwork,” analysis, and report writing, and  provided continuity, consistency, and stability  to the overall study.    the methods used by the hip in the action  research (table 1) incorporated all aspects of  knowledge management, including literature  review (clinical and policy issues), data  collection and analysis, collating and  comparing data (to help create new  knowledge and produce evidence in context),  developing tools for change, and devising  appropriate dissemination strategies. the  work therefore bridged traditional  information management (of patient data) and  the developing knowledge management roles  (as defined by the uk national health  service). the action research embraced cycles  of planning and action, followed by reflection  and learning to inform future planning.    co‐ordination was vital for management of  the multiple research cycles and a successful  outcome. the hip co‐ordinated the  development of communication networks,  such as the regional conferences and  workshops that helped establish trust among  the diverse groups involved. the hip was also  responsible for the development of improved  information systems (e.g. for the patient care  pathways, and creation of website) and  development of the rehabilitation categories  and codes that would help ehig create new  knowledge.      results    the research programme produced a  replicable service framework, service planning  tools, a valuable research resource (regional  head injury database), and a flexible  collaborative learning network with strong  leadership that included a model for the role  of a hip. the work and results of the research  are published in a series of papers in peer‐ reviewed journals (bradley et al., 2006; pickard  et al. 2004; seeley et al., 2001; seeley &  hutchinson, 2006; seeley et al., 2006). table 2  summarises the outcomes of the study in the  development of systems for a comprehensive,  co‐ordinated responsive and flexible service  framework for head injury.  it also  summarises the models, strategies, and  methodologies developed that contribute to  health services research.    the importance of good information and  communication systems, together with co‐ ordination became increasingly apparent. the  study not only developed strategies and  initiatives to address the gaps and variability  in services, but also created new and  replicable information systems for planning  and evaluation, which can be collated into the  elements of a methodology for innovative  change. this aspect is described in two further  publications (seeley et al., 2007; seeley &  urquhart, 2008).    51 evidence based library and information practice 2010, 5.2  table 1.  specific methods used  planning and action – review collaborative development of service tools 1. role of principal researcher  2. groups: research group, focus groups, head injury (hi) working groups, neurosciences strategic  group.  3. partnerships  4. literature reviews   5. initial survey (2000)  6. reviews a) a&e   b) neurorehabilitation  7. questionnaires (a&e, neurorehabilitation)  8. interviews and visits a&e, rehabilitation units  9. retrospective  audit:  head  injury  admissions  to  regional  neurosciences  centre  +  as  many  other  hospitals as possible  in region  (numbers, categories, pathways/initial mapping, resources, data  quality);  prospective  audit  of  hi  transfers;  prospective  regional  audit  of  hi  pts  with  disruptive  behaviour  10. assessment of  impact  and implications of new resource/technology/ role/ work patterns /change:   11. a) observation ward   b)minor hi clinic c)traumatic brain injury follow‐up clinic d)ct scanning,  image transfer e) hi co‐ordinator role > business plan: hi co‐ordinator post  12. evaluation /outcome studies  13. piloting of hi standards  14. piloting of rehabilitation codes        15. detailed mapping  16. development of standards      17. development of rehabilitation codes  18. development of head injury co‐ordination template  reflection and learning planning dissemination and knowledge/information-sharing 19. multidisciplinary conferences  20. workshops  21. working groups  22. reports  23. study days: hi training day for a&e staff; aggression in hi seminar; neurorehabilitation meeting  24. working documents  25. presentations at key professional bodies conferences   26. liaison with  government  and professional bodies  27. links created with other regions  28. publication of papers  29. development of regional ‘hi network’ ( database of contacts, website, database, helpline)    the hip role  a key component of the effectiveness of the  research programme was the innovative hip  role, which was diverse, multi‐functional, and  objective (neither clinical nor managerial, not  purely information management or  knowledge management). the importance of  the neutrality and the different skills were  appreciated as essential to the effectiveness of  the team, the research and the collaborative  continuum.    the role has similarities to the informationist  role, in which working in context, continuous  learning, and domain knowledge are  important (bailey & rudman, 2004; holman,  2002). in this case some domain knowledge  had been gained in clinical audit, prior to  52 evidence based library and information practice 2010, 5.2  starting in the principal co‐ordinator role, but  more domain knowledge was gained  throughout the research (continuous learning  while working in context).    more general research skills were also  developed, partly as the hip was undertaking  doctoral research based around the role.  however, these skills would have been  necessary in any case, and some informationist  roles stress them (detlefsen, 2004). literature  searching and systematic reviewing are skills  recognised in other studies (beverley, booth, &  bath, 2003), but in this study, the searching  and reviewing covered the theory and  evaluation of some of the processes for  managing change in service delivery. the  flexible nature of the post was important,  allowing development of a variety of skills in  organisation, administration, research  methods, and analysis and dissemination  (figure 3).    the hip was not working alone, and  discussion and reflection with other members  of the team helped address some limitations in  experience and knowledge. the regular input  of the research leadership group and the  collaborative action research (car)  approach, enabled triangulation of different  types of evidence, and thorough evaluation of  the various cycles of the action research.     table 2. research outcomes  head injury services strategies, methodologies and models developed/created the systems for a comprehensive, co-ordinated responsive & flexible and sustainable service framework for head injury. established a model of collaborative cross-cultural working, including a model for effective and innovative partnership working service planning tools to plan, develop, implement and evaluate an effective and comprehensive regional service for head injury. • standards for the management of head injuries in acute hospitals • rehabilitation definitions and codes • head injury co-ordination template developed a whole systems approach to knowledge transfer and dissemination established a collaborative cross-cultural research leadership model developed a model for the role of a health information professional in an academic and clinical setting templates • evaluating a new facility • evaluating a new technology • evaluating new evidence/research • contextual mapping process developed a methodology for developing and sustaining networks resources • regional head injury network, including named leads in hi amongst all key stakeholders • new and innovative posts: brain injury liaison officer linking acute and community sectors and nhs and voluntary sectors • new facilities • funding through partnerships with private/voluntary sectors contributed towards a theory of large scale transformational change in the health sector whole systems approach, drawing strength from cultural diversity, defining alternatives, looking at root causes, pushing the debate, broadening the agenda, creative direct action, contextual research leadership, approach to and theory of knowledge/evidence, comprehensive dissemination, sustaining networks   53 evidence based library and information practice 2010, 5.2  the “neutral stance” of the hip role helped in  communication among different professional  groups. the hip’s awareness of the needs of  different professional groups was helpful in  disseminating information to all those  involved. the hip had a “helicopter” view of  what was required, by whom, when, and in  what format, such as a digest, workshop, or  full report. the importance of sustaining a  network became apparent during the research  and evidence, was consulted (chisholm, 1998;  church et al. 2002) to help in designing and  creating  a website, produce regular updates  and reports and organise  regional multi‐ organisational meetings.    although part‐time doctoral research may  seem a very long process, in this case the five  year timescale suited the nature of the  research work.  there was a considerable  amount of co‐ordination required among  different organisations, each with their own  cultures and ways of working.  time was  required to understand how the different  organisations worked, and how to achieve  consensus on solutions to difficult problems.    the hip organized and made available a very  wide range of resources for learning while also  acting as a resource, or gatekeeper, to be used  by the research group and the wider  stakeholder group, feeding back learning and  research into a wider area. enabling  practitioners to enter into the process of action  and reflection is more likely to result in  learning through and about the process of  practice. for example, as new issues arose, the  research partnership was expanded to ensure  inclusion of all stakeholders (see figure 2).  key aspects facilitation communication liaison dissemination coordination between all stakeholders across sectors and cultures key skills identifying key stakeholders current awareness literature review project design questionnaire design audit case study evidencegathering/analysis research synthesis systems analysis database organising meetings organising working /focus groups interviews presentations report writing working documents mapping key features dedicated post joint funding protected time specialist knowledge/ skills objectivity continuity maintaining momentum keeping focus persistence knowledge management enabling collaborative learning negotiating involvement networking whole project coordination: dimensions geographical; chronological (multi-cycle); interdisciplinary; interprofessional; intra-organisational, inter-organisational; cross-sectoral; cross-cultural phase 1 (2000) phase 5+ (2008+) ehig r e g i o n i o n a l national fig. 3. role and scope of the health information professional in the ehig research study  54 evidence based library and information practice 2010, 5.2      fig. 4. the ehig collaborative research network    collaborative learning network    as the project developed, it was clear that a  collaborative learning network was necessary for  the diverse groups to work together with shared  purpose and values. networks of practice may  be viewed as a looser arrangement than  communities of practice (cox, 2007), and while  social networks are hardly new, how they are  enacted has changed (castells, 2000). a key aim  of the study was to provide leadership in  developing a community of key personnel in the  management of head injury. the co‐ordination,  facilitation, and continuity provided by the  research leadership group and the hip role  enabled the development of an effective research  and learning network (figure 4).  the aim of the collaborative learning network  was to implement and sustain change in health  service delivery. to help ensure that change  would happen, the hip consulted the change  management literature: the horizontal and  vertical structure within the research network  reflects the evidence on the most effective ways  of initiating and sustaining change (e.g. kaluzny,  veney, & gentry, 1974; sine, mitsuhashi &  kirsch, 2006; stinchcombe, 1965).    although collaboration and participation are  important in the change process, senior  management support is essential in  implementing organisational change in projects  (hart, 2006; meyer, 1993; pinto, pinto, & prescott,  1993). senior managers were sent research  summaries by the hip regularly, and health  service commissioners were involved in the  initiation of the research programme and kept  informed of developments.    discussion    table 3 summarises the hip roles and tasks  identified in the literature.     individual studies mapping exercise joint initiatives partnerships with: department of health national service framework voluntary organisations private sector clinical leads in head injury other regions/ organisations collaborative working in specific projects pct leads in head injury health information professional opinion leaders & ‘champions’ contextual research leadership partnership voluntary / private sectors partnerships professional bodies building capacity to use evidence getting research into practice learning generating new knowledge pct leads in head injury organic, flexible and responsive. underpinned by leadership to provide structure 55 evidence based library and information practice 2010, 5.2  table 3. the information specialist roles and tasks identified in the literature  roles  analyst   clinical librarian  communicator  critical appraiser  disseminator  educator  facilitator  gatekeeper   information architect  information scientist  knowledge manager  knowledge  counsellor  liaison   mediator  primary researcher  research  resource manager  system designer  teacher  tasks/skills  • research & development  • data storage / retrieval  • systems development / design  • systems analysis  • database development  • information management  • information scientist  • identification and ranking of information needs   • location & selection of appropriate information  sources  • appropriate manipulation and processing of  information to disseminate to target audience  • information networking:  development incl.  identification of potential participants  • design and implementation of standard methods  for acquisition, control and exchange of information  • developing and building necessary infrastructures  for sectoral and multi‐sectoral information systems  • co‐ordination of services  • training and support of new technologies    rankin et al. (2008), in a systematic review of  the informationist role, found that that domain  knowledge, continuous learning, and working  in context (embedding) were crucial to  success. this project confirmed these findings,  but also emphasised the importance of co‐ ordination across disciplines, organizations,  and cultures, plus the ability to work across  traditional library information analysis  (research literature discovery and appraisal)  and information analysis of patient data sets.    the hip role developed in this study has some  features of the role of “boundary spanner,” in  that it fulfilled an important boundary role  between a number of organisations and  cultures; the hip received, filtered, and  disseminated the flow of information between  the research group and the wider collaborative  network (figure 4). this seems critical to  sustainability and the spread and  implementation of innovative change. the hip  was proactive in finding, sorting, processing,  applying, negotiating, transmitting and  reframing and sharing knowledge.     there has been a shift from education as  training to a learner‐centred view, where the  learner constructs their own knowledge  through “collaborative learning, authentic  tasks, reflection and dialogue” (mayes, 2001,  p.16), and this shift has changed the role of the  academic subject librarian (pinfold, 2001).  health librarians are now often described as  “learning facilitators” or “knowledge  mobilizers” (brice & gray, 2004, p.82) with  new responsibilities (bury, lindsey & roberts,  2006). this research project demonstrated the  need for new structures as well as knowledge  brokerage. moore (2003) noted how  information specialists, and learning centre  services changed to meet new student learning  requirements in a university. urquhart et al.  (2006; 2007) and burdick (2004) discussed  similar changes in the role of the clinical  librarian to assist in team learning. in this  project brokerage included signposting,  sourcing, interpretation, distillation, and  commentary on quality and dissemination of  evidence and capacity building in terms of  support for collection and analysis of new  evidence.     56 evidence based library and information practice 2010, 5.2  successful knowledge networks create and  disseminate, share new knowledge, have well‐ defined management structure and  communication strategy, transcend the  boundaries between sectors (clarke, 2001),  and play an important part in innovative  processes. there is some evidence that the  difficulties in creating successful knowledge  networks can be overcome by including a  “knowledge activist” responsible for  energising and co‐ordinating the knowledge  creation (dyer & nobeoka, 2000; eisenhardt &  martin, 2000; seufert, von krogh, & bach,  1999; swan et al., 1999). von krogh, nonaka,  and khijo (1997) also found that knowledge  activists working as network co‐ordinators are  the main driving force in a successful  knowledge network. in this research, the hip  may have fulfilled the role of the knowledge  activist.    in many ways, this study confirms the diverse  nature of professional employment for health  informatics professionals (norris & brittain,  2000) as well as the particular needs of those  working within the health sector in the uk  (pearson & urquhart, 2002; urquhart et al.  2005) and in the us (giuse et al. 2005; mla,  2007), where there is an informatics impetus as  well (helms et al., 2004; hersch, 2002; oliver &  roderer, 2006; perry, roderer, & assar, 2005).  for an informationist to be successful, rankin,  grefsheim & canto (2008) conclude from their  systematic review, that subject knowledge is  essential. another study (petrinic & urquhart,  2007) suggested that scientific knowledge  certainly helps health librarians initially, but  that such knowledge can be gained while in  post.     for a hip to be accepted, their work must be  credible to the team of health professionals.  case studies of management consultants  working on knowledge transfer suggest that  there are four dimensions to credibility  (jacobsen & goering, 2006). they are the  scientific credibility (accuracy of the data), the  expertise of the communicator, the authority  of the communicator, and the neutral stance of  the communicator. in this study, there was an  emphasis on obtaining and generating  accurate data and information for planning,  with regular collaborative discussion and  comprehensive feedback. the hip’s  responsibilities include accurate data capture,  interpretation, and intelligent dissemination  (i.e. communicating it in an appropriate,  meaningful and accessible format). working  with the research partnership (a source of  expertise) helped to put a stamp of authority  on the information analysis. the neutrality of  the role, whilst important in data analysis, did  not necessarily contribute to the credibility of  the research, but as the project progressed, the  credibility of the hip probably increased, as  responsibilities became greater.    rankin, grefsheim, and canto (2008) also  conclude that an embedded informationist is  more likely to provide a credible, acceptable,  and sustainable service. shumaker and talley  (2010) reviewed the model of the embedded  information professional in the literature and  found that embedding might be physical (co‐ location), organisational, or virtual, and the  ability to build good relationships was vital.  even in the health sector, where the concept is  accepted (although names are debated), the  evidence for the effectiveness of the  informationist role is only slowly emerging in  the us (e.g giuse et al., 2008; robison, ryan &  cooper, 2009; whitmore, grefsheim, &  rankin, 2008).    however, the costs of specialist hips need to  be justified, and they need to be placed where  the need is greatest (hill, section 10.10, 2008).  health librarians themselves doubt that the  funding will be sufficient to support  information specialist in context posts (sathe,  jerome & giuse, 2007). in the ehig project,  there was a definite identified need for change  in service delivery, and specific funding was  allocated. it was perhaps fortunate the  funding continued over a period of time,  which allowed not just the development of  services, but sustainable systems that could  perhaps operate without direct intervention of  the health information professional in the  future.        57 evidence based library and information practice 2010, 5.2  conclusion    the innovative role described here is the  health information professional (hip). as a  member of the research team, the hip was  involved in health services research to change  the model of health service delivery for a  region. a theory driven approach was chosen,  with the hip providing ideas and evidence  from the literature on how to manage the  change process. an important part of the role  was to support learning, and co‐generation of  new knowledge within the research team and  the health professionals and other  organisations involved in the care of head  injury patients. the hip collected and  analysed patient data, and helped design  improved systems for that purpose.    this role helped to sustain the project  throughout the various cycles of action  research, and a measure of success is the  continuation of the role throughout the  project. with shorter projects, hips may have  less time to develop the credibility that this  setting allowed.    as evidence will increasingly come from  routinely collected patient data sets and  randomised clinical trial data sets,  informationists or hips will work with and  among health professionals to assist in  analysis and synthesis of patient data, as well  as providing the external research evidence  (from databases). in this project the process of  health care delivery was also of interest.    evaluations of informationist and clinical  librarian projects will help to delineate the  skills in information management, information  synthesis, and informatics that will be  required for embedded hips to flourish in  such interesting, but also challenging, work  environments.      acknowledgements    the authors thank all those involved in the  ehig project for their support. we thank the  referees for their helpful and constructive  comments.      references    bailey, j., & rudman, w. (2004). the  expanding role of the him  professional: where research and him  roles intersect. perspectives in health  information management,  1(7), 1‐6.    beverley, c., booth,a., & bath, p.  the role of  the information specialist in the  systematic review process: a health  information case study.  health  information libraries journal, 20(2), 65‐ 74.    bradley, l., kirker, s., corteen, e., seeley, h.,  pickard, j. & hutchinson, p.(2006).  inappropriate acute neurosurgical bed  occupancy and short falls in  rehabilitation: implications for the  national service framework. british  journal of neurosurgery, 20(1), 36‐39.    brazil, k., ozer, e., cloutier, m., levine, r., &  stryer, d. (2005). from theory into  practice: improving the impact of  health services research. bmc health  services research, 5(1) doi:10.1186/1472‐ 6963‐5‐1    brice, a., & gray, m. (2004). what is the role of  the librarian in 21st century health  care?  health information and libraries  journal,  21(2), 81‐83.    burdick, a. (2004). informationist?  internal  medicine rounds with a clinical  medical librarian. journal of hospital  librarianship, 4(1), 17‐27.     bury, r., lindsey, m., & roberts, s. (2006).  achieving change through mutual  development: supported online  learning and the evolving roles of  health and information professionals.  health information and libraries journal,  23(supplement 1), 22‐31.  58 evidence based library and information practice 2010, 5.2    castells, m. (2000). towards a sociology of the  network society.  contemporary  sociology, 29(5), 693‐699.    church, m., bitel, m., armstrong, k.,  fernando, p. et al. (2002) participation,  relationships and dynamic change:  new thinking on evaluating the work  of international networks.   development planning unit, working  paper no. 121. london: ucl, 2002.    chisholm, r. (1998). developing network  organisations: learning from practice and  theory. london: addison wesley.    clarke, h. (2001). formal knowledge networks – a  study of canadian experiences.   winnipeg, canada: internal  institution for sustainable  development, 2001.    cox, a.m. (2007). beyond information‐factors  in participation of networks of  practice: a case study of web  management in uk higher education.  journal of documentation, 63(5), 765‐ 787.    davidoff, f., & florance, v. (2000). the  informationist: a new health  profession?”  annals of international ,  132(12), 996‐998.    debons, a., home, e., & cronenweth, s.   information science: an integrated view.  boston: massachusetts,1988.    detlefsen, e. (2004). clinical research  informationist. reference services  review,  32(1), 26‐30.     dyer, j. & nobeoka, k. (2000). creating and  managing a high‐performing,  knowledge‐sharing network: the  toyota case.  strategic management,  21(3), 345‐367.  eisenhardt, k., & martin, j. (2000). dynamic  capabilities: what are they? strategic  management, 21(10/11), 1105‐1121.    fleck, i., & bawden, d. (1995). the information  specialist: attitudes and images:  examples from information services in  law and medicine. journal of  librarianship & information science,  27(4), 215‐226.    gray, m. (2001). evidence‐based healthcare.  edinburgh: churchill‐livingstone.    giuse, n. b., koonce, t.y., jerome, r.n,  cahall, m.  sathe, n.a., & williams, a.  (2005). evolution of a mature clinical  informationist model. journal of the  american medical informatics  association, 12(3), 249–255.    harrison, j., & sargeant, s. (2004). clinical  librarianship in the uk: temporary  trend or permanent profession? part  ii: present challenges and future  opportunities. health information and  libraries journal, 21(4), 220‐226.    hart, e. (2006). action research as a  professionalizing strategy: issues and  dilemmas.  journal of advanced  nursing, 23(3), 454‐461.    helms, a., bradford, k.d., warren, n.j. &  schwartz, d.g. (2004). bioinformatics  opportunities for health sciences  librarians and information  professionals. journal of the medical  library association, 92(4), 489‐493.     hersh, w. (2002). medical informatics  education: an alternative pathway for  training informationists. journal of the  medical library association, 90(1), 76‐ 79.    hill, p. (2008). report of a national review of  nhs health library services in  england: from knowledge to heath in  the 21st century. retrieved april 2,  2010, from  http://library.nhs.uk/nlhdocs/national _library_review_final_report_4feb_081 .pdf  59 evidence based library and information practice 2010, 5.2    homan, j. m., & mcgowan, j. (2002).  the  mla: promoting new roles for health  information professionals. journal of  the medical library association, 90(1),  80‐85.    houser, s.h, tesch, l., hart‐hester, s., &  dixon‐lee, c. (2008). roles and  challenges of the health information  management educator: a national  him faculty survey.  perspectives in  health information management, 6(4):1‐ 6.    house of commons health committee 2000.   head injury rehabilitation: third  report, together with the proceedings  of the committee, minutes of evidence  and appendices.  london: house of  commons papers, session 2000‐01;  307.    jacobson, n., & goering, p. (2006). credibility  and credibility work in knowledge  transfer. evidence & policy, 2(2), 151‐ 165.    kaluzny, a.d., veney, j.e., & gentry, j.t.  (1974).  innovation of health services:  a comparative study of hospitals and  health departments. milbank memorial  fund quarterly, 52(1), 51‐82.    lewis, r., urquhart, c., & rolinson, j. (1998).  health professionals’ attitudes  towards ebm and the role of the  information professional in the  exploitation of research evidence.  journal of information science, 24(5),  281‐290.    mann, m., sander,l., & weightman, a. (2006).  signposting best evidence: a role for  information professionals. health  information and libraries journal,  23(supplement 1), 61‐64.     mayes, t. (2001). learning technology and  learning relationships,  in:   stephenson j. (ed.). teaching and  learning online: pedagogies for new  technologies, p.16‐26.  london: kogan  page.    medical library association (mla) (2007).  competencies for lifelong learning  and professional success: educational  policy statement of the medical  library association. retrieved april 2,  2010 from  http://www.mlanet.org/education/poli cy/    meyer, j. e. (1993) new paradigm research  practice: the trials and tribulations of  action research.  journal of advanced  nursing, 18(7), 1066‐1072.    moore, k. (2003). building new partnerships:  changing institutional relations, in:  oysten, e. (ed.) centred on learning:  academic case studies on learning centre  development, p.127‐152.  london:  ashgate.    norris, a. & brittain, j.m. (2000). education,  training and the development of  healthcare informatics. health  informatics journal, 6(4), 189‐195.    oliver, k.b., dalrymple,p., lehmann,  h.p.,mcclellan, d.a., robinson, k.a.,  & twose, c. (2008). bringing evidence  to practice: a team approach to  teaching skills required for an  informationist role in evidence‐based  clinical and public health practice.   journal of the medical library  association, 96(1), 50‐57.    oliver, k.b., & roderer, n.k. (2006). working  towards the informationist. health  informatics journal, 12(1), 41‐48.     oswitch, p.a. (1990) the role of the  information professional in  development.  information  development, 6(1), 28‐34.         60 evidence based library and information practice 2010, 5.2  pearson, a., & urquhart, c. (2002). health  informatics education: working across  the professional boundaries. library  review, 51(3/4), 200‐210.    perry, g.j., roderer, n.k, & assar, s. (2005). a  current perspective on medical  informatics and health sciences  librarianship. journal of the medical  library association, 93(2), 199‐205.    petrinic, t. & urquhart, c. (2007). the  education and training needs of health  librarians ‐ the generalist versus  specialist dilemma. health information  and libraries journal, 24(3), 167‐176.     pickard, j., seeley, h., kirker, s., maimaris, c.,  mcglashan, k., roels, e., greenwood,  r., steward, c., hutchinson, p., &  carroll, g. (2004). mapping  rehabilitation resources for head  injury. journal of the royal society of  medicine, 97(8), 384‐389.    pinfold, s. (2001). the changing role of the  subject librarian. journal of  librarianship and information science,  33(1), 32‐38.    pinto, m., pinto, j. & prescott, j. (1993).  antecedents and consequences of  project team cross‐functional co‐ operation.  management science, 39(10),  1281‐1297.    publicover, m., giles, g., bolton, b., shemilt,  c. & urquhart, c. (2006). listening to  our clinical colleagues: how are they  finding their evidence to support  trust clinical governance.  health  information and libraries journal,  23(supplement 1), 57‐61.    rankin, j.a., grefsheim, s.f., & canto, c.c.  (2008). the emerging informationist  specialty: a systematic review of the  literature. journal of the medical library  association, 96(3), 194‐206.    robison, r.r, ryan, m.e., & cooper, i.d.  inquiring informationists: a  qualitative exploration of our role.  evidence based library and information  practice, 4(1), 4‐16.     royal college of surgeons of england. (1999)  report on the working party on the  management of head injured patients  (galasko report). london: rcse.    sathe, n.a., jerome, r., & giuse, n.b. (2007).  librarian‐perceived barriers to the  implementation of the  informationist/information specialist  in context role. journal of the medical  library association, 95(3), 270‐274.    seeley, h. (2007). developing services for head  injury: obtaining the data.  health  informatics journal, 13(2), 135‐153.    seeley, h.m., hutchinson,p., maimaris, c.,  carroll, g., kirker, s., tasker, r.,  haynes, k., & pickard, j. d. (2006). a  decade of change in regional head  injury care: a retrospective review.  british journal of neurosurgery, 20(1), 9‐ 21.    seeley, h., & hutchinson, p. (2006)  rehabilitation following traumatic  brain injury: challenges and  opportunities. advances in clinical  neurosciences and rehabilitation,  6(2),  22‐28.    seeley, h.m., maimaris, c., carroll, g.,  kellerman, j. & pickard, j.d. (2001).  implementing the galasko report on  the management of head injuries: the  eastern region approach.  emergency  medicine journal, 18(5), 358‐365.     seeley, h.m.,  maimaris, c., hutchinson, p.j.,  carroll, g., white, b., kirker, s.,  tasker, r.c., steward, c., haynes, k.,  hardy, d. & pickard, j.d. (2006).  standards for head injury  management in acute hospitals:  evidence from the six million  61 evidence based library and information practice 2010, 5.2  62 population of the eastern region.  emergency medicine journal, 23(2), 128‐ 132.    seeley, h. & urquhart, c. (2008). action  research in developing knowledge  networks. health informatics journal,  14(4), 279‐296.    seufert, a., von krogh, g. & bach, a. (1999).  towards knowledge networking.  journal of knowledge management, 3(3),  180‐190.    shipman, j. p. (2007). informationists or  information specialists in context  (isic): six years after conception. ifla  journal, 33(4), 335‐339.    shumaker, d. & talley, m. (2009).  models of  embedded librarianship. final report.  (prepared under the special libraries  association research grant, 2007).     http://www.sla.org/pdfs/embeddedli brarianshipfinalrptrev.pdf    sine, w.d., mitsuhashi, h. & kirsch, d.a.  (2006). revisiting burns and stalker:  formal structure and new venture  performance in emerging economic  sectors.  academy of management  journal, 49(1), 121‐32.    society of british neurological surgeons.  safe  neurosurgery. (2000).  london: sbns.    stinchcombe, a. (1965). social structures and  organisations.  in: march, j. (ed.)   handbook of organisations, p.142‐193.  chicago: r mcnally.    swan, j., newell, s., scarborough, j. & hislop,  d. (1999). knowledge management  and innovation: networks and  networking. journal of knowledge  management, 3(4), 262‐275.    urquhart, c., spink, s., thomas, r. & durbin,  j. (2005). systematic assessment of the  training needs of health library staff.  library and information research, 29(93),  35‐42.    urquhart, c., turner, j., durbin, j. & ryan, j.  (2006). evaluating the contribution of  the clinical librarian to a  multidisciplinary team. library and  information research, 30(94), 30‐43.    urquhart, c., turner, j., durbin j. & ryan, j.  (2007). changes in information  behavior in clinical teams after  introduction of a clinical librarian  service. journal of the medical library  association, 95(1), 14‐22.    von krogh, g., nonaka, i., & ichijo, k. (1997).   develop knowledge activists!   european management journal, 15(5),  475‐483.    wagner, k.c., & byrd, g. d. (2004). evaluating  the effectiveness of clinical medical  librarian programs. journal of the  medical library association, 92(1), 14‐ 33.    whitmore, s.c., grefsheim, s.f., & rankin,  j.a. (2008). informationist programme  in support of biomedical research: a  programme description and  preliminary findings of an evaluation.  health information and libraries journal,  25(2), 135‐141.     winning, m.a., & beverley, c.a. (2003).  clinical librarianship: a systematic  review of the literature. health  information and libraries journal,  20(supplement 1), 10‐21.  setting article   alignment of citation behaviors of philosophy graduate students and faculty   jennifer knievel associate professor / director, arts & humanities university of colorado boulder libraries boulder, colorado, united states of america jennifer.knievel@colorado.edu   received: 27 mar. 2013   accepted: 14 june 2013      2013 knievel. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – this study analyzes sources cited by graduate students in philosophy at the university of colorado boulder (ucb) in 55 phd dissertations and master’s theses submitted between 2005 and 2010, to discover their language, age, format, discipline, whether or not they were held by the library, and how they were acquired. results were compared to data previously collected about sources cited by philosophy faculty at ucb, in books published between 2004 and 2009, to identify how closely citation behaviors aligned between the two groups.   methods – citations were counted in the phd dissertations and master’s theses. citations to monographs were searched against the local catalog to determine ownership and call number. comparison numbers for faculty research were collected from a previous study. results were grouped according to academic rank and analyzed by format, language, age, call number, ownership, and method of purchase.   results – graduate students cited mostly books, though fewer than commonly found in other studies. citations were almost entirely of english language sources. master’s students cited slightly newer materials than doctoral students, who in turn cited newer materials than faculty. the library owned most cited books, and most of those were purchased on an approval plan. doctoral students most frequently cited resources outside the discipline of philosophy, in contrast to master’s students and faculty. conclusions – the citation behavior of graduate students in philosophy largely, but not entirely, mirrors that of the faculty. further study of citation behavior in humanities disciplines would be useful. understanding the behavior of philosophers can help philosophy librarians make informed choices about how to spend library funds.     introduction   librarians have long had an interest in better understanding how scholars use library resources. improved understanding of resource use can help librarians make more efficient and effective use of limited acquisitions budgets. this understanding can be somewhat elusive, and has been approached in many different ways. this particular study attempts to take a user-perspective model of looking at resource use employing a citation analysis. rather than looking at an existing library collection and asking how much it gets used, this study looks instead at resources cited by graduate students at the university of colorado at boulder (ucb), and whether or not the library owns them. a similar study of faculty research at the same institution turned up some interesting findings, and it became relevant to question whether or not graduate student research behavior matched that of the faculty (kellsey & knievel, 2012). most citation analyses, for various reasons, focus primarily or exclusively on science disciplines, but there is limited analysis in the literature of humanities fields.   objectives   this study looks specifically at graduate theses and dissertations in the field of philosophy to assess the extent to which the library collection holds the materials cited by philosophy graduate students, as well as whether or not philosophy graduate student research behaviors mirror those of philosophy faculty.   the author expected to find that graduate students in philosophy, as newer entrants to the field, would use newer materials than the faculty. since graduate students request purchase of materials from their librarian less frequently than faculty, the author expected more of the owned titles to be purchased on approval rather than firm orders (this process is further explained below). the author expected a high percentage of the cited materials to be classed within the discipline of philosophy, rather than interdisciplinary. finally, the author expected the breakdown of the percentage of monographs and journals cited, as well as the amount of non-english material used, to roughly match those of the faculty.   since most citation analyses are of scientific fields, this study can help inform collection development decisions in humanities fields, including whether or not to target older materials and foreign languages for weeding, whether to focus on disciplinary content and monographs for collection of new materials, and whether or not approval plans for collection building effectively match materials used by scholars.   literature review   a robust conversation already exists in the literature about the strengths and weaknesses of citation analysis (see, for example, burright, hahn, & antonisse, 2005; hellqvist, 2010; macroberts & macroberts, 2010; mccain & bobick, 1981; waugh & ruppel, 2004; smith, 2003; sylvia, 1998; vallmitjana & sabate, 2008; zipp, 1996). beile, boote, and killingsworth (2004), among others, make persuasive arguments against using citation analysis to develop core title lists for monographs or journals, or as a method of measuring research quality. this study, however, makes use of citation analysis for a different purpose for which the method is more effective, by employing citations as a measurement of the resources local scholars needed, and whether or not the library owns those sources.   existing literature in citation analysis (e.g., iivonen, nygren, valtari, & heikkila, 2009), focuses heavily on journal citations and on the sciences. few analyze the humanities, and even fewer specifically analyze philosophy. john east and john cullars investigate philosophy specifically. cullars (1998) found 15% of citations in philosophy materials were to foreign language resources. he also found that a large majority of citations (85%) were to books, and that a quarter of the cited sources were classed outside the area of philosophy. he concluded that older materials were likely to be considered “recent” in philosophy, including consistent use of materials up to nearly 40 years old. bandyopahyay (1999) also found that philosophy authors cited mostly books, but studies by kellsey and knievel, (2012; 2005) found that philosophy scholars tended to cite far more journals than other humanists, and that most citations were to english language materials (kellsey & knievel, 2004). a study by east (2003) also found almost no citations to non-english books in a year’s worth of citations in two philosophy journals from 2002. a recent study of graduate students included philosophy (kayongo & helm, 2012), and also found that the philosophy students cited newer books and more journals than other humanists.    various authors discuss the importance of evaluating the work of graduate students as a measurement of the usefulness of a library collection (edwards, 1999; kushkowski, parsons, & wiese, 2003; washington-hoagland & clougherty, 2002). thomas (2000) emphasizes the value of looking at local use and local scholars. zipp (1996) and mccain & bobick (1981) both found that graduate student resource use mirrors faculty usage. both studies, however, focus on science disciplines, and measure similarity of research based on lists of cited journals. neither study intended to evaluate whether graduate student research mirrors faculty research in the humanities, nor did they look at language, format, or interdisciplinarity of citations. some studies have found that graduate students tend to cite newer materials than faculty (kushkowski et al., 2003; larivière, sugimoto, & bergeron, 2013; zainab & goi, 1997).   some studies call for more research into humanities sources (sherriff, 2010; smyth, 2011), since data collected and presented in these fields can help to influence collection development policy in libraries. a few interdisciplinary citation studies included some humanities (most notably broadus, 1989; buchanan & herubel, 1994; kayongo & helm, 2012; leiding, 2005; smith, 2003; wiberley & jones, 1994; wiberley & jones, 2000; wiberley, 2003). in general, these studies found that humanists tended to cite more, and older, monographs than scientists and social scientists. smith (2003) found that ownership of monographs was going down over time in the humanities. wiberley (2002; 2003) found that most humanists tended to cite materials within their own discipline, though he did not evaluate philosophy in his studies.   this study attempts to address the question of similarity of graduate student behavior to that of faculty in a humanities discipline. it also attempts to investigate an apparent contradiction of existing studies regarding the dominance of monographs, as well as the use of foreign languages, in the research of philosophy scholars. the results of this study can inform the collection development choices of humanities librarians.   method   this study used a citation analysis approach. the author analyzed all of the dissertations and theses submitted for the department of philosophy at the university of colorado boulder (ucb) between 2005 and 2010. in that time period, there were 26 doctoral dissertations and 29 master’s theses, for a total of 55 source works. the results were compared with 9 faculty books published between 2004 and 2009 by philosophy faculty at the same institution.   most citation analyses are conducted using tools such as web of science. however, in the case of humanities disciplines like philosophy, which are comparatively poorly covered in such tools, most citation analyses have to be hand-counted. as is true of such citation analyses, it was necessary to make several choices about how to categorize citations for the purposes of the study. these decisions were made based on the study goals and characteristics of the resources.   for this study, the author followed the same process used in a 2012 study by kellsey and knievel that analyzed citations in books published by philosophy faculty at ucb during roughly the same time frame, in order to provide comparative results. the 2012 study also provided comparison data for faculty behaviors. each citation was evaluated to determine if it cited a book or a journal, and whether or not the work cited was in english or not in english. works in translation were counted in the language into which they were translated; thus, a citation to an english translation of a french philosophical text was tallied as english, since that was the language of the material actually used. chapters or articles in compiled volumes were counted as books, and counted in the language of the cited chapter or article, not the language of the volume. books with multiple citations in one bibliography (to multiple chapters, for example) were counted only once, since that measures availability, the focus of this study, rather than intensity of use. proceedings were counted as books or journals depending upon how they were published; most were published as books. newspaper articles and encyclopedia entries were counted as articles. as with the study this method emulates, law cases, dissertations, archival materials, unpublished proceedings, and other unpublished works were not counted, since unpublished materials did not provide useful analysis of overlap with the locally held collection. the university of colorado boulder (ucb) is a united states regional and federal depository, as well as a united nations depository, which means that the library automatically receives copies of all documents published by government agencies. hence it can be generally assumed that ucb owns all government documents except in unusual cases of missing or lost materials. thus determining whether or not the library owned cited government documents did not provide the enlightenment this study sought, and government documents were not counted.   many libraries work with book vendors to set up profiles of materials that the library automatically purchases. these arrangements are called approval plans, and have become commonly used in large libraries throughout the united states. this study attempted to determine whether the cited materials were purchased this way, or if they were purchased through firm orders, meaning that a librarian specifically requested a title that was not delivered via the approval plan. firm orders might be the result of specific requests by library patrons, or may simply be the result of librarians noticing a title missing from the approval plan that might be useful.   once each qualifying citation was identified, the books were checked against the local library catalog to determine: 1. if the book is owned by the library, 2. the call number (ucb uses library of congress classification for call numbers), 3. the publication date, and 4. whether it was ordered directly or via approval. in philosophy, as with many other humanities disciplines, different editions or translations are considered different works by scholars in the field. thus, only the exact edition cited was considered a match; if the library owned the same title in a different edition it was not marked as a title owned. many records, especially for titles older than about 15 years, did not indicate the method of purchase, so it could not be determined if the items were purchased directly or via an approval plan.   results   the total number of citations counted was 3,910 in 55 dissertations and theses from ucb, 3,000 of which were in the 26 phd dissertations, with the remaining 910 in the 29 master’s theses. the resulting data were grouped by graduate level to facilitate more meaningful interpretation, and were analyzed in comparison with each other, in the aggregate, and to faculty research. the faculty data for comparison were drawn from 9 faculty books from the same department, which held a total of 2,560 citations.   the average of 71 citations per dissertation is slightly higher than the 59 citations per dissertation found by zainab and goi (1997). average citations per document diverged widely when looked at by student level, with 115 citations per phd dissertation when dissertations are considered alone, and only 31 citations per master’s thesis when looked at alone. both are considerably lower than the average of 284 citations per book by philosophy faculty in the previous study.   language and format   among dissertations and theses, 36% of the citations were of journal articles, while 42% percent of the citations in faculty books were of journal articles (see figure 1). an independent samples t-test revealed a statistically significant difference between these groups. citations in faculty books were more likely to cite journal articles than those in dissertations and theses (t(8.7)=-5.0, p=.001). foreign language citations made up 0.7% of the total citations in the theses and dissertations, and 4.3% of the total citations in faculty books. there was no statistically significant difference between the two groups in the amount of foreign language they cited.   ownership   the ucb library owned 83% of the books cited by graduate students, compared to the 81% of books cited by faculty (see table 1). though these numbers are very close, there is a statistically significant difference in ownership of materials cited by graduates and faculty (t(62)=-5.5, p<.01).   figure 1 language and format of cited works   table 1 ownership of cited works type owned not owned graduate students 83% 17% faculty 81% 19%     purchase method   information about how materials were purchased was not collected by the existing system until 1995. as a result, only materials purchased after that time, regardless of their publication date, included information about whether they were purchased on an approval plan or as firm orders. of the materials cited by graduate students and owned by the library, 43% (897) included purchase information. of the materials for which purchase information was available, 82% were purchased on approval (see table 2). of the materials cited by faculty and owned by the library, a higher percentage, 84%, were ordered on approval. like the results of the owned/not owned data, though these figures are close to those of the previous study of faculty sources in philosophy, there is a statistical significance to the higher number of cited materials that were acquired via firm order for the graduate students (t(62)=-2.8, p=.01).     table 2 purchase method of cited works type approval firm order graduate students 82% 18% faculty 84% 16%     age   the age distribution of citations in theses alone shows highest usage of very new materials (5 years old or less), with a steady decline as materials age (see figure 2). even materials older than 26 years, when grouped together as a whole, proved fewer than the newest materials in master’s theses.   this distribution of age of citations is in contrast with the phd dissertations, in which the largest age group of materials cited is the 26+ year range. looking at 5 year increments up to 25, the largest age group for phd dissertations is the 6-10 year range. additionally, the dissertations cited a higher percentage of materials in all of the older ranges as well, showing a general adoption and use of older materials in dissertations than in theses (see table 3). faculty research follows this same pattern, using materials even older than those used for the dissertations (see figure 3). faculty publications show a much more pronounced jump in the 26+ age range, but are similar to the phd dissertations in that the largest 5 year span is the 6-10 year range (see table 4).   consistent with that observation is the difference in average publication date of cited materials, which was newer for theses than for dissertations, which in turn were newer than faculty materials (see table 5).   interdisciplinarity   in order to assess the interdisciplinarity of cited sources in the philosophy theses and dissertations, the library of congress call numbers were recorded for each cited book owned by the library, and then counted in groups. anything in the library of congress classification system (lccs) “b,” which includes philosophy and religion, was considered “in discipline.” everything else was considered “out of discipline.”     figure 2 age of works cited by graduate students     table 3 statistical tests: age of works cited by graduate students age of materials master’s theses phd dissertations t-value p-value 0-5 years 26% 17% t(39)=-3 p<.01 6-10 years 20% 20% t(32.3)=-5 p<.01 11-15 years 15% 17% t(30.7)=-4.8 p<.01 16-20 years 10% 12% t(32.3)=-4.6 p<.01 21-25 years 8% 10% t(29.8)=-5 p<.01 26+ years 21% 25% t(26.6)=-3.4 p<.01     of the owned books cited in the phd dissertations alone, a minority, only 42%, classified as in discipline while 58% classified as out of discipline. in the master’s theses, that breakdown was reversed, with 56% of citations in discipline (see figure 4). phd dissertation writers were more likely to cite materials published outside of the discipline than master’s thesis writers (t(33.8)=-4, p<.01).   a more detailed breakdown of the call numbers of cited works shows that the majority of out of discipline citations for both theses and dissertations is in the social science range (lccs areas g-k). after social science, the next largest discipline cited was science (q-v), though only a third as many citations were in this area. even so, science alone represented more than literature (p) and history (c-f) combined, with arts and education (l-n) and reference (a and z) almost completely absent (see figure 5).   this particular finding was dissimilar from research done with faculty citations, which found a significantly higher percentage of faculty citations within the discipline (see figure 6; t(8.5)=-4, p<.01).   discussion   language and format   of the few existing analyses of citations in humanities dissertations and theses, most ask whether scholars cited more books or journals. most other studies found a higher percentage of citations to monographs. however, inconsistent counting methods make these numbers difficult to compare, since some other studies counted duplicate citations more than once, or included government documents as books, while this study did not. in this study, though citations to monographs represent a majority among both groups, this percentage is considerably lower than is typically seen in other humanities studies or in older studies of philosophy (cullars, 1998). this higher percentage of citations to journals is consistent with more recent studies of philosophy, and may reflect a transition of the discipline toward being a more journal-reliant field than it once was (kellsey & knievel, 2012; 2005). this may have an influence on how philosophy librarians distribute their funding for materials, since it may be prudent to devote more attention to serials in order to match available resources with resource use.       figure 3 age of works cited     table 4 statistical tests: age of works cited by graduate students and faculty age of materials graduate students faculty books t-value p-value 0-5 years 19% 10% t(62)=-2.1 p=.04 6-10 years 20% 16% t(62)=-4.6 p<.01 11-15 years 16% 15% t(62)=-4.9 p<.01 16-20 years 12% 13% t(62)=-5.8 p<.01 21-25 years 10% 11% t(8.6)=-3.5 p=.01 26+ years 24% 34% t(8.6)=-3.4 p=.01                         table 5 average publication date of cited works type average pub date master’s theses 1991 phd dissertations 1988 faculty books 1984     a particularly unusual result of this study is the near absence of any foreign language citations, which made up less than 1% of the total citations. this number is much lower than some studies have shown (cullars, 1998), and yet is more consistent with some other recent studies that have shown low usage of foreign language materials by philosophy scholars (east, 2003; kellsey & knievel, 2012; kellsey & knievel, 2004). the philosophy degree at ucb has only a provisional language requirement, in which language study is required on a case-by-case basis, if the student’s topic of interest necessitates it. this, combined with the availability of translated material for study, may have an influence on the very low usage of non-english material. in addition, there is a local emphasis on applied ethics, which is a niche of philosophy that tends to eschew continental philosophical approaches where foreign language might play a larger role (cullars, 1998).     figure 4 interdisciplinarity of works cited by graduate students       figure 5 cited discipline by call number classification     figure 6 interdisciplinarity of cited works   the language and format distribution of the materials cited by graduate students mirrors very closely those cited by faculty. this finding supports zipp’s (1996) analysis that graduate student research is reflective of faculty research, but other significant factors discussed below need to be assessed to determine whether graduate student citation behavior really does align with faculty behavior in the humanities.   ownership   between 81 and 83% of cited monographs were owned locally. this number can be interpreted in various ways; 83% is very high, and clearly the library is collecting a large majority of the sources used by students. at the same time, this is an indication that nearly 1 in every 5 sources are being obtained by the students or faculty through interlibrary loan (ill) or some other mechanism, which, from the user perspective, may feel like a burden. the not-owned material may be partly explained by the number of sources cited from outside the field of philosophy, which will be further addressed below. another explanation may be a local practice of not purchasing volumes of collected articles that have been previously published elsewhere; students may not be finding the previously published versions that are in alternative locations, and instead are acquiring the volumes of collected articles. it is worth reiterating here that only exact editions were considered a match. many of the not-owned materials were held in different editions. these findings may indicate a need to purchase more duplicative material, such as the collected works, since there is reason to suspect that students and faculty are still using the collected works but attaining them through borrowing or other means. the ownership percentages are much higher than the un-weighted owned percentage of 63% of cited humanities materials in a similar study by kayongo & helm (2012). it is hard to establish a bench-mark of what percentage of cited materials should be owned by the local library. as a result of budget pressures, many libraries are moving away from the “just-in-case” philosophy of collection development, which would logically drive down the percentage of cited materials that are already owned.   purchase method   since phd dissertation topics tend to be narrow and relatively unexplored, it is logical that the library approval plan would not necessarily reflect the newer topics, so 82% seems like a reasonable percentage of titles to be ordered on approval. the faculty are more established scholars, and tend to remain at the institution for longer periods than the students. thus it is easier to establish approval profiles to provide a higher percentage of the materials of interest to the faculty. also, since more of the materials cited by faculty fall into the philosophy classification (see below), it is easier for a subject librarian to ensure coverage in the collection of topics of interest to the philosophy scholars.   age   of the three groups, master’s theses cited the newest materials, phd dissertations cited slightly older materials, and faculty books cited the oldest materials of the three. this is consistent with other studies that have shown that graduate students user newer sources than faculty, and may be a result of the fact that graduate students, by their nature, are performing comprehensive literature reviews for their projects, while faculty are building on a more mature research agenda and may be less aggressive in identifying new related literature. the results of this study are consistent with other humanities studies in showing that humanists use older materials than scientists or social scientists. librarians should take into account these differences of field of study before making choices about materials to target for weeding projects, or assuming that humanities materials lose their value as a direct function of their age, as may be more true in scientific disciplines.    interdisciplinarity   surprisingly, faculty authors were the most strict adherents to their own disciplinary material of all the groups studied. phd dissertations demonstrated the weakest tie to disciplinary material, as this was the only group for whom fewer than half of the cited sources were classified in philosophy. in this way, graduate students and faculty show more divergence in the materials they choose to cite in their research. if this citation pattern were to continue as these graduate students become members of philosophy faculties, this could have an influence on how librarians want to define their collections. in order to address the current need of graduate students, as well as the potential future needs of faculty, librarians should also be reaching across traditional disciplinary definitions to ensure that the library is collecting relevant materials in disciplines related to philosophy. in this study, those relationships are in areas not traditionally associated with philosophy: the social sciences and the sciences, rather than the other humanities. thus it may be useful for philosophy librarians to build new understandings with other librarians to ensure sufficient breadth of coverage in a library collection.   conclusion   this study took a user-perspective approach to analyzing resource use by philosophy scholars. building on the earlier study of faculty research behaviors, this study analyzed citations in philosophy master’s theses and phd dissertations from the university of colorado boulder for their format (monograph or journal), language (english or other), age, presence in the local library, method of acquisition (approval or firm order), and subject classification.   this study found that in most ways except interdisciplinarity, graduate student research mirrored faculty research. in contrast to some earlier studies, this study found almost no use of foreign language sources by philosophy scholars. generally, the percentage of cited sources owned by the library was high, over three-quarters, and of the sources with purchasing information, more than three quarters had been purchased on approval plans. the majority of citations were to monographs, with phd dissertations citing roughly two thirds monographs, and master’s theses slightly less. master’s theses cited somewhat newer materials than phd dissertations, which in turn cited newer materials than faculty publications analyzed in a previous study. the most notable separation between faculty and graduate student research behaviors was that graduate student research cited a much higher percentage of materials classed outside of philosophy than faculty research did.   further similar studies of both faculty and graduate students in other humanities disciplines would be of interest to assess whether the results found in this study reflect an average result or an outlier.   results of this study can help to develop the picture of how humanities scholars use library resources. it can be useful for humanities librarians as they evaluate their collection development policies and practices related to journals, foreign language, and approval plans, as well as provide some data to help determine policies and practices related to age and language for weeding of materials.     references   bandyopadhyay, a. k. (1999). literature use pattern in doctoral dissertations of different disciplines. international information communication and education, 18(1), 33-42.   beile, p. m., boote, d. n., & killingsworth, e. k. (2004). a microscope or a mirror?: a question of study validity regarding the use of dissertation citation analysis for evaluating research collections. journal of academic librarianship, 30(5), 347-353. broadus, r. n. (1989). use of periodicals by humanities scholars. serials librarian, 16(1), 123-131. doi: 10.1300/j12v1601_10   buchanan, a. l., & herubel, j.p. v. m. (1994). profiling phd dissertation bibliographies: serials and collection development in political science. behavioral and social sciences librarian, 13(1), 1-10. doi: 10.1300/j103v13n01_01   burright, m. a., hahn, t. b., & antonisse, m. j. (2005). understanding information use in a multidisciplinary field: a local citation analysis of neuroscience research. college & research libraries, 66(3), 198-210.   cullars, j. m. (1998). citation characteristics of english-language monographs in philosophy. library & information science research, 20(1), 41-68.   east, j. w. (2003). australian library resources in philosophy: a survey of recent monograph holdings. australian academic and research libraries, 34(2), 92-99.   edwards, s. (1999). citation analysis as a collection development tool: a bibliometric study of polymer science theses and dissertations. serials review, 25(1), 11-20.   hellqvist, b. (2010). referencing in the humanities and its implications for citation analysis. journal of the american society for information science and technology, 61(2), 310-318.   iivonen, m., nygren, u., valtari, a., & heikkila, t. (2009). library collections contribute to doctoral studies: citation analysis of dissertations in the field of economics and administration. library management, 30(3), 185-203.   kayongo, j., & helm, c. (2012). relevance of library collections for graduate student research: a citation analysis study of doctoral dissertations at notre dame. college & research libraries, 73(1), 47-67.   kellsey, c., & knievel, j. (2012). overlap between humanities faculty citation and library monograph collections 2004–2009. college & research libraries, 73(6), 569-583.   kellsey, c., & knievel, j. e. (2004). global english in the humanities? a longitudinal citation study of foreign-language use by humanities scholars. college & research libraries, 65(3), 194-204.   knievel, j. e., & kellsey, c. (2005). citation analysis for collection development: a comparative study of eight humanities fields. library quarterly, 75(2), 142-168. doi: 10.1086/431331   kushkowski, j. d., parsons, k. a., & wiese, w. h. (2003). master's and doctoral thesis citations: analysis and trends of a longitudinal study. portal: libraries and the academy, 3(3), 459-479.   larivière, v., sugimoto, c. r., & bergeron, p. (2013). in their own image? a comparison of doctoral students' and faculty members' referencing behavior. journal of the american society for information science and technology, 64(5), 1045-1054. doi:10.1002/asi.22797   leiding, r. (2005). using citation checking of undergraduate honors thesis bibliographies to evaluate library collections. college & research libraries, 66(5), 417-429.   macroberts, m. h., & macroberts, b. r. (2010). problems of citation analysis: a study of uncited and seldom-cited influences. journal of the american society for information science & technology, 61(1), 1-12. doi:10.1002/asi.21228   mccain, k. w., & bobick, j. e. (1981). patterns of journal use in a departmental library: a citation analysis. journal of the american society for information science, 32(4), 257-267. doi: 10.1002/asi.4630320405   sherriff, g. (2010). information use in history research: a citation analysis of master's level theses. portal: libraries and the academy, 10(2), 165-183. doi: 10.1353/pla.0.0092   smith, e. t. (2003). assessing collection usefulness: an investigation of library ownership of the resources graduate students use. college & research libraries, 64(5), 344-355.   smyth, j. b. (2011). tracking trends: students' information use in the social sciences and humanities, 1995-2008. portal: libraries and the academy, 11(1), 551–573. doi: 10.1353/pla.2011.0009   sylvia, m. j. (1998). citation analysis as an unobtrusive method for journal collection evaluation using psychology student research bibliographies. collection building, 17(1), 20-28.   thomas, j. (2000). never enough: graduate student use of journals-citation analysis of social work theses. behavioral & social sciences librarian, 19(1), 1-16. doi: 10.1300/j103v19n01_01   vallmitjana, n., & sabate, l. g. (2008). citation analysis of ph.d. dissertation references as a tool for collection management in an academic chemistry library. college & research libraries, 69(1), 72-81.   washington-hoagland, c., & clougherty, l. (2002). identifying the resource and service needs of graduate and professional students: the university of iowa user needs of graduate professional series. portal: libraries & the academy, 2(1), 125-143. doi: 10.1353/pla.2002.0014   waugh, c. k. & ruppel, m. (2004). citation analysis of dissertation, thesis, and research paper references in workforce education and development. journal of academic librarianship, 30(4), 276-284.   wiberley, s., & jones, w. g. (1994). humanists revisited: a longitudinal look at the adoption of information technology. college and research libraries, 55(6), 499-509.   wiberley, s., & jones, w. g. (2000). time and technology: a decade-long look at humanists' use of electronic information technology. college and research libraries, 61(5), 421-431.   wiberley, s. e. (2002). the humanities: who won the '90s in scholarly book publishing. portal: libraries and the academy, 2(3), 357-374.   wiberley, s. e. (2003). a methodological approach to developing bibliometric models of types of humanities scholarship. library quarterly, 73(2), 121-159.   zainab, a. n., & goi, s. s. (1997). characteristics of citations used by humanities researchers. malaysian journal of library and information science, 2(2), 19-36.   zipp, l. s. (1996). thesis and dissertation citations as indicators of faculty research use of university library journal collections. library resources & technical services, 40(4), 335-342. doi: 10.5860/lrts.40n4.335   article   enhancing access to e-books   karen harker collection assessment librarian university of north texas libraries denton, texas, united states of america email: karen.harker@unt.edu   catherine sassen principal catalog librarian university of north texas libraries denton, texas, united states of america email: catherine.sassen@unt.edu   received: 30 sept. 2014  accepted: 16 dec. 2014        2015 harker and sassen. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – the objective of the study was to determine if summary notes or table of contents notes in catalogue records are associated with the usage of e-books in a large university library.   methods – a retrospective cohort study, analyzing titles from three major collections of e-books was employed. titles were categorized based on the inclusion of the marc 505 note (table of contents) or marc 520 note (summary) in the catalogue record. the usage was based on standardized reports from 2012-2013. the measures of usage were the number of titles used and the number of sections downloaded. statistical methods used in the analysis included correlations and odd ratios (ors). the usage measures were stratified by publication year and subject to adjust for the effects of these factors on usage.   results – the analysis indicates that these enhancements to the catalogue record increase usage significantly and notably. the probability of an e-book with one of the catalogue record enhancements being used (as indicated by the or) was over 80% greater than for titles lacking the enhancements, and nearly twice as high for titles with both features. the differences were greatest among the oldest and the most recently published e-books, and those in science and technology. the differences were least among the e-books published between 1998 and 2007 and those in the humanities and social sciences.   conclusion – libraries can make their collections more accessible to users by enhancing bibliographic records with summary and table of contents notes, and by advocating for their inclusion in vendor-supplied records. introduction   while librarians may have suspected that a certain percentage of their collections would be more heavily used, it was the ground-breaking work in the 1960s and 1970s that brought the issue to light (morse & chen, 1975; trueswell, 1968).  trueswell (1969) found that about 20% of the collections at university of massachusetts and mount holyoke accounted for about 80% of the titles used. galvin and kent’s (1977) landmark study showed that about 40% of a large academic collection at the university of pittsburgh had never been used. more recently, a study from auburn university demonstrated that these principles of print book circulation applied as well to electronic book usage (best, 2008). but for all the circulation studies that quantified the distribution of usage, there have been relatively few studies that examined why some books are used and others are not.    the university of north texas libraries embarked on a pilot of patron-driven acquisitions (pda) through a single vendor in 2012. until then, the usage of individual e-books had not received much attention by the university librarians. it was primarily due to the direct association of usage to cost that we started carefully tracking this information. the program was quite successful, in that the percentage of titles used at least once was greater than for those e-books that librarians had previously purchased. furthermore, only 18% of librarian-selected e-books were used again after their initial use, while 57% of pda titles were used multiple times.  we began to wonder about factors associated with patron usage. e-book titles were made available through the libraries’ online catalogue, as well as through the vendor’s platform, and we determined that the primary source for discovering these titles was the catalogue. thus, patrons who searched the catalogue selected titles based solely on the contents of the marc records. we considered what characteristics of the marc records were associated with titles that were selected. two characteristics were noticeable: inclusion of tables of contents and summaries. is inclusion of tables of contents or summary statements in the marc record associated with a greater likelihood of usage and the number of uses?   literature review   the enhancement of bibliographic records with tables of contents and other features dates back to the introduction of online catalogues.  one of the earliest catalogue use studies that included this topic was the online catalog public access project (matthews & lawrence, 1984). the council on library resources (clr) conducted this study in cooperation with the library of congress (lc), oclc, the research libraries group (rlg), the university of california division of library automation, and joseph matthews and associates, a library consulting firm (matthews, 2014). information was gathered from thousands of individuals through surveys and focus group interviews. when asked about the most desired enhancements to online catalogue records, library users specified that they would like to be able to search a book’s table of contents, summary, and index. (markey, 1983, p. 141).    much later, in 2008, oclc researchers administered a pop-up survey to worldcat users and received 11,151 responses (calhoun, cantrell, gallagher & hawk, 2009, p. 7). worldcat users were asked, "what changes would be most helpful to you in identifying the item that you need?", and the top five responses included summaries/abstracts (18%) and tables of contents (18%) (calhoun et al., 2009, p. 13). oclc researchers also covered this topic in a series of focus group interviews, comprised of eight undergraduates, eight "casual searchers", and finally, eight “scholars,” including graduate students and faculty members (calhoun et al., 2009, p. 6). the focus group participants indicated that summaries, abstracts and tables of contents are helpful for "a quick verification to determine if it is worth their time to even look at an item" (calhoun et al., 2009, p. 17).   libraries attempted to address this issue as early as the 1970s through bibliographic record enhancement projects. for the subject access project, led by cochrane (1978) at the university of toronto, researchers created a test database of 1,979 catalogue records enhanced with table of contents and index information. when they performed 90 searches, they retrieved 56 relevant items in the enhanced database compared to 14 relevant items in the regular database.  users of the enhanced database reported that they were “able to find some items which would be impossible to locate with l.c. [library of congress] subject headings” (cochrane, p. 86).  cochrane identified several benefits to searching enhanced records, including “greater access to the books with relevant information” and “greater precision, insuring fewer non-relevant items in the search output” (p. 85).   researchers have tried to assess the value of enhanced catalogue records by studying the circulation rates of the items. in four separate experimental studies, researchers measured the circulation rate for materials before and after their catalogue records were enhanced.  in the first study, no association was found between catalogue records enhanced with only table of contents notes and higher circulation rates (knutson, 1991). this study involved 291 records divided into 3 groups: records enhanced with subject headings and table of contents information; records enhanced with table of contents only; and records in the control group that received no enhancements. the lack of association of enhancements with circulation rates may be explained by the small sample size. in three other experimental studies, researchers found that the circulation of titles increased from 5% to 25% after record enhancement (dinkins & kirkland, 2006; faiks, rademacher & sheehan, 2007; chercourt & marshall, 2013).    in five other investigations, researchers conducted retrospective cohort studies to determine if library materials with enhanced catalogue records circulated more than those with unenhanced records (morris, 2001; madarash-hill & hill, 2004; madarash-hill & hill, 2005; tosaka & weng, 2011; kirkland, 2013). in each of these studies, catalogue record enhancements were associated with increased circulation rates. tosaka and weng (2011) conducted one of the largest of these studies at the college of new jersey library, involving 88,538 titles in 4 subject fields (history, social sciences, language and literature, and science and technology).  the researchers found that titles published between 1990 and 2004 with enhanced records had 30% to 50% higher circulation than those with unenhanced records. however, record enhancement had no effect on the circulation of titles published between 2005 and 2008. the researchers found a correlation between recent publication dates and circulation, and suggested that library users prefer more recent publications (tosaka & weng, 2001, p. 420). the researchers also found that table of contents notes were associated with higher circulation, but not summary notes.   from the above research, it is apparent that tables of contents provide additional keywords that users need to identify resources. this enhancement increases user access to library resources.    none of the nine prior studies examined the usage of e-books in relation to catalogue record enhancements. this study was designed to fill that research gap.     aims   we began our investigation by asking why library users select certain e-books but not others after viewing their catalogue records. based on the information gleaned from the literature review, we focused our study on the following research questions:   •        what is the effect of table of contents or summary statement in the catalogue record on the number of uses of the e-book and the probability of being used at least once? •        what is the effect of year of publication and subject on total uses, as well as on the probability of being used? •        what is the effect of these catalogue record enhancements on total uses and the probability of a title being used at least once, controlling for publication year and subject?   from these questions, we developed these hypotheses:   •     titles with either the table of contents or a summary in the catalogue record would have more uses than titles with neither of these features, controlling for publication year and subject. •     titles with both table of contents and summary in the catalogue record would have more uses, controlling for publication year and subject. •     titles with either table of contents or summary in the catalogue record would have a greater probability of being used, controlling for publication year and subject. •     titles with both table of contents and summary in the catalogue record would have a greater probability of being used, controlling for publication year and subject.   methods   a retrospective cohort study design was employed.   data collection   three e-book collections were included in this study: ebrary, ebsco ebooks and netlibrary.  the ebrary collection was available on the ebrary publishing platform. the ebsco ebooks and netlibrary collections were available on the ebsco publishing platform. the marc records were downloaded from the integrated library system into a set of spreadsheets. the unit of analysis was the marc record, which represented a unique e-book title from a particular vendor. while no attempt was made to analyze the distribution of titles by language, it is clear that the vast majority of the titles were in the english language. usage data were collected from the platforms in the form of counter book reports, either br1 or br2. the counter br1 (supplied by ebsco) provides the “number of successful title requests,” while the br2 (supplied by ebrary) reports the “number of successful section requests” (counter, 2008).  the former reports the number of titles used, while the latter reports the number of chapters or other sections downloaded. while these are not the same measures, most platforms provide only one or the other. therefore, “use” in this study is defined as either a request for an entire e-book or the downloading of chapters or other sections. the total usage for calendar years 2012 and 2013 was used as the measure of total usage. titles with a usage of one or more were flagged as titles used. these two measures, total usage (counts) and titles used (binomial), were the dependent variables in the analysis. only titles continuously available from january 2012 through december 2013 were included in the study.   other data collected included the marc 505 (table of contents) and the marc 520 (summary) fields. the records were classed in the following categories, based on the inclusion of either or both of these fields (see table 1).     table 1 categories of catalogue record enhancements (cre) mutually-exclusive categories non-mutually-exclusive categories ·        neither table of contents (toc) nor summary fields. ·        toc only. ·        summary only. ·        both toc and summary.   ·        neither toc nor summary fields. ·        toc. ·        summary. ·        either toc or summary, but not both. ·        either toc or summary, or both. ·        both toc and summary.       when considering the inclusion of the enhanced content as a single categorical variable, the categories must be mutually exclusive. however, this would fail to determine the impact of having one or the other, regardless of which one. therefore, we conducted paired-comparisons analysis using the non-mutually exclusive categories, and categorical and multivariate statistical analyses using the mutually exclusive categories. the other independent variables were the publication year, as indicated in the catalogue record (marc 260 subfield c), and broad subject categories based on library of congress (lc) class.   statistical analysis   descriptive statistics were generated to evaluate the distribution of the catalogue records for each of the variables. these analyses included simple counts of records and percentages by categories, and means, medians and skew of distributions of continuous data (year of publication and  usage).  bivariate analyses were then conducted between the various factors to identify relationships.  this included cross-tabulations of categorical data, comparisons of means and distributions for continuous data, and correlations of the variables.    the statistical tests were selected based on the distributions of the data. given that the interval data (year of publication and usage) were not normally distributed, non-parametric tests were used. statistical tests of inference that are based on assumptions about the population (such as population distribution) are called parametric statistical analyses. these methods are quite commonly used, and include such well-known methods as the student’s t-test and linear regression. these methods, however, can lead to invalid results when the data does not conform to these assumptions about the distributions, such as categorical data. non-parametric statistical tests of inference do not rely on assumptions about the distribution. these “distribution free” methods are most valid for categorical data or interval data that do not have the normal distribution or the “bell curve”.    odds ratios (ors) were calculated for the cross-tabulations of titles used by each of the categories of catalogue record enhancement (cre). the or provides a simple measure of association of the exposure (the level of cre) and the binary outcome (used or not used).  it ranges from zero to infinity, and a value of one indicates no difference in outcome between the two exposure groups (has or does not have the cre). a value greater than one indicates a greater probability of use for a title with that level of cre. a value less than one indicates that the title without that feature has greater probability of being used. the or includes a 95% confidence interval (95% ci).   the or is calculated by dividing the odds of titles used among those with a catalogue record enhancement by the odds of titles used among those without that feature (see equation 1). the result is a positive number ranging from zero to infinity; the closer to 1.0, the more similar the probability of usage is between the two groups. an or between 0 and 1 indicates that the items lacking tables of contents (tocs) are more likely to be used, while an or above 1 indicates items with the tocs have greater probability of being used. a 95% confidence interval is used to test the or against random variation. if the interval spans above and below 1.0, then there is too much variation in the measure for the estimate to be valid. ors are measures of comparison of two non-overlapping groups. figure 1 provides a summary of the groups that are compared.     outcome of interest (used) not outcome of interest (not used) exposed (has cre) a b not exposed (does not have cre) c d figure 1 table (2x2) of exposures and outcomes     equation 1 odds ratio     for comparing total uses across the factors, the kruskal-wallis rank test was used to test for significance. this is a non-parametric test of significance of differences in the distributions of uses between the two or more groups (with and without the cre).    to control for differences due to subject coverage, the titles were categorized into one of three broad disciplines: humanities, social sciences and stem (science, technology, engineering and mathematics). these categories are similar to those used by michael levine-clark (2014) in his recent analysis of e-books from multiple collections. the only difference was the inclusion of ranges on geography (lc class g), which we split between stem and social sciences (see table 2). differences in the mean publication year and the inclusion of catalogue record enhancements between these broad disciplines were examined. in addition, we analyzed differences in the mean usage and ors of being used to determine the effect of discipline on usage.   to control for year of publication, the titles were grouped by year, with roughly the same number of titles in each group, and the mean uses, as well as ors of titles used, were compared between the groups. differences in the means and the ors between the groups were indications of the amount of influence the year of publication had on the effects of catalogue record enhancements on usage.     table 2 broad disciplines humanities social sciences stem b (philosophy, religion, excl. bf) c, d, e, f (history) m (music) n (fine arts) p (language and literature bf (psychology) gn-gv (human geography) h (social sciences) j (political science) k (law) l (education) u, v (military, naval sciences) g, ga-gf (geography) q (science) r (medicine) s (agriculture) t (technology)   * p<0.05     results   the data set used in the final analysis included a total of 76,467 records from 3 collections.  about 32% of the titles (24,050) had tocs in the catalogue records, and only about 8% of the titles (6,174) included summaries. the distribution of titles with tocs and summaries was not equal across platforms. similarly, the collections varied in the mean and median publication years, with titles from the ebsco or netlibrary sets more than ten years older than the ebrary titles.   usage by catalogue record enhancements (cre)   there were apparent differences in the publication year, mean usage, and number of titles used between the groups that have and do not have the enhanced catalogue content (see table 3). those with the added content were more recently published, and had much higher mean usage and greater rate of titles used. a wide range of analytical techniques was used to determine if these differences were due to publication year, subject, or chance.   the simplest method was a cross-tabulation of titles used by the exposure (i.e., the enhanced catalogue content). using such a method, we can provide an or measure that describes the probability that a title having the exposure (toc or summary in catalogue record) was used against a title lacking the exposure.    most of the ors were significant, favouring the inclusion of at least one of the catalogue record enhancements, and the highest or favoured having both (see table 3). the analysis shows that a title with at least one of the features was over 80% more likely to be used than those with neither, and over twice as likely if it has both features. titles having the keyword summary only, however, were not significantly more likely to be used. it is clear that tocs and summaries in the catalogue record had a positive effect on the title being used at least once. the publication year in all of these groups, however, was also associated with the inclusion of catalogue record enhancements, and must be taken into consideration before concluding that the enhancements were, indeed, associated with increased usage.     table 3 usage by catalogue record enhancement cre groups # titles in the cre group # titles not in the cre group uses per title in the cre group uses per title not in the cre group % titles used in cre group % titles not in the cre group used or (95% ci)) has toc only 19,704 56,763 11.3 3.8 26.4% 16.2% 1.9* (1.8-1.9) has toc 24,050 52,417 14 1.9 28.3% 14.4% 2.3* (2.25-2.4) has summary only  1,828 74,639 20.9 4.4 32.0% 17.6% 2.2*  (2.1-2.3) has summary 6,174 70,293 8 5.7 20.0% 18.8% 1.1 (0.96-1.2) has either but not both 21,532 54,935 11 3.7 25.8% 16.0% 1.8* (1.75-1.9) has either or both 25,878 50,589 13.6 1.7 27.7% 14.2% 2.3* (2.2-2.4) has both 4,346 72,121 26.3 4.5 37.0% 17.7% 2.7* (2.6-2.9) * p<0.05     distribution by publication year   because the publication year had been identified as a key factor in usage of items, we evaluated the distribution of e-books by publication year (tosaka & weng, 2011, p. 419; morris, 2001, p. 34).  interestingly, the distribution of publication years was bimodal (two “humps”) (see figure 2). there is a drop in the number of titles between 2003 and 2008.  this, combined with the severe skewing to the left, indicated that statistical tests based on normal distributions would not be appropriate. the titles were then grouped by publication year into four approximately equal groups based on their distribution: <=1998, 1999-2001, 2002-2007, >=2008.   the distribution of catalogue enhancements varied across the publication year groups. generally, the more recently published books had greater rates of these enhancements, as well as lower rates of only one of the enhancements (see table 4).   similar to tosaka and weng (2011, p. 417-418), we determined that the e-book records with tocs and those with summaries tended to be more recent than those without (see table 3). to determine if this variation could be random, we used the non-parametric tests, mann-whitney u and kolmogonov-smirnov tests, comparing the mean publication years between those with and without the tocs, and with and without summaries. in addition, we ran kruskal-wallis one-way anova, comparing the distribution of publication years across all four groups. the null hypothesis for this test was that the distributions of publication years were similar for those with and lacking content enhancements in the catalogue records.      figure 2 distribution of titles by publication year     table 4 distribution of cre by publication year categories <1998 1998-2001 2002-2007 >=2008 has toc 5.6% 8.3% 26% 60.1% has summary 8.4% 11.3% 22.5% 15% has toc only 6.5% 9.4% 27.4% 56.7% has summary only 25.5% 30.7% 10.6% 33.1% either toc, not both 8.1% 11.2% 26% 54.7% either or both 7% 9.9% 24.9% 54.2% both toc and summary 1.4% 3.4% 19.6% 75.7%     the results indicated that the differences in year of publication between having and not having the content enhancements were not likely due to chance (p<0.001) (see table 5). this clearly indicated that publication years were different between the two exposed groups of bibliographic records.      table 5 median publication year by groups   # titles median publication year overall 73,936 2002 has neither 48,104 (65%) 1999 has summary only 1,789 (2.4%) 2000 has toc only 19,697 (26.6%) 2009 has both 4,346 (6%) 2010     table 6 usage by publication year group publication year group mean uses % used <1998 0.45 1.6% 1998-2001 0.89 2.2% 2002-2007 3.09 2.7% >=2008 18.31 4.6%     table 7 usage by broad discipline n (%) mean uses % used or (95% ci) humanities 28,735 (38%) 4.5 2.4% 1.2 (1.1-1.3) social sciences 29,778 (39%) 6.1 3.1% 0.8 (0.73-0.87) stem 17,684 (23%) 7.1 2.6% 1.07 (0.96-1.2) overall 76,197 (100%) 5.7 2.7%     as bucknell (2010, p. 128-129) documented, we found that publication year was also a factor in usage of e-books (see table 6). the simplest way to determine this was through correlation analysis. because of the severe skewedness of both usage and publication year, the statistic used was spearman's correlation factor, which, although statistically significant, was quite low (spearman's r=0.191) on a scale of -1 to +1. analyzing usage by publication year group revealed more substantive differences.   the usage of these groups was compared using the non-parametric statistical test, kruskal-wallis one-way anova, setting the significance level at 0.01 and the confidence interval at 99%. this test allows comparisons of usage across multiple groups. the null hypothesis in this analysis was that the usage would be similar across all four publication year groups. this null hypothesis was rejected (p<=0.001), indicating that usage was clearly associated with publication year; more recent titles garnered more uses than earlier titles.    given the association of publication year on the exposure variables (catalogue record enhancements) and on the outcome (usage), it was clear that publication year could confound the effect of the catalogue record enhancements on usage, making it difficult to tell the difference between the effects of the age of the book and those of the catalogue record enhancements   usage across broad disciplines   as mentioned above, the titles were categorized into one of three broad disciplines based on their lc classification. there was a negligible set (270 titles, 0.4%) that were not categorized due to various reasons. the distribution of titles across these three categories was not even, with just under 40% in both humanities and social sciences, and just over 20% in stem. usage of titles within each category varied slightly (see table 7), with stem titles having the most mean uses per title (7) and social sciences titles having a greater percentage of titles used at least once (3%).  while the ors of the paired-comparisons of these groups were statistically significant, the size of their effect (20% greater or lesser odds of usage) was not very notable. it appears that broad discipline may have had a slight effect on usage.   adjusting for publication year and broad discipline   to understand the effect of publication year and broad discipline on the odds of being used, we stratified the analysis by publication year group, and separately discipline, and compared the ors (see table 10). our hypothesis was that the or of use would increase across all publication year groups with more catalogue content enhancement. only the earliest and most recent titles (publication year either before 1998 or after 2007) demonstrated this pattern (see table 8). by adjusting for publication year, there did not appear to be a clear association between the catalogue record enhancements and use, except for oldest and most recent publications.   the inclusion of the catalogue record enhancements varied across the three broad disciplines, with social sciences having the most and stem having the least percentage of enhanced records (see table 9).    to determine the combined effect of catalogue record enhancements and broad discipline on usage, we examined the ors comparing rates of usage by catalogue record enhancement for each discipline separately. if the ors did not differ substantially across the disciplines, then the effect of discipline was minimal. it was apparent that the effect of catalogue record enhancements on usage did not vary between the humanities and social sciences; however, the effect of having tocs or both was notably greater on stem titles (see table 10).     table 8 odds ratios of usage by publication year group categories <1998 1998-2001 2002-2007 >=2008 has toc 1.6 1.3 1.1 1.44 has summary 2.2 0.8 0.7 1.8 has toc only 1.7 1.3 1.2 0.9 has summary only 2.3 0.9 1.2 1.4 either toc, not both 1.9 1.2 1.2 0.9 either or both 1.9 1.2 1.1 1.6 both toc and summary 1.0 0.6 0.6 1.8     table 9 distribution of cre by discipline catalogue record enhancement humanities social sciences stem overall has toc only 27% 29% 20% 26% has summary only 3% 2% 3% 2% has toc 32% 35% 24% 32% has summary 8% 8% 8% 8% has either, not both 29% 31% 23% 28% has either or both 34% 37% 28% 34% has both 5% 7% 5% 6%     table 10 odds ratios of usage by cre and discipline catalogue record enhancement humanities social sciences stem overall has toc only 1.5 1.5 2 1.86 has summary only 1.6 1.7 2.6 1.08 has toc 1.8 1.8 2.8 2.34 has summary 2.1 1.9 1.1 2.19 has either, not both 1.5 1.6 1.9 1.82 has either or both 1.9 1.9 2.7 2.31 has both 2.2 1.9 3.5 2.73     discussion   based on previous studies, it was apparent that publication year and subject could also affect usage (tosaka & weng, 2011, pp. 419-420; morris, 2001, p. 34). we controlled for these factors by stratifying our analysis across different year groups and broad disciplines. our primary outcomes (dependent variables) were number of titles used at least once, and the total number of uses.   similar to tosaka and weng (2011, pp. 418-421) and morris (2001, pp. 33-34), we found a direct correlation between inclusion of catalogue record enhancements in the marc record and number of uses. we found that the probability of an e-book with one of the catalogue record enhancements being used (as indicated by the or) was over 80% greater than for titles lacking the enhancements, and nearly twice as high for titles with both features. the differences were greatest among the oldest and the most recently published books, and in science and technology, and least among the books published between 1998 and 2007 and those in the humanities and social sciences. the reasons for this may be due to the distribution of the cres being more balanced in the latter year groups (see table 4), or perhaps due to the decrease in the distribution of titles from this year group (see figure 2).   the limitations of which we were aware included the limited number of e-book platforms represented (two), the confluence of counter br1 (titles used) with br2 (sections used) measures, the effects of assigned readings on usage, and any preference by the patrons for platform. when instructors request the library purchase titles for assigned readings, our policy is to purchase these titles as e-books with a licence to allow multiple users access.  the data that identified such titles was not available for analysis, so this could be a factor in the results. finally, while patrons may express their preference for e-book platforms in surveys and usability studies, demonstrated preference by platform through purposeful selection and non-selection should be examined more carefully.    given the large sample size and the statistical analyses, the results demonstrate a clear and consistent relationship between catalogue record enhancements and e-book usage of any kind.  our next step is to conduct an experimental study, adding such content to randomly selected titles that had not been used.  if an increase in the usage of these titles results, we will plan to add into the workflow the addition of these fields to the catalogue records. we also plan to extend this study to the use of printed materials, particularly those housed in remote storage facilities. we would like to know if adding such content will increase their likelihood of continued usage even after resources are removed from the open stacks. finally, it would be interesting to examine the effects of providing bibliographic records with thumbnail cover images on usage.    conclusion   our primary reason for conducting this study was to better understand why certain e-books were used and others were not.  because usage data indicated that users discovered most titles in the catalogue, we focused on differences in the marc catalogue record.  the inclusion of enhancements to the catalogue records was our first target, notably the marc 505 field for tables of contents (tocs) and the marc 520 field for summary statements. our literature review indicated that such enhancements could be associated with increased likelihood of being used in print, as well as increased number of times being used. this study was a retrospective cohort study, where titles were categorized and their usage analyzed based on the inclusion of defined catalogue record enhancements. one cohort had marc 505, another had marc 520, and a final had neither.  from the first two cohorts, we analyzed subgroups, including those that had one or the other but not both, and those with both.  because of the size of our collection (more than 75,000 titles, of which nearly 10,000 were used at least once), we were able to conduct subgroup analyses using robust statistical methods and significance criteria.    by studying information seeking behaviour, it is possible to discover catalogue record enhancements that have facilitated library users’ research. by adding these enhancements to catalogue records, and by advocating for their inclusion in vendor-supplied records, libraries can make their catalogues and collections more accessible to users.   references   best, r. d. (2008). the university of pittsburgh study in an electronic environment: have e-books changed usage patterns of monographs? acquisitions librarian, 19 (3-4), 311-323. http://dx.doi.org/10.1080/08963570802026369   bucknell, t. (2010). the ‘big deal’ approach to acquiring e-books: a usage-based study, serials, 23(2), 126-134. http://dx.doi.org/10.1629/23126       calhoun, k. cantrell, j., gallagher, m., & hawk, j. (2009). online catalogs: what users and librarians want. dublin, ohio: oclc.   chercourt, m. & marshall, l. (2013). making keywords work: connecting patrons to resources through enhanced bibliographic records. technical services quarterly, 30(3), 285-295. http://dx.doi.org/10.1080/07317131.2013.785786   cochrane, p.a. (1978). books are for use: final report of the subject access project to the council on library resources. syracuse, n.y.: school of information studies, syracuse university.   counter (2008). counter code of practice: journals and databases (release 3). retrieved from http://www.project.counter.org/r3/release3d9.pdf   dinkins, d., & kirkland, l. n. (2006). it's what's inside that counts: adding contents notes to bibliographic records and its impact on circulation. college & undergraduate libraries, 13(1), 59-71. http://dx.doi.org/10.1300/j106v13n01_07   faiks, a., radermacher, a., & sheehan, a. (2007).  what about the book: google-izing the catalog with tables of contents. [special issue] library philosophy & practice (june), 1-12.   galvin, t. j., & kent, a. (1977).  use of a university library collection: a progress report on a pittsburgh study. library journal, 102(20), 2317-2320.   kirkland, l. n. (2013). the relationship of metadata to item circulation. cataloging & classification quarterly 51(5), 510-531. http://dx.doi.org/10.1080/01639374.2012.762963   knutson, g. s. (1991). subject enhancement: report on an experiment. college & research libraries, 52(1), 65-79.   levine-clark, m. (2014, march) e-book usage: navigating the swell of information. paper presented at the 9th electronic resources and libraries conference, austin, texas.  retrieved from http://www.slideshare.net/michaellevineclark/levineclark-michael-diving-into-ebook-usage-assessing-the-swell-of-information-electronic-resources-libraries-austin-march-17-2014   madarash-hill, c. & hill, j. b. (2004). enhancing access to ieee conference proceedings: a case study in the application of ieee xplore full text and table of contents enhancements. science & technology libraries, 24(3/4), 389-399. http://dx.doi.org/10.1300/j122v24n03_09   madarash-hill, c. & hill, j. b. (2005). electronically enriched enhancements in catalog records: a use study of books described on records with url enhancements versus those without. technical services quarterly, 23(2), 19-31. http://dx.doi.org/10.1300/j124v23n02_02   markey, k. (1983) online catalog use: results of surveys and focus group interviews in several libraries. final report to the council on library resources (vol. 2).  dublin, ohio: oclc online computer library center.   matthews, j. r. (2014). consulting services. retrieved from http://www.joematthews.org/consulting-services.html        matthews, j. r., & lawrence, g. s. (1984). further analysis of the clr online catalog project. information technology and libraries, 3(4), 354-376.   morris, r. c.  (2001). online tables of contents for books: effect on usage. bulletin of the medical library association, 89(1): 29-36.   morse, p. m., & chen, c. (1975). using circulation desk data to obtain unbiased estimates of book use. library quarterly, 45(2), 179-194.   tosaka, y., & weng, c. (2011). reexamining content-enriched access: its effect on usage and discovery. college & research libraries, 72(5), 412-427. http://dx.doi.org/10.5860/crl-137   trueswell, r. w. (1968). some circulation data from a research library. college & research libraries, 29(6), 493-495.   trueswell, r. w. (1969). some behavioral patterns of library users: the 80/20 rule. wilson library bulletin, 43(5), 458-461.   microsoft word art_booth.doc evidence based library and information practice 2007, 2:1  89 evidence based library and information practice     article    prediction is difficult, especially the future: a progress report     andrew booth  director of information resources and reader in evidence based information practice,   school of health and related research (scharr) university of sheffield   sheffield, great britain, united kingdom  e‐mail: a.booth@sheffield.ac.uk     anne brice  head of knowledge and information sciences, public health resource unit  university of oxford ‐ headington  oxford, great britain, united kingdom  e‐mail: anne.brice@dphpc.ox.ac.uk       received: 05 september 2006    accepted: 08 february 2007      © 2007 booth and brice. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    abstract    objective ‐ this paper reviews developments in the consolidation and diversification of the  evidence based library and information practice (eblip) paradigm since publication of the  authors’ book evidence based practice for information professionals: a handbook in 2004.    methods ‐ the authors provide an updated narrative review of key themes in the  development of evidence based librarianship within the context of the new consensual term  ‘eblip.’ sources for this thematic framework included professional literature, internet  searches, and the authors’ personal experiences.    results ‐ while considerable achievements have been realized within a three‐year period,  most notably the instigation of the journal known as eblip, a broadening of the paradigm  to other library sectors, and increased availability of implementation studies, many  challenges remain. of particular concern is the lack of international strategic foresight in  determining rotation of the biennial international conferences and distribution of influential  eblip infrastructures and initiatives.    http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:1  90 conclusion ‐ while the enthusiasms and energies of individual practitioners and work  teams have made considerable progress in meeting short‐term objectives, uncertainty  remains concerning how longer‐term objectives requiring infrastructure and resources  might be realized. from its faltering steps as a toddler eblip has developed to a ‘pre‐ pubescent’ stage with the promise of ‘growth spurts’ and ‘emotional crises.’ the next three  years should prove both challenging and demanding.     introduction    in 1997 at the cochrane colloquium in  amsterdam, dave sackett, then director of  the centre for evidence based medicine in  oxford, predicted a future for evidence  based medicine and the cochrane  collaboration (hicks 123).  by 2005, he  surmised, the cochrane collaboration  would have more than 10,000 systematic  reviews and researchers could access their  results within 15 seconds. today’s reality is  that it takes closer to 3 times as long to  locate the results section of even a unique  hit for a review in the cochrane library files.  as for the promised yield of systematic  reviews, even at its creditable total of  approximately 3,600 titles including  protocols, this number falls well short of the  predicted amount by two‐thirds. this  illustrates how even a major player within  the field of evidence based practice is unable  to predict the future with any degree of  certainty.     proponents of evidence based library and  information practice have found it equally  difficult to be prescient. in 2001 at the first  evidence based librarianship conference at  the university of sheffield, eldredge  advanced 4 predictions to be reached by a  target date of 2005:     1. research results will appear that  begin to answer the practical,  enduring questions of health  sciences librarianship.  2. cohort studies and randomised  controlled trials (rcts) will begin to  serve as major research designs for  providing answers for certain  practical questions in health  sciences librarianship.  3. structured abstracts will become the  convention for reporting research to  enable health sciences librarians to  identify and extract needed  information quickly from their  literature.  4. qualitative research will generate  valuable exploratory hypotheses.  (eldredge “evidence based  librarianship: what might we  expect”)    although, in comparison with sackett’s bold  soothsaying, such quantification‐resistant  predictions find refuge in subjectivity, we  find it difficult to argue convincingly that  any of these have been fully realized. indeed,  in 2005 at the 3rd evidence based  librarianship conference in brisbane,  australia, booth revisited eldredge’s  predictions, concluding that not a single one  had been realized by its target date. in doing  so he reminded the audience of his previous  observation that information specialists are  endowed with silicon chips not crystal balls  (booth and walton).    but surely, some might argue, there is some  support for eldredge’s predictions, at least  in quantitative terms? to this we would  respond that, as evidence based  practitioners, we have learned not to  conflate increased inputs and outputs with  increased impact (booth, “counting”).  notwithstanding an increase across the  evidence based library and information practice 2007, 2:1  91 board in research and research‐oriented  articles (table 1) the impact of evidence  based library and information practice has  fallen short of predictions. so, for example,  while crude estimates of the quantities of  research articles demonstrate an increase of  about 50% between 2001 and 2005, few of  the relevant and answerable questions of  health sciences librarianship identified by  eldredge and other international colleagues  have been satisfactorily explored, let alone  resolved (evidence based librarianship  implementation committee). similarly,  while articles discussing cohort studies and  randomised controlled trial designs occur  more plentifully than before, this simply  reflects a general increase in, and awareness  of, research production and is not reflected  in a migration to greater use of major  research designs.    structured abstracts are indeed more  plentiful, aided by their adoption for papers  being presented at the conferences of the  medical library association and the  canadian health libraries association.  however, as their prevalence within the  longer‐established field of medical journals  is only marginally greater than 60%  (nakayama et al. 239) applying the word  “convention” to a figure that likely falls well  short of 50% in health information journals  and which approaches only 3% in general  information journals (koufogiannakis et al.  232) is surely stretching accepted meanings  of that term. furthermore their adoption has  been on a one‐by‐one basis and not through  adoption of an editorial consensus statement  such as the consort convention for  reporting randomised controlled trials (plint  et al. 263). at least two major conferences ‐  the european association of health  information and libraries and the health  libraries group in the uk ‐ do not yet insist  on structured abstracts.     qualitative research continues to command  an important place in the research  methodology toolkit, but it is difficult to  identify any particularly valuable  explanatory hypotheses that distinguish the  years between 2001 and 2005. in attempting  to distinguish the more significant measures  of impact from the more readily obtainable  measures of output, the authors believe one  can paint a much less rose‐tinted picture of  the paradigm.  none of the foregoing is meant to  undermine any claims that the above  commentators might wish to make to  ‘expertness’ within evidence‐based practice  (sackett). we would do well to remind  ourselves that less than two years before he  made the first manned flight, wilbur wright  stated that man “would not fly for fifty  years.” he later noted, “ever since, i have  distrusted myself and avoided all  predictions.” (wright)     2001  medline / cinahl*  2005  medline / cinahl*  mentions of ‘research’ articles            40         /        205           66         /       323  mentions of ‘cohort studies’ and  ‘rcts’               5        /           23           10        /         30  mentions of ‘qualitative research’            77       /           18          103      /         49  * combinations of free‐text and subject  terms used in cinahl were those  deemed to be the nearest equivalent to  medline terms.         table 1: incidence of markers of evidence based practice in the health information literature  evidence based library and information practice 2007, 2:1  92 where we seem on safer ground is in  recording aspirations rather than  predictions. in the final chapter of evidence  based practice for information professionals: a  handbook, the editors, together with  colleagues, denise koufogiannakis, ellen  crumley, and jonathan eldredge, advanced  individual and collective aspirations for  evidence based information practice (brice  et al.). our thoughts were compiled in late  2003, the year before publication of that  edited volume. having already  demonstrated how ineffectual five‐year  vistas prove within evidence based practice,  these provide an alternative three‐year  benchmark for progress to date. we  welcome the opportunity in this paper to  revisit these aspirations. we acknowledge  that authorship of this article no longer  privileges the north american perspective.  yet at the same time, our contribution is  strengthened by insights gained from  workshops and seminars delivered in  denmark, finland, sweden, spain, sicily,  romania, and australia.    what’s in a name?    in beginning this narrative review we  should acknowledge our personal antipathy  to the label ‘evidence based librarianship.’  we believe that this label is limited in scope  and may carry undesirable connotations of  exclusivity. initially we were attracted to the  alternative name ‘evidence based  information practice,’ because it appears to  be more encompassing. it was this term that  we originally advocated. however, we  subsequently realized that this term loses  the strength derived from connection with  the already well‐established body of our  professional practice. in discussions at the  3rd evidence based librarianship  conference in brisbane, we came to agree on   ‘evidence based library and information  practice.’ the authors believe that this  phrase aptly characterizes the intellectual  efforts and energies expended by a growing  international community on behalf of this  model of reflective and continuously  developing professional practice. we  therefore hope that this name continues to  establish itself as a consensual term under  which future endeavours will be organised  and mobilised.    booth has opportunistically recorded a  snapshot of the concept behind this newly  preferred term. in doing so he attempted to  synthesize previous definitions (booth,  “exceeding;” eldredge, “evidence;”  crumley and koufogiannakis). writing in  performance measurement and metrics he says  that eblip :     . . .  .seeks to improve library and  information services and practice by  bringing together the best available  evidence and insights derived from  working experience, moderated by  user needs and preferences. eblip  involves asking answerable  questions, finding, critically  appraising and then utilising  research evidence from relevant  disciplines in daily practice. it thus  attempts to integrate user‐reported,  practitioner‐observed and research‐ derived evidence as an explicit basis  for decision‐making. (“counting” 65)    this definition is our starting point for the  discussion that follows. to facilitate  comparison with the original chapter, this  article follows the outline of the original and  discusses publishing, community access and  discussion, policy development,  improvement of the knowledge base,  instruction and practical tools, international  collaboration, and achievements. it revisits  various challenges, aspirations and  frustrations of eblip before examining  goals for the future. in doing this, the paper  draws on a thematic analysis populated  from the literature, internet searches, and  the authors’ personal experience. the article  evidence based library and information practice 2007, 2:1  93 concludes by extending the horizon to  accommodate ongoing expectations and  aspirations.     publishing    the original chapter outlined a “plethora of  conceptual literature” (brice et al. 280), and  there remains ongoing interest in eblip  among researchers, educators, policy  makers, and practitioners in promoting  eblip and embedding it within daily  practice. arguably, however, although these  publications are a necessary mechanism for  translating and adapting the generic  evidence based practice model to our  professional domain, they have been  subsumed in importance by a new and  exciting brand of ‘implementation’ study  (e.g., cotter et al.; abbott; yu et al.). such  studies, first presented at the 3rd  international ebl conference (brisbane,  2005) and later published in this journal,  have promulgated the key messages of  eblip more widely, across both sectoral and  cultural divides.     in the authors’ personal opinion, the single  most important development has been the  launch of the journal evidence based library  and information practice (eblip) itself, not  only because it is an open access, online  resource, but also because it is already  significantly indexed by lisa: library and  information science abstracts, library  literature and information science, lista:  library, information science and technology  abstracts and the informed librarian online.  published by the university of alberta  learning services, since march 2006, eblip  combines primary research, commentary,  and evidence summaries for current journal  literature, and it represents a considerable  achievement in planning and application.  as such it provides a tangible and highly  visible representation of the desire to put the  practice of eblip on a firmer, more rigorous  footing.  other journal publishing ventures include  the “using research in practice” column in  the health information and libraries journal  (beginning in 2003) and the medical library  association research section’s publication,  hypothesis. a more encouraging sign of  diversification of the paradigm are the  special features in existing journals, such as  the 2006 special ebl issue of library hi‐tech,  and feature articles in library and information  research that appeared in 2006.    with regard to the book literature, our own  evidence based practice for information  professionals: a handbook suffers, perhaps,  from the burden and expectation of being  the first volume on the topic. in attempting  to combine the triple roles of historical  marker, evidence source, and practical  guide it unsurprisingly falls between stools.  rightly criticised for being more than a  ‘handbook,’ unless of the arachnodactylic  variety (!), it has at least opened the way for  further niche offerings, such as case studies  in evidence based librarianship, edited by  elizabeth connor (chandos publishing,  2007). connor’s work, as indicated by its  proposed title, is located securely within the  earlier variant of the paradigm, carrying the  joint virtue/vice of being independent of  more recent eblip thinking and activity as  showcased at the recent ebl conferences. in  addition, the concept of ‘evidence based  research,’ as apparently espoused by the  editor, is doubly problematic because all  research should be evidence based – at least  in its conduct – while research‐derived  evidence is only one ingredient of an  optimal blend that also includes  ‘practitioner observed’ and ‘user‐reported’  evidence. it remains to be seen whether this  book’s principal contribution lies in  exemplification of, or fragmentation from,  the most current version of the eblip  model.     what have such publishing developments  brought, and what remains to be done? no  evidence based library and information practice 2007, 2:1  94 formal measures of the spread or depth of  awareness of the concepts and features of  eblip exist, and it is not possible to be  precise about the impact of these and other  ventures. many fundamental questions  remain about the nature of research and  scholarly publishing and its impact on the  lives of everyday library employees. will the  dearth of good quality research make the  task of identifying potentially useable  contributions unsustainable? furthermore,  what might we learn from the experiences of  other disciplines, where publication of better  quality primary evidence is likely to be  subordinate to many contextual factors that  influence implementation of research into  practice (meijers et al.)?    community access and discussion    the intention of creating a vibrant, open,  community discussion forum, to enable  information sharing, and to “promote the  movement internationally and to encourage  librarians from all subject areas to learn  more” (brice et al. 280), a deficiency  identified three years ago, is still to be  realised. although the evidence based  libraries electronic mail list (<evidence  based‐libraries@jiscmail.ac.uk>) continues to  attract a wide user base of nearly 700  members, it remains primarily a vehicle for  one‐to‐many dissemination. the same can  be said of the more recent addition, the  canadian‐originated evidence based  librarianship interest group  (<eblig@clify.ucs.mun.ca>). although this  interest group has extended its membership  beyond the narrow confines of the canadian  library association, here, too, there is little  evidence of genuine innovation and debate.  there is little evidence of more than a slow  seepage of eblip into discussions on other  library professional mailing lists. the  comment from 2004 that there is “as yet no  natural forum for international discussions  on ebl ideas, philosophies and techniques”  (brice et al. 280) reflects the current status in  2007.  this lack of discussion on conceptual  issues is tentatively addressed by articles on  the wider ramifications, such as the ethics of  eblip (booth, “where’s the harm”), eblip  narrative methods (brophy), and the utility  of eblip (pearce‐smith).     one specific attempt to engage members in  discussion has been the online,  asynchronous medical library association  research sectionʹs evidence based  librarianship (ebl) discussion group. here  a series of readings, to provide an  introduction and overview of specific areas,  helps to structure a discussion based on  topics including an overview of ebl,  finding evidence, study design and the  future of ebl. apparently aimed at laying a  foundation of fundamental principles for  evidence based librarianship, rather than  advancing controversy and debate, this  otherwise invaluable discussion necessarily  captures a retrospective view of the topic  through the aperture of historical, albeit  recent, articles.     in 2004 the authors threw down a somewhat  weary challenge to extend participation on  the public stage beyond the ‘same names’:  “in short, the innovators have done their bit  – now is the time for the early adopters to  step forward and be identified” (brice et al.  291).    encouragingly, this particular challenge has  been met, illustrated by such projects as  those mentioned earlier (cotter et al.;  abbott). there is also a heavy presence  among the eblip journal advisors of those  ‘early adopters,’ adding an impressive  breadth and depth of coverage to the  publication, albeit focused on the developed,  english‐speaking world. fortified by this  injection of talent and enthusiasm, many  self‐styled innovators have derived new  impetus.     evidence based library and information practice 2007, 2:1  95 a profession that tends towards the  anecdotal and practical, rather than the  conceptual and visionary, however,  mediates such encouraging and yet isolated  developments. this observation is equally  true of other areas of library practice, and  indeed, of other groups and disciplines.  social research informs us that people will  express their views only when they are  engaged and committed, and this requires  the building of trust and a common  language (cooper et al.).  initial indications  appear to be that it will be easier to start  small ‐ by creating virtual learning  communities locally and then building upon  early successes to reach out to the wider  librarianship community (lewis).    eblip achieves its largest measure of  success where different communities or  sectors of our profession stamp their own  unique approaches on the topic, while  retaining the spirit of the movement. this  aspiration, best exemplified by activities  within school librarianship, was recorded in  2004 as “incorporating all library sectors  into the movement and revising the  definition of ebip to capture the different  aspects of librarianship” (brice et al. 289).     what evidence is there of the spread of  evidence based practices from healthcare to  other sectors? a search of the lisa database  identified 326 citations published in 2003  with ‘evidence based’ appearing in article  titles or textwords. visual inspection of the  abstracts for these established that only 9%  were from settings outside the health  sciences (table 2). by 2006 this percentage  had increased to 12%. certainly this does  not constitute evidence of growth sufficient  to fulfil this particular aspiration. other  markers such as use of the phrases ‘critical  appraisal’ and ‘systematic review’ actually  represent a relative decrease in spread to  other disciplines, mainly due to more  prolific activities within the healthcare  information sector. at the same time, there  is some indication of increasing interest  from practitioners from other sectors in the  evidence based librarianship conference  series. the first evidence based  librarianship conference was attended  almost exclusively by health information  practitioners. the second conference  included information practitioners from  outside the health arena, but most were  principally those working within the host  institution. it was in brisbane at the third  conference that there was a truly genuine  attempt at cross‐sector participation.     2003  2006 selected eblip  terms  health  non‐ health  total     health   non‐ health    total  % increase  2003‐2006  evidence based  298  (91%)      28 (9%)   326    43 (88%)       60 (12%)      494  52%  critical appraisal    43 (41%)      63 (59%)   106    53 (44%)       67 (56%)      120  13%  systematic  review    19 (63%)      11 (37%)     30    39 (71%)       16 (29%)        55  83%  evidence  484 (20%)  1978 (80%)  2452  716 (22%)   2524 (78%)    3240  32%    table 2: prevalence in the lisa database of selective markers of evidence based practice    additionally the ifla social sciences  libraries section has recently issued a call  for papers at its 2007 meeting on the theme  “evidence based practice in social science       libraries: using research and empirical  data to improve service.”        evidence based library and information practice 2007, 2:1  96 policy development    involvement of key figures in planning and  policy discussions is of symbolic, as well as  practical, importance. however, it is  correspondingly more difficult to measure  the impact of eblip on planning and policy  – if indeed there has been any at all. not  only is it problematic to identify where such  discussions impact on day‐to‐day  practitioners, but it remains a challenge to  trace when the evidence based signal, first  imperceptible and then increasing in  strength, becomes manifest in policy  decisions. indeed, there has been no formal  analysis of the influence of the evidence  based paradigm on policy development.  areas where progress has been made  remain hidden in internal documents or in  the consciousness of senior policy makers.  relying only on published accounts, it  becomes clear that some eblip related areas  of policy development have received  attention at the highest level within specific  geographic, sectoral or professional contexts.  however it remains unclear what the impact  of these has been or, indeed, how this might  actually be measured.    notwithstanding such difficulties in  evaluation of impact, involvement of senior  and high‐profile librarians – e.g., joanne  marshall, margaret haines, and gillian  hallam ‐raising priorities and issues during  their presidency periods of mla, cilip, and  alia respectively, has provided  considerable encouragement to those  working in the field. these and other  opinion‐leaders have an important and  continuing role to play, largely because of  their ability to combine academic and  professional rigour with influence and  impact on broader professional matters.     it would be wrong to overplay the  importance of this leadership role, however.  evidence based practice has always  accommodated “top‐down” and “bottom‐ up” approaches alike. indeed, the success of  current australian endeavours may be most  attributable to energy derived from  grassroots experiences, and their successful  efforts may translate as examples for  librarians in other continents. for example,  librarians in gosford, new south wales  have produced a useful and pragmatic  document, “libraries using evidence  charter” (cotter and lewis). the speed with  which this deliverable was realised contrasts  with the corresponding delay currently  taking place as leading figures in the  movement debate the best way forward for  developing an international consensus  statement. such a statement would mirror  the “sicily statement for evidence based  practice” (dawes et al.) in agreement  regarding the scope, current status, and  priorities for development of eblip.    improving the knowledge base    the original summary chapter highlighted  improvement of the knowledge base as one  of the biggest challenges facing the  development of eblip, and this remains the  area where significant changes need to be  made. these areas can be summarised as:    • availability of research funding  • type of questions being answered  • study designs selected  • quality of research   • synthesis of findings  • dissemination of results    while publishing initiatives itemized above  help to disseminate findings more rapidly,  we might well ask whether there is a  ‘mother lode’ for a ready source of evidence  summaries.  koufogiannakis et al.  previously investigated our knowledge of  the characteristics of our evidence base, and  perryman and lu have recently augmented  this with their examination of the  terminology used by the lisa (library and  information science abstracts) database to  evidence based library and information practice 2007, 2:1  97 identify research methodologies. the  authors demonstrated that the lisa  thesaurus is neither consistent nor  sufficiently comprehensive to serve the  needs of researchers. however, in  comparison with work in healthcare, we still  have a great deal more to learn concerning  the tools and methods of evidence based  practice. for example, might methodological  filters be successfully developed to retrieve  rigorous studies from the library literature  in the same way that they have been devised  for health care research?    the ability to collect and address questions  of direct relevance and applicability to  practitioners, the ability to fund and conduct  good quality studies, and the skills and  funding to synthesise research findings to  add power to their results, should lie at the  heart of eblip, as with similar movements  in other disciplines. is there anything we can  do to improve the relevance and  applicability of research? do we have even  the basic tools to use, such as validated  instruments? a recent systematic review of  information skills training (brettle)  highlights the fact that our quest to  synthesise research results may often  conclude not simply that we lack the  research studies, but, even more  fundamentally, that we do not have the  validated tools with which to conduct such  research. our own thinking has moved on to  the extent that we consider the development  of tools to facilitate rigorous research to  have supplanted the specific areas  summarized above as a realizable priority  with regard to improving the knowledge  base.    teaching and learning and practical tools    hand‐in‐hand with the need for an  improved knowledge base come skills to  access and exploit it. the ongoing  development of eblip workshops (booth  and brice, “clear‐cut?”), the integration of  eblip principles into mainstream library  and information studies (lis) course  teaching, and the development of formal  eblip modules have been accompanied by  the development and spread of several  practical learning resources:      • setting‐perspective‐intervention‐ comparison‐evaluation (spice)  framework for analyzing  answerable library questions (booth,  “formulating”).  • readerʹs guide to the literature on  interventions addressing the need  for education and training  checklist for educational studies  (koufogiannakis et al., “reliant”)  and checklist for filter studies  (jenkins).  • inventory of research methods for  librarianship and informatics  (eldredge, “inventory”).    ad hoc individual contacts between  international collaborators have contributed  to the spread of teaching methods and  materials, such as the workshops at the 3rd  ebl conference facilitated jointly by us‐uk  and uk‐canada pairings. improved  international understanding and  collaboration are particularly evident in  europe and australia where several joint  teaching events, and an accompanying  exchange of teaching methods, have been  successfully delivered. there has been  criticism, however, that the success of these  events is overly dependent on the profiles  and efforts of a few well‐known individuals  and can only contribute to short‐term  awareness. in contrast, the european ethos  embodies a system of cascade whereby an  external facilitator comes into a single  organisation as an eblip catalyst. following  this external stimulus, internal staff  members work on taking forward issues  within their own organisations.     evidence based library and information practice 2007, 2:1  98 this model, which targets the aspiring  evidence based organisation, through work  teams, as opposed to the isolated evidence  based practitioner (booth, “from ebm”) has  been tried by the authors when conducting  training courses for libraries such as those of  loughborough university and kings  college, london in the united kingdom and  for the karolinska institute in sweden.  while it is too early to judge whether such  approaches have proved effective there is at  least some grounds for optimism that these  may provide an opportunity for greater  institutionalisation and integration of eblip  into work teams. the team at the karolinska  institute, for example, report in a personal  communication how they have taken their  journal club one step further to a more  objective discussion, adopting the critical  skills training in appraisal for librarians  (cristal) checklists developed for  appraising library studies (booth and brice,  “clear‐cut?”), and writing the minutes to  resemble an evidence summary.  interestingly, this cascade method models,  albeit on a smaller‐scale, the approach used  by the association of research libraries in  the us to increase local ownership of  performance measurement (hiller et al.).  of  course, there may be other such ‘learning  cascades,’ but these remain, as yet,  unrecorded.    international collaboration and  achievements    rotating locations for the international  conferences has provided many  opportunities for collaboration, as  colleagues from around the world plan and  attend these events. fortuitously each  conference has witnessed the appearance of  an international delegate to volunteer to  organize the subsequent event. holding the  conference in different locations also allows  proponents from each country to bring  relevant issues to a larger audience and to  stimulate local developments. it also attracts  international speakers amid awareness that  research carried out outside one’s own  immediate environment may be both  interesting and transferable.     the downside of this opportunistic  approach to hosting the conference lies in  the absence of a shared strategic vision for  development of the series. certainly the  long‐standing pedigree of the movement  within the united states might have led to  expectations that participation as host for  the biennial international conference would  come earlier than the fourth iteration. of  more concern is the possibility that this  opportunistic modus operandi might lead to  persistent exclusion of the developing world  or of non‐english speaking cultures such as  those in eastern europe. as stated three  years ago, eblip  “is still mainly in  developed countries, so involving other  librarians from all types of environments  and situations would enable us to develop  the concept for all librarians’” (brice et al.  285).    while something as tangible and focused as  a conference has survived and thrived in  this ‘strategy vacuum,’ this is not true of  broader, more long‐term, collaboration for  the movement in general. there has still not  been an explicit discussion or debate, never  mind agreement, over what can and should  be done internationally, and what is best  done nationally or locally. the model that  currently appears to hold sway for eblip  initiatives is that exemplified by the eblip  journal ‐‐ namely that it is parochial (in this  particular case, canadian) in ownership but  international in scope and impact. while  commending the foresight and opportunism  that leads to such developments, one has to  ask whether such an approach will result in  an even spread of eblip mechanisms and  infrastructures across multiple countries.  too, how will such entrepreneurialism  ensure the development of less attractive,  and yet equally necessary, structures, once  evidence based library and information practice 2007, 2:1  99 the first round of initiatives has been  ‘cherry‐picked’? further dilemmas are  reflected in divisions between sectors – advocates for eblip remain split as to the  extent to which their energies should be  focused inwards, towards running  workshops and delivering presentations  within their own sectors, or outwards, in  working across sectors on wider  propagation of the paradigm.     encouragingly, some examples of cross‐ boundary international efforts do exist. the  cochrane information retrieval methods  group provides the first example of one  aspiration, “an international register of  rigorous librarianship studies” (brice et al.  290)  (albeit as a component of the cochrane  methodology register and focusing  exclusively on issues associated with  information retrieval). the international  hta‐i information specialists group hosted  an evidence based information retrieval  workshop in adelaide, australia, that  attracted international regulars and local  enthusiasts. the european association for  health information and libraries has also  hosted an evidence based librarianship  workshop and featured eblip in a special  issue of its open access journal (booth,  “where’s the harm?”). wider collaborative  opportunities with the contiguous  disciplines of evidence based information  systems and evidence based software  engineering have been tantalisingly hinted  at but are still to be explored in detail.    challenges, aspirations and frustrations     in the opinion of the authors two particular  challenges remain as pertinent today as they  did for the original chapter: the ‘depth and  spread’ of eblip and the need for  developing an eblip ‘skills base.’  ‘depth  and spread’ represents a complex  phenomenon, although significant  developments may still exist where eblip  issues are tackled at the grassroots. whether  such depth extends to responsibility at every  level – “individual, organisational,  professional, national and international”  (brice et al 284.) ‐ is more difficult to gauge.  major professional membership associations  are comparatively quick to pay lip service to  the importance of eblip but, with the  exception of funds for conference  organisation, few have placed resources at  its disposal. associations with a prior record  of involvement in research utilisation, such  as the special libraries association and  medical library association in the united  states, the canadian health libraries  association, and the library and  information research group and health  libraries group in the united kingdom,  have found it easier to accommodate the  eblip agenda than more recent additions to  the fold. organisations such as the  australian library and information  association have had their involvement  stimulated by the proximity of eblip  conferences, in truth making non‐ participation almost impossible, and their  support has been warmly welcomed. for  many other groups, however, there persists  a sense that inclusion in conference  programmes or workshops is simply a  response to the perceived need of committee  members: “i suppose we should do  something for our members on evidence  based librarianship.” this would be  followed by the almost inevitable, “well  now we have done that, what is the next hot  topic that should command our attention?”     issues with regard to the skills base are also  problematic. eblip not only requires  production of a new generation of  practitioners, familiar with the design and  interpretation of lis research. we hope that  the encouraging initial observation that  evidence based practice apparently meets  the “unique learning preference of the  modern‐day lis student – the millennials”  (partridge and hallam 417) is ultimately  realised. however we also acknowledge the  evidence based library and information practice 2007, 2:1  100 same authors’ caution that “the current  student cohort is not comprised solely of  millennials, but also includes older students  such as generation x – and even baby  boomers”(406). eblip therefore also needs  to provide corrective or remedial education  for generations of practitioners who, like  ourselves, received little or no formal  research training. as we commented three  years ago:    a sustainable continuing  professional development policy  and funding has neither been  achieved nor implemented.  otherwise successful tools and  educational activities have,  disappointingly, not been  recognized by commitment and  funding at a policy level. (brice et al.  287)    despite interest from educators, there is still  considerable unfulfilled demand for  remedial education, hence ongoing demand  for workshops and short courses and the  need for sharing of resources.     priorities for the future    our aspirations for the future must  necessarily include both strategic necessities  and personal priorities. as briefly hinted  above, the eblip movement requires both  strategic coordination and the development  of international consensus. strategic  coordination would yield opportunities for  a more planned and systematic approach to  conference planning and the development  and allocation of resources and initiatives  according to genuine need. this would  serve to channel the prodigious energies  and enthusiasms of those already involved  in eblip more productively as well as  providing a rallying point for those  interested in future participation. parallels  have previously been drawn with the  international cochrane collaboration.  certainly were even a fraction of that  network’s efforts to be achieved within our  profession, all would benefit (booth, “will  health librarians”).    similarly the achievement of an  international consensus on what eblip is  and how it might best be taken forward is  seen as a prerequisite to other proposed  developments. while a joint international  statement of policy and intent, first  mentioned three years ago and discussed  informally at the 3rd evidence based  librarianship conference is seemingly  symbolic, we believe it would constitute a  really important first step. this would both  provide an opportunity to synchronise the  different national ‘eblip clocks,’ from  which subsequent developments and  synergies could be advanced, and also act as  a vehicle for shared values and clearer  communication across cultures and sectors.      given the limited time and resources  available, and the increasing pressure on  individuals to satisfy immediate  organisational objectives, sectoral  involvements, and national interests ‐ time  remaining for international activities will be  limited. yet the logic for collaboration is  persuasive: if every member attending a 20‐ strong eblip workshop wrote one evidence  summary over a two‐year period, the  international community would have access  to the findings of 20 new research studies.  this would supplement the 10 or more  summaries published in each issue of this  journal. collectively such evidence  summaries benefit the individuals, their  organisations, and nameless colleagues  worldwide. how effective these and other  such international efforts, could be in saving  time and energy, sharing knowledge,  reducing duplication, and supporting  learning from global experiences! a  disappointing feature of an otherwise  encouraging trend towards localisation of  eblip is the number of anecdotal accounts  evidence based library and information practice 2007, 2:1  101 of journal clubs with no visible or virtual  outputs disseminated via the rapid  publishing medium of the web. indeed,  such journal clubs do not even make  available the lists of articles they have  reviewed and appraised – maybe laying  claim to the label ‘evidence based library  invisible practice’! by way of contrast, the  libraries using evidence initiative has  recently established an eblip newsfeed  providing access to and awareness of recent  international literature in our field  (<http://del.icio.us/ebliptoolkit/current>).  this initiative has the potential for a positive  impact on our agenda.    our personal aspirations continue to  prioritise one still outstanding deficiency.  “robust methods for disseminating research  findings to practising professionals are yet  to be achieved” (brice et al. 287).   investigation of promising, but ultimately  unsatisfactory, products such as systematic  reviews, guidelines, briefings, and digests  has culminated, most recently, in the  proposal of the resolutions evidence model,  an as yet untried but holistic and pragmatic  evidence summary (booth, “route maps”).  again, our own limited powers of  prediction have seen us having to postpone  at least one aspiration,  “once we have  attained a critical mass of rigorous studies  such as randomised controlled trials and  cohort studies, we can then graduate to  conducting more systematic reviews” (brice  et al. 290).    the beguiling elegance of the effectiveness  review remains defeated, at least for the  moment, by a paucity of studies.  nevertheless, future prospects for research  synthesis lie encouragingly with qualitative  systematic reviews that may allow us to  prioritise the attitudes, perceptions, and  feelings of our users (booth, “will health  librarians”).  meanwhile there remains an ongoing need  to develop tools and products to make it  easier for practitioners to apply research  findings in the workplace.  the libraries  using evidence initiative is working on a  proposed ‘toolkit,’ while the authors  themselves are planning a more extensive  inventory of cristal checklists. however  there remains a need for “lobbying  educational institutions to increase research  and appraisal skills teaching in the  curriculum, roll‐out of educational  interventions such as appraisal workshops,  and continuing development of e‐learning  and distance based courses” (brice et al. 289).    this emphasis on the practitioner is further  seen in our aspiration for an increase in  practitioner‐led research. herein lies a  paradox – simply increasing practitioner‐ conducted research, without increasing  skills and knowledge, may improve the  relevance and applicability of research, but  could actually result in a reduction in  research quality! the lesson to be learnt  from consumer involvement in research is  that practitioners could shape the questions  but not necessarily conduct the research  themselves. the biggest challenge may yet  be to obtain funding to conduct the research  needed to make better decisions in areas  deemed most relevant by practitioners.      many suggestions for driving developments  forward in the short‐ or medium‐term carry  the underlying themes of integration and  embedding in mainstream activities. to  have eblip conferences and courses is good,  to embed eblip themes and approaches  within established forums is better. to have  an eblip journal is again important, but to  have eblip articles within every lis journal  is even more so. to have eblip modules  within lis courses is significant, but to have  an eblip approach within every module  should be the aspiration.      http://del.icio.us/ebliptoolkit/current evidence based library and information practice 2007, 2:1  102 • interactive version of cristal checklists on the web alongside other learning materials  • international online course offered on a regular basis  • complete set of “user guides to the library literature” commissioned and published   • filtered database, containing references to high quality research articles categorized by  domains or areas of common interest  • common international curriculum  • international research strategy and funding  • scholarships/fellowships  • large‐scale international multi‐centre prospective studies  • safe space on the web to encourage mentoring research and applied skills  figure 1: longer‐term objectives requiring infrastructure and/or funding (brice et al. 289‐90)      “future success will only come when ebip  becomes mainstream and is supported by  those who are grounded in the  organisational cultures of international,  national, and professional library  associations and groups” (brice et al. 291)    this rationale lay behind the most radical  statement of the original chapter:        and finally – success in extinction!  maybe the most important longer  term aim, and critical success factor,  would be the abolition of an  international association of evidence  based information practitioners  (because it is so widespread and  common place everyone strives to  be evidence based). (brice et al. 290)    it is telling that many longer‐term objectives,  brought forward from 2004, carry with them  a requirement for infrastructure and/or  funding. some of these are listed in figure 1.    like any mid‐term report card, the eye is  inextricably drawn to the ‘could do better’  rather than to the ‘good.’ we have seen  considerable progress, and this is the note  upon which we should fittingly conclude.  we have seen a small but perceptible  increase in the use of structured abstracts  internationally for journals and conferences.  our community continues to benefit not  only from the biennial conference circuit but  also, more importantly, from more frequent  inclusion of eblip in mainstream  conference programmes. we have witnessed  development and dissemination of the first  evidence based guideline on the topic of  web site usability (koyani et al.) and a  report on its subsequent utilisation in a  health library web development project  (cotter et al.). we have also observed  development of the first eblip module in an  undergraduate or postgraduate library  course (marshall and perryman). the eblip  journal has met several objectives  simultaneously – providing evidence based  digests on the web, a vehicle for systematic  reviews, a secondary journal of summaries,  and ongoing and productive international  collaboration.     the future: some conclusions     in 2004 we commented, “we’ve made the  first faltering toddler’s steps, but we are a  long way from adolescence and maturity”  (brice et al. 291). growing pains continue,  and divergent opinions are necessary in any  organic movement spanning many  continents and disciplines. the significant  growth of evidence based library and  information practice, the development of  practical projects around integration and  implementation, and increasing  geographical and sectoral spread provide  evidence based library and information practice 2007, 2:1  103 evidence that this ’toddler’ is now mobilised.  the time has come to mature from being  problem‐focused to solution‐focused. in  other words, it is time to use the evidence  based practice process to produce real and  transferable solutions to important practical  problems, rather than simply further clarify  our discussion of the existence of such  problems. similarly, there is an urgent need  to consider the adoption of evidence based  practice as a core personal, professional, and  organisational responsibility. this requires  that eblip be fully integrated into personal  development plans, professional  revalidation, and organisational  accreditation schemes. perhaps  ‘prepubescence’ is now a more fitting  analogy for the current status of our  paradigm’s biochronological development.     having consistently highlighted the dangers  inherent in prediction, it would be foolish  for us to put in place too many  developmental milestones (or even too  many arbitrary millstones!) against which to  chart ongoing success of the paradigm.  nevertheless we would confidently aspire  to some, if not all, of the following  achievements over the next three‐year time  frame:    • formation of an international  collaboration or association of  evidence based library and  information practitioners.  • production of an internationally  recognized consensus statement.  • increasing practitioner involvement  in determination of research  questions and priorities.  • continued dissemination of  significant research findings  through evidence summaries.  • increasing use of ‘implications for  practice’ from published research  findings.  • generation of tools and techniques  for increased utilisation of the  evidence and for the promotion of  reflective practice.  • integration of evidence based  practice and project management  methodologies as tools for strategic  planning.  • increasing numbers of team‐based  exemplars of local eblip initiatives.    should none of these aspirations be fulfilled  within this putative timeframe, we shall, of  course, take refuge in pointing again to the  universal truth that “prediction is difficult;  especially for the future!” of one thing we  can be certain, as we anticipate growth  spurts and accompanying emotional crises  in the further development of our eblip  adolescent, the next three years promise to  command ongoing parental attention!     works cited    abbott, wendy anne. “persuasive evidence:  improving customer service through  evidence based librarianship.”  evidence based library and  information practice 1.1 (2006): 58‐68.  24 feb. 2007  <http://ejournals.library.ualberta.ca/ind ex.php/eblip/article/view/13/84>.    booth, andrew. “counting what counts:  performance measurement and  evidence based practice.” performance  measurement and metrics 7.2 (2006):  63‐74.    ‐‐‐. “exceeding expectations: achieving  professional excellence by getting  research into practice,” lianza, 15‐ 18 oct. 2000, christchurch, new  zealand.     ‐‐‐. “formulating answerable questions.”   evidence based practice for  information professionals: a  handbook.  eds. andrew booth and  anne brice. london: facet, 2004: 61‐70.  http://ejournals.library.ualberta.ca/ind evidence based library and information practice 2007, 2:1  104 ‐‐‐. “from ebm to ebl: two steps forward  or one step back.” medical reference  services quarterly 21.3 (2002): 51‐64     ‐‐‐. “route maps for evidence based  problem solutions (resolutions):  what’s the evidence for journal  cancellation?” health information and  libraries journal 23.4 (2006): 298‐303.    ‐‐‐. “whereʹs the harm in eblip? current  perspectives, future developments.”  journal of the european association  for health information and libraries  2.3 (2006): 34‐8.    ‐‐‐. “will health librarians and related  information workers ever work  together to create an international  network?” health information and  libraries journal 18.1 (mar. 2001): 60‐3.     booth, andrew, and anne brice. “clear‐cut?  facilitating health librarians to use  information research in practice.”  health libraries review 20 supp.1  (2003): 45‐52.     booth, andrew, and graham walton, “some  concluding trends and themes.”  managing knowledge in health  services. eds. andrew booth and  graham walton.  london: library  association, 2000. 289.     brettle, alison. “information skills training:  a systematic review of the literature.”  health information and libraries  journal 20 supp. 1 (2003): 3‐9.  brice, anne, andrew booth, ellen crumley,  denise koufogiannakis, and jonathan  d. eldredge, “a future for evidence  based information practice? evidence  based practice for information  professionals: a handbook. eds.  andrew booth and anne brice.  london: facet, 2004. 279‐92.   brophy, peter. “narrative‐based  librarianship.” the area of  information and social  communication: festschrift for  professor wanda pindlova. (studies in  library and information science, vol.  10). krakow: jagiellonian university  press, 2004. 188‐95.    cooper, janet, siân spink, rhian thomas,  and christine urquhart. “evaluation of  the specialist libraries/communities of  practice: report for national library  for health.” department of  information studies, university of  wales aberystwyth (june 2005). 24  feb. 2007  <http://cadair.aber.ac.uk/dspace/bitstre am/2160/221/1/evaluation+of+the+speci alist+library+cop.pdf >.    cotter, lisa, larnich harjie, suzanne lewis,  and ingrid tonnison. “adding spice  to a library intranet site: a recipe to  enhance usability.” evidence based  library and information practice 1.1  (2006): 3‐25. 24 feb. 2007   <http://ejournals.library.ualberta.ca/ind ex.php/eblip/article/view/11/81>.     cotter, lisa, and suzanne lewis. “libraries  using evidence ‐ eblip.net.au.”  evidence based library and  information practice 1.3 (2006): 98‐100.  24 feb. 2007  <http://ejournals.library.ualberta.ca/ind ex.php/eblip/article/view/102/154>.  crumley, ellen, and denise  koufougiannakis. “developing  evidence based librarianship:  practical steps for implementation.”  health information and libraries  journal 19.2 (2002): 61‐70.    dawes martin, william summerskill, paul  glasziou, antonino cartabellotta, janet  martin, kevork hopayian, franz  http://cadair.aber.ac.uk/dspace/bitstre http://ejournals.library.ualberta.ca/ind http://ejournals.library.ualberta.ca/ind evidence based library and information practice 2007, 2:1  105 porzsolt, amanda burls, and james  osborne, second international  conference of evidence based health  care teachers and developers. “sicily  statement on evidence based practice.”  bmc medical education 5.1 (5 jan.  2005): 1. 24 feb. 2007  <http://www.biomedcentral.com/1472‐ 6920/5/1>.      eldredge, jonathan d. “evidence based  librarianship: what might we expect  in the years ahead?” health  information and libraries journal 19.2  (june 2002): 71‐7.    ‐‐‐. “inventory of research methods for  librarianship and informatics.”  journal of the medical library  association 92.1 (jan. 2004): 83‐90.     evidence based librarianship  implementation committee. “the most  relevant and answerable research  questions facing the practice of health  sciences librarianship.” hypothesis  15.1 (2001): 9‐15,17.     hicks, alison “developing information  skills training for national health  service personnel: experiences at the  trent institute for health services  research.” program: electronic library  and information systems 32.2 (1998):  123‐36.     hiller, steve, martha kyrillidou, and jim  self. “assessment in north american  research libraries: a preliminary  report card.” performance  measurement and metrics 7.2 (2006):  100‐6.     jenkins, michelle “evaluation of  methodological search filters—a  review.” health information and  libraries journal 21.3 (2004): 148‐63.    koufogiannakis, denise, linda slater, and  ellen crumley. “a content analysis of  librarianship research.” journal of  information science 30.3 (2004): 227‐39.    koufogiannakis, denise, andrew booth, and  alison brettle. “reliant: readerʹs  guide to the literature on  interventions addressing the need for  education and training.” library and  information research 30.94 (2006): 44‐51.     koyani, sanjay j., robert w. bailey, and  janice r. nall. research‐based web  design & usability guidelines.  washington, dc: us department of  health and human services, 2003.     lewis, suzanne. “creating and sharing  opportunities for lifelong learning.”  education for library and information  services: a festschrift to celebrate  thirty years of library education at  charles sturt university. eds. philip  hider and bob pymm wagga wagga,  nsw: centre for information studies  charles sturt university, occasional  publications no. 2: 121‐4, 2006. 24 feb.  2007  <http://www.csu.edu.au/faculty/sciagr/ sis/cis/epubs/libeduc/fschrift_lifelon g_learn.pdf>.     marshall, joanne gard, and carol perryman.  “inls 210‐096: evidence based  information practice.” chapel hill, nc:  university of north carolina at chapel  hill, school of information and library  science. 24 feb. 2007  <http://www.unc.edu/~cperryma/teac hing/inls210‐096%20syllabus.htm>.    meijers, judith m.m., maaike a.p. janssen,  greta g. cummings, lars wallin,  carole a. estabrooks, and ruud y.g.  halfens. “assessing the relationships  between contextual factors and  research utilization in nursing:  http://www.biomedcentral.com/1472%e2%80%90 http://www.csu.edu.au/faculty/sciagr/ http://www.unc.edu/~cperryma/teac evidence based library and information practice 2007, 2:1  106 systematic literature review.” journal  of advanced nursing 55.5 (2006): 622‐ 35.    nakayama takeo, nobuko hirai, shigeaki  yamazaki, and mariko naito.  “adoption of structured abstracts by  general medical journals and format  for a structured abstract.” journal of  the medical library association 93.2  (apr. 2005): 237‐42. 24 feb. 2007  <http://www.pubmedcentral.nih.gov/a rticlerender.fcgi?artid=1082941>.     partridge, helen, and gillian hallam.  “educating the millennial generation  for evidence based information  practice.” library hi tech 24.3 (2006):  400‐19.    pearce‐smith, nicola. “is evidence based  librarianship just an attractive  theory, or can practicing librarians  make it work in real life?“ eahil  workshop. implementation of quality  systems and certification of  biomedical libraries. palermo. 23‐25  june 2005.      perryman, carol, and dihui lu. “finding  our foundation: analysis of the  library and information science  abstracts database for research article  retrievability.” medical library  association annual meeting, phoenix,  az. 23 may 2006.    plint amy c., david moher, andra  morrison, kenneth schulz, douglas g.  altman, catherine hill, and isabelle  gaboury. “does the consort  checklist improve the quality of  reports of randomised controlled  trials? a systematic review.” medical  journal of australia. 185.5 (4 sept.  2006): 263‐7. 24 feb. 2007  <http://www.mja.com.au/public/issues/ 185_05_040906/pli11098_fm.html>.   sackett, david l. “the sins of expertness  and a proposal for redemption.” bmj.  320.7244 (6 may 2000): 1283. 24 feb.  2007  <http://www.bmj.com/cgi/content/full/ 320/7244/1283>.     wright, wilbur. (1908)  speech presented to  the aero club of france, paris, france.    yu, fei, jan sullivan, and leith woodall.  “what can students’ bibliographies  tell us? evidence based information  skills teaching for engineering  students.” evidence based library and  information practice 1.2 (2006): 12‐22.   24 feb. 2007  <http://ejournals.library.ualberta.ca/ind ex.php/eblip/article/view/8/123>.      http://www.pubmedcentral.nih.gov/a http://www.mja.com.au/public/issues/ http://www.bmj.com/cgi/content/full/ http://ejournals.library.ualberta.ca/ind why do i have to write that?: compositionists identify disconnects between student and instructor conceptions of research writing that can inform teaching   a review of: schwegler, r. a., and shamoon, l. k. (1982). the aims and process of the research paper. college english, 44(8), 817-824.   reviewed by: andrea baer undergraduate education librarian indiana university-bloomington bloomington, indiana, united states of america email: apbaer@indiana.edu   received: 30 nov. 2013 accepted: 10 march 2014      2013 baer. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objectives – this classic article discusses research-based writing assignments. schwegler and shamoon sought to identify differences between college students’ and college instructors’ conceptions of research and research paper assignments, particularly in terms of their purpose and process. the authors also sought to identify common features of academic research writing that could inform writing instruction about research writing.   design – qualitative interviews with college instructors and students about their views of the research process and about forms of research writing. instructors were also interviewed about evaluation standards for academic research papers.   setting – unspecified, though the description suggests a college or university in the united states.   subjects – college instructors and college students. (number of subjects unspecified.)   methods – the authors, a university writing program director and a writing program instructor, conducted one-on-one interviews with college instructors and students about their views of research and the research paper. questions focused on conceptions of the research process, the purposes of research, and the forms that research writing takes. instructors were also asked about standards for effective evaluation of research papers.   the limited description of the research methods and interview questions employed in this study hinder the ability to critically assess its validity and reliability. potential limitations of the study, such as selection bias or unclear wording of interview questions, cannot be adequately assessed based on the provided information. the authors also do not identify limitations of their study. as is discussed in more detail in this review’s commentary, the study does not conform to the conventions of most research studies from the behavioral, health, physical, and social sciences. the authors’ methods, however, may be better understood in light of particular disciplinary approaches and debates in composition studies.   main results – interviewees’ responses illustrated notable differences between college instructors’ and college students’ conceptions of the process, purpose, forms, and audiences of research paper assignments. while instructors understood the research paper to be argumentative, analytical, and interpretive, students generally described it as informative and factual. students, when asked why research papers are assigned, identified purposes such as learning more about a topic, demonstrating one’s knowledge, or learning to use the library. instructors indicated that the purpose of the research paper includes testing a theory, building on previous research, and exploring a problem that has been presented by other research or events (p. 819). at the same time, most instructors described research as an ongoing pursuit of “an elusive truth” (p. 819), rather than as primarily factual in nature. according to schwegler and shamoon, instructors also indicated during interviews that research and writing involve a clear though complex pattern that is evident in the structure and conventions of research papers. for example, the research process usually begins with activities like reading, note-taking, identifying problems with and gaps in current research, and conversing with colleagues. these instructors also reported that writing conventions which are implicitly understood in their fields are used by other scholars to evaluate their peers’ work.   reflecting on these interview responses, schwegler and shamoon suggest that pedagogical approaches to writing instruction can be informed both by acknowledging disparities in students’ and instructors’ conceptions of research and by identifying shared characteristics of academic writing. the authors therefore make several general observations about the nature of professional research papers and describe the structure and conventions of academic research papers. they conclude that the structure of scholarly research papers across the disciplines reflects the research process. such a paper opens with identification of a research problem and a review of current knowledge and is followed by a variation of four possible patterns: 1) review of research, 2) application or implementation of a theory, 3) refute, refine, or replicate prior research, and 4) testing a hypothesis (pp. 822-823). schwegler and shamoon indicate that the key features of scholars’ writings are also apparent in student research papers which instructors evaluate as highly-ranked and absent in lower-ranked papers. furthermore, they provide an appendix that outlines the essential textual features of a research paper (appendix a) (p. 822). it is unclear, however, if these descriptions of scholarly research writing are based on the instructor interviews or on other sources, such as previous analytical studies or an analysis of academic research papers from various disciplines. the researchers do not articulate the specific methods used to arrive at their generalizations.   conclusion – the authors conclude that students’ and instructors’ differing conceptions of the research process and the research paper have important implications for writing instruction. many of the interviewed instructors described research as involving methods that are quite different from those needed for most research paper assignments. the discrepancies between class assignments and academics’ approaches to research suggests that differences in instructors’ and students’ views of research often are not addressed in the design of research paper assignments. instructors who teach the research paper should ensure that the purpose, structure, and style of assignments reflect what content-area instructors will expect from students. schwegler and shamoon argue that because the basic conventions of the research paper generally apply across disciplines, instruction about those conventions can be integrated into composition courses and lower-level undergraduate courses. such an approach can assist students in better understanding and approaching research writing as would a scholar in the given discipline.     commentary   schwegler and shamoon’s 1982 article was published during the rise of the writing across the curriculum (wac) movement in higher education. college writing programs, many of which had been established in the 1960s and 1970s, were recognizing that for writing instruction to be most effective and meaningful it must be taken beyond the freshmen composition course and integrated throughout curricula. many college writing programs therefore were developing writing across the curriculum and writing in the disciplines (wid) programs and initiatives. for librarians this may call to mind recent and ongoing efforts to integrate information literacy into undergraduate education.   schwegler and shamoon’s 1982 study reflects concerns of writing instructors and wac proponents frustrated by the limitations of the generic research paper and the mandatory freshman composition course. the questionable value of the standard research paper assignment would gain further attention that same year with richard larson’s frequently cited article “the ‘research paper’ in the writing course: a non-form of writing” (1982). larson’s description of the standard research paper assignment as a decontextualized, artificial, and inauthentic writing task that does not foster genuine inquiry still resonates with college teachers across academic fields.   the wac movement, which would expand significantly in the 1980s and 1990s, has greatly influenced – and continues to shape – undergraduate curricula and writing programs.  nonetheless, the generic research paper remains a common assignment, and debates about if or  where “the research paper” should exist in undergraduate curricula remain part of an ongoing debate in composition studies (e.g. larson, 1982; schwegler & shamoon, 1982; ballenger, 1999; davis & shadle, 2000; mcdonald, 2000; mezler & zemliansky, 2003; hood, 2010). schwegler and shamoon’s “the aims and process of the research paper” (1982) is repeatedly cited within such discussions, as schwegler and shamoon sought to examine the deeper roots of the research paper’s flaws.   disciplinary contexts & research methods   schwegler and shamoon’s study, while most often cited in the composition studies literature, has also received the attention of some librarians and library and information science scholars interested in information literacy and student information seeking behaviors (sheridan, 1992; fister, 1993; hubbard, 1995; rabinowitz, 2000; nutefall & ryder, 2010). the implications of schwegler and shamoon’s article for information literacy and library instructional services, however, are more far reaching than is suggested by the frequency with which the article is cited in the library science literature. as information literacy education moves to more collaborative, integrated models and shifts its focus from more mechanical aspects of information seeking to higher order skills like critical thinking, rhetorical analysis, and source use, the cross-disciplinary relevance of studies like schwegler and shamoon’s becomes increasingly evident.   interdisciplinary approaches, of course, often require some understanding of critical frameworks and methods common within various disciplines. schwegler and shamoon’s work may be better understood when contextualized within its disciplinary and sociohistorical origins. many empirical researchers might give pause when considering schwegler and shamoon’s research methods, which, as the abstract above indicates, remain largely unclear. the authors provide little description of who the subjects were or how they were chosen. nor do they identify the specific interview questions posed or the duration of the interviews. because of this lack of detail, the study cannot be replicated and specific limitations in the research design and findings are difficult to pinpoint.   for most researchers in library and information sciences, and for many in composition studies, this raises questions of validity and reliability. yet, it is important to recognize the sociohistorical moment in composition studies and in higher education when this publication appears. throughout composition studies’ (relatively short) history as an academic discipline, a tension has existed among compositionists who view their research and scholarship in more humanistic or more empirical terms. many in the discipline argue that the notion of empiricism as a means for representing human experience fully and accurately is a myth sometimes used to obscure research biases, cultural biases, and the varied nature of human experience (johanek, 2000; driscoll, 2009). along with this critique, some composition researchers believe that human experiences such as literacy development are best expressed through narrative and descriptions of individual experiences rather than through quantified data (berkenkotter, 1993; roberts-miller, 2002; driscoll, 2009). others contend that for research to be most meaningful it must be replicable and data-driven (johanek, 2000; haswell, 2005; discoll & perdue, 2012). the scope of this article does not allow for a detailed discussion of this debate, but the lack of critique leveled at schwegler and shamoon’s methods may be better understood in light of these variations in composition research methods.   related research studies   despite the methodological limitations of schwegler and shamoon’s study, other research from both composition studies and library and information sciences has yielded similar results. while the research methods of these different studies vary, their findings appear fairly consistent and suggest legitimacy in schwegler and shamoon’s main assertions. among the earliest of these related studies is that of the compositionists nelson & hayes (1988). in a two-part study, they examined students’ and instructors’ views of and approaches to research through student writing process logs, instructor interviews, and analysis of research assignment prompts. similar to schwegler & shamoon, nelson and hayes found that most students view research as an act of fact-finding and apply “low-investment” strategies which reflect a fact-finding approach to information gathering. in the first of their two-part study, however, advanced students (upper classmen and graduate students) usually applied “high-investment” research strategies which were driven by inquiry into and analysis of issues, in contrast to college freshmen.   perhaps even more significant are the results from the second part of nelson and hayes (1988) study, in which student research strategies were analyzed alongside the related assignments. the results indicate that the nature of assignments and accompanying instruction powerfully influence students’ research processes. like the advanced students of the study’s first portion, students who were given scaffolded assignments that emphasized process and incorporated instructor feedback at various stages tended to take an issueand analysis-driven approach to research. these individuals also invested more time and effort in their work. this stood in contrast to the tendency of most students (whose assignments did not incorporate scaffolding or instructor feedback) to focus on information gathering and “low-investment” strategies. (nelson & hayes, 1988).   limberg, through phenomenological research, has similarly noted that students tend to understand research in terms of fact-finding. in a series of interviews with high school seniors at various stages in completing a research assignment, limberg (1999) identified three common ways students experienced information seeking and use: as fact-finding, as balancing information in order to choose the appropriate information, and as scrutinizing and analyzing. this third category of information use, scrutinizing and analyzing, was the least common conception. students’ understanding of information seeking and use appeared to correspond with their research strategies: those who focused on discrete pieces of information and “surface” approaches to research described their purpose as fact-finding; those who took a “deep or holistic approach” which analyzed and related sources to one another perceived their research purpose in those terms. limberg (1999) has noted important implications these findings have for pedagogy, particularly library instruction, which often focuses primarily on tools for locating sources and which may influence students’ understandings of information seeking (p. 11).   fortunately, student perceptions of the research process are not necessarily fixed, and instruction may facilitate more sophisticated understanding of information use. this is evident in limberg, et al.’s three related research studies (2008), each of which indicated that a focus on learning goals and content fosters more sophisticated practices of information seeking and use. instruction that encouraged more complex understanding of and approaches to research stressed the quality of research questions, negotiation of learning goals between students and teachers, and source evaluation. use of technological tools, on the other hand, tended to strengthen an orientation toward procedure and skills. pedagogy’s influence on student views of research is also supported by nelson and hayes’ (1988) observations about assignment design and further research by limberg and others (nelson, 1990; limberg & sundin, 2006; limberg, alexandersson, lantz-andersson, & folkesson, 2008; holliday & rogers, 2013,).   despite the teaching which emphasizes that an inquiry-based approach to research appears to be more effective in encouraging deeper engagement with research writing, in practice both students and instructors appear to focus more on procedure and skills than on knowledge content or learning process. in limberg and sundin’s 2006 study, instructors’ intended learning goals, which were more process-centered, usually did not align with their pedagogical practices. interviews with librarians and teachers at schools from preschool to universities reflected great discrepancies between instructional content and the assessment criteria used to determine the quality of students’ information seeking. while instruction tended to focus on the procedures of locating information, the assessment criteria centered on more complex abilities related to source use such as reading and understanding source content, critically evaluating information, and synthesizing information from various sources. (limberg & sundin, 2006). this suggests that what teachers wanted students to learn was not actually taught.   holliday and rogers’ (2013) observational study of research instruction in a college writing course is further evidence that college educators may reinforce a conception of research as fact-finding. the researchers noted that the majority of instructors’ course content and writing assignments described sources as objects (or containers of facts), while placing little emphasis on the act of learning about sources. holliday and rogers (2013), reflecting on both limberg and sundin’s findings (2006) and their own observational study, conclude that classroom discourse on the research process may influence how students view research and writing. more specifically, an emphasis on “finding sources” may limit student engagement with research as a process of inquiry.   in addition to reinforcing students’ views of research as fact-finding, tool-based instruction may also encourage students to prioritize the end product of research over its process. through interviews and process logs from college freshmen completing writing assignments, nelson (1990) found that students concentrate more on the final product than on process. in doing so, students often develop shortcuts for completing assignments that circumvent the learning processes their instructors intended for them. however, nelson (1990) also found that assignment design can facilitate student engagement with the writing and learning process, particularly through the use of evaluation criteria, instructor feedback, instructions, and other assignment-related support materials. these results align with her earlier study, discussed above (nelson & hayes, 1988).   the student concern with product over process appears closely tied to an emphasis on grades. in interviews about research paper assignments, college students identified grades as their chief concern and described strategies for determining an instructor’s expectations and the most timeand energy-efficient way to receive an acceptable grade (valentine, 2001). some students only looked at the objective criteria of their assignments, such as the number of required pages and sources (valentine, 2001, p. 110). gathering the appropriate number of sources or the appropriate types of sources (e.g., scholarly articles, books) was perceived to be more important than the process of inquiry or knowledge production. this suggests an emphasis on objective assignment criteria over the purpose and process of a research paper again mirror a fact-finding approach to research.   the idea that students tend to apply limited rhetorical analysis or critical thought to research writing assignments is further supported by studies of plagiarism. howard, rodrigue, & serviss (2010), in a detailed analysis of 18 college student papers, found that plagiarism and patchwork are commonplace. the authors contend that the frequency of student plagiarism and patchwork writing may be due more to a lack of engaging with and understanding sources, rather than to an attempt to cheat, since there was little evidence that students comprehended the content of their information sources (howard, rodrigue, & serviss, 2010). in keeping with the idea that students often approach research writing as a process of uncritically patching together facts, head and eisenberg (2010) found that students tend to consistently use the same research strategies and sources, regardless of the task at hand (head & eisenberg, 2010). the inclination to apply the same search strategies regardless of rhetorical purpose again may reflect a view of research as an act of fact-gathering.   implications for information literacy instruction   these various studies have strong, and generally consistent, implications for information literacy education. instructors across disciplines, including writing and library instructors, often experience a disconnect between how they and their students approach research and information use. this discrepancy is often evident in student research papers that fall short of instructor expectations. in practice, however, instruction often does not encourage the more inquiry-based approach that many educators hope students will apply to research.   the studies discussed above indicate that pedagogies which represent and support student research as a recursive process of inquiry, critical thinking, and knowledge production can help students understand research in the terms of inquiry and analysis which appear to concern instructors most. effective pedagogical practices include: breaking down the research process through staged assignments and learning activities, providing instructor feedback throughout the learning process, emphasizing the value of genuine questions and investigation, and inviting students to reflect on their own learning and research process. in contrast, the tool-based instruction that traditionally has characterized information literacy instruction may communicate to students that research is a mere matter of gathering sources to insert into a paper.   as instruction librarians now often argue, the complex skills needed for meaningful engagement with research indicate that for information literacy education to be most effective it must be integrated into assignments and course content, rather than limited to one or two class sessions. as librarians experience considerable challenges in shifting from traditional instructional models to more collaborative partnerships, many librarians are redefining the role and relevance of their instructional services. such efforts, occurring both within and beyond library walls, will, it is hoped, continue to grow substantially.   the need to develop more collaborative and cross-disciplinary partnerships is also true for researchers of composition studies and information sciences. the studies discussed above emerge primarily from these two fields. given the strong connections between these various studies, it is notable that research in these disciplines has not intersected more often. while interdisciplinary effort is increasing, citation patterns, along with discussions with both librarians and writing instructors, suggest that these collaborations are still limited. as the information literacy movement places increasing emphasis on critical thinking, transferable skills, and research within the disciplines and on information literacy integration, the time appears ripe for cultivating more cross-disciplinary conversations and research.     references   ballenger, b. p. (1999). beyond note cards: rethinking the freshman research paper. portsmouth, nh: boynton/cook publishers : heinemann. berkenkotter, c. (1993). a “rhetoric for naturalistic inquiry” and the question of genre. research in the teaching of english, 27(3), 293–304.   davis, r., & shadle, m. (2000). “building a mystery”: alternative research writing and the academic act of seeking. college composition and communication, 51(3), 417-446. doi:10.2307/358743 driscoll, d. l. (2009). composition studies, professional writing and empirical research: a skeptical view. journal of technical writing and communication, 39(2), 195–205. doi: 10.2190/tw.39.2.e driscoll, d, & perdue, s. w. (2012). theory, lore, and more: an analysis of rad research in the writing center journal, 1980-2009. the writing center journal, 32(1), p. 11-39.   fister, b. (1993). teaching the rhetorical dimensions of research. research strategies 11(4), 211-219. retrieved december 20, 2013, from http://homepages.gac.edu/~fister/rs.html   haswell, r. h. (2005). ncte/cccc’s recent war on scholarship. written communication, 22(2), 198-223. doi: 10.1177/0741088305275367   head, a. j. & eisenberg, m. b. (2010). truth be told: how college students evaluate and use information in the digital age. project information literacy progress report (2010). retrieved december 15, 2013, from http://projectinfolit.org/images/pdfs/pil_fall2010_survey_fullreport1.pdf   holliday, w., & rogers, j. (2013). talking about information literacy: the mediating role of discourse in a college writing classroom. portal: libraries and the academy, 13(3), 257-271. retrieved december 1, 2013, from http://www.press.jhu.edu/journals/portal_libraries_and_the_academy/portal_pre_print/current/articles/13.3holliday.pdf hood, c. l. (2010). ways of research: the status of the traditional research paper assignment in first-year writing/composition courses. composition forum, 22. retrieved december 20, 2013, from http://compositionforum.com/issue/22/ways-of-research.php .   howard, r. m., rodrigue, t. k., & serviss, t. c. (2010) “writing from sources, writing from sentences.” writing and pedagogy 2.2 (fall 2010): 177-192. retrieved november 15, 2013, from http://writing.byu.edu/static/documents/org/1176.pdf   hubbard, t. e. (1995). bibliographic instruction and postmodern pedagogy. library trends, 44(2), 439-452.   johanek, c. (2000). composing research: a contextualist paradigm for rhetoric and composition. logan, ut: utah state university press.   larson, r. l. (1982). the “research paper” in the writing course: a non-form of writing. college english, 44(8), 811-816.   louise limberg. (1999). experiencing information seeking and learning: a study of the interaction between two phenomena. information research: an international electronic journal, (1), 68.   limberg, l., alexandersson, m., lantz-andersson, a., & folkesson, l. (2008). what matters?: shaping meaningful learning through teaching information literacy. libri, 58(2), 82-91. doi: 10.1515/libr.2008.010   limberg, l. & sundin, o. (2006). teaching information seeking: relating information literacy education to theories of information behavior. information research, 12(1), paper 280. retrieved december 2, 1013, from http://informationr.net/ir/12-1/paper280.html mcdonald, j. c. (2000). beyond formalism: current alternatives to traditional research paper instruction. journal of college writing, 3(1), 5–20. melzer, d., & zemliansky, p. (2003). research writing in first-year composition and across disciplines: assignments, attitudes, and student performance. kairos, 8(1). retrieved november 20, 2013, from http://kairos.technorhetoric.net/8.1/features/melzer/kairosfront.htm nelson, j. (1990). “this was an easy assignment”: examining how students interpret academic writing tasks. berkeley, ca: center for the study of writing. technical report no. 43.   nelson, j., & hayes, j. r. (1988). how the writing context shapes college students’ strategies for writing from sources. berkeley, ca: center for the study of writing. technical report 16. retrieved november 15, 2013, from http://www.nwp.org/cs/public/print/resource/602   nutefall, j. e., & ryder, p. m. (2010). the timing of the research question: first-year writing faculty and instruction librarians’ differing perspectives. portal: libraries and the academy 10(4), 437–449. doi: 10.1353/pla.2010.0009 rabinowitz, c. (2000). working in a vacuum: a study of the literature of student research and writing. research strategies, 17(4), 337-346. roberts-miller, p. (2002). post-contemporary composition: social constructivism and its alternatives. composition studies, 30(1), 97–116.   sheridan, j. (1992). wac and libraries: a look at the literature. journal of academic librarianship, 18(2), 90.   valentine, b. (2001). the legitimate effort in research papers: student commitment versus faculty expectations. journal of academic librarianship, 27(2), 107.   evidence summary   enrollment in a library credit course is positively related to the college graduation rates of full time students   a review of: cook, j.m. (2014). a library credit course and student success rates: a longitudinal study. college & research libraries 75(3), 272-283. doi:10.5860/crl12-424   reviewed by: sara sharun assistant professor mount royal university library calgary, alberta, canada email: ssharun@mtroyal.ca   received: 27 feb. 2015   accepted: 6 apr. 2015      2015 sharun. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine the impact of a credit-bearing information literacy skills course on student success rates.   design – observational study.   setting – an academic library at a mid-sized university in georgia, united states of america.   subjects – nine cohorts of students (n=15,012) who entered the institution for the first time, on a full-time basis, each year between 1999 and 2007.   methods – aggregate data on each student cohort was gathered from the department of institutional research and planning. data included high school act and sat scores, high school graduating gpas, college graduating gpas, and college graduation dates. the nine cohorts were each divided into two groups: students who took a credit library course (libr 1101) at some point during their student career, and students who did not. for each cohort, a pearson chi-square test was used to determine statistical correlation between library course enrollment and four-, five-, and six-year graduation rates. z-tests were used to determine a difference in the average graduation gpa of students who did and did not take the course, as well as a difference in the average high school graduation gpa, act, and sat scores of the two groups in each cohort.   main results – graduation rates were positively associated with students who took the library course at some point during their studies. students who took the library course graduated at higher rates than students who did not: 56% of those students who took the library course graduated within the study’s time frame, compared to 30% of those who did not take the course. on average, there was no significant difference in college graduation gpas between students who did and did not take libr 1101. during the time period of the study, more students who took the course graduated than those who did not, but those students who took the course did not have higher graduating gpas.   conclusion – students who enrolled in libr 1101 at some point in their studies graduated at a significantly higher rate than students who did not.   commentary   this is an instructive example of how to employ institutional analysis and planning data to measure library impact on student success. this is one of many similar studies aimed at quantitatively and qualitatively proving the value of academic libraries, but the distinct contribution of this article is its use of a large secondary dataset which not only allowed the author to study 15,000 students over a 9-year period, but also allowed her to contextualize and compare data on library course registration with other institutional data (aggregate graduation rates, college graduation gpas, and high school gpas and test scores).   interestingly, this institutional data that in some ways offered unique potential for impactful evidence also presented the study’s most significant threat to the validity of that evidence. this study does not meet the standards of validity established by glynn’s (2006) critical appraisal tool, primarily because of challenges presented by the non-random, non-representative sample, and the biased population selection and inclusion/exclusion criteria. these limitations were the result of a decision to gather and present data that could be compared to the institution’s official metrics, and are clearly noted by the author. it is significant, therefore, that the author makes no mention of the impact (realized or potential) of sharing this information with institutional stakeholders, which was the initial purpose for this study and the reason for proceeding with the study in the face of methodological challenges that would potentially impact the study’s validity.   despite the methodological issues that prevent this study from being generalizable, either to other forms of library instruction or to credit courses on other campuses, it does effectively present evidence of a positive correlation between enrollment in a library instruction course and graduation rate. the author also raises good questions for future research, related to impacts of timing of library instruction in a student’s career, instructional delivery method, and instruction on students with different levels of academic achievement.   although the quality of evidence presented by this particular study is weak in itself, the author recognizes that it is only through the “gathering together an increasing number of suggestive correlative studies” (p. 282) that librarians can begin to present a stronger argument for the value of academic libraries’ instruction efforts and the impact of instructional practices on certain measures of student success. this study is potentially reproducible in other institutions that have access to similar institutional data. and, while the limitations of this study identified by the author are significant and limit its external validity, it lays the groundwork for future studies on the associations between library instruction and student outcomes using existing institutional planning data.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154 evidence summary   librarians are attracted to blogs that support professional continuing education   a review of: jackson-brown, g.m. (2013). content analysis study of librarian blogs: professional development and other uses. first monday, 18(2). retrieved 4 sept. 2013 from http://dx.doi.org/10.5210%2ffm.v18i2.4343   reviewed by: laura newton miller collections assessment librarian carleton university library ottawa, ontario, canada email: laura.newtonmiller@carleton.ca   received: 27 may 2013  accepted: 2 aug. 2013      2013 newton miller. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     objective – the purpose of the study was to examine how librarian blogs are being used for communication within the profession.   design – the method used was content analysis and unstructured interviews.   setting the researcher is based out of a state university in the united states of america.   subjects content of and communication within 12 librarian blogs were analyzed. seven of the 12 bloggers were interviewed.   methodology – there were 15 blogs identified in a list by quinn (2009) and reduced to the 12 best suited for the study. over a 24-month period (january 2009-december 2010), random samples of posts with 2 or more comments were selected for each month from the 12 blogs and analyzed. all comments related to these selected posts were also analyzed. the researcher categorized the blogs overall, plus individual posts, into one of four predominant genres (social, professional development, political, and research). content was coded based on previous coding methodology for blog content found in the research literature. requests for interviews were sent to all 12 bloggers with 7 agreeing to be interviewed. preliminary results of the content analysis for his/her own blog were shared with each blogger before the interview took place. inter-coder reliability was pretested and found to be 83.33%.   main results two hundred eighty-eight posts randomly chosen received 1936 reader comments. bloggers responded to these comments 254 times. blogs were categorized under the “social” genre most frequently (53%), followed by “professional development” (31%), “political” (14%), and “research” (2%; percentages were rounded to the nearest whole number by the reviewer). professional development was the lead genre in two of the individual blogs. all seven bloggers interviewed stated that professional development is a large focus of their blogs. reasons for blogging ranged from the importance of sharing information, contextualizing information, and (for some) satisfying personal ambition. there was a common personal enjoyment of writing and all planned to continue blogging despite increasing time constraints.   conclusion professional development is a major focus of content in librarian blogs. blog posts and comments stay on topic throughout exchanges between bloggers and readers.     commentary   social media is a vast and ever-evolving entity. many librarians use tools like blogs and twitter to keep up to date on issues that affect the profession. this paper is the latest regarding the use of blogs in academia. although others have written about library/librarian-specific blogs (see citations below), this paper categorizes blog content and reader comments into specific genres.   using the ebl critical appraisal checklist (glynn, 2006), the reviewer found the content analysis and interviews appropriate choices for data collection. one can appreciate the difficulty of narrowing down a list of blogs from the many available on the web. however, in narrowing the list available for content analysis to a list from another paper about librarian blogs (quinn, 2009), the strengths of the paper are put into question.   the author used a selected list of 12 blogs from an article entitled “learning with blogs” (quinn, 2009). first, this is a very small number in light of so many individual librarian blogs available. however, what’s more troubling is that the basis of the list from “learning with blogs” is that they are included because of their value as continuing education tools. the researcher concludes that professional development is the major focus for librarian blogs, but her source is a list of blogs valued for their professional development focus. the paper would have been much stronger had she used a variety of sources to form a list of blogs for content analysis (e.g., aharony, 2009). albeit an older paper with focus on organizational and individual blogs, bar-ilan (2007) goes into great detail regarding how blogs were chosen for content analysis.   bar-ilan (2007) was not mentioned in the researcher’s citations. nor were others of relevance, and this also limits the paper’s effectiveness. the researcher states that “genres used in the content analysis [were] . . . based on professional and scholarly literature” (research design section, para. 6) and that coding was “derived from previous conceptual constructs of blog content from professionals and academics that [were] found in previous research and published literature” (research design section, para. 9). a list of that literature would have made for a stronger paper.   it is interesting that the research genre had the smallest content area in librarian blogs. is this a reflection of the state of librarian research publishing in general? or is it more reflective of the fact that librarians are reserving their writing for more peer-reviewed/scholarly outlets? although there are benefits to blogging (e.g., collaboration, networking), there is also a hesitancy to use one’s time writing in a self-publishing venue that may not “count” in terms of tenure or promotion.   the time period studied, january 2009-december 2010, is a long time ago in terms of social media, and time is always a constraint in social media research. as mentioned by one interviewee, people’s attention is moving more toward twitter. this reviewer can tell by her rss feed reader results that many blogs that were once active a few years ago have either discontinued completely or now post significantly less content. the researcher is to be commended for trying to pinpoint categories in the moving target of social media. although there are issues with this research paper, it does contribute to the knowledge base regarding genre theory, social media, and its use for librarians as a professional development tool. a selection of recommended further readings on this topic is listed below.     references   aharony, n. (2009). librarians and information scientists in the blogosphere: an exploratory analysis. library & information science research, 31(3), 174-181. retrieved 4 sept. 2013 from http://dx.doi.org/10.1016/j.lisr.2009.02.001   bar-ilan, j. (2007). the use of weblogs (blogs) by librarians and libraries to disseminate information. information research, 12(4). 1-24. retrieved 4 sept. 2013 from http://informationr.net/ir/12-4/paper323.html   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi:10.1108/07378830610692154   quinn, m.e. (2009). learning with blogs: selected blogs that will enlighten and inform every library professional. american libraries, 40(8-9), 59-61.   stephens, m. (2008). the pragmatic biblioblogger: examining the motivations and observations of early adopter librarian bloggers. internet references services quarterly, 13(4), 311-345. doi:10.1080/10875300802326475 article   the impact of public access venue information and communication technologies in botswana public libraries   angelina totolo senior lecturer, department of library and information studies faculty of humanities, university of botswana gaborone, botswana email: totoloa@mopipi.ub.bw   jaco renken lecturer, management and information systems centre for development informatics, university of manchester manchester, england, united kingdom email: renken@manchester.ac.uk   araba sey research assistant professor technology and social change group university of washington seattle, washington, united states of america email: arabasey@uw.edu   received: 23 jan. 2015    accepted: 13 may 2015        2015 totolo, renken, and sey. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – a study on the impact of public access venue (pav) information and communication technologies (icts) was conducted in botswana libraries with internet connections. the main objective was to determine the impact of icts in public libraries.   methods –using the sustainable livelihoods framework as a theoretical lens, the study used semi-structured interviews and focus group discussions to investigate the impact of pav icts in 4 study sites, resulting in data from a total of 39 interviews and 4 focus groups.   results – the results of the study show that pav icts had a positive impact on users in the areas of education and economic benefits. within educational and economic impacts, social benefits were also found, pertaining to the use of social media and the internet for formal and informal communication. the study also revealed a slight difference between school going users and non-school going elderly users where the use and acquisition of computer skills was concerned. elderly non-school going users tended to rely on venue staff for skills more than the younger school going users.   conclusion – the study recommends that pav facilities should be improved in terms of skills offered and resources availed so as to appeal to both the younger school going generation and the older non-school going users. it is also recommended that education on ict be improved to help curb rising unemployment in botswana; such skills would enhance the income generation skills of the unemployed users as well as school leavers. introduction   a qualitative study was carried out to investigate the impact of public access venue (pav) information and communication technologies (icts) located in botswana libraries. botswana is a land-locked country in south africa that shares borders with south africa, namibia, zimbabwe, and zambia. botswana has a projected population of 1,926.872 people in 2015 (population & housing census, 2011). the government of botswana through the ministry of youth, sport and culture (mysc) and african comprehensive hiv/aids partnerships (achap), collaborated using a grant from bill and melinda gates foundation (gates foundation) to equip public and community libraries with ict facilities and training.   this paper makes a distinction between public libraries and community libraries. the former were built by the botswana national library service while the later were built and sponsored by the rothschild foundation. both public and community libraries fall under the botswana national library service (bnls). bnls was established in 1967, as a government department for the provision of library services. bnls is   responsible for community libraries, public libraries, and village reading rooms (vrrs) (isaak, 2000). the later were introduced in 1986 as an extension of public library service in rural areas, and they are smaller than public libraries in size and breadth of services (baffour-awuah & pilane, 2001).   the phrase public access venue is used to refer to libraries (public and community), village reading rooms (vrrs), cybercafés, and telecentres (the latter are known as kitsong centres in botswana). the phrase is used to show ict availability either for free, as in libraries, or at a cost in cybercafés. a national data inventory in 2009 revealed a total of 164 public access venues for the provision of information to the nation. this number included 96 public and community libraries, as well as village reading rooms (all under the botswana national library service) and 48 non library venues such as kitsong centres, private internet cafes, and other resource centres (grand et al., 2010). today the number of public access venues has increased. there are 27 public libraries, 7 community libraries (acquired through the rothschild endowment), and 69 village reading rooms (sesigo botswana public e-libraries, 2010).   this study focused on the impact of free public access icts in public and community libraries that were equipped with computers and internet access through the bill and melinda gates foundation and the sesigo project (sesigo is a botswana national language word which means a storage for harvest). by 2010 the number of public access venues under bnls had risen from 96 to 98 (sesigo botswana public e-libraries, 2010). the sesigo project aimed at computerizing and networking public and community libraries as well as building the capacity for library employees to serve in e-enabled environments across botswana (mutula, grand, zulu & sebina, 2010). about 78 out of 98 of all venues (branch libraries, village reading rooms, and community libraries) were targeted. through this project, the public is provided with free access to computers, internet, and ict training.   botswana information technology policy context   the millennium declaration of 2000, and its eight goals that are referred to as the millennium development goals (mdgs), became the point of departure for practically applying and theoretically understanding icts in socio-economic development (jensen, 2011). ict-focused development efforts can be traced back to the mid-1990s and were focused on providing or improving access to information resources in an emerging knowledge society (parkinson & ramirez, 2006). the rationale is that lack of access contributes to the marginalization of deprived people and that icts can improve that situation.   in line with the millennium declaration, the botswana government initiated projects, policies and strategies for the improvement of access to icts in botswana. to date, policy initiatives include the universal service and access (usa) policy (telecommunications and postal services, 2015), the maitlamo, botswana’s national information and communication technology policy (2004), and long term vision for botswana policy (1997). all these policies aim to improve ict access for the nation. additionally, the botswana government developed telecommunications infrastructure to support information technology access, including “the national fibre backbone infrastructure (trans-kalahari); the international connectivity with the rest of the world through the west africa festoon fibre system (wafs) and the east africa sea cable system (eassy)” (telecommunications and postal services, 2015) and the rural telecommunications strategy (rts) for the improvement of information and communication technologies in rural areas.   ict policies and projects   the botswana government developed the universal service and access (usa) policy in order to improve telecommunications in its entirety. this policy includes internet and postal services, as well as broadcasting and media services (electronic and print). it is envisioned that universal access will be achieved throughout the country through this policy (telecommunications and postal services, 2015). the government of botswana, in line with the maitlamo national information and communication technology policy, has initiated the rural telecommunications strategy as part of the rural telecommunications development initiative (mutula et al., 2010). the government set up ict infrastructure in the villages in botswana in a project known as nteletsa ii. the word “nteletsa” comes from the national language of botswana (setswana) and it means “call me”. therefore “nteletsa” can be construed to mean the enhancement of communication because through this project it is expected that internet and mobile communication access will be improved to facilitate economic activities within communities (telecommunications and postal services, 2015). by the end of the nteletsa ii project, every village in botswana is expected to have a telecentre where the public can access the internet and other telecommunication services at a fee. therefore, this access to icts in the “nteletsa” project is not free for the public. long term vision for botswana policy   the long term vision for botswana policy which has been dubbed “vision 2016” has seven vision pillars, namely: 1) ‘an educated, informed nation’; 2) ‘a prosperous, productive and innovative nation’; 3) ‘a compassionate, just and caring nation’, 4)‘a safe and secure nation’, 5)‘an open, democratic and accountable nation’, 6) ‘a moral and tolerant nation’, 7) ‘a united and proud nation’ (long term vision for botswana, 1997, p.ii). the most relevant pillar for the present paper is vision pillar 1, which aims for universal education, an informed people through access to information, and a literate nation. some of the objectives for vision pillar 1, according to the long term vision for botswana policy document are:   to acquire “the best available information technology . . . .”(p.5). “develop communication capacity . . . .”(p.5). “…the people of botswana will be able to use and apply the potential of computer equipment . . . .” (p.6). the above objectives all aim at improving access to information through enhanced icts. the provision of public access venues like libraries and telecentres fits into the “educated and informed nation” pillar. although the development of telecentres was very instrumental in extending access, this did not extend to public and community libraries; therefore the intervention by donors like bill and melinda gates foundation and the rothschild foundation to build and equip libraries has filled a gap in the information and communication technology scenario in botswana public and community libraries (churu, 2014).   objective   the impact of public access venues (pavs), located in libraries in botswana was investigated. the main areas of investigation centred on the impact of icts on education, income generation, and employment opportunities on the users of ict in free public access venues. the co-location of public access icts with public and community libraries was of particular interest in the present study. at the time of the study, the internet had already reached libraries through aid from donors. the government has contributed by creating relevant policy which enhances the availability of telecommunications and the donors brought the ict facilities for users of the libraries. the focus of the study was on assessing the impact of these facilities on the livelihoods of the users via the theoretical lens of the sustainable livelihoods framework (slf). during the fieldwork, seventeen libraries were equipped with computers and the internet and they were functioning as public access venues (located in public libraries). these libraries constituted part of the sample for this qualitative study. the study used the slf as a theoretical lens because of its popularity in studies of this nature (arun, heeks, & morgan, 2004).   this study was part of a series of global impact studies carried out in several countries around the world. the specific research objectives for this study were:   1) to find out whether pav icts enhance the education of users in botswana. 2) to investigate whether pav icts enhance the employability and income generation strategies of users in botswana.   literature review   the literature review examines the concept of sustainable livelihoods, the ict, and national library context. it covers the socio-economic status of africa and botswana, in line with the purpose of the study which is to determine the impact of icts on the income generation techniques and the education of the users. the education of users includes both school related issues and information literacy skills. income generation is in part an economic issue as well as a political phenomenon where policy is involved. the economic status literature deals with poverty and lack of income while the political situation deals with botswana policy in relation to information and communication technology. socio-economic barriers to icts therefore form part of the basis of the literature review. the sustainable livelihoods framework   the sustainable livelihoods framework (slf) originated in the social sciences as a response to the realization of the defects of conventional professional analysis of poverty: production thinking, employment thinking, and poverty-line thinking (chambers & conway, 1992). the concept of sustainable livelihoods served as a program development catalyst for organizations concerned with development such as the united nations development programme (undp), department for international development (dfid), food and agriculture organization (fao), international fund for agricultural development (ifad), and various research institutes (brocklesby & fisher, 2003). although the methods and frameworks that emanated from these organizations differ, they all use guiding principles from chambers and conway (1992). as a point of departure, sustainable livelihoods approaches propagate the need to understand the livelihoods of poor people in context (arun, heeks & morgan, 2004). they seek specific constraints which inhibit people’s ability to improve their livelihoods in a sustainable way. sustainability is defined as “the ability to maintain and improve livelihoods while maintaining or enhancing the local and global assets and capabilities on which livelihoods depends” (chambers & conway, 1992, p.5).   the slf has five parts: vulnerability context, capital assets, transforming structures and processes, livelihood strategies, and livelihood outcomes (parkinson & ramirez, 2006). the context of users is described as the everyday shocks, trends, culture, and the environment that impact on people’s use of capital assets. in the present study, the culture and environment of libraries without icts can have some influence on how people use icts for a livelihood. this context further impacts on the available capital assets in the form of physical, human, financial, social, and natural resources that shape the people’s livelihoods. the present study concentrated on the social (e.g. ict skill acquisition) and physical assets (e.g. facilities) in order to determine how available assets enhance the livelihoods of users in terms of income generation. accordingly, capital assets impact on the transforming structures and processes and this is where policies and laws are enacted by both government and the private sector to enhance ict uptake, as is the case of the present study (e.g. vision 2016). this study described enacted ict policies and showed how transforming structures and processes can create an environment that is conducive for ict use and uptake. the vulnerability context, capital assets, and transforming structures and processes, also influence the kind of livelihood strategies and livelihood outcomes people would choose for survival. the present study sought to identify impacts of public access icts by identifying the livelihood strategies and outcomes that result from ict use. for example, income generation using public access icts would be regarded as a strategy and positive outcome. the slf is instrumental in studies that seek to find out whether an intervention like ict changes the lives of the marginalized.   socio-economic barriers to ict access   the socio-economic situation with regards to information and communication technologies in botswana needs to be reexamined following the intervention in public access that has been described above. however, based on the literature on sub saharan africa, botswana has not removed all the hurdles that hamper access to icts (maswabi, sethate, sebusang, & taolo, 2011; oladokun and aina, 2011; totolo, 2014). the obstacles to technology diffusion and use in libraries and the country can be categorized as access issues. africa’s lack of communication channels, lack of it education and training, lack of information literacy, non-utilization of computers, underutilization of available computers and other technologies is documented in the literature (bose, 2004; oladokun & aina, 2011; totolo, 2014). most of the barriers to ict access and use in africa are intertwined with the socio-economic situation related to poverty and lack of education (hillborn, 2012; rice, 2003) which invariably lead to policy failure. this can be categorized as the digital divide, which is described as the socio-economic gap caused by lack of resources and opportunities to access ict and use it for various activities. the digital divide is said to prevail between individuals, households and geographic areas (papazafeiropoulou & pouloudi, 2004). the digital divide, which characterizes most of africa (gyamfi, 2005), is also prevalent in botswana (oladokun & aina, 2011; totolo, 2014).   the lack of information literacy skills amongst students, in general, is discussed by several authors (beethan, mcgill, & littlejohn, 2009; head & eisenberg, 2010; head, 2012). lack of information literacy skills has been reported in the botswana literature as well amongst elementary school teachers (bose, 2004), distance education students (oladokun & aina, 2011), and secondary school principals (totolo, 2011). the lack of education in african nations impacts the acquisition of ict skills, which are necessary in the present digital environment. according to duff (2011), the lack of digital skills leads to “socio-economic stratification” (p.604) which is similar to the digital divide.   the high costs of the it equipment and the equally high cost of maintenance of technology in places where poverty is reported to prevail as it does in botswana, (hillborn, 2012) is a real challenge. the government of botswana, through the botswana technology centre (botec) piloted three telecentres in various rural communities in july 2004. these telecentres were called “kitsong centres” (translated to mean “knowledge” centre) therefore, it was hoped that these centres will be used to “acquire knowledge” for sustainable socio-economic development (radijeng, 2009). currently the kitsong centres include those telecentres which were developed under the nteletsa ii project. the government has contracted botswana telecommunications corporation (btc) and a private telecommunications service provider, mascom, to implement these projects. the guiding sustainable livelihoods framework used in this study suggests that the transforming structures and processes (parkinson & ramirez, 2006) should be a mechanism that enhances sustainable livelihoods through government and private sector collaboration. therefore the services offered at these centres which include computer facilities, internet access, fax services, photocopying, and printing, can contribute to sustainable livelihoods. some of the centres are operated by village development committees, parents, and teachers’ association committees from schools. most of the centres, however, are operated by young potential entrepreneurs who are trained in collaboration with the local enterprise authority (lea). the kitsong centres are not free for all users, therefore only those who can afford them can have access.   public library ict access   literature consulted on the effectiveness of library services has revealed that, in general, some public libraries were underutilized in the united states at the time of the study (sin & kyung-sun, 2008). writing on this, sin and kyung-sun (2008) said “while the public library aims to facilitate information access to all, only a portion of the public uses its collections, computer facilities, reference services, or library programs” (p. 207). rosenberg (1997) and sturges (2001) writing on the condition of libraries in africa, argued that when libraries were built, it was expected that they would improve lives. unfortunately most library buildings were not maintained. shelves were either empty or full of outdated and irrelevant materials, and funding was inadequate which resulted in the underutilization of most libraries in africa. many years later, libraries in africa and botswana in particular, had still not improved significantly. in a country report compiled by isaak (2000), and studies conducted by mutshewa et al., (2010), and maswabi et al. (2011), it was discovered that some of the library service problems in botswana include a shortage of qualified staff, limited physical space, shortage of materials, and limited facilities for using information technology.   from the national government perspective, libraries as public institutions need to show the value added to the lives of the citizens (ambrozic, 2003). a baseline study was carried out by researchers from the university of botswana, department of library and information studies, under the sesigo project to find out the status of public access to ict in public libraries and other public access venues such as internet cafes and kitsong centres. the findings showed that libraries did not offer ict related access at that time (grand et al., 2010).   in recent years, researchers studied the impact of public access venues in several countries to measure the impacts of publicly available icts. such research includes studies carried out in eight countries under the leadership of technology and social change group of the university of washington’s information school, in which positive impacts were reported. positive impacts have also been realized in other reports by sesigo projects (2010), and totolo and renken (2012). sey, coward, bar, scaidas, rothschild, and koepke (2013), confirmed the positive impacts of pav icts in the eight countries mentioned above when they said “the results show that a central impact of public access is the promotion of digital inclusion through technology access, information access, and development of ict skills. … users … report positive impacts in various social and economic areas of their lives” (p.2.).   overall it can be concluded that the icts were introduced amidst digital divide challenges (oladokun & aina, 2011), an almost dysfunctional library system (maswabi et al., 2011; mutshewa et al., 2010), and a reported poverty status (hillborn, 2012), yet positive impacts were achieved.   some positive impacts in pav icts include the adoption and use of web 2.0 tools. the use of web 2.0 technologies which has been elusive for areas hard hit by the prevalence of digital divide was another yardstick for judging inclusion and exclusion to digital information in recent studies. web 2.0 technologies have changed the social fabric of society and have introduced many possibilities for improving education worldwide. the prospects of enriching education and communication using web 2.0 technologies have been discussed by several authors (al-aufi & fulton, 2014; badea, 2014; berube, 2011).   heeks, gao, and ospina (2010) introduced what they called the “ict4d value chain” which challenges the socio-economic and political stability of icts in a nation. their argument is that ict policies need to factor in four important areas of operation, namely: readiness, availability, uptake, and impact. for policy to succeed the nation must be ready to make icts available for uptake and use for purposes of impact. historically, ict policy tended to fail in africa (heeks, 2002; rice, 2003), therefore, the suggestion by heeks et al. (2010) of building a systematic impact analysis in policy formulation is useful. for example, the ict objectives which appear under the vision 2016 policy did not encompass the four areas identified by heeks et al., (2010). if they did, all libraries would have been supplied with computers and the internet, but that only happened after the intervention from donors. it is against this backdrop that the gates foundation was introduced to enhance access to icts and to combat the digital divide. in as much as the botswana ict policy context was instrumental in laying the ict framework in the country, the availability of icts in public venues like libraries lagged behind.   summary of the literature   the literature review has shown the importance of using the slf in an ict research study such as this one. it has also illustrated the socio-economic context of the present study, as well as ict policy formulation in botswana. this study, which aims to investigate the impact of icts in pavs, will benefit from the context as described in the literature review, by way of identifying those vulnerabilities and impacts of icts that exist in botswana. botswana has benefited from donors such as the bill and melinda gates foundation and the rothschild foundation and it will be interesting to assess the impacts of these ict developments in the livelihoods of batswana who use pavs. methodology the study used the sustainable livelihoods framework (slf) as a theoretical framework. several studies have employed the use of slf in studying icts and developments (albu & scott, 2001; arun, heeks, & morgan, 2004; chapman, slaymaker, & young, 2001; duncombe, 2006; parkinson & ramirez, 2006; soriano, 2007). according to the slf thinking, there is the vulnerability context (comprising of trends, shocks, culture, and environment) which influences capital assets (natural, social, physical, human, and financial). the capital assets influence the transforming structures and processes (government and private sector laws and policies) which in turn shape the livelihood strategies and outcomes of the citizens (parkinson & ramirez, 2006). the present study did not investigate all the variables in the framework but it used the vulnerability context (environment), some of the capital assets such as human capital development (ict skills) and physical assets (available ict infrastructure), transforming structures and processes (government policy structure), and finally livelihood outcomes in the form of income generation or employability.   the strength of the slf is that it considers icts as a sub-set of the more complex livelihoods of people. in this research, icts available at the public and community libraries in botswana were considered an asset available to users for integration into their livelihood strategies. the role and impact of ict skills obtained from the libraries is part of human capital. the researchers were interested in understanding how the libraries’ icts enhance the livelihoods of users. three development domains were chosen from the larger global impact study for the present study: education; employment; and income generation. these were operationalized into the slf as both livelihood strategies and livelihood outcomes. in one of the earlier studies by albu and scott (2001), the slf was used to better understand the livelihoods of people involved in micro-enterprise. a particularly interesting finding from their study is the importance of micro-enterprises’ capacity to generate and manage technological change in order to maintain a sustainable livelihood. the interviews and focus groups used a purposive sample, appropriate for this research on the “basis of the knowledge of the population” (babbie, 2001, p. 179). focus group discussions and semi-structured interviews were conducted at the four study sites spread across the country and these are gaborone (capital city in the south east), ramokgonami (remote village in central botswana), kasane (tourist township in the north east), and jwaneng (a town in the south). the sampling took into account the different parts of botswana, as well as the different situations in the different regions. in botswana, diamond mining accounts for a third of the economy therefore the mining town of jwaneng was chosen. the second highest source of income is tourism and kasane, which is a tourist site, was chosen. agriculture is another source of income so the agricultural village of ramokgonami was chosen. the urbanized, capital city of gaborone was also included (our africa, botswana economy and industry website).   a total of 39 semi-structured interviews and 4 focus group discussions were conducted. these methods of data collection are useful for qualitative research designs since they are flexible and are not dependent on the sample size as quantitative designs are (babbie, 2001). an exit interview strategy was used, after verifying library usage, with particular attention to gender representation. participants were recruited for interviews during the researcher’s first two days at the library and the focus group discussion was done on the third day. the exit interview was useful because patronage was very low in some of the selected venues and researchers had to wait for users to trickle in.   participants could partake in either english or setswana (setswana is the national language of botswana and english is the official language). the interviews and focus group discussions were recorded with permission. all user interviews and focus group discussions were transcribed verbatim in the used language. setswana speaking research assistants were used for transcribing the setswana interviews. the principal researcher, as a native setswana speaker, assumed responsibility for translating the setswana transcripts to english.   analysis of the data began in the field during data collection using computer assisted qualitative data analysis software (caqdas) to assist in the analysis process. a deductive approach was followed in conceptualizing an initial set of thematic areas and variables for qualitative analysis. the guiding slf and findings from the literature review were used for this purpose.   results   the findings are based on two variables, namely education, and employment (including income generation strategies). the objective was to find out whether the pav facilities and services enhance the education, and employability strategies of the users. positive impact was shown by enhancement in education or employability strategies in this study. the study also explored the level of the impacts, in order to find out the extent of the impact of pav icts on the users.   impact of pav icts on education   in this study, education is regarded as an increase in one’s knowledge in the form of enhanced school work, improved ict skills, and learning new things. the results of the study show a positive impact on the lives of users in terms of acquisition of ict skills, acquisition of online information for research and assignments, scholarship activities online, informal and formal communication.   acquisition of ict skills   it was found that non-school going users of the library benefitted from the ict skills imparted at the library. an example of ict skills benefit was described as follows: “i started on the 10th of october, 2010, when i started working here. somebody came to help us and teach us the computer but at the beginning i was taught by library staff” (participant from ramokgonami). another example which demonstrates the benefit from ict education was when a participant said, “ah! i learnt on my own here in ramokgonami, in this library. i was taught by the library staff here” (participant from ramokgonami). as demonstrated in these cases, the users above seemed to rely on the library staff to impart the knowledge of using computers. the results show that pav users learnt ict skills from the employees of the library and this is a positive impact, however, school going users attributed the acquisition of skills to schools and this emerged in the data. when users were asked where they obtained their ict skills they said: “at school, here in the library they don’t really train us on how to use the internet, you have to come here with an idea of how to email, type, all of which i learned from school” (participant from gaborone). another participant said: “i learnt a little bit about basic computers, about logging in and logging off, in high school when i was doing form 5” (participant from jwaneng). another participant said: “i learnt myself, up until college when i got to learn from doing computer studies” (participant from kasane). from the village, another school going participant said: “the skills … i had a little bit of computer awareness when i was still at junior school but other than that through school i got to realize the basics of computers like word processing” (participant from ramokgonami).   the findings above emphasize that users who were not in school benefited from the skills provided at the library, however, those skills were very basic. this is further confirmed by the reports of younger users who did not need skills from the library staff and mentioned that, “… i had a little bit of computer awareness”. it is apparent from the above quotes that the “basic” knowledge they brought to the library sufficed. users in the quotations above refer to basic skills such as word processing, email, typing, logging in and logging off, and no reference is made to more complicated tasks. in addition, one participant attested to the availability of microsoft word and excel lessons for novice users, when he said: “i personally had the basics before but i remember those months they had lessons for microsoft word and excel….” (participant from jwaneng). according to the above excerpts, there was positive impact on the ict education of the users but it was mostly limited to basic computer skills.   access to online information   users benefited from using the library for research and assignments. the following extracts illustrate what benefits were derived from using the library:   when it comes to studies, i was using this (the icts from the library) a lot to get question papers to study. (participant from kasane).   they (icts at the library) have been benefitting me very much because i did not pass english properly; you see   . . . . then i decided that if i can use these computers as sort of reading these books, it can help me a lot. i think i have benefited a lot because i got a “c” for english which i used to fail and i see that they have been beneficial. (participant from ramokgonami)   looking at the way of my training, there was no way these facilities could not have been important in my training. like if we look at where i come from, like my yard exactly, there is no computer, there is no power. so the fact that these facilities were here i could always carry my work from school to here. i would research and i could even type my assignment here. i could even download my notes and any kind of take-home resources from school. (participant from ramokgonami)   i like researching about things you see, so our study teacher used to tell us about things so the students used to tell him that he is lying so he told them to come and research (here at the library). when he gives assignments, i come to research you know. he even told us about these trucks that they use at the mines you know and he told us to come and research because we did not agree with him. (participant from ramokgonami)   on the assignments, the course that i do is obviously on the net, i just type www.cip.com and they just appear, study resources and so on, so i just click where i want to go. i can access the exam papers and i can even get my assessment and the results. even the answers for the exams just appear. (participant from gaborone)   the above quotations show that users benefited from the pavs by searching and obtaining the information they needed. this activity adds to the positive impact of pav facilities in the education of the users. the lack of books in school was mitigated by the presence of the pav icts since students had more access to more sources using the internet. participants who benefited from the icts in terms of access to sources said:   yes, i would say they are useful … if you don’t have enough books at school you are able to research …, you can search the internet, you are able to search for the books and read and some have free tutorials in the net, you are able to research… because in the books you cannot access everything but here you can go further. (participant from gaborone)   based on the quotations, users benefit from pav icts in three ways; they have access to facilities, information online to supplement limited information in textbooks, as well as plenty of information for doing assignments and research in general.   scholarship activities online   activities performed in the pavs included searching for scholarship opportunities in the library. it is clear from the data that they perceive the library icts to be very useful in terms of finding institutions, programs, and funding opportunities for their tertiary education. the following extracts serve as illustration:   i opted to download the form of the university and i applied using the form that i downloaded   . . . . i applied for different schools, some offered me a place there but i didn’t go…i continued to apply, downloading stuff to use then posting it back to them. last year … i applied again for the local institution, the institution offered me a place and that’s where i have been studying. (participant from ramokgonami)   right now i just want to get back to school so i’m looking for a nice course to do like in the university of botswana and maybe a combination of scholarship. (participant from jwaneng)   i have come here searching for scholarships. (participant from kasane).   one participant from ramokgonami described her use of the library icts in terms of furthering her education:   i started with the idea of nursing so i searched about different levels of nursing from different schools. and also schools by state and i started to narrow down to what i was looking for in a school. then i started looking at specific schools and their programs…um from there i was able to select the schools i want to apply to. i was also able to email some of the advisors at the schools on programs i want to do. (participant from ramokgonami)   informal and formal communication   communication is regarded as a social impact in this study because the users reported new methods of communication that would not be possible without access to the internet. participants said they use pav icts to communicate with friends via e-mail and facebook and this can be classified as personal or informal communication. they also reported that they conducted interviews on skype and retrieved bank statements, both of which can be regarded as formal communication.   a user who spoke in support of communication benefits said, “i would come here because i don’t have internet at home…so i come here every day” (participant from gaborone). such quotations attest to formal and informal communication use: “usually i communicate with friends outside the country and i email the assignments that i have been given at school” (participant from ramokgonami). it can be summarized that the internet available at pavs has had a positive impact on the communication of users.   impact of pav icts on employment and income generation   employment and income generation strategies are regarded as economic benefits in this study. these can be categorized as those activities which are likely to generate income through jobs or business, and those activities which save money and the time of the users.   time and money saving activities   the positive economic impact for participants was apparent in the availability of free services. examples include using printers and skype at the pav. printers could be used for printing curriculum vitae at no cost for the user. this constitutes the first economic benefit of pav icts. this was emphasized by the users when they said: “it is available and it is free …. in here it is free and you can stay as long as you want” (participant from gaborone). another user, in support of the above statement, said: “also imagine how much it will cost to go to south africa for an interview, it will cost more than 2,000 and yet an interview on skype here is free” (participant from gaborone). in the above quotation, the user of the pav is benefiting twice in that he gets an opportunity to do a job interview on skype (communication) and saves travelling money.   another form of saving was a comparison between the traditional way of writing letters and the modern use of the internet. as one participant put it, “also this saves us money. imagine writing a letter for a job on paper and posting it for p8.00 each and if i use internet café i would pay p10.00 and this one is totally free. i save both time and money” (participant from gaborone). not only is the user saving money, they are also benefiting from using modern facilities for communication. one user showed the usefulness of the facilities by saying: “it is convenient, you save a lot of time and there is a lot more information available on the internet. it has actually changed my way of thinking” (participant from gaborone).   online job related activities   for the unemployed, the library and all its facilities form an important part of finding and applying for jobs. a list of employment-seeking strategies that were identified in the study include: using the internet to communicate to prospective employers or schools, using e-mail for sending job application letters, using skype for job interviews, and using printers for cvs. the following quotation illustrates the positive impact of pav icts on the employment strategies of the participants:   these facilities help me in such a way that …when you are here you can google, you can search, actually let me say search for any job opportunities, so you do not need to go through a newspaper to find job openings. some companies which have websites … post the jobs on their stuff. … i search, i find jobs, and i apply. i don’t have to use the post anymore, i email from here, i do my cv here then i email it here. i am trying to get a job so they are really helping me a lot. (participant from ramokgonami)   the other type of employment benefit derived from the pavs is work related. a user who was employed by a company in another country used the pav to do his job. the user reported that: “internet works as a link between the company i work for here and its headquarters in south africa. i work in the company itself to acquire information on the market and for research purposes” (participant from gaborone). in the quotations above, it is worth noting the shift from using traditional media, such as newspapers, to the internet. this confirms that the new media has an impact on society and individuals’ communication and job seeking methods. all the excerpts above support the fact that participants’ benefit from the library in relation to the employment seeking strategies as established in the data.   discussion   the results of the study revealed two major areas of positive impact are education and income generation strategies occasioned by the availability of icts in libraries or pavs. educational impacts can be subdivided into the development of ict skills, access to online information, scholarship opportunities, and communication opportunities. income generation impacts fall into two categories; time and money saving, and job related activities. the socio-economic barriers to ict use in developing countries and the slf thinking highlight the social and economic parameters (duff, 2011; heeks et al., 2010; parkinson & ramirez, 2006). to that end, the discussion of results is divided into educational, economic, and social benefits. this is in line with the purpose of the study which sought to determine the impacts of pav icts in the lives of users.   educational benefits   development of ict skills   the results show that there was positive impact in terms of ict skills imparted; however, this was not the only positive impact. the second benefit from pav icts was registered by young and school going users who reported that the pav icts were beneficial for access to educational information and resources. access had proved to be elusive to some distance education students in botswana before pav icts (oladokun & aina, 2011). this study has proven that one of the issues in the reported digital divide, which was explained in part as lack of access to the plethora of digital information (duff, 2011; gyamfi, 2005, rezaian, 2007), is finally beginning to be resolved.   the other finding was that school going users acquired basic skills at school while non-school goers relied on the public library staff. therefore, the ict skills acquisition phenomenon reveals two types of users in the library with distinctly different needs based on their environments and socialization. also, the school goers reported that they had already acquired basic skills and did not need the skills imparted in the library. in slf thinking (parkinson & ramirez, 2006), non-school goers had a context in which some basic assets were missing such as the social and physical assets in the form of ict education and facilities, while the school goers had exposure to them. the assertion by duff (2011) that the digital divide leads to socio-economic stratification, is applicable here. the results revealed two types of users in the same facility and this has implications for the services and needs of the two user groups. the remnants of the digital divide in society are visible where non-school going users are concerned (gyamfi, 2005).   access to resources   one of the variables in the slf is capital assets under which physical assets falls. the fact that users had access to physical assets like computers and the internet means that they were able to carve better livelihood strategies for themselves. literature on poverty in botswana (hillborn, 2012; lekoko & morolong, 2007) confirmed that lack of facilities and poverty were barriers to access. pav icts helps to curb these barriers to information technology.   formal and informal communication   formal and informal communication was reported by users from the four research sites. the available icts have therefore had a positive impact on the users in that respect as shown in the discussion above.   to summarize, it has been established that with regards to education, the icts located in the pav did have a positive impact in terms of imparting skills to older users. from this finding, it is possible that older users did not have the opportunity to learn ict skills because when they were younger there were limited computers and internet opportunities in the system (e.g. in schools, homes, workplaces).this was confirmed in the literature on the barriers of icts in sub-saharan africa (gyamfi, 2005; maswabi et.al.,2011). also, today younger users have other options in addition to public access, where the internet is accessible; examples include internet cafes and mobile phones (grand et al., 2010). however, the library does seem to be very significant for non-school going older users. this is regarded as a positive impact in the lives of the users because they gained some knowledge or education from the venue staff and pav icts.   it was also established that for younger and school going users, the opportunity to carry out educational activities and communication (both formal and informal) was a positive benefit. the facilities gave them educational opportunities free of charge and this constitutes economic benefits, while the mode of communication changed the traditional methods of doing business in society. the literature in sub-saharan africa has long decried the lack of computer skills by most users in africa (bose, 2004; lekoko & morolong, 2007; totolo, 2014), therefore, the evidence of the acquisition of computer skills among botswana library users is quite significant in this study.   the positive impacts emanating from the library in relation to education include ict skills imparted at the venue, availability of facilities for ict related educational activities and communication channels. it can be concluded that the availability of library icts is slowly being incorporated in the educational process. it seems that external motivations to use the facilities are increasing the impact. there is evidence that school assignments requiring the use of the library icts encourage adoption and use of computers. the well-known history of africa’s lack of adoption and use of icts (rice, 2003, totolo, 2011), lack of ict skills (bose, 2004; gyamfi, 2005), lack of ict policy (heeks, 2002), and ict access issues (grand et al., 2010; lekoko & morolong, 2007), is being challenged by the new developments which show positive ict impacts. the positive results have been reported in eight country reports (sey et al., 2013). the ict4d literature strongly supports the notion that these intangible benefits are very important for closing the digital divide (gomez & panther, 2012).   economic benefits   the study has established the economic benefits are positive impacts on employment and income generating strategies for the users of pavs. users reported that they used the icts to communicate with existing or potential employers (e-mail; internet; websites); searching for jobs; registering on employment websites; updating cvs; and communication between the unemployed in their job seeking endeavours.   it was quite clear that users did not spend money to access services from the pav. it has also been revealed that instances where participants use e-mail to send job applications, skype to conduct a job interview, the internet to perform job related activities, and the printer to get copies of curriculum vitas are evidence of positive impact derived from the available icts. these can be regarded as actual impacts of pavs on the users. using slf analogy, the income generating activities and outcomes are visible in the study (parkinson & ramirez, 2006).   the pav seems to have additional and more efficient ways for users to look for jobs compared to relying only on traditional media. the literature on botswana mentioned poverty (hillborn, 2012) and lack of ict facilities (lekoko & morolong, 2007; maswabi et al., 2011; totolo, 2014). however, the availability of icts is slowly changing the social, economic, and educational fabric of the botswana society, with access to technology. access to facilities is proving to be one of the key factors that contribute to potential usage of icts in the botswana libraries. the slf shows that facilities or physical capital is one of the catalysts in the provision of access which can lead to sustainable livelihoods and outcomes (parkinson & ramirez, 2006).   further research is needed to establish the magnitude of the economic impacts. this study analyzed the economic benefits of icts and established the existence of positive impacts, however, there is a need to juxtapose the impacts to the statistics on unemployment in botswana. the recent botswana census established that the average unemployment rate was estimated at 17.8 percent and 116,388 persons lived below a dollar a day in 2009/10. according to the botswana statistics the age group distribution indicates that the highest unemployment rate of 41.4 percent is among the 15-19 years age group followed by the 20-24 year age group at 34.0 percent (statistics botswana: botswana core welfare indicators (poverty) survey 2009/10, 2011). therefore, the positive impacts registered in this study need to be increased in order to improve poverty and unemployment. this is clearly a policy implementation issue as suggested in the literature (heeks et al., 2010).   social benefits   a trend amongst users in this study developed where preference seemed to be given to modern ways of conducting business online. users preferred communication on the internet such as skype, e-mail, and doing examinations online rather than traditional face to face methods. indeed society is changing to adopt ict related business and social activities because of exposure to ict facilities and this can be regarded as positive impact. literature on the digital divide discussed how those who did not have access were marginalized (duff; 2011; gyamfi, 2005; oladokun and aina, 2011), therefore access to icts in the pavs is beneficial.    the literature on the library system in africa and botswana before the advent of icts showed that the libraries were underutilized and under resourced (isaak, 2000; maswabi et. al., 2011; mutshewa et.al. 2010). although this study was not investigating the impacts of the library on users, but rather the impacts of the newly introduced icts, it is interesting to see the uptake of icts by the users in a library system that was not popular before. also, the use of web 2.0 tools such as skype is a breakthrough in the lives of pav users. web 2.0 tools are well known for extending access and enhancing education (al-aufi & fulton, 2014; badea, 2014), therefore its use in the pavs is evidence of positive impact. however, this study was done on a small scale using qualitative research and it cannot be generalized to the whole population. the study has achieved its objective of uncovering rich qualitative data about the users of pavs. however further research covering the whole country using both quantitative and qualitative data would complement this study since those results can be generalized to the entire population.   recommendations   the study revealed educational, social, and economic benefits of icts, however, the level of impact tended to border on basic computer skills. it is recommended that advanced ict skills such as website design be offered to users in order to stimulate income generation and improve livelihoods of both the unemployed and the school leavers through self-employment.   library personnel, as information mediaries, play a key role in the success and impact of the library ict facilities and services. the venue staff mostly catered to the needs of novice users. however, in view of the availability of computer skills in schools and the proliferation of hand held gadgets with internet access, libraries might cease to attract young users. it is recommended that staff be equipped with advanced ict skills so as to entice young users to the library and to offer them more opportunities for learning survival skills in a world dominated by icts. continued training is needed to ensure that staff have skills to train users and assist them as they grow more sophisticated in their ict and information competence. it is strongly recommended that the training-the-trainer program receives priority attention. in addition to advanced ict skills, it is recommended that the softer skills required for effectively training users, should also receive attention. finally, staff should be trained in entrepreneurship – this would equip them to assist users more effectively in utilizing the library icts in employment and income generating strategies.   ict policy failure has characterized most of sub saharan africa before, as discussed above. the positive impacts discussed above depend on the continuity of ict access once the donors stop the aid. it is recommended that governments should enact the right policies to ensure maximum benefit from the digital era.   conclusion   the present study sought to find out whether pav icts, available free of charge for users in botswana, had any impact on their livelihoods. the use of slf as a lens indicates that the investigation of the phenomena hinges on sustainable livelihoods geared towards poverty reduction. the study established three broad categories of positive impacts: educational, economic, and social impacts.   educational impacts include skill acquisition and access to educational materials and communication. although there is no doubt that the positive impacts of icts are likely to increase the education and income generation strategies of the user, there is a need to mention the level of the impact. this study has revealed that the skills users benefited from pavs were basic. if the skills are basic, they are useful for logging in and out, typing, writing e-mail etc., however, if users are to develop business ideas and generate income, there is a need to do more sophisticated tasks using icts. in this study, it appears that users functioned with the most basic skills; therefore the level of the ict skills impact is bound to be minimal albeit present. in addition, the acquisition of skills by non-school goers in pavs has shown that there are two types of users in the library, therefore they have different needs. using the slf methodology helps to explain the two distinct environments of the school goers and the non-school goers, and this should inform the tailoring of an information skills program in the library system.   in terms of economic benefits, the pav icts positively impacted both job seekers and those who were employed. those who were seeking jobs used the pav to do so and those who preferred to work online for their employers also benefited. this is a clear indication of an opportunity emanating from the use of pavs. according to the slf, economic impacts ideally, should lead to a sustainable livelihood and possible income generation. longitudinal studies can be helpful in determining whether in the long run, the economic benefits shown in this study will lead to a sustainable life and poverty reduction.   the other impacts revealed in this study were social benefits. it appears that the pav icts are beginning to transform library users in terms of communication and conducting educational activities. however, this study was qualitative in nature; therefore, more studies are needed to explain further the social revolution that seems to be taking place amongst the users of pav icts. the findings in this study are instrumental for showing the benefits of ict access in society but the magnitude of the change will manifest itself with more exposure to icts.   the above assertion has a bearing on policy formulation and implementation. earlier in this paper, the vision 2016 policy was discussed and was shown to have minimal impact in the botswana national library service, since none of the libraries were fully equipped with technology before donors came. therefore, the positive impacts realized here will continue if the botswana government continues to enact and implement the right policies. the almost dysfunctional library system reported in the literature also has a bearing on the uptake and use of information for generations to come. the study concentrated on the positive impacts of icts in the botswana society. further studies could concentrate on the negative impacts and suggest ways of overcoming such in order to advance the evident technology benefits as discussed in this paper. this study also dealt with the impact of icts, so future studies could analyze overall impact of all library services on the user’s livelihood strategies. further scrutiny of the botswana ict policy implementation context will enhance the present study.   references   albu, m., & scott, a. (2001). understanding livelihoods that involve micro-enterprise: markets and technological capabilities in the sl framework. retrieved from http://practicalaction.org/docs/advocacy/micro-enterprise-livelihoods.pdf   al-aufi, a. s., & fulton, c. (2014). use of social networking tools for informal scholarly communication in humanities and social sciences disciplines. procedia social and behavioral sciences, 147, 436 – 445. doi:10.1016/j.sbspro.2014.07.135   ambrozic, m. (2003). a few countries measure impact and outcomes – most would like to measure at least something. performance measurement and metrics, 4(2), 64-78. http://dx.doi.org/10.1108/14678040310486909   arun, s., heeks, r., & morgan, s. (2004). researching ict-based enterprise for woman in developing countries: a livelihoods perspective. manchester: idpm, university of manchester. retrieved from http://r4d.dfid.gov.uk/pdf/outputs/ict/r8352-livelihoodsresearch.pdf   babbie, e. (2001). the practice of social research. australia: wadsworth.   badea, m. (2014). social media and organizational communication. procedia social and behavioral sciences, 149, 70 – 75. doi:10.1016/j.sbspro.2014.08.192   baffour-awuah. m., & pilane, m. (2001). village reading rooms: book outreach in botswana school libraries. school libraries worldwide, 7(2), 65-71.   beetham, h., mcgill, l., & littlejohn, a. (2009). thriving in the 21st century: learning literacies for the digital age (llida project). retrieved from http://www.webarchive.org.uk/wayback/archive/20140615060659/http://www.jisc.ac.uk/media/documents/projects/llidareportjune2009.pdf      berube. l. (2011). do you web 2.0? public libraries and social networking. oxford: chandos publishing.   bose, k. (2004). computer training programme for primary school teachers in teacher training institutions of the southern region of botswana. research in post compulsory education, 9(3), 401-415. retrieved from eric database (ej939756)   brocklesby, m. a., & fisher, e. (2003).community development in sustainable livelihoods approaches an introduction. community development journal, 38(3), 185-198. retrieved from http://agrilifecdn.tamu.edu/essm675/files/2013/01/03_brocklesby-et.pdf   central statistics office. (2011). labour statistics 2010. gaborone: statistics botswana. retrieved from http://www.cso.gov.bw/templates/cso/file/file/2010%20labour%20statistics%20report(1).pdf   chambers, r., & conway, g. r. (1992). sustainable rural livelihoods: practical concepts for the 21st century. sussex, uk: institute for development studies (ids). retrieved from  http://opendocs.ids.ac.uk/opendocs/bitstream/handle/123456789/775/dp296.pdf?seq   chapman, r., slaymaker, t., & young, j. (2003). livelihoods approaches to information and communication in support of rural poverty elimination and food security. london: overseas development institute. retrieved from http://www.odi.org/resources/docs/208.pdf   churu, j. (2014). gates, youth ministry in e-library service deal: botswana. retrieved from http://www.biztechafrica.com/article/gates-youth-ministry-e-library-service-deal/9260/?country=botswana   duff, a. s. (2011). the rawls-tawney theorem and the digital divide in postindustrial society. journal of the american society for information science and technology, 62(3), p. 604–612.  doi: 10.1002/asi.21470   duncombe, r. (2006). using the livelihoods framework to analyze ict applications for poverty reduction through microenterprise. information technologies and international development, 3(3), 81-100.   gomez, r., & pather, s. (2012). ict evaluation: are we asking the right questions. electronic journal on information systems in developing countries, 50(5), 1-14. retrieved from http://www.ejisdc.org/ojs2/index.php/ejisdc/article/view/879   grand, b., totolo, a., mutshewa, a., sebina, p., zulu, s., & jorosi, b. (2010). botswana global libraries baseline study: sesigo project impact assessment. unpublished report. department of library and information studies, university of botswana.   gyamfi, a. (2005). closing the digital divide in sub-saharan africa: meeting the challenges of the information age. information development, 21(1), 22-30. doi: 10.1177/0266666905051910   head, a. j., (2012). learning curve: how college graduates solve information problems once they join the workplace. retrieved from http://dx.doi.org/10.2139/ssrn.2165031   head, a. j. and eisenberg, m. b. (2010). truth be told: how college students evaluate and use information in the digital age. project information literacy progress report. retrieved from http://projectinfolit.org/pdfs/pil_fall2010_survey_fullreport1.pdf      heeks, r. (2002). information systems and developing countries: failure, success, and local improvisations. the information society, 18(2), 101-112. doi:10.1080/01972240290075039   heeks, r., gao, p., & ospina, a. (2010). delivering coherent ict policies in developing countries (e-develoment briefing no.14). centre for development policy, university of manchester.   hillborn, e. (2012). botswana: a development-oriented gate-keeping state. african affairs, 111(442), 67-89. doi: 10.1093/afraf/adr070   isaak, a. (2000). public libraries in africa: a report and annotated bibliography. international network for the availability of scientific publications (inasp). retrieved from http://www.inasp.info/uploads/filer_public/2013/03/08/public_libraries_in_africa.pdf   jensen, l. (ed.). (2011). the millennium development goals report 2011. new york: united nations. retrieved from http://www.un.org/millenniumgoals/pdf/(2011_e)%20mdg%20report%202011_book%20lr.pdf   lekoko, r., & morolong, b. (2007). poverty reduction through community-compatible icts: examples from botswana and other african countries. in m. gasco-hernandez, f. equiza-lopez and m. acevedo-ruiz (eds.): information communication technologies and human development: opportunities and challenges (pp.116-137). london: idea group.   long term vision for botswana: towards prosperity for all. (1997). gaborone: presidential task group.   maitlamo, botswana’s national information and communications technology policy (2004). retrieved from http://www.bits.org.bw/downloads/maitlamo_national_ict_policy.pdf   maswabi, t., sethate, t., sebusang, s. e., & taolo, r. (2011). public libraries: pathways to making botswana an educated, informed nation. library review, 60(5), 409-420. http://dx.doi.org/10.1108/00242531111135272   mutshewa, a., grand, b., totolo, a., zulu, s., sebina, p., & jorosi, b. (2010). information behaviours of non-users of libraries in botswana. african journal of library archives and information science, 20(1), 1-10. retrieved from http://ubrisa.ub.bw/xmlui/bitstream/handle/10311/1099/mutshewa_ajlais_2010.pdf?sequence=1   mutula, s. m., grand, b., zulu, s., & sebina, p. (2010). towards an information society in botswana: ict4d country report gaborone, botswana: sangonet. retrieved 8 july 2015 from http://www.ngopulse.org/article/towards-information-society-botswana   oladokun, o., & aina, l. (2011). access odl and the impact of digital divide on information in botswana. the international review of research in open and distributed learning, 12(6), 157-177. retrieved from  http://www.irrodl.org/index.php/irrodl/article/view/1053/1959   our africa: botswana, economy and industry. retrieved from http://www.ourafrica.org/botswana/economy-industry     papazafeiropoulou, a., & pouloudi, a. (2004). digital divide challenge: how stakeholder analysis can be used to formulate effective it diffusion policies. in g. doukidis, n. mylonopoulos, & n. pouloudi (eds.). social and economic transformation in the digital era. (pp. 47-55). hershey, pa: idea group publishing.   parkinson, s., & ramirez, r. (2006). using a sustainable livelihoods approach to assessing the impact of icts in development. journal of community informatics, 2(3), 1-12. retrieved from http://ci-journal.net/index.php/ciej/article/viewarticle/310/269   population & housing census. retrieved from central statistics office website: http://www.cso.gov.bw/index.php/sector-statistics/population-housing-census   radijeng, k. (2009). open access resources. in p. jain & r. nfila, p. (eds.). botswana library association (bla) 2nd national conference: information services and community development (pp.42-47). gaborone, botswana: botswana library association.   rezaian, b. (2007). integrating icts in african development: challenges and opportunities in sub-saharan africa. in m. gasco-hernandez, f. equiza-lopez and m. acevedo-ruiz (eds): information communication technologies and human development: opportunities and challenges. (pp.23-56). london: idea group.   rice, m. (2003). information and communication technologies and the global digital divide: technology transfer, development, and least developing countries. comparative technology transfer and society, 1(1), 72-88. doi: 10.1353/ctt.2003.0009   rosenberg, d. & international african institute (1997). university libraries in africa: a review of their current state and future potential. vol. 1, summary. london: international african institute.   sesigo botswana public e-libraries: project implementation progress. retieved from http://www.sesigo.org.bw/index.php?id=49   sey, a., coward, c., bar, f., scaidas, g., rothschild, c., & koepke, l. (2013). connecting people for development: why public access icts matter. seattle: technology & social change group, university of washington information school.   sin, s. j., & kyung-sun, k. (2008). use and non-use of public libraries in the information age: a logistic regression analysis of household characteristics and library services variables. library and information science research, 30(3), 207-215. doi:10.1016/j.lisr.2007.11.008   soriano, c. r. r. (2007). exploring ict and rural poverty link: community telecenters and rural livelihoods in wu'an, china. electronic journal of information systems in developing countries, 32(1), 1-15. retrieved from http://www.ejisdc.org/ojs2.../index.php/ejisdc/article/viewfile/462/230   statistics botswana: botswana core welfare indicators (poverty) survey 2009/10 (2011). retrieved from http://www.cso.gov.bw/templates/cso/file/file/bcwis%20_poverty_%20survey%20statsbrief%20nov%202011..pdf   sturges, p. (2001). the poverty of librarianship: an historical critique of public librarianship in anglophone africa. libri, 51, 38-48. doi: 10.1515/libr.2001.38   telecommunications and postal services. retrieved from http://www.gov.bw/en/ministries--authorities/ministries/ministry-of transport and communications/departments/telecommunications/   totolo, a. (2011). adoption and use of computer technology among school principals in botswana secondary schools. the international information & library review, 43(2), 70-78. doi:0.1080/10572317.2011.10762882   totolo, a. (2014). national information and communication technology policy and digital era school librarianship in botswana. the international information & library review, 46(1-2), 84-92. doi:10.1080/10572317.2014.932168   totolo, a., & renken, j. c. (2012). the impact of public access venue (pav) information and communication technologies (icts) available at libraries on sustainable livelihood strategies and outcomes in botswana. global impact study research report series. seattle: technology & social change group, university of washington information school.     evidence summary   academic librarians would benefit from instruction on conducting research   a review of: kennedy, m. r., & brancolini, k. r. (2012). academic librarian research: a survey of attitudes, involvement, and perceived capabilities. college & research libraries, 73(5), 431-448. retrieved 31 may 2013 from http://crl.acrl.org/content/73/5/431.full.pdf+html   reviewed by: annie m. hughes reference librarian wilson dental library, university of southern california los angeles, california, united states of america email: amhughes@usc.edu   received: 27 feb. 2013   accepted: 25 may 2013      2013 hughes. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objectives – to survey and ascertain the level of confidence academic librarians demonstrate with regard to performing and consuming research, as well as to gather information in order to plan a curriculum that would offer professional continuing education programming for librarians interested in enhancing their research skills.   design – web-based survey of academic librarians on their level of confidence with regard to performing and consuming research.   setting – various email lists with academic librarians as subscribers.   subjects – 918 self-selected academic librarians who subscribe to email lists.   methods – the authors chose to gather a convenience sample of academic librarians by sending a survey via various email lists. a link to an informed consent notice was sent via the request for participation and then linked to the survey. the survey consisted of 19 questions and gathered information regarding four areas: current research practices, self-evaluation of confidence in research practice, research courses in which the participants participated either during their library and information studies (lis) programs or through other means, and demographic information and information related to support provided by the librarians’ home institutions. the authors adapted their survey from other previously published surveys, and it was pre-tested for its effectiveness and reviewed by the institutional review board. question 10 included a confidence scale from 1-5 with 1 being “not at all confident” and 5 being “very confident.” the confidence scale was used to capture respondents’ self-perceptions of their research design expertise. various statistical tests were performed.   main results – the authors received 918 responses to their call for participation, with 809 completing the full survey; incomplete responses were not excluded. results indicate that the vast majority of academic librarians are focused on staying current with regard to the literature, and there is the assumption or expectation that they will do this as part of their job duties. while nearly 78% of librarians view the table of contents of journals, fewer librarians regularly read the full content, as time is a determining factor with regard to consumption of the literature. the results show that 62% of academic librarians have undertaken some type of research since graduating with an lis degree, but they do not necessarily publish or disseminate their results; only 77% of those who had done research had presented at conferences, published in a journal, or presented at their institution in an informal setting.   a confidence scale was used to determine the level of confidence in performing certain research tasks, and while respondents recorded the highest level of confidence in conducting a literature review, the lowest levels of confidence were noted for the ability to design a project to test their questions and to analyze research data. respondents were fairly confident on average with regard to turning their topics into questions, gathering data, reporting results in a written format, reporting results verbally, and identifying appropriate places to disseminate results. the authors chose to analyze two variables with regard to their question on confidence (average confidence and conduct research), as they predicted that whether librarians conduct research after completing their lis studies was dependent upon their confidence in performing evidence-based research. after running a logistic regression analysis in spss, the authors found that confidence may be a predictor for research performance.   with regard to training in research methods, only 26% of respondents felt that their lis training provided them with the background to perform research tasks. the authors analyzed the data in this instance and found that there is no statistically significant relationship between the belief that lis training prepared them and the likelihood of performing research.   the demographic section of the survey provided the authors with information regarding research support from institutions, and asked librarians if they are tenure track/promotion, promotion only, or not eligible for tenure and promotion. the results showed that 45% were eligible for tenure, 28% achieved tenure, 48% had been through the tenure process, and 40% had other degrees in addition to their lis qualification.   conclusion – by surveying a sample of academic librarians, the authors were hoping to gather information about their confidence level, training level, and current practices with regard to research. the data was collected as a way to inform the authors as to how they could best design a curriculum for continuing education in research practice. the survey results show that academic librarians are confident with regard to consumption of the literature and developing research questions, but less confident with regard to gathering and analyzing data.   the proposed curriculum would provide training on conducting research, and the authors’ eventual goal is to offer an “institute for research design in librarianship” that would help librarians, especially those who are required to do research for tenure and promotion purposes, to perform the more difficult aspects of research such as design and analysis. librarians would complete the course and be able to return to their home institutions with the capability to perform all steps in the research process.     commentary   the authors focus on a very important issue within the academic librarian community: some librarians are required to perform research tasks, but may not feel confident in performing these tasks. while the article uses a survey instrument to gather general data about the population, the main goal was to gather evidence to support a curriculum that would effectively teach research design and data analysis. the authors’ proposal to offer an institute for research design in librarianship, from the evidence provided in the article, looks to be a much needed form of continuing education, as many librarians’ jobs depend upon producing and disseminating quality research. while there is evidence that some librarians receive training in research methods during their lis program, it appears that the courses do not always go into sufficient detail regarding data gathering and analysis. a course focused mainly on conducting research, as opposed to just consuming and understanding research, could be one way of closing the gap with regard to low confidence levels in the academic librarian population. the authors should continue in this line of inquiry and gather further evidence to support their idea.    the authors are very forthright about their research limitations, and state that they used a convenience sample, gave two weeks for response time, and only sent out one call to the email lists. they also felt that further study would need to be done with regard to the idea of self-efficacy as a predictor for completion of research tasks. the authors mention that confidence as a predictor may not show the whole picture, and more would need to be done to get a better perspective. however, as stated several times throughout the article, the survey was only meant to generally inform and gather evidence in order to develop a curriculum in research design.    future research possibilities were also identified by the authors. research culture is one area of interest to them, as support from an academic librarian’s institution is important to fostering production and dissemination of research. an institution’s expectations for staff to publish or disseminate may enhance the research culture and therefore provide support through resources such as time and money to attend training workshops and conferences. one other possible area of interest noted by authors is to study academic librarians who are successful researchers and who have published their research, in order to explore their steps to success. the authors believe that a study of high level researchers in the academic library world may enable a formula for effective research practice.    the article is only an introduction to what could potentially be derived from a more substantial analysis of academic librarian research behaviors. the authors do an excellent job of utilizing the evidence they gathered in an effective way and do not make assumptions based on the data. their goal is to inform themselves so that they may develop an effective strategy for continuing education for academic librarians. they clearly state their purpose, their goals and their limitations, and hopefully will continue to develop the institute for research design in librarianship.   evidence summary   meta-synthesis of the research on information seeking behaviour of graduate students highlights different library resource needs across disciplines and cultures   a review of: catalano, a. j. (2013). patterns of graduate students' information seeking behaviour: a meta-synthesis of the literature. journal of documentation, 69(2), 243-274. doi:10.1108/00220411311300066   reviewed by: joanne l. jordan research information manager arthritis research uk primary care centre, keele university keele, staffordshire, united kingdom email: j.jordan@keele.ac.uk   received: 01 jul. 2013    accepted: 11 oct. 2013      2013 jordan. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to synthesize research on the information seeking behaviour of graduate students.   design – meta-synthesis of quantitative and qualitative research.   setting – higher education institutions mainly in the u.s. and canada, but including studies from other countries in europe, asia, and africa.   subjects – graduate students (master’s and doctoral level).   methods – the library information science and technology abstracts (lista) database was searched from 1997 to 2012. references of retrieved studies were reviewed and a google search carried out. studies were critically appraised using the evidence based librarianship (ebl) critical appraisal checklist by glynn (2006). the author extracted information from the included studies and took notes on the studies’ findings. notes were then grouped into themes according to relevant research questions that emerged. a critical interpretive synthesis approach used qualitative and quantitative information from the synthesis to answer these research questions. small user surveys were summarized in the tables but not included in the synthesis.   main results – the review included 48 studies. most studies were rated as having good study design and results, but many were thought to be weak when it came to their sampling and data collection techniques.   students often initially look on the internet for information. many acknowledged that this information may be unreliable and turn to sources recommended by their advisors. increasingly library resources are accessed remotely, rather than print versions. however, knowledge of library web resources and services is not always good, with many students using internet search engines to find information.   it is suggested that accessibility of resources in different disciplines and familiarity with technology drives information behaviour. it is not always feasible for all sources of information needed in different subjects to be made readily available electronically. professors, faculty members, and advisors were consulted most often by students, however this varied between disciplines and institutions. librarians who demonstrated and promoted their expertise to academic departments were more highly valued by students.   students used reference lists of articles to find other relevant material (citation chasing). students were more concerned about the speed of accessing material rather than the quality or reliability of the content. some students were put off by seemingly complex library systems and tools. boolean operators and advanced search strategies were rarely used and if they were used, it tended to be by students with more computer expertise.   international students may not be as aware of the library services that are available to them. differences in culture and language can affect whether a student feels comfortable asking for help with library resources.   conclusion – different types of students, such as master’s and doctoral level students or those from different disciplines, access different types of resources in different ways. graduate students may benefit from training offered in a variety of different formats to address these different needs. other people are important in helping students begin their research and therefore institutions should ensure those advising students are aware of information services and training available. it is suggested that further research should be done looking into cultural differences in information behaviours. it is also recommended that researchers should increase their use of standardized, validated questionnaires to improve consistent measurement of information behaviour.     commentary   critical interpretive synthesis (cis) was adapted from meta-ethnography by dixon-woods et al. specifically to synthesize a large number of studies that are a mix of qualitative and quantitative research (2006). this approach differs from conventional systematic reviews of quantitative studies, in that it is more dynamic, iterative, and responsive and is very much based on qualitative research methods such as grounded theory. therefore, experience in qualitative research methods is an advantage when undertaking this type of review. this type of synthesis does not follow a step-by-step predefined method and, as much of the process is driven by the author-generated theory, it is often difficult to report in a clearly reproducible way. this seems a suitable method to use in this review, because a variety of different quantitative and qualitative studies are included. it is useful to have examples of this method being used in the library and information science literature for others to refer to.   catalano begins her review with a broad search for research studies on the information-seeking behaviour of graduate students in one database, the library and information science database lista. in this type of review less formal search strategies are often used, with one article leading to another until no new ideas emerge and saturation occurs. this is different to the structured comprehensive literature search conducted in a quantitative systematic review, such as a cochrane review. catalano’s choice of a library-based resource influenced the type of studies that were retrieved in this review. catalano explicitly states, however, that she is taking a service provision approach and intends to look at the literature from this perspective. as with primary qualitative research, this is a snap-shot taken from a particular point of view and not a complete summary of all the research on this topic.   similarly, formal critical appraisal is not always thought useful in qualitative or mixed methods (including both quantitative and qualitative studies) reviews. in the original cis review by dixon-woods et al. the methodological quality was assessed and only studies that were fatally flawed were excluded on this basis (2006). the assessment of quality forms the “critical” part of cis along with the relevance of the studies in order to form theories. catalano has used a critical appraisal tool specifically designed for information and library studies and reported studies as valid (>75%) and not valid (<75%) in the summary table and this is commented on in the methods section of the article. it would be helpful to have information on each section of the critical appraisal for each study to enable readers to judge individual sources of bias across the included studies. one source of bias reported by catalano was that most studies were conducted in convenience samples. this means that the study participants may not be representative (however large the sample) and that statistical inferences in these studies are invalid, as these are based on probabilities from the larger population. catalano goes on to report inferential statistics from convenience samples in the synthesis without comment on the error in these tests. it is unclear how the critical appraisal was incorporated in the synthesis. more details could help other researchers conduct similar reviews in the future.   the chosen method of synthesis reflects the voice of the author, possibly with other authors bringing out different themes or coming to different conclusions from the included studies. a concern with this review by catalano is that she seems to be the only reviewer here with the exception of the critical appraisal of her study that was carried out by someone else. reviewing in a team can bring a variety of perspectives and, while reaching consensus can sometimes be problematic, this can result in richer and more robust conclusion.   this review has highlighted some important implications for library practice.   it is worrying that students find speed of accessing research resources more important than quality. encouraging internet use in itself is not a bad thing, but students should learn early on how and where to find information and how to assess its relevance and reliability. once these skills become automatic, then the process of finding the most appropriate and trustworthy information is much quicker. literature searching and critical appraisal skills are highly valuable and can be used on the internet, as well as more specialist research databases, throughout a person’s life. it is difficult to change this behaviour at a graduate/postgraduate level when bad habits have already set in. perhaps this could be tackled by introducing good searching techniques earlier in undergraduate programs or even in schools when children first go online. in the meantime, higher education institutions must make learning searching and critical appraisal skills a priority and ensure that library staff are able to teach this effectively.   as more students are using the internet as a research tool, libraries need to ensure that the library website is the preferred starting point for any research project by highlighting appropriate and reliable sources and useful research tools. more effort is needed to engage and teach students using the library’s website and other tools, such as social media. libraries should also find ways to help students from different cultures (e.g., providing some resources in different languages). libraries have to respond to the needs of increasing numbers of remote users and find innovative ways to reach students that do not often physically visit library buildings. the resources and services provided by university libraries should be promoted more widely to academic staff and students and instructions on their use embedded into research training.   many academic libraries have begun to address several of these issues and they have an excellent range of resources for students from different disciplines. with the increased emphasis on evidence based librarianship, this good practice must be evaluated and shared with other information professionals.     references   dixon-woods, m., cavers, d., agarwal, s., annandale, e., arthur, a., harvey, j., hsu, r., katbamna, s., olsen, r., smith, l., riley, r., & sutton, a.j. (2006). conducting a critical interpretive synthesis of the literature on access to healthcare by vulnerable groups. bmc medical research methodology, 6, 35. doi:10.1186/1471-2288-6-35   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-99. doi:10.1108/07378830610692154   article   applying kolb’s learning theory to library instruction: an observational study   chau ha librarian saskatchewan polytechnic library saskatoon, saskatchewan, canada email: chau.ha@saskpolytech.ca   nina verishagen librarian saskatchewan polytechnic library saskatoon, saskatchewan, canada email: nina.verishagen@saskpolytech.ca   received: 10 june 2015   accepted: 25 aug. 2015      2015 ha and verishagen. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – this article answers the following questions: does applying kolb’s learning theory to library instruction enhance student engagement and will it improve librarian teaching practices?   methods – this observational study analyzed four forms of qualitative data to examine the learning experience of first year nursing students and the teaching experience of two faculty librarians. the four forms of data collected were:  (1) post-class qualitative feedback to assess the students’ engagement; (2) library instructors’ shared teaching observations; (3) librarian peer feedback after observing each other teach; and (4) feedback from an instructional facilitator on the individual librarian’s teaching skills. two distinct lesson plans were developed: lesson plan one was the first attempt at incorporating kolb’s theory into practice and lesson plan two was a refinement of lesson plan one. teaching strategies were altered from one lesson plan to the next based on the instructional facilitator’s feedback. the role of the instructional facilitator was to guide the professional development of new instructors by providing them with information and feedback on their teaching skills.   results – there were perceived improvements in student engagement and teaching practice from lesson plan one to two. although the students’ reported experience remained similar from one to the next, both the librarians and instructional facilitator felt the students were more engaged and the environment seemed more collaborative when following lesson plan two. with the second lesson plan, librarian instructors experienced a positive transformation as teachers, becoming facilitators of learning rather than lecturers.   conclusion – incorporating kolb’s theory into instructional practice resulted in librarian instructors perceiving a positive effect on both instruction and on student engagement in the teaching-learning process. introduction   at our polytechnic institution, librarians are often called upon to teach learners from a variety of programs, from trades to health sciences, to business and technology programs. like the majority of teaching faculty at the institution, librarians do not have a diploma or degree in adult education. in an attempt to address this education deficiency, our institution requires instructors in all programs and services, including library services, complete the faculty certification program (fcp), an adult education teaching certificate, in order to retain their positions. the courses required to complete the certificate include adult learning theories, instructional strategies, adult development, technology in teaching, curriculum design, evaluation of learning, and leadership.  fcp is designed to provide new instructors with the basic knowledge and skills needed to teach adult learners.  the challenge that our polytechnic librarians have, that other teaching faculty do not, is that we often teach single sessions to students in a variety of programs. this means librarians lack the opportunity to get to know students’ strengths, challenges, and learning preferences over time. our participation in the fcp program prompted us to ask the question: would applying a learning theory to an information literacy workshop increase student engagement, given that librarians often teach single sessions to a variety of student groups?   in the fall of 2014, as liaison librarians for a baccalaureate nursing program, we (the authors of this article) were asked to provide four three-hour workshops on database searching and american psychological association (apa) writing style to 150 first-year nursing students. previously, we had both taught the workshop independently of one another but the students who met with us to ask follow-up questions indicated inconsistencies in their understanding resulting from two different instructors teaching the workshops. so we embarked on this project in order to bridge this gap and to apply what we learned in fcp to our instructional practice. our two main objectives were: 1) to improve student engagement with information literacy skills instruction and, 2) to grow as professionals by perfecting our teaching skills. this paper describes the process of applying kolb’s learning theory to practice and our reflection of that process towards achieving student engagement and becoming better instructors.   we chose kolb’s theory because it postulates that experience is a critical aspect of the learning process, which aligns well with library instruction as it often involves hands-on experience in order to make sense of learning (kolb, 2014). according to kolb’s theory (2014), teaching to various learning styles and facilitating the learner’s progression through the learning cycle is necessary in order to create new knowledge. because this theory highlights the importance of experimentation, reflection, and abstract conceptualization, it is suited to information literacy instruction; research skills and information literacy are more than just imitating keystrokes, they require creativity and critical thinking.   literature review   a literature search was conducted to find publications on the topic of how to incorporate kolb’s learning theory in library instruction. although librarians are applying learning theory to instructional practice, using approaches such as tiered instructional programs (bowles-terry, 2012), adult learning theory (lange, canuel, & fitzgibbons, 2011) and evaluation methodologies (schilling & applegate, 2012), we were unable to find any specific examples of the application of kolb’s theory. woods (2012) provides a list of suggestions on how to consult kolb’s cycle of learning when planning information literacy sessions by emphasizing the use of a variety of teaching strategies to meet the preferences of all learners. other than woods’ suggestions for how to incorporate kolb’s theory we were unable to find literature on librarians actually applying kolb’s theory to their instruction. one reason for this may be that as librarians generally conduct one-shot information literacy sessions in a wide array of programs, the varying subject matter and timeframes make the application of kolb’s theory difficult.  other teaching faculty see students daily or weekly and therefore have the ability to get to know the students over time. these faculty have time to build on previously-taught concepts, assess learning, and adjust their teaching strategies and materials as needed, making it easier to apply adult learning theory to instruction.   while little research exists on using kolb’s theory to guide library instruction, its use in the fields of adult education, business, social work, and nursing has been well documented. kolb’s learning theory includes learning styles and his cycle of learning (figure 1). since its conception in 1984, academics and practitioners in the field of education have analyzed and implemented kolb’s theory into their practice. even today, kolb continues to inform instructional design (bergsteiner, avery, & neumann, 2010; lisko & o’dell, 2010; d’amore, james, & mitchell, 2012; cox, clutter, sergakis, & harris, 2013).     figure 1 kolb’s experiential learning theory. adapted from experiential learning: experience as the source of learning and development, by d. a. kolb, 2014, upper saddle river: pearson. copyright 2014.     according to kolb & kolb (2005), there are four types of learning styles that instructors may encounter in every classroom:   ·         diverging style learners are able to view experiences from multiple perspectives, are creative, open minded, interested in people, imaginative, emotional, open to feedback, are able to gather information, have broad interests, and enjoy group work. these learners enjoy concrete experiences and reflective activities. ·         assimilating style learners are logical and are able to understand a wide range of information. they are less interested in people and are more interested in ideas, concepts, and theory. these learners may prefer lectures, reading, exploring models, and having time for abstract conceptualization and reflection. ·         converging style learners can put theory into practice and solve problems. these learners may prefer technical tasks and problem solving to social or interpersonal experiences. they learn best through abstract conceptualization and active experimentation. ·         accommodating style learners learn from active experimentation and concrete experiences. these learners rely on others for information and group work to achieve their goals (kolb & kolb, 2005, pp.196-197).   for learning and knowledge creation to occur, kolb & kolb (2005) contend that instructors must create a learning space that is welcoming and respectful of past experiences while meeting the learner’s current needs. they must also provide a space for conversational learning, experimenting and reflecting, which encourages intrinsic motivation and allows learners to take charge of their own learning. in kolb’s learning cycle, learners move through: (1) concrete experiences, using past and present experiential learning to inform new learning; (2) reflective observation, which leads to (3) abstract conceptualization, which is followed by (4) active experimentation (kolb, 2014). new knowledge is only created through active experimentation and reflection (lisko & o’dell, 2010).   determining which learning style is characteristic of students in a particular discipline is difficult. d’amore et al. (2012) explored the learning styles of first-year undergraduate nursing and midwifery students at an australian university, and found that the majority of the students surveyed were “divergers” (p. 507). although they were able to identify a dominant learning style, the authors conceded that as people move through various stages of growth and development, they become capable of drawing from all four learning styles.  d’amore et al. (2012) therefore concluded that students should “not rely solely on one style” (p. 514).  cox et al. (2013) identified the learning styles of senior students in four undergraduate health programs and found a variety of learning styles in both the classroom and clinical setting and that no predominant learning style emerged. the evidence therefore indicates that it is important for educators to use a variety of teaching strategies in order to engage learners with different learning preferences. accordingly, creating a lesson plan that piques the interest of all four learning styles is more important than identifying what type of learner a student might be.   there is a connection between the skills built through experiential learning and the skills required to become information literate. in their study, devasagayam, johns-masten, and mccollum (2012) found that requiring students to participate in experiential exercises that require application to reinforce learning and are strengthened through repetition, is an effective way to teach information literacy because it encourages critical thinking via problem solving. clem, mennicke, and beasley (2014) describe experiential learning as something involving an experience that engages the physical body “in an effort to holistically enhance the process of learning” (p. 491). they have found that students learn better when the teaching approach is student-centered, when students can take control over their learning, and when the lesson is relevant to them. lisko and o’dell (2010) believe that active experimentation and reflection are essential to transform learning into knowledge creation.   from the current literature on kolb’s theory being applied to nursing instruction, it can be summarized that nursing students have a variety of learning styles, that their styles can change over time and according to the learning environment (whether classroom or clinical), and that nursing students prefer a variety of experiential learning strategies. halcomb and peters (2009) changed the curriculum for an undergraduate nursing course to incorporate more reflective and active learning. they surveyed their students at the end of the course and found that there was a positive response to the variety of interactive teaching strategies introduced (halcomb & peters, 2009). lisko and o’dell (2010) changed a medical-surgical nursing curriculum to meet the varied learning styles of their students by incorporating more scenario and experiential based teaching. overall, their student and faculty feedback was positive and the authors felt that their methods enabled the “development of nursing students’ critical thinking abilities” (lisko and o’dell, p. 108). crookes, crookes, and walsh (2013) found that nursing students preferred various experiential and reflective teaching techniques: technology tools, simulations, gaming, art, problem-based learning, and narrative activities for reflection and for linking theory to practice. based on a review of the literature, the application of kolb’s theory is well-suited to the nursing classroom.   methods   this study answered the following questions: does using kolb’s theory in library instruction enhance student engagement and does it improve teaching practice? four forms of qualitative feedback were collected to determine how kolb’s theory contributed to the students’ engagement and how it improved our instruction: (1) we assessed the learning experience using a student post-class qualitative feedback form; (2) we reflected on our teaching; (3) we provided feedback to each other after observing one another teach, and; (4) an instructional facilitator, whose job description is to guide and support the growth of instructional faculty, observed and provided feedback to us on our teaching skills.   student feedback   the student post-class qualitative feedback form asked students to respond to the following questions:   1.       how did you feel about your [research and apa skills] before the session? 2.       how did you feel about your [research and apa skills] after the session? 3.       what do you attribute the cause of the change in how you feel if there was a change? 4.       what was the most impactful thing you learned? 5.       what teaching activity was least useful and why?   the feedback provided us with guidance and resulted in changes to the lesson plans as well as modifications to our teaching strategies.   librarian reflection and peer feedback   in an effort to reflect and share our teaching experiences, we met to debrief at the end of each lesson. we discussed the following questions:   1.       how did you feel about your teaching experience? 2.       what were the students’ reactions to your teaching strategies? 3.       what changes to the content or teaching strategies would you make to improve student engagement?   these questions encouraged us to reflect, to gain insight, and to become aware of the impact of our teaching practices on the learners. in the debriefing sessions after implementing the revised, second lesson plan from which we team-taught, we provided verbal feedback regarding one another’s performance. this feedback led to further reflection and discussion.    faculty facilitator feedback   the faculty facilitator’s knowledge, experience and expertise in guiding the professional development of instructors provided us with further insight into best teaching practices. she observed our teaching, made notes, and provided verbal feedback to us after each teaching session. her suggestions led to further discussion and changes to our lesson plan and teaching strategies.   all four forms of qualitative feedback were transcribed in a microsoft word document, which we used to change our lesson plans, and to track our progress as teachers. our first attempt at incorporating kolb’s theory into our teaching practice was delivered to the first group of nursing students (lesson one). then, after processing feedback, a second lesson plan (lesson two) was created to more effectively incorporate kolb’s theory.   lesson one: first attempt at incorporating kolb’s theory   we met to create one common lesson plan: learning goals, learning content, teaching materials, and teaching activities. for this lesson, we incorporated kolb’s theory based on our interpretation. we chose our teaching strategies based on our teaching experience and our learning styles. our teaching strategies included asking pre-assessment questions about past experiences, a lecture, and a demonstration followed by individual activities. we stayed at the front of the room and came to the students who asked for help. our discussion questions focused on their understanding of what was lectured on or what was demonstrated. a post-assessment form was used to assess students’ experience and the instructors debriefed afterwards to discuss our teaching experience and our observation of students’ engagement.   we asked the faculty facilitator to observe our teaching sessions and give us feedback on our instructional methods. she observed that we had not incorporated kolb’s theory into our lesson plan effectively. she commented that although our lesson plan had elements of kolb’s theory, such as reflective and experiential activities, we were not teaching to all learning styles, nor did we facilitate moving the students through the learning cycle.  she felt that our lesson plan was more traditional than experiential. using her suggestions, our reflection of our practice, our feedback to each other, and the students’ feedback, we created a new lesson plan that incorporated more experiential learning and reflection as suggested by kolb’s theory.   lesson two: effective incorporation of kolb’s theory   kolb’s theory was incorporated into our teaching material, activities and strategies in the following ways:   ·         we taught this workshop as a team in order to learn from each other through reflective observation and discussion. ·         we facilitated the class activity, discussion and learning instead of lecturing from the front of the room. we engaged all learners by walking around the room as we talked and asked reflective questions of learners sitting at the front, middle and back of the classroom. ·         we facilitated a discussion on their past concrete experiences with research and apa style, giving them time to reflect before beginning questioning. ·         we facilitated discussions and provided time for a dialogue of student observations, ideas, and opinions on their new learning. ·         through a learning activity on database searching, we encouraged students to search using their usual methods and then to try a new approach to research before coming to a conclusion. we encouraged them to use their past experiences, and to observe and reflect, as well as actively experiment with new approaches to search. ·         to teach apa style, we used paired learning activities for peer-to-peer support and peer-to-peer learning, and we encouraged them to independently search for answers using a variety of resources.   understanding that we needed to facilitate their movement through the kolb’s cycle of learning, we used a variety of teaching strategies designed to appeal to different learning styles:   ·         for assimilating style learners, the lecture combined with discussion questions encouraged reflective observations. ·         for assimilating and diverging styles, a video and a demonstration, as well as classroom discussion and a reflective post-class survey, encouraged the sharing of reflections and observations. ·         for diverging and accommodating styles, paired activities allowed for active experimentation and concrete experiences. ·         for assimilating and converging styles, individual activities allowed for active experimentation and concrete experiences.   additionally, the emphasis on reflective sharing and paired activities required that the students remain focused and accountable.   results and discussion   students’ reflection and feedback   we wanted to know if the students in lesson one, where we first attempted to incorporate kolb’s theory, had different experiences from the students in lesson two, where we more effectively incorporated kolb’s theory after student feedback and our own reflection. students completed a post class survey meant to facilitate reflection. from this survey we were able to gather some general conclusions about their experiences. there were no remarkable changes in the student feedback from lesson one to lesson two. the majority of students in both sessions responded that: (1) they felt a positive change in their level of confidence after our teaching sessions; (2) they attributed the change to what they learned in the session; and (3) they found the session to be valuable and useful, with a few students finding the content to be confusing at times. some suggestions for changes from both student groups included: increase the length of the session, decrease the length of the session, incorporate a break, and slow the pace of the lesson. these results may indicate that different students had different needs and that different aspects of our teaching appealed to each type of learner in each session regardless of teaching kolb-style or not.   the student survey was not designed to evaluate our teaching effectiveness. it was meant to encourage student reflection on their learning experience as reflective observation is a key component of kolb’s theory (kolb, 2014). this activity allowed learners to reflect on what they had learned, what they did not understand, and prompted them to take control of their learning by seeking answers or librarian support.   both librarians and the instructional facilitator perceived a change in student engagement between lessons one and two. we collectively observed the students to be more engaged when kolb’s theory was more effectively incorporated into the lesson. the students appeared more focused on their learning activities, and more involved in the paired and classroom discussions and group work, and there was more time allotted for reflection and active experimentation. we perceived them to be less distracted and more actively involved in all aspects of learning.   facilitating students’ movement through kolb’s cycle of learning   kolb’s theory is about facilitating learning by moving learners through each stage of the learning cycle so that they may be able to understand and transform their learning into new knowledge (kolb, 2014). knowledge creation is facilitated if learners are able to resolve the cognitive dissonance between their previous learning experience with new learning, between concrete experience and abstract conceptualization as well as between reflective observation and active experimentation (kolb, 2014). the following section provides an example of how we facilitated students’ movement through the learning cycle in the database searching portion of the class.   concrete experience   at this stage, learners rely on their concrete experiences as they approach a task, using knowledge and skills based on both past and present experiential learning. we gave the students time to demonstrate their current searching skills on a research question related to their course assignment. most students used internet search engines, some used google scholar, and a few used databases. although some were successful at finding journal articles, none searched in a systematic manner.   reflective observation   reflective observation is about critically analyzing the learning and considering the impact of what has been learned. first, we facilitated a reflective conversation in which they shared their approaches to finding journal articles. we taught the students how to systematically search by introducing them to the following skills: creating a search strategy, using subject terms and keywords, choosing the appropriate databases, using limiters, and applying the same search strategy across various databases. to encourage reflective observation, we provided time to conduct searches using both the new method they had just learned and their previous methods. the students compared their results, and then shared their findings with a peer, then the class. this provided another concrete experience on systematic searching and facilitated another reflective discussion on their new experience.   abstract conceptualization   at this stage, learners critically analyze the new skills and think about how it applies to them accomplishing a task. in order to facilitate this internal, personal, and individualized cognitive process, we asked the students to work with a peer to create a new a search strategy in order to find peer reviewed journal articles for their research question. this activity encouraged them to collaboratively work through a problem and think critically about their past and new learning experiences in order to create an individualized approach to systematic searching. it was our hope that they would synthesize their original method with ours to complete this search.   active experimentation   at this stage, learners experiment with what they have learned and adapt it to their individual style in order to accomplish a task. we encouraged the students to adapt what we taught them and merge it with any previously successful searching strategies as they attempted to apply the new skills to their new search. we acknowledged there are different ways to search systematically and we encouraged students to experiment in order to find what will work best for them in the future.   librarian experience: reflection and feedback   the general themes that arose from our reflection and feedback sessions with each other and with the faculty facilitator are highlighted in table 1.   our reflective practice, inspired by applying kolb’s experiential learning theory to library instruction, has been essential to our growth and development as adult educators. self-reflection and soliciting feedback from multiple sources encouraged insight and awareness of the effects of our teaching styles. through this process we learned the value of making time to debrief with each other in order to facilitate our instructional development. we learned to value our differences while challenging ourselves to try new strategies in an effort to improve student engagement with information literacy skills.     table 1 self reflection and feedback session   self-reflection peer feedback faculty facilitator feedback lesson one on our teaching: ·  well organized ·  good time management ·  lesson was relevant and applicable to students – the lesson aligned with a research assignment the students were expected to complete for their nursing course ·  overall, satisfied with lesson plan and teaching strategies   on student engagement: ·   the students were focused on learning activities. ·   most students participated in activities. we discussed: ·  the pros and cons of our different communication styles, as they had an effect on our delivery of the lesson plan. ·  the pros and cons of the content we emphasized during the session (e.g., apa references vs. apa citations might receive more emphasis depending on the teacher). on our teaching: ·  teaching styles did not appeal to all types of learners. ·  we stood at the front of the room and lectured. ·  our questioning did not encourage reflection and critical thinking from the students.   on student engagement: ·  low engagement. many students were not engaged in learning, but were searching other websites. lesson two on our teaching: ·   initially, we were reluctant to try new teaching strategies. ·   we felt a time pressure in modifying the lesson by applying kolb’s theory. ·   over time we felt we had expanded our knowledge with the new teaching strategies. ·   we felt empowered by the new strategies. ·   we perceived the lesson to have been a success. ·   the lesson was relevant because it was tied to a course assignment.   on student engagement: ·  greater student discussion, collaboration and focus on their learning activities   we discussed ·  recognition of our different teaching styles. ·  appreciation for one another’s strengths as teachers.   we noted: ·  we had become more learner-centered and less teacher-centered. we would ask ourselves questions like what is the impact of our actions on the learners? we focused less on how we liked to teach and how we liked learn. ·  we had become facilitators of learning instead of lecturers. ·  we learned how to effectively give and receive feedback. on our teaching: ·  a variety of teaching styles were used to meet the needs of various learners. ·  we moved around the classroom engaging learners from all corners. ·  the activities were more reflective, stimulating critical thinking.   on student engagement: ·  high engagement. all students participated in the activities rather than visiting other websites.       librarian experience: kolb’s cycle of learning   for our professional development, we used kolb’s theory to process what we had learned about library instruction when we used his theory to guide teaching practice. we reflected on our learning style and teaching strategies as we progressed through kolb’s cycle of learning in order to gain insight into our teaching practice. the following section outlines our movement, as instructors, through kolb’s cycle of learning.   concrete experimentation   in lesson one, we created content, teaching activities, and teaching strategies based on our past learning and teaching experiences and preferences. student feedback was generally positive and our perception of the lesson was that it was organized and well managed. the students appeared engaged in the learning activities as they all completed the assigned tasks. we later realized that we were looking for strengths in our practice that validated our bias that we were effective instructors. the objective feedback from the instructional facilitator, an experienced instructor of adult education who observed our teaching, provided us with information that challenged our thinking and our practice.    the instructional facilitator’s feedback on lesson one, our first attempt at incorporating kolb’s theory, was as follows:   ·         during the lecture some students were engaging in their own conversation or using the computer for other purposes. ·         the students appeared bored and distracted at times, especially those students sitting at the back of the room. ·         we stood at the front of the room and mostly engaged with learners at the front. we did not engage learners from all areas of the classroom. ·         we asked closed ended questions about comprehension but did not wait for responses. ·         we did not ensure student accountability for their learning activity, nor did we evaluate their search queries or their apa exercise.   the instructional facilitator recommended the following changes be made in order to more effectively incorporate kolb’s experiential theory:   ·         when lecturing, walk around the room to get the attention of learners from all corners of the classroom. ·         have paired activities for peer-to-peer learning and support as well as individual activities for those learners who prefer to work on their own. ·         encourage discussion and reflection by asking reflective questions and giving students time to answer. ·         ensure students are engaged by randomly asking reflective questions to students in all corners of the classroom. ·         ensure students are accountable for their learning by asking them to show you how they search or how they apply apa. ·         give students time to experiment and compare their past search practices with the new approaches that have been introduced.   reflective observation   transforming the lesson   we compared the feedback from the instructional facilitator to our observations of the students with our first attempt at incorporating kolb’s theory. from our point of view, the students appeared engaged and we questioned the need for change. we were reluctant to incorporate the changes due to time constraints. creating an experiential lesson plan would require the addition of new activities for reflection, abstract conceptualization, and active experimentation. all of these additions are time-consuming and we questioned the feasibility of incorporating kolb’s theory into practice. ultimately, we decided to experiment with the suggestions and assess the student’s engagement when kolb’s theory was applied properly. our learning styles   another insight we had was: as teachers, our learning styles affect our teaching styles. in our previous experiences of teaching this class, we focused on content, teaching activities, and teaching strategies that were based on what we valued, our past experiences as learners and instructors, our learning style preferences, and what we learned in our master’s of library and information studies and fcp. completing kolb’s learning style inventory revealed that one of us favors a converging learning style, while the other is a combination diverging and assimilating style learner. converging style learners tend to have a natural instruction style, which focuses on the practical application of searching skills which can be applied to the student’s assignment, whereas the assimilating/diverging learner favors reflecting in a structured way through organized activities such as lectures, readings and discussions in order to explore new ideas and skills (kolb, 2007). teaching strategies such as brainstorming sessions and using guided logical conversations to find solutions to problems may appeal to assimilating/diverging learners.   these differences in our learning styles as instructors became evident throughout the project. at the completion of each lesson, we would often view different learning activities as having been the “most impactful” in cultivating student engagement and we would often disagree on strategies for moving forward. upon reflection, we found strengths in these differences that resulted in teaching approaches neither of us had considered before.  we came to realize, as kolb states, “ideas are not fixed and immutable elements of thought but are formed and reformed through experience” (kolb, 2014, p. 36).   abstract conceptualization   using kolb’s theory in library instruction presented some challenges in terms of the time required to move students through the learning cycle. to add to this, we understood some of the criticisms of kolb’s learning style and cycle of learning. coffield, moseley, hall, and ecclestone (2004) did a systematic review of various learning styles models with the objective of evaluating the validity and reliability of the theories, claims, and applications of these models. coffield et al. (2004) found the reliability and validity of the learning style inventory and learning cycle to be questionable. for example, matching teaching style with learning style does not improve academic achievement (coffield et al., 2004). despite this, the value of using kolb’s theory to guide library instruction is that it provides a theoretical framework to guide practice to use past experiences in present teaching, and to provide time for reflection and abstract conceptualization (critical thinking) as well as time for active experimentation (testing ideas and theories). it is also learner-centered and reminds instructors to teach to a variety of learning styles in the classroom.   active experimentation   we decided to experiment with team teaching for lesson two.  having two instructors in the room enabled us to use each other’s strengths and expose the students to two different styles of teaching, thereby appealing to more learning styles in the classroom. two librarians in the classroom enabled us to monitor students’ completion of tasks and to ask them to show us how they applied what they learned to the learning activities. we and the faculty facilitator observed that the students in lesson two appeared more engaged in learning activities, reflection, and discussion. student feedback was consistently positive in both lesson one and lesson two.   limitations and next steps   the amount of time required to implement kolb’s theory was an issue. because the delivery of certain content was required, there was insufficient time to incorporate the reflection and abstract conceptualization required for a true experience of kolb’s cycle of learning. that we saw the students only once also limited our opportunities for follow-up on student engagement, and subsequent adjustment of our teaching strategies. another limitation is that we did not randomly assign students to a control group, taught without applying kolb’s theory, and an experimental group, taught applying kolb’s theory, to measure any differences in student satisfaction or achievement of learning outcomes. the results observed during this process were based on our reflection and our perception of our performance and the impact on student engagement. there was no objective measurement of student engagement or teaching effectiveness.   this paper focused on our reflective practice and perceptions of the teaching-learning process as we incorporated kolb’s theory into library instructional practice. future researchers may want to focus on gathering empirical data in order to measure student satisfaction with information literacy skills instruction when librarians incorporate various learning theories into their teaching practice. interested researchers may also want to compare the effectiveness of using kolb’s theory on student learning outcomes and comparing that to when librarians use another adult learning theory to guide teaching practice.   conclusion   in this observational study we incorporated adult learning theory into two distinct lesson plans, delivered to two groups of students from the same program, completing the same assignment.  four types of qualitative feedback appeared to affirm that there were improvements in student engagement from lesson one to lesson two. it appeared that the effective incorporation of kolb’s theory resulted in greater student engagement and a more collaborative classroom environment. additionally, we experienced a transformation as teachers. we became more thoughtful, deliberate, and aware of our teaching purpose and goals and their potential effect on student engagement. the incorporation of multiple teaching strategies that address a variety of learning styles enabled us to facilitate the students moving through the cycle of learning in order for knowledge creation to occur. applying theory to practice increased our knowledge of adult learning theory and teaching practice, challenged our beliefs that we were already effective instructors, and motivated us to try new strategies that we had not considered, such as team teaching and being observed by a peer and a faculty facilitator. this study motivated us to change our practice to enhance student engagement and to develop into more effective information literacy instructors.    references bergsteiner, h., avery, g. c., & neumann, r. (2010). kolb's experiential learning model: critique from a modeling perspective. studies in continuing education, 32(1), 29-46. http://dx.doi.org/10.1080/01580370903534355   bowles-terry, m. (2012). library instruction and academic success: a mixed-methods assessment of a library instruction program. evidence based library & information practice, 7(1), 82-95. retrieved from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/12373/13256   clem, j.m., mennicke, a.m., & beasley, c. (2014). development and validation of the experiential learning survey. journal of social work education, 50(3), 490-506. http://dx.doi.org/10.1080/10437797.2014.917900     coffield, f., moseley, d., hall, e., & ecclestone, k. (2004). should we be using learning styles? what research has to say to practice. retrieved from http://www.itslifejimbutnotasweknowit.org.uk/files/lsrc_learningstyles.pdf   cox, l., clutter, j., sergakis, g., & harris, l. (2013). learning style of undergraduate allied health students: clinical versus classroom. journal of allied health, 42(4), 223-228. retrieved from http://www.asahp.org/publications/journal-of-allied-health/   crookes, k., crookes, p. a., & walsh, k. (2013). meaningful and engaging teaching techniques for student nurses: a literature review. nurse education in practice, 13(4), 239-243. http://dx.doi.org/10.1016/j.nepr.2013.04.008   d'amore, a., james, s., & mitchell, e. k. (2012). learning styles of first-year undergraduate nursing and midwifery students: a cross-sectional survey utilizing the kolb learning style inventory. nurse education today, 32(5), 506-515. http://dx.doi.org/10.1016/j.nedt.2011.08.001   devasagayam, r., johns-masten, k., & mccollum, j. (2012). linking information literacy, experiential learning, and student characteristics: pedagogical possibilities in business education. academy of educational leadership journal, 16(4), 1-18. retrieved from http://www.alliedacademies.org/academy-of-educational-leadership-journal/   halcomb, e., & peters, k. (2009). nursing student feedback on undergraduate research education: implications for teaching and learning. contemporary nurse: a journal for the australian nursing profession, 33(1), 59-68. http://dx.doi.org/10.5172/conu.33.1.59   kolb, a.y & kolb, d.a (2005). learning styles and learning spaces: enhancing experiential learning in higher education. academy of management learning and education, 4(2), 193-212. http://dx.doi.org/10.5465/amle.2005.17268566   kolb, d.a. (2014). experiential learning: experience as the source of learning and development (2nd ed.). upper saddle river: pearson.   lange, j., canuel, r., & fitzgibbons, m. (2011). tailoring information literacy instruction and library services for continuing education. journal of information literacy 5(2), 66-80. http://dx.doi.org/10.11645/5.2.1606   lisko, s.a. & o’dell, v. (2010). integration of theory and practice: experiential learning theory and nursing education.  nursing education perspectives 31(2), 106-108. retrieved from http://www.nln.org/newsroom/newsletters-and-journal/nursing-education-perspectives-journal   schilling, k., & applegate, r. (2012). best methods for evaluating educational impact: a comparison of the efficacy of commonly used measures of library instruction. journal of the medical library association, 100(4), 258-269. http://dx.doi.org/10.3163/1536-5050.100.4.007   woods, h. b. (2012). know your ro from your ae? learning styles in practice. health information & libraries journal, 29(2), 172-176.  http://dx.doi.org/10.1111/j.1471-1842.2012.00983.x editorial   collaboration and growth   lorie kloda editor-in-chief assessment librarian, mcgill university library montreal, quebec, canada email: lorie.kloda@mcgill.ca      2015 kloda. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/) which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   after the success of the journal’s feature section on the 2010 library assessment conference which appeared in issue 8(2) in 2013, i am pleased to present the current issue, which includes a feature section on the 2012 library assessment conference. the conference took place in charlottesville, virginia, and i was lucky enough to be in attendance. it was, in fact, my first library assessment conference, as i was a newly minted assessment librarian at the time. in the present issue, six papers are included: one commentary and five peer reviewed articles, based on the original conference proceedings. in addition, our feature section guest editor, martha kyrillidou who served co-chair of the 2012 conference, has an editorial. the editorial board at of eblip is delighted to continue its collaboration with the library assessment conference and to provide an open access venue to publish scholarship on library assessment.   this issue also includes the final ebl 101 column, which made its first appearance in issue 3(3) in 2008. published every issue, the column’s first 11 installments were meant to cover all the steps of evidence based library and information practice. the following 14 installments introduced various research methods. virginia wilson, who authored 23 of the 25 columns, says goodbye to the ebl 101 column but not to the journal. read her last column to find out her plans for a new column.   given all the successes of the eblip journal over the past decade, it’s no surprise to me to see new opportunities arising and to see the journal flourish. this success is due to past and current members of our editorial board. this past year, as the board prepared for alison brettle, to step down from her role as editor-in-chief, we discussed possible ways in which previous board members could continue to serve the journal. (i’ll confess to a mild panic at the thought of alison’s sudden disappearance from editorial meetings.) what we conceived was a group of experienced editors to help guide the journal’s direction and provide support during times of high volume of submissions.   it was decided that anyone who has served for a minimum of one year as a member of the editorial board of eblip, (as editor-in-chief, associate editor, production editor, or lead copyeditor, for instance) would be invited to become an editorial advisor. i am thrilled to be able to announce that we invited four prior editorial board members who met these qualifications, and all of them readily accepted! our new editorial advisors are lindsay alcock, alison brettle, katrine mallan, and pam ryan.   lindsay alcock (formerly glynn) is head, public services at the health sciences library, memorial university of newfoundland in canada. lindsay served as editor-in-chief of eblip from its inaugural issue in 2006 until 2008 (volume 3, issue 4). she is also currently a member of the evidence summaries writing team, and has served as a peer reviewer for the journal. alison brettle is a senior lecturer and information specialist in the school of nursing, midwifery and social work at the university of salford in the united kingdom. she served as associate editor (articles) of eblip and more recently as its editor-in-chief. katrine mallan is head of acquisitions at the university of ottawa in canada. pam ryan is the director of collections and technology at edmonton public library in canada. katrine and pam both served as production editors for eblip, and pam has also been a peer reviewer.   the role of editorial advisor, like other members of the editorial team, is completely voluntary. the purpose of the editorial advisor is to act in an advisory role within the editorial team: to offer guidance, provide support when needed, and to maintain continuity in the journal as it evolves. editorial advisors serve for a three-year term, which may be renewed. the entire team here at the journal is very excited about this new development, and we look forward to expanding our group to include these experienced members in order to continue to propel eblip into another decade of high-quality content.   evidence summary   undergraduate students who use library resources are also more likely to stay enrolled   a review of: haddow, g. (2013). academic library use and student retention: a quantitative analysis. library & information science research, 35(2), 127-136. doi:10.1016/j.lisr.2012.12.002   reviewed by: cari merkley associate professor mount royal university library calgary, alberta, canada email: cmerkley@mtroyal.ca   received: 01 sep. 2013   accepted: 07 nov. 2013      2013 merkley. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine if there is an association between library use and student retention.   design – quantitative analysis.   setting – a large research university in australia.   subjects – 6330 new undergraduate students.   methods – the researcher obtained a data set on all new undergraduate students registered at the institution in april 2010 from the student enrolment system. the data set included students’ identification number, age, gender, australian postal code, and country of residence. using the students’ identification numbers, the author then retrieved information from the library’s systems on the number of physical library items borrowed, and the number of logins to authenticated electronic library resources by this cohort at three points in the first semesters of 2010 and 2011. these three points in the semester fell after the course withdrawal date, mid-semester, and after exams. the author obtained additional data sets from the student enrolment system at the end of the first semester of 2010, and after the course withdrawal date and after exams in the first semester of 2011 to determine which students from the original sample were retained over the 18 month period. the researcher then compared library use data for students still enrolled at each date to those who had withdrawn from their studies.   the researcher also coded students’ data according to age and socio-economic status to allow further analysis. all students in the sample were grouped into two age categories: students under 21 years of age, and mature students, which included all students aged 21 years and over. those students with a permanent australian address (5125) were coded as low, medium, or high socioeconomic status using the 2006 socio-economic indexes for areas. postal codes were also used to determine if a student resided in close proximity to the university library in western australia, and could be expected to access the physical collection.   main results – students who withdrew by the end of their first semester in 2010 were more likely not to access online library resources at all (39% versus 20.4% of retained students). by the end of the first semester of 2011, retained students still showed higher use of library online resources. over half of those leaving their studies did not login to library resources by the end of the first semester of their second year of study, compared to 17.6% of retained students. borrowing rates for physical library items was very low among both retained students and those who withdrew from their studies in both years.   the data did not demonstrate a strong association between a student’s socioeconomic status, library use, and their retention. the findings regarding age were more significant when it came to retention, with mature students more likely to withdraw from their studies by the end of their first semester than those under 21. in terms of their library use, retained mature students were more likely to borrow physical items from the library than younger students in both their first and second years of study.   conclusion – while students who remained enrolled over the 18 month period did demonstrate higher use of the library’s electronic and physical collections than those who withdrew, the low use of the library’s physical and electronic resources even by those retained undermines any conclusions that could be drawn about the positive associations between library use and retention. mature students may benefit from targeted library supports, as their library use seems to be more positively associated with their retention than with younger students. socio-economic status did not appear to play a major role in library use and retention, according to the study’s findings.     commentary   this study is one of many currently attempting to connect the work of the academic library to larger institutional aims like student retention. while assessing the impact of library collections and services on student success is not new, there has been increasing emphasis placed on it by organizations such as the association of college and research libraries (acrl) in recent years. in her high profile acrl sponsored report the value of academic libraries, oakleaf (2010) highlights student retention as an area where academic libraries can and have demonstrated their contributions. the author’s work here shows that north american academic libraries are not the only ones feeling the pressure to provide evidence of their value, and emphasizes the need for those engaged in this work to look beyond their own borders for best practices and strategies.   the author states that other libraries will find more value in the study’s methodology than in its findings, which cannot be easily generalized. this is, in part, due to the eccentricities of the data collected at this institution. for example, categorizing all students 21 and over as mature is problematic, but the data obtained from the university’s enrolment system could not be parsed further. logins to authenticated resources may also be one of the best data sets available for exploring library use, but as students authenticate at this institution for services ranging from chat reference to database access, it is difficult to tie results to collection spending or specific types of library support. the limitations of using postal codes to determine socioeconomic status are also acknowledged. an earlier study by the author at the same institution did find differences in the use of the library’s computers based on students’ socioeconomic status, and it would be interesting to know why that source of data was not used again or why the author relied only on the use of electronic resources as a measure of students’ library use in relation to this variable in the follow up research (haddow & joseph, 2010).   overall, the study’s transparency around the strengths and limitations of the methodology employed, as well as the detail provided about the results (for example, including the mean, median, and mode for logins) increases its utility as a model for other libraries (glynn, 2006). this study provides a useful example for how libraries can use data collected in the course of university business, like registration data, to explore library impact on students. those interested in library use and student retention should also refer to the oakleaf report, which outlines a variety of additional data points and methods that could be used for this type of research.     references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi:10.1108/07378830610692154    haddow, g., & joseph, j. (2010). loans, logins, and lasting the course: academic library use and student retention. australian academic & research libraries, 41(4), 233-244. doi:10.1108/07378830610692154   oakleaf, m. (2010). the value of academic libraries: a comprehensive research review and report. chicago, il: association of college and research libraries. retrieved 25 nov. 2013 from http://www.ala.org/acrl/sites/ala.org.acrl/files/content/issues/value/val_report.pdf     conference paper   linking information seeking patterns with purpose, use, value, and return on investment of academic library journals donald w. king honorary university professor bryant university smithfield, rhode island, united states of america and adjunct professor university of tennessee knoxville, tennessee, united states of america email: donaldwking@gmail.com   carol tenopir chancellor’s professor university of tennessee knoxville, tennessee, united states of america email: ctenopir@utk.edu      2013 king and tenopir. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to demonstrate the power of the critical incident method in studying the information seeking patterns of university faculty.   methods – faculty at five u.s. universities participated in a study concerning their information seeking and reading patterns involving scholarly journals. the surveys relied on a critical incident method of asking questions concerning the last journal article read. this method allows analysis of the relationships among the purposes of reading articles, ways in which faculty first learned about the articles, where they obtained them, aspects of their use, and the value or impact of the information read.   results – results show that journal articles were by far the most used source of the last substantive piece of information used for work. over half of article readings were from articles provided by libraries (52%, compared with 32.6% from personal subscriptions), and journal articles were the most frequent way faculty became aware of information prior to reading about it (33.9%, compared with 19.4% from informal discussions).   conclusion – this project has shown that articles read for the purpose of research, found by searching, and obtained from the library collections have the highest value to faculty by many measures. library provided articles save faculty time and effort, which can be quantified using contingent valuation. the return on investment (roi) for library collections can be calculated by measuring all library costs and establishing the monetary returns to faculty members through contingent valuation. library journal collections are estimated to have an roi of between 3.3 and 3.6 to 1.   introduction   this paper is based on data gathered from an institute of museum and library services (imls) sponsored study “maxdata” involving surveys of university faculty concerning their use of academic library scholarly journal collections. the emphasis of this paper is to demonstrate the relationship of how purposes of reading scholarly journals (e.g., research, teaching, current awareness, etc.) lead to the information seeking patterns used by them (e.g., how they identify articles that are read, where they obtain them, etc.), which dictates certain aspects of use (e.g., how much is read, age of articles read, format of the articles, etc.), which is related to the positive outcomes or value of reading (e.g., increased productivity, improved research or teaching, saving readers’ time or money, etc.), which serves as return components of the roi of academic library journal collections.   the paper gives an example of how the method of observing a critical incident of the last reading can be used to determine the above relationships. estimates are made for the number of readings made for research; of these, the number identified from searching; and of these, the number obtained from library collections, the age of the readings and whether from print or electronic versions of the articles, the amount of time spent obtaining and reading the articles (as an indicator of what readers “pay” for the content read), the ways in which the reading affected research (whether the article read is eventually cited), and the “contingent value” of the articles read from the library collection.   contingent value is an economic method used to assess the benefits of non-priced goods and services, by examining the implications of not having that product or service. in the example presented here, an estimate is made of how much more it would cost readers to obtain the article if there were no library collection. this value is compared to the relative cost of the library collection and the cost to the reader to estimate the roi. of course, this is only one way to do so. the university of tennessee and other participants are currently conducting another imls study (libvalue) to develop additional measures of “value” and “roi” of all academic library services, in addition to those provided by their journal collections.   summary   over the years there have been hundreds of studies that provide estimates of the value of all types of libraries and more recently on the roi in libraries. the university of tennessee, school of information sciences and the u.k. research team ciber (www.ciber-research.eu) were funded by the institute of museum and library services to examine how to maximize library investments in digital collections through better data gathering and analysis. the study focused on alternative means of collecting journal and article usage metrics including readership surveys, local server hits and downloads, data provided by various vendors, and deep log analysis of electronic journal usage data collected by ciber from ohiolink. the findings were intended to enable librarians to learn what conclusions can be drawn from each metric, the strengths and weaknesses of each one, how they complement one another, and what conclusions can be drawn if only one source of data is available.   this paper addresses the survey of faculty at five us universities that participated in the study concerning their information seeking and reading patterns involving scholarly journals (king, tenopir, choemprayong, & wu, 2009; tenopir, king, edwards, & wu, 2009). the surveys relied on a “critical incident” method of asking questions concerning the last journal article reading. this method allows analysis of the direct relationship of faculty purposes of reading articles, ways in which they first learned about the articles, where they obtained them, aspects of their use, and the value or impact of the information read. this information is used to establish the role academic libraries play in achieving value from their collections and, ultimately, a value contribution to the return component of journal collection roi.   examples of value include “purchase” value in terms of how much readers pay for the information in their time and money to obtain and read articles and the “use” value in the consequences of reading the information such as saving readers’ time in doing their work, improving their productivity, inspiring new thinking or ideas, improving their work, resulting in collaborations, and so on.  one indicator of the value of journal collections is how much more it would cost readers to obtain the same information, if the journal collections were not available to them. the investment component of roi is based on the relevant library cost and the cost to faculty in their time and money for browsing, searching, printing, and photocopying (king, aerni, brody, herbision, & kohberge, 2004a; king, aerni, brody, herbision & knapp, 2004b). a current imls sponsored study is developing additional measures of “value” and “roi” of all academic library services  to all who benefit from these services in a variety of ways.   context is given for the role academic library journal collections play in achieving value. it is emphasized that it is “information content” that achieves value from reading and not the journals or articles, and that academic library services facilitate access to the information content in various ways. one aspect of this context is that faculty use many information sources to do their work and journal articles are collectively only one such source. a second context is that readers can obtain articles they read from many article sources such as personal subscriptions, article reprints, colleagues, authors, free web journals, and so on – library collections serve as only one of these article sources. a third context is that article information is often known (or partially known) before an article is read, and since the surveys focus on the last article read, it is possible that the reading is only the most recent of many past readings of the article by a faculty member. these contexts for assessing academic scholarly journal collections are discussed in the next section.   a section of this paper is also devoted to the purposes for which information is read by faculty, including research, teaching, writing, keeping up with new information, continuing education, and so on. information seeking patterns are closely related to those purposes and form the focus of that section. such patterns include ways in which readers learn about the articles they read (e.g., browsing, searching, being told, etc.) and the article sources used (e.g., personal subscriptions, library collections, etc.). an example is given in which articles primarily read for research are identified through searching and subsequently obtained from library collections.   the section following purposes for reading journal articles discusses various aspects of the use of article information, such as age and format of the articles. these aspects are given for those articles read for research, which have been found by searching and obtained from library collections.   the section following that deals with the outcomes or value of information found in journal articles. an example is given for value of information read for research, identified by searching, obtained from library collections, and the age of the articles. finally, the collection roi achieved for universities is estimated for these articles and compared with other reasons for reading, information seeking methods, and use aspects.   the contexts for assessing academic scholarly journal collections   in the broadest context, one can think of all activities in research communication. for example, various authors have described the flow of research information from discovery through oral or written reports over time, with journal articles being somewhat down the chain in time. (garvey,  lin, & nelsen, 1970; crawford, hurd, & weller, 1996).   one can think of information as being a resource that faculty use to perform their work and, for that matter, the principal output from their work. faculty can choose from many information sources such as journal articles, books, personal contact, and so on. the survey asked a question about sources used by faculty, which provides an indicator of the relative importance of such sources. the question asked: “what sources did you use for the last substantive piece of information you used for work?”  (king et al., 2009 shows figures; figures are also included in the 2010 library assessment conference proceedings version of this paper). the results demonstrate the relative importance of information found in articles compared with other sources of information and supports a reason for examining journal articles further.   there are a number of sources of journal articles that are read, including personal subscriptions; library collections (e.g., central and department library subscriptions/databases, interlibrary loan or document delivery services); preprint or reprint copies; copies from colleagues, authors, etc.; and repositories. it is useful to know the relative use of library collections in order to assess the importance of them and to examine why one source is chosen over others. as shown later, one aspect is purpose of reading and another aspect is the way articles are identified. when looking at the sources used to obtain the last article read by faculty, the results show that library provided articles are found to be the prevalent source (king et al., 2009).   another context addresses the fact that a scholar often knows about the information in an article prior to reading it the first time. the surveys revealed that nearly half of the articles last read contained at least some information previously reported (king et al., 2009). there are many possible reasons that an article is read even though some of the information is known. for example, the reader might have heard about it at a conference and then waited for it to be edited and sent by a publisher to be refereed. the article might have been cited in another article in which case only some information found in the current reading is revealed.   it is abundantly clear that information found in articles is important to faculty work, libraries are used most often as a source of articles, and information cited in journal articles frequently leads to the entire information in articles being read. the importance of journal articles is well established.   purposes for reading journal articles   information in journal articles is read for many purposes. survey respondents were asked: “for what principal purpose did you use, or plan to use, the information obtained from the last article you read? (choose only the one best answer)” (king et al., 2009). research is the purpose most often given by faculty. faculty average 240 readings annually so that about 114 readings a year are addressed to research in their work. below the information seeking patterns used to obtain those 114 readings are described, as well as how these patterns compare with other purposes for reading.   it is shown that the purposes for reading determine to a large degree how readers become aware of the articles and where they obtain them. a detailed example is given for articles that are read for research, identified from searching, and obtained from libraries to show how the critical incident of the last reading can be applied.   in examining how faculty became aware of the article read for research, we see about an equal proportion of readings are found by browsing or searching and less from citations or another person. the question was worded as follows: how did you become aware of this last article you read? specific options to the response “found while i (or someone on my behalf) was searching (i.e., by subject or author’s name)” included: web search engine, electronic indexing/abstracting service, print index or abstract, online journal collection, current awareness service, and preprint/e-print service. the estimated number of readings done for research and found by searching is about 34 of the total 240 readings. browsing is used much more for teaching (44.1% vs. 30.9% of readings for research) and current awareness (64.7%). articles found through citations are used much less for teaching (12.2% vs. 21.2% of readings for research) and current awareness (only 2.5% of these readings).   about 42.3% of readings for research and found by searching involves information discussed in the article that is previously known. this compares with 56.9% of readings done for other purposes. thus, even when information is known it is necessary to search at some level for many articles read for research, but less than readings for other purposes (56.9% of these readings). faculty were asked if they had previously read the article. answers did not vary much by purpose of reading and method of identifying the article. that is, about 17.0% of readings for research and found by search were re-readings compared with 19.6% of all other readings.   across all readings (i.e., 240 annual readings per faculty), faculty tend to obtain articles most frequently from library provided sources (52.0% of readings) or personal subscriptions (32.6%)  (king et al., 2009).  these article sources depend a great deal on the purpose of reading and how the articles were initially identified.   nearly two-thirds of articles read for research are obtained from libraries (compared with 52.0% for all readings). the 114 readings done for research are often found by browsing (30.9% of these readings). however, they are also often identified through searching (26.7% of these readings), citations (15.8%) and occasionally being told by another person (less than 17.6% since that person may do this by providing a copy of the article). in these instances and, sometimes when found by browsing, the readers must look for a place to obtain the articles. most (76.9%) of the readings for research and found by search were obtained from a library provided article source (king et al., 2009).     these articles tend to be older and, therefore, are more difficult to obtain, which is why libraries play an important role. in the next section, we show that article use aspects such as age and format are dependent on information seeking patterns, particularly on how articles are identified and where they are obtained. later it is shown that value of information read is also dependent on the purpose of reading and information seeking patterns.   aspects of the use of article information   the average age of articles read is 4.1 years old with the distribution of age being highly skewed, much like a nuclear decay curve. about half of article readings are made in the first year following publication or posting, but 2.8% of the readings are over 25 years old and 2 readings observed in the surveys were published in 1943 and 1947. the age of articles read depends a great deal on how the articles were identified and where they were obtained. for example, the average age of the articles found by browsing is 1.8 years, but much higher for those found by searching (4.7 years), from citations (8.0 years) and those mentioned by another person (3.5 years). citations appear to identify particularly old articles, and contribute to 25.6% of readings from citations over 10 years old.   browsing is done from all article sources: 51.7% of browsed articles are from personal subscriptions, 37.6% from library provided articles, and 10.7% from other articles sources. the average age of readings from personal subscriptions is 1.9 years, which suggests that the age of browsed articles from other article sources must also be low since the overall average age from browsing is 1.8 years. on the other hand, articles identified by searching and from citations largely come from library provided sources, which have an average age of 4.8 years. the average age of articles identified by searching is 4.7 years and three-fourths of these are provided by libraries (only 7.8% from personal subscriptions and 17.2% from other article sources). similarly, 61.9% of cited articles are obtained from libraries and average 4.1 years old. it seems clear that there is a strong age of reading relationship between the way read articles are identified and the article sources used to obtain them.   about 24 of the 240 average readings are for research purposes, identified by search and obtained from libraries. the age of these articles becomes even older at 6.2 years (compared with 4.0 years for the rest of the readings). the information from them is shown later to have greater value to the readers.   another aspect of use of articles is the format of the articles when read, which is also somewhat dependent on information seeking patterns. for example, across all readings about 54.4% are from electronic formats with 62.5% of readings for research from electronic formats. only 12.6% of articles obtained from personal subscriptions are in electronic format while 71.2% of library provided articles are electronic and 68.5% from other article sources are as well. it is clear that libraries are a major source of electronic journal articles and provide a substantial number of readings in this format (82 of 240 average readings by faculty). very few of the library provided readings take place in the library (5.4% of readings) and most of these are from print versions (84.2%), largely from browsed journals in periodical rooms. average age of print journals is 4.4 years and electronic journals 3.9 years, where most of the difference is in articles over 10 years old. about three-fourths of articles read for research, identified by search and obtained from libraries are electronic compared with 51.5% for all other readings.   the purpose of readings, information seeking patterns, and aspects of use all have a bearing on the value of information read in articles. in turn, the value is the return component of the roi of academic library journal collections.   the value of information provided by journal articles   for this paper we differentiate value as:   purchase or exchange value: what one is willing to pay in time and/or money for information found in journal articles. use value: the favorable consequences derived from reading and using the information. (machlup, 1979)   a paradox that demonstrates the above concepts is that gems have high purchase value but low use value. on the other hand, air has low purchase value but high use value. thus, the use value of information in articles generally is much higher than the purchase value.   the purchase value of journal information   faculty pay for journal information through their time (and effort) in information seeking and reading journal articles and the price paid for personal subscriptions. faculty on average spend about 150 hours per year in information seeking and reading, or an average of 37.5 minutes per reading (based on 240 readings per year). this sub-divides into 33.1 minutes reading and 4.4 minutes in information seeking.   the reading time is a good indication of the value of articles since readers would not choose to use this valuable time if the information was not of value to them. the reading time is related to purpose of reading, source of articles read, age of articles, and format of articles as discussed below. for example, the average time spent reading for various purposes has been documented (king et al., 2009). the information used for research and writing has greater purchase value than that used for other purposes such as teaching and current awareness.   considering the source of articles, library provided articles average 35.4 minutes per reading compared with personal subscriptions (27.9 minutes), and other sources (34.0 minutes), which suggests that library provided articles have greater purchase value. older articles tend to have greater purchase value (30.1 minutes for articles one year old, and 41.1 minutes for those over 5 years old). print and electronic versions of articles have about the same value (32.9 and 33.4 minutes respectively). both personal subscriptions and library provided articles gain value the older they are. that is, personal subscriptions go from 26.8 minutes for 1-year-old articles to 38.5 minutes for those over 5 years old and library provided articles increase from 31.8 to 42.8 minutes. the time spent reading articles for research found by searching and obtained from libraries is 39.1 minutes, which suggests these articles have greater value.   interestingly, the average time spent browsing per article read is greater than that spent searching (6.9 vs. 5.3 minutes respectively). generally, browsing from electronic sources takes less time than browsing from print versions (6.7 vs. 7.1 minutes respectively). this is affected by the fact that browsing electronic personal subscriptions takes more time than browsing print copies (7.6 vs. 6.8 minutes) and, on the other hand, browsing electronic library subscriptions is somewhat less time consuming (6.4 vs. 8.0 minutes).   use value of information found in articles   in a sense the purposes of reading are an indication of the value of information in articles, but a better set of indicators is the outcome or consequences of reading the information. survey respondents were asked: “in what ways did the reading of the article affect the principal purpose? (choose all that apply).” outcomes vary by purpose of reading, how articles are discovered, and source of articles. it appears that articles read for the principal purpose of research, which were identified from searching and obtained from the library, yield greater value than the other readings in terms of outcomes. faculty were also asked: “how important is the information contained in this article to achieving your principal purpose.” coded responses are that the information is not at all important, somewhat important, or absolutely essential. across all readings, 37.6% were said to be absolutely essential compared to 45.8% of the research readings, presenting further evidence of the value of the information to research. additional evidence is that information from more readings done for the purpose of research were said to be cited in a paper or report. over half (51.4%) of articles read for research are likely to be cited, while 25.2% may be cited, and 21.5% already have been cited. in contrast, about half of readings for other purposes will never be cited.   in the past two years, faculty averaged being an author or co-author of 3.41 articles in refereed scholarly journals; 1.21 non-refereed articles; 1.07 chapters in books, proceedings, etc.; and 0.11 books. only 14.3% of faculty had not authored any publication. most faculty (74.4%) had authored at least 1 article. another indicator of value of article information includes the productivity given to readers by authorship. it is observed that those who read more are more likely to publish. for example, those who published a refereed article in the past 2 years read an average of 28.2 articles in the past month compared with 22.7 readings for those who did not publish.   contingent valuation is an economic method used to assess the benefits of non-priced goods and services by examining the implications of not having that product or service. an indicator of the value of the library journal collection is to estimate what it would cost readers in time and/or money to obtain the information read from the collection if there were no collection. this is found by asking the following multi-part question pertaining to analysis performed only on readings from library provided articles:   thinking back to the source of the [last read] article, where would you obtain the information if that source were not available? (a) i would not bother getting the information. (b) i would obtain the information from another source. if (b) is checked: in order to obtain the same information, if this source were not available, i would expect to spend _______ minutes of time and/or $ _______. (if the answer is zero, answer “0” instead of leaving blank)   results are given below from a survey conducted at the university of pittsburgh where there was an average of 125 readings from library provided articles (king et al., 2004a; king et al., 2004b). the faculty indicated they would look for alternative sources of information for 99 of these readings. they averaged 3.0 hours per year searching, 3.4 hours browsing and 6.4 hours in obtaining useful citations, and articles from elsewhere, as well as photocopying, downloading, and printing articles. that is, about 12.8 hours were taken to search, browse, and identify articles from citations, other persons, etc. at about an average $55 per hour in salaries, benefits, etc. the cost to faculty was $704. in addition it cost the library and other facilities about $65 per faculty in photocopying, downloading, and printing for a total of $769 per faculty member.   the cost of obtaining alternative sources of the information is 59 hours in time ($3,295) and $990 for subscriptions, travel, communications, etc. therefore, the additional cost to readers is 46.2 hours of time (59-12.8 hours) or $2,641 and $425 in other costs ($990-$65) or a total of $3,466 per faculty member.   a similar analysis was done for readings used for research, found by searching, and obtained through library sources. here there were about 26 readings in which faculty would seek alternative sources and spend 2.3 hours searching, about 1.5 hours downloading and printing and/or photocopying at $60 per faculty. the time spent going to alternative sources is 14.8 hours and $538 in other costs. the net cost of the alternatives is 11.0 hours (14.8-3.8 hours) at $605 and $478 in other costs ($538-$60) or a total of $1,083 per faculty.   therefore, one indicator of the value of the library collection is that it saves faculty on average about $3,466 annually. when reading is done for research, found by searching and obtained from the library collection, the savings is about $1,083. these values can be considered a return dollar component of the roi in the library collection along with other “value” components mentioned earlier.   return on investment in academic library journal collections   the university investment in the library journal collection includes all of the library costs of purchasing and maintaining print and e-journal collections (allocated to an appropriate amount of reading) and the cost to readers in their salaries, benefits, etc., and other costs. for readings obtained from the library collection the library cost is about $283 per faculty member, and the cost attributed to readers is $769 or $1,052 total investment by the university in that portion of the library collection. therefore the roi of library journal collections is $3,466 ÷ $1,052 or 3.3 to 1.   the library cost to serve reading done for research and found by searching is $75 per faculty member. this added to the cost to users of $228 results in an investment of $303. therefore, the roi is $1,083 ÷ $303 = 3.6 to 1.   conclusion   this article shows that journal articles are by far the most used information source of the last substantive piece of information used for work (articles 92.4% of the time, books 51.5%). over 50% of article readings are from articles provided by libraries (libraries 52.0%, personal subscriptions 32.6%), and journal articles are the most frequent way faculty became aware of information prior to reading about it (articles 33.9%, informal discussions 19.4%). it is clear that journal articles play an important role in faculty communication.   scholarly articles are read by faculty for many purposes, including research, teaching, writing, and current awareness. they find articles in many ways, including browsing and searching, and they obtain articles from a variety of sources, including library collections and personal subscriptions. readers benefit in many ways from using information contained in articles. every article reading is unique in its combination of purpose, ways of identifying articles, sources used to obtain articles, and benefits gained from reading them. by using the critical incident observation of the last reading of articles, the complexity of article information seeking and reading patterns can be sorted out. the purpose of this article is to demonstrate the power of the critical incident method. it also serves as a baseline for future studies.   studies of this kind can also be used by a variety of audiences, such as readers to know if their information seeking can be improved, publishers to understand their market better, and librarians to make decisions on how best to serve their users. this article focuses most on the library issues. there are several ways to use this information to measure the value of article reading and the value of library access.   faculty members spend considerable amounts of their time on discovering, obtaining, and reading articles, thus demonstrating the value of libraries for their research and teaching. the outcomes of reading are an even better indicator of value in supporting the principal purpose of reading, and a majority of articles are said to be essential or important to the principal purpose.   this project has shown that articles read for the purpose of research, found by searching, and obtained from the library collections have the highest value to faculty by many measures. library provided articles save faculty time and effort, which can be quantified using contingent valuation. if the library collections were not available, the cost in time and other costs to faculty would be high.   the roi for library collections can be calculated by measuring all library costs and by establishing the monetary returns to faculty members through contingent valuation. library journal collections are estimated to have a return of between 3.3 and 3.6 to 1.   academic librarians may be tempted to ask, why does this all matter? libraries need to consider ways to measure their value in order to prioritize services in an era of new possibilities and declining or steady-state resources. and in an era of accountability and competition, academic libraries need to employ multiple ways to demonstrate their value to their constituents and funders. measuring the use and value of scholarly journal article collections is one finding of the studies reported here, but is just one piece of the value picture. the libvalue project continues on this work and examines ways to measure other library products and services (www.libvalue.org).     references   crawford, s. y., hurd, j. m., & weller, a. c. (1996). from print to electronic: the transformation of scientific communication. medford, nj: information today.   garvey, w. d., lin, n., & nelsen, c. e. (1970). communication in the physical and social sciences. science, 170, 166-173.   king, d. w., aerni, s., brody, f., herbision, m., & kohberge, p. (2004a). comparative cost of the university of pittsburgh electronic and print library collections. pittsburg, pa: the sara fine institute for interpersonal behavior and technology. retrieved 7 june 2013 from http://web.utk.edu/~tenopir/research/pitts/pitt_cost_final.pdf   king, d. w., aerni, s., brody, f., herbision, m., & knapp, a. (2004b). the use and outcomes of university library print and electronic collections. pittsburgh, pa: the sara fine institute for interpersonal behavior and technology. retrieved 7 june 2013 from http://web.utk.edu/~tenopir/research/pitts/pitt_use_final.pdf   king, d. w., tenopir, c., choemprayong, s., & wu, l. (2009). scholarly journal information-seeking and reading patterns of faculty at five us universities. learned publishing, 22(2), 126-144. doi:10.1087/2009208   machlup, f. (1993). use, value, and benefits of knowledge. science communication, 14, 448-466. doi:10.1177/107554709301400407 (original work published 1979)   tenopir, c., king, d. w., edwards, s. & wu, l. (2009). electronic journals and changes in scholarly article seeking and reading patterns.  aslib proceedings: new information perspectives, 61(1) 5-32. doi:10.1108/00012530910932267   ebl 101   research methods: the most significant change technique   virginia wilson director, centre for evidence based library and information practice (c-eblip) university library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   received: 11 aug. 2014 accepted: 29 aug. 2014      2014 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   in this ebl101 column i am exploring a technique that has largely been used to evaluate international development programs. the most significant change technique (msc) was developed by rick davies and jessica dart in the early 2000s to evaluate complex interventions. it takes place during the lifespan of the intervention program, so it is a process of continuous evaluation. while i have not encountered this method used in library and information studies, it strikes me that a technique such as this would be useful in variety of situations: evaluating instruction, appraising public programs, assessing various initiatives in any area of the library (client services, technical services, etc.), and others that i am probably just not seeing right now.   the methodology is participatory, so its use would be a good chance to have direct contact and conversation with various stakeholders. for example, library patrons, library staff, higher administration, the public—whoever is involved with whatever is being looked at and changed. while this method could perhaps be used on smaller projects in the library, i see its usefulness as being centred more upon large, organization-wide developments and changes, as a way to continuously monitor the situation and make adjustments as the project progresses. examples of larger projects include the implementation of a different organizational structure, the design, development, and building of a library facility, advancement initiatives, or a large-scale research project that is national in scope.   dart and davies (2003) refer to msc as a “dialogical, story-based evaluation tool” that eschews “conventional monitoring against quantitative indicators” in favour of the “collection and participatory interpretations of ‘stories’ about change” (p. 138). these stories are “examples of significant program outcomes are collected and presented to designated groups of stakeholders who deliberate on the value of these outcomes in a systematic and transparent manner” (dart, 2005, p. 261). though stories are the focus of the analysis, dart and davies (2003) indicate that “the central aspect of the technique is not the stories themselves, but the deliberation and dialogue that surrounds the process of selecting significant changes” (p. 138).   on his news website, davies (2008) states that msc is most useful in the following situations:   where it is not possible to predict in any detail or with any certainty what the outcome will be where outcomes will vary widely across beneficiaries where there may not yet be agreements between stakeholders on what outcomes are the most important where interventions are expected to be highly participatory, including any forms of monitoring and evaluation of the results   msc consists of seven key steps (although the guideline document listed below under resources by the same authors outlines 10 steps):   the selection of domains of change to be monitored unlike performance indicators, which are specific and focused, the domains of change are broad and loose, allowing for program participants to define them for themselves. the domains are identified by stakeholders. the reporting period “stories of significant change are collected from those most directly involved” over a time period decided upon at the start of the project. the time period can be extended if more stories are needed. the participants the participants are those who are involved with the program in question, such as beneficiaries, clients, and field staff. phrasing the question stories are gathered by using one simple question: “during the last month, in your opinion, what was the most significant change that took place in the program?” the structure of participants the stories are then analyzed and “filtered up through the levels of authority typically found within an organization or program”, with each level selecting the most significant change stories to be sent on up the ladder. feedback continuous communication amongst stakeholders participating in and reviewing the stories is a key component, so that the feedback can be incorporated into each subsequent round of story collection. verification verification can take place by visiting the sites of the events described in the stories for follow up. (adapted from dart & davies, 2003, pp. 138-139)   for example, the above steps might be mapped on to a library or information management process in the following way when a complex intervention is chosen for continuous evaluation, e.g., the move to a programmatic approach to academic library instruction. domains to be monitored are selected by participants in this change process, and by those who the change affects, such as librarians, students, library staff, and others from the larger institution (professors from various colleges who utilize library instruction, administrators from colleges who are participating in the construction of a programmatic approach to instruction, etc.). stories are gathered from the participants regularly over the academic year to provide continuous monitoring of the change intervention. stories are the result of asking a simple question, such as “what was the most significant change that happened this month as a result of the programmatic approach to instruction recently implemented?” those monitoring the change analyse and examine the stories at every level, with the different analyses going up the organizational structure to be further analysed. all participants provide and share continuous feedback, providing more information about the change occurring. follow up should happen, with various participants talking to other participants about the change taking place, verifying the analyses of the stories.   the msc technique could be an interesting framework with which to assess and evaluate our professional practice. dart and davies (2003) state that the key strength of msc “lies in its ability to facilitate a dynamic dialogue between designated stakeholders” (p. 152). as librarians and information professionals, we should be looking for new and innovative ways to communicate with our users, clients, patrons, and with each other as we strive to provide the best services possible.   other resources and examples of msc in action   davies, r., & dart, j. (2005). the ‘most significant change’ (msc) technique: a guide to its use. retrieved from http://www.alnap.org/resource/8102   lunch, c. (2007). the most significant change: using participatory video for monitoring and evaluation. participatory learning and action, 56, 28-32. http://www.iied.org/participatory-learning-action     wilder, l., & walpole, m. (2008). measuring social impacts in conservation: experience of using the most significant change method. oryx, 42(4), 529-538.     http://dx.doi.org/10.1017/s0030605307000671   willetts, j., & crawford, p. (2007). the most significant lessons about the most significant change technique. development in practice, 17(3), 367-379. http://dx.doi.org/10.1080/09614520701336907       references   dart, j. (2005). most significant change technique. in s. mathison, (ed.), encyclopedia of evaluation (pp. 261-263). thousand oaks, california: sage.   dart, j., & davies, r. (2003). a dialogical, story-based evaluation tool: the most significant change technique. american journal of evaluation, 24(2), 137-155. http://dx.doi.org/10.1177/109821400302400202   davies, r. (2008, apr. 13). most significant change (msc). monitoring and evaluation news. retrieved from http://mande.co.uk/special-issues/most-significant-change-msc/ article   internet connectivity and accessibility in university libraries: a study of access, use and problems among faculty of natural sciences students, university of jos, nigeria   daniel abubakar university library university of jos jos, nigeria email: abubakardb1968@yahoo.com   rhoda diyoshak university library university of jos jos, nigeria email: diyoshakr@yahoo.com   received: 21 nov 2014   accepted: 18 aug 2015      2015 abubakar. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – this study has the objective of establishing whether the undergraduate students of the faculty of natural sciences, university of jos, have access to and use internet facilities in the university library.   methods – a survey research design was adopted for this study and questionnaires were used in gathering data. statistical methods used in the analysis include percentages, frequencies, and chi-square test for measuring the association of library visit and use of the internet.   results – the analysis of the data and findings indicated that there is internet connectivity in the library. the findings also revealed that few students (15.5%) use the computer and the internet on a daily basis. the problems of slow internet connection at peak periods and unsteady power supply were clearly identified. furthermore, the analysis revealed that there is no association between the students’ library visits and their use of the internet for most academic purposes, except for downloading articles.   conclusion – the presence of internet connectivity in the library, does not translate to meaningful academic behaviour among the students. therefore, sensitising and training of the students on internet usage were recommended for better academic performance and life-long learning. introduction   university students pay frequent visits to their libraries to search and retrieve relevant and current information in electronic/online format for the purpose of effective learning and research. university library patrons include undergraduate and postgraduate students, researchers, information professionals, staff, and other users from outside the university who intend to use the university library. the undergraduate students are expected to read further after class to collect and retrieve information for assignments, seminars, term papers, and projects and this information could be retrieved from the internet with adequate access and connectivity. the undergraduates of a university, particularly the final year students, need information to satisfy their needs, and to enhance their academic pursuit during their course of study in the university. in addition, they need information to write their projects for the requirements of the award of their degrees.   the mandate of the university library is therefore to provide adequate and relevant information resources both in print and online for users. the print information resources include journals, textbooks, and magazines; the internet and online resources are to support class work, assignments, research/project work, term papers and, seminar presentations by providing relevant information and services for effective and efficient achievement of academic and social purposes. internet connectivity and accessibility to information resources on the internet are an integral part of the research process for students. after discussing with their professors, students look up the references in class readings, go to the internet for resources, and begin the research process. university libraries and students in many countries use internet resources and search engines regularly as well as e-mail as a normal form of communication (kindilchie & samarraie, 2008).   one of the major problems confronting university education in nigeria has been inadequate current and relevant information materials for teaching and research (okonofua, 2008). according to (adika, 2003) efforts had been made to ameliorate the challenges through interlibrary loan and document delivery, yet the situation persisted. it was therefore, in the light of this that the use of the internet was introduced into the educational system to bridge the information gap (okonofua, 2008). the university of jos and its library have also subscribed to the internet and its facilities (university of jos library, (2015). the library has provided a computer laboratory with the sole aim of providing access to information on the internet. the information that is retrieved from the internet could promote the students ‘academic purposes. in addition, other academic databases could also be accessed in the library. help is provided in the use of the databases subscribed to. users of this facility are required to collect the respective username and password of the subscription databases from the respective subject librarians. in addition, the systems unit organizes regular training programmes for students and staff on how to access and use the electronic library resources via the internet and also the local server. the laboratory is divided into two sections: desktop computer lab and laptop computer lab for library users. the laboratory also offers printing and scanning services at affordable prices for the undergraduate students. the final year students were considered for this study because they are expected to do more on writing term papers, in addition to writing their projects in partial fulfilment for the award of their degrees. they therefore have a need for internet resources.   the faculty of natural sciences started during the 1974/75 session, as the faculty of science in the jos campus of the university of ibadan. it moved to its present site at the bauchi road campus during the 1975/76 session. the faculty of natural sciences, university of jos consists of ten academic departments, namely: botany, chemistry, computer science, geology and mining, industrial chemistry, mathematics, microbiology, physics, statistics, and zoology. these departments award first degrees, higher degrees, and diplomas.   it is globally accepted today that the internet has impacted and revolutionised lives, particularly in academic circles. students use the internet for different purposes, including their scholarly activities. although most undergraduate students pride themselves on being computer literate, the fact remains that it is only during examination periods that one observes high numbers of students in the library with few using either their personal computers or the ones provided by the library. this shows that most of the students are either unaware of the internet facility in the library or they are ignorant of its full benefits for their study and research. it is against this backdrop that the researchers wish to establish internet connectivity, access and use among students of the faculty of natural sciences of university of jos, in addition to solving the problems students encounter in using the internet.   literature review   the internet, which is also referred to as the ‘net’, is a collection of computers and computer networks located all over the world, all of which share information established upon internet protocols. therefore, it is an information highway using different computer networks. adegoke (2009) affirms that, “indeed, the internet is also a virtual library which is seen as virtual space containing a vast amount of information and documents including books, pictures, video, graphs and musical sounds that can be consulted.” thus, the internet provides a way of accessing information of all types. reddick and king (2001) agree that ‘clearly, there is a lot of information out there – too much to catalogue. in fact, nobody knows exactly what is available online and where it is. the growth represents both an opportunity and a hazard.... the opportunity is that, sitting at your desk, you can access information that you may not have known been existing [sic]’ (p.74). the risk is that you waste a lot of time looking at information that is not relevant to the projects on which you are working.   the internet is fast changing the methods for accessing and using information among various groups of students. accessing information in the library through the use of catalogue cards is a daunting task. in some cases the information may be available but, due to certain constraints, inaccessible. the internet has introduced new concepts of literature searching and is recasting the roles played by students in the educational process (omotayo, 2010). still, while many undergraduates use the internet, there are many who may not. supporting this view, aguolu and aguolu (2002) averred that information in a library may be available and yet not accessible. this could be due to the student’s ignorance of the library’s internet connectivity and proper access channels.   students’ ignorance and lack of proper access channels could be due to failure of the university’s management to provide adequate education and facilities. there are many surveys on access to the internet, and nearly all found out that internet access is a huge problem among many students. such access problems include below average computer literacy of the students, accessibility to the internet, slow connectivity and downloading, and severe network failure (udende & azeez 2010; fasae & aladeniyi 2012; otunla 2013). indeed, access to current literature is still a problem in africa, as university libraries continue to contend with problems of poor funding, and student unrest.   the use of the internet can be of prime benefit to students. it is very useful by allowing students to access different types of information and also to process this information and communicate it to their peers across the globe (eyitayo, 2008); eyitayo further asserts that the internet has become the most popular way of locating and retrieving information. the internet has become a very useful tool in the library for simplifying information location, retrieval, use, and communication. abubakar and bada (2005) observe that it provides facilities and capabilities to browse through a list of subject headings to get an idea of what is available in other places through the network. internet connectivity can facilitate the work of reference librarians to answer queries on areas in which they lack prior knowledge, and such information is often very current and up to date.   on the use of the internet in nigerian universities among students, jagboro (2003) established that two-thirds of the respondents indicated that they used it for e-mail. just over half used it to get research materials while 39.73% used it to retrieve course materials. she attributed the low level of internet use to low level of connectivity and high cost of cybercafé facilities. in addition, hanauer, dibble, fortin, and col (2004) surveyed a diverse community college to assess the use of the internet by the students of health-related information. the survey showed that 83% of internet users had access to the internet at their home and 51% of the respondents accessed the internet at the college or library.   the internet is very useful to university students in nigeria because it enables them to have access to timely, accurate, and relevant information that cannot be obtained from library shelves. bankole and oludayo (2012) note that the majority (86%) of their respondents (undergraduates) from olabisi onabanjo university, ago-iwoye, nigeria use the internet to search for information. mishra (2009) studied the use of internet at the university of maiduguri, nigeria. the findings show that the internet was very important for 60.8% of the respondents, with 74.6% using the internet for research. google was mentioned by 71.5% of participants as their preferred search engine and mishra concludes that necessary facilities should be put in place for faculty and students to make optimal use of information resources available on the internet. emphasizing the advantages of internet resources, dadzie (2005) writes that it is an invaluable research tool that complements the print-based resources in a traditional library setting. the advantages, according to her, include: access to information that might be restricted to the user due to geographical location or finances; access to more current information; and provision of extensive links to additional resources related with contents.   the visit to and use of any university library is dependent on so many variables; these include the resources provided or made available in the library, the environmental conditions, and even the student’s self-efficacy (waldman, 2003). if the resources are not sufficient for the needs of the users of the library, it is not likely that the library will be visited. by and large the accepted functions of a university, according to ifidon (1999), are to cater for the teaching of the students and for the research interests of the faculties. therefore, the purpose of any university library is to provide such resources, in the form of recorded knowledge, as are necessary for both teaching and research in the university. if such are provided, it is most likely that students would visit and use the library. the visit to the library by students could stimulate them to use the available internet resources. soria, fransen, and nackerud (2013) measured and published the impact of library visit and use on students’ academic success in the university of minnesota, usa. the study found that there are statistically relevant data showing first-year undergraduate students who visit and use the library’s resources (internet) have a higher grade point average for their first semester and higher retention rate from fall to spring than those who hardly visit the library. this indicates that there is a relationship between library visit and use of its resources such as the internet. this invariably promotes the student’s academic performance.   a reduction in the usage of conventional library services at the university level suggests that students are looking somewhere else for information resources. the association of research libraries (arl) in 2005 reported that between 1991 and 2005, reference requests dropped by an average of 4.5% per year, and book checkouts fell by 1.2% per year, though this varied depending on the type of academic library (applegate, 2008). the arl also documented a decline in reference requests and borrowing between libraries from 2009-2011 (arl, 2011). request and borrowing of library materials is a function of the visit of the students to the library. mcdonough and jimenez (2007, p.171) reported from the university of illinois and   observed that in terms of undergraduate usage of the library, “they are losing clientele; students may come in the library to study, to socialize, to strike the newly installed cafe designed to lure them in, but they are not using library materials, or library services, at anything like the rate they did even ten years ago.” this also suggests that the students visit to the library, may influence their use of the facilities (internet). however, such is dependent on the urgent tasks at hand for the students. quadri’s (2013) findings from two private nigerian universities revealed that there was significant relationship between level of study and utilization of online library resources (r= .933; p= 0.05) but that the internet was an important source of information for the undergraduate students.   diem (2013) posited that ‘…the use of libraries is influenced by the existence of important features’ (p. 005) and that these features include the provision of ict facilities. this is the case in indonesia. at penn state university in the u.s., internet access is available at students’ computing labs, residence halls, offices, most classrooms, and off-campus locations. in order to use the full range of internet services students need the penn state access account with other access requirements varying depending on location (penn state university, 2012).   internet access and its use had its own share of problems. according to olalude (2007) one of these is the initial capital outlay to install internet facilities. the reason for this is because most african countries are experiencing huge debts and foreign exchange problems. therefore, to purchase such facilities becomes a huge problem. the consequence of this is the complete lack of these facilities in many african university libraries. in addition, emejo (2009) reported that about 70 percent of nigeria’s internet capacity has been disrupted following a reported damage to one of the landing cables of sat3 submarine system. he expressed the opinion that this was the cause of the network outage being experienced by large number of internet subscribers in the country. in a study of internet access by students of faculty of sciences in two nigerian universities, fasae & aladeniyi (2012) reported that most of their respondents, (97%) had experienced the problems of slow internet speed followed by lack of skills for surfing the internet. furthermore, israel and edesiri (2014) reported that personal observation has shown that many undergraduate students lacked the skills needed to make effective use of the internet to meet their educational needs.   aims   the primary objective of the study is to investigate the connectivity and accessibility of internet services in the university of jos library among final year students of the faculty of natural sciences.   the specific objectives are:   1.       to find out the frequency of use of computer internet in the library; 2.       to know if the students are aware of internet connectivity in the library; 3.       to investigate if they access the internet in the library; 4.       to assess the purposes for which they use the internet in the library; 5.       to determine the problems they have encountered with internet access in the library.   hypothesis   there is no significant relationship between library visit and use of the internet for all purposes.   methods   a survey design was adopted for this study, and questionnaires were used to gather data. the questionnaires were distributed with the assistance of lecturers to encourage a high return rate. the population for this study was nine hundred and sixty nine (969) final year students of the faculty of natural sciences of the university of jos (academic planning unit of university of jos, 2012). the faculty has ten departments with the number of students in the final year for the 2011/12 session as follows: botany 71, chemistry 62, industrial chemistry 53, geology and mining 143, mathematics 90, computer science 55, statistics 88, microbiology 161, physics 131 and zoology 115.   the sample size was determined by using krejcie and morgan’s (1970) table for sample size.     table 1 demographics of the respondents departments number percent microbiology 34 16.4 mathematics 45 21.7 statistics 23 11.1 geology 3 1.4 chemistry 35 16.9 physics 22 10.6 zoology 19 9.2 computer science 2 1.0 plant science 19 9.2 industrial chemistry 5 2.5 total 207 100       gender     male 124 59.9 female 77 37.2 missing 6 2.9 total 207 100       age     15-19 10 4.8 20-24 95 45.9 25-29 86 41.5 30-34 10 4.8 missing 6 2.9 total 207 100     according to krejcie and morgan (1970) a population of 1000 should have a sample of 278. therefore, this sample (278) was considered appropriate for the population of 969. in addition, proportionate sampling was done to determine the sample for each department in the faculty, thus: botany 20, chemistry 18, industrial chemistry 15, geology and mining 41, mathematics 26, computer science 16, statistics 25, microbiology 46, physics 38, and zoology 33. a response rate of 207 constituting 74% was adequate because yamane (1967) stated that a response rate of 60% is adequate for making generalisation for any given population. statistics used include percentages, frequencies and chi –square statistics to test the association of the dependency of internet usage on the visits the students made to the library. furthermore, it was used because the variables in question were categorical.   results and discussion   table 1 shows the demographics of the respondents based on departments, gender, and age.   table 1 indicates that the mathematics department had most of the students in the study (21.7%), and computer science had the lowest (1%). most of the students in the study were males (59.9%). this indicates a ratio of about 60% to 40% intake for males and female students. most of the students’ ages ranged from 25 to 29 years, with an average of 27 years, and this constitutes 41.5%. the age range is normal because it is within the bracket years nigerian students are engaged in the national service. this is the accepted age at which students graduate from the universities and are engaged in free national service for their fatherland.   table 2 indicates the frequency of use of the computer and internet by the students in the library.   table 2 shows that only 15.5% of the students were using the computer and internet in the library daily, while 33.8% of them used it less than once month. perhaps these few 15.5% of the students that use the computer and the internet in the library are the ones that own personal laptops and/ or the ones that use the library’s computers which are connected to the internet. this result is contrary to the study by awoleye, siyanbola, and oladipupo (2008) which revealed that about 92% of undergraduate students have embraced the internet and are using it consistently. what is interesting, however, is that the library provides internet and computer services in all the branch libraries that are permitted to be used by all students for an interval or period of one hour due to limited computer/access points. most of the students visit and use the library once a week (36.2%) and only (14%) visit and use the library once a month. this percentage is close to those that use the library’s internet facility. it thus implies that only few of the final year students visit and also use the library’s internet. the implication of this is that the library as an information provision centre will not achieve its objective and as a result will be a colossal and wasteful investment.     table 2 use of computer and the internet in the library       item: number percent 1. i use computer and internet in unijos library       daily 32 15.5   once a week 48 23.2   once a month 45 21.7   less than once a month 70 33.8   missing 12 5.8   total 207 100 2. i visit and use the library       daily 44 21.3   once a week 75 36.2   once a month 29 14.0   less than once a month 53 25.6   never 1 0.5   missing 5 2.4   total 207 100     table 3 internet connectivity in the library   items number percent 1. there are computers with internet connectivity in the library       yes 163 78.7   no 39 18.8   missing 5 2.4   total 207 100 2. i always seek permission to use the computer internet facilities       yes 120 58.0   no 81 39.1   missing 6 2.9   total 207 100 3. i own a wireless laptop computer and connect with the library’s internet facility to use       yes 94 45.4   no 108 52.2   missing 5 2.4   total 207 100 4. i desire to acquire a laptop computer in the near future       yes 145 70.0   no 43 20.8   missing 19 9.2   total 207 100     table 3 presents 78.7% of the students indicating that there are computers with internet connectivity in the library and 58% take permission before they use it. only 45.4% used their own wireless laptops to connect with the library’s internet. the majority of the respondents affirmed that there is internet connectivity in the library. this revelation is not in consonance with the submission of baro and asaba (2010) who averred from their study of internet connectivity in university libraries in nigeria that most university libraries in nigeria lacked internet connectivity, especially the private university libraries, and that is why they  failed the national university commission’s (nuc) accreditation most of the time. up to 58% of the students will always seek permission before they use the internet connectivity, it could be those that own their personal laptops. the reason why students sought permission is because the library’s rules and regulations state that: before using any system, permission needs to be given by the system managers to avoid system break down and other misuse by students such as visiting pornographic sites. the majority of the students (78.7%) testified to the availability of internet connectivity in the library. however, only few (15.5%) used it daily for their normal academic activities. this shows that the students need to be re-orientated and re-sensitised on the need to use these facilities for academic work. part of such orientation is normally given during the use-of-library class at the point of entry into the university and subsequently in their research classes.   table 4 below presents the platforms through which students accessed the internet.   the faculty students accessed the internet mostly through their gsm phone (72%), followed by private cybercafés in town (68.6%) and the library (63.3%) (table 4). access to the internet is least through their departments (20.5 %). while the majority of the students confirmed the presence of internet connectivity in the library, the majority accessed the internet via their mobile phones. this could be the reason for the low library visits and consequent low use of the library’s internet connectivity. apprehension could be said to be another factor to the low access to the internet by the students in the library. one clear reason that could be speculated is that the students don’t want to waste their time on the queue to take their turns, and a good number of them prefer the cybercafé in the town where they waste little or no time at all. this result is in tune with that of ani (2010) who reported on access to the internet in three nigerian universities; that there is inequitable access to the internet, as students mostly rely on private/commercial internet cybercafés both on and off campuses for their access and use. access to the internet in the university libraries, departments/faculties, and university computer/ict centres is very poor. this pattern of accessing the internet outside the campus could have time implication on the part of the students; the time taken to visit the cafés in town would have been used for positive study, and also may have conserve time wasted occasionally on missing lectures.     table 4 access to the internet   items number percent 1. i access the internet in the library via its server       yes 131 63.8   no   70 33.3   missing     6   2.9   total 207 100 2. i access the internet in my faculty       yes          61 29.5   no 141 68.1   missing     5   2.4   total 207 100 3. i access the internet in my department       yes   43 20.8   no 154 74.4   missing   10   4.8   total 207 100 4. through subscription to mis wireless access       yes   83 40.1   no 117 56.5   missing     7   3.4   total 207 100 5. through gsm phone services       yes 149 72.0   no   52 25.1   missing     6   2.9   total 207 100 6. through private cybercafés on campus       yes 100 48.3   no   96 46.4   missing    11    5.3   total 207 100 7. through private cybercafés in town       yes 142 68.6   no   55 26.6   missing   10   4.8   total 207 100     table 5 shows the frequency of the use of the internet by the students.   table 5, indicates that the frequency of use of the internet is mostly occasional for all purposes, especially for the use of e-mail (45.9%).over seven percent (7.7%) frequently use the internet for chatting on a daily basis. this finding corresponds with lubans (1999) who found that a majority of respondents used the internet from “several times a week” to “often.” however, it is not in agreement with the findings of ani’s (2010) study on internet access and use by undergraduate students of nigerian universities that the internet is extensively used by undergraduate students. the most prominent uses of the internet among the students as reported by ani (2010) include the use for e-mail and downloading articles from online journals; however, such uses of the internet are of low frequencies from the current study (4.8% and 3.4% respectively on a daily basis). this scenario might have negative effects on students’ overall academic endeavour, which could restrict sharing of ideas with their peers and also their lecturers.     table 5 frequency and use of the internet in the library   use for: number percent 1. e-mail       never 76 36.7   occasionally 95 45.9   once a week 20 9.7   daily 10 4.8   missing 6 2.9   total 207 100 2. receiving e-newsletters        never 137 66.2   occasionally 48 23.2   once a week 14 6.8   daily 6 2.9   missing 2 2.1   total 207 100 3. downloading articles from online journals       never 106 51.2   occasionally 77 37.2   once a week 16 7.7   daily 7 3.4   missing 1 0.5   total 207 100 4. internet phone calls       never 134 64.7   occasionally 48 23.2   once a week 6 2.9   daily 15 7.2   missing 4 1.9   total 207 100 5. chatting       never 120 58.0   occasionally 52 25.1   once a week 13 6.3   daily 16 7.7   missing 6 2.9   total 207 100 6. listening to live radio/tv broadcasts       never 139 67.1   occasionally 41 19.8   once a week 8 3.9   daily 15 7.2   missing 4 1.9   total 207 100 7. finding romantic partners       never 162 78.3   occasionally 25 12.1   once a week 7 3.4   daily 10 4.8   missing 3 1.4   total 207 100 8. reading e-newspapers         never 135 65.2   occasionally 43 20.8   once a week 13 6.3   daily 12 5.8   missing 4 1.9   total 207 100 9. watching videos and films         never 142 68.6   occasionally 42 20.3   once a week 4 1.9   daily 12 5.8   missing 7 3.4   total 207 100         table 6 indicates the problems of internet access encountered by the students.   table 6 presents slowness of the internet at peak periods as the major problem of internet access (81.6%) followed by unsteady power supply (56.0%). however, just over one third of participants (35.2 %) attested that poor knowledge of computer skills was their problem. although the frequency of use of the internet is low for all purposes, such problems indicated by the final year students could affect the degree of use of the internet. the slowness of the internet at peak period is bound to the issue of bandwidth. this situation is in agreement with the report of womboh and abba (2008) who emphasized that nigeria, which had an information technology mission statement of its intention to become an it capable country in africa and a key player in the information society by the year 2005, had not achieved this as at 2008. they noted further that the number and quality of computer literate librarians to train the students were not adequate; this perhaps accounted for the 35.2% of the students in the current study who noted poor computer knowledge as their problem. bankole and oludayo’s (2012) findings also show that slowness of the internet at peak periods is one of the major constraints facing internet usage among olabisi onabanjo undergraduate students. this trends will prohibit and demoralise the students in their scholarly activities with consequent low performance academically.     table 6 problems of internet access   problem number percent 1. unsteady power supply       yes 116 56.0   no 77 37.2   missing 14 6.8   total 207 100 2. slowness of the internet at peak periods        yes 169 81.6   no 37 17.9   missing 1 0.5   total 207 100 3. insufficient bandwidth for effectiveness       yes 109 52.7   no 95 45.9   missing 3 1.4   total 207 100 4. space constraints in dept/faculty       yes 133 64.3   no 74 35.7   missing   total 207 100 5. space constraints at cybercafés       yes 99 47.8   no 108 52.2   missing   total 207 100 6. insufficient work stations       yes 103 49.8   no 103 49.8   missing 1 0.5   total 207 100 7. poor knowledge of internet surfing       yes 75 36.2   no 132 63.8   missing   total 207 100 8. poor knowledge of computer skills       yes 73 35.2   no 132 63.8   missing 2 1.0   total 207 100 9. unreadiness of the university library’s computer labs       yes 112 54.1   no 89 43.0   missing 6 2.9   total 207 100 10. insufficient workers to attend to customers at the cafés       yes 89 43.0   no 115 55.6   missing 3 1.4   total 207 100 11. virus contamination of saving devices       yes 100 48.3   no 99 47.8   missing 8 3.9   total 207 100     from table 7, there is no significant relationship between library visits of the students and use of the internet for most of the purposes (six), except for two activities. these are downloading articles from online journals and finding romantic partners. therefore, the null hypothesis is accepted for the six purposes. this relationship shows that these six purposes for using the internet were not influenced by the students’ visits to the library. however, there does exist a significant relationship between visiting the library and downloading articles from online journals or finding romantic partners (p. value =0.008 and 0.015 respectively). this invariable suggests that as the students are mindful of downloading relevant online journals pertinent to their studies; they are also mindful of their social life style. this is expected of final year undergraduate students they are of course adults.   conclusion   it is the conclusion of this research that internet connectivity was provided in the university of jos for faculty of natural sciences students to access and use in their academic pursuit. the study showed that only a few of the final year students use the library’s internet facility when visiting the library. this means usage of internet when visiting the library among the faculty students was extremely low. despite this, there exists a significant relationship between their visit and usage of the internet in downloading articles pertinent to their study. it is therefore recommended that the librarians in conjunction with faculty staff should sensitise and train students on accessing and using the internet for better academic performance and life-long learning, as usage of the internet by respondents was limited only to two purposes.     references   abubakar, d. & bada, b. a. (2005). application of internet facilities for research and teaching: university of jos academic experience. lagos journal of interdisciplinary studies (lajis) 3, 157-172.   academic planning unit of university of jos (2012). retrieved 23 sept. 2015 from http://www.unijos.edu.ng/academic_planning_and_management/    adegoke, l. (2009). introduction to public relations: principles, media and methods. amsterdam: homelab multiventures.   adika, g. (2003). internet use among faculty members of universities in ghana. library review. 52 (1), 22-37. doi: http://dx.doi.org/10.1108/00242530310456997   aguolu, c. c. & aguolu, i. e. (2002). libraries and information management in nigeria: seminal essays on theses and problems. maiduguri: ed-linform services.   ani, o. e. (2010). internet access and use: a study of undergraduate students in three nigerian universities. the electronic library 28(4), 555-567. retrieved 23 nov. 2014 from http://www.emeraldinsight.com/loi/el   applegate, r. (2008). whose decline? which academic libraries are "deserted" in terms of reference transactions? reference and user services quarterly 48(2), 176–189. doi: http://dx.doi.org/10.5860/rusq.48n2.176   association of research libraries (arl): empowering the research library community. (2011). arl statistics 2011-2012. retrieved 1 nov. 2015 from http://publications,arl.org.arl-statistics   awoleye, m. o., siyanbola, w. o. & oladipupo, o. f. (2008). adoption assessment of internet usage amongst undergraduates in nigeria universitiesa case study approach. journal of technology management and innovation, 3(1). retrieved 14 may 2013 from http://www.jotmi.org   bankole, o. m. & oludayo, b. s. (2012). internet use among undergraduate students of olabisi onabanjo university, ago iwoye, nigeria. library philosophy and practice. retrieved 15 oct. 2012 from http://digitalcommons.unl.edu/libphilprac/   baro, e. e. & asaba, j. o. (2010). internet connectivity in university libraries in nigeria: the present state. library hi tech news: 27(9/10), 13-19. doi: http://dx.doi.org/10.1108/07419051011110603    dadzie, p. s. (2005). electronic resources: access and usage at ashesi university college, campus-wide information systems 22(5), 290-297. doi: http://dx.doi.org/10.1108/10650740510632208   diem, c. d. (2012). how the presence of a technologically supported library influences high school students’ reading habits and skills. global advanced research journal of library, information and archival studies. 1(1), 001-005. retrieved 17 sept. 2015 from http://garj.org/garjlias/home   emejo, j. (2009, july). fibre cable cut affects 70% of nigerians’ internet capacity. thisday.   eyitayo, o. t. (2008). internet facilities and the status of africa’s connectivity. in l. o. aina, s. m. mutula & m. a. tiamiyu (eds.), information and knowledge management in the digital age: concepts, technologies and african perspectives. (pp. 2-41). ibadan: third world information service.   fasae, j. k. & aladeniyi, f. r. (2012). internet use by students of faculty of science in two nigerian universities. library philosophy and practice. retrieved 25 sept. 2015 from http://digitalcommons.unl.edu/libphilprac/   hanauer, d., dibble, e., fortin, j., & col, n. f. (2004). internet use among community college students: implications in designing health care interventions. journal of american college health 52(5):197-202. doi: http://dx.doi.org/10.3200/jach.52.5.197-202    ifidon, s. e. (1999). essentials of african university library management. lagos: the national library press. ltd.   israel, o. & edesiri, e. (2014). ict skills and internet usage among library and information science students in delta and edo states, nigeria. international journal of library and information science, 6(5), 98-107. retrieved 25 sept. 2015 from http://www.academicjournals.org/journal/ijlis   jagboro, k. (2003). a study of internet usage in nigerian universities: a case study of obafemi awolowo university, ile-ife, nigeria. first monday 8(2-3). doi: http://dx.doi.org/10.5210/fm.v8i2.1033    kindilchie, a. i. & samarraie, i. f. (2008). interaction and impact of electronic information resources on qatar university faculty. libri. 58(4), 281 – 293. doi: http://dx.doi.org/10.1515/libr.2008.028   krejcie, r. v. & morgan, d. w., (1967). determining sample size for research activities. educational and psychological measurement, 30(3), 607-610.   lubans, j. (1999). key findings on internet use among students. retrieved 15 apr. 2012 from http://www.lubans.org/    mcdonough, j & jimenez, m. (2007). video preservation and digital reformatting: pain and possibility. journal of archival organization. 4(1-2).167-191   mishra, m. k. (2009). use and importance of the internet at the university of maiduguri, nigeria. first monday. 14(3). doi: http://dx.doi.org/10.5210/fm.v14i3.2301    okonofua, e. f. (2008). university research and the challenge to national development. benin city: faculty of pharmacy, university of benin.   olalude, f. o. (2007). utilization of internet sources for research by information professionals in sub-saharan africa. african journal of library, archives, and information science 17(1): 5358. retrieved 26 sept. 2015 from http://www.ajol.info/index.php/ajlais   omotayo, b. o. (2006). a survey of internet access and usage among undergraduates in an african university. the international information & library review, 38(4), 215–224. doi: http://dx.doi.org/10.1016/j.iilr.2003.10.016    otunla, a. o. (2013). internet access and use among undergraduate students of bowen university iwo, osun state, nigeria. library philosophy and practice. retrieved 25 sept. 2015 from http://digitalcommons.unl.edu/libphilprac/   penn state university (2012). information technology resources and services at penn state. retrieved 26 sept. 2015 from http://www.psu.edu/dept/itscss/internet/connections.html   quadri, g. o. (2013). influence of demographic factors on use of online library resources by undergraduate students in two private nigerian university libraries. library philosophy and practice. retrieved 5 apr. 2015 from: http://digitalcommons.unl.edu/libphilprac/   reddick, r. & king, e. (2001). the online journalist: using the internet and other electronic resources. (3rd ed.) fort worth, tx: harcourt college publishers.   soria, k., fransen, j. & nackerud, s. (2013). library use and undergraduate student outcomes: new evidence for students’ retention and academic success. portal: libraries and the academy. doi: http://dx.doi.org/10.1353/pla.2013.0010.   udende, p. & azeez, a. l. (2010). internet access and use among students of the university of ilorin, nigeria. journal of communication and media research. 2(1) 33-42. retrieved 25 june 2015 from http://unilorin.edu.ng/publications/udendep/   university of jos library (2015). retrieved 30 sept. 2015 from: http://www.unijos.edu.ng/library/index.php   waldman, m. (2003). freshmen’s use of library electronic resources and self-efficacy. information research. 8(2). retrieved 5 july 2015 from http://www.informationr.net/ir/8-2/paper150.html   womboh, b. s. h., & abba, t. (2008) the state of information and communication technology (ict) in nigerian university libraries: the experience of ibrahim babangida library, federal university of technology, yola. library philosophy and practice. retrieved 30 sept. 2015 from http://digitalcommons.unl.edu/libphilprac/   yamane, t. (1967). statistics: an introductory analysis. 2nd ed., new york: harper and row.     ebl 101   research methods: sampling   virginia wilson librarian, murray library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   received: 16 june 2014  accepted: 19 june 2014      2014 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   research often involves choosing a certain number of people or items in order to answer a question. a frequent question is how does one choose a sample? another question is how many individuals or items is enough? what sample size will give the best results for the question at hand? well, the answer to the latter is: “it depends”. it depends on the question, on the method one uses to administer the research project, on what kind of results one is hoping for.   sampling is a consideration in both qualitative and quantitative research. survey methodology, interviews, focus groups, bibliometrics, content analysis, usability testing, etc., all rely on an appropriate number of people or items being selected and examined. for the purposes of this brief column, i’ll look at sampling as it pertains to survey methodology, as much of this information can be applied to other types of research methodologies. a valid sample must be considered in order to obtain generalizability in quantitative research and trustworthiness in qualitative research.   there are various types of sampling methods, including nonprobability sampling and probability sampling. below is a very brief examination of the methods under each, adapted from basic research methods for librarians, 5th ed. (connaway & powel, 2010). sampling is a complex exercise, depending on the type. as usual, the brevity of this column necessitates only the briefest overview of the topics.   nonprobability sampling: the researcher cannot be sure of a “specific element of the population [i.e. the particular grouping that is being looked at] being included in the sample” (p. 117). the weakness of a nonprobability sample is that it “does not permit generalizing from the sample to the population because the researcher has no reassurance that the sample is representative of the population” (p. 117). still, these types of samples are easier and often cheaper to obtain than the alternative (which we will get to later), and they can be adequate depending upon the research question.   there are various types of nonprobability samples:   •     accidental (or convenience) sampling: selecting the cases at hand until the desired number of people/items is reached. •     quota sample: the same as accidental sampling, except that “it takes steps to ensure that the significant, diverse elements of the population are included” (p. 118). •     snowball sample: a cumulative sample is generated by starting with a few people and asking them to recommend more people. •     purposive sample: based on the researcher’s “knowledge of the population and the objectives of the research” (p. 119). •     self-selected sample: people self-identify with the desired population criteria and select themselves to participate in the study.   probability sampling: this type of sampling comes closer to the objective of sampling, that is, “to select elements that accurately represent the total population from which the elements were drawn” (p. 119). the critical piece in probability sampling is that “every element in the population has a known probability of being included in the sample” (pp. 119-120).   there are various types of probability samples:   •     simple random sampling: this is the basic sampling method in survey research and it “gives each element in the population an equal chance of being included in the sample” (p. 120). the simple random sample is generated most often by using a table of random numbers. there are variations of the random sample, differentiated by the way the samples are generated. ø  systematic sample: involves “taking the every nth element from a list until the total list has been sampled” (p. 123). ø  stratified random sample:  the population elements are divided into categories, then independent random samples are selected from each category. ø  cluster sample: the population (not the population’s elements as in stratified random sampling) are divided up into clusters and samples are drawn from the clusters. this is particularly helpful when a population cannot be easily listed for sampling purposes.   this has been a whirlwind trip through types of sampling, as the other main point i would like to address is the “how many” question. how many people/items are enough to be representative of any given population? the rule of thumb for sample sizes is the larger the better. however, time, funding, and a host of other issues also have a role to play in determining how big to go. connaway and powell state that there are four criteria that you can think about to help determine the necessary sample size:   1.     the degree of precision required (the less accuracy needed, the smaller the sample you can get away with) 2.     the variability of the population (the greater the variability, the larger the sample size) 3.     the method of sampling (i.e. “stratified sampling requires fewer cases to achieve a specified degree of accuracy” (p. 129). 4.     how the results are to be analyzed (small samples have limitations in terms of the types of statistical analyses that can be used)   there are formulas that can be used to determine the ideal number. luckily, for the mathematically challenged (like me) there are tables and calculators that researchers can use that already have the formulas applied:   ·         table: http://www.research-advisors.com/tools/samplesize.htm   ·         a sample size calculator: http://www.raosoft.com/samplesize.html ·         calculator: http://www.surveysystem.com/sscalc.htm ·         simple random sample calculator: http://www.nss.gov.au/nss/home.nsf/pages/sample+size+calculator     other resources related to sampling:   beck, s.e. & manuel, k. (2008). practical research methods for librarians and information professionals. new york: neal-schuman. (includes sampling across a variety of research methods).   bouma, g. d., ling, r., & wilkinson, l. (2009). the research process, canadian edition. don mills, on: oxford university press. (includes a chapter on selecting a sample and a table of random numbers).   cochran, w.g. (1977). sampling techniques. 3rd ed. new york: wiley.   kish, l. (1995). survey sampling. new york: wiley.     references   connaway, l.s. & powell, r.r. (2010). basic research methods for librarians, 5th ed. santa barbara, ca: libraries unlimited.   evidence based library and information practice evidence based library and information practice 2011, 6.2 48 evidence based library and information practice evidence summary undergraduate science students are uncertain of how to find facts in e-books compared to print books a review of: berg, s. a., hoffmann, k., & dawson, d. (2010). not on the same page: undergraduates' information retrieval in electronic and print books. the journal of academic librarianship, 36(6), 518525. reviewed by: christina e. carter associate professor consortium library, university of alaska anchorage anchorage, alaska, united states of america email: trina@uaa.alaska.edu received: 01 mar. 2011 accepted: 21 apr. 2011 2011 carter. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. abstract objective – to observe and compare the strategies that undergraduate science students use to perform information retrieval tasks in ebooks and in print books. design – qualitative analysis, employing a “prompted think-aloud” methodology and thematic analysis. setting – taylor library (serving the faculty of science), university of western ontario, london, ontario, canada. subjects – twenty undergraduate science students (11 females, 9 males) who had completed at least two years of study in faculty of science programs at the university of western ontario. methods – participants for the study were recruited through informational posters in taylor library, science departments, and in undergraduate science classes. participants were assigned fact-finding tasks in e-book and print versions of eight health, computer science, and engineering textbooks and handbooks available in the taylor library. book titles and tasks are included in a table in the study. each student completed four tasks using e-books and four tasks using print books. half of the participants performed tasks in print books first, and half began with tasks in e-books. print books were “preselected” for each participant. the e-books mailto:trina@uaa.alaska.edu� evidence based library and information practice 2011, 6.2 49 were all from the same platform: electronic book library. participants were provided with a laptop computer to access the e-book versions, and a list of questions or facts to locate within each book. following the methodology of cotton & gresty (2006), one researcher prompted students to verbalize actions while performing assigned tasks. a second researcher captured audio and video of the laptop screen as students individually conducted their e-book searches. a third researcher took notes on each session. an exit survey was given to each participant, asking about previous use, knowledge, and attitudes towards e-books. thematic analysis was then used to examine the collected data. main results – researchers identified four major themes from the data with regard to use of print versus e-books: linear/non-linear strategies; tangible/intangible aspects of books; met/unmet expectations; and transferable/nontransferable behaviours. researchers found that participants tended to search print books in a linear fashion, whereas they approached e-books non-linearly. physicality and familiarity with print books helped participants more readily find answers, compared to e-books, where students tried less successfully to mimic techniques used in print books to locate requested information. participants used indexes in print books, versus e-books where they did not quickly identify the e-books as having them. the students expected that the e-books would behave as other web-based/online sources or search engines would (such as google books), and commented that they did not. transferable actions between print and ebooks included developing and using keywords for searching. conclusion – the authors of this study found that student participants did not know how to navigate the e-books presented to them compared with their print counterparts. there was a lack of awareness on the part of participants about e-books in general: the students were unaware that e-books were available through the library catalogue; they did not know that e-books have indexes as print books do; and did not know the differences among platforms offered by the library. all of these facts point to the importance of user education. the authors note the importance of testing of e-book platforms by students, faculty, and librarians prior to committing to purchase particular platforms. the authors note that more research is needed on user interaction with ebooks, how e-books are used to assimilate information, and how groups other than undergraduates search e-books. commentary the library literature gives many examples of think-aloud protocols in usability testing of library websites, databases, online catalogues, and other online library tools. its use here for studying how e-books are searched is notable and very timely (see also hernon, hopper, leach, saunders, & zhang, 2007). this is a qualitative study, yet there are some issues with controls: 12 of the 20 students reported having used e-books in the past, leaving eight whose use of e-books prior to the study is either absent or unknown. should novices versus those with some experience in the use of e-books have been studied separately? the authors don’t explain what a “convenience sample” is, (though it generally refers to a nonprobablistic sample of individuals selected from the population at hand) or precisely and convincingly how they determined that the 20 participants met the “point of saturation” (p. 520). the authors stated that science students are regular users of e-books, and yet none of the students had used the electronic book library platform. pilot testing of the methodology is not described. the rationale both for book titles chosen and the eight retrieval tasks students were asked to conduct is not given. demographics collected (including ages of participants) and exit survey questions and summary of answers are not included. as the authors note, science students are not necessarily representative of undergraduates evidence based library and information practice 2011, 6.2 50 from other disciplines. further research on ebook use should include students from additional disciplines, as well as other groups, such as graduate students. further studies should compare e-book platforms subscribed by libraries with each other, as well as to those available on popular kindle, nook, and ipad devices. the authors believe that study participants expected e-books they searched to perform similarly to google searching, especially with its ubiquitous hypertext linking. the authors do not devote discussion to another likely candidate with which science students would be especially aware and well-versed at searching and using: e-journal articles within aggregate databases and e-journal platforms. the authors address this in their introduction (p. 518), but curiously, do not return to this factor anywhere else in the study. this article has some obvious implications for practice. reference and instruction librarians should carefully point out differences between searching in print books and searching in library e-book platforms and other online content. along with evaluating e-book platforms for titles provided, librarians should carefully examine search capabilities of considered platforms, and continue to press publishers and e-book vendors to develop more transparent and robust search functionality. references cotton, d., & gresty, k. (2006). reflecting on the think-aloud method for evaluating e-learning. british journal of educational technology, 37(1), 45-54. hernon, p., hopper, r., leach, m. r., saunders, l. l., & zhang, j. (2007). ebook use by students: undergraduates in economics, literature, and nursing. the journal of academic librarianship, 33(1), 3-13. / evidence based library and information practice evidence summary   urban public libraries do not yet meet benchmarks for web accessibility by individuals with disabilities   a review of: maatta smith, s. l. (2014). web accessibility assessment of urban public library websites. public library quarterly, 33(3), 187-204. http://dx.doi.org/10.1080/01616846.2014.937207   reviewed by: ann glusker reference/consumer health librarian business, science and technology department the seattle public library seattle, washington, united states of america email: ann.glusker@spl.org   received: 1 mar. 2015    accepted: 7 may 2015      2015 glusker. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine the extent to which urban public libraries in the united states of america provide web sites which are readily accessible to individuals with disabilities with reference to the urban library council’s edge initiative (specifically benchmark 11, “technology inclusiveness”).   design – web site evaluation.   setting – urban public libraries in the united states of america.   subjects – the 127 library systems, which were both members of the urban libraries council at the time of the study and located in the united states of america.   methods – using the “everyday life information seeking” conceptual framework, an assessment of each of the web sites of the purposive sample of public library systems was performed by an online evaluation tool as well as visually and physically to determine web accessibility and, by extension, technology inclusiveness.   main results – the results of the online accessibility evaluation tool revealed that not one of the sites surveyed was free of errors or alerts. contrast errors (related to color combinations), missing alternative text (providing text alternatives for visual elements), and missing form labels (thereby preventing screen readers from performing searches and navigating to results) were the most common problems. results of visual and physical scans revealed that many sites lacked specific links and/or resources for persons with disabilities, as well as noting that the resources available used oblique language and required many clicks to access. in addition, the vast majority neglected to feature links to national resources such as the national library service for the blind and physically handicapped.   conclusions – the web sites of urban public libraries are not yet completely accessible for persons with disabilities. at the very least they need coding fixes and ongoing maintenance to address the kinds of issues found by the online web evaluation tool used. in addition, resources for disabled persons should be prominently and clearly linked and promoted. further research is called for, both in non-urban library systems and in testing a wider range of access technologies. improvement efforts should acknowledge that web design that improves access for persons with disabilities serves the broader community as well.   commentary   there is no question that much remains to be done to make the internet accessible to persons with disabilities (vicente & lópez, 2010; dobransky & hargittai, 2006). this study’s findings concur. using the purposive sample of urban public libraries, which are members of the urban library council, and using edge initiative benchmarks as a touchstone, it employed two methods for evaluating web sites. for this evidence summary, these methodologies were systematically assessed using the critical appraisal checklist by glynn (2006).   the first method used in the study was an online evaluation tool for web sites – the wave tool – which, while not as powerful as expert inspection (lazar et al., 2012), creates a consistent and rigorous assessment approach, and increases the quantifiability of and confidence in the evaluation results. however, there was no information about whether any comparisons were done with other tools, or which other tools might have been considered. the w3c accessibility initiative has a list of 48 tools on its site, with a detailed list of criteria for choosing the appropriate tool, so this could easily have been noted (w3c web accessibility initiative, 2015).   more importantly, there were no detailed criteria mentioned for the second method, a “visual and physical” inspection of web sites. in a similar study examining public libraries in maryland, each home page was examined by five experienced evaluators, working from an explicit set of guidelines which were included in the article (lazar et al., 2012). the study author noted only that the sites were explored for certain features such as ease of use by screen readers, with neither indication of who performed the evaluations, nor of a standard list of features which were explored on each site.   the author openly acknowledges the limitations and lack of generalizability of the study. the purposive sample used covers only 1.5% of libraries (presumably meaning library systems, but this is unclear), and while sites were tested with several operating systems and browsers, further exploration remains to be done. the section on future research is detailed and explicit.   the implications for practice are clear and concrete. there are easy, achievable ways to make sites more accessible, if a library has the will and the funds. librarians understand both their users and the uniqueness of accessed library resources, so they do a better job of ensuring accessibility compared with jurisdictional it staff. improving web sites’ accessibility helps everyone, not just the disabled; it would be useful to hear more about that, and also to have a resource list. also, library users with various disabilities should be consulted for input. as web sites have more and more interactive content, and as they are increasingly accessed on mobile devices, the need for accessibility improvement is ever more urgent.     references   accessibility evaluation resources (2015). w3c web accessibility initiative. retrieved from http://www.w3.org/wai/eval/overview.html   dobransky, k. & hargittai, e. (2006). the disability divide in internet access and use. information, communication and society 9(3), 313–34.                 http://dx.doi.org/10.1080/13691180600751298   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154    lazar, j., wentz, b., akeley, c., almuhim, m., barmoy, s., beavan, p., …yatto, t. (2012). equal access to information? evaluating the accessibility of public library web sites in the state of maryland. in langdon, p., clarkson, j., robinson, p., lazar, j., & heylighen, a. (eds.), designing inclusive systems: designing inclusion for real-world applications (pp. 185-194). london: springer-verlag.    vicente, m. r. & lópez, a.j. (2010). a multidimensional analysis of the disability digital divide: some evidence for internet use. the information society 26 (1), 48–64. http://dx.doi.org/10.1080/01615440903423245     evidence summary   exploring the disconnect between information literacy skills and self-estimates of ability in first-year community college students   a review of: gross, m., & latham, d. (2012). what’s skill got to do with it?: information literacy skills and self-views of ability among first-year college students. journal of the american society for information science and technology, 63(3), 574-583. doi: 10.1002/asi.21681   reviewed by: heather coates digital scholarship & data management librarian iupui university library indianapolis, indiana, united states of america email: hcoates@iupui.edu   received: 27 feb. 2013   accepted: 25 may 2013      2013 coates. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective to explore the relationships between information literacy (il) test scores and self-estimated ability both prior to and after completing the test.   design – information literacy test (ilt) with preand post-test surveys of self-estimated ability.   setting – two community colleges: a small institution in a rural area and a large institution in an urban area.   subjects – first-year community college students enrolled in entry-level english courses.   methods – the authors conducted a replication study of their earlier work using a larger sample from two community colleges. information literacy (il) skills were assessed using the information literacy test (ilt) developed and validated by researchers at james madison university. during the spring and fall semesters of 2009 and 2011, the authors administered in a single session the ilt, pre-, and post-test survey instruments to 580 participants. participants self-selected via sign-up sheet. the first hundred students to sign up per enrollment period were scheduled. participants received incentives for participation, with an additional incentive offered for scoring in the top 15%.   main results – the majority of students at both schools (95% at school 1, 80% at school 2) scored in the below-proficient range on the ilt, a few scored in the proficient range (5% at school 1, 20% at school 2), but no students scored in the advanced range. the mean of the few scores in the proficient range was closer to the below-proficient range (≤65%) than the advanced range (≥90%). for students at both schools, significant differences were found between their self-estimated and actual test score. while students at both schools adjusted their self-estimated scores downward after completing the ilt, post-test self-estimates remained significantly inflated in relation to their test performance. in particular, students scoring in the below-proficient range demonstrated a large and significant gap. the difference between the self-estimated comparisons to peers and actual scores was significant for students from both schools who scored in the below-proficient range. only the proficient students at school 1 were able to accurately estimate their il skill level. most students completed the ilt remaining unaware of their poor performance.   conclusion – the study revealed a significant disconnect between students’ perceptions of their information literacy skills and their actual performance. students scoring in the proficient range demonstrated a stronger post-test correction response than students scoring at below-proficient levels. generally, the authors of the find that the results support the dunning-kruger effect theory that people lacking skills in a particular domain demonstrate a miscalibration between self-estimated and actual skill. specifically, it confirms that this effect occurs in the domain of information literacy.   there is a need for tools to diagnose information literacy competence. most students are unable to self-assess accurately and competency should not be assumed. meeting the needs of this population will be challenging, given that they do not recognize the need for instruction or assistance.     commentary   student self-perception of skill level is a relatively unstudied aspect of information literacy. this expanded replication study contributes to the body of evidence suggesting that students are entering college with inadequate information literacy skills. the comparison of actual performance with students’ self-estimates is a useful contribution to our understanding of information literacy.   in general, the study was executed well and addressed the initial research questions. one concern pertaining to the use of the ilt to estimate information literacy is the exclusion of acrl standard 4, which addresses the use of information. the exclusion of standard 4 may result in inaccurate measurement of student skill level. despite that limitation, the ilt is generally accepted as a valid tool for assessing information literacy.   the procedures section is generally strong with some details missing. more complete description of the preand post-test survey development, data characterization and screening, and rationale for the statistical methods used would be helpful in assessing the validity of the data. some of these details are provided in the findings section rather than in the methods section. the authors fail to state whether an institutional review board reviewed the study and whether informed consent was provided. overall, the results are presented logically and correspond to the initial research questions. the sample size was sufficient for the reported analyses and level of precision. the authors clearly state the primary limitations of the study: 1) a non-random sample of self-selected participants at two community colleges; and, 2) a lack of data on students scoring in the proficient range. thus, the sample may not be representative of community college or university students in general.   this study describes an interesting approach for understanding the role of perceived ability in information literacy instruction. this area of research is exploratory, so immediate implications for practice are few. we can conclude that librarians and faculty should not rely on students’ self-reported ability to guide il instruction. diagnostic tools for identifying students with deficient il skills are necessary so that appropriate instruction can be provided. such diagnostic tools should attempt to address all five acrl standards, particularly standard #4 (“…uses information effectively…”), which is not assessed by the ilt.   additional replication studies carried out at other types of institutions and using a random sample of students are needed for further replication. if this disconnect is present in the general student population, it speaks to the need for integrating information literacy instruction into program curricula, rather than expecting students to self-select for optional il instruction. an approach to engaging students in il learning opportunities might be through a certificate or badge program. some institutions provide certification for skills in particular software applications or programming languages. libraries could provide certification or badges for application of information literacy skills to relevant tasks; these could be included in student portfolios to demonstrate real-world skills.     evidence summary   undergraduate students still experience difficulty interpreting library of congress call numbers   a review of: murphy, j., long, d., & macdonald, j. b. (2013). students' understanding of the library of congress call number system. the reference librarian, 54(2), 103-117. doi:10.1080/02763877.2013.755418   reviewed by: michelle dalton liaison librarian university college dublin dublin, ireland email: michelle.dalton@ucd.ie      received: 04 july 2013   accepted: 18 oct. 2013      2013 dalton. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to explore how undergraduate students interpret library of congress call numbers when trying to locate books.   design – multiple case study.   setting – a public, residential university in illinois, united states of america.   subjects – 11 undergraduate students (10 upper division, 1 freshman; no transfer students included).   methods – a qualitative approach was adopted, with a multiple case study design used to facilitate the collection of data from several sources. students were recruited for the study via convenience and snowball sampling. participants who volunteered were interviewed and requested to complete a task that required them to organize eight call numbers written on index cards in the correct order. interviewees were also asked about any instruction they had received on interpreting call numbers, and their experiences locating materials in other libraries and bookstores. responses were then coded using colours to identify common themes.   main results – the study reported that there was little correlation between the students’ own estimation of their ability to locate materials and their actual performance in the index card test. five students who reported that they could find materials 75-100% of the time performed poorly in the test. of the 11 participants, only 4 ordered the cards correctly, and in 1 such case this was by fortune rather than correct reasoning. of these, three self-reported a high level of confidence in their ability to locate material, whilst one reported that he could only find the material he was looking for approximately half of the time. of the seven students who incorrectly ordered the cards, no two students placed their cards in the same order, indicative that there is no clear pattern in how students misinterpret the numbers. during the interview process, five students stated that they experienced more difficulty locating books in bookstores compared with the library.   conclusion – based on the findings of the study, the authors recommend several interventions which could help students to locate material within the library, namely through improved signage in shelving areas including the listing of subjects and colour-coding, as well as integrating training on understanding call numbers into subject-based instruction. the possibility of using online directional aids such as qr codes and electronic floor maps is also suggested as a strategy to help orient students.     commentary   the study addresses an extremely fundamental and important question in academic libraries: can students easily locate the reading materials they require? indeed, the fact that the study was prompted by responses to a more general user satisfaction survey highlights the pervasiveness of the problem. in spite of this, it is an issue that is often overlooked in instructional design in favour of broader and deeper information literacy skills and behaviours, such as search skills, critical thinking, and personal information management.   as well as highlighting an important basic skill that is critical to successful library use, the study itself appears well-designed and could also be replicated or adapted for use at a local level in other libraries. at one point the authors state that 10 undergraduates participated (p.107); however, in the methodology section it transpires that there were actually 11 participants in total, which is initially somewhat confusing. in this context, a slightly larger sample size would have been welcome, but is understandable given the case study methodology adopted. the index card test is an interesting idea that could be used in the future as a standalone preand post-test activity to measure learning outcomes in information skills training. the authors clearly recognise the limitations of the convenience sampling used, and also provide detailed information regarding the positionality of the researchers involved in the study, and the validity and reliability of the instruments used. it is refreshing and most welcome to see such rigour applied to methodological issues. helpfully, the authors also include the interview script in full as an appendix. however, a clearer presentation of the results in a tabular format alongside the narrative description would make it easier for the reader to see at a glance the performance of each student in the index card test relative to the self-assessment of their ability in finding materials.    the study also raises questions over which classification scheme libraries should adopt, although the broader literature reviewed by the authors indicates that this is not just a problem unique to lcc; a comparative study exploring undergraduates’ interpretation of dewey decimal classification would provide an interesting counterpoint in this context. moreover, the results of this study could be used as a control group to measure the success of future interventions such as improved signage or directional assistance.   most interesting perhaps are the clear implications for information literacy instruction. the authors note that relatively little time is given over to discussing call numbers and print holdings during first-year sessions, with the main focus tending to be on e-resources and library services more generally. it is likely that this is also replicated in many other institutions, particularly as an increasing proportion of library resources are shifting to electronic formats. however, given the findings of the study, perhaps it may still be of value to offer a session dedicated to locating print materials within the library for new undergraduates who find themselves trying to navigate a very unfamiliar terrain. indeed the results of the study serve as a useful reminder that we cannot make any assumptions about our users’ prior knowledge.   we must also remember that user experience is a key component of effective service design, and the evidence here suggests that libraries may still be struggling as a result of frequent “navigation fail points” (hahn & zitron, 2011, p. 31). indeed difficulties using catalogues and shelfmarks have been encountered by users and reported on in the literature for many years now. that the problem still persists today may suggest an inability to find more creative solutions for our users in how they can locate materials. improving user experience has real implications for our signage, instructional design, layout, and indeed the classification systems and ways that we organize our collections. it also offers opportunities for new tools and technologies such as augmented reality applications, which can provide maps, floorplans and navigational directions to specific collections, or even key core texts, direct to a user’s mobile device at the point of need.     references   hahn, j. & zitron, l. (2011). how first-year students navigate the stacks: implications for improving wayfinding. reference & user services quarterly, 51(1): 28–35. doi:10.5860/rusq.51n1.28     evidence based library and information practice commentary   moving the eblip community’s research agenda forward   jonathan d. eldredge associate professor biomedical informatics research, training and scholarship health sciences library and informatics center/family and community medicine albuquerque, new mexico, united states of america email: jeldredge@unm.edu     heather n. holmes clinical informationist summa health system akron city & st. thomas hospitals instructor, department of internal medicine northeastern ohio medical university akron, ohio, united states of america email: holmesh@summahealth.org   marie t. ascher lillian hetrick huber endowed library director health sciences library new york medical college valhalla, new york, united states of america email: marie_ascher@nymc.edu   received: 20 mar. 2015  accepted: 14 may 2015      2015 eldredge, holmes, and ascher. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   four times a year a new issue of eblip appears on the scene. each issue brims with evidence summaries, original research reports, and commentaries. the eight editors, nine copyeditors, nearly 100 regular peer reviewers, and numerous authors contribute voluntarily to the production of this peer reviewed, open access journal. in any given year there are thousands of downloads from the eblip site, attesting to the wide use of eblip.   participation in the production and use of eblip spans six continents and includes every type of library or information practitioner. many of those either publishing in eblip or benefitting from its contents have never met one another. only a fraction of those colleagues loosely affiliated with eblip have participated in the international biannual eblip conferences. these small conferences, with only a couple of hundred attendees, provide face-to-face contacts that build a common sense of purpose, although those contacts might translate into some ongoing or future associations with our eblip journal.   our profession’s version of evidence based practice (ebp) as an interest area and decision-making method might be thought of as a movement (glynn, 2007; koufogiannakis, 2012; lewis, 2011). a movement is an important concept for understanding the place of ebp in fostering professional accountability, which seems to be an underlying theme in the great interest expressed in ebp by diverse professions (eldredge, 2014). yet, a movement is an abstract concept. the conferences are more tangible, but these occur at only two-year intervals and are attended by only a minority of interested practitioners.   that leaves the eblip journal as the most tangible forum for colleagues with an interest in ebp for our profession. on the surface, eblip might be a journal; it also represents a mostly virtual community that shares the goal of making ebp a firmly established reality. the authors define our profession’s version of ebp, namely evidence based library and information practice, in the following way:   a sequential, structured process for integrating the best available evidence into making important decisions. the practitioner applies this decision making process by using the best available evidence while informed by a pragmatic perspective developed from working in the field, critical thinking skills, an awareness of different research designs, and modulated by knowledge of the affected user population’s values or preferences. (eldredge, 2012, p. 139)   producers and consumers affiliated with the journal might constitute what wenger (1998) describes as a “community of practice.” when one reads wenger closely, it quickly becomes apparent that those colleagues associated with our eblip journal resemble a community of practice. wenger notes that a community of practice fosters dynamic learning through professional practice. eblip certainly advances that goal.   over the last decade our journal as well as our conference planners clearly have resisted institutionalization. wenger recognizes that “institutionalization consumes energy. it requires continual maintenance” (p. 243). to the extent that any institutionalization does exists, wenger reports that it should exist solely to serve the community of practice. provided that we might even define our journal as an “institution” in wenger’s sense of the term, eblip certainly subsumes itself to the goal of applying research evidence in practice.   how should we define our future direction as a community of practice? perhaps a research agenda might inform where we devote much of our energy as a community? fortunately, some members of our community of practice have experience with defining a research agenda. a team of librarians in sweden conducted a delphi study to define the research agenda for their colleagues in that country (maceviciute & wilson, 2009). unaware of our swedish colleagues’ efforts, the same year members of our team in the united states conducted a different type of delphi study to define a research agenda for the medical library association (eldredge, harris, & ascher, 2009). a delphi study allows members of a group who hold diverse views to reach a satisfactory group consensus. in the u.s. study we focused upon generating important and answerable research questions. our experiences matched the observations of others regarding the question formulation process (booth, 2006; kloda, 2008; sutton, a., booth, a. & evans, p., 2013). we believed earnestly that once we had defined the research agenda, applied researchers would implement an action plan to answer its practical research questions. instead, we were bewildered by the apparent inaction among researchers to answer these top-ranked research questions.   after conducting a second delphi study (eldredge, ascher, holmes, & harris, 2012), we tried to find a way to implement the new research agenda. we developed the idea of linking each of the 15 top-ranked research questions to a systematic review. this new dimension would help build the growing body of systematic reviews in library and information science (koufogiannakis, 2012). we were happily surprised when over 200 colleagues, a third of them from outside the united states and some even outside health sciences libraries, volunteered for our 15 systematic review teams. we reported on this project at eblip7 in saskatoon, saskatchewan (eldredge, ascher, & holmes, 2013), and we will be reporting on the project during eblip8 in brisbane this july. we will be happy to answer your specific questions in brisbane or via email.   first, however, our community of practice needs to determine if it wants to define a research agenda. second, it needs to define the scope of the project. will such a project include all registered users of eblip supplemented with the attendees at recent eblip conferences? third, it has to define the methodology. we employed the delphi method because it allowed for electronically assisted asynchronous consensus formation. while we think that this methodology to be best suited to such a research inquiry, we all need to be open to other methodologies (brettle, 2012). once a research agenda takes shape, will systematic reviews be the next best step?   we the authors have experience with all three aspects of such a project and are willing to serve as resource persons for any project team with an interest. finally, we are happy to serve more tangentially as an information-sharing clearinghouse to help potentially interested colleagues to organize a possible project.   references   booth, a. (2006). clear and present questions: formulating questions for evidence based practice. library hi tech, 24(3), 355-368. http://dx.doi.org/10.1108/07378830610692127   brettle, a. (2012). learning from others about research evidence. evidence based library and information practice, 7(2), 1-3. http://ejournals.library.ualberta.ca/index.php/eblip   eldredge, j. d. (2012). the evolution of evidence based library and information practice, part i: defining eblip. evidence based library and information practice, 7(4), 139-145. http://ejournals.library.ualberta.ca/index.php/eblip   eldredge, j. d. (2014). the evolution of evidence based library and information practice, part iii: revitalizing the profession through eblip. evidence based library and information practice, 9(1), 62-73. http://ejournals.library.ualberta.ca/index.php/eblip   eldredge, j. d., harris, m. r., & ascher, m.t. (2009). defining the medical library association research agenda: methodology and final results from a consensus process. journal of the medical library association, 97(3), 178-185. http://dx.doi.org/10.3163/1536-5050.97.3.006   eldredge, j. d., ascher m. t., holmes, h. n., & harris, m. r. (2012). the new medical library association research agenda: final results from a three-phase delphi study. journal of the medical library association, 100(3), 214-218. http://dx.doi.org/10.3163/1536-5050.100.3.012   eldredge, j. d., ascher m. t., & holmes, h. n. (2013, july 17). implementing a national eblip research agenda. poster presented at the 7th international evidence based library and information practice conference (eblip7), saskatoon, saskatchewan.       glynn, l. (2007). defending evidence based practice. evidence based library and information practice, 2(2), 1-2. http://ejournals.library.ualberta.ca/index.php/eblip   kloda, l. (2008). asking the right question. evidence based library and information practice, 3(4), 79-81. http://ejournals.library.ualberta.ca/index.php/eblip   koufogiannakis, d. (2012). the state of systematic reviews in library and information studies. evidence based library and information practice, 7(2), 91-95. http://ejournals.library.ualberta.ca/index.php/eblip   lewis, s. (2011). evidence based library and information practice in australia: defining skills and knowledge. health information and libraries journal, 28 (2), 152-155. http://dx.doi.org/10.1111/j.1471-1842.2011.00937.x   maceviciute, e. & wilson, t. d. (2009). a delphi investigation into the research needs in swedish librarianship. information research, 14 (4), paper 419. http://www.informationr.net/ir/index.html   sutton, a., booth, a. & evans, p. (2013). “ask, acquire, appraise”: a study of lis practitioners participating in an eblip continuing education course. evidence based library and information practice, 8(2), 200-213. http://ejournals.library.ualberta.ca/index.php/eblip   wenger, e. (1998). communities of practice: learning, meaning, and identity. cambridge: cambridge university press.   evidence summary   the majority of high-impact science journals would accept manuscripts derived from open access electronic theses and dissertations   a review of: ramírez, m. l., mcmillan, g., dalton, j. t., hanlon, a., smith, h. s., & kern, c. (2014). do open access electronic theses and dissertations diminish publishing opportunities in the sciences? college & research libraries, 75(6), 808-821. http://dx.doi.org/10.5860/ crl.75.6.808   reviewed by: lisa shen head of reference and instructional services newton gresham library sam houston state university huntsville, texas, united states of america email: lshen@shsu.edu   received: 3 jun. 2015     accepted: 28 jul. 2015      2015 shen. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to assess science journal publishers’ attitudes and policies regarding open access electronic theses and dissertations (etds).   design – survey questionnaire.   setting – science journal publications.   subjects – editorial team members from 290 high-impact science journals.   methods – the 16,455 science journals listed in the 2005-09 thompson reuter’s journal performance indicators (jpi) were identified as the base population for this study. the top five journals, as ranked by relative impact factor, from each of the 171 jpi-defined science disciplines were selected for the sampling frame. after the removal of duplicates, defunct titles, and pretest participants, the 715 resulting journals were grouped into 14 broader subject groups defined by the researchers. randomized systematic sample was then employed to select a final sample size of 300 journals. ten additional titles were later removed due to publication scope.   email invitations to participate in the survey were sent to the selected journals on august 9, 2012. after two email reminders, the web survey closed on august 27. six phone follow-ups were made to a random sample of 100 out of the 246 non-responders between september 7 and 14 to increase the response rate.                 main results – the final response rate for the survey was 24.8% (72 out of 290), and the findings had an 11.5% margin of error with 95% confidence interval.   only 12.5% of the journals surveyed indicated they would “never accept” manuscripts derived from open access etds, while 51.4% indicated revised edts are “always welcome.” the rest of the respondents had some acceptance restrictions, including case-by-case review (19.4%), accept only if the content differs significantly from the original (8.3%), accept or only if access to the original etd was limited (1.4%). five of the 72 respondents (6.9%) did not have a policy for accepting etds. of the 17 researcher-created discipline categories, engineering titles had the highest (85.7%, or 12 out of 14) and medical journals had the lowest (25%, or 3 out of 14) proportion of respondents who would “always welcome” manuscripts derived from open access etds.   at least 50% of the journals from every type of publishing entity indicated they would “always welcome” revised etds. however, there are differences between the entities: university presses were most likely to “always welcome” revised etds (87.5%), commercial publishers were more likely to have some acceptance restrictions (41.7%), and academic societies were the most likely entity to “never welcome” revised etds (12.7%).   lastly, in a comparison of the results of this study with the results from a similar 2013 study conducted on social science, arts and humanities (ss&h) journals, the authors found statistically significant differences (p=0.025, α=0.05) between the editorial policies regarding revised etds of science and ss&h journals.   conclusion – the study results suggest that, contrary to common perceptions, the majority of high-impact science journals would actually welcome revised open access etds submissions. therefore, science scholars would not greatly reduce their chances for publishing manuscripts derived from edts by making the original etds accessible online.   commentary   this article is a valuable contribution to the ongoing discussion about perceptions regarding open access scholarship. an examination of the study using the glynn’s critical appraisal checklist (2006) indicated an overall validity of 81%, above the accepted threshold (75%). validities for the individual sections also met the threshold. the survey instrument was included in the article and the research methodology was clear.   even so, the article had some areas for improvement. in particular, the researchers had drawn a number of generalizations about science journals as a whole without fully addressing the representativeness of the data. when choosing the survey recipients, the researchers first selected the top five ranked journals in each jpi subject categories, then used stratified sampling to select the final sample of 300 titles. consequently, while the results represented how top ranked, high-impact science journals treated etd-derived works, one cannot comfortably apply the same conclusion to all science journals.   in addition, the researchers did not fully explain the method for condensing the 171 jpi science disciplines into 14 subject groups. this omission could be problematic for others who wish to replicate or conduct similar studies. moreover, since there were notable differences between the subject groups’ perceptions toward revised etds, it would be valuable to know how interdisciplinary jpi subjects were treated. for instance, was the jpi subject category “biophysics” grouped into the researcher-defined subject group of biology or physics?   moreover, while the researchers are commended for conducting a pretest, it was unclear how the pretest findings affected the actual study. specifically, editors-in-chief were identified as the most suitable survey respondents from the pretest. however, while editors-in-chief did compose 68.6% of the actual survey respondents, the researchers did not disclose whether specific efforts were made to contact the editors-in-chief, nor did they examine any potential impact of the respondents’ position on their responses. it is possible that the pretest finding was biased and therefore not adopted, since all the pretest participants were editors-in-chief. however, such considerations were also not addressed.   lastly, the interpretation of the results called for further scrutiny. this study generated solid evidence to demonstrate the level of publisher acceptance towards manuscripts derived from open access etds. however, the findings did not necessarily suggest, as the researchers concluded, that “publishers as a whole are accepting of [such] manuscripts” (p. 818). after all, 48.6% of those surveyed would not “always welcome” such manuscripts, and the level of acceptance also varied greatly by discipline. therefore, readers are advised to interpret the findings with caution.   nonetheless, and despite these minor issues, this study demonstrated the value of evidence-based practices and provided a good foundation for future research on the perception and impact of open access etds.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154 researchers’ perceptions and experiences with an open access subvention fund research article   researchers’ perceptions and experiences with an open access subvention fund   jylisa doney social sciences librarian university of idaho library moscow, idaho, united states of america email: jylisadoney@uidaho.edu   jeremy kenyon research librarian university of idaho library moscow, idaho, united states of america email: jkenyon@uidaho.edu   received: 29 july 2021                                                               accepted: 20 jan. 2022      2022 doney and kenyon. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30015     abstract   objective – this study investigated researchers’ perceptions of open access publishing and the ways in which the university’s open access subvention fund could evolve to meet the campus community’s needs.   methods – in spring 2021, two librarians conducted an anonymous survey using a convenience sample to recruit participants. the survey was directly distributed to 113 university of idaho (u of i) affiliates who had received funding from, or expressed interest in, the open access subvention fund during the previous three years (fy 2019 to fy 2021). other u of i affiliates were also offered the opportunity to participate in the survey via a link shared in the u of i’s daily email newsletter as well across the u of i’s graduate student email list. the researchers received 42 usable survey responses. the survey included 26 closed and open-ended questions and analysis included cross-tabulations based on fund applicant status as well as respondent role. of the 26 questions, 4 were modified from a colleague’s previous study with u of i faculty members (gaines, 2015).   results – survey responses showed that interest in and support for open access were common among respondents. although a majority of respondents had published an open access journal article and would like to continue to publish open access in the future, only 17% agreed that they had departmental support to do so. results also demonstrated that researchers were less willing to pay article processing charges (apcs) out-of-pocket and preferred for funding to come from grant budgets first, followed by office of research budgets, department or college budgets, and library budgets. respondents expressed support for many of the open access subvention fund’s current criteria and processes, but they also indicated an interest in establishing a more equitable fund distribution cycle and allowing researchers to seek pre-approval once their article was accepted for peer-review. findings related to open access publishing perspectives built upon previous research conducted at the u of i (gaines, 2015) and across other institutions.   conclusion – this study confirmed the importance of evaluating and assessing library programs and services to ensure that they continue to meet the needs of campus communities. through the study results, the researchers demonstrated that respondents were interested in open access publishing and the continuation of the open access subvention fund, as well as offering the u of i an opportunity to adjust the open access subvention fund’s processes to better serve researchers. these results also highlighted the need for those involved in open access publishing support to investigate new open access advocacy and education efforts to ensure that researchers receive the philosophical and financial support they need to pursue different models of scholarly publishing.     introduction   open access subvention funds are “pool[s] of money set aside by an institution or other research-sponsoring entity specifically to defray or cover processing fees for articles published by members of the institution in open-access journals” (sparc, n.d., “what is an open-access fund?” section). these funds are often upheld as opportunities for libraries and other campus units to heighten the visibility of affiliated research, help change the scholarly publishing landscape, reduce barriers to access, encourage diversity and competition in the publishing services market, and support authors who would otherwise be unable to publish their research in an open access journal. between fy 2019 and fy 2021, the university of idaho (u of i) library, office of the provost, and office of research and economic development (hereafter, “oapf fund partners”) invested a combined total of $110,000 into their open access subvention fund, the u of i open access publishing fund (oapf). during this three-year pilot period, the oapf allocated between $30,000 and $50,000 each fiscal year, and supported the publication of 80 open access articles, authored by a total of 144 u of i researchers. use of the oapf was consistent each year, with the fund disbursing its entire allocation by the end of february (fy 2019), the beginning of february (fy 2020), and the end of march (fy 2021). although fund performance data demonstrated that the oapf filled a need on campus, the researchers were interested in building upon prior research and utilizing other data sources in their assessment of the oapf.   literature review   open access funding models   numerous models for funding open access publications are in the midst of experimentation and development by academic institutions, research funders, non-profit organizations, and others.  models include allowing direct charges to grants by funders (solomon & björk, 2016; springer nature, 2021), creating open access subvention funds (click & borchardt, 2019), academic libraries pursuing institutional open access memberships (björk & solomon, 2012), and pursuing transformative agreements (borrego et al., 2021), among others. one of the reasons given for a shift away from subvention funds by some libraries is the unsustainability of those funds (university of arizona, 2021). some argue that scaling up a fund based solely on paying for institutional researchers’ apcs will eventually exceed the funding available to the library. for large research institutions, this appears mathematically correct. the under-appreciated pay it forward report from the university of california (2016) system highlighted this, showing that for its larger institutions and their comparables, subvention funds were not a scalable approach to ensuring its articles are open access. thus, these institutions have moved toward transformative agreements, which achieve a similar commitment of funds to apcs (albeit capped at a specific level) while reinvesting in a “big deal” approach to journal subscriptions and other methods, such as institutional memberships, to discount publishing fees. however, buried in the report was the implicit suggestion that smaller (or “low-output”) research institutions, e.g., those with under approximately $200 million in research expenditures or that publish around 1,000 articles per year or less, may actually find a subvention fund sustainable, if not transformational (university of california, 2016). this finding suggested that smaller libraries may find value in subvention funds. if anything, this highlights the problematic class divide in library science literature – the largest institutions drive the conversation and trends (for obvious reasons), but their models may not apply beyond the largest institutions or to the long tail of smaller academic libraries. exactly what the right mix of funding strategies ought to be, relative to local institutional dynamics, is a question for further research.   subvention funds   literature on open access subvention funds has taken the form of institution-specific case studies that detail factors related to fund performance, uptake, and sustainability (beaubien et al., 2016; gyore et al., 2015; hampson & stregger, 2017; korolev, 2018, 2020; mcmillan et al., 2020; nabe & imre, 2015; otto, 2019; pinfield & middleton, 2016; sinn et al., 2017; teplitzky & phillips, 2016; zuniga & hoffecker, 2016) as well as multi-institution research (click & borchardt, 2019; fernandez & nariani, 2011; monson et al., 2014; tananbaum, 2014; yates et al., 2015). of most interest to this project were the studies and reports that looked beyond descriptive statistics and investigated fund recipients’ perspectives on open access publishing, their opinions about the criteria and processes used by open access subvention funds, and changes made to open access subvention funds based on their feedback (beaubien et al., 2016; gyore et al., 2015; korolev, 2018, 2020; mcmillan et al., 2020; nabe & imre, 2015; sinn et al., 2017; teplitzky & phillips, 2016; zuniga & hoffecker, 2016).   perspectives on open access publishing   when choosing to publish an open access article, previous research demonstrated that open access subvention fund recipients were interested in increasing the visibility of their work; when asked, many indicated that articles published with open access subvention fund support were more visible and accessible (beaubien et al., 2016; gyore et al., 2015; nabe & imre, 2015; teplitzky & phillips, 2016). another positive outcome of receiving an open access subvention fund award was that recipients reported being willing to publish an open access article again in the future (beaubien et al., 2016; nabe & imre, 2015). even though fund recipients experienced benefits from open access publishing and were interested in pursuing open access publishing again, they believed that authors should not pay the associated article processing charges (apcs) out-of-pocket. when prior open access subvention fund recipients were asked who should pay these apcs, they listed grants, the university, and the library/open access subvention fund as their top three choices; almost no recipients indicated that authors or co-authors should be responsible for paying apcs out-of-pocket (teplitzky & phillips, 2016). related to the desire of having agencies or institutional entities pay for apcs was the finding that the availability of an open access subvention fund was one of the primary factors in a recipient’s decision to publish an open access article (beaubien et al., 2016; nabe & imre, 2015; teplitzky & phillips, 2016). as described by teplitzky and phillips (2016), fund recipients “[were] not necessarily open access advocates but [were] willing to try open access” (p. 573) if funds were made available to them.   opinions on open access subvention fund criteria and processes   overall, open access subvention fund recipients expressed positive opinions about the existence of the funds and wanted them to continue, while also suggesting ways to improve the criteria and processes (gyore et al., 2015; mcmillan et al., 2020; teplitzky & phillips, 2016). these suggestions ranged from allocating more money each year, seeking out money from other campus units, helping potential applicants identify reputable journals, streamlining the process of paying apcs, allowing applicants with active grant funding to apply for an award, making hybrid journals eligible, and removing the requirement that articles must be accepted for publication before submitting an application (gyore et al., 2015; mcmillan et al., 2020; teplitzky & phillips, 2016).   changes made to open access subvention funds   multiple researchers also reported the changes made to open access subvention funds based on fund performance and feedback from fund recipients. some open access subvention funds made changes to their fund caps to ensure that funding was available throughout the year (gyore et al., 2015) or to ensure that authors received enough funding to cover the majority of an apc (korolev, 2018; mcmillan et al., 2020; zuniga & hoffecker, 2016). feedback and fund performance data also led some open access subvention fund operators to prioritize authors with certain roles, such as graduate students and early career researchers, for funding (gyore et al., 2015; sinn et al., 2017; zuniga & hoffecker, 2016). one open access subvention fund reported adding the requirement that articles must be accepted for publication before they could be considered for funding (korolev, 2020).   aims   through this study, we sought to increase the u of i’s awareness of their campus community’s perspectives on open access publishing and the ways in which the oapf’s criteria and processes could evolve to meet campus needs.   the research questions used to frame this study were:   ·        what beliefs do those who applied for or expressed interest in the oapf hold about open access and open access publishing, and have they changed since gaines’ (2015) study? ·        why did u of i affiliates choose or choose not to apply for the oapf during the three-year pilot period? ·        are the oapf’s current criteria and processes supported by those who applied for or expressed interest in the oapf? ·        how can the oapf’s criteria and processes change to better support u of i affiliates?   by conducting this study, the researchers wanted to build upon previous research and assessment related to open access subvention funds and highlight the needs and experiences of campus affiliates at a carnegie r2 institution (doctoral university – high research activity), with approximately 10,790 enrolled students, and a land-grant mission to share research and information with the general public and engage with our local communities. the u of i publishes approximately 1,000 articles per year and expends approximately $110 million in research expenditures annually. although the current study is not the first to examine this topic, documenting how to gather campus affiliates’ perspectives and use their feedback to evaluate and inform changes to an open access subvention fund provides other libraries with a framework to do the same on their own campuses.   methods   in spring 2021, the researchers sent a qualtrics survey to 113 current u of i affiliates who received funding from or expressed interest in the oapf during the previous three years (fy 2019 to fy 2021). these two groups were the main target respondents as the researchers sought to learn more about their experiences with open access publishing and the oapf in order to investigate whether changes to the fund were necessary. however, because those who received funding or expressed interest in funding likely had an open access-positive response bias, the researchers also chose to share the survey twice in the u of i’s daily email newsletter and via the u of i’s graduate student email list.   the 26-item anonymous qualtrics survey included closed and open-ended questions about open access publishing, the oapf and its criteria and processes, as well as demographic information (appendix a). these included four modified questions used by gaines’ (2015) to investigate u of i faculty perceptions and knowledge about open access. the researchers chose to conduct a survey because it allowed us to gather quantitative data, in a timely manner, and compare the results to previous survey research that examined open access subvention funds. the survey research described in this article was approved by the u of i’s institutional review board (protocol # 21-110), who classified this study as exempt under category 2 at 45 cfr 46.104(d)(2), meaning that the ethical standards of respect for persons, beneficence, and justice must be followed, but expedited or full irb review of this project was not required (protection of human subjects, 2018). survey responses were collected between may 3, 2021, and may 27, 2021. data were initially analyzed using the qualtrics cross-tabs feature, which is similar to the pivottables feature available in excel. this feature allowed the researchers to review trends in the data based on respondents’ oapf applicant status. the qualtrics cross-tabs were then imported into excel to create the charts and tables included in this article. to ensure respondent anonymity, all data presented in this article have been aggregated. all direct quotes included in this article are not reported in connection with the respondent’s demographic information to ensure that specific respondents cannot be identified by readers.   results   demographics   forty-nine respondents completed the survey. six incomplete survey responses and one response from a non-u of i respondent were excluded from analysis, resulting in a total of 42 usable responses. respondents included current u of i faculty (n = 28), staff (n = 1), postdoctoral researchers (n = 5), and enrolled graduate students (n = 8) of various ages, with most respondents being between 26 and 45 years old (n = 27). respondents included researchers from agriculture and natural resources (n = 22); science and engineering (n = 13); as well as education and human sciences, humanities and social sciences, and other (n = 7). of the 42 survey respondents, 15 had not applied to the oapf (36%), while 27 reported that they had previously applied (64%), totaling 25% of all currently affiliated fund recipients contacted. while these results can be analyzed across these categories in different ways, we chose to focus on the differences between the oapf applicants and non-applicants in the results below because this is the most pertinent lens for evaluating the fund.   open access publishing: perspectives and experiences   journal selection criteria   when asked about the importance of specific criteria when selecting a journal to publish in, more than 85% of respondents indicated that the journal’s relevance and prestige within their discipline were important when considering both open access and non-open access journals. slight differences in the importance of specific factors were seen when comparing respondents based on oapf applicant status (table 1). for example, across open access and non-open access journals, a higher percentage of those who had applied to the oapf rated the anticipated speed of peer-review to publication as important when compared to those who had not applied to the oapf.   table 1 percentage of respondents rating criteria as important for journal selection criteria applied to oapf didn’t apply to oapf   oa journal non-oa journal oa journal non-oa journal prestige w/n discipline 85% 92% 86% 93% whether is oa or not 58% 42% 77% 29% cost to publish 78% 67% 79% 73% relevance to your discipline 93% 100% 92% 93% impact factor 70% 74% 62% 80% speed of review and publication 66% 67% 54% 50%   perspectives on publishing systems   all but one respondent (98%, n = 41) believed that their discipline or research field benefits from open access journal articles. however, slight differences emerged between respondents when asked whether they preferred a system of publishing in which it is free to publish, but requires subscriptions to read or one in which it is free to read, but requires fees to publish. those who had applied to the oapf (74%, n = 20) expressed a stronger preference for a system in which it is free to read, but requires fees to publish. those who had not applied to the oapf also expressed this preference over the alternative, but at a lower rate (47%, n = 7). in the “other” text-box associated with this question, four respondents stated that they saw the benefits of both systems; two discussed the challenges of both systems; two stated that they preferred a system that is free to publish and free to read; and one stated that they preferred “non-profit based open access,” where content is “free to read, with lower apc than for-profit.”   perspectives on and experiences with open access   respondents also agreed that it was important for the general public to be able to access and read their research (93%, n = 39) and that publishing open access would increase the visibility of their work (88%, n = 37). however, when respondents were asked to indicate their agreement with “i have departmental support to publish open access,” 55% (n = 23) disagreed, 17% (n = 7) agreed, and 29% (n = 12) selected neutral. yet, the majority of respondents reported publishing an open access journal article before (79%, n = 33) and almost all respondents (95%, n = 40) would like to publish in open access journals in the future. differences in prior experiences with open access publishing were seen across respondents, with a higher percentage of oapf applicants (93%, n = 25) indicating that they had previously published an open access journal article when compared to those who had not applied to the oapf (53%, n = 8).   of the respondents who had previously published an open access journal article, most estimated that they had published between one and five articles (70%, n = 23). a smaller number of respondents reported publishing 6 to 10 (n = 4), 11 to 15 (n = 4), or 16 or more open access articles (n = 2). respondents who selected “no” or “not applicable” to publishing an open access journal article before were asked to share some of the reasons why. responses included concerns about the cost of open access publishing, apprehension about lower impact factors and standards for open access journals, as well as statements about career stage and that they had not published any journal articles yet.   open access funding sources   when asked to rank potential funding sources for apcs, respondents listed grant budgets as the first source, followed by office of research budgets, department/college budgets, library budgets, and “other” (figure 1). when ranking the “other” funding source, respondents were given the option of filling-out a text-box to specify the funding source, but all respondents left this box blank. the ranking of funding source preferences did not differ significantly across respondents based on oapf applicant status.   when respondents were asked how much they would be willing to pay out-of-pocket to publish an article in an open access journal, a plurality indicated that they would only be willing to pay up to $250 (36%, n = 15), while a majority selected various options that added up to $1,000 (64%, n = 27). of the respondents who selected “other” (n = 7), 6 listed $0 in the open text-box and one stated they had “never paid out of pocket” (figure 2). no respondents selected $2,501 to $3,000.   experiences with the u of i – oapf   when those who had applied to the oapf (64%, n = 27) were asked about their motivations for seeking funding, a slight majority indicated that it was a practical decision (n = 20) and a smaller number of respondents indicated that it was a desire to change publishing for the better (n = 12). respondents could select multiple answers. when those who had not applied to the oapf before (36%, n = 15) were asked why, responses included that they did not know it existed (n = 7), the oapf was already closed (n = 4), and that their application did not meet all of the eligibility criteria (n = 4). six of these respondents also selected “other” and shared more information in an open text-box, stating that they “never intend to publish in open access vanity journals,” “journals covered by the fund were too restrictive,” and “i have been able to cover the costs through grants, however, costs seem to be going up and i would like to publish more in open access journals.”   figure 1 funding source preference for apcs, by respondent count (n = 40).   figure 2 what is the highest amount you would be willing to pay out-of-pocket to publish an article in an open access journal? (by respondent count).   feedback on the u of i – oapf’s current eligibility criteria and processes   during the three-year pilot period, applicants had to meet specific article, journal, author, and documentation criteria and requirements in order to qualify for the fund. these were as follows: 1) articles must be accepted for publication in the current fiscal year and funding per article is capped at $2,000.00; 2) journals must be listed in the directory of open access journals (doaj), utilize peer-review, and make articles fully open for publication; 3) journal publishers must be members of or comply with the open access scholarly publishing association (oaspa); 4) hybrid journals are ineligible for funding; 5) the applicant must be a current u of i affiliate and funding per fiscal year is capped at $3,000.00 per author; and 6) priority will be given to oapf applications without other sources of funding available. these criteria and requirements were established following a review of open access subvention funds at other colleges and universities as well as discussions with the oapf fund partners about their goals for the fund.   the requirements that received the highest levels of support from respondents were that a journal must utilize peer-review (95%, n = 40) and make articles fully open upon publication (95%, n = 40), the affirmation that authors may apply for funding multiple times until their cap is met or funds are exhausted (85%, n = 35), only current u of i affiliates can apply for funding (83%, n = 34), and the prioritization of oapf applications without other sources of funding available (71%, n = 29). the funding cap of $3,000.00 per author per fiscal year (66%, n = 27) and the per article cap of $2,000.00 (57%, n = 24) were also supported by a majority of respondents.   in comparison, the ineligibility of subscription journals that publish only certain articles open access (hybrid journals) was opposed by a plurality of respondents (oppose = 45%, n = 19). this was the only current criterion that received more opposition than support from respondents. other requirements that received lower levels of support included that a journal must be listed in the doaj (support = 50%, n = 21), an article must be accepted for publication before submitting an oapf application (support = 50%, n = 21), and that a journal publisher must be a member of or comply with the oaspa (support = 41%, n = 17).   slight differences in the levels of support or opposition for specific criteria were seen based on oapf applicant status. as seen in figure 4, a higher percentage of those who had applied to the oapf (67%, n = 18) than those who had not applied (20%, n = 3) expressed support for the requirement that articles must be accepted for publication before submitting an oapf application. a higher percentage of oapf applicants (96% n = 25) than non-applicants (67%, n = 10) also expressed support for the affirmation that authors may apply for funding multiple times until their cap is met or funds are exhausted. while a higher percentage of those who had not applied to the oapf (87%, n = 13) than those who had applied (62%, n = 16) expressed support for the prioritization of applications without other sources of funding available (figure 3).   figure 3 feedback on oapf criteria and processes.   feedback on proposed changes to the u of i – oapf’s eligibility criteria   within this survey, respondents were also asked to indicate their support for changes suggested by members of the u of i community during the oapf’s three-year pilot period. the proposed changes with the highest levels of support from respondents were splitting the fund allocation during the fiscal year (63%, n = 26), allowing applications to be submitted for pre-approval when an article is undergoing peer-review (59%, n = 24), and making subscription journals that publish only certain articles open access (hybrid journals) eligible for funding (54%, n = 22). the proposed change with the highest level of opposition was that authors could only receive funding once per fiscal year (oppose = 56%, n = 23).   slight differences in the levels of support or opposition for potential changes were seen based on oapf applicant status. as seen in figure 4, a higher percentage of those who had not applied to the oapf (53%, n = 8) expressed support for setting aside or earmarking a portion of the funding for u of i affiliates at specific career stages when compared to those who had applied to the oapf (31%, n = 8). while a higher percentage of those who had applied to the oapf (62%, n = 16) expressed support for making hybrid journals eligible when compared to those who had not applied to the oapf (40%, n = 6). when asked about setting aside or earmarking a portion of the funding for u of i affiliates in non-stem disciplines, a slight plurality of those who had applied to the oapf (46%, n = 12) opposed this change while a majority of those who had not applied to the oapf (60%, n = 9) were neutral (figure 4).   figure 4 feedback on proposed changes to oapf criteria and processes.   discussion/limitations   findings from this survey demonstrated that respondents agreed their research field/discipline benefits from open access journal articles, that publishing open access would increase the visibility of their work, and that the general public should be able to access and read their research. these findings are in line with previous research on open access publishing (beaubien et al., 2016; boock et al., 2020; dallmeier-tiessen et al., 2011; dalton et al., 2020; gaines, 2015; greussing et al., 2020; gyore et al., 2015; jamali et al., 2020; mcdonald et al., 2016; nabe & imre, 2015; odell et al., 2017; rowley et al., 2017; segado-boj et al., 2018; tenopir et al., 2017; teplitzky & phillips, 2016; togia & korobili, 2014). current survey respondents also affirmed that a journal’s relevance to their discipline and its prestige were important factors when choosing both open access and non-open access journals (blankstein & wolff-eisenberg, 2019; boock et al., 2020; gaines, 2015; greussing et al., 2020; mcdonald et al., 2016; odell et al., 2017; rowley et al., 2017). as found in prior research, respondents also appeared to hold positive opinions about the oapf, and when asked to share their feedback, many commented that they wanted the fund to continue and that it would be difficult to publish in high quality journals without it (gyore et al., 2015; mcmillan et al., 2020; teplitzky & phillips, 2016). respondents to this survey also indicated a willingness to pursue open access publishing in the future, which is consistent with earlier research (beaubien et al., 2016; nabe & imre, 2015).   even though respondents were interested in open access publishing, they were less willing to pay the associated fees out-of-pocket. most stated they would only be willing to pay up to $1,000 out-of-pocket and preferred for apc funding to come from grant budgets first, followed by institutional funding sources, and finally library sources; these results are similar to those found in prior research (sheikh, 2019; tenopir et al., 2017; teplitzky & phillips, 2016; togia & korobili, 2014). one hypothesis as to why current respondents ranked agency and institutional funding sources higher than library funding sources is that authors may be prone to a status quo bias. if authors view paying apcs as the final stage in their research process, they may look to funding sources that are traditionally associated with research to pay these charges, such as grants, the office of research, and their own departments. even though library and out-of-pocket funding sources may be available, authors may not view these sources as viable funding options for apcs if they do not typically use them for their own professional research activities. a librarian’s tendency to see apcs as the “flipside” of a subscription – both are types of funds that go to supporting a journal’s operations – is not viewed in the same way by researchers. additional research is necessary to investigate the relationship between funding source preferences for apcs and general support for library-funded open access subvention funds.   overall, respondents indicated that they sought funding from the oapf for practical reasons, but some were interested in changing publishing for the better. those who had not sought funding from the oapf admitted that they did not know it existed, that it was already closed when they attempted to apply, and that their applications did not meet all of the eligibility criteria. these results confirm that some of the oapf’s education and outreach is working, but additional work needs to be done to ensure that the entire campus community knows that this program exists and is kept informed about the fund’s status and criteria. survey responses and open text-box comments demonstrated that respondents supported most of the u of i oapf’s current criteria and processes, but were also interested in specific changes. although u of i affiliates were interested in receiving financial support to publish in hybrid journals, the oapf decided not to make hybrid open access journals eligible for funding. this decision was made following discussions with the oapf fund partners in which they reaffirmed their commitment to only support articles that are published in fully open access journals. oapf fund partners felt that hybrid open access journals did not support the oapf’s goals of promoting diversity and competition in the publishing services market nor would they advance open access publishing models that embrace equitable access to research. they also recognized that the consistently higher apcs charged by hybrid open access journals (see budzinski et al., 2020; pinfield et al, 2017) could result in fewer authors receiving support each year.   u of i affiliates were also interested in allowing authors to seek pre-approval for funding, but because of the potential issues associated with earmarking funding throughout the fiscal year, this suggestion has not been implemented at this time either. however, the oapf will continue to investigate the “pre-approval for funding” option, in concert with the library’s and university’s financial experts, as the current “accepted for publication” criterion is likely a barrier that discourages some authors from pursuing open access publishing as they would be unable to pay the apcs without guaranteed financial support.   the change with the most support from survey respondents, splitting the fund allocation during the fiscal year, was something that seemed feasible and beneficial, if subsequent data also demonstrated that this change was necessary. after examining oapf application data, the researchers discovered that 61% of the allocation in fy 2019, 89% of the allocation in fy 2020, and 100% of the allocation in fy 2021 had been disbursed by the end of the first six months of the fiscal year. the researchers were able to secure an additional $20,000.00 in fy 2021 and reopen the fund in january 2021, but these funds were disbursed by the end of march 2021. as a result of this data and support from survey respondents, the oapf proposed a change to its funding distribution cycle for fy 2022. with approval from its three original funding partners and a new donor, the oapf’s allocation was split 50/50, with half of the allocation ($17,500) made available for articles accepted for publication between july 1, 2021, and december 31, 2021, and the other half ($17,500) made available for articles accepted for publication between january 1, 2022, and june 15, 2022. with this change, the oapf seeks to create a more equitable funding cycle and give those who publish their research during the second half of the fiscal year an opportunity to receive financial support.   looking ahead to future funding cycles, it will be necessary for the oapf and its fund partners to continue to take the needs and suggestions of the u of i campus community into account, while also considering whether their perspectives correspond with the needs, interests, and goals of the oapf and its fund partners. at times, as evinced by u of i affiliates’ suggestion to make hybrid open access journal eligible for funding, the needs of researchers might be in conflict with the perspectives of the oapf and its fund partners. the u of i’s faculty senate committee on library affairs gives us a means to discuss and evaluate these conflicting perspectives in a forum designated for handling faculty preferences with respect to library and institutional goals. navigating conflicting perspectives is never an easy task; the fact that a conflict exists gives the oapf fund partners an opportunity to listen and learn more about the needs and experiences of their affiliates and potentially find common ground.   comparison to previous research at the u of i   this survey also built upon and modified four questions used in a previous study of u of i faculty members (gaines, 2015). these questions focused on the benefits of open access to a researcher’s discipline, prior open access publishing experience, the factors used when selecting where to publish, and perspectives on open access. in 2013, 57% of survey respondents believed that their research field/discipline “currently benefits or would benefit from journals that publish open access” (gaines, 2015, p. 7); 96% (n = 27) of faculty respondents to the current survey believed the same. in total, 41% of faculty in 2013 had published an open access journal article before (gaines, 2015, p. 7) compared to 89% (n = 25) of faculty respondents to the current survey. although both survey samples were small, these data demonstrated that in the last eight years, it is likely that perspectives on open access have grown more positive and more faculty are engaging in open access publishing across campus; however, it is also necessary to recognize that the 2021 respondents may have an open access-positive response bias as more than half of respondents had sought support from the oapf. across both surveys, a higher percentage of faculty selected “relevance of the journal for their discipline” as important for both open and non-open access journals when compared to other factors (gaines, 2015, p. 9). in 2013, 80% of faculty “strongly believed that publicly funded research should be made available to the public without barriers” (gaines, 2015, p. 8), while 93% (n = 26) of faculty in the current survey agreed that “it is important to me that the general public can access and read my research.” gaines’ (2015) findings demonstrated that 67% of respondents agreed that “publishing in open access journals would increase the visibility of their research,” while 72% of respondents agreed that they “would like to publish in open access journals in the future” (p. 9). in the current survey, 89% (n = 25) and 93% (n = 26) of faculty respondents agreed with these statements, respectively. these data indicated that interest in open access appears to be growing at the u of i and more respondents agreed with many common philosophical and credit-based arguments about open access.   however, all of the findings were not as positive. when asked in 2013 whether they “felt they had institutional or departmental support to publish open access,” 15% of faculty respondents agreed, 48% were unsure, and 35% disagreed (gaines, 2015, p. 9). in the current survey, 14% (n = 4) of faculty agreed, 29% (n = 8) selected neutral, and 57% (n = 16) disagreed. these data showed that for some u of i faculty, departmental support for open access publishing might have decreased over the last eight years.   as neither survey defined “support,” it is possible that respondents answered this question with different types of departmental support in mind, such as philosophical support, financial support, tenure/promotion support, or other types of support. even though faculty respondents’ may have interpreted “support for open access publishing” differently, the researchers affirm that these results are still relevant when examined in light of other findings. although faculty respondents are interested in publishing open access and see how this decision could potentially lead to both professional and broader public benefits, they do not feel that their departments are aligned with these pursuits. looking ahead, additional research is necessary to investigate researchers’ experiences and perceptions of various types of departmental support for open access publishing, as any efforts to increase departmental support for open access publishing will need to be tailored to the types of support that are currently lacking.   limitations   although these data added to the u of i’s knowledge of affiliates’ perspectives on open access and the oapf and led to proposed changes to the funding distribution cycle, it is important to acknowledge that there were limitations. one limitation is that the convenience sample of respondents—those who had applied for or expressed interest in the oapf—likely had an open access-positive response bias. this bias did not negate the usefulness of these findings, but it did suggest that additional research with the u of i campus community is necessary to confirm whether these findings apply to the broader community. other limitations included a small sample size, which limits both the local and broader generalizability of the results; a limited ability to ask follow-up questions; and the potential for respondents to interpret questions differently. to address the small sample size, researchers could seek additional irb approval and pursue further engagement by asking library liaisons to share a link to the survey with faculty and students in their departments. researchers could also ask the oapf fund partners to send a joint email to campus affiliates, encouraging them to participate in the survey. to increase participation among an expanded convenience sample, the researchers could also add a link to the survey within the oapf award decision emails and share a link to the survey with anyone who expressed interest in the fund. the lack of follow-up questions available when using the survey methodology could be addressed by adding open-ended “tell us why you selected that response” survey questions in specific locations. the researchers could also conduct interviews or focus groups to give participants an opportunity to share their perspectives in a less structured way. to ensure that all respondents approached questions in a consistent manner, researchers could also add definitions or clarify questions, when necessary. for example, the researchers could change the general phrase “departmental support” to “financial support from your department,” “philosophical support from your department,” or other options.   conclusion   as the oapf’s three-year pilot period has come to an end, it is evident that this program met a campus need by providing financial and philosophical support for those who were interested in open access publishing. findings demonstrated that interest in open access journal publishing was high among a subset of the u of i campus community, but a perceived lack of departmental support was also common. respondents supported most of the u of i oapf’s current criteria and processes, but they also supported changes that sought to make the fund more equitable. these included disbursing the allocation biannually to ensure that funding was available for researchers throughout the year and utilizing a pre-approval award process to ensure that funding was still available once an article was officially accepted for publication. during its fy 2022 award cycle, the oapf was able to split the total allocation in half, ensuring that researchers publishing in either the first or the second half of the fiscal year had a similar opportunity to seek funding. more research is needed to determine whether this change to the disbursement cycle will be effective. the researchers will also continue investigating whether and how pre-approval of applications prior to article acceptance could be implemented. in investigating the feasibility of the pre-approval process, the researchers will leverage the expertise of the library’s and university’s financial experts as well as seek additional input from the oapf’s funding partners and members of the campus community.   this survey is just one piece of the data-driven assessment strategy that is necessary to evaluate the impact and effectiveness of an open access subvention fund. although these results are specific to the local setting, the survey protocol as well as the experiences and needs of respondents could potentially inform oa subvention fund assessment and library practices at other institutions. for example, other institutions with open access subvention funds may choose to replicate this survey and add questions that reflect their unique institutional contexts. they might also consider investigating whether their fund could benefit from some of the changes suggested by our respondents and whether these changes would be feasible on their own campuses. overall, these results demonstrated that the provision of a subvention fund is just one step in the process; libraries and institutions must continue to assess whether the fund’s criteria and processes reflect the practices and needs of their own unique scholarly communities. ongoing assessment should include a combination of data related to fund performance; the attitudes, perceptions, and behaviours of campus community members; and the impact of funded articles. gathering data from a variety of sources will allow open access subvention funds and their fund partners to gain a well-rounded perspective on the impact of these funds; such work is already occurring at the u of i. to determine whether the open access subvention fund met its user-centred program outcomes, research analyzing citation data and researchers’ publishing behaviours before and after receiving an oapf award is also ongoing. looking ahead to the ever-changing open access landscape, it will be necessary for those involved in open access subvention funds to continue their assessment efforts to ensure that these funds consistently meet the needs of their campus communities.   author contributions   jylisa doney: conceptualization (equal), methodology (equal), investigation (equal), formal analysis (lead), writing – original draft (lead), writing – review & editing (equal) jeremy kenyon: conceptualization (equal), methodology (equal), investigation (equal), formal analysis (supporting), writing – original draft (supporting), writing – review & editing (equal)   references   beaubien, s., garrison, j., & way, d. (2016). evaluating an open access publishing fund at a comprehensive university. journal of librarianship and scholarly communication, 3(3), ep1204. https://doi.org/10.7710/2162-3309.1204   björk, b.-c., & solomon, d. (2012). pricing principles used by scholarly open access publishers. learned publishing, 25(2), 132-137. https://doi.org/10.1087/20120207   blankstein, m., & wolff-eisenberg, c. (2019). ithaka s+r us faculty survey 2018. ithaka s+r. https://doi.org/10.18665/sr.311199   boock, m., todorova, t. y., trencheva, t. s., & todorova, r. (2020). bulgarian authors’ open access awareness and preferences. library management, 41(2/3), 91-102. https://doi.org/10.1108/lm-08-2019-0059   borrego, á., anglada, l., abadal, e. (2021). transformative agreements: do they pave the way to open access? learned publishing, 34(2), 216-232. https://doi.org/10.1002/leap.1347   budzinski, p., grebel, t., wolling, j., & zhang, x. (2020). drivers of article processing charges in open access. scientometrics, 124, 2185-2206. https://doi.org/10.1007/s11192-020-03578-3   click, a. b., & borchardt, r. (2019). library supported open access funds: criteria, impact, and viability. evidence based library and information practice, 14(4), 21-37. https://doi.org/10.18438/eblip29623   dallmeier-tiessen, s., darby, r., goerner, b., hyppoelae, j., igo-kemenes, p., kahn, d., lambert, s., lengenfelder, a., leonard, c., mele, s., nowicka, m., polydoratou, p., ross, d., ruiz-perez, s., schimmer, r., swaisland, m., & van der stelt, w. (2011, january 28). highlights from the soap project survey: what scientists think about open access publishing. arxiv. https://arxiv.org/abs/1101.5260v2   dalton, e. d., tenopir, c., & björk, b.-c. (2020). attitudes of north american academics toward open access scholarly journals. portal: libraries and the academy, 20(1), 73-100. https://doi.org/10.1353/pla.2020.0005   fernandez, l., & nariani, r. (2011). open access funds: a canadian library survey. partnership: the canadian journal of library and information practice and research, 6(1). https://doi.org/10.21083/partnership.v6i1.1424   gaines, a. m. (2015). from concerned to cautiously optimistic: assessing faculty perceptions and knowledge of open access in a campus-wide study. journal of librarianship and scholarly communication, 3(1), ep1212. https://doi.org/10.7710/2162-3309.1212   greussing, e., kuballa, s., taddicken, m., schulze, m., mielke, c., & haux, r. (2020). drivers and obstacles of open access publishing. a qualitative investigation of individual and institutional factors. frontiers in communication, 5. https://doi.org/10.3389/fcomm.2020.587465   gyore, r., reeve, a. c., cameron-vedros, c., ludwig, d., & emmett, a. (2015). campus open access funds: experiences of the ku “one university” open access author fund. journal of librarianship and scholarly communication, 3(1), ep1252. https://doi.org/10.7710/2162-3309.1252   hampson, c., & stregger, e. (2017). measuring cost per use of library-funded open access article processing charges: examination and implications of one method. journal of librarianship and scholarly communication, 5(1), ep2182. https://doi.org/10.7710/2162-3309.2182   jamali, h. r., nicholas, d., herman, e., boukacem-zeghmouri, c., abrizah, a., rodríguez-bravo, b., xu, j., świgon’, m., polezhaeva, t., & watkinson, a. (2020). national comparisons of early career researchers’ scholarly communication attitudes and behaviours. learned publishing, 33(4), 370-384. https://doi.org/10.1002/leap.1313   korolev, s. (2018). evaluating the impact of the university open access publication fund (uwm libraries other staff publications). https://dc.uwm.edu/lib_staffart/11   korolev, s. (2020). 2020 update of the university open access publication fund (uwm libraries other staff publications). https://dc.uwm.edu/lib_staffart/13   mcdonald, b., gibson, i., yates, e., & stephenson, c. (2016). an exploration of faculty experiences with open access journal publishing at two canadian comprehensive universities. partnership: the canadian journal of library and information practice and research, 11(2). https://doi.org/10.21083/partnership.v11i2.3703   mcmillan, g., o’brien, l., & lener, e. f. (2020, august 31). guest post—assessing user perceptions of an open access subvention fund. the scholarly kitchen. https://scholarlykitchen.sspnet.org/2020/08/31/guest-post-assessing-user-perceptions-of-an-open-access-subvention-fund/   monson, j., highby, w., & rathe, b. (2014). library involvement in faculty publication funds. college & undergraduate libraries, 21(3-4), 308-329. https://doi.org/10.1080/10691316.2014.933088   nabe, j., & imre, a. (2015, january 31). library funding of open access publication fees: effects on faculty behavior and attitude. ala midwinter: alcts scholarly communications interest group. https://opensiuc.lib.siu.edu/morris_confs/22   odell, j., palmer, k., & dill, e. (2017). faculty attitudes toward open access and scholarly communications: disciplinary differences on an urban and health science campus. journal of librarianship and scholarly communication, 5(1), ep2169. https://doi.org/10.7710/2162-3309.2169   otto, r. w. (2019). six years of running a campus open access publishing fund. where are we? charleston conference. https://krex.k-state.edu/dspace/handle/2097/40201   pinfield, s., & middleton, c. (2016). researchers’ adoption of an institutional central fund for open-access article-processing charges: a case study using innovation diffusion theory. sage open, 6(1). https://doi.org/10.1177/2158244015625447   pinfield, s., salter, j., & bath, p. a. (2017). a “gold-centric” implementation of open access: hybrid journals, the “total cost of publication,” and policy development in the uk and beyond. journal of the association for information science and technology, 68(9), 2248-2263. https://doi.org/10.1002/asi.23742   protection of human subjects, 45 c.f.r. § 46.104(d)(2). (2018). https://www.ecfr.gov/on/2018-07-19/title-45/subtitle-a/subchapter-a/part-46#p-46.104(d)(2)   rowley, j., johnson, f., sbaffi, l., frass, w., & devine, e. (2017). academics’ behaviors and attitudes towards open access publishing in scholarly journals. journal of the association for information science and technology, 68(5), 1201-1211. https://doi.org/10.1002/asi.23710   segado-boj, f., martín-quevedo, j., & prieto-gutiérrez, j. j. (2018). attitudes toward open access, open peer review, and altmetrics among contributors to spanish scholarly journals. journal of scholarly publishing, 50(1), 48–70. https://doi.org/10.3138/jsp.50.1.08   sheikh, a. (2019). faculty awareness, use and attitudes towards scholarly open access: a pakistani perspective. journal of librarianship and information science, 51(3), 612-628. https://doi.org/10.1177/0961000617742455   sinn, r. n., woodson, s. m., & cyzyk, m. (2017). the johns hopkins libraries open access promotion fund: an open and shut case study. college & research libraries news, 32-35. https://doi.org/10.5860/crln.78.1.9605   solomon d., & björk, b.-c. (2016). article processing charges for open access publication—the situation for research intensive universities in the usa and canada. peerj, 4, e2264. https://doi.org/10.7717/peerj.2264   sparc. (n.d.). campus open access fund faq. https://sparcopen.org/our-work/oa-funds/faq/   springer nature. (2021). funding for open access articles. https://www.springernature.com/gp/open-research/funding/articles   tananbaum, g. (2014). north american campus-based open access funds: a five-year progress report. sparc. https://sparcopen.org/wp-content/uploads/2016/01/oa-fund-5-year-review.pdf   tenopir, c., dalton, e. d., christian, l., jones, m. k., mccabe, m., smith, m., & fish, a. (2017). imagining a gold open access future: attitudes, behaviors, and funding scenarios among authors of academic scholarship. college & research libraries, 78(6), 824-843. https://doi.org/10.5860/crl.78.6.824   teplitzky, s., & phillips, m. (2016). evaluating the impact of open access at berkeley: results from the 2015 survey of berkeley research impact initiative (brii) funding recipients. college & research libraries, 77(5), 568-581. https://doi.org/10.5860/crl.77.5.568   togia, a., & korobili, s. (2014). attitudes towards open access: a meta-synthesis of the empirical literature. information services & use, 34(3-4), 221-231. https://doi.org/10.3233/isu-140742   university of arizona. (2021). open access investment fund. https://new.library.arizona.edu/about/awards/oa-fund   university of california libraries. (2016). pay it forward: investigating a sustainable model of open access article processing charges for large north american research institutions. https://www.library.ucdavis.edu/wp-content/uploads/2018/11/icis-uc-pay-it-forward-final-report.rev_.7.18.16.pdf   yates, e., hampson, c., hatherill, j., lavigne, j., nariani, r., paterson, j., shires, m., & tiessen, r. (2015). library open access funds in canada: review and recommendations. canadian association of research libraries: open access working group. https://www.carl-abrc.ca/doc/carloawglibraryoafundsfinalreport-jan%202016.pdf   zuniga, h., & hoffecker, l. (2016). managing an open access fund: tips from the trenches and questions for the future. journal of copyright in education & librarianship, 1(1). https://doi.org/10.17161/jcel.v1i1.5920   appendix survey questions   1.      do you believe your discipline or research field benefits from open access journal articles? a.      yes   b.      no   c.      unsure     2.      which system of publishing do you prefer? a.      one in which it is free to publish, but requires subscriptions to read   b.      one in which it is free to read, but requires fees to publish   c.      other [text-box]   3.      have you published an open access journal article before? a.      yes   b.      no   c.      not applicable [text-box]     4.      approximately how many open access journal articles have you published? (if yes on q3) [text-box]   5.      what are some of the reasons you haven't published an open access journal article before? (if no on q3) [text-box]   6.      when you select a journal to publish in, how important are the following factors? (likert scale: not important, neutral, important) a.      non-open access journal                                                     i.     prestige of the journal within your discipline                                                                                                        ii.     whether the journal is open access                                                                                                                                       iii.     the cost to publish in the journal                                                                                                                                          iv.     relevance of the journal for your discipline                                                                                                                          v.     the journal's impact factor                                                                                                                                      vi.     the anticipated speed of the peer-review to publication process b.      open access journal                                                     i.     prestige of the journal within your discipline                                                                                                        ii.     whether the journal is open access                                                                                                                                       iii.     the cost to publish in the journal                                                                                                                                          iv.     relevance of the journal for your discipline                                                                                                                          v.     the journal's impact factor                                                                                                                                      vi.     the anticipated speed of the peer-review to publication process                                                                                              7.      what is the highest amount you would be willing to pay out-of-pocket to publish an article in an open access journal if you had no financial support from grants, the u of i – oapf, your department, etc.? a.      up to $250.00   b.      $250.00 to $500.00   c.      $501.00 to $1000.00    d.      $1001.00 to $1500.00   e.      $1501.00 to $2000.00   f.       $2001.00 to $2500.00   g.      $2501.00 to $3000.00   h.      more than $3001.00   i.       other [text-box]   8.      with 1 (one) being your top preference, please rank these common funding avenues based on where you believe funding for article processing charges (fees to publish in an open access journal) should come from. a.      grant budgets b.      department/college budgets c.      library budgets d.      office of research budgets e.      other [text-box]   9.      please indicate your agreement with the following statements. (likert scale: disagree, neutral, agree) a.      i have departmental support to publish open access                                        b.      i prefer to publish in open access journals                                           c.      i would like to publish in open access journals in the future                                         d.      important researchers in my field are publishing open access                                      e.      publishing open access would increase the visibility of my work                                               f.       it is important to me that the general public can access and read my research                g.      there are open access journals that match with my research interests/field                                10.   have you applied for the u of i – oapf during the last three years? a.      yes   b.      no     11.   what motivated you to apply for funding via the u of i – oapf? (select all that apply) (if yes on q10) a.      desire to change publishing for the better   b.      a practical decision   c.      other [text-box]   12.    did you receive funding for at least one article from the u of i – oapf? (if yes on q10) a.      yes   b.      no     13.   did you ultimately publish your article through other means? (if no on q12) a.      yes: in an open access journal   b.      yes: in a non-open access journal   c.      no   d.      other [text-box]   14.   what are some of the reasons you haven't applied to the u of i – oapf before? (select all that apply) (if no on q10) a.      didn't know it existed   b.      oapf was already closed   c.      my application didn't meet all of the eligibility criteria   d.      other [text-box]   15.   what feedback do you have about the u of i – oapf's application, notification, invoice payment and reimbursement, or other aspects of the process? questions about criteria and processes appear on the next page of the survey. [text-box]   please indicate your support for the u of i – oapf’s current criteria and processes.   16.   article criteria (likert scale: oppose, neutral, support) a.      articles must be accepted for publication before submitting an oapf application         b.      funding per article is capped at $2000.00                                               17.   journal criteria (likert scale: oppose, neutral, support) a.      the journal must be listed in the directory of open access journals (doaj)                 b.      the journal must utilize peer-review                                     c.      the journal must make articles fully open upon publication                                        d.      subscription journals that make certain articles open access are ineligible for funding                              e.      the journal publisher must be a member of or comply with the open access scholarly publishers association (oaspa)                                           18.   author criteria (likert scale: oppose, neutral, support) a.      only current u of i affiliates are able to apply for the oapf           b.      funding per author is capped at $3000.00 per fiscal year                                c.      authors may apply for funding multiple times until their cap is met or funds are exhausted    d.      priority will be given to oapf applications without other sources of funding available (such as grants or contracts)                                      19.   award process (likert scale: oppose, neutral, support) a.      applications are evaluated on a first come, first served basis                                       b.      the oapf will pay invoices directly                                      c.      the oapf will reimburse funded applicants for out-of-pocket invoice payments                                          20.   please indicate your support for the following changes to the u of i – oapf, as suggested by members of the u of i community. (likert scale: oppose, neutral, support) a.      an oapf application can be submitted for pre-approval when an article is undergoing peer-review                              b.      a portion of funding will be set aside/earmarked for u of i affiliates at specific career stages      c.      authors can only receive funding once per fiscal year                                     d.      a portion of funding is set aside/earmarked for u of i affiliates in non-stem disciplines                         e.      subscription journals that make certain articles open access would be eligible for funding         f.       split oapf allocation within the fiscal year (e.g. half the allocation is available for applications submitted between july and december; half the allocation is available for applications submitted between january and june)                                       21.   what other changes would you like to see the u of i – oapf make to its criteria and processes? [text-box]   22.   do you have any additional feedback about the u of i – oapf? [text-box]   23.   are you a current u of i ... a.      faculty member   b.      staff member   c.      postdoctoral researcher   d.      enrolled graduate student   e.      enrolled undergraduate student   f.       other [text-box]   24.   what is your age? a.      18 25   b.      26 35   c.      36 45   d.      46 55   e.      56 65   f.       66 75   g.      75 +     25.    which college do you primarily work in? a.      agricultural and life sciences   b.      art and architecture   c.      business and economics   d.      education, health and human sciences   e.      engineering   f.       law   g.      letters, arts and social sciences   h.      natural resources   i.       science   j.        other [text-box]   26.    which department do you primarily work in? [text-box]   evidence summary   gender, race, and age of librarians and users have an impact on the perceived approachability of librarians   a review of: bonnet, j. l., & mcalexander, b. (2012). structural diversity in academic libraries: a study of librarian approachability. the journal of academic librarianship, 38(5), 277-286. doi:10.1016/j.acalib.2012.06.002   reviewed by: dominique daniel humanities librarian for history and modern languages oakland university rochester, michigan, united states of america email: daniel@oakland.edu   received: 28 may 2013  accepted: 1 aug. 2013      2013 daniel. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to assess how the age, gender, and race characteristics of library users affect their perceptions of the approachability of reference librarians with similar or different demographic characteristics.   design – image rating survey.   setting – large, three-campus university system in the united states.   subjects – there were 449 students, staff, and faculty of different ages, gender, and race.   methods – in an online survey respondents were presented with images of hypothetical librarians and asked to evaluate their approachability, using a scale from 1 to 10. the images showed librarians with neutral emotional expressions against a standardized, neutral background. the librarians’ age, gender, and race were systematically varied. only white, african american, and asian american librarians were shown. afterwards respondents were asked to identify their own age, gender, race, and status.   main results – respondents perceived female librarians as more approachable than male librarians, maybe due to expectations caused by the female librarian stereotype. they found librarians of their own age group more approachable. african american respondents scored african american librarians as more approachable, whereas whites expressed no significant variation when rating the approachability of librarians of different races. thus, african americans demonstrated strong in-group bias but whites manifested colour blindness – possibly a strategy to avoid the appearance of racial bias. asian americans rated african american librarians lower than white librarians.   conclusion – this study demonstrates that visible demographic characteristics matter in people’s first impressions of librarians. findings confirm that diversity initiatives are needed in academic libraries to ensure that all users feel welcome and are encouraged to approach librarians. regarding gender, programs that deflate the female librarian stereotype may help improve the approachability image of male librarians. academic libraries should staff the reference desk with individuals covering a wide range of ages, including college-aged interns, whom traditional age students find most approachable. libraries should also build a racially diverse staff to meet the needs of a racially diverse user population. since first impressions have lasting effects on the development of social relationships, structural diversity should be a priority for libraries’ diversity programs.     commentary   whereas most library diversity studies focus on librarians or users separately, this project provides a compelling analysis of their interactions, especially the impact of their respective demographic characteristics on librarians’ perceived approachability. it relies on recent psychological models for stereotypical impressions. an extensive bibliography gives the study a strong theoretical and methodological foundation.   the survey methodology and statistical analysis are solid, and the instrument was validated by pilot tests (glynn, 2006, p.393). for more nuanced results, additional ethno-racial categories could have been included: respondents were categorized as white (around 60%), african american (only 6%), and asian american (18%), with the remainder being lumped under “other.” for this reason it would have been useful to include an appendix with the demographic and personality questions that were asked in the survey and detailed information about the respondents’ demographic profiles. the authors acknowledge that they could not include hispanic librarians, because they could not find enough librarian images that would match the purpose of the study.   some of the interpretations rely on appealing but untested hypotheses, such as the positive impact of the female librarian stereotype on women’s approachability or white respondents’ strategic colour blindness when evaluating black librarians. the authors confirm the need for diversity in library staff. their findings lend credence to the proposal to place student workers at the reference desk. however their conclusions overestimate the external validity of the study (glynn, 2006, p. 398). the survey was administered at a midwestern university, where the ethno-racial makeup and culture are somewhat different from other regions, but the authors offer seemingly uniform, nationwide recommendations about the need for diversity programs. regarding gender, they rightly recognize that stereotypes can vary as they are influenced by factors like “societal norms, cultural factors and context” (p. 283). yet they do not consider that race and ethnicity are also social constructions that are context-dependent. curiously, their finding that female librarians are seen as more approachable leads them to conclude that more male librarians are needed, but  the finding that young or black librarians are preferred by young or black users prompts them to call for an increased number of young persons and african americans on the reference desk. it looks like the authors determined some of their conclusions before starting their study.   finally, reliance on a neutral, hypothetical situation is sound practice but it would be useful to test librarian approachability in real-life or naturalistic settings, where the impact of demographic factors relative to affective and situational factors could be evaluated. since this 2012 article, the authors have published a new study testing the influence of affect and clothing on approachability and recommend research into combinations of variables that should lead to valuable insight for library services (bonnet & mcalexander 2013, p. 10).     references   bonnet, a. & mcalexander, b. (2013) first impressions and the reference encounter: the influence of affect and clothing on librarian approachability. journal of academic librarianship, doi:10.1016/j.acalib.2012.11.025   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi:10.1108/07378830610692154   news/announcements   eblip returns down under      2014. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   the international evidence based library and information practice conference is returning to australia. ten years after the third conference ventured to the southern hemisphere, brisbane is again host city for eblip8 from july 6th to 8th 2015. this biennial event aims to promote the use of the best available evidence in practices and processes particular to the library and information profession. the conference provides an opportunity to bring together researchers and practitioners from all sectors to discuss and explore contemporary issues and topics of relevance to evidence based practice. the conference will be held at queensland university of technology, located adjacent to brisbane’s central business district and surrounded by the brisbane river and the botanic gardens. the organising committee, chaired by professor helen partridge, is planning a vibrant and stimulating conference onsite, and for those who cannot attend in person, a virtual conference is planned to extend the eblip conversation as widely as possible. to keep up to date with all eblip8 news, check out the conference website at eblip8.info or follow us on twitter @eblip8. for sponsorship opportunities, email eblip8@gmail.com for your exclusive prospectus.    evidence based library and information practice evidence based library and information practice 2011, 6.1 101 evidence based library and information practice news/announcements eblip6 news – registration open 2011. this is an open access article distributed under the terms of the creative commons-attributionnoncommercial-share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. eblip6 is fast approaching. we hope you will be able to join us 27th to 30th june at the university of salford, salford, greater manchester, u.k. we have an exciting programme with pre-conference workshops, over 50 confirmed paper presentations, four keynotes, posters, and three “madness sessions” where delegates, poster presenters and sponsors have an opportunity to address the audience for one minute! social events include an evening reception in the local museum and art gallery and the conference dinner will be held at old trafford football stadium (home of manchester united an impressive venue, even for non football fans!). for those who cannot make it to the event, we are hoping to make a limited part of the conference available online interactively. registration opens on march 1st 2011, with early bird rates until april 15th. we look forward to seeing you soon! for more information and to register, please visit http://www.eblip6.salford.ac.uk/ alison brettle and maria grant – eblip6 conference chairs http://www.eblip6.salford.ac.uk/� / evidence based library and information practice editorial   keeping evidence alive   alison brettle editor-in-chief reader in evidence based practice school of nursing, midwifery, social work and social sciences university of salford, uk email: a.brettle@salford.ac.uk      2014 brettle. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   welcome to the december 9(4) issue of eblip, our last issue for 2014. our first issue of the year began by describing change in the journal editorial team and this last issue of the year will do the same. i would like to say goodbye and thank you to our intern archana deshmukh, who has graduated from her ma and has taken up a post of research officer at the university of brighton, uk. archana was a great asset to the team, and i wish her well in her new post. our new intern, melissa griffiths, is another uk librarian and also an lis student at the university of aberystwyth, uk, and has already contributed with her work toward the production of this issue. i would also like to thank and say goodbye to our copyeditor molly des jardin and welcome on board two new copyeditors, julie evener and heather healy.   finally, at the end of a 3-year term of office, i am stepping down as editor-in-chief. i’ve been involved with the journal since its inception 9 years ago, and although i’ve thoroughly enjoyed my role as editor-in-chief, i think it is time to take a step back and enable another member of the team to take the journal forward into the next decade. i’m immensely proud of what the journal has achieved. as a bottom-up, practitioner focused journal, we have a good user base, along with a strong editorial team made up of practitioner researchers supported by a large team of peer reviewers. we are a zero budget publication that relies solely on volunteer professionals, and despite this, the journal has continued to grow year over year (measured by the numbers of users registered and article downloads). our focus continues to extend across library sectors from health to academic, to school and public libraries, and we incorporate evidence in various forms.   in my first editorial (brettle, 2012) i reflected on my own professional journey with eblip that began by examining and researching my own practice before i realized eblip as a concept even existed. although i will retain a link and role within the editorial team, i’m looking forward to having more time to undertake research and encourage others to question and evaluate their own practice. i also noted that it had been questioned whether eblip had a future (booth, 2011). i’m relieved that in the intervening years eblip continued and i see no signs of it going away. i believe the eblip journal is a crucial part in fostering debate within the eblip movement as well as ensuring that library practitioners are able to easily access evidence that is relevant to their practice. i look forward to seeing this journal continue to flourish under the new editor-in-chief, dr lorie kloda, and i wish lorie well in the role. lorie is well equipped to take over as she has previously held associate editor roles, initially for evidence summaries and more recently for articles.   “no librarians will die” is a phrase we often use within our editorial team, so we don’t always worry about achieving deadlines exactly on time. within healthcare, where the evidence based movement began, if evidence isn’t acted upon, at best resources will be wasted, but at worst patients will die. although “no librarians will die” if we don’t implement eblip in our practice, we do need to ensure that our services are healthy by being efficient and provide what our stakeholders need and value. so it is important that we continue to ask questions, to find evidence and implement it in our practice – thus keeping evidence based practice alive. i look forward to watching this happen within the eblip journal over the years to come.   references   booth, a. (2011). is there a future for evidence based library and information practice? evidence based library and information practice, 6(4), 22-27. retrieved from http://ejournals.library.ualberta.ca/index.php/eblip/   brettle, a. (2012). looking forwards and looking back. evidence based library and information practice, 7(1), 1-3. retrieved from http://ejournals.library.ualberta.ca/index.php/eblip/   conference paper   climatequal® and thinklets: using climatequal® with group support systems to facilitate discussion and set priorities for organizational change at criss library   audrey defrank associate dean dr. c.c. and mabel l. criss library university of nebraska omaha omaha, nebraska, united states of america email: adefrank@unomaha.edu   nora hillyer director of research services dr. c.c. and mabel l. criss library university of nebraska omaha omaha, nebraska, united states of america email: nhillyer@unomaha.edu      2013 defrank and hillyer. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – this article discusses a series of actions taken by the criss library at the university of nebraska at omaha to implement organizational change, using the climatequal® survey and facilitated discussions with thinktank™ group decision software. the library had experienced significant changes over a five-year period, with a renovation of the facility and three reorganizations resulting in a 50% staff turnover. recognizing the strain that years of construction and personnel changes had placed on the organization, there was a desire to uncover the mood of the employees and reveal the issues behind low morale, uneasiness, and fear.   methods – in november 2009, the library conducted a climatequal® survey to develop a baseline to assess the effectiveness of any changes. after the results were distributed to library faculty and staff, a series of two-hour facilitated discussions was held to gather opinions and ideas for solutions using thinklets, a pattern language for reasoning toward a goal. the group support system thinktank™ software was loaded onto computers, and employees were able to add their ideas anonymously during the sessions. finally, 12 employees (29%) completed a four-question survey on their perceptions of the facilitated discussions.   results – the facilitated discussions returned 76 sub-themes in 12 categories: staffing and scheduling issues, staff unity/teamwork, communication, goodwill/morale, accountability, decision-making, policy issues, skills and training, leadership, ergonomics/physical work environment, respect, and bullying. an advisory team culled the 76 sub-themes into 40 improvement strategies. five were implemented immediately, and the remaining 35 were scheduled to be presented to the faculty and staff via an online survey. participants’ perceptions of the facilitated discussions were mixed. eighty-three percent of respondents reported that they did not feel safe speaking out about issues, most likely because a supervisor was present.   conclusion – improving organizational climate is a continuous and iterative process that leads to a healthy environment.   introduction   the criss library at the university of nebraska at omaha (uno) has experienced exceptional level of change within the last five years. the library has undergone a complete physical transformation; a 30,000 square foot addition was completed in 2006 and a total renovation of the facility was completed in 2009. throughout the construction, the facility remained open and all services available to patrons.   not only did the library faculty and staff endure the environmental stress of a renovation, they were also affected by three reorganizations in a three-year time frame. the reorganizations changed job descriptions for 30% of the employees and resulted in a 50% turnover in staff from resignations, layoffs, and retirements. the personnel changes left the remaining employees feeling uneasy; and while there is a high level of achievement among the staff, an undercurrent of low morale, distrust, and fear remained.   after the completion of the building renovation and a change in leadership, the organizational focus returned to collections, services, and employees after long being on facilities issues.   recognizing the strain that years of construction and personnel changes had placed on the organization, there was a desire to uncover the mood of the employees and reveal the true issues behind the low morale, uneasiness, and fear. after doing some research on organizations, change, and the effects of change on employees, it was decided to use the climatequal® survey for assessment of the library staff.   overview/background and climatequal®   the criss library set out to determine its organizational health by measuring the diversity and climate of the organization. as lowry and hanges (2008) indicate, the climate of an organization helps employees interpret and understand what behavior is rewarded, supported, and expected. a healthy organization creates climates that show that teamwork, diversity, and justice are valued and there is a strong concern for customers.   beckhard described the genesis of organizational development in organization development: strategies and models:   “today there is a need for longer-range, coordinated strategy to develop organization climates, ways of work, relationships, communication systems, and information systems. it is out of those needs that systematic planned change efforts – organizational development – have emerged” (beckhard, 1969, p. 8).   the father of organizational development in academic and research libraries, duane webster, identified the following principles for improvement of organizations: “the importance of interpersonal competence; participation leading to commitment; the importance of groups and teamwork; and importance of those who will implement a change being involved in the planning of that change” (sullivan, 2009, p. 314). some of these same principles were repeated as elements of organizational development described by karen holloway: putting decision-making closer to people doing the work; improving group dynamics, organizational structure, and organizational culture; learning how to work collaboratively and across hierarchies; and building trust (2004). the organizational climate and diversity assessment (ocda) used the principles and elements of organizational development and described them as climates (lowry & hanges, 2008). questions were developed for the ocda, which later evolved into the climatequal® assessment, to help libraries discover their strengths and weaknesses within each principle or climate.   the criss library used climatequal® tool to survey library employees and develop a baseline to assess the effectiveness of any changes. the climatequal® survey addresses climates for diversity, teamwork, learning, and fairness. the survey was administered in november 2009 and results were received in december 2009. the results were based on a seven-point likert scale.  with some exceptions, a higher mean score indicates a stronger or healthier climate. the criss library results showed healthy climates in several areas but also indicated three areas where changes were warranted. the criss library work environment scored well on interpersonal justice (m = 5.86), informational justice (m = 5.02), a healthy climate for leadership (m = 5.69), a healthy climate for deep diversity (m = 5.18) and demographic diversity for race (m = 6.74), gender (m = 6.47), rank (m = 5.20), and sexual orientation (m = 6.40), organizational citizenship behaviors (m = 5.06), interpersonal conflict (m = 2.66, note scale with reversed coding), and task conflict (m = 3.35, note scale with reversed coding). the three areas where the mean scores were low included distributive justice (m = 3.29), procedural justice (m = 3.98), and structural facilitation of teamwork (m = 3.79).   criss library’s climatequal® results   with a better understanding of organizational development, research was conducted for additional clarification on the three climates with the lowest mean scores at the criss library: distributive and procedural justice and the structural facilitation of teamwork. the climatequal® tool defines distributive justice as the degree to which staff perceive that rewards are fairly distributed upon performance, and procedural justice as the degree to which staff perceive the procedures that determine the distribution of rewards are uniformly applied. (association of research libraries, n.d.). the climate for teamwork and the structural facilitation of teamwork is the degree to which staff members perceive that teamwork is valued by the organization and to which they perceive that they are valued as team members.   in general, distributive justice is related to specific attitudes or perceptions of the fairness of organizational outcomes or processes received in a given transaction such as pay satisfaction and job satisfaction (greenberg & colquitt, 2005). individuals evaluate and compare the outcome they receive to a standard or rule or to the outcome received by a coworker. distributive justice perceptions are positively associated with job satisfaction, organizational commitment, and trust, and negatively associated with organizational withdrawal (chory & kingsley westerman, 2009). negative associations of distributive justice can contribute to rumor spreading, counter-productive work behaviors, conflict at work, faking illness, and damaging or wasting organizational resources or equipment (chory & kingsley westerman, 2009).    procedural justice is more strongly related to global attitudes such as organizational commitment and group commitment (greenberg & colquitt, 2005). procedural justice in the group context demonstrates that individuals care about fairness because of their relationships with the groups to which they belong (greenberg & colquitt, 2005). procedural justice can be defined as the perception of the fairness of the processes used to arrive at outcomes. it is the individual’s perception of the fairness of the process components of the social system that regulates the distribution of resources. procedures are judged on their consistency of application, alignment with prevailing ethical standards, the degree of bias present, their accuracy and correctability in application, and the extent to which they represent all people concerned. fair procedures ensure acceptance of policies such as smoking bans, pay systems, parental leave policies, and disciplinary actions. positive procedural justice is associated with trust in management, job satisfaction, and organizational commitment. negative or low procedural justice can lead to counterproductive work behaviors, conflict at work, and the use of organizational revenge strategies (chory & kingsley westerman, 2009).   structural facilitation of teamwork was another opportunity area where the criss library scored lower than other academic libraries. the criss library work environment scored a mean of 3.79 compared to 4.24 for all institutions, placing uno below the average. only 40% of criss library employees responded positively to the question about the structural facilitation of teamwork, which compares to the mean of 48% for all institutions. teams, as defined by baughman, are “small groups of staff working on a common purpose” and teamwork is the environment that is created to foster how the members of a group work together” (2008, p. 294). moreover, baughman describes a true team as one empowered to make decisions, improve processes, and implement strategies to better serve the user.  a team can add to the success of an organization by taking ownership of identifying ways to improve processes, promote continuous learning and development, and increase innovation and risk-taking. she goes on to explain that libraries that develop into learning organizations focus on customer needs and building a culture of continuous learning for team members.   the criss library scored the highest on the climatequal® survey in the climates for diversity. the valuing diversity climate, defined as the degree to which equality between minorities and majorities is valued, showed 71% of criss employees responded positively. in response to race, which is the extent to which the library supports racial diversity, 96% responded positively. another climate where criss library employees responded positively was interpersonal justice (84%) – the degree to which one perceives there is fairness and respectfulness between employees and supervisors. finally, organizational citizenship behaviors (71%) – the degree to which employees perceive that “professionalism,” politeness, and care is exhibited within the organization, was another positive climate at the criss library. some comments:   ·         “overall this is a very good place to work. folks are generally helpful, good natured and open minded.” ·         “our library caters greatly to the patrons. there is a great working atmosphere at the service desks, and you know that other employees are friendly and ready to help you, should you require it.”   in contrast to the healthy climates, three areas from the survey indicated opportunities for improvement: distributive justice, procedural justice, and structural facilitation of teamwork. in the climates for justice/fairness, distributive justice, and procedural justice the criss library ranked lower (22% and 35% employees responded positively respectively) than compared to all institutions (30% and 47% respectively). in the area of climate for teamwork, the structural facilitation of teamwork, the criss library received a lower score (m=3.79) than all institutions (m=4.26). furthermore, 43% of criss library employees perceive they do not have as much influence over their teams as other institutions’ employees, where 62% responded positively. some comments on the teamwork issue:   ·         “staff members, librarians, and administrators need to be more open to helping other departments within the library when asked.” ·         “i feel communication and teamwork are two areas at the library that need to be addressed.”   criss library employees also expressed concern in the climate for psychological safety, defined as the degree to which employees feel the organization is a safe environment for offering opinions and taking risks. the mean score for the criss library was 4.52 compared with 4.95 for all institutions. criss library employees expressed concerns regarding expressing ideas and opinions, and fear that theirs is not a safe environment for risk-taking:   ·          “there is a great deal of fear in this organization.” ·         “this organization is a mess. people don’t trust. communication is the pits.” ·         “. . . they were out of favor with administration. it created a climate of fear across the library. this is why people are still afraid to try new things or offer dissenting opinions.”   there were several comments regarding the absence of rewards in the organization. the mean score for the climate for continual learning shows that criss library employees felt they were not as encouraged to express new ideas and that their ideas were not accepted or rewarded as those from other libraries. the mean score for criss library was 5.05 compared to 5.28 for all institutions.   ·         “the rewards questions were very hard to answer because the library doesn’t give reward.” ·         “there are attempts at saying thank you but i’d say most people do not feel personally rewarded for their work.” ·         “it would be nice if the directors or the dean provided greater recognition and/or rewards (not just monetary, but treats, prizes or even paper certificates) to those departments or individuals who go ‘above and beyond’ to serve our patron population.”   tables 1 and 2 break out the lowest and highest mean scores, by percentage of respondents assigning a ranking of 5 or above on each 7 point scale.   table 3 shows the top three opportunity areas for all departments and the range of mean scores. all departments, with the exception of one (who did not have the minimum number of responses for reporting), had the same three lowest scoring climates (opportunity areas), but in varying rank order.   after the survey: group support systems (gss) and thinklets   the receipt of the survey results coincided with the semi-annual climatequal® partners meeting at the 2010 ala midwinter meeting in boston. a number of partners spoke informally on their experiences with survey administration and the common theme running through those discussions was the importance of library staff involvement in the identification of interventions and solutions. this concept was communicated to the criss library climatequal® advisory team, and the group began to discuss ways to garner feedback from library staff. one of the team’s members is a senior fellow at the university of nebraska at omaha’s center for collaboration science (ccs), an experienced facilitator, and knowledgeable about the ccs’ use of group decision software to facilitate meetings both on-campus and in the omaha business community.   table 1 all library organization climate lowest five ranked climatequal® climate percentage of respondents assigning a ranking of 5 or above (7 point scale) organizational climate for justice      distributive justice   22.22%      procedural justice 34.62% climate for teamwork      structural facilitation of teamwork   40.00% climate for customer service 62.26% climate for psychological safety 62.26%   table 2 all library organization climate highest five ranked climatequal® climate percentage of respondents assigning a ranking of 5 or above (7 point scale) climate for demographical diversity      race   95.74%      gender 90.38%      sexual orientation 90.00% organizational climate for justice      interpersonal justice   84.44% leadership climate       leader-member relationship quality 83.67%   table 3 top three opportunity areas for all departments climatequal® climate range of mean scores organizational climate for justice      procedural justice    2.00 – 4.88 climate for teamwork      structural facilitation of teamwork    2.75 – 4.36 organizational climate for justice      distributive justice    2.44 – 4.14   the advisory team chose to use the group decision software based on prior experience using it in other contexts at uno.  in addition to being a very productive and successful system, it is fun and engaging to use.  there is a level of anonymity that can provide psychological safety to participants, which they may not experience using other traditional brainstorming systems, as well as providing a focus on the quality of the feedback and not on the personality of the person providing it. we felt the anonymity was an important factor given the general feeling of mistrust among library faculty and staff.   the system developed at the ccs uses “thinklets.” a “thinklet” is “the smallest unit of intellectual capital required to create one repeatable pattern of thinking among people working toward a goal” (briggs, de vreede, nunamaker, & tobey, 2001. p. 2).   briggs and de vreede (2009) have developed over sixty thinklets that can be configured and used within a group decision system and can “encapsulate the components of a stimulus used to create a single repeatable, predictable, pattern of thinking among people working toward a goal” (briggs, de vreede, nunamaker, & tobey, 2001, p. 2). it was decided that the criss library would use the thinktanktm group collaboration software, and employ the freebrainstorm, fastfocus, and priorityvote thinklets.   facilitated discussion process   the climatequal® survey was administered to the following library departments, which align with the current organizational reporting structure: administrative services, collections, leadership team, patron services, research services, and virtual services. likewise, the facilitated discussions were conducted among these same departmental groups, with the exception of student assistants, who did not participate in the initial facilitated discussions. there are plans to hold conversations with student assistants later in the process.   prior to the scheduled discussions, each departmental group was provided a summary report of climatequal® results. the report included both the highest and lowest scoring climates for their department as well as the library as a whole. faculty and staff were asked to reflect on the lowest-scoring climates, referred to as “opportunity areas” and to begin thinking of possible answers to this question: over the next year, what can we do to improve our work environment? given the complexity of organizational development and possible interventions to address opportunity areas, the one-year time frame was presented in order to provide a manageable time frame for our initial work.   two-hour blocks were scheduled to maximize participation from faculty and staff.  sessions were facilitated by faculty and graduate students affiliated with the center for collaboration science as well as faculty colleagues from the university of nebraska-lincoln libraries climatequal® team. the gss software was installed on library laptops and each participant was given a computer with which to work. facilitators used the thinktank™ group facilitation software to garner answers to the aforementioned question. a page was displayed for each participant in the session and the freebrainstorm thinklet was used to provide participants the opportunity to share their particular points of view; and it also enabled them to quickly see the bigger picture and to diverge from comfortable patterns of thinking (see figure 1). participants were instructed to move to another page where they could either enter a new idea or comment on the other ideas that were entered onto that page by another participant. this thinklet activity varied by the size of the group, but ranged from 20 minutes to over an hour in length.     the fastfocus thinklet was used in the next step of the process to quickly extract a clean list of key issues (see figure 2). each participant was assigned a page and given the opportunity to choose the idea they felt was most important from that page. each participant was given two “turns” to choose important ideas. once each participant had identified their two most important ideas, the facilitator verbally engaged the group to refine this list to eliminate duplication and to ensure that all agreed on and understood the idea presented.    the final thinklet employed was priorityvote which is a simple ranking of the most important ideas (see figure 3). the groups were asked to individually rank the list and the top five or six ideas for each group session remained.   employee survey perceptions of the facilitated discussions   to gain more understanding and insight of employees’ perception of the facilitated discussions, a four-question survey was distributed to all library employees, via surveymonkey®. twelve employees answered the survey, a 29% response rate. three short-answer essay questions were asked: “in your experience during the climatequal® facilitated discussion, what worked well?”; “what did not work well?”; and “what could have been done differently?” the fourth question was a likert-scale matrix question where the respondents were asked to strongly agree, agree, disagree or strongly disagree with four statements: 1) differing opinions were openly discussed; 2) it was safe to speak up without fear of a negative effect; 3) i am satisfied with my involvement at the facilitated discussion; and 4) there was good cooperation within my group.   figure 1 freebrainstorm thinklet   figure 2 fastfocus thinklet   the response for questions 1 and 2 was mixed. for question 1, five respondents stated they felt the anonymity of the process worked well. five respondents for question 2 answered that anonymity did not work well with one comment stating anonymity was compromised in the facilitated discussions. additional comments provided from the survey indicated participants could tell who was typing; others were uncomfortable expressing any opinions if their supervisor attended the same facilitated discussion. additionally, 33% of the respondents (n=4) felt nothing worked well in the discussions.   question 3 asked what could have been done differently in the facilitated discussions. most people responded by writing that they wished they could have chosen their own group rather than joining their department in the discussions. several reasons explaining this response can be found in the agree/disagree matrix questions. a large number (83%) did not feel safe speaking out about issues, most likely because of a supervisor present. only 50% of the respondents felt opinions were openly discussed and were satisfied with their involvement in the discussions. even though people did not feel safe speaking in their group, a majority of respondents agreed that there was good cooperation in their group.    results of thinktank™ sessions in all groups   reports were returned for each departmental session, which included transcripts from the freebrainstorm sessions and results from the priorityvote. all departmental sessions were combined to provide 12 general themes for the library as a whole:   staffing and scheduling issues (5) staff unity/teamwork (5) communication (5) goodwill/morale (4) accountability(4) decision-making(4) policy issues (4) skills and training (3) leadership(3) ergonomics/physical work environment(3) respect(3) bullying (2)   the numbers in parentheses represent the number of groups identifying the theme as a priority (total number of groups, n=6). each of the 12 themes had between 3 and 10 related sub-themes and strongly corroborated sub-themes (priority ranked by over one-half of the generating group) were noted. figure 3 priorityvote   thinkthank™ sessions and climatequal® results   recall that the question asked in the facilitated discussions was “over the next year, what can we do to improve our work environment?” while some of the groups answered that question in the context of the opportunity areas (lowest scoring climates) identified in the climatequal® report for their department, some did not. thus, it is somewhat difficult to draw parallels between the feedback from the facilitated discussion to the climatequal® results. however, based on keywords and concepts delivered in facilitated discussions, some associations were made (see table 4). for example discussion related to “staff unity and teamwork” were associated with the climatequal® concept of “structural facilitation of teamwork,” “communication” relates to “climate for psychological safety,” “goodwill/morale” relates to a number of different climatequal® scales such as “climate for procedural justice,” “job satisfaction,” “climate for psychological safety,” and “organizational citizenship behavior.”  similarly facilitated discussion themes on “policy issues” relate to “climate for procedural justice,” “leadership” to “climate for leadership,” “respect” to “team psychological empowerment” and “bullying” to “climate for interpersonal justice.”   strategies for improvement: the next steps   the facilitated discussions returned 76 sub-themes under the 12 general themes. the advisory team culled the 76 sub-themes into 40 statements, or improvement strategies by removing duplicates such as “make people accountable” and “develop a way to make people accountable” and combining like statements such as “reorganize circ area” and “optimize work spaces” into “optimize work spaces for all departments as needed so staff can do their job tasks effectively and efficiently.”   of the 40 improvement strategies, there were five that could be implemented immediately: the courtesy committee was reinstated, and reconceived as the positive employee recognition committee (perc). this committee would not only oversee the social activities but also organize and advise the leadership team and the dean on a staff recognition program. a mechanism for staff to provide anonymous ideas, comments, and feedback was developed by the communications advisory group (cag), which was also formed with representatives from each library department. several members of the leadership team and library supervisors have completed or are scheduled to participate in a new campus leadership program; and lastly, a current project to collate policies on the library’s internal wiki will be followed by an internal review of all policies.   table 4 mapping themes from facilitated discussions to climatequal® core concepts general themes from facilitated discussions climatequal® core concepts staff unity/teamwork structural facilitation of teamwork communication climate for psychological safety goodwill/morale climate for procedural justice; job satisfaction; climate for psychological safety; organizational citizenship behavior policy issues climate for procedural justice leadership climate for leadership respect team psychological empowerment bullying climate for interpersonal justice   the remaining 35 improvement strategies are scheduled to be presented to the faculty and staff via an online survey where they will be asked to rank the strategies in order of importance. the resulting list is where the criss library will take the first solid steps toward organizational change and a healthier organizational climate.   conclusion   to sum up, the criss library conducted the climatequal® survey during the fall semester in 2009. the library had experienced numerous changes due to a three-year library renovation and several personnel resignations and library reorganizations. there was an over-riding perception of mistrust, fear, and uncertainty that needed to be addressed. the first step in addressing these negative perceptions was to administer the climatequal® survey to gather data for a better understanding of staff perceptions. the next step was to report the climatequal® results to library staff and begin discussions on goals and solutions for improving the organizational climate. the third step was to identify areas of concern within the organization to address. once those organizational areas were identified, they were prioritized and goals with associated solutions were developed. improving organizational climate is a continuous and iterative process that leads to a healthy environment.   acknowledgement   the authors would like to acknowledge contributions of the members of the criss library climatequal® advisory group: lynn harland, elaine westbrooks, and sarah rowe as well as the faculty and facilitators from uno center for collaboration science: andy callens, roni reiter-palmon, and ross rippe.     references   association of research libraries. (n.d.). core scales—concepts measured in the survey. retrieved 24 may 2013 from http://www.climatequal.org/about/concepts/core-scales   baughman, m. s. (2008). assessment of teams and teamwork in the university of maryland libraries.  portal: libraries and the academy, 8(3), 293-312. doi:10.1353/pla.0.0003   beckhard, r. (1969). organization development: strategies and models. reading, ma: addison-wesley.   briggs, r. o., de vreede, g.-j., nunamaker, j. f., jr., & tobey, d. (2001). thinklets: achieving predictable, repeatable patterns of group interaction with group support systems (gss). proceedings of the 34th annual hawaii international conference on system sciences, 9, 1057-1065. doi:10.1109/hicss.2001.926238   briggs, r. o., & de vreede, g.-j. (2009). thinklets: building blocks for concerted collaboration. nebraska: briggs en de vreede.   chory, r. m., & kingsley westerman, c. y. (2009). feedback and fairness: the relationship between negative performance feedback and organizational justice. western journal of communication, 73(2), 157-181. doi:10.1080/10570310902856055   greenberg, j. & colquitt, j. a. (2005). handbook of organizational justice, mahwah, nj: lawrence erlbaum associates.   holloway, k. (2004). the significance of organizational development in academic research libraries. library trends, 53(1), 5-16.   lowry, c. b. & hanges, p. j. (2008). what is the healthy organization? organizational climate and diversity assessment: a research partnership. portal: libraries and the academy, 8(1), 1-5.   sullivan, m. (2009). duane webster's contribution to organization development in academic and research libraries. portal: libraries and the academy, 9(3), 313-316.   thinktank (version 2.4.1) [computer software]. (n.d.). greenwood village, co: groupsystems.   evidence summary   frequent internet users may prefer more health care information and participation in decision-making   a review of: xie, b., wang, m., feldman, r., & zhou, l. (2013). internet use frequency and patient-centered care: measuring patient preferences for participation using the health information wants questionnaire. journal of medical internet research, 15(7), e132. doi:10.2196/jmir.2615   reviewed by: eamon c. tewell reference & instruction librarian long island university, brooklyn campus brooklyn, new york, united states of america email: eamon.tewell@liu.edu   received: 15 sep. 2013   accepted: 17 feb. 2014      2014 tewell. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine whether there is a significant relationship between patients’ frequency of internet use and their health care information and decision-making preferences.   design – cross-sectional questionnaire survey.   settings – undergraduate classes at a large state university and senior-oriented computer classes at public libraries and senior centers.   subjects – 438 respondents, including 226 undergraduates (mean age 20) and 212 community-dwelling older adults (mean age 72).   methods – respondents were administered the health information wants questionnaire (hiwq), a 21-item instrument designed to measure preferences for 7 types of health information and decision-making, in group settings.   main results – the younger age group spent significantly more time online compared to the older age group. frequent internet users in both populations expressed an overall preference for more information regarding diagnosis, but less information for psychosocial and health care provider concerns. internet use was positively correlated to the overall preference rating, leading the researchers to suggest that, as a whole, regular internet users prefer more information and independence in decision-making.   conclusions – the study concludes that internet use frequency is associated with an overall preference for obtaining health information and participating in decision making. internet use as related to different types of preferences is inconsistent. age was not found to be associated with the overall preference rating, and time spent online is proposed to be a stronger indicator of respondents’ health information preferences. the authors suggest that future studies utilizing the hiwq take a longitudinal approach in order to better track how patient preferences for information may evolve over time.     commentary   the internet has revolutionized the ways in which patients seek and obtain health information. due to the decentralization of information that traditionally required the consult of a medical specialist, patients have the ability to select their level of participation in personal health care decisions. the internet introduced two primary challenges to the medical profession, as noted by blumenthal (2002): those that are cognitive (creating unmediated access to medical information), and those that are collegial (allowing users to make decisions regarding the health care professional they choose). this democratization of access has led to a conceivable shift in patient preferences and involvement regarding one’s health care. while a considerable body of research addresses the health information seeking behavior of various populations, the authors of this study attempt to compare the internet use of two age groups and their preferences for health information. by finding a marginal correlation between online activity and patient participation, this paper’s contribution to the literature is that it provides some degree of support for the significance of internet use as it relates to the provision of patient-centered health care.   the study’s strengths lie in its description of both methodology and process for data collection, which with the exception of the survey instrument’s omission, are presented adequately. the discussion of the results is grounded in the literature on changes brought to the medical profession by online information. statistical significance of the results was tested and verified, and appropriate tables and graphs visualizing the data accompany the text. while these factors contribute favorably to the strength of the evidence presented, shortcomings in the sampling technique, data collection, and data analysis significantly reduce the validity of the findings.   limitations of the study include two that are common to research relying upon survey data: the use of a convenience sample and biases inherent in self-reported responses. the convenience sample, which the authors acknowledge as an issue for external validity, can lead to unrepresentative results and the inability to generalize findings. because one population selected was older adults in computer workshops, this group is likely to have a greater interest in improving their computer skills and may not reflect the attitudes of adults their age not in computer workshops. additionally, the study notes that a survey respondent’s stated preference may not reflect actual practice, given that the survey can measure only how subjects state they would act in a hypothetical situation. more intrinsically problematic for the study at hand is the authors’ selection of independent variables, as the sampling is based on the single demographic variable of age. selecting two age groups at each extreme of internet use not only ignores other age groups that use the internet, but fails to account for a host of other factors that have been shown to be more robust predictors of online health information seeking, including socioeconomic status, gender, and internet use experience.   the slightly unusual method for data collection presents additional issues. surveys are frequently conducted online, or if in print, distributed through postal mail, yet the authors opted to collect survey responses in group settings. this potentially introduces issues of respondent anonymity and confidentiality, as well as researcher conflict or bias if involved in the survey distribution and collection process. the data analysis would be strengthened by the use of regression analysis instead of analysis of variance, as the former allows for the proper investigation of the relationships between the myriad variables that should be accounted for. the researchers’ conclusions and recommendations do not account for several factors, including the omission of suitable predictors; the assumption that individuals rely solely on the internet when seeking health information; the fact that the internet represents a variety of information sources in itself, each of varying quality and type; and the fact that information source characteristics, such as reliability, have a considerable impact on its use.   the significance of this research lies less in its findings than in the usefulness of the survey instrument for further research. in terms of recommendations for health care professionals, the researchers suggest that internet use can be an advantageous way to consider the dimensions of patient interest in participating in their own care process. to the benefit of researchers, this study contributes to the further validation of hiwq, a survey that expands upon previous instruments used for measuring patient preferences and that would be of use in future work conducted on the topic. at its best, the internet’s impact on how the general public accesses health information is one of democratization and empowerment, as the authors of this study propose. while the evidence is modest due to the limitations of the survey methodology, sampling technique, and data analysis reported, it represents a step towards considering patient involvement with and confidence in the medical decision-making process.     references   blumenthal, d. (2002). doctors in a wired world: can professionalism survive connectivity? milbank quarterly, 80(3), 525-546. doi:10.1111/1468-0009.00021   microsoft word ed.doc evidence based library and information practice       editorial    interested in research? start here.      lindsay glynn  editor‐in‐chief    public services librarian and instruction coordinator, health sciences library   memorial university of newfoundland   st. johnʹs, newfoundland, canada  e‐mail: lglynn@mun.ca      © 2007 glynn. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      hey, there it is.  an idea.  a question.   finding an answer to that question would  be really useful, wouldn’t it?  i’ll bet that  other people would find the answer useful  too, since they probably have asked  themselves that very same question.  has  anyone else made an attempt to answer that  question in the literature?  nope.   hmmm...you’ve found a gap in the  literature on an idea that you have identified  and in which you are interested.  looks like  you have a research project on your hands.      it is at this point that many people become  intimidated.  the thought of designing a  study, collecting and interpreting data and  maybe even publishing the results can be  daunting, whether you are familiar with  research methods or not.  where do you  begin? how do you determine the study  design?  how do you narrow down your  idea to a manageable hypothesis?  what  statistical tests should you utilize?  in other  words, help!       even the most seasoned researchers don’t  do it all on their own.  they, just like new  researchers, consult guides and textbooks  and enlist the help of statisticians and coders.   a research project is a huge undertaking,  but it is more manageable than you might  think.  in this issue of evidence based library  and information practice you will find a  feature section on research.  the articles in  this section are designed to act as guides for  those new to research or for those needing a  refresher.  diane lorenzetti and lisa given  take a close look at qualitative and  quantitative research design in their articles.   both these articles will help you choose the  most appropriate study design based on  your question type.  research can be costly  and therefore, funding is often sought.   lynne langille and theresa mackenzie  provide valuable and practical advice on  how to write a successful grant application.   in her two articles, lisa goddard identifies a  wealth of data that is available to you in  web server logs and specific library  applications.  the possibilities of using such  data for research are endless.  once you  have all your data collected, gillian byrne  has some tips on statistical analysis to make  sense of your information.  the expert  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:1    2 advice in these feature articles will prove to  be useful to you time and time again in  planning your project.  good luck!    while the feature articles provide guidance,  information and advice, also useful in  helping you to design a research study are  the articles and evidence summaries in this  (and every other) issue of eblip.  as critical  appraisals of original research articles, the  evidence summaries describe and identify  issues related to methodology, analysis and  conclusions.  the insight that they provide  in terms of how well a study is designed  and carried out is invaluable and can be  inferred when planning your own study.   the same, of course, applies to the original  research articles: it is helpful and prudent to  find a research article that addresses a  question type similar to yours.  while your  project is likely not exactly the same, you  will undoubtedly obtain ideas and  inspiration on how to design your own.  perhaps one of the most useful resources,  however, is your colleagues.  colleagues  will provide different views, point out  things that you hadn’t thought of or  previously noticed, identify gaps or  redundancies, and most importantly,  provide support and encouragement.     evidence summaries in this issue of eblip  will take you to public libraries in norway,  african american graduate students with  library anxiety, and interlibrary lending  departments feeling the effects of electronic  journals.  articles will examine movement  and progress in evidence based library and  information practice over the last several  years and the eblip research‐practice gap.   the commentary examines ebp’s close  relative, narrative based practice.     fill up your coffee cup, find a comfortable  chair and read on.         article   a comparison of beginning and advanced engineering students’ description of information skills   kerrie anna douglas visiting assistant professor school of engineering education, purdue university west lafayette, indiana, united states of america email: douglask@purdue.edu   amy s. van epps associate professor of library science, engineering librarian purdue university, siegesmund engineering library west lafayette, indiana, united states of america email: vanepa@purdue.edu   brittany mihalec-adkins research assistant educational studies, purdue university west lafayette, indiana, united states of america email: bpadkins@purdue.edu   michael fosmire professor of library science head of physical science, engineering, and technology division of purdue libraries purdue university libraries west lafayette, indiana, united states of america email: fosmire@purdue.edu   şenay purzer assistant professor school of engineering education, purdue university west lafayette, indiana, united states of america email: purzer@purdue.edu   received: 8 dec. 2014      accepted: 13 may 2015      2015 douglas, van epps, adkins, fosmire, and purzer. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – the purpose of this research was to examine how beginning and advanced level engineering students report use of information when completing an engineering design process. this information is important for librarians seeking to develop information literacy curricula in the context of engineering design.   methods – researchers conducted semi-structured interviews about information strategies used in engineering design with 21 engineering students (10 first and second year; 11 senior and graduate). researchers transcribed interviews and developed an inductive coding scheme. then, from the coding scheme, researchers extracted broader themes.   results – beginning level engineering students interviewed: (a) relied primarily on the parameters explicitly given in the problem statement; (b) primarily used general search strategies; (c) were documentation oriented; and (d) relied on external feedback to determine when they had found enough information. advanced level engineering students interviewed: (a) relied on both their own knowledge and the information provided in the problem statement; (b) utilized both general and specific search strategies; (c) were application oriented; and (d) relied on self-reflection and problem requirements to determine when they had found enough information.   conclusion – beginning level students describe information gathering as externally motivated tasks to complete, rather than activities that are important to inform their design. advanced level students describe more personal investment in their use of information through consideration of information based on their prior knowledge and questioning information.  future research should consider how to best support beginning level engineering students’ personal engagement with information. introduction   with the vast amount of information readily available to current students on the open web and through library resources, the skillset necessary to navigate through information and use it appropriately is arguably one of the most important factors for educational success (bruce, 2004). all students, both undergraduate and graduate, need to possess information literacy skills to manage the rapidly changing technological environment. in particular, future and current engineers are challenged to be adept at information literacy as rapid research and technological advances in their fields generate new and changing information that directly impacts their daily work. throughout their careers, engineers must stay current within their field and incorporate new information to inform their own professional development (fosmire & radcliffe, 2014). innovation and future technology are, at least in part, influenced by information literacy skills (fosmire & radcliffe, 2014).   in this qualitative study, we report data collected through interviews with students rather than relying on surveys or other quantitative measures of students’ thoughts. this approach is especially helpful in uncovering assumptions, conceptions, and strategies students might employ as they solve open ended engineering problems that might be difficult to capture, for example, in a more formal survey or analysis of work products. by discovering the information strategies students use in an engineering context, the results will provide insight into the misconceptions that need to be corrected, as well as areas of strength that can be built upon in instructional interventions.    information literacy skills are an important part of undergraduate and graduate education. for example, the association of college and research libraries (acrl) standards state an information literate person should be able to effectively search for, identify, evaluate, use, and document information (acrl, 2000). in addition, the document criteria for accrediting engineering programs, created by the body which accredits engineering programs, abet (accreditation board for engineering and technology), includes a requirement that students demonstrate life-long learning skills (2013). as we prepare future engineers, it is necessary to cultivate habits of information literacy that will serve as the prerequisites for life-long learning. according to a 2006 report, 91% of responding engineering employers rated life-long learning as either essential, highly important, or moderately important (lattuca, terenzini, & volkwein, 2006). yet, lattuca and colleagues note that in the years since this criterion was originally released, growth in preparing engineering students in life-long learning skills has been relatively stagnant.   research directed specifically at engineering students’ information literacy is sparse. of the research conducted, some inconsistency has been noted between first-year engineering students’ self-assessment of their skills and actual task performance (atman, cardella, turns, & adams, 2005; atman, chimka, bursic, & nachtmann, 1999; douglas, wertz, fosmire, purzer, & van epps, 2014; wertz, purzer, fosmire, & cardella, 2013). it can be difficult to ascertain whether inconsistencies are genuine self-inflation or artifacts of the surveys used. as davidson (2005) noted, there is a sea of qualitative assumptions behind every quantitative measure. the purpose of this current research is to gain a deeper understanding into how engineering students discuss use of information literacy principles in the context of engineering design. the implications of this study include specific recommendations to inform decision-making among academic librarians and areas to target for curricular interventions.    literature review   information strategies of engineers   dym, little, orwin, and spjut (2004) define engineering design as “a systematic, intelligent process in which designers generate, evaluate and specify designs for devises, systems or processes whose form(s) and function(s) achieve clients’ objectives and users’ needs while satisfying a specified set of constraints” (p. 6). while there are many models of the engineering design process, it is commonly understood to result in the creation or adaptation of a product, system, or service after a process of feedback loops and iterations (fosmire & radcliffe, 2014).   the use of information in engineering design is context dependent. one of the roles of the engineer is to determine the type and depth of resources needed to complete the project (ellis & haugan, 1997; tenopir & king, 2004). engineers have been characterized as using a “least effort” approach to information gathering. that is, their goal is to find “reliable answers to specific questions,” in contrast to scientists who are motivated more by deeper understanding of concepts (pinelli, 1991). further, engineers try to minimize loss (e.g., of time or performance) rather than maximize gain of finding the perfect solution (pinelli, 1991). engineers value accessibility above all else and rely on colleagues, personal knowledge, and personal collections as the most desirable sources of information (e.g., engel, robbins, & kulp, 2011; gerstberger & allan, 1968; leckie, pettigrew, & sylvain, 1996). however, engineers with advanced degrees (kwasitsu, 2003) and exposure to information literacy coursework (holland & powell, 1995) consult formal information sources (e.g., journals, standards, patents) at a higher rate than those without. in recent years, google has become an important first-resort method of professional engineers as well (allard, levine, & tenopir, 2009).     ingwersen and jarvelin (2005) conceptualize information use as centered on work tasks, where an engineer determines knowledge gaps between the task and their ability to complete it. ellis and haugan’s (1997) behavioral model for engineers includes processes of surveying, chaining, monitoring, browsing, distinguishing, filtering, extracting, and ending. recently, there has been a movement to combine information literacy and engineering design processes in an effort to cultivate information literacy skills and life-long learning among engineering students (fosmire & radcliffe, 2014). fosmire (2012) applied kuhlthau’s (2004) information search process (isp), which comprises the stages of initiation, selection, exploration, formulation, collection, presentation, and reflection to engineering design. certainly, the engineering design process extends beyond information literacy; however, the skills needed to seek, evaluate, apply, and document information are essential to successful design (purzer & wertz, 2014). johnson and simonsen (2015) condense the above (and additional) information process models into the core tasks of start, survey, evaluate, gather, and finish. the above process models show a great deal of similarity, which gave the current authors confidence in using kuhlthau’s isp model as the basis of interview questions. while all the process models show similar stages, kuhlthau’s model is more highly tuned to the information components, so provides the opportunity for more focused questions.    engineering students’ information literacy skills   prior research related to undergraduate students’ information literacy has shown that while students report an overall confidence in skills associated with il, such as evaluating information, there are aspects in which students do not feel adequately prepared. for example, head and eisenberg (2010) found 84% of undergraduates surveyed had difficulty getting started in the search process and synthesizing information. similarly, in a study of first-year students, 74% struggled with online searches and 43% reported problems making sense of all of the information gathered (head, 2013). engineering students tend to use library resources less frequently than in other disciplines (collins & stone, 2014) yet, is unclear why, highlighting the need for more research to understand engineering students’ information literacy skills and habits.   emerging engineering education research has pointed to areas where librarians and instructors could further support engineering students in information literacy. for example, researchers have found differences between novice engineers and expert engineers in how they use information to make design decisions. in particular, expert engineers seem place a higher value on the role of information in solving design problems than do beginning students. mosberg et al., (2005) found that when engineering experts were asked to select the most important aspect of design, they ranked gathering information as the fourth highest of 24 design activities. atman, adams, cardella, turns, mosborg, & saleem (2007) found experts collected substantially more information, over a larger breadth of topics, than students when engaged in design. although engineering students have a strong self-perception of their information literacy skills (ross, fosmire, wertz, cardella, & purzer, 2011), their actual performance was poor when asked to identify reliable sources and appropriate use of information to support a design decision (wertz, purzer, fosmire, & cardella, 2013).  even masters-level engineering students did not demonstrate an effective search process or awareness of library sources or services (johnson & simonsen, 2015).    most studies of the information skills of engineering students have focused on citation analysis, specific search strategies and database use (bhatt, dennick, & layton, 2010; hensel, brown, & strife, 2012; mcalpine & uddin, 2009; welker, mccarthy, komlos, & fry, 2012; wertz et al., 2013). these research studies, when taken together, speak not only to the challenges associated with methods of self-report measurement, but also to challenges associated with engineering students’ development and recognition of their own information literacy skillset. without recognition that their own skills could be further developed, students are unlikely to be motivated to actively engage in increasing their information literacy. this study, on the other hand, asks less directed questions regarding library services than what has previously been done (george et al., 2006) and concentrates on the students’ general conception of their processes (see appendix).   aims   while there have been some quantitative studies of engineering students’ information literacy, a deeper look into the varying skill levels between advanced and beginning students is needed. furthermore, there is limited research directly related to how engineering students describe their use of information when faced with a design project. before curriculum can be developed to target engineering students’ information literacy needs, there must be an exploration into how students report using information specifically when faced with engineering design. therefore, a qualitative approach is needed to inform the conceptualization of how beginning and advanced students approach information when faced with an engineering design project. according to peshkin (1993), one of the key outcomes of qualitative research is the ability to make interpretations that “explain or create generalizations, develop new concepts, elaborate existing concepts, provide insights, clarify complexity, and develop theory” (p. 25). our study seeks to provide insight into how engineering students describe their use of information, for the purpose of informing future curricular efforts that target information literacy in engineering design. in particular, we are interested in informing first-year and senior level curriculum for engineering students.   methods   setting, participants, and interview structure   we developed the research design based on the responsive interviewing method of qualitative research, as discussed by rubin and rubin (2012). to inform how to support information literacy for first-year and senior engineering students, we purposefully chose to interview two groups of students. “beginning” level students are defined as being in their first or second year of their undergraduate engineering program and “advanced” level students are defined as being either a senior or a graduate student in an engineering field. we included second year and graduate engineering students in order to capture skill development gained through the current academic year. therefore, a purposeful sampling strategy was used to create comparable groups, in an effort to fairly examine beginning and advanced engineering students with regard to information literacy.   we conducted the study at a large mid-western university with a large college of engineering. after obtaining human subjects research approval, we recruited interviewees through posted fliers in campus buildings where first-year and advanced engineering courses are routinely held.  the flier stated that students would be paid $15 for participating in the study. students went to an online survey and entered their year in school and email information. only students that fit our study criteria (first-year, second year, senior, and graduate engineering student) were interviewed. we described to interviewees the purpose of the interview, the procedures that would occur during the interview, and reiterated the voluntary nature of the interview. interviewees were asked to give informed consent prior to interviews. qualitative studies do not have rules on sample size. patton (2002) states the information richness of the cases selected is more important than the number. we interviewed a total of 21 engineering students, 10 beginning and 11 advanced, for approximately 30 minutes each. all engineering students at the university are admitted as first-year engineering students, then they apply to engineering disciplines in the sophomore year. the students’ majors represented include: first-year, aerospace, chemical, civil, computer and technology, electrical, industrial, materials, and mechanical engineering. throughout this work, we refer two groups of students. “beginning” refers to the eight first-year and 2 second-year engineers while the “advanced” group refers to four seniors and seven graduate engineering students.  the interviewee demographics are shown in table 1.     table 1 interviewee demographic information   beginning (n=10) advanced (n=11) residency u.s. citizen 10 4 international 0 7       gender male 7 9 female 3 2       ethnicity white 8 3 asian pacific 2 6 other 0 2     we created a semi-structured, responsive interview protocol based on guidelines from rubin and rubin (2012). responsive interviewing allows researchers to probe for deeper insights based on information provided by the interviewee, compared to a semi-structured or structured interview. in responsive interviewing, the emphasis is on hearing what the interviewee is saying and in the moment having the flexibility to ask follow-up questions that may not have been predetermined, as well as to foster an informal, conversational environment where students felt comfortable discussing their information literacy practices without feeling like they need to provide a “correct” answer.    the initial set of questions were written to include each stage of the information search process (isp) (kuhlthau, 2004) in the context of an engineering design project (see appendix). the isp is an evidence-based process model for research, and thus it was used as the underlying framework to make sure all phases of research were addressed in the interview. multiple follow-up questions were asked to encourage students to think deeper about each topic. as part of the interview protocol development process and training for the undergraduate research assistants, two pilot interviews were conducted and revisions were made to the protocol based on feedback from the interviewees. the first author and two research assistants, trained in the responsive interviewing technique, conducted interviews. none of the interviewees had interacted with the research team prior to this research project. interviews were conducted in a private conference room located in a campus engineering building. student interviewees were instructed to think about a recent class-related design project when answering the questions. each interview was audio recorded and transcribed.   analysis   we followed patton’s (2002) guidelines for qualitative data analysis following a thematic analysis approach, which allows findings to emerge from the data. three of the authors began by open coding a subset of 10 interviews (five beginning and five advanced), one question at a time. next, our research team reached consensus on a coding scheme which consisted of 159 total codes that spanned the 11 main questions. based on that coding scheme, two researchers coded all interviews (one question at a time) and compared results. discrepancies in coding were then discussed until consensus was reached. our team examined the categories and found broader categories to collapse codes into, resulting in 17 consolidated categories: application, consider options, little reflection, documentation, group process, information discovered, information given, people, pragmatic, reflection, saturation, selecting information, solution, solution found, structured information, synthesis, uncertainty, and unstructured information. from these broader patterns, themes regarding beginning and advanced level students’ information literacy strategies emerged. once identified, we carefully examined the data for any examples that contradicted or diverged from the asserted theme. any themes that had contradictory examples were removed from the presented results. in addition, we considered the strength of each category by examining whether every student in the group (beginning and advanced) had coding reflective of the category. this process resulted in five themes that are presented and discussed in the results.   results and discussion   in this section the five themes are broken out below, compared and contrasted between beginning and advanced groups, then each group is further characterized with supporting examples.   awareness of information needs   beginning students in our sample limited their discussion of information needs to those needs given to them through assignment guidelines or talking with instructors. advanced level engineering students more frequently discussed the need to identify information for the purpose of completing a successful design.   beginning   beginning level engineering students were aware of their information needs based on information given to them by instructors. students discussed reading and re-reading the design problem to determine what information to find. for example, one beginning level student said “i look over, i guess, all the guidelines and make sure i’m clear on everything”. the students discussed that they were new to engineering and instructors must guide them into design projects by providing detailed instructions related to the design problem. for example one beginning student said, “right, well, i’m just a freshman obviously so i’m in first year so a lot of the times they will pretty much just lay it out and say ‘these are the constraints’.” another beginning student phrased knowing what information to find “mostly using the guidelines given to me.”   advanced   advanced level students recognized information needs based on information provided by instructors, but they were also able to identify what to learn and then used information they found to identify additional gaps. in this way, they involved themselves in raising their awareness of information needs beyond what instructors had explicitly stated.   so when i’m given a problem, i try to understand what it means…what background is required for the project.  like….i had to work…..not right now but before this i had to work on an engine related project.  and i didn’t understand much about it so i went on to read about engines.  try to figure out what my part is in that and then see if i can understand it properly so that i can take it up as a project and go ahead with it.    strategies for searching   beginning engineering students in our sample almost exclusively focused on general search strategies, whereas advanced level students discussed specific information sources used to locate technical information. in addition, advanced level students used their own personal judgment to determine the appropriateness of a specific information source, rather than only relying on taught heuristics.   beginning   both in terms of locating and evaluating information, beginning level students identified efficiency as guiding their search strategies. they utilized general search engines to locate sources (e.g., google). one beginning level student said, “we don’t really have too well of a process for initially just coming up with a good source.  it’s more a trial and error, i guess”, and that they “tend to just search for the main keywords that [they’re] trying to look for”.  in evaluating sources, they discussed readily available information, such as urls, to determine the quality of a source. the beginning students placed a high value on where the information originated: “…usually try to look for like reputable sources like anything with like a .gov or a .org tend to be better than like a blog or something”. another student explained:   you can look at the url and determine like if it’s .gov then it’s usually credible compared to a .com and we also look at the author information and determine if they’re a nobody… and they’re talking about things that a phd student could be talking about and it’s not necessarily as credible as it could be.   advanced   advanced level students discussed reliance on general search engines and readily available evaluation criteria (e.g., domain name, host), similar to beginning level students, but they also reported utilization of their prior knowledge to make decisions about the usefulness of information. for example, they considered their knowledge gained from previous designs projects. they reported consulting forums, filtering information they found through their own experience to see if it was reasonable, and cross-checking information from multiple sources. some students reported techniques to evaluate reliability such as replication of the source’s information through modeling. a student said a source was credible   if you can replicate the results.  so, if you get an equation that says this and then they give like a sample…..some other places will give you a sample……if you use the equation and you get the same value and you can find the same equation somewhere else, it’s good.  it’s basically a two-step check.  so, if you can find the information somewhere else, then you know it’s good.   another student said:   …if you like come across any website or page that you haven’t seen up to now, you try to get some sort of a feedback about that website…..maybe through the reviews given in the bottom or ask somebody about the credibility of the paper.   extraction of information   when it comes to extracting information from sources to apply to a design problem, beginning level students utilized less sophisticated reading strategies, and they had a utilitarian purpose for using information sources.  advanced level students tended to report advanced reading strategies and to independently replicate results before using the information in their project.    beginning   in terms of actual use of information, students described using information as a way to justify their solutions, for example, fulfilling an assignment’s requirements for sources, rather than as a way to generate new ideas to solve a problem. one student described citing as a way of increasing credibility:   i feel like that if we would just do that as much as possible [cite sources] then that helps the audience know that we’re not just like throwing stuff out there. we’re not just making up stuff and that we’re actually using sources and that we try to base our entire project off of credibility of sources and stuff like that.   beginning level students also discussed reading entire sources to understand information, rather than reading pertinent parts, indicating that they are having a hard time understanding the structure and perhaps content of papers, using brute force measures to extract meaning. “i try to read the entire source at least two to three times so i thoroughly can understand everything they’re saying and make sure that i understand it so i can actually apply it without overlooking little details in it.”    advanced   advanced level students discussed scanning for information in pertinent paragraphs, rather than reading straight through. advanced level students also discussed attempts to replicate the results or use data from sources for calculations. the results of those activities inform whether the information was used in the design. students described pulling out the relevant parts of an information source:   generally i’ll read probably half of it. well see, like, if it’s a good source i’ll read the first two paragraphs and then continue reading word for word almost. if it’s not exactly what i need i’ll skim the first couple sentences in each paragraph and skip until i find something that’s relevant.   and,   for this year i did project [and found] 60 or more articles that total 400 pages. there’s not enough time. so i, like…..you know you find…..you know “ctrl + f” for the terms you want and you read the page above and below it and then move on with your life.   sufficiency of information   students in both groups discussed some level of uncertainty in describing how they knew they had found enough information; however, how they determine the amount of information needed varied.  beginning level students more frequently discussed external factors to terminate their search for information, such as time constraints or lack of new results from a search engine, while advanced level students discussed determining whether their questions were completely answered as the mechanism for deciding when to stop looking for information.   beginning   beginning level students mentioned external feedback as a way of knowing when they had found enough information. students discussed ending the search process based on pragmatic issues such as an answer found, group agreement, and time constraints. one student said, “i think when the information that we have starts to overlap more and more and we don’t find anything new in, i don’t know, ten….fifteen minutes then….well that’s just saying that we run out of things to find.” another student described getting search results as ending the process, rather than thinking about other search terms or strategies, “i guess it’s more when i’ve exhausted all the sources i can find and because if i type something in google, first couple of pages will be real useful but after a while it’s completely irrelevant to what i’m doing.”   advanced   advanced level students reported the use of internally constructed measures to determine whether they had sufficient information. they discussed internal decision-making related to sufficiency of information. for example, one advanced level student said, “it’s hard to say. it’s very hard to say. maybe when i try my idea, i find i’m not….i have not enough information so i go back to search again.” other students described the internal process they go through, such as “when you’re not asking yourself questions, i guess, when you are at a point when you have gone far enough to continue self-reliantly, i think, is a point where you can say you have enough information.”   organization of information   beginning level students more frequently reported organization of their sources in a simple word file, although both advanced and beginning level students did describe using that technique. advanced level students also reported the use of additional methods, such as bibliographic management tools or filing papers by subject within folders. none of the respondents utilized advanced strategies such as rating or tagging sources within bibliographic systems. sharing information among project team members was also mentioned more frequently by advanced level students than beginning level students.   beginning   beginning level students focused on the documentation aspects of the information search process, such as securing citations and references, rather than focusing on how they interact with information (e.g. collecting, using, and applying information). strategies for maintaining information were not as clearly described as the role of citation in presentation. for example, in response to the question, “what do you do with information?” one student replied:   copy down what i think would be really useful into a google doc, bookmark or just find the link and write down the link so what i think are the more useful sites that i can just refer back to them.   advanced   advanced level students discussed systems for saving the information about sources throughout the duration of the project, through tools such as electronic folders, bibliographies, and notes. one student describes one such organizational method as: “i keep a little folder for the specific project on firefox or whatever browser i’m using. professors, i write down their name, what the information was, the date…i keep specific tabs inside the folder for online ones...”   summary   through the analysis process several themes regarding similarities and differences between beginning and advanced level engineering students’ information literacy strategies emerged.   all beginning level engineering students interviewed described:   (a)     reliance on the problem statement and parameters as explicitly given, (b)     use of primarily basic search strategies, (c)     orientation toward documentation purposes of information, and (d)    reliance on external feedback to determine when they had found enough information.   all advanced level engineering students interviewed described:   (a)     integration of both their own knowledge and information provided in the problem statement, (b)     use of both basic and advanced search strategies, (c)     application purposes of information, and (d)    integration of self-reflection and problem requirements to determine when they had found enough information.   conclusion   in this study, we were able to identify several differences between the information habits of beginning and advanced level students. as might be expected, beginning level students exhibited less sophisticated strategies, relying more on rote or external operations and measures of success. they treated information gathering more as an activity to be completed as a mechanistic part of an assignment (e.g., collect five sources), rather than as a means of improving their work products or increasing their understanding. indeed, beginning level students treated the projects themselves more as externally motivated tasks to complete rather than activities in which they were personally invested. advanced level students, on the other hand, had internalized the purpose of information gathering. they compared information found to their prior knowledge and asked questions of the information, for example, whether it met their needs and whether unanswered questions remained.    in this respect, the advanced level students do seem to be acting more like experts than novices (wankat & oreovicz, 1993). among other characteristics, wankat and oreovicz (1993) observe that experts can “chunk” information and identi­­fy fundamental information.  rather than treating each bit of information as separate and independent, they can identify relevant information and draw inferences from incomplete information, and they take time to define (and redefine) and explore a problem, rather than jump to a conclusion. novice students use trial and error instead of coherent strategies.    it should be noted that expertise is typically limited to a particular field (e.g., an expert chess player is not necessarily a good dancer), so there is no a priori reason expert engineers would be expert information gatherers. thus, it is encouraging that the advanced engineering students did in fact show well-developed information abilities and attitudes. some of advanced skills follow from a greater disciplinary knowledge, i.e., students with a well-developed personal knowledgebase of engineering principles can more easily identify relevant and reasonable information from a source. other skills, for example, information organization, are less obviously related to disciplinary knowledge, and they do show weaker improvement between the two samples, i.e., some advanced level students used advanced knowledge management strategies, but others showed similar strategies as the beginning level students.    the implications for instructors are clear. beginning students need to develop the “expert” information literacy skills they will need as professionals. they do not demonstrate an awareness or internalization of the importance and purpose of information in solving of engineering problems, so problems should be posed and feedback given that provides direct guidance where unsupported claims or suboptimal solutions could be strengthened by the search for and appropriate application of additional information. this study identified several areas where there are marked differences in the quality of information strategies used by beginning and advanced students: awareness of information need, search strategies, extraction of information, organization of information, and determining the sufficiency of information.   the qualitative summaries of these concepts show the thinking processes of beginning students, so instruction can directly target those conceptions. for example, discussions of why information is needed in problem solution, or practice creating mind-maps of a topic to identify what information students know and what is not, and what questions they want to explore further, will help students engage with the concept of being aware of their information need. in that way, they can internalize the importance and use of that concept. similarly, for the other concepts, exercises that reinforce higher quality strategies and the better results that follow from using them will allow for the development of expert information literacy skills in tandem with expert disciplinary skill development. ideally, these concepts need to be reinforced consistently over several courses so that students practice transfer of information literacy concepts across different problem contexts.   as with all research, this study has limitations. as a qualitative study, we interviewed a small number of beginning and advanced level engineering students to develop a deeper characterization of how they differ in their information literacy. the students interviewed were from one large research university, and it is unknown how similar engineering students at other campuses are to those in this sample. however, only findings that could represent every student interviewed were included in this study to strengthen the implications of the results. future research should consider whether these findings are consistent with engineering students at different institutions.     references   abet engineering accreditation commission. (2010). criteria for accrediting engineering programs: effective for evaluations during the 2011–2012 accreditation cycle. retrieved from http://www.abet.org/accreditation-criteria-policies-documents/   allard, s., levine, k. j., & tenopir, c. (2009). design engineers and technical professionals at work: observing information usage in the workplace. journal of the american society for information science and technology, 60(3), 443-454. http://dx.doi.org/10.1002/asi.21004   association of college and research libraries (acrl). (2000). information literacy competency standards for higher education. retrieved from http://www.ala.org/acrl/standards/informationliteracycompetency     atman, c. j., adams, r. s., cardella, m. e., turns, j., mosborg, s., & saleem, j. (2007). engineering design processes: a comparison of students and expert practitioners. journal of engineering education, 96(4), 359-379. http://dx.doi.org/10.1002/j.2168-9830.2007.tb00945.x      atman, c. j., cardella, m. e., turns, j., & adams, r. (2005). comparing freshman and senior engineering design processes: an in-depth follow-up study. design studies, 26(4), 325-357. http://dx.doi.org/10.1016/j.destud.2004.09.005   atman, c. j., chimka, j. r., bursic, k. m., & nachtmann, h. l. (1999). a comparison of freshman and senior engineering design processes. design studies, 20(2), 131-152. http://dx.doi.org/10.1016/s0142-694x(98)00031-3   bhatt, j., dennick, d., & layton, b. e. (2010). citation analysis of engineering design reports for information literacy assessment. proceedings of the american society for engineering education, louisville, ky.   bruce, c. (2004). information literacy as a catalyst for educational change. a background paper. in danaher, p. a. (ed.), proceedings of the 3rd international lifelong learning conference. lifelong learning: whose responsibility and what is your contribution? (pp. 8-19). yeppoon, queensland.   collins, e., & stone, g. (2014). understanding patterns of library use among undergraduate students from different disciplines. evidence based library and information practice, 9(3), 51–67. retrieved from http://ejournals.library.ualberta.ca/index.php/eblip/index   creswell, j. w. (2012). qualitative inquiry and research design: choosing among five approaches (3rd ed.). thousand oaks, ca: sage.   davidson, e. j. (2005). evaluation methodology basics: the nuts and bolts of sound evaluation. thousand oaks, ca: sage.   douglas, k. a., wertz, r. e. h., fosmire, m., purzer, s., & van epps, a. s. (2014). first year and junior engineering students’ self-assessment of information literacy skills. proceedings of the 121st american society of engineering education conference and exposition, indianapolis, in. retrieved from http://www.asee.org/public/conferences/32/papers/9576/view   dym, c. l., little, p., orwin, e. j., & spjut, r. e. (2009). engineering design: a project-based introduction. new york: wiley.   ellis, d., & haugan, m. (1997). modelling the information seeking patterns of engineers and research scientists in an industrial environment. journal of documentation, 53(4), 384-403. http://dx.doi.org/10.1108/eum0000000007204   engel, d., robbins, s., & kulp, c. (2011). the information-seeking habits of engineering faculty. college and research libraries, 72(6), 548-567.    fosmire, m. (2012). information literacy and engineering design:  developing an integrated conceptual model. ifla journal, 38(1), 47-52.   fosmire, m., & radcliffe, d. (eds.). (2014). integrating information into the engineering design process. west lafayette, in: purdue university press.   george, c., bright, a., hurlbert, t., linke, e. c., st. clair, g., & stein, j. (2006). scholarly use of information: graduate students’ information seeking behaviour. information research, 11(4). retrieved from http://www.informationr.net/ir/index.html    gerstberger, p. g., & allen, t. j. (1968). criteria used by research and development engineers in the selection of an information source. journal of applied psychology, 52, 272-279.   head, a. j. (2013). learning the ropes: how freshmen conduct course research once they enter college. retrieved from project information literacy website: http://projectinfolit.org/images/pdfs/pil_2013_freshmenstudy_fullreport.pdf   head, a. j., & eisenberg, m. b. (2010). truth be told: how college students evaluate and use information in the digital age. retrieved from project information literacy website: http://projectinfolit.org/images/pdfs/pil_fall2010_survey_fullreport1.pdf   hensel, r. a. m., brown, o., & strife, m. l. (2012). engineering an information literacy program for first-year engineering students. proceedings of the american society for engineering education annual conference & exposition, san antonio, tx.   holland m. p., & powell, c. k. (1995). a longitudinal survey of the information seeking and use habits of some engineers. college and research libraries, 56(1), 7-15.   holliday, w., & li, q. (2004). understanding the millenials: updating our knowledge about students. reference services review, 32(4), 356-366. http://dx.doi.org/10.1108/00907320410569707   ingwersen, p., & järvelin, k. (2005). the turn: integration of information seeking and retrieval in context. dordrecht: springer.   johnson, p. c., & simonsen, j. e. (2015). do engineering master’s students know what they don’t know?: exploring abstracting & indexing service use and non-use. library review, 64(1/2), 36-57.   kuhlthau, c. c. (2004). seeking meaning: a process approach to library and information services. (2nd ed.). westport, ct: libraries unlimited.   kwasitsu, l. (2003). information-seeking behavior of design, process, and manufacturing engineers. library & information science research, 25(4), 459-476.   lattuca, l. r., terenzini, p. t., & volkwein, j. f. (2006). engineering change: a study of the impact of ec2000: executive summary. retrieved from abet website: http://www.abet.org/uploadedfiles/publications/special_reports/engineeringchange-executive-summary.pdf   leckie, g. j., pettigrew, k. e., & sylvain, c. (1996). modeling the information seeking of professionals: a general model derived from research on engineers, health care professionals, and lawyers. library quarterly, 66, 161-193.   mcalpine, b., & uddin, m. (2009). integrating information literacy across the engineering design curriculum. proceedings of the american society for engineering education annual conference & exposition, austin, tx.   mosberg, s., adams, r., kim, r., atman, c. j., turns, j., & cardella, m. (2005). conceptions of the engineering design process: an expert study of advanced practicing professionals. proceedings of the 2005 american society for engineering education annual conference & exposition.   patton, m. q. (2002). qualitative research and evaluation methods (3rd ed.). thousand oaks, ca: sage publications.   peshkin, a. (1993). the goodness of qualitative research. educational researcher, 22(2), 23-29. http://dx.doi.org/10.3102/0013189x022002023     pinelli, t. e. (1991). the information-seeking habits and practices of engineers. science & technology libraries, 11(3), 5-21.   purzer, s., & wertz, r. (2014). scaffold and assess: preparing students to be informed designers. in fosmire, m., & radcliffe, d. (eds.), integrating information into the engineering design process. west lafayette, in: purdue university press.   ross, m. c., fosmire, m. j., wertz, r. e. h., cardella, m. e., & purzer, s. (2011). life-long learning and information literacy skills and the first year engineering undergraduate: report of a self-assessment. proceedings of the 118th american society of engineering education conference and exposition, vancouver, bc, canada. retrieved from http://www.asee.org/public/conferences/1/papers/1275/view   rubin, h. j., & rubin, i. s. (2012). qualitative interviewing: the art of hearing data (3rd ed.). thousand oaks, ca: sage publications.   tenopir, c., & king, d. w. (2004). communication patterns of engineers. hoboken, nj: wiley.   wankat, p. c., & oreovicz, f. s. (1993). teaching engineering. new york: mcgraw-hill.   welker, a. l., mccarthy, l. a., komlos, j., & fry, a. a. (2012). information literacy: a field guide for the path of life-long learning. proceedings of the american society for engineering education annual conference & exposition, san antonio, tx.   wertz, r. e. h., purzer, s., fosmire, m. j., & cardella, m. e. (2013). assessing information literacy skills demonstrated in an engineering design task. journal of engineering education, 102(4), 577-602. http://dx.doi.org/10.1002/jee.20024     appendix interview protocol   part 1–prior to the interview instructions for interviewer:  there are 11 main questions, each with follow-up prompts to elicit depth of information from the interviewee. please read each main question and the follow-up prompts prior to beginning.  the point of this semi-structured interview is to gather as much information about what students actually do. make sure to allow interviewee plenty of time to answer each question. once one question is asked and answered, ask follow-up prompts if those areas where not brought up. if the interviewee says something unclear, ask questions to clarify or further elaborate what was said.   part 2–pre-interview read to interviewee:  the purpose of this interview is to find out more about how engineering students approach finding and using information for their engineering projects. i will ask you questions related to how you find and use information. there are no right or wrong answers, this interview is simply to learn more about what students do.  please answer each question as best you can and any answer is okay. the interview will be recorded and transcribed with no identifying information. participation in this study is voluntary and you may stop at any time.  we have an informed consent for you to review. in addition, since you will be receiving a case incentive to participate in this study, we will also need your signature indicating that you received the cash when the interview is over. if you agree to participate in this study, please sign the form and let me know when you are ready to proceed.   instructions for interviewer: allow time for participant to read and sign informed consent agreement.   read to interviewee:  do you have any questions about the interview before we begin?   part 3–during the interview (begin recording) instructions for interviewer:  read each question, one at a time, and allow interviewee to speak until finished.  ask every question as written. read to interviewee:  i’m going to ask you a series of questions about how you complete course projects. please think about a recent engineering design project when answering these questions.   1.       when you first approach an assigned project, what do you do? -         what do you do to prepare yourself for completing the project? -         how do you determine the criteria/requirements for the project? -         how do you determine the constraints?   2.       when do you begin searching for information? -         before you begin searching, what do you do? -         how do you know what information you need?   3.       where do you go to get your information? (what sources of information do you search?)  -         how do you know where to go for the type of information needed? -         what type of search strategies do you use? -         name three places that you go to for information.   4.       how do you determine whether a source is credible or not? -         what makes a source high quality?   5.       once you’ve found information that is relevant and you’ve determined credible, what do you do with it?  (how do you use the information you’ve found?) -         how much do you read? -         after you’ve read it, what do you do? -         before going to the next source, what do you do?   6.       when gathering information, how do you keep track of what you are finding? -         how do you synthesize all the information?   7.       how do you know when you have found enough information?   8.       how do you consider alternative approaches to the problem?   9.       when you initially test your solution, how do you decide what type of revisions are needed?   10.    when presenting your project, when do you refer to a source of information? -         how do you reference where the information came from? -         how do you separate your ideas from ideas that you found elsewhere?                        11.    once a task is finished, how much thought do you spend on evaluating your efforts? -         how often do you think about how you can improve in future tasks? article   evidence based practice using formative assessment in library research support   jackie wolstenholme research services librarian information & research services eddie koiki mabo library james cook university townsville, australia qld 4811 email: jackie.wolstenholme@jcu.edu.au   received: 28 jan. 2015    accepted: 17 jun 2015      2015 wolstenholme. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – the purpose of this study was to develop and review the effectiveness of a new evidence-based approach for teaching library research support.   methods – formative assessment, through two variations of the one minute paper model, is used to poll the experiences of university researchers in library research support sessions. prior to a session, polling one minute papers (pomps) assess what researchers know about topics that will be covered in the session. after a session, reflective one minute papers (romps) review whether university researchers achieved the intended learning outcomes of the session. pomps were used for 16 sessions and romps were used for a subset of 11 of these sessions. examples of responses from the pomps and romps were presented to describe and analyse the effectiveness of this approach for library support of research.   results – pomp and romp responses were remarkably informative given their simplicity and the little effort required on the part of the instructing librarian or researchers. the completion rate of pomps was 72.7%. they gave researchers the opportunity to self-assess their current level of knowledge or skills about the topic to be covered in the upcoming session. the librarian could then tailor the session content to this level of knowledge. pomp responses were shared as part of the session content, enabling researchers to benchmark themselves against their peers. completion rate of romps was 20.9%, with the level of reflection in the individual researchers’ responses varying from shallow to insightful. deeper responses stated how the researchers would use what they learned or pose new questions which emerged from their learning.   conclusion – polling one minute papers (pomps) and reflective one minute papers (romps) are an effective and efficient approach for guiding the learning of researchers and closing the feedback loop for librarians. these tools extend the opportunity for librarians to engage with researchers and, through tailoring of session content, assist to maximise the benefit of library research support sessions for both librarians and researchers. sharing of pomp and romp responses can assist librarians to coordinate the teaching of the researchers that they support. at an institutional level, evidence in pomps and romps can be used to demonstrate the value that the library has contributed to improving awareness and performance of its researchers. introduction   researchers in universities are working in an increasingly complex and competitive environment (e.g., frances, fletcher, & harmer, 2011; kennan, corrall, & afzal, 2014; richardson, nolan-brown, loria, & bradbury, 2012). factors driving these changes include internet and digital technologies and greater accountability through performance management and institutional benchmarking. these changes are requiring researchers to adapt faster than most would achieve through their traditional discipline-based networks, including information sharing among colleagues.   the internet and digital technologies have transformed scholarly communication. research outputs, although still published as books and journals, are now also made available in an array of other digital options including blogs and other social media, multimedia formats, and data files which may be displayed through sophisticated visualization tools. the numbers of research outputs have vastly increased and are distributed through a growing range of publishing models, many offering some form of open access. researchers, as creators of research outputs, need to consider copyright and licensing for managing their rights, in balance with maximizing accessibility to their research outputs. the quality of publishers also needs to be assessed, to ensure that researchers avoid unethical publishers (e.g., see beall, 2014).   to measure and benchmark performance, researchers and their institutions rely on citation ranking metrics. researchers need to understand how these metrics are calculated and how citation indexes (e.g., the h-index) are calculated. researchers are also expected to have an online presence, ideally as a professional profile to promote their research interests and achievements. altmetrics are emerging as an additional measure of impact, by measuring the online activity of a researcher or their outputs (priem, taraborelli, groth, & neylon, 2010).   university libraries can assist researchers to work in this environment and make the most of emerging opportunities. to provide this support, university libraries are moving their core business from provision of information resources to provision of information services and information solutions (association of college and research libraries, 2010; kaufman, 2009; parsons, 2010). information resources have traditionally involved the development and management of collections. in contrast, information services and solutions include infrastructure such as repositories (simons & richardson, 2013) and instructional support on topics such as scholarly and open access publishing, managing research data, maximizing research visibility, and measuring research performance (e.g., auckland, 2012; haddow, 2012; kennan, et al., 2014).   effective communication skills are essential for building a rapport with researchers and providing a valued service (auckland, 2012; creaser & spezi, 2013; parker, 2012). research support librarians need to be confident in talking about the range of topics that researchers need to learn, match the information they provide with the skill level of the researcher, and explain the information in a way that is understandable for the researcher. this study investigates a teaching and learning approach which can assist librarians to achieve this.   literature review   teaching and learning in library research support   teaching and learning for researchers is best suited to the learning theory of andragogy. this theory is based on the assumption that adults are self-directed learners who are interested in immediate application of knowledge (merriam, 2001). according to this theory, adults take the initiative in diagnosing their learning needs, including formulating objectives, identifying resources, implementing strategies, and evaluating outcomes (knowles, 1975). other learning theories of relevance to library research support include problem-based learning (knowles, 1975), experiential learning (kolb, 1984), and informed learning (hughes & bruce, 2012). in problem-based and experiential learning, learners draw on their prior knowledge and experience (brodie, 2012), enabling learning to be built on a researcher’s existing practices. informed learning describes how learners develop flexibility and confidence to use information in constantly evolving information environments, shifting the focus of information literacy education from mastering skills to learning to use information critically, ethically, and creatively (hughes & bruce, 2012).   in addition to self-directed learning, another key aspect of researcher learning is that researchers frequently learn from their peers. the peer is a defining figure in research practice. for example, it is implicit in the institution of “peer review” (boud & lee, 2005). as described for higher degree research students, research learning can be usefully construed in terms of entry into communities of practice, where peer learning becomes a powerful tool for describing and developing a rich understanding of the learning resources available (boud & lee, 2005). peer interaction can enhance learning by stimulating the production of deeper thought through the desire to know what a colleague knows, prompting self-assessment and clarification of uncertainties (draper, 2009).   assessment in library research support   library research support needs to contribute to improving research performance without adding additional burden to a researcher’s workload. researchers operate in a constant environment of research performance assessment (parker, 2012), e.g., through funding or promotion applications, performance management acquittal, or as part of institutional assessment exercises such as those in australia and the united kingdom (australian research council, 2014; ref2014, 2014). research librarians need to be acutely aware of this research assessment landscape (parker, 2012). the learning needs of researchers must form the core content of library research support material, in terms of what researchers need to know as well as their current status of understanding a particular topic.   within a teaching and learning framework, assessment of the performance of researchers is analogous to summative assessment, and library support is analogous to formative assessment. summative assessment tasks focus a student’s learning on “what counts,” while formative assessment provides a fine tuning mechanism which guides the learner’s learning progress (boud, 2000). summative assessment occurs after the learning process for the purpose of certification (sadler, 1989). in contrast, formative assessment occurs as part of the learning process. through formative assessment, the learner gains feedback which is intended to shape and improve their learning, leading to independent learners who are able to self-monitor their learning needs (sadler, 1989). to implement this analogy of formative assessment in library research support, research librarians will be most effective if they develop teaching materials which incorporate the established practices of researchers’ self-directed and peer learning.   learning is a cyclical process, as explained in experiential learning theory (kolb, 1984). in the context of library research support, there is no starting point in the researchers’ learning process, but rather, they build on what they already know or have experienced. within the learning cycle of assessment (figure 1, crisp, 2009), there may be many feedback loops between the phases of diagnostic, learning, and formative assessment (sadler, 1989). often, formative assessment leads to summative assessment, as researchers take on tasks to advance their careers or to meet institutional requirements.   research framework: formative assessment of researchers   the one minute paper (omp) is a formative assessment tool that has been successful in improving the teaching of, and learning by, undergraduate students (bartlett & morrow, 2001; chizmar & ostrosky, 1998). omps are effective for gaining student feedback in return for a modest amount of student and instructor effort (bartlett & morrow, 2001; chizmar & ostrosky, 1998; drummond, 2007; stead, 2005). the omp is a questionnaire which asks:   1.     what was the most important thing you learned today? 2.     what was the most confusing point in today’s lecture? 3.     what was the most interesting fact that you learned today?     figure 1 relationship between diagnostic, formative, and summative assessments (redrawn from crisp, 2009) the addition of a third question was recommended by bartlett and morrow (2001:     the omp benefits both instructors and students, regardless of their teaching or learning ability (chizmar & ostrosky, 1998). omps can provide specific and immediate feedback to the instructor about student learning, helping to set the pace and content of future instruction. this is useful for inexperienced instructors or instructors of new material (stead, 2005), as is often the case in library research support. instructors can also use the feedback to identify and then address misconceptions (bartlett & morrow, 2001). this closing of the feedback loop demonstrates that the instructor values student opinion and encourages students to actively contribute to their own learning experience (stead, 2005). class discussion of issues raised in omp’s have reassured students by enabling them to benchmark their learning against their classmates, often revealing that the problems that others are experiencing are the same as their own (bartlett & morrow, 2001).   the simplicity of the omp makes it an ideal tool for identifying the learning needs and learning outcomes of researchers. content of library research support sessions can then be tailored accordingly. the omp is typically assigned at the end of a class, but could also be adapted for implementation prior to a class (stead, 2005). pre-class formative quizzes encourage students to think critically about course content prior to a session (dobson, 2008), offering the benefits of identifying current learning needs or learning gaps, providing an indication of what will be covered in the upcoming session, and creating an opportunity for self-assessment.   aims   this study describes a method, adapted from formative assessment in teaching and learning, to assist research support librarians to develop an evidence-based foundation to support their teaching. two variations of the one minute paper (omp), polling omps and reflective omps, are developed in this study. a case study approach, from a series of multiple workshops on a range of topics, is used to investigate whether the two variations of the omp are an effective and efficient approach for guiding the learning of researchers and closing the feedback loop for librarians.   questions asked are:   1.       do pomps stimulate researcher engagement and interest? 2.       can pomps identify learning needs of researchers? 3.       are pomps or romps effective tools for gaining feedback about researcher learning?   methods   this study reports on the outcomes of a series of case studies, to explore the effectiveness of using the one minute paper (omp) model for the purpose of library research support sessions. two variations of the omp were developed in this study: pomps, i.e., polling one minute papers, and romps, i.e., reflective one minute papers.   both the pomps and romps are intended as tools which guide the formative learning of researchers. pomps were distributed prior to a session and romps were distributed after a session. sessions were organized in response to specific requests from researchers on behalf of a research group, rather than according to a specific schedule.   overview of the james cook university research profile   the omps described in this study were developed for library research support sessions at james cook university (jcu). the strategic intent of jcu is to create a brighter future for life in the tropics world-wide (james cook university, 2015).the number, distribution and turnover of jcu researchers make it challenging to identify and meet their evolving library research support needs. there are approximately 2600 academic staff and more than 600 higher degree research students (james cook university, 2014) across townsville, cairns, singapore and other smaller, regional centres.   research needs vary with disciplinary research practices and career stage. at jcu, the largest and fastest growing area of research is in the medical disciplines, with many of these researchers having a strong applied knowledge but limited research experience. in contrast, internationally recognized researchers in the biological and environmental sciences tend to have metrics-driven library support needs. humanities and social sciences, including law and creative arts, have the most discipline-centric research needs. career stage also influences library research support requirements: postgraduate students and early career researchers need to develop their research skills, mid-career researchers may be concerned about keeping up with technological changes, and senior researchers may be under pressure to maintain their high research standing.   polling one minute papers (pomps)   pomps are a self-assessment tool. the questions asked in a pomp were structured around the topic of a library research support session, polling researchers to gauge their understanding of the topic. session content was then tailored for this level of understanding. the questions and response options in pomps are listed in the appendix. pomps were also intended to promote a session and stimulate interest about the content that would be covered in that session.   sessions were organized in collaboration with research leaders, e.g., key researchers or research managers. this strategy helped to increase attendance and facilitate discussion because participants shared common research interests and usually knew each other prior to the session. pomps were distributed approximately one week prior to a session in an email. this email was sent by the research leader to all researchers who he felt should attend the session. pomps were voluntary, with participants being encouraged to submit their responses prior to the session. a summary of the pomp results were shared during the corresponding session to enable peer benchmarking and therefore further self-assessment. pomp responses were presented as descriptive data in tables, histograms or pie charts.   reflective one minute papers (romps)   romps are a feedback tool which encouraged voluntary reflection about a session. the same three questions were asked in all romps:   1.       what was the most important thing you learnt? 2.       what was the most confusing thing i covered? 3.       what was the most interesting thing you learnt?   romps were completed on a voluntary basis. by responding to these three questions, researchers were able to provide feedback about the session and their learning to the librarian. romps were developed partway through this project in order to close the formative assessment loop. romps were used for 11 sessions, as listed in table 1. a link to the three romp questions was usually distributed on the last slide of a session presentation or immediately after a session. in one exception (session 16), the romp link was sent out 2 weeks after the session. romp responses were in an unstructured, free text format. thematic analysis, also known as analytic coding, was used to interpret the responses and quantify them according to themes (guest, macqueen, & namey, 2012; richards, 2015).   data collection   pomps and romps were created in google forms (google, 2014). the selection of google forms was based on a number of criteria: google forms is a free service with no limit on the number of questions that can be asked; checklists of multiple options per question can be selected, enabling quick standardized responses; and, the omps could be shared through an online link, enabling participation by both local and remote researchers.   pomps and romps were anonymous, a decision based on the presumption that anonymity would make researchers more likely to provide honest responses and therefore enable more realistic benchmarking amongst peers (dillman, smyth, & christian, 2009). respondents were also not required to log in using google forms, building researcher confidence that the omps were anonymous.   results   the results report on a series of case studies to explore the effectiveness of pomps and romps to respond to the three questions stated in the aims.   response rates of pomps and romps   the numbers of researchers attending the sessions for which pomps and romps were distributed are summarized in table 1.   the completion rate of pomps was high, at 72.7%, i.e., 136 from a pool of 187 researchers who attended the 16 sessions. a likely factor contributing to this high rate of completion was the simplicity of the pomp form. in one click from a link in an email, it was immediately evident what the researcher needed to respond to. the entire pomp could be viewed on a desktop screen without scrolling, visually emphasizing that the form would be quick to complete, with the format of all or most responses being checklists. promoting the pomp as a “1 minute quiz” with endorsement from a research leader was also effective. for example, the director of a research centre for one of the life and environmental science sessions included the following statement of support in his email:   as part of the prep for the planning day, and to help with our understanding of the use of research profiles, can i ask you to take this 1 minute quiz? i took it and it took even less than 1 minute.   the completion rate of romps was lower, at 20.9%, i.e., 23 from a pool of 110 researchers who attended the 11 sessions for which romps were distributed. a major factor contributing to this low response rate may have been that the response format was free text.   researchers were from the disciplines of life and environmental science, health science, social science and humanities, and mixed disciplines (table 1). the librarian was present in the same room with participants for sessions 1-10, 15 and 16. sessions 11-14 were conducted remotely via videoconference (table 1). sessions 1-15 were presented by the author. session 16 was presented by another research support librarian who provided peer feedback about the value of omps.     table 1 summary of attendance, pomps and romps for each session session # discipline / client group session topic remote* session date attendance # pomps submitted % pomps submitted # romps submitted % romps submitted 1 life and environmental sciences: post-docs altmetrics no 9 october 2013 23 17 73.9 --- --- 2 life and environmental sciences managing research profiles no 11 october 2013 28 28 100.0 --- --- 3 life and environmental sciences research profile update and management no 12 august 2014 8 9 112.5 6 75.0 4 life and environmental sciences research profile update and management no 29 august 2014 8 6 75.0 1 12.5 5 life and environmental sciences research profile update and management no 19 september 2014 5 4 80.0 1 20.0 6 life and environmental sciences research profile update and management no 9 october 2014 4 4 100.0 0 0.0 7 life and environmental sciences research profile update and management no 10 october 2014 4 2 50.0 0 0.0 8 life and environmental sciences research profile update and management no 24 november 2014 9 8 88.9 0 0.0 9 health sciences publishing academic research no 23 october 2013 11 6 54.5 --- --- 10 health sciences research and social media no 27 august 2014 14 15 107.1 4 28.6 11 health sciences quality publishing yes 28 august 2014 3 2 66.7 3 100.0 12 social sciences and humanities researcher identifiers yes 9 september 2014 2 3 150.0 2 100.0 13 social sciences and humanities altmetrics yes 18 september 2014 5 4 80.0 4 80.0 14 social sciences and humanities promoting and maximising research impact yes 2 december 2013 2 6 300.0 --- --- 15 mixed disciplines: academic teaching staff research impact and publishing no 13 february 2014 13 9 69.2 --- --- 16 mixed disciplines: higher degree research students intellectual property & copyright no 2 and 9 october 2014** 48 13 27.1 2 4.2 total attendance and number of pomps/romps submitted for all sessions 187 136   23       figure 2 display of results for the pomp question "have you used any of the following to promote or discuss your research?" (responses from 78 submitted pomps distributed to 89 researchers, compiled progressively and presented in sessions 1-8).     aim 1: do pomps stimulate researcher engagement and interest?   pomps enhanced the engagement of researchers. completion of a pomp and then the sharing of pomp results provided researchers with two opportunities for self-assessment. firstly, pomps could provide additional incentives to go to a session if researchers identified gaps in their knowledge or skills based on questions asked in the pomp. in at least one case, a researcher started to use some of the tools listed in the pomp prior to the session. secondly, in sessions, researchers showed great interest in the activities and tools being used by their peers. figure 2 displays the combined responses for eight sessions on the same topic (managing research profiles) for researchers from the same discipline (life and environmental sciences). the cumulative responses were shared in each of the eight sessions. the opportunity to benchmark against peers and resulting discussion gave endorsement to the advice provided by the librarian. for example, one researcher reported that he now has a new international collaborator with substantial funding through a linkedin™ connection. hearing how their peers were using such tools gave context-relevant evidence, making it easier to sell the concept of using social media tools in a research context. researchers were also able to see which tools were widely used by their peers, giving an indication of where to get started, or a confidence boost if they were already using those tools. learning about lesser used tools such as the conversation (http://theconversation.com/au) gave researchers ideas for how they could increase awareness of their research.   aim 2: can pomps identify learning needs of researchers?   pomps were effective and efficient in identifying current learning needs of researchers and what the focus of the session content should be prior to a session. pomp responses indicated what researchers understood or were doing well and also gaps in their current knowledge or activities.   responses from life and environmental science researchers attending one of the eight sessions represented in figure 2 (session 2, as listed in table 1) indicated that participants in this session were using some online tools and almost half had used traditional media to communicate and promote their research (figure 3). this suggested that they were generally aware of the importance of communicating and promoting their research. despite this apparent awareness, the moderate to nil use of more than half the listed tools (figure 3) also highlighted areas for further instruction.     figure 3 display of results for the pomp question "have you used any of the following to promote or discuss your research?" (responses from 28 submitted pomps distributed to 28 researchers, presented in session 2).     as a further example of how pomps were useful for identifying learning needs, health science and life and environmental science researchers were mostly aware of whether an article processing charge (apc) had been paid for their article, but were less clear about whether they had signed copyright ownership over to the publisher. approximately one third of responding researchers from each discipline had authored a paper in which an apc had been paid and less than 10% were not sure (figure 4). for authors who had paid an apc, a serious issue for discussion was highlighted: eight of the life and environmental science researchers (figure 5) and the four health science researchers who indicated they had paid an apc were not sure if they had retained copyright ownership of their work. this finding highlighted the need to focus on the importance of understanding the conditions of a publisher copyright agreement in the session. this was pertinent given that researchgate™, a site which facilitates the sharing of research publications, was a frequently used social media tool (figures 2 and 3).     figure 4 relative proportions of researchers, based on pomp responses, who have paid an article processing charge (apc) to make an article open access; health science (15 responses, session 10) and life and environmental science (33 responses, sessions 3-8)     figure 5 display of results for the pomp questions 5a: "have you or a co-author paid an article processing charge to make any of your articles open access?" and 5b "if you have paid an article processing charge, do you know if you retained copyright ownership of your article?”. (responses from 38 submitted pomps distributed to 38 researchers, compiled progressively and presented in sessions 3-8).     despite the anonymity of pomp responses, examining answers to each question from individual responses was also useful for identifying learning needs. in relation to scholarly publishing, researchers attending session 9 each had some publishing experience, mostly with journals and conferences (table 2). however, only three researchers were familiar with the australian government higher education research data collection (herdc) (department of education, 2014) and excellence in research for australia (era) (australian research council, 2014). although this group of researchers was aware of the importance of publishing, most were not using the herdc and era specifications, a form of summative assessment for researchers in australian universities, to guide their publishing decisions. session content was therefore adapted to explain the specifications at an introductory level. similarly, of the 12 researchers in figure 3 who had used traditional media to promote their research, only 6 identified researchonline@jcu, the institutional repository, as a tool for communicating and promoting their research. this identified a point for discussion in the session, revealing that some researchers had only considered the institutional repository as an administrative reporting tool and not an avenue for communicating and promoting their research.     table 2 pomp responses from health science researchers about their experience in publishing academic research* have you published any of the following types of journal articles? have you published any of the following types of conference works? have you published any of the following types of books or book chapters? have you co-authored other types of works that are derived from your research? familiarity with herdc or era use of journal ranking tools peer reviewed article peer reviewed paper, poster, abstract or summary     era: excellence in research for australia   peer reviewed article, short note or commentary abstract or summary       journal citation reports peer reviewed article, non-refereed article, case study peer reviewed paper, non-refereed paper, poster         peer reviewed article, non-refereed article, short note or commentary poster, abstract or summary, edited a conference proceedings teaching material   herdc: higher education research data collection, era: excellence in research for australia journal citation reports peer reviewed article, non-refereed article, short note or commentary abstract or summary non-commercial   herdc: higher education research data collection journal citation reports non-refereed article         journal citation reports, scopus analytics     aim 3: are omps an effective tool for gaining feedback about researcher learning?   polling one minute papers (pomps)   pomps created an opportunity to learn about the audience prior to an upcoming session, freeing up time in a session for discussion and teaching additional content. pomps also created the opportunity to ‘hear’ from researchers who would normally not speak up in a session or who were only prepared to share information anonymously. even when the number of responses was low, due to a small number of researchers attending a session, pomps were still useful because all or most of the audience responded.   pomp responses were useful for refining how sessions were taught. for example, only a small proportion (14.7%) of researchers indicated that they had a good understanding of the term altmetrics, but all who selected this option provided an appropriate description of the term (table 3). in contrast, of the 75 researchers who responded to this question, 42% indicated they had no understanding and 22% indicated they had some understanding of altmetrics. given this range, discussion was encouraged in sessions to facilitate learning from peers who provided explanations that other session participants could easily relate to.   pomps created an unexpected insight from one group of remote researchers (session 14). enquiry as to why only one researcher had responded to the pomp revealed cultural differences as the underlying issue. the group’s research manager explained that the researchers were embarrassed that they had no or few publications and so were reluctant to respond to the pomp. this information was useful in itself as it indicated that the session content needed to be directed at getting the researchers started with publishing. further explanation about the pomp boosted the response rate to 6, although only 2 researchers attended the session. these insights will be considered in the planning of future sessions with this group of researchers.   feedback from a colleague who trialed the use of a pomp (session 16) reported that it was very interesting and useful to see the range of disciplines and prior knowledge of the researchers who had registered for the session. in this pomp, respondents were asked to table any prior questions, which gave the librarian time to prepare for complex questions.   reflective one minute papers (romps)   romp responses gave an insight into what researchers gained from their session. they were used by the librarian to determine whether intended learning outcomes were achieved, and consider how the teaching or content of future sessions could be improved. romps were particularly effective in facilitating reflective feedback from remote sessions with small numbers of participants. in discussions at the end of sessions with 2-5 participants (sessions 11-13), responses were received from all or most participants. in some cases, immediate clarification was given in the session by the librarian. for more complex issues, future sessions were offered as a response. although response rates were lower for larger sessions, the romps were still beneficial for encouraging reflection, receiving feedback, and continuing the conversation with session participants. romps were least effective for sessions 4-8, which were hands-on computer sessions with substantial discussion and feedback throughout the sessions. when asked to complete the romp, the researchers repeated comments they had made during the session, but only 2 of 30 recorded their feedback in a romp. feedback from these hands-on sessions was mostly positive, with the exception of one researcher who was frustrated with the work she needed to do to manage her online presence.   most researchers responding to the romps gave a response for each of the three questions. within the 23 romp responses, 17 researchers responded to all 3 questions, 4 responded to 2 questions, and 2 responded to only 1 question. using thematic analysis, responses were grouped into one of five categories (table 4). the number of responses to each of the three questions is presented in table 5. the categories “topic named” and “positive statement” suggest a relatively shallow level of learning and limited engagement with the session content. “reflective statement” suggests some level of engagement, while responses coded as “reflective statement with further insight” and “reflective question” each provide evidence of deeper learning. most responses to the question about the most important things learned were reflective statements. the majority of responses to the question about the most confusing thing covered were positive, indicating that respondents felt that they understood the session content. responses to the question about the most interesting thing learned indicated a range of levels of engagement from positive statements or a reflective statement with some demonstrating further insight.     table 3 pomp responses from researchers who indicated that they had some or a good understanding of altmetrics discipline understanding of altmetrics if you have heard of the term altmetrics, briefly describe what you understand it to mean? health sciences good stats of research acknowledgement health sciences good it appears to be a count of twitter mentions (although it might include more than that) health sciences some heard it but not sure of it's meaning health sciences some no much i understand it health sciences good a measure of the social media impact of your paper health sciences good means of measuring research impact life and environmental sciences good non-traditional metrics, number of mentions on websites, social media, media etc life and environmental sciences good proposed/potential alternative to impact factor, as a measure of influence life and environmental sciences good another way of measuring research 'impact' mixed disciplines good a number that represent the amount of attention an article receives from blogs, twitter, etc. mixed disciplines good measure of attention an article has received relative to 'lifespan'     table 4 categories used to group romp responses response category explanation of category example responses from this study topic named simple listing of an aspect covered in the session, with no insight to the researcher's learning ·  open access ·  altmetrics positive statement indicates session was worthwhile, with no insight to the researcher's learning ·  nothing was confusing excellently done ·  all of it. informative and interesting presentation. thank you. no negative feedback was recorded in any responses. reflective statement repeats content from the session, highlighting specific aspects ·  labouring the points about "dodgy" journals. i liked the tips to improve your altmetric score. ·  i also didn't know that twitter can be so useful. reflective statement with further insight what the researcher learnt and how it applies to their personal situation or how they will use what they have learnt ·  the most interesting thing i learn was about the importance of open access. i am going to bring some of this information to my lab group and postgrads. overall, thank you so much for taking the time and going over these things you are so knowledgeable and kind, it was a wonderful workshop! ·  the importance of twitter to academics. i always thought it was meant for teenagers. i will definitely sign up for a twitter account after this session. reflective question indicates deeper thinking by new issues that the session content raised for the researcher ·  probably not covered entirely, but i am interested how publications in journal with lower impact, but receive higher citations, might influence the indices we looked at during the session ·  nothing was confusing. however would have liked an example of how to tweet a publication. i have tweeted a publication but there are no doughnuts associated with the corresponding author publication list in research portfolio.   table 5 number of responses to each of the three romp questions   romp question and number of responses response category what was the most important thing you learnt? what was the most confusing thing i covered? what was the most interesting thing you learnt? topic named 2 1 1 positive statement 1 10 6 reflective statement 13 3 8 reflective statement with further insight 4 2 5 reflective question 0 4 1     discussion   this study describes a new approach to providing library support for researchers. the defining feature of this approach is that it is simple yet informative. the approach adapts and combines two strategies derived from a formative assessment framework. the first strategy is polling researchers using the one minute paper concept to (1) identify their learning needs and (2) increase levels of engagement (e.g., hoppenfeld, 2012). the polling one minute paper (pomp) is designed to be quick for researchers to complete, and easy for librarians to interpret and gain a snapshot of current learning needs of the target group of researchers. completing the pomp prior to the session gives responding researchers an indication of the session content, raising their interest and allowing them to self-assess their understanding. during the session, a summary of the anonymous pomp responses is presented.   sharing the pomp responses is effective in that it provides context relevant information against which session participants can benchmark themselves, and prompts peer to peer discussion within the context of the participants’ discipline. increased levels of discussion provide informal evidence of the effectiveness of this strategy, which would be unlikely to occur through didactic delivery of the session content. at the end of the session, participants are asked to complete a reflective one minute paper (romp), encouraging researchers to reflect on the session content and provide feedback which enables the librarian to review what the researchers gained from the session. this second strategy is adapted from the original one minute paper (chizmar & ostrosky, 1998), which was first used in library instruction by choinski and emmanuel (2006).   the pomp-romp approach can improve the value of library research support sessions for researchers. it offers a responsive approach to the current learning needs of researchers. variation in undergraduate students’ competencies represents a significant challenge in information literacy pedagogy (dunaway & orblych, 2011). researcher learning needs may be more diverse than that of undergraduate students, varying with career stage, discipline, current research priorities or activities, and previous training. the increasingly complex and competitive environment that researchers now work in (richardson, et al., 2012) also makes it difficult to identify and track researcher learning needs. given this variability, the two complementary omp tools described in this study are an effective means of rapidly obtaining a snapshot of the current learning status of a specific research group, immediately prior to and following a library support training session. librarians can use this snapshot to determine the level of detail that they teach in a session, with the aim of meeting the learning needs of researchers and providing the right amount of challenge to spark engagement in the topic. using a tailored approach also creates proactive rather than passive library services, providing “just-in-time” and “just-for-me” assistance (association of college and research libraries, 2010).   the pomp-romp approach is designed to slot into researchers’ workflow. the simplicity of both the pomps and romps mean that they can be prepared at short notice and used for small, large, remote, face-to-face, lecture, or hands-on sessions. the flexibility of these tools enables the library to contribute to improving researcher learning without adding additional burden to a researcher’s workload (parker, 2012). the pomps and romps also extend the opportunity for discussion with researchers beyond the defined period of a library research support session, creating further opportunity to build and strengthen the researcher-librarian relationship (auckland, 2012; parker, 2012). in this study, romp feedback received in discussion at the end of sessions was responded to directly, with the offer of future sessions where relevant. further consideration is needed for how to respond to written responses. one option would be to ask researchers to include their name on the romp if they would like further information. another option would be to provide a single response, shared with all session participants, which responds to all issues raised in each romp from the corresponding session.   many academic libraries are now developing or offering support programs for their researchers (auckland, 2012; richardson, et al., 2012). maximizing the benefit for researchers attending library research support sessions will also benefit libraries by efficiently using the time that librarians spend in supporting researchers. efficient use of the time of librarians is important because it is likely that libraries will develop research support services with no or little additional resources (kennan, cole, willard, wilson, & marion, 2006; kennan, et al., 2014). therefore, as more librarians become involved in supporting researchers in their institution, a coordinated approach will be necessary to make efficient use of limited library resources. aggregating pomp and romp data from all library research sessions could be used as a professional development tool for research support librarians, as also described for peer review of teaching data (drew & klopper, 2014). from the perspective of academic libraries, aggregated pomp and romp data could be used for identifying learning gaps, sharing evolving perspectives in researcher feedback, and optimizing content in library research support programs. such a community of practice could fast track the development of research librarians (drew & klopper, 2014). as more librarians become involved, the opportunity for peer review of library instruction could also lead to new insights (drew & klopper, 2014) which may improve pomps, romps and other strategies for teaching library support to researchers.   one of the most important but often overlooked parts of the assessment cycle is for teachers or instructors to close the loop by reflecting on results and making appropriate changes, such as adjusting teaching methodologies or changing the structure of a program (oakleaf, 2009; swoger, 2011). ultimately, the goal of all instruction and assessment efforts is to engage in reflective practice (oakleaf, 2014). teaching programs can nearly always be improved (swoger, 2011) and should evolve to keep pace with new teaching strategies and content. pomps and romps are a viable means of gaining regular, systematic feedback from researchers to assist with developing and improving library research support programs. if implemented as a routine practice, the pomp-romp model could support a cyclical process of quality control and improvement.   academic libraries and librarians must demonstrate their value. libraries and librarians can no longer rely on an assumed belief by stakeholders that they are important (association of college and research libraries, 2010). this is particularly true for the context of library research support. in this relatively new enterprise of research support, libraries are trialing various models of operation and entering spaces that were previously the domain of other sections of the university, e.g., research offices. assessing impact, to demonstrate value, is made more complicated because academic libraries operate in a changing environment in which people, services, and needs are constantly evolving (association of college and research libraries, 2010). it is also difficult to prove that actions taken by the library contributed to improvements in the performance of researchers (association of college and research libraries, 2010). implementing the pomp-romp approach as standard operating practice for library research support will develop a data source which provides evidence of a library’s value and how its contribution to the support of the institution’s researchers has changed over time.   limitations   the strength but also limitation of the pomp-romp model is that collected information applies to a specific situation, with respect to a topic and point in time. pomp-romp responses should not be extrapolated to other contexts.   each pomp-romp dataset provides a snapshot of the status of the learning of a specific group of researchers. no control groups are used and sample sizes are often small and not randomly selected. all responses are analyzed and samples are comprised of researchers with similar interests or skill levels. questions asked in pomps are tailored to the topic of an upcoming session, so are not intended to provide a detailed or comprehensive insight to library research support issues.   it should also be recognised that pomp and romp responses may not reflect the learning or experience of all researchers in a corresponding session. the needs of researchers who do not submit a pomp may not be considered in the preparation of session content. non-responses are most likely due to the researcher having other priorities, not seeing a personal benefit in submitting a response, or being reluctant to share his thoughts. in the most extreme situation, non-response could be due to a lack of understanding of the question. given these assumptions, an unexpected finding was that the number of pomp responses for sessions 3, 10, 12 and 14 was higher than session attendance. as a result, the needs of the responding researchers may be addressed but not actually apply to the researchers who attended the session. in the current study, non-response to romps seemed mainly due to researchers not having anything to report that was not already expressed during the session. unfortunately, this verbal feedback was not recorded for longer term analysis and comparison. in future sessions, the value of completing a romp will be emphasized, to encourage researchers to reflect on the session content and to record feedback to the librarian.   conclusions   polling one minute papers (pomps) and reflective one minute papers (romps) offer a new approach for librarians to guide the learning process of the researchers they support. this study demonstrates that pomps and romps have the potential to be remarkably informative, despite requiring little effort on the part of the researchers or librarians. pomps and romps are adaptations of conventional formative assessment tools which extend the opportunity for librarians to engage with researchers, both before and after a library research support session. pomps allow researchers to benchmark the status of their learning needs and assist librarians to identify learning gaps. romps encourage researchers to reflect on what they learned in library research support sessions and assist librarians to determine whether intended learning outcomes were achieved. the simplicity of pomps and romps enable them to be slotted into researchers’ workflow. as librarians take on research support duties, these tools can be used to share recorded evidence of the evolving learning needs of researchers. responses to pomps and romps also document evidence of the value that a library has contributed to supporting its researchers.   references   association of college and research libraries. (2010). the value of academic libraries: a comprehensive research review and report. researched by megan oakleaf. chicago: association of college and research libraries. retrieved from http://www.acrl.ala.org/value/page_id=21   auckland, m. (2012).  re-skilling for research an investigation into the role and skills of subject and liaison librarians required to effectively support the evolving information needs of researchers. retrieved from http://www.rluk.ac.uk/files/rluk%20re-skilling.pdf   australian research council. (2014). excellence in research for australia (era). retrieved from http://www.arc.gov.au/era/default.htm   bartlett, m. g., & morrow, k. a. (2001). method for assessing course knowledge in a large classroom environment: an improved version of the minute paper. american journal of pharmaceutical education, 65(3), 264-267.   beall, j. (2014). scholarly open access: critical analysis of scholarly open-access publishing. retrieved from http://scholarlyoa.com/   boud, d. (2000). sustainable assessment: rethinking assessment for the learning society. studies in continuing education, 22(2), 151-167. http://dx.doi.org/10.1080/713695728   boud, d., & lee, a. (2005). ‘peer learning’ as pedagogic discourse for research education. studies in higher education, 30(5), 501-516. http://dx.doi.org/10.1080/03075070500249138   brodie, l. (2012). problem-based learning. in l. hunt & d. chalmers (eds.), university teaching in focus: a learning-centred approach. hoboken: taylor and francis.   chizmar, j. f., & ostrosky, a. l. (1998). the one-minute paper: some empirical findings. the journal of economic education, 29(1), 3-10. http://dx.doi.org/10.1080/00220489809596436   choinski, e., & emmanuel, m. (2006). the one-minute paper and the one-hour class: outcomes assessment for one-shot library instruction. reference services review, 34(1), 148-155. http://dx.doi.org/10.1108/00907320610648824   creaser, c., & spezi, v. (2013). improving perceptions of value to teaching and research staff: the next challenge for academic libraries. journal of librarianship and information science, 46(3), 191-206. http://dx.doi.org/10.1177/0961000613477678   crisp, g. (2009). designing and using e-assessments. milperra, nsw: higher education research and development society of australasia.   department of education. (2014). higher education research data collection (herdc). retrieved from http://education.gov.au/higher-education-research-data-collection   dillman, d. a., smyth, j. d., & christian, l. m. (2009). internet, mail, and mixed-mode surveys: the tailored design method. new jersey: wiley.   dobson, j. l. (2008). the use of formative online quizzes to enhance class preparation and scores on summative exams. advances in physiology education, 32, 297-302. http://dx.doi.org/10.1152/advan.90162.2008   draper, s. w. (2009). catalytic assessment: understanding how mcqs and evs can foster deep learning. british journal of educational technology, 40(2), 285-293. http://dx.doi.org/10.1111/j.1467-8535.2008.00920.x   drew, s., & klopper, c. (2014). evaluating faculty pedagogic practices to inform strategic academic professional development: a case of cases. higher education, 67(3), 349-367.  http://dx.doi.org/10.1007/s10734-013-9657-1   drummond, g. (2007). programme feedback: it only takes a minute! international journal of educational management, 21(1), 29-36. http://dx.doi.org/10.1108/09513540710716803   dunaway, m. k., & orblych, m. t. (2011). formative assessment: transforming information literacy instruction. reference services review, 39(1), 24-41. http://dx.doi.org/10.1108/00907321111108097   frances, m., fletcher, j., & harmer, s. (2011). reshaping and rescoping university libraries to fit changing academic requirements. paper presented at the 32nd iatul annual conference: libraries for an open environment: strategies, technologies and partnerships, warsaw, poland. retrieved from http://www.bg.pw.edu.pl/iatul2011/proceedings/ft/frances_m.pdf   google. (2014). create a survey using google forms. retrieved from http://support.google.com/drive/answer/87809?hl=en   guest, g., macqueen, k. m., & namey, e. e. (2012). applied thematic analysis. los angeles: sage publications.   haddow, g. (2012). research support in a research assessment environment: the experience of 'new' universities. library and information research, 36(113), 62-80.   hoppenfeld, j. (2012). keeping students engaged with web-based polling in the library instruction session. library hi tech, 30(2), 235-252. http://dx.doi.org/10.1108/07378831211239933   hughes, h., & bruce, c. (2012). informed learning in online environments: supporting the higher education curriculum beyond web 2.0. in p. p. godwin & j. parker (eds.), information literacy beyond library 2.0 (pp. 65-79). london: facet.   james cook university. (2014). facts & figures 2014: data snapshot. retrieved from http://www.jcu.edu.au/about/public/groups/everyone/documents/annual_report/jcu_140345.pdf   james cook university. (2015). about jcu: strategic intent. retrieved from http://www.jcu.edu.au/about/strategic-intent/   kaufman, p. t. (2009). carpe diem: transforming services in academic libraries. in ideals. retrieved from http://www.ideals.illinois.edu/handle/2142/12032   kennan, m. a., cole, f., willard, p., wilson, c., & marion, l. (2006). changing workplace demands: what job ads tell us. aslib proceedings: new information perspectives, 58(3), 179-196. http://dx.doi.org/10.1108/00012530610677228   kennan, m. a., corrall, s., & afzal, w. (2014). "making space" in practice and education: research support services in academic libraries. library management, 35(8/9), 666-683. http://dx.doi.org/10.1108/lm-03-2014-0037   knowles, m. s. (1975). self-directed learning: a guide for learners and teachers. englewood cliffs, n.j.: cambridge adult education.   kolb, d. a. (1984). experiential learning: experience as the source of learning and development. upper saddle river, nj: prentice-hall.   merriam, s. b. (2001). andragogy and self-directed learning: pillars of adult learning theory. new directions for adult and continuing education, 2001(89), 3-13. http://dx.doi.org/10.1002/ace.3   oakleaf, m. (2009). the information literacy instruction assessment cycle: a guide for increasing student learning and improving librarian instructional skills. journal of documentation, 65(4), 539-560. http://dx.doi.org/10.1108/00220410910970249   oakleaf, m. (2014). a roadmap for assessing student learning using the new framework for information literacy for higher education. the journal of academic librarianship, 40(5), 510-514. http://dx.doi.org/10.1016/j.acalib.2014.08.001   parker, r. (2012). what the library did next: strengthening our visibility in research support. paper presented at em-powering efutures, the 16th biennial vala conference and exhibition, melbourne, victoria, australia.  retrieved from http://hdl.handle.net/1959.3/214486   parsons, a. (2010). academic liaison librarianship: curatorial pedagogy or pedagogical curation? ariadne, 65, 1-11. retrieved from http://www.ariadne.ac.uk/issue65/parsons/   priem, j., taraborelli, d., groth, p., & neylon, c. (2010). altmetrics: a manifesto. retrieved from http://altmetrics.org/manifesto/   ref2014. (2014). ref2014: research excellence framework. retrieved from http://www.ref.ac.uk/   richards, l. (2015). handling qualitative data: a practical guide (3rd ed.). los angeles: sage publications.   richardson, j., nolan-brown, t., loria, p., & bradbury, s. (2012). library research support in queensland: a survey. australian academic & research libraries, 43(4), 258-277. http://dx.doi.org/10.1080/00048623.2012.10722287   sadler, d. r. (1989). formative assessment and the design of instructional systems. instructional science, 18(2), 119-144. http://dx.doi.org/10.1007/bf00117714   simons, n., & richardson, j. (2013). new content in digital repositories: the changing research landscape. oxford: chandos.   stead, d. r. (2005). a review of the one-minute paper. active learning in higher education, 6(2), 118-131. http://dx.doi.org/10.1177/1469787405054237   swoger, b. j. m. (2011). closing the assessment loop using preand post-assessment. reference services review, 39(2), 244-259. http://dx.doi.org/10.1108/00907321111135475     appendix session # discipline / client group questions response options   1 life and environmental sciences: post-docs how would you rate your understanding of altmetrics? none; low; good   1 life and environmental sciences: post-docs which of the following do you use to do your research, or to talk about your research interests? academia.edu; blogging; citeulike; comments on other sites; facebook; mendeley; newspaper interviews; radio interviews; researchgate; slideshare; twitter; wikipedia; youtube; other   1 life and environmental sciences: post-docs which of the following sites do you use to manage your research profile? jcu research portfolio; researchonline@jcu; researcherid; scopus; orcid; google scholar; other   2 life and environmental sciences is the information in your jcu research portfolio profile up-to-date? yes; no; other   2 life and environmental sciences which of the following do you use to do your research, or to talk about your research interests? researchonline@jcu; researcherid; scopus; orcid; google scholar; other   2 life and environmental sciences which of the following sites do you use to manage your research profile? academia.edu; blogging; citeulike; comments on other sites; facebook; mendeley; newspaper interviews; radio interviews; researchgate; slideshare; twitter; wikipedia; youtube; other   3, 4, 5, 6, 7, 8 life and environmental sciences have you used any of the following to promote or discuss your research? researcherid; scopus; google scholar; orcid; the conversation; traditional media newspaper, radio, tv; blogging; comments on the blogs or articles by others; twitter, researchgate; youtube; linkedin; mendeley; other   3, 4, 5, 6, 7, 8 life and environmental sciences have you heard of the term altmetrics? yes; no; not sure   3, 4, 5, 6, 7, 8 life and environmental sciences if you have heard of the term altmetrics, briefly describe what you understand it to mean? [free text]   3, 4, 5, 6, 7, 8 life and environmental sciences if you have paid an article processing charge, do you know if you retained copyright ownership of your article? yes; no; not sure   3, 4, 5, 6, 7, 8 life and environmental sciences have you or a co-author paid an article processing charge to make any of your articles open access? yes; no; not sure   9 health sciences have you published any of the following types of journal articles? peer reviewed article; non-refereed article; case study; short note or commentary; other   9 health sciences have you published any of the following types of conference works? peer reviewed paper; non-refereed paper; poster; abstract or summary; edited a conference proceedings; other   9 health sciences have you published any of the following types of books or book chapters? research; non-research; teaching material; non-commercial; reference; later edition; report   9 health sciences tick the box if you are you familiar with the following government reporting or assessment exercises: herdc: higher education research data collection; era: excellence in research for australia; other   9 health sciences tick the box if you have used any of the following for ranking the value of a journal: journal citation reports; scopus analytics; beall's list of predatory publishers; other   10 health sciences have you used any of the following to promote or discuss your research? researcherid; scopus; google scholar; orcid; the conversation; traditional media newspaper, radio, tv; blogging; comments on the blogs or articles by others; twitter, researchgate; youtube; linkedin; mendeley; other   10 health sciences have you heard of the term altmetrics? yes; no; not sure   10 health sciences if you have heard of the term altmetrics, briefly describe what you understand it to mean? [free text]   10 health sciences have you or a co-author paid an article processing charge to make any of your articles open access? yes; no; not sure   10 health sciences if you have paid an article processing charge, do you know if you retained copyright ownership of your article? yes; no; not sure   11 health sciences have you published any of the following types of journal articles? peer reviewed article; non-refereed article; case study; short note or commentary; other             11 health sciences have you published any of the following types of conference works? peer reviewed paper; non-refereed paper; poster; abstract or summary; edited a conference proceedings; other   11 health sciences have you published any of the following types of books or book chapters? research; non-research; teaching material; non-commercial; reference; later edition; report   11 health sciences would you consider that you have a fair understanding of: herdc: higher education research data collection; era: excellence in research for australia; other   11 health sciences are any of your publications open access: in researchonline@jcu?; on the publisher's website?; other   12 social sciences and humanities are you the author of a peer reviewed journal article? no; yes   12 social sciences and humanities are you the author of a peer reviewed conference paper? no; yes   12 social sciences and humanities are you the author of a commercially published book or book chapter about a research topic? no; yes   12 social sciences and humanities do you have publications available in researchonline@jcu? no; yes   12 social sciences and humanities do you have a google scholar profile that lists your research outputs? no; yes   12 social sciences and humanities if you have answered yes to questions 1,2 or 3, have you searched for your publications in web of science? no; yes   12 social sciences and humanities if you have answered yes to questions 1,2 or 3, have you searched for your publications in scopus? no; yes   12 social sciences and humanities is there any topic or question that you would like me to talk about in the session? [free text]   13 social sciences and humanities have you used any of the following to promote or discuss your research? researcherid; scopus; google scholar; orcid; the conversation; traditional media newspaper, radio, tv; blogging; comments on the blogs or articles by others; twitter, researchgate; youtube; linkedin; mendeley; other   13 social sciences and humanities have you heard of the term altmetrics? yes; no; not sure   13 social sciences and humanities if you have heard of the term altmetrics, briefly describe what you understand it to mean? [free text]   13 social sciences and humanities have you or a co-author paid an article processing charge to make any of your articles open access? yes; no; not sure   13 social sciences and humanities if you have paid an article processing charge, do you know if you retained copyright ownership of your article? yes; no; not sure   14 social sciences and humanities how many herdc eligible publications do you have? this includes peer reviewed publications or commercially published books or book chapters. [give number]   14 social sciences and humanities have you heard of the term altmetrics, and how would you rate your understanding of altmetrics? i have not heard of the term; i have some understanding; i have a good understanding   14 social sciences and humanities which of the following do you use to manage your research profile? jcu research portfolio; researchonline@jcu; researcherid; scopus; orcid; google scholar; other   14 social sciences and humanities which of the following do you use to do your research, or to talk about your research? academia.edu; blogs; comments on any sites; facebook; mendeley; newspaper interviews; radio interviews; researchgate; slideshare; twitter; wikipedia; youtube; other   15 mixed disciplines: academic teaching staff have you published any of the following outputs about your research? peer reviewed journal article; non-peer reviewed journal article; non-peer reviewed conference paper; book or book chapter; report; other   15 mixed disciplines: academic teaching staff have you published any of the following outputs about your teaching? peer reviewed journal article; non-peer reviewed journal article; non-peer reviewed conference paper; book or book chapter; report; other   15 mixed disciplines: academic teaching staff do you use any of the following social media tools to talk about or promote your teaching and learning? blogs; comments on other people's posts; twitter; facebook; academic.edu or researchgate; slideshare; wikipedia; youtube; (your) jcu research portfolio; (your) google scholar profile; other   16 mixed disciplines: higher degree research students what is the discipline or subject area of your research?   16 mixed disciplines: higher degree research students are you a: research masters student; phd student; early career researcher i.e. are you within 5 years of starting your post-phd research career?; other 16 mixed disciplines: higher degree research students are you using (or planning to use) material in your thesis for which the copyright may have expired? select other if you would like to provide more information. yes; no; not sure; other 16 mixed disciplines: higher degree research students are you using (or planning to use) material in your thesis which is still under copyright? select other if you would like to provide more information. yes; no; not sure; other 16 mixed disciplines: higher degree research students have you ever requested copyright permission to use someone else's work in your research? yes; no; not sure; other 16 mixed disciplines: higher degree research students are you planning to publish your research in: journal article(s); conference paper(s); book or book chapter(s); other 16 mixed disciplines: higher degree research students have you heard of creative commons? yes; no; not sure 16 mixed disciplines: higher degree research students do you have a copyright question that you would like answered in the workshop? please provide more details here. [free text]   article   a holistic look at reference statistics: whither librarians?   b. jane scales reference team leader / e-projects librarian washington state university libraries pullman, washington, united states of america email: scales@wsu.edu   lipi turner-rahman faculty washington state university pullman, washington, united states of america email: ilipi@wsu.edu   feng hao visiting lecturer of sociology university of richmond richmond, virginia, united states of america email: fhao@richmond.edu   received: 22 june 2015   accepted: 17 nov. 2015       2015 scales, turner-rahman, and hao. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – washington state university (wsu) pullman campus librarians track a diverse set of reference statistics to gain a “holistic” look at local reference transaction trends. our aim was to aggregate virtual, reference desk and office transaction data over the course of three years to determine staffing levels. specifically, we asked “where should reference librarians be to answer questions?”   methods – using springshare’s libanalytics, we generated longitudinal (2012-2014) statistics and data, to help us assess the patterns and trends of patron question numbers, types, communication modes, and locations in the terrell library. with this data, we considered current staffing patterns and how we could best address patron needs.   results – researchers found that compiling data across modalities of location, communication, question type, and the read scale led to a better understanding of user behavior trends.   conclusion – examining and interpreting a more inclusive and richer set of transaction statistics gives reference managers a better picture of how patrons are seeking help, and can serve as a basis for making staffing decisions. introduction   washington state university (wsu), a land grant institution, was established in 1890. its main campus is located in pullman. the largest library on campus, holland and terrell, houses the humanities and social sciences collection as well as the only traditional reference desk on campus.  the library is contained in two buildings, with the reference desk located in terrell.   reference services on the wsu pullman campus are coordinated by the libraries’ reference steering committee. this group establishes service hours and staffing for desk and online chat services, and co-ordinates services for the email-based libanswers. with tight budgets and a changing student population, the committee was tasked with assessing the demand for services. in 2014, the group looked at a comprehensive set of statistics covering three years of reference services to get a better picture of the behavior trends of patrons seeking assistance.   we had been aware that our reference questions had declined in number for several years. between 2012 and 2014, the committee made incremental adjustments to the schedule and staffing of the reference desk, based on a cursory review of data.   we implemented a tiered-reference model, a term which massey-burzio (huling, 2002) articulated. tiered or stratified reference models use paraprofessionals as the first point of contact for patrons needing help. these were, in our case, a mix of undergraduate and graduate students, who were instructed to refer patrons on to librarians and subject specialists (the next tier) if their needs required more expertise to address and answer.   after three years taking this tactic, however, we lacked a clear picture of how those changes met and fulfilled (or not) patrons’ needs. to understand what patrons wanted, and where reference was happening, the committee reviewed a comprehensive set of statistics spanning several years. the question the committee posed to the data was, “where is reference happening in terrell library, and at what level of complexity? in which location(s) are the librarians most needed to answer reference questions, and how does the data show this?”   literature review   there is little debate that academic library reference services have changed in the last several years. tyckoson (2012) points out that, while the concepts of what constitutes reference services have remained stable since the 1870s, the tools and skills used to deliver those services have evolved dramatically. our means of communication with patrons, staffing pressures, assessment practices, and information access have all contributed to a more complex and nuanced view of reference.   many have also described the downturn in the use of traditional academic reference services. reference transactions have declined significantly. martin (2009) cites statistics from the association of research libraries (arl) that document this downturn. coffman (2012) describes these changes as a “decline of the library empire.” the result of this has been dramatic. some universities have abandoned the reference desk, and replaced it with other models of patron assistance (lederer & feldmann, 2012). others have tried new models such as combined reference desks and tiered services, to form new collaborative models (meserve, belanger, bowlby, & rosenblum, 2009; deineh, middlemas, & morrison, 2011; dinkins & ryan, 2010).   these changes challenge library managers. multiple models of reference mean different methods of collecting transaction statistics, and require a more intensive and inclusive look at data. king and christensen-lee (2014) prefaced their study by reviewing longitudinal trends of patrons’ question-types, as well as overall trends specific to email and online chat reference, at the valley library at oregon state university.    baro, efe, and oyeniran (2014) looked at a more expansive set of possible reference channels used within the universities of nigeria by surveying librarians. specifically, they considered: in-person visits, facebook, telephone, short message service (sms), instant messaging, and email. the authors’ consideration of so many means of communication is unique, and necessary in order to discover the preferred method of their patrons to ask questions.   when the wsu’s libraries’ reference steering committee decided to evaluate services provided by the largest, most used library on the pullman campus, members looked to the literature for guidance. the committee found the recommendations in kern’s 2006 article a good basis for the study. kern advises librarians to think “...about your reference services as a single reference service with many modes of communication.” she asks researchers to delineate a clear and precise question to ask of their statistics before embarking on surveying the data.   methodology   in order to understand where questions were being answered, and the nature and complexity of those questions, the wsu libraries’ reference steering committee looked at a multifaceted set of data: im (instant messaging), libanswers, email, phone, in-person reference desk, and in-person office visits. implicit to this research was an examination of how the tiered reference model was working.   since 2012, the reference team has used springshare’s libanalytics, a popular tool reviewed by dworak (2011). libanalytics facilitates the customization and consolidation of reference transaction data across communication modes. both flatley and jensen (2012) and gossett, stephan, and marrall (2012) describe this tool’s flexibility.    after every transaction, library staff (both librarians and student employees) record their location, type of question, mode of communication, read scale difficulty, length of the transaction, and whether or not the exchange required the use of government documents. the interface provides reference workers multiple forms, including text boxes, check boxes, radio buttons, likert scale, and multiple column categories of information to characterize a transaction (see figure 1).   table 1 outlines the various data points that staff record after each reference transaction. only a few points need to be further explained. the contact type includes the various communication modes patrons use to request information from us. currently, there are five options available.     figure 1 libanalytics transaction screen as configured by the wsu pullman libraries.     table 1 data types recorded for reference transactions contact type person-to-person, telephone, instant messaging (im), email, libanswers (a variation of email) level of question difficulty read scale (reference effort assessment data) – indicates the complexity of the question, and the amount of effort necessary to answer the question on a scale of 1 to 6 question type policy, technology, directional, reference location & service points terrell library was the consistent “location.”  service points included in this study included reference desk and office     read scale assessment, devised by gerlich and berard (2007), is a qualitative measurement of the amount of effort and knowledge necessary to answer a question. questions deemed a read level 1 require no specialized knowledge, so that staff can answer them without consulting a database or our libguides. questions assessed at levels 2 and 3 require increased knowledge and effort to answer. student employees, who participate in our tiered reference model, are trained to recognize the point at which a question should be passed off or referred to a subject specialist librarian. these are considered the higher-end level 3 questions.   in addition to tracking the read scale levels of questions, we track a set of locally identified and defined “question types.”   question type   policy queries regarding library policies, such as book loan periods, fines, purchase requests technology requests for help with printing, assistance with software on public computers or thumb drives directional questions regarding the location of a book, a study room, a classroom, the restrooms, etc. reference research assistance; demonstrating to others how to search a database, develop a topic, evaluate and identify useful information   locations used by reference staff for this study included the terrell reference desk and the librarians’ offices. those options can be seen illustrated in figure 1 under the location and service points columns (see table 1).   the authors extracted the data from libanalytics, to increase the reliability and correct interpretation of the information. the charts, tables, and figures were organized in a single document before our analysis began. lastly, we reviewed terrell library gate counts between 2012 and 2014.   the committee also combined these different data points, figuring monthly and semester averages of terrell reference transactions. in an effort to account for any other factors that affect the number and quality of reference transactions, we gathered data on how our springshare libguides were accessed. while it is not reasonable to conclude that a reference question was answered with every access of a libguide, the group saw value in looking at overall usage trends. similarly, we looked at any changes in foot traffic into the terrell library by gathering gate count data.   results   our first interest was documenting the change in the number of reference transactions from the two most common location and service points: the terrell reference desk and librarian offices. table 2 contains the monthly average number of questions answered at the terrell reference desk over the last several years.   between 2012 and 2013, the number of average monthly questions answered at the desk declined by 16%. a more significant decline in reference desk transactions was recorded in 2014, when the average number of questions librarians received at the terrell reference desk every month dropped to 584: a 35% decline from 2012, and a 22.5% decrease from 2013.   during the same time period, the average number of questions librarians answered in their offices declined 33% from 2012 to 2013, but increased slightly by 4.4% from 2013 to 2014.     table 2  monthly average number of office and reference desk questions by for 2012, 2013, and 2014 year monthly average of reference desk questions answered monthly average of office questions answered 2012 898 135 2013 754 90 2014 584 94     table 3 temporal look at communication modes in librarian offices transactions in 2012 transactions in 2013 transactions in 2014 in-person 427 (22%) 71 (7%) 85 (8%) telephone 164 (8%) 148 (14%) 87 (8%) im 640 (33%) 311 (29%) 87 (8%) email 323 (17%) 244 (23%) 450 (40%) libanswers 395 (20%) 304 (28%) 415 (37%)     using libanalytics, the reference committee then looked for trends in the data points represented in table 1. contact types did not significantly change at the reference desk between 2012 and 2014. for example, while the number of questions answered at the desk declined over those years, the percentage occurring “in-person” changed only 1%, from 93% in 2012 and 2013 to 94% in 2014. telephone calls hovered between 4% and 5%. im reference and email reference at the desk remained stable around 3% and 0%, respectively.   reference transactions that took place within librarian offices, however, changed much more (see table 3). the percentage of questions answered via email and libanswers has risen significantly, from a combined 37% in 2012 to 77% in 2014.   next, we looked for trends in the read levels recorded by library staff at the terrell reference desk and terrell offices over the same three year time period. we noted that in 2012, it was more common for staff to forget to record the read level, so data for many transactions were not recorded. over time, we became better at understanding the use of the read scale, and used it more frequently. for example, in 2012 a monthly average of 53 transactions, which accounts for 33% of the total transactions that took place in librarian offices, was not assigned a read number. in 2013, the number of unassigned transactions fell to 5%. by 2014, the percentage of office transactions that did not have a read scale number assigned was only 3%. the averages in table 4 below include only those transactions that were recorded with a read scale number.   read value 1-3 questions (those which require less effort to answer) comprise the bulk of questions at the reference desk, while those read values 4 and 5 have decreased. this is an inverse of the situation in the librarian offices. read value 1 in librarians office fell dramatically, whilst value 2 remained constant. read values 3, 4 and 5 increased from 49% in 2012 and 58% in 2013, to 62% in 2014. temporary employees (tes) providing reference service give assistance for questions including some with read value 3. read value 3 questions which are out of the tes’ area of study, and all of those of read value 4 and up are transferred or referred to a librarian by the temporary employees. referral can happen either by email or by furnishing contact information for the appropriate subject specialist.     table 4  monthly averages of reference desk questions and office questions by read values, 2012-2014 reference desk questions   read values monthly read value averages for 2012 monthly read value averages for 2013 monthly read value averages for 2014 1 187 (37%) 287 (40%) 248 (44%) 2 161 (32%) 252 (35%) 179 (31%) 3 125 (25%) 160 (22%) 127 (22%) 4 30 (6%) 24 (3%) 14 (2%) 5 6 (1%) 2 (0%) 1 (0%) 6 0 0 0 office questions 1 33 (30%) 4 (5%) 5 (5%) 2 27 (25%) 22 (26%) 25 (27%) 3 38 (35%) 43 (51%) 47 (51%) 4 9 (8%) 12 (14%) 12 (13%) 5 2 (2%) 3 (4%) 3 (3%) 6 0 0 0     table 5 monthly averages of question types at the terrell reference desk and librarian offices january 2012-december 2014 reference desk monthly averages for 2012 monthly averages for 2013 monthly averages for 2014 policy 12 (1%) 32 (4%) 13 (2%) technology 125 (14%) 150 (20%) 113 (19%) directional 301 (34%) 258 (34%) 186 (32%) reference 459 (51%) 313 (42%) 272 (47%) offices policy 4 (2%) 11 (12%) 15 (16%) technology 28 (17%) 22 (24%) 32 (34%) directional 49 (30%) 9 (10%) 7 (8%) reference 83 (51%) 48 (53%) 39 (42%)     the next set of tables displays trends in the types of questions asked. table 5 shows a slight dip in reference questions – those queries related to searching for and finding information. technology questions at the desk, on the other hand, have increased by 5%. the greater percentage of policy questions in 2013 was partly due to some confusion as to what constituted that type of question. after some training and discussion, the group of librarians and staff reached a broader consensus on what constitutes policy questions, which changed how those were recorded. generally, table 5 shows that the percentages of each of the question types have remained consistent at the terrell reference desk.   table 5 demonstrates real changes occurring in the type of questions librarians are seeing in their offices: increasingly, a higher percentage concerns policy and technology. the percentage of reference and directional questions has dipped.   we examined how reference hours (the number of hours that the terrell reference desk offered services) and the staffing level (number of librarians and staff hours spent at the reference desk) changed over the three years, comparing numbers by like semester, to track the trends. table 6 shows that the hours of service have dropped 9% during spring and fall semesters.   table 7 shows that between 2012 and 2014, the average number of hours the desk was staffed during the spring semester dropped almost 12% (from 223 to 198). average hours for staffing during the fall dropped 13% (239 to 210   next, we looked at the changes in reference desk staffing in terms of library staff vs. graduate student worker or temporary employees (tes) assigned to the desk. the tes consisted of primarily graduate students, as well as a few select undergraduates. table 8 shows that the role of tes at the desk has increased and librarian time has decreased. tes now staff the desk at almost the same level as librarians. there are rarely two librarians on the desk simultaneously. more commonly, one librarian and a te, or two tes, are at the desk at any time.     table 6 number of regularly scheduled hours per week in which the terrell reference desk provided service year semester number of regularly scheduled hours per week 2012 spring 46   summer 20   fall 46 2013 spring 46   summer 20   fall 42 2014 spring 42   summer 20   fall 42     table 7 terrell reference desk staffing levels 2012-2014 spring 2012 spring 2013 spring 2014 total hours staffed 1112 1299 988 total questions 9632 6855 5729 average hours staffed per month 223 260 198 hours to question ration .12 .19 .17 summer 2012 summer 2013 summer 2014 total hours staffed 285 302 295 total questions 4605 2870 1173 average hours staffed per month 95 101 98 hours to question ration .06 .11 .25 fall 2012 fall 2013 fall 2014 total hours staffed 955 742 843 total questions 8203 5945 2943 average hours staffed per month 239 186 210 hours to question ration .12 .12 .29     table 8 also compares staffing levels for librarians and tes, and provides an hours-to-questions ratio. the ratios have increased between 2012 and 2014, meaning that more time was spent per question (including time between questions).   the researchers also decided to incorporate data not previously considered in past reference service assessment. by collecting data on the use of libguides, a springshare product which facilitates the creation of online content by non-programmers, we tracked additional patron activity (see table 9). over the past five years, libguides have replaced many of the libraries’ web sites, and serve as informational resources for instruction and research. between 2012 and 2014, the use of the wsu libraries’ libguides increased 6.4%.   finally, we looked at annual gate counts for the terrell library, to see if there was any possible correlation between in-library reference traffic and overall traffic. between 2012 and 2014, foot traffic in the terrell library actually increased by 3.3%. some part of this increase stems from the library going to a 24/7 operational schedule in 2014. see table 10.     table 8 librarian hours vs. temporary employee hours at the terrell reference desk for spring and fall semesters 2012-1014 average librarian hours per week average te hours per week percent of weekly staffed hours by tes spring 2012 46.5 21 31% fall 2012 53 12 18% spring 2013 54 24 31% fall 2013 32.5 23 41% spring 2014 33 27 45% fall 2014 33 23 41%     table 9 libguide views 2012-2014 year views of published libguides in thousands 2012 217.4 2013 242.2 2014 231.3     table 10 gate counts for terrell library 2012-2014 year gate counts (in the millions) 2012 1.033 2013 1.037 2014 1.067     discussion   the committee took considerable time to understand how this data informed answers to our research questions:  “where is reference happening, and at what level of complexity? where are librarians most needed?”   the university population seems increasingly comfortable accessing online information from the libraries. evidence of this can be seen in a) the increased use of libguides, and b) the rise in number of email and libanswers transactions. the latter composed 77% of questions answered in librarian offices in 2014. this indicates that librarians are more needed in their offices where libguide maintenance is more likely to occur, and where the other online transactions can happen without the interruption or time constraints one experiences at the desk.   the steep drop of in-person transactions at the terrell reference desk seen in the data occurred when building hours actually expanded, and gate counts were rising. students are entering the library to use it as a study space, without seeking research assistance from traditional services.   it also marks the time we introduced a tiered reference model at the terrell desk. evidence that the tiered reference model functions as we envisioned can be seen in the decreasing difficulty of the questions answered at the reference desk, and the increased difficulty of those addressed from offices. we hypothesize that the increase of read value 3, 4, and 5 questions in the library offices is a result of the bifurcation of reference service.   staffing changes at the reference desk have also contributed to this transition. we’ve noted (see table 8) that more time is spent per question at the reference desk, which economically, is usually not optimal. however, with heavier reliance on student workers to field these questions, this has become less costly, because they do not earn as much per hour as a librarian.   staffing changes over the years have been justified, as they have allowed librarians more time to maintain libguides and address complex questions, and tes to field simpler ones. the data suggests that we continue the trend of using graduate students on the desk, and encourage librarians to provide more specialized assistance from their offices. by incorporating kern’s (2006) call for a holistic approach, and articulating a research question before beginning our analysis, the reference committee was better able to identify the data that address our research questions, and plan ahead.    conclusions   a holistic look at reference statistics means considering all modes of reference service delivery. kern (2006) recognized reference as a system of communication modes, which should be considered as a whole.   the authors have looked at a comprehensive set of data from reference transactions over multiple communication modes and staffing configurations to ask, essentially, “what is happening to reference?” an analysis of our data has demonstrated the growing significance of online transactions occurring in librarian offices, despite im chat reference numbers being low. the tiered-reference model has largely facilitated this change, allowing librarians more time to work in their offices creating online guides, and address more complex questions from patrons.   many questions remain unanswered. for example, we cannot say conclusively why reference transactions dropped so quickly during the study period, but we know that the trend is not unique to wsu. the data does not necessarily support any cause-and-effect hypothesis, but rather provides us a few snapshots of our reference services over three years.   there are other factors affecting reference services that are difficult to quantify, and are outside the realm of this paper: for example, library instruction sessions, changes in course assignments, and changing student demographics and skill sets.   we will continue tier-modeled reference, with layered points of discovery, complexity and specialization. the increased use of email, libanswers, and libguides suggests a developing library user who is very comfortable engaging and interacting with multiple sites within multiple tiers to discover information. it indicates that the university community is comfortable finding their information from the libraries, but values it being at their fingertips.   references   baro, e. e., efe, b. u., & oyeniran, g. k. (2014). reference inquiries received through different channels: the challenges reference librarians face in university libraries in nigeria. reference services review, 42(3), 514-529. http://dx.doi.org/10.1108/rsr-09-2013-0049   coffman, s. (2012). the decline and fall of the library empire. searcher, 20(3). retrieved from http://www.infotoday.com/searcher/apr12/coffman--the-decline-and-fall-of-the-library-empire.shtml   deineh, s., middlemas, j., & morrison, p. (2011). a new service model for the reference desk: the student research center. library philosophy and practice (e-journal), (554). retrieved from http://digitalcommons.unl.edu/libphilprac/554   dinkins, d., & ryan, s. m. (2010). measuring referrals: the use of paraprofessionals at the reference desk. the journal of academic librarianship, 36(4), 279-286. http://dx.doi.org/10.1016/j.acalib.2010.05.001   dworak, e. (2011).  libanalytics. charleston advisor, 13(2), 41-44. http://dx.doi.org/10.5260/chara.13.3.41   flatley, r., & jensen, r. b. (2012). implementation and use of the reference analytics module of libanswers. journal of electronic resources librarianship, 24(4), 310-315. http://dx.doi.org/10.1080/1941126x.2012.732838   gerlich, b. k., & berard, l. g. (2007). introducing the read scale: qualitative statistics for academic reference services. georgia library quarterly, 43(4), 7-13. retrieved from http://digitalcommons.kennesaw.edu/glq/vol43/iss4/4   gossett, j. g., stephan, e., & marrall, r. (2012). implementing reference statistics collection software at multiple library service points. new library world, 113(5/6), 235-248. http://dx.doi.org/10.1108/03074801211226328   huling, n. (2002). reference services and information access. in schement, j. r. (ed.), encyclopedia of communication and information (2nd ed.) (pp. 867-874). new york: macmillan reference usa.   kern, k. m. (2006). looking at the bigger picture: an integrated approach to evaluation of chat reference services. the reference librarian, 46(95-96), 99-112. http://dx.doi.org/10.1300/j120v46n95_07   king, v., & christensen-lee, s. (2014). full-time reference with part-time librarians. reference & user services quarterly, 54(1), 34-43. http://dx.doi.org/10.5860/rusq.54n1.34   lederer, n., & feldmann. l. m. (2012). interactions: a study of office reference statistics. evidence based library and information practice, 7(2), 5-19. retrieved from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/12282   martin, p. n. (2009). societal transformation and reference services in the academic library: theoretical foundations for re-envisioning reference. library philosophy and practice, (260). retrieved from http://www.webpages.uidaho.edu/~mbolin/pamelamartin.htm   meserve, h. c., belanger, s. e., bowlby, j., & rosenblum, l. (2009). developing a model for reference research statistics. reference & user services quarterly, 48(3), 247–258. http://dx.doi.org/10.5860/rusq.48n3.247      tyckoson, d. a. (2012). issues and trends in the management of reference services: a historical perspective. journal of library administration, 52(6/7), 581–600. http://dx.doi.org/10.1080/01930826.2012.707953     evidence summary   canadian healthcare practitioners’ access to evidence based information is inequitable   a review of: chatterley, t., storie, d., chambers, t., buckingham, j., shiri, a., & dorgan, m. (2012). health information support provided by professional associations in canada. health information & libraries journal, 29(3), 233-241. doi: 10.1111/j.1471-1842.2012.00993.x   reviewed by: maria melssen medical librarian, independent contractor port clinton, ohio, united states of america email: mariamelssen@gmail.com   received: 28 feb. 2013   accepted: 25 may 2013      2013 melssen. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine what services and resources are available to health professionals through national canadian and alberta based health professional associations and licensing colleges and if those resources and services are being used. also, to assess the associations’ perceptions of what resources and services canadian health professionals actually need and if those needs are being met, membership satisfaction with the resources and services provided, and challenges the associations have with providing resources and services.   design – structured telephone interview.   setting – health professional associations and licensing colleges in canada.   subjects – 23 health professional associations: 9 alberta-based associations and 14 national-level professional associations and licensing colleges.   methods – a librarian, communications officer, or another individual in a comparable position at each association was invited via email to participate in the study. individuals willing to participate in the interview were emailed the interview questions in advance. telephone interviews were conducted in july and august of 2009. for those who did not respond to the email request or who did not wish to participate in the interviews, information was collected from the association’s website.   main results – of the 23 contacted associations 12 agreed to be interviewed: less than 50% response rate. data was collected from websites of seven associations that either declined to be interviewed or did not respond to the authors’ email request. data were unavailable for four associations due to data being in members only sections of the websites. data were analyzed both qualitatively and quantitatively.     resources and services provided by the associations and licensing colleges range from none to reference services provided by a librarian and access to licensed databases.     none of the three licensing colleges or the two provincial associations interviewed maintains usage statistics or surveys their members. nor do they grant access to licensed databases or offer information services, such as having a librarian or other information professional available to answer reference questions or to perform mediated literature searches. the two provincial associations and the three licensing colleges interviewed do supply information pertinent to health professionals, for example insurance information and funding.     seven national associations were interviewed: two permit access to databases developed by that association and three grant access to licensed databases such as medline. all seven national associations provide access to journals (four of the seven only provide access to their own association’s journal) and five offer information services. four maintain usage statistics and five survey their members.    of the seven associations not interviewed, none grant access to licensed databases and one permits access to databases developed by that association. five provide access to their own association’s journal and one provides book loans. only one offers information services. cost and the priority to provide resources to staff over members are barriers when trying to provide association members’ services and resources.    conclusion – health professionals’ access to health information varies depending on the professional’s area of specialization, location in canada, and particular association memberships. there is no consistency as to what health information is available to all health professionals in canada, specifically alberta. the majority of the associations do not provide resources and services, nor do they survey members to assess their usage, desires, needs, or satisfaction with resources and services. usage rates are low for the associations that do track resource and service usage.   a resource list of freely available online health information should be generated to mitigate existing disparities without accruing additional cost factors. also, a partnership between hospital and academic libraries with various associations is needed to promote the usage of licensed and freely available resources accessible at institutions.   this study has several limitations. the low response rate and excluding associations and licensing colleges in other provinces make this an incomplete assessment of all associations which provide resources and services to health professionals in canada, specifically alberta. to compensate for this deficit, the authors had collected information from seven associations’ websites; however, because much of the needed information was within members-only pages, some data may be missing. due to the study’s limitations, further research is needed to better assess health professionals’ information needs and barriers to their use of available resources and services.     commentary   the impetus for this study was the struggle health sciences alumni at the university of alberta face when trying to access evidence based information once they are no longer affiliated with the university. in this study, the authors were able to determine which health professional associations in canada could meet their alumni’s needs as well as those of other health professionals throughout canada. the findings support the current literature and underscore the significant disparity between accessible resources for canadian healthcare professionals.    critical appraisal of this study was completed using the evidence based library and information practice critical appraisal checklist (glynn, 2006). the study’s validity was analyzed in four content areas: population, data collection, study design, and results. the data collection methods, study design, and results are valid; however, the population selection validity is questionable.   the selection of study participants is problematic due to a small sample size. though the associations selected for the interviews do meet the authors’ participant selection criteria, this group may not include all of the potential study participants. according to the canadian information centre for international credentials (2013), there are over 30 health professional associations in alberta and over 70 national health professional associations in canada. the authors contacted only 14 national associations and 9 provincial associations. considering the number of associations available, the sample size is very small. the data collected could have been richer if more associations were interviewed.   another issue is the data from several associations is missing. contact was made 23 associations and data was obtained from 19 associations, thereby lacking data from 4 associations. the authors do acknowledge that this is due to data being in members only sections of the websites.   the data collection method is also a concern. there is a potential for intra-observer bias because multiple individuals where responsible for collecting the data. also, one interview was conducted via email rather than telephone. this variance in data collection could yield different responses from the phone interview data.     finally, presenting specific numbers opposed to saying “many” of university of alberta alumni continue to practice in alberta would have provided better support for the researchers’ decision to focus on alberta-based associations.     despite these issues, the study design is clearly outlined and appropriate for the authors’ established objectives. the authors include the interview questions in the article which would allow another researcher to replicate this study as secondary verification. inclusion and exclusion criteria were clearly outlined, and although there is no indication that the study design was validated, ethics approval was obtained. regarding the data itself, the response rate for the sample size contacted was appropriate, data collection is clearly defined, and the timing of data collection is appropriate. moreover, the study’s results were clearly explained and could be applied at similar institutions.   this study stresses the need to address limited access to evidence based information for health professionals in canada. the ability to quickly and efficiently locate reliable evidence based health information is critical for health professionals to provide optimal patient care.  librarians can play a crucial role in facilitating this information need. the opportunities for librarians to work closely with professional associations identified by the authors could be employed to bridge the evidence based health information access gap.      references   canadian information centre for international credentials (2013). national professional organizations and provincial and territorial professional organizations. retrieved 31 may 2013 from http://www.cicic.ca/en/profess.aspx?sortcode=2.19.21.21#alberta   glynn, l. (2006). a critical appraisal tool for library and information research.  library hi tech, 24 (3), 387-399. evidence summary   selecting which databases to teach students in communication disorders by considering database pairs that index core journals in the field   a review of: grabowsky, a. (2015). library instruction in communication disorders: which databases should be prioritized? issues in science and technology librarianship (79). http://dx.doi.org/10.5062/f4707zfb   reviewed by: giovanna badia liaison librarian, schulich library of science & engineering mcgill university montreal, quebec, canada email: giovanna.badia@mcgill.ca   received: 3 jun. 2015     accepted: 23 jul. 2015      2015 badia. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – there are two objectives in this research article. the first is to identify databases that librarians usually recommend to students for searching topics in communication disorders. the second is to compare these databases’ indexing of core journals in communication disorders, with the purpose of ascertaining which databases should be taught first in a one-shot information literacy session.   design – a comparative database evaluation using citation analysis.   setting – 10 universities in the united states of america offering libguides for their audiology or speech language pathology programs.   subjects – six databases: cinahl, eric, linguistics and language behavior abstracts (llba), psycinfo, pubmed/medline, and web of science/web of knowledge.   methods – the author selected 10 universities from the top 20 included in the graduate school rankings for audiology and/or speech language pathology from u.s. news & world report. the 10 universities selected were chosen because their librarians published online subject guides using libguides that suggest databases students can use for searching topics in communication disorders. the libguides were then examined to identify the most popular recommended databases that the author subsequently used for comparing coverage of core journals in communication disorders. the author generated a core journals list by selecting the top 20 audiology and speech-language pathology journals from journal citation reports, scimago journal & country rank, and google scholar top publications. these three sources produced lists of influential journals in different subject areas by looking at the number of citations the journals have received, alongside other factors. the author searched for 33 journals in total in each of the subject databases previously identified.   main results – the author found six databases that were mentioned in the libguides of at least half the universities investigated. none of the 6 databases indexed all 33 core journals. the breakdown of the number of journals indexed in each database is as follows: web of science/web of knowledge indexed 32 of 33 core journals (97%); pubmed/medline indexed 28 (85%); psycinfo indexed 27 (82%); cinahl indexed 25 (76%); llba indexed 23 (70%); and eric indexed 9 journals (27%).   conclusion – the author discovered that pairing certain databases allows for coverage of all 33 core journals. these pairings are: pubmed/medline with psycinfo, pubmed/medline with llba, pubmed/medline with web of science, web of science with psycinfo, and web of science with llba. the author suggests that librarians can create instructional materials for all recommended databases, “but use information from this study together with institution-specific factors to decide which databases to prioritize in face-to-face instruction sessions for speech-language pathology and audiology students” (conclusion).   commentary   other studies have employed libguides for identifying recommended resources, or have used core journal lists for comparing database coverage. this study is unique in that it is the only one thus far that combines both methodologies to compare database coverage of the literature in communication disorders for the purpose of helping librarians decide which databases to teach first in information literacy workshops.   the article scores well for the applicable data collection and study design questions in glynn’s (2006) ebl critical appraisal checklist. the author provides a clear description of the steps used to conduct the research, cites other studies to support the appropriateness of the methodology used, and presents the results succinctly, making it easy for readers to understand and replicate the study.   further directions for research are not included in the article, and this omission draws more attention to the limitations of the study. these limitations are the creation of a core journals list that concentrates on communication disorders in general, thus deemphasizing subspecialties of the field and excluding journals from related disciplines such as psychology and child development. another limitation is the inclusion of database suggestions from only those top ranked universities that use libguides (10 of the 20 top ranked schools). according to the author, “it would also be possible to examine library web sites of all top ranked schools to determine databases recommended by any means and that method could produce different results” (limitations). similarly, all of the journals from the lists created by black (2012) and slater (1997) in communication disorders and speech language pathology, both of which the author cites, could have been used to create the core journals list the author utilized for database comparison, so that the sample list would have been more representative of the field of communication disorders. the number of journals checked across databases would also affect the results.   furthermore, this reviewer has questions about the criteria the author used to determine whether a journal was indexed in a database or not. the author states: “indexing had to be current in order for the journal to be included” (results). did the author also take into account the database’s depth of coverage for the indexing of a particular journal title, specifically whether the database provides full or partial indexing of a journal and how far back the indexing goes for that journal? this information is not included in the article, and combined with the study’s limitations, makes it difficult for readers to completely trust the results.   the easily replicable methodology presented in this article might be of interest to librarians trying to find an evidence based approach to help them decide which database(s) to teach in a one-shot information literacy workshop for a specific discipline. readers should keep in mind that the core journals list used for comparing databases is the primary data source in this type of study and will directly affect the results.   references    black, s. (2012). how much do core journals change over a decade? library resources and technical services, 56(2), 80-93. https://journals.ala.org/lrts/index   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154     slater, l. g. (1997). mapping the literature of speech-language pathology. bulletin of the medical library association 85(3), 297-302. retrieved from http://www.ncbi.nlm.nih.gov/pmc/journals/72/ evidence based library and information practice editorial responsibilities      2014. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     editor-in-chief: alison brettle   associate editor (evidence summaries): heather pretty   associate editor (articles): lorie kloda   associate editor (reviews): denise koufogiannakis   associate editor (classics): jon eldredge   associate editor (commentaries, using evidence in practice, news): alison brettle   production editor: michelle dunaway   editorial intern: archana deshmukh   copyeditors: richard hayman (lead copyeditor), georgianne bordner, molly des jardin, linda ferguson, richard hayman, stacey penney, maria tan, mary virginia taylor, alison yeoman   indexing support: meral alakus   evidence summary   increasing the authenticity of group assignments in an online research course may lead to higher academic achievement   a review of: finch, j. l., & jefferson, r. n. (2013). designing authentic learning tasks for online library instruction. journal of academic librarianship, 39(2), 181-188. http://dx.doi.org/10.1016/j.acalib.2012.10.005   reviewed by: cari merkley associate professor mount royal university library calgary, alberta, canada email: cmerkley@mtroyal.ca   received: 1 mar. 2015    accepted: 7 may 2015      2015 merkley. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to explore what impact assigning authentic tasks to students deliberately grouped by their majors in an online library research course has on student perceptions of teaching quality (teaching presence) and satisfaction.   design – empirical comparative study.   setting – medium-size (10,500 full-time students) liberal arts college in the united states of america.   subjects – 33 undergraduate students enrolled in a library research course.   methods – the study focusses on two sections of a one-credit online library research course taught by library faculty. the 17 students in the spring “express” section were randomly assigned to groups and asked to complete a group annotated bibliography project using mla style (class random). the 16 students registered in the summer section of the same course were grouped by their majors, and asked to complete a modified version of the annotated bibliography group project in which they were asked to identify and then utilize the citation style most appropriate for their discipline (class deliberate). students in class deliberate also received instruction around the role of subject specific citation styles in scholarly communication. both sections completed a final assignment in which they developed a portal of resources to support their future studies or careers. all 33 students in both sections were invited to complete a modified online version of the community of inquiry (coi) survey consisting of 16 questions relating to student perceptions of the course’s teaching and cognitive presences. questions relating to social presence were not administered. the final grades awarded to all students in both sections were also analyzed.   main results – a total of 59% of the students in class random (10/17) and 67% of the students in class deliberate (11/16) completed the online survey. there were no statistically significant differences in the survey responses between the two sections with both groups of students rating the instructor’s teaching presence and the course’s cognitive presence highly. only 40% of the respondents from class random and 46% from class deliberate agreed that working with peers facilitated their learning. the mean final grade received by students in class deliberate was 95.27 versus 86.15 in class random, a statistically significant difference (p<0.10).   conclusion – assigning authentic tasks has a positive impact on academic achievement, but differences in course timing and the structure and the higher number of seniors enrolled in class deliberate may partly account for the differences in the mean grade. the coi theoretical framework is useful for understanding the complex multidisciplinary nature of information literacy instruction, particularly in an online environment. areas for future research include the role of social presence and its relationship to the age of participants in online library instruction.   commentary   this study illustrates the challenges faced by those involved in the scholarship of teaching and learning (sotl). in their discussion of the “fallacies of sotl,” grauerholz and main (2013) write, “there is no such thing as a control group when it comes to research on a classroom” (p. 155). a classroom is not a petri dish in which exposures can be easily isolated and controlled. the unpredictable mix of student personalities and levels of experience in each course is likely to have an impact on the success (or failure) of any intervention (grauerholz & main, 2013).   finch and jefferson acknowledge many of the confounding factors that may have influenced levels of student success and satisfaction, many of which were beyond their ability to control without jeopardizing the student experience. however, some decisions regarding study design undermine the study’s conclusions. for example, the decision to compare two different sections of students that took place at different points in the year diminishes the study’s internal validity (bartsch, 2013). more detail on the length of each term (i.e., what made the spring section “express”) and a clear statement that both sections were taught by the same instructor would help readers determine how comparable the two groups truly were. the use of the final course grade rather than the specific grade awarded for the assessment also makes it more difficult to link student performance to the increased authenticity of the assignment.   the use of the previously validated coi survey allows for easier comparison of the findings to other studies. the decision to remove the questions on social presence in the version administered to students is surprising, particularly when the intervention under study involved group work. only 40% of the students in class random and 46% in class deliberate felt that working with their peers helped their learning on the one question included on group work. this is such a departure from the otherwise positive ratings of the course that it raises questions about what data the questions on social presence might have elicited if they had been included. that less than half of the students saw value in working with their peers may also point to a need to revisit the group assignment design or how it is framed within the course.   finch and jefferson’s overall openness around the limitations of their study design and results are helpful for readers looking to determine the applicability of the results to their own teaching contexts or considering their own research into teaching and learning. their discussion of the coi framework and how the online research course was designed to highlight teaching presence may be of interest to those involved in the delivery of online information literacy instruction.   references   bartsch, r. a. (2013). designing sotl studies—part i: validity. new directions for teaching and learning, 2013(136), 17-33. http://dx.doi.org/10.1002/tl.20073   grauerholz, l., & main, e. (2013). fallacies of sotl: rethinking how we conduct our research. in k. mckinney (ed.), the scholarship of teaching and learning in and across disciplines (pp. 152-168). bloomington: indiana university press. evidence summary   various factors may influence high school student use of public libraries   a review of: sin, s.-c. j. (2012). modeling the impact of individuals’ characteristics and library service levels on high school students’ public library usage: a national analysis. library & information science research, 34(3), 228–237. doi:10.1016/j.lisr.2012.01.002   reviewed by: robin e. miller assistant professor and research & instruction librarian mcintyre library university of wisconsin-eau claire eau claire, wisconsin, united states of america email: millerob@uwec.edu   received: 28 may 2013  accepted: 2 aug. 2013      2013 miller. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to discover the factors that influence frequency of high school students’ usage of public libraries.   design – structural equation modeling (sem) using the person-in-environment (pie) framework to test latent variables and direct and indirect relationships between variables.    setting – public and school libraries in the united states.   subjects – three datasets: educational longitudinal study of 2002, the national center for education statistics (nces), provides data about individual students; public libraries survey of 2004, then conducted by nces, provides data about public libraries in the united states; and summary files 1 and 3 of u.s. census 2000, provide neighborhood-level demographic data.   methods – using arcgis, the researcher prepared and linked three datasets. data were analyzed using factor analysis, regression, weighted least squares, and path analysis in order to test relationships between variables exposed in three large datasets.   main results – frequency of public library use by high school students may be influenced by several factors, including race and/or ethnicity and access to resources like school libraries, home computers, and public libraries with adequate service levels.   conclusion – increased funding for public library spaces and resources may be warranted by the finding that high levels of public library service may increase high school students’ use of public libraries, particularly in socioeconomically disadvantaged neighborhoods.     commentary   in an effort to understand barriers to public library use among high school students, sin set out to explore how teen information behaviour is influenced by personal characteristics and by characteristics of their schools and public libraries.   this research analyzes three large nationally representative datasets: the education longitudinal study of 2002; summary files 1 and 3 of census 2000; and the 2004 public libraries survey. the author acknowledges that the conclusions cannot be generalized to all public library users. sin used these datasets to test variables derived from the “person-in-environment” (pie) framework. pie is a conceptual and methodological framework for information behaviour (ib) research that sin introduced in a previous publication (2011). describing the development of the pie framework in greater detail would have given the author the ability to compare pie to other conceptual frameworks used in ib research, potentially strengthening the credibility of the relatively-unknown pie framework for this research.   data were prepared and linked using arcgis, and structural equation modeling (sem) was utilized to test the influence of many variables on three library use outcomes: school work, leisure, and internet access. an extensive description of the conceptual model and data analysis procedures bolster the validity of this unique and complex research. unfortunately, the lengthy procedural narrative leaves less time for thorough discussion of the findings.   in linking outcomes to community-level characteristics, including access to information resources in schools and ready access to public libraries, the author seeks to demonstrate the link between information behaviour and library resources. the author reports descriptive data to indicate that high school students use school libraries more frequently than public libraries, though the 67% who reported using public libraries were more likely to do so for school work than for leisure or internet access. in modeling latent and single indicator variables, the author reveals positive indicators for use of the public library, including limited information resources at school, race or ethnicity, and to a lesser extent, public library “environment.” negative indicators include high use of a school library and access to a computer and internet at home.   the author concludes that greater “service levels” at public libraries encourage high school students to use public libraries more. indeed, the structural model shows a statistically significant correlation between frequency of use and two variables: public library environment and public library accessibility. left to assume that these variables are defined by data drawn from the public libraries survey, one must also wonder which budgetary, programmatic, and personnel data informs “public library environment.” this omission leaves librarians, administrators, and policy makers without enough information to act on the findings. librarians, administrators, teachers, and public funding agencies may be intrigued by the author’s findings that school information environment and race/ethnicity influence public library use frequency. these findings might inspire new partnerships between school and public librarians, new programming and collection development in public libraries, cultural competency training for public librarians and staff, or other initiatives.   as exploratory research, sin makes a compelling case that large nationally representative datasets can be used to model some aspects of information behaviour. the author’s unique approach merits further examination and application in the area of information behaviour research, and the wide-ranging findings may inspire deeper investigation of specific aspects of teen information behaviour.     references   sin, s.-c. j. (2011). towards agency-structure integration: a person-in-environment (pie) framework for modelling individual-level information behaviours and outcomes. in a. spink, & j. heinström (eds.), new directions in information behaviour (library and information science, volume 1) (pp. 181-209). bingley, england: emerald group publishing. doi:10.1108/s1876-0562(2011)002011a011   evidence based library and information practice editorial responsibilities      2013. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.       editor-in-chief: alison brettle   associate editor (evidence summaries): heather pretty   associate editor (articles): lorie kloda; derek rodriguez   associate editor (reviews): denise koufogiannakis   associate editor (commentaries, using evidence in practice, news): alison brettle   production editor: michelle dunaway   editorial intern: archana deshmukh   copyeditors: richard hayman (lead copyeditor), georgianne bordner, molly des jardin, linda ferguson, richard hayman, stacey penney, maria tan, mary virginia taylor, alison yeoman   indexing support: pam morgan   article   the contributions of medline, other bibliographic databases and various search techniques to nice public health guidance   paul levay information specialist national institute for health and care excellence manchester, united kingdom email: paul.levay@nice.org.uk   michael raynor information specialist national institute for health and care excellence manchester, united kingdom email: michael.raynor@nice.org.uk   daniel tuvey information specialist national institute for health and care excellence london, united kingdom email: daniel.tuvey@nice.org.uk   received: 22 oct. 2014    accepted: 8 feb. 2015           2015 levay, raynor, and tuvey. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – to make recommendations for the national institute for health and care excellence (nice) on the factors to consider when choosing databases and search techniques when producing systematic reviews to support public health guidance development.   methods – retrospective analysis of how the publications included in systematic reviews commissioned by nice on obesity, spatial planning, and tuberculosis were retrieved. the   included publications were checked to see if they were found from searching medline, another database or through other search techniques.   results – medline contributed 24.2% of the publications included in the obesity review, none of the publications in the spatial planning review and 72% of those in the tuberculosis review. other databases accounted for 9.1% of included publications in obesity, 20% in spatial planning and 4% in tuberculosis. non-database methods provided 42.4% of the included publications in the obesity review, compared to 5% in the spatial planning review and 24% in the tuberculosis review. it was not possible to establish retrospectively how 24.2% of the publications in the obesity review and 75% in the spatial planning review were found.   conclusions – topic-specific databases and non-database search techniques were useful for tailoring the resources to the review questions. the value of medline in these reviews was affected by the degree of overlap with clinical topics, the domain of public health, and the need to find grey literature. the factors that nice considers when planning a systematic search are the multidisciplinary nature of public health and the different types of evidence required. introduction   a systematic review is designed to reduce bias, synthesize the available evidence, and answer a specific research question (higgins & green, 2011). it is essential that a systematic review begins with a systematic search, in order that the evidence to be appraised is gathered in an unbiased, transparent, and robust manner. the purpose of this study was to explore the challenges that the national institute for health and care excellence (nice) has experienced when applying the principles of systematic searching to public health topics. the purpose of public health is “promoting and protecting health and well-being, preventing ill health and prolonging life through the organised efforts of society” (faculty of public health, 2010). the definition encapsulates three key domains in public health: health improvement, improving services, and health protection.   methods for nice public health guidance   nice is an independent public body responsible for producing public health guidance to promote good health and prevent ill health in england. nice has produced guidance across all three domains of public health, including behaviour change interventions (health improvement), smoking cessation services (improving services), and preventing infectious diseases (health protection). the guidance is developed following rigorous and transparent methods, as set out in a manual that incorporates public health, social care, safe staffing, medicines practice, and clinical guidelines (nice, 2014). once the government has referred a topic to nice, the process starts with a scope document that outlines the key questions and the populations, settings, and interventions that will, and will not, be covered by the guidance. the next stage is for the evidence to be gathered and appraised in a systematic review. evidence statements are prepared to demonstrate the quantity, strength, appropriateness, and applicability of the findings. the evidence is then presented to an independent committee of experts who use it to inform a series of recommendations. the draft recommendations are subject to public consultation with stakeholders before final publication and implementation.   searching is an integral part of the process as nice is explicit that its recommendations are made “using the best available evidence” (nice, 2014, p. 5). the chapter on identifying the evidence (nice, 2014, chapter 5) describes the main stages in the process, which starts with a series of exploratory searches to support writing of the scope. once the scope has been finalized, a protocol is developed setting out the parameters of the search and a draft strategy for the principal database. the search strategy is subjected to a rigorous quality assurance check involving information specialists and the technical team. the agreed search strategy is run across the sources and the results are gathered in reference manager or similar bibliographic management software. the manual has clear requirements on recording searches to ensure that the methods are transparent to the committee, stakeholders, and the wider public (nice, 2014, section 5.9).   at the time of this study, all of the systematic reviews were written independently of nice by external review teams who were commissioned to run the searches, extract the data, and write the report. the review teams commissioned by nice have included universities, research institutes, and business consultancies. a nice team including information specialists, technical analysts, and health economists is responsible for quality assuring the review team’s protocols, searches and reports.   it is important for nice to develop its own methods as the purpose is to produce guidance for england and not just to write a systematic review for academic purposes. nice has an ongoing commitment to keeping its methods up to date and the manuals have undergone several revisions to ensure they follow current best practice. the periodic process of updating the manuals draws on research conducted by nice itself, published papers, and the recommendations from other relevant organisations, such as the campbell collaboration and the evidence for policy & practice information and co-ordinating (eppi) centre. the methods are aligned to, and draw on, the recommendations made by other organizations. for example the current manual refers to the cochrane collaboration, the intertasc information specialists' sub-group, the canadian agency for drugs and technologies in health (cadth), and the centre for reviews and dissemination (nice, 2014, sections 5.3, 5.4, 5.7, 5.11).   nice has been considering for several years how systematic review methods originally developed in the medical field can be used to review evidence in public health (kelly et al., 2010). the current study extends a previous case study on cardiovascular disease (bayliss, davenport, & pennant, 2014) to a wider set of topics.   literature review   all systematic searches need to target the right sources of information to find the right types of evidence. authors of cochrane systematic reviews are advised that using medline alone does not constitute a systematic search, as it may retrieve biased or unrepresentative sets of results (lefebvre, manheimer, & glanville, 2011, section 6.1.1.2). the question for the searcher is how to balance the need to use a range of resources beyond medline with the need to complete the review on time and to budget. there are several issues that make these challenges more pronounced in public health searching, as opposed to, say, finding evidence on the efficacy of a pharmaceutical product (alpi, 2005). these issues are explored further in relation to two main themes, showing that public health reviews require a variety of databases covering a multidisciplinary evidence base, and a range of search techniques to locate different types of evidence.   multidisciplinary evidence base   medical sources should not be overlooked when starting to review a public health topic, even if evidence from other disciplines will need to be considered. this means that medline can be an important source of evidence for public health. a case study on multi-factor interventions to prevent cardiovascular disease at a population level (bayliss et al., 2014) found that medline contributed 91% of the programmes. most of the programmes were identified through medline, although it was necessary to search for clinical trials (cochrane central register of controlled trials, central), the social sciences (applied social sciences index and abstracts, assia), and psychology (psycinfo) to retrieve all of the evidence. the value of medline was boosted when efforts were made to improve its public health coverage, such as adding 10 main headings to medical subject headings (mesh) in 2003 (whitener, van horne, & gauthier, 2005). it is also feasible to draw on the health services research indexed on medline to inform the service delivery elements of public health guidance, even though this can be challenging (wilczynski, haynes, lavis, ramkissoosingh, & arnold-oatley, 2004).   it is instructive to contrast public health topics with clinical topics. a study of clinical guidelines found that medline accounted for 94.72% of references in head and neck cancer, 88.94% in hepatitis c, 88.18% in bronchiolitis, 82.68% in cervical cancer, and 78.02% in autism (kelly, 2008). medline made a similarly high contribution to a systematic review of diabetes epidemiology, accounting for 94% of references (royle, bain, & waugh, 2005). combining medline with several other medical databases achieved 90% recall in breast cancer (lemeshow, blum, berlin, stoto, & colditz, 2005), 100% in nursing studies (subirana, sola, garcia, gich, & urrutia, 2005), and 97% in orthopaedics (slobogean, verma, giustini, slobogean, & mulpuri, 2009). booth (2010) has suggested that medline consistently delivers up to 80% of the relevant references in health technology assessments. the importance of medline is clear in these reviews.   by contrast, it is not possible to limit searches for public health topics to a defined and regular set of sources in the way that might be possible for clinical topics. it would be unfair to characterize clinical topics as requiring the same resources every time a review is conducted (beyer & wright, 2013; crumley, wiebe, cramer, klassen, & hartling, 2005). the difference with literature searches in public health is the breadth of potential sources that might need to be captured, as it draws its evidence from a number of disciplines, including psychology, education, sociology, housing, transport, and architecture (nice, 2012b). alpi (2005) has shown that a wider range of databases is needed to ensure each of these disciplines is covered adequately. nice public health guidance has covered topics as diverse as domestic violence (requiring access to criminal justice sources), promoting cycling schemes (transport), body mass index thresholds (epidemiology), looked after children (social care), workplace health (business), and behaviour change (psychology).   the issue for the searcher is that the evidence from other disciplines is likely to be indexed on specialized databases, which may increase the time and costs required for the review (alpi, 2005). specialized databases were valuable to a review on exercise therapy, where they were used alongside medline and other techniques (stevinson & lawlor, 2004). a review of interventions on encouraging walking and cycling concluded that 4 of the 69 included publications came from “first line health databases”, compared to 8 from social science sources and 33 from a topic-specific transport database (ogilvie, hamilton, egan, & petticrew, 2005). further evidence on the value of searching beyond medline is available on a number of topics relevant to public health, including: occupational health (rollin, darmoni, caillard, & gehanno, 2010), social welfare (taylor, wylie, dempster, & donnelly, 2007), maternal health (betran, say, gulmezoglu, allen, & hampson, 2005), mental health (lohonen, isohanni, nieminen, & miettunen, 2010), mental illness (brettle & long, 2001), psychiatry (mcdonald, taylor, & adams, 1999), and injury prevention (lawrence, 2008).   range of search techniques   it would not be appropriate to focus a public health review on retrieving evidence from randomized controlled trials (kelly et al., 2010). it would be difficult to blind patients to some public health interventions, such as exercise therapy. interventions can be organized at a population level (such as taxation) and some trials might be unethical in areas where interventions are known to be beneficial (e.g., smoking cessation). this means that nice public health guidance does not just assess whether an intervention works, it also addresses “when, why, how, and for whom an approach does (and does not) work” (nice, 2012b, p. 17). these types of questions are often answered by grey literature, such as reports, case studies, theses, surveys, audits, and other guidelines, which can be difficult to locate and retrieve (benzies, premji, hayden, & serrett et al., 2006; turner, liddy, bradley, & wheatley, 2005). grey literature has previously been found to be a “productive way” of identifying further studies for a different nice programme that appraises drugs and other health technologies (royle & waugh, 2003). the issue in public health is that this type of contextual evidence is key to understanding the interventions and whether it would be appropriate for nice to recommend them, rather than it just being useful supplementary information to what can be retrieved from databases.   the emphasis on grey literature means that non-database searching methods, such as citation searching, website searching, pearl growing, and contacting experts are an essential component of comprehensive searches in public health (papaioannou, sutton, carroll, booth & wong, 2010). a search that is too focused on trawling a standard list of databases, covering only peer-reviewed journal articles, is unlikely to leave sufficient time to use additional methods for locating grey literature (booth, 2010).   a comparison of 12 cochrane systematic reviews in public health found that 9 of them cited evidence that could not be retrieved in 5 large health databases and had benefited from supplementary methods (morgan, bauschmann, & weightman, 2011). this particular study helps to confirm that the recommendation in the nice manual to search for grey literature is consistent with the findings of the cochrane public health group. stansfield, brunton and rees (2013) examined qualitative reviews on transport, motherhood, and obesity, finding that around a third of the 229 studies would have been missed if only databases had been used, no matter how many were searched. similar figures were found in: a review deriving 21% of the included publications from citation searching, websites, and hand-searching (stansfield, kavanagh, rees, gomersall, & thomas, 2012); a search on childhood obesity where 13% of reviews came from websites, library catalogues, and bibliographies (woodman et al., 2010); and a review on the built environment where a quarter of studies required specialized sources and grey literature (weaver et al., 2002).   the aim of bayliss et al. (2014) was to examine the suitability of different databases for searching on a public health topic and they did not consider other search techniques. the purpose of the current study was to examine a range of search techniques across a larger sample of subjects.   aims   the aim of the study was to make recommendations on the factors to consider when choosing databases and search techniques when producing systematic reviews to support public health guidance development at nice.   the objectives of the study were to:   ·         assess the value of bibliographic databases (particularly in relation to medline) for identifying the evidence for the guidance in an appropriate sample ·         evaluate the contribution of different search techniques for identifying the evidence for the guidance in an appropriate sample ·         make recommendations on the factors to consider when planning the systematic searches to support public health guidance development.   methods   personnel   three information specialists, each with at least two years’ experience of literature searching to support nice public health guidance development, carried out the sampling and analysis between february and october 2011. the information specialists involved in this study had not actually conducted the searches, although they had been involved in the procurement process to select the external review teams and had quality assured the protocols and strategies.   sampling of guidance   selection was carried out in february 2011 and guidance was eligible for inclusion if:   ·         the search was conducted in accordance with the then current edition of the nice methods manual (nice, 2009) ·         the review was completed and had been presented to an expert committee meeting on or before 1 february 2011.   the resources were available to analyze three reviews for this study. maximum variation sampling was chosen so that a full range of issues would be experienced with the three reviews. it was felt that analyzing three reviews from the same review team, taking the same approach or in the same domain of public health would limit the lessons learnt in this study. the value of choosing these three was to help in assessing the feasibility of scaling up the study to a larger sample of reviews.   seven guidance topics were eligible for inclusion. nice often commissions several systematic reviews on each topic and a total of 17 had been completed by the cut-off date for the 7 topics. the three purposively chosen reviews covered obesity (garside, pearson, hunt, moxham, & anderson, 2010), spatial planning (gray et al., 2010), and tuberculosis (o’mara et al., 2010). table 1 shows that they cover at least two different public health domains, both clinical and non-clinical topics, a range of search approaches, and were conducted by different types of review team.     table 1 attributes of reviews selected for study topic title type of review team public health domain clinical/non-clinical search approach obesity: working with local communities university health improvement non-clinical iterative spatial planning for health: local authorities and primary care trusts university health improvement non-clinical comprehensive identifying and managing tuberculosis among hard-to-reach groups business consultancy health protection clinical comprehensive     data analysis   the three reviews were analyzed to determine how the publications included in them had been retrieved. the unit of assessment in this study is the “included publication”, which refers to the publications cited in the evidence statements contained in the reviews. an evidence statement is an aggregated summary of all the relevant studies, regardless of their findings, reflecting the balance of the evidence, its strength, applicability, and any gaps (nice, 2014, p. 107). this means that the analysis was not concerned with citations contributing to the background context (such as epidemiology) or the methods.   this study obtained the list of sources used from the original search protocols. the included publications were extracted from the evidence statements for each of the three reviews identified in table 1 and the numbers were checked against the literature flow diagram in the systematic review. review teams usually supply nice with a database of search results in reference manager or similar bibliographic software as part of their contract to produce the systematic review. these databases should show how each publication has been located, whether it was included in the evidence statements, or why it was excluded. the reference manager files were checked to determine the format of each included publication, the method used to locate it, the database on which it was found, whether it was ordered in full text, and any annotations on the inclusion decision. any queries were referred to the review teams who had conducted the searches.   results   table 2 shows that 39.4% (13 of 33) of the included publications in the obesity review were journal articles, compared to 80% (16 of 20) in the spatial planning review, and 76% (19 of 25) in the tuberculosis review. the obesity review, with 42.4%, had the highest proportion of included publications classified as grey literature, compared with 20% in the spatial planning review, and 24% in the tuberculosis review.   table 3 shows that the obesity review had 33 included publications and 33.3% of these were from databases and 42.4% were from a variety of non-database techniques (24.2% from checking the reference lists of relevant publications, 15.2% from the call for evidence on the nice website, and 3% from website searching). the search records did not describe how eight (24.2%) of the publications in the obesity review had been located. database searches retrieved 76% of the 25 included publications in the tuberculosis review, while non-database methods retrieved 24% (16% from contact with experts and 8% from the national research register website).     table 2 format of included publications format obesity spatial planning tuberculosis book or book chapter 6 (18.2%) 0 (0) 0 (0) grey literature 14 (42.4%) 4 (20.0%) 6 (24.0%) journal article 13 (39.4%) 16 (80.0%) 19 (76.0%) total 33 (100%) 20 (100%) 25 (100%)     table 3 how the included publications were found search method obesity spatial planning tuberculosis call for evidence 5 (15.2%) 1 (5.0%) 0 (0) contact with experts 0 (0) 0 (0) 4 (16.0%) database (core)* 11 (33.3%) 0 (0) 19 (76.0%) database (topic specific) 0 (0) 4 (20.0%) 0 (0) national research register 0 (0) 2 (8.0%) reference list harvesting 8 (24.2%) 0 (0) website 1 (3.0%) 0 (0) unknown 8 (24.2%) 15 (75.0%) 0 (0) total 33 (100%) 20 (100%) 25 (100%) * core database is defined as those listed in the second edition of the nice methods manual (nice, 2009, appendix a).     table 3 shows that it was only possible to determine how 5 of the 20 publications included in the spatial planning review had been retrieved, with 4 from database searching and 1 from the call for evidence that was posted on the nice website at time the searches were being conducted. the sources were not recorded in reference manager and so the original 11 files downloaded from the bibliographic databases were obtained from the review team. table 5 shows that three of the included publications were found in the planex file and one in the urbadoc file; these are specialist databases on town and country planning. none of the included publications were found in the medline file. it is unlikely that the other 15 publications were found through database searching if they were not contained in these 11 files. it is impossible to replicate the non-database methods to see how the 15 were found; for example, citation searching would now retrieve a different set of results.   tables 4-6 show the contribution of each database in terms of the publications found through database searching and the total number of included publications. it was felt that only reporting how the journal articles had been found would hide the importance of grey literature and unfairly boost the contribution of medline. table 4 shows that medline contributed 72.7% of the publications found using databases and 24.2% of the total number of included publications in the obesity review. on the other hand, as table 6 shows, medline contributed 94.7% of the publications derived from databases and 72% of the total number in the tuberculosis review. medline did not contribute any publications to the spatial planning review, as shown in table 5. the purpose of this study was to determine the value of databases in relation to medline and the retrieval of the same publication from multiple sources was not considered. the resources were not available, and it would require another study, to determine how many of the databases index the included publications, whether the search strategies retrieved them, and the degree of database overlap.     table 4 databases used to locate included publications in the obesity review database no. of publications % of included publications found by database searching % of all included publications assia* 1 9.1% 3.0% bibliomap 0 0 0 cumulative index to nursing and allied health (cinahl)* 0 0 0 database of promoting health effectiveness reviews (dopher) 0 0 0 health management information consortium (hmic)* 2 18.2% 6.1% intute 0 0 0 medline* 8 72.7% 24.2% obesity and sedentary studies database 0 0 0 social science citation index 0 0 0 trials register of promoting health interventions (trophi) 0 0 0 total 11 100% 33.3%     discussion   multidisciplinary evidence base   tables 4-6 show that six different databases contributed unique publications to the three reviews (assia, cinahl, hmic, medline, planex and urbadoc). these databases demonstrate the multidisciplinary nature of the resources required, with coverage of the social sciences, nursing, health management, and urban planning, as well as medline. the manual used at the time of the three reviews (nice, 2009) included a core list of databases that was to be considered for all topics. the findings from this study suggest that enforcing a standard list of sources on all searches might be of limited value, as some topics will draw heavily on medline and others will need topic-specific sources. it might be legitimate to prioritize topic-specific resources in some reviews, for example the transport database might be more valuable than embase in a review of interventions to increase the uptake of cycling. the nice manual now encourages searchers to choose sources “depending on the subject of the review question” (nice, 2014, p. 78). choosing the right databases for the review question suggests that it would be useful to conduct scoping searches early in the process to map out where the evidence lies and to identify any major gaps.   the cardiovascular case study (bayliss et al., 2014) suggested that, as some public health topics overlap with clinical issues more than others, the “medical public health topics” might rely on medline more than “non-medical” ones. the results of this study seem to confirm this finding, as medline contributed 24.2% to the obesity review, none to the spatial planning review, and 72% to the tuberculosis review. the tuberculosis review was the only topic approaching the benchmark that “medline consistently delivers an average of 80% of included papers for systematic reviews” (booth, 2010, p. 432). it would be difficult to classify consistently public health topics as either “medical” or “non-medical” and so it would be worth exploring whether the domains of public health are a more fruitful framework for analysis.   the tuberculosis review dealt with an infectious disease, which tends towards the “medical” end of the public health scale but it is also a health protection topic. the tuberculosis review suggests health protection might be the domain of public health most likely to overlap with clinical topics and benefit from medline. this could be tested against the publications included in nice guidance on other health protection topics, such as hiv testing, hepatitis testing, and controlling hospital-acquired infections.     table 5 databases used to locate included publications in the spatial planning review database no. of publications % of included publications found by database searching % of all included publications cab abstracts 0 0 0 embase* 0 0 0 geobase 0 0 0 hmic* 0 0 0 international construction database (iconda) 0 0 0 medline* 0 0 0 planex 3 75.0% 15.0% pscyinfo* 0 0 0 social science citation index* 0 0 0 transport research information systems (tris) 0 0 0 urbadoc 1 25.0% 5.0% total 4 100% 20.0% * core databases     table 6 databases used to locate included publications in the tuberculosis review database no. of publications % of included publications found by database searching % of all included publications assia* 0 0 0 british library direct 0 0 0 british nursing index* 0 0 0 cinahl* 1 5.3% 4.0% cochrane database of systematic reviews (cdsr)* 0 0 0 community abstracts 0 0 0 current contents 0 0 0 embase* 0 0 0 education resources information center (eric) 0 0 0 hmic* 0 0 0 medline* 18 94.7% 72.0% psycinfo* 0 0 0 social policy and practice* 0 0 0 sociological abstracts 0 0 0 social services abstracts 0 0 0 web of science 0 0 0 total 19 100% 76.0% * core databases           the second highest ranked review, in terms of medline contribution, was obesity, which again reflects the type of topic being reviewed. obesity is itself a topic likely to be indexed in considerable depth on a medical database such as medline. the review was not, however, concerned with clinically treating obesity and it examined the value of local authorities adopting a “whole-systems approach”. it is difficult to define this topic as either “medical” or “non-medical” and it seems to fit more comfortably in the health improvement domain of public health. the public-health nature of this topic is reflected in the contribution of other databases, with two included publications coming from hmic and one from assia (table 4). the lack of specific evidence on the topic meant that the obesity angle was dropped from some searches and they were broadened out to explore how the concept of the whole-systems approach had been applied in other domains, such as ecology. this may go some way to explaining why one study was found on a social science source (assia), while databases that might be expected to be useful on the topic of obesity, such as cinahl or embase, did not yield any included publications.   range of search techniques   the results from these three reviews highlight the need to leave sufficient time to use a range of techniques to search for grey literature. table 2 shows that in the obesity review, 39.4% of the publications were journal articles, 42.4% were grey literature, and a further 18.2% were books or book chapters. the conclusions of this review would have been affected if the search had been restricted to the large medical databases that only index journal articles, such as medline and embase. this point is emphasized by the finding that a number of databases did not contribute any included publications to the three reviews and there might have been scope to focus the searches further. tables 4-6 show that no included publications were retrieved by 7 of the 10 databases used in the obesity review, 9 of the 11 in the spatial planning review, and 14 of the 16 in the tuberculosis review. it was not possible to know in advance which databases would not find any unique publications but this does suggest that a targeted approach may be as productive as searching a long list of sources.   the obesity searches progressed iteratively to explore the available literature one step at a time. the concept of a “whole-systems approach” had not been well developed, it did not have an agreed definition, and there was little consensus on how the terminology could be applied. a comprehensive search could not be planned at the outset of the project if the definition of the main concept was contested. the definition emerged from reviewing some of the evidence and this, in turn, led the search in new directions. this approach to searching builds on the “berrypicking model” defined by bates (1989), which shows how it is legitimate to build a series of results at each stage of an “ever-modifying search” instead of expecting to find a “single final retrieved set”.   table 4 shows that the medline searches contributed 24.2% of the 33 publications included in the obesity review but this overall number masks the iterative nature of the searching. medline searches were undertaken at several phases of the process and not as one large search strategy. the first phase involved scoping out the topic and medline was used alongside website searching and contact with experts. the relevant publications identified during the first phase formed a cluster of documents worthy of further investigation during the second phase. the first search identified several projects that had adopted a relevant approach to obesity and the second phase involved trying to find more evidence on these named interventions. three of the eight publications identified from medline were contained in the phase 1 searches and five were only found when searching medline in phase 2 for the project names found in phase 1. reference harvesting was an important method, contributing eight publications to the final review (table 3). however, the reference harvesting could only take place in phase 2 using relevant publications found by medline in phase 1. the numbers for each search technique in tables 3 and 4 mask the complexity of the obesity search, where the iterative approach meant the methods were dependent on each other and they should not be treated in isolation.   table 6 shows that medline contributed 94.7% of the included publications found through databases in the tuberculosis review and 72% of the total number. this suggests that the type of evidence required for the review could influence the usefulness of the search techniques. the review required qualitative evidence on the barriers and facilitators to increasing the uptake of tuberculosis testing. this qualitative evidence might be published in reports, audits, surveys, and other types of grey literature which would not be indexed on medline. a separate study has shown that designing a search filter on the hard-to-reach population of interest reduced the volume of medline hits to be screened by 64% and this was crucial to freeing time for more productive activities for exploring the grey literature (cooper, levay, lorenc, & craig, 2014). medline was important, as would be expected in the health protection domain, but this was balanced by the need to find qualitative evidence. the results from one review cannot be extrapolated across public health but it does suggest that targeting resources on grey literature could be particularly beneficial when searching for qualitative evidence.   limitations and further research   the tuberculosis guidance (nice, 2012a) and the obesity guidance (nice, 2012c) have been published since this study was completed, although the spatial planning guidance-development process was subsequently suspended. this change in status did not affect the literature searching that had already taken place, which was fully compliant with nice methods, and so spatial planning was still considered eligible for this study.   this study has built on bayliss et al. (2014) by extending it to a wider range of topics, although the small sample of reviews still limits the generalizability of the results. the data required had not been collected at the time of the searches and it was difficult to analyze the three reviews retrospectively. this study does, however, provide a framework for the analysis of a larger sample of reviews and highlights the data requirements to carry out such research successfully. the main practical issues encountered related to the removal of duplicate records, resource constraints, and recording of iterative searches.   the study was difficult to perform retrospectively because of the way that duplicate records had been processed, as bayliss et al. (2014) also experienced. the review teams provide nice with a reference manager file with the duplicates removed and so there was no way of knowing if each publication had actually been found on several databases. it would only be possible to know how many times a publication had been retrieved by re-running the exact search across all of the databases used by the review team. it would not be reasonable for nice to expect the review team to undertake the additional work required to record the source of the duplicates within current budgets for guidance production. there would be a further difficulty, given that the contractual relationship with nice has ended and any queries or requests for data would depend on the good will of the review team.   the duplicates issue did not affect the results in this study but it did show that it would be difficult to conduct a larger retrospective study. the risks were mitigated in this study by checking the original files downloaded from medline for the included publications, rather than just relying on the review team’s annotations in reference manager. this established how many publications medline retrieved, although there was not sufficient time to test how many were unique to this source or the overlap with other databases. the obesity review was not affected in the same way, given the iterative nature of the evidence gathering described above. it was time consuming to analyze the spatial planning review as the 11 files containing the results from each database had to be checked 20 times: once for each of the included publications.   table 3 shows that it was not possible to establish retrospectively how 75% of the included publications in the spatial planning review and 24% in the obesity review were found. it is unlikely that these publications were found by medline, given the checking that has been conducted, but it does mean that the non-database search techniques could not be fully assessed. previous studies have acknowledged that non-database sources can be difficult to record in a standardized format (rader, mann, stansfield, cooper, & sampson, 2014). one method to improve recording is to provide a narrative description of how the search progressed to explain the decisions made during each iteration, as was found in a different nice guidance programme (craven & levay, 2011). the narrative is a transparent record of the decision-making process during the search but it is not replicable in the way that a database strategy can be re-run exactly as before.   the corollary of tailoring sources to the review question is that the searcher must ensure that maximum benefit is derived from each one. it was beyond the scope of the current study to investigate whether medline could have been searched more effectively in the three reviews but it is an area worthy of further investigation. the obesity review did not aim to search medline comprehensively as an iterative approach was adopted. the spatial planning review balanced a specific search on medline with searches on topic-specific databases, which is consistent with the finding that “undertaking very sensitive searches across a number of databases may not be an effective use of resources” (bayliss et al., 2014, p. 310). the one publication retrieved from cinahl in the tuberculosis review (kelly-rossini, perlman, & mason, 1996) was actually indexed on medline at the time of searching, although it was felt that retrieving 94.7% of the available publications indicated the search strategy was of sufficient quality. a further study would be required to measure how the quality of the medline strategy affects the need to search other resources.   conclusions   the practical issues experienced in this study suggest it would be difficult to analyze retrospectively a larger sample of the public health systematic reviews commissioned by nice. it would be worth collecting the necessary data at the outset of guidance development to facilitate a larger study. the findings do suggest several issues that are worth considering when planning future public health guidance development searches. a range of databases was required, confirming the conclusion that public health questions cannot be answered with a “one size fits all solution” (bayliss et al., 2014, p. 304). grey literature contributed 42% of included publications in the obesity review, 20% in the spatial planning review, and 24% in the tuberculosis review, showing that sufficient time should be allowed to use additional search techniques to find this type of evidence. scoping searches will take on greater importance in the guidance development process, as they are useful for choosing sources, identifying topic-specific databases, and locating publications that will be useful in the later phases (such as in citation searching).   further work is required to determine how far the domain of public health influences the choice of sources. the tuberculosis review suggests that health protection, where there is overlap with clinical issues, may benefit more from medline than health improvement or improving services. this could be a more useful framework of analysis than attempting to make a binary choice between “medical” and “non-medical” reviews.   the findings from this study have been incorporated into the searching chapter in the current nice methods manual (nice, 2014). the manual no longer recommends a core list of databases that should be considered for all review questions. the manual recommends a list of databases to help searchers without making any of them mandatory (nice, 2014, appendix g). the manual also suggests that non-database methods should be used when it is “reasonably likely” they will be a productive source of evidence (nice, 2014, section 5.4). the manual emphasises the need to tailor the search approach to the topic of the research question and the type of evidence required to answer it. the factors that nice considers when planning which databases and search techniques to use in a systematic search are the multidisciplinary nature of public health and the different types of evidence required.   recommendations   the following recommendations are particularly pertinent to nice and the review teams it commissions, although they are applicable to anyone planning a systematic review on a public-health topic.   1.       search results should be tagged with the names of all sources where the record was found before removing duplicates. 2.       searches should be fully recorded and accompanied by a narrative outlining the key decisions, especially when an iterative approach is being used. 3.       the list of sources to be searched should be tailored to the review question. 4.       the appropriateness of databases for a topic should be tested at the start of a project. 5.       the value of topic-specific databases relevant to the review question should be explored early in a project. 6.       scoping searches can be a useful method for informing the later phases of evidence gathering. 7.       the efficiency and appropriateness of non-database search methods should be tested at the beginning of a project. 8.       research teams and those commissioning reviews should ensure adequate priority is given to using non-database methods. 9.       further research is required to compare the contribution of medline across the three domains of public health. 10.    the feasibility of a prospective study of this type should be investigated.   references   alpi, k. (2005). expert searching in public health. journal of the medical library association, 93(1), 97-103. retrieved from http://www.ncbi.nlm.nih.gov/pmc/articles/pmc545128/   bates, m. (1989). the design of browsing and berrypicking techniques for the online search interface. online information review, 13(5), 407-424. http://dx.doi.org/10.1108/eb024320   bayliss, s. e., davenport, c. f., & pennant, m. e. (2014). where and how to search for information on the effectiveness of public health interventions a case study for prevention of cardiovascular disease. health information & libraries journal, 41(4), 303-313. http://dx.doi.org/10.1111/hir.12075   benzies, k., premji, s., hayden, k. a., & serrett, k. (2006). state-of-the-evidence reviews: advantages and challenges of including grey literature. worldviews on evidence-based nursing, 3(2), 55-61. http://dx.doi.org/10.1111/j.1741-6787.2006.00051.x   betran, a., say, l., gulmezoglu, a. m., allen, t., & hampson, l. (2005). effectiveness of different databases in identifying studies for systematic reviews: experience from the who systematic review of maternal morbidity and mortality. bmc medical research methodology, 5(6). http://dx.doi.org/10.1186/1471-2288-5-6   beyer, f., & wright, k. (2013). can we prioritise which databases to search? a case study using a systematic review of frozen shoulder management. health information & libraries journal, 30(1), 49-58. http://dx.doi.org/10.1111/hir.12009   booth, a. (2010). how much searching is enough? comprehensive versus optimal retrieval for technology assessments. international journal of technology assessment in health care, 26(4), 431-435. http://dx.doi.org/10.1017/s0266462310000966   brettle, a. j., & long, a. f. (2001). comparison of bibliographic databases for information on the rehabilitation of people with severe mental illness. bulletin of the medical library association, 89(4), 353-362. retrieved from http://www.ncbi.nlm.nih.gov/pmc/articles/pmc57964/   cooper, c., levay, p., lorenc t., & craig g. (2014). a population search filter for hard-to-reach populations increased search efficiency for a systematic review. journal of clinical epidemiology, 67(5), 554-559. http://dx.doi.org/10.1016/j.jclinepi.2013.12.006   craven, j., & levay, p. (2011). recording database searches for systematic reviews what is the value of adding a narrative to peer-review checklists? a case study of nice interventional procedures guidance. evidence based library and information practice, 6(4). retrieved from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/11594   crumley, e., wiebe, n., cramer, k., klassen, t., & hartling, l. (2005). which resources should be used to identify rct/ccts for systematic reviews: a systematic review. bmc medical research methodology, 5(24). http://dx.doi.org/10.1186/1471-2288-5-24   faculty of public health. (2010). what is public health? retrieved from http://www.fph.org.uk/what_is_public_health   garside, r., pearson, m., hunt, h., moxham, t., & anderson, r. (2010). identifying the key elements and interactions of a whole system approach to obesity prevention. exeter: peninsula technology assessment group. retrieved from: http://www.nice.org.uk/guidance/ph42/resources/review-1-identifying-the-key-elements-and-interactions-of-a-whole-system-approach-to-obesity-prevention2   gray, s., barton, h., mytton, j., lease, h., carmichael, l., & joynt, j. (2010). the effectiveness and cost effectiveness of health appraisal processes currently in use to address health and wellbeing during plan appraisal. bristol: university of the west of england.   higgins, j. p. t., & green, s. (eds.). (2011). cochrane handbook for systematic reviews of interventions (version 5.1.0). oxford: the cochrane collaboration. retrieved from http://www.cochrane-handbook.org/   kelly, j. (2008, october). the value of searching multiple databases for guideline development. poster presented at the 16thcochrane colloquium, freiburg, germany.   kelly, m., morgan, a., ellis, s., younger, t., huntley, j., & swann, c. (2010). evidence based public health: a review of the experience of the national institute of health and clinical excellence (nice) of developing public health guidance in england. social science & medicine, 71(6), 1056-1062. http://dx.doi.org/10.1016/j.socscimed.2010.06.032   kelly-rossini, l., perlman, d., & mason, d. (1996). the experience of respiratory isolation for hiv-infected persons with tuberculosis. journal of the association of nurses in aids care, 7(1), 29-36. http://dx.doi.org/10.1016/s1055-3290(96)80035-5   lawrence, d. w. (2008). what is lost when searching only one literature database for articles relevant to injury prevention and safety promotion? injury prevention, 14, 401-404. http://dx.doi.org/10.1136/ip.2008.019430   lefebvre c., manheimer e., & glanville j. (2011) chapter 6: searching for studies. in: cochrane handbook for systematic reviews of interventions (version 5.1.0), eds. jpt higgins, s green. oxford: cochrane collaboration. retrieved from: http://handbook.cochrane.org/   lemeshow, a., blum, r., berlin, j., stoto, m., & colditz, g. (2005). searching one or two databases was insufficient for meta-analysis of observational studies. journal of clinical epidemiology, 58(9), 867-873. http://dx.doi.org/10.1016/j.jclinepi.2005.03.004   lohonen, j., isohanni, m., nieminen, p., & miettunen, j. (2010). coverage of the bibliographic databases in mental health research. nordic journal of psychiatry, 64(3), 181-188. http://dx.doi.org/10.3109/08039480903337378   mcdonald, s., taylor, l., & adams, c. (1999). searching the right database. a comparison of four databases for psychiatry journals. health libraries review, 16(3), 151-156. retrieved from http://www.ingentaconnect.com/content/bsc/hlr/1999/00000016/00000003/art00222   morgan, h., bauschmann, m., & weightman, a. (2011, october). finding public health evidence: how many stones need to be turned? poster presented at the 19th cochrane colloquium, madrid, spain. retrieved from http://www.cochrane.org/sites/default/files/uploads/abstracts_book_def.pdf   nice. (2009). methods for the development of nice public health guidance (2nd edition). london: nice.   nice. (2012a). identifying and managing tuberculosis among hard-to-reach groups (ph37). london: nice. retrieved from http://www.nice.org.uk/guidance/ph37   nice. (2012b). methods for the development of nice public health guidance (3rd edition). london: nice. retrieved from http://publications.nice.org.uk/methods-for-the-development-of-nice-public-health-guidance-third-edition-pmg4   nice. (2012c). obesity: working with local communities (ph42). london: nice. retrieved from http://www.nice.org.uk/guidance/ph42   nice. (2014). developing nice guidelines: the manual. london: nice. retrieved from: http://www.nice.org.uk/article/pmg20/    ogilvie, d., hamilton, v., egan, m., & petticrew, m. (2005). systematic reviews of health effects of social interventions: 1. finding the evidence: how far should you go? journal of epidemiology and community health, 59, 804-808. http://dx.doi.org/10.1136/jech.2005.034181   o’mara, a., marrero-guillamon, i., jamal, f., lehmann, a., cooper, c., & lorenc, t. (2010). tuberculosis evidence review 1. review of barriers and facilitators. london: matrix knowledge group. retrieved from http://www.nice.org.uk/guidance/ph37/resources/review-1-review-of-barriers-and-facilitators2   papaioannou, d., sutton, a., carroll, c., booth, a., & wong, r. (2010). literature searching for social science systematic reviews: consideration of a range of search techniques. health information & libraries journal, 27(2), 114-122. http://dx.doi.org/10.1111/j.1471-1842.2009.00863.x   rader, t., mann, m., stansfield, c., cooper, c., & sampson, m. (2014). methods for documenting systematic review searches: a discussion of common issues. research synthesis methods, 5(2), 98-115. http://dx.doi.org/10.1002/jrsm.1097   rollin, l., darmoni, s., caillard, j. f., & gehanno, j. f. (2010). searching for high-quality articles about intervention studies in occupational health: what is really missed when using only the medline database? scandinavian journal of work, environment & health, 36(6), 484-487. http://dx.doi.org/10.5271/sjweh.3082   royle, p., & waugh, n. (2003) literature searching for clinical and cost-effectiveness studies used in health technology assessment reports carried out for the national institute for clinical excellence appraisal system. health technology assessment, 7(34). retrieved from http://www.journalslibrary.nihr.ac.uk/__data/assets/pdf_file/0019/65125/fullreport-hta7340.pdf   royle, p., bain, l., & waugh, n. (2005). systematic reviews of epidemiology in diabetes: finding the evidence. bmc medical research methodology, 5(2). http://dx.doi.org/10.1186/1471-2288-5-2   slobogean, g., verma, a., giustini, d., slobogean, b., & mulpuri, k. (2009). medline, embase, and cochrane index most primary studies but not abstracts included in orthopedic meta-analyses. journal of clinical epidemiology, 62(12), 1261-1267. http://dx.doi.org/10.1016/j.jclinepi.2009.01.013   stansfield, c., kavanagh, j., rees, r., gomersall, a., & thomas, j. (2012). the selection of search sources influences the findings of a systematic review of people's views: a case study in public health. bmc medical research methodology, 12(55). http://dx.doi.org/10.1186/1471-2288-12-55   stansfield, c., brunton, g., & rees, r. (2013). search wide, dig deep: literature searching for qualitative research. an analysis of the publication formats and information sources used for four systematic reviews in public health. research synthesis methods, 5(2), 142-151. http://dx.doi.org/10.1002/jrsm.1100   stevinson, c., & lawlor, d. a. (2004). searching multiple databases for systematic reviews: added value or diminishing returns? complementary therapies in medicine, 12(4), 228-232. http://dx.doi.org/10.1016/j.ctim.2004.09.003   subirana, m., sola, i., garcia, j., gich, i., & urrutia, g. (2005). a nursing qualitative systematic review required medline and cinahl for study identification. journal of clinical epidemiology, 58(1), 20-25. http://dx.doi.org/10.1016/j.jclinepi.2004.06.001   taylor, b., wylie, e., dempster, m., & donnelly, m. (2007). systematically retrieving research: a case study evaluating seven databases. research on social work practice, 17(6), 697-706. http://dx.doi.org/10.1177/1049731507304402   turner, a., liddy, e., bradley, j., & wheatley, j. (2005). modeling public health interventions for improved access to the gray literature. journal of the medical library association, 93(4), 487-494. retrieved from http://www.ncbi.nlm.nih.gov/pmc/articles/pmc1250325/   weaver, n., williams, j. l., weightman, a. l., kitcher, h. n., temple, j. m., jones, p., & palmer, s. (2002). taking stox: developing a cross disciplinary methodology for systematic reviews of  research on the built environment and the health of the public. journal of epidemiology and community health, 56, 48-55. http://dx.doi.org/10.1136/jech.56.1.48   whitener, b. l., van horne, v. v., & gauthier, a. k. (2005). health services research tools for public health professionals. american journal of public health, 95(2), 204-207. http://dx.doi.org/10.2105/ajph.2003.035030   wilczynski, n. l., haynes, r. b., lavis, j. n., ramkissoosingh, r., & arnold-oatley, a.e. (2004) optimal search strategies for detecting health services research in medline. canadian medical association journal, 171(10), 1179-1185. http://dx.doi.org/10.1503/cmaj.1040512   woodman, j., harden, a., thomas, j., brunton, j., kavanagh, j., & stansfield, c. (2010). searching for systematic reviews of the effects of social and environmental interventions: a case study of children and obesity. journal of the medical library association, 98(2), 140-146. http://dx.doi.org/10.3163/1536-5050.98.2.006   evidence based library and information practice evidence based library and information practice 2010, 5.2  evidence based library and information practice       article    evaluating qualitative research studies for evidence based library and information  practice    doug suarez  liaison librarian  brock university, ontario, canada  email: dsuarez@brocku.ca    received:  21 jan. 2010          accepted: 21 apr. 2010       2010 suarez. this is an open access article distributed under the terms of the creative commons‐attribution‐ noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which  permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly  attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the  same or similar license to this one.    abstract    objective ‐ research studies in the literature that may be useful for solving  professional practice questions are frequently based on findings from studies that use  qualitative methods. criteria used to appraise qualitative research are still evolving  and often lack the readily understood precision of the numerical criteria used for  quantitative research. qualitative research studies can often be more valuable than  quantitative studies for a given situation. this article offers a template to assess  qualitative methods used in practitioner‐led research for library and information  science.    methods – this paper presents a narrative scenario of a library management problem.   after conducting a literature search, the author identified an article with apparent  relevance and potential to help resolve the problem. the author then evaluated the  article using an assessment framework to illustrate how qualitative library research  can be assessed. the paper examines the components of the framework, and explores  the process.    results ‐ the appraisal of the selected article demonstrates that qualitative methods  used in library research can be critically evaluated for evidence to assist librarians in  addressing their professional practice questions.      conclusions ‐ results obtained from qualitative research projects can be applied as  evidence to support library practice. qualitative methods are useful, and for many  75 evidence based library and information practice 2010, 5.2  76 library practice issues, the assessment process illustrated here will help librarians  evaluate the evidence and assess its appropriateness for practice.    introduction     library professionals often find themselves in  situations where they need answers to  questions that emerge from their professional  practice. one method to obtain the answers is  to search the professional literature. the  ability to assess the quality of the research  studies found in a literature search is essential  in order to evaluate and apply the findings to  answer these questions in a meaningful and  useful manner. depending on the research  design used in these studies, the methods of  collecting data and their analysis may include  either quantitative, or qualitative, or a  combination of both methods.  many library  research studies use a variety of methods that  are selected for their appropriateness to best  suit the research design (fidel, 2008).  for  example, to determine how effectively the  library’s journal collection is meeting faculty  and student needs, quantitative methods  might include surveys as well as qualitative  methods such as interviews, focus groups, and  observation studies with selected student and  faculty populations within a library.  to find  out if a specific change in service has had an  effect (e.g., adding specific journal titles), this  service intervention could be measured before  and after implementation by using survey or  other instruments to collect numerical data.  in  this example data could be obtained for the  number of journal titles used before and after  the introduction of new titles. if librarians also  wanted to assess the perceptions of students  and faculty to this service intervention, then  interviews and focus groups could be  conducted to collect this sort of data.    no matter what methods are employed to  gather and analyse research data, the findings  have to be assessed before the evidence can be  used for practice. given (2006) argued  persuasively for the acknowledgment of  qualitative research within evidence based  practice library and information practice and  also outlined the qualitative paradigm and the  importance of its approach. booth (2009b)  recently acknowledged that where assessment  is based on good practical client‐centered  evaluation models, it can contribute usefully  to well‐informed decisions at the local level.  while librarians have always been cognizant  of the inherent value of library research, the  recent trend for evidence based practice has  provided a focus for improved research  quality.  evidence based practice in library  science has been associated largely with  research in health sciences librarianship and is  often referred to as evidence based  librarianship or, most recently, evidence based  library and information practice (eblip).   eldredge (2002) and others (booth, 2003;  booth, 2009b; booth & brice, 2004;  koufogiannakis & crumley, 2006) have noted  that reliable and valid evidence has become  the expected norm for evidence based practice  in most library and information studies  research, regardless of specific library  contexts.     the assessment criteria used in eblip are  more arguably appropriate for quantitative  methods, perhaps because of the fundamental  difference between the paradigms with which  evidence based practice has traditionally been  associated.  according to loiselle, profetto‐ mcgrath, polit, and beck (2007), the  positivistic paradigm is “the traditional  paradigm underlying the scientific approach,  which assumes that there is a fixed, orderly  reality that can be objectively studied; often  associated with quantitative research” (p. 508).  the naturalistic paradigm is an alternative to  this, in that there are “multiple interpretations  of reality, and the goal of research is to  understand how individuals construct reality  within their context; often associated with  qualitative research” (p. 506).  the credibility  and accuracy of data generated from  quantitative methods may well be more  readily understood because of their numeric  nature and subject to interpretation and  analysis by statistical methods, whereas the  evidence based library and information practice 2010, 5.2  data generated by qualitative studies are  interpreted and analyzed almost  simultaneously and integrated into a unified  whole. because of this fundamental difference,  the transferability, or generalizability, of  findings from quantitative methods may be  more easily accepted as valid, compared to the  findings of qualitative methods. quantitative  studies are part of the traditional scientific  method, subject to numerical analysis, and  reported in a style based on the traditional  scientific report format that is normally  accepted as being an objective description of  the research.  bazerman (1988) referred to this  style as “prescriptive rhetoric” (p. 275).  sandelowski and barroso (2002) argued that  research findings are more difficult to locate  and communicate in qualitative studies,  because the methods and findings are not as  easily suited to this style.  qualitative methods  and the results obtained from using such  methods have sometimes been suspect,  regarded as less rigorous than other methods,  and even devalued (booth, 2009b; bradley,  1993; johannsen, 2004; fitzgerald & galloway,  2001).  as a result, research outcomes  generated from qualitative studies may not be  regarded as evidence, and they may not be  disseminated or used as widely as they could  be (devers, 1999).    how do librarians know when qualitative  research findings provide evidence that can be  used to assist them in their library practice?   one way is to use an assessment framework.  when assessing qualitative research, the seven  criteria should be considered and incorporated  in the process, and these are generally  included in some format in many assessment  frameworks (cohen & crabtree, 2008):     • maintaining an ethical research policy.  • importance of the research.  • clarity and coherence of the research  proposal.  • use of appropriate and rigorous  methods.  • incorporating reflexivity (critical self‐ reflection).   • attending to researcher bias.  • establishing validity (i.e., credibility)  and reliability (i.e., verification).    beck and manuel (2008) list some of the most  common errors in research assessment that  frameworks help identify.  some of those  errors include:    • not asking the right question or not  asking the question in the right way.  • gathering data at the wrong time or  place.  • using unrepresentative samples, or  failing to recognize possible response  bias among those members of the  sample who do respond.  • failing to control for or consider  possible experimenter expectancy  effects that arise when researchers’  measurements are shaped to match  their own hypotheses or expectations.  • not allowing for research biases (e.g.,  the hawthorne effect, the effect on  data results from research subjects’  awareness that they are being  studied).  • over‐generalizing to conclusions not  directly supported by the research  data (generalizability validity).     this article outlines the process of reviewing  qualitative research methods by presenting a  dilemma as part of a library practice scenario.  part of this scenario will include an article  from a literature search that will be evaluated  in the course of the assessment exercise. the  article will  discuss and review a range of  assessment tools and present a possible  framework for assessing library research.  in  the critique of the article, this paper will use  the framework’s component parts to identify  the process of discovering and evaluating the  evidence. while it is not the intention here to  provide a comprehensive review of qualitative  and quantitative methods, the assessment  framework will address key elements of  evaluation, particularly as these elements  pertain to qualitative methods. the paper will  also discuss these elements of evaluation in  relation to quantitative methods in this  context.   77 evidence based library and information practice 2010, 5.2  78 academic training in anthropology and  working experience in academic libraries have  contributed to the author’s commitment to the  use of qualitative methods in professional  library practice (suarez, 2007).  the discipline  of anthropology is well known for its  productive use of qualitative methods, and it  shares many of the same theoretical and  methodological procedures and methods used  in other social science and health science  disciplines. (berg, 2004; bryman, 2004;  creswell, 2009; griffiths, 2009; sandelowski,  2004).  the purpose of this article is to  demonstrate that there is a place for  qualitative research methods within evidence  based library and information practice and to  help librarians determine whether the research  has been conducted rigorously.     methodology    the following library scenario illustrates a  typical academic library practice situation  requiring resolution:     a liaison librarian for social science and applied  health science disciplines at a mid‐sized university  has a dilemma. the university has recently  expanded its department of graduate studies.   two of the librarian’s subject specialties, sociology  and kinesiology, currently have high  undergraduate enrollments. with the expansion of  the graduate program, the demands for library  services are expected to grow exponentially. both  the university and library mission statements  include statements about taking positive steps to  meet the needs of graduate students.  while there  have always been graduate programs offered at the  university, this expansion is expected to cause  serious strains on programs and services. how this  change will affect the liaison librarian’s role and  effectiveness, particularly from the point of view of  teaching information seeking skills to graduate  students, will be challenging.  the librarian  suspects that graduate students will need different  levels of instruction, but is not sure if this can be  substantiated, or if it is just an intuitive  observation. initially the librarian thinks that a  literature search would be appropriate as part of a  needs assessment. the librarian conducts a search  in standard subject related databases using the  terms “graduate students” and “information  literacy.” the search is limited to recent peer‐ reviewed journals published in the last 5 years; the  search retrieves 17 article citations.  the librarian  browses the abstracts and selects one article that  looks particularly relevant. titled “development of  information search expertise: postgraduates’  knowledge of searching skills” (chu and law  2007), the article appears to address her main areas  of interest. the librarian notes that the study  employs qualitative methods in its research design.   the librarian downloads a copy of the article and  proceeds to read and assess its worth.      assessment tools    there are a number of readily available tools  to help evaluate qualitative research.  examples include the joanna briggs institute  qari assessment tool (briggs, 2009) and the  critical appraisal skills programme (casp)  worksheet created by the public health resource  unit, national health service (2006). these were  developed and have been used to assess the  rigor of research method and the credibility and  relevance of research findings in studies that use  qualitative methods.  other assessment tools  have referred to library guidelines or checklists  developed by national organizations (american  library association, 2004) or in‐house (ward,  2004; fitzgerald & galloway, 2001).  these in‐ house assessment tools often have limited value  because of their specificity. the critical skills  training in appraisal for librarians (cristal)  checklists developed for appraising library  studies (booth & brice, 2003) and the ebl critical  appraisal checklist (glynn, 2006) are good  examples of tools used for critical appraisal of  eblip.      the particular tool used here for illustrative  purposes has been selected on the basis of its  brevity, clarity, and adaptability for library  practice. in their article on evaluating qualitative  research studies in nursing practice, russell and  gregory (2003) presented a list of questions, to  use in assessing qualitative research. this has  been adapted for use in library practice and is  reproduced in table 1 below.     evidence based library and information practice 2010, 5.2  table 1  questions to help critically appraise qualitative research (adapted from russell and gregory, 2003)  1. are the findings valid?  a. is the research question clear and adequately substantiated?  b. is the design appropriate for the research question?  c. was the sampling method appropriate for the research question?  d. were data collected and managed systematically?  e. were the data analyzed appropriately?  2. what are the findings?  a. is the description of findings thorough?  3. how can the findings be applied to library practice?  a. what meaning and relevance does the study have for library practice?  b. does the study help me understand the context of library practice?   c. does the study enhance my knowledge about library practice?    description of the assessment framework  and article critique     librarians can use the three main criteria  headings of russell and gregory’s framework  as a model to evaluate qualitative library  research:  • are the findings valid?   • what are the findings?   • how can the findings be applied to  library practice?     1. are the findings valid?     the librarian should start by looking for a  clear statement of the question that the study  sought to answer, as well as looking for factors  that make this relevant for the librarian’s  particular library practice question.  the  question should be clearly stated within the  context of prior research and should include  relevant literature referents.  the research  design should be clearly defined and  explained and the narrative should include a  clear discussion of the qualitative approaches  that have been used and why.  possible  research approaches might include grounded  theory, phenomenology, case studies, or  narrative and historical research, among  others.  whatever approach is used should fit  the research question, and the methods of  sampling should be appropriate for the  research question (creswell, 2009).      sampling tends to be purposive, rather than  random, in qualitative studies; i.e., the sample  tends to be selected for the potential richness  of its contribution, rather than randomly as in  quantitative studies.  the sample size tends to  be small to allow for the tendency of  qualitative research to enfold and provide rich  descriptive data.  the reasons for the sample  selection, its size, and the description of the  process used to select the sample, should be  thoroughly presented and discussed, because  the study results may not be meaningful, or  easily transferable, outside the study’s  immediate context.  researchers judge the  relevance of a sample in relation to the study  results, and the librarian should be able to  assess the validity of the results by their  comprehensiveness and by the extent to which  the research questions were answered.    how the data were collected, whether by field  observation, document analysis, or interviews,  should be thoroughly described and judged to  avoid bias and misleading results.  typically  the article should explain to the reader the  number of observations, interviews, or  documents used. it should also explain how  the data were collected ‐‐ by whom, when, and  for how long a period of time.  finally, the  data analysis phase of the study should be  clearly described.  this is extremely important  in qualitative research, because the research  question is often exploratory. in the analysis,  the researcher describes the data retrieved,  explains how the data is collected, discusses  patterns observed, and organizes the results  into a conceptual framework.  data collection  continues in a cyclical pattern until saturation  79 evidence based library and information practice 2010, 5.2  (informational redundancy) is observed, and  the researchers decide that no new  information is likely to emerge to change the  conceptual framework. (this is often assumed  and not elaborated upon in most qualitative  studies.) data analysis software packages such  as nvivo are sometimes used to store,  organize, and retrieve qualitative data, but  they do not actually analyze the data.   researchers often demonstrate data  collaboration with the use of other information  resources (often referred to as triangulation),  and this includes the use of methods such as  surveys to add weight to the analysis.  the  methods used and how the data were  analyzed need to be thoroughly discussed and  presented to ensure their depth and breadth so  that the librarian can judge how meaningful  they appear in relation to the research  question and to her own library practice  situation.    article critique:    chu, s. k.‐w., & law, n. (2007).  development  of information search expertise:  postgraduates’ knowledge of searching skills.   portal: libraries and the academy, 7(3), 295‐316.    this assessment of chu and law’s article  utilizes the five subheadings in this category :  a. is the research question clear and  adequately substantiated?  chu and law (2007) explored the information  searching skills used by two groups of  doctoral students when they searched  databases to find research studies to support  their academic program requirements  (engineering and education). they had three  research questions: what were the information  searching behaviours and skills of two groups  of graduate students, and how had these skills  developed over time? were these skills used  more often when the students became more  familiar with searching techniques and  databases? what similarities and differences  exist between the two groups in terms of  information searching behaviour and skills?  the literature review provided references to  previous studies where results indicated  graduate students’ research skills were  inadequate. two major approaches used by  study participants were noted in the literature  review: novice‐expert comparison; and  developmental studies of expertise. in the first  instance, other researchers had studied the  characteristics of experts in a given subject  field but had not, for the most part, looked at  the development of these skills over time.  in  the second instance, more studies were  identified that looked at various components  of the search process, including search tactics  and operators, keyword searching, boolean  searching, subject searching, author and title  searching, and browsing. the population  samples of these studies were largely  comprised of undergraduate students.       b. is the design appropriate?    chu and law wanted to investigate graduate  students’ development of information  searching expertise over time, but they  discovered there was a lack of longitudinal  searching behaviour studies. they designed a  longitudinal study with a small participant  sample in order to gain an in‐depth  understanding of the processes used by two  groups of graduate students as they moved  from novice to expert information searchers.    c. was the sampling method appropriate for  the research question and design?   twelve graduate students (six from  engineering and six from education) were  purposively selected for this study.  all but  one were first‐year graduate students and  were appropriate participants for establishing  a baseline of novice searchers. the remaining  student was a part‐time third‐year student  who self‐identified as someone who needed  more help with searching techniques.   selecting three male and three female  participants from each of the two academic  disciplines minimized gender effect bias.  it  was not entirely clear why these two  disciplines were more appropriate to study  than others, apart from their disparate natures  (science versus social science), but this did not  compromise the study’s value, as the study  80 evidence based library and information practice 2010, 5.2  made no claim to be comparing specific  disciplines.  any idiosyncrasies associated  with searching behaviour within each  discipline were not the main aims of the  research and did not compromise the study.     d. were data collected and managed  systematically?    four kinds of data were collected for analysis:  written search queries; think‐aloud strategies;  self‐reporting questionnaires; and interviews.   students used written search statements when  they searched databases, and when students  performed their searches they used think‐ aloud strategy protocol verbalizations. these  two data sets were collected from a series of  five meetings with individual students over a  one‐year period.  the authors provided a good  description of each meeting: first, a student  would perform a search for twenty minutes,  followed by the researcher demonstrating how  he would do the same search (also twenty  minutes).  after the researcher’s  demonstration, the student’s second search  period followed, and again that would be  followed by a second demonstration by the  researcher.  to avoid undue influence from the  second search period and the researcher’s  demonstrations, only the first search period  data was used for analysis.    a self‐reporting questionnaire was  administered at the end of search meetings  one, three, and five to help the researchers  determine what searching knowledge and  skills each student had.  the authors used a  likert scale to ascertain the level of  importance of (and familiarity with) various  searching techniques and strategies.  the  researchers conducted interviews with each  student to help augment perceptions of the  searching process. the researchers realized  that the data from the five search meetings  was limited and amounted to only  approximately 90 minutes of research time per  student. in a future study it would be prudent  to add other methods of analysis to add more  depth and breadth to this research and to  discover more about the students’ searching  knowledge and skills. it was not clear whether  the interviews were semi‐structured, what the  questions were, or when they were conducted  during the study.    the researchers used common qualitative data  collecting strategies. they observed the  searching behaviours and think‐aloud  statements of search strategies that were  exhibited by the students in the search  meetings.  they used document analysis when  they obtained the initial written search query  statements from each student prior to their  searches.  finally, they used interviews to  obtain student search experiences.    e. were the data analyzed appropriately?  the authors analyzed their data and provided  a series of tables that listed the results  obtained from their data analysis: search types  (keyword, subject, author, title, call number,  and browse); subject searches (searches that  use the automatic mapping feature of a  database); keyword searches; students’  familiarity with search knowledge/skills  (average rating of student search skills);  importance of search knowledge/skills  (average rating of student acknowledgment of  their importance); students’ use of search  operators and parentheses; students’  perceived importance of information search  skills at the end of the study in meeting five;  and lastly, the number of students who had  not obtained the search skills they had  identified as lacking in the initial meeting.    the authors provided a thorough description  and discussion of the tables. they used  consistent terminology and also provided the  results within a context that could be  understood easily by the librarians engaged in  instruction.  although they did not provide  details of the interview questions, sufficient  information about the interview process was  provided. presumably these interview  questions were closely related to the survey  questions, and the data obtained through  observation were sufficient to provide good  data saturation.       81 evidence based library and information practice 2010, 5.2  2. what are the findings?     a well‐written article with clearly  communicated methods and results will  provide a thorough understanding of the  research. qualitative research is often context  specific and purports to make sense within  narrow boundaries, as defined in the research  question. it is imperative that the researcher  describes and justifies the findings within that  context. this will help the librarian to make  informed decisions about the applicability of  those findings to the research problem in  question.  a clear and thorough discussion  about the limitations of the study is  particularly important so that librarians can  judge the relevance of the findings.    article critique:    chu, s. k.‐w., & law, n. (2007).  development  of information search expertise:  postgraduates’ knowledge of searching skills.   portal: libraries and the academy, 7(3), 295‐316.    chu and law’s article identified several  research findings that can be assessed:    the authors were primarily interested in  discovering if students’ searching skills  improved over time and if their familiarity  with searching skills contributed to the  positive development of their expertise.  the  authors claimed that students’ skills advanced  as they became more familiar with the various  search techniques. they identified three major  findings:     • library instruction was necessary  even at the postgraduate level.   • being computer literate was not  the same as being information  literate.   • offering library tailored  instruction by subject was  necessary to address different  academic disciplines.      these findings were clearly articulated and  based on the evidence that was described and  presented in the article’s methods section.  the  research question and findings were  congruous. their research articulated how  students’ knowledge of searching skills  developed over time and influenced changes  in searching behaviour as students became  more familiar with searching techniques. the  authors concluded that further studies with  more participants, as well as further studies of  individual student search strategies within an  academic discipline, would be useful to obtain  more reliable data.     3. how can the findings be applied to library  practice?    once the librarian has thoroughly read and  examined the article, its applicability can be  assessed. it is necessary to reflect on the study  to make sure the research question, design,  and the results are congruent.  using the  above assessment framework criteria,  common sense, and personal professional  experience, a determination must be made  whether the results can be used as evidence  for library practice.  this decision will depend  on professional judgment as to how reliable  and useful these findings will be for the  specific library practice setting.    article critique:    chu, s. k.‐w., & law, n. (2007).  development  of information search expertise:  postgraduates’ knowledge of searching skills.   portal: libraries and the academy, 7(3), 295‐316.    finally, chu and law’s article can be assessed  for its applicability under the respective  subheadings in this section:    a. what meaning and relevance does this  study have for library practice?    chu and law made a strong case for the  relevance and importance of addressing the  changing service needs of library clientele. the  study clearly articulated the library context of  student learning and emerging instructional  skills that students were required to have in  order to succeed in their academic careers. the  findings provided evidence that graduate  82 evidence based library and information practice 2010, 5.2  students would and could benefit from  discipline focused library instruction.  subsequently, liaison librarians at their  university could more realistically target  groups of students from different faculties and  emphasize different skills depending on their  students’ perceived needs.     b. does the study help me understand the  context for library practice?   c. does the study enhance my knowledge  about library practice?    from their research results, the authors noted  the following limitations: small sample size,  limited number of sampled subject disciplines,  and recognition that students within a specific  discipline will exhibit varying search expertise  and will need an assortment of search skills.  therefore, it would be prudent to not  generalize at the group level and to avoid any  assumption that the same results would relate  to sociology and kinesiology exactly. but on  the whole, this study provides the liaison  librarian with good evidence for solving the  scenario problem by planning for tailored  instruction.   discussion     librarians need to have evidence on which to  base their informed decisions.  whenever they  consult the library literature for the purpose of  finding evidence to help answer service  problems in their library practice, the articles  they choose need to be scrutinized to ensure  that findings are valid and relevant to their  practice.  the emerging standards of eblip  require this approach, and librarians in health  science disciplines have arguably embraced  this concept more thoroughly than those in  other academic disciplines (bayley &  mckibbon, 2006; booth, 2003). the intention in  this article has been to offer an immediate and  practical method of assessing library research  that uses qualitative methods, in the broader  library research context.    if librarians use data from findings that are  reported in the literature, they need to be  assured that these findings are valid and  reliable. librarians can and should make  informed decisions about the quality of  research, no matter what methods are used.   qualitative methods need to be assessed  differently than quantitative methods, because  of the different paradigms with which they are  associated, naturalistic and positivistic,  respectively.  the constructivist nature of the  naturalistic paradigm, the different research  problems that lend themselves to this  paradigm, the different methods of data  collection and analysis, and the reporting style  of research studies that use this paradigm can  make evidence difficult to interpret and  appraise (sandelowski & barroso, 2002).   nevertheless, it is possible to appraise  naturalistic research evidence using a  framework, such as the one used here, to help  librarians identify common research errors as  outlined earlier by beck and manuel (2008).   more research needs to be done in library  contexts to identify and construct library‐ specific research assessment frameworks that  address library research problems (booth,  2003; booth, 2009a). in the meantime,  librarians can use the assessment framework  illustrated here or similar frameworks to help  assess evidence to assist them in their ongoing  library practices.  conclusion    the aim of this article has been to help  librarians evaluate qualitative research so they  can use it in their daily practice. evidence  derived from qualitative methods may be  disregarded because it can sometimes be  overlooked. this can be because the research  results of qualitative studies are compared to  the findings and rigor associated with  quantitative methods, or because of the  inherently different paradigms with which  each is associated. taking into account the  suitability of different approaches and  methods for specific research problems,  librarians may not find the evidence or use it  to best advantage in their professional  practice. to address this omission, the  assessment framework suggested here  illustrates the criteria and process for assessing  qualitative research, while simultaneously  83 evidence based library and information practice 2010, 5.2  applying the assessment method as a template  to critique a research article. the findings have  been assessed, and the results deemed  valuable for use in resolving the library  scenario that was initially provided.    it is anticipated that this article will provide a  reliable and practical demonstration of how  library research based on qualitative methods  can be evaluated and judged on its worthiness  and then used for evidence to guide eblip.   by employing an assessment framework such  as the one offered here, and by combining its  use with personal work experience and  common sense, librarians will be able to judge  the inherent value of published library  research and to use it as evidence for practice.    references  american library association.  (2004).  rusa  guidelines for behavioral performance of  reference and information services providers.   retrieved 17 may 2010 from  http://www.ala.org/  bayley, l., & mckibbon, a. (2006).  evidence‐ based librarianship: a personal  perspective from the medical/nursing  realm.  library hi tech, 24(3), 317‐323. doi:  10.1108/07378830610692091   bazerman, c. (1988).  shaping written  knowledge: the genre and activity of the  experimental article in science.  madison,  wi: university of wisconsin press.    beck, s. e. & manuel, k. (2008).  practical  research methods for librarians and  information professionals.  new york, ny:  neal‐schuman.    berg, b. l. (2004).  qualitative research methods  for the social sciences (5th ed.).  boston, ma:  pearson.  booth, a. (2009a). a bridge too far?  stepping  stones for evidence based practice in an  academic context.  new review of academic  librarianship, 15(1), 3‐34.  doi:  10.1080/13614530903123534  booth, a. (2009b).  research or evaluation?  does it matter?  health information and  libraries journal, 26(3), 255‐258. doi:  10.1111/j.1471‐1842.2009.00858.x  booth, a. (2003).  bridging the research‐ practice gap? the role of evidence based  librarianship.  new review of information  and library research, 9(1), 3‐23. doi:  10.1080/13614550410001687909  booth, a., & brice, a. (2004).  why evidence‐ based information practice?  in a. booth  & a. brice (eds.), evidence‐based practice  for information professionals: a handbook (pp.  1‐13).  london: facet publishing.  booth, a., & brice, a. (2003).  clear‐cut?   facilitating health librarians to use  information research in practice.  health  information and libraries journal, 20(suppl.  1), 45‐52. doi: 10.1046/j.1365‐ 2532.2051.10.x  bradley, j. (1993). methodological issues and  practices in qualitative research.  library  quarterly, 63(4), 431‐449. doi:  10.1086/602620  briggs, j. (2009). jbi qari: qualitative  assessment and review instrument.   retrieved 28 may 2010 from  http://www.jbiconnect.org/connect/down loads/qari_crit_apprais.pdf  bryman, a. (2004).  quantitative and  qualitative research, debate about.  in m.  lewis‐beck, a. bryman, & t. futing lao  (eds.), the sage encyclopedia of social science  research methods (pp. 895‐896).  thousand  oaks, ca: sage.  chu, s. k.‐w.,  & law, n. (2007).   development of information search  expertise: postgraduates’ knowledge of  searching skills.  portal: libraries and the  academy, 7(3), 295‐316.  84 evidence based library and information practice 2010, 5.2  85 cohen, d. j. & crabtree, b. f. (2008).   evaluative criteria for qualitative research  in health care: controversies and  recommendations.  annals of family  medicine, 6(4), 331‐339. doi:  10.1370/afm.818  creswell, j. w. (2009).  research design:  qualitative, quantitative, and mixed methods  approaches (3rd ed.).  los angeles, ca:  sage.  devers, k. j. (1999).  how will we know  “good” qualitative research when we see  it? beginning the dialogue in health  services research.  health services research,  34(5 pt. 2), 1153‐1188.    eldredge, j. d. (2002).  evidence‐based  librarianship: what might we expect in  the years ahead?  health information and  libraries journal, 19(2), 71‐77.    fidel, r. (2008).  are we there yet?  mixed  methods research in library and  information science.  library & information  science research, 30(4), 265‐272. doi:  10.1016/j.lisr.2008.04.001  fitzgerald, m. a., & galloway, c. (2001).   relevance judging, evaluation, and  decision making in virtual libraries: a  descriptive study.   journal of the american  society for information science &  technology, 52(12), 989‐1010.  given, l. (2006).  qualitative research in  evidence‐based practice: a valuable  partnership.  library hi tech, 24(3), 376‐ 386.  doi: 10.1108/07378830610692145    glynn, l. (2006).  a critical appraisal tool for  library and information research.  library  hi tech, 24(3), 387‐399.  doi:  10.1108/07378830610692154    griffiths, f. (2009).  research methods for health  care practice. los angeles, ca: sage.    johannsen, c. g. (2004).  managing fee‐based  public library services: values and  practices. library management, 25(6/7),  307‐315. doi: 10.1108/01435120410547977    koufogiannakis, d. & crumley, e. (2006).   research in librarianship: issues to  consider.  library hi tech, 24(3), 324‐340.   doi: 10.1108/07378830610692109    loiselle, c. g., profetto‐mcgrath, j., polit, d.  f., & beck, c. t.  (2007). canadian  essentials of nursing research (2d ed.).   philadelphia, pa: lippincott, williams &  wilkins.     public health resource unit, national health  service. (2006). critical appraisal skills  programme (casp): making sense of  evidence. retrieved 28 may 2010 from  http://www.phru.nhs.uk/pages/phd/reso urces.htm  russell, c. k., & gregory, d. m. (2003).  evaluation of qualitative research studies.   evidence based nursing, 6(2), 36‐40.  doi:  10.1136/ebn.6.2.36  sandelowski, m., (2004).  qualitative research.   in m. lewis‐beck, a. bryman, & t. futing  lao (eds.), the sage encyclopedia of social  science research methods (pp. 893‐894).   thousand oaks, ca: sage.  sandelowski, m., & barroso, j. (2002).  reading  qualitative studies.  ijom: international  journal of qualitative methods, 1(1), 74‐108.  retrieved 28 may 2010 from  http://ejournlas.library.ualberta.ca  suarez, d. (2007).  what students do when  they study in the library: using  ethnographic methods to observe student  behavior.  e‐jasl:the electronic journal of  academic and special librarianship, 8(3).  retrieved 28 may 2010 from  http://southernlibrarianship.icaap.org/    ward, d., (2004).  measuring the completeness  of reference transactions in online chats:  results of an unobtrusive study.   reference & user services quarterly, 44(1),  46‐56.    abstract discussion conclusion references evidence summary   no pedagogical advantage found between libguides and other web page information literacy tutorials   a review of: bowen, a. (2014). libguides and web-based library guides in comparison: is there a pedagogical advantage? journal of web librarianship, 8(2), 147-171. doi:10.1080/19322909.2014.903709   reviewed by: kimberly miller research & instruction librarian for emerging technologies albert s. cook library towson university towson, maryland, united states of america email: kimberlymiller@towson.edu   received: 24 nov. 2014 accepted: 26 jan. 2015      2015 miller. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – this study compares two versions of an online information literacy tutorial – one built with springshare’s libguides and one built as a series of web pages – in order to determine if either platform provides a pedagogical advantage in delivering online instruction.   design – experimental, posttest only.   setting – large, public, primarily undergraduate four-year university in the western united states of america with 16,000 full time equivalent student enrollment.   subjects – the sample consists of 812 students enrolled in 25 sections of a 100-level communications studies course. of those students, 89 responded to the study’s posttest survey (11% response rate).   of the 89 respondents, 53 viewed the libguide tutorial: 12 respondents were male, 33 respondents were female, and 8 respondents did not report their gender. of the 53 libguide participants, 47 responded to other demographic questions, and were primarily 18-20 years old (94%), first-year students (79%), and non-communication studies majors (91%).   the remaining 36 respondents viewed the web page tutorial: 7 respondents were male, 25 respondents were female, and 4 did not report their gender. of the 32 respondents that provided demographic information, all participants were 18-20 years old, 31 of 32 were first-year students, and the majority were non-communication studies majors (78%).   methods – students completed an online tutorial designed to teach them information literacy skills necessary to find resources for a class debate. each section was randomly assigned to one of two information literacy tutorials: 12 sections viewed a tutorial built with libguides and 13 sections viewed a web page tutorial. the two tutorials included identical instructional content and worksheet. each of the tutorials’ six sections were tied to the acrl information literacy competency standards for higher education. a seventh section in both tutorials administered a voluntary survey. six knowledge-based survey questions tested students’ abilities on the six skills covered in the tutorials. three affective questions asked students to use a four-point likert scale to report ease (1 = very easy, 4 = very difficult), clarity (1 = very clear, 4 = very unclear), and convenience (1 = very convenient, 4 = very inconvenient) of six research skills, including: identifying keywords and main concepts in a topic, identifying scholarly versus non-scholarly sources, finding relevant scholarly articles, locating a book’s call number in the library catalog and on the shelf, finding newspaper articles, and constructing an annotated bibliography. two affective survey questions asked students to use a four-point likert scale (1 = very significant increase, 4 = no increase) to rate the impact the tutorial had on their knowledge of and satisfaction with using the library in each of the six areas of research.   main results – the overall response patterns for the six information literacy knowledge-based questions were similar for both groups. students who viewed the libguides tutorial performed better than the web page group on four of the six knowledge-based questions. the web page group performed better than the libguides group on two of the six knowledge-based questions. across the board, students performed poorly on the first question, which measured students’ abilities to form a search string (39.2% correct in the libguides group; 25.7% correct in the web page group), and on the fifth question which asked students to identify the best source of current information from a list of resources (32% correct in the libguides group; 17% correct in the web page group).   response means on the first three affective questions indicate that students in both groups found searching for relevant scholarly articles and constructing an annotated bibliography to be more difficult than the other four skills. additionally, students in the libguide group reported slightly higher means than the web page group concerning the clarity of finding newspaper articles, and were therefore less clear on the task. students in the web page group reported slightly higher means than the libguide group when reporting the convenience of constructing an annotated bibliography, suggesting they found creating a bibliography more inconvenient. students in both groups also responded similarly to the final two affective questions measuring the perceived impact the tutorial had on their knowledge of and satisfaction with using library resources.   conclusion – the author concludes that there is no evidence of a pedagogical advantage for either the libguide or web page information literacy tutorials. students’ poor performance on the first knowledge-based question led the author to revise the tutorial content in order to emphasize matching a search strategy to the research topic. responses to the fifth question resulted in modifying the survey question to emphasize the importance of selecting a current source of information and to deemphasize format. the author suggests revising the tutorials to include a site map and reorder the materials, as well as pretesting survey questions and collecting data across multiple semesters to assess the validity and reliability of the survey instrument. libguides is recognized as a platform that reduces barriers to creating online learning materials with a pedagogical value similar to other web-based tutorials.   commentary   cited as a “cms for busy librarians” (verbit & kline, 2011), springshare’s libguides remains a popular option for librarians looking to create subject guides, course guides, and other online learning materials. librarians have created over 400,000 libguides (springshare, 2014), but there is little literature addressing the efficacy of such guides as learning objects. the current study seeks to fill this gap.   a review of the study using the reliant critical appraisal instrument for educational interventions (koufogiannakis, booth, & brettle, 2006) suggests questions remain about the study’s intervention, population, and interpretation of results. the study’s literature review emphasizes information literacy tutorial assessment, libguides popularity, and the lack of literature related to libguides’ pedagogical value. a review of findings from libguides’ usability literature (e.g., hintz et al., 2010; sonsteby & dejonghe, 2013) would add theoretical support for the study intervention and the hypothesis that information architecture differences between the two platforms affect student performance.   the study addresses limitations related to response rate and generalizability, yet states the respondents “were not likely to be significantly different from their peers who did not complete the survey” (p. 165). only respondent demographics are known; there is no comparable information for sample demographics, and it is difficult to determine whether the respondents reflect the rest of the sample. although survey questions underwent librarians’ review, pretesting with students (as noted in the study’s “limitations and further directions”) will strengthen survey design and allow for establishing the instrument’s validity and reliability. alternatively, rather than relying on an additional survey, collecting and analyzing the worksheets that accompanied the tutorials might increase response rates and provide an opportunity for authentic assessment.   there is little discussion of student groups’ comparative performances, even though libguides students outperformed on four of six knowledge-based questions. the performance differences appear numerically small, and no analyses are reported to determine whether these differences are statistically significant. student performance on knowledge-based questions is rarely attributed to the tutorial design, as hypothesized in the study’s aim. for instance, with regard to student responses on knowledge-based question number five, the study concludes, “[i]t is unclear what caused this difference between the two versions of the assignment…” (p. 164) even though the study hypothesized that presentation differences between the libguide and the web page tutorial may lead to differing student performances.   while the study’s limitations make the evidence difficult to interpret, the lack of meaningful differences between the libguide and web page tutorial suggests that librarians may feel comfortable relying on a librarian-friendly cms to create information literacy tutorials. while no pedagogical advantage was revealed, there is also no evidence to suggest the additional time or expertise required to build the web page version of this tutorial resulted in benefits to students.   references   hintz, k., farrar, p., esghi, s., sobol, b., naslund, j., lee, t., …& mccauley, a. (2010). letting students take the lead: a user-centred approach to evaluating subject guides. evidence based library and information practice, 5(4), 39-52. retrieved from http://ejournals.library.ualberta.ca/index.php/eblip/     koufogiannakis, d., booth, a., & brettle, a. (2006). reliant: reader’s guide to the literature on interventions addressing the need of education and training. library and information research, 30(94), 44-51. retrieved from http://www.lirgjournal.org.uk/lir/ojs/index.php/lir/index   springshare. (2014). libguides. retrieved from http://www.springshare.com/libguides   sonsteby, a., & dejonghe, j. (2013). usability testing, user-centered design, and libguides subject guides: a case study. journal of web librarianship, 7, 83-94. doi:10.1080/19322909.2013.747366   verbit, d., & kline, v. l. (2011). libguides: a cms for busy librarians. computers in libraries, 31(6), 21-25. retrieved from http://www.infotoday.com/cilmag/ciltop.htm   article   understanding patterns of library use among undergraduate students from different disciplines   ellen collins research consultant research information network london, united kingdom email: ellen.collins@researchinfonet.org   graham stone information resources manager university of huddersfield huddersfield, united kingdom email: g.stone@hud.ac.uk   received: 17 jan. 2014    accepted: 16 july 2014      2014 collins and stone. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to test whether routinely-generated library usage data could be linked with information about students to understand patterns of library use among students from different disciplines at the university of huddersfield. this information is important for librarians seeking to demonstrate the value of the library, and to ensure that they are providing services which meet user needs. the study seeks to join two strands of library user research which until now have been kept rather separate – an interest in disciplinary differences in usage, and a methodology which involves large-scale routinely-generated data.   methods – the study uses anonymized data about individual students derived from two sources: routinely-generated data on various dimensions of physical and electronic library resource usage, and information from the student registry on the course studied by each student. courses were aggregated at a subject and then disciplinary level. kruskal-wallis and mann whitney tests were   used to identify statistically significant differences between the high-level disciplinary groups, and within each disciplinary group at the subject level.   results – the study identifies a number of statistically significant differences on various dimensions of usage between both high-level disciplinary groupings and lower subject-level groupings. in some cases, differences are not the same as those observed in earlier studies, reflecting distinctive usage patterns and differences in the way that disciplines or subjects are defined and organised. while music students at huddersfield are heavy library users within the arts subject-level grouping arts students use library resources less than those in  social science disciplines, contradicting findings from studies at other institutions, computing and engineering students were relatively similar, although computing students were more likely to download pdfs, and engineering students were more likely to use the physical library.    conclusion – the technique introduced in this study represents an effective way of understanding distinctive usage patterns at an individual institution. there may be potential to aggregate findings across several institutions to help universities benchmark their own performance and usage; this would require a degree of collaboration and standardisation. this study found that students in certain disciplines at huddersfield use the library in different ways to students in those same disciplines at other institutions. further investigation is needed to understand exactly why these differences exist, but some hypotheses are offered.   introduction   libraries and librarians have often been accused of deciding on what’s best for the user without consultation (wells, 1996; wilson, 2000; tilley, 2013). “one of the most complex issues to deal with in acquiring knowledge about students is concerned with the assumptions library staff make about student behaviour” (tilley, 2013, p.84).   however, in times of austerity in higher education funding, increased competition for financial resources within a university as well as increased competition between universities this approach is no longer adequate. simply counting data, such as anonymized usage statistics, or assuming that librarians and libraries know ‘best’ is no longer enough. libraries must justify both their value and impact to university senior management and to the student body who want to see their fees are invested in services that will add value to their studies. however, as oakleaf suggests, “librarians can develop systems that will allow data collection on individual user library behaviour”… …“until librarians do that, they will be blocked in many of their efforts to demonstrate value” (oakleaf, 2010, p.96).   one important aspect of this work is recognizing different patterns of usage among different groups of library patrons. we have long known that information behaviours are very different in different disciplines (covi, 1999; whitmire, 2002). in order to develop services which meet these different needs, and to thereby show that the library has value, librarians must first understand patterns of need and usage among different groups.   the first stage of the library impact data project (lidp), based at the university of huddersfield, established that a statistically significant relationship existed across a number of uk universities between library activity data and student attainment (stone & ramsden, 2013). the second phase of the project looked at the data in more detail to establish whether there is a relationship between subject discipline and undergraduates’ use of academic libraries. the paper will outline the methodology of the research and present findings that show that there is a statistically significant difference between various disciplines on several different dimensions of physical and electronic library usage. the paper concludes with a discussion of the findings and recommendations for further study.   literature review   the literature shows a longstanding interest in the differences between disciplines, and how these affect the way students and researchers use the library. a large number of approaches, methodologies, and definitions were used in order to try to understand the answer to this question. studies have used surveys, both purpose-built (chrzastowski & joseph, 2006; housewright, schonfeld, & wulfson, 2013) and re-analysis of pre-existing responses (whitmire, 2002), case studies (meyer et al., 2011; bulger et al., 2011), or a combination of the two (maughan, 1999) to try to understand disciplinary differences. the specific definitions of disciplines have been shaped to fit the needs of research methods or of organisational structures. for example, the case-study approach adopted by both meyer et al. (2011) and bulger et al. (2011) demanded an intense focus on very small and tightly-defined groups of researchers, while housewright et al.’s 2013 survey used high-level categories to define disciplines in order to permit statistical analysis. chrzastowski and joseph (2006) use high-level categories in order to fit with their university’s organisational structure, but whitmire (2002) is forced to exclude the life scientists at her institution from her analysis, because the theoretical structure of the study does not allow for them. studies have also looked at different groups of library users: undergraduates (wells, 1996; bridges, 2008; cox & jantti, 2012), postgraduates (chrzastowski & joseph, 2006), and researchers at all stages of their careers (meyer et al., 2011; bulger et al., 2011; housewright et al., 2013; tenopir & volentine, 2012). finally, they have adopted various definitions of what constitutes library use – from gate entries to e-resource usage, book borrowing to searching behaviours – to explore how different groups engage with the library and its services.   the differences in methodology and approach limit librarians’ ability to make use of the findings in their own context. in some cases, findings are relatively consistent across studies: for example, arts and humanities are usually found to be the biggest users of library materials (de jager, 2002; maughan, 1999; whitmore, 2002). nackerud et al. (2013) found, at a more granular level, that college of design undergraduates were the highest library borrowers in their study. but in other instances, different ways of defining subjects and user groups can lead to confusion in understanding exactly how findings may apply in other settings. for example, many studies found engineering students to be the least engaged library users across resources (kramer & kramer, 1968; bridges, 2008; cox & jantti, 2012; nackerud et al., 2013). however chrzastowski and joseph (2006) found that graduate students from the physical sciences and engineering used online resources more than graduates in other disciplines. this study looks at a smaller group of students (graduate students only) but across a bigger selection of disciplines (physical sciences and engineering). how is a reader to tell which change has made the difference, or whether there is something inherent to the university of illinois at urbana-champaign, where their study was carried out, that is affecting the results?   in recent years, a new group of studies have begun to take a more data-driven approach to understanding library usage, deriving value from data that is routinely generated by people who use the library – gate entries or e-resource logins for example (jisc, 2012). this data is then linked with information from student registry or central administration systems, including degree classifications, demographic characteristics, and discipline. the advantage of this methodology is twofold. first, unlike survey or interview-based studies, it does not rely upon self-reported data to understand the phenomenon being investigated. second, it can capture data for every student in the institution, removing the possibility of bias on the part of either the researcher or the survey respondents.   most studies using this methodology were directed towards understanding the relationship between student library usage and degree result, usually in order to engage university management with the importance and value of the library. so, for example, wong and webb (2011), cox and jantti (2012), stone and ramsden (2013), and soria, fransen, and nackerud (2013) have looked at various measures of library usage to understand their relationship with final degree outcome. all of these studies have demonstrated a statistically significant relationship, though they hold back from inferring what kinds of cause and effect mechanisms may be at work.   some of these studies have begun to incorporate other variables into their work such as the demographic characteristics of library users (stone and collins, 2013). other studies have looked specifically at usage by discipline. nackerud et al. (2013) showed use by college of all types of library use, finding, for example, that 100% of pharmacy students visited the library in one semester. nonetheless, much of this work continues to examine usage in the context of attainment. jantti and cox (2013) broke down their analysis by department in order to show that the science faculty got the most academic benefit from books and electronic resources, while health and behavioural sciences obtained the least academic benefit from books, and creative arts the least from electronic resources. while very informative for librarians seeking to demonstrate the impact of their work, this analysis does not provide information to identify how different groups use the library.   this study attempts to fill a hole in the literature by using routinely-generated data to understand different usage patterns across disciplines within a single institution. studies based upon a survey methodology do not typically achieve high response rates: 14% in the case of chrzastowski and joseph (2006) and 7.8% in the case of housewright et al. (2013). there can also be problems around recollection: tenopir and volentine (2012) deal with this through a critical incident technique which asks about the last time the respondent used the library in a particular way, but this relies upon large numbers of respondents. case study techniques, while providing considerable depth of understanding, have similar problems around recall, and cannot always be generalised to wider communities of interest. using routinely-generated data circumvents the problems of generalizability and recall, and presents an interesting opportunity to understand exactly how students at a particular institution use their library.   aims   this study explores how full-time undergraduate students in a range of disciplines at the university of huddersfield use the library and information resources. the aims are twofold: first, to explore whether routinely-generated usage data can be used to provide an insight into working patterns, and second, to analyze the different patterns of usage to inform librarian practice and the support services offered to students.   methods   there were two sources of data for this analysis. the first was data that are routinely generated when students use huddersfield’s physical or electronic library resources, such as library gate entries, logins to e-resources, or hours spent on library computers. e-resource data do not relate to a specific resource used, but that the student logged into a database. this methodology was also used by the minnesota study (nackerud et al., 2013). the second were data from huddersfield’s student registry, such as information on demographic characteristics, course and mode of study, and final degree result (where available). these datasets were amalgamated using unique identifiers and then anonymized.   both datasets underwent considerable processing before analysis could be undertaken. only full-time undergraduate students based at huddersfield’s main campus were included. the usage data were restructured to create new variables that permitted more sensitive analysis. for example, the data on e-resource logins were aggregated to give the hours spent logged into e-resources, counting the number of hours in a year when students logged into e-resources at least once.   the analysis method required the 105 full-time undergraduate courses offered by huddersfield at the time of the research to be grouped into a small number of categories; ideally no more than six. upon discussion with project stakeholders, we established that in doing this we would lose a great deal of detail and produce findings that, while useful, would be too broad an approach. to permit both rigorous analysis and useful outputs we adopted a two-tier approach; grouping courses into subject-level groups, and then aggregating these subject groups into higher-level disciplinary groupings. we could then compare subject groups within each disciplinary grouping, and also compare the disciplinary groupings for some high-level results. note that it is not possible to compare subjects from different disciplinary groupings using the results we have provided here.   these groupings reflect the distribution of students and courses within huddersfield and were determined by library staff. in some cases, only a top-level disciplinary grouping exists, because there is no logical way to subdivide into smaller groups – usually because huddersfield does not offer many courses in this area. universities wishing to replicate this study will need to identify a disciplinary structure which suits the profile of courses at their institution.   complete lists of library usage variables and their definitions are shown in table 1. a list of disciplines and their respective student enrolment by course is shown in table 2.   the data were analyzed using spss. they were tested for normality and found to be non-normal. we therefore used kruskal-wallis and mann-whitney tests to establish whether a relationship existed between discipline and the usage variables. on disciplinary groups with three or more variables, we used an initial kruskal-wallis test to identify whether a statistically significant difference existed followed by mann-whitney tests to identify which variables differed from each other. a bonferroni correction was applied to these mann-whitney tests to compensate for the increased chance of type 1 errors from multiple mann-whitney tests. for groups with two variables, we simply used the mann-whitney test.   for tests with six or more groups, we used a control group in our second stage of testing (the mann-whitney tests). this was to ensure we did not have an unacceptably small p value for the significance testing, following the bonferroni correction. in each case, we selected the largest group as our control, in order to identify differences from the majority which might not be noticed by librarians in their day-to-day work. at the disciplinary level, social sciences was selected as the control as it was the largest group (contained the highest number of students). there was no need to use a control group for any of the subject-level analysis as these all contained five or fewer groups.   throughout our analysis, we have followed cohen (1992) in classifying effect sizes:   .1 – small effect .3 – medium effect .5 – large effect     table 1 library usage definitions variable definition number of items borrowed items checked out from the library; not limited to books number of library visits measured via gate entries – all students must swipe their id card to enter the library, this data is recorded on library systems hours logged into library pc number of hours in a year in which a student was logged into a library pc (maximum possible number of pc hours per year is 8, 760 = 24 hours x 365 days). multiple logins within a single hour on a single day are not counted hours logged into e-resources number of hours in a year in which a student was logged into e-resources, both on-site and remote logins (maximum possible number of e-resource hours per year is 8,760 = 24 hours x 365 days). multiple logins within a single hour on a single day are not counted number of pdf downloads   total number of e-resources accessed the number of different e-resources accessed both on-site and through remote logins. within huddersfield’s data, a single e-resource varies from an individual journal subscription to a large multi-journal platform or database, so this data must be treated with some caution number of e-resources accessed 5 or more times   number of e-resources accessed 25 or more times     table 2 course enrolment discipline subject number of students science science 30 discipline total 30 health health 138 discipline total 138 computing and engineering computing 74 engineering 43 discipline total 257 arts music 74 architecture 59 fashion 130 2d design 29 3d design 47 discipline total 339 humanities english 70 drama 41 media and journalism 111 discipline total 222 social sciences business, management and accountancy 352 law 60 behavioural sciences 236 social work 85 education 70 discipline total 803                                                           results   table 3 shows the median values for each measure of library usage at the discipline level. table 4 shows the effect sizes, in a range from 0 to 1, and the statistical significance of mann-whitney tests on each measure when comparing the discipline to the control group of social sciences. social sciences has been used as a control because it is the largest disciplinary group (containing the highest number of students). a light grey cell indicates that usage in the group under examination was lower than in the control group of social sciences, while a darker grey cell indicates that it was higher than the control group. cells that have no highlighting indicate no significant difference between the group and the control group. all results are significant at the .005 level, which is the value generated by the bonferroni correction for a .05 significance level.   table 4 shows that students within the social science grouping are, in most respects, significantly higher users of library content and resources than any other disciplinary grouping. arts students are the lowest users, with a large effect size for the number of pdf downloads, and medium effect sizes for most of the variables associated with e-resource use. the courses which make up arts disciplines may explain this lower level of usage. many of them rely upon visual or audio content rather than the journal articles available via huddersfield’s e-resources.   tables 5 and 6 show the breakdown of the arts group in more detail. in this case, we compared all of the groups against each other, so table 6 is slightly more complex. the top line shows the two groups that we are comparing, and the letter in the cell indicates which group was higher as per the key below the figure. as before, a blank cell indicates no significant difference between the two groups. all results are significant at the .001 level, which is the value generated by the bonferroni correction for a .05 significance level.     table 3 median values for library usage measures, by discipline library usage measure  number of items borrowed number of library visits hours logged into library pc hours logged into e-resources number of pdf downloads number of e-resources accessed number of e-resources accessed 5 or more times number of e-resources accessed 25 or more times science 14.0 180.5 11.5 16.0 32.0 11.0 1.5 0.0 computing and engineering 10.0 48.0 4.0 6.0 10.0 6.0 0.0 0.0 arts 29.0 132.0 18.0 5.0 1.0 5.0 0.0 0.0 humanities 43.0 116.5 16.0 28.5 46.0 14.0 3.0 0.0 health 57.5 111.5 13.0 47.0 111.5 26.5 6.0 0.0 social sciences 43.0 112.0 16.0 26.0 47.0 14.0 2.0 0.0   table 4 effect sizes and statistical significance of mann-whitney tests by discipline library usage measure number of items borrowed number of library visits hours logged into library pc hours logged into e-resources number of pdf downloads number of e-resources accessed number of e-resources accessed 5 or more times number of e-resources accessed 25 or more times science .232               computing and engineering .337 .214 .106   .283 .281 .272 .157 arts .193     .435 .559 .485 .432 .183 humanities   .113 .064   .138     .087 health .064 .295 .147   .057 .114   .147   table 5 median values for library usage measures for arts discipline, by subject library usage measure number of items borrowed number of library visits hours logged into library pc hours logged into e-resources number of pdf downloads number of e-resources accessed number of e-resources accessed 5 or more times number of e-resources accessed 25 or more times music 107.0 162.0 10.5 17.5 5.0 8.0 1.0 0.0 architecture 26.0 81.0 21.0 12.0 18.0 10.0 1.0 0.0 fashion and textiles 21.0 124.5 18.0 2.0 0.0 2.0 0.0 0.0 2d design 2.0 162.0 42.0 4.0 0.0 2.0 0.0 0.0 3d design 43.0 164.0 18.0 8.0 1.0 5.0 0.0 0.0   table 6 effect sizes and statistical significance of mann-whitney tests in arts discipline, by subject* library usage measure number of items borrowed number of library visits hours logged into library pc hours logged into e-resources number of pdf downloads number of e-resources accessed number of e-resources accessed 5 or more times number of e-resources accessed 25 or more times music /architecture .646 (m) .434 (m)   .300 (m)     .322 (m) .256 (m) music /fashion .524 (m)       .315 (m) .292 (m) .248 (m)   music /2d design .621 (m)     .361 (m) .293 (m) .322 (m) .401 (m) .363 (m) music /3d design .676 (m)   .280 (3d) .430 (m) .488 (m) .427 (m) .428 (m) .316 (m) architecture /fashion   .352 (f)             architecture /2d design   .328 (2d)             architecture /3d design         .324 (3d) .299 (3d)     fashion /3d design   .363 (f)             *music (m); architecture (a); fashion (f); 2d design (2d); 3d design (3d)     clearly, music dominates usage against all other subjects on a number of variables and, in relation to the number of items borrowed, with a large effect size. this may be because the music subject group includes some courses that might have fitted alongside english or drama in the humanities group, as well as some that are more technology-focused and rightly belong in the arts group. it is also worth noting that fashion students visit the library frequently; this may be because they are making extensive use of the art and design resource area which has traditionally been strong in their discipline. architects have a separate resource area outside the library, which may explain their lower levels of usage. we found no statistically significant differences in usage when comparing 2d design with fashion and with 3d design.   table 7 shows the breakdown for subject groups within the social science discipline, and table 8 shows the results of the statistical tests. again, all the groups are compared with each other. all results are significant at the .001 level, which is the value generated by the bonferroni correction for a .05 significance level.   many of the effect sizes in this group are large, indicating very different patterns of usage between subjects. overall, students in behavioural sciences tend to show the highest usage on most measures, when compared to other subjects. business students have higher usage than law, social work, and education students on several dimensions but not on the number of items borrowed, which is consistently lower (and with a large effect size). lawyers are extremely low users of library resources, particularly e-resources; we hypothesize that this may be because, more than any other discipline, they rely upon a few core texts which they purchase for themselves. we observed no difference in usage for social work and education, which may reflect a similarity in how these two groups of vocational courses are taught.     table 7 median values for library usage measures for social sciences discipline, by subject library usage measure number of items borrowed number of library visits hours logged into library pc hours logged into e-resources number of pdf downloads number of e-resources accessed number of e-resources accessed 5 or more times number of e-resources accessed 25 or more times business 26.0 113.0 17.0 33.0 74.5 13.5 3.0 0.0 law 24.0 159.5 25.5 0.0 0.0 0.0 0.0 0.0 behavioural sciences 89.0 132.5 22.0 34.5 74.0 18.0 3.0 0.0 social work 81.0 74.0 8.0 18.0 29.0 16.0 1.0 0.0 education 72.0 76.5 4.0 21.0 42.0 17.0 2.0 0.0   table 8 effect sizes and statistical significance of mann-whitney tests in social sciences discipline, by subject* library usage measure number of items borrowed number of library visits hours logged into library pc hours logged into e-resources number of pdf downloads number of e-resources accessed number of e-resources accessed 5 or more times number of e-resources accessed 25 or more times business /law       .477 (b) .456 (b) .459 (b) .421 (b) .200 (b) business /behavioural sciences .590 (bs)         .175 (bs)     business /social work .409 (sw) .264 (b) .185 (b) .155 (b) .168 (b)   .139 (b)   business /education .405 (e) .154 (b) .177 (b)           law /behavioural sciences .537 (bs)     .573 (bs) .549 (bs) .576 (bs) .477 (bs) .188 (bs) law /social work .642 (sw) .354 (l) .265 (l) .636 (sw) .626 (sw) .679 (sw) .565 (sw) .257 (sw) law /education .715 (e)   .276 (l) .744 (e) .713  (e) .775 (e) .724 (e) .358 (e) behavioural sciences/ssocial work   .358 (bs) .220 (bs)           behavioural sciences /education   .213 (bs) .219 (bs)         *business (b); law (l); behavioural sciences (bs); social work (sw); education (e)     the computing and engineering subgroups had very few differences between them. computing students were more likely to visit the library (median = 61.0, r=.362) and spent more hours logged into the library pcs (median = 8.0, r=.235). we think that this may be because computing students are more likely to use their own personal computing equipment, compared to the engineers.   among the humanities subgroups, there were no statistically significant differences in usage between students on the english and drama courses. however, both groups showed higher levels of usage than media students on most of the e-resource dimensions, english students with slightly bigger effect sizes. this probably reflects the way that the courses are taught, and in particular the importance of written texts and criticisms to english and drama students.   discussion   our results demonstrate the value of a data-driven approach for librarians seeking to understand usage patterns among library users from different disciplines. comparing our findings to previous studies, several disparities appear. arts and humanities students are not particularly heavy library users, as they have been found to be in earlier work (de jager, 2002; nackerud et al., 2013; maughan, 1999); in fact, they are lower users than social scientists on most dimensions. earlier research found computing and engineering students to be relatively low users of library resources (kramer & kramer, 1968; bridges, 2008; cox & jantti, 2012; nackerud et al., 2013), although chrzastowski and joseph (2006) found that graduate students from the physical sciences and engineering used online resources more than graduates in other disciplines. again, our results show that although students from the computing and engineering discipline are low users (relative to the control group of social sciences), they are not particularly different from some other disciplines, such as arts, in this respect.    this study was also able to show quite nuanced differences in library usage within the high-level subject groupings. this information, for example – showing the high usage level of musicians compared to other “arts” subjects, or the strong usage by behavioural scientists compared to other social science groupings – helps librarians develop a more realistic understanding of how students use resources and to target areas of particularly low uptake which may be masked by the behaviour of bigger groups within a subject. this is a distinct advantage of this methodology over earlier survey-based methodologies, where response numbers were too small to permit statistical analysis at this level of granularity.   findings from this phase of lidp regarding subject disciplines gives the library evidence that a one size fits all approach, such as information literacy sessions could be enhanced by intelligence from library analytics. for example, known “low-use” subjects could be targeted differently from known “high use” subjects in order to give a more personalized boutique service to the end user. this addresses one of tilley’s (2013) success factors of the boutique model, “[k]nowledge of users’ needs and activity-their preferences, the irritants-and their methods of working” (p.82). however, using library analytics and making the assumption that increased use of library resources may lead to increased achievement, knowledge of subject cohorts methods of working could be used to guide them to appropriate resources.   of course, this methodology retains some limitations. the usage measures are very accurate representations of student behaviour but we must be cautious about how we interpret them. for example, we cannot claim that students only entered the library in order to study, as other student services were also located there at the time of the study; gate entries recorded by library systems might represent students seeking help with issues completely outside the library. interestingly, previous research indicated that gate entries are one of the library usage measures that are not correlated with student outcomes (stone & ramsden, 2013).   we must also be cautious about over-interpreting why usage patterns look the way they do. qualitative methodologies are more useful in understanding this kind of issue. face to face discussions with the cohort provides a much richer seam of information. tilley explores this in her discussion on the knowledge about english students at the university of cambridge and the implications for the library service. but library analytics can help to identify the “context” that tilley (2013) describes, which, “… allows us to prioritize areas of our service for improvement” (p. 91). this is also supported by poll (2012) who suggests a mixed methods approach as the most effective way of exploring library impact. at huddersfield, this mixed methods approach has been adopted and used to support the findings of the lidp. towards the end of the study, a focus group was held with a cohort of computing students – a cohort that had been identified as low users in the study. this proved valuable as a way to evidence the data from the project in a real life situation, where students could explain their reasons for library use. as tilley (2013) states, this should not be a one off conversation, but the beginning of frequent knowledge collection.   there have also been two spin off projects at huddersfield that were heavily influenced by the study. the first is the ‘roving librarian’ project, which was being piloted at the time of the study, and was continued using the findings of lidp in order to target areas of low use. “the statistics gathered showed that many students are not using our resources…” therefore the roving librarian project extended its roving “… to take it to social spaces and resource centers within all schools to reach students who may otherwise be library non-users.” (sharman & walsh, 2012) the other project to come out of lidp was lemontree (running in the halls, 2012), which was designed to be a fun, innovative, low input way of engaging students through new technologies and increasing use of library resources. when registering for lemontree, students sign terms and conditions that allow their student number to be passed to computing and library services (cls), which allows cls to track usage of library resources by lemontree gamers versus students who do not take part. this study only planned to come up with a proof of concept, however, over 850 users registered by october 2012, thus providing a solid base for further analysis in order to establish whether intervention using gamification can have an impact throughout a student’s academic course. since completion of the study, lemontree, now known as librarygame (running in the halls, 2013), is being used by the universities of huddersfield, glasgow, and manchester.   just as this study identifies findings that contradict earlier research, we would not expect that the findings at huddersfield will necessarily translate into other institutions. the subject groupings reflect huddersfield’s structure and strengths, and may not be typical of other universities in the u.k., let alone in the wider higher education sector. the specificity which makes our findings so useful at huddersfield make them much less useful to other institutions, and mean that it can be rather difficult to benchmark the library’s strengths and weaknesses against comparable institutions, or to aggregate data to get a better picture of usage patterns across institutions (a strength of the first phase of the work, which worked with eight institutions altogether) (stone & ramsden, 2013). with this in mind, towards the end of the study, the project collaborated with colleagues at mimas (2013) to produce a library analytics survey in order to assess the demand for a national library analytics tool. the survey found that 94.6% of those who replied wanted to benchmark their data with other institutions and that 87.7% were interested in the richer data that was used as part of this study (showers & stone, 2014). as a result of the lidp findings and the lidp-copac survey, jisc have commissioned a new project, the library analytics and metrics project (jisclamp), which in 2013 produced, “a prototype shared library analytics service for u.k. academic libraries” (jisc, 2013).   conclusions   this study examined whether large datasets could be used to understand disciplinary differences in student library usage. it used statistical analysis to explore routinely-generated data from the university of huddersfield’s library, linked to information about students from the student registry.   this technique revealed significant differences among groups of students and found that these differences were not always the same as those identified by previous studies. in doing so, it demonstrated the value of undertaking this analysis on an institution-by-institution basis in order to avoid developing services based upon information from other universities or studies which may not reflect usage patterns across all institutions.   unlike more qualitative methods, the technique is unable to say much about why these different usage patterns exist. however, findings could be followed up with focus groups or interviews with the groups of students in question, in order to gain a greater depth of understanding.   the jisc-funded (2013) library analytics and metrics project (lamp) is an interesting attempt to automate this analytics service for libraries that are able to supply the relevant data; it also offers opportunities to develop standardised definitions for subject, ethnicity, country of residence, and other demographic variables so that they can analyse their data on their own terms or compare it against other institutions. in 2014 lamp produced an “ugly prototype”, which was able to manipulate the raw data from this study and other partner institutions (showers, palmer & stone, 2014). lamp has now received additional funding to produce a shared service for the u.k., which will enable libraries to submit their own data for analysis, which will include statistical significance testing. this will allow follow up research to be conducted by libraries that join the service.   both phases of the lidp have produced toolkits to aid institutions wishing to collect and analyze their own data (stone and collins, 2012; stone, ramsden & pattern, 2011), in addition a value impact starter kit (oakleaf, 2012) comprising 52 exercises for librarians, an outcome of the value of academic libraries project (oakleaf, 2010), is also available. the lamp project is also considering a toolkit approach in order to address concern over the level of statistical knowledge required by users in order to interpret the outputs of the system. one possible outcome would be to collaborate with oakleaf on a new toolkit and initial discussions are underway.   at the university of huddersfield, discussions are now underway to consider how the results of the study can be used to improve the student experience. now that the library can evidence the results of the study, a set of briefing papers are planned for specific subject areas that shows the evidence in areas that relate specifically to academic staff it was decided at an early stage that low usage is not acceptable in any discipline. furthermore, longitudinal data is required to look at usage over time so that the library can start to benchmark and show whether interventions have made a difference.     references   bridges, l. m. (2008). who is not using the library? a comparison of undergraduate academic disciplines and library use, portal. libraries and the academy, 8(2), 187–196. http://dx.doi.org/10.1353/pla.2008.0023     bulger. m., meyer, e. t., de la flor, g., terras, m., wyatt, s., jirotka, m., eccles, k., & madsen, c. (2011). reinventing research? information practices in the humanities. london, research information network. retrieved from http://www.rin.ac.uk/our-work/using-and-accessing-information-resources/information-use-case-studies-humanities   chrzastowski, t. e., & joseph, l. (2006). surveying graduate and professional students’ perspectives on library services, facilities and collections at the university of illinois at urbana-champaign: does subject discipline continue to influence library use? issues in science and technology librarianship, 45(winter). retrieved from http://www.istl.org/06-winter/refereed3.html     cohen, j. (1992). a power primer, psychological bulletin, 112(1), 155-159. doi:10.1037/0033-2909.112.1.155   covi, l.m. (1999). material mastery: situating digital library use in university research practices. information processing & management, 35(3), 293-316. http://dx.doi.org/10.1016/s0306-4573(98)00063-6     cox, b.l., & jantti, m. (2012). capturing business intelligence required for targeted marketing, demonstrating value, and driving process improvement. library and information science research, 34(4), 308-316. http://dx.doi.org/10.1016/j.lisr.2012.06.002   de jager, k. (2002). impacts and outcomes: searching for the most elusive indicators of academic library performance. in meaningful measures for emerging realities, proceedings of the 4th northumbria international conference on performance measurement in libraries and information services. washington, dc, association of research libraries. retrieved from http://www.libqual.org/documents/admin/4np_secure.pdf   housewright, r., schonfeld, r. c., & wulfson, k. (2013). ithaka s+r, jisc, rluk, uk survey of academics 2012. retrieved from http://www.ulrls.lon.ac.uk/wesline/documents/uk_survey_of_academics_2012.pdf      jantti, m., & cox, b. (2013). measuring the value of library resources and student academic performance through relational datasets. evidence based library and information practice, 8(2), 163-171. retrieved from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/19574/15217   jisc (2012). activity data: delivering benefits from the data deluge. retrieved from http://www.jisc.ac.uk/publications/reports/2012/activity-data-delivering-benefits.aspx   jisc (2013). library analytics and metrics project. retrieved from http://jisclamp.mimas.ac.uk/   kramer, l. a., & kramer m. b. (1968). the college library and the drop-out. college and research libraries, 29(4), 310-12. retrieved from http://crl.acrl.org/content/29/4/310   maughan, p. d. (1999). library resources and services: a cross-disciplinary survey of faculty and graduate student use and satisfaction. journal of academic librarianship, 25(5), 354-66. http://dx.doi.org/10.1016/s0099-1333(99)80054-8   meyer, e. t., bulger, m., kyriakidou-azcharoudiou, a., power, l., willians, p., venters, w., terras, m., & wyatt, s. (2011). collaborative yet independent: information practices in the physical sciences. london, research information network. retrieved from http://www.iop.org/publications/iop/2012/page_53560.html     mimas. (2013). copac collections management. retrieved from http://copac.ac.uk/innovations/collections-management/   nackerud, s., fransen, j., peterson, k., & mastel, m. (2013). analyzing demographics: assessing library use across the institution. portal: libraries and the academy, 13(2), 131-145. retrieved from http://www.press.jhu.edu/journals/portal_libraries_and_the_academy/portal_pre_print/current/articles/13.2nackerud.pdf   oakleaf, m. (2010). the value of academic libraries, chicago, il., association of college & research libraries. retrieved from http://www.ala.org/ala/mgrps/divs/acrl/issues/value/val_report.pdf     oakleaf, m. (2012). academic library value: the impact starter kit. syracuse, ny: della graphics, 2012.   poll, r. (2012). can we quantify the library's influence? creating an iso standard for impact assessment. performance measurement and metrics, 13(2), 121–130. http://dx.doi.org/10.1108/14678041211241332     running in the halls (2012). lemontree. retrieved from http://librarygame.co.uk/      running in the halls (2013). librarygame. retrieved from https://library.hud.ac.uk/lemontree/      sharman, a., & walsh, a. (2012). roving librarian at a mid-sized, uk based university. library technology reports, 48 (8). 28-34. http://dx.doi.org/10.5860/ltr.48n8      showers, b., palmer, j., & stone, g. (2014). jisclamp: shining a light on our analytics and usage data. paper presented at uksg 37th annual conference and exhibition, hic, harrogate. retrieved from http://eprints.hud.ac.uk/19260/       showers, b., & stone, g. (2014). safety in numbers: developing a shared analytics service for academic libraries. performance measurement and metrics, 15(1/2). 13-22. http://dx.doi.org/10.1108/pmm-03-2014-0008     soria, k. m., fransen, j., & nackerud, s. (2013). library use and undergraduate student outcomes: new evidence for students' retention and academic success. portal: libraries and the academy, 13(2), 147-164. retrieved from http://www.press.jhu.edu/journals/portal_libraries_and_the_academy/portal_pre_print/current/articles/13.2soria.pdf   stone, g., & collins, e. (2012). library impact data project toolkit: phase 2. manual. university of huddersfield, huddersfield. retrieved from http://eprints.hud.ac.uk/16316/   stone, g., & collins, e. (2013). library usage and demographic characteristics of undergraduate students in a uk university. performance measurement and metrics, 14(1), 25-35. http://dx.doi.org/10.1108/14678041311316112   stone, g., & ramsden, b. (2013). library impact data project: looking for the link between library usage and student attainment. college and research libraries, 74(6), 546-559. retrieved from http://crl.acrl.org/content/74/6/546.abstract   stone, g., ramsden, b., & pattern, d. (2011). library impact data project toolkit. manual. university of huddersfield, huddersfield. retrieved from http://eprints.hud.ac.uk/11571/   tenopir, c., & volentine, r. (2012). uk scholarly reading and the value of library resources: summary results of the study conducted spring 2011, knoxville, university of tennessee, center for information and communication studies. retrieved from http://www.jisc-collections.ac.uk/reports/ukscholarlyreadingreport/   tilley, e. (2013). personalized boutique service: critical to academic library success? new review of academic librarianship, 19(1), 78-97. http://dx.doi.org/10.1080/13614533.2012.753464     wells, j. (1996). the influence of library usage on undergraduate academic success. australian academic and research libraries, 26(2), 121-128. http://dx.doi.org/10.1080/00048623.1995.10754923   wilson, m. (2000). understanding the needs of tomorrow’s library user: rethinking library services for the new age. australasian public libraries and information services, 13(2), 81-86. retrieved from http://library.sau.edu/jpollitz/reserves/libraryforthenewage.htm     whitmire, e. (2002). disciplinary differences and undergraduates’ information-seeking behaviour. journal of the american society for information science and technology, 53(8), 631-638. http://dx.doi.org/ 10.1002/asi.10123   wong, s. h. r., & webb, t. d. (2011). uncovering meaningful correlation between student academic performance and library material usage. college and research libraries, 72(4), 361-370. retrieved from http://crl.acrl.org/content/72/4/361.abstract   evidence based library and information practice commentary   counting what cannot be counted: bringing the humanities to eblip   heidi lm jacobs information literacy librarian leddy library university of windsor windsor, ontario, canada email: hjacobs@uwindsor.ca   denise koufogiannakis collections and acquisitions coordinator university of alberta libraries edmonton, alberta, canada email: denise.koufogiannakis@ualberta.ca   received: 14 aug. 2014 accepted: 21 aug. 2014      2014 jacobs and koufogiannakis. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   introduction   evidence based practice in librarianship (eblip) has evolved since its beginnings in 1997 (eldredge, 1997) when the model was closely based on evidence based medicine, which itself had only begun as a movement five years earlier (evidence-based medicine working group, 1992). the focus of eblip at that time was on using research for decision making, and within that focus, quantitative research was privileged within a hierarchy that positioned certain types of research as more rigorous, reliable, and valid (eldredge, 2000).   such a model for evidence based practice diminished the importance of qualitative research and other forms of evidence frequently used by librarians in their practice. criticism (banks, 2008; given, 2006; hjørland, 2011; hunsucker, 2007) led to a rethinking of the model within librarianship, and new approaches and ways of thinking about the value of different types of evidence for use in decision making within librarianship were proposed (booth, 2009; howard and davis, 2011; koufogiannakis, 2013). the conversation about what constitutes evidence in the context of librarianship has broadened as a result, but still requires further discussion, debate, and examination.   one area that has not yet been noted, or really even questioned, is the absence of dialogue regarding the place of humanities research within an evidence based approach to practice. although widely regarded as a social science, librarianship is closely aligned with humanities; many librarians have humanities backgrounds, and many questions related to libraries and librarianship have roots in humanities thinking. it is somewhat perplexing that the absence of humanities research in eblip has not been raised as an issue.   the absence of humanities research in eblip could be explained by the fact that many librarians with humanities backgrounds have felt that the evidence based approach does not recognize or include them because the forms of evidence humanists use “are not being recognized as important” (koufogiannakis, 2012, p. 6). overlooking or ignoring the kind of evidence humanists value may have led librarian scholars with humanities backgrounds to ignore or drop out of the eblip conversation because it does not seem open to their work, ideas, or approaches.   this commentary attempts to redress this real or perceived exclusion by exploring how humanities research fits within evidence based practice in librarianship. does humanities research have a place within a model using evidence for practice-based decision making? can the humanities’ forms of evidence—theory and reflection—be useful in librarian decision making? can theory and reflection as forms of evidence push eblip in new directions by asking different questions and by asking questions differently?   since one cannot quantify theoretical thoughts or measure reflective practice, some researchers might view the kind of evidence humanists value as soft at best, inadmissible at worst. in this commentary we will provide an overview of humanities research, consider the properties of humanities research, and argue that humanities research not only fits within eblip, it offers a much needed approach to our decision making processes.   humanities research   if one asked 100 humanists “what exactly are the humanities?” most scholars would likely begin by mentioning the connections to disciplines such as english literature, philosophy, history, modern languages, classics, and so on. likely, this is where the commonalities would end and 100 different answers would then emerge regarding research topics, approach, or methodologies. for most, if not all, humanists, the discipline’s open-endedness in terms of subject, approach, methodology, and forms of “evidence” is what is appealing to them.  it is this openness or flexibility, however, which makes humanities research seem suspicious, non-rigorous, and conjectural to those outside the discipline. most humanities research is different from most social science research. we must remember though that the word “different” should be used as a descriptor not a value judgment. it is because humanities research is different from social science research that it has so much to offer eblip.   for most scholars working in the humanities, it is not the quest for a single or definitive answer that fuels their work but rather the process of asking difficult and complex questions and working through those questions. perhaps this is why it is so difficult to find a single, stable definition of the humanities: those working in humanities tend to be suspicious of any answer that tries to be absolute or definitive. this is not to say that humanities scholars do not consider, problematize, or theorize their field of study: they do, often in great length and detail. however, when humanities scholars set out to consider their field of study, they tend to do so at a disciplinary level rather than a categorical level: what is philosophical research? what drives literary scholarship? nevertheless, through describing their specific fields, humanities scholars often reveal key elements of humanities research.    jordanova (2000), for example, provides what could be read more broadly as an excellent overview of humanities research. she writes:   historians study human nature in operation. they do not observe this directly as anthropologists, sociologists and psychologists have the chance to do, but mediated through sources. they are interested in both the abstract and the concrete features of past societies, and in the connections between them . . . history involves intricate dialogues between the specific and the general. any given text, image, activity or experience is set in contexts—the plural is important since historians typically consider a range of contexts, including those in which their sources were produced, received, and used and those in which complex phenomena take place and are given. customarily, such contexts involve structural elements, that is, the systems through which a given society functions—the distribution of wealth and of power, forms of social difference, institutions, administrations, governance and so on. (p. 197)   literature scholar graff’s (2007) description of literary theory is similarly useful in understanding humanities research. he contends that we must think of literary theory not “as a set of systematic principles necessarily, or a founding philosophy, but simply as an inquiry into assumptions, premises and legitimizing principles and concepts” (p. 252). literary theory, he goes on to argue:   treats literature in some respects as a problem and seeks to formulate that problem in general terms. theory is generated when some aspect of literature, its nature, its history, its place in society, its conditions of production and reception, its meaning in general, or the meanings of particular works, ceases to be given and it becomes a question to be argued in a general way. theory is what inevitably arises when literary conventions and critical definitions once taken for granted have become objects of generalized discussion and dispute. (p. 252)   jordanova’s and graff’s descriptions of their fields offer us several key concepts that are worth keeping in mind as we ponder what the humanities are about and what humanists do in their research. humanists tend to explore connections between the abstract and concrete, create dialogues between the specific and general, consider context, engage with complex phenomena and structural elements, and investigate assumptions and premises we often take for granted.   much of humanities research is also about synthesizing, by pulling diverse elements together, placing them in dialogue, looking for relationships, and articulating a cohesive, well-argued narrative about those elements and relationships. to go back to jordanova’s description of her field, historical research involves:   using historical materials and ideas in a coherent argument, showing their significance, especially in the light of other accounts, making convincing, plausible claims based upon research findings, and employing concepts, theories, frameworks appropriately. these are dependent on other skills: clear, logical and evocative writing, critical reading, making connections, and the ability to see patterns and links, that is, to think laterally, integrating different kinds of materials. (pp. 185-186)   key components of humanities research are questions and the act of questioning. jordanova describes the role of questions in historical research thusly:   the way into any historical work—whether undergraduate essays or the most advanced research—must be through a question, a puzzle, a conundrum, an anomaly, a surprise, a hypothesis. these can take many different forms, but most often they involve some kind of comparison, which provides a context for the question. (p. 174)   by foregrounding questions as the starting place of research, the humanities are no different from any other discipline; every discipline uses questions as their starting point for research. however, what distinguishes the humanities from other disciplines are the ways questions are approached and the role “answers” play within the scholarly process. finite answers are rarely the goal within most humanities research. instead, what is important is the process of working through questions and posing new questions.   as graff has pointed out about the teaching of literature, “[t]he assumption has been that students should be exposed to the results of the disagreements between their instructors—results presumably representing settled knowledge not the debates that produced them” (p. vii). instead of showing students the results of scholarly debates, graff argues we need to teach the controversies:   controversial issues are not tangential to academic knowledge but part of that knowledge. that is, controversy is integral to the subject matter of subjects or disciplines – it is the object of knowledge or is inseparable from it. debates about what a literary work means, or whether it deserves classic status or not are internal to the study of that work, if only because such debates are part of the awareness of literate readers. (p. xv)   questions such as what does shakespeare’s “sonnet 18” mean, or does jane austen’s pride and prejudice deserve “classic status,” are questions that cannot be proved definitively and they are not intended to be proved. instead, they are questions that ask us to consider issues such as: where does meaning come from? how do we make meaning or arrive at meaning from reading a poem? or, what makes a literary text worthy of classic status? what qualities should that work possess? who decides what is a classic or not a classic? as these brief examples suggest, humanities research often answers questions with other questions, and it values the process of working through questions perhaps more than arriving at an answer.   as noted above, answers in humanities research are rarely finite, fixed, or definitive: rather, the answers are always contextual, open to further synthesis and interpretation, and demanding of new questions. as graff reflects:   the better historians never forget that any reconstruction of the past is always problematic and open to challenge, that historical interpretation is not simply a matter of accumulating facts but a hermeneutical weighing of inferences and hypotheses whose results are conjectural, tentative, and subject to refutation. (pp. 203-204)   as graff’s quotation suggests, questions are not asked for the sake of asking questions: questions are asked as a way to further knowledge, interrogate controversies, understand the complexities within a particular topic or issue, and synthesize different, differing and often contradictory forms of evidence.   one final aspect of some humanities research we would like to draw attention to is the articulation of explicit or implicit connections with the world outside of the discipline. frequently, humanities research has elements of social justice or social change within the research question. some progressive humanities researchers view their work as ammons (2010) describes:   our task (as progressive humanists) is to open young people’s eyes to oppressive systems of human power, how they work, and how we are all involved in them. we expose the injustices and the ideologies driving them…. we help others to see the importance of interrogating the bases of contemporary thought in order to understand destructive forces in the world today such as racism, environmental devastation, and economic imperialism. (pp. 11-12)   a progressive humanistic approach, she further argues, should not only show “what’s wrong in the world but also how we might fix it—what actions, personal and collective, we might take to change the world for the better” (p. 14). in progressive humanities research, we are reminded that the work we do, the research we conduct can—and indeed should—make a difference in the world. certainly, there are similar social justice elements within librarianship and within our individual and collective practice. humanities research could help us nurture and further develop work in the areas of progressive librarianship. even if we do not work with students in the ways that ammons describes, it is worth considering if there are ways we can ask critical questions of ourselves and our profession regarding the ways we do our work and run our libraries in relation to the “systems of human power" (p. 11).   while some scholars have attempted to pin down humanities research methodologies (ochsner, hug, & daniel, 2013), one inherent merit of the humanities is its flexibility and openness depending on what a particular question or situation requires. jordanova calls this flexibility “eclecticism,” a word she hastens to note is “sometimes treated as a dirty word” (p. 198). “at the very least,” she continues:   it sounds untidy—just so: if historians treat the past in too tidy a manner they lose a great deal. ... it is precisely the ability to embrace complexities while making sense of them, and to think flexibly about diverse phenomenon at distinct analytical levels that characterises historians’ purchase on the past. (p. 198)   certainly, the humanities’ methodological openness can, from the outside, seem soft, questionable, and hardly rigorous. however, it is important to remember that humanities research uses a different set of critical paradigms and asks different questions. as william bruce cameron (1963) writes, “not everything that can be counted counts, and not everything that counts can be counted” (p. 13). if humanities scholarship asks us to “count” different things, what might those different things be and how might they contribute to our practice in librarianship, and more specifically, evidence based practice?   incorporating humanities research into evidence based practice   as a movement within librarianship, eblip is focused on the practical integration of the best sources of evidence to answer questions that arise in practice within our profession. eblip provides a model for librarians to use as a guide in order to more thoroughly adapt and be successful with such an approach to decision making. the stages within a revised eblip model are:   articulate – come to an understanding of the problem and articulate it. assemble – assemble evidence from multiple sources that are most appropriate to the problem at hand. assess – place the evidence against all components of the wider overarching problem. assess the evidence for its quantity and quality. agree – determine the best way forward and if working with a group, try to achieve consensus based on the evidence and organisational goals. adapt –revisit goals and needs. reflect on the success of the implementation. (booth, 2009, p. 342; koufogiannakis, 2013, pp. 189-190)   throughout these stages there is not only room for humanities research, there is a distinct need for a humanities approach to be incorporated. in the beginning stage of articulate, one asks, “what do i already know” and places the question or problem in a wider context. while there may be a very specific question to be answered at this point in time, it is valuable to explore the wider issues and understand the many other questions that arise in conjunction with the problem to be addressed. it is here where we may incorporate professional knowledge and the wider concerns and principles of the profession into our thinking about the problem. we need, as graff (2007) might argue, to engage with the controversies of our profession. a humanities approach at this stage would push us to seek wider understandings of the various issues arising, and urge us to go into problems with knowledge that answers may not be easy or tidy. humanities research will also remind us that the absence of definitiveness is not only to be expected but also acceptable.   questioning what we do in practice ultimately leads us to determine what kind of practice we collectively want to have as librarians, strengthening our knowledge of what we believe in, how we progress in changing times, and where we ultimately set our priorities and goals. within this questioning process, humanities approaches will remind us that the very notion of “best” is highly contextual and extremely subjective. we need to remember that words like “good” or “best” are words that we have come to take for granted and not problematize: we must make words like these “objects of generalized discussion and dispute” (graff, 2007, p. 252). in so doing, we remember that whatever is “best” is highly subjective, extremely context-dependent, and very likely to change.   when assembling evidence to answer the question or problem at hand, humanities thinking reminds us to use whatever evidence we need to answer the question. it also asks us to be flexible and incorporate the many sources of evidence into our desire to come to a “good” decision, rather than rigidly following a hierarchy or set path. further, it demands that we evaluate and critically think about all pieces of evidence and all kinds of evidence with an open mind. as we assess the various forms of evidence, we draw upon the humanities’ skills of synthesizing and drawing together connections between differing pieces of evidence in order to gain a more holistic understanding of the evidence than we could if we isolated individual sources. this type of consideration of evidence would facilitate dialogue leading to the agree stage of the process, wherein the group would consider all elements and appreciate what all members bring to the table. by taking the time to consider the whole of what is presented and how it fits together—or does not fit together—we rigorously evaluate what the “best” way forward might be at this particular point in time. we understand that whatever answer we arrive at is not final, but is sufficient for the time being and changeable in the future.   the final stage in the cyclical eblip process is adapt, and this stage calls for reflection upon one’s role and actions within the decision making process, paying attention to new questions or problems arising. it encourages immediate questioning and placement of the process back into the wider context surrounding the decision. the eblip process does not simply end when a decision is made, but encourages openness and curiosity: a humanities perspective reminds us there are always more questions than answers. a humanities perspective invites open, innovative, and creative thought, while loosening rigidity and static absolutes.   for eblip to be successful, librarians must acknowledge that uncertainty is acceptable and that questioning practice is a healthy part of growth and, as such, is a valid form of research inquiry. a humanities approach to research helps us see questioning as the norm and understand that any decision does not need to be closed or made final in order for it to be a “successful” decision. as practitioners we need to know that changing a decision based on new evidence is not a failure, but rather a successful progression that shows adaptability and growth. if we focus, as humanists do, on continually questioning, we will understand why we are moving in a certain direction and better understand what makes a “good” decision. perhaps rather than thinking that eblip will lead us to a “best” and final answer to a particular practice question, we need to acknowledge that eblip will help us to make the best answer in a particular context, at a particular point in time.   humanities research may not fit into a tidy checklist regarding validity and reliability of method, but it is perhaps best suited to answer the bigger, most important questions within librarianship. questions such as “what do we mean by ‘best’?”, or “for whom does this ‘best’ solution best serve?”, are questions that make us think differently about our processes, practices, and decisions. further, questions such as, “what are the unstated principles or assumptions that we are operating under?”, or “how are these practices or decisions responding to larger institutional, cultural, economic, or global initiatives?”, help us to locate our decision making within a broader context. these types of questions need to draw upon processes that synthesize and contextualize evidence, place concrete and abstract in dialogue with each other, and problematize the accepted and given assumptions and practices: these are processes that are central to humanities research. we need to remember that asking and seeking answers to these types of questions is also part of being an evidence based practitioner. without considering the bigger picture and questioning the core values that inform our practices and decisions, we will lose sight of the potentials within our profession. humanities research can help us pull new kinds of evidence into our practice and to think about evidence in new ways. in so doing, humanities research can help us articulate complex readings of issues within our profession and the role these issues play in our profession as a whole.   if we use humanities research in eblip, we need to be able to accept that theory and reflection are valid and reliable forms of evidence. certainly most theories or reflections cannot be empirically proved or quantified, but that is not to say that they are not rigorous or reliable forms of evidence. to revisit jordanova and graff, theoretical and reflective thinking requires a range of rigorous modes of thinking, analysis, and evaluation as they explore the connections between the abstract and the concrete, the specific and the general and consider how context, complex phenomena and structural elements work together. in their inquiry into “assumptions and premises we often take for granted” (graff, 2007, p. 252), humanities researchers synthesize diverse and often contradictory forms of information and evidence and put them into dialogue with each other and with broader social, political, economic, cultural, or disciplinary contexts. finally, humanities researchers present their “findings” in carefully constructed, well-argued, well-supported, logical, compelling written arguments. as jordanova writes, “these are complex and subtle skills” (2000, p.186). to understand what humanities research can contribute to eblip and studies of the profession, we need to count these “complex and subtle” skills as valuable, rigorous, reliable, and intellectually useful. additionally, we need to count the questions humanities research raises as useful ways to consider our practice and to inform our decision-making process.    a practical example of what humanities research could contribute to decision making processes can be drawn from the broad topic of open access (oa). oa possesses a wide-ranging complexity and is of concern to librarians, as well as faculty, publishers, funding bodies, and the public. it is an issue that some librarians have embraced as one that is core to our profession, and one for which they want to play a role in shaping future development. if we look at oa from an evidence based perspective, we would start with the problem we are facing or trying to solve, and pose that problem as a question. there are probably thousands of questions one could ask in relation to oa, but let us consider the example related to oa author fees that might arise in practice:   what is the most efficient way to manage an open access author’s fund at my institution? what do faculty think about paying author fees, and would they welcome library support? should libraries be paying author fees?   an eblip approach asks that you consider what evidence would be the best to answer the question at hand. the question about efficiency would probably be best answered by quantitative research study that can be applied to your own situation, or lacking that, an examination of how others have been managing their author funds. through comparison, you could determine what would work best at your own institution. the question related to what faculty think about paying author fees would likely look to qualitative studies already published, and be supplemented with local information to obtain a better understanding of local needs and the library’s role within the larger organizational context.   on the surface, the question “should libraries be paying author fees” could be examined qualitatively or quantitatively through surveys or interviews asking for librarian, faculty, and administrators’ opinions on the topic. however, this question could also be approached as a humanities question. instead of treating it as a “yes or no” question that aims to find out what most people think, it would take a broader approach dealing with principles and sub-questions relating to the library’s role and support of oa and the broader nature and context of oa. if we begin thinking further about libraries’ roles related to oa, additional questions would emerge such as: why is oa important to libraries? from where does that commitment emerge? with what values of librarianship does oa connect? what are the broader social, professional, economic, cultural contexts for the oa movement? humanities research would, in this instance, push us to interrogate some of the principles behind oa.   within librarianship today there is an overwhelming consensus that oa is, overall, a “good” thing and that libraries should be involved in oa. however, there are multiple nuances and implications to oa that we urgently need to take into account and assumptions we need to examine and unpack. oa has become something unquestionably “good” to our profession and as such it has become something we take for granted and often do not question or problematize. because librarianship has so actively endorsed oa, there have been very few critics of oa as a concept, principle, or practice. even though there are infinitely more reasons to support oa than to reject it, we still need to question our trust of it and ask the difficult questions so that we are confident we are making the “best” possible decisions locally, nationally, and globally. we need to consider questions like: are there downsides to oa? have we considered what oa does or might do to the economies of scholarly publishing or university presses? who benefits from oa? do our notions of oa take into account the economies of scholarly publishing? could oa become undemocratic or oppressive in other ways? who pays for oa journals? are oa journals staffed by volunteers or paid employees? if unpaid labour, what does oa do to the de-professionalization of trained and skilled copyeditors, editors, and production staff? if paid labour, who is paying? if authors are expected to pay to be published in a particular journal, there are questions we need to ask about scholars’ abilities to pay. would scholars have to pay fees themselves, or would their universities cover these fees? what about sessional faculty or graduate students? could author payments mean that someday only scholars from elite institutions could afford to publish in the top journals? are there ethical issues related to paying to be published in a scholarly forum? further, we could also ask questions of a progressive humanities nature: how could (or does) oa make a difference in the world in terms of power, equity, and equality? are there ways that oa could contribute to larger social justice issues?   none of the above mentioned questions are easy but they are questions we need to ask of ourselves, of our practice, and of our profession. we need to ask these questions to work through all aspects of oa and understand how it fits in with broader contexts. we may not arrive at a single answer or a definitive answer, but the process of asking these tough questions makes us confident that, whatever decisions we arrive at, we have considered all of the nuances of the question and that our decisions our decisions are solid, rigorously considered, thoroughly contextualized, and forward looking.   conclusion   in 2013, horgan published a blog post on the scientific american site called “why study humanities? what i tell engineering freshman.” horgan tells his students that the humanities are subversive. they undermine the claims of all authorities, whether political, religious or scientific. ... the humanities are more about questions than answers, and we’re going to wrestle with some ridiculously big questions in this class” (n.p).   horgan’s phrase “wrestle with some ridiculously big questions” summarizes much humanities research. it is possible that wrestling with ridiculously big questions may be the best definition of humanities research we can find. humanities research is crucial to an evidence based approach to practice because the humanities can wrestle with those ridiculously big questions in librarianship. the type of evidence used in research always depends on the question at hand, and humanities research will help us navigate and consider some of the larger issues that inform, contextualize, problematize, and develop questions about our profession and practice which eblip, in its current approach, cannot. answers to questions about how we move forward within our profession and what directions we should take must be guided by principles that have been questioned, interrogated, theorized, problematized, and reflected upon using a wide spectrum of evidence and critical approaches.    as scholars and practitioners, it is unlikely we will ever be able to provide answers to the ridiculously big questions our profession asks of us. however, we must see this open-endedness as not only acceptable but necessary. if we limit our research inquiries to questions that can be answered definitively, we scale back the kinds of questions we ask about our practice and our profession. in so doing, we miss some of the major issues, controversies, and ideas that make our profession engaging and vital. further, if we only rely on evidence that is countable or quantifiable, we are excluding a whole segment of evidence that can help us think critically, creatively, and innovatively about our profession and our practice both today and in the future.   horgan concludes by arguing the point of the humanities is that they “keep us from being trapped by our own desire for certainty” (n.p.). librarianship is at a point in time where we are bombarded by some ridiculously big questions and we need all manner of thinking and all manner of thinkers working on these questions. the quest for certainty through uncertainty and an ability to count what cannot be counted are the types of contradictions that eblip should embrace. bringing the humanities to eblip is not only possible – it is necessary.     references   ammons, e. (2010). brave new words: how literature will save the planet. iowa city, ia: university of iowa press.   banks, m. (2008). friendly skepticism about evidence based library and information practice. evidence based library and information practice, 3(3), 86-90. http://ejournals.library.ualberta.ca/index.php/eblip   booth, a. (2009). eblip five-point-zero: towards a collaborative model of evidence-based practice. health information and libraries journal, 26(4), 341-344. http://dx.doi.org/10.1111/j.1471-1842.2009.00867.x   bruce cameron, w. (1963). informal sociology, a casual introduction to sociological thinking. new york, ny: random house.   eldredge, j. d. (1997). evidence-based librarianship: a commentary for hypothesis. hypothesis, 11(3), 4-7.   eldredge, j. d. (2000). evidence-based librarianship: an overview. bulletin of the medical library association, 88(4), 289-302. https://www.mlanet.org/publications/jmla   evidence-based medicine working group. (1992). evidence-based medicine: a new approach to teaching the practice of medicine. jama: journal of the american medical association, 278(17), 2420-2425. http://dx.doi.org/10.1001/jama.1992.03490170092032   given, l. (2006). qualitative research in evidence-based practice: a valuable partnership. library hi tech, 24(3), 376-386. http://dx.doi.org/10.1108/07378830610692145   graff, g. (2007). professing literature: an institutional history, twentieth anniversary edition. chicago, il: university of chicago press.   hjørland, b. (2011). evidence-based practice: an analysis based on the philosophy of science. journal of the american society of information science, 62(7), 1301-1310. http://dx.doi.org/10.1002/asi.21523   horgan, j. (2013, june 20). why study humanities? what i tell engineering freshmen. scientific american. retrieved from http://blogs.scientificamerican.com/cross-check/2013/06/20/why-study-humanities-what-i-tell-engineering-freshmen/   howard, z., & davis, k. (2011). from solving puzzles to designing solutions: integrating design thinking into evidence based practice. evidence based library and information practice, 6(4), 15-21. http://ejournals.library.ualberta.ca/index.php/eblip   hunsucker, r. l. (2007). the theory and practice of evidence-based information work–one world? paper presented at the 4th international evidence based library and information practice conference, university of north carolina-chapel hill, durham, nc. retrieved from http://www.eblip4.unc.edu/papers/hunsucker.pdf   jordanova, l. (2000). history in practice. london, uk: arnold.   koufogiannakis, d. (2012). academic librarians’ conception and use of evidence sources in practice. evidence based library and information practice, 7(4), 5-24. http://ejournals.library.ualberta.ca/index.php/eblip   koufogiannakis, d. a. (2013). how academic librarians use evidence in their decision making: reconsidering the evidence based practice model (doctoral dissertation). aberystwyth university, aberystwyth, uk. retrieved from http://cadair.aber.ac.uk/dspace/handle/2160/12963   ochsner, m., hug, s. e., & daniel, h. d. (2013). four types of research in the humanities: setting the stage for research quality criteria in the humanities. research evaluation, 22(2), 79-92. http://dx.doi.org/10.1093/reseval/rvs039   evidence based library and information practice editorial responsibilities      2015. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     editor-in-chief: lorie kloda   associate editor (evidence summaries): heather pretty   associate editor (articles): rebekah (becky) willson; lisl zach; with additional assistance from denise koufogiannakis   associate editor (classics): jonathan eldredge   associate editor (reviews): denise koufogiannakis   associate editor (commentaries, using evidence in practice, news): lorie kloda   production editor: michelle dunaway   editorial intern: melissa griffiths   editorial advisor: alison brettle   copyeditors: richard hayman (lead copyeditor), georgianne bordner, julie evener, linda ferguson, heather healy, stacey penney, maria tan, mary virginia taylor, alison yeoman   indexing support: meral alakus evidence summary   mixed-method survey research is useful to incrementally improve library homepage design   a review of: deschenes, a. (2014). improving the library homepage through user research – without a total redesign. weave, 1(1). http://dx.doi.org/10.3998/weave.12535642.0001.102   reviewed by: kathleen reed assessment & data librarian vancouver island university nanaimo, british columbia, canada email: kathleen.reed@viu.ca   received: 3 jun. 2015     accepted: 11 aug. 2015      2015 reed. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to assess content organization and wording of links on the library’s homepage.   design – mixed-methods survey.   setting – small college, united states of america.   subjects – 57 library users.   methods – library staff distributed paper surveys at the entrance to the library, with the goal of collecting a minimum of 30 surveys. the survey directed participants to indicate their preferred terms from a list, and their preference for ordering the menu items on the library’s homepage. qualitative survey data was also collected via several open-ended questions that began with prompts such as “i really love…” and “i can never find…”   main results – the search box tab labelled “library catalogue” was preferred over “books and media,” which the staff believed to be a more user-friendly term. using a pre-defined list, participants ranked the library catalogue as the most important tab, followed by e-resources, articles, and library guides. a link to the library catalogue was also selected as the most important resource sidebar link, followed by e-resources, full-text journals, library guides, and refworks. the service sidebar links by order of importance were found to be: library hours, group study rooms, writing & citing, interlibrary loan, and chat with a librarian. qualitative feedback received demonstrated a lack of understanding what the terms “library guides” and “a-z list” mean, and difficulty finding a complete list of databases. library staff received feedback that the library hours and account log in should be made more prominent.   conclusion – library staff updated the website to reflect user preferences for wording and order of links on the homepage. google analytics showed a decrease of 30 seconds per average visit after the changes, which the author attributes to better wording and organization. there were no complaints about the website in the first three months after the change. the author concludes that a paper survey is an effective tool for librarians who would like to make incremental changes on their homepages.   commentary   website design and usability are much-discussed topics both within libraries and more broadly. with so much research already done on particular wording and design for creating an optimal library homepage, this study has few original findings to add to the conversation and made few links to previous research findings. rather, the value of this article is in the type of evidence collection that the work discusses and models.   this study is an excellent example of a small, incremental assessment activity that was undertaken between major user experience studies. it exemplifies the difference between research and assessment. as a research study, this work has significant flaws which the authors do acknowledge. the sample size is small and not representative of all students. as the college is home to a library and information school, there is the potential for students in this program to skew results. the survey design assumes that people who physically visit the library also use the website, and those who use the website visit the physical library. the survey results are not generalizable to other institutions, failing to pass glynn’s critical appraisal checklist (2006).   however, if understood as an assessment activity, a short mixed-methods survey can be helpful to an institution. instead of leaving website design up to library staff, the college was able to update website terms and link order using some evidence. for example, library hours were identified in the survey as being the most important link related to services and thus listed at the top of the navigation sidebar. without the survey, the homepage would have been completely influenced by library staff’s opinions. this information was gained with a small amount of staff time (20 hours) and funds, and did not conflict with institutionally-mandated branding or content management systems. not every activity intended to collect evidence to assist decision-making can or should be a thorough research study.   an exercise like the one outlined in this article has significant benefit for collecting evidence supporting small, continual changes. the process discussed would be beneficial in any type of library and does not require staff to be well versed in web design or user experience testing. the author does a thorough job of detailing the process, making it easily actionable for librarians at other institutions to do similar work. the author is to be commended for publishing an example of an evidence-based practice that any librarian could pick up and use, regardless of his or her familiarity with research methods.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi: 10.1108/07378830610692154   conference paper   “what’s so special about special collections?” or, assessing the value special collections bring to academic libraries   christian dupont aeon program director atlas systems virginia beach, virginia, united states of america email: cdupont@atlas-sys.com   elizabeth yakel professor school of information, university of michigan ann arbor, michigan, united states of america email: yakel@umich.edu      2013 dupont and yakel. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – the objective of this study was to examine and call attention to the current deficiency in standardized performance measures and usage metrics suited to assessing the value and impact of special collections and archives and their contributions to the mission of academic research libraries and to suggest possible approaches to overcoming the deficiency.   methods – the authors reviewed attempts over the past dozen years by the association of research libraries (arl) and the association of college and research libraries (acrl) to highlight the unique types of value that special collections and archival resources contribute to academic research libraries. they also examined the results of a large survey of special collections and archives conducted by oclc research in 2010. in addition, they investigated efforts by the society of american archivists (saa) dating back to the 1940s to define standardized metrics for gathering and comparing data about archival operations. finding that the library and archival communities have thus far failed to develop and adopt common metrics and methods for gathering data about the activities of special collections and archives, the authors explored the potential benefits of borrowing concepts for developing user-centered value propositions and metrics from the business community.   results – this study found that there has been a lack of consensus and precision concerning the definition of “special collections” and the value propositions they offer, and that most attempts have been limited in their usefulness because they were collections-centric. the study likewise reaffirmed a lack of consensus regarding how to define and measure basic operations performed by special collections and archives, such as circulating materials to users in supervised reading rooms. the review of concepts and metrics for assessing value in the business community, however, suggested new approaches to defining metrics that may be more successful.   conclusion – the authors recommend shifting from collection-centric to user-centric approaches and identifying appropriately precise metrics that can be consistently and widely applied to facilitate cross-institutional comparisons. adopting a user-centric perspective, they argue, will provide a broader picture of how scholars interact with special collections at different points in the research process, both inside and outside of supervised reading rooms, as well as how undergraduate students change their thinking about evidence through interaction with primary sources. they authors outline the potential benefits of substituting the commonly used “reader-day” metric for tabulating reading room visits with a “reader-hour” metric and correlating it with item usage data in order to gauge the intensity of reading room use. they also discuss the potential benefits of assessing impact of instructional outreach in special collections and archives through measures of student confidence in pursuing research projects that involve primary sources.   introduction   what’s so special about special collections? what kinds of value do they contribute to the overall mission of academic libraries and their parent institutions in terms of support for research, teaching, and learning? how should we measure return on the immense investment that it takes to maintain their secure, climate-controlled facilities, provide salaries and wages for staff, and support the various costs associated with acquiring, processing, and preserving rare and unique materials? how should we measure value and assess impact?   “what’s so special about special collections?” was the title chosen for the inaugural issue of the association of college and research libraries (acrl) journal rbm: a journal of rare books, manuscripts and cultural heritage, which appeared in spring 2000. although the most frequent answer to the question in that issue was the collections themselves (boyd, 2000; de hamel, 2000), some articles also pointed to the distinctive qualities of special collections researchers (howarth, 2000) and staff and their interactions (katchen, 2000). the same title was also used for a special section of an issue of american libraries (2000) that highlighted several collections as well as collaborative collecting projects.   in june 2001, the association of research libraries (arl) held a working symposium on the future of special collections in research libraries at brown university (association of research libraries, 2001; hewitt & panitch, 2003). this led to the creation of an arl task force charged with engaging the agenda that emerged from the symposium. following the task force’s final report (association of research libraries, 2006), a new arl special collections working group was assembled and given a charge that included “contributing to the work underway within arl to develop qualitative and quantitative measures for the evaluation of special collections” (association of research libraries, 2010b). in october 2009, arl partnered with the coalition for networked information (cni) to host a two-day forum on special collections, “an age of discovery: distinctive collections in the digital age,” which opened with a panel session titled “why are special collections so important? exploring the value proposition of special collections” (association of research libraries, 2009a). participants at the forum focused on expanding the research potential and value of special collections through the creation of virtual collections of digitized materials. nichols (2009) explored various aspects of this issue, taking up both the utilitarian argument that special collections do not deserve support unless they are widely used, as well as the scholarly perspective that digitization represents a natural evolution which promises to keep the collections alive through new (if not yet fully discovered) transformative uses.   in january 2010, arl announced that it was collaborating in a three-year imls grant-funded study titled “value, outcomes, and return on investment of academic libraries (lib-value),” the aim of which is to “enrich, expand, test, and implement methodologies measuring the return on investment (roi) in academic libraries” (association of research libraries, 2010a). whether or how this study will address special collections is not yet known, but it seems worthwhile to suggest some possible areas of engagement, especially because library discourse around value propositions appears to have reached a critical juncture.   value propositions for special collections   defining customer-oriented value propositions emerged as a business strategy in the early 2000s. in business parlance, a “value proposition is an analysis and quantified review of the benefits, costs, and value that an organization can deliver to customers and other constituent groups within and outside of the organization” (value proposition, 2012; barnes, blake, & pinder 2009, p. 28). as we have noted, value propositions for special collections – although they have not often been labeled as such – typically have been framed around inherent features of the collections themselves or their use by scholars. summarizing these viewpoints, waters (2009) stated at the arl-cni symposium:   at its most simplistic, the value proposition for special collections is that scholarship broadly across fields in the humanities, social sciences, and the sciences just cannot proceed without corollary investment in the acquisitions and carrying costs of the primary source evidence needed to sustain and advance those scholarly fields. (waters, 2009, p. 32)   others, meanwhile, have attempted to articulate the value of special collections in terms of their impact upon a wide range of functions and indicators. these include not only contributions to research and the creation of new knowledge, but also their usefulness for teaching and learning (particularly through the development of critical thinking skills), and even the enhancement of an institution’s reputation and prestige – the latter being a function highlighted by koda (2008) but implicitly critiqued by waters (2009).   measuring and assessing value   although the special collections and archives communities by and large have not attempted to articulate explicit value propositions, they have engaged in various efforts to define collection and usage metrics.   the metrics movement in the archival community dates back to the 1960s, when the society of american archivists established a committee on uniform archival statistics. its functions were:   to collect and analyze information about existing archival statistical systems with a view towards (a) isolating and describing these aspects of archival activity which are measurable, i.e., can be expresses in numerical terms; (b) defining these characteristics with a precision that will eliminate confusion wherever a particular term is used; (c) developing standards for archival statistics that will permit meaningful comparisons and studies of archival institutions throughout the country; and (d) encouraging general adoption of these standards by archival agencies. (campbell, 1967)   unfortunately, no statistical standards for measuring processing activities or usage emerged from this early effort or other subsequent attempts within the archival community, with the possible exception of the archival metrics project (n.d.), whose user-based evaluation tools for evaluating the quality of archival services and facilities are beginning to see some adoption (duff et al., 2010). meanwhile, the topic has been emerging in the special collections community, as evidenced, for example, by sessions held at recent acrl rare books and manuscripts section (rbms) preconferences, for example, the presentations given at the 2009 rbms preconference seminar on “public services and ‘un-hidden’ collections” (schreyer, schaffner, bowen, & steele, 2009).   a pressing need for special collections and archival metrics therefore remains, as witnessed by the report that oclc research has published concerning its 2010 survey of special collections and archives in 275 academic and research libraries throughout the united states and canada. the most comprehensive and detailed investigation of its kind to date, the study builds upon the “exposing hidden collections” survey conducted by panitch (2001) in 1998. report authors dooley and luce remark that their findings “convey how difficult it is to evaluate data usefully without standard metrics in use across the special collections community. we cannot demonstrate the level of value delivered to primary constituencies, unless we can reliably characterize our users” (2010, p. 35)  – and, we would add, their use.   while recurrent interest in both metrics and values has been expressed in the special collections and archival communities for some time, little traction thus far has been gained by efforts to define and operationalize the momentum.   a stalemate   we are thus faced with a stalemate. on the one hand, academic libraries have been focusing renewed attention on special collections over the past decade based on an assumption that rare and uniquely held materials will serve to distinguish research libraries as they rapidly move into a future in which their core collections and services will be constituted by a commonly held array of licensed content and other distributed electronic resources. on the other hand, special collections have done little either to articulate their distinctive value or identify metrics that demonstrate how they have been contributing to the mission of their parent institutions and the larger academic enterprise they serve. at the same time, the uniqueness of special collections vis-à-vis main library collections has been disputed (lavoie, connaway, & dempsey, 2005). and yet, turning the uniqueness argument on its head, waters (2009) – like nichols (2009) – has argued that the value proposition for special collections is enhanced through digitization and the resultant opportunities to perform cross-collection analyses and comparisons.   despite these disconnects, special collections have been producing relevant value on an increasing scale. in addition to pursuing a range of digitization activities, special collections librarians and archivists have also been working diligently to bring primary resources into the classroom to support teaching and learning. the 2010 arl spec kit survey on “special collections engagement” documents the greater levels of effort that special collections have been putting into instructional outreach, exhibits, and public programs in recent years. nevertheless, the otherwise encouraging report admits that “institutions feel they are not able to quantify the success of their efforts, and this in turn limits the ability to compare activities within the institution or across institutions, to plan further outreach effectively, or to communicate the results of those outreach activities to the larger special collections community” (berenbak et al., 2010, pp. 16-17).   it seems evident that the inability of institutions to quantify their successes, let alone describe them qualitatively, stems from a lack of standardized metrics for measuring special collections usage and or even commonly agreed-upon values. following are some perspectives and practical proposals that we hope will prove helpful in surmounting the current impasse.   approaching definitions   one problem that immediately arises in trying to identify either the value proposition or appropriate metrics for special collections is defining just what one means by “special collections.” this is typically done with reference to the collections themselves, and includes a list of materials formats and qualifiers concerning rarity and uniqueness and sometimes age and physical vulnerability. for instance, the 2003 arl statement of principles research libraries and the commitment to special collections takes this approach. according to this statement, special collections   comprise manuscripts and archival collections unduplicated elsewhere and one-of-a-kind or rarely held books. they also include items precious through their rarity, monetary value, or their association with important figures or institutions in history, culture, politics, sciences, or the arts. special collections extend beyond paper to other formats of cultural significance, for example photographs, moving pictures, architectural drawings, and digital archives. special collections are also significant for their focused assemblages of published materials so comprehensive as to constitute unparalleled opportunities for scholarship. (association of research libraries, 2003, n.p.)   the 2009 report on special collections in arl libraries pursued a decidedly different definitional course and identified special collections as “any kind of vehicle for information and communication that lacks readily available and standardized classification schemes, and any that is vulnerable to destruction or disappearance without special treatment” (association of research libraries, working group on special collections, 2009).   these approaches reflect and perpetuate varying degrees of ambiguity. they both also reflect a collections-centric approach. might it be possible and perhaps more useful to take a user-centric approach, especially when it comes to defining value? doing so would likely lead us to a different set of metrics and different algorithms for assessing quantitatively and qualitatively the values derived from special collections. by adopting a user-centric perspective, we may be able to look more broadly at how scholars interact with special collections at different points in the research process, both inside and outside of supervised reading rooms, as well as how undergraduate students change their thinking about evidence through interaction with primary sources.   values and metrics converge and diverge in various ways across the spectrum of library services. while collections that are used in a controlled reading room or staff-mediated situation are typically described as “non-circulating,” this is an imprecise and unhelpful term when it comes to defining metrics. it is, in fact, perfectly appropriate to refer to the process of requesting, consulting, and returning special collections materials in a reading room environment as “circulation” for the metaphor and processes it represents are essentially the same as those employed for the circulation of materials from open stacks in the main library. the only substantial difference is that the user is not allowed to remove the materials from a defined location. likewise, just as raw circulation counts can serve as a basic indicator of the frequency of the use of main library collections, so, too, they can provide a similar index of special collections usage. thus, a well-defined circulation metric for special collections and archives could also contribute toward a goal of integrating them more fully into the operational perspectives of their parent institutions (whittaker, 2008).   taking another tack, speedier circulation in special collections could help researchers work more efficiently and perhaps also more effectively by reducing lag times between requesting and receiving materials and the associated disruptions to study and concentration. accordingly, retrieval time could constitute a benchmark metric for special collections library service that could be correlated with increased scholarly productivity – an important component of the special collections value proposition, as we have seen above. in fact, the national archives (united kingdom) closely monitors retrieval times using its internal electronic paging system to meet its goal of fulfilling all requests within twenty minutes or less (d. priest, personal communication, december 8, 2010).   yet these types of linkages between circulation metrics and end-user values are not currently possible beyond the confines of individual institutions because there is no generally accepted definition for counting many of the types of materials that are found in special collections. although it is not difficult in principle to apply common standards for counting the use of published print materials, such as books and serials, where a count of circulated volume units is the established norm (association of research libraries, 2009b; international organization for standardization, 2008), there is no standard or best practice for tabulating the usage of collections of unpublished manuscripts and archives. special collections repositories that include university archives or otherwise function as archives tend to count circulation use at the box or container level. by contrast, special collections whose strengths lie in historical and literary manuscripts tend to count circulation use at the folder or even individual item level. some special collections that include both manuscript and archival collections apply a multiplier to circulation counts of archival boxes – thus, a single box may be tallied as representing 250-500 items if it is a 5-inch wide document case or even a thousand items if it is a full-size record storage container. such extreme variations in practice tend to render circulation figures for special collections rather meaningless, even for the special collections librarians who collect them.   another factor that complicates and skews circulation counts for special collections is the concept of a visit. arl statistics follow the definition and methodology prescribed by the niso z39.7 standard for calculating visits, which is an averaged gate count over a typical week (national information standards organization, n.d.). special collections libraries and archives, on the other hand, typically require their users to complete a registration form and present some type of identity document as a general security precaution before admitting them to the reading room. the registration record, or another system, such as a daily sign-in sheet, is also often used to track the date and time of the user’s visits and occasionally the materials consulted during each visit.   special collections libraries that track the materials consulted during each visit generally tabulate the use of the same item on different days as distinct circulation counts, whereas other repositories count only the first time the item is requested, especially if they base their count on the number of callslips submitted rather than the number of times the item is actually delivered to the researcher in the reading room. obviously, this divergence in practice results in statistical counts that cannot be used for cross-institutional comparisons.   to compensate for this lack of consistency, some institutions employ a “reader-day” metric that considers simply the total number of visits per month or per year. this type of broad metric has its shortcomings as well. for instance, a visit by a local user who requests one volume and looks at it for ten minutes is given equal weight to a visit by a professional scholar who has traveled from across the ocean to look at a cartful of material and who spends every moment the reading room is open doing so. likewise, the metric itself does not readily tell us how many unique visitors have used the reading room or how many days on average the typical user has stayed.   taking flight   to overcome this last defect, it would be enough to apply a more precise measure. looking at the basic metrics other industries have developed to measure their business performance can be instructive. for example, two basic metrics that the airline industry uses to measure overall business capacity and volume are “available seat-miles” (asms), which is equal to the number of available seats multiplied by the number of miles flown, and “revenue passenger-miles” (rpms), which equals the number of filled seats multiplied by the number of miles flown. dividing rpm by asm yields a third metric, “load factor,” which represents the percentage of airline seating capacity that is actually used (massachusetts institute of technology, global airline industry program, n.d.). because these metrics are simple in concept and can be equally and objectively applied across all airline companies, they are useful for assessing the performance of individual airlines from quarter to quarter, comparing the respective performance of multiple airlines over a single quarter, and benchmarking the overall performance of the industry over time.   although special collections reading rooms are seldom arranged or oriented to calculating seating availability or measuring their performance as a quotient of seats filled, it may be useful to consider adopting a measurement strategy akin to revenue passenger-miles in order to enable libraries to engage in meaningful longitudinal and comparative assessments. as shown above, the reader-day metric fails to adequately convey the amount of time that researchers actually spend in the reading room. it also does not take into account differences in reading room schedule. if a reading room is open for four hours on a saturday afternoon should that be considered equivalent to a weekday when the reading room is open for six or eight or ten hours? most special collections that employ a reader-day metric do in fact treat all of these as equivalent, which is to say that they consider a “day” to be any day when they maintain at least some reading room hours.   taking a lesson from the airlines, these shortcomings could be remedied by simply refining the basic reader-day metric to instead count reader-hours. just as airlines use seat-miles rather than, say, seat-segments to gauge capacity and profitability, so, too, special collections libraries could achieve a more precise, consistent, and objective measure of their use by counting the actual hours that researchers spend in the reading room.   and it would not be hard to do. to facilitate the tabulation and calculation, the manual tally sheets and reading room logs that most special collections employ to track usage could be replaced with simple electronic databases that staff would use to record the time that researchers enter and leave the reading room. if the log also linked visits with individual researchers, reports could be constructed to calculate the number of unique visitors during a given time period and analyses of the average visit lengths of various categories of users (such as students, faculty members, visiting scholars, and members of the general public). visitors could even be given “smart” cards to scan upon entry and exit, like those the national archives and records administration has begun issuing at some of its research facilities.   furthermore, reader-hour data could be correlated with circulation or item usage data to provide a kind of “load factor” indicator of reading room use. how many items, on average, do various categories of researchers consult when they visit the reading room? to facilitate basic comparisons, it would be enough to divide the total number of items used during a given time period by the total number of reader-hours.   having precise metrics and consistent data collection methods would enable managers to assess the adequacy of their services and staffing over time. are special collections reading rooms in fact getting busier as evidence from some libraries, largely anecdotal, would suggest? have changes in policy, such as allowing researchers to use personal digital cameras in the reading room, had an effect on the way researchers are using their time in the reading room? are researchers spending less time in reading rooms because they can now come in and make their own digital copies at no charge and then consult these copies at home on their own time? or is the opposite occurring: because it is now quicker and easier to obtain copies, are more researchers spending more time in the reading room and requesting more materials?   the impact of such changes in policy and practice on researchers and staff alike is potentially significant and therefore should be assessed. how else will libraries be able to make informed decisions about service delivery and staffing unless they have reliable measures and data to guide them? yet such assessments are not currently possible because special collections have yet to define and employ adequate metrics and data collection methods. in an unpublished study, yakel and goldman (2002) found that although all repositories have at least some mechanisms for data collection and there are some commonalities in the kinds of statistics collected, there are fundamental differences in the reasons why archives and special collections amass data, how they collect it, and what they do with it. these differences have persisted for a long time and have previously prevented standardization and circumscribed what can be done with this information. the interview data from the study also revealed the limitations of current data collection methods.   in terms of value, a corollary concept that could extend the application of reader-hours and load factor metrics is “intensity of use,” an idea introduced by miller. in his study, miller (1986) proposed four levels of intensity: incidental use, substantive use, important use, and fundamental use. miller based his analysis on seven data elements pertaining to characteristics of the resource consulted and the nature of the citation. goggin (1986) similarly attempted to demonstrate the value of collection through usage by examining callslips and citations to materials from the library of congress. these early attempts to understand the impact of special collections on scholarship have not been followed up in more recent years even though methods of citation analysis and visualizations of scholarly networks have become more sophisticated.   the interplay between metrics and values can also be demonstrated by looking at how archives and special collections support the teaching mission of the university. special collections generally collect data on the number of instructional sessions presented to visiting classes and the numbers of local students who use the reading room. nevertheless, as the arl spec kit survey on “special collections engagement” cited above has shown, repositories have struggled to assess the impact of special collections on learning outcomes (berenbak et al., 2010).   the impact is potentially large. in the final beta testing of the archival metrics project (n.d.) “student researcher questionnaire” in 11 classes at two universities, we found that 92% (n=444) of the students enrolled in these courses had never used archives or special collections before. ninety-six percent said that they would return if they had another project that would benefit from the use of primary sources (daniels & yakel, in press). although one of the primary arguments levied against special collections has been that their collections are esoteric and outreach insignificant, these findings indicate that much value can be gained from having undergraduates engage with special collections.   the archival metrics project along with other studies by duff and cherry (2008) and krause (2010) provide further options for demonstrating the impact of special collections and archives on student learning. duff and cherry measured the effect of archival orientation programs on student confidence in undertaking archival research. the archival metrics “student researcher questionnaire” was similarly designed to measure confidence and also asks whether skills learned as a part of archival assignments are transferrable to other courses. krause (2010) conducted a large-scale field experiment to gauge the effect of archival instruction. her results showed that such instruction helps students develop their critical thinking about evidence. a wider diffusion of these and other evaluation and impact measures are needed to more fully understand the value of special collections to higher education.   discussion   special collections and archives can and do contribute unique value to research and learning, but their value has not been effectively communicated due to a lack of standards and best practices for measuring and assessing their impact. although past efforts to define and operationalize special collections and archival metrics have not met with much success, the current focus of research libraries on value propositions and return on investment provides a new opportunity to remedy the deficiency. as we have shown with our proposal for a reader-hour metric, some solutions may only require identifying appropriately precise variations of existing measures that can be applied objectively and universally. in other cases, the solution may simply involve making wider use of available tools, such as the archival metrics user surveys.   the key in every case is to define metrics and assessment techniques that are user-centric, that is, defined around user perceptions and demonstrations of impact on the user. in 1997, saracevic and kantor introduced the idea of a user-centered approach concentrating on the impact of library service, which they called “value as results” or “value in use” (p. 540). in the value of academic libraries, oakleaf expands on this concept and argues for the importance of demonstrating value-on-investment measures that show impact or the differences that libraries make in the lives of their users (2010). creating an identifiable link between the value proposition and demonstrated value for the user also needs to be done for special collections and archives. quantitative approaches that measure intensity of use offer one possibility, while qualitative interview and sampling techniques offer another. chapman and yakel have observed that although neither value nor impact “can be determined solely by quantitative means, operational metrics can be structured to better support research into value propositions and impact” (2012, p. 149). they have likewise elaborated a rationale for making better use of operational data collected by special collections and archives reading rooms by highlighting several successful case studies. nevertheless, they have also recognized, like dooley and luce (2010), that “[t]he absence of data definitions makes it difficult for repositories to model and collect data in a coherent and consistent way over time” (p. 150). to address this deficiency and the need to formulate more appropriate usage metrics for special collections, rbms formed a metrics and assessment task force in july 2011 and charged it with “examining current practices for gathering and reporting information to demonstrate the value and impact of special collections and archives” (association of college and research libraries, rare books and manuscripts section, 2011). the task force will provide a report and initial set of recommendations to the rbms executive committee by june 2013, including the identification of specific guidelines and best practices that should be developed to support meaningful assessment activities. in the meantime, dupont (2012) has served as guest editor of special issue of rbm: a journal of rare books, manuscripts and cultural heritage devoted to assessment in special collections and archives. the goal of defining usage metrics for special collections and archives at academic institutions is ultimately to better assess and articulate their value propositions in the context of the rapidly evolving landscape of research libraries.     references   american libraries. (2000). what’s so special about special collections? [special section]. american libraries, 31(7).   archival metrics project. (n.d.). retrieved 24 may 2013 from http://archivalmetrics.org/   association of college and research libraries, rare books and manuscripts section. (2011). metrics and assessment task force. retrieved 24 may 2013 from http:// http://www.rbms.info/committees/task_force/metrics_assessment/   association of research libraries. (2001). building on strength: developing an arl agenda for special collections. program and selected presentations, june 27, 2001, john hay library, brown university. retrieved 18 mar. 2013 from http://www.arl.org/storage/documents/publications/building-on-strength-developing-an-arl-agenda-for-special-collections-june2001.pdf   association of research libraries. (2003). research libraries and the commitment to special collections. washington, dc: association of research libraries. retrieved 18 mar. 2013 from http://www.arl.org/storage/documents/publications/special-collections-statement-of-principles-2003.pdf   association of research libraries. (2006). special collections task force final report. washington, dc: association of research libraries. retrieved 18 mar. 2013 from http://www.arl.org/storage/documents/publications/special-collections-task-force-final-status-report-july2006.pdf   association of research libraries. (2009a). an age of discovery: distinctive collections in the digital age. proceedings of the arl-cni fall forum, october 14–15, 2009, washington, dc. retrieved 27 may 2013 from http://www.arl.org/publications-resources/search-publications/search/summary?term[0]=17&term[1]=&term[2]=&term[3]=73&term[4][]=103   association of research libraries. (2009b). arl statistics questionnaire, 2008-09: instructions for completing the questionnaire. washington, dc: association of research libraries. retrieved 27 may 2013 from http://www.libqual.org/documents/admin/09instruct.pdf   association of research libraries. (2010a). arl partners in grant to study value of academic libraries [press release]. retrieved 29 may. 2013 from http://www.arl.org/news/arl-news/2747-arl-partners-in-grant-to-study-value-of-academic-libraries   association of research libraries. (2010b). transforming special collections in the digital age working group [working group charge]. retrieved 24 may 2013 from http://www.arl.org/storage/documents/publications/scwg-charge-final-11feb10.pdf   association of research libraries, working group on special collections. (2009). special collections in arl libraries: a discussion report. washington, dc: association of research libraries. retrieved 24 may 2013 from http://www.arl.org/storage/documents/publications/scwg-report-mar09.pdf   barnes, c., blake, h., & david pinder, d. (2009). creating and delivering your value proposition: managing customer experience for profit. london & philadelphia: kogan page.   berenbak, a., putirskis, c., o’gara, g., ruswick, c., cullinan, d., dodson, j. a., walters, e., & and brown, k. (2010). special collections engagement [arl spec kit 317]. retrieved 24 may 2013 from http://www.arl.org/news/pr/spec317-august10.shtml   boyd, b. (2000). serendipity of the new. rbm: a journal of rare books, manuscripts, and cultural heritage, 1(1), 36-37.   campbell, e. g. (1967, april 9). [letter to philip p. mason]. university of wisconsin-milwaukee, archives and special collections (collection rg 200/01/06, box 1, folder 56: society of american archivists records, president, herbert e. angel, uniform archival statistics, 1966-67).   chapman, j. & yakel, e., (2012). data-driven management and interoperable metrics for special collections and archives user services. rbm: a journal of rare books, manuscripts, and cultural heritage, 13(2), 129-151.   daniels, m. g., & yakel, e. (in press). uncovering impact: the influence of archives on student learning. journal of academic librarianship.   de hamel, c. (2000). tangible artifacts, rbm: a journal of rare books, manuscripts, and cultural heritage, 1(1), 27-29.   dooley, j. m., & luce, k. (2010). taking our pulse: the oclc research survey of special collections and archives. dublin, oh: oclc research. retrieved 24 may 2013 from http://www.oclc.org/research/publications/library/2010/2010-11.pdf   duff, w. and cherry, j. (2008) archival orientation for undergraduate students: an exploratory study of impact. american archivist, 71(2), 499-530.   duff, w., yakel, e., tibbo h., cherry, j., krause, m., & sheffield, r. (2010). the development, testing, and evaluation of the archival metrics toolkits. american archivist, 73(2), 569-599.   dupont, c. (ed.). (2012). assessment in special collections and archives. [special issue]. rbm: a journal of rare books, manuscripts, and cultural heritage, 13(2). retrieved 24 may 2013 from http://rbm.acrl.org/content/13/2.toc   goggin, j. (1986). the indirect approach: a study of scholarly users of black and women’s organizational records in the library of congress manuscript division. midwestern archivist, 11(1): 57-67.   hewitt, j. a., & panitch, j. m. (2003). the arl special collections initiative. library trends, 52(1), 157-171.   howarth, r. (2000). what’s the use? rbm: a journal of rare books, manuscripts, and cultural heritage, 1(1), 42-43.   international organization for standardization. (2008). iso 11620:2008, information and documentation: library performance indicators. available from http://www.iso.org/iso/catalogue_detail.htm?csnumber=37853   katchen, m. (2000). archiving the avant-garde. rbm: a journal of rare books, manuscripts, and cultural heritage, 1(1), 38-41.   koda, paul s. (2008). the past is more than prologue: special collections assume central role in historical research and redefine research library collections. library quarterly, 78(4), 473-482.   krause, m. (2010). undergraduates in the archives: using an assessment rubric to measure learning. american archivist, 73(2), 507-534. retrieved 24 may 2013 from http://archivists.metapress.com/content/72176h742v20l115/   lavoie, b. f., connaway, l. s., & dempsey, l. (2005). anatomy of aggregate collections: the example of google print for libraries. d-lib magazine, 11(9). retrieved 24 may 2013 from http://www.dlib.org/dlib/september05/lavoie/09lavoie.html   massachusetts institute of technology, global airline industry program. (n.d.). airline data project: glossary. retrieved 24 may 2013 from http://web.mit.edu/airlinedata/www/res_glossary.html   miller, f. (1986). use, appraisal, and research: a case study of social history. american archivist, 49(4), 371-392.   national information standards organization. (n.d.). niso z39.7-201x, information services and use: metrics & statistics for libraries and information providers—data dictionary. retrieved 24 may 2013, from http://www.niso.org/dictionary/appendices/appendixa   nichols, s. g. (2009). why special collections matter now more than ever before. paper presented at arl-cni 2009 fall forum. washington, dc: association of research libraries. retrieved 29 may 2013 from http://www.arl.org/storage/documents/publications/ff09-nichols.mp3   oakleaf, m. (2010). the value of academic libraries: a comprehensive research review and report. chicago: association of college and research libraries. retrieved 29 may 2013 from http://www.ala.org/ala/mgrps/divs/acrl/issues/value/val_report.pdf   panitch, j. m. (2001). special collections in arl libraries: results of the 1998 survey sponsored by the arl research collections committee. washington, dc: association of research libraries.   saracevic, t., & kantor, p. b. (1997). studying the value of library and information services; part i: establishing a theoretical framework. journal of the american society for information science, 48(6), 527-542.   schreyer, a., schaffner, j., bowen, s. & steele, v. (2009). public services and ‘un-hidden’ collections: what we know and what we need to know. panel presentation at the 2009 preconference of the rare books and manuscripts section of the association of college & research libraries, charlottesville, virginia, 17-20 june 2009. retrieved 24 may 2013 from http://www.rbms.info/conferences/preconfdocs/2009/2009docs.shtml   value proposition. (2012). in wikipedia, the free encyclopedia. retrieved 4 dec. 2012 from http://en.wikipedia.org/w/index.php?title=value_proposition&oldid=556435992   waters, d. (2009). the changing role of special collections in scholarly communications. research library issues, 267(3). retrieved 29 may 2013 from http://publications.arl.org/rli267/31   whittaker, b. m. (2008). using circulation systems for special collections: tracking usage, promoting the collection, and addressing the backlogs. college and research libraries, 69(1), 28-35. retrieved 24 may 2013 from http://crl.acrl.org/content/69/1/28.full.pdf   yakel, e., & goldman, e. (2002). archival metrics: an overview of current practices. unpublished manuscript. evidence summary   the launch of a joint library/writing centre online course on academic integrity   a review of: greer, k., swanberg, s., hristova, m., switzer, a. t., daniel, d., & perdue, s. w. (2012). beyond the web tutorial: development and implementation of an online, self-directed academic integrity course at oakland university. the journal of academic librarianship, 38(5), 251-258. doi: 10.1016/j.acalib.2012.06.010   reviewed by: cari merkley associate professor mount royal university library calgary, alberta, canada email: cmerkley@mtroyal.ca   received: 3 mar. 2013    accepted: 21 apr. 2013      2013 merkley. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to outline the collaborative development of an online course addressing academic integrity by a university’s library system and writing centre.   design – case study.   setting – a public research university in the midwestern united states.   subjects – 1650 students who completed the online module.   methods – oakland university (ou) libraries and the writing centre began to collaborate on the development of a new online course on academic integrity in 2011. it was felt that an existing online library tutorial on plagiarism no longer met the needs of students and faculty. the development of the course was informed by the association of college and research libraries’ information literacy competency standards for higher education (2000) as well as a research study investigating students’ use of sources in their scholarly writing across several institutions. moodle, the institution’s learning management system (lms), was used to develop the learning object.   main results – ou libraries and the writing centre launched the six-part online course entitled “using and citing sources” in january 2012. they developed modules around learning outcomes in five broad categories: defining academic integrity and plagiarism; the use of sources in academic writing; paraphrasing; quoting; and citation. the final module provided students with an opportunity to practise lessons learned in the first five modules. the use of the lms to design and host the course limited the tutorial to registered students, but provided developers with access to additional course functionality without labour-intensive coding. it also allowed writing centre staff to access students’ performance data on the modules prior to their appointments. improvements over the previous online tutorial included expanded content on academic ethics and referencing, more active learning elements, video content, and the opportunity for students to choose discipline-specific examples. in the first four months of its availability, 1650 students completed the course, with 3330 attempts overall.   conclusion – the diverse perspectives and expertise that individuals from ou libraries and the writing center brought to their collaboration greatly informed the development of the course. the time and effort saved by using the university’s existing lms to develop interactive content and the focus on providing students with opportunities for active learning within the course contributed to the project’s success.     commentary   plagiarism and teaching students how to avoid it are issues academic librarians and their allies continue to grapple with. the authors’ advice to look to outside partners in educating students about academic integrity is well taken, particularly as many libraries find themselves trying to do more with fewer resources. designing learning objects within a university’s existing lms rather than expending the time and money to build them from scratch is also a possibility other libraries should consider.   the fact that this information comes in the form of a case study does prove problematic for the critical appraisal process. one of the most popular study designs for information researchers, the case study has been a target for criticism in evidence based library practice for the lack of rigour and positive bias many such studies display (eldredge, 2004; glynn, 2006). as such, it is not surprising that case studies score low on critical appraisal tools when compared to systematic reviews or randomized controlled trials, and practitioners should keep this in mind when weighing such evidence in their decision making process.   a general critical appraisal tool, however, may fail to highlight best practices in those circumstances where the case study may be the most appropriate study design. in her discussion of quality improvement case studies in the medical sciences, greenhalgh (2010) offers a checklist of ten questions that writers and readers of library case studies may find useful when considering the quality of the research presented. such studies in medicine, like many in library and information science, are often tasked with detailing attempts to improve the overall experience of the patient/client within a complex system of services. several questions stand out when considering the work of greer et al. through this lens. greenhalgh writes, “was the intended quality improvement evidence based?” (192). while the authors of the study reference previous work on the role of libraries in addressing questions of academic integrity and the role of collaborations with other departments on campus in this work, little outside evidence on best practices in designing online education is cited to support the pedagogical decisions made. greer et al. admit that the choice to deliver the content online was for practical reasons (lack of face-to-face time with students), but there are lessons to be drawn from the literature both within and outside of our discipline on how to best go about this.   greenhalgh also asks, “how did the authors measure success and was this reasonable?” (p. 193). the only evidence presented of student performance in the online course is the large number of students (1650) who completed it in the first four months of its availability. this figure does suggest that ou faculty saw value in the content and were assigning it to their students, although more information on the size and composition of the student body at the institution would provide readers with a better sense of the scale of this achievement. the explanation provided by the authors as to why an even larger number of students (3330) tried and failed to complete the course is unsatisfying. for example, the large number of unsuccessful attempts could point to problems with overall course design rather than simply students’ desire for a higher score. student performance data from the individual modules is not provided, even though the study notes that that this information is being collected and used internally to inform future changes to the course. there is no mention of ethics clearance in the study, so one possibility is that the authors did not have permission to release the data. it would also have enriched the study had more information been included as to how the previous online tutorial was assessed and found wanting.   the absence of such assessment data, and the article’s focus on a description of the course’s development rather than an analysis of its impact limits its overall value to the rest of the library community. improving students’ understanding of the ethical issues around their use of information is a worthy goal, but it is not yet clear that this course accomplishes what it sets out to do. there is an indication that the authors’ future reporting on the project will include measures of student performance and faculty feedback, and further details would be welcomed by other libraries looking for ideas on how to best address issues of academic integrity in their own institutions.     references   american library association (2000) information literacy competency standards for higher education. retrieved 1 may 2013 from http://www.ala.org/acrl/standards/informationliteracycompetency   eldredge, j. d. (2004). inventory of research methods for librarianship and informatics. journal of the medical library association, 92(1), 83-90.   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi: 10.1108/07378830610692154   greenhalgh, t. (2010). how to read a paper: the basics of evidence-based medicine (4th ed.). retrieved from ebrary.   microsoft word es_hook.doc evidence based library and information practice 2006, 1:1  98 evidence based library and information practice       evidence summary    use and access of grey literature in special libraries may be hindered by lack of  visibility and cataloguing    a review of:   ranger, sara l. “grey literature in special libraries: access in use.”  publishing research  quarterly 21.1 (spring 2005): 53‐63.    reviewed by:   david hook  manager, operations information and configuration management, mda  brampton, ontario, canada  e‐mail: david.hook@mdacorporation.com    received: 20 december 2005  accepted: 21 february 2006      © 2006 hook. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to examine the barriers to  making grey literature (literature not  controlled by commercial publishers) easier  to access in special libraries.    design – interviews.    setting – variety of special libraries  (government, corporate and specialized  academic) in the united states.    subjects – sixteen librarians from fourteen  organizations in washington, michigan and  texas were interviewed.  four of the  organizations were government libraries,  four were corporate libraries and five were  specialized academic libraries.  one of the  interviews was not used because the  organization did not maintain a collection of  paper‐based grey literature.    methods – librarians were selected as  possible interview subjects via three  methods: some were previously familiar  with the author; some were referred to the  author by friends, family and colleagues;  two candidates volunteered in response to a  presentation of the project at a professional  meeting.  interviews were conducted  between february 2002 and may 2003.  a  standard set of seven questions were used,  but often followed with further questions.   the interviews were conducted either in the  library or the librarian’s office.  the  interviews were tape‐recorded and the  answers were written down.  interviews  typically lasted between fifteen and thirty  minutes and asked about the current state,  holdings, access and use of grey literature in  the special library.    http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2006, 1:1  99 main results – results from the interviews  suggest a wide variance in the percentage of  users that access grey literature.  grey  literature was used less in the corporate  libraries than the academic and government  libraries.  the percentage of the collection  made up of grey literature also varied  widely between the different libraries.   reports were found to be the most popular  form of grey literature, although most of the  libraries reported owning conference  proceedings and newsletters in addition to  reports.  one interesting observation found  during the interviews was that most of the  users of grey literature are also producers of  grey literature.  the librarians surveyed  reported that some of the reasons for using  grey literature included use in research, to  write (often more grey literature), interest in  the topic, for class assignments, as records of  previous practices, for localized studies, and  for creating models and practices.  results  found that for the libraries surveyed, much  of the grey literature remains uncatalogued  and what has been catalogued was done  using a variety of methods.  over half of the  libraries surveyed had their grey literature  accessible online.    conclusion – two main reasons were cited  as explanations for why grey literature was  not used as much as it should be: lack of  cataloguing and visibility.  in many of the  libraries surveyed, much of the grey  literature had not been catalogued, making  it difficult to find and use the resources.   reasons cited for not cataloguing grey  literature include lack of time, funds and/or  knowledge.  as well, in many of the  libraries surveyed, it was found that the  holdings of grey literature were not readily  visible to the users, so users were not even  aware that it existed.    to improve the awareness and accessibility  of grey literature, the author recommends  regional depositories for grey literature,  international standards for cataloguing and  more cooperation between special libraries.    commentary    the study attempts to address not only the  research question of how grey literature is  used in special libraries, but also the reasons  why grey literature is not used as much as it  should be.  the methodology, criteria and  survey instrument are all well described.    the major concern with this study is the  selection and size of the respondent group.   the group should have been more  randomly selected, as the author previously  knew most of the respondents.  with groups  of only four corporate, four government and  five academic libraries, the group sizes are  not large enough to accurately compare the  different types of special library or to draw  precise conclusions.      one factor that was not considered in the  study was the relationship between the  subject specialization of the libraries and  grey literature usage.  one would expect  certain subject fields such as business or  science to use grey literature more than  others, such as legal or medical.  the subject  specialization of the libraries was not  indicated in the study.  similarly, the size of  the special libraries selected was not  indicated.    of the thirteen libraries surveyed about their  collections of grey literature, only one  respondent based holding sizes on actual  recent inventories.  for the other twelve  libraries, the librarians estimated the  percentages.  it is not clear how reliable the  estimated percentages are.  the paper does  not state whether the percentage of users  who use grey literature are based on  estimates or actual statistics.  explanations for low use of grey literature  are examined, such as lack of cataloguing  and visibility.  in the case of lack of visibility,  evidence based library and information practice 2006, 1:1  100 however, the conclusions were not tied to  the interview results.    the author states from the outset that  electronic grey literature would not be  considered for the purposes of this paper.   while this may have been necessary to limit  the scope of the study, it would certainly  have been useful to include electronic grey  literature considering the increase in blogs,  wikis, ‘born digital’ documents and other  types of literature that would be considered  grey.  not only is electronic grey literature  increasing in volume, but also in its  availability.  the rise of grey literature could  also be a possible explanation for the low  use of paper‐based grey literature.  this  possibility, however, was not investigated.  considering the importance, availability and  growth of electronic grey literature in  organizations today, excluding it from the  study severely limits its usefulness.    overall, this study is limited in scope and  uses a sample group that is too small, non‐ representative and non‐random to be  considered valid. as such, the study only  contributes some anecdotal information to  our understanding of how grey literature is  used in special libraries.  if the study were to  be reworked to include electronic grey  literature usage with a more representative  sample group, the results would give a  clearer understanding of grey literature  usage that readers would be able to apply in  their own practice.    evidence summary   uk library and information science research is having a significant influence on research in other subject disciplines   a review of: hessey, r., & willett, p. (2013). quantifying the value of knowledge exports from librarianship and information science research. journal of information science, 39(1), 141-150. doi:10.1177/0165551512442476   reviewed by: mathew stone assistant librarian bradford teaching hospitals nhs foundation trust bradford, united kingdom email: mathew.stone@bthft.nhs.uk   received: 30 may 2013  accepted: 10 jan. 2014      2014 stone. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to quantify the value of librarianship and information science (lis) exports knowledge to other subject disciplines.   design – bibliometric study.   setting – lis departments in u.k. universities.   subjects – 232 lis research articles published between 2001 and 2007.   methods – data from the 2008 u.k. research assessment exercise were checked to identify 405 research articles submitted by 10 selected university departments (out of a total of 21), which submitted research in the lis category. the web of science database was then searched to see how many of these articles had been cited in other articles (n=232). if the citing article was published in a non-lis journal it was considered a knowledge export. journals were defined as non-lis if they had not been assigned the subject category of information science & library science by the journal of citation reports. the journal impact factors (ifs) of citing journals were then normalized to measure the value of individual knowledge exports to their respective subject disciplines. this was done by comparing a citing journal’s if with the median journal if within that subject category. if the citing journal’s if was above this median it was considered to be a valuable knowledge export.   main results – the sample of lis research articles produced a total of 1,061 knowledge exports in 444 unique non-lis journals. these non-lis journals covered 146 unique subject categories of which those related to computer science and chemistry/pharmacology cited lis research with the greatest frequency. just over three-quarters (n=798) of these citations were considered to be valuable knowledge exports. a sub-analysis showed that lis articles published in non-lis journals were significantly more valuable than the knowledge exports published in lis journals.   conclusion – the validity of bibliometric studies can be improved by adopting the two methodological innovations presented in this study. the first innovation is to avoid over-estimating the number of knowledge exports by discounting “part exports” (i.e., where the citing journal is assigned to multiple subject categories, one of which includes the same as that of the cited reference). the second innovation introduced by this study is to add an extra dimension to the analysis by measuring the value of each knowledge export by taking into account the “normalized” impact factor of citing journals.       commentary   previous studies have shown that knowledge transfers between subject disciplines are not simply reciprocal in nature but are instead part of a hierarchical structure with certain subjects exerting more influence than others (urata, 1990). since the 1980s knowledge transfers have been increasing across all subject areas with lis reporting the largest increase of all (levitt, thelwall, & oppenheim, 2011). the bibliometric import-export study used here is well established in the literature as a means of quantifying such knowledge transfers. using terms borrowed from economics, a knowledge export is defined as when published research from one subject area is cited by research from another subject area.   defining lis research is less straightforward than one might think. other studies have defined lis research as that which is published in lis journals (defined as such by a journal classification system such as journal citation reports). the authors reject this definition as too narrow arguing that, because of the inherently inter-disciplinary character of lis research and those who produce it, we would expect a good deal of it to be published in non-lis journals (as is confirmed by this study). hence a potential source of selection bias is avoided in this study by collecting a set of articles from recognized lis researchers regardless of where they will eventually be published. the advantage of collecting the articles by this method is that we can be confident that this study is truly quantifying the value of lis research rather than research from other disciplines which may have been published in journals over-sensitively classified as belonging to the lis discipline.   the study sample was adequate in size to produce statistically significant results when comparing the value of knowledge exports published in lis journals with those published in non-lis journals. another aspect of the study sample is that it only included lis research published by academics and so the findings may not be applicable to research published by practising librarians.   a major finding of this study is the association of lis research published in non-lis journals with a higher value of knowledge exports. however, the observational study method employed here by hessey and willett cannot prove that publishing in non-lis journals causes these improved outcomes, even if it is plausible.   the lis community can take encouragement from the news that knowledge exports from their specialist subject area have been shown to be of high value to those areas to which they have been exported. lis researchers who wish to attract the interest of researchers from other subject fields will be more specifically encouraged to publish their work in non-lis journals as this has been associated with a greater value of knowledge export.         references   levitt, j. m., thelwall, m., & oppenheim, c. (2011). variations between subjects in the extent to which the social sciences have become more interdisciplinary. journal of the american society for information science and technology. 62(6). 1118-1129. doi:10.1002/asi.21539   urata, h. (1990). information flows among academic disciplines in japan. scientometrics. 18(3-4). 309-319. doi:10.1007/bf02017767   microsoft word es_corkett.doc evidence based library and information practice 2007, 2:2  113 evidence based library and information practice       evidence summary    struggling to improve our understanding of nursing student information needs          a review of:  gannon‐leary, pat, graham walton, raffik cader, julie derbyshire, and ann smith. “use of  evidence by nursing students: an interdisciplinary study.” library & information  science research 28.2 (mar. 2006): 249‐64.    reviewed by:   michael corkett  information manager, british heart foundation  london, england, united kingdom  e‐mail: corkettm@bhf.org.uk      received: 05 march 2007  accepted: 10 april 2007      © 2007 corkett. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to identify the sources used by  student nurses when undertaking a health  needs analysis of particular client groups,  and to determine students’ ease in accessing  source materials.     design – case series, incorporating citation  analysis and focus groups.    setting – honours nursing studies program  (adult branch), northumbria university,  newcastle upon tyne, united kingdom.    subjects – citation analysis: 40 students  undertaking health needs analysis  assignment. focus groups: 2 groups of 8  students.    methods – subjects for the citation analysis  were not randomly selected. citation  analysis was performed and references  tabulated using a previously piloted pro‐ forma. distribution of references between  sources was measured using quantitative  analysis. focus groups consisted of self‐ selecting volunteers from the student cohort  undertaking the assignment. issues of  potential bias/coercion relating to research  team members being group participant  tutors were addressed. results were  analysed through the categorisation of key  themes.    main results – the mean number of sources  cited per assignment was 16. more than half  (56%) of the sources were published after  1999. twenty‐nine percent of citations were  published between 1996 and 1999, and 12%  between 1990 and 1995. only small  percentages were published in the 1980s  (2%) and 1970s (1%). journals (37%) formed  the largest body of references, followed by  mailto:corkettm@bhf.org.uk http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:2  114 books at 27%. uk government publications  accounted for 14% of total references; web‐ based resources totalled 12%. students  acknowledged the importance of supporting  assignments with references. government  publications were recognised as high‐ quality pieces of evidence. difficulty was  experienced extrapolating evidence from  different care settings and moving outside  of usual parameters. students indicated that  statistical data was hard to find. those  without home internet facilities had greater  difficulty in accessing information than  students with such facilities. support from  information professionals was received, but  quality was variable.       conclusion – doubts exist as to whether  opportunities for student learning to inform  educators are being fully exploited. upon  completion of assignments students are  arguably in a better position to debate  subjects more deeply and so advance  understanding. potential for peer group  development may be lost. collaboration  between nursing academics and information  professionals offers potential for nursing  students and health professionals to  advance their exploitation and evaluation of  information resources.  commentary      the rise of evidence based medicine has  brought into focus the importance of  information literacy skills for nurses.  knowing how to access relevant  information and apply it is a necessary skill  in today’s knowledge and information‐age  world. the primary goal for clinical nurse  educators is to enhance students’ abilities to  identify, read, critique, and apply literature  from multiple disciplines to a clinical realm,  while maintaining their focus on their  health‐related discipline.      the exploration of student nurse informatics  requirements and the endeavours of  information professionals to understand and  satisfy them sustain a consistently steady  volume of literature. this study fails to  supplement what is becoming an  increasingly rich evidence base. caution  should be exercised when considering this  study, and the authors do acknowledge the  limitations of their research.     the study design is appropriate and clearly  described. however, the small population  size undermines the validity of the overall  study. while small scale studies can provide  benefits to a single institution, results from  larger populations permit meaningful  conclusions to be drawn and applied in a  wider environment.      basic quantitative analysis of citations  revealed correlations with previous studies,  both in terms of mean number of citations  and date ranges of cited material.  distribution of references by media type  again correlated with earlier studies.  unfortunately, interesting questions that  emerged, for instance whether journals  utilised by students were paper‐based or  electronic, do not appear to have been have  been followed up in focus group discussions.      qualitative analysis of focus group  discussion is limited, and there is little  evidence of meaningful interpretation of key  themes. again, interesting issues are  touched upon – the utilisation of online  databases by students, identification of  statistical data, extrapolation of evidence  between care settings, and internet access –  but analysis and insight is lacking.      this is a distinctly weak piece of research,  compounded by a reporting style rich on  citation of other studies, but short of  meaningful primary data and analysis.  whilst the study’s subjects indicated that to  satisfy their objectives they had to think  more laterally, this is something that the  authors themselves have struggled to  evidence based library and information practice 2007, 2:2  115 achieve. consequently, they have offered  the wider health community a work of  marginal interest that fails to significantly  build on previous research.    conference paper   business intelligence infrastructure for academic libraries   joe zucca director for planning and organizational analysis university of pennsylvania libraries philadelphia, pennsylvania, united states of america email: zucca@upenn.edu      2013 zucca. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to describe the rationale for and development of metridoc, an information technology infrastructure that facilitates the collection, transport, and use of library activity data.   methods – with the help of the institute for museum and library services, the university of pennsylvania libraries have been working on creating a decision support system for library activity data. metridoc is a means of “lighting up” an array of data sources to build a comprehensive repository of quantitative information about services and user behavior. a data source can be a database, text file, extensible markup language (xml), or any binary object that contains data and has business value. metridoc provides simple tools to extract useful information from various data sources; transform, resolve, and consolidate that data; and finally store them in a repository.   results – the penn libraries completed five reference projects to prove basic concepts of the metridoc framework and make available a set of applications that other institutions could test in a deployment of the metridoc core. these reference projects are written as configurable plugins to the core framework and can be used to parse and store ez-proxy log data, counter data, interlibrary loan transactional data from illiad, fund expenditure data from the voyager integrated library system, and transactional data from the relais platform, which supports the borrowdirect and ezborrow resource sharing consortiums. the metridoc framework is currently undergoing test implementations at the university of chicago and north carolina state university, and the kuali-ole project is actively considering it as the basis of an analytics module.   conclusion – if libraries decide that a business intelligence infrastructure is strategically important, deep collaboration will be essential to progress, given the costs and complexity of the challenge.   introduction   since the late 1990s, the academic library community has held a wide-ranging discussion on library metrics for the digital age.  beginning in 1998, this conversation took on formal dimensions with two noteworthy developments:  first, the guidelines for measuring the use of electronic resources issued by the international coalition of library consortia (icolc); and second, the emergence in europe of equinox, a project to create performance indicators for the “hybrid” library (international coalition of library consortia, 2006). soon after, the association of research libraries (arl) (2001) identified electronic use statistics as a key priority for its statistics program, and launched the e-metrics project.    the arl effort eventually broadened into an attempt to restructure the canon of statistics that describes and tracks the value of library services in the 21st century. it has long been recognized that the traditional arl statistical corpus – holdings, expenditure, and staff size – cannot adequately represent library contributions to academic outcomes, or engagement with the strategic interests of the academic community, such as library support for collaborative methods of teaching and learning, e-science and e-research, and the globalization of higher education.   even as the search for more relevant metrics has unfolded, academic libraries have been buffeted by paradigm-altering events. they have seen their purchasing power erode, their budgets constrict, and their audiences shift to powerful new commercial information services, such as google and amazon. in their planning, libraries have had to tackle difficult questions about their very nature and purpose in the academy.  to quote one study: “unless libraries take action…they risk being left with responsibility for low-margin services that no one else (including the commercial world) wants to provide” (“a continuing discussion”, 2008, p. 4).   academic libraries, regardless of carnegie designation, share a common mission to support the teaching and learning enterprise, and the fulfillment of that mission amid today’s pressures is increasingly linked to intelligence about resource consumption, service quality, and the library’s impact on research and student learning.  clearly, libraries have entered a period where measurement and mission are inextricably linked, where effective management is evidence based management (wilson, 2008).   the challenges of the past decade have sparked a keen interest in assessment and an even sharper focus on accountability and the elusive questions of what to measure and how (luce, 2008).  arl’s commendable reevaluation of the statistical canon notwithstanding, only nominal progress has been made on new metrics or on the critical problem of assembling data for effective, cost-efficient, and sustainable assessment. further, some of the most promising work has originated outside the arl community, for example, in the los alamos digital library’s mesur initiative and project counter. jisc is another source of good recent work that sheds light on tools, methods, and developmental pathways for business intelligence in libraries (kay &van harmelen, 2012).   arl has had notable success at building a nascent community of practice around library assessment, elevating quantitative methods employed within the community through libqual®  and other initiatives. but if libraries are to link evidence to management and planning effectively, the assessment effort will require additional focus, leadership, tools, and technical infrastructure.  the thrust toward evidence based management has been particularly hobbled by the problem of gathering and mining information from data—vast amounts of data arising from service and user interaction with librarians. until data can be quickly and routinely harvested and made ready for study, the evolving community of practice, along with effective leadership in assessment, will struggle to coalesce.   this situation seems paradoxical given that nearly every library service leaves some kind of data trail to mine, from circulation records to e-journal logs to emails about research questions. enormous in size and potential, these trails of evidence are as inaccessible as they are ubiquitous; they are locked up in silos that bar retrieval and thwart investigation; they are expensive and complicated to render usable. at the present time, assessment’s most critical assets are, in effect, the detritus of library systems–traces in the clickstream captured by some log or millions of transaction records stored in an esoteric database table.    libraries are not wanting for analytical methods, even if the data they need are hard to reach. a variety of protocols has been developed in recent years including: a means to analyze the depth of reference services, to measure the impact of networked electronic resources, and to estimate return on investment (roi) in academic libraries (gerlich & berard, n.d.; franklin & plum, 2008; kaufman, 2008). but in each case, the commodity most critical to sustained, productive use of these methods is also the hardest resource to muster. liberating an institution’s data and converting them into knowledge which informs budgetary decisions, staff allocations, new service models, and a sophisticated understanding of research output and scholarly workflows is fundamentally important to evidence based practice and, by extension, to the course of libraries and the universities they serve. duderstadt argues that the evolution of the library in the digital age prefigures the evolution of the university: “in a sense the university library may be the most important observation post for studying how students really learn. if the core competency of the university is the capacity to build collaborative spaces, both real and intellectual, then the changing nature of the library may be a paradigm for the changing nature of the university itself.” (2009, p. 220) this reasoning underscores the critical need for an improved understanding of how scholars interact with and use the services that libraries provide.   meaning and scope of a decision support system   as an enterprise approach to systematic decision support, the university of pennsylvania libraries (penn libraries) is developing metridoc to provide an information technology (it) infrastructure that facilitates the collection and transport of data. as such, our goal is to address the assessment challenge cited above, specifically to unlock the vast and rich data reserves that libraries possess and to tap them for planning and decision-making.    metridoc constitutes several layers of a tiered decision support system (dss). in the literature, the concept of dss has many connotations, which encompass technology but also speak to the non-technical facets of data administration and evidence based management. for present purposes, i follow turban, leidner, mclean, and wetherbe (2004) in describing a dss as:   “a computer-based information system that combines models and data in an attempt to solve semi-structured and some unstructured problems with extensive user involvement.” (pp. 550)   again, following turban et al. the metridoc approach to a dss possesses these features:   1)       data management layer: the range of data that originate from disparate sources and are targeted for harvest into a database or repository layer of the dss. (as turban points out, extract into a database is not a prerequisite of the dss, but that is the method we employ with metridoc.). 2)       model management and data governance: structural components of data that form the building blocks of dss applications and require continuous coordination with the production systems that generate transactional data. 3)       data warehouse: repository of refined, normalized data from raw sources. 4)       user interface layer: a discovery interface that aids users in identifying and isolating relevant data, performs basic aggregation and analysis, and outputs results to dashboards, feeds (rss and/or atom), structured reports, or even integrates with third-party applications such as excel, sas, r, or software environment for the advancement of scholarly research seasr.   as lakos and phipps (2004) have noted, the management of library services employs multiple data sources that often have overlapping relationships, such as the linkages between expenditure and use, or the more complex interconnections between user populations and resource consumption. for this reason, a single, integrated dss should be developed that supports sophisticated use of both descriptive and inferential statistics. the dss should make quantitative information readily available and easy to access by all levels of staff. data should be routinely harvested, modeled, updated, and archived. a management structure should be in place with sufficient staffing and executive support to deal with data governance issues and manage the flow of quantitative information throughout the organization.   options for developing dss capabilities   the case for developing decision support systems for libraries dates back to at least the 1980s (mcclure, 1980). by the late 1990s, the idea had found a prolific champion in amos lakos (1998), whose work with shelley phipps (lakos & phipps, 2004) gives a prominent place to the dss in furthering what is commonly termed the culture of assessment in libraries.   though the need for such systems is well established in the literature, there has been little institutional investment in their creation. lakos cites automated dss systems in some stage of development at only a handful of universities, including the penn libraries’ data farm project, which we discuss in more detail below (1998).   the rarity of dss projects in the academic library community, particularly given the need to clarify mission, optimize finances, and cultivate new services and management methods, testifies to the difficulty and expense of the endeavor.   the commercial development sphere   for the majority of library administrators, keeping pace with mission-critical technologies, such as their integrated library systems (ils) and web applications, absorbs most of the staff and technical expertise available to them. as a result, the appeal of vendor support in this realm is especially strong.   all ils vendors provide some level of report writing, but these capabilities are deeply integrated into the architecture of proprietary systems and thus fail to provide the flexibility or richness of data analysis that libraries need. oclc’s worldcat collection analysis tool is yet another of these “blackbox” solutions. regardless of their strengths or flaws, both the ils and oclc provide business intelligence primarily about print collections; gathering and processing data on other aspects of library services would involve a multiplicity of systems, which works against the need for economy and integration in a dss solution. the dss space is also occupied by commercial firms active in the university market. here the need is for enterprise level data warehousing to provide metrics related to admissions, student performance, retention, and the like. firms in data warehousing have not made a foothold in libraries due to the expense of implementation and support.    whether the commercial sphere is prepared to engage with libraries and the complicated mix of data sources they handle is unclear. libraries need to integrate budgetary data, bibliographic measures, web analytics, personnel information, courseware measures, and a wide range of usage data from local and licensed sources. while library-oriented data warehousing systems have appeared from vendors, they require substantial contributions and start-up costs involving a range of library staff to implement. the ongoing costs for a commercial solution are uncertain, but clearly, libraries will have no control or proprietary stake in the products they are helping vendors to design and market. in the end, a proprietary solution will struggle to satisfy the scope of library needs, but it will add extraordinary new costs and slow deployment of dss technology. the commercial option is also apt to inhibit prospects for multi-institutional collaboration around metrics, just as the commercial ils inhibits cooperative efforts by hardening the silos around data and systems architecture.    community development model   a development role in dss, under an open or community source model, would be advantageous to the library community, specifically enabling:   maximization of local data reserves, effective use and development of domain expertise, financial and functional sustainability, and infrastructure required for collaborative research and development.   community-sourcing does not exclude commercial interests, but changes the fundamental dynamics of the library market, allowing vendors and libraries to forge new relationships around the support of software and the extension of that intellectual property for the best interests of the community. open development of a metrics framework insulates libraries from a destabilizing reliance on vendors for product development and support, while also building a knowledge base that strengthens intraand inter-institutional cooperation around strategic problems.  open development can also spur competency-building within the library community, encouraging the acquisition of statistical skills and creating professional opportunities around data modeling, metadata design, and data governance, in addition to statistical methods and presentation.   metridoc: a system overview   with the help of the institute for museum and libraries services (imls), the penn libraries have been working on the feasibility of creating a dss for library activity data, and have developed a deployable, extensible technology, metridoc, that other libraries can use to broach the challenge. metridoc is a means of “lighting up” an array of data sources to build a comprehensive repository of quantitative information about services and user behavior. a data source can be a database, text file, xml, or any binary object that contains data that has business value. metridoc provides simple tools to extract useful information from various data sources, transform, resolve and consolidate that data, and finally store them in a repository. the repository is comprised of various storage mechanisms to make it easy to extract data for reports and statistical processes. with this in mind, the penn libraries are designing metridoc to meet the following requirements:   create a simple framework that handles the complexities of extracting, resolving and storing data provide hooks into the framework so non-enterprise programmers can use metridoc with a combination of scripting languages, xml and project schemas create reusable solutions specific to the library space, such as extracting data from popular ils systems, handling counter data, resolving ezproxy logs, etc. follow best practices when storing and curating data in the repository to enable the widest possible distribution of decision-support information so that data analysis can become a routine and continuous facet of organizational administration and culture.   metridoc must be understood within the context of the penn libraries’ data farm initiative. the data farm website (http://metridoc.library.upenn.edu/) has authentication controls, but this page suggests features available to staff. that said, a number of data farm functions deliver data on schedules directly to managers and do not required interaction with the web. in addition, penn libraries management information services provide considerable ad hoc analyses from data farm sources.   a program that began in 2000, the data farm represents a substantial institutional investment in assessment. in brief, the data farm is a "collection" of dds functions that run on a common oracle instance and output to the web or excel (cullen, 2005; zucca, 2003). the underlying data come from a variety of sources, for example: the voyager ils system, apache web server logs, a local database that powers segments of the penn libraries website for metrics on e-resource usage, counter data from vendors (this includes a penn-designed sushi harvester which we deploy in metridoc), and input from public services staff who consult with students and do bibliographic instruction. the data farm is also the reporting utility for the borrowdirect and ez-borrow programs (two large-scale resource sharing cooperatives in the northeast). the data farm is used heavily by more than 70 members of these cooperatives, as well as penn libraries bibliographers, public service managers, and strategic planning team. but for all of that, in certain fundamental respects the data farm is a prototype for study and experimentation.   metridoc represents a more rigorous phase of data farm development, and leverages the knowledge the penn libraries have gained since 2000. the key points of distinction between data farm and metridoc are represented in table 1.   the four service layers comprising metridoc support the following functions:   1)       extraction of raw data sources. routines within metridoc are designed to “recognize” specific data structures and extract what is of primary interest to measurement, for example, relevant information from a log or database. 2)       transformation of the raw extract into normalized, decoded information (such as the resolving of issn numbers into a serial title, or an sfx object identifier into citation elements). transformation is a complex but critical process that sets the stage for the third function, 3)       the loading of normalized and anonymized data into a query-able data repository. the fourth metridoc tier sits above the other three (etl) service layers and allows for the integration of the data repository with statistical analysis and visualization tools, or the distribution of flat files for use with statistical programs.   the metridoc service layers are more fully described below and illustrated in figure 1.   1. extraction service – the extraction api, or application programming interface, can be accessed directly with code via scripts. this process creates the payload for ingestion by the metridoc repository – in most cases a data construct that defines a database table and rules for validation.   2. transformation service – data elements within a log stream often include encoded or identity information. encoded data must be resolved to capture the meaningful information for analysis and reporting. for example, digital object identifiers (doi) or issn numbers are commonly used to identify specific instances of articles or journal titles. identity information provides useful demographic class descriptions about a user’s department, status, and rank. the metridoc resolution service consists of processes that tap external data sources, such as national bibliographic utilities or the university data warehouse, and query for matching content from these sources. once deployed, these resolvers can be linked in order to resolve data points iteratively within a log or other data source. the metridoc document is returned to the messaging channel with enriched data about the bibliographic and demographic components of service events.   3. metridoc repository service – metridoc provides a repository service that houses metridoc event data processed from source files and exposes that data for user query and retrieval. this service abstracts the actual data store to provide scalability and flexibility, and can comprise a wide variety of repositories, from relational databases such as oracle or mysql to a mere file system. additionally, abstraction allows storage to be distributed across physical locations for improved resiliency and fault tolerance.   4. data farm service layer – the metridoc architecture abstracts user interaction from the etl components of the framework. in the penn libraries context, interactive services are supported by the data farm service layer, which can be developed using a variety of commercial tools or locally designed solutions. by design, the metridoc repository can be exposed to report-building applications via a restful interface, or to scripts that generate dashboard pages, datasets in excel format for download, or comma delimited files for ingestion into a third-party analytics repository such as ethority. in this last scenario, the data farm service can contain an extensible repository with a library of datasets and data visualizations, and the ability to create refined datasets for analysis, using a statistical language such as r or sas. this service can support analysis tools that are shared across domains to assist in comparison, reporting, and analysis.   table 1 data farm and metridoc structural features data farm structural features metridoc structural features builds a specific extraction and ingestion tool for each type of data source. abstracts the ingestion process and delegates specific extraction to small pieces of code. builds source-specific data structures in an oracle tablespace. generalizes each log transaction into an abstract representation of an “event.” resolves identity and bibliographic data after ingestion. resolves identity data on the fly from a rich and diverse set of resolution sources. exposes a single discovery interface, tightly coupled with the end-user tool. isolates discovery of datasets and provides workflow tools to combine, refine, analyze, and augment data, and then expose it through a multifaceted delivery service layer. comprises a single technology stack. composed of loosely coupled service layers consisting of four distinct services that are integrated through easy-to-use, restful interfaces. figure 1 metridoc tiered architecture   the four metridoc service layers are an orchestrated chain of services that ingest, resolve, normalize, store, index, query, deliver and transform event data regardless of their native structures. it is designed to provide flexibility, extensibility, and consistency to data flows. the technologies used are common in enterprise applications including spring, hibernate, java, and grails.   current development   with funding from the imls received in 2010/11, the penn libraries completed five reference projects to prove basic concepts of the metridoc framework and make available a set of applications that other institutions could test in a deployment of the metridoc core. these reference projects are written as configurable plugins to the core framework and can be used to parse and store ez-proxy log data, counter data (with and accompanying plugin for data harvest with sushi), ill transactional data from illiad, fund expenditure data from the voyager ils, and transactional data from the relais platform which supports the borrowdirect and ezborrow resource sharing consortiums. the projects represent a range of challenges and repository concepts that a dds will encounter in a library setting.   as of this writing the penn libraries are also developing a metridoc module for data related to research consultations and bibliographic instruction services. the metridoc framework is currently undergoing test implementations at the university of chicago and north carolina state university, and the kuali-ole (open library environment) project is actively considering it as the basis of an analytics module that will ship with ole.    benefits of collaboration   the purpose of metridoc is to make available vast, unutilized quantitative information in support of library strategic planning and decision-making. success in this endeavor opens a range of partnership opportunities. deployed in a collective environment, a metridoc-like framework can:   ·         provide libraries a tool for conducting the foundational research leading to new performance metrics; ·         aid cross-institutional study of collections, which advances collaborative collection development; ·         be deployed in resource-sharing initiatives which will help partners identify best practices and optimize the distribution of physical materials; ·         increase an institution’s knowledge of local research interests and patterns through the demographic analysis of transaction records; ·         expose metadata based on resource use to discovery systems for improved resource access and research intelligence; ·         enable the integration of usage and expenditure data to identify cost efficiencies and help libraries apportion budgets more effectively across communities; ·         gather electronic use data on both locally created and licensed digital resources; and ·         provide a platform for relating usage information to customer satisfaction and other parametric measures of quality.   conclusion   powerful new tools for visualizing and distributing data are available to the assessment community. measurement standards for library performance and the potential for creating a robust canon of library metrics are also within reach. the challenge remaining is posed by the data: by the complex and ornery problem of harvesting, structuring, and storing the vast troves of activity data resting dormant in the systems libraries all use to conduct business. metridoc, and etl solutions generally provide an answer to this problem.   the academic library community faces some tough decisions with regard to business intelligence. first, this is not an assessment project, but a matter of technical and staff infrastructure, on the level of our commitments to ils technology and similar it supported functions. it is, additionally, an area requiring development resources, as there are no shrink-wrap solutions for our particular challenges. infrastructure creation and development are expensive activities and will test the importance of business intelligence in the spectrum of this community’s strategic priorities. in the end, libraries will or will not rank this as strategically important. if it is, deep collaboration will be essential to progress, given the costs and complexity of the challenge. a community effort on business intelligence infrastructure can expedite innovation and instigate new relationships among academic institutions and between the academy and commercial sector.  but how will this deep collaboration come about? one wonders if this is an area where arl can be an effective broker, providing a space for potential partners to begin addressing the challenge of creating and governing a critical new infrastructure for managing library services. such an effort is afoot in the u.k. where, under jisc sponsorship, the focus by libraries on activity data is picking up steam and maturing faster than here in the states.  the metridoc effort has joined that conversation even as it looks for development partners closer to home (zucca, 2012).     references   a continuing discussion on research libraries in the 21st century. (2008). in no brief candle: reconceiving research libraries for the 21st century (pp. 1-12). washington, dc: council on library and information resources (clir).   association of research libraries (arl). (2001). e-metrics: measures for electronic resources. in association of research libraries. retrieved january 2010. retrieved 18 may 2013 from http://www.arl.org/stats/initiatives/emetrics/index.shtml   cullen, k. (2005). delving into data: businesses have used data mining for years, now libraries are getting into the act. library journal, 130 (13), 30-32.   duderstadt, j. j. (2009). possible futures for the research library in the 21st century. journal of library administration, 49(3), 217-225. doi:10.1080/01930820902784770   franklin, b., & plum, t. (2008). assessing the value and impact of digital content. journal of library administration, 48(1), 41-47. doi:10.1080/01930820802029334   gentleman, r., & ihaka r., et al. (n.d.) r project for statistical computing. retrieved 18 may 2013 from http://www.r-project.org/   gerlich, b. k., & berard, g. l. (n.d) reference effort assessment data (read) scale. retrieved 18 may 2013 from http://readscale.org/   international coalition of library consortia. (2006). revised guidelines for statistical measures of usage of web-based information resources. in international coalition of library consortia (icolc). retrieved 18 may 2013 from http://icolc.net/statement/revised-guidelines-statistical-measures-usage-web-based-information-resources   kaufman, p. t. (2008, jan.). university investment in the library: what’s the return? a case study at the university of illinois at urbana–champaign. american library association midwinter meeting, philadelphia, pa, usa. retrieved 18 may 2013 from http://hdl.handle.net/2142/3587   kay, d., & van harmelen, m. (2012). activity data: delivering benefits from the data deluge. in jisc. retrieved 18 may 2013 from http://www.jisc.ac.uk/publications/reports/2012/activity-data-delivering-benefits.aspx   lakos, a. (1998). building a culture of assessment in academic libraries: obstacles and possibilities. living the future ii, tucson, az, usa. retrieved 18 may 2013 from http://www.library.arizona.edu/conferences/ltf/1998/proceeds.htm   lakos, a., & phipps, s. (2004). creating a culture of assessment: a catalyst for organizational change. in portal: libraries & the academy, 4(3), 345-361. retrieved 18 may 2013 from http://muse.jhu.edu/journals/portal_libraries_and_the_academy/v004/4.3lakos.html   luce, r. (2008). raising the assessment bar: a challenge to our community. in s. hiller, k. justh, m. kyrilladou, & j. self (eds.), proceedings of the 2008 library assessment conference (pp. 7-11). seattle: association of research libraries. retrieved 18 may 2013 from http://libraryassessment.org/bm~doc/proceedings-lac-2008.pdf   mcclure, c. (1980). information for academic library decision making: the case for organizational information management. westport, ct: greenwood press.   turban, e., leidner, d., mclean, e., & wetherbe, j. (2004). information technology for management: transforming organizations in the digital economy. hoboken, nj: j. wiley.   wilson, b. (2008). accelerating relevance. in s. hiller, k. justh, m. kyrilladou, & j. self (eds.), proceedings of the 2008 library assessment conference (p. 14). seattle: association of research libraries. retrieved 18 may 2013 from http://libraryassessment.org/bm~doc/proceedings-lac-2008.pdf   zucca, j. (2003). traces in the clickstream: early work on a management information repository at the university of pennsylvania. information technology & libraries, 22(4), 175-179.   zucca, j. (2012). metridoc is an extensible framework that supports library assessment and analytics, using a wide variety of activity data collected from heterogeneous sources. in jisc. retrieved 18 may 2013 from http://www.jisc.ac.uk/publications/reports/2012/activity-data-delivering-benefits.aspx#penn     microsoft word news_bob.doc evidence based library and information practice 2007, 2:1  167 evidence based library and information practice     news    call for papers: 16th bobcatsss symposium         © 2007 eblip. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      symposium theme: providing access to  information for everyone    access to information is vital for the  personal and social growth of every  individual. however, many people are  facing barriers in accessing information. the  goal of the bobcatsss 2008 symposium is  to explore access to information for  everybody from the technical, political, legal,  social and economical point of view. the  symposium would like to identify the  existing barriers in accessing information  and to offer an overview of new concepts,  ideas and examples of good practice in  order to raise awareness and achieve  possible solutions. invited are contributions  in a form of a paper, workshop or a poster  covering the following topics:    technical aspects:  • infrastructures, standards and  interoperability for universal access to  information  • open access supported by open source  • digital repositories and documentserver    (long term preservation and access)  • usability, service and interface design        political aspects:  • repression via information ‐  withdrawal/censorship  • bills and laws for lis ‐ an international  composition  • copyright aspects of open access and  open source initiative  • role of national/regional/local  governments in the development of  information institutions    social aspects:  • digital divide  • information professionals in the  information society  • information literacy (a holistic approach)  • information ethics  • information behavior of different users  groups    economical aspects:  • costs of information  • costs of electronic resources  • licensing agreements  • digital library costs    the target group is students. we are  especially inviting workshop contributions  or papers given by students or teams of  students and teachers. further information  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:1  168 and guidelines for the abstracts can be  found on the bobcatsss website  www.bobcatsss2008.org.    abstracts should be sent by 15 may 2007 to  papers@bobcatsss2008.org.    bobcatsss 2008 will be held on january  28‐30 in zadar, croatia.  article   summon, ebsco discovery service, and google scholar: a comparison of search performance using user queries   karen ciccone director of the natural resources library and research librarian for science informatics north carolina state university libraries raleigh, north carolina, united states of america email: kacollin@ncsu.edu   john vickery analytics coordinator & collection manager for social science north carolina state university libraries raleigh, north carolina, united states of america email: john_vickery@ncsu.edu   received: 11 dec. 2014   accepted: 6 feb. 2015           2015 ciccone and vickery. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objectives to evaluate and compare the results produced by summon and ebsco discovery service (eds) for the types of searches typically performed by library users at north carolina state university. also, to compare the performance of these products to google scholar for the same types of searches.   methods a study was conducted to compare the search performance of two web-scale discovery services: proquest’s summon and ebsco discovery service (eds). the performance of these services was also compared to google scholar. a sample of 183 actual user searches, randomly selected from the ncsu libraries’ 2013 summon search logs, was used for the study. for each query, searches were performed in summon, eds, and google scholar. the results of known-item searches were compared for retrieval of the known item, and the top ten results of topical searches were compared for the number of relevant results.   results there was no significant difference in the results between summon and eds for either known-item or topical searches. there was also no significant difference between the performance of the two discovery services and google scholar for known-item searches. however, google scholar outperformed both discovery services for topical searches.   conclusions there was no significant difference in the relevance of search results between summon and eds. thus, any decision to purchase one of those products over the other should be based upon other considerations (e.g., technical issues, cost, customer service, or user interface). introduction   the north carolina state university (ncsu) libraries is a large academic library at a major land-grant research university, serving over 34,000 students. like many similar libraries, it has invested a significant amount of money and staff time in implementing a web-scale discovery service for its collections, and it continues to invest a significant amount of resources in managing its discovery service. we therefore consider it important to periodically evaluate competing products that could potentially provide a less expensive and more effective replacement for our current service, summon.   like other web-scale discovery products, summon provides a pre-harvested central index allowing users to search across a library’s book and journal holdings through a single search box. at the ncsu libraries, a web-scale discovery product team tests and evaluates ongoing upgrades to summon and provides critical feedback to the vendor, proquest. it also investigates alternatives to the libraries' current discovery service and reference linking products and makes recommendations for changes or upgrades as needed. the web-scale discovery product team is composed of nine librarians representing the library’s public services, technical services, collection management, and information technology departments.   the ncsu libraries purchased the summon discovery service in 2009, at which time there were few competitors on the market, none offering all of the features of summon. specifically, we needed a product that had an application programming interface (api) that could be used to populate the articles portion of our quicksearch application (http://search.lib.ncsu.edu/). a search in this tool presents separate results for articles, books & media, our website, and other categories of information (figure 1; see also lown, sierra, & boyer, 2013). in 2009, summon was the only discovery service with this feature. since then, other products, in particular ebsco discovery service (eds), have added an api that can be used this way. this and other developments led the web-scale discovery product team to decide that eds warranted fresh investigation and comparison to summon. we obtained a trial to eds in april and may of 2014, and it was during that time that this study was conducted.   literature review   ellero (2013) offers a literature review on the evaluation and assessment of web-scale discovery services. this literature focuses primarily on usability studies and criteria for choosing a web-scale discovery service, with little or no emphasis on search performance. there are few studies specifically comparing the search performance of web-scale discovery services with each other, or with google scholar. of those, most base their evaluation of search performance on a very small sample of searches (e.g., timpson & sansom, 2011; zhang, 2013). the studies below represent more extensive attempts to compare the search performance of these products.     figure 1 the ncsu libraries’ quicksearch interface.     asher, duke, and wilson (2012) compared the search performance of google scholar, summon, and eds. in their study, quality was judged on the basis of whether each article was from a scholarly source or from “non-peer-reviewed newspapers, magazines, and trade journals” (p. 470). performance scores were given to each product based on librarian quality ratings of the resources selected by test subjects, who had been asked to perform typical search tasks. using this methodology, the authors found that eds produced the “highest quality” results.   while article quality is important, there is more than one way to ensure that library patrons receive scholarly results. at the ncsu libraries, we pre-filter our users’ summon results to include only journal articles and book chapters. thus, the relevance – i.e., the degree of relatedness to the topic being searched – of the remaining results is a more important factor for our users. there is therefore a need for a study comparing the relevance of results of various web-scale discovery products, regardless of format or peer review.   while the asher et al. (2012) study was based on searches performed by test subjects, more accurate assessments of user behavior and experience can be made using actual queries from search logs. tasks given in test situations are artificial, and the behavior of test subjects is influenced by the testing situation. in contrast, search logs contain the searches library users actually perform. for this reason, search log data are likely to provide better information about search performance as experienced by users in their day-to-day use of web-scale discovery services.   rochkind (2013) compared user preference for search results produced by eds, summon, ebscohost “traditional” api, ex libris primo, and elsevier’s scopus. his survey tool allowed subjects to enter search terms of their own choosing and view side-by-side results, within the survey window, for two randomly chosen products. each product was configured to exclude non-scholarly content. users were asked to indicate which set of results they preferred, with an option for “can’t decide/about the same.” the study found no significant difference in preference between products, with the exception of scopus (which was less preferred).   as with the asher et al. study, the artificial testing situation and use of test subjects in the rochkind study is problematic. as rochkind (2013) noted,   when i experimented with the evaluation tool, i found that if i just entered a hypothetical query, i really had no way to evaluate the results. i needed to enter a query that was an actual research question i had, where i actually wanted answers. then i was able to know which set of results was better. however, when observing others using the evaluation tool, i observed many entering just the sort of hypothetical sample queries that i think are hard to actually evaluate realistically.   rochkind also noted that,   this issue does not apply to known-item searches, where either the item you are looking for is there at the top of the list, or it isn’t. looking through the queries entered by participants, there seem to be very few ‘known item’ searches (known title/author), even though we know from user feedback that users want to do such searches. so the study may not adequately cover this use case.   the use of search log data solves both of these problems by providing queries representing users’ actual research questions as well as a sample of searches that accurately reflects the relative frequency of known-item to topical searches.   objectives   our primary objective was to answer the question of whether summon or eds produced better results for the types of searches typically performed by our users. we know, from examining user search logs, that about 24% of the searches performed by our users are for known items – specific articles or books that the user attempts to retrieve through title keywords, a combination of title and author keywords, or a pasted-in citation. we also know that about 74% of the searches performed by our users are topical, with subjects often defined through the use of only two or three keywords (table 1). approximately 52% of the topical queries in our sample, or 42% of the total sample, used three or fewer words. known-item and topical searches represent very different use cases and present very different challenges for a discovery layer. we therefore wanted to separately evaluate how well summon and ebsco performed for each of these types of searches.   while discovery services offer several advantages over google scholar (e.g., api available, ability to save and email results, ability to limit to peer-reviewed articles), we know that the latter is the first go-to search tool for many researchers. for this reason, a secondary objective of this study was to compare the search performance of summon and eds to that of google scholar. google scholar’s terms of use do not allow for its results to be presented in a context outside of google scholar, so replacing our discovery service with google’s free service is not currently tenable. nonetheless, we were interested to know whether or not our users have as much success searching with our discovery service as they do when searching google scholar. we hoped that our discovery service would perform at least as well as, if not better than, google scholar, for the types of known-item and topical searches our users typically perform.     table 1 examples of known-item and topical search queries from our sample known-item search queries: topical search queries: personal characteristics of the ideal african american marriage partner a survey of adult black men and women adderall edema national cultures and work related values the hofstede study solar power coating nanoparticle adalimumab induces and maintains clinical remission in patients with moderateto-severe ulcerative colitis experimentation on animals hill bond mulvey terenzio conjugated ethylene uv vis sullivan a, nord ce (2005) probiotics and gastrointestinal diseases. j intern med 257: 78–92 religiosity among phd bryan; griffin et al; tierney and jun sleep deprivation emotional effects bone graft substitutes expert rev med devices 2006 czech underground ann. appl. bioi. 33: 14-59 toxicology capsaicin     methods   in order to truly measure how well the products performed for our users, we used actual user search queries from our summon search logs. these searches sometimes contained typos, punctuation errors, extraneous words, and other characters. they were also sometimes overly broad or otherwise problematic from the standpoint of obtaining a useful list of article results (table 2). we kept these search terms as they were entered, both in order to compare how well the two products dealt with the types of errors typically found in user searches and to get a fair idea of our users’ actual search experiences.     table 2 examples of problematic search queries from our sample problematic search queries: compendium of transgenic crop gplants suicide collegte the over use of vaccinations new class of drugs patent technology behind online gaming disney world facts divorce effects on childreen abortion is not morally permissible     by using a random sample of queries from our summon search logs, we hoped to create a dataset truly representative of our users’ searches, both in terms of the ratio of known-item to topical searches and in the types of known-item and topical searches entered. the dataset also reflects the range of disciplines represented by our users.   a computer-generated random sample of 225 search queries was obtained from the approximately 664,000 summon searches performed between january 1 and december 31, 2013. the sample was obtained using proc surveyselect in sas software. the sample size of 225 was deemed large enough to be representative of the population yet small enough to be manageable by the team doing the testing. the nine members of the web-scale discovery product team were each given a portion of this sample (25 queries) to analyze. two team members were unable to complete their portion of the testing, and two queries in the sample were found to be uninterpretable, resulting in 183 queries being tested.   team members classified queries as topical search queries or known-item search queries. for each query, team members performed the search in summon, eds, and google scholar and entered the data into the spreadsheet. the spreadsheets were then combined and the data was analyzed using sas software.   for topical search queries, the number of relevant results within the first ten results was recorded. team members were instructed to consider a result relevant if it matched the presumed topic of the user’s search. relevance was judged based on information in the title and abstract only.   for known-item search queries, team members coded “yes” or “no” responses to the following questions:   ●        did you find the item? ●        was it in the top three results?   two versions of the analysis were performed, in order to compare summon directly to eds as well as to compare both discovery services to google scholar. the first analysis compared only the summon and eds data. the second analysis also included the google scholar data.   for the summon and eds comparison of performance for topical search queries, we used three methods. we graphically compared the distribution of the number of relevant results. we performed a matched pair t-test to assess whether the mean numbers of relevant results for the two discovery services were statistically different. lastly, we performed a bootstrap permutation test to compare the means of the paired data.   for the summon and eds comparison of performance for known-item search queries, we graphically compared the number of found known items.   for the comparison including google scholar, the topical search queries analysis was expanded to include a permutation test for repeated measures analysis of variance and pairwise permutation tests for comparing the means of the paired data. the known-item search queries analysis consisted of a graphical comparison of the number of found known items and a mantel-haenszel analysis to examine the relationship between discovery product and success of a known-item search.   results   summon and eds comparison for topical searches   of the 183 queries in our sample, 137 were classified as topical search queries. graphical comparison of the distributions of the number of relevant results from summon and eds shows similar performance (figure 2). while summon had a greater number of queries with ten relevant results, it also had a greater number of queries with zero relevant results.     figure 2 frequency distribution of the number of relevant results from eds and summon.     as each topical search query was tested in both summon and eds, the number of relevant results for each query was considered a matched pair. the matched pair t-test tests the hypothesis that the difference between sample means for the paired data is significantly different from zero (“t test for related samples,” 2004). there was not a significant difference in the mean number of relevant results for eds (m=4.83, sd=3.62) and summon (m=4.76, sd=3.81); t(137)=0.26, p=0.7924. figure 3 shows the distribution of the difference in the mean number of relevant results between summon and eds. a post-hoc power analysis using sas software showed that the sample size of n=137 was sufficient to detect an effect of at least 1 mean difference at  (1 β) of  > 0.99. this is above the standard power (1 β) of 0.80.   in order to account for possible violation of assumptions for the t-test, we performed a bootstrap permutation test (good, 2005; anderson, 2001). the permutation test also showed that there was no significant difference in the mean number of relevant results for summon and eds. figure 4 shows that 10,000 simulations of the difference between summon and eds agree with the matched-pair t-test.     figure 3 distribution of the difference in the mean number of relevant results between eds and summon shows no significant difference, with 95% confidence interval for mean.     figure 4 bootstrap distribution under null hypothesis for 10,000 resamples shows that the observed difference in the mean number of relevant results between summon and eds was not significant.   summon and eds comparison for known-item searches   forty-four queries in our sample were classified as known-item search queries. between summon and eds, the frequency of items found and not found was exactly equal (figure 5).   the team also recorded whether or not a found known item was returned within the top three results for each discovery service. summon and eds performed nearly identically here as well. all but one of the found known items for summon and all but two for eds were in the top three results.   topical search comparison including google scholar   as with the analysis of only summon and eds results, graphical comparison of the distributions of the number of relevant results across all three products shows similar performance. figure 6 shows the frequency of the number of relevant results for summon, eds, and google scholar.   google scholar had the highest number of queries with ten relevant results. it also had the lowest number of queries with zero relevant results.   the mean number of relevant results for each product is listed in table 3.     figure 5 frequency of known items found for eds and summon.     figure 6 frequency distribution of the number of relevant results from eds, google scholar, and summon.     table 3 mean number of relevant results for each discovery product discovery product mean number of relevant results summon 4.76 eds 4.83 google scholar 5.68     figure 7 bootstrap distribution of the f-statistic under the null hypothesis for 10,000 resamples indicates an overall difference between the mean numbers of relevant results for eds, google scholar, and summon.     a permutation test for repeated measures analysis of variance was used to detect any overall difference between the three related means (good, 2005; howell, 2006). ten thousand simulations of the f-statistic indicate that there is an overall difference (figure 7).   given the indication of an overall difference in the mean number of results between the three products, pairwise permutation tests were done to confirm where the difference occurred. these tests compared summon to eds, eds to google scholar, and summon to google scholar. given the agreement between the matched-pair t-test and permutation test for summon and eds, and the potential violation of t-test assumptions, we felt the permutation tests alone would be appropriate for pairwise comparisons of summon, eds, and google scholar. as with the original comparison between summon and eds, there was no significant difference in the mean number of relevant results between those two products. there was, however, a significant difference between the mean number of relevant results for google scholar and both eds and summon.   in the observed data, google scholar outperformed eds by an average of 0.85 relevant results. as shown in figure 8, 10,000 simulations of the data indicate that it is highly unlikely that this difference was due to chance alone.     figure 8 bootstrap distribution under the null hypothesis for 10,000 resamples shows that the observed difference in the mean number of relevant results between google scholar and eds was significant.     in the observed data, google scholar outperformed summon by an average of 0.91 relevant results. as with the eds and google comparison above, 10,000 simulations of the data indicate that it is highly unlikely that the difference was due to chance alone (figure 9).     figure 9 bootstrap distribution under the null hypothesis for 10,000 resamples shows that the observed difference in the mean number of relevant results between google scholar and summon was significant.     known-item search comparison including google scholar   as shown in figure 10, the proportion of known-items found by summon, eds, and google scholar was essentially the same.   the team also recorded whether or not a found known item was returned within the top three results for each product. all three performed nearly identically in this regard. all but two of the found known items were in the top three results for eds and google scholar, and all but one was in the top three for summon.     figure 10 frequency of known items found for eds, google scholar, and summon.     adjusting or controlling for the sample query, no significant difference was found between summon, eds, and google scholar success rates, χ2 (2, n = 132) = 0.08, p = 0.96. however, the small sample size for known items (n=44) means that our test did not have the power to detect small differences (<40%) in the performance of the products.   discussion   very few studies have compared the search performance of web-scale discovery services, such as summon and eds, to each other or to google scholar. of those that have been conducted, most base their evaluation on a very small sample of searches, and all rely on the use of test subjects in artificial testing situations. this study contributes to our knowledge of the comparative search performance of these products by using a large number of actual user search queries as the basis for analysis.   the relevance of search results is of primary importance in comparing the performance of search engines. while there are ways to pre-filter search results to ensure that patrons receive scholarly results, all web-scale discovery products must deliver results that patrons recognize as related to the topic of their query. by focusing our evaluation of topical queries upon relevance, this study fills a need for comparative information about the relevance ranking algorithms of various web-scale discovery products and google scholar.   our analysis showed no significant difference in the search performance of summon and eds for either topical or known-item searches. the number of relevant results for topical searches was the same, and the number of known items found was the same. any decision to purchase one product or the other, therefore, should be based upon other considerations (e.g., technical issues, cost, customer service, or user interface).   google scholar performed similarly to summon and eds for known-item searches, but outperformed both discovery products for topical searches. this finding has implications for how users may perceive the effectiveness of google scholar in comparison to purchased library databases.   in our study, we looked only at the top ten results for each product. this focus is justified by studies showing that users of library databases rarely look beyond the first page of results for information (e.g., asher, duke, & wilson, 2012). our own knowledge of user behavior corroborates this finding. click-through statistics for the articles portion of quicksearch show that 57% of users click on the first result in that module, and that 74% click on one of the top three results. only 21% of users click on the “see all results” option.   our methodology required members of the research team to make educated assumptions about what users were actually looking for when they entered the search terms in our sample into summon. it also required them to judge whether each search result was relevant in relation to the presumed search topic. this methodology is similar to that used by google’s search evaluators to improve its relevance ranking algorithm (google, 2012). while this methodology introduced a certain amount of subjectivity into this study, the effect on the results was likely small. in practice, it was generally easy to interpret the intent behind each search query, and only two uninterpretable search queries were removed from the sample. (see examples of search queries in tables 1 and 2). it was also generally easy to decide whether a specific search result was relevant (i.e., on topic). for subsequent studies, the authors would suggest including a measure of intercoder reliability.   a limitation of this study was the small sample size for known item queries. while the proportion of known item queries in our random sample (24%) matched our expectations, it resulted in a sample size of only n=44 for known items. for subsequent studies, the authors suggest using a larger initial sample size in order to obtain a sample of known item queries large enough to discriminate small performance differences between the products.   while the relevance of results is an important search engine evaluation criterion, it is useful to keep in mind that other factors could be of equal or greater importance to our users. our study did not take into consideration other potential advantages or disadvantages of google scholar, e.g., its familiar and clean user interface, lack of ability to limit to peer-reviewed articles, or inability to pull results into the library’s quicksearch interface. similarly, our study did not take into consideration interface design, usability, and feature differences between summon and eds.   unlike most institutions subscribing to a web-scale discovery product, the ncsu libraries does not use summon to create a single search box for articles, books, and other formats of material. instead, it uses summon primarily to help novice library users find scholarly articles. of key importance to us is the ability to use summon to populate the articles portion of our quicksearch interface. because the majority of our users search summon through quicksearch over half (72%) never even see the summon interface. because of this, the relevance of the top three summon results is particularly important to us, and we will continue to evaluate products that could potentially provide the same functionality at lower cost.   conclusion   a study was conducted to compare the search performance of two web-scale discovery services, proquest’s summon and ebsco discovery service (eds). the performance of these services was also compared to google scholar. a sample of 183 actual user searches, randomly selected from the ncsu libraries’ 2013 summon search logs, was used for the study. there was found to be no significant difference in performance between summon and eds for either known-item or topical searches. there was also no significant performance difference between the two discovery services and google scholar for known-item searches. however, google scholar outperformed both discovery services for topical searches. because there was no significant difference in the search performance of summon and eds, any decision to purchase one product or the other should be based upon other considerations (e.g., technical issues, cost, customer service, or user interface).   references   anderson, m. (2001). permutation tests for univariate or multivariate analysis of variance and regression. canadian journal of fisheries & aquatic sciences, 58(3), 626-639. http://dx.doi.org/10.1139/cjfas-58-3-626   asher, a. d., duke, l. m., & wilson, s. (2013). paths of discovery: comparing the search effectiveness of ebsco discovery service, summon, google scholar, and conventional library resources. college & research libraries, 75(5), 464-488.   ellero, n. (2013). an unexpected discovery: one library's experience with web-scale discovery service (wsds) evaluation and assessment. journal of library administration, 53(5/6), 323-343.   good, p. (2005). permutation, parametric and bootstrap tests of hypotheses (3rd ed.). new york: springer.   google. (2012). search quality rating guidelines version 1.0. in inside search: how search works. retrieved 2 march 2015 from https://static.googleusercontent.com/media/ www.google.com/en/us/insidesearch/howsearchworks/assets/searchqualityevaluatorguidelines.pdf   howell, d. c. (2006). repeated measures analysis of variance via randomization. retrieved 2 march 2015 from https://www.uvm.edu/~dhowell/statpages/resampling/randomrepeatmeas/repeatedmeasuresanova.html   lown, c., sierra, t., & boyer, j. (2013). how users search the library from a single search box. college & research libraries, 74(3), 227-241. http://dx.doi.org/10.5860/crl-321   rochkind, j. (2013). a comparison of article search apis via blinded experiment and developer review. code4lib journal, 19. retrieved from http://journal.code4lib.org/articles/7738   t test for related samples. (2004). in d. cramer and d. howitt (eds.), the sage dictionary of statistics (p. 168). london, england: sage publications, ltd. retrieved from http://srmo.sagepub.com/view/the-sage-dictionary-of-statistics/sage.xml   timpson, h., & sansom, g. (2011). a student perspective on e-resource discovery: has the google factor changed publisher platform searching forever? the serials librarian, 61(2), 253-266.   zhang, t. (2013). user-centered evaluation of a discovery layer system with google scholar. in design, user experience, and usability. web, mobile, and product design (pp. 313-322). berlin: springer.   evidence summary   usage data of images from a digital library informs four areas of digital library management: metadata creation, system design, marketing and promotion, and content selection   a review of: reilly, m., & thompson, s. (2014). understanding ultimate use data and its implication for digital library management: a case study. journal of web librarianship, 8(2), 196-213. http://dx.doi.org/10.1080/19322909.2014.901211   reviewed by: aoife lawton systems librarian health service executive dr. steevens’ library, dr. steevens’ hospital dublin 8, ireland email: aoife.lawton@hse.ie   received: 9 mar. 2015    accepted: 7 may 2015      2015 lawton. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to investigate the implications of intended and actual usage data retrieved from a digital library on digital library management and design.   design – case study.   setting – a digital library of predominantly high resolution images based at a large research university in the united states of america.   subjects – responses from 917 users of an open access digital library.   methods – researchers used a literature review to identify previous research on this topic and to inform the methodology for their research. two distinct studies informed the methodology: research by beaudoin (2009) that identified categories of both users and questions around usage was incorporated, and the ultimate use categories suggested by chung and yoon (2011) to compare against those used in this research. researchers used data extracted via recorded system logs that are part of the statistics feature of the digital library. this feature is an in-house developed system, the digital cart service (dcs). the logs tracked usage of 917 images recorded over a three year period, from 2011-2013. after eliminating personal information, researchers examined three fields: university affiliation, intended use, and description. after exporting the data from these three fields to a microsoft access database for text analysis, researchers normalized the data using a series of codes assigned to the responses. it is unclear how many description fields were used to yield more information.   main results – researchers identified five user-types among users of the digital library. the biggest user group was visitors, followed by university staff, while university faculty had the lowest usage. visitors were found to use images for personal use, such as inspirational and artistic purposes. the products developed from images in the digital library were-wide ranging, and included image albums, research, artwork, and video productions. these findings have implications for four areas of practical management of digital libraries: metadata creation, system design, marketing and promotion, and content selection. among the eight categories of intended uses recorded, the highest uses were found to be for personal use, followed by ‘other’ use. researchers examined the ‘other’ use category and further divided it into 12 sub-categories. of these sub-categories, the highest use was for publication and research, while the lowest use was for ‘gift’ and ‘industry.’   conclusion – incorporating user-generated metadata and distributing it to digital library managers is found to produce enhanced metadata and to aid the promotion and awareness of collections. usage data may inform marketing efforts, as it provides a more comprehensive picture of who uses digital libraries and why they use images retrieved from those libraries. equally, usage data may reveal the least frequent users of digital libraries, which informs targeted user marketing campaigns. finally, the authors find that usage data combined with user-generated metadata should form part of content selection criteria for digital library managers.   commentary   this research adds to a small but growing body of evidence about the use of digital images from digital libraries and how usage may inform digital library management. at least one study has found that the needs of users of digital images are not yet met (kandiuk & lupton, 2012). the findings point to the analysis of usage data as a way of identifying user needs.   overall, this case study adheres to a good level of validity when checked against glynn’s (2006) critical appraisal checklist. the authors note the limitations of the study, including the possibility of the sample not being representative of the overall user population of the digital library, or of all the uses of the data. the literature review successfully builds on previous research methodologies, including coding schemes for image use (chung & yoon, 2011), and the replication of three research questions previously asked by beaudoin (2014). despite the integration of relevant methodologies, the authors do not appear to have integrated recommendations of previous research into their own digital library management practice. for example, two recommendations found by the current study come from mccay-peet & toms’ (2009), who advised that information professionals develop more categories of access points to aid image retrieval and to describe objects using both conceptual and non-conceptual attributes. these recommendations are highlighted in the literature review, and it may have been useful to revisit their relevance in the discussion area of the article.   the dcs appears to be an innovative feature of the digital library and may prove useful for other digital library managers. since the dcs is customized software, the inclusion of a screenshot of the dcs may have enhanced the readers’ understanding of the practical use of image retrieval from the digital library. a good description of the data extraction and analysis is provided, and the categories used could be replicated by other digital library managers. results are clearly presented in six tables, and the authors incorporate quotations from users about images used in practice.   the authors have made some important findings in this case study that will be of interest to digital library managers and of special interest to those responsible for digital images. specifically, the integration of user-generated metadata compliments indexing by library professionals. where this feature is possible, the metadata should be sent to digital repository managers. user data informs system design and involves users in marketing and promotion of digital image collections. low usage statistics among categories of users, as indicated by usage data, could directly inform targeted marketing initiatives. the authors argue that ultimate use should be used as a selection criterion for content inclusion. this may become more important over time, with more usage data collected over longer periods of time. the authors continue to investigate how usage of images can aid effective retrieval of images using improved metadata. they have identified further topics relevant to digital library management for future research.   references   beaudoin, j. e. (2009). an investigation of image users across professions: a framework of their image needs, retrieval and use (unpublished doctoral dissertation). drexel university, philadelphia, pa. available at http://hdl.handle.net/1860/3160   beaudoin, j. e. (2014). a framework of image sse among archaeologists, architects, art historians and artists. journal of documentation, 70(1), 119-147. http://dx.doi.org/10.1108/jd-12-2012-0157       chung, e., & yoon, j. (2011). image needs in the context of image use: an exploratory study. journal of information science, 37(2), 163-177. http://dx.doi.org/10.1177/0165551511400951   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154   harris, v., & hepburn, p. (2013). trends in image use by historians and the implications for librarians and archivists. college & research libraries, 74(3), 272-287. http://dx.doi.org/10.5860/crl-345   kandiuk, m., & lupton, a. (2012). digital images in teaching and learning at york university: are the libraries meeting the needs of faculty members in fine arts? evidence based library and information practice, 7(2), 20-48. retrieved from http://ejournals.library.ualberta.ca/index.php/eblip   mccay‐peet, l., & toms, e. (2009). image use within the work task model: images as information and illustration. journal of the american society for information science and technology, 60(12), 2416-2429. http://dx.doi.org/10.1002/asi.21202 review article   are best practices really best? a review of the best practices literature in library and information studies   jackie druery head, learning and research services queen’s university library kingston, ontario, canada email: drueryj@queensu.ca   nancy mccormack librarian and associate professor of law queen’s university library kingston, ontario, canada email: nm4@queensu.ca   sharon murphy head, academic services division queen’s university library kingston, ontario, canada email: murphys@queensu.ca   received: 08 july 2013    accepted: 04 nov. 2013      2013 druery, mccormack, and murphy. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective the term “best practice” appears often in library and information science literature, yet, despite the frequency with which the term is used, there is little discussion about what is meant by the term and how one can reliably identify a best practice.   methods – this paper reviews 113 articles that identify and discuss best practices, in order to determine how “best practices” are distinguished from other practices, and whether these determinations are made on the basis of consistent and reliable evidence. the review also takes into account definitions of the term to discover if a common definition is used amongst authors.   results – the “evidence” upon which papers on “best practices” are based falls into one of the following six categories: 1) opinion (n=18, 15%), 2) literature reviews (n=13, 12%), 3) practices in the library in which the author works (n=19, 17%), 4) formal and informal qualitative and quantitative approaches (n=16, 14%), 5) a combination of the aforementioned (i.e., combined approaches) (n=34, 30%), and 6) “other” sources or approaches which are largely one of a kind (n=13, 12%). there is no widely shared or common definition of “best practices” amongst the authors of these papers, and most papers (n=94, 83%) fail to define the term at all. the number of papers was, for the most part, split evenly amongst the six categories indicating that writers on the subject are basing “best practices” assertions on a wide variety of sources and evidence.   conclusions – library and information science literature on “best practices” is rarely based on rigorous empirical methods of research and therefore is generally unreliable. there is, in addition, no widely held understanding of what is meant by the use of the term.   introduction   it is generally agreed that the term “best practice” grew out of the manufacturing industry’s interest in and implementation of benchmarking. the process of benchmarking began in earnest in the 1970s and increased in popularity in the 1980s during which time companies became increasingly intent on discovering how they rated amongst their competitors as well as on determining why some companies were more successful than others in specific areas. today, benchmarking is defined as, “the process of identifying the best practice in relation to products and processes, both within an industry and outside it, with the object of using this as a guide and reference point for improving the practice of one's own organization” (law, 2009, “benchmarking,” para. 1). “best practice,” correspondingly, is defined as “a practice that has been shown to produce superior performance,” and the adoption of best practices is viewed as a mechanism for improving the performance of a process, business unit, product, service, or entire organization (szwejczewski, 2011, “best practice”, para. 1).   not surprisingly, the interest in benchmarking and best practices is no longer confined to the manufacturing sector and the term “best practices” has entered the vocabulary in a number of fields including library and information science. yet, despite the frequency with which the term is used in library and information science literature, it is not obvious that there is a standard or even a widely-shared meaning of the term amongst the professionals who use it. clearly, without an understanding in the profession of what is meant when we use the term, there is some question about how meaningful the body of “best practices” literature is and what insights may be gleaned from it.   problems of definition aside, there remains the further problem of what constitutes solid evidence upon which to ground an assertion that a process or practice is a best practice. by what means do we determine that something is a best practice? is “best practices” literature in library and information science based on a particular type of evidence or are its foundations as varied as definitions of the term itself? the question is not merely academic — in our current political context, which more than ever demands accountability and measurable outcomes, it is clear that we must focus our attention on establishing meaningful ways to evaluate our performance and work towards greater consistency and clarity in our discourse.   literature review   while there is a considerable amount of literature on “best practices” in a number of disciplines and professions from the humanities to engineering, relatively few articles survey the literature to explore how the claim that something is a “best practice” is defined and determined across a specific discipline or profession. the study and practice of organizational change, notably, is one area in which surveys of the literature have been conducted to determine the basis for the claims of “best practice.”   hallencreutz and turner (2011), for example, explored the literature of organizational change to determine if consistent models and definitions had been used to declare a process or practice a best practice. they surveyed 160 articles from the emerald database of management literature that contained some combination of the words organizational change and/or change management and best practice. the authors concluded that “many popular management practices labeled as best practices…are based on anecdotal evidence rather than empirical data” (p. 65). importantly, this work builds on a previous study in which the authors determined that that no coherent models or definitions of best practice in organizational change were to be found in the literature (turner, haley and hallencreutz, 2009).   reay, berta and kohn (2009) conducted a systemic review of the literature — that of evidence-based management (ebmgt) — and asked three questions: 1) is there a substantial literature concerning the concept of ebmgt?, 2) what is the quality of evidence (where it exists) for ebmgt?, and, 3) is there evidence that ebmgt improves organizational performance? these researchers reviewed 169 journal articles written in english which were current up to 2008 and which were available using electronic journal databases. their study found that “a large number of articles are published on the topic, but most provide encouragement to adopt ebmgt based on opinion and anecdotal evidence” (p. 5). they were also surprised to find that there was almost no evidence presented in the articles that ebmgt improves performance.   simon (2011) conducted a literature review of best practices in corporate libraries in the united states. she found that “although there are many articles in the body of library literature focusing on the importance of incorporating benchmarking and best practices into practice, there is a distinct lack of case studies detailing actual benchmarking/best practices experiences and there are no articles proposing a set of generally accepted best practices for corporate libraries” (p. 139). simon speculated that the reason for this lies in the work of early library theorists including shera (1944), wasserman (1958) and lancaster and joncich (1977) who argued that a set of standards could not be developed for special libraries because by their very nature they are not homogeneous and therefore not natural or easy candidates for standardization.   more studies exist across a variety of disciplines which question the concept of best practice in a more general way along with the lack of standardized models and definitions (for examples see sanwal, 2008; reay, berta & kohn, 2009; wellstein & kieser, 2011), and still others suggest models that could be used to evaluate and determine that something is a best practice (for examples see bardach, 2003; turner, haley & hallencreutz, 2009). while the full exploration of these works across the spectrum of academic and other disciplines is beyond the scope of this paper, what these studies make clear is that this paper is not the first to question these ideas and, of course, that further research is necessary in the field of library and information science to determine whether any of the suggested models and/or definitions would be applicable to our own services and processes.   objectives   what is the extent to which best practices library literature fails to define the term “best practices”?  do assertions made regarding best practices rest less on detailed empirical studies and far more on opinion, individual experience and anecdotal information? we wondered if the term “best practice” would be interpreted in any one of a variety of ways including 1) practices carried out by most organizations, 2) practices carried out by “successful” organizations, 3) practices based on observation and experience, 4) practices based on opinion alone, and, finally, 5) practices based on empirical research.   our goals in this study were first, to identify “best practices” library and information literature across all types of libraries; second, to understand the evidence used by the writers of these papers in declaring that a practice is a best practice; and, third, to determine the extent to which there exists a common understanding of the term “best practice” in the literature.   methods   on february 5, 2013, the library, information science & technology abstracts (lista) database was searched for articles with the words “best practice” or “best practices” in the title. we chose to search the title field only, given the number of papers retrieved (more than 2,000) when a broader search using “best practice(s)” was run in the database.   lista, published by ebsco, was selected for several reasons. the first had to do with the amount of journal overlap amongst library and information science databases and the sheer volume of papers we expected to retrieve. by using one specific database covering “…librarianship, classification, cataloging, bibliometrics, online information retrieval, information management and more” (lista, 2013) and indexing “…more than 560 core journals, nearly 50 priority journals, and nearly 125 selective journals; plus books, research reports and proceedings” (lista, 2013), we expected to find the key papers central to the profession. the database was also chosen because articles indexed extend back more than half a century to the mid-1960s.   the search revealed 450 such titles. brief news items, book reviews, and calls for papers as well as cursory reports of conference talks were then eliminated from this group. papers with subject keywords which were associated with papers we believed to be too remote from those areas central to the profession were generally eliminated, e.g., computer and software stores, freestanding ambulatory surgical and emergency centers. no paper, however, even those with one of the aforementioned subject headings, was eliminated from the group until it had first been examined to ensure that it was not a substantive paper dealing specifically with best practices in libraries.   the remaining articles were reviewed to determine if they dealt with best practices in libraries. papers on best practices in medical libraries, for example, were included but those in the practice of medicine itself were not. papers primarily about programming computer systems and network optimization were, similarly, removed from the group. in addition, publications by professional associations which merely announced the existence of best practice guidelines developed by various associations were excluded unless they proceeded to present the substance of those guidelines at some length.   using this method, 113 papers remained of the 450 originally retrieved. each researcher read one third of the papers and recorded whether “best practice” was defined by the author, and if defined, we recorded the definition. the papers were then classified with respect to the evidence used by these authors to determine a best practice, and like papers were grouped together. researchers then re-read their own and, often, each other’s papers during this exercise. the results of the classification exercise are discussed below. all percentages are rounded up or down to the nearest whole number.   results   categories of best practices within the library literature   best practices papers have been written on a variety of subjects in the library and information science field, but even more so in those areas central to the profession itself. publications on the matters of teaching and instruction made up the largest group at 21 (out of 113) papers. this topic was followed by cataloguing and metadata (10 papers), digital collections (9 papers), and reference (6 papers). several categories were tied at 5 papers each, including distance education, management and leadership, multicultural services, and social media outreach. as table 1 illustrates, the remainder of the best practices papers were spread thinly amongst a variety of other subject categories   our results also confirmed our sense that the term “best practice” had indeed gained popularity in library discourse. of the 113 papers we reviewed, the first instance of best practice in a title was in one article published in 1997. in 2001 that number had tripled to 3; in 2006, 14 such articles were published and, by 2011, that number had grown to 17.   approaches used in the lis literature for determining “best practices”   papers were categorized according to the approach used for determining the best practice, and were placed in one of the following categories: 1) literature reviews, 2) practices derived from the library in which the author works, 3) qualitative and quantitative research approaches, 4) combined approaches, 5) opinion and 6) other. most papers involved initiatives in a single library rather than a comparison of services or practices across libraries.   figure 1 shows the percentage of the total number of papers in each category. a discussion of each category follows.   1)    literature reviews   thirteen papers (12%) consulted the literature and used this information to determine that a process or practice constitutes a best practice. literature reviews were rarely systematic; where other published papers declared that a particular method was a “best practice,” this was accepted as being accurate. one of the few exceptions to this is the paper by shaw and spink (2009), in which the authors attempted to distinguish between the types of papers they consulted, noting that “twenty three were identified as empirical papers, and sixteen were opinion pieces” (p. 192). this distinction is, at least, a first step towards pinpointing works with reliable content.   the papers using literature reviews dealt with a variety of topics including teaching (saunders, 2002), protecting patrons’ confidentiality (maji, 2007), managing organizational diversity (kreitz, 2008), cataloguing (alexander, 2008), reference services (meyer, 2008; shaw & spink, 2009), the history of “story time” (albright, delecki & hinkle, 2009), creating an online tutorial (blummer & kritskaya, 2009), hiring processes (shaffer, 2011), and integrating e-books into a collection (blummer & kenton, 2012).   2)    practices derived from the library in which the author works nineteen papers (17%) in our review chose this method to determine whether a practice was a “best practice.” papers in this category explained a system or a method used in the author’s library, discussed why it seemed to work well, and declared it to be a “best practice” without any empirical pre-and post-method evaluation. the category covered a variety of topics including teaching (campbell & fyfe, 2002; tempelmen-kluit & ehrenberg, 2003), reference (johnson, 2009), managing media centers (“best practices for managing,” 2001), weeding (gushrowski, 2007), evaluating classroom teaching material (johnson & reynolds, 2007), leadership skills (choh, 2008), offering multicultural library services (crichton, 2008; glass & sheffield, 2008), fostering library as place (coonin, williams & steiner, 2011) and more. as with a number of the papers based on literature reviews, several papers in this category used the term “best practice” in the title without any further reference to it in the article. interestingly, not one paper in this category defined the term.   table 1 best practices papers by subject subject number of papers teaching/ instruction 21 cataloguing and metadata 10 digital collections 9 reference 6 distance education 5 management and leadership 5 multicultural services 5 social media outreach 5 collection development 4 interlibrary loan 4 associations, library 3 media, managing 3 outreach programs 3 hiring 2 public relations 2 students with disabilities 2 websites, management/assessment 2 best practices, why they matter 1 citation management 1 construction projects 1 disaster recovery 1 email management 1 funding 1 friends of the library programming 1 impact, library services of 1 indexing 1 information technology departments 1 laptop lending program 1 knowledgebase data transfer 1 protecting library patrons' confidentiality 1 preservation 1 story reading/telling 1 student achievement 1 student engagement 1 students, remedial 1 teams, working in 1 vendors, selection 1 weeding 1 wikis 1   3)    empirical research (qualitative and quantitative)   sixteen papers (14%) appeared in this category. most of the articles involved surveys and interviews with responses collected through a variety of means including online and print questionnaires, and interviews with specific people. one notable exception to this group dealt with a preand post-measurement of library-instruction-learning in which students filled out multiple-choice knowledge tests both before and after library instruction (stec, 2006). the purpose of this study was to identify best practices in instruction by assessing student learning with different instructor types and instruction methods.   brown (1998) used surveys to identify best practices in vendor-selection criteria. hodge’s (2000) study of digital archiving identified best practices using a survey of managers of select projects. cowen and edson (2002) sent questionnaires to hospital librarians via a listserv asking, among other things, about “successful techniques” for it collaboration and communication. shelton’s (2003) study about cooperative collection development pre-selected organizations already employing best practice as indicated by “viability, track record and longevity” (p. 192) and then surveyed them about conditions that either facilitated or worked against that cooperation.   clair’s (2012) study on the use of metadata for web content management systems involved a survey to determine responsibility for content, the workflows that managed it, the standards that were in place, and the barriers which existed to using metadata in this context. while the purpose, methodology, literature review and discussion of results offered seemingly useful information, little evidence was provided to support what made this information qualify as a best practice.   figure 1 percentage of each approach within the best practices literature   renner, vardaman and norton (2007) administered an online survey to determine how health libraries delivered services to distance learners. although their discussion of the results sought to relate practice to established association of college & research libraries (acrl) guidelines, common practice, for the most part, was deemed to be best practice.   finally, in a departure from the survey format, butler (2009) solicited, analyzed, and compared twelve job descriptions for interlibrary loan supervisors from different types of law libraries, and used the results to provide a sample job description outlining “…the elements that should be included, and the core functions and best practices described in this article” (p. 30). like many of the papers mentioned above, the practices were determined to be best because they were common.   papers based on such surveys, interviews, and related methods were problematic because of the bias inherent in the research design. for example, interviewees or survey participants were generally chosen because they are in “cutting edge” programs (hodge, 2000) or programs with a reputation for success (shelton, 2003). how “cutting edge” or “success” was defined is unclear. in addition, common practices (which turned out most often to be practices reported by those who responded to the survey) were simply translated as best practices without any rigorous analyses of practices or the designation. similarly troubling, authors may have selected replies that they deemed “useful,” and reported these as best practices (cowen & edson, 2002). the papers which fell into our next category, “combined approaches,” followed, to various extents, these same problematic practices.   4)    combined approaches   this category, in which authors use two or more approaches to identify best practices, contained thirty-four papers (30%). the two or more approaches most often consisted of a literature review and a survey of other information professionals (briscoe, selden & nyberg, 2003; buck, islam & syrkin, 2006; farelly, fisher & kurmann, 2006), or a literature review and a description of how things were done in the author’s library (benjes-small, dorner & schroeder, 2009; boule, 2008; wheeler, johnson & manion, 2008). other approaches included a combination of literature and document (project reports, procedures, policies) reviews (mellinger & starmer, 2002), self-evaluation accompanied by a survey (bentley, 2006), or literature review, personal experience and a study of practices at a small number of institutions (hoffman & ramin, 2010).   5)    opinion   opinion pieces were generally one of two kinds — those based on feelings or beliefs (discussed later in this paper), and those based, in part, on how practices are carried out in a particular library. eighteen papers (15%) in this study were the product of opinion alone. these articles covered a variety of topics, including outreach (huwe, 2006), coaching staff (will, 2006), public relations success (postar, 2006), social software (stephens, 2007), and virtual reference (perret, 2011). the papers in this category were based on the author’s personal thoughts or opinions; in general, few and often no sources were cited.   6)     other   in this final category, thirteen papers (12%) used an assortment of approaches (which did not fall under any of the previously discussed groups) to identify best practices or to discuss best practices as a general concept. these included self-studies (i.e., comparing a current practice against library association guidelines) (hunt & birks, 2004), discussions about what best practices are, the issues surrounding the adoption of them and the steps that libraries might take to implement them (todaro, 2002) or why best practices matter (no particular study involved), how librarians can apply them and how they can become advocates to help others in their organizations to do the same (leandri, 2005), and various reviews of library policies (thomas, 2007), protocols and standards (kasprowski, 2008).   definitions within the best practices literature   as table 2 illustrates, only 20 of the 113 papers (17%) included in this study presented any definition of the term “best practices;” indeed, in some, the term was used in the title only and never appeared again in the text as though its meaning or its use in the title alone is self-evident.   of the papers that did attempt to define “best practices,” what emerged was far from a common or shared definition. definitions included practices resulting in better results, standards drafted by associations or organizations, criteria derived through benchmarking and comparison with “successful” organizations, standards appropriate given the circumstances, and practices which have been shown to lead to best outcomes. samson (2011) defines best practice as those meeting federal regulations. some authors relied on existing definitions while other authors created their own. for the purposes of illustration, we have included these definitions under the single most descriptive heading — see appendix.   patricia kreitz (2008), relying on an existing definition, cited webster's new millennium dictionary of english which defines best practices as “practices which are most appropriate under the circumstances, esp. as considered acceptable or regulated in business; techniques or methodologies that, through experience and research, have reliably led to desired or optimum results” (p. 103). she noted, however, that best practice literature too often relies on “brief case studies or anecdotal stories to support the authors' assertions” and that the body of empirical research is often too small to determine whether particular practices will “produce desired or optimum results” (p. 103).   table 2 best practices papers sorted by approach approach best practice defined best practice not defined total number of papers literature reviews 3 10 13 (12%) practices from author’s library 0 19 19 (17%) qualitative and quantitative approaches 3 13 16 (14%) combined approaches 9 25 34 (30%) opinion 2 16 18 (15%) other 3 10 13 (12%)   20 (17%) 93 (83%) 113 (100%)   other definitions relied on “better results” criteria include “highly effective or innovative operating procedures and philosophies that produce outstanding performance when implemented” (leon, deweese & kochan, 2003, p. 420), or even “any procedure which, when properly applied, consistently yields superior results” (liu, 2004, p. 339). the latter definition is, of course, particularly problematic in that phrases such as “any procedure,” “properly applied” and “superior results” are themselves open to debate and interpretation. in addition, they shed little light on what separates a better practice from one that is best.   a small number of definitions employ the language of best. davis (2009), for example, defines best practices as “the best way of carrying out a function or process” (p. 7). best practices come about, the author reports, once one has assessed internal needs and identified current practices, and then identified alternate ways of doing things and modeled best practices on those alternate methods. comparing one’s operations to the competition is key in order to see what can be used in one’s own operation.   for one other author, “best practices are simply the best ways to perform a specific … function or process” (leandri, 2005, p. 20). they bring about results desired by the organization; others seek to emulate them, and the results are often measurable. they also, according to this author, transcend boundaries, which means that best practices in customer service in unrelated industries or organizations (e.g., hotels or healthcare) can be employed in others (libraries) with similar results. hurst-wahl (2009), in contrast, defines the term as that which “has been determined to work well”. “in some circles,” this author notes, “they are called traditions” (p. 22).   shaw and spink (2009) cite morin (2004) who suggests that “best practices and guidelines outline a process, practice, or method that can improve effectiveness and efficiency in several situations" (p. 193). a best practice, according to morin, becomes evident when applied to a specific task rather than larger or more general areas. the term “best practices” is used in these papers as a synonym for “tips” or “good ideas,” and this is borne out by postar (2006) who writes, “…the term implies success; that certain actions, attitudes, and programs are the most efficient and effective way of doing business and that the same measures can be used with successful outcomes in all similar organizations” (p. 12). what is “most efficient and effective” for this author, can be derived from having seen, first hand, certain ideas or principles which, when applied in various libraries, have produced successful results.   as the definitions above indicate, these authors are nowhere near a shared definition. adding to the difficulty are those papers which rely on existing literature as evidence of best practices, particularly where articles consulted are treated as correct and authoritative merely because they have been published. virtually no discussion takes place on why the papers selected should be deemed authoritative or reliable.   in addition to “best practices,” some authors tackle the definition of benchmarking. melo, pires and taveira (2008), for example, cite foot’s definition of benchmarking as “a process of measuring your service’s processes and performance and systematically comparing them to the performance of others in order to seek best practice” (p. 50). as we saw earlier, best practices according to this view, arise from the measurement of one’s own services and the subsequent comparison with the services of others. along the same lines, other authors define best practices as encompassing “quality frameworks, benchmarking and performance measurement of products, processes and services” (farelly, fisher & kurmann, 2006, p. 7). finally, many authors do not define best practice specifically but instead use terms such as “practices commonly employed” (albright, delecki & hinkle, 2009, p. 15), or “common practices...that appear to foster success in their use” (blummer & kritskaya, 2009, p. 200) as synonyms for “best practices.” for these authors, literature reviews and standards drafted by academic bodies seem to offer ideal evidence for “current best practices.” it is also suggested that best practices may be identified through “brainstorming” sessions (campbell & fyfe, 2002), through newly introduced processes or practices which appear to be doing well or which have garnered positive user responses (buzzard, teetor & travis, 2011); or through traditional practices that have been used successfully for many years (johnson, 2009).   discussion   the limitations of this study have to do with the selection of papers. the entire body of library and information science literature was not included — only those papers having to do with libraries and library services and practices were included. also, only one database was searched.   the findings of this research are similar to those of hallencreutz and turner (2011) and reay, berta and kohn (2009) who concluded in their studies of the literature of best practice in organizational change, and evidence-based management, respectively, that there were no consistent ways of determining a best practice, nor were there consistent definitions in the literature. as with these previous studies, many claims of best practice were based on opinion and anecdotal evidence.   our theory that best practices papers in the library and information science literature would be based far less on empirical data and far more on opinion, individual experience and anecdotal information was borne out by this study. eighty-three percent of papers on the topic of best practices left the term undefined, and those that did define the term did not agree on the definition. as a result, “best practice” was used as a synonym for standards, guidelines, good ideas, common practices, practices derived through benchmarking, traditions, and recommended practices. this research underscores the variation in definitions of this term. it also points to the difficulty in building a body of literature around best practices when our understanding of “best practices” ranges from “traditions” and good ideas to the “most efficient and effective way of doing business.” (postar, 2006, p.12)   also important in providing libraries with reliable evidence on best practices is proper training — training which should begin long before professionals reach the workplace. so, what does this mean for library and information science (lis) school curricula? clearly, more attention should be paid, during the education of lis professions, to the processes involved in the assessment of library services along with the reliability of methods used. likewise, suitable training requires a more rigorous use of the vocabulary within the profession, so that, as a group, we understand the difference between, say, a good idea, a better outcome, and a best practice. some papers in this mix did attempt to adopt a more rigorous methodology by describing a process or service, establishing goals for it, identifying at least two methods of attaining those goals, conducting pre-and post-assessment of the process or service when using those methods (stec, 2006), and identifying methods that met benchmarked goals (farrelly, fisher & kurmann, 2006). these types of studies, in theory, are those which should be most likely to provide libraries with reliable evidence on best practices.   what does all this mean for practitioners where this type of training has either been completely absent or, perhaps, incomplete? clearly, good ideas are shared in the library and information literature, and we should continue to adopt and adapt those that are transferable to our own organizations. at the same time, we must read the literature with caution and ask ourselves the same questions that have been asked in this paper. we need to look critically at anything that claims to be a best practice before we attempt to implement or adapt it in our own library. when we write about best practice we should ask ourselves whether the process or practice is a best practice based on sound research or whether is it just a “good idea,” that is, something that we do in our library that works for us. the call is to place a moratorium on the term unless we truly have the evidence to call what we do a “best practice.”   future research in this area might include a similar study of best practice guidelines developed by library associations. stein (2008) in her discussion of the ifla guidelines for best practice for interlibrary loan and document delivery outlines the relatively rigorous process that was followed to develop the guidelines, and she offers advice for implementing the guidelines in any individual library. one could compare best practice guidelines issued by other library associations to determine the process that was followed to develop such guidelines. questions one might ask could include: is best practice defined, and if so, how is it defined in these documents? are there similarities and differences in the practice used to develop the guidelines? does the degree of rigour in developing these guidelines differ between library services and processes? one might also seek out and survey libraries which have followed such guidelines to determine if implementing them not only improved service, introduced efficiencies and/or increased impact but were, in fact, a best practice.   as reay, berta and kohn (2009) noted in their study of evidence-based management, more research comparing and contrasting local efforts and their outcomes would help to build a body of knowledge that could be transferable to other organizations and situations. it is worthy of note that many of the services and processes described in the papers discussed in this study are initiatives implemented in a single library.   as a final consideration, it is important to keep in mind that professionals are increasingly being asked to be more accountable and to quantify and document the impact of services. as such, libraries of all types, more than ever, must engage in a more rigorous assessment of services and programs using sound assessment methodologies and techniques. among the various outcomes would be a larger body of literature, based in sound evidence, which reliably documents and determines best practices in any one of a number of processes or services.   conclusion   increasingly libraries are required by their institutions and funding agencies not only to demonstrate return on investment but also to provide reliable statistics and other evidence-based data. as such, understanding what we mean when we use terms such as “best practices” and increasing the rigour of our analysis is critical to our future.   at the same time, we should recognize the danger in assuming that there is some universal yardstick for libraries’ practice. for example, applied too rigidly given our complex and unique environments, a best practice might stifle critical problem solving and creativity and, ironically, bring about inferior results. it may reduce us to merely adopting a practice whose results are not replicable in our own institutions. this is not to suggest, however, that we abandon our search for best practices. instead, it is to approach with caution that which we conclude is “best,” and to remember, as one writer has suggested, that we may, in our pursuit, be chasing a mirage (liu, 2004).   acknowledgement   an earlier abbreviated version of this paper was presented at the eblip6 conference, university of salford, manchester, uk, june 27-30, 2011.     references   albright, m., delecki k., & hinkle, s. (2009). the evolution of early literacy: a history of best practices in storytimes. children and libraries, 7(1), 13-18.   alexander, m.s. (2008). core cataloging and metadata standards and best practices. science & technology libraries, 28(1/2), 63-85. doi.org/10.1080/01942620802096945   association of college & research libraries (acrl) information literacy best practices committee. (2012). characteristics of programs of information literacy that illustrate best practices: a guideline. college & research libraries news, 73(6), 355-359.   bardach, e. (2003). creating compendia of “best practice”. journal of policy analysis and management, 22(4), 661-665. doi.org/10.1002/pam.10160   benjes-small, c., dorner, j.l., & schroeder, r. (2009). surveying libraries to identify best practices for a menu approach for library instruction requests. communications in information literacy, 3(1), 31-44.   bentley, m. (2006). best practice or selling out? public library journal, 21(3), 25-26.   best practices for managing library media centers. (2001). media & methods, 37(4), 34-35.   blummer, b. & kenton, j. (2012). best practices for integrating e-books in academic libraries: a literature review from 2005 to present. collection management, 37(2), 65-97. doi.org/10.1080/01462679.2012.660851   blummer, b.a. & kritskaya, o. (2009). best practices for creating an online tutorial: a literature review. journal of web librarianship, 3(3), 199-216. doi.org/10.1080/19322900903050799   boule, m. (2008). best practices for working in a virtual team environment. library technology reports, 44(1), 28-31.   briscoe, g., selden, k., & nyberg, c.r. (2003). the catalog vs the home page? best practices in connecting to online resources. law library journal, 95(2), 151-174.   brown, l. (1998). approval vendor selection — what’s best practice? library acquisitions: practice & theory, 22(3), 341-351. doi.org/10.1016/s0364-6408(98)00055-6   buck, s., islam, r., & syrkin, d. (2006). collaboration for distance information literacy instruction: do current trends reflect best practices? journal of library administration, 45(1/2), 63-79. doi.org/10.1300/j111v45n01_04   butler, m. (2009). job descriptions for interlibrary loan supervisors: core functions and best practices. journal of interlibrary loan, document delivery and electronic reserve, 19, 21-31. doi.org/10.1080/10723030802533804   buzzard, p., teetor, c., & travis, s. (2011). best practices for a university laptop lending program. code4lib journal, 15, 1.   campbell, s. & fyfe, d. (2002). teaching at the computer: best practices for one-on-one instruction in reference. feliciter, 48(1), 26-28.   choh, n.l. (2008, august). leadership skills and advocacy for libraries: best practice in library association management, the singapore experience. paper presented at the world library and information congress, international federation of library associations, québec, canada. retrieved from http://archive.ifla.org/iv/ifla74/papers/083-choh-en.pdf   clair, k. (2012). metadata best practices in web content management systems. journal of library metadata, 12(4), 362-371. doi.org/10.1080/19386389.2012.735562   colburn, s. & haines, l. (2012). measuring libraries’ use of youtube as a promotional tool: an exploratory study and proposed best practices. journal of web librarianship, 6(1), 5-31. doi.org/10.1080/19322909.2012.641789   coonin, b., williams, b., & steiner, h. (2011). fostering library as a place for distance students: best practices from two universities. internet reference services quarterly, 16(4), 149-159. doi.org/10.1080/10875301.2012.618796   cowen, j.l. & edson, j. (2002). best practice in library/information technology. journal of hospital librarianship, 2(4), 1-15. doi.org/10.1300/j186v02n04_01   crichton, d. (2008). best practices in library services for aboriginal peoples in saskatchewan. paper presented at the world library and information congress, international federation of library associations, québec, canada. retrieved from http://archive.ifla.org/iv/ifla74/papers/118-lee_crichton-en.pdf   davis, s. (2009). best practices for information services: achieving operational excellence. scitech news, 63(4), 7.   farrelly, j., fisher, t., & kurmann, e. (2006). did we get it right? post evaluation of new zealand's interloans best practice workshop. paper presented at the world library and information congress, international federation of library associations, seoul, korea. retrieved from http://archive.ifla.org/iv/ifla72/papers/073-farrelly_fisher_kurmann-en.pdf   fiegan, a. (2011). business information literacy: a synthesis for best practices. journal of business & finance librarianship; 16(4), 267-288. doi.org/10.1080/08963568.2011.606095   glass, e. & sheffield, d. (2008). best practices in multicultural library services: library settlement (lsp) at toronto public library. paper presented at the world library and information congress, international federation of library associations, québec, canada. retrieved from http://archive.ifla.org/iv/ifla74/papers/118-glass_sheffield-en.pdf   gushrowski, b. (2007). moving from good effort to best practice — refining a weeding process in a dental school library. against the grain, 19(3), 26-32.   hallencreutz, j. & turner d. (2011). exploring organizational change best practice: are there any clear-cut models and definitions? international journal of quality and service sciences, 3(1), 60-68. doi.org/10.1108/17566691111115081   hodge, g.m. (2000). an information life-cycle approach: best practices for digital archiving. journal of electronic publishing, 5(4). retrieved from http://www.journalofelectronicpublishing.org/ doi.org/10.3998/3336451.0005.406   hoffman, s. & ramin, l. (2010). best practices for librarians embedded in online courses. public services quarterly, 6, 292-305. doi.org/10.1080/15228959.2010.497743   hunt, f. & birks, j. (2004). best practices in information literacy. libraries and the academy, 4(1), 27-38. doi.org/10.1353/pla.2004.0010   hurst-wahl, j. (2009). digitization: how many best practices, guidelines, and standards do we need? information standards quarterly, 21(4), 22-24.   huwe, t. (2006). some best practices for personalizing outreach. computers in libraries, 26(2), 36-38.   johnson, n.p. (2009). best practices: what first year law students should learn in a legal research class. legal reference services quarterly, 28, 77-99. doi.org/10.1080/02703190902961494   johnson, a.m. & reynolds, l. (2007). trust but verify: using best practices to create an evaluation-of-information exercise. kentucky libraries, 71(4), 8-10.   karshmer, e. & bryan, j. (2011). perspectives on building a first-year information literacy experience: integrating best practices in education and acrl il competency standards for higher education. journal of academic librarianship, 37(3), 255-266.   kasprowski, r. (2008). best practice and standardization initiatives for managing electronic resources. bulletin of asist, 35(1), 13. doi.org/10.1002/bult.2008.1720350106   kreitz, p. (2008). best practices for managing organizational diversity. journal of academic librarianship, 34(2), 101-120. doi.org/10.1016/j.acalib.2007.12.001   lancaster, f. & joncich, m. (1977). the measurement and evaluation of library services. information resources press, washington, dc.   law, j. (ed.). (2009). benchmarking. a dictionary of business and management. oxford university press. retrieved july 5 2013 from http://www.oxfordreference.com/view/10.1093/acref/9780199234899.001.0001/acref-9780199234899-e-624?rskey=kgb75z&result=639&q=   leandri, s.j. (2005). why best practices still matter. information outlook, 9(7), 20-22.   leon, l., deweese, j., kochan, c. & peterson-lugo, b. (2003). enhanced resource sharing through group interlibrary loan best practices: a conceptual, structural and procedural approach. libraries and the academy, 3(3), 419-430. doi.org/10.1353/pla.2003.0063   lista (2013). retrieved from http://www.ebscohost.com/academic/library-information-science-technology-abstracts-lista.   liu, y.q. (2004). best practices, standards and techniques for digitizing library materials: a snapshot of library digitization practices in the usa. information review, 28(5), 338-345. doi.org/10.1108/14684520410564262   luo, l. (2011). text reference service: delivery, characteristics, and best practices. reference services review, 39(3), 482-496. doi.org/10.1108/00907321111161449   maurer, m., gammon, j., & pollock, b. (2013). developing best practices for technical services cross-institutional collaboration. cataloging & classification quarterly, 51(1-3), 179-193. doi.org/10.1080/01639374.2012.733795   maji, t.j. (2007). protecting library patrons confidentiality: checklist of best practices. ila reporter, 25(6), 14-16.   mccutcheon, c. (2008). notes from the field: best practices for psychology collection management. against the grain, 20(4), 58-61.   mellinger, m. & starmer, m.e. (2002). preservation in an agriculture library: balancing best practices with practical considerations. journal of agricultural & food information, 4(3), 51-59. doi.org/10.1300/j108v04n03_06   melo, l.b., pires, c., & taveira, a. (2008). recognizing best practice in portuguese higher education libraries. ifla journal, 34(1), 34-50.   meyer, c.a. (2008). reference accuracy: best practices for making the links. journal of electronic publishing, 11(2). doi.org/10.3998/3336451.0011.206   morin, a.c. (2004). approaching best practices and guidelines for digital reference. in r. lankes, j. janes, l.c. smith, & c.m. finneran (eds.), the virtual reference experience: integrating theory into practice, london: neal-schuman publishers inc.   oberman, c. (2002). what the acrl institute for information literacy best practices initiative tells us about the librarian as teacher". paper presented at the world library and information congress, international federation of library associations, glasgow, scotland. retrieved from http://archive.ifla.org/iv/ifla68/papers/082-119e.pdf   perret, r. (2011). developing policies and best practices for virtual reference. pnla quarterly, 76(1), 82-85.   postar, a. (2006). use best practices to promote your library. aall spectrum, 10(7), 12-13.   reay, t., berta, w., & kohn, m. (2009). what’s the evidence on evidence-based management? academy of management perspectives, 23(5), 5-18. doi.org/10.5465/amp.2009.45590137   renner, b.r., vardaman, a.s., & norton, m.j. (2007). best practices for medical libraries to deliver materials to distance learners. journal of interlibrary loan, document delivery & electronic reserves, 17(3), 85-97. doi.org/10.1300/j474v17n03_12   samson, s. (2011). best practices for serving students with disabilities. reference services review, 39(2), 260-277. doi.org/10.1108/00907321111135484   sanwal, a. (2008). the myth of best practices. journal of corporate accounting and finance, 19(5), 51-60. doi.org/10.1002/jcaf.20417   saunders, l. (2002). teaching the library: best practices. library philosophy & practice, 4(2).   shaffer, c. (2011). best practices for hiring academic librarians with faculty status and rank. southeastern librarian, 59(3), 2-9.   shaw, k. & spink, a. (2009). university library virtual reference services: best practices and continuous improvement. australian academic & research libraries, 40(3), 192-205. doi.org/10.1080/00048623.2009.10721404   shelton, c. (2003). best practices in cooperative collection development: a report prepared by the center for research libraries working group on best practices in cooperative collection development. collection management, 28(3), 191-222. doi.org/10.1300/j105v28n03_02   shera, j. (1944). special library objectives and their relation to administration. special libraries, 35(3), 91-4.   simon, c. (2011). an examination of best practices and benchmarking in corporate libraries.  journal of management development, 30(1), 134-141.   stec, e.m. (2006). using best practices: librarians, graduate students and instruction. reference services review, 34(1), 97-116. doi.org/10.1108/00907320610648798   stein, j.e. (2008). ifla guidelines for best practice for interlibrary loan and document delivery. journal of access services, 5(1-2), 295-303. doi.org/10.1080/15367960802199067   stephens, m. (2007). best practices for social software in libraries. library technology reports, 43(5), 67.   szwejczewski, m. (2008). best practice. in c. l. cooper (ed.), the blackwell encyclopedia of management. blackwell publishing, blackwell reference online. retrieved from http://www.blackwellreference.com.proxy.queensu.ca/subscriber/tocnode?id=g9780631233176_chunk_g97814051109696_ss3-1   tempelman-kluit, n. & ehrenberg, e. (2003). library instruction and online tutorials: developing best practices for streaming desktop video capture. feliciter, 49(2), 89-90.   thomas, m. (2007). an evaluation of selected aserl web pages: "best practices" for serving distance learners. journal of access services, 5(1/2), 37-45. doi.org/10.1080/15367960802235283   todaro, j.b. (2002). change for the right reason: what is a best practice? community & junior college libraries, 11(1), 27-36. dx.doi.org/10.1300/j107v11n01_03   turner, d., haley, h. & hallencreutz, j. (2009). towards a global definition of best practice in change management. international journal of knowledge, culture and change management, 9(8), 188-192.   wasserman, p. (1958). measuring performance in a special library – problems and prospects. special libraries, 49(8), 377-82.   wellstein, b. & kieser, a. (2011). trading “best practices” – a good practice? industrial and corporate change, 20(3), 683-719. doi.org/10.1093/icc/dtr011   wheeler, r.e., johnson, n.p., & manion, t.k. (2008). choosing the top candidate: best practices in academic law library hiring. law library journal, 100(1), 117-135.   will, l. (2006). sales skills for information professionals: best practices and other musings. legal information alert, 25(3), 1-6.       appendix definitions by category better results   “ … common practices that led to success” (maurer, gammon & pollock, 2013)   “a technique or methodology that, through experience and research, has reliably led to a desired or optimum result.” (mccutcheon, 2008)   “practices which are most appropriate under the circumstances, esp. as considered acceptable or regulated in business; techniques or methodologies that, through experience and research, have reliably led to desired or optimum results.” (kreitz, 2008)   “ … any procedure which, when properly applied, consistently yields superior results, and is therefore used as a reference point in evaluating the effectiveness of alternative methods of accomplishing the same task.” (liu, 2004)   “ … are highly effective or innovative operating procedures and philosophies that produce outstanding performance when implemented (leon, deweese, kochan, & peterson-lugo, 2003)   benchmarking   “because the characteristics are descriptive in nature and the result of a meta-analysis of many programs, they may also be used for benchmarking program status, improvement, and long-term development” (“characteristics of programs,” association of college & research libraries (acrl) information literacy best practices committee, 2012, “purpose and use”, para. 2)   “a process of measuring your service’s processes and performance and systematically comparing them to the performance of others in order to seek best practice” (melo, pires, & taveira, 2008)   “… benchmarking … criteria for turnaround time; fill-rate; unit cost; patron satisfaction ..” (farrelly, fisher & kurmann, 2006) “best practice benchmarking is the process of the “quest” of looking for, identifying, studying the best practices that produce superior performance in specific areas and then applying or transferring the best practice to the organization in need of change.” (todaro, 2002)   best way   “ … characteristics are closely aligned with many of the best practices in education … the well-known mcrel study, a theory-based meta-analysis of research on instruction (1998), which resulted in the identification of instructional techniques for more effective teaching” (karshmer & bryan, 2011)   “a best practice is the best way of carrying out a function or process.” (davis, 2009)   “… what has been determined to work well, in some circles they are called traditions.” (hurst-wahl, 2009)   “best practices are simply the best ways to perform a specific business function or process, such as developing or marketing a product. they are performance standards that others seek to emulate.” (leandri, 2005)   evidence-based   “the quantitative and qualitative measures developed for this study are offered as possible metrics …. based on these measures …. the authors have derived a set of evidence-based best practices …” (colburn & haines, 2012)   “… met some level of evidence-based research. the research quality of those articles was then verified through a paired interrater reliability test that checked them against the ebl evidence based checklist” (fiegan, 2011)   goals/standards   “ … guidelines for reference exchanges as a set of goals …” (luo, 2011)   “quality services … if the needs [of the americans with disabilities act] are proactively met in the first place. best practices incorporate this singular premise.” (samson, 2011)   transferable   "best practices and guidelines outline a process, practice, or method that can improve effectiveness and efficiency in several situations." (shaw and spink, 2009)   “essentially the term implies success; that certain actions, attitudes and programs are the most efficient and effective way of doing business and that the same measures can be useful with successful outcomes in all similar organizations.” (postar, 2006)   “… a unique set of criteria that exists for all effective information literacy programs despite the differences in the types and sizes of institutions” (oberman, 2002)     evidence summary   demand-driven acquisition e-books have equal cost per use as print, but dda has much more active use overall   a review of: downey, k., zhang, y., urbano, c., & klinger, t. (2014). a comparative study of print book and dda e-book acquisition and use. technical services quarterly, 31 (2), 139-160. http://dx.doi.org/10.1080/07317131.2014.875379   reviewed by: laura newton miller assessment librarian carleton university library ottawa, ontario, canada email: laura.newtonmiller@carleton.ca   received: 26 nov. 2014 accepted: 26 jan. 2015      2015 miller. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to compare usage of demand-driven acquisition (dda) e-books with print books to help determine if one acquisition model better serves the needs of library users and return on investment.   design – case study.   setting – library system of a large american public university.   subjects – 22,018 dda e-book discovery records, 456 purchased e-book records, and 20,030 print item records were examined.   method – the researchers examined usage statistics, circulation statistics, and cost measures of dda e-books and print books. e-books were purchased in 2012 and print books were purchased by the start of the dda project (january 2012).   main results – all but one of the 456 dda-triggered e-books had repeated use within the first year, totalling 2,484 user sessions. 90% of the triggered e-books had 2-9 user sessions, and over half had at least 4 user sessions. e-books were most used in classes n (fine arts), p (language and literature), and r (medicine). e-books in t (technology) had a lower percentage of user sessions compared to other subject areas. 712 (3.2%) of the e-books in the discovery pool were used without triggering a purchase. usage of e-books in the discovery pool (those used but not triggering a purchase) showed a consistent use of e-books by subject. e-books in class b (philosophy, psychology, religion) were used more in the discovery pool without actually being purchased, suggesting a light use of a wide range of books in this subject area. in contrast, class r (medicine) saw less use in the discovery pool than what was actually purchased, suggesting heavier and more focused use of triggered e-books in this area. only 62.5% of the 20,030 purchased print books included in the study were used in the first 1 to 2.5 years they were added to the collection (i.e., 37.5% were not used in that time period). half of the print books were used no more than once (once or no use), and more than 90% were used fewer than 10 times. print books in class q (science) contributed to only 7.5% of the total circulations, suggesting print books are underused in this subject area. 10.2% of total circulation of print books in class r (medicine) suggests print books are better used in this area. print acquisition and use occur more often in classes n (fine arts) and p (language and literature). the average cost for dda e-books was of $98.52 per book. the average price per print book was $59.53. the unit cost per print book was $17.73 per use. depending on various measures, cost per use for e-books ranged from $17.73 to $29.15 per use. (if the measurement included the free use of non-triggered dda books, the cost per use was $18.07, essentially the same as the print cost).   conclusion – both print books and dda e-books are proportionately distributed across most subject areas. although dda and print cost per use are equal, dda leads to much more active use overall.   commentary   although there is a growing number of papers about dda programs, very few include the addition of print books for comparison. this article makes a very good effort in contributing to this subject.   the eblip critical appraisal checklist (glynn, 2006) was used to determine that a case study was an appropriate tool for this research. the methodology (for the most part) was clearly explained and those interested in replicating the study should be able to do so with relative ease.   this was a very thought-provoking read, however there were some items that should be noted. what led to some confusion for this reader (but was explained later in the article) was the definition of a “discovery pool” and what actually constituted a “trigger.” these definitions would have been very helpful closer to the beginning of the discussion, as some libraries have other criteria for initiating a trigger.   one shortcoming of the methodology is the comparison of dda in 2012 and print books purchased up to 2012. although it is very intriguing and an interesting way of examining “like” books, one might question the publication date of the print for comparison purposes. we know that dda books were purchased during 2012 (and one can perhaps assume they were published close to that time), but it is unclear what the publication dates were for the print. how long does it take to purchase 20,000 print books? a year? more? it may or may not be a minor detail, depending on how big the difference actually is. stating what years of print were included in the study would have strengthened the article.   university libraries are struggling to find acquisition models that best serve the needs of their users while also trying to stay within budget. it was interesting to see that books were used more in n (fine arts) and p (language and literature) and r (medicine) regardless of format. cost per use was essentially the same or very similar, whether print or e-book. e-books purchased were more expensive than print books, but e-books were more likely to be used than the print books. the famous kent (1979) study concludes that 40% of print books do not circulate within the first six years on the shelf. with decreasing budgets and relatively slow uptake and varying buy-in for e-books over print books, libraries struggle with the fact that many of the print books they purchase do not get used. with dda, at least a purchase gets used once. some e-books were used without triggering an actual purchase. those libraries who already use dda but have a trigger option after only one use may want to investigate other options that could allow usage, but not necessarily the financial commitment. the authors mention some hidden time costs to the dda model (such as technical services workload issues and weeding discovery records). future research on these hidden costs would be extremely helpful for libraries to determine the true costs of dda. although every library will have its unique subject collections and unique users, this article is an important read for those in libraries who are struggling to determine what they should be doing (or attempting to do) with their acquisition monograph budgets.   references   kent, a. (1979). use of library materials: the university of pittsburgh study. new york, ny: m. dekker.   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154     evidence summary   follow-up study on free document delivery and interlibrary loan service demonstrates customer satisfaction and generates improvements   a review of: yang, z. y. (l.), hahn, d., & thornton, e. (2012). meeting our customers’ expectations: a follow-up customer satisfaction survey after 10 years of free document delivery and interlibrary loan services at texas a&m university libraries. journal of interlibrary loan, document delivery & electronic reserve, 22(2), 95-110. doi:10.1080/1072303x.2012.708390   reviewed by: kathleen reed assessment & data librarian vancouver island university nanaimo, british columbia, canada email: kathleen.reed@viu.ca   received: 1 june 2013    accepted: 21 aug. 2013      2013 reed. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – texas a&m university libraries have delivered free documents and interlibrary loans for ten years via the get it for me service. this study explores whether the needs of customers are being met, areas for improvement, acceptable turnaround times, why some resources are never picked up, preferred format and steps to obtaining resources, places searched before submitting a request, and whether users ever purchased resources after obtaining them through get it for me.   design – online questionnaire.   setting – large academic library system located in texas, united states.   subjects – researchers used responses from 735 registered users of the get it for me service (12% undergraduates, 49% graduate students, 21% faculty, 15% staff, 1% distance education, 2% other).   methods – the authors emailed all currently registered users of the get it for me service (n=23,063) inviting them to participate in a survey. the survey ran for two weeks, with no follow-up emails sent.   main results – the return rate of 3.18% (n=735/23,063) surpassed the participant goal to achieve a confidence level of 95%, with a confidence interval of 4%.   researchers found that 79% of respondents are satisfied with turnaround time, with 54% of respondents desiring items within three days. expectations increased with position in the academy. time is the significant factor in users not retrieving ordered items; items are no longer needed after deadlines pass or other related materials are found. responses revealed that 55% of users prefer print to e-books, although 70% of participants would accept an e-book version if print is not available. participants were evenly split between reading documents online and printing them to read offline. about one quarter of respondents bought or suggested that the library purchase an item requested via get it for me. when participants encountered a problem, 55% of respondents would contact library staff and 45% would check the service faq. of those that contacted staff, there is a 94% satisfaction rate. overall, 95% of respondents checked the libraries’ online catalogue for availability, 83% looked in e-journal collections, and 74% checked google or google scholar. get it for me was complimented on its user-friendly interfaces and policies, and the money and time it saves its users. in terms of criticism, users requested better quality scanned documents, longer interlibrary loan times, and a pdf instead of a link when an article is found by staff.   conclusion – the author concludes that the document delivery and interlibrary loan services delivered by get it for me are meeting the expectations of users, with 99% of respondents reporting that the get it for me service meets or somewhat meets their needs. areas that required improvement were identified and strategies put in place to improve service. this questionnaire can be applied to other libraries to assist them in learning about document delivery and interlibrary loan service users and their expectations.     commentary   studies on interlibrary loan and document delivery services are not new in the library community. this survey updates previous work by including questions never before asked about preferred formats of books and methods of receiving electronic documents, as well as whether service use inspires personal purchase.   the participation rate of only 3.18% is low, but well within the authors’ desired confidence range. the method used for data collection makes sense for this type of study, and the questions reflect clearly stated objectives. the authors do a good job of explaining how results differ by subsets of participants based on academic rank (undergraduate, graduate, professor, etc). however, the authors do not compare the respondent demographics to those of the university population.   while this study is assessing a local service, the authors ask questions and provide suggestions that are relevant to other institutions. unfortunately, the authors never give a complete list of questions utilized, leaving people wishing to duplicate their efforts having to comb through the article in order to piece together questions to ask. the service improvements made are clear and transferable across many institutions, including reiteration of the importance of request details to staff, high standards for scanned documents, addition of ability of users to re-submit a request for a paper resource that might first come to them in e-book format, highlighting the service’s tutorial link, and reducing the number of times customers need to log in to access library records.   this study is an example of one step in an excellent assessment cycle of this service. when the document delivery and interlibrary loan service first debuted in 2002, it was the subject of a user survey a year later. in the decade it has been in existence, tweaks have been made to the service and its name. instead of simply leaving the service alone, it was assessed again and clear improvements were made. as technologies change, it is critical to review past decisions and explore new opportunities to ensure the best service for library users.       enhancing users’ perceived significance of academic library with mooc services research article   enhancing users’ perceived significance of academic library with mooc services   flora charles lazarus research scholar, department of library and information science banasthali vidyapith niwai, rajasthan, india email: flora.charles4@gmail.com   rajneesh suryasen faculty, department of library and information science, banasthali vidyapith niwai, rajasthan, india email: rajnishsuryasen15@gmail.com   received: 28 july 2021                                                               accepted: 24 mar. 2022      2022 lazarus and suryasen. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip30016     abstract   objective – academic libraries have been impacted by the tremendous changes taking place in higher education due to the arrival of the internet and web-based technologies. several articles have shown the decline in library usage and user need for electronic resources. the entry of moocs into higher education has repurposed the library’s roles and services. this research aims to explore the possible mooc services of academic libraries and their effect on the user perception towards the significance of academic libraries.   methods – the academic library’s mooc services are derived from the extensive literature review and subsequently a research model based on extant literature has been developed to evaluate user behaviour. the research model is evaluated using confirmatory factor analysis methods. results – the academic library’s services for moocs have been categorized as, (a) user support services, (b) information services, and (c) infrastructure services. the study shows that each of these service categories have a positive impact on the library usage intention of the users. this in turn has a positive effect on the library’s perceived significance.   conclusion – the library services for mooc users defined in this research and the findings are useful for librarians to develop new service strategies to stay relevant for the user.     introduction   online education and distance education has been available for many years now, but many experts agree that massive open online courses (moocs) have been a driver of change in higher education by providing innovative ways of learning (zhang et al., 2019). according to a report published in 2017 by the european association of distance teaching institutions (eadtu), the number of higher education institutions offering moocs is increasing steadily, and the number of students opting for such courses in europe is significantly higher than in the us. in this report, jansen & konings (2017) also underline that the cooperation of libraries is an important factor in open education.   studies indicate that academic libraries are facing increased competition like every other business entity due to technological advances in information and technology. they are striving harder to maintain their role as an information provider in academic communities (iwu-james et al., 2020). academic libraries are not considered as the heart of the university anymore by the top leadership as academic and research information is also available from other sources (cox, 2018). osman & ahlijah (2021), studied to examine the relevance of university libraries in the 21st century. they found that user expectations from the academic library have changed, and the traditional roles of the library need to adapt to the new learning behaviour of users. the study showed that less than 10 percent of users prefer to visit the library but most of them prefer to use the library’s electronic resources, due to their easy access and availability. this study argues that the library is the centre of information and knowledge for the students and the academic library is an integral part of the university set-up. hence, the academic library must fulfil the core objectives of the parent institution for the curricular needs of the learners, teachers, and researchers. the library is a service-based institution that must strive to upgrade its potential users to habitual users. providing greater access to resources and user-centric services can help achieve this.   moocs are perceived as a disruptive innovation in higher education, with reach and potential much higher than traditional online courses. according to patru & balaji (2016), moocs are different from traditional online courses in four ways, (1) it is highly scalable, and designed for a theoretically unlimited number of users, (2) it is accessible without any fees, (3) there are no pre-requisites, and (4) entire course is online.  moocs offer an opportunity for academic librarians to have a greater influence on the faculty and students. academic libraries can involve themselves in moocs in many forms, ranging from traditional roles of information, instruction, and reference services, or in the form of advanced services like copyright check, oers, content creation, policy framework, and guidelines (wu, 2013).   moocs have gained importance in emerging economies like china (zhang et al., 2019), india (mahanta, 2020), malaysia (albelbisi, 2020), africa (rambe & moeti, 2017), etc. due to their potential to reduce the burden on university infrastructure, increase enrollments, improve quality of education and creating opportunities with equal access through digital means (badi & ali, 2016). academic libraries and moocs have yet to be examined together in the recent academic literature. most published articles on this topic appeared in the years 2013-17 (as per the current literature review), focusing on issues like copyright and licensing, open educational resources (oers), production of new courses, and policy issues. the goals of this exploratory study are to explore the suggested academic library services for moocs in the available literature; to propose moocs as a library service; to create a research model to find out possible library services for mooc users, and to determine its effects on the library’s perceived significance for users.   literature review   an extensive literature survey was carried out using the following keywords: library and moocs; mooc services; library services; mooc success; library in mooc era; mooc and higher education; moocs and librarian; user significance of library; library significance; academic library trends for a period of 2010 to 2021. most appropriate research articles were selected for carrying out the literature review. relevant citations from the primary literature survey were also explored for broadening the understanding of the research issues. research articles in the english language have only been considered for this review, although a considerable amount of research literature is available in the chinese language, mostly for which the abstracts were only available in english. such vernacular articles haven’t been considered in this research. this section can be discussed in two parts: academic library mooc services, and user-perceived significance of academic library.   mooc services of academic libraries   higher education institutions globally have included moocs in their curricula in various forms (fox, 2013). based on current trends in higher education, moocs are going to be integrated into the academic curriculum of higher education in the coming years (yanxiang, 2016).   the advent of moocs means change not only for the ways universities operate, but also the function of academic libraries. due to the different needs in diverse courses, libraries need to revive their services as the present ones are not enough to fulfil the emerging needs of mooc-based curricula. new services related to copyright, intellectual property, information literacy education, data synthesis, metadata, information sharing services, and others will be needed by the users to complete these courses (liu, 2016).   the relationship between moocs and academic libraries has been emphasized in the literature by several authors such as mahraj (2012), creed-dikeogu & clark (2013), gore (2014), yanxiang (2016), and others. the logic of relating these two entities is based on the following similarities (deng, 2019):   table 1 similarities between moocs and academic libraries objectives information sharing and dissemination of knowledge. users the students/ learners are the primary users. focus knowledge services. freedom there is freedom to select the kind of resource and knowledge acquired by the user.   the academic library specializes in information and services. this makes it the most suitable organization in the higher education system to drive the inclusion of moocs in the curriculum (luan, 2015). from an extensive literature review on the relationship between moocs and academic libraries, we realized that although many researchers have discussed the importance of academic libraries in the mooc era, the literature does not provide a consolidated account of possible mooc services of an academic library concerning its current roles and functions.   the discourse on academic libraries’ mooc services was started by becker (2013) of san jose state university, california. becker states that the mooc literature is ‘sparse’, and there needs to be an exploration of the possible involvement of academic libraries in mooc-based education. the primary focus in becker’s research was the development of a collection of open access resources for mooc users, as moocs have an international appeal, and the resource distribution seemed to be the most important issue on moocs.   gore (2014) also supported this idea and discussed the issues and challenges for academic libraries due to moocs. they are considered a disruptive technology in the field of education and gore suggests that librarians cannot have any subordinate role in mooc-based education. information literacy, involvement in the mooc production process, influencing instructors, copyright and licensing issues, the role of it infrastructure in mooc distribution and the scale of the mooc courses were some of the issues proposed in gore’s research, which directly concerned academic libraries.   in other words, the stage for academic library mooc services started getting prepared right after moocs arrived in 2012 (sanchez-gordon & luján-mora, 2014). followed by many other research articles on the relationship between moocs and academic libraries, as mentioned in table 2, these possible mooc services have been carefully collected from the literature and have been summarized to form the possible academic library services for mooc users.   in the following section, the research literature on issues pertaining to mooc-based higher education curriculum has been explored and mapped against the features and roles of academic libraries. based on this method, this study proposes the possible roles of any traditional academic library in providing services to mooc users. table 2 summarizes these library services for mooc users.   table 2 academic library services for mooc users   current roles and features of academic library possible mooc services roles citation(s) roles citation(s) 1 technical infrastructure kassim, 2009 broadband and technical infrastructure marrhich et al., 2020 2 constant upgradation of technology for changing information needs kaushik & kumar, 2016 managing moocs for various departments, meeting needs of different users. mune, 2015 3 cataloging and classification services kassim, 2009 cataloging and classification of moocs jie, 2019 4 information services for all departments kassim, 2009 moocs for all departments wang, 2017 5 use of integrated library system (ils) and online catalogs (opac) kassim, 2009 need of integrated platform for managing mooc information, instruction, evaluation and support services to all the users jie, 2019 6 procurement, distribution, management, preservation of reading and multi-media resources. kaushik & kumar, 2016 open educational resources, online resources, embedded content for moocs yanxiang, 2016; shapiro et al., 2017 7 services like reprography, document search and delivery, plagiarism check, printing, research assistance etc. gardner and eng, 2005 users also need all these services for successful completion of moocs. shapiro et al., 2017 8 library advisory committee for planning, developing and managing information needs of all the departments. liu, 2010 library can provide administrative services for moocs to all the departments. marrhich et al., 2020 9 library services are available at all times for its users. gardner and eng, 2005 mooc services on mobile platforms, self-support services and technical assistance for remote users. wang, 2017; kaushik, 2020 10 instruction support services kaushik & kumar, 2016 mooc instruction support services luan, 2015 11 inter-library networks for resource sharing kassim, 2009 resource sharing on library networks wang, 2017 12 training and orientation programs for library users gardner and eng, 2005 language training, technology training, information retrieval training, gulatee and nilsook, 2016; marrhich et al., 2020 13 publicity and awareness programs kaushik & kumar, 2016 publicity and awareness of moocs jie, 2019 14 departmental libraries and special libraries kaushik & kumar, 2016 departmental needs for advanced and customized information for specific moocs. mune, 2015 15 copyrights and licencing of library resources kaushik & kumar, 2016 copyrights and licencing of library resources for moocs kaushik & kumar, 2016   users’ perceived significance of the academic library   the current research needs to evaluate the effect on the perceived significance of academic libraries for its users if the mooc services are offered to them. according to the merriam-webster dictionary (n.d.), the definition of the word “significance”, is the “quality of being important”. to measure the significance of the library for its users, which is an abstract idea, the current research proposes to measure the user’s desire to use the library, as has been discussed in the concept of e-commerce systems success by molla & licker (2001). the higher the user’s intention to use a service, the higher the perceived significance of the academic library (the service provider).   this correlation between the library service usage and its perceived significance is in line with the research document academic library impact: improving practice and essential areas to research prepared by the association of college and research libraries (acrl) and the online computer library center (oclc) and authored by connaway et al., (2017). in this report, the code for “how library services need to be measured”, is its ‘usage and attendance’.                users’ intention to use an information system is a widely researched topic. the information systems-success model was proposed by delone and mclean (1992). this feedback model is applicable to information systems (is) applications. an updated model was proposed by delone and mclean in 2003 owing to the high acceptance of their earlier model and the vastly changing landscape of the information industry due to the onset of e-commerce businesses in the 2000s.   the quality antecedents of this new is-success model are service, systems and information. these three independent variables of this model can be altered individually. together these three independent variables influence the user’s derived satisfaction and usage intention of the information service. this model is explained in the form of a line diagram in figure 1.   figure 1 information systems success model (delone and mclean, 2003).   the mooc service of an academic library is also an information system, where the main users are the learners. so, it is logical to analyze the academic library’s mooc services in the light of the d&m iss model.   the academic library services for mooc users are also categorized into primary antecedents like in the updated d&m iss model (2003), namely, (i) system quality, (ii) information quality, and (iii) service quality. the adopted primary antecedents for this study in the context of mooc services are, (i) infrastructure services, (ii) information services, and (iii) user support services. they together influence the perceived significance of the library for its users.   the three adopted primary service categories for the library services for mooc users are displayed in table 3, with more details included. in all, a total of eighteen mooc user services have been listed in this table, classified into three primary service categories.   gaps identified from literature   the literature review on academic libraries and moocs in higher education has shown two research gaps that are addressed in this research:   ·        research gap: the library services for the mooc users have been discussed in the literature but there has been no available record of classifying them according to the traditional roles and functions of the library. ·        research gap: the diminishing perceived significance of academic libraries due to the internet and social media and the change in the learning and information-seeking behaviour of the students has been discussed in the literature (luan, 2015). also, the shift in the role of an academic library from passive academic support to active service and information provider for a mooc-based curriculum has been discussed (yanxiang, 2016). but, the change in the perception of the library’s significance for users due to this changing role in the mooc era has not been properly addressed in the available literature.   table 3 primary antecedents and measures (mooc services of academic library) primary antecedents measures citation(s) infrastructure services technical facilities of the academic library marrhich et al., 2020 infrastructure facilities of the academic library ning et al., 2016 embedded content in online courses luan, 2015 broadband connection chen, 2014 library resources on mobile platforms yang, 2015 user support services technical support for mooc users jie, 2019 user specific information services yang, 2015 information literacy programs for mooc users ning et al., 2016 technology training for users marrhich et al., 2020 training users in english language gulatee and nilsook, 2016 support services for mooc users kaushik, 2020 mooc specific question and answers for user self service mune, 2015 inter-library resource sharing wang, 2017 information services digital resources shapiro et al., 2017 open educational resources yanxiang, 2016 course material ackerman et al., 2016 continuous updation and mooc resources yanxiang, 2016 classification and cataloging of moocs jie, 2019   aims   this exploratory study has two main objectives:   1.    to explore the possible services of an academic library for mooc users. 2.    to establish the relationship between the library’s mooc services and the perceived significance of the library for its users.   hypotheses and research model   the three categories of academic library services for mooc users form the primary antecedents. these antecedents as described in table 3, are infrastructure services, user support services, and information services. these independent variables are proposed to influence the library user’s desire to use the library services, the “intention to use” is proposed to have a positive influence on the perceived significance of academic library for its users. the research model indicating these relationships is illustrated in figure 2.   figure 2 research model.   the primary antecedent of “system quality” proposed in the d&m model (2003) has been modified in the current research as “infrastructure services” for mooc users. this refers to the consistency of service and the features of the service provided to users to support mooc consumption. this encapsulates the performance characteristics and features of the physical and technical infrastructure provided and maintained by academic libraries. this would also include making moocs accessible to students with disabilities, or for students without sufficient hardware and software (bohnsack & puhl, 2014). mooc infrastructure should be scalable and modular, making it suitable for long-term maintenance (chunwijitra et al., 2020). providing mooc infrastructure services is easier said than done, as traditional universities globally are not equipped to support such a highly demanding and ever-evolving environment. many outsourcing companies are now moving quickly to provide such e-learning infrastructure (baggaley, 2013). there would be challenges regarding quality assurance and standards, and training of teachers and students on the e-learning systems, to ensure the quality of the mooc-based education (baggaley, 2013). the study intends to explore whether the ‘mooc infrastructure services’ positively influence the user’s desire to use the library services. the subsequent hypothesis can be stated as:   h1: the mooc user’s desire to use the library services depends upon user’s attitude towards the features and consistency of its infrastructure services.   the primary antecedent of “service quality” in the d&m model (2003) has been modified in the current research to “user support services” for the users of an academic library. it refers to academic library services, which could facilitate and ease the mooc consumption and assimilation by the library users. the onset of moocs has challenged the traditional concepts of formal education. the learners, teachers, and universities are not equipped and trained enough to assimilate moocs in their current form. technical assistance or training for information search and retrieval are the primary challenges in making moocs inclusive. the primary objective of introducing moocs in higher education have been their ability to democratize quality education, but the technical and information divide acts as a barrier to achieving this objective. the mooc support services have been given due importance in the research literature. the role of libraries has evolved from information provider to knowledge provider. this change needs to be supported by advanced it-based technologies such as machine learning and ai to provide customized knowledge services to various user profiles (luan, 2015). this would require highly trained library professionals, a specialized technical team, trainers, and counsellors. the role of academic librarian would change drastically, probably a new generation of information professionals would be required to adapt to the new roles.       mooc user support services assist users in completing moocs (gregori et al., 2018). this study proposes that an academic library’s user support services have a direct effect on the user’s desire to use the library services. the subsequent hypothesis can be stated as:   h2:  the mooc user’s desire to use the library services depends upon the user’s attitude towards its user support services.   “information services” is derived from the d&m model’s antecedent of “information quality”. this antecedent may be defined as the nature and significance of the information offered by academic libraries to the mooc learners. the mooc model of curriculum is based on the concept of “embedded content” based learning (yanxiang, 2016). mooc courses generally require multiple reading or reference materials. currently, the library resources consist of electronic versions of textbooks and e-books. moreover, these resources are scattered across various databases in the library. hence, the most challenging task for the libraries would be to integrate these distributed learning resources into the mooc platforms with seamlessly embedded links.                another challenge with resource content for a mooc's reference needs is the copyright check. the license terms prohibit the use of copyrighted content without permission or payment. the use of open educational resources (oers) becomes inevitable in such cases, or the need to re-negotiate the license terms with the resource providers and databases, for the use of their copyrighted content for mooc-based curricula in the university (luan, 2015). oers are educational content available for public access (atkins et al., 2007). if oers are used as the building blocks of moocs, the library would have to spend less time and resources on copyright management of the content. course-specific self-help faqs or the need for sufficient focus on each of the university’s offered courses for their required content, along with a regular update of the references makes the mooc information service even more challenging.   one more dimension in this context is the need to establish inter-library cooperation through the network for information resource sharing (wang, 2017). the establishment of a library network involves several operational issues which govern its functionality. these issues are described by kaul (2010), as 5 c’s: connectivity, cost, computers, client, and content. the library networks in the knowledge economies also involve sharing of tacit (non-published) knowledge acquired by the different institutions. research shows that only a few library networks sustain after the initial phase of development and initiation. resource sharing within an international library network is even more difficult with geographic, technical, and institutional barriers (butler et al., 2006). the subsequent hypothesis can be stated as:   h3:  the mooc user’s desire to use the library services depends upon the user’s attitude towards its information services.   the current research needs to evaluate the perceived significance of academic libraries for their users. to measure the significance of the library for its users, which is an abstract idea, the current research measures the user’s intention to use the library services, as has been discussed in the concept of e-commerce systems success by molla & licker (2001). according to academic library impact: improving practice and essential areas to research, the code for how library services need to be measured, the provided value is its “usage and attendance” (connaway et al., 2017). the higher the user’s intention to use, the higher would be the perceived significance of library services. hence, the hypothesis can be formed as: h4:  the mooc user’s desire to use the library services influences the user’s perceived significance of academic library.   methods   survey design   the relationships between the independent and the dependent variables of the research model have been tested using an empirical approach, using feedback from library users on a structured questionnaire. a printed schedule was used with a likert scale for measuring attitude. the likert scale ranged from “strongly disagree” to “strongly agree”, ranging from a corresponding response of 1 to 5 respectively. similar scales have been used in previous studies for evaluating information success scales. the questionnaire was prepared in the english language as it is the primary language for teaching and instruction for indian higher education students. the following scales were used in this questionnaire, derived from the extant literature: mooc infrastructure services (5 items), mooc user support services (8 items), mooc information services (5 items), user’s perceived significance of academic library (6 items). demographic data were collected on age, gender, and education. the full scales can be found in the appendix.   the scale’s content validity was determined with the help of a review done by three subject area experts. the experts’ direct personal experience and familiarity with the construct help establish content validity. deciding upon the number of subject area experts depends upon the researcher’s discretion. a greater number of experts may reduce the possibility of reaching a common conclusion. generally, no less than three and no more than five experts are referred to in the process (zamanzadeh et al., 2015). this step is essential to ensure that proper language and questions are used and that the design of the research instrument is as per the desired objectives. the validity of the survey instrument is done at several stages of research through many available methods. in this research, the content validity is determined before the implementation of the survey on the survey frame. following this, a test run on 50 library users was done to ensure the ability of the questionnaire to properly evaluate the research model and its appropriateness for the target respondents, before implementing it in a large-scale survey. the respondents for this pilot study were university students who have enrolled for or completed at least one mooc course and are academic library users.   mooc services of library – evaluation scale   mooc services of library evaluation scale, given below in table 4, is derived from table 3 given above, which forms the basis of the survey scales of this study. the scale is designed based on the assertion that academic library’s decision-making regarding suggested mooc services should be based on user experience. the user’s desire to use the library services and the user’s perception of the usefulness of the provided services forms the basis of this evaluation scale. the library services for the mooc users are divided into three categories, as described earlier in this article. these three categories are ‘infrastructure services’, ‘information services’, and ‘user support services’ table 4 presents this evaluation scale for the users.   table 4 mooc services of library – evaluation scale category of service mooc services of library poor (1) below average (2) average (3) good (4) excellent (5) infrastructure services technical facilities of the academic library           infrastructure facilities of the academic library           embedded content in moocs           broadband connection           library resources on mobile platforms           information services e-learning resources           open educational resources           learning resources           continuous updation and mooc resources           classification and cataloging of moocs           user support services technical support for mooc users           customized information services           mooc information literacy programs for users           technology training for users           english language training for users           support services for mooc users           mooc specific faqs for user self service           inter-library resource sharing             sampling and method   this survey engaged university students who are academic library users from ten universities and institutions from the capital territory of rajasthan state in india. a survey method is used for this research because of its potential for generalizing the findings for a larger population with similar characteristics. the survey used a tailored design method as proposed by dillman (2011). this method was used to increase the response rates. the respondents were provided with a pre-notice intimation from their subject instructors. dillman proposed that by using this technique the response rates are positively affected. the pre-notice primes the respondents about the upcoming survey followed by a gratitude message. the survey was administered in print form after a gap of 2-3 days after the priming. 30 respondents from each university were included in this survey who have enrolled for or completed at least one mooc course and are academic library users. the respondents were first briefed about the purpose and usefulness of the study and were assured that their responses would be kept confidential. the respondents were guided through the questionnaire followed by a short gratitude message. this data collection was a part of a larger study done by the researchers, and out of the sample size of 300 participants, 257 forms were included in the study. the forms were selected based on their completeness. hence, 85.67 percent of the response rate was recorded. the survey participants had a recorded mean age of 21.3 years. in terms of gender distribution, there were 168 males and 89 females. 144 respondents were undergraduates and 113 respondents were postgraduates.   results   to understand the relationship between the multiple latent variables, confirmatory factor analysis (cfa) was done using a 5-point likert scale with ‘1’= strongly disagree to ‘5’= strongly agree. the reliability of the research instrument was determined by using composite reliability (cr) values. the discriminant validity is determined using the ave validity method. it determines that the constructs are independent of each other and are unrelated. the average variance extracted value’s positive square root needs to be higher when compared against the highest value of the correlation of each factor against all other factors. the fornell-larcker ratio (1981) has been used to identify the convergent validity of the instrument. it gives us the level of confidence in how well the constructs are measured by the survey items. ave values of more than 0.50 are considered acceptable and values more than 0.70 are considered good. composite reliability (cr) values of more than 0.70 are considered acceptable (chin, 1998). the scale properties shown in table 5 are under acceptable limits. so, it can be concluded that the research instrument has achieved discriminant validity successfully.   table 5 scale properties factors information services (is) user support services (ss) infrastructure services (if) perceived significance of library (sig) flr 0.88 0.87 0.74 0.82 ave 0.58 0.72 0.61 0.68 cr 0.83 0.68 0.76 0.77   the fit indices have been calculated in the confirmatory factor analysis for this model. the indices considered for this study are recorded in table 6. the acceptable value for ‘root means square approximation’ is less than 0.08, and for all other indices, the acceptable values are equal to or greater than 0.90. the values for all the cfa fit indices are significant.   the regression coefficients of the dependent and independent variables are indicated by gamma (γ) values, as shown in figure 3 with (***). the model shows that all the three primary antecedents of library mooc services, namely, “information services”, “infrastructure services”, and “user support services” have a positive influence on the library user’s desire to reuse the library services, and this also has a direct relationship with the perceived significance of the library for its users.   table 6 model fit values chi-square value 188.545 significance value 0.110 degrees of freedom 164 chi square/ degrees of freedom 1.149 root mean square error of approximation 0.072 goodness of fit index 0.937 adjusted goodness of fit index 0.932 tucker lewis index 0.874 comparative fit index 0.956 incremental fit index 0.945 normed fit index 0.903       figure 3 ‘library’s perceived significance’ structural equation model.   academic library services for mooc users – evaluation scale   current research on the user perception of the significance of academic libraries allows us to form an evaluation scale for the library’s mooc services. this measurement scale has a total of 18 mooc services of the academic library. the highest score possible for this scale is 90 (18 * 5), and the possible lowest score is 18 (18 * 1). so, the scores can be easily categorized into three categories, (1) the low score (18 to 42; least 1/3rd cumulative value of scores), (2) medium score (43 to 66; median 1/3rd cumulative value of scores), and (3) high score (67 to 90; highest 1/3rd cumulative value of scores). the respondents of this mooc service evaluation scale are the learners, preferably from every academic department, to have an equal representation of the library users in this survey. contrarily, this scale can also be applied to the library users of any specific academic department, to scale the mooc service perception of any particular department.                the cumulative value of scores received on this evaluation scale would assist in evaluating and benchmarking the library’s services to its mooc users. this tool can be useful for the policymakers, to plan library activities and budgets, for a higher education institution using mooc based curriculum. the national educational rating agencies and certification bodies can also use this instrument to determine the level of preparedness of any institution with a mooc-based curriculum. many issues about the library’s mooc services can be easily addressed through national knowledge infrastructure and policy initiatives (yuan et al., 2014).   discussion   academic libraries were gradually losing their importance of being the heart of the university. the information collection and services were facing a decline in usage, primarily due to the increasing penetration of the internet and the availability of mobile devices (cox, 2018). the information and learning resources being available to the learners at any time and from anywhere had diminished the role of the libraries (luan, 2015).   moocs have entered the educational landscape in the year 2012 (also known as the year of moocs) (pappano, 2012), and since then, the mooc movement has been joined by the elite institutions, private and non-profit organizations, and are now getting rapidly promoted by the government’s world-over to increase the reach and quality of higher education (albelbisi & yusop, 2020). the adoption of moocs by universities across the globe has led their libraries to provide mooc information services. the academic libraries are specialized bodies for information services within any university, hence, their role in mooc based higher education curriculum is pivotal (luan, 2015).   acrl (2000) has defined information literacy as “the set of abilities requiring individuals to recognize when information is needed and can locate, evaluate and use effectively the needed information”. moocs have been broadly classified as x-moocs (extended moocs) and c-moocs (connectivist moocs). x-moocs are more popular and require a lower level of information literacy as the course content is generally prescribed by the developer and the understanding of the content is evaluated through tests. conversely, c-moocs are more participatory with learners required to aggregate, remix, repurpose and feed forward the information, based on the acrl information literacy standards (bond, 2015). libraries can play an important role in providing information literacy for mooc users. likewise, many different library services for mooc users have been proposed, like providing a collection of mooc resources, copyright services, providing it infrastructure, mining of mooc resources, mooc production, and providing online and offline space for mooc users (yanxiang, 2016). the library services for the mooc learners have been discussed and commented upon by many authors in the available literature. in this article, a comprehensive list of possible library services for moocs have been curated, based on the extant literature, and to keep them in perspective these services have been compared and segregated according to the traditional roles and features of an academic library. such a list would prove extremely useful for the libraries, institutions, and policymakers to decide upon the development and inclusion of mooc services for their users.   furthermore, to understand the effect of mooc services of the academic library, on the user’s perceived significance of the library, an empirical study has been conducted using cfa. the research model is based on the premise that the “significance of library” being an abstract idea, can be measured using the user’s desire to use the library service, as has been proposed in the academic library impact report by connaway et al. (2017).   the three categories of academic library services for mooc users form the primary antecedents, namely, “infrastructure services”, “user support services”, and “information services”. these exogenous variables are proposed to influence the library user’s desire to use the library services, and, as derived from the information systems success model (delone and mclean, 2003), the “intention to use” is proposed to have a positive influence on the perceived significance of academic library for its users (endogenous variables).   this study on the user perception of the significance of academic libraries makes it possible, to form an evaluation scale for the library’s mooc services. this evaluation scale can be used by the university administration and the national educational policymakers for evaluation, planning and budgeting of knowledge resources.   conclusion   this research attempts to establish an argument that mooc services of academic libraries increase the library user’s perceived significance of the library. these services, although they seem very logical and feasible due to the current technological developments, have their challenges and difficulties in adoption. this research also presents the issues and challenges for the universities, academic libraries, and information professionals for information needs while adopting mooc based higher education curriculum.   this research was conducted in the context of indian higher education, with a generalization of the concepts for developing and emerging economies. another possible limitation of this research is that it is based on delone and mclean’s information systems success model, where the user’s “intention to use”, which is an attitude has been related to ‘use’, which is a behaviour trait. in real world situations, attitude and behaviour are not always related. the administration of similar studies in other countries and educational systems would improve the findings and generalizations. suggested future research directions are:   1.      studies to explore the organizational and leadership challenges to be faced by library management for delivering mooc services. 2.      studies to understand the possibilities and dynamics of international library networks for content and knowledge sharing for offering mooc services. 3.      to keep moocs manageable by the libraries and to provide access to the public, oers play a very crucial role. oers make moocs more accessible. ideally, oers should form the building blocks for the mooc framework to truly democratize higher education. however, challenges regarding worldwide accreditation and adherence to standards with oers need to be explored. 4.      moocs face a high student dropout rate, and several reasons for this have been pointed out in the literature (onah, sinclair & boyatt, 2014). studies have shown that a better planned mooc instructional design can accommodate the diversity of students with the scope of personalized learning (guàrdia, maina & sangrà, 2013). the use of artificial intelligence and technologies such as machine learning can assist in better understanding students’ learning behaviour. librarians can assist instructors in profiling the learners and developing a better instructional design.   author contributions   flora charles lazarus: conceptualization (lead), methodology (lead), investigation (lead), formal analysis (lead), writing – original draft (lead), writing – review & editing (lead) rajneesh suryasen: conceptualization (supporting), methodology (supporting), investigation (supporting), formal analysis (supporting), writing – original draft (supporting), writing – review & editing (supporting)   references   ackerman, s., mooney, m., morrill, s., morrill, j., thompson, m., & balenovich, l. k. (2016). libraries, massive open online courses, and the importance of place: partnering with libraries to explore change in the great lakes. new library world, 117(11/12), 688-701. https://doi.org/10.1108/nlw-08-2016-0054   acrl. (2000). information literacy competency standards for higher education. http://www.ala.org/acrl/standards/informationliteracycompetency   albelbisi, n. a., & yusop, f. d. (2020). systematic review of a nationwide mooc initiative in malaysian higher education system. electronic journal of e-learning, 18(4), 287-298. https://doi.org/10.34190/ejel.20.18.4.002   atkins, d. e., brown, j. s., & hammond, a. l. (2007). a review of the open educational resources (oer) movement: achievements, challenges, and new opportunities. report to the william and flora hewlett foundation. http://www.hewlett.org/uploads/files/reviewoftheoermovement.pdf   badi, s., & ali, m. e. a. (2016). massive open online courses (mooc): their impact on the full quality in higher education institutions "rwaq: saudi educational platform for mooc". journal of library and information sciences, 4(1), 73-101. https://doi.org/10.15640/jlis.v4n1a6   baggaley, j. (2013). mooc rampant. distance education, 34(3), 368-378. https://doi.org/10.1080/01587919.2013.835768   becker, b. w. (2013). connecting moocs and library services. behavioral & social sciences librarian, 32(2), 135-138. https://doi.org/10.1080/01639269.2013.787383   bohnsack, m., & puhl, s. (2014). accessibility of moocs. in miesenberger, k., fels, d., archambault, d., peňáz, p. & zagler, w. (eds.) international conference on computers for handicapped persons (pp. 141-144). springer. https://doi.org/10.1007/978-3-319-08596-8_21   bond, p. (2015). information literacy in moocs. current issues in emerging elearning, 2(1), 6. https://scholarworks.umb.edu/ciee/vol2/iss1/6/   butler, b. a., webster, j., watkins, s. g., & markham, j. w. (2006). resource sharing within an international library network: using technology and professional cooperation to bridge the waters. ifla journal, 32(3), 189-199. https://doi.org/10.1177/0340035206070165   chen, y. (2014). investigating moocs through blog mining. the international review of research in open and distributed learning, 15(2), 85-106. https://doi.org/10.19173/irrodl.v15i2.1695   chin, w. w. (1998). commentary: issues and opinion on structural equation modelling. mis quarterly, 22(1), 7-16. https://www.jstor.org/stable/249674   chunwijitra, s., khanti, p., suntiwichaya, s., krairaksa, k., tummarattananont, p., buranarach, m., & wutiwiwatchai, c. (2020). development of mooc service framework for life long learning: a case study of thai mooc. ieice transactions on information and systems, 103(5), 1078-1087. https://doi.org/10.1587/transinf.2019edp7262   connaway, l. s., harvey, w., kitzie, v., & mikitish, s. (2017). academic library impact: improving practice and essential areas to research. http://www.ala.org/acrl/sites/ala.org.acrl/files/content/publications/whitepapers/academiclib.pdf   cox, j. (2018). positioning academic library within the institution: a literature review. new review of academic librarianship, 24(3-4), 1-25. https://doi.org/10.1080/13614533.2018.1466342   creed-dikeogu, g., & clark, c. (2013). are you mooc-ing yet? a review for academic libraries. kansas library association college and university libraries section proceedings, 3(1), 9-13. https://doi.org/10.4148/culs.v1i0.1830   delone, w. h., & mclean, e. r. (1992). information systems success: the quest for the dependent variable. information systems research, 3(1), 60-95. https://doi.org/10.1287/isre.3.1.60   delone, w. h., & mclean, e. r. (2003). the delone and mclean model of information systems success: a ten-year update. journal of management information systems, 19(4), 9-30. https://doi.org/10.1080/07421222.2003.11045748   deng, y. (2019). construction of higher education knowledge map in university libraries based on mooc. the electronic library, 37(5), 811-829. https://doi.org/10.1108/el-01-2019-0003   dillman, d. a. (2011). mail and internet surveys: the tailored design method-2007 update with new internet, visual, and mixed-mode guide. john wiley & sons.   fornell, c., & larcker, d. f. (1981). evaluating structural equation models with unobservable variables and measurement error. journal of marketing research, 18(1), 39-50. https://doi.org/10.1177/002224378101800104   fox, a. (2013). from moocs to spocs. communications of the acm, 56(12), 38-40. https://doi.org/10.1145/2535918   gardner, s., & eng, s. (2005). what students want: generation y and the changing function of the academic library. portal: libraries and the academy, 5(3), 405-420. https://doi.org/10.1353/pla.2005.0034   gore, h. (2014). massive open online courses (moocs) and their impact on academic library services: exploring the issues and challenges. new review of academic librarianship, 20(1), 4-28. https://doi.org/10.1080/13614533.2013.851609   gregori, e. b., zhang, j., galván-fernández, c., & de asís fernández-navarro, f. (2018). learner support in moocs: identifying variables linked to completion. computers & education, 122, 153-168. https://doi.org/10.1016/j.compedu.2018.03.014   guàrdia, l., maina, m., & sangrà, a. 2013. mooc design principles: a pedagogical approach from the learner's perspective. elearning papers, 33. https://r-libre.teluq.ca/596/1/in-depth_33_4.pdf   gulatee, y., & nilsook, p. (2016). mooc's barriers and enables. international journal of information and education technology, 6(10), 826-830. https://doi.org/10.7763/ijiet.2016.v6.800   iwu-james, j., haliso, y., & ifijeh, g. (2020). leveraging competitive intelligence for successful marketing of academic library services. new review of academic librarianship, 26(1), 151-164. https://doi.org/10.1080/13614533.2019.1632215   jansen, d. & konings, l. (2017). mooc strategies of european institutions. status report based on a mapping survey conducted in november 2016 february 2017. eadtu. http://eadtu.eu/documents/publications/oeenm/mooc_strategies_of_european_institutions.pdf   jie, s. u. n. (2019). innovative work of university libraries for assisting mooc instruction. cross-cultural communication, 15(1), 7-12.   kassim, n. a. (2009). evaluating users' satisfaction on academic library performance. malaysian journal of library & information science, 14(2), 101-115.   kaul, s. (2010). delnet-the functional resource sharing library network: a success story from india. interlending & document supply, 38(2), 93-101. https://doi.org/10.1108/02641611011047169   kaushik, a., & kumar, a. (2016). mooc-ing through the libraries: some opportunities and challenges. international journal of information dissemination and technology, 6(1), 21-26.   liu, h. (2016, november). analysis on the current situation of university library service under mooc environment. 3rd international conference on management, education technology and sports science (metss 2016). (pp. 425-427). atlantis press. https://doi.org/10.2991/metss-16.2016.86   liu, r. (2010). the value of a library advisory board in a research library. in the new face of value: 2010 best practices for government libraries (pp. 20-24). http://www.lexisnexis.com/tsg/gov/best_practices_2010.pdf   luan, x. (2015, june). research on service innovation of university library under moocs. in international conference on education, management and computing technology (icemct-15). (pp. 1496-1499). atlantis press. https://doi.org/10.2991/icemct-15.2015.315   mahanta, s. (2020). paradigm shift in higher education through ict: conventional to moocs-a case study of dibrugarh university. indian journal of educational technology, 2(2), 41.   mahraj, k. (2012). using information expertise to enhance massive open online courses. public services quarterly, 8(4), 359-368. https://doi.org/10.1080/15228959.2012.730415   marrhich, a., lafram, i., berbiche, n., & el alami, j. (2020). a khan framework-based approach to successful moocs integration in academic context. international journal of emerging technologies in learning (ijet), 15(12), 4-19. https://doi.org/10.3991/ijet.v15i12.12929   molla, a., & licker, p. s. (2001). e-commerce systems success: an attempt to extend and respecify the delone and maclean model of is success. journal of electronic commerce research, 2(4), 131-141. http://www.jecr.org/node/290   mune, c. (2015). massive open online librarianship: emerging practices in response to moocs. journal of library & information services in distance learning, 9(1-2), 89-100. https://doi.org/10.1080/1533290x.2014.946350   ning, q., jiyong, l., yongming, m., & bin, w. (2016). research on the college library information literacy education in mooc environment. international conference on education, management, computer and society (emcs 2016). (pp. 782-784). atlantis press.   onah, d. f., sinclair, j., & boyatt, r. 2014. dropout rates of massive open online courses: behavioural patterns. in edulearn14 proceedings. (pp. 5825-5834).          iated academy. http://library.iated.org/view/onah2014dro   osman, h., & ahlijah, s. a. (2021). the relevance of library to students of the school of public health of the university of health and allied sciences, in ho, ghana. library philosophy and practice, 4631. https://www.proquest.com/docview/2492710407   pappano, l. (2012, november 2). the year of the mooc. the new york times. retrieved from https://www.nytimes.com/2012/11/04/education/edlife/massive-open-online-courses-are-multiplying-at-a-rapid-pace.html   patru, m., & balaji, v. (2016). making sense of moocs: a guide to policy makers in developing countries. unesco. https://unesdoc.unesco.org/ark:/48223/pf0000245122    rambe, p., & moeti, m. (2017). disrupting and democratising higher education provision or entrenching academic elitism: towards a model of moocs adoption at african universities. educational technology research and development, 65(3), 631-651. https://doi.org/10.1007/s11423-016-9500-3   sanchez-gordon, s., & luján-mora, s. (2014). moocs gone wild. in proceedings of the 8th international technology, education and development conference (inted 2014). (pp. 1449-1458).   shapiro, h. b., lee, c. h., roth, n. e. w., li, k., çetinkaya-rundel, m., & canelas, d. a. (2017). understanding the massive open online course (mooc) student experience: an examination of attitudes, motivations, and barriers. computers & education, 110, 35-50. https://doi.org/10.1016/j.compedu.2017.03.003   merriam-webster dictionary (n.d.). significance. retrieved april 26, 2019 from https://www.merriam-webster.com/dictionary/significance   wang, y. (2017, february). current situation, problems, and countermeasures of mooc service in university library. in international conference on humanities science, management and education technology (hsmet 2017). (pp. 602-606). atlantis press. https://doi.org/10.2991/hsmet-17.2017.118   wu, k. (2013). academic libraries in the age of moocs. reference services review, 41(3), 576-587. https://doi.org/10.1108/rsr-03-2013-0015 yang, j. (2015, november). study on the development strategy of the future library under information environment. in international conference on industrial technology and management science. (pp. 964-966). atlantis press. https://doi.org/10.2991/itms-15.2015.232   yanxiang, l. (2016, december). service innovations of university libraries in the mooc era. in 8th international conference on information technology in medicine and education (itme). (pp. 744-747). ieee. https://doi.org/10.1109/itme.2016.0173   yuan, l., powell, s. j., & olivier, b. (2014). beyond moocs: sustainable online learning in institutions. https://e-space.mmu.ac.uk/619736/1/beyond-moocs-sustainable-online-learning-in-institutions.pdf   zamanzadeh, v., ghahramanian, a., rassouli, m., abbaszadeh, a., alavi-majd, h., & nikanfar, a. r. (2015). design and implementation content validity study: development of an instrument for measuring patient-centered communication. journal of caring sciences, 4(2), 165. https://doi.org/10.15171/jcs.2015.017   zhang, j., sziegat, h., perris, k., & zhou, c. (2019). more than access: moocs and changes in chinese higher education. learning, media and technology, 44(2), 108-123. https://doi.org/10.1080/17439884.2019.1602541     appendix survey items constructs items measures infrastructure services (if) if1 the technical facilities of academic library are important for the success of my mooc course. if2 infrastructure facilities of academic library are important for the success of my mooc course. if3 embedded content in moocs would increase the success of my mooc course. if4 high-speed internet access is important for the success of my mooc course. if5 library resources on mobile devices would increase the success of my mooc course. user support services (ss) ss1 technical support is important for the success of my mooc course. ss2 customized information services are important for the success of my mooc course. ss3 mooc information literacy programs would increase the success of my mooc course. ss4 technology training would increase the success of my mooc course. ss5 english language training would increase the success of my mooc course. ss6 support services for moocs are important for the success of my mooc course. ss7 mooc specific faqs for user self service would increase the success of my mooc course. ss8 inter-library resource sharing would increase the success of my mooc course. information services (is) is1 e-learning resources of the academic library would help me in my mooc course. is2 availability of a collection of open educational resources for moocs is important for the success of my mooc course. is3 availability of learning resources for mooc users is important for the success of my mooc course. is4 continuous updation and mooc resources is highly desirable for my moocs. is5 indexed, ranked, and organized mooc courses would be highly desirable. user’s perceived significance of academic library (sig) sig1 library’s mooc services would increase my reliance on the library. sig2 library’s mooc services would increase my usage of the library services. sig3 library’s mooc services would increase my chances of completion of moocs. sig4 library’s mooc services would help me in enhancing my academic performance. sig5 library’s mooc services would help me become more employable. sig6 library’s mooc services would increase the overall significance of the library for my academic journey.     evidence based library and information practice commentary   a 2cul collaborative ethnographic assessment of humanities doctoral students: design, implementation and analysis   gabriela castro gessner research and assessment analyst cornell university ithaca, new york, united states of america email: agc24@cornell.edu   damon e. jaggars associate university librarian for collections and services columbia university new york, new york, united states of america email: djaggars@columbia.edu   jennifer rutner consultant new york, new york, united states of america email: jenrutner@gmail.com   kornelia tancheva associate university librarian for research and learning services cornell university ithaca, new york, united states of america email: kt18@cornell.edu   received: 7 feb. 2015     accepted: 14 may 2015      2015 gessner, jaggars, rutner, and tancheva. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   introduction   ethnographic studies of various user groups have flourished within libraries in recent years. most of these studies focus on planning service programs, facilities, and end-user interfaces, following a foundational tenet of participatory design – that systems and tools are best designed with engaged input from their users (foster & randall, 2007). the pioneering effort to design library spaces on the basis of ethnographic research findings at the university of rochester, since extended to other areas of library service (foster & gibbons, 2007), has led a number of academic research libraries to ground planning efforts in similar research methodologies.   the advantages of utilizing ethnographic research as a planning tool derive from observing subjects in their work process and capturing their experiences in their own words. combined with data measuring actual user behaviour, qualitative information gathered from interviews and observations provide a powerful tool for improving customer service and the end-user experience. while many early efforts centered on undergraduate academic work practices, more recent studies focus on the work of “serious researchers,” a frequently used catchall denoting faculty and graduate students. examples include case studies produced at the broad discipline level by the research information network, design projects concentrated on advanced researchers (foster, clark, tancheva, & kilzer, 2011), and efforts by scholars themselves to examine their own research workflows and the library’s role within those processes (abbott, 2008).   graduate students, and specifically doctoral students in the humanities, represent fertile ground for libraries interested in using ethnographic inquiry for service improvement and planning. humanities doctoral students are some of the most frequent and dedicated library users, given the nature of their research programs. a number of recent studies show that these students take longer to complete their programs and drop out at a higher rate than those in the sciences and social sciences (ehrenberg, zuckerman, groen, & brucker, 2010; national research council, 2010; hoffer & welch, 2006). contributing factors are numerous and include the availability of adequate funding, prospects for employment after completion, and the quality of students’ relationships with their faculty advisors – all important variables in completing a doctoral degree in a timely fashion (ehrenberg et al.). this intense interest in doctoral student completion and retention is underpinned by a growing anxiety about graduate education and the future of the academy (ehrenberg & kuh, 2009) which has, in turn, spawned a cottage industry of guide books for both current and future graduate students (hume (2005) and semenza (2010) were two guidebooks often cited by study subjects).   most research on doctoral student success does not discuss the library as a factor affecting completion or retention. in an attempt to fill this gap, the research libraries at columbia and cornell universities (2cul) conducted a collaborative ethnographic user needs study investigating the needs of doctoral students in the humanities, focusing specifically on the question of whether the library could positively impact student success (gessner, jaggars, rutner, & tancheva, 2011). the study was supported by grants from the gladys krieble delmas foundation, the council on library and information resources, and funding from the respective graduate schools at cornell and columbia. this funding covered equipment purchases, incentives for interview participants, training, and some modest staffing support for the project.   in summary, the study focused on doctoral students in the humanities at any stage of their programs. between the two institutions, the research team conducted 5 focus groups with 27 participants and 45 individual interviews. data gathered from the focus groups were used to refine the two protocols used in the interviews. written questionnaires were developed and administered at the end of each focus group and interview session. the interviews lasted between 60 and 90 minutes and were conducted in person by teams of two library staff members, except for two interviews, which were conducted via telephone.   the study concentrated initially on students enrolled in english, religion, history, and classics doctoral programs, but participation was expanded to include other humanities disciplines at both institutions. history and english were the only two disciplines to overlap at both institutions, and they also contributed the highest number of participants. the subjects varied in age from 21 to 75 years old, and their academic backgrounds and experience with libraries, archives, and academic writing ranged dramatically. almost two-thirds of all participants had advanced to doctoral candidacy. over half of the interviewees had earned advanced degrees (typically a master’s degree) prior to starting their doctoral program.   interviews revealed that even though there is no “typical” humanities doctoral student, there are institutional and library-related concerns that these students share and consider important in their pursuit of advanced degrees. while interviewees confirmed the importance of other factors already identified in the literature (funding, future employment prospects, and the faculty advisor relationship), their comments on what the library does and might do to contribute to their success were of particular interest. the opportunities for libraries that emerged from the study included providing work and social space, fostering community, ensuring access to deep research collections, providing assistance in supporting both research and teaching, and nurturing the development of doctoral students as scholars.   the detailed results of the study, including an in-depth demographic analysis, are reported elsewhere (gessner et al., 2011). the current paper will focus on the process of conducting a collaborative ethnographic study between two research libraries and student populations. the paper will examine the processes taken to design and administer the study and analyze the resulting data within an inter-institutional, collaborative framework. the project leaders identified both opportunities and challenges while completing the project, including addressing differences in institutional review board (irb) procedures and crafting instruments, and analyzing results collaboratively, across two research teams and different institutional cultures.   project organization   team structure and project management   by the end of the project, a total of 22 individuals (including 7 students) across both campuses had contributed in some way to the success of the study. the core research team consisted of 11 library staff members who contributed their time in addition to their regular duties (see appendix for a listing of team members). only the project manager from cornell received a 25% leave from regularly assigned duties to support the study.   the columbia team consisted of the associate university librarian for collections and services (the co-principle investigator (pi) from columbia), the assessment and planning librarian, who managed the overall project and the local irb process, five staff members from across the organization, including four subject specialist librarians and a paraprofessional access services supervisor, and a graduate student research assistant.   as the project manager for columbia, the assessment and planning librarian served as the primary liaison with cornell. working with the project manager, the research assistant coordinated the many daily tasks, scheduled interviews, ensured that interviewers were assigned for each interview, prepared interview materials, organized and filed interview recordings, and shared data with cornell.   the columbia team met routinely throughout the course of the 18-month project. team members were recruited to participate based on their experience with or interest in user assessment, familiarity with the population to be researched, and ability to dedicate time to a long-term project. the supervisors of each team member were consulted to ensure that time would be made available to dedicate to the project without negatively impacting their primary job responsibilities. team members were responsible for conducting interviews, data analysis, and the drafting of preliminary results. they were also asked to familiarize themselves with relevant research on the state of graduate education in the humanities (via a literature review assembled by the research assistant), and to complete training in ethnographic interview techniques.   the cornell team consisted of several staff members from across the social sciences and humanities library: the library’s director (the co-pi from cornell), two reference librarians, a staff member from access services, an administrative assistant, and a reference specialist/assessment analyst. in addition, two access services staff members and five students served as transcriptionists. two members of the cornell team had previous exposure to ethnographic research methodologies through an earlier project (foster et. al., 2011), and additional participants were recruited based on their subject expertise and experience with or interest in ethnographic research. prior to the launch of the study, team members researched the issues surrounding doctoral student success and attrition in humanities programs, collecting the research in a collaboratively maintained online bibliography.   at cornell, the reference specialist/assessment analyst served as the local project manager and primary liaison with columbia. as was the case at columbia, core team members were responsible for conducting interviews, data analysis, and drafting preliminary results. the core team met weekly or more as needed, depending on the evolving needs of the project. a project wiki was created at cornell to manage and distribute project documentation, and email was relied on heavily to communicate between meetings.   the cornell and columbia teams met jointly a total of five times over the course of the project. the initial face-to-face meeting at columbia included a one-day training workshop on ethnographic interviewing techniques. the four subsequent meetings were conducted via videoconference and occurred during data analysis and the drafting of preliminary results. the joint team meetings were planned by the project managers during numerous telephone calls and email exchanges that began a full three months in advance of the official launch of the project.   process   institutional review boards   before launching the study, both teams obtained approval from the irbs at their local institutions. the teams discovered divergent irb requirements and procedures between the two universities, probably due to the fact that the review process for cornell’s ithaca campus does not routinely interact with human subject research for medical/clinical trials while columbia’s does. luckily, the only significant impact of these differences was on the timing of data collection, as the study could not begin before approval was obtained at both institutions.   at cornell, the normal procedure is to request an exemption from the irb for library-related studies that pose no risk to human subjects and are usually considered “service improvement” activities. for this study, the normal procedure was initially followed but because of the open-ended nature of the instrument questions, an exemption was not granted.   because many members of the columbia team were new to human subjects research, it was decided that the entire team would complete the local irb training process, obtaining certification as researchers on the project. similarly to cornell, the normal procedure at columbia is to request an exemption for library-related studies. unlike at cornell, the columbia team received an exemption for the study protocol, most likely due to the fact that it was decided not to include former students in the study at columbia, thus reducing the necessary layers of review and documentation.   training   training in ethnographic research methods was supported by the grants and institutional funding that financed the project. the project teams from both universities received training jointly from anthropologist nancy fried foster, who had worked with members of the cornell team on a previous project (the members of the cornell team who had completed similar training earlier did not participate in the workshop). the training proved valuable not only for its content but because of the successful team building accomplished across the two local teams during the workshop. for the project managers in particular, this was an important opportunity to meet and make a face-to-face connection after months of planning and before a year of working together intensively at a considerable distance. this training provided the requisite skills for team members new to ethnographic research and laid a solid foundation for the teams to collaborate effectively during the subsequent phases of the project.    the training was based on the study goals, which had been developed jointly by the two teams. relying heavily on the protocol the teams drafted for individual interviews, the training covered techniques and best practices for conducting effective ethnographic interviews, as well as approaches for analyzing qualitative research data. live interviews with graduate students were incorporated into the workshop, which team members found both engaging and extremely helpful in their preparation.   instruments and written questionnaires   three instruments were developed for the study: a focus group protocol and preand postqualifying exams for the individual interviews. a written questionnaire was also created to collect additional demographic, funding, and other relevant information (see appendices of gessner et al. (2012) for examples of the interview protocols and questionnaire). the process of developing these instruments was an interesting collaborative process because the cornell and columbia teams had different applications for the data in mind, as well as differing sets of available data about their local graduate student populations. the columbia team was chiefly interested in gathering information about the research process for humanists within the local context, whereas significant research of this type had already been completed at cornell. the cornell team’s goals centered on finding points of convergence between graduate students’ needs and opportunities for the library to engage those needs. to accommodate the collaborative nature of the project, the interview protocol balanced the goals of the two institutions, which ultimately benefited both teams.   following a best practice in qualitative data gathering, the teams collected data from study participants using multiple approaches. a written post-interview questionnaire was used in addition to the interview protocol. a pre-interview questionnaire was initially considered, but the teams decided on using a post-interview questionnaire so as not to bias the interviews themselves. the questionnaires were administered on paper following each interview, ensuring a 100% completion rate by the participants. the questionnaires were developed from a combination of questions that each of the local teams had used previously in other assessments.  for example, the columbia team included a set of technology-usage and library satisfaction questions in order to provide context for each participant's responses. these questions were relevant to the aims of the current study, and as they had been used in other assessments could be used in comparisons between local user populations.   a subset of the two teams, led by the project managers, developed and edited the focus group instrument collaboratively over a period of three weeks. the teams agreed during the initial study design process that the focus groups would be used to gather preliminary data about the population being studied and to gather information to help refine the individual interview protocol. the collaborators shared documents via the project wiki and held regular conference calls to discuss how to best develop the instrument. this iterative process of development and revision proved rigorous and engaging for those involved.   a similar process was used in developing the interview protocol, where the same cross-institutional subset of team members worked to ensure that the protocol would cover research questions and gather data useful for both teams. the resulting protocol was reviewed by all members of both teams for their perspectives and feedback. the cornell team consulted with their irb to refine all instruments, which were subsequently pre-tested with students. as previously discussed, cornell had conducted earlier studies gathering information about the research processes of humanists within the local context, whereas columbia had not yet gathered this information from this particular population. through extensive discussion, a compromise was struck on the areas to be covered in the interviews, resulting in a rather comprehensive protocol covering research processes for humanities doctoral students, as well as other environmental and behavioural elements.   focus groups and interviewing   the initial plan was to conduct focus groups and individual interviews simultaneously at both institutions; but given staff schedules and other demands on team members’ time, this proved impossible. instead, the cornell team conducted focus groups a month ahead of columbia and shared initial results and suggestions for refining interview questions. similarly, individual interviews began at columbia a month ahead of cornell, with both teams completing interviews by a mutually agreed-upon deadline.   at both institutions, focus groups and individual interviews were conducted by team members in pairs, with one person facilitating the focus group/interview and another taking notes with a laptop and an audio recorder. these audio recordings were subsequently transcribed by cornell team members. the project managers kept both teams apprised of the focus group and interview schedules via the wiki, posting updated information as this phase of the project progressed.   at columbia, recruitment for the individual interviews was a collaborative effort between the graduate school of arts and sciences and the local project manager. administrators from the graduate school sent recruitment emails to doctoral students in target departments, alerting them to the opportunity to participate in the study. the columbia team also placed fliers requesting participation in high-traffic locations throughout the campus, which turned out to be an effective recruitment tool. at cornell, recruitment for the focus groups and interviews also relied on email invitations sent to students in target departments. recruitment was facilitated by close collaboration between the cornell pi, department chairs, and administrators from the graduate school, who encouraged students to participate. the cornell team also used invitational fliers posted throughout key buildings on campus, but this method did not prove to be as effective at cornell as at columbia.   transcription   undergraduate students at cornell transcribed the audio recordings of the focus groups and interviews using the start-stop universal system. the time initially budgeted for transcription was significantly underestimated, as was the number of students needed. ultimately, three additional students had to be hired, for a total of five. in addition, two cornell library staff members were diverted from other duties to complete the task. given the large number of transcriptionists and potentially uneven work product, the cornell team closely reviewed and revised the transcripts in pairs before coding began.    coding   again, a cross-institutional subset of the local teams, led by the project managers, worked collaboratively to develop the codebook and procedures for analysis of the approximately 900 pages of transcripts that resulted from 45 90-minute interviews. a grounded-theory approach was utilized to analyze the transcripts and develop the codebook (mansourian, 2006). four team members read each transcript independently, developing a preliminary code structure and definitions. team members then came together to share their work and debate the most appropriate, practical, coding structure, considering the original research questions posed for the study and local goals for applying findings. from this exercise, a codebook was developed, providing the agreed upon coding structure, definitions for each code, and examples of a statement describing a code for some complex cases. instructions were also developed, so that all team members would use a consistent approach for coding the transcripts.   although the teams considered a variety of software packages for coding, such as atlas.ti or nvivo, due to cost restrictions (project funding did not cover the purchase of software for all team members tasked with coding), the time necessary to train team members in these software packages, and computer hardware considerations (eight individuals on the columbia team were using five different computer operating systems), the team chose a coding approach using microsoft word, developed at the brown university library (neurohr, ackermann, o’mahony, & white, 2011).   to ensure inter-coder reliability, two-person teams coded each transcript. each member of these teams would read and code a transcript independently; then the two would come together to compare codes and collaboratively decide on a final coding. each coded transcript was compiled into a single microsoft word file, and the aggregate of these files was used to create a master index document. the master index allowed team members to discover, via the coding structure, quotes from any transcript with a specific code, conveniently compiled together.   analysis and writing   members from the cornell and columbia teams paired up for the analysis and writing phases of the project, despite some initial questions about working across organizations from a distance. this early anxiety gave way to productive working relationships, and team members enjoyed working with their colleagues from the partner institution. these pairs were assigned a set of themes, for which they would analyze the raw data using the master index produced in the coding phase of the project. each pair was responsible for drafting a section of the report, outlining findings and recommendations, which the larger group then reviewed, discussed, and edited.   tools   the columbia and cornell teams used a variety of tools to communicate, facilitate collaboration, and gather and analyze data over the course of the project. some were used only in a local context and others were supported for team members on both campuses by one of the partner institutions. tools important for the successful completion of the project included:   wiki   cornell provided a confluence (atlassian news) wiki to support the project. guest accounts were created for the columbia team, which enhanced overall communication and enabled all project documentation to be stored and shared in one location. the wiki served as both a document repository for both teams, aggregating irb protocols, meeting minutes, draft questionnaires, and other documents, and as the main communication vehicle for the project, providing project timelines, interview schedules, team member information, and status updates on different phases of the project.   telephone and email   the project managers communicated almost daily via email and held weekly meetings via telephone. conference calls for larger groups were used frequently throughout the project, especially when sub-teams needed to come together. sometimes it is the simple technologies that facilitate frequent and open communication, building the trust and understanding that enable a collaborative project to run effectively.   video conferencing   the cornell and columbia libraries had invested in video conferencing systems (polycom hdx 7000 series) to support the larger 2cul collaboration. the teams were able to utilize these systems during the analysis phase of the project, coming together to discuss the data as a full group. team members at both institutions were initially skeptical about the quality of interaction that would be possible via video conferencing but were pleasantly surprised by the experience. after a series of icebreakers facilitated by the project managers, the teams felt comfortable, and the meetings were productive and engaging.   microsoft word   unexpectedly, the teams used microsoft word to code the interview transcripts. while several team members had previous experience using software packages such as nvivo or atlast.ti, it was not possible to acquire one of these packages for all team members due to the financial, time, and technological constraints previously mentioned. instead, the team successfully used the indexing function in microsoft word to code the transcripts.   audio recording & playback   audio recorders (olympus ls10 linear pcm) were used to record focus group discussions and individual interviews by both teams. the audio quality produced by this equipment aligned with project needs, and thus optional external microphones were deemed unnecessary. the goal was to create crisp, high quality reproductions of every interaction, so the recorders were augmented with flash storage cards to support large file sizes (kingston 8gb micro sdhc flash cards). anticipating the need to review hundreds of hours of audio, samson sr850 professional studio reference headphones were purchased for both teams. to ensure technological compatibility, the cornell team purchased and distributed all equipment for the project.   data backup   audio recordings were burned to dvd, and data from columbia was sent to cornell for transcription. both teams purchased external hard drives to save all data gathered from the project, which was stored in accordance with local irb requirements.   video tutorial   the columbia team employed a video tutorial, created in camtasia, covering proper coding procedures. while this proved an effective training method at columbia, team members were not able to successfully share the tutorial with colleagues at cornell because of file size restrictions on the project wiki.   transcription software   a start-stop software system was utilized during transcription, enabling the transcriptionists to pause recordings with a foot pedal, freeing their hands for uninterrupted typing. this system substantially sped the transcription process.   google calendar   the columbia team used google calendar to schedule interviews, ensuring that both an interviewer and note taker were available for each interview. each team member had access to the project calendar and was able to accept or reject appointment invitations.   citation management software   the cornell team used a citation management application (refworks) to manage and share a bibliography and articles relevant to the project. a direct feed from refworks to the project wiki ensured up-to-date information available to both project teams in one location.   successes and challenges   any discussion of the relative success of conducting a collaboratively managed assessment of this scale must start with acknowledging the importance of clear, flexible, and constant communication, especially between the project managers. the ability of the project managers to effectively negotiate potential points of conflict between the teams’ goals and work styles was crucial. project managers were empowered by the co-pis to make daily operational decisions, which enabled an easy flow of communication and positively contributed to maintaining the project’s momentum. daily email exchanges and weekly phone calls kept the information flowing and both teams informed of the project’s progress.   the positive, supportive working relationship modeled by the project managers spread to and across the project teams as the project progressed. team members at both institutions were almost uniformly engaged and responsive. successful completion would have proven difficult if team members had not been fully committed to the project’s goals and flexible in how those goals were to be met. an important example of this operational flexibility was the extent to which the teams employed various technologies to work at a distance. collaborating via technology worked much better than expected, and team members from both institutions reported enjoying the experience.     while ultimately considered a worthwhile activity, the project required a substantial time commitment from team members from both institutions. this was time away from their routine job functions, so clear communication with supervisors about the time commitment on the part of the project managers and co-pis was critical. in fact, one team member was unable to meet the time commitment and was released from the project after a discussion with his supervisor. as the activities comprising ethnographic assessment represent a new type of work for many library staff members, the initial comfort level and skill sets of team members varied widely. it was important for project leaders to recruit team members with an active interest in and a proclivity for both qualitative assessment and working collaboratively.   as the project progressed, time management became increasingly important. the project managers performed well in terms of keeping local teams focused and on task. but as with most projects, more could have been accomplished with more time on task, especially during the data analysis and writing phases. project leaders and team members alike commented on the need for more time to analyze and discuss data before drafting results; and in retrospect, more time should have been allotted for those tasks, given the added complexities of collaborating across distance and organizational boundaries. as discussed earlier, the process of transcribing the massive corpus of interview transcripts took much longer than anticipated. looking back, project leaders would consider outsourcing this task to a professional transcription service rather than relying on student workers, whose work had to be augmented by support staff diverted from their normal duties.    project impact   the overall project was judged a clear success by the administrations from the libraries and graduate schools at both institutions. much was learned about humanities doctoral students and their research behaviours, and the results from the study were used on both campuses to improve services and launch new initiatives targeted at this user population. results were used at cornell to plan and implement a pilot immersion program for humanities graduate students and at columbia as impetus to relocate the graduate student teaching center within the library, among several other initiatives at both universities.    the immersion of a large number of library staff members in such a project, supported by high-quality training, and followed by visible outcomes based on the study’s results, has deepened interest in and enthusiasm for user assessment and data-driven decision making within the partner organizations. in this sense, the project was a positive, effective vehicle for staff and organizational development. in fact, following the completion of the project, library leaders and staff on both campuses actively discussed extending the study to other disciplines, possibly in the sciences or the social sciences. although this post-completion zeal has been somewhat tempered by the reality of how time consuming and staff intensive a project of this type can be, as of this writing, some members of the cornell team are in the early stages of planning another ethnographic study.   of greatest importance strategically, the execution of the project and resulting service improvements facilitated a deeper engagement not only with an important user group but also with local academic leadership, most notably department chairs and administrators within the graduate schools on both campuses. the conversations enabled by the planning and reporting phases of the project offered invaluable opportunities to position the library as an effective partner in addressing issues affecting students and faculty on both campuses and across the broader higher education sector. project leaders began this process answering questions from academic administrators and potential funders about why the library was concerned about the broader issues surrounding student success. at the end of the project, the libraries at cornell and columbia emerged with not only an improved understanding of an important constituent group, but were also better positioned as active, visible contributors to solving some of the difficult problems their parent institutions face in fulfilling their research and teaching missions.   references   abbott, a. (2008). the traditional future: a computational theory of library research. college & research libraries, 69(6), 524-545.   ehrenberg, r. g., & kuh, c. v. (eds.). (2009). doctoral education and the faculty of the future. ithaca, ny: cornell university press.   ehrenberg, r. g., zuckerman, h., groen, j. a., & brucker, s. m. (2010). educating scholars: doctoral education in the humanities. princeton. nj: princeton university press.   foster, n. f., clark, k., tancheva, k., & kilzer, r. (eds.). (2011). scholarly practice, participatory design and the extensible catalog. chicago, il: acrl publications.   foster, n. f., & gibbons, s. (eds.). (2007). studying students: the undergraduate research project at the university of rochester. chicago, il: acrl publications.   foster, n. f., & randall, r. (2007). designing the academic library catalog: a review of relevant literature and projects. university of rochester. retrieved from http://hdl.handle.net/1802/8409   gessner, g. c., jaggars, d.e, rutner, j., & tancheva, k. (2011). supporting humanities doctoral student success: a collaborative project between cornell university library and columbia university libraries. retrieved from http://www.clir.org/pubs/ruminations/02cornellcolumbia/report.html/report.pdf   hoffer, t. b., & welch, v. (2006). time to degree of u.s. research doctorate recipients. washington, dc: national science foundation. retrieved from http://www.nsf.gov/statistics/infbrief/nsf06312/nsf06312.pdf   hume, k. (2005). surviving your academic job hunt: advice for humanities phds. new york: palgrave macmillan.   mansourian, y. (2006). adoption of grounded theory in lis research. new library world, 107(9/10), 386 402.   national research council. (2010). assessment of research doctoral programs. washington, dc: national academies press. retrieved from http://sites.nationalacademies.org/pga/resdoc/index.htm   neurohr, k., ackermann, e., o’mahony, d., & white, l. (2011). coding practices for libqual+ comments: survey findings. in s. hiller, k. justh, m. kyrillidou, & j. self, (eds.), proceedings of the 2010 library assessment conference: building effective, sustainable, practical assessment (pp. 131-145). washington, dc: association of research libraries.   semenza, g. m. c. (2010). graduate study for the 21st century: how to build an academic career in the humanities. new york: palgrave macmillan.     appendix research team columbia cornell amanda bielskas, team member gabriela castro gessner, project manager yogesh chandrani, research assistant michelle hubbell, team member   jim crocamo, team member tanjina islam, transcription fadi dagher, team member antonio james , transcription victoria gross, research assistant tiwonge kayenda, transcription damon jaggars, co-pi rob kotaska, transcription alysse jordan, team member deb muscato, revision   jennifer rutner, project manager susette newberry, team member john tofanelli, team member dilara ozbek , transcription   jeremiah trinidad-christensen, team member deborah schmidle, team member sana siddiqui, transcription kornelia tancheva, co-pi   jill ulbricht, administrative support     microsoft word es_haigh.doc evidence based library and information practice 2007, 2:2  110 evidence based library and information practice       evidence summary    comparing the use of books with enhanced records versus those without  enhancements: methodology leads to questionable conclusions    a review of:  madarash‐hill, cherie and j.b. hill. “electronically enriched enhancements in catalog records:  a use study of books described on records with url enhancements versus those  without.” technical services quarterly 23.2 (2005): 19‐31.     reviewed by:   susan haigh   senior policy officer, library and archives canada  ottawa, ontario, canada  e‐mail: susan.haigh@lac‐bac.gc.ca      received: 23 february 2007  accepted: 12 april 2007      © 2007 haigh. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to compare the use of books  described by catalogue records that are  enhanced with url links to such  information as dust jackets, tables of  contents, sample text, and publishers’  descriptions, with the use of books  described by records that are not enhanced  with such links.      design – use study.    setting – academic library (southeastern  louisiana university, sims memorial  library).    subjects – 180 records with enhancements  and 180 records (different titles) without  enhancements.      methods – the study identified the sample  of unenhanced records by conducting  searches of the broad subject terms  “history”, “united states”, “education”,  and “social” and limiting the searches to  books. the enhanced sample was derived in  the same manner, but with additional search  limiters to identify only those records that  had url enhancements.  an equal sample  of enhanced and unenhanced records (50 or  30 of each) was tracked for each of four  search terms. only records for books that  could be checked out were included, as use  statistics were based on whether or not a  book was borrowed.  while half of the  enhanced records had full‐text elements  (such as descriptions) that were indexed and  thus searchable, the rate of use for these  records was not tracked separately from the  mailto:haigh@lac%e2%80%90bac.gc.ca http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:2  111 enhanced records that only had url  enhancements.    main results – books described on records  with url enhancements for publisher  descriptions, tables of contents, book  reviews, or sample text had higher use than  those without url enhancements. only 7%  of titles with urls, compared with 21% of  those without, had not been borrowed.   74.67% of titles with urls had been checked  out one or two times, compared with 69.5%  of those without urls. the number of titles  with enhanced records that had 3 or more  checkouts was almost double the rate of  unenhanced titles (18% to 9.5%).       conclusion – the authors conclude that  catalogue records that have electronic links  to book reviews, cover jackets, tables of  contents, or publisher descriptions can lead  to higher use of books, particularly if textual  enhancements such as descriptions are also  searchable.     commentary      the topic of this study is of considerable  interest to librarianship as the profession  reconsiders, reworks, and repositions the  library catalogue in the context of amazon  and google.     yet, while the findings of this study seem  intuitively valid, they should be accepted  with extreme caution. if all catalogue  records were enhanced, would the overall  volume of loans increase? the evidence is  not clear.     the authors acknowledge three limitations  of the study:     the sample sizes are small owing to  the relatively small numbers of  enhanced records, and the need to  keep the samples of enhanced  versus unenhanced records equal.   some of the works included in the  sample were on ‘hot topics’ and  therefore might have attracted high  use whether or not they were  enhanced. (the example given was  of a book, john kerry, with an  enhanced record. it is not clear from  the write‐up whether this  uncontrolled variable was thought  to apply equally to both enhanced  and unenhanced records).     in‐house use was not included in  the assessment of use.  (for some  reason, the authors seem to suggest  that this may have resulted in an  undercounting of only the  electronically enhanced sample.)     however, there may be other limitations, as  the control of variables is difficult with this  study methodology.     record enhancements may have  been disproportionately added to  the ‘best books’ records (whether  ‘best’ be defined as most recent,  most topical, best book jacket, most  authoritative author, best title, or  whatever).      it is possible that users choose  enhanced records over unenhanced  records. but given the methodology,  their choices of records are not  known – only their choices of books  to borrow are documented. the user  may have decided which titles to  borrow at the shelf based on  qualitative assessment of the books  in hand rather than deciding from  their records.     it is possible that the order of  records in the users’ search results  would influence their choice of  records to examine (a likely  precursor to borrowing the books).  evidence based library and information practice 2007, 2:2  112 this would be especially true for the  broad subject searches that were  part of the study methodology, as  the user would end up with large  search results. records appearing  on the first screen, likely the first 10‐ 20 titles, could be by their placement  the most likely to be borrowed. we  do not know whether enhanced  records were likely to predominate  on these screens, or not.     the same variable, the order of  search results, would likely be  affected if the tables of contents and  summaries in enhanced records  were full‐text indexed. exactly half  of the enhanced marc records (90  of 180) had textual enhancements in  the 505 (formatted table of  contents) or 520 (summary) fields,  which were full‐text indexed and  therefore searchable. this is  described in a curiously  inconclusive manner in one section  of the article. the authors do not  address the possibility that all, or a  disproportionate amount, of the  increase in use was on the books for  which their records had indexed  enhancements.     it is not clear whether from a results  screen the user would have been  able to identify enhanced records  and might therefore be attracted to  examine those first, possibly  influencing the likelihood that those  books would be borrowed.    a more valid methodology would have  been to study the usage based on a sample  of unenhanced records for a period of time,  then add enhancements to those records and  measure the usage on the same set of books  for a subsequent equal period of time.  (of  course, this may have not been operationally  feasible, and such a methodology would not  control the variable of the order of search  results and could introduce new variables  such as the diminishing likelihood of a book  being used over time.)      a different methodology could also have  tracked whether textual enhancements such  as tables of contents were being full‐text  indexed, and if so whether that affected  those records’ rankings in search results,  and as a direct consequence further  influenced the comparative rate at which  usage increased.     the statistical analysis of the findings also  falls short. the statistical significance of the  differences in loan rates between books with  enhanced versus unenhanced records  should have been measured and reported,  both for the samples presented in the tables  and for the entire records sample.   article   linking library to student retention: a statistical analysis   sidney eng chief librarian randolph memorial library borough of manhattan community college new york, ny, usa email: seng@bmcc.cuny.edu   derek stadler web services coordinator randolph memorial library borough of manhattan community college new ny, usa email: dstadler@bmcc.cuny.edu   received: 15 april 2015 accepted: 23 june 2015      2015 eng and stadler. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective this study analyses both library expenditure and student retention.  it seeks to determine if positive correlations found in a former study endure using more recent data or if alternative interpretations can be made.  it includes the associate degree-granting colleges and examines whether library instruction has a greater significance on student retention over expenditure and if library instruction at the two-year college correlates to retention.   methods the colleges and universities included in the study grant associate, bachelor, masters, and doctoral degrees, based on carnegie foundation classification.  data was analysed to determine if a correlation exists between the library and student persistence.  library statistics were drawn from the association of college and research libraries (acrl) metrics database which provides reports collected from academic institutions.  when aggregated, the acrl report yielded total library expenditures, total salaries of professional staff, the professional staff full-time equivalent (fte), fall semester student enrolment and data from a library instruction category of acrl surveys for associate degree-granting institutions.    results after replicating the same mathematical approach, the single category that has remained constant for all institutions is professional staff.  while the former study’s analysis suggested that a relationship between library expenditure and retention existed in every carnegie category, this study asserts that the same argument cannot be made for master’s degree-granting institutions.  the findings here indicate that total library and professional salary expenditure had a negative correlation.  also, while an analysis of instruction at the two-year school level cannot make the case that expenditure and staffing significantly influence retention, they can justify that instruction plays a factor in whether a student persists with their education.   conclusion the current research posits that there is no longer a relationship between library expenditure per se and student retention.  further research is needed to resolve the differences in the results of the study.  since there is a correlation between library instruction and retention at the two-year college, high-impact information literacy activities can form a bond between the student and the institution.  considering the low retention rates at the two-year school, a customised library instruction approach may be a solution to improving retention. introduction   recently there has been extensive discussion in the press regarding free tuition for community colleges.  part of the debate centered on the question of whether there is an enrolment or retention issue in higher education.  some have suggested that if we are concerned with educational policy or resource allocations, we should focus on the retention part of the equation.  considering that 61.1% of undergraduate students were retained in 2012 (u.s. department of education, national center for education statistics (2012) there are some merits to such a suggestion.  when so many students leave before finishing their studies it poses serious educational and financial problems to the individual and the institution.   what is the role of the library in student retention? for the last twenty years the library community has begun to empirically examine the potential connection of library service and student success.  early studies suggested that academic difficulty was the most significant cause of student withdrawal.  these research activities focused on correlating library use and retention (mezick, 2007).  more investigation quickly pointed to the fact that a student’s integration into the social and academic structure of the campus played a larger role than library use.  in either case, the variety and quality of library service was essential to student performance and persistence.  the question remained as how to identify what services contributed to whether a student returned the following semester and how to measure the potential contribution of such services.   in 2010 the association of college and research libraries commissioned the value of academic libraries: a comprehensive research review and report (val) to identify the value of libraries and establish a research agenda.  “student retention and graduation” is chief among them (p. 12).  in the report, acrl conveyed that libraries need to provide analytical evidence that students who engage in library instruction are more likely to graduate on time.  since 2010, several studies have examined the potential of library instruction.  correlations have been made between students’ participation in library classes and grade point average (gpa) (wong & cmor, 2011).  other studies have concluded that library technology instruction improves student retention.  indeed, the more technologically-prepared students are, the more likely they will persist (haddow & joseph, 2010).  the val report also pointed out the importance of the collegiate experience, which is evident in the attention placed on a student’s sense of belonging in recent literature.  often excluding questions directly related to libraries, experience studies aim attention at the entire student experience.  to facilitate this campus experience and give students a sense of belonging, librarians can create institutional environments that foster retention and eventual graduation.  focusing on instruction, librarians can affect a student’s decision whether to return to the campus the following fall (kuh, 2008).     student retention and graduation is important to higher education.  a returning customer is the raison d’être for all businesses.  in academia, the returning customer is the student whom the college wishes to retain for the complete duration of his or her academic career.  former retention studies have shown that both academic library expenditure and staff-to-student ratio contribute to student perseverance.  however, current research suggests that a student’s adjustment to an institution’s academic setting contributes to a greater commitment to the college and the goal of graduation.  while qualitative studies provide a context for meaning and interpretation, quantitative analysis may establish a potential correlation between library instruction and retention.    aims   this article re-examines library expenditure research methods and investigates library instruction class participation.  mezick (2007) analyzed both acrl and association of research libraries (arl) data at academic institutions, as well as retention information, and asserted that there was a positive correlation between library expenditure and student retention percentage.  she also advanced the notion that professional staff-to-student ratio was related to student retention.  reproducing mezick’s methodology with more recent data, our study analyzes both library expenditure and student retention to determine if the positive correlations found by mezick endure at baccalaureate, master, and doctoral degree-granting institutions or if alternative interpretations can be made.  secondly, using the same approach it will calculate if a relationship exists between these variables at associate degree-granting colleges.  this category was omitted in the previous study.  using two fields of supplementary data and bivariate analysis, the study will also determine whether library instruction had a greater impact on student retention over expenditure.  lastly, data will be analyzed to determine if library instruction at the two-year college correlates to retention.   literature review   studies of academic libraries and retention can be categorized as either single or cross-sectional.  early retention studies concluded that students who used the library generally performed better academically than those that did not and had a higher percentage of persistence.  one of the earliest studies was at california state polytechnic college, pomona.  kramer and kramer (1968) uncovered a connection between library circulation and retention.  it was determined that while 73.7% of freshmen students who borrowed books returned the following fall, only 57% who never checked out books returned (p. 310).  in another single institution study four years later, breivik (1977) discovered the retention potential in library instruction at brooklyn college.  of students who received weekly library instruction, 77% completed course work the following semester compared with 68.75 percent who did not, with a difference of 8.25% (p. 46).    in the past decade, retention studies and literature suggest that a student’s level of integration into the social and academic structure of campus life play a larger role than library use in the determination of persistence.  this approach follows the theory advocated by tinto (1993) who argued for the importance of social integration.  mezick’s study analyzed library expenditures, student enrolment, and professional staff data against student retention rates. academic library trends and statistics: 2003 (association of college and research libraries, 2003) provided raw library data and fall-to-fall retention percentage rates were obtained from the integrated postsecondary educational data system (ipeds).  since retention data for canadian post-secondary institutions were not provided by ipeds, canadian libraries were omitted, as well as institutions that did not report enrolment, expenditures, or retention rates.  in the end, the total study population was 586 or 47% of the population represented in arl/acrl publications (p.563).    the specific expenditure categories were total library expenditures, total library materials, monographs, serials, and professional salaries.  data was standardized on a per student basis to minimize the effect of institutional size but this step was not performed for professional library staff data.  correlations between expenditure per student and retention rates were determined by calculating pearson correlation coefficients (r) for each category of library expenditure within carnegie classification using ibm spss.  levels of significance were also ascertained using the rules of thumb for interpreting the bivariate correlation.  coefficients of determination (r²) were calculated to identify the percentage of variance in student retention rates that is explained by library expenditures.  a similar method was performed to investigate the relationship between the number of professional library staff and retentions rates (p.563-564).   mezick uncovered that positive relationships exist between each independent variable category of expenditure and the dependent variable of student retention within every carnegie category, with the strongest at baccalaureate colleges.  mezick also noted that personalized library service, particularly at doctoral granting institutions, may account for a relationship between library staff salary expenditures and student persistence.  while data suggested that students continue to demand increased library hours and quiet study space, it also hinted that a student has a greater chance to persist if more funds are allocated to library staffing.  a larger, experienced staff has more of a chance to interact with students and guide them in the academic setting (p.564-565).    emmons and wilkinson performed a cross-sectional study to investigate library instruction.  rather than bivariate, emmons and wilkinson (2011) utilized univariate statistics, developing a scatterplot in order to perform a regression analysis of each independent and control variable against each dependent variable.  controlling for socio-economic status (ses) and gender their conclusion was that the independent variables of staff-to-student ratio and students receiving instruction had an impact on student persistence.  the more library staff available per student provided for a greater opportunity at welcoming interactions.  therefore, students who were engaged were more likely to persist.          two other institutions examined ses more closely.  a curtin university study hypothesized that library data pointed to a relationship among library use, student engagement, and retention.  but more importantly, the authors wanted to link these variables with student age and ses.  derived from the library’s management system, the library use data set included number of loans, workstation logins, and other logins such as catalogue, database, and electronic reserve (haddow, 2013).  ultimately, there were higher than expected rates of library workstation logins by students from low ses backgrounds.  the contention was that low ses students may have less access to information technology in their homes and rely on campus resources, the library in particular (haddow & joseph, 2010, p. 240).   at california state university, monterey bay, reference librarians initiated an ongoing informal study and focused on non-research-related questions asked at library service desks.  it was discovered that 47% of questions did not directly relate to library research.  in fact, a majority was about the use of computer hardware and software since the college served primarily first-generation students who were at a low ses status and possibly the first in their families to go to college.  grallo, chalmers and baker (2012) hypothesized that the academic library could assist in student retention through the development of programs and services geared to help students become accustomed to academic life.   based on acrl’s recommendation, studies have examined correlations between library instruction and gpa.  one of the largest was at hong kong baptist university.  in the study, student library workshop attendance and graduation gpa were examined for over 8,000 students.  results suggested that if several workshops were offered, students had a higher gpa and were more likely to return the following semester (wong & cmor, 2011, p. 464).   another study at the university of minnesota – twin cities examined the association between a variety of library services and gpa.  based on student logins and those who participated in instruction sessions and reference interactions, results suggested that freshmen first-semester undergraduate students who used the library had a higher gpa in their second year and were more likely to return than non-users.  the mean average gpa for students who used the library was 3.18 compared with 2.98 for those who did not use the library (soria, fransen & nackerund, 2013, p. 151).   methods   culling information from the years 2010 and 2011, the current study employed methods similar to mezick but also extracted data from pre-baccalaureate institutions granting the associate degree.  raw numbers were drawn from the acrl metrics database which provides reports collected from academic institutions.  the colleges and universities included in the study based on carnegie classification grant associate, bachelor, masters, and doctoral degrees.  when aggregated, the acrl report yielded total library expenditures, total salaries of professional staff, the professional staff full-time equivalent (fte), and the fall semester student enrolment.   to provide an accurate comparison, some institutions were omitted or deleted.  for example, if data was erroneous, such as negative numbers for full-time professional staff, or not included at all, the college was removed.  another criterion for removal was if institutions reported some fields but not others.  the final list yielded full data in all fields for all colleges and universities.  for 2010 the number of schools was 1,179 and for the year 2011 it was 1,194 (see table 1).  similar to mezick’s methodology to minimize the effect of institutional size, expenditure per student was calculated using fall semester student enrolment.     table 1 number of institutions by carnegie classification   2010 2011 total 1,179 1,194 associates 316 339 bachelors 273 248 masters/professional 351 375 doctorate 239 232     data was also selected from a library instruction category of acrl surveys for associate degree-granting institutions to seek an argument for improved instruction at the two-year school.  the first set of instruction data analyzed was the number of instruction presentations to groups.  it may also be defined as the total number of sessions during the academic year in the category of bibliographic instruction programs and other scheduled class presentations, orientation sessions, and library tours.  beyond instruction, it may be for cultural, recreational, or educational purposes, outside of the physical library as long as it is library-sponsored.  if the library sponsors multi-session or semester credit courses, each individual session was counted as separate events.  however, meetings sponsored by other groups, using library space, were not included.  neither was training for library staff.  some of the counts are based on a full tabulation but sampling was also acceptable.  libraries are allowed to use numbers based on a typical week that may be extrapolated to a full year.  the other instruction data was previously used by emmons and wilkinson the number of participants in the instruction presentations.  it does not however include personal, one-on-one consultation.  for multi-session classes with a constant enrolment, each student was counted only once. similar to the previous instruction question, in addition to the data set, data also included if the number was based on sampling.   the decision to use both 2010 and 2011 was based on the latest entry in retention study by crawford (2015).  while the study agreed with previous findings, suggesting that library expenses per student had the highest correlation with graduation and retention rates, it also pointed out that doctoral institutions pay the most to provide library instruction.  the author noted that his study was limited by using only one year’s worth of data (p. 16).   while we attempted to replicate mezick’s analysis, which is not normally done in library science, we also introduced a slightly different strategy (see table 2).     table 2 comparison of retention studies mezick (2003) eng/stadler (2010/2011) analyzed total library expenditures, professional salaries, and staff  fte same method studied institutions that grant the bachelors, masters, and doctorate degrees same used carnegie classification   same calculated the correlation coefficient  (r) and the coefficient of determination (r²) same used the rules of thumb for interpreting bivariate correlation coefficients same analyzed data from 2003   data from 2010 and 2011 utilized arl, acrl, and ipeds data relied on the acrl metrics since all data is now available from this database analyzed total library expenditures as well as the four subcategories that comprise it used total library expenditures analyzed only institutions that grant the bachelors, masters, and doctorate degrees added the associate degree-granting institution did not analyze library instruction categories examined two instruction variables     analysis   for the study, data was analyzed to determine if a correlation exists between the library and student persistence.  the independent variables were library service and the dependent variable was retention.  using ibm spss, a pearson correlation coefficient (r) was calculated to determine any interrelation between each selected category and student retention, in degree, direction, and significance.  in replicating mezick’s methodology, a coefficient of determination (r²) was also computed to establish the percentage of variance in retention that is explained by library independent variables or more simply, identifies the impact that the independent variable may have on the dependent variable (hamilton, 1990, p. 355).   by definition, the value of r is a measure of the covariance of two variables divided by the product of their standard deviation.  analysis focuses on how two variables vary in relationship to each other.  calculation of the correlation coefficient returns a value between -1 and +1, with “0” indicating no relationship at all.  the closer to 1 or -1 represents a strong relationship (prion & haerling, 2014, p. 535).  similarly, the closer to zero the coefficient of determination is the less likely there is a relationship between variables.  the coefficient of determination is the square of the correlation coefficient.  mezick utilized this to estimate the percentage of variance of the dependent variable explained by its relationship independent variables (cheng, shalabh & garg, 2014, p. 137-138).   results   to replicate the mezick study, the value of r was calculated for the categories of total library expenditure, professional staff fte, and professional salaries for the years 2010 and 2011.  they are displayed in table 3.  results were analyzed using the rules of thumb for interpreting the bivariate correlation coefficient and the coefficient of determination.  while    social and physical scientists interpret values differently, this study, like mezick’s, made use of linear relationships as defined in hamilton’s (1990) modern data analysis (p. 481).  for the value r, the closer to zero there is no relationship between variables.  weak positive or negative relationships range from r=0.2 to r=0.49 while moderate are r=0.5 or greater.   in review, for the bachelor degree-granting institution the correlation coefficient has remained relatively constant for total library expenditure, professional salaries, and professional staff fte.  mezick’s calculations for the value of r are listed in table 4.  a comparison of all three years reveals that the numbers are similar.  total library expenditure and retention had a moderate positive relationship in 2003, 2010, and 2011, indicating that they are directly related.  while the value of r for professional salaries and professional staff fte is a weak relationship, it is however positive.  the same cannot be said about the master degree-granting institution.  in 2003, the total expenditure and professional salary coefficient revealed a weak positive relationship.  however, the value of r for the master degree-granting college was negative in 2010 and 2011.  in fact, for the year 2011 the coefficient reveals a weak negative relationship at r=-0.220.  the doctoral degree-granting institution calculations are an oddity.  for the categories of expenditure and salaries the value of r for 2010 was slightly negative while in 2011 it was positive.  actually, expenditure and retention had a moderate positive relationship in 2011 at r=+0.500.   on the other hand, the professional staff fte correlation is consistent between the years.  for example, in 2003, mezick calculated the coefficient to be +0.458, +0.231, and +0.536 for the bachelor, master, and doctoral degree-granting institutions, respectively, while in 2011, they were +0.432, +0.297, and +0.513.      table 3 values of r in retention for 2010 and 2011   bachelors masters/professional doctorate 2010 2011 2010 2011 2010 2011 total library expenditure + 0.531 + 0.592 0.002 0.220 0.033 + 0.500 professional salaries + 0.376 + 0.447 0.046 0.196 0.037 + 0.486 professional staff fte + 0.447 + 0.432 + 0.311 + 0.297 + 0.242 + 0.513   table 4 values of r in retention from mezick study (2003)   bachelors masters/professional doctorate total library expenditure + 0.505 + 0.318 + 0.476 professional salaries + 0.411 + 0.255 + 0.421 professional staff fte + 0.458 + 0.231 + 0.536   table 5 values of r² in retention for 2010 and 2011   bachelors masters/professional doctorate 2010 2011 2010 2011 2010 2011 total library expenditure 0.282 0.350 0 0.049 0.001 0.0250 professional salaries 0.141 0.200 0.002 0.038 0.001 0.236 professional staff fte 0.200 0.187 0.097 0.088 0.059 0.263     the value of r² was also calculated and the results appear in table 5.  using the same rules of thumb, a weak positive or negative relationship is 0.04 or greater, while moderate is 0.25 or higher.  the figures will be used to summarize the data in the next section.   in addition to replicating mezick’s analysis, this study looked at the associate degree-granting college.  the values of r and r² for the years 2010 and 2011 are in table 6.  for total library expenditure and professional salaries, the value of r indicates there is only slight or no relationship at all, either positive or negative.  in each year, the number was very near zero and reveals that no correlation exists between them or retention.  however, the calculation for professional staff fte was positive.  for example, in 2010 the value of r was +0.185 nearly indicating a weak relationship between the number of staff within the two-year college library and retention.   the supplementary acrl data examined shows the number of library instruction classes and participants.  the number of participants in instruction classes and retention are directly related.  looking at the first year, the value of r for number of participants was +0.207 illustrating that there is weak positive relationship between the independent and dependent variables.  in the second it was just shy of weak at +0.132.  it can be hypothesized that the more students enrolled in library instruction in a given year the greater the student retention percentage the following fall.  the value of r for the number of library instructions revealed a very slight correlation, although positive.  in 2010 it was +0.167, nearly weak, but in 2011 it was +0.091.   similar to the emmons and wilkinson study, a case can be made that library instruction positively impacts retention at the two-year college.  all correlation coefficients were positive numbers.  for both years the value of r was somewhat greater for number of participants.  while the number of instructional presentations was important, the number of participants was of greater significance to retention.     table 6 associate degree-granting institutions   value of r value of r² 2010 2011 2010 2011 total library expenditure 0.031 + 0.007 0.001 0 professional salaries 0.041 0.039 0.002 0.002 professional staff fte + 0.185 + 0.102 0.034 0.010 number of presentations + 0.167 + 0.091 0.028 0.008 number of participants + 0.207 + 0.132 0.043 0.017     discussion   mezick made the argument that for 2003 data analysis suggested that a relationship between library expenditure and retention existed in every carnegie classification category.  it was strongest for the baccalaureate college.  indeed, that argument, along with professional salary expenditure, can be made for both the years 2010 and 2011.  using the value of r², total library and professional salary expenditure in 2010 reveal 28% and 14% of the total variation in student retention, respectively.  in the year 2011 it was 35% for library expenditure and 20% for professional salary.    however, the same argument cannot be made for master’s degree-granting institutions.  the findings here indicate that total library and professional salary expenditure had a negative correlation for both years.  while not a significant negative correlation, the case can be made that neither category affected student persistence.  one possible explanation for the change in correlation is the growth in online learning.  by the fall of 2012, students taking at least one online class surpassed 7.1 million (holzweiss, joyner, fuller & young, 2014, p. 311).  in the same year, the united stated department of education (2012) estimated that almost 30% of students enrolled in distance learning were at the graduate level; while only 26% were at the undergraduate level. currently, the expenditure of in-house library resources is of less significance for a student to return the following semester.  therefore, the validity of mezick’s 2007 hypothesis is questionable in the present time.   the anomaly of the doctoral calculations makes any assumptions unjustified.  the significant difference between the years may be attributed to the fact that only 147 institutions reported data to acrl in 2010 compared with 231 in 2011.  are doctoral students more independent and self-sufficient in later years?  are more databases and tools contributing to this independency?      the single category that has remained constant for all institutions is professional staff fte.  for all years studied the correlation was positive but for the most part a weak relationship.  mezick noted that the strongest relationship between professional staff and retention was at the doctoral-granting institution.  the value of r² for 2003 was a moderate relationship at +0.287 or 29%.  the same applies to 2011 when 231 institutions reported to acrl.  the value of r² was +0.263 or 26%.  while not a comparable positive relationship for the years 2010 and 2011, the value of r² for professional staff at associate degree-granting institutions was 0.185 and 0.102 respectively.  indeed, there was only a 3% (2010) or 1% (2011) variance in student retention based on professional staff fte.  although the percentage is modest, it can be argued that the professional staff-to-student ratio is directly related to retention.   analysis of instruction at the two-year school raises intriguing conversation.  while the associate degree-granting colleges cannot make the case that expenditure and staffing significantly influence retention, they can justify that instruction plays a factor in whether a student persists.  when analyzing the year 2010 the number of library instruction classes given influenced retention by 2%.  furthermore, the more students enrolled in those instructions shaped the variance in retention by 4%.  though not as strong, the values of r² for 2011 were also positive.   can regression analysis be applied to the variables of library instruction and retention?   in their study emmons and wilkinson argued that there is a positive but weak relationship between instruction and retention at ninety-nine institutions. in the current study a case can be made that instruction at the two-year school plays a minor role in persistence. our analysis shows that the number of students enrolled is a stronger correlation than the number of classes given. such a hypothesis is important when looking at the two-year school where persistent rates are comparatively lower than at the baccalaureate institution. even though students are already less likely to return, they are more likely to persist if given library instruction. further studies will be needed to understand this phenomenon.      by controlling for the two-year college, this study echoes current library literature.  recent retention studies focus on single institutions, controlling for students with a low socio-economic status. in general, findings indicate that a higher proportion of retained students were logging into authenticated library resources more often. in the curtin university study, it was hypothesized that this was the result of the awareness of library resources through the instruction program (haddow, 2013, p. 130). there were also higher rates of logins from students from low ses backgrounds.  the two-year school is typically attended by students who may be from a low ses. the contention is that these students may have less access to information technology in their homes and rely on campus resources and the library in particular.  a city university of new york (cuny) library study revealed that while other low ses students may have access to information and communications technology, whether at home or in the library, they may not have skills to perform course-related research. often students shared home computer resources with other family members thus constraining access to academic technology. while plentiful resources are available on campus, students do not have the necessary instruction to research efficiently (smale & regalado, 2014).   in 2010 acrl called attention to the student experience and the sense of belonging to an institution.  library instruction serves as a valuable asset in two ways. through technology training, library instruction is an ancillary student experience assisting retention. also, attention to the first-year student needs can gear students to become accustomed to academic life and increase their sense of belonging to the institution. the notion opens up the possibility of expanding library service to those students who may need the technology help and thus increase retention. by focusing on the type of questions asked at reference desks and gearing instruction towards technology, the library can also adjust a student to academic life and further increase persistence.   conclusion   overall, available information suggests that retention is aided by a good support network and relationships with faculty, administrators, and yes librarians.  they provide direct research support and education.  the national survey of student engagement in 2014 indicated that while an overwhelming majority of instructors emphasized library skills only 37% of first-year students and 36% of seniors critically evaluated the quality of an information source (p. 14).  high-impact information literacy activities can support student success and promote retention by emphasizing the value of creditable information.  the library can serve as a bridge between social and academic engagement to produce learning outcome.   bell (2014) argues that when librarians become part of a student’s support network a student performs better academically.  the quality of the service is therefore vital to student persistence.  also, current retention studies pay particular focus on graduation.  in fact, fewer than half of students who entered college in 2007 finished school where they started.  bell offers the notion of an alt-higher ed, which is based on the new scenario wherein multiple and more affordable paths to graduation reduce the significance of single-institution retention.  his reasoning is that no single provider retains a monopoly on a student’s college education but rather what really counts is if the student graduates.  under the model, institutions that wish to retain students must create an “educational ecosystem” that matches students to the type and level of education that allows them to graduate (p. 12).  successful transfer and completion should be counted towards retention.    while related studies differ in sample group, there is one common theme.  it can be best summarized by a guide for both librarians and libraries: “there’s very strong evidence to suggest that students tend to be more engaged with learning…if they engage with library services, interact with library staff, and spend more time using libraries” (as cited in haddow & joseph, 2010, p. 234).  such students are more likely to persist.  hagel, horn, owen and currie (2012) provided five worthy recommendations for the library to assist in student retention.  one of these is a close working relationship between librarian and student, and the introduction of programs that help students commit to and engage with their library studies.  another is collaborative teamwork with other support services across the campus to provide students with integrated support (p. 221).  in looking to the future, bell (2014) suggests that academic librarians can emphasize the delivery of individualized research assistance and focus on building research skills.  they can demonstrate how the library can contribute to student retention by providing data that links student persistence and satisfaction to the library’s services, resources, and people (p. 14).  these guidelines advance the notion that the library is no longer simply bricks and mortar but rather a place where the constructive interaction of staff and student is a catalyst for retention.   contrary to former research, library expenditure is no longer directly related to student retention at all levels of academia.  regardless of how much is spent on materials and collection support, student persistence is not a guaranteed reflection of expenditures.  while online learning could be one of the reasons, higher education’s approach to the acclimation of a student to college life and society in general is of greater significance.  today, the college or university stresses the interaction of the faculty and the student body.  the trend is apparent considering that professional staff-to-student ratio is directly related and has remained constant in the past dozen years.  at the associate degree-granting college, instruction focused on technology training, or simply providing an academic location for computing, will form a bond between the student and the institution.  since the library may no longer be the “heart of the university,” it must conform to current learning paradigm and make itself marketable (association of college and research libraries, 2010, p. 11).  lastly, considering the low retention rates at the two-year school, a customized library instruction approach may be a solution to retaining students. with a focused program the library can focus on the basic needs of freshmen students who may be from a low ses.  by focusing on common non-library related questions asked at the reference desk, librarians can make the student more adapted to the campus and college life, and of course, compel them to return the following fall. it is high time for the academic library to align its mission with student success by reconsidering its functions and service.  at borough of manhattan community college we hope to drill deeper into the nexus between library instruction and retention by tracking a cohort of students who have received library instruction over their entire career in the college.  we may find certain activities to be conducing to student engagements.   references   association of college and research libraries (2003).  academic library trends and statistics: 2003.  http://www.ala.org/acrl/publications/trends    association of college and research libraries (2010). value of academic libraries: a comprehensive research review and report. chicago: association of college and research libraries. retrieved from http://www.ala.org/acrl/sites/ala.org.acrl/files/content/issues/value/val_report.pdf   bell, s. j. (2014). retention matters, but it's not the only thing that counts: as higher education shifts its focus away from retention toward graduation, librarians are seeking new ways to connect with students and ensure their success. information outlook, 18(1), 11-14.   breivik, p. s. (1977). resources: the fourth r. community college frontiers, 5(2), 46-50.   cheng, c.l., shalabh, & garg, g. (2014). coefficient of determination for multiple measurement. journal of multivariate analysis¸126, 137-152. http://dx.doi.org/10.1016/j.jmva.2014.01.006   crawford, g. a. (2015). the academic library and student retention and graduation: an exploratory study. portal: libraries and the academy, 15(1), 41-57. http://dx.doi.org/10.1353/pla.2015.0003   emmons, m., & wilkinson, f. c. (2011). the academic library impact on student persistence. college & research libraries, 72(2), 128-149. http://dx.doi.org/10.5860/crl-74r1    grallo, j.d., chalmers, m. & baker, p., (2012). how do i get a campus id? the other role of the academic library in student retention and success. reference librarian, 53(2), 182-193.                 http://dx.doi.org/10.1080/02763877.2011.618787   haddow, g., & joseph, j. (2010). loans, logins, and lasting the course: academic library use and student retention. australian academic and research libraries, 41(4), 233-244. http://dx.doi.org/10.1080/00048623.2010.10721478   haddow, g. (2013). academic library use and student retention: a quantitative analysis. library and information science research, 35(2), 127-136. http://dx.doi.org/doi:10.1016/j.lisr.2012.12.002   hagel, p., horn, a., owen, s., & currie, m. (2012). “how can we help?” the contribution of university libraries to student retention. australian academic & research libraries, 43(3), 214-230. http://dx.doi.org/10.1080/00048623.2012.10722278   hamilton, l. c. (1990). modern data analysis: a first course in applied statistics. pacific grove, ca: brooks/cole publishing company.   holzweiss, p. c., joyner, s. a., fuller, m. b., henderson, s., & young, r. (2014). online graduate students’ perceptions of best learning experiences. distance education, 35(3), 311-323. http://dx.doi.org/10.1080/01587919.2015.955262   kramer, l. a., & kramer, m. b. (1968). the college library and the drop-out. college & research libraries, 29(4), 310-312. http://dx.doi.org/10.5860/crl_29_04_310   kuh, g.d., & gonyea, r.m. (2003). the role of the academic library in promoting student engagement in learning. college and research library, 64(4), 256-282.                 http://dx.doi.org/doi:10.5860/crl.64.4.256   kuh, g. d. (2008). high-impact educational practices: what they are, who has access to them, and why they matter. washington, dc: association of american colleges and universities.   mezick, e. m. (2007). return on investment: libraries and student retention. journal of academic librarianship, 33(5), 561-566. http://dx.doi.org/ 10.1016/j.acalib.2007.05.002   national survey of student engagement. (2014). bringing the institution into focus—annual results 2014. bloomington, in: indiana university center for postsecondary research.   prion, s. & haerling, k. a. (2014). making sense of methods and measurement: spearman-rho ranked-order correlation coefficient. clinical simulation in nursing¸10 (10), 535-536.                 http://dx.doi.org/10.1016/j.ecns.2014.07.005   smale, m.a., & regalado, m. (2014). commuter students using technology. retrieved from http://www.educause.edu/ero/article/commuter-students-using-technology   soria, k. m., fransen, j., & nackerud, s. (2013). library use and undergraduate student outcomes: new evidence for students' retention and academic success. libraries and the academy, 13(2), 147-164. http://dx.doi.org/10.1353/pla.2013.0010   tinto, v. (1994). leaving college: rethinking the causes and cures of student attrition. 2nd ed. chicago, il: university of chicago press.   u.s. department of education, national center for education statistics (2012). table 326.30: retention of first-time degree-seeking undergraduates at degree-granting postsecondary institutions, by attendance status, level and control of institution, and percentage of applications accepted: 2006 to 2012. in u.s. department of education, national center for education statistics (ed.), digest of education statistics (2012 ed.). retrieved from http://nces.ed.gov/programs/digest/d13/tables/dt13_326.30.asp.   u.s. department of education, national center for education statistics. (2014). enrollment in distance education courses, by state: fall 2012. retrieved from http://nces.ed.gov/pubs2014/2014023.pdf   wong, s.h.r., & cmor, d. (2011). measuring association between library instruction and graduation gpa. college & research libraries, 72(5), 464-473. http://dx.doi.org/10.5860/crl-151   evidence based library and information practice using evidence in practice   evidence for development and enhancement of a popular reading collection in an academic library   timothy hackman head of resource sharing and access services university of maryland libraries college park, maryland, united states of america email: thackman@umd.edu   kelsey corlett-rivera research commons librarian, liaison to the school of languages, literatures and cultures university of maryland libraries college park, maryland, united states of america email: kcr1@umd.edu   elizabeth larson information services university of maryland libraries college park, maryland, united states of america email: elarson3@umd.edu   received: 14 aug. 2014  accepted: 27 nov. 2014      2014 hackman, corlett-rivera, and larson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   setting   the university of maryland is a major public research university located in college park, maryland, usa. it is the flagship institution of the university system of maryland and offers 127 undergraduate majors and 112 graduate degrees through programs in 12 colleges and schools. the university has a total enrollment of 36,102 (26,474 undergraduate and 9,628 graduate) and a tenured/tenure-track faculty of 1,464 (4,410 total faculty). the university of maryland libraries includes eight campus libraries, the largest and most central of which is mckeldin library, with 1.2 million volumes in the humanities, social sciences, life sciences, business, government documents, and east asia collection.   in 2011, the librarian for english and linguistics proposed the creation of a popular reading collection in mckeldin library, in response to frequent requests from users for non-academic reading material and audiobooks. because of the nature of these materials, the librarian decided to lease, rather than buy, them, and chose brodart’s mcnaughton plan as the best way to receive new titles that could be returned once they were no longer popular. under the plan, the library receives 30 books per month (up to 330 per year) and approximately 60 audiobooks per year. the librarian set up a selection profile for books which identified the genres that the library did and did not wish to receive, and selected audiobooks individually from the mcnaughton catalogue. the first monthly shipment of books and audiobooks arrived in december 2011 and were shelved in the busy learning commons on the library’s second floor. the collection was promoted heavily at first via social media, the libraries website, posters in mckeldin library, and a feature in the campus newspaper; ongoing promotion has been through inclusion in the libraries’ printed promotional materials and occasional website news items. popular reading collection materials can be identified through the libraries’ aleph catalogue, but not through worldcat local. students, faculty, and staff at the university of maryland can borrow items for three weeks at a time, plus one three-week renewal.   problem   the purpose of the popular reading collection is to provide a variety of current reading and audiobook materials that can be continually updated to reflect our users’ changing interests.   the problem we face is how to identify those interests and predict what will be popular with our users, so that we can assess whether we are receiving the “right” titles from the vendor. in the interest of efficiency, selection of new titles has been ceded to brodart staff, who, in theory, have a better understanding of popular publishing trends. however, they serve a variety of libraries and user communities across the country, so they cannot predict what will be popular among a heterogeneous group of students, faculty, and staff at one particular university. once items are received at the library, there is also the problem of weeding the collection appropriately to retain the items that are still popular and to keep it to a browse-able size. evidence   the primary evidence used in managing the popular reading collection is circulation data. since may 2012, we have exported reports from our aleph integrated library system (ils) on a quarterly basis, which show identifying information, format (book or audio cd), when the item was added to the collection, the number of times the item has circulated, and the date the item was last returned. this last piece of data was not originally included, but as one of our goals is to keep the collection fresh, we began including it in 2014 in order to identify items that have not circulated recently. as our mcnaughton selection profile is genre-based, it was also necessary to capture genre information for every title in our collection. this information is not tracked by our library system, so it is added manually by looking up titles in goodreads, a social media platform for sharing and receiving book recommendations that includes crowd-sourced genre information.   initially, this evidence was gathered to demonstrate the collection’s popularity to administrators and funders. in reviewing the data, it became apparent that circulation statistics would be the best way to identify items that were not popular (had not circulated) and therefore should be returned to allow for new items. popular reading collection items circulate at a much higher rate than our general collections, and the statistics show that they are providing a needed service to our users. all our data can be viewed online at http://hdl.handle.net/1903/15567.   secondary evidence used to manage the collection includes questions asked through our online reference system or at the library services desk, and requests for particular titles submitted via email. implementation   at first, using the circulation data for weeding decisions was rather straightforward: every quarter, we returned those items that had not been checked out. after nearly a year, however, there were fewer and fewer items with zero checkouts appearing on the reports. at that point, we began to rely on how long items had been on the shelf combined with number of circulations (e.g. items that have been on the shelf for at a least six months and have circulated fewer than five times). now we also consider the last time an item has been checked out.   in the summer of 2013, we decided to review returned books by genre to determine whether our selection profile was meeting users’ needs, and also by format to generate a profile for audiobooks (which had been selected individually up to that point). genre was manually added to the weeded book reports, and subsequent calculations based on circulation rates showed that we should decrease the percentage of mysteries/thrillers and westerns and increase romance and sci-fi/fantasy.   the circulation data is obtained from aleph reports that are provided in microsoft excel. we have utilized the built-in sorting and filtering features a great deal in determining which items to weed. when we began carrying out more in-depth calculations and assessing the entire collection as opposed to just the weeded books, we combined the data using open refine (formerly google refine), which is a free and open source tool that facilitates cleaning and organizing irregular data. in this case, the circulation reports had been generated over two years and so column headings and cell formats varied slightly. open refine also allowed us to easily combine the quarterly circulation reports into one large table with uniform data. we could then use that clean data to create an access database to facilitate the addition of genre information through a user-friendly form, and the generation of complex queries such as the percentage of items from each genre that had circulated more than five times.   outcome   location had the largest impact on circulation statistics, which increased by over 10% when the collection was moved to a prime spot by the entrance on the first floor of mckeldin library, even though advertising had ceased almost entirely. not only does every person entering and exiting the library see the collection, it is also next to an elevator, where users often browse while they wait. users looking for “the fiction section” get something much closer to what they were expecting.   adjustments to our profile to include more of the popular genres and fewer or none of the genres with lower circulation mean that 75.5% of items have circulated at least once. popular non-fiction has proved more popular than anticipated, while westerns have been dropped completely.   the popularity of the collection led to requests for a dvd lending collection and a graphic novel collection by the library’s student advisory committee. the evidence from the popular reading collection gave weight to these proposals. the collection has also helped us promote other, related collections. questions about the graphic novels in the collection are often a jumping off point to introducing users to the other graphic novels in our regular collections.   reflection   adding genre information to all 928 popular reading item records was time-consuming and, unfortunately, of limited use in the end. after we collected the genre information, we were disheartened to learn that brodart does not use genre designations in the way we had thought.   selections are made by our account representative from a list of titles that brodart believes will become popular based on past sales by the author, pre-release publicity, and other factors. that list does not include genre information; it is up to our representative to judge whether or not a book fits into a genre we want. (this may explain why we ended up with a number of christian romances in our collection, despite the fact that we had asked specifically to exclude christian fiction, after we increased the percentage of romances in our profile.) in practice, this means that the data we have collected on circulation by genre is of limited use; we can adjust the selection profile but have little control over what titles are actually sent based on that profile. working with brodart to improve the selection profile and process will be one important outcome of this assessment.   future topics for investigation include:  the effects of location changes or promotional efforts on circulation statistics; comparisons of popular reading and regular stacks items with similar call numbers; identifying an ideal size for the collection (e.g., do circulation statistics stop growing when the collection becomes too large to browse easily?); and circulation statistics for various users types (faculty, graduate students, undergraduates).   news/announcements   call for volunteers: peer reviewers      2015. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   the evidence based library and information practice journal would like to announce an opportunity to become a peer reviewer for the eblip journal.   the role involves:   providing in-depth peer review of original research articles, evidence summaries, classic evidence summaries, or review articles. the total number of peer review requests will vary depending upon content submitted to the journal, but reviewers, typically, are asked to review 2 to 4 submissions per year. contributing to the overall success of the journal by providing unbiased, fair, and timely reviews of submissions that are assigned. communicating with the editorial board about areas where the journal could improve. promoting eblip as an avenue of publication to colleagues.   the ideal candidate is a professional or researcher in the area of librarianship and information practice and well-versed in evidence based practice and research methods. interested persons should send a statement of interest, indicating areas of strength they would bring to the role, as well as a brief resume to dr. lorie kloda, editor-in-chief at eblipjournal@gmail.com.       peer reviewers for the journal follow detailed guidelines supplied by the editorial board. the double-blind peer review process is managed through the online journal system for notification, acceptance, and reviewers’ comments.   the deadline for individuals to indicate their interest is july 15, 2015.   about the journal: published quarterly by the university of alberta, this peer reviewed, open access journal is targeted at all library and information professionals interested in an evidence based model of practice. by facilitating access to library and information studies research via original research articles and evidence summaries of relevant research from the library literature, evidence based library and information practice enables information professionals to practice their profession in an evidence-based manner. please visit the evidence based library and information practice website (http://ejournals.library.ualberta.ca/index.php/eblip) for further information about the journal. news/announcements   cilip & library and information research group (lirg) research award    2014. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   the deadline for this award has been extended to 31 january 2015. the purpose of the award is to encourage research and innovation in library and information science. particular attention will be paid to proposals intended to improve the accessibility, retrieve-ability and usefulness of information. the maximum value of the award is £2500. the award is co-sponsored by health libraries group and ptfs europe. the award may be used to defray research expenses (e.g. travel, postage costs), to fund attendance at high level meetings or to fund a study tour. however, a clear overall aim for the research must be stated. visits for general professional development will not be considered.   for the evaluation criteria to be used and further information about the award, please see the lirg website at the link below. for a guide on writing research proposals, please see 'writing a research proposal' by juliet eve in our journal library and information research. applications should be sent to: dr alison brettle, lirg awards co-ordinator: a.brettle@salford.ac.uk further information: http://www.cilip.org.uk/library-and-information-research-group/awards/research-award news/announcements   call for contributions now open for eblip8      2014. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   the 8th international evidence based library and information practice conference (eblip8) committee is calling for submissions for both papers and posters.  the conference is being held in brisbane, australia from july 6-9 2015.   the theme of this conference evidence and practice: working together reflects our focus on narrowing the gap between practice and evidence and in promoting evidence based practice in all parts of the information profession.  evidence based library and information practice (eblip), once based on the medical paradigm, is changing and evolving. we are developing practices and processes particular to the library and information profession and reconfiguring the traditional evidence based philosophy to suit individual circumstances and locations. collaboration across institutions and sectors is fundamental to learning through sharing experiences, publication of quality research results, and a sustainable and connected eblip community.   submission of abstracts closes october 13, 2014. find out more at: http://eblip8.info/call-for-contributions/   many thanks,   clare thorpe on behalf of the eblip8 committee   guest editorial   current themes in academic library assessment: select papers from the 2010 library assessment conference   martha kyrillidou senior director, statistics and service quality programs association of research libraries washington, district of columbia, united states email: martha@arl.org   damon jaggars associate university librarian for collections and services columbia university new york, new york, united states email: djaggars@columbia.edu      2013 kyrillidou and jaggars. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     the 2010 library assessment conference was the third iteration of an ongoing partnership of three institutions, the association of research libraries (arl), the university of washington libraries, and the university of virginia library. the event was attended by more than 450 registrants and featured more than 68 peer reviewed papers (hiller, justh, kyrillidou, & self, 2011) and enticing keynotes (hiller, kyrillidou, & self, 2011). the library assessment conferences build upon arl’s rich history of describing research libraries typically with its statistics collection and more recently with the qualitative arl profiles (potter, cook, & kyrillidou, 2011). these forums advance the cause of assessment and data-driven decision making by engaging the academic library community in an active and elevated discourse about the strategic and policy issues that demonstrate the value of the library and its linkages to the academy.   arl’s historical trajectory includes decades of gathering the arl statistics and the arl annual salary survey, but the need to investigate new assessment measures began fermenting in the 1990s. in 1999, the chair of the arl statistics and assessment committee, carla stoffle, initiated a gathering of arl directors interested in defining “new measures” (blixrud, 2003), which became the foundation for the follow-up activities arl pursued, as well as the “living the future” conferences (bowlby, 2011) launched at the university of arizona. in fall 2000, arl sponsored the measuring service quality forum (heath & kyrillidou, 2001), where the latest thinking on assessment was brought together and immortalized in a special issue of library trends. at this gathering, innovative developments in library assessment were presented, notably the new survey protocol branded as libqual+®. in early 2002, the arl e-metrics gathering was launched in arizona (shim & mcclure, 2002) with follow up work that influenced the data collected on electronic resources and the will for arl to be one of the founding members of counter. in early 2001, arl and oclc co-sponsored a forum on academic library performance in the digital age, which led to the implementation of the balanced scorecard at the university of virginia library (self, 2003). later that year steve hiller and jim self presented a review of library surveys at the 4th northumbria international conference on performance measurement in libraries and information services, an event co-sponsored by arl (stein, kyrillidou, & davis, 2002). following this event, a collaboration between the university of virginia, the university of washington, and arl was initiated with the establishment of an onsite consultation service, initially known as “making library assessment work” and later recast as “effective, sustainable, and practical assessment” (hiller, kyrillidou, & self, 2008). in 2005 martha kyrillidou, steve hiller and jim self realized the growing library assessment community needed a venue for exchanging information and ideas, both formally and informally. thus was born the library assessment conference (see the conference website at www.libraryassessment.org). the success of the conference reflects the growth and the success of academic libraries – institutions that are navigating transformative changes and are positioning themselves to help users achieve both short-term research and learning goals and longer-term outcomes and life-long achievement.   the papers presented at the 2010 library assessment conference demonstrated that the areas where evidence based decision making is needed are multi-faceted and complex. the following selection of articles features several emerging facets of library assessment and highlights activities that are shaping and have shaped the arl agenda. starting with an article by dupont and yakel entitled “what's so special about special collections? or, assessing the value special collections bring to academic libraries,” one of the major challenges research libraries face in the 21st century, justifying and increasing the value of special collections activities in an era of diminishing resources, is discussed. the authors “recommend shifting from collection-centric to user-centric approaches and identifying appropriately precise metrics that can be consistently and widely applied to facilitate cross-institutional comparisons.” they suggest the potential benefits of “substituting the commonly used “reader-day” metric with a “reader-hour” metric and correlating it with item usage data in order to gauge the intensity of reading room use.” they also “discuss attempts to assess the impact of instructional outreach through measures of student confidence in pursuing research projects that involve primary sources.” this discussion is timely as the assessment of special collections is a key element of the arl agenda and efforts are underway to capture their value in more effective ways by developing a set of simple annual indicators.   the transformations taking place in academic libraries are deep and issues of organizational culture are explored in a set of three articles that focus on implementations of the climatequal protocol, a standardized staff survey that arl has developed in collaboration with the university of maryland that focuses on organizational culture and diversity assessment. phipps, franklin and sharma examine a recent implementation at the university of connecticut; defrank and hillyer describe the climatequal implementation at the university of nebraska-omaha; and mengel, smith, and uzelac discuss their experience at johns hopkins university. our society is changing rapidly and our professional ranks are facing pressures to reflect the increasing diversity of the larger society – issues of fairness, justice and psychological safety are some of the areas on which these assessment efforts focus. the overarching argument in this set of articles is that a healthy organizational climate ultimately links to healthy customer service.   library service is key. collections and access have merged in the minds of our users, and product and process when it comes to information discovery are seen as one and the same. in this environment it is imperative for libraries to have good data and to use it effectively for service improvement and effective marketing. the article by porat highlights some emerging organizational themes and activities at the university of haifa while discussing the relationship of marketing and assessment functions and demonstrating how this relationship has been strengthened. marketing and assessment are two sides of the same coin – one needs the other and yet the two do not face in the same direction. assessment “tries” to be objective while marketing (or advocacy) is shameless in being biased and partial to the message we want to promote. having a straight face on both sides of the coin is important!   a set of four articles focuses on the measurement of library service quality from the libqual+® perspective (harvey & lindstrom; fox & doshi; neurohr, ackerman, o’mahony, & white; and rutner & self). library as place, a key dimension of service quality measured by libqual+® is explored by harvey and lindstrom, and fox and doshi, while neurohr et al. focus on the analysis of the qualitative comments received through this survey protocol. rutner and self replicate an earlier study that demonstrated the insatiable appetite of faculty for journals, especially in a research library setting – no research library in the world will ever have enough journals for the voracious researchers at the top research institutions, or do we foresee a world where this situation might change? in the area of standardized protocols for assessing the merged environment where library and information technology have come together stands miso – measuring information services outcomes. an effort developed within the context of a group of liberal arts educational institutions where libraries and it services have merged, miso provides a way to assess these merged services from the perspectives of faculty and students. a group of five collaborators from sponsoring institutions analyze data collected by 38 colleges and smaller universities that participated in the miso survey between 2005 and 2010. the survey gathers input from faculty, staff, and students about the importance, use, and satisfaction with campus library and computing services.   there is an increasing interest in capturing the value that libraries provide to parent institutions, and a number of methods of assessing library value are being tested both the association of college and research libraries and arl have broad-ranging initiatives under development in capturing value. a set of three papers, jubb, rowlands and nicholas, king and tenopir, and jantti and cox present examples of efforts to shift the focus of library assessment in this direction. these articles, representing three different continents and countries (uk/europe, us/north america, and australia) reflect the global perspective and movement towards the perceived need to capture library value to the fulfillment of institutional missions. these articles focus on library outcomes in relation to research, faculty productivity, and student performance respectively. work in this area is emerging and most of it taking place at the research and development stage, though the example from australia stands as a strong application of theory put into practice, with a disciplined and collaborative process that demonstrates where the future of library assessment activities may be heading.   this work on measuring library value also demonstrates the importance of having a robust, stable, and well-architected technical infrastructure that enables a better understanding of user behavior in an era when users increasingly enter the “library” through its website. the work of joe zucca at the university of pennsylvania stands as a fine example of capturing useful data through sound data architecture and demonstrates how linkages between resource allocation and user behavior can be drawn. in his article, “business intelligence infrastructure for academic libraries,” zucca poses a strategic challenge to arl to be “an effective broker, providing a space for potential partners to begin addressing the challenge of creating and governing a critical new infrastructure for managing library services.” this is a world where libraries can share assessment data more easily to understand their users and serve them and their institutional missions more effectively.   lastly, lewis, mengel, hiller, and tolson describe the experience of four arl libraries in building library scorecards using the balanced scorecard framework. the article covers “an introduction to the balanced scorecard and its key components; an overview of the arl initiative and the process used to develop scorecards at each library; an exploration of the concept of a standardized suite of measures for arl libraries based on a commonality of key objectives; and a review of organizational challenges faced by the sites during their implementations.” the authors emphasize the importance of communication and organizational development activities and conclude that “the balanced scorecard forces an organization to have new, sometimes challenging, conversations and to analyze aspects of its current and future state that may have otherwise gone unexamined. ultimately, the scorecard may substantially shift an organization’s strategic direction or dramatically change how its human capital and other resources are allocated. the scorecard is, by its very nature, a change driver.”   leading change is a theme articulated in all of the articles in this collection. the authors demonstrate the continued commitment of libraries and their staffs to push towards more “effective, sustainable and practical assessment,” the enduring subtitle and underlying purpose of the library assessment conferences.   acknowledgement   preliminary versions of the papers published in this feature section were originally published in the proceedings of the 2010 library assessment conference. see: http://libraryassessment.org/bm~doc/proceedings-lac-2010.pdf     references   blixrud, j. (2003). mainstreaming new measures. arl: a bimonthly report on research libraries issues and actions from arl, cni and sparc, (230/231), 1-7. retrieved 18 may 2013 from http://www.arl.org/storage/documents/publications/arl-br-230-231.pdf   bowlby, r. (2011). living the future: organizational performance assessment. journal of library administration, 51(7-8), 618-644. doi:10.1080/01930826.2011.601267   heath, f., & kyrillidou, m. (2001). introduction. library trends, 49(4): 541-547.   hiller, s., justh, k., kyrillidou, m., &self, j. (eds.) (2011). proceedings of the 2010 library assessment conference. washington, dc: association of research libraries. retrieved 18 may 2013 from http://libraryassessment.org/bm~doc/proceedings-lac-2010.pdf   hiller, s., kyrillidou, m., & self, j. (2008). when the evidence isn’t enough: organizational factors that influence effective and successful library assessment. performance measurement and metrics, 9(3), 223-330. doi:10.1108/14678040810928444   hiller, s., kyrillidou, m., & self, j. (eds.) (2011a). library quarterly, 81(1), 3-128. retrieved 18 may 2013 from http://www.jstor.org/stable/10.1086/657499   potter, w. g., cook, c., & kyrillidou, m. (2011). arl profiles: research libraries 2010. washington, dc: association of research libraries. retrieved 18 may 2013 from http://www.libqual.org/documents/admin/arl_profiles_report_2010.pdf   self, j. “using data to make choices: the balanced scorecard at the university of virginia library." arl: a bimonthly report on research libraries issues and actions from arl, cni and sparc, (230/231): 28-29. retrieved 18 may 2013 from http://www.arl.org/storage/documents/publications/arl-br-230-231.pdf   shim, w., & mcclure, c. r. (2002) improving database vendors’ usage statistics reporting through collaboration between libraries and vendors. college & research libraries, 63(6): 499-514. retrieved 18 may 2013 from http://crl.acrl.org/content/63/6/499.full.pdf   stein, j., kyrillidou, m., & davis, d. (eds.). (2002). proceedings of the 4th northumbria international conference on performance measurement in libraries and information services, pittsburgh, pennsylvania, august 12-16, 2001. washington, dc: association of research libraries. retrieved 18 may 2013 from http://www.libqual.org/documents/admin/4np_secure.pdf   evidence summary   additional search strategies may not be necessary for a rapid systematic review   a review of: westphal, a., kriston, l., hölzel, l.p., härter, m., & von wolff, a. (2014). efficiency and contribution of strategies for finding randomized controlled trials: a case study from a systematic review on therapeutic interventions of chronic depression. journal of public health research, 3(2), 177. doi: 10.4081/jphr.2014.177   reviewed by: joanne l. jordan research information manager arthritis research uk primary care centre, keele university keele, staffordshire, united kingdom email: j.jordan@keele.ac.uk   received: 10 mar. 2015  accepted: 12 may 2015      2015 jordan. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to evaluate the efficiency and contribution of additional searching strategies for finding randomized controlled trials (rcts) in a systematic review.   design – a methodological case study.   setting – biomedical literature.   methods – a sensitive search (defined as “the ratio of the number of relevant reports identified to the total number of relevant reports in existence”) was conducted of electronic databases, cochrane central database, medline, embase, psycinfo, cinahl, biosis, and web of science databases (science and social science citation indexes). the following additional searching strategies were conducted: hand-searching contents of relevant journals (archives of general psychiatry, journal of consulting and clinical psychology, and journal of affective disorders), citation tracking (forwards tracking using social science and science citation index and backwards tracking by looking through reference lists of included studies), screening reference lists of relevant systematic reviews, searching clinical trials registers (clinicaltrials.gov and ictrp registers), and contacting first authors of included studies to find any similar unpublished studies.   the number of articles identified by each of these methods was recorded and screened for inclusion in the systematic review. the authors calculated what they labelled as the ‘efficiency’ of each searching strategy (the number of included studies identified by the search method as a proportion of the full text articles screened) and the ‘contribution’ of the search strategies (the ratio of included studies identified by that method to the final number of included studies in the systematic review). the methodological quality of each included study was assessed using the cochrane risk of bias tool, which is a critical appraisal checklist used to judge the study’s value in the systematic review. the meta-analysis in the systematic review was conducted with and without the studies identified by the additional searching strategies to assess their impact on the review’s findings.   main results – in total 50 studies were identified, 42 from electronic database searches and 8 from additional search strategies. as illustrated by the results in table 1, the most useful additional search strategy was screening reference lists of relevant systematic reviews. journal hand-searching and contacting authors also contributed to the review.   of the eight studies identified by the additional search strategies none were judged to have a low risk of bias (four had high risk of bias and four were unclear). of the 42 included studies from electronic searches only 11 were judged to have a low risk of bias, whereas 9 studies had a high risk of bias and 22 were unclear.   excluding the eight studies retrieved from additional search strategies in the systematic review meta-analysis did not influence the results on the effectiveness of the different interventions for chronic depression. these studies were found to be indexed correctly on the electronic databases, but were not identified in the initial search.   conclusion – additional search strategies, especially screening reference lists of systematic reviews and hand-searching relevant journals, retrieved a substantial number of relevant studies for a systematic review of interventions for treating chronic depression. however, results of the review’s meta-analysis did not differ when these additional studies (rated as either high or unclear risk of bias) were not included and search methods were time consuming. it might be reasonable to rely on electronic searching strategies when resources for conducting a systematic review are limited or when doing a “rapid review.” the benefits and limitations of additional search strategies should be considered particularly when resources or time for conducting a systematic review are limited.     table 1 the numbers of articles retrieved and included studies from each of the searching strategies. search strategy search results full texts screened included in review efficiency contribution electronic database search 2417 276 42 9.8% 84% journal hand-search 19076 33 2 6.1% 4% forward citation tracking 2979 7 0 0% 0% backward citation tracking 1692 15 0 0% 0% review reference screening 1191 16 5 31.3% 10% clinical trial registers 2053 0 0 0% 0% contacting authors 16 11 1 9.1% 2%     if the electronic database search is sensitive and includes the cochrane central database additional search strategies may not be necessary, but these findings should be tested in other research areas.   commentary   this study is highly relevant for information professionals who often conduct search strategies for systematic reviews. systematic reviews frequently take 8 to 12 months to complete and studies evaluating rapid review methods that reduce production time without compromising rigour are needed.   the cochrane central database (the cochrane library) consists of rcts retrieved from a variety of sources, including hand-searching journals and searches of trial registries. as the authors of the study rightly point out, by including central in their electronic database searches, additional strategies are likely to be less influential. in this study, journals with high impact factors and those containing well-known potentially relevant studies were hand searched. the authors did not mention if they checked the list of journals already hand-searched for the central database, which could have avoided the duplication of effort required in screening several thousand article titles and abstracts.   however, cochrane review standards (cochrane editorial unit, 2013) state it is mandatory for cochrane reviews (the gold standard systematic reviews), in addition to searching central, to search clincaltrials.gov, who international clinical trials registry platform (ictrp) trial registers, and screen reference lists of included studies and relevant systematic reviews. these standards ensure that searches for systematic reviews are extensive and sensitive. the authors found that studies indexed in the electronic databases were missed by their search strategies. by using a variety of methods systematic reviewers can be more confident about identifying all the relevant studies.   studies found in this review by additional search strategies did not influence the results of the meta-analysis. however, as pointed out by the authors, where results of studies are sparse or conflicting, missing even the smaller studies will have a large impact and bias the results of the systematic review. this study needs to be repeated in a broad range of reviews to fully assess the impact of additional searching strategies. including unpublished studies (grey literature), which were excluded in this study due to limited searching, also have the potential to affect systematic review results. searching for grey literature requires different resources and their value needs to be assessed.   the authors of this study did not count studies that had already been included by a previous strategy. they comment that the order in which searching strategies were applied will have affected the level of contribution. therefore the contribution of the searching strategies may be underestimated and the impact of this needs further exploration.   a criticism of the study is that the electronic search strategies are not sensitive enough and may be the reason eight studies were missed. the strategies do not include indexed terms (e.g., mesh terms) that would help to identify relevant studies. the authors state they did not consult an expert librarian or information specialist when constructing their search strategies, which might have improved their search retrieval.   reference   cochrane editorial unit. (2013) methodological expectations of cochrane intervention reviews (mecir): methodological standards for the conduct of new cochrane intervention reviews. version 2.3. available from http://editorial-unit.cochrane.org/mecir   evidence summary   library research courses that follow universal design principles and best practices for online education of special needs students improve student learning experiences   a review of: catalano, a. (2014). improving distance education for students with special needs: a qualitative study of students’ experiences with an online library research course. journal of library & information services in distance learning, 8(1-2): 17-31. doi: 10.1080/1533290x.2014.902416   reviewed by: dominique daniel humanities librarian for history and modern languages oakland university rochester, michigan, united states of america email: daniel@oakland.edu   received: 26 nov. 2014 accepted: 26 jan. 2015      2015 daniel. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to evaluate student experience with an online library research course that follows best practices about distance education for special needs students.   design – questionnaire and semi-structured interviews.   setting – a large private college in the united states of america.   subjects – seven female students, both undergraduate and graduate, each with different physical and cognitive disabilities.   methods – students were recruited from respondents to a survey about accessible library services, with a $50 gift card incentive. they took an online information literacy course that had been adapted for students with special needs, using universal design for learning and best practices in distance education for special needs students and in library instruction. upon completion, students answered a questionnaire about the course learning activities. students were then asked to participate in in-depth, semi-structured interviews on their learning preferences and study skills.   main results – students expressed overall satisfaction with the course, especially the clear organization and the ability to choose from various types of assignments for their final project. they expressed a preference for click-through, step-by-step instructions for tutorials. five of the seven students participated in in-depth interviews, which revealed some common themes in their overall online learning experience: the challenge of obtaining extended time on tests; overcoming reluctance to participate in online discussions; the need for regular communication with instructors; and the need for clearly stated expectations and timely feedback.   conclusion – student feedback confirms best practices identified in the literature on distance learning and on special needs students. the need for clear instructor expectations, clear course organization, and frequent interaction with the professor are common to all distance learning situations, but students with special needs are particularly in need of such well-structured instruction. librarians should always determine accessibility before selecting software and tools to be used in online instruction. accessible online library instruction should include information about resources for students with special needs; it should provide the same content in varied formats; and it should offer students options for assignment formats. much research remains to be done to compare students with special needs in online and face-to-face courses, and to determine factors that improve the success of students with special needs in online courses.   commentary   this article provides a thorough review of the literature on library instruction to students with special needs, as well as a useful summary of best practices for online teaching to students with special needs and of the principles of universal design for learning (udl). but the main contribution of the article is the empirical study of special needs students’ perceptions of an online library research course, revised according to these principles and best practices. the study opens the way to new research on the effectiveness of online library instruction regarding accessibility.   however, the authors provide little information about the methods used for the study. the questionnaire is not included and the reader does not know who administered it and how, nor the types of questions used. regarding the follow-up interviews, no details are given about the interviewers, whether the questions were pre-tested, or how the answers were analyzed. there were only seven respondents, each with a different disability, which does not allow for generalization and limits the results obtained from the interviews. furthermore, as the author recognizes, the respondents were self-selected and had external motivation. this does not allow for replication of the study (koufogiannakis, booth, & brettle, 2006).   the author points out that there is a need for both structured and diverse online instruction – instruction that provides guidance through learning steps, but also offers a range of optional formats for learning objects and assignments. the findings are consistent with the udl approach, and no reader can disagree with the author’s conclusion that consideration of learner differences should drive all instruction, not just that addressing students with disabilities.   regrettably, the study is confined to student perceptions of online instruction and does not attempt to measure how well the students actually did in the class. a pre-test was done (p. 23), but there is no mention of a comparison with final student achievements. the reader can only conclude that when the author affirms that universal design principles can improve distance education, she is referring to student satisfaction rather than actual performance.   finally, one interesting contribution of this article is its suggestion that online library instruction is unique because it requires the use of complex information systems that may be especially challenging for students with disabilities. yet the article does not dwell on the aspects of the redesigned course that included instruction on such information systems. at the same time, the article also emphasizes that many of the needs of students with disabilities are not specific to library instruction but apply to all online courses. in fact it is striking that the interviewees’ comments are often similar to those of students without a disability. future exploration of what makes online library instruction unique or similar to online courses in other fields would be welcome.   references   koufogiannakis, d., booth, a., & brettle, a. (2006). reliant: reader’s guide to the literature on interventions addressing the need of education and training. library and information research, 30(94), 44-51. retrieved from http://www.lirgjournal.org.uk/lir/ojs/index.php/lir/article/view/271 conference paper   still bound for disappointment? another look at faculty and library journal collections   jennifer rutner assessment and planning librarian columbia university libraries new york city, new york, united states of america email: jenrutner@gmail.com   james self director, management information services university of virginia library charlottesville, virginia, united states of america email: self@virginia.edu      2013 rutner and self. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to examine why faculty members at columbia university are dissatisfied with the library’s journal collections and to follow up on a previous study that found negative perceptions of journal collections among faculty at association of research libraries (arl) member institutions in general.   methods – in 2006, jim self of the university of virginia published the results of an analysis of libqual+®  survey data for arl member libraries, focusing on faculty perceptions of journal collections as measured by libqual+®  item ic-8: “print and/or electronic journal collections i require for my work.” the current analysis includes data from 21 arl libraries participating in the libqual+® survey from 2006 through 2009. notebooks for each library were accessed and reviewed for the information control and overall satisfaction scores. at columbia, the results were used to identify departments with negative adequacy gaps for the ic-8 item. follow-up phone interviews were conducted with 24 faculty members in these departments, focusing on their minimum expectation for journal collections, their desired expectations, and preferences for print or electronic journals. results – analysis of the 2009 libqual+® scores shows that faculty across arl libraries remain dissatisfied with journal collections. none of the libraries achieved a positive adequacy gap, in which the perceived level of service exceeded minimum expectations. there was no significant change in the adequacy gap for the ic-8 item since 2006, and satisfaction relative to expectations remained consistent, showing neither improvement nor decline. while most of the faculty members interviewed at columbia stated that the journal collections met their minimum expectations, 15 of 24 reported that the library did not meet their desired level of service in this area. key issues identified in the interviews included insufficient support from library staff and systems regarding journal acquisition and use, the need for work-arounds for accessing needed journals, problems with search and online access, collection gaps, insufficient backfile coverage, and the desire for a discipline-specific “quick list” to provide access to important journals.   conclusion – the issue of satisfaction with journal collections is complex, and faculty members have little tolerance for faulty systems. the evolution of the electronic journal collections and the inherent access challenges will continue to play a critical role in faculty satisfaction as libraries strive to provide ever-better service.   introduction   in 2006, jim self from the university of virginia (u.va.) published the results of an analysis of libqual+® data for association of research libraries (arl) members, focusing on faculty perceptions of journal collections. the libqual+® item in question was ic-8: “print and/or electronic journal collections i require for my work.” findings included the observation of negative adequacy gaps for this item across arl institutions from 2006, regardless of the level of expenditures on journals (self, 2008). adequacy gaps, when dealing with libqual+®, measure the difference between the minimum score and the perceived score of a given survey item. a negative adequacy gap, such as -0.5, tells us that perceived service ratings were lower than minimum desired service ratings, i.e., that respondents are dissatisfied with the level of service. a correlation of 0.84 was determined for the journal collection item and the overall satisfaction item, confirming the importance of journal collections on faculty’s overall satisfaction with library services. the study also reviewed ic-8 scores for faculty at arl institutions from 2004, demonstrating a consistently negative adequacy gap. follow-up phone interviews with faculty at u.va. shed some light on the complex topic. issues of access – both physical and electronic, missing backfiles, and coverage of foreign titles were disclosed by faculty.   since the study in 2006, u.va. has worked to improve search interfaces, most notably by introducing a new version of the online catalog in july 2010. there has also been an ongoing effort to inform and instruct teaching faculty. individual libraries have made improvements in their journal holdings and facilities. the fine arts library transferred monographic funds to serials and devoted more physical space to journal use. the music library conducted a comprehensive review of all subscriptions, analyzing use and accessibility, and identifying gaps in holdings. the study has educated library staff at u.va. as a whole; there is a deeper recognition of the profound importance of journals to faculty.   columbia university libraries (cul), which was included in self’s original 2006 analysis, participated in libqual+® on a three-year basis from 2003, making 2009 the third administration of the survey at columbia. u.va., however, has not participated in libqual+® since 2006. response to the survey at columbia increased dramatically in 2009, with more than 3,800 completed surveys (a vast improvement over the response level in 2006: around 250 surveys completed).   this study follows up on self’s initial inquiry, “given the substantial investment in journals at arl libraries, why are faculty at these institutions consistently dissatisfied with their library’s journal collections?” in 2009, the collections budget at columbia saw electronic resources outpace print for the first time. more than 50% of the collections budget now funds electronic resources, including e-journals. before 2010, the collections budget at columbia continued to grow at a healthy pace, accounting for inflation and then some. why, then, do faculty continue to report dissatisfaction with journal collections at columbia? this study addresses the following questions:   are faculty at institutions with arl libraries more or less satisfied with journal collections in 2009 than in 2006? why are faculty at columbia continually dissatisfied with journal collections, as observed from libqual+® scores? does ic-8 continue to be the area of greatest dissatisfaction for faculty at institutions with arl libraries, according to libqual+® data?   the libqual+® survey   the libqual+® survey was developed by the association of research libraries and the texas a&m university libraries. the survey is administered online and collects demographic, library use, overall satisfaction, and perception feedback from library users. libqual+®’s central measures are the 22 core questions that approach library services from 3 perspectives: affect of service (as), information control (ic), and library as place (lp). respondents are asked to rate each of the 22 items on a scale of 1-9 in 3 ways: their minimum level of service, their desired level of service, and their perceived level of service. these scores together provide a rich view of user perceptions of library services. one of the key benefits to this rating scale is the analysis of the adequacy gap (i.e., the difference between the minimum ratings and the perceived ratings). this adequacy gap allows libraries to gauge whether or not they are meeting their users’ expectations in each of the 22 areas of library service. an open-ended comment box, in which respondents are invited to share any additional feedback with the library, follows the 22 core items. these free-text comments provide context to the 22 survey items.   methodology   the methodology for this study was based directly on that used at u.va. in 2006. analysis includes data from arl libraries participating in the libqual+® surveys from 2006 through 2009. notebooks for each arl library were accessed and reviewed for the information control and overall satisfaction scores (cook, 2009). unlike the 2006 u.va. study, only faculty scores were analyzed in this study; graduate students were not included. arl institutions with fewer than 50 faculty survey participants were excluded from the analysis.   at columbia, the 2009 libqual+® results were used to identify departments with negative adequacy gaps for the ic-8 item. these departments were targeted with follow-up phone interviews using the identical interview protocol developed at u.va. in 2006. interview participants were asked about their minimum expectation for journal collections, their desired expectations for journal collections, and preferences for print or electronic journals. a series of themes were identified as significant. this paper includes a preliminary analysis based on the grounded theory methodology (strauss, 1987). in total, 24 faculty members were interviewed over the phone.   results at columbia   2009 faculty scores for ic-8 were no surprise at columbia. as seen in figure 1, the perceived score is well below the minimum, with an adequacy gap of -0.34. figure 1 displays the faculty scores for the 22 libqual+® items, including affect of service (as), library as place (lp), and information control (ic). the top of each bar illustrates the mean desired score, the bottom of the bar illustrates the mean minimum score, and the black dot illustrates the mean perceived score for each survey item. while the affect of service items show a relatively comfortable adequacy gap (other than as-9, which has consistently garnered low scores at columbia), nearly all of the ic items show perceived scores falling below the minimum. library as place items show a level of satisfaction, with lower desired scores. it is clear that from the high desired scores that faculty place the highest priority on information control items and report that the libraries are not meeting minimum expectations in these areas. this is consistent with columbia’s scores from 2003 and 2006. figure 2 displays the scores for ic-8 by faculty discipline, illustrating that health sciences, law, architecture, math, engineering, education, humanities, computer science, history, and business faculty reported a negative adequacy gap for ic-8 in 2009 at columbia. these departments were targeted for follow-up interviews with faculty, excluding the health sciences, law, and education departments, as these populations were not included in the initial survey sample. many of the response counts for individual departments were low. however, it was felt that this was a sufficient way to identify which departments were relatively less satisfied than others, and all were included for the sake of consistency.   results at arl institutions   figure 3 displays the composite faculty scores from the 21 arl libraries participating in libqual+® 2009, included in this study. in 2006, self concluded that there was no correlation between expenditures and faculty desired scores for journal collections (r = -0.14) (self, 2008). this analysis was not revisited in the current study.   figure 1 libqual+®  2009, columbia university faculty   figure 2 libqual+® 2009, columbia university faculty, ic-8 by discipline.   figure 3 libqual+® 2009, arl libraries faculty   looking at the 2009 scores for journal collections across these arl libraries, it is clear that faculty across these institutions remain dissatisfied with journal collections. figure 4 displays the scores for each of the 21 arl libraries included in this analysis. libraries are arranged from largest library (on the left) to smallest (on the right), based on total library expenditures as reported by arl (kyrillidou & bland, 2009). columbia is the first from the left in figure 4. none of these 21 libraries achieved a positive adequacy gap in 2009. it can be observed that the desired scores appear relatively consistent between institutions (generally around 8.5) and not remarkably higher than the same desired scores for arl libraries in 2006 (cook, 2006).   figure 4 libqual+® 2009, faculty ratings of journal collections, arl libraries   comparisons over time   in his original study, self illustrated that 2006 was not a unique year for negative adequacy gaps on ic-8, showing similar scores for arl institutions from 2004 through 2006. figure 5 displays ic-8 scores for faculty from 2006 through 2009, further demonstrating the trend. the chart is labeled with the mean and standard deviation for each data point.   taking this analysis further, another question was asked: has there been a statistically significant change in ic-8 scores – indicating a change in faculty satisfaction – since 2006? after conducting an anova (analysis of variance) using the mean adequacy gaps from each arl institution for ic-8 from 2006 through 2009, the significance was calculated to be 0.119, which is not deemed statistically significant. in other words, there has been no significant change in the adequacy gap for ic-8 since 2006, and satisfaction relative to expectations remains consistent, showing neither improvement nor decline. faculty members are just as dissatisfied with journal collections today as in 2006. a more meaningful evaluation of change over time might involve the use the individual respondent scores from each institution rather than means; unfortunately, this data is not available. anova analysis was not conducted on the item scores (minimum, desired, perceived), and would be recommended to explore the topic further. looking at the data in figure 5, it can be observed that the desired scores have remained relatively stable, while the minimum scores and perceived scores have increased slightly. is the zone of tolerance shrinking?   figure 5 libqual+® 2006-09, faculty ratings of journal collections, arl libraries   information control   information control items have consistently shown the highest desired scores (indicating high-priority) among faculty, as well as the largest negative adequacy gaps. how does ic-8 compare to the other ic items in libqual+®? charting the adequacy gaps over time illustrates the change in the size of the gaps, and whether the gaps are positive or negative. figure 6 shows that the adequacy gaps for ic-1, ic-5, ic-6, and ic-7 have remained relatively stable. items ic-2, ic-3, ic-4, and ic-8, show greater change over time. the observed change in these scores (aside from ic-8) has not been evaluated for statistical significance.   ic-4, addressing electronic resources, has had a consistently negative adequacy gap, indicating faculty dissatisfaction with service in this area. looking at the ic-2 scores from arl libraries since 2006, a similar trend to ic-8 can be observed. faculty perceptions are consistently negative: libraries are not meeting faculty’s minimum expectations for “a library web site enabling me to locate information on my own.” due to the increasingly digital nature of journal collections as well as faculty dependence on the library website to access them, future analysis should explore the correlation between ic-2, ic-4 and ic-8. at the very least, it appears that ic-2 may be “the next ic-8,” in terms of consistently negative adequacy gaps. further, the website may play a critical role in improving journal collection and e-resource scores over time.   figure 6 libqual+®  2006-09, information control adequacy gaps over time   journal ratings and overall satisfaction   following up on self’s correlation analysis of ic-8 and overall satisfaction (“how would you rate the overall quality of the service provided by the library?”), additional correlations were run as part of this study. in 2006, self found a strong correlation of 0.84 between journal collections and overall satisfaction. in 2009, a correlation of 0.71 was found (figure 7).   when reviewing the correlation between each item and overall satisfaction with library services over time, ic-8 does show the highest mean correlation (0.67) with the smallest standard deviation between years (0.11). looking at the correlations by year, 2008 shows the strongest correlations between ic items and overall satisfaction, with a mean of 0.78 and a standard deviation of 0.10. these correlation calculations would be stronger, and perhaps more accurate, were they computed using the individual scores for each faculty respondent from each institution, rather than the mean scores of all faculty respondents at each institution.   in table 1, there appear to be 2 clusters in the correlations. ic-2, ic-3, ic-4, and ic-7 show correlations in the high 50s across time. ic-1, ic-5, and ic-6 show correlations in the high 40s and low 50s. this may indicate that the collections, both print and electronic, and the ability to access them easily, are of greater importance to faculty’s overall satisfaction with library services.   following up at columbia   twenty-four follow-up phone interviews were conducted with faculty from departments identified via libqual+® scores as being dissatisfied with library journal collections (table 2). history faculty members were not included in recruitment for this phase of the study. faculty members at columbia were asked identical questions to those used at u.va. in 2006. faculty members were asked about whether journal collections were meeting their minimum and desired service levels, as well as their preferences for print or electronic journals.   figure 7 libqual+®  2009, correlation of faculty satisfaction with journal collections (ic-8) and overall library service from 21 libraries   table 1 libqual+® 2006-09, correlation coefficient of ic-8 adequacy gap and overall library service score ic item 2006 2007 2008 2009   standard deviation (by question) mean (by question) ic-1 0.44 0.61 0.44 0.55   0.08 0.51 ic-2 0.71 0.55 0.61 0.42   0.12 0.57 ic-3 0.86 0.42 0.31 0.73   0.26 0.58 ic-4 0.71 0.67 0.4 0.61   0.14 0.60 ic-5 0.72 0.49 0.27 0.45   0.19 0.48 ic-6 0.73 0.58 0.42 0.12   0.26 0.46 ic-7 0.81 0.46 0.40 0.67   0.19 0.59 ic-8 0.80 0.60 0.55 0.71   0.11 0.67                 standard deviation (by year)) 0.13 0.08 0.11 0.20       mean (by year)) 0.72 0.55 0.43 0.53         table 2 libqual+® 2009, columbia disciplines with negative adequacy gaps columbia discipline phone interviews conducted n for libqual+® 2009 2009 ic-8 departmental mean adequacy gap architecture 5 8 -1.375 business 6 8 -0.125 computer science 4 3 -0.333 engineering 4 12 -0.583 history 0 22 -0.318 humanities 5 60 -0.379 math 1 4 -0.750   overall, participants responded positively regarding the libraries’ journal collections, stating that, yes, the collections meet the minimum expectations. however, 15 of the 24 participants stated that, no, the library is not meeting their desired level of service for journal collections. further probing uncovered some key issues: support, work-arounds, search and online access, collection gaps, coverage, quick list, and resources. (see appendix a for additional quotes from each category.)   support: service provided by library staff and systems regarding journal acquisition, use or problems   these statements generally focused on the quality of automated responses from library systems, or lack thereof. service issues could indicate a correlation between certain affect of service items and satisfaction with journal collections. as the online collections continue to become more complex to navigate, expert support from library staff will become more important.   “what would be great for faculty would be if when things are not available, there was one source in the library, extraordinarily skilled at tracking down items. [. . .] this happens about once a week for me that i need this service. [. . .] these people would be specialists in working the electronic and journal capabilities.”   work-arounds: faculty’s alternate methods for accessing the journals they need   there was some discussion about barriers to access when using library resources. expectedly, faculty will find their own ways to access the articles they need, and are generally comfortable with their work-arounds. these work-arounds seemed rather common, and often complex or expensive. librarians rarely played a role in these processes, as reported in the interviews. while a primary concern is that faculty find access to materials, through the library or otherwise, there are some clear disadvantages to the work-arounds.   “i just buy them individually from my research funds, so it’s coming out of my research money. i can afford to buy only individual subscriptions, so i can’t share with my students.”   one professor reported an elaborate process of seeking out articles for a course (after using cul’s search tools without success) and working with a colleague at another institution to get copies of the needed articles. “it was kind of unwieldy, but i got her on the phone and i needed six articles from the journal from different years. we got on the phone and i would tell her the citation, and she would go to her collections, download the pdf, and sent it to me.” this anecdote is striking for two reasons. one: it has since been confirmed that cul had subscribed to the journal in question. two: this professor did reach out to a librarian for assistance, but remembers receiving no response.   search and online access: use of online tools to identify and access needed information   libraries typically present users with a series of search tools developed by various vendors, based on widely differing search processes. it is no wonder that search and interface design are key issues for faculty. the comments on this topic reflect concerns about the libraries’ catalog (clio), the journal search interface, and specific e-journal interfaces. there was also some discussion of the quality of indexing for journals – both print and electronic, and the ability to easily and efficiently use the libraries’ website to find them.   “i think just having free text search, like google book search, would be something that would be very, very useful to have. i still feel like we are living 20 years behind where the rest of the world is in terms of being able to search these databases and large collections of books that we have.”   collection gaps: instances where the libraries do not subscribe to a particular title, or type of journal   foreign language journals were mentioned regularly. when participants were asked if they request titles that the libraries does not currently subscribe to, most said no. the general sentiment was that the process for requesting could be streamlined.   “there are things published around the world we don’t have. things that are between journals and edited books [. . .]. university publications or things like that. foreign journals.”   interlibrary loan (ill) also plays an important role in managing collection gaps. a few interview participants noted that they would not be satisfied with the collections if ill could not get items from other libraries. “i don’t recall not being able to get something at columbia. and, when i needed it and they didn’t have it, it was there through ill.” the consequences of these collections gaps are uncertain. one participant stated, “fifty percent of the time [that i can’t locate an item] i go without. fifty percent of the time i will email the authors or i will go to the author’s website.”   coverage: within a journal title, there are gaps in the back-file coverage   complete coverage of a held title is consistently desired. “even if they’ve stopped issues for a year or two, i think it’d be good if they could at least get the back issues.”   there are also issues of coverage currentness, particularly with e-journals. “one obvious problem is that the [electronic] journals are always behind. we’re sending students to the library to read more recent issues.”   quick list: desire for a discipline-specific “quick list” that would provide easy access to the most important online journals   these comments spoke directly to a relationship between the online search interfaces (perhaps indicated in ic-2) and the collections.   “if i was to give a suggestion, maybe to have discipline-specific pointers that could help each discipline find things. [. . .] we need help remembering how to use the interface. it’s more of an interface issue than a collections issue.”   pdfs (portable document format) were mentioned frequently and have clearly become the preferred format for accessing electronic content. given that the libraries’ website provides links to multiple vendors for a particular title (each with its own access caveats) there is a desire to know which one is “best.” “best” would be, according to interviews, the one vendor that provides complete coverage of a title and pdfs for download.   resources: the libraries’ allocation of resources   startlingly, two participants implied that they would prefer to have library funds diverted from acquiring additional materials for the collection, to making the collection more easily accessible.   “the size of the collection is not as important as getting the current collection working as smooth as possible. before, when we used to go to the library, we got service.”   print vs. electronic   regarding the preference for print or electronic, two of twenty-four participants stated a clear preference for print. a small number of participants responded that they would prefer to have both print and electronic available (as is often the case, currently) or that print is preferable for historic or archival materials only. overwhelmingly, the flexibility and access to electronic journals was highly desired and praised, particularly when pdfs are available.   some faculty stated that because some journals are currently available in print, they expect to keep accessing them in this manner. this may change over time, as more materials are digitized at higher quality and made available online. one participant stated their preference for electronic materials, noting “a few years ago, i wouldn’t have said that. but, i guess things have changed.”   remote access   there were far fewer complaints about connecting to online resources from off-campus than expected. the topic came up a handful of times but was not at a “crisis” level for the majority of participants. in general, this did not seem to be a barrier for using journal collections for the majority of study participants.   moving forward at columbia   as of the writing of this paper, the collections & services directors, along with the collection development unit, are reviewing the results of this study. it is expected that the interview information, along with formal usability studies, will be useful in the upcoming redesign of the libraries’ website, as well as in the implementation of future search tools. columbia will continue to engage faculty in discussions about journal collections. their active involvement will be crucial in improving this area of library service. libqual+® scores and comments will continue to play a role in tracking this issue at columbia.   conclusion   returning to our initial motivation for this study – “given the substantial investment in journals at arl libraries, why are faculty at these institutions consistently dissatisfied with their library’s journal collections?” – what have we learned? without question, faculty at arl libraries continue to show dissatisfaction with journal collections, despite the continuing evolution of access tools, delivery services, and growing collections which arl libraries provide. given the economic downturn of 2010, libqual+® scores may show a noticeable decrease in satisfaction with journal collections, depending on the impact of budget cuts throughout arl libraries. faculty at columbia are satisfied with many aspects of the journal collections the libraries provide. of course, they also want easier access to online journals, reliable pdf downloads, and better support from library systems and staff. providing faculty with discipline-specific “quick lists” may be one way to bridge the gap in satisfaction with journal collections. what, if anything, has changed since 2006? relative satisfaction with journal collections at arl libraries has not changed significantly. faculty continue to show dissatisfaction with journals collections across arl libraries. while it may be observed that desired scores for journal collections remain consistent and minimum scores are on the rise, the gap between the minimum and “reality” remains the same. and, it is a complex reality to navigate.   information control issues, as measured by libqual+®, continue to be top priority for faculty at arl libraries, as well as a consistent area of dissatisfaction. as seen from the libqual+® scores, print and electronic collections, including journals, and the ability of a library website to provide easy access to materials, are critically important to overall satisfaction with library services. as noted in the interviews, access and use of journal collections is dramatically more complex when dealing with electronic resources. libraries will need to continue to address these needs by re-allocating resources and staff to this growing area of service.   finally, are there other information control items that libraries should be watching? yes: the library website – typically the sole tool for accessing and using journal collections – is becoming an area of consistent dissatisfaction among faculty. journal collections, however, continue to be the area of least satisfaction for faculty at institutions with arl libraries. it is expected that the relationship between the website and collections will only strengthen over time, for better or worse.   clearly, the issue of satisfaction with journal collections is complex, ever-more technical, and faculty have little tolerance for faulty systems. the evolution of the electronic journal collections and the inherent access challenges will continue to play a critical role in faculty satisfaction as libraries strive to provide ever-better service.   acknowledgement   special thanks to rebecca chovnick for her work as a research assistant for this research project, and to shanna jaggars for her assistance and advice regarding data analysis.     references   cook, c., davis, m., heath, f., kyrillidou, m., thompson, b., and roebuck, g.  (2009). libqual+® 2009 survey notebook. washington, dc: association of research libraries. cook, c., kyrillidou, m., heath, f., sousa, j., thompson, b., and webster, d. (2006). libqual+® 2006 survey notebook. washington, dc: association of research libraries.   kyrillidou, m., & bland, l. (2009). arl statistics 2007-2008.  washington, dc: association of research libraries. retrieved 30 may 2013 from http://publications.arl.org/arl-statistics-2007-2008/   self, j. (2008). bound for disappointment: faculty and journals at research institutions. arl: a bimonthly report, 257, 7-11. retrieved 30 may 2013 from http://www.arl.org/storage/documents/publications/arl-br-257.pdf   strauss, a. l. (1987). qualitative analysis for social scientists. cambridge, england: cambridge university press.   appendix a additional quotes arranged by key issues   collection gaps   “i tried to actually put in a request for this, but i couldn’t find the web form.” “i don’t know how easy it is to recommend a journal you don’t have. […] maybe that could be a little easier.” “[…] we’re global, so for us not to have various european or asian journals, particularly older ones, is problematic.” “i’ve started to tailor my searches to the journals i know the library has.” “especially in foreign languages. some of them are quite new, you know, no one has probably ever requested them before.” “send out a list every year; we could submit a list of what the library should [have].”   interlibrary loan   “there are so many other means of access, whether it’s via jstor or borrow direct or some other resource. i always manage to get the article i’m looking for.” “i will say, it’s rare – ill is wonderful; it’s rare that they [ill] can’t find a copy of an article somewhere or a journal somewhere. but, it can take a very long time and sometimes on rare occasions, they can’t.”   quick list   “[…] one thing that we used to have a long time ago that’s no longer there is a list of the most commonly used journals, or the top journals in economics and finance. so a page on the website that would then be a collection of links to maybe the top thirty or so journals in economics and finance, and just go there. it saves a few steps. it would be good to reinstate that, and maybe have different lists for different departments.” “on my web page i have my favorite journals, and it worked fine until you changed it. […] so, originally, my idea of a home page was one stop. everything’s here. you don’t have to do anything else.”  “it would be nice if there was just a quick list and easy guide to the most popular titles.” “to create real interface pages for particular disciplines and fields that are annotated connections to journals online and databases.” “if there’s an opportunity to improve the web interfaces, and add discipline-specific hints for navigating to the online collections. i don’t do it often enough to always remember how to do it.” “you know, top ten lists of highly recommended books or articles that someone has really benefitted from. and, i suppose if the library had some way of selecting things that would be specifically of interest and very targeted, that might make a difference.”   resources   “don’t touch the collection, and put the money into figuring out how to be able to use it off campus or on, so it’s transparent.” “the collection is a pain. it’s throwing away money until i can use it.” “that would be something i would have the field of librarianship work on. how to use the technology to make it more user-friendly, rather than spend resources trying to collect print.”   search and online access   “in clio, if i want to do a search, it’s not always accurate as to what journals are available and what we don’t have. so, you have to go through multiples resources to know for sure.”  “there might be a way to streamline ways of downloading articles. sometimes you go to the web site and see five different sources for the same journal. one’s cleaner, some are better formatted, some are the same, and it’s not always easy to tell which ones are best. […] if there’s five sources someone could go through them and see which ones are best.” “i counted once how many times i had to click to get to what i wanted, finally in pdf form, and it was something like ten clicks.”  “i had the impression that, depending on how i searched [in the catalog], i would get completely different results.” “i’ll be browsing nature.com and then i’d like to read an article and there’s no easy way for me to suddenly be at that same page but logged in with my cu credentials. instead i have to open a new browser and go through columbia’s interface to get back to the same page […] that’s certainly consistent with the minimum required.” “sometimes it’s a bit difficult. sometimes i just go and talk to a reference librarian because i’m not very good at navigating the system.”   support   “i want a response that someone got my request. and if nothing happens, someone to call.” “i need training.” “i do always feel that if i have any questions that i always get a sense of helpfulness in all the libraries.”   work-arounds   “i’ve never had any problems getting what i wanted, but that’s also because i have a research assistant.” “if it’s directly in my work, i’ll just buy it myself.” “when i can’t find something through ill, i ask colleagues in the field if they know of a place to get it, and then to resources of their university. can they make a copy of the given article or something like that.”  “i personally subscribe to several journals which are the ones that i most read. which are the ones that i would most read if they were in the library, but i have them myself as part of, in some cases, society memberships.” evidence summary   patron-driven acquisition of e-books satisfies users’ needs while also building the library’s collection   a review of: fischer, k. s., wright, m., clatanoff, k., barton, h., & shreeves, e. (2012). give ’em what they want: a one-year study of unmediated patron-driven acquisition of e-books. college & research libraries, 73(5), 469-492.    reviewed by: giovanna badia liaison librarian, schulich library of science and engineering mcgill university montreal, quebec, canada email: giovanna.badia@mcgill.ca   received: 2 jun. 2013     accepted: 1 aug. 2013      2013 badia. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract objective – to present the initial results of an academic library’s one-year pilot with patron-driven acquisition of e-books, which was undertaken “to observe how user preferences and the availability of e-books interacted with [the library’s] traditional selection program” (p. 469). design – case study. setting – the university of iowa, a major urban research university in the united states. subjects – original selection of 19,000 e-book titles from ebrary at the beginning of the pilot in october 2009. to curb spending during the pilot, the number of e-book titles available for purchase was reduced to 12,000 titles at the end of december 2009, and increased to nearly 13,000 titles in april 2010. methods – these e-book titles were loaded into the library’s catalogue. the goal was for the university of iowa’s faculty, staff, and students to search the library catalogue, discover these e-book titles, and purchase these books unknowingly by accessing them. the tenth click by a user on any of the pages of an e-book caused the title to be automatically purchased for the library (i.e., ebrary charged the library for the e-book). main results – from october 2009 to september 2010, the library acquired 850 e-books for almost $90,000 through patron-driven acquisition. the average amount spent per week was $1,848 and the average cost per book was $106. researchers found that 80% of the e-books purchased by library patrons were used between 2 to 10 times in a 1-year period. e-books were purchased in all subject areas, but titles in medicine (133 titles purchased, 16%), sociology (72 titles purchased, 8%), economics (58 titles purchased, 7%), and education (54 titles purchased, 6%) were the most popular. two of the top three most heavily used titles were standardized test preparation workbooks. in addition, 166 of the e-books purchased had print duplicates in the library, and the total number of times the print copies circulated dropped 70% after the e-versions of these books were obtained.   the authors also examined usage data for their subscription to ebrary’s academic complete collection from september 2009 to july 2010, which consisted of 47,367 e-books. together with the 12,947 book titles loaded into the catalogue for the patron-acquisition pilot, there were a grand total of 60,314 ebrary e-book titles in the library catalogue that were accessible to the iowa university community. the study revealed that 15% of these titles were used during this 11-month period, and the used titles were consulted 3 or more times. the authors sorted the user sessions by publisher and found that patrons used e-books from a wide variety of publishing houses, of which numerous university presses together constituted the majority of uses. the five most heavily used e-books were in the fields of medicine, followed by economics, sociology, english-american literature, and education. conclusion – the authors’ experience has shown that patron-driven acquisition “can be a useful and effective tool for meeting user needs and building the local collection” (p. 490). incomplete coverage of academic publications makes patron-driven acquisition only one tool among others, such as selection by liaison librarians, which may be employed for collection development. according to the authors, patron-driven acquisition “does a good job of satisfying the sometimes unrecognized demand for interdisciplinary materials often overlooked through traditional selection methods,” (p. 491) and alerts librarians to new research areas.   commentary   this study is not unique, in that the authors cite other research that investigated patron-driven acquisition for books, both in print and electronic formats (e.g., anderson, freeman, hérubel, mykytiuk, & ward, 2010; bracke, 2010; price & mcdonald, 2009). the authors do not state how this study supports, differs from, or adds to existing literature on the topic, thereby making it difficult to place it in its wider research context.   this reviewer used glynn’s ebl critical appraisal checklist (2006) to assess the study’s methodology. an original selection of 19,000 e-book titles from ebrary were loaded into the library’s catalogue at the beginning of the pilot, but this was reduced to almost 13,000 titles a few months later in order to curb patrons’ spending. the original inclusion or exclusion criteria for selecting which e-books should be entered in the catalogue were not provided, but would have been helpful for readers who wish to replicate this pilot at their own institutions. this reviewer would also have liked to know if there were any differences between the marc records for the e-books, which were supplied by ebrary and loaded into the library catalogue as is, and the marc records for all the other items in the catalogue. any differences between the records may have affected the results, especially the data regarding the uses of the print versus electronic versions of the same book titles, since one set of records may have contained more descriptive information that could have been discovered in a search.   this study provides data to support the idea that patron-driven acquisition of e-books is useful for collection development, and also offers some details for its implementation. taken together, the data and the literature review contained in this article present a strong case for readers to try patron-driven acquisition at their own institutions.   references   anderson, k. j., freeman, r. s., hérubel, j-p. v. m., mykytiuk, l. j., nixon, j. m., & ward, s. m. (2010). liberal arts books on demand: a decade of patron-driven collection development, part 1. collection management, 35(3/4), 125-141. doi:10.1080/01462679.2010.486959   bracke, m. s. (2010). science and technology books on demand: a decade of patron-driven collection development, part 2. collection management, 35(3/4), 142-150. doi:10.1080/01462679.2010.486742   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi:10.1108/07378830610692154   price, j., & mcdonald, j. (2009, nov.). beguiled by bananas? a retrospective study of the usage & breadth of patron vs. librarian acquired ebook collections. proceedings of the charleston library conference, charleston, sc. doi: 10.5703/1288284314741   article   grey literature searching for health sciences systematic reviews: a prospective study of time spent and resources utilized   ahlam a. saleh assistant librarian arizona health sciences library university of arizona tucson, arizona, united states of america email: asaleh@ahsl.arizona.edu   melissa a. ratajeski reference librarian health sciences library system university of pittsburgh pittsburgh, pennsylvania, united states of america email: mar@pitt.edu   marnie bertolet assistant professor graduate school of public health university of pittsburgh pittsburgh, pennsylvania, united states of america email: mhb12@pitt.edu   received: 4 june 2014     accepted: 7 aug. 2014         2014 saleh, ratajeski, and bertolet. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to identify estimates of time taken to search grey literature in support of health sciences systematic reviews and to identify searcher or systematic review characteristics that may impact resource selection or time spent searching.   methods – a survey was electronically distributed to searchers embarking on a new systematic review. characteristics of the searcher and systematic review were collected along with time spent searching and what resources were searched. time and resources were tabulated and resources were categorized as grey or non-grey. data was analyzed using kruskal-wallis tests.   results – out of 81 original respondents, 21% followed through with completion of the surveys in their entirety. the median time spent searching all resources was 471 minutes, and of those a median of 85 minutes were spent searching grey literature. the median number of resources used in a systematic review search was four and the median number of grey literature sources searched was two. the amount of time spent searching was influenced by whether the systematic review was grant funded. additionally, the number of resources searched was impacted by institution type and whether systematic review training was received.   conclusions – this study characterized the amount of time for conducting systematic review searches including searching the grey literature, in addition to the number and types of resources used. this may aid searchers in planning their time, along with providing benchmark information for future studies. this paper contributes by quantifying current grey literature search patterns and associating them with searcher and review characteristics. further discussion and research into the search approach for grey literature in support of systematic reviews is encouraged. introduction   a properly conducted systematic review summarizes the evidence from all relevant studies on a topic concisely and transparently (cook, mulrow, & haynes, 1997). the searches to support these reviews need to be extensive often including extended searches of the grey literature.   grey literature can be described in a number of ways, but commonly has been defined by the 1997 luxembourg convention on grey literature definition as literature: “which is produced on all levels of government, academics, business and industry in print and electronic formats, but which is not controlled by commercial publishers” (farace, 1998, p.iii). in 2004, at the sixth international conference on grey literature, a postscript was added to further expand on the “commercial publishers” aspects of this definition. in recent years, the definition has stimulated new discussion due to changes in the environment such as the evolving landscape of information dissemination and the introduction of new avenues of scientific communication. according to two-thirds of respondents in the grey literature survey (boekhorst, farace, & frantzen, 2005), “grey literature is best described by the type of document it embodies” (p.6). some examples of grey literature include: reports, conference abstracts, dissertations, and white papers (greynet international, 2013).   systematic review support seems to be of growing interest to health sciences information professionals. a 2013 survey of librarians and directors about emerging roles of biomedical librarians found that support of systematic reviews was one of the top six most common reported new roles (crum & cooper, 2013). furthermore, in a 2013 systematic review of the literature from 1990-2012 on changing roles for health sciences librarians, systematic review librarian was identified as one of the newer roles in the field (cooper & crum, 2013).   in recent years, education opportunities relating to systematic review searching have transpired. this is no surprise, given that information professionals planning to get involved in systematic reviews would need to familiarize themselves and learn more about the process and specifically about the search process, as it is distinct from routine literature searches.  as an example of these emerging education opportunities, a search of the medical library association’s educational clearinghouse (http://cech.mlanet.org/), found eight continuing education (ce) courses with the words “systematic review” in the title, and there were at least three ce courses found listed on the canadian health libraries association website (http://www.chla-absc.ca/node/119). systematic review as a topic is also emerging in library science curricula as can be found in select content covered in the certificate of advanced study in health sciences librarianship (healthcas), reference services and instruction in healthcare environments course previously offered through the university of pittsburgh. the university of alberta, school of library information studies has a course entitled “systematic review searching”, and the texas woman’s university medical library curriculum  planned a course for systematic reviews that would  be launched in spring 2014 (c. perryman, personal communication, july 7, 2013).   in 2011, the institute of medicine (iom) released standards for systematic reviews which indicate that researchers should “take action to address potentially biased reporting of research results” (p.84). to address this bias, standards 3.2.1 and 3.2.4 call for the inclusion of grey literature searches in all systematic reviews and handsearching of selected journals and conference abstracts (institute of medicine [iom], 2011). these standards may lead to wider acceptance of including grey literature searching in systematic review methodology.  as librarians become increasingly integrated into systematic reviews, as called for by the iom standard 3.1.1 which specifically states “work with a librarian or other information specialist trained in performing systematic reviews to plan the search strategy”, they must be prepared to search the grey literature or at least provide guidance on resources and search strategies (iom, 2011, p.84).  locating grey literature can often be challenging, requiring librarians to utilize a number of databases from various host providers or various websites, some of which they may not be familiar with or aware of.  additionally, searchers may need to spend time learning various search interfaces and the nuances of each resource, such as how to download references or if the search query boxes have  term limits (wright, cottrell, & mir, 2014).   investigation into the grey literature search process for systematic reviews may reveal useful information that can be applied by information specialists planning and preparing for systematic review searches. there is limited information on the time it takes to search grey literature in support of systematic reviews and we are not aware of any studies which relate searcher or systematic review characteristics to either time spent searching the grey literature or which grey literature resources are selected for the search. therefore, we sought to explore these aspects of the process of grey literature searching in support of systematic reviews.   literature review   acceptance of the inclusion of grey literature in systematic reviews has varied over time. a 2006 survey showed approximately 90% of systematic reviewers and approximately 70% of editors felt grey literature probably or definitely should be eligible for inclusion in systematic reviews, while a prior 1993 survey showed that 78% of meta-analyst and methodologist respondents felt that unpublished material should definitely or probably be included in systematic reviews, and that only 47% of journal editors felt this way (cook et al., 1993; tetzlaff, moher, pham, & altman, 2006). the biggest concerns were about the lack of peer review and quality of the studies found in the grey literature. however, other studies detail the benefit of including grey literature in systematic reviews (crumley, wiebe, cramer, klassen, & hartling, 2005; savoie, helmer, green, & kazanjian, 2003). two cochrane reviews further support why it is important to search for grey literature. a 2007 cochrane systematic review on the use of grey literature in meta-analyses of randomized trials found that non-grey literature trials tended to be larger and showed an overall larger treatment effect when compared to grey literature trials (hopewell, mcdonald, clarke, & egger, 2007). another cochrane systematic review on time to publication for results of clinical trials found that positive result trials tended to be published earlier than negative or null result trials and positive result trials were more likely to be published than negative or null result trials (hopewell, clarke, stewart, & tierney, 2007). therefore, when conducting searches for systematic reviews searching for grey literature may be used as a means to minimize the introduction of bias such as publication and time-lag bias.   there is a time cost for including a grey literature search in a review. how much time is difficult to estimate however and, because librarians in an academic or hospital role are often juggling other responsibilities, time management is crucial.  there seems to be limited research specifically reporting on the time taken to conduct literature searching for systematic reviews. if time is reported, it is often grouped with other tasks such as article retrieval and screening, or the search time is listed as one number, not denoting the time differences for various resources such as pubmed vs. a grey literature resource such as clinicaltrials.gov. in an examination of 37 meta-analyses, allen and olkin (1999) found that the average systematic review took 1139 hours to complete (with a range of 216 to 2518 hours). of this time, 588 hours accounted for protocol development, searches, retrieval, abstract management, paper screening, blinding, data extraction and quality scoring and data entry. in a single meta-analysis conducted by steinberg et al. (1997), a description of the time to complete various systematic review tasks, including screening, extracting data, and writing the manuscript was reported. they estimated the total hours for conducting the review to be 1046 hours (26 weeks) of which 24 hours was used to conduct the literature search. guise and viswanathan (2011) estimate that it would take 1-4 weeks to run comparative effectiveness review searches.   greenhalgh and peackock (2005) reported the time taken for electronic database searches for their systematic review (including developing the search, refining, and adapting to other databases) as approximately two weeks of a librarian specialist’s time. this article also looked at how productively the time was spent. the two weeks of a librarian’s time “yielded only about a quarter of the sources an average of one useful paper every 40 minutes of searching” (greenhalgh & peacock, 2005, p. 1065). greenhalgh and peackock (2005) go into further detail and compares electronic searching with handsearching. a handsearch of 271 journals took approximately a month of time resulting in 24 papers that were included in the final report – “an average of one paper per nine hours of searching” (greenhalgh & peacock, 2005, p. 1065). using traditional electronic databases to search the literature does not always identify all relevant studies. this can be due to a number of reasons including lack of appropriate indexing terms, lack of indexing all sections of a journal, or research methods not being fully described in the abstract (hopewell, clarke, lefebvre, & scherer, 2007). therefore handsearching of journals or conference proceedings would be particularly relevant to include in the systematic review search methodology. the cochrane handbook for systematic reviews of interventions defines handsearching as follows: “handsearching involves a manual page-by-page examination of the entire contents of a journal issue or conference proceedings to identify all eligible reports of trials” (cochrane collaboration, 2011, section 6.2.2.1). handsearching of conference proceedings could be considered as a type of grey literature searching.   studies other than greenhalgh and peackock have also looked at the time required for handsearching. additional reported time ranged from 30 minutes per journal issue to 45 minutes 3 hours per year of a title and this may vary based on subject matter (adams, power, frederick, & lefebvre, 1994; armstrong, jackson, doyle, waters, & howes, 2005; croft, vassallo, & rowe, 1999; jadad, carroll, moore, & mcquay, 1996).   there are a variety of studies that have examined how characteristics such as search experience, relate to search quality, search speed, search effectiveness and the search process (al-maskari & sanderson, 2011; debowski, 2001; fenichel, 1981; hsieh-yee, 1993; kuhlthau, 1999). many of these studies seem to focus on varying experience levels within end-user groups. though, tabatabai and shore (2005) explored how experts (highly experienced librarian professionals), intermediates (final year master of library information studies students) and novices (undergraduate teachers) searched the web. significant differences in patterns of search among the different groups were found in cognition, metacognition, and prior knowledge strategies (tabatabai & shore, 2005). previous literature also demonstrates interest in exploring different experience levels in relation to performance of a systematic review. a study (riaz, sulayman, salleh, & mendes, 2010) reporting on systematic reviewers noted that new systematic reviewers experienced problems with time taken to conduct the review, defining the research question and inclusion/exclusion criteria, and data management that were not faced by experienced systematic reviewers. given this insight from the literature, it seems useful to consider subject characteristics as variables that may potentially impact outcome searches.   aims the purpose of this study was to explore grey literature searching for health sciences systematic reviews. more specifically:   to explore the time taken to conduct grey literature searches for systematic reviews to explore the resources selected for grey literature searches for systematic reviews to evaluate whether any relationship exists between searcher and systematic review characteristics and time to search or number of resources selected for grey literature searches in support of systematic reviews   methods   study recruitment   university of pittsburgh institutional review board (irb) approval was obtained to conduct the study. participants were recruited through listservs, social media and email contacts. a total of 19 listservs, including information professional types and those whose subscribers were thought to have an interest in systematic reviews, were used for recruitment. social media sources included posting on facebook, posting on popular librarian blogs, and having fellow librarians post tweets. the prerequisite for study enrolment was: the participant must currently be embarking on a literature search to support a systematic review or plan to in the near future. however, searching should not have taken place prior to enrolment. the searcher did not have to be a librarian but had to be the person responsible for building the searches and running them in each database used.    data collection   a survey instrument was developed and pilot tested on a small sample which included two systematic review course instructors (a clinician and a biostatistician) and five information specialists. revisions to the survey forms were made based on feedback. the survey forms were distributed in two parts (see appendix for surveys). part one of the survey collected demographic information about the searcher and their systematic review experience and was required to be submitted directly after study enrolment. part two of the survey was provided to participants after they submitted part one. return of the part two form was expected upon completion of the systematic review searches. this part of the survey collected information about the systematic review including topic, population, and if the systematic review was grant funded. participants were also asked to document the name of the resources searched, platform or vendor of the resource, resource url when applicable, and the time taken to search the resource (electronic database, website searching or handsearching). reported time was to account for choosing terminology, developing the strategy, refining and running the search. in order to minimize the potential of bias in the searching, it was not disclosed to subject participants that this was a survey specifically focused on grey literature searching in systematic reviews. rather, participants were asked to include all resources searched for the systematic review including handsearching. other means of searching for studies, such as citation tracking (i.e. snowballing) or contacting key authors or experts were not specifically requested in our data collection form. email reminders were sent twice to participants who did not submit their completed data forms.   data analysis   from the completed surveys, a list of the resources participants used in their systematic review searches was compiled. for some resources various interfaces were used to complete the search i.e. ovid medline vs. pubmed. in these cases the resources were classified as the same despite the interface used with the exception of cochrane library and the centre for review and dissemination (crd) resources. some users searched both cochrane library and the crd version of the databases: database of abstracts of reviews of effects (dare), nhs economic evaluation database (nhseed) and health technology assessment (hta), for the same review. cochrane library individual resources were all grouped under “cochrane library,” and if a crd search platform was also used, that resource was categorized separately as “crd database.”   once the list of resources was compiled, each resource was labeled as grey literature or non-grey literature. to categorize the resources, the librarian authors used the 1997 luxembourg definition of grey literature, reviewed the database content, contacted the database content producers as necessary, and used their own expert opinion. resources that predominately included literature from journals were categorized as non-grey. resources that included citations mostly from book chapters, theses, reports, conference materials or other type of grey literature were categorized as grey. because many resources may cover both grey and non-grey literature for these resources, assignment of grey vs. non-grey was based on the authors’ estimation of which type of content was the majority. handsearching was denoted as a grey literature resource because conference proceedings were primarily handsearched for participant systematic reviews which included handsearching. once categorization was completed, the number of grey literature vs. non-grey literature resources and the time taken to search each type of resource was tallied.   the six outcomes analyzed were the number of grey, non-grey and total literature resources along with the amount of time searching each of those resources. these outcomes were compared across the searcher and the systematic review characteristics. continuous variables were compared across groups using a kruskal-wallis test instead of a t-test because non-normality in some of the variables violated t-test assumptions. boxplots comparing the amount of time searching the literature across groups were graphed on the log scale due to a few extremely large values. kruskal-wallis tests were performed on the log scales for the amount of time searching the literature across groups to be consistent with the boxplots. sensitivity analysis without the log transformations yielded similar results.   results   out of 81 initial respondents, 17 (21%) completed both parts of the study and were included in the data analysis. nineteen respondents were excluded because they did not meet the study prerequisites. the remaining respondents withdrew from the study or did not complete both parts of the survey. of the 17 final participants, 15 reported that their primary professional role was a librarian/information professional.   searcher characteristics   most study participants were from an academic environment (figure 1). hospital was reported by 24% and other, which included one non-government agency and one independent research company, made up 11% (n= 2) of participants’ institution. the participant country representation was mostly comprised of united states (us), canada, and the united kingdom (uk) with a few additional countries represented by other (figure 2).     figure 1 type of institution where searcher was employed (n= 17).   figure 2 country where searcher was employed (n= 17).   figures 3 and 4 show that most study participants had greater than 10 years of experience in their profession and at least 5 years of experience in assisting in systematic reviews. the number of systematic review searches that respondents had contributed to ranged from 0 to greater than 50 (figure 5).     figure 3 searchers’ years of experience in profession (n= 17).     systematic review characteristics   half of the searchers responded that they would be a co-author on the systematic review that they completed the searches for, while the others responded they would not (33%) or were not sure (17%).   with regards to study population, the survey asked about age of the target population. thirty-nine percent of the systematic reviews focused on adults, 17% pediatric, and 44% focused on both. the breakdown for the methodological focus of the systematic reviews included therapy (33%), diagnosis (11%), prognosis (11%), other (39%), and unsure (6%).    an attempt was made to explore whether any relationships existed between systematic review topic and time spent searching and resources used. however, because of the wide range of topics for a small sample size, we were not able to analyze the data in a meaningful way.   most of the reviews were completed under the guidance of a systematic review producing entity (56%; n= 18). also the majority of systematic reviews were not grant funded (67%) (figure 6).     figure 4 searchers’ years of experience contributing to systematic reviews (n= 17).   figure 5 number of systematic reviews searcher has contributed to in the past (n= 17).   figure 6 grant funded systematic reviews (n= 18).     time spent searching and number of resources searched   tables 1 and 2 show the time survey participants reported searching resources for their systematic review and the mean number of resources used per systematic review.   using the kruskal-wallis test we examined whether the time spent searching resources for a systematic review or the number of resources used for a systematic review search varied by the characteristics of a systematic review (grant funded, under guidance of a systematic review producing entity, etc.) or of an individual searcher (institution, country, systematic review training, etc.).   figures 7-10 use boxplots to visualize the statistically significant (p ≤ 0.05) findings. in the boxplots the thick dark horizontal line represents the median value. the bottom and top of the box are the first (q1) and third (q3) quartiles where 25% of the data is below q1 and 75% is below q3. the “whiskers” extending from the boxes show the spread of the data with outliners represented as dots (as in figures 9 and 10).    the time spent searching can be obtained by exponentiating the log (minutes) in figures 7 and 8. specifically, whether or not the systematic review was grant funded was associated with the amount of time spent searching for both grey literature (median [q1, q3] for grant funded: 544.5 [211.4, 1339.4] minutes and not grant funded: 66.7 [36.6, 109.9] minutes, p=0.03)   and non-grey literature (median [q1, q3] for grant funded: 1480.3 [365.0, 3294.5] minutes and not grant funded: 270.4 [164.0, 492.7] minutes, p=0.05) (figures 7 and 8). the number of resources searched for the systematic reviews varied by the searchers’ institution (median [q1-q3] for academic: 2.5 [2.0, 4.5]; hospital: 1.0  [1.0, 1.3]; and other: 8.5 [7.8, 9.3], p=0.02) and whether the searcher received systematic review training (median [q1-q3] for trained: 3.0 [3.0, 5.0]; and untrained: 6.0 [4.0, 7.0], p=0.045) (figures 9 and 10).   resources searched   figures 11 and 12 show the most common grey and non-grey literature resources reported as being used by the 17 study participants.   discussion   the results of this study found that the average total time spent searching electronic databases and handsearching the literature for a systematic review was 24 hours with a range of 2 to 113 hours with 50% of the participants reporting spending less than 8 hours. our study also more specifically identified time taken to search the grey literature for a systematic review. all systematic reviews reported (n= 18) included some form of grey literature searching. the average time taken to conduct the grey literature search was approximately 7 hours, with range of 20 minutes to 58 hours, with 50% of the participants reporting spending less than 1.5 hours. the grey literature search represents 27% of the total time of the literature search on average, with 50% of the participants spending 20% or less of their time searching grey literature.     table 1 time spent searching resources for systematic review searches     time range (minutes) time mean (minutes) time median (minutes) quartiles (q1,q3) grey literature resources 20-3480 395 85 (45,240) all resources 96-6780 1457 471 (255,2104)   table 2 number of resources utilized for systematic review searches   resources range (number) resources mean (number) resources median (number) quartiles (q1,q3) total resources searched 3-27 9 8 (5,10) grey literature resources searched 1-14 4 2 (1.25,5.5) non-grey literature resources searched 2-13 5 4 (3,5.75)     figure 7 time spent searching grey literature for a systematic review (n= 17).   figure 8 time spent searching non-grey literature for a systematic review (n=17).   figure 9 number of grey literature resources used for a systematic review (n= 18).   figure 10 number of non-grey literature resources used for a systematic review (n= 18).   figure 11 top grey literature resources searched for a systematic review (n= 18).   figure 12 top non-grey literature resources searched for a systematic review (n= 18).     when reporting time spent searching, survey respondents were asked to account for choosing terminology, developing the strategy, refining, and running the search. depending on how the instruction “running the search” was interpreted response times could be skewed low. although not specifically stated, we aimed to have the time requested on the survey to capture the time required for searchers to navigate and learn the nuances of the databases used for grey literature searching as they prepare the search approach. grey literature electronic resources are often limited and crude in search capability. truncation of terms may not be possible, search boxes may be limited in the number of characters they accept, little or no search help documentation may be provided and no export feature may be available. because of these limitations the searcher may spend more time searching the database than anticipated. grey literature resources are also not routinely searched by the average health sciences librarian for everyday work. because of this, database unfamiliarity may also require searchers to spend more time searching the grey literature.    as previously mentioned, all reviews reported included some form of grey literature searching. an average of four grey literature resources were searched per review, with a range of one to 14 resources and with 50% of the participants reporting using one or two grey literature resources. the number of resources selected for the grey literature search may be restricted due to time constraints if there is pressure to complete the systematic review within a short time period. also resource selection may be impacted by resource access which may be limited depending on institutional subscriptions. we did not ask searchers what, if anything, impacted or limited the number of resources selected, especially for grey literature. we did ask if a resource was purchased as a one-time paid subscription for purposes of a search for the systematic review, however no respondents reported this to be the case. locating resources to search for grey literature is important and worth noting as one of the challenges with searching the grey literature. it is possible that problems locating grey literature resources impacted the number of resources used by participants in the study.   as shown in figures 11 and 12, medline and cochrane library were used in all the reviews reported in this study. following the cochrane library, the next most commonly used grey literature resources were the world health organization international clinical trials registry platform (who ictrp) (n= 6) and the metaregister of controlled trials (mrct) (n= 5).    it is not surprising that the cochrane library, produced by the cochrane collaboration, led the ranking of the grey literature resource searched. the cochrane collaboration, which recently celebrated their 20th anniversary is widely known for generating systematic reviews and is respected by both researchers and librarians alike (friedrich, 2013). the cochrane library provides access to the central database, which includes randomized and controlled clinical trials obtained not only from electronic database searches but also from the journal and conference proceeding handsearching efforts of the cochrane collaboration. furthermore, the cochrane library includes access to the hta database and the nhseed which contain indexed reports. . for this reason, the cochrane library was classified as a source of grey literature for the purpose of this study. it should be noted that 56% of the systematic reviews reported in this survey were conducted under the guidance of a systematic review entity. this may include the cochrane collaboration, for which documentation in the editorial policies requires the search of central for all cochrane systematic reviews (chandler, churchill, higgins, lasserson, & tovey, 2013). it is possible that this could have influenced why the cochrane library was found to be the top grey literature resource used. the two grey literature resources that were the next most commonly used were the who ictrp and mrct. both of these resources allow federated searching across multiple trial registries including clinicaltrials.gov. searching such resources would ideally reduce the need to search several trial registries, a possible explanation for their appearance in our top grey literature resources searched list and perhaps why they may have appeared higher on the list compared to clinicaltrials.gov.   also evaluated was whether time spent searching or the number of resources selected varied by searcher or review characteristics. as shown in figures 7 and 8 the amount of time spent searching for both grey and non-grey literature was impacted by whether or not the systematic review was grant funded. more time was spent searching both types of resources if the systematic review was grant funded. the explanation for this finding is unclear. it may be that there are differences as to how funded systematic reviews are conducted compared to non-funded systematic reviews. a 2007 study (reed et al., 2007) on the association of funding and quality of published medical education research found differences in the quality of funded studies. perhaps a funded systematic review has more resources available in terms of manpower and technology because of funding.     another finding was that the number of resources searched was impacted by the searcher characteristics. the number of literature resources varied by institution (academic, hospital or other). when performing all two-way comparisons, we found that those who work in academic settings used fewer grey literature resources than those who work in other settings (p = 0.03) and that those who work in hospitals use fewer numbers of grey literature resources than those who work in other settings (p = 0.049). academicand hospital-affiliated information professionals are often juggling a multitude of responsibilities and therefore may not have as much time to devote to each systematic review search and therefore it is possible that this leads to fewer grey literature resources being used in the search. the number of non-grey literature resources searched also varied by whether the individual received systematic review training. if systematic review training was received, the number of non-grey literature resources was decreased. it is unclear why such a relationship is evident. perhaps those with training can more readily select the pertinent databases for the systematic review topic or feel more confident with being selective. our survey cannot conclude this however. searchers with less training may take precaution and therefore select a large number of resources to search and spend a long time searching. again, there is no certain explanation for our results. many variables may contribute, including whether there is a fair amount of time allotted to locating studies.   limitations   there are several limitations in this study that are worth noting. we utilized convenience sampling, recruiting mainly through known listservs. of the respondents who started the survey, only 21% completed the survey. the reasons for this non-completion are not known, although it is possible that the second part of the survey was viewed as too onerous or that the planned systematic review never progressed to the searching phase. there was limited power in the analysis due to the small sample size, so results should be viewed as suggestive rather than predictive. also, because the small sample size consisted of mainly library professionals, results may not be generalizable to all those undertaking a systematic review search. part two of this survey study utilized prospective methodology, asking participants to record information about their systematic review searching as they worked. perhaps if a retrospective methodology was used more potential participants would have met our inclusion criteria, resulting in a larger sample size. however, it was the feeling of the authors that a prospective survey would allow the least biased capture of time spent searching. a further limitation regarding methodology of the study was the subjective categorization of resources into grey literature or non-grey literature by the authors.   the time for searching that we obtained through this study may be underreported due to the following: five survey participants reported that a portion of the searching was completed by another individual but only reported the name of the resource, not time spent searching. two survey participants did not report the time spent searching each resource individually; they only reported the total time as whole which was spent searching all resources for the review.   future directions   this study draws attention to the need for further research on search methodology in systematic reviews and grey literature. furthermore, there is a need for guidelines on conducting systematic review searches including grey literature searching. some earlier literature demonstrates searchers’ recognition of issues with searching grey literature, and with the approach to searching for comprehensive literature reviews. in a canadian report (dobbins, robeson, jentha & desmeules, 2008), a methodology for the grey literature search to support evidence syntheses in public health was explored. tyndall (2008) opens up discussion of the grey literature search process by suggesting a hierarchy for acceptable grey literature. bidwell‘s cosi model (bidwell & jensen, 2003), a model proposed for searches to support technology assessment reports is an example of an approach that can be revisited to address this matter. the cosi model relates to the whole search process and uses a framework or protocol to categorize resources into three levels of priority based on expectation of yield. the acronym cosi, and levels of priority, include co for core search, s for standard search, and i for ideal search (bidwell & jensen, 2003). if there is sufficient time to undertake the search, the ideal search of resources is proposed. information professionals conducting searches for systematic reviews have little guidance to assist them with the approach and how to conduct the searches. for information specialists embarking on systematic review searches, questions may arise as to which resources to search, how many, how far to go in breadth of resources. perhaps, a similar concept to the cosi model can be used to develop a framework for guidance in grey literature searching in support of systematic reviews.   hopewell, mcdonald, et al. (2007) found that studies included in systematic reviews, located through grey literature searching, were more commonly conference abstracts or unpublished data (trial registers, file drawer data, data from individual trialists). additional types of grey literature identified in the review include: book chapters, unpublished reports, pharmaceutical company data, in press publications, and letters and theses (hopewell, mcdonald, et al., 2007). the study findings could potentially be used to support development of a framework that would prioritize resources by the document types that they include, with the thought that certain types of documents are more often found useful for inclusion into systematic reviews. the aforementioned are just several examples for illustration purposes. a more national/international collaborative effort by an authoritative body would be best to propose systematically developed guidance.   it should be noted that a number of guides exist which list grey literature resources to search and explain the importance of searching grey literature (canadian agency for drugs and technologies in health, 2013; centre for reviews and dissemination, 2009; cochrane collaboration, 2011). many of these guides can be helpful to identify possible resources but some are merely lists, and while some explain the importance of select types of items they do not highlight an approach or framework for following through with the entire grey literature search process. furthermore, only a select few are designed specifically for information professionals.   exploration of the systematic review search process regarding grey literature including time taken to search, resources used, and investigation of any relating characteristics which may impact these factors is one step forward toward engaging the health sciences library community in further discussion about grey literature. many information professionals are multi-tasking, such as is the case with academic health science librarians and hospital librarians, and therefore time management is of great interest in order to efficiently integrate systematic review searching into one’s routine responsibility.   conclusion   we sought to prospectively explore the time taken to conduct grey literature searches, via electronic database searching and handsearching, for systematic reviews and to evaluate whether any relationship exists between searcher and systematic review characteristics. the mean time taken to conduct grey literature searches was approximately 7 hours, with 50% of the searchers reporting less than 1.5 hours spent. this mean time represents 27% of the total time taken to complete the systematic review literature search. time spent searching both the grey and non-grey literature was influenced by whether or not the systematic review was grant funded. the time estimates given in this study are for searching-related activities only and do not include other potential librarian efforts involved in participating in the synthesis of a review such as meetings with the review requestor and the systematic review team and managing citations. however, the time estimate provided for searching both the grey literature and non-grey literature resources can provide direction for librarians when meeting with researchers, writing a grant for a completing a systematic review, or simply managing their own time. the top resources used by the participants in this study might provide a reference point for librarians working on a systematic review.    in light of recently established systematic review standards, we expect some changes in the landscape of systematic review searching. additionally, we hope that in the near future grey literature searching standards for systematic reviews are developed by the librarian community for information professionals. acknowledgements   this project is supported in part by the national institutes of health through grant numbers ul1 rr024153 and ul1tr000005.   we would like to thank dave piper for review of the manuscript.     references   adams, c. e., power, a., frederick, k., & lefebvre, c. (1994). an investigation of the adequacy of medline searches for randomized controlled trials (rcts) of the effects of mental health care. psychological medicine, 24(3), 741-748. doi: http://dx.doi.org/10.1017/s0033291700027896   al-maskari, a., & sanderson, m. (2011). the effect of user characteristics on search effectiveness in information retrieval. information processing & management, 47(5), 719-729. doi: http://dx.doi.org/10.1016/j.ipm.2011.03.002   allen, i. e., & olkin, i. (1999). estimating time to conduct a meta-analysis from number of citations retrieved. journal of the american medical association, 282(7), 634-635.   armstrong, r., jackson, n., doyle, j., waters, e., & howes, f. (2005). it's in your hands: the value of handsearching in conducting systematic reviews of public health interventions. journal of public health, 27(4), 388-391. doi: http://dx.doi.org/10.1093/pubmed/fdi056   bidwell, s., & jensen, m. f. (2003). using a search protocol to identify sources of information: the cosi model. etext on health technology assessment (hta) information resources. in national information center on health services research and health care technology (nichsr). retrieved from http://www.nlm.nih.gov/archive/20060905/nichsr/ehta/chapter3.html   boekhorst, a. k., farace, d. j., & frantzen, j. (2005). grey literature survey 2004: a research project tracking developments in the field of grey literature. in d.j. farace & j. frantzen (eds). proceedings from sixth international conference on grey literature: work on grey in progress, 6-7 december 2004. (pp. 1-9). amsterdam: textrelease.   canadian agency for drugs and technologies in health. (2013). grey matters: a practical deep-web search tool for evidence-based medicine, retrieved from http://www.cadth.ca/media/pdf/grey-matters_a-practical-search-tool-for-evidence-based-medicine.doc      centre for reviews and dissemination. (2009). systematic reviews: crd's guidance for undertaking reviews in healthcare. retrieved from http://www.york.ac.uk/inst/crd/pdf/systematic_reviews.pdf   chandler, j., churchill, r., higgins, j., lasserson, t., & tovey, d. (2013). methodological standards for the conduct of new cochrane intervention reviews. in cochrane editorial unit (ed.) methodological expectations of cochrane intervention reviews (mecir) | retrieved from http://www.editorial-unit.cochrane.org/sites/editorial-unit.cochrane.org/files/uploads/mecir_conduct_standards%202.3%2002122013.pdf   cochrane collaboration. (2011). in j. p. t. higgins & s. green (eds.) cochrane handbook for systematic reviews of interventions, retrieved from http://www.cochrane-handbook.org     cook, d. j., guyatt, g. h., ryan, g., clifton, j., buckingham, l., willan, a., mcilroy, w., oxman, a. d. (1993). should unpublished data be included in meta-analyses? current convictions and controversies. journal of the american medical association, 269(21), 2749-2753. doi: http://dx.doi.org/10.1001/jama.1993.03500210049030     cook, d. j., mulrow, c. d., & haynes, r. b. (1997). systematic reviews: synthesis of best evidence for clinical decisions. annals of internal medicine, 126(5), 376-380. doi: http://dx.doi.org/10.7326/0003-4819-126-5-199703010-00006     cooper, i. d., & crum, j. a. (2013). new activities and changing roles of health sciences librarians: a systematic review, 1990-2012. journal of the medical library association, 101(4), 268-277. doi: http://dx.doi.org/10.3163/1536-5050.101.4.008     croft, a. m., vassallo, d. j., & rowe, m. (1999). handsearching the journal of the royal army medical corps for trials. journal of the royal army medical corps, 145(2), 86-88. doi: dx.doi.org/10.1136/jramc-145-02-09   crum, j. a., & cooper, i. d. (2013). emerging roles for biomedical librarians: a survey of current practice, challenges, and changes. journal of the medical library association, 101(4), 278-286. doi: http://dx.doi.org/10.3163/1536-5050.101.4.009   crumley, e. t., wiebe, n., cramer, k., klassen, t. p., & hartling, l. (2005). which resources should be used to identify rct/ccts for systematic reviews: a systematic review. bmc medical research methodology, 5, 24. doi: http://dx.doi.org/10.1186/1471-2288-5-24   debowski, s. (2001). wrong way: go back! an exploration of novice search behaviours while conducting an information search. the electronic library, 19(6), 371-382.   dobbins, m.,robeson, p,jentha, n., & desmeules , m (2008). grey literature: a methodology for searching the grey literature for effectiveness evidence syntheses related to public health: a report from canada.health inform, 17(1), 9-12.   farace, d. j. (1998). forward. in d.j. farace (ed.).  gl'97 conference proceedings:  perspectives on the design and transfer of scientific and technical information, amsterdam: transatlantic.   fenichel, c. h. (1981). online searching: measures that discriminate among users with different types of experiences. journal of the american society for information science, 32(1), 23-32. doi: http://dx.doi.org/10.1002/asi.4630320104   friedrich, m. j. (2013). the cochrane collaboration turns 20: assessing the evidence to inform clinical care. journal of the american medical association, 309(18), 1881-1882. doi: http://dx.doi.org/10.1001/jama.2013.1827     greenhalgh, t., & peacock, r. (2005). effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. british medical journal, 331(7524), 1064-1065. doi: http://dx.doi.org/10.1136/bmj.38636.593461.68   greynet international. (2013). document types in grey literature. retrieved 07/09/2013, from http://www.greynet.org/greysourceindex/documenttypes.html   guise, j.-m., & viswanathan, m. (2011). overview of best practices in conducting comparative-effectiveness reviews. clinical pharmacology &therapeutics, 90(6), 876-882. doi: http://dx.doi.org/10.1038/clpt.2011.239   hopewell, s., clarke, m.j., lefebvre, c., & scherer, r.w. (2007). handsearching versus electronic searching to identify reports of randomized trials. in cochrane database of systematic reviews, mr000001. doi: http://dx.doi.org/10.1002/14651858.mr000001.pub2   hopewell, s., clarke, m.j., stewart, l., & tierney, j. (2007). time to publication for results of clinical trials. in cochrane database of systematic reviews, mr000011. doi: http://dx.doi.org/10.1002/14651858.mr000011.pub2   hopewell, s., mcdonald, s., clarke, m.j., & egger, m. (2007). grey literature in meta-analyses of randomized trials of health care interventions. in cochrane database of systematic reviews, mr000010. doi: http://dx.doi.org/10.1002/14651858.mr000010.pub3   hsieh-yee, i. (1993). effects of search experience and subject knowledge on the search tactics of novice and experienced searchers. journal of the american society for information science, 44(3), 161-174. doi: http://dx.doi.org/10.1002/(sici)1097-4571(199304)44:3<161::aid-asi5>3.0.co;2-8   institute of medicine. (2011). finding what works in health care : standards for systematic reviews. washington dc: national academies pressretrieved from www.nap.edu/openbook.php?record_id=13059     jadad, a. r., carroll, d., moore, a., & mcquay, h. (1996). developing a database of published reports of randomised clinical trials in pain research. pain, 66(2-3), 239-246.   kuhlthau, c. c. (1999). the role of experience in the information search process of an early career information worker: perceptions of uncertainty, complexity, construction, and sources. journal of the american society for information science, 50(5), 399-412. doi: http://dx.doi.org/10.1002/(sici)1097-4571(1999)50:5<399::aid-asi3>3.0.co;2-l   reed, d. a., cook, d. a., beckman, t. j., levine, r. b., kern, d. e., & wright, s. m. (2007). association between funding and quality of published medical education research. journal of the american medical association, 298(9), 1002-1009. doi: http://dx.doi.org/10.1001/jama.298.9.1002   riaz, m., sulayman, m., salleh, n., & mendes, e. (2010). experiences conducting systematic reviews from novices' perspective. in m. turner & m. niazi (eds.) proceedings of the 14th international conference on evaluation and assessment in software engineering. (pp. 44-53). swinton: british computer society.   savoie, i., helmer, d., green, c. j., & kazanjian, a. (2003). beyond medline: reducing bias through extended systematic review search. international journal of technology assessment in health care, 19(1), 168-178. doi: http://dx.doi.org/10.1017/s0266462303000163   steinberg, k. k., smith, s. j., stroup, d. f., olkin, i., lee, n. c., williamson, g. d., & thacker, s. b. (1997). comparison of effect estimates from a meta-analysis of summary data from published studies and from a meta-analysis using individual patient data for ovarian cancer studies. american journal of epidemiology, 145(10), 917-925.   tabatabai, d., & shore, b. m. (2005). how experts and novices search the web. library & information science research, 27(2), 222-248. doi: http://dx.doi.org/10.1016/j.lisr.2005.01.005   tetzlaff, j., moher, d., pham, b., & altman, d. (2006, october). survey of views on including grey literature in systematic reviews. xiv cochrane colloquium, dublin, ireland.   tyndall, j. (2008, september). how low can you go? toward a hierarchy of grey literature. dreaming08: australian library and information association biennial conference, alice springs, australia.   wright, j. m., cottrell, d. j., & mir, g. (2014). searching for religion and mental health studies required health, social science, and grey literature databases. journal of clinical epidemiology, 67(7), 800-810. doi: http://dx.doi.org/10.1016/j.jclinepi.2014.02.017   appendix   survey -part one   contact information solely used for communication with you regarding the study first name last name institution/organization e-mail phone   information about the searcher   1.       country of employment   2.       list academic credentials (e.g., phd, mlis, mph)   3.       type of institution   ·         academic ·         hospital ·         industry/corporation ·         government agency ·         organization/agency (non-government) ·         other (if other, specify)   4.       position title   5.       select an option which represents your primary professional role   librarian/information professional researcher healthcare professional statistician student other (if other, specify)   6.       years of experience in your profession (that you selected in question 5)?   7.       have you had any formal training on how to conduct literature searching for systematic reviews? yes/no   8.       approximately how many years of experience do you have contributing to systematic reviews?   9.       approximately how many systematic reviews have you contributed to in the past?   survey -part two   information about the systematic review (sr)   1.       please list the title or the topic of the sr. notethis information will only be used to classify the srs into subject categories.   2.       please select the population age group included in the sr.   adults pediatrics both   3.       which category best describes the type of systematic review?   therapy diagnosis prognosis etiology adverse effects methodology other not sure   4.       is this sr grant funded? yes/no   5.       is this sr being produced under the guidance of a systematic review funding or producing agency/organization? yes/no/not sure   6.       should the sr be published, are there plans for you to be a coauthor? yes/no/not sure   7.       if you are not a librarian/information professional, is there one involved in the sr? if you are a librarian/information professional select n/a yes/no/ n/a if yes, in what primary role?   manage project overall search responsibility assist with search strategy design suggest resources provide general guidance assist with full text acquisition other   information about searching for the systematic review (sr)   8.       is cited reference searching (checking who cited a paper) or checking the reference lists of papers of interest, planned as part of the search effort for this sr?   cited reference  searching checking reference lists both none   9.       will a methodology filter be used in the sr search? (a methodology filter is a pre-designed search strategy with terms related to research methodology. examples include the cochrane highly sensitive search strategy and clinical queries.) yes/no if yes, for which resources?   10.    were any resources purchased as a one-time paid subscription for purposes of a search for the sr? yes/no   11.    may we contact you to ask for the final number of studies included and where the citations were originally found? yes/no   12.    use the following table to document all resources searched for the sr.   if any journals or conference proceedings were handsearched, label with handsearch followed by a dash and the title of the resource in the resource name column. indicate in resource platform column whether the print or electronic version was handsearched   if a resource was purchased as a one-time paid subscription for purposes of a search for this sr, please mark with an asterisk (*) preceding the resource name in the resource name column.   use n/a if an item does not apply, to indicate that you are unable to identify the information, or if the information is not available   three examples are provided, although the time is not documented   another1 date searched resource name resource platform/ interface/ vendor resource url time in minutes2   1/22/2010   * biosis previews dialog   n/a insert time here   2/10/2010   clinicaltrials.gov n/a clinicaltrials.gov   insert time here x 1/4/20103/1/2010   handsearchproceedings of the nutritional society (2007-2009)   print   n/a   insert time here                                                                                       1 mark with an x resources you did not search yourself but that were searched for the sr 2 time should account for choosing terminology, developing the strategy, and running the search   evidence summary   lis periodicals contain a low percentage of articles that qualify as research   a review of: turcios, m. e., agarwal, n. k., & watkins, l. (2014). how much of library and information science literature qualifies as research? journal of academic librarianship, 40(5), 473-479. http://dx.doi.org/10.1016/j.acalib.2014.06.003   reviewed by: richard hayman assistant professor & digital initiatives librarian mount royal university calgary, alberta, canada email: rhayman@mtroyal.ca   received: 29 apr. 2015  accepted: 17 sept. 2015      2015 hayman. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract    objective – to determine how much of the literature in a library and information science (lis) periodical collection qualifies as research.   design – content analysis.   setting – the lis periodicals collection of an academic library that supports an established lis graduate program at a college in the united states of america.   subjects – of the 177 identified periodicals with lis content that fell within project scope from the local collection, researchers analyzed 101 journals that include academic/scholarly content and an additional 4 journals with relevant trade content. this study excluded open access (oa) journals.   methods – using the most recent issue of each subject journal from the fiscal year 2012-2013, the authors performed a content analysis on all indexed content items, and then classified each content item as research or non-research. for content identified as research, researchers identified the research method (or methods) used. the data collection tool also captured identifying information and keywords for all content.   main results – within the journals meeting the scope of this study, researchers identified 1,880 articles from 105 individual journal issues. only 16% (n=307) of articles met the authors’ established definition to qualify as research. within the subset of research articles, the authors further identified 45% (n=139) that used a single research method. an additional 36% (n=112) of identified research articles used two research methods and 15% (n=46) used three methods, with the remainder using four or more methods.   surveys were the most frequently used research method, accounting for 49% (n=66) of the single method studies. the researchers discovered that surveys remained popular even in mixed-method studies, with 21% (n=117) of all identified research articles using surveys. this is closely followed by 20% (n=109) of studies reported as using the general category of “other” methods, for research that did not meet one of the predefined methods. the next two most popular identified methods were case studies at 13% (n=73), followed by content analyses at 13% (n=71). for the eight other research methods identified, none saw a frequency above 10%. focus groups and usability studies tied for the least frequently used method among the 307 articles, both at 2% (n=9).   the keyword analysis focused on two categories, one for research article keywords and another for non-research article keywords, for all 1,880 articles identified. non-research articles had less reliance on keywords, with authors reporting keywords appearing on 73% (n=1156). within these, authors discovered 120 separate keywords used 10 or more times across non-research articles. the top ten keywords among non-research articles were reported as primarily related to books and publishing, with “non-fiction,” “adult,” and “libraries” as the top three. by comparison, research articles heavily favour the use of keywords, with 94% (n=290) of research articles having keywords. analysis of the individual keywords found 56 keywords appearing 10 or more times across research articles. the top ten keywords are primarily practice related, with “information,” “libraries,” and “library” being the top three. when comparing shared keywords across both categories, the same top three keywords reported for research in the previous sentence apply to the collective set.   conclusion – the authors note that the nature and size of the local collection both benefited and limited this study. compiling and maintaining a comprehensive list of lis periodicals is a challenging task across a large body of potential sources. within the resulting periodicals studied, a mere 16% of analyzed lis literature met the criteria to qualify as research, and that only after the study had eliminated virtually all trade periodicals from the population. had that trade literature been included, the percentage qualifying as research would have been even lower. the popularity of surveys as a research method among lis research reflects other recent findings, though the frequency of studies falling into the general “other” category suggests that lis research is changing. based on this research, the authors conclude that there is still much to be learned from content analysis of literature published in lis periodicals. future analyses could further examine the frequency of research methods used within lis research.   commentary   this study identified trends within lis literature, identifying a number of lis journals that contain academic/scholarly content as a starting place. by examining content from single periodical issues, the authors achieved a broad, though not deep, analysis of the quantity of lis research as compared to all lis literature. a recent comparative example is chu’s (2015) content analysis study that has depth but not breadth, in that it focuses on just 3 lis journals, analyzing 1,162 research articles published over a 10-year period.   for researchers interested in replicating this study or employing similar methods for other collections, the authors provide their list of lis periodicals that include academic content. they also include an example data collection form alongside a coding document that defines different research methods. together these aid the reader in understanding what content is included in the analysis. something missing from this documentation and the report is a full explanation of the “other” category of research methods. given the frequency of these undefined methods, across 20% of all research articles, this study would be strengthened by addressing the various approaches captured in this category. the authors recognize that more detail is required and suggest this as an avenue for further studies. in this reviewer’s opinion the keyword analysis yields little of use to practitioners. the top results within research articles (“information,” “libraries,” and “library”) and within non-research articles (“non-fiction,” “adult,” and “libraries”) are all so generic that they are of little use for comprehensive topic searching.   the study population is tied to a specific library collection that supports an lis graduate program. the authors infer that the sample population is representative of lis literature and conclude that their analysis provides “vital statistical data pertaining to the current state of lis research and periodicals” (p. 479). yet the authors acknowledge that their sample population focuses on subscription periodicals and does not include open access journals, and this represents a significant lost opportunity. among others, yuan and hua (2011) have illustrated that lis has fully adopted oa literature as mainstream, so the exclusion of non-subscription content excludes a relevant body of literature from the study population.    ultimately the low frequency of research in the literature, just 16%, might seem alarming. based on the research presented here it is possible to infer that the body of lis literature is crowded with non-research articles, suggesting that there are opportunities for journals to adjust the balance of research versus non-research content. however, we must consider that the authors examined all indexed content within the periodicals under consideration, without accounting for whether the content itself was presented as research. many publications include non-research content in formats that are indexed: editorials, reviews, commentaries, interviews, and even scholarly evidence summaries are treated as indexed content, but they are not written as, nor typically presented as, original research. a fundamental question, and one needing more study, would be to ask: how much of lis literature presented as research actually qualifies as research? further research might help us understand whether journal editors need to prioritize and include more research content in their publications, and also whether the lis profession itself should be responsible for generating more research overall.   references   chu, h. (2015). research methods in library and information science: a content analysis. library & information science research, 37(1), 36-41. http://dx.doi.org/10.1016/j.lisr.2014.09.003   yuan, s., & hua, w. (2011). scholarly impact measurements of lis open access journals: based on citations and links. the electronic library, 29(5), 682-697. http://dx.doi.org/10.1108/02640471111177107 news/announcements   call for applications: institute for research design in librarianship (irdl) summer workshop      2014. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   “irdl not only set me on the path to producing and properly documenting original research, but it set me up with a network of like-minded librarians who will help me achieve my goals.” — 2014 irdl scholar   call for applications for the irdl 2015. the second summer workshop will be held on july 13-23, 2015, with arrival on campus on sunday, july 12, and departure on friday, july 24.   we are seeking novice librarian researchers who are employed by academic libraries or research libraries outside an academic setting in the united states to participate in the institute. novice researchers typically may have conducted research but have not yet had a peer-reviewed article published as the primary author or had an individual presentation accepted by a peer-reviewed conference. we define “novice” broadly; if you feel that you would benefit from being guided throughout the entire research design process, we encourage your application. librarians of all levels of professional experience are welcome to apply for the scholarship which includes, instruction in research design and a full year of peer/mentor support to complete a research project at their home institutions; the learning experience, travel to and from los angeles, ca, accommodations, and food will be supplied to scholars free of charge.   we seek librarians with a passion for research and a desire to improve their research skills. irdl is designed to bring together all that the literature tells us about the necessary conditions for librarians to conduct valid and reliable research in an institutional setting. the cohort will be chosen from a selective submission process, with an emphasis on enthusiasm for research and diversity from a variety of perspectives, including ethnicity and type and size of library.   the advisory board, comprised of librarians with research experience in a variety of settings, will use an open review process to select participants who will travel to los angeles, california, participate in all institute activities, and conduct a research project during the 2015-2016 academic year. the online application requires a proposed research project that will be revised at the workshop and will be the basis for the study to be completed in the coming year.   selection criteria:   commitment to the year-long process of communicating with other participants and conducting the proposed study; significance of the research problem to the operational success of the applicant’s library or to the profession of librarianship; thoughtfulness, thoroughness, and clarity of the research proposal; enthusiasm for research and a desire to learn.   we will be accepting applications from december 1, 2014 to february 1, 2015.   scholars accepted to the institute will be notified in mid-april, 2015.   please contact project directors with any questions about the institute or the application process: kristine brancolini, dean of the library, loyola marymount university: brancoli@lmu.edu   marie kennedy, serials & electronic resources librarian, loyola marymount university: marie.kennedy@lmu.edu   for more information about the project, including the project partners, the san josé state university school of information and the statewide california electronic library consortium (scelc), please see the project website: http://irdlonline.org.   news/announcements   announcing ishimr 2015!      2014. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   the organizing committee of the ishimr 2015 committee are pleased to announce the 17th international symposium for health information management research (ishimr), which will take place on 25-26 june 2015, at york st john university, in york, england, uk. ishimr 2015 is a collaboration between york st john university and the university of sheffield. ishimr 2015 will bring together researchers and professionals in the general area of e-health, health information management and librarianship, and health informatics and will provide a forum for the presentation and discussion of their research activities.   ishimr 2015 will feature oral and poster presentations and a doctoral symposium.  we are also delighted to announce that our keynote speakers will be professor frances mair from the university of glasgow, and dr andrew booth from scharr at the university of sheffield. ishimr 2015 will include a multi-disciplinary audience and the opportunity for discussions and research collaborations between academia, government and industry.   please see the ishimr 2015 web-site (www.ishimr2015.com) for a list of themes and topics the conference covers.   we invite you to consider submitting your research contributions or practical experience reports to the conference when the 1st call for papers is made in september 2014. all papers will be blind peer-reviewed by at least two members of the programme committee.  the deadline for submission of papers and poster abstracts is: 15th january 2015.   all papers presented and abstracts for posters will be published in the conference proceedings. we expect the best papers to be published in a special issue of an international journal. please circulate this announcement to your colleagues and fellow researchers. further information about ishimr 2015 is available at www.ishimr2015.com.   on behalf of the organising committee, we look forward to seeing you at ishimr 2015. dr. hannah spring dr. barbara sen professor peter bath (conference chairs)   article   identifying the visible minority librarians in canada: a national survey   maha kumaran liaison librarian leslie and irene dubé health sciences library university of saskatchewan saskatoon, saskatchewan, canada email: maha.kumaran@usask.ca   heather cai information technology services librarian mcgill university library montréal, quebec, canada email: heather.cai@mcgill.ca   received: 30 oct. 2014    accepted: 8 apr. 2015          2015 kumaran and cai. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – this paper is based on a national survey conducted in late 2013 by the authors, then co-moderators of the visible minority librarians of canada (vimloc) network of the canadian library association (cla). it is a first survey of its kind, aiming to capture a snapshot of the demographics of the visible minority librarians working in canadian institutions. the authors hoped that the data collected from the survey and the analysis presented in this paper would help identify the needs, challenges and barriers of this group of librarians and set future directions for vimloc. the authors also hoped that the findings would be useful to library administrators, librarians, and researchers working on multicultural issues, diversity, recruitment and retention, leadership, library management, and other related areas.   methods – an online survey questionnaire was created and the survey invitation was sent to visible minority librarians through relevant library association electronic mail lists and posted on vimloc’s electronic mail list and website. the survey consisted of 12 questions: multiple-choice, yes/no questions, and open-ended. the survey asked if the participants were visible minority librarians. if they responded “no,” the survey closed for them. respondents who did not identify themselves as minority librarians were excluded from completing the survey.   results – of the 192 individuals that attempted, 120 who identified themselves as visible minority librarians completed the survey. of these, 36% identified themselves as chinese, followed by south asian (20%) and black (12%). there were 63% who identified themselves as first generation visible minorities and 28% who identified themselves as second generation. a total of 84% completed their library degree in canada. equal numbers (38% each) identified themselves as working in public and academic libraries, followed by 15% in special libraries. although they are spread out all over canada and beyond, a vast majority of them are in british columbia (40%) and ontario (26%). there were 38% who identified themselves as reference/information services librarians, followed by “other” (18%) and “liaison librarian” (17%). a total of 82% responded that they worked full time. the open-ended question at the end of the survey was answered by 42.5% of the respondents, with responses falling within the following broad themes: jobs, mentorship, professional development courses, workplace issues, general barriers, and success stories.   conclusions – there are at least 120 first, second, and other generation minority librarians working in (or for) canadian institutions across the country and beyond. they work in different kinds of libraries, are spread out all over canada, and have had their library education in various countries or in canada. they need a forum to discuss their issues and to have networking opportunities, and a mentorship program to seek advice from other librarians with similar backgrounds who have been in similar situations to themselves when finding jobs or re-pursuing their professional library degrees. getting support from and working collaboratively with cla, vimloc can be proactive in helping this group of visible minority librarians. introduction   in december 2011, the visible minority librarians of canada (vimloc) network (http://vimloc.wordpress.com/) was established through the canadian library association (cla). the focus of the network was to create a forum for visible minority librarians in canada. in january 2013, vimloc was invited to participate in a panel presentation at the ontario library association (ola) super conference. at the presentation, vimloc sought ideas from the attendees on future directions for vimloc. based on the feedback, two future directions were identified and added to vimloc’s agenda: that vimloc 1) gather statistical information of visible minority librarians working in canadian institutions; and 2) create a mentorship program for this group. in december 2013, the authors, then co-moderators of vimloc, distributed an online survey to gather statistical information on visible minority librarians at canadian institutions both in and outside of canada.    the principal aim of the survey was to collect foundational data on the number of visible minority librarians working in canadian institutions. the authors hoped that any additional information collected through this survey would help vimloc identify the needs, challenges and barriers of this group of librarians and set future directions for vimloc.    this is the first survey of its kind in canada designed to learn directly from visible minority groups for such information as:   which ethnic groups these librarians belong to whether they were first or second generation canadians their educational qualifications and experiences the types of institutions in which they are currently employed the types of positions they hold, and, whether they are employed full-time or part-time.    the authors hope that the quantitative data and the qualitative analysis of the survey presented in this paper will be useful to library administrators, librarians, and researchers working on multicultural issues, diversity, recruitment and retention, leadership, library management, and other related areas.   research background   the canadian employment equity act (government of canada, 2014) uses the term “visible minorities”  and defines them as “persons, other than aboriginal peoples, who are non-caucasian in race or non-white in colour” (statistics canada, 2012a). statistics canada’s 2011 national household survey (2013) states that there are 13 categories that make up the visible minority variable chinese, south asian, black, arab, west asian, filipino, southeast asian, latin american, japanese, korean, visible minority not included elsewhere, multiple visible minority, and not a visible minority. south asian includes east indian (from india), pakistani, and sri lankan. southeast asian includes vietnamese, cambodian, malaysian, laotian, and others, and west asian includes iranian, afghan, and others (statistics canada, 2013).   the immigration and ethnocultural diversity in canada report (statistics canada, 2014) states that 20.6% of the total population is foreign born and that 19.1% are visible minorities. the three largest minority groups are south asians, chinese, and blacks. statistics canada also projects that close to 30% of the nation’s population will consist of visible minorities by 2031, including both foreign-born and canada-born visible minorities (statistics canada, 2010). does our profession reflect this current demographic? the 8rs study (ingles et al., 2005) states that visible minorities “are under-represented across all types of libraries” and make up “only 7% of the library professional librarian labour force” (compared to 14% of canada’s entire labour force).   it was vimloc’s intention to learn directly from visible minority librarians first, about the number of them working in canadian institutions and second, through an open-ended question, their professional needs, challenges, and barriers, so that vimloc can partner with cla and other interested entities and collaborate to assess, plan, and implement the needed frameworks to work with these librarians and address their concerns.    knowing the number of librarians and their needs will help vimloc create pathways and partnerships to address their needs, find ways to eliminate barriers, provide ways to network, and create a mentorship program. gathering statistical evidence was the first step towards achieving vimloc’s aims to help visible minority librarians receive professional support and network. armed with this data, vimloc representatives can make a case to libraries and library schools in canada to work with vimloc to design and initiate programs to help visible minority librarians achieve ala accreditation, progress faster in their careers, and become employable sooner, and also work with professional organizations to request funding to support their education and enable them to attend conferences. if vimloc continues gathering these statistics every 3-5 years, the change in the data will help them understand the rate at which this population number is changing, the reasons for such a change, the immigration patterns of this group (where they come from), and how vimloc can help or support this particular group.   literature review   a literature review was conducted to ascertain if there were any similar studies done among visible minority librarians. none were found. the literature review was expanded to find any material on visible minority librarians and their experiences. there is a dearth of literature in canada that focuses on the challenges and barriers faced and overcome by visible minority librarians. a search in library and information science abstracts (lisa) and library literature and information science full text & retrospective limited to canadian publications retrieves very few results. there are career and demographic profiles of librarians that also feature minority librarian data (fox, 2007; ingles et al., 2005; mckenna, 2007), papers on the importance of diversity among canadian librarians (kandiuk, 2014; leong, 2013), papers on multicultural populations, services, and collections (berry, 2008; chilana, 2001; cho & con, 2012; dilevko & dali, 2002; kumaran & salt, 2010; paola picco, 2008), and articles on information seeking behaviour or information practices of immigrants (caidi, 2008; caidi & macdonald, 2008; hakim silvio, 2006) and reading practices of canadian immigrants (dali, 2012).   there are very few scholarly works that speak directly about librarians who are visible minorities and their experiences as canadian librarians, except for the recent publications on leadership (or lack thereof) among minority librarians (kumaran, 2012) and the collection of chapters co-edited by lee and kumaran (2014), which includes papers on experiences of going through the tenure process (majekodunmi, 2014); struggles and success stories of assimilating themselves into the canadian library system (dakshinamurti, 2014; lau, 2014; li, 2014; maestro, 2014; shrivatsava, 2014); challenges specifically as chinese born or chinese-canadian librarians (li, 2014; cho, 2014); and struggles as new immigrants in this new country (gupta, 2014; kumaran, 2014).   a major challenge in finding and accessing information about these librarians lies in the fact that there is no universally accepted terminology or subject heading by which these librarians can be identified (aspinall, 2002). various terms such as “ethnic minorities,” “visible minorities,” “diverse librarians,” and “librarians of colour” are used to identify this group of librarians. adding to the confusion is the united nations’ request to canada to not use the term “visible minorities” as it is considered racist terminology (government of canada, 2011; national post, 2007).    lack of literature on and by minority librarians in canada themselves could be due to many reasons: they are in positions that do not require them to publish; lack of training in writing academic papers, especially if they are first generation minority librarians; lack of support for writing for publication; lack of time or funding; not having a dedicated canadian library journal that allows them to voice their thoughts; and perhaps fear of bringing attention to themselves by expressing their opinions.    methodology   an online survey questionnaire was created using fluidsurvey (see appendix 1: questionnaire). after ethics approvals from the authors’ respective institutions, the survey was made available between december 9, 2013 and january 31, 2014. it was a nation-wide survey with participation from visible minority librarians working in canadian institutions both within and outside of canada. the online survey invitation was sent to visible minority librarians through relevant library association electronic mail lists, such as the canadian library association, canadian medical libraries interest group, and special libraries association. the invitation was also posted on vimloc’s electronic mail list and website.    the survey consisted of 12 questions: multiple-choice, yes/no, and open-ended. the survey provided a definition of visible minorities as defined by the canadian employment equity act (government of canada, 2014) and asked if the participant was a visible minority librarian. if the response was “no,” the survey closed. respondents who did not identify themselves as minority librarians were excluded from completing the survey. the rest of the survey was divided into personal and professional questions.    results   of the 192 individuals that attempted, 120 who identified themselves as visible minority librarians were permitted by the system to complete the survey.   once participants identified themselves as visible minorities, they were asked to identify which ethnic groups they belong to or which group fits them best, with options provided. over 36% of the respondents identified themselves as chinese, followed by south asian and black. while many of the respondents for the “other” category identified themselves as “mixed race,” one respondent identified as “canadian.” it is possible that this respondent is a second or third generation canadian and does not identify with any other groups offered as options.   question: what group do you belong to or which group fits you the best?   participants were also asked if they were first or second generation canadians. (first generation visible minority would mean that they were born elsewhere and moved to canada at some point during their lives. second generation would mean they were born in canada to immigrant parents.) of the librarians who completed the survey, 63% identified themselves as first generation visible minorities and 28% identified themselves as second generation. another 9% of them identified themselves under the “other” category. comments from “other” included “one parent was an immigrant to canada,” “third generation,” “yonsei” (fourth generation japanese canadian) and “us permanent resident born to second generation americans.”     table 1 ethnic groups chinese 36% 43 south asian (includes bangladeshi, indian, sri lankan) 20% 24 black 12% 15 filipino 3% 4 latin american 1% 1 south east asian (includes vietnamese, cambodian, malaysian and laotian) 2% 3 arab (includes egyptian, kuwaiti and libyan) 2% 2 west asian (includes afghan, assyrian and iranian) 5% 6 korean 1% 1 japanese 6% 7 other, please specify... 12% 14     table 2 generation first generation 63% 76 second generation 28% 33 other 9% 11     table 3 where library degree was received     space was provided for respondents to provide further information if they wanted to: the year in which they came to canada or their age when they came to canada. there were a few who came to canada in or after 2000 (n=7), a few who came in the 70s, 80s and 90s (n=7), some who came before they were 11 years old (n=6), and the rest identified themselves as second, third or fourth generation canadians.   the next questions focused on educational and professional accomplishments and status, particularly where participants received their professional library degree; whether it was ala accredited; the province where they currently work; number of hours they work per week; and the type of institution where they work.   when asked if they completed their library degree in canada, 84% responded “yes” and 19% responded “no.” when asked to reveal from which university or college they received their library professional degree, only 101 out of 120 responded. it is possible that some respondents were still in library school or simply did not want to identify where they received their degrees for unknown reasons. almost 40% of the 101 respondents stated that they had graduated from the university of british columbia, vancouver, british columbia, followed by the university of western ontario, london, ontario, at 31%. of the 19 respondents who indicated that they had foreign degrees, 7 respondents had received their degrees in africa and asia (outside of united kingdom and united states), but only 3 identified their degrees as non-ala accredited.   when asked what types of library they were employed in, an equal number of respondents identified themselves as working in public and academic libraries (38% each), followed by special libraries (15%). those who chose “other” identified themselves as “unemployed,” “library consultant with provincial government,” “federal agency,” or “federal government.”   the authors wanted to know how widely these minority librarians were spread across canada and beyond. the table below indicates that although they are spread out all over canada and beyond, a vast majority of them are in british columbia and ontario.    this is not surprising because these two provinces have the highest numbers of immigrant populations, as illustrated in the table below (statistics canada, 2009).   the authors used a list of job categories from the american library association’s (ala) website (accessible only to its members) with some modifications for the creation of the next question about types of jobs in which these librarians were employed. when asked to identify themselves to the closest job  category(ies), 38 % identified themselves as reference/information services librarians, followed by “other” (18%) and “administration” (15%). the authors did not ask what “other” meant in this category and it is also not clear what types of administrative positions these respondents hold.     table 4 employment by library type public library 38% 46 regional library 1% 1 academic library 38% 45 special library 15% 18 school library 1% 1 in library school 2% 2 other please specify 6% 7     table 5 geographic distribution alberta 8% 10 british columbia 40% 48 manitoba 6% 7 new brunswick 2% 2 newfoundland and labrador 1% 1 nova scotia 5% 6 nunavut 1% 1 ontario 27% 32 quebec 4% 5 saskatchewan 6% 7 other (if you are working for a canadian library outside of the country) 1% 1     table 6 visible minority population, by province and territory (2006 census)   canada que. ont. man. sask.   alta. b.c. total population 31,241,030 7,435,900 12,028,895 1,133,515 953,850   3,256,355 4,074,380 total visible minority population 5,068,095 654,350 2,745,205 109,100 33,895   454,200 1,008,855 south asian 1,262,865 72,850 794,170 16,565 5,130   103,885 262,290 chinese 1,216,565 79,825 576,980 13,705 9,505   120,270 407,225 black 783,795 188,070 473,765 15,660 5,090   47,075 28,315 filipino 410,700 24,200 203,220 37,785 3,770   51,090 88,075 latin american 304,245 89,510 147,135 6,275 2,520   27,265 28,960 arab 265,550 109,020 111,405 2,320 1,710   26,180 8,635 southeast asian 239,935 50,460 110,045 5,670 2,555   28,605 40,690 west asian 156,695 16,115 96,615 1,960 1,020   9,655 29,810 korean 141,890 5,310 69,540 2,190 735   12,045 50,490 multiple visible minority 133,120 11,310 77,405 3,265 810   13,250 25,415 japanese 81,300 3,540 28,080 2,010 645   11,030 35,060 visible minority, not included elsewhere 71,420 4,155 56,845 1,690 405   3,850 3,880     when asked if they worked part-time (less than 30 hours/week), full-time (30 or more hours/week) or casual hours, an overwhelming 82% responded that they worked full time. the number of hours for part-time and full-time were derived from statistics canada (2012b).   those who chose “other” responded that they were “unemployed,” “volunteering,” “part time contract,” or “casual.”   at the panel presentation at the ontario library association super conference, attending visible minority librarians mentioned that they faced various challenges. in light of this, our final question was an open-ended question that asked them to provide information on anything they deemed relevant to visible minority librarians. of the 120 participants, 50 (47%) responded to this question.   the open-ended question at the end of the survey helped elicit more information on these challenges. the question was “please use the box below to comment on anything else topics could be on the challenges of finding the right job, the need for another degree, lack of support through mentorship or networking possibilities, etc.” below is a list of selected open-ended responses:   “networking and mentoring among minorities is lacking in the field but something is needed.”   “there is definitely a lack of support and lack of any access to information on how to succeed as a visible minority librarian in canada. am glad that vimloc is changing that.”     table 7 type of work     table 8 hours worked part time   9% 11 full time 82% 99 casual hours 5% 6 other 3% 4     “looking forward to a time when upper management reflects the diversity of canada.”   “there is definite lack of mentoring for new and early career librarians, for both those of minority status and non-minorities.”   “i think that having a professional group within cla dedicated to visible minorities is quite important as it gives us an opportunity to discuss issues not often confronted by our majority counterparts. “   “lack of cultural diversity and inclusive practices in the workplace, particularly at the leadership levels.”   “no programs encouraging/assisting minorities in library studies.”   “lack of leadership opportunities.”   the challenge of moving up the career ladder.”   “i think the professional as a whole lacks visible minority presence at all levels of the organization.”   “my working experience in canada has been rewarding. there was great support but i would certainly advocate for a stronger presence of minority librarians in all aspects of the library communities in canada.”   “it's important for librarians new to canada to learn the culture and norms here. their experience and qualifications do not always translate easily and the onus is on them to get up to speed. employers should also recognize talent and potential and be willing to take a chance on librarians new to the country.”   “lack of experience and opportunities to gain library experience is one of the biggest challenges.”   some of the broad themes that evolved through the qualitative analysis of the open-ended question are: jobs, mentorship, professional development courses, workplace issues, general barriers, and success stories.    in the job category, respondents stated that they don’t believe visible minorities are being hired in considerable numbers in spite of being encouraged to apply. some have problems designing their cv or resume according to canadian standards. finding the first job was often the biggest challenge. when speaking of their own workplaces, there were positive and negative comments. there was mention of lack of diversity at their own workplaces, where subtle ways of stereotyping and alienating exist. they speak of being passed over for promotions and being told that they do not have canadian work experience. there are cases where patrons ignore the visible minority librarian and seek out a caucasian librarian. there are others who have not experienced any of these issues and continue to have successful careers.   in the mentorship category, responses reiterated the lack of mentorship and networking opportunities with other minorities in canada. participants stated that they use the american networking opportunities.   there were suggestions that visible minority librarians take courses on client engagement, customer service skills, writing reports, special collections (unique language collections), project management, social media tools, human resources, and budgeting or financial management.    minority librarians in managerial or supervisory roles have identified some issues they perceive as barriers among other minority librarians in order to succeed in their careers. the barriers they have observed among fellow minority librarians are lack of communication skills, lack of customer service skills, and lack of knowledge of the canadian work environment. it was also noted that some of these minority librarians are not willing to accept feedback.    other barriers that respondents noted are lack of an initiative to recruit more minority students at library schools, evaluation of credentials, lack of role models, and lack of leadership courses for minorities.    key findings and conclusions   this is a landmark study capturing a snapshot of the demographics of visible minority librarians working for canadian institutions. the authors hope that the findings of this study will help set future strategic directions of vimloc.    as evidence from our survey suggests, there are at least 120 first, second and other generation minority librarians working in (or for) canadian institutions across the country and beyond. they work in different kinds of libraries, are spread out all over canada, and have had their library education in various countries or in canada. many of the respondents have reiterated what vimloc learned at ola in 2013 – that they need a forum to discuss their issues, a mentorship program to seek advice from other librarians with similar backgrounds who have been in similar situations to themselves when finding jobs or re-pursuing their professional library degrees, and to have networking opportunities.   it was interesting to note that a majority of the respondents graduated from either the university of british columbia (ubc) or the university of western ontario (uwo). there may be many reasons for this: more immigrants living in these two provinces, easier admission requirements, larger cohort sizes, higher intake per year, the option to earn a degree faster through a one year program, and others. it would be beneficial for vimloc to explore this further in future iterations of a similar survey to see if there is consistency in this data. if the trend to enroll at ubc or uwo continues to stay higher, this will be an opportunity for vimloc to partner with cla to initiate conversations with these schools encouraging them to work with ala and minority librarians to find ways to accredit their foreign degrees, so these librarians can be employable sooner than later.   the vimloc mentoring program is already underway, and vimloc hopes to work with the cla and vimloc members to address scholarship and leadership training possibilities. cla needs to consider offering many of these training sessions free of charge through their online forum, the educational institute. these topics are often covered in library schools, but for first generation librarians who did not attend library schools in canada and are not yet employed in a library, these free workshops will be particularly beneficial.     many librarians have expressed a need for networking opportunities. although vimloc has been active in professional conferences, such as cla and ola, and has a website, facebook page, linkedin page and a google electronic mail list, through which members can interact with each other, an interest in creating other networking opportunities still remains.   limitations and recommendations   although the survey was sent out to various electronic mail lists, it is possible that there were librarians who did not take the time to complete the survey. due to financial constraints some minority librarians may not be members of any of the library associations, and due to lack of awareness first generation librarians may not be part of library electronic mail lists. these members may have been missed in the survey. unfortunately, there is no way to know the number of visible minority librarians that did not take the survey.    this survey is also only a snapshot in time and these demographics are likely to change with changes in government policies on immigration. vimloc should aim to conduct these surveys every 3-5 years to compare statistics and learn more about demographic changes, the needs of this population, their challenges and barriers, and their continuous evolution in the library field. data from this survey will help vimloc to find and implement ways to create better networking opportunities for visible minority librarians to connect, collaborate, and create a sense of community.   it will also be beneficial for vimloc to learn from this group of librarians the types of administrative positions they hold and what “other” positions they hold in their libraries, as well as to learn more about their leadership experiences, trials, and successes.   vimloc should consider collaborating with cla and its education institute more closely so both entities can be proactive in helping this group of visible minority librarians.   another initiative for vimloc to consider is establishing connections with canadian library schools and partnering with them to identify and recruit minorities in the community to join their schools.    while vimloc has initiated and worked on various projects, funding is always an issue for this group of librarians, especially if they are first generation immigrants from asian and african countries. all vimloc representatives are volunteers interested in this topic and have worked hard on many of the initiatives mentioned. funding help from cla will go a long way in supporting this network that can help these librarians, particularly those not yet employed. apart from helping vimloc to offer free web-based workshops on various topics addressed previously, cla funding can be used to attract visible minorities to join library schools and attend library conferences.   future surveys undertaken by vimloc should not only collect quantitative data, but also gather more qualitative responses. these qualitative responses will assist in setting further strategic directions for vimloc.   acknowledgements   the authors would like to thank cla for its sponsorship of one-year free personal cla membership to one of the lucky respondents of the survey.    the authors would also like to thank lyn currie (retired librarian, university of saskatchewan) and norda majekodunmi (york university) for taking the time to read previous versions of the manuscript and offer suggestions for improvement.    references   aspinall, p. j. (2002). collective terminology to describe the minority ethnic population: the persistence of confusion and ambiguity in usage. sociology, 36(4), 803-816. http://dx.doi.org/10.1177/003803850203600401   berry, e. (2008). multicultural services in canadian public libraries. bibliothek forschung und praxis, 32(2), 237 242. http://dx.doi.org/10.1515/bfup.2008.036   caidi, n. (2008). information practices of ethno-cultural communities (ipec). toronto, on, can: ceris the ontario metropolis centre. retrieved from http://ceris.metropolis.net/virtual%20library/rfpreports/caidiallard2005.pdf   caidi, n., & macdonald, s. (2008). information practices of canadian muslims post-9/11. toronto, on, can: ceris the ontario metropolis centre.   chilana, r. s. (2001). delivering multilingual services in public libraries in british columbia. pnla quarterly, 65(3), 18-20.   cho, a. (2014). diversity pathways in librarianship: some challenges faced and lessons learned as a canadian-born chinese male librarian. in d. lee & m. kumaran (eds.), aboriginal and visible minority librarians: oral histories from canada (pp. 89-102). lanham, maryland: scarecrow.   cho, a., & con, a. (2012). partnerships linking cultures: multicultural librarianship in british columbia’s public libraries. in c. b. smallwood & k. becnel (eds.), library services for multicultural patrons: strategies to encourage library use (p. 338). lanham, md: scarecrow press.   dakshinamurti, g. b. (2014). reflections on my experience in manitoba as a visible minority librarian: a personal perspective and review of future challenges for visible minority librarians. in d. lee & m. kumaran (eds.), aboriginal and visible minority librarians: oral histories from canada (pp. 17-26). lanham, maryland: scarecrow.   dali, k. (2012). reading their way through immigration: the leisure reading practices of russian-speaking immigrants in canada. library & information science research, 34(3), 197-211. http://dx.doi.org/10.1016/j.lisr.2012.02.004   dilevko, j., & dali, k. (2002). the challenge of building multilingual collections in canadian public libraries. library resources & technical services, 46(4), 116-137.   fox, d. (2007). a demographic and career profile of canadian research university librarians. the journal of academic librarianship, 33(5), 540-550. http://dx.doi.org/10.1016/j.acalib.2007.05.006   government of canada. (2011). international convention on the elimination of all forms of racial discrimination: 19th and 20th reports of canada. retrieved from http://www.canadianheritage.gc.ca/damassetpub/dam-drtprs-humrts/staging/texte-text/19-20_1362690863117_eng.pdf?wt.contentauthority=3.1   government of canada. (2014). employment equity act. retrieved from http://laws-lois.justice.gc.ca/eng/acts/e-5.401/page-1.html   gupta, a. (2014). a minority librarian's journey: challenges and issues along the way. in d. lee & m. kumaran (eds.), aboriginal and visible minority librarians: oral histories from canada (pp. 147-155). lanham, maryland: scarecrow.   hakim silvio, d. (2006). the information needs and information seeking behaviour of immigrant southern sudanese youth in the city of london, ontario: an exploratory study. library review, 55(4), 259-266. http://dx.doi.org/10.1108/00242530610660807   ingles, e., de long, k., humphrey, c., sivak, a., sorensen, m., & de peuter, j. (2005). the 8rs research team. canadian library human resource study.  retrieved from http://www.ls.ualberta.ca/8rs/8rsfutureofhrlibraries.pdf   kandiuk, m. (2014). promoting racial and ethnic diversity among canadian academic librarians. college & research libraries, 75(4), 492-556. http://dx.doi.org/10.5860/crl.75.4.492   kumaran, m. (2012). leadership in libraries: a focus on ethnic-minority librarians. oxford, uk: chandros publishing.   kumaran, m. (2014). the immigrant librarian: challenges big and small. in d. lee & m. kumaran (eds.), aboriginal and visible minority librarians: oral histories from canada (pp. 171-184). lanham, maryland: scarecrow.   kumaran, m., & salt, l. (2010). diverse populations in saskatchewan: the challenges of reaching them. partnership: the canadian journal of library & information practice & research, 5(1), 1-25.   lau, k. e. (2014). becoming the rhizome: empowering librarians and archivists of color. in d. lee & m. kumaran (eds.), aboriginal and visible minority librarians: oral histories from canada (pp. 123-134). lanham, maryland: scarecrow.   lee, d., & kumaran, m. (2014). aboriginal and visible minority librarians: oral histories from canada. lanham, maryland: scarecrow.   leong, j. h. t. (2013). ethnic diversity at the university of toronto libraries. paper presented at the international federation of library associations (ifla), singapore. retrieved from http://library.ifla.org/67/   li, l. (2014). from china to canada: experiences of a college librarian in the canadian prairies. in d. lee & m. kumaran (eds.), aboriginal and visible minority librarians: oral histories from canada (pp. 49-56). lanham, maryland: scarecrow.   maestro, e. (2014). proud to be a filipino librarian. in d. lee & m. kumaran (eds.), aboriginal and visible minority librarians: oral histories from canada (pp. 27-36). lanham, maryland: scarecrow.   majekodunmi, n. (2014). from recruitment to tenure: a reflectionon race and culture in a canadian acadmiec library. in d. lee & m. kumaran (eds.), aboriginal and visible minority librarians: oral histories from canada (pp. 199-211). lanham, maryland: scarecrow.   mckenna, j. (2007). canadian library human resources short-term supply and demand crisis is averted, but a significant long-term crisis must be addressed. evidence based library and information practice, 2(1), 121-127.   national post. (2007). canada told not to use term 'visible minorities.' national post. retrieved from http://www.nationalpost.com/news/story.html?id=99f7e1e1-4424-429c-908a-125822989a97&k=18408   paola picco, m. a. (2008). multicultural libraries' services and social integration: the case of public libraries in montreal canada. public library quarterly, 27(1), 41-56. http://dx.doi.org/10.1080/01616840802122443   shrivatsava, a. (2014). observations of a new immigrant library professional: career journey from india to canada via the netherlands. in d. lee & m. kumaran (eds.), aboriginal and visible minority librarians: oral histories from canada (pp. 103-116). lanham, maryland: scarecrow.   statistics canada. (2009). visible minority population, by province and territory (table). 2006 census. retrieved from http://www.statcan.gc.ca/tables-tableaux/sum-som/l01/cst01/demo52a-eng.htm   statistics canada. (2010, march 9). study: projections of the diversity of the canadian population. the daily. retrieved from http://www.statcan.gc.ca/daily-quotidien/100309/dq100309a-eng.htm   statistics canada. (2012a). visible minority of person. retrieved from http://www.statcan.gc.ca/concepts/definitions/minority-minorite1-eng.htm   statistics canada. (2012b). classification of full-time and part-time work hours. retrieved from http://www.statcan.gc.ca/concepts/definitions/labour-travail-class03b-eng.htm        statistics canada. (2013). visible minority and population group reference guide. national household survey, 2011. statistics canada catalogue no. 99-010-xwe2011009. retrieved from http://www12.statcan.gc.ca/nhs-enm/2011/ref/guides/99-010-x/99-010-x2011009-eng.cfm   statistics canada. (2014). immigration and ethnocultural diversity in canada. statistics canada analytical document 99-010-x. retrieved from http://www12.statcan.gc.ca/nhs-enm/2011/as-sa/99-010-x/99-010-x2011001-eng.cfm   vimloc. (2014). about us. retrieved from http://vimloc.wordpress.com/terms-of-reference/     appendix survey of visible minority librarians statistics in canada   page 1 project title: collecting statistics of visible minority librarians in canada researcher(s): maha kumaran (liaison, mcgill university) are current co-moderators and founding members of the visible minority librarians of canada (vimloc) network, canadian library association (2011-2013).   compensation: this survey will take about 5 minutes to complete. by participating in the survey you will have a chance to win a one-year free personal membership to cla. if you would like to enter your name for this draw, please make sure you enter the information as necessary when redirected at the end of the survey. your chance of winning is estimated at approximately 1%, depending on the actual number of participants who entered their names into the draw.   purpose and objective: currently there is no data on the number of visible minority librarians working in canadian libraries. the visible minority librarians of canada (vimloc) network wants to gather statistics on the number of visible minority librarians working in or for canadian libraries. the results of this survey will serve as foundational data that will help vimloc identify the needs of visible minority librarians and propose projects or initiatives to empower them in their current positions or their future career development initiatives.    research background: according to statistics canada, the current visible minority population in canada is over 6 million. this is 19.1% of canada’s total population[1]. statistics canada also projects that close to 30% of the nation’s population will consist of visible minorities by 2031 that will include both foreign-born and canadian born visible minorities[2]. does our current profession reflect this demographic?  the 8rs study states that visible minorities “are under-represented across all types of libraries” and make up “only 7% library professional librarian labour force (compared to 14% of canada’s entire labour force)[3]. with this survey, vimloc intends to determine the number of visible minority librarians in canadian libraries. the information collected through this survey will serve as foundational data that enables vimloc to identify the needs of this particular group of librarians.   the canadian employment equity act defines visible minorities as "persons, other than aboriginal peoples, who are non-caucasian in race or non-white in colour.[4]" the visible minority population consists mainly of the following groups: chinese, south asian, black, arab, west asian, filipino, southeast asian, latin american, japanese and korean.   confidentiality: all data collected will be maintained, managed and stored by the two researchers. after you complete the survey, you will be redirected to another form to provide your personal information for the draw to win a personal one-year cla membership. the survey is set up so that researchers will not be able to associate the personal information with the rest of the data.   right to withdraw: your participation in this survey is voluntary. you may withdraw from this survey any time by closing the browser. once all the information is entered and submitted, the data is anonymous and withdrawal will not be possible.   follow up: the researchers intend to publish the survey results of this study through a canadian open access journal, such as the partnership journal, and if you are interested, you will be sent a link to this publication. if you would like to know the results of the study prior to publication, please contact either one of the researchers.   questions or concerns: this project has been approved on ethical grounds by the university of saskatchewan and mcgill university. if you have any questions or concerns regarding your rights or welfare as a participant in this research study, please contact the university of saskatchewan research ethics office at 306-966-2084 or ethics.office@usask.ca or the manager, research ethics, mcgill university at 514-398-6831 or lynda.mcneil@mcgill.ca. by completing and submitting this survey, your free and informed consent is implied.    if you have any questions, please contact: maha at maha.kumaran@usask.ca or heather at heather.cai@mcgill.ca.    on the next page, you will see a pdf/word icon. please use this icon to print and save a copy of the consent form if you need it for your records.   page 2 the canadian employment equity act defines visible minorities as "persons, other than aboriginal peoples, who are non-caucasian in race or non-white in colour." the visible minority population consists mainly of the following groups: chinese, south asian, black, arab, west asian, filipino, southeast asian, latin american, japanese and korean. are you a visible minority librarian currently working in canada?         yes no   what group do you belong to or which group fits you the best?   chinese south asian (includes bangladeshi, indian, pakistani, srilankan) black filipino latin american south east asian (includes vietnamese, cambodian, malaysian, and laotian) arab (includes egyptian, kuwaiti and libyan) west asian (includes afghan, assyrian and iranian) korean japanese other (please specify) ________________________   tell us if you are a first generation minority librarian or not. first generation would mean that you were born else where but moved to canada at some point in your life. second generation would mean you were born in canada to immigrant parents. if you would like to add an explanation about this, please use the text box beside, such as your age or the year when you came to canada.   first generation  ______________________ second generation _______________________ other _____________________________ second generation  ______________________ other ______________________   did you complete your professional library degree in canada? yes no   if no, tell us which country you completed your professional degree?     was the professional library degree considered ala accredited?      if yes, tell us which university /college you received your ala accredited library degree.       what type of library are you currently working at? if you are at a special library, please specify the type of library government, religious organization, etc. public library regional library academic library college library special library ______________________________ school library other (please specify)   which province / territory do you currently work in? alberta british columbia manitoba new brunswick newfoundland and labrador northwest territories nova scotia nunavut ontario prince edward island quebec saskatchewan yukon other (if you are working for a canadian library outside of canada) ___________________   select the job category(ies) that matches your current job responsibilities acquisitions / collection development administration adult services archives assessment automation / systems / it services cataloging / metadata management children’s services circulation consultant / knowledge management/ researchers digitization and preservation e-resources and serials government documents instruction services interlibrary loan services liaison librarian licensing marketing/outreach/community services media specialist rare books and special collections reference / information services school / teacher librarian web services youth services other (please specify) _______________________   do you work part-time (less than 30 hours/week), full-time (30 or more hours/week), or casual hours?  please indicate below: part time full time casual hours other _________________________________   please use the box below to comment on anything else: topics could be on the challenges of finding the right job, the need for another degree, lack of support through mentorship or networking possibilities, etc.     if you would like to save a copy of the survey for your own records please click on the pdf/word icon below now.   cla one year personal membership draw: this is optional, if you do not want to enter the information click submit.   your full name: your institution: your mailing address with postal code: your email: your day time phone number:   thank you for taking the time to complete this survey.     [1] http://www12.statcan.gc.ca/nhs-enm/2011/as-sa/99-010-x/99-010-x2011001-eng.cfm#a4 [2] http://www.statcan.gc.ca/pub/12-581-x/2012000/pop-eng.htm#c01 [3] http://www.ls.ualberta.ca/8rs/8rsfutureofhrlibraries.pdf [4] http://laws-lois.justice.gc.ca/eng/acts/e-5.401/page-1.html research article   a comparison of traditional book reviews and amazon.com book reviews of fiction using a content analysis approach   christy sich research & instructional librarian western university london, on, canada email: csich@uwo.ca   received: 29 june 2015  accepted: 14 feb. 2017      2017 sich. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective this study compared the quality and helpfulness of traditional book review sources with the online user rating system in amazon.com in order to determine if one mode is superior to the other and should be used by library selectors to assist in making purchasing decisions.   methods for this study, 228 reviews of 7 different novels were analyzed using a content analysis approach. of these, 127 reviews came from traditional review sources and 101 reviews were published on amazon.com.   results using a checklist developed for this study, a significant difference in the quality of reviews was discovered. reviews from traditional sources scored significantly higher than reviews from amazon.com. the researcher also looked at review length. on average, amazon.com reviews are shorter than reviews from traditional sources. review rating—favourable, unfavourable, or mixed/neutral—also showed a lack of consistency between the two modes of reviews.   conclusion although amazon.com provides multiple reviews of a book on one convenient site, traditional sources of professionally written reviews would most likely save librarians more time in making purchasing decisions, given the higher quality of the review assessment.   introduction   collection development training often promotes the use of book reviews to assist librarians in choosing fiction for library collections. book reviewing as a systematic evaluation of literature first began with the publication of the journal des savants in 1665 (boaz, 1958). traditional book review sources for libraries include such periodicals as book review digest, booklist, book world, kirkus reviews, and library journal. other traditional book review sources include saturday review, observer, new york times book review, and the new yorker.   in recent years, with the decline of the newspaper industry, the place for book reviews has come into question. many stand-alone publications have folded and book review sections have been amalgamated with other sections or they have been slashed completely (ciabattari, 2011). at the same time, with the rise of the web, independent book reviewers have cropped up in various online manifestations including google books, goodreads, librarything and amazon, just to name a few. these new online platforms are convenient and allow readers and librarians quick, point-of-need access to reviews.   not surprisingly, those in the formal book reviewing industry scorn these types of reader review and ratings systems, criticizing reviews published on amazon for being anonymous or sometimes even fraudulent with claims made that authors seek out friends and family to write glowing reviews (harmon, 2004). pool (2007) argues that “amazon has created a system that not only allows but encourages ethical and literary standards far lower than those we find in print reviewing” (p. 100).   still, despite the criticism from the book reviewing industry, amazon seems like a quick and easy way for librarians to get a sense of a book’s quality and whether or not it might be worthy of selection. in order to test the validity of the anecdotal claims about the flaws with amazon reviews, and to be able to recommend review sources as tools for collections librarians, this study aims to measure the quality of reviews found on amazon and compares the results to the quality of reviews in traditional publications, using a content analysis approach. by examining the texts of the reviews, the researcher assessed the quality and helpfulness of reviews found on the amazon.com website to test the claims about the poor quality of amazon reviews.   in order to determine the quality, the author scored each individual review using a checklist of 12 categories developed for this study (see appendix). the categories of analysis were adapted from john e. drewry’s writing book reviews (1974), using elements that make up a helpful review. the 12 categories that the researchers looked for in each review included character source, character treatment, plot elements, plot devices, plot summary, theme, setting, four categories of style, and the inclusion of an evaluative statement about the book. in this study, quality was determined by looking at the total helpfulness score. results of the study will help librarians responsible for collections of fiction decide whether or not to make use of amazon as a collections development tool.   literature review   the deluge of the number of books published is something that librarians and readers alike must grapple with. the impetus and context for this study was formed, in part, through an analysis of some of the literature on book reviewing. in particular, this research was motivated by the apparent anxiety of critics, researchers, and writers of reviews who have attempted to make a case for formal, professional book reviews by condemning the free, online book review sources for encouraging amateur book reviewing. the literature reviewed in this study can be organized thematically by the role or purpose of book reviews, authority and anonymity of reviews—especially in online formats and the quality of reviews.   role or purpose of the book review   there has been tension between critics and writers regarding the purpose of the book review from the early days of periodical publishing. in her essay, reviewing (1939), virginia woolf unabashedly unleashed her contempt of book reviews and the critics. woolf proposed a new system and questioned the purpose of reviews: “why bother to write reviews or to read them or to quote them if in the end the reader must decide the question for himself?” (p. 12).   interestingly, when reviewing was published, woolf’s husband contributed his own contradictory commentary, defending the role of reviews to some degree, by suggesting that with the burgeoning of readers and the number of books published that the function of the reviewer is “to give to readers a description of the book and an estimate of its quality in order that he may know whether or not it is the kind of book which he may want to read” (woolf, 1939, p. 29).   just as with readers needing some guidance on what books to buy and read, librarians responsible for building large collections of fiction in both academic libraries and public libraries can also benefit from tools to help make purchasing decisions, especially when faced with dwindling budgets. some articles, including natowitz and carlo (1997), palmer (1991), burchette (1992), greene and spornick (1995), touch on the impact of reviews on collection development in libraries as well as on individual purchasing decisions. in particular, natowitz and carlo examined book reviews published in choice, journal of american history, and american historical review to determine the degree to which book reviews provide assessments to help with acquisition choices. natowitz and carlo’s research showed that different variables involved in book reviewing depend upon the journal the review is written for, making the case that awareness of different types of reviews in journals allows librarians to make informed collections decisions.   in reviewing the literature on book reviewing, blake (1989) covered the role of the review extensively and noted that there is a connection between the number of book reviews of a single title and the number of library collections in which that title can be found. blake indicated that research on the reviewing of non-print materials had largely been neglected at that time and was an area that needed further study. in another study, palmer (1991) pointed to the inconsistent role that reviews play in collection development in libraries by demonstrating that reviews are relied upon heavily by some libraries and not at all by others.   pinfold (2007) focused on book reviews as a tool for collections librarians, explaining that, despite the prevalence of approval plans whereby libraries automatically receive books based on pre-set parameters rather than reviews, he heavily relies on reviews to help with selection decisions. pinfold also commented on reviews from amazon, stressing that they should be ignored because of the lack of peer review.   authority and anonymity   as revealed in pinfold’s paper, a major critique of book reviews from online platforms such as amazon is that they lack authority and the peer review process. ciabattari (2011) explored this further by discussing the resurrection of the book review, arguing that just as there is a proliferation of books being published each year, so too is there a proliferation of reviewers commenting on books in the online world.  reliance on these reviews then is all the more difficult because of the amount of reviews available. ciabattari notes that “readers can find book news and reviews in formats ranging from a hundred forty characters to six thousand words and up, online and in print. … despite the flood of friendly recommendations coming from amazon and the social networking sites, many readers still turn to familiar gatekeepers for curatorial guidance” (2011, p. 122).   pool (2007) emphasized the lack of quality control in the free online world: “in self-published reviews on the web … critical failings are and are bound to be exacerbated. … unscreened, anonymous, and unedited, self-published reviewers can be—and often are—as biased, uninformed, ungrammatical, and critically illiterate as they like” (p. 122). to highlight the uncritical aspect of online reviews further, harmon (2004) reported on a glitch that occurred in 2004 when amazon reviewers’ real names were displayed for a short time, revealing that some authors had given their own works and works of friends glowing, five-star reviews.   quality of reviews   in a study on the sunny book review, katz (1985) discusses the disproportionate number of positive reviews to negative reviews. katz suggests that doubling the number of reviews produced with an emphasis on negative reviews would be more beneficial to librarians.   some studies have focused on the evaluative aspects of reviews. bilhartz (1984) looks at the changes over time by reviewing works about history and notes that certain periods seemed to produce more critical reviews than other periods. authors before 1960 could expect uncritical praise, but by the sixties, a new breed of reviewers had arrived. they were not so accepting and who began to write much more critically.   boaz (1958) similarly points out that the nature of book reviews changes over time but suggests that this change has to do with an increase of educated readers. book reviews were quite scathing in the earlier days of reviewing, speaking to a small educated few. by the mid-twentieth century the number of educated readers increased and so too did the volume of book production. at this time, book reviews became more favourable as the book reviews served increasingly as a means of notifying readers of books and their contents. it was left to the reader or librarian to make the ultimate decision about reading or purchasing the work.   in a more recent study, hartley, cowan, deeson, and thomas (2016) looked at the quality of individuals' writing to see if any changes occur over time. the researchers compared the writing styles of 5 different academic writers of book reviews over a 20-year span to see if there were any changes in style over time. they used automated measures of spelling, grammar, and readability found in microsoft word 8. they found that, generally, each reviewer remained relatively consistent over time.   interestingly, witucke (1982) compares reviews from several traditional reviewing publications, examining various aspects, such as coverage and promptness, and concludes that selectors cannot rely on a single reviewing publication to get a comprehensive view of a given book.   liu, chen, and chiu (2013) propose a book review recommendation system that scrapes the web for all available reviews and systematically presents the reviews to the user in a ranked order based on quality. in order to evaluate the quality of the reviews for this proposed system, the researchers discuss two approaches. one was to look at textual features of the reviews and the other was to look at non-textual features of reviews. liu and co-researchers chose to focus their study on non-textual features. these included book review length, time factor, book rating, and reviewers' reputation as ways to measure the book review quality. this author would argue that there are some flawed assumptions built into the features used to score reviews. for instance, the study gives more credit to lengthier reviews, based on comments made from an earlier study (jurca, garcin, talwar, & faltings, 2010) that looked at product reviews and found that lengthier reviews were perceived by other users as being written by someone with more authority on the product. it is unclear as to how the length would necessarily contribute to the quality of a review. similarly, the timeliness of the review was given a higher score stating that earlier reviews are more influential. again, it is unclear how a review's date can indicate its level of quality.   aims   this study looks at the textual features of reviews and provides an alternative approach to assessing the quality of reviews. the purpose is to assess the quality of the reviews found on amazon.com and determine whether or not amazon is an appropriate tool for librarians, who are responsible for building collections of fiction to use to inform purchase decisions. the study compares book reviews posted on amazon.com with book reviews published in more traditional book review sources, such as periodicals and newspapers. the author hypothesized a significant difference between the quality of reviews published in traditional sources and reviews found on amazon.com. therefore, the null hypothesis is no significant difference between the quality of traditional reviews and the quality of amazon reviews. this study was undertaken with several research questions in mind:   is there a significant difference between the quality of reviews found in traditional reviews and reviews published on amazon?   is there a difference in rating agreement across the two types of reviews? does one type of source have a higher number of favourable, neutral, or unfavourable reviews than the other?   is the average length of reviews from one type of source an indication of the quality of the reviews from that type of source?   to this author’s knowledge no studies have yet compared the quality and helpfulness of traditional book review sources with online user rating systems such as amazon.com. a comparison of the two modes of reviews will help to clarify anecdotal claims that one mode may be superior to the other. this has the potential of assisting library selectors in identifying appropriate sources for reviews, saving time, and making the most of library collections budgets.   methods   for this study, the author determined that analyzing reviews of the same titles to look at rating agreement across the different modes of reviews would be important. as such, randomly selecting reviews from each type of review source was not possible. similarly, selecting titles at random was difficult because often not enough reviews of the randomly selected titles existed in either the traditional book review sources or on amazon. around the time of this study’s inception, the lost man booker prize was announced. the purpose of this special booker prize was to acknowledge retrospectively novels that had been published in 1970 and had missed an opportunity to compete when the booker prize changed its rules in 1971 to look at works of fiction in the current year only. this seemed to be an opportunity to make use of a selected list of books that would most likely have been reviewed in the traditional review sources.   two types of reviews were selected for analysis: published reviews that had been indexed in book review index and amazon.com reviews that were written, in order to avoid bias, prior (pre-2008) to the announcement of the lost man booker prize. this study used reviews from seven books long-listed for the lost man booker prize because they each had six or more reviews from each type of review source. table 1 shows the books and number of reviews used in the study. in total, 127 reviews from traditional published review sources and 101 reviews from amazon.com were analyzed.     table 1 books reviewed title total number of reviews published in traditional sources total number of amazon reviews bomber 22 11 troubles 8 16 the bay of noon 22 6 i’m the king of the castle 8 13 a fairly honourable defeat 23 15 fire from heaven 24 34 the driver’s seat 20 6 total 127 101     in order to assess the quality of each review, the review’s helpfulness was scored using a coded checklist developed for this study that involved 12 categories (see appendix). the higher the helpfulness score, the greater the quality of the review, and the more helpful it is to a librarian making a purchase decision. the categories were developed and adapted from a list of recommended elements that reviewers should address when reviewing fiction from john e. drewry’s writing book reviews (1974). written as a guide for review practitioners, drewry's work was selected for this study because it was thorough and included a chapter on reviewing fiction. the goal of drewry's book was to encourage competent reviewing. drewry was a former dean of the university of georgia's school of journalism and not only wrote reviews for various publications but also taught a course in book reviewing.   for this study, one point was given to a review for each of the defined categories that it addressed. the categories included character source (fictional or historical), character treatment (or character development), plot elements (how elements are handled, e.g., introduction, suspense, climax, conclusion), plot devices (e.g., catastrophe, accident, fate, mystery, sub-plots etc.), plot summary (the plot is described), theme (a comment about the overall theme is made), setting (a comment is made about the setting or background), four categories of style (clarity, sentence structure, emotional qualities, and narrator perspective), and inclusion of a statement of judgment about the book. the length (word count) of the review and the evaluative rating—whether the review was favorable, unfavorable or mixed/neutral—were also documented for analysis. the coders looked for specific words such as recommended or not recommended to determine a rating.   best practices in content analysis encourage inter-coder reliability testing. to follow these principles of best practice and to establish objectivity in content analysis, two coders were used in the study. the author coded the entire sample of reviews, and a second coder, a librarian colleague, was enlisted and trained. the author and second coder met several times. in two training sessions, they coded together reviews not included in this study, using the checklist. where they disagreed, they made changes to the checklist. they met a third time after independently coding 10 non-study reviews, and they tweaked the chart again.   the second coder ultimately coded a random sample of 50 of the 228 reviews using the checklist. initially, the author supplied the coder with 10 reviews from the study and did a preliminary calculation of percent agreement to make sure that an acceptable level of agreement was being met. at this point, the coder then coded an additional 40 reviews from the study. a free, online program called recal was used to establish inter-coder reliability by calculating percent agreement. cohen’s kappa was selected for this study, as it takes chance agreement into consideration. according to lombard, snyder-duch, and campenella-brachen (2010), a coefficient of .70 agreement is appropriate for some studies because some indices, such as cohen’s kappa, are known to be more conservative; therefore, lower criteria can be used. in this study, acceptable levels of agreement were achieved in 10 of the 12 categories (see table 2). additional training may have improved the agreement in the theme and clarity categories.   results   quality   the quality of a review was determined by how well a review scored on the helpfulness checklist (see table 3). to compare traditional reviews with amazon reviews, a two-sample t-test was performed that does not assume equal variances. the test revealed that the probability of variance is less than 0.05, therefore we can reject the null hypothesis, that there is no significant difference between the quality of traditional reviews and the quality of amazon reviews. the mean score for helpfulness for the traditional reviews (m = 6.57, sd = 2.399, n = 127) is significantly higher than the scores for the amazon reviews (m = 4.81, sd = 1.999, n = 101), using the sample t-test for unequal variances, t = 5.9, p < 0.0001.     table 2 inter-coder reliability testing elements of a helpful review cohen’s kappa character source .88 character treatment .89 plot elements .70 plot devices .78 plot summary .92 theme .60 setting .87 sentence structure .80 clarity .53 emotional qualities .70 narrator perspective 1.0 judgement .81     table 3 review helpfulness score by book title averagea score for traditional reviews averagea score for amazon reviews bomber 7.0 4.8 troubles 6.3 6.1 the bay of noon 7.2 5.7 i’m the king of the castle 5.0 3.4 a fairly honourable defeat 6.2 4.5 fire from heaven 6.1 4.4 the driver’s seat 7.3 6.5 weighted average score 6.6 4.8     review length   a pearson product-moment correlation coefficient was computed to assess the relationship between the length of a review and the quality of a review as measured by the helpfulness score. there was a positive correlation between the two variables, r = 0.326, n = 228, p = 0. increases in review length were correlated with an increase in the helpfulness score. length of review was determined by an individual word count. interestingly, despite the unlimited space available on its online platform, amazon reviews tend to be more succinct on average than traditional reviews. the average word count for traditional reviews was 394 and the average word count for amazon reviews was 244. a t-test reveals that this difference is significant (p = 0.0089). there is a definite tendency toward shorter reviews on amazon, with well over half being 300 words or less. the traditional reviews that were analyzed had a fairly even distribution of varying lengths. some of the publications containing the traditional reviews reserved very little room for reviews, while others had lengthier essays.   rating   further research questions that were asked in this study include the following: is there a difference in rating across the platforms? does amazon have a higher percentage of positive ratings than the traditional sources? on average, do the two types of sources agree on rating?   a chi-square test was performed to look at whether a difference exists between ratings across the two modes. a significant difference was noted between the number of unfavourable reviews and favourable reviews, depending on whether the review was from amazon or a traditional source (see tables 4 and 5). traditional sources had more unfavourable reviews and fewer favourable reviews than expected. the opposite relationship was observed for reviews on amazon, which had more favourable reviews and fewer unfavourable reviews than expected, no significant difference existed between the expected number of unfavourable and neutral/mixed reviews for either modes, also, the sources had no significant difference between the expected number of favourable and mixed reviews,   ranking each book by the highest percentage of favourable views shows disagreement between the two types of reviews. the driver’s seat had the highest percentage of favourable reviews on amazon. conversely, this novel had the lowest percentage of favourable reviews among the traditional review sources. instead, the bay of noon received the highest percentage of favourable reviews in the traditional review sources. interestingly, the novel that eventually won the lost man booker prize, troubles, was given the second lowest percentage of favorable reviews by amazon raters and had only the fifth highest number of favourable reviews among the traditional review sources.     table 4 amazon review ratings title number of reviews by rating total favourable mixed/neutral unfavourable a fairly honourable defeat 11 (73%) 3 (20%) 1 (7%) 15 bomber 9 (82%) 2 (18%) 0 (0%) 11 fire from heaven 26 (76%) 6 (18%) 2 (6%) 34 i’m the king of the castle 8 (62%) 4 (31%) 1 (8%) 13 the bay of noon 5 (83%) 1 (17%) 0 (0%) 6 the driver’s seat 6 (100%) 0 (0%) 0 (0%) 6 troubles 12 (75%) 3 (19%) 1 (6%) 16 total 77 19 5 101     table 5 traditional review ratings title number of reviews by rating total favourable mixed/neutral unfavourable a fairly honourable defeat 7 (30%) 9 (39%) 7 (30%) 23 bomber 10 (45%) 6 (27%) 6 (27%) 22 fire from heaven 16 (67%) 6 (25%) 2 (8%) 24 i’m the king of the castle 2 (25%) 5 (63%) 1 (13%) 8 the bay of noon 17 (77%) 0 (0%) 5 (23%) 22 the driver’s seat 4 (20%) 12 (60%) 4 (20%) 20 troubles 2 (25%) 3 (38%) 3 (38%) 8 total 58 41 28 127     discussion   although amazon has many benefits as a review source and some quality reviews do exist, this study shows that the best quality reviews are found in traditional book review sources. based on the results of this study, the null hypothesis that no significant difference exists between the quality of traditional reviews and the quality of amazon reviews can be rejected. the findings of this study reveal that the mean score for helpfulness for traditional reviews is significantly higher that the score for the amazon reviews; therefore, collections librarians would be better equipped for making purchasing decisions if they avoid amazon reviews and read reviews found in the traditional sources, such as those indexed by book review index. although tracking down these reviews may take more time, the product is more helpful for making purchasing decisions than reviews found on amazon.   findings in this study also reveal a positive correlation between length of review with the quality of review. it makes sense that if 12 elements are needed for a high-quality score, a minimum length is necessary to achieve this. this study also found that despite having no limits on length in the online environment, the average word count of the amazon reviews was significantly less than the average word count of the reviews from traditional sources.   one limitation of the study is that the traditional book review sources were written decades before the reviews on amazon.com. as bilhartz (1984) pointed out, times change, and so do the ways in which writers write reviews. a follow-up study that compares reviews on amazon.com to online sources of contemporary reviews found in proprietary or subscribed sources would offer additional insight.   conclusion   amazon is a tempting source for librarians to use in book selection largely due to its ease of use and the plethora of reviews that are often available. with the significant difference in quality that was found in comparing the amazon reviews to the available traditional reviews, this researcher would urge selectors to use amazon in a limited fashion, and if used, used in conjunction with more traditional sources. as with other studies, this research shows that agreement is not consistent even across the traditional sources; therefore, no single source should be relied upon. this researcher would recommend that collections librarians seek out traditional sources of reviews that involve at least an editor in the publication process. full text databases that index book reviews from many different traditional sources are a good alternative to amazon.com.        acknowledgments   the author would like to thank dan sich for his assistance coding reviews for the interrater reliability testing, ken meadows for his assistance with the data analysis of the study, and lise doucette for reviewing a draft of this manuscript.   references   bilhartz, t. d. (1984). in 500 words or less: academic book reviewing in american history. history teacher, 17(4), 525–536. retrieved from https://eric.ed.gov/?id=ej304820   blake, v. l. p. (1989). the role of reviews and reviewing media in the selection process. collection management, 11(1-2), 1–40. http://dx.doi.org/10.1300/j105v11n01_01   boaz, m. (1958). some historical sidelights on book reviewing. in l. c. merritt (ed.), reviews in library book selection (pp. 179-182). detroit: wayne state university press.   burchette, r. b. (1992). an examination of children’s book review media (masters thesis). university of north carolina at chapel hill, chapel hill.   ciabattari, j. (2011). back from the dead: the state of book reviewing. poets & writers, 39(5), 121. retrieved from https://www.pw.org/content/back_from_the_dead_the_state_of_book_reviewing_0?cmnt_all=1   drewry, j. e. (1974). writing book reviews. westport, conn.: greenwood press.   greene, r. j., & spornick, c. d. (1995). favorable and unfavorable book reviews: a quantitative study. the journal of academic librarianship, 21(journal article), 449–453. http://dx.doi.org/10.1016/0099-1333(95)90088-8   harmon, a. (2004, feb 14). amazon glitch unmasks war of reviewers. the new york times. retrieved from http://www.nytimes.com/2004/02/14/us/amazon-glitch-unmasks-war-of-reviewers.html   hartley, j., cowan, j., deeson, c., & thomas, p. (2016). book reviews in time. scientometrics, 109(3), 2123–2128. http://dx.doi.org/10.1007/s11192-016-2114-z   jurca, r., garcin, f., talwar, a., & faltings, b. (2010). reporting incentives and biases in online review forums. acm transactions on the web, 4(2), 1–27. https://dx.doi.org/10.1145/1734200.1734202   katz, b. (1985). the sunny book review. technical services quarterly, 3(1-2), 17–25. http://dx.doi.org/10.1300/j124v03n01_03   liu, d. r., chen, w. h., & chiu, p. h. (2013). recommending quality book reviews from heterogeneous websites. internet research, 23(1), 27–46. http://dx.doi.org/10.1108/10662241311295764   lombard, m., snyder-duch, j., & campenella-brachen, c. (2010). practical resources for assessing and reporting intercoder reliability in content analysis research projects. retrieved from http://matthewlombard.com/reliability/   natowitz, a., & carlo, p. w. (1997). evaluating review content for book selection: an analysis of american history reviews in choice, american historical review, and journal of american history. college & research libraries, 58(4), 323–336. http://dx.doi.org/10.5860/crl.58.4.322    palmer, j. w. (1991). fiction selection in ontario public libraries—how important are reviews? public library quarterly, 10(4), 39–48. http://dx.doi.org/10.1300/j118v10n04_04   pinfold, j. (2007). reviews: a librarian’s view. african research & documentation, (102), 31–35.   pool, g. (2007). faint praise: the plight of book reviewing in america. columbia, mo.: university of missouri press.   witucke, v. (1982). the performance of juvenile book review media. serials review, 8(1), 49–55. http://dx.doi.org/10.1016/0098-7913(82)90030-2   woolf, v. (1939). reviewing. london: hogarth press.     appendix     helpfulness code category descriptiona score 1 char_source comment on sources of characters—made up? historical? 2 char_treatment comment on treatment of characters, comment on attitude of writer toward his characters, character development? simple/complex? comment on character personality 3 plot_elements comment on how (one or more) elements of the plot are handled, e.g., introduction, conflict, suspense, climax, conclusion (not just mentioned), tension, action 4 plot_devices comment on plot devices generally used (e.g., catastrophe, accident, fate, mystery, sub-plots, journey), is plot primary or secondary? (letters, diaries, flashbacks, etc.)   5 plot_summary plot is described or outlined   6 theme comment on overall theme, over-arching ideas in the novel, novel’s message, or purpose of the novel, e.g., betrayal, love conquers all, good vs. evil   7 setting comment on setting (historical/local, occupational/institutional, ethereal/esoteric), background, atmosphere, locale, scenic effects   8 style_elements comment on elements of style, e.g., words, prose, language used, figures of speech, sentence structure, paragraphs, tone, allusions, metaphors, symbolism, pace, aphorisms, truisms   9 style_intellect comment on intellectual qualities such as simplicity (e.g., written for children) or clearness of writing (e.g., triteness, clichés, satire)   10 style_emotional comment on emotional qualities, e.g., pathos, humour, force, tragedy, pity, horror, terror, darkness, irony, sarcasm, mysterious, quirky   11 style_perspective comment on narration or narrator perspective   12 judgement some kind of judgement about the book is made—either favorable, neutral/mixed, or unfavorable    13 rating overall review's rating of the book (favourable, neutral/mixed, or unfavourable) abased on the elements that make up a helpful review from writing book reviews (drewry, 1974)   evidence summary   first-time use books are frequently available for patron-driven acquisition   a review of: herrera, g. (2015). testing the patron-driven model: availability analysis of first-time use books. collection management, 40(1), 3-16. doi: http://dx.doi.org/10.1080/01462679.2014.965863   reviewed by: eamon c. tewell reference & instruction librarian brooklyn campus library long island university brooklyn, new york, united states of america email: eamon.tewell@liu.edu   received: 17 mar. 2015  accepted: 23 jul. 2015      2015 tewell. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine whether a hypothetical patron-driven acquisition (pda) purchasing model is acceptable in terms of making available print monographs after their initial publication.   design – quantitative data analysis.   setting – a large public university located in the southern united states of america.   subjects – 8,020 item records representing books used at the author’s institution for the first time in 2012. non-circulating monographs and items such as personal copy reserve materials and government documents were excluded from the sample.   methods – using the libraries’ ils, a listing of the titles of monographs that received first-time use in 2012 was generated and exported to microsoft excel. the getting it system toolkit (gist) was used to batch-search possibilities for acquisition and/or access, including purchase (including amazon and better world books) and free access (such as haithitrust and google books).   main results – a total of 76% (6,130) of titles from the sample of 8,020 were available for purchase. a total of 3% (165) of these titles were both available for purchase and freely available online. books not available either freely or by purchase represented 21% (1,682) of the sample. when participation in a regional resource-sharing consortium was accounted for, only 1% (101) of the titles could not be obtained. books published before the 1920s were more likely to be freely available due to being in the public domain; however a majority of the titles (64%; 5,127) had a publication date of 1990 forward. the humanities represented the largest disciplinary grouping at 57% (4,563), with social sciences (31%; 2,472) and stem (11%; 879) following.   conclusions – in sum, the results indicated a very low margin of unavailability for titles. the author notes that, based on the findings, there should be no pda purchase restrictions according to publication date if a large-scale program were to be implemented at their institution, and that researchers requiring humanities titles would be likely to benefit most from such a program (p. 14). it should be noted that a significant budget for pda was allocated at the author’s institution.   commentary   patron-driven acquisitions (pda), also known as demand-driven acquisitions (dda), is a collection development method that allows patrons to indicate their need for an item that can then be acquired. pda has been rapidly adopted as an alternative method for acquisitions due in part to many libraries’ fiscal constraints imposed by declining budgets and the rapidly escalating costs of electronic subscriptions. while there exist a large number of case studies on pda models in a range of settings, a majority of this work focuses upon e-books instead of print. anderson et al. (2010) is a noteworthy exception, finding that over the course of ten years their library’s pda program contributed appropriate and cross-disciplinary titles to the collection. the study at hand investigates the question of what proportion of first-time use titles could have hypothetically been purchased on-demand, and determines that the retrospective purchase of titles as patrons require them is frequently possible in terms of availability.   the strengths of this study include the appropriateness of the methodology to provide evidence regarding the research question, well-defined criteria for the sample, and the clear presentation of data collection methods and the results. in addition, the study provides context for the library’s print collections so that readers may apply this information to their own setting. one area where the reviewer found the research to be lacking was in the absence of suggested areas for future work. however, this omission does not negatively impact the quality of the findings presented.   this research holds implications for libraries considering or participating in pda. the findings are encouraging for librarians interested in pursuing or expanding a print pda program but concerned that they would be unable to obtain books after their initial publication. as the author observes, the results may be used to inform and guide collection development decisions, including whether to implement a pda model for print books. future studies in this area might consider the broader implications of pda in terms of collection diversity and other ramifications, positive and negative, of moving towards patron-driven models. for instance, walters (2012) expresses concern that pda programs may not distinguish between users’ “immediate desires and their long-term educational needs,” do not take advantage of librarians’ expertise, and may fail to represent marginalized stakeholders and the educational mission of the university (pp. 204-206). moreover, pda is part of a larger tendency that prioritizes “just-in-time” delivery over the ownership of materials, which is itself a continuing trend in library collections that deserves additional scrutiny and deliberation if we are to develop and maintain collections that do not just fill an immediate information need, but aim to provide access to a variety of perspectives and materials.   references   anderson, k. j., freeman, r. s., hérubel, j.-p. v. m., mykytiuk, l. j., nixon, j. m., & ward, s. m. (2010). liberal arts books on demand: a decade of patron-driven collection development, part 1. collection management, 35(3-4), 125–141. doi: http://dx.doi.org/10.1080/01462679.2010.486959   walters, w. h. (2012). patron-driven acquisition and the educational mission of the academic library. library resources & technical services, 56(3), 199–213. doi: http://dx.doi.org/10.5860/lrts.56n3.199   evidence summary   a systematic review of librarian-provided services delivered in healthcare settings finds a positive effect on clinician and patient relevant outcomes and identifies gaps in the evidence   a review of: perrier, l., farrell, a., ayala, a. p., lightfoot, d., kenny, t., aaronson, e., . . . weiss, a. (2014). effects of librarian-provided services in healthcare settings: a systematic review. journal of the american medical informatics association, 21(6), 1118-1124. http://dx.doi.org/10.1136/amiajnl-2014-002825   reviewed by: elizabeth stovold information specialist, cochrane airways group st george’s, university of london tooting, london, united kingdom email: estovold@sgul.ac.uk   received: 30 nov. 2014 accepted: 20 feb. 2015      2015 stovold. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to assess the effects of librarian-provided services, in any healthcare setting, on outcomes important to patients, healthcare providers, and researchers. design – systematic review and narrative synthesis. setting – medline, cinahl, eric, lisa, and central databases; library-related websites, conference proceedings, and reference lists of included studies. subjects – twenty-five studies identified through a systematic literature search. methods – in consultation with the review team, a librarian designed a search to be run in medline that was peer-reviewed against a published checklist. the team then conducted searches in the five identified databases, adapting the search as appropriate for each database. authors also checked the websites of library and evidence based healthcare organisations, along with abstracts of relevant conference proceedings, to supplement the electronic search. two authors screened the literature search results for eligible studies, and reached agreement by consensus. studies of any librarian-delivered service in a healthcare setting, directed at either patients, clinicians of any type, researchers, or students, along with studies reporting outcomes relevant to clinicians, patients, or researchers, were eligible for inclusion. the authors assessed results initially on the titles and abstracts, and then on the full-text of potentially relevant reports. the data from included studies were then extracted into a piloted data extraction form, and each study was assessed for quality using the cochrane epoc risk of bias tool or the newcastle-ottawa scale. the results were synthesised narratively. main results – the searches retrieved a total of 25 studies that met the inclusion criteria, comprised of 22 primary papers and 3 companion reports. authors identified 12 randomised trials, 4 controlled before-and-after studies, 3 cohorts, 2 non-randomised trials, and 1 case-control study. they identified three main categories of intervention: librarians teaching search skills; providing literature searching as a service; and a combination of the teaching and provision of search services. the interventions were delivered to a mix of trainees, clinicians, and students. none of the studies examined services delivered directly to patients or to researchers. the quality assessment found most of the studies had a midto high-risk of bias due to factors such as lack of random sequence generation, a lack of validated tools for data collection, or a lack of statistical analysis included in the study.   two studies measured patient relevant outcomes and reported that searches provided by librarians to clinicians had a positive impact on the patient’s length of stay in hospital. five studies examined the effect of librarian provided services on outcomes important to clinicians, such as whether a literature search influenced a clinical decision. there was a trend towards a positive effect, although two studies found no significant difference.   the majority of studies investigated the impact of training delivered to trainees and students on their literature search skills. twelve of these studies found a positive effect of training on the recipients’ search skills, while three found no difference. the secondary outcomes considered by this review were satisfaction with the service (8 studies), relevance of the answers provided by librarians (2), and cost (3). the majority reported good satisfaction, and relevance. a cost benefit was found in 2 of 3 studies that reported this outcome.   conclusion – authors report a positive effect of training on the literature search skills of trainees and students, and identified a benefit in the small number of studies that examined librarian services to clinicians. future studies should use validated data collection tools, and further research should be conducted in the area of services provided to clinicians. research is needed on the effect of librarian-provided services to patients and researchers as no studies meeting the inclusion criteria examining these two groups were identified by the literature search. commentary   librarians work in a variety of clinical and healthcare environments, providing valuable support services to clinicians, patients, students, and researchers. the authors of this review identified four previously conducted systematic reviews seeking to evaluate librarian-provided services. these reviews were narrower in scope, and none of them considered services offered to patients. this review updates and widens the evidence base on the impact of librarian-delivered services in healthcare settings.   the article was evaluated using the amstar tool for assessing the quality of systematic reviews (shea et al., 2007). it scored highly, meeting 10 out of a possible 11 criteria. authors established their research question prior to conducting the review, and registered the title on the prospero database of systematic reviews. two authors independently carried out the study selection and data. the study includes a comprehensive literature search, outlines the medline strategy in the supplementary material, and indicates that the strategy was peer-reviewed. all types of publications were eligible for inclusion in the study, and the authors provide a table of included studies and their characteristics within the paper.   the authors assessed the studies for quality, and considered the appropriate method of combining the study results. due to the different methods, interventions, and populations used in the studies, the review authors decided it was not appropriate to combine the results in a meta-analysis. efforts were made to identify additional studies through grey literature sources such as conference proceedings and websites, although authors did not explore the possibility of publication bias from a lack of studies from unpublished. finally, the review authors provided an explicit conflict of interest statement.   overall, this is a high-quality systematic review conducted with methodological rigour. however, the decision to include studies based on their reported outcomes may have restricted the number of eligible studies. the authors did not contact the researchers who published the included studies for additional or unpublished data, though while it was not possible to combine the study results statistically, this may not have added anything to the findings of the review. only the medline search strategy was included in full in the supplementary material. the authors indicated that the other search strategies were available on request, so this is a minor limitation in the reporting of the search methods. the review is a useful overview of librarian-provided services in clinical settings. it highlights the need for validated data collection tools to evaluate these services, and the difficulties in combining results from a diverse collection of studies of different interventions. the findings of this review need to be interpreted in this context. it demonstrates the value of librarian-provided services in healthcare settings, such as search skills training, and suggests new areas for research, such as services aimed directly at researchers or patients. however, it is not clear from this review if there are studies that did not report the relevant outcomes, or if there are no studies in this area. in either case, further work is needed to identify and assess these studies, if they exist, or to conduct such studies if they do not yet exist. future systematic reviews could consider a more focussed question and seek to compare individual interventions, such as face-to-face versus online training, to gain a coherent idea of the efficacy of specific elements of librarian-provided services in healthcare settings. references   shea, b. j., grimshaw, j. m., wells, g. a., boers, m., andersson, n., hamel, c.,   . . . bouter, l. m. (2007). development of amstar: a measurement tool to assess the methodological quality of systematic reviews. bmc medical research methodology, 7(10). http://dx.doi.org/10.1186/1471-2288-7-10   news/announcements   call for submissions: information: interactions and impact (i³)      2014. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   call for submissions: information: interactions and impact (i³)   deadline: monday 19 january 2015   i3 focuses on the quality and effectiveness of the interaction between people and information and how this interaction can bring about change. the conference will look beyond the issues of use and accessibility of technology to questions about the way people interact with the information and knowledge content of today's systems and services, and how this might ultimately affect the impact of that information on individuals, organisations and communities.   we invite the submission of high quality papers that report original research or critically discuss underlying methodological issues associated with the main themes of the conference. papers may reflect ongoing or completed research studies and should not have been previously published or be currently under consideration for publication elsewhere. we would particularly welcome papers which address two or more of the following conference themes:   •     the quality and effectiveness of user/information interactions (e.g. information literacies); •     patterns of information behaviour in different contexts (e.g. creativity, ethics, surveillance, ownership, information recycling/reuse); •     the social, cultural and economic impacts of engagement with information, including the assessment of impact; •     the value of information and knowledge as enablers of resilience and change in organisations and communities.   by looking at these issues i3 seeks to influence the development of research towards a fuller understanding of the role of information in a complex, fast-moving information society.   submissions are invited for:   •     full papers (40 minutes duration: 35 minutes, 5 minutes for questions); •     short papers (15 minutes duration: questions at end of session); and •     round table discussions (60-80 minutes duration).   authors who are accepted to give full papers at i3 will also be invited to submit a full paper for consideration by the editorial board of the journal of information science (http://jis.sagepub.com) for a special issue in spring 2016.   contributors should submit abstracts of 1000 words (excluding the list of references) for full papers, and 300-500 words (excluding the list of references) for short papers and round table discussions.  these should be submitted electronically by downloading the online submissions form, located on the call for papers section of the website (http://www.rgu.ac.uk/i3conference2015). this should then be emailed (as an attachment) to i3submissions@rgu.ac.uk. the conference language will be english.   submissions will be anonymously reviewed by two members of the international programme committee specialising in one or more of the conference research streams.  notification of acceptance will be emailed to authors and will also include the comments of the reviewers.   full details of the requirements can be found in the call for papers section of the website. the deadline for receipt of abstracts is 19 january 2015.   for any further information please see http://www.rgu.ac.uk/i3conference2015, or contact the conference team at i3information@rgu.ac.uk.   professor dorothy williams, i³ conference director article   assessing the library’s influence on freshman and senior level outcomes with user surveys   john k. stemmer director of library services bellarmine university louisville, kentucky, united states of america email: jstemmer@bellarmine.edu   david m. mahan executive director of institutional research and assessment manhattan college riverdale, new york, united states of america email: david.mahan@manhattan.edu   received: 14 feb. 2015    accepted: 30 apr. 2015      2015 stemmer and mahan. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objectives – this study seeks to identify areas where relationships exist between a student’s library usage and student outcomes at bellarmine university, a private master’s level institution. the study has two primary aims. the first is to see if an operationally oriented user survey can be used to provide evidence of the library’s support for institutionally important student outcomes. the second is to develop a regression model that provides a big picture with multiple variables to determine if library factors are still significant in student outcomes when controlling for significant demographic factors.   methods – the library regularly conducts student user surveys, and this study examines the results of the first three surveys, from 2007, 2008 and 2010. these surveys include individually identifiable data on why students come to the library and how often they use it in person and online. researchers aggregated student responses into class-based cohorts and used regression analysis to analyze the extent and significance of the relationships, if any, that exist between student use of the library and student outcomes such as retention, graduation and cumulative gpa. the study takes into consideration known significant student demographic factors such as american collect testing (act) composite score, fullor part-time status, and their session gpa.   results – the study identifies specific library services and resources that have significant correlations with the selected student learning measures and outcomes. for freshman students, the ability to access the library online influences both retention and graduation. in looking at freshman learning outcomes represented by gpa, the results again indicate that the library has a positive influence on a student’s gpa. the library’s influence appears through two factors that highlight the library as a place: providing a place to study alone and as a place that has specialized equipment available to students. the library influences seniors’ cumulative gpa differently than for freshmen, primarily through the library’s role as an information resource. the variable check out books had a positive impact on senior’s gpa.   conclusions – this study indicates that the library does have an influence on student outcomes, whether learning outcomes, represented by cumulative gpa, or more typical student success outcomes, represented by second-year retention and graduation. this is true even when controlling for certain demographics, including the student’s act score, whether the student is part-time or full-time, and their session gpa. the factors that influence an individual student’s outcome change depending on the point in time in the undergraduate experience. these statistical analyses provide significant evidence for the value the library provides in support of institutionally important student outcome goals. introduction   at bellarmine university the administration is creating a culture of assessment. as part of this effort, bellarmine university library conducts user surveys of faculty or students as a matter of course, starting in 2007. these surveys are a direct result of the need to provide assessment measures to outside stakeholders, in this case for the southern association of colleges and schools accreditation review. they are also used as a focused supplement to benchmark studies that had already been conducted. in many ways this practice reflects the state of traditional library assessment, in that it is generally input/output oriented and, more recently, operationally oriented. in both of these assessment efforts, benchmarks of inputs/outputs or user satisfaction surveys, whether locally or nationally (e.g., libqual), the assessment has remained focused on the library and its internal operations. they do not provide particularly compelling evidence for other interested parties.   the library’s satisfaction survey is used to gauge the effectiveness of the library’s operations as seen by its users. traditional library assessment has little correlation with the outcomes and positive work of the institution; rather it provides data that indicates strengths and weaknesses on the library’s part that the library administration can then take efforts to further develop or correct as the case may be. operationally these surveys can be helpful to a library administration; they provide valuable data that the library and its resources and services are being used and even appreciated. however, they do not demonstrate to institutional stakeholders the impact of the library on institutional goals and objectives.   increasingly, academic libraries are being asked to provide data and assessments that demonstrate the library’s connection with the institution’s desired outcomes. they are facing increasing demands for accountability to a broader audience than ever before. libraries have moved from being the only information resource to being a quality information resource. in the current economic climate, parent institutions seek to save every dollar they can and face difficult budget choices, often including decreased library budgets. meanwhile, the political climate calls for increased accountability and greater connections between the university’s efforts and student learning and student success. this is the changing environment faced by higher education institutions and therefore academic libraries.   in response, librarians are more frequently seeking to identify areas where they can demonstrate a relationship exists between a student’s library interactions and student outcomes, in contexts relevant to a wider audience at the university and beyond. to accomplish this oakleaf (2010) indicates that the library must link its data to individual students. matthews (2012) has likewise recently noted that success in efforts to link the library and student outcomes will require the use of individual student data. one of the reasons that student unit data is such a constant issue in these efforts by librarians is that generally libraries do not track what resources are used by which students, or how students use library resources and services. as a matter of professional ethics, personally identifiable information is often deleted or not even collected to maintain an individual’s privacy. so while offices of institutional research have access to large amounts of individually identifiable outcome data, most libraries do not.   the need for library assessment efforts that show the influence of the library in terms of institutionally relevant outcomes, such as student learning and student success, is highlighted in the acrl report the value of academic libraries (oakleaf, 2010). much work and many attempts have been made to demonstrate the library’s direct impact on student learning and other student outcomes. recent work includes studies by wong and her colleagues to examine the connections between library material usage and student gpa (wong & webb, 2011), and library instruction and graduating gpa (wong & cmor, 2011). emmons and wilkinson (2011) examine the library’s impact on student persistence. tenopir (2012) reports on multiple methods being used to try and measure library value in her status report on the lib-value project. more thought and effort has been put forth to developing appropriate assessment measures that address the library’s impact on students, particularly the impact on student outcomes and student learning. oakleaf (2011) addresses the challenges librarians face in trying to assess the library’s connection with student learning and outcomes. rodriguez (2011) reports on his efforts to develop one such tool, the protocol for understanding library impacts, to show library impact on student learning, and his first results are promising (rodriguez, 2012). in the uk, the library impact data project has been examining the issue of developing appropriate measures to assess the library’s role in student learning and outcomes, finding significant correlation between library usage and student attainment of the final degree (stone, pattern, & ramsden).   aims   this exploratory study seeks to identify areas where relationships exist between a student’s library usage and relevant student outcomes at bellarmine university. the two primary aims are to see if an operationally oriented user survey can be used to provide evidence of the library’s role in institutionally important student outcomes, and to move beyond the simple, one to one, correlations and use a regression model that provides a bigger picture with multiple variables to determine if, when significant demographic factors are controlled, library factors are still significant in student outcomes.   methods   like many academic libraries, bellarmine university library uses a student user survey as part of its assessment efforts. between 2007 and 2010 it conducted three such surveys, and now does one every other year. these surveys provide data on why students come to the library and how often they use the library. the library survey asked questions in two groupings (see appendix a). the first grouping asks the reasons that students come to the library, with a list of 18 possible reasons with checkboxes. respondents were asked to check all reasons that applied. the second group of questions requests information on how often a student came to the library and how often the student used the library online, with response options ranging from daily to never. most importantly, the survey is not anonymous. the library can identify how students report using the library at an individual level.   researchers paired the library survey data with the university’s office of institutional research student data to first determine if there were any significant relationships between a student’s self-reported library usage and known student outcomes. they used a logistic regression of all the library input variables against selected student outcomes: retention, graduation, and gpa. they also determined that the data would be viewed in undergraduate class-based cohorts to minimize variations among the respondent experiences. in addition to the library variables, researchers tested a number of demographic factors drawn from institutional research data against selected student outcomes to identify the most significant demographic factors for the individual cohorts. using these control factors and the library variables the researchers developed three research questions:   does library usage influence whether a freshman student returns in his/her next year of undergraduate study? does library usage influence whether a freshman student graduates within four or five years in undergraduate study? does library usage influence cumulative gpa for freshmen and seniors?   undergraduate students participated in library surveys in the years 2007, 2008 and 2010. bellarmine university is a small, private, catholic university located in louisville, kentucky. total university enrollment is approximately 2,000 undergraduate students and nearly 800 graduate students. there were over 1,000 students living in the residence halls. the institution currently offers over 50 undergraduate bachelor degree programs and over 20 graduate programs. over 80% of bellarmine’s undergraduate students attend full-time and are under 25 years of age.   bellarmine university has an incoming class of about 600 students every fall. the following demographic and academic preparedness measures are consistent across freshman cohorts:   24-25 act composite average 65% from kentucky, 35% out of state 40% first-generation students (defined as neither parent earning a bachelor’s degree) 20-30% pell grant recipients 10-15% students of colour 1% international students   due to these similar demographics across the student body, researchers collapsed the survey responses for 2007, 2008, and 2010 and studied the data as class-based cohorts.   due to the limited number of undergraduate students at the institution, sampling was unnecessary because the entire population could easily be requested to participate. on designated years, the institution invited the undergraduate populations to participate in the library survey. most members of the freshmen and senior populations start at the university as first-time, traditional age freshmen; however, all freshman and senior students were invited to participate in the survey, regardless of whether they enrolled as traditional freshmen, transfer students, or re-admitted students. the response rate for the three surveys ranged from a low of 20% to a high of 26%.   results   the variables considered, full-time/part-time status, session gpa, race, and act composite score, either singly or in combinations, were identified by a logistic regression as having a significant relationship to outcome variables, whether a student graduated or the cumulative gpa (p<.05). these variables were controlled in our analyses to consider whether library usage variables independently influenced student outcomes (see appendix b).   analysis one: second-year retention of freshman students   does library usage influence whether a freshman student returns in his/her next year of undergraduate study?   researchers conducted a forward entry logistic regression using freshmen student data only (n=370), considering any significant library variables related to outcome variable, retained in a students’ second year. there were 336 freshmen returning in the second year (91%) and 34 students did not (9%). the omnibus test of model coefficients was significant, with a chi-square test result of 8.227 (p<.05). one library variable was determined to be a significant positive predictor of returning for the second year: access library online (see table 1). the nagelkerke r-squared was 0.048.     researchers conducted second logistic regression in two steps. first, by identifying the control variables full-time/part-time status, act composite score, and session gpa, which were selected because correlational analysis revealed significance relationship to return (p<.05). second, they used a forward entry technique to consider library variables after the control variables. the omnibus test of model coefficients was significant, with a resulting chi-square of 31.021 (df=3, p<.001). they found that session gpa significantly predicted return. after the control variables were entered, the library variable access library online again significantly predicted whether a student returned in their second year using the same forward entry method (p<.05) (see table 2). the nagelkerke r-squared was 0.206.     table 1 significant library variables related to retention     measure      b s.e.  wald df sig. exp(b)       access library online .432  .158 7.504  1 .006 1.54       constant 1.242  .388 10.236  1 .001 3.463           table 2 significant control and library variables related to retention            b  s.e.   wald  df sig. exp(b) ft or pt 21.868 40192.7 .000 1    1      .000 act .074 .067 1.238 1   .266 1.077 session gpa .998 .224 19.859 1   .000 2.712 access library online .311 .157 3.947 1    .047 1.365 constant 18.721 40192.7 000 1     1 1.4e+08     table 3 significant library variables related to graduation   b s.e. wald df sig. exp (b) access library online .245 .117 4.411 1 .036 1.277 constant .240 .335 .513 1 .474 1.271     analysis two: graduation of freshman students from the same institution   does library usage influence whether a freshman student graduates within four or five years in undergraduate study?   researchers conducted a forward entry logistic regression using freshmen student data only (n=220, after combining freshmen from surveys in 2007 and 2008), considering any library variables related to outcome variable to identify any significant library variables related to graduation. this included respondents who graduated within four years for 2008 freshman surveys or within five years for 2007 freshman surveys. respondents were predominantly traditional-aged freshmen, and primarily full-time students. there were 156 freshmen that graduated (71%) and 64 students did not (29%). the omnibus test of model coefficients was significant, with a resulting chi-square of 4.584 (p<.05). researchers found that one library variable, access library online (p<.05), was a significant predictor of graduation it was a positive predictor, and confirmation of this was associated with students graduating (see table 3). the nagelkerke r-squared was .029.    researchers conducted a second logistic regression in two steps. first using the following control variables: full-time/part-time status, act score, session gpa, and race, which were selected because correlational analysis revealed significance relationship to graduate (p<.05). second, using a forward entry technique to consider library variables after including the demographic control variables. the omnibus test of model coefficients was significant, with a chi-square result of 37.943 (p<.001). after entering the control variables, the library variable access library online still significantly predicted whether a student graduated, using the forward entry method (p<.05) (see table 4). the nagelkerke r-squared was 0.255.    analysis three: first-year cumulative gpa   does library usage influence cumulative gpa for freshmen?   researchers conducted an analysis of freshman student data only (n=370), considering any significant variables correlated with the outcome variable cumulative gpa. they employed a forward entry ordinary least squares (ols) regression to consider what variables, if any, predict cumulative gpa. researchers identified two significant library variables, study alone and use of printer or photocopier, as positive predictors, for agreement with them was associated with a higher cumulative gpa. the adjusted r-squared was .030. controlling for the significant non-library variable, act composite, entering it in the model before library survey items, the r-squared is .15 for the act variable; however study alone and use of printer and photocopier still present themselves as significant factors for cumulative gpa (see tables 5 and 6).     table 4 significant control and library variables related to graduation   b s.e. wald df sig. exp(b) ft or pt 22.999 40193.2 .000 1 1 9.74e+09 act 0.065 0.054 1.44 1 0.23 1.067 session gpa 1.09 0.282 14.892 1 0 2.974 race 0.954 0.437 4.766 1 0.029 2.595 access library online 0.262 0.131 3.986 1 0.046 1.299 constant -28.586 40193.2 .000 1 0.999 .000     table 5 control and library variables related to first-year gpa std. error of the estimate change statistics model r r square adjusted r square r square change f change df1 df2 sig. f change act 0.396 0.157 0.154 0.561 0.157 67.240 1 362 0.000 act, study alone 0.426 0.182 0.177 0.553 0.025 11.014 1 361 0.001 act, study alone, use printer or photocopier 0.437 0.191 0.184 0.551 0.009 4.005 1 360 0.046     additional library survey items were not significant in the model, but had significant positive correlation with students’ cumulative gpa at the end of the their freshman year (p<.01):   ·         access library online ·         use computer for academic purposes ·         consult a reference librarian   analysis four: senior cumulative gpa   does library usage influence cumulative gpa for seniors?   researchers conducted an analysis of senior student data only (n=360), considering any significant library variables correlated with the outcome variable cumulative gpa.  researchers used a forward entry ordinary least squares (ols) regression to consider what variables, if any, predict cumulative gpa. they identified two significant library predictors: check out books and use of group study rooms. check out books was a positive predictor, and confirmation of this was associated with a higher cumulative gpa. use of group study rooms was a negative predictor. the adjusted r-squared was .045. when the control variable act composite is entered in the model before library survey items, the r-squared is .275 for the act variable; however while check out books still presents as significant, use of group study room is no longer a significant factor (see tables 7 and 8).     table 6 first-year gpa model coefficients unstandardized coefficients standardized coefficients model b std. error beta t sig. (constant) 1.443 0.218 6.617 0.000 act 0.071 0.009 0.396 8.200 0.000 (constant) 1.243 0.223 5.565 0.000 act 0.070 0.009 0.391 8.198 0.000 study alone 0.265 0.080 0.158 3.319 0.001 (constant) 1.201 0.224 5.372 0.000 act 0.069 0.008 0.385 8.099 0.000 study alone 0.231 0.081 0.138 2.845 0.005 use printer or photocopier 0.133 0.066 0.097 2.001 0.046     table 7 control and library variables related to senior gpa std. error of the estimate change statistics model r r square adjusted r square r square change f change df1 df2 sig. f change act 0.525 0.275 0.273 0.353 0.275 104.147 1 274 0.000 act, check out books 0.540 0.292 0.287 0.350 0.016 6.333 1 273 0.012     table 8 senior year cumulative gpa model coefficients unstandardized coefficients standardized coefficients model b std. error beta t sig. (constant) 1.780 0.155 11.456 0.000 act 0.063 0.006 0.525 10.205 0.000 (constant) 1.743 0.155 11.276 0.000 act 0.062 0.006 0.513 10.029 0.000 check out books 0.111 0.044 0.129 2.516 0.012     discussion   the analysis indicates that student usage of the library manifests itself in a number of ways, including use as an academic or information resource, use as an information service, as a place to study alone or in a group, a place to use equipment available in the library, and for social reasons. many of these have previously appeared in the library literature as recognizable parts of a library’s offerings to the university community. the results illustrate that library factors consistently show a significant relationship to student outcome variables, even when control factors are considered.   the study’s findings confirm other studies that link the use of library services and resources to a student’s learning and success outcomes. wong and webb (2011) identified a single aspect of library usage (checking out books) that correlates with a student outcome (gpa). for seniors, checking out books was a significant predictor in the model for their gpa. emmons and wilkinson (2011) identify that the ratio of professional staff to full-time students has an impact on student persistence. this study also found that the library has an influence on freshman retention and graduation, through the use of online resources.   in examining the library’s influence on student success outcomes of second-year retention and graduation, this study indicates that the library’s support does favourably influence student success. in looking at second-year retention of freshmen, accessing the library online was identified as the most significant library factor and even after controlling for significant demographic variables, this factor was still identified by the model as significant influence on retention. for freshman students the same factor influenced both retention and graduation: the availability of online access to the library. freshman students that accessed the library online more frequently were more likely to return for their second year and to graduate. using the library’s academic information resources appears to favourably influence a freshman’s student success outcomes.   in looking at student learning outcomes represented by gpa, the results again indicate that the library does have a consistent and positive influence on a student’s gpa, though specific library factors change depending on where in their academic career the student is located. for first-year students the library’s influence appears through two variables that highlight the library as a place: providing a place to study alone and as a place that has specialized equipment available to students. in a sense, as new students they are getting acclimated to the place of the library in their academic efforts. interestingly, the impact of studying alone repeats a finding by arum and roska (2011) showing that time spent studying alone is more academically beneficial than time spent studying in groups. seniors cumulative gpa was influenced by the library differently than first-year students. checking out books had a positive impact on senior’s gpa. interestingly using group study rooms, which had a negative effect when considering the library variables alone, was not significant when control variables were entered. the consistent presentation of library variables after the use of significant control variables shows that library factors do have significant connections to student outcomes and success.   limitations   researchers employed a number of different non-library control variables in this study, but there are many that were not considered. the researchers wanted a “bigger picture” view of the library’s relationship with student outcomes more than a one-to-one correlation, but this is not intended to imply causation. it should be noted that this study relies on self-reported data, the value of which has recently been challenged (porter, 2011). it is also worth noting that this study focuses on acquiring factual information on specific activities of the students. this study does not seek information on students’ perception of ability or growth over time, merely what students were doing the year the survey was taken. it was designed within the parameters outlined by gonyea (2005) as an appropriate means of creating and using self-reported data.     conclusion   not surprisingly, not all correlations identifying the library’s influence on student outcomes were found to be significant when using regression analysis to further examine the relationship in conjunction with non-library control variables. the study confirms previous correlation studies that identify the library as contributing to student outcomes, and indicates that the library’s relationship with student outcomes is not eliminated by control variables. the library has an influence on first-year and senior student outcomes but does not affect them in the same manner. the study indicates that the library provides support for students in a number of ways: as a resource, through services, and as a place. all of these aspects of the library’s operations are shown to be influential on student outcomes, but no single aspect is consistent from year to year as an individual students progress through their studies. rather, as individual students develop they seem to rely on different aspects of the library offerings and their use of the library in order to succeed.   this study indicates that the library does have an influence on freshman and senior level outcomes, whether expressed as learning outcomes represented by cumulative gpa, or more typical student success outcomes, such as retention and graduation rates. this is true even when controlling for certain demographic characteristics, including the student’s act score, whether the student is part-time or full-time, and their session gpa. the factors that impact a student’s outcomes changes depending on where in their academic career a student is located. these statistical analyses provide significant evidence for the value provided by the library in support of institutionally important student outcome goals. additional study into what students identify as different library factors over time would be useful. another area for further research would be to consider if there are any differences in library usage factors based on the student’s field of study.   references   arum, r., & roksa, j. (2011). academically adrift: limited learning on college campuses. chicago, ill: university of chicago press.   emmons, m., & wilkinson, f. c. (2011). the academic library impact on student persistence. college & research libraries, 72(2), 128-149. http://dx.doi.org/10.5860/crl-74r1   gonyea, r. m. (2005). self-reported data in institutional research: review and recommendations. new directions for institutional research, 2005(127), 73-89.  http://dx.doi.org/10.1002/ir.156   matthews, j. r. (2012). assessing library contributions to university outcomes: the need for individual student level data. library management, 33(6/7), 389-402.  http://dx.doi.org/10.1108/01435121211266203      oakleaf, m. (2010). the value of academic libraries: a comprehensive research review and report. chicago, il: association of college and research libraries.   oakleaf, m. (2011). demonstrating the value of the library: assessment tools & techniques. paper presented at the american library association annual conference new orleans, la. http://meganoakleaf.info/ala2011valuepanel.pdf porter, s. r. (2011). do college student surveys have any validity? review of higher education, 35(1), 45-76. doi: http://dx.doi.org/10.1353/rhe.2011.0034   rodriguez, d. (2011). understanding library impacts on student learning. in the library with the lead pipe. retrieved from: http://www.inthelibrarywiththeleadpipe.org/2011/understanding-library-impacts-on-student-learning/   rodriguez, d. (2012). answering questions about library impact on student learning. in the library with the lead pipe. retrieved from: http://www.inthelibrarywiththeleadpipe.org/2012/answering-questions-about-library-impact-on-student-learning/   stone, g., pattern, d., & ramsden, b. (2011). does library use affect student attainment? a preliminary report on the library impact data project. liber quarterly, 21(1), 5-22. doi: http://liber.library.uu.nl/index.php/lq/index   tenopir, c. (2012). the lib-value project: status report. paper presented at the american library association annual conference, anaheim, ca. http://libvalue.cci.utk.edu/sites/default/files/arl%20alatenopir20126%2022%20rev%20final.pdf   wong, s. h. r., & cmor, d. (2011). measuring association between library instruction and graduation gpa. college & research libraries, 72(5), 464-473. doi: http://dx.doi.org/10.5860/crl-151   wong, s. h. r., & webb, t. d. (2011). uncovering meaningful correlation between student academic performance and library material usage. college & research libraries, 72(4), 361-370.  http://dx.doi.org/10.5860/crl-129      appendix a 2010 library usage survey   reasons you come to the library (check all that apply): to check out books to check out media (cds, dvds, etc.) to locate journal/newspaper articles to get help with research papers or other course assignments to read newspapers or current magazines to use items (books/articles/videos) placed on reserve by your professor to use media equipment (e.g. video cameras, digital cameras, scanners, video editing, video viewing) to study alone to study with a group to use the group study rooms to use a printer or photocopier to use the computers for academic purposes to use the computers for recreational/personal use to use the mac lab (apple macintosh computers) to visit the help desk to visit the academic resource center (arc) to visit the merton center to use a laptop to meet friends to look for information in online databases (ebscohost, proquest etc.) other (please specify)   if you never use the library, why don't you?   please rate the following:               on average, how often do you use the library in person? daily 2 to 4 times a week once a week 2 to 3 times a month once a month or less never               on average, how often do you access library materials, services and databases (such as proquest and ebscohost) without visiting the library? daily 2 to 4 times a week once a week 2 to 3 times a month once a month or less never                   appendix b variables considered   demographic and academic preparedness   act composite (max. 36) 1st generation college student (y/n) session grade point average (gpa) pell recipient (y/n) athlete (y/n) race (white, black, hispanic, pacific islander/native hawaiian, asian, american indian/alaska native, multi-race, unknown) sex (f, m) high school type (public or private) radius (miles from home) full-time or part-time student   library usage variables   check out books locate articles  read newspapers or magazines use items on reserve study alone study with a group use the group study rooms use printer or photocopier use computers for academic purposes visit the help desk visit the academic resource center (arc) visit the merton center use a laptop use the computers for personal use meet friends use library at all use library in person access library online   outcome variables   graduated in four or five years (y/n) returned in the second year (y/n) cumulative gpa in freshman year (0-4.0) cumulative gpa in senior year (0-4.0)     ebl 101   riding into the sunset   virginia wilson director, centre for evidence based library and information practice (c-eblip) university library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   received: 2 may, 2015   accepted: 13 may 2015      2015 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   time flies when you’re having fun and when you’re writing the ebl 101 column! and with that pithy sentiment, i’m pleased to let you know that this is the final ebl 101 column in evidence based library and information practice (eblip). i’m pleased because ebl 101 has had an exciting run spanning eight years! it has become a go-to resource for those beginning to think about evidence based library and information practice (eblip) and for overviews of research methods. if the citations of the research methods columns are any indication, scholars from within library and information studies (lis) and without have found value in the overviews. i’m also pleased for another reason that i’ll get to in a moment.   i started writing the ebl 101 column in 2009 (vol 4, no 1), continuing the steps of eblip.   when those steps were finished, i moved on to writing concise overviews of research methods, including examples of the methods from the lis literature and resources for further information as there’s only so much one can do in 2 or 3 pages of a column. here’s an overview of all the columns as a whole. aside from the first two ebl 101 columns, the rest where written by me: ebl 101: steps of eblip   ·         vol. 3, no. 3 (2008) evidence based librarianship backgrounder – su cleyle, julie mckenna ·         vol. 3, no. 4 (2008) asking the right question – lorie kloda   ·         vol. 4, no. 1 (2009) matching question types to study designs ·         vol. 4, no. 2 (2009) looking to the literature: domains to help determine where to look ·         vol. 4, no. 3 (2009) looking to the literature: open access and free sources of lis evidence ·         vol. 4, no. 4 (2009) conducting your own research: something to consider   ·         vol. 5, no. 1 (2010) an introduction to critical appraisal ·         vol. 5, no. 2 (2010) applicability: what is it? how do you find it? ·         vol. 5, no. 3 (2010) evaluating the results of evidence application, part one ·         vol. 5, no. 4 (2010) evaluating the results of evidence application, part two: at the practice level   ·         vol. 6, no. 1 (2011) disseminating your research ebl 101: research methods   ·         vol. 6, no. 2 (2011) a new path: research methods ·         vol. 6, no. 3 (2011) design, methods, case study...oh my! ·         vol. 6, no. 4 (2011) content analysis ·         vol. 7, no. 1 (2012) focus groups ·         vol. 7, no. 2 (2012) interviews ·         vol. 7, no. 3 (2012) bibliometrics ·         vol. 8, no. 1 (2013) altmetrics ·         vol. 8, no. 2 (2013) mixed methods research ·         vol. 8, no. 3 (2013) systematic reviews ·         vol. 8, no. 4 (2013) action research ·         vol. 9, no. 1 (2014) triangulation ·         vol. 9, no. 2 (2014) sampling ·         vol. 9, no. 3 (2014) the most significant change technique ·         vol. 9, no. 4 (2014) scoping studies   quite the list! there are plans in the works for a "virtual issue" that would bring all the ebl 101 columns together in one place for readers and we’ll keep you posted about that. also (and this is the part i’m really pleased about), on the horizon and taking ebl 101’s place will be a new column that i’m very excited to begin research in practice. this column will be designed for practicing librarians who are using research in their practice, conducting research for their practice, and otherwise interested in the varied and vast topics pertaining to research in practice. the scope of the new column is broad, allowing for a variety of topics to be explored in a number of ways. there could be interviews, guest columnists, co-authored columns...it boggles the mind!   i would like to thank my past editors of the ebl 101 column, denise koufogiannakis and alison brettle, for their patience, help, and guidance, and for the opportunity to explore my thoughts in a column format. and thank you to lorie kloda, eblip’s new editor-in-chief, for being open to the idea of reconfiguring the column. eblip’s editorial board has had some exciting ideas for the column and i look forward to continuing in this capacity with the journal. look for research in practice in the next issue of eblip.   ebl 101   research methods: scoping studies   virginia wilson director, centre for evidence based library and information practice (c-eblip) university library university of saskatchewan saskatoon, saskatchewan, canada email: virginia.wilson@usask.ca   received: 2 nov. 2014    accepted: 16 nov. 2014      2014 wilson. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   this time around, i’m going to take a look at scoping studies or scoping reviews. a scoping study consists of a fairly comprehensive search of the literature around a particular topic. when i came across this methodology, my first question was, “how are these different from systematic reviews?” which i’ve written about previously (wilson, 2013). as i looked deeper, i’ve discovered that a scoping study seems to sit somewhere between a literature review and a systematic review and is “one method among many that might be used to review literature” (arksey & o’malley, 2005, p. 20). this column will provide an overview of the scoping study methodology, some further reading on the subject, and some citations of examples of scoping studies in library and information studies.   but what exactly is a scoping study? a number of definitions have been put forward and several have been collected in a paper by levac, colquhoun, and o’brien (2010). for this column, i’m going to use the definition quoted by arskey and o’malley (2005): a scoping study aims “to map rapidly the key concepts underpinning a research area and the main sources and types of evidence available” (mays, roberts & popay, 2001). levac, colquhoun, and o’brien (2010) also include a definition from the canadian institutes of health research which states that scoping studies “are exploratory projects that systematically map the literature available on a topic, identifying the key concepts, theories, sources of evidence and gaps in the research” and are often “preliminary to full syntheses” (p. 2 of 9). the name of this methodology has been a bit confusing, as it has been referred to as “scoping study”, “scoping review”, “scoping literature review”, and “scoping exercise” in various studies. perhaps its relative newness as a defined methodology means that standard terminology has yet to be adopted.     table 1 adapted from arksey & o’malley (2005) and grant & booth (2009) systematic reviews scoping studies ·     focus on a well-defined research question ·    address broader topics ·     specific study designs can be identified prior to searching based on the question ·    different study designs may be applicable and included ·     attempt to provide answers from a narrow range of “quality assessed studies” (arksey & o’malley, p. 20) ·    less likely to assess quality of studies included ·     the goal is thorough, comprehensive searching ·    how complete the searching is depends on time and scope constraints     in order to illustrate how a scoping study differs from a systematic review, i have put together a table (table 1).   the literature outlining and advancing the methodology of scoping studies is fairly recent. arksey and o’malley (2005) presented a framework that they adopted for the undertaking of a scoping study (p.22), to which i have added clarifying points:   stage 1: identifying the research question – this guides the development of search strategies stage 2: identifying relevant studies – the depth of this often depends of time and budget constraints stage 3: study selection – some search results will not be applicable to the research question stage 4: charting the data – material is sorted according to key issues and themes stage 5: collating, summarizing and reporting the results   an optional, although strongly recommended, stage 6 was also proposed, consisting of a consultation exercise “to inform and validate findings from the main scoping review” (p. 23). levac et al (2010) also recommend incorporating this stage, as it “adds methodological rigor and should be considered a required component” (p. 7 of 9).  stage 6 involves consultations with stakeholders who may be able to “provide additional references about potential studies to include in the review as well as valuable insights” about other issues pertinent to the review (arksey & o’malley, 2005, p. 29). in 2013, daudt, van mossel, and scott published an article further enhancing the methodology from the perspective of a large, inter-professional team’s experience using the original arksey and o’malley steps. they too agree that stage 6 should be a fully incorporated rather than optional step.   why might one conduct a scoping study? arskey & o’malley (2005) offer up four reasons:   1.       to examine the extent, range and nature of research activity... [and when] mapping fields of study 2.       to determine the value of undertaking a full systematic review... 3.       to summarize and disseminate research findings... 4.       to identify research gaps in the existing literature... (pp. 21-22)   a scoping study is a viable and useful methodology for a rapid scan of literature on a specific topic. i strongly recommend delving into the papers that i have consulted for this column; these are listed in the reference list. and, if you do conduct a scoping study, you might consider contributing your own experience to the literature as well.  to get a broad overview of the many review strategies you might explore, check out the article by grant and booth (2009).   the following are some examples of the scoping study methodology used in library and information studies.   ·         baxter, g. j. & connolly, t.m. (2014). implementing web 2.0 tools in organisations: feasibility of a systematic approach. the learning organization, 21(1), 6-25. http://dx.doi.org/10.1108/tlo11-2012-0069 ·         gardois, p., colombi, n., grillo, g., & villanacci, m.c. (2012). implementation of web 2.0 services in academic, medical and research libraries: a scoping review. health information and libraries journal, 29, 90-109. http://dx.doi.org/10.1111/j.1471-1842.2012.00984.x ·         norwood, j. & skinner, b. (2012). implementing rfid in a hospital library: a scoping study. health information and libraries journal, 29(2), 162-165. http://dx.doi.org/10.1111/j.1471-1842.2012.00987.x ·         younger, p. (2010). internet-based information-seeking behaviour amongst doctors and nurses: a short review of the literature. health information and libraries journal, 27(1), 2-10. http://dx.doi.org/10.1111/j.1471-1842.2010.00883.x   references   arksey, h. and o’malley, l. (2005). scoping studies: towards a methodological framework. international journal of social research methodologies, 8(1), 19-32. http://dx.doi.org/10.1080/1364557032000119616   daudt, h. m. l., van mossel, c., & scott, s. j. (2013). enhancing the scoping study methodology: a large, inter-professional team’s experience with arksey and o’malley’s framework. bmc medical research methodology, 13(48). http://dx.doi.org/10.1186/1471-2288-13-48   grant, m. j. & booth, a. (2009). a typology of reviews: an analysis of 14 review types and associated methodologies. health information and libraries journal, 26, 91-108. http://dx.doi.org/10.1111/j.1471-1842.2009.00848.x   levac, d., colquhoun, h., & o’brien, k. k. (2010). scoping studies: advancing the methodology. implementation science, 5(69). http://dx.doi.org/10.1186/1748-5908-5-69   mays. n., roberts, e., popay, j. (2001). synthesizing research evidence. in: fulup, n., allend, p., clarke, a., black, n. (eds). studying the organisation and delivery of health services: research methods (pp.188-220).
london: routledge.   wilson, v. (2013). research methods: systematic reviews. evidence based library and information practice, 8(3), 83-84. retrieved from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/20437/15740   article   electronic resource availability studies: an effective way to discover access errors   sanjeet mann arts and electronic resources librarian armacost library university of redlands redlands, ca email: sanjeet_mann@redlands.edu   received: 18 aug. 2014  accepted: 19 june 2015      2015 mann. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – the availability study is a systems research method that has recently been used to test whether library users can access electronic resources. this study evaluates the availability study’s effectiveness as a troubleshooting tool by comparing the results of two availability studies conducted at the same library before and after fixing access problems identified by the initial study.   methods – the researcher developed a six-category conceptual model of the causes of electronic resource errors, modified nisonger’s e-resource availability method to more closely approximate student information-seeking behaviour, and conducted an availability study at the university of redlands armacost library to estimate how many resources suffered from errors. after conducting troubleshooting over a period of several months, he replicated the study and found increased overall availability and fewer incidences of most error categories. he used z tests for the difference of two proportions to determine whether the changes were statistically significant.   results – the 62.5% availability rate in the first study increased after troubleshooting to 86.5% in the second study. z tests showed that troubleshooting had produced statistically significant improvements in overall availability, in the number of items that could be downloaded from the library’s online collection or requested through interlibrary loan (ill), and in three of six error categories (proxy, target database and ill).   conclusion – availability studies can contribute to successful troubleshooting initiatives by making librarians aware of technical problems that might otherwise go unreported. problems uncovered by an availability study can be resolved through collaboration between librarians and systems vendors, though the present study did not demonstrate equally significant improvements across all types of errors. this study offers guidance to librarians seeking to focus troubleshooting efforts where they will have the greatest impact in improving access to full-text. it also advances the availability research method and is the first attempt to quantify its effectiveness as a troubleshooting tool. introduction   electronic resource failure is a significant and multifaceted problem for libraries. encountering an error frustrates library users, and e-resource errors cost the library in terms of lost subscription value and staff time spent troubleshooting problems. inaccessible resources cause patrons to place unnecessary interlibrary loans or settle for lower-quality information sources that are readily available. since unavailable subscriptions do not accumulate usage statistics, they bias a common measure used by libraries to gauge the value of their electronic collections. more insidiously, persistent problems with electronic resources undermine staff confidence in the reliability of their own systems, and undercut the library’s image in the eyes of the administrators and funding stakeholders to which libraries hope to demonstrate their value. even when total breakdowns in access do not occur, resources may still suffer from such issues as sub-optimal interface design. the impact is felt in library instruction when instruction librarians emphasize search interfaces and error workarounds, taking valuable classroom time away from discussion of how to develop research topics and evaluate sources.   when resources fail, libraries usually turn to their systems or electronic resource units for a solution. depending on the size of the library, this may be a team or a single individual. some errors can be fixed in-house, while others require collaboration with systems vendors – and all the while, the patron is waiting. given the time-consuming nature of technical troubleshooting, it is advantageous for libraries to identify the most significant access problems and proactively address what can be fixed immediately.   problems are often discovered one at a time through interaction with users during a class or at the reference desk, or because other library staff stumbled across the problem during their regular workflows. conducting evidence-based troubleshooting by auditing resources through an availability study can help libraries take a more proactive approach to identifying and solving problems.   literature review   libraries have used availability studies for decades to evaluate their ability to provide patrons with desired materials. mansbridge (1986) and nisonger (2007) have published review articles describing the development of the availability study. this systems analysis research method uses a sample of items to estimate the proportion of the library’s collection that users can immediately access. researchers can obtain samples by interacting directly with patrons (“real” availability) or by using their judgment to compile a list of items that approximates patron needs (“simulated” availability). early availability studies often involved surveying library users to find out which books they wanted during a library visit but could not find (gaskill, dunbar, & brown, 1934). library staff then searched for those books themselves and categorized the reasons why they could not be found: the library never purchased a copy, the books were checked out or misshelved, the patron looked in the wrong place, etc. kantor (1976) and saracevic, shaw and kantor (1977) used binomial probability statistics to prioritize these reasons according to how often they occurred, and depicted the results in a “branching” diagram, making it easier for libraries to act on the findings. overall, mansbridge and nisonger reported 60% average availability across the studies they reviewed.   more recently, the availability technique has been applied to study access to electronic resources. nisonger (2009) conducted the first known electronic resource availability study by creating a list of 50 scholarly journals that he considered to reflect the curriculum of indiana university. from a handful of recently published articles in each journal, he randomly selected 10 citations to other journal articles, and tried to obtain the full-text of each citation from the library catalogue or a search box on the library website tied into its ex libris sfx knowledge base. nisonger found the full-text of 65.4% of these 500 citations in the indiana university libraries’ electronic collections. acquisitions “errors” in which the library did not hold a subscription, or the holdings entitlement did not include the citation being tested, were the most common reasons for nonavailability. crum (2011) used the catalogue and link resolver at oregon health & science university library to test a sample of 414 citations requested by patrons and recorded in the library’s resolver log file. she found just under 80% availability and observed that the link resolver was a special point of failure.   link resolver performance is also the focus of a related body of research involving the classification of openurl errors in order to improve the systems that manage electronic resources. wakimoto, walker and dabbour (2006)(2006) conducted a mini-availability study by running 224 likely searches in abstracting databases at california state university northridge and san marcos, as part of a mixed-methods project that also included surveys of sfx users and librarian focus groups. the availability study shed quantitative light on the dissatisfaction with sfx expressed in the user surveys, demonstrating that sfx gave erroneous results 20% of the time at the two campuses. trainor and price (2010) found linking errors 29% of the time in a similar study conducted at eastern kentucky university and the claremont colleges. university of texas southwestern medical library researchers jayaraman and harker (2009) tested 380 randomly selected citations from a&i databases where full-text was known to exist in another subscribed resource, and found that ebsco linksource failed to make the connection 9% of the time (above their target 5% goal). chen (2012) categorized 432 linking errors reported by bradley university patrons over a four year period, and found that the most common reasons for link failure involved missing content, incorrect metadata, or knowledge base collections that did not support article level linking. finally, stuart, varnum and ahronheim (2015) analyzed 430 user-reported errors and randomly tested over 2,000 openurls from university of michigan link resolver log files over a three year period, concluding that openurls failed with a discouraging 20% frequency.   research into openurl failure provides compelling evidence that e-resource linking sorely needs improvement. the national information standards organization (niso) has established two relevant working groups to coordinate efforts among librarians, publishers and vendors to address these problems. the improving openurl through analytics (iota) initiative built off adam chandler’s earlier research into the relationship between missing metadata elements and openurl failure (chandler, leblanc, & wiley, 2011; pesch, 2012). iota developed a “completeness index” to measure the quality of link metadata and recommended essential fields for content providers to include (kasprowski, 2012; niso, 2013). meanwhile, the knowledge bases and related tools (kbart) initiative recommended a data format and best practices for content providers to use when sending metadata to a knowledge base (culling, 2007; niso & uksg, 2010, 2014). the research involved in the creation of these standards has greatly advanced librarians’ understanding of openurl linking failure; universal compliance with the standards could rectify many linking errors, thereby increasing electronic resource availability for all libraries.   in addition to traditional availability studies and openurl accuracy studies, a third type of research – usability studies – also provides insight into why users cannot obtain the electronic resources they seek. ciliberti, radford, radford and ballard (1998) established an early connection between availability research and user studies when they included “catalog use failures” and “user retrieval failures” in their branching diagram. their study of patron searches at adelphi university libraries attributed one third of unsuccessful searches to difficulty searching the catalogue or finding items on shelf. later, as librarians became aware of usability research methods, they applied these techniques to test patron access to electronic resources. to note but a few examples from this ample literature: cockrell and jayne (2002) asked students to find e-resources using library systems; cummings and johnson (2003) observed students using openurl linking; wrubel (2007) offered a concise overview of e-resource usability testing; o’neill (2009) considered how best to instruct patrons in openurl linking based on usability results; next generation catalogues (majors, 2012) and discovery services (asher, duke, & wilson, 2013; fagan, mandernach, nelson, paulo, & saunders, 2012; williams & foster, 2011) have been studied thoroughly; kress, del bosque and ipri (2011) conducted a study to find out why students placed unnecessary ill requests; and imler and eichelberger (2014) investigated how confusing vocabulary acts as a barrier to full-text. considered as a whole, this usability research demonstrates librarians’ awareness that the problems leading to full-text nonavailability are complex, arising from both library systems and human error.   the question of why users cannot obtain the sources they need is persistent and vexing. researchers have improved their understanding of this issue by conducting availability studies, link failure studies, and e-resource usability studies. as a result, the traditional availability technique can now be modified to better track linking errors, while accounting for real-life user behaviours.   since this research method was only recently adapted to measure electronic resource availability, gaps exist in the literature. most availability studies were conducted at libraries with intensive research collections, leaving smaller libraries without peers to benchmark against. additionally, no researchers have conducted paired availability studies before and after troubleshooting, as a way to quantify the method’s effectiveness at detecting errors. finally, all electronic resource availability studies published to date have been simulated studies that did not measure availability as experienced by actual library users.   aims   the present study had a threefold purpose: 1) document electronic resource availability at a smaller academic library; 2) update the availability technique to reflect librarians’ current understanding of why electronic resources fail and how students search for information; and 3) determine whether troubleshooting efforts informed by an availability study can produce a statistically significant improvement in full-text access.   the present study took place at armacost library, university of redlands. the user population is approximately 4,800 full time equivalent (fte) students, faculty and staff. the library’s annual acquisitions budget is $950,000; the physical collection includes just under 500,000 volumes; and the e-resource knowledge base tracks 75,000 unique titles.   mansbridge (1986) and nisonger (2007) commented that availability researchers have historically used inconsistent methods, making it difficult to compare results from different studies. thus, it is important to discuss the rationale behind the researcher’s methodological choices in the present study.   the present study, like nisonger’s, relies on a judgment sample rather than a randomly selected sample of citations, and so introduces some risk of sampling bias. statistical validity is a unique attribute of availability studies, but truly random samples would not reflect the way library users interact with electronic resources. the present study derives the sample from student research topics, since students often begin their research with a topic rather than a list of sources.   the researcher also chose to search topics in abstracting and indexing (a&i) databases commonly taught to students at armacost library instruction sessions rather than full-text databases, in order to include openurl linking to full-text as part of the study. he chose to run searches as simple keyword searches and tested access to only the first screen of search results in keeping with findings from the ethnographic research in illinois academic libraries (erial) study about how students interact with databases (duke & asher, 2012, pp. 76, 80). he classified situations where the link resolver did not connect directly to full-text as an error, since students expect direct linking to full-text (connaway & dickey, 2010; trainor & price, 2010; stuart et al., 2015). finally, he chose to test items for availability in the library’s electronic collection first, followed by the library’s physical collection if no electronic access was present, and finally via interlibrary loan if no access through the library’s local collection was present. these resources take increasingly longer amounts of time to return full-text, and students prefer to take the quickest route possible (connaway & dickey, 2010).     figure 1 full-text availability model     the researcher diverged from traditional practice in the availability literature by classifying nonlocal items as available through interlibrary loan, rather than as “acquisitions errors.” smaller libraries have historically relied on interlibrary loan to extend their collections, and today even large research libraries are acknowledging that they can no longer expect to acquire the entire scholarly output “just in case.” ill can be considered a core operational function, rather than an option of last resort, for libraries interested in adopting “just in time” approaches to collection development. free-lender and courier delivery networks and the ability to leverage knowledge bases to automatically select lenders and receive articles make resource sharing an increasingly practical supplement to subscriptions.   the researcher developed a simple full-text availability model that classified all items as available online, available in the physical collection, available through interlibrary loan, or experiencing an error (figure 1).   the researcher also created a conceptual model of e-resource failure, depicting the various systems that must work together to allow users to get the full-text of a citation (figure 2).   the model depicts a typical full-text request process with the following steps:   1)       users authenticate to the library proxy server to run off campus database searches (proxy misconfigurations here will also block access to full-text for all users, regardless of location, later in the process) 2)       clicking the openurl icon causes the a&i database to send a “source” or incoming openurl link to the link resolver. 3)       the link resolver compares the source openurl against the library’s self-reported subscription holdings in the knowledge base. 4)       if the library reported a full-text holding for the desired item in the knowledge base, the link resolver then creates a “target” or outbound openurl link to route the user to full-text. if no full-text holding was reported, libraries can configure the resolver to provide users with options to search the library catalogue for a physical copy or request the item through ill. 5)       full-text providers receive the target openurl and deliver the full-text of the item, or 6)       the ill software receives the target openurl, assigns each metadata element to appropriate fields in the request form, and allows the user to submit the request for staff processing.   each step in the request process carries the potential for error. proxy errors occur because a resource’s internet domain name was not registered in the library proxy server’s forwarding table or included in the library’s secure sockets layer (ssl) certificate (figures 3-4). librarians can edit the forward table at will, but may need to re-purchase the ssl certificate or wait until the next renewal period to solve certificate errors.     figure 2 conceptual model of electronic resource failure points   figure 3 proxy error caused by missing domain in forward table   figure 4 proxy error caused by missing domain in ssl certificate   source errors can be caused by inaccurate information in an abstracting record (such as a missing issn) or because the database lacks interface elements required to access full-text (such as the clickable icon needed to trigger openurl linking) (figure 5). often the solution involves collaboration with the database vendor.   when a library’s subscription entitlement does not match holdings reported in the knowledge base (kb), a kb error may result (figure 6). some of these problems are not under the library’s control (for example, if the knowledge base vendor defines a collection in insufficient detail to allow article-level linking).   link resolver errors occur in rare situations where the logic used by the link resolver fails to retrieve the desired item. figure 7 depicts one link resolver error, which occurred when an outbound link for a journal article landed on a book in the library catalogue instead. one volume of the journal had been purchased for the library collection and catalogued as part of a monographic series. since the link resolver was matching on title rather than issn, it resolved to this item rather than routing the request to ill. this problem was fixed by configuring the link resolver to match on issn instead.   target errors typically occur because content is unavailable from the full-text provider; resolution usually requires collaboration between librarians and vendors (figure 8).   finally, ill errors appear when the library’s ill system fails to correctly populate openurl metadata into the online request form (figure 9). libraries can correct some problems by adjusting configuration settings in ill software.     figure 5 source metadata error caused by missing issn   figure 6 knowledge base collection that does not support article-level linking   figure 7 link resolver erroneously matching on title   figure 8 target error due to missing content   figure 9 ill error caused by misconfigured request form and lack of unicode support     methods   the present study involved gathering a realistic sample of citations generated by likely keyword searches, testing them for online, physical or interlibrary loan availability, and attributing any errors to one of these six categories. after performing troubleshooting, the study was repeated and the results were compared using a test for statistical significance.   the researcher obtained a 400 item sample by selecting 10 a&i databases, running four keyword searches in each database, and testing the first 10 items in each search result for availability or error. this sample size was chosen because statistical calculations based on a pilot study of 100 items showed that a 400-item sample size would approximate the overall collection’s availability rate with a 97% confidence interval and +/5% margin of error (brase and brase, 1987, pp. 284–287). databases were chosen to represent a variety of disciplines in the humanities, social sciences, and natural sciences. the researcher derived search terms by querying the library’s libstats reference desk software for four reference questions related to each discipline and isolating key concepts from each question (see table 1 for the list of search terms). citations in the sample were classified according to item type (book, article, book chapter, dissertation, other).   the researcher conducted the initial availability study by searching databases from his office at armacost library over a one-month time period in april 2012. he tested for electronic availability by clicking the serials solutions “get article” link (or “get journal” if no article level link existed) for each of the first 10 search results. if the item was not available online, he tested the library catalogue for physical access by clicking the “search the library catalog” link. if items were not locally available in print or online, he clicked the “submit an interlibrary loan request” link and verified that the request form was correctly filled out (without actually submitting a request). item metadata, inbound and outbound openurls, availability, and error codes were tracked on a spreadsheet (http://works.bepress.com/sanjeet_mann). testing the results of a typical search took about 25 minutes.     table 1 databases and search terms used     availability was defined as a binomial (yes/no) variable. items were additionally classified into one of three availability categories (local online, local print, and ill) or one of six error categories (proxy, source, kb, resolver, target, and ill). assigning errors to a category often involved comparing metadata in the source database with inbound and outbound openurl links, looking for discrepancies or missing metadata.   the first round of availability testing revealed numerous system errors, so the researcher pursued troubleshooting over a period of several months. he addressed the most frequent category, ill errors, by working with the web librarian to update the illiad customization manager tables and online request forms. he addressed knowledge base errors by updating the serials solutions knowledge base; in particular, one problematic consortial e-journal collection was switched to a different collection that was more accurate. proxy errors were fixed by adding domains for e-journal providers to the innovative web access management (wam) forward table. the researcher also opened numerous support tickets with database vendors to address source metadata errors and missing content in target databases, though these categories of errors were not pursued exhaustively.   after the initial study took place, oclc upgraded the illiad software to use unicode to correctly render diacritic marks or non-roman characters embedded in an openurl. several database vendors also made interface changes and updated content.   were these changes enough to improve full-text access for armacost library users? to find out, the researcher conducted a second round of availability testing using the same databases and search terms in march 2013, producing another 400-citation sample recorded in its own spreadsheet (http://works.bepress.com/sanjeet_mann). availability was higher in the second study, so the researcher used z tests for the difference of two proportions to determine whether the differences were statistically significant (kanji, 2006). this statistical test compared percentages from the second study against percentages from the first study to determine whether the changes were large enough to be unlikely to occur by chance.   results   availability increased from 250 of 400 items (62.5%) in the first study to 346 of 400 items (86.5%) in the second study. a comparison of the branching diagrams from the two studies in figures 10-11 clearly illustrates the gains in full-text downloads and fillable ill requests.   since overall availability increased in the follow up study, overall error frequency decreased. source errors were the only error category that did not decline (table 2).   however, z tests showed that the changes in print availability, kb errors and resolver errors were not statistically significant (α= 0.05) (table 3).       figure 10 branching diagram, 2012 study   figure 11 branching diagram, 2013 study   table 2 availability rates   table 3 z test results     most items in the sample were journal articles. articles displayed higher availability than books or book chapters even after troubleshooting (table 4).   availability improved in the second study for all but one discipline. music and english displayed the greatest gains, while philosophy was the only discipline not to reach at least 75% availability after troubleshooting (figure 12).    table 4 availability by item type     figure 12 availability by discipline     discussion   the significant improvement in overall availability and in three of six error categories found in the 2013 availability study suggests that the initial 2012 study effectively alerted the researcher to errors that were systematically blocking access to electronic resources. libraries must be proactive in seeking out electronic resource errors; according to one study, “relying solely on user reports of errors to judge the reliability of full-text links dramatically underreports true problems by a factor of 100” (stuart et al., 2015, p. 74). conducting an availability study gives e-resource librarians reliable evidence to focus their efforts.   it is important to note some issues inherent to e-resource troubleshooting which will limit libraries’ ability to maximize their improvements. based on the results of this study, the problems that affect the most resources are not the problems that can be fixed most efficiently. the quickest fixes – proxy errors – only accounted for 1% of errors in the original study. illiad errors improved the most, but it is unclear how much of this success came from the library’s own actions, versus the system upgrade coincidentally implemented by oclc. the researcher had little influence over target errors (the other category to show significant improvement) beyond reporting problems to the vendor and setting reminders to follow up with the assigned support agent at regular intervals. many kb errors found in the 2012 study could be corrected in-house, but this task required complex troubleshooting skills and did not produce statistically significant improvement. as chen (2012, p. 223) observed, many open access journals do not support openurl linking at the item level, limiting the knowledge base’s ability to connect to these titles. the most common problems – source errors – were the most difficult to fix. it was often unclear whether the publisher, a&i database vendor, or full-text vendor was responsible for correcting these problems, which still comprised 8% of sampled items in the 2013 study.   since availability studies are time consuming (requiring about 25 minutes per search), and the most productive troubleshooting strategies listed here (such as customizing illiad) will likely yield one time improvements, there may be diminishing returns for libraries that attempt multiple full-scale availability studies. however, smaller-scale availability studies could be effectively incorporated into a library’s e-resource workflows on an annual basis. a ten-search study of 100 items still has a 79% confidence interval with +/5% error, which may be “good enough.” a student worker could conduct the searches, leaving it to the electronic resources librarian or well-trained staff to fix problems.   this research has intriguing implications for other areas of library operations. those interested in benchmarking availability as an assessment metric should note that local availability is affected by a library’s acquisitions budget and the size of its physical and electronic collections. future studies at other institutions can place the present study’s results in proper context. the 62.5% availability rate in the 2012 study is comparable to the typical 63% availability rate at other libraries reported by nisonger (2007), though those studies could have reported 25% higher availability rates if they did not count ill-requestable items as “acquisitions errors.” troubleshooting appears to have raised armacost library’s availability rate to a similar level (86.5%) in the 2013 study.   collection development librarians may wish to further increase full-text availability rates at their institutions by adding subscriptions and switching a&i databases to full-text. this study presumed that nonlocal availability was not an obstacle, because libraries can now efficiently provide users with items at the point of need. however, one could argue that local electronic downloads best satisfy library users’ demand for immediate access to full-text. wakimoto et al. found that nearly 50% of students expected to get full-text from an openurl click “always” or “most of the time” (2006). imler and hall (2009) found that penn state students rejected sources whose full-text was not immediately available online. erial researchers reported illinois wesleyan students abandoning sources which had triggered a system error: “virtually any obstacle they encountered would cause them to move on to another source or change their research topic” (duke & asher, 2012, p. 82). the finding that even after troubleshooting, clicking an openurl link in armacost library databases did not produce a full-text download two out of three times suggests that online availability may be not be meeting user expectations.   instruction librarians could reduce students’ frustration with unavailable full-text by waiting to introduce a&i databases until students are ready to conduct advanced research in their discipline. instruction should include explanations of how to place interlibrary loan requests, how to exercise reciprocal borrowing rights, or how to refine the search to find a different source that is locally available. these skills can be “scaffolded” atop other concepts such as question formulation and source evaluation, which could be introduced to lower-division students through full-text resources.   metadata-related errors persisted in the 2013 study despite troubleshooting; if these problems are unavoidable, librarians must consider how and when to teach error workarounds. conversations surrounding the association of college and research libraries (acrl) framework for information literacy and the critical information literacy (cil) movement demonstrate librarians’ desire to de-emphasize instruction in search mechanics and engage students in discussion of how scholarly communities construct notions of authority, or the consequences of inequitable access to information in our society. yet the dispositions of “persistence, adaptability and flexibility” described in the acrl framework (association of college and research libraries, 2015) can be strengthened by classroom examination of why e-resources fail and what students can do about it.   subject liaisons should note that library users will have varying experiences with a&i searching across the disciplines. these differences could be related to the a&i database vendors and full-text content providers chosen, or the type of items indexed. some databases like rilm (répertoire international de littérature musicale) indexed many error-prone books and chapters, while others like america: history and life only returned journal articles, which were less likely to produce errors. this simulated study also did not account for search strategies naturally employed by researchers in various disciplines.   finally, it is necessary to acknowledge limits to the availability technique as a way of studying real-life user interaction with electronic resources. however refined the methodology, simulated availability studies conducted by a librarian can only detect system errors, not “human errors” which arise as library users navigate databases while trying to make sense of their information need. in theory, recruiting patrons to test items themselves would allow availability researchers to expand the conceptual model of error causes given above to include problems with search strategy or source evaluation, which librarians could address by improving interfaces or changing what they teach. however, the researcher’s first attempt at a “real” e-resource availability study revealed a methodological problem with adapting the availability technique in this manner (mann, 2014). simulated availability studies produce a sample of database citations, while studies of library users produce a sample of user interactions with a resource – two different types of data that could not be compared directly. furthermore, the challenges of recruiting students resulted in a sample too small to support significance testing. at what point must researchers give up the ability to make a statistical inference about the entire library collection in order to learn realistic and actionable information about user behaviour? perhaps availability studies should remain simulated and limited to observation of system errors, but be conducted alongside e-resource usability studies as part of a mixed-methods research project.   conclusion   a 400 item sample of electronic resource citations allowed the researcher to accurately estimate the availability of items in armacost library a&i databases. z tests showed that overall availability improved significantly after troubleshooting, though only 1 in 3 items were available as electronic downloads. electronic resource availability studies produce evidence that can inform discussions and address concerns felt in various library units. however, there are limits to how well a simulated electronic resource availability study can approximate the behaviour of library users. further directions for this type of research include conducting availability studies at other types of libraries, and combining availability studies with usability studies to account for both technical and human errors.   references   asher, a. d., duke, l. m., & wilson, s. (2013). paths of discovery: comparing the search effectiveness of ebsco discovery service, summon, google scholar, and conventional library resources. college & research libraries, 74(5), 464–488. http://dx.doi.org/10.5860/crl-374   association of college and research libraries. (2015). framework for information literacy for higher education. chicago: acrl. retrieved from http://www.ala.org/acrl/standards/ilframework   brase, c. h. & brase, c. p. (1987). understandable statistics: concepts and methods (3rd ed.). lexington, mass: heath.   chandler, a., wiley, g., & leblanc, j. (2011). towards transparent and scalable openurl quality metrics. d-lib magazine, 17(3/4). http://dx.doi.org/10.1045/march2011-chandler   chen, x. (2012). broken-link reports from sfx users: how publishers, vendors and libraries can do better. serials review, 38(4), 222–227.  http://dx.doi.org/10.1016/j.serrev.2012.09.002   ciliberti, a., radford, m. l., radford, g. p., & ballard, t. (1998). empty handed? a material availability study and transaction log analysis verification. journal of academic librarianship, 24(4), 282–289. http://dx.doi.org/10.1016/s0099-1333(98)90104-5   cockrell, b. j., & jayne, e. a. (2002). how do i find an article? insights from a web usability study. journal of academic librarianship, 28(3), 122-132. http://dx.doi.org/10.1016/s0099-1333(02)00279-3   connaway, l. s., & dickey, t. j. (2010). the digital information seeker: report of the findings from selected oclc, rin, and jisc user behaviour projects. dublin: oclc. retrieved from http://www.jisc.ac.uk/media/documents/publications/reports/2010/digitalinformationseekerreport.pdf   crum, j. a. (2011). an availability study of electronic articles in an academic health sciences library. journal of the medical library association, 99(4), 290–296. http://dx.doi.org/10.3163/1536-5050.99.4.006   culling, j. (2007). link resolvers and the serials supply chain. oxford: uksg. retrieved from http://www.uksg.org/sites/uksg.org/files/uksg_link_resolvers_final_report.pdf   cummings, j., & johnson, r. (2003). the use and usability of sfx: context-sensitive reference linking. library hi tech, 21(1), 70–84.   duke, l. m., & asher, a. d. (2012). college libraries and student culture: what we now know. chicago: american library association.   fagan, j. c., mandernach, m. a., nelson, c. s., paulo, j. r., & saunders, g. (2012). usability test results for a discovery tool in an academic library. information technology and libraries, 31(1), 83–112. http://dx.doi.org/10.6017/ital.v31i1.1855   gaskill, h. v., dunbar, r. m., & brown, c. h. (1934). an analytical study of the use of a college library. the library quarterly, 4(4), 564–587.   imler, b., & eichelberger, m. (2014). commercial database design vs. library terminology comprehension: why do students print abstracts instead of full-text articles? college & research libraries, 75(3), 284-297. http://dx.doi.org/10.5860/crl12-426   imler, b., & hall, r. a. (2009). full-text articles: faculty perceptions, student use, and citation abuse. reference services review, 37(1), 65–72. http://dx.doi.org/10.1108/00907320910935002   jayaraman, s., & harker, k. (2009). evaluating the quality of a link resolver. journal of electronic resources in medical libraries, 6(2), 152–162. http://dx.doi.org/10.1080/15424060902932250   kanji, g. k. (2006). 100 statistical tests (3rd ed.). london: sage publications.   kantor, p. b. (1976). availability analysis. journal of the american society for information science, 27(5), 311–319. http://dx.doi.org/10.1002/asi.4630270507   kasprowski, r. (2012). niso’s iota initiative: measuring the quality of openurl links. the serials librarian, 62(1-4), 95–102. http://dx.doi.org/10.1080/0361526x.2012.652480   kress, n., del bosque, d., & ipri, t. (2011). user failure to find known library items. new library world, 112(3/4), 150–170. http://dx.doi.org/10.1108/03074801111117050   majors, r. (2012). comparative user experiences of next-generation catalogue interfaces. library trends, 61(1), 186–207. http://dx.doi.org/10.1353/lib.2012.0029   mann, s. (2014). why can’t students get the sources they need? results from a real availability study. presented at the 29th annual nasig conference, fort worth, tx. retrieved from http://works.bepress.com/sanjeet_mann   mansbridge, j. (1986). availability studies in libraries. library & information science research, 8(4), 299–314.   niso. (2013). improving openurls through analytics (iota): recommendations for link resolver providers. baltimore, md: national information standards organization. retrieved from http://www.niso.org/apps/group_public/download.php/10811/rp-21-2013_iota.pdf   niso, & uksg. (2010). kbart: knowledge bases and related tools (no. niso-rp-9-2010). baltimore, md: national information standards organization. retrieved from http://www.niso.org/publications/rp/rp-2010-09.pdf   niso, & uksg. (2014). knowledge bases and related tools (kbart) recommended practice (no. niso rp-9-2014). baltimore, md: national information standards organization. retrieved from http://www.niso.org/apps/group_public/download.php/12720/rp-9-2014_kbart.pdf   nisonger, t. e. (2007). a review and analysis of library availability studies. library resources & technical services, 51(1), 30–49.   nisonger, t. e. (2009). a simulated electronic availability study of serial articles through a university library web page. college & research libraries, 70(5), 422–445. http://dx.doi.org/10.5860/crl.70.5.422   o’neill, l. (2009). scaffolding openurl results: a call for embedded assistance. internet reference services quarterly, 14(1-2), 13–35. http://dx.doi.org/10.1080/10875300902961940   pesch, o. (2012). improving openurl linking. serials librarian, 63(2), 135–145. http://dx.doi.org/10.1080/0361526x.2012.689465   saracevic, t., shaw, w.m., & kantor, p. (1977). causes and dynamics of user frustration in an academic library. college & research libraries, 38(1), 7-18. http://dx.doi.org/10.5860/crl_38_01_7   stuart, k., varnum, k., & ahronheim, j. (2015). measuring journal linking success from a discovery service. information technology and libraries, 34(1), 52–76. http://dx.doi.org/10.6017/ital.v34i1.5607   trainor, c., & price, j. (2010). rethinking library linking: breathing new life into openurl. chicago: american library association.   wakimoto, j. c., walker, d., & dabbour, k. (2006). myths and realities of sfx in academic libraries. journal of academic librarianship, 32(2), 127-136. http://dx.doi.org/10.1016/j.acalib.2005.12.008   williams, s. c., & foster, a. k. (2011). promise fulfilled? an ebsco discovery service usability study. journal of web librarianship, 5(3), 179–198. http://dx.doi.org/10.1080/19322909.2011.597590   wrubel, l. s. (2007). improving access to electronic resources (er) through usability testing. collection management, 32(1-2), 225–234. http://dx.doi.org/10.1300/j105v32n01_15   evidence summary   hands on digital information literacy training from peers is preferred by public service library staff   a review of: robertson, r. (2014). reframing ourselves: digital information literacy skills of frontline public library staff. new zealand library and information management journal, 53(3). doi:10.1080/00048623.2011.10722203   reviewed by: cari merkley associate professor mount royal university library calgary, alberta, canada email: cmerkley@mtroyal.ca   received: 9 sept. 2014    accepted: 8 oct. 2014      2014 merkley. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to explore how and where public library employees acquire digital information literacy (dil) skills.   design – qualitative study using semi-structured interviews.   setting – two public libraries in new zealand.   subjects – nine front line public library staff members.   methods – a convenience sample of nine library employees was interviewed about their existing dil skills, how and where they learned them, any barriers to this learning, and how they defined dil in others. interviewees ranged in age from 40 to 64 and included both those new to libraries and those with over 25 years in the profession. the interview transcripts were analyzed for key themes and placed in the theoretical framework of kolb’s experiential learning cycle (robertson, 2014).   main results – five participants described their own dil skills as average or below average. the remaining participants classified their skills as above average. participants recounted acquiring dil skills in the course of their work through formal workplace training sessions, peer support, or individual exploration; through personal exploration of tools on their own time; or through a mix of work and personal learning opportunities. the barriers they identified to their learning included insufficient time to train and practice the skills learned and the lack of access to relevant technologies. participants noted problems such as accessing key hardware and insufficient internet connectivity at work because of issues with organizational infrastructure and at home due to personal financial constraints. participants largely preferred informal hands-on training by peers to formal training sessions, which were described by some as too general or held too far in advance of the implementation of new technology. the data suggested participants largely fell into kolb’s accommodating or diverging learning styles because of their preference for “concrete experience” (robertson, 2014).   conclusion – libraries may improve staff acquisition of dil skills by increasing hands on learning opportunities and providing dedicated time to review and practice skills learned. other suggestions included identifying potential digital peer mentors among staff and providing them with the necessary resources (time, money, and a defined role) to support their colleagues, breaking training into parts allowing time for practice, creating training plans tied to performance evaluation, and using incentives to encourage staff to participate in self-directed training.     commentary   the large body of research studies and professional literature that exists on maintaining the currency of digital/technical skills among library staff suggests that it is a challenge shared by all types of libraries. the author’s decision to approach this question qualitatively through interviews sets it apart from most previous studies and creates a rich data set around employee training preferences and barriers to learning.   while the small sample size is appropriate to the method chosen, the sampling technique used bears scrutiny. the use of a convenience sample is identified by greenhalgh (2010) as an area for concern when evaluating qualitative studies. in this case, the fact that none of the participants were under the age of 40 may mean that a significant group of frontline library employee training needs and experiences with dil outside of work were not captured by the study. this issue should have been acknowledged along with any other limitations of the study, and the resulting data and conclusions framed appropriately. it is also unclear how large a pool of possible participants existed, as the total number of front line staff was not provided. the discussion of results refers to a questionnaire administered to participants, but no mention is made of this aspect of the data gathering in the too brief methodology section nor is the questionnaire itself included. further detail on the process of data analysis or how the data was validated would have strengthened the study and clarified what role, if any, kolb’s theory of experiential learning, identified as the study’s theoretical framework, played in the study design.   most of the conclusions drawn by the author flow clearly from the findings presented, with the exception of the recommendation to provide incentives to staff who engage in self-directed learning programmes. lack of motivation did not emerge as a key barrier among those interviewed, and incentives did not appear to have been raised by participants. the suggested incentives to participate in self-directed learning are unlikely to significantly address the impact that the lower wages and underemployment experienced by front line staff has on potential dil skill acquisition, an issue that the author has flagged elsewhere.   the study does offer some concrete suggestions for improving staff digital information literacy training that will be of interest to those in both public and academic libraries, although the narrow demographic represented will limit its applicability to groups such as student employees. the study also serves as an important reminder that those individuals who are the most visible to patrons are often the lowest paid. library managers and supervisors cannot assume that opportunities to practice dil skills outside of work will be available. if staff dil is a priority, it must be appropriately resourced in terms of dedicated work time and resources.   references   greenhalgh, t. (2010). how to read a paper: the basics of evidence-based medicine (4th ed.). hoboken, nj: wiley-blackwell/bmj books. news/announcements   international symposium for health information management research (ishimr) 2015    2014. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   on behalf of the ishimr 2015 committee, we invite you to participate in the 17th international symposium for health information management research (ishimr), which will take place at york st. john university on 25-26 june 2015, in the beautiful city of york, england, uk.   the theme of this meeting will focus on "health informatics for enhancing health and well-being".   ishimr 2015 will bring together researchers in the general area of e-health, health information management and health informatics and will provide a forum for the presentation and discussion of their research activities. we therefore welcome paper (oral presentation) / abstract (poster) submissions and delegates from the health and research community, particularly those with teaching/ research responsibilities in health information management and health informatics, those with clinical responsibilities, and those involved in provision of health information and knowledge services.   ishimr 2015 will feature oral and poster presentations, a doctoral symposium, a demonstration of health informatics software applications and invited speakers from industry, government organisations and academia. ishimr 2015 will include a multidisciplinary audience.   topics of interest   the main themes of interest include, but are not restricted to:   health data management strategies health information management strategies health knowledge management strategies health informatics systems evidence-based decision making emerging trends in health informatics    paper submission   you and your colleagues are invited to submit your research contributions or practical experience reports. all papers will be blind peer-reviewed by at least two members of the programme committee. the deadline for submission of papers and poster abstracts is: 15th january 2015. all papers presented and abstracts for posters, will be published in the conference proceedings. as with previous ishimr conferences, the best papers will be considered for publication in a special issue of the health informatics journal and in the international journal of health information management research (ijhimr).   please see the ishimr 2015 web-site (www.ishimr2015.com) for further information.   we look forward to your contribution and to seeing you at ishimr 2015.   hannah spring, barbara sen, peter bath, dimitri raptis, organising committee  evidence summary   assessment of undergraduate-driven acquisitions at a small college library shows both costs and benefits   a review of: waller, j. h. (2013). undergrads as selectors: assessing patron-driven acquisition at a liberal arts college. journal of interlibrary loan, document delivery & electronic reserve, 23(3), 127-148. http://dx.doi.org/10.1080/1072303x.2013.851052   reviewed by: laura costello head of library materials & acquisitions gottesman libraries teachers college, columbia university new york, new york, united states of america email: lac2184@columbia.edu   received: 30 nov. 2014 accepted: 26 jan. 2015      2015 costello. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to examine the viability of an undergraduate-focused, patron-driven acquisitions strategy in a small college library and to evaluate the titles acquired through this program for collection appropriateness, patron satisfaction, and cost effectiveness.   design – case study.   setting – a small, catholic college in the eastern united states with 1,850 undergraduate students.   subjects – acquisitions of 432 print monographs selected by students and 18,624 print monographs selected by librarians and faculty members.   methods – the author compared purchases selected from a pool of undergraduate interlibrary loan requests acquired from 2004 to 2013 to purchases acquired during the same time period through traditional means, including collection development work by librarians and selections by academic departments. the author evaluated titles for use based on circulation figures, for suitability using overlap analysis with the collections of four peer libraries, for patron satisfaction based on turnaround time, and for cost compared to items obtained through interlibrary loan.   main results – student selection had some advantages, including moderately increased circulation. traditionally acquired titles were less likely to circulate initially and only 20.46% of these titles circulated two or more times compared to 24.77% of student-selected titles. student selections were less likely to be acquired by peer libraries, and 63.66% of student-selected titles were unique, though they had a similar subject distribution to traditionally acquired titles. compared to interlibrary loan, student-selected purchases had similar turnaround times and in the most recent three-year period had an average turnaround time that was one day faster than interlibrary loan. however, student acquisitions were far costlier than interlibrary loan. items acquired through this program cost the library $39.70 on average while borrowing cost $6.18 on average.   conclusion – the student selection process was found to be moderately successful, and the library will continue the program. based on the analysis of peer library holdings, the author suggests more librarian intervention in the selection process. instead of purchasing any requests that meet the criteria for student selection, the author recommends an intermediary selection step of evaluation by librarians. student selection did not show the dramatic advantages represented in studies conducted in larger academic libraries, and this disparity could potentially be due to a difference in selection quality between the undergraduate students at this college and the graduate and research populations of larger institutions.   commentary   as diverse libraries adopt patron-driven and demand-driven acquisitions strategies, it is important to evaluate these programs for their suitability to individual libraries and groups. the most significant research on these strategies has been conducted in large research libraries with strong graduate student and postgraduate populations and a variety of demand-driven acquisitions strategies. undergraduate-focused liberal arts institutions have different collection development goals, and this study aims to address the longand short-term advantages and disadvantages of a selection strategy driven by undergraduate interlibrary loan demand.   the short-term circulation advantages are dramatic in existing literature and, though this study revealed more modest gains, convincing evidence still exists that even undergraduate-selected titles have more initial and subsequent circulations than traditionally selected titles. evaluating on cost per use alone, these items potentially have a more significant benefit since the study limited student-selected purchases to $75 while facultyand librarian-selected titles were unlimited. cost was highlighted as a disadvantage to student-selected purchases because these titles were drawn from and compared to interlibrary loan requests rather than traditionally acquired items.   the study emphasized the importance of quality in selection, which the authors measured by comparing items acquired through student selection to the collections of four peer institutions. very little overlap occurred between student selections and traditionally acquired titles held in peer libraries, but it's significant that the author did not compare the much larger list of traditionally acquired titles to these libraries. because traditional selection occurred near the publication date and the library did not receive subsequent interlibrary loan requests from which to draw student selections for these titles, it is possible that many of these titles were already held in the library and student selection represented holes in traditional acquisitions strategies that were missed by all five peer libraries. assessing incoming interlibrary loan requests for student-selected titles might help clarify this issue.   student-driven acquisitions policies are difficult to evaluate. they are inexorably bound up with other types of acquisitions and borrowing. the statistical inconsistencies in the results of this study could be due to the close integration of patron-driven acquisitions with other types of acquisitions in the library. evaluating the results of these policies is important, but establishing criteria for evaluation is even more important. for a small liberal arts library, balancing budget, patron satisfaction, usability, and collection quality pull acquisitions strategies in many different directions at once. student-driven strategies might be a part of this balanced acquisitions ecosystem, but the significance of that part depends on the values of the library. this study found clear benefits for circulation and turnaround time, which might mean a collection that is highly useful for contemporary patrons, but cost considerations and value comparisons to other institutions could push smaller libraries to de-emphasize student-selected acquisitions.   review article   the life and times of aboutness: a review of the library and information science literature   sophie rondeau library technical specialist syracuse university syracuse, new york, united states of america email: smrondeau@hotmail.com   received: 03 mar. 2013   accepted: 16 feb. 2014      2014 rondeau. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – this paper explores the concept of “aboutness”, its related terms, and the process of aboutness determination as found in the library and information science literature.   methods – a review of the literature pertaining to aboutness determination and related terms specific to library and information science was undertaken, borrowing extensively from the literature review of daniel joudrey’s (2005) dissertation, building puzzles and growing pearls: a qualitative exploration of determining aboutness, as well as examining a small selection of research and articles not discussed by joudrey. in addition, a concept map was developed to outline many of the concepts and theories found.   results – the lis literature demonstrates conflicting positions surrounding the term aboutness and its correlates. despite the lack of firm agreement on terminologies, the notion of subject is explored because it featured prominently in the literature. as well, intensional and extensional aboutness are explored in contrast to a more subjectivist perspective which asserts that a document's aboutness cannot be separated from its reader. aboutness determination is also examined through the lens of theme and rheme, that is what is presupposed in a document versus what is new. aboutness from the user perspective featured considerably in the literature, with many authors asserting both the importance and the challenge of mediating knowledge on behalf of the user. the stage of aboutness determination in the complete process of subject indexing is also presented here, and the findings demonstrate that the stages are anything but linear and sequential.   conclusion – while the findings are more expositional than conclusive, they demonstrate the complexities and challenges surrounding the concept of “aboutness” and the process of its determination. the value of this review is in its ability to present the ways in which scholars and practitioners have attempted to grapple with this conundrum. although indexers may find temporary solace in cataloguing manuals that outline aboutness determination procedures, underneath these “safe surfaces” is an ambiguous concept further complicated by obscure and incomplete processes. this review provides an opportunity to reflect on those challenges and to further the discussion.     introduction   the philosophical problems of information organization may appear inconsequential and inflated within the context of slashed library budgets, the rise of information exhaustion (a.k.a. information overload), and the popularization surrounding the notion of the library as an “endangered species”. yet, underneath what appear to be musings among scholars and researchers, there is a sincere desire to understand the complexities of information organization.   in a discipline such as library and information science (lis) the philosophical does not operate within the theoretical realm alone, but is deeply engaged in practice. when scholars, researchers, and librarians discuss concepts within the field, they are discussed within the context of a practice. more particularly, discussions surrounding information organization are considered in a purposive context, even if they are based in the theoretical realm. in other words, information organization would be based in hollow ponderings if not for its usefulness. the theoretical basis is an essential component to the daunting task of contemporary information organization. it assures that the implications of the practice are considered in light of their various influences and potentialities.   in this light, the problem of aboutness determination in information organization is significant and worthy of exploration. it does not exist merely in the “clouds”, but penetrates into the “heart” of organization. this researcher was once warned that “writing a paper about ‘about’ is a recipe for a brain burnout feedback loop” (c. read, personal communication, august 6, 2011). while this may prove true, it is not sufficient grounds to avoid the discussion. one can acknowledge the challenge, and near impossibility, of reaching a universal definition, and at the same time “chew” on the various colourful ways in which lis has dealt with the complexity. this is precisely the purpose of this paper.   aims   while the term about is commonly employed among speakers, the term aboutness is less common. this may have to do with a certain je ne sais quoi or abstract condition of the word. to explain, the term is constructed of the preposition, about, followed by the suffix, -ness. a preposition is commonly used to link relationships between other words in a sentence (e.g. the book is about cats). a preposition is generally followed by a determiner, which is exemplified above as “cats”. as well, prepositions are generally difficult to define because they are not used in isolation. although the word about is frequently used at the end of a sentence, it is often in the form of a question (i.e. what is the book about?). when the suffix –ness is added to the preposition, about, the preposition becomes an abstract noun and refers to a quality or condition of being about something (oxford english dictionary, aboutness). thus, the example above becomes: the aboutness of the book is cats; or the book’s aboutness is cats. while this example appears somewhat clumsy, h.h. joachim (1906) in the nature of truth iv 174, provides a rather more eloquent application of the term: knowledge, so far as that is judgement and inference, is primarily and explicitly thinking “about” an other. and even though discursive thought may find its concentrated fulfilment in immediate or intuitive knowledge, its character of ‘aboutness’ is not thereby eliminated.   according to the oxford english dictionary online (oed) (2011), joachim (1906) is credited as being the first to coin this term, and his application of the term in the example above is synonymous with the definition provided in the oed.   in the lis literature, robert fairthorne (1969) was the first to employ the term aboutness. fairthorne had specific ideas surrounding aboutness in subject indexing, and he further narrows his understanding into two types of aboutness: extensional aboutness and intensional aboutness. these two types will be explored more deeply at a later point in this paper. what is useful here is to understand that the term aboutness is fairly recent to the general discourse, and even more so in the lis literature. as a result of its newness, the term can either be highly simplified or made otherworldly. the proceeding review seeks to critically examine its use in  the lis literature, and especially as it applies to the task of subject indexing.   this paper borrows from the literature review of daniel joudrey’s (2005) dissertation, building puzzles and growing pearls: a qualitative exploration of determining aboutness, as well as a few articles not presented by joudrey, but considered important in the discussion. reference to joudrey’s literature review, however, is not exhaustive. textbooks were not employed due to their pedagogical structure in presenting established concepts. in other words, this paper is more interested in exploring concepts aside from their position within an established framework of practice. as well, some texts are neglected in this discussion because they did not prove useful, or do not fall within the realm of its boundaries. for example, joudrey presents numerous references to works related to facet analysis. despite the fact that facet analysis has made significant contributions within lis, it is not within the parameters of this paper to present a detailed exposition on the topic. rather the focus remains as a broad review of aboutness, as opposed to an exposition on specific systems. the author also acknowledges that the topic of aboutness is much broader than its application in lis, and that the literature exploring link patterns through search engines and recommendation tools on discovery engines could offer additional insight into this complex phenomenon. however, the purpose of this paper is to interact with the broad lis literature exclusively. future research could extend the discussion beyond the lis literature to see if useful connections could be made, and greater insights gained through alternate methods of naming and discovery.   methods   joudrey’s review was chosen based on its intentional examination into the nature of aboutness and the conceptual analysis processes involved in the determination of aboutness. the relatively recent publication of his dissertation made it possible to examine a broad range of accessible literature, both contemporary and classic. as well, joudrey is recognized for his work within the lis community, and specifically for his contribution to information organization. he has co-authored the latest edition of the organization of information (2009), alongside professor emeritus, arlene taylor. thus, joudrey’s review was deemed a valuable source for research into the topic of aboutness. while the researcher does not necessarily share the same philosophical perspectives as the author of the review, she acknowledges his contribution to scholarship, and in a “pearl-growing” like fashion, extends the discussion.   one challenge in writing a paper on aboutness in lis is the difficulty in separating the theoretical from the practical. while every attempt has been made to provide the clearest explanations of concepts and practices, it may so happen that the two overlap. for the understanding surrounding the notion of aboutness cannot happen in isolation from the practice of its application.   many of the issues presented in this review are also outlined in a concept map at the end of the document (see figure 1). the map provides a visual display of the concepts explored here.   findings   the findings are presented under the following headings: terminology debates; aboutness as a two-type distinction; aboutness from the user perspective; theme and rheme, the notion of subject; and stages in organization. these headings are intended to gather and organize literature that explores a particular aspect of aboutness determination. as is the case with many classification systems, these headings distinguish the various ideas presented here. however, it may be additionally fruitful to examine them as a whole, and the concept map in figure 1 provides a visual representation of the literature explored in this review.   terminology debates   it may be a simple matter of convenience that the term aboutness was drawn into the library science community. after all, it is difficult to talk about about because of its prepositional nature. by converting the preposition to a noun, it becomes possible to describe it, despite it being an abstraction. in other words, as a noun, aboutness is “capable of functioning as the subject and direct object in a sentence, and as the object of a preposition” (oed, 2011, noun). converting this preposition to a noun may, however, only create a semblance of precision. in reality, the term is fraught with numerous contradicting definitions as scholars and librarians attempt to grapple with this abstraction in application and theory.                 joudrey (2005), himself, claims that the movement towards the use of the term aboutness in lis literature relates to its being more “pragmatic” and “practice-oriented” in definition (p. 3). he makes a distinction between the terms subject and aboutness. due to its “richer and longer history” the term subject, is riddled with complexities and ambiguities that relate to its use within the philosophical and literary theoretical realms (p. 3). “thus”, he explains, “it is not difficult to leap from the term subject to the related but more philosophical terms: meaning, understanding, interpretation, and idea” (p. 3). metcalfe (1973) also comments on the unsatisfactory use of the term subject in lis because of “conflicts and confusions of meaning, particularly with distinctions of general and specific, and of object and aspect” (p. 336).                 hjørland (2001) does not share joudrey or metcalfe’s perspective. he states that the “concept of aboutness did not remove this inherent unclarity, it only changed its name” (p. 774). likewise, albrechtsen (1993) states that the previous vagueness surrounding the term subject was eventually transferred to the term aboutness (p. 220). rather, says hjørland, “we should regard subject (including the compound subject matter) and aboutness as synonymous concepts (and prefer the former). the subject of a document is that ‘something’ that subject analysis and retrieval are supposed to identify. this is closely related to the questions that a document should provide answers to” (p. 776). the value of this perspective is that the indexer considers the document’s aboutness alongside potential information seeking queries. the indexer engages with both the document and its potential users as she considers the questions it may seek to answer. she is not simply a neutral automaton extracting the inherent aboutness from the document, but an engaged and active participant in the discussion.                 hjørland (2001) attempts to find a theoretical basis for defining synonymous terms that are not “biased towards any specific kind of ir-system” (p. 775). “if we are going to compare different approaches and develop general theories of information science and information retrieval,” says hjørland, “we have to develop concepts that do not give priority to certain kinds of systems at the expense of others” (p. 775). for hjørland, it is the theoretical basis behind the usage of terms like aboutness that defines them. he examines the most important terms that are used both synonymously and in distinction from one another. these include: subject (subject matter; subject-predicate); aboutness; topic (topicality; topic/comment); theme (with central theme and the german leitmotiv); domain (cognitive domain, scientific domain); field (information field, field of knowledge, field of research); content; and information (p. 775).                 also in contrast to joudrey’s claim that the term aboutness is free of ambiguities, swift, winn, and bramer (1978) suggest that the notion of aboutness is far from clear, despite the fact that it forms the basis for models of indexing (p. 182). the problem with the application of aboutness as an indexing model relates to their view that aboutness statements are usually oversimplified, and rarely describe the complexities surrounding searches carried out within the social sciences. they call instead for a “multi-modal” approach that involves characterising documents in a way that will serve a variety of search formulations. their multi-modal system is a means to achieve a small-scale approach to indexing that acknowledges specific needs within specific disciplines, such as the social sciences and education (1978, p. 94). this disciplinary focused approach to subject indexing provides the opportunity for specific disciplines to define their methods of aboutness determination and document characterisation. winn and bramer’s methodology is a distinct move away from universal, “one size fits all” approaches to subject indexing. the value of this approach is that it serves subject specialization by creating a meaningful and relevant vocabulary that is defined by and speaks to those who understand their disciplines best.   hutchins (1977) states “that the subject description is merely one form of expression of some part of what the document is about” (p. 2). he distinguishes between subject description and topics. topics act as a summarization of a document’s content for purposes of information retrieval (p. 2) and are based on references to relationships within a particular textual context. “in isolation a word has a sense, but it has no actual referent; it can have a referent only in a particular context” (p. 2). while subject descriptions, as exhorted by hutchins, act as isolated parts, topics function within a particular context. hutchins seems to be advocating a more holistic understanding of aboutness determination; one whereby topics point to the resource and provide a glimpse of its contents.   aboutness as a two-type distinction   aboutness as a two-type distinction was discovered in the lis literature. in these readings, the aboutness of a resource was seen to have intrinsic and extrinsic properties. this section of the paper will examine and analyse aboutness as a two-type distinction, and the issues associated with such a division.   as mentioned above, the term aboutness was first coined in the lis literature by robert fairthorne (1969). fairthorne (1969) bifurcates aboutness into two types: extensional aboutness and intensional aboutness. fairthorne’s two “camps” of aboutness distinguish between what is inherent in the document (extensional aboutness), and what is inferred from it (intensional aboutness) (p. 79). the former is fixed, stable, and unchanging, while the latter is meaning-based, fluctuating, and subject to interpretation. extensional aboutness is sometimes considered the concern of the indexer, and some would argue that it is the indexer’s purpose is to extract the document’s innate aboutness without interpolation of bias, subjective meaning, or interpretation. intensional aboutness implies a relationship between the inanimate resource and the user engaged with its content. as a result, meaning is derived. since users come to resources from various perspectives and for various purposes, the interpretations and meanings drawn from a resource will be from numerous vantage points.   fairthorne’s two types of aboutness can be found in the literature under different headings. beghtol (1986) recognizes two kinds of document aboutness, but names them aboutness and meaning to create a sharper distinction in terminology (p. 84). joudrey (2005) supports the distinction between extensional and intensional aboutness, and uses the terms aboutness and subject to distinguish the two (p. 59).   boyce (1982) uses the terms topicality and informativeness from a two-stage view of relevance and the retrieval process. while considered “operationally necessary” (p. 106), topicality is considered an insufficient condition for user based relevance. it may or may not serve user relevance judgements. informativeness, however, is based on the premise that the knowledge acquired from information retrieval is both “understandable and novel” (p. 106). while boyce’s two stages relate to fairthorne’s two types of aboutness, his description of informativeness is also similar to hutchins’s (1977) notion of theme and rheme, which is based on the idea that documents consist of a general knowledge base supplemented by an exposition of something new. theme and rheme will be discussed in more detail ahead.   some argue against the concepts of extensional and intensional aboutness. todd (1992) challenges the notion of a document’s permanent aboutness. he argues that the current low measure of inter-indexer consistency in subject determination does not support the view of extensional aboutness (p. 102). wilson (1968), too, demonstrates the indeterminacy and elusive nature surrounding the notion of subjects. he argues that we “cannot expect to find one absolutely precise description of one thing which is the description of the subject (p. 90). wilson’s approach to the notion of subject will also be explored in greater detail later in this paper.   mai (1999), who does not talk specifically about the distinct types of aboutness, offers an alternative perspective by challenging the notion that words and their meanings can be separated. instead, he argues that language is not merely words pointing to an objective reality, but is defined by the discourse of the “people or the community in which the words are used” (p. 553). therefore, knowledge organization becomes more concerned with the “interpretive process” and “the cultural and social context which the knowledge organization is a part of” (p. 555). merrell (as cited in mai, 1999) says that “classifications are never innocent but constantly streaked with arbitrariness and motivated by preconceptions and prejudices. besides they are constantly shifting, whether by design or in spite of our efforts to capture them” (p. 92). in other words, neutral and objective knowledge organization is a “pie in the sky”, merely a fanciful idea that is impossible to attain.   as a premise to their research, bertrand-gastaldy, lanteigne, giroux, and david (1995) claim that “all reading is subjective and oriented in terms of a project” (p. 15). similar to mai’s social construction, bertrand-gastaldy et al. assert that reading is both an individual and a social act of interpretation. as a social act, reading reflects certain shared characteristics among individuals within their respective communities. reading as an individual act reflects instances of differences between individuals within their communities which are based on their goals and needs.   if indexers are considered members of a specific community, it becomes possible to study the shared characteristics of their ‘readings’ of a document for the purpose of indexing, as well as to examine individual deviations or differences within the community, and to engage in an ongoing discussion surrounding those deviations and/or differences.   campbell (2000) asserts that the knowledge organization community has become sceptical of the idea that a document has an intrinsic aboutness, and concurs with mai (1999) that knowledge organization is a social construction. as a result, knowledge organization has come to embrace multiplicity and community-identity, while acknowledging that previous notions of universal neutrality were in fact laced with cultural bases and bias (p. 123). by admitting to the problem of bias, knowledge organizers will inevitably look to communities (such as the gay and lesbian communities) for their response to updated vocabularies and classifications. (p. 123). however, this approach does not leave indexes and classification schemes free from further scrutiny. campbell explains:                   the makers of new classification systems will be expected to articulate their positions relative to the community for whom the system is designed. this position will make a fundamental part of the tool’s nature, and will be the means whereby readers, users, and critics of the system will rebel, and find their own provisional categories (p. 129).   even as organization systems endeavour to include a multiplicity of communities within their systems, contentions can and do arise within those same communities as to how they understand themselves in relation to the labels used to define them (p. 130). this fits with bertrand-gastaldy et al.’s (1995) assertion that reading is both an individual and a social construction. in other words, the idea of universality within communities is itself a myth. as campbell says, “categories are fluid and unstable” (p. 130). this does not negate the importance of acknowledging the concerns of communities. rather, “that the tough questions are here to stay, and that complexity, debate, and controversy can be negotiated, but not banished” (p. 130).                   while joudrey (2005) acknowledges the validity of the “subjective, interpretivist viewpoint” his acknowledgement is cast-off with a statement that the “work of subject analysis must continue to be done”, and that “those arguing against the concept of extensional aboutness appear to be embracing an extreme view of relativism, one that is ultimately untenable for information organization” (p. 58). his assertion suggests that the rejection of extensional aboutness is a mere philosophical musing that has no place in the “real” world. joudrey’s statements seem fractional, incomplete, and complacent. to acknowledge the subjective and social constructs that inform knowledge organization is to work within the confines of an ever-changing reality. this reality is not built of absolutes waiting to be discovered and applied in the completion of what was once incomplete. rather, it is a recognition that organizational processes are themselves temporary and unstable precisely because the language and societies within which they function are also of this nature.   aboutness from the user perspective   the notion of the user and her requirements also featured prominently in the literature examining aboutness determination. since indexers serve a community of users, however broad or narrow that may be, a focus on users, their behaviour, and requirements was thematic to the literature and paramount to the findings.   in addition to supporting the notion of extensional and intensional aboutness, wellisch (1996) extends the discussion by bringing in fairthorne (1971) and lancaster’s (1991) call to consider not only what the document is about, but its intended use (p. 7). he uses the term aboutness to describe the conceptual analysis of a document from a “two principal criteria – what does it mention and for whom is it intended?” (p. 7). topics are assigned following the aboutness determination process which is based on the response to the questions above. for wellisch, relevance is an additional consideration in the selection of topics, especially in the case of depth indexing. the indexer must ask, is this “relevant to the aboutness of the document…and will this statement, fact, issue, problem, opinion or belief expressed in the text of the document be relevant to the prospective users of indexes” (p. 7). wellisch’s discussion of user relevance raises an important point in the discussion. the degree of depth in indexing relates to the level of specificity desired by index users. the indexer must have an idea about who the users of the database are to inform her decisions about relevance and specificity.   maron (1977) examines the concept of about from its perspective within information retrieval. maron identifies three types of about: s-about (subjective about); o-about (objective about); and r-about (retrieval about). s-about is defined as “a relationship between a document and the resulting inner experience of its reader” (p. 41). in other words, for s-about to be successful, a document’s aboutness must relate to the reader’s personal experience of it. s-about is a complex psychological concept that “cannot be examined further in objective terms” (p. 41). o-about, however, is based on observable, individual behaviour, and “refers to the (actual or potential) behaviour of asking or searching for writings. it is the external correlate of s-about because it would be actually (or potentially) observable by an external observer in a retrieval situation” (p. 41). r-about relates to the observable behaviour of groups or classes of individuals, such as a group of people with who actively engage with an information retrieval system, in contrast to individual behaviour observed through the notion of o-about. it is based on the probability that what a group of users find most relevant within a given document is the term or concept most likely to be used in searching. in others words, a document is about dragonflies if most of the people who found it relevant used the term dragonflies during their search (olson, 2010, slide 13). the merit to maron’s argument is the direct correlation between aboutness determination and user perception and behaviour. current subject indexing practice of library catalogs, for example, involves the consideration of a user, but in practice, few links are being made between aboutness determination and information retrieval behaviour. indexers are encouraged to consider audience and/or their users when determining aboutness without having any real sense of their information seeking behaviour. one way that indexers could access their users is through analysis of folksonomies. there are challenges to analysing folksonomies, most notably that tagging serves two functions: the user's personal collection, and the community of users (gerolimos, 2013, p. 42). what has been observed is that tags attributed within personal information spaces do not necessarily aid in information retrieval for a community of users. for folksonomies to provide insight into how users attribute aboutness to a resource, it would be necessary to assign tags with the community purpose in mind. for a detailed review of the effectiveness of tagging systems for library catalogs, see gerolimos.   albrechtsen (1993) advocates a requirements-oriented approach to subject analysis. the focus of the requirements-oriented approach is based on the intent to convey the knowledge of the document to those who may be interested or find it useful. in other words, the document is analysed for the potential knowledge it offers to prospective groups of users. the ability of indexers to predict or forecast user behaviour and to determine the priority of subjects based on supposition is indeed a tricky assignment. regardless, albrechtsen argues that indexers should pick-up the “challenges posed by the social and cultural reality within which we operate…new frameworks like requirements-oriented approaches have potentials for supporting a broad and open transfer of knowledge, which is a primary responsibility of our profession” (p. 223).   user based approaches to aboutness determination are inherently problematic. while librarians do have a certain knowledge surrounding their community of users, they cannot know the full extent of information use, nor can they be certain of when individuals deviate from standard user behaviour within their respective communities. still, user based approaches provide challenges that present great opportunities for information interaction. indexers are presented with the opportunities to move beyond “modest, value-free ethics for dissemination of knowledge” towards a “new consciousness of the impact of our profession for mediating knowledge” (albrechtsen, 1993, p. 223). the task of mediating knowledge provides indexers with a level of responsibility that compels them to engage in an ongoing discussion with their communities.   theme and rheme   another two-type distinction, borrowed from linguistics, and applied within lis, is the notion of theme and rheme. hutchins’s (1977) article is frequently cited with reference to theme and rheme. the oed defines theme as “that part of a sentence that indicates what is being talked about”, and rheme as “that part of a sentence or utterance giving new information about the theme” (oed, 2011, theme and rheme).   hutchins offers a reading into the notion of aboutness within the context of text linguistics which is beyond the limits of this discussion. of interest, however, is hutchins’s description of the problem of relevance, which relates to the “persistent and perhaps inherent conflict between what readers regards as the ‘aboutness’ of a document and what indexers define as its ‘aboutness’” (p. 34). readers are interested in what is new to them in a document, whereas indexers are concerned with the “given framework” (p. 34). “for the reader , relevance is a function of his current interests and his personal ‘state of knowledge’; for the indexer, relevance is a function of the place of the document in the current ‘state of knowledge’ as a whole” (p. 34). hutchins concludes that the indexer can only concern herself with those parts of the document that form the “knowledge base upon which the writer builds the ‘new’ information she tends to convey” (p.34).   akin to the notion of theme and rheme is weinberg’s (1988) discussion surrounding the concepts, aboutness and aspect. weinberg claims that indexing fails the scholar or researcher because it places its emphasis on aboutness while neglecting aspect. to better explain, she compares aboutness and aspect to their linguistic analogs, topic and comment. topics are defined as “subjects of discourse”, while comment as “that part of the utterance that adds something new” (p.4). weinberg claims that topics are particularly relevant for the student or general reader looking for information on a given subject. the scholar or researcher, however, is rarely in need of topical information. rather, she seeks content that comments on a particular aspect of a given topic (p. 3). through repeated observation, weinberg has observed that scholars rarely use subject indexes because they lack aspectual perspective.   hutchins (1978) also distinguishes between two types of document need. the first concerns the reader who is interested in acquiring information on a specific topic that she may know nothing or very little about. her need is based on satisfying a basic understanding of a subject. the second type concerns readers who are already well-versed in a subject, but are looking for a new approach or perspective on it. both share the need for something new within the information they seek, but the first type is looking for something foundational, while the second pre-supposes a certain level of knowledge, that while the information may speak to that level of knowledge, it also offers something new (p. 178).   weinberg’s dissatisfied subjects, however, would likely not be fulfilled with hutchins’s suggestion for indexing. what hutchins suggests is a “definition of the ‘aboutness’ of documents which is formulated in terms of the knowledge presupposed by the authors of the texts” (p. 178). hutchins lays the burden of expressing the newness of a document onto abstracting services since the use of summarization provides a greater opportunity to inform users in greater detail as to what the author has had to say on a particular topic (p. 180). the pangs of neglecting what is new in a document may no longer be felt as readily given new discovery tools. for example, worldcat provides a summary of the resource, tags, table of contents, and reviews, and looks much more like the amazon interface than a library catalog. the responsibility of aboutness is not relegated to subject heading attribution alone, but can be determined through a number of fields. if the subject index is formulated by the knowledge presupposed by the authors of the text, than it can only serve as a starting point, not a direct link, for those seeking what is new in a document. here additional discovery tools may be necessary.   the notion of subject   joudrey’s (2005) argument was that the term subject was riddled with complexity as a result of its longstanding position within the philosophical and literary realms. while it is true that in the oed the term subject clearly has a greater list of definitions and historical references, it remains unclear whether the complexities are thereby eliminated by changing the reference to the term aboutness.   to accept, or to not accept, the distinction between aboutness and subject, while certainly a noteworthy debate, is not the intention of this paper. rather this paper seeks to provide an analysis of the issue of aboutness in lis, and the distinction between terms is merely one component. to neglect the notion of subject because of disputes in terminology would be to do injustice to the literature. therefore the following section provides an overview of the notion of subject within the literature. in some examples, the notion of subject is intricately linked to the practice of subject determination. for that reason, the following discussion will also include details on various approaches to subject determination as a manner of understanding the notion of subject.   ranganathan (2006) discusses the notion of subject as an “organized or systematized body of ideas, whose extension and intension are likely to fall coherently within the field of interest and comfortably within the intellectual competence and the field of inevitable specialization of a normal individual” (p. 82). ranganathan’s definition falls very much within the framework of his classification, and the term is broken down further into basic subject, compound subject, complex subject, micro and macro subjects, and spot subject.   according to the classification research group (crg) (1957) the notion of subject as a simple isolated concept or theme “that can be neatly tucked away in a single pigeon-hole in the vast cabinet of knowledge” is insufficient (p. 139). rather, according to the group, it is a compound of simpler concepts. they explain that co-ordinate indexing systems and analytico-synthetic classification schemes recognize the compound nature of subject, the former by listing multiple subject terms in reference to one another, and the latter by use of classification symbols which form an aggregate of the one subject (p. 139).   reynolds (1989) also discusses the inadequacy of the “pigeonhole” approach to subject determination that attempts, through various clues such as table of contents, title, and summary statements, to find an overall theme and where it fits into the subject (p. 232). she states the ability to do so depends, not only on the work, but also on the reader (indexer). from a subjectivist perspective she states:                   we like to think that the text conveys a message, but that is only part true. readers (or cataloguers) project their own perceptions, experiences, and level of comprehension onto the text. each reading experience, even by the same person, is unique. there is always a degree of tension between the new information and what the reader already knows or believes (p. 232).   in an attempt to find a subject cataloguing code, reynolds (1989) acknowledges the need for it to have a theoretical basis, not simply a list of arbitrary procedures and rules. for this theoretical basis to be useful, however, it must be put to good use. at the same time, she claims the near impossibility of establishing such a code given that subjects are defined by individual perceptions that we can neither fully understand, nor describe (p. 234). despite, and in fact, given, the elusive nature of subjects, she suggests we re-evaluate the use of concepts such as relevance and specificity because their current role within subject determination is limited. in other words, is it possible to define specificity within a variety of contexts, or to discard the notion of relevance given the unpredictability of future information needs (p. 236)?   in response, it can be said that the terms are themselves not fixed, but function within a milieu of potentialities. seeking to find a stable definition of a term such as relevance in all circumstances is, as donovan leitch (1965) aptly said, like “trying to catch the wind”. the concepts that surround subject cataloguing must be malleable if we aim to eliminate philosophical frustration.   hjørland (1992) takes on the task, considered elusive by reynolds, of investigating the theoretical notion of subject and subject matter as it has been used in lis. he argues that an explicit investigation into the theoretical underpinnings is necessary in order to understand the process of subject determination. he explores five epistemological positions surrounding the concept of subject: the naïve conception; subjective idealism; objective idealism; the pragmatic concept of subject matter; and a realist/materialist subject theory. it is evident in hjørland’s examination, that the notion of subject is affected by the epistemological framework through which subject analysis takes place. in order to illustrate, a brief description of each is provided.   the naïve conception is not particularly burdened by the notion of subject, but understands subjects to be fairly obvious. there are gradations within the naïve conception, as hjørland talks about “a slightly less naïve viewpoint” that “would recognise that there need not be a correspondence between for example, the title of the book and its actual subject” (p. 172). still, the naïve view-point does not “differentiate between linguistic forms and meanings” (p. 173).   idealism is a fundamental philosophical tenet that claims that what exists in the mental realm is of greater consequence, or serves as a primary function and determinant of the external, material world. from this perspective, the concept of subject is first an idea, whether in a subjective or an objective sense. therefore, subjective idealism concerns points of view, whether they are from the author, reader, or indexer, while objective idealism takes the position that ideas, or subjects, are, in fact, objective realities with universal or fixed properties (pp. 173-179).   the pragmatic concept is concerned with the notion of subject based on the needs of users. the concept of subject is applied within the practical realm, anticipating the varying user requirements. “user-oriented or need-oriented indexing is a description of a subject which must be perceived as the relation between the properties of a document and a real or anticipated user need” (p. 180).         the last theory presented by hjørland, and the one he appears to endorse, is the realist/materialist subject theory, which is based on the “viewpoint that things exist objectively and encompass objective properties” (p. 181). similar to the pragmatic theory, the materialist theory suggests that the extent to which subjects represent the potential of documents will have an effect on “optimising the potential perception of the document” (p. 185). in contrast, however, realistic and materialistic theory has as its aim to penetrate the “innermost essence” of reality, so that subjects represent the “general and the significant aspects of reality” (p. 185). thus a subject description of a document is, in one way or another, an expression of the epistemological potentials of the document. the better the description predicts the potentials of the document, the more correct, more objective, the description of the subject is (p. 186). while hjørland’s examination may appear to abide within the philosophical realm, his investigation is applicable to the discussion of subject analysis because it raises to the forefront the epistemological positions that influence the process of subject determination. as hjørland says, “a very close connection exists between what subjects are, and how we are to know them” (p. 172).   according to langridge (1989), there is a greater need for precision surrounding the term subject. he claims that the term subject has been used exclusively in what are invariably two distinct senses. these two distinct meanings are related to the following two questions asked of a work: what is it? and what is it about? (p. 9). the first question is answered by reference to what langridge terms forms of knowledge, which may include history, philosophy, music, science, and so on. the second question is answered by observable facts or events, known as phenomena. langridge employs the term topic to describe this approach. he clarifies the distinction between forms of knowledge and topics based on their aims and methods. to explain further:                   philosophy aims to clarify by examining beliefs, science to reveal natural laws, criticism to evaluate, technology to facilitate production. there are very obvious differences between such forms of knowledge, ways of looking at the world, and the topics they discuss, such as morals, animals, symphonies and steam engines (p. 31).                 wilson (1968) is an important contributor in the discussion surrounding aboutness determination and the concept of subject. his discourse on subject determination is concerned with the movement from understanding the parts of a writing to knowing what the writing is about as a whole (p. 78). he outlines four possible methods in the determination of subject, and as a result, demonstrates the complexities surrounding the notion of subject. usefully, his exposition of each method also describes the potential problems and deficiencies therein.   the first he calls the “purposive way”, which relates to the identification of the author’s purpose in writing. in some cases the author’s intentions are clearly outlined by the author, but other instances may require a detailed examination in order to be deciphered. there are problems with both approaches. the author may misrepresent her purpose, fail to express her purpose, or fail to achieve a definite purpose. the analysis may also be thwarted in attempts at finding a primary purpose to the writing. writings may, in fact, be constructed of subjects independent of one another, and “recognition of those (independent aims) requires an ability to see which of the things done or attempted in the writing are done only because necessary as a means to an end, and which are done ‘for their own sake’” (p. 80).   his second approach is known as the “figure ground” way. this approach is based on the idea that there is dominant subject in a writing which “stands out” or is “most emphasized” (p. 82). the problems herein apply to the argument surrounding what constitutes the dominant subject. wilson explains, “dominance is not simple omnipresence; what we recognize as dominant is what captures or dominates our attention, but we cannot expect that everyone’s attention will be dominated by the same things” (p. 83).   again, wilson offers another approach to subject determination, this one based on the notion that dominance can be determined from an objective stand-point. more specifically explained, the objective way involves counting references to items within a writing. it is an objective correlate to the notion of dominance. however, wilson is quick to contend, the objective way is plagued with the problem that the item most frequently represented may not be the dominant subject of the writing, but rather background. as well, the primary concept might be signified in various ways, or may not be expressed concretely within the writing. in fact, says wilson,                   one can always rewrite a text in such a way as to reduce the number of references to any item and increase the number of references to any other without materially altering           the general sense of the writing or even, if one were skilful enough, changing the balance of impressions of dominance and subordination (p. 83).           wilson’s final method is based on the “appeal to unity” (p. 86). for a work to hold together, there must be something that binds it; that holds it together; that makes it complete. the binding component is based on that which is necessary versus that which is dispensable in a writing. once again, wilson is forthright in his acknowledgement of the problems of this ideal. after all, not all writers attain unified writings. in certain circumstances this may be the result of a certain deficiency in ability, but in other cases, the writer may lack a subject simply because questions are left unanswered. wilson explains that our efforts to take the unified whole, what he calls the “cast of characters”, and extract one subject from the cast may not reveal what the subject really is, but “may result in a piece of artistry on our (librarians) part, rather than on the part of the writer” (p. 88).   in conclusion, wilson argues that “the notion of the subject is indeterminate” (p. 89). he acknowledges that things are what they are, but that descriptions are vague, nebulous, and inexact (p. 90). while there can be incorrect descriptions, there are also no perfect descriptions of the subject. “the uniqueness implied in our constant talk of the subject is non-existent” (p. 90). therefore, the position that a writing is given in an organizational scheme is based on the methods used to determine its subject. it is possible to deduce that items positioned in subject proximity share some characteristic in common. this does not suggest that these writings belong exclusively within selected positions. on the contrary, the indeterminacy surrounding the notion of subject means that their assignment is not definite. thus, “the place has no definite sense” (p. 91).   what is particularly revelatory in wilson’s analysis is his demonstration of the flaws inherent in each of his methods. he does not attempt to hide the defects, nor “brush them under the table” as though insignificant. rather, his four methods illustrate the impossibility of finding the perfect subject through the perfect method. at no point, however, does wilson suggest that the search for methodologies is in vain. after all, he, himself, provides four approaches to subject determination. instead, wilson’s exposition suggests that there is great benefit in understanding what is insufficient in our methods and impossible in our attempts.   stages in organization   several identify stages to subject indexing, and aboutness determination is often considered the preliminary stage. although stages are outlined, the literature demonstrates that the movement from aboutness determination to subject heading attribution is anything but linear, especially among seasoned indexers. this section of the paper examines how these stages have been identified, and their inter-influence.   according to iso 5963-1985, there are, in fact, three stages to indexing (p. 2). they are:   1. examining the document and establishing the subject content; 2. identifying the principal concepts present in the subject;                3. expressing these concepts in the terms of the indexing language.   perhaps it is for the sake of convenience and analysis that these stages are identified as three separate processes within this international standard. ranganathan (2006) also discusses the notion of stages in what he calls “the three planes of a work” (p. 327). ranganathan’s three planes are known as the ideas plane; the verbal plane; and the notation plane, and share similarities with iso 5963-1985.   while iso 5963-1985 acknowledges that these processes may overlap, they do not concern themselves with the influence of intersection between stages. ranganathan does acknowledge the potential influence that may take place between the planes, although he also claims that it is possible to separate the three planes. of course, ranganathan is working within the framework of his own particular system, while iso is providing guidelines for indexing as a general phenomenon, thus making it difficult to deal with too many specifics.   while it is not within the realm of this analysis to examine this issue in great detail, it is certainly an area that requires additional research, for it is indeed rare for processes so closely connected to be separable without some degree of influence, unless, of course, purposefully calculated. the following section illustrates research performed in one or more stages of the iso 8963-1985. in this paper, a link is made between iso 8963-1985 and the research presented herein.   joudrey’s (2005) dissertation research focuses on the conceptual analysis processes involved in aboutness determination. his research was set forth for the purpose of finding clues into how humans analyze documents to determine aboutness. according to him, there is a distinction between conceptual analysis and subject analysis, for the latter involves both the conceptual analysis stage and the translation process.   joudrey expected to find patterns emerging in the conceptual analysis processes of his subjects, as well as the use of bibliographic, content, or visual features such as table of contents, chapter headings, titles, illustrations, and so on (p. 10). he makes a distinction between the processes involved in aboutness determination and the bibliographic features used in conceptual analysis. the distinction appears to relate to the first two stages presented in iso 5963-1985. the first stage involves an examination of various features of the work, followed by the analytical processes involved in the identification of concepts.   reynolds (1989) makes a similar distinction between the mental processes involved in aboutness determination and the examination of clues provided in the work advocated by cataloguing texts (p. 232). wilson, too, comments on the manuals of library practice that outline common recommendations in the examination of a work to determine its subject. these manuals, “full of references to ‘the subject’of a writing", are “curiously uninformative about how one goes about identifying the subject of a writing” (p. 73).   the examples above demonstrate an interest in moving from the application process to the intellectual process as a manner of understanding what informs decision-making. understanding how cataloguers determine or identify topics and select appropriate subject descriptions was also the basis for šauperl’s (1999) dissertation research. the background to her research was based on the lack of illumination surrounding the cognitive processes involved in identifying the topic of a work followed by the selection of subject headings. similar to joudrey, šauperl mentions that cataloguing manuals provide lists of parts of a work that should be consulted in the determination of subjects, but fail to address the selection process itself. in other words, cataloguing guidelines provide us with clues in the identification of topics, but do not explain how cataloguers select topics for subject representation. the goal of her research was to investigate the cognitive processes of topic identification and subject description from a holistic perspective. while she does not define holistic, it is assumed to mean a complete examination surrounding the conscious intellectual activity of aboutness determination, and is usually based on or capable of being reduced to empirical factual knowledge. research of this nature is concerned with the cognitive processes in information organization, and not merely in the application of applied techniques by catatonic cataloguers.   although not the basis of her dissertation research, nonetheless, šauperl did indeed observe the overlapping of the three stages mentioned above, particularly with respect to the last two stages. in fact, šauperl’s samples were found to have selected tentative headings following an examination of the document, and to have applied these headings through a search of titles or subject headings, followed by an analysis of the search results to determine the appropriateness of the subject headings in relation to the tentative headings. in other words, her observations revealed that the identification and translation processes were not conducted separately. as noted by šauperl, this approach “is sensible in terms of collocation” but may be problematic “in the assumption that all the documents described with the same subject heading or a set of subject headings actually address the same topic” (p. 255).   šauperl’s samples seem to exemplify hickey’s (1976) claim that american librarians have no clear philosophy surrounding the issue of subject control, in part due to their reliance on lists of headings and classificatory divisions that are centrally issued and updated (p. 275, 288). hickey says that “since the library of congress has only infrequently published any official explanation of the principles underlying the maintenance of its list and schemes, it is not surprising that most librarians are unable to state with any assurance the basis for selection of subject terms and classification symbols beyond the general rule of ‘specificity’” (p. 275). hickey raises an important point in the discussion of aboutness, namely the issue of applying preferred terms that may not be explicitly defined or understood by the cataloguer. as well, the preferred terms themselves may be insufficient in their descriptions of the contents. there is indeed a problem in applying the preferred term, feminism, to a work written from a feminist perspective. the two are clearly not the same thing (olson, personal correspondence, july 20, 2011).   in a similar manner to joudrey (2005), chu and o’brien’s (1993) study focuses on the initial process of subject analysis, which involves analysing the text and expressing the subjects in natural language. their study did not include the translation stage from natural language to indexing terms so as to not hinder the analytical process by the act of trying to fit terms into a controlled setting. considering the activities of šauperl’s samples, it was intuitive of chu and o’brien to neglect the translation stage.   interestingly, their investigation revealed that bibliographic elements were a major factor in determining aboutness, but that the level of difficulty in using these elements depended in great part on the discipline of the resource. for example, bibliographic elements for texts within the humanities, (their study employed documents in the humanities, social sciences, and sciences), were much less factual than the sciences, and, as a result, more difficult to decipher subject content. another point of interest relates to the difficulty participants had in isolating primary and secondary topics. their problems related to the issue of prioritisation, as some texts seem to have two subjects of equal priority, especially within the sciences (p. 453). this observation fits with wilson’s (1968) discussion of dominance, and the problems associated with determining one dominant subject over another. as well, their research suggests that subject analysis carries different issues based on the discipline of the subject being analysed. it begs further investigation into the study of the conceptual analysis process as it applies within varying disciplines. the rules that guide one may not be suitable for another. in other words, “one size may not fit all”.   research in the cognitive structure of classification by hovi (1988) demonstrates that classifiers most commonly approach a document from a theme-oriented approach, and references to other parts of the text, such as its “newness”, or to the user and his/her purpose with the text were less common (p. 127). her samples were nearly unanimous in the determination of a book’s main subjects. hovi does not question the unanimity of the theme-oriented aboutness determination of her subjects, but is more interested in the influence that classification systems can have on the conceptualization of the subjects derived from a book. her research, which included all three stages, was predominantly concerned with the translation stage. she explains that “in spite of the fact that the classifiers were unanimous about the main subjects of the books, they picked up the ‘important concepts’ in different ways according to the classification system” (p. 130). for hovi’s participants, the classification system influenced the selection of important concepts.   this researcher, rondeau (2012), also conducted qualitative research using a hermeneutic phenomenological approach with the intention of seeking to understand the cataloger's lived experience of aboutness determination. in addition to outlining degrees of variability in determining the aboutness of a resource, as well assuming the role as mediator, the catalogers that were interviewed also demonstrated a predisposition towards the systems and structures that they used in later stages of subject work. in other words, the systems and structures they used influenced their work in determining the aboutness of a resource. not only would catalogers consult with subject heading systems, but depending on their familiarity with the subject matter, and years of cataloguing experience, they would immediately start thinking of headings in their initial examinations. this research, like that of hovi and šauperl, is significant because it demonstrates that the movement between stages is not necessarily a linear one. from the researcher’s perspective this suggests that the systems may not be serving analysis, but defining it.   discussion   as mentioned earlier, an aboutness concept map (see figure 1) has been drafted to provide a visual summary of the paper's findings. the findings of this review began by examining how the lis community has applied and responded to the term aboutness in relation to its correlates. the literature demonstrates that for every agreement on terminology there is disagreement; some claiming aboutness more fitting to the task of indexing than its correlate subject, others arguing no difference, and others choosing different terminology altogether. although arguments of terminology can be helpful in understanding frameworks of meaning, this review also sought to examine how the term aboutness and its correlates have been understood and defined.     aboutness as a two type distinction featured prominently in the literature. the notion of a document's inherent aboutness has been guiding indexers since the advent of the objective ideal. in other words, indexers have been taught to put aside interpretation, or tendencies to become subjectively involved with the resource, in favour of a detached and stable approach that extracts what is intensional in the document. however, not all agree that a document has an inherent aboutness. some authors argue that words and their meanings cannot be separated from the community of readers, and that there cannot be a precise and singular description of what a document is about, not only because of the difference in readers, but also because authors may not be singular or have clear and easily distinguishable subject-matter in their writings. instead of favoring a subjective approach over a two-type distinction that seeks to eliminate the possibility of too much interpretation, this researcher suggests a more holistic approach to aboutness determination. with this approach, the indexer is engaged with the resource in a manner that does not deny her interpretation, but through openness and questioning, continually seeks to clarify when that response has become distorted. in other words, the relationship between indexer and resource becomes inter-subjective, involving an engaged manner of relating to the resource that recognizes the indexer's ability to extrapolate meaning, but within a framework that remains pointed towards the resource.   aboutness from the user perspective also featured prominently in the literature. through this approach, indexers consider not only what the resource is about, but also in consideration of the user. the challenges of this approach are evident in that indexers may not have a full understanding of their user communities or be able to anticipate all user needs. some argue that despite these concerns, indexers would do well to mediate that knowledge on behalf of users. perhaps it need not be a "guessing game." studies in user behaviour may help to identify how user communities think about unknown item needs. analysing community-oriented folksonomies may be one way of gaining access to user behaviour, as well as search-log analysis, and staying abreast of the community's discourse through active engagement. relegating the indexer to the backroom without providing insight into her community of users may create an unnecessary divide. this divide may be further expounded by using subject heading systems that don't accurately represent the discourse of the community or the resource. this researcher suggests that, despite the initial cost of implementation, it may be time for disciplinary communities and indexers to define their discourse, that indexers continuously engage in this dialogue, and become active mediators between the resource and their community of users.   theme and rheme is another two-type distinction in the discussion surrounding aboutness determination. despite discontent among scholars that subject indexes fail to address what's new in the document, some argue that the indexer cannot concern herself with what is new in the resource, but must be satisfied to determine its placement within a framework of knowledge. this viewpoint provides a clear definition of the task of aboutness determination as it relates to the subject index. however, it also restricts the benefits of the index to an important group of users, many of whom have the skills and knowledge to effectively exploit the subject index. while scholars may be grateful for summarization and new discovery tools that allow them to garner a more detailed understanding of the surrogate records, as a tool for discovery, the subject index only proves useful as a starting place for those seeking what is new in the document.   despite differences of opinion surrounding terminology, the notion of subject was also explored in this paper because of its prominent and noteworthy place within the literature. what was paramount to the literature on the notion of subject was the need to extend beyond simple concepts and practices related to subject identification towards an epistemological framework. as well, the issues associated with methodologies surrounding the determination of a document's subject were presented, as well as how methodologies may be stunted by the document itself. revealing the limitations of subject indexing may leave some feeling defeated by the task. however, rather than submitting to defeat, it challenges the lis community to devise a broad range of opportunities for discovery that may not be achievable through a singular notion of subject.   lastly, this paper explored the literature examining stages in the subject indexing process. this literature was found important because aboutness determination is defined as a distinct stage in indexing, and yet proved to be influenced by later stages in the process, especially among those with some level of experience. given the influence of later stages, it raises questions around the meaningfulness and relevance of aboutness determination as a distinct stage if it is being influenced by subject heading systems. is it possible to "meet" the document freshly within a framework whereby subject headings are ascribed by external bodies? in what ways do indexers lose their power to mediate on behalf of their users by ascribing to such systems, and how do these systems reflect the needs of their users? these are questions that require further exploration.   conclusion   one of the key issues is the complexity of terminology and the various definitions applied to terms in an attempt to understand phenomena presented in documents. this relates to the transient and ever-changing quality of language as a result of its subservience within human communities. the debates within communities further highlight the inconsistency of language use. this is exemplified in lis with respect to the term aboutness. as discussed, some find the term aboutness most fitting, while others regard it as unclear, and even problematic. as alfred korzybski said, “the word is not the thing”. language is used to describe the thing, and the number of ways that it may be described is indeed remarkable. remarkable…and tricky, especially in light of considerations surrounding consistency, relevancy, and specificity, and under the framework of various epistemological viewpoints. as well, as wilson (1968) points out in his discussion surrounding the indeterminacy of the notion of subject, there may not even be a single “thing” that can be clearly determined.   while the issue of terminology may be left unsolved, nonetheless, as joudrey (2005) says, the “work of subject analysis must continue to be done” (p. 58). the process in which this task is realized is indeed an additional anomaly. while it appears to be obvious and straightforward as expounded by iso 5963-1985, in reality, it is a process of inter-influential stages that lack clarity and specific direction. as well, the lack of clarification surrounding references created by distant sources further complicates the issue and may create a hierarchical divide between indexers and those in charge of the creation and maintenance of headings. libraries have been willing to pay for a central body to provide controlled headings, but at what cost to their communities? might there be greater benefit in giving disciplinary communities the power to name their subject matter accordingly? within these communities, the indexer acts as the mediator, bridging the gap between resource and user in a way that is meaningfully relevant to those using the index. this may involve on-going interaction between indexers and those they serve; one that invites conversation, and potentially even controversy, but one that draws the indexer out of the backroom and into the discourse.     figure 1 aboutness concept map.     references   (references marked with an asterisk indicate studies included in the concept map).   aboutness. (n.d.). in oxford english dictionary. retrieved 21 feb. 2014 from http://www.oed.com .   *albrechtsen, h. (1993). subject analysis and indexing: from automated indexing to domain analysis. the indexer, 18(4), 219-224. retrieved 21 feb. 2014 from http://www.theindexer.org/files/18-4/18-4_219.pdf   *beghtol, c. (1986). bibliographic classification theory and text linguistics: aboutness analysis, intertextuality, and the cognitive act of classifying documents. journal of documentation. 42(2), 84-113. doi:10.1108/eb026788   *bertrand-gastaldy, s., lanteigne, d., giroux, l., & david, c. (1995). convergent theories: using a multidisciplinary approach to explain indexing results. in t. kinney (ed.), proceedings of the american society for information science, vol. 32. 58th annual meeting, chicago. (pp. 56-60) medford, new jersey: information today.   *boyce, b. (1982). beyond topicality: a two-stage view of relevance and the retrieval process. information processing and management, 18(3), 105-109. doi:10.1016/0306-4573(82)90033-4   *campbell, g. (2000). queer theory and the creation of contextual subject access tools for gay and lesbian communities. knowledge organization, 27(3), 122-131.   chu, c., & o’brien, a. (1993). subject analysis: the critical first stage in indexing. journal of information science, 19(6), 439-454. doi:10.1177/016555159301900603   *classification research group. (1957). the need for a faceted classification as the basis of all methods of information retrieval. in proceedings of the international study conference on classification for information retrieval, beatrice webb house, dorking, england.   *fairthorne, r. (1969). content analysis, specification, and control. annual review of information science and technology 4, 73-109   gerolimos, m. (2013). tagging for libraries: a review of the effectiveness of tagging systems for library catalogs. journal of library metadata,13(1), 36-58. doi:10.1080/19386389.2013.778730   hickey, d.j. (1976). subject analysis: an interpretive survey. library trends, 25(1), 273-292.   * hjørland, b. (1992). the concept of subject in information science. journal of documentation, 48(2), 172-200. doi:10.1108/eb026895   *hjørland, b. (2001). towards a theory of aboutness, subject, topicality, theme, domain, field, content…and relevance. journal of the american society for information science, 52(9), 774-778. doi:10.1002/asi.1131   hovi, i. (1989). the cognitive structure of classification work. in s. koskiala & r. launo (eds.), information, knowledge, evolution: proceedings from the forty-fourth fid congress held in helsinki, finland, 28 august – 1 september 1988. amsterdam: fid.   hutchins, w.j. (1977). on the problem of aboutness in document analysis. journal of informatics, 1(1), 17-35.   hutchins, w.j. (1978). the concept of aboutness in subject indexing. aslib proceedings, 30(5), 172-181.   international organization for standardization (1985). iso 5963-1985 documentation methods for examining documents, determining their subjects, and selecting indexing terms. geneva: international organization for standardization.   joachim, h. h. (1906). the nature of truth iv 174. in oxford english dictionary. retrieved from http://www.oed.com .   *joudrey, d. j. (2005). building puzzles and growing pearls: a qualitative exploration of determining aboutness (doctoral dissertation). retrieved from proquest dissertations and theses database. (umi no. 3206794)   lancaster,f.w. (1991). indexing and abtsracting in theory and practice. champaign, il: university of illinois, graduate school of library and information science.   *langridge, d. w. (1989). subject analysis: principles and procedures. london: bowker-saur.   leitch, d. (1965). catch the wind [vinyl single]. u.s.a: hickory.   *mai, j. e. (1999). a postmodern theory of knowledge organization. in woods, l. (ed.), information science at the dawn of the millennium, proceedings of the annual conference of the canadian association of library science, toronto: cais.   *maron, m. e. (1977). on indexing, retrieval, and the meaning of about. journal of the american society for information science, 28(1), 38-43. doi:10.1002/asi.4630280107   metcalfe, j. (1973). when is a subject not a subject? in c.h. rawski (ed.), towards a theory of librarianship. new york: scarecrow press.   noun. (n.d.). in oxford english dictionary. retrieved 21 feb. 2014 from http://www.oed.com.     olson, h. (2010). notes from a lecture on maron’s three different kinds of aboutness. unit 9: subject analysis, l&i sci 531: organization of information. university of wisconsin-milwaukee.   *ranganathan, s. r. (2006). prolegomena to library classification. new delhi: ess ess.   *reynolds, s. j. (1989). in theory there is no solution: the impediments to a subject cataloguing code. library quarterly, 59(3), 223-238. retrieved 21 feb. 2014 from http://www.jstor.org/stable/4308378   rheme. (n.d.). in oxford english dictionary. retrieved 21 feb. 2014 from http://www.oed.com.   rondeau, w.g. (2012). the lifeworld in the library's backroom: a hermeneutic phenomenological study of the cataloguer's lived experience of aboutness determination (master's thesis). retrieved from theses and dissertations. (paper 42) http://dc.uwm.edu/etd/42/   šauperl, a. (1999). subject determination during the cataloguing process. (doctoral dissertation). retrieved from proquest dissertations and theses database. (umi no. 9943257)   swift, d. f., winn, v., & bramer, d. (1977). a multi-modal approach to indexing and classification. international classification, 4(2), 90-94.   swift, d. f., winn, v., & bramer, d. (1978). aboutness as a strategy for retrieval in the social sciences. aslib proceedings, 30(5), 182-187. doi:10.1108/eb050630   theme. (n.d.). in oxford english dictionary. retrieved from 21 feb. 2014 from http://www.oed.com.   the practical philosopher. (2011, april 20) alfred korzbyski: the word is not the thing [web log post]. retrieved from http://thepracticalphilosopher.com/2011/04/20/the-word-is-not-the-thing/   *todd, r. j. (1992). academic indexing: what’s it all about? the indexer, 18(2), 101-104. retrieved 21 feb. 2014 from http://www.theindexer.org/files/18-2/18-2_101.pdf   *weinberg, b. h. (1988). why indexing fails the researcher. the indexer, 16(1), 3-6. retrieved 21 feb. 2014 from http://www.theindexer.org/files/16-1/16-1_003.pdf   *wellisch, h. h. (1996). aboutness and selection of topics. keywords, 4(2), 7-9.   *wilson, p. (1968). two kinds of power: an essay in bibliographical control. berkeley: university of california press. retrieved from http://books.google.com/    evidence based library and information practice       article   library website visits and enrollment trends   linda anderson library website coordinator library information technology services iowa state university ames, iowa, united states of america email: landerso@iastate.edu   received: 4 june 2015                                                                     accepted: 27 oct. 2015         2016 anderson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – measures of trends in iowa state university library website visits per student/faculty/staff headcount show decreased use. analysis was conducted to test for a relationship between this decrease and decreasing graduate/undergraduate enrollment ratios and decreasing visits to a popular digital collection. the purpose was to measure the influence of these factors and to produce an adjusted measure of trend which accounts for these factors.   methods – website transaction log data and enrollment data were modelled with box and jenkins time series analysis methods (regression with arma errors).   results – a declining graduate to undergraduate enrollment ratio at iowa state university explained 23% of the innovation variance of library website visits per headcount over the study period, while visits to a popular digital collection also declined, explaining 34% of the innovation variance. rolling windows analysis showed that the effect of the graduate/undergraduate ratio increased over the study period, while the effect of digital collection visits decreased. in addition, estimates of website usage by graduate students and undergraduates, after accounting for other factors, matched estimates from a survey.   conclusion – a rolling windows metric of mean change adjusted for changes in demographics and other factors allows for a fairer comparison of year-to-year website usage, while also measuring the change in influence of these factors. adjusting for these influences provides a baseline for studying the effect of interventions, such as website design changes. box-jenkins methods of analysis for time series data can provide a more accurate measure than ordinary regression, demonstrated by estimating undergraduate and graduate website usage to corroborate survey data. while overall website usage is decreasing, it is not clear it is decreasing for all groups. inferences were made about demographic groups with data that is not tied to individuals, thus alleviating privacy concerns.     introduction   library use is a measure of implied value (tenopir, 2013). measuring changes in usage over time and the impact of internal and external factors on usage is of interest as libraries are looking for ways to demonstrate continued library value.   one aspect of library usage is library website usage. changes to the website may or may not affect the number of visits to the website, but it could affect usage of specific services or resources by making them more visible than before. while design changes may be based on research prior to the redesign, the effect of changes can also be evaluated after they are made, using both qualitative methods, such as usability studies, and quantitative methods, such as transaction log analysis.   the iowa state university library website includes a discovery service through ex libris’ primo, lists of article indexes and databases, e-journals, course reserves, “ask us!” online reference service, digital collections, special collections, and general information about the library and library services. to evaluate website usage over time, enrollment levels, which have been increasing at iowa state university, must be taken into account.   library website usage data (visits as defined by ip address) from server transaction logs are analyzed in this paper. three factors are included in the analysis: increasing enrollment (using website visits per headcount as the dependent variable); graduate to undergraduate enrollment ratio; and visits starting on the george washington carver digital collections pages.   in general, sources of visits to the website include robots, people unaffiliated with the university, and faculty, staff, undergraduate students, and graduate students from the university. robot visits are filtered by the awstats software. otherwise, ip addresses do not identify the group of the visitor. it would be possible to filter by on-campus or off-campus ips, but faculty, staff, undergraduates, and graduate students can all access the website from off-campus, and people unaffiliated with the university could access the website from an on-campus ip address. the graduate to undergraduate enrollment ratio is included in the model as this ratio is decreasing due to increasing undergraduate enrollment (from 21,607 in fall 2008 to 27,659 in fall 2013) and flat graduate enrollment (figure 1). the ratio should have an influence on visits per headcount as survey data shows that graduate students report more frequent library website usage than do undergraduates.   the george washington carver digital collections pages contain digitized photos, letters, and other documents related to botanist and inventor george washington carver, iowa state agricultural college's (later iowa state university) first black student and faculty member. the george washington carver visits are included because it seems plausible that many of these visits originate in the primary and secondary schools and many of the visitors are not affiliated with the university. these visits peak in february, which is black history month in the schools, and have been decreasing. this variable is included to remove a portion of the non-affiliated visits from the estimate, leaving an estimate of the mean yearly change that more closely reflects usage by students, faculty, and staff. worldwide, george washington carver-related search engine searches have been declining over the last several years, according to google trends.     figure 1 headcount enrollment, february 2008 through july 2014.     another analysis was conducted, with the goal of estimating the average marginal effect on number of library website visits per additional student from each of these groups: graduate students, level 2, 3, and 4 undergraduates, and freshmen, after accounting for other factors, and comparing this result with estimates of usage from survey results.   literature review   is it true that college students think (and act on) the statement "everything needed for research is available free on the web" (cochrane, 2007)? if so, is this tendency increasing?   the value of the library as a source of information has competition. students and faculty have choices besides the library website for starting their research, such as internet search engines, google scholar, google books, wikipedia, and hathi trust (education advisory board, 2011). liu (2008) wrote that academic library websites have to compete with many other sites which may seem more entertaining or easier to use, such as amazon, google, or youtube, although libraries provide higher quality scholarly information. connaway, dickey, and radford (2011) found that users chose sources which were convenient and “good enough,” with search engines as the most frequently used sources for graduate and undergraduate students.   accordingly, usage of library websites may be in decline. a study from oclc shows such a decrease among college students, from 61% in 2005 to 57% in 2010, although 22% of students who do use the website use it at least weekly, an increase of 7% over 2005 (de rosa, cantrell, carlson, gallagher, hawk, & sturtz, 2010). the measuring information services outcomes (miso) survey found a decrease in student usage of library websites between 2008 and 2010 (allen, baker, wilson, creamer, & consiglio, 2013). wood and walther (2000) reminds us that, although there is a wealth of free information on the internet, the profit motive remains strong for publishers, and patrons will need libraries to receive free access to subscription material.   nackerud, fransen, peterson, and mastel (2013) collected demographic data on licensed database, e-journal, and e-book usage and website logins at the university of minnesota via a “click-thru” script, and found that 65% of undergraduates used electronic resources or logged into the website, while 82% of graduate students did so.   marek (2011) offers comprehensive advice on setting up and using web analytics in a library. cohen (2003), jansen (2006), and goddard (2007) discuss technical details of web server transaction log analysis. transaction log analysis is more often used to measure cross-sectional aspects of website usage than trends over time (asunka, chae, hughes, & natriello, 2009; ke, kwakkelaar, tai, & chen, 2002; li, 1999; park & lee, 2013).   time series regression and autoregressive integrated moving average (arima) methods (box, jenkins, & reinsel, 2008) are usually used for forecasting. ahiakwo (1988), brooks (1984a and 1984b), and naylor and walsh (1994) have used these methods for forecasting circulation. all of these researchers included regression variables to improve their forecasting models. in this study, rather than being used to improve forecasting, the magnitude of the effect of the regression variables are of interest in explaining trends in website visits.   methods   transaction log analysis   the iowa state university library has been capturing and parsing transaction log data with awstats software and has data available on website usage since february 2008. awstats defines a library website visit as one or more page accesses during an hour by a single ip address. a unique visitor is defined by ip address as well.   visits data were cleaned and partitioned by using the counts for entry for each page. an entry page is the first page visited during a session. the total count for entries should equal the count of visits. some counts were discarded as they showed the entry page to be a url not belonging to the library, such as “http://www.styleusagroupco.com/.” visits starting on staff intranet pages were also discarded. two days had counts of zero and were assumed missing. interpolated values were added to the cleaned monthly count.   a plot of the cleaned total library website visits from february 2008 through july 2014 is shown with a plot of visits starting on special collections and isu digital collections george washington carver pages in figure 2.   a visits-per-headcount statistic was created by dividing the number of visits by the sum of enrollment and employment (students, faculty, and professional & scientific (p&s) staff headcount). this leaves out website visitors who are currently unaffiliated with the university, and other groups, such as university retirees and classified staff. the number of unaffiliated website visitors could vary substantially over time.   an average monthly student count was calculated for the months of august, december, january, and may for each year, which includes weeks when school is not in session. for weeks between semesters the number of undergraduates was set to zero, while the number of graduate students was set to the enrollment for the next semester.   usage rates of undergraduates, graduates, and faculty   the pattern in the graph of all visits (figure 2) is inverted in the graph of visits per headcount (figure 3). while the number of visits drops markedly in the summer and between semesters when there are few undergraduates around, the number of visits per headcount goes up markedly in the summer.       figure 2 plot of all library website visits with plot of visits starting on george washington carver pages.     figure 3 visits per month per total headcount of students, faculty, and p&s staff.     although in figure 2, the highest number of website visits coincides with the highest number of students present, it does not necessarily follow that students are the source of most of the visits. anecdotally, some undergraduates never or rarely use the library website. alternative explanations for the increase in visits could include an increase in usage by faculty or staff during the semesters, perhaps in preparation for classes or for research; it is also possible that there is some usage from the primary and secondary schools, which are also in session at roughly the same time.   in 2012, the iowa state university library conducted a survey to measure satisfaction, importance, and usage levels for library services and resources. from this survey, a rough estimate of the self-reported number of visits per month can be made for each of these groups: lower and upper division undergraduates, graduate and professional students, faculty, and p&s staff. freshmen were not included in the survey, so classification year 2 comprised lower division, and classification years 3 and 4 comprised upper division.   for answers to the question “how often do you use the e-library (i.e., library website)”, visits per month were assigned as follows to the answer choices:   daily: 16 weekly: 4 monthly: 1 once a semester: 0.3 less often: 0.2 never: 0   while graduate students and faculty clearly use the library website more than undergraduates (p<.0001), the evidence is weak that faculty members use the website more than graduate students (p=0.11), or that upper level undergraduates use the website more than sophomores (p=0.27) (table 1). it is unknown if freshmen would be different, since they were not included in the survey, which was conducted in the fall. contrast statements were used to test the differences between the groups.     table 1. estimate of average library website visits, by group, from 2012 survey   mean visits per month faculty 6.4 grad and professional students 5.7 second year undergrads 1.4 upper division undergrads 2.0 p & s 1.5     since graduate students are more frequent users of the library website than undergraduates (by self-report), the declining graduate to undergraduate enrollment ratio may be contributing to declining visits per person. the ratio is also seasonal, with peaks in the summer when undergraduate enrollment is much smaller (figure 4).   seasonal differences   seasonality in the data needs to be accounted for, either by eliminating it by seasonal differencing, or by including other variables, such as indicator variables for months. in this analysis, all variables were seasonally differenced: for each value, the value from twelve months before was subtracted. the resulting estimates from the model include an estimate of the mean yearly change, after controlling for each of the included explanatory variables.   regression with arma errors   ordinary regression applied to time series data presents problems, as residuals from the model are often correlated (a value at one point in time is likely to be similar to its neighbor), thus violating the assumption of independent residuals needed for regression analysis.   if the residuals are correlated, then some available information won’t be used in the model, resulting in inaccurate estimates of coefficients (granger & newbold, 1986; hyndman & athanasopoulos, 2014). other problems include invalid statistical tests, as the residual variance is estimated incorrectly, and misleading correlations, or spurious regressions (pankratz, 1991, p. 12, or for absurd examples see the website spurious correlations).   autocorrelation in the residuals can be removed by using regression with arma errors (called dynamic regression by pankratz, 1991, also called transfer function or arimax). the residuals are modelled as a time series with terms referencing past history of the series, leaving white noise, independent residuals. these terms can be autoregressive (ar), which are portions of past values, and/or moving average (ma), which are portions of past random shocks.       figure 4 ratio of graduate and professional student to undergraduate headcount enrollment.     autoregressive terms for lags 1 and 12, with a multiplicative term for lag 13 were added, but there were still significant autocorrelations at lags 3 and 6 (second row of figure 5). this suggests a trading day effect (pankratz, 1991, pp. 115-118).   a trading day effect (a count of the number of weekdays in each month) is included to remove remaining autocorrelation in the residuals. weekdays have more website visits than weekends. the number of weekdays can vary. for example, a month might have four or five wednesdays in different years. adding the weekdays term lowered the akaike information criterion (aic) from -21 to -42 and the autocorrelation function (acf) and partial autocorrelation function (pacf) display no significant autocorrelation. in each succeeding model, autocorrelation is removed from the residuals, the model fits the observed values more closely, and the confidence interval gets smaller (gray bands) (figure 5).   in figure 5, the top row is a regression model of visits per headcount, seasonally differenced, with two independent variables, graduate to undergraduate enrollment ratio and visits starting on carver pages (both seasonally differenced). the residuals from the model are autocorrelated, as seen by the serial grouping of observations above or below the predicted line, and as shown on the acf plot on the right. the second row adds autoregressive terms for lags 1 and 12, with a multiplicative effect for lag 13. this removes the autocorrelation in lags 1 and 2, but lags 3 and 6 in the acf indicate a trading day effect. in the third row, another variable for number of weekdays per month was added, leaving no significant autocorrelation in the residuals.   the final model is:   y’t = µ + β1ratio’t + β2gwc’t + β3weekday’s3,t + n’t, and n’t=φ1y’t-1 + φ12y’t-12 φ1φ12y’t-13 + et where y’t = visits per headcount at month t (seasonally differenced), µ is the mean change adjusted for other factors in model, et are uncorrelated residuals, and all independent variables are seasonally differenced.     figure 5 autoregressive terms at lags 1 and 12 (middle row) removes most of the residual autocorrelation. adding a variable for the number of weekdays in a month leaves uncorrelated residuals (bottom row).     to provide a comparison, in case enrollment sizes are not the true driver of higher website visits during the semester, an indicator variable for fall and spring semesters, replacing the graduate to undergraduate enrollment ratio, was included in an alternative model. for the months of january, may, august, and december, which were partially included in the semester, an average indicator was calculated.   to get a new measure every year of the adjusted mean change, and to see how the effect of the explanatory variable changes over time, the analysis was repeated for rolling time windows of equal length. the results are a smoothed and more easily interpretable metric that can identify correlations that change over time (zivot & wang, 2006). a four-year (academic year) rolling window with 48 observations was chosen. seasonal differencing leaves 36 observations available to estimate the model, resulting in a three-year average of differences.   another analysis estimated how often students from different groups visit the website, on average, after past history, the effect of the other groups, and other factors are taken into account. this marginal effect is estimated by the coefficient of the variable in the regression model.   the data were not seasonally differenced. instead, a number of other variables besides enrollment are included to account for seasonality: the number of george washington carver entry page visits, a count of library closed days for the two weekdays of thanksgiving break and weekdays closed during winter break, and a count of weekdays minus the other holidays and break days per month. additionally, december 24 and the days between christmas and new year’s, if the library was not closed, are counted as holidays. the model includes autoregressive terms for lags 1 and 12. again, this model was compared to a model containing an indicator variable for fall and spring semesters, rather than enrollment variables.   results   effect sizes   in time series models, most of the month-to-month variation is explained by past history (top row of figure 6). pierce (1979) developed a regression r2 that measures how much of the remainder of the variation (the innovation variance) is explained by the independent variables. the weekdays adjustment is excluded from the regression r2 in this analysis.   the graduate/undergraduate ratio explains 23% of the innovation variance (the variance that is not explained by past history and the weekdays adjustment). the adjusted mean estimate is  -0.17. the carver visits explain 34% of the innovation variance, with adjusted mean of -0.19. both variables together explain 58% of the innovation variance (figure 6).   the adjusted mean with both variables is -0.14, with a 95% confidence interval of -0.24 to -0.03, compared to the raw mean of -0.23. the magnitude of the adjusted mean decrease is 61% of the magnitude of the raw decrease.   the alternate model with the averaged fall/spring semester indicator variable fit slightly worse than the final model, with an aic of -37, compared to -42, and a regression r2 of 55%.   redesign effect   after a website redesign in august 2010, there appears to be a drop in both visits and visits per headcount (figures 2 and 3). fitting a model with a dummy variable set to 0 before that date and 1 afterwards, there is an effect of -0.20 (p=0.07). adding first order and seasonal autoregressive terms reduces the effect to a nonsignificant -0.07 (p=0.68). including the other variables (graduate/undergraduate ratio, george washington carver visits and weekdays) changes the effect to 0.04 (p=0.77).   rolling windows estimates of visits per headcount adjusted mean change   for the period ending in 2012, the adjusted mean change is -0.14, for the period ending in 2013, the adjusted mean change is -0.10, and for 2014, -0.16 (figure 7.)   the relative importance of the two independent variables changes over time, with the graduate/undergraduate ratio becoming more important and the carver pages visits becoming less important, shown by the regression r2. the regression r2 for the complete model increases over time, from 42% in the period ending in 2012 to 59% in the period ending in 2014 (figure 8).   the left panel of figure 8 shows the observed values and model fitted for each window, while the right panel shows the estimates of the regression coefficients. the bottom panel shows the regression r2 for the rolling windows for the model containing both of the variables of interest and for models containing one of the variables of interest. regression r2 is the percent of innovation variance (variance not explained by past history).     figure 6 comparison of effect size, using adjusted mean, aic, and regression r2: the first two rows show results from baseline models (autoregressive terms only and autoregressive adjusted for weekdays). the third row includes the graduate/undergraduate ratio, with a regression r2 of 23%; the fourth row includes the carver visits (but not the ratio), with a regression r2 of 34%; the last row includes both independent variables, with a regression r2 of 58%. in other words, including both the graduate/undergraduate ratio and the carver visits explains 58% of the variance in website visits per headcount that is not explained by past history of the series and a weekdays adjustment.     figure 7 seasonal differences of website visits per headcount, compared to unadjusted 3-year rolling averages, and rolling averages adjusted for graduate/undergraduate enrollment ratio and visits starting on george washington carver pages.     estimate of undergraduate and graduate student marginal effects on library website visits   freshmen enrollment and other undergraduate enrollment follow different patterns (figure 9). freshmen have a lower enrollment during the spring, in contrast to the other undergraduates. graduate student enrollment (figure 4) exhibit less seasonal change and less trend than undergraduate enrollment, making it more difficult to estimate the effect with precision. faculty headcount is flat, so faculty effect can’t be estimated separately.   website visits from faculty, staff, and all others are included in the estimate of 1556 visits attributed to each additional weekday. visits due to visits starting on the carver pages were restricted to be 1. each library closed day had an effect of -2369 fewer visits, after accounting for other factors.   an average of 5.4 visits per month is attributed to each additional graduate student, after all other variables are taken into account. similarly, 2.0 visits per month are attributed to freshmen, and 2.5 visits per month are attributed to other undergraduates. the 95% confidence intervals are quite large and overlapping (figure 10).   this model had a lower aic of 1520 compared to 1553 for a model containing an indicator variable for fall and spring semesters instead of enrollment variables, indicating a better fit.   in the rolling windows analysis, the variation in the marginal visits attributed to weekdays minus holidays varies widely, from 1015 in the first rolling window ending in august 2012, to 1723 and 1571 in the next two. the decrease attributed to closed days ranges from -1836 in the first period to -2444 in the third period.   while the point estimates for graduate students show an increase from 3.8 to 5.4, the broad and nearly completely overlapping confidence intervals make it difficult to say whether there was actually an increase. the same is true for the increase for freshmen and the decrease for other undergraduates, although the confidence intervals for the other undergraduates are much narrower (figure 11).     figure 8 rolling windows analysis using three years of seasonal differences for each window.     the decline in library website usage over this study period is small. students and faculty may be using resources the library has paid for but not accessing them through the library website. perhaps fewer individuals are using the library website but the individuals who are using it are more intensive users, as seen in the oclc study (de rosa, et al., 2011).     figure 9 undergraduate headcount enrollment, freshmen, and all others.     figure 10 coefficients of student group variables estimates of marginal effect of adding one student on number of website visits, for each student group, with 95% confidence limits.     figure 11 rolling windows estimates of marginal effects of adding one student from each student group on number of website visits (top row) and marginal effects of each additional closed day during thanksgiving and winter breaks, and each additional weekday that is not a holiday per month, with 95% confidence bars.     there is some support for the idea that fewer individuals are using the library website but they are more intensive users. the number of unique visitors per headcount decreased, but the number of visits per unique visitor (as defined by ip address) increased until the 2011-2012 academic year, then plateaued. at the beginning of the period, unique visitors per headcount is 1.5 or greater, perhaps partly attributable to non-affiliate use for george washington carver pages. in the last two years of the study period, it stays mostly between 1.0 and 1.5, with a dip below 1.0 during fall 2013 (figure 12).   there are caveats with visits and unique visitor statistics – ip address is used to define website visits and visitors but there isn’t a one-to-one relationship between ip addresses and individuals. there are also people who are not included in the headcount who may use the library website. some of these individuals may not be affiliated with the university.   figure 12 the number of unique visitors per total headcount is decreasing but the number of library website visits per unique visitor has increased.     figure 13 page views of the articles indexes & databases main page as a percentage of visits. aid/bb reference links marks implementation date of a redesigned articles indexes & databases page and of blackboard mylibrary tab.     further analysis   although a redesign in 2010 did not result in any change in visitors per headcount, an emphasis on the ask us! feature did result in an increase in the chat and email service usage. in the 2014 redesign, a new link to interlibrary loan and document delivery (ill/dd) was placed on the home page. ill/dd data could be analyzed for an effect on number of ill/dd requests and number of patrons who used the service.   intensive library website visitors may make more use of certain features of the website, such as the article indexes and databases page, which shows an increase in page views as a percentage of visits to the library website in the last two years (figure 13). page view statistics for article indexes & databases pages could be analyzed in conjunction with both database and journal usage data and website design changes. two events happened in january 2013: a change in the design of the article indexes & databases page, and the implementation of a “my library” tab with a link to this page in the campus blackboard course management system. the recent trend upwards, if it is not a short-lived fluctuation, could be due to either or both of these changes, and/or perhaps to recent visitors being more intensive users of the website.   even though there is a general decline in interest in george washington carver, improved search engine optimization for this and other digital collections could continue to bring both affiliated and unaffiliated people to the website.   effect on number of website visits by student groups: graduate and professional students, freshmen, and other undergraduates   the effect of adding autoregressive terms to the model, rather than using an ordinary regression, was quite marked. a model with all of the variables except the autoregressive terms resulted in parameter estimates of 12.8 for graduate and professional students, 0.9 for undergraduates (class 2, 3, and 4) and 2.6 for freshmen, illustrating the need to remove correlation from regression residuals. using regression with arma errors allows making inferences about demographic groups, even without having data that is directly tied to demographics.   these analyses assume that students, both graduate and undergraduate, visit the library website. there is self-reported evidence of that but no direct evidence. there are clearly more website visits during the fall and spring semesters when there are also many more students, but behaviour by other possible visitors, including faculty and staff, and teachers and students from the public schools, could change then as well. models including enrollment variables fit slightly better than models including a fall/spring semester indicator variable instead of the enrollment variables.   conclusion   trend in library website visits   time series analysis (regression with arma errors) was conducted to evaluate trend in library website visits, while accounting for factors such as increased enrollment, decreasing graduate to undergraduate enrollment ratios, and decreasing visits to a popular george washington carver digital collection.   the sample mean change in monthly visits per headcount over the study period (february 2008 to august 2014) is -0.23. the mean change adjusted for graduate to undergraduate ratio and george washington carver visits is -0.14. together these two factors explain 58% of the variance of the seasonal differences in visits per headcount that is not explained by past history of a time series. rolling windows analysis shows the effect of the undergraduate/graduate ratio increasing over time, while the effect of the george washington carver visits decreases.   a decrease in visits per headcount coinciding with a design change in 2010 was found to be nonsignificant after including autoregressive terms. the decrease also coincided with a drop in george washington carver pages visits. according to google trends, searches for george washington carver have been decreasing worldwide.   comparison of usage estimates by student group from survey data and from web log data   regression with arma errors was used to estimate marginal effects on library website visits by three student groups. each additional freshman enrolled marginally increased the number of website visits per month by 2, after taking into account george washington carver visits, the number of other undergraduates, the number of graduate students, the number of weekdays minus holidays per month, and library closed days at thanksgiving and winter break. similarly, the regression analysis attributes 2.5 visits to each additional level 2, 3, and 4 undergraduate, and 5.4 visits for each additional graduate student. the confidence intervals for freshmen and graduate students are quite wide. the point estimates for graduates’ and other undergraduates’ marginal usage match closely (within confidence intervals and within one visit) with estimates taken from survey results in 2012. these estimates were made without demographic data tied to individual records in the transaction logs.   library websites are a gateway to library resources, services, contact information, and events. changes in the website may affect awareness and usage of these resources and services. this analysis can be extended to evaluate the impact of changes on usage and understand the effect of background data such as enrollment changes and other events. the methods can be applied to any time series data libraries have, such as electronic resource usage, attendance, or number of reference transactions.   acknowledgements   many thanks to karen lawson and kristine stacy-bates for reading earlier versions of the manuscript and contributing greatly to its improvement.   references   ahiakwo, o. n. (1988). forecasting techniques and library circulation operations: implications for management. library and information science research, 10(2), 195-210.   allen, l., baker, n., wilson, j., creamer, k., & consiglio, d. (2013). analyzing the miso data: broader perspectives on library and computing trends. evidence based library and information practice, 8(2), 129-138. retrieved from https://ejournals.library.ualberta.ca/index.php/eblip   asunka, s., chae, h. s., hughes, b., & natriello, g. (2009). understanding academic information seeking habits through analysis of web server log files: the case of the teachers college library website. the journal of academic librarianship, 35(1), 33-45. http://dx.doi.org/10.1016/j.acalib.2008.10.019   box, g. e. p., jenkins, g. m., & reinsel, g. c. (2008). time series analysis: forecasting and control (4th ed.). hoboken, nj: john wiley & sons.   brooks, t. a. (1984a). naive vs. sophisticated methods of forecasting public library circulations. library & information science research, 6(2), 205-214.   brooks, t. a. (1984b). using time-series regression to predict academic library circulations. college & research libraries, 45(6), 501-505.   cochrane, l. s. (2007). if the academic library ceased to exist, would we have to invent it? educause review, 42(1), 6-7. retrieved from  http://er.educause.edu/~/media/files/article-downloads/erm0714.pdf   cohen, l. b. (2003). a two-tiered model for analyzing library website usage statistics, part 1: web server logs. portal: libraries and the academy, 3(2), 315-326. http://dx.doi.org/10.1353/pla.2003.0028   connaway, l. s., dickey, t. j., & radford, m. l. (2011). “if it is too inconvenient i'm not going after it”: convenience as a critical factor in information-seeking behaviors. library & information science research, 33(3), 179-190. http://dx.doi.org/doi:10.1016/j.lisr.2010.12.002   de rosa, c., cantrell, j., carlson, m., gallagher, p., hawk, j., sturtz, c. (2011). perceptions of libraries, 2010: context and community. retrieved from oclc website: http://www.oclc.org/reports/2010perceptions.en.html   education advisory board. (2011). redefining the academic library: managing the migration to digital information services. retrieved from https://www.eab.com/research-and-insights/academic-affairs-forum/studies/2011/redefining-the-academic-library   goddard, l. (2007). getting to the source: a survey of quantitative data sources available to the everyday librarian: part i: web server log analysis. evidence based library and information practice, 2(1), 48-67. retrieved from https://ejournals.library.ualberta.ca/index.php/eblip   google. (2014). google trends: george washington carver. retrieved from http://www.google.com/trends/explore#q=george%20washington%20carver   granger, c. w. j., & newbold, p. (1977). forecasting economic time series. new york: academic press.   hyndman, r. j., & athanasopoulos, g. (2014). forecasting: principles and practice. heathmont, vic: otexts. retrieved from https://www.otexts.org/fpp   jansen, b. j. (2006). search log analysis: what it is, what's been done, how to do it. library & information science research, 28(3), 407-432. http://dx.doi.org/doi:10.1016/j.lisr.2006.06.005   ke, h. r., kwakkelaar, r., tai, y. m., & chen, l. c. (2002). exploring behavior of e-journal users in science and technology: transaction log analysis of elsevier's sciencedirect onsite in taiwan. library & information science research, 24(3), 265-291. http://dx.doi.org/doi:10.1016/s0740-8188(02)00126-3   liu, s. (2008). engaging users: the future of academic library web sites. college & research libraries, 69(1), 6-27. http://dx.doi.org/10.5860/crl.69.1.6   marek, k. (2011). using web analytics in the library: a library technology report. chicago, il: american library association.   nackerud, s., fransen, j., peterson, k., & mastel, k. (2013). analyzing demographics: assessing library use across the institution. portal: libraries and the academy, 13(2), 131-145. http://dx.doi.org/doi:10.1353/pla.2013.0017   naylor, m., & walsh, k. (1994). a time-series model for academic library data using intervention analysis. library & information science research, 16(4), 299-314. http://dx.doi.org/doi:10.1016/0740-8188(94)90054-x   pankratz, a. (1991). forecasting with dynamic regression models. new york: wiley. park, m., & lee, t. s. (2013). understanding science and technology information users through transaction log analysis. library hi tech, 31(1), 123-140. http://dx.doi.org/10.1108/07378831311303976   pierce, d. a. (1979). r 2 measures for time series. journal of the american statistical association, 74(368), 901-910.   tenopir, c. (2013). building evidence of the value and impact of library and information services: methods, metrics and roi. evidence based library and information practice, 8(2), 270-274. retrieved from https://ejournals.library.ualberta.ca/index.php/eblip   vigen, t. spurious correlations. retrieved 19 feb 2015 from http://www.tylervigen.com/spurious-correlations   wood, p. a., & walther, j. h. (2000). the future of academic libraries: changing formats and changing delivery. the bottom line, 13(4), 173-182. http://dx.doi.org/10.1108/08880450010355904   zivot, e., & wang, j. (2007). modeling financial time series with s-plus® (2nd ed.). new york: springer.   conference paper   after the data: taking action on climatequal® results   elizabeth mengel associate director, scholarly resources and special collections the sheridan libraries, johns hopkins university baltimore, maryland, united states of america email: emengel@jhu.edu   judith smith liaison librarian university of michigan ann arbor, michigan, united states of america email: judsmith@umich.edu   elizabeth uzelac law clerk, united states district court for the northern district of illinois  chicago, illinois, united states of america email: euzelac@gmail.com      2013 mengel, smith, and uzelac. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – this paper discusses the actions taken by the staff development and training (sd&t) team at the sheridan libraries and johns hopkins university museums in response to results of a climatequal survey.   methods – the team administered the climatequal organizational climate and diversity assessment in march 2009 to the 150 staff members of the museums and libraries, and 80% responded. to get at the root of some of the results, the team conducted 23 focus group sessions over the course of two months. in each 90-minute session, 8 open-ended questions were used to probe the staff’s thoughts on the survey results and elicit concrete suggestions for moving forward. participants were asked to discuss their personal experiences with six areas of concern: procedural justice, distributive justice, structural facilitation of teamwork, psychological safety, communication, and leadership. one year after the original climatequal survey, the team administered a one-question follow-up survey.   results – the team analyzed and coded the notes taken during the focus group sessions and developed three discrete written summaries for each session: a brief summary of themes, a list of specific actionable suggestions, and a general description of specific scenarios aired in the sessions. from these analyses, the team developed two types of recommendations: quick tactical actions and long-term strategic recommendations. strategic recommendations were developed in three main areas: fostering a sense of global ownership of organizational issues, improving organizational communication, and improving leadership and facilitation of teamwork. with these recommendations, the team charged managers to take broad ownership of a plan for individual actions. the results of the one-year follow-up survey were mixed. staff perceived positive change in communication, but indicated that the areas of procedural and distributive justice, psychological safety, and transparency in decision making continued to require improvement.   conclusion – the work of the sd&t team continues, and it is hoped that climatequal will serve as the foundation for future assessments of organizational health.   introduction   in 2008, the sheridan libraries and johns hopkins university (jhu) museums staff development and training team (sd&t) found itself wrestling with ways to facilitate constructive organizational change. the team was at a regrouping point in terms of its strategic direction. it sought to go beyond anecdotal evidence in identifying ways to support library and museum staff. sd&t, a small committee charged with supporting the staff of the libraries and museums with training and organizational development matters, did not want to make decisions based on unfounded assumptions in the course of implementing change in the organization. to ensure its programming was rooted in the actual needs of the organization, the team administered the climatequal organizational climate and diversity assessment (ocda) in 2009 to its 150 staff members. climatequal, a confidential, third party organizational health and diversity survey, is designed to assess the shared culture of an organization. it gathers data and assesses overall staff perceptions of the organizational climate of a library. developed at the university of maryland libraries in 2000, climatequal is now supported in partnership between the university of maryland and the association of research libraries (arl).   little did the team realize that participating in climatequal would begin a year-long odyssey of building organizational trust and championing change! the official climatequal report turned out to be merely the beginning, and the team discovered that getting results is by far the easiest part of the process.   this paper discusses the organization’s process of responding to data measuring organizational climate. after outlining the survey preparation and administration processes, the paper describes the climatequal results report. the paper then discusses the issues inherent in acting upon the report, and the processes taken to respond to these issues and act to improve the organization. it explores and explains the steps that came next in responding to data: staff focus groups, in-depth interviews with library leadership, qualitative and quantitative data analysis, re-evaluating the meaning of communicating well, finding ways to get staff and management to hear one another, and developing short and long range recommendations. the authors hope that explicating the process will aid other organizations in taking effective action in response to their own climatequal data.   survey preparation and administration   the libraries and museums ran the climatequal survey march 2-23, 2009. while many organizational culture assessments exist, the team chose climatequal because of its library-based context and the support offered through a community of peers. there was an 80% return rate to the survey, a rate in line with the high return rates at other climatequal institutions. the high return rate was a byproduct of the significant preparation the sd&t team led to prepare the organization and foster a sense of both security and ownership among staff.   prior to administering the survey, the sd&t team spent a considerable amount of time planning the rollout and grappling with organizational questions. how would we communicate the survey to staff? how would we motivate them to take it? how do we manage expectations about what can be done with the results to create change? how do we get staff to understand change is a long process and most organizational problems do not have quick fixes? the team realized that a critical success factor of the survey administration would be the existence of trust among staff in the confidential nature of the survey. as even good changes can be disruptive, part of the role as potential change agents was to find ways to effectively and comfortably discuss organizational issues without losing the trust or participation of people along the way. the team wanted to emphasize the reasons for doing the survey: assessing the libraries and museums’ organizational health, making people part of the process of improving the organization, and stimulating thinking about everyone’s role in broader organizational improvements.   three weeks prior to the administration of the survey, sd&t team members held meetings with each department to explain the survey’s importance to the organization and outline how organizational demographics were mapped to the climatequal demographic categories. the team emphasized the survey’s safeguards to protect someone’s identity. for example, if there was a low response rate in any particular department, these responses would be rolled up into the next larger category. in these meetings, the team told staff how the raw data would be handled (i.e., no one in the institution would be able to see it or manipulate it to determine who said what), how the incentive would be administered (the team chose to have arl administer the incentive so no one at jhu would know who submitted a survey or who won the incentive), and how the results would be distributed (the full report of everything received would be sent to staff; nothing would be held back).   these meetings were the first step in building trust with staff. once the sd&t team established this trust, it was vigilant throughout the rest of the process to make sure we did nothing to break these bonds. staff members showed an interest in change and looked to the team to facilitate changes many had hoped would happen. feeling a strong sense of responsibility to the organization to do this process well, the team focused on following up on promises and finding ways to keep staff informed along the way throughout the survey administration period and beyond.   survey results   the sd&t team received the climatequal results report several weeks after the survey closed. the results are divided into four sections: demographics, organizational climate scales, analysis of variables, and comments.   demographics   the demographics section provides breakdowns by library team, position, full or part time, librarian vs. non-librarian, age, ethnicity, religion, sexual orientation, gender, length of service at jhu, and length of overall library service.   organizational climate scales   after the demographics, the results provide tables with the mean, standard deviation, and standard error for the organizational climate scales. the scales include measures for organizational justice, leadership climate, diversity, climate for continual learning, climate for teamwork, and more. definitions of these terms can be found at: http://www.climatequal.org/concepts/core-scales/index.shtml. because the results contain a considerable amount of statistical data, one is also provided definitions and some interpretation of mean, standard deviation, and standard error to help non-statisticians understand the significance of those measures.   analysis of variables   following the tables on organizational climate, the results include another set of statistical tables on the demographic differences of the climate scales. these tables were perhaps the most confusing to people who were unfamiliar with reading and interpreting statistics especially analysis of variance (anova) tables. in the simplest of terms, the analysis of variance (anova) compares the means for different groups and asks whether such differences are systematic and likely to be replicated if we had another sample of respondents. if the probability is less than .05 or .01, we typically conclude that there is a systematic difference across the different groups and the difference is likely to be replicated across different samples. even though anova tests do not indicate importance, they do indicate how likely it is to see such differences replicated and can serve as an initial stepping stone for evaluating group differences for the climate scales in climatequal.   comments   after the anova tables were the comments section. twenty-one percent of staff members supplied some type of comment. those comments ranged from in-depth, serious discussions about issues in the library to comments on the structure and phrasing of the survey itself. based on comments the team heard once the results were released, we suspect the comments section was the most popular with staff and where they spent the majority of their time.   challenges   while the results were full of important data, they lacked a roadmap for our next steps. to develop a plan, the sd&t team spent focused, intensive time understanding the data, analyzing the comments, and posing questions to ourselves. what comes next? what actions are needed? one of the challenges the team faced, as is common when attempting to use any survey’s results, was determining what is actionable based on the data.   another challenge was to determine the best way to convey the results to staff. the team had promised to provide all raw data to staff members, but there needed to be some kind of explanation to help guide them through the results, especially the statistical information. the sd&t team decided to craft an executive summary to the report including indications of which pages staff should spend their time investigating. the team also included information about the team’s next steps to continue the dedication to a transparent communication process.   choosing focus issues   the team’s first step was to isolate specific issues on which it felt the organization should concentrate. because the team could not address every possible issue in the report at once, it was important to prioritize. it used the analysis in the report to identify focus areas: 1) where a high percentage of staff gave positive responses; and, 2) where a high percentage of staff gave negative areas. these areas became, in turn, areas highlighted for celebration and targeted for constructive change.   the organization scored well in the climatequal areas of benefits of teamwork (93% of respondents giving positive responses), task engagement (87%), and both valuing diversity (84%) and climate for demographic diversity (91%). in addition, the organization had welcome low scores is areas such as organizational withdrawal (i.e., desire to leave the organization, 13% of respondents answering affirmatively) and work unit conflict at both the interpersonal and task level (18% and 24% respectively). finally, the comments revealed that staff had very positive things to say about their colleagues and the dedication and work ethic exhibited daily in the organization. the team took these to be very positive signs.   similar to other organizations, there were areas where the team needed to focus some attention. the team identified a small number of indicators for which fewer than 60% of our organization’s staff gave a positive response or where the mean score was below 5.0 on a 7-point scale. scores for distributive justice (25%), procedural justice (55%), structural facilitation of teamwork (48%), and climate for psychological safety (66%, but a mean score of 4.94) all pointed to areas where organizationally we needed improvement.   to incorporate the free text comments into a plan of action, the team coded them to analyze any patterns or trends. rather than use a grounded theory approach in analyzing the comments, the team used the climatequal core concept terms as provisional categories to which it mapped clauses of each comment. first, we reduced the data in the concepts by labeling them with climatequal terms relevant to what each comment conveyed (miles & humberman, 1994). this process focused, simplified, and abstracted the comments, enabling us to work with an organized, compressed display of what they communicated (miles & humberman, 1994). after analyzing the comments in relation to the climatequal core concept terms, we were able to conclude that the comments mirrored the focus issues identified above (distributive justice, procedural justice, structural facilitation of teamwork, and psychological safety). in addition to these areas, the comments also showed a pattern of concern surrounding leadership and communication. coding the comments enabled us to explicitly connect the qualitative data available to us to the quantified conclusions in the climatequal report. moving forward, we were then able to use all of the patterns in our results to inform an action plan.   taking data-driven action   the climatequal report helped identify the most pressing organizational issues, allowing the sd&t team to avoid guessing at underlying organizational strengths and weaknesses when creating an action plan. initial internal response to the report, though, involved far more questions than comprehension when it came to taking action. why do people feel psychologically unsafe? how does our organization aid or impede teamwork? what does distributive justice mean at jhu? how do you reconcile positive scores on the benefits of teamwork with a lack of structural facilitation of team work? to develop specific next steps, the sd&t team, library leadership, and managers needed to better understand the specific landscapes of target issues. the sd&t team realized that it needed to dig deeper and go beyond the climatequal results. after many discussions about the best method for delving further, the team decided to engage in in-depth focus groups with staff to contextualize focus issues in the organizational landscape.    contextualizing issues through focus groups   to get to the root of some of the climatequal results, the team conducted focus groups with each department in the organization. members of the sd&t team paired up, one person assuming a facilitator role and the other assigned to take detailed notes. the sd&t pairs scheduled ninety-minute sessions with each department, without their manager, as well as one session each for managers, supervisors, the staff development and training committee, and those unable to attend with their department. in all, the team held 23 focus groups over the course of 2 months.   in each session, the team used structured conversations to probe the staff’s thoughts on the survey results and ask for concrete suggestions for moving forward. the facilitator opened each focus group by explaining that individual comments would be kept confidential, and asked that each attendee similarly refrain from sharing their colleagues’ comments outside the session. the facilitator also tried to manage expectations about the focus group and its purpose: the team was there to hear more, and gather ideas to inform actions, but not every idea could necessarily be implemented. each attendee received a one-page handout listing the definitions of the six terms (procedural justice, distributive justice, psychological safety, facilitation of teamwork, communication, and leadership), and the questions that were to anchor the session’s conversation. the group briefly reviewed the overall results and six issues of concern from climatequal in order to set a common stage for those who may not have internalized the vocabulary and conclusions from the assessment, or were perhaps simply overwhelmed by the amount of statistical data provided.   the facilitator then asked for all suggestions, concerns, and responses in answer to the following eight questions, which anchored the conversation for the duration of the focus group sessions:   what did you think (were your impressions) of the survey itself (taking it, questions, timing, how it was rolled out)? what were your first impressions of the results? what are your expectations now, having read the results? what would you like to see done? after reading the comments in the survey, is there anything else you wish had been said? what makes you feel valued? (what types of rewards, recognitions, processes, or other factors?) how do you feel the organization might better foster or facilitate teamwork? of the issues we identified earlier (procedural justice, distributive justice, structural facilitation of teamwork, psychological safety, communication, leadership), which do you think needs to be addressed first? what changes in the organization would you like to see in the organization when we repeat the survey?   the team chose to create open-ended questions to better encourage staff to explore and share their thoughts on the survey and potential follow-up actions. the eight questions were designed to progress first from helping participants remember the assessment questionnaire, to exploring their personal experiences with the six issues of concern – to discussing concrete ideas for action. facilitators inhabited a neutral questioning role, and refrained from agreeing or disagreeing with any statements made. to help guide attendees from venting concerns to making tangible suggestions, facilitators used a series of follow-up prompts, including “if you were put in charge of fixing that issue, what would be your first step?” the team wanted to ensure that staff understood that they not only had the ability to present ideas, but that it is their responsibility to be part of the change process. this was a first step toward having staff take ownership of future organizational change.   by emphasizing confidentiality and constructive engagement, the team heard an enormous amount of information, even from individuals who had not previously felt comfortable actively engaging in global organizational issues. overall, participants showed remarkable candor. some staff members aired specific anecdotes of concern to them. many were responsive to the above prompts and the discussion that followed. as the focus groups progressed, the team heard directly from staff members who were appreciative of the chance to participate so directly in organizational change. in a few cases, staff members who had at first elected not to participate in a focus group changed their mind after hearing from their colleagues about their focus group experience. staff members passed the word along about the benefits of the focus group sessions, but fully complied with the confidential nature of the content discussed. through their actions, they helped reinforce and perpetuate the underlying trust that the sd&t team sought to engender.   developing action items   some concerns turned out to be common to almost every focus group. there was a clear overall message from the focus groups that it would be detrimental to staff morale if action was not taken in response to issues identified by the climatequal survey. to recommend concrete actions, the team evaluated and dissected the notes of every focus group. common themes for each question emerged alongside concrete recommendations and historical anecdotes. the sd&t team analyzed and coded the notes of each of the eight focus group questions. the team developed three discrete written summaries for each session: a brief summary of the themes and sentiment of focus groups participants, a list of specific actionable suggestions elicited during the sessions, and a general description of any specific scenarios that focus group participants aired in the sessions.   from these analyses, the team developed two major types of recommendations: quick tactical actions and long-term strategic recommendations. long-term strategic recommendations were developed in three main areas: fostering a sense of global ownership of our organizational issues, improving organizational communication, and improving leadership and facilitation of teamwork. many of the final quick and long-term recommendations, listed below, came from the data developed through focus group sessions.   those recommendations were:   ·         to address overall organizational climate and leadership skills we need to foster a sense of ownership of organizational issues: o    develop leadership skills on all levels of the organization. o    facilitate conversation across the organization about leadership ·         to address organizational communication: o    develop and publish each of the following, in series: a) a complete organizational chart, b) a map of organizational workflows, and c) a matrix of how decisions are made. o    charge management team with designing an explicit set of managerial communication principles and hold each other accountable to those principles. o    charge a cross functional, cross departmental working group with developing a set of communication principles to use across the organization. ·         to address both leadership and facilitation of teamwork practices: o    bring in talent management and organizational development (an internal jhu unit which provides a suite of human resources, organizational development, and talent management services) to assist in developing the management team into a high performing team. o    charge the executive committee and management team with defining delegated authority and work with talent management and organizational development to move toward organizational practices that empower teams and remove ambiguity about authority in team related issues. o    develop institutional teamwork checklists that address issues such as participation, accountability, roles and responsibilities, team communication expectations, and annual reporting practices. o    develop sheridan libraries and johns hopkins university museums 101: have the organization collaboratively and openly design its own cross-training program.   the team worked to articulate the scope of what to address when specific solutions were trickier to find or outside the scope of the team’s expertise. for instance, executive and managerial level staff members were charged as a group with defining “delegated authority” to move toward practices that empower and remove ambiguity about authority in team-related situations. additionally, managers were charged with designing an explicit set of managerial communication principles and holding each other accountable to those principles. the principles included prompt sharing of information, structuring decision making around a process based on the strategic plan, logic and data, and endeavoring to operate by consensus.   with these recommendations, the team charged managers to take broad ownership of a plan for individual actions. in cases where an issue was entwined with the daily work of managers, it was necessary to define overall expectations directed toward all managers. the team shifted into a role as a source of program-wide momentum and reporting, and managers were expected to engage with the full set of recommendations. managers were charged with employing organizational-level thinking and were encouraged to make direct ties between the recommendations and their program areas. managers subsequently articulated the first set of actions that would be undertaken, complete with timeline and a point person.   engaging executive leadership   after the recommendations were presented to the management team, the sd&t team felt it was important to discuss them in more depth with the executive leadership. the team held one-hour individual sessions with the dean and each of the other members of the executive council. to get the conversations started the following questions were emailed prior to the discussion.   based on the climatequal focus group recommendations, which of the recommendations really resonates with you? for us to be in a position to most efficiently attain our strategic goals and achieve success what would our library organization look like? or how would a successful organization differ from our organization today? as an executive level leader, how do you suggest starting to address some of these issues?   executive leadership, like other staff members, were open and honest about their perceptions and were willing to give their insights. they were able to be honest because they also knew that their comments and observations would be kept confidential, and would only go towards helping the sd&t team develop a viable action plan.   maintaining momentum   after the baton was officially passed to the management team, the staff development and training team’s role changed to one of maintaining momentum. there were still many ways the team needed to continue the change process through follow-up actions. based on comments in the survey and focus group sessions, the team discovered that staff members have a long organizational memory – especially for projects that once started with fanfare and were then not spoken of again. although the team was not directly responsible for many of the action items, it needed to track the identified actions holistically and ensure that tasks were completed by the management team and communicated to all staff. the actions themselves and the continuous communication helped maintain the solid level of trust developed. staff members looked to the team to take action, so any perception of lack of action would have left staff feeling that their confidences and trust was betrayed.   the sd&t team facilitated follow-up in multiple ways. one way was to have the management team report on progress at the libraries’ and museums’ staff exchanges (i.e., all staff meetings). since perceptions of the staff from the climatequal survey results were that management in the library were not responsive to staff needs, it was especially important to have those responsible report and be visible to staff members. the task did not even need to be complete at the time of reporting out – there just needed to be a noticeable effort to keep staff members apprised of the project’s status.   another initiative focused on better communication across the organization. using the management teams’ communication principles as the basis, the sd&t team repurposed them to be appropriate for all staff members. the communication principles focused on the libraries and museums’ values, including integrity and openness, innovation and constructive engagement, and stewardship and trust. the sd&t members presented these principles at a staff exchange, using examples from their own work to illustrate main points at an all-staff, open meeting. staff members eagerly participated in the staff exchange and responded positively, indicating that they not only enjoyed the session but felt that they learned a great deal. one staff member mentioned how grateful he was that these issues were being examined. overall, the session bolstered the level of trust that the team had built with the staff and illustrated how we were working towards common goals.   in addition, the team became deeply involved in developing a new performance appraisal program. climatequal indicated levels of dissatisfaction with distributive and procedural justice. a way to address this issue was through a new university led performance partnership program (i.e., performance appraisal system). highlights of this new system include: a single anniversary date, a much stronger focus on year-round coaching and development, and the creation of defined, measurable goals. this new performance appraisal system was a huge initiative in the team’s workload. it required the team to evaluate a new system, allay staff concerns, and create staff “buy in.” the sd&t team utilized the communication techniques from the climatequal rollout: communicating through multiple venues; communicating repeatedly; and meeting with every department to describe the rationale for the new system. overall, the team reinforced the idea to staff that the new system was a result of listening to their needs. we worked closely with the university’s talent management and organizational development department to train staff on the new system using a “train the trainer approach.”   one year after the original climatequal survey, the team administered a simple, one question follow up survey via zoomerang. the question asked was: “last year the issues below were identified from climatequal and the follow up focus groups as organizational issues that needed to be addressed. please reflect back on the past year. how do you think the sheridan libraries and johns hopkins university museums are doing on these issues at this point?” the issues identified were the six issues highlighted throughout this paper: distributive justice, procedural justice, psychological safety, facilitation of teamwork, communication, and leadership. the team also added a comment box to the survey. by administering this survey, the team wanted to better understand perceptions of organizational improvement one year after taking climatequal to help inform what items to work on next. we had a response rate of 44%, and the results were mixed. as anticipated, the perceived pace of change on issues as core as those raised by climatequal is gradual, yet staff expected faster results. the team recognized that none of these issues will be “fixed” without long term attention and willingness of the organization to change.   there were areas where staff believed there had been change, and areas that indicate a desire for more or faster improvement. areas where staff perceived improvement include communication, where 66% of survey respondents perceived positive change. there were also indications of areas in which we continue to need improvement, such as procedural and distributive justice. some comments indicated dissatisfaction with lack of change overall. survey results also indicated that there have not been significant changes in psychological safety and transparency in decision making – leading the team to note, yet again, that organizational change takes a significant amount of time and continuous effort.   overall lessons   several practical lessons follow the team’s experiences with climatequal and inform how it will handle current and future data and initiatives.   start with a strong team   when undertaking a large initiate such as climatequal, the importance of a proactive and dedicated team is crucial. the sd&t team is comprised of members who volunteered to be on the committee because of their avid interest in organizational development issues. without that interest and commitment to helping staff members succeed at their jobs, this type of assessment and follow-up would not succeed. given the issues raised by the climatequal survey, absolute discretion of each team member was critical.   communication   so much of what the team learned throughout the climatequal implementation, analysis, and follow up is the importance of a clear, proactive, and multi-pronged approach to communication. as in real estate where the motto is “location, location, location,” the team found that it consistently returned to “communication, communication, communication” as the foundation for everything needing to be accomplished. instead of making assumptions about staff members “hearing” the team’s message, the team started with the premise that “hearing” is challenging. the team focused on ways to have the message about the survey and its follow up activities simple and clear. people have different communication styles. the team was consistently reminded of this fact as it communicated aspects of climatequal. more often than not the team still had staff members ask us questions that in our minds, we addressed. sd&t found it critical to communicate along each step of the process in multiple ways and through multiple venues: e-mails, meetings with departments individually, postings on the wiki, and answering questions individually, or presenting at staff exchanges. by proactively communicating in many different ways, the team was able have people “hear” the message because the active communication built a solid level of trust. staff members felt that there was nothing being hidden from them, and they were receptive to the information given.   data needs context   whatever results you begin with will need to be interpreted and internalized for them to have meaning. the data from climatequal provide a starting point for analyzing institutional perceptions. however, the results do not provide the analysis that only you and your colleagues can provide through your institutional lens. more discussion is often needed, as the team discovered when it held focus groups. other organizations may find different ways to tease out important themes from their climatequal results, but our team found that having climatequal as the jumping off point for continuous discussion (not the end point) to be what propelled it forward and helped it to understand the libraries and museums strengths and areas to address.   determine the level of data desired through the survey   climatequal offers a range of granularity in its results data. prior to implementing your survey, consider the level of data that you are seeking. is it at the unit level or the broader departmental level? there are various costs associated with the results received, so it is important to determine your organizations needs ahead of time and think about the results you want in the long run. the team wanted to start with results based around very broad demographic categories at the departmental level, and because of that, there were some questions about how applicable the data was to a supervisor’s individual unit. however, because many units in the libraries and museums are very small (2-3 people) this would leave individuals’ responses overly exposed the broader organization.   create a clear process   an open and defined process laid out for staff helps answer the perpetual “what now?” questions that follow such an assessment. a group-oriented process can give staff a non-confrontational group voice to management. the team also found that even with setting up a clear process and communication, there were still many questions about what was actually being accomplished, and how quickly. staff were eager for change, and it required reporting out on expectations and continuous management of expectations about timeframe. long-term, deep change takes time, and this idea has to be restated often.   focus and tenacity is required to repair and build trust   through the course of this process, the team found that there can never be enough trust in an organization, and that it takes conscious efforts and tangible actions, such as getting “out there” and speaking to colleagues, usually face-to-face, to build or repair trust.   organization-level thinking is crucial   at all levels, but especially in leadership, a broad organizational outlook is crucial for intentional change and organizational health. to succeed in trying to assess and implement change, there needs to be a strong ”we” at all levels to move ideas forward. the team found, in conversations at all levels, that more often than not no one spoke of the organization as a whole. staff, including management, mentioned “their team,” “their staff,” or “their department.” the team found that the concept of “we” needs to be continuously emphasized in daily communications and in larger initiatives. this change in perspective takes time, but is vital for breaking down silos and fostering deep collaborations across units.   the work of the sd&t team continues, and we hope that climatequal will serve as the foundation for future assessments for organizational health. our plans in the future involve, not only assessments of the organization as a whole, but also evaluations of how we as the staff development and training team can continue to improve to meet organizational needs.     references miles, m. b., & humberman, a. m. (1994). qualitative data analysis. thousand oaks: sage. evidence based library and information practice using evidence in practice   streamlining workflows in the now: using evidence to select a dvd vendor for the university of alberta libraries   jessica thorlakson academic library intern university of alberta libraries edmonton, alberta, canada email: jthorlak@ualberta.ca   denise koufogiannakis collections and acquisitions coordinator university of alberta libraries edmonton, alberta, canada email: denise.koufogiannakis@ualberta.ca   sharon marshall manager, bibliographic services (retired) university of alberta libraries edmonton, alberta, canada email: sharon.marshall@ualberta.ca   received: 13 nov. 2014  accepted: 31 jan. 2015      2015 thorlakson, koufogiannakis, and marshall. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   setting   located in edmonton, alberta, canada, the university of alberta libraries (ual) serve a robust community of users made up of over 10,000 staff and over 39,000 students across 200 undergraduate and 170 graduate programs from 18 faculties at the university of alberta (university of alberta, 2013a; university of alberta, 2013b). programs of study are wide ranging, and serve all levels of study. the ual have 13 locations, with librarians and support staff at all locations involved with collections work. the acquisitions and bibliographic services units serve a central role within the libraries to oversee and streamline processes for functions related to the acquisition and discovery of library materials. each year, the ual normally hire paid academic library interns who are recent graduates (within the last two years) from an accredited library school program, and in the 2013-2014 academic year one of the interns was assigned to work with acquisitions and bibliographic services on a project related to the acquisition of video materials on dvd.   problem   with an influx of online video resources, many institutions are increasing their partnerships with video streaming providers such as films on demand, filmakers library online, the national film board of canada, and more, providing ubiquitous access to these resources. although the ual are implementing many video streaming services, there is still a sizable demand for individual physical dvds in cases where there are requests for specific, often difficult to locate video resources that are not available via streaming. the acquisition of video materials (generally in dvd format) at ual had never been fully examined in the past and there was no preferred vendor in place. unit libraries within ual normally purchased dvds online from various suppliers using a credit card. such a process was cumbersome and not cost effective, nor effective from a workflow perspective. the availability of a librarian intern initiated the process of further investigation into possible vendors and methods for obtaining this particular format of information across nine ual units, covering the complete range of disciplines served by the ual.   the acquisitions and bibliographic services department strives to streamline workflows within an environment of constant change (both in terms of staffing and new technologies), and also to meet the university’s procurement standards in dealings with vendors. as part of the evaluation of business relationships with materials vendors, the current landscape of materials supply must be examined, and a determination made regarding the best options for provision of service. the current increased reliance on digital media suggests that libraries’ video needs will be met with more streaming services, but digital streaming does not yet provide full coverage, necessitating the use of dvd vendors in the meantime. hence, this project set out to uncover a suitable dvd vendor for the ual in an effort to decrease credit card spending and to streamline acquisitions and cataloguing workflows.   evidence   several sources of evidence were used to make a decision on which vendor would be best for the acquisition of video materials at the ual. first, a literature review was conducted in order to examine the landscape of possible vendors and understand the acquisition and cataloguing activities at other academic libraries. current literature on academic libraries’ dvd acquisitions is limited, focusing instead on the shift to video streaming services (alonso-regalado, 2010; handman, 2010; mckenzie, 2006; widzinski, 2010) or on the development of a video collection (bergman, peters, & schomberg, 2007; walters, 2003). walters (2003), however, identified some possible vendors who distribute materials internationally, such as baker & taylor and ambassador media (now called ambassador education solutions) (p. 166). to supplement the lack of literature, information regarding possible video vendors was obtained through some academic library websites (north seattle community college library & media services, http://facweb.northseattle.edu/mpaz/media/vendors.html, and university of calgary selected interdisciplinary video vendors, http://libguides.ucalgary.ca/content.php?pid=100718&sid=817240) and the periodical video librarian online: the video review magazine for libraries, http://www.videolibrarian.com. based on content coverage and services, a short-list of possible vendors was established: the book house inc., action! library media, and coutts information services.   in order to have a better understanding of the types of dvds purchased by the nine unit libraries at ual via credit card, an examination and analysis of three months of credit card purchases was undertaken, looking at orders from june to august 2013. data recorded from each purchase included: date, merchant or source (such as amazon.ca, abebooks, and alibris), format (monograph, dvd, or cd), cost (including shipping and handling), publisher, title, and, for dvds, the film’s director. such data were collected to allow further analysis of credit card purchase trends both within and across library units in addition to providing information on dvd purchases. overall, 562 items were recorded and 30% of these were dvds. of those, 78% were purchased from amazon. the next biggest vendors of the purchased dvds were new day films, accounting for 5%, and the national film board with 4%. seeing so many dvd purchases confirmed the need for a formal, vetted, dvd vendor.   interviews were also undertaken with each library collection manager to ascertain video ordering preferences and procedures. librarians elaborated on their extensive use of and reliance on credit cards to purchase dvds. as no current video vendor was formally established, they cited a great reliance on amazon.   data from the examination of video titles recently purchased via credit card was used to test whether the short-list of vendors would be able to meet the ual’s dvd needs. a sample of 50 titles, covering a selection of diverse titles previously ordered from each library, including foreign language, popular, and independent films, was created and sent to three vendors that in our initial estimation would be best suited to meet the ual’s need for procurement of materials and shelf-ready cataloguing services. each vendor was asked to indicate whether or not they could supply these items (on an item by item basis), how much it would cost, an approximation of how long it would take to obtain the items, if they could provide cataloguing/shelf-ready processing services, and if so, how much those services would cost.   the evidence showed that no one vendor was able to supply all titles on the sample list; however, all three were able to supply much more than we initially supposed. one of the three short-listed vendors (book house inc.) did not wish to solely supply video materials, so they were dropped from consideration. of the remaining two, one was also our preferred supplier for english language monographs, coutts information services, and it was therefore decided that ual would use them as the primary supplier, with the second vendor, action! library media, being used as the secondary dvd supplier.   implementation   the process of investigation and selection of video suppliers was communicated to library staff in a presentation and via the ual collection development committee. acquisitions staff were informed of the change and procedures were drafted to detail the order of operations, detailing which vendor to use first and the processes for ordering with each vendor. since action! library media was a new vendor for the ual, a new account was opened and test orders were placed in order to further verify that the company could fulfil the orders.   outcome   the selection of two vendors to fulfil ual’s dvd acquisition needs has led to a streamlining of processes around the acquisition of this format. in practical terms this has meant a reduction in the use of credit card based purchasing in favour of ordering within established library vendor systems and receiving regular invoices. the changes are still new and will need to be monitored and adjusted over time. we plan to monitor fulfilment and turnaround rates, as well as refine processes as issues arise, such as how to handle rush acquisition of these types of materials. the changes have meant increasing ual’s use of an existing vendor, coutts information services, to a format not previously purchased, and the creation of a relationship with a vendor new to ual, action! library media. as ual monitor how both of these companies work with dvds, we will gain a better sense of which is best in handling specific types of material or publishers, and can tailor our ordering accordingly.   reflection   implementing this evidence based change was straightforward, encountering little resistance. initially there were doubts as to whether, in this digital climate and with such a plethora of dvd needs, a primary vendor could be established. the availability of an academic library intern ensured time would be spent finding and assessing dvd vendors, establishing confidence in the outcome of this project. having the evidence — comparing the cost of credit card purchases to vendors’ pricing, and seeing how much the ual were actually relying on credit cards — provided the rationale for giving preference to the identified vendors.   further, this project is a reminder that although digital technology has changed the way libraries serve their users, it is important to note that there is still a transitional phase — a time of learning and adjusting to trends. libraries are participating in video streaming services, providing access to films online, but demand still exists for physical video collections and should be met efficiently, especially in a restricted budgetary climate.   references   alonso-regalado, j. (2010). latin american and spanish online videos: vendor offerings for us academic libraries. collection building, 29(1), 31–36. http://dx.doi.org/10.1108/01604951011015286   bergman, b., peters, v., & schomberg, j. (2007). video collecting for the sometimes media librarian: tips and tricks for selecting, purchasing, and cataloging videos for an academic library. college & undergraduate libraries, 14(1), 57-77. http://dx.doi.org/10.1300/j106v14n01_05   handman, g. (2010). license to look: evolving models for library video acquisition and access. library trends, 58(3), 324-334. http://dx.doi.org/10.1353/lib.0.0094   mckenzie, r. (2006). academic media collections -a look at trends through one florida university library. florida libraries, 49(2), 10-12.   university of alberta. (2013a). statistical reports: u of a staff data book. retrieved from: http://uofa.ualberta.ca/reporting/statistical-reports   university of alberta. (2013b). students: by the numbers. retrieved from: http://uofa.ualberta.ca/why-ualberta/ualbertafacts/students   walters, w. h. (2003). video media acquisitions in a college library. library resources & technical services, 47(4), 160-170.  http://dx.doi.org/10.5860/lrts.47n4.160   widzinski, l. j. (2010). 'step away from the machine': a look at our collective past. library trends, 58(3), 358-377.  http://dx.doi.org/10.1353/lib.0.0092   evidence summary   graduate students reference open access content in literature review assignments   a review of: allen, e. j., & weber, r. a. (2014). the library and the web: graduate students’ selection of open access journals for empirical literature searches. journal of web librarianship, 8(3), 243-262. http://dx.doi.org/10.1080/19322909.2014.927745   reviewed by: kimberly miller learning technologies librarian albert s. cook library towson university towson, maryland, united states of america email: kimberlymiller@towson.edu   received: 14 may 2015  accepted: 23 july 2015      2015 miller. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – this study seeks to understand to what degree education graduate students cite open access, peer-reviewed journals in their coursework and whether patterns of open access referencing change over time.   design – longitudinal reference list analysis.   setting – public university in the united states of america.   subjects – reference lists collected from final literature review papers in a graduate-level education class (n = 382).   methods – the authors collected reference lists from final literature review papers in a graduate-level education class for a nine-year time frame from the 2005 to 2013 academic years. the authors analyzed 2,594 reference entries from the 382 reference lists in the sample.   the authors organized reference entries into spreadsheets for analysis, creating one spreadsheet per class section and sorting references by type (e.g., book, journal article, website, etc.) and source. the authors also created a cumulative list of journal titles that they analyzed for open access status and how often the journals appeared in the sample. other information collected about each journal included "issns, national origin, journal sponsorship, frequency and years of student usage, presence of empirical research, [and] peer review status" (p. 249). finally, the authors organized open access journals into the following four categories based on their access method:   category a "oa platform and publisher" category b "publisher only" category c "delay or hybrid from host or publisher" category d "subscription based, but self archived" (p. 249)   main results – a total of 594 unique journals appeared in reference lists over the study period, and 11.5% (n=68) of the total were open access journals. of the open access journals, 96% (n=65) included original research articles, and the majority (n=51) fell into category a "oa platform and publisher." nine, six, and two journal titles fell into categories b, c, and d, respectively. the authors found no pattern or change in the use of open access titles during the nine-year study period. open access journals appeared in reference lists an average of 14 times per year with the highest usage observed in 2009.   conclusion – the results show that graduate students in the sample used a range of open access journals. the presence of open access resources in reference lists signals that students may use both library subscription databases and open web search tools to complete their literature review assignments. the authors suggest potential reasons why open access use did not grow during the study period, including a possible mismatch between student research interests and the topics present in open access titles, the lack of discussion about open access publishing during library instruction, or student satisfaction with the resources provided through library-sponsored subscriptions. librarians are encouraged to include high-quality open access resources within their catalogues or other electronic resources to increase open access discoverability and to include popular open web search tools as a means to retrieve open access materials during information literacy instruction.     commentary   open access (oa) publishing presents a variety of opportunities and challenges to academic libraries. while some authors call for libraries to take on an active role in publishing oa works (chadwell & sutton, 2014), others discuss strategies for educating students about scholarly information economics (warren & duckett, 2010) and the oa policies that affect research funding (keane, 2012). the current study adds to this body of knowledge related to student interactions with oa research publications.   the study's primary strength, when examined with the cat: a generic critical appraisal tool (perryman & rathbun-grubb, 2014), lies in the choice of authentic, longitudinal assessment of student work. collecting reference lists over a nine-year period allows the authors to illustrate oa usage trends in a typical educational setting. this evidence may be valuable for librarians looking to understand students’ natural inclination toward finding and using oa content. librarians may also wish to consult the concise review of literature related to past, present, and future oa publishing trends. librarians should interpret results with the study limitations in mind, avoiding generalizations to other populations without considering whether the study’s population (graduate-level education students), setting (large public university), and disciplinary publishing patterns and valuing of empirical research is appropriate for their purposes.   the presence of oa materials in the study sample leads the authors to the conclusion that students rely on open web search tools to conduct research and to the recommendation that these tools should be incorporated into information literacy instruction. the article's literature review references prior studies that highlight graduate students' preferences for open web search tools like google scholar; however, the current study's evidence does not account for oa materials that are indexed in library-subscribed databases. without understanding the likelihood that students encounter oa materials through library-provided search tools or the methods students in the current sample used to discover oa materials, the conclusion that open web search tools were the discovery point for oa materials is not well supported. additionally, the balance between oa sources and subscription sources in the publishing industry over time may influence student citation patterns.   understanding the scope of oa content found in the study sample is difficult because the four oa content categories outlined in the methodology are not used to relay the results. instead, readers must align the findings with categories a through d for themselves, using data in the appendix to confirm category breakdowns.   regardless of discovery methods, whether through library subscriptions or the open web, this study demonstrates that students are finding and using open access content. the study’s recommendation to include discussions about oa content in information literacy instruction aligns with the association of college and research libraries’ (2015) recently approved framework for information literacy for higher education, which includes information as a commodity and oa publishing in the “information has value” frame. librarians may use findings that graduate students naturally include oa content in their work as another talking point in conversations about oa across campus.   references   association of college & research libraries. (2015). framework for information literacy for higher education. http://www.ala.org/acrl/standards/ilframework   chadwell, f., & sutton, s. c. (2014). the future of open access and library publishing. new library world, 115(5/6), 225-236. http://dx.doi.org/10.1108/nlw-05-2014-0049   keane, e. p. (2012). librarian viewpoints on teaching open access publishing principles to college students. the serials librarian, 63(3/4), 333-349. http://dx.doi.org/10.1080/0361526x.2012.724377    perryman, c., & rathbun-grubb, s. (2014). the cat: a generic critical appraisal tool. http://www.jotform.us/cp1757/thecat   warren, s., & duckett, k. (2010). “why does google scholar sometimes ask for money?” engaging science students in scholarly communication and the economics of information. journal of library administration, 50(4), 349-372. http://dx.doi.org/10.1080/01930821003667021 evidence based library and information practice editorial responsibilities      2014. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     editor-in-chief: alison brettle   associate editor (evidence summaries): heather pretty   associate editor (articles): lorie kloda; lisl zach   associate editor (reviews): denise koufogiannakis   associate editor (commentaries, using evidence in practice, news): alison brettle   production editor: michelle dunaway   editorial intern: melissa griffiths   copyeditors: richard hayman (lead copyeditor), georgianne bordner, julie evener, linda ferguson, heather healy, stacey penney, maria tan, mary virginia taylor, alison yeoman   indexing support: meral alakus   editorial   evidence in crisis?   alison brettle editor-in-chief school of nursing, midwifery and social work university of salford, united kingdom email: a.brettle@salford.ac.uk      2014 brettle. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   a recent headline “evidence based medicine: a movement in crisis” (greenhalgh et al, 2014), caught my eye.  eek, i thought, if evidence based medicine (ebm) is in crisis, what about evidence based library and information practice?  greenhalgh et al (2014) put forward a thought provoking argument suggesting that 20 years down the line despite a number of successes, there are a number of problems with evidence based medicine.  in addition to long standing criticisms of an emphasis on experimental research over clinical experience based on tacit knowledge, these include: misappropriation of the “evidence based” research agenda by vested interests, an unmanageable volume of evidence (including guidelines), a focus on statistically significant benefits rather than clinical ones, management rather than patient driven care based on inflexible rules and the inability of evidence based guidelines to deal with complex morbidity.  although these may be problems for ebm, i don’t think this is the case for librarians.  most of the problems highlighted stem from ebm’s emphasis on experimental research and focus on the “hierarchy of evidence” which lends itself to the creation of guidelines and rules. various authors have debated evidence and research evidence in relation to eblip (eg koufogiannakis and crumley, 2003; eldredge, 2002). and although we have bemoaned a lack of high quality research evidence in our field (e.g. brettle,2003: 2011), this does not prevent us from practicing in an evidence based manner and may well have led to a broader concept of evidence and model of evidence based practice as proposed by koufogiannakis (2013).   the solutions, proposed to the ebm crisis, however are of far more interest and relevance to librarians, as well as being aligned to the reality of eblip.  greenhalgh et al (2014) suggest that it is time to return to “real evidence based medicine” which is: individualised for the patient, based on judgement not rules, built on strong clinician-patient relationships and shared decision making.  these suggestions are akin to koufogiannakis’s (2013) proposition that the eblip model should consider all types of evidence with the librarian and professional decision making at the centre, and that the applicability of the evidence and the local context is taken into account when the decision is made.   furthermore the actions proposed to rescue ebm are also relevant to eblip.  these include: a demand for better evidence, training which combines critical appraisal with judgement and decision making, usable and robust evidence and a broad research agenda.  i think as librarians we should demand the same.  hopefully this september (2014) issue will help you do that.  it is full of a wide variety of research, from user surveys to routine data collection as ever with the aim of providing you with useful and applicable evidence to help in your local decision making.     references   brettle, a. (2003). information skills training: a systematic review of the literature. health information & libraries journal, 20 (suppl. 1), 3-9.   brettle, a., maden-jenkins, m., & anderson, l. (2011). evaluating clinical librarian services: a systematic review, health information and libraries journal, 28(1): 2-32.   crumley, e., & koufogiannakis, d. (2002). developing evidence-based librarianship: practical steps for implementation. health information and libraries journal, 19(2), 61-70. doi:10.1046/j.1471-1842.2002.00372.x   eldredge, j. (2002). evidence-based librarianship: levels of evidence. hypothesis, 16(3): 10-13. retrieved 6 dec. 2012 from http://research.mlanet.org/hypothesis/hyp_v16n3.pdf   greenhalgh, t., howick, j., & maskrey, n. (2014). evidence based medicine: a movement in crisis? bmj, 348:g3725. doi: http://dx.doi.org/10.1136/bmj.g3725     koufogiannakis, d. (2013). eblip7 keynote: what we talk about when we talk about evidence. evidence based library and information practice, 8(4), 6-17. retrieved from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/20486/15965   evidence summary   awareness of open access issues differs among faculty at institutions of different sizes   a review of: kocken, g. j. & wical, s. h. (2013). “i’ve never heard of it before”: awareness of open access at a small liberal arts university. behavioral & social sciences librarian, 32(3), 140-154. http://dx.doi.org/doi:10.1080/01639269.2013.817876   reviewed by: kathleen reed assessment & data librarian vancouver island university nanaimo, british columbia, canada email: kathleen.reed@viu.ca   received: 6 may 2014    accepted: 19 aug. 2014      2014 reed. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – this study surveyed faculty awareness of open access (oa) issues and the institutional repository (ir) at the university of wisconsin. the authors hoped to use findings to inform future ir marketing strategies to faculty.   design – survey.   setting – university of wisconsin-eau claire, a small, regional public university (approximately 10,000 students).   subjects – 105 faculty members.   methods – the authors contacted 397 faculty members inviting them to participate in an 11 question online survey. due to anonymity issues on a small campus, respondents were not asked about rank and discipline, and were asked to not provide identifying information. a definition of oa was not provided by the authors, as survey participants were queried about their own definition.   main results – approximately 30% of the faculty were aware of oa issues. of all the definitions of oa given by survey respondents, “none . . . came close” to the definition favoured by the authors (p. 145). more than 30% of the faculty were unable to define oa at a level deemed basic by the authors.   a total of 51 (48.57%) of the survey respondents indicated that there are oa journals in their disciplines. another 6 (5.71%) of the faculty members claimed that there are no oa journals in their disciplines, although most provided a definition of oa and several considered oa publishing to be “very important.”   the remaining 48 participants (46%) were unsure if there are oa journals in their disciplines. of these survey respondents, 38 answered that they have not published in an oa journal, 10 were unsure, and 21 believed that their field benefits or would benefit from oa journals.   survey respondents cited quality of the journal, prestige, and peer review as extremely important in selecting a journal in which to publish.   conclusion – the authors conclude that the level of awareness related to oa issues must be raised before irs can flourish. they ponder how university and college administrators regard oa publishing, and the influence this has on the tenure and promotion process.     commentary    as the authors point out, oa is most often discussed by librarians and faculty at large research universities, leaving a void in the literature related to oa awareness and motivations at smaller schools. this paper attempts to address this gap, but falters in communication of methods and results.   a stronger methods section would benefit the paper; the authors never specifically state a list of the questions they asked participants, and mention of the recruitment process is briefly referred to only in a footnote. while the authors place survey respondents into groups based on vague, basic, or advanced understandings of oa, they never define a rubric to explain how these decisions were made. understanding how various groups were defined is key to replicating the study or for comparison purposes.    aside from issues with methods, a further flaw in this article relates to gaps in the reporting of results. the authors never specifically state the level of oa unawareness found. instead they write, “the results of the authors’ survey indicates that a greater percentage of faculty members . . . do not know, or simply have a limited understanding of, what open access is” (p. 149). later, they mention the opposite of unawareness: “. . . the findings of our research suggest that open access awareness . . . was closer to 30%” (p. 152). if awareness is near 30%, than unawareness must be approximately 70% – a significant difference from the 15% found by xia (2010) in a study that the authors frequently cite. the authors fail to address this significant difference.    adding to the issues with the results is that the authors fail to make a distinction between “not understanding” oa and “unawareness” of oa. additionally, despite mentioning that the participants’ level of awareness of their institutional ir would be investigated, these results are never discussed.    while there are significant issues with this study, a helpful feature is the authors’ suggestions of potential strategies librarians might employ to better engage faculty in discussions about oa and contributing to irs. the authors suggest focusing outreach on specific disciplines (p. 153), and highlighting specific oa journals that would be beneficial to particular departments (p. 148). overall, this study fails to contribute in any significant way to the literature. perhaps the most useful aspect of this article is that it raises the question of how oa understanding among faculty differs based on institution size, and calls for further research from the perspective of smaller schools.   references   xia, j. (2010). a longitudinal study of scholars’ attitudes and behaviors toward open-access journal publishing. journal of the american society for information science and technology, 61(3), 615-624. http://dx.doi.org/doi:10.1002/asi.21283    news/announcements   conference announcement – 4th information: interactions and impact (i³) conference, 25–28 june 2013      2013. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     registration is now open for i³ 2013 – june 25-28, 2013 at robert gordon university in aberdeen, scotland, uk.   i³ focuses on the quality and effectiveness of the interaction between people and information and how this interaction can bring about change. the conference will look beyond the issues of use and accessibility of technology to questions about the way people interact with the information and knowledge content of today's systems and services, and how this might ultimately affect the impact of that information on individuals, organisations and communities.   conference themes include:   the quality and effectiveness of user/information interactions (e.g. information literacy); patterns of information behaviour in different contexts; impact of information or information services on people, organisations, communities and society (e.g. social, learning, cultural and economic outcomes of engagement with information); more effective decision making.   the organisers of i³ are delighted to announce the following keynote speakers:   dr. annemaree lloyd (school of information studies, charles sturt university) prof. jane mckenzie (director of the km forum, henley business school, university of reading) dr. adam rutherford (editor of nature, science writer and broadcaster for the bbc and channel 4) dr. christine urquhart (department of information studies, aberystwyth university)   for further details please visit www.i3conference.org.uk.  if you would like to register your interest and receive further alerts, please email the conference team at i3information@rgu.ac.uk.  you can also follow us on twitter (@i3_conference).   professor dorothy williams, i³ conference director   conference paper   libqual+® and the information commons initiative at buffalo state college: 2003 to 2009   eugene j. harvey assessment librarian suny buffalo state buffalo, new york, united states of america email: harveyej@buffalostate.edu   maureen lindstrom associate director for information commons suny buffalo state buffalo, new york, united states of america email: lindstma@buffalostate.edu      2013 harvey and lindstrom. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to examine the effect of a transition to an information commons model of service organization on perceptions of library service quality. in 2003, the e. h. butler library at buffalo state college began development of an information commons, which included moving the computing help desk to the library, reorganizing the physical units in the library around functional service areas, and moving the reference desk to the lobby.   methods – in 2003, 2006, and 2009, the library administered the libqual+ survey, which measures the relationship between perceived library service delivery and library user satisfaction. the 2003 survey was conducted before the implementation of the information commons initiative. analyses of variance were conducted to compare the effect of the service changes on users’ perceptions of library service quality between the three data collection points, as well as to explore differences between undergraduate and graduate students.   results – the analyses revealed significant differences between the three data points, with significantly more positive perceptions of library service quality in 2006 and 2009 than in 2003. comparisons between 2006 and 2009 were not statistically significant. in 2003, no significant differences were found between undergraduate and graduate students’ perceptions. however, in 2006, undergraduate students perceived higher levels of service quality after the development of the information commons than graduate students. this difference was maintained in 2009.   conclusion – the information commons has become a popular place for new programming, exhibits, workshops, and cultural events on campus. the library staff and administration have regained the respect of the campus community, as well as an appreciation for user-driven input and feedback and for ongoing assessment and evaluation.   introduction across numerous types of service businesses and organizations, of which libraries are a part, evaluation and measurement of service methodologies and outcomes has become a common, multifaceted necessity. the era of accountability has arrived, and libraries are no longer viewed simply as separate entities providing “inputs” into larger systems. rather, library systems naturally are part of these systems, and they must be defined and evaluated accordingly, including their processes, outputs, and outcomes in relation to larger systemic structures.   library measurement and evaluation evolved significantly throughout the 20th century and especially into the 21st century. several key contributors, as individuals and as members of larger library associations, enriched the field of library measurement and evaluation, and their contributions will be discussed briefly to provide a chronological context to undergird a portion of the literature review, particularly as it relates to the selection of the libqual+® survey instrument. more importantly, though, their contributions led to the recognition and acceptance of the need for library evaluation, which helped spur attempts to strengthen library evaluation research. one such attempt stemmed from an initiative from the association of research libraries (arl): a pilot project designed to examine and assess service quality among academic and research libraries. this project led to the development of libqual+®, a psychometric survey instrument designed to measure the relationship between perceived library service delivery and library user satisfaction. successive attempts to strengthen and expand the research base in this field continue today.   throughout the past six years, libqual+® played a special evaluative role at butler library at buffalo state college. in 2003, the e. h. butler library engaged in an extensive physical and virtual reorganization of service provision and delivery. specifically, the library initiated the development of and transition to an information commons model of service organization.  prior to this transition, however, butler library collected libqual+® data from its user groups for two primary reasons: 1) to establish a baseline (i.e., pre-test) for measurement of changes to users’ perception of library service quality over time, and 2) to receive concrete feedback from its constituencies to help guide the direction of development of the information commons. after completion of the information commons, libqual+® surveys were administered again in 2006 and 2009 for purposes of benchmarking against peers, self-benchmarking, and post-testing user perceptions of service changes.   this paper will present the evaluative, practical findings related to butler library’s journey of developing an information commons. a literature review will be presented, which will cover: 1) a brief acknowledgement of key contributors to the field of library evaluation research, and 2) an overview of libqual+®. further literature about the information commons model will be touched upon in the methodology section of this paper. the purposes of this research are simple: 1) to provide other academic libraries with a documentation of our successes and challenges in developing an information commons; 2) to illustrate changes in users’ perceptions of library services between 2003, 2006, and 2009; and 3) to contribute to the bodies of practice-based library research and service evaluation, particularly in relation to information commons case studies and libqual+® research.   literature review   library evaluation   most fields respectfully acknowledge the early works of their key contributors, and the field of library evaluation should be no exception. three prominent individuals wove a common thread in this field throughout the past century: james thayer gerould, a library administrator; f. wilfrid lancaster, a library educator, and duane webster, a library association executive (kyrillidou & cook, 2008). the efforts and contributions of these individuals highlight the evolution of library evaluation practices, and each brought different perspectives into the assessment and measurement of library services. their endeavors serve as the foundation for how future research would supplement their practices and findings and further improve upon library service evaluation models and methodologies.   servqual: the origins of libqual+®   arl reports of descriptive statistics fill a critical need in evaluative library research, even today. decades of statistics pinpoint practices of collection investment, (in)stability of library funding, and declines and improvements in resource allocation. trends in these areas can be monitored, and initiatives can be instituted when deemed important or necessary to the arl membership. however, these trends and practices make an assumption that has yet to be proven empirically: the relationship between expenditures and service quality (cook, heath, thompson, & thompson, 2001) “a measure of library quality based solely on collections has become obsolete” (nitecki, 1996, p. 181).   recognizing the lack of instruments that directly measure service quality from the user point of view, arl approved a membership-centered pilot project in 1999 to respond to college and university administration demands nationwide for accountability (cook, thompson, heath, & thompson, 2001). part of arl’s new measures program, this project represented a paradigm shift away from descriptive, collection-input driven measures toward service evaluation, user satisfaction, and formalized, standardized measurement initiatives grounded in scientific methodology. these efforts promoted the need to rely less on the arl index (arl statistics) as the primary, most important assessment tool; rather, this project represented a collective, collaborative effort of many arl-member libraries and librarians to adopt a new way of conceptualizing and conducting library evaluation.   to begin the collaborative efforts, arl accepted the adoption of texas a&m university’s research in servqual (service quality), a psychometric survey instrument that addressed user assessments of service delivery (cook, heath, thompson, & thompson, 2001). although it is beyond the scope of this paper to address servqual in-depth, one important point should be noted. the servqual instrument was designed in the 1980s to assess service quality in the for-profit business world (cook, heath, thompson, & thompson, 2001). thus, in order to utilize and incorporate this research into the field of library evaluation, arl requested the instrument be re-conceptualized, re-designed and re-tested to better address service delivery to users of libraries. the new instrument would need to be tailored to library users, rightly presumed to be a distinctly different population than traditional “business customers.” also, the instrument needed to be grounded in college and university library settings and environments; after all, libraries typically are non-profit entities focusing more on service provision (as compared to for-profit settings, possibly focusing on resource provision or production). nevertheless, servqual represented a promising survey model, a foundation from which a more library-oriented survey could be developed.   libqual+®: an overview   in general terms, libqual+® is a 22-core-item “total market” survey instrument designed to assess library service quality of an academic library from the point of view of the library user (thompson, kyrillidou, & cook, 2008). factor analytic studies and item analyses reveal that libqual+® measures the single overarching dimension of perceived library service satisfaction and quality (thompson, cook, & heath, 2001). however, this should not be confused with its three subscales: affect of service, information control, and library as place. these three “dimensions” measure components of library service satisfaction:   affect of service this aspect of user satisfaction examines the helpfulness and responsiveness of library employees to users. early libqual+® research indicates three components to this subscale dimension (cook, heath, thompson, & thompson, 2001). assurance is “the knowledge and courtesy of employees and their ability to convey confidence and trust” (cook, thompson, heath, & thompson, 2001, p. 265).  empathy includes the caring, compassionate, individualized attention of employees toward their users. responsiveness is the ability and willingness to provide efficient service to its users.   information control this aspect of user satisfaction examines the availability, timeliness and appropriateness of library resources. components of this subscale dimension include user perceptions of the comprehensiveness of collections, barrier-free access to information at the time of need, and information formats (e.g., print, digital, etc.) (cook, heath, thompson, & thompson, 2001).   library as place the final subscale measurement examines how well physical library facilities serve users’ needs for space and technology. this concept assesses the ability to meet needs for community socialization, utilitarian space (e.g., for study, collaboration, etc.), and space for creative and scholarly inquiry and rumination (cook, heath, thompson, & thompson, 2001).   although validity issues will be discussed later, it is important to note two potential shortcomings of these subscale areas. first, library as place is a continuously changing phenomenon, especially as technology demands force a shift from print-based resources to digital and web-based resources. loudly and clearly, users have expressed an overwhelming need for resources to be available anytime, anywhere, from any location (thompson et al., 2008). this demand has fostered technological changes in the ways in which resources are accessed, particularly from remote locations using computing and web-based technologies. thus, library as place is becoming less “physical.” as more resources become available as online digital full-text, the “dependency” on a library’s physical space for information resources becomes lessened. in fact, it may become possible in the not-so-near future for users to complete library research activities entirely in an online, digital environment. if this becomes the case, this aspect of user satisfaction may shift dramatically, if not be eliminated altogether.    secondly, and on a similar note, information formats are shifting toward digital, electronic versions. however, one particular item in the information control subscale inquires about “the printed library materials i need for my work” (thompson et al., 2008, p. 14).  again, this item may become less relevant given the shift toward digital formats. if the question is asked, it may “plant the seed” in the mind of the survey respondent that printed materials should be a part of a library’s collection. if a library shifts to a digital-based collection (which butler library has done—90% of journals are digital), then the respondent may perceive the library is deficient in this area. consequently, this item could threaten the validity of libqual+® data. this is why it is important for libqual+® researchers to monitor these trends and make necessary item modifications or deletions accordingly (e.g., delete the word “printed”).   libqual+®: psychometric properties and integrity   in 2007, libqual+® collected data from the one-millionth library user and the one-thousandth institution; and since its conception in the early 2000s, surveys have been administered to library users in 20 countries in 12 different languages (thompson et al., 2008). the sheer number of data collected is massive and expansive, lending to a richly diverse longitudinal collection of statistical information. what started out as a need for stronger evaluative measures in north american academic libraries has expanded to a global  scale, a truly remarkable representation of libraries both nationally and internationally.   validity   some libqual+® studies have engaged in rigorous statistical testing to determine criterion-related validity (thompson, cook, & kyrillidou, 2005; heath, cook, kyrillidou, & thompson, 2002). however, since libqual+® was a unique instrument, convergent validity, or statistical comparisons between instruments measuring the same or similar concepts, could not possibly be tested (shadish, cook, & campbell, 2002). instead, heath et al. investigated libqual+®’s concurrent validity, or the distinct ability to distinguish concepts from one another in order to measure each concept separately, as compared to the arl index, a predominantly collection-and-expenditure-based reporting instrument (heath et al., 2002; shadish et al., 2002). as expected, the “strongest” correlation between libqual+® and the arl index involved information access (r 2 = .147 = 2.2%), and this correlation was small. the reason the two instruments did not correlate presumably is due to each instrument measuring distinctly different concepts—libqual+® measures user satisfaction, and the arl index measures collection holdings and expenditures. thus, in a fascinating way, this study showed and strengthened libqual+® ’s validity by disproving its correlation with a conceptually different measure.   one other potential threat to validity is self-selection bias. libqual+® surveys rely on the voluntary completion of the survey by respondents. due to confidentiality, a library would not be able to access personally identifiable information (such as email addresses) for the purposes of conducting research using random-sampling methods. instead, libraries market the survey to its users utilizing whatever means available to them. libraries rely on these marketing efforts to “attract” users (and non-users) to participate in the typically web-driven survey. self-selection is not a random sampling method and, thus, carries with it the potential flaws of such a bias—the most general concerns being: “do respondents differ from non-respondents?” for example, a user who is greatly satisfied with library services may be more than willing to complete a survey “to help the library.” alternatively, a user who is greatly dissatisfied may be more likely, too, to complete a survey to voice their concerns. however, what about users who are “in the middle”—maybe only somewhat satisfied? are they more, less, or just as likely to participate in this survey? also, what about the likelihood of non-users to complete the survey? are library non-users just as likely to complete the libqual+® survey (or not complete it) than library users? these questions and concerns inherently could impact the validity of any research findings, including those of libqual+®.   reliability   a plethora of research studies have examined the stability of libqual+® ’s reliability, including longitudinal analyses, and most reliability correlation coefficients reach at least .85, .90, or even higher (thompson, cook, & thompson, 2002; thompson & cook, 2002; cook, heath, thompson, & thompson, 2001; cook & thompson, 2001). although it is beyond the scope of this paper to cover all reliability studies in depth, the research of thompson, cook, and thompson (2002) is most indicative of libqual+® ’s reliability.  their research reported a cronbach’s alpha coefficient of .948, a remarkably high internal reliability indicator.   item response scoring—the “gap measurement” model   given its roots in attitude measurement, libqual+® utilizes a gap-measurement model for item response scoring. for each survey item, respondents provide three different ratings; these ratings include:   the minimum level of library service that is deemed acceptable the perceived level of library service seen as being offered the desired level of library service. (thompson, cook, & heath, 2000, p. 166)   gap measurement relies on the perceived scores of respondents as indicators of service quality (thompson et al., 2000). specifically, the difference between perceived levels of service and minimum and desired levels of service is calculated to determine positive and negative scores. if levels of perceived service are greater than or equal to minimum levels of services, users typically are “tolerant” and accepting of the library’s service in that area. if it falls below that minimum, however, then the user believes the library is not performing up to their minimal expectations in that area, which typically results in dissatisfaction. similarly, if perceived service meets or exceeds their desired level of service, then typically a user is “satisfied.” anything below desired levels of service may be an indication of dissatisfaction. however, libqual+® posits that service quality may still be acceptable as long as the library meets users’ perceived minimal levels of service, even if they are not functioning at the desired level. this “gap” indicates a threshold known as the zone of tolerance.  ideally, libraries should attempt to meet users’ desired levels of service, but, even if they meet their minimal levels of service, libraries generally will be met with at least somewhat satisfied users.   gap measurement carries its own set of pros and cons. one positive outcome of gap measurement is an inherent “lie detection” and random response scale. “logically . . . a user’s rating of desired performance should never be below . . . minimally acceptable performance [ratings]” (thompson et al., 2000, p. 168). if so, especially if persistent throughout a respondent’s cumulative scores, it likely is an indication of random response (and, thus, a threat to score validity). consequently, such aberrances are determined through simple counting, and once aberrances for an individual survey reach a predetermined threshold, that survey is deemed invalid and subsequently is deleted from data inclusion.   another positive outcome also happens to be related to multiple ratings. gap measurement carries an “intuitive” appeal, a “complex simplicity,” if you will. assuming a respondent understands the nature of the rating methods and how they are related to one another, a respondent can provide very important, powerfully reliable data (thompson et al., 2000). one con of gap measurement involves the user directly. instead of responding to one 22-item likert-type scale, the gap measurement model “forces” users to complete three likert-type scales, one for each perceived service rating. this results in, minimally, a user completing over 60 responses. this reality may have been beyond their expectation and, consequently, may result in mid-completion respondent attrition, which typically is another threat to validity.   similarly, another con involves the user’s comprehension of an item’s concepts and/or constructs. for example, a respondent reaches the item: “library space that inspires study and learning.” if they do not understand the concept “library space” (or if it is not applicable to them, such as only accessing the library through remote digital access), they may be confused as to how to answer. then when they attempt to provide a score for each rating, the chances of computing imperfect scores are compounded (thompson et al., 2000). interpretation problems magnify inaccuracies when multiple ratings for one item are involved.   the information commons initiative at buffalo state college   historical background   2003 was the year of the perfect storm of bad news for butler library. as was the case in hundreds of academic libraries across the country, 2003 was a year of an unprecedented decrease in gate counts, reference desk statistics, and library material circulation. at butler library it also was the year of an unprecedented increase in technology-related questions and complaints: usernames did not work, e-mail accounts needed to be activated, passwords needed to be reset, printers were jammed, work was not saved, discs were lost, and software could not be loaded. students with these types of problems had such a confusing time resolving them that the process was given a name—“the buffstate shuffle.” in 2003 users’ frustration levels were high on all fronts, and staff morale seemed to be at an all-time low. library administrators were scrambling to justify filling vacant lines for functions that appeared to be in decline. as scott carlson noted in his 2001 article in the chronicle of higher education, “gate counts and circulation of traditional materials are falling at many college libraries across the country, as students find new study spaces in dorm rooms or apartments, coffee shops, or nearby bookstores” (p. a35). new technologies, increased automation, and of course the web, improved access to information and empowered users. it also kept users away from the library. the silence was deafening . . . but only for a while. we needed to find a way to get our users back.   our first formal step was to confirm what we suspected: users were staying away because they were unsatisfied with the library on many fronts. hence, in 2003, we administered the libqual+® survey to formally measure library patron satisfaction and, according to the data received, library user groups perceived butler library as falling short in all three dimensions/service areas. scores for overall satisfaction, affect of service, information control, and library as place ranged from the 40th to the 42nd percentiles. (baseline percentiles were determined through comparisons against 2003 libqual+® norms.)   william m. sullivan, senior scholar at the carnegie foundation for the advancement of teaching, stated, “thinking of a library as an information center is the first step toward losing it” (carlson, 2001, p. a35). what really was the library then if not an information center? the disappointing results of libqual+® served as a wake-up call for butler library to redefine itself. what resulted was the creation of the information commons and, seven years later, a library that had reclaimed its place as the academic and cultural heart of the buffalo state college campus.   college & library overview   buffalo state college, a carnegie master’s-l level institution, is the largest four-year urban college in the state university of new york (suny) system. enrollment for fall 2009 was 11,714 students: 9822 undergraduate and 1892 graduate students. five schools, the school of arts and humanities, the school of education, the school of natural and social sciences, the school of the professions, and the graduate school, offer 162 undergraduate programs with 11 honors options and 60 graduate programs including 17 post-baccalaureate teacher certification programs. first-year undeclared students are enrolled in university college, which provides support programs and specific opportunities to foster student success. the top five majors at the college are business, elementary education & reading, technology, criminal justice, and history.   butler library is a medium-sized academic library which houses more than 675,000 printed books, over 174,000 electronic books, and access to full-text articles from over 57,000 unique print and electronic journals. the library is open 110 hours each week during regular semesters and within our building we have two extended-hours facilities, studyquad and quietquad, which are open and staffed 24/7 during regular semesters. butler library is the largest open computer lab on the campus, housing more than 200 computers, which provide full access to library resources, the web, the microsoft office suite, and various specialized software applications. access to the wireless network and secure networked printing is also available in the library. the library has a café and several lounge areas. security cameras are installed for safety and the building is routinely patrolled by university police student assistants.   the beginning of a developmental plan   credit must be given to the seminal article by donald beagle, conceptualizing an information commons, for giving librarians at butler library a vision for the future. (1999) librarians by nature tend to be excellent organizers, visionaries, and adept at seeing the bigger picture. the road to revitalization of the library required a new way of defining the library’s purpose and its responsibility to provide support to the greater academic community. the information commons concept defined by donald beagle provided an excellent framework. of particular interest were beagle’s new descriptions for use of library space and his redefinitions of library services. butler library’s front line staff could clearly articulate many instances of poor or confusing service on campus. if we could consolidate the provision of essential services within the library itself, students would be better served by a “one-stop shop.” the plan was for that one-stop shop to become an information commons.   implementation: building an information commons   the look and feel of the butler library of seven years ago is but a distant memory—so much has changed. below is a summary of the major highlights of the library’s reorganization:   the computing help desk moves into the library   a review of the literature on restructuring academic libraries is full of information and case studies about the marriage of computing services and library services. in butler library this was the most obvious service to include in the information commons. this move allowed for support to be available at the point of need—most students discover they need password resets or specialized computer assistance when they using library computers. having the computing help desk in the library also raised user satisfaction levels as this service was physically more accessible and visible. the help desk staff instantly became supportive partners, fully participating in technology and customer service planning within the information commons.   continuous assessment/continuous improvement (ca/ci)   two librarians participated in a year-long ca/ci training workshop during which public service areas were evaluated and a structure for change was developed. continuous improvement continues to be the philosophy within the information commons.    use of an outside facilitator   during times of change staff can become nervous or concerned about their future role in the organization. the entire library staff needed to come together around an understanding and vision for the creation of an information commons. an outside facilitator was hired and helped aggregate input to create a newly envisioned mission statement for the information commons.  in our session, the facilitator did an excellent job of rallying the staff around a common goal. in retrospect, this activity proved to be extremely productive and worthwhile.   library reorganization   physical units in the library, such as microforms, media services, interlibrary loan, were re-organized around functional service areas. librarians had responsibility for functional areas but were encouraged to develop interdisciplinary partnerships and scholarship. the associate director for information commons position was created to oversee all public areas of the library including the web site and online and print resources. an information commons supervisor was appointed to oversee all clerical and student staff. all clerical staff were cross-trained in all functional service areas.   perhaps the most visible change, and the most controversial, was the move of the reference desk from the back reference room to the library lobby. librarians initially disagreed with this move, indicating the potential of compromised privacy and that the area was too noisy and too visible. however, within a week, reference desk statistics in all categories increased. reference librarians were busy again and librarians’ concerns soon subsided.   managing expectations   with little additional, direct fiscal expense, the concept of the information commons seemed to be a risk worth taking. this implementation, in a sense, could even be considered a trial phase, if necessary—enabling the library to try something new, yet leaving open the option of returning to the previous structure of services. even with some resistance and dissension, expectations remained cautiously optimistic. however, all agreed that increased visibility and aligning with user expectations was a positive step in the right direction.   post-implementation evaluation: the second data collection point (2006)   the year-long process of creating an information commons was well-grounded and justified by the disappointing results of the 2003 libqual+® data. in 2006, butler library administered a second collection of libqual+® data. although detailed results will be presented later, it is worth noting that users’ perception of overall library service quality changed significantly in a positive direction. across the board, libqual+® scores showed improvement in all three service dimensions. these results helped justify and confirm the direction of library service reorganization into the information commons model.   the services   almost immediately after the information commons was opened and marketed, typical library usage statistics (e.g., reference desk, gate counts, circulation) indicated the library was becoming busier, and campus offices and departments seemed to realize that conducting their business in the library could be more practical, more efficient and effective, and could reach more students. hence, the information commons became the site for new services such as:   the writing help center academic skills remote location advisement bengal id card office transfer and new student orientation the application support and training desk (a new technology and software service which the library itself decided to oversee and incorporate into the information commons)   as a direct result of the success of the information commons, the library received funding to create and staff this area to provide software and application support and training for students, faculty, and campus staff. this is the only area on campus that provides this much-needed service, its value indicated by the over 16,500 questions that were answered by this area in 2009.   equipment loan   students need to borrow equipment for use in their coursework. previous to the library taking on this service, equipment loan was located in a secluded office, which provided limited hours of service. the library identified space adjacent to the application support and training desk, purchased new equipment, created a web site to reserve and track this equipment, created video tutorials for proper use of this equipment, and as a result logged over 3,000 loans that year.   the bengal id card office   along with agreeing to print id cards and bus passes for all faculty, staff, and students, the library has become the site for the administration of all id card functions, including dining, vending, and printing.    professional development center   this new space opened in september 2010 and is the site for faculty and professional staff development programming and training. requests for space in the library continue to be made, again indicative of the excellent reputation of the information commons.    studyquad and quietquad   these areas were constructed in the library specifically because of student requests for late night collaborative and quiet study spaces. these areas are open 24/7 during regular semesters and are extremely popular for those students who have jobs or cannot study in the dorms.   methodology   this non-experimental, practice-oriented research study utilized the well-established libqual+® survey instrument as the primary means of collecting baseline data in 2003 and for two subsequent tri-annual data collections (2006 & 2009). after the three-year initiative to develop the information commons, the 2006 data collection, hypothetically, would highlight positive changes in users’ perceptions of overall service quality as measured by the libqual+®  instrument. finally, the 2009 data collection would indicate whether or not users’ satisfaction with the development of the information commons could be sustained or if it simply was the result of a dramatic short-term effect.   although libqual+® provides numerous demographic variables worthy of additional study, additional analyses were narrowed solely to differences between undergraduate and graduate students. examination of these differences happened quite serendipitously, mostly due to one of the researcher’s statistical background. such “data mining” techniques typically are frowned upon in the scholarly community as most sound research is perceived as deriving from theories or models and the development of research questions hypotheses before data collection and analysis (i.e., experimental research). however, for the purposes of practice-oriented library service evaluation, examination of data from a multitude of facets, dimensions, and variables truly gives practitioners a greater understanding of their users’ needs. ultimately, greater insight into user needs could equate to better provision of library services. thus, this data, despite being discovered through happenstance, will be presented, too.   participants beginning in 2003, butler library utilized a cross-sectional sampling plan to collect libqual+® survey data from its constituents in three-year intervals, the most recent in 2009. recruitment of volunteers occurred through three primary channels: direct outreach (reference desk interactions, classrooms, student & faculty contacts), marketing (campus newspapers, announcements on the web site, bookmarks, departmental and campus emails), and incentives (the chance to win an ipod). volunteers were asked to visit the library’s libqual+® survey page to complete the survey. only fully completed surveys were used for data analysis; imputation of missing data was not utilized. with the exception of undergraduate and graduate student status, most sampling demographic variables were not as crucial for the purposes of these evaluations. thus, they will not be reported in this paper. however, table 1 illustrates frequencies of undergraduate and graduate student participation based on year; this demographic variable was found to be important in some analyses.   formal analyses of other demographic differences for each tri-annual data collection point were never calculated, but demographics in libqual+® reports were reviewed and, roughly estimating, showed no substantive differences from the overall buffalo state college population.   all participants were from various user groups of buffalo state college: students, faculty and staff. library staff members were excluded from all analyses due to the potential for biased results (i.e., vested interests). faculty were included in analyses related to changes in perceived library service quality over the development of the information commons, but they were excluded from other analyses relating to undergraduate and graduate student groups.   testing instrument (libqual+®)   despite methodological flaws inherent to almost any testing instrument, including libqual+®, library faculty at buffalo state college selected libqual+® based upon its well-documented psychometric properties, which were discussed previously in the literature review, and for its value in collecting the same data over time, longitudinally. beagle, bailey, and tierney point out the lack of explicit evaluative instruments focusing specifically on the effectiveness of information commons services (beagle, bailey, & tierney, 2006). instead, like libqual+®, most evaluative instruments implicitly, or indirectly, measure said services. technically, libqual+® measures perceptions of library service quality, not information commons service quality, yet beagle and other scholars tend to accept the administration of libqual+® for such a purpose.   table 1 undergraduate and graduate libqual+® participation – 2003 to 2009   2003 2006 2009 undergraduate 266 423 380 graduate 50 54 76 total 316 477 456   score data   only the mean adequacy gap scores were selected from libqual+® data for use in most statistical analyses. these scale scores reflect the difference between a user’s expected minimum level of service and their perceived level of service. larger, positive adequacy gap scores indicate greater satisfaction, while negative scores indicate dissatisfaction.   results   a one-way, between-subjects anova was conducted to compare the effect of the aforementioned service changes on users’ perceptions of library service quality between three tri-annual data collection points (2003, 2006, and 2009). the levene test of homogeneity of variances indicated equal variance and, thus, supports the usage of anova (f [2, 1598] = 2.62, p > .05). results of the one-way anova revealed significant differences between the tri-annual data collection points (f [2, 1598] = 7.07, p = .001). post-hoc comparisons using scheffe’s test indicated significantly more positive perceptions of library service quality for the 2006 data point (m = .32, 95% ci [.09, .55]) and the 2009 data point (m = .307, 95% ci [.07, .54]) as compared to the 2003 data point. comparisons between the 2006 and 2009 data points were not statistically significant at p < .05.   the impact of these service changes on undergraduate and graduate student groups’ perceptions of service quality was explored also using one-way anovas. (post-hoc comparisons will not be necessary due to having only two factorial conditions: undergraduate or graduate student status. statistically significant differences will be between those two groups only.) in 2003, results of one-way anova indicated no significant differences between undergraduate and graduate students and their perceptions of library service quality (f [1, 314] = .014, p < .05).the levene test of homogeneity of variance indicated equal variance and supported the usage of anova (f [1, 314] = .724, p > .05). however, in 2006, results of one-way anova indicated that undergraduate students’ perceived higher levels of service quality after the development of the information commons than graduate students (f [1, 475] = 5.024, p = .025). equal variance was indicated through the levene test (f [1, 475] = .553, p > .05). this difference was maintained in 2009 as well, as shown through one-way anova (f [1, 454] = 4.013, p = .046) (levene test: f [1, 454] = .163, p > .05).   discussion   as hypothesized, the development of the information commons between 2003 and 2006 had a significantly positive impact on users’ overall perceptions of service quality, including in each of libqual+®’s three service dimensions. interestingly, the information commons model would seem to fit more into the “library as place” dimension, yet scores in affect of service and information control also improved significantly. perhaps the physical, virtual, and cultural “repackaging” of services indirectly affected users’ perceptions of these two areas. for example, a medical office seen as clean, comfortable, nurturing, etc. may influence patients’ expectations of the quality and competence of staff there (i.e., affect of service), whereas a less clean, uncomfortable environment would result in a different opinion or expectation of staff and service. a similar effect may have happened with butler library patrons. after revitalizing the environment with the information commons model of service organization and delivery, patrons’ perceptions of library staff and interactions with them (i.e., affect of service) may have improved as an indirect coincidence. a similar phenomenon may have occurred with the dimension of information control (e.g., perceptions of having better ability to access and retrieve information).   besides the inferential statistics applied in this paper, the scores for all three data sets were compared against libqual+® norms (cook, heath, & thompson, 2002; thompson, cook, & kyrillidou, 2006). this enabled butler library to benchmark results to that of other libraries as a means of comparison. also, it enabled the library to self-benchmark longitudinally over three years utilizing the same testing instrument. figure 1 illustrates this data.   this data further supports the findings from the statistical analysis section. butler library showed significant, positive gains in percentile scores between 2003 and 2006.   difference in results between 2006 and 2009 were not statistically significant. although the percentile for overall perceived service quality increased slightly, statistical analysis indicates that it could not be ruled out due to chance. however, one very important point should be noted: perceived service quality did not decrease. despite the economic downturn and subsequent fiscal “crunching” between 2006 and 2009, users’ satisfaction with service quality did not diminish significantly. the gains resulting from the development of the information commons were maintained, which suggests a long-term, sustained impact from developing such a model of service delivery. the butler library staff and administration were pleased overall with this result since it was hoped this model would not be a one-time “shot in the arm” or a dramatic fad. results from 2006-2009 comparisons support sustained, positive gains.     figure 1 butler library benchmarking & self-benchmarking from 2003 to 2009   statistical analyses for undergraduate and graduate students revealed no differences in their perceptions of service quality prior to the development of the information commons; without disagreement, it was apparent they were both equally dissatisfied with library services in 2003. however, for both the 2006 and 2009 data, analyses revealed that the development of the information commons had more of an impact on undergraduate students’ perceptions of service quality than graduate students. to help understand this difference, correlations between all 2009 libqual+® survey items and the overall mean adequacy gap scores were computed for both the undergraduate and graduate student groups. for each group, table 2 illustrates the five libqual+® items that most highly correlate with the mean adequacy gap score:   the development of an information commons best fits with the library as place service dimension. using table 2 as a guide, this dimension appears to be of more value to undergraduate students than graduate students. for undergraduates, three of the top five items stem from this service dimension. one explanation is that undergraduate students see the information commons and/or library as a necessity for their learning, study, and research. with a multitude of information, technological, cultural, and recreational services and activities, they may view the information commons as a place to “get away” and relax and/or a place to be nurtured when they need assistance.   library as place seems to be less relevant to graduate students, as evidenced in table 2; only one item stems from this service dimension. instead, more of their top items relate to information control and affect of service. many graduate students have families, careers, and other responsibilities outside of the college environment and, thus, might be less reliant on the information commons to fill the role of a “second home.” also, since many of their responsibilities and activities may center more on advanced research than undergraduates, the information control dimension is more important to graduate students.   table 2 top five libqual+® items for undergraduate and graduate students service element service dimension pearson r coefficient undergraduate students employees who are consistently courteous. affect of service 0.756 a comfortable and inviting location. library as place 0.755 library space that inspires study and learning. library as place 0.739 a getaway for study, learning, or research. library as place 0.724 employees who have the knowledge to answer user questions. affect of service 0.71 graduate students a library website enabling me to locate information on my own. information control 0.827 readiness to respond to user questions. affect of service 0.781 a getaway for study, learning, or research. library as place 0.779 employees who have the knowledge to answer questions. affect of service 0.776 employees who are consistently courteous. affect of service 0.774 the electronic information resources i need. information control 0.769   these findings sparked much debate among library faculty and staff, and they likely will guide future planning and services for the information commons. after all, graduate students are a very important user group too; and the planning of services must take into account their unique needs and interests, particularly in relation to their research interests and information requests. these findings would not have been identified without the libqual+® data and methods related somewhat to data mining. certainly this information is of critical importance and will be addressed in future endeavors.   conclusion   the information commons has become a popular place for new programming, exhibits, workshops, and cultural events on campus. one exciting new initiative, which has received extensive local and national recognition, was the creation of the rooftop poetry club. other new initiatives are the implementation of a digital commons, the library green initiative, the software virtualization project, google docs workshops, and the library blog.   beagle describes three manifestations integral to an information commons: the physical commons, the virtual commons, and the cultural commons (beagle et al., 2006). in butler library, the physical and virtual had been deliberately and consciously created. however, it was the cultural component that developed last, almost organically, and likely a result of our physical and virtual changes. beagle lists creative expression, public speech, popular and academic publishing, and scholarly inquiry as pieces of the cultural commons. butler library’s cultural developments and progressions include examples such as:   ·         new programming ·         new exhibits (e.g., a faculty publications showcase; campus and community art exhibits) ·         workshops (e.g., google docs; software programs) ·         the implementation of a digital commons for scholarly works and publications ·         the creation of a rooftop poetry club ·         the library’s green initiative ·         a software virtualization project ·         the library blog and newsletter   new partners   the information commons now partners with student affairs, graduate studies, orientation, instructional resources, college relations, events management, university college, the registrar, and computing and technology services to provide ancillary services to the campus.   recognition   since the creation of the information commons, butler library librarians have been awarded a chancellor’s award for excellence in librarianship, an excellence in library service award, and a library of the year award. our library director was promoted to associate vice president for library and instructional technology.  a new reporting structure, split between the provost and the chief information officer, reflects the collaborative nature and common goals of computing and technology services and the library.    benefits for students   seven years ago, a student coming to the library to complete a homework assignment would need to log into the library’s computers with her assigned username. if this student forgot her username, she needed to walk across campus to a different building to get assistance at the computer help desk. at this desk the student would be asked to show her id card. if this student did not have an id card, she needed to walk back to the library to the id card office where she might have to wait until the next business day to receive her id. the student would then have to walk back across campus to the help desk for a username and then finally back to the library to access the library’s computers and use the library’s resources.   seven years ago, there was no place to go for word processing assistance nor was there any equipment such as voice recorders, projectors, or laptops available for loan. there was no place for quiet study during late night hours as the library closed at 11:00 pm. meal plan services were in another building, the writing center was across campus, and coming to the library for a sandwich and a quick look at e-mail was unheard of.   today every student has access to all the following services in butler library:   id cards bus passes meal/dining/vending plans and funds computing help, including username look-ups and password resets class registration assistance advisement research paper writing assistance equipment loan specialized software assistance microsoft office assistance and instruction google docs assistance and instruction printing assistance library instruction and lunch!!   the process of revitalizing e. h. butler library through the implementation of an information commons has been an immensely rewarding experience for the entire staff.  not only has the butler library staff and administration regained the respect of the campus community, they also have regained an invaluable appreciation for user-driven input and feedback and for ongoing assessment and evaluation, including the well-established, multidimensional libqual+® instrument. most importantly, though, the users of the information commons have responded loudly and clearly – they approved of the changes in service structure, and their satisfaction with the information commons and its service quality has sustained over time.     references   beagle, d. (1999). conceptualizing an information commons. the journal of academic librarianship, 25(2), 82-89. doi:10.1016/s0099-1333(99)80003-2   beagle, d. r., bailey, d. r., & tierney, b. (2006). the information commons handbook. new york: neal-schuman.   carlson, s. (2001). as students work online, reading rooms empty out -leading some campuses to add starbucks. the chronicle of higher education, 48(12), a35-a37.   cook, c., heath, f., & thompson, b. (2002). score norms for improving library service quality: a libqual+® study. portal: libraries and the academy, 2(1), 13-26. doi:10.1353/pla.2002.0007   cook, c., heath, f., thompson, b., & thompson, r. (2001). the search for new measures: the arl libqual+® project—a preliminary report. portal: libraries and the academy, 1(1), 103-112. doi:10.1353/pla.2002.0007   cook, c., & thompson, b. (2001). psychometric properties of scores from the web-based libqual+® study of perceptions of library service quality. library trends, 49(4), 585-604.   cook, c., thompson, b., heath, f., & thompson, r. (2001). libqual+®: service quality assessment in research libraries. ifla journal, 27(4), 264-268. retrieved 20 may 2013 from http://archive.ifla.org/v/iflaj/art2704.pdf   heath, f., cook, c., kyrillidou, m., & thompson, b. (2002). arl index and other validity correlates of libqual+® scores. portal: libraries and the academy, 2(1), 27-42. doi:10.1353/pla.2002.0017   kyrillidou, m., & cook, c. (2008). the evolution of measurement and evaluation of libraries: a perspective from the association of research libraries. library trends, 56(4), 888-909.   nitecki, d. a. (1996). changing the concept and measure of service quality in academic libraries. journal of academic librarianship, 22(3), 181-190. doi:10.1016/s0099-1333(96)90056-7   shadish, w. r., cook, t. d., & campbell, d. t. (2002). experimental and quasi-experimental designs for generalized causal inference. boston: houghton mifflin.   thompson, b., & cook, c. (2002). stability of the reliability of libqual+® scores a reliability generalization meta-analysis study. educational and psychological measurement, 62(4), 735-743. doi:10.1177/0013164402062004013   thompson, b., cook, c., & heath, f. (2000). the libqual+® gap measurement model: the bad, the ugly, and the good of gap measurement. performance measurement and metrics, 1(3), 165-178. doi:10.1108/eum0000000007216   thompson, b., cook, c., & heath, f. (2001). how many dimensions does it take to measure users’ perceptions of libraries?: a libqual+® study. portal: libraries and the academy, 1(2), 129-138. doi:10.1353/pla.2001.0030   thompson, b., cook, c., & kyrillidou, m. (2005). concurrent validity of libqual+® scores: what do libqual+® scores measure? journal of academic librarianship, 31(6), 517-522. doi:10.1016/j.acalib.2005.08.002   thompson, b., cook, c., & kyrillidou, m. (2006, april). stability of library service quality benchmarking norms across time and cohorts: a libqual+® study.  paper presented at the asia-pacific conference of library and information education and practice (a-liep), singapore. retrieved 20 may 2013 from http://www.coe.tamu.edu/~bthompson/libq2005.htm   thompson, b., cook, c., & thompson, r. l. (2002). reliability and structure of libqual+® scores: measuring perceived library service quality. portal: libraries and the academy, 2(1), 3-12. doi: 10.1353/pla.2002.0022   thompson, b., kyrillidou, m., & cook, c. (2008). library users’ service desires: a libqual+® study. library quarterly, 78(1), 1-18.       article   evaluation of an academic library’s liquid designed website   holt zaugg, ph.d. assessment librarian harold b. lee library brigham young university provo, utah email: holt_zaugg@byu.edu   vera terekhova (nan)  doctoral student instructional psychology and technology brigham young university provo, utah email: verochkaterekhova@gmail.com   brian rennick web development manager harold b. lee library brigham young university provo, utah email: brian_rennick@byu.edu   received: 2014 aug 18   accepted: 2015 jun 19      2015 zaugg, terekhova, and rennick. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – when the harold b. lee library (hbll) at brigham young university released a new website with same-look capabilities for computers, tablets, and smartphones, we undertook a summative assessment to review website features and to determine baseline measures of website access via device and patron group.   methods – the study used a mixed methods approach using three levels of assessment (focus groups, an online survey, and a usability test), with each level informing the subsequent level.   results – the website changes were well-received by the overwhelming majority of patrons. device usage was associated with the type of task for which patrons were accessing the website. computers were used primarily for research-related tasks (e.g., accessing journals, databases, and the main search bar). smartphones were used primarily for on-the-go tasks (e.g., accessing personal accounts, finding library hours, and reserving group study rooms). tablets fell between these two. several website services were identified as being underused. study results were moderated by time of release (i.e., only half of survey participants had viewed the new website) and access to device (i.e., many patrons did not have access to a tablet or a smartphone).   conclusions – the summative assessment of the hbll’s new website was well-received and viewed as a positive change. while most patrons were initially unaware of the same-look feature across devices, this was considered to be a positive change. as devices become more accessible for patrons, it is believed that website access by device will change. a follow-up study is planned to assess any changes in use patterns or use of access devices. keywords: website design, website assessment, website evaluation, fluid design, mixed methods, usability test, same-look design introduction   in an effort to meet patrons’ needs, academic libraries continue to develop a strong virtual presence to provide online library services. these efforts include granting access to traditional services (e.g., journal and book searches, room reservations, instruction, consultations) using a variety of mobile applications and web 2.0 tools (e.g., social networking sites, blogs, wikis). the intent is to create and maintain intuitive and effective websites that meet users’ needs and preferences adapted to a wide variety of devices (aldrich, 2010; fang, 2007; george, 2005; houghton, 2000; kroski, 2008; tullis & stetson, 2004).   this study evaluates the release of a new website for a private, mid-western university library. the university has a student population of approximately 30,000 to 35,000 students and grants undergraduate and graduate degrees in a wide variety of disciplines (e.g. engineering, humanities, education, and nursing). while designated as a teaching university, the institution has a strong history of academic research. this study is a summative assessment of the new library website meant to meet the needs of university patrons (faculty, undergraduate and graduate students) on a variety of access devices (computer, tablet, and smart phone). it is summative in that it represents the point of release to patrons following development that included a needs assessment and multiple formative assessments to inform the redesign of the library’s webpage.   literature review   some examples of new technologies adapted for academic libraries’ website design are web 2.0 tools, built-in analytics software that tracks users’ behaviors, and a website layout that is consistent across mobile devices (i.e., tablets and smartphones). iterative website evaluations and continuous improvement are essential to meet users’ needs in times of rapid changes in technology. researchers describe the importance of identifying current patrons’ needs and providing responsive designs so evolving library websites are effective and relevant to patrons (aldrich, 2010; george, 2005; kroski, 2008). iterative website upgrades and improvements require constant evaluation of patrons’ use patterns and needs (george, 2005). evaluation tools used for such efforts include observations, built-in analytics, surveys, talk-along tasks, and usability testing. these tools provide data that indicate best practices for designing, developing, and improving library websites (fang, 2007; houghton, 2000; tullis & stetson, 2004; vandecreek, 2005).   george (2005) discussed how to improve an academic library website’s navigation through the use of strategic use of color and graphics to better attract users’ attention. she also recommended improving visibility by adjusting fonts, labels, and placement. readability is boosted by chunking and using keywords. finally, she suggested increasing usability with a consistent design throughout the website.   raward (2001) stated that libraries’ websites are designed with the goal of providing reliable content and an interface that is intuitive and easy for patrons to use. using human-computer interface research, she examined the main challenges of designing an effective academic library website and established a list of best practices for website development. the checklist includes 100 questions from four main areas: supporting user tasks and finding, understanding, and presenting the information. she suggests that user-centered design principles are the most successful for user experiences and these areas have become the main foci of academic website developers.   many researchers stress the importance of end user feedback during all stages of development, especially since traditional academic websites were based mostly on the experience and expectations of librarians and developers but not the users (crowley, leffel, ramirez, hart, & armstrong, 2002). in their study, crowley et al. (2002) conducted focus groups in order to gather users’ feedback on the library’s webpage. they were surprised to find out how many of the patrons were confused and frustrated with their website experience. they used this feedback to better understand user experiences to improve the website and make it more intuitive and user friendly for the patrons.   duncan and gerard (2011) described a process of integrating an academic library’s reference system into its website. one initial goal was to discover user needs and then rework the reference system to fit those needs. the process resulted in system-wide changes, not only to the virtual reference system, but also to all aspects of the reference system and customer service delivery. understanding user needs first was the foundation of their success.   evans (2012) similarly recommended using focus group panels of end users to make sure librarians receive users’ input about the new technologies to be implemented on the library’s website. instead of asking librarians to discuss what they think patrons’ technological needs are, patrons were asked directly. evans modified the focus group approach by creating a focus group panel and letting library staff sit in and observe patrons’ discussions. making sure the information finds its way from users to librarians is the key to success in the library’s website development.   while the website development is important, another consideration is how patrons are accessing the website. internet access via mobile devices is a growing trend as website users want to find something now instead of looking it up later (kroski, 2008). this results in patrons accessing information as they are moving through their daily tasks or during a momentary break. however, it is difficult for libraries to adopt technology to access virtual services that may be the flavor of the month (evan, 2012). while on-going evaluations during development are important, a summative evaluation is also important to know how the entire package works together. that is, how well do all changes made to the website integrate with access devices to meet patrons’ needs?   aims   following an extensive, iterative design and development process, the harold b. lee library (hbll) revised its website to better serve the needs of its patrons. a “liquid” design of the website that provides a same-look view across different devices (e.g., smartphone, tablet, and computer) was released for patron use. this study is a summative evaluation of the efficacy of hbll’s new website in meeting the needs of all patrons (faculty and undergraduate and graduate students). it also seeks to determine how each patron group accesses the website via the mobile devices for which it was designed. finally, the study seeks to establish a baseline for preference of device access by patron group use so that future assessments will indicate trends and patterns of library website access.   methods   this study used a three-part approach to determine the website and device usage patterns of library patrons. the three parts in our evaluation included focus groups, a university-wide survey, and usability test interviews. all data collection methods were approved by the university’s institutional review board (irb). each step is described below.   focus groups. a total of seven focus groups, consisting of male and female hbll patrons and employees 18 years and older were conducted. participants were recruited from a pool of library patrons used for library assessment purposes. an employee is any full or part time, non-student employee who works at byu. this group includes faculty who work at the hbll and faculty who work elsewhere on campus (non-library faculty), but both of which use library services. focus groups lasted 20–50 minutes and focused primarily on employees’ and patrons’ experiences with the new website. questions included demographic information (e.g., participants’ major or department affiliation, year in school or position) and experience with the new website (e.g., usage frequency, task preference, current and ideal device preference, problems encountered, and website satisfaction). focus group questions are found in appendix a. all focus groups were video recorded and transcribed. data from the focus groups informed the online survey and usability study.   survey. the second step of the evaluation included developing and distributing a survey to library patrons. focus groups’ responses and website developers identified areas of interest that were used to create the survey questions. survey questions are found in appendix b. the survey sought to confirm the patron usage patterns, current and ideal device preferences, and website services satisfaction as identified by the information from focus groups and website developers. the survey was administered via an online survey tool (qualtrics) to approximately 6,000 male and female participants 18 years of age and older who were students (undergraduate and graduate) and non-library faculty. response time to the survey varied depending on individual responses, but average response time was approximately 10 minutes.   usability test. as a final step of the study, a usability test was administered to 21 patrons (faculty, undergraduate and graduate students). participants were asked to complete 14 website-related tasks on three different devices (computer, tablet, and smartphone). the top 14 uses of the website, discovered during previous two steps, were chosen as usability tasks. all participants completed each task, but the tasks were randomized by device. each participant switched devices as he or she completed the 14 tasks. all usability tests were video recorded but not transcribed as comments generally referred to specific actions taken on each device. transcripts would not have provided the context of the comment without the image of what the participant was doing. participants for the usability test volunteered via the online survey and were provided with a gift card as compensation for the length of time (approximately 1 hour) for participation.     table 1 list of library website functions identified by website designers and patrons. 1 use main search bar 8 find an e-book 2 access databases 9 locate library floor maps 3 find a journal 10 reserve a group study room 4 use library catalog 11 find and reserve a movie 5 logon to account/holds 12 access course reserve 6 request interlibrary loan 13 add and delete a bookmark 7 find library hours 14 identify library events     funding was provided for this effort was provided by the hbll.   results   the results focus on three areas: (1) how well the new website was received, (2) what device patrons used to access website functions, and (3) how access devices (e.g., computer, tablet, and smartphone) were used to access the new website. the results combine findings from the focus groups, survey (with a 23% response rate), and the usability study.   response to the website. three-quarters of all patrons responding to the survey were happy with the new website design. undergraduate students were the most pleased and graduate students the least satisfied. of all patrons:   77% agree or strongly agree that they can easily find what they need on the new website, 75% report that the hbll’s new website is clear and understandable, and 78% state that the new website is visually appealing.   when disaggregated by patron group, faculty responses focused on learning how to use the new website for teaching and research responsibilities. graduate students were solely focused on the website’s functionality in their specific area of research. undergraduates welcomed the changes the most, viewing the website as more modern looking as they used it for course assignments (e.g., researching via databases, peer-reviewed journals, and books), collaboration activities (e.g., booking group study rooms), or recreational activities (e.g., finding a movie). the broader use of the website by a specific patron group, the more satisfied the patron was with the changes. comments typical of patrons were:   i find it better than the old website. fonts and colors are visually more appealing, and it’s easier to access the content you are looking for through the library. there is always a learning curve when using something new, but after exploring for a short while i figured things out easily enough. that tells me that it is a well laid out website.   i like the look of the new website. it is easier to search for articles and books by their category than to just search aimlessly.   function access by device. a main goal of website developers was to create a fluid design with a same look across multiple devices.   during this evaluation, a set of 14 website functions were identified by website developers and users as primary website functions. a list of each of these functions is provided in table 1.   using this list of website functions, patrons were asked to use access devices (e.g., computer, tablet, and smartphone) to complete usability tasks for each function. this usability study provided further indication of how well the website is accessed via the device by each patron group. results for the usability test are shown in table 2.   these results, along with comments made by patrons, indicate that patrons are generally able to access functions on the website easily. five of the functions were easily accessed regardless of patron or device. another three functions were easily accessed using two of the three devices.   however, when viewed by device, there is an increasing trend in the number of tasks that were increasingly difficult for patrons to do (computer = 0, tablet = 12, smartphone = 21). in these cases, size does matter as patrons commented that doing some tasks on the smartphone became increasingly difficult because the font size got smaller or patrons were required to zoom in and out to complete the task. in addition to this, the tablet and smartphone orientation (portrait or landscape) required patrons to scroll down to find website features. patrons reported that the need to scroll down to see features was not intuitive. while the look was the same, switching from portrait to landscape obscured obvious links, resulting in more difficulty for patrons to access functions.   faculty and graduate students were able to access the website more easily, while undergraduate students demonstrated a third more instances where the functionality caused problems.   other comments related to difficulty of access focused on the intuitiveness of the webpage link. as mentioned earlier, the mobile device orientation could obscure the link. however, in some cases, comments indicated that the link button simply did not look like a link or needed to be located in a more prominent place on the webpage. for example, several patrons did not find the link to their user account via their name (after log on) to be intuitive, regardless of device.     table 2 summary of successful use of website functionality by patron and device use website function faculty graduate students undergraduate students c t s c t s c t s 1 + + = + + = + + = 2 + + = + + = + + = 3 + – = + + = + + = 4 – – – – – – – – – 5 + + + + + + + + + 6 = = = = = = = = = 7 + + + + + + + + + 8 + = = + = = + = = 9 + + + + + + + + + 10 + + + + + + + + + 11 + = + + = + + = + 12 + = = + = = + = = 13 = + = = + = = + = 14 + + + + + + + + + c = computer, t = tablet, s = smartphone, + used with ease, = used with some difficulty, – did not use or difficulty using prevented success   table 3 preference of patron accessing website functions by device   accessed by not accessed website function (by importance) computer tablet smartphone main search bar 76% 6% 11% 6% databases 80% 5% 4% 11% journals 75% 4% 3% 19% my account (renewals, holds, etc.) 67% 6% 10% 18% interlibrary loan 57% 3% 3% 36% special collections 31% 2% 2% 66% course reserve 31% 2% 2% 65% group study room reservations 49% 4% 8% 40% library chat 25% 1% 2% 72% bookmarks 23% 1% 4% 72% library hours 55% 5% 11% 29% library floor maps 50% 4% 7% 39% library events 28% 2% 4% 67% worldcat 23% 2% 1% 74% library catalog 62% 5% 7% 26% physical books 62% 3% 7% 28% e-books 50% 5% 5% 40% movies 32% 2% 4% 62% audio books 20% 2% 3% 76% media equipment (e.g. cameras, sound booth, etc.) 15% 1% 2% 82% boldface indicates a high rate of use or non-use (70% or greater).     current website access. as this is the first website release that has the same look on multiple devices, one objective of the study was to determine current use patterns. this data serves two purposes. first, it indicates what devices patrons are currently using to access the website functions. second, it determines a baseline of use patterns for future reference. table 3 summarizes the percentage of all patrons accessing the website function by device. it also indicates the degree to which a website function is not accessed by any patrons.   while this information offers a better picture of how all patrons are accessing website functions, it is instructive to examine website function access by device and user group to establish trends and patterns among key user groups. tables 4, 5, and 6 indicate the top five rank-ordered, website functions accessed by each patron group as indicated on the survey. in this case, graduate students were further disaggregated into master’s and doctorate students to determine whether differences existed in use patterns for this specific patron group.   table 4 top five accessed services on a computer by patron group undergraduate students master’s students doctorate students faculty main search bar main search bar journals databases databases journals main search bar journals journals databases databases main search bar my account my account my account interlibrary loan physical books interlibrary loan interlibrary loan my account     table 5 top five accessed services on a tablet by patron group undergraduate students master’s students doctorate students faculty main search bar my account journals main search bar my account main search bar databases library catalog library hours databases main search bar e-books e-books journals interlibrary loan databases library catalog interlibrary loan my account my account     table 6 top five accessed services on a smartphone by patron group undergraduate students master’s students doctorate students faculty main search bar main search bar e-books main search bar library hours my account my account library hours my account e-books physical books my account gsr reservations library hours journals library catalog library floor maps gsr reservations main search bar library floor maps gsr=group study room     while there is considerable overlap between patron groups, undergraduate students’ access has a strong relation to website access and the physical library. each subsequent groups’ website access is more tied to virtual functions rather than to physical resources. the rank order also speaks to patron group preferences when accessing library resources. this listing provides the opportunity to examine use patterns of patrons and to determine how patrons can be directed to other complementary resources helpful for learning, instruction, and research needs.   limitations of the study   while a needs assessment and on-going formative assessment took place during the development of the new website and promotion of the website prior to its release occurred, two factors limited the assessment of the new website. while the limitations were of concern, we feel that the samples obtained were representative of the library website patrons and provided a strong baseline of the website’s utility and patron access.   lack of experience with the website. regardless of when the new website was released, there would be a period of adjustment to the new website features. one of the obvious findings of this study was users’ lack of experience with using the new website. forty-seven percent of the students and faculty surveyed online reported that they did not have a chance to yet utilize the website. several participants during the focus group used mobile devices and laptop computers to access the website to make comments on the website during the focus group. this first time access indicated the limited exposure patrons had with the new website. the lack of experience using the website was evident but expected as patrons learned the new functionality.   lack of experience with access devices. many participants did not have experience using some devices to access the website. for example, 65% of the survey respondents had not used a smartphone and 72% had not used a tablet to access the website. faculty and graduate students reported rarely using a tablet or a smartphone to access the website.   choice of device to access the website depended on patrons being familiar with the functionality of the device and having access to the device. for example, when asked if the website was accessed via smartphone, several undergraduate students expressed the sentiment, “i wish i had a smartphone.” faculty also reported only rarely seeing some students using a tablet in class to access the library’s website. the lack of experience using a specific device may have influenced patrons’ access of specific website features.   discussion   this study reiterates the importance of continuous feedback from patrons regarding library products and their delivery (aldrich, 2010; george, 2005; kroski, 2008). while a needs assessment and continuous patron feedback occurred during the development of the new website, a summative evaluation was required upon the website’s release. such an evaluation provides a broader look at the library’s services or products and how they integrate into the overall organization. while this evaluation is primarily summative at this point in time, it inevitably uncovers additional information to inform future developments. it helps to embody the attitude that there is no limit to improvement.   in this specific case the look, feel, and access to the website is an improvement over the previous website. the changes in font, color, and link placement serve to make the website more functional, easier to navigate, and shortens the learning time, especially when using different devices. this finding is in line with previous suggestions from george (2005) on how to change fonts, colors, and placement to improve website usability. comments from patrons throughout the data collection indicated that the website provided a cleaner, sharper appearance that facilitated use. the changes make the website use more reliable and intuitive to find library products (raward, 2001).   the study also highlights several use patterns among patrons. computers (both laptop and desktop) continue to be the dominant device for website access, but there is potential to use mobile devices more in accessing library services. the key is using the device as a tool that is best suited for accessing a specific website function. mobile devices appear to be beneficial for on-the-go tasks related to attending the physical library (e.g., booking group study rooms, finding library hours) or housekeeping tasks (e.g., accessing one’s library account to renew books). while research tasks may be done on mobile devices, these tasks are better done on computer since the smaller text size makes mobile access more difficult. tablets may become a happy mid-point between mobility and functionality of device access, as patrons gain increased access to and experience with them.   having a fluid design enables patrons to learn only the website features and their location on one website once, but care needs to be taken with limitations of device use. for example, mobile device orientation may limit users’ views and patrons need to be aware of the need to scroll up or down or to zoom in or out to fully access the page views. the findings also indicate that device ownership does not translate to device use for accessing the library services via the specific device. it is important that librarians provide instruction using each device so patrons are better able to understand how each device may be effectively used.   other findings indicate which device is commonly used to access website functions. the computer is the dominant device used by patrons for accessing website functions associated with research activities (e.g., main search bar, databases, and journals). the results also establish smartphones as a more dominant access device than tablets. however, this pattern may be affected by patron ownership of or access to a specific device. greater numbers of patrons owned smartphones than owned tablets.   the dominant activities associated with smartphones indicate a pattern toward easily accessed functions. the on-the-go functions accessed include, but are not exclusive to, accessing patron accounts, reserving group study rooms, finding library hours, and viewing library floor maps. if access via smartphone was easier than using a computer, patrons used the smartphone. a common comment indicative of patron smartphone use was:   if i can access the service faster than it takes to get out my computer and look it up, i use my smartphone.   use patterns also provide indications of how library services are used. for example, physical library services are accessed more by undergraduate students. graduate students and faculty tend toward a greater use of virtual access of library services. while this is helpful for device access, it also provides insights into the type of patron services that should be developed for virtual use. the use patterns also indicate services that are rarely or underutilized (e.g., library chat, bookmarks, worldcat, and media). this knowledge enables librarians to change instructional and promotional efforts to increase patron exposure to these services. it also enables librarians to examine the service to see if it is working as intended through additional input from patrons or if additional instruction is needed (duncan & gerard, 2011; fang, 2007; houghton, 2000; tullis & stetson, 2004; vandecreek, 2005).   instruction from librarians and faculty provides the opportunity to make patrons more aware of website services and how these services may be accessed on each device. the instruction may be imbedded in current discipline instruction, added on to library research instruction, or provided as stand-alone instruction. in the latter case, instruction may come in the form of short tutorial videos to help patrons learn or remember how to use the website better. this instruction helps patrons understand how services may be accessed via different devices and which services are best accessed with which device. it will help to shorten the learning curve on how to use different website functions and increase the usability of the website.   future directions of research include a follow-up study to continue to revise and expand the website’s functionality (e.g., updating floor maps). as patrons gain more access to mobile devices, it will be of interest to determine if and how website access changes across devices and patron groups. since one purpose of this study was to determine a baseline of use, determining future changes in website access will influence how the website adapts to those changes.   conclusion   this assessment determined the usage patterns of patrons of the harold b. lee library’s new website via a variety of devices. we conducted a three-step evaluation including focus groups, an online survey, and usability tests using computers, tablets, and smartphones. each stage of the study helped inform the next stage, and the data gathered at each stage was used to triangulate the results and conclusions. about a half of our respondents did not have experience with the new website and were unable to provide us feedback because the website only became live at the beginning of the summer term when many students and faculty do not have classes.   the changes to the website were well-liked and well-received by a vast majority of the undergraduate students as well as most graduate students and faculty. the majority of the patrons with website experience reported high levels of satisfaction with the website’s look and aspects of its functions. most areas of dislike were deemed to be issues of personal preference, issues that are easily fixed, or issues beyond the responsibilities of library website designers.   another of our purposes was to explore patrons’ device preference for using the website. computers are owned and used the most of the three devices, followed by smartphones and tablets. patrons’ device preference for accessing website function was also determined and disaggregated by patron group and device. there are certain tasks patrons would perform on any device (e.g., search for materials or manage their account). there are tasks most participants would only perform on their computer (e.g., research or accessing course reserve). finally, patrons prefer accessing specific functions of the website on their mobile devices (e.g., checking the library’s hours or reserving group study rooms) because of the ease and convenience.   the summative evaluation provides a broad view of the library’s new website. it increases the understanding of how well the website is working to meet patron needs. finally, it adds to the feedback knowledge for future modifications to help the website be more functional, intuitive, and useful to patrons.     references   aldrich, a. w. (2010, june). universities and libraries move to the mobile web. educause quarterly. retrieved from http://www.educause.edu/ero/article/universities-and-libraries-move-mobile-web.   crowley, g. h., leffel, r., ramirez, d., hart, j. l., & armstrong, t. s. i. (2002). user perceptions of the library’s web pages: a focus group study at texas a&m university. journal of academic librarianship, 28(4), 205–211.   duncan, v., & gerrard, a. (2011). all together now! integrating virtual reference in the academic library. reference and user services quarterly, 50(3), 280–292.   evans, g. (2012). emerging technologies: how do we know what’s happening “on the ground”? public services quarterly, 8(2), 164–170. doi:10.1080/15228959.2012.675284   fang, w. (2007). using google analytics for improving library website content and design: a case study. library philosophy and practice (e-journal), paper 121.   george, c. a. (2005). usability testing and design of a library website: an iterative approach. oclc systems & services, 21(3), 167–180. doi:10.1108/10650750510612371   houghton, d. (2000). building an academic library website: experiences at de montfort university program. electronic library and information systems, 34(3), 269–280. doi:10.1108/eum0000000006934   kroski, e. (2008). on the move with the mobile web: libraries and mobile technologies. library technology reports, 44(5), 5–9.   raward, r. (2001). academic library website design principles: development of a checklist. australian academic & research libraries, 32(2), 123–136. doi:10.1080/00048623.2001.10755151   tullis, t. s., & stetson, j. n. (2004, june). a comparison of questionnaires for assessing website usability., usability professionals association conference (pp. 1–12). retrieved from: http://home.comcast.net/~tomtullis/publications/upa2004tullisstetson.pdf   vandecreek, l. m. (2005). usability analysis of northern illinois university libraries’ website: a case study. oclc systems & services, 21(3), 181–192. doi:10.1108/10650750510612380   appendix a questions for the focus groups   1.       please tell us your major (students) or faculty position or staff position, and year in school (or position). 2.       how frequently do you use the library website? 3.       describe your typical uses of the library website. 4.       what are the most important features of the website? 5.       what device do you usually use when accessing the website? 6.       given the choice, what device would you use for the website? 7.       why would you choose that device? 8.       have you tried the new version of the library website? what have you noticed is different? 9.       how easy is navigation of the new website? 10.    what problems or issues have you encountered in using the new website? 11.    given your list of typical uses, how do you feel about the new website meeting your expectations for achieving the tasks? 12.    please share any other thoughts or comments about the library new website.   please note that questions are numbered for convenience of reference. depending on the course and comments of the focus group, questions were discussed in different order. not all questions were addressed in some focus groups because patrons’ responses extended longer than the scheduled time of the focus group. finally, additional questions were asked to help clarify or add explanation to given patrons responses, such as, “would you please elaborate more on that?” and “please explain that idea a bit further for me?”   appendix b:  questions for the online survey   1. what is your affiliation with brigham young university? ·         undergraduate student ·         graduate student masters ·         graduate student doctoral ·         faculty member ·         staff   1a. how many years have you been in: a. undergraduate studies? b. your graduate program? c. a byu faculty position? d. a byu staff position?   pull down menu (1-45)   1b. are you/have you been             a. hbll student employee?                                                                 b. hbll faculty/staff?   2. what is your major/department affiliation?   pull down menu (list of departments)   3. in the last six months, approximately how frequently have you logged onto the harold b. lee library website? ·         about once a day ·         about 2-3 times a week ·         about once a week ·         about 2-3 times a month ·         about once a month ·         less than once a month ·         never   4. how proficient would you rate yourself when using the library website? ·         very proficient ·         proficient ·         a little proficient ·         not at all proficient ·         other ______________   5. what device do you usually use to access the website? (select all that apply) ·         personal computer ·         library’s desktop computers ·         university (non-library) computer ·         tablet ·         cell phone ·         other ____________________   6. please select the device you use to access the harold b. lee library website to access:    computer   tablet   cell phone   i do not access this hbll service databases         journals         physical books         e-books         bookmarks         interlibrary loan         media services (e.g. dvds, audio books, video cameras, etc.)         library catalog         worldcat         special collections         my account         library chat         course reserve         group study rooms reservation         search bar         library hours         library maps         library events           7a. do you use your smart phone to access hbll website? ·         yes ·         no ·         i do not have a smart phone if no or i do not have a smart phone, go to question 8   if yes, go to question 7b.   7b. to what degree do you access the following services on your smart phone? 1=very rarely, 5=all the time   very rarely rarely sometimes often very often study room reservations           hours           book check-out           bookmarks/check-out list, call numbers           course reserve           my account (e.g. book renewal)           media services (e.g. dvds or audio books)           databases           events           search bar           maps             8. please rate the following statements about the library website by choosing from 0 to 5 with 0 being no experience, 1 being strongly disagree, 5 being strongly agree   ·         the information offered on the website is clear and understandable ·         the website is easy to navigate to find what i need  ·         the website is visually appealing to me ·         my expectations of accessible information were met by the new website ·         what is available on the mobile website right now is not what i would do on my smart phone ·         the things i would like to do on my smart phone are difficult to access on the mobile website ·         i never used the site on my phone because i didn't know it became more mobile friendly ·         it is difficult to access my account-based things (e.g. bookmarks or renewals) ·         i want to be able to customize the library webpage links to meet my needs ·         when reserving group study rooms, i want them to be categorized by size and equipment, and not by location ·         i want to know how search results are grouped and categorized ·         often when search results show the book is available, it is not found on the floor ·         i would like to be able to access my check out history ·         i would like to be able to access my past searches ·         i do not love the new design – the colors and theme are not consistent with all byu websites ·         the color scheme and fonts make it hard for me to see things clearly on the new website ·         what would you like to tell the library about the new website? (open-ended)   10. in our efforts to fully evaluate the library’s new website we are looking for participants for interviews and usability studies. would you be willing to participate in an:   interview (15-20 minutes, volunteer)? yes no   usability test (30-40 minutes, compensated, need access to smart phone)? yes no   if no, end of survey.   if yes, please provide your name and contact information below.   name: phone: email:   end of survey. evidence based library and information practice commentary   newcastle libraries’ evaluation strategy: evidence based practice in challenging times   becky cole project officer: digital inclusion newcastle libraries newcastle upon tyne, united kingdom email: rebecca.cole@newcastle.gov.uk   received: 12 nov. 2014 accepted: 16 nov. 2014      2014 cole. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   challenging times   in the last twelve years, newcastle libraries has undergone a significant period of organizational change. this commentary describes how evidence based practice has informed and focused these developments.   newcastle libraries is the statutory public library service for the city of newcastle upon tyne in the northeast of england. in 2002, the service began a wide ranging modernization program, and by 2009, it was recognized as a leading exemplar in its field with a large, newly built destination city library, seventeen satellite branches, a mobile library, and a home delivery service. in 2010, the election of a new national government transformed the political and financial landscape in which public libraries operate, precipitating further reinvention and accelerating the need for innovative service delivery models, multi-functional library spaces, and new ways of working.   in 2014, newcastle’s organizational structure reflects these developments. four years of public sector budget cuts and a revolutionary shift in user behaviour brought about by the ascendancy of the networked information landscape have challenged the service to reinvent itself as invisible intermediary, memory institution, learning centre, and community resource (brophy, 2008, p. 8). some core services have disappeared and been replaced by new services, such as the business and intellectual property centre. after the adoption of a hybrid model, library services in the city library and several branches share premises with complementary organizations, such as social housing, adult education, and youth employment support providers.   in brief summary, this may seem like a reversal of fortunes, but in a period of widespread national library closures, it could have been a great deal worse. the chartered institute of public finance and accountancy (cipfa) have calculated that 201 u.k. library service points were lost in financial year 2011–12 with a further 74 closing in 2012–13. another 411 buildings and 80 mobiles are “currently reported as either likely to be closed or passed to volunteers or have been closed/left council control from 1/4/13 to 31/3/14” (“reasons”).   in newcastle, the comparative preservation of services has been achieved through resilient leadership, an energetic political and public advocacy program, and a solid foundation of service delivery. this relative survival is also due in part to the organization’s implementation of a dedicated evaluation strategy, the proactive collection and use of evidence demonstrating impact, and the libraries’ willingness to respond to this evidence (within the constraints of budget and capacity) when determining how the service can and should evolve.   types of evidence   today, the types of evidence that newcastle collects—like the buildings it inhabits—reflect both the political and financial demands on the sector and the wider societal and informational zeitgeist. the way that newcastle thinks about and uses evidence is innovative and multi-faceted. standard quantitative measures (% computer usage, library memberships, books and e-book issues, and visits) still play a part, but these are supplemented by client-specific statistics, such as numbers of people attending work clubs or requesting assistance with online job search or social housing platforms.   recording the types of support patrons currently seek in public libraries builds a picture of the social uses of library buildings, which is of academic interest to information professionals. it also enables library managers and advocacy groups to demonstrate to stakeholders (and in the case of newcastle’s close neighbours, gateshead libraries, to the local media; see proctor 2014 in the reference list) the economic value of libraries as trusted information providers experiencing an increasing demand for assisted services.   to complement this quantitative data, newcastle collects visual and multimedia evidence, including photographic records of exhibitions, launches, installations, and celebratory events, that is used to illustrate promotional materials, such as the quarterly “page turner” brochure and the “annual guide.” the organization is active on social media, promoting events and campaigns via blogs, facebook, and twitter. as a result, it creates a digital record of the breadth of its operations and the way that it interacts with service users.   the libraries record in-depth, qualitative evidence by means of a bespoke evaluation toolkit devised and implemented in 2011–12 and used to collect, store, and present rich, explicit evidence of impact. this evidence includes general trends and preferences, quotes and comments from users, empirical evidence (where staff provide feedback of their observations), and reactions and responses to library services. the evidence is discussed further in the article “rich emotive, evidence of impact” (cole, 2014).   partnership work and the delivery of discrete, externally funded projects demand an additional layer of rigorous data collection and reporting. here, the primary focus is the “use value” of the resource, i.e., “the favourable consequences” of using the “information products or service” (tenopir, 2013, p. 271). in newcastle, the european regional development fund-funded business and intellectual property centre and the chartered institute of library and information professionals information literacy group-backed go digital newcastle digital inclusion project maintain case studies, financial risk registers, and qualitative and quantitative research outputs to demonstrate to stakeholders the benefits to small and medium sized enterprises and digitally excluded residents of interacting with their services.   the toolkit’s use of open-ended questions captures evidence pertaining to the function and value of “library space as a service . . . for cultural events (poetry readings, book launchings, displays of artwork . . .)” (delong, 2008, para. 5). in 2013–14, users’ opinions, including “simple praise or complaints” of events and the spaces in which they were delivered, were used to support decision making by instigating specific courses of action (e.g., adjusting the room temperature) and informing general approaches, such as liaising with the library café to reduce noise pollution (tenopir, 2013, p. 272).   evidence in practice   in times of change, the way an organization approaches evidence collection and use becomes even more crucial. in the last five years, evidence based practice has had radical implications for newcastle libraries by informing decisions over areas of expansion (where evidence has been used to secure external project funding that adds value to the service), and, inevitably, contraction. difficult decisions over library closures were based on stark roi calculations (visits and issues against building and staffing costs) and the physical distance of one library from the next.   the evaluation toolkit has enabled newcastle to take a structured, integrated, and user-focused approach to evidence collection by following the principles outlined by brettle (2014): “specify the outcomes you are measuring, so that you know what evidence you need to collect . . . [and] be aware of your stakeholders so that you can ensure you collect evidence that is important and relevant to each” (para. 2). newcastle’s primary stakeholders are its users (actual and potential), parent organization (newcastle city council), funders, staff, staff representatives, managers, and national bodies and advocacy groups—each of whom has a different perception of the libraries’ quality.   for the service user, quality often means exceeding expectations and delivering satisfaction. to help achieve this outcome, the toolkit asks questions that invite critical feedback from library users that is utilised in future planning. in 2012–13, parents attending story time sessions were asked to suggest improvements and indicate other activities they would like to see. responses were aggregated and analyzed as a word cloud that highlighted the terms toddlers and more. this data indicated a demand for increased provision for under-five’s and led to the development of little bears story and rhyme sessions that were rolled across the service.   the current volatility of the u.k. public library sector means that effective practice is sometimes as rudimentary as survival, and survival depends (at least in part) on the ability to prove value to the parent organization. in practice, this means alignment with “wider organisational objectives and priorities” and asserting the library’s significance within the overall structure (grant, sen, & spring, 2013, para. 15). newcastle’s toolkit achieves the former by linking harvested data (at the collection stage) with the city council’s four key performance priorities. the toolkit achieves the latter by collecting “‘explicit’ measures of value that come directly from testimonials” and that are used to support accreditation in areas such as customer service excellence (tenopir, 2013, p. 271).   with in-house resources scarce, public libraries are increasingly reliant on external funding, and the relationship between evidence and practice in this area is particularly complex. for the potential funder, quality often translates as evidence of need, originality, and value for money (poll & te boekhorst, 2008, p. 20). a library’s ability to demonstrate such characteristics determines whether or not a grant or bursary is awarded. thus, practice (the types of projects or services the library is able to deliver) is arguably determined by the cache of evidence it holds. once in delivery, the project itself is expected to “actively contribute to the building of an evidence base that both supports decision-making and is actively deployed in practice” (hall, 2011, p. 12).   for the sponsoring government department for libraries (currently culture, media and sport), quality is measured in terms of value for money and the extent to which an organization supports government policy and meets the service standards it prescribes for the sector. in newcastle, this manifests as tailoring service delivery and collecting evidence pertaining to “reading and informal learning,” “access to digital skills and services,” and social inclusion (bawden, petuchovaite, and vilar, 2005, p. 459). it also requires active and visible participation in national initiativessuch as the society of chief librarians’universal offers; the go on uk campaign for digital skills; access to research and the reading agency’s books on prescription scheme—all of which help to raise the public profile of the service.   with 2015 approaching, u.k. public libraries are neither comfortable nor secure, and each raft of budget cuts requires further reinvention. in newcastle, at least for the present, the library service remains largely intact, even managing to maintain its customer service excellence accreditation due in part to its conscious placement of the “needs of existing and potential customers . . . at the heart of planning” (d. fay, personal communication, october 28, 2014). in these immensely challenging times, library services must be savvy and stakeholder conscious. they must adapt to survive, and the ways in which they approach the collection and proactive use of evidence to inform practice is key to this survival.   references   bawden, d., petuchovaite, r., & vilar, p. (2005). are we effective? how would we know? new library world, 106(9/10), 454-463. http://dx.doi.org/10.1108/03074800510623137   brettle, a. (2014). evidence of impact [editorial]. evidence based library and information practice, 9(2), 1-2. retrieved from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/22182/16558   brophy, p. (2008). telling the story: qualitative approaches to measuring the performance of emerging library services. performance measurement and metrics, 9(1), 7-17.   cole, b. (2014, june). rich, emotive evidence of impact. cilip update. 42-44.   delong, l. (2008). book review: the evaluation and measurement of library services [review of the book the evaluation and measurement of library services, by j.r. matthews]. the canadian journal of library and information practice and research, 3(1). retrieved from https://journal.lib.uoguelph.ca/index.php/perj/article/view/447/737#.vf_napmswso   grant, m. j., sen, b., & spring, h. (2014, october 27). demonstrating your value: 13 tips for library and information professionals [web log post]. retrieved from http://www.cilip.org.uk/cilip/blog/demonstrating-your-value-13-tips-library-and-information-professionals      hall, h. (2011). project output versus influence in practice: impact as a dimension of research quality. evidence based library and information practice, 6(4), 12-14. retrieved from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/12138/9373   poll, r., & te boekhorst, p. (ed.). (2008). measuring quality: performance measurement in libraries (rev. 2nd ed.). berlin, boston: k. g. saur. retrieved from http://www.degruyter.com/view/product/37583   proctor, k. (2014, october 8). gateshead library staff shouldering the burden of welfare changes. newcastle chronicle. retrieved from http://www.chroniclelive.co.uk/news/north-east-news/gateshead-library-staff-shouldering-burden-7898865   reasons for libraries: false economy. (n.d.). public libraries news. retrieved from http://www.publiclibrariesnews.com/   tenopir, c. (2013). building evidence of the value and impact of library and information services: methods, metrics and roi. evidence based library and information practice, 8(2), 270-274. retrieved from http://ejournals.library.ualberta.ca/index.php/eblip/article/view/19527/15262 microsoft word es_hannigan.doc evidence based library and information practice 2007, 2:2  104 evidence based library and information practice       evidence summary    users’ awareness of electronic books is limited      a review of:  levine‐clark, michael. “electronic book usage: a survey at the university of denver.” portal:  libraries and the academy 6.3 (jul. 2006): 285‐99.     reviewed by:   gale g. hannigan  professor, texas a&m medical sciences library  college station, texas, united states of america  e‐mail: g‐hannigan@tamu.edu      received: 28 february 2007  accepted: 25 april 2007      © 2007 hannigan. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      abstract    objective – to determine if university  library users are aware of electronic books,  and how and why electronic books are used.     design – survey.    setting – university of denver.    subjects – two thousand sixty‐seven  graduate and undergraduate students,  faculty, and staff.    methods – in spring 2005, the university of  denver faculty, and graduate and  undergraduate students were invited to  participate in a survey about awareness and  use of electronic books. a link to the survey  was also posted on the library’s home page  and on the university’s web portal. the 19‐ question survey consisted of 11 questions to  get feedback about electronic books in  general, five questions focused on netlibrary,  and the remaining were demographic  questions. eligibility to win one of two  university bookstore gift certificates  provided incentive to complete the survey.    main results – surveys were completed by  2,067 respondents, including undergraduate  students (30.1%), graduate students (39.1%),  faculty (12.5%), and staff (11.8%). results  were reported by question, broken out by  status (undergraduate students, graduate  students, faculty) and/or by discipline  (business, humanities, nontraditional,  professional, sciences, social sciences), and  presented in tables or in the text. in general,  most respondents (59.1%) were aware that  the library provides access to electronic  books. the library catalog and professors  were the main ways respondents learned  about electronic books. approximately half  mailto:hannigan@tamu.edu http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:2  105 (51.3%) indicated they had used an  electronic book. of those who indicated that  they used electronic books (1,061  respondents), most (72%) had used  electronic books more than once. the main  reasons mentioned for choosing to use an  electronic book included: no print version  available, working from home makes  getting to the library difficult, and searching  text in an electronic book is easier. when  asked about typical use of electronic books,  most respondents indicated they read only a  part of an electronic book; only 7.1% of 1,148  respondents indicated they read the entire  electronic book. in answer to a question  about choosing the print or electronic  version of the same book, 60.7% responded  that they would always or usually use print,  and 21.5% indicated they would always or  usually use electronic. the amount of  material to read, the need to refer to the  material at a later time, and the desire to  annotate or highlight text are all factors that  influence whether users read electronic  books on a computer or pda, or print out  the material. u.s. government publications  and netlibrary were the electronic resources  used the most by survey participants.     conclusion – the results of this survey  suggest the need to market availability of  the library’s electronic books. problems  associated with the use of electronic books  are related to reading large amounts of text  on a computer screen, but a reported benefit  is that searching text in an electronic book is  easier. responses to the survey suggest that  the use of electronic resources may not be  generic, but rather depends on the type of  resource (content) being used. the author  notes that this finding should lead to further  investigation of which items will be  preferred and used in which format.     commentary      as the author points out, libraries spend a  considerable amount of money on electronic  books with very little information about  how they are used. an excellent literature  review illustrates the need for research to  discover this information. unfortunately,  this research is not methodologically sound  enough to contribute much to answer the  questions of how and why users use  electronic books. it does suggest that at  approximately 60%, university users’  awareness of electronic books is not high.  this number may be an overestimate since  as the author points out, “a small but  significant portion” of survey respondents’  comments illustrated confusion between  electronic books and electronic journals  (289).  user surveys are common research  tools in libraries. it is a challenge to develop  good survey questions and get a good  response rate from a representative sample.  this survey targeted the university faculty  and students, but included responses from  staff, which constituted 11.8% of the total  respondents. it is not clear how the staff  data fit into the analysis. a response rate  was not provided, but using the student and  faculty population at the time of the survey  as the denominator and including staff  (which overestimates response), the  response rate was approximately 20% of the  targeted population.  the survey was  voluntary and respondents self‐reported  their demographic information; there was  no mention of verification or tracking of  who completed the survey. this is typical,  but not a rigorous survey implementation  design.     unfortunately, web links to the survey  questions and to some of the analyses are no  longer valid, so information about the  survey itself and results are limited to the  published article. the question asked to  assess awareness of electronic books was  somewhat leading: “are you aware that  penrose library provides access to  electronic books?” the response categories  for the question “how often do you use  electronic books?” were “one time only,”  evidence based library and information practice 2007, 2:2  106 “occasionally,” and “frequently.”  “occasionally” and “frequently” are too  open to various interpretations to be useful  – once a week may be occasionally for one  person and frequently for another (291).    some data are reported by discipline, some  are reported by status, some by both  discipline and status. by the time the author  reported subset percentages, it was easy to  be confused. the stated purpose of the  study was to determine awareness of  electronic books and to discover how and  why they are used and the general  satisfaction with the medium, but the  analysis focused more on who was doing  what with regard to electronic books. since  approximately only 10‐12% of the targeted  population indicated awareness of electronic  books, it is difficult to draw significant,  valid conclusions from this study. there is  evidence, however, of the need for library  promotion of electronic books and of the  need for more research into the cost  effectiveness of delivering content in this  format.       editorial   outcomes assessment in select papers from the 2012 library assessment conference   martha kyrillidou senior director, statistics and service quality programs association of research libraries washington, district of columbia, united states of america email: martha@arl.org      2015 kyrillidou. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/) which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   we continue the tradition of featuring select articles from the library assessment conference in eblip. we first featured a few from the 2010 event (kyrillidou & jaggars, 2013), demonstrating our commitment to open access publishing. we continue in this issue by having five of the six papers presented here go through peer review, with a sixth that focuses on the process of the research project included as commentary since the findings of this work have already been published in another venue.   it is with great pleasure that we present these six articles, some of them with slightly different and more precise titles compared to the original submissions included in the 2012 library assessment conference proceedings (hiller, kyrillidou, pappalardo, self, & yeager, 2012). these articles feature a diverse array of topics and methods, a trend that can be observed over the last ten years that assessment has been documenting through the conference proceedings (hiller, kyrillidou, & oakleaf, 2014), demonstrating that assessment speaks to all parts and processes of a library organization.   other papers presented at the 2012 library assessment conference that we invited to consider for publication in eblip are already in the process of being published in other peer reviewed journals (e.g., plum & franklin, 2015; lupton & davidson, 2015). seeing conference papers successfully published in this and other peer reviewed journals speaks to the quality of the work presented at the conference itself, which boasts ‘effective, sustainable, and practical’ assessment as its tagline.   the feature articles in this issue of eblip speak to the two key aspects of the academic mission: research and teaching. three papers speak to the relation of libraries and research. rawls writes from an institutional perspective, relating expenditures on electronic resources to scholarly productivity. gessner, jaggars, rutner, and tancheva write from the perspective of improving library services for humanities doctoral students, while mcgowan and namachchivaya write from the perspective of sponsored research pursued by library staff and linked to organizational strategies. the other three articles (stemmer and mahan; askew; and donahue) focus on student learning. the authors offer ways to understand and implement different approaches for outcomes assessment, which has become the golden standard in practical and tangible ways for libraries as their future survival is tied to demonstrating the value and benefits they deliver to their constituencies.   rawls’ article brings five years of data related to libraries and universities, as well as citation data from the web of science, to understand the relationship between institutional characteristics and investments and research productivity. he tested a variety of models to examine whether:   the number of journal articles produced by the faculty journal article output of each institution correlated with their libraries’ investment in electronic materials. this inquiry is based on the concept that the speed of access and convenience of use offered by electronic library materials creates efficiencies that should increase research productivity by saving the researcher’s time. thus the expectation is that institutions investing more in electronic materials should generate more journal articles over a given period.   rawls’ analysis captures the emerging effect and relationship of research article production and electronic journal spending and has implications for disciplinary perspectives. electronic journal spending has risen dramatically over the last decade and he documents the relationship between spending levels and scholarly productivity, with a positive and statistically significant correlation. he notes that “expenditures for [electronic] materials have a positive and statistically significant correlation with journal article production,” finding that an increase of 511 additional journal articles produced “from 2005 through 2010 for each additional $1,000,000 spent on electronic materials on average per year.” rawls’s study captures a baseline prototype:   a university that attracts an average of $1 billion per year in revenue, employs 3,500 faculty members, 100 professional librarians, spends $200 million on research, and spends $5 million apiece on both electronic and nonelectronic library materials is predicted to produce 1,801 articles each year.   we hope to see more refinement of these models in future years as well as a deeper understanding of the relationship between library staff and increased article productivity.   gessner, jaggars, rutner and tancheva discusses in detail the extensive collaborative ethnographic research methods cornell and columbia undertook to complete a study that helped improve services for humanities doctoral students. the results of the study have been published elsewhere (see gessner, jaggars, rutner, and tancheva, 2011), so the feature published here outlines in detail the methodological rigor behind the qualitative methods utilized, and this is primarily why this piece appears as a commentary. in particular, this work:   examines the processes taken to design and administer a collaborative ethnographic study of humanities doctoral students within an inter-institutional, collaborative framework. project organization and management, including the creation of instruments and analysis of results across two local research teams and institutional cultures is discussed. effective communications, among and between project teams, and time management were identified as critical factors for success. benefits resulting from the project included an improved understanding of the needs of a key user group, a heightened interest in user assessment and data-driven decision making among staff within the partner organizations, and a deeper engagement with important academic administrators on both campuses.   the study utilized focus groups and in depth interviews and utilized a suite of collaborative tools available to columbia and cornell, and the authors emphasize the importance of clear, flexible and engaging communication. according to the authors, results from this research:   were used on both campuses to improve services and launch new initiatives targeted at this user population. results were used at cornell to plan and implement a pilot immersion program for humanities graduate students and at columbia as impetus to relocate the graduate student teaching center within the library, among several other initiatives at both universities.   namachchivaya and mcgowan present a mixed methods analysis of seven years of sponsored research projects at the university of illinois library at urbana-champaign. they analyzed proposals and funding trends in a quantitative way and provided context based on in-depth interviews with principal investigators. the authors were seeking to understand:   the research trends and themes over that period. the analysis was aimed at identifying future research trends and corresponding support opportunities. goals included developing institutional research themes that intersect with funding priorities; demystifying grant writing and project management through professional development programs, increasing communication about grant successes; and bringing new faculty and academic staff into these processes.   the authors report that their research has been valuable for the library’s institutional practices and its grant writing initiatives, concluding that “the most important outcome of the assessment was that it revealed the need for the library to support grant efforts as an integral component of the research process.”   in stemmer and mahan we see one of the emerging studies that link library use to student outcomes. they explored a) whether library use influences freshman retention, b) whether it is linked to four-year graduate rates, and c) whether it affects the cumulative gpa for freshmen and seniors. in all cases they found positive relations indicating the importance of library as place for freshmen and the importance of developing good study habits as students progress through a four-year higher education program. as the factors that influence student outcomes differ depending on what stage of their program students find themselves, questions are raised of how this information can be used by libraries to offer interventions and improvements in the students’ learning trajectories. the authors have plans to continue this line of inquiry, supplementing a rich set of studies that are taking place in this area.   regarding developing specific intervention services, donahue reports on an evaluation study of a peer2peer service at the university of new hampshire manchester for approximately ten years. this service is a collaboration between the library and the college’s center for academic enrichment (cae). a critical component of this collaboration is the incorporation of peer writing tutors trained in basic library research skills who work side-by-side with instruction librarians in the classroom over the course of a semester. the study uses a mixed methods approach and provides valuable baseline evidence for informing teaching and learning practices. furthermore, it has led to exploring future approaches for deepening the understanding of peer2peer mentoring during a first-year writing course for imparting critical information literacy skills.   finally, askew discusses the implementation of assessment of roaming reference services. she reviews results from a pilot study at florida international university through inputs, outputs, qualitative data, and organizational perspectives offered by the roaming librarians. she emphasizes the need to iteratively improve new services and the critical importance of taking advantage of new mobile technologies to offer roaming services throughout the campus, even outside library buildings.   these curated feature articles speak to the maturation of library assessment in shifting towards outcomes perspectives, from the summative to the formative, by engaging mixed methods, and by addressing the need to relate to research and learning outcomes. learning and research are ultimately intertwined, and depend on past experiences, knowledge, and the perspectives our students and faculty bring as they approach the information rich environments they are constantly exposed to.    references   gessner, c. g., jaggars, d. e., rutner, j., & tancheva, k. (2011). supporting humanities doctoral student success: a collaborative project between cornell university library and columbia university libraries. retrieved from the council on library and information resources website: http://www.clir.org/pubs/ruminations/02cornellcolumbia/report.html/report.pdf   hiller, s., kyrillidou, m., & oakleaf, m. (2014). the library assessment conference – past, present and near future! journal of academic librarianship, 40(3-4), 410-412. http://dx.doi.org/10.1016/j.acalib.2014.05.013   hiller, s., kyrillidou, m., pappalardo, a., self, j., & yeager, a. (2012). proceedings of the 2012 library assessment conference: building effective, sustainable and practical assessment. washington, dc. association of research libraries.   kyrillidou, m., & jaggars, d. (2013). current themes in academic library assessment: select papers from the 2010 library assessment conference. evidence based library and information practice, 8(2), 4-8. http://ejournals.library.ualberta.ca/index.php/eblip/index   lupton, a., & davidson, c. (2013). assessing the value of e-resources to york university faculty using the mines for libraries protocol: an evolving landscape. journal of web librarianship, 7(4), 422-433. http://dx.doi.org/10.1080/19322909.2013.839849   plum, t., & franklin, b. (2015). what is different about e-books?: a mines for libraries® analysis of academic and health sciences research libraries’ e-book usage. portal: libraries and the academy, 15(1), 93-121. http://dx.doi.org/10.1353/pla.2015.0007   news/announcements   eplip 7 conference update      2013. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     the 7th international evidence based library and information practice conference (eblip7) is just months away. make plans to attend the 7th iteration of this successful and innovative conference series, july 15 18, 2013, at the university library, university of saskatchewan, saskatoon, saskatchewan, canada.   why choose eblip7 when there are so many conferences to attend and funding can be tight?   the international evidence based library and information practice community is a vibrant and inventive group of library and information professionals. from lis scholars to practitioners to researchers in many forms, you will find interesting conversation, abundant networking opportunities, and peer reviewed posters and presentations to stimulate and energize your own thinking and practice. eblip7 keynote speakers: denise koufogiannakis, who will open the conference on tuesday, july 16, is a key driver in the international eblip movement. she co-founded the evidence based library and information practice journal and was one of the first librarians to think systematically about eblip and how it can be implemented in libraries. dan gardner, a canadian journalist, author, and speaker, will close the conference on thursday, july 18. he is the author of two acclaimed books: risk and future babble. he has strong opinions and an interest in the evidence based movement. evidence based practice is prevalent in many areas and disciplines. if you are new to it, come and see what it is and (more importantly) what it isn’t. there is a lot of information out there about eblip. you can enhance that with face-to-face conversations with leaders and innovators as well as peers who have the same questions that you do. if travel to saskatoon, saskatchewan isn’t in your bucket list[i], it should be! saskatoon is a beautiful city in the middle of the canadian prairies. the natural river bank, many summer activities and events, and friendly hospitality will make you glad you came. not to mention the elbip7 conference!   registration will open in march 2013. at that time, a preliminary program will be available. for continuing conference information, bookmark the eblip7 website: http://eblip7.library.usask.ca/   you can follow us on twitter: @eblip7 join us on facebook: http://www.facebook.com/groups/eblip7/   [i] bucket list: informal; a number of experiences or achievements that a person hopes to have or accomplish during their lifetime (http://oxforddictionaries.com/). evidence summary   undergraduates may prefer to learn about the library informally   a review of: murphy, j. a. (2014). library learning: undergraduate students’ informal, self-directed, and information sharing strategies. partnership: the canadian journal of library and information practice and research, 9(1), 1-20. https://journal.lib.uoguelph.ca/index.php/perj/index   reviewed by: robin e. miller assistant professor and research & instruction librarian mcintyre library university of wisconsin-eau claire email: millerob@uwec.edu   received: 12 dec. 2014  accepted: 27 jan. 2015      2015 miller. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine undergraduate student approaches to learning about research and to seeking assistance with resources and services offered by the library.   design – three face-to-face focus groups received the same 12 questions to discuss over 90 minutes.   setting – academic library in saskatchewan, canada.   subjects – a total of 14 undergraduate students majoring in a social science or humanities subject area. of these, four subjects were in their second year of undergraduate study, four in their third year, and six in their fourth year. subjects participated in focus groups with other students in their year of study. the researcher recruited subjects through printed advertisements distributed in areas frequented by social science and humanities students. 12 female students and 2 male students participated. 13 participants had attended a library instruction session in the past. subjects were offered pizza, but were not otherwise incentivized to participate.   methods – the researcher and an assistant conducted three focus groups with undergraduate students, eliciting qualitative comments later transcribed and coded manually for analysis. requirements for participation included being engaged in an undergraduate major in the social sciences or humanities, and previous experience using the library. subjects answered open-ended questions about their studies, research activities, use of the library for a variety of tasks, and help seeking preferences.   main results – regardless of year of study, focus group participants reported informal approaches to learning about and conducting research. all participants were confident about using the library’s online resources, and preferred learning about library resources through self-directed practice and trial and error. participants revealed that learning about the library informally was preferable to library instruction. most participants indicated they had sought help from the library at one time or another. participants prized sharing information with classmates, especially through collaboration and social networks, and they valued the expertise of professors, peers, friends, and family when doing research. three factors may influence their choice to consult and exchange information with other trusted advisers outside of the library: convenience, familiarity, and knowledge.   conclusion – findings from this study align with previous findings about student approaches to seeking research assistance. the author reveals that assistance from the library, including library instruction, is less important to focus group participants than the research strategies they have developed informally, including trial and error and information sharing within one’s personal network. the author observes that the informal learning strategies implemented by undergraduates in this study mirror the strategies of adult learners, especially in the workplace. the author suggests that intentional, course integrated library instruction in the early years of undergraduate education would strengthen students’ preferred self-directed learning about research.   commentary   the author elicits a rich discussion of undergraduate library use, painting a picture of undergraduate students engaged in self-directed learning about library resources.   the author administered a focus group questionnaire to three groups of undergraduate students. while the author hoped for more, a total of 14 subjects participated in the study. the researcher and an assistant recorded and transcribed the comments, themes, and observations from the focus group sessions, though methods of coding and analysis were not described. a test of inter-coder reliability would strengthen the evidence.   three themes emerged from the focus groups: students conduct library research through self-directed research and trial and error; they share and exchange information with peers; and “convenience, familiarity and knowledge” (p. 7) may influence whether they seek help from the library or another trusted advisor such as a professor, peer, or family member. the author suggests that the themes discussed represent the majority of focus group participants, and direct quotes from individual subjects further illustrate the themes reported. subjective terms like “some” or “many” are not quantified. in contrast to the themes listed above, subjects expressed less agreement about actual use of the library. consultations with a librarian, service desks, and help guides were some of the ways subjects had sought help in the library, but the majority of subjects did not favour any one method.   social science and humanities students who had previously used the library were specifically recruited to participate in this study. some, but not all, subjects had extensive experience with this particular library. including students who reported using the library at least once may have ensured that data were collected from subjects capable of deciding whether to work independently or to seek help from the library. however, further study with a group of students who reported never using the library would offer more complete insights about undergraduate help-seeking strategies.   the size of this study, and exclusion of both non-library users and students from other disciplines, prevents generalization of the author’s findings. the literature review highlights findings about student preference for working independently, consulting with peers, and using resources that are familiar and convenient. however, the literature review omits large-scale studies about student research practices, such as the ethnographic research in illinois academic libraries (erial) project (duke & asher, 2012), and research from project information literacy, including head’s (2008) study involving humanities and social science majors.   thirteen of the subjects had participated in a library instruction session in the past, although the author does not note whether the social sciences and humanities are major users of library instruction at the institution in question. in the discussion, the author argues for course-integrated library instruction in the early years of undergraduate education. one focus group participant indicated she appreciated library-faculty collaboration (pp. 13-14); however, the author cites no other evidence to support the conclusion, and the main results indicate that subjects did not favour formal library instruction over self-directed methods.   the author also concludes that practitioners could offer self-directed programs like “student peer mentorship, internship, and first year experience programs” (p. 14), though focus group subjects were not questioned about whether they would avail themselves of these opportunities. nonetheless, librarians may be inspired by the author’s research to consider experimenting with information literacy skill-building activities outside the formal classroom.   references   duke, l. m. & asher, a. d. (eds.). (2012). college libraries and student culture: what we now know. chicago: american library association.   head, a. j. (2008). information literacy from the trenches: how do humanities and social science majors conduct academic research? college & research libraries, 69(5), 427-446. http://dx.doi.org/10.5860/crl.69.5.427 evidence summary   reference desk employees need both research knowledge and technical skills for successful reference transactions   a review of: chan, e. k. (2014). analyzing recorded transactions to extrapolate the required knowledge, skills, and abilities of reference desk providers at an urban, academic/public library. journal of library administration, 54(1), 23-32. doi:10.1080/01920836.2014.893113   reviewed by: lisa shen reference librarian and assistant professor newton gresham library, sam houston state university huntsville, tx, united states email: lshen@shsu.edu   received: 2 sep. 2014     accepted: 11 nov. 2014      2014 shen. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine the essential knowledge and skills required by reference positions serving academic and public library patrons.   design – data analysis of recorded reference transactions using author-created categories.   setting – the reference desk of a joint academic and public library in downtown san josé, california.   subjects – a total of 9,683 in-person and phone reference transactions recorded between august 20 and december 29, 2012.   methods – all reference transactions recorded in the tracking software gimlet during the fall 2012 semester were downloaded and analyzed in excel using 17 author-created reference service categories. of the original 13,827 transaction entries, 4,135 were eliminated because the actual reference questions, an optional entry in gimlet, were not recorded. thus these transactions could not be properly categorized for analysis.   main results – the most frequently occurred type of reference transaction (16.6%, or 1,607 out of 9,683) out of the 17 categories was assistance for printing, copying, scanning, and wireless network assistance. the next most regularly recorded categories were catalog searching for non-known items (15.0%) and general research (10.9%), which included formulating research questions and selecting the appropriate resources for searching.   when clustering the 17 reference question categories into 4 broader thematic groups, “research-oriented assistance,” including question categories for catalog searching and general research, emerged as the most common question type (31.7%). technical and equipment assistance (30.8%) was the second most popular category group, followed by facility and policy questions (19.2%), and quick search requests (18.3%).   conclusion – the study findings suggest that successful reference desk transactions would require library employees to master research knowledge as well as technical computer and equipment skills.     commentary   this study uncovered a number of training considerations for reference desk employees’ professional development. unfortunately, flaws in the study design limited the usefulness of the findings. a close examination of the research using the ebl critical appraisal checklist (glynn, 2006) indicated an overall validity (65%) below the accepted threshold of 75%.   one major validity issue concerns the author-created question categories. it is unclear whether these categories were developed based on existing literature, the researcher’s personal experience, or other sources. moreover, the author appeared to be the only coder for categorizing all 9,683 reference questions. such ambiguities in the coding process raise questions about rater bias and reliability of the category assignments.   in addition, while the author should be commended for providing detailed descriptions for every question category, the broader thematic groupings of these categories would have benefited from similarly detailed explanations. for instance, the question category for circulation policies was assigned the theme of “quick searches” (group 3), instead of “policies” (group 1). likewise, transactions for assisting patrons with downloading e-books on supported devices were grouped under “research-oriented assistance” (group 4) instead of “technical/equipment assistance” (group 2). unfortunately, the rationale for these seemingly contradictory assignments was unavailable.   moreover, almost 30% of all recorded transactions were eliminated because the original reference questions were unavailable. it is unclear whether reference staff failed to follow the researcher’s instructions, or proper recording instructions were not provided. in either case, omission of such a significant portion of reference transactions raises concerns about representativeness of the results. also, as the author had noted, the types and numbers of questions sent directly to liaison librarians from academic patrons were also excluded, thus further impacting reliability of the data.   lastly, even though the tracking software gimlet required other metrics for each reference transaction, none of these data was used in the study. some of the data points, such as transaction duration and question format, could have complemented the study results by demonstrating the amount of time reference staff spent addressing different types of questions. likewise, differentiations between academic and public patron transactions could have strengthened the findings, but user type was not recorded.   therefore, despite this study’s unique setting in a hybrid public and academic library, further research is needed to solidify its findings. nonetheless, this article has provided a good starting point for future research in core reference skills and training development.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi: 10.1108/07378830610692154 evidence summary   library fines make a difference in academic library book return behaviour   a review of: sung, j. s., & tolppanen, b. p. (2013). do library fines work?: analysis of the effectiveness of fines on patron’s return behavior at two mid-sized academic libraries. journal of academic librarianship, 39(6), 506-511. http://dx.doi.org/10.1016/j.acalib.2013.08.011   reviewed by: sue f. phelps reference librarian washington state university vancouver library vancouver, washington, united states of america email: asphelps@vancouver.wsu.edu   received: 21 may 2015  accepted: 23 jul. 2015      2015 phelps. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objectives – to quantify library fines and their impact on patron return behaviour.   design – hypothesis testing of data extracted from integrated library systems.   setting – two midsize academic libraries, including one from the pacific, university of hawaii at manoa (uhm), and one from the midwest, eastern illinois university (eiu).   subjects – undergraduates, graduate students, and faculty.   methods – the authors collected data from two midsized universities. the universities have identical integrated library systems, which allowed for uniform data extraction. the authors counted book returns in each population group (undergraduates, graduate students, and faculty) for those books that were returned before and after the due dates with a focus on late fees as the primary variable. the authors tested the following five hypotheses:   hypothesis 1: “there is no difference in return rates before due dates among the uhm patron groups because the fine policy is the same for all patron groups” (p. 507). hypothesis 2: “before 2006, the eiu undergraduate students’ return rates before due dates should be the highest among the three eiu groups because this was the only group which had overdue fines. . . . there should be no difference in the return rates before due dates between eiu faculty and eiu graduate students (both groups had no overdue fines)” (p. 507). hypothesis 3: “eiu graduate students’ return rates before due dates was lower for 2002–2006 than 2007–2011” (p. 507). this hypothesis tests the impact of a change in fine policy that the library implemented in the fall of 2006. hypothesis 4: “uhm undergraduate students’ return rates before due dates is higher than that of eiu undergraduate students” because there is no grace period for uhm undergraduates (p. 507). eiu undergraduate students have a 10-day grace period. hypothesis 5: “uhm faculty’s return rates before due dates is higher than that of eiu faculty” (p. 507). uhm faculty incur overdue fines, but eiu faculty encounter no penalty for overdue materials.   from the integrated library systems, the authors extracted data for the number of books returned before due dates and after overdue notices and for the number of books borrowed by the different populations for the time period starting with fall 2002 and ending with spring 2011. the authors analyzed the data using statistical package for social science (spss) and made comparisons using analysis of variance (anova) expressed with an f-ratio and p-value < 0.01 as the level of significance.   main results – the findings did not support hypotheses 1 or 2. for hypothesis 1, in which fines were the same for undergraduates, graduate students, and faculty, return rates increased with academic level and faculty groups. the rates were 90.4%, 93.9%, and 95.7%, respectively (f = 112.1, p < 0.001). for hypothesis 2, the return rate was 88.8% for undergraduates, 92.6% for graduate students, and 80.1% for faculty. the group difference was small but still statistically significant (f = 171.4, p < 0.001). the graduate students, who were not fined, had significantly higher return rates before due dates than undergraduates who incurred fines. graduate students had higher return rates than faculty, though both groups had no fines.   the data did not support hypotheses 3 and 4. for hypothesis 3, no significant change occurred in return rates before and after imposing fines (f = 5.75, p = .031). for hypothesis 4, the return rates of undergraduates at the university with a grace period showed no statistically significant difference in return rates from those undergraduates with no grace period (f = 4.355, p = .044).   the findings supported hypothesis 5. the return rates indicated a statistically significant difference between faculty with fines for overdue books and those with no fines (f = 1701, p < 0.001). for those hypotheses for which the differences were not significant, the authors cite other variables, including reminders, grace periods, maturity of the borrower, withholding of privileges, fees, and lost book charges, that may contribute to return rates.   conclusions – in answer to the main research question, the authors conclude that “fines indeed make a difference” (p. 511) in patron book return conduct. however, they also note that fines can mar the reputation of the library creating a barrier to access and that courtesy notices and overdue notices are also effective ways to ensure timely return of materials.   commentary   this study provides an excellent example of how librarians strive to make evidence based decisions about fines to ensure timely return of materials, although confounding factors affected the data analysis. with little published on the efficacy of library fines, and even less that is data driven, sung and tolppanen make a valuable contribution to professional library literature.   though the target population of this study is limited to the current users of the library, the circulation data is large enough to generalize to patrons who are not currently borrowing from the libraries. additionally, the study used hypothesis testing, which allows for sample data to be generalized to a larger population (gravetter & wallnau, 2008) as does the use of anova to compare data between population groups.   the author of this review appraised the study using glynn’s critical appraisal checklist (2006). the score was 80% overall, which is within the scale range for validity. the population, data collection, and study design sections rated 100%, 100%, and 100%, respectively. however, the results section rated 33%, indicating that the conclusions should be called into question. the lower score on the conclusions was due to significant confounding variables discussed in the results section that were not accounted for in the conclusions. therefore, the conclusions do not completely reflect the analysis.   though there are strengths to this study, many confounding factors affected interpretation of the data. some of the findings could also be attributed to the differences in patron groups and the differing lengths of loan periods. the grace period studied was not advertised to students; therefore, it was less likely to make a difference in student behaviour. lost book charges and fees also have an impact on return rates, as does the policy that some students are prohibited from registering for the following semester or receiving their diplomas if they have unreturned books or outstanding fines. the differences are not in the fines alone.   the fifth hypothesis, from which the researchers drew their conclusion about fines, shows a significant increase in returns for faculty with fines versus those without; however, in addition to fines, faculty with accumulated fines over $10 lose database access. this is certainly a compelling reason to stay current with library material.   the conclusions would be less ambiguous if the only variable was overdue fines, though this level of homogeneity among library return policies is unlikely. given the policies outlined in this study, librarians could use the detailed information to develop or revise existing policies with these results in mind. other researchers have cited this study, indicating a clear interest in evaluating the validity of fines to manage returns of library materials. the researchers made excellent use of data already available in an integrated library system. with the use of spss to analyze the data, the research design is one well worth replicating.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154   gravetter, f. j., & wallnau, l. b. (2008). essentials of statistics for the behavioral sciences. belmont, ca: wadsworth.   evidence summary   a multiyear curriculum-integrated information literacy program increases the confidence and research skills of nursing students, although not as much as expected   a review of: farrell, a., goosney, j., & hutchens, k. (2013). evaluation of the effectiveness of course integrated library instruction in an undergraduate nursing program. journal of the canadian health libraries association, 34(3), 164-175. doi: 10.5596/c13-061   reviewed by: elaine sullo coordinator, information and instructional services himmelfarb health sciences library the george washington university washington, district of columbia, united states of america email: elainej@gwu.edu   received: 12 nov. 2014 accepted: 27 jan. 2015      2015 sullo. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to evaluate nursing students’ information literacy (il) confidence and competence after a four year implementation of a curriculum-integrated information literacy program.   design – web-based and paper-based information literacy questionnaire.   setting – two schools of nursing in newfoundland, canada.   subjects – 422 undergraduate nursing students.   methods – a 20-item questionnaire was distributed to nursing students at two school of nursing programs. questions were designed to test one or more of the nine learning outcomes set forth for the il program. at one school of nursing, web-based questionnaires were sent via email, while paper-based questionnaires were distributed in class at the second school of nursing.   main results – because response rates were low at one of the nursing schools, these questionnaire responses were not examined, nor were they included in the data analysis. the 422 completed questionnaires were divided into groups based on the number of il sessions attended. the class of 2011 attended two out of three il sessions, the class of 2012 attended all three il sessions, and the classes of 2014 and 2015 did not attend any il classes.   there were 109 responses from class of 2011 students, 98 from class of 2012 students, and 215 from the classes of 2014 and 2015 (98 and 117, respectively). results were reported according to the two main goals of the study – evaluating confidence levels and il skills. regarding confidence level, the class of 2012 felt more prepared than the other classes to begin evidence-informed practice (eip). when asked about their confidence in the ability to perform research tasks, confidence levels rose between first year students and the class of 2011, but decreased between the class of 2011 and class of 2012.   in regard to information literacy skills, when asked to identify appropriate tools for locating scholarly research articles, only 2.8% of the first year students correctly identified two resources. within the class of 2011, 33.9% could correctly identify tools, while 57.1% of the class of 2012 correctly identified appropriate tools. for a series of questions that asked if students could make appropriate decisions about the use of scholarly and popular literature, there was overall improvement between the first year students and fourth year students.   several questions asked students about their understanding of different study types; students in all classes had difficulty with these questions. additionally, there were questions related to identifying components of pico (problem/population, intervention, comparison, outcome), and none of the students in the first year classes or in the 2011 class were able to correctly identify all four parts although 11.2 % of the class of 2012 identified all components correctly. students were asked to create an answerable question using the pico format, and 11.9% of the class of 2011, along with 11.2% from the class of 2012 and 3.7% from the first year class were able to do so.   when asked to identify criteria for evaluating websites, 28.4% of students in the class of 2011 and 30.6% of the class of 2012 were able to list three correct criteria. only 8.3% of the first year students answered this question correctly. finally, students were given a research question and asked to select the best search statement; those who answered correctly included 28.4% of the class of 2011, 27.6% from the class of 2012, and 20.8% of first year students.   conclusions – the authors concluded that overall, il skills improved from first to fourth year, although not as much as the anecdotal information from nursing faculty would indicate. this ongoing, four year study has prompted discussion on ways to improve the current il component of the undergraduate nursing degree curriculum, and changes will be implemented in a future version of this curriculum.   commentary   in their literature review, the authors note that the norm in il seems to be the “one-shot” library instruction session; research has shown that this type of session does not have a lasting impact on students, as they are not likely to retain skills taught in the session. further research has supported the idea that il instruction should be at the point of need, should be similar to real world situations, and should be integrated throughout the curriculum. while there are numerous published studies evaluating the il skills of nursing students, very few have been done in canada, and none were found that evaluate students’ skills with regard to evidence informed practice (eip). the two nursing programs in the current study incorporate searching strategies for eip into their il curriculum; as such, the authors’ research is unique and possibly the first to be conducted on this distinct aspect of il.   this study was evaluated using the reliant instrument (koufogiannakis, booth, & brettle, 2006). the significant strengths of this research related to study design include the clearly explained research methodology, the researchers’ strategies to ensure efficacy of the research instrument, the measurement of il skills over multiple years, and the detailed description of the il curriculum and learning outcomes, with inclusion of the bn (bachelor of nursing) il questionnaire as appendix a.   regarding educational content, the authors describe the il topics taught during each academic year and list the learning outcomes for the bn il program and how these map to the questions on the evaluative questionnaire. results of the study are clearly explained and the researchers provide the percentage as well as actual number of responses for each question component of the evaluation. however, no statistical analysis was performed on results; conclusions were solely based on descriptive statistics.   because acceptable response rates for data analysis were only acquired at one site, the research results and conclusions were based on the 422 responses from a single site. the main reason for this issue was the mode of questionnaire delivery. due to unexpected changes in the timing of the il sessions, evaluations were delivered electronically at one site, which resulted in a poor response rate. at the other site, the questionnaires were distributed in class in hard copy, resulting in a much higher response rate.   even though the study’s research results are specific to the course-integrated il at one of the two nursing program sites, the information gained can be useful to libraries and librarians that work with nursing programs and who are thinking about curriculum integrated il. librarians struggle with justifying value in “one-shot” instructional sessions and may face resistance when suggesting cumulative, curriculum integrated il to faculty members, as these sessions take time away from the course content, and may be difficult to include based on a class’s schedule. the most valuable aspect of this research is that it provides evidence that this type of instruction is effective and has long-term benefits to nursing students.    the article notes that anecdotal evidence from nursing faculty suggests that il sessions have a positive impact on student papers and also results in more confident students. this information could certainly be considered as an area for further scholarly inquiry.   references   koufogiannakis, d., booth, a., & brettle, a. (2006). reliant: reader’s guide to the literature on interventions addressing the need of education and training. library and information research, 30(94), 44-51. retrieved from http://www.lirgjournal.org.uk/lir/ojs/index.php/lir/article/view/271   article   a mixed methods approach to assessing roaming reference services   consuella askew director john cotton dana library rutgers university–newark newark, new jersey, united states of america email: consuella.askew@rutgers.edu   received: 15 feb. 2015    accepted: 13 may 2015        2015 askew. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract objective – the objectives of this research are threefold: a) to assess the students’ perception of the roaming service at the point of service; b) to assess the librarians’ perception of the service; and, c) to solicit librarian feedback and observations on their roaming experience and perceived user reactions. ultimately, this data was used to inform and identify best practices for the improvement of the roaming service.   methods – a combination of quantitative and qualitative survey methodologies were used to collect data regarding patron and librarian service perceptions. patrons and librarians were asked to complete a survey at the conclusion of each reference transaction. in addition at the end of the first semester of the implementation, librarians were asked to provide feedback on the overall program by responding to five open-ended questions.   results – the findings indicate that our students typically seek assistance from the librarians once a term (58%), but the majority (71%) indicated that they would seek a librarian’s assistance more frequently, if one were available on the various floors of the library. overall, our users indicated that they were “satisfied” (36%) to “very satisfied” (43%) with the roaming service. librarian responses indicate overall enthusiasm and positive feelings about the program, but cautioned that additional enhancements are needed to ensure the continued development and effectiveness of the service.   conclusion – overall, patrons were satisfied with the service delivered by the roaming reference librarian. the roaming librarians also provided positive feedback regarding the delivery of service. data collected from both groups is also in agreement on two major program aspects needing improvement: marketing of the service and a means by which to easily identify the roaming librarian. introduction   reference service delivery has centered on the physical service desk since the late nineteenth century (miles, 2013, p. 323). however, even pioneering library thinkers such as samuel s. green saw the need to decentralize reference service delivery and untether the reference librarian from the desk. in one of his classic publications concerning patron–librarian relations, green asserts: “one of the best means of making a library popular is to mingle freely with its users, and help them in every way.” (pena & green, 2006, p. 164). over the years, reference service delivery has evolved in tandem with emerging information and communications technology. reference librarians have expanded their reach beyond the desk by interacting with patrons using multiple modalities such as by phone, fax, email and now via the web, text and sms. as noted by askew and ball (2014): “the use of technology has empowered reference librarians to move away from reference ‘as place’ services and enabled them to provide focused service at point of need” (p. 119). today, mobile technologies such as ipads, cell phones, smartphones, and laptops are being employed successfully to deploy roaming or roving services in public and academic libraries to provide reference services to the patrons where they are.   such is the case at florida international university (fiu) libraries. to better accommodate the information needs of the students in a library building with severely limited available seating, the information & research services librarians instituted a roaming reference service. in the literature, the terms roaming and roving have been used interchangeably when referring to reference services physically delivered beyond the desk. as a professional preference, the fiu reference librarians preferred to be referred to as “roamers” rather than “rovers”. therefore the term “roam” and its variant forms will be used throughout this article when referring to the fiu roaming service.   fiu libraries roaming reference   the fiu libraries system is comprised of two libraries, the steven and dorothea green library located on the modesto a. maidique campus and the glenn hubert library situated on the biscayne bay campus – approximately 30 miles apart. despite the different geographic locations, the libraries share common service challenges that are inherent to primarily commuter-based populations. reference services provided across both libraries include the traditional desk, in-depth one-on-one research consultations, phone, email, and growing chat and texting services. despite this array of service options, results from a previous internal library survey attempting to discern users’ preferred mode of interaction revealed users still preferred face-to-face interaction. these types of interactions have become increasingly difficult as seating in the libraries – particularly, in the green library – has become even scarcer. students are reluctant to leave their seats to seek the assistance of a librarian at the reference desk for fear of not being able to reclaim their seat upon their return and have taken to twitter to express their concerns about the lack of seating in the libraries. the number of in-house initiated chat sessions serves as further evidence of their reluctance to leave their study space. growing user expectations for ubiquitous service and the continued evolution of information and communications technologies has dictated the need for increased flexibility and mobility in the delivery of the libraries’ reference services. the “ask-us-anywhere” roaming reference pilot ipad program was developed in an attempt to respond to user needs and expectations of the libraries’ reference services, with the added benefit of providing at-point-of-need service.   a total of 12 volunteers for the pilot ipad roaming service were recruited from across the libraries: 5 at the hubert library and 7 at the green library. in order to participate in the program, librarians agreed to roam for two hours each weekday during the peak hours, between 10:00 a.m. to 2:00 p.m. on the day(s) of their choosing. a shared calendar was created to facilitate the scheduling of the service across libraries. due to limited weekend staffing, roaming was not provided on saturdays and sundays. roamers were encouraged to roam within or outside of the library buildings and were expected to represent the libraries at student and faculty orientations and other events across the university. in order to receive their ipads, participants were required to attend a training session to familiarize themselves with the ipad and recommended software applications prior to their first day of roaming.   the ipad 2 was selected as the device of choice for the roamers primarily because of its easy mobility. the device allowed full access to the web, the online catalogue and other library resources including, research guides, library faqs, databases, and the libraries website. funding for the ipads was secured through a student technology fee grant awarded by the university. given the nature of the ipads as personal devices, as well as the scheduling difficulties that would arise from sharing devices across campuses and busy schedules, the program coordinators decided to assign each librarian their own ipad. therefore, grant funds were used to purchase 12 ipad 2s, 12 wireless keyboards, and 12 otterbox protective cases.   to supplement the training sessions, an ask us anywhere: ipad roving/roaming libguide was created. the workshop, as well as the guide, covered the basics of device usage, the setup of their individual accounts, network and wireless access, installation of apps, bookmark suggestions, and how to collect service assessment data. guidelines for best practices on how to approach patrons and what to do when roaming were also addressed using roaming etiquette and techniques compiled from researching the library literature and business-related literature (askew & ball, 2013).   aim   the objectives of this research were threefold: a) to assess the students’ perception of roaming service at the point of service; b) to assess the librarians’ perception of the service at point-of-need; and, c) to solicit librarian feedback and observations on their roaming experience and user reactions. ultimately, the findings were used to inform and identify best practices for the improvement of the roaming service.   review of the literature   in its earliest form, as described by samuel green, roaming reference consisted of librarians who would walk around the library to identify and assist patrons in need. however, the proliferation of electronic and web-based information impeded the librarian’s ability to easily access information while away from the reference desk – and their computer work station. kramer (1996) notes that this challenge was resolved as libraries increased their numbers of stand-alone opac terminals, which were strategically scattered throughout the library buildings. as electronic information became mobile in the first decade of the new millennium, tablet pcs were incorporated into the delivery of roaming reference services with mixed results (hibner, 2005; smith & pietraszewski, 2004). next came the integration of smartphones, particularly the iphone, but problems with connectivity, screen size, non-standardization, formatting, and functionality prevented the early generations of this technology from being adopted on a long term basis (murray, 2008). however, apple’s introduction of the ipad in 2010 provided a mobile and lightweight technology that librarians were quick to adopt for their roaming services. given its relatively short lifespan, a review of the literature published during the years 2010-2015, reveals little has been published about utilizing ipads for roaming reference services substantiating the findings of maloney and wells (2012) who noted in their literature review, that they found only a “handful of scholarly titles, with most focusing on roving reference” (p. 12).   perhaps, the most thorough ipad roaming reference study to date was conducted by mccabe and macdonald (2011) at the university of northern british columbia. using roaming reference as a way to address their declining reference statistics, their librarians staffed the service for six months, during which time they collected transaction data for query type, location and approach. two iterations of the roaming service were implemented: one integrated with the traditional service desk duties and the other as a standalone service. the latter iteration required librarians to provide roaming service in addition to their reference desk hours. patrons were asked to complete an optional e-questionnaire at the end of the roaming transaction to collect data related to past use of reference services, provide thoughts on the service and to find out whether or not the service made them more apt to contact a librarian for help.   the library realized an overall increase of 228 reference questions with the roaming service; the majority of which (67%) were research-related. the results indicated that the roving reference service with ipads proved to be very successful when librarians were only assigned to rove, but less successful when they combined desk hours with their roving duties. they found that the integration of roaming and reference desk services resulted in a 56% decline in the total number of roaming reference questions from the previous iteration where roaming was implemented in addition to desk hours. although they indicated that they did collect patron data, an analysis of that patron data was not presented.   the youth services (ys) division at the boise main public library received a library services and technology act (lsta) just-in-time grant that allowed them to acquire four ipad2s for nine staff members to provide a roving reference service (may, 2011). the intended goal was to increase staff interaction with patrons, by giving them tools that allowed them to move away from the reference desk. although they kept their traditional reference desk, the use of the desk was minimized as they added more roving personnel.   as a result of the service, they were able to have multiple librarians assisting multiple patrons at the same time using the staff features of the catalogue. they also learned that their web-based public access catalogue was not optimized to work with mobile technologies. other complaints such as ergonomic issues with long-term use of the device, the lack of ease when switching back and forth between applications and cutting and pasting were also common. based on their experience, it was recommended that each librarian should have her or his own device to allow for the personalization of the applications and other customization (may, 2011, p. 14). unfortunately, may did not provide any assessment data regarding this program.   at the university of warwick library, widdows (2011) recounts their roving reference experiences using the mobile phone and their trial of the ipad as a potential roving tool. the warwick library does not have a traditional reference service desk, but utilizes helpdesks, which deal primarily with circulation and account questions as a means of proving query “triage”. the helpdesks refer patrons to “specialists staff”, or rovers, as needed. the rovers also provide backup support to the helpdesks during peak times.   their ipad trial lasted one week (35 service hours). fifty-six of the total 230 helpdesks queries were handled by the rovers and 26 of these required the use of the ipad. widdows noted the major challenge with using the ipad was the lack of a phone feature which prohibited the rovers from contacting a specialist for more complex queries. as with the boise library, the warwick librarians also ran into problems accessing the full features of their web-based catalogue on the ipad. although widdows states that they collected data on their roaming program, other than the few transaction statistics shared above, there was no other data presented to illustrate an assessment of the users’ or the rovers’ perspectives about the program.   at southern illinois university-carbondale, morris library, three ipads were integrated into an existing roaming program (lotts & graves, 2011). nine reference librarians shared usage of the ipads, which were checked out in shifts. the benefits of using the ipads included the virtual librarian being mobile while staffing the virtual reference service and the multi-functionality of the ipad which was ideal for reference, enabling access to the online catalogue, reference tools, and serving as an ebook reader. the drawbacks noted by the authors presented some surprises. the literature typically reflects that roaming librarians tend to prefer the lighter, more mobile ipad, to the laptop. however, at this library, the librarians reported feeling “uncomfortable” with the ipad as a replacement for the laptop. in agreement with may’s recommendation, the authors thought that each roamer should have his or her own ipad to minimize the need for continual account management and allowing individual to customization for their specific needs. lotts and graves did not present any transactional or assessment data of any kind, they explained the omission in their “next steps and the future” section of the article, by saying that assessment and usage data will be compiled and analyzed as part of their next steps in determining how the library moves forward with their roaming service (p. 220).   librarians at the albin o. kuhn library & gallery collected two semesters of data about their ipad roaming service, which operated for four hours per week in predefined campus locations (gadsby & qian, 2012). the roaming locations were identified through observing traffic patterns in their 24-hour library study space, the commuter lounge, the university center and academic department offices.   using transaction data for 60 queries, they determined that more than 75% of the service users were students, the busy times of the week were tuesday through thursday from 2:00 p.m. – 4:00 p.m., and more than half of the questions they received were library-related. although they mention anecdotal feedback they received from their campus community, there was no attempt to formalize a data collection effort to capture and analyze this qualitative data. furthermore, there was no mention of future assessment efforts.   the roaming programs identified in the literature shared a number of commonalities across library types such as: reference librarians being able to provide services beyond the reference desk; the provision of just-in-time service; the ability to access web-based library resources away from the desk; and, library staff being able to access multiple instances of their online catalogue in order to assist large crowds of users (may, 2011; mccabe & mcdonald, 2011; widdows, 2011). the statistics collection and tracking for these ipad roaming programs vary widely by method and scope. the noted challenges of these programs included using mobile devices to access full functionality of the online catalogue, unstable wifi connectivity, and statistics recording (may, 2011; mccabe & mcdonald, 2011; widdows, 2011).   the literature indicates that the experimenting libraries – academic and public – have had overall positive experiences with integrating the ipad into their roaming services. however, it also reveals that very little assessment data has been collected on ipad roaming programs. while three of the five articles discussed above present mostly transactional or usage data, none provided any type of assessment data – empirical or otherwise – to represent program effectiveness, user satisfaction or feedback. this study attempts to fill this existing gap in the literature and create a foundation upon which to build assessment techniques for roaming services using mobile devices.   methodology   surveys were used to collect data from the user and the librarian immediately after the roaming transaction was completed. the survey instruments were created using qualtrics, a web-based survey tool licensed by the university and were bookmarked on the roamers’ ipads for easy access. in order to encourage participation, the instruments were designed to be very brief; the user survey consisted of four items and the librarian survey had two items. the survey items were piloted by a small group of faculty and students before they were put to use. to allay any concerns about privacy, users were advised that their responses were confidential and that once they clicked on the survey submit button, all responses would be recorded and disappear before the ipad was handed back to the librarian. after the user completed the survey, the librarian would then complete the corresponding librarian survey for that transaction. the data collected from both surveys reside behind a firewall on a secure university server.   the roaming service coordinators collected feedback about the program from the librarians via email asking them to respond to five questions concerning the service implementation, user reception, suggestions for improvement and an open-ended question for any additional comments they may have had about the program.   results   the survey data collection period lasted 52 days (approximately 10 weeks) for a total of 208 service hours during the beginning of fall 2011. the roaming service and data collection efforts were conducted during the libraries’ peak hours between 10:00 a.m. 2:00 p.m., monday through friday. during that time the reference librarians responded to a total of 2,850 (n) queries via our virtual/mobile services that include chat/im, sms/text, telephone, email, and phone. reported roaming reference transactions totalled 168 queries (n=168), which represents 5.9% of the total number of these virtual/mobile transactions.   quantitative results   a deeper analysis of transaction data recorded in our libanswers system provided us with useful information not only about the program, but also about our library users. the data show that our roamers were most often inside the library (89%), when a transaction occurred. the majority (79%) of the service users were undergraduates. the nature of the roaming queries were most likely to be directional/informational (73%), followed by research-related (20%) and least likely to be technology-related (7%). the overwhelming majority (88%) of the transactions took between one to ten minutes to complete.   the student surveys (n=15) completed upon the conclusion of a transaction, provided insight to user behaviour and their satisfaction levels with the service. when asked why they were in the library on that day, the largest percentage (33%) of users responded that they were there to check out a book or reserve an item. the second most frequent response was that they were in the library to “study by themselves” (27%) or to “research an assignment for a class or class project” (27%). the third highest response showed that the reason users were in the library that day was to check out an electronic device (21%) such as a laptop, ipad, kindle etc. (see figure 1)   figure 2 shows the majority of the respondents indicated that they asked a librarian for assistance about once a month.   the third survey item collected data regarding user satisfaction with elements of the roaming program using a likert type satisfaction scale (see table 1) with 5=very satisfied and 1=very dissatisfied. in particular, we wanted to know if the roaming librarians were friendly and approachable, if they were easily identifiable and, of course, if the user received the help they needed. since there were no negative responses, only the positive responses are represented in the table. forty-seven percent of the respondents indicated that they were “very satisfied” and the librarian who assisted them was approachable and friendly; however, 20% of the respondents indicated they were “neutral”. over half of the respondents (60%) indicated satisfaction with the ease by which they could recognize the librarian as a library employee and with the help that they received. however, only a third (33%) of the respondents indicated being “very satisfied” with the ease by which they could recognize the librarian as a library employee and 7% gave “neutral” rating to this same item indicating a program need.     figure 1 purpose of library visit. percentages do not equal 100%, since users were asked to check all responses that applied to them.     figure 2 how often user asks for librarian assistance     the last survey item asked if the respondent would be more willing to ask for assistance if a librarian were available on the various floors of the library, to which 73% responded “yes”. the low number of user responses prevents us from gathering any meaningful information from a cross tabulation of the responses to this item with survey item #2 regarding their frequency of asking assistance, to find out if there is a relationship between the respondents who tended to ask a librarian for help more often and those who would be more likely to seek assistance from a librarian posted on the various floors of the library throughout the day.   the responses to the librarian surveys (n=23) provided insight into their behaviour while roaming. the librarians indicated that they typically approached the student (74%). there seemed to be two most common roaming locations between both campuses: the second floor of green library (35%) and the third floor of hubert library (30%). (see figure 3)     table 1 user level of service satisfaction question very satisfied satisfied neutral librarian was approachable and friendly 47% 33% 20% librarian was easily recognized as a library employee 33% 60% 7% i got the help i needed 40% 60% 0%     figure 3 roamer frequented locations     the librarians’ response to the survey item asking them to rank how they felt the user’s level of satisfaction was with their assistance indicated that a little over half (52%) felt their user was “satisfied” with the services received, a few indicated their users were “very satisfied” (22%) with their assistance.   qualitative results   while the quantitative data primarily focused on the users’ behaviours, the qualitative data does the same for the librarians who roamed. the qualitative data collected from all 12 librarians provided useful suggestions for changes to the services from the people on the front line interacting with the patrons. as the quantitative data showed, roamers indicated a preference for roaming in one of two places in and around the green and hubert libraries. surprisingly the comments for the favourite spot in the green library, the third floor, differed from the recorded transaction locations, which mostly took place on the second floor of the library where the reference desk was located. the green library librarians indicated that they liked to roam on the third and seventh floors, as these floors have no service desk. since the hubert library has fewer floors than the green library, the librarians roamed in the library, as well as in the nearby academic buildings and the student center where the students tend to congregate. one of hubert library librarians shared that they roamed   …through the library and around the wuc [wolf university center]. sometimes through ac1 [academic center 1]…because the library is too small and i often find i get more questions outside of [sic] library.   the challenges identified from the fiu experiences were unique compared to those found in the literature and included excessive noise levels, extreme temperatures in certain locations and poor recognition or visibility of the service. one librarian commented on feeling a “little intrusive” when roaming a floor where the students are quietly studying saying:   i must admit that sometimes, when it is very quiet and students are busily engaged, i feel a little intrusive and somewhat like a floor walker.     figure 4 roamers' rating of user's service satisfaction level. using the same likert type satisfaction scale as with the previous items, there were no negative responses recorded and these are therefore not represented in this figure.     although at the end of the comment, the concluding sentiment was that perhaps it was “just my hang up, of course.” another librarian expressed the difficulty of having students feel comfortable with approaching them for help stating:   the most challenging aspect so far has been having the students approach us for help. you can usually find students who need help if you ask them, but they will not approach us themselves.   a number of free applications were suggested and recommended for use by the roamers during the roaming service training session and the roamers were taught how to install and use these apps on their ipad. most of the roamers report that rather than using the apps, they used bookmarks more frequently instead. one of the more ambitious and tech savvy roamers indicated using applications such as “prezi viewer, dropbox (most often), and adobe reader” as well.   the majority of the roaming librarians agreed that the service needed better publicity and marketing to raise the students’ awareness of the service and help them easily identify roamers when they needed one. as one librarian commented, the service needed to be more “high profile”. however it became clear through other comments that along with the high profile there was a need to implement a “consistent schedule”.   when asked to look into the future and share their vision of our roaming service one to two years from now, all but one roamer indicated that they saw this service existing alongside the traditional reference desk as opposed to a standalone service. in comment after comment, it was clearly and strongly expressed that the traditional reference desk should continue to be a point of service for reference. such assertions included the following:   i believe the desk will always be needed   as a traditionalist, i like the idea of having a reference desk. i think people need to identify a specific place where they can go for help.   roaming should not replace the reference desk: it’s an extra way to help people.   however, one librarian saw things a bit differently:   i see reference increasingly decentralized, online, ubiquitous, and continuous….   when asked to provide any additional comments they had about the service, they unanimously presented overall positive and enthusiastic feelings about their service experience:   i’ve enjoyed it quite a bit, and believe this and online help are closer to the future of reference services than sitting at a desk.   there is great potential with this service. we just have to keep tweaking.   the students are always very happy when they receive help right where they are.   discussion   the reference transaction data recorded in libanswers showed that the respondents who received assistance from a roamer were more likely to be an undergraduate student and were in the library to check out a book or reserve material. this indicates that they were more likely to interact with library staff at the access services desk(s) than with those stationed at the reference desk. this also means they were less likely to need to seek out a reference librarian for research assistance. when users were provided assistance by a roamer, it was the roaming librarian who approached the student to initiate the reference transaction more often than not.  student responses concerning their recognition of the roamers as a library employee validated the librarians’ suggestions regarding the need to improve the identification of librarians while away from the reference desk and to improve the service publicity and marketing strategies.   although library personnel have nametags they are not required to wear them. the “ask me” tag attached to the lanyards worn by the roamers tended to hang lower than the line of sight and was therefore easily overlooked by potential users. alternatives discussed included creating a button, wearing hats, or wearing other outerwear that would clearly show the “ask me” logo to encourage users to approach the roamers for assistance. the program publicity consisted of an announcement on the libraries’ website, social network venues and advertising the service on the libraries’ internal digital signage displays. the roamers agreed that more should be done to raise the visibility of the program. in addition to the above, ideas included creating a more attractive and engaging sign for the libraries’ digital display, highlighting this service more prominently on the libraries’ homepage as well as promoting the service in the student newspaper.   given the students’ reluctance to ask a librarian for help, it was encouraging to see users respond that they were most often satisfied with the help they received from the roamers and with their overall experience. while most (80%) of the respondents indicated that they were “very satisfied” and “satisfied” with the librarian being approachable and friendly, 20% responded “neutral” to this item. these responses may suggest that roamers be more aware of their body language and facial expressions when approached by a student, or when approaching them. what was most encouraging was that the respondents indicated they would be more likely to seek assistance from a librarian if one were available on the various floors of the library. this indicates that the roaming service has high impact potential and signifies a need to redefine the program service strategy. as several roamers noted, the service needs to be provided on a more consistent schedule and perhaps in conjunction with the reference service desk schedule.   all of the above factors, along with the abbreviated service hours, most likely contributed to the low response rate to the quantitative surveys. while the data presented in this article may not be generalizable to other libraries, it does serve as an indicator for students’ receptiveness and potential use of a fully implemented roaming service by the fiu libraries. overall, the data indicates that the fiu libraries’ roaming service fulfilled a need and that students would use the service if were offered as part of a suite of reference services.   based on the librarians’ survey responses it is noteworthy that the gl librarians, unlike the hl librarians, preferred to roam on the floor where the reference desk is located. especially so, since the green library has eight floors six of them providing open study spaces – whereas, the hubert library only has three floors. a comparison of the service data between the roamers and the users presented an interesting revelation. the users reported being much more satisfied with the service they received than the librarians perceived them to be. this suggests that as service professionals, librarians set a higher bar for service delivery for themselves, than is actually expected by the patrons.   although there were a number of common challenges cited in the literature about providing an ipad roaming service, very few of these challenges were mirrored by the data collected from the fiu roamers. the challenges experienced by the librarians were unique to the fiu libraries and included excessive noise levels, extreme temperatures in certain locations, and poor recognition or visibility of the service. the latter challenge was further exemplified by the statistics indicating that the librarian most often initiated the roaming transactions.   there was quite of bit of time spent on identifying appropriate and relevant ipad applications for the service along with the appropriate training for their use. however, the majority of the librarians’ responses revealed that they preferred to use bookmarks instead of the apps. this preference bears further investigation to determine why that was the case.   a subsequent iteration of the roaming service model implemented in the following academic year which integrated the service with the reference desk as was suggested after the pilot was considered unsuccessful. of particular concern with the new model was determining an easy and reliable method of communication (i.e., realtime chat, sms/text, facetime, etc.) between the user and the librarian at the reference desk so that a roamer can be efficiently dispatched. in the spring of 2014, the roaming service was placed on hiatus until the information & research services departments can identify and come to a mutually agreed-upon solution.   conclusion   roaming reference is not new in academic libraries and the integration of mobile technologies has provided even more opportunity for academic librarians to become “unchained” from the traditional desk to meet their users at the point-of-need. as reference services become more decentralized and personalized, researching the effect of roaming services may be valuable to inform the overall quality of service as perceived by the user. askew and ball (2013) identify a need for further research to determine to what extent does culture, language or gender impact a library user’s willingness and comfort level to approach a librarian for help. they also state more research is needed to determine how these same factors affect librarians’ comfort level when approaching users. when focusing on the technologies employed in roaming reference services such as ipads, there is need to determine what functionalities, features and apps are most necessary or useful when responding to queries at the point-of-need.   there are always two sides to every story. in addition to gathering data from our patrons, there is also a need to gather data from roaming librarians (staff) in a more formal way. askew and ball (2013) note: “…academic libraries should consider not so much the ‘what’ we do, as illustrated by the traditional reference transactional data collected, but should also incorporate data collection to describe who we serve, how we serve them, and where we serve them” (p. 98). conversely, we should also take into account what services our users tell us they want, along with how and where they want to receive them. the two may not always be in agreement. in order to accomplish this in a comprehensive fashion necessitates using assessment methods and measures looking from the outside in, by obtaining data not only about the patron, but also about the librarian to capture and reveal the complete story.   references   askew, c., & ball, m. (2013). telling the whole story: a mixed methods approach to assessing roaming reference services. in s. hiller, m. kyrillidou, a. pappaloardo, j. self, & a. yeager (eds). proceedings of the 2012 library assessment conference building effective, sustainable, practical assessment. association of research libraries library assessment conference, charlottesville, va. (pp. 91-101). washington, dc: association of research libraries. retrieved from http://libraryassessment.org/bm~doc/proceedings-lac-2012.pdf   askew, c., & ball, m. (2014). transforming reference services: more than meets the eye. in c. forbes & j. bowers, (eds.). rethinking reference services in academic libraries, (pp. 117-134). lanham, md: rowman & littlefield.   gadsby j. & qian, s. (2012). using an ipad to redefine roving reference service in an academic library. library hi tech news, 29(4), 1-5. doi: http://dx.doi.org/10.1108/07419051211249446   hibner, h. (2005). the wireless librarian: using tablet pcs for ultimate reference and customer service: a case study. library hi tech news, 22(5), 19-22. doi: http://dx.doi.org/10.1108/07419050510613819   kramer, e. h. (1996). why roving reference: a case study in a small academic library. reference services review, 24(3), 67-80. doi:  http://dx.doi.org/10.1108/eb049290   lotts m., & graves s. (2011). using the ipad for reference service: librarians go mobile,” college & research library news, 72(4), 217220.   maloney, m. m., & wells, v. a. (2012). “ipads to enhance user engagement during reference interactions. library technology reports, 48(8), 11-16.   may, f. (2011). roving reference, ipad-style. the idaho librarian, 61(2): retrieved from https://theidaholibrarian.wordpress.com   mccabe, k. m., & macdonald, j. r. (2011). roaming reference: reinvigorating reference through point of need service. partnership: the canadian journal of library & information practice & research, 6(2), 1-15. retrieved from https://journal.lib.uoguelph.ca/index.php/perj/index#.vvzvm-dbliu   miles, d. b. (2013). shall we get rid of the reference desk?” reference & user services quarterly, 52(4), 320-33.  http://dx.doi.org/10.5860/rusq.52n4.320   murray, d. c. (2008). “ireference: using apple’s iphone as a reference tool. the reference librarian, 49(2), 167-170. http://dx.doi.org/10.1080/02763870802101419   pena, d. s., & green, s. s. (2006). personal relations between librarians and readers. journal of access services, 4(1/2), 157-167. http://dx.doi.org/10.1300/j204v04n01_12   smith, m. m., & pietraszewski, b. a. (2004). enabling the roving reference librarian: wireless access with tablet pcs. reference services review, 32(3), 249-255.  http://dx.doi.org/10.1108/00907320410553650   widdows, k. (2011). mobile technology for mobile staff: roving enquiry support. multimedia information & technology, 37(2), 12-15. retrieved from http://www.cilip.org.uk/multimedia-information-and-technology-group/mmit-journal-0     microsoft word art_goddard1.doc evidence based library and information practice 2007, 2:1  48 evidence based library and information practice     feature article    getting to the source: a survey of quantitative data sources available to the everyday  librarian:  part 1: web server log analysis       lisa goddard  emerging services librarian/division head for systems  memorial university of newfoundland libraries  st. john’s, newfoundland, canada  email: lgoddard@mun.ca      received : 01 december 2006  accepted : 07 february 2007      © 2007 goddard. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    abstract    this is the first part of a two‐part article that provides a survey of data sources which are  likely to be immediately available to the typical practitioner who wishes to engage in  statistical analysis of collections and services within his or her own library.  part i outlines  the data elements which can be extracted from web server logs, and discusses web log  analysis tools. part ii looks at logs, reports, and data sources from proxy servers, resource  vendors, link resolvers, federated search engines, institutional repositories, electronic  reference services, and the integrated library system.     introduction    so you’re a librarian who has decided to  embrace evidence based decision‐making.  your reasons might be to justify a purchase  decision, to measure your organization’s  effectiveness, to identify more efficient ways  of organizing your library’s limited  resources, or to provide improved services  to your patrons. finding a place to start this  undertaking may seem to be an  overwhelming task. however, there is an  enormous amount of data about patron  activity available for your review and  analysis logged on servers in your library  every day. each of these servers produces  reams of transaction information in the form  of text files and databases, as do those of  your remote resource providers. every link  you follow, every search strategy you  develop, every online form you complete,  and every byte you download is recorded  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:1  49 and stored among billions of lines of similar  data on servers all over the world. in  addition to all this data, computing systems  operate on high capacity networks and  utilise powerful computers capable of  correlating, processing, and organising  much of this data into neatly formatted  reports for human consumption.     there are several requirements to  harnessing this data, and one of the first is  simply knowing what kinds of information  may be available and how they can be  accessed. server logs are usually protected  behind firewalls and strict authentication  mechanisms, and reporting tools are often  hidden in restricted administration  interfaces. this article is intended to help  identify some of the data sources and tools  that can aid in planning and decision‐ making.      web server log analysis    all web‐based electronic resources and  services are made available through web  servers.  in this context ‘web server’ refers  to a software application such as microsoft  internet information server or apache  http server. the web server operates by  accepting http requests from a browser  (typically in the form of a url) and  providing an http response (typically an  html document) to the requester. each  web server application has the ability to log  each of the requests it receives, and the  responses that were made to each request.  these web server logs are usually stored on  the server as plain text files. web server logs  provide a rich source of quantitative data for  any librarian who wishes to gather  information about usage of electronic  services or resources.     because almost all electronic resources  produce web server logs, it is useful to look  at the elements of these logs in greater detail.  the first section of this article will outline  the kinds of information that can be gleaned  from web server logs, discuss several web  log analysis tools to help librarians  aggregate web log data for user‐friendly  reports, and provide some caveats about  interpreting web server log data.     what’s in a web server log?    each time a web page, image, or object on a  site is accessed through a browser, a record  of the transaction is written to the log. the  server administrator determines the amount  of information written to the log and will  sometimes choose to log less information  about each request in order to prevent log  files from growing too large. a library’s  server logs may have more or less data,  depending on the log format chosen by the  administrator.     one of the critical aspects of web log  analysis is to ensure that logs are collecting  the kind of data that will be required to  make evidence based decisions at a later  time. if data is collected in order to  understand patron needs and to create a  more usable and visible site, then it is  necessary to determine which statistics can  be used to measure those needs. you must  know ahead of time what kinds of questions  you hope to answer with web log analysis,  and carefully choose the log elements to  provide sufficient and accurate measures.     web log data elements    there are nine types of basic web log data  that may be useful to librarians when  conducting quantitative analysis of  electronic service and resource usage:  • date/time  • requested item  • url query  • referrer  • client ip address  • host name  • unique session id  evidence based library and information practice 2007, 2:1  50 • client side user agent  • client side username    this section will describe each type and will  provide examples of each from raw log data.     date/time: the year, month, day, hour,  minute, and second that a request is made.    2006‐11‐04 21:47:33  [16/may/2006:00:18:41 ‐0230]    the date/time stamp is a basic element  included in all standard log formats. it helps  to determine the times of year, days of the  week, or hours of the day that an electronic  resource or service is most heavily used.  this element will identify increasing or  declining patterns of use over time for the  entire site or for a specific page, resource, or  object. the date/time stamp is also used in  any report that attempts to gauge the length  of a patron session or the length of time a  patron spent looking at a particular web  page.     requested item: the url of the requested  item.    /guides/howto/index.php  /qeii/cns/photos/cnsphoto0108002.jpg    identifying the requested item’s url allows  one to see which resources, objects, or  services are being accessed most often by  patrons. it is used to generate top n analysis  and to provide counts of hits on specific  pages or objects.     url query: the portion of the request that  appears after the question‐mark (?) in a  dynamic url (uniform resource locator).    /eindex/dbsearchresults.asp?subhead=en vironmental%20science    /eindex/alphasearchresults.asp?searchtex t=w  /viewnews.php?item=265    /query.html?col=spidert&la=en&qt=+dictio nary+of+newfoundland+english    /cgi‐ bin/docitemview.exe?cisoroot=/newfo undlandquarterly&cisoptr=602    /?genre=article&isbn=&issn=00332917&title =psychological+medicine&volume=36&iss ue=8&date=20060801&atitle=the+temporal +relationship+of+the+onsets+of+alcohol+de pendence+and+major+depression%3a+usi ng+a+genetically+informative+study+desig n.&aulast=kuo%2c+pohsiu&spage=1153&s id=ebsco:psycinfo    if users have the option to search against a  back‐end database, the query portion of the  url contains the search term entered or the  search option selected from a menu.  this is  important if an e‐index or e‐journal title  search box is present on the page. it would  also be helpful to analyse queries entered  into the library’s ‘site search’ engine.   resolvers rely on the openurl standard  and full citation information for a requested  resource in the query portion of each url  generated.     referrer: the site and page that referred a  visitor to the site.    http://www.library.mun.ca/guides/howto/ write_book_review.php    http://images.google.com/imgres?imgurl=h ttp://www.library.mun.ca/hsl/bates/ch17p4 90b.jpg&imgrefurl=http://www.library.mu n.ca/hsl/docs/bates.php&h=370&w=400&sz =38&tbnid=oqadkkwsz3muxm:&tbnh=1 15&tbnw=124&hl=en&prev=/images%3fq %3dmusculoskeletal%2bsystem%26svnu m%3d10%26hl%3den%26lr%3d&frame=s mall    http://www.library.mun.ca/guides/howto/ http://images.google.com/imgres?imgurl=h ttp://www.library.mun.ca/hsl/bates/ch17p4 http://www.library.mu evidence based library and information practice 2007, 2:1  51 http://www.google.com/search?hl=en&lr=& q=sample+annotated+bibliographies&btn g=search    http://web.ebscohost.com/ehost/results?vid =42&hid=123&sid=60138425‐1342‐4032‐ 811c‐7121f3daf2e1%40sessionmgr102    it is useful to know the referring page to  determine the methods by which users  discover resources on the web site ‐‐  whether they are following links from  within the site, from other sites, or from  search engine results.  referrer data is also  useful to measure the impact of web site  metadata initiatives and search engine  optimization (seo) projects.     client ip address:  the unique ip address of  the computer making the request.     134.153.184.170  66.249.72.4    the client ip address is included by default  in all standard log formats. the ip address  helps to identify each individual computer  using the web site, and it can be used to  track repeat visitors. along with date/time  stamps, the ip address can help to  determine the path a user has taken through  a site and the resources viewed during that  visit. some of the difficulties involved in  using ip addresses to track unique user  sessions are discussed below (“interpreting  general summary statistics: terms and  definitions”).    host name:  the computer host name and  domain to which an ip address belongs.     beluga.library.mun.ca  wiley‐411‐2130.roadrunner.nf.net  crawl‐66‐249‐64‐54.googlebot.com    if a web server is configured to perform  reverse domain name server (dns)  lookups, it will automatically translate each  requesting ip address into a full dns host  name in the log. if, for example, the ip  address ‘134.153.184.70’ appeared in the log  file, the web server could perform a reverse  dns look‐up on the fly. the server could  also record the host name of the requesting  computer, which in this case would be  ‘<proxy1.library.mun.ca>.’     knowing the domain suffix of each user’s  internet service provider (isp) makes it  possible to run reports that may provide  information about a client’s organisational  membership and geography based on the  domain to which the ip is registered.  reverse dns look‐ups can slow down the  performance of a busy web server. the  server administrator will need to determine  whether or not it is advisable to log this  information. many web log analysis  packages will perform dns look‐ups  against ip addresses when running their  reports, permitting domain name analysis  even, if this information does not exist in the  log file.     unique session id: uniquely identifies  each client session.    phpsessid=8aa4a615a20382e917731ffc8c6 e6bd5    ip addresses are an unreliable means to use  to identify individual visitors. dynamic  scripting languages such as php can be set  to automatically generate a unique id for a  user session which can be propagated in  each url selected by the user during that  session. this provides an easy way to track a  user’s path through the site during a session.  unique session ids are useful for click‐path  analysis, for generating statistics about the  duration of each user session, and for  counting the number of resources viewed  during each session. cookies are another  method frequently used for tracking  individual sessions. more information on  the use of cookies is presented below under  http://www.google.com/search?hl=en&lr=& http://web.ebscohost.com/ehost/results?vid evidence based library and information practice 2007, 2:1  52 the heading: “interpreting general  summary statistics: terms and definitions”.    client side user agent: records  information about the client’s browser type,  version, operating system, and language    mozilla/4.0+(compatible;+msie+6.0;+wind ows+nt+5.1;+infopath.1)    mozilla/5.0+(windows;+u;+windows+nt+ 5.1;+enus;+rv:1.8.0.1)+gecko/20060111+fire fox/1.5.0.1    mozilla/5.0+(x11;+u;+linux+i686;+en‐ us;+rv:1.7.13)+gecko/20060418+fedora/1.0. 8‐1.1.fc4+firefox/1.0.8    mozilla/5.0+(macintosh;+u;+ppc+mac+os +x;+en)+applewebkit/312.8+(khtml,+lik e+gecko)+safari/312    the client side user agent provides  information on which operating systems,  browser types, and versions are used by  patrons. this information can be helpful  when making decisions about site  functionality and design. it also indicates  whether patrons frequently access the site  using small screen devices such as mobile  phones or handheld computers.     client side username: authenticated  username entered into a name and  password dialog when some portion of the  site is restricted, such as    lgoddard    sites that require authentication can also log  usernames, so resource requests can be  tracked according to the username of the  individual making a specific request. if a site  does not require a login, then the username  field in the log will contain a dash (‐)  indicating that username information is not  available. most libraries ask users to login  only to those services that require personal  identification, (e.g., personal account  information or licensed resources). as a  result, this field is rarely available as a  general statistical measure.     what’s not in a web server log?    although web server log analysis can  provide a great deal of quantitative  information about web site usage, there are  also some very real limitations to the  available data:    • external links ‐ web logs track requests  only for resources that reside on the  web server. many library web sites  contain links that point to external sites,  such as those of e‐journal and e‐index  providers, online reference sources, and  useful sites available elsewhere on the  web. when a patron clicks one of these  links, that selection is not recorded in  your web log, but in the log of the  target server. as a result, web log  analysis cannot determine which  external links or resources patrons are  accessing from the library’s web site.  there are other strategies to help track  use of external links on your web site.  several are examined later in this paper  under the heading: “where did they  go?”    • user profile information – unless a  user login is required, web logs contain  no information that allows the  identification of a particular individual  who has visited the site. web logs  cannot analyze usage according to  personal characteristics such as age,  gender, or affiliation.    • qualitative data – it is difficult to draw  conclusions about a patron’s reasons for  visiting a site. the log data will not help  determine whether the resources found  there met his or her information needs  or how the data was used.   evidence based library and information practice 2007, 2:1  53 • cached pages – most web browsers  have their own cache where they store  and serve frequently viewed pages. if  the browser replies to a user request by  returning a page cached on the user’s  hard drive, the web server log will have  no record for that request. this makes it  difficult to obtain accurate data about  the number of times pages are being  viewed by patrons.     tools and guidelines for the analysis of  web server logs    web logs contain a great deal of useful  information about resource use, but this  information is presented in enourmous  plain text files with thousands of entries for  any given day. every single object returned  is recorded as a separate hit. for example, a  single web page with nine individual  embedded images will generate ten entries  in the web log – one entry for the page, and  an additional entry for each of the images. .  ‘server‐side includes’ are a way of pulling  images, side‐bars, menus and other common  elements into pages as they load. if ‘server‐ side includes’ are used in web site design  then each of these items also generates a log  entry when a page is viewed  a single user  session can generate hundreds or thousands  of individual lines in a log file. web logs can  contain millions of lines and achieve file  sizes well over 500 mb.    very large text files are extremely difficult to  manipulate. it is often impossible to read log  files in text editors (e.g., notepad, wordpad,  or ms word), as these applications are not  optimised for huge files. readers with  programming experience may be able to  write scripts to extract pertinent data  elements from log files. perl, a general‐ purpose programming language originally  developed for text manipulation, is excellent  for parsing and manipulating log files.  however, there are many commercial and  freeshare software packages available to  extract information from web server logs for  those without this expertise.    a web log analyser is a piece of software  that parses the information from web logs,  and uses it to generate different types of  reports that may be delivered as html  pages and charts, in text files, or in excel  spreadsheets, depending on user preference  and the software package. two of the more  popular commercial log analysis packages  are webtrends  (<http://www.webtrends.com/>) and  clicktracks (<http://www.clicktracks.com/>).     there are also several excellent open source  solutions released under the gnu general  public license and freely available from the  internet, including:  • analog: http://www.analog.cx/  • webalizer:  http://www.mrunix.net/webalizer/  • awstats:  http://awstats.sourceforge.net/    one of the drawbacks of web log analyzer  software is that it requires access to logs  from the server. this can be difficult if your  library is a branch of a large library or multi‐ departmental system with central it  support providing web hosting. librarians  wishing to perform detailed or custom web  log analysis should consult with their web  site hosts about log formats and the  availability of log data. some hosts will offer  web‐based reporting systems. work with  the server administrator to be sure that the  reports will be configured in a format useful  to answer your questions    at memorial university of newfoundland  (mun) the web server is housed in the  library, and the commercial webtrends  software package is used to generate web  page statistical reports. the following report  examples are all generated from webtrends,  although the features shown here are  common to many web log analysis  http://www.webtrends.com/ http://www.clicktracks.com/ http://www.analog.cx/ http://www.mrunix.net/webalizer/ http://awstats.sourceforge.net/ evidence based library and information practice 2007, 2:1  54 applications. note that the data contained in  the following tables and reports may have  been truncated or altered. the data  provided does not necessarily represent  actual activity for any mun library service  and is included only for the purpose of  example.      interpreting general summary statistics:  terms and definitions    it is not difficult to generate numbers about  web site usage to add to annual reports or  to requests for increased funding. it is  somewhat more difficult to create actionable  metrics for the library web site. a ‘metric’ is  any type of measurement used to gauge  some quantifiable component of an  organization’s performance. a ‘key  performance indicator’ is a metric tied to an  objective set by the organisation. web log  analysers provide numbers – lots of them.  they do not, however, indicate which of  those numbers is meaningful in a particular   environment, nor do they help to set  objectives for the improvement of electronic  services. understanding the way in which   this data is generated can help to develop  key performance indicators centered on the  goals of your own organisation.     notice that the report in table 1 contains  several levels of information about  site activity. the first section, ‘hits,’  measures every single resource requested  from the server. this includes requests for  images, menus, and style sheets that may be  embedded in any given page. this number  is usually artificially inflated and should not  be taken as an indicator of pages viewed.  the element labelled ‘home page’  inaccurately reports no activity, because the  server administrator failed to include the  url of the library home page when  configuring the reporting software. the  second section contains two separate  measures: ‘page views’ and ‘document  views.’ these elements are a much closer  approximation of actual resource usage on  the site. in this case ‘page views’ counts hits  on all documents, including forms and  dynamic pages.      hits  entire site (successful)  19,364,751    average per day  212,799    home page  0  page views  page views  832,949    average per day  9,153    average per unique visitor  4    document views  305,827  visits  visits  530,214    average per day  5,826    average visit length  0:18:04    median visit length  0:00:01    international visits  56.14%    visits of unknown origin  4.00%    visits from united states  39.84%    visits referred by search engines  128,744  visitors  unique visitors  178,065    visitors who visited once  139,072    visitors who visited more than once  38,993    table 1. general statistics report (jan. – mar.  2006). generated from webtrends analysis suite v.7.  evidence based library and information practice 2007, 2:1  55 supporting graphics and other non‐page  files are not counted. webtrends considers  any page retreived with either a post  command or a get command with a  question‐mark (‘?’) to be a dynamic page.  the ‘document views’ element is defined  according to locally‐configurable criteria  established by the server administrator.  typically ‘document views’ counts hits on  complete html pages, pdf documents,  and other static content retrieved with a  get command. it does not include form  submissions, nor does it count dynamic  pages that require user input (e.g., a search  query) to construct results from a back‐end  database. both ‘page’ and ‘document’ views  are susceptible to undercounting, because  many web browsers cache frequently  viewed pages. these pages are then served  back to the user from his or her own hard  drive, rather than being requested from the  server, so no evidence of cached views exists  in the server log.     the third section indicates statistics for  ‘visits.’ in this report group the log analyser  attempts to isolate specific session  information. a visit may be comprised of a  single page view or of many pages viewed  by an individual from a particular ip  address without an idle time of more than  30 minutes occuring between views. the  idle time can be configured according to  preference.     the fourth section, ‘visitors,’ records the  number of single or repeat visits according  to the ip address of the visiting computer.  be careful when using an ip address as a  measure of individual visitors to your site. if,  for example, there are a large number of  public stations from which people can  access your web‐based resources, these  station ips will appear frequently in the logs.   statistical analysis based on ip address will  show these as repeat visitors, even though a  different person may have been using that  station for each session.   the opposite problem is true of users who  access the site from a public isp such as  aol or roadrunner. in most cases home  broadband providers do not assign a static  ip to each computer on the network, but  assign an ip from a pool each time a  connection is made. in some cases the isp  will channel many users through a proxy  server, and the ip appearing in your log will  therefore be that of the proxy, and not of the  end user computer. this means that a user  may visit from home on a regular basis, but  cannot be identified as a repeat visitor,  because the ip trace left in the server log will  be different each time that person visits your  site.     many web site administrators implement  tracking cookies to solve this latter problem.  a cookie is a small file written to a patron’s  computer during a visit to a web site. web  sites sometimes use cookies to personalize  the information seen by a user during a  given internet session, or during subsequent  sessions. in order to track repeat visitors  who do not have static ips, a persistent  cookie can be served from the site. a  persistent cookie is one that never expires  and stays in the patron’s cookie folder until  he or she chooses to delete it. each time that  patron visits the site, the value held in the  cookie is incremented to indicate a repeat  visit. the cookie information is sent back to  the server, and it can be logged to help  identify unique visitors who may not have a  unique or stable ip address.     cookies are often perceived as a privacy  threat, and browsers can be configured to  not accept cookies of various types. first  party cookies that do not collect personal  information (such as those left by the library  server when a patron views the site) are  accepted by most web browsers. cookie  information can be read only by the server  that sets the cookie, so there is no risk that a  cookie from the library web server can be  read by other web sites.   evidence based library and information practice 2007, 2:1  56 who are they? geographic location and  organisational membership of site  visitors    the information about ‘international visits’  in the third section of table 1 should be  examined more closely, as this particular  report has been configured to assume that  visits from countries other than the united  states are international. the administrator  should configure these reports to indicate  the country where the library is situated.  further geographic breakdowns are  available to identify activity from specific  countries, states, or provinces.    table 2 demonstrates more detailed reports  where we see a high level of activity from  various canadian provinces, as one might  expect at a canadian library. how do we  account for the large amount of traffic out of  viriginia? the ‘most active cities’ report in  table 3 shows that the usage is  predominantly from herndon, virginia.    the high‐speed internet provider  roadrunner has its company headquarters  in herndon, virginia. these log hits occur  because internet traffic from across north  america is channeled through herndon.  other reports can help to futher untangle  the provenance of site visitors.    most active countries    countries  visits  1  canada  245,429  2  united states  211,256  3  australia  7,998  4  united kingdom  5,684  5  europe  5,446  6  france  3,214  7  china  2,138  8  netherlands  1,988  9  united kingdom  1,985  10  germany  1,930    north american states &provinces   state  visits  1  newfoundland  167,543  2  ontario  56,215  3  virginia  38,073  4  california  14,269  5  new brunswick  3,152  6  nova scotia  2,981  7  alberta  2,943  8  massachusetts  2,915  9  new york  2,868  10  british columbia  2,590  table 2.  most active countries and north american states and provinces reports (jan. – mar.,  2006). generated from webtrends analysis suite v. 7.      evidence based library and information practice 2007, 2:1  57   city, state, country  visits  1  st johns, newfoundland, canada  162,458  2  ottawa, ontario, canada  45,841  3  herndon, virginia, united states  35,868  4  mtn view, california, united states  4,533  5  toronto, ontario, canada  4,530  6  marina del rey, california, united states  4,349  7  milton, australia  4,068  8  halifax, nova scotia, canada  2,221  9  st john, new brunswick, canada  1,783  10  calgary, alberta, canada  1,652    table 3. most active cities report  (jan. – mar., 2006). generated from webtrends analysis suite v. 7.        organizations  hits  % of total hits  visits  1  library.mun.ca  4,872,809  25.16%  44,563  2  aliant.net  1,972,426  10.18%  24,617  3  nf.net  1,483,644  7.66%  14,016  4  med.mun.ca  619,718  3.20%  15,022  5  pcglabs.mun.ca  547,373  2.82%  9,813  6  wsr.mun.ca  321,116  1.65%  7,949  7  googlebot.com  293,935  1.51%  16,737  8  comcast.net  238,617  1.23%  6,557  9  rogers.com  235,181  1.21%  5,096  10  wst.mun.ca  168,942  0.87%  4,165    table 4. most active organizations report  (jan.‐ mar., 2006). generated from webtrends analysis suite v.  7.    by the translation of the ip addresses of  visitors into dns names, log analysers can  help group them according to the domain  extensions of their isps (table 4). in this case,  it is apparent that many of the visitors to the  library’s web site come from within the  ‘mun.ca’ domain, and are, therefore,  members of the university community.  more granular information is also available,  as demonstrated by table 5, where details of  site activity are listed according to the host  name of each frequent visitor:  another issue becomes clear from this  report: many of the site’s most frequent  visitors are not people at all, but are web  bots and spiders that crawl the site to  provide indexing information to web search  engines. examples from table 5 include  “crawler.bloglines.com” and  “msnbot.msn.com”. while these visitors are  a welcome means to increase site visibility  in search engines, these hits should not be  included in use data reports. spiders often  repeat visits to the same sites, and they look  at many pages very quickly, generating a  huge number of hits on each web site. the  report on visits and hits from spiders (table  6) can help to isolate spider activity, so one  can be sure not to include this data in patron  statistics.    when do they come? time dimensions  of web server activity   evidence based library and information practice 2007, 2:1  58   visitor  visits  hits  % of total hits  1  vhost.ucs.mun.ca  2,134  6,362  0.03%  2  crawler.bloglines.com  2,120  4,187  0.02%  3  altair.ucs.mun.ca  1,474  1,596  0.00%  4  msnbot.msn.com  1,074  33,482  0.17%  5  bentley.mha.mun.ca  945  1,203  0.00%  6  wiley‐411‐2130.roadrunner.nf.net  843  1,165  0.00%  7  egspd42141.ask.com  705  15,315  0.07%  8  hslcircbehind.med.mun.ca  696  54,905  0.28%  9  bastion.hccsj.nf.ca  629  90,674  0.46%  10  med‐sur1831b.med.mun.ca  596  1,040  0.00%  11  public01.med.mun.ca  589  14,360  0.07%  12  lib‐mason.library.mun.ca  583  18,291  0.09%    table 5. top visitors report  (jan. – mar., 2006). generated from webtrends analysis suite v. 7.      spider  visits  hits  % of total hits  1  mozilla/5.0 (compatible; yahoo! slurp)  48,073  52,454  12.01%  2  googlebot  15,822  101,427  23.24%  3  mozilla/5.0 (compatible; yahoo! slurp china)  1,013  1,044  0.23%  4  mozilla/5.0 (compatible; googlebot/2.1l)  894  177,967  40.77%  5  gigabot  777  2,455  0.56%  6  yahoo‐mmcrawler  240  2,901  0.66%  7  http:  205  502  0.11%  8  baiduspider (http:  187  188  0.04%  9  gsa‐crawler (enterprise; gix‐03519)  141  286  0.06%  10  mozilla/4.0 (compatible; msie crawler)  94  1,917  0.43%    …  …  …  …    total for spiders above   68,454  435,090  99.69%    table 6. visiting spiders report  (jan. – mar., 2006). generated from webtrends analysis suite v. 7.      this report (table 7) aggregates activity  according to the date/time stamp in the  web log. this information may be used to  inform decisions on chat and e‐mail  reference service availability, library and  service desk hours of operation,  opportunities for server maintenance, and  windows for running processor intensive  tasks, such as back‐ups or large analysis  reports like those shown here. each of the  temporal elements can be examined to show  activity summaries by week, day of the  week, and hour of the day.       date/time analysis, combined with session  information based on a user’s ip address,  sessionid, or cookie information, can  provide data on the length of user visits and  the duration of page views. many hits will  be less than a minute in length, if bot  activity has not been filtered out of the  report, or if many public or staff stations are  set to show the library home page as their  browser default. short session times are also  normal when patrons are using one specific  page on the site to gain access to external  resources (e.g., electronic indexes and  journals). if the web site, library catalogue,  evidence based library and information practice 2007, 2:1  59                          summary of activity for report period    average number of visits per day on weekdays  6,388  average number of hits per day on weekdays  239,871  average number of visits per weekend  8,843  average number of hits per weekend  290,239  most active day of the week  tue  least active day of the week  sat  most active date  16‐mar‐06  number of hits on most active date  426,614  least active date  1‐jan‐06  number of hits on least active date  57,945  most active hour of the day  13:00‐13:59  least active hour of the day  04:00‐04:59    table 7. activity by time summary report  (jan. – mar., 2006). generated from webtrends analysis suite v.  7.    metasearch interface, and resolver interface  reside on different web servers, then  patrons using the web site as an entry point  to other tools will also show very short visit  durations, even though they are continuing  to use library resources during their session.     if the library houses a large number of  public or kiosk type stations, there may be  many extremely long sessions. stations used  by many individuals can skew use data, as  several consecutive patrons can use the  same machine to view the web site. unless  an idle‐time period of 30 minutes occurs  between these sessions, the report is  configured to assume that that all of the  activity comprises a single visit.      another metric used by commercial web  sites is the number of pages viewed per  session, which indicates whether visiting  patrons usually look at only a single page on  the site or follow a path through many  pages. unlike commercial enterprises,  which usually want to keep people on their  own sites as long as possible, library web  sites are often designed to function as  portals to other sites and services.  for  libraries short session durations and low  numbers of page views per sessions may be  common and are not necessarily undesirable.  metrics designed around user session time  and the number of pages viewed in a  session are, therefore, difficult to interpret.  did the user spend a lot of time on the site  because he couldn’t find the correct  information, or because he found a useful  link to another site? did he view many  pages because navigation of the site was  difficult for him, or because there were  many pages relevant to his information need?    what do they want? resources accessed,  search strategies    there are data sources that help assess the  reasons that patrons are visiting the library  web site, including reports on the resources  or groups of resources accessed most often.  top n analysis can be run against the whole  site, or against resources grouped together  in specific directories.     table 8 represents usage on the directory  containing the library’s user guides.  this report helps to identify those guides  that are in heavy demand and the average  amount of time spent viewing each page.  the ‘time viewed’ variable may help to  determine whether or not a patron has evidence based library and information practice 2007, 2:1  60   table 8. top n pages report ‐ guides  (jan. – mar., 2006). generated from webtrends analysis suite v. 7.      found the resource useful. in the case of  guides, it is likely that a view of 5 seconds  means that the document did not meet the  user’s information need, but a view of 30  seconds could indicate that the patron found  the document useful enough to print for  reading at a later date. of course, just  because a page is open in the user’s browser,  it is still not possible to determine if she is  engaged with the information on the page.    in addition to specific page view data, it is  possible to generate a top directories report,  which provides a useful snapshot of the  areas of the site accessed most frequently.  the first entry in table 9 indicates hits on  the root directory, the main directory where  the homepage for the site is located. the  root directory is represented by a slash (/).     just knowing that patrons are viewing pages  or collections doesn’t provide enough  information to determine whether or not the  information that they find there has met  their needs. one clear indicator that a patron  found a resource useful is the number of  times that a document is downloaded,       meaning that it has been saved to the  patron’s local computer. presumably a  patron who has bothered to download a  document from your web site has found a  resource that she considers relevant and  useful.     frequently accessed files may give an  indication of a patron’s information needs,  but another important source of data is  contained in search strategies, where the  patron’s own words express his information  needs. when a search engine refers a patron  to a site, the query portion of the referrer  url contains the words that were entered  in the patron’s original search.  this data  can be aggregated in a number of ways,  including a basic report that shows the  search phrases that most commonly lead a  user to the library site.    referrals from search engines on the web,  however, do not necessarily indicate how  patrons are using the web site. search  engine users did not choose the library web  page as a starting point for their research,  even if that site meets their information  need. strategies entered into a ‘site search’     top pages: guides         pages  views  % total views  avg. time viewed  1  /guides/howto/annotated_bibl.php  1,054  12.10%  00:01:53  2  /guides/howto/  write_book_review.php  450  5.17%  00:02:35  3  /guides/howto/turabian.php  312  3.58%  00:04:17  4  /guides/howto/apa.php  229  2.63%  00:02:38  5  /guides/howto/mla.php  134  1.53%  00:04:58  6  /guides/howto/evaluation.php  126  1.44%  00:02:00  7  /guides/howto/tips.php  155  1.78%  00:02:38  8  /guides/howto/offcampus.php  188  2.15%  00:03:07  9  /guides/howto/primary.php  98  1.12%  00:02:52  10  /guides/howto/reserve.php  130  1.49%  00:01:51  evidence based library and information practice 2007, 2:1  61                             table 9. top directories report  (jan – march, 2006). generated from webtrends analysis suite v. 7.    table 10.  most downloaded files report  (jan. – mar., 2006). generated from webtrends analysis suite v.  7.        phrases  phrases found  % of total  how to write an annotated  bibliography  1,926  1.49%  how to write a book review  1,711  1.32%  newfoundland map  1,042  0.80%  mun library  937  0.72%  how to write citations  651  0.50%  how to write bibliographies  642  0.49%  queen elizabeth ii  539  0.41%  free scores  376  0.29%  newfoundland newspapers  333  0.25%  male physical exam  296  0.22%  ship drawings  248  0.19%    table 11. top search phrases report  (jan .– mar., 2006). generated from webtrends analysis suite v. 7.        path to directory  visits  hits  1  /  272,117  918,972  2  /guides/howto  48,991  135,371  3  /eindex  45,288  296,900  4  /hsl/bates  40,551  339,321  5  /eindex/images  37,000  559,921  6  /qeii/cns  32,233  131,857  7  /hsl/images  32,186  126,310  8  /hsl/docs  29,334  61,378  9  /hsl  27,517  112,816  10  /swgc/music  22,366  180,713      file  no. downloads  1  http://www.library.mun.ca/guides/howto/apa.pdf  2,199  2  http://www.library.mun.ca/guides/howto/mla.pdf  1,709  3  http://www.library.mun.ca/guides/howto/annotated_bibl.pdf  1,570  4  http://www.library.mun.ca/qeii/holidays.pdf  945  5  http://www.library.mun.ca/guides/howto/music_citations.pdf  717  6  http://www.library.mun.ca/qeii/cns/waterpoweredsawmills.pdf  880  7  http://www.library.mun.ca/guides/howto/turabian.pdf  419  8  http://www.library.mun.ca/guides/howto/offcampus.pdf  394  9  http://www.library.mun.ca/hsl/ guides/cinahlsearchguide.pdf  424  10  http://www.library.mun.ca/qeii/maps/1‐138b.pdf  472  http://www.library.mun.ca/guides/howto/apa.pdf http://www.library.mun.ca/guides/howto/mla.pdf http://www.library.mun.ca/guides/howto/annotated_bibl.pdf http://www.library.mun.ca/qeii/holidays.pdf http://www.library.mun.ca/guides/howto/music_citations.pdf http://www.library.mun.ca/qeii/cns/waterpoweredsawmills.pdf http://www.library.mun.ca/guides/howto/turabian.pdf http://www.library.mun.ca/guides/howto/offcampus.pdf http://www.library.mun.ca/hsl/ http://www.library.mun.ca/qeii/maps/1%e2%80%90138b.pdf evidence based library and information practice 2007, 2:1  62       table 12. top search our site requests (jan. – mar., 2006). generated from web server logs using a custom  perl script.          table 13. top electronic index subject selections report. (jan – mar. 2006). generated from web server logs  using a custom perl script.            evidence based library and information practice 2007, 2:1  63 box on the library web page provide more  information about the kinds of information  being sought by patrons, and may also help  to identify pages that need to be more  visible on the site. table 12 is an example of  a perl report, with aggregated search  strategies from the library web site.    any dynamic search box on a web site that  searches a local backend database will also  provide query strings in the web server log.  table 13 is a perl‐based report that indicates  the most frequent selections from the subject  and a‐z search menus on the library’s  electronic index site.    where did they come from? referrers     if the server administrator has configured a  web server to log referring urls, it is  possible to see which sites are driving traffic  to the library web site. referrer data can be  loosely grouped according to the referring  site (table 14).     this report helps to determine the number  of patrons who came to the site through  search engines (e.g., google in line 4), those  who were following links found on other  web sites that may be part of the   institutional web presence (e.g.,  <http://thecommons.mun.ca> in line 12), and  those who came to the site through pages  hosted outside of your own institution (e.g.,  myspace in line 8). if few visitors arrive via  major search engines, then it is possible to  experiment with the use of different ‘meta‐ tag’ keywords and descriptions in page  headers to try to increase the site’s visibility  in search engines.    it is also possible to generate reports that  show the full address of the referring pages.  as patrons follow click‐paths through a  web site, the referrer information indicates  which pages are generating a lot of internal  clicks to other resources on the web site.     table 14 indicates a large number of visits  with ‘no referrer.’ a referrer is only  registered in the server log when a user  clicks through to the site using a link.  referrers are not registered when a patron  types the web site url into a browser,  when the page is loaded as the browser  home page, or if a patron uses a bookmark  to access the site. in some cases the referrer       site  visits  1  no referrer  230,388  2  http://www.library.mun.ca  86,325  3  http://www.mun.ca  57,071  4  http://www.google.com  38,622  5  http://www.google.ca  23,225  6  http://images.google.com  7,694  7  http://www.swgc.mun.ca  6,853  8  http://profile.myspace.com  6,771  9  http://www.google.co.uk  4,158  10  http://qe2a‐proxy.mun.ca  3,903  11  http://images.google.ca  2,185  12  http://thecommons.mun.ca  1,734    table 14. top referring sites report (jan. ‐ mar., 2006). generated from webtrends analysis  suite v. 7.    http://thecommons.mun.ca http://www.library.mun.ca http://www.mun.ca http://www.google.com http://www.google.ca http://images.google.com http://www.swgc.mun.ca http://profile.myspace.com http://www.google.co.uk http://qe2a%e2%80%90proxy.mun.ca http://images.google.ca http://thecommons.mun.ca evidence based library and information practice 2007, 2:1  64 information is also lost if the link from  another web page opens in a new window.     where did they go?    reports on top exit pages will provide clues  as to why patrons left the site. did a patron  leave because she couldn’t find what she  was looking for, or did she leave because  she found a link to a resource which met her  information need? ideally, the top exit pages  on your site are those which provide links to  recommended services and resources such  as the catalogue entry page, lists of useful  external resources, or e‐journal and e‐index  search pages.     one of the most frustrating aspects of web  log analysis is the inability to tell where a  patron has gone once he has left the site.  library web sites are often designed to help  users find information, resources, and  services that exist in other places. it is  difficult to determine whether this objective  has been met if one cannot see the resource  selections that users make from the library  web site. additionally, libraries invest  significant amounts of money to license  electronic reference materials including  indexes, journals and other content that  does not reside on library servers.  measuring the use of this material is  important for on‐going cost‐benefit  evaluation of subscribed electronic content.     although standard web log analysis cannot  provide information about the external links  selected by patrons, there is a way to track  this information through the use of small  programs that log and redirect requests for  external links. these programs can be  written in languages such as php, asp, or  cgi/perl. external link‐tracking is  implemented by changing all of the urls  on a web site to point to the redirect  program on the local server, rather than  pointing the user directly to a remote  resource. following are examples of two  ways to use hrefs to redirect users to local  servers:     • href pointing to an external site:  o <a  href=ʺhttp://agricola.nal.u sda.gov/ʺ>agricola  citation index<a>  • href pointing to a cgi/perl script  named ‘getit’ on your own server.   o <a  href=ʺ/cgi/getit/http://agri cola.nal.usda.gov/ʺ>agric ola citation index</a>   when the latter link is invoked, the url of  the remote resource is sent to the cgi  program. the cgi can log information  about the request, including the date/time of  the request, the page from which a user  clicked the link, and the url of the remote  resource. the cgi then redirects the user to  the requested url. the cgi may store  logged information in a delimited text file,  or in an sql‐compliant database. similar  cgis can be added to urls in marc  records to help track resource usage from  the library catalogue.    there are two drawbacks to tracking  external links selected from the library web  site. the first is that all urls to external  resources will have to be re‐written to point  to your cgi, instead of directly to the url.  the second is that the cgi adds a fractional  delay in the time that it will take patrons to  see the external page which has been  requested. performance hits are best  measured in a specific environment, as they  have to do with the hardware resources  available on the web server, the amount of  information logged by the cgi, and the  amount of traffic on a site. if a link checker  is used to find broken urls, then ensure  that the cgi returns appropriate error codes  when it cannot contact a site.     http://agricola.nal.u http://agri evidence based library and information practice 2007, 2:1  65 page tagging and traffic analysis    if web site logs are not easily accessible, but  you have access to the source code, e.g.,  ‘hosted solution,’ then traffic analysis  programs may be the solution. these  include:    • google analytics  <http://www.google.com/analytics/>  • onestat <http://www.onestat.com/>  • statcounter  <http://www.statcounter.com/>    each of these applications relies on client‐ side data collection through javascript and  cookies. these programs require that a small  piece of javascript code is added to each  page that will be tracked. in web analytics  literature the process of adding this  javascript to web pages is called ‘page  tagging,’ (not to be confused with ‘tagging’  offered by social bookmarking sites such as  <del.icio.us>). the google analytics ‘tag,’  for example, looks like this:    <script src=ʺhttp://www.google‐ analytics.com/urchin.jsʺ  type=ʺtext/javascriptʺ></script>  <script type=ʺtext/javascriptʺ>          _uacct = ʺ<tracking number goes  here>ʺ;          urchintracker();  </script>    once the javascript has been added to the  page header, activity can be tracked by  signing into a web‐based interface that  allows the generation of reports, charts, and  graphs that contain information about site  traffic. the type and level of analysis  possible will depend on the chosen vendor  solution, but most of the types of reports  that are generated through web log analysis  are also available from page tags.     analysis performed through page tagging is  close to real‐time, so a snapshot of site  activity is quickly available. it is possible to  monitor take‐up of a new service on the day  it is launched, or to watch the effects of a  publicity campaign or a policy change from  its inception.     one of the main advantages of page tagging  is that the javascript code runs every time  the page is loaded, whether from the server  or from a browser cache. this means that  usage information is updated even when a  user looks at a cached page.  in some cases  tagging solutions can collect data that  cannot be found in web logs. tags store  data in cookies that can be configured to log  additional information, such as javascript  and flash events triggered as a user  navigates through the site. another  advantage is that the software used to view  and analyze traffic is often hosted on a  third‐party server, so technical expertise is  not required in the administration of these  reports.     some of the drawbacks of page tagging  include the fact that the javascript header  has to be added to every page that is tracked.  ‘server‐side includes’ are a means by which  a common piece of code can be inserted into  web pages as they load.  if a site has server‐ side includes that put header information on  pages, then it may be a simple matter of  adding the javascript to the single header  file read into all pages. if a site does not  have includes for header information, then  some manual labour will be required to add  this information to each page of the web site.  the execution of javascript code is  performed by a users’ web browser ,’ and  although it may not be detectable to the  casual user, javascript creates a slight lag in  page load time. some users may have  javascript disabled in their browsers, in  which case the appropriate information  cannot be collected. ‘page‐tag vendors’ may  also rely on third‐party cookies that are  frequently blocked in modern browsers.  third party cookies are those which do not  http://www.google.com/analytics/ http://www.onestat.com/ http://www.statcounter.com/ http://www.google%e2%80%90analytics%00 evidence based library and information practice 2007, 2:1  66 originate from the server on which the web  page is located, but are pulled in from a web  server at a different location. this renders  the cookie information available to the third  part server, and so can constitute as a  privacy threat. it is preferable to select a  vendor that supports first‐party cookies.     due to storage restrictions on the remote  server, data may not be available on the  vendor site indefinitely. data may also  disappear if the page tag subscription with  that vendor is terminated. ideally the data  store can be periodically exported to a local  server for long‐term storage and additional  analysis.     other concerns for libraries include privacy  issues. in most tagging solutions, web site  traffic data is held on the server of the page  tag vendor. it is important to understand the  vendor’s privacy policy up front, including  whether or not the data may be used for  purposes other than your own. privacy laws  are applied according to the country in  which the vendor server is located.     part i of this article has introduced concepts  which will be important for any librarian  who wishes to engage in webserver log  analysis. these examples have been  developed around the library’s main  website, however there are many other  library resources which produce server logs.  in part ii the author provides an overview of  server log data from library‐specific  applications including proxy servers, link  resolvers, and integrated library system  (ils) servers.       works cited    beitzel,  steven, eric jensen, abdur  chowdhury, david grossman, and  ophir frieder. “hourly analysis of a  very large topically categorized web  query log.” proceedings of sigir  (special interest group in information  retrieval), july 25–29, 2004, sheffield,  south yorkshire, uk.  321‐8.    breeding, marshall. “analyzing web server  logs to improve a siteʹs usage,”  computers in libraries. 25.9 (oct.  2005):26, 28‐9.    carter, david s. “web server transaction  logs: dave’s ula project.” 7 mar. 1996.  25 feb. 2007 <http://www‐ personal.umich.edu/~superman/ap/>.    clifton, brian. whitepapers: web traffic  data sources & vendor comparison.  omega digital media ltd. 7 dec. 2006.  25 feb. 2007 <http://www.ga‐ experts.co.uk/web‐data‐sources.pdf>.     coombs, karen a. “using web server logs  to track users through the electronic  forest,” computers in libraries 25.1  (jan. 2005): 16‐20.    cram, jeff. “building a web site for  analytics,” digital web magazine 16  oct. 2006. 25 feb. 2007  <http://www.digital‐ web.com/articles/building_a_web  site_for_analytics/>.     davis, philip m. “information‐seeking  behavior of chemists: a transaction  log analysis of referral urls.”  journal of the association for  information and library science and  technology (2004) 55.4: 326‐32.    eisenberg, bryan. “accurate analytics  require cookies,” clickz networks 5  mar. 2004. 25 feb. 2007  <http://www.clickz.com/showpage.ht ml?page=3319891>.     flaherty, p. “transaction logging systems:   a descriptive summary,” library hi  tech 11.2 (1993): 67‐78.  http://www%e2%80%90personal%00 http://www.ga%e2%80%90experts%00 http://www.digital%e2%80%90 http://www.clickz.com/showpage.ht evidence based library and information practice 2007, 2:1  67 jasra, manoj. “web analytics comparison –   google vs. visistat.” web analytics  association. 16 aug. 2006.  9 nov. 2006   <http://www.webanalyticsassociation. org/en/art/?119>.     kurth, martin. “the limits and limitations  of transaction log studies,” library hi  tech 42 (1993): 98‐104.     menasalvas, ernestina, socorro millán, josé  m. peña, michael hadjimichael, oscar  marbán. subsessions: a granular  approach to click path analysis,”  international journal of intelligent  systems 19(2004): 619‐37.    sandor, b. “applying the results of  transaction log analysis,” library hi  tech 11 (1993): 87‐97.    schwartz, randal l. “clicking‐through  tracking in perl,” new architect (may  1998). 25 feb. 2007  <http://www.webtechniques.com/archi ves/1998/05/perl/>.    sterne, jim. web metrics: proven methods  for measuring web site success. ny:  wiley, 2002.    web analytics tutorial. summary.net (18   apr. 2002). 25 feb 2007   <http://www.summary.net/manual/tut o rial/toc.html>.    software    webtrends 7.0, c2004. webtrends, inc.,  portland, or. 12 february 2007  <http://www.webtrends.com/>.    clicktracks optimizer 3.0 c2005.  clicktracks analytics, inc. santa cruz,  ca 12 february 2007.  <http://www.clicktracks.com/>.     analog 6.0 c.2004. written by stephen  turner. released under the gnu  general public license. february 12  2007.  <http://www.analog.cx/>    webalizer 2.01 c.1997‐1999. written by  bradford l. barrett . released under  the gnu general public license.  februrary 12 2007.  <http://www.mrunix.net/webalizer/>    awstats 6.6 c 2006. written by  laurent  destailleur. released under the gnu  general public license. february 12  2007. <http://awstats.sourceforge.net/>    google analytics c.2006. google inc.  mountain view, ca . february 12 2006.  <http://www.google.com/analytics/>    onestat enterprise 3.0 c2006. onestat.com,   netherlands. febrarury 12 2006.   <http://www.onestat.com/>    statcounter c2005. statcounter.com, dublin,  ireland. february 12, 2006.  <http://www.statcounter.com/>    http://www.webanalyticsassociation http://www.webtechniques.com/archi http://www.summary.net/manual/tut http://www.webtrends.com/ http://www.clicktracks.com/ http://www.analog.cx/ http://www.mrunix.net/webalizer/ http://awstats.sourceforge.net/ http://www.google.com/analytics/ http://www.onestat.com/ http://www.statcounter.com/ classics   developing a measure of library goodness   a review of: orr, r. h. (1973). measuring the goodness of library services: a general framework for considering quantitative measures. journal of documentation, 29(3), 315-332.     reviewed by: gregory a. crawford interim director penn state harrisburg school of humanities middletown, pennsylvania, united states of america email: gac2@psu.edu   received: 9 feb. 2016     accepted: 20 june 2016      2016 crawford. this is an open access article distributed under the terms of the creative commons-attribution-noncommercial-share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to discuss the theoretical design of a measure of library quality and value that could be used across functional areas of a library in order to justify and maximize the allocation of resources.   design – this theoretical article provides background on how to conceptualize and develop a quantitative measure of library goodness.   setting – the process delineated is applicable to any library, whether public, academic, or special.   subjects – the intended audience is library management, both at the director and the department head levels.   methods – the author provided examples and questions in the development of appropriate variables.   main results – the author presented a discussion of potential variables. these variables include library capability and utilization.   conclusion – the article concluded with a discussion of the major desiderata for an effective measure of library goodness: appropriateness, informativeness, validity, reproducibility, comparability, and practicality.   commentary   although many think of assessment of libraries as a recent phenomenon, assessment of libraries has been on the minds of library administrators for many years. the chief problem was and still remains how to measure library effectiveness or “goodness”, as orr called it. although some of his discussion is naturally dated, orr developed a theoretical framework for measures of library goodness for directors or department heads to utilize for assessments of the effectiveness of their own libraries.   orr’s chief goal in this article was to provide a method of developing quantitative variables so that libraries can achieve better services at an acceptable cost. specifically, these variables could serve as management tools in the tasks of justification (obtaining resources needed to meet specific goals), allocation (allocating these resources effectively), and maximization (using resources to maximal advantage). in his concept of “goodness”, orr argued that two questions must be answered: “how good is the service?” and “how much good does it do?” (p. 317). he referred to the first question as “quality”, meaning how well the service meets the needs of the user. he called the second measure “value”, which can be judged by the beneficial effects that result from its use. for orr, the goal of the library, in simple terms, was to maximize the quality of its services and the value that it provided to its community in relation to the resources provided.    orr proposed four criterion variables that can be measured and used as surrogates for measuring quality and value directly: resources, capability, utilization, and beneficial effects. he asserted the following basic relationships between these variables: as resources increase, the capability of a service increases; as the capability increases, the total uses (utilization) will increase depending also on demand; as the utilization increases, the beneficial effects will increase; and as the beneficial effects increase, the resources will increase, thereby closing the feedback loop. while resources and use are generally the easiest variables to measure, capability (quality) and beneficial effects (value) are more difficult to measure directly.      one of the major problems in measuring goodness, according to orr, is determining user needs. users are often not aware of their true needs, making it difficult to use the direct measure of needs as a basis for measuring library goodness. as he noted, it is only possible to record needs that have been recognized and acted upon. similarly, the capability of the library in meeting needs of the user is complicated by many factors including the limitation that the perception of library patrons affects their use of library resources. if a user perceives the library as having the capability of answering a need, the service will more likely be used.      in the final section of the article, orr proposed five desiderata for measures of library goodness: appropriateness, informativeness, validity, reproducibility, comparability, and practicality. for appropriateness, he listed several questions that must be answered, including: “is the measure of the proper type?”, “are the units and scale suitable?”, and “are the operations required to implement the measure compatible with the library’s procedures, physical layout, etc.?” (p. 329). by informativeness, orr meant a measure that helps suggest sources of operational problems or possible solutions. validity indicates that the measure is free from bias and that the variable truly measures what it is intended to measure. reproducibility can also be called reliability and indicates that standard techniques were used in order to reduce error. comparability seeks to answer the question “to what extent will results be affected by factors one would wish to control?”, so that changes can be monitored. finally, practicality is of great importance since the overall purpose of the measure is to provide data for use in managing the library.   conclusion   although this article is now over 40 years old and the services provided by libraries have changed greatly, orr still provides helpful insights into developing serviceable measures of library quality and value which would assist to any director who must justify the budget of the library and the allocation of its resources. libraries continue to excel at measuring inputs such as budgets and the size of collections and simple output measures such as circulation and the number of reference questions. these measures, however, do not assist managers in providing a true picture of the importance of libraries to the lives of our users that will resonate with our funders. any current measure must take into account newer services such as the provision of electronic resources (databases, e-journals, e-books, datasets, etc.), new references services such as chat, and new delivery mechanisms. but, most importantly, orr does not take into consideration the satisfaction of users with the services and resources provided. this must be incorporated into any discussion of the effectiveness and value of libraries. thus, the quest for a truly meaningful measure library goodness continues.   evidence summary   a user survey finds that a hospital library literature search service has a direct impact on patient care   a review of: farrell, a., mason, j. (2014). evaluating the impact of literature searching services on patient care through the use of a quick-assessment tool. journal of the canadian health libraries association, 35(3),116-123. doi: 10.5596/c14-030   reviewed by: elizabeth stovold information specialist, cochrane airways group st george’s, university of london tooting, london, united kingdom email: estovold@sgul.ac.uk   received: 21 may 2015  accepted: 27 jul. 2015      2015 stovold. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to assess the impact of a library provided literature search service on patient care. design – multiple choice questionnaire survey. setting – hospital library. subjects – 54 library users who had requested a literature search and indicated the primary purpose of their request was patient care. methods – a multiple choice questionnaire survey was designed, building on previously published library impact surveys and best practice guidelines, with input from staff in the local research department. the survey was reviewed by library staff, researchers and prospective respondents and piloted. the survey was sent out with the answers to literature search requests and a small incentive was offered to those who completed the survey. the survey was followed up with reminders. main results – the response rate was 57.5% (n=54/94). the most common staff groups requesting literature searches were physicians (33.3%), nurses (22.2%), therapists (16.7%), pharmacists (11.1%) and residents (7.4%). the majority stated that their questions had been answered (77.8%), while 18.5% indicated their questions had been partially answered, for reasons such as the answer leading to more questions, or parts of the question had not been addressed. two (3.7%) of the respondents’ questions were not answered, either because no answer existed, or because the question didn’t contain enough detail.   of those who replied that their question had been answered, 64.3% judged the information to have had an immediate impact on patient treatment or management. other uses of the information included refreshing memory, avoiding an adverse event, diagnosing a patient, or preventing a referral. the percentage of respondents judging there to be no immediate impact on patient care was 16.7%. the impact on diagnosis and treatment was further investigated in those who had said there had been an immediate impact, with 22.2% saying the information determined their choice of drug, 29.6% saying the information confirmed their choice of drug, while 18.5% stated the information changed their choice of drug. all respondents replied that they intended to use this information in the future, regardless of whether the information had an immediate impact, or if their question had been answered. conclusion – the authors concluded that the survey results show hospital libraries can have an impact on patient care through a literature searching service. they also found that the library was answering its literature service users’ questions. commentary   the article was assessed with the cristal checklist for appraising a user study (booth & brice, 2003). the strengths of this study lie in its rationale, design, and reproducibility. there is a tradition of measuring the value of health library services (marshall, 2007) and the motivation for this study was recognition that the hospital library was not currently measuring its impact on patient care. the authors recognise that issues surrounding the measurement of impact aren’t unique to their institution, raising a wider need for the development of a common tool.   the study targeted a clearly focussed group of users of a literature search service with the defined objective of measuring the impact of this service on patient care. the authors built on previous work in survey design and best practice to develop a tool which they piloted and critically reviewed. based on the number of literature search requests in the previous year, they estimated that 50 responses were necessary for a representative sample, and this was achieved. the authors provide the full questionnaire in the article, together with a web link to the survey, to enable other institutions to reproduce and adapt the tool. the authors acknowledge that while they found that the service provided by their library had a direct impact on patient care, this may not be generalizable to other populations.   the results raise issues not discussed in the article. reasons for non-response were not considered, or the possibility that these reasons may be skewing the results. each respondent’s profession was analysed, but it is not clear whether the proportion of individual professions is representative of the whole library user base, or if some staff groups were under-represented. there wasn’t a control group, so we do not know how the impact of a library-provided literature search compares with a self-conducted literature search, for example. library users were only surveyed if they indicated that patient care was the reason for their inquiry, so it is not possible to say anything about the value of the service overall.   the study has provided a validated tool for other health libraries to measure their own impact on patient care, and the authors encourage others to do so. the tool could be adapted to assess other outcomes, such as impact on teaching, learning, and policy decisions by the hospital, and other services not limited to health libraries, such as information skills training. by demonstrating their impact and value, libraries will be in a stronger position to justify funding to provide these services.   future research opportunities include work to further validate the tool, and to answer some of the unknowns, such as how a library literature search service compares with a self-conducted literature search.   references   booth, a., & brice, a. (2003). clear-cut?: facilitating health librarians to use information research in practice. health information & libraries journal, 20(suppl 1), 45-52. http://dx.doi.org/10.1046/j.1365-2532.20.s1.10.x   marshall, j.g. (2007). measuring the value and impact of health library and information services: past reflections, future possibilities. health information & libraries journal, 24(suppl 1), 4-17. http://dx.doi.org/10.1111/hir.12079       conference paper   analyzing the miso data: broader perspectives on library and computing trends   laurie allen coordinator for digital scholarship and services haverford college libraries haverford, pennsylvania, united states of america email: lallen@haverford.edu   neal baker library director earlham college richmond, indiana, united states of america email: bakerne@earlham.edu   josh wilson director for academic support and user services brandeis university waltham, massachusetts, united states of america email: jwilson@brandeis.edu   kevin creamer director for teaching, learning and technology boatwright memorial library university of richmond richmond, virginia, united states of america email: kcreamer@richmond.edu   david consiglio head of research support and educational technology bryn mawr college bryn mawr, pennsylvania, united states of america email: dconsiglio@brynmawr.edu      2013 allen, baker, wilson, creamer, and consiglio. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to analyze data collected by 38 colleges and universities that participated in the measuring information services outcomes (miso) survey between 2005 and 2010.   methods – the miso survey is a web-based quantitative survey designed to measure how faculty, students, and staff view library and computing services in higher education. since 2005, over 10,000 faculty, 18,000 students, and 15,000 staff have completed the survey. to date, the miso survey team has analyzed the data by faculty age group and student cohort. much of the data analysis has focused on changes in the use, importance, and satisfaction with services over time.   results – analysis of the data collected during 2008-2010 reveals marked differences in how faculty and students use the library. the most frequently used services by faculty are the online library catalog (3.39 on a 5-point scale), library databases (3.34), and the library website (3.29). in contrast, the most frequently used services by students are public computers in the library (3.61) and quiet work space in the library (3.29). faculty reported a much higher use of online resources from off campus. analysis of data from schools where the survey was administered more than once during 2005-2010 reveals that both faculty and students increased their utilization of databases over time. all other significant faculty trends reflected declines in usage, whereas, with the exception of use of the library website, all other student trends reflected no change or increased usage.   conclusion – as the miso survey has continued and expanded over the years, the usefulness of rich comparable data from a set of peer institutions over time has increased tremendously. in addition to providing a rich source of data, miso can serve as a model for how a group of schools can collaborate on a share assessment tool that meets the needs of individual institutions and provides a robust, aggregated dataset for deeper analysis.   introduction   as higher education changes in response to budgetary, technological, and political pressures, library and technology leaders increasingly look for meaningful ways to assess how and to what extent our organizations support scholarship, teaching, and learning. the measuring information services outcomes (miso) survey is a web-based quantitative survey designed to measure how faculty, students, and staff view library and computing services in higher education.   the core of the miso survey consists of questions designed to measure the use of library and it services, their importance to the campus community, and the level of satisfaction with which the community views these services. the survey also measures the ownership of technology tools and their use for academic and personal purposes, as well as participants’ perceptions of their own technology skills and preferred learning methods. in addition, it measures overall attitude toward library and technology services on campus.   by looking at computing and library services together, the miso survey provides a richer context for each set of services while acknowledging the shared nature of many of the services as seen from the perspective of our constituents. while there are many distinct services offered by library and computing organizations on campuses, librarians and technologists also frequently work together to support instructional and academic computing needs on campus and to provide resources to off-campus students and faculty. in addition, library buildings are, on many campuses, the site of many computing resources.   launched in 2005, the miso survey has been taken more than 43,000 times at 38 participating institutions, 26 of whom have responded to the survey more than once and 8 more than twice. overall, more than 10,000 faculty, 18,000 students, and 15,000 staff have completed the survey.   the precursor to the miso survey was designed by david consiglio and his colleagues at bryn mawr college to assess the effectiveness of the college’s recently merged information services department. when the survey proved extremely useful, a group of chief information officers from the council of library and information resources (clir) agreed to use the bryn mawr survey as the basis for a common survey to be administered across schools. this would allow each school to learn from the data gathered on its campus and also compare itself to a group of peer institutions. in addition, by conducting the survey every year, each institution would be able to evaluate its services over time. bates college, middlebury college, the university of richmond, and wellesley college graciously agreed to donate a significant amount of a top manager’s time toward the project. in january of 2005, the team members met for the first time at bryn mawr college to begin this process.   during the spring and summer of 2005, the miso survey team prepared and tested the instrument. their five institutions participated in a pilot administration in fall 2005. additional schools administered the miso survey in spring 2006 and in each spring since.   the miso survey team works together to develop long-term strategies, to conduct in-depth analysis of data, and to complete biennial revisions to the instrument. the co-investigators also liaise with participating institutions during the survey administration season to ensure that the survey administration goes well. the team has developed high standards for data quality by using tested questions, ensuring high response rates and customizing the survey instrument so that participating institutions can address local concerns. each participating school receives a summary dataset representing all institutions for the survey year for comparison.   this article will focus on analysis of the larger dataset of all schools and years, offering deeper analysis of user needs than any one school could conduct using its own data. to date, the miso survey team has analyzed the data by faculty age group and student cohort and is now examining how views on services are affected by academic discipline. finally, the survey team combined use and importance trends to provide a richer look at longitudinal changes and better predict how constituents will view services in the future.   survey method, structure, and process at each participating institution, the survey is administered to all teaching faculty, all staff members who are not members of the library or it organizations, and a stratified sample of students selected randomly from the population. the survey is generally administered starting on the fourth thursday of each institution’s spring semester. this approach helps ensure that each institution’s data can be compared to data gathered at other institutions.   the survey’s strategy of regular outreach to respondents enables each campus to achieve high response rates compared to other surveys. in addition, surveying a sample of each institution’s student body helps to avoid student survey fatigue and further increases the student response rate. these methods helped to achieve response rates in 2010 of 48.8% for faculty, 44.9% for students, and 50.3% for staff.   in addition to the core questions included in the base instrument, most participating schools include an expanded set of optional questions and many include custom questions that ask about local services. most of the core and optional questions can be customized to reflect the service names in use at each institution (e.g., those about the online catalog or the course management system).   once a school has agreed to participate in the survey in the coming year, its leadership selects a campus survey administrator (csa) from among the library or it staff. this individual is responsible for all aspects of survey administration at his or her institution. a member of the miso survey team liaises with each institution, helps its csa prepare for upcoming administration deliverables, guides the csa in working with the school’s institutional review board, and answers questions as the process unfolds. these preparations for survey administration take place largely during the fall semester so that the survey is ready to go live early in the spring semester. a more detailed timeline for miso survey administration is available on the miso website (http://www.misosurvey.org).   once all participating schools have concluded their survey administration, the results are summarized and analyzed during the spring and summer months. each participating school receives a comprehensive spreadsheet that includes the mean values for questions included in its survey instrument for each population surveyed (faculty, staff, and students), as well as comparable mean values for all other participating schools. these spreadsheets include results from the current year as well as all previous years. the spreadsheets allow for easy comparison of schools and cohorts to show where statistically significant differences exist. each institution also receives its raw data as well as an spss file for further data analysis.   what is unique about miso?   while each institution has a rich collection of data to analyze from their own survey and from the spreadsheet of mean data for all schools, the miso team has also spent considerable time analyzing results from all schools and cohorts to see broader patterns from within the data. this broader analysis is one of the unique features of the miso survey, as it is done in a statistically rigorous way that allows us to differentiate between patterns that seem emergent based on anecdotal evidence or changes at a single school and those that are truly widespread. the team has been able to view changes in student attitudes about services as they move from freshman year to senior year, as well as some changes that are happening in student attitudes over time without regard to class year. we have also looked closely at trends in the use, importance, and satisfaction with our services as it relates to the age of our faculty members. beginning with the 2010 survey (table 1), we will look at how faculty and students within the various disciplines interact differently with our services as well. below, we have provided one example of the kind of trend analysis possible with the miso survey instrument by taking a deeper look at how the use of library services has changed over time, and how those changes are different for faculty as compared with students.   examples of how analyzing the miso data provides broader perspectives on library and technology services   much of the data analysis has focused on changes in the use, importance, and satisfaction with services over time. in this section, we look more closely at trends in the reported use of library services, without consideration of importance or satisfaction, as an example of one kind of analysis possible with the data. this section first presents the mean frequency of use for faculty and students 2008-2010 as a benchmark about current use patterns, followed by time trends taken from all institutions which participated in the survey more than once from 2005-2010 (n=27). it is important to underline at the outset that an analysis of frequency of use alone is not a sufficient gauge of a service’s value to faculty and students. such an analysis does, however, provide one informative, broader perspective on the it landscape in higher education.   2010 benchmarks: faculty and student frequency of use   frequency of use in the miso survey is set on a five-point scale:   1 (never) 2 (once or twice a semester) 3 (one to three times a month) 4 (one to three times a week) 5 (more than three times a week)   it should be noted that while the numbers used in the scale increase in a linear fashion the categories do not increase linearly. each successive category represents an increase in use that is three or four times greater than the previous category. as a result, a person selecting category four uses a service about 16 times as much as a person selecting category two, even if the numbers “4” and “2” suggest there is only twice as much use.   below are tables illustrating the frequency of use of all library and technology services 2008-2010 (figures 1 & 2). no attempt is made to isolate what constitutes a library service per se, so that nominal “library” services can be viewed in the context of all services. it is of course difficult to decouple such increasingly linked terms.   selecting from the overall array of services, various combinations can be grouped under a more focused rubric labeled “the library.” any attempt to do so is potentially problematic given local conditions at each institution. librarian position descriptions at some colleges involve campus course management system duties, for example, while other librarians elsewhere help to maintain access to online resources from off-campus via software proxy servers.   despite differences in local conditions, there will likely be wide consensus as to what represents a typical library service. these standard library functions are grouped together for comparative analysis (table 2).   comparison of the data reveals marked differences in how faculty and students use “the library.”   the most frequently used services by faculty are the online library catalog (3.39), library databases like jstor (3.34), and the library website (3.29). these are the only library services that faculty use at least one to three times a month, on average.   in contrast, the most frequently used services by students are public computers in the library (3.61) and quiet work space in the library (3.29). these are the only library services that students use at least one to three times a month on average.   table 1 sample sizes and response rates population sample size responses response rate total institutions faculty 9,482 4,707 49.6% 38 students 22,757 8,605 37.8% 38   figure 1 faculty use benchmarks   figure 2 student use benchmarks   the implications for “library as place” are worth serious consideration. across the board, students report using library facilities more than faculty (table 3).   when planning library facilities upgrades, decision makers might do well to consider design with students foremost in mind. they could also synthesize miso frequency of use data with other empirical research that yields similar results about faculty and library facilities (schonfeld and housewright, 2010).   whereas students use a location-based library, faculty turn to online library services with greater frequency (table 4). in addition, faculty report a much higher use of “access to online resources from off campus” (3.54 vs. 2.38), which presumably includes the use of proxy services which allow access to library materials outside of library facilities.   note that it is difficult to determine the extent to which some library services are perceived as location-based or online. for example, library reference services can occur at a physical desk on campus or via email and/or chat. likewise, the provision of interlibrary loan services occurs via online forms embedded in proprietary databases and at location-based service points. furthermore, library patrons can typically use circulation services either online (i.e., a “renew books” option available in the online library catalog) or in a physical facility. overall, faculty use most of these hybrid online/place-based library services with greater frequency than students. however, library reference services are used to basically the same extent by students and faculty (table 5).   table 2 comparison of all library services use benchmarks service name faculty mean student mean interlibrary loan 2.32 1.76 library circulation services 2.70 2.25 library reference services 2.18 2.22 library website 3.29 2.93 online library catalog 3.39 2.80 library collections 2.92 2.49 library databases (e.g. jstor) 3.34 2.90 digital image collections (e.g. artstor) 1.48 1.49 library liaison/contact 1.91 not asked online course reserves not asked 2.96 study carrels in the library 1.28 2.78 quiet work space in the library 1.55 3.29 group study spaces in the library 1.24 2.76 the library café 2.32 2.93 public computers in the library 1.82 3.61   table 3 comparison of “place-based” library services use benchmarks service name student mean faculty mean public computers in the library 3.61 1.82 quiet work space in the library 3.29 1.55 the library café 2.93 2.32 study carrels in the library 2.78 1.28 group study spaces in the library 2.76 1.24   table 4 comparison of online library services use benchmarks service name faculty mean student mean online library catalog 3.39 2.80 library databases (e.g. jstor) 3.34 2.90 library website 3.20 2.93   2010 trends: faculty and student frequency of use   to analyze trends in the use of library services, the following analysis relies only on data from schools where the survey was administered more than once since 2005 (n=27). new questions have been added to the miso survey since 2005, stemming from changes in the wider library and technology services landscape. as a result, trend data is available for a smaller number of services because not all survey questions have yet to be answered more than once by enough institutions to provide generalizeable trends (denoted by “n.a.” in table 6).   this section only reports on services where the change in use over time was statistically significant for faculty or students and where the change was large enough (+/-.025) to merit attention. consequently, an “--“ value in the table below denotes a slope (i.e., a possible change over time) that is not statistically significant or not large enough to be of real practical significance.   the only library services use trend common to both faculty and students is increased utilization of databases like jstor (0.0300 and 0.0348, respectively).   with the exception of library database use, all other significant faculty library services trends reflect declines in usage for faculty: reference services (-0.0380), circulation services (-0.0430), and the online library catalog (-0.0430).   with the exception of the library website (-0.0338), all other student library services trends reflect no change, or reflect an increased usage (a rise in digital images collections like artstor [0.0711]) with less pronounced but still significant growth in interlibrary loan (0.0338).   taken as a whole, these divergent trends also suggest important differences in faculty and student library use patterns.   to focus only on notional “library” services is to occlude important developments of interest to librarians, and this is where miso data distinguishes itself relative to more circumscribed assessment tools. by means of conclusion, one additional technology frequency of use trend deserves careful attention. both faculty and students increasingly turn to the course management system (0.2110 and 0.1399). the usage slopes for products like blackboard and moodle are much steeper than any increased library use trend. librarians ought to consider embedding their services in their course management system since that is where their patrons are to be increasingly found.   table 5 comparison of hybrid online/”place-based” library services use service name faculty mean student mean library collections 2.92 2.49 library circulation services 2.70 2.25 interlibrary loan 2.32 1.76 library reference services 2.22 2.18   table 6 comparison of statistically significant library services use trends service name faculty trend student trend interlibrary loan - 0.0338 library circulation services -0.0430 - library reference services -0.0380 - library website - -0.0337 online library catalog -0.0430 - library collections n.a. n.a. library databases (e.g. jstor) 0.0300 0.0348 digital image collections (e.g. artstor) - 0.0711 library liaison/contact - not asked online course reserves not asked - study carrels in the library n.a. n.a. quiet work space in the library n.a. n.a. group study spaces in the library n.a. n.a. the library café n.a. n.a. public computers in the library n.a. n.a.   conclusion   the data analyzed provide evidence of trends in stakeholder interactions with libraries for 2010. faculty, for example, decreasingly use the online library catalog, library circulation services, and library reference services, and view these three service categories as decreasingly important. of these three service categories, the online library catalog and library circulation services experienced slight drops in perceived importance among faculty while library reference services experienced a somewhat larger drop. on the other hand, faculty increasingly use library databases and are increasingly likely to access online resources from off-campus, which potentially speaks to an increased importance of proxy services. at the same time, faculty consider library research instruction, library liaisons, the library website, and interlibrary loan to be increasingly important, in that order. as for undergraduates, they are slightly less inclined to use library reference services and much less inclined to use the library website over time. conversely, and more so than faculty, undergraduates increasingly use interlibrary loan, library databases, and particularly digital image collections. like faculty but even more so, undergraduates consider library research instruction and interlibrary loan to be increasingly important, in that order. unlike faculty, the undergraduate trend is to view the library website as slightly less important. consistent with faculty, undergraduates view library databases and off-campus access as increasingly important.   the analysis above provides one look at the miso data. by examining the use values for the subset of variables representing library services across time and institutions, we can see trends and patterns that would not have been as meaningful if taken from a single school. as the miso survey has continued and expanded over the years, the usefulness of rich comparable data from a set of peer institutions over time has increased tremendously. the miso annual summary data help participant schools in identifying their relative strengths and weaknesses, creating peer groups for analysis, and determining whether a problem is a local concern or a nationwide trend. the analysis of micro data provided by the survey team allows library and technology decision makers a wider perspective on trends and relationships between services.   in addition to providing a rich source of data, miso can serve as a model for how a group of schools can collaborate on a shared assessment tool that meets the needs of individual institutions and provides a robust, aggregated dataset for deeper analysis. the process of designing, updating, and customizing the miso survey by a team of library and computing leaders from within participating institutions ensures that the instrument remains relevant to decision making, and that the survey is easy to conduct. as the dataset becomes larger, and a greater variety of institutions participate, we will continue to plan for ways to increase the usefulness and scope of analysis, while ensuring that all participating institutions continue to find useful measures of their own service.     references   schonfeld, r. c., & housewright, r. (2010). faculty survey 2009: key strategic insights for libraries, publishers, and societies. new york: ithaka s+r. retrieved 24 may 2013 from http://cyber.law.harvard.edu/communia2010/sites/communia2010/images/faculty_study_2009.pdf   evidence summary   undergraduates have difficulty distinguishing formats based on discovery tool search results   a review of: gewirtz, s. r., novak, m., & parsons, j. (2014). evaluating the intersection between worldcat local and student research. journal of web librarianship, 8(2), 113-124. http://dx.doi.org/10.1080/19322909.2014.877312   reviewed by: diana k. wakimoto associate librarian california state university, east bay hayward, california, united states of america email: diana.wakimoto@csueastbay.edu   received: 7 jan. 2015      accepted: 20 apr. 2015      2015 wakimoto. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to evaluate the usability of worldcat local for undergraduate students conducting research.   design – usability study.   setting – two small, liberal arts colleges in central minnesota.   subjects – 13 undergraduates (7 females and 6 males).   methods – to simulate an authentic research process, researchers created a thesis statement and a list of materials students needed to find using worldcat local. the students were video recorded and instructed to use the “think aloud” protocol as they worked through the list of materials to find. researchers analyzed the recordings and evaluated the efficiency of the students’ searching processes using a rubric with scores from 1 to 5.   main results – students were able to find books relevant to their topic, but had difficulty in identifying a book that their college library did not own. students had more difficulty finding current scholarly journal articles and encyclopedias. additionally, students had trouble distinguishing different formats in the results list.   conclusion – the worldcat local results interface confused students, especially when they tried to determine the types of materials found (e.g., article, book, etc.). the students showed little understanding of relevance sorting and facets, although they did attempt to use them while searching. despite the difficulties, the colleges will keep worldcat local as their discovery tool while exploring alternative options. the researchers suggest the need for future research to confirm their findings and determine what changes to the discovery tool interface would be most beneficial for the users.   commentary   as the popularity of discovery services increases, usability studies become important. this study fits into the larger context of understanding how undergraduates use, or are confused by, result interfaces when conducting research.   the study adds to the growing literature on usability testing of discovery tools. the study is valid when evaluated with glynn’s (2006) critical appraisal checklist. the sample size, though small, fits into the accepted practice of having between 10 and 20 users for usability testing (emanuel, 2013). the strengths of the study include providing the research assignment given to the participants, acknowledging limitations of the study, and not overgeneralizing the results. the authors noted how they achieved inter-rater reliability using their created efficiency score rubric, although the rubric may prove difficult for others to use given some vagueness in the definitions in the scoring categories. including the average length of time that students took to complete or abandon the assigned tasks would strengthen the rubric.   a few clarifications in the methodology would have strengthened the study. there is no mention of a pilot study and it is unclear whether the “sample recordings” noted by the authors were considered part of the main study data set or excluded and simply used for norming purposes.  there is also no explanation for including only 13 participants when over 300 volunteered to participate and the authors had planned for 16 participants. while the sample size is aligned with other usability studies, there is no justification for the number of participants or note of reaching a saturation point in data collection, as is a standard practice.   the authors make a convincing case for needing more authentic tests of students’ ability to effectively use discovery tools. reviewing further literature specific to conducting usability studies with worldcat local (boock, chadwell, & reese, 2009; thomas & buck, 2010), in addition to the literature cited, may have helped in designing methodology and in contextualizing results. additionally, a more thorough literature review would have allowed the authors to compare their methodology with other worldcat local-specific evaluative methodologies, better supporting their stated outcome of designing an evaluative methodology for discovery systems.    while small in scale and scope, this study should give confidence to other librarians designing and executing their own usability testing. as discovery tools evolve and more libraries come to depend on them for a single, streamlined portal to resources, librarians should conduct usability testing and improve these services in ways that allow students to use the tools most effectively. librarians can use the results to suggest local interface changes and to develop better training and instruction methods. in addition, discovery tool vendors can use usability test results to improve their products.   references   boock, m., chadwell, f., & reese, t. (2009, march 27). worldcat local task force report to lamp. retrieved from http://hdl.handle.net/1957/11167   emanuel, j. (2013). usability testing in libraries: methods, limitations, and implications. oclc systems & services: international digital library perspectives, 29(4), 204-217. http://dx.doi.org/10.1108/oclc-02-2013-0009   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154   thomas, b., & buck, s. (2010). oclc’s worldcat local versus iii’s webpac: which interface is better at supporting common user tasks? library hi tech, 28(4), 648-671. http://dx.doi.org/10.1108/07378831011096295     article   bringing in the experts: library research guide usability testing in a computer science class   laura cobus-kuo health sciences librarian ithaca college library ithaca, new york, united states of america email: lkuo@ithaca.edu   ron gilmour web services librarian ithaca college library ithaca, new york, united states of america email: rgilmour@ithaca.edu   paul dickson assistant professor department of computer science ithaca college ithaca, new york, united states of america email: pdickson@ithaca.edu   received: 19 july 2013    accepted: 19 oct. 2013      2013 cobus-kuo, gilmour, and dickson. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     abstract   objective – we sought to develop best practices for creating online research guides in an academic library.   methods – we performed usability tests of particular library research guides in order to determine how to improve them. students in a human-computer interaction (hci) class (n=20) participated in the studies both as subjects of the tests and as evaluators of the results. the students were each interviewed and then asked to review the interviews recorded of four other classmates. based on their own experience with the guides and their viewing of their classmates using the guides, the students worked with librarians to develop best practices.   results – students were generally unfamiliar with the library's research guides prior to the study. they identified bibliographic databases as the most important links on the guides and felt that these should be prominently placed. opinions about many specific features (e.g., images, length of guide, annotations) varied widely, but students felt strongly that there should be some organizational consistency among the guides.   conclusions – the importance that students placed on consistency led the library to adopt guidelines dictating the inclusion of a table of contents and short list of major databases at the top of each guide, as well as uniform placement of certain other elements.     introduction   academic librarians have been creating research guides in a variety of formats for years. such guides are intended to serve as a starting point for research in a particular subject or course. “pathfinders,” a term associated with paper guides to library resources, was coined in the 1970s by librarians from mit (little, fallon, dauenhauer, balzano, & halquist, 2010; vileno, 2007). in the mid-1990s electronic guides first made their appearance (vileno, 2007), and eventually became the web-based guides of today.   at ithaca college, research guides are used by librarians, but we questioned whether they were used by students. the library tracks guide use via google analytics, but numbers for individual guides are low. this is to be expected, since no guide is relevant to all our users. also, analytics do not show us who is using the guides: are the visitors students, faculty members, librarians, or external users?   our study aimed to better understand how, or if, students are using the guides. based on that knowledge, we hoped to improve the guides. our study was unique in that we worked with students enrolled in a human-computer interaction (hci) course to identify areas that need improvement. we investigated student preferences in terms of guide layout, organization, internal navigation, hierarchy, images and video, and content. our results were used to create simple guidelines for local implementation.   literature review   in academic libraries, research guides are as common as books (ghaphery & white, 2012). in spite of their ubiquity and the amount of time librarians devote to creating such guides (gonzalez & westbrock, 2010; hintz et al., 2010; jackson & pellack, 2004; mcmullin & hutton, 2010; sinkinson, alexander, hicks, & kahn, 2012; sonsteby & dejonghe, 2013), there is little research assessing how students use these guides (hintz et al., 2010; ouellette, 2011; sinkinson et al., 2012; staley, 2007; vileno, 2007). the “if you build it they will come” approach has been disputed, as research shows students are not using research guides (mcmullin & hutton, 2010; ouellette, 2011; reeb & gibbons, 2004; staley, 2007).   the literature discussing web-based research guides dates back to the late 1990s. in the early 2000s one study surveyed students on the overall helpfulness of guides and found that 40% of students found guides “unhelpful” or “a little helpful” (courtois, higgins, & kapur, 2005). research has since shown that the more specific the guide, the better (ouellette, 2011; reeb & gibbons, 2004). moreover, research indicates that students use guides more often and find them more useful after receiving library instruction (ouellette, 2011; staley, 2007). it has also been found that students prefer course guides (reeb & gibbons, 2004) and that they are used more often than the more general subject guides (staley, 2007; strutin, 2008).     more recently, the data has shown that research guides should mirror students’ mental models of research rather than the librarian’s approach or expectation of how research should be performed (sinkinson et al., 2012). some studies suggest that a student perspective should influence how guides are created (hintz et al., 2010; ouellette, 2011; strutin, 2008). santos, dias, silva, ferreira, & madeira (2011) found that working with students in an undergraduate hci course as both subjects and designers was a great way to gather student-centered evidence and an opportunity for students to learn about the research process.   subjectsplus and the ithaca college library   our library uses subjectsplus (www.subjectsplus.com), an open source subject guide tool. this software has undergone significant changes in recent years. the early versions of subjectsplus (pre-0.9) were fairly simple: the librarian added resources to the database and tagged them as being associated with a particular subject and having a particular type (e.g., “encyclopedia,” “handbook,” “database”). the resource then automatically appeared on the appropriate guides. each guide consisted of a list of resources, organized by type.   with the release of subjectsplus 0.9 in 2011, all of this changed. the new version had a drag-and-drop interface that let librarians insert labeled chunks of content (known as “pluslets”) into their guides in whatever order they preferred. a librarian could choose the “all items by type” pluslet, which mimicked the functionality of the pre-0.9 versions, or they could build their guides entirely of customized content that might or might not be related to the resources in the subjectsplus database.     figure 1 a “pluslet”: design view (top) and public view (bottom).   these changes to subjectsplus were very popular with the librarians, as they now had the freedom to organize guides in whatever way they felt was most appropriate to the subject matter. they were no longer limited to an organizational scheme based on resource type. with freedom came diversity. as more librarians exploited the 0.9 features, the research guides began to look less and less uniform. this created a tension between the pedagogical freedom desired by the librarians and the standardization that might be helpful to users of the guides (hintz et al., 2010; jackson & pellack, 2004; ouellette, 2011; sonsteby & dejonghe, 2013; strutin, 2008).   the research guides at ithaca college fall into two categories: subject and course guides. subject guides often correspond to academic majors and minors. course guides are targeted at students in a particular class and are usually supplemented by in-class library instruction. we will be following this naming convention throughout this paper, with “research guide” referring to all guides produced by subjectsplus and “subject guide” and “course guide” referring to specific types.   in september 2012, the library’s web team was in a position to perform usability testing on the guides for a number of reasons. first, the team was in the midst of a redesign to make the website responsive. a responsive website is one that can alter its display based on the size of the screen on which it is viewed. another reason was the vast number of research guides that had been produced and the rate at which more were being produced. at the time of the study the library had just over 400 guides. this number has continued to increase rapidly as course guides become more popular and are requested by more instructors. if the library was to continue producing guides on this scale, it was imperative that they be constructed in a manner that met the needs of users. the final reason was simply that research guides had not been the focus of any of the library’s prior usability studies.   methodology   for this study, the web team partnered with a computer science faculty member and his hci class. the professor and web team created an assignment, which served as a usability study and provided the students the experience of being both a research subject and a data analyst on a practical, real-world use case.   we began by choosing sample research guides to test. we tested the two guide types (subject and course) separately. for each testing group, we chose guides that seemed very different from one another in terms of design and organization. the two subject guides tested were the anthropology guide and the biology guide. the anthropology guide was created using an early version of subjectsplus where the guide was assembled automatically from records. the biology guide was created using sp 0.9, so the author had more options for layout and organization. the primary difference between the two guides is the presence of the long “all items by type” pluslet in the anthropology guide and the subdivision into smaller pluslets in the biology guide. the three course guides tested were “we are what we’ve eaten,” “journalism history,” and “the blues” (see table 1). since all course guides were created after sp 0.9 was launched, the three guides were chosen based on their stylistic and organizational differences. given the diversity of the library’s course guides, we selected three guides to represent the varying styles as illustrated in table 1.   two usability questionnaires were developed—one for the subject guides and one for the course guides (see appendix a). these were similar except for references to the guides being tested. the questions were written to determine which features students considered most useful or appealing. for each guide we provided a research scenario for the students to consider.   table 1 course guide features we are what we’ve eaten journalism history the blues header image yes (not linked) no yes (linked) table of contents no yes no links to individual book or ebook titles yes yes yes links to individual dvds yes yes no links to individual databases no yes yes catalog search box yes no no article search box yes no no related guides yes yes yes lc subject headings no yes yes source descriptions no yes yes pluslets grouped beneath headings no no yes link to plagiarism tutorial yes no no citation information yes no yes length short long long     we hoped that offering a scenario would make this a realistic representation of the research process. in addition to asking questions about the sample guides, the questionnaires were used to gather basic demographic information, as well as information about prior experience with the library’s research guides and the student’s typical research habits. the questionnaire was piloted with two library student employees to test the wording of the questions.   our twenty research participants were from the hci course titled, “user interface design and development” at ithaca college. they were required to participate in the study as part of an assignment. we obtained irb approval from ithaca college to conduct the usability study. more demographic detail is provided in the results section, below.   the testing took place in a private room in the library. the students were randomly assigned to either the subject or course guide testing group. two members of the web team conducted each interview. the guides were shown to the participant on an imac computer in the firefox browser at a 1280 x 720 window size. the action on the screen and the audio of the interview were recorded using camtasia. students were informed that they were being recorded and that these recordings would be viewed by their classmates, but assured that no one beyond the class and the research team would be able to view them.   two team members were present at each interview: one set up the workstation and conducted the interview while the other took notes. at the start of each usability interview, one web team member read a script explaining the purpose of the study (see appendix a). students were encouraged to think aloud and ask questions during the interviews. each interview took 20-35 minutes.   during the two weeks after the interviews were completed, each student was required to view four of the recorded sessions. students who had served as participants in the course guide group viewed only other course guide sessions, while those in the subject group viewed only subject guide sessions.   as the final phase of the study, members of the web team were invited by the professor to attend four 50-minute class sessions. during the first two sessions, each student gave an analysis of the research guides based on the usability test that she or he reviewed. on the third day, members of the web team met with the students in small groups to discuss possible design changes to the guides and to develop a list of best practices. during the final class period, the class came together to combine their design changes and best practices. the web team then summarized their findings for the class and also asked for feedback about the interview procedures used in the usability test.   results   what the usability interviews told us   demographics   the student participants (n=20) represented more than ten different majors and six minors. fifteen majored or minored in computer science. there were 13 men and seven women. the majority of the students were upperclassmen: 18 were juniors and seniors. just under half (n=9) of the students had been in a class with a librarian before, but only seven had visited the library’s research help desk (no correlation to those who had had library instruction). a little more than half (n=12) of the students stated that they knew there was a subject librarian for their major.   students’ research process   when the students were asked to describe their approach to research, 11 students mentioned using google as a first step, and 15 mentioned library resources such as databases, journal articles, and books. the latter number might have been inflated because the students were talking to librarians. when asked if they had used a subject or course guide before, only five participants answered yes. this number may not be generalizable to the campus as a whole, given that nearly one third of the students (n=6) were computer science majors. this department does not often request library instruction. those who had used guides discovered them through various methods including library instruction, recommendation by a professor, or the library website.   the answers to the open-ended questions about the research process were varied, as subjects interpreted the questions differently. the only general trend was that students expect to find library related resources (e.g., books, journals, and databases) on the subject and course guides. the students found databases to be the most useful tools on the subject and course guides.   comparisons between guides   the last section of the questionnaire asked the students to compare the guides (within either the subject or course group) side-by-side in terms of the use of images, multimedia, internal navigation, length of guide, and resource description. again, there was no clear signal in these results—the students were split on what they liked and did not like.   images & video   this area of the study was noteworthy for the sharp division of opinions among students. some students (n=13) appreciated images (“it makes me feel comfortable, like i’m in the right place”), while others (n=7) considered them wasted space (“i don’t think it adds much”). images that served to aid navigation (e.g., biology guide, right side) were generally approved of, but purely decorative images (e.g., biology guide, top) were sometimes questioned.   video had a similarly mixed reception. a video showing how to use the microfilm machine garnered some praise, but a mislabeled video from youtube caused considerable confusion. several students stated categorically that they would not click on videos (hintz et al., 2010, found similar results).   icons indicating database features were not popular. they were regarded as either confusing (“i know what ‘get it’ means when it’s next to an article, but i don’t understand why it’s here”) or just unnecessary (regarding the lock icon indicating that authentication is needed: “i have to log into everything anyway”).   internal navigation & organization   just over half of the students (n=12) liked having a table of contents (toc). one student noted that a toc “really helps to break down the page so i don’t have to scroll through.” regarding organization, students appreciated that there was organization, but they noted the great variance and disharmony between the schemes used in different guides.   length   we asked about students’ preferences regarding length of guides and length of resource annotations. in some cases, a single student would espouse different viewpoints depending on the context in which she was asked.   regarding the length of guides, many students (n=10) preferred shorter guides whereas some (n=7) preferred longer guides. when presented with very long guides, students sometimes felt “overwhelmed” (this word came up frequently) but they also felt greater confidence in the thoroughness of the guide (“i don’t feel like i’d have to go elsewhere”). interestingly, half of the students (n=10) pointed out that length is not an issue as long as there is good navigation and organization.   similarly, many students (n=13) said they preferred minimal annotations, but when they encountered cases where there were longer annotations with search tips, etc., they tended to react positively. one student suggested that the length of a description might depend on the resource: “if a paragraph is necessary, okay, but for common knowledge like the new york times, don’t bother.”   what the classroom discussion told us   the four class sessions in which students presented their analyses and recommendations proved very helpful. below are some themes that emerged from the discussions.   consistency of layout   students attached more importance to consistency of layout than expected. they repeatedly stressed the need for at least some commonality of experience in going from one guide to the next. specific areas where students felt greater consistency would be helpful included:   •     navigational elements toc, back-to-top link. •     contact information in a consistent location. •     common supplemental information e.g., citation styles, plagiarism tutorial. •     common search boxes catalog, article quick search (they liked the ability to perform a search right from the page itself). •     overall format guides should avoid the “all items by type” pluslet.   organization   the most common criticism of the guides was that they were poorly organized, or at least that the organizational scheme was neither apparent to the user nor consistent with other guides of the same type.   in the case of the subject guides, one guide that we studied was organized by resource type (handbook, almanac, encyclopedia, etc.), while the other was organized topically. the students noted this inconsistency, many of them favoring the latter organizational scheme. this tendency has been previously noted in the literature (sinkinson et al., 2012; sonsteby & dejonghe, 2013).   another organizational issue concerned the role of the narrower (right-hand) column. students could not detect any pattern for why some things were in the left (main) column and others were in the right. several students found themselves ignoring the right column. this is consistent with studies showing that people read a screen in an f-shaped pattern (nielsen, 2006). the students suggested that the right column be used primarily for supplemental information.   students appreciated strong visual divisions between organizational units (i.e., smaller pluslets rather than single long ones). on the other hand, students did not like a large number of pluslets that each contained only a link or two.   hierarchy   students felt the most important content should be at the top. by “most important,” they usually meant databases. they appreciated the short “principal databases” boxes at the tops of some guides (e.g., anthropology). one suggestion was to list three top databases and have a “more” link that would reveal additional databases.   internal navigation   navigation within the page was very important to the students. they appreciated tocs, but mentioned some ways that they could be better:   •     tocs should be set off such that they are distinct from other pluslets. •     tocs should be consistent across guides. •     for complex pages, tocs could appear as a collapsible, windows explorer-style tree.   discussion   limitations   while working with students in a course that focused on interface design and usability testing provided valuable feedback, this created a very non-representative sample. for example, 30% of the participants were majors and seventy-five percent were minors in computer science, a very small department at ithaca college. also, 90% of the students were upperclassmen. it is possible that freshman and sophomores may interact with our research guides differently.   we discovered during the usability testing and analysis that we should have piloted the questionnaire with more students. using only two students who worked in the library did not help to uncover problems with many of our questions. for example, we later learned that using a search scenario for each subject or course page (e.g., “muckrakers” was the topic for the journalism course guide; “polygamy” for the anthropology guide) was not helpful. it was stressful for students, as they often felt limited by their knowledge of that particular topic. some students scrolled through the guides or used the browser’s “find on page” feature to look for the specific topic word. more thorough pre-testing of the questions could have avoided this problem.   libraries & computer science   the ubiquity of smartphones with their touch screen interfaces has led to a renewed focus on interface design within the fields of computer science and computer science education. while once primarily the focus of web developers, interface design is now a major component of software development. this has led to an increased number of computer science departments offering courses in hci and integrating hci into more general courses.   one of the specific skills covered in an hci course is usability testing. usability testing involves setting up a testing location with the software to be tested and bringing people in to use said software. the testers are given a basic introduction and asked to perform various tasks. the best way to give students an opportunity to learn about usability tests is to have them take part in the process. for this reason, computer science classes often bring people in from outside to act as clients for the students.   given the increasing number of hci classes and the desirability of real world clients for the students in these classes to work with, collaborations between the library and computer science departments should be possible at many institutions.   this study benefitted both the web team and the hci class. by taking part in the usability tests described in this paper, students learned many of the skills needed to run their own tests, which they were required to do later in the semester. they had an increased understanding of the awkwardness felt by subjects and the importance of the testing environment. the transition to analyzing the data showed them the difference between what they felt during the tests, what they said in response to questions, and what the testers saw. by working with the library’s web team, the students were able to get a better grasp of how usability tests happen in the real world instead of just an academic description of the best case scenario or a toy example in class.   running the usability tests described here was time consuming but otherwise relatively inexpensive. the only software purchased was camtasia for mac. screen capture is not absolutely essential, though it can prove useful for later review of material. what we describe in this paper is only one way to run these tests and what we discovered about the best way for ithaca to develop research guides. any studies run to improve research guides are likely to prove beneficial.   changes resulting from the study   following this study, the web team created simple guidelines that all research guide authors must follow (see appendix b). these guidelines were greatly influenced by the classroom discussions with the students.   the web team decided that a few pluslet types should be included in every guide, with a fixed location for each:   •     “dashboard” (see below) •     contact information •     table of contents •     best bets   the “dashboard” is a newly designed pluslet that contains the following elements:   •     “article quick search” search box •     link to the catalog •     link to ebrary •     link to citation information page •     link to interlibrary loan •     link to the plagiarism tutorial     figure 2 dashboard pluslet.     these were elements that the team felt should be on every guide, and already were on many guides, but in different locations and contexts. placing features in a recognizable configuration and in the same place on every guide makes these important services easier for students to discover (roth, tuch, mekler, bargas-avila, & opwis, 2013). a consistent background image was used for this pluslet to make it stand out. librarians can add this pluslet to a guide by a simple drag-and-drop on the subjectsplus back end.   the contact information pluslet includes the subject librarian’s email, title, phone number, and a link to more guides created by that author. prior to the usability study, librarians could place this information anywhere on the guide. the students informed us that it should be placed prominently and in the same location on all guides.   the toc pluslet auto-populates with internal links to all other pluslets on a guide. the toc was an item that students found highly desirable, so the team wanted to make this a consistent element and easy for librarians to implement.   students consistently told us that they found databases to be the most useful resources on the research guides (similar results were observed by ouellette, 2011; sonsteby & dejonghe, 2013; staley, 2007). they also liked the set of “principle databases” located on the top of the anthropology guide. as a result, each guide is required to include a “best bets” pluslet just below the toc. this pluslet should contain links to a few of the most important databases. for very short guides, this feature is optional.   regarding organization, the subjectsplus administrator disabled the “all items by type” pluslet. as a result, librarians will have to determine their own organizational scheme for each guide, based on the needs of the particular class or discipline. this should limit the use of a type-based organizational scheme, which students did not find helpful.   the web team instituted a rule that primary content should be in the left (larger) column, with the right (smaller) column reserved for supplementary information. of course, the opinions of librarians as to what is primary versus supplemental may vary, so examples were provided in the guidelines. relegating less important material to the right hand column makes sense for a responsive site, since the right column will drop below the left when viewed at a narrow screen width, for instance on a smartphone.   due to the divided opinion of the students with regard to images, they were neither required nor discouraged in the guides. the guidelines do specify a recommended aspect ratio for images used at the top of a guide. this allows librarians the creative freedom to use images if they feel it is appropriate, but encourages a standard practice that produces visual consistency across guides.   visit http://ithacalibrary.net/research/lkuo/2013/ to view images of the research guides evaluated in the study and their revisions using the new guidelines.   conclusion   the research guide usability testing and classroom discussions were successful as they helped the web team to generate simple guidelines for all librarians at ithaca college to follow. working with the hci course provided invaluable insight into both design and organizational issues. it is hoped that other libraries will consider some of these suggested practices.   student responses during the usability testing were highly varied. the classroom focus groups helped clarify and underscore what the participants were actually trying to say. with the usability study we were able to observe the students interacting with our guides, while the discussion allowed for an in-depth conversation about students’ preferences. we recommend the combination of a usability study and group discussion.   a study of this nature is very time consuming, but justified by the work that librarians devote to the construction and maintenance of research guides. linking this study to work with members of the computer science department also proved valuable as it provided feedback from an outside source. it also offered some additional knowledge of how to run such studies that should benefit future usability testing. collaboration with academic departments is a great marketing opportunity for a library, since it allows librarians the opportunity for interaction with students and faculty members, and stresses that librarians are actively working to better meet their needs.   perhaps the most interesting finding from this study is that the students value consistency across guides. doing research is hard work for both the novice and expert. providing research guides with a consistent layout simplifies the initial steps. however, students have diverse preferences and personalities, so studies like this one are unlikely to reveal a single path to successful research that works for all students. therefore, guides should be designed with these varying needs and skills in mind (sinkinson et al., 2012). content of the guides is dependent on the discipline and should be left to the expertise of the subject librarian.   we hope that the students at ithaca college will benefit from the newly designed subject guides. we will continue to test the implemented changes with students to ensure the guides’ usefulness.     references   courtois, m. p., higgins, m. e., & kapur, a. (2005). was this guide helpful? users’ perceptions of subject guides. reference services review, 33(2), 188–196. doi: 10.1108/00907320510597381   ghaphery, j., & white, e. (2012). library use of web-based research guides. information technology & libraries, 31(1), 21-31.   gonzalez, a. c., & westbrock, t. (2010). reaching out with libguides: establishing a working set of best practices. journal of library administration, 50(5/6), 638–656. doi: 10.1080/01930826.2010.488941   hintz, k., farrar, p., eshghi, s., sobol, b., naslund, j., lee, t., stephens, t., & mccauley, a. (2010). letting students take the lead: a user-centred approach to evaluating subject guides. evidence based library & information practice, 5(4), 39–52.   jackson, r., & pellack, l. j. (2004). internet subject guides in academic libraries: an analysis of contents, practices, and opinions. reference & user services quarterly, 43(4), 319–327.   little, j. j., fallon, m., dauenhauer, j., balzano, b., & halquist, d. (2010). interdisciplinary collaboration: a faculty learning community creates a comprehensive libguide. reference services review, 38(3), 431–444. retrieved 5 nov. 2013 from http://www.emeraldinsight.com/10.1108/00907321011070919   mcmullin, r., & hutton, j. (2010). web subject guides: virtual connections across the university community. journal of library administration, 50(7-8), 789–797. doi: 10.1080/01930826.2010.488972   nielsen, j. (2006). f-shaped pattern for reading web content. retrieved 6 nov. 2013 from http://www.nngroup.com/articles/f-shaped-pattern-reading-web-content/   ouellette, d. (2011). subject guides in academic libraries: a user-centred study of uses and perceptions. canadian journal of information & library sciences, 35(4), 436–451.   reeb, b., & gibbons, s. (2004). students, librarians, and subject guides: improving a poor rate of return. portal: libraries and the academy, 4(1), 123–130. doi: 10.1353/pla.2004.0020   roth, s. p., tuch, a. n., mekler, e. d., bargas-avila, j. a., & opwis, k. (2013). location matters, especially for non-salient features–an eye-tracking study on the effects of web object placement on different types of websites. international journal of human-computer studies, 71(3), 228–235. doi: 10.1016/j.ijhcs.2012.09.001   santos, b. s., dias, p., silva, s., ferreira, c., & madeira, j. (2011). integrating user studies into computer graphics-related courses. ieee computer graphics and applications, 31(5), 14–17. doi: 10.1109/mcg.2011.78   sinkinson, c., alexander, s., hicks, a., & kahn, m. (2012). guiding design: exposing librarian and student mental models of research guides. portal: libraries & the academy, 12(1), 63–84. doi: 10.1353/pla.2012.0008   sonsteby, a., & dejonghe, j. (2013). usability testing, user-centered design, and libguides subject guides: a case study. journal of web librarianship, 7(1), 83–94. doi: 10.1080/19322909.2013.747366   staley, s. m. (2007). academic subject guides : a case study of use at san josé state university. college & research libraries, 68(2), 119–139.   strutin, m. (2008). making research guides more useful and more well used. issues in science and technology librarianship, fall(55). retrieved 7 nov. 2013 from http://www.istl.org/08-fall/article5.html   vileno, l. (2007). from paper to electronic, the evolution of pathfinders: a review of the literature. reference services review, 35(3), 434–451. doi: 10.1108/00907320710774300     appendix a questionnaires   usability testing fall 2012 subject guides   thank you for participating in our usability study of the library’s course and subject guides. our purpose is not to test you personally but to uncover problems all our users face when conducting course related research. so try not to feel self-conscious about any difficulties you run into, since these are exactly what we’re trying to identify. if at any point you are not sure what we are asking, please let us know so that we can clarify our question. we value your honest opinion tremendously and believe that student feedback is what’s needed to help us improve our research guides. don’t be shy! we really want to know what you think of our guides.   0. what is your major/minor?   1. what is your year of study? 2. have you had a class with a librarian before at ithaca college?   3. have you been to the research help desk for assistance?   4. did you know that there is a subject librarian for your major?   5. what is your research strategy for beginning a term paper? (you have to write a paper on fracking. how would you start?) subject librarians at ic library construct guides to particular subject areas to help students who are writing papers in those areas. [bring up a random subject guide to demonstrate.]   6. have you used a library subject guide before? [if no, skip to *] when did you use it? (early in the research process or later?)   6a. which guides have you used before?   6b. are there guides that you have used repeatedly?   6c. can you show me the guide(s)?   6d. how did you find out about the guide?   6e. have you ever searched for a guide that wasn’t first shown to you? (e.g., if you’d used a music guide, and were assigned a psychology paper, did you look for a psychology guide?) [if no, skip to *]   6f. did you find the guide useful? what did you find useful about it?   *anthropology guide   7a. what would you expect to find on a subject guide for anthropology?   [bring up anthropology guide] you have to write a research paper on polygamy. please take your time to look over this guide.   7b. how might you use this guide as part of your research process?   7c. what are the three most useful tools for you on this guide? biology guide   8a. what would you expect to find on a subject guide for biology? [bring up biology guide] you have to write a research paper on rna. please take a moment to look over this guide. 8b. how might you use this guide as part of your research process? 8c. what are the three most useful tools on this guide? comparison of guides   [show anthropology and biology guides in the same browser in different tabs] which guides do you prefer in terms of: 9a. use of images and/or multimedia   9b. internal navigation (toc)/organization   9c. length of guide   9d. resource descriptions   10. content: do you feel the resources you need to do research are there?   navigation  11. please find a guide on psychology.   usability testing fall 2012 course guides   thank you for participating in our usability study of the library’s course and subject guides. our purpose is not to test you personally but to uncover problems all our users face when conducting course related research. so try not to feel self-conscious about any difficulties you run into, since these are exactly what we’re trying to identify. if at any point you are not sure what we are asking, please let us know so that we can clarify our question. we value your honest opinion tremendously and believe that student feedback is what’s needed to help us improve our research guides. don’t be shy! we really want to know what you think of our guides.   0. what is your major and minor?   1. what is your year of study? 2. have you had a class with a librarian before at ithaca college?   3. have you been to the research help desk for assistance? 4. did you know that there is a subject librarian for your major? 5. what is your research strategy for beginning a term paper? (you have to write a paper on fracking. how would you start?) subject librarians at ic library often construct guides for particular classes that highlight resources that students may find useful. [bring up a random course guide to demonstrate.] 6. have you used a library course guide before? [if no, skip to *] when did you use it? (early in the research process or later?) 6a. if yes, which guides have you used before?   6b. are there guides that you have used repeatedly?   6c. can you show me the guide(s)? 6d. how did you find out about the guide? 6e. have you ever searched for a guide that wasn’t first shown to you? (e.g., if you’d used a sociology course guide, and were assigned a psychology paper, did you look for a psychology course guide?) [if no, skip to *]   6f. did you find the guide useful? what did you find useful about it? *we are what we’ve eaten   you have to write a research paper on the banana trade in central america. please take a moment to look over this guide.   [bring up “we are what we’ve eaten” guide.]   7a. how might you use this guide as part of your research process?   7b. what are the three most useful tools on this guide?   journalism history   you have to write a research paper on early-20th century american “muckrakers.” please take a moment to look over this guide.   [bring up journalism history guide]   8a. how might you use this guide as part of your research process?   8b. what are the three most useful tools on this guide?   blues   you have to write a research paper on the influence of african american sacred music on the blues. please take a moment to look over this guide. [bring up the blues guide]   9a. how might you use this guide as part of your research process?   9b. what are the three most useful tools on this guide? comparison of guides   [show the three guides side by side in same browser in different tabs]   which guides do you prefer in terms of:   10a. use of images and/or multimedia   10b. internal navigation (toc)/ organization   10c. length of guide   10d. resource descriptions 11. content: do you feel the resources you need to do research are there   navigation   12. please find a psychology course guide.     appendix b recommendations/guidelines for subject and course guides   “contact us” pluslet in the upper right corner table of contents is the first non-image pluslet. use the toc pluslet; don’t make your own. optional if guide is less than 900px tall. there should be a “best bets” area near top of guide (but below toc). this should contain links to major resources and/or custom search boxes. optional for very short guides. revise all guides to not use the “all items by source” pluslet. left column should contain primary content. right column should contain supplemental content including, but not limited to: dashboard (directly under subject specialist) custom content may include related guides, selected journals/rss, associations, help documents.   figure 3 post-revision version of the “we are what we’ve eaten” guide, showing implementation of the new guidelines.     news/announcements   the 7th international evidence based library and information practice conference (eblip7) is just weeks away!      2013. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by‐nc‐sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   the 7th international evidence based library and information practice conference (eblip7) is just a few weeks away! preparations are being made to welcome the international eblip community to saskatoon, sk, canada. there are a few of things to make note of at this time:   the registration deadline for eblip7 is tuesday, july 2, 2013. register online at http://eblip7.library.usask.ca/#registration/index . eblip7 lightning strikes! lightning strikes are short presentations on the topic “how i’ve been inspired by eblip.” speakers have 5 minutes and 20 slides. however, the slides automatically advance every 15 second! the lightning strikes plenary event will be held thursday, july 18 at 9:45am. there’s nothing too serious here, though. we want your stories, your perspective, and a broad interpretation of the topic. there is still room available to present your own lightning strikes talk. contact virginia wilson to sign up: virginia.wilson@usask.ca the lightning strikes sign up deadline is june 21. what’s a hootenanny? well, the evidence based hootenanny is the eblip7 conference banquet. we’ll be traveling by bus outside of the saskatoon city limits to a saskatchewan farm. there, we’ll dine in rural splendor, draw for door prizes, and enjoy a prairie sunset. dress code is casual; enjoyment is guaranteed. the centre for evidence based library and information practice (c-eblip) will see its grand opening during eblip7. c-eblip is a newly established research centre based at the university library, university of saskatchewan. its focus is supporting librarians as researchers and promoting evidence based library and information practice.   eblip7, july 15-18, 2013, university library, university of saskatchewan. the possibilities are endless!   evidence summary   academic libraries should consider deselection of some electronic books   a review of: waugh, m., donlin, m., & braunstein, s. (2015). next-generation collection management: a case study of quality control and weeding e-books in an academic library. collection management, 40(1), 17-26. http://dx.doi.org/10.1080/01462679.2014.965864   reviewed by: robin e. miller assistant professor and research & instruction librarian mcintyre library university of wisconsin-eau claire eau claire, wisconsin, united states of america email: millerob@uwec.edu   received: 22 may 2015  accepted: 23 jul. 2015      2015 miller. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to describe and advocate for the development of a procedure to discard electronic books from an academic library collection.   design – case study.   setting – academic library in the united states of america.   subjects – 514 electronic books purchased from netlibrary, a subset of 52,000 netlibrary titles collected by the investigating library 2001-2007.   methods – the researchers examined a set of 514 electronic books in the health sciences and medical field, specifically for qualities such as currency and content relevance. an anecdotal case with limited validity, the goal was to articulate why a particular set of electronic books failed to meet the investigating library’s collection standards, and to remove these e-books.   main results – a set of 514 e-books published by icon health publications were found to be mass-produced, and displayed other notable problems, including age over seven years, outdated or irrelevant content, quality issues, and inclusion in an older platform no longer favored for e-books. the icon health e-books were removed from the library collection and, with some difficulty, the items were also removed from the vendor platform. the authors recommended an e-book weeding procedure that considers six potential problems: publication date; inclusion of defunct internet links; mass production; low quality works by the same authors or publishers; e-book packages that appear to feature multiple low quality works; and e-books from early packages, which may have integration problems.   conclusion – electronic books may take up little physical space but libraries should not ignore them when making deselection decisions because their content may be inappropriate for a library or for the disciplines the library serves. the icon health publications e-book package is an egregious example of low-quality e-book content that the authors discovered and subsequently removed from their collection, offering a set of recommendations based on the experience.   commentary   a thought-provoking read for any practitioner considering the problem of e-book retention and deselection; the authors argued that e-books are candidates for discard, just like print material.   as the authors explained, a package of 514 e-books from icon health publications came to their attention in 2013. upon reviewing the titles in this set, the authors determined that the e-books were candidates for deselection because they were mass-produced, having been compiled by algorithm to create formulaic texts that shared similar wording, layout, and source citations. each text shared similar templates, sources, and sentences. to illustrate this point, the authors listed sample titles, and included a table showing a side-by-side comparison of two texts from the collection. the authors omitted the total number of icon texts they examined, leaving the reader to wonder if they drew their conclusions from a sample of e-books or from a review of every title in the package. in addition to mass-production, the authors identified other specific problems with the icon health titles: age greater than seven years, outdated content, and low relevance to institutional collection priorities.   the authors described the unexpectedly difficult task of removing the e-books from the vendor platform so that their users did not continue to encounter the texts. this problem may interest many practitioners, though the article did not address whether or not this problem is common. the icon health package appeared to be a particularly egregious example of low quality e-book content, however, the authors did not compare it to other packages in their library’s collection.   many library professionals regularly encounter troublesome material and choose to discard it. however, the line between interesting anecdote and significant case is blurred by the authors’ own account of how they discovered the icon health package and the recommendations they developed after weeding it. though subject selection is as important to a case study as it is in empirical research, the authors stated clearly that the “case” was selected when it came to the attention of a group of librarians, who identified reasons to deselect the material and subsequently wrote recommendations based on the experience. the authors made logical arguments in favor of systematic weeding of library e-book collections, using the example of the icon health package to illustrate their point. they recommended e-book weeding procedures that librarians could apply in any e-book deselection process, but the article omitted discussion of how these procedures were replicated or modified in smaller or larger, more systematic efforts. the absence of replication, or at least re-application of the authors’ recommendations, limits the validity of the authors’ recommendations.   the glynn’s critical appraisal checklist (2006) was used to determine that this study lacked validity as a case study. the methodology was not clearly defined and the rationale for selecting the icon health package as the single subject was unclear. the authors’ recommendations would be strengthened if they had drawn them from at least one additional case of e-books. in the absence of a comparative approach, an explanation of how the authors have replicated their recommendations in other collection development decisions would have strengthened the report and aided library practitioners in applying the same procedure to their own collection development. nonetheless, in the absence of robust literature about best practices for e-book selection and deselection, the authors’ experience may be instructive to many practitioners as they shape their e-book collections.   references   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. http://dx.doi.org/10.1108/07378830610692154 article   looking for links: how faculty research productivity correlates with library investment and why electronic library materials matter most   michael m. rawls budget and assessment director virginia commonwealth university richmond, virginia, united states of america email: rawlsmm@vcu.edu   received: 15 feb. 2015    accepted: 30 apr. 2015      2015 rawls. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – this paper summarizes two studies that share the same research question: do universities produce more scholarly research when they invest more in their libraries? research libraries spend a great deal of effort reporting their expenditures, collections statistics, and other measures that serve as a basis for interlibrary comparison and even rankings. the straightforward assumption implied by this activity is that libraries better serve their student and research communities when they are well-funded and well-resourced. the studies examined here both ask if that notion can be validated empirically, not because research libraries require some sort of justification, but because in an environment of tough budget decisions and shifting opinions about the changing role of libraries, it may be useful to demonstrate that sustained investment in libraries offers tangible returns or that the failure to do so can result in tangible costs.   methods – a cross-sectional design featuring ordinary least squares regression analysis was used in both studies to estimate the relationship between scholarly research productivity at u.s. doctoral institutions and an array of institutional characteristics presumed to influence that productivity. the concept of research productivity is operationalized as the total number of scholarly journal articles produced by each institution over a five year period – as journal articles represent the most common form of scholarly expression across the greatest number of academic fields. serving as the dependent variable, this data was regressed against a variety of institutional characteristics including faculty size, research expenditures, and grant awards, and several library variables centered mostly on expenditures. the concept behind this design is that to realistically explore the relationship between levels of library investment and research productivity, all other institutional drivers of research productivity must also be represented in the dataset. while the design was similar for both studies, they each drew on different data sources and marginally different populations.   results – both studies found that an institution’s research productivity is positively and significantly correlated with the level of investment it makes in its libraries. furthermore, both studies found electronic library material expenditures to be particularly associated with increased productivity. this relationship was so strong that an institution’s level of research productivity appears to be sensitive to how its library’s collection budget is allocated between print and electronic materials. as the portion of the budget dedicated to non-electronic material grew, research productivity decreased in statistically significant fashion in both studies.   conclusion – while both studies succeeded in demonstrating the existence of an empirical relationship between library investment and research productivity, the most intriguing finding is that both studies observed a decrease in number of journal articles being produced as expenditures for non-electronic library materials increased. the conclusion is that the efficiencies of electronic resources offer such advantages over the use of traditional library materials in supporting scholarly research that productivity suffers as institutions dedicate a greater portion of their collection budgets to print materials at the expense of electronic materials. introduction   a 2009 membership survey conducted by the association of college and research libraries identified “concern about demonstrating library value and effectiveness” as one of the most important considerations on the minds of responding library directors. according to michael germano (2010), “the ultimate goal is a demonstrable strengthening of support from user populations that will translate into the avoidance of deeper or ongoing cuts during the current economic climate.” yet, the call to demonstrate library value can be a gauntlet cast down more often than picked up, due to the difficulty in linking a library’s contributions to campus-wide outcomes that are more manifold than manifest. this paper summarizes two studies conducted by the author in 2012 and 2013 that were designed to overcome this challenge by incorporating representative measures of as many of the drivers of scholarly   productivity as practicable for more than 200 institutions. this approach allows for the examination of how library characteristics relate to scholarly output while also accounting for other relevant campus factors that are likely influences. by using this type of design, both studies can offer insight into how libraries contribute to scholarly productivity in an empirical sense – something that cannot be achieved by examining any single institution. while finding a linkage between library investment and scholarly productivity can only imply a return on investment to the institution (no research design can prove causality so long as we are unable to confine libraries and universities to a laboratory), an empirically established relationship is still preferable to the absence of evidence. furthermore, if a reasonable theory can establish a context for interpreting the correlation, it can provide a reasonable basis for the claim that the correlation being measured represents an actual impact of libraries’ services.   literature review   many studies have explored the relationship between library resources and faculty research productivity. the two research projects featured in this paper are what oakleaf has categorized as “input/output assessments” of library impact on faculty research productivity (acrl, 2010, p. 48). other examples of this type of approach include budd’s work in the 1990s that compared the number of journal publications produced by institutions to their library’s volume count (1995, 1999). more recently, wilson and tenopir (2008) conducted local citation analysis that compared library holdings to faculty member citations to determine the percentage of referenced items that were available from the faculty member’s library. further examples of input/output assessment studies related to research productivity can be found in the value of academic libraries (acrl, 2010, p. 48).   while these works examine the relationship between library resources and faculty research productivity, no u.s. studies have focused explicitly on how electronic library material expenditures relate to research productivity or other institutional outcomes. there are two groups in the united kingdom, however, who have launched empirical investigations analyzing the link between electronic resources and higher education outcomes in that nation. ciber research ltd conducted a study that found a strong correlation between e-journal spending and usage at u.k. universities (ciber, 2008). the study found e-journal spending was correlated with such “downstream” effects as the number scholarly journal publications, phds awarded, and research grant awards at each institution. these results were corroborated by another u.k. study conducted the following year by the research information network (rin), a policy organization funded by the u.k. higher education funding council (rin, 2009). rin later developed a structural modelling technique to test the directionality of the relationship between spending and use, determining that spending drove usage (rin, 2011).   the studies presently examined in this paper were largely influenced by budd’s work linking research productivity to volume counts, mentioned earlier, and weiner’s work examining the library’s impact on institutional reputation. budd’s work relied on citation indexes to attribute the number of journal articles produced by individual research universities and then compared that total to each institution’s volume count using arl and acrl library survey data (budd, 1995, 1999). budd also accounted for the effect of faculty size on productivity by standardizing scholarly output on per-faculty basis. however, budd did not account for the effect that other institutional characteristics – such as research expenditures, financial strength, and grant awards – might have on research productivity. weiner, on the other hand, employed a variety of institutional characteristics to explore the relationship between libraries and institutional reputation, as ranked by the u.s news and world report (2009). she used regression analysis whereby an institution’s ordinal ranking served as the dependent variable and a variety of library and non-library measures served as the independent variables. her goal was to determine if any library characteristics were positively correlated with institutional reputation, but she also recognized that the prestige of a university is not centered solely on the library. therefore she included expenditure data for instruction, research, and student services; levels of alumni, corporate, and foundation giving; measures for graduate rate, retention rate, and the number of grants received; as well as library expenditures, staffing, and transactional data. the two studies explored in this paper essentially amalgamate budd’s comparison of library characteristics to scholarly output with weiner’s use of a regression model that features both library and non-library institutional characteristics to determine their relation to a campus-wide outcome.   this type of research design has a precedent in the field of economics, where actual firm-level output data for a particular industry is regressed against firm-level inputs to form an industry-specific production function equation. known as the cobb-douglas model, this approach is used to study the relationship between a set of inputs and the quantity of output produced, which in turn can be used to measure production efficiency, including the impact of technological improvements (biddle, 2011). the two studies examined in this paper take a similar approach by identifying the institutional inputs that go into producing scholarly research and regressing those measures against actual scholarly output – producing an industry production function of sorts for academic scholarship. furthermore, both studies’ findings regarding the potential efficiencies that electronic library materials introduce into the scholarly production process are consistent with the cobb-douglas model’s ability to identify the impact of technological improvements on production.   methodology and results   first study   the original study sought any evidence suggesting that libraries confer value to the research mission of their host institutions when properly resourced. at the time, there was no particular focus on the role of electronic library materials. instead, a wide array of library measures was assembled to determine which aspects, if any, of libraries are correlated with scholarly output. this array of library variables was drawn from acrl’s annual trends and statistics survey using counting opinion’s acrlmetrics service (www.acrlmetrics.com) and included such measures as total expenditures, library material expenditures, electronic library material expenditures, volume counts, staffing levels, interlibrary loan borrowing, and others. non-library institutional characteristics that might also influence scholarly productivity were collected using the u.s. department of education’s integrated post-secondary education data system (ipeds). these variables included such measures as research expenditures, grant funding, faculty count, total university revenue, year-end value of the endowment, the number of phds awarded, and others. all told, more than 25 different library and non-library institutional measures were represented in the study as potential explanatory variables for scholarly research productivity at each doctoral institution.   the concept of scholarly research productivity was operationalized using the total number of scholarly journal articles produced by each u.s. doctoral institution. journal articles were selected over other forms of scholarly expression because they are common to most academic fields. the number of journal articles attributable to each doctoral institution was established using thomson reuter’s isi web of knowledge citation index. the article count for each school could then be linked to that institution’s library and non-library explanatory variables for analysis.    the choice was made to aggregate the data over a period of five years, rather than relying on data from one particular year. the decision was based on the rationale that it is too imprecise to tie a specific year’s inputs to a specific year’s outputs. instead, by examining a short range of years, it is possible to get a more representative indication of the amount of resources that each institution typically dedicates to scholarly research as well as the amount of productivity that it typically achieves. for the ipeds and acrl data, this involved collecting the reported figures for each measure from 2005 through 2009 and then calculating an average (e.g., average library expenditures per year or average number of faculty per year). this average was compared to the total number of journal articles produced from 2006 to 2010. the range of years was staggered between the explanatory variables and the dependent variable data based on the assumption that inputs must necessarily precede outputs.   the carnegie classifications (2010) were used as the basis for identifying doctoral institutions, though several were excluded due to a lack of reported data. ultimately, 234 institutions were included in the study. a full discussion of this study, including an exhaustive list of the variables, data limitations, and iterative details, can be found in the proceedings of the 2012 library assessment conference (rawls, 2013).    potential correlations were explored using ordinary least squares regression analysis, where the number of journal articles served as the dependent variable and the institutional characteristics served as the explanatory variables. after exploring several different combinations of explanatory variables in a number of iterations, the factors deemed to be most strongly, consistently, and significantly related to journal article output were as follows: total university revenue, number of faculty members, research expenditures, the number of professional librarians, electronic library material expenditures, and non-electronic library material expenditures. other explanatory variables also proved to be significantly related to journal article output, but had to be excluded due to the issue of multicollinearity. this occurs when two or more explanatory variables are so highly related to each other that the scope of their relationship with the dependent variable cannot be precisely measured. for example, both total library material and electronic library material expenditures had statistically significant relationships with journal article output. both variables, however, increased or decreased from one institution to the next in a very similar manner. this similarity was so strong that when both variables were included simultaneously in the same model, the analysis was unable to distinguish the effect that one variable had from the other on the corresponding changes in each institution’s article count. this development meant that some variables needed to be excluded in order to gain an understanding of the degree to which different characteristics related to scholarly productivity. level and consistency of statistical significance as well as size of standardized coefficients were used as a basis for which significant variables were excluded or retained.   finally, it was necessary to include an indicator variable for harvard university to control for the outlier effects that that institution’s unparalleled personnel expenditures and staffing levels were exerting on the rest of the dataset. prior to adding this “dummy” variable, the regression results had mostly indicated that the library variables were not significant. after it was introduced into the dataset, however, most major library expenditures categories were consistently significant. another option would have been to exclude harvard altogether, as both methods would have reduced the residual effect of harvard to zero. the decision was made to retain harvard, however, because it seemed appropriate to include the highest-spending library, given the goals of the study.   the unstandardized coefficient for each variable contained in the regression results represents its estimated relationship to the number of journal articles produced by an institution (see table 1). for example, these results estimate that for each dollar dedicated to electronic library materials, a u.s. doctoral institution is expected to produce .00052 journal articles. likewise, it estimates the publication of .78292 journal articles per faculty member. when the coefficients for the model’s variables are multiplied by the actual numbers belonging to a particular institution and then added together, it provides an estimate for the total number of journal articles that the institution is predicted to produce given these inputs. the implication is that a change to any one of these variables should result in a corresponding change to the number of journal articles that an institution produces. for example, this model suggests that a $1,000,000 increase in electronic library materials spending should result in 520 additional articles.   the model produced an adjusted r-squared value of .925, which was roughly consistent with other iterations. among the library-related measures, the number of professional librarians had the largest standardized coefficient, suggesting that this measure was more strongly associated with increased scholarly productivity than electronic material expenditures (.218 to .184). while this finding was very encouraging, and deserving of additional study, the second study was unable to replicate a linkage between staffing levels and productivity.     table 1 "best fit" model from first study independent variables* unstandardized coefficients   standardized coefficients t sig.   b std. error beta     (constant) -1401.94136 303.274   -4.623 .000 total university revenue 0.00000212 .000 .21251 3.961 .000 faculty fte 0.78292 .355 .10485 2.203 .029 research expenditures 0.00002 .000 .33949 8.250 .000 number of professional librarians 30.98683 7.519 .21828 4.121 .000 electronic library material expenditures 0.00052 .000 .18403 4.661 .000 non-electronic library material expenditures -0.00026 .000 -.09610 -2.739 .007 harvard 21924.60497 3282.390 .17972 6.679 .000 *dependent variable: total number of articles published by faculty and other researchers associated with each us doctoral institution from 2006 to 2010 according to isi web of knowledge.     second study   after the positive results of the first study, a follow-up study was conducted to determine if similar results would be replicated using a different data source. to achieve this, the new study relied on the academic analytics (www.academicanalytics.com) database tool. academic analytics (aa) is a subscription-based system that university administrators can use to measure faculty scholarly productivity. it attempts to do this by attributing scholarly works, citations, grants awards, and honorific awards to individual faculty members and then aggregating that information at the phd program level and again at the institutional level. this allows administrators to analyze the faculty scholarly productivity of each phd program or the overall university within the context of other programs and institutions around the nation.   the general methodology of this study was very similar to its predecessor. the main differences were that the aa system provided a different source of journal count data (crossref), a slightly different time frame (2008-2011), and it drew from a subpopulation of researchers at each institution (only those faculty members associated with phd programs are tracked in aa) instead of the entire research community. the second study also necessitated changes in the explanatory variable data. the ipeds and acrl data used to represent library and other institutional characteristics were re-collected for the years 2007 to 2010 to synchronize with the new time frame of the dependent variable data.   additional explanatory variable data from aa was also introduced into the dataset. this included the system’s own count for faculty, grants awards, and grant dollars – all of which were lower than similar measures from ipeds due to aa’s singular focus on just those professors associated with doctoral programs. the reason for adding this additional data from aa was that it was more proportionally scaled to the dependent variable data. in other words, given that only journal articles published by faculty members associated with a phd program were being counted at each institution, it was logical to count only those faculty members associated with such programs, instead of the entire faculty, when measuring how faculty size relates to this study’s measure of scholarly output. likewise, the same logic applies for the grant-related measures collected from aa over ipeds grant and research expenditures data. in this way, variations in the size of each university’s phd enterprise relative to the overall institution’s size would not skew results.   again, ordinary least squares regression analysis was used to test the relationship between journal output and the variety of institutional and library characteristics represented in the dataset. the results of the final model bore a resemblance to those of the first study, particularly where electronic and non-electronic material expenditures were concerned, though some notable differences occurred as well. the combination of independent variables observed to most strongly correlate with journal article output were: grant dollars, number of phd faculty, number of phds awarded in research fields, electronic library material expenditures, and non-electronic library material expenditures. the model produced an adjusted r-squared value of .969.   the grant dollars and phd faculty count variables in this model can be seen as more relevant substitutes for the research expenditure and faculty count variables found in the first study. the variables for total university revenue and the number of professional librarians were not statistically significant. both revenue and the number of librarians are more realistically driven by overall institution size than by the number of phd programs, suggesting that these measures could simply be out of synch with the dependent variable data used in the study. likewise, once professional librarians were no longer included in the model, the indicator variable for harvard proved unnecessary and was dropped.   electronic and non-electronic materials expenditures each had a similar relationship to journal articles as in the first study, with the former being positively correlated and the latter being negatively correlated, with both relationships being statistically significant. the coefficients were lower, but this too could be a result of scale, produced by comparing overall material expenditures to a subset of each institution’s scholarly output, as opposed to all scholarly output in the first study. a more detailed discussion of the second study is available in the proceedings of the 10th northumbria conference on international performance measurement in libraries and information services (rawls, 2014).   discussion   the inverse correlation between non-electronic material expenditures and journal article output was unforeseen, in that the general expectation for explanatory variables was that each one would have a relationship that was either significantly positive or one that was not statistically significant at all. but these results suggest that for each additional dollar invested in traditional library materials, scholarly productivity decreases. how could this be? it is not as though print materials offer no usefulness to researchers, let alone serve as a hindrance. furthermore, volume counts and other measures of the physical collections did not register a significant or negative correlation. instead, a plausible interpretation is that electronic library resources are more efficient in supporting research needs than print materials. to illustrate the obvious, think of a researcher in her office conducting a single, well-worded search on the library’s website and gaining instant access to a dozen relevant titles for her literature search. contrastingly, think of her at a poorly resourced institution, finding only some of her needed articles and having to work through interlibrary loan or make a trip to the library to wade through the bound periodicals in order to access the remaining portion of the same titles. the time difference between these two scenarios is likely measured in hours or days. likewise, access to digital archives, databases, and secondary datasets may preclude a trip to far-flung archives or the need to collect data, potentially speeding up a research project by days, weeks, or months, or even allowing the research project to take place at all. when all of these time savings, however great or small, are multiplied by each member of the institution’s research community, it is not surprising that those institutions that are better endowed with electronic materials are able to produce more scholarship over a given period of time than those that are not.     table 2 "best fit" model from academic analytics study independent variables* unstandardized coefficients   standardized coefficients t sig.   b std. error beta     (constant) -317.09038 89.028   -3.562 .000 phd faculty count 2.32040 .287 .29340 8.077 .000 grant dollars .00002 .000 .57484 23.070 .000 phds awarded research fields 2.40900 .669 .11100 3.598 .000 electronic library material expenditures .00011 .000 .07997 3.210 .002 non-electronic library material expenditures -.00005 .000 -.03317 -1.983 .049 *dependent variable: total number of journal articles published by faculty members associated with a phd program at us doctoral institutions from 2008 to 2011, according to crossref.     yet, the efficiency alone does not entirely explain why print expenditures would be significantly negative. to illustrate why this is the case, it is important to point out that non-electronic library materials expenditures is a not a measure collected in the arl or acrl surveys. rather the variable was derived by subtracting each institution’s reported electronic library material expenditures from their total library material expenditures. this means that the non-electronic and electronic materials variables serve as two components that comprise the library’s overall collection budget. therefore, as electronic material expenditures grew as a total portion of the budget from one institution to the next, the non-electronic material expenditures necessarily shrank. conversely, as the ratio of non-electronic library materials grew, it was at the expense of electronic materials. the suggestion is that those institutions deciding to invest more in non-electronic materials – or perhaps those that experienced a slower transition from print to electronic during the span of this study – paid an opportunity cost in terms of journal article production. thus those universities that spent more on non-electronic library materials experienced a loss in scholarly productivity instead of realizing a potential gain. these results are in line with the manner in which the cobb-douglas model detects production efficiency in economic production theory. the model does this by identifying firms that are producing more output than the sum of their inputs suggest that they should be able to produce, when compared to an industry average as established by a regression equation (biddle, 2011). this suggests that the excess production is attributable to a technological efficiency that the highly productive firm is employing and that the average and lower producing firms are not. in the case of these two studies, the institutions allocating more of their collection budgets toward electronic resources experienced greater productivity – presumably because they offered their research communities more efficient inputs that reduced the time needed to complete the research cycle.   the nature of the relationship between non-electronic materials and scholarly output offers unique evidence in support of the study’s original hypothesis. recalling that the initial intent was to demonstrate empirically whether well-supported libraries are generally associated with higher levels of scholarly production, the strong positive correlations that both electronic library materials and the number of professional librarians exhibited with journal articles arguably achieved that goal (total library material expenditures and total library expenditures were also strongly related to journal articles, but again, were removed due to multicollinearity). while these results realize the original objective of detecting linkages between library inputs and scholarly output, they cannot prove causality – as is the case with a quasi-scientific research design. in fact, were it not for the negative coefficient associated with the print materials, it would be simple to challenge these results with the argument that the findings only prove that well-off doctoral institutions have more of everything than less well-to-do universities. it follows that institutions of greater prestige and deeper funding are simultaneously in a better position to support research, to spend more lavishly in support of their libraries, and to produce more scholarship. that all of these factors can be identified to correlate with one another in a regression equation could be interpreted simply as a rising tide that lifts all boats. the print material expenditure results, however, confound this notion of a rising tide by going in the opposite direction of every other statistically significant measure associated with scholarly productivity. when coupled with the theory that print material expenditures represent an opportunity cost to scholarly productivity, a basis is provided for contending that some degree of causality is being measured between electronic materials and scholarly output in this model.   conclusion   the studies described here each provide empirical evidence that scholarly research productivity increases at u.s. doctoral institutions as they invest more in their libraries. the primary finding both studies share in common is that growth in electronic library material expenditures has an especially strong association with growth in research productivity. these findings satisfy the original research question and provide a credible argument that universities can realize a detectable return on their investment in libraries, depending on how that investment is spent. this argument would be less plausible if print materials had not proven to be so spectacularly less productive than electronic resources. but because scholarly productivity seems to ebb and flow so significantly based on how an institution comprises its collections budget, the contention that scholarly output is actually affected by library spending is much more persuasive.   applying regression analysis to the question of whether universities produce more scholarship when they invest more in their libraries allowed both studies to control for other important institutional characteristics that also drive scholarly productivity. this means that the effects that an institution’s faculty size, research expenditures, or grant awards might have on scholarly output were accounted for and incorporated into the study alongside the library-related variables. this approach makes the results more meaningful than simple correlations. as such, it may have applications in other areas where libraries would like to demonstrate their value, yet face the challenge of being one factor among many that contribute to an important institutional outcome.   because both studies found such a sharp contrast between how electronic and print materials expenditures each relate to scholarly research productivity, this topic merits further inquiry. one approach may be to explore the relationship between library investment and scholarly productivity at the discipline level, to determine if these relationships persist across different subject areas. such a study might also benefit from substituting other forms of scholarly expression in place of journal articles in order to further develop this line of inquiry.   references   association of college and research libraries. (2009). building capability for demonstrating value of academic libraries. chicago, il: american library association.   association of college and research libraries. (2010). value of academic libraries: a comprehensive research review and report. chicago, il: american library association.   biddle, j. e. (2011). the introduction of the cobb-douglas regression and its adoption by agricultural economists. history of political economy, 43(suppl. 1), 235-257. http://dx.doi.org/10.1215/00182702-1158745   budd, j. m. (1995) faculty publishing productivity: an institutional analysis and comparison with library and other measures. college and research libraries, 56(6), 547-554. http://dx.doi.org/10.5860/crl_56_06_547   budd, j. m. (1999). increases in faculty publishing activity: an analysis of arl and acrl institutions. college & research libraries, 60(4), 308-315. http://dx.doi.org/10.5860/crl.60.4.308   budd, j. m. (2006). faculty publishing productivity: comparisons over time. college & research libraries, 67(3), 230-239. http://dx.doi.org/10.5860/crl.67.3.230   carnegie foundation for the advancement of teaching (2010). classification downloads. retrieved from: http://carnegieclassifications.iu.edu/resources/   ciber research ltd. (2008). evaluating the usage and impact of e-journals in the uk.  report. retrieved from: http://ciber-research.eu/download/20110119-references.pdf   counting opinions. (2015). acrlmetrics statistics portal. retrieved from http://www.acrlmetrics.com/   germano, m. a. (2010). narrative-based library marketing: selling your library’s value during tough economic times. the bottom line: managing library finances, 23(1), 5-17. http://dx.doi.org/10.1108/08880451011049641   rawls, m. (2013). measuring the impact of electronic library materials on the university’s research mission. in s. hiller, m. kyrillidou, a. pappalardo, j. self, & a. yeager. (eds.), proceedings of the 2012 library assessment conference: building effective, sustainable, practical assessment. (pp. 67-76). washington, d.c.: association of research libraries.   rawls, m. (2014). looking for links: do universities produce more scholarship when their libraries invest more in electronic resources? in i. hall, s. thornton, & s. town (eds.), proceedings of the 10th northumbria international conference on performance measurement in libraries and information services. (pp. 253-257). york, uk: university of york.   research information network. (2009). e-journals: their use, value, and impact.  report available from: http://www.rin.ac.uk/our-work/communicating-and-disseminating-research/e-journals-their-use-value-and-impact   research information network. (2011). e-journals: their use, value, and impact.  report available from: http://www.rin.ac.uk/our-work/communicating-and-disseminating-research/e-journals-their-use-value-and-impact   weiner, s. (2009). the contribution of the library to the reputation of a university. journal of academic librarianship, 35(1), 3-13. http://dx.doi.org/10.1016/j.acalib.2008.10.003   wilson, c. s., & tenopir, c. (2008). local citation analysis, publishing and reading patterns: using multiple methods to evaluate faculty use of an academic library's research collection. journal of the american society for information science and technology, 59(9), 1393-1408. http://dx.doi.org/10.1002/asi.20812   evidence summary   first-year library student assistants have better information literacy skills, and demonstrate a larger increase in these skills, compared to their first-year peers   a review of: folk, a. l. (2014). how well are we preparing them?: an assessment of first-year library student assistants’ information literacy skills. college & undergraduate libraries, 21(2), 177-192. http://dx.doi.org/10.1080/10691316.2013.829377   reviewed by: elaine sullo coordinator, information and instructional services himmelfarb health sciences library the george washington university washington, district of columbia, united states of america email: elainej@gwu.edu   received: 29 may 2015  accepted: 17 sept. 2015      2015 sullo. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to examine the information literacy skills of first year library student assistants, in comparison to first year students who are not library assistants. additionally, the study investigates whether information literacy skills of library student assistants increased more than those of the general student population during their first semester at college.    design – pretest/posttest.   setting – two regional campuses of a research university in the united states of america.   subjects – first-year students, including library student assistants and students in the freshman seminar course. at one regional campus, 103 first-year students, including 5 library student assistants, completed the pretest. at the same campus, 75 first year students, including 5 library student assistants, completed the posttest. at the other campus, 30 first-year students, including 3 library student assistants, completed the pretest, and 26 first-year students, including 2 library student assistants, completed the posttest.   methods – the researcher distributed a pretest and posttest that included demographic questions and 11 items related to information literacy to first-year students. the pretest was given within the first two weeks of the fall semester, before the students attended library instructional sessions. at one campus, the library student assistants took the pretest at the beginning of their first shift, while at the second campus, the library student assistants completed the pretest within the first two weeks of the semester. the posttest was given to participants within the last two weeks of classes during the fall semester.   main results – on the pretest, the library student assistant scores ranged from 6 to 10, out of a maximum of 11 points. for the posttest, these students had scores that ranged from 8 to 11. both of these score ranges were higher than the mean score of the general first-year students.   the mean of the pretest scores of the general first-year students was 5.95 points out of 11 points with a mean score of 54.1%, while the mean of the pretest scores for the library student assistants was 8.13, or 73.9%. the mean of the posttest scores for the general first-year students was 7.29, or 66.3%, while the mean of the posttest scores for the library student assistants was 9.43, or 85.7%. no students earned a perfect score on the pretest. on the posttest, 29% of the library student assistants scored a perfect 11 points, while only 4% of the general first-year students earned a perfect score.   in comparing pretest to posttest scores, the general first-year students’ mean score increased 1.34 points, while the mean score of the library student assistants increased by 1.3 points. the library student assistants scored higher than the general first-year students on both the pretest and posttest; these numbers are statistically significant. the author reports that the increase in the mean scores from the pretest to the posttest for the library student assistants is not statistically significant. on the other hand, the increase of the mean scores from the pretest to the posttest for the general first-year students is statistically significant (p. 186).   conclusions – the author concludes that the information literacy skills of first-year library student assistants are better than general first-year students. this information is valuable to librarians who wish to gauge how well they are doing in regards to teaching information literacy skills to library student assistants. additionally, librarians can better understand how their instruction is contributing to the library student assistants’ educational experiences in general as well as their future as lifelong learners.   commentary   in the literature review, the author notes that there has been no other research conducted that directly links increased information literacy skills to college-level student employment in the library. considering this gap in the published literature, the author has clearly contributed to the baseline knowledge on this topic and filled a void on the subject.   this study was evaluated using the reliant instrument (koufogiannakis, booth, & brettle, 2006). the strengths of this research related to study design include the clearly explained research methodology, the detailed description of the study population, and the inclusion of the survey instrument in the article’s appendix. furthermore, the author notes that the survey questions were based on and linked to acrl’s information literacy competency standards for higher education.   the results of the study are explained in detail, but in some cases the explanation may be too detailed for some readers. fortunately, the author provides an easy-to-read summary of the results, including which test scores are statistically significant.   the study had several limitations. first, a convenience sample was used to select participants and collect data for the assessment. second, the survey was not pre-tested or piloted, so the researcher did not have an opportunity to test whether the questions made sense to the first-year students, and did not know if the instrument included the correct questions to obtain the desired information. additionally, the author did not provide an overview of the specific job responsibilities of library student assistants or include an outline of the training schedule for these students at the institution where the research was conducted. such job responsibilities and training may play a role in the library assistants’ information literacy fluency. in regard to data collection, the author did not indicate if the students completed the questionnaire online or in paper format. the author also did not state if the surveys were given during class time. furthermore, the data was unpaired (i.e., the assessment did not track the performance of individual participants). other limitations include the small sample size of the library student assistants, and also that the assessment only asked about basic information literacy skills.   despite these limitations, the study results should be considered at universities beyond the regional campuses under examination. most universities employ library student assistants, and so understanding how their information literacy skills compare to those of the general student population can certainly help librarians and other library staff with training, and provide insight into how these skills translate into customer service for patrons. additionally, library student assistants are already more knowledgeable about information literacy concepts than their first-year peers prior to instruction. training at a level that is more advanced than the introductory information literacy sessions aimed at first-year students may be more appropriate for library student assistants.   as the article suggests, there are several opportunities to conduct further research in the area of information literacy and library student assistants. future studies could include a larger number of library student assistants, a longer time period for data collection, including paired data, inclusion of targeted information literacy education for library student assistants, and examination of information literacy skills of junior and senior library student assistants.   reference   koufogiannakis, d., booth, a., & brettle, a. (2006). reliant: reader’s guide to the literature on interventions addressing the need of education and training. library and information research, 30(94), 44-51. retrieved from http://www.lirgjournal.org.uk/lir/ojs/index.php/lir/       microsoft word art_lupien.doc evidence based library and information practice 2007, 2:2  67 evidence based library and information practice     article    out of the question!... how we are using our students’ virtual reference questions to  add a personal touch to a virtual world      pascal lupien   academic liaison librarian  university of guelph library  guelph, ontario   plupien@uoguelph.ca      lorna evelyn rourke   academic liaison librarian, university of guelph library  guelph, ontario   lecturer, faculty of information & media studies  university of western ontario  lrourke@uoguelph.ca      received: 08 february 2007    accepted: 18 may 2007      © 2007 lupien and rourke. this is an open access article distributed under the terms of the creative  commons attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.    abstract    objective ‐ to investigate the types of questions students ask and the language they use in  virtual reference. it is hoped that this examination will provide understanding of students’  needs and thus improve/enhance library services.    methods ‐ over 600 virtual reference transcripts were reviewed, analysed and categorised.  this work was focused on three levels of analysis: broad categories based on the general  type of question being asked, subcategories based on the specific question and the language  that students used to ask their questions.    results ‐ students are primarily using the library’s virtual reference service for higher‐level  research assistance rather than using the tool to obtain quick answers to simple questions.  the two most common types of questions involved staff providing detailed information or  instruction on a topic. more specifically, the most frequently occurring type of question was  related to finding journal articles on a given topic. our analysis of the words students use to  mailto:plupien@uoguelph.ca mailto:lrourke@uoguelph.ca http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:2  68  ask their questions confirmed that students and librarians often do not speak the same  language.    conclusion ‐ the results of our analysis of students’ needs and language can help us  understand our users. this study demonstrated that our library can enhance services in five  areas: online services, collections, relationships, staff skills, and the library as place.     introduction    the university of guelph, located in  southern ontario, canada, is a research‐ intensive and learner‐centred institution  with 17,000 full‐time students and 16,000  distance education course enrollments. the  library has offered a virtual reference (vr)  service since 2001; docutek vr software is  used and, since 2006, windows live  messenger (msn) service has also been  implemented. ten staff participate in this  service, which is available during daytime  hours monday to friday. this paper  describes the university of guelph’s virtual  reference (vr) transcript analysis project  and discusses how the evidence from the  analysis can be used to enhance and  implement changes to library services. the  study involved a two‐year examination of  600 docutek vr transcripts, with a focus on  the questions being asked by users. the  analysis looked at the words being used by  students in their questions, as well as the  information and library‐related needs  demonstrated by these questions.  the  findings from the vr transcript analysis (the  only written record of reference interactions  with users) have been complemented by  other information, including anecdotal  evidence gleaned from reference desk  interactions. in the past, decisions about  library services were often made based on  what librarians thought was happening in  reference transactions, but examining the  transcripts has provided concrete evidence.  in some cases, the analysis confirmed what  was already known, and in some cases the  results were surprising!  review of the literature    as vr has become increasingly common,  there has been an explosion of research and  case studies in the library and information  science literature. a number of studies have  focused on analysing the transcripts of vr  sessions. these have looked at various  issues such as the types of questions asked,  the types of users and resources used to  answer questions, and the quality of  responses based on standards such as acrl.  a number of studies have attempted to  understand what users are asking by  classifying vr questions into categories.     diamond and pease reviewed e‐mail  reference questions over a period of two  years and found that similar questions were  asked in the virtual format compared to the  traditional reference desk. questions were  grouped under 11 categories including  database mechanics, catalogue questions,  information literacy and non‐library  questions (diamond and pease 210‐219).     sears analysed four months of transcripts to  identify the types of users, types of  questions, the resources used by staff to  answer the questions and the extent to  which library resources were used (sears).     broughton analysed usage data and user  surveys for one academic year at bowling  green state university (bgsu). included in  this study was an analysis of question type  according categories such as finding articles  on various topics, patron records, university  information, off‐campus access and referrals.  evidence based library and information practice 2007, 2:2  69 these categories were created by the  librarians who provide virtual reference  (broughton 190‐192).     another review of virtual reference  transcripts was undertaken by curtis and  greene, who analysed the university of  nevada’s university‐wide chat service run  by librarians. this chat service provides an  opportunity for prospective students and  registered students to ask questions about  the university in general or library‐related  questions. this study identified that 25% of  the questions were library‐related and  categorized as reference, authentication,  reserves and interlibrary loan (curtis and  greene 220‐233).          others have conducted similar studies for  instant messenger (im) transcripts. foley  describes a university of buffalo pilot  project to determine the viability of  providing a chat reference service. the  project utilized america online’s instant  messenger software, and data was collected  for a period of one academic year. in order  to protect privacy, transcripts were not kept;  however, after each session staff were  responsible for paraphrasing the question  and entering it into a database. the  questions were reviewed and they were  categorized as information literacy,  catalogue, navigation of library web site,  general library information, technical issues,  general web navigation, reserves, university  information, research, ready reference and  questions about virtual reference (foley 41‐ 43).      desai describes the use of im at southern  illinois university by identifying the usage,  the types of questions asked and the impact  of the technology on the reference  interaction. similar to other studies,  questions were categorized based on katz’s  categorization of reference questions with  the addition of an ‘other’ category (desai 23‐ 25).    some studies have compared the use of the  different mediums for delivering online  reference services. lee conducted a  comparison of chat and email reference  questions, including an analysis of the types  of questions asked, the number of words  required to answer the questions, the  number of calls and their duration, as well  as the metrics of the chat transcripts.  questions were categorized as  administrative or reference (lee 98‐108).     a number of studies have analysed  transcripts to evaluate the responses  provided by reference staff. a team at  central missouri state university used the  reference and user services association  (rusa) guidelines as an evaluation tool  (zhuo, love and norwood 75‐88). another  team applied the rusa guidelines to an  analysis of transcripts generated by a  tutor.com homework help service and  found that the transactions were severely  lacking in the qualities required for effective  reference service (walter and mediavilla  214‐221). smyth, in addition to analysing the  transcripts for question type using sears’  classification, also looked at the quality of  the response based on acrl information  literacy competency standards (smyth 28‐ 29).     a few studies have examined problems with  the use of “library terminology” and  differences in online communication styles  between librarians and young people.  kupersmith used web usability testing and  focus group methodology to study library  web site terminology. he asserted that the  average user success rate for finding journal  articles or article databases from a library  website is 53% due to the use of library  terminology. he lists commonly used  library terms not understood by users, such  as database, index and periodical.  kupersmith’s web site also provides links to  a number of studies that look at issues such  evidence based library and information practice 2007, 2:2  70 as terminology used on library web sites  (kupersmith).     fagan and desai discuss common  communication problems in the online  environment and examine how effective  communication strategies can be adapted  and applied to this medium. using  transcripts to provide examples of good  communication strategies, they argue that  the skills needed to communicate effectively  with patrons are different in the online  environment. they suggest that librarians  must avoid using library jargon and ʺrobot‐ like instructionsʺ by “speaking the patron’s  language.” they provide tips on adding a  human touch to im reference transactions,  including the use of more natural language  and emoticons. part of their article focuses  specifically on language and grammar, and  the authors warn that spelling and  grammar‐conscious librarians may have to  lower their standards when chatting with  patrons in an im environment. they argue  that using common im language, which  may include abbreviations, shortcuts and  misspellings, can make the librarian seem  “more approachable and less robotic”  (fagan and desai 125‐143).     similarly, janes suggests that librarians  must understand instant messaging lingo  and culture to communicate effectively with  young people (janes 451). while many  studies have focused on analysing the  transcripts of vr and im sessions, few  articles have gone to the next level by  examining the language students use to  communicate with the library. furthermore,  while much of the research is useful in that  it tells us how students are using online  reference services, there has been little  discussion in the literature about how this  information can be used to make libraries  more user‐friendly. the current study will  attempt to address these issues.      methodology and focus    approximately 600 virtual reference  transcripts were reviewed from transactions  that took place over a five‐month period  from september 2005 to february 2006.  although using two people to categorise the  questions would have increased the  reliability of the results, in an attempt to  maintain as much consistency as possible,  one person was responsible for the question  categorisation. while the focus was on the  questions rather than on the answers, it was  often necessary to examine the entire  transcript to determine what the student  was asking. classification of reference  questions is a difficult task, as the question  asked is not necessarily the question that  needs to be answered. furthermore, many  sessions involved more than one question.  in this case, the session was categorized  according to the original question and each  session was counted in only one category.     analysis of the transcripts was based of  three levels of analysis. questions were first  classified into broad categories based on the  type of question. they were then separated  into subcategories based on what the  student was asking. finally, the language  that students used to ask their questions was  analysed. in other words, the researchers  wanted to know what students were asking  and how they were asking it. the  researchers used excel spreadsheets to keep  track of the questions and categories.    categorization of questions    the broad categories were based loosely on  sears’ adaptation of katz’s categories of  reference questions, namely directional,  ready reference, specific‐search, research  and policy and procedure (sears; katz 14‐18).  directional questions involve directing the  user to a geographical location or to a place     evidence based library and information practice 2007, 2:2  71 on the library web site, policy and  procedure questions involve inquiries about  such topics as borrowing periods and fines,  and ready reference questions are related to  factual information and can be answered  using reference materials. specific search  questions require detailed information or  instruction on a topic and users are often  referred to various sources. an example of  this would be questions regarding finding  journal articles on a given topic. the  research category involves higher‐level  research, generally of the type that would be  conducted by faculty or graduate students.    the categories were broken down into sub‐ categories based on the topics of the  individual sessions. a total of 39 sub‐ categories were created across three of the  broader categories (directional, policy and  procedural and specific search).  subcategories were created by the authors  based on the content of recurring questions.  if there were a sufficient number of  questions on a particular topic, a  subcategory was created to classify similar  questions. the intention was to break each  topic down as much as possible to see what  kinds of things students were asking. this  helped to determine, among other things,  the most commonly asked questions.     following the classification of questions into  broad categories and topic‐based  subcategories, the researchers re‐examined  the language used by students in each  question to identify common terms or  “lingo.” transcripts were scanned for  examples of library lingo and for recurring  terms in students’ questions, and these were  highlighted. the researchers also used the  search capabilities in excel to determine the  number of times a particular term was used.  for example, with respect to questions  about finding journal articles, the various  terms students used to ask for journal  articles were closely examined and a list of  all the words they used to ask for help  finding articles was developed.     results    types of questions    the highest percentage of questions (41%)  can be classified as specific search. policy  and procedure questions accounted for  39.4% of the transactions, directional  questions accounted for 14.87% and ready  reference for only 4.59%. higher‐level  questions that could be classified as  “research” were not asked on vr.      directional policy and procedure ready reference specific search   table 1. types of questions asked      evidence based library and information practice 2007, 2:2  72 question type  frequency (number of times asked  / 600 questions)  how to find journal articles  106  general research strategies  71  off‐campus access   60  locating library web resources  36  library accounts    32  table 2. the 5 most frequently occurring questions and how often they were asked      specific search questions were broken down  into the following subcategories:  archives/special collections, data,  geographic information systems (gis), e‐ books, finding books, finding  articles/journal searching, getting started  with research, government documents and  statistics, index‐specific questions, using  newspaper sources, referencing, finding  theses and finding videos. policy and  procedures questions were broken down  into alumni/community borrowers access,  audio‐visual material, document delivery  procedures, collections policy, holds/recalls  procedures, user accounts, general library  services, non‐student access, off‐campus  access, open learning/distance education,  ill procedures, reciprocal borrowing,  renewals, reserve procedure, special  collections, technology issues and general  university questions. finally, the directional  category was broken down into questions  about the physical library, web resources  and inquiries about the university campus  or web site.    these subcategories were used to determine  the most commonly occurring questions.  the two most common types of questions  were search‐specific. of these, the most  frequently occurring question was related to  finding journal articles. questions were  classified in this subcategory when a student  specifically mentioned that journal articles  must be used as a source of information. a  straightforward example of this type of  question is, “i am looking for journal articles     on women’s rights in nigeria. can you  help?” of course, students expressed this  need for information in many different ways,  and often did not understand what they  were being asked to look for. in these cases,  a more complex online reference interview  was often required to determine what the  student was being asked to do. table 2  provides an illustration of the five most  frequently occurring questions and how  often they were asked.  the next most frequently occurring  questions revolved around general search  strategies (n=71). questions were classified  here when the focus was on finding any  information on a topic, not necessarily  specific to journal articles. an example of  this type of question would be, “i would like  some information on something referred to  as sickle cell disease.” general search  questions that required instruction but were  not linked to a specific type of source were  also classified here.     the third most frequently occurring  questions were related to off‐campus access  (n=60). these involved, for example,  questions about how to access databases  and e‐journals from home. the fourth most  frequently asked questions were directional  questions about locating resources on the  library web site (n=36). the frequency of  this question can perhaps be attributed to a  change in the library web page at the  beginning of the semester during which the  transcripts were analysed. the fifth most   evidence based library and information practice 2007, 2:2  73   term used  occurrence  term used  occurrence  articles   27  full text journals   2  journals (often used when  they       really mean “articles”)   22  magazines   2  journal article  –   17  refereed journals   2  e‐journals (ejournals, etc.)   8  empirical research articles   1  online journals   7  journal entry   1  peer reviewed articles   6  periodicals   1  research articles   6  popular articles   1  academic journals   5  scientific articles   1  scholarly journals   4  scientific journals   1  scholarly articles   3  academic journal articles   2  empirical articles   2        table3.  terms students used when asking for journal articles       frequently asked questions were policy and  procedure questions related to library  accounts (n=32). an example of a typical  question is “i’m not being allowed to charge  out books. the library says i have an  overdue recalled book, but the only recalled  books i have aren’t due yet. do you know  why this is happening?”      status and location of users    information regarding the status of users  was also collected, as users are asked to  provide this when they log into the service.  the following results include repeat users,  as there was no way of separating out the  results for unique users. as this field was  not required,  14%  of respondents (n=86)  provided no information. of vr users who  did respond, undergraduate students made  up the largest percentage (47%, n=268),  followed by graduate students at 17% (n=97).  about 13% (n=74) of respondents identified  themselves as not affiliated with the  university, while only 5% (n=29) were  faculty. only one respondent self‐identified  as a distance education student.      in terms of physical location, approximately  57% (n=336) of vr users were physically  present in the library, while another 15%  (n=89) were located somewhere else on  campus and, 27% (n=159) were located off‐ campus.     language used    following classification of the questions, the  language or words within each question was  then examined. the most frequently asked  question on finding journal articles was  asked over 100 times using 19 different  variations of words!  table 3 illustrates this  by showing the number of different ways  (and their occurrences) that students asked  for journal articles.    the majority of students asked for “articles”  or “journals.” sometimes they used the  words “e‐journals” and “online journals,” or  added the phrase “peer‐reviewed” or  “research” to their requests. out of all of the  questions about “journal articles”, only one  person used the word “periodical.” none of  the students used either of the words     evidence based library and information practice 2007, 2:2  74 librarian language  student language  periodical  journal  databases, indexes  get an article!  faculty, faculty member  prof, professor, teacher  catalogue  trellis (name of library catalogue)  resources  articles, books, web sites  reference collection  background information  citing (for a bibliography)  referencing  interlibrary loan  borrow from another library  stacks  shelves    table 4. variations in librarian and student language      “index” or “database,” two words  frequently used by librarians!      another example of student language was  discovered in the words they use to define  journal articles that are appropriate for a  university paper. half of the students who  asked specifically about academic articles  used the word “academic,” and half of them  used the term “scholarly.” similarly,  librarians tend to use the word “faculty”  when referring to the students’ instructors;  students use the words “prof,” “professor”  and “teacher,” but did not ever use the word  “faculty.”     the library catalogue provided another  example. the library catalogue at the  university of guelph is called trellis.  students asked for ‘the catalogue’ twice, but  they asked for “trellis” 30 times.  however, there were many more questions  about finding books, dvds, archival  materials, etc. in which students did not  know where to start. for them, neither the  term ʺtrellisʺ nor ʺcatalogueʺ may be  meaningful.     librarians often speak of “resources,” but  students only used that word 11 times. they  consistently asked for the product they were  looking for  –  books, articles, or websites.  the phrase ʺinterlibrary loanʺ was used only  twice, but there were over 30 questions  about the interlibrary loan service. students  tend to say “borrow from another library”  or some variation of that, and may not easily  understand the phrase “interlibrary loan.”  table 4 highlights the language used by  librarians and the terms used by students to  mean the same.  discussion    categorizing the questions confirmed  anecdotal evidence of what librarians  already know, or think they know, about the  types of reference questions being asked.  staff members who work on virtual  reference also help students at the  traditional reference desk, and most were  not particularly surprised by the results.  questions requiring students to find journal  articles on a given topic are extremely  common at the reference desk. tracking the  reference questions at the physical desk is  difficult, however, as staff members do not  have the time to write every question down  and need to consider privacy issues. the vr  transcripts therefore provide a written  record of at least a part of the library’s  reference services. this helps to confirm and  better understand what our users are asking.     the results of the transcript analysis should  help librarians to better understand their  evidence based library and information practice 2007, 2:2  75 students and to develop a sense of needs  that are not being met. analysis of the  transcripts demonstrates that students are  primarily using the library’s virtual  reference service for research assistance with  tasks such as finding various sources of  information on a topic and finding journal  articles for course assignments. they are  also asking for help with technical issues  and with navigating the library web site.     the relatively small number of ready  reference questions is not surprising in an  academic environment, as the library’s  mandate is to teach students how to  understand and find information rather  than to simply direct them to answers. still,  it is interesting to note that contrary to what  many librarians originally expected of  virtual reference, it appears that students are  asking research questions in the online  environment as opposed to simply using the  tool to get quick answers to simple  questions. the literature suggests that these  results seem to be representative of the  types of questions being asked at other  libraries (broughton 190‐192; smyth 27‐28;  sears).     the fact that only one user self‐identified as  a distance education student is perhaps  surprising, considering the large numbers of  course enrollments in distance education  courses, indicating a possible need for  marketing and promotion to off‐campus  students. faculty (5% of users) may also  benefit from marketing. another related and  unexpected result of this analysis revealed  that close to 60% of vr users were  physically present in the library and fewer  than 30% were located off‐campus.     this raises questions about why students  were using vr when located in close  proximity to a reference desk. perhaps it is  simply a preference for the online  environment on the part of these individuals,  yet there are a number of other possibilities  to consider. is there a problem with the  physical layout of the reference area,  making the desk difficult to identify? are  some students intimidated about asking a  reference question and the desk? are these  issues related to signage? it should be noted  that at the university of guelph library, the  reference desk was named “research help  desk.”     the transcript analysis also confirmed that  librarians and students sometimes have  different goals and that they do not always  speak the same language. librarians focus  on databases and indexes, and students  focus on articles: one extrapolation from this  finding is that students want, and ask for, a  product — they want an article — rather  than a tool or method, like a database or an  index. librarians tend to focus on the  process and on teaching the students  everything we think they should know  about databases and indexes. students’  focus is on getting the articles and the  information they need to write a paper. in  addition to this realization about what  students really want, when student words  and traditional librarian language were  examined, it appeared that “librarian  language” is often very different from  “student language.” when designing web  sites, knowledge bases and other online  tools and when teaching classes, librarians  need to be aware that students may use  different phrases for the same thing, and  they may not understand an alternative  phrase. libraries may wish to examine how  to move away from “librarian language,” or  at least add in “student language,” in tools  such as library web sites. student  terminology could also be used when  creating metadata for our knowledge base.     the information gleaned from this type of  study can be used to improve library  services and access. by assessing the types  of questions students are asking and  examining the language they understand  evidence based library and information practice 2007, 2:2  76 and use, librarians can think about how this  information can help libraries become more  user‐friendly. with this in mind, the authors  have identified five areas in which libraries  can use this evidence to improve their  services. in some cases, improvements have  already been made, and in some cases,  potential changes are suggested.    1. enhancing online services      library services are increasingly moving to  a virtual environment and the web site and  online services are an essential means of  communicating with users. it is therefore  essential that these tools reflect and use  language that users understand.     the information gleaned from the  transcripts has been used to improve the  library web site. as mentioned earlier, one  of the most commonly recurring questions  at the university of guelph involved  accessing electronic resources from off  campus. staff realized that the link to off‐ campus access login information was not  prominently displayed on the library  homepage, and a more noticeable link was  added. library web sites are perhaps the  most glaring example of poor  communication between librarians and  users. while many library homepages still  use terms such as “journal indexes” or  “databases” to direct users to these tools,  our research indicated that few students use  these terms, and few are likely to  understand what they mean. to ensure that  users make the connection between what  they are looking for (journal articles) and the  tools they must use to find them, library  web sites can use simple user‐friendly  language such as “find journal articles.” the  university of guelph changed the wording  of the “journal indexes” link to a prominent  “journal articles” link, based on the  knowledge that this is the term students use  most frequently when they ask about  finding articles.   changes to language and terminology could  also be used to improve access to other  online tools on the library web site, such as  tutorials, subject guides and also to develop  a knowledge base that students can use to  find answers to common questions. a  knowledge base is a searchable database of  frequently asked questions that users can  use to get quick answers to their queries.  information about how students  communicate with the library can be used to  create a user‐friendly tool that users can  interact with in their own language and that  will respond to users in their own language.  many faq pages require students to know  which words the library uses to find the  response. for example, a student would  have to know that a journal index must be  used to find scholarly articles to find the  relevant question on the faq page. some  knowledge bases created from vr  transcripts simply take questions directly  from transcripts and dump them into a  database, which presents the same problem.  virtual reference transcripts allow librarians  to create more student‐friendly knowledge  bases and faq bases by presenting  questions and answers that are easier to  understand and by developing metadata  based on the words students use in the  transcripts. rather than simply dumping  questions from a vr software package into a  knowledge base, it is possible to take a  number of similar questions, edit them to  make one question and response, and then  attach metadata so that students can ask  questions in their language and get a  response. there are commercial tools on the  market, developed for customer service  purposes, which offer a great deal of  customization, including editing questions  and adding metadata to ensure that users  can retrieve an answer using a variety of  search words.       2. enhancing our relationships    evidence based library and information practice 2007, 2:2  77 student questions on the vr prompted the  creation of new tutorials in information  literacy as students were perceived to be  having difficulties. for example, the  university of guelph offers geography 1220,  a first‐year course on the impact of humans  on the environment. the assignment for this  course is complex and it was apparent from  the vr questions that many students were  not able to complete it without assistance.  the geography librarian created an online  tutorial specifically for these students and  this assignment, and alerted the faculty and  research help desk staff to this tutorial. a  further example occurred when sociology  1100 students began using vr in large  numbers. their librarian intervened by  offering drop‐in research clinics for these  students. once again, a need was identified  from the vr transcripts and students took  advantage of the service that was offered.  because of the number of general questions  from undergraduate users (who comprise  vast majority of guelph’s vr users)  initiatives were established to help them  navigate the library. for example, students  frequently came into the library and into vr  to ask for help in differentiating between  our main library building (mclaughlin), the  agricultural economics building  (maclachlan), the physics building  (mcnaughton), and the arts building  (mackinnon). with so many similar scottish  building names, they were understandably  confused!  to address such problems, one  initiative involved locating a “welcome”  table in the library foyer, staffed by friendly  library staff, where pens and candy were  handed out and students were helped with  finding their way around the library and the  campus.      as guelph’s vr numbers continue to rise  and in‐person reference desk transactions  decrease, the need to reach out to the  students has become even more became  apparent. in addition to offering vr and  expanding it to include the student‐friendly  msn chat, a roving reference service has  been implemented, in which staff are  encouraged to get up from the desks and  approach students with an offer of help.  this service is being very slowly introduced  at guelph, but the need for it seems clear  given the apparent reticence of so many  students to approach a staff member at a  desk. in conjunction with the introduction of  a roving reference program, workshops on  customer service have been held for all  reference staff to ensure the best possible  service is offered—whether in vr, roving,  email, or traditional reference service.    3. enhancing our skills     another result of examining the student  questions was the realization that some of  them are very difficult for staff to answer!   for example, questions related to gis and  government publications are often  challenging for reference staff, and the vr  transcripts proved that this is the case at  guelph. an awareness of staff difficulties  can indicate a need for staff training. it may  also indicate that some questions are too  complex to answer in the confines of a vr  session and that referrals to staff outside of  vr should take place.     4. enhancing our collections    vr transcripts have been used to help  identify gaps in library collections and  support acquisitions requests and decisions.  for example, many of the questions related  to difficulty in finding business journals and  other resources, particularly those related to  leadership. enrollment in business‐related  courses at guelph has increased by 88% in  recent years (university of guelph college  of management & economics, 2007). the  questions indicated that guelph’s library  holdings had not kept up with the interest in  business and commerce. the business  librarians strongly recommended the  purchase of many new resources related to  evidence based library and information practice 2007, 2:2  78 their subject area, supported by tangible  evidence such as statistics from the vr  questions in this subject area. with many  different subject areas competing for limited  acquisitions funding, it is useful to have  tangible evidence to back up requests for  new resources; vr questions and statistics  can provide some of this evidence.    vr questions can also illustrate which  courses and assignments are creating  problems for students. when many students  in a particular course are asking questions in  vr, it becomes apparent that there are  library‐related problems or issues that need  to be addressed. in some cases, students  may benefit from a library instruction  session to help them with their assignments.  in others, assignments may contain  inaccurate information about the library; for  example, incorrect citations are sometimes  supplied or references made in course  outlines to materials not held by the library.  if  students are having difficulties creating  bibliographies, liaison librarians can help by  contacting faculty and offering some library  instruction, by ordering materials listed on  course outlines that are not in the library  collection, by letting faculty know about  incorrect citations listed on assignments, or  by altering reference staff to problems with  assignments and course outlines. if it  becomes apparent that students could  benefit from a library service or tool, such as  writing clinics, online tutorials, or refworks  (bibliographic software) classes, faculty can  be asked to let their students know about  these offerings.     5. enhancing the library as place    library signs are important. as jeannette  woodward notes, “signs are an absolutely  essential ingredient in your library’s success  formula.” (118). unfortunately, especially  considering that libraries are in the business  of information, signs are often inadequate:   they may be unprofessional looking;  sometimes there are too many; sometimes  there are too few; sometimes they are  confusing. sometimes all of those  descriptions apply. on flickr, the photo‐ sharing website, there are groups devoted  solely to library signs, many of them posted  as an example of what not to do. (for  example, see  http://www.flickr.com/groups/librarysignag e/). the vr transcript examination pointed  out some of the gaps in guelph’s library  signage. for example, even though students  were sitting at library computers a few feet  away from the centralized printers, they  could not figure out where they could print.  when one actually looked around the  library, existing signs were observed to be  small and obscured, and some of printers  had no signs at all!  other directional  questions being asked in vr point out other  signage needs as well—to more effectively  direct people to pencil sharpeners,  photocopiers, and classrooms.    one of the most important things learned  from the vr project relates back to the  discussion of language. like many libraries,  guelph has always displayed signs reading  “periodicals” and “indexes.” as discovered  through the students’ questions, they don’t  use those words; in fact, they may not  understand what those words mean. it is  time to change library signs, so that they  reflect what users say and understand. signs  for “periodicals” need to say “journals!”      conclusion    vr transcript analysis, particularly the  examination of the questions and the words  students use to ask them, has provided a  tremendous amount of information and has  led to a change in some aspects of library  services at the university of guelph. future  work could develop the analysis further to  include an analysis and comparison of the  more recently offered instant messaging  (msn) reference service. a comparison of  http://www.flickr.com/groups/librarysignag evidence based library and information practice 2007, 2:2  79 the ways in which students use the two  services, the language they use in each, and  the effectiveness of each type of service may  also provide information about how we can  better meet the needs of our users. by  reading students’ words, librarians are  really trying to listen to what they are  saying.    acknowledgement    this article is based on a presentation from  the “computers in libraries 2006”  conference in washington, d.c.     the authors gratefully acknowledge the  assistance of caroline stewart, formerly a  co‐op student from the faculty of  information & media studies, university of  western ontario, and now a librarian at  seneca college.       works cited    broughton, kelly m. “usage and user  analysis of a real‐time digital  reference service.” the reference  librarian 79/80 (2003): 183‐200.    coffman, steve. going live: starting and  running a virtual reference service.  chicago: american library association,  2003.     curtis, donnelyn and araby green. “a  university‐wide, library‐based chat  service.” reference services review  32.3 (2004):  220‐33.    desai, christina m. “instant messaging  reference: how does it compare?”  electronic library 21.1 (2003): 21‐30.     diamond, wendy, and barbara pease.  “digital reference: a case study of  question types in an academic  library.” reference services review  29.3 (2001): 210‐219.     fagan jody condit, and christina m. desai.  “communication strategies for instant  messaging and chat reference  services.” the reference librarian  79/80 (2003): 121‐55.     foley, marianne. “instant messaging  reference in an academic library: a  case study.” college and research  libraries 63.1 (2002): 36‐45.    janes, joseph. “digital reference for teens.”  voice of youth advocates 25.6 (2003):  451.    katz, william a. introduction to reference  work, volume 1: basic information  service. new york: mcgraw‐hill, 1997.    kenney, brian. “live, digital reference.”  library journal 127.16 (2002): 46‐50.     kupersmith, john. library terms that users  understand. 30 may 2007  <http://www.jkup.net/terms.html>.    lee, ian j. “do virtual reference librarians  dream of digital reference questions?:  a qualitative and quantitative  analysis of email and chat reference.”  australian academic and research  libraries 35.2 (2004): 95‐110.    sears, joann. “chat reference service: an  analysis of one semesterʹs data.”  issues in science & technology  librarianship 32 (2001): 30 may 2007  http://www.library.ucsb.edu/istl/01‐ fall/article2.html    smyth, joanne. “virtual reference  transcript analysis: a few models.”  searcher 11.3 (2003): 26‐30.     http://www.jkup.net/terms.html http://www.library.ucsb.edu/istl/01%e2%80%90fall/article2.html http://www.library.ucsb.edu/istl/01%e2%80%90fall/article2.html evidence based library and information practice 2007, 2:2  80 university of guelph. college of  management and economics. 30 may  2007  http://www.uoguelph.ca/cme/current‐ students.cfm    walter, virginia, and cindy mediavilla.  “teens are from neptune, librarians  are from pluto: an analysis of online  reference transactions.” library  trends 54.2 (2005): 209‐227.    ward, david. “using virtual reference  transcripts for staff training.”  reference services review 31.1 (2003):  46‐56.                                                                  woodward, jeannette. creating the  customer‐driven library: building on  the bookstore formula. washington:  american library association, 2004.    zhuo, fu, mark love, and scott norwood.  “applying rusa guidelines in the  analysis of chat reference  transcripts.” college & undergraduate  libraries 13.1 (2006): 75‐88. http://www.uoguelph.ca/cme/current%e2%80%90students.cfm http://www.uoguelph.ca/cme/current%e2%80%90students.cfm http://www.uoguelph.ca/cme/current%e2%80%90students.cfm evidence summary   formal mentoring programs can provide organizational and professional benefits, but are few and far between in canadian academic libraries   a review of: harrington, m. r., & marshall, e. (2014). analyses of mentoring expectations, activities, and support in canadian academic libraries. college & research libraries, 75(6), 763-790. doi: http://dx.doi.org/10.5860/crl.75.6.763     reviewed by: lindsay alcock head, public services health sciences library memorial university of newfoundland st. john’s, newfoundland, canada email: lalcock@mun.ca   received: 17 feb. 2015   accepted: 12 may 2015      2015 alcock. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – to determine the extent to which formal and informal mentoring is present in canadian academic libraries and how such activities meet expectations for new librarians, practising librarians, and library administrators.   design – online surveys.   setting – canadian college and university libraries.   subjects – three groups were defined and surveyed: graduates from one canadian mlis program; librarians practising in a canadian academic library, and library administrators directing a canadian academic library.   methods – participants were selected using stratified, purposeful sampling and were invited to participate in an online survey in december 2010. three surveys were distributed in total; one for each of the three target groups defined. the surveys contained both closedand open-ended questions. students from one specific mlis program graduating in december 2010 and spring 2011 were directly invited to participate. an incentive was offered to the student group and the librarian group to encourage participation.   main results – mentoring programs in canadian colleges and universities are present but are largely informal. mentoring activities are positively correlated with student population, how long a mentor has worked professionally, and whether or not librarians had been mentored early in their careers. 83% of new graduates expected to be mentored when hired by a canadian academic library while less than one quarter of librarians and no administrators reported having similar expectations when starting their professional academic careers. over 50% of the students who responded to the survey reported that they had experienced some form of mentoring while completing their mlis, though that may be related to the cooperative placement component of their educational program. all respondents, with the exception of university administers, indicated that library colleagues are appropriate mentors, while academic librarians and administrators felt that the mentorship relationship would not benefit were the mentor also the supervisor.   respondents placed the greatest weight of importance on mentoring activities related to academic expertise, career guidance, psychosocial support and role models, while indicating that evaluation was not considered to have a significant place in the mentor-mentee relationship. networking, cultural complexities, general encouragement, and career counseling were seen as important mentorship aspects for both new graduates and practising librarians. administrators in both college and university libraries were less likely to support mentoring particularly in the areas of assisting with grant writing, evaluation, and career counseling. in general, administrators were reluctant to develop or support formal mentor programming even though a significant percentage agreed that it would benefit succession planning.   conclusion – new graduates and practising librarians expect to participate in some form of mentoring activities yet there are very few canadian academic libraries providing formal mentorship programming. the value of the mentor-mentee relationship with respect to organizational planning, recruitment and retention, as well as career planning, is perceived as high amongst new graduates, practising librarians, and administrators.   commentary   the authors provide a thorough literature review on librarian mentoring activities in academic institutions, noting a gap in the literature in terms of formal and informal mentoring programs in canadian academic libraries. this is especially relevant given the importance placed on the 8rs study (2005) that highlights recruitment, retention, and restructuring, which the authors view as directly intertwined with mentoring. using a critical appraisal checklist (glynn, 2006), the study is determined to be valid.   three data collection instruments for three different groups provided the authors with a large amount of data that was compared and described clearly, thus providing a multi-faceted view of the varying perceptions of expectations and the importance placed on mentoring. it is unclear whether or not the surveys were validated or exactly how they were distributed, however the surveys were included thus allowing further transparency. the response rates for the administrators and practising librarians was stated, however no information was provided as to the expected response rate or the power of the study. one may surmise that the exact number of practising librarians and directors was known to the authors in order to calculate an exact response rate, though this information is not clear. given the low response rate and the use of purposeful stratified sampling (typically used for small sample sizes), it may not be possible to generalize results to the broader population.   the choice to survey recent graduates from one specific program is both interesting and controversial. while the program was not directly identified, it is easy to glean which one it is given the author’s description of the program. knowing both the institution as well as the graduating class dates compromises the privacy of this particular study group. in addition, it is possible that students attending a program with a strong placement component may respond differently than students from other programs, thus presenting a potential bias. the authors note this as a study limitation, implying that external validity may have been affected.   the perceived value of mentoring is clear for the new graduates and practising librarians in terms of academic expertise and career guidance, yet the primary benefits for these two groups relate to psychosocial support and role modeling. it is interesting that administrators did not indicate strong support for formal or informal mentoring programs in spite of the fact that librarians at universities reported more informal mentoring activities. it may be that administrators are unaware of informal mentoring activities amongst librarians. that said, administrators provided valid reasons for not supporting formal programs including a lack of available time and resources, confines related to collective agreements, difficulty matching mentors with mentees, and the general absence of a structured program with defined outcomes. without formal mentoring programs in place, informal mentoring practices depend on individual time, interest and commitment to continue. the conundrum presented is that this and other research points to the fact that mentoring has a positive effect on organizational restructuring and succession planning, suggesting that it may be in a library’s best interest to encourage and support such formal programming.   references   8rs research team. (2005) the future of human resources in canadian libraries. edmonton ab: university of alberta.   glynn, l. (2006). a critical appraisal tool for library and information research. library hi tech, 24(3), 387-399. doi: http://dx.doi.org/10.1108/07378830610692154      article   information literacy articles in science pedagogy journals   cara bradley teaching and learning librarian university of regina regina, saskatchewan, canada email: cara.bradley@uregina.ca   received: 24 july 2013    accepted: 19 nov. 2013      2013 bradley. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 2.5 canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – this study sought to determine the extent to which articles about information literacy-related topics have been published in science pedagogy journals. it also explored the nature of these references, in terms of authorship, association of college and research libraries (acrl) information literacy competency standards addressed, and degree of emphasis on information literacy topics. in addition to characterizing information literacy in the science pedagogy literature, the study presents a methodology that can be adopted by future efforts to explore representations of information literacy in the literature of additional academic disciplines. methods – the 2011 journal citation reports® science edition was used to identify the 15 journals with the highest impact factor in the “education—scientific disciplines” subject category. initially web of science was searched to identify occurrences of “information literacy” and related terms in the journals of interest during the 10 year period 2002-2011. this was supplemented by a title scan of the articles to ensure inclusion of relevant items that did not include library-centric terminology. abstracts and, where necessary, full papers were reviewed to confirm relevance. only articles were included: editorials, news items, letters, and resource reviews were excluded from the analysis. articles selected for inclusion were read in their entirety. professional designations for each author were identified to characterize the authorship of this body of literature. articles were also classified according to levels developed by o’connor (2008), to indicate whether information literacy was a “major topic,” “substantive focus,” “incidental mention,” or “not explicitly named.” further analysis mapped each article to the acrl information literacy competency standards (2000), to provide more detailed insight into which standards are most frequently addressed in this body of literature. results – articles on information literacy-related topics appear only sporadically in science pedagogy journals, and that frequency varies depending on the specific subject area. overall, librarians contribute a relatively small proportion of these articles, and are more likely to co-author with teaching faculty/graduate students than to publish alone or with other librarians. the degree of focus on information literacy topics (o’connor level) varies depending on article authorship, with librarians more likely to treat information literacy as the “major focus” of their work. additionally, the articles tend to cluster around acrl information literacy standards two, three, and especially four, rather than addressing them equally. conclusions – the presence of some articles on information literacy-related topics in science pedagogy journals suggests that there is a willingness among these journals to publish work in this area. despite this, relatively few librarians have pursued this publication option, choosing instead to publish articles on information literacy topics within the library and information studies (lis) literature. as a result, librarians are missing out on the opportunity to share their published work in venues more likely to be seen and valued by subject faculty, and on the chance to familiarize science educators with information literacy topics. future research should focus on: librarians’ rationale when selecting target publications for their information literacy writing; science educator interest in writing and reading about information literacy topics in their pedagogical journals; and the impact of articles about information literacy in these journals on subject faculty perceptions of the topic’s importance. the methods used in this research have implications for the study of information literacy in other academic disciplines, and demonstrate that the study of information literacy in the literature of academic disciplines can provide valuable insights into representations and characterizations of information literacy in diverse fields of study. a better understanding of how subject faculty think and write about information literacy in their scholarly literature could have a significant impact on how librarians approach and collaborate with faculty in all fields of study.   introduction   thousands of journal articles, books, standards, and other documents have been written on the topic of information literacy over the past two decades. library and information studies (lis) venues have published the vast majority of this work, where it is read primarily by librarians with a pre-existing interest in the topic. this body of literature is certainly important, as it has promoted information literacy to a wider lis audience and helped to refine the profession’s understanding of the concept.   of particular interest to the current study, however, is the degree to which information literacy has permeated the pedagogical literature in the academic disciplines; that is, the literature that is most likely to be read by teaching faculty. research has repeatedly demonstrated that curricular integration of information literacy competency development is essential to its success, and such integration cannot happen without the willing participation of faculty teaching in the disciplines (kearns & hybl, 2005; lampert, 2005). this study approaches pedagogical literature in the disciplines as one measure of the interest in and uptake of information literacy among educators. in addition to quantifying information literacy’s presence in the science pedagogical literature, it also attempts to characterize the nature of this work. it is hoped that the results will provide guidance and insight to librarians, whether they are considering publication venues for their own information literacy writing, or trying to identify connection points between information literacy and pedagogical discussions in the disciplines.   literature review   in 1992, jacobson and vallely undertook a study to determine the prevalence and authorship of articles about “library instruction” in the “journals that faculty members read” (p. 360). they did not specify any subject area limitations in their research, and the databases searched indicate that they included a wide range of disciplines. the term “information literacy” was not yet in widespread use in the period covered by their search (1980-1990), so they used “bibliographic instruction” and other keywords and subject headings (outlined in detail in appendix a of their article). they found 74 articles about library instruction in non-library journals, with approximately 50% written by librarians alone, 25% written collaboratively by librarians and faculty members, and another 25% authored by faculty members alone. jacobson and vallely expressed general disappointment, not only with the small numbers, but also with the quality of the articles retrieved, noting that there was “not much . . . novel or surprising” (p. 360) in the articles by librarians, and that faculty-authored articles revealed, “a remarkably superficial notion of who we [librarians] are and what we do” (p. 362). they ended with a call for librarians to increase publication about the value of library instruction in journals read by faculty members.   still’s 1998 article followed six years after jacobson and vallely’s early effort to use non-library literature as a barometer for interest in and uptake of library instruction. like her predecessors, still looked at subject-specific pedagogical journals across disciplines, and found that only 33 of 13,016 articles discuss library instruction or library-related assignments. she highlighted specific articles within four broad subject categories: sciences, humanities, social sciences, and nursing/social work, but did not characterize the literature in any systematic way. in sciences, the category most relevant to the present study, she lauded the creation of the “chemical information instructor” column, edited by a librarian, in the journal of chemical education. her conclusion, however, was sobering: “if the library and library instruction have been integrated into the academic curriculum, there is little evidence of it in the discipline specific teaching journals studied” (still, 1998, p. 229).   nearly a decade after still’s article, stevens (2007) was the first author to analyze discipline-specific pedagogical literature in the era of widespread adoption of the term “information literacy” and the acrl information literacy competency standards. even with the broadening of her focus from “library instruction” to “information literacy,” stevens found only 25 information literacy articles published from 2000-2005 in the 54 pedagogy journals included in her study. like jacobson and vallely, stevens was particularly interested in the authorship of these 25 articles, and found that 7 were written by librarians, 12 were faculty/librarian collaborations, and 6 were written exclusively by faculty. she concluded that, while information literacy had not made significant inroads into the disciplinary pedagogy literature overall, there were some bright spots. she noted the growing presence of information literacy in the nursing pedagogy literature, presenting it as an example that illustrates the value of publishing this work in disciplinary journals. stevens was also the first author to mention the acrl standards in her analysis; while she didn’t delve down to use of specific standards, she did note that some articles, “use the acrl standards as a framework for defining il competencies, designing assignments, and assessing student learning” (p. 262). ultimately, like jacobson and vallely and also stills, stevens concluded with a call for librarians (either alone or collaboratively with faculty) to capitalize on the potential of discipline-specific pedagogy journals to interest faculty in information literacy.   o’connor (2008) was the first, and to date only, author to conduct a more in-depth study of information literacy in the literature of a specific discipline. she searched business literature (broadly, not just pedagogical journals) in order to assess the “diffusion” of information literacy in business studies. she located 159 relevant works (unlike previous studies, o’connor included trade publications in addition to scholarly journals) and her analysis revealed that disappointingly few were written by librarians. she also developed and applied a scale for delineating the extent to which the works addressed information literacy topics, a scale that has been adopted for the current study. she found that most of the information literacy articles she had identified gave the topic “incidental mention,” although there were also a significant number in which information literacy was the “major focus.” o’connor’s application of rogers’ diffusion of innovations theory led her to the conclusion that the low and relatively stable level of information literacy publications appearing in the business literature over time indicated that it is in the “earliest phases of adoption . . . and has not yet reached the tipping point” (2008, p. 120).   the present study, like o’connor’s work, is based on the premise that detailed examination of the literature of specific fields provides a deeper and more nuanced understanding of information literacy, while also recognizing that there is value in being able to conduct some comparisons among similar fields. it shares earlier authors’ interest in the number of information literacy-related articles appearing in non-lis literature over time and their curiosity about the authorship of these articles. it also offers a deeper level of analysis, not only characterizing this body of literature by applying o’connor’s levels, but also, for the first time in this type of study, mapping journal articles to the acrl information literacy competency standards.   aims   this study aims to answer the following research questions: to what extent are information literacy competencies addressed by science pedagogy journals? what is the nature of these references, in terms of authorship, acrl standards addressed, and degree of emphasis on information literacy?   methods   the term “information literacy,” despite its widespread use among librarians, lacks a single, accepted definition. professional associations in different countries have variously defined it as “knowing when and why you need information, where to find it, and how to evaluate, use and communicate it in an ethical manner” (chartered institute of library and information professionals, 2004), or as being “able to recognize when information is needed and have the ability to locate, evaluate, and use effectively the needed information” (american library association, 1989). the author of this study used the acrl’s five competency standards as the basis for her definition and conceptualization of information literacy, primarily because they have been so widely adopted in the north american academic library sector where she works.   the researcher used the 2011 journal citation report (jcr) science® edition to identify high impact journals in the “education—scientific disciplines” subject category. while the problems inherent in using journal impact factors as a proxy for journal quality have been well-documented (lozano, larivière, & gingras 2012; mcveigh & mann 2009, among others), this provided a convenient way to identify journals across scientific disciplines whose reach is significant, thereby serving the purpose of this study.  the analysis included the 15 non-medical journals with the highest impact factors, and the study considered articles published in the most recent 10-year period for which complete data was available (2002-2011), with the sample obtained in april 2013. only articles were included; editorials, news items, letters, and resource reviews were excluded from the analysis. this created a large pool of 10,743 articles for analysis, providing broad coverage across scientific disciplines, and covering a time period of great change in information retrieval and usage practices. the study used the online versions of the articles, relying on print only in cases of missing content or access restrictions.   efforts to identify articles from this pool of 10,743 publications that address information literacy-related topics were two-fold. the researcher initially conducted keyword searches of web of science to identify occurrences of the terms “information litera*” or “information fluen*” or “library instruction” or “information competen*” from within the large pool of articles. she then supplemented this by personally scanning the titles and, where necessary for clarification, the abstracts and/or full text, of the original pool of 10,743 articles. this ensured the inclusion of relevant articles that did not contain library-centric terminology in describing information-literacy related concepts, as well as articles that focused on specific competencies (such as plagiarism, searching) without including an umbrella information literacy term. in an attempt to ensure consistency and quality control in the selection process, the author selected only articles that could be correlated to specific components of the acrl competencies/ performance indicators/outcomes. additionally, articles had to focus on inculcating these competencies in students, rather than mentioning them in other contexts. for example, an article about instructor strategies for detecting plagiarism was also excluded as the focus was not on educating students about the topic.   the researcher read articles selected for inclusion in their entirety to ensure accurate categorization. she created a standardized template in excel and extracted data from each article as it was read. the entry for each article included details about the professional designations for every author in order to characterize the authorship of this body of literature; in cases where this information was not provided in the article, the researcher located it through web searches and, in a small number of cases, email follow-up. the template also required entry of the publication year and the broad scientific subject area of the journal in which each article was published. additionally, the template also required that the researcher classify each article according to the levels developed by o’connor to indicate the nature of the work’s attention to information literacy concepts. the four levels are:   texts in which il [information literacy] is explicit and a major focus (il–major topic); texts in which il is explicit and treated substantively, but is not the focus of the article (il-substantive treatment); texts in which il is explicit, but only mentioned in passing, possibly with a very brief definition provided (il–incidental mention); and texts in which il competencies are clearly being discussed, yet il is never explicitly named. (o’connor 2008, p. 113)   finally, the template required the researcher to assign each article to the appropriate acrl information literacy competency standards for higher education (2000) to provide more detailed insight into which standards are most frequently addressed in this body of literature. the researcher coded articles as addressing up to four individual standards, while coding those addressing all five standards or information literacy generally as “il—general.”   results   a total of 10,743 journal articles met the criteria of articles published from 2002 to 2011 in the target 15 journals. the researcher first reviewed article titles in order to determine relevance of the papers, which revealed that the vast majority of these did not address information literacy topics in any notable way. in 430 instances where titles were ambiguous or suggested information literacy-related content, the researcher read abstracts to glean a better understanding of the article. this allowed further refinement of the article set, and left 218 articles to be read in their entirety. ultimately, only 156 of the original 10,743 articles (or 1.5%) addressed information literacy-related topics. the names of the journals included in the analysis, the number of citations under review from each journal, and the number/percentage of information literacy-related references found in each journal are outlined in table 1.  these numbers clearly indicate that articles on information literacy-related topics appeared only sporadically in science pedagogy journals.   table 1 information literacy articles by journal journal total articles number of information literacy articles % of information literacy articles out of total journal of engineering education 228 6 2.6% advances in physiology education 358 15 4.2% studies in science education 52 1 1.9% cbe—life sciences education 325 18 5.5% ieee transactions on education 659 13 2.0% physical review special topics—physics education research* 143 0 0 journal of science education and technology 459 13 2.8% chemistry education research and practice 277 4 1.4% biochemistry and molecular biology education 566 19 3.4% european journal of physics 1213 0 0 journal of chemical education 3057 39 1.3% american journal of physics 1702 3 .2% international journal of engineering education 1260 8 .6% international journal of technology and design education 198 7 3.5% journal of biological education 246 10 4.1% total 10,743 156 1.5% * publication began in 2005   journals were also grouped according to specific scientific discipline, based on “subject category” assigned in the 2011 journal citation report (jcr) science® edition, in an effort to uncover any differences in the frequency of information literacy articles by subject area. figure 1 shows that, of the scientific disciplines represented in the 15 journals under review, biology/life sciences journals were most likely to have addressed information literacy topics (4.2% of articles). science (general), chemistry, and engineering journals published somewhat fewer articles on information literacy topics, and information literacy articles were virtually non-existent in the physics education literature, with only .2% of journal articles under review addressing the topic.   the researcher also analyzed journal articles to determine the level or depth with which they focus on information literacy topics. application of the levels developed by o’connor (1998) revealed that, when addressed in science pedagogy journals, information literacy is most frequently the “major topic” of a journal article, with o’connor’s category “il substantive treatment” a close second (figure 2).  this is not to imply that the term “information literacy” itself was used in the articles; in fact, this phrase is absent in the vast majority of articles. instead, it indicates that the concept or its constituent parts (as articulated in the acrl information literacy competency standards for higher education) were represented at the specified level. thus, the category “il not explicit” does not refer to the absence of the term “information literacy,” but most often indicates that a learning activity was developed to foster several skills, of which information literacy is one.   the researcher further broke down the journal articles on information literacy-related topics by publication year over the 10 year period. figure 3 reveals a general increase in the number of articles addressing information literacy topics from 2002-2008. after peaking in 2008, the number of articles on information literacy topics declined precipitously in 2009, followed by a more gradual decrease in 2010 and 2011. figure 3 also reveals that over the years, information literacy topics have become increasingly likely to be a “major focus” of journal articles, whereas in the past there was a more even split in the depth with which articles addressed the topic.   figure 1 information literacy articles by subject area.   figure 2 number of articles by o'connor level.     analysis of authorship patterns of the information literacy-related journal articles revealed that subject faculty and/or graduate students wrote the vast majority of these works. figure 4 shows that librarians (writing either alone or with other librarians) wrote only 4 of the 156 information literacy-related articles. collaborations between librarians and subject faculty/graduate students were somewhat more productive, resulting in 13 articles. one unexpected finding was the number of publications written by educational developers and other teaching centre employees (either alone or in conjunction with subject faculty/graduate students). this category was not included in the initial analysis but was added when it become apparent that these staff contributed information literacy-related articles in numbers comparable to librarians.   results also revealed that the o’connor level of articles varied depending on the authorship of the article. articles written by librarians (either alone or in collaboration with others) were much more likely to address information literacy topics in depth, as demonstrated in table 2. table 2 also shows that articles written by those other than librarians were spread much more evenly over the o’connor levels, particularly “major focus,” “substantive treatment,” and “incidental mention.”   the researcher also categorized the articles under review according to the acrl information literacy competency standards that they addressed. she assigned up to four standards for each article as applicable, assigning articles addressing all five standards or information literacy generally to the category “il--general.” standard two (accessing information effectively and efficiently), standard three (evaluation of information) and particularly standard four (using information to accomplish a specific purpose) were most frequently the topic of the articles under review. standard five (ethical and legal use of information) was addressed less often, and standard one (identifying an information need) was only infrequently the focus of articles in the science pedagogy literature.   figure 3 articles by publication year and o'connor level.   figure 4 authorship of articles.   table 2 o'connor level of articles by authorship   librarians (#) librarians (%) non-librarians (#) non-librarians (%) major focus 16 94% 48 35% substantive treatment 1 6% 52 37% incidental mention 0 _ 32 23% not explicitly named 0 _ 7 5%   figure 5 acrl standards addressed by articles. note: as some articles address more than one standard, the sum of the numbers in this chart exceeds the 156 articles categorized.     discussion   the number of information literacy-related journal articles published in the science pedagogy literature is quite small and in this regard similar to findings in earlier studies of information literacy in the non-lis pedagogical literature. it is difficult, however, to draw more detailed comparisons between these results and those of previous studies because their searching practices did not provide a denominator that gave a sense of the percentage of articles that were information literacy-related.  the one study by still (1998) that did determine that less than .5% of articles studied addressed information literacy initially seems to suggest that the current study’s findings of 1.5% of articles in the science pedagogical literature may be an improvement. in fact, the present study’s use of the broader concept of information literacy (rather than still’s use of library instruction), as well as its supplementation of keyword searches with more inclusive title/abstract scans, may in fact mask a decline in the overall percentage of articles being published on information literacy topics in pedagogical literature. despite the difficulties of comparing rates between studies, it can be claimed with certainty that the rate of information literacy-related articles in non-lis journals always has been and remains disappointingly low.   variation in the prevalence of information literacy-related journal articles among the specific scientific disciplines represented in the journal set are likely the result of several factors.  undoubtedly, some of these are complex issues embedded in the nature of the disciplines themselves; awareness of differences among practices and beliefs in different subject areas has been growing ever since the 1989 publication of becher’s book, academic tribes and territories: intellectual enquiry and the culture of disciplines. although the disciplines included in this study are all “sciences,” there are likely deep-rooted differences in the ways that researchers use the literature of the field, as well as varied expectation levels about the information literacy capacity of incoming students. some disciplines may simply not view developing student information literacy levels as part of a university instructor’s responsibility. on a more pragmatic level, some differences in information literacy-related publication levels seem to be related to publication practices of the journals in the various subject areas. “special” topic issues of journals, examples of which include a “special issue on plagiarism” from ieee transactions on education, and regular columns (for example “chemical information instructor” in the journal of chemical education) account for a significant proportion of the relevant articles published by these journals. future research, focussed on developing a better understanding of the different perceptions of information literacy among faculty in these disciplines, may provide insight into how information literacy can have more of an impact, on both teaching practices and the pedagogical literature in these fields of study.   the data revealed that librarians contribute only a small proportion of the information literacy-related articles in science pedagogy journals, with subject faculty or graduate students responsible for the lion’s share of this work. this finding suggests that interest in the topic extends beyond the lis sphere, and to at least some of our target audience of teaching faculty in the disciplines. viewed less positively, it also indicates that librarians may not be leading the charge (or even be visible) in efforts to improve student information literacy levels and advocate for the importance of these competencies. the few faculty or graduate student-authored articles that do mention a librarian simply do so in passing “after the librarian taught students how to x,” or credit them in the acknowledgements section of the paper. without further follow-up, it is unclear whether this is because librarians declined further involvement in the writing of the journal article, or because they are regarded simply as a “service” rather than an academic partner. it is, however, an important question for future research to ask as it gets to the heart of how librarians see themselves and are seen by teaching faculty as contributing to the teaching mission of the university.   librarian collaboration with subject faculty and graduate students was more fruitful in terms of publication output than librarians working alone or with other librarians. as thorough course integration is integral to the success of information literacy competency development (acrl, 2012), the collaborative librarian/faculty projects documented in these publications are a positive development. the wide variation of o’connor level by authorship (librarians involved in articles in which information literacy is a “major focus” vs. a more even spread among “major focus,” “substantive treatment” and “incidental mention” in non-librarian articles) is telling of the difference perspectives on information literacy held by these populations. it suggests that librarians treat information literacy as a standalone activity, while subject faculty view and write about it as an integrated component of coursework and the larger curriculum. the unexpected finding that educational developers and those working in teaching centres contribute a significant portion (more than librarians writing alone) of the information literacy-related articles in these journals suggests that this is another campus group with whom librarians could be collaborating, both in terms of program development/delivery and of co-authorship.   the heavy focus on acrl information literacy competency four, “uses information effectively to accomplish a specific purpose,” in the journal articles under review was interesting. standard two, “searching for information,” is the information literacy element that librarians “most often address and teach within our professional domain, as illustrated by the large body of professional literature addressing methods for teaching and assessing skills in information search and retrieval” (adams, in press, p. 11).  while standards two and three are addressed by a large number of the journal articles, it is standard four that receives the most attention in these works. this illustrates a disconnect between subject faculty emphasis and that of librarians, as there is little evidence in the literature that librarians at large engage in teaching or assessment related to this information literacy competency. the reasons for and implications of this finding are unclear but important; it may mean that librarians and/or subject faculty view standard four as best left to the subject experts, or it could suggest that librarians and faculty members have not figured out the best way to collaborate on student development of this competency. further research into the roles around this standard may help subject faculty to recognize potential librarian contributions to the development of this competency, as well as help them to situate it in the larger context of information literacy.   it is difficult to draw connections and see trends between this study and past works because the concept of interest (“information literacy”) did not exist or was not in widespread use when past works explored the narrower notion of “library instruction.” as well, the decision to scan all article titles in addition to searching for specific terms changes the nature of the study; while it helps to illustrate the occurrence of the information literacy concept independent of variations in terminology, it does make it almost impossible to compare the occurrence of information literacy in this study with findings of previous works. another limitation to this study is that a single researcher conducted the searches and screened the titles; while efforts were made to apply standard criteria, a second selector may have reduced any potential bias or consistencies. it should also be noted that journal articles in the science pedagogy literature are just one measure of information literacy uptake in the sciences; there are undoubtedly instances of information literacy competency development that are not written up in articles in traditional journals (such as editorials, news items, letters, and resource reviews, which were all excluded from this analysis), or are published in other formats (e.g., news items, conferences, blogs). finally, authorship is just one way that librarians may contribute to information literacy articles; passing references to their contributions to projects, or acknowledgements, are not accounted for in this study.   conclusions   this paper set out to discover the extent to which science pedagogy journals address information literacy competencies, and to characterize these information literacy articles in terms of authorship, acrl standards addressed, and level of detail. results indicate that information literacy has a very low profile in science pedagogy journals overall, with only 1.5% of articles addressing information literacy competencies. perhaps even more concerning is the fact that the number of information literacy-related articles in these journals appears to be declining over recent years, suggesting that more needs to be done to keep information literacy on the minds of those reading these publications.   the study also reveals some potential avenues to increase the prevalence of information literacy publications in these journals. rates varied by specific scientific field, from a high of 4.2% in the biological/life sciences, to a low of .2% in physics. further exploration of the reasons for the relatively high levels in some subject areas may provide clues to increasing interest in information literacy in other subjects where levels were low. the study also found that special issues of journals as well as dedicated columns help to increase the number of information literacy-related journal articles, findings that may encourage those interested in the topic to seek these publication opportunities, or even to suggest or implement them themselves in science pedagogy journals.   findings also highlighted tensions between librarian and subject faculty conceptions of information literacy. subject faculty, sometimes aided by graduate students, have written the vast majority of articles about information literacy in these journals, although they very rarely adopt the lis term “information literacy” to describe their work. terminology is not the only difference; while librarians tend to write articles about information literacy as a “standalone” major topic, subject faculty more often favour “substantive treatment” or “incidental mention,” characterizing information literacy as one component of student learning experiences. librarians and subject faculty also seem to focus on different information literacy competencies, with the latter particularly interested in “using information effectively.”   while this study focused on information literacy in the science pedagogy literature, it may also have implications for other disciplines. the methods used in this work suggest that the study of information literacy in the literature of academic disciplines can provide valuable insights into representations and characterizations of information literacy within diverse fields of study. future research into information literacy in the literature of other fields of study has the potential to illuminate faculty perceptions across disciplines. it would be interesting to learn if the science literature paints an entirely different picture of information literacy than that found, for instance, in the humanities or social sciences. a better understanding of how subject faculty think and write about information literacy in their scholarly literature could have a significant impact on how librarians approach and collaborate with faculty in all fields of study.   librarians need to redouble their efforts to publish and raise the profile of information literacy in science pedagogy journals, either alone or collaboratively with subject faculty and even with educational developers/teaching centre staff. in doing so, they need to remain mindful that the term “information literacy” may not resonate with those outside of lis, ensuring that their profession’s preferred language doesn’t become a barrier to collaboration. librarians need to extend the reach of their information literacy work by using language and publishing in venues that will turn the lis profession’s information literacy monologue into a dialogue with subject faculty.     references   adams, n. e. (in press). a comparison of evidence-based practice and the acrl information literacy standards: implications for information literacy practice. college and research libraries. advance online publication, retrieved 12 jul. 2013 from http://crl.acrl.org/content/early/2012/12/19/crl12-417.full.pdf+html   american library association. (1989).  presidential committee on information literacy: final report. chicago, il: american library association. retrieved 12 jul. 2013 from http://www.ala.org/acrl/publications/whitepapers/presidential   association of college and research libraries. (2000). information literacy competency standards for higher education. chicago, il: american library association.  retrieved 12 jul. 2013 from http://www.acrl.org/ala/mgrps/divs/acrl/standards/standards.pdf   association of college and research libraries. (2012). characteristics of programs of information literacy that illustrate best practices: a guideline. chicago, il: american library association. retrieved 12 jul. 2013 from http://www.ala.org/acrl/standards/characteristics      becher, t. (1989). academic tribes and territories: intellectual enquiry and the culture of disciplines. milton keynes: the society for research into higher education and open university press.   chartered institute of library and information professionals. (2004). “information literacy: definition.”  chartered institute of library and information professionals. retrieved 12 jul. 2013 from http://www.cilip.org.uk/cilip/advocacy-campaigns-awards/advocacy-campaigns/information-literacy/information-literacy   jacobson, t. e., & vallely, j. r. (1992). a half-built bridge: the unfinished work of bibliographic instruction. the journal of academic librarianship, 17(6), 359-363.   kearns, k., & hybl, t. t. (2005). a collaboration between faculty and librarians to develop and assess a science literacy laboratory module. science & technology libraries 25(4): 39-59. doi: 10.1300/j122v25n04_04   lampert, l. (2005). “getting psyched” about information literacy: a successful faculty-librarian collaboration for educational psychology and counseling. the reference librarian 43(89/90): 5-23. doi: 10.1300/j120v43n89_02   lozano, g. a., larivière, v., & gingras, y. (2012). the weakening relationship between the impact factor and papers' citations in the digital age. journal of the american society for information science and technology 63(11), 2140–2145. doi: 10.1002/asi.22731   mcveigh, m. e., & mann, s. j. (2009). the journal impact factor denominator: defining citable (counted) items.  journal of the american medical association 302(10), 1107-1109. doi:10.1001/jama.2009.1301   o’connor, l.g. (2008). the diffusion of information literacy in academic business literature. journal of business and finance librarianship 13(2), 105-125. doi: 10.1300/15470640802119455   stevens, c. r. (2007). beyond preaching to the choir: information literacy, faculty outreach, and disciplinary journals. the journal of academic librarianship 33(2), 254-267. doi: 10.1016/j.acalib.2006.08.009   still, j.  (1998). the role and image of the library and librarians in discipline-specific pedagogical journals. the journal of academic librarianship 24(3), 225-231. doi: 10.1016/s0099-1333(98)90043-x   <html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/tr/rec-html40"> <head> <meta http-equiv=content-type content="text/html; charset=unicode"> <meta name=progid content=word.document> <meta name=generator content="microsoft word 15"> <meta name=originator content="microsoft word 15"> <link rel=file-list href="art_hebert_29404_files/filelist.xml"> <link rel=edit-time-data href="art_hebert_29404_files/editdata.mso"> <!--[if !mso]> <style> v\:* {behavior:url(#default#vml);} o\:* {behavior:url(#default#vml);} w\:* {behavior:url(#default#vml);} .shape {behavior:url(#default#vml);} </style> <![endif]--><!--[if gte mso 9]><xml> <o:documentproperties> <o:author>mallan, katrine</o:author> <o:template>normal</o:template> <o:lastauthor>hinrichs, rachel</o:lastauthor> <o:revision>4</o:revision> <o:totaltime>10</o:totaltime> <o:created>2018-09-10t17:23:00z</o:created> <o:lastsaved>2018-09-10t17:34:00z</o:lastsaved> <o:pages>1</o:pages> <o:words>42417</o:words> <o:characters>241777</o:characters> <o:company>microsoft</o:company> <o:lines>2014</o:lines> <o:paragraphs>567</o:paragraphs> <o:characterswithspaces>283627</o:characterswithspaces> <o:version>16.00</o:version> </o:documentproperties> <o:officedocumentsettings> <o:allowpng/> </o:officedocumentsettings> </xml><![endif]--> <link rel=datastoreitem href="art_hebert_29404_files/item0009.xml" target="art_hebert_29404_files/props010.xml"> <link rel=themedata href="art_hebert_29404_files/themedata.thmx"> <link rel=colorschememapping href="art_hebert_29404_files/colorschememapping.xml"> <!--[if gte mso 9]><xml> <w:worddocument> <w:trackmoves>false</w:trackmoves> <w:trackformatting/> <w:punctuationkerning/> <w:validateagainstschemas/> <w:saveifxmlinvalid>false</w:saveifxmlinvalid> <w:ignoremixedcontent>false</w:ignoremixedcontent> <w:alwaysshowplaceholdertext>false</w:alwaysshowplaceholdertext> <w:donotpromoteqf/> <w:lidthemeother>en-ca</w:lidthemeother> <w:lidthemeasian>x-none</w:lidthemeasian> <w:lidthemecomplexscript>x-none</w:lidthemecomplexscript> <w:compatibility> <w:breakwrappedtables/> <w:snaptogridincell/> <w:wraptextwithpunct/> <w:useasianbreakrules/> <w:useword2010tablestylerules/> <w:dontgrowautofit/> <w:splitpgbreakandparamark/> <w:enableopentypekerning/> <w:dontflipmirrorindents/> <w:overridetablestylehps/> </w:compatibility> <w:browserlevel>microsoftinternetexplorer4</w:browserlevel> <m:mathpr> <m:mathfont m:val="cambria math"/> <m:brkbin m:val="before"/> <m:brkbinsub m:val="&#45;-"/> <m:smallfrac m:val="off"/> <m:dispdef/> <m:lmargin m:val="0"/> <m:rmargin m:val="0"/> <m:defjc m:val="centergroup"/> <m:wrapindent m:val="1440"/> <m:intlim m:val="subsup"/> <m:narylim m:val="undovr"/> </m:mathpr></w:worddocument> </xml><![endif]--><!--[if gte mso 9]><xml> <w:latentstyles deflockedstate="false" defunhidewhenused="false" defsemihidden="false" defqformat="false" defpriority="99" latentstylecount="371"> <w:lsdexception locked="false" priority="0" qformat="true" name="normal"/> <w:lsdexception locked="false" priority="3" qformat="true" name="heading 1"/> <w:lsdexception locked="false" priority="3" semihidden="true" unhidewhenused="true" qformat="true" name="heading 2"/> <w:lsdexception locked="false" priority="3" semihidden="true" unhidewhenused="true" qformat="true" name="heading 3"/> <w:lsdexception locked="false" priority="3" semihidden="true" unhidewhenused="true" qformat="true" name="heading 4"/> <w:lsdexception locked="false" priority="3" semihidden="true" unhidewhenused="true" qformat="true" name="heading 5"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 6"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 7"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 8"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="heading 9"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 6"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 7"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 8"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index 9"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 1"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 2"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 3"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 4"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 5"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 6"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 7"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 8"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" name="toc 9"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="normal indent"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="footnote text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="annotation text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" qformat="true" name="header"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="footer"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="index heading"/> <w:lsdexception locked="false" priority="35" semihidden="true" unhidewhenused="true" qformat="true" name="caption"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table of figures"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="envelope address"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="envelope return"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" qformat="true" name="footnote reference"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="annotation reference"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="line number"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="page number"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="endnote reference"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="endnote text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table of authorities"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="macro"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="toa heading"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="list bullet"/> <w:lsdexception locked="false" priority="9" semihidden="true" unhidewhenused="true" qformat="true" name="list number"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list bullet 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list number 5"/> <w:lsdexception locked="false" priority="1" qformat="true" name="title"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="closing"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="signature"/> <w:lsdexception locked="false" priority="1" semihidden="true" unhidewhenused="true" name="default paragraph font"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text indent"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="list continue 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="message header"/> <w:lsdexception locked="false" priority="11" qformat="true" name="subtitle"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="salutation"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="date"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text first indent"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text first indent 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="note heading"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text indent 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="body text indent 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="block text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="hyperlink"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="followedhyperlink"/> <w:lsdexception locked="false" priority="22" qformat="true" name="strong"/> <w:lsdexception locked="false" priority="3" qformat="true" name="emphasis"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="document map"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="plain text"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="e-mail signature"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html top of form"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html bottom of form"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="normal (web)"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html acronym"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html address"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html cite"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html code"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html definition"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html keyboard"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html preformatted"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html sample"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html typewriter"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="html variable"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="normal table"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="annotation subject"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="no list"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="outline list 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="outline list 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="outline list 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table simple 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table simple 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table simple 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table classic 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table classic 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table classic 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table classic 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table colorful 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table colorful 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table colorful 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table columns 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 6"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 7"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table grid 8"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 4"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 5"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 6"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 7"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table list 8"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table 3d effects 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table 3d effects 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table 3d effects 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table contemporary"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table elegant"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table professional"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table subtle 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table subtle 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table web 1"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table web 2"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table web 3"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="balloon text"/> <w:lsdexception locked="false" priority="39" name="table grid"/> <w:lsdexception locked="false" semihidden="true" unhidewhenused="true" name="table theme"/> <w:lsdexception locked="false" semihidden="true" name="placeholder text"/> <w:lsdexception locked="false" priority="3" qformat="true" name="no spacing"/> <w:lsdexception locked="false" priority="60" name="light shading"/> <w:lsdexception locked="false" priority="61" name="light list"/> <w:lsdexception locked="false" priority="62" name="light grid"/> <w:lsdexception locked="false" priority="63" name="medium shading 1"/> <w:lsdexception locked="false" priority="64" name="medium shading 2"/> <w:lsdexception locked="false" priority="65" name="medium list 1"/> <w:lsdexception locked="false" priority="66" name="medium list 2"/> <w:lsdexception locked="false" priority="67" name="medium grid 1"/> <w:lsdexception locked="false" priority="68" name="medium grid 2"/> <w:lsdexception locked="false" priority="69" name="medium grid 3"/> <w:lsdexception locked="false" priority="70" name="dark list"/> <w:lsdexception locked="false" priority="71" name="colorful shading"/> <w:lsdexception locked="false" priority="72" name="colorful list"/> <w:lsdexception locked="false" priority="73" name="colorful grid"/> <w:lsdexception locked="false" priority="60" name="light shading accent 1"/> <w:lsdexception locked="false" priority="61" name="light list accent 1"/> <w:lsdexception locked="false" priority="62" name="light grid accent 1"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 1"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 1"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 1"/> <w:lsdexception locked="false" semihidden="true" name="revision"/> <w:lsdexception locked="false" priority="34" qformat="true" name="list paragraph"/> <w:lsdexception locked="false" priority="29" qformat="true" name="quote"/> <w:lsdexception locked="false" priority="30" qformat="true" name="intense quote"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 1"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 1"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 1"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 1"/> <w:lsdexception locked="false" priority="70" name="dark list accent 1"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 1"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 1"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 1"/> <w:lsdexception locked="false" priority="60" name="light shading accent 2"/> <w:lsdexception locked="false" priority="61" name="light list accent 2"/> <w:lsdexception locked="false" priority="62" name="light grid accent 2"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 2"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 2"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 2"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 2"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 2"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 2"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 2"/> <w:lsdexception locked="false" priority="70" name="dark list accent 2"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 2"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 2"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 2"/> <w:lsdexception locked="false" priority="60" name="light shading accent 3"/> <w:lsdexception locked="false" priority="61" name="light list accent 3"/> <w:lsdexception locked="false" priority="62" name="light grid accent 3"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 3"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 3"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 3"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 3"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 3"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 3"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 3"/> <w:lsdexception locked="false" priority="70" name="dark list accent 3"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 3"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 3"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 3"/> <w:lsdexception locked="false" priority="60" name="light shading accent 4"/> <w:lsdexception locked="false" priority="61" name="light list accent 4"/> <w:lsdexception locked="false" priority="62" name="light grid accent 4"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 4"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 4"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 4"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 4"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 4"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 4"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 4"/> <w:lsdexception locked="false" priority="70" name="dark list accent 4"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 4"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 4"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 4"/> <w:lsdexception locked="false" priority="60" name="light shading accent 5"/> <w:lsdexception locked="false" priority="61" name="light list accent 5"/> <w:lsdexception locked="false" priority="62" name="light grid accent 5"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 5"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 5"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 5"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 5"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 5"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 5"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 5"/> <w:lsdexception locked="false" priority="70" name="dark list accent 5"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 5"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 5"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 5"/> <w:lsdexception locked="false" priority="60" name="light shading accent 6"/> <w:lsdexception locked="false" priority="61" name="light list accent 6"/> <w:lsdexception locked="false" priority="62" name="light grid accent 6"/> <w:lsdexception locked="false" priority="63" name="medium shading 1 accent 6"/> <w:lsdexception locked="false" priority="64" name="medium shading 2 accent 6"/> <w:lsdexception locked="false" priority="65" name="medium list 1 accent 6"/> <w:lsdexception locked="false" priority="66" name="medium list 2 accent 6"/> <w:lsdexception locked="false" priority="67" name="medium grid 1 accent 6"/> <w:lsdexception locked="false" priority="68" name="medium grid 2 accent 6"/> <w:lsdexception locked="false" priority="69" name="medium grid 3 accent 6"/> <w:lsdexception locked="false" priority="70" name="dark list accent 6"/> <w:lsdexception locked="false" priority="71" name="colorful shading accent 6"/> <w:lsdexception locked="false" priority="72" name="colorful list accent 6"/> <w:lsdexception locked="false" priority="73" name="colorful grid accent 6"/> <w:lsdexception locked="false" priority="19" qformat="true" name="subtle emphasis"/> <w:lsdexception locked="false" priority="21" qformat="true" name="intense emphasis"/> <w:lsdexception locked="false" priority="31" qformat="true" name="subtle reference"/> <w:lsdexception locked="false" priority="32" qformat="true" name="intense reference"/> <w:lsdexception locked="false" priority="33" qformat="true" name="book title"/> <w:lsdexception locked="false" priority="37" semihidden="true" unhidewhenused="true" qformat="true" name="bibliography"/> <w:lsdexception locked="false" priority="39" semihidden="true" unhidewhenused="true" qformat="true" name="toc heading"/> <w:lsdexception locked="false" priority="41" name="plain table 1"/> <w:lsdexception locked="false" priority="42" name="plain table 2"/> <w:lsdexception locked="false" priority="43" name="plain table 3"/> <w:lsdexception locked="false" priority="44" name="plain table 4"/> <w:lsdexception locked="false" priority="45" name="plain table 5"/> <w:lsdexception locked="false" priority="40" name="grid table light"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light"/> <w:lsdexception locked="false" priority="47" name="grid table 2"/> <w:lsdexception locked="false" priority="48" name="grid table 3"/> <w:lsdexception locked="false" priority="49" name="grid table 4"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 1"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 1"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 1"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 1"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 1"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 1"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 1"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 2"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 2"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 2"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 2"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 2"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 2"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 2"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 3"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 3"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 3"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 3"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 3"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 3"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 3"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 4"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 4"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 4"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 4"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 4"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 4"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 4"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 5"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 5"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 5"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 5"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 5"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 5"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 5"/> <w:lsdexception locked="false" priority="46" name="grid table 1 light accent 6"/> <w:lsdexception locked="false" priority="47" name="grid table 2 accent 6"/> <w:lsdexception locked="false" priority="48" name="grid table 3 accent 6"/> <w:lsdexception locked="false" priority="49" name="grid table 4 accent 6"/> <w:lsdexception locked="false" priority="50" name="grid table 5 dark accent 6"/> <w:lsdexception locked="false" priority="51" name="grid table 6 colorful accent 6"/> <w:lsdexception locked="false" priority="52" name="grid table 7 colorful accent 6"/> <w:lsdexception locked="false" priority="46" name="list table 1 light"/> <w:lsdexception locked="false" priority="47" name="list table 2"/> <w:lsdexception locked="false" priority="48" name="list table 3"/> <w:lsdexception locked="false" priority="49" name="list table 4"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 1"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 1"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 1"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 1"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 1"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 1"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 1"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 2"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 2"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 2"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 2"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 2"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 2"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 2"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 3"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 3"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 3"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 3"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 3"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 3"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 3"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 4"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 4"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 4"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 4"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 4"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 4"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 4"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 5"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 5"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 5"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 5"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 5"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 5"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 5"/> <w:lsdexception locked="false" priority="46" name="list table 1 light accent 6"/> <w:lsdexception locked="false" priority="47" name="list table 2 accent 6"/> <w:lsdexception locked="false" priority="48" name="list table 3 accent 6"/> <w:lsdexception locked="false" priority="49" name="list table 4 accent 6"/> <w:lsdexception locked="false" priority="50" name="list table 5 dark accent 6"/> <w:lsdexception locked="false" priority="51" name="list table 6 colorful accent 6"/> <w:lsdexception locked="false" priority="52" name="list table 7 colorful accent 6"/> </w:latentstyles> </xml><![endif]--> <style> <!- /* font definitions */ @font-face {font-family:wingdings; panose-1:5 0 0 0 0 0 0 0 0 0; mso-font-charset:2; mso-generic-font-family:auto; mso-font-pitch:variable; mso-font-signature:0 268435456 0 0 -2147483648 0;} @font-face {font-family:simsun; panose-1:2 1 6 0 3 1 1 1 1 1; mso-font-alt:宋体; mso-font-charset:134; mso-generic-font-family:auto; mso-font-pitch:variable; mso-font-signature:3 680460288 22 0 262145 0;} @font-face {font-family:latha; panose-1:2 0 4 0 0 0 0 0 0 0; mso-font-charset:1; mso-generic-font-family:roman; mso-font-format:other; mso-font-pitch:variable; mso-font-signature:262144 0 0 0 0 0;} @font-face {font-family:"cambria math"; panose-1:2 4 5 3 5 4 6 3 2 4; mso-font-charset:0; mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:-536870145 1107305727 0 0 415 0;} @font-face {font-family:calibri; panose-1:2 15 5 2 2 2 4 3 2 4; mso-font-charset:0; mso-generic-font-family:swiss; mso-font-pitch:variable; mso-font-signature:-536870145 1073786111 1 0 415 0;} @font-face {font-family:cambria; panose-1:2 4 5 3 5 4 6 3 2 4; mso-font-charset:0; mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:-536870145 1073743103 0 0 415 0;} @font-face {font-family:tahoma; panose-1:2 11 6 4 3 5 4 4 2 4; mso-font-charset:0; mso-generic-font-family:swiss; mso-font-pitch:variable; mso-font-signature:-520081665 -1073717157 41 0 66047 0;} @font-face {font-family:georgia; panose-1:2 4 5 2 5 4 5 2 3 3; mso-font-charset:0; mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:647 0 0 0 159 0;} @font-face {font-family:"segoe ui"; panose-1:2 11 5 2 4 2 4 2 2 3; mso-font-charset:0; mso-generic-font-family:swiss; mso-font-pitch:variable; mso-font-signature:-469750017 -1073683329 9 0 511 0;} @font-face {font-family:consolas; panose-1:2 11 6 9 2 2 4 3 2 4; mso-font-charset:0; mso-generic-font-family:modern; mso-font-pitch:fixed; mso-font-signature:-536869121 64767 1 0 415 0;} @font-face {font-family:"palatino linotype"; panose-1:2 4 5 2 5 5 5 3 3 4; mso-font-charset:0; mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:-536870265 1073741843 0 0 415 0;} @font-face {font-family:"\@simsun"; panose-1:2 1 6 0 3 1 1 1 1 1; mso-font-charset:134; mso-generic-font-family:auto; mso-font-pitch:variable; mso-font-signature:3 680460288 22 0 262145 0;} /* style definitions */ p.msonormal, li.msonormal, div.msonormal {mso-style-unhide:no; mso-style-qformat:yes; mso-style-parent:""; margin:0in; margin-bottom:.0001pt; line-height:normal; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; color:windowtext; mso-ansi-language:en-ca; mso-fareast-language:en-us;} h1 {mso-style-priority:3; mso-style-unhide:no; mso-style-qformat:yes; mso-style-link:"heading 1 char"; mso-style-next:normal; margin-top:12.0pt; margin-right:0in; margin-bottom:0in; margin-left:0in; margin-bottom:.0001pt; line-height:normal; mso-pagination:widow-orphan lines-together; page-break-after:avoid; mso-outline-level:1; font-size:14.0pt; font-family:"cambria",serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:#365f91; mso-font-kerning:0pt; mso-ansi-language:en-ca; mso-fareast-language:en-us; font-weight:bold;} h2 {mso-style-noshow:yes; mso-style-priority:3; mso-style-qformat:yes; mso-style-link:"heading 2 char"; mso-style-next:normal; margin-top:10.0pt; margin-right:0in; margin-bottom:0in; margin-left:0in; margin-bottom:.0001pt; line-height:normal; mso-pagination:widow-orphan lines-together; page-break-after:avoid; mso-outline-level:2; font-size:13.0pt; font-family:"cambria",serif; mso-ascii-font-family:cambria; mso-ascii-theme-font:major-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:major-fareast; mso-hansi-font-family:cambria; mso-hansi-theme-font:major-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:major-bidi; color:#4f81bd; mso-themecolor:accent1; mso-ansi-language:en-ca; mso-fareast-language:en-us; font-weight:bold;} h3 {mso-style-noshow:yes; mso-style-priority:3; mso-style-qformat:yes; mso-style-link:"heading 3 char"; mso-margin-top-alt:auto; margin-right:0in; mso-margin-bottom-alt:auto; margin-left:0in; line-height:normal; mso-pagination:widow-orphan; mso-outline-level:3; font-size:13.5pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; color:windowtext; mso-fareast-language:en-us; font-weight:bold;} h4 {mso-style-noshow:yes; mso-style-priority:3; mso-style-qformat:yes; mso-style-link:"heading 4 char"; mso-style-next:normal; margin:0in; margin-bottom:.0001pt; text-indent:.5in; line-height:200%; mso-pagination:widow-orphan lines-together; page-break-after:avoid; mso-outline-level:4; font-size:12.0pt; font-family:"cambria",serif; mso-ascii-font-family:cambria; mso-ascii-theme-font:major-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:major-fareast; mso-hansi-font-family:cambria; mso-hansi-theme-font:major-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:major-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja; font-weight:bold; font-style:italic;} h5 {mso-style-noshow:yes; mso-style-priority:3; mso-style-qformat:yes; mso-style-link:"heading 5 char"; mso-style-next:normal; margin:0in; margin-bottom:.0001pt; text-indent:.5in; line-height:200%; mso-pagination:widow-orphan lines-together; page-break-after:avoid; mso-outline-level:5; font-size:12.0pt; font-family:"cambria",serif; mso-ascii-font-family:cambria; mso-ascii-theme-font:major-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:major-fareast; mso-hansi-font-family:cambria; mso-hansi-theme-font:major-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:major-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja; font-weight:normal; font-style:italic;} h6 {mso-style-noshow:yes; mso-style-priority:9; mso-style-qformat:yes; mso-style-link:"heading 6 char"; mso-style-next:normal; margin-top:2.0pt; margin-right:0in; margin-bottom:0in; margin-left:0in; margin-bottom:.0001pt; line-height:200%; mso-pagination:widow-orphan lines-together; page-break-after:avoid; mso-outline-level:6; font-size:12.0pt; font-family:"cambria",serif; mso-ascii-font-family:cambria; mso-ascii-theme-font:major-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:major-fareast; mso-hansi-font-family:cambria; mso-hansi-theme-font:major-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:major-bidi; color:#243f60; mso-themecolor:accent1; mso-themeshade:127; mso-fareast-language:ja; font-weight:normal;} p.msoheading7, li.msoheading7, div.msoheading7 {mso-style-noshow:yes; mso-style-priority:9; mso-style-qformat:yes; mso-style-link:"heading 7 char"; mso-style-next:normal; margin-top:2.0pt; margin-right:0in; margin-bottom:0in; margin-left:0in; margin-bottom:.0001pt; line-height:200%; mso-pagination:widow-orphan lines-together; page-break-after:avoid; mso-outline-level:7; font-size:12.0pt; font-family:"cambria",serif; mso-ascii-font-family:cambria; mso-ascii-theme-font:major-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:major-fareast; mso-hansi-font-family:cambria; mso-hansi-theme-font:major-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:major-bidi; color:#243f60; mso-themecolor:accent1; mso-themeshade:127; mso-fareast-language:ja; font-style:italic;} p.msoheading8, li.msoheading8, div.msoheading8 {mso-style-noshow:yes; mso-style-priority:9; mso-style-qformat:yes; mso-style-link:"heading 8 char"; mso-style-next:normal; margin-top:2.0pt; margin-right:0in; margin-bottom:0in; margin-left:0in; margin-bottom:.0001pt; line-height:200%; mso-pagination:widow-orphan lines-together; page-break-after:avoid; mso-outline-level:8; font-size:10.5pt; font-family:"cambria",serif; mso-ascii-font-family:cambria; mso-ascii-theme-font:major-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:major-fareast; mso-hansi-font-family:cambria; mso-hansi-theme-font:major-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:major-bidi; color:#272727; mso-themecolor:text1; mso-themetint:216; mso-fareast-language:ja;} p.msoheading9, li.msoheading9, div.msoheading9 {mso-style-noshow:yes; mso-style-priority:9; mso-style-qformat:yes; mso-style-link:"heading 9 char"; mso-style-next:normal; margin-top:2.0pt; margin-right:0in; margin-bottom:0in; margin-left:0in; margin-bottom:.0001pt; line-height:200%; mso-pagination:widow-orphan lines-together; page-break-after:avoid; mso-outline-level:9; font-size:10.5pt; font-family:"cambria",serif; mso-ascii-font-family:cambria; mso-ascii-theme-font:major-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:major-fareast; mso-hansi-font-family:cambria; mso-hansi-theme-font:major-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:major-bidi; color:#272727; mso-themecolor:text1; mso-themetint:216; mso-fareast-language:ja; font-style:italic;} p.msoindex1, li.msoindex1, div.msoindex1 {mso-style-update:auto; mso-style-noshow:yes; mso-style-priority:99; mso-style-next:normal; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:12.0pt; margin-bottom:.0001pt; line-height:normal; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msoindex2, li.msoindex2, div.msoindex2 {mso-style-update:auto; mso-style-noshow:yes; mso-style-priority:99; mso-style-next:normal; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:24.0pt; margin-bottom:.0001pt; line-height:normal; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msoindex3, li.msoindex3, div.msoindex3 {mso-style-update:auto; mso-style-noshow:yes; mso-style-priority:99; mso-style-next:normal; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.5in; margin-bottom:.0001pt; line-height:normal; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msoindex4, li.msoindex4, div.msoindex4 {mso-style-update:auto; mso-style-noshow:yes; mso-style-priority:99; mso-style-next:normal; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:48.0pt; margin-bottom:.0001pt; line-height:normal; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msoindex5, li.msoindex5, div.msoindex5 {mso-style-update:auto; mso-style-noshow:yes; mso-style-priority:99; mso-style-next:normal; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:60.0pt; margin-bottom:.0001pt; line-height:normal; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msoindex6, li.msoindex6, div.msoindex6 {mso-style-update:auto; mso-style-noshow:yes; mso-style-priority:99; mso-style-next:normal; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:1.0in; margin-bottom:.0001pt; line-height:normal; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msoindex7, li.msoindex7, div.msoindex7 {mso-style-update:auto; mso-style-noshow:yes; mso-style-priority:99; mso-style-next:normal; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:84.0pt; margin-bottom:.0001pt; line-height:normal; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msoindex8, li.msoindex8, div.msoindex8 {mso-style-update:auto; mso-style-noshow:yes; mso-style-priority:99; mso-style-next:normal; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:96.0pt; margin-bottom:.0001pt; line-height:normal; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msoindex9, li.msoindex9, div.msoindex9 {mso-style-update:auto; mso-style-noshow:yes; mso-style-priority:99; mso-style-next:normal; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:1.5in; margin-bottom:.0001pt; line-height:normal; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msotoc4, li.msotoc4, div.msotoc4 {mso-style-update:auto; mso-style-noshow:yes; mso-style-priority:39; mso-style-next:normal; margin-top:0in; margin-right:0in; margin-bottom:5.0pt; margin-left:.5in; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msotoc5, li.msotoc5, div.msotoc5 {mso-style-update:auto; mso-style-noshow:yes; mso-style-priority:39; mso-style-next:normal; margin-top:0in; margin-right:0in; margin-bottom:5.0pt; margin-left:48.0pt; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msotoc6, li.msotoc6, div.msotoc6 {mso-style-update:auto; mso-style-noshow:yes; mso-style-priority:39; mso-style-next:normal; margin-top:0in; margin-right:0in; margin-bottom:5.0pt; margin-left:60.0pt; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msotoc7, li.msotoc7, div.msotoc7 {mso-style-update:auto; mso-style-noshow:yes; mso-style-priority:39; mso-style-next:normal; margin-top:0in; margin-right:0in; margin-bottom:5.0pt; margin-left:1.0in; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msotoc8, li.msotoc8, div.msotoc8 {mso-style-update:auto; mso-style-noshow:yes; mso-style-priority:39; mso-style-next:normal; margin-top:0in; margin-right:0in; margin-bottom:5.0pt; margin-left:84.0pt; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msotoc9, li.msotoc9, div.msotoc9 {mso-style-update:auto; mso-style-noshow:yes; mso-style-priority:39; mso-style-next:normal; margin-top:0in; margin-right:0in; margin-bottom:5.0pt; margin-left:96.0pt; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msonormalindent, li.msonormalindent, div.msonormalindent {mso-style-noshow:yes; mso-style-priority:99; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.5in; margin-bottom:.0001pt; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msofootnotetext, li.msofootnotetext, div.msofootnotetext {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"footnote text char"; margin:0in; margin-bottom:.0001pt; text-indent:.5in; line-height:normal; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msocommenttext, li.msocommenttext, div.msocommenttext {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"comment text char"; margin-top:0in; margin-right:0in; margin-bottom:10.0pt; margin-left:0in; line-height:normal; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:windowtext; mso-fareast-language:en-us;} p.msoheader, li.msoheader, div.msoheader {mso-style-priority:99; mso-style-qformat:yes; mso-style-link:"header char"; margin:0in; margin-bottom:.0001pt; line-height:normal; mso-pagination:widow-orphan; tab-stops:center 3.25in right 6.5in; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; color:windowtext; mso-ansi-language:en-ca; mso-fareast-language:en-us;} p.msofooter, li.msofooter, div.msofooter {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"footer char"; margin:0in; margin-bottom:.0001pt; line-height:normal; mso-pagination:widow-orphan; tab-stops:center 3.25in right 6.5in; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; color:windowtext; mso-ansi-language:en-ca; mso-fareast-language:en-us;} p.msoindexheading, li.msoindexheading, div.msoindexheading {mso-style-noshow:yes; mso-style-priority:99; mso-style-next:"index 1"; margin:0in; margin-bottom:.0001pt; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"cambria",serif; mso-ascii-font-family:cambria; mso-ascii-theme-font:major-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:major-fareast; mso-hansi-font-family:cambria; mso-hansi-theme-font:major-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:major-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja; font-weight:bold;} p.msocaption, li.msocaption, div.msocaption {mso-style-noshow:yes; mso-style-priority:35; mso-style-qformat:yes; mso-style-next:normal; margin-top:.25in; margin-right:0in; margin-bottom:0in; margin-left:0in; margin-bottom:.0001pt; line-height:normal; mso-pagination:widow-orphan; tab-stops:255.15pt; font-size:11.0pt; mso-bidi-font-size:12.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:#3e9ed0; mso-fareast-language:en-us;} p.msotof, li.msotof, div.msotof {mso-style-noshow:yes; mso-style-priority:99; mso-style-next:normal; margin:0in; margin-bottom:.0001pt; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msoenvelopeaddress, li.msoenvelopeaddress, div.msoenvelopeaddress {mso-style-noshow:yes; mso-style-priority:99; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:2.0in; margin-bottom:.0001pt; line-height:normal; mso-pagination:widow-orphan; mso-element:frame; mso-element-frame-width:5.5in; mso-element-frame-height:99.0pt; mso-element-frame-hspace:9.0pt; mso-element-wrap:auto; mso-element-anchor-horizontal:page; mso-element-left:center; mso-element-top:bottom; font-size:12.0pt; font-family:"cambria",serif; mso-ascii-font-family:cambria; mso-ascii-theme-font:major-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:major-fareast; mso-hansi-font-family:cambria; mso-hansi-theme-font:major-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:major-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msoenvelopereturn, li.msoenvelopereturn, div.msoenvelopereturn {mso-style-noshow:yes; mso-style-priority:99; margin:0in; margin-bottom:.0001pt; line-height:normal; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"cambria",serif; mso-ascii-font-family:cambria; mso-ascii-theme-font:major-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:major-fareast; mso-hansi-font-family:cambria; mso-hansi-theme-font:major-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:major-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} span.msofootnotereference {mso-style-noshow:yes; mso-style-priority:99; mso-style-qformat:yes; vertical-align:super;} span.msocommentreference {mso-style-noshow:yes; mso-style-priority:99; mso-ansi-font-size:8.0pt; mso-bidi-font-size:8.0pt;} span.msoendnotereference {mso-style-noshow:yes; mso-style-priority:99; vertical-align:super;} p.msotoa, li.msotoa, div.msotoa {mso-style-noshow:yes; mso-style-priority:99; mso-style-next:normal; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:12.0pt; margin-bottom:.0001pt; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msomacrotext, li.msomacrotext, div.msomacrotext {mso-style-noshow:yes; mso-style-priority:99; mso-style-parent:""; mso-style-link:"macro text char"; margin:0in; margin-bottom:.0001pt; line-height:200%; mso-pagination:widow-orphan; tab-stops:24.0pt 48.0pt 1.0in 96.0pt 120.0pt 2.0in 168.0pt 192.0pt 3.0in; font-size:10.0pt; font-family:consolas; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:consolas; color:black; mso-themecolor:text1; mso-font-kerning:12.0pt; mso-fareast-language:ja;} p.msotoaheading, li.msotoaheading, div.msotoaheading {mso-style-noshow:yes; mso-style-priority:99; mso-style-next:normal; margin-top:6.0pt; margin-right:0in; margin-bottom:0in; margin-left:0in; margin-bottom:.0001pt; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"cambria",serif; mso-ascii-font-family:cambria; mso-ascii-theme-font:major-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:major-fareast; mso-hansi-font-family:cambria; mso-hansi-theme-font:major-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:major-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja; font-weight:bold;} p.msolist, li.msolist, div.msolist {mso-style-noshow:yes; mso-style-priority:99; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.25in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistcxspfirst, li.msolistcxspfirst, div.msolistcxspfirst {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.25in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistcxspmiddle, li.msolistcxspmiddle, div.msolistcxspmiddle {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.25in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistcxsplast, li.msolistcxsplast, div.msolistcxsplast {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.25in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistbullet, li.msolistbullet, div.msolistbullet {mso-style-noshow:yes; mso-style-priority:9; mso-style-qformat:yes; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.25in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:-.25in; line-height:normal; mso-pagination:widow-orphan; mso-list:l9 level1 lfo1; font-size:11.0pt; mso-bidi-font-size:12.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:windowtext; mso-fareast-language:en-us;} p.msolistbulletcxspfirst, li.msolistbulletcxspfirst, div.msolistbulletcxspfirst {mso-style-noshow:yes; mso-style-priority:9; mso-style-qformat:yes; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.25in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:-.25in; line-height:normal; mso-pagination:widow-orphan; mso-list:l9 level1 lfo1; font-size:11.0pt; mso-bidi-font-size:12.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:windowtext; mso-fareast-language:en-us;} p.msolistbulletcxspmiddle, li.msolistbulletcxspmiddle, div.msolistbulletcxspmiddle {mso-style-noshow:yes; mso-style-priority:9; mso-style-qformat:yes; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.25in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:-.25in; line-height:normal; mso-pagination:widow-orphan; mso-list:l9 level1 lfo1; font-size:11.0pt; mso-bidi-font-size:12.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:windowtext; mso-fareast-language:en-us;} p.msolistbulletcxsplast, li.msolistbulletcxsplast, div.msolistbulletcxsplast {mso-style-noshow:yes; mso-style-priority:9; mso-style-qformat:yes; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.25in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:-.25in; line-height:normal; mso-pagination:widow-orphan; mso-list:l9 level1 lfo1; font-size:11.0pt; mso-bidi-font-size:12.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:windowtext; mso-fareast-language:en-us;} p.msolistnumber, li.msolistnumber, div.msolistnumber {mso-style-noshow:yes; mso-style-priority:9; mso-style-qformat:yes; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.75in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:-.25in; line-height:200%; mso-pagination:widow-orphan; mso-list:l8 level1 lfo2; tab-stops:list .75in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistnumbercxspfirst, li.msolistnumbercxspfirst, div.msolistnumbercxspfirst {mso-style-noshow:yes; mso-style-priority:9; mso-style-qformat:yes; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.75in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:-.25in; line-height:200%; mso-pagination:widow-orphan; mso-list:l8 level1 lfo2; tab-stops:list .75in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistnumbercxspmiddle, li.msolistnumbercxspmiddle, div.msolistnumbercxspmiddle {mso-style-noshow:yes; mso-style-priority:9; mso-style-qformat:yes; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.75in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:-.25in; line-height:200%; mso-pagination:widow-orphan; mso-list:l8 level1 lfo2; tab-stops:list .75in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistnumbercxsplast, li.msolistnumbercxsplast, div.msolistnumbercxsplast {mso-style-noshow:yes; mso-style-priority:9; mso-style-qformat:yes; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.75in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:-.25in; line-height:200%; mso-pagination:widow-orphan; mso-list:l8 level1 lfo2; tab-stops:list .75in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolist2, li.msolist2, div.msolist2 {mso-style-noshow:yes; mso-style-priority:99; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.5in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolist2cxspfirst, li.msolist2cxspfirst, div.msolist2cxspfirst {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.5in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolist2cxspmiddle, li.msolist2cxspmiddle, div.msolist2cxspmiddle {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.5in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolist2cxsplast, li.msolist2cxsplast, div.msolist2cxsplast {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.5in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolist3, li.msolist3, div.msolist3 {mso-style-noshow:yes; mso-style-priority:99; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.75in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolist3cxspfirst, li.msolist3cxspfirst, div.msolist3cxspfirst {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.75in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolist3cxspmiddle, li.msolist3cxspmiddle, div.msolist3cxspmiddle {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.75in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolist3cxsplast, li.msolist3cxsplast, div.msolist3cxsplast {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.75in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolist4, li.msolist4, div.msolist4 {mso-style-noshow:yes; mso-style-priority:99; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:1.0in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolist4cxspfirst, li.msolist4cxspfirst, div.msolist4cxspfirst {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:1.0in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolist4cxspmiddle, li.msolist4cxspmiddle, div.msolist4cxspmiddle {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:1.0in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolist4cxsplast, li.msolist4cxsplast, div.msolist4cxsplast {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:1.0in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolist5, li.msolist5, div.msolist5 {mso-style-noshow:yes; mso-style-priority:99; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:1.25in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolist5cxspfirst, li.msolist5cxspfirst, div.msolist5cxspfirst {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:1.25in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolist5cxspmiddle, li.msolist5cxspmiddle, div.msolist5cxspmiddle {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:1.25in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolist5cxsplast, li.msolist5cxsplast, div.msolist5cxsplast {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:1.25in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistbullet2, li.msolistbullet2, div.msolistbullet2 {mso-style-noshow:yes; mso-style-priority:99; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.5in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:0in; line-height:200%; mso-pagination:widow-orphan; mso-list:l7 level1 lfo3; tab-stops:list .5in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistbullet2cxspfirst, li.msolistbullet2cxspfirst, div.msolistbullet2cxspfirst {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.5in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:0in; line-height:200%; mso-pagination:widow-orphan; mso-list:l7 level1 lfo3; tab-stops:list .5in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistbullet2cxspmiddle, li.msolistbullet2cxspmiddle, div.msolistbullet2cxspmiddle {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.5in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:0in; line-height:200%; mso-pagination:widow-orphan; mso-list:l7 level1 lfo3; tab-stops:list .5in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistbullet2cxsplast, li.msolistbullet2cxsplast, div.msolistbullet2cxsplast {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.5in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:0in; line-height:200%; mso-pagination:widow-orphan; mso-list:l7 level1 lfo3; tab-stops:list .5in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistbullet3, li.msolistbullet3, div.msolistbullet3 {mso-style-noshow:yes; mso-style-priority:99; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.75in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:0in; line-height:200%; mso-pagination:widow-orphan; mso-list:l6 level1 lfo4; tab-stops:list .75in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistbullet3cxspfirst, li.msolistbullet3cxspfirst, div.msolistbullet3cxspfirst {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.75in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:0in; line-height:200%; mso-pagination:widow-orphan; mso-list:l6 level1 lfo4; tab-stops:list .75in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistbullet3cxspmiddle, li.msolistbullet3cxspmiddle, div.msolistbullet3cxspmiddle {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.75in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:0in; line-height:200%; mso-pagination:widow-orphan; mso-list:l6 level1 lfo4; tab-stops:list .75in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistbullet3cxsplast, li.msolistbullet3cxsplast, div.msolistbullet3cxsplast {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.75in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:0in; line-height:200%; mso-pagination:widow-orphan; mso-list:l6 level1 lfo4; tab-stops:list .75in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistbullet4, li.msolistbullet4, div.msolistbullet4 {mso-style-noshow:yes; mso-style-priority:99; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:1.0in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:0in; line-height:200%; mso-pagination:widow-orphan; mso-list:l5 level1 lfo5; tab-stops:list 1.0in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistbullet4cxspfirst, li.msolistbullet4cxspfirst, div.msolistbullet4cxspfirst {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:1.0in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:0in; line-height:200%; mso-pagination:widow-orphan; mso-list:l5 level1 lfo5; tab-stops:list 1.0in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistbullet4cxspmiddle, li.msolistbullet4cxspmiddle, div.msolistbullet4cxspmiddle {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:1.0in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:0in; line-height:200%; mso-pagination:widow-orphan; mso-list:l5 level1 lfo5; tab-stops:list 1.0in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistbullet4cxsplast, li.msolistbullet4cxsplast, div.msolistbullet4cxsplast {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:1.0in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:0in; line-height:200%; mso-pagination:widow-orphan; mso-list:l5 level1 lfo5; tab-stops:list 1.0in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistbullet5, li.msolistbullet5, div.msolistbullet5 {mso-style-noshow:yes; mso-style-priority:99; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:1.25in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:0in; line-height:200%; mso-pagination:widow-orphan; mso-list:l4 level1 lfo6; tab-stops:list 1.25in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistbullet5cxspfirst, li.msolistbullet5cxspfirst, div.msolistbullet5cxspfirst {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:1.25in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:0in; line-height:200%; mso-pagination:widow-orphan; mso-list:l4 level1 lfo6; tab-stops:list 1.25in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistbullet5cxspmiddle, li.msolistbullet5cxspmiddle, div.msolistbullet5cxspmiddle {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:1.25in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:0in; line-height:200%; mso-pagination:widow-orphan; mso-list:l4 level1 lfo6; tab-stops:list 1.25in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistbullet5cxsplast, li.msolistbullet5cxsplast, div.msolistbullet5cxsplast {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:1.25in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:0in; line-height:200%; mso-pagination:widow-orphan; mso-list:l4 level1 lfo6; tab-stops:list 1.25in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistnumber2, li.msolistnumber2, div.msolistnumber2 {mso-style-noshow:yes; mso-style-priority:99; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.5in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:0in; line-height:200%; mso-pagination:widow-orphan; mso-list:l3 level1 lfo7; tab-stops:list .5in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistnumber2cxspfirst, li.msolistnumber2cxspfirst, div.msolistnumber2cxspfirst {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.5in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:0in; line-height:200%; mso-pagination:widow-orphan; mso-list:l3 level1 lfo7; tab-stops:list .5in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistnumber2cxspmiddle, li.msolistnumber2cxspmiddle, div.msolistnumber2cxspmiddle {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.5in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:0in; line-height:200%; mso-pagination:widow-orphan; mso-list:l3 level1 lfo7; tab-stops:list .5in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistnumber2cxsplast, li.msolistnumber2cxsplast, div.msolistnumber2cxsplast {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.5in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:0in; line-height:200%; mso-pagination:widow-orphan; mso-list:l3 level1 lfo7; tab-stops:list .5in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistnumber3, li.msolistnumber3, div.msolistnumber3 {mso-style-noshow:yes; mso-style-priority:99; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.75in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:0in; line-height:200%; mso-pagination:widow-orphan; mso-list:l2 level1 lfo8; tab-stops:list .75in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistnumber3cxspfirst, li.msolistnumber3cxspfirst, div.msolistnumber3cxspfirst {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.75in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:0in; line-height:200%; mso-pagination:widow-orphan; mso-list:l2 level1 lfo8; tab-stops:list .75in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistnumber3cxspmiddle, li.msolistnumber3cxspmiddle, div.msolistnumber3cxspmiddle {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.75in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:0in; line-height:200%; mso-pagination:widow-orphan; mso-list:l2 level1 lfo8; tab-stops:list .75in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistnumber3cxsplast, li.msolistnumber3cxsplast, div.msolistnumber3cxsplast {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.75in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:0in; line-height:200%; mso-pagination:widow-orphan; mso-list:l2 level1 lfo8; tab-stops:list .75in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistnumber4, li.msolistnumber4, div.msolistnumber4 {mso-style-noshow:yes; mso-style-priority:99; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:1.0in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:0in; line-height:200%; mso-pagination:widow-orphan; mso-list:l1 level1 lfo9; tab-stops:list 1.0in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistnumber4cxspfirst, li.msolistnumber4cxspfirst, div.msolistnumber4cxspfirst {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:1.0in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:0in; line-height:200%; mso-pagination:widow-orphan; mso-list:l1 level1 lfo9; tab-stops:list 1.0in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistnumber4cxspmiddle, li.msolistnumber4cxspmiddle, div.msolistnumber4cxspmiddle {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:1.0in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:0in; line-height:200%; mso-pagination:widow-orphan; mso-list:l1 level1 lfo9; tab-stops:list 1.0in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistnumber4cxsplast, li.msolistnumber4cxsplast, div.msolistnumber4cxsplast {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:1.0in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:0in; line-height:200%; mso-pagination:widow-orphan; mso-list:l1 level1 lfo9; tab-stops:list 1.0in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistnumber5, li.msolistnumber5, div.msolistnumber5 {mso-style-noshow:yes; mso-style-priority:99; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:1.25in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:0in; line-height:200%; mso-pagination:widow-orphan; mso-list:l0 level1 lfo10; tab-stops:list 1.25in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistnumber5cxspfirst, li.msolistnumber5cxspfirst, div.msolistnumber5cxspfirst {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:1.25in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:0in; line-height:200%; mso-pagination:widow-orphan; mso-list:l0 level1 lfo10; tab-stops:list 1.25in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistnumber5cxspmiddle, li.msolistnumber5cxspmiddle, div.msolistnumber5cxspmiddle {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:1.25in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:0in; line-height:200%; mso-pagination:widow-orphan; mso-list:l0 level1 lfo10; tab-stops:list 1.25in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistnumber5cxsplast, li.msolistnumber5cxsplast, div.msolistnumber5cxsplast {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:1.25in; margin-bottom:.0001pt; mso-add-space:auto; text-indent:0in; line-height:200%; mso-pagination:widow-orphan; mso-list:l0 level1 lfo10; tab-stops:list 1.25in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msotitle, li.msotitle, div.msotitle {mso-style-noshow:yes; mso-style-priority:1; mso-style-unhide:no; mso-style-qformat:yes; mso-style-link:"title char"; mso-style-next:normal; margin-top:120.0pt; margin-right:0in; margin-bottom:0in; margin-left:0in; margin-bottom:.0001pt; mso-add-space:auto; text-align:center; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"cambria",serif; mso-ascii-font-family:cambria; mso-ascii-theme-font:major-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:major-fareast; mso-hansi-font-family:cambria; mso-hansi-theme-font:major-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:major-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msotitlecxspfirst, li.msotitlecxspfirst, div.msotitlecxspfirst {mso-style-noshow:yes; mso-style-priority:1; mso-style-unhide:no; mso-style-qformat:yes; mso-style-link:"title char"; mso-style-next:normal; mso-style-type:export-only; margin-top:120.0pt; margin-right:0in; margin-bottom:0in; margin-left:0in; margin-bottom:.0001pt; mso-add-space:auto; text-align:center; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"cambria",serif; mso-ascii-font-family:cambria; mso-ascii-theme-font:major-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:major-fareast; mso-hansi-font-family:cambria; mso-hansi-theme-font:major-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:major-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msotitlecxspmiddle, li.msotitlecxspmiddle, div.msotitlecxspmiddle {mso-style-noshow:yes; mso-style-priority:1; mso-style-unhide:no; mso-style-qformat:yes; mso-style-link:"title char"; mso-style-next:normal; mso-style-type:export-only; margin:0in; margin-bottom:.0001pt; mso-add-space:auto; text-align:center; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"cambria",serif; mso-ascii-font-family:cambria; mso-ascii-theme-font:major-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:major-fareast; mso-hansi-font-family:cambria; mso-hansi-theme-font:major-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:major-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msotitlecxsplast, li.msotitlecxsplast, div.msotitlecxsplast {mso-style-noshow:yes; mso-style-priority:1; mso-style-unhide:no; mso-style-qformat:yes; mso-style-link:"title char"; mso-style-next:normal; mso-style-type:export-only; margin:0in; margin-bottom:.0001pt; mso-add-space:auto; text-align:center; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"cambria",serif; mso-ascii-font-family:cambria; mso-ascii-theme-font:major-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:major-fareast; mso-hansi-font-family:cambria; mso-hansi-theme-font:major-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:major-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msoclosing, li.msoclosing, div.msoclosing {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"closing char"; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:3.0in; margin-bottom:.0001pt; line-height:normal; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msosignature, li.msosignature, div.msosignature {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"signature char"; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:3.0in; margin-bottom:.0001pt; line-height:normal; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msobodytext, li.msobodytext, div.msobodytext {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"body text char"; margin:0in; margin-bottom:.0001pt; line-height:normal; mso-pagination:none; mso-layout-grid-align:none; text-autospace:none; font-size:12.0pt; font-family:"arial",sans-serif; mso-fareast-font-family:calibri; mso-bidi-font-family:latha; color:windowtext; mso-fareast-language:en-us;} p.msobodytextindent, li.msobodytextindent, div.msobodytextindent {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"body text indent char"; margin-top:0in; margin-right:0in; margin-bottom:6.0pt; margin-left:.25in; line-height:115%; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:windowtext; mso-fareast-language:en-us;} p.msolistcontinue, li.msolistcontinue, div.msolistcontinue {mso-style-noshow:yes; mso-style-priority:99; margin-top:0in; margin-right:0in; margin-bottom:6.0pt; margin-left:.25in; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistcontinuecxspfirst, li.msolistcontinuecxspfirst, div.msolistcontinuecxspfirst {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.25in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistcontinuecxspmiddle, li.msolistcontinuecxspmiddle, div.msolistcontinuecxspmiddle {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.25in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistcontinuecxsplast, li.msolistcontinuecxsplast, div.msolistcontinuecxsplast {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:6.0pt; margin-left:.25in; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistcontinue2, li.msolistcontinue2, div.msolistcontinue2 {mso-style-noshow:yes; mso-style-priority:99; margin-top:0in; margin-right:0in; margin-bottom:6.0pt; margin-left:.5in; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistcontinue2cxspfirst, li.msolistcontinue2cxspfirst, div.msolistcontinue2cxspfirst {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.5in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistcontinue2cxspmiddle, li.msolistcontinue2cxspmiddle, div.msolistcontinue2cxspmiddle {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.5in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistcontinue2cxsplast, li.msolistcontinue2cxsplast, div.msolistcontinue2cxsplast {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:6.0pt; margin-left:.5in; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistcontinue3, li.msolistcontinue3, div.msolistcontinue3 {mso-style-noshow:yes; mso-style-priority:99; margin-top:0in; margin-right:0in; margin-bottom:6.0pt; margin-left:.75in; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistcontinue3cxspfirst, li.msolistcontinue3cxspfirst, div.msolistcontinue3cxspfirst {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.75in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistcontinue3cxspmiddle, li.msolistcontinue3cxspmiddle, div.msolistcontinue3cxspmiddle {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.75in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistcontinue3cxsplast, li.msolistcontinue3cxsplast, div.msolistcontinue3cxsplast {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:6.0pt; margin-left:.75in; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistcontinue4, li.msolistcontinue4, div.msolistcontinue4 {mso-style-noshow:yes; mso-style-priority:99; margin-top:0in; margin-right:0in; margin-bottom:6.0pt; margin-left:1.0in; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistcontinue4cxspfirst, li.msolistcontinue4cxspfirst, div.msolistcontinue4cxspfirst {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:1.0in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistcontinue4cxspmiddle, li.msolistcontinue4cxspmiddle, div.msolistcontinue4cxspmiddle {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:1.0in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistcontinue4cxsplast, li.msolistcontinue4cxsplast, div.msolistcontinue4cxsplast {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:6.0pt; margin-left:1.0in; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistcontinue5, li.msolistcontinue5, div.msolistcontinue5 {mso-style-noshow:yes; mso-style-priority:99; margin-top:0in; margin-right:0in; margin-bottom:6.0pt; margin-left:1.25in; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistcontinue5cxspfirst, li.msolistcontinue5cxspfirst, div.msolistcontinue5cxspfirst {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:1.25in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistcontinue5cxspmiddle, li.msolistcontinue5cxspmiddle, div.msolistcontinue5cxspmiddle {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:1.25in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msolistcontinue5cxsplast, li.msolistcontinue5cxsplast, div.msolistcontinue5cxsplast {mso-style-noshow:yes; mso-style-priority:99; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:6.0pt; margin-left:1.25in; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msomessageheader, li.msomessageheader, div.msomessageheader {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"message header char"; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.75in; margin-bottom:.0001pt; line-height:normal; mso-pagination:widow-orphan; background:#cccccc; mso-shading:windowtext; mso-pattern:gray-20 auto; border:none; mso-border-alt:solid windowtext .75pt; padding:0in; mso-padding-alt:1.0pt 1.0pt 1.0pt 1.0pt; font-size:12.0pt; font-family:"cambria",serif; mso-ascii-font-family:cambria; mso-ascii-theme-font:major-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:major-fareast; mso-hansi-font-family:cambria; mso-hansi-theme-font:major-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:major-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msosalutation, li.msosalutation, div.msosalutation {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"salutation char"; mso-style-next:normal; margin:0in; margin-bottom:.0001pt; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msodate, li.msodate, div.msodate {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"date char"; mso-style-next:normal; margin:0in; margin-bottom:.0001pt; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msobodytextfirstindent, li.msobodytextfirstindent, div.msobodytextfirstindent {mso-style-noshow:yes; mso-style-priority:99; mso-style-parent:"body text"; mso-style-link:"body text first indent char"; margin:0in; margin-bottom:.0001pt; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msobodytextfirstindent2, li.msobodytextfirstindent2, div.msobodytextfirstindent2 {mso-style-noshow:yes; mso-style-priority:99; mso-style-parent:"body text indent"; mso-style-link:"body text first indent 2 char"; margin-top:0in; margin-right:0in; margin-bottom:6.0pt; margin-left:.25in; text-indent:10.5pt; line-height:normal; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; color:windowtext; mso-fareast-language:en-us;} p.msonoteheading, li.msonoteheading, div.msonoteheading {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"note heading char"; mso-style-next:normal; margin:0in; margin-bottom:.0001pt; line-height:normal; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msobodytext2, li.msobodytext2, div.msobodytext2 {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"body text 2 char"; margin-top:0in; margin-right:0in; margin-bottom:6.0pt; margin-left:0in; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msobodytext3, li.msobodytext3, div.msobodytext3 {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"body text 3 char"; margin-top:0in; margin-right:0in; margin-bottom:6.0pt; margin-left:0in; line-height:200%; mso-pagination:widow-orphan; font-size:8.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msobodytextindent2, li.msobodytextindent2, div.msobodytextindent2 {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"body text indent 2 char"; margin-top:0in; margin-right:0in; margin-bottom:6.0pt; margin-left:.25in; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msobodytextindent3, li.msobodytextindent3, div.msobodytextindent3 {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"body text indent 3 char"; margin-top:0in; margin-right:0in; margin-bottom:6.0pt; margin-left:.25in; line-height:200%; mso-pagination:widow-orphan; font-size:8.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msoblocktext, li.msoblocktext, div.msoblocktext {mso-style-noshow:yes; mso-style-priority:99; margin-top:0in; margin-right:.8in; margin-bottom:0in; margin-left:.8in; margin-bottom:.0001pt; line-height:200%; mso-pagination:widow-orphan; border:none; mso-border-alt:solid #4f81bd .5pt; mso-border-themecolor:accent1; padding:0in; mso-padding-alt:10.0pt 10.0pt 10.0pt 10.0pt; mso-border-shadow:yes; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:#4f81bd; mso-themecolor:accent1; mso-fareast-language:ja; font-style:italic;} a:link, span.msohyperlink {mso-style-noshow:yes; mso-style-priority:99; color:blue; text-decoration:underline; text-underline:single;} a:visited, span.msohyperlinkfollowed {mso-style-noshow:yes; mso-style-priority:99; color:purple; mso-themecolor:followedhyperlink; text-decoration:underline; text-underline:single;} p.msodocumentmap, li.msodocumentmap, div.msodocumentmap {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"document map char"; margin:0in; margin-bottom:.0001pt; line-height:normal; mso-pagination:widow-orphan; font-size:8.0pt; font-family:"segoe ui",sans-serif; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msoplaintext, li.msoplaintext, div.msoplaintext {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"plain text char"; margin:0in; margin-bottom:.0001pt; line-height:normal; mso-pagination:widow-orphan; font-size:10.5pt; font-family:consolas; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:consolas; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msoautosig, li.msoautosig, div.msoautosig {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"e-mail signature char"; margin:0in; margin-bottom:.0001pt; line-height:normal; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p {mso-style-noshow:yes; mso-style-priority:99; margin:0in; margin-bottom:.0001pt; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; color:black; mso-themecolor:text1; mso-fareast-language:ja;} address {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"html address char"; margin:0in; margin-bottom:.0001pt; line-height:normal; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja; font-style:italic;} pre {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"html preformatted char"; margin:0in; margin-bottom:.0001pt; line-height:normal; mso-pagination:widow-orphan; tab-stops:45.8pt 91.6pt 137.4pt 183.2pt 229.0pt 274.8pt 320.6pt 366.4pt 412.2pt 458.0pt 503.8pt 549.6pt 595.4pt 641.2pt 687.0pt 732.8pt; font-size:10.0pt; font-family:consolas; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:consolas; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.msocommentsubject, li.msocommentsubject, div.msocommentsubject {mso-style-noshow:yes; mso-style-priority:99; mso-style-parent:"comment text"; mso-style-link:"comment subject char"; mso-style-next:"comment text"; margin-top:0in; margin-right:0in; margin-bottom:10.0pt; margin-left:0in; line-height:normal; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:windowtext; mso-fareast-language:en-us; font-weight:bold;} p.msoacetate, li.msoacetate, div.msoacetate {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"balloon text char"; margin:0in; margin-bottom:.0001pt; line-height:normal; mso-pagination:widow-orphan; font-size:8.0pt; font-family:"tahoma",sans-serif; mso-fareast-font-family:"times new roman"; color:windowtext; mso-ansi-language:en-ca; mso-fareast-language:en-us;} span.msoplaceholdertext {mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; color:gray;} p.msonospacing, li.msonospacing, div.msonospacing {mso-style-name:"no spacing\,no indent"; mso-style-noshow:yes; mso-style-priority:3; mso-style-unhide:no; mso-style-qformat:yes; mso-style-parent:""; margin:0in; margin-bottom:.0001pt; line-height:normal; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; color:windowtext; mso-ansi-language:en-ca; mso-fareast-language:en-us;} p.msormpane, li.msormpane, div.msormpane {mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-parent:""; margin:0in; margin-bottom:.0001pt; line-height:normal; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:windowtext; mso-fareast-language:en-us;} p.msolistparagraph, li.msolistparagraph, div.msolistparagraph {mso-style-noshow:yes; mso-style-priority:34; mso-style-unhide:no; mso-style-qformat:yes; mso-style-link:"list paragraph char"; margin-top:0in; margin-right:0in; margin-bottom:10.0pt; margin-left:.5in; mso-add-space:auto; line-height:115%; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:windowtext; mso-fareast-language:en-us;} p.msolistparagraphcxspfirst, li.msolistparagraphcxspfirst, div.msolistparagraphcxspfirst {mso-style-noshow:yes; mso-style-priority:34; mso-style-unhide:no; mso-style-qformat:yes; mso-style-link:"list paragraph char"; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.5in; margin-bottom:.0001pt; mso-add-space:auto; line-height:115%; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:windowtext; mso-fareast-language:en-us;} p.msolistparagraphcxspmiddle, li.msolistparagraphcxspmiddle, div.msolistparagraphcxspmiddle {mso-style-noshow:yes; mso-style-priority:34; mso-style-unhide:no; mso-style-qformat:yes; mso-style-link:"list paragraph char"; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.5in; margin-bottom:.0001pt; mso-add-space:auto; line-height:115%; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:windowtext; mso-fareast-language:en-us;} p.msolistparagraphcxsplast, li.msolistparagraphcxsplast, div.msolistparagraphcxsplast {mso-style-noshow:yes; mso-style-priority:34; mso-style-unhide:no; mso-style-qformat:yes; mso-style-link:"list paragraph char"; mso-style-type:export-only; margin-top:0in; margin-right:0in; margin-bottom:10.0pt; margin-left:.5in; mso-add-space:auto; line-height:115%; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:windowtext; mso-fareast-language:en-us;} p.msoquote, li.msoquote, div.msoquote {mso-style-noshow:yes; mso-style-priority:29; mso-style-unhide:no; mso-style-qformat:yes; mso-style-link:"quote char"; mso-style-next:normal; margin-top:10.0pt; margin-right:.6in; margin-bottom:8.0pt; margin-left:.6in; text-align:center; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:#404040; mso-themecolor:text1; mso-themetint:191; mso-fareast-language:ja; font-style:italic;} p.msointensequote, li.msointensequote, div.msointensequote {mso-style-noshow:yes; mso-style-priority:30; mso-style-unhide:no; mso-style-qformat:yes; mso-style-link:"intense quote char"; mso-style-next:normal; margin-top:.25in; margin-right:.6in; margin-bottom:.25in; margin-left:.6in; text-align:center; line-height:200%; mso-pagination:widow-orphan; border:none; mso-border-top-alt:solid #4f81bd .5pt; mso-border-top-themecolor:accent1; mso-border-bottom-alt:solid #4f81bd .5pt; mso-border-bottom-themecolor:accent1; padding:0in; mso-padding-alt:10.0pt 0in 10.0pt 0in; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:#4f81bd; mso-themecolor:accent1; mso-fareast-language:ja; font-style:italic;} p.msobibliography, li.msobibliography, div.msobibliography {mso-style-noshow:yes; mso-style-priority:37; mso-style-qformat:yes; mso-style-next:normal; margin-top:0in; margin-right:0in; margin-bottom:10.0pt; margin-left:0in; line-height:115%; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:windowtext; mso-fareast-language:en-us;} span.htmladdresschar {mso-style-name:"html address char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"html address"; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:black; mso-themecolor:text1; mso-ansi-language:en-us; mso-fareast-language:ja; font-style:italic;} span.heading1char {mso-style-name:"heading 1 char"; mso-style-priority:3; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"heading 1"; mso-ansi-font-size:14.0pt; mso-bidi-font-size:14.0pt; font-family:"cambria",serif; mso-ascii-font-family:cambria; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:cambria; mso-bidi-font-family:"times new roman"; color:#365f91; font-weight:bold;} span.heading2char {mso-style-name:"heading 2 char"; mso-style-noshow:yes; mso-style-priority:3; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"heading 2"; mso-ansi-font-size:13.0pt; mso-bidi-font-size:13.0pt; font-family:"cambria",serif; mso-ascii-font-family:cambria; mso-ascii-theme-font:major-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:major-fareast; mso-hansi-font-family:cambria; mso-hansi-theme-font:major-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:major-bidi; color:#4f81bd; mso-themecolor:accent1; font-weight:bold;} span.heading3char {mso-style-name:"heading 3 char"; mso-style-noshow:yes; mso-style-priority:3; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"heading 3"; mso-ansi-font-size:13.5pt; mso-bidi-font-size:13.5pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; mso-ansi-language:en-us; font-weight:bold;} span.heading4char {mso-style-name:"heading 4 char"; mso-style-noshow:yes; mso-style-priority:3; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"heading 4"; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"cambria",serif; mso-ascii-font-family:cambria; mso-ascii-theme-font:major-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:major-fareast; mso-hansi-font-family:cambria; mso-hansi-theme-font:major-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:major-bidi; color:black; mso-themecolor:text1; mso-ansi-language:en-us; mso-fareast-language:ja; font-weight:bold; font-style:italic;} span.heading5char {mso-style-name:"heading 5 char"; mso-style-noshow:yes; mso-style-priority:3; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"heading 5"; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"cambria",serif; mso-ascii-font-family:cambria; mso-ascii-theme-font:major-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:major-fareast; mso-hansi-font-family:cambria; mso-hansi-theme-font:major-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:major-bidi; color:black; mso-themecolor:text1; mso-ansi-language:en-us; mso-fareast-language:ja; font-style:italic;} span.heading6char {mso-style-name:"heading 6 char"; mso-style-noshow:yes; mso-style-priority:9; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"heading 6"; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"cambria",serif; mso-ascii-font-family:cambria; mso-ascii-theme-font:major-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:major-fareast; mso-hansi-font-family:cambria; mso-hansi-theme-font:major-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:major-bidi; color:#243f60; mso-themecolor:accent1; mso-themeshade:127; mso-ansi-language:en-us; mso-fareast-language:ja;} span.htmlpreformattedchar {mso-style-name:"html preformatted char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"html preformatted"; mso-ansi-font-size:10.0pt; mso-bidi-font-size:10.0pt; font-family:consolas; mso-ascii-font-family:consolas; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:consolas; mso-bidi-font-family:consolas; color:black; mso-themecolor:text1; mso-ansi-language:en-us; mso-fareast-language:ja;} p.msonormal0, li.msonormal0, div.msonormal0 {mso-style-name:msonormal; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; margin:0in; margin-bottom:.0001pt; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; color:black; mso-themecolor:text1; mso-fareast-language:ja;} span.heading7char {mso-style-name:"heading 7 char"; mso-style-noshow:yes; mso-style-priority:9; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"heading 7"; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"cambria",serif; mso-ascii-font-family:cambria; mso-ascii-theme-font:major-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:major-fareast; mso-hansi-font-family:cambria; mso-hansi-theme-font:major-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:major-bidi; color:#243f60; mso-themecolor:accent1; mso-themeshade:127; mso-ansi-language:en-us; mso-fareast-language:ja; font-style:italic;} span.heading8char {mso-style-name:"heading 8 char"; mso-style-noshow:yes; mso-style-priority:9; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"heading 8"; mso-ansi-font-size:10.5pt; mso-bidi-font-size:10.5pt; font-family:"cambria",serif; mso-ascii-font-family:cambria; mso-ascii-theme-font:major-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:major-fareast; mso-hansi-font-family:cambria; mso-hansi-theme-font:major-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:major-bidi; color:#272727; mso-themecolor:text1; mso-themetint:216; mso-ansi-language:en-us; mso-fareast-language:ja;} span.heading9char {mso-style-name:"heading 9 char"; mso-style-noshow:yes; mso-style-priority:9; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"heading 9"; mso-ansi-font-size:10.5pt; mso-bidi-font-size:10.5pt; font-family:"cambria",serif; mso-ascii-font-family:cambria; mso-ascii-theme-font:major-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:major-fareast; mso-hansi-font-family:cambria; mso-hansi-theme-font:major-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:major-bidi; color:#272727; mso-themecolor:text1; mso-themetint:216; mso-ansi-language:en-us; mso-fareast-language:ja; font-style:italic;} span.footnotetextchar {mso-style-name:"footnote text char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"footnote text"; mso-ansi-font-size:10.0pt; mso-bidi-font-size:10.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:black; mso-themecolor:text1; mso-ansi-language:en-us; mso-fareast-language:ja;} span.commenttextchar {mso-style-name:"comment text char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"comment text"; mso-ansi-font-size:10.0pt; mso-bidi-font-size:10.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:calibri; mso-bidi-font-family:"times new roman"; mso-ansi-language:en-us;} span.headerchar {mso-style-name:"header char"; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:header; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman";} span.footerchar {mso-style-name:"footer char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:footer; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman";} span.macrotextchar {mso-style-name:"macro text char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"macro text"; mso-ansi-font-size:10.0pt; mso-bidi-font-size:10.0pt; font-family:consolas; mso-ascii-font-family:consolas; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:consolas; mso-bidi-font-family:consolas; color:black; mso-themecolor:text1; mso-font-kerning:12.0pt; mso-ansi-language:en-us; mso-fareast-language:ja;} span.titlechar {mso-style-name:"title char"; mso-style-priority:1; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:title; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"cambria",serif; mso-ascii-font-family:cambria; mso-ascii-theme-font:major-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:major-fareast; mso-hansi-font-family:cambria; mso-hansi-theme-font:major-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:major-bidi; color:black; mso-themecolor:text1; mso-ansi-language:en-us; mso-fareast-language:ja;} span.closingchar {mso-style-name:"closing char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:closing; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:black; mso-themecolor:text1; mso-ansi-language:en-us; mso-fareast-language:ja;} span.signaturechar {mso-style-name:"signature char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:signature; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:black; mso-themecolor:text1; mso-ansi-language:en-us; mso-fareast-language:ja;} span.bodytextchar {mso-style-name:"body text char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"body text"; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"arial",sans-serif; mso-ascii-font-family:arial; mso-fareast-font-family:calibri; mso-hansi-font-family:arial; mso-bidi-font-family:latha; mso-ansi-language:en-us;} span.bodytextindentchar {mso-style-name:"body text indent char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"body text indent"; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:calibri; mso-bidi-font-family:"times new roman"; mso-ansi-language:en-us;} span.messageheaderchar {mso-style-name:"message header char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"message header"; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"cambria",serif; mso-ascii-font-family:cambria; mso-ascii-theme-font:major-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:major-fareast; mso-hansi-font-family:cambria; mso-hansi-theme-font:major-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:major-bidi; color:black; mso-themecolor:text1; background:#cccccc; mso-shading:windowtext; mso-pattern:gray-20 auto; mso-ansi-language:en-us; mso-fareast-language:ja;} span.salutationchar {mso-style-name:"salutation char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:salutation; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:black; mso-themecolor:text1; mso-ansi-language:en-us; mso-fareast-language:ja;} span.datechar {mso-style-name:"date char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:date; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:black; mso-themecolor:text1; mso-ansi-language:en-us; mso-fareast-language:ja;} span.bodytextfirstindentchar {mso-style-name:"body text first indent char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-parent:"body text char"; mso-style-link:"body text first indent"; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"arial",sans-serif; mso-ascii-font-family:arial; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:arial; mso-bidi-font-family:latha; color:black; mso-themecolor:text1; mso-ansi-language:en-us; mso-fareast-language:ja;} span.bodytextfirstindent2char {mso-style-name:"body text first indent 2 char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-parent:"body text indent char"; mso-style-link:"body text first indent 2"; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; mso-ansi-language:en-us;} span.noteheadingchar {mso-style-name:"note heading char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"note heading"; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:black; mso-themecolor:text1; mso-ansi-language:en-us; mso-fareast-language:ja;} span.bodytext2char {mso-style-name:"body text 2 char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"body text 2"; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:black; mso-themecolor:text1; mso-ansi-language:en-us; mso-fareast-language:ja;} span.bodytext3char {mso-style-name:"body text 3 char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"body text 3"; mso-ansi-font-size:8.0pt; mso-bidi-font-size:8.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:black; mso-themecolor:text1; mso-ansi-language:en-us; mso-fareast-language:ja;} span.bodytextindent2char {mso-style-name:"body text indent 2 char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"body text indent 2"; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:black; mso-themecolor:text1; mso-ansi-language:en-us; mso-fareast-language:ja;} span.bodytextindent3char {mso-style-name:"body text indent 3 char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"body text indent 3"; mso-ansi-font-size:8.0pt; mso-bidi-font-size:8.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:black; mso-themecolor:text1; mso-ansi-language:en-us; mso-fareast-language:ja;} span.documentmapchar {mso-style-name:"document map char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"document map"; mso-ansi-font-size:8.0pt; mso-bidi-font-size:8.0pt; font-family:"segoe ui",sans-serif; mso-ascii-font-family:"segoe ui"; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:"segoe ui"; mso-bidi-font-family:"segoe ui"; color:black; mso-themecolor:text1; mso-ansi-language:en-us; mso-fareast-language:ja;} span.plaintextchar {mso-style-name:"plain text char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"plain text"; mso-ansi-font-size:10.5pt; mso-bidi-font-size:10.5pt; font-family:consolas; mso-ascii-font-family:consolas; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:consolas; mso-bidi-font-family:consolas; color:black; mso-themecolor:text1; mso-ansi-language:en-us; mso-fareast-language:ja;} span.e-mailsignaturechar {mso-style-name:"e-mail signature char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"e-mail signature"; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:black; mso-themecolor:text1; mso-ansi-language:en-us; mso-fareast-language:ja;} span.commentsubjectchar {mso-style-name:"comment subject char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-parent:"comment text char"; mso-style-link:"comment subject"; mso-ansi-font-size:10.0pt; mso-bidi-font-size:10.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:calibri; mso-bidi-font-family:"times new roman"; mso-ansi-language:en-us; font-weight:bold;} span.balloontextchar {mso-style-name:"balloon text char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"balloon text"; mso-ansi-font-size:8.0pt; mso-bidi-font-size:8.0pt; font-family:"tahoma",sans-serif; mso-ascii-font-family:tahoma; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:tahoma; mso-bidi-font-family:tahoma;} span.listparagraphchar {mso-style-name:"list paragraph char"; mso-style-priority:34; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"list paragraph"; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-fareast-font-family:"times new roman"; mso-hansi-font-family:calibri; mso-bidi-font-family:"times new roman"; mso-ansi-language:en-us;} span.quotechar {mso-style-name:"quote char"; mso-style-priority:29; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:quote; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:#404040; mso-themecolor:text1; mso-themetint:191; mso-ansi-language:en-us; mso-fareast-language:ja; font-style:italic;} span.intensequotechar {mso-style-name:"intense quote char"; mso-style-priority:30; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"intense quote"; mso-ansi-font-size:12.0pt; mso-bidi-font-size:12.0pt; font-family:"times new roman",serif; mso-ascii-font-family:"times new roman"; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:#4f81bd; mso-themecolor:accent1; mso-ansi-language:en-us; mso-fareast-language:ja; font-style:italic;} p.heading11, li.heading11, div.heading11 {mso-style-name:"heading 11"; mso-style-noshow:yes; mso-style-priority:9; mso-style-unhide:no; mso-style-qformat:yes; mso-style-next:normal; margin-top:24.0pt; margin-right:0in; margin-bottom:0in; margin-left:0in; margin-bottom:.0001pt; line-height:115%; mso-pagination:widow-orphan lines-together; page-break-after:avoid; mso-outline-level:1; font-size:14.0pt; font-family:"cambria",serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:#365f91; mso-fareast-language:en-us; font-weight:bold;} p.titlemain, li.titlemain, div.titlemain {mso-style-name:"title main"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-qformat:yes; margin:0in; margin-bottom:.0001pt; line-height:80%; mso-pagination:widow-orphan; font-size:24.0pt; mso-bidi-font-size:12.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:black; mso-fareast-language:en-us; font-weight:bold; mso-bidi-font-weight:normal;} p.titlesubtitle, li.titlesubtitle, div.titlesubtitle {mso-style-name:"title subtitle"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-qformat:yes; margin-top:0in; margin-right:0in; margin-bottom:30.0pt; margin-left:0in; line-height:110%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"georgia",serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:black; mso-fareast-language:en-us; font-style:italic; mso-bidi-font-style:normal;} p.titleauthors, li.titleauthors, div.titleauthors {mso-style-name:"title authors"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-qformat:yes; margin:0in; margin-bottom:.0001pt; line-height:normal; mso-pagination:widow-orphan; font-size:10.0pt; mso-bidi-font-size:12.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:black; mso-fareast-language:en-us; font-weight:bold; mso-bidi-font-weight:normal;} p.references, li.references, div.references {mso-style-name:references; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-qformat:yes; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.5in; margin-bottom:.0001pt; text-indent:-.5in; line-height:normal; mso-pagination:widow-orphan; font-size:11.0pt; mso-bidi-font-size:12.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:windowtext; mso-fareast-language:en-us;} p.heading1nonumber, li.heading1nonumber, div.heading1nonumber {mso-style-name:"heading 1 \(no number\)"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-qformat:yes; mso-style-parent:"heading 1"; mso-style-next:"heading 1"; margin-top:12.0pt; margin-right:0in; margin-bottom:0in; margin-left:0in; margin-bottom:.0001pt; line-height:normal; mso-pagination:widow-orphan lines-together; page-break-after:avoid; mso-outline-level:1; font-size:14.0pt; font-family:"cambria",serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:#365f91; mso-ansi-language:en-ca; mso-fareast-language:en-us; font-weight:bold;} p.listbulletspaces, li.listbulletspaces, div.listbulletspaces {mso-style-name:"list bullet \(+spaces\)"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-qformat:yes; mso-style-parent:"list bullet"; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.25in; margin-bottom:.0001pt; text-indent:-.25in; line-height:normal; mso-pagination:widow-orphan; mso-list:l9 level1 lfo1; font-size:11.0pt; mso-bidi-font-size:12.0pt; font-family:"calibri",sans-serif; mso-fareast-font-family:"times new roman"; mso-bidi-font-family:"times new roman"; color:windowtext; mso-fareast-language:en-us;} p.sectiontitle, li.sectiontitle, div.sectiontitle {mso-style-name:"section title"; mso-style-noshow:yes; mso-style-priority:2; mso-style-unhide:no; mso-style-qformat:yes; mso-style-next:normal; margin:0in; margin-bottom:.0001pt; text-align:center; line-height:200%; page-break-before:always; mso-pagination:widow-orphan; mso-outline-level:1; font-size:12.0pt; font-family:"cambria",serif; mso-ascii-font-family:cambria; mso-ascii-theme-font:major-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:major-fareast; mso-hansi-font-family:cambria; mso-hansi-theme-font:major-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:major-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.title2, li.title2, div.title2 {mso-style-name:"title 2"; mso-style-noshow:yes; mso-style-priority:1; mso-style-unhide:no; mso-style-qformat:yes; margin:0in; margin-bottom:.0001pt; text-align:center; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.tablefigure, li.tablefigure, div.tablefigure {mso-style-name:"table\/figure"; mso-style-noshow:yes; mso-style-priority:4; mso-style-unhide:no; mso-style-qformat:yes; margin-top:12.0pt; margin-right:0in; margin-bottom:0in; margin-left:0in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.tablefigurecxspfirst, li.tablefigurecxspfirst, div.tablefigurecxspfirst {mso-style-name:"table\/figurecxspfirst"; mso-style-noshow:yes; mso-style-priority:4; mso-style-unhide:no; mso-style-qformat:yes; mso-style-type:export-only; margin-top:12.0pt; margin-right:0in; margin-bottom:0in; margin-left:0in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.tablefigurecxspmiddle, li.tablefigurecxspmiddle, div.tablefigurecxspmiddle {mso-style-name:"table\/figurecxspmiddle"; mso-style-noshow:yes; mso-style-priority:4; mso-style-unhide:no; mso-style-qformat:yes; mso-style-type:export-only; margin:0in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.tablefigurecxsplast, li.tablefigurecxsplast, div.tablefigurecxsplast {mso-style-name:"table\/figurecxsplast"; mso-style-noshow:yes; mso-style-priority:4; mso-style-unhide:no; mso-style-qformat:yes; mso-style-type:export-only; margin:0in; margin-bottom:.0001pt; mso-add-space:auto; line-height:200%; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; color:black; mso-themecolor:text1; mso-fareast-language:ja;} p.xmsonormal, li.xmsonormal, div.xmsonormal {mso-style-name:x_msonormal; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-margin-top-alt:auto; margin-right:0in; mso-margin-bottom-alt:auto; margin-left:0in; line-height:normal; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; color:windowtext; mso-fareast-language:en-us;} span.st {mso-style-name:st; mso-style-unhide:no;} span.apple-converted-space {mso-style-name:apple-converted-space; mso-style-unhide:no;} span.slug-doi {mso-style-name:slug-doi; mso-style-unhide:no;} span.heading1char1 {mso-style-name:"heading 1 char1"; mso-style-priority:9; mso-style-unhide:no; mso-ansi-font-size:16.0pt; mso-bidi-font-size:16.0pt; font-family:"cambria",serif; mso-ascii-font-family:cambria; mso-ascii-theme-font:major-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:major-fareast; mso-hansi-font-family:cambria; mso-hansi-theme-font:major-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:major-bidi; color:#365f91; mso-themecolor:accent1; mso-themeshade:191;} span.mention {mso-style-name:mention; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; color:#2b579a; background:#e6e6e6;} span.hlfld-title {mso-style-name:hlfld-title; mso-style-unhide:no;} span.unresolvedmention {mso-style-name:"unresolved mention"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; color:#605e5c; background:#e1dfdd;} .msochpdefault {mso-style-type:export-only; mso-default-props:yes; font-size:10.0pt; mso-ansi-font-size:10.0pt; mso-bidi-font-size:10.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:calibri; mso-fareast-theme-font:minor-latin; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"times new roman"; mso-bidi-theme-font:minor-bidi; mso-ansi-language:en-ca;} /* page definitions */ @page {mso-footnote-separator:url("art_hebert_29404_files/header.htm") fs; mso-footnote-continuation-separator:url("art_hebert_29404_files/header.htm") fcs; mso-endnote-separator:url("art_hebert_29404_files/header.htm") es; mso-endnote-continuation-separator:url("art_hebert_29404_files/header.htm") ecs; mso-footnote-position:beneath-text;} @page wordsection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in; mso-header-margin:.5in; mso-footer-margin:.5in; mso-columns:2 even .5in; mso-header:url("art_hebert_29404_files/header.htm") h1; mso-first-header:url("art_hebert_29404_files/header.htm") fh1; mso-paper-source:0;} div.wordsection1 {page:wordsection1; mso-footnote-position:beneath-text;} @page wordsection2 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in; mso-header-margin:.5in; mso-footer-margin:.5in; mso-header:url("art_hebert_29404_files/header.htm") h1; mso-first-header:url("art_hebert_29404_files/header.htm") fh1; mso-paper-source:0;} div.wordsection2 {page:wordsection2; mso-footnote-position:beneath-text;} /* list definitions */ @list l0 {mso-list-id:-132; mso-list-type:simple; mso-list-template-ids:1859161492;} @list l0:level1 {mso-level-style-link:"list number 5"; mso-level-tab-stop:1.25in; mso-level-number-position:left; margin-left:1.25in; text-indent:-.25in;} @list l1 {mso-list-id:-131; mso-list-type:simple; mso-list-template-ids:1028785134;} @list l1:level1 {mso-level-style-link:"list number 4"; mso-level-tab-stop:1.0in; mso-level-number-position:left; margin-left:1.0in; text-indent:-.25in;} @list l2 {mso-list-id:-130; mso-list-type:simple; mso-list-template-ids:-220424468;} @list l2:level1 {mso-level-style-link:"list number 3"; mso-level-tab-stop:.75in; mso-level-number-position:left; margin-left:.75in; text-indent:-.25in;} @list l3 {mso-list-id:-129; mso-list-type:simple; mso-list-template-ids:-1798125002;} @list l3:level1 {mso-level-style-link:"list number 2"; mso-level-tab-stop:.5in; mso-level-number-position:left; text-indent:-.25in;} @list l4 {mso-list-id:-128; mso-list-type:simple; mso-list-template-ids:398257062;} @list l4:level1 {mso-level-number-format:bullet; mso-level-style-link:"list bullet 5"; mso-level-text:; mso-level-tab-stop:1.25in; mso-level-number-position:left; margin-left:1.25in; text-indent:-.25in; font-family:symbol;} @list l5 {mso-list-id:-127; mso-list-type:simple; mso-list-template-ids:2100850686;} @list l5:level1 {mso-level-number-format:bullet; mso-level-style-link:"list bullet 4"; mso-level-text:; mso-level-tab-stop:1.0in; mso-level-number-position:left; margin-left:1.0in; text-indent:-.25in; font-family:symbol;} @list l6 {mso-list-id:-126; mso-list-type:simple; mso-list-template-ids:-684031746;} @list l6:level1 {mso-level-number-format:bullet; mso-level-style-link:"list bullet 3"; mso-level-text:; mso-level-tab-stop:.75in; mso-level-number-position:left; margin-left:.75in; text-indent:-.25in; font-family:symbol;} @list l7 {mso-list-id:-125; mso-list-type:simple; mso-list-template-ids:179339368;} @list l7:level1 {mso-level-number-format:bullet; mso-level-style-link:"list bullet 2"; mso-level-text:; mso-level-tab-stop:.5in; mso-level-number-position:left; text-indent:-.25in; font-family:symbol;} @list l8 {mso-list-id:-120; mso-list-type:simple; mso-list-template-ids:-689962352;} @list l8:level1 {mso-level-style-link:"list number"; mso-level-tab-stop:.75in; mso-level-number-position:left; margin-left:.75in; text-indent:-.25in;} @list l9 {mso-list-id:-119; mso-list-type:simple; mso-list-template-ids:877445814;} @list l9:level1 {mso-level-number-format:bullet; mso-level-style-link:"list bullet"; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:.25in; text-indent:-.25in; font-family:wingdings;} @list l10 {mso-list-id:34239747; mso-list-type:hybrid; mso-list-template-ids:-1527472752 67698713 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;} @list l10:level1 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.75in; text-indent:-.25in;} @list l10:level2 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.25in; text-indent:-.25in;} @list l10:level3 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:3.75in; text-indent:-9.0pt;} @list l10:level4 {mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.25in; text-indent:-.25in;} @list l10:level5 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.75in; text-indent:-.25in;} @list l10:level6 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:5.25in; text-indent:-9.0pt;} @list l10:level7 {mso-level-tab-stop:none; mso-level-number-position:left; margin-left:5.75in; text-indent:-.25in;} @list l10:level8 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:6.25in; text-indent:-.25in;} @list l10:level9 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:6.75in; text-indent:-9.0pt;} @list l11 {mso-list-id:64767349; mso-list-type:hybrid; mso-list-template-ids:-1489367586 67698713 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;} @list l11:level1 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l11:level2 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l11:level3 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; text-indent:-9.0pt;} @list l11:level4 {mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l11:level5 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l11:level6 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; text-indent:-9.0pt;} @list l11:level7 {mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l11:level8 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l11:level9 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; text-indent:-9.0pt;} @list l12 {mso-list-id:271938444; mso-list-type:hybrid; mso-list-template-ids:825255144 67698713 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;} @list l12:level1 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l12:level2 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l12:level3 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; text-indent:-9.0pt;} @list l12:level4 {mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l12:level5 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l12:level6 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; text-indent:-9.0pt;} @list l12:level7 {mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l12:level8 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l12:level9 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; text-indent:-9.0pt;} @list l13 {mso-list-id:358552042; mso-list-type:hybrid; mso-list-template-ids:-881687556 67698713 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;} @list l13:level1 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.0in; text-indent:-.25in;} @list l13:level2 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.5in; text-indent:-.25in;} @list l13:level3 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:2.0in; text-indent:-9.0pt;} @list l13:level4 {mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.5in; text-indent:-.25in;} @list l13:level5 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.0in; text-indent:-.25in;} @list l13:level6 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:3.5in; text-indent:-9.0pt;} @list l13:level7 {mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.0in; text-indent:-.25in;} @list l13:level8 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.5in; text-indent:-.25in;} @list l13:level9 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:5.0in; text-indent:-9.0pt;} @list l14 {mso-list-id:584727393; mso-list-type:hybrid; mso-list-template-ids:1483505488 67698713 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;} @list l14:level1 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.0in; text-indent:-.25in;} @list l14:level2 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.5in; text-indent:-.25in;} @list l14:level3 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:2.0in; text-indent:-9.0pt;} @list l14:level4 {mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.5in; text-indent:-.25in;} @list l14:level5 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.0in; text-indent:-.25in;} @list l14:level6 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:3.5in; text-indent:-9.0pt;} @list l14:level7 {mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.0in; text-indent:-.25in;} @list l14:level8 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.5in; text-indent:-.25in;} @list l14:level9 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:5.0in; text-indent:-9.0pt;} @list l15 {mso-list-id:751898990; mso-list-type:hybrid; mso-list-template-ids:-1196370326 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;} @list l15:level1 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:symbol;} @list l15:level2 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:"courier new";} @list l15:level3 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:wingdings;} @list l15:level4 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:symbol;} @list l15:level5 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:"courier new";} @list l15:level6 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:wingdings;} @list l15:level7 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:symbol;} @list l15:level8 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:"courier new";} @list l15:level9 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:wingdings;} @list l16 {mso-list-id:1008752568; mso-list-type:hybrid; mso-list-template-ids:-356637682 67698713 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;} @list l16:level1 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.0in; text-indent:-.25in;} @list l16:level2 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.5in; text-indent:-.25in;} @list l16:level3 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:2.0in; text-indent:-9.0pt;} @list l16:level4 {mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.5in; text-indent:-.25in;} @list l16:level5 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.0in; text-indent:-.25in;} @list l16:level6 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:3.5in; text-indent:-9.0pt;} @list l16:level7 {mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.0in; text-indent:-.25in;} @list l16:level8 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.5in; text-indent:-.25in;} @list l16:level9 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:5.0in; text-indent:-9.0pt;} @list l17 {mso-list-id:1038311107; mso-list-type:hybrid; mso-list-template-ids:780463474 67698713 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;} @list l17:level1 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.0in; text-indent:-.25in;} @list l17:level2 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.5in; text-indent:-.25in;} @list l17:level3 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:2.0in; text-indent:-9.0pt;} @list l17:level4 {mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.5in; text-indent:-.25in;} @list l17:level5 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.0in; text-indent:-.25in;} @list l17:level6 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:3.5in; text-indent:-9.0pt;} @list l17:level7 {mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.0in; text-indent:-.25in;} @list l17:level8 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.5in; text-indent:-.25in;} @list l17:level9 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:5.0in; text-indent:-9.0pt;} @list l18 {mso-list-id:1113864892; mso-list-type:hybrid; mso-list-template-ids:35173894 67698713 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;} @list l18:level1 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.0in; text-indent:-.25in;} @list l18:level2 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.5in; text-indent:-.25in;} @list l18:level3 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:2.0in; text-indent:-9.0pt;} @list l18:level4 {mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.5in; text-indent:-.25in;} @list l18:level5 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.0in; text-indent:-.25in;} @list l18:level6 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:3.5in; text-indent:-9.0pt;} @list l18:level7 {mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.0in; text-indent:-.25in;} @list l18:level8 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.5in; text-indent:-.25in;} @list l18:level9 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:5.0in; text-indent:-9.0pt;} @list l19 {mso-list-id:1161120208; mso-list-type:hybrid; mso-list-template-ids:81272358 67698713 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;} @list l19:level1 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.0in; text-indent:-.25in;} @list l19:level2 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.5in; text-indent:-.25in;} @list l19:level3 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:2.0in; text-indent:-9.0pt;} @list l19:level4 {mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.5in; text-indent:-.25in;} @list l19:level5 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.0in; text-indent:-.25in;} @list l19:level6 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:3.5in; text-indent:-9.0pt;} @list l19:level7 {mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.0in; text-indent:-.25in;} @list l19:level8 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.5in; text-indent:-.25in;} @list l19:level9 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:5.0in; text-indent:-9.0pt;} @list l20 {mso-list-id:1291932031; mso-list-type:hybrid; mso-list-template-ids:-2054132642 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;} @list l20:level1 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:symbol;} @list l20:level2 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:"courier new";} @list l20:level3 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:wingdings;} @list l20:level4 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:symbol;} @list l20:level5 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:"courier new";} @list l20:level6 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:wingdings;} @list l20:level7 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:symbol;} @list l20:level8 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:"courier new";} @list l20:level9 {mso-level-number-format:bullet; mso-level-text:; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in; font-family:wingdings;} @list l21 {mso-list-id:1405567142; mso-list-type:hybrid; mso-list-template-ids:-1133324984 67698713 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;} @list l21:level1 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.0in; text-indent:-.25in;} @list l21:level2 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.5in; text-indent:-.25in;} @list l21:level3 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:2.0in; text-indent:-9.0pt;} @list l21:level4 {mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.5in; text-indent:-.25in;} @list l21:level5 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.0in; text-indent:-.25in;} @list l21:level6 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:3.5in; text-indent:-9.0pt;} @list l21:level7 {mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.0in; text-indent:-.25in;} @list l21:level8 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.5in; text-indent:-.25in;} @list l21:level9 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:5.0in; text-indent:-9.0pt;} @list l22 {mso-list-id:1414816484; mso-list-type:hybrid; mso-list-template-ids:56913722 67698713 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;} @list l22:level1 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.0in; text-indent:-.25in;} @list l22:level2 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.5in; text-indent:-.25in;} @list l22:level3 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:2.0in; text-indent:-9.0pt;} @list l22:level4 {mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.5in; text-indent:-.25in;} @list l22:level5 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.0in; text-indent:-.25in;} @list l22:level6 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:3.5in; text-indent:-9.0pt;} @list l22:level7 {mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.0in; text-indent:-.25in;} @list l22:level8 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.5in; text-indent:-.25in;} @list l22:level9 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:5.0in; text-indent:-9.0pt;} @list l23 {mso-list-id:1667857516; mso-list-type:hybrid; mso-list-template-ids:-80428308 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;} @list l23:level1 {mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l23:level2 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l23:level3 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; text-indent:-9.0pt;} @list l23:level4 {mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l23:level5 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l23:level6 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; text-indent:-9.0pt;} @list l23:level7 {mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l23:level8 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l23:level9 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; text-indent:-9.0pt;} @list l24 {mso-list-id:1688097652; mso-list-type:hybrid; mso-list-template-ids:-1797354100 67698713 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;} @list l24:level1 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.0in; text-indent:-.25in;} @list l24:level2 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:1.5in; text-indent:-.25in;} @list l24:level3 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:2.0in; text-indent:-9.0pt;} @list l24:level4 {mso-level-tab-stop:none; mso-level-number-position:left; margin-left:2.5in; text-indent:-.25in;} @list l24:level5 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:3.0in; text-indent:-.25in;} @list l24:level6 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:3.5in; text-indent:-9.0pt;} @list l24:level7 {mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.0in; text-indent:-.25in;} @list l24:level8 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:4.5in; text-indent:-.25in;} @list l24:level9 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:5.0in; text-indent:-9.0pt;} @list l25 {mso-list-id:1840997160; mso-list-type:hybrid; mso-list-template-ids:-1633243102 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;} @list l25:level1 {mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l25:level2 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l25:level3 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; text-indent:-9.0pt;} @list l25:level4 {mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l25:level5 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l25:level6 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; text-indent:-9.0pt;} @list l25:level7 {mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l25:level8 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l25:level9 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; text-indent:-9.0pt;} ol {margin-bottom:0in;} ul {margin-bottom:0in;} --> </style> <!--[if gte mso 10]> <style> /* style definitions */ table.msonormaltable {mso-style-name:"table normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; mso-ansi-language:en-ca;} table.msotablegrid {mso-style-name:"table grid"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-priority:39; mso-style-unhide:no; border:solid windowtext 1.0pt; mso-border-alt:solid windowtext .5pt; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-border-insideh:.5pt solid windowtext; mso-border-insidev:.5pt solid windowtext; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"times new roman",serif; mso-fareast-font-family:simsun; mso-bidi-font-family:arial; mso-ansi-language:en-ca; mso-fareast-language:zh-cn; mso-bidi-language:hi;} table.msotablegridlight {mso-style-name:"table grid light"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-priority:40; mso-style-unhide:no; border:solid #bfbfbf 1.0pt; mso-border-themecolor:background1; mso-border-themeshade:191; mso-border-alt:solid #bfbfbf .5pt; mso-border-themecolor:background1; mso-border-themeshade:191; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-border-insideh:.5pt solid #bfbfbf; mso-border-insideh-themecolor:background1; mso-border-insideh-themeshade:191; mso-border-insidev:.5pt solid #bfbfbf; mso-border-insidev-themecolor:background1; mso-border-insidev-themeshade:191; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; text-indent:.5in; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; color:black; mso-themecolor:text1; mso-ansi-language:en-ca; mso-fareast-language:ja;} table.apareport {mso-style-name:"apa report"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-priority:99; mso-style-unhide:no; border-top:solid windowtext 1.5pt; border-left:none; border-bottom:solid windowtext 1.5pt; border-right:none; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:12.0pt; font-family:"calibri",sans-serif; mso-ascii-font-family:calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"times new roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:calibri; mso-hansi-theme-font:minor-latin; color:black; mso-themecolor:text1; mso-ansi-language:en-ca; mso-fareast-language:ja;} table.apareportfirstrow {mso-style-name:"apa report"; mso-table-condition:first-row; mso-style-priority:99; mso-style-unhide:no; mso-tstyle-border-top:1.5pt solid windowtext; mso-tstyle-border-left:cell-none; mso-tstyle-border-bottom:1.5pt solid windowtext; mso-tstyle-border-right:cell-none; mso-tstyle-diagonal-down:cell-none; mso-tstyle-diagonal-up:cell-none; mso-tstyle-border-insideh:cell-none; mso-tstyle-border-insidev:cell-none;} </style> <![endif]--><!--[if gte mso 9]><xml> <o:shapedefaults v:ext="edit" spidmax="1026"/> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext="edit"> <o:idmap v:ext="edit" data="1"/> </o:shapelayout></xml><![endif]--> </head> <body lang=en-us link=blue vlink=purple style='tab-interval:.5in'> <div class=wordsection1> <p class=msonormal style='mso-outline-level:1'><i style='mso-bidi-font-style: normal'><span lang=en-ca style='font-family:"palatino linotype",serif; mso-bidi-font-weight:bold'>research article<o:p></o:p></span></i></p> <p class=msonormal><b><span lang=en-ca style='font-size:10.0pt;font-family: "palatino linotype",serif'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal style='mso-pagination:none'><b style='mso-bidi-font-weight: normal'><span lang=en-ca style='font-size:11.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>information literacy skills of first-year library and information science graduate students: an exploratory study<o:p></o:p></span></b></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-weight:bold'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>andrea hebert<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>human sciences, education, and distance learning librarian<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>lsu libraries<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>louisiana state university<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>baton rouge, louisiana, united states of america<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>email: <span class=msohyperlink><a href="mailto:ahebert@lsu.edu">ahebert@lsu.edu</a></span><o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-bidi-font-weight:bold'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b><span lang=en-ca style='font-size:10.0pt;font-family: "palatino linotype",serif'>received</span></b><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-weight: bold'>: </span><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-bidi-font-weight:bold'>7 feb. 2018</span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-weight:bold'><span style='mso-tab-count:1'>     </span><b>accepted</b>: </span><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-us;mso-bidi-font-weight:bold'>17 july 2018</span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-weight:bold'><o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:9.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:arial;mso-ansi-language:en-us;mso-no-proof:yes'><!--[if gte vml 1]><v:shapetype id="_x0000_t75" coordsize="21600,21600" o:spt="75" o:preferrelative="t" path="m@4@5l@4@11@9@11@9@5xe" filled="f" stroked="f"> <v:stroke joinstyle="miter"/> <v:formulas> <v:f eqn="if linedrawn pixellinewidth 0"/> <v:f eqn="sum @0 1 0"/> <v:f eqn="sum 0 0 @1"/> <v:f eqn="prod @2 1 2"/> <v:f eqn="prod @3 21600 pixelwidth"/> <v:f eqn="prod @3 21600 pixelheight"/> <v:f eqn="sum @0 0 1"/> <v:f eqn="prod @6 1 2"/> <v:f eqn="prod @7 21600 pixelwidth"/> <v:f eqn="sum @8 21600 0"/> <v:f eqn="prod @7 21600 pixelheight"/> <v:f eqn="sum @10 21600 0"/> </v:formulas> <v:path o:extrusionok="f" gradientshapeok="t" o:connecttype="rect"/> <o:lock v:ext="edit" aspectratio="t"/> </v:shapetype><v:shape id="picture_x0020_4" o:spid="_x0000_i1025" type="#_x0000_t75" alt="cc-ca_logo_xl" style='width:7.5pt;height:7.5pt;visibility:visible; mso-wrap-style:square'> <v:imagedata src="art_hebert_29404_files/image005.png" o:title="cc-ca_logo_xl" cropright="49941f"/> </v:shape><![endif]--><![if !vml]><img border=0 width=10 height=10 src="art_hebert_29404_files/image001.png" alt="cc-ca_logo_xl" v:shapes="picture_x0020_4"><![endif]></span><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:arial'><span style='mso-spacerun:yes'> </span>2018 </span><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif; mso-bidi-font-weight:bold'>hebert</span><span lang=en-ca style='font-size:9.0pt; font-family:"palatino linotype",serif'>. this is an open access article distributed under the terms of the creative commons</span><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: "cambria math"'>‐</span><span lang=en-ca style='font-size:9.0pt;font-family: "palatino linotype",serif'>attribution</span><span lang=en-ca style='font-size: 9.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family:"cambria math"'>‐</span><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif'>noncommercial</span><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:"cambria math"'>‐</span><span lang=en-ca style='font-size: 9.0pt;font-family:"palatino linotype",serif'>share alike license 4.0 international (<span class=msohyperlink><a href="http://creativecommons.org/licenses/by-nc-sa/4.0/">http://creativecommons.org/licenses/by-nc-sa/4.0/</a></span>), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.<o:p></o:p></span></p> <div style='mso-element:para-border-div;border:none;border-bottom:solid black 1.0pt; mso-border-bottom-alt:solid black .25pt;padding:0in 0in 1.0pt 0in'> <p class=msobodytext style='border:none;mso-border-bottom-alt:solid black .25pt; padding:0in;mso-padding-alt:0in 0in 1.0pt 0in'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-ca'><o:p>&nbsp;</o:p></span></p> <p class=msobodytext style='border:none;mso-border-bottom-alt:solid black .25pt; padding:0in;mso-padding-alt:0in 0in 1.0pt 0in'><span lang=en-ca style='font-size:9.0pt;font-family:"palatino linotype",serif;color:blue; mso-ansi-language:en-ca'>doi</span><span lang=en-ca style='font-size:9.0pt; font-family:"palatino linotype",serif;mso-ansi-language:en-ca'>: 10.18438/eblip29404<o:p></o:p></span></p> <p class=msobodytext style='border:none;mso-border-bottom-alt:solid black .25pt; padding:0in;mso-padding-alt:0in 0in 1.0pt 0in'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-ca'><o:p>&nbsp;</o:p></span></p> </div> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msobodytext style='margin-left:.5in'><b style='mso-bidi-font-weight: normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif'>abstract<o:p></o:p></span></b></p> <p class=msobodytext style='margin-left:.5in'><b style='mso-bidi-font-weight: normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></b></p> <p class=msobodytext style='margin-left:.5in'><b style='mso-bidi-font-weight: normal'><span lang=en-gb style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-gb'>objective </span></b><span lang=en-gb style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-gb'>– </span><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>this cross-sectional, descriptive study seeks to address a gap in knowledge of both information literacy (il) self-efficacy and il skills of students entering louisiana state university’s master of library and information science (mlis) program.</span><b style='mso-bidi-font-weight:normal'><span lang=en-gb style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-gb'><o:p></o:p></span></b></p> <p class=msobodytext style='margin-left:.5in'><b style='mso-bidi-font-weight: normal'><span lang=en-gb style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-gb'><o:p>&nbsp;</o:p></span></b></p> <p class=msobodytext style='margin-left:.5in'><b style='mso-bidi-font-weight: normal'><span lang=en-gb style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-gb'>methods </span></b><span lang=en-gb style='font-size: 10.0pt;font-family:"palatino linotype",serif;mso-ansi-language:en-gb'>– </span><span style='font-size:10.0pt;font-family:"palatino linotype",serif'>an online survey testing both il self-efficacy and skills was administered through qualtrics. the online survey instrument used items from existing instruments </span><!--[if supportfields]><span style='font-size:10.0pt;font-family:"palatino linotype",serif'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;abstract&quot; : &quot;few constituencies exist where it is more important to produce information literate individuals than teacher candidates, yet rarely is it suggested that practitioners newly entering the field are adequately prepared to teach and model information literacy to their students. as a result, information literacy has been established as a key outcome by a number of teacher education accrediting bodies and professional associations. corollary to this initiative is the effort to develop valid instruments that assess information literacy skills of teacher candidates. this paper describes the development and validation of the information literacy assessment scale in education (ilas-ed). this instrument can be used to inform curricular and instructional decisions and to provide evidence of institutional effectiveness for program reviews.&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;beile&quot;, &quot;given&quot; : &quot;penny m&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;society for information technology &amp; teacher education international conference&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;1&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2007&quot; ] ] }, &quot;page&quot; : &quot;1-27&quot;, &quot;title&quot; : &quot;the ilas-ed: a standards-based instrument for assessing pre-service teachers\u2019 information literacy levels&quot;, &quot;type&quot; : &quot;paper-conference&quot;, &quot;volume&quot; : &quot;2007&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=4e7969cf-c30c-4be7-abc3-11ba4e156a90&quot; ] }, { &quot;id&quot; : &quot;item-2&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1080/08963568.2016.1145787&quot;, &quot;issn&quot; : &quot;08963568&quot;, &quot;abstract&quot; : &quot;in 2015, the information literacy (il) team designed an instrument to determine international graduate business students' perceptions of their il skills and their actual test-assessed il skills. the purpose of this research was to compare international graduate students' perceptions of their il skills versus their test-assessed competencies, with the aim of creating training modules to improve test-identified deficiencies. results demonstrate that wide discrepancies exist between students' perceptions of their own skills and their actual skills on all six ila-topic areas assessed by the internally developed test instruments, with students overestimating their abilities in all areas. [abstract from publisher]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;michalak&quot;, &quot;given&quot; : &quot;russell&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;rysavy&quot;, &quot;given&quot; : &quot;monica&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of business &amp; finance librarianship&quot;, &quot;id&quot; : &quot;item-2&quot;, &quot;issue&quot; : &quot;2&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2016&quot;, &quot;4&quot; ] ] }, &quot;note&quot; : &quot;accession number: 116263533; michalak, russell 1; rysavy, monica d. t. 1; affiliations: 1 : goldey-beacom college, wilmington, delaware, usa; source info: apr-jun2016, vol. 21 issue 2, p152; thesaurus term: information literacy -research; thesaurus term: business information services; subject term: instructional systems design; subject term: international business enterprises; subject term: graduate students; author-supplied keyword: business information literacy; author-supplied keyword: information literacy assessment; author-supplied keyword: information literacy skills; author-supplied keyword: instructional design; author-supplied keyword: international business graduate students; number of pages: 23p; document type: article&quot;, &quot;page&quot; : &quot;152-174&quot;, &quot;title&quot; : &quot;information literacy in 2015: international graduate business students' perceptions of information literacy skills compared to test-assessed skills.&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;21&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=79e910db-f957-4128-857a-5ce4b0a15542&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(beile, 2007; michalak &amp; rysavy, 2016)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(beile, 2007; michalak &amp; rysavy, 2016)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(beile, 2007; michalak &amp; rysavy, 2016)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span style='font-size:10.0pt;font-family:"palatino linotype",serif'>(beile, 2007; michalak &amp; rysavy, 2016)</span><!--[if supportfields]><span style='font-size:10.0pt;font-family:"palatino linotype",serif'><span style='mso-element:field-end'></span></span><![endif]--><span style='font-size: 10.0pt;font-family:"palatino linotype",serif'> and was distributed to two cohorts of incoming students; the first cohort entered the mlis program in fall 2017, and the second entered in spring 2018.<o:p></o:p></span></p> <p class=msobodytext style='margin-left:.5in'><span style='font-size:10.0pt; font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msobodytext style='margin-left:.5in'><b style='mso-bidi-font-weight: normal'><span lang=en-gb style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-gb'>results </span></b><span lang=en-gb style='font-size: 10.0pt;font-family:"palatino linotype",serif;mso-ansi-language:en-gb'>– </span><span style='font-size:10.0pt;font-family:"palatino linotype",serif'>data varied between cohorts and between survey instruments for both il self-efficacy and skills; however, bivariate analysis of data indicated a moderate positive correlation between overall il self-efficacy and demonstrated il skill scores in both fall 2017 and spring 2018 cohorts. <o:p></o:p></span></p> <p class=msobodytext style='margin-left:.5in'><b style='mso-bidi-font-weight: normal'><span lang=en-gb style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-gb'><o:p>&nbsp;</o:p></span></b></p> <p class=msobodytext style='margin-left:.5in'><b style='mso-bidi-font-weight: normal'><span lang=en-gb style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-ansi-language:en-gb'>conclusion </span></b><span lang=en-gb style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-ansi-language: en-gb'>–</span><span lang=en-gb style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-fareast-theme-font:minor-fareast; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-gb;mso-fareast-language:ja'> </span><span style='font-size:10.0pt; font-family:"palatino linotype",serif'>the study indicates a need for a larger, multi-institutional study using a rigorously validated instrument to gather data and make generalizable inferences about the il self-efficacy and skills of incoming lis graduate students.<o:p></o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>introduction</span></b><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>students enrolled in u.s. library and information science (lis) graduate programs are an understudied population in lis literature. most articles focus on lis curricula and teaching methodologies. very few published studies focus on the fundamental information literacy (il) skill set of students entering library school. because of this lack of data, research and instruction services librarians who work with lis graduate students are unable to anticipate accurately these students’ information needs and information literacy proficiencies, making it a challenge to provide support and instruction. <o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>lis students in the united states are a heterogeneous mix. lis graduate programs pull students from a wide range of undergraduate majors </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;issn&quot; : &quot;10949054&quot;, &quot;abstract&quot; : &quot;our paper presents the results of a survey of mlis students' motivations for choosing a library career, as well as their outlook on the job market, preferences for various subfields, and dreams about the future. in 2004 several researchers conducted a survey of mlis students at the university of alabama's school of library and information studies and reported the results in a 2006 rusq article. in a field of constant change, it is essential that a new study is conducted to glean current motivations for pursuing a mlis degree. new technologies, economic issues, and other factors could affect a new generation of librarians' mindsets. therefore we replicated the earlier survey, added some new questions, and compared our results. this article describes the results of the current survey. it shows that myriad reasons motivate students to pursue the mlis, and librarians who have an interest in their new colleagues will find this paper of interest. we also suggest some practical steps that reference librarians (as well as human resource officers and library school administrators) can follow to recruit new librarians. this paper is based on a poster presented at the alabama library association conference (alla), april 10, 2009, in auburn, alabama. [abstract from author]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;taylor&quot;, &quot;given&quot; : &quot;stephanie d&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;perry&quot;, &quot;given&quot; : &quot;r alexander&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;barton&quot;, &quot;given&quot; : &quot;jessica l&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;spencer&quot;, &quot;given&quot; : &quot;brett&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;reference &amp; user services quarterly&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;1&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2010&quot; ] ] }, &quot;note&quot; : &quot;accession number: 53897400; taylor, stephanie d. 1; perry, r. alexander 1; barton, jessica l. 1; spencer, brett 1,2; affiliations: 1 : university of alabama's school of library.; 2 : reference librarian, university of alabama, tuscaloosa.; source info: fall2010, vol. 50 issue 1, p35; thesaurus term: library education; thesaurus term: librarians -employment; thesaurus term: librarian recruiting; thesaurus term: reference librarians; thesaurus term: surveys; subject term: vocational guidance; subject term: wages; number of pages: 13p; illustrations: 7 graphs; document type: article&quot;, &quot;page&quot; : &quot;35-47&quot;, &quot;publisher&quot; : &quot;american library association&quot;, &quot;title&quot; : &quot;a follow-up study of the factors shaping the career choices of library school students at the university of alabama&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;50&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=44a0f9cc-898d-4922-a9a7-355c75c6b022&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(taylor, perry, barton, &amp; spencer, 2010)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(taylor, perry, barton, &amp; spencer, 2010)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(taylor, perry, barton, &amp; spencer, 2010)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(taylor, perry, barton, &amp; spencer, 2010)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>, and approximately 49% of students enrolled in american library association (ala) accredited master’s programs in the united states are 30 years of age or more </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;albertson&quot;, &quot;given&quot; : &quot;dan&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;spetka&quot;, &quot;given&quot; : &quot;kaitlyn&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;snow&quot;, &quot;given&quot; : &quot;kristen&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;id&quot; : &quot;item-1&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2015&quot; ] ] }, &quot;publisher-place&quot; : &quot;seattle, wa&quot;, &quot;title&quot; : &quot;alise library and information science education statistical report 2015&quot;, &quot;type&quot; : &quot;report&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=1b9c57c9-5f33-436c-b9d3-147bbf90d2f0&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(albertson, spetka, &amp; snow, 2015)&quot;, &quot;manualformatting&quot; : &quot;(albertson, spetka, &amp; snow, 2015, table ii-8-c-2-ala)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(albertson, spetka, &amp; snow, 2015)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(albertson, spetka, &amp; snow, 2015)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(albertson, spetka, &amp; snow, 2015, table ii-8-c-2-ala)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>, suggesting that they are returning to academia after professional employment. the varied academic and professional backgrounds of lis graduate students make it hard to predict what il skills incoming students may possess. at louisiana state university (lsu), it is not uncommon to encounter new lis graduate students who cannot look up a book in an opac, cannot distinguish a citation for a journal article from that of a monograph, and who are unfamiliar with peer review, but librarians who work with lis graduate students need more than anecdotal information about these students to serve them efficiently and effectively. <o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>likewise, understanding students’ il self-efficacy can guide librarians in their outreach and instruction to this population. students with low self-efficacy need additional encouragement and guidance </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1016/j.acalib.2013.08.008&quot;, &quot;issn&quot; : &quot;00991333&quot;, &quot;abstract&quot; : &quot;this study investigates distance learners' information literacy skills in using digital library resources and the factors (online learning and information manipulation) that correlate with learners' information seeking self-efficacy. in addition, distance learners' preferences with regard to digital resources selection and interests of developing information seeking skills were examined. 3517 students enrolled in one or more distance education courses were invited to participate in the online survey; 219 students completed the survey, for a response rate of 6.2%. the results revealed that distance learners who have higher self-efficacy for information seeking and proficiency in information manipulation exhibited higher self-efficacy for online learning. moreover, students with high self-efficacy demonstrated superior knowledge of digital resources selection. students who have low self-efficacy with regard to information seeking were more likely to express interest in learning how to use the library resources, although learning techniques for database searching was the exception. [copyright &amp;y&amp; elsevier]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;tang&quot;, &quot;given&quot; : &quot;yingqi&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;tseng&quot;, &quot;given&quot; : &quot;hung wei&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of academic librarianship&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;6&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2013&quot; ] ] }, &quot;note&quot; : &quot;accession number: 95607608; tang, yingqi 1; email address: tang@jsu.edu; tseng, hung wei 2; email address: htseng@jsu.edu; affiliations: 1 : houston cole library, jacksonville state university, 700 pelham rd. n, jacksonville, al 36256-1602, united states; 2 : office of distance education, jacksonville state university, 700 pelham rd. n, 212a self hall, jacksonville, al 36256-1602, united states; source info: nov2013, vol. 39 issue 6, p517; thesaurus term: research; thesaurus term: information literacy; thesaurus term: digital libraries -use studies; thesaurus term: electronic information resources; subject term: distance education students; subject term: self-efficacy in students; author-supplied keyword: academic library; author-supplied keyword: e-learning; author-supplied keyword: information manipulation; author-supplied keyword: information seeking; author-supplied keyword: self-efficacy; number of pages: 5p; document type: article&quot;, &quot;page&quot; : &quot;517-521&quot;, &quot;title&quot; : &quot;distance learners' self-efficacy and information literacy skills&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;39&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=600c7dbf-2ce2-45d4-b4be-26fbaf8e2ea7&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(tang &amp; tseng, 2013)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(tang &amp; tseng, 2013)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(tang &amp; tseng, 2013)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(tang &amp; tseng, 2013)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>. however, if students’ self-efficacy is higher than their actual skill level, students may be unaware of their weaknesses and may be unlikely to seek help </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1002/asi.21681&quot;, &quot;issn&quot; : &quot;15322882&quot;, &quot;abstract&quot; : &quot;this study replicates a previous study based on work in psychology, which demonstrates that students who score as below proficient in information literacy (il) skills have a miscalibrated self-view of their ability. simply stated, these students tend to believe that they have above-average il skills, when, in fact, an objective test of their ability indicates that they are below-proficient in terms of their actual skills. this investigation was part of an institute of museum and library services-funded project and includes demographic data about participants, their scores on an objective test of their information literacy skills, and self-estimates of their ability. findings support previous research that indicates many students come to college without proficient il skills, that students with below-proficient il skills have inflated views of their ability, and that this miscalibration can also be expressed by students who test as proficient. implications for research and practice are discussed. [abstract from author]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;gross&quot;, &quot;given&quot; : &quot;melissa&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;latham&quot;, &quot;given&quot; : &quot;don&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of the american society for information science &amp; technology&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;3&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2012&quot;, &quot;3&quot; ] ] }, &quot;note&quot; : &quot;accession number: 71885232; gross, melissa 1; latham, don 2; affiliations: 1: school of library and information studies, florida state university, 246 louis shores building, tallahassee, florida; 2: school of library and information studies, florida state university, 248 louis shores building, tallahassee, florida; issue info: mar2012, vol. 63 issue 3, p574; thesaurus term: research -finance; subject term: asians; subject term: blacks; subject term: college students; subject term: community colleges; subject term: hispanic americans; subject term: indigenous peoples of the americas; subject term: self-perception; subject term: t-test (statistics); subject term: whites; subject term: information literacy; subject term: data analysis -software; naics/industry codes: 611210 junior colleges; number of pages: 10p; document type: article&quot;, &quot;page&quot; : &quot;574-583&quot;, &quot;publisher&quot; : &quot;john wiley &amp; sons, inc.&quot;, &quot;title&quot; : &quot;what's skill got to do with it?: information literacy skills and self-views of ability among first-year college students.&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;63&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=dcaf775b-3b49-4177-963f-e32619beda96&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(gross &amp; latham, 2012)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(gross &amp; latham, 2012)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(gross &amp; latham, 2012)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(gross &amp; latham, 2012)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>. librarians may need to promote their expertise and services more heavily not only to students with low self-efficacy but also to those students who have high levels of il self-efficacy but lower levels of demonstrated il skills.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>literature review<o:p></o:p></span></b></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>bandura defines perceived self-efficacy “as people’s judgments of their capabilities to organize and execute courses of action required to attain designated types of performances” </span><!--[if supportfields]><span lang=en-ca style='font-size: 10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin; mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;issn&quot; : &quot;0033295x&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;bandura&quot;, &quot;given&quot; : &quot;albert&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;psychological review&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;1977&quot;, &quot;3&quot; ] ] }, &quot;note&quot; : &quot;accession number: 520769948; authors:bandura, albert; physical description: bibliography; subject: learning, psychology of -social learning; subject: adaptability (psychology); subject: avoidance (psychology); subject: expectation (psychology); number of pages: 25p; language: undetermined&quot;, &quot;page&quot; : &quot;191-215&quot;, &quot;title&quot; : &quot;self-efficacy: toward a unifying theory of behavioral change.&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;84&quot; }, &quot;locator&quot; : &quot;391&quot;, &quot;suppress-author&quot; : 1, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=1acc8a6f-0c88-4077-b115-c958154a0c27&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(1977, p. 391)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(1977, p. 391)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(1977, p. 391)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(1977, p. 391)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>. people with positive self-efficacy beliefs are more likely to engage in activities that improve actual competencies, but bandura (1986) is careful to note that misjudgments of self-efficacy (overestimating or underestimating one’s talents) can cause a negative impact. people who underestimate their self-efficacy often limit themselves and underperform because of self-doubt, while those who greatly overestimate their abilities expose themselves to frustration and failure </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;isbn&quot; : &quot;013815614x&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;bandura&quot;, &quot;given&quot; : &quot;albert&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;collection-title&quot; : &quot;prentice-hall series in social learning theory&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;1986&quot; ] ] }, &quot;note&quot; : &quot;accession number: lalu.1945446; other notes: includes indexes.; bibliography: p. 523-582.; publication type: book; physical description: xiii, 617 p. ; 25 cm.; language: english; lccn: 85009310; oclc: 12080269&quot;, &quot;publisher&quot; : &quot;englewood cliffs, n.j. : prentice-hall&quot;, &quot;title&quot; : &quot;social foundations of thought and action: a social cognitive theory&quot;, &quot;type&quot; : &quot;book&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=ef41b833-5b94-4ee2-bac6-8e05f1ed7d99&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(bandura, 1986)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(bandura, 1986)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(bandura, 1986)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(bandura, 1986)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>. <o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>definitions of il vary widely and continue to evolve. the association of college and research libraries’ <i style='mso-bidi-font-style:normal'>framework for information literacy for higher education</i> </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;url&quot; : &quot;http://www.ala.org/acrl/standards/ilframework&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;association of college and research libraries&quot;, &quot;given&quot; : &quot;&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;id&quot; : &quot;item-1&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2016&quot; ] ] }, &quot;title&quot; : &quot;framework for information literacy for higher education&quot;, &quot;type&quot; : &quot;webpage&quot; }, &quot;suppress-author&quot; : 1, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=fdbe2e17-510e-4ea9-9287-bc5510645871&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(2016)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(2016)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(2016)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(2016, para. 5)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'> stresses its more conceptual aspects: “information literacy is the set of integrated abilities encompassing the reflective discovery of information, the understanding of how information is produced and valued, and the use of information in creating new knowledge and participating ethically in communities of learning.” for the purpose of this study, il will refer to a narrower, more traditional, and concrete definition—the ability “to recognize when information is needed and . . . the ability to locate, evaluate, and use effectively the needed information” </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;american library association&quot;, &quot;given&quot; : &quot;&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;id&quot; : &quot;item-1&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;1989&quot; ] ] }, &quot;publisher-place&quot; : &quot;chicago, il&quot;, &quot;title&quot; : &quot;presidential committee on information literacy: final report.&quot;, &quot;type&quot; : &quot;report&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=8fe5f22d-4b67-45a3-b8e4-631f8e26ec85&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(american library association, 1989)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(american library association, 1989)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(american library association, 1989)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(american library association, 1989, para. 3)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>there is a growing body of literature focused on il self-efficacy and its relationship to demonstrated il skills.<span style='color:red'> </span>a systematic review of literature revealed that out of 53 studies, 41 clearly or partially indicated that students overestimated their il skills </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;issn&quot; : &quot;19335954&quot;, &quot;pmid&quot; : &quot;1865189998&quot;, &quot;abstract&quot; : &quot;this systematic review has analyzed 53 english language studies that assessed and compared peoples' self-reported and demonstrated information literacy (il) skills. the objective was to collect empirical evidence on the existence of dunning-kruger effect in the area of information literacy. the findings clearly show that this theory works in this area. it is concluded that there is no calibration in peoples' perceived and actual il skills; in most cases low-performers overestimate their skills in self-assessments. the findings have theoretical and practical implications for librarians and il educators. [abstract from author]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;mahmood&quot;, &quot;given&quot; : &quot;khalid&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;communications in information literacy&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;2&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2016&quot;, &quot;9&quot; ] ] }, &quot;note&quot; : &quot;from duplicate 1 (do people overestimate their information literacy skills? a systematic review of empirical evidence on the dunning-kruger effect. mahmood, khalid)\n\naccession number: 120468655; authors:mahmood, khalid 1; affiliations: 1: university of the punjab; subject: information literacy; subject: information science; subject: dunning-kruger effect; subject: self-evaluation; subject: librarians; author-supplied keyword: assessment; author-supplied keyword: dunning-kruger effect; author-supplied keyword: information literacy; number of pages: 22p; record type: article&quot;, &quot;page&quot; : &quot;199-213&quot;, &quot;title&quot; : &quot;do people overestimate their information literacy skills? a systematic review of empirical evidence on the dunning-kruger effect&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;10&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=633ac203-b4fa-46c8-830e-1d4d71284c9a&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(mahmood, 2016)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(mahmood, 2016)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(mahmood, 2016)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(mahmood, 2016)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>. the review included studies dealing with high school, undergraduate, graduate, and professional students, but only 4 of the 53 studies focused solely on graduate students </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;issn&quot; : &quot;19335954&quot;, &quot;pmid&quot; : &quot;1865189998&quot;, &quot;abstract&quot; : &quot;this systematic review has analyzed 53 english language studies that assessed and compared peoples' self-reported and demonstrated information literacy (il) skills. the objective was to collect empirical evidence on the existence of dunning-kruger effect in the area of information literacy. the findings clearly show that this theory works in this area. it is concluded that there is no calibration in peoples' perceived and actual il skills; in most cases low-performers overestimate their skills in self-assessments. the findings have theoretical and practical implications for librarians and il educators. [abstract from author]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;mahmood&quot;, &quot;given&quot; : &quot;khalid&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;communications in information literacy&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;2&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2016&quot;, &quot;9&quot; ] ] }, &quot;note&quot; : &quot;from duplicate 1 (do people overestimate their information literacy skills? a systematic review of empirical evidence on the dunning-kruger effect. mahmood, khalid)\n\naccession number: 120468655; authors:mahmood, khalid 1; affiliations: 1: university of the punjab; subject: information literacy; subject: information science; subject: dunning-kruger effect; subject: self-evaluation; subject: librarians; author-supplied keyword: assessment; author-supplied keyword: dunning-kruger effect; author-supplied keyword: information literacy; number of pages: 22p; record type: article&quot;, &quot;page&quot; : &quot;199-213&quot;, &quot;title&quot; : &quot;do people overestimate their information literacy skills? a systematic review of empirical evidence on the dunning-kruger effect&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;10&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=633ac203-b4fa-46c8-830e-1d4d71284c9a&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(mahmood, 2016)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(mahmood, 2016)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(mahmood, 2016)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(mahmood, 2016)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>. mahmood </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;issn&quot; : &quot;19335954&quot;, &quot;pmid&quot; : &quot;1865189998&quot;, &quot;abstract&quot; : &quot;this systematic review has analyzed 53 english language studies that assessed and compared peoples' self-reported and demonstrated information literacy (il) skills. the objective was to collect empirical evidence on the existence of dunning-kruger effect in the area of information literacy. the findings clearly show that this theory works in this area. it is concluded that there is no calibration in peoples' perceived and actual il skills; in most cases low-performers overestimate their skills in self-assessments. the findings have theoretical and practical implications for librarians and il educators. [abstract from author]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;mahmood&quot;, &quot;given&quot; : &quot;khalid&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;communications in information literacy&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;2&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2016&quot;, &quot;9&quot; ] ] }, &quot;note&quot; : &quot;from duplicate 1 (do people overestimate their information literacy skills? a systematic review of empirical evidence on the dunning-kruger effect. mahmood, khalid)\n\naccession number: 120468655; authors:mahmood, khalid 1; affiliations: 1: university of the punjab; subject: information literacy; subject: information science; subject: dunning-kruger effect; subject: self-evaluation; subject: librarians; author-supplied keyword: assessment; author-supplied keyword: dunning-kruger effect; author-supplied keyword: information literacy; number of pages: 22p; record type: article&quot;, &quot;page&quot; : &quot;199-213&quot;, &quot;title&quot; : &quot;do people overestimate their information literacy skills? a systematic review of empirical evidence on the dunning-kruger effect&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;10&quot; }, &quot;suppress-author&quot; : 1, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=633ac203-b4fa-46c8-830e-1d4d71284c9a&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(2016)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(2016)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(2016)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(2016)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'> found that 83% of the studies focusing on undergraduates indicated that undergraduate students frequently overestimate their il skills. the results of studies dealing with graduate students were less conclusive. boucher, davies, glen, dalziel, and chandler </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;boucher&quot;, &quot;given&quot; : &quot;clare&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;dalziel&quot;, &quot;given&quot; : &quot;katrina&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;davies&quot;, &quot;given&quot; : &quot;michele&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;glen&quot;, &quot;given&quot; : &quot;susan&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;chandler&quot;, &quot;given&quot; : &quot;jed&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;id&quot; : &quot;item-1&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2009&quot; ] ] }, &quot;publisher-place&quot; : &quot;cardiff, uk&quot;, &quot;title&quot; : &quot;are postgraduates ready for research? poster presented at the librarians information literacy annual conference (lilac)&quot;, &quot;type&quot; : &quot;article&quot; }, &quot;suppress-author&quot; : 1, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=815b527b-3c1e-47d8-86c8-c27ae26719cc&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(2009)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(2009)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(2009)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(2009)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'> noted that graduate students both underand overestimated their skill levels, and those who rated their skills highest often had the lowest performance scores. likewise, jackson </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.11645/7.2.1848&quot;, &quot;issn&quot; : &quot;17505968&quot;, &quot;abstract&quot; : &quot;the article discusses the need for confidence in a research student. it states that research students' success depends on their knowledge, behaviour and attributes. researchers are advised to identify their strengths and weaknesses for personal development planning. some self-assessment tools for testing the skills of research students are presented.&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;jackson&quot;, &quot;given&quot; : &quot;cathie&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of information literacy&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;2&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2013&quot;, &quot;12&quot; ] ] }, &quot;note&quot; : &quot;from duplicate 1 (confidence as an indicator of research students' abilities in information literacy: a mismatch. jackson, cathie)\n\naccession number: 93325037; authors:jackson, cathie 1 email address: cjackson@dinkycrew.com; affiliations: 1: senior consultant: information literacy, cardiff university; subject: information literacy; subject: research; subject: students; subject: self-evaluation; subject: information skills; author-supplied keyword: competence; author-supplied keyword: confidence; author-supplied keyword: diagnostic tests; author-supplied keyword: information literacy; author-supplied keyword: phd students; author-supplied keyword: postgraduate research students; author-supplied keyword: researcher development framework; author-supplied keyword: self-assessment; author-supplied keyword: training needs; author-supplied keyword: uk; number of pages: 4p; record type: article\n\nfrom duplicate 2 (confidence as an indicator of research students' abilities in information literacy: a mismatch. jackson, cathie)\n\naccession number: 93325037; jackson, cathie 1; email address: cjackson@dinkycrew.com; affiliations: 1 : senior consultant: information literacy, cardiff university; source info: dec2013, vol. 7 issue 2, p149; thesaurus term: information literacy; thesaurus term: research; thesaurus term: students; subject term: self-evaluation; subject term: information skills; author-supplied keyword: competence; author-supplied keyword: confidence; author-supplied keyword: diagnostic tests; author-supplied keyword: information literacy; author-supplied keyword: phd students; author-supplied keyword: postgraduate research students; author-supplied keyword: researcher development framework; author-supplied keyword: self-assessment; author-supplied keyword: training needs; author-supplied keyword: uk; number of pages: 4p; document type: article&quot;, &quot;page&quot; : &quot;149-152&quot;, &quot;publisher&quot; : &quot;journal of information literacy&quot;, &quot;title&quot; : &quot;confidence as an indicator of research students' abilities in information literacy: a mismatch.&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;7&quot; }, &quot;suppress-author&quot; : 1, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=06ec81e1-d8d7-4a4f-a37c-857c20343214&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(2013)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(2013)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(2013)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(2013)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'> found that although some graduate students accurately predict their skill levels, others overestimated them; in short, there was no clear correlation. other studies indicated a weak positive correlation </span><!--[if supportfields]><span lang=en-ca style='font-size: 10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin; mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.3928/01484834-20150218-03&quot;, &quot;issn&quot; : &quot;01484834&quot;, &quot;pmid&quot; : &quot;25692245&quot;, &quot;abstract&quot; : &quot;maintaining evidence-based nursing practice requires information literacy (il) skills that should be established prior to completing an undergraduate nursing degree. based on bandura's social cognitive theory, this cross-sectional descriptive correlational study assessed the perceived and applied il skills of graduate nursing students from two family nurse practitioner (fnp) programs in the midwestern united states. results showed that although the 26 newly admitted fnp students demonstrated a high level of confidence in their il skills, the students did not perform well in the actual il skills test. according to bandura, the students' confidence in their il knowledge should allow students to be engaged in course activities requiring il skills. nurse educators teaching in undergraduate or graduate programs are in key positions to incorporate il experiences into class activities to allow for skill assessment and further practice. further research is needed on nursing students' il self-efficacy and performance. [j nurs educ. 2015;54(3, suppl.):s26-s30.]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;robertson&quot;, &quot;given&quot; : &quot;d susie&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;felicilda-reynaldo&quot;, &quot;given&quot; : &quot;rhea faye d&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of nursing education&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;3, suppl&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2015&quot; ] ] }, &quot;note&quot; : &quot;accession number: 25692245. language: english. date created: 20150228. date completed: 20151215. update code: 20151215. publication type: journal article. journal id: 7705432. publication model: print-electronic. cited medium: internet. nlm iso abbr: j nurs educ. linking issn: 01484834. subset: im; n; date of electronic publication: 2015 feb 19. current imprints: publication: thorofare, n.j. : charles b. slack; original imprints: publication: new york, n.y. : blackiston division, mcgraw-hill, [c1962-&quot;, &quot;page&quot; : &quot;s26-30&quot;, &quot;publisher&quot; : &quot;charles b. slack&quot;, &quot;publisher-place&quot; : &quot;united states&quot;, &quot;title&quot; : &quot;evaluation of graduate nursing students' information literacy self-efficacy and applied skills&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;54&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=e598323d-e01e-4782-aa5a-317b3a02abb7&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(robertson &amp; felicilda-reynaldo, 2015)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(robertson &amp; felicilda-reynaldo, 2015)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(robertson &amp; felicilda-reynaldo, 2015)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(robertson &amp; felicilda-reynaldo, 2015)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'> or mixed results </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1080/00049670.2004.10721622&quot;, &quot;issn&quot; : &quot;0004-9670&quot;, &quot;abstract&quot; : &quot;this article looks at the information literacy (ils) skills of graduate students and is based on an audit at the anu in 2003 which included database searching, web searching, information management and word processing skills--vital to all graduate students. each student who completed the audit received a \&quot;training needs profile\&quot;. staff also benefited as they had a better understanding of the skills of the students that they were providing training for. (contains 9 figures and 2 endnotes.)&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;perrett&quot;, &quot;given&quot; : &quot;valerie&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;australian library journal&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;2&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2004&quot;, &quot;5&quot;, &quot;1&quot; ] ] }, &quot;note&quot; : &quot;accession number: ej997487; acquisition information: australian library and information association. p.o. box 6335, kingston 2604, australia. tel: +61-2-6215-8222; fax: +61-2-6282-2249; e-mail: enquiry@alia.org.au; web site: http://www.alia.org.au; journal code: feb2014; level of availability: not available from eric; publication type: journal articles; publication type: reports descriptive; entry date: 2013&quot;, &quot;page&quot; : &quot;161-171&quot;, &quot;publisher&quot; : &quot;australian library journal&quot;, &quot;title&quot; : &quot;graduate information literacy skills: the 2003 anu skills audit&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;53&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=d6df846e-338c-4993-a121-6edbdb20ba38&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(perrett, 2004)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(perrett, 2004)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(perrett, 2004)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(perrett, 2004)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>. mahmood’s </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;issn&quot; : &quot;19335954&quot;, &quot;pmid&quot; : &quot;1865189998&quot;, &quot;abstract&quot; : &quot;this systematic review has analyzed 53 english language studies that assessed and compared peoples' self-reported and demonstrated information literacy (il) skills. the objective was to collect empirical evidence on the existence of dunning-kruger effect in the area of information literacy. the findings clearly show that this theory works in this area. it is concluded that there is no calibration in peoples' perceived and actual il skills; in most cases low-performers overestimate their skills in self-assessments. the findings have theoretical and practical implications for librarians and il educators. [abstract from author]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;mahmood&quot;, &quot;given&quot; : &quot;khalid&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;communications in information literacy&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;2&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2016&quot;, &quot;9&quot; ] ] }, &quot;note&quot; : &quot;from duplicate 1 (do people overestimate their information literacy skills? a systematic review of empirical evidence on the dunning-kruger effect. mahmood, khalid)\n\naccession number: 120468655; authors:mahmood, khalid 1; affiliations: 1: university of the punjab; subject: information literacy; subject: information science; subject: dunning-kruger effect; subject: self-evaluation; subject: librarians; author-supplied keyword: assessment; author-supplied keyword: dunning-kruger effect; author-supplied keyword: information literacy; number of pages: 22p; record type: article&quot;, &quot;page&quot; : &quot;199-213&quot;, &quot;title&quot; : &quot;do people overestimate their information literacy skills? a systematic review of empirical evidence on the dunning-kruger effect&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;10&quot; }, &quot;suppress-author&quot; : 1, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=633ac203-b4fa-46c8-830e-1d4d71284c9a&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(2016)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(2016)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(2016)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(2016)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'> review covered the years from 1986 to 2015, but in two recent articles, international graduate students in business were found to overestimate their il skills </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1080/08963568.2016.1145787&quot;, &quot;issn&quot; : &quot;08963568&quot;, &quot;abstract&quot; : &quot;in 2015, the information literacy (il) team designed an instrument to determine international graduate business students' perceptions of their il skills and their actual test-assessed il skills. the purpose of this research was to compare international graduate students' perceptions of their il skills versus their test-assessed competencies, with the aim of creating training modules to improve test-identified deficiencies. results demonstrate that wide discrepancies exist between students' perceptions of their own skills and their actual skills on all six ila-topic areas assessed by the internally developed test instruments, with students overestimating their abilities in all areas. [abstract from publisher]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;michalak&quot;, &quot;given&quot; : &quot;russell&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;rysavy&quot;, &quot;given&quot; : &quot;monica&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of business &amp; finance librarianship&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;2&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2016&quot;, &quot;4&quot; ] ] }, &quot;note&quot; : &quot;accession number: 116263533; michalak, russell 1; rysavy, monica d. t. 1; affiliations: 1 : goldey-beacom college, wilmington, delaware, usa; source info: apr-jun2016, vol. 21 issue 2, p152; thesaurus term: information literacy -research; thesaurus term: business information services; subject term: instructional systems design; subject term: international business enterprises; subject term: graduate students; author-supplied keyword: business information literacy; author-supplied keyword: information literacy assessment; author-supplied keyword: information literacy skills; author-supplied keyword: instructional design; author-supplied keyword: international business graduate students; number of pages: 23p; document type: article&quot;, &quot;page&quot; : &quot;152-174&quot;, &quot;title&quot; : &quot;information literacy in 2015: international graduate business students' perceptions of information literacy skills compared to test-assessed skills.&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;21&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=79e910db-f957-4128-857a-5ce4b0a15542&quot; ] }, { &quot;id&quot; : &quot;item-2&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1016/j.acalib.2017.02.003&quot;, &quot;issn&quot; : &quot;0099-1333&quot;, &quot;abstract&quot; : &quot;in the 2015 summer session i, the information literacy team combined two instruments, the information literacy assessment (ila) and the students' perceptions of their information literacy skills questionnaire (spil-q), into one survey and distributed it to the college's international graduate students through a google form. it was distributed to 932 international graduate students, and 172 valid respondents completed the survey. the purpose of this research was to compare the confidence gap in information literacy skills between men and women, particularly in international graduate students. data collected illustrated that female international business students (n=70) tended to be slightly more confident than their male counterparts (n=102) regarding their perceived information literacy skills as evidenced by their spil-q average score across all six il topics, 3.78, vs. male student's average score of 3.58.&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;michalak&quot;, &quot;given&quot; : &quot;russell&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;rysavy&quot;, &quot;given&quot; : &quot;monica d t&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;wessel&quot;, &quot;given&quot; : &quot;alison&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of academic librarianship&quot;, &quot;id&quot; : &quot;item-2&quot;, &quot;issue&quot; : &quot;2&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2017&quot;, &quot;3&quot;, &quot;1&quot; ] ] }, &quot;note&quot; : &quot;accession number: s0099133316301872; author: michalak, russell (\u204e); author: rysavy, monica d.t.; author: wessel, alison; affiliation: goldey-beacom college, united states; number of pages: 5; language: english;&quot;, &quot;page&quot; : &quot;100-104&quot;, &quot;publisher&quot; : &quot;elsevier inc.&quot;, &quot;title&quot; : &quot;students' perceptions of their information literacy skills: the confidence gap between male and female international graduate students&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;43&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=a1fb1f73-5d6a-4986-a041-055a9e6a27fc&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(michalak &amp; rysavy, 2016; michalak, rysavy, &amp; wessel, 2017)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(michalak &amp; rysavy, 2016; michalak, rysavy, &amp; wessel, 2017)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(michalak &amp; rysavy, 2016; michalak, rysavy, &amp; wessel, 2017)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(michalak &amp; rysavy, 2016; michalak, rysavy, &amp; wessel, 2017)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>these contradictory findings echo the conflicting research about graduate student il as a whole. some librarians believe that graduate students’ need for il instruction exceeds that of undergraduates because of the intensive research required by many graduate programs </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;crosetto&quot;, &quot;given&quot; : &quot;alice&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;wilkenfeld&quot;, &quot;given&quot; : &quot;polly&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;runnestrand&quot;, &quot;given&quot; : &quot;diane&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;chapter-number&quot; : &quot;3&quot;, &quot;container-title&quot; : &quot;information literacy collaborations that work&quot;, &quot;editor&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;jacobson&quot;, &quot;given&quot; : &quot;trudi&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;mackey&quot;, &quot;given&quot; : &quot;thomas&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;id&quot; : &quot;item-1&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2007&quot; ] ] }, &quot;page&quot; : &quot;41-56&quot;, &quot;publisher&quot; : &quot;neal-shuman publishers&quot;, &quot;publisher-place&quot; : &quot;new york, ny&quot;, &quot;title&quot; : &quot;responding to the needs of our graduate students: a pilot information literacy course in graduate education&quot;, &quot;type&quot; : &quot;chapter&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=dba102c4-f190-4e07-ab4d-f5fba937b3e8&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(crosetto, wilkenfeld, &amp; runnestrand, 2007)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(crosetto, wilkenfeld, &amp; runnestrand, 2007)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(crosetto, wilkenfeld, &amp; runnestrand, 2007)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(crosetto, wilkenfeld, &amp; runnestrand, 2007)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>. catalano’s </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.18438/b8v62b&quot;, &quot;issn&quot; : &quot;1715720x&quot;, &quot;abstract&quot; : &quot;objective this study investigates the information literacy of graduate education students, including those in doctoral cohorts. the association for research and college libraries information literacy standards were used a baseline for measurement. methods a survey was sent to all graduate students in the school of education; it asked a combination of questions measuring students' perceptions of their information literacy skills and testing their knowledge of information literacy. results a total of 172 surveys were returned. the results indicated that while there is a heavy reliance on internet sources, many students were able to determine which sources were reliable and which were not. after attending information instruction sessions, students were more familiar with library services and more inclined to use them. conclusion it was determined that a one credit course or multiple sessions of library instruction would better serve graduate students completing capstone projects. [abstract from author]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;catalano&quot;, &quot;given&quot; : &quot;amy&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;evidence based library &amp; information practice&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;4&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2010&quot; ] ] }, &quot;page&quot; : &quot;21-38&quot;, &quot;title&quot; : &quot;using acrl standards to assess the information literacy of graduate students in an education program.&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;5&quot; }, &quot;suppress-author&quot; : 1, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=5ec351db-fe2d-45f9-a626-ec4191abcacc&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(2010)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(2010)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(2010)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(2010)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'> research indicates that graduate students are generally able to evaluate information but lack advanced search skills; other research indicates that graduate students actually have sophisticated il skills </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin; mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1016/j.acalib.2010.05.005&quot;, &quot;issn&quot; : &quot;0099-1333&quot;, &quot;abstract&quot; : &quot;this paper explores the notion of information illiteracy in relation to doctoral students' information literacy activities. findings from a qualitative study of the doctoral literature review process portray learners as competent, rather than information illiterate, even though they may not have received information literacy interventions.&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;green&quot;, &quot;given&quot; : &quot;rosemary&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of academic librarianship&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;4&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2010&quot; ] ] }, &quot;note&quot; : &quot;accession number: s009913331000100x; author: green, rosemary; affiliation: shenandoah university, winchester va; number of pages: 7; language: english;&quot;, &quot;page&quot; : &quot;313-319&quot;, &quot;title&quot; : &quot;information illiteracy: examining our assumptions&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;36&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=948e525d-0322-4d9c-9fdd-d9b5677abcd6&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(green, 2010)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(green, 2010)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(green, 2010)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(green, 2010)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>. a study asking graduate students to rate their feelings of engagement, affirmation, and puzzlement during an information literacy instruction session revealed conflicting responses, pointing to a wide range of abilities and competencies among graduate students </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;http://10.1016/j.acalib.2016.08.008&quot;, &quot;issn&quot; : &quot;00991333&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;saunders&quot;, &quot;given&quot; : &quot;laura&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;severyn&quot;, &quot;given&quot; : &quot;jenny&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;freundlich&quot;, &quot;given&quot; : &quot;shanti&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;piroli&quot;, &quot;given&quot; : &quot;vivienne&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;shaw-munderback&quot;, &quot;given&quot; : &quot;jeremy&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of academic librarianship&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;6&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2016&quot;, &quot;11&quot; ] ] }, &quot;note&quot; : &quot;accession number: 119652422; saunders, laura 1; email address: laura.saunders@simmons.edu severyn, jenny 1; email address: jennifer.severyn@simmons.edu freundlich, shanti 2; email address: shanti.freundlich@mcphs.edu piroli, vivienne 3; email address: vivienne.piroli@simmons.edu shaw-munderback, jeremy 3; email address: jeremy.shaw-munderback@simmons.edu; affiliation: 1: simmons college school of library and information science, 300 the fenway, boston, ma 02115, united states 2: massachusetts college of pharmacy libraries 179 longwood ave, boston, ma 02115, united states 3: simmons college beatley library, 300 the fenway, boston, ma 02115, united states; source info: nov2016, vol. 42 issue 6, p655; subject term: information literacy; subject term: instructional systems; subject term: academic librarianship; subject term: information science; subject term: library science; number of pages: 9p; document type: article\n\nsa lit review&quot;, &quot;page&quot; : &quot;655-663&quot;, &quot;title&quot; : &quot;assessing graduate level information literacy instruction with critical incident questionnaires&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;42&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=8ce5779e-c7e4-4868-8e14-984f6cc42d3b&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(saunders, severyn, freundlich, piroli, &amp; shaw-munderback, 2016)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(saunders, severyn, freundlich, piroli, &amp; shaw-munderback, 2016)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(saunders, severyn, freundlich, piroli, &amp; shaw-munderback, 2016)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(saunders, severyn, freundlich, piroli, &amp; shaw-munderback, 2016)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>. even when graduate students are aware that they need research help, they are hesitant to approach librarians </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin; mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1108/00220410710723911&quot;, &quot;isbn&quot; : &quot;10.1108/00220410710723911&quot;, &quot;issn&quot; : &quot;0022-0418&quot;, &quot;abstract&quot; : &quot;purpose \u2013 this study seeks to apply ecological psychology\u2019s concept of \u201caffordance\u201d to graduate students\u2019 information behavior in the academic library, and to explore the extent to which the affordances experienced by graduate students differed from the affordances librarians were attempting to provide. design/methodology/approach \u2013 in-depth, qualitative interviews with graduate students and academic librarians explored how the students perceived and used the library\u2019s various \u201copportunities for action\u201d (e.g. books, databases, instructional sessions, librarians, physical space, etc.) and compared these perceptions and behavior with librarians\u2019 intentions and expectations. findings \u2013 findings indicate a disparity between expectations and experience and point to graduate students as an underserved population in this context, especially in terms of the library\u2019s outreach efforts. in addition, because graduate students are increasingly teaching introductory undergraduate courses, communication methods that bypass graduate students tend to miss undergraduate students as well. practical implications \u2013 practical implications discussed in this paper include possible methods of improving communication channels between graduate students and academic librarians, and considerations for information literacy instruction. originality/value \u2013 this paper presents a unique perspective by using affordance theory to frame students and librarians\u2019 expectations about library services. the findings are particularly valuable for their implications for library-patron communication and information literacy. keywords ecology, graduates, academic libraries, librarians, information media, qualitative research paper type research paper&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;sadler&quot;, &quot;given&quot; : &quot;elizabeth (bess)&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;given&quot;, &quot;given&quot; : &quot;lisa m.&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of documentation&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;1&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2007&quot; ] ] }, &quot;note&quot; : &quot;item citation: journal of documentation, 2007, vol. 63, issue 1, pp. 115-141.\n\naccession number: edsemr.10.1108.00220410710723911; publication type: academic journal; source: journal of documentation; language: english; publication date: 20070123; rights: \u00a9 emerald group publishing limited 2007; imprint: emerald group publishing limited, january 23, 2007.&quot;, &quot;page&quot; : &quot;115-141&quot;, &quot;publisher&quot; : &quot;emerald group publishing limited&quot;, &quot;title&quot; : &quot;affordance theory: a framework for graduate students' information behavior&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;63&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=449185d8-f052-4546-b972-cc706f845fa3&quot; ] }, { &quot;id&quot; : &quot;item-2&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1080/01639260903272778&quot;, &quot;issn&quot; : &quot;0163-9269&quot;, &quot;abstract&quot; : &quot;this article investigates how psychology graduate students find information for coursework and research, who teaches them how to find it, and whether differences emerge over the course of their graduate careers. findings indicate that these graduate students are comfortable using campus libraries, prefer electronic resources, ask supervisors when they need assistance locating information, and have some interest in furthering their information literacy knowledge. finally, the master's students use different information management skills than do the phd students, as evidenced by the use of bibliographic management software. suggestions for furthering the role of research librarians focus on being more proactive, rather than expecting students to come to librarians for assistance. (contains 1 table and 1 note.)&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;harrington&quot;, &quot;given&quot; : &quot;marni r&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;behavioral &amp; social sciences librarian&quot;, &quot;id&quot; : &quot;item-2&quot;, &quot;issue&quot; : &quot;4&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2009&quot;, &quot;1&quot;, &quot;1&quot; ] ] }, &quot;note&quot; : &quot;accession number: ej870453; acquisition information: routledge. available from: taylor &amp;amp; francis, ltd. 325 chestnut street suite 800, philadelphia, pa 19106. tel: 800-354-1420; fax: 215-625-2940; web site: http://www.tandf.co.uk/journals; education level: higher education; reference count: 17; journal code: jan2017; level of availability: not available from eric; publication type: journal articles; publication type: reports evaluative; entry date: 2010&quot;, &quot;page&quot; : &quot;179-201&quot;, &quot;publisher&quot; : &quot;behavioral &amp; social sciences librarian&quot;, &quot;title&quot; : &quot;information literacy and research-intensive graduate students: enhancing the role of research librarians&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;28&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=21f0ef77-ccf0-4863-a50c-eeda13f07ca9&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(harrington, 2009; sadler &amp; given, 2007)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(harrington, 2009; sadler &amp; given, 2007)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(harrington, 2009; sadler &amp; given, 2007)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(harrington, 2009; sadler &amp; given, 2007)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>few articles address lis graduate students’ il self-efficacy or il skills, but there is a suggestion that lis graduate students have high il self-efficacy but lower than expected performance. several studies point to lis students having positive il self-efficacy </span><!--[if supportfields]><span lang=en-ca style='font-size: 10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin; mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1108/00220410310506295&quot;, &quot;issn&quot; : &quot;0022-0418&quot;, &quot;abstract&quot; : &quot;this paper is primarily concerned with self?efficacy in the context of information literacy. the focus is first on the concept of self?efficacy, followed by attainment of self?efficacy beliefs. finally, findings of the research, the aim of which was to explore students? (who enrolled in the department of information management, hacettepe university, ankara, turkey) perceived self?efficacy for information and computer literacy, are scrutinized. results of the research indicate no significant year?to?year changes, although the students have a positive perceived self?efficacy for information literacy. students? self?efficacy beliefs regarding information literacy and computers are correlated.&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;kurbanoglu&quot;, &quot;given&quot; : &quot;serap&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of documentation&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;6&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2003&quot;, &quot;12&quot;, &quot;1&quot; ] ] }, &quot;note&quot; : &quot;doi: 10.1108/00220410310506295&quot;, &quot;page&quot; : &quot;635-646&quot;, &quot;publisher&quot; : &quot;emerald&quot;, &quot;title&quot; : &quot;self-efficacy: a concept closely linked to information literacy and lifelong learning&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;59&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=2c7c4aba-912a-42b2-bd6a-ad84b1efa489&quot; ] }, { &quot;id&quot; : &quot;item-2&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.12783/issn.2328-2967/56/s1/9&quot;, &quot;issn&quot; : &quot;07485786&quot;, &quot;abstract&quot; : &quot;librarians are expected to be expert searchers, and developing information literacy skills to navigate the vast world of information is a focus of most library and information science (lis) programs. it is important to understand the information literacy and behaviors of lis students to see if they are employing the skills they will need to assist and educate their future patrons, yet there is relatively little research into the information literacy and behaviors of librarians and library students. using a survey on information behaviors, this study addresses this gap in the literature by examining how lis students in 18 countries search for, evaluate, and use information in various contexts, and on whom they rely for help. the results of this study demonstrate library science students' information literacy skills within an international context. faculty in lis programs will be interested to see how their students approach information problems, and might use this understanding to develop or strengthen courses and inform curricular decisions. [abstract from author]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;saunders&quot;, &quot;given&quot; : &quot;laura&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;kurbanoglu&quot;, &quot;given&quot; : &quot;serap&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;boustany&quot;, &quot;given&quot; : &quot;joumana&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;dogan&quot;, &quot;given&quot; : &quot;guleda&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;becker&quot;, &quot;given&quot; : &quot;peter&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;blumer&quot;, &quot;given&quot; : &quot;eliane&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;chowdhury&quot;, &quot;given&quot; : &quot;sudatta&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;dobreva&quot;, &quot;given&quot; : &quot;milena&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;gendina&quot;, &quot;given&quot; : &quot;natalia&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;grgic&quot;, &quot;given&quot; : &quot;ivana hebrang&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;haddow&quot;, &quot;given&quot; : &quot;gaby&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;koltay&quot;, &quot;given&quot; : &quot;tibor&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;kortelainen&quot;, &quot;given&quot; : &quot;terttu&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;krakowska&quot;, &quot;given&quot; : &quot;monika&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;majid&quot;, &quot;given&quot; : &quot;shaheen&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;mezhova&quot;, &quot;given&quot; : &quot;marina&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;repanovici&quot;, &quot;given&quot; : &quot;angela&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;rud\u017eioniene&quot;, &quot;given&quot; : &quot;jurgita&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;schneider&quot;, &quot;given&quot; : &quot;rene&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;terra&quot;, &quot;given&quot; : &quot;ana lucia&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of education for library &amp; information science&quot;, &quot;id&quot; : &quot;item-2&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2015&quot;, &quot;3&quot;, &quot;2&quot; ] ] }, &quot;note&quot; : &quot;accession number: 108403317; saunders, laura 1; email address: laura.saunders@simmons.edu; kurbanoglu, serap 2; email address: kurbanogluserap@gmail.com; boustany, joumana 3; email address: jboustany@gmail.com; dogan, guleda 2; email address: guledaduzyol@gmail.com; becker, peter 4; email address: p.g.becker@hhs.nl; blumer, eliane 5; email address: eliane.blumer@hesge.ch; chowdhury, sudatta 6; email address: sudatta@hotmail.co.uk; dobreva, milena 7; email address: milena.dobreva@gmail.com; gendina, natalia 8; email address: gendina@inbox.ru; grgic, ivana hebrang 9; email address: ihgrgic@ffzg.hr; haddow, gaby 10; email address: g.haddow@curtin.edu.au; koltay, tibor 11; email address: koltay.tibor@abk.szie.hu; kortelainen, terttu 12; email address: terttu.kortelainen@oulu.fi; krakowska, monika 13; email address: monika.krakowska@uj.edu.pl; majid, shaheen 14; mezhova, marina 15; email address: mezhova75@mail.ru; repanovici, angela 16; email address: arepanovici@gmail.com; rud\u017eioniene, jurgita 17; email address: jurgita.rudzioniene@kf.vu.lt; schneider, rene 5; email address: rene.schneider@hesge.ch; terra, ana lucia 18; email address: analuciaterra@yahoo.com; affiliations: 1 : school of library and information science, simmons college; 2 : department of information management, hacettepe university; 3 : department of information communication, universit\u00e9 paris descartes; 4 : hague university of applied sciences, netherlands; 5 : department of information science, haute ecole de gestion; 6 : department of computer &amp;amp; information sciences, university of strathclyde; 7 : library information and archive sciences department, malta university; 8 : scientific and research institute of information technologies, kemerovo state university of culture and arts kemerrovo; 9 : department of information and communication sciences, faculty of humanities and social sciences, university of zagreb; 10 : department of information studies, curtin university; 11 : department of information and library studies, szent istv\u00e1n university,; 12 : department of information studies, oulu university; 13 : department of management and social communication institute of library and information science, jagiellonian university; 14 : division of information studies, wee kim wee school of communication &amp;amp; information, nanyang technological university; 15 : scientific and research institute of information technologies in kemerovo state university of culture and arts kemerrovo; 16 : transilvania university of brasov; 17 : faculty of communication institute of library and information science, vilnius university; 18 : information science department, superior school of industrial studies and management, polytechnic institute of porto; source info: 2015 supplement 1, vol. 56, ps80; thesaurus term: library education; thesaurus term: research; thesaurus term: information literacy -research; thesaurus term: library school students; thesaurus term: library schools -curricula; subject term: academic programs; number of pages: 20p; illustrations: 3 charts, 5 graphs; document type: article; full text word count: 7907&quot;, &quot;page&quot; : &quot;s80-s99&quot;, &quot;publisher&quot; : &quot;association for library &amp; information science education&quot;, &quot;title&quot; : &quot;information behaviors and information literacy skills of lis students: an international perspective&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;56&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=1ea827af-d9ca-44f8-8664-0a731c11718c&quot; ] }, { &quot;id&quot; : &quot;item-3&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1016/j.acalib.2012.08.006&quot;, &quot;isbn&quot; : &quot;00991333&quot;, &quot;abstract&quot; : &quot;the study was carried out with students of official doctoral programs of information science in four universities in spain and latin america with the purpose of finding out, through self-assessments, student perceptions of their own information competence. a survey was designed to determine self-perceptions of knowledge, skills and attitudes regarding information competence, and it was distributed to students of two universities in spain, one in cuba and one in mexico. student perceptions of the levels of their own information competence in terms of knowledge, skills and attitudes are, in general, high. nevertheless, despite being immersed in research processes that require these information competencies, they recognize that they are lacking in certain skills related tothe organization, evaluation and presentation of information. having a self-assessment that identifies those areas of information competence in which students consider themselves to be weak will be a significant help in planning activities designed to strengthen these areas in doctoral programs.&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;pinto&quot;, &quot;given&quot; : &quot;maria&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;fernandez-ramos&quot;, &quot;given&quot; : &quot;a&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;sanchez&quot;, &quot;given&quot; : &quot;g&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;meneses&quot;, &quot;given&quot; : &quot;g&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of academic librarianship&quot;, &quot;id&quot; : &quot;item-3&quot;, &quot;issue&quot; : &quot;2&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2013&quot; ] ] }, &quot;page&quot; : &quot;144-154&quot;, &quot;title&quot; : &quot;information competence of doctoral students in information science in spain and latin america: a self-assessment&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;39&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=011b0714-3e8a-4c3c-a334-04ee76e35bae&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(kurbanoglu, 2003; pinto, fernandez-ramos, sanchez, &amp; meneses, 2013; saunders et al., 2015)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(kurbanoglu, 2003; pinto, fernandez-ramos, sanchez, &amp; meneses, 2013; saunders et al., 2015)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(kurbanoglu, 2003; pinto, fernandez-ramos, sanchez, &amp; meneses, 2013; saunders et al., 2015)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(kurbanoglu, 2003; pinto, fernandez-ramos, sanchez, &amp; meneses, 2013; saunders et al., 2015)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>. kurbanoglu’s </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1108/00220410310506295&quot;, &quot;issn&quot; : &quot;0022-0418&quot;, &quot;abstract&quot; : &quot;this paper is primarily concerned with self?efficacy in the context of information literacy. the focus is first on the concept of self?efficacy, followed by attainment of self?efficacy beliefs. finally, findings of the research, the aim of which was to explore students? (who enrolled in the department of information management, hacettepe university, ankara, turkey) perceived self?efficacy for information and computer literacy, are scrutinized. results of the research indicate no significant year?to?year changes, although the students have a positive perceived self?efficacy for information literacy. students? self?efficacy beliefs regarding information literacy and computers are correlated.&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;kurbanoglu&quot;, &quot;given&quot; : &quot;serap&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of documentation&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;6&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2003&quot;, &quot;12&quot;, &quot;1&quot; ] ] }, &quot;note&quot; : &quot;doi: 10.1108/00220410310506295&quot;, &quot;page&quot; : &quot;635-646&quot;, &quot;publisher&quot; : &quot;emerald&quot;, &quot;title&quot; : &quot;self-efficacy: a concept closely linked to information literacy and lifelong learning&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;59&quot; }, &quot;suppress-author&quot; : 1, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=2c7c4aba-912a-42b2-bd6a-ad84b1efa489&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(2003)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(2003)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(2003)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(2003)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'> study of undergraduates enrolled in an information management program (hacettepe university, ankara, turkey) suggests that although there is a slight increase in il self-efficacy between students’ first and second years, there is little gain in succeeding years. although kurbanoglu’s </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin; mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1108/00220410310506295&quot;, &quot;issn&quot; : &quot;0022-0418&quot;, &quot;abstract&quot; : &quot;this paper is primarily concerned with self?efficacy in the context of information literacy. the focus is first on the concept of self?efficacy, followed by attainment of self?efficacy beliefs. finally, findings of the research, the aim of which was to explore students? (who enrolled in the department of information management, hacettepe university, ankara, turkey) perceived self?efficacy for information and computer literacy, are scrutinized. results of the research indicate no significant year?to?year changes, although the students have a positive perceived self?efficacy for information literacy. students? self?efficacy beliefs regarding information literacy and computers are correlated.&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;kurbanoglu&quot;, &quot;given&quot; : &quot;serap&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of documentation&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;6&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2003&quot;, &quot;12&quot;, &quot;1&quot; ] ] }, &quot;note&quot; : &quot;doi: 10.1108/00220410310506295&quot;, &quot;page&quot; : &quot;635-646&quot;, &quot;publisher&quot; : &quot;emerald&quot;, &quot;title&quot; : &quot;self-efficacy: a concept closely linked to information literacy and lifelong learning&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;59&quot; }, &quot;suppress-author&quot; : 1, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=2c7c4aba-912a-42b2-bd6a-ad84b1efa489&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(2003)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(2003)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(2003)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(2003)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'> study is valuable, it is not longitudinal—different students were tested over the course of four years. the differences in self-efficacy for each program year could be due to the group of students tested instead of an actual increase in self-efficacy. there is evidence that lis graduate students have limited il skills, including difficulty formulating boolean search queries </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1177/0340035210388243&quot;, &quot;isbn&quot; : &quot;8802966192&quot;, &quot;issn&quot; : &quot;03400352&quot;, &quot;abstract&quot; : &quot;the aim of this study is to assess the information literacy competency of information science and library management (islm) graduate students at the university of dhaka. bangladesh, and to determine their strengths and weaknesses. in general it was found that students had limited skills in the area of information literacy, as it is not discussed extensively in their academic course curriculum this study urges the incorporation of an information literacy programme in the course curriculum, and more writing discussion and other relevant issues that will make the students more information literate. reprinted by permission of the publisher.&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;islam&quot;, &quot;given&quot; : &quot;mohammed anwarul&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;tsuji&quot;, &quot;given&quot; : &quot;keita&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;ifla journal&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;4&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2010&quot; ] ] }, &quot;note&quot; : &quot;accession number: 503005925; authors:islam, mohammed anwarul;\u00a0tsuji, keita; physical description: bibliography; diagram; graph; table; subject: library orientation; subject: surveys; subject: library orientation for college students; subject: library school students; subject: library schools -india; subject: university of dhaka (dhaka, bangladesh); number of pages: 17p; record type: article&quot;, &quot;page&quot; : &quot;300-316&quot;, &quot;title&quot; : &quot;assessing information literacy competency of information science and library management graduate students of dhaka university&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;36&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=1d405c85-8a89-4b74-91cc-37c185194eac&quot; ] }, { &quot;id&quot; : &quot;item-2&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1080/00048623.2011.10722218&quot;, &quot;isbn&quot; : &quot;00048623&quot;, &quot;issn&quot; : &quot;00048623&quot;, &quot;abstract&quot; : &quot;this paper describes a study comparing the information literacy skills of commencing undergraduate ahttp://eds.a.ebscohost.com/eds/citations/fulltextlinkclick?sid=74f97223-0987-46b4-be5e-2e40a731670d@sessionmgr4005&amp;vid=1&amp;id=pdffulltextnd postgraduate information studies students at curtin university. the study was conducted by means of an online multiple choice survey, which students completed during the first few weeks of their course. the results show that a number of both undergraduates and postgraduates have problems with basic information literacy skills, particularly those related to the use of traditional library tools, such as library catalogues. the study found that the only factors associated with differences in performance were postgraduate and undergraduate status (the primary variable examined), age, and previous work experience. [abstract from author]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;conway&quot;, &quot;given&quot; : &quot;kate&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;australian academic &amp; research libraries&quot;, &quot;id&quot; : &quot;item-2&quot;, &quot;issue&quot; : &quot;2&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2011&quot;, &quot;6&quot; ] ] }, &quot;note&quot; : &quot;from duplicate 1 (how prepared are students for postgraduate study? a comparison of the information literacy skills of commencing undergraduate and postgraduate information studies students at curtin university. conway, kate)\n\naccession number: 72101036; conway, kate 1; email address: kate.conway@health.wa.gov.au; affiliations: 1 : king edward memorial hospital, po box 134, subiaco wa 6904; source info: jun2011, vol. 42 issue 2, p121; thesaurus term: academic libraries -use studies; thesaurus term: information literacy; subject term: undergraduates; subject term: graduate students; number of pages: 15p; document type: article&quot;, &quot;page&quot; : &quot;121-135&quot;, &quot;title&quot; : &quot;how prepared are students for postgraduate study? a comparison of the information literacy skills of commencing undergraduate and postgraduate information studies students at curtin university&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;42&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=84111873-0ef2-43a8-892e-11381032a253&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(conway, 2011; islam &amp; tsuji, 2010)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(conway, 2011; islam &amp; tsuji, 2010)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(conway, 2011; islam &amp; tsuji, 2010)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(conway, 2011; islam &amp; tsuji, 2010)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>. in fact, a study of students entering curtin university’s (perth, australia) information studies graduate program found that “33% of postgraduates were unable to identify a citation as indicating a journal article; 59% were unable to select the best method of searching for a specific journal article; 48% were unaware of how to find a book chapter using a library catalogue; and 33% were unable to identify the boolean operator ‘and’ as a means to narrow a search” </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1080/00048623.2011.10722218&quot;, &quot;isbn&quot; : &quot;00048623&quot;, &quot;issn&quot; : &quot;00048623&quot;, &quot;abstract&quot; : &quot;this paper describes a study comparing the information literacy skills of commencing undergraduate ahttp://eds.a.ebscohost.com/eds/citations/fulltextlinkclick?sid=74f97223-0987-46b4-be5e-2e40a731670d@sessionmgr4005&amp;vid=1&amp;id=pdffulltextnd postgraduate information studies students at curtin university. the study was conducted by means of an online multiple choice survey, which students completed during the first few weeks of their course. the results show that a number of both undergraduates and postgraduates have problems with basic information literacy skills, particularly those related to the use of traditional library tools, such as library catalogues. the study found that the only factors associated with differences in performance were postgraduate and undergraduate status (the primary variable examined), age, and previous work experience. [abstract from author]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;conway&quot;, &quot;given&quot; : &quot;kate&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;australian academic &amp; research libraries&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;2&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2011&quot;, &quot;6&quot; ] ] }, &quot;note&quot; : &quot;from duplicate 1 (how prepared are students for postgraduate study? a comparison of the information literacy skills of commencing undergraduate and postgraduate information studies students at curtin university. conway, kate)\n\naccession number: 72101036; conway, kate 1; email address: kate.conway@health.wa.gov.au; affiliations: 1 : king edward memorial hospital, po box 134, subiaco wa 6904; source info: jun2011, vol. 42 issue 2, p121; thesaurus term: academic libraries -use studies; thesaurus term: information literacy; subject term: undergraduates; subject term: graduate students; number of pages: 15p; document type: article&quot;, &quot;page&quot; : &quot;121-135&quot;, &quot;title&quot; : &quot;how prepared are students for postgraduate study? a comparison of the information literacy skills of commencing undergraduate and postgraduate information studies students at curtin university&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;42&quot; }, &quot;locator&quot; : &quot;130-131&quot;, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=84111873-0ef2-43a8-892e-11381032a253&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(conway, 2011, pp. 130\u2013131)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(conway, 2011, pp. 130\u2013131)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(conway, 2011, pp. 130\u2013131)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(conway, 2011, pp. 130–131)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>. <o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>a study of lis students in 18 countries revealed that although students were confident in their search skills, the students’ self-reported information behaviors and attitudes raised “some concerns as to whether lis students are moving beyond the general population in their location, search, evaluation, and use of resources” </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.12783/issn.2328-2967/56/s1/9&quot;, &quot;issn&quot; : &quot;07485786&quot;, &quot;abstract&quot; : &quot;librarians are expected to be expert searchers, and developing information literacy skills to navigate the vast world of information is a focus of most library and information science (lis) programs. it is important to understand the information literacy and behaviors of lis students to see if they are employing the skills they will need to assist and educate their future patrons, yet there is relatively little research into the information literacy and behaviors of librarians and library students. using a survey on information behaviors, this study addresses this gap in the literature by examining how lis students in 18 countries search for, evaluate, and use information in various contexts, and on whom they rely for help. the results of this study demonstrate library science students' information literacy skills within an international context. faculty in lis programs will be interested to see how their students approach information problems, and might use this understanding to develop or strengthen courses and inform curricular decisions. [abstract from author]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;saunders&quot;, &quot;given&quot; : &quot;laura&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;kurbanoglu&quot;, &quot;given&quot; : &quot;serap&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;boustany&quot;, &quot;given&quot; : &quot;joumana&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;dogan&quot;, &quot;given&quot; : &quot;guleda&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;becker&quot;, &quot;given&quot; : &quot;peter&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;blumer&quot;, &quot;given&quot; : &quot;eliane&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;chowdhury&quot;, &quot;given&quot; : &quot;sudatta&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;dobreva&quot;, &quot;given&quot; : &quot;milena&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;gendina&quot;, &quot;given&quot; : &quot;natalia&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;grgic&quot;, &quot;given&quot; : &quot;ivana hebrang&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;haddow&quot;, &quot;given&quot; : &quot;gaby&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;koltay&quot;, &quot;given&quot; : &quot;tibor&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;kortelainen&quot;, &quot;given&quot; : &quot;terttu&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;krakowska&quot;, &quot;given&quot; : &quot;monika&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;majid&quot;, &quot;given&quot; : &quot;shaheen&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;mezhova&quot;, &quot;given&quot; : &quot;marina&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;repanovici&quot;, &quot;given&quot; : &quot;angela&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;rud\u017eioniene&quot;, &quot;given&quot; : &quot;jurgita&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;schneider&quot;, &quot;given&quot; : &quot;rene&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;terra&quot;, &quot;given&quot; : &quot;ana lucia&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of education for library &amp; information science&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2015&quot;, &quot;3&quot;, &quot;2&quot; ] ] }, &quot;note&quot; : &quot;accession number: 108403317; saunders, laura 1; email address: laura.saunders@simmons.edu; kurbanoglu, serap 2; email address: kurbanogluserap@gmail.com; boustany, joumana 3; email address: jboustany@gmail.com; dogan, guleda 2; email address: guledaduzyol@gmail.com; becker, peter 4; email address: p.g.becker@hhs.nl; blumer, eliane 5; email address: eliane.blumer@hesge.ch; chowdhury, sudatta 6; email address: sudatta@hotmail.co.uk; dobreva, milena 7; email address: milena.dobreva@gmail.com; gendina, natalia 8; email address: gendina@inbox.ru; grgic, ivana hebrang 9; email address: ihgrgic@ffzg.hr; haddow, gaby 10; email address: g.haddow@curtin.edu.au; koltay, tibor 11; email address: koltay.tibor@abk.szie.hu; kortelainen, terttu 12; email address: terttu.kortelainen@oulu.fi; krakowska, monika 13; email address: monika.krakowska@uj.edu.pl; majid, shaheen 14; mezhova, marina 15; email address: mezhova75@mail.ru; repanovici, angela 16; email address: arepanovici@gmail.com; rud\u017eioniene, jurgita 17; email address: jurgita.rudzioniene@kf.vu.lt; schneider, rene 5; email address: rene.schneider@hesge.ch; terra, ana lucia 18; email address: analuciaterra@yahoo.com; affiliations: 1 : school of library and information science, simmons college; 2 : department of information management, hacettepe university; 3 : department of information communication, universit\u00e9 paris descartes; 4 : hague university of applied sciences, netherlands; 5 : department of information science, haute ecole de gestion; 6 : department of computer &amp;amp; information sciences, university of strathclyde; 7 : library information and archive sciences department, malta university; 8 : scientific and research institute of information technologies, kemerovo state university of culture and arts kemerrovo; 9 : department of information and communication sciences, faculty of humanities and social sciences, university of zagreb; 10 : department of information studies, curtin university; 11 : department of information and library studies, szent istv\u00e1n university,; 12 : department of information studies, oulu university; 13 : department of management and social communication institute of library and information science, jagiellonian university; 14 : division of information studies, wee kim wee school of communication &amp;amp; information, nanyang technological university; 15 : scientific and research institute of information technologies in kemerovo state university of culture and arts kemerrovo; 16 : transilvania university of brasov; 17 : faculty of communication institute of library and information science, vilnius university; 18 : information science department, superior school of industrial studies and management, polytechnic institute of porto; source info: 2015 supplement 1, vol. 56, ps80; thesaurus term: library education; thesaurus term: research; thesaurus term: information literacy -research; thesaurus term: library school students; thesaurus term: library schools -curricula; subject term: academic programs; number of pages: 20p; illustrations: 3 charts, 5 graphs; document type: article; full text word count: 7907&quot;, &quot;page&quot; : &quot;s80-s99&quot;, &quot;publisher&quot; : &quot;association for library &amp; information science education&quot;, &quot;title&quot; : &quot;information behaviors and information literacy skills of lis students: an international perspective&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;56&quot; }, &quot;locator&quot; : &quot;s94&quot;, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=1ea827af-d9ca-44f8-8664-0a731c11718c&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(saunders et al., 2015, p. s94)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(saunders et al., 2015, p. s94)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(saunders et al., 2015, p. s94)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(saunders et al., 2015, p. s94)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>. doctoral students enrolled in information science programs in spain, cuba, and mexico generally ranked their il knowledge as high, but the authors of the study commented, “although the results of the self-assessments are encouraging, the authors of this article, as a result of their extensive experience in training doctoral students and directing doctoral dissertations, believe that the real world reality is not, however, as encouraging” </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin; mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1016/j.acalib.2012.08.006&quot;, &quot;isbn&quot; : &quot;00991333&quot;, &quot;abstract&quot; : &quot;the study was carried out with students of official doctoral programs of information science in four universities in spain and latin america with the purpose of finding out, through self-assessments, student perceptions of their own information competence. a survey was designed to determine self-perceptions of knowledge, skills and attitudes regarding information competence, and it was distributed to students of two universities in spain, one in cuba and one in mexico. student perceptions of the levels of their own information competence in terms of knowledge, skills and attitudes are, in general, high. nevertheless, despite being immersed in research processes that require these information competencies, they recognize that they are lacking in certain skills related tothe organization, evaluation and presentation of information. having a self-assessment that identifies those areas of information competence in which students consider themselves to be weak will be a significant help in planning activities designed to strengthen these areas in doctoral programs.&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;pinto&quot;, &quot;given&quot; : &quot;maria&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;fernandez-ramos&quot;, &quot;given&quot; : &quot;a&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;sanchez&quot;, &quot;given&quot; : &quot;g&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;meneses&quot;, &quot;given&quot; : &quot;g&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of academic librarianship&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;2&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2013&quot; ] ] }, &quot;page&quot; : &quot;144-154&quot;, &quot;title&quot; : &quot;information competence of doctoral students in information science in spain and latin america: a self-assessment&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;39&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=011b0714-3e8a-4c3c-a334-04ee76e35bae&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(pinto et al., 2013)&quot;, &quot;manualformatting&quot; : &quot;(pinto, fernandez-ramos, sanchez, &amp; meneses, 2013, p. 151)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(pinto et al., 2013)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(pinto et al., 2013)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(pinto, fernandez-ramos, sanchez, &amp; meneses, 2013, p. 151)</span></span><!--[if supportfields]><span lang=en-ca style='font-size: 10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>. although this discrepancy has been noted, there are no direct measurements to confirm it.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>aims<o:p></o:p></span></b></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>this paper describes an exploratory study to address this gap in knowledge by gathering data to answer the following questions:<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msolistparagraphcxspfirst style='text-indent:-.25in;mso-list:l23 level1 lfo11'><![if !supportlists]><span style='font-size:10.0pt;line-height:115%;font-family:"palatino linotype",serif; mso-fareast-font-family:"palatino linotype";mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>1.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;line-height:115%; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'>what level of information literacy self-efficacy do first-year mlis students have?<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='text-indent:-.25in;mso-list:l23 level1 lfo11'><![if !supportlists]><span style='font-size:10.0pt;line-height:115%;font-family:"palatino linotype",serif; mso-fareast-font-family:"palatino linotype";mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>2.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;line-height:115%; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'>what information literacy skills do first-year mlis students demonstrate?<o:p></o:p></span></p> <p class=msolistparagraphcxsplast style='text-indent:-.25in;mso-list:l23 level1 lfo11'><![if !supportlists]><span style='font-size:10.0pt;line-height:115%;font-family:"palatino linotype",serif; mso-fareast-font-family:"palatino linotype";mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>3.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;line-height:115%; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'>is there a relationship between first-year mlis students’ perceived and demonstrated information literacy skills?<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>methods<o:p></o:p></span></b></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>the study used an online survey to determine first-semester mlis students’ levels of self-efficacy and to test their il skills. each il skill was keyed to a self-efficacy belief, allowing the author to compare discrete beliefs and skills. <span style='color:#212121'>the </span>author submitted an application for exemption to lsu’s institutional review board (irb) to use the survey in fall 2017. the irb chair reviewed the application for this project (lsu irb# e10534) and determined that the project did not require a formal review.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>the author repeated the study in spring 2018 for additional data collection using a different instrument and a streamlined distribution method. the librarian submitted another irb exemption application, which reflected the use of a new instrument, the change in distribution method, and an updated consent script. the irb exemption was granted (lsu irb# e10817).<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style: normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>study population and sampling design<o:p></o:p></span></i></b></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style: normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></i></b></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>the study included students entering the mlis degree program at lsu’s school of library and information science (slis). participants had to be enrolled in their first semester of the mlis degree program and could have no more than 3 graduate-level credits in lis. <o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><i style='mso-bidi-font-style:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>fall 2017<o:p></o:p></span></i></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>in fall 2017, 42 students eligible for the study entered the mlis program at lsu (b. antie, personal communication, sept. 5, 2017); because the study population was small (<i style='mso-bidi-font-style:normal'>n</i> = 42), the author used a census survey instead of a sample survey. the study relied on a voluntary response, but the director of the slis program encouraged students to complete the survey. respondents who completed the survey received an amazon.com egift code worth $5.00 as an incentive after the survey closed; the incentives were funded by lsu libraries.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>qualtrics recorded 61 survey attempts. the author determined that of the 61 responses, 35 were from students who met the inclusion criteria. data from ineligible students were deleted. of the 35 responses from students who met inclusion criteria, 3 respondents to the survey took the survey twice. in these cases, the author deduped the responses using the following criteria: <o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msolistparagraphcxspfirst style='text-indent:-.25in;mso-list:l25 level1 lfo12'><![if !supportlists]><span style='font-size:10.0pt;line-height:115%;font-family:"palatino linotype",serif; mso-fareast-font-family:"palatino linotype";mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>1.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;line-height:115%; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'>retain the attempt that is most complete (fewest skipped questions).<o:p></o:p></span></p> <p class=msolistparagraphcxsplast style='text-indent:-.25in;mso-list:l25 level1 lfo12'><![if !supportlists]><span style='font-size:10.0pt;line-height:115%;font-family:"palatino linotype",serif; mso-fareast-font-family:"palatino linotype";mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>2.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;line-height:115%; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'>if both attempts are complete, keep the first attempt and delete the second.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>this left 32 valid responses—32 out of 42 eligible students responded to the survey for a response rate of 76%.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><i style='mso-bidi-font-style:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>spring 2018<o:p></o:p></span></i></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>the spring 2018 study used the same inclusion and exclusion criterion as the fall 2017 study. on the first day of classes (january 10, 2018), the author obtained a list of the 30 incoming mlis students and their university email addresses from the office of the university registrar (b. antie, personal communication, january 10, 2018). the study once again used a census survey. the study relied on a voluntary response and no incentive was offered for participating in the survey. qualtrics recorded 23 survey attempts; 22 students indicated that they met the inclusion criteria, and 3 attempts were incomplete. after data from the ineligible student and from incomplete surveys were deleted, 19 valid responses were left for a response rate of 65.5%.<o:p></o:p></span></p> <p class=msonormal><i style='mso-bidi-font-style:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></i></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style: normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>study design<o:p></o:p></span></i></b></p> <p class=msonormal><i style='mso-bidi-font-style:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-fareast-font-family: "times new roman";mso-fareast-theme-font:major-fareast;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-bidi-font-weight:bold'><o:p>&nbsp;</o:p></span></i></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>the survey was created in qualtrics, a web-based survey platform. five graduate assistants at lsu libraries took the survey to ensure the survey’s functionality and provide an estimated completion time.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>on the first day of the fall 2017 semester (august 21, 2017), slis’s administrative coordinator of academic services emailed a link to the survey along with a short introduction explaining the purpose of the study. there were three reminders for participation after the initial survey distribution on august 21 with the survey closing on september 6. <o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>the spring 2018 survey was distributed on january 11, 2018, to newly enrolled slis mlis students (<i style='mso-bidi-font-style:normal'>n</i> = 30) to their university email addresses using qualtrics. the author sent email reminders through qualtrics. the survey closed on january 25, 2018. <o:p></o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style: normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></i></b></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style: normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>data collection instruments<o:p></o:p></span></i></b></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>respondent data were collected through qualtrics. a statement containing information required by lsu’s irb prefaced both surveys. <o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><i style='mso-bidi-font-style:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>fall 2017<o:p></o:p></span></i></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>in fall 2017, the instrument consisted of 4 questions to measure il self-efficacy, 18 questions to measure specific il skills, and 5 demographic questions. the author gained permission to use questions 2, 3, 4, and 6 of michalak and rysavy’s </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1080/08963568.2016.1145787&quot;, &quot;issn&quot; : &quot;08963568&quot;, &quot;abstract&quot; : &quot;in 2015, the information literacy (il) team designed an instrument to determine international graduate business students' perceptions of their il skills and their actual test-assessed il skills. the purpose of this research was to compare international graduate students' perceptions of their il skills versus their test-assessed competencies, with the aim of creating training modules to improve test-identified deficiencies. results demonstrate that wide discrepancies exist between students' perceptions of their own skills and their actual skills on all six ila-topic areas assessed by the internally developed test instruments, with students overestimating their abilities in all areas. [abstract from publisher]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;michalak&quot;, &quot;given&quot; : &quot;russell&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;rysavy&quot;, &quot;given&quot; : &quot;monica&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of business &amp; finance librarianship&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;2&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2016&quot;, &quot;4&quot; ] ] }, &quot;note&quot; : &quot;accession number: 116263533; michalak, russell 1; rysavy, monica d. t. 1; affiliations: 1 : goldey-beacom college, wilmington, delaware, usa; source info: apr-jun2016, vol. 21 issue 2, p152; thesaurus term: information literacy -research; thesaurus term: business information services; subject term: instructional systems design; subject term: international business enterprises; subject term: graduate students; author-supplied keyword: business information literacy; author-supplied keyword: information literacy assessment; author-supplied keyword: information literacy skills; author-supplied keyword: instructional design; author-supplied keyword: international business graduate students; number of pages: 23p; document type: article&quot;, &quot;page&quot; : &quot;152-174&quot;, &quot;title&quot; : &quot;information literacy in 2015: international graduate business students' perceptions of information literacy skills compared to test-assessed skills.&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;21&quot; }, &quot;suppress-author&quot; : 1, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=79e910db-f957-4128-857a-5ce4b0a15542&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(2016)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(2016)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(2016)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(2016)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'> students’ perceptions of their information literacy skills questionnaire (spil-q) (m. rysavy, personal communication, june 16, 2017). spil-q measures perceived il self-efficacy with a 5-point likert scale. although there are other well-known and validated il self-efficacy instruments, in particular kurbanoglu, akkoyunlu, and umay’s information literacy self-efficacy scale (ilses) </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1108/00220410610714949&quot;, &quot;isbn&quot; : &quot;00220418&quot;, &quot;issn&quot; : &quot;00220418&quot;, &quot;abstract&quot; : &quot;purpose the main aim of this paper is to describe the development of a scale designed to measure self-efficacy for information literacy.&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;kurbanoglu&quot;, &quot;given&quot; : &quot;serap&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;akkoyunlu&quot;, &quot;given&quot; : &quot;buket&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;umay&quot;, &quot;given&quot; : &quot;aysun&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of documentation&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;6&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2006&quot;, &quot;12&quot; ] ] }, &quot;note&quot; : &quot;from duplicate 1 (developing the information literacy self-efficacy scale kurbanoglu, s serap; akkoyunlu, buket; umay, aysun)\n\nfrom duplicate 1 (developing the information literacy self-efficacy scale kurbanoglu, s serap; akkoyunlu, buket; umay, aysun)\n\naccession number: 502975837; authors:kurbanoglu, s. serap;\u00a0akkoyunlu, buket;\u00a0umay, aysun; physical description: bibliography; illustration; table; subject: research -methodology; subject: library orientation -evaluation; number of pages: 14p; record type: article&quot;, &quot;page&quot; : &quot;730-743&quot;, &quot;title&quot; : &quot;developing the information literacy self-efficacy scale&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;62&quot; }, &quot;suppress-author&quot; : 1, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=113ffd09-ac44-4c7c-a271-5e76217b2798&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(2006)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(2006)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(2006)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(2006)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>, spil-q allows users to rate their self-efficacy with just six questions, allowing the author to keep the survey brief. questions 2, 3, 4, and 6 measure locating information, accessing information, evaluating information, and citing, respectively. this modified spil-q will be referred to as m-spil-q for clarity in this paper.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>the author adapted questions from the information literacy assessment for education (ilas-ed) to measure il skills. ilas-ed, also known as b-tiled </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;abstract&quot; : &quot;few constituencies exist where it is more important to produce information literate individuals than teacher candidates, yet rarely is it suggested that practitioners newly entering the field are adequately prepared to teach and model information literacy to their students. as a result, information literacy has been established as a key outcome by a number of teacher education accrediting bodies and professional associations. corollary to this initiative is the effort to develop valid instruments that assess information literacy skills of teacher candidates. this paper describes the development and validation of the information literacy assessment scale in education (ilas-ed). this instrument can be used to inform curricular and instructional decisions and to provide evidence of institutional effectiveness for program reviews.&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;beile&quot;, &quot;given&quot; : &quot;penny m&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;society for information technology &amp; teacher education international conference&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;1&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2007&quot; ] ] }, &quot;page&quot; : &quot;1-27&quot;, &quot;title&quot; : &quot;the ilas-ed: a standards-based instrument for assessing pre-service teachers\u2019 information literacy levels&quot;, &quot;type&quot; : &quot;paper-conference&quot;, &quot;volume&quot; : &quot;2007&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=4e7969cf-c30c-4be7-abc3-11ba4e156a90&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(beile, 2007)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(beile, 2007)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(beile, 2007)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(beile, 2007)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>, assesses basic il skills with multiple-choice questions. during ilas-ed’s development, the instrument demonstrated reasonable reliability and validity </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;beile&quot;, &quot;given&quot; : &quot;penny m&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;id&quot; : &quot;item-1&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2005&quot; ] ] }, &quot;publisher&quot; : &quot;university of central florida&quot;, &quot;title&quot; : &quot;development and validation of the beile test of information literacy for education (b-tiled)&quot;, &quot;type&quot; : &quot;thesis&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=2051e984-6393-4461-b718-e3e5811ebeff&quot; ] }, { &quot;id&quot; : &quot;item-2&quot;, &quot;itemdata&quot; : { &quot;abstract&quot; : &quot;few constituencies exist where it is more important to produce information literate individuals than teacher candidates, yet rarely is it suggested that practitioners newly entering the field are adequately prepared to teach and model information literacy to their students. as a result, information literacy has been established as a key outcome by a number of teacher education accrediting bodies and professional associations. corollary to this initiative is the effort to develop valid instruments that assess information literacy skills of teacher candidates. this paper describes the development and validation of the information literacy assessment scale in education (ilas-ed). this instrument can be used to inform curricular and instructional decisions and to provide evidence of institutional effectiveness for program reviews.&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;beile&quot;, &quot;given&quot; : &quot;penny m&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;society for information technology &amp; teacher education international conference&quot;, &quot;id&quot; : &quot;item-2&quot;, &quot;issue&quot; : &quot;1&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2007&quot; ] ] }, &quot;page&quot; : &quot;1-27&quot;, &quot;title&quot; : &quot;the ilas-ed: a standards-based instrument for assessing pre-service teachers\u2019 information literacy levels&quot;, &quot;type&quot; : &quot;paper-conference&quot;, &quot;volume&quot; : &quot;2007&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=4e7969cf-c30c-4be7-abc3-11ba4e156a90&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(beile, 2005, 2007)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(beile, 2005, 2007)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(beile, 2005, 2007)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(beile, 2005, 2007)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>. it is freely available and has been used in several il studies </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin; mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;issn&quot; : &quot;1556-8881&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;alfonzo&quot;, &quot;given&quot; : &quot;paige&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;batson&quot;, &quot;given&quot; : &quot;jennifer&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;international journal of doctoral studies&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2014&quot; ] ] }, &quot;note&quot; : &quot;item citation: international journal of doctoral studies, vol 9, pp 061-071 (2014)\n\nrelated material: http://www.informingscience.org/publications/1973\n\nrelated material: https://doaj.org/toc/1556-8881\n\nrelated material: https://doaj.org/toc/1556-8873\n\naccession number: edsdoj.022df300a8e14433a33e30656594326a; publication type: journal article; source: international journal of doctoral studies, vol 9, pp 061-071 (2014); language: english; format: electronic resource; publication date: 20140101; rights: journal licence: publisher's own license; imprint: informing science institute, 2014.&quot;, &quot;page&quot; : &quot;61-71&quot;, &quot;title&quot; : &quot;utilizing a co-teaching model to enhance digital literacy instruction for doctoral students&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;9&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=fd2affda-883b-462c-b70f-bd643895e63e&quot; ] }, { &quot;id&quot; : &quot;item-2&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;doi:10.1007/978-3-319-14136-7_54&quot;, &quot;issn&quot; : &quot;9783319141350&quot;, &quot;abstract&quot; : &quot;it is commonly assumed that the self-efficacy is a predictor of the student motivation and learning and therefore could influence self-regulated lifelong learning. hence, the role of self-efficacy in the formation of information literate person is of special interest for this study. the main objective of this paper is to investigate the relation between the student information literacy self-efficacy and actual information literacy competence, while placing the research study results in the context of the current literature on the information literacy self-efficacy and information literacy competence. the results of the statistical correlation analysis indicate the relationship between results on the information literacy self-efficacy scale and the information literacy competence assessment scale consisting of four content clusters. it was found that information literacy self-efficacy is significantly and positively related to information literacy competence. in addition, the scores on the advanced ilse are correlated with each of the four information-literacy content clusters.&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;batarelo koki\u0107&quot;, &quot;given&quot; : &quot;ivana&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;novosel&quot;, &quot;given&quot; : &quot;vi\u0161nja&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;information literacy. lifelong learning &amp; digital citizenship in the 21st century&quot;, &quot;id&quot; : &quot;item-2&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2014&quot;, &quot;1&quot; ] ] }, &quot;note&quot; : &quot;102739858&quot;, &quot;page&quot; : &quot;512-520&quot;, &quot;title&quot; : &quot;the ball is in your court: information literacy self-efficacy and information literacy competence relation&quot;, &quot;type&quot; : &quot;paper-conference&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=2d65ae5e-fc76-48c4-9cea-ee04928f0c13&quot; ] }, { &quot;id&quot; : &quot;item-3&quot;, &quot;itemdata&quot; : { &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;cannon&quot;, &quot;given&quot; : &quot;tyrone&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;id&quot; : &quot;item-3&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2007&quot; ] ] }, &quot;publisher&quot; : &quot;university of san francisco&quot;, &quot;title&quot; : &quot;closing the digital divide: an assessment of urban graduate teacher education students' knowledge of information literacy and their readiness to integrate information literacy into their teaching&quot;, &quot;type&quot; : &quot;thesis&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=c786842b-85a2-4297-aeba-93457a32875f&quot; ] }, { &quot;id&quot; : &quot;item-4&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.18438/b82k7w&quot;, &quot;issn&quot; : &quot;1715-720x&quot;, &quot;abstract&quot; : &quot;objective the authors investigated the impact of library instruction on information literacy (il) skills as part of acrl\u2019s aia initiative. additionally, the researchers sought to determine whether there was a relationship between il tests scores and research experiences with student success outcomes such as retention. methods the researchers administered a standardized il test to 455 graduate and undergraduate students in multiple disciplines. they then collected outcome data on gpa, retention, and graduation three years later. results while there were no significant differences between those students who had instruction and those who did not on the il test, a regression analysis revealed that experience writing research papers that required library resources and an individual\u2019s use of library boks throughout their academic career demonstrated significant, positive relationships with whether a student passed the information literacy test. additionally, using the longitudinal data on gpa, retention, graduation, and employment, the researchers found that students\u2019 il scores were significantly correlated with their gpas, and that students who passed the il test were more likely to be retained or graduate within six years. conclusion the ability to demonstrate il skills appears to contribute to retention and graduation and, therefore, may be an integral part of one\u2019s academic success. further, experience writing research papers and other meaningful assignments contributes to student success.&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;catalano&quot;, &quot;given&quot; : &quot;amy&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;phillips&quot;, &quot;given&quot; : &quot;sharon r&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;evidence based library and information practice&quot;, &quot;id&quot; : &quot;item-4&quot;, &quot;issue&quot; : &quot;4&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2016&quot; ] ] }, &quot;note&quot; : &quot;b-tiled\n\nitem citation: evidence based library and information practice, vol 11, iss 4, pp 2-13 (2016)\n\nrelated material: https://ejournals.library.ualberta.ca/index.php/eblip/article/view/28006/21052\n\nrelated material: https://doaj.org/toc/1715-720x\n\naccession number: edsdoj.f6961ef047db4e8a8c73b2b02c0db45a; publication type: journal article; source: evidence based library and information practice, vol 11, iss 4, pp 2-13 (2016); language: english; format: electronic resource; publication date: 20161201; rights: journal licence: cc by-nc-sa; imprint: university of alberta, 2016.&quot;, &quot;page&quot; : &quot;2-13&quot;, &quot;publisher&quot; : &quot;university of alberta&quot;, &quot;title&quot; : &quot;information literacy and retention: a case study of the value of the library&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;11&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=f48e46d7-0f6d-4a7f-88c8-c01bd4158626&quot; ] }, { &quot;id&quot; : &quot;item-5&quot;, &quot;itemdata&quot; : { &quot;issn&quot; : &quot;04123131&quot;, &quot;abstract&quot; : &quot;academic libraries have been changing the traditional instructional framework of library instruction teaching modules to information literacy teaching modules. national standards for information literacy increased the possibility to unify such efforts throughout the country and clarify for librarians, administrators, and faculty the desired student learning outcomes. this paper presents findings of a quantitative research study developed to provide documentation for a regional accrediting body, college administration, and faculty on the efficacy of a subject specific information literacy curriculum and assessment instrument. the study took place within a seminary and began with an initial needs assessment. a previously developed instrument, b-tiled, which had been through a rigorous process of reliability and validity testing was applied to conduct the needs assessment. the findings of the needs assessment indicated a requirement for intervention which led to the development and implementation of a formal course of instruction in information literacy. the course was developed and taught by the researcher in the fall of 2010. all incoming 1st year students were required to take, complete, and pass a one-unit class in information literacy. in order to assess the effectiveness of the course, and to provide supportive documented data to the accrediting body, pretests and posttests were administered. the instrument used, b-tiled, was the same as that used in the needs assessment study. [abstract from author]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;jesse&quot;, &quot;given&quot; : &quot;sis&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;christian librarian&quot;, &quot;id&quot; : &quot;item-5&quot;, &quot;issue&quot; : &quot;1&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2012&quot;, &quot;7&quot; ] ] }, &quot;note&quot; : &quot;accession number: 77251479; source info: 2012, vol. 55 issue 1, p02; thesaurus term: information literacy standards; thesaurus term: universities &amp;amp; colleges; thesaurus term: academic library research; thesaurus term: needs assessment; thesaurus term: information literacy education; subject term: information literacy; number of pages: 15p; document type: article&quot;, &quot;page&quot; : &quot;2-16&quot;, &quot;title&quot; : &quot;subject specific information literacy curriculum and assessment&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;55&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=68046d97-b971-4fd7-9b96-fe93e7d001b4&quot; ] }, { &quot;id&quot; : &quot;item-6&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.3928/01484834-20150218-03&quot;, &quot;issn&quot; : &quot;01484834&quot;, &quot;pmid&quot; : &quot;25692245&quot;, &quot;abstract&quot; : &quot;maintaining evidence-based nursing practice requires information literacy (il) skills that should be established prior to completing an undergraduate nursing degree. based on bandura's social cognitive theory, this cross-sectional descriptive correlational study assessed the perceived and applied il skills of graduate nursing students from two family nurse practitioner (fnp) programs in the midwestern united states. results showed that although the 26 newly admitted fnp students demonstrated a high level of confidence in their il skills, the students did not perform well in the actual il skills test. according to bandura, the students' confidence in their il knowledge should allow students to be engaged in course activities requiring il skills. nurse educators teaching in undergraduate or graduate programs are in key positions to incorporate il experiences into class activities to allow for skill assessment and further practice. further research is needed on nursing students' il self-efficacy and performance. [j nurs educ. 2015;54(3, suppl.):s26-s30.]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;robertson&quot;, &quot;given&quot; : &quot;d susie&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;felicilda-reynaldo&quot;, &quot;given&quot; : &quot;rhea faye d&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of nursing education&quot;, &quot;id&quot; : &quot;item-6&quot;, &quot;issue&quot; : &quot;3, suppl&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2015&quot; ] ] }, &quot;note&quot; : &quot;accession number: 25692245. language: english. date created: 20150228. date completed: 20151215. update code: 20151215. publication type: journal article. journal id: 7705432. publication model: print-electronic. cited medium: internet. nlm iso abbr: j nurs educ. linking issn: 01484834. subset: im; n; date of electronic publication: 2015 feb 19. current imprints: publication: thorofare, n.j. : charles b. slack; original imprints: publication: new york, n.y. : blackiston division, mcgraw-hill, [c1962-&quot;, &quot;page&quot; : &quot;s26-30&quot;, &quot;publisher&quot; : &quot;charles b. slack&quot;, &quot;publisher-place&quot; : &quot;united states&quot;, &quot;title&quot; : &quot;evaluation of graduate nursing students' information literacy self-efficacy and applied skills&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;54&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=e598323d-e01e-4782-aa5a-317b3a02abb7&quot; ] }, { &quot;id&quot; : &quot;item-7&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.18438/b8b60x&quot;, &quot;issn&quot; : &quot;1715720x&quot;, &quot;abstract&quot; : &quot;objective -to determine whether playing library-related online games during information literacy instruction sessions improves student performance on questionnaires pertaining to selected research practices: identifying citation types and keyword and synonym development. methods -86 students in seven introductory english composition classes at a large urban university in the northeastern united states served as participants. each class visited the library for library instruction twice during a given semester. in the experimental group students received information literacy instruction that incorporated two online games, and the control group received the same lesson plan with the exception of a lecture in place of playing games. a six-item preand posttest questionnaire was developed and administered at the outset and conclusion of the two-session classes. the 172 individual tests were coded, graded, and analyzed using spss. results -a paired sample t-test comparing the control and experimental groups determined that that there was a statistically significant difference between scores on pre-tests and post-tests in the experimental group but not the control group. conclusion -students who played the online games improved significantly more from pre-test to post-test than students who received a lecture in lieu of playing online games, suggesting that participating in games related to the instruction they received resulted in an improved ability to select appropriate keywords and ascertain citation formats. these findings contribute to the evidence that online games concerning two frequently challenging research practices can be successfully applied to library instruction sessions to improve student comprehension of such skills. [abstract from author]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;tewell&quot;, &quot;given&quot; : &quot;eamon&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;angell&quot;, &quot;given&quot; : &quot;katelyn&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;evidence based library &amp; information practice&quot;, &quot;id&quot; : &quot;item-7&quot;, &quot;issue&quot; : &quot;1&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2015&quot;, &quot;1&quot; ] ] }, &quot;note&quot; : &quot;questions adapted from b-tiled&quot;, &quot;page&quot; : &quot;20-33&quot;, &quot;title&quot; : &quot;far from a trivial pursuit: assessing the effectiveness of games in information literacy instruction&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;10&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=9b58b9bb-27f7-4fa2-8473-60d86b6db54d&quot; ] }, { &quot;id&quot; : &quot;item-8&quot;, &quot;itemdata&quot; : { &quot;abstract&quot; : &quot;the aim of this sequential integrated mixed model design study was to examine information literacy (il) levels and needs of graduate students in education, social studies, and humanities at the mid-size canadian university. this was done through surveying 201 graduate students who volunteered to fill-in a quantitative questionnaire that included supplementary open-ended questions. to triangulate data and as part of the chosen methodological approach, 16 graduate students also took part in the semi-structured follow-up interviews which included observation of the participants on-task behaviour. in order to consider the il of graduate students in the larger context of a library information ecosystem, the researcher incorporated the technology acceptance model (tam) and the affordance theory (at) frameworks. the quantitative component of the study was based on the modified beile test of information literacy for education (b-tiled) survey as an instrument to measure the participants' il. the survey questions were organized to address the participants' demographic, academic and departmental characteristics. the statistically significant results were found for the b-tiled scores on the following three independent variables: (i) first language of participants (i.e., non-native english speakers performed lower), (ii) minimum course requirements completed for the master's degree (i.e., students who did not complete the minimum number of courses performed lower), and (iii) the department of study (i.e., master's of education and master's of social work students performed lower). the data from the follow-up interviews confirmed that graduate students perceived that they need more il-related instruction, as well as a discipline-specific instruction. findings suggest that graduate students may benefit from differentiated methods for gaining the il skills, through frequent and more hands-on in-library, in-class, and on-line il instruction. the conclusion of this study, points out that those who need sophisticated search and research skills, require sustained and individualized support in order to achieve the necessary comfort and mastery in doing so. thus, with increased technological development of library tools, a generic onetime library instruction, usually given in the first semester of graduate program is not sufficient to provide the most needed il skills.&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;magliaro&quot;, &quot;given&quot; : &quot;jelena&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;id&quot; : &quot;item-8&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2011&quot; ] ] }, &quot;publisher&quot; : &quot;university of windsor&quot;, &quot;title&quot; : &quot;comparing information literacy needs of nraduate students in selected graduate programs through the technology acceptance model and affordance theory&quot;, &quot;type&quot; : &quot;thesis&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=63e8d8fd-a0af-48b5-a517-cf824e9477c9&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(alfonzo &amp; batson, 2014; batarelo koki\u0107 &amp; novosel, 2014; cannon, 2007; catalano &amp; phillips, 2016; jesse, 2012; magliaro, 2011; robertson &amp; felicilda-reynaldo, 2015; tewell &amp; angell, 2015)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(alfonzo &amp; batson, 2014; batarelo koki\u0107 &amp; novosel, 2014; cannon, 2007; catalano &amp; phillips, 2016; jesse, 2012; magliaro, 2011; robertson &amp; felicilda-reynaldo, 2015; tewell &amp; angell, 2015)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(alfonzo &amp; batson, 2014; batarelo koki\u0107 &amp; novosel, 2014; cannon, 2007; catalano &amp; phillips, 2016; jesse, 2012; magliaro, 2011; robertson &amp; felicilda-reynaldo, 2015; tewell &amp; angell, 2015)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(alfonzo &amp; batson, 2014; batarelo kokić &amp; novosel, 2014; cannon, 2007; catalano &amp; phillips, 2016; jesse, 2012; magliaro, 2011; robertson &amp; felicilda-reynaldo, 2015; tewell &amp; angell, 2015)</span></span><!--[if supportfields]><span lang=en-ca style='font-size: 10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>. although the instrument was developed in 2005, the terminology used in the questions is still current.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>ilas-ed consists of 35 questions. questions 1 and 2 deal with general self-efficacy, questions 3 through 6 deal with students’ library instruction history, questions 7 through 28 test il skills, and questions 29 through 35 collect demographic data. for the purpose of this study, the author excluded questions 1 and 2 because self-efficacy was measured with more granularity by m-spil-q. question 3 was omitted as irrelevant to slis’s online students because it dealt with attending “a tour or physical orientation of the library” </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;abstract&quot; : &quot;few constituencies exist where it is more important to produce information literate individuals than teacher candidates, yet rarely is it suggested that practitioners newly entering the field are adequately prepared to teach and model information literacy to their students. as a result, information literacy has been established as a key outcome by a number of teacher education accrediting bodies and professional associations. corollary to this initiative is the effort to develop valid instruments that assess information literacy skills of teacher candidates. this paper describes the development and validation of the information literacy assessment scale in education (ilas-ed). this instrument can be used to inform curricular and instructional decisions and to provide evidence of institutional effectiveness for program reviews.&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;beile&quot;, &quot;given&quot; : &quot;penny m&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;society for information technology &amp; teacher education international conference&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;1&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2007&quot; ] ] }, &quot;page&quot; : &quot;1-27&quot;, &quot;title&quot; : &quot;the ilas-ed: a standards-based instrument for assessing pre-service teachers\u2019 information literacy levels&quot;, &quot;type&quot; : &quot;paper-conference&quot;, &quot;volume&quot; : &quot;2007&quot; }, &quot;locator&quot; : &quot;19&quot;, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=4e7969cf-c30c-4be7-abc3-11ba4e156a90&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(beile, 2007, p. 19)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(beile, 2007, p. 19)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(beile, 2007, p. 19)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(beile, 2007, p. 19)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>. the study also omitted questions 4 through 6, which concerned receiving instruction in the library, in the classroom, and one-on-one; without contextual information about how long ago the instruction took place, by whom or at which institution the instruction was given, or what the instruction covered, this data would provide limited insight about the impact of the instruction on il self-efficacy or skills.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>demographic data about age, ethnicity/race, and gender was collected with the intent to identify patterns (see appendix a for demographic questions), but preliminary analysis of data about age, gender, and race/ethnicity provided little insight. the survey also solicited information about the highest degree obtained and the number of years since respondents received their most recent degree. the results section details the collected demographic data.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>because ilas-ed was designed for students in education programs, some of the original ilas-ed questions were modified for use with lis students with the consent of ilas-ed’s author (p. beile, personal communication, june 20, 2017). (appendix b presents the modified questions along with the corresponding ilas-ed question number.) the modified form of the ilas-ed will be referred to as m-ilas-ed.<o:p></o:p></span></p> <p class=msonormal style='mso-outline-level:1'><i style='mso-bidi-font-style: normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></i></p> <p class=msonormal style='mso-outline-level:1'><i style='mso-bidi-font-style: normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>spring 2018<o:p></o:p></span></i></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>although the data from the m-spil-q and m-ilas-ed instruments gave the author valuable insights, the fall 2017 assessment measured some aspects of il with multiple questions, while others were measured with only a few; for example, seven questions were used to assess the ability of students to access information, but only two questions were used to assess students’ citation skills (see table 1). <o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>when the study was repeated for additional data collection in spring 2018, the author chose to use michalak and rysavy’s </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin; mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1080/08963568.2016.1145787&quot;, &quot;issn&quot; : &quot;08963568&quot;, &quot;abstract&quot; : &quot;in 2015, the information literacy (il) team designed an instrument to determine international graduate business students' perceptions of their il skills and their actual test-assessed il skills. the purpose of this research was to compare international graduate students' perceptions of their il skills versus their test-assessed competencies, with the aim of creating training modules to improve test-identified deficiencies. results demonstrate that wide discrepancies exist between students' perceptions of their own skills and their actual skills on all six ila-topic areas assessed by the internally developed test instruments, with students overestimating their abilities in all areas. [abstract from publisher]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;michalak&quot;, &quot;given&quot; : &quot;russell&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;rysavy&quot;, &quot;given&quot; : &quot;monica&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of business &amp; finance librarianship&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;2&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2016&quot;, &quot;4&quot; ] ] }, &quot;note&quot; : &quot;accession number: 116263533; michalak, russell 1; rysavy, monica d. t. 1; affiliations: 1 : goldey-beacom college, wilmington, delaware, usa; source info: apr-jun2016, vol. 21 issue 2, p152; thesaurus term: information literacy -research; thesaurus term: business information services; subject term: instructional systems design; subject term: international business enterprises; subject term: graduate students; author-supplied keyword: business information literacy; author-supplied keyword: information literacy assessment; author-supplied keyword: information literacy skills; author-supplied keyword: instructional design; author-supplied keyword: international business graduate students; number of pages: 23p; document type: article&quot;, &quot;page&quot; : &quot;152-174&quot;, &quot;title&quot; : &quot;information literacy in 2015: international graduate business students' perceptions of information literacy skills compared to test-assessed skills.&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;21&quot; }, &quot;suppress-author&quot; : 1, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=79e910db-f957-4128-857a-5ce4b0a15542&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(2016)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(2016)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(2016)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(2016)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'> unmodified spil-q to measure self-efficacy and their information literacy assessment (ila) instrument to measure il skills.<b style='mso-bidi-font-weight:normal'> </b>michalak and rysavy granted the author permission (r. michalak, personal communication, dec. 7, 2017) to use the unmodified spil-q instrument and a minimally modified version of their ila instrument </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin; mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1080/08963568.2016.1145787&quot;, &quot;issn&quot; : &quot;08963568&quot;, &quot;abstract&quot; : &quot;in 2015, the information literacy (il) team designed an instrument to determine international graduate business students' perceptions of their il skills and their actual test-assessed il skills. the purpose of this research was to compare international graduate students' perceptions of their il skills versus their test-assessed competencies, with the aim of creating training modules to improve test-identified deficiencies. results demonstrate that wide discrepancies exist between students' perceptions of their own skills and their actual skills on all six ila-topic areas assessed by the internally developed test instruments, with students overestimating their abilities in all areas. [abstract from publisher]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;michalak&quot;, &quot;given&quot; : &quot;russell&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;rysavy&quot;, &quot;given&quot; : &quot;monica&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of business &amp; finance librarianship&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;2&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2016&quot;, &quot;4&quot; ] ] }, &quot;note&quot; : &quot;accession number: 116263533; michalak, russell 1; rysavy, monica d. t. 1; affiliations: 1 : goldey-beacom college, wilmington, delaware, usa; source info: apr-jun2016, vol. 21 issue 2, p152; thesaurus term: information literacy -research; thesaurus term: business information services; subject term: instructional systems design; subject term: international business enterprises; subject term: graduate students; author-supplied keyword: business information literacy; author-supplied keyword: information literacy assessment; author-supplied keyword: information literacy skills; author-supplied keyword: instructional design; author-supplied keyword: international business graduate students; number of pages: 23p; document type: article&quot;, &quot;page&quot; : &quot;152-174&quot;, &quot;title&quot; : &quot;information literacy in 2015: international graduate business students' perceptions of information literacy skills compared to test-assessed skills.&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;21&quot; }, &quot;suppress-author&quot; : 1, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=79e910db-f957-4128-857a-5ce4b0a15542&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(2016)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(2016)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(2016)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(2016)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>. the spil-q and ila instruments were developed together, so each question on the ila corresponds to a spil-q item, and each il skill was measured by the same number of questions. <o:p></o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></b></p> <p class=msolistparagraph style='margin-left:0in;mso-add-space:auto;line-height: normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>there were only two modifications to the ila instrument. module 1, question 5 was modified to reflect lsu’s name and library’s name. module 2, question 8 was changed from “materials in the hirons library are organized . . .” to “materials in most major university libraries in the united states are organized . . .” to make the question applicable to u.s. universities in general. the survey used the same demographic questions used in the fall 2017 study; the results section reports the collected demographic data.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>table 1<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>fall 2017 il self-efficacy beliefs keyed to m-ilas-ed questions <o:p></o:p></span></p> <table class=apareport border=1 cellspacing=0 cellpadding=0 style='border-collapse: collapse;border:none;mso-border-alt:solid windowtext .5pt;mso-yfti-tbllook: 1184;mso-padding-alt:0in 5.4pt 0in 5.4pt;mso-border-insideh:.5pt solid windowtext; mso-border-insidev:.5pt solid windowtext'> <tr style='mso-yfti-irow:-1;mso-yfti-firstrow:yes;mso-yfti-lastfirstrow:yes'> <td valign=top style='border:solid windowtext 1.0pt;mso-border-alt:solid windowtext .5pt; padding:0in 5.4pt 0in 5.4pt'> <p class=msonormal style='mso-yfti-cnfc:1'><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin;color:black;mso-themecolor:text1;mso-ansi-language: en-us;mso-fareast-language:ja'>m-spil-q self-efficacy belief<o:p></o:p></span></p> </td> <td valign=top style='border:solid windowtext 1.0pt;border-left:none; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:0in 5.4pt 0in 5.4pt'> <p class=msonormal style='mso-yfti-cnfc:1'><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin;color:black;mso-themecolor:text1;mso-ansi-language: en-us;mso-fareast-language:ja'>corresponding m-ilas-ed questions <o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:0'> <td valign=top style='border:solid windowtext 1.0pt;border-top:none; mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:0in 5.4pt 0in 5.4pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;color:black; mso-themecolor:text1;mso-ansi-language:en-us;mso-fareast-language:ja'>locating information<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:0in 5.4pt 0in 5.4pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;color:black; mso-themecolor:text1;mso-ansi-language:en-us;mso-fareast-language:ja'>questions 8, 9, 10, 12, 20<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:1'> <td valign=top style='border:solid windowtext 1.0pt;border-top:none; mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:0in 5.4pt 0in 5.4pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;color:black; mso-themecolor:text1;mso-ansi-language:en-us;mso-fareast-language:ja'>accessing information <o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:0in 5.4pt 0in 5.4pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;color:black; mso-themecolor:text1;mso-ansi-language:en-us;mso-fareast-language:ja'>questions 11, 13, 14, 15, 16, 17, 18<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:2'> <td valign=top style='border:solid windowtext 1.0pt;border-top:none; mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:0in 5.4pt 0in 5.4pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;color:black; mso-themecolor:text1;mso-ansi-language:en-us;mso-fareast-language:ja'>evaluating information<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:0in 5.4pt 0in 5.4pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;color:black; mso-themecolor:text1;mso-ansi-language:en-us;mso-fareast-language:ja'>questions 7, 19, 21, 23<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:3;mso-yfti-lastrow:yes'> <td valign=top style='border:solid windowtext 1.0pt;border-top:none; mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:0in 5.4pt 0in 5.4pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;color:black; mso-themecolor:text1;mso-ansi-language:en-us;mso-fareast-language:ja'>citing<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:0in 5.4pt 0in 5.4pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;color:black; mso-themecolor:text1;mso-ansi-language:en-us;mso-fareast-language:ja'>questions 24, 25<o:p></o:p></span></p> </td> </tr> </table> <p class=msolistparagraph style='margin-left:0in;mso-add-space:auto;line-height: normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style: normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>data analysis techniques<o:p></o:p></span></i></b></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style: normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></i></b></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>the author transferred the data collected in qualtrics to spss. each item from m-ilas-ed was keyed to one of the four self-efficacy beliefs (locate, access, evaluate, and cite) measured by the four questions from m-spil-q, allowing individual skills to be measured against self-efficacy beliefs for possible correlations. table 1 provides a breakdown of skills keyed to questions. <o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>the total m-spil-q and m-ilas-ed scores were used to calculate the pearson correlation coefficient to determine whether there was a possible correlation between students’ il perceived self-efficacy and demonstrated il skills. <o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>the same procedure was followed in spring 2018, this time using the spil-q and ila scores. again, each question in the ila was keyed to a self-efficacy belief (developing a topic, locating information, accessing information, evaluating information, writing, and citing) in spil-q. the librarian keyed the questions as described by michalak and rysavy </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1080/08963568.2016.1145787&quot;, &quot;issn&quot; : &quot;08963568&quot;, &quot;abstract&quot; : &quot;in 2015, the information literacy (il) team designed an instrument to determine international graduate business students' perceptions of their il skills and their actual test-assessed il skills. the purpose of this research was to compare international graduate students' perceptions of their il skills versus their test-assessed competencies, with the aim of creating training modules to improve test-identified deficiencies. results demonstrate that wide discrepancies exist between students' perceptions of their own skills and their actual skills on all six ila-topic areas assessed by the internally developed test instruments, with students overestimating their abilities in all areas. [abstract from publisher]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;michalak&quot;, &quot;given&quot; : &quot;russell&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;rysavy&quot;, &quot;given&quot; : &quot;monica&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of business &amp; finance librarianship&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;2&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2016&quot;, &quot;4&quot; ] ] }, &quot;note&quot; : &quot;accession number: 116263533; michalak, russell 1; rysavy, monica d. t. 1; affiliations: 1 : goldey-beacom college, wilmington, delaware, usa; source info: apr-jun2016, vol. 21 issue 2, p152; thesaurus term: information literacy -research; thesaurus term: business information services; subject term: instructional systems design; subject term: international business enterprises; subject term: graduate students; author-supplied keyword: business information literacy; author-supplied keyword: information literacy assessment; author-supplied keyword: information literacy skills; author-supplied keyword: instructional design; author-supplied keyword: international business graduate students; number of pages: 23p; document type: article&quot;, &quot;page&quot; : &quot;152-174&quot;, &quot;title&quot; : &quot;information literacy in 2015: international graduate business students' perceptions of information literacy skills compared to test-assessed skills.&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;21&quot; }, &quot;suppress-author&quot; : 1, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=79e910db-f957-4128-857a-5ce4b0a15542&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(2016)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(2016)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(2016)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(2016)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>results<o:p></o:p></span></b></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style: normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>fall 2017<o:p></o:p></span></i></b></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style: normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></i></b></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>of the 31 respondents to the gender question, the majority of respondents were female (<i style='mso-bidi-font-style:normal'>n</i> = 24, 75%); the remainder were male (<i style='mso-bidi-font-style:normal'>n</i> = 7, 22%) or preferred not to answer (<i style='mso-bidi-font-style:normal'>n</i> = 1, 3%). of the 32 respondents to the race question, 20 (63%) were white, 5 (16%) were black or african american, 2 (6%) identified themselves as hispanic of any race, 3 (9%) identified themselves as two or more races, and 2 (6%) preferred not to answer. of the 19 respondents to the age question, 2 (11%) were between 20 and 24 years of age, 7 (37%) were between 25 and 29 years of age, 2 (11%) were between 30 and 34 years of age, 2 (11%) were between 35 and 39 years of age, 3 (16%) were between 40 and 44 years of age, and 3 (16%) were between 45 and 49 years of age.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>the highest degree obtained by respondents was a bachelor’s degree (<i style='mso-bidi-font-style: normal'>n</i> = 23, 72%), followed by a master’s degree (<i style='mso-bidi-font-style: normal'>n</i> = 8, 25%), and a doctoral degree (<i style='mso-bidi-font-style: normal'>n</i> = 1, 3%). twenty respondents (63%) had earned their degree within the last 5 years, 6 (19%) had earned their most recent degree within the last 6 to 10 years, 5 (16%) within the last 11 to 15 years, and 1 (3%) within the last 16 to 20 years.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>table 2 reports the mean m-spil-q and m-ilas-ed scores. the highest possible m-spil-q score was 20; the highest possible m-ilas-ed score was 18. <o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>tables 3 and 4 summarize the mean scores of each of the four areas (locating information, accessing information, evaluating information, and citing) tested by m-spil-q and m-ilas-ed. <o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>the author used spss to calculate pearson’s <i style='mso-bidi-font-style:normal'>r</i> to determine if a correlation existed between m-spil-q scores and m-ilas-ed scores (table 5). the results indicate a moderate positive correlation that is statistically significant (<i style='mso-bidi-font-style:normal'>p</i> &lt;.005), meaning that as self-efficacy scores increased so did il scores.<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-us'>table 2<o:p></o:p></span></p> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-us'>fall 2017 m-spil-q and m-ilas-ed scores<sup>a<o:p></o:p></sup></span></p> <table class=msonormaltable border=1 cellspacing=0 cellpadding=0 style='margin-left:-.25pt;border-collapse:collapse;border:none;mso-border-alt: solid windowtext .5pt;mso-yfti-tbllook:1184;mso-padding-alt:2.15pt 2.15pt 2.15pt 2.15pt; mso-border-insideh:.5pt solid windowtext;mso-border-insidev:.5pt solid windowtext'> <tr style='mso-yfti-irow:0;mso-yfti-firstrow:yes'> <td valign=top style='border:solid windowtext 1.0pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-us'><o:p>&nbsp;</o:p></span></p> </td> <td width=19 valign=top style='width:.3pt;border:solid windowtext 1.0pt; border-left:none;mso-border-left-alt:solid windowtext .5pt;mso-border-alt: solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal><i style='mso-bidi-font-style:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-us'>n<o:p></o:p></span></i></p> </td> <td width=65 valign=top style='width:.3pt;border:solid windowtext 1.0pt; border-left:none;mso-border-left-alt:solid windowtext .5pt;mso-border-alt: solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-us'>minimum<o:p></o:p></span></p> </td> <td width=67 valign=top style='width:.3pt;border:solid windowtext 1.0pt; border-left:none;mso-border-left-alt:solid windowtext .5pt;mso-border-alt: solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-us'>maximum<o:p></o:p></span></p> </td> <td width=101 style='width:76.1pt;border:solid windowtext 1.0pt;border-left: none;mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-us'>mean<o:p></o:p></span></p> </td> <td width=56 style='width:42.2pt;border:solid windowtext 1.0pt;border-left: none;mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-us'>sd<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:1'> <td valign=top style='border:solid windowtext 1.0pt;border-top:none; mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-us'>m-spil-q <o:p></o:p></span></p> </td> <td width=19 valign=top style='width:.3pt;border-top:none;border-left:none; border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-us'>32<o:p></o:p></span></p> </td> <td width=65 valign=top style='width:.3pt;border-top:none;border-left:none; border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-us'>8<o:p></o:p></span></p> </td> <td width=67 valign=top style='width:.3pt;border-top:none;border-left:none; border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-us'>20<o:p></o:p></span></p> </td> <td width=101 style='width:76.1pt;border-top:none;border-left:none; border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-us'>15.37 (77%)<o:p></o:p></span></p> </td> <td width=56 style='width:42.2pt;border-top:none;border-left:none;border-bottom: solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;mso-border-top-alt: solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt;mso-border-alt: solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-us'>3.28<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:2;mso-yfti-lastrow:yes'> <td valign=top style='border:solid windowtext 1.0pt;border-top:none; mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-us'>m-ilas-ed<o:p></o:p></span></p> </td> <td width=19 valign=top style='width:.3pt;border-top:none;border-left:none; border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-us'>32<o:p></o:p></span></p> </td> <td width=65 valign=top style='width:.3pt;border-top:none;border-left:none; border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-us'>6<o:p></o:p></span></p> </td> <td width=67 valign=top style='width:.3pt;border-top:none;border-left:none; border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-us'>15<o:p></o:p></span></p> </td> <td width=101 style='width:76.1pt;border-top:none;border-left:none; border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-us'>10.88 (60%)<o:p></o:p></span></p> </td> <td width=56 style='width:42.2pt;border-top:none;border-left:none;border-bottom: solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;mso-border-top-alt: solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt;mso-border-alt: solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-us'>2.34<o:p></o:p></span></p> </td> </tr> </table> <p class=msonormal><sup><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-us'>a</span></sup><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-us'>numbers are rounded to the nearest hundredth; percentages are rounded to the nearest percent.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>table 3<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>fall 2017 m-spil-q scores (<i style='mso-bidi-font-style:normal'>n</i> = 32) by subcategory<sup>a</sup> <o:p></o:p></span></p> <table class=msonormaltable border=1 cellspacing=0 cellpadding=0 style='border-collapse:collapse;border:none;mso-border-alt:solid windowtext .5pt; mso-yfti-tbllook:1184;mso-padding-alt:2.15pt 2.15pt 2.15pt 2.15pt;mso-border-insideh: .5pt solid windowtext;mso-border-insidev:.5pt solid windowtext'> <tr style='mso-yfti-irow:0;mso-yfti-firstrow:yes;page-break-inside:avoid; height:19.8pt'> <td width=159 valign=bottom style='width:119.3pt;border:solid windowtext 1.0pt; mso-border-alt:solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:19.8pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>m-spil-q subcategory<o:p></o:p></span></p> </td> <td width=70 valign=bottom style='width:52.35pt;border:solid windowtext 1.0pt; border-left:none;mso-border-left-alt:solid windowtext .5pt;mso-border-alt: solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:19.8pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>minimum<o:p></o:p></span></p> </td> <td width=72 valign=bottom style='width:54.3pt;border:solid windowtext 1.0pt; border-left:none;mso-border-left-alt:solid windowtext .5pt;mso-border-alt: solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:19.8pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>maximum<o:p></o:p></span></p> </td> <td width=78 valign=bottom style='width:58.3pt;border:solid windowtext 1.0pt; border-left:none;mso-border-left-alt:solid windowtext .5pt;mso-border-alt: solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:19.8pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>mean<o:p></o:p></span></p> </td> <td width=34 valign=bottom style='width:25.3pt;border:solid windowtext 1.0pt; border-left:none;mso-border-left-alt:solid windowtext .5pt;mso-border-alt: solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:19.8pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>sd<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:1;page-break-inside:avoid;height:19.8pt'> <td width=159 valign=bottom style='width:119.3pt;border:solid windowtext 1.0pt; border-top:none;mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:19.8pt'> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'> locate<o:p></o:p></span></p> </td> <td width=70 valign=bottom style='width:52.35pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:19.8pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>1<o:p></o:p></span></p> </td> <td width=72 valign=bottom style='width:54.3pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:19.8pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>5<o:p></o:p></span></p> </td> <td width=78 valign=bottom style='width:58.3pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:19.8pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>3.81 (76%)<o:p></o:p></span></p> </td> <td width=34 valign=bottom style='width:25.3pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:19.8pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>1.00<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:2;page-break-inside:avoid;height:19.8pt'> <td width=159 valign=bottom style='width:119.3pt;border:solid windowtext 1.0pt; border-top:none;mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:19.8pt'> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'> access<o:p></o:p></span></p> </td> <td width=70 valign=bottom style='width:52.35pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:19.8pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>1<o:p></o:p></span></p> </td> <td width=72 valign=bottom style='width:54.3pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:19.8pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>5<o:p></o:p></span></p> </td> <td width=78 valign=bottom style='width:58.3pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:19.8pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>3.56 (71%)<o:p></o:p></span></p> </td> <td width=34 valign=bottom style='width:25.3pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:19.8pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>1.05<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:3;page-break-inside:avoid;height:19.8pt'> <td width=159 valign=bottom style='width:119.3pt;border:solid windowtext 1.0pt; border-top:none;mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:19.8pt'> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'> evaluate<o:p></o:p></span></p> </td> <td width=70 valign=bottom style='width:52.35pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:19.8pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>1<o:p></o:p></span></p> </td> <td width=72 valign=bottom style='width:54.3pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:19.8pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>5<o:p></o:p></span></p> </td> <td width=78 valign=bottom style='width:58.3pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:19.8pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>4.03 (81%)<o:p></o:p></span></p> </td> <td width=34 valign=bottom style='width:25.3pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:19.8pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>.90<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:4;mso-yfti-lastrow:yes;page-break-inside:avoid; height:19.8pt'> <td width=159 valign=bottom style='width:119.3pt;border:solid windowtext 1.0pt; border-top:none;mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:19.8pt'> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'> cite<o:p></o:p></span></p> </td> <td width=70 valign=bottom style='width:52.35pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:19.8pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>1<o:p></o:p></span></p> </td> <td width=72 valign=bottom style='width:54.3pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:19.8pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>5<o:p></o:p></span></p> </td> <td width=78 valign=bottom style='width:58.3pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:19.8pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>3.97 (79%)<o:p></o:p></span></p> </td> <td width=34 valign=bottom style='width:25.3pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:19.8pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>1.03<o:p></o:p></span></p> </td> </tr> </table> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><sup><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>a</span></sup><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>numbers are rounded to the nearest hundredth; percentages are rounded to the nearest percent. <o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>table 4<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>fall 2017 m-ilas-ed scores (<i style='mso-bidi-font-style:normal'>n</i> = 32) by subcategory<sup>a</sup><o:p></o:p></span></p> <table class=msonormaltable border=1 cellspacing=0 cellpadding=0 width=0 style='width:350.9pt;border-collapse:collapse;border:none;mso-border-alt:solid windowtext .5pt; mso-yfti-tbllook:1184;mso-padding-alt:2.15pt 2.15pt 2.15pt 2.15pt;mso-border-insideh: .5pt solid windowtext;mso-border-insidev:.5pt solid windowtext'> <tr style='mso-yfti-irow:0;mso-yfti-firstrow:yes;page-break-inside:avoid; height:15.55pt'> <td valign=bottom style='border:solid windowtext 1.0pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:15.55pt'> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>m-ilas-ed<sup> </sup>subcategory<sup>b</sup><o:p></o:p></span></p> </td> <td valign=bottom style='border:solid windowtext 1.0pt;border-left:none; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt;height:15.55pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>minimum<o:p></o:p></span></p> </td> <td valign=bottom style='border:solid windowtext 1.0pt;border-left:none; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt;height:15.55pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>maximum<o:p></o:p></span></p> </td> <td valign=bottom style='border:solid windowtext 1.0pt;border-left:none; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt;height:15.55pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>mean<o:p></o:p></span></p> </td> <td valign=bottom style='border:solid windowtext 1.0pt;border-left:none; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt;height:15.55pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>sd<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:1;page-break-inside:avoid;height:15.55pt'> <td valign=bottom style='border:solid windowtext 1.0pt;border-top:none; mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:15.55pt'> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'> locate<o:p></o:p></span></p> </td> <td valign=bottom style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt;height:15.55pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>1<o:p></o:p></span></p> </td> <td valign=bottom style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt;height:15.55pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>4<o:p></o:p></span></p> </td> <td valign=bottom style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt;height:15.55pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>2.88 (58%)<o:p></o:p></span></p> </td> <td valign=bottom style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt;height:15.55pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>.94<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:2;page-break-inside:avoid;height:15.55pt'> <td valign=bottom style='border:solid windowtext 1.0pt;border-top:none; mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:15.55pt'> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'> access<o:p></o:p></span></p> </td> <td valign=bottom style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt;height:15.55pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>1<o:p></o:p></span></p> </td> <td valign=bottom style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt;height:15.55pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>7<o:p></o:p></span></p> </td> <td valign=bottom style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt;height:15.55pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>3.91 (56%)<o:p></o:p></span></p> </td> <td valign=bottom style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt;height:15.55pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>1.45<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:3;page-break-inside:avoid;height:15.55pt'> <td valign=bottom style='border:solid windowtext 1.0pt;border-top:none; mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:15.55pt'> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'> evaluate<o:p></o:p></span></p> </td> <td valign=bottom style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt;height:15.55pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>0<o:p></o:p></span></p> </td> <td valign=bottom style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt;height:15.55pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>4<o:p></o:p></span></p> </td> <td valign=bottom style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt;height:15.55pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>2.28 (57%)<o:p></o:p></span></p> </td> <td valign=bottom style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt;height:15.55pt'> <p class=msonormal align=center style='text-align:center;mso-outline-level: 1'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>.99<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:4;mso-yfti-lastrow:yes;page-break-inside:avoid; height:15.55pt'> <td valign=bottom style='border:solid windowtext 1.0pt;border-top:none; mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:15.55pt'> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'> cite<o:p></o:p></span></p> </td> <td valign=bottom style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt;height:15.55pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>0<o:p></o:p></span></p> </td> <td valign=bottom style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt;height:15.55pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>2<o:p></o:p></span></p> </td> <td valign=bottom style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt;height:15.55pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>1.81 (91%)<o:p></o:p></span></p> </td> <td valign=bottom style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt;height:15.55pt'> <p class=msonormal align=center style='text-align:center'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>.47<o:p></o:p></span></p> </td> </tr> </table> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><sup><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>a</span></sup><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>numbers are rounded to the nearest hundredth; percentages are rounded to the nearest percent. <o:p></o:p></span></p> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><sup><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>b</span></sup><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>the total possible points for each subsection of the m-ilas-ed are as follows: locate 5, access 7, evaluate 4, cite 2.<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='page-break-after:avoid'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>table 5<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>correlation between m-ilas-ed and m-spil-q scores<o:p></o:p></span></p> <table class=msonormaltable border=1 cellspacing=0 cellpadding=0 width=0 style='width:359.45pt;border-collapse:collapse;border:none;mso-border-alt: solid windowtext .5pt;mso-yfti-tbllook:1184;mso-padding-alt:2.15pt 2.15pt 2.15pt 2.15pt; mso-border-insideh:.5pt solid windowtext;mso-border-insidev:.5pt solid windowtext'> <tr style='mso-yfti-irow:0;mso-yfti-firstrow:yes;page-break-inside:avoid'> <td width=296 colspan=2 valign=bottom style='width:221.85pt;border:solid windowtext 1.0pt; mso-border-alt:solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> </td> <td width=86 valign=bottom style='width:64.2pt;border:solid windowtext 1.0pt; border-left:none;mso-border-left-alt:solid windowtext .5pt;mso-border-alt: solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>m-spil-q<o:p></o:p></span></p> </td> <td width=98 valign=bottom style='width:73.4pt;border:solid windowtext 1.0pt; border-left:none;mso-border-left-alt:solid windowtext .5pt;mso-border-alt: solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>m-ilas-ed<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:1;page-break-inside:avoid'> <td width=163 rowspan=3 valign=top style='width:1.7in;border:solid windowtext 1.0pt; border-top:none;mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>m-spil-q<o:p></o:p></span></p> </td> <td width=133 valign=top style='width:99.45pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>pearson correlation<o:p></o:p></span></p> </td> <td width=86 valign=top style='width:64.2pt;border-top:none;border-left:none; border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>1<o:p></o:p></span></p> </td> <td width=98 valign=top style='width:73.4pt;border-top:none;border-left:none; border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>.561<sup>a</sup><o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:2;page-break-inside:avoid'> <td width=133 valign=top style='width:99.45pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>sig. (2-tailed)<o:p></o:p></span></p> </td> <td width=86 style='width:64.2pt;border-top:none;border-left:none;border-bottom: solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;mso-border-top-alt: solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt;mso-border-alt: solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> </td> <td width=98 valign=top style='width:73.4pt;border-top:none;border-left:none; border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>.001<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:3;page-break-inside:avoid'> <td width=133 valign=top style='width:99.45pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><i style='mso-bidi-font-style:normal'><span lang=en-ca style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin'>n</span></i><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'><o:p></o:p></span></p> </td> <td width=86 valign=top style='width:64.2pt;border-top:none;border-left:none; border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>32<o:p></o:p></span></p> </td> <td width=98 valign=top style='width:73.4pt;border-top:none;border-left:none; border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>32<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:4;page-break-inside:avoid'> <td width=163 rowspan=3 valign=top style='width:1.7in;border:solid windowtext 1.0pt; border-top:none;mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>m-ilas-ed<o:p></o:p></span></p> </td> <td width=133 valign=top style='width:99.45pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>pearson correlation<o:p></o:p></span></p> </td> <td width=86 valign=top style='width:64.2pt;border-top:none;border-left:none; border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>.561<sup>a</sup><o:p></o:p></span></p> </td> <td width=98 valign=top style='width:73.4pt;border-top:none;border-left:none; border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>1<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:5;page-break-inside:avoid'> <td width=133 valign=top style='width:99.45pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>sig. (2-tailed)<o:p></o:p></span></p> </td> <td width=86 valign=top style='width:64.2pt;border-top:none;border-left:none; border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>.001<o:p></o:p></span></p> </td> <td width=98 style='width:73.4pt;border-top:none;border-left:none;border-bottom: solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;mso-border-top-alt: solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt;mso-border-alt: solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:6;mso-yfti-lastrow:yes;page-break-inside:avoid'> <td width=133 valign=top style='width:99.45pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><i style='mso-bidi-font-style:normal'><span lang=en-ca style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin'>n</span></i><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'><o:p></o:p></span></p> </td> <td width=86 valign=top style='width:64.2pt;border-top:none;border-left:none; border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>32<o:p></o:p></span></p> </td> <td width=98 valign=top style='width:73.4pt;border-top:none;border-left:none; border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;background:white;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>32<o:p></o:p></span></p> </td> </tr> </table> <p class=msonormal><sup><span lang=en-ca style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'>a</span></sup><span lang=en-ca style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'>correlation is significant at the 0.01 level (2-tailed).</span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p></o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style: normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></i></b></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style: normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>spring 2018<o:p></o:p></span></i></b></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>of the 19 respondents, 14 (74%) were female and 5 (26%) were male. the majority of respondents were white (n =13, 68%). three respondents (16%) identified themselves as black or african american, 1 respondent (5%) identified as “hispanic of any race,” 1 respondent (5%) selected american indian or alaskan native, and 1 respondent (5%) preferred not to answer. the largest percentage of respondents (n = 7, 37%) were between 25 and 29 years of age, 2 respondents (11%) were between 20 and 24 years of age, 2 respondents (11%) were between 30 and 34 years of age, 2 respondents (11%) were between 35 and 39 years of age, 3 respondents (16%) were between 40 and 44 years of age, and 3 respondents (16%) were between 45 and 49 years of age.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>of the 19 respondents, 9 (47%) held a bachelor’s as their highest degree, 8 (42%) held a master’s degree as their highest degree, and 2 (11%) held a doctoral degree. over half of respondents (<i style='mso-bidi-font-style:normal'>n </i>= 10, 53%) earned their most recent degree within the last 5 years, almost a third (<i style='mso-bidi-font-style:normal'>n</i> = 6, 32%) had earned their most recent degree within the last 6 to 10 years, 2 respondents (11%) within 11 to 15 years, and 1 respondent (5%) 21years ago or more. <o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>table 6 reports the mean spil-q and ila scores. the highest possible spil-q score was 30; the highest possible ila score was 60.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>table 6<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>spring 2018 spil-q and ila scores<sup>a</sup><o:p></o:p></span></p> <table class=msonormaltable border=1 cellspacing=0 cellpadding=0 width=0 style='width:231.4pt;border-collapse:collapse;border:none;mso-border-alt:solid windowtext .5pt; mso-yfti-tbllook:1184;mso-padding-alt:2.15pt 2.15pt 2.15pt 2.15pt;mso-border-insideh: .5pt solid windowtext;mso-border-insidev:.5pt solid windowtext'> <tr style='mso-yfti-irow:0;mso-yfti-firstrow:yes;page-break-inside:avoid; height:13.8pt'> <td valign=bottom style='border:solid windowtext 1.0pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:13.8pt'> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> </td> <td valign=bottom style='border:solid windowtext 1.0pt;border-left:none; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:13.8pt'> <p class=msonormal align=center style='text-align:center;mso-layout-grid-align: none;text-autospace:none'><i style='mso-bidi-font-style:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>n</span></i><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p></o:p></span></p> </td> <td valign=bottom style='border:solid windowtext 1.0pt;border-left:none; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:13.8pt'> <p class=msonormal align=center style='text-align:center;mso-layout-grid-align: none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin'>minimum<o:p></o:p></span></p> </td> <td valign=bottom style='border:solid windowtext 1.0pt;border-left:none; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:13.8pt'> <p class=msonormal align=center style='text-align:center;mso-layout-grid-align: none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin'>maximum<o:p></o:p></span></p> </td> <td valign=bottom style='border:solid windowtext 1.0pt;border-left:none; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:13.8pt'> <p class=msonormal align=center style='text-align:center;mso-layout-grid-align: none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin'>mean<o:p></o:p></span></p> </td> <td valign=bottom style='border:solid windowtext 1.0pt;border-left:none; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:13.8pt'> <p class=msonormal align=center style='text-align:center;mso-layout-grid-align: none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin'>sd<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:1;page-break-inside:avoid;height:29.2pt'> <td valign=top style='border:solid windowtext 1.0pt;border-top:none; mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:29.2pt'> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>spil-q total<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:29.2pt'> <p class=msonormal align=center style='text-align:center;mso-layout-grid-align: none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin'>19<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:29.2pt'> <p class=msonormal align=center style='text-align:center;mso-layout-grid-align: none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin'>12<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:29.2pt'> <p class=msonormal align=center style='text-align:center;mso-layout-grid-align: none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin'>30<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:29.2pt'> <p class=msonormal align=center style='text-align:center;mso-layout-grid-align: none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin'>24.53 (82%)<o:p></o:p></span></p> </td> <td valign=bottom style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:29.2pt'> <p class=msonormal align=center style='text-align:center;mso-layout-grid-align: none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin'>5.23<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:2;mso-yfti-lastrow:yes;page-break-inside:avoid; height:28.4pt'> <td valign=top style='border:solid windowtext 1.0pt;border-top:none; mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:28.4pt'> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>ila total<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:28.4pt'> <p class=msonormal align=center style='text-align:center;mso-layout-grid-align: none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin'>19<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:28.4pt'> <p class=msonormal align=center style='text-align:center;mso-layout-grid-align: none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin'>37.33<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:28.4pt'> <p class=msonormal align=center style='text-align:center;mso-layout-grid-align: none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin'>56.49<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:28.4pt'> <p class=msonormal align=center style='text-align:center;mso-layout-grid-align: none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin'>49.59 (83%)<o:p></o:p></span></p> </td> <td valign=bottom style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:28.4pt'> <p class=msonormal align=center style='text-align:center;mso-layout-grid-align: none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin'>5.42<o:p></o:p></span></p> </td> </tr> </table> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><sup><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>a</span></sup><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>all numbers are rounded to the nearest hundredth. <o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>mean scores were calculated in each of the six tested areas for both the spil-q and ila (tables 7 and 8).<o:p></o:p></span></p> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>the author used spss to calculate pearson’s <i style='mso-bidi-font-style:normal'>r</i> to determine if a correlation existed between spil-q and ila scores (table 9). the results indicate a moderate positive correlation that is statistically significant (<i style='mso-bidi-font-style:normal'>p</i> &lt; .005).<o:p></o:p></span></p> </div> <span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"times new roman";mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca;mso-fareast-language: en-us;mso-bidi-language:ar-sa'><br clear=all style='page-break-before:auto; mso-break-type:section-break'> </span> <div class=wordsection2> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>table 7<o:p></o:p></span></p> <p class=msonormal style='mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>spring 2018 spil-q scores (<i style='mso-bidi-font-style:normal'>n</i> = 19) by subcategory<sup> a</sup><o:p></o:p></span></p> <table class=msonormaltable border=1 cellspacing=0 cellpadding=0 width=0 style='width:326.15pt;border-collapse:collapse;border:none;mso-border-alt: solid windowtext .5pt;mso-yfti-tbllook:1184;mso-padding-alt:2.15pt 2.15pt 2.15pt 2.15pt; mso-border-insideh:.5pt solid windowtext;mso-border-insidev:.5pt solid windowtext'> <tr style='mso-yfti-irow:0;mso-yfti-firstrow:yes;page-break-inside:avoid; height:31.3pt'> <td valign=bottom style='border:solid windowtext 1.0pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:31.3pt'> <p class=msolistparagraphcxspfirst align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>spil-q subcategory<o:p></o:p></span></p> </td> <td valign=bottom style='border:solid windowtext 1.0pt;border-left:none; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:31.3pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>minimum<o:p></o:p></span></p> </td> <td valign=bottom style='border:solid windowtext 1.0pt;border-left:none; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:31.3pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:12.95pt; mso-add-space:auto;text-align:center;text-indent:-12.95pt;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-ca'>maximum<o:p></o:p></span></p> </td> <td width=103 valign=bottom style='width:77.5pt;border:solid windowtext 1.0pt; border-left:none;mso-border-left-alt:solid windowtext .5pt;mso-border-alt: solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt;height:31.3pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>mean<o:p></o:p></span></p> </td> <td width=50 valign=bottom style='width:37.45pt;border:solid windowtext 1.0pt; border-left:none;mso-border-left-alt:solid windowtext .5pt;mso-border-alt: solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt;height:31.3pt'> <p class=msolistparagraphcxsplast align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>sd<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:1;page-break-inside:avoid;height:18.95pt'> <td valign=top style='border:solid windowtext 1.0pt;border-top:none; mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:18.95pt'> <p class=msolistparagraphcxspfirst style='margin-left:0in;mso-add-space:auto; line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin;mso-ansi-language:en-ca'>develop a topic<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:18.95pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>2<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:18.95pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:12.95pt; mso-add-space:auto;text-align:center;text-indent:-12.95pt;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-ca'>5<o:p></o:p></span></p> </td> <td width=103 valign=top style='width:77.5pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:18.95pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>4.05 (81%)<o:p></o:p></span></p> </td> <td width=50 valign=top style='width:37.45pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:18.95pt'> <p class=msolistparagraphcxsplast align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>1.08<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:2;page-break-inside:avoid;height:22.1pt'> <td valign=top style='border:solid windowtext 1.0pt;border-top:none; mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:22.1pt'> <p class=msolistparagraphcxspfirst style='margin-left:0in;mso-add-space:auto; line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin;mso-ansi-language:en-ca'>locate<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:22.1pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>1<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:22.1pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:12.95pt; mso-add-space:auto;text-align:center;text-indent:-12.95pt;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-ca'>5<o:p></o:p></span></p> </td> <td width=103 valign=top style='width:77.5pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:22.1pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>4.05 (81%)<o:p></o:p></span></p> </td> <td width=50 valign=top style='width:37.45pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:22.1pt'> <p class=msolistparagraphcxsplast align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>1.18<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:3;page-break-inside:avoid;height:16.25pt'> <td valign=top style='border:solid windowtext 1.0pt;border-top:none; mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:16.25pt'> <p class=msolistparagraphcxspfirst style='margin-left:0in;mso-add-space:auto; line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin;mso-ansi-language:en-ca'>access<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:16.25pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>2<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:16.25pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:12.95pt; mso-add-space:auto;text-align:center;text-indent:-12.95pt;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-ca'>5<o:p></o:p></span></p> </td> <td width=103 valign=top style='width:77.5pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:16.25pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>4.05 (81%)<o:p></o:p></span></p> </td> <td width=50 valign=top style='width:37.45pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:16.25pt'> <p class=msolistparagraphcxsplast align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>.91<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:4;page-break-inside:avoid;height:31.3pt'> <td valign=top style='border:solid windowtext 1.0pt;border-top:none; mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:31.3pt'> <p class=msolistparagraphcxspfirst style='margin-left:0in;mso-add-space:auto; line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin;mso-ansi-language:en-ca'>evaluate<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:31.3pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>2<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:31.3pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:12.95pt; mso-add-space:auto;text-align:center;text-indent:-12.95pt;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-ca'>5<o:p></o:p></span></p> </td> <td width=103 valign=top style='width:77.5pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:31.3pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>4.16 (83%)<o:p></o:p></span></p> </td> <td width=50 valign=top style='width:37.45pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:31.3pt'> <p class=msolistparagraphcxsplast align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>1.02<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:5;page-break-inside:avoid;height:24.35pt'> <td valign=top style='border:solid windowtext 1.0pt;border-top:none; mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:24.35pt'> <p class=msolistparagraphcxspfirst style='margin-left:0in;mso-add-space:auto; line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin;mso-ansi-language:en-ca'>write<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:24.35pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>2<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:24.35pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:12.95pt; mso-add-space:auto;text-align:center;text-indent:-12.95pt;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-ca'>5<o:p></o:p></span></p> </td> <td width=103 valign=top style='width:77.5pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:24.35pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>4.11 (82%)<o:p></o:p></span></p> </td> <td width=50 valign=top style='width:37.45pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:24.35pt'> <p class=msolistparagraphcxsplast align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>.94<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:6;mso-yfti-lastrow:yes;page-break-inside:avoid; height:19.85pt'> <td valign=top style='border:solid windowtext 1.0pt;border-top:none; mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:19.85pt'> <p class=msolistparagraphcxspfirst style='margin-left:0in;mso-add-space:auto; line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin;mso-ansi-language:en-ca'>cite<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:19.85pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>2<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:19.85pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:12.95pt; mso-add-space:auto;text-align:center;text-indent:-12.95pt;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-ca'>5<o:p></o:p></span></p> </td> <td width=103 valign=top style='width:77.5pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:19.85pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>4.11 (82%)<o:p></o:p></span></p> </td> <td width=50 valign=top style='width:37.45pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:19.85pt'> <p class=msolistparagraphcxsplast align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>.99<o:p></o:p></span></p> </td> </tr> </table> <p class=msonormal><sup><span lang=en-ca style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'>a</span></sup><span lang=en-ca style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'>numbers are rounded to the nearest hundredth; percentages are rounded to the nearest percent.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>table 8<o:p></o:p></span></p> <p class=msonormal style='mso-outline-level:1;mso-layout-grid-align:none; text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>spring 2018 ila scores (<i style='mso-bidi-font-style:normal'>n</i> = 19) by subcategory<sup> a</sup><o:p></o:p></span></p> <table class=msonormaltable border=1 cellspacing=0 cellpadding=0 width=0 style='width:341.3pt;border-collapse:collapse;border:none;mso-border-alt:solid windowtext .5pt; mso-yfti-tbllook:1184;mso-padding-alt:2.15pt 2.15pt 2.15pt 2.15pt;mso-border-insideh: .5pt solid windowtext;mso-border-insidev:.5pt solid windowtext'> <tr style='mso-yfti-irow:0;mso-yfti-firstrow:yes;page-break-inside:avoid; height:25.15pt'> <td valign=bottom style='border:solid windowtext 1.0pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:25.15pt'> <p class=msolistparagraphcxspfirst style='margin-left:0in;mso-add-space:auto; line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin;mso-ansi-language:en-ca'>ila subcategory<o:p></o:p></span></p> </td> <td valign=bottom style='border:solid windowtext 1.0pt;border-left:none; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:25.15pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>minimum<o:p></o:p></span></p> </td> <td valign=bottom style='border:solid windowtext 1.0pt;border-left:none; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:25.15pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:12.95pt; mso-add-space:auto;text-align:center;text-indent:-12.95pt;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-ca'>maximum<o:p></o:p></span></p> </td> <td width=114 valign=bottom style='width:85.7pt;border:solid windowtext 1.0pt; border-left:none;mso-border-left-alt:solid windowtext .5pt;mso-border-alt: solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt;height:25.15pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>mean<o:p></o:p></span></p> </td> <td width=41 valign=bottom style='width:30.55pt;border:solid windowtext 1.0pt; border-left:none;mso-border-left-alt:solid windowtext .5pt;mso-border-alt: solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt;height:25.15pt'> <p class=msolistparagraphcxsplast align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>sd<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:1;page-break-inside:avoid;height:24.35pt'> <td valign=top style='border:solid windowtext 1.0pt;border-top:none; mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:24.35pt'> <p class=msolistparagraphcxspfirst style='margin-left:0in;mso-add-space:auto; line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin;mso-ansi-language:en-ca'> develop a topic<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:24.35pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>5<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:24.35pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:12.95pt; mso-add-space:auto;text-align:center;text-indent:-12.95pt;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-ca'>10<o:p></o:p></span></p> </td> <td width=114 valign=top style='width:85.7pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:24.35pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>8.26 (83%)<o:p></o:p></span></p> </td> <td width=41 valign=top style='width:30.55pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:24.35pt'> <p class=msolistparagraphcxsplast align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>1.52<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:2;page-break-inside:avoid;height:25.15pt'> <td valign=top style='border:solid windowtext 1.0pt;border-top:none; mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:25.15pt'> <p class=msolistparagraphcxspfirst style='margin-left:0in;mso-add-space:auto; line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin;mso-ansi-language:en-ca'> locate<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:25.15pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>5<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:25.15pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:12.95pt; mso-add-space:auto;text-align:center;text-indent:-12.95pt;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-ca'>10<o:p></o:p></span></p> </td> <td width=114 valign=top style='width:85.7pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:25.15pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>7.53 (75%)<o:p></o:p></span></p> </td> <td width=41 valign=top style='width:30.55pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:25.15pt'> <p class=msolistparagraphcxsplast align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>1.12<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:3;page-break-inside:avoid;height:25.15pt'> <td valign=top style='border:solid windowtext 1.0pt;border-top:none; mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:25.15pt'> <p class=msolistparagraphcxspfirst style='margin-left:0in;mso-add-space:auto; line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin;mso-ansi-language:en-ca'> access <o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:25.15pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>2.84<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:25.15pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:12.95pt; mso-add-space:auto;text-align:center;text-indent:-12.95pt;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-ca'>10<o:p></o:p></span></p> </td> <td width=114 valign=top style='width:85.7pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:25.15pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>7.68 (77%)<o:p></o:p></span></p> </td> <td width=41 valign=top style='width:30.55pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:25.15pt'> <p class=msolistparagraphcxsplast align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>2.04<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:4;page-break-inside:avoid;height:25.15pt'> <td valign=top style='border:solid windowtext 1.0pt;border-top:none; mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:25.15pt'> <p class=msolistparagraphcxspfirst style='margin-left:0in;mso-add-space:auto; line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin;mso-ansi-language:en-ca'> evaluate<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:25.15pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>5.5<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:25.15pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:12.95pt; mso-add-space:auto;text-align:center;text-indent:-12.95pt;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-ca'>10<o:p></o:p></span></p> </td> <td width=114 valign=top style='width:85.7pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:25.15pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>7.87 (79%)<o:p></o:p></span></p> </td> <td width=41 valign=top style='width:30.55pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:25.15pt'> <p class=msolistparagraphcxsplast align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>1.25<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:5;page-break-inside:avoid;height:24.35pt'> <td valign=top style='border:solid windowtext 1.0pt;border-top:none; mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:24.35pt'> <p class=msolistparagraphcxspfirst style='margin-left:0in;mso-add-space:auto; line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin;mso-ansi-language:en-ca'> write<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:24.35pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>6.5<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:24.35pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:12.95pt; mso-add-space:auto;text-align:center;text-indent:-12.95pt;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-ca'>10<o:p></o:p></span></p> </td> <td width=114 valign=top style='width:85.7pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:24.35pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>9.26 (93%)<o:p></o:p></span></p> </td> <td width=41 valign=top style='width:30.55pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:24.35pt'> <p class=msolistparagraphcxsplast align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>.96<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:6;mso-yfti-lastrow:yes;page-break-inside:avoid; height:25.15pt'> <td valign=top style='border:solid windowtext 1.0pt;border-top:none; mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:25.15pt'> <p class=msolistparagraphcxspfirst style='margin-left:0in;mso-add-space:auto; line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin;mso-ansi-language:en-ca'> cite<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:25.15pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>7<o:p></o:p></span></p> </td> <td valign=top style='border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt; border-right:solid windowtext 1.0pt;mso-border-top-alt:solid windowtext .5pt; mso-border-left-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt;height:25.15pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:12.95pt; mso-add-space:auto;text-align:center;text-indent:-12.95pt;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-ca'>10<o:p></o:p></span></p> </td> <td width=114 valign=top style='width:85.7pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:25.15pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>9.00 (90%)<o:p></o:p></span></p> </td> <td width=41 valign=top style='width:30.55pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt; height:25.15pt'> <p class=msolistparagraphcxsplast align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>1.00<o:p></o:p></span></p> </td> </tr> </table> <p class=msonormal><sup><span lang=en-ca style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'>a</span></sup><span lang=en-ca style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'>numbers are rounded to the nearest hundredth; percentages are rounded to the nearest percent. <o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>table 9<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>correlation between spil-q and ila scores<o:p></o:p></span></p> <table class=msonormaltable border=1 cellspacing=0 cellpadding=0 width=0 style='width:328.5pt;border-collapse:collapse;border:none;mso-border-alt:solid windowtext .5pt; mso-yfti-tbllook:1184;mso-padding-alt:2.15pt 2.15pt 2.15pt 2.15pt;mso-border-insideh: .5pt solid windowtext;mso-border-insidev:.5pt solid windowtext'> <tr style='mso-yfti-irow:0;mso-yfti-firstrow:yes;page-break-inside:avoid'> <td width=240 colspan=2 valign=bottom style='width:2.5in;border:solid windowtext 1.0pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msolistparagraphcxspfirst style='line-height:normal;page-break-after: avoid'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-ca'><o:p>&nbsp;</o:p></span></p> </td> <td width=84 valign=bottom style='width:63.0pt;border:solid windowtext 1.0pt; border-left:none;mso-border-left-alt:solid windowtext .5pt;mso-border-alt: solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal;page-break-after: avoid'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-ca'>spil-q<o:p></o:p></span></p> </td> <td width=114 valign=bottom style='width:85.5pt;border:solid windowtext 1.0pt; border-left:none;mso-border-left-alt:solid windowtext .5pt;mso-border-alt: solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msolistparagraphcxsplast align=center style='margin-top:0in; margin-right:-.75pt;margin-bottom:10.0pt;margin-left:.05pt;mso-add-space: auto;text-align:center;text-indent:-.05pt;line-height:normal;page-break-after: avoid'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-ca'>ila <o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:1;page-break-inside:avoid'> <td width=83 rowspan=3 valign=top style='width:61.9pt;border:solid windowtext 1.0pt; border-top:none;mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msolistparagraphcxspfirst style='margin-left:0in;mso-add-space:auto; text-align:justify;line-height:normal;page-break-after:avoid'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-ca'>spil-q<o:p></o:p></span></p> </td> <td width=157 valign=top style='width:118.1pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msolistparagraphcxspmiddle style='text-indent:-.5in;line-height: normal;page-break-after:avoid'><span lang=en-ca style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>pearson correlation<o:p></o:p></span></p> </td> <td width=84 valign=top style='width:63.0pt;border-top:none;border-left:none; border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal;page-break-after: avoid'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-ca'>1<o:p></o:p></span></p> </td> <td width=114 valign=top style='width:85.5pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msolistparagraphcxsplast align=center style='margin-top:0in; margin-right:-.7pt;margin-bottom:10.0pt;margin-left:0in;mso-add-space:auto; text-align:center;line-height:normal;page-break-after:avoid'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-ca'>.668<sup>a</sup><o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:2;page-break-inside:avoid'> <td width=157 valign=top style='width:118.1pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msolistparagraphcxspfirst style='text-indent:-.5in;line-height:normal; page-break-after:avoid'><span lang=en-ca style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin;mso-ansi-language:en-ca'>sig. (2-tailed)<o:p></o:p></span></p> </td> <td width=84 style='width:63.0pt;border-top:none;border-left:none;border-bottom: solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;mso-border-top-alt: solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt;mso-border-alt: solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal;page-break-after: avoid'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-ca'><o:p>&nbsp;</o:p></span></p> </td> <td width=114 valign=top style='width:85.5pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msolistparagraphcxsplast align=center style='margin-top:0in; margin-right:-.7pt;margin-bottom:10.0pt;margin-left:0in;mso-add-space:auto; text-align:center;line-height:normal;page-break-after:avoid'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-ca'>.002<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:3;page-break-inside:avoid'> <td width=157 valign=top style='width:118.1pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msolistparagraphcxspfirst style='text-indent:-.5in;line-height:normal; page-break-after:avoid'><i style='mso-bidi-font-style:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-ca'>n</span></i><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-ca'><o:p></o:p></span></p> </td> <td width=84 valign=top style='width:63.0pt;border-top:none;border-left:none; border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal;page-break-after: avoid'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-ca'>19<o:p></o:p></span></p> </td> <td width=114 valign=top style='width:85.5pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msolistparagraphcxsplast align=center style='margin-top:0in; margin-right:-.75pt;margin-bottom:10.0pt;margin-left:.05pt;mso-add-space: auto;text-align:center;line-height:normal;page-break-after:avoid'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-ca'>19<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:4;page-break-inside:avoid'> <td width=83 rowspan=3 valign=top style='width:61.9pt;border:solid windowtext 1.0pt; border-top:none;mso-border-top-alt:solid windowtext .5pt;mso-border-alt:solid windowtext .5pt; padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msolistparagraphcxspfirst style='margin-left:0in;mso-add-space:auto; text-align:justify;line-height:normal'><span lang=en-ca style='font-size: 10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>ila <o:p></o:p></span></p> </td> <td width=157 valign=top style='width:118.1pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msolistparagraphcxspmiddle style='text-indent:-.5in;line-height: normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-ca'>pearson correlation<o:p></o:p></span></p> </td> <td width=84 valign=top style='width:63.0pt;border-top:none;border-left:none; border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>.668<sup>a</sup><o:p></o:p></span></p> </td> <td width=114 valign=top style='width:85.5pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msolistparagraphcxsplast align=center style='margin-top:0in; margin-right:-.75pt;margin-bottom:10.0pt;margin-left:.05pt;mso-add-space: auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size: 10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>1<o:p></o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:5;page-break-inside:avoid'> <td width=157 valign=top style='width:118.1pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msolistparagraphcxspfirst style='text-indent:-.5in;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-ca'>sig. (2-tailed)<o:p></o:p></span></p> </td> <td width=84 valign=top style='width:63.0pt;border-top:none;border-left:none; border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>.002<o:p></o:p></span></p> </td> <td width=114 style='width:85.5pt;border-top:none;border-left:none; border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msolistparagraphcxsplast align=center style='margin-top:0in; margin-right:-.75pt;margin-bottom:10.0pt;margin-left:.05pt;mso-add-space: auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size: 10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'><o:p>&nbsp;</o:p></span></p> </td> </tr> <tr style='mso-yfti-irow:6;mso-yfti-lastrow:yes;page-break-inside:avoid'> <td width=157 valign=top style='width:118.1pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msolistparagraphcxspfirst style='text-indent:-.5in;line-height:normal'><i style='mso-bidi-font-style:normal'><span lang=en-ca style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>n</span></i><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language: en-ca'><o:p></o:p></span></p> </td> <td width=84 valign=top style='width:63.0pt;border-top:none;border-left:none; border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msolistparagraphcxspmiddle align=center style='margin-left:0in; mso-add-space:auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>19<o:p></o:p></span></p> </td> <td width=114 valign=top style='width:85.5pt;border-top:none;border-left: none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt; mso-border-top-alt:solid windowtext .5pt;mso-border-left-alt:solid windowtext .5pt; mso-border-alt:solid windowtext .5pt;padding:2.15pt 2.15pt 2.15pt 2.15pt'> <p class=msolistparagraphcxsplast align=center style='margin-top:0in; margin-right:-.75pt;margin-bottom:10.0pt;margin-left:.05pt;mso-add-space: auto;text-align:center;line-height:normal'><span lang=en-ca style='font-size: 10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin;mso-ansi-language:en-ca'>19<o:p></o:p></span></p> </td> </tr> </table> <p class=msonormal><sup><span lang=en-ca style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'>a</span></sup><span lang=en-ca style='font-size:10.0pt;font-family: "palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'>correlation is significant at the 0.01 level (2-tailed).</span><span lang=en-ca style='mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></b></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>discussion<o:p></o:p></span></b></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style: normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>research question 1: self-efficacy<o:p></o:p></span></i></b></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style: normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></i></b></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>michalak and rysavy </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1080/08963568.2016.1145787&quot;, &quot;issn&quot; : &quot;08963568&quot;, &quot;abstract&quot; : &quot;in 2015, the information literacy (il) team designed an instrument to determine international graduate business students' perceptions of their il skills and their actual test-assessed il skills. the purpose of this research was to compare international graduate students' perceptions of their il skills versus their test-assessed competencies, with the aim of creating training modules to improve test-identified deficiencies. results demonstrate that wide discrepancies exist between students' perceptions of their own skills and their actual skills on all six ila-topic areas assessed by the internally developed test instruments, with students overestimating their abilities in all areas. [abstract from publisher]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;michalak&quot;, &quot;given&quot; : &quot;russell&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;rysavy&quot;, &quot;given&quot; : &quot;monica&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of business &amp; finance librarianship&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;2&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2016&quot;, &quot;4&quot; ] ] }, &quot;note&quot; : &quot;accession number: 116263533; michalak, russell 1; rysavy, monica d. t. 1; affiliations: 1 : goldey-beacom college, wilmington, delaware, usa; source info: apr-jun2016, vol. 21 issue 2, p152; thesaurus term: information literacy -research; thesaurus term: business information services; subject term: instructional systems design; subject term: international business enterprises; subject term: graduate students; author-supplied keyword: business information literacy; author-supplied keyword: information literacy assessment; author-supplied keyword: information literacy skills; author-supplied keyword: instructional design; author-supplied keyword: international business graduate students; number of pages: 23p; document type: article&quot;, &quot;page&quot; : &quot;152-174&quot;, &quot;title&quot; : &quot;information literacy in 2015: international graduate business students' perceptions of information literacy skills compared to test-assessed skills.&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;21&quot; }, &quot;suppress-author&quot; : 1, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=79e910db-f957-4128-857a-5ce4b0a15542&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(2016)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(2016)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(2016)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(2016)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'> defined students who felt they had adequate skills in an area as those who selected 4 or 5 (agree or strongly agree). in fall 2017, the only mean score above 4 was for evaluating information, although the mean scores for citing information (3.97) and locating information (3.81) were close to this cutoff. the mean score for students’ confidence in accessing information (3.56) suggests more ambivalence.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>the students in the spring 2018 cohort were more confident; the mean score for each area was above 4.0, indicating most students felt like they had adequate skills in all six areas. <o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>the majority of students in both cohorts felt like their skills were adequate, supporting the findings of pinto, fernandez-ramos, sanchez, and meneses </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1016/j.acalib.2012.08.006&quot;, &quot;isbn&quot; : &quot;00991333&quot;, &quot;abstract&quot; : &quot;the study was carried out with students of official doctoral programs of information science in four universities in spain and latin america with the purpose of finding out, through self-assessments, student perceptions of their own information competence. a survey was designed to determine self-perceptions of knowledge, skills and attitudes regarding information competence, and it was distributed to students of two universities in spain, one in cuba and one in mexico. student perceptions of the levels of their own information competence in terms of knowledge, skills and attitudes are, in general, high. nevertheless, despite being immersed in research processes that require these information competencies, they recognize that they are lacking in certain skills related tothe organization, evaluation and presentation of information. having a self-assessment that identifies those areas of information competence in which students consider themselves to be weak will be a significant help in planning activities designed to strengthen these areas in doctoral programs.&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;pinto&quot;, &quot;given&quot; : &quot;maria&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;fernandez-ramos&quot;, &quot;given&quot; : &quot;a&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;sanchez&quot;, &quot;given&quot; : &quot;g&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;meneses&quot;, &quot;given&quot; : &quot;g&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of academic librarianship&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;2&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2013&quot; ] ] }, &quot;page&quot; : &quot;144-154&quot;, &quot;title&quot; : &quot;information competence of doctoral students in information science in spain and latin america: a self-assessment&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;39&quot; }, &quot;suppress-author&quot; : 1, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=011b0714-3e8a-4c3c-a334-04ee76e35bae&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(2013)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(2013)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(2013)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(2013)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'> and saunders et al. </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.12783/issn.2328-2967/56/s1/9&quot;, &quot;issn&quot; : &quot;07485786&quot;, &quot;abstract&quot; : &quot;librarians are expected to be expert searchers, and developing information literacy skills to navigate the vast world of information is a focus of most library and information science (lis) programs. it is important to understand the information literacy and behaviors of lis students to see if they are employing the skills they will need to assist and educate their future patrons, yet there is relatively little research into the information literacy and behaviors of librarians and library students. using a survey on information behaviors, this study addresses this gap in the literature by examining how lis students in 18 countries search for, evaluate, and use information in various contexts, and on whom they rely for help. the results of this study demonstrate library science students' information literacy skills within an international context. faculty in lis programs will be interested to see how their students approach information problems, and might use this understanding to develop or strengthen courses and inform curricular decisions. [abstract from author]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;saunders&quot;, &quot;given&quot; : &quot;laura&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;kurbanoglu&quot;, &quot;given&quot; : &quot;serap&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;boustany&quot;, &quot;given&quot; : &quot;joumana&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;dogan&quot;, &quot;given&quot; : &quot;guleda&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;becker&quot;, &quot;given&quot; : &quot;peter&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;blumer&quot;, &quot;given&quot; : &quot;eliane&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;chowdhury&quot;, &quot;given&quot; : &quot;sudatta&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;dobreva&quot;, &quot;given&quot; : &quot;milena&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;gendina&quot;, &quot;given&quot; : &quot;natalia&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;grgic&quot;, &quot;given&quot; : &quot;ivana hebrang&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;haddow&quot;, &quot;given&quot; : &quot;gaby&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;koltay&quot;, &quot;given&quot; : &quot;tibor&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;kortelainen&quot;, &quot;given&quot; : &quot;terttu&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;krakowska&quot;, &quot;given&quot; : &quot;monika&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;majid&quot;, &quot;given&quot; : &quot;shaheen&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;mezhova&quot;, &quot;given&quot; : &quot;marina&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;repanovici&quot;, &quot;given&quot; : &quot;angela&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;rud\u017eioniene&quot;, &quot;given&quot; : &quot;jurgita&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;schneider&quot;, &quot;given&quot; : &quot;rene&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;terra&quot;, &quot;given&quot; : &quot;ana lucia&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of education for library &amp; information science&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2015&quot;, &quot;3&quot;, &quot;2&quot; ] ] }, &quot;note&quot; : &quot;accession number: 108403317; saunders, laura 1; email address: laura.saunders@simmons.edu; kurbanoglu, serap 2; email address: kurbanogluserap@gmail.com; boustany, joumana 3; email address: jboustany@gmail.com; dogan, guleda 2; email address: guledaduzyol@gmail.com; becker, peter 4; email address: p.g.becker@hhs.nl; blumer, eliane 5; email address: eliane.blumer@hesge.ch; chowdhury, sudatta 6; email address: sudatta@hotmail.co.uk; dobreva, milena 7; email address: milena.dobreva@gmail.com; gendina, natalia 8; email address: gendina@inbox.ru; grgic, ivana hebrang 9; email address: ihgrgic@ffzg.hr; haddow, gaby 10; email address: g.haddow@curtin.edu.au; koltay, tibor 11; email address: koltay.tibor@abk.szie.hu; kortelainen, terttu 12; email address: terttu.kortelainen@oulu.fi; krakowska, monika 13; email address: monika.krakowska@uj.edu.pl; majid, shaheen 14; mezhova, marina 15; email address: mezhova75@mail.ru; repanovici, angela 16; email address: arepanovici@gmail.com; rud\u017eioniene, jurgita 17; email address: jurgita.rudzioniene@kf.vu.lt; schneider, rene 5; email address: rene.schneider@hesge.ch; terra, ana lucia 18; email address: analuciaterra@yahoo.com; affiliations: 1 : school of library and information science, simmons college; 2 : department of information management, hacettepe university; 3 : department of information communication, universit\u00e9 paris descartes; 4 : hague university of applied sciences, netherlands; 5 : department of information science, haute ecole de gestion; 6 : department of computer &amp;amp; information sciences, university of strathclyde; 7 : library information and archive sciences department, malta university; 8 : scientific and research institute of information technologies, kemerovo state university of culture and arts kemerrovo; 9 : department of information and communication sciences, faculty of humanities and social sciences, university of zagreb; 10 : department of information studies, curtin university; 11 : department of information and library studies, szent istv\u00e1n university,; 12 : department of information studies, oulu university; 13 : department of management and social communication institute of library and information science, jagiellonian university; 14 : division of information studies, wee kim wee school of communication &amp;amp; information, nanyang technological university; 15 : scientific and research institute of information technologies in kemerovo state university of culture and arts kemerrovo; 16 : transilvania university of brasov; 17 : faculty of communication institute of library and information science, vilnius university; 18 : information science department, superior school of industrial studies and management, polytechnic institute of porto; source info: 2015 supplement 1, vol. 56, ps80; thesaurus term: library education; thesaurus term: research; thesaurus term: information literacy -research; thesaurus term: library school students; thesaurus term: library schools -curricula; subject term: academic programs; number of pages: 20p; illustrations: 3 charts, 5 graphs; document type: article; full text word count: 7907&quot;, &quot;page&quot; : &quot;s80-s99&quot;, &quot;publisher&quot; : &quot;association for library &amp; information science education&quot;, &quot;title&quot; : &quot;information behaviors and information literacy skills of lis students: an international perspective&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;56&quot; }, &quot;suppress-author&quot; : 1, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=1ea827af-d9ca-44f8-8664-0a731c11718c&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(2015)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(2015)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(2015)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(2015)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'> that lis students have positive il self-efficacy. <o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style: normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>research question 2: demonstrated il skills <o:p></o:p></span></i></b></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>in fall 2017, the mean m-ilas-ed scores in the four tested il skill areas showed that students performed best in citing, followed by locating, evaluating, and accessing information. <o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>mean scores present a general overview of skills, but item level analysis gives granular insight into the specific skills of the incoming students and indicates specific weaknesses. in fall 2017, at least half of the respondents incorrectly answered 7 of 18 questions. the seven items and percentage of students answering incorrectly follow: <o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msolistparagraphcxspfirst style='margin-bottom:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:-.25in;line-height:normal;mso-list:l15 level1 lfo13'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>72% were unable to identify the best source to locate a brief history and summary of a topic (ilas-ed, question 8).<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-bottom:0in;margin-bottom: .0001pt;mso-add-space:auto;text-indent:-.25in;line-height:normal;mso-list:l15 level1 lfo13'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>78% were unable to identify options offered in advanced search interfaces (ilas-ed, question 11).<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-bottom:0in;margin-bottom: .0001pt;mso-add-space:auto;text-indent:-.25in;line-height:normal;mso-list:l15 level1 lfo13'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>50% were unable to identify the best place to find recent scholarly articles in a particular subject (ilas-ed, question 13). <o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-bottom:0in;margin-bottom: .0001pt;mso-add-space:auto;text-indent:-.25in;line-height:normal;mso-list:l15 level1 lfo13'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>62% were unable to select the best set of synonyms and terms related to a concept (ilas-ed, question 15).<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-bottom:0in;margin-bottom: .0001pt;mso-add-space:auto;text-indent:-.25in;line-height:normal;mso-list:l15 level1 lfo13'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>66% respondents were unable to identify a citation for chapter in a book (ilas-ed, question 19).<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-bottom:0in;margin-bottom: .0001pt;mso-add-space:auto;text-indent:-.25in;line-height:normal;mso-list:l15 level1 lfo13'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>59% were unable to select the best way to locate a journal article using the library’s catalog (ilas-ed, question 20).<o:p></o:p></span></p> <p class=msolistparagraphcxsplast style='margin-bottom:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:-.25in;line-height:normal;mso-list:l15 level1 lfo13'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>59% were unable to able to determine the reliability of a story on the internet (ilas-ed, question 23).<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>five of the questions on which 50% or fewer respondents answered correctly had been modified (questions 8, 13, 15, 20, and 23 on the original ilas-ed). although the changes to the questions were minor (see appendix b), the possibility of poor adaptation may have contributed to the respondents’ lower performance. despite the modifications, the findings point to a gap in knowledge to some fundamental skills used in locating, accessing, and evaluating information.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>in spring 2018, the mean scores of entering mlis students in developing a topic; locating, accessing, and evaluating information; writing; and citing as measured by the ila instrument were all 75% or above (see table 8). however, looking at the results on the item level highlights weaknesses: <o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msolistparagraphcxspfirst style='margin-bottom:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:-.25in;line-height:normal;mso-list:l20 level1 lfo14'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>53% of respondents did not identify that information in a library is selected through a review process as the best description of what distinguishes the information in the library from information on the web </span><!--[if supportfields]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin; mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1080/08963568.2016.1145787&quot;, &quot;issn&quot; : &quot;08963568&quot;, &quot;abstract&quot; : &quot;in 2015, the information literacy (il) team designed an instrument to determine international graduate business students' perceptions of their il skills and their actual test-assessed il skills. the purpose of this research was to compare international graduate students' perceptions of their il skills versus their test-assessed competencies, with the aim of creating training modules to improve test-identified deficiencies. results demonstrate that wide discrepancies exist between students' perceptions of their own skills and their actual skills on all six ila-topic areas assessed by the internally developed test instruments, with students overestimating their abilities in all areas. [abstract from publisher]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;michalak&quot;, &quot;given&quot; : &quot;russell&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;rysavy&quot;, &quot;given&quot; : &quot;monica&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of business &amp; finance librarianship&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;2&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2016&quot;, &quot;4&quot; ] ] }, &quot;note&quot; : &quot;accession number: 116263533; michalak, russell 1; rysavy, monica d. t. 1; affiliations: 1 : goldey-beacom college, wilmington, delaware, usa; source info: apr-jun2016, vol. 21 issue 2, p152; thesaurus term: information literacy -research; thesaurus term: business information services; subject term: instructional systems design; subject term: international business enterprises; subject term: graduate students; author-supplied keyword: business information literacy; author-supplied keyword: information literacy assessment; author-supplied keyword: information literacy skills; author-supplied keyword: instructional design; author-supplied keyword: international business graduate students; number of pages: 23p; document type: article&quot;, &quot;page&quot; : &quot;152-174&quot;, &quot;title&quot; : &quot;information literacy in 2015: international graduate business students' perceptions of information literacy skills compared to test-assessed skills.&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;21&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=79e910db-f957-4128-857a-5ce4b0a15542&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(michalak &amp; rysavy, 2016)&quot;, &quot;manualformatting&quot; : &quot;(module 2, question 1)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(michalak &amp; rysavy, 2016)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(michalak &amp; rysavy, 2016)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(module 2, question 1)</span></span><!--[if supportfields]><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>. <o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-bottom:0in;margin-bottom: .0001pt;mso-add-space:auto;text-indent:-.25in;line-height:normal;mso-list:l20 level1 lfo14'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>47% of respondents were unable to identify the library of congress classification system as that most often used in major u.s. universities (module 2, question 8).<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-bottom:0in;margin-bottom: .0001pt;mso-add-space:auto;text-indent:-.25in;line-height:normal;mso-list:l20 level1 lfo14'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>58% of respondents did not know how to search for different endings of a word by using truncation </span><!--[if supportfields]><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1080/08963568.2016.1145787&quot;, &quot;issn&quot; : &quot;08963568&quot;, &quot;abstract&quot; : &quot;in 2015, the information literacy (il) team designed an instrument to determine international graduate business students' perceptions of their il skills and their actual test-assessed il skills. the purpose of this research was to compare international graduate students' perceptions of their il skills versus their test-assessed competencies, with the aim of creating training modules to improve test-identified deficiencies. results demonstrate that wide discrepancies exist between students' perceptions of their own skills and their actual skills on all six ila-topic areas assessed by the internally developed test instruments, with students overestimating their abilities in all areas. [abstract from publisher]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;michalak&quot;, &quot;given&quot; : &quot;russell&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;rysavy&quot;, &quot;given&quot; : &quot;monica&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of business &amp; finance librarianship&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;2&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2016&quot;, &quot;4&quot; ] ] }, &quot;note&quot; : &quot;accession number: 116263533; michalak, russell 1; rysavy, monica d. t. 1; affiliations: 1 : goldey-beacom college, wilmington, delaware, usa; source info: apr-jun2016, vol. 21 issue 2, p152; thesaurus term: information literacy -research; thesaurus term: business information services; subject term: instructional systems design; subject term: international business enterprises; subject term: graduate students; author-supplied keyword: business information literacy; author-supplied keyword: information literacy assessment; author-supplied keyword: information literacy skills; author-supplied keyword: instructional design; author-supplied keyword: international business graduate students; number of pages: 23p; document type: article&quot;, &quot;page&quot; : &quot;152-174&quot;, &quot;title&quot; : &quot;information literacy in 2015: international graduate business students' perceptions of information literacy skills compared to test-assessed skills.&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;21&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=79e910db-f957-4128-857a-5ce4b0a15542&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(michalak &amp; rysavy, 2016)&quot;, &quot;manualformatting&quot; : &quot;(module 3, question 3)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(michalak &amp; rysavy, 2016)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(michalak &amp; rysavy, 2016)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(module 3, question 3)</span></span><!--[if supportfields]><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>. <o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-bottom:0in;margin-bottom: .0001pt;mso-add-space:auto;text-indent:-.25in;line-height:normal;mso-list:l20 level1 lfo14'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>58% of respondents were unable to identify the least important action in evaluating a resource when writing about the history of a topic (module 4, question 2).<o:p></o:p></span></p> <p class=msolistparagraphcxsplast style='margin-bottom:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:-.25in;line-height:normal;mso-list:l20 level1 lfo14'><![if !supportlists]><span style='font-size:10.0pt;font-family:symbol;mso-fareast-font-family:symbol; mso-bidi-font-family:symbol'><span style='mso-list:ignore'>·<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>37% of respondents indicated that not every website needs to be evaluated before using information found on it (module 4, question10).<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>in both fall 2017 and spring 2018, a high percentage of respondents demonstrated a lack of basic knowledge and skills. these findings are similar to conway’s </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1080/00048623.2011.10722218&quot;, &quot;isbn&quot; : &quot;00048623&quot;, &quot;issn&quot; : &quot;00048623&quot;, &quot;abstract&quot; : &quot;this paper describes a study comparing the information literacy skills of commencing undergraduate ahttp://eds.a.ebscohost.com/eds/citations/fulltextlinkclick?sid=74f97223-0987-46b4-be5e-2e40a731670d@sessionmgr4005&amp;vid=1&amp;id=pdffulltextnd postgraduate information studies students at curtin university. the study was conducted by means of an online multiple choice survey, which students completed during the first few weeks of their course. the results show that a number of both undergraduates and postgraduates have problems with basic information literacy skills, particularly those related to the use of traditional library tools, such as library catalogues. the study found that the only factors associated with differences in performance were postgraduate and undergraduate status (the primary variable examined), age, and previous work experience. [abstract from author]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;conway&quot;, &quot;given&quot; : &quot;kate&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;australian academic &amp; research libraries&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;2&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2011&quot;, &quot;6&quot; ] ] }, &quot;note&quot; : &quot;from duplicate 1 (how prepared are students for postgraduate study? a comparison of the information literacy skills of commencing undergraduate and postgraduate information studies students at curtin university. conway, kate)\n\naccession number: 72101036; conway, kate 1; email address: kate.conway@health.wa.gov.au; affiliations: 1 : king edward memorial hospital, po box 134, subiaco wa 6904; source info: jun2011, vol. 42 issue 2, p121; thesaurus term: academic libraries -use studies; thesaurus term: information literacy; subject term: undergraduates; subject term: graduate students; number of pages: 15p; document type: article&quot;, &quot;page&quot; : &quot;121-135&quot;, &quot;title&quot; : &quot;how prepared are students for postgraduate study? a comparison of the information literacy skills of commencing undergraduate and postgraduate information studies students at curtin university&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;42&quot; }, &quot;suppress-author&quot; : 1, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=84111873-0ef2-43a8-892e-11381032a253&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(2011)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(2011)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(2011)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(2011)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'> experience with lis graduate students. librarians who serve lis graduate students cannot assume that incoming students possess skills and knowledge that are considered fundamental in the lis discipline. librarians should keep this gap in mind when constructing libguides and other resources for lis graduate students. if librarians are providing one-shot instruction or embedding in an lis graduate course, they may want to consider administering a pretest before designing instruction and activities so they can address gaps in knowledge and skills.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style: normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>research question 3: correlation<o:p></o:p></span></i></b></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>in the first phase of the study (fall 2017), there was a modest positive correlation between m-spil-q and m-ilad-ed scores (r = .561, p &lt; .005). there was also a moderate positive correlation (r = .668, p &lt; .005) between spil-q and ila scores in spring 2018, again indicating a possible positive correlation between perceived il self-efficacy and actual il skills. these moderate positive correlations echo the correlation between il self-efficacy and skills found by robertson &amp; felicilda-reynaldo’s </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.3928/01484834-20150218-03&quot;, &quot;issn&quot; : &quot;01484834&quot;, &quot;pmid&quot; : &quot;25692245&quot;, &quot;abstract&quot; : &quot;maintaining evidence-based nursing practice requires information literacy (il) skills that should be established prior to completing an undergraduate nursing degree. based on bandura's social cognitive theory, this cross-sectional descriptive correlational study assessed the perceived and applied il skills of graduate nursing students from two family nurse practitioner (fnp) programs in the midwestern united states. results showed that although the 26 newly admitted fnp students demonstrated a high level of confidence in their il skills, the students did not perform well in the actual il skills test. according to bandura, the students' confidence in their il knowledge should allow students to be engaged in course activities requiring il skills. nurse educators teaching in undergraduate or graduate programs are in key positions to incorporate il experiences into class activities to allow for skill assessment and further practice. further research is needed on nursing students' il self-efficacy and performance. [j nurs educ. 2015;54(3, suppl.):s26-s30.]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;robertson&quot;, &quot;given&quot; : &quot;d susie&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;felicilda-reynaldo&quot;, &quot;given&quot; : &quot;rhea faye d&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of nursing education&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;3, suppl&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2015&quot; ] ] }, &quot;note&quot; : &quot;accession number: 25692245. language: english. date created: 20150228. date completed: 20151215. update code: 20151215. publication type: journal article. journal id: 7705432. publication model: print-electronic. cited medium: internet. nlm iso abbr: j nurs educ. linking issn: 01484834. subset: im; n; date of electronic publication: 2015 feb 19. current imprints: publication: thorofare, n.j. : charles b. slack; original imprints: publication: new york, n.y. : blackiston division, mcgraw-hill, [c1962-&quot;, &quot;page&quot; : &quot;s26-30&quot;, &quot;publisher&quot; : &quot;charles b. slack&quot;, &quot;publisher-place&quot; : &quot;united states&quot;, &quot;title&quot; : &quot;evaluation of graduate nursing students' information literacy self-efficacy and applied skills&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;54&quot; }, &quot;suppress-author&quot; : 1, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=e598323d-e01e-4782-aa5a-317b3a02abb7&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(2015)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(2015)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(2015)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>(2015)</span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'> study of graduate nursing students.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>although there is a positive correlation between il self-efficacy and skills, there are indications of discrepancies between perceived and actual il skills. this study reveals specific examples of students misjudging their skill level. for example, in fall 2017, the mean m-spil-q score of 15.38 (sd 3.28) indicates that students were confident about their il skills, but the mean m-ilas-ed skill score, 9.5 (sd 2.578) out of a possible score of 18, indicates a low skill level (see table 2). in addition, in fall 2017, 84% of respondents rated their ability to evaluate information as adequate, but the mean score for demonstrated ability to evaluate information was 2.28/4.0 (57%) (see table 4). <o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>these discrepancies confirm the concern that lis graduate students overestimate their il skills put forth by pinto, fernandez-ramos, sanchez, and meneses </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1016/j.acalib.2012.08.006&quot;, &quot;isbn&quot; : &quot;00991333&quot;, &quot;abstract&quot; : &quot;the study was carried out with students of official doctoral programs of information science in four universities in spain and latin america with the purpose of finding out, through self-assessments, student perceptions of their own information competence. a survey was designed to determine self-perceptions of knowledge, skills and attitudes regarding information competence, and it was distributed to students of two universities in spain, one in cuba and one in mexico. student perceptions of the levels of their own information competence in terms of knowledge, skills and attitudes are, in general, high. nevertheless, despite being immersed in research processes that require these information competencies, they recognize that they are lacking in certain skills related tothe organization, evaluation and presentation of information. having a self-assessment that identifies those areas of information competence in which students consider themselves to be weak will be a significant help in planning activities designed to strengthen these areas in doctoral programs.&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;pinto&quot;, &quot;given&quot; : &quot;maria&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;fernandez-ramos&quot;, &quot;given&quot; : &quot;a&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;sanchez&quot;, &quot;given&quot; : &quot;g&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;meneses&quot;, &quot;given&quot; : &quot;g&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of academic librarianship&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;2&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2013&quot; ] ] }, &quot;page&quot; : &quot;144-154&quot;, &quot;title&quot; : &quot;information competence of doctoral students in information science in spain and latin america: a self-assessment&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;39&quot; }, &quot;suppress-author&quot; : 1, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=011b0714-3e8a-4c3c-a334-04ee76e35bae&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(2013)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(2013)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(2013)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(2013)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>, which was based on personal observations, and by saunders et al. </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.12783/issn.2328-2967/56/s1/9&quot;, &quot;issn&quot; : &quot;07485786&quot;, &quot;abstract&quot; : &quot;librarians are expected to be expert searchers, and developing information literacy skills to navigate the vast world of information is a focus of most library and information science (lis) programs. it is important to understand the information literacy and behaviors of lis students to see if they are employing the skills they will need to assist and educate their future patrons, yet there is relatively little research into the information literacy and behaviors of librarians and library students. using a survey on information behaviors, this study addresses this gap in the literature by examining how lis students in 18 countries search for, evaluate, and use information in various contexts, and on whom they rely for help. the results of this study demonstrate library science students' information literacy skills within an international context. faculty in lis programs will be interested to see how their students approach information problems, and might use this understanding to develop or strengthen courses and inform curricular decisions. [abstract from author]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;saunders&quot;, &quot;given&quot; : &quot;laura&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;kurbanoglu&quot;, &quot;given&quot; : &quot;serap&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;boustany&quot;, &quot;given&quot; : &quot;joumana&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;dogan&quot;, &quot;given&quot; : &quot;guleda&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;becker&quot;, &quot;given&quot; : &quot;peter&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;blumer&quot;, &quot;given&quot; : &quot;eliane&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;chowdhury&quot;, &quot;given&quot; : &quot;sudatta&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;dobreva&quot;, &quot;given&quot; : &quot;milena&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;gendina&quot;, &quot;given&quot; : &quot;natalia&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;grgic&quot;, &quot;given&quot; : &quot;ivana hebrang&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;haddow&quot;, &quot;given&quot; : &quot;gaby&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;koltay&quot;, &quot;given&quot; : &quot;tibor&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;kortelainen&quot;, &quot;given&quot; : &quot;terttu&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;krakowska&quot;, &quot;given&quot; : &quot;monika&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;majid&quot;, &quot;given&quot; : &quot;shaheen&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;mezhova&quot;, &quot;given&quot; : &quot;marina&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;repanovici&quot;, &quot;given&quot; : &quot;angela&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;rud\u017eioniene&quot;, &quot;given&quot; : &quot;jurgita&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;schneider&quot;, &quot;given&quot; : &quot;rene&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;terra&quot;, &quot;given&quot; : &quot;ana lucia&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of education for library &amp; information science&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2015&quot;, &quot;3&quot;, &quot;2&quot; ] ] }, &quot;note&quot; : &quot;accession number: 108403317; saunders, laura 1; email address: laura.saunders@simmons.edu; kurbanoglu, serap 2; email address: kurbanogluserap@gmail.com; boustany, joumana 3; email address: jboustany@gmail.com; dogan, guleda 2; email address: guledaduzyol@gmail.com; becker, peter 4; email address: p.g.becker@hhs.nl; blumer, eliane 5; email address: eliane.blumer@hesge.ch; chowdhury, sudatta 6; email address: sudatta@hotmail.co.uk; dobreva, milena 7; email address: milena.dobreva@gmail.com; gendina, natalia 8; email address: gendina@inbox.ru; grgic, ivana hebrang 9; email address: ihgrgic@ffzg.hr; haddow, gaby 10; email address: g.haddow@curtin.edu.au; koltay, tibor 11; email address: koltay.tibor@abk.szie.hu; kortelainen, terttu 12; email address: terttu.kortelainen@oulu.fi; krakowska, monika 13; email address: monika.krakowska@uj.edu.pl; majid, shaheen 14; mezhova, marina 15; email address: mezhova75@mail.ru; repanovici, angela 16; email address: arepanovici@gmail.com; rud\u017eioniene, jurgita 17; email address: jurgita.rudzioniene@kf.vu.lt; schneider, rene 5; email address: rene.schneider@hesge.ch; terra, ana lucia 18; email address: analuciaterra@yahoo.com; affiliations: 1 : school of library and information science, simmons college; 2 : department of information management, hacettepe university; 3 : department of information communication, universit\u00e9 paris descartes; 4 : hague university of applied sciences, netherlands; 5 : department of information science, haute ecole de gestion; 6 : department of computer &amp;amp; information sciences, university of strathclyde; 7 : library information and archive sciences department, malta university; 8 : scientific and research institute of information technologies, kemerovo state university of culture and arts kemerrovo; 9 : department of information and communication sciences, faculty of humanities and social sciences, university of zagreb; 10 : department of information studies, curtin university; 11 : department of information and library studies, szent istv\u00e1n university,; 12 : department of information studies, oulu university; 13 : department of management and social communication institute of library and information science, jagiellonian university; 14 : division of information studies, wee kim wee school of communication &amp;amp; information, nanyang technological university; 15 : scientific and research institute of information technologies in kemerovo state university of culture and arts kemerrovo; 16 : transilvania university of brasov; 17 : faculty of communication institute of library and information science, vilnius university; 18 : information science department, superior school of industrial studies and management, polytechnic institute of porto; source info: 2015 supplement 1, vol. 56, ps80; thesaurus term: library education; thesaurus term: research; thesaurus term: information literacy -research; thesaurus term: library school students; thesaurus term: library schools -curricula; subject term: academic programs; number of pages: 20p; illustrations: 3 charts, 5 graphs; document type: article; full text word count: 7907&quot;, &quot;page&quot; : &quot;s80-s99&quot;, &quot;publisher&quot; : &quot;association for library &amp; information science education&quot;, &quot;title&quot; : &quot;information behaviors and information literacy skills of lis students: an international perspective&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;56&quot; }, &quot;suppress-author&quot; : 1, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=1ea827af-d9ca-44f8-8664-0a731c11718c&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(2015)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(2015)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(2015)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(2015)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>, which was based on the self-reported information behaviors of lis graduate students. in both cases, the researchers did not have data about demonstrated il skills. when working with individual lis graduate students, practitioners should remember that a student’s skill level may not may measure up to the student’s confidence; librarians should probe to identify the student’s actual competence or knowledge instead of relying on the student’s self-reported understanding and ability.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style: normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>research limitations<o:p></o:p></span></i></b></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>the study’s most significant limitation is its small, self-selected sample size. a larger sample of students from multiple mlis programs across the country would yield more reliable data and generalizable results. <o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>the use of two different instruments in fall 2017 and fall 2018 introduced additional limitations. although using two different instruments gave the author insight into which test might be more suitable for large scale use, it did prevent the author from establishing a clear baselines of il self-efficacy and skills for mlis students, and although general trends could be identified, results between the two cohorts could not be directly compared.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>the timing of the survey in fall 2017 was problematic. to collect students’ answers before they were exposed to il instruction in graduate lis classes, the survey needed to be distributed at the beginning of the semester; however, the beginning of the semester corresponded to the catastrophic destruction caused by hurricane harvey. although most of hurricane harvey’s destruction was in texas, portions of louisiana also experienced flooding. slis’s mlis program in an online degree program, and students are scattered across the united states; however, many of them live in louisiana. some eligible students may have been affected directly, and those living in other areas of the state may have had family in devastated areas. there is no way to measure the effect that hurricane harvey had on the response rate or on respondents’ performance. it is impossible to quantify the emotional impact of the storm on those it affected either directly or indirectly. completing an optional survey would have been a low priority for affected students.<b style='mso-bidi-font-weight:normal'><o:p></o:p></b></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>funding limited the measurement instruments available for use. many standardized measurement instruments for il with rigorous testing for reliability and validity, such as the research readiness self-assessment (rrsa) </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1300/j111v41n01_13&quot;, &quot;abstract&quot; : &quot;summary. librarians and learning researchers at central michigan university collaboratively developed an online tool that assesses how student research attitudes and perceptions correlate to their actual re-search skills in order to educate them about state-of-the-art library re-sources and prepare them to write high-quality research papers. this article describes the reasons for developing the assessment as well as the design process and technical characteristics.&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;ivanitskaya&quot;, &quot;given&quot; : &quot;lana&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;laus&quot;, &quot;given&quot; : &quot;ryan&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;casey&quot;, &quot;given&quot; : &quot;anne marie&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of library administration&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;1/2&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2004&quot; ] ] }, &quot;page&quot; : &quot;167-183&quot;, &quot;title&quot; : &quot;research readiness self-assessment&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;41&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=c7b82240-2d70-30df-bc74-0c19acdd77dc&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(ivanitskaya, laus, &amp; casey, 2004)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(ivanitskaya, laus, &amp; casey, 2004)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(ivanitskaya, laus, &amp; casey, 2004)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><span style='mso-no-proof:yes'>(ivanitskaya, laus, &amp; casey, 2004)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'> and the standardized assessment of information literacy skills (sails) </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;isbn&quot; : &quot;978-1-59407-967-2&quot;, &quot;abstract&quot; : &quot;the institute of museum and library services (imls) has funded three major information literacy projects. this paper describes the projects and addresses these questions: what difference have those expenditures made? what more do we know about student information literacy? about assessment of information literacy? the projects are: \u2022 project sails\u2014the standardized assessment of information literacy skills \u2022 rails\u2014rubric assessment of information literacy skills \u2022 pil\u2014project information literacy&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;radcliff&quot;, &quot;given&quot; : &quot;carolyn&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;oakleaf&quot;, &quot;given&quot; : &quot;megan&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;hoeck&quot;, &quot;given&quot; : &quot;michele van hoeck&quot;, &quot;non-dropping-particle&quot; : &quot;van&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;proceedings of the 2014 library assessment conference: building effective, sustainable, practical assessment&quot;, &quot;editor&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;sarah durso&quot;, &quot;given&quot; : &quot;&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;steve hiller&quot;, &quot;given&quot; : &quot;&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;martha kyrillidou&quot;, &quot;given&quot; : &quot;&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;angela pappalarado&quot;, &quot;given&quot; : &quot;&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;202&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2014&quot; ] ] }, &quot;page&quot; : &quot;801-809&quot;, &quot;publisher&quot; : &quot;association of research libraries&quot;, &quot;publisher-place&quot; : &quot;seattle, wa&quot;, &quot;title&quot; : &quot;so what? the results and impact of a decade of imls-funded information literacy assessments&quot;, &quot;type&quot; : &quot;paper-conference&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=aae0bc8f-f6fd-3b43-ad5a-db58ce21940d&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(radcliff, oakleaf, &amp; van hoeck, 2014)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(radcliff, oakleaf, &amp; van hoeck, 2014)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(radcliff, oakleaf, &amp; van hoeck, 2014)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><span style='mso-no-proof:yes'>(radcliff, oakleaf, &amp; van hoeck, 2014)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'> are fee-based </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1002/ets2.12118&quot;, &quot;abstract&quot; : &quot;digital information literacy (dil)--generally defined as the ability to obtain, understand, evaluate, and use information in a variety of digital technology contexts--is a critically important skill deemed necessary for success in higher education as well as in the global networked economy. to determine whether college graduates possess the requisite knowledge and skills in dil, higher education institutions must be able to administer and use results from valid assessments of dil. in this paper, we provide a comprehensive review of existing definitions of this construct in major frameworks from higher education and the workforce and propose an operational definition of dil. next, we provide a review of existing assessments of information literacy and related constructs, including features of the assessments, construct alignment, and psychometric properties (i.e., reliability and validity evidence). finally, we discuss challenges and considerations surrounding the design, implementation, and use of next-generation assessments of dil. we offer this review as a resource for higher education institutions in selecting among existing assessments or in designing their own measures.&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;sparks&quot;, &quot;given&quot; : &quot;jesse r&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;katz&quot;, &quot;given&quot; : &quot;irvin r&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;beile&quot;, &quot;given&quot; : &quot;penny m&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;ets research report series&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2016&quot;, &quot;12&quot;, &quot;1&quot; ] ] }, &quot;note&quot; : &quot;accession number: ej1124778; acquisition information: educational testing service. rosedale road, ms19-r princeton, nj 08541. tel: 609-921-9000; fax: 609-734-5410; e-mail: rdweb@ets.org; web site: https://www.ets.org/research/policy_research_reports/ets; education level: higher educationpostsecondary education; reference count: 103; journal code: may2017; level of availability: available online; publication type: journal articles; publication type: reports research; entry date: 2017&quot;, &quot;publisher&quot; : &quot;ets research report series&quot;, &quot;publisher-place&quot; : &quot;princeton, nj&quot;, &quot;title&quot; : &quot;assessing digital information literacy in higher education: a review of existing frameworks and assessments with recommendations for next-generation assessment. research report. ets rr-16-32&quot;, &quot;type&quot; : &quot;report&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=2665154f-cb3c-4041-be71-46756b34ea29&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(sparks, katz, &amp; beile, 2016)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(sparks, katz, &amp; beile, 2016)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(sparks, katz, &amp; beile, 2016)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><span style='mso-no-proof:yes'>(sparks, katz, &amp; beile, 2016)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>. with additional funding, the study could be repeated with an established instrument across multiple lis graduate programs. <o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><i style='mso-bidi-font-style: normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>future considerations<o:p></o:p></span></i></b></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>repeating the study across multiple institutions would yield a larger sample size that could help librarians target specific groups for outreach. for example, in fall 2017, students having completed their most recent degree in the last five years (<i style='mso-bidi-font-style:normal'>n</i> = 20) had the lowest mean il score, 10.70 (sd 2.54), and students between 20 and 24 years of age (<i style='mso-bidi-font-style:normal'>n</i> = 6) had the lowest mean il score of all age groups. this finding suggests that outreach and instruction efforts should focus on younger students and more recent graduates, but the results apply only to this small cohort of students at a single university. a large sample size that includes students from different institutions would make analysis of the data on age, highest degree earned, and years since most recent degree useful for librarians planning outreach to incoming lis graduate students. <o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>additional demographic questions could reveal useful insights into student needs. for example, questions about previous areas of study could indicate whether students beginning mlis programs with degrees in particular subjects enter with higher or lower il skills. questions about library work experience could give insight into its impact on il. working in a library is often cited as an motivation to enroll in an lis graduate program </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;issn&quot; : &quot;10949054&quot;, &quot;abstract&quot; : &quot;our paper presents the results of a survey of mlis students' motivations for choosing a library career, as well as their outlook on the job market, preferences for various subfields, and dreams about the future. in 2004 several researchers conducted a survey of mlis students at the university of alabama's school of library and information studies and reported the results in a 2006 rusq article. in a field of constant change, it is essential that a new study is conducted to glean current motivations for pursuing a mlis degree. new technologies, economic issues, and other factors could affect a new generation of librarians' mindsets. therefore we replicated the earlier survey, added some new questions, and compared our results. this article describes the results of the current survey. it shows that myriad reasons motivate students to pursue the mlis, and librarians who have an interest in their new colleagues will find this paper of interest. we also suggest some practical steps that reference librarians (as well as human resource officers and library school administrators) can follow to recruit new librarians. this paper is based on a poster presented at the alabama library association conference (alla), april 10, 2009, in auburn, alabama. [abstract from author]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;taylor&quot;, &quot;given&quot; : &quot;stephanie d&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;perry&quot;, &quot;given&quot; : &quot;r alexander&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;barton&quot;, &quot;given&quot; : &quot;jessica l&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;spencer&quot;, &quot;given&quot; : &quot;brett&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;reference &amp; user services quarterly&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;1&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2010&quot; ] ] }, &quot;note&quot; : &quot;accession number: 53897400; taylor, stephanie d. 1; perry, r. alexander 1; barton, jessica l. 1; spencer, brett 1,2; affiliations: 1 : university of alabama's school of library.; 2 : reference librarian, university of alabama, tuscaloosa.; source info: fall2010, vol. 50 issue 1, p35; thesaurus term: library education; thesaurus term: librarians -employment; thesaurus term: librarian recruiting; thesaurus term: reference librarians; thesaurus term: surveys; subject term: vocational guidance; subject term: wages; number of pages: 13p; illustrations: 7 graphs; document type: article&quot;, &quot;page&quot; : &quot;35-47&quot;, &quot;publisher&quot; : &quot;american library association&quot;, &quot;title&quot; : &quot;a follow-up study of the factors shaping the career choices of library school students at the university of alabama&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;50&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=44a0f9cc-898d-4922-a9a7-355c75c6b022&quot; ] }, { &quot;id&quot; : &quot;item-2&quot;, &quot;itemdata&quot; : { &quot;issn&quot; : &quot;10949054&quot;, &quot;abstract&quot; : &quot;hearing why the librarians of tomorrow chose the library profession is vital, because understanding their motivations provides librarians with a glimpse of the future as well as ways to improve that future for the profession. this article therefore seeks to communicate the career motivations of mlis students by reporting the results of a survey conducted at the university of alabama's school of library and information studies. library administrators and others responsible for hiring information professionals will want to hear the library students' reasons for entering the field, their salary expectations, geographic preferences, perceptions of the job market, and interest in various subfields. however, all librarians should seek to learn more about students' career motivations and professional interests so that they can recruit more people to the profession and accommodate the needs of new colleagues. by using the survey results as a springboard, we propose several recruitment strategies with an emphasis on the key role of reference librarians in future campaigns. we also recommend that rusa sponsor annual surveys similar to our own questionnaire so that the profession can adopt an informed approach to the recruitment dilemma. [abstract from author]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;ard&quot;, &quot;given&quot; : &quot;allyson&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;clemmons&quot;, &quot;given&quot; : &quot;susan&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;morgan&quot;, &quot;given&quot; : &quot;nathan&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;sessions&quot;, &quot;given&quot; : &quot;patrick&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;spencer&quot;, &quot;given&quot; : &quot;brett&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;tidwell&quot;, &quot;given&quot; : &quot;tracy&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;west&quot;, &quot;given&quot; : &quot;patricia j&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;reference &amp; user services quarterly&quot;, &quot;id&quot; : &quot;item-2&quot;, &quot;issue&quot; : &quot;3&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2006&quot; ] ] }, &quot;note&quot; : &quot;accession number: 20770697; ard, allyson 1; clemmons, susan 2; morgan, nathan 3; sessions, patrick 4; spencer, brett 5; tidwell, tracy 6; west, patricia j. 7; affiliations: 1 : training specialist, ebsco industries; 2 : technical information specialist, centers for disease control and prevention, atlanta; 3 : reference and instructional librarian, reinert/alumni memorial library, creighton university, omaha, nebraska; 4 : graduate assistant, school of library and information studies, university of alabama; 5 : reference librarian/graduate assistant coordinator, amelia gayle gorgas library--information services, university of alabama; 6 : part-time temporary instructor, department of modern languages and classics, university of alabama; 7 : reference librarian, bender library, american university; source info: spring2006, vol. 45 issue 3, p236; thesaurus term: surveys; thesaurus term: librarians; thesaurus term: reference librarians; thesaurus term: information science; subject term: vocational guidance; subject term: libris (information retrieval system); number of pages: 13p; illustrations: 9 graphs; document type: article&quot;, &quot;page&quot; : &quot;236-248&quot;, &quot;publisher&quot; : &quot;american library association&quot;, &quot;title&quot; : &quot;why library and information science?&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;45&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=14639c83-2ede-4341-897d-b421ee37dcd3&quot; ] }, { &quot;id&quot; : &quot;item-3&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.5860/crl.68.6.533&quot;, &quot;isbn&quot; : &quot;00100870&quot;, &quot;abstract&quot; : &quot;in libraries and other related organizations, information professionals with subject and cultural/language expertise are invaluable as they can contribute to the provision of better service for users who are becoming increasingly diverse. recent statistics indicate that there is a great need to recruit more librarians with subject specialties and with diverse ethnic/cultural backgrounds, especially in academic libraries. two separate studies were conducted to examine career decisions of such individuals who are already working, or likely to work, in academic libraries. suggestions are made to develop effective recruitment strategies and to ensure that the supply of the two special groups of the library profession will effectively meet the demand.&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;kim&quot;, &quot;given&quot; : &quot;kyung-sun&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;chiu&quot;, &quot;given&quot; : &quot;ming-hsin&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;sin&quot;, &quot;given&quot; : &quot;sei-ching joanna&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;robbins&quot;, &quot;given&quot; : &quot;louise&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;college &amp; research libraries&quot;, &quot;id&quot; : &quot;item-3&quot;, &quot;issue&quot; : &quot;6&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2007&quot; ] ] }, &quot;page&quot; : &quot;533-552&quot;, &quot;title&quot; : &quot;recruiting a diverse workforce for academic/research librarianship: career decisions of subject specialists and librarians of color&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;68&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=6aac750d-63bd-4c56-ae77-f614cda98103&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(ard et al., 2006; kim, chiu, sin, &amp; robbins, 2007; taylor et al., 2010)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(ard et al., 2006; kim, chiu, sin, &amp; robbins, 2007; taylor et al., 2010)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(ard et al., 2006; kim, chiu, sin, &amp; robbins, 2007; taylor et al., 2010)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(ard et al., 2006; kim, chiu, sin, &amp; robbins, 2007; taylor, perry, barton, &amp; spencer, 2010)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>. data could substantiate or refute the assumption that students with library work experience may score higher in both self-efficacy and demonstrated il skills than students with no history of working in a library. the number of online mlis students has grown rapidly. in the 2003–2004 academic year, approximately 67% of lis programs responding to the alise survey reported offering internet or web-based classes </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;saye&quot;, &quot;given&quot; : &quot;jerry d.&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;id&quot; : &quot;item-1&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2008&quot; ] ] }, &quot;publisher-place&quot; : &quot;chicago, il&quot;, &quot;title&quot; : &quot;library and information science education statistical report 2005&quot;, &quot;type&quot; : &quot;report&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=4583d164-6dc9-4428-8e95-b5df5d33d7b4&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(saye, 2008)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(saye, 2008)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(saye, 2008)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(saye, 2008)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>; by 2013–2014 , 96% of programs responding to the survey reported offering online courses </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;albertson&quot;, &quot;given&quot; : &quot;dan&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;spetka&quot;, &quot;given&quot; : &quot;kaitlyn&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;snow&quot;, &quot;given&quot; : &quot;kristen&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;id&quot; : &quot;item-1&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2015&quot; ] ] }, &quot;publisher-place&quot; : &quot;seattle, wa&quot;, &quot;title&quot; : &quot;alise library and information science education statistical report 2015&quot;, &quot;type&quot; : &quot;report&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=1b9c57c9-5f33-436c-b9d3-147bbf90d2f0&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(albertson et al., 2015)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(albertson et al., 2015)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(albertson et al., 2015)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(albertson, spetka, &amp; snow, 2015)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>. research suggests that online mlis students have a unique profile </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin; mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.12783/issn.2328-2967/56/3/4&quot;, &quot;isbn&quot; : &quot;07485786&quot;, &quot;issn&quot; : &quot;07485786&quot;, &quot;abstract&quot; : &quot;this paper presents a large scale study of online mlis students (n = 910), who completed at least one online course and were enrolled in 36 of the 58 ala-accredited mlis programs in canada and the united states. the results indicate that the typical student is female, white, lives in an urban setting, and is in her mid-30s. online students were found to be quite diverse, with statistically significant differences in their preferences and satisfaction across five demographic variables: age (generational cohort), employment status, urban status, commute distance, and program modality. three motivations emerged: accommodation, predisposition, and selectivity, which influenced the respondents to choose online learning. the prevalent issues online mlis students experienced were a sense of isolation from peers and instructors, and a lack of professional development and networking opportunities with peers. the findings have implications for enhancing mlis online education including marketing, course offerings, and student support services. [abstract from author]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;oguz&quot;, &quot;given&quot; : &quot;fatih&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;chu&quot;, &quot;given&quot; : &quot;clara m&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;chow&quot;, &quot;given&quot; : &quot;anthony s&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of education for library &amp; information science&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;3&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2015&quot;, &quot;7&quot;, &quot;19&quot; ] ] }, &quot;note&quot; : &quot;from duplicate 1 (studying online: student motivations and experiences in ala-accredited lis programs oguz, fatih; chu, clara m; chow, anthony s)\n\naccession number: 109938520; oguz, fatih 1; email address: f_oguz@uncg.edu; chu, clara m. 2; email address: cmchu@illinois.edu; chow, anthony s. 1; email address: aschow@uncg.edu; affiliations: 1 : department of library and information studies, the university of north carolina at greensboro; 2 : mortenson center for international library programs, university of illinois at urbana-champaign; source info: jul2015, vol. 56 issue 3, p213; thesaurus term: library education; thesaurus term: research; thesaurus term: information science -study &amp;amp; teaching; thesaurus term: library school students; thesaurus term: information science students; thesaurus term: distance education; subject term: online courses; number of pages: 19p; illustrations: 6 charts, 3 graphs; document type: article; full text word count: 8970&quot;, &quot;page&quot; : &quot;213-231&quot;, &quot;publisher&quot; : &quot;association for library &amp; information science education&quot;, &quot;title&quot; : &quot;studying online: student motivations and experiences in ala-accredited lis programs&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;56&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=3a7ff75b-959d-4c59-b9d5-38fb1f6517be&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(oguz, chu, &amp; chow, 2015)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(oguz, chu, &amp; chow, 2015)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(oguz, chu, &amp; chow, 2015)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(oguz, chu, &amp; chow, 2015)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>. it is possible that the scores of online students could differ from those of face-to-face students.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>although lis graduate students have been reported to consult librarians more frequently than graduate students in other programs </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1016/j.acalib.2014.05.004&quot;, &quot;issn&quot; : &quot;0099-1333&quot;, &quot;abstract&quot; : &quot;library and information science (lis) students at the university of illinois, urbana-champaign, were surveyed to ascertain their patterns of library use and their perceptions of the value of various library services. the data reveals similarities and differences to previously studied general graduate student populations, at both illinois and other universities. the analysis highlights differences between distance and on-campus students, an important division in lis professional education. beyond the utility of the findings for understanding the information needs and behaviors of the target population, the study sheds light on pre-professional lis students' use of libraries and their attitudes toward services they are being trained to provide.&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;tracy&quot;, &quot;given&quot; : &quot;daniel g&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;searing&quot;, &quot;given&quot; : &quot;susan e&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of academic librarianship&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;3/4&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2014&quot;, &quot;5&quot;, &quot;1&quot; ] ] }, &quot;note&quot; : &quot;from duplicate 1 (lis graduate students as library users: a survey study tracy, daniel g; searing, susan e)\n\naccession number: s0099133314000706; author: tracy, daniel g. (a, \u204e); author: searing, susan e. (b); affiliation: university library, university of illinois at urbana-champaign, 100 main library, 1408 w. gregory dr., urbana, il 61801, usa; affiliation: university library, university of illinois at urbana-champaign, 246e main library, 1408 west gregory dr., urbana, il 61801, usa; number of pages: 12; language: english;\n\nfrom duplicate 2 (lis graduate students as library users: a survey study. tracy, daniel g; searing, susan e)\n\naccession number: 97305994; tracy, daniel g. 1; email address: dtracy@illinois.edu searing, susan e. 2; email address: searing@illinois.edu; affiliation: 1: university library, university of illinois at urbana-champaign, 100 main library, 1408 w. gregory dr., urbana, il 61801, usa 2: university library, university of illinois at urbana-champaign, 246e main library, 1408 west gregory dr., urbana, il 61801, usa; source info: may2014, vol. 40 issue 3/4, p367; subject term: academic libraries -use studies; subject term: library school students; subject term: research; subject term: professional education; subject term: library school students -attitudes; author-supplied keyword: graduate students; author-supplied keyword: library and information science students; author-supplied keyword: library use patterns; author-supplied keyword: user surveys; company/entity: university of illinois at urbana-champaign; naics/industry codes: 611310 colleges, universities, and professional schools; naics/industry codes: 611430 professional and management development training; number of pages: 2p; document type: article&quot;, &quot;page&quot; : &quot;367-378&quot;, &quot;publisher&quot; : &quot;elsevier inc.&quot;, &quot;title&quot; : &quot;lis graduate students as library users: a survey study&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;40&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=26760086-5f07-4ced-a128-fe30b68cc5a7&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(tracy &amp; searing, 2014)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(tracy &amp; searing, 2014)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(tracy &amp; searing, 2014)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(tracy &amp; searing, 2014)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>, lis graduate students in the united states are still more likely to consult with their instructors and classmates than with librarians </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin; mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.12783/issn.2328-2967/56/s1/9&quot;, &quot;issn&quot; : &quot;07485786&quot;, &quot;abstract&quot; : &quot;librarians are expected to be expert searchers, and developing information literacy skills to navigate the vast world of information is a focus of most library and information science (lis) programs. it is important to understand the information literacy and behaviors of lis students to see if they are employing the skills they will need to assist and educate their future patrons, yet there is relatively little research into the information literacy and behaviors of librarians and library students. using a survey on information behaviors, this study addresses this gap in the literature by examining how lis students in 18 countries search for, evaluate, and use information in various contexts, and on whom they rely for help. the results of this study demonstrate library science students' information literacy skills within an international context. faculty in lis programs will be interested to see how their students approach information problems, and might use this understanding to develop or strengthen courses and inform curricular decisions. [abstract from author]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;saunders&quot;, &quot;given&quot; : &quot;laura&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;kurbanoglu&quot;, &quot;given&quot; : &quot;serap&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;boustany&quot;, &quot;given&quot; : &quot;joumana&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;dogan&quot;, &quot;given&quot; : &quot;guleda&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;becker&quot;, &quot;given&quot; : &quot;peter&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;blumer&quot;, &quot;given&quot; : &quot;eliane&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;chowdhury&quot;, &quot;given&quot; : &quot;sudatta&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;dobreva&quot;, &quot;given&quot; : &quot;milena&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;gendina&quot;, &quot;given&quot; : &quot;natalia&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;grgic&quot;, &quot;given&quot; : &quot;ivana hebrang&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;haddow&quot;, &quot;given&quot; : &quot;gaby&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;koltay&quot;, &quot;given&quot; : &quot;tibor&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;kortelainen&quot;, &quot;given&quot; : &quot;terttu&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;krakowska&quot;, &quot;given&quot; : &quot;monika&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;majid&quot;, &quot;given&quot; : &quot;shaheen&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;mezhova&quot;, &quot;given&quot; : &quot;marina&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;repanovici&quot;, &quot;given&quot; : &quot;angela&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;rud\u017eioniene&quot;, &quot;given&quot; : &quot;jurgita&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;schneider&quot;, &quot;given&quot; : &quot;rene&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;terra&quot;, &quot;given&quot; : &quot;ana lucia&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of education for library &amp; information science&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2015&quot;, &quot;3&quot;, &quot;2&quot; ] ] }, &quot;note&quot; : &quot;accession number: 108403317; saunders, laura 1; email address: laura.saunders@simmons.edu; kurbanoglu, serap 2; email address: kurbanogluserap@gmail.com; boustany, joumana 3; email address: jboustany@gmail.com; dogan, guleda 2; email address: guledaduzyol@gmail.com; becker, peter 4; email address: p.g.becker@hhs.nl; blumer, eliane 5; email address: eliane.blumer@hesge.ch; chowdhury, sudatta 6; email address: sudatta@hotmail.co.uk; dobreva, milena 7; email address: milena.dobreva@gmail.com; gendina, natalia 8; email address: gendina@inbox.ru; grgic, ivana hebrang 9; email address: ihgrgic@ffzg.hr; haddow, gaby 10; email address: g.haddow@curtin.edu.au; koltay, tibor 11; email address: koltay.tibor@abk.szie.hu; kortelainen, terttu 12; email address: terttu.kortelainen@oulu.fi; krakowska, monika 13; email address: monika.krakowska@uj.edu.pl; majid, shaheen 14; mezhova, marina 15; email address: mezhova75@mail.ru; repanovici, angela 16; email address: arepanovici@gmail.com; rud\u017eioniene, jurgita 17; email address: jurgita.rudzioniene@kf.vu.lt; schneider, rene 5; email address: rene.schneider@hesge.ch; terra, ana lucia 18; email address: analuciaterra@yahoo.com; affiliations: 1 : school of library and information science, simmons college; 2 : department of information management, hacettepe university; 3 : department of information communication, universit\u00e9 paris descartes; 4 : hague university of applied sciences, netherlands; 5 : department of information science, haute ecole de gestion; 6 : department of computer &amp;amp; information sciences, university of strathclyde; 7 : library information and archive sciences department, malta university; 8 : scientific and research institute of information technologies, kemerovo state university of culture and arts kemerrovo; 9 : department of information and communication sciences, faculty of humanities and social sciences, university of zagreb; 10 : department of information studies, curtin university; 11 : department of information and library studies, szent istv\u00e1n university,; 12 : department of information studies, oulu university; 13 : department of management and social communication institute of library and information science, jagiellonian university; 14 : division of information studies, wee kim wee school of communication &amp;amp; information, nanyang technological university; 15 : scientific and research institute of information technologies in kemerovo state university of culture and arts kemerrovo; 16 : transilvania university of brasov; 17 : faculty of communication institute of library and information science, vilnius university; 18 : information science department, superior school of industrial studies and management, polytechnic institute of porto; source info: 2015 supplement 1, vol. 56, ps80; thesaurus term: library education; thesaurus term: research; thesaurus term: information literacy -research; thesaurus term: library school students; thesaurus term: library schools -curricula; subject term: academic programs; number of pages: 20p; illustrations: 3 charts, 5 graphs; document type: article; full text word count: 7907&quot;, &quot;page&quot; : &quot;s80-s99&quot;, &quot;publisher&quot; : &quot;association for library &amp; information science education&quot;, &quot;title&quot; : &quot;information behaviors and information literacy skills of lis students: an international perspective&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;56&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=1ea827af-d9ca-44f8-8664-0a731c11718c&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(saunders et al., 2015)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(saunders et al., 2015)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(saunders et al., 2015)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(saunders et al., 2015)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>. tracy and searing’s (2014) survey study on lis graduate students as library users found that lis students “need to learn search strategies and resources as much as other graduate students” </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1016/j.acalib.2014.05.004&quot;, &quot;issn&quot; : &quot;0099-1333&quot;, &quot;abstract&quot; : &quot;library and information science (lis) students at the university of illinois, urbana-champaign, were surveyed to ascertain their patterns of library use and their perceptions of the value of various library services. the data reveals similarities and differences to previously studied general graduate student populations, at both illinois and other universities. the analysis highlights differences between distance and on-campus students, an important division in lis professional education. beyond the utility of the findings for understanding the information needs and behaviors of the target population, the study sheds light on pre-professional lis students' use of libraries and their attitudes toward services they are being trained to provide.&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;tracy&quot;, &quot;given&quot; : &quot;daniel g&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;searing&quot;, &quot;given&quot; : &quot;susan e&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of academic librarianship&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;3/4&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2014&quot;, &quot;5&quot;, &quot;1&quot; ] ] }, &quot;note&quot; : &quot;from duplicate 1 (lis graduate students as library users: a survey study tracy, daniel g; searing, susan e)\n\naccession number: s0099133314000706; author: tracy, daniel g. (a, \u204e); author: searing, susan e. (b); affiliation: university library, university of illinois at urbana-champaign, 100 main library, 1408 w. gregory dr., urbana, il 61801, usa; affiliation: university library, university of illinois at urbana-champaign, 246e main library, 1408 west gregory dr., urbana, il 61801, usa; number of pages: 12; language: english;\n\nfrom duplicate 2 (lis graduate students as library users: a survey study. tracy, daniel g; searing, susan e)\n\naccession number: 97305994; tracy, daniel g. 1; email address: dtracy@illinois.edu searing, susan e. 2; email address: searing@illinois.edu; affiliation: 1: university library, university of illinois at urbana-champaign, 100 main library, 1408 w. gregory dr., urbana, il 61801, usa 2: university library, university of illinois at urbana-champaign, 246e main library, 1408 west gregory dr., urbana, il 61801, usa; source info: may2014, vol. 40 issue 3/4, p367; subject term: academic libraries -use studies; subject term: library school students; subject term: research; subject term: professional education; subject term: library school students -attitudes; author-supplied keyword: graduate students; author-supplied keyword: library and information science students; author-supplied keyword: library use patterns; author-supplied keyword: user surveys; company/entity: university of illinois at urbana-champaign; naics/industry codes: 611310 colleges, universities, and professional schools; naics/industry codes: 611430 professional and management development training; number of pages: 2p; document type: article&quot;, &quot;page&quot; : &quot;367-378&quot;, &quot;publisher&quot; : &quot;elsevier inc.&quot;, &quot;title&quot; : &quot;lis graduate students as library users: a survey study&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;40&quot; }, &quot;locator&quot; : &quot;377&quot;, &quot;suppress-author&quot; : 1, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=26760086-5f07-4ced-a128-fe30b68cc5a7&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(2014, p. 377)&quot;, &quot;manualformatting&quot; : &quot;(p. 377)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(2014, p. 377)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(2014, p. 377)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(p. 377)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>. lis library liaisons can use the data collected from assessments of skill and self-efficacy to guide their outreach efforts to the areas of greatest weakness, especially if self-efficacy exceeds assessed skills. <o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>results could be used in collaborations between lis professors and lis librarian liaisons to address gaps in knowledge in a systematic way, such as the program described by lamb </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.0.49.239/issn.2328-2967/58/1/2&quot;, &quot;issn&quot; : &quot;07485786&quot;, &quot;abstract&quot; : &quot;information workers are not born information fluent. like other students, incoming library science students enter graduate programs with a broad range of information and technology skills. the aim of this study was to determine if systematically designed online tutorials would be effective in preparing university students with information literacy skills. a needs assessment was conducted to identify what information and technology skills faculty expected of entering library students. a series of 46 online tutorials were used to address the required competencies. pre-tests were designed to determine whether a particular student needed to complete a given tutorial. post-tests and proficiency projects were used to determine whether students reached mastery. the results of the study indicated that this type of individualized instruction was effective in preparing library science students with information literacy skills. while the study focused on the library science program, the results may have applications for other types of information literacy instruction. there is a need to expand this individualized, self-paced tutorial approach to other information literacy courses in other disciplines. [abstract from author]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;lamb&quot;, &quot;given&quot; : &quot;annette&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of education for library &amp; information science&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;1&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2017&quot;, &quot;1&quot; ] ] }, &quot;note&quot; : &quot;accession number: 121769959; lamb, annette 1; email address: anlamb@iupui.edu; affiliations: 1 : indiana university at indianapolis (iupui); source info: jan2017, vol. 58 issue 1, p15; thesaurus term: knowledge workers; thesaurus term: library science; thesaurus term: literacy education; subject term: graduate education; subject term: tutors &amp;amp; tutoring; author-supplied keyword: curriculum development; author-supplied keyword: information literacy; author-supplied keyword: information skills; author-supplied keyword: instructional development; author-supplied keyword: lis education; author-supplied keyword: online tutorials; author-supplied keyword: technology skills; number of pages: 12p; illustrations: 1 chart; document type: article&quot;, &quot;page&quot; : &quot;15-26&quot;, &quot;publisher&quot; : &quot;association for library &amp; information science education&quot;, &quot;title&quot; : &quot;debunking the librarian 'gene': designing online information literacy instruction for incoming library science students&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;58&quot; }, &quot;suppress-author&quot; : 1, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=cbf6dbe5-6b16-4e36-a7f8-db9ff9cb8b1f&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(2017)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(2017)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(2017)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(2017)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'> at the department of library science at indiana university at indianapolis. in this program, students are given diagnostic pretests that are used to prescribe a series of self-paced tutorials designed to address the varying degrees of technological proficiency of incoming lis students </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin; mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.0.49.239/issn.2328-2967/58/1/2&quot;, &quot;issn&quot; : &quot;07485786&quot;, &quot;abstract&quot; : &quot;information workers are not born information fluent. like other students, incoming library science students enter graduate programs with a broad range of information and technology skills. the aim of this study was to determine if systematically designed online tutorials would be effective in preparing university students with information literacy skills. a needs assessment was conducted to identify what information and technology skills faculty expected of entering library students. a series of 46 online tutorials were used to address the required competencies. pre-tests were designed to determine whether a particular student needed to complete a given tutorial. post-tests and proficiency projects were used to determine whether students reached mastery. the results of the study indicated that this type of individualized instruction was effective in preparing library science students with information literacy skills. while the study focused on the library science program, the results may have applications for other types of information literacy instruction. there is a need to expand this individualized, self-paced tutorial approach to other information literacy courses in other disciplines. [abstract from author]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;lamb&quot;, &quot;given&quot; : &quot;annette&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of education for library &amp; information science&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;1&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2017&quot;, &quot;1&quot; ] ] }, &quot;note&quot; : &quot;accession number: 121769959; lamb, annette 1; email address: anlamb@iupui.edu; affiliations: 1 : indiana university at indianapolis (iupui); source info: jan2017, vol. 58 issue 1, p15; thesaurus term: knowledge workers; thesaurus term: library science; thesaurus term: literacy education; subject term: graduate education; subject term: tutors &amp;amp; tutoring; author-supplied keyword: curriculum development; author-supplied keyword: information literacy; author-supplied keyword: information skills; author-supplied keyword: instructional development; author-supplied keyword: lis education; author-supplied keyword: online tutorials; author-supplied keyword: technology skills; number of pages: 12p; illustrations: 1 chart; document type: article&quot;, &quot;page&quot; : &quot;15-26&quot;, &quot;publisher&quot; : &quot;association for library &amp; information science education&quot;, &quot;title&quot; : &quot;debunking the librarian 'gene': designing online information literacy instruction for incoming library science students&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;58&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=cbf6dbe5-6b16-4e36-a7f8-db9ff9cb8b1f&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(lamb, 2017)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(lamb, 2017)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(lamb, 2017)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(lamb, 2017)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>. students who score 85% or above on a pretest are exempt from completing the corresponding tutorial, so students only need to complete the tutorials for skills in which they are not deemed proficient </span><!--[if supportfields]><span lang=en-ca style='font-size: 10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin; mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.0.49.239/issn.2328-2967/58/1/2&quot;, &quot;issn&quot; : &quot;07485786&quot;, &quot;abstract&quot; : &quot;information workers are not born information fluent. like other students, incoming library science students enter graduate programs with a broad range of information and technology skills. the aim of this study was to determine if systematically designed online tutorials would be effective in preparing university students with information literacy skills. a needs assessment was conducted to identify what information and technology skills faculty expected of entering library students. a series of 46 online tutorials were used to address the required competencies. pre-tests were designed to determine whether a particular student needed to complete a given tutorial. post-tests and proficiency projects were used to determine whether students reached mastery. the results of the study indicated that this type of individualized instruction was effective in preparing library science students with information literacy skills. while the study focused on the library science program, the results may have applications for other types of information literacy instruction. there is a need to expand this individualized, self-paced tutorial approach to other information literacy courses in other disciplines. [abstract from author]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;lamb&quot;, &quot;given&quot; : &quot;annette&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of education for library &amp; information science&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;1&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2017&quot;, &quot;1&quot; ] ] }, &quot;note&quot; : &quot;accession number: 121769959; lamb, annette 1; email address: anlamb@iupui.edu; affiliations: 1 : indiana university at indianapolis (iupui); source info: jan2017, vol. 58 issue 1, p15; thesaurus term: knowledge workers; thesaurus term: library science; thesaurus term: literacy education; subject term: graduate education; subject term: tutors &amp;amp; tutoring; author-supplied keyword: curriculum development; author-supplied keyword: information literacy; author-supplied keyword: information skills; author-supplied keyword: instructional development; author-supplied keyword: lis education; author-supplied keyword: online tutorials; author-supplied keyword: technology skills; number of pages: 12p; illustrations: 1 chart; document type: article&quot;, &quot;page&quot; : &quot;15-26&quot;, &quot;publisher&quot; : &quot;association for library &amp; information science education&quot;, &quot;title&quot; : &quot;debunking the librarian 'gene': designing online information literacy instruction for incoming library science students&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;58&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=cbf6dbe5-6b16-4e36-a7f8-db9ff9cb8b1f&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(lamb, 2017)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(lamb, 2017)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(lamb, 2017)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(lamb, 2017)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>. <o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;color:#0070c0'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>there are indications that lis faculty are aware of that some incoming lis graduate students lack foundational il skills </span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin csl_citation { &quot;citationitems&quot; : [ { &quot;id&quot; : &quot;item-1&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.0.49.239/issn.2328-2967/58/1/2&quot;, &quot;issn&quot; : &quot;07485786&quot;, &quot;abstract&quot; : &quot;information workers are not born information fluent. like other students, incoming library science students enter graduate programs with a broad range of information and technology skills. the aim of this study was to determine if systematically designed online tutorials would be effective in preparing university students with information literacy skills. a needs assessment was conducted to identify what information and technology skills faculty expected of entering library students. a series of 46 online tutorials were used to address the required competencies. pre-tests were designed to determine whether a particular student needed to complete a given tutorial. post-tests and proficiency projects were used to determine whether students reached mastery. the results of the study indicated that this type of individualized instruction was effective in preparing library science students with information literacy skills. while the study focused on the library science program, the results may have applications for other types of information literacy instruction. there is a need to expand this individualized, self-paced tutorial approach to other information literacy courses in other disciplines. [abstract from author]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;lamb&quot;, &quot;given&quot; : &quot;annette&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of education for library &amp; information science&quot;, &quot;id&quot; : &quot;item-1&quot;, &quot;issue&quot; : &quot;1&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2017&quot;, &quot;1&quot; ] ] }, &quot;note&quot; : &quot;accession number: 121769959; lamb, annette 1; email address: anlamb@iupui.edu; affiliations: 1 : indiana university at indianapolis (iupui); source info: jan2017, vol. 58 issue 1, p15; thesaurus term: knowledge workers; thesaurus term: library science; thesaurus term: literacy education; subject term: graduate education; subject term: tutors &amp;amp; tutoring; author-supplied keyword: curriculum development; author-supplied keyword: information literacy; author-supplied keyword: information skills; author-supplied keyword: instructional development; author-supplied keyword: lis education; author-supplied keyword: online tutorials; author-supplied keyword: technology skills; number of pages: 12p; illustrations: 1 chart; document type: article&quot;, &quot;page&quot; : &quot;15-26&quot;, &quot;publisher&quot; : &quot;association for library &amp; information science education&quot;, &quot;title&quot; : &quot;debunking the librarian 'gene': designing online information literacy instruction for incoming library science students&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;58&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=cbf6dbe5-6b16-4e36-a7f8-db9ff9cb8b1f&quot; ] }, { &quot;id&quot; : &quot;item-2&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.12783/issn.2328-2967/56/s1/9&quot;, &quot;issn&quot; : &quot;07485786&quot;, &quot;abstract&quot; : &quot;librarians are expected to be expert searchers, and developing information literacy skills to navigate the vast world of information is a focus of most library and information science (lis) programs. it is important to understand the information literacy and behaviors of lis students to see if they are employing the skills they will need to assist and educate their future patrons, yet there is relatively little research into the information literacy and behaviors of librarians and library students. using a survey on information behaviors, this study addresses this gap in the literature by examining how lis students in 18 countries search for, evaluate, and use information in various contexts, and on whom they rely for help. the results of this study demonstrate library science students' information literacy skills within an international context. faculty in lis programs will be interested to see how their students approach information problems, and might use this understanding to develop or strengthen courses and inform curricular decisions. [abstract from author]&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;saunders&quot;, &quot;given&quot; : &quot;laura&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;kurbanoglu&quot;, &quot;given&quot; : &quot;serap&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;boustany&quot;, &quot;given&quot; : &quot;joumana&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;dogan&quot;, &quot;given&quot; : &quot;guleda&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;becker&quot;, &quot;given&quot; : &quot;peter&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;blumer&quot;, &quot;given&quot; : &quot;eliane&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;chowdhury&quot;, &quot;given&quot; : &quot;sudatta&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;dobreva&quot;, &quot;given&quot; : &quot;milena&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;gendina&quot;, &quot;given&quot; : &quot;natalia&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;grgic&quot;, &quot;given&quot; : &quot;ivana hebrang&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;haddow&quot;, &quot;given&quot; : &quot;gaby&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;koltay&quot;, &quot;given&quot; : &quot;tibor&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;kortelainen&quot;, &quot;given&quot; : &quot;terttu&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;krakowska&quot;, &quot;given&quot; : &quot;monika&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;majid&quot;, &quot;given&quot; : &quot;shaheen&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;mezhova&quot;, &quot;given&quot; : &quot;marina&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;repanovici&quot;, &quot;given&quot; : &quot;angela&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;rud\u017eioniene&quot;, &quot;given&quot; : &quot;jurgita&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;schneider&quot;, &quot;given&quot; : &quot;rene&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;terra&quot;, &quot;given&quot; : &quot;ana lucia&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of education for library &amp; information science&quot;, &quot;id&quot; : &quot;item-2&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2015&quot;, &quot;3&quot;, &quot;2&quot; ] ] }, &quot;note&quot; : &quot;accession number: 108403317; saunders, laura 1; email address: laura.saunders@simmons.edu; kurbanoglu, serap 2; email address: kurbanogluserap@gmail.com; boustany, joumana 3; email address: jboustany@gmail.com; dogan, guleda 2; email address: guledaduzyol@gmail.com; becker, peter 4; email address: p.g.becker@hhs.nl; blumer, eliane 5; email address: eliane.blumer@hesge.ch; chowdhury, sudatta 6; email address: sudatta@hotmail.co.uk; dobreva, milena 7; email address: milena.dobreva@gmail.com; gendina, natalia 8; email address: gendina@inbox.ru; grgic, ivana hebrang 9; email address: ihgrgic@ffzg.hr; haddow, gaby 10; email address: g.haddow@curtin.edu.au; koltay, tibor 11; email address: koltay.tibor@abk.szie.hu; kortelainen, terttu 12; email address: terttu.kortelainen@oulu.fi; krakowska, monika 13; email address: monika.krakowska@uj.edu.pl; majid, shaheen 14; mezhova, marina 15; email address: mezhova75@mail.ru; repanovici, angela 16; email address: arepanovici@gmail.com; rud\u017eioniene, jurgita 17; email address: jurgita.rudzioniene@kf.vu.lt; schneider, rene 5; email address: rene.schneider@hesge.ch; terra, ana lucia 18; email address: analuciaterra@yahoo.com; affiliations: 1 : school of library and information science, simmons college; 2 : department of information management, hacettepe university; 3 : department of information communication, universit\u00e9 paris descartes; 4 : hague university of applied sciences, netherlands; 5 : department of information science, haute ecole de gestion; 6 : department of computer &amp;amp; information sciences, university of strathclyde; 7 : library information and archive sciences department, malta university; 8 : scientific and research institute of information technologies, kemerovo state university of culture and arts kemerrovo; 9 : department of information and communication sciences, faculty of humanities and social sciences, university of zagreb; 10 : department of information studies, curtin university; 11 : department of information and library studies, szent istv\u00e1n university,; 12 : department of information studies, oulu university; 13 : department of management and social communication institute of library and information science, jagiellonian university; 14 : division of information studies, wee kim wee school of communication &amp;amp; information, nanyang technological university; 15 : scientific and research institute of information technologies in kemerovo state university of culture and arts kemerrovo; 16 : transilvania university of brasov; 17 : faculty of communication institute of library and information science, vilnius university; 18 : information science department, superior school of industrial studies and management, polytechnic institute of porto; source info: 2015 supplement 1, vol. 56, ps80; thesaurus term: library education; thesaurus term: research; thesaurus term: information literacy -research; thesaurus term: library school students; thesaurus term: library schools -curricula; subject term: academic programs; number of pages: 20p; illustrations: 3 charts, 5 graphs; document type: article; full text word count: 7907&quot;, &quot;page&quot; : &quot;s80-s99&quot;, &quot;publisher&quot; : &quot;association for library &amp; information science education&quot;, &quot;title&quot; : &quot;information behaviors and information literacy skills of lis students: an international perspective&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;56&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=1ea827af-d9ca-44f8-8664-0a731c11718c&quot; ] }, { &quot;id&quot; : &quot;item-3&quot;, &quot;itemdata&quot; : { &quot;doi&quot; : &quot;10.1016/j.acalib.2012.08.006&quot;, &quot;isbn&quot; : &quot;00991333&quot;, &quot;abstract&quot; : &quot;the study was carried out with students of official doctoral programs of information science in four universities in spain and latin america with the purpose of finding out, through self-assessments, student perceptions of their own information competence. a survey was designed to determine self-perceptions of knowledge, skills and attitudes regarding information competence, and it was distributed to students of two universities in spain, one in cuba and one in mexico. student perceptions of the levels of their own information competence in terms of knowledge, skills and attitudes are, in general, high. nevertheless, despite being immersed in research processes that require these information competencies, they recognize that they are lacking in certain skills related tothe organization, evaluation and presentation of information. having a self-assessment that identifies those areas of information competence in which students consider themselves to be weak will be a significant help in planning activities designed to strengthen these areas in doctoral programs.&quot;, &quot;author&quot; : [ { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;pinto&quot;, &quot;given&quot; : &quot;maria&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;fernandez-ramos&quot;, &quot;given&quot; : &quot;a&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;sanchez&quot;, &quot;given&quot; : &quot;g&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; }, { &quot;dropping-particle&quot; : &quot;&quot;, &quot;family&quot; : &quot;meneses&quot;, &quot;given&quot; : &quot;g&quot;, &quot;non-dropping-particle&quot; : &quot;&quot;, &quot;parse-names&quot; : false, &quot;suffix&quot; : &quot;&quot; } ], &quot;container-title&quot; : &quot;journal of academic librarianship&quot;, &quot;id&quot; : &quot;item-3&quot;, &quot;issue&quot; : &quot;2&quot;, &quot;issued&quot; : { &quot;date-parts&quot; : [ [ &quot;2013&quot; ] ] }, &quot;page&quot; : &quot;144-154&quot;, &quot;title&quot; : &quot;information competence of doctoral students in information science in spain and latin america: a self-assessment&quot;, &quot;type&quot; : &quot;article-journal&quot;, &quot;volume&quot; : &quot;39&quot; }, &quot;uris&quot; : [ &quot;http://www.mendeley.com/documents/?uuid=011b0714-3e8a-4c3c-a334-04ee76e35bae&quot; ] } ], &quot;mendeley&quot; : { &quot;formattedcitation&quot; : &quot;(lamb, 2017; pinto et al., 2013; saunders et al., 2015)&quot;, &quot;manualformatting&quot; : &quot;(lamb, 2017; pinto et al., 2013)&quot;, &quot;plaintextformattedcitation&quot; : &quot;(lamb, 2017; pinto et al., 2013; saunders et al., 2015)&quot;, &quot;previouslyformattedcitation&quot; : &quot;(lamb, 2017; pinto et al., 2013; saunders et al., 2015)&quot; }, &quot;properties&quot; : { &quot;noteindex&quot; : 0 }, &quot;schema&quot; : &quot;https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot; }<span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>(lamb, 2017; pinto, </span>fernandez-ramos, sanchez, and meneses<span style='mso-no-proof:yes'>, 2013)</span></span><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>. this suggests an additional opportunity for research comparing how lis professors rate lis graduate students’ information literacy proficiency to how lis graduate students rate their own skill level.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'>conclusion<o:p></o:p></span></b></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>this exploratory, cross-sectional, descriptive study measured both the il self-efficacy and demonstrated il skills of students entering an mlis program. the collected data suggests that a moderate positive correlation exists between il self-efficacy and skills.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>this study also tests the feasibility of a larger, multi-institution study that would fill a gap in the literature about lis graduate students and provide other librarians who support these students with data to inform their instruction and outreach plans. this study may also be the first part of a longitudinal study of how mlis students’ il self-efficacy and skills develop as students progress through their graduate program.<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>references<o:p></o:p></span></b></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><!--[if supportfields]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-element:field-begin;mso-field-lock:yes'></span>addin mendeley bibliography csl_bibliography <span style='mso-element:field-separator'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><span style='mso-no-proof:yes'>albertson, d., spetka, k., &amp; snow, k. (2015). <i>alise library and information science education statistical report 2015</i>. seattle, wa. retrieved from <span class=msohyperlink><a href="http://www.alise.org/assets/documents/statistical_reports/2015/alise_2015_statistical_report.pdf%20">http://www.alise.org/assets/documents/statistical_reports/2015/alise_2015_statistical_report.pdf</a></span> <o:p></o:p></span></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>alfonzo, p., &amp; batson, j. (2014). utilizing a co-teaching model to enhance digital literacy instruction for doctoral students. <i>international journal of doctoral studies</i>, <i>9</i>, 61–71. retrieved from <span class=msohyperlink><a href="https://pdfs.semanticscholar.org/ddc5/79e2dcbf825d7a11e122615a89a27646f482.pdf">https://pdfs.semanticscholar.org/ddc5/79e2dcbf825d7a11e122615a89a27646f482.pdf</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><em><span lang=en-ca><o:p>&nbsp;</o:p></span></em></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>american library association. (1989). <i>presidential committee on information literacy: final report.</i> chicago, il. retrieved from <span class=msohyperlink><a href="http://www.ala.org/acrl/publications/whitepapers/presidential">http://www.ala.org/acrl/publications/whitepapers/presidential</a></span></span><span lang=en-ca style='mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin; mso-no-proof:yes'><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>ard, a., clemmons, s., morgan, n., sessions, p., spencer, b., tidwell, t., &amp; west, p. j. (2006). why library and information science? <i>reference &amp; user services quarterly</i>, <i>45</i>(3), 236–248. retrieved from <span class=msohyperlink><a href="https://www.jstor.org/stable/20864520">https://www.jstor.org/stable/20864520</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>association of college and research libraries. (2016). framework for information literacy for higher education. retrieved from <span class=msohyperlink><a href="http://www.ala.org/acrl/standards/ilframework">http://www.ala.org/acrl/standards/ilframework</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>bandura, a. (1977). self-efficacy: toward a unifying theory of behavioral change. <i>psychological review</i>, <i>84</i>, 191–215. retrieved from <span class=msohyperlink><a href="https://doi.org/10.1037/0033-295x.84.2.191">https://doi.org/10.1037/0033-295x.84.2.191</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>bandura, a. (1986). <i>social foundations of thought and action: a social cognitive theory</i>. englewood cliffs, n.j.: prentice-hall.<o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>batarelo kokić, i., &amp; novosel, v. (2014). the ball is in your court: information literacy self-efficacy and information literacy competence relation. in s. kurbanoglu, s. špiranec, e. grassian, d. mizrachi, r. catts (eds.), <i>information literacy. lifelong learning &amp; digital citizenship in the 21st century</i> (pp. 512–520). <span class=msohyperlink><a href="https://doi.org/doi:10.1007/978-3-319-14136-7_54">https://doi.org/doi:10.1007/978-3-319-14136-7_54</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>beile, p. m. (2005). <i>development and validation of the beile test of information literacy for education (b-tiled)</i>. university of central florida. retrieved from <span class=msohyperlink><a href="http://works.bepress.com/penny-beile/18/">http://works.bepress.com/penny-beile/18/</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>beile, p. m. (2007). the ilas-ed: a standards-based instrument for assessing pre-service teachers’ information literacy levels. in <i>society for information technology &amp; teacher education international conference</i> (vol. 2007, pp. 1–27). retrieved from <span class=msohyperlink><a href="http://eprints.rclis.org/16928/">http://eprints.rclis.org/16928/</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>boucher, c., dalziel, k., davies, m., glen, s., &amp; chandler, j. (2009). are postgraduates ready for research? poster presented at the librarians information literacy annual conference (lilac). cardiff, uk. retrieved from <span class=msohyperlink><a href="http://rrsa.cmich.edu/documents/rrsa_poster_lilac_2009.pdf">http://rrsa.cmich.edu/documents/rrsa_poster_lilac_2009.pdf</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>cannon, t. (2007). <i>closing the digital divide: an assessment of urban graduate teacher education students’ knowledge of information literacy and their readiness to integrate information literacy into their teaching</i>. university of san francisco. retrieved from <span class=msohyperlink><a href="https://repository.usfca.edu/diss/252">https://repository.usfca.edu/diss/252</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>catalano, a. (2010). using acrl standards to assess the information literacy of graduate students in an education program. <i>evidence based library &amp; information practice</i>, <i>5</i>(4), 21–38. <span class=msohyperlink><a href="https://doi.org/10.18438/b8v62b">https://doi.org/10.18438/b8v62b</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>catalano, a., &amp; phillips, s. r. (2016). information literacy and retention: a case study of the value of the library. <i>evidence based library and information practice</i>, <i>11</i>(4), 2–13. <span class=msohyperlink><a href="https://doi.org/10.18438/b82k7w">https://doi.org/10.18438/b82k7w</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>conway, k. (2011). how prepared are students for postgraduate study? a comparison of the information literacy skills of commencing undergraduate and postgraduate information studies students at curtin university. <i>australian academic &amp; research libraries</i>, <i>42</i>(2), 121–135. <span class=msohyperlink><a href="https://doi.org/10.1080/00048623.2011.10722218">https://doi.org/10.1080/00048623.2011.10722218</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>crosetto, a., wilkenfeld, p., &amp; runnestrand, d. (2007). responding to the needs of our graduate students: a pilot information literacy course in graduate education. in t. jacobson &amp; t. mackey (eds.), <i>information literacy collaborations that work</i> (pp. 41–56). new york, ny: neal-shuman publishers.<o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>green, r. (2010). information illiteracy: examining our assumptions. <i>journal of academic librarianship</i>, <i>36</i>(4), 313–319. <span class=msohyperlink><a href="https://doi.org/10.1016/j.acalib.2010.05.005">https://doi.org/10.1016/j.acalib.2010.05.005</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>gross, m., &amp; latham, d. (2012). what’s skill got to do with it?: information literacy skills and self-views of ability among first-year college students. <i>journal of the american society for information science &amp; technology</i>, <i>63</i>(3), 574–583. <span class=msohyperlink><a href="https://doi.org/10.1002/asi.21681">https://doi.org/10.1002/asi.21681</a></span><o:p></o:p></span></p> <p class=msonormal style='mso-pagination:none;mso-layout-grid-align:none; text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>harrington, m. r. (2009). information literacy and research-intensive graduate students: enhancing the role of research librarians. <i>behavioral &amp; social sciences librarian</i>, <i>28</i>(4), 179–201. <span class=msohyperlink><a href="https://doi.org/10.1080/01639260903272778">https://doi.org/10.1080/01639260903272778</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>islam, m. a., &amp; tsuji, k. (2010). assessing information literacy competency of information science and library management graduate students of dhaka university. <i>ifla journal</i>, <i>36</i>(4), 300–316. <span class=msohyperlink><a href="https://doi.org/10.1177/0340035210388243">https://doi.org/10.1177/0340035210388243</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>ivanitskaya, l., laus, r., &amp; casey, a. m. (2004). research readiness self-assessment: assessing students' research skills and attitudes. <i>journal of library administration</i>, <i>41</i>(1/2), 167–183. <span class=msohyperlink><a href="https://doi.org/10.1300/j111v41n01_13">https://doi.org/10.1300/j111v41n01_13</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>jackson, c. (2013). confidence as an indicator of research students’ abilities in information literacy: a mismatch. <i>journal of information literacy</i>, <i>7</i>(2), 149–152. <span class=msohyperlink><a href="https://doi.org/10.11645/7.2.1848">https://doi.org/10.11645/7.2.1848</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>jesse, s. (2012). subject specific information literacy curriculum and assessment. <i>christian librarian</i>, <i>55</i>(1), 2–16. retrieved from <span class=msohyperlink><a href="http://digitalcommons.georgefox.edu/cgi/viewcontent.cgi?article=1477&amp;context=tcl">http://digitalcommons.georgefox.edu/cgi/viewcontent.cgi?article=1477&amp;context=tcl</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>kim, k.-s., chiu, m.-h., sin, s.-c. j., &amp; robbins, l. (2007). recruiting a diverse workforce for academic/research librarianship: career decisions of subject specialists and librarians of color. <i>college &amp; research libraries</i>, <i>68</i>(6), 533–552. <span class=msohyperlink><a href="https://doi.org/10.5860/crl.68.6.533">https://doi.org/10.5860/crl.68.6.533</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>kurbanoglu, s. (2003). self-efficacy: a concept closely linked to information literacy and lifelong learning. <i>journal of documentation</i>, <i>59</i>(6), 635–646. <span class=msohyperlink><a href="https://doi.org/10.1108/00220410310506295">https://doi.org/10.1108/00220410310506295</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>kurbanoglu, s., akkoyunlu, b., &amp; umay, a. (2006). developing the information literacy self-efficacy scale. <i>journal of documentation</i>, <i>62</i>(6), 730–743. <span class=msohyperlink><a href="https://doi.org/10.1108/00220410610714949">https://doi.org/10.1108/00220410610714949</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>lamb, a. (2017). debunking the librarian “gene”: designing online information literacy instruction for incoming library science students. <i>journal of education for library &amp; information science</i>, <i>58</i>(1), 15–26. <span class=msohyperlink><a href="https://eric.ed.gov/?id=ej1150595%20">https://eric.ed.gov/?id=ej1150595</a></span> <o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>magliaro, j. (2011). <i>comparing information literacy needs of graduate students in selected graduate programs through the technology acceptance model and affordance theory</i>. university of windsor. retrieved from <span class=msohyperlink><a href="https://scholar.uwindsor.ca/etd/424">https://scholar.uwindsor.ca/etd/424</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>mahmood, k. (2016). do people overestimate their information literacy skills? a systematic review of empirical evidence on the dunning-kruger effect. <i>communications in information literacy</i>, <i>10</i>(2), 199–213. retrieved from <span class=msohyperlink><a href="https://doaj.org/article/cf4e25b16ec2420c9168033977cdcd3d">https://doaj.org/article/cf4e25b16ec2420c9168033977cdcd3d</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>michalak, r., &amp; rysavy, m. d. t. (2016). information literacy in 2015: international graduate business students’ perceptions of information literacy skills compared to test-assessed skills. <i>journal of business &amp; finance librarianship</i>, <i>21</i>(2), 152–174. <span class=msohyperlink><a href="https://doi.org/10.1080/08963568.2016.1145787">https://doi.org/10.1080/08963568.2016.1145787</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>michalak, r., rysavy, m. d. t., &amp; wessel, a. (2017). students’ perceptions of their information literacy skills: the confidence gap between male and female international graduate students. <i>journal of academic librarianship</i>, <i>43</i>(2), 100–104. <span class=msohyperlink><a href="https://doi.org/10.1016/j.acalib.2017.02.003">https://doi.org/10.1016/j.acalib.2017.02.003</a></span><o:p></o:p></span></p> <p class=msonormal style='mso-pagination:none;mso-layout-grid-align:none; text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>oguz, f., chu, c. m., &amp; chow, a. s. (2015). studying online: student motivations and experiences in ala accredited lis programs. <i>journal of education for library &amp; information science</i>, <i>56</i>(3), 213–231. <span class=msohyperlink><a href="https://files.eric.ed.gov/fulltext/ej1074636.pdf">https://files.eric.ed.gov/fulltext/ej1074636.pdf</a></span> <o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>perrett, v. (2004). graduate information literacy skills: the 2003 anu skills audit. <i>australian library journal</i>, <i>53</i>(2), 161–171. <span class=msohyperlink><a href="https://doi.org/10.1080/00049670.2004.10721622">https://doi.org/10.1080/00049670.2004.10721622</a></span><o:p></o:p></span></p> <p class=msonormal style='mso-pagination:none;mso-layout-grid-align:none; text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>pinto, m., fernandez-ramos, a., sanchez, g., &amp; meneses, g. (2013). information competence of doctoral students in information science in spain and latin america: a self-assessment. <i>journal of academic librarianship</i>, <i>39</i>(2), 144–154. <span class=msohyperlink><a href="https://doi.org/10.1016/j.acalib.2012.08.006">https://doi.org/10.1016/j.acalib.2012.08.006</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>radcliff, c., oakleaf, m., &amp; van hoeck, m. (2014). so what? the results and impact of a decade of imls-funded information literacy assessments. in s. durso, s. hiller, m. kyrillidou, &amp; a. pappalarado (eds.), <i>proceedings of the 2014 library assessment conference: building effective, sustainable, practical assessment</i> (pp. 801–809). seattle, wa: association of research libraries. retrieved from <span class=msohyperlink><a href="http://old.libraryassessment.org/bm~doc/24radcliffpanel.pdf">http://old.libraryassessment.org/bm~doc/24radcliffpanel.pdf</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>robertson, d. s., &amp; felicilda-reynaldo, r. f. d. (2015). evaluation of graduate nursing students’ information literacy self-efficacy and applied skills. <i>journal of nursing education</i>, <i>54</i>(3, suppl), s26–30. <span class=msohyperlink><a href="https://doi.org/10.3928/01484834-20150218-03">https://doi.org/10.3928/01484834-20150218-03</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>sadler, e. (bess), &amp; given, l. m. (2007). affordance theory: a framework for graduate students’ information behavior. <i>journal of documentation</i>, <i>63</i>(1), 115–141. <span class=msohyperlink><a href="https://doi.org/10.1108/00220410710723911">https://doi.org/10.1108/00220410710723911</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>saunders, l., kurbanoglu, s., boustany, j., dogan, g., becker, p., blumer, e., … terra, a. l. (2015). information behaviors and information literacy skills of lis students: an international perspective. <i>journal of education for library &amp; information science</i>, <i>56</i>, s80–s99. retrieved from <span class=msohyperlink><a href="https://eric.ed.gov/?id=ej1073544">https://eric.ed.gov/?id=ej1073544</a></span><span style='mso-spacerun:yes'>  </span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>saunders, l., severyn, j., freundlich, s., piroli, v., &amp; shaw-munderback, j. (2016). assessing graduate level information literacy instruction with critical incident questionnaires. <i>journal of academic librarianship</i>, <i>42</i>(6), 655–663. </span><span class=msohyperlink><span lang=en-ca style='font-size: 10.0pt;font-family:"palatino linotype",serif'><a href="https://doi.org/10.1016/j.acalib.2016.08.008">https://doi.org/10.1016/j.acalib.2016.08.008</a></span></span><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>saye, j. d. (2008). <i>library and information science education statistical report 2005</i>. chicago, il.<o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>sparks, j. r., katz, i. r., &amp; beile, p. m. (2016). <i>assessing digital information literacy in higher education: a review of existing frameworks and assessments with recommendations for next-generation assessment. research report. ets rr-16-32</i>. <i>ets research report series</i>. princeton, nj: ets research report series. <span class=msohyperlink><a href="https://doi.org/10.1002/ets2.12118">https://doi.org/10.1002/ets2.12118</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>tang, y., &amp; tseng, h. w. (2013). distance learners’ self-efficacy and information literacy skills. <i>journal of academic librarianship</i>, <i>39</i>(6), 517–521. <span class=msohyperlink><a href="https://doi.org/10.1016/j.acalib.2013.08.008">https://doi.org/10.1016/j.acalib.2013.08.008</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>taylor, s. d., perry, r. a., barton, j. l., &amp; spencer, b. (2010). a follow-up study of the factors shaping the career choices of library school students at the university of alabama. <i>reference &amp; user services quarterly</i>, <i>50</i>(1), 35–47. retrieved from <span class=msohyperlink><a href="http://www.jstor.org/stable/20865334">http://www.jstor.org/stable/20865334</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>tewell, e., &amp; angell, k. (2015). far from a trivial pursuit: assessing the effectiveness of games in information literacy instruction. <i>evidence based library &amp; information practice</i>, <i>10</i>(1), 20–33. <span class=msohyperlink><a href="https://doi.org/10.18438/b8b60x">https://doi.org/10.18438/b8b60x</a></span><o:p></o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-left:24.0pt;text-indent:-24.0pt;mso-pagination: none;mso-layout-grid-align:none;text-autospace:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin;mso-no-proof:yes'>tracy, d. g., &amp; searing, s. e. (2014). lis graduate students as library users: a survey study. <i>journal of academic librarianship</i>, <i>40</i>(3/4), 367–378. <span class=msohyperlink><a href="https://doi.org/10.1016/j.acalib.2014.05.004">https://doi.org/10.1016/j.acalib.2014.05.004</a></span><o:p></o:p></span></p> <p class=msonormal style='mso-pagination:none;mso-layout-grid-align:none; text-autospace:none'><!--[if supportfields]><span lang=en-ca style='font-size: 10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin'><span style='mso-element:field-end'></span></span><![endif]--><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='margin-bottom:10.0pt;line-height:115%'><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt; line-height:115%;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>appendix a<o:p></o:p></span></b></p> <p class=msolistparagraph style='margin-left:0in;mso-add-space:auto;line-height: normal'><b style='mso-bidi-font-weight:normal'><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'>demographic questions<o:p></o:p></span></b></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>to which gender do you most identify? (radio button)<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>female<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>male<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>non-binary/third gender<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>prefer to self-describe _____<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>prefer not to answer <o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>age (drop down)<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>19 or less<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>20-24<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>25-29<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>30-34<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>35-39<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>40-44<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>45-49<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>50-54<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>55 or over<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>race ethnicity (drop down)<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>hispanic of any race<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>american indian or alaskan native<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>asian<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>black or african american<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>native hawaiian or pacific islander<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>white<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>two or more races<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>international<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>race or ethnicity unknown<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>prefer not to answer<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>years since obtaining your most recent degree (radio button)<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>5 or less<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>6-10<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>11-15<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>16-20<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>21 or more<o:p></o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>highest degree earned (radio button)<o:p></o:p></span></p> <p class=msolistparagraphcxspfirst style='line-height:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>bachelors<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='line-height:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>masters<o:p></o:p></span></p> <p class=msolistparagraphcxsplast style='line-height:normal'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>doctorate<o:p></o:p></span></p> <p class=msonormal style='mso-pagination:none;mso-layout-grid-align:none; text-autospace:none'><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>appendix b<o:p></o:p></span></b></p> <p class=msonormalcxspmiddle style='margin-left:24.0pt;mso-add-space:auto; text-indent:-24.0pt;mso-pagination:none;mso-layout-grid-align:none;text-autospace: none'><b style='mso-bidi-font-weight:normal'><span lang=en-ca style='font-size: 10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin'>questions modified from ilas-ed<o:p></o:p></span></b></p> <p class=msonormal style='mso-pagination:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='mso-pagination:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>question 7<o:p></o:p></span></p> <p class=msonormal style='mso-pagination:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>which of the following characteristics best indicates scholarly research? <o:p></o:p></span></p> <p class=msolistparagraphcxspfirst style='margin-bottom:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:-.25in;line-height:normal;mso-pagination:none; mso-list:l11 level1 lfo15'><![if !supportlists]><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-fareast-font-family:"palatino linotype"; mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>a.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>available in an academic library <o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-bottom:0in;margin-bottom: .0001pt;mso-add-space:auto;text-indent:-.25in;line-height:normal;mso-pagination: none;mso-list:l11 level1 lfo15'><![if !supportlists]><span style='font-size: 10.0pt;font-family:"palatino linotype",serif;mso-fareast-font-family:"palatino linotype"; mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>b.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>indexed by an academic database <o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-bottom:0in;margin-bottom: .0001pt;mso-add-space:auto;text-indent:-.25in;line-height:normal;mso-pagination: none;mso-list:l11 level1 lfo15'><![if !supportlists]><span style='font-size: 10.0pt;font-family:"palatino linotype",serif;mso-fareast-font-family:"palatino linotype"; mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>c.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>reviewed by experts for publication <o:p></o:p></span></p> <p class=msolistparagraphcxsplast style='margin-bottom:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:-.25in;line-height:normal;mso-pagination:none; mso-list:l11 level1 lfo15'><![if !supportlists]><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-fareast-font-family:"palatino linotype"; mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>d.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>written by university faculty<o:p></o:p></span></p> <p class=msonormal style='mso-pagination:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='mso-pagination:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>question 8<o:p></o:p></span></p> <p class=msolistparagraphcxspfirst style='margin-left:0in;mso-add-space:auto; line-height:normal;mso-pagination:none'><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'>you are unfamiliar with the topic of the whole language movement, so you decide to read a brief history and summary about it. which of the following sources would be best?<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-bottom:0in;margin-bottom: .0001pt;mso-add-space:auto;text-indent:-.25in;line-height:normal;mso-pagination: none;mso-list:l12 level1 lfo16'><![if !supportlists]><span style='font-size: 10.0pt;font-family:"palatino linotype",serif;mso-fareast-font-family:"palatino linotype"; mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>a.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>a book on the topic, such as <i style='mso-bidi-font-style:normal'>perspectives on whole language learning: a case stud</i>y<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-bottom:0in;margin-bottom: .0001pt;mso-add-space:auto;text-indent:-.25in;line-height:normal;mso-pagination: none;mso-list:l12 level1 lfo16'><![if !supportlists]><span style='font-size: 10.0pt;font-family:"palatino linotype",serif;mso-fareast-font-family:"palatino linotype"; mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>b.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>a general encyclopedia, such as <i style='mso-bidi-font-style:normal'>encyclopedia britannica</i><o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-bottom:0in;margin-bottom: .0001pt;mso-add-space:auto;text-indent:-.25in;line-height:normal;mso-pagination: none;mso-list:l12 level1 lfo16'><![if !supportlists]><span style='font-size: 10.0pt;font-family:"palatino linotype",serif;mso-fareast-font-family:"palatino linotype"; mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>c.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>an article on the topic, such as “whole language in the classroom: a student teacher’s perspective” <o:p></o:p></span></p> <p class=msolistparagraphcxsplast style='margin-bottom:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:-.25in;line-height:normal;mso-pagination:none; mso-list:l12 level1 lfo16'><![if !supportlists]><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-fareast-font-family:"palatino linotype"; mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>d.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>an education encyclopedia, such as <i style='mso-bidi-font-style:normal'>encyclopedia of education</i><o:p></o:p></span></p> <p class=msonormal style='mso-pagination:none'><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin;mso-ansi-language:en-us'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='mso-pagination:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>question 10<o:p></o:p></span></p> <p class=msolistparagraphcxspfirst style='margin-left:0in;mso-add-space:auto; line-height:normal;mso-pagination:none'><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'>you are looking for a peer-reviewed article about the librarian’s role in open education resources and textbook affordability efforts. the most appropriate place to look is:<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:.25in;line-height:normal;mso-pagination:none; mso-list:l10 level1 lfo17'><![if !supportlists]><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-fareast-font-family:"palatino linotype"; mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>a.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>a library &amp; information science database<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:.25in;line-height:normal;mso-pagination:none; mso-list:l10 level1 lfo17'><![if !supportlists]><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-fareast-font-family:"palatino linotype"; mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>b.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>wikipedia<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:.25in;line-height:normal;mso-pagination:none; mso-list:l10 level1 lfo17'><![if !supportlists]><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-fareast-font-family:"palatino linotype"; mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>c.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>a news resources database<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:.25in;line-height:normal;mso-pagination:none; mso-list:l10 level1 lfo17'><![if !supportlists]><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-fareast-font-family:"palatino linotype"; mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>d.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>both (a) and (c)<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-left:0in;mso-add-space:auto; line-height:normal;mso-pagination:none'><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-left:0in;mso-add-space:auto; line-height:normal;mso-pagination:none'><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'>question 12<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-left:0in;mso-add-space:auto; line-height:normal;mso-pagination:none'><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'>research studies in library and information science are generally first communicated through: <o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:.25in;line-height:normal;mso-pagination:none; mso-list:l19 level2 lfo18'><![if !supportlists]><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-fareast-font-family:"palatino linotype"; mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>a.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>books published by library associations <o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:.25in;line-height:normal;mso-pagination:none; mso-list:l19 level2 lfo18'><![if !supportlists]><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-fareast-font-family:"palatino linotype"; mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>b.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>library science encyclopedia entries <o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:.25in;line-height:normal;mso-pagination:none; mso-list:l19 level2 lfo18'><![if !supportlists]><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-fareast-font-family:"palatino linotype"; mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>c.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>newsletters of library associations <o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:.25in;line-height:normal;mso-pagination:none; mso-list:l19 level2 lfo18'><![if !supportlists]><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-fareast-font-family:"palatino linotype"; mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>d.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>professional conferences and journal articles<o:p></o:p></span></p> <p class=msolistparagraphcxsplast style='margin-left:0in;mso-add-space:auto; line-height:normal;mso-pagination:none'><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='mso-pagination:lines-together;page-break-after:avoid'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>question 13<o:p></o:p></span></p> <p class=msolistparagraphcxspfirst style='margin-left:0in;mso-add-space:auto; line-height:normal;mso-pagination:lines-together;page-break-after:avoid'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>you have been assigned to write a short class paper on effective library instruction techniques. your professor indicated three recent scholarly sources would be sufficient. which strategy is best to locate items?<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-bottom:0in;margin-bottom: .0001pt;mso-add-space:auto;text-indent:-13.5pt;line-height:normal;mso-pagination: lines-together;page-break-after:avoid;mso-list:l22 level2 lfo19'><![if !supportlists]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-fareast-font-family: "palatino linotype";mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>a.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>search a general academic database and a library and information science database for journal articles <o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-bottom:0in;margin-bottom: .0001pt;mso-add-space:auto;text-indent:-13.5pt;line-height:normal;mso-pagination: lines-together;page-break-after:avoid;mso-list:l22 level2 lfo19'><![if !supportlists]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-fareast-font-family: "palatino linotype";mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>b.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>search a library and information science database for journal articles <o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-bottom:0in;margin-bottom: .0001pt;mso-add-space:auto;text-indent:-13.5pt;line-height:normal;mso-pagination: lines-together;page-break-after:avoid;mso-list:l22 level2 lfo19'><![if !supportlists]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-fareast-font-family: "palatino linotype";mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>c.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>search the library catalog for books <o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-bottom:0in;margin-bottom: .0001pt;mso-add-space:auto;text-indent:-13.5pt;line-height:normal;mso-pagination: lines-together;page-break-after:avoid;mso-list:l22 level2 lfo19'><![if !supportlists]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-fareast-font-family: "palatino linotype";mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>d.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>search the library catalog for encyclopedias<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-left:0in;mso-add-space:auto; line-height:normal;mso-pagination:none'><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-left:0in;mso-add-space:auto; line-height:normal;mso-pagination:none'><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'>question 14<o:p></o:p></span></p> <p class=msolistparagraphcxsplast style='margin-left:0in;mso-add-space:auto; line-height:normal;mso-pagination:none'><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'>select the set of search terms that best represent the main concepts in the following:<o:p></o:p></span></p> <p class=msonormal style='margin-left:22.5pt;mso-add-space:auto;mso-pagination: none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>what are the benefits associated with library use for low-income students?<o:p></o:p></span></p> <p class=msolistparagraphcxspfirst style='margin:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:22.5pt;line-height:normal;mso-pagination:none; mso-list:l24 level2 lfo20'><![if !supportlists]><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-fareast-font-family:"palatino linotype"; mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>a.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>library use, benefits, low-income students <o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:22.5pt;line-height:normal;mso-pagination:none; mso-list:l24 level2 lfo20'><![if !supportlists]><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-fareast-font-family:"palatino linotype"; mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>b.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>library use, benefits, students <o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:22.5pt;line-height:normal;mso-pagination:none; mso-list:l24 level2 lfo20'><![if !supportlists]><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-fareast-font-family:"palatino linotype"; mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>c.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>library use, low income, students <o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:22.5pt;line-height:normal;mso-pagination:none; mso-list:l24 level2 lfo20'><![if !supportlists]><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-fareast-font-family:"palatino linotype"; mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>d.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>library, low-income students, use<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-left:0in;mso-add-space:auto; line-height:normal;mso-pagination:none'><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-left:0in;mso-add-space:auto; line-height:normal;mso-pagination:none'><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'>question 15<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-left:0in;mso-add-space:auto; line-height:normal;mso-pagination:none'><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'>select the set that best represents synonyms and related terms for the concept “college students.”<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:22.5pt;line-height:normal;mso-pagination:none; mso-list:l13 level2 lfo21'><![if !supportlists]><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-fareast-font-family:"palatino linotype"; mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>a.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>colleges, universities, community colleges… <o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:22.5pt;line-height:normal;mso-pagination:none; mso-list:l13 level2 lfo21'><![if !supportlists]><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-fareast-font-family:"palatino linotype"; mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>b.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>millennials, students, undergraduates… <o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:22.5pt;line-height:normal;mso-pagination:none; mso-list:l13 level2 lfo21'><![if !supportlists]><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-fareast-font-family:"palatino linotype"; mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>c.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>graduate students, freshmen, sophomores... <o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:22.5pt;line-height:normal;mso-pagination:none; mso-list:l13 level2 lfo21'><![if !supportlists]><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-fareast-font-family:"palatino linotype"; mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>d.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>university, adult learners, educational attendees...<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-left:0in;mso-add-space:auto; line-height:normal;mso-pagination:none'><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-left:0in;mso-add-space:auto; line-height:normal;mso-pagination:none'><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'>question 16<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-left:0in;mso-add-space:auto; line-height:normal;mso-pagination:none'><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'>while researching library patrons, you find that they are also sometimes called “library customers” or “library clients.” you decide to look for information on the subject in a database that indexes library science literature. to save time you write a search statement that includes all three terms. which of the following is the best example to use when you have fairly synonymous terms and it does not matter which of the terms is found in the record?<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:22.5pt;line-height:normal;mso-pagination:none; mso-list:l21 level2 lfo22'><![if !supportlists]><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-fareast-font-family:"palatino linotype"; mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>a.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>patrons and customers and clients<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:22.5pt;line-height:normal;mso-pagination:none; mso-list:l21 level2 lfo22'><![if !supportlists]><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-fareast-font-family:"palatino linotype"; mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>b.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>patrons or customers or clients<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:22.5pt;line-height:normal;mso-pagination:none; mso-list:l21 level2 lfo22'><![if !supportlists]><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-fareast-font-family:"palatino linotype"; mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>c.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>patrons, customers and clients<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:22.5pt;line-height:normal;mso-pagination:none; mso-list:l21 level2 lfo22'><![if !supportlists]><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-fareast-font-family:"palatino linotype"; mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>d.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>patrons, customers or clients<o:p></o:p></span></p> <p class=msolistparagraphcxsplast style='margin-left:0in;mso-add-space:auto; line-height:normal;mso-pagination:none'><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='mso-pagination:lines-together;page-break-after:avoid'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>question 18<o:p></o:p></span></p> <p class=msolistparagraphcxspfirst style='margin-left:0in;mso-add-space:auto; line-height:normal;mso-pagination:lines-together;page-break-after:avoid'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>you have a class assignment to investigate how summer reading programs impact student achievement. a keyword search in an academic database on “summer reading programs” has returned over 600 items. to narrow your search, which of the following steps would you next perform?<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:22.5pt;line-height:normal;mso-pagination:lines-together; page-break-after:avoid;mso-list:l16 level2 lfo23'><![if !supportlists]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-fareast-font-family: "palatino linotype";mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>a.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>add “impact” as a keyword <o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:22.5pt;line-height:normal;mso-pagination:lines-together; page-break-after:avoid;mso-list:l16 level2 lfo23'><![if !supportlists]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-fareast-font-family: "palatino linotype";mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>b.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>add “student achievement” as a keyword <o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:22.5pt;line-height:normal;mso-pagination:lines-together; page-break-after:avoid;mso-list:l16 level2 lfo23'><![if !supportlists]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-fareast-font-family: "palatino linotype";mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>c.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>limit search results by date <o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:22.5pt;line-height:normal;mso-pagination:lines-together; page-break-after:avoid;mso-list:l16 level2 lfo23'><![if !supportlists]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-fareast-font-family: "palatino linotype";mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>d.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>limit search results by publication type<o:p></o:p></span></p> <p class=msolistparagraphcxsplast style='margin-left:0in;mso-add-space:auto; line-height:normal;mso-pagination:none'><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='mso-pagination:none;page-break-after:avoid'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>question 20<o:p></o:p></span></p> <p class=msolistparagraph style='margin-left:0in;mso-add-space:auto;line-height: normal;mso-pagination:none;page-break-after:avoid'><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'>your professor suggested you read a particular article and gave you the following citation:<o:p></o:p></span></p> <p class=msonormalcxspfirst style='margin-left:.5in;mso-add-space:auto; mso-pagination:none;page-break-after:avoid'><span lang=en-ca style='font-size: 10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family:calibri; mso-bidi-theme-font:minor-latin'>thomas, w., &amp; shouse, d. (2014). this is not a dumpsite: the problem of evaluating gift books. <i style='mso-bidi-font-style: normal'>library collections, acquisitions &amp; technical services</i>, <i style='mso-bidi-font-style:normal'>38</i>(3-4), 63-69.<o:p></o:p></span></p> <p class=msonormal style='mso-pagination:none;page-break-after:avoid'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>which of the following would you type into the library’s catalog to locate the actual article?<o:p></o:p></span></p> <p class=msolistparagraphcxspfirst style='margin:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:22.5pt;line-height:normal;mso-pagination:none; page-break-after:avoid;mso-list:l14 level1 lfo24'><![if !supportlists]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-fareast-font-family: "palatino linotype";mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>a.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>author search: thomas <o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:22.5pt;line-height:normal;mso-pagination:none; page-break-after:avoid;mso-list:l14 level1 lfo24'><![if !supportlists]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-fareast-font-family: "palatino linotype";mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>b.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>journal title search: library collections, acquisitions &amp; technical services <o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:22.5pt;line-height:normal;mso-pagination:none; page-break-after:avoid;mso-list:l14 level1 lfo24'><![if !supportlists]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-fareast-font-family: "palatino linotype";mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>c.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>journal title search: this is not a dumpsite: the problem of evaluating gift books <o:p></o:p></span></p> <p class=msolistparagraphcxsplast style='margin:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:22.5pt;line-height:normal;mso-pagination:none; page-break-after:avoid;mso-list:l14 level1 lfo24'><![if !supportlists]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-fareast-font-family: "palatino linotype";mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>d.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>subject search: gift books<o:p></o:p></span></p> <p class=msonormal style='mso-pagination:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='mso-pagination:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>question 21<o:p></o:p></span></p> <p class=msolistparagraph style='margin-left:0in;mso-add-space:auto;line-height: normal;mso-pagination:none'><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>the following item was retrieved from a database search. what kind of source is it? <o:p></o:p></span></p> <p class=msonormalcxspfirst style='margin-left:.5in;mso-add-space:auto; mso-pagination:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>title: the effect of library instruction learning environments on self-efficacy levels and learning outcomes of graduate students in education <o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto; mso-pagination:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>author(s): beile, penny<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto; mso-pagination:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>publication year: 2002<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto; mso-pagination:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>abstract: the purpose of the study was to examine the effectiveness of three learning environments: (1) campus-based students who attended a classroom library instruction session; (2) campus-based students who completed a web-based library tutorial; and (3) distance students who completed a web-based library tutorial on library skills self-efficacy levels and learning outcomes among graduate students of education.<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto; mso-pagination:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>notes: presented at the annual meeting of the american educational research association (new orleans, la, april 1-5, 2002)<o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto; mso-pagination:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>number of pages: 8 <o:p></o:p></span></p> <p class=msonormalcxspmiddle style='margin-left:.5in;mso-add-space:auto; mso-pagination:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>accession number: ed453084<o:p></o:p></span></p> <p class=msonormal style='text-indent:22.5pt;mso-pagination:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>a. a book <o:p></o:p></span></p> <p class=msonormal style='text-indent:22.5pt;mso-pagination:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>b. a book chapter <o:p></o:p></span></p> <p class=msonormal style='text-indent:22.5pt;mso-pagination:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>c. a conference paper <o:p></o:p></span></p> <p class=msonormal style='text-indent:22.5pt;mso-pagination:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>d. a journal article<o:p></o:p></span></p> <p class=msonormal style='mso-pagination:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='mso-pagination:lines-together;page-break-after:avoid'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>question 23<o:p></o:p></span></p> <p class=msolistparagraph style='margin-left:0in;mso-add-space:auto;line-height: normal;mso-pagination:lines-together;page-break-after:avoid'><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>while researching the u.s. legislative system, you find the following story on the internet:<o:p></o:p></span></p> <p class=msonormalcxspfirst style='margin-left:.5in;mso-add-space:auto; mso-pagination:lines-together;page-break-after:avoid'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>congress launches national congress-awareness week washington, dc—hoping to counter ignorance of the national legislative body among u.s. citizens, congressional leaders named the first week in august national congress awareness week. “this special week is designed to call attention to america’s very important federal lawmaking body,” speaker of the house dennis hastert said. the festivities will kick off with a 10-mile walk for congress awareness. the item is from a newspaper web site, which states it is “america’s finest news source.” <o:p></o:p></span></p> <p class=msonormal style='mso-pagination:lines-together;page-break-after:avoid'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msonormal style='mso-pagination:lines-together;page-break-after:avoid'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>given this, the following action is in order:<o:p></o:p></span></p> <p class=msolistparagraphcxspfirst style='margin:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:22.5pt;line-height:normal;mso-pagination:lines-together; page-break-after:avoid;mso-list:l17 level1 lfo25'><![if !supportlists]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-fareast-font-family: "palatino linotype";mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>a.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>you can use the story as it’s obviously from a reputable news source<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:22.5pt;line-height:normal;mso-pagination:lines-together; page-break-after:avoid;mso-list:l17 level1 lfo25'><![if !supportlists]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-fareast-font-family: "palatino linotype";mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>b.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>you decide to investigate the reputation of the publisher by looking at their web site<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:22.5pt;line-height:normal;mso-pagination:lines-together; page-break-after:avoid;mso-list:l17 level1 lfo25'><![if !supportlists]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-fareast-font-family: "palatino linotype";mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>c.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>you decide to investigate the reputation of the publisher by looking at other web sites<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin:0in;margin-bottom:.0001pt; mso-add-space:auto;text-indent:22.5pt;line-height:normal;mso-pagination:lines-together; page-break-after:avoid;mso-list:l17 level1 lfo25'><![if !supportlists]><span style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-fareast-font-family: "palatino linotype";mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>d.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>you should not use the story because web information is not always trustworthy<o:p></o:p></span></p> <p class=msolistparagraphcxspmiddle style='margin-left:0in;mso-add-space:auto; line-height:normal;mso-pagination:none'><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'><o:p>&nbsp;</o:p></span></p> <p class=msolistparagraphcxsplast style='margin-left:0in;mso-add-space:auto; line-height:normal;mso-pagination:none'><span style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'>question 24<o:p></o:p></span></p> <p class=msonospacing style='mso-pagination:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif;mso-bidi-font-family: calibri;mso-bidi-theme-font:minor-latin'>based on the following paragraph, which sentence should be cited? <o:p></o:p></span></p> <p class=msonospacingcxspmiddle style='margin-left:.5in;mso-add-space:auto; mso-pagination:none'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-bidi-font-family:calibri;mso-bidi-theme-font:minor-latin'>(1) libraries were once quiet spaces reserved for readers. (2) as libraries increased their community programming, they began to shift to the more social (and unquiet) places with which we are familiar today. (3) many libraries try to preserve some aspects of their quiet past while continuing to offer engaging programing. (4) the public seems to want this as well; in a pew research poll, 61% of americans said that they believe libraries should have completely separate locations or spaces for quiet and social activities.<o:p></o:p></span></p> <p class=msonospacingcxspmiddle style='margin-left:0in;mso-add-space:auto; text-indent:22.5pt;mso-pagination:none;mso-list:l18 level2 lfo26'><![if !supportlists]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"palatino linotype";mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>a.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span lang=en-ca style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'>1 <o:p></o:p></span></p> <p class=msonospacingcxspmiddle style='margin-left:0in;mso-add-space:auto; text-indent:22.5pt;mso-pagination:none;mso-list:l18 level2 lfo26'><![if !supportlists]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"palatino linotype";mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>b.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span lang=en-ca style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'>2 <o:p></o:p></span></p> <p class=msonospacingcxspmiddle style='margin-left:0in;mso-add-space:auto; text-indent:22.5pt;mso-pagination:none;mso-list:l18 level2 lfo26'><![if !supportlists]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"palatino linotype";mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>c.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span lang=en-ca style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'>3 <o:p></o:p></span></p> <p class=msonospacingcxsplast style='margin-left:0in;mso-add-space:auto; text-indent:22.5pt;mso-pagination:none;mso-list:l18 level2 lfo26'><![if !supportlists]><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif; mso-fareast-font-family:"palatino linotype";mso-bidi-font-family:"palatino linotype"'><span style='mso-list:ignore'>d.<span style='font:7.0pt "times new roman"'>&nbsp;&nbsp;&nbsp; </span></span></span><![endif]><span lang=en-ca style='font-size:10.0pt; font-family:"palatino linotype",serif;mso-bidi-font-family:calibri;mso-bidi-theme-font: minor-latin'>4<o:p></o:p></span></p> <p class=msonormal style='page-break-after:avoid;mso-outline-level:2'><span lang=en-ca style='font-size:10.0pt;font-family:"palatino linotype",serif'><o:p>&nbsp;</o:p></span></p> </div> </body> </html> article   charting success: using practical measures to assess information literacy skills in the first-year writing course   ann elizabeth donahue interim dean dimond library university of new hampshire durham, new hampshire, united states of america email: annie.donahue@unh.edu   received: 21 feb. 2015    accepted: 13 may 2015        2015 donahue. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   abstract   objective – the aim was to measure the impact of a peer-to-peer model on information literacy skill-building among first-year students at a small commuter college in the united states. the university of new hampshire (unh) is the state’s flagship public university and unh manchester is one of its seven colleges. this study contributed to a program evaluation of the research mentor program at unh manchester whereby peer writing tutors are trained in basic library research skills to support first-year students throughout the research and writing process.   methods – the methodology employed a locally developed pre-test/post-test instrument with fixed-choice and open-ended questions to measure students’ knowledge of the library research process. anonymized data was collected using an online survey with surveymonkey™ software.  a rubric was developed to score the responses to open-ended questions.   results – the study indicated a positive progression toward increased learning for the three information literacy skills targeted: 1) using library resources correctly, 2) building effective search strategies, and 3) evaluating sources appropriately. students scored higher in the fixed-choice questions than the open-ended ones, demonstrating their ability to more effectively identify the applicable information literacy skill than use the language of information literacy to describe their own research behavior.   conclusions – the assessment methodology used was an assortment of low-key, locally-developed instruments that provided timely data to measure students understanding of concepts taught and to apply those concepts correctly. although the conclusions are not generalizable to other institutions, the findings were a valuable component of an ongoing program evaluation.  further assessment measuring student performance would strengthen the conclusions attained in this study. introduction   due to limited budgetary and staffing issues, small academic libraries within the united states face a cornucopia of challenges when delivering a broad spectrum of services to their constituents. these challenges often engender innovative and creative solutions that yield delightful and unexpected outcomes. the research mentor program at the university of new hampshire (unh) manchester is one of those happy circumstances. through this program, research mentors become the conduit whereby the librarians are able to extend academic support beyond the library walls to reach first-year students at each stage of the research process – from brainstorming topics; developing effective search strategies;  evaluating sources to preparing outlines; developing thesis statements; and drafting through the writing/revision cycle.   in the research mentor program, the library partnered with the college’s center for academic enrichment (cae) to improve students’ information literacy skills in all first-year writing courses. one critical component of this collaboration was the incorporation of peer writing tutors trained in basic library research skills who worked side-by-side with the instruction librarians in the classroom as research mentors to first-year students. the unh manchester librarians recognize research and writing as an integrated process and used this approach to provide these students with essential support throughout the research process. within the classroom, research mentors worked with librarians to model effective research strategies. outside the classroom, they worked directly with students in individualized tutorials.   small class size and teaching excellence are hallmarks of unh manchester. first-year writing courses are capped at 15 students and generally six sections of the course are offered each semester. the library’s information literacy instructional plan includes three 90-minute sessions per section to scaffold learning in manageable units each building upon the previous unit. this intense delivery model is a deliberate effort to meet students' developmental readiness levels and to embed information literacy into the curriculum of the composition program.   the genesis for the research mentor program came from an idea presented in a poster session at an association of college and research libraries (acrl) annual conference. the original design utilized students trained in basic library research techniques to assist other students with their research projects at evening and week-end drop-in sessions held in the residence halls. by modifying the delivery method to accommodate a commuter campus, capitalizing upon the college's collaborative culture and partnering with the cae’s successful peer tutoring enterprise, the unh manchester library was able to experiment with an innovative, student-centered approach to increasing information literacy competencies (fensom, mccarthy, rundquist, sherman, & white, 2006; white & pobywajlo, 2005).   the program has evolved since its inception in 2004. although originally focused on serving the students in the first-year writing course, in the fall semester of 2013 the program reach was extended to include the use of peer research mentors across the disciplines and in upper-level courses. each of the three members of the library's instruction team had a significant role in ensuring the success of the program. the information literacy instruction coordinator partnered with the director of the cae to design and teach the two to four credit-bearing tutor development course required of each peer writing tutor. the information literacy specialist developed the course objectives and delivered instruction for all sections, partnering with the research mentors to include modeling of best practice techniques through a peer-to-peer lens. the library director collaborated with the instruction team to craft effective assessment instruments, liaised with the teaching faculty and administration to ensure adherence to research protocol, and analyzed the data collected.   during the first seven years of the program, anecdotal evidence suggested the program was a successful one, but a systematic evaluation that provided clear evidence was long overdue. in the academic year 2011, the library instruction team planned and implemented the first phase of a program evaluation to gather data to assess the impact of this peer-to-peer model on student learning. beginning with a pilot study in the spring 2011 semester, the study continued through the next two semesters resulting in data that highlighted strengths and indicated areas for improvement. this paper discusses selected quantitative and qualitative findings from this eighteen-month study measuring the effectiveness of delivering information literacy through a peer-to-peer approach, replacing the traditional one-shot library instruction methodology with semester-long engagement in information literacy skill-building.   literature review   the professional literature describes a variety of collaborations that exist between the academic library and the college writing centre. some examples defined shared-space arrangements leading to mutually beneficial opportunities that enhanced student services (currie & eodice, 2005; foutch, 2010; giglio & strickland, 2005). other examples described joint workshops led by instruction librarians and the professional writing staff focused on improving student learning outcomes (artz, 2005; boff & toth, 2005; cooke & bledsoe, 2008; leadley & rosenberg, 2005). further examples discussed the use of peer tutors serving in an assortment of roles from marketing ambassadors to basic research support assistants (cannon & jarson, 2009; deese-roberts & keating, 2000; furlong & crawford, 1999; gruber, knefel & waelchli, 2008; lowe & lea, 2004; millet & chamberlain, 2007).   when library collaborations with writing centres utilized student peer tutors rather than professional staff a new dimension – peer-to-peer learning – made it possible to extend the reach of the librarians beyond the instruction class. when these collaborations involved an aspect of research or instruction assistance, various levels of training were incorporated to prepare these student peer tutors to develop the basic skills necessary for engaging with research strategies and processes. this training provided the peer tutors with critical foundational skills that enabled them to directly respond to research questions that arose during writing tutorials.   a classroom clinic, co-led by instruction librarians and student peer tutors, is described in an article by gruber et al. (2008). this collaboration was crafted to respond to assignment-specific objectives that reflected information literacy standards and effective writing criteria. the alliance between librarian, faculty, and peer tutor enabled the students in the course to participate in small group experiences, facilitated by either the librarian or peer tutor, in order to grapple with identifying the key elements of scholarly inquiry and evaluating academic journal articles. at the university of new mexico, deese-roberts and keating (2000) discussed the collaboration between the library and the writing centre whereby peer writing tutors were trained by librarians in “five key concept areas: (1) library services and policies; (2) search strategies; (3) boolean logic, search logic, and limits; (4) vocabulary (controlled vs. natural); and (5) database structure” (p. 225). peer writing tutors then worked with students on research and writing projects. assessment of the pilot program indicated positive feedback from all stakeholders. the assessment focused on user satisfaction and participation. student participation in the program “increased 100% from the first to the second semester” (p. 228) inspiring the authors to declare the pilot program a success.     elmborg (2005) suggested that peer tutors work effectively because they “understand the student perspective . . . they live that perspective” (p. 15). nelson (1995) proposed that peer tutors were well situated to assist less capable students because they empathized and guided comprehension more effectively since they “speak the language of other undergraduates more distinctly than graduate students and professors” (p. 45). lowe and lea (2004) defined the peer tutor in an academic setting as “a person who helps you over bumps and makes you realize that you really can do it – whatever it is – by yourself” (p. 134).   several academic libraries have incorporated undergraduate students in their instruction programs. the role of these students varied from facilitating small group discussions (gruber et al., 2008) to roaming the classroom providing assistance during hands-on activities (deese-roberts & keating, 2000) to teaching mini-seminars on specific library resources (holliday & nordgren, 2005). as the demand for library instruction in lower-division general education courses grew to unsustainable levels, librarians at california polytechnic state university implemented a “student-based solution” (bodemer, 2013, p. 578). undergraduate students serving as reference assistants received additional training in instructional design, were designated as peer instructors, and worked alongside the librarian in the classroom. the online evaluations for each session showed that students ranked these peer instructors higher than the librarians on an affective scale (bodemer). based on these evaluations, the student peer instructors were assigned to lead basic information literacy sessions independently.   at unh manchester, the peer tutor program was already a college reading and learning association certified program that was highly effective and recognized the benefits of students helping students. by enhancing the writing tutor’s toolkit with information literacy skills and integrating them into the instruction sessions to model good research behaviour, these research mentors became better equipped to guide first-year students through the entire research process.        aims   the impetus for undertaking a program evaluation study was the imminent retirement of the director of the cae. as the search for a new director began, it became apparent that there was no measurable evidence available to support continuation of a program deemed valuable to the stakeholders. whenever the program's value was discussed, its success was attributed to the connections forged through "a network of people dedicated to helping [students] achieve their academic goals” (white & pobywajlo, 2005). yet no data existed to support this claim as no evidence that students' achieved their goals was ever collected. it was time to formalize assessment and develop a plan that would measure the impact of the program. in fall 2009, the information literacy instruction team began building an assessment plan to evaluate the program. although it was agreed that improving teaching and learning were important goals for this evaluation, demonstrating the program's effectiveness and value to ensure the continuation of the program was an essential purpose for this study.   a review of the program objectives identified by both the library and the cae suggested a three-phased approach for the program evaluation plan: 1) measure change in students' information literacy skills in first-year writing courses and their self-perceptions of confidence with the research process, 2) examine peer tutor experiences and their perceptions of self-development as a result of participating in the program, and 3) investigate faculty perceptions of their students' learning outcomes attributable to the program's peer-to-peer model.    both departments shared common objectives for student success that focused primarily on increasing critical thinking, improving research and writing skills, and giving students the tools to become information literate. these objectives became the goals measured during the initial phase of the program evaluation. the aim of the program evaluation was to measure the impact of a peer-to-peer model on information literacy skill-building among first-year students. this paper presents selected results from the initial phase of the program evaluation which measured the impact on information literacy skills.   methodology   the study received institutional review board  protocol approval in january 2011, and a pilot study was implemented that spring semester. all students enrolled in a first-year writing course were invited to participate in the study. the size of the college (approximately 900 undergraduates) resulted in a small pool of potential participants. although random sampling was a desired method, the capped enrolments in these courses made convenience sampling the most logical approach to obtain a reasonably-sized data pool. participation was voluntary, and students could opt to leave the study at any time during the semester.   several quantitative and qualitative measures were designed to assess the goals identified for this study. a pre-test/post-test instrument (appendix a) measured students' knowledge about the library research process by asking students to respond to questions, both fixed-choice and open-ended, thereby demonstrating competency levels for defining, investigating, and evaluating an information need.   the pre-test instruments were administered on the first day of the course during the pilot semester, but in subsequent semesters pre-tests were given during the second week of classes. this brief delay was designed to allow students time to understand course expectations before making a decision about participating in the study. results of the pre-test formed a baseline measure of students' abilities and were available to the librarian prior to the first information literacy instruction session. then, in the penultimate class, the post-test instruments were administered. assessment instruments were administered online using surveymonkey™ software in one of the college's computer classrooms during normal class hours.   a rubric (appendix b) was used to measure the open-ended questions, but with limited experience in designing and using rubrics a review of the literature was a necessary first step (brown, 2008; crowe, 2010; daniels, 2010; diller & phelps, 2008; fagerheim & shrode, 2009; gardner & acosta, 2010; knight, 2006; oakleaf, 2008, 2009a, 2009b; oakleaf, millet & kraus, 2011). in the rubric design, aligning the criteria to the objectives of the first-year information literacy curriculum provided the framework within which to craft the measures. a valuable source for examples of designing and using rubrics was found at the rails (rubric assessment of information literacy skills) website (http://railsontrack.info/).   results   the sample size was small for each semester but consistent with enrolment patterns for the college. during the pilot semester (spring 2011), 54 students enrolled in the first-year writing course but only 31 students agreed to participate in the study. the 57% participation rate was disappointing and attributed to asking students to participate by completing the pre-test on the first day of class before students had any understanding of the class expectations. in each subsequent semester, the invitation to participate and the administration of the pre-test occurred during the second week of class resulting in a 100% participation rate each semester. in fall 2011, the sample size was 76 students and in spring 2012, the sample size was 48 students. attrition rates for first-year writing significantly affected the post-test sample size in every semester. in spring 2011, only 28 students remained in the study. in fall 2011, the post-test was completed by 55 students and in spring 2012, the post-test sample size numbered 32.   the pre-test/post-test instrument included six questions designed to identify students' previous library research experiences and an additional nine questions focused on three acrl information literacy competency standards: 1) the information literate student identifies a variety of types and formats of potential sources of information; 2) the information literate student constructs and implements effectively-designed search strategies; and 3) the information literate student articulates and applies initial criteria for evaluating both the  information and its sources (acrl, 2000).   among the nine information literacy questions were three clusters of three questions that directly mapped these standards as learning objectives assigned to the information literacy instruction sessions delivered in the first-year writing course. using a cluster approach enabled students to demonstrate knowledge of each learning objective by answering a set of three questions that explored a single information literacy competency from multiple perspectives. each cluster included two fixed-choice questions and one open-ended question. a fixed-choice question was written as an informational inquiry while the second was placed within the context of a potential research scenario. the open-ended question required students to describe the research activities they would complete to accomplish the task presented in the question. the results of these cluster questions are discussed here.   table 1 shows the results for the two fixed-choice questions in each cluster. findings indicated improvement each semester in five out of six questions. the question that indicated a lack of improvement was the question that measured the ability to evaluate sources in the research scenario format. in post-test results for this question, students in spring 2011 scored an 11% increase over pre-test results, but fall 2011 students scored a 7% decrease from their pre-test results. in spring 2012, this question yielded no change in students’ pre-test to post-test results.   results for the remaining five questions point toward an increase in knowledge over the baseline measure; the percent of change across the remaining cluster questions ranged from a 6% to 57% increase. table 1 visually depicts the quantitative results for each semester for both the informational inquiry and the scenario based formats.    tables 2, 3 and 4 show the results of the final question in each cluster set; an open-ended question requiring students to demonstrate the research skills they would employ in response to the task described. once again, each cluster question mapped to one of the information literacy competency standards identified above.     table 1 results of the fixed-choice questions cluster sets: il standards 1-3 pretest spring 2011 post-test spring 2011 pre-test fall 2011 post-test fall 2011 pre-test spring 2012 post-test spring 2012 library resources – info inquiry 68% 86% 48% 82% 62% 81% library resources – scenario based 74% 100% 61% 82% 75% 81%               search strategies – info inquiry 32% 89% 43% 84% 53% 90% search strategies – scenario based 16% 68% 28% 47% 28% 44%               source evaluation – info inquiry 55% 79% 76% 82% 64% 84% source evaluation – scenario based 74% 85% 80% 73% 78% 78%     table 2 information literacy standard one – determine the nature and extent of information needed ratings pre-test spring 2011 post-test spring 2011 pre-test fall 2011 post-test fall 2011 pre-test spring 2012 post-test spring 2012 novice 71% 57% 38% 22% 30% 31% emerging 23% 36% 43% 27% 35% 50% intermediate 6% 7% 14% 36% 28% 16% advanced n/a n/a 5% 15% 7% 3% expert n/a n/a n/a n/a n/a n/a     a rubric was developed to translate qualitative responses into quantitative scores. the rubric scored students’ results on a five-point scale from novice to expert, based on the number of criteria students identified for each competency.   the first cluster set measured students’ ability to define their information need. the seven criteria identified in acrl’s information literacy standard one (acrl, 2000) were incorporated into the rubric used to score students’ responses. the rubric allowed for five rating levels determined by the number of criteria students listed in their responses. the rankings of novice to expert were based on students’ naming the criteria associated with the standard. when students described their research process by articulating one or no criteria they ranked at the novice level, two criteria ranked at the emerging level, three criteria ranked at the intermediate level, four or five criteria ranked at the advanced level, and six or more criteria ranked at the expert level.   table 2 shows the rankings for information literacy standard one. results indicated students’ skill levels improved across most semesters, as noted by a drop in novice rankings and a rise in emerging or intermediate rankings. among the seven criteria measured, students demonstrated notable growth in three areas: 1) explores general information sources to increase familiarity with the topic, 2) identifies key concepts and terms that describe the information need, and 3) defines and modifies the information need to achieve a manageable focus.   the second cluster set measured students' ability to construct an effective search strategy. four criteria identified in acrl’s information literacy standard two (acrl, 2000) were incorporated into the rubric used to score students’ responses. although students in each semester scored well in the pre-test on one criterion, identified keywords, synonyms, and related terms for information need, approximately one-third of students' responses denoted no search strategy at all. post-test scores demonstrated that "no search strategy" responses were reduced by 50% and that search strategies using a combination of keywords with boolean operators increased significantly; by 33% in spring 2011, 47% in fall 2011, and 19% in spring 2012.   table 3 demonstrates the change in rankings across the three semesters. when students described their search strategy, if they merely repeated the topic phrase or gave no answer they ranked at the novice level; if they identified keywords and related terms they ranked at the emerging level; and if they identified keywords and used boolean operators they ranked at the intermediate level. although no students incorporated all four criteria denoted for this information literacy standard, results demonstrated improvement as novice rankings decreased and intermediate rankings increased.   the third cluster set asked students to name the criteria they used to evaluate sources. five criteria identified in acrl’s information literacy standard three (acrl, 2000) were incorporated into the rubric used to score students’ responses. when students described the criteria used to evaluate sources, a response with one or no criteria was ranked at the novice level, two criteria ranked at the emerging level, three criteria ranked at the intermediate level, four criteria ranked at the advanced level, and five criteria ranked at the expert level.   table 4 shows the rankings for information literacy standard three. in both spring 2011 and fall 2011 semesters, rankings indicated that students increased skill levels, however, spring 2012 results reflected no improvement for this competency. across all semesters in pre-test results, most students identified a single criterion as sufficient to evaluate a resource. the top three criteria noted were: 1) accuracy and authority, 2) timeliness, and 3) relevancy. post-test scores for these three criteria remained strong in each semester, but the notable change was that students regularly identified more than one criterion for evaluating sources in the post-test data.     table 3 information literacy standard two – access needed information effectively and efficiently ratings pre-test spring 2011 post-test spring 2011 pre-test fall 2011 post-test fall 2011 pre-test spring 2012 post-test spring 2012 novice 32% 14% 27% 11% 14% 12% emerging 68% 54% 57% 27% 72% 50% intermediate 0 32% 16% 62% 14% 38% advanced n/a n/a n/a n/a n/a n/a expert n/a n/a n/a n/a n/a n/a     table 4 information literacy standard 3 – evaluate information and its sources critically ratings pre-test spring 2011 post-test spring 2011 pre-test fall 2011 post-test fall 2011 pre-test spring 2012 post-test spring 2012 novice 65% 39% 39% 27% 33% 34% emerging 32% 39% 32% 40% 35% 34% intermediate 0 22% 26% 27% 32% 32% advanced 3% 0 3% 6% n/a n/a expert n/a n/a n/a n/a n/a n/a     discussion   the data collected in this phase of the evaluation study indicated a positive progression in student learning. students demonstrated growth of information literacy skills throughout the semester. however, there are several limitations in this study that make generalization of the findings impractical. the overall sample size was small and the use of convenience sampling, rather than random sampling, may not capture a true representation of first-year students' abilities. high attrition rates in first-year writing courses led to lower post-test responses which can impact accurate analysis of pre-test/post-test comparison data leading to a potentially false conclusion.   the fixed-choice test methodology incorporates further potential limitations. the questions measure students' knowledge of facts, but tend to “measure recognition rather than recall” (oakleaf, 2008, p. 236) which is an indirect assessment of students’ knowledge but not necessarily a measure of students’ ability to apply that knowledge appropriately. on the positive side, this methodology is easily administered and analyzed; it is locally-specific and allows for timely measurement of the objectives from each information literacy instruction session. with the data collected in this study, the librarian can adapt lesson plans and activities to respond to students' developmental readiness level more fully.   the open-ended questions gave students the opportunity to articulate their research behaviour, enabling a more direct measurement of their ability to apply information literacy skills. a rubric was an effective scoring mechanism to convert the qualitative responses to a quantitative measure that could be analyzed against the results of the other two cluster set questions. although the rubric made scoring results possible, the process was considerably more time-consuming than anticipated. this methodology also contributed to potential limitations in the study due to the use of a single rater to score results. although effort was employed to maintain an objective scoring plan, it was challenging to interpret students' responses consistently when scoring at "different points in time" (oakleaf, 2009b, p. 970). use of trained student raters has been an efficient and effective approach at other institutions and may be appropriate in future rubric scoring to increase reliability of the results (knight, 2006).   this 18-month study was undertaken beginning in spring 2011 and the results of this study were presented at the library assessment conference in october 2012. the positive results of this study encouraged the unh manchester librarians to expand the reach of the research mentor program beyond the first-year writing courses. the credit-bearing tutor development course was revised to include training in subject-specific databases. this study used the acrl information literacy competency standards as criteria for evaluating students’ information seeking skills. in february 2015, the acrl board affirmed the framework for information literacy for higher education. as librarians incorporate the six concepts of the framework into the information literacy curriculum, a further study of this peer-to-peer learning approach would be a valuable addition to the research mentor program evaluation.    conclusion   this paper examined the findings from a selected section of the pre-test/post-test instrument used to measure change in student learning in our first-year writing course. through this study, an historical snapshot of the effectiveness of employing a peer-to-peer learning approach with first-year students emerged. the primary assessment instrument incorporated three cluster sets of fixed-choice and open-ended questions mapped to the curriculum objectives for information literacy instruction, and the findings demonstrated a positive progression toward increased learning in the three targeted areas identified: 1) using library resources correctly, 2) building effective search strategies, and 3) evaluating sources appropriately. students scored higher in the fixed-choice questions than the open-ended ones, demonstrating the ability to more effectively identify the applicable information literacy skill than use the language of information literacy to describe their own research behavior. the findings, although specific to the college’s local situation and not generalizable, are a valuable baseline for informing teaching and learning practice.   the method used was a low-key, locally-developed instrument that provided timely data to measure students understanding of concepts taught and to apply those concepts correctly. this instrument provided an indirect assessment of students’ learning by relying on their ability to recognize the correct response from a selection of possible options. this approach is easily administered and analyzed but results demonstrated that students were better able to recognize components of the research process when given choices than articulate the steps they would undertake when conducting research. further assessment that directly measured student performance would strengthen the conclusions attained in this study. although the conclusions are not generalizable to other institutions, the findings were a valuable component of an ongoing program evaluation.   references   artz, j. (2005). libraries and learning center collaborations: within and outside the walls. in j. k. elmborg & s. hook (eds.), centers for learning: writing centers and libraries in collaboration (pp.93-115). chicago, il: association of college & research libraries.   association of college & research libraries. (2000). information literacy competency standards for higher education. chicago, il: association of college & research libraries.   bodemer, b. b. (2013, apr.). they not only can but they should: why undergraduates should provide basic il instruction. paper presented at the association of college and research libraries annual conference, indianapolis, il, usa.   boff, c., & toth, b. (2005). better-connected student learning: research and writing project clinics at bowling green state university. in j.k. elmborg & s. hook (eds.), centers for learning: writing centers and libraries in collaboration (pp. 148-157). chicago, il: association of college & research libraries.   brown, c. a. (2008). building rubrics: a step-by-step process. library media connection, 26(4), 16-18.   cannon, k., & jarson, j. (2009). information literacy and writing tutor training at a liberal arts college. communications in information literacy, 3(1), 45-55. retrieved from http://www.comminfolit.org/index.php?journal=cil   cooke, r., & bledsoe, c. (2008). writing centers and libraries: one-stop shopping for better term papers. the reference librarian, 49 (2), 119-127. http://dx.doi.org/10.1080/02763870802101310   crowe, k. (2010, oct.). assessment = improved teaching and learning: using rubrics to measure information literacy skills. paper presented at the 2010 library assessment conference in baltimore, md, usa. retrieved from: http://libraryassessment.org/archive/2010.shtml#2010_schedule   currie, l., & eodice, m. (2005). roots entwined: growing a sustainable collaboration. in j.k. elmborg & s. hook (eds.), centers for learning: writing centers and libraries in collaboration (pp. 42-60). chicago, il.: association of college & research libraries.   daniels, e. (2010). using a targeted rubric to deepen direct assessment of college students' abilities to evaluate the credibility of sources. college & undergraduate libraries, 17(1), 31-43. http://dx.doi.org/10.1080/10691310903584767   deese-roberts, s., & keating, k. (2000). integrating a library strategies peer tutoring program. research strategies, 17(2/3), 223-229. http://dx.doi.org/10.1016/s0734-3310(00)00039-2   phelps, s. f. (2008). learning outcomes, portfolios, and rubrics, oh my! authentic assessment of an information literacy program. portal: libraries and the academy, 8(1), 75-89. http://dx.doi.org/10.1353/pla.2008.0000   elmborg, j. k. (2005). libraries and writing centers in collaboration: a basis in theory. in j.k. elmborg & s. hook (eds.), centers for learning: writing centers and libraries in collaboration (pp. 1-20). chicago, il.: association of college & research libraries.   fagerheim, b. a., & shrode, f. g. (2009). information literacy rubrics within the disciplines. communications in information literacy, 3(2), 158-170. retrieved from http://www.comminfolit.org/index.php?journal=cil   fensom, g., mccarthy, r., rundquist, k., sherman, d., & white, c. b. (2006). navigating research waters: the research mentor program at the university of new hampshire at manchester. college & undergraduate libraries, 13(2), 49-74. http://dx.doi.org/10.1300/j106v13n02_05           foutch, l. j. (2010). joining forces to enlighten the research process: a librarian and writing studio integrate. college & research libraries news, 71 (7), 370-373.retrieved from: http://crln.acrl.org/   furlong, k., & crawford, a. b. (1999). marketing your services through your students. computers in libraries, 19 (8), 22-26.   gardner, s., & acosta, e. s. (2010, oct.). using a rubric to assess freshman english library instruction. paper presented at the 2010 library assessment conference in baltimore, md, usa. retrieved from: http://libraryassessment.org/archive/2010.shtml#2010_schedule      giglio, m. r. & strickland, c. f. (2005). the wesley college library and writing center: a case study in collaboration. in j. k. elmborg & s. hook (eds.), centers for learning: writing centers and libraries in collaboration (pp. 138–147). chicago, il.: association of college & research libraries.   gruber, a. m., knefel, m. a., & waelchli, p. (2008). modeling scholarly inquiry: one article at a time. college & undergraduate libraries, 15 (1-2), 99125. http://dx.doi.org/10.1080/10691310802177085   holliday, w. & nordgren, c. (2005, april).  extending the reach of librarians: library peer mentor program at utah state university. college & research libraries news, 66(4), 282-284. retrieved from: http://crln.acrl.org/   knight, l. a. (2006). using rubrics to assess information literacy. reference services review, 34(1), 43-55. http://dx.doi.org/10.1108/00907320610640752   leadley, s., & rosenberg, b. r. (2005). yours, mine, and ours: collaboration among faculty, library, and writing center. in j. k. elmborg & s. hook (eds.), centers for learning: writing centers and libraries in collaboration (pp. 61-77). chicago: association of college & research libraries.   lowe, m., & lea, b. (2004). when worlds collide: libraries & writing centers. academic exchange quarterly, 8(1), 134-138. retrieved from: http://rapidintellect.com/aeqweb/      millet, m. s., & chamberlain, c. (2007). word-of-mouth marketing using peer tutors. the serials librarian, 53(3), 95-105. http://dx.doi.org/10.1300/j123v53n03_07   nelson, r. r. (1995/96). peer tutors at the collegiate level: maneuvering within the zone of  proximal development. journal of college reading and learning, 27(1), 43-51. http://dx.doi.org/10.1080/10790195.1996.10850030   oakleaf, m. (2008). dangers and opportunities: a conceptual map of information literacy assessment approaches. portal: libraries and the academy, 8(3), 233-253. http://dx.doi.org/10.1353/pla.0.0011   oakleaf, m. (2009a). the information literacy instruction assessment cycle: a guide for increasing student learning and improving librarian instructional skills. journal of documentation, 65(4), 539-560. http://dx.doi.org/10.1108/00220410910970249   oakleaf, m. (2009b). using rubrics to assess information literacy: an examination of methodology and inter-rater reliability. journal of the american society for information science & technology, 60(5), 969-983. http://dx.doi.org/10.1002/asi.21030   oakleaf, m., millet, m. s., & kraus, l. (2011). all together now: getting faculty, administrators, and staff engaged in information literacy assessment. portal: libraries and the academy, 11(3), 831-852. http://dx.doi.org/10.1353/pla.2011.0035   white, c. b., & pobywajlo, m. (2005). a library, learning center, and classroom collaboration: a case study.  in j.k. elmborg & s. hook (eds.), centers for learning: writing centers and libraries in collaboration (pp. 175-203). chicago, il: association of college & research libraries.     appendix a questionnaire                 appendix b information literacy rubric evidence based library and information practice using evidence in practice   terminology for librarian help on the home page   leni matthews user experience librarian university of texas at arlington arlington, texas, united states of america email: leni@uta.edu   received: 7 feb. 2018      accepted: 17 apr. 2018      2018 matthews. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29405   setting   the university of texas at arlington (uta) is a public university serving over 40,000 students, reaching nearly 50,000 when online students are calculated. there are three libraries on campus: central library, architecture and fine arts library, and science and engineering library. central library has six floors including a basement. students in this study were recruited from all three libraries. uta libraries have a diverse user community and uta itself was named one of the most diverse campuses in the us in a study done by the us news and world report (2018).   problem   librarians at uta began an email conversation about terminology to use on the home page that is easy to understand for users. one of our librarians stated that we should “include more areas of expertise that work directly with our users” while creating language that would make librarians findable on the home page. the librarians began to propose terms they thought users would choose to search for them on the library home page. instead of weighing in on librarian-proposed terms from only librarians, we decided to ask our user community since they are users or potential users of the home page. to improve the way users access our “expertise” for assistance, we must include them to understand their search process.   our librarians have backgrounds in areas such as open educational resources (oer) and data literacy that exist outside their normal duties. we want to market those skills to make our librarians more accessible to users. first, we needed to figure out which labels users preferred from the librarian-proposed terms when they searched for a librarian on our home page. were we generating user-friendly terminology that was intuitive for librarian discoverability? we also wanted to figure out what they expected to “see” once a particular label was chosen by asking their expectations for the landing page. determining this expectation would help us discover users’ understanding of these labels.   evidence   at uta, the digital creation and assessment departments collaborated to conduct a usability study to find out the terms users preferred when they searched for librarian assistance on the home page. a paper prototype of the library home page was used where students pointed to terms they thought would help them find librarians. term refers to the labels or language used on the home page that link to other webpages. librarians proposed terms they thought would help users to locate them. these terms were shared via email and open to all twenty-one of our librarians. some librarians proposed new terms, while others agreed with some of the suggested terms. we added these terms on the home page.   the paper prototype looked similar to figure 1.   there were a total of 26 labels for the right side of the home page, including the terms proposed by librarians. the dropdown menus were expanded on the paper prototype making it easier to “navigate.” this paper will focus on the usability study for the right-side of the home page. this is where the original term (librarians by subject) that linked directly to librarians was located.   librarians proposed seven terms/labels they thought would be useful when users searched for a librarian:   librarians by subject (previous – before the usability test)   librarians by academic subject (current – after the usability test) librarians by expertise librarians by academic discipline librarians by area of expertise librarians by specialty areas of expertise assistance by expertise   we recruited 14 students to participate in the usability testing. we asked the participants to point to the label they would click on to find a librarian under two circumstances: 1) seeking assistance in their major and 2) seeking assistance outside their major.     figure 1 paper prototype of the home page.       we presented the paper prototype to each student and used a semi-structured approach, using a set of questions as a guide, not a prescriptive, rigid survey. we asked questions to identify their preferred labels and what they expected to see on that landing page. we identified two possible search questions that correspond with the above circumstances. students may have these questions in mind when seeking librarian assistance:   when looking for help from a librarian for information related to your major, what link would you choose? when looking for help from a librarian for information outside your major, what link would you choose?   with this user-centered approach we were able to gather students’ preferences for labels when seeking librarian assistance. while we considered asking student workers, we knew that students who work for the library may be biased or have a better understanding of searching for librarian assistance, since they work with librarians almost daily. because of this knowledge, we decided to ask random students in the library.   “librarians by academic subject” was chosen most often (table 1). this label was chosen at a higher rate for question one, relating to their major, which may be obvious because of the word “academic.” however, the amount of clicks it received was unexpected. language made a significant difference in how students searched.   there was no difference when comparing the current and original labels for question 2: searching for librarian help outside their major (figure 2. red bars). we can assume that there needs to be specific language that affords this type of search, especially if librarians want to market their various skills to students.   once a term was “clicked,” students were asked what they expected to see on the following page. we wanted to find out what they thought they would discover once they chose a link. students expected to see similar information on these landing pages across all librarian-proposed terms.  expectations included seeing a list of librarians, college departments by major, and librarian contact information. when “librarians by expertise” was chosen, one student commented that they expected the librarian to have a ma or phd in that subject.   fessenden’s (2010) eye tracking study found that 80% of web users look at the left side of the screen while 20% look at the right side of the screen. our usability study showed that students chose labels most often from the right side of the screen. however, this does not negate the fact that they may have mostly looked on the left side of the screen. eye tracking could have been useful to find out where students look for discoverability of librarians, but it was not part of our study. according to pernice (2017) users’ motivation impacts what they click. so, our users’ choice in labels may have been impacted by the two questions asked as opposed to finding labels under actual circumstances.   involving students in this study to gather feedback about how they search the library home page was more user-centered than allowing librarians to assume how users search. our actual or potential users were our best resource for evidence.       implementation   on the uta library website, “librarians by subject” was changed to “librarians by academic subject” based on the usability testing conducted. the digital creation department made changes to the website within weeks of providing the findings from the paper prototypes. no other website changes were made at that time.   the evidence from students showed us their preferred language when searching for librarians. the implementation of this new, or rather improved label was intended to clarify language and meaning as it relates to seeking librarians on the home page.     table 1 the percentage of label clicksa proposed terms total percentage of clicks librarians by academic subject (current term) 30% librarians by expertise 15% librarians by academic discipline 3% librarians by area of expertise 6% librarians by specialty 9% areas of expertise 6% assistance by expertise 6% librarians by subject (original term) 12% ask us 9% request an appointment 3% alibrarian-proposed terms are shaded       figure 2 preference for terms on library home page.     table 2 page views for 2016 and 2017 for subject librarians   2016 june – september 2017 june – september page views 1557 2199       outcome   there was a 39% increase in clicks for “librarians by academic subject” from summer 2016 to summer 2017 (see table 2).   “librarians by academic subject” received 32% of all page views, reiterating that the label change was successful. however, when it came to finding librarian help outside the student’s major, none of the librarian-proposed terms were strong contenders. “librarians by expertise” received the greatest number of clicks (3 of 19 total clicks). since “librarians by academic subject” received the most page views overall, it is a starting point for placing those extra skills librarians have on that page, as well as for better accessibility.   students chose other labels from the website as well, such as “ask us.” this shows that we are losing a portion of our student population (12%) when they are seeking librarian assistance. some are not using the labels we expect them to use. also, searching for librarian assistance outside their major (question 2) held no significant click rate for any one particular label; the librarian-proposed labels were chosen one to two times. this finding shows that we need better labels that are intuitive and that resonate with students when they search for a librarian on the home page. there was not a significant amount for any other librarian-proposed term chosen outside of “librarians by academic subject.” was this because most of the proposed terminology did not resonate with our users when searching for librarian assistance?   students also provided their own labels, such as “librarians by research interest,” “librarian assistance by expertise” and “librarian assistance by specialty.” further research would allow students to create labels instead of choosing from terms generated by librarians. according to gillis (2017) knowing certain terms does make navigating easier, but we must be aware that jargon can create obstacles for new users. allowing our users to create labels may make their search easier.   reflection   the data collection process was straightforward due to the consistency in questioning and methodology.   this study could be improved. the librarian-proposed terms were shown as a list and “librarians by academic subject” could have created a bias since it was the first listed. to limit the amount of paper illustrating the various places for the terminology, we decided to list the terms together, using one sheet. we could also improve the way the study was implemented. for example, in the future, we should ask students what they prefer, without presenting them with labels to choose from. their language, or suggestions for labels, strongly reflected ours. they used similar terms as ours such as “expertise” and “specialty.” students’ suggested labels could have been influenced by the labels that were presented to them, even those that were already on the website.   optimally, the usability study would be done on a computer, randomizing the locations of the librarian proposed terms with students doing real-time research in the context of what they are studying. relating to our goal, the literature talks about marketing or “effectively sharing” the services and expertise of librarians (benedetti, 2017). we want the best labels to be able to communicate to users the range of skills that uta librarians possess.   our results are satisfying, but more work should be done to simplify language for improved online accessibility to meet users’ needs.   references   benedetti, a. r. (2017). promoting library services with user-centered language. portal: libraries and the academy, 17(2), 217-234. retrieved from https://doi.org/10.1353/pla.2017.0013   fessenden, t. (2017). horizontal attention leans left. retrieved from https://www.nngroup.com/articles/horizontal-attention-leans-left/   gillis, r. (2017). watch your language: word choice in library website usability. partnership: the canadian journal of library and information practice research, 12(1). retrieved from http://dx.doi.org/10.21083/partnership.v12i1.3918   pernice, k. (2017).  eyetracking shows how task scenarios influence where people look. nielsen norman group. video. retrieved from https://www.nngroup.com/videos/eyetracking-task-scenarios/?utm_source=alertbox&utm_campaign=eea3f842de-newsletter_imagery_leading_questions_2017_12_18&utm_medium=email&utm_term=0_7f29a2b335-eea3f842de-40289501   us news & world report (2018). campus ethnic diversity. retrieved from https://www.usnews.com/best-colleges/rankings/national-universities/campus-ethnic-diversity   microsoft word news_irg.doc evidence based library and information practice 2007, 2:1  165 evidence based library and information practice     news    information resources group (irg) workshop: pushing the frontiers of hta  information management        © 2007 eblip. this is an open access article distributed under the terms of the creative commons  attribution license (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,  distribution, and reproduction in any medium, provided the original work is properly cited.      make the irg pre‐conference workshop part  of your annual htai (health technology  assessment international) meeting plans!  join us in barcelona on sunday, june 17th  for a full day of training and lively  discussions. our workshop includes a  mixture of presentations and interactive  sessions highlighting important and  emerging developments in hta  information management.    julie glanville, is manager and associate  director at the centre for reviews and  dissemination in york, will outline recent  research into search filters to identify  diagnostic tests and assessments of which  databases and other resources should be  searched for diagnostic test studies.     greg bak, an information specialist from the  canadian agency for drugs and  technologies in health, will discuss a  cadth‐adapted critical appraisal  instrument that can be used to rank and  assess the quality of methodological search  filters. his presentation will provide  participants with the skills needed to make  informed decisions about when and which  methodological search filters to use.    sari ormstad, a research librarian at the  norwegian knowledge centre for the  health services, oslo, will present on a new  database that offers local decision‐makers  and health personnel an overview of all  current international assessments of new  health technologies and summarises those  that are most relevant for them.     miquel mayer, director of web mèdica  acreditada, an international accreditation  program of medical websites, will discuss  the quality and use of health‐related  websites, different strategies to select and  improve their quality, and finally how  metadata and semantic web technologies  could help users find the best health  information.    elaine alligood, an information specialist  from veterans affairs technology  assessment program, will update  participants on current search engines and  how they can be used in hta searches to  complement traditional database searching.    andrew booth, director of information  resources at the school of health &  http://creativecommons.org/licenses/by/2.0 evidence based library and information practice 2007, 2:1  166 related research (scharr), and amanda  burls, senior clinical lecturer and director  of west midlands health technology  assessment collaboration, will deliver a  half‐day bilingual (english and spanish)  critical appraisal workshop in which they  provide a ʺtapasʺ of critical appraisal skills  and techniques. the workshop will  demonstrate that skills in critical appraisal  can be modeled and facilitated by library,  information and other health staff working  locally.   full details regarding conference  registration are available at  http://www.htai.org/barcelona‐2007/.    for more information, contact becky  skidmore, irg chair (bskidmore@sogc.com)  or malene fabricius jensen, irg chair‐elect  (mfj@sst.dk).        http://www.htai.org/barcelona%e2%80%902007/%00 news   call for applicants for eblip journal: production editor      2018. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29406     evidence based library and information practice (eblip), a peer reviewed open access journal published since 2006 seeks to fill the position of production editor. the successful candidate will be expected to begin their role by may 2018 and serve a 3 year term (with possibility of renewal).   the production editor will be responsible for:   layout of all journal content in pdf and html formats. working with associate editors to ensure adherence to the journal’s publication manual and to ensure the publication maintains a professional and consistent appearance. supervising the editorial intern to ensure proper formatting and accurate metadata for each article. management of the journal’s masthead and website content, with the assistance of the communications officer. communicating and consulting with the editor-in-chief on a regular basis. attending editorial team meetings, via skype, on a monthly basis and participating in the overall governance of the journal. ensuring that all necessary deadlines are met. communicating with potential authors and responding to queries.   the ideal candidate for production editor will be experienced in microsoft word with working knowledge of html. attention to detail and the ability to manage tight deadlines are essential. familiarity with pkp’s online journal system is beneficial. knowledge of evidence based practice is also an asset.   this position requires dedicated time on a regular basis, and it is therefore essential that interested persons ensure available time to devote to this position prior to applying. it is estimated that the workload is approximately 10 hours per month, with the bulk of work occurring on a quarterly basis prior to journal publication.   interested persons should send a cover letter, indicating areas of strength they would bring to the role, and resume/cv as a single pdf file to lorie kloda, editor-in-chief, at lorie.kloda@concordia.ca by april 15, 2018. for informal enquiries and more information about the role, contact the current production editor, michelle dunaway, at mdun9430@gmail.com).   **please note that evidence based library and information practice is a non-profit, open access journal and all positions are voluntary and unpaid.   about the journal:   published quarterly, and hosted by the university of alberta, this peer-reviewed, open access journal is targeted at all library and information professionals interested in an evidence based model of practice. by facilitating access to librarianship research via original research articles and evidence summaries of relevant research from the library literature, evidence based library and information practice will enable librarians to practice their profession in an evidence based manner.   please visit the evidence based library and information practice web site (https://journals.library.ualberta.ca/eblip/index.php/eblip) for further information about the journal.   research article   assessment of multilingual collections in public libraries: a case study of the toronto public library   valentina ly research assistant mount sinai hospital toronto, ontario, canada email: vly@ualberta.ca   received: 8 feb. 2018     accepted: 24 july 2018      2018 ly. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29408     abstract   objective – the toronto public library has been frequently identified as having an exemplary multilingual collection to serve the information needs of the most diverse population in canada; however, there is no evidence or collection assessment information available in the literature to validate those claims. this research sought to gain an understanding of the current state of their multilingual collection and compare it to the most recent multicultural population demographics.   methods – this was a case study of the toronto public library multilingual collection using data collected from their online public access catalogue in november 2017. data was collected about all languages available, with english, french, and the 17 most spoken mother tongues explored in more detail. language results from the statistics canada 2016 census of population were also collected. data was used to calculate and compare the english, french, and language collections to the population of reported mother tongues spoken in toronto.   results – it was found that the toronto public library has items in 307 languages. while the collection comprises many languages, there is far more focus on official language items than any other language compared to the population in terms of number of items and variety of formats. all 17 non-official languages that were studied had fewer items proportionally available in the catalogue than the proportion of speakers with that mother tongue.   conclusion – the high circulation rates of the toronto public library’s multilingual collection indicate that it has had some success in meeting the needs of its community. however, as the largest library system in canada with a highly regarded multilingual collection and with many resources for collection development, the toronto public library falls short of having a language collection that is proportional to the languages spoken within the community. while it may not be possible to have a multilingual collection that is entirely representative of the community, this study shows that libraries can use census data to monitor population shifts in order to be responsive to the information needs of their changing communities.   introduction   public libraries are institutions that serve the community’s information needs, regardless of language barriers. it is vital for libraries to reflect the full range of community diversity, which is often delivered through multilingual (ml) services such as collection items in non-official languages (chilana, 2001; rodrigues, jacobs, & cloete, 2006). language is a major part of a person’s identity and sense of belonging. when a library serves a linguistic minority, it shows consideration and respect for that language in the community, resulting in improved relationships between the library and that group. the provision of these materials also allows the group to preserve their mother tongue and cultural diversity, and stay connected to their roots while they settle into life in a new country (atlestam, brunnström, & myhre, 2011; bezugloff, 1980; picco, 2008; tanackovic, lacovic, & stanarevic, 2012). however, the variation in community demographics makes it difficult to prescribe a one size fits all solution for how each public library should meet those needs. what makes it more challenging is that many communities are constantly changing due to migration and globalization, but it is imperative for public libraries to be aware of these changes in demographics and act accordingly.   in a survey of public libraries across canada, dilevko and dali (2002) found that many libraries reported that their ml collections are underused. as such, there is mixed response about the direction that public librarians want to take their ml collection, especially when there is a requirement for them to justify the development of ml collections (chilana, 2001; dilevko & dali, 2002). while collections should be representative of the linguistic diversity of the community, it is not necessarily feasible for public libraries to build a perfect ml collection (atlestam et al., 2011). the toronto public library (tpl) has reported high circulation of their ml materials and has been lauded for how they build their ml collection (oder, 2003; rogers, 2003; sathi, ngan, & debenham, 2016). as suggested by godin (1994), smaller libraries can learn from larger libraries that have the resources to focus on ml collection development. this case study of the tpl ml collection sought to reveal how representative the collection is of the immigrant population and demonstrate how ml collections can be developed in practice.   literature review   the ml collection is often the first service libraries start with when developing services that cater to immigrant populations (hoyer, 2011). early in public library service, there was some evidence of collection development of non-english items. as migration and movement of populations became common, more libraries began to include materials in other languages to meet demand (bezugloff, 1980). the national library of canada used to maintain the multilingual biblioservice so that public libraries could offer their community books in non-official languages without having to develop the service themselves. towards the end of the twentieth century there was a push for ml collections with nationwide efforts and guidelines, but there have been fewer updates in the last decade (godin, 1994; international federation of library associations and institutions, 2009; library and archives canada, 2005; reference and user services association, 2007).   many varieties of ml collections assessments have been explored globally to determine the breadth and depth of the collection, and it has been consistently found that public libraries are not providing equitable service to linguistic minorities (berger, 2002; hansson, 2011; picco, 2008; rodrigues et al., 2006). some of these studies are based on user information collected through focus groups and questionnaires and others are based on evaluating the collection, circulation statistics, or presence of key titles. overall, researchers have found that the official language of the country is the dominant language collected at a library. in canada, the official languages are english and french, however, the majority of canadians speak english.   a library collection consists of a variety of formats and the same should apply to the ml collection as well. official guidelines and recommendations for ml collections stress the need to have print and other media formats (ifla, 2009; library and archives canada, 2005; reference and user services association, 2007). the study of collection formats is an important part of ml collection assessment (chilana, 2001; dilevko & dali, 2002; rodrigues et al., 2006). online formats are also important as more libraries begin to deliver digital content. the internet is a major gateway to ml information, with many foreign newspapers accessed through this route (atlestam et al., 2011; berger, 2002; dilevko & dali, 2002; tanackovic et al., 2012).   a community profile or needs assessment is an important step to knowing who a library’s users are and applying that information to build library services. authorities on library services suggest that this process can take many forms, but the simplest and most accessible method for a public library to find its population demographics is to use national census results (library and archives canada, 2005; reference and user services association, 2007). although these results may not provide information about community needs, they can provide information about immigration, distribution within an area, and languages spoken by residents. most assessments of ml collections include this type of demographic information as a measure of the population (atlestam et al., 2011; dilevko & dali, 2002; garrison, 2013; hoyer, 2011; rodrigues, 2006). libraries that have not utilized a community profile, such as census results, also have been found to have very imbalanced ml collections (hansson, 2011; hoyer, 2011; rodrigues, 2006).   as canada’s largest and most multicultural city, toronto plays a significant role in receiving immigrants to canada (kumaran & salt, 2010; quirke, 2007). in 2016, 47.03% of torontonians were immigrants, with 6.98% of those immigrants having come within the past five years (statistics canada, 2017). soon after arrival, many newcomers to toronto seek out memberships to the tpl, with double the usage by immigrants compared to those born in canada, which correlates with other immigrant library usage studies (berger, 2002; dali, 2016; oder, 2003; quirke, 2007). tpl has been noted for rising to the challenge of serving the most linguistically diverse population through continued investment in newcomer programming, particularly for their responsive building of ml collections and devotion to its development for the past few decades (dilevko & dali, 2002; quirke, 2007; rogers, 2003; zielinska, 1980). as such, the tpl can be used as an exemplar for other public libraries since many have noticed its influence, compared to any other canadian public library (kumaran & salt, 2010; rogers, 2003; zielinska, 1980).   while the tpl may be known for actively collecting and curating their ml collection for the large immigrant population, only the broader picture of delivering ml services and programs is addressed. other canadian public libraries have been evaluated for their linguistic diversity; for example, picco (2008) determined that the items at three montreal libraries do not reflect the diversity of the population. however, there are no similar studies of the tpl, and there is a need to quantify its ml collections. many may want to model after the tpl ml collection, but there is no collection assessment information available. this tpl ml collection assessment will allow for comparative analytics across public libraries in canada through the collection count process.   aims   the purpose of this case study is to determine how well the tpl ml collection development reflects the diversity of languages spoken by the community. the following four research questions were posed:   ·         how many different language collections are available at the library? ·         how many items are available in each language collection? ·         what formats are available in each language collection? ·         is the size of each language collection equitable to the size of the minority language community?   method   using statistics canada’s (2017) 2016 census profile, the toronto census division map was aligned with the tpl branch map and locations to confirm that they are contained within the same geographic boundary. this confirmed that the demographics for that census division are relevant to the tpl system.   from the census demographics, data from “mother tongue” and “language spoken most often at home” was collected and arranged based on the number of respondents. when ranked, the spoken mother tongues and the languages spoken most often at home did not correspond in ranking or proportions of the population; however, the top 18 languages listed in both categories were the same. these 18 most commonly spoken languages were selected: cantonese, mandarin, tagalog, spanish, italian, portuguese, tamil, persian, urdu, russian, korean, arabic, bengali, greek, gujarati, polish, vietnamese, and punjabi. these languages were also listed in tpl’s ml collection directory, as part of the languages that they actively collect and advertise (tpl, 2018). these languages correspond to searchable languages in the tpl (2017) online public access catalogue (opac), except that only “chinese” can be selected on the tpl catalogue, rather than mandarin and cantonese individually. the census value for “chinese languages” was used in lieu, which includes cantonese, mandarin, hakka, min dong, min nan, wu, chinese not otherwise specified, and chinese not listed elsewhere (statistics canada, 2017). therefore, 17 non-official languages were explored, as well as the official languages of canada, which are french and english.   using the tpl opac on november 12, 2017, the total number of items available in every language was collected, as well as the number of items in each of the formats of the selected 17 languages and in english and french. next, the proportions of mother tongue language groups, library items, and formats were analyzed and compared. calculations for the number of items per capita were determined, but altered to consist of the number of items for a specific language and the number of people who speak it.   results   language collection at tpl   through the advanced search in the tpl opac, 362 language categories were available for selection at the time of data collection. of those, there were 307 language categories (including english and french) that had at least one item in the collection. however, this includes languages that are not spoken in modern society or that are not considered as one of the 215 languages that are quantified by statistics canada. figure 1 displays the languages and number of items available from tpl for non-official languages that had more than 100 items catalogued. there were 229 non-official language categories that have not been included since they had 100 items or less.     figure 1 the number of items in each tpl non-official language collection that contains more than 100 items.     table 1 number of non-official language collections within each item range number of items in language collection non-official language collection 1-9 156 10-99 72 100-999 37 1,000-9,999 34 10,000+ 6     in total, tpl had 2,234,867 items listed in their opac. of that, there were 1,849,095 english items and 130,377 french items. the remaining 255,395 items were split up amongst 305 languages. table 1 presents the number of items in all the non-official tpl language collections that fit within the designated item ranges. there are six non-official languages that had 10,000 or more items, 34 languages that had 1,000-9,999 items, 37 languages that had 100-999 items, 72 languages with 10-99 items, and 156 languages with 1-9 items. this shows that most of the languages collected by the tpl comprised very few items, while few languages had substantial collection sizes.   collection representation   further investigation of non-official languages was limited to the top 17 mother tongues in toronto. table 2 presents the proportion of items listed in these languages and the proportion of people in toronto that speak that mother tongue. all tpl items considered, 1,849,095 items (82.74%) in the collection were in english, 130,777 (5.83%) items were in french, and 255,395 (11.43%) items were in a non-official language. table 2 reveals a portion of the distribution of non-official languages.   based on the toronto census 2016 results, out of the total 2,704,415 responses, there were 1,375,905 (50.88%) people who said their mother tongue is english, and 35,440 (1.31%) indicated that french is their mother tongue. comparatively, 1,186,885 (43.89%) of the population had a mother tongue that is not considered an official language of canada. the remainder of respondents listed multiple mother tongue languages. despite the high number of toronto residents reporting a non-official language native tongue, fewer people primarily spoke these languages at home. with the exception of english, the study languages dropped by 26% to 58% in usage at home (statistics canada, 2017).   a common collection assessment calculation is items per capita. figure 2 displays the language items per mother tongue speaker, which is the item number available in that language, divided by the number of respondents with that mother tongue. the mean item per mother tongue speaker was 0.17 items per capita, with a standard deviation of 0.11. of the 17 study languages, tagalog was the most underrepresented at 0.03 items per tagalog speaker, while polish had the largest rate of the non-official study languages at 0.50 items per speaker. english speakers had a very large collection with 1.43 items per speaker, but it was the french language that far surpassed that at 3.68 items per speaker.   similarly, figure 3 presents the items per capita based on the number of respondents that indicated that that language was spoken most often at home. since there were fewer people speaking languages other than english at home, the items per capita value was higher compared to figure 2, with a mean of 0.31 items per capita and a standard deviation of 0.25. again, french language items per capita were far more represented than any other language. interestingly, there were 1.18 polish items per capita, compared to the 1.13 english items per capita.     table 2 proportion of language collection items in comparison to the population languages proportion of collection proportion of mother tongue speakers proportion of languages spoken most often at home english 82.74% 50.88% 64.33% french 5.83% 1.31% 0.63% chinese 1.41% 9.07% 6.70% spanish 0.86% 2.69% 1.54% italian 0.71% 2.32% 1.00% polish 0.56% 0.93% 0.39% russian 0.48% 1.34% 0.85% portuguese 0.30% 2.20% 1.24% tamil 0.30% 2.13% 1.55% korean 0.23% 1.25% 0.87% gujarati 0.21% 0.98% 0.61% greek 0.20% 1.03% 0.48% urdu 0.20% 1.38% 0.87% punjabi 0.18% 0.74% 0.45% persian 0.17% 1.82% 1.23% bengali 0.17% 1.05% 0.70% vietnamese 0.15% 0.92% 0.63% arabic 0.14% 1.10% 0.61% tagalog 0.12% 3.08% 1.57% total 94.96% 86.22% 86.25%     figure 2 items per capita for each mother tongue language group.     figure 3 items per capita for each language group spoken most often at home.       available formats   table 3 shows that out of the possible 32 formats listed by the tpl opac, french had items in 29 formats and english had items available in 31 formats, which was the largest format variety. for the 17 study languages, there was a mean of 15.53 formats, standard deviation of 5.35, and a range of 8 to 26 formats.   table 4 presents the formats available at tpl and the proportion of the collection for each format. from the 17 language collections explored in this study, 71.40% of these items were catalogued as regular print books; however, this ranged from 40.47% to 88.15%. in comparison, 64.85% of official language items (71.54% french and 64.41% english) were catalogued as regular print books. some items had very few items present within format types; however, dvds and music cds were categories that consistently had several items available in each language, especially for the 17 non-official languages.       table 3 number of format types available in each language languages format types english 31 french 29 italian 26 spanish 25 portuguese 20 chinese 19 korean 19 russian 18 arabic 16 greek 16 polish 16 persian 15 urdu 14 vietnamese 13 tamil 12 bengali 10 tagalog 9 gujarati 8 punjabi 8     table 4 distribution of language collections in each format type physical formats study non-official languages french english regular print books 71.41% 71.54% 64.41% large print books 0.05% 0.16% 1.05% multimedia kits 0.12% 0.01% 0.01% braille books 0.03% 0.02% 0.01% tactile books 0.02% 0.02% 0.00% cloth books 0.01% 0.00% 0.00% images 0.01% 0.06% 5.07% dvds 8.02% 2.11% 1.60% vhs 0.02% 0.01% 0.06% music cds 9.69% 1.91% 1.19% vinyl 0.49% 0.29% 0.06% cassettes 0.00% 0.00% 0.00% sheet music 0.66% 0.65% 0.39% pictures and photographs 0.00% 0.01% 4.97% image maps 0.01% 0.03% 0.02% posters and printed ephemera 0.00% 0.02% 0.07% photo albums 0.00% 0.00% 0.00% audiobook cassettes 0.04% 0.04% 0.03% audiobook cds 0.77% 0.72% 0.68% talking books 0.00% 0.00% 0.63% magazines, newspapers, and journals 0.40% 1.25% 1.78% microforms 0.12% 15.78% 4.55% atlases 0.00% 0.00% 0.00% maps 0.10% 0.13% 0.16% computer disks 0.01% 0.01% 0.06% digital formats ebooks 1.00% 3.10% 9.22% eaudiobooks 0.10% 0.03% 1.42% emusic 4.31% 1.44% 0.73% evideo 0.43% 0.20% 0.66% ejournals 2.12% 0.44% 1.13% databases and online tools 0.02% 0.01% 0.00% downloadable magazines 0.05% 0.02% 0.02% total 100% 100% 100%     as collections are shifting towards electronic availability, it is important to determine whether non-official languages are being collected in digital formats as well. these would be items that are catalogued as an ebook, eaudiobook, emusic, ejournal, evideo, database and online tool, or downloadable magazine in table 4. from the 17 language collections explored in this study, 8.04% of the total items were available in an electronic format. in comparison, 12.69% of official language items (5.22% french and 13.19% english) were digital content.   discussion   the tpl can potentially serve as a model library for other institutions through their ml collection development strategies. in their article, kumaran and salt (2010) concluded that compared to other large, urban library systems, tpl was the most dedicated to their diverse community based on their non-english services and community partnerships. nevertheless, the results of this study reveal that at the time of data collection the tpl ml collection was far from proportional in representation to the population. further explorations of these results reveal the many challenges that exist with ml collection development.   some might assume that if the library is responding to community demographics, then there should be equitable representation of the linguistic diversity in the collection as there is in the population (rodrigues et al., 2006). however, that is not the case with the tpl, nor of any ml collection assessment in the literature so far. older ml collection development models recommend having equitable and fair services per capita if there are 300 or more minority language speakers within a community (godin, 1994). the reference and user services association’s (1990) “guidelines for multilingual materials collection and development and library services” echo that statement, relating the size of the collection to the size of the group; however, the updated 2007 “guidelines for the development and promotion of multilingual collections and services” version removes that wording from their guidelines. while it is important to compare the language collection to the population that speaks that language, there is a move away from what a “balanced” collection may refer to; rather, the goal of collection development is about meeting the needs of the user groups within the given constraints (dali, 2010; kumaran & salt, 2010).    tpl has the added challenge of collecting for the most linguistically diverse population in canada, let alone north america. the toronto 2016 census results reveal that while there are many people with a non-official language mother tongue, there are fewer people indicating that this is the language spoken most often at home. there were 64.96% respondents who noted they speak an official language at home, whereas 25.95% said that the primary language spoken at home was a non-official language, and 8.85% said that they equally speak an official and a non-official language at home (statistics canada, 2017). while a large portion of the toronto population was born outside of canada and grew up with a different mother tongue, english still seems to be the dominant speaking language, which may explain why it is tpl’s main budgetary and developmental focus. in addition, dilevko and dali (2002) point out that in 2001 the tpl was allocating 12% of their book budget for the french and ml collection, which is far below the proportion of the community that that budget was meant to serve.for more detailed analysis in this study, only the most commonly spoken mother tongues in toronto were explored. large disparities were revealed in table 2 between the proportions of the tpl ml collection compared to the population in toronto speaking that language. although the tpl has actively funded and built their ml collection over the years, toronto’s 17 most spoken mother tongues and languages spoken most often at home are not as highly represented in the collection compared to the official languages of canada. since the study languages were not randomly selected, the 17 languages are more likely to be better represented than a random sample of the ml collections because these languages have a larger population of speakers and more representation in the community. also, the 17 study languages are part of the 40 languages that the tpl actively collects (oder, 2003; rogers, 2003). the results from this study may over-represent the state of the total ml collection. however, zielinska (1980) recommends looking at the more popular languages for ml collection building, so analysis of the most popular languages may make sense for libraries with less diverse communities.     according to figure 2, of the non-official languages, polish speakers have the most selection of library items in their language, while tagalog speakers had the least available selection. this imbalance between individual language collections may be due to migration changes and library response time. the polish are a more settled population in the community compared to immigrants from the philippines. according to the 1996 census, there was a burst of immigrants from poland in toronto (statistics canada, 1997). more recently, immigrants from the philippines living in toronto have increased by 15.86% between 2011 and 2016, while polish immigrant populations have dropped by 11.93% (statistics canada, 2013, 2017). within the span of five years between census collections, that is a significant change in population demographics, whereas collection changes happen at a slower pace. although the tpl relies on information from settlement workers, supplemented with census data to stay up to date on population shifts, the tpl and other libraries face difficulty when collecting for newer immigrant populations, especially those that have been affected by international events like wars. this is an issue since these groups have a need for information in their languages, they may not have any knowledge of english or french, and fewer materials in their language may be available (bowles, 2016; quirke, 2007; rodrigues et al., 2006; rogers, 2003).   based on ml collection assessment trends in which the dominant spoken language of the country is the one that is heavily collected, it is surprising to see in figure 2 that the french collection has the most items per capita. this far surpasses the rate for any other language, including english. this may be due to the fact that although there are few people who have french as a mother tongue, it is a protected language in canada and supported in government and school systems (mougeon, 2015). furthermore, there may be more ease in locating french resources compared to other non-english materials (rogers, 2003). dilevko and dali (2002) note that the ml budget at tpl includes money to be spent on french resources as well, so most of the ml budget could be spent on developing french over the other non-english languages.   based on the results, english and french collection users have had the most variety of formats available to them. considering this study investigated the most popular non-official languages, there were significantly fewer formats to choose from in those study languages. across all languages, the collections in this study consisted mostly of regular print books, which ifla (2009) states is the most likely format available in ml collections. non-english languages had fewer digital resources available for its language groups. tpl has cited the difficulty of locating databases in languages other than english, which may be the issue for other digital resources as well. as a way of combatting this issue, tpl has built their own internet subject gateway to newspapers worldwide; however, this was not considered as part of the ml collection in the tpl opac (rogers, 2003).   besides print books, dvds and music cds were abundant in foreign languages compared to its availability in english and french. this vast ml dvd collection may be due to the previously high circulation rates and demand of foreign dvds at tpl, which contributed to the overall high circulation of the ml collection (bowles, 2016). this fits with garrison’s (2013) findings that some libraries may build their collection with more emphasis on circulation rather than on equitable service. recently, there have been fewer dvd loans, which has negatively impacted the ml collection circulation, with a decline of 12.6% in the past year. the tpl attributes the drop in dvd loans to the ease of online streaming services and to certain language minority groups moving out of the tpl service area (bowles, 2016).         certain tpl staff who speak another language are in charge of that language collection in terms of selecting items and connecting with vendors (oder, 2003; rogers, 2003). however, the ml collection selection process takes place on a branch level, with only the local residents’ needs in mind (martin prosperity institute, 2013). some of the variation of the ml collection stock could be due to that branch-level selection and having individual employees independently focused on one language. over time, this could result in an imbalance of collection development since individual purchasers may not reflect on the balance of the whole collection. investigation of the ml collection on a branch level may reveal that the collection is more balanced to each catchment area than it does when explored as a whole.   many libraries cite the challenge of building ml collections due to lack of staff knowledge of other languages. libraries struggling to develop and catalogue their ml collections can learn from other library systems that share their expertise and knowledge, one of which is the tpl. rogers’ (2003) article reveals that even librarians from large metropolitan cities such as the seattle public library reach out to other library systems, including the tpl, to better inform their collection building processes. kumaran and salt (2010) also highlighted the partnerships between the tpl and organizations that directly deal with immigrant populations to better cater their services to different language speakers. furthermore, rogers (2003) noted that the tpl has excelled in locating ml vendors and have staff who speak 34 of the 40 active language collections, so the tpl should face fewer challenges than other library systems.   conclusion   while the official languages are important collections to focus on, in toronto there were 4.91% of residents who have no knowledge of either official language at the time of this study. they will face a much more difficult experience accessing information if the public library does not develop collections or build services to meet their information needs (statistics canada, 2017). circulation of ml materials in 2015 made up 7.7% of total tpl circulations, showing that there was a need for those materials (bowles, 2016; sathi et al., 2016).   there is a difference between proportions of the population of foreign language speakers in toronto and the items available in those languages at the tpl. of the 17 non-official languages that were studied in november 2017,  all were underrepresented in the collection. given the constraints of collecting in other languages, it should not be expected that public libraries have completely representative ml collections. however, libraries can use census data to monitor the population and migration patterns in their local communities and strive to collect an adequate number of items to satisfy the information needs of those who speak other languages besides english and french.   this study has a few limitations. of note, this study investigates items listed in the tpl opac. the item numbers used represent the number of unique items, not copies of each item, so may not be completely representative of the actual selection available at tpl. errors can also arise through the way items are catalogued. furthermore, the study does not consider the currency or quality of the ml items, which should be a major consideration of any collection assessment.   libraries may be unsure of how to meet the linguistic needs of their growing and changing communities. due to the number of resources needed to build ml library collections, librarians are reaching out to other library systems to better inform their collection building practices. as toronto has been acknowledged as a diverse city and the tpl has a reputation for their ml collection development, it may be more feasible for public libraries to follow the example of the tpl ml collection, rather than to pursue an equally representative collection.   acknowledgments   the author thanks tami oliphant for guidance and proofreading.   references   atlestam, i., brunnström, a.-c., & myhre, r. (2011). collection development for immigrants what to purchase, and why? findings in gothenburg, sweden. ifla journal, 37(1), 69-79. https://doi.org/10.1177/0340035210397308   berger, å. (2002). recent trends in library services for ethnic minorities – the danish experience. library management, 23(1/2), 79-87. https://doi.org/10.1108/01435120210413878   bezugloff, n. b. (1980). library services to non-english-language ethnic minorities in the united states. library trends, 29(2), 259-274.   bowles, v. (2016, apr.). 2015 annual performance measures and benchmarking. toronto public library. retrieved from http://www.torontopubliclibrary.ca/content/about-the-library/pdfs/board/meetings/2016/apr25/15-2015-annual-performance-measures-and-benchmarking.pdf   chilana, r. (2001). delivering multilingual services in public libraries in british columbia: a case study of the fraser valley regional library. pnla quarterly, 65(3), 18-20.   dali, k. (2010). readers’ advisory interactions with immigrant readers. new library world, 111(5/6), 213-222. https://doi.org/10.1108/03074801011044089   dali, k. (2016). readers’ histories as a way of studying and understanding multicultural library communities. library review, 65(8/9), 519-534. https://doi.org/10.1108/lr-04-2016-0035   dilevko, j., & dali, k. (2002). the challenge of building multilingual collections in canadian public libraries. library resources & technical services, 46(4), 116-137. https://doi.org/10.5860/lrts.46n4.116   garrison, k. l. (2013, aug.). “this intense desire to know the world”: cultural competency as a personal and professional disposition in collection development practices. paper presented at the meeting of ifla world library and information congress: 79th ifla general conference and assembly, singapore. retrieved from http://library.ifla.org/66/1/101-garrison-en.pdf   godin, j. (ed.). (1994). a world of information: creating multicultural collections and programs in canadian public libraries. library and archives canada. retrieved from http://www.nlc-bnc.ca/obj/005007/f2/005007-210.1-e.pdf   hansson, j. (2011). public libraries and the national minorities of sweden – results of a national survey. new library world, 112(9/10), 406-415. https://doi.org/10.1108/03074801111182003   hoyer, j. (2011). reshaping to serve evolving communities: addressing immigrant needs in the library. journal of organisational transformation & social change, 8(3), 297-312. https://doi.org/10.1386/jots.8.3.297_1   international federation of library associations and institutions [ifla]. (2009). multicultural communities: guidelines for library services (3rd ed.). retrieved from https://www.ifla.org/files/assets/library-services-to-multicultural-populations/publications/multicultural-communities-en.pdf   kumaran, m., & salt, l. (2010). diverse populations in saskatchewan: the challenges of reaching them. partnership: the canadian journal of library & information practice & research, 5(1), 1-25. https://doi.org/10.21083/partnership.v5i1.1012   library and archives canada. (2005). multicultural resources and services toolkit. retrieved from https://www.collectionscanada.gc.ca/obj/005007/f2/005007-300.1-e.pdf   martin prosperity institute. (2013, dec.). so much more: the economic impact of the toronto public library on the city of toronto. retrieved from http://www.torontopubliclibrary.ca/content/about-the-library/pdfs/board/meetings/2013/dec09/10_1.pdf   mougeon, r. (2015). french language in canada. in the canadian encyclopedia. retrieved from http://thecanadianencyclopedia.ca/en/article/french-language/   oder, n. (2003). growing into a changing city. library journal, 128(10), 40-42.   picco, m. a. p. (2008). multicultural libraries' services and social integration: the case of public libraries in montreal canada. public library quarterly, 27(1), 41-56. https://doi.org/10.1080/01616840802122443   quirke, l. c. (2007). more than books: examining the settlement services of the toronto and windsor public libraries. our diverse cities, 4, 156-160.   reference and user services association. (1990). guidelines for multilingual materials collection and development and library services. retrieved from http://www.ala.org/rusa/sections/rss/rsssection/rsscomm/spanishspeaking/guidelinesmultilingual   reference and user services association. (2007). guidelines for the development and promotion of multilingual collections and services. reference & user services quarterly, 47(2), 194-197. https://doi.org/10.5860/rusq.47n2.198   rodrigues, a. (2006). serving culturally diverse communities in south african public libraries. libri, 56(4), 213-226. https://doi.org/10.1515/libr.2006.213   rodrigues, a., jacobs, l., & cloete, l. (2006). providing equitable public library services to south africa's multicultural communities. mousaion, 24(2), 211-231.   rogers, m. (2003). serving up world languages. library journal, 128(11), 42-44.   sathi, s., ngan, e., & debenham, s. (2016, jan.). building responsive multilingual collections at toronto public library [powerpoint slides]. paper presented at the ontario library association super conference, toronto. retrieved from http://accessola2.com/superconference2016/sessions/1200ser%20f.pdf   statistics canada. (1997, november 4). 1996 census: immigration and citizenship. the daily. retrieved from http://www.statcan.gc.ca/daily-quotidien/971104/dq971104-eng.htm   statistics canada. (2013). nhs profile, toronto, c, ontario, 2011. (statistics canada catalogue no. 99-004-xwe). retrieved from http://www12.statcan.gc.ca/nhs-enm/2011/dp-pd/prof/index.cfm?lang=e   statistics canada. (2017). census profile, 2016 census: toronto, cdr [census division], ontario and ontario [province]. (statistics canada catalogue no. 98-316-x2016001). retrieved from http://www12.statcan.gc.ca/census-recensement/2016/dp-pd/prof/index.cfm?lang=e   tanackovic, s. f., lacovic, d., & stanarevic, s. (2012). public libraries and linguistic diversity: a small scale study on the slovak ethnic minority in eastern croatia. libri, 62(1), 52-66. https://doi.org/10.1515/libri-2012-0004   toronto public library. (2017). advanced search. retrieved from http://www.torontopubliclibrary.ca/advanced/   toronto public library. (2018). material in your language. retrieved from https://www.torontopubliclibrary.ca/books-video-music/your-language/   zielinska, m. (1980). public library services to canadian ethnocultural communities: an overview. library trends, 29(2), 275-292.     evidence based library and information practice commentary   collect with intent: craft meaningful questions that drive evidence based assessment strategies   melissa goertzen information management consultant halifax, nova scotia, canada email: goertzen.melissa@gmail.com   received: 20 feb. 2018   accepted: 16 apr. 2018      2018 goertzen. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29410     librarians work in an information environment that is highly competitive and provides users with many alternatives to traditional library services. despite the exponential growth of available information sources, collection budgets remain fixed or are reduced because of factors like the economy, greater competition for institutional resources, and assumptions that electronic content is low-cost or freely available (goertzen, 2017). information needs on university campuses surpass available resources, and librarians are required to justify annual collection budgets with evidence of use and overall value. now more than ever, it is essential for professional to demonstrate evidence-based collection practices to support users’ research, teaching, and learning needs.   developing collection assessment strategies in the current landscape is both an exciting and daunting task. the opportunities for experimentation are enormous but the complexities involved, like the dynamic nature of formats and technologies, present significant challenges. on top of this, librarians face pressures from administrators to produce evidence that justifies collection decisions or demonstrates impact.   data analysis is a relatively new skill set required of librarians. many articles published over the past several years focused on the fact that training opportunities are not widely available, and this disparity has prevented the standardization of assessment practices within the profession. from usage statistics to impact factors, there are myriad tools available to help librarians understand the strengths and weaknesses of their collections. the only problem is that there is not an agreed upon method to arrive at, compare, or act on assessment results (schmidt, 2010).   i spent five years working as a collection development librarian, and from my perspective, it seems that the profession directs its focus to solving a problem that has not been properly defined. there is pressure to present recommendations and evidence to administrators, but sustainable solutions will only come from a well-defined assessment strategies, goals, and objectives. i propose that the key to developing sustainable assessment strategies is to first uncover the correct questions to guide investigations. the inquiry process provides a focus to assessment work, ensures that the proper data is collected, and dictates how to conduct analysis activities in order to arrive at answers that support collection decisions. when librarians locate the central questions at the heart of evidence-based collection assessment, they create a roadmap which leads to correct answers and guides efforts to standardize assessment practices across the professional community as a whole.   developing questions that drive evidence based collection practices over time   my experience developing evidence based collection strategies started five years ago when i was hired by columbia university libraries (cul) to conduct the e-book program development study, a two-year assessment project that provided collection policies and best practices for e-book collections at cul (goertzen, 2016). when i read the project description, the opportunity seemed both exciting and daunting; the opportunities for experimentation were enormous, but i knew that the complexities involved with e-book collection development would present significant challenges.   several months into the study i realized that i was operating on the assumption that users prefer electronic content for research, teaching, and learning activities. however, as i started to collect usage statistics, examine search terms, analyze cost data, and speak with patrons, i realized that my initial impressions of content use were far too simplistic and did not tell the full story. i started to ask more and more questions about when, how, and why users gravitate towards certain formats (e.g., print, electronic, archival materials) to support scholarly activities and build knowledge around specific subject areas.   the inquiry process provided a focus and pulled everything i had observed into one overarching question: what is the intended use of e-book content? users interact with information for a variety of reasons including course use, research pursuits, and general reference. all of these activities serve different functions within a research community, rely on different levels of engagement with content, and support different information needs. in my investigation, identifying the intent of information use allowed me to provide evidence required to allocate budgets, negotiate license agreements, and make cases for information product acquisition.   when i consider the question of intent of use from a collection development perspective, my mind automatically separates activities into the categories of ‘current use’ and ‘future use’ (yale university library, 2013). this seems appropriate as there are few business models or collection development strategies that address both requirements at once. i think this separation points to a general shift in the way libraries in the 21st century must approach collection development activities: successful initiatives rely on a balance between ‘just in case’ and ‘just in time’ strategies. this balance allows information professionals to determine when it makes economic sense to invest resources in high use materials for current users and when it is appropriate to purchase materials that may have low use but add to the long-term value and legacy of the collection (yale university library, 2013). again, having a strong understanding of how patrons intend to use collection materials provides the insight required to make these decisions.      intent of use within the context of a long-term collection development strategy   essentially, “data stands in place of a reality we wish to study. we cannot simply know a phenomenon, but we can attempt to capture it as data which represents the reality we have experienced…and are trying to explain” (matthews & ross, 2010, p. 45). in the age of big data, there are seemingly endless data streams to examine and analyze. in order to prevent scope creep and collect evidence that is relevant to the needs of local user communities, it is essential to design a quantitative research framework around the central assessment question: what is the intent of information use? this process allows librarians to sketch a roadmap that leads to the intent of use, and ultimately to present a case for budget requests and support collection development decisions (goertzen, 2017).   intent of information use is not a static investigation. as new technologies are developed and users’ needs shift, intent of information use will evolve as well. building an assessment strategy that informs evidence based collection decisions is similar to building a long-term relationship with the user community. success relies on librarians’ abilities to create assessment plans that are flexible, sustainable, and can be replicated year after year. when this is accomplished, annual results provide evidence of trends that support ‘just in case’ and ‘just in time’ collection development decisions, especially in cases where information products do not support both simultaneously.   time spent planning is never wasted. in fact, the time invested in developing a strategy, particularly during the first year of an investigation, will create efficiencies in the long-run and develop baselines that provide evidence of collection use and value over time.   clearly articulated objectives are the engine that drives the assessment process (bakkalbasi, sundre, & fulcher, 2012). below is a checklist that i used to sketch out a roadmap that answered my central assessment question: what is the intent of information use?   review strategic plans at the library system and host institution; state project goals and objectives; create a list of internal and external stakeholders; develop a project timeline around the annual budget cycle; identify data sources that will answer the assessment question; define data analysis methods; develop a strategy to present results and evidence of collection value (goertzen, 2017).   by developing a standardized template for collection investigations, librarians essentially create a bridge between the current information landscape and a future vision for collection development activities. linking current work to future goals also allows librarians to effectively allocate budgets as research interests shift and ensure that information needs are met. collection assessment becomes less about proving the value of the collection today, and more about demonstrating the impact of the collection over time.   moving beyond data analysis: mapping assessment results to collection policies   when i consider how the intent of information use informs ‘just in time’ and ‘just in case’ collection decisions, my assessment activities take on new significance. i organize my activities so that results either confirm existing collection policies and practices, or flag areas where improvements can be made. by considering current best practices through the lens of assessment results, projects move the profession closer to standardized practices that benefit collection decisions over time.   with this being said, i organize data around five performance measures in order to understand how i can measure return on investment, value, or impact (goertzen, 2017). these measures are not confined to electronic resources and allow for assessment across the full collection, providing a more holistic view of trends and resource allocations.   domain measures: captures the user community served by the library. includes data related to demographic information, population size, and documented information needs. input cost measures: demonstrates how funds are allocated across collections. includes cost data pulled from the library’s integrated library system. collection output measures: relates to the quantity and quality of output, like the number of titles in a subscription package, or the number or bibliographic records acquired over a given time. includes data from title lists, overlap data, or bibliographic records. effectiveness measures and indicators: accounts for both collection input and output. includes data from counter reports, resolver statistics, consortial reports, turnaway statistics, publication counts, or google analytics data that provide insight into discovery, access, or usage trends. cost-effectiveness indicators: documents the return on investment or perceived value of a collection. includes data from counter reports, turnaway statistics, publication number, or citation analyses.   when i organized data analysis activities around the abovementioned performance measures, i discovered an important trend regarding the intent of information use. print and electronic materials supported different forms of reading activities: continuous (e.g., reading for extended periods of time, conducting in-depth research, exploring subjects in depth) or discontinuous reading (e.g. reference, citation confirmation, searching for keywords, skimming chapters). the results were consistent across the major disciplines observed during this study (i.e., humanities, social sciences, sciences, and fine arts).   based on results organized around the five performance measures, i went back to cul’s collection policies and recommended that print serve continuous reading needs and electronic serve discontinuous reading needs. essentially, i recommended that ‘just in case’ collection development activities focused on electronic materials, and that ‘just in time’ activities focus on print materials. finally, i mapped these policy recommendations against collection depth indicators (goertzen, 2016).   basic collection: e-books recommended  supports lower-division undergraduate research; includes the core of the discipline or sub-discipline as it relates to the curriculum. this level describes materials that serve to introduce and define subjects including selected databases, fundamental materials, introductory works, historical surveys, and reference works. extensive collection: e-books recommended supports graduate course work; information is adequate to maintain knowledge of a subject required at less than research intensity. examples of content include primary and critical resources, reference resources, specialized databases, and bibliographical resources. research collection: print recommended  supports research leading to a doctorate, faculty research, or independent study. it includes resources supporting the framework for the methodology and implementation of original doctoral research.   by framing my assessment strategy around a central question and placing results within the context of overarching collection development policies, cul not only received an understanding of how collections are valued today, but implemented strategies to measure intent of information use over time.   conclusion   when librarians challenge assumptions, look at issues from multiple perspectives, and test beliefs against performance measures, they pull back the layers of a problem to uncover the core issues that pull seemingly disconnected elements together through one investigation. in my work, this core issue has been identifying and understanding the intent that drives information use. by beginning assessment work with a strong research question, librarians provide a starting point for strategic plans and collaborative relationships that define how collections and services will be delivered in the future.   references   bakkalbasi, n., sundre, d., & fulcher, k. (2013). assessing assessment: a framework to evaluate assessment practices and progress for library collections and services. in s. hiller, m. kyrillidou, a. pappalardo, j. self, & a. yeager, (eds.), proceedings of the 2012 library assessment conference: building effective, sustainable, practical assessment, october 29-31, 2012 (pp. 533-547). washington, dc: association of research libraries.   goertzen, m. (2017). introduction to quantitative research and data. in m. goertzen (ed.), applying quantitative methods to e-book collections (pp. 12-18). chicago, il: american library association.   goertzen, m. (2016). e-book program development study: results and recommendations, 2013-2015. retrieved from https://doi.org/10.7916/d81z44c3   matthews, b., & ross, l. (2010). research methods: a practical guide for the social sciences. toronto: pearson education.   schmidt, j. (2010). musings on collection analysis and its utility in modern collection development. evidence based library and information practice, 5(3).  https://doi.org/10.18438/b8w330   yale university library. (2013, march 3). the ebook strategic plan task force: report of findings and recommendations. retrieved from http://www.library.yale.edu/departments/collection-development/yale-ebook-task-force-rpt.pdf   evidence summary   lis students at a japanese university use smartphones for social communication more often than for educational purposes   a review of: lau, k. p., chiu, d. k. w., ho, k. k. w., lo, p., & see-to, e. w. k. (2017). educational usage of mobile devices: differences between postgraduate and undergraduate students. the journal of academic librarianship, 43(3), 201-208. https://doi.org/10.1016/j.acalib.2017.03.004     reviewed by: stephanie krueger head, office of specialized academic services czech national library of technology prague, czech republic email: stephanie.krueger@techlib.cz   received: 23 feb. 2018  accepted: 13 june 2018      2018 krueger. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.   doi: 10.18438/eblip29412   abstract   objective – to discover how undergraduate (ug) and graduate (g; “postgraduate” [pg] in the original article) students of library and information science (lis) use mobile devices and to understand preferences and perceived barriers to educational use.   design – survey questionnaire.   setting – university in japan.   subjects – ninety undergraduate students (30 male, 60 female) and 30 graduate students (13 male, 17 female). nineteen additional recruits were excluded from the study due to incomplete surveys. almost all subjects (>98%) were born between 1982 and 2002.   methods – subjects were recruited without incentives from one lis department. an online survey was conducted with the purpose of gathering information on how often devices were used for various activities, perceived barriers to mobile learning (m-learning), and demographic data. the survey was modeled on a 2015 study of lis students in hong kong, japan, and taiwan (ko, chiu, lo, & ho, 2015). the mann-whitley u test was used to investigate possible significant differences between ug and g responses.   main results – 94.2% of participants had smartphones with internet access; both ug and g subjects reported weekly to daily use for social communications (email, short message service [sms], chat, and social media) and for querying search engines. both ug and g subjects reported using finance and banking services less than once a month. other activities (shopping, finding locations, entertainment, sports, tools and productivity software, casual reading, academic reading, accessing reference materials, accessing libraries) for both groups fell within the range of less than once per month to weekly use. unlike g subjects, ug subjects reported significant (p < 0.05) engagement with social media and marginal (p < 0.10) engagement with accessing libraries, and productivity tools.   in terms of educational use, neither ug nor g subjects reported daily m-learning behaviors, instead reporting monthly to weekly browsing of online information and social networking sites, with far less (i.e., less than once a month) engagement with professional articles, e-books, learning management platforms, and several other activities (listening to podcasts, viewing videos, “other”). ug subjects reported significant marginal (p < 0.10) engagement with “other” materials, unlike g subjects. library catalogs and databases were less likely to be used when compared to reference sources, with ug and g subjects reporting monthly or less use for these. when asked if they would use mobile library services, respondents answered “maybe interested if available”, with ug subject reporting significant marginal (p < 0.10) engagement vs. g subjects for several of these services. regarding productivity activities, both ug and g subjects reported monthly or less use of note taking, word processing, and scheduling tools. for communication and sharing activities, subjects reported monthly or less activity for communicating with classmates, using email for study-related issues, posting to discussions on learning management platforms, posting or commenting about their studies on social networking sites, sending photos or videos to social media, moving document files, and scanning quick response (qr) codes. ug subjects were marginally (p < 0.10) more engaged in communicating with classmates than g subjects.   barriers to m-learning were not considered “high” barriers, with “low” to “medium” barriers for both ug and g subjects being small screen size, non-mobile format, difficulty typing, challenges with authentication, no wi-fi, difficulty reading, lack of specialized apps, and slow loading times.   conclusion – this study provides a snapshot of how participants used mobile devices at the time the survey was conducted. both ug and g subjects used their devices for social communication more than for educational purposes.   commentary   this study sheds light on the question of how mobile devices are used in a particular educational setting. it contributes to the multidisciplinary literature regarding m-learning in education (chee, yahaya, ibrahim, & hasan, 2017), as well as to research on the acceptance of mobile library service technologies (saravani & haddow, 2015).   this study fulfills the basic requirements for a user study (booth & brice, 2003). the tables summarizing activities are clearly presented and provide a sense of which questions appeared in the original survey. the original survey instrument is not included as an appendix and the citation to the prior survey (ko, chiu, lo, & ho, 2015) is missing from the reference list, meaning that the survey could not be replicated solely on the basis of this article. furthermore, information about the reliability and validity of the instrument (e.g., results of reliability testing to measure internal consistency) is not provided. such information, together with more detail regarding survey administration (including the time needed to the complete survey, as well as whether informed consent was sought), would improve confidence in this study’s findings and should be included in future studies. the authors note that they recruited “sufficient subjects” to perform the mann-whitley u test, without stating how they determined this (p. 206). as recognized by the authors, additional investigation would be required to make any generalizations beyond this study (p. 207).   it would be difficult to apply findings from this study to practice because the survey did not tie barriers of use to specific activities, and did not delve into why some activities were performed more often than others. for example, it is clear from the data presented that mobile library services were infrequently accessed, but the reasons behind this are a matter of conjecture. future studies would be greatly enriched by linking questions about activities to questions about barriers and context, including open-ended questions about activity choices.   future research could also benefit from allowing subjects to provide commentaries about perceived educational utility. for example, the “viewing video clips” activity was included in the “general m-learning” table (p. 204). however, one can imagine scenarios in which subjects watched non-educational videos. specifically describing how activities were assigned to the m-learning category and more deeply examining the perspectives of the participants would strengthen the arguments made about educational vs. non-educational use.   another interesting point of departure for future studies would be an exploration of various types of learning taking place via mobile devices. for example, informal learning can be defined as “any activity involving the pursuit of understanding, knowledge or skill…without the presence of externally imposed curricular criteria” (bilandzic, 2013, p. 159). might, therefore, reading about an aspect of finance and banking on a smartphone represent “informal m-learning” and therefore be educational? what do subjects think? such questions were not part of this study but could be considered in future investigations.   overall, lis professionals planning their own local surveys can use this study as an example and as a basis for comparison.   references   bilandzic, m. (2013). connected learning in the library as a product of hacking, making, social diversity and messiness. interactive learning environments, 24(1), 158-177. https://doi.org/10.1080/10494820.2013.825811   booth, a. & brice, a. (2003). clear‐cut?: facilitating health librarians to use information research in practice. health information & libraries journal, 20, 45-52. https://doi.org/10.1046/j.1365-2532.20.s1.10.x   chee, k. n., yahaya, n., ibrahim, n. h., & hasan, m. n. (2017). review of mobile learning trends 2010-2015: a meta-analysis. journal of educational technology & society, 20(2), 113-126. http://www.jstor.org/stable/90002168   ko, e. h. t., chiu, d. k. w., lo, p., ho, & k. k. w. (2015). comparative study on m-learning usage among lis students from hong kong, japan and taiwan. the journal of academic librarianship, 41(5), 567-577. https://doi.org/10.1016/j.acalib.2015.07.005   lau, k. p., chiu, d. k. w., ho, k. k. w., lo, p., & see-to, e. w. k. (2017). educational usage of mobile devices: differences between postgraduate and undergraduate students. the journal of academic librarianship, 43(3), 201-208. https://doi.org/10.1016/j.acalib.2017.03.004   saravani, s-j. & haddow, g. (2015). a theory of mobile library service delivery. journal of librarianship and information science, 49(2), 131-143. https://doi.org/10.1177/0961000615595854 evidence summary   ongoing and multifaceted assessment of academic library professional development programs enhances their efficacy   a review of: harker, k. r., o'toole, e., & sassen, c. (2018). assessing an academic library professional development program. portal: libraries and the academy, 18(1), 199-223.  https://doi.org/10.1353/pla.2018.0010   reviewed by: rachel elizabeth scott integrated library systems librarian university libraries university of memphis memphis, tennessee, united states of america email: rescott3@memphis.edu   received: 23 feb. 2018   accepted: 24 apr. 2018      2018 scott. this is an open access article distributed under the terms of the creative commons‐attribution‐noncommercial‐share alike license 4.0 international (http://creativecommons.org/licenses/by-nc-sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.     doi: 10.18438/eblip29413       abstract   objective – to analyze various measures of need, participation, satisfaction, and impact of an academic library professional development program.   design – multi-modal; surveys, curriculum vitae (cv) analysis, and attendance statistics.   setting – academic library in the united states.   subjects – library faculty of all ranks.   methods – assessment of the career development program began with an interest survey conducted at the beginning of the fiscal year in which participants ranked their interest in professional development topics. attendance statistics were collected at all program sessions and participants were emailed post-event surveys comprised of three likert-scale questions and an open-ended question. participants in the peer-review service were emailed a survey with two likert-scale questions and an open-ended question. all programs and surveys were voluntary.   an “activities survey” attempted to document counts of scholarly publications and presentations according to geographic scope, format, and peer-review. however, due to low response rates, the activities survey was replaced after two years with an analysis of library faculty member cvs on a publicly-accessible university website. the final assessment was a narrative annual report that drew on and summarized all of the previously conducted assessments.   main results – multi-modal assessment of the professional development program improved its relevance and quality while also documenting its impact.   conclusion – continuous and multi-faceted assessment of professional development programs not only leads to improved efficacy, but also provides accountability and details the value of the program to stakeholders. professional development programs promote scholarly productivity, which has implications for the career satisfaction of academic librarians. further research should investigate the validity of professional development program assessment instruments and identify which assessment methods are most effective for evaluating professional development programs and measuring the impact of this programming on scholarship.   commentary   the study opens by acknowledging the increasing demands for faculty librarians to publish and their lack of preparation to do so. these findings—as well as the proposed solution of professional development programs—are well documented in the literature. vilz and poremski (2015) surveyed academic librarians regarding their perceptions of and satisfaction with support for tenure requirements and found that librarians were moderately satisfied with support mechanisms that varied considerably. sullivan et al. (2013) published one of several case studies detailing how an academic library’s professional development program was established or expanded and assessed. the study at hand is unique in its variety of interventions and the agility of the assessment.   the koufogiannakis, booth, and brettle reliant critical appraisal tool (2006) provides a useful framework for investigating the study design, the educational context, results, and relevance. all of these will be briefly considered in the subsequent paragraphs.   the objective and need for the study were clearly stated, however the exact number of participants was not. the data reported was collected over several years; accordingly, the authors did not identify the exact number of subjects or describe them. instead, “sample” data was provided from an unspecified year. a variety of research methodologies were employed in the study and the entirety of the surveys employed were provided in the article’s appendices. assessment instruments were not validated and the authors identified that as an area for future research.   the academic library setting was similar to large, public academic libraries throughout the united states. the program content was determined in part by participant ranking of their professional development needs. the learning objectives and the amount of instructional contact time for individual programs were not specified. the reported data supported the author's conclusions. “sample” results are clearly presented in tables and were positive in respect to the intervention, though not significantly so.   the reproducibility of the study as presented is impaired by the small and shifting population and multifaceted methodology. nonetheless, several aspects of the study could be employed to enhance professional development programming in similar settings. the authors succeeded in opening “a discussion on the value of continuous assessment using multiple measures” (p. 211). their model of multi-modal, iterative, and responsive professional development program assessment offers a substantial and practical contribution to the academic library literature.   references   koufogiannakis, d., booth, a., & brettle, a. (2006). reliant: reader’s guide to the literature on interventions addressing the need for education and training. library & information research, 30(94), 44-51. retrieved from http://www.lirgjournal.org.uk/lir/ojs/index.php/lir/index   sullivan, d., leong, j., yee, a., giddens, d., & phillips, r. (2013). getting published: group support for academic librarians. library management, 34(8/9), 690-704. https://doi.org/10.1108/lm-03-2013-0026   vilz, a. j., & poremski, m. d. (2015). perceptions of support systems for tenure-track librarians. college & undergraduate libraries, 22(2), 149-166. https://doi.org/10.1080/10691316.2014.924845